9. Admin command line reference

The vipu-admin command line utility has the following syntax:

vipu-admin [GlobalOptions] Command ResourceType Arguments [LocalOptions]

Command: Specifies the operation that are to be executed on one or more resources. The commands supported are listed in Table 9.1.
ResourceType: Specifies the resource type on which the given command is to be executed. The resource types are described in Table 9.2. The commands supported for each resource type are listed in Table 9.3. Note that not all resource types are supported for all commands.
Options: Specify options on the command line to provide additional information globally or for specific commands. The global options are listed in Table 9.4. The local options are shown in the description for each command.
Arguments: Specify resource names or other information supplied to the command line. For example, the name of the agent (“myagent”) in the command below:
```
$ vipu-admin create agent myagent --host myhost --port 8080
```

Table 9.1 Command summary
Command	Description
bash-completion	Generate a Bash completion script
create	Create entities Create an agent Create an allocation Create a cluster Create a partition Create a user
discover	Discover entities Discover agents
get	Get info about an entity Get info for an agent Get allocation Get info for a cluster Get info for a partition
list	List entities List connected agents List allocations List available clusters List IPUs List partitions in an allocation List active users
remove	Remove entities Remove an agent Remove an allocation Remove a cluster Remove a partition Remove a user
rename	Rename entities Rename an agent Rename a cluster
reset	Reset entities Reset a partition
test	Test entities Test a cluster
zsh-completion	Generate a Zsh completion script

Table 9.2 Resource types
Resource	Description
agent	A V-IPU agent running on an IPU-Machine (see Section 6.2.1, Agent entity).
user	A user of the V-IPU software (see Section 5.1, Users).
cluster	A V-IPU cluster. The IPUs in a single cluster are capable of directly communicating and synchronising (see Section 6.2.2, Clusters).
ipu	Represents a single IPU in an IPU-Machine.
allocation	An allocation represents a set of IPUs bundled together (see Section 5.2, Allocations).
partition	A partition defines a set of IPUs within a cluster that provides an isolated domain for running user applications (see Section 6.2.4, Partition).

Table 9.3 Commands for each resource type
Resource type	Description
Agent	Create an agent Discover agents Get info for an agent List connected agents Remove an agent Rename an agent
Allocation	Create an allocation Get allocation List allocations Remove an allocation
Cluster	Create a cluster Get info for a cluster List available clusters Remove a cluster Rename a cluster Test a cluster
IPU	List IPUs
Partition	Create a partition Get info for a partition List partitions in an allocation Remove a partition Reset a partition
User	Create a user List active users Remove a user

9.1. Global options

Global options work with all vipu commands. You can also pass global options as environment variables or in a config file.

Table 9.4 Global options
Option	Description
`--api-access-key string`	API access key
`--api-host string`, `-H string`	Controller hostname or IP address
`--api-port int`, `-P int`	Controller port number (default 8090)
`--api-user-id string`	API user ID
`--config string`	Choose a configuration file explicitly; do not look in the default locations
`--help`, `-h`	Show help for the `vipu-admin` command
`--quiet`, `-q`	Set quiet mode for the utility
`--secure`	Connect to the V-IPU controller in secure mode
`--server-socket string` `-S string`	Choose the local Unix socket to connect to the V-IPU controller. If this option is used the `--api-host` option is ignored
`--server-version`	Show version of the remote server
`--showjson`	Select JSON output
`--timeout int`	Set the time-out for client calls in seconds (default 1600)
`--tls-cert-dir string`	Directory for the TLS certificates (default `./vipu-certs`)
`--version`, `-v`	Show version

9.1.1. Using a configuration file

All the global options listed in Section 9.1, Global options can be overridden with a vipu-cli configuration file. The option names in the configuration file are identical but without the the leading hyphens. For example, if you use vipu-admin with the following command from the shell:

$ vipu-admin --api-host my-host --api-port 8100 list agents

The corresponding vipu-admin options can be overridden in a configuration file that looks like this:

// vipu-admin utility will try to connect to the admin API endpoint at myhost:8100
api-host = "my-host"
api-port = "8100"

The command line utilities (vipu-admin and vipu) will look for a configuration file in the following ordered list of directories and will load the first configuration file found:

Configuration file provided with the --config option
./.vipu-cli.hcl
$HOME/.vipu-cli.hcl
/etc/vipu/vipu-cli.hcl

9.1.2. Using environment variables

All the options listed in Section 9.1, Global options can be overridden using environment variables using the syntax: VIPU_CLI_OPTION_NAME.

Everything should be upper case without the leading hyphens. The remaining hyphens (-) should be replaced by underscores (_). For example, if you run vipu-admin with the following command from the shell:

$ vipu-admin --api-host my-host --api-port 8100 list agents

The corresponding vipu-admin options can be overridden with the following environment variables:

$ export VIPU_CLI_API_HOST="my-host"
$ export VIPU_CLI_API_PORT="8100"

9.2. Script generation

9.2.1. Generate a Bash completion script

$ ./vipu-admin bash-completion [options]

To configure your Bash shell to load completions for each session you can add the following line at the end of your ~/.bashrc file:

$ source <(vipu-admin bash-completion)

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.2.2. Generate a Zsh completion script

$ ./vipu-admin zsh-completion [options]

Generates zsh completion scripts.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.3. Create entities

Create entities in a V-IPU controller.

9.3.1. Create an agent

$ ./vipu-admin create agent AGENT_NAME [options]

Add an agent in the V-IPU controller.

Supported options

Option	Description
`--exporter-metrics-path string`	Provide the metrics path for the exporter associated with this agent (default “/metrics”). See Section 7, V-IPU monitoring.
`--exporter-port uint16`	Provide the port for the exporter associated with this agent (default 2112). See Section 7, V-IPU monitoring.
`--force`, `-f`	Creates agent without validating connection and agent platform
`--help`, `-h`	Help with syntax and options
`--host string`	Specify the host for the agent
`--port uint32`	Provide the port for the agent (default 8080)

9.3.2. Create an allocation

$ ./vipu-admin create allocation ALLOCATION_NAME [options]

Create an allocation in the V-IPU controller. Users can create partitions from the allocations that are assigned to them (see Section 5.2, Allocations) or an administrator can create partitions for use by users.

Supported options

Option	Description
`--cluster string`	Specify a cluster ID
`--help`, `-h`	Help with syntax and options
`--index int`	Choose an allocation from the list returned by `--options`
`--max-allocation-options uint32`	Specify the maximum number of allocation options to be returned. Used in conjunction with `--options` (0 returns all options, default 10)
`--options`	Get a list of all possible allocations
`--size int`	Desired size (number of IPUs) of the new allocation (must be either 4, 8, 16, 32 or a multiple of 64 IPUs)

9.3.3. Create a cluster

$ ./vipu-admin create cluster CLUSTER_NAME [options]

Create a cluster in the V-IPU controller.

Supported options

Option	Description
`--agents strings`	Specify a comma separated list of agents
`--cluster-topology string`	Specify the GW-Link topology: `looped` (default) or `switched`
`--create-default-allocation boolean`	Create a default allocation upon successful creation of the cluster (default: true)
`--help`, `-h`	Help with syntax and options
`--num-ipulinkdomains int`	Specify the number of IPU-Link domains when creating cluster by agents (default 1)
`--topology string` `-t string`	Specify the IPU-Link topology: `mesh` (default) or `torus`

Warning

For backward compatibility with older V-IPU installations, a default allocation with the same name as the cluster will be created unless --create-default-allocation=false. This default allocation ensures that users and administrators can create partitions without the need to create allocations explicitly. With V-IPU 1.18, this behaviour will change and allocation will have to be created explicitly.

9.3.4. Create a partition

$ ./vipu-admin create partition [PARTITION_NAME] [options]

Create a partition in the V-IPU controller. Users can create partitions from the allocations that are assigned to them (see Section 5.2, Allocations) or an administrator can create partitions for use by users.

IPUoF configuration file

In order to communicate with the Pod, Poplar needs a configuration file with details of how to connect to the IPU-Machines. This file is created when you create a partition. The default location is the directory $HOME/.ipuof.conf.d/. You can change the directory using the --ipuof-config-location option (see Section 9.3.4, Create a partition).

There must only be one configuration file in this directory. If Poplar finds more than one file, then it will throw an exception.

By default, the automatically-created file is named partition-name.conf but Poplar will read any file that it finds in that location.

If you, as an administrator, create partitions for users, then you might want to put a symbolic link to the configuration file in their $HOME/.ipuof.conf.d directory. By making sure that the user has only read access to the file and the symbolic link you can securely manage access to the IPU resources.

The file contains information about:

The number of IPUs assigned to this user
Networking details for how to connect to those IPUs

For example:

{
  "devices": [
    {
      "ip": "10.1.5.1",
      "server_port": 50052,
      "rdma_port": 7471,
      "device_id": 3,
      "ipulink_segment_id": 0,
      "routing_id": 0,
      "ipu_arch_id": 2,
      "ipu_id": 2
    }
  ],
  "attributes": {
    "IPU version": "ipu2",
    "Routing Type": "DNC"
  }
}

There is a single device (which may correspond to one or more IPUs) in the “devices” array that can be accessed from the specified IP address (10.1.5.1 in this example). The rest is additional information used by IPUoF to connect to the server-side IPUoF component (see Section 3.1, V-IPU agents).

Supported options

Option	Description
`--all`	Show all possible partitions, including those that have already been used
`--allocation string`	Specify an allocation ID (create the partition in any of the allocations when not specified)
`--help`, `-h`	Help with syntax and options
`--index int`	Choose a partition from the list returned by `--options`
`--ipuof-config-attrs boolean`	Enable or disable writing attributes to IPUoF configuration file (default true)
`--ipuof-config-enable boolean`	Enable or disable writing IPUoF configuration file (default true)
`--ipuof-config-location string`, `-s string`	Directory to write IPUoF configuration file to (default `$HOME/.ipuof.conf.d`)
`--ipuof-config-ports boolean`	Enable or disable support for IPUoF with non-default ports (default true)
`--max-partition-options uint32`	Specify the maximum number of partition options to be returned. Used in conjunction with `--options` (0 returns all options, default 10)
`--num-gcds uint32`	Specify the number of GCDs in the partition (default 1)
`--options`	Get a list of all possible partitions that could be created
`--reconfigurable`	Enables reconfiguration for fabric multi-IPU partitions
`--relocatable`	Enables relocatable flag for the partition. A relocatable partition can be relocated to a different set of IPUs by the V-IPU controller, if needed.
`--routing string`	Routing to use for IPU-Links: “dnc” (default), “btnc”, “swnc” or “ringswnc”
`--size int`	Desired size (number of IPUs) of the new partition (must be a power of 2)
`--sync-type string`	Sync type for the partition: “pod-native-default” (default), “c2-compatible” (default if `--reconfigurable` is set), “independent-rails”, “pod-native-repgraph”
`--total-num-replicas uint32`	Specify the total number of replicas across all GCDs (default: 1 replica per GCD)

9.3.5. Create a user

$ ./vipu-admin create user USER_NAME [options]

Create a user in the V-IPU controller.

Supported options

Option	Description
`--adminuser`	Specify that the new user is an admin user
`--allocations strings`	Specify a comma separated list of allocations the user has access to
`--default-allocation string`	Specify the default allocation the user has access to (use the first one in the list of allocations when not specified)
`--dont-write-user-config-file`	Specify that the user V-IPU config file will not be written
`--help`, `-h`	Help with syntax and options
`--user-config-file string`	Specify the file to write the user configuration to (default is `USERID.vipu-cli.hcl`)

9.4. Discover entities

Discover entities from the V-IPU controller.

9.4.1. Discover agents

$ ./vipu-admin discover agents [options]

With no options, this will list all the agents discovered.

Supported options

Option	Description
`--auto-add`	Automatically add all agents discovered (see Agent auto-discovery)
`--help`, `-h`	Help with syntax and options

9.5. List entities

List entities from the V-IPU controller.

9.5.1. List connected agents

$ ./vipu-admin list agent [options]

List connected agents in the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.5.2. List allocations

$ ./vipu-admin list allocation [options]

List allocations.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.5.3. List available clusters

$ ./vipu-admin list cluster [options]

List available clusters in the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.5.4. List IPUs

$ ./vipu-admin list ipu [options]

List IPUs available on the agents.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.5.5. List IPU attributes

$ ./vipu-admin list ipu-attrs [options]

List IPU attributes from all agents.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.5.6. List partitions in an allocation

$ ./vipu-admin list partition [options]

List partitions in an allocation

Supported options

Option	Description
`--allocation string`	Specify an allocation ID (list the partitions in all allocations when not specified)
`--help`, `-h`	Help with syntax and options

9.5.7. List active users

$ ./vipu-admin list user [options]

List active users in the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.6. Get info about an entity

Get information about an entity from the V-IPU controller.

9.6.1. Get info for an agent

$ ./vipu-admin get agent AGENT_ID [options]

Get information for an agent in the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options
`--ipu-attributes`	Show all IPU attributes

9.6.2. Get allocation

$ ./vipu-admin get allocation [options]

Get information about an allocation.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.6.3. Get info for a cluster

$ ./vipu-admin get cluster CLUSTER_ID [options]

Get information for a cluster in the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.6.4. Get info for a partition

$ ./vipu-admin get partition PARTITION_ID [options]

Get information for a partition in the V-IPU controller.

Supported options

Option	Description
`--gcd id`	Only output information for the specified GCD.
`--ipuof-configs`	Outputs only IPUoF configuration information. The IPUoF information for each GCD is output in JSON format (as used in IPUoF configuration files). For a multi-GCD partition, the output will only be valid JSON if the `--gcd id` option is also used.
`--help`, `-h`	Help with syntax and options.

9.7. Remove entities

Remove entities from V-IPU controller.

9.7.1. Remove an agent

$ ./vipu-admin remove agent AGENT_NAME [options]

Remove an agent from the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.7.2. Remove an allocation

$ ./vipu-admin remove allocation ALLOCATION_NAME [options]

Remove an allocation

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.7.3. Remove a cluster

$ ./vipu-admin remove cluster CLUSTER_NAME [options]

Remove a cluster from the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.7.4. Remove a partition

$ ./vipu-admin remove partition PARTITION_NAME [options]

Remove a partition in the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options
`-s string` `--ipuof-config-location string`	Directory to check for removing IPUoF config files (default `$HOME/.ipuof.conf.d`)
`--force`, `-f`	Force the removal of the partition ignoring errors

9.7.5. Remove a user

$ ./vipu-admin remove user USER_NAME [options]

Remove a user from the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.8. Rename entities

Rename entities.

9.8.1. Rename an agent

$ ./vipu-admin rename agent CURRENT_AGENT_NAME NEW_AGENT_NAME [options]

Rename an agent from the V-IPU controller.

Note

The renamed agent must exist and not be part of any partition nor allocation.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.8.2. Rename a cluster

$ ./vipu-admin rename cluster CURRENT_CLUSTER_NAME NEW_CLUSTER_NAME [options]

Rename a cluster from the V-IPU controller.

Note

The renamed cluster must exist and not contain any partition nor allocation.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options

9.9. Reset entities

Reset entities on V-IPU controller.

9.9.1. Reset a partition

$ ./vipu-admin reset partition PARTITION_NAME [options]

Reset a partition in the V-IPU controller.

Supported options

Option	Description
`--all`	Reset all partitions
`--relocatable`	Enable relocatable flag for the partition. Existing flag will be overridden.
`--help`, `-h`	Help with syntax and options

9.10. Test entities

Test entities from V-IPU controller. See Section 6.4, Cluster tests for more details of the tests performed.

9.10.1. Test a cluster

$ ./vipu-admin test cluster CLUSTER_NAME [options]

Run tests on a cluster configured in the V-IPU controller.

Supported options

Option	Description
`--help`, `-h`	Help with syntax and options
`--cabling`	Run a OSFP cabling verification test
`--gwlink`	Run GW-Link test
`--gw-traffic`	Run GW traffic test
`--ipulink`	Run IPU-Link test
`--pfc-settings`	Check whether agent RNIC PFC settings are as expected
`--start`	Dispatch an asynchronous cluster test request to the server
`--status`	Dispatch an asynchronous cluster test request to the server
`--stop`	Dispatch an asynchronous cluster test request to the server
`--sync`	Run sync-link test
`--traffic`	Run traffic test
`--versions`	Run a version consistency test and report if mismatched versions of same components are found

Search help

9. Admin command line reference

9.1. Global options

9.1.1. Using a configuration file

9.1.2. Using environment variables

9.2. Script generation

9.2.1. Generate a Bash completion script

Supported options

9.2.2. Generate a Zsh completion script

Supported options

9.3. Create entities

9.3.1. Create an agent

Supported options

9.3.2. Create an allocation

Supported options

9.3.3. Create a cluster

Supported options

9.3.4. Create a partition

IPUoF configuration file

Supported options

9.3.5. Create a user

Supported options

9.4. Discover entities

9.4.1. Discover agents

Supported options

9.5. List entities

9.5.1. List connected agents

Supported options

9.5.2. List allocations

Supported options

9.5.3. List available clusters

Supported options

9.5.4. List IPUs

Supported options

9.5.5. List IPU attributes

Supported options

9.5.6. List partitions in an allocation

Supported options

9.5.7. List active users

Supported options

9.6. Get info about an entity

9.6.1. Get info for an agent

Supported options

9.6.2. Get allocation

Supported options

9.6.3. Get info for a cluster

Supported options

9.6.4. Get info for a partition

Supported options

9.7. Remove entities

9.7.1. Remove an agent

Supported options

9.7.2. Remove an allocation

Supported options

9.7.3. Remove a cluster

Supported options

9.7.4. Remove a partition

Supported options

9.7.5. Remove a user

Supported options

9.8. Rename entities

9.8.1. Rename an agent

Supported options

9.8.2. Rename a cluster

Supported options

9.9. Reset entities

9.9.1. Reset a partition

Supported options

9.10. Test entities

9.10.1. Test a cluster

Supported options