8. Admin command line reference

The vipu-admin command line utility has the following syntax:

vipu-admin [GlobalOptions] Command ResourceType Arguments [LocalOptions]

  • Command: Specifies the operation that are to be executed on one or more resources. The commands supported are listed in Table 8.1.

  • ResourceType: Specifies the resource type on which the given command is to be executed. The resource types are described in Table 8.2. The commands supported for each resource type are listed in Table 8.3. Note that not all resource types are supported for all commands.

  • Options: Specify options on the command line to provide additional information globally or for specific commands. The global options are listed in Table 8.4. The local options are shown in the description for each command.

  • Arguments: Specify resource names or other information supplied to the command line. For example, the name of the agent (“myagent”) in the command below:

    $ vipu-admin create agent myagent --host myhost --port 8080
    
Table 8.1 Command summary

Command

Description

bash-completion

create

Create entities

discover

Discover entities

get

Get info about an entity

list

List entities

remove

Remove entities

rename

Rename entities

reset

Reset entities

test

Test entities

zsh-completion

Table 8.2 Resource types

Resource

Description

agent

A V-IPU agent running on an IPU-Machine (see Section 6.2.1, Agent entity).

user

A user of the V-IPU software (see Section 5.1, Users).

cluster

A V-IPU cluster. The IPUs in a single cluster are capable of directly communicating and synchronising (see Section 6.2.2, Clusters).

ipu

Represents a single IPU in an IPU-Machine.

allocation

An allocation represents a set of IPUs bundled together (see Section 5.2, Allocations).

partition

A partition defines a set of IPUs within a cluster that provides an isolated domain for running user applications (see Section 6.2.4, Partitions).

Table 8.3 Commands for each resource type

Resource type

Description

Agent

Allocation

Cluster

IPU

Partition

User

8.1. Global options

Global options work with all vipu commands. You can also pass global options as environment variables or in a config file.

Table 8.4 Global options

Option

Description

--api-access-key string

API access key

--api-host string, -H string

Controller hostname or IP address

--api-port int, -P int

Controller port number (default 8090)

--api-user-id string

API user ID

--config string

Choose a configuration file explicitly; do not look in the default locations

--help, -h

Show help for the vipu-admin command

--quiet, -q

Set quiet mode for the utility

--secure

Connect to the V-IPU controller in secure mode

--server-socket string -S string

Choose the local Unix socket to connect to the V-IPU controller. If this option is used the --api-host option is ignored

--server-version

Show version of the remote server

--showjson

Select JSON output

--timeout int

Set the time-out for client calls in seconds (default 1600)

--tls-cert-dir string

Directory for the TLS certificates (default ./vipu-certs)

--version, -v

Show version

8.1.1. Using a configuration file

All the global options listed in Section 8.1, Global options can be overridden with a vipu-cli configuration file. The option names in the configuration file are identical but without the the leading hyphens. For example, if you use vipu-admin with the following command from the shell:

$ vipu-admin --api-host my-host --api-port 8100 list agents

The corresponding vipu-admin options can be overridden in a configuration file that looks like this:

// vipu-admin utility will try to connect to the admin API endpoint at myhost:8100
api-host = "my-host"
api-port = "8100"

The command line utilities (vipu-admin and vipu) will look for a configuration file in the following ordered list of directories and will load the first configuration file found:

  1. Configuration file provided with the --config option

  2. ./.vipu-cli.hcl

  3. $HOME/.vipu-cli.hcl

  4. /etc/vipu/vipu-cli.hcl

8.1.2. Using environment variables

All the options listed in Section 8.1, Global options can be overridden using environment variables using the syntax: VIPU_CLI_OPTION_NAME.

Everything should be upper case without the leading hyphens. The remaining hyphens (-) should be replaced by underscores (_). For example, if you run vipu-admin with the following command from the shell:

$ vipu-admin --api-host my-host --api-port 8100 list agents

The corresponding vipu-admin options can be overridden with the following environment variables:

$ export VIPU_CLI_API_HOST="my-host"
$ export VIPU_CLI_API_PORT="8100"

8.2. Script generation

8.2.1. Generate a Bash completion script

$ ./vipu-admin bash-completion [options]

To configure your Bash shell to load completions for each session you can add the following line at the end of your ~/.bashrc file:

$ source <(vipu-admin bash-completion)

Supported options

Option

Description

--help, -h

Help with syntax and options

8.2.2. Generate a Zsh completion script

$ ./vipu-admin zsh-completion [options]

Generates zsh completion scripts.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.3. Create entities

Create entities in a V-IPU controller.

8.3.1. Create an agent

$ ./vipu-admin create agent AGENT_NAME [options]

Add an agent in the V-IPU controller.

Supported options

Option

Description

--exporter-metrics-path string

Provide the metrics path for the exporter associated with this agent (default “/metrics”). See Section 7, V-IPU monitoring.

--exporter-port uint16

Provide the port for the exporter associated with this agent (default 2112). See Section 7, V-IPU monitoring.

--force, -f

Creates agent without validating connection and agent platform

--help, -h

Help with syntax and options

--host string

Specify the host for the agent

--port uint32

Provide the port for the agent (default 8080)

8.3.2. Create an allocation

$ ./vipu-admin create allocation ALLOCATION_NAME [options]

Create an allocation in the V-IPU controller. Users can create partitions from the allocations that are assigned to them (see Section 5.2, Allocations) or an administrator can create partitions for use by users.

Supported options

Option

Description

--cluster string

Specify a cluster ID

--help, -h

Help with syntax and options

--index int

Choose an allocation from the list returned by --options

--max-allocation-options uint32

Specify the maximum number of allocation options to be returned. Used in conjunction with --options (0 returns all options, default 10)

--options

Get a list of all possible allocations

--size int

Desired size (number of IPUs) of the new allocation (must be either 4, 8, 16, 32 or a multiple of 64 IPUs)

8.3.3. Create a cluster

$ ./vipu-admin create cluster CLUSTER_NAME [options]

Create a cluster in the V-IPU controller.

Supported options

Option

Description

--agents strings

Specify a comma separated list of agents

--cluster-topology string

Specify the GW-Link topology: looped (default) or switched

--create-default-allocation boolean

Create a default allocation upon successful creation of the cluster (default: true)

--help, -h

Help with syntax and options

--num-ipulinkdomains int

Specify the number of IPU-Link domains when creating cluster by agents (default 1)

--topology string -t string

Specify the IPU-Link topology: mesh (default) or torus

Warning

For backward compatibility with older V-IPU installations, a default allocation with the same name as the cluster will be created unless --create-default-allocation=false. This default allocation ensures that users and administrators can create partitions without the need to create allocations explicitly. With V-IPU 1.18, this behaviour will change and allocation will have to be created explicitly.

8.3.4. Create a partition

$ ./vipu-admin create partition [PARTITION_NAME] [options]

Create a partition in the V-IPU controller. Users can create partitions from the allocations that are assigned to them (see Section 5.2, Allocations) or an administrator can create partitions for use by users.

IPUoF configuration file

In order to communicate with the Pod, Poplar needs a configuration file with details of how to connect to the IPU-Machines. This file is created when you create a partition. The default location is the directory $HOME/.ipuof.conf.d/. You can change the directory using the --ipuof-config-location option (see Section 8.3.4, Create a partition).

There must only be one configuration file in this directory. If Poplar finds more than one file, then it will throw an exception.

By default, the automatically-created file is named partition-name.conf but Poplar will read any file that it finds in that location.

If you, as an administrator, create partitions for users, then you might want to put a symbolic link to the configuration file in their $HOME/.ipuof.conf.d directory. By making sure that the user has only read access to the file and the symbolic link you can securely manage access to the IPU resources.

The file contains information about:

  • The number of IPUs assigned to this user

  • Networking details for how to connect to those IPUs

For example:

{
  "devices": [
    {
      "ip": "10.1.5.1",
      "server_port": 50052,
      "rdma_port": 7471,
      "device_id": 3,
      "ipulink_segment_id": 0,
      "routing_id": 0,
      "ipu_arch_id": 2,
      "ipu_id": 2
    }
  ],
  "attributes": {
    "IPU version": "ipu2",
    "Routing Type": "DNC"
  }
}

There is a single device (which may correspond to one or more IPUs) in the “devices” array that can be accessed from the specified IP address (10.1.5.1 in this example). The rest is additional information used by IPUoF to connect to the server-side IPUoF component (see Section 3.1, V-IPU agents).

Supported options

Option

Description

--all

Show all possible partitions, including those that have already been used

--allocation string

Specify an allocation ID (create the partition in any of the allocations when not specified)

--help, -h

Help with syntax and options

--index int

Choose a partition from the list returned by --options

--ipuof-config-attrs boolean

Enable or disable writing attributes to IPUoF configuration file (default true)

--ipuof-config-enable boolean

Enable or disable writing IPUoF configuration file (default true)

--ipuof-config-location string, -s string

Directory to write IPUoF configuration file to (default $HOME/.ipuof.conf.d)

--ipuof-config-ports boolean

Enable or disable support for IPUoF with non-default ports (default true)

--max-partition-options uint32

Specify the maximum number of partition options to be returned. Used in conjunction with --options (0 returns all options, default 10)

--num-gcds uint32

Specify the number of GCDs in the partition (default 1)

--options

Get a list of all possible partitions that could be created

--reconfigurable

Enables reconfiguration for fabric multi-IPU partitions

--relocatable

Enables relocatable flag for the partition. A relocatable partition can be relocated to a different set of IPUs by the V-IPU controller, if needed.

--routing string

Routing to use for IPU-Links: “dnc” (default), “btnc”, “swnc” or “ringswnc”

--size int

Desired size (number of IPUs) of the new partition (must be a power of 2)

--sync-type string

Sync type for the partition: “pod-native-default” (default), “c2-compatible” (default if --reconfigurable is set), “independent-rails”, “pod-native-repgraph”

--total-num-replicas uint32

Specify the total number of replicas across all GCDs (default: 1 replica per GCD)

8.3.5. Create a user

$ ./vipu-admin create user USER_NAME [options]

Create a user in the V-IPU controller.

Supported options

Option

Description

--adminuser

Specify that the new user is an admin user

--allocations strings

Specify a comma separated list of allocations the user has access to

--default-allocation string

Specify the default allocation the user has access to (use the first one in the list of allocations when not specified)

--dont-write-user-config-file

Specify that the user V-IPU config file will not be written

--help, -h

Help with syntax and options

--user-config-file string

Specify the file to write the user configuration to (default is USERID.vipu-cli.hcl)

8.4. Discover entities

Discover entities from the V-IPU controller.

8.4.1. Discover agents

$ ./vipu-admin discover agents [options]

With no options, this will list all the agents discovered.

Supported options

Option

Description

--auto-add

Automatically add all agents discovered (see Agent auto-discovery)

--help, -h

Help with syntax and options

8.5. List entities

List entities from the V-IPU controller.

8.5.1. List connected agents

$ ./vipu-admin list agent [options]

List connected agents in the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.5.2. List allocations

$ ./vipu-admin list allocation [options]

List allocations.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.5.3. List available clusters

$ ./vipu-admin list cluster [options]

List available clusters in the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.5.4. List IPUs

$ ./vipu-admin list ipu [options]

List IPUs available on the agents.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.5.5. List IPU attributes

$ ./vipu-admin list ipu-attrs [options]

List IPU attributes from all agents.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.5.6. List partitions in an allocation

$ ./vipu-admin list partition [options]

List partitions in an allocation

Supported options

Option

Description

--allocation string

Specify an allocation ID (list the partitions in all allocations when not specified)

--help, -h

Help with syntax and options

8.5.7. List active users

$ ./vipu-admin list user [options]

List active users in the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.6. Get info about an entity

Get information about an entity from the V-IPU controller.

8.6.1. Get info for an agent

$ ./vipu-admin get agent AGENT_ID [options]

Get information for an agent in the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

--ipu-attributes

Show all IPU attributes

8.6.2. Get allocation

$ ./vipu-admin get allocation [options]

Get information about an allocation.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.6.3. Get info for a cluster

$ ./vipu-admin get cluster CLUSTER_ID [options]

Get information for a cluster in the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.6.4. Get info for a partition

$ ./vipu-admin get partition PARTITION_ID [options]

Get information for a partition in the V-IPU controller.

Supported options

Option

Description

--gcd id

Only output information for the specified GCD.

--ipuof-configs

Outputs only IPUoF configuration information. The IPUoF information for each GCD is output in JSON format (as used in IPUoF configuration files). For a multi-GCD partition, the output will only be valid JSON if the --gcd id option is also used.

--help, -h

Help with syntax and options.

8.7. Remove entities

Remove entities from V-IPU controller.

8.7.1. Remove an agent

$ ./vipu-admin remove agent AGENT_NAME [options]

Remove an agent from the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.7.2. Remove an allocation

$ ./vipu-admin remove allocation ALLOCATION_NAME [options]

Remove an allocation

Supported options

Option

Description

--help, -h

Help with syntax and options

8.7.3. Remove a cluster

$ ./vipu-admin remove cluster CLUSTER_NAME [options]

Remove a cluster from the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.7.4. Remove a partition

$ ./vipu-admin remove partition PARTITION_NAME [options]

Remove a partition in the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

-s string --ipuof-config-location string

Directory to check for removing IPUoF config files (default $HOME/.ipuof.conf.d)

--force, -f

Force the removal of the partition ignoring errors

8.7.5. Remove a user

$ ./vipu-admin remove user USER_NAME [options]

Remove a user from the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.8. Rename entities

Rename entities.

8.8.1. Rename an agent

$ ./vipu-admin rename agent CURRENT_AGENT_NAME NEW_AGENT_NAME [options]

Rename an agent from the V-IPU controller.

Note

The renamed agent must exist and not be part of any partition nor allocation.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.8.2. Rename a cluster

$ ./vipu-admin rename cluster CURRENT_CLUSTER_NAME NEW_CLUSTER_NAME [options]

Rename a cluster from the V-IPU controller.

Note

The renamed cluster must exist and not contain any partition nor allocation.

Supported options

Option

Description

--help, -h

Help with syntax and options

8.9. Reset entities

Reset entities on V-IPU controller.

8.9.1. Reset a partition

$ ./vipu-admin reset partition PARTITION_NAME [options]

Reset a partition in the V-IPU controller.

Supported options

Option

Description

--all

Reset all partitions

--relocatable

Enable relocatable flag for the partition. Existing flag will be overridden.

--help, -h

Help with syntax and options

8.10. Test entities

Test entities from V-IPU controller. See Section 6.4, Cluster tests for more details of the tests performed.

8.10.1. Test a cluster

$ ./vipu-admin test cluster CLUSTER_NAME [options]

Run tests on a cluster configured in the V-IPU controller.

Supported options

Option

Description

--help, -h

Help with syntax and options

--cabling

Run a OSFP cabling verification test

--gwlink

Run GW-Link test

--gw-traffic

Run GW traffic test

--ipulink

Run IPU-Link test

--pfc-settings

Check whether agent RNIC PFC settings are as expected

--start

Dispatch an asynchronous cluster test request to the server

--status

Dispatch an asynchronous cluster test request to the server

--stop

Dispatch an asynchronous cluster test request to the server

--sync

Run sync-link test

--traffic

Run traffic test

--versions

Run a version consistency test and report if mismatched versions of same components are found