3. Installation

This section describes how to install and configure the V-IPU management software. It assumes you have access to suitable IPU hardware such as a Pod.

You can download the V-IPU software tarball from the Graphcore software download portal.

For a functional V-IPU cluster the components that were introduced in Section 2, Concepts and architecture are required: the V-IPU agents, the V-IPU controller and a front-end V-IPU admin interface that is able to communicate with the V-IPU controller through the V-IPU admin API.

This section describes how and where to install these components.

Note

All the components of the V-IPU management software should be at the same version. This is something to keep in mind when upgrading the system as all the components will have to upgraded together if any component changes version.

3.1. V-IPU agents

The V-IPU agent software consists of the following components:

  • The V-IPU agent itself: an agent process that the V-IPU controller talks to

  • The V-IPU exporter: an agent for exporting critical application-monitoring metrics

  • The IPUoF server: the server-side component of IPUoF

The V-IPU agents are pre-installed in the IPU-Machines. To know which version of V-IPU agent is installed, please check the release notes for the IPU-Machine software.

The V-IPU controller and V-IPU admin client should be at the same version as the V-IPU agents.

3.2. V-IPU controller

This section describes the deployment of the V-IPU controller software which is distributed in the form of a command line utility called vipu-server. vipu-server is a statically compiled binary without external dependencies to third party libraries and should be able to run on any Linux-based system running on x86_64 or Arm 64-bit CPU architectures.

For convenience, it is recommended that you copy the binary to a location listed in the $PATH environment variable of the user who is going to execute it. No elevated privileges are required to run vipu-server, and it is strongly recommended that you run the software as a non-root user.

A shell script is provided to simplify the installation. See Section 3.4, Installation script.

See Section 4, Securing the installation for details of how to improve the security of the V-IPU software.

For complete information on the vipu-server command please refer to Section 9, Server command line reference.

3.2.1. Initialisation

The vipu-server implements the V-IPU controller, the heart of the V-IPU management system. This needs a database to maintain system state that has to live across service restarts. You must explicitly initialise the database the very first time vipu-server is executed. The supported database format is a json formatted file, called the storage file. If you haven’t initialised the storage file, vipu-server will complain with the following error message:

$ vipu-server
Storage error: open vipu-server.json: no such file or directory
    (vipu-server --storage-init may be required,
    or you may need to specify the path to the storage file vipu-server.json with --storage)

You can initialise the storage using the --storage-init option, and this must be done only once in the lifetime of the system. By default, the storage file will be initialised in the current working directory, in a file named vipu-server.json:

$ vipu-server --storage-init
Initialised storage: vipu-server.json

If you need to initialise the storage file in a different directory, you can use the --storage option. For example:

$ vipu-server --storage-init --storage /opt/vipu-server/my-vipu-server.json
Initialised storage: /opt/vipu-server/my-vipu-server.json

3.2.2. Service execution

Once you have initialised the server, the simplest way to start the service is by running the command vipu-server without any options:

$ vipu-server
{"t":"2020-10-13T16:03:19.942834Z","sv":"notice","ev":"vipu_server","tags":{"app":"vipu-server"},"detail":{"version":"1.11.0"}}
{"t":"2020-10-13T16:03:19.942926Z","sv":"notice","ev":"startup_storage","tags":{"app":"vipu-server"},"detail":{"path":"/opt/vipu-server/vipu-server.json"}}
{"t":"2020-10-13T16:03:19.943196Z","sv":"notice","ev":"listening_local","tags":{"app":"vipu-server"},"detail":{"address":"vipu-server.sock","protocol":"unix","service":"Admin-LocalSocket"}}
{"t":"2020-10-13T16:03:19.943309Z","sv":"notice","ev":"listening_remote","tags":{"app":"vipu-server"},"detail":{"address":":8090","protocol":"tcp","service":"Admin"}}
Starting GRPC server at [::]:8090
Starting GRPC server at vipu-server.sock

vipu-server supports several configuration options that can either be passed directly in the command line, through environment variables, or through a configuration file. The server will look for a configuration file in the following ordered list of directories and will load the first configuration file found:

  1. Configuration file provided with the --config option

  2. ./.vipu-server-config.hcl

  3. $HOME/.vipu-server-config.hcl

  4. /etc/vipu/vipu-server-config.hcl

Note that in order to enable some advanced features of vipu-server (such as the agent autodiscovery or V-IPU prometheus exporter) you have to explicitly configure the server to do so. Please refer to Section 9, Server command line reference to see the available options and configuration options.

3.2.3. Upgrading

Before upgrading to a newer version of vipu-server please read the changelog carefully. Since all the V-IPU system components (V-IPU agent, vipu-server and vipu-admin) should be the same version you should plan a maintenance window to upgrade all the components together.

The simplest way to upgrade vipu-server is to stop the service (if running), replace the vipu-server binary and start the service again. Note that if the storage file is not located in the current working directory you will need to point the newly started server to the correct storage file that the service was using before it was stopped.

You can check whether the database (DB) schema needs to be migrated to a new version with the following command:

$ vipu-server --check-db-migration --storage "/opt/vipu-server/vipu-server.json" --config "/etc/vipu/config.hcl"
Storage error:
  vipu-server 1.16.0 expects DB schema version(s): 6. Current vipu-server DB schema version: 5.

  *** Please read the V-IPU Admin documentation and the changelog for more details about the DB migration procedure. ***

  You can migrate your DB (one version at a time) with the following instructions:
  1. Backup existing database file: "/opt/vipu-server/vipu-server.json"
  2. Run the command as user "ipuuser": 'vipu-server --migrate-db --storage "/opt/vipu-server/vipu-server.json" --config "/etc/vipu/config.hcl"'

If the DB schema has not been migrated, vipu-server will exit immediately with the same message:

$ vipu-server --storage "/opt/vipu-server/vipu-server.json" --config "/etc/vipu/config.hcl"
Storage error:
  vipu-server 1.16.0 expects DB schema version(s): 6. Current vipu-server DB schema version: 5.

  *** Please read the V-IPU Admin documentation and the changelog for more details about the DB migration procedure. ***

  You can migrate your DB (one version at a time) with the following instructions:
  1. Backup existing database file: "/opt/vipu-server/vipu-server.json"
  2. Run the command as user "ipuuser": 'vipu-server --migrate-db --storage "/opt/vipu-server/vipu-server.json" --config "/etc/vipu/config.hcl"'

To safeguard against database corruption and to make sure it is possible to revert to a previous vipu-server version, it is your responsibility to backup the existing storage file and explicitly run the vipu-server command with the --migrate-db option to upgrade the database:

$ vipu-server --migrate-db --storage "/opt/vipu-server/vipu-server.json" --config "/etc/vipu/config.hcl"
Migration complete

# Now you can start the vipu-server as usual
$ vipu-server
{"t":"2020-10-13T16:03:19.942834Z","sv":"notice","ev":"vipu_server","tags":{"app":"vipu-server"},"detail":{"version":"1.11.0"}}
{"t":"2020-10-13T16:03:19.942926Z","sv":"notice","ev":"startup_storage","tags":{"app":"vipu-server"},"detail":{"path":"/opt/vipu-server/vipu-server.json"}}
{"t":"2020-10-13T16:03:19.943196Z","sv":"notice","ev":"listening_local","tags":{"app":"vipu-server"},"detail":{"address":"vipu-server.sock","protocol":"unix","service":"Admin-LocalSocket"}}
{"t":"2020-10-13T16:03:19.943309Z","sv":"notice","ev":"listening_remote","tags":{"app":"vipu-server"},"detail":{"address":":8090","protocol":"tcp","service":"Admin"}}
Starting GRPC server at vipu-server.sock
Starting GRPC server at [::]:8090

Note that downgrading of the database is not supported. Moreover, only single DB schema version upgrades are supported. For instance, if the current DB schema version is 3 and the new vipu-server requires version 5, you should execute the vipu-server --migrate-db command twice: once to upgrade from version 3 to version 4 and once more to upgrade from version 4 to version 5.

3.3. V-IPU admin client

To configure a V-IPU cluster, a V-IPU admin interface that is able to communicate with the V-IPU controller through the V-IPU admin API is required.

A command line utility to perform this task, called vipu-admin, is bundled in the same V-IPU distribution tarball as the vipu-server binary (discussed in Section 3.2, V-IPU controller).

vipu-admin is a statically compiled binary without external dependencies to third-party libraries and should be able to run on any Linux-based system running on x86_64 or Arm 64-bit CPU architectures by unpacking the provided V-IPU distribution tarball.

For convenience, it is recommended that you copy the binary to a location listed in the $PATH environment variable of the user who is going to execute it. No elevated privileges are required to run vipu-admin, and it is strongly recommended that you run the software as a non-root user.

vipu-admin is your primary interface to configure the V-IPU management system. It supports several configuration options that can either be passed directly on the command line, through environment variables, or through a configuration file. The command will look for a configuration file in the following ordered list of directories and will load the first configuration file found:

  1. Configuration file provided with the --config option

  2. ./.vipu-cli.hcl

  3. $HOME/.vipu-cli.hcl

  4. /etc/vipu/vipu-cli.hcl

For a complete reference to the vipu-admin command and configuration options please refer to Section 8, Admin command line reference.

Tip

vipu-admin does not need to be executed from the same host that vipu-server is running on. However, if the vipu-admin utility is located on the same host as vipu-server, a local Unix socket exposed by vipu-server can be used to access the V-IPU admin API endpoint. That allows access to the TCP/IP port for the V-IPU admin API endpoint to be blocked with firewall rules.

Note: if you block the V-IPU admin API endpoint with firewall rules you should expose the V-IPU user API endpoint with the vipu-server --listen-user ":port" option; otherwise users will not be able to use V-IPU to request resources. See Section 9, Server command line reference for more details.

3.4. Installation script

An installation script called install.sh is included with the V-IPU distribution tarball. The installation script has been tested and verified to work with Ubuntu and CentOS distros that use systemd as the default service manager.

The installation script needs to be executed with root privileges (sudo ./install.sh) as it copies the vipu-server, vipu-admin and vipu binaries to /usr/local/bin.

Note

You may need to log in as user itadmin to run the script because the default configuration for the username ipuuser does not have permission to run as root.

Also, based on your responses when executing the install.sh script, vipu-server can be configured to start at boot time as a systemd service. The script expects the user ipuuser to be present in the system, as the vipu-server systemd service is configured to be executed by ipuuser. If that user does not exist, the script will display the following error with instructions on how a different service user can be configured in the install.sh script:

$ sudo ./install.sh
This service must be started as user 'ipuuser' but 'ipuuser' does not exist.
Please either create the user 'ipuuser' or change the SERVICE_USER variable in this script.

If the ipuuser user (or the selected SERVICE_USER) is present in the system, the installation will ask you if vipu-server should be configured to run as a service in this host and which interface should be used for auto-discovery. If the answer is yes to the former and a valid interface is entered for the latter, the script will configure and start vipu-server.service:

$ sudo ./install.sh
Do you want to start the vipu-server as a service in this host?

Note that you should have vipu-server running only in one host
and use vipu/vipu-admin to connect to it from all other hosts.

(N/y) y

Choose an interface to use for agent auto-discovery:
eth0
eth1
docker0
lo

Enter disable to deactivate the auto-discovery

Which interface should be used for auto-discovery? eth0

- vipu-server will be configured to be run as a service in this host
'vipu-server' -> '/usr/local/bin/vipu-server'
'vipu' -> '/usr/local/bin/vipu'
'vipu-admin' -> '/usr/local/bin/vipu-admin'
- Initialising /etc/vipu/config.hcl
If you want to limit (or disable) the autodiscovery service please edit the
configuration file /etc/vipu/config.hcl
- Configuring vipu-server systemd service
- Enabling vipu-server systemd service to start at system boot
- Starting vipu-server service
- Use the command 'systemctl status vipu-server.service' to check the status of the service

As seen from the above output, the vipu-server database will be auto-initialised in /opt/vipu-server/vipu-server.json. The vipu-server behaviour can be tuned through the configuration file found in /etc/vipu/config.hcl. If you change the configuration of vipu-server, the systemd service should be restarted. For available configuration options please refer to Section 9, Server command line reference.

For a promptless installation, certain flags can be set when executing the installation script (see output below). Set the --yes flag to automatically create vipu-server as a service. The interface to use for auto-discovery of agents can be set with -a INTERFACE, where INTERFACE is the interface to be used (for example, -a eth0). If INTERFACE is set to disabled, the auto-discovery mechanism is disabled.

$ sudo ./install.sh --yes -a eth0

- vipu-server will be configured to be run as a service in this host
'vipu-server' -> '/usr/local/bin/vipu-server'
'vipu' -> '/usr/local/bin/vipu'
'vipu-admin' -> '/usr/local/bin/vipu-admin'
- Initialising /etc/vipu/config.hcl
If you want to limit (or disable) the autodiscovery service please edit the
configuration file /etc/vipu/config.hcl
- Configuring vipu-server systemd service
- Enabling vipu-server systemd service to start at system boot
- Starting vipu-server service
- Use the command 'systemctl status vipu-server.service' to check the status of the service

When the installation is over, and if vipu-server was configured to start as a service, you can use the vipu-admin command from the same host to configure the vipu-server without any additional configuration. If the vipu-admin command is installed in a different host, then it has to be configured to talk to the correct vipu-server with the --api-host/--api-port options (see Section 8, Admin command line reference for more details).

3.4.1. Uninstall

The installation script also allows you to uninstall components installed by the script. To invoke this option, pass the --uninstall flag to the script.

$ sudo ./install.sh --uninstall
This action will stop vipu all the associated services

Are you sure you want to proceed? (N/y) y
vipu-server removed
vipu removed
vipu-admin removed
Would you like to remove the V-IPU configuration in /etc/vipu ?  (N/y) y
config folder removed at /etc/vipu
Would you like to remove the V-IPU storage at /opt/vipu-server ?  (N/y) y
storage folder removed at /opt/vipu-server
Removed /etc/systemd/system/multi-user.target.wants/vipu-server.service.

As shown, the script will prompt you for certain actions, for instance, the removal of the configuration directory (/etc/vipu). For automation purposes, the prompt can be skipped with the --yes flag. This will automatically accept any prompts triggered.

$ sudo ./install.sh --yes --uninstall
vipu-server removed
vipu removed
vipu-admin removed
config folder removed at /etc/vipu
storage folder removed at /opt/vipu-server
Removed /etc/systemd/system/multi-user.target.wants/vipu-server.service.