5. Management server software installation

This section describes the installation and configuration of software on the IPU‑POD64 management server for management and control of IPU resources and for software required to upgrade the software and firmware on the IPU-M2000.

Note

In an IPU‑POD64 with multiple servers, all servers must have the identical setup (including OS, packages, access control roles, and Poplar SDK). In addition to these, the software described in this chapter must be installed on the server nominated as the management server. The IPU-POD64 reference design contains a single server for both management and Poplar.

5.1. Overview of management server

One server in the IPU‑POD64 is nominated as the management server. In an IPU‑POD64 with more than one server, the lowest server in the rack is usually the management server. A single management server can function as a central repository for a site-wide setup of all IPU-M2000s.

The following Graphcore software packages need to be installed on the management server:

  • V-IPU software: contains management and control software for IPU resource control, testing of the communications infrastructure and monitoring of the IPU-M2000s and IPUs.

  • IPU-M software: contains the latest software and firmware for the IPU-M2000s and a utility, rack_tool, to upgrade the software and firmware on the IPU-M2000. This upgrade is performed from the management server.

Note

For large deployments, these management functions can be provided by a separate high-availability server cluster outside the IPU‑POD64. Please contact Graphcore for more details.

5.2. V-IPU software

The V-IPU software package contains a set of components that run on the management server for management and control of IPU resources.

5.2.1. V-IPU release notes

Before installing the V-IPU software, carefully read the release notes for the latest version of V-IPU that you download from the Graphcore download portal (under the IPU-POD Systems tab).

The release notes describe the following:

  • Software version numbers.

  • Compatibility changes that may need to be understood before upgrading the V-IPU software.

  • Details to any special upgrade handling for this specific release, if required.

  • An overview of fixed problems.

  • An overview of remaining known issues with proposed workarounds, if any.

5.2.2. V-IPU installation

The V-IPU software should be installed as follows:

  1. Log into the management server as the itadmin user.

    • The V-IPU controller is installed to run as a service in the context of the root user.

  2. Download the latest version of the V-IPU tarball from the Graphcore download portal (under the IPU-POD Systems tab). Save the tarball in the home directory.

  3. Extract the tarball, which contains an installation script, with:

    $ tar xvfz <tar-ball.tgz>

    An example of <tar-ball.tgz> is vipu-1.14.3.x86_64.tar.gz. The tarball is extracted into a directory named vipu-<release_version>. For the example, the directory is vipu-1.14.3.

  4. Run the installation script with the command:

    $ cd vipu-<release version>
$ sudo ./install.sh

  5. V-IPU is installed.

  6. Delete the tarball.

5.3. IPU-M software

The IPU-M software package contains the software required to upgrade the software and firmware on the IPU-M2000. The package also contains the utility rack_tool which executes the upgrade of software on the IPU-M2000s, among other rack-related tasks.

Note

You will need to install the latest version of the IPU-M software onto the management server each time you wish to upgrade the IPU-M2000s.

5.3.1. IPU-M release notes

Before installing the IPU-M software, carefully read the release notes for the latest version of IPU-M that you download from the Graphcore download portal (under the IPU-POD Systems tab).

The release notes describe the following:

  • Software sub-component version numbers.

  • Compatibility statements for Poplar SDK versions.

  • Compatibility changes from earlier releases that may need to be understood before upgrading the IPU-M2000s.

  • Details of how to handle special upgrades for this specific release, if required.

  • An overview of bug fixes.

  • An overview of improvements.

  • An overview of known issues with proposed workarounds, if any.

5.3.2. IPU-M installation

The IPU-M installation script configures rack_tool (see details in Section 5.4, Rack tool) and installs each release in its own directory under the IPU-M_releases directory (in the home directory of ipuuser):

$HOME/IPU-M_releases/IPU-M_<release version>

The IPU-M software should be installed as follows:

  1. Log in to the management server as ipuuser.

  2. Download the latest version of the IPU-M tarball from the Graphcore download portal (under the IPU-POD Systems tab). Save the tarball in the home directory.

  3. Extract the tarball, which contains an installation script, with:

    $ tar xvfz <tar-ball.tgz>

    An example of <tar-ball.tgz> is IPU_M_SW-2.2.2.tar.gz. The tarball is extracted into a directory named IPU_M_SW-<release_version>. For the example, the directory is IPU_M_SW-2.2.2.

  4. Run the installation script with the command:

    $ cd IPU_M_SW-<release_version>
$ ./install.sh

  5. This release of IPU-M is installed.

  6. Delete the tarball.

5.4. Rack tool

rack_tool is a utility that is supplied with the IPU-M software package to support execution of operations on the IPU-M2000s.

rack_tool is used to:

  1. Upgrade the IPU-Machines.

  2. Troubleshoot the rack, like perform built-in hardware tests or to check cable connectivity.

  3. Show the status of IPU-Machines in a rack.

  4. Execute operations for the BMC, like set hostname, run BMC-specific commands, set a logging server or perform a factory reset.

  5. Execute operations for the IPU-Gateway, like set hostname, run commands specific to the IPU-Gateway or set a logging server.

  6. Power on/off/cycle the IPU-Gateway and IPU.

  7. Update specific IPU-Gateway OS config files by copying a set of root file system overlay files stored on the management server to the IPU-Machines.

rack_tool is installed with the IPU-M software and is found in:

$HOME/IPU-M_releases/IPU-M_<release-version>/rack_tool.py.

The IPU-M installation script configures the shell’s command search path to the rack_tool utility, creates a symbolic link to allow the rack_tool utility to be used without the .py extension and creates the necessary rack_tool directory ($HOME/.rack_tool) and rack configuration file ($HOME/.rack_tool/rack_config.json) if these do not already exist.

For more details on specific rack_tool commands or about rack_tool in general, refer to the rack_tool man page.