5. Integration with Slurm

The section describes integration of the V-IPU with Slurm. Slurm is a popular open source cluster management and job scheduling system. The integration of V-IPU with Slurm is provided through a custom V-IPU resource selection plugin for Slurm systems.

For more details about Slurm and it’s architecture, please refer to the Slurm website.

A Slurm plugin is a dynamically linked code object providing customized implementation of well-defined Slurm APIs. Slurm plugins are loaded at runtime by the Slurm libraries and the customized API callbacks are called at appropriate stages.

Resource selection plugins are a type of Slurm plugins which implement the Slurm resource/node selection APIs. The resource selection APIs provide rich interfaces to allow for customized selection of nodes for jobs, as well as performing any tasks needed for preparing the job runs (such as partition creation in our case), and appropriate clean-up code at job termination (such as partition deletion in our case).

A preview of the V-IPU Slurm plugin is available by request. Please contact Graphcore support if you are interested.

5.1. Configuring Slurm to use V-IPU select plugin

Note

This document assumes that you have access to pre-compiled Slurm binaries with the V-IPU plugin support or you have already patched and recompiled your Slurm installation with the V-IPU support.

In order to enable V-IPU resource selection in Slurm, you need to configure the SelectType as select/vipu in the Slurm configuration. The V-IPU Slurm plugin is a layered plugin, which means it can enable V-IPU support for existing resource selection plugins. Options pertaining to selected secondary resource selection plugins can be specified under SelectTypeParameters.

The following is an example of the Slurm configuration enabling the V-IPU resource selection plugin layered on top of a consumable resource allocation plugin (select/cons_res) with the CPU as a consumable resource:

SelectType=select/vipu
SelectTypeParameters = other_cons_res,CR_CPU

Note the other_ prefix added to the layered resource selection plugin’s name. In the same way, other_cons_tres, and other_linear can also be configured. For SelectTypeParameters supported by each of the existing resource selection plugin, refer to the Slurm documentation.

5.2. Configuration parameters

Configuration parameters for the V-IPU resource selection plugin are set in a separate configuration file, vipu.conf, that needs to be in the same directory as slurm.conf.

The following configuration options are supported:

  • ApiHost: The host name or IP address for the V-IPU controller.

  • ApiPort: The port number for the V-IPU controller. Default port is 8090.

  • IpuofDir: The directory where IPUoF configuration files for user jobs will be stored.

  • MaxIpusPerJob: Maximum IPUs allowed per job. Default value is 256.

  • ApiTimeout: Timeout in seconds for the V-IPU client. The default value is 50.

The V-IPU resource selection plugin also supports secure communication with mTLS between the V-IPU controller and the plugin. For setting up that option, IsSecure needs to be enabled in the vipu.conf. Please note that V-IPU controller must have been configured to run in the secure mode for this option to work.

The following parameters are needed for secure mode:

  • ApiUser: The user name of the API user that the plugin will use.

  • ApiAccessKey: Unique access key assigned to the API user.

  • ClientCertFile: Client certificate to be used for the TLS.

  • ClientKeyFile: Client key file to be used for the TLS.

  • ServerCertFile: Server certificate to be used for the TLS.

5.3. Job submission and parameters

The V-IPU resource selection plugin supports the following options:

  • --ipus: Number of IPUs requested for the job

  • -n / --ntasks: Number of tasks for the job. This will correspond to the number of GCDs requested for the job partition.

  • --num-replicas: Number of model replicas for the job.

These parameters can be configured in both sbatch and srun scripts as well as provided on the command line:

$ sbatch --ipus=2 --ntasks=1 --num-replicas=1 myjob.batch

5.3.1. Job script examples

The following is an example of a single gcd job script:

#!/bin/bash
#SBATCH  --job-name single-gcd-job
#SBATCH --ipus 2
#SBATCH -n 1
#SBATCH --time=00:30:00

srun ./ipu_program.sh

wait

You can configure a multi-GCD job in the same way, except for indicating the number of GCDs requested by setting the number of tasks:

#!/bin/bash
#SBATCH --job-name multi-gcd-job
#SBATCH --ipus 2
#SBATCH -n 2
#SBATCH --time=00:30:00

srun ./ipu_program.sh

wait