3. PopRun features

3.1. Usage

The PopRun launch format is as follows:

$ poprun [options] program [program args]

3.1.1. Generic options

Option

Description

Default Value

--help

Produce help message.

--version

Print version information. See --version-format for options.

--version-format

Print version information of PopRun, PopDist, and all of the dependencies (when =full-json or =full-text).

text

--import-config

File to read configuration from. Arguments specified on the command line override values stored in this file.

--export-config

File to write the current configuration to.

--overwrite-config

Allow or disallow overwriting exported configuration file.

0

3.1.2. Configuration

Option

Description

Default Value

-H

A list of hosts on which to launch instances. Example: host0,host1 or host[0-2,4]. The instances are divided evenly among the hosts.

--num-replicas

The total number of replicas.

1

--ipus-per-replica

The number of IPUs per replica.

1

--num-instances

The number of instances. The replicas are divided evenly among the instances.

1

--num-ilds

The number of IPU-Link domains. If not set, determined by the V-IPU server by default.

--vipu-server-host

The host (hostname or IP address) of the V-IPU server.

localhost

--vipu-server-port

The port number of the V-IPU server.

8090

--vipu-server-timeout

The timeout for the V-IPU server requests (in seconds).

120

--vipu-partition

Create/use the specified V-IPU partition.

--vipu-partition-attempts

The number of times PopRun will attempt to bind to the specified partition.PopRun will retry when the partition does not end up in ACTIVE state.

1

--update-partition

Update the V-IPU partition if needed.

0

--reset-partition

Reset the V-IPU partition before execution. If not set, non-reconfigurable partitions are reset by default, while reconfigurable partitions are not.

--remove-partition

Remove PopRun created V-IPU partition after execution.

1

--vipu-cluster

Use the specified V-IPU cluster when creating a partition.

--vipu-allocation

Use the specified V-IPU allocation when creating a partition.

--executable-cache-path

Use the specified directory as an executable cache path to avoid redundant compilations. If no file name is specified, executable caching will be disabled.

--mpi-global-args

Global options to be passed to mpirun. Example: --tag-output to tag the output from each instance with its index.

--mpi-local-args

Local options to be passed to mpirun. Example: -x ENV_VAR=VALUE to set an environment variable on all the instances.

--process-placement

Control process affinity by using one of the pre-defined named policies: Disabled, SpreadNuma, ScatterSocket, ScatterNuma, Close.

SpreadNuma

--instance-mpi-local-args

Local options to be passed to mpirun for a given instance. Format: <instance-index>:<arg>. Example: 0:-x ENV_VAR=VALUE to set an environment variable on instance 0.

--numa-aware

Bind instances to NUMA nodes by distributing all instances across available NUMA nodes on each host. If not set, enabled by default if there is more than one instance per host.

--ipu-link-routing-type

The IPU-Link routing type to use when creating V-IPU partitions.

IRT_UNDEFINED

--gw-link-routing-type

The GW-Link routing type to use when creating V-IPU partitions.

GRT_UNDEFINED

--sync-type

The sync type to use when creating V-IPU partitions.

ST_UNDEFINED

--tag-output

Whether poprun should pass –tag-output to –mpi-global-args or not.

1

--allow-run-as-root

Whether poprun should pass –allow-run-as-root to –mpi-global-args or not.

1

--print-topology

Print topology table.

0

--only-output-from-instance

Suppress the output (both stdout and stderr) from all instances except the given one (zero-indexed).

--only-stdout-from-instance

Suppress the output (only stdout) from all instances except the given one (zero-indexed).

--offline-mode

Run without requiring or attaching to any IPUs.

0

--autoreport-dir

Set Poplar profiler output directory and enable profiling.

-v

Verbose output.

0

--vv

Very verbose output.

0

--validate-command

Perform basic input validation on the user command.

1

Warning

You should be very careful when using the --only-output-from-instance option, as this will hide potential errors from the other instances.

3.2. Tips and tricks

In this section we share tips and tricks for more advanced users.

3.2.1. Passing environment variables

Often it can be useful to pass specific environment variables to each of the instances. To pass your own custom environment variables to the various instances, simply add the following option to PopRun:

$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS=VALUE"

If you want to pass one or more environment variables to only one of the instances, this can be done using --instance-mpi-local-args=<instance-index>:<arg>. For example, this will set specified environment variables only on instances 0, 1, and 7:

$ poprun --instance-mpi-local-args=0:"-x ENV_VAR=VALUE -x PYTHONPATH" \
  --instance-mpi-local-args=1:"-x ENV_VAR=ANOTHER_VALUE -x ENV_VAR_2=VALUE2" \
  --instance-mpi-local-args=7:"-x ENV_VAR=VALUE"

3.2.2. Character escaping

If you are passing custom environmental variables and its value contains special characters such as the quote () character, it might be cumbersome to escape such character. An alternative is to use a technique called environment variable forwarding. It can be applied like this:

$ export POPLAR_ENGINE_OPTIONS=".."
$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS"

In the case shown above, the external value of POPLAR_ENGINE_OPTIONS is forwarded to all instances.

3.2.3. File output forwarding

Another useful feature, especially in connection with debugging, is the ability to capture <stdout> and <stderr> and forward it to separate files. Preferably, where one file corresponds to each instance. With PopRun’s file output forwarding this is possible. Simply pass the additional option to PopRun to enable this feature:

$ poprun --mpi-global-args="--output-filename output"

3.2.4. Topology table

PopRun can print a table describing the topology of the job you are launching. This can be useful to get a better understanding of how the replicas are spread across instances, IPU-Link domains (ILDs) and hosts. The table will be printed if verbose logging is enabled (for example by passing -v), and it can also be enabled explicitly by passing --print-topology=yes.

$ poprun --print-topology=yes --num-replicas=8 --num-instances=4 \
  --num-ilds=2 --host=host1,host2 --offline-mode=yes echo test
 ===========================================
|              poprun topology              |
|===========================================|
| hosts     |     host1     |     host2     |
|-----------|-------------------------------|
| ILDs      |       0       |       1       |
|-----------|-------------------------------|
| instances |   0   |   1   |   2   |   3   |
|-----------|-------------------------------|
| replicas  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
 -------------------------------------------

3.2.5. Generating auto-completion file

PopRun can generate an output that can be used for auto-completion in the following shells:

  • bash

  • zsh

To generate a Bash auto-completion function or update an existing one, execute the following:

$ poprun --help --options-output-format=bash > \
  ~/.local/share/bash-completion/completions/_poprun

Please reload your existing shell either by logging out and in or by executing the following command:

$ exec bash

zsh loads its functions from files that are specified in $fpath. To generate a zsh auto-completion function, you need to do one of the following:

  • Find a directory in the current $fpath that is writable by your user.

  • Create a subdirectory (for example, .my-completions) under your user’s home directory, and add it to the fpath like fpath=( ~/.my-completions "${fpath[@}]}" ) in your user’s .zshrc file.

  • Make a system-wide installation by copying the generated file in a system directory that is already in $fpath (for example /usr/share/zsh/vendor-completions or /usr/share/zsh/functions/Misc).

% poprun --help --options-output-format=zsh > \
  ~/.my-completions/_poprun

If you are installing auto-completion function for the first time, you need to run compinit.

% autoload -U compinit && compinit

If you are updating an existing auto-completion, execute the following:

% unfunction _poprun && autoload -U _poprun

Note

PopRun doesn’t keep your auto-completion functions up-to-date. Therefore, it is recommended to run the above commands after updating PopRun.