3. PopRun features

3.1. Usage

The PopRun launch format is as follows:

$ poprun [options] program [program args]

You can use the --help option to get full usage information for poprun.

Warning

You should be very careful when using the --only-output-from-instance option, as this will hide potential errors from the other instances.

3.2. Tips and tricks

In this section we share tips and tricks for more advanced users.

3.2.1. Passing environment variables

Often it can be useful to pass specific environment variables to each of the instances. To pass your own custom environment variables to the various instances, simply add the following option to PopRun:

$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS=VALUE"

If you want to pass one or more environment variables to only one of the instances, this can be done using --instance-mpi-local-args=<instance-index>:<arg>. For example, this will set specified environment variables only on instances 0, 1, and 7:

$ poprun --instance-mpi-local-args=0:"-x ENV_VAR=VALUE -x PYTHONPATH" \
  --instance-mpi-local-args=1:"-x ENV_VAR=ANOTHER_VALUE -x ENV_VAR_2=VALUE2" \
  --instance-mpi-local-args=7:"-x ENV_VAR=VALUE"

3.2.2. Character escaping

If you are passing custom environmental variables and its value contains special characters such as the quote () character, it might be cumbersome to escape such character. An alternative is to use a technique called environment variable forwarding. It can be applied like this:

$ export POPLAR_ENGINE_OPTIONS=".."
$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS"

In the case shown above, the external value of POPLAR_ENGINE_OPTIONS is forwarded to all instances.

3.2.3. File output forwarding

Another useful feature, especially in connection with debugging, is the ability to capture <stdout> and <stderr> and forward it to separate files. Preferably, where one file corresponds to each instance. With PopRun’s file output forwarding this is possible. Simply pass the additional option to PopRun to enable this feature:

$ poprun --mpi-global-args="--output-filename output"

3.2.4. Topology table

PopRun can print a table describing the topology of the job you are launching. This can be useful to get a better understanding of how the replicas are spread across instances, IPU-Link domains (ILDs) and hosts. The table will be printed if verbose logging is enabled (for example by passing -v), and it can also be enabled explicitly by passing --print-topology=yes.

$ poprun --print-topology=yes --num-replicas=8 --num-instances=4 \
  --num-ilds=2 --host=host1,host2 --offline-mode=yes echo test
 ===========================================
|              poprun topology              |
|===========================================|
| hosts     |     host1     |     host2     |
|-----------|-------------------------------|
| ILDs      |       0       |       1       |
|-----------|-------------------------------|
| instances |   0   |   1   |   2   |   3   |
|-----------|-------------------------------|
| replicas  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
 -------------------------------------------

3.2.5. Generating auto-completion file

PopRun can generate an output that can be used for auto-completion in Bash and Zsh shells.

To generate a Bash auto-completion function or update an existing one, execute the following:

$ poprun --help --options-output-format=bash > \
  ~/.local/share/bash-completion/completions/_poprun

Reload your existing shell either by logging out and in or by executing the following command:

$ exec bash

Note

PopRun doesn’t keep your auto-completion functions up-to-date. Therefore, it is recommended to run the above commands after updating PopRun.