3. PopRun features

3.1. Usage

The PopRun launch format is as follows:

$ poprun [options] program [program args]

You can use the --help option to get full usage information for poprun.

Warning

You should be very careful when using the --only-output-from-instance option, as this will hide potential errors from the other instances.

3.2. Tips and tricks

In this section we share tips and tricks for more advanced users.

3.2.1. Passing environment variables

Often it can be useful to pass specific environment variables to each of the instances. To pass your own custom environment variables to the various instances, simply add the following option to PopRun:

$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS=VALUE"

If you want to pass one or more environment variables to only one of the instances, this can be done using --instance-mpi-local-args=<instance-index>:<arg>. For example, this will set specified environment variables only on instances 0, 1, and 7:

$ poprun --instance-mpi-local-args=0:"-x ENV_VAR=VALUE -x PYTHONPATH" \
  --instance-mpi-local-args=1:"-x ENV_VAR=ANOTHER_VALUE -x ENV_VAR_2=VALUE2" \
  --instance-mpi-local-args=7:"-x ENV_VAR=VALUE"

3.2.2. Character escaping

If you are passing custom environmental variables and its value contains special characters such as the quote (”) character, it might be cumbersome to escape such character. An alternative is to use a technique called environment variable forwarding. It can be applied like this:

$ export POPLAR_ENGINE_OPTIONS=".."
$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS"

In the case shown above, the external value of POPLAR_ENGINE_OPTIONS is forwarded to all instances.

3.2.3. File output forwarding

Another useful feature, especially in connection with debugging, is the ability to capture <stdout> and <stderr> and forward it to separate files. Preferably, where one file corresponds to each instance. With PopRun’s file output forwarding this is possible. Simply pass the additional option to PopRun to enable this feature:

$ poprun --mpi-global-args="--output-filename output"

3.2.4. Topology table

PopRun can print a table describing the topology of the job you are launching. This can be useful to get a better understanding of how the replicas are spread across instances, IPU-Link domains (ILDs) and hosts. The table will be printed if verbose logging is enabled (for example by passing -v), and it can also be enabled explicitly by passing --print-topology=yes.

$ poprun --print-topology=yes --num-replicas=8 --num-instances=4 \
  --num-ilds=2 --host=host1,host2 --offline-mode=yes echo test
 ===========================================
|              poprun topology              |
|===========================================|
| hosts     |     host1     |     host2     |
|-----------|-------------------------------|
| ILDs      |       0       |       1       |
|-----------|-------------------------------|
| instances |   0   |   1   |   2   |   3   |
|-----------|-------------------------------|
| replicas  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
 -------------------------------------------

3.2.5. Generating auto-completion file

PopRun can generate an output that can be used for auto-completion in Bash and Zsh shells.

To generate a Bash auto-completion function or update an existing one, execute the following:

$ poprun --help --options-output-format=bash > \
  ~/.local/share/bash-completion/completions/_poprun

Reload your existing shell either by logging out and in or by executing the following command:

$ exec bash

Zsh loads its functions from files that are specified in $fpath. To generate a Zsh auto-completion function, you need to do one of the following:

Find a directory in the current $fpath that is writable.
Create a subdirectory (for example, .my-completions) under your home directory, and add it to the fpath in your .zshrc file, for example:
```
fpath=( ~/.my-completions "${fpath[@}]}" )`` .
```
Make a system-wide installation by copying the generated file to a system directory that is already in $fpath (for example /usr/share/zsh/vendor-completions or /usr/share/zsh/functions/Misc).

% poprun --help --options-output-format=zsh > \
  ~/.my-completions/_poprun

If you are installing auto-completion for the first time, you need to run compinit.

% autoload -U compinit && compinit

If you are updating an existing auto-completion, execute the following:

% unfunction _poprun && autoload -U _poprun

Note

PopRun doesn’t keep your auto-completion functions up-to-date. Therefore, it is recommended to run the above commands after updating PopRun.

Search help