3. PopRun features

3.1. Usage

The PopRun launch format is as follows:

$ poprun [options] program [program args]

3.1.1. Generic options



Default Value


Produce help message.


Print version information. See --version-format for options.


Print version information of PopRun, PopDist, and all of the dependencies (when =full-json or =full-text).



File to read configuration from. Arguments specified on the command line override values stored in this file.


File to write the current configuration to.


Allow or disallow overwriting exported configuration file.


3.1.2. Debug options



Default Value


Enables debug mode that will help diagnose problems.


3.1.3. Configuration



Default Value


A list of hosts on which to launch instances. Example: host0,host1 or host[0-2,4]. The instances are divided evenly among the hosts.


The name of a file containing a newline seperated list of hosts. Mutually exclusive with –host/–H.


The total number of replicas.



The number of IPUs per replica.



The number of instances. The replicas are divided evenly among the instances.



The number of IPU-Link domains. If not set, determined by the V-IPU server by default.


The host (hostname or IP address) of the V-IPU server.



The port number of the V-IPU server.



The timeout for the V-IPU server requests (in seconds).



Create/use the specified V-IPU partition.


The number of times PopRun will attempt to bind to the specified partition.PopRun will retry when the partition does not end up in ACTIVE state.



Update the V-IPU partition if needed.



Reset the V-IPU partition before execution. If not set, non-reconfigurable partitions are reset by default, while reconfigurable partitions are not.


Remove PopRun created V-IPU partition after execution.



Use the specified V-IPU cluster when creating a partition (deprecated, use --vipu-allocation instead).


Use the specified V-IPU allocation when creating a partition.


Use the specified directory as an executable cache path to avoid redundant compilations. If no file name is specified, executable caching will be disabled.


Global options to be passed to mpirun. Example: --tag-output to tag the output from each instance with its index.


Local options to be passed to mpirun. Example: -x ENV_VAR=VALUE to set an environment variable on all the instances.


Control process affinity by using one of the pre-defined named policies: Disabled, SpreadNuma, ScatterSocket, ScatterNuma, Close.



Local options to be passed to mpirun for a given instance. Format: <instance-index>:<arg>. Example: 0:-x ENV_VAR=VALUE to set an environment variable on instance 0.


The IPU-Link routing type to use when creating V-IPU partitions.



The GW-Link routing type to use when creating V-IPU partitions.



The sync type to use when creating V-IPU partitions.



Whether poprun should pass --tag-output to --mpi-global-args or not.



Whether poprun should pass --allow-run-as-root to --mpi-global-args or not.



Print topology table.



Suppress the output (both stdout and stderr) from all instances except the given one (zero-indexed).


Suppress the output (only stdout) from all instances except the given one (zero-indexed).


Run without requiring or attaching to any IPUs.



Specify a target used for offline compilation.


Set Poplar profiler output directory and enable profiling.


Specify a key to be distributed to all hosts. If this flag is set PopRun will terminate after distribution. If path is ommited it dafaults to $HOME/.ssh/.


If distribute-ssh-key is set, this allows for generating a key of this type to the address specified by distribute-ssh-key, if it should not exit.



The name of a file containing a newline seperated list of hosts. Mutually exclusive with –host/–H.


The Poplar SDK containing this PopRun binary will be synchronized over all hosts provided by --host or -H.



If a Python virtual environment is enabled, it will be synchronized over all hosts provided by --host or -H.



Subnet or network interface name to use for communicating with other hosts. Default: Network address of either an RNIC that can reach to all hosts or the default gateway.


Verbose output.



Very verbose output.



Perform basic input validation on the user command.



You should be very careful when using the --only-output-from-instance option, as this will hide potential errors from the other instances.

3.2. Tips and tricks

In this section we share tips and tricks for more advanced users.

3.2.1. Passing environment variables

Often it can be useful to pass specific environment variables to each of the instances. To pass your own custom environment variables to the various instances, simply add the following option to PopRun:

$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS=VALUE"

If you want to pass one or more environment variables to only one of the instances, this can be done using --instance-mpi-local-args=<instance-index>:<arg>. For example, this will set specified environment variables only on instances 0, 1, and 7:

$ poprun --instance-mpi-local-args=0:"-x ENV_VAR=VALUE -x PYTHONPATH" \
  --instance-mpi-local-args=1:"-x ENV_VAR=ANOTHER_VALUE -x ENV_VAR_2=VALUE2" \
  --instance-mpi-local-args=7:"-x ENV_VAR=VALUE"

3.2.2. Character escaping

If you are passing custom environmental variables and its value contains special characters such as the quote () character, it might be cumbersome to escape such character. An alternative is to use a technique called environment variable forwarding. It can be applied like this:

$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS"

In the case shown above, the external value of POPLAR_ENGINE_OPTIONS is forwarded to all instances.

3.2.3. File output forwarding

Another useful feature, especially in connection with debugging, is the ability to capture <stdout> and <stderr> and forward it to separate files. Preferably, where one file corresponds to each instance. With PopRun’s file output forwarding this is possible. Simply pass the additional option to PopRun to enable this feature:

$ poprun --mpi-global-args="--output-filename output"

3.2.4. Topology table

PopRun can print a table describing the topology of the job you are launching. This can be useful to get a better understanding of how the replicas are spread across instances, IPU-Link domains (ILDs) and hosts. The table will be printed if verbose logging is enabled (for example by passing -v), and it can also be enabled explicitly by passing --print-topology=yes.

$ poprun --print-topology=yes --num-replicas=8 --num-instances=4 \
  --num-ilds=2 --host=host1,host2 --offline-mode=yes echo test
|              poprun topology              |
| hosts     |     host1     |     host2     |
| ILDs      |       0       |       1       |
| instances |   0   |   1   |   2   |   3   |
| replicas  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |

3.2.5. Generating auto-completion file

PopRun can generate an output that can be used for auto-completion in the following shells:

  • bash

  • zsh

To generate a Bash auto-completion function or update an existing one, execute the following:

$ poprun --help --options-output-format=bash > \

Please reload your existing shell either by logging out and in or by executing the following command:

$ exec bash

zsh loads its functions from files that are specified in $fpath. To generate a zsh auto-completion function, you need to do one of the following:

  • Find a directory in the current $fpath that is writable by your user.

  • Create a subdirectory (for example, .my-completions) under your user’s home directory, and add it to the fpath like fpath=( ~/.my-completions "${fpath[@}]}" ) in your user’s .zshrc file.

  • Make a system-wide installation by copying the generated file in a system directory that is already in $fpath (for example /usr/share/zsh/vendor-completions or /usr/share/zsh/functions/Misc).

% poprun --help --options-output-format=zsh > \

If you are installing auto-completion function for the first time, you need to run compinit.

% autoload -U compinit && compinit

If you are updating an existing auto-completion, execute the following:

% unfunction _poprun && autoload -U _poprun


PopRun doesn’t keep your auto-completion functions up-to-date. Therefore, it is recommended to run the above commands after updating PopRun.