3. PopRun features

The PopRun launch format is as follows:

$ poprun [options] program [program args]

To list all the available command line options in PopRun, type:

$ poprun --help

This will print out the following table of options:



Default Value


Produce help message

-H [ --host ] arg

A comma-separated list of hosts on which to invoke instances

--num-replicas arg

The total number of replicas


--ipus-per-replica arg

The number of IPUs per replica


--num-instances arg

The number of instances

No default

--num-ilds arg

The number IPU-Link domains


--vipu-server-host arg

The host of the V-IPU server


--vipu-server-port arg

The port number of the V-IPU server


--vipu-server-timeout arg

The timeout for the V-IPU server requests (in seconds)


--vipu-partition arg

Create/use the specified V-IPU partition

No default

--update-partition arg

Update the V-IPU partition if needed


--reset-partition arg

Reset the V-IPU partition before execution. If not set, non-reconfigurable partitions are reset by default, while reconfigurable partitions are not

--remove-partition arg

Remove a V-IPU partition after execution if the --vipu-partition options was used to create a partition


--vipu-cluster arg

Use the specified V-IPU cluster when creating a partition

--executable-cache-path arg

Use the specified directory as an executable cache path

--mpi-global-args arg

Global options to be passed to mpirun

--mpi-local-args arg

Local (per-instance) options to be passed to mpirun

--numa-aware arg

Bind instances to NUMA nodes in a round-robin fashion


--print-topology arg

Print topology table


--only-output-from-instance arg

Suppress the output from all other instances

--offline-mode arg

Run without requiring or attaching to any IPUs


-v [ --verbose ]

Verbose output



Very verbose output


You should be very careful when using the --only-output-from-instance option, as this will hide potential errors from the other instances.

3.1. Tips and tricks

In this section we share tips and tricks for more advanced users.

3.1.1. Passing environment variables

Often it can be useful to pass specific environment variables to each of the instances. To pass your own custom environment variables to the various instances, simply add the following option to PopRun:

$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS=VALUE"

3.1.2. Character escaping

If you are passing custom environmental variables and its value contains special characters such as the quote () character, it might be cumbersome to escape such character. An alternative is to use a technique called environment variable forwarding. It can be applied like this:

$ poprun --mpi-local-args="-x POPLAR_ENGINE_OPTIONS"

In the case shown above, the external value of POPLAR_ENGINE_OPTIONS is forwarded to all instances.

3.1.3. Tagging

It can sometimes be useful to trace what is printed out on the screen back to its originating instance. This is particularly true when debugging parallel and distributed applications. Luckily, PopRun provides a feature called tagging, which makes it possible to map what is printed out to an associated instance. This is done by prefixing the instance identifier to the text that is output to screen. Here is a simple example:

$ [1,1]<stdout> iteration: 6396, epoch: 81.79, lr: 0.1764, loss: 2.131
$ [1,0]<stdout> iteration: 6396, epoch: 81.79, lr: 0.1764, loss: 2.131

In the code excerpt shown above, the output is piped to <stdout> is prefixed with a pair of brackets ([]). The instance identifiers appear inside the brackets.

Tagging can be enabled by simply passing the following option to PopRun:

$ poprun --mpi-global-args="--tag-output"

3.1.4. File output forwarding

Another useful feature, especially in connection with debugging, is the ability to capture <stdout> and <stderr> and forward it to separate files. Preferably, where one file corresponds to each instance. With PopRun’s file output forwarding this is possible. Simply pass the additional option to PopRun to enable this feature:

$ poprun --mpi-global-args="--output-filename output"