10. Environment variables

There are several environment variables which you can use to control the behaviour of PopART.

10.1. Logging

PopART can output information about its activity as described in Turning on execution tracing. You can control the level of logging information using environment variables.

10.1.1. POPART_LOG_LEVEL

POPART_LOG_LEVEL controls the amount of information written to the log output for all modules. Finer control can be achieved with POPART_LOG_CONFIG.

10.1.2. POPART_LOG_DEST

POPART_LOG_DEST defines the output for the logging information. The value can be stdout, stderr or a file name.

The default is stderr.

10.1.3. POPART_LOG_CONFIG

If set, POPART_LOG_CONFIG defines the name of a configuration file which specifies the logging level for each module. This is a JSON format file with pairs of module:level strings. For example, you can specify a file called conf.py by setting the environment variable:

export POPART_LOG_CONFIG=conf.py

To set the logging level of the devicex and session modules, conf.py would contain:

{
  "devicex":"INFO",
  "session":"WARN"
}

These values override the value specified in POPART_LOG_LEVEL.

10.2. Generating DOT files

PopART can output a graphical representation of the graph, in DOT format.

10.2.1. POPART_DOT_CHECKS

PopART can export DOT files of your graphs at different points. By default, no DOT files are created, but you can use the POPART_DOT_CHECKS environment variable to control what DOT files to create. POPART_DOT_CHECKS is set using predefined checkpoint names. For example to create a DOT file of all checkpoints, you would set:

export POPART_DOT_CHECKS=ALL

The predefined checkpoint names are:

FWD0: Initial IR immediately after lowering from ONNX to the IR.
FWD1: After the pre-alias patterns have been applied to FWD0.
BWD0: After growing the backward pass (including the optimiser step). Note this happens before optimiser decomposition, so the optimiser will appear as a single special op rather than the many ops that implement it.
PREALIAS: After pre-alias transforms have been applied to BWD0.
MAINLOOPS: After the MainLoops transform has been applied. This transform adds explicit loop ops to the IR for device iterations (batches per step) and gradient accumulation.
FINAL: The final IR after preparation.
The following checkpoint names only apply if you are using explicit pipelining (Note: The default is implicit pipelining). Explicit pipelining happens between MAINLOOPS and FINAL:
- EXPLICITPIPELINEBEFOREOUTLINE: Before the outline stage of the transform.
- EXPLICITPIPELINEAFTEROUTLINE: After the outline stage of the transform.
- EXPLICITPIPELINEAFTERDECOMPOSE: After the decompose stage of the transform.
ALL: All checkpoints are selected.

Multiple checkpoint names can be passed, and should be separated using “:”. For example to create a DOT file for the FWD0 and PREALIAS checkpoints:

export POPART_DOT_CHECKS=FWD0:PREALIAS

The values in POPART_DOT_CHECKS will be combined with any values that are defined in the session options.

10.3. Caching of compiled executables

It can take a long time to compile a large graph into an executable for the IPU. You can enable caching of compiled executables to avoid re-compiling the same graph every time it is run.

To enable the cache, set the environment variable POPART_CACHE_DIR to the path where the compiled executables for the PopART graph will be stored. For example:

$ export POPART_CACHE_DIR="/tmp/cachedir"

An executable binary file with the extension .popef will be saved for each Poplar graph required to execute the PopART program.

The cache does not manage the files within the directory. It is your responsibility to delete out-of-date files. No index is kept of the files, so they can be deleted without risk.

10.4. Preloading of compiled executables

Loading PopEF executable files may exhibit poor performance if the the files are stored remotely, for example on Amazon Simple Storage Service (Amazon S3) and mounted through S3FS. This is caused by making random accesses to uncached remote files.

You can optimise the loading process in such cases by enabling a preload mechanism. This can significantly improve the speed of the initial loading of the file and also enables subsequent operations to be carried out on a cached file. This makes it viable to store PopEF files on systems such as Amazon S3.

To enable the preload mechanism, set the value of the environment variable POPART_PRELOAD_POPEF to full-preload.

$ export POPART_PRELOAD_POPEF=full-preload

10.5. Inspecting the IR

You can also set an environment variable to inspect the IR.

10.5.1. POPART_IR_DUMP

If set, POPART_IR_DUMP defines the name of a file where the serialised IR will be written. For example, to write the serialised IR to the file ir_dump.log, set:

export POPART_IR_DUMP=ir_dump.log

The IR will be written either at the end of the IR preparation phase, or when an exception is thrown during the IR preparation phase.

10.6. Manipulating outliner behaviour for PopLiner

Note

It is assumed that everything in this section will only be used in conjunction with PopLiner - the guided partitioning tool.

10.6.1. POPART_POPLINER_OUTLINER_REGEX

Used to help reproduce multi-IPU outlining when compiling for a single IPU. This is helpful for use with PopLiner - the guided partitioning tool. When set, the value will be used as a regular expression for extracting the layer from the operation name. The layer must be numerical and must be within parenthesis in the regular expression for the capture group. The following value is recommended for this environment variable:

export POPART_POPLINER_OUTLINER_REGEX="(?:^|\/)(?:[L|l]ayer|blocks|encoder)[\/_\.]?(\d+)"

When set, you should see warning-level log messages to indicate which layer names are found, allowing you to check that you regular expression works as intended.

Please remember to unset this environment variable when you no longer wish to generate a profile for PopLiner.