4. Poplar distributed configuration library (PopDist)

This section provides information on how you can use the PopDist library to make the appropriate changes to your application so that it can be launched in a distributed environment using PopRun. In short, PopDist provides an API which you can use to write a distributed application. As the examples (Section 4.3, PopDist examples) will demonstrate, very few modifications are needed in order to prepare an application for distributed execution.

PopRun uses mpirun for multiple instance creation, while PopDist uses mpi for communication between multiple hosts. This is not to be confused with communication between IPUs, which is realised using the Graphcore Communication Library (GCL) over either the IPU-Links or the GW-Links.

There are several packages that enable mpi communication in Python applications, but we recommend Horovod due to its simplicity. More details about the Horovod package can be found in the Horovod documentation.

4.1. Horovod Installation

The implementation of Horovod that you install depends on your machine learning framework:

  • TensorFlow 1 and TensorFlow 2: You do not have to install Horovod separately as it has been bundled in the Graphcore TensorFlow wheel.

  • PyTorch: You must install the official Horovod package.

  • PopART: You must install the Graphcore implementation of Horovod that is included as a wheel file with the Poplar SDK.

Note

You must install and enable the Poplar SDK following the instructions in the Getting Started guide for your IPU system. Do this before installing Horovod to ensure that it uses the OpenMPI version that comes with the SDK.

Note

It is recommended to create Python virtual environments for each framework before installing any packages.

4.1.1. TensorFlow 1 and TensorFlow 2

Horovod is bundled in the Graphcore TensorFlow wheel and so you do not have to install it separately. For instructions on how to install the TensorFlow wheel, refer to the section on setting up TensorFlow for the IPU in the Getting Started guide for your IPU system.

Once you have installed the TensorFlow wheel, you can validate that Horovod has been installed correctly by using the code snippet in Listing 4.1.

Listing 4.1 Validating that the Horovod for TensorFlow has been installed correctly.
1# Copyright (c) 2021 Graphcore Ltd. All rights reserved.
2
3import popdist
4import popdist.tensorflow
5from tensorflow.python.ipu import horovod as hvd
6
7hvd.init()

4.1.2. PyTorch

If you are using PyTorch, then you must install the official Horovod package if you plan to use mpi for host-side communication. Make sure you have activated your PyTorch virtual environment and installed PopTorch. For instructions on how to install PopTorch, refer to the Installation section in the PyTorch for the IPU: User Guide.

You can install the official Horovod package with the command:

$ pip install horovod

You can validate that Horovod has been installed correctly by using the code snippet in Listing 4.2.

Listing 4.2 Validating that the Horovod for PyTorch has been installed correctly.
1# Copyright (c) 2021 Graphcore Ltd. All rights reserved.
2
3import torch
4import poptorch
5import popdist
6import horovod.torch as hvd
7
8hvd.init()

4.1.3. PopART

If you are using PopART, then you must install the Graphcore implementation of Horovod.

$ pip install <sdk_path>/horovod.x.y.z.whl

Note

Since PopART has to use the version of Horovod that is bundled with the Poplar SDK, there is no need to validate the installation.

4.2. PopDist API

PopDist contains some universal functions which are useful in all frameworks. They can be used for doing things such as the following dynamically, based on the arguments provided to PopRun:

  • Evaluating the global batch size

  • Sharding your dataset

  • Setting the size of buffer to use for shuffling

See Section 5, PopDist C++ API reference and Section 6, PopDist Python API reference for more information.

4.3. PopDist examples

In this section we will detail how you can use PopDist to distribute your application. Examples for PyTorch and TensorFlow are shown.

4.3.1. PyTorch

The code example below outlines the most common steps involved in adding PopDist to an application. For the sake of brevity, the example code shown below assumes that the application is launched using PopRun like this:

$ poprun --vipu-partition=MyPartition --vipu-server-host=127.0.0.1
         --num-replicas=4 --num-instances=1 -v python main.py
Listing 4.3 Simple PopDist Example with PyTorch
 1# Copyright (c) 2021 Graphcore Ltd. All rights reserved.
 2
 3import torch
 4import argparse
 5import poptorch
 6import popdist
 7import popdist.poptorch
 8import horovod.torch as hvd
 9
10
11def command_line_arguments():
12    parser = argparse.ArgumentParser(description='PopDist PyTorch example')
13    parser.add_argument(
14        '--replicas', type=int, default=1, help='Number of replicas')
15    return parser.parse_args()
16
17
18def init_popdist(args):
19    hvd.init()
20    args.use_popdist = True
21    if popdist.getNumTotalReplicas() != args.replicas:
22        print(f"The number of replicas is overridden by PopRun. "
23              f"The new value is {popdist.getNumTotalReplicas()}.")
24    args.replicas = int(popdist.getNumLocalReplicas())
25
26    args.popdist_rank = popdist.getInstanceIndex()
27    args.popdist_size = popdist.getNumInstances()
28
29
30def create_model(opts):
31    if opts.use_popdist:
32        model_opts = popdist.poptorch.Options()
33    else:
34        model_opts = poptorch.Options()
35
36    return model_opts
37
38
39if __name__ == '__main__':
40
41    opts = command_line_arguments()
42
43    # Initialise PopDist
44    if popdist.isPopdistEnvSet():
45        init_popdist(opts)
46    else:
47        opts.use_popdist = False
48
49    create_model(opts)

The PopTorch PopDist package is imported on line 5-6, while Horovod is imported on line 7. Next, the PopRun command line parameters are parsed and passed to init_popdist, which is used to initialise both Horovod and PopDist.

Note

The use of Horovod is optional. It is not a requirement to initialise PopDist.

Some of the applications in our GitHub examples repository, such as our PyTorch CNNs training application make use of PopDist and PopRun.

4.3.2. TensorFlow 1

When using PopDist with TensorFlow 1, there is an additional class and function that you need to use that are not part of the PopDist API:

  • PopDistStrategy: This is a context manager for targeting the IPU in a distributed manner.

  • tf.data.Dataset.shard(): This allows you to split your dataset between instances using PopDist API methods. This is done as follows:

    dataset = dataset.shard(num_shards=popdist.getNumInstances(), index=popdist.getInstanceIndex())
    

    The data is automatically split between replicas within each instance, so no further manual splitting is required.

See the TensorFlow 1 PopDist feature example.

Our TensorFlow 1 CNNs training application also makes use of PopDist and PopRun.

4.3.3. TensorFlow 2

When using PopDist with TensorFlow 2, there is an additional class and function that you need to use that are not part of the PopDist API:

  • PopDistStrategy: This is a new context manager for targeting the IPU in a distributed manner.

  • tf.data.Dataset.shard(): This allows you to split your dataset between instances using PopDist API methods. This is done as follows:

    dataset = dataset.shard(num_shards=popdist.getNumInstances(), index=popdist.getInstanceIndex())
    

    The data is automatically split between replicas within each instance, so no further manual splitting is required.

See the TensorFlow 2 PopDist feature example.

Note

When using Keras methods such as tf.keras.Model.fit() in a multi-instance session, the progress bar that summarises the metrics and loss for each epoch is a summary across all replicas. Do not be alarmed that the logs of each instance are claiming to process the whole epoch, each instance only processes its respective dataset shard.

Note

When using tf.keras.Model() to define your model, the batch_size argument for tf.keras.Input() expects the global batch size. You do not need to explicitly provide this parameter, but if you do, be aware of this behaviour.

4.4. Conclusion

A PopDist application can be organised into the following parts:

1. Parsing phase: where PopRun runtime command line parameters are parsed and handled.

2. Initialisation phase: where much needed variables such as rank and size are stored by making PopDist API calls. If MPI is to be used between the various instances, Horovod is also initialised.

3. Digestion phase: where PopDist variables are actively used to make the application distributed.

The steps above are just rough outlines and may vary from application to application.