5. Managing devices

To manage connections with IPU devices, Model Runtime defines a set of helper classes that create an abstraction of a device to allow for easier device management.

5.1. Device

The Device class is a thin wrapper around the Poplar Device. Device extends poplar::Device with the following characteristics:

IPU version declared by the user (see: ipuVersion()),
an extra step in binding to Session: if there is a Session bound to Device already, it gets unloaded from the device first (see: unloadFromDevice()).

Note

The Device class is not intended to be used directly by the user. To create a Device use DeviceManager (Section 5.2, Device Manager).

5.2. Device Manager

The DeviceManager class provides an interface for managing devices available in the user system. From a programmer’s perspective, it is a builder class that creates Device objects for the user.

Note

DeviceManager does not own the Device objects it creates. Full ownership is transferred to the user.

To create a Device suitable for the user model and representing a physical IPU, one of the following DeviceManager methods can be used:

getDevice(std::shared_ptr<popef::Model>, const DeviceWaitConfig &): the function accepts a user Model object and a DeviceWaitConfig object and returns a newly created Device.

Note

DeviceWaitConfig is a helper class that stores the configuration of how DeviceManager should wait for the specified device to become available. By default, if there are no devices suitable to run the model on, an exception is thrown. More information about DeviceWaitConfig can be found in the description of the Session constructor “wait_config” param.

DeviceManager reads the model’s Metadata to figure out the parameters of a suitable Device. If there is no suitable device available in the system, an exception is thrown.
```
// Assuming there is a model (popef::Model) prepared/loaded in advance
model_runtime::DeviceManager mgr;
auto device = mgr.getDevice(model); // acquire device suitable for the model
```
getDevice(int64_t, const poplar::OptionFlags &, const DeviceWaitConfig &, const DeviceConstraints &): is an overloaded version of getDevice. The user can choose to manually determine the set of requested Device parameters: number of IPUs requested, device options (see: Poplar OptionFlags), DeviceWaitConfig and DeviceConstraints.

Note

DeviceConstraints is a class that gathers constraints for the requested device. Currently, there is only one constraint defined - requiresRemoteBuffersSupport (see: Poplar remote memory buffers).
```
DeviceManager mgr;
const int64_t number_of_ipus = 1;
const model_runtime::DeviceConstraints constraints{true /*requiresRemoteBuffersSupport*/};
const model_runtime::DeviceWaitConfig wait_config{};
const poplar::OptionFlags options{};

std::shared_ptr<model_runtime::Device> device;

try {
  device = mgr.getDevice(number_of_ipus, options, wait_config, constraints);
} catch (std::runtime_error &err) {
  // catch and handle the exception if thrown
}
```
tryGetDevice(std::shared_ptr<popef::Model>, const DeviceWaitConfig &) and tryGetDevice(int64_t, const poplar::OptionFlags &, const DeviceWaitConfig &, const DeviceConstraints &): functions perform the same tasks as the corresponding getDevice methods, but do not throw exceptions in the case of a failure. tryGetDevice also does not check the IPU version.
```
// Assuming there is a model (popef::Model) prepared/loaded in advance
model_runtime::DeviceManager mgr;
auto device = mgr.tryGetDevice(model); // acquire device suitable for the model

if (device == nullptr) {
  // no Device object created, handle error
}
```

getSpecificDevice(int64_t, std::shared_ptr<popef::Model>, const DeviceWaitConfig &) and getSpecificDevice(int64_t, const poplar::OptionFlags &, const DeviceWaitConfig &) are other variations of the Device getter methods. Specific to these getSpecificDevice methods is their first accepted argument — device_id that represents the numerical ID of the device.

model_runtime::DeviceManager mgr;
int64_t id;
{
  model_runtime::Device device = mgr.getDevice(1);
  id = device->device().getId();
}
// should have been released when device went out of scope
std::shared_ptr<model_runtime::Device> device;
try {
  device = mgr.getSpecificDevice(id);
} catch (std::runtime_error &err) {
  // catch and handle the exception if thrown
}

tryGetSpecificDevice(int64_t, std::shared_ptr<popef::Model>, const DeviceWaitConfig &) and tryGetSpecificDevice(int64_t, const poplar::OptionFlags &, const DeviceWaitConfig &): functions perform the same tasks as the corresponding getSpecificDevice methods, but do not throw exceptions in the case of a failure.

model_runtime::DeviceManager mgr;
int64_t id;
{
  model_runtime::Device device = mgr.getDevice(1);
  id = device->device().getId();
}
// should have been released when device went out of scope
std::shared_ptr<model_runtime::Device> device = mgr.tryGetSpecificDevice(id);

if (device == nullptr) {
  // no Device object created, handle error
}

If the user does not have any IPUs available in the system, or for any other reason does not want to operate on a physical IPU, DeviceManager still can be used to create Device objects representing IPU models.

Methods dedicated for that task are createIpuModelDevice(int64_t, int64_t, int64_t) and createIpuModelDevice(std::shared_ptr<popef::Model>, int64_t).

Note

Both versions of the createIpuModelDevice function accept the tiles_per_ipu parameter as the last arguments. The value of the parameter sets up the number of tiles per IPU to be simulated. The larger this value set up, the longer the simulated computations take.

model_runtime::DeviceManager mgr;
const int64_t number_of_ipus = 1;
const int64_t ipu_version = 2;
const int64_t tiles_per_ipu = 100;
std::shared_ptr<model_runtime::Device> device =
  mgr.createIpuModelDevice(number_of_ipus, ipu_version, tiles_per_ipu);

The last two methods: createSmallIpuModelDevice(int64_t, int64_t) and createSmallIpuModelDevice(std::shared_ptr<popef::Model>) deliver shortcuts for creating a Device referring to a “small” IPU model (only 4 tiles per IPU).

model_runtime::DeviceManager mgr;
const int64_t number_of_ipus = 1;
const int64_t ipu_version = 2;
std::shared_ptr<model_runtime::Device> device =
  mgr.createSmallIpuModelDevice(number_of_ipus, ipu_version);