4. Model handling

Each framework is capable of exporting and importing programs designed for the IPU architecture. PopEF was created to unify and facilitate this process. This chapter describes the general model export and import workflows and links to the relevant information for each framework.

The Poplar SDK supports three high-level frameworks that allow you to create, export, import and execute a model. These are PyTorch for the IPU, TensorFlow for the IPU and PopART.

4.1. Exporting models

4.1.1. Overview

Each framework must be able to save its state after the model compilation is complete, so that it can be restored at a later time. To make this possible, it is necessary to save such elements as:

• a serialised Poplar executable

• its associated metadata

• tensor data blobs if model parameters have not been frozen (which means are not constant and saved inside the PopEF file)

• a framework-specific opaque blob to store information only relevant to that framework.

The general export process is shown in Fig. 4.1.

Fig. 4.1 Overview of export process

4.1.2. How to export

Exported models are stored as PopEF files. Each framework supports pre-compilation of executables for the IPU on a machine that doesn’t necessarily have access to IPUs.

Instructions for exporting models in the frameworks supported by the Poplar SDK are:

• PyTorch for the IPU:

  • Enable caching allows you to implicitly export a model to the specified directory after the compilation process.

  • Precompilation allows you to to compile the model and run it later. This is useful for when you don’t have immediate access to a machine with IPUs.

  • Explicit API functions for exporting a model:

    • poptorch.PoplarExecutor.compileAndExport()

    • poptorch.PoplarExecutor.save()

• PopART:

  • C++ API:

    • Implicit export of a model to the specified directory with popart::SessionOptions is possible using two options:

      • popart::SessionOptions::enableEngineCaching

      • popart::SessionOptions::cachePath

    • Explicit export is possible using:

      • popart::Session::compileAndExport()

      • popart::Session::saveExecutable()

      • popart::Session::saveVariables()

  • The same operations are also available in the PopART Python API

• TensorFlow for the IPU:

  • Export and import in TensorFlow 1

  • Export and import in TensorFlow 2

  • Export model for TensorFlow Serving (TensorFlow 1)

  • Export model for TensorFlow Serving (TensorFlow 2)

4.1.3. Poplar executable

The Poplar library allows you to create programs that are executed on an IPU. You can create a PopEF file using the poplar::Executable::serialize() function.

Note

In this case, you must prepare the Metadata structure yourself and any other data for the program, such as tensor data blobs, to make the PopEF data complete.

4.2. Importing models

As with exporting models, all frameworks also provide model import functionality.

Instructions for importing models into the frameworks supported by the Poplar SDK are:

• PyTorch for the IPU:

  • Implicitly importing a model is possible by setting the caching option. Details are provided in the How to export section.

  • Explicit model import is available using two functions:

    • poptorch.load()

    • poptorch.PoplarExecutor.loadExecutable()

• PopART only provides an implicit way of importing a model by setting proper options through popart::SessionOptions. Details are provided in the How to export section.

• For TensorFlow, details are given in the How to export section.

• You can also import and run precompiled models using:

  • Poplar Triton Backend See how configure a model repository.

  • IPU TensorFlow Serving for TensorFlow 1

  • IPU TensorFlow Serving for TensorFlow 2

  • Model Runtime

4.2.1. Model management

To facilitate the management of imported models, PopEF provides two classes: Model and ModelBuilder. They let you bind blobs from PopEF files to models. Based on the data from the Model class, it is possible to run the model using the Poplar library. To do this, you must first deserialize the contents of the blob the model contains. This is possible with the poplar::Executable::deserialize() function. The return value is an object of type poplar::Executable that can be used to create a poplar::Engine object. Using the remaining data from Model and the poplar::Engine object, you can run the model yourself.