1. Introduction

The PopVision Analysis Library (libpva) is part of the Poplar SDK and can be used for programmatic analysis of Poplar profiling information.

The library provides both C++ and Python APIs that can be used to query the Poplar profiling information.

This document describes the features of libpva and will use Python for most of the examples.

For details of the API refer to Section 2, PopVision Analysis library C++ API or Section 3, PopVision Analysis library Python API.

1.1. Capturing IPU reports

In order to use libpva you must first generate a profile of your application. The simplest way is to use the POPLAR_ENGINE_OPTIONS environment variable and set the autoReport.all option to true.

For instance to collect a profile report from my_application.py, set the environment variable and the profile.pop file will be created in the current working directory:

$ POPLAR_ENGINE_OPTIONS='{"autoReport.all":"true"}' python3 my_application.py

1.2. Opening a report

Once you have generated a report, you can open it by calling the openReport() function and passing the location of the report:

import pva

report = pva.openReport("./profile.pop")

If you are using Poplar, there is a convenience method, poplar::Engine::getReport(), that you can use to open a report. This is useful because the report has a temporary filename until the engine is destroyed, when it will be renamed to profile.pop

auto report = engine.getReport();

1.3. Reading a report

After you have opened the report you can query it for information on the compilation and execution of your application. The following sections describe common queries.

1.3.1. Reading target information

You can read information about the target from the target object. More information can be found in the description of the Target class.

numIpus = report.compilation.target.numIPUs
numReplicas = report.compilation.target.numReplicas

1.3.2. Reading memory usage by tile

You can read information about how much memory is used on each tile:

report.compilation.tiles[0].memory.total.includingGaps
report.compilation.tiles[1].memory.total.includingGaps
...

Or sum the total across all tiles:

sum(tile.memory.alwaysLiveBytes for tile in report.compilation.tiles)

1.3.3. Reading the liveness information

To look at liveness information for each program step you can iterate over the livenessProgramSteps array

for step in report.compilation.livenessProgramSteps:
    print(step)
    totalNotAlwaysLive += step.notAlwaysLiveBytes

Each step relates to a Poplar program instance. To view the details of the programs you have to create a ProgramVisitor class. The appropriate visit method will then be called depending on the type of the program.

class MyVisitor(pva.ProgramVisitor):
    def visitOnTileExecute(self, onTileExecute):
        # Details of the on tile execute
        totalNumVertices += len(onTileExecute.computeset.vertices)

    def visitDoExchange(self, doExchange):
        # Details of the exchange program
        pass


myVisitor = MyVisitor()
for step in report.compilation.livenessProgramSteps:
   step.program.accept(myVisitor)

1.3.4. Reading the execution steps

The preceding examples read information based on the compilation of an application. When an application is executed and the Poplar instrumentation is enabled then the report will also contain the execution profile. Instrumentation is enabled by default when autoReport.all is set to true.

You can iterate over the execution steps and query how many cycles each step took. A step refers to an execution of a Poplar Program object, for instance an OnTileExecute or a DoExchange program.

for step in report.execution.steps:
    # It is possible that a step could contain more than 1 compute set
    if len(step.computeSets) > 1 :
        totalCycles = sum(step.computeSets[0].cyclesByTile)
    step.program.accept(myVisitor)

Again you can use the visitor, as described above, to inspect the details of the program.

1.3.5. Debug contexts

In addition to the profile.pop file, Poplar can output a debug.cbor file which contains information about debug contexts. This file is written if the autoReport.outputDebugInfo Poplar engine option is set to true or if the autoReport.enable option is set to true. libpva can load debug.cbor alongside profile.pop and report information about debug contexts:

import pva
report = pva.openReport("profile.pop", "debug.cbor")

Debug contexts provide information about higher level operations in your application and how they map to Poplar programs. The debug contexts form a tree structure. For example, a TensorFlow operation has a corresponding debug context, each PopLibs API call generated by that TensorFlow operation also has a debug context, and the PopLibs debug contexts are children of the TensowFlow operation’s debug context.

Viewing the top-level debug contexts gives an overview of the operations carried out by the application:

# Print all top-level debug contexts, i.e. those without a parent
debugContextFilter = pva.pva_core.DebugContextFilter(False)
for debugContext in report.compilation.debugContexts(debugContextFilter):
    print(debugContext.layer)

By looking at the debug contexts of a program, the parents of those debug contexts, and so on, it is possible to see which high-level operation this program is a part of:

program = report.compilation.programs[0]
for debugContext in program.debugContexts:
    print(debugContext)

Alternatively, looking at what programs are contained within a debug context gives information about what lower-level processing is required to carry out a high-level operation:

debugContextFilter = pva.pva_core.DebugContextFilter(False)
debugContexts = report.compilation.debugContexts(debugContextFilter)
debugContext = debugContexts[0]
for program in debugContext.programs():
    print(program.type)