5. Hardware and software monitoring

The IPU hardware, and any programs running on it, can be monitored and analysed in various ways.

5.1. Command line tools

The Poplar SDK includes a number of software tools that provide information on the current status of the connected hardware. These include:

  • gc-info: determine which IPU devices are present in the system.

  • gc-monitor: passively monitor IPU activity and telemetry such as power draw, temperature and clock speed.

  • gc-reset: reset one or more IPU devices.

These are in the directory poplar-os-version/bin under the Poplar SDK directory (where os is the host operating system, and version is the current software version number).

The use of these tools is described in the IPU Command Line Tools document.

5.2. Trace output

When running a program, you have the option to output a trace from the running code which allows you to see the phases that the graph compiler uses when preparing and compiling the graph processes to execute on the IPU. Going back to the simple adder example in Section 4, Running a Poplar program on your Pod, try running:

$ POPLAR_LOG_LEVEL=DEBUG ./adder_ipu

You will see each stage of the process listed before the program output appears. The logging options are documented in the Poplar & PopLibs User Guide.

The Profiling Quick Start contains more information about outputting a trace.

5.3. Execution profiling

The PopVision Graph Analyser provides a graphical view of the graph execution trace, showing memory use, tile mapping and other vital information to help you optimise your application. For more information see the PopVision Graph Analyser User Guide.

The PopVision System Analyser provides information about the behaviour of the host-side application code. It shows an interactive graphical view of the timeline of execution steps, helping you to identify any bottlenecks between the CPUs and IPUs. For more information see the PopVision System Analyser User Guide.

You can download the Graph Analyser and System Analyser from the PopVision Tools web page. There are downloads for Ubuntu, Mac and Windows.

The Profiling Quick Start contains more information about execution profiling.