2. User guide

2.1. Overview

The Graphcore PopVision™ System Analyser is a desktop tool for analysing the execution of IPU-targeted software on your host system processors. It shows an interactive timeline visualisation of the execution steps involved, helping you to identify any bottlenecks between the CPUs and IPUs. This is particularly useful when you are scaling models to run on multiple CPUs and IPUs.

Used in combination with the PopVision™ Graph Analyser application, the System Analyser allows you to identify exactly how long execution events take to run, from the main program itself all the way down to individual IPU execution steps.

Poplar - and the machine learning frameworks it supports such as PopART, TensorFlow and PyTorch - use the Poplar libpvti library to capture profiling information from your code. This information is saved to a file which you can then open and analyse in the System Analyser application. User APIs in C++ and Python are available to instrument your own application code.

Note

The majority of the functionality in the System Analyser app requires at least Poplar SDK 1.4, but there are some features (for example line graphs and metadata) that only become available in later SDK versions. These features are highlighted within this help document.

2.1.1. End User Licence Agreement

Before you can use the System Analyser, you must first agree to the End User License Agreement (EULA) that is displayed when the app is first opened (assuming that you have not already agreed to it in a previous version). Clicking on the ‘Disagree’ button will quit the application immediately.

You can re-read the EULA at any time after you’ve agreed to it. Select ‘View EULA’ from the Help menu.

You can toggle the EULA dialog between modal and full-screen view by clicking on the icon to the left of the window’s title.

2.2. Capturing execution information

To capture execution data from your program to a file, use the following options when executing your program. These are specified at the same time as the POPLAR_ENGINE_OPTIONS :

PVTI_OPTIONS='{"enable":"true"}'

Additional options include:

directory - to specify where the .pvti file will be saved,
channels - to specify a list of channels to capture from, as described Using the libpvti API , below.

Note

.pvti files are streamed to disk as the program executes, so it should not have a significant effect on your host system’s memory (although it may affect the speed of execution).

The pvti.hpp file, in the libpvti library in Poplar, gives more details.

2.2.1. Capturing function entry and exit

For most basic cases, there are POPLAR_TRACEPOINT macros that can be used inside your program to capture the timings of function entry and exit. For example:

void Engine::prepareForStreamAccess(StreamAndIndex stream) const {
  POPLAR_TRACEPOINT();
  const DataStream &streamInfo = getStreamInfo(stream.id);
  logging::engine::debug("prepareForStreamAccess {} \\"{}\\" (id {}, index {}))",
                          isHostToDevice(streamInfo.type) ? "host write of",
                                                          : "host read of",
                          streamInfo.handle, stream.id, stream.index);
}

This will capture the name of the method, and, using object construction and destruction, record the entry and exit time.

Similar macros are available for PopART ( POPART_TRACEPOINT ) and TensorFlow ( TENSORFLOW_TRACEPOINT ).

2.2.2. Using the libpvti API

The API allows you to ‘mark’ the begin and end of a ‘section of code’. This is not limited to functions - markers can be placed anywhere in your code. More information on the instrumentation API can be found in the Poplar libpvti.hpp header file documentation & README.

To use the API to indicate the beginning and end of trace events, you must first create a channel, as the example below shows. In C++:

// Create channel
pvti::TraceChannel channel = {"MyChannel"};

// Functional implementation
void foo() {
  pvti::TracePoint::begin(channel, "foo");
  ...
  pvti::TracePoint::end(channel, "foo");
}

// Scoped tracepoint object
void bar() {
  pvti::Tracepoint tp(channel, "bar");
  ...
}

And in Python:

import libpvti as pvti
channel = pvti.createTraceChannel("MyChannel")

# Functional implementation
def foo ():
  pvti.Tracepoint.begin(channel, "foo")
  ...
  pvti.Tracepoint.end(channel, "foo")

# context manager implementation
def bar():
  with pvti.Tracepoint(channel, "bar"):
    ...

# wrapped object
class Bob:
  def somemethod(self):
    ...

bob = Bob()
pvti.instrument(bob ["somemethod"], channel)

# decorator (later)
@pvti_instrument()
def cat():
  ...

2.2.3. Capturing scalar values

The API also allows you to capture scalar values over time. This can be used to log any scalar value, potential uses could be to capture host memory usage or CPU load. More information on the instrumentation API can be found in the Poplar libpvti.hpp header file documentation & README.

To use the API to capture scalar values, you must first create a graph (with units) and then a series against that graph, as the example below shows. In C++:

// Create graph
pvti::Graph graph("MyGraph", "%");

// Create series
auto series1 = graph.addSeries("series1");
auto series2 = graph.addSeries("series2");

// Capture values
series1.add(0.1);
series2.add(0.3);

And in Python:

import libpvti as pvti
# Create graph
graph = pvti.Graph("MyGraph", "%")

# Create series
series1 = graph.addSeries("series1")
series2 = graph.addSeries("series2")

# Capture values
series1.add(0.1)
series2.add(0.3)

You can capture default driver monitoring information by setting the environment variable GCDA_MONITOR=1 .

2.3. Opening reports

After starting the System Analyser, you’re presented with a ‘landing page’ from which you can open reports and view various topics within this online help. You can open report files on your local machine, or from a remote server over SSH. The application remembers which tab you last used (‘local’ or ‘remote’) and selects it automatically whenever the Open dialog is displayed.

If you’ve previously opened a report, it will appear in the Recent list of report files. Click on one to open it again.

Note

You can open a .pvti file in the System Analyser by double-clicking it in your OS’s file browser. (Not supported on Linux.)

2.3.1. Opening the Demo Report

If you don’t have any reports to hand, you can open and browse the demo report that’s included with the System Analyser. It’s a small report, but contains all of the features that are supported in the current application.

Click on the ‘Open Demo Report’ button on the main screen to open the demo report.

2.3.2. Opening recent reports

If you’ve used the application before, the Home screen displays a list of recently opened reports in the ‘Recent’ box.

Click on a recent report to open it again. That report will automatically move to the top of the Recent list.
If you want to remove a report from the Recent list, click on the ‘trash’ icon that appears to the left of a report when you hover the mouse over it.
If you attempt to open a report file from the Recent reports list that has been moved or deleted since it was last opened, an error dialog appears indicating that the file can’t be found. You can either replace the file, in which case the corresponding link in the Recent box will work again, or you can click on the ‘Remove from recent files’ button in the error dialog to remove it from the Recent list.

The Recent list can contain up to 64 previous file selections.

2.3.3. Local reports

You can open report files stored on your local machine as described below.

To open a local report on your machine:

Click on the Open Report link. You’ll be presented with a file selection dialog, and the ‘local’ tab at the top will be selected by default. You’ll see listings of the directories and files on your local machine.
The System Analyser application can open .pvti files and .json files that support the Chromium trace format. When the PopVision™ System Analyser identifies a directory in which any .pvti files are found, those files are listed on the right-hand side.
When typing a path into the box at the top of the dialog, a drop-down list shows the directories within the current directory. You can use the keyboard up and down arrows to navigate this list, and choose the next path element by pressing the Return key.
You can sort these files by name or modified date, in ascending or descending order, by clicking on the appropriate column header. Your sorting preference is saved.
You can toggle the Open dialog between modal and full-screen view by clicking on the icon to the left of the window’s title.
After navigating to the desired directory, you can select a file by clicking on it. Multiple files can be selected at once.
Once you’ve selected the file(s) you wish to open, click on the ‘Open’ button to load the report data from the file(s).

While the report file(s) load into the application, you’ll see a ‘Loading’ progress bar, then the main view is displayed, as detailed below .

Note

While you have a report open, you can add other reports to it. Click the ‘Open’ button in the top left-hand corner of the application.

2.3.4. Remote reports

If you are using an IPU system on a remote server, for example on a cloud service, any reports generated will be saved to that server, so you cannot open them ‘locally’. You can, however, open them remotely by specifying the server address, and connecting to the machine over SSH. The reports are analysed and the output is streamed back to the PopVision™ System Analyser application on your local machine, allowing you to view the reports.

Note

When the PopVision™ System Analyser opens report files on a remote machine, it downloads a small binary app to it which pre-processes the report data and sends it back over SSH to the PopVision™ System Analyser application running on your local machine. If you’re running other performance-critical processes on that remote machine, you should be aware of any effects this process may have on the capacity of the remote machine’s hardware to run any other tasks. As server performance varies a great deal, the only way to know how much processor speed it takes is to try a small sample, and monitor the CPU usage.

To open a remote report on another machine:

Click on the Open Report link. You’ll be presented with a file selection dialog, and the ‘local’ tab at the top will be selected by default.
Click on the ‘remote’ tab at the top, and you’ll see a login dialog that allows you to connect to a remote server. Enter your username, and the address of the remote machine.
If you just want to log in with a password for the remote machine, enter it in the Password field.
Alternatively, you can use your local machine’s SSH key to authorise your connection. Enter its file path in the Preferences dialog .
Once logged in, you’ll see listings of the directories and files on the remote machine. You can sort these files by name, modified date or size, in ascending or descending order, by clicking on the appropriate column header.
The System Analyser application can open .pvti files and .json files that support the Chromium trace format. Multiple files can be opened together from this dialog by clicking to select all the files you want to open. You’ll notice that when the PopVision™ System Analyser identifies a directory in which any .pvti files are found, those files are listed on the right-hand side.
Once you’ve selected the file(s) you wish to open, click on the ‘Open’ button to load the report data from the file(s).

If you’ve previously opened a report, it will appear in the Recent list of report files. Click on one to open it again.

While the report file(s) load into the application, you’ll see a ‘Loading’ progress bar, then the main view is displayed, as detailed below .

Note

The System Analyser does not currently support encrypted SSH private keys, i.e. keys that are protected by a passphrase. However it does support SSH agents. If your key is passphrase protected you will need to make sure to add it to your SSH agent before the PopVision™ System Analyser can use it, by using the ssh-add command-line tool and ensure ‘SSH Agent mode’ is set correctly in the Preferences.

To configure SSH Agent, from a terminal you can run the following.

# Start the ssh-agent in the background.
eval "$(ssh-agent -s)"

# Add your SSH private key to the ssh-agent
ssh-add -K ~/.ssh/id_rsa

Then edit your Preferences to remove your SSH private key path. Make sure that SSH agent mode is set to “Automatically obtain ssh-agent socket path from environment”.

2.3.5. Connection Errors

A number of errors can occur when connecting to a remote server. This section lists the most common and gives some troubleshooting steps.

Error

Error: getaddrinfo ENOTFOUND server.example.com

This error occurs when the specified server could not be found (DNS lookup failed). Check that you have typed the server’s name correctly. If a VPN connection is required, check that it is connected and working correctly.

Error

Error: Password authentication failed

If SSH agent and SSH key authentication fail then password authentication is attempted. If you normally use a public key to connect to the server, check that you have correctly specified the key in SSH preferences . Otherwise, check that password authentication is enabled on the server and that you have typed your password correctly.

Error

Error: Cannot create directory ‘.cache/popvision_system_analyser’ because a file already exists there.

PopVision™ System Analyser will attempt to create a number of directories on the remote server if they do not already exist. If a file already exists with the same name then attempting to create the directory will fail. Check what the file is and either delete or rename it to allow the directory to be created.

Error

Error: Could not create directory ‘.cache’: Permission denied

PopVision™ System Analyser will attempt to create a number of directories on the remote server if they do not already exist. This error indicates that your user did not have permission to create one of these directories. This usually indicates a problem with how your home directory is setup on the server: either you do not have a home directory and don’t have permission to create one, or you have not been given adequate permissions on your own home directory. Contact the server administrator to ask for a home directory to be created or for its permissions to be corrected.

Error

Error: Could not write to ‘.cache/popvision_system_analyser/backend_…’ on the remote. This could be caused by a full filesystem

This error usually occurs because there was not enough disk space on the server to upload the required binary file to your home directory (around 20MB is required). You may be able to free up some space by deleting unused files.

In some cases servers have a home directory filesystem which is very limited in size but have a much larger local disk or “scratch space” available. PopVision™ can upload to the scratch space if you create a symbolic link from your home directory to the scratch drive, for example:

mv ~/.cache /localdata/username/.cache
ln -s /localdata/username/.cache ~/.cache

2.4. Viewing reports

The application’s main view displays the Timeline information of the execution events recorded in the files you opened, together with a scaled-down overview above, which shows the entire set of events irrespective of your current zoom and pan state.

In the main window, the following actions are available:

Pan and zoom in and out of the timeline, viewing every event in the execution of the program.
Select individual events to view their details, and view the durations of selected sections of the timeline.
Search for events by entering an event name to match in the Search box.
Save report images to your machine.

2.4.2. Timeline flamegraphs

Events in the timeline are grouped and layered according to the file where they originated, and beneath that to the process and thread in which they occurred. If there’s room to display it, the event’s name is shown within each event block. Hovering your mouse over an event block displays a pop-up that shows the details of the event, as described below .

The number of events within the currently displayed portion of the timeline is displayed in the top left-hand corner, as well as the duration of that portion. You’ll see these numbers change as you pan and zoom around the timeline.

The time scale is displayed across the top of the timeline, showing elapsed time from the start of the first event. This is displayed in hours and minutes and seconds, and more significant digits of the seconds are displayed as you zoom in. You can choose to display the time scale in relative terms (starting at 0:00 at the beginning of the first captured event), or in absolute terms, where a real time is displayed. Choose the display you want by clicking the Options button at the top of the application window.

Note

Once you’ve opened a report, you can open additional reports to display on the same timeline. Just click the Add file button at the top-left of the application window, and add files as above .

Events in the timeline are coloured as follows:

Poplar - events triggered from the Poplar libraries are coloured red.
Framework - events triggered from the PopART libraries, or any of the machine learning frameworks such as TensorFlow, are coloured orange.
Driver - events triggered by the driver layer are coloured blue.
Others - user-generated event categories are then assigned a different colour.

2.4.3. Viewing instant PVTI events

Instant PVTI events are now displayed in the timeline as small ‘lollipop’ icons above the main timeline event blocks. Instant events are set in your program code using, for example:

pvti.Tracepoint.event(channelName, "Here's an instant event")

This event will then show up on the timeline when that command was encountered as the program executes.

Click on the event to reveal its details in the tabs below the timeline display.

2.4.4. Viewing zero-length blocks

PVTI files may contain sub-microsecond events which are rounded down to zero duration. To make these visible on the timeline, we have set their length to 0.5µs (providing they do not overlap the next event), and changed their label to read “<1µs”.

2.4.5. Timeline line graphs

If any scalar values have been captured, the timeline will also show a collection of line graphs grouped at the bottom of a file. These can be used to view these scalar values over time, and compare to the events shown in the flamegraphs above.

Hovering over a line graph will display a tooltip giving each separate series’ value at that timestamp.

You can change whether to display a line graph as a line or as a filled-in area:

Click on the Line Graph Type drop-down list at the top of the screen to select which type you’d like displayed.

Note

This feature is only available in reports generated with Poplar SDK 2.1 and later.

IPU Utilisation

The data used to populate the IPU Utilisation line graph is sourced from the drivers when the GCDA MONITORING option is enabled. It is the average amount of time spent waiting for one or more IPU syncs in the previous second. This measurement is a rough guide, and is not intended to be used for performance measurement.

See the online documentation for the gc-monitor command for more details of the data provided by the drivers.

Note

IPU Utilisation is not the same as the derived IPU execution, which shows when each IPU is active, as described here .

2.4.6. Timeline options

Above the timeline overview there are the following buttons:

Add file - this reopens the file browsing dialog, allowing you to add more files to the current timeline as above .
Graph Type - this allows you to switch between graph display types:
- Timeline - displays a flamegraph showing the position and duration of each event in the call stack. Each event on the timeline view corresponds directly to an event logged in the PVTI file.
- Aggregated - displays a flamegraph showing each event aggregated by its unique call stack. The starting timestamp at which a block appears in this view has no link to the timestamp that its events took place at within the trace. However, its duration does correspond to the total duration of the events that combine to make the block.
Options - this shows the following options to apply to your timeline:
- Absolute timing - checking this option will position files on your timeline using their absolute times rather than positioning all files to start at time 0.
- Collapse all charts - checking this option will collapse all charts in your timeline, it will leave parent nodes in their current state. Unchecking this will similarly expand all charts in your timeline.
- Derive IPU Execution - checking this option will display an additional chart at the top of the timeline. It shows intervals of IPU activity that may be derived from host synchronisation events in the other charts.
- Hide charts below threshold duration - checking this option will hide all charts in your timeline with a total duration (sum of all top-level event durations) less than or equal to the threshold percentage of the total time-window of the file. You can specify the threshold percentage using the radio buttons below the option.
Key - this allows you to see which channels the different colours represent on your timeline. Clicking one of these will reload the timeline without that channel shown.
Save (camera icon) - this allows you to save the current view as a PNG file. This can either be saved to a location on your machine, or to your clipboard.

2.4.7. Panning and zooming

You can move around the timeline using the mouse to bring events into view:

In the overview at the top of the timeline, click and drag a section to zoom into the corresponding are of the timeline. The section of the timeline you’re currently viewing is highlighted in the overview.
Drag the mouse left and right in the timeline to shift it left and right at the current zoom level.
Use the mouse wheel to zoom in and out of the timeline. If the timeline is too deep to fit into the application window, a scrollbar is displayed to enable you to move the timeline up and down in the window. As the mouse wheel is used for zooming in and out, you can hold down the Control key to scroll the timeline up and down using the mouse wheel.
Note that you can choose how you want your scroll wheel to behave by setting the Scroll behaviour setting preference in the Preferences dialog.

Note

You can switch between window and full-screen display by selecting Toggle Full screen from the View menu, or by pressing Control/Command and F.

2.4.8. Using the Navigation panel

In the top left-hand corner of the report graph, there is a small compass icon that, when clicked, opens out into a navigation panel that allows you to move around the report easily with the keyboard, or if you don’t have a scrolling mouse or mouse pad.

Note

When you hover your mouse over each icon in the navigation panel, you can see the equivalent quick-key for that action.

Click the compass icon to open the navigation panel. You can click it again to close it, or click on the < icon at the panel’s right-hand end.
Click on the magnifying glass icons to zoom in or out of the graph, or type in the magnification percentage you’d like to jump to (and press the Enter key).
Click the « and » icons to pan the graph left and right.
Click the ? icon to jump to the in-app help, where you can see all the keyboard commands listed.

2.4.9. Selecting events

You can select any individual event in the timeline by clicking on it. Tabs giving further event information are then displayed beneath the timeline. By holding down the Shift key whilst clicking on an event, you can select multiple events.

Summary:
- Name - for events dispatched through one of the libraries (Poplar, PopART, TensorFlow, etc.) this is a concatenated list of the namespaces from which the event originated. For user-created functions (for example, in your Python programs), it is the name of the function.
- Channel - the channel is where this event has been called. This can be one of the predefined channels (Drivers, Poplar, Framework) or user-created.
- Timestamp - this is the execution time at which the event occurred in the timeline, measured in hours, minutes and microseconds.
- Absolute Timestamp - same as the timestamp but in absolute terms rather than relative to the start of the event.
- Duration - the amount of time the event took to execute.
- Metadata - any metadata information that has been associated with the selected trace event. See below .
Call Tree:

The call tree shows the events that are descendants of the selected event in a tree structure. You can expand to see the children of an event by clicking the caret on the left. Percentages given in the call tree are a percentage of the total time of the selected event. Hovering on a node in the call tree will highlight the respective events in the timeline above.
- Self Time - this is the total time of an event minus the time taken by its children.
- Total Time - this is the total time that the event took.
- Activity - this is the name of the event.
The call tree also has the following options that apply to it:
- Aggregate - when checked, this option aggregates all nodes with the same event name at the same level, summing their times and combining their children all under one node.

2.4.10. Searching for events

Use the Search box at the top of the report window to find events by their name. The search targets events within all processes and threads.

Click in the Search box, enter some search text, and press the Return key to start the search.
If any events have names that match your search text, the number of matches is displayed below the search box, and they are highlighted in the timeline. Events that do not match the search text are ‘greyed out’ in the timeline display, making it simple to see at a glance which events match your search.
Press the Return key repeatedly to step through the matches one-by-one, or click on the small arrows either side of the match count to move forward and backward through the matches. The report scrolls to the location of the current match.
You can cancel the search by deleting the text in the search box, or by clicking the small cross icon at its right-hand end.

2.4.11. Metadata

Metadata information (in JSON format) can be recorded by the libpvti library and associated with trace events in the pvti report file. This data can be added to the Begin phase, the End phase or both phases of a function. Metadata information is displayed in the summary tab if the selected event has any associated with it.

This metadata capability has been added to the Poplar software stack in two places:

to show information regarding the state of the synchronization with the IPU in the GDCA layer;
to show the stream that is being requested by Poplar in the prefetch and fetch calls.

Note

Metadata information for trace events is only supported for pvti files of version 1.2 or higher, and only with Poplar SDK version 2.3 or later.

2.4.12. Expanding and collapsing nodes

The timeline is described by a tree of nodes at its left-hand edge. Each of these nodes may be expanded or collapsed by clicking the caret icon at the left of the node’s label.

Note

A chart node that is collapsed displays only the top-level events for that node. By collapsing nodes, you can view more threads at once which may help to understand the events that occurred.

2.4.13. Pinning charts

An empty pin icon is displayed at the right of each chart node in the timeline (flame and line graphs). To pin a chart to the top of the timeline, click its pin. The chart will move from its initial position to a highlighted region above the rest of the timeline named “Pinned”, and its pin icon will now appear filled.

To return the chart to its initial position, click the filled icon again and it will be removed from the “Pinned” region.

Note

All charts are returned to their initial positions when the graph type or options change. Any pinned charts will remain in the “Pinned” region when files are added to the timeline.

The “Pinned” region may be expanded and collapsed, like the other nodes in the timeline. Charts appear beneath it in the order in which they were pinned.

Note

Pinning a chart does not affect the timeline overview.

To return all charts to their initial positions, click the pin icon at the right of the “Pinned” region, which will also then disappear.

2.4.14. Viewing selected duration

You can view a duration on the timeline by holding down the

Shift key and dragging the mouse from side to side anywhere in the timeline. This displays a timing duration marker at the top of the timeline, showing you the duration of the timeline you’ve selected. The selected duration is highlighted in pink in the overview.

2.4.15. Saving reports

To save the currently displayed portion of the timeline as a PNG image file:

Click on the Save button in the top, right-hand corner of the main screen.
Your system’s file browser dialog appears. Select the directory in which you want the image file saved.
Click Save .

{MODE === Mode.Development && ( <>

2.5. Dimensioning Calculator

After starting the System Analyser, you’re presented with a “landing page” from which you can open the Dimensioning Calculator. The Dimensioning Calculator helps identify when you are likely to be limited by one of two common bottlenecks when executing an application on IPU systems:

The compute required to execute the application
The I/O bandwidth required to transfer the data to and from the IPUs

2.6. Usage

You can use the Dimensioning Calculator when you have an idea of an application you want to run on the IPU system, but do not have a program yet. This helps you become aware of potential bottlenecks before writing your program.

Note

You should have a good understanding of your application, as you will need to estimate values such as input size and output size.

2.6.1. Configuration

In the About your system section, you can choose which IPU system to run your application on via the dropdown menu, and how many IPUs your application needs using the slider.

You then need to enter:

Input size - the input size of your application.
Output size - the output size of your application.

When an IPU is selected from the list on the left of the IPU section, a card is displayed in which you can customise the configuration of each IPU in the system:

Number of TFLOPS per iteration - this is the number of FLoating-point OPerations per Second (FLOPS) that will be executed on this IPU for each iteration of your application (in Tebiflops).
Compute FLOPS ratio - a scaling factor for the IPU compute to prevent an overestimation of performance.
Bandwidth ratio - a scaling factor to prevent overestimation of the I/O transfer rate.

Note

In the advanced settings, when Parallel I/O Transfer is selected, the values are not configurable per IPU, and instead are configurable for the entire system.

There are also advanced settings, which are collapsed by default:

I/O Transfer

Pipelined - data is transferred to IPU 0 and then pipelined through to the other IPUs in the system. Data is then transferred back to the host via the last IPU in the system.
Parallel - data is transferred from the host to each IPU in the system, and vice versa, at the same time.

Overlapping I/O - whether the I/O to and from the IPUs is overlapped with the IPU compute.
Tiles reserved for I/O - when I/O is overlapped, the number of tiles on each IPU that must be reserved for I/O.
Transfer rate - the rate at which data is transferred between the host and the IPUs, which depends on the hardware used.
Number format - whether your application primarily uses half-precision floating-point (FP16) or single-precision floating-point (FP32).
MAX TFLOP/S - the theoretical compute assuming 100% efficiency. This depends on the IPU system type and number format.

2.6.2. Results

Click the Calculate button to view the results corresponding to the details of your IPU system and application. The results indicate whether compute or I/O to and from the IPUs is the limiting factor in your application.

Bottleneck classification

Bandwidth - transfer time is longer than compute time
Compute - compute time is longer than transfer time

The results section also includes a visualisation of your system to illustrate the structure of the IPUs and host CPU.

The visualisation includes information about each IPU:

Frequency - IPU frequency is a constant that depends on the system type. This value is the same for each IPU.
FLOPS per iteration - this is the number of FLoating-point OPerations per Second (FLOPS) that will be executed on this IPU for each iteration of your application. This value is input by the user.
Max FLOPS - the theoretical compute assuming 100% efficiency. This depends on the IPU system type and number format. It is customisable in the advanced settings. This value is the same for each IPU.
Duration - the amount of compute time spent on each IPU. It is calculated by taking the number of FLOPS in your application model and dividing it by the IPU compute (with an applied scaling factor).

)}

2.7. Application preferences

2.7.1. Colour theme

The PopVision™ System Analyser supports light and dark colour themes, and you can select a preference here. There are three options:

Auto - this is the default setting, and allows the application to follow your machine’s system-wide theme setting for light or dark mode.
Light - this forces the application into light mode, irrespective of your machine’s theme settings.
Dark - this forces the application into dark mode, irrespective of your machine’s theme settings.

2.7.2. SSH preferences

You can store your SSH preferences in the Preferences dialog to allow authorisation when opening reports on remote machines.

SSH private key path - enter the file path of your your machine’s private SSH key here. This filepath will be used to authenticate you on remote machines during the connection process. The default path is

~/.ssh/id_rsa/ .
SSH agent mode - this dropdown-list allows you to choose whether you want to specify an ssh-agent socket path, and, if so, how you want to do so:
- Disabled - do not use an ssh-agent socket.
- Manually specify - enter file path to the ssh-agent socket in the field that appears below this option.
- Automatically obtain from environment - obtain the ssh-agent path from an environment variable.

2.7.3. Scroll behaviour

Use this preference to set the default behaviour for your mouse’s scroll wheel (or using two-finger drag on on a laptop trackpad). You can choose either:

‘Scroll by default’, where the mouse wheel will scroll the window content up and down. Holding down the Ctrl key while using the scroll wheel then zooms the window content in and out, or.
‘Zoom by default’, where the mouse wheel zooms the window content in and out. Holding down the Ctrl key while using the scroll wheel then scrolls the window content up and down.

2.7.4. Experimental features

Each version of the PopVision™ System Analyser contains some experimental features that are hidden by default. These features are not fully release-capable, and will have limited support and may change or be removed in future. You can enable them here, by toggling the button next to this option.

2.7.5. Send telemetry

When you first install the System Analyser, you will be asked for your consent for Graphcore to collect telemetry information that helps Graphcore improve the application and monitor performance. Your response to this dialog is stored in your preferences, and you can turn telemetry on or off with this option.

See this section for full details.

2.7.6. Check for updates

Check this option if you’d like the PopVision™ System Analyser to check periodically for updates. If enabled, and a newer version is detected, a pop-up notification appears, prompting you to download and install it. You can choose to download and install immediately, or have the application prompt you again later. All your preferences will be carried across to the new version.

If you’ve previously delayed the update, you can manually check again at any time by selecting ‘Check For Updates…’ from the menu.

Sometimes, network issues may cause a download to take too long to finish, in which case a dialog is diaplayed that allows you to cancel the current download, and retry later on.

2.8. About System Analyser

To see the details of the System Analyser application, select About PopVision™ System Analyser from the application’s main menu. A dialog window appears showing:

Version - the version number of the application.
Commit - the unique commit hash of this release version.
Date - the data and time this version was released.
Component version numbers - the version numbers of the main software components used by the application, including Electron, Node, Chrome and the V8 engine.

2.9. Release notes

To see details of the latest features and updates to the System Analyser application, select Release Notes from the application’s Help menu.

You can toggle the Release notes dialog between modal and full-screen view by clicking on the icon to the left of the window’s title.

2.10. License information

To see license details of third-party software used in the System Analyser application, select License from the application’s Help menu.

You can toggle the Licenses dialog between modal and full-screen view by clicking on the icon to the left of the window’s title.

2.11. Data we collect

Graphcore’s PopVision™ tools collect data from you about the way in which you use them, as described here . The data we collect depends on how you interact with the tools and may change between releases. This data helps us to develop and improve the tools.

We do not obtain any personal data when you use the PopVision™ tools. On installation, we randomly generate a unique identifier to link your interactions with one of the tools together. This identifier is stored with your preferences for the tool and can be seen at any time in the About dialog. We also randomly generate an identifier each time you open the tool to distinguish between sessions of usage.

Note

We do not collect any data about the reports or models you analyse, such as the names of variables and events.

2.12. Frequently Asked Questions (FAQ)

Question : How can I use the System Analyser over X-Forwarding on MacOS?

Answer : To view the System Analyser over X-forwarding on MacOS, follow these steps:

On your MacOS machine, download and install XQuartz from https://www.xquartz.org/.
Start the XQuartz app, and start a terminal session from within it.
In the terminal, enter ssh -X [username]@[host] , supplying the username and host for the remote machine.
In the SSH session, run the following commands (assuming you want to use v2.7.2):

$ wget https://github.com/graphcore/popvision_system_analyser/releases/download/v2.7.2/popvision-system-analyser-2.7.2.AppImage
$ chmod +x ./popvision-system-analyser-2.7.2.AppImage
$ ./popvision-system-analyser-2.7.2.AppImage

The System Analyser application should then start up and work normally over X.

2. User guide

2.1. Overview

2.1.1. End User Licence Agreement

2.1.2. Telemetry Consent

2.2. Capturing execution information

2.2.1. Capturing function entry and exit

2.2.2. Using the libpvti API

2.2.3. Capturing scalar values

2.3. Opening reports

2.3.1. Opening the Demo Report

2.3.2. Opening recent reports

2.3.3. Local reports

2.3.4. Remote reports

2.3.5. Connection Errors

2.4. Viewing reports

2.4.1. Using the sidebar buttons

2.4.2. Timeline flamegraphs

2.4.3. Viewing instant PVTI events

2.4.4. Viewing zero-length blocks

2.4.5. Timeline line graphs

IPU Utilisation

2.4.6. Timeline options

2.4.7. Panning and zooming

2.4.8. Using the Navigation panel

2.4.9. Selecting events

2.4.10. Searching for events

2.4.11. Metadata

2.4.12. Expanding and collapsing nodes

2.4.13. Pinning charts

2.4.14. Viewing selected duration

2.4.15. Saving reports

2.5. Dimensioning Calculator

2.6. Usage

2.6.1. Configuration

2.6.2. Results

2.7. Application preferences

2.7.1. Colour theme

2.7.2. SSH preferences

2.7.3. Scroll behaviour

2.7.4. Experimental features

2.7.5. Send telemetry

2.7.6. Check for updates

2.8. About System Analyser

2.9. Release notes

2.10. License information

2.11. Data we collect

2.12. Frequently Asked Questions (FAQ)