5. Viewing reports
The application’s main reports view displays the Timeline information of the execution events recorded in the files you opened, together with a scaled-down overview above, which shows the entire set of events irrespective of your current zoom and pan state.
The total number of events is displayed at the top of the timeline, along with the total duration of the report.
There is also a button named Collapse all charts (or Expand all charts) that either collapses all the charts to their most compact form in the timeline, or expands them all.
In the main reports view, the following actions are available:
Pan and zoom in and out of the timeline, viewing every event in the execution of the program.
Select individual events to view their details, and view the duration of selected sections of the timeline.
Search for events by entering an event name in the Search box.
Save report images to your local machine.
5.2. Timeline flamegraphs
Events in the timeline are grouped and layered according to the file where they originated, and beneath that to the process and thread in which they occurred. If there’s room to display it, the event’s name is shown within each event block. Hovering your mouse over an event block displays a pop-up that shows the details of the event.
The number of events within the currently displayed portion of the timeline is displayed in the top left-hand corner, as well as the duration of that portion. You’ll see these numbers change as you pan and zoom around the timeline.
The time scale is displayed across the top of the timeline, showing the elapsed time from the start of the first event. This is displayed in hours, minutes and seconds, and more significant digits of the seconds are displayed as you zoom in. You can choose to display the time scale in relative terms (starting at 0:00 at the beginning of the first captured event), or in absolute terms, where a real time is displayed. Choose the display you want by clicking the Options button at the top of the application window.
Once you’ve opened a report, you can open additional reports to display on the same timeline. Just click the Add file button at the top-left of the application window, and follow the standard method to open reports.
Events in the timeline are coloured as follows:
Poplar: Events triggered from the Poplar libraries are coloured red.
Framework: Events triggered from the PopART libraries, or any of the machine learning frameworks such as PyTorch or TensorFlow, are coloured orange.
Driver: Events triggered by the driver layer are coloured blue.
Others: User-generated event categories are then assigned a different colour.
5.3. Viewing instant PVTI events
Instant PVTI events are now displayed in the timeline as small lollipop icons above the main timeline event blocks. Instant events are set in your program code using, for example:
pvti.Tracepoint.event(channelName, "Here's an instant event")
This event will then show up on the timeline when that command was encountered as the program executes.
Clicking on an event displays its details in the tabs below the timeline.
5.3.1. IPU utilisation
The data used to populate the IPU Utilisation line graphs are sourced from the drivers when the GCDA_MONITOR environment variable is set. It is the average amount of time spent waiting for one or more IPU syncs in the previous second, and therefore a measure of whether IPUs are active (that is, executing code or stream copies) or idle. This measurement is a rough guide, and is not intended to be used for performance measurement.
Two types of IPU Utilisation graph are displayed in the timeline:
Total IPU Utilisation (%) shows what proportion of the time any IPU has been executing code over the lifetime of your application up to a particular moment.
Current IPU Utilisation (%) shows what proportion of IPU tiles are performing compute activity at a particular moment in the timeline.
See the online documentation for the gc-monitor command for more details of the data provided by the drivers.
IPU utilisation is not the same as the derived IPU execution, which shows when each IPU is active, as described in Timeline options.
5.4. Viewing zero-length blocks
PVTI files may contain sub-microsecond events which are rounded down to zero duration. To make these visible on the timeline, we have set their length to 0.5µs (providing they do not overlap the next event), and changed their label to read “<1µs”.
5.5. Timeline line graphs
If any scalar values have been captured, the timeline will also show a collection of line graphs grouped at the bottom of a file. These can be used to view these scalar values over time, for comparison to the events shown in the flamegraphs above.
Hovering over a line graph will display a tooltip with the value for each series at that timestamp.
You can display a line graph as a line or as a filled-in area:
Click on the Line Graph Type drop-down list at the top of the screen to select the display type.
This feature is only available in reports generated with Poplar SDK 2.1 and later.
5.6. Heatmaps of tensor data
You can collect data from any tensor in your model and save it in the PVTI reports that are generated by Poplar, allowing you to analyse the numerical stability of your model. See Capturing tensor data for heatmaps for instructions on how to capture tensor data from your model.
Tensor data is collected into a time-series of histogram bins, with each bin’s occupancy plotted as a coloured bar on the System Analyser trace at the moment it was sampled.
All the heatmaps are collected into a Heatmaps section on the left of the timeline, and you can show, hide and pin them just as you can with the other charts.
Each heatmap has a name and is coloured according to the scale displayed down the left-hand side of its chart. You can change these colours using the Heatmap Options dialog, described below.
See Selecting heatmap data for more information on visualising heatmap data.
5.6.1. Changing the heatmap options
You can change the colour palette used for displaying the heatmaps and histograms. Click the Heatmap Options button at the top of the application window, and select a palette. If you have more than one heatmap in the report, select Alternate colours per group, and a different colour palette will be used for each heatmap.
You can also change the range and scale that is used to colour the heatmaps. Click the Heatmap Options button at the top of the application window, and enter a maximum and minimum exponent. The colour range displayed will then be scaled across that frequency density range. Any bins with a density outside that range will be clamped to the max/min density bin.
5.7. Timeline options
Above the timeline overview there are the following buttons:
Add file: This reopens the file browsing dialog, allowing you to add more files to the current timeline following the standard method to open reports.
Graph Type: This allows you to switch between graph display types:
Timeline: Displays a flamegraph showing the position and duration of each event in the call stack. Each event on the timeline view corresponds directly to an event logged in the PVTI file.
Aggregated: Displays a flamegraph showing each event aggregated by its unique call stack. The starting timestamp at which a block appears in this view has no link to the timestamp that its events took place at within the trace. However, its duration does correspond to the total duration of the events that combine to make the block.
Options This shows the following options to apply to your timeline:
Absolute timing: Checking this option will position files on your timeline using their absolute times rather than positioning all files to start at time 0.
Derive IPU Execution: Checking this option will display an additional chart at the top of the timeline. It shows intervals of IPU activity that may be derived from host synchronisation events in the other charts.
The derived Ipu activity is more accurate in Poplar SDK 2.3 or later.
Hide charts below threshold duration This allows you to remove from the timeline any data whose duration is less than a certain proportion of the timeline duration. The options are 0% (the default), 1%, 5% and 10%.
Merge non-overlapping events This option allows you to collapse a number of threads into a smaller number of separate tracks by combining events that don’t overlap in time. This can save vertical space in the timeline if you have a number of non-synchronous or sequential events in your data. Each track displayed on the left contains a small blue i icon to show that it is the result of merged events. A rough count of the events merged is also shown.
5.7.1. Other options
Key: This allows you to see which channels the different colours represent on your timeline. Clicking one of these will reload the timeline without that channel shown.
Save (camera icon): This allows you to save the current view as a
PNGfile. This can either be saved to a file on your local machine, or to the clipboard.
5.8. Panning and zooming
You can move around the timeline using the mouse to bring events into view:
In the overview at the top of the timeline, click and drag a section to zoom into the corresponding area of the timeline. The section of the timeline you’re currently viewing is highlighted in the overview.
Drag the mouse left and right in the timeline to shift it left and right at the current zoom level.
Use the mouse wheel to zoom in and out of the timeline. If the timeline is too deep to fit into the application window, a scrollbar is displayed to enable you to move the timeline up and down in the window. As the mouse wheel is used for zooming in and out, you can hold down the
Ctrl/Commandkey to scroll the timeline up and down using the mouse wheel.Note that you can choose how you want your scroll wheel to behave by setting the Scroll behaviour in the Preferences dialog.
You can switch between window and full-screen display by selecting
Toggle Full screen from the View menu, or by pressing Ctrl/Command and F.
5.10. Selecting events
You can select any individual event in the timeline by clicking on it. Tabs giving further event information are then displayed beneath the timeline. By holding down the Shiftkey whilst clicking on an event, you can select multiple events.
Clicking on a timeline event displays the tabs described below. If you click on a heatmap element, you’ll see a histogram displayed. See Selecting heatmap data
Summary tab:
Name: For events dispatched through one of the libraries (for example PyTorch, TensorFlow, Poplar or PopART) this is a concatenated list of the namespaces from which the event originated. For user-created functions (for example, in your Python programs), it is the name of the function.
Channel: The channel is where this event has been called. This can be one of the predefined channels (Drivers, Poplar, Framework) or user-created.
Timestamp: This is the execution time at which the event occurred in the timeline, measured in hours, minutes and microseconds.
Absolute Timestamp: Same as the timestamp but in absolute terms rather than relative to the start of the event.
Duration: The amount of time the event took to execute.
Metadata: Any metadata information that has been associated with the selected trace event.
Call Tree tab: The call tree shows the events that are descendants of the selected event in a tree structure. You can expand to see the children of an event by clicking the caret on the left. Percentages given in the call tree are a percentage of the total time of the selected event. Hovering on a node in the call tree will highlight the respective events in the timeline above.
Self Time: This is the total time of an event minus the time taken by its children.
Total Time: This is the total time that the event took.
Activity: This is the name of the event.
The call tree also has the following options that apply to it:
Aggregate: When checked, this option aggregates all nodes with the same event name at the same level, summing their times and combining all their children under one node.
5.11. Selecting heatmap data
You can get more information about any element plotted in a heatmap:
Hover the mouse over a heatmap, and you’ll see the following information displayed:
Timestamp: This is the moment in the timeline that you’re hovering the mouse over.
Bin: Which histogram bin the sample you’re hovering over has been placed into.
Quantity: How many samples were collected for this bin, out of the total number of samples that were collected at that moment.
Click on a point on the heatmap, and you’re effectively selecting a moment in time and a histogram of the tensor’s data sampled at that moment. The histogram of the tensor data is displayed beneath the main timeline, showing the size of each bin along the x-axis and the number of samples falling in that bin’s range on the y-axis.
5.12. Searching for events
Use the Search box at the top of the report window to find events by their name. The search looks for events within all processes and threads.
Click in the Search box, enter some search text, and press the
Enterkey to start the search.If any events have names that match your search text, the number of matches is displayed below the search box, and they are highlighted in the timeline. Events that do not match the search text are greyed out in the timeline display, making it simple to see at a glance which events match your search.
You can cancel the search by deleting the text in the search box, or by clicking the small x icon at its right-hand end.
5.13. Metadata
Metadata information (in JSON format) can be recorded by the libpvti library and associated with trace events in the .pvti report file. This data can be added to the Begin phase, the End phase or both phases of a function. Metadata information is displayed in the summary tab if the selected event has
any metadata associated with it.
This metadata capability has been added to the Poplar software stack in two places:
to show information regarding the state of the synchronization with the IPU in the GDCA layer;
to show the stream that is being requested by Poplar in the prefetch and fetch calls.
Metadata information for trace events is only supported for .pvti files of version 1.2 or higher, and only with
Poplar SDK version 2.3 or later.
5.14. Expanding and collapsing nodes
The timeline is described by a tree of nodes at its left-hand edge. Each of these nodes may be expanded or collapsed by clicking the caret icon at the left of the node’s label. By collapsing nodes, you can view more threads at once which may help to understand the events that occurred.
There are two types of node that can be collapsed or expanded:
chart nodes (for example, threads or individual line graphs) - these have no sub-nodes, and when collapsed they display only the top-level events for that node.
group nodes (for example an entire Process, any Derived Info or the Line Graphs group) - these nodes have lots of chart nodes inside them, and when they are collapsed, they display a graphical overview of the nodes within them, similar to the timeline overview at the top of the report window.
5.15. Pinning charts
An empty pin icon is displayed at the right of each chart node in the timeline (flame and line graphs). To pin a chart to the top of the timeline, click its pin. The chart will move from its initial position to a highlighted region above the rest of the timeline named Pinned, and its pin icon will now appear filled.
To return the chart to its initial position, click the filled icon again and it will be removed from the Pinned region.
All charts are returned to their initial positions when the graph type or options change. Any pinned charts will remain in the Pinned region when files are added to the timeline.
The Pinned region may be expanded and collapsed, like the other nodes in the timeline. Charts appear beneath it in the order in which they were pinned.
Pinning a chart does not affect the timeline overview.
To return all charts to their initial positions, click the pin icon at the right of the Pinned region, which will also then disappear.
5.16. Viewing selected duration
You can view a duration on the timeline by holding down the Shift key and dragging the mouse from side to side anywhere in the timeline. This displays a timing duration marker at the top of the timeline, showing you the duration of the timeline you’ve selected. The selected duration is highlighted in pink in the overview.
5.17. Saving reports
To save the currently displayed portion of the timeline as a PNG image file:
Click on the Save button in the top, right-hand corner of the main screen.
Your system’s file browser dialog appears. Select the directory in which you want the image file saved.
Click Save