15. gc-monitor

You can use this command to monitor IPU activity without affecting users of the IPUs. This can be used to:

  • Check and monitor what’s currently running on which IPU in shared systems.

  • Make sure code is correctly running on an IPU.

  • Monitor performance: the power and temperature will increase, and the clock rate will drop, when an IPU is heavily loaded.

15.1. Output

gc-monitor displays a device information table followed by information about any processes that are using these IPUs. The structure of the device information table is different depending on whether your system uses IPU-Machines or PCIe cards.

By default, gc-monitor only displays information about the current active partition. To see the IPUs in other partitions, and the processes using those IPUs, use the --all-partitions command line option.

15.1.1. IPU-Machine device information

The example below is from an IPU-Machine installation that has been configured with a single 8-IPU partition.

+---------------+--------------------------------------------------------------------------------+
|  gc-monitor   |            Partition: p1 (gcd:0) [active] has 8 reconfigurable IPUs            |
+-------------+--------------------+--------+--------------+----------+------+----+------+-------+
|    IPU-M    |       Serial       |IPU-M SW|Server version|  ICU FW  | Type | ID | IPU# |Routing|
+-------------+--------------------+--------+--------------+----------+------+----+------+-------+
|  10.1.5.10  | 0024.0002.8203321  | 2.5.0  |    1.9.0     |  2.4.4   |M2000 | 0  |  3   |  DNC  |
|  10.1.5.10  | 0024.0002.8203321  | 2.5.0  |    1.9.0     |  2.4.4   |M2000 | 1  |  2   |  DNC  |
|  10.1.5.10  | 0024.0001.8203321  | 2.5.0  |    1.9.0     |  2.4.4   |M2000 | 2  |  1   |  DNC  |
|  10.1.5.10  | 0024.0001.8203321  | 2.5.0  |    1.9.0     |  2.4.4   |M2000 | 3  |  0   |  DNC  |
+-------------+--------------------+--------+--------------+----------+------+----+------+-------+
|  10.1.5.11  | 0013.0002.8204921  | 2.5.0  |    1.9.0     |  2.4.4   |M2000 | 4  |  3   |  DNC  |
|  10.1.5.11  | 0013.0002.8204921  | 2.5.0  |    1.9.0     |  2.4.4   |M2000 | 5  |  2   |  DNC  |
|  10.1.5.11  | 0013.0001.8204921  | 2.5.0  |    1.9.0     |  2.4.4   |M2000 | 6  |  1   |  DNC  |
|  10.1.5.11  | 0013.0001.8204921  | 2.5.0  |    1.9.0     |  2.4.4   |M2000 | 7  |  0   |  DNC  |
+-------------+--------------------+--------+--------------+----------+------+----+------+-------+
Table 15.1 IPU-Machine device table fields

Label

Description

IPU-M

IPU-Machine address.

Serial

Each IPU-Machine has two serial numbers, one for each pair of IPUs.

IPU-M SW

IPU-Machine overall software version.

Server version

Fabric server software version.

ICU FW

ICU firmware revision.

Type

IPU-Machine type — M2000 in this case.

ID

Device ID of the IPU, used by applications and other tools to address an IPU in a partition.

IPU#

Index from 0-3 that identifies the IPU chip on an IPU-Machine.

Routing

Link configuration for partition. In this case, “DNC”.

Partition information is displayed in the table header.

Note: if the Type field displays M2000, this is the abbreviation for both the IPU-M2000 and Bow-2000 IPU-Machines.

15.1.2. PCIe card device information

The example below is from a system built from 8 C600 PCIe cards:

+---------------+---------------------------------------------------------------------------+
|  gc-monitor   |                          Installed driver: 1.1.7                          |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
| Slot |       Serial       |  ICU FW  | Type  |  Speed   | Ln | ID |    PCIe ID     | IPU# |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
|  10  |  0057.0063.822351  |  2.6.7   | C600  | 8.0 GT/s | 8  | 0  |  0000:1b:00.0  |  0   |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
|  11  |  0041.0063.822351  |  2.6.7   | C600  | 8.0 GT/s | 8  | 1  |  0000:1c:00.0  |  0   |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
|  12  |  0014.0063.822351  |  2.6.7   | C600  | 8.0 GT/s | 8  | 2  |  0000:48:00.0  |  0   |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
|  13  |  0045.0063.822351  |  2.6.7   | C600  | 8.0 GT/s | 8  | 3  |  0000:49:00.0  |  0   |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
|  7   |  0053.0063.822351  |  2.6.7   | C600  | 8.0 GT/s | 8  | 4  |  0000:8a:00.0  |  0   |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
|  6   |  0029.0063.822351  |  2.6.7   | C600  | 8.0 GT/s | 8  | 5  |  0000:8c:00.0  |  0   |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
|  8   |  0062.0063.822351  |  2.6.7   | C600  | 8.0 GT/s | 8  | 6  |  0000:c4:00.0  |  0   |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
|  9   |  0016.0063.822351  |  2.6.7   | C600  | 8.0 GT/s | 8  | 7  |  0000:c5:00.0  |  0   |
+------+--------------------+----------+-------+----------+----+----+----------------+------+
+--------------------------------------------------------------------------------------------------+
|                                      No attached processes                                       |
+--------------------------------------------------------------------------------------------------+
Table 15.2 PCIe device table fields

Label

Description

Slot

Physical PCIe slot location, if available.

Serial

Serial number of the board.

ICU FW

ICU firmware revision.

Type

IPU system type — C600 PCIe cards in this example.

Speed

PCIe speed.

Ln

Number of PCIe lanes in use.

ID

Device ID of the IPU. Used by applications and other tools to address an IPU.

PCIe ID

PCI domain:bus:device.function

IPU#

Index from 0-1 that identifies the IPU chip on a two-IPU PCIe card.

The kernel driver version is displayed in the table header.

15.1.3. Process information

The example below shows the default process information table displaying a gc-hosttraffictest process running across 4 IPUs.

+------------------------------------------------------------------------+------------------------+-----------------+
|                     Attached processes in partition p1                 |          IPU           |      Board      |
+--------+-----------------------------------------+--------+------------+----+----------+--------+--------+--------+
|  PID   |                 Command                 |  Time  |    User    | ID |  Clock   |  Temp  |  Temp  | Power  |
+--------+-----------------------------------------+--------+------------+----+----------+--------+--------+--------+
| 81816  |           gc-hosttraffictest            |  10s   |  ipuuser   | 14 | 1300MHz  | 33.7 C | 35.0 C |113.7 W |
| 81816  |           gc-hosttraffictest            |  10s   |  ipuuser   | 15 | 1300MHz  | 37.4 C |        |        |
+--------+-----------------------------------------+--------+------------+----+----------+--------+--------+--------+
| 81816  |           gc-hosttraffictest            |  10s   |  ipuuser   | 12 | 1300MHz  | 31.4 C | 32.8 C |107.1 W |
| 81816  |           gc-hosttraffictest            |  10s   |  ipuuser   | 13 | 1300MHz  | 34.7 C |        |        |
+--------+-----------------------------------------+--------+------------+----+----------+--------+--------+--------+
Table 15.3 Default process information fields

Label

Description

PID

The process ID of the program using the IPU.

Command

The program using the IPU.

Time

How long the program has been running. Note, not necessarily how long it has been using IPUs.

User

The user that started the process.

ID

Device ID of the IPU.

Clock

The current measured IPU clock rate.

IPU Temp

IPU temperature in degrees Celsius measured from on-chip sensors. May be less accurate than board temperature.

Board Temp

Board temperature in degrees Celsius.

Board Power

Board power consumption in watts. This is the total power of the IPUs on the board. It does not include the power used by fans and other components on the board. Note that the power is sampled and may not accurately reflect either peak or average power.

There are a number of additional process information fields which can be enabled with the -f option. If -f is specified without a parameter all process information fields will be displayed, as below:

+------+---------------+---------------+------------------+----------+----------+----------+----------+---------------+----------------+----------------+--------+--------+------------+
|  ID  |  Board Power  |  Board Temp   |     Command      | Exch Mem | IPU Code | IPU Data |IPU Stack |   IPU Temp    |    IPU-util    | IPU-util-sess  |  PID   |  Time  |    User    |
+------+---------------+---------------+------------------+----------+----------+----------+----------+---------------+----------------+----------------+--------+--------+------------+
|  0   |    188.1 W    |    38.2 C     |     python3      | 0.00 GB  |  40.90%  | 265.12%  |  0.79%   |    39.1 C     |     98.75%     |     95.68%     | 22844  | 1m22s  |  ipuuser   |
+------+---------------+---------------+------------------+----------+----------+----------+----------+---------------+----------------+----------------+--------+--------+------------+

The complete set of fields may be slightly unwieldy, so -f can take a set of comma-separated arguments to specify which fields should be displayed. Use -f list to display all available fields.

Table 15.4 All process fields

Label

Description

ID

Device ID of the IPU.

Board Power

Board power consumption in watts. This is the total power of the IPUs on the board. It does not include the power used by fans and other components on the board. Note that the power is sampled and may not accurately reflect either peak or average power.

Board Temp

Board temperature in degrees Celsius.

Command

The program using the IPU.

Exch Mem

Total external memory usage in GB.

IPU Code

Total application code size expressed as a percentage of tile memory on a single IPU.

IPU Data

Total application data size expressed as a percentage of tile memory on a single IPU.

IPU Stack

Total application stack size expressed as a percentage of tile memory on a single IPU.

IPU Temp

IPU temperature in degrees Celsius measured from on-chip sensors. May be less accurate than board temperature.

IPU-util

A rough indication of “IPU utilisation” based on the amount of time spent waiting for one or more IPU syncs in the last second. Not intended to be used for performance measurement.

IPU-util-sess

A rough indication of “IPU utilisation” based on the amount of time spent waiting for one or more IPU syncs since the HSPs were set up. Not intended to be used for performance measurement.

PID

The process ID of the program using the IPU.

Time

How long the program has been running. Note, not necessarily how long it has been using IPUs.

User

The user that started the process.

15.2. Usage

15.2.1. Allowed options

--no-card-info

Don’t display card information

--all-partitions

Show information about all partitions (default is active partition only)

-s, --sensors

Show sensor data even if no process is attached

-j, --json-output

Emit JSON output

-x, --xml-output

Emit XML output

-c, --csv-output

Emit CSV output

-f {arg}, --fields {arg}

Comma separated field names to be displayed in attached process info. Use -f list to see the choices. If empty all supported fields are displayed.

--no-headers

Don’t display headers (csv only)

--version

Version number

-h, --help

Produce help message

15.2.2. Notes

  • By default, IPU applications do not read power and temperature sensors, so this information will not be available in gc-monitor. To enable sensor reading the application must be launched with the GCDA_MONITOR environment variable set.

15.2.3. Examples

### Run main.py, enabling gc-monitor power/temp output
$ GCDA_MONITOR=1 python main.py
### Refresh gc-monitor output every second
$ watch -n1 gc-monitor
### Append one JSON gc-monitor reading to a file
$ gc-monitor -j >> data.json