13. gc-links
This tool displays the status and connectivity of each of the IPU-Links that connect the IPUs. It does this by “training” the links with some data, and then checking that the data can be retrieved across all the links.
To use it, run:
gc-links -j
Note: On IPU-POD systems, the IPU-Links are trained when the partition is created. When running gc-links, the output will look something like this:
{
"iteration_0": {
"warning": "Links already trained - no training performed",
"overall_result": "passed",
"discovery_method": "VIRM"
},
"requested_iterations": "1",
"iterations_with_failures": "0",
"overall_result": "passed"
}
On a system where link training is available, the output will look similar to the following example:
{
"ipu to ipu": {
"from pcie id": "5",
"to pcie id": "7",
"channel": {
"from": "NLC_E_2A",
"to": "NLC_E_3A",
"status": "passed",
"gen": "4",
"lanes": "8"
},
"channel": {
"from": "NLC_E_2B",
"to": "NLC_E_3B",
"status": "passed",
"gen": "4",
"lanes": "8"
}
},
"num ipus": "16",
"overall result": "passed",
"training fails": "0"
}
The “status” field shows the training status for each link. The “lanes” field shows the number of lanes being trained, and the “gen” field shows what generation of link is tested.
When gc-links finds that a link fails to train, the output looks like
this:
{
"from_pcie_id": "7",
"to_pcie_id": "6",
"channels": [
{
"from": "NLC_W_0B",
"to": "NLC_W_0B",
"num_tries": "1",
"status": "passed",
"gen": "4",
"lanes": "8",
"lowest_fom": "154"
},
{
"from": "NLC_W_0C",
"to": "NLC_W_0C",
"num_tries": "1",
"status": "failed",
"gen": "4",
"lanes": "8",
"lowest_fom": "140"
}
]
}
This output shows that it failed to train the link from device 7 to device 6, using link NLC_W_1C.
You can run gc-inventory to show more information on devices 6 and 7:
Device
"id": "6",
"target": "PCIe",
"average board temp": "26.6 C",
"average die temp": "20.1 C",
"board ipu index": "0",
"board serial number": "0057.0063.822351",
"board type": "C600",
"board variant": "C600",
"clock": "20MHz",
"driver version": "1.3.0",
"firmware major version": "2",
"firmware minor version": "6",
"firmware patch version": "7",
"firmware version": "2.6.7",
"hexoatt active size (bytes)": "0",
"hexoatt total size (bytes)": "16911433728",
"hexopt active size (bytes)": "0",
"hexopt total size (bytes)": "268435456",
"icu bootloader version": "2.6.0",
"ipu architecture": "ipu21",
"ipu error state": "no errors",
"ipu power": "N\/A",
"ipu utilisation": "0.00%",
"ipu utilisation (session)": "0.00%",
"link correctable error count": "0",
"link correctable error count (session)": "0",
"link speed": "8.0 GT\/s",
"link width": "8",
"numa node": "0",
"parity error count threshold": "4",
"parity error threshold interval": "7776000 seconds",
"parity initialised": "1",
"pci id": "0000:1b:00.0",
"pcie physical slot": "PCIe Slot 12",
"remote buffers supported": "1",
"sysfs file id": "0",
"total board power": "12.4 W"
Device
"id": "7",
"target": "PCIe",
"average board temp": "26.6 C",
"average die temp": "20.1 C",
"board ipu index": "0",
"board serial number": "0057.0063.822352",
"board type": "C600",
"board variant": "C600",
"clock": "20MHz",
"driver version": "1.3.0",
"firmware major version": "2",
"firmware minor version": "6",
"firmware patch version": "7",
"firmware version": "2.6.7",
"hexoatt active size (bytes)": "0",
"hexoatt total size (bytes)": "16911433728",
"hexopt active size (bytes)": "0",
"hexopt total size (bytes)": "268435456",
"icu bootloader version": "2.6.0",
"ipu architecture": "ipu21",
"ipu error state": "no errors",
"ipu power": "N\/A",
"ipu utilisation": "0.00%",
"ipu utilisation (session)": "0.00%",
"link correctable error count": "0",
"link correctable error count (session)": "0",
"link speed": "8.0 GT\/s",
"link width": "8",
"numa node": "0",
"parity error count threshold": "4",
"parity error threshold interval": "7776000 seconds",
"parity initialised": "1",
"pci id": "0000:1b:00.0",
"pcie physical slot": "PCIe Slot 13",
"remote buffers supported": "1",
"sysfs file id": "0",
"total board power": "12.4 W"
You can see, in this example, that there’s an issue with the link between the card in slot 12 and 13.
When training a chassis full of IPUs, the tool outputs additional information on IPU-Link failures, for example:
{
"failures": [
{
"cable_id": "IPUL-00"
},
{
"cable_id": "IPUL-24"
},
{
"cable_id": "IPUL-25"
}
]
}
The physical location of these failing cables is shown in
ipu_link_channel_mapping.
13.1. Usage
13.1.1. Allowed options
|
Emit JSON output |
|
Number of link training retries (default: 3) |
|
Device id (default is largest group) |
|
Number of times to train each link (default: 1) |
|
Verbose output |
|
Print PHY summary after all training runs |
|
Produce help message |
|
Run the training until it succeeds |
|
Train links to specified generation, default is gen 4 |
|
Train links using single IPU loopback config |
|
Version number |