13. gc-links
This tool displays the status and connectivity of each of the IPU-Links that connect the IPUs. It does this by “training” the links with some data, and then checking that the data can be retrieved across all the links.
To use it, run:
gc-links -j
Note: On IPU-POD systems, the IPU-Links are trained when the partition is created. When running gc-links
, the output will look something like this:
{
"iteration_0": {
"warning": "Links already trained - no training performed",
"overall_result": "passed",
"discovery_method": "VIRM"
},
"requested_iterations": "1",
"iterations_with_failures": "0",
"overall_result": "passed"
}
On a system where link training is available, the output will look similar to the following example:
{
"ipu to ipu": {
"from pcie id": "5",
"to pcie id": "7",
"channel": {
"from": "NLC_E_2A",
"to": "NLC_E_3A",
"status": "passed",
"gen": "4",
"lanes": "8"
},
"channel": {
"from": "NLC_E_2B",
"to": "NLC_E_3B",
"status": "passed",
"gen": "4",
"lanes": "8"
}
},
"num ipus": "16",
"overall result": "passed",
"training fails": "0"
}
The “status” field shows the training status for each link. The “lanes” field shows the number of lanes being trained, and the “gen” field shows what generation of link is tested.
When gc-links
finds that a link fails to train, the output looks like
this:
{
"ipu to ipu": {
"from pcie id": "7",
"to pcie id": "6",
"channel": {
"from": "NLC_W_1B",
"to": "NLC_W_1B",
"status": "passed",
"gen": "4",
"lanes": "8"
},
"channel": {
"from": "NLC_W_1C",
"to": "NLC_W_1C",
"status": "failed",
"gen": "0",
"lanes": "0"
}
}
}
This output shows that it failed to train the link from device 7 to device 6, using link NLC_W_1C.
You can run gc-inventory to show more information on devices 6 and 7:
Device:
id: 6
type: C2
Firmware Major Version: 1
Firmware Minor Version: 0
Firmware Patch Version: 4
IPU: 1
IPU version: ipu0
PCI Id: 0000:43:00.0
link speed: 8 GT/s
link width: 8
physical slot: PCIe Slot 12
serial number: 0020.0004.018471
Device:
id: 7
type: C2
Firmware Major Version: 1
Firmware Minor Version: 0
Firmware Patch Version: 4
IPU: 1
IPU version: ipu0
PCI Id: 0000:44:00.0
link speed: 8 GT/s
link width: 8
physical slot: PCIe Slot 13
serial number: 0027.0004.018481
You can see, in this example, that there’s an issue with the link between the card in slot 12 and 13.
When training a chassis full of IPUs, the tool outputs additional information on IPU-Link failures, for example:
{
"failures": [
{
"cable_id": "IPUL-00"
},
{
"cable_id": "IPUL-24"
},
{
"cable_id": "IPUL-25"
}
]
}
The physical location of these failing cables is shown in IPU-Link channel mapping.
13.1. Usage
13.1.1. Allowed options
|
Emit JSON output |
|
Number of link training retries (default: 3) |
|
Device id (default is largest group) |
|
Number of times to train each link (default: 1) |
|
Verbose output |
|
Print PHY summary after all training runs |
|
Produce help message |
|
Run the training until it succeeds |
|
Train links to specified generation, default is gen 4 |
|
Train links using single IPU loopback config |
|
Version number |