13. gc-links

This tool displays the status and connectivity of each of the IPU-Links that connect the IPUs. It does this by “training” the links with some data, and then checking that the data can be retrieved across all the links.

To use it, run:

gc-links -j

Note: On IPU-POD systems, the IPU-Links are trained when the partition is created. When running gc-links, the output will look something like this:

{
  "iteration_0": {
      "warning": "Links already trained - no training performed",
      "overall_result": "passed",
      "discovery_method": "VIRM"
  },
  "requested_iterations": "1",
  "iterations_with_failures": "0",
  "overall_result": "passed"
}

On a system where link training is available, the output will look similar to the following example:

{
  "ipu to ipu": {
      "from pcie id": "5",
      "to pcie id": "7",
      "channel": {
          "from": "NLC_E_2A",
          "to": "NLC_E_3A",
          "status": "passed",
          "gen": "4",
          "lanes": "8"
      },
      "channel": {
          "from": "NLC_E_2B",
          "to": "NLC_E_3B",
          "status": "passed",
          "gen": "4",
          "lanes": "8"
      }
  },
  "num ipus": "16",
  "overall result": "passed",
  "training fails": "0"
}

The “status” field shows the training status for each link. The “lanes” field shows the number of lanes being trained, and the “gen” field shows what generation of link is tested.

When gc-links finds that a link fails to train, the output looks like this:

{
  "ipu to ipu": {
    "from pcie id": "7",
    "to pcie id": "6",
    "channel": {
        "from": "NLC_W_1B",
        "to": "NLC_W_1B",
        "status": "passed",
        "gen": "4",
        "lanes": "8"
    },
    "channel": {
        "from": "NLC_W_1C",
        "to": "NLC_W_1C",
        "status": "failed",
        "gen": "0",
        "lanes": "0"
    }
  }
}

This output shows that it failed to train the link from device 7 to device 6, using link NLC_W_1C.

You can run gc-inventory to show more information on devices 6 and 7:

Device:
  id: 6
  type: C2
  Firmware Major Version: 1
  Firmware Minor Version: 0
  Firmware Patch Version: 4
  IPU: 1
  IPU version: ipu0
  PCI Id: 0000:43:00.0
  link speed: 8 GT/s
  link width: 8
  physical slot: PCIe Slot 12
  serial number: 0020.0004.018471
Device:
  id: 7
  type: C2
  Firmware Major Version: 1
  Firmware Minor Version: 0
  Firmware Patch Version: 4
  IPU: 1
  IPU version: ipu0
  PCI Id: 0000:44:00.0
  link speed: 8 GT/s
  link width: 8
  physical slot: PCIe Slot 13
  serial number: 0027.0004.018481

You can see, in this example, that there’s an issue with the link between the card in slot 12 and 13.

When training a chassis full of IPUs, the tool outputs additional information on IPU-Link failures, for example:

{
  "failures": [
    {
        "cable_id": "IPUL-00"
    },
    {
        "cable_id": "IPUL-24"
    },
    {
        "cable_id": "IPUL-25"
    }
  ]
}

The physical location of these failing cables is shown in IPU-Link channel mapping.

13.1. Usage

13.1.1. Allowed options

-j, --json-output	Emit JSON output
-n {arg}, --num-retries {arg}	Number of link training retries (default: 3)
-d {id}, --device-id {id}	Device id (default is largest group)
-i {arg}, --num-iterations {arg}	Number of times to train each link (default: 1)
-v, --verbose	Verbose output
-p, --phy-summary	Print PHY summary after all training runs
-h, --help	Produce help message
--until-trained	Run the training until it succeeds
--train-to-gen {arg}	Train links to specified generation, default is gen 4
--single-ipu-loopback	Train links using single IPU loopback config
--version	Version number