21. C600 PCIe Device IDs and channel map

The following information applies to C600 cards in a PCIe server. For IPU-Machine and IPU Pod systems, see the appropriate product documentation.

21.1. Device IDs

The diagram below shows how single IPUs and then, hierarchically, groups of IPUs are numbered in an 8-IPU system. For example, device ID 12 corresponds to the four IPUs numbered 0 to 3. These numbers are the IDs used by tools such as gc-info.

_images/id-mapping-c600.png

Fig. 21.1 IPU device IDs

21.2. PCIe ID to slot mapping

The Graphcore tools typically deduce which IPU cards are in which PCIe slot via a table in the SMBIOS. Alternatively, a JSON configuration file may be provided by setting the environment variable GCDA_CONFIG_FILE. The file must have the following format:

{
  "ipu_card_mapping": [
   "0000:8c:00.0",
   "0000:8a:00.0",
   "0000:c4:00.0",
   "0000:c5:00.0",
   "0000:1b:00.0",
   "0000:1c:00.0",
   "0000:48:00.0",
   "0000:49:00.0"
  ]
}

Where “ipu_card_mapping” is an ordered list of the PCIe IDs for the Graphcore cards, from one side of the chassis to the next. Each PCIe ID should match a single card, and the next entry should be in the next physical slot.

21.3. Manually preparing the mapping config File

To discover the ordering, it is necessary to visually inspect the C600 cards within the chassis. Open up the chassis and determine the serial numbers of the cards in the slot order they appear in. For example:

  • Slot 6 - 0029.0063.822351

  • Slot 7 - 0053.0063.822351

  • Slot 8 - 0062.0063.822351

  • Slot 9 - 0016.0063.822351

  • Slot 10 - 0057.0063.822351

  • Slot 11 - 0041.0063.822351

  • Slot 12 - 0014.0063.822351

  • Slot 13 - 0045.0063.822351

Now use gc-inventory to determine the PCIe ID of these cards:

$ gc-inventory |grep 'pci id:\|board serial'
  board serial number: 0057.0063.822351
  pci id: 0000:1b:00.0
  board serial number: 0041.0063.822351
  pci id: 0000:1c:00.0
  board serial number: 0014.0063.822351
  pci id: 0000:48:00.0
  board serial number: 0045.0063.822351
  pci id: 0000:49:00.0
  board serial number: 0053.0063.822351
  pci id: 0000:8a:00.0
  board serial number: 0029.0063.822351
  pci id: 0000:8c:00.0
  board serial number: 0062.0063.822351
  pci id: 0000:c4:00.0
  board serial number: 0016.0063.822351
  pci id: 0000:c5:00.0

So taking this information you can generate the ordering of the PCIe IDs using the JSON file:

{
  "ipu_card_mapping": [
   "0000:8c:00.0",
   "0000:8a:00.0",
   "0000:c4:00.0",
   "0000:c5:00.0",
   "0000:1b:00.0",
   "0000:1c:00.0",
   "0000:48:00.0",
   "0000:49:00.0"
  ]
}