7. gc-gwlinkstraffictest
This tool tests data transfer between IPU-Machines over GW-Links. It sends data from IPUs on one IPU-Machine across the GW-Links to an IPU-Machine on the other side of the link. Receiving IPUs verify the data is correct.
This program is only applicable to IPU-POD systems where IPU-POD racks have been connected by GW-Links.
This program has two completely separate modes of use:
Standard mode, which tests entire IPU Fabric multi-ILD (IPU-Link Domain) partitions.
Single IPU mode, which provides a low-level mechanism to configure individual IPUs to test specific GW-Link configurations. This mode is not meant for general use. It is primarily intended to be used by the V-IPU diagnostic tools.
7.1. Standard mode
To use GW-Links, an IPU-POD system must have been configured with a cluster and non-reconfigurable partition containing at least two IPU Link Domains (ILDs) (Note that GW-Links cannot be used with reconfigurable partitions).
For example, the simplest such configuration would be two IPU-Machines, sitting side by side in racks connected only by GW-Links.
You could make a cluster and partition containing two ILDs, where each ILD contains one GCD.
The V-IPU commands to do this from agents ag1 and ag2 would look like:
vipu-admin create cluster cl1x2 --agents ag1,ag2 --num-ipulinkdomains 2 --cluster-topology looped
vipu-admin create partition pa1x2 --size 8 --num-gcds 2 --gw-routing ring
This will create two IPUoF configuration files for the pa1x2 partition, one per ILD/GCD.
In this example they would be named pa1x2_gcd0.conf and pa1x2_gcd1.conf.
You can now use gc-gwlinkstraffictest to check this configuration is working and measure its performance.
For each ILD/GCD, you must run one instance of gc-gwlinkstraffictest.
Each instance needs to be configured to attach to the correct partition and ILD. This can be done by setting the IPUOF_CONFIG_PATH environment variable to specify the relevant configuration file.
If your system uses a separate headnode per ILD then you would run one instance on the first headnode and the other on the second. Each instance will attach to the multi-IPU device containing all the IPUs in that ILD, and configure those devices to exchange data with devices in the other ILD. You should start both instances at roughly the same time.
IPUOF_CONFIG_PATH=/home/ipuuser/.ipuof.conf.d/pa1x2_gcd0.conf gc-gwlinkstraffictest
IPUOF_CONFIG_PATH=/home/ipuuser/.ipuof.conf.d/pa1x2_gcd1.conf gc-gwlinkstraffictest
Each instance will finish and report results separately. If the test was successful the results will look something like:
$ IPUOF_CONFIG_PATH=/home/ipuuser/.ipuof.conf.d/pa1x2_gcd0.conf gc-gwlinkstraffictest
gwlinks: 0.7188 GiB in 0.08 seconds, 9.54 GiB/s, 81.93 Gbps
$ IPUOF_CONFIG_PATH=/home/ipuuser/.ipuof.conf.d/pa1x2_gcd1.conf gc-gwlinkstraffictest
gwlinks: 0.7188 GiB in 0.08 seconds, 9.53 GiB/s, 81.83 Gbps
To get more detailed information use the verbose option, -v.
Detailed JSON formatted results are available by specifying the -j option.
On each IPU-Machine in the partition, data is sent from IPUs #0 and #1 from the first ILD to IPUs #2 and #3 in the corresponding IPU-Machine in the second ILD, and vice versa. By default, IPU #0 will transmit through GW-Link port 1 and IPU #2 will use port 2. This behaviour can be changed with the -p option.
To run multiple iterations of the test, specify the -i option.
To test that the transfer rate does not fall below a specific threshold, use the --min-gwlinks-bandwidth option.
Note
If the message Warning: SPCR.GSBGS2DIR is not set to SREQ_OUTPUT is displayed, it means that the sync configuration of the partition is incorrect and the test will most likely fail. This can occur if another application changes the sync configuration from its original state. The vipu-admin reset partition command will restore the sync configuration to the correct state.
If an error occurs, the tool will display an error message and set a non-zero exit code.
7.2. Single IPU mode
This mode is not intended for general use, as such it will only be described briefly. Single IPU mode is only available on native IPU-Machine hardware and cannot be used over fabric.
This mode is selected with the --single-ipu-mode option.
Each IPU in an IPU-Machine must be configured individually with one of the following commands:
tx: Configure the IPU to transmit to a device on the other side of the GW-Link connection. Requires a--rx-device-idparameter to specify the target IPU.rx: Configure the IPU to expect to receive data. One of these devices should be configured as the sync master, by specifying--master-rx.dummy: Take part in the sync mechanism but do not transfer any data.run: Use the sync mechanism to drive execution of the transmitter and receiver programs. Must be ran on the IPU that is the sync master. All 4 devices must have already been configured in either tx, rx or dummy mode for run to execute correctly.check: Check the received data is as expected. Must be ran on an IPU that was previous configured as a receiver.
7.3. Usage
7.3.1. Allowed options
|
Number of tiles. Default all (default: 0) |
|
Iterations per tile. (default: 1) |
|
128 byte packets instead of 256 byte packets. |
|
Don’t verify the result. |
|
0: alternate ports, 1: always port 1, 2: always port 2 (default: 0) |
|
Minimum gwlinks bandwidth (Gbps) expected - fails if not reached |
|
Verbose output. |
|
help |
|
Emit JSON output |
|
Timeout in seconds (default: 300) |
|
Version |