This tool tests data transfer between IPU-Machines over GW-Links.
It sends data from IPUs on one IPU-Machine across the GW-Links to an IPU-Machine on the other side of the link.
Receiving IPUs verify the data is correct.
This program is only applicable to IPU-POD systems where IPU-POD racks have been connected by GW-Links.
5.1. Standard mode
To use GW-Links, an IPU-POD system must have been configured with a cluster and non-reconfigurable partition containing
at least two IPU Link Domains (ILDs) (Note that GW-Links cannot be used with reconfigurable partitions).
For example, the simplest such configuration would be two IPU-Machines, sitting side by side in racks connected only by GW Links.
You could make a cluster and partition containing two ILDs, where each ILD contains one GCD.
The V-IPU commands to do this from agents
ag2 would look like:
vipu-admin create cluster cl1x2 --agents ag1,ag2 --num-ipulinkdomains 2 --cluster-topology looped
vipu-admin create partition pa1x2 --size 8 --num-gcds 2 --gw-routing ring
This will create two IPUoF configuration files for the
pa1x2 partition, one per ILD/GCD.
In this example they would be named
You can now use
gc-gwlinkstraffictest to check this configuration is working and measure its performance.
For each ILD/GCD, you must run one instance of
Each instance needs to be configured to attach to the correct partition and ILD. This can be done by setting the
IPUOF_CONFIG_PATH environment variable to specify the relevant configuration file.
If your system uses a separate headnode per ILD then you would run one instance on the first headnode and the other on the second.
Each instance will attach to the multi-IPU device containing all the IPUs in that ILD, and configure those devices to exchange data with devices in the other ILD.
You should start both instances at roughly the same time.
Each instance will finish and report results separately.
If the test was successful the results will look something like:
$ IPUOF_CONFIG_PATH=/home/justina/.ipuof.conf.d/pa1x2_gcd0.conf gc-gwlinkstraffictest
gwlinks: 0.7188 GiB in 0.08 seconds, 9.54 GiB/s, 81.93 Gbps
$ IPUOF_CONFIG_PATH=/home/justina/.ipuof.conf.d/pa1x2_gcd1.conf gc-gwlinkstraffictest
gwlinks: 0.7188 GiB in 0.08 seconds, 9.53 GiB/s, 81.83 Gbps
To get more detailed information use the verbose option,
Detailed JSON formatted results are available by specifying the
On each IPU-Machine in the partition, data is sent from IPUs #0 and #1 from the first ILD to IPUs #2 and #3 in the corresponding IPU-Machine in the second ILD, and vice versa. By default, IPU #0 will transmit through GW-Link port 1 and IPU #2 will use port 2. This behaviour can be changed with the
To run multiple iterations of the test, specify the
To test that the transfer rate does not fall below a specific threshold, use the
Note, if the message
Warning: SPCR.GSBGS2DIR is not set to SREQ_OUTPUT is displayed, it means that the sync configuration of the partition is incorrect and the test will most likely fail. This can occur if another application changes the sync configuration from its original state. The
vipu-admin reset partition command will restore the sync configuration to the correct state.
If an error occurs, the tool will display an error message and set a non-zero exit code.