8. IPU‑POD₁₂₈ network configuration

This section describes how to set up the network configuration on the management server and IPU-M2000s for your IPU‑POD₁₂₈.

8.1. Overview

This section describes how two IPU‑POD₆₄ racks can be merged into a new IPU‑POD₁₂₈. The two IPU‑POD₆₄ racks will be referred to as lrack1 and lrack2 in this section of the document for illustration purposes. Lrack1 has the management server role for the BMC and IPU-Gateway management network.

An RNIC spine switch can be used in an IPU‑POD₁₂₈ if required. With an RNIC spine switch installed in the system, all the IPU-M2000s in the IPU‑POD₁₂₈ can be accessed by both the lrack1 and lrack2 servers in the RDMA network (data plane). Without the spine switch, only local IPU-POD servers can access their local IPU-M2000s over the RDMA network - so the lrack1 server can only access the IPU-M2000s in lrack1 and the lrack2 server can only access the IPU-M2000s in lrack2. Since the lrack1 server is used as the management server, it can access the lrack2 IPU-M2000s using the management network, with or without a spine switch.

You need to upgrade the IP addressing on both IPU‑POD₆₄ racks to the IP address scheme required for IPU-PODs larger than IPU‑POD₆₄. The default factory IPU‑POD₆₄ setup has no rack number indication in the IP addresses, however this is required for larger IPU-PODs. With this updated IP address scheme multiple IPU‑POD₆₄ racks can be connected together and can also form part of a larger IPU-POD later on without having to change their IP addresses.
- The default factory scheme for IPU‑POD₆₄ racks as shipped is 10.1.x.z.
- The z values are for the IPU-M2000s in the rack
- The updated IP address scheme required for the IPU‑POD₁₂₈ uses 10.x.y.z where y = the logical rack number (lrack) which is normally identical to the IPU-POD number
- The x values stay the same: x=1 for BMC, x=2 for IPU-Gateway, x=3 for management server port, x=5 for data RNIC
Lrack1 hosts a management server running the V-IPU Management Controller, 1GbE management DHCP server, as well as NTP and syslog servers for the IPU‑POD₁₂₈.
Lrack2 has a limited management server role if no spine switch is being used to fully connect all the IPU‑POD₁₂₈ servers with all the IPU-M2000s across the two racks. In this case (not fully connected) the RNIC interfaces on the IPU-M2000s are still served by their local (existing) DHCP server on each IPU‑POD₆₄ rack, but this time with a new IP address that identifies the lrack number.

Note

If there is no spine switch you will not be able to reach the RNICs on lrack2 when checking the status with rack_tool from lrack1 since there is no inter-rack 100GbE connectivity.

Note

If any of your IPU‑POD₆₄ racks have DHCP config files named vlan-14.conf these should be changed to vlan-13.conf. You should also check the port names for the rack switches - any that are named vlan-14 should be changed to vlan-13.

8.2. Useful resources

For more details on using V-IPU you can reference the V-IPU user guide and V-IPU administrator guide. The BMC user guide also contains relevant information.

8.3. IP addressing

The previous template for management network IP addresses (10.x.y.z) has been modified since logical rack (lrack) numbers are now needed in the IP address. This information was previously given as the second octet of the IP address (x), but is now moved to the third octet (y) to ease subnetting based on interface types.

This means that new IP addresses are of the form:

10.<interface type>.<lrack#>.<IPU-M2000 number>

The interface types are as follows:

1 – net 10.1.0.0/16 - IPU-M2000 BMC network interfaces, including management server(s)’ interface(s) facing this network
2 – net 10.2.0.0/16 - IPU-M2000 IPU-Gateway network interfaces, including management server(s)’ interface(s) facing this network
3 – net 10.3.0.0/16 - IPU-POD server´s management network interfaces (OS – not BMC) - not used for interface type 1 or 2 networks, but closed management server network
4 – Not used
5 – net 10.5.0.0/16 - RDMA interfaces on IPU-M2000s and IPU-POD servers
6 – net 10.6.0.0/16 - IPU-POD server’s own BMC network interfaces, including management server(s)’ interface(s) facing this network
7 – net 10.7.0.0/16 - switch management network interfaces, including management server(s)’ interface(s) facing this network
8 – net 10.8.0.0/16 – PDU network interfaces, including management server(s)’ interface(s) facing this network

From this you can see that:

The IPU-M2000 BMCs have IP addresses: 10.1.<lrack#>.<IPU-M2000 number=1..16>
The IPU-M2000 IPU-Gateway chips have IP addresses: 10.2.<lrack#>.<IPU-M2000 number=1..16>
The IPU-M2000 RNIC ports have IP addresses: 10.5.<lrack#>.<IPU-M2000 number=1..16>

The management server ports that face the IPU-M2000 BMC, IPU-Gateway and V-IPU management subnet (V-IPU CLI from one host) should have IP addresses that match the IP subnets they will reach:

10.1.<lrack#>.150 (BMC network)
10.2.<lrack#>.150 (IPU-Gateway network)
10.3.<lrack#>.150 (V-IPU management subnet)

as shown in this example:

user: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f4:02:70:b9:18:3e brd ff:ff:ff:ff:ff:ff
    inet 10.1.7.150/16 brd 10.1.255.255 scope global eno1
    valid_lft forever preferred_lft forever
    inet 10.2.7.150/16 brd 10.2.255.255 scope global eno1
    valid_lft forever preferred_lft forever
    inet 10.3.7.150/16 brd 10.3.255.255 scope global eno1
    valid_lft forever preferred_lft forever
    ....

8.4. Merging IPU‑POD₆₄ racks to create IPU‑POD₁₂₈

8.4.1. Networking pre-requisites

Make sure that the lrack1 and lrack2 1GbE management switches are trunked together on a site-specific switch that takes VLAN 13 uplinks from each rack’s management switch and connects VLAN13 across both IPU‑POD₆₄ racks as one L2 broadcast domain.

Warning

These trunks must be disabled for VLAN13 traffic forwarding during the phased merge to a single IPU‑POD₁₂₈ since the migration steps require the IPU‑POD₆₄ racks to be fully isolated during the upgrade to the new IP address scheme.

If your IPU‑POD₆₄ 1GbE switches have the VLAN13 ports named as VLAN-14 or VLAN14 then you should correct them.
Make sure the DHCP files in both the lrack1 and lrack2 management servers are named vlan-13.conf not vlan-14.conf:
$ sudo find . -name vlan-14.conf /etc/dhcp/dhcpd.d/vlan-14.conf
If you do have to correct this then any references to these files should also be fixed in /etc/dhcp/dhcpd.conf.
Turn off V-IPU services and DHCP services
To turn off the V-IPU services and DHCP services, use these commands:
sudo systemctl stop vipu-server.service sudo systemctl stop isc-dhcp-server.service

8.4.2. Phase 1: Edit configuration files

A user with sudo rights needs to edit the configuration files first.

Warning

It is important that you do NOT apply the new configuration until you have carried out all the steps in this section and are ready to embark on Section 8.4.3, Phase 2: Activate new configuration.

Step 1: For lrack1 and lrack2

1a) Copy rack_config.json to rack_config.json_pod128 in the same directory:

cp  /home/ipuuser/.rack_tool/rack_config.json /home/ipuuser/.rack_tool/rack_config.json_pod128

You then need to rewrite the rack_config.json_pod128 config file to match the IPU‑POD₁₂₈ IP addresses as shown in Section 8.7.3, Lrack1: rack_config.json file.

Doing this means that you will be able to run rack_tool commands from the lrack1 management server for all the IPU-M2000s, in both IPU‑POD₆₄ racks (32 IPU-M2000s). Upgrading IPU-M2000 software for all 32 IPU-M2000s can then be run as a single operation.

1b) RNIC entries in lrack1 and lrack2 rack_tool.json files

System fully connected with spine switches

If your IPU‑POD₁₂₈ is using spine switches then you need to keep all the entries for the RNIC interfaces on both lrack1 and lrack2 rnic_ip in the lrack1 rack_tool.json file. This file will also contain all the IPU-Gateway and BMC interfaces for both lrack1 and lrack2. The lrack2 rack_tool.json file is not required as lrack1 will be used for all rack_tool operations.

System NOT fully connected with spine switches

If you are not using spine switches then you need to keep the entries for all the IPU-Gateway and BMC interfaces for both lrack1 and lrack2 in the lrack1 rack_tool.json file, but only the lrack1 RNIC interfaces rnic_ip. You need to list the lrack2 RNIC interfaces rnic_ip in the lrack2 rack_tool.json file instead.

Note

There is no inter-rack RDMA connectivity (data plane) between the two IPU‑POD₆₄ racks unless there is a RoCE spine switch that brings the leaf switches together. This means that, on the data plane, the lrack1 server(s) cannot access the lrack2 IPU-M2000s, and the lrack2 server(s) cannot access the lrack1 IPU-M2000s. The lrack1 server(s) can reach the lrack2 IPU-M2000s using the management network (control plane) for software updates.

Note

Kubernetes will not be supported on the IPU‑POD₁₂₈ unless there are spine switches providing inter-rack RDMA connectivity.

Step 2: DHCP config files for lrack1 and lrack2

The DHCP config file vlan-13.conf contains the ports for the IPU-M2000 IPU-Gateway and BMC ports. You need to add a copy of this file into a directory called /etc/dhcp/dhcpd.d/lrack#, where # denotes the rack (so 7 or 8 in our example). You also need to copy over the vlan-11.conf file.

2a) Lrack1

sudo mkdir /etc/dhcp/dhcpd.d/lrack1
sudo mkdir /etc/dhcp/dhcpd.d/lrack2
sudo cp /etc/dhcp/dhcpd.d/vlan-13.conf /etc/dhcp/dhcpd.d/lrack1
sudo cp /etc/dhcp/dhcpd.dvlan-11.conf /etc/dhcp/dhcpd.d/lrack1

2b) Lrack2

sudo mkdir /etc/dhcp/dhcpd.d/lrack2
sudo cp /etc/dhcp/dhcpd.d/vlan-11.conf /etc/dhcp/dhcpd.d/lrack2
sudo scp /etc/dhcp/dhcpd.d/vlan-13.conf ipuuser@pod7:/etc/dhcp/dhcpd.d/lrack2

2c) Lrack1 (required for spine switches)

You will need to run the following command if you have spine switches. If you don’t have spine switches then it is not necessary, however, if you might add spine switches in the future then it is a good idea to run it now so that lrack1 is prepared.

sudo scp /etc/dhcp/dhcpd.d/vlan-11.conf ipuuser@pod7:/etc/dhcp/dhcpd.d/lrack2

Step 3: Edit DHCP config files for lrack1

You need to edit the following management network DHCP files on lrack1:

/etc/dhcp/dhcpd.d/lrack1/vlan-13.conf
/etc/dhcp/dhcpd.d/lrack2/vlan-13.conf

Step 4: Netplan setup

There are examples of the nNtplan files that you are required to edit in Section 8.7, /etc/netplan files. There are also descriptions of how to edit them. You need to carry out the Netplan setup changes in this step in the order they are given (4a - 4d).

4a) BMC management

Create a Netplan setup for the BMC management subnet interface of the lrack1 management server using the new management network address 10.1.7.150.

4b) IPU-Gateway management

Create a Netplan setup for the IPU-Gateway management subnet interface of the lrack1 management server using the new management network address 10.2.7.150.

4c) Lrack1 RNIC interface

Create a Netplan setup for the lrack1 management server RNIC interface using the new management network address 10.5.7.150.

4d) lrack2 RNIC interface

Note

This part of the step is only required if there is no spine switch enabling full RDMA connectivity between the two IPU‑POD₆₄ racks.

Create a Netplan setup for the lrack2 management server RNIC interface using the new management network address 10.5.8.150.

Step 5: Update vlan-11.conf files

Modify /etc/dhcp/dhcpd.d/lrack1/vlan-11.conf on lrack1 using the new network addresses. For example:

# server RNIC addresses also set by the 150 server’s DHCP server
# ONLY: if rack has 4 servers
# lr7-server1mx is setup using netplan to 10.5.7.150
host lr7-server2mx { hardware ethernet 1c:36:da:4b:ea:ef; fixed-address 10.5.7.151; }
host lr7-Server3mx { hardware ethernet 0c:44:a1:20:7c:83; fixed-address 10.5.7.152; }
host lr7-Server4mx { hardware ethernet 0c:44:a1:20:80:a3; fixed-address 10.5.7.153; }

Once you have edited this file then you follow 2) or 3) depending on whether you have spine switches or not.

Either: 2) Modify /etc/dhcp/dhcpd.d/lrack2/vlan-11.conf on lrack1 using the new network addresses.

Note

This is only required if you have spine switches enabling full RDMA connectivity between the two IPU‑POD₆₄ racks.

# server RNIC addresses also set by the Pod7 management server’s DHCP server
# ONLY: if rack has 4 servers
host lr8-server1mx { hardware ethernet 1c:36:da:4b:ea:ef; fixed-address 10.5.8.150; }
host lr8-server2mx { hardware ethernet 1c:36:da:4b:ea:ef; fixed-address 10.5.8.151; }
host lr8-Server3mx { hardware ethernet 0c:44:a1:20:7c:83; fixed-address 10.5.8.152; }
host lr8-Server4mx { hardware ethernet 0c:44:a1:20:80:a3; fixed-address 10.5.8.153; }

Or: 3) Modify /etc/dhcp/dhcpd.d/lrack2/vlan-11.conf on lrack2 using the new network addresses.

Note

This is only required if you do NOT have spine switches enabling full RDMA connectivity between the two IPU‑POD₆₄ racks.

# server RNIC addresses also set by the Pod7 management server’s DHCP server
# ONLY: if rack has 4 servers
# lr8-server1mx is setup using netplan to 10.5.8.150
host lr8-server2mx { hardware ethernet 1c:36:da:4b:ea:ef; fixed-address 10.5.8.151; }
host lr8-Server3mx { hardware ethernet 0c:44:a1:20:7c:83; fixed-address 10.5.8.152; }
host lr8-Server4mx { hardware ethernet 0c:44:a1:20:80:a3; fixed-address 10.5.8.153; }

Next section requires changing so it no longer describes a live migration , but should describe migration from a reference POD64

8.4.3. Phase 2: Activate new configuration

This section describes how to activate the new configuration to combine the two IPU‑POD₆₄ racks into an IPU‑POD₁₂₈. While this is ongoing users will not be able to use either of the two IPU‑POD₆₄ racks.

Warning

It is important that you do NOT apply the new configuration in this phase until you have carried out all the steps in Section 8.4.2, Phase 1: Edit configuration files.

Step 6: Inform users of down time

If the IPU‑POD₆₄ rack(s) are in use you need to inform all users that they will be unavailable for use while they are switched over to form a single IPU‑POD₁₂₈. This activiation of the new configuration will generally take a couple of hours.

Warning

You need to follow these instructions in the order they are given to avoid problems with the configuration activation. Do NOT enable VLAN13 trunking between the lrack1 and lrack2 management switches at this stage.

Step 7: lrack2 DHCP server

7a) The local lrack2 DHCP server must not service requests from the 1GbE management interface vlan-13.conf. You therefore need to disable vlan-13.conf by editing the current DHCP setup ipum-dhcp.conf and removing the vlan-13.conf file, as shown in Section 8.6.3, Lrack2: /etc/dhcp/dhcpd.d/ipum-dhcp.conf.

Note

The following is only required if you do NOT have spine switches enabling full RDMA connectivity between the two IPU‑POD₆₄ racks.

7b) Use the DHCP setup and file location /etc/dhcp/dhcpd.d/lrack2/ for vlan-11.conf to use the new IP address scheme for lrack2. Then restart the DHCP server on lrack2 with this command:

sudo systemctl restart isc-dhcp-server

Step 8: lrack1 DHCP server

On lrack1 you need to edit /etc/dhcp/dhcpd.conf and include /etc/dhcp/dhcpd.d/ipum-dhcp.conf as described in Section 8.6.1, Lrack1 and lrack2: /etc/dhcp/dhcpd.conf. This dhcpd.conf file will then start pointing to the new /etc/dhcp/dhcpd.d/lrack1 and /etc/dhcp/dhcpd.d/lrack2 config files.

You then need to split the vlan-13.conf file into two files, one containing IPU-Gateway ports only, and the other containing BMC ports only.

Restart the DHCP server on lrack1 with this command:

sudo systemctl restart isc-dhcp-server

Step 9: Restart IPU-M2000s on lrack1

Next you need to restart all the IPU-M2000s on lrack1 by using the BMC controller to manually power cycle them with the following commands:

rack_tool.py power-cycle
rack_tool.py run-command –c reboot –d bmc

This will use the old /home/ipuuser/.rack_tool/rack_config.json file so no network setup change should have been activated before this.

The IPU-M2000s will reboot and request new IP, as enabled above.

Step 10: Netplan configuration on lrack1

You need to change to the new Netplan configuration for management interfaces on lrack1 and then activate it.

Step 11: Update rack_config files on lrack1

11a) Save the old rack_config.json file on lrack1 as this is not required any more:

cp /home/ipuuser/.rack_tool/rack_config.json /home/ipuuser/.rack_tool/rack_config.json_pod64

11b) Update the rack_config.json file on lrack1 to an IPU‑POD₁₂₈ setup:

cp /home/ipuuser/.rack_tool/rack_config.json_pod128 /home/ipuuser/.rack_tool/rack_config.json

More details about the contents of rack_config.json_pod128 can be found in Step 1: For lrack1 and lrack2.

11c) Use rack_tool on lrack1 to verify that the new lrack1 IPU-M2000, IPU-Gateway and BMC IP addresses have been set up correctly:

rack_tool.py status

The lrack2 IPU-M2000s will fail at this point as they still have their old IP addresses and there is no trunking over VLAN13 between the switches enabled yet.

Step 12: Restart IPU-M2000s on lrack2

12a) Restart the IPU-M2000s on lrack2 by using the BMC controller to manually power cycle them with the following commands run on lrack2:

rack_tool.py power-cycle
rack_tool.py run-command –c reboot –d bmc

This will use the old /home/ipuuser/.rack_tool/rack_config.json file.

12b) Save the old rack_config.json file on lrack2 as this is not required any more:

cp /home/ipuuser/.rack_tool/rack_config.json /home/ipuuser/.rack_tool/rack_config.json_pod64

12c) Update the rack_config.json file on lrack2 to an IPU‑POD₁₂₈ setup:

cp /home/ipuuser/.rack_tool/rack_config.json_pod128 /home/ipuuser/.rack_tool/rack_config.json

More details about the contents of rack_config.json_pod128 can be found in Step 1: For lrack1 and lrack2.

12d) Use rack_tool on lrack2 to verify that the new lrack2 IPU-M2000 IP addresses have been set up correctly:

rack_tool.py status

Step 13: Verify IPU-M2000 interface access

Note

this step is only required if the IPU‑POD₁₂₈ is NOT fully connected with spine switches.

Run the following rack_tool command on lrack1 to verify that there is no access to the IPU-M2000 interfaces on lrack2 from lrack1:

rack_tool.py status

The IPU-M2000 RNICs on lrack2 will fail as they are not reachable from lrack1 unless there are spine switches.

Step 14: Create V-IPU cluster on lrack1

Create a new V-IPU cluster on lrack1 and add all the V-IPU agents from both IPU‑POD₆₄ racks (lrack1 and lrack2) to the cluster. There is one V-IPU agent per IPU-Gateway on each IPU-M2000 so there will be 32 in total. Make sure the cluster is added as a torus if the IPU-Link and GW-Link cables are connected as a loop. To do this you need to use the --cluster-topology looped and --topology torus options. The --cluster-topology argument defines the GW-Link topology (horizontal torus) and the --topology argument defines the IPU-Link topology (vertical torus).

For example:

vipu-admin create cluster cl128 --num-ilds 2 --topology torus --cluster-topology looped --agents ${ALL_IPUM_NAMES_FROM_vipu_list_agents}

Step 15: Test access to V-IPU agents

Use the V-IPU test command to test access to V-IPU agents for both lrack1 and lrack2. For example:

vipu-admin test cluster cl128

For more details on how to use this command refer to the V-IPU guides (V-IPU user guide and V-IPU administrator guide.)

Step 16: Create partitions on lrack1

Use the IPUOF_VIPU_API environment variables to create user specific partitions on lrack1 with the new IP address setup. These environment variables are:

IPUOF_VIPU_API_HOST: The IP address of the server running the V-IPU controller. Required.
IPUOF_VIPU_API_PORT: The port to connect to the V-IPU controller. Optional. The default is 8090.
IPUOF_VIPU_API_PARTITION_ID: The name of the partition to use. Required.
IPUOF_VIPU_API_GCD_ID: The ID of the GCD you want to use. Required for multi-GCD systems.
IPUOF_VIPU_API_TIMEOUT: Set the time-out for client calls in seconds. Optional. Default 200.

More details about how to use these environment variables are given in the V-IPU user guide.

Use gc-info on the management server to check that the IPUs in the partition are visible. More details about how to use gc-info are given here.

Step 17: lrack1 RNIC addresses

Run netplan apply on all lrack1 servers so that they get new RNIC IP addresses. Then use gc-info to check that they have access to the partition setup and that all lrack1 servers can reach all the IPU-M2000s they should have access to.

Step 18: lrack2 RNIC addresses

Run netplan apply on all lrack2 servers so that they get new RNIC IP addresses. Then use gc-info to check that they have access to the partition setup and that all lrack2 servers can reach all the IPU-M2000s they should have access to.

Step 19: rsyslog.d

Edit /home/ipuuser/.rack_tool/root-overlay/etc/rsyslog.d to point to lrack1 IP (10.2.7.150).

Step 20: chrony.conf

Edit /home/ipuuser/.rack_tool/root-overlay/etc/chrony/chrony.conf to point to lrack1 IP (10.2.7.150).

Step 21: Refresh overlay files on lrack1

Use rack_tool to refresh the overlay files.

rack_tool.py update-root-overlay

Step 22: Check IPU-M2000s logging to lrack1

You need to check whether all 32 IPU-M2000s are logging to the lrack1 syslog. This will either be located in /var/log/syslog or in a specified location if you have filters in place for IPU-M2000 logs. A common syslog filter is /etc/rsyslog.d/99_ipum.conf:

# Graphcore.ai - Config Management
# This file is managed by puppet, so any manual changes will be overwritten automatically!
$template tplremote,"%timegenerated% %HOSTNAME% %fromhost-ip% %syslogtag%%msg:::drop-last-lf%\n"
$template bmclogdir,"/var/log/ipumlogs/bmclogs/%fromhost-ip%.log"
$template gwlogdir,"/var/log/ipumlogs/gwlogs/%fromhost-ip%.log"

if $fromhost-ip startswith '10.1' then ?bmclogdir;tplremote
if $fromhost-ip startswith '10.2' then ?gwlogdir;tplremote
& ~

Run this command to check if the syslog server is running on lrack1’s management server (server1):

systemctl | grep syslog

If the syslog server is not running then you need to start it.

Step 23: Check IPU-M2000s have NTP date and time

You need to check whether all 32 IPU-M2000s are taking their date and time from the NTP server on lrack1.

This can be checked by ssh to itadmin @ IPU-Gateway and checking the system time/date.

Step 24: Run ML application

Run a machine learning application on the IPU‑POD₁₂₈ to check that is it correctly using both lrack1 and lrack2 resources. This will verify that the merge to an IPU‑POD₁₂₈ has been successful. You can find examples of ML applications to run in Graphcore’s GihHub examples repo.

8.5. IPU-M2000 setup files

8.5.1. Syslog and chrony on the IPU-Gateway

Update the chrony.conf file in the root_overlay files.

Below is a truncated example of a chrony.conf file.

(venv) ipuuser@lr23-poplar1:~/IPU_M_SW-2.3.0$ more ~/.rack_tool/root-overlay/etc/chrony/chrony.conf
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usuable directives.
pool 10.2.7.150 iburst

Update the rsyslog.d configuration file in the root_overlay files.

(venv) ipuuser@lr23-poplar1:~/IPU_M_SW-2.3.0$ more ~/.rack_tool/root-overlay/etc/rsyslog.d/99-loghost.conf
*.* @10.2.7.150:514

Update the root_overlay files with rack_tool:

rack_tool.py update-root-overlay

8.5.2. Syslog on BMC

8.6. DHCP files

8.6.1. Lrack1 and lrack2: /etc/dhcp/dhcpd.conf

This setup assumes that the IPU‑POD₆₄ switches, server BMCs and PDUs are connected to the IT network (see Section 3, IPU-POD64 rack assembly), leaving the BMC, IPU-Gateway and RDMA networking to connect to the management server(s). The management server in lrack1 is for all BMC, IPU-Gateway, and lrack1 RDMA networking; the management server in lrack2 is the DHCP server for lrack2 RDMA networking.

root@gbnwp-pod013-1:/home/user1# cat /etc/dhcp/dhcpd.conf
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
authoritative;
log-facility local7;

include "/etc/dhcp/dhcpd.d/ipum-dhcp.conf";

8.6.2. Lrack1: /etc/dhcp/dhcpd.d/ipum-dhcp.conf

root@gbnwp-pod007-1:/home/user1# cat /etc/dhcp/dhcpd.d/ipum-dhcp.conf
shared-network "L2" {
# IPU-M BMC subnet
subnet 10.1.0.0  netmask 255.255.0.0 {
    option subnet-mask         255.255.0.0;
    default-lease-time         600;
    max-lease-time             7200;
    option ntp-servers         10.1.7.150;
}

# IPU-M IPU-Gateway subnet
subnet 10.2.0.0  netmask 255.255.0.0 {
    option subnet-mask         255.255.0.0;
    default-lease-time         600;
    max-lease-time             7200;
    option ntp-servers         10.2.7.150;
}
# V-IPU management subnet i.e. V-IPU CLI from one host
# to V-IPU Controller on management server
subnet 10.3.0.0  netmask 255.255.0.0 {
    option subnet-mask         255.255.0.0;
    default-lease-time         600;
    max-lease-time             7200;
    option ntp-servers         10.3.7.150;
}
}

# list all lrack numbers that are served by this host here
# lrack1
include "/etc/dhcp/dhcpd.d/lrack1/ipum-bmc.conf";
include "/etc/dhcp/dhcpd.d/lrack1/ipum-gw.conf";
include "/etc/dhcp/dhcpd.d/lrack1/poplar-mgmt.conf";
# lrack2
include "/etc/dhcp/dhcpd.d/lrack2/ipum-bmc.conf";
include "/etc/dhcp/dhcpd.d/lrack2/ipum-gw.conf";
include "/etc/dhcp/dhcpd.d/lrack2/poplar-mgmt.conf";

# IPU-M IPUoF subnet
subnet 10.5.0.0 netmask 255.255.0.0 {
    option subnet-mask         255.255.0.0;
    default-lease-time         600;
    max-lease-time             7200;
    range                      10.5.7.240 10.5.7.244;
}
# list all lrack numbers that is served by this host here
include "/etc/dhcp/dhcpd.d/lrack1/ipum-rnic.conf";

8.6.3. Lrack2: /etc/dhcp/dhcpd.d/ipum-dhcp.conf

root@gbnwp-pod8-1:/home/hhoeg# cat /etc/dhcp/dhcpd.d/ipum-dhcp.conf
shared-network "L2" {
# IPU-M BMC subnet
# V-IPU management subnet i.e. V-IPU CLI from one host
# to V-IPU Controller on management server
subnet 10.3.0.0  netmask 255.255.0.0 {
    option subnet-mask         255.255.0.0;
    default-lease-time         600;
    max-lease-time             7200;
    option ntp-servers         10.3.8.150;
}
}

# list all lrack numbers that is served by this host here
# lrack2
include "/etc/dhcp/dhcpd.d/lrack2/poplar-mgmt.conf";

# IPU-M IPUoF subnet
subnet 10.5.0.0 netmask 255.255.0.0 {
    option subnet-mask         255.255.0.0;
    default-lease-time         600;
    max-lease-time             7200;
    range                      10.5.8.240 10.5.8.244;
}
# list all lrack numbers that is served by this host here
include "/etc/dhcp/dhcpd.d/lrack2/ipum-rnic.conf";

8.6.4. Lrack1 and lrack2: /etc/dhcp/dhcpd.d/ files

root@gbnwp-pod013-1:/etc/dhcp/dhcpd.d# ll
total 16
drwxrwx--- 4 root dhcpd  164 Mar  1 09:57 ./
dr-xrwxr-x 6 root dhcpd 4096 Feb 18 16:46 ../
-rw-r--r-- 1 root root  1449 Mar  1 09:55 ipum-dhcp.conf
drwxr-xr-x 2 root root    93 Mar 11 15:28 lrack1/
drwxr-xr-x 2 root root    93 Mar  1 09:57 lrack2/

8.6.5. Lrack1: /etc/dhcp/dhcpd.d/lrack1 files

ipum-bmc.conf

root@gbnwp-pod7-1:/etc/dhcp/dhcpd.d/lrack1# cat ipum-bmc.conf

host lr7-ipum1bmc {  hardware ethernet 70:69:79:20:24:D0; fixed-address 10.1.7.1; }
host lr7-ipum2bmc {  hardware ethernet 70:69:79:20:14:48; fixed-address 10.1.7.2; }
host lr7-ipum3bmc {  hardware ethernet 70:69:79:20:12:BC; fixed-address 10.1.7.3; }
host lr7-ipum4bmc {  hardware ethernet 70:69:79:20:13:2C; fixed-address 10.1.7.4; }
host lr7-ipum5bmc {  hardware ethernet 70:69:79:20:13:04; fixed-address 10.1.7.5; }
host lr7-ipum6bmc {  hardware ethernet 70:69:79:20:14:68; fixed-address 10.1.7.6; }
host lr7-ipum7bmc {  hardware ethernet 70:69:79:20:14:D4; fixed-address 10.1.7.7; }
host lr7-ipum8bmc {  hardware ethernet 70:69:79:20:14:BC; fixed-address 10.1.7.8; }
host lr7-ipum9bmc {  hardware ethernet 70:69:79:20:14:38; fixed-address 10.1.7.9; }
host lr7-ipum10bmc { hardware ethernet 70:69:79:20:14:94; fixed-address 10.1.7.10; }
host lr7-ipum11bmc { hardware ethernet 70:69:79:20:13:F4; fixed-address 10.1.7.11; }
host lr7-ipum12bmc { hardware ethernet 70:69:79:20:12:74; fixed-address 10.1.7.12; }
host lr7-ipum13bmc { hardware ethernet 70:69:79:20:12:6C; fixed-address 10.1.7.13; }
host lr7-ipum14bmc { hardware ethernet 70:69:79:20:14:84; fixed-address 10.1.7.14; }
host lr7-ipum15bmc { hardware ethernet 70:69:79:20:16:BC; fixed-address 10.1.7.15; }
host lr7-ipum16bmc { hardware ethernet 70:69:79:20:17:A4; fixed-address 10.1.7.16; }

ipum-gw.conf

root@gbnwp-pod7-1:/etc/dhcp/dhcpd.d/lrack1# cat ipum-gw.conf
host lr7-ipum1gw {   hardware ethernet 70:69:79:20:24:D1; fixed-address 10.2.7.1; }
host lr7-ipum2gw {   hardware ethernet 70:69:79:20:14:49; fixed-address 10.2.7.2; }
host lr7-ipum3gw {   hardware ethernet 70:69:79:20:12:BD; fixed-address 10.2.7.3; }
host lr7-ipum4gw {   hardware ethernet 70:69:79:20:13:2D; fixed-address 10.2.7.4; }
host lr7-ipum5gw {   hardware ethernet 70:69:79:20:13:05; fixed-address 10.2.7.5; }
host lr7-ipum6gw {   hardware ethernet 70:69:79:20:14:69; fixed-address 10.2.7.6; }
host lr7-ipum7gw {   hardware ethernet 70:69:79:20:14:D5; fixed-address 10.2.7.7; }
host lr7-ipum8gw {   hardware ethernet 70:69:79:20:14:BD; fixed-address 10.2.7.8; }
host lr7-ipum9gw {   hardware ethernet 70:69:79:20:14:39; fixed-address 10.2.7.9; }
host lr7-ipum10gw {  hardware ethernet 70:69:79:20:14:95; fixed-address 10.2.7.10; }
host lr7-ipum11gw {  hardware ethernet 70:69:79:20:13:F5; fixed-address 10.2.7.11; }
host lr7-ipum12gw {  hardware ethernet 70:69:79:20:12:75; fixed-address 10.2.7.12; }
host lr7-ipum13gw {  hardware ethernet 70:69:79:20:12:6D; fixed-address 10.2.7.13; }
host lr7-ipum14gw {  hardware ethernet 70:69:79:20:14:85; fixed-address 10.2.7.14; }
host lr7-ipum15gw {  hardware ethernet 70:69:79:20:16:BD; fixed-address 10.2.7.15; }
host lr7-ipum16gw {  hardware ethernet 70:69:79:20:17:A5; fixed-address 10.2.7.16; }

ipum-rnic.conf

root@gbnwp-pod7-1:/etc/dhcp/dhcpd.d/lrack1# cat ipum-rnic.conf
host lr7-ipum1mx {  hardware ethernet 0c:42:a1:78:90:35; fixed-address 10.5.7.1; }
host lr7-ipum2mx {  hardware ethernet 0c:42:a1:78:90:d5; fixed-address 10.5.7.2; }
host lr7-ipum3mx {  hardware ethernet 0c:42:a1:78:7e:55; fixed-address 10.5.7.3; }
host lr7-ipum4mx {  hardware ethernet 0c:42:a1:78:82:65; fixed-address 10.5.7.4; }
host lr7-ipum5mx {  hardware ethernet 0c:42:a1:84:ec:3d; fixed-address 10.5.7.5; }
host lr7-ipum6mx {  hardware ethernet 0c:42:a1:78:95:65; fixed-address 10.5.7.6; }
host lr7-ipum7mx {  hardware ethernet 0c:42:a1:78:96:c5; fixed-address 10.5.7.7; }
host lr7-ipum8mx {  hardware ethernet 0c:42:a1:84:f5:85; fixed-address 10.5.7.8; }
host lr7-ipum9mx {  hardware ethernet 0c:42:a1:84:ea:85; fixed-address 10.5.7.9; }
host lr7-ipum10mx { hardware ethernet 0c:42:a1:78:7e:65; fixed-address 10.5.7.10; }
host lr7-ipum11mx { hardware ethernet 0c:42:a1:78:8a:0d; fixed-address 10.5.7.11; }
host lr7-ipum12mx { hardware ethernet 0c:42:a1:78:85:55; fixed-address 10.5.7.12; }
host lr7-ipum13mx { hardware ethernet 0c:42:a1:78:89:f5; fixed-address 10.5.7.13; }
host lr7-ipum14mx { hardware ethernet 0c:42:a1:78:8a:4d; fixed-address 10.5.7.14; }
host lr7-ipum15mx { hardware ethernet 0c:42:a1:84:e7:bd; fixed-address 10.5.7.15; }
host lr7-ipum16mx { hardware ethernet 0c:42:a1:78:87:25; fixed-address 10.5.7.16; }

8.6.6. Lrack1: /etc/dhcp/dhcpd.d/lrack2 files

ipum-bmc.conf

root@gbnwp-pod7-1:/etc/dhcp/dhcpd.d/lrack2# cat ipum-bmc.conf
host lr8-ipum1bmc {  hardware ethernet 70:69:79:20:24:D0; fixed-address 10.1.8.1; }
host lr8-ipum2bmc {  hardware ethernet 70:69:79:20:14:48; fixed-address 10.1.8.2; }
host lr8-ipum3bmc {  hardware ethernet 70:69:79:20:12:BC; fixed-address 10.1.8.3; }
host lr8-ipum4bmc {  hardware ethernet 70:69:79:20:13:2C; fixed-address 10.1.8.4; }
host lr8-ipum5bmc {  hardware ethernet 70:69:79:20:13:04; fixed-address 10.1.8.5; }
host lr8-ipum6bmc {  hardware ethernet 70:69:79:20:14:68; fixed-address 10.1.8.6; }
host lr8-ipum7bmc {  hardware ethernet 70:69:79:20:14:D4; fixed-address 10.1.8.7; }
host lr8-ipum8bmc {  hardware ethernet 70:69:79:20:14:BC; fixed-address 10.1.8.8; }
host lr8-ipum9bmc {  hardware ethernet 70:69:79:20:14:38; fixed-address 10.1.8.9; }
host lr8-ipum10bmc { hardware ethernet 70:69:79:20:14:94; fixed-address 10.1.8.10; }
host lr8-ipum11bmc { hardware ethernet 70:69:79:20:13:F4; fixed-address 10.1.8.11; }
host lr8-ipum12bmc { hardware ethernet 70:69:79:20:12:74; fixed-address 10.1.8.12; }
host lr8-ipum13bmc { hardware ethernet 70:69:79:20:12:6C; fixed-address 10.1.8.13; }
host lr8-ipum14bmc { hardware ethernet 70:69:79:20:14:84; fixed-address 10.1.8.14; }
host lr8-ipum15bmc { hardware ethernet 70:69:79:20:16:BC; fixed-address 10.1.8.15; }
host lr8-ipum16bmc { hardware ethernet 70:69:79:20:17:A4; fixed-address 10.1.8.16; }

ipum-gw.conf

root@gbnwp-pod7-1:/etc/dhcp/dhcpd.d/lrack2# cat ipum-gw.conf
host lr8-ipum1gw {   hardware ethernet 70:69:79:20:24:D1; fixed-address 10.2.8.1; }
host lr8-ipum2gw {   hardware ethernet 70:69:79:20:14:49; fixed-address 10.2.8.2; }
host lr8-ipum3gw {   hardware ethernet 70:69:79:20:12:BD; fixed-address 10.2.8.3; }
host lr8-ipum4gw {   hardware ethernet 70:69:79:20:13:2D; fixed-address 10.2.8.4; }
host lr8-ipum5gw {   hardware ethernet 70:69:79:20:13:05; fixed-address 10.2.8.5; }
host lr8-ipum6gw {   hardware ethernet 70:69:79:20:14:69; fixed-address 10.2.8.6; }
host lr8-ipum7gw {   hardware ethernet 70:69:79:20:14:D5; fixed-address 10.2.8.7; }
host lr8-ipum8gw {   hardware ethernet 70:69:79:20:14:BD; fixed-address 10.2.8.8; }
host lr8-ipum9gw {   hardware ethernet 70:69:79:20:14:39; fixed-address 10.2.8.9; }
host lr8-ipum10gw {  hardware ethernet 70:69:79:20:14:95; fixed-address 10.2.8.10; }
host lr8-ipum11gw {  hardware ethernet 70:69:79:20:13:F5; fixed-address 10.2.8.11; }
host lr8-ipum12gw {  hardware ethernet 70:69:79:20:12:75; fixed-address 10.2.8.12; }
host lr8-ipum13gw {  hardware ethernet 70:69:79:20:12:6D; fixed-address 10.2.8.13; }
host lr8-ipum14gw {  hardware ethernet 70:69:79:20:14:85; fixed-address 10.2.8.14; }
host lr8-ipum15gw {  hardware ethernet 70:69:79:20:16:BD; fixed-address 10.2.8.15; }
host lr8-ipum16gw {  hardware ethernet 70:69:79:20:17:A5; fixed-address 10.2.8.16; }

8.6.7. Lrack2: /etc/dhcp/dhcpd.d/lrack2 files

Note

This setup is ONLY required on lrack2 if there is no spine switch. In this situation you will also need a DHCP server to run on lrack2.

ipum-rnic.conf

root@gbnwp-pod8-1:/etc/dhcp/dhcpd.d/lrack2# cat ipum-rnic.conf
host lr8-ipum1mx {  hardware ethernet 0c:42:a1:78:90:35; fixed-address 10.5.8.1; }
host lr8-ipum2mx {  hardware ethernet 0c:42:a1:78:90:d5; fixed-address 10.5.8.2; }
host lr8-ipum3mx {  hardware ethernet 0c:42:a1:78:7e:55; fixed-address 10.5.8.3; }
host lr8-ipum4mx {  hardware ethernet 0c:42:a1:78:82:65; fixed-address 10.5.8.4; }
host lr8-ipum5mx {  hardware ethernet 0c:42:a1:84:ec:3d; fixed-address 10.5.8.5; }
host lr8-ipum6mx {  hardware ethernet 0c:42:a1:78:95:65; fixed-address 10.5.8.6; }
host lr8-ipum7mx {  hardware ethernet 0c:42:a1:78:96:c5; fixed-address 10.5.8.7; }
host lr8-ipum8mx {  hardware ethernet 0c:42:a1:84:f5:85; fixed-address 10.5.8.8; }
host lr8-ipum9mx {  hardware ethernet 0c:42:a1:84:ea:85; fixed-address 10.5.8.9; }
host lr8-ipum10mx { hardware ethernet 0c:42:a1:78:7e:65; fixed-address 10.5.8.10; }
host lr8-ipum11mx { hardware ethernet 0c:42:a1:78:8a:0d; fixed-address 10.5.8.11; }
host lr8-ipum12mx { hardware ethernet 0c:42:a1:78:85:55; fixed-address 10.5.8.12; }
host lr8-ipum13mx { hardware ethernet 0c:42:a1:78:89:f5; fixed-address 10.5.8.13; }
host lr8-ipum14mx { hardware ethernet 0c:42:a1:78:8a:4d; fixed-address 10.5.8.14; }
host lr8-ipum15mx { hardware ethernet 0c:42:a1:84:e7:bd; fixed-address 10.5.8.15; }
host lr8-ipum16mx { hardware ethernet 0c:42:a1:78:87:25; fixed-address 10.5.8.16; }

8.7. /etc/netplan files

8.7.1. 1GbE management interface on lrack1 server

The 1GbE management interface is required to have the setup described in this section. The eno1 interface has three IP subnets for communicating with: BMC ports, IPU-Gateway ports, and between servers (for example V-IPU CLI to V-IPU controller on the management server, and Poplar instance to V-IPU controller).

3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether bc:97:e1:46:00:b6 brd ff:ff:ff:ff:ff:ff
    inet 10.1.7.150/16 brd 10.1.255.255 scope global eno1
    valid_lft forever preferred_lft forever
    inet 10.2.7.150/16 brd 10.2.255.255 scope global eno1
    valid_lft forever preferred_lft forever
    inet 10.3.7.150/16 brd 10.3.255.255 scope global eno1
    valid_lft forever preferred_lft forever
    inet6 fe80::be97:e1ff:fe46:b6/64 scope link
    valid_lft forever preferred_lft forever

8.7.2. RNIC interfaces on the servers

The IPU‑POD₁₂₈ contains two IPU‑POD₆₄ racks, and each of the IPU‑POD₆₄ racks can contain up to four servers. Each of these servers has an RNIC interface which needs to be configured.

You need to setup the RNIC interfaces as described in this section. If there are multiple servers on lrack1 (up to four), the management server (lrack1 server 1) will control the remaining servers (lrack1 servers 2, 3 and 4) with DHCP. lrack1 server 1 will be running the v-ipu server and also a DHCP server to control the IP addressing for the lrack1 IPU-M2000s and servers (lrack1 servers 2, 3 and 4). Server 1 on lrack2 only runs a DHCP server for the lrack2 IPU-M2000s and servers (lrack2 servers 2, 3 and 4).

7: enp161s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 0c:42:a1:1e:79:27 brd ff:ff:ff:ff:ff:ff
    inet 10.5.7.150/16 brd 10.5.255.255 scope global enp161s0f1
    valid_lft forever preferred_lft forever
    inet6 fe80::e42:a1ff:fe1e:7927/64 scope link
    valid_lft forever preferred_lft forever

This can be achieved by setting up the /etc/netplan/01-netcfg.yaml file as follows:

network:
version: 2
renderer: networkd
ethernets:
    eno1:
    addresses:
        - 10.1.7.150/16
        - 10.2.7.150/16
        - 10.3.7.150/16
    eno2:
    dhcp4: yes
    enp161s0f0:
    dhcp4: yes
    enp161s0f1:
    addresses:
        - 10.5.7.150/16

If there are more servers in the lrack1 rack (up to four), each will require an /etc/netplan/01-netcfg.yaml file with the enp161s0f1 interface edited accordingly. For example. server 2 in the rack would have /etc/netplan/01-netcfg.yaml as follows:

network:
version: 2
renderer: networkd
ethernets:
    eno2:
    dhcp4: yes
    enp161s0f0:
    dhcp4: yes
    enp161s0f1:
    addresses:
        - 10.5.7.151/16

Server 3 would have enp161s0f1 as 10.5.7.152/16 and server 4 10.5.7.153/16.

8.7.3. Lrack1: rack_config.json file

This file on lrack1 is required for software upgrades and connectivity checks. With a spine switch installed, this file will contain entries for all 32 IPu-M2000s in the IPU‑POD₁₂₈, as shown in the code example below. If there is no spine switch, this file will only contain entries for lrack1 IPU-M2000s and a similar rack_config.json file on lrack2, containing entries for lrack2 IPU-M2000s, will be required for RNIC connectivity checks on lrack2 IPU-M2000s.

For your own BMC and IPU-Gateway passwords, please ask on the Graphcore support portal https://www.graphcore.ai/support.

root@gbnwp-pod7-1:/etc/dhcp/dhcpd.d/lrack13# cat /home/ipuuser/.rack_tool/rack_config.json
{
    "gw_root_overlay": "/home/ipuuser/.rack_tool/root-overlay",
    "global_credentials": {
        "bmc_username": "root",
        "bmc_passwd": "AskGCForBMCPwd",
        "gw_username": "itadmin",
        "gw_passwd": "AskGCForGWPwd"
    },
    "machines": [
        {
            "name": "lr7_ipum1",
            "bmc_ip": "10.1.7.1",
            "gw_ip": "10.2.7.1",
            "rnic_ip": "10.5.7.1"
        },
        {
            "name": "lr7_ipum2",
            "bmc_ip": "10.1.7.2",
            "gw_ip": "10.2.7.2",
            "rnic_ip": "10.5.7.2"
        },
        {
            "name": "lr7_ipum3",
            "bmc_ip": "10.1.7.3",
            "gw_ip": "10.2.7.3",
            "rnic_ip": "10.5.7.3"
        },
        {
            "name": "lr7_ipum4",
            "bmc_ip": "10.1.7.4",
            "gw_ip": "10.2.7.4",
            "rnic_ip": "10.5.7.4"
        },
        {
            "name": "lr7_ipum5",
            "bmc_ip": "10.1.7.5",
            "gw_ip": "10.2.7.5",
            "rnic_ip": "10.5.7.5"
        },
        {
            "name": "lr7_ipum6",
            "bmc_ip": "10.1.7.6",
            "gw_ip": "10.2.7.6",
            "rnic_ip": "10.5.7.6"
        },
        {
            "name": "lr7_ipum7",
            "bmc_ip": "10.1.7.7",
            "gw_ip": "10.2.7.7",
            "rnic_ip": "10.5.7.7"
        },
        {
            "name": "lr7_ipum8",
            "bmc_ip": "10.1.7.8",
            "gw_ip": "10.2.7.8",
            "rnic_ip": "10.5.7.8"
        },
        {
            "name": "lr7_ipum9",
            "bmc_ip": "10.1.7.9",
            "gw_ip": "10.2.7.9",
            "rnic_ip": "10.5.7.9"
        },
        {
            "name": "lr7_ipum10",
            "bmc_ip": "10.1.7.10",
            "gw_ip": "10.2.7.10",
            "rnic_ip": "10.5.7.10"
        },
        {
            "name": "lr7_ipum11",
            "bmc_ip": "10.1.7.11",
            "gw_ip": "10.2.7.11",
            "rnic_ip": "10.5.7.11"
        },
        {
            "name": "lr7_ipum12",
            "bmc_ip": "10.1.7.12",
            "gw_ip": "10.2.7.12",
            "rnic_ip": "10.5.7.12"
        },
        {
            "name": "lr7_ipum13",
            "bmc_ip": "10.1.7.13",
            "gw_ip": "10.2.7.13",
            "rnic_ip": "10.5.7.13"
        },
        {
            "name": "lr7_ipum14",
            "bmc_ip": "10.1.7.14",
            "gw_ip": "10.2.7.14",
            "rnic_ip": "10.5.7.14"
        },
        {
            "name": "lr7_ipum15",
            "bmc_ip": "10.1.7.15",
            "gw_ip": "10.2.7.15",
            "rnic_ip": "10.5.7.15"
        },
        {
            "name": "lr7_ipum16",
            "bmc_ip": "10.1.7.16",
            "gw_ip": "10.2.7.16",
            "rnic_ip": "10.5.7.16"
        },
        {
            "name": "lr8_ipum1",
            "bmc_ip": "10.1.8.1",
            "gw_ip": "10.2.8.1",
            "rnic_ip": "10.5.8.1"
        },
        {
            "name": "lr8_ipum2",
            "bmc_ip": "10.1.8.2",
            "gw_ip": "10.2.8.2",
            "rnic_ip": "10.5.8.2"
        },
        {
            "name": "lr8_ipum3",
            "bmc_ip": "10.1.8.3",
            "gw_ip": "10.2.8.3",
            "rnic_ip": "10.5.8.3"
        },
        {
            "name": "lr8_ipum4",
            "bmc_ip": "10.1.8.4",
            "gw_ip": "10.2.8.4",
            "rnic_ip": "10.5.8.4"
        },
        {
            "name": "lr8_ipum5",
            "bmc_ip": "10.1.8.5",
            "gw_ip": "10.2.8.5",
            "rnic_ip": "10.5.8.5"
        },
        {
            "name": "lr8_ipum6",
            "bmc_ip": "10.1.8.6",
            "gw_ip": "10.2.8.6",
            "rnic_ip": "10.5.8.6"
        },
        {
            "name": "lr8_ipum7",
            "bmc_ip": "10.1.8.7",
            "gw_ip": "10.2.8.7",
            "rnic_ip": "10.5.8.7"
        },
        {
            "name": "lr8_ipum8",
            "bmc_ip": "10.1.8.8",
            "gw_ip": "10.2.8.8",
            "rnic_ip": "10.5.8.8"
        },
        {
            "name": "lr8_ipum9",
            "bmc_ip": "10.1.8.9",
            "gw_ip": "10.2.8.9",
            "rnic_ip": "10.5.8.9"
        },
        {
            "name": "lr8_ipum10",
            "bmc_ip": "10.1.8.10",
            "gw_ip": "10.2.8.10",
            "rnic_ip": "10.5.8.10"
        },
        {
            "name": "lr8_ipum11",
            "bmc_ip": "10.1.8.11",
            "gw_ip": "10.2.8.11",
            "rnic_ip": "10.5.8.11"
        },
        {
            "name": "lr8_ipum12",
            "bmc_ip": "10.1.8.12",
            "gw_ip": "10.2.8.12",
            "rnic_ip": "10.5.8.12"
        },
        {
            "name": "lr8_ipum13",
            "bmc_ip": "10.1.8.13",
            "gw_ip": "10.2.8.13",
            "rnic_ip": "10.5.8.13"
        },
        {
            "name": "lr8_ipum14",
            "bmc_ip": "10.1.8.14",
            "gw_ip": "10.2.8.14",
            "rnic_ip": "10.5.8.14"
        },
        {
            "name": "lr8_ipum15",
            "bmc_ip": "10.1.8.15",
            "gw_ip": "10.2.8.15",
            "rnic_ip": "10.5.8.15"
        },
        {
            "name": "lr8_ipum16",
            "bmc_ip": "10.1.8.16",
            "gw_ip": "10.2.8.16",
            "rnic_ip": "10.5.8.16"
        }
    ]
}

Search help

8. IPU‑POD128 network configuration

8.1. Overview

8.2. Useful resources

8.3. IP addressing

8.4. Merging IPU‑POD64 racks to create IPU‑POD128

8.4.1. Networking pre-requisites

8.4.2. Phase 1: Edit configuration files

Step 1: For lrack1 and lrack2

Step 2: DHCP config files for lrack1 and lrack2

Step 3: Edit DHCP config files for lrack1

Step 4: Netplan setup

Step 5: Update vlan-11.conf files

8.4.3. Phase 2: Activate new configuration

Step 6: Inform users of down time

Step 7: lrack2 DHCP server

Step 8: lrack1 DHCP server

Step 9: Restart IPU-M2000s on lrack1

Step 10: Netplan configuration on lrack1

Step 11: Update rack_config files on lrack1

Step 12: Restart IPU-M2000s on lrack2

Step 13: Verify IPU-M2000 interface access

Step 14: Create V-IPU cluster on lrack1

Step 15: Test access to V-IPU agents

Step 16: Create partitions on lrack1

Step 17: lrack1 RNIC addresses

Step 18: lrack2 RNIC addresses

Step 19: rsyslog.d

Step 20: chrony.conf

Step 21: Refresh overlay files on lrack1

Step 22: Check IPU-M2000s logging to lrack1

Step 23: Check IPU-M2000s have NTP date and time

Step 24: Run ML application

8.5. IPU-M2000 setup files

8.5.1. Syslog and chrony on the IPU-Gateway

8.5.2. Syslog on BMC

8.6. DHCP files

8.6.1. Lrack1 and lrack2: /etc/dhcp/dhcpd.conf

8.6.2. Lrack1: /etc/dhcp/dhcpd.d/ipum-dhcp.conf

8.6.3. Lrack2: /etc/dhcp/dhcpd.d/ipum-dhcp.conf

8.6.4. Lrack1 and lrack2: /etc/dhcp/dhcpd.d/ files

8.6.5. Lrack1: /etc/dhcp/dhcpd.d/lrack1 files

ipum-bmc.conf

ipum-gw.conf

ipum-rnic.conf

8.6.6. Lrack1: /etc/dhcp/dhcpd.d/lrack2 files

ipum-bmc.conf

ipum-gw.conf

8.6.7. Lrack2: /etc/dhcp/dhcpd.d/lrack2 files

ipum-rnic.conf

8.7. /etc/netplan files

8.7.1. 1GbE management interface on lrack1 server

8.7.2. RNIC interfaces on the servers

8.7.3. Lrack1: rack_config.json file

8. IPU‑POD₁₂₈ network configuration

8.4. Merging IPU‑POD₆₄ racks to create IPU‑POD₁₂₈