2. Expand an IPU‑POD₄ DA to an IPU‑POD₁₆ DA

These instructions describe how to expand an IPU‑POD₄ to an IPU‑POD₁₆.

2.1. Overview of expansion

1. Obtain the extra equipment as described in Table 2.1. All additional cables are provided when you buy the IPU-M2000 Founder’s Edition.

   • An IPU‑POD₁₆ DA system has four IPU-M2000s installed.

2. Ensure that the existing power distribution units (PDU) meet the power requirements for an IPU‑POD₁₆ system. If not, please upgrade your PDUs.

   • The installation of PDUs is not covered in this guide since that will depend upon the PDUs selected.

3. Move the server and/or the existing IPU-M2000 (Section 2.3, Uninstall existing equipment) to make space in the rack for the additional equipment.

4. Install the IPU-M2000s, the network switches and the server(s) into the correct rack slot locations (Section 2.4, Install equipment, Section 2.5, Cable the system, Section 2.6, Power cabling, Section 2.7, Complete the rack).

   • All IPU-M2000s and the server should be adjacent to each other in the rack, with the server installed above the IPU-M2000s. If you are using horizontal PDUs, then you need 7 adjacent rack slots (Fig. 2.1). If you are using vertical PDUs, then you need 5 adjacent rack slots.

   

   Fig. 2.1 Schematic of locations of components in the rack for an IPU‑POD₁₆ with horizontal PDUs.

5. Upgrade the configuration on the server and the software on the IPU-M2000s (Section 2.8, Upgrade software on host server and IPU-M2000).

2.2. Equipment needed

Table 2.1 shows the additional equipment needed for the expansion. This is based on the assumption that you already have the equipment listed in Table 2.2.

Table 2.1 Additional equipment to expand IPU‑POD₄ DA to IPU‑POD₁₆ DA

Description	Additional Quantity	Notes
Blanking panels (APC AR8136BLK)		for every unoccupied rack slot at the front of the rack
Graphcore IPU-M2000	3	You will need to buy 3 additional IPU-M2000s Founder’s Edition kits.
IPU-M2000 slider kits	3	Included with IPU-M2000 Founder’s Edition kits
0.3m blue Ethernet	3	Included with IPU-M2000 Founder’s Edition kits
0.15m red Ethernet	6	Included with IPU-M2000 Founder’s Edition kits
1.5m QSFP28	3	Included with IPU-M2000 Founder’s Edition kits
0.3m OSFP	12	Included with IPU-M2000 Founder’s Edition kits

Table 2.2 Existing equipment (from the IPU‑POD₄)

Description	Existing quantity
Rack (AR3300SP)	1
PDU	2
PDU bracket kit	2
Graphcore IPU-M2000	1
IPU-M2000 slider kits	1
Dell R6525 server	1
1.5m blue Ethernet	1
1.5m QSFP28	1

2.3. Uninstall existing equipment

Warning

Turn the power off on all systems in the rack before starting with the expansion.

Note

If the IPU‑POD₄ was installed without space for expansion, then you will have to move the IPU-M2000 and the server in order to make space for the new IPU-M2000s.

2.3.1. Remove rack doors and side panels

1. Disconnect the earth straps (Fig. 2.2).

   

   Fig. 2.2 Disconnect rack earth straps before removing doors

2. Remove the front and rear doors from the rack.

3. Remove the top and bottom side panels.

2.3.2. Remove blanking panels

Remove as many blanking panels as needed to create the space for the additional units.

2.3.3. Remove cables

Disconnect the following cables from the IPU‑POD₄:

• PDU cables

• Data and server management cables between IPU-M2000 and host server (Fig. 2.3)

  

  Fig. 2.3 Unplug both cables connecting the IPU-M2000 to the host server

2.3.4. Remove server

If you need to remove the Dell R6525 server from the rack:

1. Press the slide-release lock buttons on both rails and slide the server completely out of the rack.

2. There are lock levers on the sides of the inner rails. Rotate each lever upwards to release the server from the rails.

3. Firmly holding the sides of the server, and pull it forward until the tabs are at the front of slots. Lift the server up and remove from the rails.

4. Place the server on a level surface.

5. To remove the rails from the rack, pull the latch release button on the midpoint of the front and back of each rail. This releases the rail from the rack. Remove the rail.

For removing other servers from the rack, please refer to their manufacturer instructions.

2.3.5. Remove IPU-M2000

If you need to remove an IPU-M2000 from the rack:

1. Prepare an appropriate server lift and adjust the height such that it is suitable for the IPU-M2000 sliders. If a lift is not available, then this is a two-person operation.

2. Unscrew the captive thumb screws at the front of both the inner rack rails (Fig. 2.4).

   

   Fig. 2.4 Remove the thumb screws at the front

3. Completely slide out the IPU-M2000 on the rails.

4. Pull on the white tabs (Fig. 2.5) on both sides of the IPU-M2000 to release it. Pull the IPU-M2000 forwards until it starts sliding out of the outer rails.

   

   Fig. 2.5 Location of white release tab

5. Slide the IPU-M2000 onto the server lift.

2.3.6. Uninstall IPU-M2000 rails from the rack

If you need to move the IPU-M2000, then you need to remove the IPU-M2000 rails from the rack.

1. For the rear end of the outer rail, remove the screw with the washer from the midpoint of the vertical rack rail (Fig. 2.6).

   

   Fig. 2.6 Remove the screw from the centre of the endpiece of the rail.

2. Release the rear end of the outer rail by slightly opening the large metal latch (pull on the white tab in Fig. 2.7), then lifting the upper and lower locating pins from the square holes on the vertical rack rail. The rear end of the outer rail will now be free of the rack.

   

   Fig. 2.7 Open large metal latch to release rear end of outer rail

3. Release the front end of the outer rail by opening the latching mechanism (Fig. 2.8). Lift the rail and pull/push it towards/away from you. The rail will now be free of the rack.

   

   Fig. 2.8 Open large metal latch to release front end of outer rail

4. Repeat for the other outer rack rail.

2.4. Install equipment

2.4.1. Install the PDUs

The method for installing the PDUs will depend on the choice of PDU type and location. If the PDUs are to be installed horizontally within the rack, the recommendation is that these are positioned beneath the lowest IPU-M2000. Allowing some space between the lowest IPU-M2000 and the PDUs may make power cabling easier.

2.4.2. Install the IPU-M2000 rails

The IPU-M2000 rails should be installed in the rack so that there is enough space for all IPU-M2000s, the PDUs and the host server.

The IPU-M2000 rail kit comprises two mated inner and outer rack rails and an accessory bag containing screws. The inner rail affixes to the body of the IPU-M2000 and the outer rail affixes to the vertical rack rails in the server cabinet.

Repeat these instructions for each IPU-M2000 to be installed.

1. Separate the mated inner and outer rails.

   1. Fully extend the rails by pulling on the end which has the captive thumb screw attached (Fig. 2.9).

      

      Fig. 2.9 M2000 rack rail kit: extended rails

   2. Whilst pulling on the thumb screw end of the rails, push the white plastic release tab towards the thumb screw end (Fig. 2.10).

      

      Fig. 2.10 M2000 rack rail kit: white release tab

   3. The inner and outer rails will now separate (Fig. 2.11).

      

      Fig. 2.11 M2000 rack rail kit: inner and outer rails separated

2. To affix the inner rail to the body of the IPU-M2000, mate the inner rails (the thinner of the two separated rails which has a captive thumb screw at one end) to the body of the IPU-M2000. Note that the inner rails are identical. As such, the procedure for inner rail fixing is the same for the left and right hand inner rails.

   The inner rail should be oriented such that the captive thumb screw end is at the end of the IPU-M2000 containing the network ports.

   1. Place the inner rail to the side of the IPU-M2000 and ensure that all fixing pins are sitting within the enlarged opening of the retention channel (Fig. 2.12).

      

      Fig. 2.12 M2000 rack rail kit: fixing pins

   2. Push the inner rail towards the end of the IPU-M2000 containing the network ports, you should hear a click as the latching mechanism locks behind the head of a fixing pin (Fig. 2.13).

      

      Fig. 2.13 M2000 rack rail kit: fixing pins locked

   3. Ensure all fixing pins are correctly engaged with their respective retention channels.

   4. Locate the four flat head fixing screws from the rack rail accessory bag (Fig. 2.14).

      

      Fig. 2.14 M2000 rack rail kit: flat head fixing screws

   5. Using these four screws, affix the inner rail to the body of the M2000 (Fig. 2.15). There will be two screws per side.

      

      Fig. 2.15 M2000 rack rail kit: attaching inner rail to M2000

      The inner rails are now securely affixed to the IPU-M2000 body.

3. Install the outer rack rails in the rack.

   1. Locate the front and rear of the outer rail. The front of the outer rail is embossed with the word “FRONT” and the rear of the outer rail has a large metal latch (as shown in Fig. 2.16).

      

      Fig. 2.16 Attachment point for the rear of the IPU-M2000 outer rail

   2. Pull on each end of the outer rail to adjust the rail length to suit your rack

   3. Hold the front end of the outer rail (embossed with the word “FRONT”) behind the square holes in the vertical rack rail. Pull the outer rail towards the vertical rack rail and the latching mechanism will click and hold the outer rail in place (Fig. 2.17).

      

      Fig. 2.17 IPU-M2000 outer rail front latching mechanism

   4. Hold the rear end of the outer rail and slightly open the large metal latch, then press the upper and lower locating pins into the square holes in the vertical rack rail. Release the large metal latch and the outer rail will now be secured to the vertical rack rail (Fig. 2.18).

      

      Fig. 2.18 Rear of IPU-M2000 outer rail secured to vertical rack rail

   5. Find the screws and washers (Fig. 2.19) in the rack rail accessory bag. One screw and one washer should be screwed through the vertical rack rail and into the outer rack rail threaded hole. The washer should be used in such a way that the washer sits flush with the head of the screw –like a cup (Fig. 2.20).

      

      Fig. 2.19 IPU-M2000 rack rail kit: outer rail screws and washers

      

      Fig. 2.20 IPU-M2000 rack rail kit: outer rail screws and washers attached.

   6. This should be repeated for both outer rack rails.

2.4.3. Install the IPU-M2000s

1. Pull the sliding rail located within the outer rack rail completely forward such that it locks into the fully extended position (Fig. 2.21).

   

   Fig. 2.21 IPU-M2000 rack rail kit: sliders fully extended

2. Place the IPU-M2000 onto an appropriate server lift and adjust the height such that it is suitable for the sliders (Fig. 2.22). If a lift is not available, this is a two person operation.

   

   Fig. 2.22 Server lift for IPU-M2000

3. Slide the protruding inner rails (on the IPU-M2000) into the receiving channel of the extended outer rails (Fig. 2.23).

   

   Fig. 2.23 Slide IPU-M2000 inner rails into outer rails

4. Whilst the server lift is supporting the full weight of the IPU-M2000, slide the IPU-M2000 into the extended outer rails until you feel both sides engage a stopping mechanism (Fig. 2.25).

5. Then, simultaneously pull on the blue tabs for the release mechanism at each side of the IPU-M2000 and then push the IPU-M2000 unit fully into the rack (Fig. 2.24 and circled in Fig. 2.25).

   

   Fig. 2.24 Blue tab release mechanism

   

   Fig. 2.25 Location of blue tab release mechanism

6. Finally, screw the captive thumb screw into the inner rack rail. (Fig. 2.26).

   

   Fig. 2.26 Screwing captive thumb screw into inner rack rail

7. The IPU-M2000 is now installed.

2.4.4. Install the host server

The server should be installed above the four IPU-M2000s.

This section describes the installation of a Dell R6525 server. A list of approved and qualified servers is available. To find out more, speak to Graphcore sales or your Graphcore channel partner.

Note

The R6525 is installed such that the rear of the server (containing the management and data ports) is on the opposite side of the rack to the ports on the front of the IPU-Machine(s). This is due to the airflow direction.

1. Install the tool-less sliding rail kit(s).

   Pull out the rail and fit the server to the rail ensuring the T pins on the side of the server locate in the slots on the rail. Ensure that the power supplies on the server face the rear of the rack (Fig. 2.27).

   

   Fig. 2.27 Sliding rail kit for server installation

   Note

   Use an appropriate server lift or have two people installing the server to ensure correct fitting

2. Push the server gently from the front to lock it into the slides then press the tab on the side of the slides and push the server fully home in the rack. Repeat the above process for each server if installing multiple servers.

2.5. Cable the system

The cabling is very straightforward, as indicated by diagram of the completed system Fig. 2.28. The following sections take you through wiring up each group of cables in turn.

Fig. 2.28 Completed cabling for an IPU‑POD₁₆ DA system

2.5.1. IPU-M2000 to IPU-M2000 IPU-Link connectivity (OSFP)

There are 8 OSFP IPU-Link ports on the right side of each IPU-M2000.

Fig. 2.29 IPU-M2000 IPU-Link ports

Using 0.3M OSFP cables, link the top row of four IPU-link ports (5-8) to the bottom row of four IPU-link ports (1-4) in the IPU-M2000 that is installed directly above (see Table 2.3 and Fig. 2.30). The top row (5-8) of the top-most IPU-M2000 (#4) and the bottom row (1-4) of the bottom-most IPU-M2000 (#1), are left unconnected.

Fig. 2.30 IPU-M2000 to IPU-M2000 IPU-Link cabling for an IPU‑POD₁₆ DA system

1. Before attempting to install the OSFP cables, it is beneficial to manipulate the cable to form a tight loop such that the white side of the connector tabs face away from each other. During manufacture and shipping, the cables can form quite a stiff shape, so manipulating the cables (Fig. 2.31) before installing them reduces stresses on the socket during install.

   

   

   Fig. 2.31 Manipulate the OSFP cables before installing

2. After installing a cable, pull gently on the black cable to ensure the plugs are firm in the sockets on the IPU-M2000.

   

   Fig. 2.32 Pull gently on the black cable to ensure that they are firmly connected

   Note

   The white tab is on the top of the cable when inserted into the IPU-M2000.

Table 2.3 IPU-M2000 IPU-Link port mapping

From IPU-M2000 OSFP port	To IPU-M2000 OSFP port	Cables
IPU-M2000 # 3 IPU-Link ports 5,6,7,8	IPU-M2000 # 4 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 # 2 IPU-Link ports 5,6,7,8	IPU-M2000 # 3 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 # 1 IPU-Link ports 5,6,7,8	IPU-M2000 # 2 IPU-Link ports 1,2,3,4	OSFP 0.3m

2.5.2. IPU-M2000 to IPU-M2000 Sync-Link cabling (Cat5e)

There are 8 Cat5e Sync-Link ports in the middle of each IPU-M2000.

Fig. 2.33 IPU-M2000 Sync-Link ports

Using 0.15m red Ethernet Cat5e cables, link the top two central Sync-Link ports (6-7) of each IPU-M2000 to the bottom two central Sync-Link ports (2-3) of the IPU-M2000 that is installed directly above (as described in Table 2.4 and shown in Fig. 2.34). The top row (5-8) of the top-most IPU-M2000 (#4) and the bottom row (1-4) of the bottom-most IPU-M2000 (#1), are left unconnected.

Fig. 2.34 IPU-M2000 Sync-Link cabling for an IPU‑POD₁₆ DA system

Table 2.4 IPU-M2000 Sync-Link port mapping

From IPU-M2000 Sync-Link connections	To IPU-M2000 Sync-Link connections	Cables
IPU-M2000 # 3 Sync-Link ports 6-7	IPU-M2000 # 4 Sync-Link ports 2-3	2x Cat5e 0.15m red
IPU-M2000 # 2 Sync-Link ports 6-7	IPU-M2000 # 3 Sync-Link ports 2-3	2x Cat5e 0.15m red
IPU-M2000 # 1 Sync-Link ports 6-7	IPU-M2000 # 2 Sync-Link ports 2-3	2x Cat5e 0.15m red

2.5.3. IPU-M2000 to IPU-M2000 management cabling (Cat5e)

There are 2 Cat5e management ports in the middle of each IPU-M2000.

Fig. 2.35 IPU-M2000 Cat5e management ports

Using 0.3m blue Ethernet Cat5e cables, link the top management port (2) of each IPU-M2000 to the bottom management port (1) of the IPU-M2000 that is installed directly above (as described in Table 2.5 and shown in Fig. 2.36). The top management port (2) of the top-most IPU-M2000 (#4) and the bottom management port (1) of the bottom-most IPU-M2000 (#1), are left unconnected.

Fig. 2.36 IPU-M2000 management cabling for an IPU‑POD₁₆ DA system

Table 2.5 IPU-M2000 management port mapping

From IPU-M2000 management connections	To IPU-M2000 management connections	Cables
IPU-M2000 # 3 management ports 2	IPU-M2000 # 4 management ports 1	Cat5e 0.3m blue
IPU-M2000 # 2 management ports 2	IPU-M2000 # 3 management ports 1	Cat5e 0.3m blue
IPU-M2000 # 1 management ports 2	IPU-M2000 # 2 management ports 1	Cat5e 0.3m blue

2.5.4. IPU-M2000 to server management cabling (Cat5e)

There are 2 Cat5e management ports in the middle of each IPU-M2000.

Fig. 2.37 IPU-M2000 Cat5e management ports

Using a 1.5m blue Ethernet Cat5e cable, link the top management port (port #2) of the top IPU-M2000 to management port #1 of the R6525 server, as described in Table 2.6 and shown in Fig. 2.38.

Fig. 2.38 IPU‑POD₁₆ DA system (IPU-M2000 to server management cabling)

Table 2.6 IPU-M2000 management port mapping

From IPU-M2000 management connection	To server management connection	Cables
Top IPU-M2000 management port 2	R6525 server management port 1	Cat5e 1.5m blue

2.5.5. IPU-M2000 to server data cabling (QSFP)

There are two QSFP RNIC data ports on the left of each IPU-M2000.

Fig. 2.39 IPU-M2000 RNIC ports

Only one of these (port #2) should be connected from each IPU-M2000 to the server with 1.5m QSFP cables, as described in Table 2.7 and shown in Fig. 2.40.

Fig. 2.40 IPU‑POD₁₆ DA system (IPU-M2000 data cabling)

Note

IPU-M2000 #1 (the lowest in the stack) is always connected to Port 1 in server slot 3.

Table 2.7 IPU-M2000 RNIC port mapping

IPU-M2000 RNIC port	Server port	Cables
IPU-M2000 # 4 port 2	Server NIC slot 2, port 2	QSFP 1.5m
IPU-M2000 # 3 port 2	Server NIC slot 2, port 1	QSFP 1.5m
IPU-M2000 # 2 port 2	Server NIC slot 3, port 2	QSFP 1.5m
IPU-M2000 # 1 port 2	Server NIC slot 3, port 1	QSFP 1.5m

2.6. Power cabling

The method for cabling the PDUs will depend on the choice of PDU and location. Each IPU-M2000 has two power inputs and these should be supplied from separate power distribution units for redundancy.

2.7. Complete the rack

The following steps describe completing the rack: fitting blanking panels and re-installing the doors and side panels, if required.

2.7.1. Blanking panels

For proper airflow, we recommend installing 1U blanking panels in every unoccupied rack slot at the front of the rack.

Fig. 2.41 1U blanking panels

2.7.2. Rack completion

Re-install the top and bottom side panels on each side of the rack. Ensure the earth cables are reconnected to the cable on the rack. Re-install the front and rear doors.

2.8. Upgrade software on host server and IPU-M2000

To complete the expansion from an IPU‑POD₄ to an IPU‑POD₁₆, you need to upgrade the software on the host server and on the IPU-M2000s. This process is described in the Server platform system configuration chapter of the IPU-POD DA build and test guide.

Note

Because you have a configured system, when you run the IPU-POD DA install script, it will start by asking you to accept the current configuration. You must answer “n” so that you can increase the number of IPU-M2000s in your system from 1 to 4.