2. Expand a single Bow-2000 system to a Bow Pod₁₆ DA

These instructions describe how to expand a single Bow-2000 system to a Bow Pod₁₆.

2.1. Overview of expansion

1. Obtain the extra equipment as described in Table 2.1. All additional cables are provided when you buy the Bow-2000 Founder’s Edition.

   • A Bow Pod₁₆ DA system has four Bow-2000s installed.

2. Ensure that the existing power distribution units (PDU) meet the power requirements for a Bow Pod₁₆ system. If not, please upgrade your PDUs.

   • The installation of PDUs is not covered in this guide since that will depend upon the PDUs selected.

3. Move the server and/or the existing Bow-2000 (Section 2.3, Uninstall existing equipment) to make space in the rack for the additional equipment.

4. Install the Bow-2000s, the network switches and the server(s) into the correct rack slot locations (Section 2.4, Install equipment, Section 2.5, Cable the system, Section 2.6, Power cabling, Section 2.7, Complete the rack).

   • All Bow-2000s and the server should be adjacent to each other in the rack, with the server installed above the Bow-2000s. If you are using horizontal PDUs, then you need 7 adjacent rack slots (Fig. 2.1). If you are using vertical PDUs, then you need 5 adjacent rack slots.

   

   Fig. 2.1 Schematic of locations of components in the rack for a Bow Pod₁₆ with horizontal PDUs.

5. Upgrade the configuration on the server and the software on the Bow-2000s (Section 2.8, Upgrade software on host server and Bow-2000).

2.2. Equipment needed

Table 2.1 shows the additional equipment needed for the expansion. This is based on the assumption that you already have the equipment listed in Table 2.2.

Table 2.1 Additional equipment to expand a single Bow-2000 system to Bow Pod₁₆ DA

Description	Additional Quantity	Notes
Blanking panels (APC AR8136BLK)		for every unoccupied rack slot at the front of the rack
Graphcore Bow-2000	3	You will need to buy 3 additional Bow-2000s Founder’s Edition kits.
Bow-2000 slider kits	3	Included with Bow-2000 Founder’s Edition kits
0.3m blue Ethernet	3	Included with Bow-2000 Founder’s Edition kits
0.15m red Ethernet	6	Included with Bow-2000 Founder’s Edition kits
1.5m QSFP28	3	Included with Bow-2000 Founder’s Edition kits
0.3m OSFP	12	Included with Bow-2000 Founder’s Edition kits

Table 2.2 Existing equipment (from the single Bow-2000 system)

Description	Existing quantity
Rack (AR3300SP)	1
PDU	2
PDU bracket kit	2
Graphcore Bow-2000	1
Bow-2000 slider kits	1
Dell R6525 server	1
1.5m blue Ethernet	1
1.5m QSFP28	1

2.3. Uninstall existing equipment

Warning

Turn the power off on all systems in the rack before starting with the expansion.

Note

If the single Bow-2000 system was installed without space for expansion, then you will have to move the Bow-2000 and the server in order to make space for the new Bow-2000s.

2.3.1. Remove rack doors and side panels

1. Disconnect the earth straps (Fig. 2.2).

   

   Fig. 2.2 Disconnect rack earth straps before removing doors

2. Remove the front and rear doors from the rack.

3. Remove the top and bottom side panels.

2.3.2. Remove blanking panels

Remove as many blanking panels as needed to create the space for the additional units.

2.3.3. Remove cables

Disconnect the following cables from the single Bow-2000 system:

• PDU cables

• Data and server management cables between Bow-2000 and host server (Fig. 2.3)

  

  Fig. 2.3 Unplug both cables connecting the Bow-2000 to the host server

2.3.4. Remove server

If you need to remove the Dell R6525 server from the rack:

1. Press the slide-release lock buttons on both rails and slide the server completely out of the rack.

2. There are lock levers on the sides of the inner rails. Rotate each lever upwards to release the server from the rails.

3. Firmly holding the sides of the server, and pull it forward until the tabs are at the front of slots. Lift the server up and remove from the rails.

4. Place the server on a level surface.

5. To remove the rails from the rack, pull the latch release button on the midpoint of the front and back of each rail. This releases the rail from the rack. Remove the rail.

For removing other servers from the rack, please refer to their manufacturer instructions.

2.3.5. Remove Bow-2000

To remove an IPU-Machine from the rack:

1. Prepare an appropriate server lift and adjust the height such that it is suitable for the IPU-Machine sliders. If a lift is not available, then this is a two-person operation.

2. Unscrew the captive thumb screws at the front of both the inner rack rails (Fig. 2.4).

   

   Fig. 2.4 Remove the thumb screws at the front

3. Completely slide out the IPU-Machine on the rails.

4. Pull on the white tabs (Fig. 2.5) on both sides of the IPU-Machine to release it. Pull the IPU-Machine forwards until it starts sliding out of the outer rails.

   

   Fig. 2.5 Location of white release tab

5. Slide the IPU-Machine onto the server lift, if available, or two people should carry the IPU-Machine.

2.3.6. Uninstall Bow-2000 rails from the rack

If you need to move the IPU-Machine, then you need to remove the IPU-Machine rails from the rack.

1. For the rear end of the outer rail, remove the screw with the washer from the midpoint of the vertical rack rail (Fig. 2.6).

   

   Fig. 2.6 Remove the screw from the centre of the endpiece of the rail.

2. Release the rear end of the outer rail by slightly opening the large metal latch (pull on the white tab in Fig. 2.7), then lifting the upper and lower locating pins from the square holes on the vertical rack rail. The rear end of the outer rail will now be free of the rack.

   

   Fig. 2.7 Open large metal latch to release rear end of outer rail

3. Release the front end of the outer rail by opening the latching mechanism (Fig. 2.8). Lift the rail and pull/push it towards/away from you. The rail will now be free of the rack.

   

   Fig. 2.8 Open large metal latch to release front end of outer rail

4. Repeat for the other outer rack rail.

2.4. Install equipment

2.4.1. Install the PDUs

The method for installing the PDUs will depend on the choice of PDU type and location. If the PDUs are to be installed horizontally within the rack, the recommendation is that these are positioned beneath the lowest Bow-2000. Allowing some space between the lowest Bow-2000 and the PDUs may make power cabling easier.

2.4.2. Install the Bow-2000 rails

The Bow-2000 rails should be installed in the rack so that there is enough space for all Bow-2000s, the PDUs and the host server.

The IPU-Machine rail kit comprises two mated inner and outer rack rails and an accessory bag containing screws. The inner rail affixes to the body of the IPU-Machine and the outer rail affixes to the vertical rack rails in the server cabinet.

Repeat these instructions for each IPU-Machine to be installed.

1. Separate the mated inner and outer rails.

   1. Fully extend the rails by pulling on the end which has the captive thumb screw attached (Fig. 2.9).

      

      Fig. 2.9 IPU-Machine rack rail kit: extended rails

   2. Whilst pulling on the thumb screw end of the rails, push the white plastic release tab towards the thumb screw end (Fig. 2.10).

      

      Fig. 2.10 IPU-Machine rack rail kit: white release tab

   3. The inner and outer rails will now separate (Fig. 2.11).

      

      Fig. 2.11 IPU-Machine rack rail kit: inner and outer rails separated

2. To affix the inner rail to the body of the IPU-Machine, mate the inner rails (the thinner of the two separated rails which has a captive thumb screw at one end) to the body of the IPU-Machine. Note that the inner rails are identical. As such, the procedure for inner rail fixing is the same for the left and right hand inner rails.

   The inner rail should be oriented such that the captive thumb screw end is at the end of the IPU-Machine containing the network ports.

   1. Place the inner rail to the side of the IPU-Machine and ensure that all fixing pins are sitting within the enlarged opening of the retention channel (Fig. 2.12).

      

      Fig. 2.12 IPU-Machine rack rail kit: fixing pins

   2. Push the inner rail towards the end of the IPU-Machine containing the network ports, you should hear a click as the latching mechanism locks behind the head of a fixing pin (Fig. 2.13).

      

      Fig. 2.13 IPU-Machine rack rail kit: fixing pins locked

   3. Ensure all fixing pins are correctly engaged with their respective retention channels.

   4. Locate the four flat head fixing screws from the rack rail accessory bag (Fig. 2.14).

      

      Fig. 2.14 IPU-Machine rack rail kit: flat head fixing screws

   5. Using these four screws, affix the inner rail to the body of the IPU-Machine (Fig. 2.15). There will be two screws per side.

      

      Fig. 2.15 IPU-Machine rack rail kit: attaching inner rail to IPU-Machine

      The inner rails are now securely affixed to the IPU-Machine body.

3. Install the outer rack rails in the rack.

   1. Locate the front and rear of the outer rail. The front of the outer rail is embossed with the word “FRONT” and the rear of the outer rail has a large metal latch (as shown in Fig. 2.16).

      

      Fig. 2.16 Attachment point for the rear of the IPU-Machine outer rail

   2. Pull on each end of the outer rail to adjust the rail length to suit your rack

   3. Hold the front end of the outer rail (embossed with the word “FRONT”) behind the square holes in the vertical rack rail. Pull the outer rail towards the vertical rack rail and the latching mechanism will click and hold the outer rail in place (Fig. 2.17).

      

      Fig. 2.17 IPU-Machine outer rail front latching mechanism

   4. Hold the rear end of the outer rail and slightly open the large metal latch, then press the upper and lower locating pins into the square holes in the vertical rack rail. Release the large metal latch and the outer rail will now be secured to the vertical rack rail (Fig. 2.18).

      

      Fig. 2.18 Rear of IPU-Machine outer rail secured to vertical rack rail

   5. Find the screws and washers (Fig. 2.19) in the rack rail accessory bag. One screw and one washer should be screwed through the vertical rack rail and into the outer rack rail threaded hole. The washer should be used in such a way that the washer sits flush with the head of the screw –like a cup (Fig. 2.20).

      

      Fig. 2.19 IPU-Machine rack rail kit: outer rail screws and washers

      

      Fig. 2.20 IPU-Machine rack rail kit: outer rail screws and washers attached.

   6. This should be repeated for both outer rack rails.

2.4.3. Install the Bow-2000s

1. Pull the sliding rail located within the outer rack rail completely forward such that it locks into the fully extended position (Fig. 2.21).

   

   Fig. 2.21 IPU-Machine rack rail kit: sliders fully extended

2. Place the IPU-Machine onto an appropriate server lift and adjust the height such that it is suitable for the sliders (Fig. 2.22). If a lift is not available, this is a two person operation.

   

   Fig. 2.22 Server lift for IPU-Machine

3. Slide the protruding inner rails (on the IPU-Machine) into the receiving channel of the extended outer rails (Fig. 2.23).

   

   Fig. 2.23 Slide IPU-Machine inner rails into outer rails

4. Whilst the server lift is supporting the full weight of the IPU-Machine (or with two people carrying the IPU-Machine if not using a server lift), slide the IPU-Machine into the extended outer rails until you feel both sides engage a stopping mechanism (Fig. 2.25).

5. Then, simultaneously pull on the blue tabs for the release mechanism at each side of the IPU-Machine and then push the IPU-Machine unit fully into the rack (Fig. 2.24 and circled in Fig. 2.25).

   

   Fig. 2.24 Blue tab release mechanism

   

   Fig. 2.25 Location of blue tab release mechanism

6. Finally, screw the captive thumb screw into the inner rack rail. (Fig. 2.26).

   

   Fig. 2.26 Re-attach the IPU-Machine to the inner rack rail by tightening the captive thumb screws”

7. The IPU-Machine is now installed.

2.4.4. Install the host server

The server should be installed above the four Bow-2000s.

This section describes the installation of a Dell R6525 server. A list of approved and qualified servers is available. To find out more, speak to Graphcore sales or your Graphcore channel partner.

Note

The R6525 is installed such that the rear of the server (containing the management and data ports) is on the opposite side of the rack to the ports on the front of the IPU-Machine(s). This is due to the airflow direction.

1. Install the tool-less sliding rail kit(s).

   Pull out the rail and fit the server to the rail ensuring the T pins on the side of the server locate in the slots on the rail. Ensure that the power supplies on the server face the rear of the rack (Fig. 2.27).

   

   Fig. 2.27 Sliding rail kit for server installation

   Note

   Use an appropriate server lift or have two people installing the server to ensure correct fitting

2. Push the server gently from the front to lock it into the slides then press the tab on the side of the slides and push the server fully home in the rack. Repeat the above process for each server if installing multiple servers.

2.5. Cable the system

The cabling is very straightforward, as indicated by diagram of the completed system Fig. 2.28. The following sections take you through wiring up each group of cables in turn.

Fig. 2.28 Completed cabling for a Bow Pod₁₆ DA system

2.5.1. Bow-2000 to Bow-2000 IPU-Link connectivity (OSFP)

There are 8 OSFP IPU-Link ports on the right side of each Bow-2000.

Fig. 2.29 Bow-2000 IPU-Link ports

Using 0.3M OSFP cables, link the top row of four IPU-link ports (5-8) to the bottom row of four IPU-link ports (1-4) in the Bow-2000 that is installed directly above (see Table 2.3 and Fig. 2.30). The top row (5-8) of the top-most Bow-2000 (#4) and the bottom row (1-4) of the bottom-most Bow-2000 (#1), are left unconnected.

Fig. 2.30 Bow-2000 to Bow-2000 IPU-Link cabling for a Bow Pod₁₆ DA system

1. Before attempting to install the OSFP cables, it is beneficial to manipulate the cable to form a tight loop such that the white side of the connector tabs face away from each other. During manufacture and shipping, the cables can form quite a stiff shape, so manipulating the cables (Fig. 2.31) before installing them reduces stresses on the socket during install.

   

   

   Fig. 2.31 Manipulate the OSFP cables before installing

2. After installing a cable, pull gently on the black cable to ensure the plugs are firm in the sockets on the Bow-2000.

   

   Fig. 2.32 Pull gently on the black cable to ensure that they are firmly connected

   Note

   The white tab is on the top of the cable when inserted into the Bow-2000.

Table 2.3 Bow-2000 IPU-Link port mapping

From Bow-2000 OSFP port | To Bow-2000 OSFP port		Cables
Bow-2000 # 3 IPU-Link ports 5,6,7,8 | Bow-2000 # 4 IPU-Link ports 1,2,3,4		OSFP 0.3m
Bow-2000 # 2 IPU-Link ports 5,6,7,8 | Bow-2000 # 3 IPU-Link ports 1,2,3,4		OSFP 0.3m
Bow-2000 # 1 IPU-Link ports 5,6,7,8 | Bow-2000 # 2 IPU-Link ports 1,2,3,4		OSFP 0.3m

2.5.2. Bow-2000 to Bow-2000 Sync-Link cabling (Cat5e)

There are 8 Cat5e Sync-Link ports in the middle of each Bow-2000.

Fig. 2.33 Bow-2000 Sync-Link ports

Using 0.15m red Ethernet Cat5e cables, link the top two central Sync-Link ports (6-7) of each Bow-2000 to the bottom two central Sync-Link ports (2-3) of the Bow-2000 that is installed directly above (as described in Table 2.4 and shown in Fig. 2.34). The top row (5-8) of the top-most Bow-2000 (#4) and the bottom row (1-4) of the bottom-most Bow-2000 (#1), are left unconnected.

Fig. 2.34 Bow-2000 Sync-Link cabling for a Bow Pod₁₆ DA system

Table 2.4 Bow-2000 Sync-Link port mapping

From Bow-2000 Sync-Link connections | To Bow-2000 Sync-Link connections		Cables
Bow-2000 # 3 Sync-Link ports 6-7 | Bow-2000 # 4 Sync-Link ports 2-3		2x Cat5e 0.15m red
Bow-2000 # 2 Sync-Link ports 6-7 | Bow-2000 # 3 Sync-Link ports 2-3		2x Cat5e 0.15m red
Bow-2000 # 1 Sync-Link ports 6-7 | Bow-2000 # 2 Sync-Link ports 2-3		2x Cat5e 0.15m red

2.5.3. Bow-2000 to Bow-2000 management cabling (Cat5e)

There are 2 Cat5e management ports in the middle of each Bow-2000.

Fig. 2.35 Bow-2000 Cat5e management ports

Using 0.3m blue Ethernet Cat5e cables, link the top management port (2) of each Bow-2000 to the bottom management port (1) of the Bow-2000 that is installed directly above (as described in Table 2.5 and shown in Fig. 2.36). The top management port (2) of the top-most Bow-2000 (#4) and the bottom management port (1) of the bottom-most Bow-2000 (#1), are left unconnected.

Fig. 2.36 Bow-2000 management cabling for a Bow Pod₁₆ DA system

Table 2.5 Bow-2000 management port mapping

From Bow-2000 management connections	To Bow-2000 management connections	Cables
Bow-2000 # 3 management ports 2	Bow-2000 # 4 management ports 1	Cat5e 0.3m blue
Bow-2000 # 2 management ports 2	Bow-2000 # 3 management ports 1	Cat5e 0.3m blue
Bow-2000 # 1 management ports 2	Bow-2000 # 2 management ports 1	Cat5e 0.3m blue

2.5.4. Bow-2000 to server management cabling (Cat5e)

There are 2 Cat5e management ports in the middle of each Bow-2000.

Fig. 2.37 Bow-2000 Cat5e management ports

Using a 1.5m blue Ethernet Cat5e cable, link the top management port (port #2) of the top Bow-2000 to management port #1 of the R6525 server, as described in Table 2.6 and shown in Fig. 2.38.

Fig. 2.38 Bow Pod₁₆ DA system (Bow-2000 to server management cabling)

Table 2.6 Bow-2000 management port mapping

From Bow-2000 management connection	To server management connection	Cables
Top Bow-2000 management port 2	R6525 server management port 1	Cat5e 1.5m blue

2.5.5. Bow-2000 to server data cabling (QSFP)

There are two QSFP RNIC data ports on the left of each Bow-2000.

Fig. 2.39 Bow-2000 RNIC ports

Only one of these (port #2) should be connected from each Bow-2000 to the server with 1.5m QSFP cables, as described in Table 2.7 and shown in Fig. 2.40.

Fig. 2.40 Bow Pod₁₆ DA system (Bow-2000 data cabling)

Note

Bow-2000 #1 (the lowest in the stack) is always connected to Port 1 in server slot 3.

Table 2.7 Bow-2000 RNIC port mapping

Bow-2000 RNIC port	Server port	Cables
Bow-2000 # 4 port 2	Server NIC slot 2, port 2	QSFP 1.5m
Bow-2000 # 3 port 2	Server NIC slot 2, port 1	QSFP 1.5m
Bow-2000 # 2 port 2	Server NIC slot 3, port 2	QSFP 1.5m
Bow-2000 # 1 port 2	Server NIC slot 3, port 1	QSFP 1.5m

2.6. Power cabling

The method for cabling the PDUs will depend on the choice of PDU and location. Each Bow-2000 has two power inputs and these should be supplied from separate power distribution units for redundancy.

2.7. Complete the rack

The following steps describe completing the rack: fitting blanking panels and re-installing the doors and side panels, if required.

2.7.1. Blanking panels

For proper airflow it is recommended to install 1U blanking panels in every unoccupied rack slot at the front of the rack.

Fig. 2.41 1U blanking panels

2.7.2. Rack completion

Re-install the top and bottom side panels on each side of the rack. Ensure the earth cables are reconnected to the cable on the rack. Re-install the front and rear doors.

2.8. Upgrade software on host server and Bow-2000

To complete the expansion from a single Bow-2000 system to a Bow Pod₁₆, you need to upgrade the software on the host server and on the Bow-2000s. This process is described in the System platform software configuration section of the Bow Pod DA Build and Test Guide.

Note

Because you have a configured system, when you run the Bow Pod DA install script, it will start by asking you to accept the current configuration. You must answer “n” so that you can increase the number of Bow-2000s in your system from 1 to 4.