3. Expand an IPU‑POD16 DA to an IPU‑POD64

These instructions describe how to expand an IPU‑POD16 DA to an IPU‑POD64.

3.1. Overview of expansion

  1. Obtain the extra equipment as described in Table 3.1. Some cables are provided when you buy the IPU-M2000 Founder’s Edition, but you will have to supply the rest.

    • An IPU‑POD64 system has 16 IPU-M2000s installed.

  2. Move the server and/or the existing IPU-M2000s (Section 3.3, Uninstall existing equipment) to make space in the rack for the additional IPU-M2000s and the network switches to be installed into the correct rack slot locations.

  3. Install the IPU-M2000s, the network switches and the server(s) into the correct rack slot locations (Section 3.4, Install the equipment, Section 3.5, Cabling the rack, Section 3.6, Power cabling, Section 3.7, Completing the rack)

    • All IPU-M2000s, the network switches and the server(s) should be adjacent to each other in the rack, with the server installed above the IPU-M2000s You need 22 adjacent rack slots (Fig. 3.1).

    _images/pod64-rack-layout.png

    Fig. 3.1 Schematic of locations of components in the rack for an IPU‑POD64

  4. Upgrade the configuration on the server and the software on the IPU-M2000s (Section 3.8, Upgrade software on management server).

3.2. Equipment needed

Table 3.1 shows the additional equipment needed for the expansion. This is based on the assumption that you already have the equipment list in Table 3.2.

Table 3.1 Additional equipment required to expand the IPU‑POD4 DA to the IPU‑POD16 DA

Description

Quantity (1 server)

Quantity (4 server)

Notes

AP8886 PDU

2

2

PDU bracket kit APC AR7711

2

2

Graphcore IPU-M2000

12

12

IPU-M2000 slider kits

12

12

All included with IPU-M2000 Founder’s Edition kits

Dell R6525 server

0

3

Arista DCS-7010T-48-F switch

1

1

Arista DCS-7060CX-32S-F switch

1

1

2m purple Ethernet

2

2

1.5m blue Ethernet

11

17

16 provided with IPU-M2000 Founder’s Edition kits, 1 more needed for 4-server version

1m blue Ethernet

9

9

1m yellow Ethernet

12

12

1.5m yellow Ethernet

4

4

1m red Ethernet

2

2

0.15m red Ethernet

24

24

All included with IPU-M2000 Founder’s Edition kits

1m QSFP28

8

8

1.5m QSFP28

6

12

All included with IPU-M2000 Founder’s Edition kits

0.3m OSFP

48

48

All included with IPU-M2000 Founder’s Edition kits

1m OSFP

4

4

0.5m red 10A C14 to C15

12

12

1m red 10A C14 to C15

4

4

0.5m blue 10A C14 to C15

12

12

1m blue 10A C14 to C15

4

4

1m red C13 to C14

2

5

1.5m red C13 to C14

1

1

1m blue C13 to C14

2

5

1.5m blue C13 to C14

1

1

Velcro

1

1
Table 3.2 Existing equipment (from IPU‑POD16 DA)

Description

Existing quantity

Rack (AR3300SP)

1

Blanking panels (APC AR8136BLK)

Graphcore IPU-M2000

4

IPU-M2000 slider kits

4

Dell R6525 server

1

1.5m blue Ethernet

1

0.3m blue Ethernet

3

0.15m red Ethernet

6

1.5m QSFP28

4

0.3m OSFP

12

3.3. Uninstall existing equipment

Warning

Turn the power off on all systems in the rack before starting with the expansion.

3.3.1. Remove rack doors and side panels

  1. Disconnect the earth straps (Fig. 2.2).

    _images/earth_straps.jpg

    Fig. 3.2 Disconnect rack earth straps before removing doors

  2. Remove the front and rear doors from the rack.

  3. Remove the top and bottom side panels.

3.3.2. Remove blanking panels

Remove as many blanking panels as needed to create the space for the additional units.

3.3.3. Remove cables

Disconnect the following cables from the IPU‑POD16 (Fig. 3.3):

  • PDU cables

  • Cables connecting an IPU-M2000 to its neighbour(s)

  • Data and server management cables between IPU-M2000 and host server

    _images/m2000_da_x4_cabling.png

    Fig. 3.3 Unplug all cables connecting the IPU-M2000 to the host server and to other IPU-M2000s.

3.3.4. Remove server

If you need to remove the Dell R6525 server from the rack:

  1. Press the slide-release lock buttons on both rails and slide the server completely out of the rack.

  2. There are lock levers on the sides of the inner rails. Rotate each lever upwards to release the server from the rails.

  3. Firmly holding the sides of the server, and pull it forward until the tabs are at the front of slots. Lift the server up and remove from the rails.

  4. Place the server on a level surface.

  5. To remove the rails from the rack, pull the latch release button on the midpoint of the front and back of each rail. This releases the rail from the rack. Remove the rail.

For removing other servers from the rack, please refer to their manufacturer instructions.

3.3.5. Remove IPU-M2000

If you need to remove an IPU-M2000 from the rack:

  1. Prepare an appropriate server lift and adjust the height such that it is suitable for the IPU-M2000 sliders. If a lift is not available, then this is a two-person operation.

  2. Unscrew the captive thumb screws at the front of both the inner rack rails (Fig. 2.4).

    _images/m2000_rail_11.png

    Fig. 3.4 Remove the thumb screws at the front

  3. Completely slide out the IPU-M2000 on the rails.

  4. Pull on the white tabs (Fig. 2.5) on both sides of the IPU-M2000 to release it. Pull the IPU-M2000 forwards until it starts sliding out of the outer rails.

    _images/m2000_removal_white_release_tab.jpg

    Fig. 3.5 Location of white release tab

  5. Slide the IPU-M2000 onto the server lift.

3.4. Install the equipment

Note the correct orientation of the IPU-M2000, server and switch units in the rack to ensure correct airflow.

The front interface of the IPU-M2000 units (connectivity ports) should be matched with the front door of the rack (cold aisle). The rear interface of the server and switches (power and fans) should be matched with the rear door of the rack (hot aisle).

Note

You can click on any of the images below to view them full size

_images/completed_rack_cold_four_server.png

Fig. 3.6 Completed rack: cold aisle (four-server version)

Note that Fig. 3.6 shows a four-server version of the IPU‑POD64. The default reference design has one server which would be the server in the lowest position, closest to the switches.

_images/completed_rack_hot_four_server.png

Fig. 3.7 Completed rack: hot aisle (four-server version)

Note that Fig. 3.7 shows three blue RJ45 cables in each R6525 server. In the default build, servers 2 to 4 only have two blue RJ45 cables. See Section 3.5.7, Dell R6525 server(s) cabling for more information about server cabling.

Note also that Fig. 3.7 shows a four-server version of the IPU‑POD64. The default reference design has one server which would be the server in the lowest position, closest to the switches.

3.4.1. Document reproduction

Ideally you should reference this document from a tablet device to allow you to zoom in on photographs. If you reproduce this document on paper it should be done in colour otherwise you will not be able to see cable colours and other notations properly.

3.4.2. Required tools

You will need the following tools and equipment:

  • No 2 Phillips screwdriver

  • Torx TX30 screwdriver

  • Scissors suitable for cutting Velcro

3.4.3. Preparing the rack

3.4.3.1. Installing the rack rails

The IPU-M2000 rail kit comprises two mated inner and outer rack rails and an accessory bag containing screws. The inner rail affixes to the body of the IPU-M2000 and the outer rail affixes to the vertical rack rails in the server cabinet.

Repeat these instructions for each IPU-M2000 to be installed.

  1. Separate the mated inner and outer rails.

    1. Fully extend the rails by pulling on the end which has the captive thumb screw attached (Fig. 2.9).

      _images/m2000_rack_rail_kit_2.png

      Fig. 3.8 M2000 rack rail kit: extended rails

    2. Whilst pulling on the thumb screw end of the rails, push the white plastic release tab towards the thumb screw end (Fig. 2.10).

      _images/m2000_rack_rail_kit_3.png

      Fig. 3.9 M2000 rack rail kit: white release tab

    3. The inner and outer rails will now separate (Fig. 2.11).

      _images/m2000_rack_rail_kit_4.png

      Fig. 3.10 M2000 rack rail kit: inner and outer rails separated

  2. To affix the inner rail to the body of the IPU-M2000, mate the inner rails (the thinner of the two separated rails which has a captive thumb screw at one end) to the body of the IPU-M2000. Note that the inner rails are identical. As such, the procedure for inner rail fixing is the same for the left and right hand inner rails.

    The inner rail should be oriented such that the captive thumb screw end is at the end of the IPU-M2000 containing the network ports.

    1. Place the inner rail to the side of the IPU-M2000 and ensure that all fixing pins are sitting within the enlarged opening of the retention channel (Fig. 2.12).

      _images/m2000_rack_rail_kit_5.png

      Fig. 3.11 M2000 rack rail kit: fixing pins

    2. Push the inner rail towards the end of the IPU-M2000 containing the network ports, you should hear a click as the latching mechanism locks behind the head of a fixing pin (Fig. 2.13).

      _images/m2000_rack_rail_kit_6.png

      Fig. 3.12 M2000 rack rail kit: fixing pins locked

    3. Ensure all fixing pins are correctly engaged with their respective retention channels.

    4. Locate the four flat head fixing screws from the rack rail accessory bag (Fig. 2.14).

      _images/m2000_rack_rail_kit_7.png

      Fig. 3.13 M2000 rack rail kit: flat head fixing screws

    5. Using these four screws, affix the inner rail to the body of the M2000 (Fig. 2.15). There will be two screws per side.

      _images/m2000_rack_rail_kit_8.png

      Fig. 3.14 M2000 rack rail kit: attaching inner rail to M2000

      The inner rails are now securely affixed to the IPU-M2000 body.

  3. Install the outer rack rails in the rack.

    1. Locate the front and rear of the outer rail. The front of the outer rail is embossed with the word “FRONT” and the rear of the outer rail has a large metal latch (as shown in Fig. 2.16).

      _images/m2000_rail_1.png

      Fig. 3.15 Attachment point for the rear of the IPU-M2000 outer rail

    2. Pull on each end of the outer rail to adjust the rail length to suit your rack

    3. Hold the front end of the outer rail (embossed with the word “FRONT”) behind the square holes in the vertical rack rail. Pull the outer rail towards the vertical rack rail and the latching mechanism will click and hold the outer rail in place (Fig. 2.17).

      _images/m2000_rail_2.png

      Fig. 3.16 IPU-M2000 outer rail front latching mechanism

    4. Hold the rear end of the outer rail and slightly open the large metal latch, then press the upper and lower locating pins into the square holes in the vertical rack rail. Release the large metal latch and the outer rail will now be secured to the vertical rack rail (Fig. 2.18).

      _images/m2000_rail_3.png

      Fig. 3.17 Rear of IPU-M2000 outer rail secured to vertical rack rail

    5. Find the screws and washers (Fig. 2.19) in the rack rail accessory bag. One screw and one washer should be screwed through the vertical rack rail and into the outer rack rail threaded hole. The washer should be used in such a way that the washer sits flush with the head of the screw –like a cup (Fig. 2.20).

      _images/m2000_rail_4.png

      Fig. 3.18 IPU-M2000 rack rail kit: outer rail screws and washers

      _images/m2000_rail_5.png

      Fig. 3.19 IPU-M2000 rack rail kit: outer rail screws and washers attached.

    6. This should be repeated for both outer rack rails.

3.4.3.2. Installing PDU brackets

Install four cage nuts on the outside at the top and bottom to both of the accessory channels as shown below (Fig. 3.20 and Fig. 3.21).

_images/top_pdu_bracket_cage_nuts.png

Fig. 3.20 Top PDU bracket cage nuts

_images/bottom_pdu_bracket_cage_nuts.png

Fig. 3.21 Bottom PDU bracket cage nuts

Screw the PDU support brackets to the inside of the cabinet. The PDU brackets should be installed at the rear of the rack: one bracket on top with 9cm distance from the top of rack and one bracket on the bottom with 12cm distance from the bottom of rack. Fig. 3.22 illustrates this. Follow the PDU bracket installation instructions.

_images/pdu_support_bracket.png

Fig. 3.22 PDU support bracket

3.4.4. Installing the IPU-M2000s

Install each of the 16 IPU-M2000s in rack slots #1 to #16..

  1. Pull the sliding rail located within the outer rack rail completely forward such that it locks into the fully extended position (Fig. 2.21).

    _images/m2000_rail_6.png

    Fig. 3.23 IPU-M2000 rack rail kit: sliders fully extended

  2. Place the IPU-M2000 onto an appropriate server lift and adjust the height such that it is suitable for the sliders (Fig. 2.22). If a lift is not available, this is a two person operation.

    _images/m2000_rail_7.png

    Fig. 3.24 Server lift for IPU-M2000

  3. Slide the protruding inner rails (on the IPU-M2000) into the receiving channel of the extended outer rails (Fig. 2.23).

    _images/m2000_rail_8.png

    Fig. 3.25 Slide IPU-M2000 inner rails into outer rails

  4. Whilst the server lift is supporting the full weight of the IPU-M2000, slide the IPU-M2000 into the extended outer rails until you feel both sides engage a stopping mechanism (Fig. 2.25).

  5. Then, simultaneously pull on the blue tabs for the release mechanism at each side of the IPU-M2000 and then push the IPU-M2000 unit fully into the rack (Fig. 2.24 and circled in Fig. 2.25).

    _images/m2000_rail_10.png

    Fig. 3.26 Blue tab release mechanism

    _images/m2000_rail_9.png

    Fig. 3.27 Location of blue tab release mechanism

  6. Finally, screw the captive thumb screw into the inner rack rail. (Fig. 2.26).

    _images/m2000_rail_11.png

    Fig. 3.28 Screwing captive thumb screw into inner rack rail

  7. The IPU-M2000 is now installed.

3.4.5. Installing the management switch

Insert 2 cage nuts, inside the rack, on either side of the rack into the top and bottom positions of rack slot #17.

Place the management switch on top of the last IPU-M2000 and screw it into position using four M6 screws (Fig. 3.29).

_images/mng_switch_install.png

Fig. 3.29 Installing management switch with four M6 screws

3.4.6. Installing the ToR switch

Fit the sliders for the ToR switch into rack slot #18 ensuring both ends of the slider are pushed firmly into the mounting slots on the rack rail (Fig. 3.30).

_images/ToR_switch_install.png

Fig. 3.30 Sliders for installing ToR switch

Insert the ToR switch ensuring the wheel on the switch is located in the groove on the slider.

3.4.7. Installing the PDUs

Install the two PDUs vertically at the rear of the rack, one on the left side and one on the right side. Push the mains cable through the roof of the rack and then clip the PDUs onto the PDU bracket as shown in Fig. 3.31.

_images/pdu_install.png

Fig. 3.31 PDU installation

3.4.8. Installing the Dell R6525 server(s)

The IPU‑POD64 reference design has a single server installed in rack slot #19; in four-server configuration they are installed in rack slots #19 to #22.

This section describes the installation of a Dell R6525 server. A list of approved and qualified servers is available. To find out more, speak to Graphcore sales or your Graphcore channel partner.

Note

The R6525 is installed such that the rear of the server (containing the management and data ports) is on the opposite side of the rack to the ports on the front of the IPU-Machine(s). This is due to the airflow direction.

  1. Install the tool-less sliding rail kit(s).

    Pull out the rail and fit the server to the rail ensuring the T pins on the side of the server locate in the slots on the rail. Ensure that the power supplies on the server face the rear of the rack (Fig. 2.27).

    _images/server_install_2.jpg

    Fig. 3.32 Sliding rail kit for server installation

    Note

    Use an appropriate server lift or have two people installing the server to ensure correct fitting

  2. Push the server gently from the front to lock it into the slides then press the tab on the side of the slides and push the server fully home in the rack. Repeat the above process for each server if installing multiple servers.

3.5. Cabling the rack

The following sections detail the cabling of the rack and the dressing of the cables within the rack. For reference, the IPU-M2000s and server(s) are numbered as shown in Fig. 3.33.

_images/rack_numbering.png

Fig. 3.33 Server and IPU-M2000 numbering

3.5.3. IPU-M2000 to management switch cabling (RJ45)

There are two Ethernet ports in the middle of each IPU-M2000 (see Fig. 3.40). One of them is a BMC + IPU-Gateway port (upper port) and the other is a BMC port (lower port).

_images/m2000_ports_1.png

Fig. 3.40 IPU-M2000 management switch ports

These are connected from each IPU-M2000 to the management switch (Fig. 3.41) with RJ45 cables. The cables that are required are:

  • 12x RJ45/Yellow 1.0m and 4x RJ45/Yellow 1.5m (BMC)

  • 8x RJ45/Blue 1.0m and 8x RJ45/Blue 1.5m (BMC + IPU-Gateway)

This cabling is described in detail in Section 3.5.4, Management switch: BMC cabling and Section 3.5.5, Management switch: BMC + IPU-Gateway cabling.

_images/mng_switch.png

Fig. 3.41 Management switch ports

Table 3.5 shows the management switch port allocation.

Table 3.5 Management switch port allocation

Ports

Allocation

1

Customer datacentre uplink for BMC + IPU-Gateway

2

Customer datacentre uplink for BMC-only (future update)

3

Customer management interface

5,6

PDU management

7

Server 1 PDU management switch

8

ToR to management switch

9-12

1GbE server management (iDRAC)

13-16

Server data ports

17-32

IPU-M2000 BMC + IPU-Gateway combined management

33-48

IPU-M2000 BMC-only management (future update)

3.5.4. Management switch: BMC cabling

Start cabling using a 1.0m yellow cable and insert one end into port 48 of the management switch.

Run the cable through the loop in the OSFP connector as shown in the left hand picture in Fig. 3.42 and connect it to the BMC port on IPU-M2000 #16 (top IPU-M2000 in the rack).

_images/BMC_wiring.png

Fig. 3.42 Management switch BMC cabling

Repeat the process for ports 47 to 37 using 1.0m yellow cables and ports 33 to 36 using 1.5m cables.

When all cables have been connected, dress the loom down the side of the cabinet and secure the bundle with a Velcro strip as shown in the right hand picture in Fig. 3.42.

The port mapping between the management switch and the IPU-M2000 BMC sockets is given in Table 3.6.

Table 3.6 IPU-M2000 management BMC-only mapping

From

To

Cables

IPU-M2000 #16 BMC port

Management switch port 48

RJ45 1.0m yellow

IPU-M2000 #15 BMC port

Management switch port 47

RJ45 1.0m yellow

IPU-M2000 #14 BMC port

Management switch port 46

RJ45 1.0m yellow

IPU-M2000 #13 BMC port

Management switch port 45

RJ45 1.0m yellow

IPU-M2000 #12 BMC port

Management switch port 44

RJ45 1.0m yellow

IPU-M2000 #11 BMC port

Management switch port 43

RJ45 1.0m yellow

IPU-M2000 #10 BMC port

Management switch port 42

RJ45 1.0m yellow

IPU-M2000 #9 BMC port

Management switch port 41

RJ45 1.0m yellow

IPU-M2000 #8 BMC port

Management switch port 40

RJ45 1.0m yellow

IPU-M2000 #7 BMC port

Management switch port 39

RJ45 1.0m yellow

IPU-M2000 #6 BMC port

Management switch port 38

RJ45 1.0m yellow

IPU-M2000 #5 BMC port

Management switch port 37

RJ45 1.0m yellow

IPU-M2000 #4 BMC port

Management switch port 36

RJ45 1.5m yellow

IPU-M2000 #3 BMC port

Management switch port 35

RJ45 1.5m yellow

IPU-M2000 #2 BMC port

Management switch port 34

RJ45 1.5m yellow

IPU-M2000 #1 BMC port

Management switch port 33

RJ45 1.5m yellow

3.5.5. Management switch: BMC + IPU-Gateway cabling

Start cabling using a 1.0m blue RJ45 cable and insert one end into port 32 of the management switch.

Run the cable through the loop in the OSFP connector as shown in the left hand picture in Fig. 3.43 and connect it to the BMC + IPU-Gateway port on IPU-M2000 #16.

_images/BMC_GW_wiring.png

Fig. 3.43 Management switch BMC + IPU-Gateway cabling

Repeat the process for ports 31 to 25 using 1.0m blue cables, and ports 24 to 17 using 1.5m blue cables. When all cables have been connected, dress the loom down the side of the cabinet and secure the bundle with a Velcro strip, as shown in the right hand picture in Fig. 3.43. Using a 1.0m blue cable, connect port 7 of the management switch to the top RJ45 connector on the ToR Switch (Fig. 3.44).

_images/ToR_switch_top_RJ5.png

Fig. 3.44 ToR switch: top RJ45 connector

With the IPU-M2000s numbered from bottom to top, the BMC + IPU-Gateway ports should be connected to the management switch as given in Table 3.7.

Table 3.7 IPU-M2000 management BMC + IPU-Gateway port mapping

From

To

Cables

IPU-M2000 #16 BMC + IPU-Gateway port

Management switch port 32

RJ45 1.0m blue

IPU-M2000 #15 BMC + IPU-Gateway port

Management switch port 31

RJ45 1.0m blue

IPU-M2000 #14 BMC + IPU-Gateway port

Management switch port 30

RJ45 1.0m blue

IPU-M2000 #13 BMC + IPU-Gateway port

Management switch port 29

RJ45 1.0m blue

IPU-M2000 #12 BMC + IPU-Gateway port

Management switch port 28

RJ45 1.0m blue

IPU-M2000 #11 BMC + IPU-Gateway port

Management switch port 27

RJ45 1.0m blue

IPU-M2000 #10 BMC + IPU-Gateway port

Management switch port 26

RJ45 1.0m blue

IPU-M2000 #9 BMC + IPU-Gateway port

Management switch port 25

RJ45 1.0m blue

IPU-M2000 #8 BMC + IPU-Gateway port

Management switch port 24

RJ45 1.5m blue

IPU-M2000 #7 BMC + IPU-Gateway port

Management switch port 23

RJ45 1.5m blue

IPU-M2000 #6 BMC + IPU-Gateway port

Management switch port 22

RJ45 1.5m blue

IPU-M2000 #5 BMC + IPU-Gateway port

Management switch port 21

RJ45 1.5m blue

IPU-M2000 #4 BMC + IPU-Gateway port

Management switch port 20

RJ45 1.5m blue

IPU-M2000 #3 BMC + IPU-Gateway port

Management switch port 19

RJ45 1.5m blue

IPU-M2000 #2 BMC + IPU-Gateway port

Management switch port 18

RJ45 1.5m blue

IPU-M2000 #1 BMC + IPU-Gateway port

Management switch port 17

RJ45 1.5m blue

3.5.6. IPU-M2000 to ToR switch cabling (QSFP)

The next step is to connect the IPU-M2000s to the ToR switch. There are two RNIC ports on the left side of each IPU-M2000, as shown in Fig. 3.45. Only one of them should be connected from each IPU-M2000 to the ToR switch with either the 1m or 1.5m QSFP cables supplied.

_images/m2000_RNIC_ports.png

Fig. 3.45 IPU-M2000 RNIC ports (QSFP)

In order to manage the cables, the QSFP cables are divided into two different lengths:

  • 8x QSFP 1.0m from IPU-M2000 #9-16 to ToR switch ports 16 to 9

  • 8x QSFP1.5m from IPU-M2000 #1-8 to ToR switch ports 17 to 24

Start from port 9 on the ToR switch and connect the cable to port 2 of IPU-M2000 #16.

Continue cabling from ports 10 to 24 on the ToR switch to port 2 on IPU-M2000 #15 to IPU-M2000 #1 respectively.

Dress the cables down the side of the cabinet. A set of four cables can be looped into cut-outs on the side of the cabinet. Tie the bundle together at the top with a Velcro strip as shown in Fig. 3.46.

_images/ToR_cabling.png

Fig. 3.46 IPU-M2000 to ToR switch cabling

The IPU-M2000s should be connected to the ToR switch as given in Table 3.8, note that the IPU-M2000s are numbered from bottom to top #1 to #16.

Table 3.8 IPU-M2000 RNIC port mapping

IPU-M2000 port

ToR switch port

Cables

IPU-M2000 #16 port 2

ToR switch port 9

QSFP 1.0m

IPU-M2000 #15 port 2

ToR switch port 10

QSFP 1.0m

IPU-M2000 #14 port 2

ToR switch port 11

QSFP 1.0m

IPU-M2000 #13 port 2

ToR switch port 12

QSFP 1.0m

IPU-M2000 #12 port 2

ToR switch port 13

QSFP 1.0m

IPU-M2000 #11 port 2

ToR switch port 14

QSFP 1.0m

IPU-M2000 #10 port 2

ToR switch port 15

QSFP 1.0m

IPU-M2000 #9 port 2

ToR switch port 16

QSFP 1.0m

IPU-M2000 #8 port 2

ToR switch port 17

QSFP 1.5m

IPU-M2000 #7 port 2

ToR switch port 18

QSFP 1.5m

IPU-M2000 #6 port 2

ToR switch port 19

QSFP 1.5m

IPU-M2000 #5 port 2

ToR switch port 20

QSFP 1.5m

IPU-M2000 #4 port 2

ToR switch port 21

QSFP 1.5m

IPU-M2000 #3 port 2

ToR switch port 22

QSFP 1.5m

IPU-M2000 #2 port 2

ToR switch port 23

QSFP 1.5m

IPU-M2000 #1 port 2

ToR switch port 24

QSFP 1.5m

Fig. 3.47 shows the final IPU-M2000 to ToR switch cabling.

_images/ToR_cabling_final.png

Fig. 3.47 Final IPU-M2000 to ToR switch cabling

3.5.7. Dell R6525 server(s) cabling

All cables should be routed from the rear of the server to the right-hand side when viewed from the rear (see Fig. 3.48), then along the side of the rack using the cable management holes in the vertical rack rails.

Details on how to install these cables are given in the followings sections - Section 3.5.8, ToR switch to Dell server(s), Section 3.5.9, Management switch to Dell server(s): iDRAC and Section 3.5.10, Management switch to Dell server(s): network connector.

_images/R6525_wiring.png

Fig. 3.48 Server cabling (4 server version)

Note that Fig. 3.48 shows the four-server version. The default build has one server (the one in the lowest position).

Note

Picture to be updated: Fig. 3.48 shows three blue RJ45 cables in each R6525 server. In the default build, servers 2 to 4 only have two blue RJ45 cables. The cables removed in the default build are marked with a red cross. The additional cable in server 1 moves to the port on the right, as indicated by the red arrow.

3.5.8. ToR switch to Dell server(s)

Using two of the 1.5m QSFP cables per server, connect the ToR switch to the server(s) as follows:

  1. Feed the cables through the cut-outs in the side of the rack first (Fig. 3.49).

    _images/ToR_switch_R6525_1.png

    Fig. 3.49 ToR switch to server cabling: feed cables through side cut-outs in rack

  2. Connect the cables to the ToR switch and server as given in Table 3.9.

    Table 3.9 Server RNIC port mapping

    Server RNIC port

    ToR switch port

    Cables

    Server # 4 RNIC port 1

    ToR switch port # 1

    QSFP 1.5m

    Server # 4 RNIC port 2

    ToR switch port # 2

    QSFP 1.5m

    Server # 3 RNIC port 1

    ToR switch port # 3

    QSFP 1.5m

    Server # 3 RNIC port 2

    ToR switch port # 4

    QSFP 1.5m

    Server # 2 RNIC port 1

    ToR switch port # 5

    QSFP 1.5m

    Server # 2 RNIC port 2

    ToR switch port # 6

    QSFP 1.5m

    Server # 1 RNIC port 1

    ToR switch port # 7

    QSFP 1.5m

    Server # 1 RNIC port 2

    ToR switch port # 8

    QSFP 1.5m

    ToR switch ports 8 and 7 connect to the bottom server (server 1). Therefore, if you are only using 1 server then only ports 8 and 7 of the ToR switch are used.

    The final ToR switch to server cabling is shown in Fig. 3.50 for a four server version.

    _images/ToR_switch_R6525_2.png

    Fig. 3.50 ToR switch to server cabling: cable connections (4 server version)

3.5.9. Management switch to Dell server(s): iDRAC

Using four of the 1.5m blue RJ45 cables connect the management switch to the Dell server(s) as follows:

  • Ports 9 to 12 on the management switch connect to the iDRAC connector on the server(s) - Fig. 3.51.

  • Port 9 is connected to the bottom server (server 1). If you only have one server then only port 9 on the management switch is used.

_images/mng_switch_R6525_1.png

Fig. 3.51 Management switch to server cabling: iDRAC for 4 server version

3.5.10. Management switch to Dell server(s): network connector

Using four of the 1.5m blue RJ45 cables connect the management switch to the Dell server(s) as follows:

  • Ports 13 to 16 on the management switch connect to the network connector on the server(s) - Fig. 3.52.

  • Port 13 is connected to the bottom server (server 1). If you only have one server then only port 13 on the management switch is used.

_images/mng_switch_R6525_2.png

Fig. 3.52 Management switch to server cabling: network connector(s) for 4 server version

3.5.11. Management switch to Dell server(s): switch management

Using one of the 1.5m blue RJ45 cables connect the management switch to server 1 as follows:

  • Port 8 from the management server is connected to the lowest server (server 1) - the correct port to connect to on the server is shown in Fig. 3.53. This is used for control of the PDUs in the case where server 1 is used as the management server.

_images/mng_switch_R6525_3.png

Fig. 3.53 Management switch to server cabling: switch management

3.5.12. Management switch to PDUs

Using the two 2m purple RJ45 cables connect the management switch to the two PDUs as given in Table 3.10.

Table 3.10 Management switch to PDU connections

Ethernet port | Management switch port

Cables

Ethernet port on left side PDU | Management switch port 5

RJ45 2m

Ethernet port on right side PDU

Management switch port 6

RJ45 2m

Allow all the cables going to the server to hang down in the rack as shown in Fig. 3.54. This allows the cables to be pulled slightly if it is necessary to remove them from a server.

_images/mng_switch_PDU.png

Fig. 3.54 Management switch cabling to PDUs

3.6. Power cabling

The designated sockets on the PDUs for servers and IPU-M2000s are shown in Fig. 3.55.

_images/power_cabling_1.png

Fig. 3.55 Designated PDU sockets for server(s) and IPU-M2000s

Fig. 3.56 and Fig. 3.57 show the final power cabling.

_images/power_cabling_2.png

Fig. 3.56 Final power cabling (direct view)

_images/power_cabling_3.png

Fig. 3.57 Final power cabling (angled view)

3.6.1. IPU-M2000 power cabling

Start by cabling IPU-M2000 #1 using 0.5m power cables. Ensure that only three IPU-M2000s are connected to the same bank on the PDU. Red cables to the left PDU and blue cables to the right PDU (as seen looking at the rear of the rack). Table 3.11 defines the power cable lengths for each IPU-M2000.

Table 3.11 Power cables for IPU-M2000s

Cable colour

Cable length

IPU-M2000

Blue

1m

IPU-M2000 #13 to IPU-M2000 #16

Red

1m

IPU-M2000 #13 to IPU-M2000 #16

Blue

0.5m

IPU-M2000 #1 to IPU-M2000 #12

Red

0.5m

IPU-M2000 #1 to IPU-M2000 #12

Fig. 3.58 and Fig. 3.59 show part of the IPU-M2000 power cabling for the red PDU and blue PDU, respectively.

_images/power_cabling_4.png

Fig. 3.58 IPU-M2000 power cabling (red PDU)

_images/power_cabling_5.png

Fig. 3.59 IPU-M2000 power cabling (blue PDU)

3.6.2. Server power cabling: Dell R6525

Using 1m C13 to C14 power cables (selecting the correct coloured cable to match the PDU colour), connect the server(s) to the PDUs as shown in Fig. 3.60.

_images/server_power_wiring.png

Fig. 3.60 Server power cabling

Note

The photo above shows the four-server version. The default reference design has one server (the one in the lowest position).

3.6.3. Switch power cabling

Connect the mains cables to the management switch and ToR switch using C13 to C14 power cables as given in Table 3.12.

Table 3.12 Management switch and ToR cables

Switch

Cable length (red/blue)

Management

1.0m

ToR

1.5m

3.7. Completing the rack

The following steps describe completing the rack: fitting blanking panels and re-installing the doors and side panels.

3.7.1. Blanking panels

Install the supplied APC 1U blanking panels in every unoccupied rack slot at the front of the rack.

_images/blanking_panels.png

Fig. 3.61 1U blanking panels

3.7.2. Front and rear doors

Re-install the front and rear doors. Ensure the earth cables are reconnected to the cable on the rack.

_images/earth_straps.jpg

Fig. 3.62 Rack earth cable

3.7.3. Side panels

Re-install the top and bottom side panels on each side of the rack.

3.7.4. PDU plugs

The power plugs for the PDUs must not exit the top of the rack, they must stay inside the rack and be secured to the vertical mounting flanges. You need to ensure that there are no cables on the outside of the rack.

3.8. Upgrade software on management server

For a server that was previously configured for an IPU‑POD16 DA system, you need to uninstall the direct attach configuration and then install the software as for an IPU‑POD64 system.

3.8.1. Uninstall direct attach configuration

You run the IPU-POD DA uninstall script from the ipuuser account as root by using sudo. There is no need to do an uninstall before upgrading to a new release.

cd  ~/IPU-M_releases/IPU_M_SW-<release>
sudo ./direct-attach uninstall

3.8.2. Install IPU‑POD64 software

Install the rest of the software as described in the IPU‑POD64 build and test guide.