3. IPU-POD64 rack assembly

You will need to follow the instructions in this section for each of the two IPU‑POD64 logical racks you need for the IPU‑POD128.

Note the correct orientation of the IPU-M2000, server and switch units in the rack to ensure correct airflow.

The front interface of the IPU-M2000 units (connectivity ports) should be matched with the front door of the rack (cold aisle). The rear interface of the server and switches (power and fans) should be matched with the rear door of the rack (hot aisle).

Note

You can click on any of the images below to view them full size

_images/completed_rack_cold_four_server.jpg

Fig. 3.1 Completed rack: cold aisle (four-server version)

Note that Fig. 3.1 shows a four-server version of the IPU‑POD64. The default reference design has one server which would be the server in the lowest position, closest to the switches.

_images/completed_rack_hot_four_server.jpg

Fig. 3.2 Completed rack: hot aisle (four-server version)

Note that Fig. 3.2 shows three blue RJ45 cables in each R6525 server. In the default build, servers 2 to 4 only have two blue RJ45 cables. See Section 3.6.7, Dell R6525 server(s) cabling for more information about server cabling.

Note also that Fig. 3.2 shows a four-server version of the IPU‑POD64. The default reference design has one server which would be the server in the lowest position, closest to the switches.

3.1. Equipment checklist

Table 3.1 Equipment checklist

Description

Quantity (1 server)

Quantity (4 server)

Rack (AR3300SP)

1

1

Blanking panels (APC AR8136BLK)

23 pieces for 42U reference rack (delivered in packs of 10)

20 pieces for 42U reference rack (delivered in packs of 10)

AP8886 PDU

2

2

Hardware mounting kit APC AR8100

1

1

PDU bracket kit APC AR7711

2

2

Graphcore IPU-M2000

16

16

IPU-M2000 slider kits

16

16

Dell R6525 server

1

4

Arista DCS-7010T-48-F switch

1

1

Arista DCS-7060CX-32S-F switch

1

1

2m purple Ethernet

2

2

1.5m blue Ethernet

11

17

1m blue Ethernet

9

9

1m yellow Ethernet

12

12

1.5m yellow Ethernet

4

4

1m red Ethernet

2

2

0.15m red Ethernet

30

30

1m QSFP28

8

8

1.5m QSFP28

10

16

0.3m OSFP

60

60

1m OSFP

4

4

0.5m red 10A C14 to C15

12

12

1m red 10A C14 to C15

4

4

0.5m blue 10A C14 to C15

12

12

1m blue 10A C14 to C15

4

4

1m red C13 to C14

2

5

1.5m red C13 to C14

1

1

1m blue C13 to C14

2

5

1.5m blue C13 to C14

1

1

Velcro

1

1

3.2. Document reproduction

Ideally you should reference this document from a tablet device to allow you to zoom in on photographs. If you reproduce this document on paper it should be done in colour otherwise you will not be able to see cable colours and other notations properly.

3.3. Required tools

You will need the following tools and equipment:

  • Pallet truck with wide fork spacing (at least 32cm between forks) and at least 750KG SWL

  • Knife suitable for cutting plastic tape

  • No 2 Phillips screwdriver

  • Torx TX30 screwdriver

  • Scissors suitable for cutting Velcro

If you will be re-packaging the rack after building it then you will also need:

  • Pallet banding tape, banding clips and banding crimping tool

  • Packaging tape

3.4. Preparing the rack

3.4.1. Rail distance

The IPU-M2000 mounting system requires a rail-to-rail distance of 720mm. This document describes the adjustments required for an AR3300SP rack. If using a different rack this rail distance must be observed.

3.4.2. Unpacking the rack

Follow the instructions to remove the outer packaging of the APC AR3300SP rack, ensuring that you safely store these materials for later repackaging. Do not remove the rack from the shock pallet. Remove the white bag (Fig. 3.3) from the rack. This contains screws and cage nuts to be used in the assembly of the components into the rack.

_images/white_bag.jpg

Fig. 3.3 White bag with rack

3.4.3. Removing the side panels and doors

Remove the front and rear doors from the rack. Ensure the earth straps (Fig. 3.4) are disconnected before the doors are removed.

_images/earth_straps.jpg

Fig. 3.4 Rack earth straps

Remove the top and bottom side panels. The vertical accessory channels should be positioned at the very front and very rear of the rack. If necessary, move these from their shipping positions (Fig. 3.5).

_images/top_bottom_side_panels.png

Fig. 3.5 Top and bottom side panel removal

3.4.4. Removing the vertical accessory channels

Using a Torx TX30 screwdriver, remove two accessory channels from the rack (Fig. 3.6).

_images/accessory_channel_removal.png

Fig. 3.6 Accessory channel removal

3.4.5. Adjusting the rear accessory channels

Set the rear accessory channel to the furthest position in the rack. Tighten up the screws ensuring the teeth engage into the slots in the rail, as shown in Fig. 3.7.

_images/rear_accessory_channel.png

Fig. 3.7 Rear accessory channel (ensure teeth engage in slots)

3.4.6. Adjusting the rear vertical rails

Using a Torx TX30 screwdriver, make both rear vertical rack rails loose and freely movable.

Position the rear vertical rack rails such that there is 20mm of distance between the rear face of the vertical rack rail and the racks rear frame. This should result in a square symbol being visible through the alignment window at the top and bottom of the rail, as shown in Fig. 3.8.

_images/rear_vertical_rails.jpg

Fig. 3.8 Rear vertical rails

Secure the rail into position by moving the TX30 screws back upwards such that the teeth engage with both the supporting rails. This must be done at the top and bottom of the bracket.

3.4.7. Adjusting the front vertical rails

Using a Torx TX30 screwdriver, make both front vertical rack rails loose and freely movable. Install the accessory channels in the front of the rack (one on the left hand side, one on the right hand side) at the frontmost position possible, moving the TX30 screws back upwards such that the teeth engage with both the supporting rails. This must be done at the top and bottom of the bracket.

Note

To ensure the clips on the accessory channels align with the channel in the rack, lift the accessory channels through the cut-out in the top of the rack and then drop them down onto the channels.

Move the vertical rack rails tight against the vertical cable organisers such that only a single diamond symbol is visible through the alignment window at the top and bottom of the rail, see Fig. 3.9.

_images/front_vertical_rails.jpg

Fig. 3.9 Front vertical rail alignment

Secure the rail into position by moving the TX30 screws back upwards such that the teeth engage with both the supporting rails. This must be done at the top and bottom of the bracket.

3.4.8. Installing the rack rails

_images/m2000_rack_rail_kit_1.png

Fig. 3.10 M2000 rack rail kit: unboxed

The IPU-M2000 rail kit comprises two mated inner and outer rack rails and an accessory bag containing screws. The inner rail affixes to the body of the IPU-M2000 and the outer rail affixes to the vertical rack rails in the server cabinet. Firstly, separate the mated inner and outer rails:

  1. Fully extend the rails by pulling on the end which has the captive thumb screw attached (Fig. 3.11):

_images/m2000_rack_rail_kit_2.png

Fig. 3.11 M2000 rack rail kit: extended rails

  1. Whilst pulling on the thumb screw end of the rails, push the white plastic release tab towards the thumb screw end (Fig. 3.12):

_images/m2000_rack_rail_kit_3.png

Fig. 3.12 M2000 rack rail kit: white release tab

  1. The inner and outer rails will now separate (Fig. 3.13):

_images/m2000_rack_rail_kit_4.png

Fig. 3.13 M2000 rack rail kit: inner and outer rails separated

Mate the inner rails (the thinner of the two separated rails which has a captive thumb screw at one end) to the body of the IPU-M2000. Note that the inner rails are mirrored and are not handed. As such, the procedure for inner rail fixing is the same for the left and right hand inner rails.

The inner rail should be oriented such that the captive thumb screw end is at the end of the IPU-M2000 containing the network ports.

To affix the inner rail to the body of the IPU-M2000:

  1. Offer up the inner rail to the side of the IPU-M2000 and ensure that all fixing pins are sitting within the enlarged opening of the retention channel (Fig. 3.14):

_images/m2000_rack_rail_kit_5.png

Fig. 3.14 M2000 rack rail kit: fixing pins

  1. Push the inner rail towards the end of the IPU-M2000 containing the network ports, you should hear a click as the latching mechanism locks behind the head of a fixing pin (Fig. 3.15):

_images/m2000_rack_rail_kit_6.png

Fig. 3.15 M2000 rack rail kit: fixing pins locked

  1. Ensure all fixing pins are correctly engaged with their respective retention channels.

  2. Locate the four flat head fixing screws from the rack rail accessory bag (Fig. 3.16):

_images/m2000_rack_rail_kit_7.png

Fig. 3.16 M2000 rack rail kit: flat head fixing screws

  1. Using the above screws, affix the inner rail to the body of the M2000 (Fig. 3.17):

_images/m2000_rack_rail_kit_8.jpg

Fig. 3.17 M2000 rack rail kit: attaching inner rail to M2000

The inner rails are now securely affixed to the IPU-M2000 body. Place the outer rails to one side for later use (Fig. 3.18):

_images/m2000_rack_rail_kit_9.png

Fig. 3.18 M2000 rack rail kit: outer rails

3.4.9. Installing PDU brackets

Install four cage nuts on the outside at the top and bottom to both of the accessory channels as shown below (Fig. 3.19 and Fig. 3.20).

_images/top_pdu_bracket_cage_nuts.png

Fig. 3.19 Top PDU bracket cage nuts

_images/bottom_pdu_bracket_cage_nuts.png

Fig. 3.20 Bottom PDU bracket cage nuts

Screw the PDU support brackets to the inside of the cabinet. The PDU brackets should be installed at the rear of the rack: one bracket on top with 9cm distance from the top of rack and one bracket on the bottom with 12cm distance from the bottom of rack. Fig. 3.21 illustrates this. Follow the PDU bracket installation instructions.

_images/pdu_support_bracket.png

Fig. 3.21 PDU support bracket

3.5. Installing the equipment

The following sections describe the installation of the IPU-M2000s, PDUs, servers, ToR and management switches into the rack.

3.5.1. Installing the IPU-M2000s

Earlier in the guide we affixed the inner rack rails to the IPU-M2000 body. The next step in installing the IPU-M2000s is to install the outer rack rails into the rack.

It is possible to identify the front and rear of the outer rail by finding the large metal latching mechanism. This is to be located at the rear of the rack. The outer rail is also embossed with the text “FRONT” at the front end of the rail.

The outer rail large metal latch end is to be installed at the rear of the rack, as shown in Fig. 3.22.

_images/m2000_rail_1.png

Fig. 3.22 Attachment point for IPU-M2000 outer rail

For each rack U in U1 through U16 (inclusive), perform the steps below with both the left hand and right hand outer rack rails:

  1. Pull on each end of the outer rail to adjust the rail length to suite your rack

  2. Locate the front end of the outer rail and hold it behind the square holes in the vertical rack rail for your installation U. Pull the outer rail towards the vertical rack rail and the latching mechanism will click and hold the outer rail in place (Fig. 3.23):

_images/m2000_rail_2.png

Fig. 3.23 IPU-M2000 outer rail latching mechanism

  1. Locate the rear end of the outer rail and slightly open the large metal latch, then press the upper and lower locating pins into the square holes in the vertical rack rail. Release the large metal latch and the outer rail will now be secured to the vertical rack rail (Fig. 3.24):

_images/m2000_rail_3.png

Fig. 3.24 IPU-M2000 outer rail secured to vertical rack rail

  1. Included in the rack rail accessory bag are two screws and two washers. One screw with one washer should be screwed through the vertical rack rail and into the outer rack rail threaded hole. The washer should be used in such a way that the washer sits flush with the head of the screw –like a cup. The screws and how they fit are shown in Fig. 3.25 and Fig. 3.26.

This should be repeated for both outer rack rails.

_images/m2000_rail_4.png

Fig. 3.25 IPU-M2000 rack rail kit: outer rail screws and washers

_images/m2000_rail_5.png

Fig. 3.26 IPU-M2000 rack rail kit: outer rail screws and washers attached

Next, to install an IPU-M2000 unit into the rack rails, work your way through the following steps in order:

  1. Pull the sliding rail located within the outer rack rail completely forward such that it locks into the fully extended position (Fig. 3.27):

_images/m2000_rail_6.png

Fig. 3.27 IPU-M2000 rack rail kit: sliders fully extended

  1. Place the IPU-M2000 onto an appropriate server lift and adjust the height such that it is suitable for the sliders (Fig. 3.28). If a lift is not available, this is a two person operation.

_images/m2000_rail_7.png

Fig. 3.28 Server lift for IPU-M2000

  1. Slide the protruding inner rails into the receiving channel of the extended outer rails (Fig. 3.29):

_images/m2000_rail_8.png

Fig. 3.29 Slide IPU-M2000 inner rails into outer rails

  1. Whilst the server lift is supporting the full weight of the IPU-M2000, slide the IPU-M2000 into the extended outer rails until you feel both sides engage a stopping mechanism (Fig. 3.30).

  2. Simultaneously pull on the blue tabs for the release mechanism at each side of the IPU-M2000 and then push the IPU-M2000 unit fully into the rack (Fig. 3.31):

_images/m2000_rail_9.png

Fig. 3.30 Sliding IPU-M2000 into rack

_images/m2000_rail_10.png

Fig. 3.31 Blue tab release mechanism

  1. Screw the captive thumb screw into the inner rack rail (Fig. 3.32):

_images/m2000_rail_11.png

Fig. 3.32 Screwing captive thumb screw into inner rack rail

3.5.2. Installing the management switch

Insert 2 cage nuts, inside the rack, on either side of the rack into the top and bottom positions of location 17.

Place the management switch on top of the last IPU-M2000 and screw it into position using four M6 screws (Fig. 3.33).

_images/mng_switch_install.png

Fig. 3.33 Installing management switch with four M6 screws

3.5.3. Installing the ToR switch

Fit the sliders for the ToR switch into position 18 on the rack ensuring both ends of the slider are pushed firmly into the mounting slots on the rack rail (Fig. 3.34).

_images/ToR_switch_install.jpg

Fig. 3.34 Sliders for installing ToR switch

Insert the ToR switch ensuring the wheel on the switch is located in the groove on the slider.

3.5.4. Installing the PDUs

Install the two PDUs vertically at the rear of the rack, one on the left side and one on the right side. Push the mains cable through the roof of the rack and then clip the PDUs onto the PDU bracket as shown in Fig. 3.35:

_images/pdu_install.jpg

Fig. 3.35 PDU installation

3.5.5. Installing the Dell R6525 server(s)

Remove and discard the cable management arm brackets from the rear of each tool-less sliding rail (Fig. 3.36).

_images/server_install_1.jpg

Fig. 3.36 Cable management arm brackets - to be removed

Install the tool-less sliding rail kit(s). The reference design has a single server installed in rack slot #19; in four server configurations they are installed in rack slots #19-#22.

Pull out the rail and fit the server to the rail ensuring the T pins on the side of the server locate in the slots on the rail. Ensure that the power supplies on the server face the rear of the rack (Fig. 3.37).

_images/server_install_2.jpg

Fig. 3.37 Sliding rail kit for server installation

Note

Use an appropriate server lift or have two people installing the servers to ensure correct fitting

Push the server gently from the front to lock it into the slides then press the tab on the side of the slides and push the server fully home in the rack. Repeat the above process for each server if installing multiple servers.

Remove the Velcro tape from the light pipes on the rear of the servers.

Remove the small plastic tab on the left front side of the server bezel and clip the bezel in place on the front of the server ensuring the connection pins on the right hand side of the bezel line up with the connector on the server, as shown in Fig. 3.38.

_images/server_install_3.png

Fig. 3.38 Bezel placement on front of server

3.6. Cabling the rack

The following sections detail the cabling of the rack and the dressing of the cables within the rack. For reference, the IPU-M2000s and server(s) are numbered as shown in Fig. 3.39:

_images/rack_numbering.jpg

Fig. 3.39 Server and IPU-M2000 numbering

3.6.3. IPU-M2000 to management switch cabling (RJ45)

There are two Ethernet ports in the middle of each IPU-M2000 (see Fig. 3.46). One of them is a BMC + IPU-Gateway port (upper port) and the other is a BMC port (lower port).

_images/m2000_ports_1.png

Fig. 3.46 IPU-M2000 management switch ports

These are connected from each IPU-M2000 to the management switch (Fig. 3.47) with RJ45 cables. The cables required are:

  • 12x RJ45/Yellow 1.0m and 4x RJ45/Yellow 1.5m (BMC)

  • 8x RJ45/Blue 1.0m and 8x RJ45/Blue 1.5m (BMC + IPU-Gateway)

This cabling is described in detail in Section 3.6.4, Management switch: BMC cabling and Section 3.6.5, Management switch: BMC + IPU-Gateway cabling.

_images/mng_switch.png

Fig. 3.47 Management switch ports

Table 3.4 shows the management switch port allocation.

Table 3.4 Management switch port allocation

Ports

Allocation

1

Customer datacentre uplink for BMC + IPU-Gateway

2

Customer datacentre uplink for BMC-only (future update)

3

Customer management interface

5,6

PDU management

7

Server 1 PDU management switch

8

ToR to management switch

9-12

1GbE server management (iDRAC)

13-16

Server data ports

17-32

IPU-M2000 BMC + IPU-Gateway combined management

33-48

IPU-M2000 BMC-only management (future update)

3.6.4. Management switch: BMC cabling

Start cabling using a 1.0m yellow cable and insert one end into port 48 of the management switch.

Run the cable through the loop in the OSFP connector as shown in the left hand picture in Fig. 3.48 and connect it to the BMC port on IPU-M2000 #16 (top IPU-M2000 in the rack).

_images/BMC_wiring.jpg

Fig. 3.48 Management switch BMC cabling

Repeat the process for ports 47 to 37 using 1.0m yellow cables and ports 33 to 36 using 1.5m cables.

When all cables have been connected, dress the loom down the side of the cabinet and secure the bundle with a Velcro strip as shown in the right hand picture in Fig. 3.48.

The port mapping between the management switch and the IPU-M2000 BMC sockets is given in Table 3.5.

Table 3.5 IPU-M2000 management BMC-only mapping

From

To

Cables

IPU-M2000 #16 BMC port

Management switch port 48

RJ45 1.0m yellow

IPU-M2000 #15 BMC port

Management switch port 47

RJ45 1.0m yellow

IPU-M2000 #14 BMC port

Management switch port 46

RJ45 1.0m yellow

IPU-M2000 #13 BMC port

Management switch port 45

RJ45 1.0m yellow

IPU-M2000 #12 BMC port

Management switch port 44

RJ45 1.0m yellow

IPU-M2000 #11 BMC port

Management switch port 43

RJ45 1.0m yellow

IPU-M2000 #10 BMC port

Management switch port 42

RJ45 1.0m yellow

IPU-M2000 #9 BMC port

Management switch port 41

RJ45 1.0m yellow

IPU-M2000 #8 BMC port

Management switch port 40

RJ45 1.0m yellow

IPU-M2000 #7 BMC port

Management switch port 39

RJ45 1.0m yellow

IPU-M2000 #6 BMC port

Management switch port 38

RJ45 1.0m yellow

IPU-M2000 #5 BMC port

Management switch port 37

RJ45 1.0m yellow

IPU-M2000 #4 BMC port

Management switch port 36

RJ45 1.5m yellow

IPU-M2000 #3 BMC port

Management switch port 35

RJ45 1.5m yellow

IPU-M2000 #2 BMC port

Management switch port 34

RJ45 1.5m yellow

IPU-M2000 #1 BMC port

Management switch port 33

RJ45 1.5m yellow

3.6.5. Management switch: BMC + IPU-Gateway cabling

Start cabling using a 1.0m blue RJ45 cable and insert one end into port 32 of the management switch.

Run the cable through the loop in the OSFP connector as shown in the left hand picture in Fig. 3.49 and connect it to the BMC + IPU-Gateway port on IPU-M2000 #16.

_images/BMC_GW_wiring.jpg

Fig. 3.49 Management switch BMC + IPU-Gateway cabling

Repeat the process for ports 31 to 25 using 1.0m blue cables, and ports 24 to 17 using 1.5m blue cables. When all cables have been connected, dress the loom down the side of the cabinet and secure the bundle with a Velcro strip, as shown in the right hand picture in Fig. 3.49. Using a 1.0m blue cable, connect port 7 of the management switch to the top RJ45 connector on the ToR Switch (Fig. 3.50).

_images/ToR_switch_top_RJ5.png

Fig. 3.50 ToR switch: top RJ45 connector

With the IPU-M2000s numbered from bottom to top, the BMC + IPU-Gateway ports should be connected to the management switch as given in Table 3.6.

Table 3.6 IPU-M2000 management BMC + IPU-Gateway port mapping

From

To

Cables

IPU-M2000 #16 BMC + IPU-Gateway port

Management switch port 32

RJ45 1.0m blue

IPU-M2000 #15 BMC + IPU-Gateway port

Management switch port 31

RJ45 1.0m blue

IPU-M2000 #14 BMC + IPU-Gateway port

Management switch port 30

RJ45 1.0m blue

IPU-M2000 #13 BMC + IPU-Gateway port

Management switch port 29

RJ45 1.0m blue

IPU-M2000 #12 BMC + IPU-Gateway port

Management switch port 28

RJ45 1.0m blue

IPU-M2000 #11 BMC + IPU-Gateway port

Management switch port 27

RJ45 1.0m blue

IPU-M2000 #10 BMC + IPU-Gateway port

Management switch port 26

RJ45 1.0m blue

IPU-M2000 #9 BMC + IPU-Gateway port

Management switch port 25

RJ45 1.0m blue

IPU-M2000 #8 BMC + IPU-Gateway port

Management switch port 24

RJ45 1.5m blue

IPU-M2000 #7 BMC + IPU-Gateway port

Management switch port 23

RJ45 1.5m blue

IPU-M2000 #6 BMC + IPU-Gateway port

Management switch port 22

RJ45 1.5m blue

IPU-M2000 #5 BMC + IPU-Gateway port

Management switch port 21

RJ45 1.5m blue

IPU-M2000 #4 BMC + IPU-Gateway port

Management switch port 20

RJ45 1.5m blue

IPU-M2000 #3 BMC + IPU-Gateway port

Management switch port 19

RJ45 1.5m blue

IPU-M2000 #2 BMC + IPU-Gateway port

Management switch port 18

RJ45 1.5m blue

IPU-M2000 #1 BMC + IPU-Gateway port

Management switch port 17

RJ45 1.5m blue

3.6.6. IPU-M2000 to ToR switch cabling (QSFP)

The next step is to connect the IPU-M2000s to the ToR switch. There are two RNIC ports on the left side of each IPU-M2000, as shown in Fig. 3.51. Only one of them should be connected from each IPU-M2000 to the ToR switch with either the 1m or 1.5m QSFP cables supplied.

_images/m2000_RNIC_ports.png

Fig. 3.51 IPU-M2000 RNIC ports (QSFP)

In order to manage the cables, the QSFP cables are divided into two different lengths:

  • 8x QSFP 1.0m from IPU-M2000 #9-16 to ToR switch ports 16 to 9

  • 8x QSFP1.5m from IPU-M2000 #1-8 to ToR switch ports 17 to 24

Start from port 9 on the ToR switch and connect the cable to port 2 of IPU-M2000 #16.

Continue cabling from ports 10 to 24 on the ToR switch to port 2 on IPU-M2000 #15 to IPU-M2000 #1 respectively.

Dress the cables down the side of the cabinet. A set of four cables can be looped into cut-outs on the side of the cabinet. Tie the bundle together at the top with a Velcro strip as shown in Fig. 3.52.

_images/ToR_cabling.png

Fig. 3.52 IPU-M2000 to ToR switch cabling

The IPU-M2000s should be connected to the ToR switch as given in Table 3.7, note that the IPU-M2000s are numbered from bottom to top #1 to #16.

Table 3.7 IPU-M2000 RNIC port mapping

IPU-M2000 port

ToR switch port

Cables

IPU-M2000 #16 port 2

ToR switch port 9

QSFP 1.0m

IPU-M2000 #15 port 2

ToR switch port 10

QSFP 1.0m

IPU-M2000 #14 port 2

ToR switch port 11

QSFP 1.0m

IPU-M2000 #13 port 2

ToR switch port 12

QSFP 1.0m

IPU-M2000 #12 port 2

ToR switch port 13

QSFP 1.0m

IPU-M2000 #11 port 2

ToR switch port 14

QSFP 1.0m

IPU-M2000 #10 port 2

ToR switch port 15

QSFP 1.0m

IPU-M2000 #9 port 2

ToR switch port 16

QSFP 1.0m

IPU-M2000 #8 port 2

ToR switch port 17

QSFP 1.5m

IPU-M2000 #7 port 2

ToR switch port 18

QSFP 1.5m

IPU-M2000 #6 port 2

ToR switch port 19

QSFP 1.5m

IPU-M2000 #5 port 2

ToR switch port 20

QSFP 1.5m

IPU-M2000 #4 port 2

ToR switch port 21

QSFP 1.5m

IPU-M2000 #3 port 2

ToR switch port 22

QSFP 1.5m

IPU-M2000 #2 port 2

ToR switch port 23

QSFP 1.5m

IPU-M2000 #1 port 2

ToR switch port 24

QSFP 1.5m

Fig. 3.53 shows the final IPU-M2000 to ToR switch cabling.

_images/ToR_cabling_final.png

Fig. 3.53 Final IPU-M2000 to ToR switch cabling

3.6.7. Dell R6525 server(s) cabling

All cables should be routed from the rear of the server to the right-hand side when viewed from the rear (see Fig. 3.54), then along the side of the rack using the cable management holes in the vertical rack rails.

Details on how to install these cables is given in the followings ections - Section 3.6.8, ToR switch to Dell server(s), Section 3.6.9, Management switch to Dell server(s): iDRAC and Section 3.6.10, Management switch to Dell server(s): network connector.

_images/R6525_wiring.png

Fig. 3.54 Server cabling (4 server version)

Note that Fig. 3.54 shows the four-server version. The default build has one server (the one in the lowest position).

Note

Picture to be updated: Fig. 3.54 shows three blue RJ45 cables in each R6525 server. In the default build, servers 2 to 4 only have two blue RJ45 cables. The cables removed in the default build are marked with a red cross. The additional cable in server 1 moves to the port on the right, as indicated by the red arrow.

3.6.8. ToR switch to Dell server(s)

Using 2 of the 1.5m QSFP cables per server, connect the ToR switch to the server(s) as follows.

  1. Feed the cables through the cut-outs in the side of the rack first (Fig. 3.55).

_images/ToR_switch_R6525_1.png

Fig. 3.55 ToR switch to server cabling: feed cables through side cut-outs in rack

  1. Connect the cables to the ToR switch and server as given in Table 3.8:

    Table 3.8 Server RNIC port mapping

    Server RNIC port

    ToR switch port

    Cables

    Server # 4 RNIC port 1

    ToR switch port # 1

    QSFP 1.5m

    Server # 4 RNIC port 2

    ToR switch port # 2

    QSFP 1.5m

    Server # 3 RNIC port 1

    ToR switch port # 3

    QSFP 1.5m

    Server # 3 RNIC port 2

    ToR switch port # 4

    QSFP 1.5m

    Server # 2 RNIC port 1

    ToR switch port # 5

    QSFP 1.5m

    Server # 2 RNIC port 2

    ToR switch port # 6

    QSFP 1.5m

    Server # 1 RNIC port 1

    ToR switch port # 7

    QSFP 1.5m

    Server # 1 RNIC port 2

    ToR switch port # 8

    QSFP 1.5m

    ToR switch ports 8 and 7 connect to the bottom server (server 1). Therefore, if you are only using 1 server then only ports 8 and 7 of the ToR switch are used.

    The final ToR switch to server cabling is shown in Fig. 3.56 for a four server version.

    _images/ToR_switch_R6525_2.png

    Fig. 3.56 ToR switch to server cabling: cable connections (4 server version)

3.6.9. Management switch to Dell server(s): iDRAC

Using four of the 1.5m blue RJ45 cables connect the management switch to the Dell server(s) as follows:

  • Ports 9 to 12 on the management switch connect to the iDRAC connector on the server(s) - Fig. 3.57.

  • Port 9 is connected to the bottom server (server 1). If you only have one server then only port 9 on the management switch is used.

_images/mng_switch_R6525_1.png

Fig. 3.57 Management switch to server cabling: iDRAC for 4 server version

3.6.10. Management switch to Dell server(s): network connector

Using four of the 1.5m blue RJ45 cables connect the management switch to the Dell server(s) as follows:

  • Ports 13 to 16 on the management switch connect to the network connector on the server(s) - Fig. 3.58.

  • Port 13 is connected to the bottom server (server 1). If you only have one server then only port 13 on the management switch is used.

_images/mng_switch_R6525_2.png

Fig. 3.58 Management switch to server cabling: network connector(s) for 4 server version

3.6.11. Management switch to Dell server(s): switch management

Using one of the 1.5m blue RJ45 cables connect the management switch to server 1 as follows:

  • Port 8 from the management server is connected to the lowest server (server 1) - the correct port to connect to on the server is shown in Fig. 3.59. This is used for control of the PDUs in the case where server 1 is used as the management server.

_images/mng_switch_R6525_3.jpg

Fig. 3.59 Management switch to server cabling: switch management

3.6.12. Management switch to PDUs

Using the two 2m purple RJ45 cables connect the management switch to the two PDUs as given in Table 3.9:

Table 3.9 Management switch to PDU connections

Ethernet port | Management switch port

Cables

Ethernet port on left side PDU | Management switch port 5

RJ45 2m

Ethernet port on right side PDU

Management switch port 6

RJ45 2m

Allow all the cables going to the server to hang down in the rack as shown in Fig. 3.60. This allows the cables to be pulled slightly if it is necessary to remove them from a server.

_images/mng_switch_PDU.png

Fig. 3.60 Management switch cabling to PDUs

3.7. Power cabling

The designated sockets on the PDUs for servers and IPU-M2000s are shown in Fig. 3.61.

_images/power_cabling_1.png

Fig. 3.61 Designated PDU sockets for server(s) and IPU-M2000s

Fig. 3.62 and Fig. 3.63 show the final power cabling.

_images/power_cabling_2.png

Fig. 3.62 Final power cabling (direct view)

_images/power_cabling_3.png

Fig. 3.63 Final power cabling (angled view)

3.7.1. IPU-M2000 power cabling

Start by cabling IPU-M2000 #1 using 0.5m power cables. Ensure that only three IPU-M2000s are connected to the same bank on the PDU. Red cables to the left PDU and blue cables to the right PDU (as seen looking at the rear of the rack). Table 3.10 defines the power cable lengths for each IPU-M2000:

Table 3.10 Power cables for IPU-M2000s

Cable colour

Cable length

IPU-M2000

Blue

1m

IPU-M2000 #13 to IPU-M2000 #16

Red

1m

IPU-M2000 #13 to IPU-M2000 #16

Blue

0.5m

IPU-M2000 #1 to IPU-M2000 #12

Red

0.5m

IPU-M2000 #1 to IPU-M2000 #12

Fig. 3.64 and Fig. 3.65 show part of the IPU-M2000 power cabling for the red PDU and blue PDU, respectively.

_images/power_cabling_4.png

Fig. 3.64 IPU-M2000 power cabling (red PDU)

_images/power_cabling_5.png

Fig. 3.65 IPU-M2000 power cabling (blue PDU)

3.7.2. Server power cabling: Dell R6525

Using 1m C13 to C14 power cables (selecting the correct coloured cable to match the PDU colour), connect the server(s) to the PDUs as shown in Fig. 3.66.

_images/server_power_wiring.png

Fig. 3.66 Server power cabling

Note

The photo above shows the four-server version. The default reference design has one server (the one in the lowest position).

3.7.3. Switch power cabling

Connect the mains cables to the management switch and ToR switch using C13 to C14 power cables as given in Table 3.11:

Table 3.11 Management switch and ToR cables

Switch

Cable length (red/blue)

Management

1.0m

ToR

1.5m

3.8. Completing the rack

The following steps describe completing the rack: fitting blanking panels and re-installing the doors and side panels.

3.8.1. Blanking panels

Install the supplied APC 1U blanking panels in every unoccupied rack slot at the front of the rack.

_images/blanking_panels.png

Fig. 3.67 1U blanking panels

3.8.2. Front and rear doors

Re-install the front and rear doors. Ensure the earth cables are reconnected to the cable on the rack.

_images/earth_cables.png

Fig. 3.68 Rack earth cable

3.8.3. Side panels

Re-install the top and bottom side panels on each side of the rack.

3.8.4. PDU plugs

The power plugs for the PDUs must not exit the top of the rack, they must stay inside the rack and be secured to the vertical mounting flanges. You need to ensure that there are no cables on the outside of the rack.

3.8.5. Packaging

If you need to re-package the rack then follow the reverse of the steps taken when you unpacked the rack. It is important to ensure that the large clear plastic bag has no tears or holes and is refitted correctly as this bag provides protection from water ingress. It is also important that you ensure that the plastic bag containing the bolts is still affixed to the foam packaging. You also need to fit sufficient straps to secure the packaging – there should be a total of four vertical straps (two per 180 degrees) and two horizontal straps, as shown in Fig. 3.69.

_images/re_package.jpg

Fig. 3.69 Re-packaging the rack