3. Rack assembly

Note the correct orientation of the IPU-M2000, server and switch units in the rack to ensure correct airflow.

The front interface of the IPU-M2000 units (connectivity ports) should be matched with the front door of the rack (cold aisle). The rear interface of the server and switches (power and fans) should be matched with the rear door of the rack (hot aisle).

Note

You can click on any of the images below to view them full size

Fig. 3.1 Completed rack: cold aisle (four-server version)

Note that Fig. 3.1 shows a four-server version of the IPU‑POD₆₄. The default reference design has one server which would be the server in the lowest position, closest to the switches.

Fig. 3.2 Completed rack: hot aisle (four-server version)

Note that Fig. 3.2 shows three blue RJ45 cables in each R6525 server. In the default build, servers 2 to 4 only have two blue RJ45 cables. See Section 3.6.7, Dell R6525 server(s) cabling for more information about server cabling.

Note also that Fig. 3.2 shows a four-server version of the IPU‑POD₆₄. The default reference design has one server which would be the server in the lowest position, closest to the switches.

3.1. Equipment checklist

Table 3.1 Equipment checklist

Description	Quantity (1 server)	Quantity (4 server)
Rack (AR3300SP)	1	1
Blanking panels (APC AR8136BLK)	23 pieces for 42U reference rack (delivered in packs of 10)	20 pieces for 42U reference rack (delivered in packs of 10)
AP8886 PDU	2	2
Hardware mounting kit APC AR8100	1	1
PDU bracket kit APC AR7711	2	2
Graphcore IPU-M2000	16	16
IPU-M2000 slider kits	16	16
Dell R6525 server	1	4
Arista DCS-7010T-48-F switch	1	1
Arista DCS-7060CX-32S-F switch	1	1
2m purple Ethernet	2	2
1.5m blue Ethernet	11	17
1m blue Ethernet	9	9
1m yellow Ethernet	12	12
1.5m yellow Ethernet	4	4
1m red Ethernet	2	2
0.15m red Ethernet	30	30
1m QSFP28	8	8
1.5m QSFP28	10	16
0.3m OSFP	60	60
1m OSFP	4	4
0.5m red 10A C14 to C15	12	12
1m red 10A C14 to C15	4	4
0.5m blue 10A C14 to C15	12	12
1m blue 10A C14 to C15	4	4
1m red C13 to C14	2	5
1.5m red C13 to C14	1	1
1m blue C13 to C14	2	5
1.5m blue C13 to C14	1	1
Velcro	1	1

3.2. Document reproduction

Ideally you should reference this document from a tablet device to allow you to zoom in on photographs. If you reproduce this document on paper it should be done in colour otherwise you will not be able to see cable colours and other notations properly.

3.3. Required tools

You will need the following tools and equipment:

• Pallet truck with wide fork spacing (at least 32cm between forks) and at least 750KG SWL

• Knife suitable for cutting plastic tape

• No 2 Phillips screwdriver

• Torx TX30 screwdriver

• Scissors suitable for cutting Velcro

If you will be re-packaging the rack after building it then you will also need:

• Pallet banding tape, banding clips and banding crimping tool

• Packaging tape

3.4. Preparing the rack

3.4.1. Rail distance

The IPU-M2000 mounting system requires a rail-to-rail distance of 720mm. This document describes the adjustments required for an AR3300SP rack. If using a different rack this rail distance must be observed.

3.4.2. Unpacking the rack

Follow the instructions to remove the outer packaging of the APC AR3300SP rack, ensuring that you safely store these materials for later repackaging. Do not remove the rack from the shock pallet. Remove the white bag (Fig. 3.3) from the rack. This contains screws and cage nuts to be used in the assembly of the components into the rack.

Fig. 3.3 White bag with rack

3.4.3. Removing the side panels and doors

Remove the front and rear doors from the rack. Ensure the earth straps (Fig. 3.4) are disconnected before the doors are removed.

Fig. 3.4 Rack earth straps

Remove the top and bottom side panels. The vertical accessory channels should be positioned at the very front and very rear of the rack. If necessary, move these from their shipping positions (Fig. 3.5).

Fig. 3.5 Top and bottom side panel removal

3.4.4. Removing the vertical accessory channels

Using a Torx TX30 screwdriver, remove two accessory channels from the rack (Fig. 3.6).

Fig. 3.6 Accessory channel removal

3.4.5. Adjusting the rear accessory channels

Set the rear accessory channel to the furthest position in the rack. Tighten up the screws ensuring the teeth engage into the slots in the rail, as shown in Fig. 3.7.

Fig. 3.7 Rear accessory channel (ensure teeth engage in slots)

3.4.6. Adjusting the rear vertical rails

Using a Torx TX30 screwdriver, make both rear vertical rack rails loose and freely movable.

Position the rear vertical rack rails such that there is 20mm of distance between the rear face of the vertical rack rail and the racks rear frame. This should result in a square symbol being visible through the alignment window at the top and bottom of the rail, as shown in Fig. 3.8.

Fig. 3.8 Rear vertical rails

Secure the rail into position by moving the TX30 screws back upwards such that the teeth engage with both the supporting rails. This must be done at the top and bottom of the bracket.

3.4.7. Adjusting the front vertical rails

Using a Torx TX30 screwdriver, make both front vertical rack rails loose and freely movable. Install the accessory channels in the front of the rack (one on the left hand side, one on the right hand side) at the frontmost position possible, moving the TX30 screws back upwards such that the teeth engage with both the supporting rails. This must be done at the top and bottom of the bracket.

Note

To ensure the clips on the accessory channels align with the channel in the rack, lift the accessory channels through the cut-out in the top of the rack and then drop them down onto the channels.

Move the vertical rack rails tight against the vertical cable organisers such that only a single diamond symbol is visible through the alignment window at the top and bottom of the rail, see Fig. 3.9.

Fig. 3.9 Front vertical rail alignment

Secure the rail into position by moving the TX30 screws back upwards such that the teeth engage with both the supporting rails. This must be done at the top and bottom of the bracket.

3.4.8. Installing the rack rails

Fig. 3.10 M2000 rack rail kit: unboxed

The IPU-M2000 rail kit comprises two mated inner and outer rack rails and an accessory bag containing screws. The inner rail affixes to the body of the IPU-M2000 and the outer rail affixes to the vertical rack rails in the server cabinet. Firstly, separate the mated inner and outer rails:

1. Fully extend the rails by pulling on the end which has the captive thumb screw attached (Fig. 3.11):

Fig. 3.11 M2000 rack rail kit: extended rails

2. Whilst pulling on the thumb screw end of the rails, push the white plastic release tab towards the thumb screw end (Fig. 3.12):

Fig. 3.12 M2000 rack rail kit: white release tab

3. The inner and outer rails will now separate (Fig. 3.13):

Fig. 3.13 M2000 rack rail kit: inner and outer rails separated

Mate the inner rails (the thinner of the two separated rails which has a captive thumb screw at one end) to the body of the IPU-M2000. Note that the inner rails are mirrored and are not handed. As such, the procedure for inner rail fixing is the same for the left and right hand inner rails.

The inner rail should be oriented such that the captive thumb screw end is at the end of the IPU-M2000 containing the network ports.

To affix the inner rail to the body of the IPU-M2000:

1. Offer up the inner rail to the side of the IPU-M2000 and ensure that all fixing pins are sitting within the enlarged opening of the retention channel (Fig. 3.14):

Fig. 3.14 M2000 rack rail kit: fixing pins

2. Push the inner rail towards the end of the IPU-M2000 containing the network ports, you should hear a click as the latching mechanism locks behind the head of a fixing pin (Fig. 3.15):

Fig. 3.15 M2000 rack rail kit: fixing pins locked

3. Ensure all fixing pins are correctly engaged with their respective retention channels.

4. Locate the four flat head fixing screws from the rack rail accessory bag (Fig. 3.16):

Fig. 3.16 M2000 rack rail kit: flat head fixing screws

5. Using the above screws, affix the inner rail to the body of the M2000 (Fig. 3.17):

Fig. 3.17 M2000 rack rail kit: attaching inner rail to M2000

The inner rails are now securely affixed to the IPU-M2000 body. Place the outer rails to one side for later use (Fig. 3.18):

Fig. 3.18 M2000 rack rail kit: outer rails

3.4.9. Installing PDU brackets

Install four cage nuts on the outside at the top and bottom to both of the accessory channels as shown below (Fig. 3.19 and Fig. 3.20).

Fig. 3.19 Top PDU bracket cage nuts

Fig. 3.20 Bottom PDU bracket cage nuts

Screw the PDU support brackets to the inside of the cabinet. The PDU brackets should be installed at the rear of the rack: one bracket on top with 9cm distance from the top of rack and one bracket on the bottom with 12cm distance from the bottom of rack. Fig. 3.21 illustrates this. Follow the PDU bracket installation instructions.

Fig. 3.21 PDU support bracket

3.5. Installing the equipment

The following sections describe the installation of the IPU-M2000s, PDUs, servers, ToR and management switches into the rack.

3.5.1. Installing the IPU-M2000s

Earlier in the guide we affixed the inner rack rails to the IPU-M2000 body. The next step in installing the IPU-M2000s is to install the outer rack rails into the rack.

It is possible to identify the front and rear of the outer rail by finding the large metal latching mechanism. This is to be located at the rear of the rack. The outer rail is also embossed with the text “FRONT” at the front end of the rail.

The outer rail large metal latch end is to be installed at the rear of the rack, as shown in Fig. 3.22.

Fig. 3.22 Attachment point for IPU-M2000 outer rail

For each rack U in U1 through U16 (inclusive), perform the steps below with both the left hand and right hand outer rack rails:

1. Pull on each end of the outer rail to adjust the rail length to suite your rack

2. Locate the front end of the outer rail and hold it behind the square holes in the vertical rack rail for your installation U. Pull the outer rail towards the vertical rack rail and the latching mechanism will click and hold the outer rail in place (Fig. 3.23):

Fig. 3.23 IPU-M2000 outer rail latching mechanism

3. Locate the rear end of the outer rail and slightly open the large metal latch, then press the upper and lower locating pins into the square holes in the vertical rack rail. Release the large metal latch and the outer rail will now be secured to the vertical rack rail (Fig. 3.24):

Fig. 3.24 IPU-M2000 outer rail secured to vertical rack rail

4. Included in the rack rail accessory bag are two screws and two washers. One screw with one washer should be screwed through the vertical rack rail and into the outer rack rail threaded hole. The washer should be used in such a way that the washer sits flush with the head of the screw –like a cup. The screws and how they fit are shown in Fig. 3.25 and Fig. 3.26.

This should be repeated for both outer rack rails.

Fig. 3.25 IPU-M2000 rack rail kit: outer rail screws and washers

Fig. 3.26 IPU-M2000 rack rail kit: outer rail screws and washers attached

Next, to install an IPU-M2000 unit into the rack rails, work your way through the following steps in order:

1. Pull the sliding rail located within the outer rack rail completely forward such that it locks into the fully extended position (Fig. 3.27):

Fig. 3.27 IPU-M2000 rack rail kit: sliders fully extended

2. Place the IPU-M2000 onto an appropriate server lift and adjust the height such that it is suitable for the sliders (Fig. 3.28). If a lift is not available, this is a two person operation.

Fig. 3.28 Server lift for IPU-M2000

3. Slide the protruding inner rails into the receiving channel of the extended outer rails (Fig. 3.29):

Fig. 3.29 Slide IPU-M2000 inner rails into outer rails

4. Whilst the server lift is supporting the full weight of the IPU-M2000, slide the IPU-M2000 into the extended outer rails until you feel both sides engage a stopping mechanism (Fig. 3.30).

5. Simultaneously pull on the blue tabs for the release mechanism at each side of the IPU-M2000 and then push the IPU-M2000 unit fully into the rack (Fig. 3.31):

Fig. 3.30 Sliding IPU-M2000 into rack

Fig. 3.31 Blue tab release mechanism

6. Screw the captive thumb screw into the inner rack rail (Fig. 3.32):

Fig. 3.32 Screwing captive thumb screw into inner rack rail

3.5.2. Installing the management switch

Insert 2 cage nuts, inside the rack, on either side of the rack into the top and bottom positions of location 17.

Place the management switch on top of the last IPU-M2000 and screw it into position using four M6 screws (Fig. 3.33).

Fig. 3.33 Installing management switch with four M6 screws

3.5.3. Installing the ToR switch

Fit the sliders for the ToR switch into position 18 on the rack ensuring both ends of the slider are pushed firmly into the mounting slots on the rack rail (Fig. 3.34).

Fig. 3.34 Sliders for installing ToR switch

Insert the ToR switch ensuring the wheel on the switch is located in the groove on the slider.

3.5.4. Installing the PDUs

Install the two PDUs vertically at the rear of the rack, one on the left side and one on the right side. Push the mains cable through the roof of the rack and then clip the PDUs onto the PDU bracket as shown in Fig. 3.35:

Fig. 3.35 PDU installation

3.5.5. Installing the Dell R6525 server(s)

Remove and discard the cable management arm brackets from the rear of each tool-less sliding rail (Fig. 3.36).

Fig. 3.36 Cable management arm brackets - to be removed

Install the tool-less sliding rail kit(s). The reference design has a single server installed in rack slot #19; in four server configurations they are installed in rack slots #19-#22.

Pull out the rail and fit the server to the rail ensuring the T pins on the side of the server locate in the slots on the rail. Ensure that the power supplies on the server face the rear of the rack (Fig. 3.37).

Fig. 3.37 Sliding rail kit for server installation

Note

Use an appropriate server lift or have two people installing the servers to ensure correct fitting

Push the server gently from the front to lock it into the slides then press the tab on the side of the slides and push the server fully home in the rack. Repeat the above process for each server if installing multiple servers.

Remove the Velcro tape from the light pipes on the rear of the servers.

Remove the small plastic tab on the left front side of the server bezel and clip the bezel in place on the front of the server ensuring the connection pins on the right hand side of the bezel line up with the connector on the server, as shown in Fig. 3.38.

Fig. 3.38 Bezel placement on front of server

3.6. Cabling the rack

The following sections detail the cabling of the rack and the dressing of the cables within the rack. For reference, the IPU-M2000s and server(s) are numbered as shown in Fig. 3.39:

Fig. 3.39 Server and IPU-M2000 numbering

3.6.1. IPU-M2000 to IPU-M2000 IPU-Link connectivity (OSFP)

There are eight OSFP IPU-Link ports on the right side of each IPU-M2000. Using the supplied 60x 0.3M OSFP cables, and starting at IPU-M2000 #1 (bottom-most IPU-M2000 in the rack), link the top row of four ports (5-8) to the bottom row of four ports (1-4) in the IPU-M2000 that is installed directly above (see Fig. 3.40 and Table 3.2 below). This applies to all IPU-M2000s except for the top row (5-8) of the top-most IPU-M2000 (#16) and the bottom row (1-4) of the bottom-most IPU-M2000 (#1), which are connected together using the 1m OSFP cables.

Fig. 3.40 IPU-M2000 IPU-Link port numbering and IPU-Link cables

Before attempting to install the OSFP cables, it is beneficial to manipulate the cable to form a tight loop (Fig. 3.41). During manufacture and shipping, the cables can form quite a stiff shape, so manipulating the cables before installing them reduces stresses on the socket during install.

Fig. 3.41 Manipulating IPU-Link cables

After installing a cable, pull gently on the black cable (Fig. 3.42) to ensure the plugs are firm in the sockets on the IPU-M2000.

Fig. 3.42 Check IPU-M2000 to IPU-M2000 IPU-Link cabling

Note

The white tab is on the top of the cable when inserted into the IPU-M2000.

Table 3.2 shows the IPU-M2000 OSFP port mapping.

Table 3.2 OSFP IPU-M2000 to IPU-M2000 IPU-Links

From	To	Cables
IPU-M2000 #15 IPU-Link ports 5,6,7,8	IPU-M2000 #16 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #14 IPU-Link ports 5,6,7,8	IPU-M2000 #15 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #13 IPU-Link ports 5,6,7,8	IPU-M2000 #14 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #12 IPU-Link ports 5,6,7,8	IPU-M2000 #13 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #11 IPU-Link ports 5,6,7,8	IPU-M2000 #12 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #10 IPU-Link ports 5,6,7,8	IPU-M2000 #11 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #9 IPU-Link ports 5,6,7,8	IPU-M2000 #10 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #8 IPU-Link ports 5,6,7,8	IPU-M2000 #9 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #7 IPU-Link ports 5,6,7,8	IPU-M2000 #8 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #6 IPU-Link ports 5,6,7,8	IPU-M2000 #7 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #5 IPU-Link ports 5,6,7,8	IPU-M2000 #6 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #4 IPU-Link ports 5,6,7,8	IPU-M2000 #5 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #3 IPU-Link ports 5,6,7,8	IPU-M2000 #4 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #2 IPU-Link ports 5,6,7,8	IPU-M2000 #3 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #1 IPU-Link ports 5,6,7,8	IPU-M2000 #2 IPU-Link ports 1,2,3,4	OSFP 0.3m
IPU-M2000 #1 IPU-Link ports 1,2,3,4	IPU-M2000 #16 IPU-Link ports 5,6,7,8	OSFP 1m

Fig. 3.43 shows the final IPU-M2000 to IPU-M2000 IPU-Link cabling.

Fig. 3.43 Final IPU-M2000 to IPU-M2000 IPU-Link cabling

3.6.2. IPU-M2000 to IPU-M2000 Sync-Link cabling

Using the 0.15m red Ethernet RJ45 cable, wire the 16 IPU-M2000s as follows:

Starting from IPU-M2000 #1 at the bottom of the rack, insert one side of a cable into port 6 and one side of another cable into port 7.

Insert the other side of the cable from port 6 of IPU-M2000 #1 into port 2 of IPU-M2000 #2, and the other side of the cable from port 7 of IPU-M2000 #1 into port 3 of IPU-M2000 #2.

Fig. 3.44 IPU-M2000 Sync-Link ports

Continue the cabling for all IPU-M2000s. When completed the top row (6,7) of the top-most IPU-M2000 (#16) connectors should be connected to the bottom row (2,3) of the bottom-most IPU-M2000 (#1) connectors with 1m red Ethernet RJ45 cables.

Fig. 3.45 shows the final IPU-M2000 in-rack Sync-Link cabling.

Fig. 3.45 Final IPU-M2000 in-rack Sync-Link cabling

Table 3.3 gives the IPU-M2000 Sync-Link port mapping.

Table 3.3 IPU-M2000 to IPU-M2000 Sync-Links

From	To	Cables
IPU-M2000 #15 Sync-Link ports 6,7	IPU-M2000 #16 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #14 Sync-Link ports 6,7	IPU-M2000 #15 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #13 Sync-Link ports 6,7	IPU-M2000 #14 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #12 Sync-Link ports 6,7	IPU-M2000 #13 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #11 Sync-Link ports 6,7	IPU-M2000 #12 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #10 Sync-Link ports 6,7	IPU-M2000 #11 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #9 Sync-Link ports 6,7	IPU-M2000 #10 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #8 Sync-Link ports 6,7	IPU-M2000 #9 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #7 Sync-Link ports 6,7	IPU-M2000 #8 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #6 Sync-Link ports 6,7	IPU-M2000 #7 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #5 Sync-Link ports 6,7	IPU-M2000 #6 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #4 Sync-Link ports 6,7	IPU-M2000 #5 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #3 Sync-Link ports 6,7	IPU-M2000 #4 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #2 Sync-Link ports 6,7	IPU-M2000 #3 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #1 Sync-Link ports 6,7	IPU-M2000 #2 Sync-Link ports 2,3	RJ45 0.15 red
IPU-M2000 #1 Sync-Link ports 2,3	IPU-M2000 #16 Sync-Link ports 6,7	RJ45 1m red

3.6.3. IPU-M2000 to management switch cabling (RJ45)

There are two Ethernet ports in the middle of each IPU-M2000 (see Fig. 3.46). One of them is a BMC + IPU-Gateway port (upper port) and the other is a BMC port (lower port).

Fig. 3.46 IPU-M2000 management switch ports

These are connected from each IPU-M2000 to the management switch (Fig. 3.47) with RJ45 cables. The cables required are:

• 12x RJ45/Yellow 1.0m and 4x RJ45/Yellow 1.5m (BMC)

• 8x RJ45/Blue 1.0m and 8x RJ45/Blue 1.5m (BMC + IPU-Gateway)

This cabling is described in detail in Section 3.6.4, Management switch: BMC cabling and Section 3.6.5, Management switch: BMC + IPU-Gateway cabling.

Fig. 3.47 Management switch ports

Table 3.4 shows the management switch port allocation.

Table 3.4 Management switch port allocation

Ports	Allocation
1	Customer datacentre uplink for BMC + IPU-Gateway
2	Customer datacentre uplink for BMC-only (future update)
3	Customer management interface
5,6	PDU management
7	Server 1 PDU management switch
8	ToR to management switch
9-12	1GbE server management (iDRAC)
13-16	Server data ports
17-32	IPU-M2000 BMC + IPU-Gateway combined management
33-48	IPU-M2000 BMC-only management (future update)

3.6.4. Management switch: BMC cabling

Start cabling using a 1.0m yellow cable and insert one end into port 48 of the management switch.

Run the cable through the loop in the OSFP connector as shown in the left hand picture in Fig. 3.48 and connect it to the BMC port on IPU-M2000 #16 (top IPU-M2000 in the rack).

Fig. 3.48 Management switch BMC cabling

Repeat the process for ports 47 to 37 using 1.0m yellow cables and ports 33 to 36 using 1.5m cables.

When all cables have been connected, dress the loom down the side of the cabinet and secure the bundle with a Velcro strip as shown in the right hand picture in Fig. 3.48.

The port mapping between the management switch and the IPU-M2000 BMC sockets is given in Table 3.5.

Table 3.5 IPU-M2000 management BMC-only mapping

From	To	Cables
IPU-M2000 #16 BMC port	Management switch port 48	RJ45 1.0m yellow
IPU-M2000 #15 BMC port	Management switch port 47	RJ45 1.0m yellow
IPU-M2000 #14 BMC port	Management switch port 46	RJ45 1.0m yellow
IPU-M2000 #13 BMC port	Management switch port 45	RJ45 1.0m yellow
IPU-M2000 #12 BMC port	Management switch port 44	RJ45 1.0m yellow
IPU-M2000 #11 BMC port	Management switch port 43	RJ45 1.0m yellow
IPU-M2000 #10 BMC port	Management switch port 42	RJ45 1.0m yellow
IPU-M2000 #9 BMC port	Management switch port 41	RJ45 1.0m yellow
IPU-M2000 #8 BMC port	Management switch port 40	RJ45 1.0m yellow
IPU-M2000 #7 BMC port	Management switch port 39	RJ45 1.0m yellow
IPU-M2000 #6 BMC port	Management switch port 38	RJ45 1.0m yellow
IPU-M2000 #5 BMC port	Management switch port 37	RJ45 1.0m yellow
IPU-M2000 #4 BMC port	Management switch port 36	RJ45 1.5m yellow
IPU-M2000 #3 BMC port	Management switch port 35	RJ45 1.5m yellow
IPU-M2000 #2 BMC port	Management switch port 34	RJ45 1.5m yellow
IPU-M2000 #1 BMC port	Management switch port 33	RJ45 1.5m yellow

3.6.5. Management switch: BMC + IPU-Gateway cabling

Start cabling using a 1.0m blue RJ45 cable and insert one end into port 32 of the management switch.

Run the cable through the loop in the OSFP connector as shown in the left hand picture in Fig. 3.49 and connect it to the BMC+IPU-Gateway port on IPU-M2000 #16.

Fig. 3.49 Management switch BMC + IPU-Gateway cabling

Repeat the process for ports 31 to 25 using 1.0m blue cables, and ports 24 to 17 using 1.5m blue cables. When all cables have been connected, dress the loom down the side of the cabinet and secure the bundle with a Velcro strip, as shown in the right hand picture in Fig. 3.49. Using a 1.0m blue cable, connect port 7 of the management switch to the top RJ45 connector on the ToR Switch (Fig. 3.50).

Fig. 3.50 ToR switch: top RJ45 connector

With the IPU-M2000s numbered from bottom to top, the BMC + IPU-Gateway ports should be connected to the management switch as given in Table 3.6.

Table 3.6 IPU-M2000 management BMC + IPU-Gateway port mapping

From	To	Cables
IPU-M2000 #16 BMC + IPU-Gateway port	Management switch port 32	RJ45 1.0m blue
IPU-M2000 #15 BMC + IPU-Gateway port	Management switch port 31	RJ45 1.0m blue
IPU-M2000 #14 BMC + IPU-Gateway port	Management switch port 30	RJ45 1.0m blue
IPU-M2000 #13 BMC + IPU-Gateway port	Management switch port 29	RJ45 1.0m blue
IPU-M2000 #12 BMC + IPU-Gateway port	Management switch port 28	RJ45 1.0m blue
IPU-M2000 #11 BMC + IPU-Gateway port	Management switch port 27	RJ45 1.0m blue
IPU-M2000 #10 BMC + IPU-Gateway port	Management switch port 26	RJ45 1.0m blue
IPU-M2000 #9 BMC + IPU-Gateway port	Management switch port 25	RJ45 1.0m blue
IPU-M2000 #8 BMC + IPU-Gateway port	Management switch port 24	RJ45 1.5m blue
IPU-M2000 #7 BMC + IPU-Gateway port	Management switch port 23	RJ45 1.5m blue
IPU-M2000 #6 BMC + IPU-Gateway port	Management switch port 22	RJ45 1.5m blue
IPU-M2000 #5 BMC + IPU-Gateway port	Management switch port 21	RJ45 1.5m blue
IPU-M2000 #4 BMC + IPU-Gateway port	Management switch port 20	RJ45 1.5m blue
IPU-M2000 #3 BMC + IPU-Gateway port	Management switch port 19	RJ45 1.5m blue
IPU-M2000 #2 BMC + IPU-Gateway port	Management switch port 18	RJ45 1.5m blue
IPU-M2000 #1 BMC + IPU-Gateway port	Management switch port 17	RJ45 1.5m blue

3.6.6. IPU-M2000 to ToR switch cabling (QSFP)

The next step is to connect the IPU-M2000s to the ToR switch. There are two RNIC ports on the left side of each IPU-M2000, as shown in Fig. 3.51. Only one of them should be connected from each IPU-M2000 to the ToR switch with either the 1m or 1.5m QSFP cables supplied.

Fig. 3.51 IPU-M2000 RNIC ports (QSFP)

In order to manage the cables, the QSFP cables are divided into two different lengths:

• 8x QSFP 1.0m from IPU-M2000 #9-16 to ToR switch ports 16 to 9

• 8x QSFP1.5m from IPU-M2000 #1-8 to ToR switch ports 17 to 24

Start from port 9 on the ToR switch and connect the cable to port 2 of IPU-M2000 #16.

Continue cabling from ports 10 to 24 on the ToR switch to port 2 on IPU-M2000 #15 to IPU-M2000 #1 respectively.

Dress the cables down the side of the cabinet. A set of four cables can be looped into cut-outs on the side of the cabinet. Tie the bundle together at the top with a Velcro strip as shown in Fig. 3.52.

Fig. 3.52 IPU-M2000 to ToR switch cabling

The IPU-M2000s should be connected to the ToR switch as given in Table 3.7, note that the IPU-M2000s are numbered from bottom to top #1 to #16.

Table 3.7 IPU-M2000 RNIC port mapping

IPU-M2000 port	ToR switch port	Cables
IPU-M2000 #16 port 2	ToR switch port 9	QSFP 1.0m
IPU-M2000 #15 port 2	ToR switch port 10	QSFP 1.0m
IPU-M2000 #14 port 2	ToR switch port 11	QSFP 1.0m
IPU-M2000 #13 port 2	ToR switch port 12	QSFP 1.0m
IPU-M2000 #12 port 2	ToR switch port 13	QSFP 1.0m
IPU-M2000 #11 port 2	ToR switch port 14	QSFP 1.0m
IPU-M2000 #10 port 2	ToR switch port 15	QSFP 1.0m
IPU-M2000 #9 port 2	ToR switch port 16	QSFP 1.0m
IPU-M2000 #8 port 2	ToR switch port 17	QSFP 1.5m
IPU-M2000 #7 port 2	ToR switch port 18	QSFP 1.5m
IPU-M2000 #6 port 2	ToR switch port 19	QSFP 1.5m
IPU-M2000 #5 port 2	ToR switch port 20	QSFP 1.5m
IPU-M2000 #4 port 2	ToR switch port 21	QSFP 1.5m
IPU-M2000 #3 port 2	ToR switch port 22	QSFP 1.5m
IPU-M2000 #2 port 2	ToR switch port 23	QSFP 1.5m
IPU-M2000 #1 port 2	ToR switch port 24	QSFP 1.5m

Fig. 3.53 shows the final IPU-M2000 to ToR switch cabling.

Fig. 3.53 Final IPU-M2000 to ToR switch cabling

3.6.7. Dell R6525 server(s) cabling

All cables should be routed from the rear of the server to the right-hand side when viewed from the rear (see Fig. 3.54), then along the side of the rack using the cable management holes in the vertical rack rails.

Details on how to install these cables is given in the followings ections - Section 3.6.8, ToR switch to Dell server(s), Section 3.6.9, Management switch to Dell server(s): iDRAC and Section 3.6.10, Management switch to Dell server(s): network connector.

Fig. 3.54 Server cabling (4 server version)

Note that Fig. 3.54 shows the four-server version. The default build has one server (the one in the lowest position).

Note

Picture to be updated: Fig. 3.54 shows three blue RJ45 cables in each R6525 server. In the default build, servers 2 to 4 only have two blue RJ45 cables. The cables removed in the default build are marked with a red cross. The additional cable in server 1 moves to the port on the right, as indicated by the red arrow.

3.6.8. ToR switch to Dell server(s)

Using two of the 1.5m QSFP cables per server, connect the ToR switch to the server(s) as follows.

1. Feed the cables through the cut-outs in the side of the rack first (Fig. 3.55).

Fig. 3.55 ToR switch to server cabling: feed cables through side cut-outs in rack

2. Connect the cables to the ToR switch and server as given in Table 3.8:

   Table 3.8 Server RNIC port mapping

   Server RNIC port	ToR switch port	Cables
   Server # 4 RNIC port 1	ToR switch port # 1	QSFP 1.5m
   Server # 4 RNIC port 2	ToR switch port # 2	QSFP 1.5m
   Server # 3 RNIC port 1	ToR switch port # 3	QSFP 1.5m
   Server # 3 RNIC port 2	ToR switch port # 4	QSFP 1.5m
   Server # 2 RNIC port 1	ToR switch port # 5	QSFP 1.5m
   Server # 2 RNIC port 2	ToR switch port # 6	QSFP 1.5m
   Server # 1 RNIC port 1	ToR switch port # 7	QSFP 1.5m
   Server # 1 RNIC port 2	ToR switch port # 8	QSFP 1.5m

   ToR switch ports 8 and 7 connect to the bottom server (server 1). Therefore, if you are only using 1 server then only ports 8 and 7 of the ToR switch are used.

   The final ToR switch to server cabling is shown in Fig. 3.56 for a four server version.

   

   Fig. 3.56 ToR switch to server cabling: cable connections (4 server version)

Note

The figures in this section only show one black 1.5m QSFP cable per server. In a normal installation there would be two black 1.5m QSFP cables per server.

3.6.9. Management switch to Dell server(s): iDRAC

Using four of the 1.5m blue RJ45 cables connect the management switch to the Dell server(s) as follows:

• Ports 9 to 12 on the management switch connect to the iDRAC connector on the server(s) - Fig. 3.57.

• Port 9 is connected to the bottom server (server 1). If you only have one server then only port 9 on the management switch is used.

Fig. 3.57 Management switch to server cabling: iDRAC for 4 server version

3.6.10. Management switch to Dell server(s): network connector

Using four of the 1.5m blue RJ45 cables connect the management switch to the Dell server(s) as follows:

• Ports 13 to 16 on the management switch connect to the network connector on the server(s) - Fig. 3.58.

• Port 13 is connected to the bottom server (server 1). If you only have one server then only port 13 on the management switch is used.

Fig. 3.58 Management switch to server cabling: network connector(s) for 4 server version

3.6.11. Management switch to Dell server(s): switch management

Using one of the 1.5m blue RJ45 cables connect the management switch to server 1 as follows:

• Port 8 from the management server is connected to the lowest server (server 1) - the correct port to connect to on the server is shown in Fig. 3.59. This is used for control of the PDUs in the case where server 1 is used as the management server.

Fig. 3.59 Management switch to server cabling: switch management

3.6.12. Management switch to PDUs

Using the two 2m purple RJ45 cables connect the management switch to the two PDUs as given in Table 3.9:

Table 3.9 Management switch to PDU connections

Ethernet port | Management switch port			Cables
Ethernet port on left side PDU | Management switch port 5			RJ45 2m
Ethernet port on right side PDU	Management switch port 6		RJ45 2m

Allow all the cables going to the server to hang down in the rack as shown in Fig. 3.60. This allows the cables to be pulled slightly if it is necessary to remove them from a server.

Fig. 3.60 Management switch cabling to PDUs

3.7. Power cabling

The designated sockets on the PDUs for servers and IPU-M2000s are shown in Fig. 3.61.

Fig. 3.61 Designated PDU sockets for server(s) and IPU-M2000s

Fig. 3.62 and Fig. 3.63 show the final power cabling.

Fig. 3.62 Final power cabling (direct view)

Fig. 3.63 Final power cabling (angled view)

3.7.1. IPU-M2000 power cabling

Start by cabling IPU-M2000 #1 using 0.5m power cables. Ensure that only three IPU-M2000s are connected to the same bank on the PDU. Red cables to the left PDU and blue cables to the right PDU (as seen looking at the rear of the rack). Table 3.10 defines the power cable lengths for each IPU-M2000:

Table 3.10 Power cables for IPU-M2000s

Cable colour	Cable length	IPU-M2000
Blue	1m	IPU-M2000 #13 to IPU-M2000 #16
Red	1m	IPU-M2000 #13 to IPU-M2000 #16
Blue	0.5m	IPU-M2000 #1 to IPU-M2000 #12
Red	0.5m	IPU-M2000 #1 to IPU-M2000 #12

Fig. 3.64 and Fig. 3.65 show part of the IPU-M2000 power cabling for the red PDU and blue PDU, respectively.

Fig. 3.64 IPU-M2000 power cabling (red PDU)

Fig. 3.65 IPU-M2000 power cabling (blue PDU)

3.7.2. Server power cabling: Dell R6525

Using 1m C13 to C14 power cables (selecting the correct coloured cable to match the PDU colour), connect the server(s) to the PDUs as shown in Fig. 3.66.

Fig. 3.66 Server power cabling

Note

The photo above shows the four-server version. The default reference design has one server (the one in the lowest position).

3.7.3. Switch power cabling

Connect the mains cables to the management switch and ToR switch using C13 to C14 power cables as given in Table 3.11:

Table 3.11 Management switch and ToR cables

Switch	Cable length (red/blue)
Management	1.0m
ToR	1.5m

3.8. Completing the rack

The following steps describe completing the rack: fitting blanking panels and re-installing the doors and side panels.

3.8.1. Blanking panels

Install the supplied APC 1U blanking panels in every unoccupied rack slot at the front of the rack.

Fig. 3.67 1U blanking panels

3.8.2. Front and rear doors

Re-install the front and rear doors. Ensure the earth cables are reconnected to the cable on the rack.

Fig. 3.68 Rack earth cable

3.8.3. Side panels

Re-install the top and bottom side panels on each side of the rack.

3.8.4. PDU plugs

The power plugs for the PDUs must not exit the top of the rack, they must stay inside the rack and be secured to the vertical mounting flanges. You need to ensure that there are no cables on the outside of the rack.

3.8.5. Packaging

If you need to re-package the rack then follow the reverse of the steps taken when you unpacked the rack. It is important to ensure that the large clear plastic bag has no tears or holes and is refitted correctly as this bag provides protection from water ingress. It is also important that you ensure that the plastic bag containing the bolts is still affixed to the foam packaging. You also need to fit sufficient straps to secure the packaging – there should be a total of four vertical straps (two per 180 degrees) and two horizontal straps, as shown in Fig. 3.69.

Fig. 3.69 Re-packaging the rack