4. Server and switch configuration

This chapter describes how to configure the server(s), and the network switches in an IPU‑POD₆₄.

Note

Ansible playbooks for the server installation and files for switch configuration can be provided on request: please contact Graphcore support. These templates may need to be adapted to match any site-specific differences.

4.1. Server configuration

The server and POD configuration consists of the following main steps:

Provide appropriate server hardware, according to the approved server list available from Graphcore. This should include supported storage and memory configurations.
Configure server BIOS appropriately
Install the appropriate OS and packages
Setup default user accounts and permissions
Setup a DHCP server
Setup a Syslog service
Setup an NTP service
Final configuration: security and default folders

More details of these steps are provided in the following sections.

4.1.1. Hardware recommendations

The IPU‑POD₆₄ reference design uses a single PowerEdge R6525 server but up to four servers can be connected. Contact Graphcore sales for details of other supported server types. This document describes the default server (PowerEdge R6525) installation only. Other servers may have different installation requirements.

The recommended configuration of the Dell R6525 is as follows:

Dell R6525 containing dual AMD EPYC 7742 processors
16x 32GbE RDIMM PC4-25600 ECC registered dual-rank X4 1.2v
2x 480GbE SSD-SATA 6Gbps 2.5 inch hot-swap
7x 1TB NVME SSD PCIe 4x 3.1
Dual port Gigabit BASE-T PCIe
Single/dual port Mellanox ConnectX-5 EN 100Gb/s Ethernet

4.1.2. Storage configuration recommendations

The recommendation is to have two types of server storage: SSD-SATA for the operating system and NVME SSD for data storage.

Operating system:

2x 480GbE SSD-SATA units as a RAID 1 via hardware controller
Partitioned to use ext4 file system

Data storage:

7x 1TB NVME SSD units as a logical RAID 6 managed with MDADM
Partitioned to use xfs file system

4.1.3. Memory configuration

The DIMMs should be installed in a fully symmetric configuration, as recommended by Dell for maximum performance. The recommended configuration has 8 DIMMs per processor, as shown below.

_images/mem_config.png — Fig. 4.1 DIMM memory configuration

4.1.4. BIOS configuration

Various BIOS settings can impact the performance of the system. The recommended settings are shown in the table below.

Table 4.1 BIOS settings
Setting	BIOS 1.2.11	BIOS 1.4.8 or later
LogicalProc	Enabled	Enabled
ProcVirtualization	Enabled	Enabled
IommuSupport	Disabled	See below
L1StreamHwPrefetcher	Enabled	Enabled
L2StreamHwPrefetcher	Enabled	Enabled
MadtCoreEnumeration	Linear	Linear
NumaNodesPerSocket	4	4
CcxAsNumaDomain	Disabled	Disabled
CpuMinSevAsid	1	1
ProcCcds	All	All
CcdCores	All	All
EmbSata	AhciMode	AhciMode

BIOS 1.4.8 or later does not configure IOMMU directly. Instead, you need to add iommu=off to the kernel boot params in /etc/default/grub and then run update-grub.

4.1.5. Operating system recommendations

Please contact your Graphcore representative or use the support portal for information about operating system support.

This document describes the following operating systems:

Ubuntu 18.04.4 LTS (bionic)
CentOS 7.2

4.1.6. Ubuntu 18.04 packages

In order to have a stable system where IPU related software can run, several packages need to be installed via the Aptitude package manager tool. Other packages may be required for local requirements.

Table 4.2 Required Ubuntu packages
autoconf	golang-go	minicom	rsyslog
automake	htop	moreutils	screen
bc	ibverbs-utils	net-tools	sshpass
build-essential	ipmitool	netcat	subversion
ccache	jq	ntp	swig
clang	kcachegrind	openjdk-8-jdk	sysfsutils
cmake	libaio-dev	parallel	tar
direnv	libboost-all-dev	pciutils	tmux
dkms	libeigen3-dev	perl	u-boot-tools
emacs	libjson-c-dev	php-cli	unzip
ethtool	libjson-c-doc	php-curl	valgrind
exuberant-ctags	libpci-dev	policykit-1	vim
flex	libpixman-1-dev	protobuf-compiler	virtualenv
g++	libprotobuf-dev	python3	wdiff
gawk	libtool	python3-dev	wget
gcc	lldpad	python3-pip	zip
gdb	lldpd	qtcreator
git	m4	rdma-core

4.1.7. CentOS 7 packages

In order to have a stable system where IPU related software can run, several packages need to be installed via the yum configuration manager. Other packages may be required for local requirements.

Table 4.3 Required CentOS 7 packages
bc	htop	moreutils	rdma-core
boost-devel	ipmitool	nano	rsyslog
centos-release-scl	java-latest-jdk	nc	screen
clang	jq	net-tools	snapd
cmake	json-c-devel	ntp	sshpass
containerd.io	json-c-doc	parallel	sysfsutils
devtoolset-7	libaio-devel	pciutils-devel	tmux
dhcp	libibverbs-utils	php-cli	vim
dkms	libuser	protobuf-devel	wdiff
eigen3	lldpad	python3-pip	wget
emacs	lldpd	python36
golang-go	minicom	qt5-qbase

4.1.8. Python packages

Several python packages are also required for both operating system installations. They can be installed using the pip installation tool.

Table 4.4 Required python packages
autograd	numpy	pytest	setuptools
boto3	paramiko	pyyaml	virtualenv
jstyleson	pep8	recommonmark	wheel
lxml	pexpect	requests	yapf
mock	pylint	scp

4.1.9. User accounts and groups

The following accounts are required as part of the default server configuration:

Table 4.5 Account creation
Account	Description
root	A root user account secured with a password is recommended.
itadmin	An admin account secured with a password is recommended. Home directory located at `/home/itadmin`, using Bash shell.
ipuuser	An account dedicated to IPU software and IPU-POD management software is mandatory. Home directory located at `/home/ipuuser`, using Bash shell.
poplaruser	An account dedicated to Poplar software is mandatory. Home directory located at `/home/poplaruser`, using Bash shell.

The following table gives the default usernames provided on the IPU‑POD₆₄.

Table 4.6 Default usernames
Log in to	Username
IPU-M2000 BMC OS	root
IPU-M2000 GW OS	itadmin
Server as Poplar SDK user	poplaruser
Server as IPU-POD admin user	ipuuser
Server as IT admin user	itadmin
Server iDRAC port	root
100GbE RDMA switch	admin
1GbE Management switch	admin
PDU	apc

The default passwords are available from Graphcore support portal.

The following table gives the required groups provided on the IPU‑POD₆₄.

Table 4.7 Required user groups
Groups	Description
root	A root group to locate the root account is mandatory.
dhcpd	A group to allocate the DHCP service is mandatory (this is usually configured automatically while installing the DHCP service).
ipugroup	A group to allocate ipuuser is mandatory.
poplargroup	A group to allocate poplaruser is mandatory.
ipupodgroup	A group to allocate both ipuuser and poplaruser is mandatory.

Note that all users need to have unique user IDs and group IDs.

4.1.10. DHCP Service (Dynamic Host Configuration Protocol)

An ISC-DHCP-Server service is recommended to provide DHCP network configuration to IPU-M2000s. It can be installed from the Ubuntu or CentOS public repositories.

Table 4.8 DHCP service
Name	Type	User	Group	Access	Description
/etc/dhcp/	folder	root	dhcpd	0755	DHCP related files
/etc/dhcp/dhcpd.d/	folder	root	dhcpd	0755	IPU-M2000 network configuration files
Ubuntu: /etc/default/isc-dhcp-server	file	root	root	0644	Network interfaces which DHCP will use
CentOS: /etc/sysconfig/dhcpd	file	root	root	0644	Network interfaces which DHCP will use
/etc/dhcp/dhcpd.d/vlan-11.conf	file	root	dhcpd	0644	Mapping between desired IPs and IPU-M2000 100GbE RNIC
/etc/dhcp/dhcpd.d/vlan-13.conf	file	root	dhcpd	0644	Mapping between desired IPs and IPU-M2000 1GbE BASE-T interfaces
/etc/dhcp/dhcpd.d/vlan-99.conf	file	root	dhcpd	0644	Mapping between desired IPs and management 1GbE BASE-T interfaces MAC addresses
/etc/dhcp/dhcpd.conf	file	root	dhcpd	0644	Main DHCP server configuration file

File templates

Ubuntu: /etc/default/isc-dhcp-server
```
INTERFACES=”eno3 enp59s0f1”
```
CentOS: /etc/sysconfig/dhcpd
```
DHCPDARGS="em4 p1p2
```

/etc/dhcp/dhcpd.d/vlan-11.conf (file)

#Example of content. Add all RNICs below using this format
#
#host ipum1mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.1;}
#host ipum2mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.2;}
#host ipum3mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.3;}
#host ipum4mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.4;}
#host ipum5mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.5;}
#host ipum6mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.6;}
#host ipum7mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.7;}
#host ipum8mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.8;}
#host ipum9mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.9;}
#host ipum10mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.10;}
#host ipum11mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.11;}
#host ipum12mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.12;}
#host ipum13mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.13;}
#host ipum14mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.14;}
#host ipum15mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.15;}
#host ipum16mx {   hardware ethernet aa:bb:cc:dd:ee:ff;    fixed-address 10.1.5.16;}

/etc/dhcp/dhcpd.d/vlan-13.conf

#Example of content. Add all BMCs and Gws below using this format
#
#host ipum1bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.1;}
#host ipum1gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.1;}
#host ipum2bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.2;}
#host ipum2gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.2;}
#host ipum3bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.3;}
#host ipum3gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.3;}
#host ipum4bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.4;}
#host ipum4gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.4;}
#host ipum5bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.5;}
#host ipum5gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.5;}
#host ipum6bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.6;}
#host ipum6gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.6;}
#host ipum7bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.7;}
#host ipum7gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.7;}
#host ipum8bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.8;}
#host ipum8gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.8;}
#host ipum9bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.9;}
#host ipum9gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.9;}
#host ipum10bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.10;}
#host ipum10gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.10;}
#host ipum11bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.11;}
#host ipum11gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.11;}
#host ipum12bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.12;}
#host ipum12gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.12;}
#host ipum13bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.13;}
#host ipum13gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.13;}
#host ipum14bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.14;}
#host ipum14gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.14;}
#host ipum15bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.15;}
#host ipum15gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.15;}
#host ipum16bmc {   hardware ethernet a0:b0:c0:d0:e0:f0;    fixed-address 10.1.1.16;}
#host ipum16gw  {   hardware ethernet a1:b1:c1:d1:e1:f1;    fixed-address 10.1.2.16;}

/etc/dhcp/dhcpd.d/vlan-99.conf

#Example of content. Add all Management devices below using this format
#
#host MgmtDevice1 {   hardware ethernet a2:b2:c2:d2:e2:f2;    fixed-address 10.1.6.x;}

/etc/dhcp/dhcpd.conf

default-lease-time 600;
max-lease-time 1200;
ddns-update-style none;
authoritative;
log-facility local7;
subnet 10.1.4.0 netmask 255.255.254.0 {
    option subnet-mask         255.255.254.0;
    range                      10.1.5.200 10.1.5.254;
    option ntp-servers         10.1.5.101;
}
include "/etc/dhcp/dhcpd.d/vlan-11.conf";
subnet 10.1.0.0  netmask 255.255.252.0 {
    option subnet-mask         255.255.252.0;
    range                      10.1.3.200 10.1.3.254;
    option ntp-servers         10.1.5.101;
}
include "/etc/dhcp/dhcpd.d/vlan-13.conf";
subnet 10.1.6.0 netmask 255.255.254.0 {
    option subnet-mask         255.255.254.0;
    range                      10.1.6.200 10.1.6.254;
}
include "/etc/dhcp/dhcpd.d/vlan-99.conf";

The DHCP service is started using:

$ sudo systemctl enable dhcp
$ sudo systecmtl start dhcp

4.1.11. Rsyslog service

This is a software utility for forwarding log messages in an IP network.

Table 4.9 Rsyslog service
Name	Type	User	Group	Access	Description
/etc/rsyslog.d	folder	root	root	0755	Rsyslog tool configuration folder
/etc/rsyslog.conf	file	root	root	0744	IPU-M2000 network configuration
/etc/rsyslog.d/99_ipum.conf	file	root	root	0744	Rsyslog rules configuration
/etc/rsyslog.d/99_dhcpd.conf	file	root	root	0744	Rsyslog rules configuration

File templates

/etc/rsyslog.conf

module(load="imuxsock")
module(load="imudp")
input(type="imudp" port="514")
module(load="imklog" permitnonkernelfacility="on")
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$FileOwner ipuuser
$FileGroup ipugroup
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$IncludeConfig /etc/rsyslog.d/*.conf

/etc/rsyslog.d/99_ipum.conf

$template tplremote,"%timegenerated% %HOSTNAME% %fromhost-ip% %syslogtag%%msg:::drop-last-lf%\n"
$template bmclogdir,"/var/log/ipumlogs/bmclogs/%fromhost-ip%.log"
$template gwlogdir,"/var/log/ipumlogs/gwlogs/%fromhost-ip%.log"

if $fromhost-ip startswith '10.1.1' then ?bmclogdir;tplremote
if $fromhost-ip startswith '10.1.2' then ?gwlogdir;tplremote

& ~

/etc/rsyslog.d/99_dhcpd.conf
```
local7.*         /var/log/dhcpd.log
```

4.1.12. NTP service (Network Time Protocol)

NTP service is recommended to provide network time configuration to IPU-M2000s. It can be installed from the Ubuntu or CentOS public repositories.

File structure

/etc/ntp.conf (file)

NTP tool configuration file.
root:root 0744

disable monitor
driftfile /var/lib/ntp/drift
fudge 127.127.1.0 stratum 10
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
restrict ::1
restrict 127.0.0.1
restrict default nomodify notrap nopeer noquery
server 127.127.1.0
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst

The NTP service is started using:

sudo systemctl enable ntpd
sudo systecmtl start ntpd

4.1.13. Other configuration files and folders

/etc/security/access.conf (file)

Configure access permissions
Root:root:0744

+ : root : cron crond :0 tty1 tty2 tty3 tty4 tty5 tty6
+ : poplargroup : ALL
+ : poplaruser : ALL
+ : ipugroup : ALL
+ : ipuuser : ALL
+ : itadmin : ALL
+ : root : ALL
- : ALL : ALL

/etc/security/limits.conf (file)
- Configure access permissions
- Root:root:0744
```
*       soft    memlock unlimited
*       hard    memlock unlimited
```

Create directories required for logging

Table 4.10 Logging directories
Directory	User	Group	Access
/localdata	root	ipupodgroup	0775
/localdata/log	root	ipupodgroup	0770
/localdata/log/ipumlogs	root	ipupodgroup	0750
/var/log/ipumlogs	root	ipupodgroup	0750
/var/log/ipumlogs/bmclogs	root	ipupodgroup	0750
/var/log/ipumlogs/gwlogs	root	ipupodgroup	0750
/etc/ipuof.conf.d	root	ipupodgroup	0770

Mount and bind logging volume
- /var/log to /localdata/logs/ipumslogs

4.2. Network configuration

4.2.1. Overview

The following figure gives a logical overview of the network setup within the IPU‑POD₆₄.

_images/network_overview.png — Fig. 4.2 IPU‑POD₆₄ network overview

`*` Option: VLAN 14 and the cabling to the dedicated BMC port can be provided as an upgrade for customers that want an isolated BMC network. This separates the BMC and GW traffic inside the IPU-M2000 onto the two ports, BMC on lower port, GW on upper port. With this option enabled, VLAN 13 becomes a GW only VLAN. Please contact Graphcore Sales for more information.

4.2.2. IPU‑POD₆₄ network interfaces

Table 4.11 Network configuration
Port	Role	Speed	IP address	Config from	VLAN (see note)
IPU-M2000: BMC	BMC only management (future)	1GbE	10.1.1.1-16/22	Static DHCP	14
IPU-M2000: GW	BMC+GW management	1GbE	10.1.2.1-16/22	Static DHCP	13
Server: Port1	Mgmt IPU-M2000	1GbE	10.1.3.101/22	Local netplan file	13
Server: Port2	Div management iDRAC, switches & PDUs	1GbE	10.1.6.1/22	Local netplan file	99
Server: iDRAC	Server BMC	1GbE	10.1.6.4-7/22	Manual setup	99
Server: RNIC/Port0	RDMA IPU-M2000	100GbE	10.1.5.101/23	Local netplan file	11
Server: RNIC/Port1	RDMA NAS	100GbE	Site specific	Site specific	Site specific
48x 1GbE + (4x 10G) Switch management port	CLI + Switch BMC management	1GbE	10.1.6.2/22	Manual setup	99
32x 100GbE + (4x 10G) switch management port	CLI + Switch BMC management	1GbE	10.1.6.3/22	Manual setup	99
PDU	Power dist. unit	1GbE	10.1.6.8-11	Dynamic DHCP	99

Note: Port based VLAN in switches (VLAN 13,14 and 99 in 1GbE switch, VLAN 11 in 100GbE switch)

4.2.3. Management switch configuration

Several VLANs need to be configured on the switches to separate traffic for the different hardware and integrate DHCP properly on the system.

The default management switch is Arista DCS-7010T-48-F.

Up to 4 interfaces per rack for PDUs and switches management
Up to 4 interfaces per rack for server management interfaces
Up to 4 interfaces per rack for server facing BMCs and gateways
Up to 16 interfaces per rack for combined IPU-M2000 Gateway and BMC
Up to 16 interfaces per rack for BMC only connection (future option)

_images/mng_switch_VLAN.png — Fig. 4.3 Management switch VLAN assignments

These port groups are members of the switch’s internal port based VLANs. The VLAN assignments are given below.

VLAN 13: IPU-M2000 BMC and GW traffic
VLAN 14: IPU-M2000 BMC only traffic (option)
VLAN 99: Device management

Device management is intended for switch management, PDU remote management and any other device in the system with remote management interfaces.

The reference configuration for the management switch is as follows:

Table 4.12 Management switch configuration
Port	Description	Vlan	Options
Hostname MGMT-Switch
Switchport default mode access
Spanning-tree mode mstp
No aaa root
No ip routing
1	Datacentre BMC	13	no snmp trap link-status
2	Datacentre GW	14	no snmp trap link-status
3	Datacentre management	99	no snmp trap link-status
4	–	–	shutdown
5-6	PDU management	99	no snmp trap link-status
7	ToR switch management	99	no snmp trap link-status
8	Server 1 POD management	99	no snmp trap link-status
9-12	Servers BMC - iDrac	99	no snmp trap link-status
13-16	Servers 1Gbe traffic	13	no snmp trap link-status
17-32	IPU GW and BMC	13	no snmp trap link-status
33-48	IPU BMC Only	14	shutdown

An example switch configuration file can be provided – please contact Graphcore support for details. This configuration file can be applied using one of the standard methods described by the switch manufacturer.

4.2.4. ToR switch configuration

The default ToR switch is an Arista DCS-7060CX-32S-F. The ports are allocated as follows:

One port per server for traffic between the server(s) and IPU-M2000s, on VLAN11
One port per server for uplink traffic from the server(s) to datacentre network-attached storage. This is disabled in the default setup file since NAS configuration will be site specific.
16 ports for connection to the IPU-M2000s on VLAN 11
This is disabled in the default setup file since the uplink configuration will be site specific.

_images/ToR_switch_VLAN.png — Fig. 4.4 ToR switch VLAN assignments

The reference configuration for the ToR switch is as follows:

Table 4.13 ToR switch configuration
Port	Description	Vlan	Options
Hostname MGMT-Switch
Switchport default mode access
Spanning-tree mode mstp
No aaa root
lldp run
No ip routing
1/1 3/1 5/1 7/1	Server Uplink	12	dcbx mode ieee priority-flow-control on priority-flow-control priority [0-7] no-drop no snmp trap link-status
2/1 4/1 6/1 8/1	Server IPUoF	11	dcbx mode ieee priority-flow-control on priority-flow-control priority [0-7] no-drop no snmp trap link-status
9/1 to 24/1	IPU-M2000	11	dcbx mode ieee priority-flow-control on priority-flow-control priority [0-7] no-drop no snmp trap link-status
25/1 to 32/1	Datacentre Uplink	See note	dcbx mode ieee priority-flow-control on priority-flow-control priority [0-7] no-drop no snmp trap link-status

Note

The VLAN assignment for the datacentre uplink ports will be site and configuration specific. In the default configuration file these ports are disabled.

A switch configuration file can be provided – please contact Graphcore support for details. This configuration file can be applied using one of the standard methods described by the switch manufacturer.

4.2.5. IPU‑POD₆₄ VLAN assignments

Each switch is configured independently based on the number of interfaces needed for the IPU-POD size (in this case IPU‑POD₆₄). This section describes the interfaces used in an IPU‑POD₆₄ with figures showing interface allocation.

The following figure shows the VLAN assignments for the server connecting between the server(s) and the switches. Four server-facing ports are shown for each group on the switches to allow for up to 4 servers in the POD.

_images/server_VLAN.png — Fig. 4.5 Dell R6525 server VLAN connectivity

The following figure shows the VLAN assignments for the IPU-M2000s connecting to the switches.

_images/M2000_VLAN.jpg — Fig. 4.6 IPU-M2000 server VLAN connectivity

4.2.6. Server network configuration

It is recommended to use the Netplan network manager to configure the server using netplan configuration files
Two 1GbE baseT connections are required to the 1GbE management switch for Server 1. Only one 1GbE baseT connection is required to the 1GbE management switch for additional servers. Fixed IP addresses are required.
One 100Gb connection is required to the 100GbE switch. A fixed IP address is required.
Subnets for each interface should be capable to contain the number of devices of the installation.

Example netplan configuration file

The default location for this file is /etc/netplan/01-netcfg.yaml.
In this example:
- Interface eno3 is facing the 1GbE network with IPU-M2000 BMCs and gateways
- Interface eno4 is facing the user’s network
- Interface enp59s0f1 is facing the IPU-M2000 100GbE RNICs

network:
  version: 2
  renderer: networkd
  ethernets:
    eno1np0:
      dhcp4: yes
    eno2np1:
      dhcp4: yes
    eno3:
      addresses:
        - 10.1.3.101/22
    eno4:
      dhcp4: yes
    enp59s0f0:
      dhcp4: yes
    enp59s0f1:
      addresses:
        - 10.1.5.101/23

4. Server and switch configuration

4.1. Server configuration

4.1.1. Hardware recommendations

4.1.2. Storage configuration recommendations

4.1.3. Memory configuration

4.1.4. BIOS configuration

4.1.5. Operating system recommendations

4.1.6. Ubuntu 18.04 packages

4.1.7. CentOS 7 packages

4.1.8. Python packages

4.1.9. User accounts and groups

4.1.10. DHCP Service (Dynamic Host Configuration Protocol)

File templates

4.1.11. Rsyslog service

File templates

4.1.12. NTP service (Network Time Protocol)

File structure

4.1.13. Other configuration files and folders

4.2. Network configuration

4.2.1. Overview

4.2.2. IPU‑POD64 network interfaces

4.2.3. Management switch configuration

4.2.4. ToR switch configuration

4.2.5. IPU‑POD64 VLAN assignments

4.2.6. Server network configuration

Example netplan configuration file

4.2.2. IPU‑POD₆₄ network interfaces

4.2.5. IPU‑POD₆₄ VLAN assignments