Server requirements
This chapter describes the reference baseline configuration for the server in an IPU-POD DA system before running the IPU-M2000 system software installation script. The high-level steps for configuring the server are as follows:
Select appropriate server specifications
Configure the server storage and RAID arrays
Install the server operating system and packages
Install required python packages
Server hardware and storage
Hardware requirements
This document describes building a system using a PowerEdge R6525 server. Contact Graphcore support for details of other supported server types or check the Approved Servers list. Other servers may have different installation requirements.
The recommended configuration of the Dell R6525 is as follows:
Dell R6525 containing dual AMD EPYC 7742 processors
16x32GB RDIMM PC4-25600 ECC registered dual-rank X4 1.2v
2x 480GB SSD-SATA 6Gbps 2.5 inch hot-swap
7x 1TB NVME SSD PCIe 4x 3.1
Dual port Gigabit BASE-T PCIe
1x single port Mellanox ConnectX-5 EN 100Gb/s Ethernet (IPU-POD4 DA system)
2x dual port Mellanox ConnectX-5 EN 100Gb/s Ethernet (IPU-POD16 DA system)
Storage configuration
The recommendation is to configure two types of server storage: SSD-SATA for the operating system and NVME SSD for data storage.
Operating system:
2x 480GB SSD-SATA drives as a RAID 1 via hardware controller
Partitioned to use ext4 file system
Data storage:
7x 1TB NVME SSDs as a logical RAID 6 managed with MDADM
Partitioned to use xfs file system
Memory configuration
The DIMMs should be installed in a fully symmetric configuration, as recommended by Dell for maximum performance. The recommended configuration has 8 DIMMs per processor, as shown below.
Fig. 25 Recommended DIMM configuration
BIOS configuration
Various BIOS settings can impact the performance of the system. The recommended settings are shown in the table below.
Setting |
BIOS 1.2.11 |
BIOS 1.4.8 or later |
|---|---|---|
LogicalProc |
Enabled |
Enabled |
ProcVirtualization |
Enabled |
Enabled |
IommuSupport |
Disabled |
See below |
L1StreamHwPrefetcher |
Enabled |
Enabled |
L2StreamHwPrefetcher |
Enabled |
Enabled |
MadtCoreEnumeration |
Linear |
Linear |
NumaNodesPerSocket |
4 |
4 |
CcxAsNumaDomain |
Disabled |
Disabled |
CpuMinSevAsid |
1 |
1 |
ProcCcds |
All |
All |
CcdCores |
All |
All |
EmbSata |
AhciMode |
AhciMode |
BIOS 1.4.8 or later does not configure IOMMU directly. Instead, you need to add iommu=off to the kernel boot params in /etc/default/grub and then run update-grub.
Operating system and packages
Please contact your Graphcore representative or use the support portal for information about operating system support. This document describes the following operating systems:
Ubuntu 18.04.4 LTS (bionic)
CentOS 7.2 / 8
Ubuntu OS installation and packages
In order to have a stable system where IPU related software can run, several packages need to be installed on the system via the Aptitude package manager tool:
autoconf |
golang-go |
M4 |
rdma-core |
automake |
htop |
minicom |
rsyslog |
bc |
ibverbs-utils |
moreutils |
screen |
build-essential |
ipmitool |
net-tools |
sshpass |
ccache |
jq |
netcat |
subversion |
clang |
kcachegrind |
ntp |
swig |
cmake |
libaio-dev |
openjdk-8-jdk |
sysfsutils |
direnv |
libboost-all-dev |
parallel |
tar |
dkms |
libeigen3-dev |
pciutils |
tmux |
emacs |
libjson-c-dev |
perl |
u-boot-tools |
ethtool |
libjson-c-doc |
php-cli |
unzip |
exuberant-ctags |
libpci-dev |
php-curl |
valgrind |
flex |
libpixman-1-dev |
policykit-1 |
vim |
g++ |
libprotobuf-dev |
protobuf-compiler |
virtualenv |
gawk |
libssl-dev |
python3 |
wdiff |
gcc |
libtool |
python3-dev |
wget |
gdb |
lldpad |
python3-pip |
zip |
git |
lldpd |
qtcreator |
Python packages
Several python packages are required for both OS installations. They can be installed using the pip installation tool.
autograd |
numpy |
pytest |
setuptools |
boto3 |
paramiko |
pyyaml |
virtualenv |
jstyleson |
pep8 |
recommonmark |
wheel |
lxml |
pexpect |
requests |
yapf |
mock |
pylint |
scp |