100gbe-rdma-setup-intel-e810

Setup kernel dependencies

sudo vim /etc/default/grub

make sure grub cmdline looks similar, edit the isolcpus.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=5-15 nohz_full=5-15 rcu_nocbs=5-15 intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll numa_balancing=disable"

Install dependencies

sudo apt-get install libpci-dev libtool libibverbs-dev librdmacm-dev libibumad-dev automake git wget rdma-core ethtool pciutils

install perftest

git clone http://github.com/linux-rdma/perftest.git

build perftest

cd perftest
./autogen.sh
./configure
make
sudo make install

install kernel module

install ice drivers

wget  https://downloadmirror.intel.com/869925/ice-2.3.15.tar.gz

build

tar -xvzf ice-*-*.tar.gz
cd ice-*-*/src
sudo make -j10 CFLAGS_EXTRA='-DADQ_PERF_COUNTERS' CFLAGS_EXTRA='-DUNIFIED_STATS'  install

install irdma

wget https://downloadmirror.intel.com/869928/irdma-2.0.34.tgz

build

tar -xvzf irdma-*.tar.gz
cd irdma-*/src
sudo make install

setting up environment

change mtu to 9000

sudo ip link set dev enp23s0f0np0 mtu 9000

enable flow control and chage to max size for E810 CQDA2

sudo ethtool -A enp23s0f0np0 rx on tx on
sudo ethtool -G enp23s0f0np0 rx 8160 tx 8160

Turn Off Adaptive Moderation and set Rx interrupt delay to 0

ethtool -C enp23s0f0np0 adaptive-rx off rx-usecs 0

This is required for rdma

sudo modprobe irdma
sudo modprobe ice

iperf test

Kernel Buffer Tuning

# This sets the maximum socket receive/send buffer sizes to 256MB
sudo sysctl -w net.core.rmem_max=268435456
sudo sysctl -w net.core.wmem_max=268435456

# This increases the TCP auto-tuning buffer limits to 256MB
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"

test run on client side

iperf3 -c <server_ip> -P 10 -Z

RDMA test

Server side

ib_write_bw -d iwp23s0f0 -R -a -b -i 1 -m 8160

Client side

ib_write_bw -d iwp23s0f0 -R -a -b -i 1 -m 8160 <server_ip>

The RDMA dev name can be found here:

ls -la /sys/class/infiniband