The steps mentioned here were tested on various Cloudlab machines. If these steps don't work, refer to the DPDK documentation
Before starting, if you're NIC uses flow bifurcation (mostly used by Mellanox/NVIDIA NICs. Verify here), it is better to follow step 2.2 first and then return to step 1. DPDK selects which Poll Mode Drivers to install based on your machine's configuration. So, it is important that you configure NVIDIA OFED before installing DPDK. You can start from step 1 if you are using
vfio-pci
drivers.
- Run
sudo bash dpdk_prereq.sh
- Update $PATH variable using
export PATH=$PATH:$HOME/.local/bin
- Install meson and ninja using
pip3 install meson ninja pyelftools
- Run
bash dpdk_setup.sh
- IOMMU is required to make full use of the VFIO functionality. To verify if IOMMU is enabled, run
sudo dmesg | grep -i -e DMAR -e IOMMU
. You should see a message similar to this which specifies which IOMMU group a PCI port belongs to
[ 11.385867] pci 0000:ff:16.0: Adding to iommu group 87
[ 11.405414] pci 0000:ff:16.1: Adding to iommu group 87
[ 11.416050] pci 0000:ff:16.2: Adding to iommu group 87
[ 11.472783] DMAR: Intel(R) Virtualization Technology for Directed I/O
- If IOMMU is not enabled, we have to update the GRUB file and restart the machine.
- Edit the GRUB file using
sudo nano /etc/default/grub
- Add the following options to the line
GRUB_CMDLINE_LINUX_DEFAULT=""
- For Intel CPU:
intel_iommu=on
- For AMD CPU:
amd_iommu=on
- For Intel CPU:
- Update GRUB
sudo update-grub
and reboot the systemsudo reboot
.
- Edit the GRUB file using
- Use
./dpdk-23.07/usertools/dpdk-devbind.py --status
to check the status of the network interfaces. If there are no devices listed in 'Network devices using DPDK-compatible drive,' we need to bind them tovfio-pci
.
We cant update the driver when a network interface is active. Use
ifconfig <if_name> down
to deactivate it.
- To bind the interface to the vfio-pci driver, use
sudo ./dpdk-23.07/usertools/dpdk-devbind.py --bind=vfio-pci <PCI-port-num>
and verify its status by running devbind using the--status
flag
If you still get errors when binding, try using the
uio_pci_generic
module instead of thevfio-pci
module. The details can be found in the documentation
- Download the appropriate Linux drivers based on your device's configuration from NVIDIA's website
This sample includes the steps for installing OFED version 5.8 for an Ubuntu 22.04 machine and kernel version 5.15.0-86-generic. Just updating the parameters based on your machine's configuration should work fine. However, if there are some issues, the detailed installation instructions can be found here
mkdir /mnt/mlnx
mount -o ro,loop MLNX_OFED_LINUX-5.8-4.1.5.0-ubuntu22.04-x86_64.iso /mnt/mlnx
/mnt/mlnx/mlnxofedinstall --dpdk --without-dkms --add-kernel-support --kernel 5.15.0-86-generic --without-fw-update --force
The
--dpdk
flag will be required irrespective of any configuration changes to ensure that the appropriate PMDs are configured.
Based on the output of the installation step, you might have to run some commands (which will be explained in the terminal output). If there are any doubts, refer to the installation steps
Typically, you have to mount huge pages before running any DPDK application. You might also have to mount hugepages after restarting the machine.
- DPDK has a program which mounts hugepages and reserves them.
sudo python3 ./dpdk-23.07/usertools/dpdk-hugepages.py -p 1G --setup 2G
- This will setup 2 - 1GB hugepages.
- Go to the DPDK build directory
$ meson configure -Dexamples=all
to enable examples compilation
To compile a specific example, use
$ meson configure -Dexamples=<example_folder_name>
instead
$ ninja
to build the examples- To run the sample applications, refer to the user guides
Important: Verify the connection of the traffic generator and the DPDK application before running the examples.
- Inside the build directory, run the following command to start testpmd
sudo ./app/dpdk-testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2
- To start the port
port start <port-no>
- To show the port status
show port info <port-no>
- To set the packet forwarding mode
set fwd txonly
- To set the destination
set eth-peer <port-no> <dest-mac-addr>
- To start the transmission
start
- To stop the transmission
stop