Running tests with DPDK interfaces

[marthin-vandyk]

Hi all,

We are looking to run bngblaster with DPDK and have followed all the docs and How-To's that we could find.

Our status at the moment:

• bngblaster DPDK support and version

root@dell630:~# bngblaster -v
Version: DEV
Compiler: GNU (11.3.0)
GIT:
  REF: main
  SHA: 42975adeeac7944ddecb7cba441925531088d321
IO Modes: packet_mmap_raw (default), packet_mmap, raw, dpdk

• Network drivers

root@dell630:~# dpdk-devbind.py -s
[truncated output]
Network devices using kernel driver
===================================
0000:01:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno1 drv=tg3 unused=vfio-pci *Active*
0000:01:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno2 drv=tg3 unused=vfio-pci 
0000:02:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno3 drv=tg3 unused=vfio-pci 
0000:02:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno4 drv=tg3 unused=vfio-pci 
0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp129s0f0 drv=i40e unused=vfio-pci 
0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp129s0f1 drv=i40e unused=vfio-pci 
0000:81:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp129s0f2 drv=i40e unused=vfio-pci 
0000:81:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp129s0f3 drv=i40e unused=vfio-pci

root@dell630:~# dpdk-devbind.py --bind=vfio-pci 0000:81:00.0
root@dell630:~# dpdk-devbind.py --bind=vfio-pci 0000:81:00.1
root@dell630:~# dpdk-devbind.py -s
[truncated output]
Network devices using DPDK-compatible driver
============================================
0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e
0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e

• To verify the correct selection of NICs for the DPDK bind, here is an extract of our current working test json. This is a dual-stack scenario.

{
    "interfaces": {
	"rx-threads": 8,
	"tx-threads": 8,
	"io-slots": 32768,
	"network": {
            "interface": "enp129s0f0",
            "address": "100.127.6.2/24",
            "gateway": "100.127.6.1",
            "address-ipv6": "2001:db8::2/64",
            "gateway-ipv6": "2001:db8::1",
            "vlan": 99
        },
        "access": {
            "interface": "enp129s0f1",
            "vlan-mode": "N:1",
	    "outer-vlan": 100,
            "type": "pppoe",
            "stream-group-id": 1
        }
    },
    "streams": [
	{
	    "name": "general4",
	    "stream-group-id": 1,
	    "type": "ipv4",
	    "direction": "downstream",
	    "bps": 2048000,
	    "length": 1024,
	    "start-rate": 30,
	    "stop-rate": 30,
	    "network-interface": "enp129s0f0:99"
        },
	{
            "name": "general6",
            "stream-group-id": 1,
            "type": "ipv6",
            "direction": "downstream",
            "bps": 2048000,
            "length": 1024,
            "start-rate": 30,
            "stop-rate": 30,
            "network-interface": "enp129s0f0:99"
        }
    ],

• When we run it with the following test json, the network interfaces never resolve. With the normal json config, we are able to ping the bngblaster network interface from the BNG DUT, however when we are running it with DPDK we cannot ping. Note this config is simplified to be v4 only, also verified to be working when not using DPDK mode.

{
    "interfaces": {
	"io-mode": "dpdk",
	"tx-threads": 8,
	"rx-threads": 8,
	"io-slots": 32768,
	"network": {
            "interface": "0000:81:00.0",
            "address": "100.127.6.2/24",
            "gateway": "100.127.6.1",
            "vlan": 99
        },
        "access": {
            "interface": "0000:81:00.1",
            "vlan-mode": "N:1",
	    "outer-vlan": 100,
            "type": "pppoe",
            "stream-group-id": 1
        }
    },
    "streams": [
	{
	    "name": "general4",
	    "stream-group-id": 1,
	    "type": "ipv4",
	    "direction": "downstream",
	    "bps": 1536000,
	    "length": 1440,
	    "network-interface": "0000:81:00.0:99"
        }
    ],

Any thoughts perhaps plse?

many thanks
Marthin

[GIC-de]

Could you start BNG Blaster with DPDK logging (-l dpdk) in non-interactive (without -I) mode and share output?

Some customers also reported issues if they have virtual functions configured. Please also ensure that driver and intel firmware is updated and that correct DPDK version is used (21.11.2).

[GIC-de]

I'm also currently working on the DPDK documentation.

[marthin-vandyk]

Hi Christian,

Apologies for the late reply here. We had to change hosts and thus the NICs and now cannot seem to get the vfio-pci bind going.

Once we have supported NICs will try again and post to this thread. For now I will mark as outdated and re-open later.

root@bng-poc1-bngblaster:~/bngblaster/buid# bngblaster -v
Version: DEV
Compiler: GNU (11.3.0)
GIT:
  REF: main
  SHA: 6d08211740ab43777f477e40a40f5c91c1c9a076
IO Modes: packet_mmap_raw (default), packet_mmap, raw, dpdk

root@bng-poc1-bngblaster:~# dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:01:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno1 drv=tg3 unused=vfio-pci 
0000:01:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno2 drv=tg3 unused=vfio-pci 
0000:02:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno3 drv=tg3 unused=vfio-pci 
0000:02:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno4 drv=tg3 unused=vfio-pci *Active*
0000:04:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=enp4s0f0 drv=tg3 unused=vfio-pci 
0000:04:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=enp4s0f1 drv=tg3 unused=vfio-pci 
0000:04:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=enp4s0f2 drv=tg3 unused=vfio-pci 
0000:04:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=enp4s0f3 drv=tg3 unused=vfio-pci 
0000:81:00.0 'T62100-LP-CR Unified Wire Ethernet Controller 6007' if= drv=cxgb4 unused=vfio-pci 
0000:81:00.1 'T62100-LP-CR Unified Wire Ethernet Controller 6007' if= drv=cxgb4 unused=vfio-pci 
0000:81:00.2 'T62100-LP-CR Unified Wire Ethernet Controller 6007' if= drv=cxgb4 unused=vfio-pci 
0000:81:00.3 'T62100-LP-CR Unified Wire Ethernet Controller 6007' if= drv=cxgb4 unused=vfio-pci 
0000:81:00.4 'T62100-LP-CR Unified Wire Ethernet Controller 6407' if=enp129s0f4,enp129s0f4d1 drv=cxgb4 unused=vfio-pci *Active*
0000:82:00.0 'T6225-CR Unified Wire Ethernet Controller 6001' if= drv=cxgb4 unused=vfio-pci 
0000:82:00.1 'T6225-CR Unified Wire Ethernet Controller 6001' if= drv=cxgb4 unused=vfio-pci 
0000:82:00.2 'T6225-CR Unified Wire Ethernet Controller 6001' if= drv=cxgb4 unused=vfio-pci 
0000:82:00.3 'T6225-CR Unified Wire Ethernet Controller 6001' if= drv=cxgb4 unused=vfio-pci 
0000:82:00.4 'T6225-CR Unified Wire Ethernet Controller 6401' if=enp130s0f4d1,enp130s0f4 drv=cxgb4 unused=vfio-pci 

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================
root@bng-poc1-bngblaster:~# 

root@bng-poc1-bngblaster:~# dpdk-devbind.py --bind=vfio-pci 0000:82:00.4
Error: bind failed for 0000:82:00.4 - Cannot bind to driver vfio-pci: [Errno 22] Invalid argument
Error: unbind failed for 0000:82:00.4 - Cannot open /sys/bus/pci/drivers//unbind: [Errno 13] Permission denied: '/sys/bus/pci/drivers//unbind'
root@bng-poc1-bngblaster:~#