Cannot add MAC address to a VF interface

[lcongio]

Hi, I have encountered an issue in TRex v3.04 with the Broadcom NetXtreme-E NICs.

If I bind the PFs to DPDK I can run the TRex, but if I bind the VFs I get this error:

bnxt_stats_reset_op(): Device Initialization not complete!
bnxt_mac_addr_add_op(): Cannot add MAC address to a VF interface
Failed setting MAC for port 0
Killing Scapy server... Scapy server is killed

Using TestPMD I can reach the VFs.

The full log with TRex is bellow (the error is at the end):

$ sudo ./t-rex-64 -i -v 7 --cfg /etc/trex_cfg.yaml
Starting Scapy server.... Scapy server is started
The ports are bound/configured.
Starting  TRex v3.04 please wait  ...
Using configuration file /etc/trex_cfg.yaml
 port limit     :  not configured
 port_bandwidth_gb    :  10
 port_speed           :  0
 port_mtu             :  0
 if_mask        : None
 is low-end : 0
 stack type :
 thread_per_dual_if      : 1
 if        :  65:02.0, 65:03.0,
 enable_zmq_pub :  1
 zmq_pub_port   :  4500
 m_zmq_rpc_port    :  4501
 src     : 00:00:00:00:00:00
 dest    : 00:00:00:00:00:00
 src     : 00:00:00:00:00:00
 dest    : 00:00:00:00:00:00
 memory per 2x10G ports
 MBUF_64                                   : 16380
 MBUF_128                                  : 8190
 MBUF_256                                  : 8190
 MBUF_512                                  : 8190
 MBUF_1024                                 : 8190
 MBUF_2048                                 : 4095
 MBUF_4096                                 : 128
 MBUF_9K                                   : 512
 TRAFFIC_MBUF_64                           : 65520
 TRAFFIC_MBUF_128                          : 32760
 TRAFFIC_MBUF_256                          : 8190
 TRAFFIC_MBUF_512                          : 8190
 TRAFFIC_MBUF_1024                         : 8190
 TRAFFIC_MBUF_2048                         : 32760
 TRAFFIC_MBUF_4096                         : 128
 TRAFFIC_MBUF_9K                           : 512
 MBUF_DP_FLOWS                             : 524288
 MBUF_GLOBAL_FLOWS                         : 5120
 master   thread  : 0
 rx  thread  : 15
 dual_if : 0
    socket  : 0
   [   1   2   3   4   5   6   7   8   9   10   11   12   13   14     ]
CTimerWheelYamlInfo does not exist
 flags           : 8010f00
 write_file      : 0
 verbose         : 7
 realtime        : 1
 flip            : 0
 cores           : 1
 single core     : 0
 flow-flip       : 0
 no clean close  : 0
 zmq_publish     : 1
 vlan mode       : 0
 client_cfg      : 0
 mbuf_cache_disable  : 0
 cfg file        :
 mac file        :
 out file        :
 client cfg file :
 duration        : 0
 factor          : 1
 mbuf_factor     : 1
 latency         : 0 pkt/sec
 zmq_port        : 4500
 telnet_port     : 4501
 expected_ports  : 2
 tw_bucket_usec  : 20.000000 usec
 tw_buckets      : 1024 usec
 tw_levels       : 3 usec
 port : 0 dst:00:00:00:00:00:00  src:00:00:00:00:00:00
 port : 1 dst:00:00:00:00:00:00  src:00:00:00:00:00:00
 port : 2 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 3 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 4 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 5 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 6 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 7 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 8 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 9 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 10 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 11 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 12 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 13 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 14 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 15 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 16 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 17 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 18 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 19 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 20 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 21 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 22 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 23 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 24 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 25 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 26 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 27 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 28 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 29 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 30 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 port : 31 dst:00:00:00:01:00:00  src:00:00:00:00:00:00
 Total Memory :
 MBUF_64                                   : 81900
 MBUF_128                                  : 40950
 MBUF_256                                  : 16380
 MBUF_512                                  : 16380
 MBUF_1024                                 : 16380
 MBUF_2048                                 : 36855
 MBUF_4096                                 : 1024
 MBUF_DP_FLOWS                             : 524288
 MBUF_GLOBAL_FLOWS                         : 5120
 get_each_core_dp_flows                    : 524288
 Total memory                              :     248.40 Mbytes
 core_list : 0,15,1
 sockets : 0
 active sockets : 1
 ports_sockets : 1
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
 phy   |   virt
 1      1
DPDK args
 xx  -l  0,15,1  -n  4  --log-level  8  --main-lcore  0  -a  0000:65:02.0  -a  0000:65:03.0  --legacy-mem
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:65:02.0 (socket 0)
EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:65:03.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
                 input : [65:02.0, 65:03.0]
                 dpdk : [0000:65:02.0, 0000:65:03.0]
             pci_scan : [0000:65:02.0, 0000:65:03.0]
                  map : [ 0, 1]
 TRex port mapping
 -----------------
 TRex vport: 0 dpdk_rte_eth: 0
 TRex vport: 1 dpdk_rte_eth: 1
 set driver name net_bnxt
 driver capability  : TCP_UDP_OFFLOAD  TSO  LRO
 set dpdk queues mode to ONE_QUE
 DPDK devices 2 : 2
-----
 0 : vdev 0000:65:02.0
 1 : vdev 0000:65:03.0
-----
 Number of ports found: 2


if_index : 0
driver name : net_bnxt
min_rx_bufsize : 1
max_rx_pktlen  : 9600
max_rx_queues  : 63
max_tx_queues  : 63
max_mac_addrs  : 128
rx_offload_capa : 0xd265f
tx_offload_capa : 0x19faf
rss reta_size   : 128
flow_type_rss   : 0xc000000000d34
tx_desc_max     : 4096
tx_desc_min     : 16
rx_desc_max     : 8192
rx_desc_min     : 16
zmq publisher at: tcp://*:4500
 rx_data_q_num : 1
 rx_drop_q_num : 0
 rx_dp_q_num   : 0
 rx_que_total : 1
 --
 rx_desc_num_data_q   : 4096
 rx_desc_num_drop_q   : 4096
 rx_desc_num_dp_q     : 0
 total_desc           : 4096
 --
 tx_desc_num     : 1024
port 0 desc: NetXtreme-E Ethernet Virtual Function
 rx_qid: 0 (4096)
bnxt_stats_reset_op(): Device Initialization not complete!
bnxt_mac_addr_add_op(): Cannot add MAC address to a VF interface
Failed setting MAC for port 0
Killing Scapy server... Scapy server is killed

Below some other infos:

 Open Source Components / Libraries 
 DPDK       (BSD)       
 YAML-CPP   (BSD)       
 JSONCPP    (MIT)       
 BPF        (BSD)       
 HDR-HISTOGRAM-C (CC0)  
 
 Open Source Binaries 
 ZMQ        (LGPL v3plus) 
 
 Version : v3.04   
 DPDK version : DPDK 23.03.0   
 User    : root   
 Date    : Apr 24 2024 , 20:39:54 
 Uuid    : 393570d6-027a-11ef-900d-73f27c253397    
 Git SHA : 64cd404ed8614ad9d5e5a86f0334100e134647d8    

Compiled with GCC     :   9.4.0
Compiled with glibc   :   2.31 (host: 2.31)
Sanitized image       :   no

$ sudo ./dpdk_setup_ports.py -t
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| ID | NUMA |   PCI   |        MAC        |                             Name                             |  Driver  | Linux IF  |  Active  |
+====+======+=========+===================+==============================================================+==========+===========+==========+
| 0  | 0    | 04:00.0 | 2c:ea:7f:b6:8a:1e | NetXtreme BCM5720 2-port Gigabit Ethernet PCIe               | tg3      | eno1      | *Active* |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 1  | 0    | 04:00.1 | 2c:ea:7f:b6:8a:1f | NetXtreme BCM5720 2-port Gigabit Ethernet PCIe               | tg3      | eno2      |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 2  | 0    | 65:00.0 | b0:26:28:79:e4:20 | BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller | bnxt_en  | ens2f0np0 |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 3  | 0    | 65:00.1 | b0:26:28:79:e4:21 | BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller | bnxt_en  | ens2f1np1 |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 4  | 0    | 65:02.0 | 62:52:b3:8c:eb:0f | NetXtreme-E Ethernet Virtual Function                        | vfio-pci |           |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+
| 5  | 0    | 65:03.0 | 8a:82:0e:c2:bf:9e | NetXtreme-E Ethernet Virtual Function                        | vfio-pci |           |          |
+----+------+---------+-------------------+--------------------------------------------------------------+----------+-----------+----------+

$ sudo ./dpdk_setup_ports.py -s

Network devices using DPDK-compatible driver
============================================
0000:65:02.0 'NetXtreme-E Ethernet Virtual Function' drv=vfio-pci unused=bnxt_en,igb_uio,uio_pci_generic
0000:65:03.0 'NetXtreme-E Ethernet Virtual Function' drv=vfio-pci unused=bnxt_en,igb_uio,uio_pci_generic

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=igb_uio,vfio-pci,uio_pci_generic 
0000:65:00.0 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller' if=ens2f0np0 drv=bnxt_en unused=igb_uio,vfio-pci,uio_pci_generic 
0000:65:00.1 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller' if=ens2f1np1 drv=bnxt_en unused=igb_uio,vfio-pci,uio_pci_generic 

Other network devices
=====================
<none>

/etc/trex_cfg.yaml

### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['65:02.0', '65:03.0']
  port_info:
      - ip: 10.60.1.45
        default_gw: 10.60.1.1
      - ip: 10.30.1.45
        default_gw: 10.30.1.1

  platform:
      master_thread_id: 0
      latency_thread_id: 15
      dual_if:
        - socket: 0
          threads: [1,2,3,4,5,6,7,8,9,10,11,12,13,14]

[lcongio]

I tried other versions of TRex and the last version without error in the bxnt driver with the VFs is v2.87.