Benchmarking results

[terryzbai]

Setup

1. QEMU/KVM: Running a guest VM on top of Linux
2. Pure Linux: Linux kernel running on physical board
3. Client VM: Guest Linux running with native sDDF ethernet driver
4. Passthrough VM: Guest Linux running with passthrough ethernet device

ipbench_queue command:

iq.sh run -c vb04 -c vb05 -c vb06 -c vb07 -b -f linux_benchmark.py -- <server_ip_address> --throughputs  <loads_list> --clients vb04 vb05 vb06 vb07

iperf3 command to run on the server (client VM):

iperf3 -s

iperf3 command ro run on the client (vb machine):

iperf3 -c <server_ip_address> --udp -bider --bandwidth=200M --length=1472

Overview of Results

1. QEMU/KVM: (running at loads=100Mb/s)

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,CPU_Utilsation
100000000,79512691,100000139,1472,26389,31086,68475,2926.40,30384,98.3

[terryzbai]

QEMU/KVM Setup

Host: Linux gentoo-maaxboard-1 6.14.6-gentoo (This one does not work anymore for some reason, so I re-built one for running on machine queue for reproducilibity)
Host: Linux kernel 6.1. built from scratch
Guest: Linux kernel 6.1 built from scratch

QEMU command to create the guest VM:

qemu-system-aarch64 -machine virt -cpu cortex-a53 \
        -nographic \
        -kernel linux_image \
        -initrd qemu_merged_rootfs.cpio.gz \
        -serial mon:stdio \
        -netdev user,id=net0,hostfwd=udp::1235-:1235,hostfwd=tcp::8037-:8037 \
        -device virtio-net-device,netdev=net0,mac=52:54:00:12:34:56 \
        -m 1G \
        -enable-kvm

The images are archived under tftp:/tftpboot/maaxboard_terryb/linux_perf for reproducing.

ipbench results

4cores, CPU 1.5GHz:

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,CPU_Utilsation
100000000,99999139,100000139,1472,488,975,4705,263.71,907,91.7
200000000,171305073,199998763,1472,3271,6426,16374,1746.17,5931,98.3

single core, CPU 1GHz:

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,Idle_Cycles,Total_Cycles
100000000,35969018,100000036,1472,12230,14197,16510,851.12,14180,0
200000000,34209477,199999840,1472,12541,15453,18756,1130.50,15438,0

iperf3 results

- - - - - - - - - - - - - - - - - - - - - - - - -bandwidth=100M
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   119 MBytes   100 Mbits/sec  0.000 ms  0/84928 (0%)  sender
[  5]   0.00-10.01  sec   119 MBytes   100 Mbits/sec  0.264 ms  0/84928 (0%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -bandwidth=200M
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   238 MBytes   200 Mbits/sec  0.000 ms  0/169856 (0%)  sender
[  5]   0.00-10.00  sec   235 MBytes   197 Mbits/sec  0.091 ms  2502/169826 (1.5%)  receiver

TODO

• Upgrade the host's kernel version
• Using vhost-user protocol with DPDK?
• Fix CPU Frequency to 1GHz
• Running the host on single core

[terryzbai]

Client VM

Guest: Linux kernel 6.1 built from scratch

ipbench results

Running ipbenchd in client VM and a script listening on the output file to notify VMM with the benchmark status (READY/START/STOP). The status READY is for re-enabling userspace access to PMU counters, which is disabled by the guest Linux at boot time.

Results:

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,CPU_Utilisation
200000000,131072926,199998799,1472,12338,14471,39756,1536.38,14205,99.8

Per-component stats

Utilisation details for VM:
{
KernelUtilisation:  79eba5f2
KernelEntries:  eaa3e
NumberSchedules:  e8191
TotalUtilisation:  2b67555a0
}
{
L1 i-cache misses: 359207751
L1 d-cache misses: 143430776
L1 i-tlb misses: 3342629
L1 d-tlb misses: 8184672
Instructions: 4174951190
Branch mispredictions: 96227493
}
Total utilisation details: 
{
KernelUtilisation: 7815118156
KernelEntries: 6750500
NumberSchedules: 4152282
TotalUtilisation: 25063889433
}
Utilisation details for PD: uart_driver (0)
{
KernelUtilisation: 9089
KernelEntries: 9
NumberSchedules: 9
TotalUtilisation: 47287
}
Utilisation details for PD: serial_virt_tx (1)
{
KernelUtilisation: 15305
KernelEntries: 17
NumberSchedules: 16
TotalUtilisation: 31868
}
Utilisation details for PD: serial_virt_rx (2)
{
KernelUtilisation: 0
KernelEntries: 0
NumberSchedules: 0
TotalUtilisation: 0
}
Utilisation details for PD: ethernet_driver (3)
{
KernelUtilisation: 957237595
KernelEntries: 757768
NumberSchedules: 685351
TotalUtilisation: 2388799598
}
Utilisation details for PD: net_virt_tx (4)
{
KernelUtilisation: 476819466
KernelEntries: 677446
NumberSchedules: 346885
TotalUtilisation: 1015262031
}
Utilisation details for PD: net_virt_rx (5)
{
KernelUtilisation: 734525802
KernelEntries: 917767
NumberSchedules: 556923
TotalUtilisation: 1412588666
}
Utilisation details for PD: client0 (6)
{
KernelUtilisation: 3243357352
KernelEntries: 2820082
NumberSchedules: 1249179
TotalUtilisation: 5335646329
}
Utilisation details for PD: client0_net_copier (7)
{
KernelUtilisation: 357658861
KernelEntries: 616314
NumberSchedules: 363242
TotalUtilisation: 3260399801
}
Utilisation details for PD: timer_driver (8)
{
KernelUtilisation: 0
KernelEntries: 0
NumberSchedules: 0
TotalUtilisation: 0
}

iperf3 results

- - - - - - - - - - - - - - - - - - - - - - - - -bandwith=200M
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   238 MBytes   200 Mbits/sec  0.000 ms  0/169842 (0%)  sender
[  5]   0.00-10.00  sec   238 MBytes   200 Mbits/sec  0.055 ms  120/169842 (0.071%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -bandwith=300M
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   358 MBytes   300 Mbits/sec  0.000 ms  0/254780 (0%)  sender
[  5]   0.00-10.00  sec   218 MBytes   183 Mbits/sec  0.105 ms  99131/254657 (39%)  receiver

[terryzbai]

Passthrough VM

Guest: Linux kernel 6.1 built from scratch

Clock driver needs to be emulated for booting the Linux ethernet driver and prevents the clocks of serial device from being disabled by the guest VM.

ipbench results

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,CPU_Utilisation
100000000,100001537,100000037,1472,169,734,3373,272.36,728,25.6
200000000,200004647,199998647,1472,220,726,2661,212.01,735,50.1
300000000,299993404,300000903,1472,272,793,3813,184.08,799,71.3
400000000,399037949,399999630,1472,391,937,11512,420.57,904,92.8
500000000,432651225,499995548,1472,2843,3726,10405,620.84,3653,99.5
600000000,390206014,600000276,1472,3205,4180,11921,664.28,4070,99.6

iperf3 results

- - - - - - - - - - - - - - - - - - - - - - - - -100Mb/s
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   119 MBytes   100 Mbits/sec  0.000 ms  0/84929 (0%)  sender
[  5]   0.00-10.00  sec   119 MBytes   100 Mbits/sec  0.176 ms  0/84929 (0%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -200Mb/s
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   238 MBytes   200 Mbits/sec  0.000 ms  0/169840 (0%)  sender
[  5]   0.00-10.00  sec   225 MBytes   189 Mbits/sec  0.075 ms  9285/169840 (5.5%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -300Mb/s
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   358 MBytes   300 Mbits/sec  0.000 ms  0/254783 (0%)  sender
[  5]   0.00-10.00  sec   308 MBytes   258 Mbits/sec  0.067 ms  35307/254783 (14%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -400Mb/s
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   477 MBytes   400 Mbits/sec  0.000 ms  0/339710 (0%)  sender
[  5]   0.00-10.00  sec   365 MBytes   306 Mbits/sec  0.026 ms  79597/339710 (23%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -500Mb/s
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   596 MBytes   500 Mbits/sec  0.000 ms  0/424637 (0%)  sender
[  5]   0.00-10.00  sec   408 MBytes   342 Mbits/sec  0.025 ms  133892/424637 (32%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -600Mb/s
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   715 MBytes   600 Mbits/sec  0.000 ms  0/509563 (0%)  sender
[  5]   0.00-10.00  sec   440 MBytes   369 Mbits/sec  0.014 ms  196292/509563 (39%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -700Mb/s
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   835 MBytes   700 Mbits/sec  0.000 ms  0/594489 (0%)  sender
[  5]   0.00-10.00  sec   481 MBytes   404 Mbits/sec  0.009 ms  251688/594489 (42%)  receiver
- - - - - - - - - - - - - - - - - - - - - - - - -800Mb/s
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   954 MBytes   800 Mbits/sec  0.000 ms  0/679417 (0%)  sender
[  5]   0.00-10.00  sec   399 MBytes   335 Mbits/sec  0.012 ms  394620/679144 (58%)  receiver

[terryzbai]

Pure Linux on physical board

Image: Linux kernel 6.1 built from scratch
CPU Frequency: 1GHz
Cores: single-core
Scheduler: Round-robin Real-time (see TS Wiki - Benchmarking on Linux)

ipbench results

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,CPU_Utilisation
10000000,10000159,10000159,1472,328,774,1482,315.10,629,8.9
20000000,20000234,20000234,1472,265,809,1508,281.72,675,16.4
50000000,50000298,50000298,1472,152,559,1363,270.98,472,32.0
100000000,100000675,100000175,1472,167,774,1374,283.20,791,38.9
200000000,200001400,199999399,1472,170,793,1353,225.26,792,58.1
300000000,300008020,300003520,1472,225,793,1191,180.28,795,70.7
400000000,399999268,400001268,1472,370,857,1241,142.26,861,85.8
500000000,500019876,499997373,1472,705,981,1254,111.79,981,98.6
600000000,490624080,600002218,1472,6085,6591,13983,1290.03,6295,96.0
700000000,490705612,699996552,1472,6248,6672,14104,1264.80,6356,95.9
800000000,490253287,800002635,1472,6346,6744,14183,1262.98,6426,96.1
900000000,490145457,899999193,1472,6357,6798,14299,1268.34,6478,96.0
1000000000,491987023,999995504,1472,6934,7393,14869,1279.85,7081,95.9

iperf3 results

[terryzbai]

DPDK on QEMU/KVM

Host Image: Linux kernel 6.1.55 built from scratch (NXP repo)
Guest Image: Linux kernel 6.1 built from scratch
DPDK: v25.03
CPU Frequency: 1GHz
Cores: single-core

ipbench results

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,Idle_Cycles,Total_Cycles
100000000,63264462,100000064,1472,5637,13754,23947,2660.75,13355,0
200000000,59382830,199999875,1472,7882,15043,28703,2492.18,14262,0

[terryzbai]

DPDK on pure Linux

Host Image: Linux kernel 6.1.55 built from scratch (NXP repo)
DPDK: v25.03
CPU Frequency: 1GHz
Cores: single-core

ipbench results

CPU frequency: 1GHz

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,Idle_Cycles,Total_Cycles
10000000,10000184,10000184,1472,15,126,220,20.45,129,0
20000000,20000344,20000344,1472,48,125,247,20.42,115,0
50000000,50000671,50000671,1472,39,119,639,15.23,113,0
100000000,100000751,100000751,1472,39,107,218,3.74,108,0
200000000,200003000,199999000,1472,9,108,224,5.20,109,0
300000000,300000779,300000779,1472,52,107,185,5.57,109,0
400000000,400003651,399997650,1472,70,118,184,13.19,113,0
500000000,500008198,499998198,1472,64,113,195,6.40,115,0
600000000,593394477,600002191,1472,2589,2709,2866,32.28,2709,0
700000000,593105541,699997982,1472,2623,2710,2848,24.10,2710,0
800000000,593118122,800005570,1472,2640,2748,2895,41.98,2746,0
900000000,593180109,900002472,1472,2639,2773,3150,56.16,2771,0
1000000000,592548172,999991433,1472,3179,3359,3736,63.84,3359,0

CPU frequency: 1.5GHz

Requested_Throughput,Receive_Throughput,Send_Throughput,Packet_Size,Minimum_RTT,Average_RTT,Maximum_RTT,Stdev_RTT,Median_RTT,Idle_Cycles,Total_Cycles
600000000,600003908,600003908,1472,31,102,320,6.40,103,0
700000000,700009136,699995136,1472,75,107,311,5.74,105,0
800000000,800002308,800006308,1472,69,111,525,12.41,109,0
900000000,824010714,900000826,1472,1903,1962,2472,15.43,1961,0
1000000000,829783752,999990213,1472,2455,2535,3022,18.11,2534,0