Benchmark Dell Pro Max with GB10 (Nvidia Spark OEM version)

[geerlingguy]

See: geerlingguy/sbc-reviews#92

To get llama.cpp to compile with CUDA, I had to specify the CUDA compiler path:

cmake -B build -DGGML_CUDA=1 -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc

I also got the error:

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes
MESA: error: Opening /dev/dri/card0 failed: Permission denied
TU: error: ../src/freedreno/vulkan/tu_knl.cc:385: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA Tegra NVIDIA GB10 (NVIDIA) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 0 | matrix cores: KHR_coopmat
...

So I added my user account to the video and render groups:

sudo usermod -aG video,render jgeerling

[geerlingguy]

SINGLE NODE RESULTS

All tests run with build: 028f93ef9 (7129), built as shown above.

tinyllama-1.1b-1t-openorca.Q4_K_M.gguf

model	size	params	backend	ngl	test	t/s
llama 1B Q4_K - Medium	636.18 MiB	1.10 B	CUDA,Vulkan	99	pp512	10736.67 ± 73.01
llama 1B Q4_K - Medium	636.18 MiB	1.10 B	CUDA,Vulkan	99	pp4096	4686.03 ± 5.29
llama 1B Q4_K - Medium	636.18 MiB	1.10 B	CUDA,Vulkan	99	tg128	264.76 ± 1.22
llama 1B Q4_K - Medium	636.18 MiB	1.10 B	CUDA,Vulkan	99	pp4096+tg128	2850.36 ± 0.79

Power consumption: 116W

Llama-3.2-3B-Instruct-Q4_K_M.gguf

model	size	params	backend	ngl	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	CUDA,Vulkan	99	pp512	5972.32 ± 40.32
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	CUDA,Vulkan	99	pp4096	3484.70 ± 6.40
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	CUDA,Vulkan	99	tg128	97.93 ± 0.95
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	CUDA,Vulkan	99	pp4096+tg128	1542.26 ± 1.44

Power consumption: 128.5W

llama-2-13b.Q4_K_M.gguf

model	size	params	backend	ngl	test	t/s
llama 13B Q4_K - Medium	7.33 GiB	13.02 B	CUDA,Vulkan	99	pp512	1567.99 ± 2.77
llama 13B Q4_K - Medium	7.33 GiB	13.02 B	CUDA,Vulkan	99	pp4096	1040.27 ± 2.79
llama 13B Q4_K - Medium	7.33 GiB	13.02 B	CUDA,Vulkan	99	tg128	26.11 ± 0.03
llama 13B Q4_K - Medium	7.33 GiB	13.02 B	CUDA,Vulkan	99	pp4096+tg128	391.64 ± 0.21

Power consumption: 146.4W

DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf

model	size	params	backend	ngl	test	t/s
qwen2 14B Q4_K - Medium	8.37 GiB	14.77 B	CUDA,Vulkan	99	pp512	1462.06 ± 5.14
qwen2 14B Q4_K - Medium	8.37 GiB	14.77 B	CUDA,Vulkan	99	pp4096	965.01 ± 0.61
qwen2 14B Q4_K - Medium	8.37 GiB	14.77 B	CUDA,Vulkan	99	tg128	23.92 ± 0.01
qwen2 14B Q4_K - Medium	8.37 GiB	14.77 B	CUDA,Vulkan	99	pp4096+tg128	407.31 ± 0.41

Power consumption: 138.4W

Qwen3-30B-A3B-Q4_K_M.gguf

model	size	params	backend	ngl	test	t/s
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	CUDA,Vulkan	99	pp512	629.93 ± 0.30
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	CUDA,Vulkan	99	pp4096	431.52 ± 0.11
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	CUDA,Vulkan	99	tg128	10.46 ± 0.00
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	CUDA,Vulkan	99	pp4096+tg128	184.79 ± 0.32

Power consumption: 143.6W

Llama-3.1-70B-Instruct-Q4_K_M.gguf

model	size	params	backend	ngl	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	CUDA,Vulkan	99	pp512	276.17 ± 5.37
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	CUDA,Vulkan	99	pp4096	223.82 ± 0.30
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	CUDA,Vulkan	99	tg128	4.71 ± 0.00
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	CUDA,Vulkan	99	pp4096+tg128	89.70 ± 0.05

Power consumption: 154.9W

Llama-3.3-70B-Instruct-Q4_K_M.gguf

model	size	params	backend	ngl	test	t/s
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	CUDA,Vulkan	99	pp512	284.07 ± 0.56
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	CUDA,Vulkan	99	pp4096	225.31 ± 0.52
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	CUDA,Vulkan	99	tg128	4.72 ± 0.00
llama 70B Q4_K - Medium	39.59 GiB	70.55 B	CUDA,Vulkan	99	pp4096+tg128	90.48 ± 0.00

Power consumption: 156W

gpt-oss-20b-Q4_K_M.gguf

model	size	params	backend	ngl	test	t/s
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	CUDA,Vulkan	99	pp512	2725.32 ± 20.31
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	CUDA,Vulkan	99	pp4096	1919.12 ± 3.70
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	CUDA,Vulkan	99	tg128	87.97 ± 0.26
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	CUDA,Vulkan	99	pp4096+tg128	1142.92 ± 1.50

Power consumption: 122.8W

gpt-oss-120b-Q4_K_M-00001-of-00002.gguf

model	size	params	backend	ngl	test	t/s
gpt-oss 120B Q4_K - Medium	58.45 GiB	116.83 B	CUDA,Vulkan	99	pp512	990.96 ± 23.22
gpt-oss 120B Q4_K - Medium	58.45 GiB	116.83 B	CUDA,Vulkan	99	pp4096	865.14 ± 0.48
gpt-oss 120B Q4_K - Medium	58.45 GiB	116.83 B	CUDA,Vulkan	99	tg128	52.86 ± 0.07
gpt-oss 120B Q4_K - Medium	58.45 GiB	116.83 B	CUDA,Vulkan	99	pp4096+tg128	591.09 ± 0.56

Power consumption: 137.1W

[geerlingguy]

CLUSTER RESULTS

llama.cpp RPC

I'm going to first test out llama.cpp RPC mode, as @eugr did here: https://community.frame.work/t/dgx-spark-vs-strix-halo-initial-impressions/77055/39?u=geerlingguy

e.g.:

build/bin/llama-bench -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf --rpc 10.0.2.225:50052 -fa 1 -n 128 -p 512,4096 -pg 4096,128 -ngl 99 -r 2 -mmp 0

I will work on testing this 2-node cluster under this issue on my Beowulf AI Cluster project:

geerlingguy/beowulf-ai-cluster#15

[eugr]

Any reason you are using gpt-oss-120b in Q5_K_M and not native MXFP4? I don't see any benefit?
BTW, here is MXFP4 benchmark on my Spark:

model	size	params	backend	test	t/s
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	pp2048	1932.64 ± 10.54
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	tg32	60.36 ± 0.18
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	pp2048 @ d4096	1845.57 ± 9.27
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	tg32 @ d4096	56.84 ± 0.42
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	pp2048 @ d8192	1758.40 ± 7.59
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	tg32 @ d8192	53.20 ± 0.46
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	pp2048 @ d16384	1549.70 ± 5.52
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	tg32 @ d16384	49.15 ± 0.25
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	pp2048 @ d32768	1245.74 ± 5.48
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	tg32 @ d32768	42.41 ± 0.21

build: 583cb8341 (7157)

[geerlingguy]

@eugr - I was actually using gpt-oss-120b-Q4_K_M-00001-of-00002.gguf, no clue how I copy-pasted Q5 for the heading there. The llama-bench result output showed gpt-oss 120B Q4_K - Medium which is correct.

I'm using bartowski's version, is there any particular reason to choose MXFP4_MOE vs Q4_K_M?

[eugr]

@geerlingguy - MXFP4 is a native quant - the one the model was originally post-trained with, I don't see any reason to re-quantize it. It also gives better speeds on both Spark and Strix Halo.