Reproduce of mvbench

[yuxwind]

Hi Cosmos3 authors,

Thank you for the great work and for releasing the Cosmos3 models and code!

I am trying to reproduce the reported MVBench performance for nvidia/Cosmos3-Nano (reasoner). However, my result is much lower than the number reported in the paper:

• Reported MVBench: 73.2%
• My reproduction: 62.0%

My evaluation configuration is:

Param	Value
model	nvidia/Cosmos3-Nano (reasoner)
thinking	on (temp=0.6, top_p=0.95, top_k=20, seed=1234)
max_new_tokens	8192
max_time	180 s/sample hard cap
attn	sdpa, bf16
fps	2.0
max_frames	128
max_tokens_per_frame	768
frame rounding	even (FRAME_FACTOR=2)

Could you please clarify the exact evaluation configuration used for the MVBench result in the paper?

In particular, I would like to ask:

1. Was the reported MVBench result evaluated with thinking enabled or disabled?
2. What FPS, maximum number of frames, and maximum tokens per frame were used?
3. Was there any benchmark-specific prompt template or answer extraction rule?
4. Were any special decoding settings used, such as greedy decoding vs. sampling?
5. Is max_frames=128 too small for reproducing the reported MVBench result?

Any suggestions on the correct configuration would be greatly appreciated.

Thank you very much!

Best,
Xin