Thanks for your work, can I use "Qwen/Qwen2.5-32B-Instruct" instead of gpt-4o-mini for evaluation? If yes, what codes need to be changed?

[LiZhangMing]

Thanks for your work, can I use "Qwen/Qwen2.5-32B-Instruct" instead of gpt-4o-mini for evaluation? If yes, what codes need to be changed?

[Muennighoff]

yes you can change the eval code to do that; it its pretty cheap with gpt-4o-mini though / you can use gpt-4.1-nano

[LiZhangMing]

yes you can change the eval code to do that; it its pretty cheap with gpt-4o-mini though / you can use gpt-4.1-nano

Thanks, when I use the code as follow :" CUDA_VISIBLE_DEVICES=0,1,2,3 OPENAI_API_KEY=XXXXXXXXXXXXXX PROCESSOR=gpt-4o-mini lm_eval --model vllm --model_args pretrained=/home/lizhangming/lzm_project/s1/ckpts/s1_20250717_014215/merge,tokenizer=/home/lizhangming/lzm_project/s1/ckpts/s1_20250717_014215/merge,dtype=float32,tensor_parallel_size=4 --tasks aime25_nofigures --batch_size auto --apply_chat_template --output_path results/aime24_nofigures_full_fintune --log_samples --gen_kwargs "max_gen_toks=32768,max_tokens_thinking=auto,thinking_n_ignore=2,thinking_n_ignore_str=Wait""。

When using gpt4o-mini, the terminal keeps prompting that the number of requests is too many and the request is being re-requested. Have you ever encountered this problem? How did you solve it?

[Muennighoff]

Hmm maybe your API key has low limits