| Model | MMLU -Redux |
ZebraLogic | CRUX | MATH -L5 |
Average |
|---|---|---|---|---|---|
| o1-preview-2024-09-12 | 92.84 | 71.40 | 93.50 | 84.47 | 85.55 |
| o1-mini-2024-09-12 | 86.72 | 52.60 | 91.38 | 89.32 | 80.00 |
| gemini-1.5-pro-exp-0827 | 86.14 | 30.50 | 77.62 | 68.10 | 65.59 |
| gpt-4o-2024-08-06 | 88.26 | 31.70 | 85.00 | 55.34 | 65.08 |
| chatgpt-4o-latest-24-09-07 | 88.88 | 29.90 | 84.25 | 53.12 | 64.04 |
| gpt-4o-2024-05-13 | 88.01 | 28.20 | 83.62 | 54.79 | 63.66 |
| claude-3-5-sonnet-20240620 | 86.00 | 33.40 | 78.75 | 51.87 | 62.51 |
| Qwen2.5-72B-Instruct | 85.57 | 26.60 | 71.88 | 60.19 | 61.06 |
| Llama-3.1-405B-Inst@sambanova | 86.21 | 30.10 | 71.25 | 49.79 | 59.34 |
| gpt-4-turbo-2024-04-09 | 85.31 | 28.40 | 76.75 | 46.46 | 59.23 |
| gemini-1.5-flash-exp-0827 | 82.11 | 25.00 | 72.50 | 54.51 | 58.53 |
| Mistral-Large-2 | 82.97 | 29.00 | 72.88 | 48.54 | 58.35 |
| gpt-4o-mini-2024-07-18 | 81.50 | 20.10 | 73.50 | 52.15 | 56.81 |
| deepseek-v2.5-0908 | 80.35 | 22.10 | 68.12 | 44.66 | 53.81 |
| claude-3-opus-20240229 | 82.54 | 27.00 | 68.62 | 36.89 | 53.76 |
| Meta-Llama-3.1-70B-Instruct | 82.97 | 24.90 | 62.62 | 43.13 | 53.41 |
| gemini-1.5-pro | 82.76 | 19.40 | 66.25 | 39.81 | 52.05 |
| gpt-4-0314 | 81.64 | 27.10 | 72.38 | 26.07 | 51.80 |
| Qwen2-72B-Instruct | 81.61 | 21.40 | 57.38 | 38.28 | 49.67 |
| gemini-1.5-flash | 77.36 | 19.40 | 61.88 | 34.81 | 48.36 |
| Qwen2.5-7B-Instruct | 75.13 | 12.00 | 51.25 | 51.46 | 47.46 |
| Meta-Llama-3-70B-Instruct | 78.01 | 16.80 | 57.12 | 25.10 | 44.26 |
| gemma-2-27b-it | 75.67 | 16.30 | 55.88 | 26.63 | 43.62 |
| Athene-70B | 76.64 | 16.70 | 49.75 | 20.67 | 40.94 |
| reka-core-20240501 | 76.42 | 13.00 | 45.00 | 21.91 | 39.08 |
| claude-3-haiku-20240307 | 72.32 | 14.30 | 53.62 | 15.12 | 38.84 |
| gemma-2-9b-it | 72.82 | 12.80 | 44.88 | 19.42 | 37.48 |
| gpt-3.5-turbo-0125 | 68.36 | 10.10 | 53.25 | 13.73 | 36.36 |
| Yi-1.5-34B-Chat | 72.79 | 11.50 | 42.88 | 18.17 | 36.34 |
| Phi-3-mini-4k-instruct | 70.34 | 11.60 | 43.50 | 16.23 | 35.42 |
| Meta-Llama-3.1-8B-Instruct | 67.24 | 12.80 | 38.75 | 22.19 | 35.24 |
| Qwen2-7B-Instruct | 66.92 | 8.40 | 36.75 | 23.86 | 33.98 |
| Phi-3.5-mini-instruct | 67.67 | 6.40 | 40.88 | 18.72 | 33.42 |
| Yi-1.5-9B-Chat | 65.05 | 2.30 | 43.75 | 19.97 | 32.77 |
| Qwen2.5-3B-Instruct | 64.25 | 4.80 | 32.12 | 25.52 | 31.67 |
| Meta-Llama-3-8B-Instruct | 61.66 | 11.90 | 36.62 | 7.91 | 29.52 |
| gemma-2-2b-it | 51.94 | 4.20 | 20.75 | 4.30 | 20.30 |