[whatmodel.sh](../../README.md) / Mistral Small 3.1 24B (2503)
# Mistral Small 3.1 24B (2503)
Mistral · released 2025-03 · 24.01B params · apache-2.0
**score 54.7** ±0.9 · #288 of 732 · from 28 benchmark results across 13 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Mistral Small 3.1 24B (2503) among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃▆●▄▅▅▄▄▂▁ #231 of 313 5.6% 3
coding not ranked yet — 2
chat & writing ▁▁▁▃▆▄▆●▆▇▇▃ #228 of 402* 11% 2
math ▁▁▃▅●▆▄█▄▂▁▁ #276 of 431 26% 4
vision & documents ▁▁▂▂▁▃●▂▆▇█▄ #111 of 153 14% 5
web search ▁▁▁▅▄█▃●·▁·▁ #12 of 84* 2.6% 1
agent tasks · #231 of 313 · 5.6% of its score
Beats [o1 (2024-12-17)](o1-2024-12-17.md), [o3-mini](o3-mini.md), [DeepSeek R1 (Jan '25)](deepseek-r1.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 10.8% █·▃▃·●▅▅▅▅▃▅ beats 5/18 very hard · avg 5.5% 2.6% sharp
BFCL v3 57.7% ▁▁▁▁▁▃▅●▅▄▁▁ beats 72/114 medium · avg 51% 1.7% blunt · at its level
Galileo Agent Leaderboard v1 (too… 91.2% ▃▃▃▃▆▅▆▅●▂▅▂ beats 26/31 easy · avg 78% 1.3% typical
coding · 0.0% of its score
test result vs the others in its run counts why
MBPP+ pass@5 (maker) 74.6% ●··········█ beats 0/1 — not counted yet: no other model in the …
HumanEval+ pass@5 (maker) 89.0% ●··········█ beats 0/1 — not counted yet: no other model in the …
chat & writing · #228 of 402 · 11% of its score
Beats [Granite 4.2 8B](granite-4-2-8b.md), [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md), [Olmo 3.1 32B Think](olmo-3-1-32b-think.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [Gemma 3 4B](gemma-3-4b.md), [Olmo 3 7B Instruct](olmo-3-7b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 1302.6 ▁▁▁▂▃▃▃●▆▇█▃ beats 128/385 avg 1260 7.7% blunt
Arena-Hard v2 aggregate (maker) 19.6% ▁●··▄▃▁▃█▁▁▇ beats 1/22 hard · avg 21% 3.0% sharp · at its level · small run
math · #276 of 431 · 26% of its score
Beats [GPT-4 Turbo (2024-04-09)](gpt-4-turbo-2024-04-09.md), [Molmo 2 8B](molmo-2-8b.md), [Llama 3.3 70B Instruct](llama-3-3-70b.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 46.8% ▂▂▅▆▄●▆▃▅▂▃▄ beats 40/92 hard · avg 28% 17% sharp · at its level
OTIS mock AIME 2024–2025 3.9% ▃▃●▃▃▃▇▇█▃▂▅ beats 25/197 very hard · avg 7.2% 3.9% sharp · noisy
MATH (subset unspecified) (maker) 69.3% ▂▄▄▃█●▅▆▄▃▄▁ beats 41/80 medium · avg 60% 3.2% at its level · small run
MathVista (testmini) (maker) 68.9% ▁··▁▄▄▆●▄▅█▄ beats 24/48 medium · avg 62% 2.2% at its level · small run
vision & documents · #111 of 153 · 14% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-4o (2024-08-06)](gpt-4o-2024-08-06.md), [GPT-4 Turbo (2024-04-09)](gpt-4-turbo-2024-04-09.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
Arena vision (style controlled) 1127.4 ▁▁▂▂▂▂●▄▅▅█▇ beats 38/152 avg 1061 5.1% blunt
MMMU (val) (maker) 64.0% ▁····▁▁▃█●▆▅ beats 27/53 medium · avg 56% 4.3% at its level · small run · several runs
MathVista (testmini) (maker) 68.9% ▁··▁▄▄▆●▄▅█▄ beats 24/48 medium · avg 62% 2.2% at its level · small run
ChartQA (maker) 86.2% ▁··▁▁▁▁▁▂▅●█ beats 27/44 easy · avg 81% 1.4% blunt · small run · several runs
DocVQA (maker) 94.1% ▁··▃▁▁▁▇●▇▄▅ beats 12/28 very easy · avg 93% 0.8% blunt · maxed out · small run
web search · #12 of 84 · 2.6% of its score
Beats [Claude Opus 5.5](claude-opus-5-5.md), [Claude Fable 5.1](claude-fable-5-1.md), [Claude Sonnet 5.5](claude-sonnet-5-5.md) here, though they rank higher overall. Trails [GPT-4o mini (2024-07-18)](gpt-4o-mini-2024-07-18.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 10.8% █·▃▃·●▅▅▅▅▃▅ beats 5/18 very hard · avg 5.5% 2.6% sharp
other tests (14)
test result vs the others in its run difficulty counts why
GPQA Diamond 41.9% ▁▁▁▁▂▄●▄▅▇█▅ beats 44/211 hard · avg 39% 17% sharp · at its level · several runs
CPTUB average (SpeakLeash, 0-5 ju… 78.0% ▁▂▂▂▃█▃▆●▄▂▃ beats 75/95 medium · avg 65% 5.1% at its level
EQ-Bench 3 (emotional intelligenc… 1055.9 ▁·▂●▄▅▅▃█▄▁▁ beats 5/62 avg 1034 4.7% typical
BRIDGE clinical text, zero-shot a… 39.7% ▁▁▁▁▁▂▂▃▄▆●▃ beats 68/97 hard · avg 31% 4.6% blunt · at its level
MMLU (maker) 80.6% ▁▁▁▁▁▂▂▅██●▄ beats 170/214 medium · avg 72% 3.6% at its level · small run
MMLU-Pro (maker) 66.8% ▁▁·▁▁▁▃▂●▅█▁ beats 78/188 medium · avg 57% 2.8% at its level · small run
SimpleQA (maker) 10.4% ▄▄▄▆●▆███·▂█ beats 12/32 very hard · avg 7.4% 2.6% small run
DiagnosticIQ industrial maintenan… 61.2% ▁··▃▁▁▁▅●▅▁▅ beats 10/24 hard · avg 51% 1.6% blunt · at its level
EuroEval HellaSwag (English), tes… 93.1% ▃·▁▁▂▁▂▅█▇▄● beats 99/100 medium · avg 67% 1.3% sharp · at its level · noisy
EuroEval Life in the UK (English … 94.4% ▁▁▁▁▁▁▁▂▄█▄● beats 139/146 easy · avg 82% 0.7% noisy
Chess Puzzles (Epoch) 1.0% ●▁▁▁▁▂▃▃▃▂▁▁ beats 4/137 very hard · avg 5.8% 0.6% sharp · too hard for it · noisy
Vectara hallucination leaderboard… 94.4% ▁▁▂▃▄●█▆▃▂▂▁ beats 52/140 very easy · avg 94% 0.4% blunt · maxed out · noisy
WildBench v2 (maker) 55.6% ●··········█ beats 0/1 — not counted yet: no other model in the …
GPQA Main (maker) 44.4% █··········● beats 1/1 — not counted yet: no other model in the …
source results share of the score
Epoch AI 4 33% ▇▇▇▇
mistralai/Mistral-Small-3.2-24B-Instruc… 9 25% ▇▇▇
LMArena leaderboard dataset 2 13% ▇▇
SpeakLeash CPTUB 1 5% ▇
Mistral AI vendor-reported benchmark sc… 3 5% ▇
EQ-Bench 3 canonical leaderboard results 1 5% ▇
BRIDGE Medical Leaderboard 1 5% ▇
Finance Agent Benchmark 1 3% ▇
EuroEval leaderboards 2 2% ▇
Berkeley Function Calling Leaderboard r… 1 2% ▇
DiagnosticIQ: A Benchmark for LLM-Based… 1 2% ▇
Galileo Agent Leaderboard 1 1% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 54.7
Licences: Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · mistralai/Mistral-Small-3.2-24B-Instruct-2506 model card ([Apache-2.0](../../licenses/mistralai--Mistral-Small-3.2-24B-Instruct-2506.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · SpeakLeash CPTUB ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · EQ-Bench 3 canonical leaderboard results ([MIT](../../licenses/eqbench3-MIT.txt)) · BRIDGE Medical Leaderboard ([MIT](../../licenses/bridge-medical-leaderboard-MIT.txt)) · Finance Agent Benchmark ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · EuroEval leaderboards ([MIT](https://github.com/EuroEval/leaderboards/blob/1a77f1d2f8ba1ba05ab73287fb090d76bf41b2e0/LICENSE)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Galileo Agent Leaderboard ([Apache-2.0](../../licenses/galileo-agent-leaderboard-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
Not counted (4)
source test score why
mistralai/Mistral-Sma… WildBench v2 55.6% too few models have it yet
mistralai/Mistral-Sma… GPQA Main 44.4% too few models have it yet
mistralai/Mistral-Sma… MBPP+ pass@5 74.6% too few models have it yet
mistralai/Mistral-Sma… HumanEval+ pass@5 89.0% too few models have it yet