[whatmodel.sh](../../README.md) / o4-mini
# o4-mini
OpenAI · released 2025-04 · closed weights
**score 65.7** ±0.6 · #132 of 732 · from 54 benchmark results across 18 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks o4-mini among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃▆█●▅▅▄▄▂▁ #161 of 313 17% 6
coding ▁▁▁▃▅▇█▇●▄▃▁ #91 of 469 13% 5
chat & writing ▁▁▁▃▆▄▆█▆●▇▃ #121 of 402 10% 5
math ▁▁▃▅█▆▄●▄▂▁▁ #82 of 431 23% 8
vision & documents ▁▁▂▂▁▃▄▂●▇█▄ #79 of 153* 2.1% 1
web search ▁▁▁▅▄█▃▄·▁·● #2 of 84* 2.5% 1
agent tasks · #161 of 313 · 17% of its score
Beats [o1 (2024-12-17)](o1-2024-12-17.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [Jamba 1.5 Large](jamba-1-5-large.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
τ²-bench Telecom (maker) 40.5% ▃·▅●▃▃··█▃▃▅ beats 3/13 very hard · avg 7.7% 7.8% sharp · at its level · several runs
τ²-bench Retail 68.3% ▁····▂▁▃▆●▃▁ beats 27/44 medium · avg 40% 2.8% at its level · several runs
Finance Agent Benchmark v1 (537 q… 37.3% █·▃▃·█▅▅▅▅●▅ beats 16/18 very hard · avg 5.5% 2.5% sharp · at its level
τ²-bench Airline 52.1% ▁▁·▂▁▃▂▁█●▂▃ beats 28/42 hard · avg 28% 2.2% at its level · several runs
BFCL V4 (December 2025 archive) 53.2% ▁▁▁▂██▅▅●▂▂▂ beats 61/77 hard · avg 31% 1.5% at its level
Galileo Agent Leaderboard v1 (too… 88.9% ▃▃▃▃▆▅▆●█▂▅▂ beats 20/31 easy · avg 78% 0.3% typical
coding · #91 of 469 · 13% of its score
Beats [Grok 4.20 (0309 reasoning)](grok-4-20.md), [GPT-5.4 mini](gpt-5-4-mini.md), [GPT-5.1](gpt-5-1.md) here, though they rank higher overall. Trails [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Aider polyglot (225 exercises, tw… 72.0% ▂·▂▃▅▂▃█▆●▂▂ beats 41/49 very hard · avg 7.6% 7.7% sharp · at its level
LiveCodeBench official (Aug 1, 20… 80.2% ▃▃▅▃··▃▅█▅▅● beats 15/15 hard · avg 24% 2.0% sharp · at its level
WebApp1K-Duo-React pass@1 (onekq-… 72.6% ▁··▁·▁▁▁▆█●█ beats 34/45 hard · avg 29% 1.7% at its level
LiveCodeBench v6 (Feb–May 2025) 71.8% ▁▁▃▁█▅▄●▅▅▃█ beats 22/38 medium · avg 36% 0.9% at its level
HumanEval (simple-evals) (maker) 99.3% ▆█▆▆▂··▂···● beats 15/15 easy · avg 83% 0.6% maxed out
chat & writing · #121 of 402 · 10% of its score
Beats [LongCat 2.0 (1.6T-A48B)](longcat-2-0.md), [Step 3.5 Flash (196B-A11B)](step-3-5-flash.md), [GPT-5 mini](gpt-5-mini.md) here, though they rank higher overall. Trails [LFM2.5 2.6B](lfm2-5-2-6b.md), [Nemotron-Cascade-8B](nemotron-cascade-8b.md), [Mistral Large 3 (675B-A41B)](mistral-large-3.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena-Hard v2.0 hard prompts (sty… 79.1% ▂·▃▅▅·▃█▆▂●▂ beats 21/22 very hard · avg 12% 3.6% sharp · at its level
LMArena (text) 1391.0 ▁▁▁▂▃▃▃▆▆●█▃ beats 235/385 avg 1260 3.2% blunt
Arena-Hard v2 aggregate 59.3% ▁▁··▄▃▁●█▁▁▇ beats 9/22 hard · avg 28% 1.8% sharp · at its level
Arena-Hard v2.0 creative writing … 58.7% ▆▂▂▆▂▂▃●▂▃·▂ beats 14/22 hard · avg 19% 0.8% sharp · at its level · noisy
IFEval (aggregation unspecified) 92.4% ▁▂▄▂▂▄▆██●▂▁ beats 128/145 easy · avg 77% 0.7% typical
math · #82 of 431 · 23% of its score
Beats [Claude Sonnet 4.6](claude-sonnet-4-6.md), [Claude Opus 4.5](claude-opus-4-5.md), [Qwen3.5 397B-A17B](qwen3-5-397b-a17b.md) here, though they rank higher overall. Trails [Granite 4.2 30B](granite-4-2-30b.md), [Qwen3-Next 80B-A3B Instruct](qwen3-next-80b-a3b-instruct.md), [EXAONE 4.5 33B](exaone-4-5-33b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
FrontierMath tiers 1–3 v2 36.1% ▂··▁▁▃▄●█▆▄▃ beats 26/86 very hard · avg 1.2% 11% sharp
FrontierMath tier 4 v2 4.9% ▁▂●▂█▅▂▃▂▁▁▁ beats 9/67 very hard · avg 0.1% 2.7% sharp · too hard for it
HMMT 2025 (edition unspecified) 66.7% ▃█▇●▇▅▁·▁··▁ beats 18/32 hard · avg 28% 2.3% sharp
MATH level 5 97.8% ▂▂▅▆▄█▆▃▅▂▃● beats 90/92 hard · avg 28% 2.2% sharp
OTIS mock AIME 2024–2025 81.7% ▃▃▄▃▃▃▇●█▃▂▅ beats 117/197 very hard · avg 7.2% 2.0% sharp · at its level · noisy
MATH-500 (simple-evals) (maker) 98.2% ▃··▅▅▃···▃·● beats 9/9 medium · avg 61% 1.3% sharp · maxed out
PolyMATH 48.7% ▁···▃▁·▇●▄▄▄ beats 10/23 hard · avg 28% 0.8% at its level
MGSM (simple-evals) (maker) 93.5% ▁····▁▅▁█▁▃● beats 16/16 easy · avg 80% 0.8% typical
vision & documents · #79 of 153 · 2.1% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-5.4 nano](gpt-5-4-nano.md), [GPT-5 mini](gpt-5-mini.md) here, though they rank higher overall. Trails [Gemma 4 12B](gemma-4-12b.md), [Qwen3.5 9B](qwen3-5-9b.md), [GPT-4.1](gpt-4-1.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena vision (style controlled) 1201.0 ▁▁▂▂▂▂▃▄●▅█▇ beats 75/152 avg 1061 2.1% blunt
web search · #2 of 84 · 2.5% of its score
Beats [GPT-6 Astra](gpt-6-astra.md), [Claude Opus 5.5](claude-opus-5-5.md), [GPT-6.1 Sol](gpt-6-1-sol.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 37.3% █·▃▃·█▅▅▅▅●▅ beats 16/18 very hard · avg 5.5% 2.5% sharp · at its level
other tests (25)
test result vs the others in its run difficulty counts why
GPQA Diamond (simple-evals) (make… 81.3% █▃▃█·▃▃▅▃▃▃● beats 15/16 hard · avg 39% 5.3% sharp · at its level
GPQA Diamond 79.6% ▁▁▁▁▂▄▆▄▅●█▅ beats 126/211 hard · avg 39% 4.6% sharp · at its level
KCL-Essay vanilla 31.3% ▁···▁▃▂●▂▃▃▂ beats 9/23 hard · avg 17% 2.5% at its level
EQ-Bench 3 (emotional intelligenc… 1224.1 ▁·▂▁▄▅▅●█▄▁▁ beats 33/62 avg 1034 1.9% typical
AraGen-v3 3C3H score (Inception, … 70.6% ▁▁·▁▂▂▄█▄▅●▄ beats 56/66 hard · avg 29% 1.7% at its level
MMLU (simple-evals, zero-shot CoT… 90.3% ▂▂·█▄▄▂·●▂·▂ beats 12/16 easy · avg 79% 1.5% typical
INCLUDE 80.8% ▂····▂▂▄▄▄●▄ beats 30/36 medium · avg 59% 1.5% typical
MMLU-Pro 81.9% ▁▁·▁▁▁▃▂▄▅●▁ beats 141/188 medium · avg 58% 1.4% typical
MMLU-Redux 92.8% ▁▁▃▁▂▃▄▅▇▅●▄ beats 50/63 easy · avg 81% 1.3% maxed out
KCL-MCQA vanilla 37.7% ▂···█▂▆●██▆▂ beats 10/23 very hard · avg 29% 1.3% at its level
Mystery Game Puzzles (Epoch) 5.0% ●··▂▃▇▄█▂▄▂▁ beats 14/77 very hard · avg 9.4% 1.3% sharp · too hard for it · noisy
Chess Puzzles (Epoch) 26.0% █▁▁▁▁▂▃●▃▂▁▁ beats 98/137 very hard · avg 5.8% 1.2% sharp · noisy
HindiGen-v1 3C3H score (Inception… 75.5% ▁▁▁▁·▂·█▆█●▁ beats 28/31 medium · avg 46% 1.2% typical
OJBench 33.3% ▁···▁▃·▅▄▅●█ beats 19/26 very hard · avg 11% 1.1% at its level
SimpleQA Verified 18.8% ▁▁▃●▃▃▆▆█▄▂▅ beats 10/81 very hard · avg 8.0% 1.0% at its level
MMLU-ProX 79.0% ▁·▁▁▁▂▁▃▃●▅▄ beats 21/36 medium · avg 60% 0.9% typical
Multi-IF (aggregation unspecified) 78.0% ▁▁▁▁▁▁▃▄●▄▁▂ beats 28/41 medium · avg 60% 0.8% typical
Creative Writing v3 78.8% ▂····▂●··▆█▆ beats 3/14 medium · avg 65% 0.8% typical
SuperGPQA 56.4% ▂▁▂▂▅▆▃▇●▇█▃ beats 29/46 medium · avg 37% 0.7% at its level · small run
SimpleQA (maker) 24.0% ▄▄▄▆█▆█●█·▂█ beats 22/32 very hard · avg 9.2% 0.7% at its level · noisy
WritingBench 78.4% ▁·▁●▁··▅█▂▂▁ beats 2/20 easy · avg 66% 0.7% typical
SimpleQA (simple-evals) (maker) 19.3% ▄·▁▁●▁··█▁·▁ beats 5/14 hard · avg 17% 0.3% at its level · noisy
DROP F1, 3-shot (simple-evals) (m… 78.1% ▂●▄▆▆█·▂···▄ beats 1/16 easy · avg 78% 0.3% blunt
Vectara hallucination leaderboard… 95.4% ▁▁▂▃▄▄●▆▃▂▂▁ beats 72/140 very easy · avg 94% 0.1% blunt · maxed out · noisy
Vectara hallucination leaderboard… 81.4% ▁●▁▂▅█▂▃▄▃▁▁ beats 7/98 very easy · avg 90% 0.0% blunt
source results share of the score
Epoch AI 8 26% ▇▇▇
Qwen/Qwen3-235B-A22B-Thinking-2507 mode… 17 22% ▇▇▇
OpenAI simple-evals README benchmark re… 7 10% ▇
Aider polyglot benchmark runs 1 8% ▇
LMArena leaderboard dataset 2 5% ▇
τ²/τ³-bench leaderboard: Sierra's own e… 3 5% ▇
Arena-Hard-Auto v2.0-Preview leaderboard 2 4% ▇
Korean Canonical Legal Benchmark: Towar… 2 4% ▇
Inception Arabic/Hindi leaderboards 2 3% ▇
Finance Agent Benchmark 1 2% ▇
LiveCodeBench official website results 1 2% ▇
EQ-Bench 3 canonical leaderboard results 1 2% ▇
WebApp1K models leaderboard 1 2% ▇
Berkeley Function Calling Leaderboard r… 1 2% ▇
OpenAI vendor-reported benchmark scores… 1 1% ▇
OpenAI (submitted / maker-reported) 1 1% ▇
Galileo Agent Leaderboard 1 0% ▇
Vectara hallucination leaderboard 2 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 65.7
Licences: Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Qwen/Qwen3-235B-A22B-Thinking-2507 model card ([apache-2.0](../../licenses/Qwen--Qwen3-235B-A22B-Thinking-2507.txt)) · OpenAI simple-evals README benchmark results ([MIT](../../licenses/openai-simple-evals-MIT.txt)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · τ²/τ³-bench leaderboard: Sierra's own evaluation runs ([MIT](https://opensource.org/license/mit)) · Arena-Hard-Auto v2.0-Preview leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-Apache-2.0.txt)) · Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities ([CC BY 4.0](../../licenses/cards-final-NOTICE.txt)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Finance Agent Benchmark ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · EQ-Bench 3 canonical leaderboard results ([MIT](../../licenses/eqbench3-MIT.txt)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · Galileo Agent Leaderboard ([Apache-2.0](../../licenses/galileo-agent-leaderboard-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
### Through an API
api (OpenAI) · $1.93/M blended · $1.1 in / $4.4 out per 1M tokens · [platform.openai.com](https://platform.openai.com/docs/models)
[price source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/providers/openai/models/o4-mini.toml): Standard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ. · [context source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/models/openai/o4-mini.toml)
[interactive version](https://whatmodel.sh/model/o4-mini) · [JSON](https://whatmodel.sh/api?format=json&all=1)