[whatmodel.sh](../../README.md) / o3
# o3
OpenAI · released 2025-04 · closed weights
**score 68.4** ±0.5 · #94 of 732 · from 68 benchmark results across 17 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks o3 among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃▆█▄●▅▄▄▂▁ #146 of 313 19% 7
coding ▁▁▁▃▅▇█▇●▄▃▁ #102 of 469 14% 6
chat & writing ▁▁▁▃▆▄▆█▆▇●▃ #74 of 402 8.0% 6
math ▁▁▃▅█▆▄●▄▂▁▁ #88 of 431 21% 9
vision & documents ▁▁▂▂▁▃▄▂●▇█▄ #72 of 153* 1.7% 1
web search ▁▁▁▅▄█▃▄·▁·● #1 of 84* 3.1% 2
agent tasks · #146 of 313 · 19% of its score
Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Large](jamba-1-5-large.md), [GPT-4o (2024-05-13)](gpt-4o-2024-05-13.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
τ²-bench Telecom (maker) 58.2% ▃·▅▅●▃··█▃▃▅ beats 5/13 very hard · avg 7.7% 5.6% sharp · at its level · several runs
SWE-bench Verified 62.3% ▁··▁·▁●▁▄█▄▁ beats 5/29 hard · avg 19% 5.5% sharp · at its level
τ²-bench Retail 72.8% █···█·█·██·● beats 5/5 hard · avg 35% 2.6% at its level · several runs
Finance Agent Benchmark v1 (537 q… 46.8% █·▃▃·█▅▅▅▅▃● beats 18/18 very hard · avg 5.5% 2.2% sharp · at its level
τ²-bench Airline 62.5% ▄·····█··●·█ beats 3/5 hard · avg 28% 1.6% at its level · several runs
BFCL V4 (December 2025 archive) 48.6% ▁▁▁▂██▅●▆▂▂▂ beats 55/77 hard · avg 31% 1.2% at its level
Galileo Agent Leaderboard v1 (too… 86.1% ▃▃▃▃▆▅●▅█▂▅▂ beats 15/31 easy · avg 78% 0.2% typical
coding · #102 of 469 · 14% of its score
Beats [Grok 4.20 (0309 reasoning)](grok-4-20.md), [Grok 4.3](grok-4-3.md), [Muse Glimmer 30B](muse-glimmer-30b.md) here, though they rank higher overall. Trails [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [Qwen3 235B-A22B Thinking 2507](qwen3-235b-a22b-thinking-2507.md), [o4-mini](o4-mini.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
SWE-bench Verified 62.3% ▁··▁·▁●▁▄█▄▁ beats 5/29 hard · avg 19% 5.5% sharp · at its level
Aider polyglot (225 exercises, tw… 81.3% ▂·▂▃▅▂▃█▆▄●▂ beats 46/49 very hard · avg 7.6% 4.9% sharp
LiveCodeBench official (Aug 1, 20… 75.8% ▃▃▅▃··▃▅█▅●▃ beats 14/15 hard · avg 24% 1.4% sharp · at its level
LiveCodeBench (version unspecifie… 76.2% ▄······▄█●·▄ beats 4/5 medium · avg 41% 1.2% typical
LiveCodeBench v6 (Feb–May 2025) 58.6% ▁▁▃▁█●▄▇▅▅▃█ beats 13/38 medium · avg 36% 0.7% at its level
HumanEval (simple-evals) (maker) 88.4% ▆●▆▆▂··▂···▂ beats 5/15 easy · avg 83% 0.4% maxed out
chat & writing · #74 of 402 · 8.0% of its score
Beats [Claude Haiku 5.5](claude-haiku-5-5.md), [LongCat 2.0 (1.6T-A48B)](longcat-2-0.md), [GPT-5](gpt-5.md) here, though they rank higher overall. Trails [Gemma 4 26B-A4B](gemma-4-26b-a4b.md), [Nemotron Cascade 2 30B-A3B](nemotron-cascade-2-30b-a3b.md), [Gemma 4 31B](gemma-4-31b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 1431.7 ▁▁▁▂▃▃▃▆▆●█▃ beats 288/385 avg 1260 2.5% blunt
Arena-Hard v2.0 hard prompts (sty… 85.9% ▂·▃▅▅·▃█▆▂▂● beats 22/22 very hard · avg 12% 2.4% sharp · at its level
Arena-Hard v2 aggregate 80.8% ▁▁··▄▃▁▃█▁▁● beats 20/22 hard · avg 28% 1.2% sharp
IFEval prompt-level strict 90.2% ███··█·●···█ beats 4/5 easy · avg 73% 0.7% typical
Arena-Hard v2.0 creative writing … 88.8% ▆▂▂▆▂▂▃█▂▃·● beats 22/22 hard · avg 19% 0.5% sharp · at its level · noisy
IFEval (aggregation unspecified) 92.1% ▁▂▄▂▂▄▆██●▂▁ beats 124/145 easy · avg 77% 0.5% typical
math · #88 of 431 · 21% of its score
Beats [Claude Sonnet 4.6](claude-sonnet-4-6.md), [Claude Opus 4.5](claude-opus-4-5.md), [Qwen3.5 397B-A17B](qwen3-5-397b-a17b.md) here, though they rank higher overall. Trails [Gemini 2.5 Flash](gemini-2-5-flash.md), [Granite 4.2 30B](granite-4-2-30b.md), [Qwen3-Next 80B-A3B Instruct](qwen3-next-80b-a3b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
FrontierMath tiers 1–3 v2 33.3% ▂··▁▁▃●▄█▆▄▃ beats 22/86 very hard · avg 1.2% 11% sharp · at its level
HMMT 2025 (edition unspecified) 71.9% ●··▄··▄·▄··█ beats 0/5 hard · avg 31% 3.4% sharp · several runs
AIME 2024 (maker) 90.0% ▁▁▁▂▁▃▁▅██●▃ beats 73/85 medium · avg 50% 1.5% sharp · small run
OTIS mock AIME 2024–2025 83.9% ▃▃▄▃▃▃▇●█▃▂▅ beats 121/197 very hard · avg 7.2% 1.1% sharp · noisy
MATH level 5 97.8% ▂▂▅▆▄█▆▃▅▂▃● beats 89/92 hard · avg 28% 1.1% sharp · maxed out
MATH-500 (simple-evals) (maker) 98.1% ▃··▅▅▃···▃·● beats 8/9 medium · avg 61% 0.8% sharp · maxed out
PolyMATH 49.7% ▁···▃▁·▇●▄▄▄ beats 11/23 hard · avg 28% 0.6% at its level
MGSM (simple-evals) (maker) 92.0% ▁····▁▅▁█▁●▁ beats 15/16 easy · avg 80% 0.5% maxed out
MATH-500 98.4% █···█●·█·█·█ beats 2/5 very easy · avg 92% 0.5% maxed out · noisy
vision & documents · #72 of 153 · 1.7% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-5](gpt-5.md) here, though they rank higher overall. Trails [Gemma 4 26B-A4B](gemma-4-26b-a4b.md), [Claude Haiku 4.5](claude-haiku-4-5.md), [Qwen3.6 Flash](qwen3-6-flash.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena vision (style controlled) 1215.8 ▁▁▂▂▂▂▃▄▅●█▇ beats 82/152 avg 1061 1.7% blunt
web search · #1 of 84 · 3.1% of its score
Beats [GPT-6 Astra](gpt-6-astra.md), [Claude Opus 5.5](claude-opus-5-5.md), [GPT-6.1 Sol](gpt-6-1-sol.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 46.8% █·▃▃·█▅▅▅▅▃● beats 18/18 very hard · avg 5.5% 2.2% sharp · at its level
Arena search systems (style contr… 1196.2 ▁·▁▁▂▂▂●▆▃▂▂ beats 19/36 avg 1069 0.9% blunt
other tests (29)
test result vs the others in its run difficulty counts why
GPQA Diamond 81.8% ▁▁▁▁▂▄▆▄▅●█▅ beats 131/211 hard · avg 38% 10% sharp · several runs
GPQA Diamond (simple-evals) (make… 83.4% █▃▃█·▃▃▅▃▃▃● beats 16/16 hard · avg 39% 3.7% sharp · at its level
MMLU-Pro 85.3% ▄·▄···█·●··▄ beats 4/5 medium · avg 55% 2.6% several runs
MMLU-Redux 93.1% ████··█····● beats 5/5 medium · avg 73% 2.3% several runs
OJBench 38.4% ▄······▄▄·●█ beats 3/5 very hard · avg 12% 2.0% at its level · several runs
KCL-Essay vanilla 51.2% ▁···▁▃▂█▂▃●▂ beats 20/23 hard · avg 17% 2.0% at its level
Mystery Game Puzzles (Epoch) 29.0% █··▂▃▇▄●▂▄▂▁ beats 47/77 very hard · avg 9.4% 1.7% sharp · noisy
EQ-Bench 3 (emotional intelligenc… 1288.3 ▁·▂▁▄▅▅▃●▄▁▁ beats 43/62 avg 1034 1.5% typical
Chess Puzzles (Epoch) 34.0% █▁▁▁▁▂▃▃●▂▁▁ beats 112/137 very hard · avg 5.8% 1.4% sharp · noisy
AraGen-v3 3C3H score (Inception, … 82.2% ▁▁·▁▂▂▄█▄▅▄● beats 64/66 hard · avg 29% 1.2% at its level
KCL-MCQA vanilla 50.2% ▂···█▂▆▆█●▆▂ beats 17/23 very hard · avg 29% 1.1% at its level
VitaBench 35.3% █····██·██·● beats 5/5 very hard · avg 8.2% 1.1% at its level
MMLU (simple-evals, zero-shot CoT… 93.3% ▂▂·█▄▄▂·█▂·● beats 16/16 easy · avg 79% 1.1% typical
INCLUDE 86.6% ▂····▂▂▄▄▄█● beats 36/36 medium · avg 59% 1.1% typical
ZebraLogic 94.3% ██··●·██···█ beats 2/5 easy · avg 75% 1.1% maxed out
SimpleQA Verified 49.4% ▁▁▃▂▃▃▆▆●▄▂▅ beats 54/81 very hard · avg 8.0% 0.9% at its level
HindiGen-v1 3C3H score (Inception… 85.6% ▁▁▁▁·▂·█▆█▇● beats 31/31 medium · avg 46% 0.8% typical
MMMLU (maker) 88.8% ▂▁▁▂▃▂▄▄▅█●▃ beats 43/46 medium · avg 64% 0.8% small run
BeyondAIME 63.2% ●··▄····▄··█ beats 1/5 medium · avg 42% 0.7% at its level
MMLU-ProX 83.3% ▁·▁▁▁▂▁▃▃█●▄ beats 31/36 medium · avg 60% 0.6% typical
SimpleQA (maker) 54.0% ▄▄▄▆█▆███·▂● beats 30/32 very hard · avg 9.2% 0.6% at its level · noisy
Multi-IF (aggregation unspecified) 80.3% ▁▁▁▁▁▁▃▄█●▁▂ beats 32/41 medium · avg 60% 0.6% typical
Creative Writing v3 87.7% ▂····▂▆··▆█● beats 14/14 medium · avg 65% 0.6% typical
WritingBench 85.3% ▁·▁▂▁··▅●▂▂▁ beats 14/20 easy · avg 66% 0.5% typical
HealthBench Hard (maker) 31.6% ▁····▁▁▂●·▁▁ beats 7/13 hard · avg 16% 0.3% blunt · at its level · small run
SimpleQA (simple-evals) (maker) 48.6% ▄·▁▁▁▁··█●·▁ beats 13/14 hard · avg 17% 0.3% at its level · noisy
DROP F1, 3-shot (simple-evals) (m… 89.8% ▂▂▄▆▆█·▂···● beats 15/16 easy · avg 78% 0.2% blunt
miniF2F (pass@1) 15.2% █●······▄··▄ beats 3/5 very hard · avg 14% 0.1% blunt · at its level · noisy
Vectara hallucination leaderboard… 93.2% ▁▁▂▃▄●█▆▃▂▂▁ beats 44/140 very easy · avg 94% 0.1% blunt · maxed out · noisy
source results share of the score
Epoch AI 8 26% ▇▇▇
Meituan LongCat-Flash-Thinking model ca… 15 18% ▇▇
Qwen/Qwen3-235B-A22B-Thinking-2507 mode… 16 16% ▇▇
OpenAI simple-evals README benchmark re… 7 7% ▇
OpenAI vendor-reported benchmark scores… 3 6% ▇
LMArena leaderboard dataset 3 5% ▇
Aider polyglot benchmark runs 1 5% ▇
Korean Canonical Legal Benchmark: Towar… 2 3% ▇
Arena-Hard-Auto v2.0-Preview leaderboard 2 3% ▇
Finance Agent Benchmark 1 2% ▇
Inception Arabic/Hindi leaderboards 2 2% ▇
OpenAI (submitted / maker-reported) 3 2% ▇
EQ-Bench 3 canonical leaderboard results 1 2% ▇
LiveCodeBench official website results 1 1% ▇
Berkeley Function Calling Leaderboard r… 1 1% ▇
Galileo Agent Leaderboard 1 0% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 68.4
Licences: Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Meituan LongCat-Flash-Thinking model card ([MIT](https://huggingface.co/meituan-longcat/LongCat-Flash-Thinking/blob/7c9aad1e2daceaeab3a5a03deedb8e11e7a7819f/LICENSE)) · Qwen/Qwen3-235B-A22B-Thinking-2507 model card ([apache-2.0](../../licenses/Qwen--Qwen3-235B-A22B-Thinking-2507.txt)) · OpenAI simple-evals README benchmark results ([MIT](../../licenses/openai-simple-evals-MIT.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities ([CC BY 4.0](../../licenses/cards-final-NOTICE.txt)) · Arena-Hard-Auto v2.0-Preview leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-Apache-2.0.txt)) · Finance Agent Benchmark ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · EQ-Bench 3 canonical leaderboard results ([MIT](../../licenses/eqbench3-MIT.txt)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · Galileo Agent Leaderboard ([Apache-2.0](../../licenses/galileo-agent-leaderboard-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
### Through an API
api (OpenAI) · $3.50/M blended · $2 in / $8 out per 1M tokens · [platform.openai.com](https://platform.openai.com/docs/models)
[price source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/providers/openai/models/o3.toml): Standard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ. · [context source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/models/openai/o3.toml)
[interactive version](https://whatmodel.sh/model/o3) · [JSON](https://whatmodel.sh/api?format=json&all=1)