[whatmodel.sh](../../README.md) / o1 (2024-12-17)
# o1 (2024-12-17)
OpenAI · released 2024-12 · proprietary
**score 65.8** ±0.6 · #130 of 732 · from 39 benchmark results across 19 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks o1 (2024-12-17) among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃▆●▄▅▅▄▄▂▁ #237 of 313 4.5% 4
coding ▁▁▁▃▅▇█▇●▄▃▁ #118 of 469 21% 5
chat & writing ▁▁▁▃▆▄▆█▆●▇▃ #113 of 402 8.1% 3
math ▁▁▃▅█▆▄●▄▂▁▁ #141 of 431 23% 8
vision & documents ▁▁▂▂▁▃▄▂●▇█▄ #88 of 153 6.5% 4
web search ▁▁▁▅▄█▃▄·●·▁ #4 of 84* 2.7% 1
agent tasks · #237 of 313 · 4.5% of its score
Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 21.4% █·▃▃·█▅▅●▅▃▅ beats 12/18 very hard · avg 5.5% 2.7% sharp · at its level
Multi-SWE-bench, 7 languages (Mag… 6.4% ▄···▄▄·▄▄·▄● beats 7/7 very hard · avg 2.6% 0.9% blunt · too hard for it
BFCL v3 56.1% ▁▁▁▁▁▃▅●▅▄▁▁ beats 62/114 medium · avg 51% 0.7% blunt · at its level
Galileo Agent Leaderboard v1 (too… 87.6% ▃▃▃▃▆▅●▅█▂▅▂ beats 18/31 easy · avg 78% 0.3% typical
coding · #118 of 469 · 21% of its score
Beats [Grok 4.3](grok-4-3.md), [Qwen3.5 Plus](qwen3-5-plus.md), [DeepSeek V3.2](deepseek-v3-2.md) here, though they rank higher overall. Trails [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [Solar Pro 4](solar-pro-4.md), [Laguna XS 2.1 (33B-A3B)](laguna-xs-2-1.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Aider polyglot (225 exercises, tw… 61.7% ▂·▂▃▅▂▃█●▄▂▂ beats 37/49 very hard · avg 7.6% 8.2% sharp · at its level
LiveCodeBench official (Jan 1–Nov… 72.4% ▁▁·▁▂▄▆█▅▂·● beats 31/31 hard · avg 23% 7.9% sharp · at its level
Aider code editing (133 Exercism … 84.2% ▁·▁▂▄▁▆▄█▆▂● beats 61/62 medium · avg 51% 3.2% sharp
HumanEval (Vector Institute, Insp… 97.0% ▄·▄·█·█·█·●▄ beats 8/9 easy · avg 77% 1.2% sharp · maxed out
Multi-SWE-bench, 7 languages (Mag… 6.4% ▄···▄▄·▄▄·▄● beats 7/7 very hard · avg 2.6% 0.9% blunt · too hard for it
chat & writing · #113 of 402 · 8.1% of its score
Beats [LongCat 2.0 (1.6T-A48B)](longcat-2-0.md), [GPT-5.4 nano](gpt-5-4-nano.md), [Step 3.5 Flash (196B-A11B)](step-3-5-flash.md) here, though they rank higher overall. Trails [Nemotron-Cascade-8B](nemotron-cascade-8b.md), [Mistral Large 3 (675B-A41B)](mistral-large-3.md), [Gemini 2.5 Flash](gemini-2-5-flash.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena-Hard v2.0 hard prompts (sty… 61.0% ▂·▃▅▅·▃█●▂▂▂ beats 19/22 very hard · avg 12% 3.8% sharp · at its level
LMArena (text) 1402.3 ▁▁▁▂▃▃▃▆▆●█▃ beats 248/385 avg 1260 3.4% blunt
Arena-Hard v2.0 creative writing … 59.9% ▆▂▂▆▂▂▃●▂▃·▂ beats 15/22 hard · avg 19% 0.8% sharp · at its level · noisy
math · #141 of 431 · 23% of its score
Beats [Claude Sonnet 4.6](claude-sonnet-4-6.md), [MiniMax-M3](minimax-m3.md), [MiMo-V2.5 (310B-A15B)](mimo-v2-5.md) here, though they rank higher overall. Trails [Olmo 3 32B Think](olmo-3-32b-think.md), [Nemotron-Cascade-8B](nemotron-cascade-8b.md), [Olmo 3.1 32B Think](olmo-3-1-32b-think.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
FrontierMath tiers 1–3 v2 14.7% ▂··▁▁●▄▄█▆▄▃ beats 8/86 very hard · avg 1.2% 12% sharp
MATH level 5 94.7% ▂▂▅▆▄█▆▃▅▂●▄ beats 81/92 hard · avg 28% 2.3% sharp
MATH (Vector Institute, Inspect E… 95.9% ▄··██▄··▄·●█ beats 7/9 medium · avg 50% 2.1% sharp
OTIS mock AIME 2024–2025 73.3% ▃▃▄▃▃▃▇●█▃▂▅ beats 108/197 very hard · avg 7.2% 2.1% sharp · noisy
MATH-500 (simple-evals) (maker) 96.4% ▃··▅▅▃···●·█ beats 6/9 medium · avg 61% 1.4% sharp · maxed out
AIME 2025 71.4% █··▄··●··▄·▄ beats 4/6 hard · avg 32% 1.2% sharp · noisy
MGSM (simple-evals) (maker) 89.3% ▁····▁▅▁●▁▃▁ beats 7/16 easy · avg 80% 0.8% typical
AIME 2024 74.6% ▄···▄●·▄▄··▄ beats 2/6 medium · avg 43% 0.7% sharp · noisy
vision & documents · #88 of 153 · 6.5% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-5 mini](gpt-5-mini.md) here, though they rank higher overall. Trails [GPT-4.1 mini](gpt-4-1-mini.md), [Mistral Large 3 (675B-A41B)](mistral-large-3.md), [Gemini 2.5 Flash](gemini-2-5-flash.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena vision (style controlled) 1192.7 ▁▁▂▂▂▂▃▄●▅█▇ beats 69/152 avg 1061 2.3% blunt
CharXiv (reasoning questions) (ma… 55.1% ▄··▂●··▂···▄ beats 3/9 hard · avg 32% 2.0% at its level
CharXiv descriptive questions (ma… 88.9% ▄▄·····▄·▄●▄ beats 5/6 easy · avg 71% 1.7% typical
PhysBench physical-world understa… 55.1% ▅▃█··▅▃····● beats 9/9 hard · avg 42% 0.5% blunt · at its level
web search · #4 of 84 · 2.7% of its score
Beats [GPT-6 Astra](gpt-6-astra.md), [Claude Opus 5.5](claude-opus-5-5.md), [GPT-6.1 Sol](gpt-6-1-sol.md) here, though they rank higher overall. Trails [GPT-4.1](gpt-4-1.md), [o4-mini](o4-mini.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 21.4% █·▃▃·█▅▅●▅▃▅ beats 12/18 very hard · avg 5.5% 2.7% sharp · at its level
other tests (15)
test result vs the others in its run difficulty counts why
GPQA Diamond 76.8% ▁▁▁▁▂▄▆▄▅●█▅ beats 119/211 hard · avg 39% 9.3% sharp · at its level · several runs
ZebraLogic (ZeroEval, 1,000 puzzl… 81.0% ▁▁▃▆██▁▁·▁●▁ beats 56/57 hard · avg 15% 7.2% sharp · at its level
GPQA Diamond (simple-evals) (make… 75.7% █▃▃█·▃▃▅▃●▃▅ beats 13/16 hard · avg 39% 5.7% sharp · at its level
GPQA Diamond (Vector Institute, I… 75.5% ▃··▃▃▃▃·▃·▃● beats 8/9 hard · avg 40% 4.3% sharp · at its level
MMLU-Pro (Vector Institute, Inspe… 84.5% ▃····▃▅·▃▅·● beats 9/9 medium · avg 58% 2.1% typical
AraGen-v3 3C3H score (Inception, … 84.3% ▁▁·▁▂▂▄█▄▅▄● beats 66/66 hard · avg 29% 1.8% at its level
MMLU (simple-evals, zero-shot CoT… 91.8% ▂▂·█▄▄▂·█●·▂ beats 15/16 easy · avg 79% 1.7% typical
Chess Puzzles (Epoch) 15.0% █▁▁▁▁●▃▃▃▂▁▁ beats 69/137 very hard · avg 5.8% 1.3% sharp · noisy
HindiGen-v1 3C3H score (Inception… 79.6% ▁▁▁▁·▂·█▆█●▁ beats 30/31 medium · avg 46% 1.3% typical
SimpleQA Verified 41.1% ▁▁▃▂▃▃▆●█▄▂▅ beats 37/81 very hard · avg 8.0% 1.1% at its level
DiagnosticIQ industrial maintenan… 65.4% ▁··▃▁▁▁▅█●▁▅ beats 18/24 hard · avg 51% 0.7% blunt · at its level
SimpleQA (simple-evals) (maker) 42.6% ▄·▁▁▁▁··●▁·▁ beats 12/14 hard · avg 17% 0.4% at its level · noisy
DROP F1, 3-shot (simple-evals) (m… 90.2% ▂▂▄▆▆█·▂···● beats 16/16 easy · avg 78% 0.3% blunt
Omni-MATH 67.5% █·█·●·██··██ beats 2/6 medium · avg 64% 0.2% blunt · noisy
Vectara hallucination leaderboard… 97.6% ▁▁▂▃▄▄█▆●▂▂▁ beats 116/140 very easy · avg 94% 0.1% blunt · maxed out · noisy
source results share of the score
Epoch AI 6 24% ▇▇▇
OpenAI simple-evals README benchmark re… 6 10% ▇
Vector Institute Evaluation Leaderboard 4 10% ▇
Aider polyglot benchmark runs 1 8% ▇
LiveCodeBench official website results 1 8% ▇
WildEval ZeroEval published results 1 7% ▇
Microsoft 4 6% ▇
LMArena leaderboard dataset 2 6% ▇
Arena-Hard-Auto v2.0-Preview leaderboard 2 5% ▇
OpenAI vendor-reported benchmark scores… 2 4% ▇
Aider code editing benchmark runs 1 3% ▇
Inception Arabic/Hindi leaderboards 2 3% ▇
Finance Agent Benchmark 1 3% ▇
Multi-SWE-bench experiments 1 1% ▇
DiagnosticIQ: A Benchmark for LLM-Based… 1 1% ▇
Berkeley Function Calling Leaderboard r… 1 1% ▇
PhysBench: Benchmarking and Enhancing V… 1 1% ▇
Galileo Agent Leaderboard 1 0% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 65.8
Licences: Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · OpenAI simple-evals README benchmark results ([MIT](../../licenses/openai-simple-evals-MIT.txt)) · Vector Institute Evaluation Leaderboard ([Apache-2.0](../../licenses/vector-inspect-evals-leaderboard-Apache-2.0.txt)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Microsoft ([CC BY 4.0](../../licenses/report-2504.21318v1.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Arena-Hard-Auto v2.0-Preview leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-Apache-2.0.txt)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Finance Agent Benchmark ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Multi-SWE-bench experiments ([Apache-2.0](../../licenses/multi-swe-bench-experiments-Apache-2.0.txt)) · DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Galileo Agent Leaderboard ([Apache-2.0](../../licenses/galileo-agent-leaderboard-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
### Through an API
api (OpenAI) · $26.25/M blended · $15 in / $60 out per 1M tokens · [developers.openai.com](https://developers.openai.com/api/docs/models/o1)
[price source](https://developers.openai.com/api/docs/models/o1): USD per 1M tokens, OpenAI API standard processing, model page 'o1' whose default snapshot is o1-2024-12-17: input $15, output $60; cached-input, Batch and other tiers excluded. Currency: USD. OpenAI official model page, observed 2026-09-28. · [context source](https://developers.openai.com/api/docs/models/o1): OpenAI model page for o1 (default snapshot o1-2024-12-17): 200,000 context window. Observed 2026-09-28.
[interactive version](https://whatmodel.sh/model/o1-2024-12-17) · [JSON](https://whatmodel.sh/api?format=json&all=1)