[whatmodel.sh](../../README.md) / o1-mini
# o1-mini
OpenAI · released 2024-09 · proprietary
**score 61.2** ±0.5 · comparison only: no current run option verified · from 41 benchmark results across 18 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks o1-mini among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▂▆█●▅▄▄▃▁▁ #186 of 354 6.0% 4
coding ▁▁▁▃▄▆█●▅▄▃▁ #170 of 539 41% 12
chat & writing ▁▁▁▃▅▄▇█●▇▇▃ #208 of 474* 5.9% 2
math ▁▁▃▆██●█▃▁▁▁ #187 of 530 14% 9
agent tasks · #186 of 354 · 6.0% of its score
Beats [o1 (2024-12-17)](o1-2024-12-17.md), [o3-mini](o3-mini.md), [DeepSeek R1 0528](deepseek-r1-0528.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [Jamba 1.5 Large](jamba-1-5-large.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
BrowserGym WebArena (AgentLab Gen… 28.6% █··▄·●█·▄··▄ beats 3/7 hard · avg 17% 1.7% at its level
BrowserGym WorkArena-L1 (AgentLab… 56.7% █···█▄▄·▄●·█ beats 8/10 hard · avg 28% 1.7% at its level
BrowserGym WorkArena-L2 (AgentLab… 14.9% ▃▅▃·▅▃●··█·▃ beats 7/11 very hard · avg 3.6% 1.5% sharp · noisy
BrowserGym MiniWoB (AgentLab Gene… 67.8% ▅····▅▅▃●·▃█ beats 7/11 medium · avg 53% 1.0% blunt · at its level
coding · #170 of 539 · 41% of its score
Beats [Qwen3-Coder-Next (80B-A3B)](qwen3-coder-next.md), [Gemini 2.5 Pro](gemini-2-5-pro.md), [Gemini 3.1 Flash-Lite](gemini-3-1-flash-lite.md) here, though they rank higher overall. Trails [Phi-4-reasoning 14B](phi-4-reasoning.md), [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [Devstral Small 2 24B](devstral-small-2.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Aider polyglot (225 exercises, tw… 32.9% ▂·▂▃▅●▃█▆▄▂▂ beats 17/49 very hard · avg 7.6% 7.2% sharp · at its level
LiveCodeBench official (Jan 1–Nov… 67.2% ▁▁·▁▂▄▆█▅▂·● beats 30/31 hard · avg 23% 7.0% sharp · at its level
HumanEval+ 89.0% ▁·▁▁▁▃▄▄▄█▃● beats 98/99 easy · avg 70% 5.9% sharp
HumanEval 96.3% ▁·▁▁▃▄▆▅█▃▁● beats 98/99 easy · avg 76% 5.6% sharp · maxed out
CRUXEval (ZeroEval, 800 problems) 93.8% ▁▂▆▃▆█▆▃▂·●▁ beats 46/47 medium · avg 57% 3.9% sharp
Aider code editing (133 Exercism … 70.7% ▁·▁▂▄▁▆▄█●▂▂ beats 49/62 medium · avg 51% 3.6% sharp
SciCode main problems (official l… 1.5% ▃···●··▂·▃·▂ beats 7/17 very hard · avg 1.0% 2.4% sharp · too hard for it
MBPP+ 78.8% ▁···▁▁▂▅▇█▆● beats 74/75 easy · avg 66% 2.0% typical
WebApp1K-Duo-React pass@1 (onekq-… 66.7% ▁··▁·▁▁▁▆●▇█ beats 24/45 hard · avg 29% 1.6% at its level
WebApp1K-React pass@1 (onekq-ai) 93.9% ▃▁▁█▁▄▅▃▄▇▁● beats 29/31 medium · avg 40% 0.8% sharp · noisy
HumanEval (simple-evals) (maker) 92.4% ▆█▆●▂··▂···▂ beats 10/15 easy · avg 83% 0.7% typical
LiveCodeBench v2 72.8% ▄··█·▄···▄▄● beats 7/7 medium · avg 50% 0.6% at its level
chat & writing · #208 of 474 · 5.9% of its score
Beats [MiniCPM5 2B](minicpm5-2b.md), [Seed-OSS 36B Instruct](seed-oss-36b.md), [GPT-5 nano](gpt-5-nano.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena-Hard (v0.1, GPT-4-1106 judg… 92.0% ▂▁▁▅█▅▄▃▃▅▃● beats 60/60 medium · avg 47% 3.2% sharp
LMArena (text) 1336.8 ▁▁▁▂▃▃▃▆●▇█▃ beats 175/385 avg 1260 2.8% blunt
math · #187 of 530 · 14% of its score
Beats [MiMo-V2.5 (310B-A15B)](mimo-v2-5.md), [MiniMax-M2.7](minimax-m2-7.md), [MiniMax-M2.1](minimax-m2-1.md) here, though they rank higher overall. Trails [OpenThinker3 7B](openthinker3-7b.md), [Olmo 3 7B Think](olmo-3-7b-think.md), [Nemotron Nano 9B v2](nemotron-nano-9b-v2.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 89.2% ▂▂▅▆▄█▆▃●▂▃▄ beats 77/92 hard · avg 28% 3.9% sharp
MATH Level 5 (ZeroEval, 721 probl… 89.3% ▁▁▁▆█▂▇▇▄··● beats 38/38 hard · avg 30% 3.2% sharp · at its level
OTIS mock AIME 2024–2025 46.9% ▃▃▄▃▃●▇▇█▃▂▅ beats 75/197 very hard · avg 7.2% 2.3% sharp · at its level · noisy
MATH-500 (simple-evals) (maker) 90.0% ▃··▅▅●···▃·█ beats 5/9 medium · avg 61% 1.8% sharp
HMMT February 2025 38.0% ▁▁▁●▁▅▅▃██▁▄ beats 3/31 hard · avg 32% 1.0% at its level · small run
GSM8K (ZeroEval, zero-shot) 96.1% ▁▄·▁▄█▅▅▅██● beats 41/43 very easy · avg 87% 0.9% sharp · maxed out
MGSM (simple-evals) (maker) 89.9% ▁····▁▅▁●▁▃▁ beats 10/16 easy · avg 80% 0.8% typical
AIME 2024 64.0% ▁▁▁▂▁▃▁▅●██▃ beats 30/85 medium · avg 40% 0.3% sharp · at its level · noisy
MATH-500 85.6% ▁▃▂▃▄●▅▇█▄▂▁ beats 28/85 easy · avg 84% 0.2% maxed out · noisy
other tests (13)
test result vs the others in its run difficulty counts why
ZebraLogic (ZeroEval, 1,000 puzzl… 52.6% ▁▁▃▆██▁●·▁▁▁ beats 53/57 hard · avg 15% 6.5% sharp · at its level
MMLU-Redux (ZeroEval, 2,778 quest… 86.7% ▁·▁▁▄▄▄▅█●▂▁ beats 48/53 easy · avg 75% 6.3% typical
GPQA Diamond 62.4% ▁▁▁▁▂▄▆●▅▇█▅ beats 90/211 hard · avg 39% 6.0% sharp · at its level · several runs
GPQA Diamond (simple-evals) (make… 60.0% █▃▃█·●▃▅▃▃▃▅ beats 8/16 hard · avg 39% 5.1% sharp · at its level
MMLU (simple-evals, zero-shot CoT… 85.2% ▂▂·●▄▄▂·█▂·▂ beats 3/16 easy · avg 79% 1.7% typical
AraGen-v3 3C3H score (Inception, … 70.5% ▁▁·▁▂▂▄█▄▅●▄ beats 55/66 hard · avg 29% 1.6% at its level
LiveBench 2024-08-31 59.1% ▄█·▄·▄▄·●··▄ beats 6/7 medium · avg 40% 1.5% at its level
HindiGen-v1 3C3H score (Inception… 60.0% ▁▁▁▁·▂·█●█▇▁ beats 14/31 medium · avg 46% 1.3% at its level
MMMLU (maker) 78.6% ▂▁▁▂▃▂▄▄▅●█▃ beats 27/46 medium · avg 64% 1.0% at its level · small run
EvalPerf correctness (118 tasks, … 88.8% ▂·▂▂▂▆▅▆▆▆▃● beats 29/29 easy · avg 74% 0.9% typical
SimpleQA (simple-evals) (maker) 7.6% ●·▁▁▁▁··█▁·▁ beats 1/14 hard · avg 17% 0.3% at its level · noisy
DROP F1, 3-shot (simple-evals) (m… 83.9% ▂▂▄▆▆●·▂···▄ beats 13/16 easy · avg 78% 0.2% blunt
Vectara hallucination leaderboard… 98.6% ▁▁▂▃▄▄█▆▃●▂▁ beats 130/140 very easy · avg 94% 0.1% blunt · maxed out · noisy
source results share of the score
WildEval ZeroEval published results 5 21% ▇▇▇
EvalPlus / EvalPerf published evaluatio… 4 14% ▇▇
Epoch AI 3 11% ▇
OpenAI simple-evals README benchmark re… 7 11% ▇
Aider polyglot benchmark runs 1 7% ▇
LiveCodeBench official website results 1 7% ▇
ServiceNow BrowserGym leaderboard 4 6% ▇
Aider code editing benchmark runs 1 4% ▇
Arena-Hard-Auto v0.1 leaderboard 1 3% ▇
Inception Arabic/Hindi leaderboards 2 3% ▇
LMArena leaderboard dataset 1 3% ▇
WebApp1K models leaderboard 2 2% ▇
SciCode leaderboard 1 2% ▇
open-thoughts/OpenThinker-7B model card 4 2% ▇
LiveBench model judgments 1 1% ▇
Microsoft 1 1% ▇
OpenAI (submitted / maker-reported) 1 1% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 61.2
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · OpenAI simple-evals README benchmark results ([MIT](../../licenses/openai-simple-evals-MIT.txt)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · ServiceNow BrowserGym leaderboard ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · open-thoughts/OpenThinker-7B model card ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Microsoft ([CC BY 4.0](../../licenses/report-2504.21318v1.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/o1-mini) · [JSON](https://whatmodel.sh/api?format=json&all=1)