[whatmodel.sh](../../README.md) / o1-preview # o1-preview OpenAI · released 2024-09 · proprietary **score 63.2** ±0.6 · comparison only: no current run option verified · from 27 benchmark results across 13 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks o1-preview among every model on that task or test, worst to best. 
task            vs the others            rank  counts  tests
agent tasks                    not ranked yet       —      1
coding          ▁▁▁▃▄▆█▇●▄▃▁      #152 of 539     42%     11
chat & writing  ▁▁▁▃▅▄▇█▇●▇▃     #146 of 474*    7.2%      2
math            ▁▁▃▆██●█▃▁▁▁      #208 of 530     14%      5
agent tasks · 0.0% of its score 
test                        result  vs the others in its run   difficulty      counts  why
SWE-bench Verified (maker)   41.3%  ▁▁▁▁▁▁▂●█▆▁▁ beats 28/138  hard · avg 19%       —  not counted yet: no other model in the …
coding · #152 of 539 · 42% of its score Beats [Qwen3.5 Plus](qwen3-5-plus.md), [Qwen3-Coder-Next (80B-A3B)](qwen3-coder-next.md), [Gemini 2.5 Pro](gemini-2-5-pro.md) here, though they rank higher overall. Trails [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [Gemma 4 12B](gemma-4-12b.md), [Seed-OSS 36B Instruct](seed-oss-36b.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty            counts  why
LiveCodeBench official (Jan 1–Nov…   52.5%  ▁▁·▁▂▄▆█▅●·▂ beats 29/31   hard · avg 23%           10%  sharp · at its level
HumanEval+                           89.0%  ▁·▁▁▁▃▄▄▄█▃● beats 99/99   easy · avg 70%          6.8%  sharp · maxed out
HumanEval                            96.3%  ▁·▁▁▃▄▆▅█▃▁● beats 99/99   easy · avg 76%          6.2%  sharp · maxed out
CRUXEval (ZeroEval, 800 problems)    95.9%  ▁▂▆▃▆█▆▃▂·▁● beats 47/47   medium · avg 57%        4.5%  sharp
Aider code editing (133 Exercism …   79.7%  ▁·▁▂▄▁▆▄█▆●▂ beats 59/62   medium · avg 51%        4.5%  sharp
SciCode main problems (official l…    7.7%  ▃···█··▂·▃·● beats 16/17   very hard · avg 1.0%    4.1%  sharp · too hard for it
MBPP+                                80.2%  ▁···▁▁▂▅▇█▆● beats 75/75   easy · avg 66%          2.6%  typical
WebApp1K-Duo-React pass@1 (onekq-…   65.2%  ▁··▁·▁▁▁▆●▇█ beats 21/45   hard · avg 29%          2.2%  at its level
WebApp1K-React pass@1 (onekq-ai)     95.2%  ▃▁▁█▁▄▅▃▄▇▁● beats 30/31   medium · avg 40%        0.9%  sharp · noisy
HumanEval (simple-evals) (maker)     92.4%  ▆█▆●▂··▂···▂ beats 11/15   easy · avg 83%          0.8%  maxed out
SWE-bench Verified (maker)           41.3%  ▁▁▁▁▁▁▂●█▆▁▁ beats 28/138  hard · avg 19%             —  not counted yet: no other model in the …
chat & writing · #146 of 474 · 7.2% of its score Beats [LongCat 2.0 (1.6T-A48B)](longcat-2-0.md), [o3-mini](o3-mini.md), [MiMo-V2-Flash (309B-A15B)](mimo-v2-flash.md) here, though they rank higher overall. Trails [LFM2.5 2.6B](lfm2-5-2-6b.md), [Nemotron-Cascade-8B](nemotron-cascade-8b.md), [DeepSeek V3 0324](deepseek-v3-0324.md) here, though they rank lower overall. 
test                                result  vs the others in its run    difficulty        counts  why
LMArena (text)                      1388.6  ▁▁▁▂▃▃▃▆▆●█▃ beats 231/385  avg 1260            3.9%  blunt
Arena-Hard (v0.1, GPT-4-1106 judg…   90.4%  ▂▁▁▅█▅▄▃▃▅▃● beats 59/60    medium · avg 47%    3.3%  sharp
math · #208 of 530 · 14% of its score Beats [MiniMax-M2.1](minimax-m2-1.md) here, though they rank higher overall. Trails [Phi-4-mini-reasoning 3.8B](phi-4-mini-reasoning.md), [OpenThinker3 7B](openthinker3-7b.md), [Olmo 3 7B Think](olmo-3-7b-think.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty            counts  why
MATH level 5                         81.6%  ▂▂▅▆▄█▆●▅▂▃▄ beats 67/92   hard · avg 28%          4.1%  sharp
MATH Level 5 (ZeroEval, 721 probl…   84.5%  ▁▁▁▆█▂▇▇▄··● beats 37/38   hard · avg 30%          4.0%  sharp
OTIS mock AIME 2024–2025             31.1%  ▃▃▄▃▃●▇▇█▃▂▅ beats 66/197  very hard · avg 7.2%    3.0%  sharp · at its level · noisy
MATH-500 (simple-evals) (maker)      85.5%  ▃··▅●▃···▃·█ beats 3/9     medium · avg 61%        2.1%  sharp
MGSM (simple-evals) (maker)          90.8%  ▁····▁▅▁█●▃▁ beats 13/16   easy · avg 80%          1.0%  typical
other tests (10) 
test                                result  vs the others in its run   difficulty           counts  why
ZebraLogic (ZeroEval, 1,000 puzzl…   71.4%  ▁▁▃▆██▁▁·●▁▁ beats 54/57   hard · avg 15%         9.0%  sharp · at its level
MMLU-Redux (ZeroEval, 2,778 quest…   92.8%  ▁·▁▁▄▄▄▅█▄▂● beats 53/53   easy · avg 75%         7.6%  typical
GPQA Diamond (simple-evals) (make…   73.3%  █▃▃█·▃▃▅●▃▃▅ beats 12/16   hard · avg 39%         7.0%  sharp · at its level
GPQA Diamond                         50.3%  ▁▁▁▁▂▄●▄▅▇█▅ beats 70/211  hard · avg 39%         6.3%  sharp · at its level
MMLU (simple-evals, zero-shot CoT…   90.8%  ▂▂·█▄▄▂·●▂·▂ beats 14/16   easy · avg 79%         2.1%  typical
LiveBench 2024-08-31                 66.0%  ▄█·▄·▄▄·▄··● beats 7/7     medium · avg 40%       2.0%  at its level
MMMLU (maker)                        86.7%  ▂▁▁▂▃▂▄▄▅█●▃ beats 39/46   medium · avg 64%       1.3%  small run
SimpleQA (simple-evals) (maker)      42.4%  ▄·▁▁▁▁··●▁·▁ beats 11/14   hard · avg 17%         0.4%  at its level · noisy
DROP F1, 3-shot (simple-evals) (m…   74.8%  ●▂▄▆▆█·▂···▄ beats 0/16    easy · avg 78%         0.3%  blunt
Vectara hallucination leaderboard…   96.7%  ▁▁▂▃▄▄█●▃▂▂▁ beats 98/140  very easy · avg 94%    0.1%  blunt · maxed out · noisy
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
WildEval ZeroEval published results             4   25% ▇▇▇
EvalPlus / EvalPerf published evaluatio…        3   16% ▇▇
OpenAI simple-evals README benchmark re…        7   14% ▇▇
Epoch AI                                        3   13% ▇▇
LiveCodeBench official website results          1   10% ▇
Aider code editing benchmark runs               1    4% ▇
SciCode leaderboard                             1    4% ▇
LMArena leaderboard dataset                     1    4% ▇
Arena-Hard-Auto v0.1 leaderboard                1    3% ▇
WebApp1K models leaderboard                     2    3% ▇
LiveBench model judgments                       1    2% ▇
OpenAI (submitted / maker-reported)             1    1% ▇
Vectara hallucination leaderboard               1    0% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 63.2
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · OpenAI simple-evals README benchmark results ([MIT](../../licenses/openai-simple-evals-MIT.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
Not counted (1) 
source  test                score  why
OpenAI  SWE-bench Verified  41.3%  too few models have it yet
## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/o1-preview) · [JSON](https://whatmodel.sh/api?format=json&all=1)