[whatmodel.sh](../../README.md) / GPT-4o (2024-08-06)
# GPT-4o (2024-08-06)
OpenAI · released 2024-08 · proprietary
**score 58.0** ±0.5 · #226 of 732 · from 56 benchmark results across 29 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks GPT-4o (2024-08-06) among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃▆●▄▅▅▄▄▂▁ #202 of 313 6.6% 6
coding ▁▁▁▃▅▇█●▅▄▃▁ #191 of 469 16% 7
chat & writing ▁▁▁▃▆▄▆●▆▇▇▃ #183 of 402 7.2% 3
math ▁▁▃▅█●▄█▄▂▁▁ #253 of 431 16% 8
vision & documents ▁▁▂▂▁▃●▂▆▇█▄ #113 of 153 7.4% 7
web search ▁▁▁▅▄█▃▄·●·▁ #6 of 84* 1.0% 1
agent tasks · #202 of 313 · 6.6% of its score
Beats [o1 (2024-12-17)](o1-2024-12-17.md), [o3-mini](o3-mini.md), [DeepSeek R1 0528](deepseek-r1-0528.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [Mellum2 12B-A2.5B Instruct](mellum2-12b-a2-5b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
ContractBench success rate (33 ob… 23.2% ▆··▆●▂▄▂▆▆█▄ beats 7/23 very hard · avg 13% 1.5% sharp · at its level
BrowserGym WorkArena-L1 (AgentLab… 45.5% █···█▄●·▄█·█ beats 5/10 hard · avg 28% 1.3% at its level
BrowserGym WebArena (AgentLab Gen… 31.4% █··▄·▄●·▄··▄ beats 5/7 hard · avg 17% 1.2% at its level
Finance Agent Benchmark v1 (537 q… 20.0% █·▃▃·█▅●▅▅▃▅ beats 10/18 very hard · avg 5.5% 1.0% sharp
BrowserGym WorkArena-L2 (AgentLab… 8.5% ▃▅▃·●▃▃··█·▃ beats 5/11 very hard · avg 3.6% 0.8% sharp · noisy
BrowserGym MiniWoB (AgentLab Gene… 63.8% ▅····●▅▃▃·▃█ beats 2/11 medium · avg 53% 0.8% blunt · at its level
coding · #191 of 469 · 16% of its score
Beats [Gemini 2.5 Flash](gemini-2-5-flash.md), [Devstral 2 (2512)](devstral-2.md), [Qwen3 30B-A3B Thinking 2507](qwen3-30b-a3b-thinking-2507.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 8B Reasoning 2512](ministral-3-8b-reasoning.md), [OpenThinker 32B](openthinker-32b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Aider polyglot (225 exercises, tw… 23.1% ▂·▂▃●▂▃█▆▄▂▂ beats 14/49 very hard · avg 7.6% 4.7% sharp · at its level
CRUXEval (ZeroEval, 800 problems) 87.0% ▁▂▆▃▆█▆▃●·▁▁ beats 45/47 medium · avg 57% 3.9% sharp
Aider code editing (133 Exercism … 71.4% ▁·▁▂▄▁▆▄█●▂▂ beats 52/62 medium · avg 51% 3.4% sharp · at its level
LiveCodeBench official (Aug 1, 20… 29.5% ▃▃●▃··▃▅█▅▅▃ beats 3/15 hard · avg 24% 1.4% sharp · at its level
HumanEval (Vector Institute, Insp… 90.9% ▄·▄·█·●·█·▄▄ beats 4/9 easy · avg 77% 1.4% sharp
WebApp1K-React pass@1 (onekq-ai) 88.5% ▃▁▁█▁▄▅▃▄●▁▄ beats 27/31 medium · avg 40% 0.8% sharp · at its level · noisy
HumanEval (simple-evals) (maker) 90.2% ▆█●▆▂··▂···▂ beats 7/15 easy · avg 83% 0.6% typical
chat & writing · #183 of 402 · 7.2% of its score
Beats [Seed-OSS 36B Instruct](seed-oss-36b.md), [Gemma 2 9B IT SimPO](gemma-2-9b-simpo.md), [Nemotron-Cascade-14B-Thinking](nemotron-cascade-14b-thinking.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena-Hard (v0.1, GPT-4-1106 judg… 77.9% ▂▁▁▅█▅▄▃▃●▃▁ beats 49/60 medium · avg 47% 3.6% sharp
LMArena (text) 1335.6 ▁▁▁▂▃▃▃●▆▇█▃ beats 173/385 avg 1260 2.1% blunt
Arabic IFEval prompt-level (Incep… 70.8% ▁·▁▁▂▁▄▄▆▄█● beats 38/38 medium · avg 46% 1.5% sharp · at its level
math · #253 of 431 · 16% of its score
Trails [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md), [Phi-2 (2.7B)](phi-2.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 53.3% ▂▂▅▆▄●▆▃▅▂▃▄ beats 47/92 hard · avg 28% 4.1% sharp · at its level
MATH Level 5 (ZeroEval, 721 probl… 55.3% ▁▁▁▆█▂▇●▄··▂ beats 31/38 hard · avg 30% 2.8% sharp · at its level
MATH (Vector Institute, Inspect E… 70.5% ▄··█●▄··▄·▄█ beats 4/9 medium · avg 50% 2.7% sharp
FrontierMath tiers 1–3 v2 0.4% ●··▁▁▃▄▄█▆▄▃ beats 0/86 very hard · avg 1.2% 2.3% sharp · too hard for it
OTIS mock AIME 2024–2025 6.4% ▃▃●▃▃▃▇▇█▃▂▅ beats 36/197 very hard · avg 7.2% 1.5% sharp · at its level · noisy
GSM8K (ZeroEval, zero-shot) 96.2% ▁▄·▁▄█▅▅▅██● beats 42/43 very easy · avg 87% 1.0% sharp · maxed out
MATH (simple-evals, original test… 75.9% █····▄··▄··● beats 4/5 medium · avg 62% 0.8% at its level
MGSM (simple-evals) (maker) 90.0% ▁····▁▅▁●▁▃▁ beats 11/16 easy · avg 80% 0.7% typical
vision & documents · #113 of 153 · 7.4% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md) here, though they rank higher overall. Trails [Gemma 3 27B](gemma-3-27b.md), [Mistral Small 3.1 24B (2503)](mistral-small-3-1.md), [Llama 4 Scout (109B-A17B)](llama-4-scout.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena vision (style controlled) 1117.9 ▁▁▂▂▂▂●▄▅▅█▇ beats 33/152 avg 1061 1.4% blunt
CharXiv (reasoning questions) 45.9% ▂▅▃●▅▆▆▅▆█▆▂ beats 23/101 hard · avg 31% 1.2% at its level · small run
MMMU val (VHELM, zero-shot exact … 63.0% ▄▄··▄▄█··▄●█ beats 7/9 medium · avg 50% 1.1% at its level
MathVista testmini (VHELM, zero-s… 56.7% ▃·▅···▃▃▃·▃● beats 7/9 medium · avg 42% 1.0% at its level
CharXiv descriptive questions 79.5% ▂▄▄▅▇●██▇▂▁▂ beats 25/59 medium · avg 64% 1.0% small run
CapArena-Auto detailed image capt… 44.0000 ▃▃▃·▃█·▅▃▃●▃ beats 11/12 avg -4.498 0.9% typical
Kaleidoscope (in-language multimo… 58.3% █·█···█·█·●█ beats 4/5 medium · avg 46% 0.8% at its level
web search · #6 of 84 · 1.0% of its score
Beats [GPT-6 Astra](gpt-6-astra.md), [Claude Opus 5.5](claude-opus-5-5.md), [GPT-6.1 Sol](gpt-6-1-sol.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 20.0% █·▃▃·█▅●▅▅▃▅ beats 10/18 very hard · avg 5.5% 1.0% sharp
other tests (25)
test result vs the others in its run difficulty counts why
LiveBench 2024-07-26 56.7% ▁▁▁▂▂▃▂█▆▃▅● beats 60/61 medium · avg 38% 9.0% at its level
MMLU-Redux (ZeroEval, 2,778 quest… 88.3% ▁·▁▁▄▄▄▅█▄●▁ beats 51/53 easy · avg 75% 6.1% typical
ZebraLogic (ZeroEval, 1,000 puzzl… 31.7% ▁▁▃▆█●▁▁·▁▁▁ beats 49/57 hard · avg 15% 4.7% sharp · at its level
GPQA Diamond (simple-evals) (make… 53.1% █▃▃●·▃▃▅▃▃▃▅ beats 7/16 hard · avg 39% 3.8% sharp · at its level
GPQA Diamond 49.2% ▁▁▁▁▂▄●▄▅▇█▅ beats 69/211 hard · avg 39% 3.5% sharp · at its level
MMLU-Pro 74.7% ▃▂▅█▃·█▃●··▂ beats 20/23 medium · avg 54% 3.1% at its level
GPQA Diamond (Vector Institute, I… 51.0% ▃··▃▃▃●·▃·▃█ beats 4/9 hard · avg 40% 2.7% sharp · at its level
MMLU-Pro (Vector Institute, Inspe… 74.5% ▃····▃▅·●▅·█ beats 4/9 medium · avg 58% 1.7% at its level
MMLU (simple-evals, zero-shot CoT… 88.7% ▂▂·█▄▄●·█▂·▂ beats 10/16 easy · avg 79% 1.4% typical
KCL-Essay vanilla 24.3% ▁···▁▃●█▂▃▃▂ beats 5/23 hard · avg 17% 1.4% at its level
BRIDGE clinical text, zero-shot a… 44.2% ▁▁▁▁▁▂▂▃▄▆█● beats 91/97 hard · avg 31% 1.3% blunt · at its level
HindiGen-v1 3C3H score (Inception… 74.5% ▁▁▁▁·▂·█▆█●▁ beats 26/31 medium · avg 46% 1.1% at its level
ARC-AGI-1 public evaluation set (… 11.4% ▁·●▃▁·█▇▁·▇▃ beats 4/27 very hard · avg 4.0% 1.0% sharp · noisy
Polish EQ-Bench v2 (SpeakLeash) 75.1% ▁··▁▁▂▂▆▅▆█● beats 66/70 medium · avg 62% 0.9% typical
Open Persian LLM Leaderboard v2 a… 76.3% ▂▄▂▃█▄▃▆▃▅▂● beats 80/81 medium · avg 57% 0.9% blunt · at its level
EvalPerf correctness (118 tasks, … 86.7% ▂·▂▂▂▆▅▆▆▆●█ beats 24/29 easy · avg 74% 0.8% typical
KCL-MCQA vanilla 43.3% ▂···█▂▆▆●█▆▂ beats 12/23 very hard · avg 29% 0.7% typical
Icelandic LLM Leaderboard average… 41.1% ▁▁▃▂▂▄▁●▄▄▄█ beats 15/38 very hard · avg 14% 0.6% typical
SimpleQA Verified 26.0% ▁▁▃▂▃●▆▆█▄▂▅ beats 17/81 very hard · avg 8.0% 0.4% typical
Open PT-LLM Leaderboard average (… 83.8% ▃▁▁▁▁▁▁▁▂█▅● beats 272/276 medium · avg 72% 0.4% at its level · noisy
Chess Puzzles (Epoch) 13.0% █▁▁▁▁●▃▃▃▂▁▁ beats 63/137 very hard · avg 5.8% 0.3% sharp · too hard for it · noisy
DROP F1, 3-shot (simple-evals) (m… 79.8% ▂▂▄●▆█·▂···▄ beats 4/16 easy · avg 78% 0.2% blunt
ARC-AGI-2 public evaluation set (… 0.0% ●··▂▅▁▃▁▃·▁▃ beats 3/24 very hard · avg 0.3% 0.2% sharp · too hard for it · noisy
SimpleQA (simple-evals) (maker) 40.1% ▄·▁▁▁▁··●▁·▁ beats 9/14 hard · avg 17% 0.2% at its level · noisy
Vectara hallucination leaderboard… 90.4% ▁▂▁▂▅●▂▃▄▃▁▁ beats 47/98 very easy · avg 90% 0.0% blunt
source results share of the score
WildEval ZeroEval published results 5 19% ▇▇
Epoch AI 6 12% ▇
LiveBench model judgments 1 9% ▇
Vector Institute Evaluation Leaderboard 4 9% ▇
OpenAI simple-evals README benchmark re… 7 8% ▇
Aider polyglot benchmark runs 1 5% ▇
ServiceNow BrowserGym leaderboard 4 4% ▇
Arena-Hard-Auto v0.1 leaderboard 1 4% ▇
LMArena leaderboard dataset 2 4% ▇
Aider code editing benchmark runs 1 3% ▇
MMLU-Pro author-released evaluation runs 1 3% ▇
Inception Arabic/Hindi leaderboards 2 3% ▇
VHELM: A Holistic Evaluation of Vision … 2 2% ▇
Meridian Labs inspect_charxiv evaluatio… 2 2% ▇
Korean Canonical Legal Benchmark: Towar… 2 2% ▇
ContractBench: Can LLM Agents Preserve … 1 1% ▇
LiveCodeBench official website results 1 1% ▇
BRIDGE Medical Leaderboard 1 1% ▇
ARC Prize Foundation model_baseline out… 2 1% ▇
Finance Agent Benchmark 1 1% ▇
SpeakLeash Polish EQ-Bench leaderboard 1 1% ▇
CapArena: Benchmarking and Analyzing De… 1 1% ▇
Open Persian LLM Leaderboard v2 1 1% ▇
EvalPlus / EvalPerf published evaluatio… 1 1% ▇
WebApp1K models leaderboard 1 1% ▇
Kaleidoscope: In-language Exams for Mas… 1 1% ▇
Miðeind Icelandic LLM Leaderboard 1 1% ▇
Open Portuguese LLM Leaderboard 1 0% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 58.0
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Vector Institute Evaluation Leaderboard ([Apache-2.0](../../licenses/vector-inspect-evals-leaderboard-Apache-2.0.txt)) · OpenAI simple-evals README benchmark results ([MIT](../../licenses/openai-simple-evals-MIT.txt)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · ServiceNow BrowserGym leaderboard ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · MMLU-Pro author-released evaluation runs ([Apache-2.0](../../licenses/mmlu-pro-author-runs-Apache-2.0.txt)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · VHELM: A Holistic Evaluation of Vision Language Models ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Meridian Labs inspect\_charxiv evaluation report ([MIT](https://github.com/meridianlabs-ai/inspect_charxiv/blob/317e4ff4335d7d4242d8e3c803898b7345973bd2/LICENSE)) · Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities ([CC BY 4.0](../../licenses/cards-final-NOTICE.txt)) · ContractBench: Can LLM Agents Preserve Observation Contracts? ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · BRIDGE Medical Leaderboard ([MIT](../../licenses/bridge-medical-leaderboard-MIT.txt)) · ARC Prize Foundation model\_baseline outputs on ARC-AGI-1/2 public evaluation sets ([MIT](../../licenses/arcprize-model-baseline-outputs-MIT.txt)) · Finance Agent Benchmark ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · SpeakLeash Polish EQ-Bench leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Open Persian LLM Leaderboard v2 ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Miðeind Icelandic LLM Leaderboard ([CC BY 4.0](../../licenses/mideind-icelandic-llm-leaderboard-CC-BY-4.0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
### Through an API
api (OpenAI) · $4.38/M blended · $2.5 in / $10 out per 1M tokens · [developers.openai.com](https://developers.openai.com/api/docs/models/gpt-4o)
[price source](https://developers.openai.com/api/docs/models/gpt-4o): USD per 1M tokens, OpenAI API standard processing, model page 'gpt-4o' whose default snapshot is gpt-4o-2024-08-06: input $2.5, output $10; cached-input, Batch and other tiers excluded. Currency: USD. OpenAI official model page, observed 2026-09-28. · [context source](https://developers.openai.com/api/docs/models/gpt-4o): OpenAI model page for gpt-4o (default snapshot gpt-4o-2024-08-06): 128,000 context window, 16,384 max output. Observed 2026-09-28.
[interactive version](https://whatmodel.sh/model/gpt-4o-2024-08-06) · [JSON](https://whatmodel.sh/api?format=json&all=1)