[whatmodel.sh](../../README.md) / GPT-4o (2024-05-13)
# GPT-4o (2024-05-13)
OpenAI · released 2024-05 · proprietary
**score 57.8** ±0.4 · #230 of 732 · from 59 benchmark results across 25 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks GPT-4o (2024-05-13) among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃▆█▄●▅▄▄▂▁ #138 of 313* 0.3% 1
coding ▁▁▁▃▅▇█●▅▄▃▁ #190 of 469 16% 8
chat & writing ▁▁▁▃▆▄▆█●▇▇▃ #177 of 402 17% 6
math ▁▁▃▅█●▄█▄▂▁▁ #257 of 431 11% 9
vision & documents ▁▁▂▂▁▃●▂▆▇█▄ #102 of 153 11% 14
agent tasks · #138 of 313 · 0.3% of its score
Beats [o3](o3.md), [Gemini 2.5 Pro](gemini-2-5-pro.md), [Gemini 3.1 Flash-Lite](gemini-3-1-flash-lite.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Large](jamba-1-5-large.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
BFCL (version not stated) 82.9% █·········█● beats 5/5 easy · avg 77% 0.3% blunt
coding · #190 of 469 · 16% of its score
Beats [Gemini 2.5 Flash](gemini-2-5-flash.md), [Devstral 2 (2512)](devstral-2.md), [Qwen3 30B-A3B Thinking 2507](qwen3-30b-a3b-thinking-2507.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 8B Reasoning 2512](ministral-3-8b-reasoning.md), [OpenThinker 32B](openthinker-32b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LiveCodeBench official (Jan 1–Nov… 40.1% ▁▁·▁▂▄▆█●▂·▂ beats 23/31 hard · avg 23% 4.8% sharp · at its level
CRUXEval (ZeroEval, 800 problems) 86.1% ▁▂▆▃▆█▆▃●·▁▁ beats 44/47 medium · avg 57% 3.6% sharp
Aider code editing (133 Exercism … 72.9% ▁·▁▂▄▁▆▄█●▂▂ beats 56/62 medium · avg 51% 3.1% sharp · at its level
HumanEval+ 84.9% ▁·▁▁▂▃▄▄█▅●▃ beats 118/145 easy · avg 77% 1.5% sharp
SciCode main problems (official l… 1.5% ▃···●··▂·▃·▂ beats 9/17 very hard · avg 1.0% 1.1% sharp · too hard for it
WebApp1K-React pass@1 (onekq-ai) 87.0% ▃▁▁█▁▄▅▃▄●▁▄ beats 24/31 medium · avg 40% 0.8% sharp · at its level · noisy
HumanEval (simple-evals) (maker) 91.0% ▆█●▆▂··▂···▂ beats 9/15 easy · avg 83% 0.6% typical
LiveCodeBench v2 50.5% ▄··●·▄···▄▄█ beats 2/7 medium · avg 50% 0.4% at its level
chat & writing · #177 of 402 · 17% of its score
Beats [MiniCPM5 2B](minicpm5-2b.md), [Seed-OSS 36B Instruct](seed-oss-36b.md), [GPT-5 nano](gpt-5-nano.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
AlpacaEval 2 length-controlled 57.5% ▁▂▂▃▆▆█▆▄●▁▁ beats 115/122 medium · avg 35% 8.4% sharp · at its level
Arena-Hard (v0.1, GPT-4-1106 judg… 79.2% ▂▁▁▅█▅▄▃▃●▃▁ beats 52/60 medium · avg 47% 3.3% sharp
LMArena (text) 1346.1 ▁▁▁▂▃▃▃▆●▇█▃ beats 187/385 avg 1260 1.9% blunt
Polyrating rating (bias-aware re-… 1297.0 ▂▁▂▁▃▇█▆▇▄▂● beats 95/95 avg 1187 1.8% typical
Arena-Hard (version/judge unspeci… 69.0% ▃▃▂▅▃▆▅●▅▅▂▃ beats 19/30 medium · avg 59% 1.0% at its level
IFEval (aggregation unspecified) 81.8% ▁▂▄▂▂▄●██▅▂▁ beats 59/145 easy · avg 74% 0.3% typical
math · #257 of 431 · 11% of its score
Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 51.0% ▂▂▅▆▄●▆▃▅▂▃▄ beats 44/92 hard · avg 28% 3.8% sharp · at its level
MATH Level 5 (ZeroEval, 721 probl… 54.8% ▁▁▁▆█▂▇●▄··▂ beats 30/38 hard · avg 30% 2.6% sharp · at its level
OTIS mock AIME 2024–2025 6.3% ▃▃●▃▃▃▇▇█▃▂▅ beats 35/197 very hard · avg 7.2% 1.4% sharp · at its level · noisy
GSM8K (ZeroEval, zero-shot) 95.4% ▁▄·▁▄█▅▅▅█●█ beats 36/43 very easy · avg 87% 1.0% sharp · maxed out
MATH (simple-evals, original test… 76.6% █····▄··▄··● beats 5/5 medium · avg 62% 0.7% at its level
MathVista (testmini) 64.6% ▂····▂···▂█● beats 8/9 medium · avg 51% 0.7% at its level · noisy · several runs
MGSM (simple-evals) (maker) 89.9% ▁····▁▅▁●▁▃▁ beats 9/16 easy · avg 80% 0.7% typical
AIME 2024 8.7% ▁▁▁●▁▃▁▅███▃ beats 8/85 medium · avg 40% 0.2% sharp · at its level · noisy
MATH-500 75.8% ▁▃▂▃●▇▅▇█▄▂▁ beats 19/85 easy · avg 84% 0.1% noisy
vision & documents · #102 of 153 · 11% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-4o (2024-08-06)](gpt-4o-2024-08-06.md) here, though they rank higher overall. Trails [Llama 4 Maverick (402B-A17B)](llama-4-maverick.md), [Qwen3-VL 8B Thinking](qwen3-vl-8b-thinking.md), [GPT-4o (2024-11-20)](gpt-4o-2024-11-20.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
CharXiv descriptive questions 84.5% ▄█▄▄▄·██·▄·● beats 12/12 easy · avg 69% 1.6% typical
CharXiv (reasoning questions) 47.1% ▂▂▂█▃▂·▂●··▂ beats 12/13 hard · avg 34% 1.3% at its level
Arena vision (style controlled) 1161.8 ▁▁▂▂▂▂▃●▅▅█▇ beats 57/152 avg 1061 1.3% blunt
MMMU (val) 68.6% ▂·▂······█▄● beats 9/9 medium · avg 51% 1.1% at its level · several runs
MMMU val (VHELM, zero-shot exact … 64.0% ▄▄··▄▄█··▄▄● beats 8/9 medium · avg 50% 1.0% at its level
MathVista testmini (VHELM, zero-s… 55.1% ▃·▅···▃▃▃·●█ beats 6/9 medium · avg 42% 0.9% at its level
CharXiv descriptive questions (te… 84.9% █··▄··█··▄·● beats 6/6 easy · avg 69% 0.9% typical
MathVista (testmini) 64.6% ▂····▂···▂█● beats 8/9 medium · avg 51% 0.7% at its level · noisy · several runs
CharXiv reasoning questions (test… 47.0% ▄··▄▄▄█····● beats 6/6 hard · avg 30% 0.7% at its level
MM-MT-Bench (Mistral, GPT-4o judg… 7.7200 MM-MT-Bench score (1-10) ▅··▃·█·▃▃··● beats 9/9 avg 5.605 MM-MT-Bench s… 0.6% sharp · noisy
VQAv2 (val) 77.8% ▃···█▃··▅●▃▃ beats 7/9 easy · avg 73% 0.3% blunt
TextVQA 75.6% ▁▁▁▄▅▁●·▃▃▃█ beats 12/24 easy · avg 75% 0.1% blunt
ChartQA 85.1% ▄·▄▄▄··▄▄▄▄● beats 8/9 easy · avg 78% 0.1% blunt · noisy · several runs
AI2D test 82.8% ▁▁▅▇▃●·▄·█·▃ beats 13/28 easy · avg 82% 0.0% blunt
other tests (17)
test result vs the others in its run difficulty counts why
LiveBench 2024-07-26 54.6% ▁▁▁▂▂▃▂█▆▃▅● beats 59/61 medium · avg 38% 8.2% at its level
MMLU-Redux (ZeroEval, 2,778 quest… 88.0% ▁·▁▁▄▄▄▅█▄●▁ beats 50/53 easy · avg 75% 5.6% typical
ZebraLogic (ZeroEval, 1,000 puzzl… 28.2% ▁▁▃▆█●▁▁·▁▁▁ beats 45/57 hard · avg 15% 4.3% sharp · at its level
GPQA Diamond 48.9% ▁▁▁▁▂▄●▄▅▇█▅ beats 64/211 hard · avg 39% 4.1% sharp · at its level · several runs
MMLU-Pro 72.1% ▃▂▅█▃·█●▅··▂ beats 19/23 medium · avg 54% 4.0% at its level · several runs
GPQA Diamond (simple-evals) (make… 49.9% █▃▃●·▃▃▅▃▃▃▅ beats 5/16 hard · avg 39% 3.4% sharp · at its level
CLIcK Korean (Daekeun Kim) 80.5% ▆▄▄▄█●▂·▄█·▄ beats 13/23 medium · avg 56% 2.7% at its level
BigCodeBench (Instruct) 49.9% ▃▂·▆▆██▆▅▃▃● beats 33/33 medium · avg 39% 2.4% at its level
HAE-RAE 1.0 (Daekeun Kim) 85.7% ▄▄▄▂▂█▂●▂█▂▄ beats 13/22 medium · avg 61% 2.2% typical
BigCodeBench (Complete) 60.2% ▂▁▁▄▂▅▇█▆▂▂● beats 58/58 medium · avg 48% 2.1% at its level
WildBench v2 WB-Score (task macro) 59.2982 WB-Score ▁·▁▁·▃█▄▇▆▅● beats 51/52 avg 43.019 WB-Score 2.1% sharp
KMMLU-HARD (Daekeun Kim, zero-sho… 39.6% █···▁▁●▂▃·▆▂ beats 10/23 very hard · avg 29% 1.4% sharp
MMLU (simple-evals, zero-shot CoT… 87.2% ▂▂·█▄●▂·█▂·▂ beats 8/16 easy · avg 79% 1.3% typical
HindiGen-v1 3C3H score (Inception… 74.0% ▁▁▁▁·▂·█▆█●▁ beats 25/31 medium · avg 46% 1.0% at its level
FlenQA (3K-token subset) 90.8% ▄···●····▄·█ beats 1/4 easy · avg 86% 0.6% sharp · noisy
DROP F1, 3-shot (simple-evals) (m… 83.7% ▂▂▄▆▆●·▂···▄ beats 12/16 easy · avg 78% 0.2% blunt
SimpleQA (simple-evals) (maker) 39.0% ▄·▁▁▁▁··●▁·▁ beats 8/14 hard · avg 17% 0.2% at its level · noisy
source results share of the score
WildEval ZeroEval published results 5 17% ▇▇
Epoch AI 3 8% ▇
AlpacaEval 1.0 and 2.0 leaderboards 1 8% ▇
LiveBench model judgments 1 8% ▇
OpenAI simple-evals README benchmark re… 7 7% ▇
Daekeun Kim Korean LLM benchmark runs 3 6% ▇
LiveCodeBench official website results 1 5% ▇
Microsoft 5 5% ▇
BigCodeBench paper 2 4% ▇
Arena-Hard-Auto v0.1 leaderboard 1 3% ▇
LMArena leaderboard dataset 2 3% ▇
Aider code editing benchmark runs 1 3% ▇
CharXiv leaderboard 2 3% ▇
MMLU-Pro author-released evaluation runs 1 3% ▇
WildBench v2 leaderboard data 1 2% ▇
VHELM: A Holistic Evaluation of Vision … 2 2% ▇
Polyrating: A Cost-Effective and Bias-A… 1 2% ▇
Pixtral 12B 5 2% ▇
open-thoughts/OpenThinker-7B model card 4 2% ▇
CharXiv: Charting Gaps in Realistic Cha… 2 2% ▇
Microsoft Phi-3.5-vision-instruct model… 5 1% ▇
SciCode leaderboard 1 1% ▇
Inception Arabic/Hindi leaderboards 1 1% ▇
WebApp1K models leaderboard 1 1% ▇
ChatGLM: A Family of Large Language Mod… 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 57.8
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · AlpacaEval 1.0 and 2.0 leaderboards ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · OpenAI simple-evals README benchmark results ([MIT](../../licenses/openai-simple-evals-MIT.txt)) · Daekeun Kim Korean LLM benchmark runs ([MIT](../../licenses/korean-personal-benchmark-runs-MIT.txt)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · Microsoft ([CC BY 4.0](../../licenses/report-2504.21318v1.txt)) · BigCodeBench paper ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · CharXiv leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · MMLU-Pro author-released evaluation runs ([Apache-2.0](../../licenses/mmlu-pro-author-runs-Apache-2.0.txt)) · WildBench v2 leaderboard data ([Apache-2.0](https://github.com/allenai/WildBench/blob/d6b8dcaf377d173d031980f97c16e1a82618c03d/LICENSE)) · VHELM: A Holistic Evaluation of Vision Language Models ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Pixtral 12B ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · open-thoughts/OpenThinker-7B model card ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs ([CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)) · Microsoft Phi-3.5-vision-instruct model card: Benchmarks table ([MIT](https://huggingface.co/microsoft/Phi-3.5-vision-instruct/blob/12b77fb40b63a2c73c68243d3f767aab688a1b2a/LICENSE)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/))
## How to run it
### Through an API
api (OpenAI) · $7.50/M blended · $5 in / $15 out per 1M tokens · [developers.openai.com](https://developers.openai.com/api/docs/models/gpt-4o)
[price source](https://developers.openai.com/api/docs/pricing): USD per 1M tokens, OpenAI API Standard processing, model gpt-4o-2024-05-13 (listed as its own row in the text-token pricing table; deprecated snapshot, shutdown scheduled 2026-10-23): input $5, output $15; no cached-input rate listed. Batch ($2.50/$7.50) and Priority ($8.75/$26.25) tiers excluded. Currency: USD. OpenAI official pricing page, observed 2026-10-07. · [context source](https://developers.openai.com/api/docs/models/gpt-4o): Context limit from the maker's documentation, observed 2026-10-06.
[interactive version](https://whatmodel.sh/model/gpt-4o-2024-05-13) · [JSON](https://whatmodel.sh/api?format=json&all=1)