[whatmodel.sh](../../README.md) / Claude 3 Opus
# Claude 3 Opus
Anthropic · released 2024-02 · proprietary
**score 53.6** ±0.4 · comparison only: no current run option verified · from 37 benchmark results across 21 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Claude 3 Opus among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▂▆●▄▅▄▄▃▁▁ #270 of 354* 0.4% 1
coding ▁▁▁▃▄▆●▇▅▄▃▁ #280 of 539 35% 7
chat & writing ▁▁▁▃▅▄▇●▇▇▇▃ #253 of 474 16% 4
math ▁▁▃▆●█▅█▃▁▁▁ #356 of 530 8.8% 4
vision & documents ▁▁▂▃●▃▅▄▆██▄ #160 of 189 8.2% 9
agent tasks · #270 of 354 · 0.4% of its score
Beats [o3-mini](o3-mini.md), [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [Nova 2 Lite](nova-2-lite.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
BFCL v3 57.0% ▁▁▁▁▁▃▅●▅▄▁▁ beats 68/114 medium · avg 51% 0.4% blunt · at its level
coding · #280 of 539 · 35% of its score
Beats [Qwen3.5 4B](qwen3-5-4b.md), [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Solar Open 100B (102B-A12B)](solar-open-100b.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 3B Reasoning 2512](ministral-3-3b-reasoning.md), [GPT-4 (0613)](gpt-4-0613.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 82.9% ▁·▁▁▃▄▆▅█●▁▁ beats 88/99 easy · avg 76% 13% sharp · several runs
HumanEval+ 77.4% ▁·▁▁▁▃▄▄▄█●▁ beats 88/99 easy · avg 70% 7.5% sharp
CRUXEval (ZeroEval, 800 problems) 70.4% ▁▂▆▃▆●▆▃▂·▁▁ beats 30/47 medium · avg 57% 4.5% sharp · at its level
LiveCodeBench official (Jan 1–Nov… 31.9% ▁▁·▁▂▄▆●▅▂·▂ beats 16/31 hard · avg 23% 4.1% sharp · at its level
Aider code editing (133 Exercism … 68.4% ▁·▁▂▄▁▆▄●▆▂▂ beats 45/62 medium · avg 51% 3.4% sharp · at its level
MBPP+ 73.3% ▁···▁▁▂▅▇█●▂ beats 68/75 easy · avg 66% 1.8% at its level
SciCode main problems (official l… 1.5% ▃···●··▂·▃·▂ beats 6/17 very hard · avg 1.0% 0.7% sharp · too hard for it
chat & writing · #253 of 474 · 16% of its score
Beats [Gemma 2 9B IT SimPO](gemma-2-9b-simpo.md), [Granite 4.2 8B](granite-4-2-8b.md), [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [Granite 4.0 H Micro 3B](granite-4-0-h-micro.md), [Gemma 3n E4B](gemma-3n-e4b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
AlpacaEval 2 length-controlled 40.5% ▁▂▂▃▆▆█▆●▄▁▁ beats 98/122 medium · avg 35% 8.0% sharp · at its level
Arena-Hard (v0.1, GPT-4-1106 judg… 60.4% ▂▁▁▅█▅▄●▃▅▃▁ beats 41/60 medium · avg 47% 4.3% sharp · at its level
LMArena (text) 1322.2 ▁▁▁▂▃▃▃●▆▇█▃ beats 156/385 avg 1260 1.8% blunt
Polyrating rating (bias-aware re-… 1273.0 ▂▁▂▁▃▇█▆▇▄▂● beats 93/95 avg 1187 1.7% typical
math · #356 of 530 · 8.8% of its score
Beats [Molmo 2 8B](molmo-2-8b.md), [DeepSeek-Coder-V2 Instruct (0724)](deepseek-coder-v2-0724.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [Mistral 7B v0.1 (base)](mistral-7b-v0-1.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 37.5% ▂▂▅▆●█▆▃▅▂▃▄ beats 29/92 hard · avg 28% 4.1% sharp · at its level
MATH Level 5 (ZeroEval, 721 probl… 36.9% ▁▁▁▆█●▇▇▄··▂ beats 18/38 hard · avg 30% 2.4% sharp · at its level
GSM8K (ZeroEval, zero-shot) 95.6% ▁▄·▁▄█▅▅▅██● beats 38/43 very easy · avg 87% 1.5% sharp
OTIS mock AIME 2024–2025 4.7% ▃▃●▃▃▃▇▇█▃▂▅ beats 31/197 very hard · avg 7.2% 0.8% sharp · noisy
vision & documents · #160 of 189 · 8.2% of its score
Trails [Qwen3.5 2B](qwen3-5-2b.md), [Qwen3-VL 2B Thinking](qwen3-vl-2b-thinking.md), [Qwen2.5-VL 7B Instruct](qwen2-5-vl-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
CharXiv descriptive questions 71.5% ▄█▄▄▄·●█·▄·█ beats 6/12 easy · avg 69% 1.9% typical
Arena vision (style controlled) 1061.5 ▁▁▂▂▂●▃▄▅▅█▇ beats 23/152 avg 1061 1.2% blunt
CharXiv (reasoning questions) 30.2% ▂▂●█▃▂·▂▂··▂ beats 2/13 hard · avg 34% 1.2% at its level
MMMU val (VHELM, zero-shot exact … 53.2% ▄▄··●▄█··▄▄█ beats 2/9 medium · avg 50% 1.0% at its level
CharXiv descriptive questions (te… 72.2% █··▄··●··▄·▄ beats 3/6 easy · avg 69% 1.0% at its level
MathVista testmini (VHELM, zero-s… 40.5% ▃·●···▃▃▃·▃█ beats 2/9 medium · avg 42% 0.9% at its level
CharXiv reasoning questions (test… 28.8% ▄··▄●▄█····▄ beats 2/6 hard · avg 30% 0.6% at its level
PhysBench physical-world understa… 37.0% ●▃█··▅▃····▃ beats 1/9 hard · avg 42% 0.2% blunt · at its level
Ai2 Molmo human preference Elo (S… 971.0 ●▄·▄·▄▄▄·▄·█ beats 0/8 avg 1032 0.2% blunt
other tests (11)
test result vs the others in its run difficulty counts why
LiveBench 2024-07-26 50.6% ▁▁▁▂▂▃▂█▆▃●▃ beats 53/61 medium · avg 38% 7.8% at its level
MMLU-Redux (ZeroEval, 2,778 quest… 82.5% ▁·▁▁▄▄▄▅●▄▂▁ beats 37/53 easy · avg 75% 6.8% at its level
EvoEval (overall) 62.9% ▂▂▃▄▄█▁▄▅▃▁● beats 40/42 medium · avg 52% 5.0% at its level
ZebraLogic (ZeroEval, 1,000 puzzl… 27.0% ▁▁▃▆█●▁▁·▁▁▁ beats 42/57 hard · avg 15% 3.2% sharp
GPQA Diamond 47.2% ▁▁▁▁▂▄●▄▅▇█▅ beats 56/211 hard · avg 39% 2.5% sharp · at its level
BigCodeBench (Instruct) 45.2% ▃▂·▆▆██▆▅▃●▃ beats 30/33 medium · avg 39% 2.2% at its level
BigCodeBench (Complete) 56.8% ▂▁▁▄▂▅▇█▆▂▂● beats 56/58 medium · avg 48% 2.1% at its level
WildBench v2 WB-Score (task macro) 51.7140 WB-Score ▁·▁▁·▃█▄▇▆●▆ beats 41/52 avg 43.019 WB-Score 2.0% sharp
SimpleQA Verified 12.6% ▁▁●▂▃▃▆▆█▄▂▅ beats 5/81 very hard · avg 8.0% 0.3% typical
Chess Puzzles (Epoch) 5.0% ●▁▁▁▁▂▃▃▃▂▁▁ beats 16/137 very hard · avg 5.8% 0.1% sharp · too hard for it · noisy
Vectara hallucination leaderboard… 89.9% ▁▁▂●▄▄█▆▃▂▂▁ beats 25/140 very easy · avg 94% 0.1% blunt · maxed out · noisy
source results share of the score
WildEval ZeroEval published results 5 18% ▇▇
EvalPlus / EvalPerf published evaluatio… 3 17% ▇▇
Top Leaderboard Ranking = Top Coding Pr… 2 10% ▇
AlpacaEval 1.0 and 2.0 leaderboards 1 8% ▇
LiveBench model judgments 1 8% ▇
Epoch AI 5 8% ▇
Arena-Hard-Auto v0.1 leaderboard 1 4% ▇
BigCodeBench paper 2 4% ▇
LiveCodeBench official website results 1 4% ▇
Aider code editing benchmark runs 1 3% ▇
CharXiv leaderboard 2 3% ▇
LMArena leaderboard dataset 2 3% ▇
WildBench v2 leaderboard data 1 2% ▇
VHELM: A Holistic Evaluation of Vision … 2 2% ▇
Polyrating: A Cost-Effective and Bias-A… 1 2% ▇
CharXiv: Charting Gaps in Realistic Cha… 2 2% ▇
SciCode leaderboard 1 1% ▇
Berkeley Function Calling Leaderboard r… 1 0% ▇
PhysBench: Benchmarking and Enhancing V… 1 0% ▇
Ai2 Molmo-72B-0924 model card: Human Pr… 1 0% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 53.6
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · Top Leaderboard Ranking = Top Coding Proficiency ([CC BY 4.0](../../licenses/paper-2403.19114v1-CC-BY-4.0.txt)) · AlpacaEval 1.0 and 2.0 leaderboards ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · BigCodeBench paper ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · CharXiv leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · WildBench v2 leaderboard data ([Apache-2.0](https://github.com/allenai/WildBench/blob/d6b8dcaf377d173d031980f97c16e1a82618c03d/LICENSE)) · VHELM: A Holistic Evaluation of Vision Language Models ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs ([CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column ([Apache-2.0](https://huggingface.co/allenai/Molmo-72B-0924/blob/ab586dd9640de059566975b8e5bcd0c6fdc73956/README.md)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/claude-3-opus-20240229) · [JSON](https://whatmodel.sh/api?format=json&all=1)