[whatmodel.sh](../../README.md) / Claude 3 Sonnet
# Claude 3 Sonnet
Anthropic · released 2024-02 · proprietary
**score 50.0** ±0.4 · comparison only: no current run option verified · from 30 benchmark results across 18 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Claude 3 Sonnet among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁▁▃▄●█▇▅▄▃▁ #357 of 539 38% 7
chat & writing ▁▁▁▃▅▄●█▇▇▇▃ #307 of 474 17% 4
math ▁▁▃●██▅█▃▁▁▁ #422 of 530 6.6% 3
vision & documents ▁▁▂●▂▃▅▄▆██▄ #166 of 189 9.3% 8
coding · #357 of 539 · 38% of its score
Beats [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Solar Open 100B (102B-A12B)](solar-open-100b.md), [Qwen3-VL 30B-A3B Instruct](qwen3-vl-30b-a3b-instruct.md) here, though they rank higher overall. Trails [DeepSeek LLM 67B Chat](deepseek-llm-67b-chat.md), [WizardLM 70B V1.0](wizardlm-70b-v1-0.md), [Yi 1.5 9B Chat](yi-1-5-9b-chat.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 70.7% ▁·▁▁▃▄▆●█▃▁▁ beats 59/99 easy · avg 76% 12% sharp
HumanEval+ 64.0% ▁·▁▁▁▃▄▄▄●▃▁ beats 58/99 easy · avg 70% 10% sharp · at its level
CRUXEval (ZeroEval, 800 problems) 66.6% ▁▂▆▃▆●▆▃▂·▁▁ beats 24/47 medium · avg 57% 5.6% sharp · at its level
Aider code editing (133 Exercism … 54.9% ▁·▁▂▄▁●▄█▆▂▂ beats 22/62 medium · avg 51% 4.0% sharp · at its level
LiveCodeBench official (Jan 1–Nov… 25.0% ▁▁·▁▂▄●█▅▂·▂ beats 10/31 hard · avg 23% 3.8% sharp
MBPP+ 69.3% ▁···▁▁▂▅▇█●▂ beats 62/75 easy · avg 66% 2.2% at its level
SciCode main problems (official l… 1.5% ▃···●··▂·▃·▂ beats 5/17 very hard · avg 1.0% 0.5% sharp · too hard for it
chat & writing · #307 of 474 · 17% of its score
Beats [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md), [Qwen2.5 Coder 32B Instruct](qwen2-5-coder-32b.md), [Ministral 3 14B Reasoning 2512](ministral-3-14b-reasoning.md) here, though they rank higher overall. Trails [Flan-T5 XXL](flan-t5-xxl.md), [Granite 4.0 H 1B](granite-4-0-h-1b.md), [Granite 4.0 1B](granite-4-0-1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
AlpacaEval 2 length-controlled 34.9% ▁▂▂▃▆▆█●▄▄▁▁ beats 89/122 medium · avg 35% 8.2% sharp · at its level
Arena-Hard (v0.1, GPT-4-1106 judg… 46.8% ▂▁▁▅█▅●▃▃▅▃▁ beats 36/60 medium · avg 47% 5.1% sharp · at its level
LMArena (text) 1281.4 ▁▁▁▂▃▃●▆▆▇█▃ beats 112/385 avg 1260 2.0% blunt
Polyrating rating (bias-aware re-… 1220.0 ▂▁▂▁▃▇█▆▇●▂▂ beats 87/95 avg 1187 1.9% typical
math · #422 of 530 · 6.6% of its score
Beats [Athene-70B](athene-70b.md) here, though they rank higher overall. Trails [OPT-2.7B](opt-2-7b.md), [Gemma 2B (base)](gemma-2b-base.md), [Stable Code 3B](stable-code-3b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 18.2% ▂▂▅●▄█▆▃▅▂▃▄ beats 16/92 hard · avg 28% 3.7% sharp · at its level
GSM8K (ZeroEval, zero-shot) 91.5% ▁▄·▁▄█▅▅●███ beats 25/43 very easy · avg 87% 2.5% sharp
OTIS mock AIME 2024–2025 2.5% ▃●▄▃▃▃▇▇█▃▂▅ beats 19/197 very hard · avg 7.2% 0.4% sharp · too hard for it · noisy
vision & documents · #166 of 189 · 9.3% of its score
Beats [Nova Pro](nova-pro.md) here, though they rank higher overall. Trails [Qwen3.5 2B](qwen3-5-2b.md), [InternVL2-26B](internvl2-26b.md), [Qwen3-VL 2B Thinking](qwen3-vl-2b-thinking.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
CharXiv descriptive questions 73.7% ▄█▄▄▄·█●·▄·█ beats 8/12 easy · avg 69% 2.4% at its level
Arena vision (style controlled) 1016.3 ▁▁▂●▂▂▃▄▅▅█▇ beats 13/152 avg 1061 1.4% blunt
CharXiv (reasoning questions) 32.2% ▂▂▂●▃▂·▂▂··▂ beats 6/13 hard · avg 34% 1.3% at its level
CharXiv descriptive questions (te… 72.8% █··▄··●··▄·▄ beats 4/6 easy · avg 69% 1.2% at its level
MMMU val (VHELM, zero-shot exact … 44.7% ●▄··▄▄█··▄▄█ beats 0/9 medium · avg 50% 1.2% at its level
MathVista testmini (VHELM, zero-s… 39.3% ▃·●···▃▃▃·▃█ beats 1/9 medium · avg 42% 1.0% at its level
CharXiv reasoning questions (test… 32.4% ▄··▄▄▄●····▄ beats 4/6 hard · avg 30% 0.6% at its level
PhysBench physical-world understa… 36.2% ●▃█··▅▃····▃ beats 0/9 hard · avg 42% 0.2% blunt
other tests (8)
test result vs the others in its run difficulty counts why
MMLU-Redux (ZeroEval, 2,778 quest… 74.9% ▁·▁▁▄▄●▅█▄▂▁ beats 19/53 easy · avg 75% 8.9% at its level
LiveBench 2024-07-26 38.7% ▁▁▁▂▂▃▂█●▃▅▃ beats 42/61 medium · avg 38% 8.3% at its level
ZebraLogic (ZeroEval, 1,000 puzzl… 18.7% ▁▁▃▆●█▁▁·▁▁▁ beats 24/57 hard · avg 15% 2.6% sharp
BigCodeBench (Instruct) 42.5% ▃▂·▆▆██▆▅●▃▃ beats 28/33 medium · avg 39% 2.4% at its level
BigCodeBench (Complete) 53.4% ▂▁▁▄▂▅▇█▆▂●▂ beats 53/58 medium · avg 48% 2.3% at its level
WildBench v2 WB-Score (task macro) 45.4815 WB-Score ▁·▁▁·▃█▄▇●▅▆ beats 34/52 avg 43.019 WB-Score 2.2% sharp
GPQA Diamond 40.6% ▁▁▁▁▂●▆▄▅▇█▅ beats 40/211 hard · avg 39% 2.1% sharp
Vectara hallucination leaderboard… 83.7% ▁▁●▃▄▄█▆▃▂▂▁ beats 10/140 very easy · avg 94% 0.1% blunt · maxed out · noisy
source results share of the score
EvalPlus / EvalPerf published evaluatio… 3 24% ▇▇▇
WildEval ZeroEval published results 4 20% ▇▇
LiveBench model judgments 1 8% ▇
AlpacaEval 1.0 and 2.0 leaderboards 1 8% ▇
Epoch AI 3 6% ▇
Arena-Hard-Auto v0.1 leaderboard 1 5% ▇
BigCodeBench paper 2 5% ▇
Aider code editing benchmark runs 1 4% ▇
LiveCodeBench official website results 1 4% ▇
CharXiv leaderboard 2 4% ▇
LMArena leaderboard dataset 2 3% ▇
WildBench v2 leaderboard data 1 2% ▇
VHELM: A Holistic Evaluation of Vision … 2 2% ▇
Polyrating: A Cost-Effective and Bias-A… 1 2% ▇
CharXiv: Charting Gaps in Realistic Cha… 2 2% ▇
SciCode leaderboard 1 0% ▇
PhysBench: Benchmarking and Enhancing V… 1 0% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 50.0
Licences: EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · AlpacaEval 1.0 and 2.0 leaderboards ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · BigCodeBench paper ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · CharXiv leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · WildBench v2 leaderboard data ([Apache-2.0](https://github.com/allenai/WildBench/blob/d6b8dcaf377d173d031980f97c16e1a82618c03d/LICENSE)) · VHELM: A Holistic Evaluation of Vision Language Models ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs ([CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/claude-3-sonnet-20240229) · [JSON](https://whatmodel.sh/api?format=json&all=1)