[whatmodel.sh](../../README.md) / Claude 3.5 Sonnet (2024-06-20)
# Claude 3.5 Sonnet (2024-06-20)
Anthropic · released 2024-06 · proprietary
**score 57.7** ±0.4 · comparison only: no current run option verified · from 44 benchmark results across 22 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Claude 3.5 Sonnet (2024-06-20) among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁▁▃▄▆█●▅▄▃▁ #228 of 539 30% 9
chat & writing ▁▁▁▃▅▄▇█●▇▇▃ #212 of 474 17% 4
math ▁▁▃▆█●▅█▃▁▁▁ #309 of 530 10% 5
vision & documents ▁▁▂▃▂▃●▄▆██▄ #133 of 189 11% 14
coding · #228 of 539 · 30% of its score
Beats [Gemini 2.5 Flash](gemini-2-5-flash.md), [Qwen3 30B-A3B Instruct 2507](qwen3-30b-a3b-instruct-2507.md), [Qwen3.5 4B](qwen3-5-4b.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 8B Reasoning 2512](ministral-3-8b-reasoning.md), [OpenThinker 32B](openthinker-32b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval+ 81.7% ▁·▁▁▁▃▄▄▄█●▁ beats 93/99 easy · avg 70% 6.2% sharp
HumanEval 87.2% ▁·▁▁▃▄▆▅█●▁▁ beats 92/99 easy · avg 76% 6.2% sharp
LiveCodeBench official (Jan 1–Nov… 42.6% ▁▁·▁▂▄▆█●▂·▂ beats 27/31 hard · avg 23% 5.3% sharp · at its level
CRUXEval (ZeroEval, 800 problems) 80.8% ▁▂▆▃▆█▆●▂·▁▁ beats 42/47 medium · avg 57% 4.0% sharp
Aider code editing (133 Exercism … 77.4% ▁·▁▂▄▁▆▄█▆●▂ beats 58/62 medium · avg 51% 3.4% sharp · at its level
MBPP+ 74.3% ▁···▁▁▂▅▇█●▂ beats 70/75 easy · avg 66% 1.8% typical
WebApp1K-Duo-React pass@1 (onekq-… 68.0% ▁··▁·▁▁▁▆█●█ beats 26/45 hard · avg 29% 1.3% at its level
SciCode main problems (official l… 4.6% ▃···█··▂·●·▂ beats 14/17 very hard · avg 1.0% 1.2% sharp · too hard for it
WebApp1K-React pass@1 (onekq-ai) 88.1% ▃▁▁█▁▄▅▃▄●▁▄ beats 26/31 medium · avg 40% 0.8% sharp · at its level · noisy
chat & writing · #212 of 474 · 17% of its score
Beats [Seed-OSS 36B Instruct](seed-oss-36b.md), [Gemma 2 9B IT SimPO](gemma-2-9b-simpo.md), [Nemotron-Cascade-14B-Thinking](nemotron-cascade-14b-thinking.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
AlpacaEval 2 length-controlled 52.4% ▁▂▂▃▆▆█▆▄●▁▁ beats 110/122 medium · avg 35% 9.1% sharp · at its level
Arena-Hard (v0.1, GPT-4-1106 judg… 79.3% ▂▁▁▅█▅▄▃▃●▃▁ beats 53/60 medium · avg 47% 3.7% sharp
LMArena (text) 1343.3 ▁▁▁▂▃▃▃▆●▇█▃ beats 184/385 avg 1260 2.1% blunt
Polyrating rating (bias-aware re-… 1286.0 ▂▁▂▁▃▇█▆▇▄▂● beats 94/95 avg 1187 1.9% typical
math · #309 of 530 · 10% of its score
Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 51.7% ▂▂▅▆▄●▆▃▅▂▃▄ beats 45/92 hard · avg 28% 4.2% sharp · at its level
MATH Level 5 (ZeroEval, 721 probl… 51.9% ▁▁▁▆█▂▇●▄··▂ beats 27/38 hard · avg 30% 2.8% sharp · at its level
OTIS mock AIME 2024–2025 6.5% ▃▃●▃▃▃▇▇█▃▂▅ beats 37/197 very hard · avg 7.2% 1.5% sharp · at its level · noisy
GSM8K (ZeroEval, zero-shot) 95.6% ▁▄·▁▄█▅▅▅██● beats 39/43 very easy · avg 87% 1.1% sharp · maxed out
MathVista (testmini) 64.4% ▂····▂···▂█● beats 7/9 medium · avg 51% 0.8% at its level · noisy · several runs
vision & documents · #133 of 189 · 11% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-4o (2024-08-06)](gpt-4o-2024-08-06.md) here, though they rank higher overall. Trails [Llama 4 Scout (109B-A17B)](llama-4-scout.md), [Mistral Small 3.2 24B](mistral-small-3-2.md), [Llama 4 Maverick (402B-A17B)](llama-4-maverick.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
CharXiv descriptive questions 84.3% ▄█▄▄▄·██·▄·● beats 11/12 easy · avg 69% 1.8% typical
CharXiv (reasoning questions) 60.2% ▂▂▂█▃▂·▂▂··● beats 13/13 hard · avg 34% 1.4% at its level
Arena vision (style controlled) 1145.2 ▁▁▂▂▂▂▃●▅▅█▇ beats 45/152 avg 1061 1.4% blunt
MMMU (val) 68.0% ▂·▂······█▄● beats 8/9 medium · avg 51% 1.2% at its level · several runs
MMMU val (VHELM, zero-shot exact … 65.5% ▄▄··▄▄█··▄▄● beats 9/9 medium · avg 50% 1.1% at its level
MathVista testmini (VHELM, zero-s… 57.5% ▃·▅···▃▃▃·▃● beats 8/9 medium · avg 42% 1.0% at its level
CapArena-Auto detailed image capt… 21.5000 ▃▃▃·▃█·▅●▃▃▃ beats 9/12 avg -4.498 0.9% typical
MathVista (testmini) 64.4% ▂····▂···▂█● beats 7/9 medium · avg 51% 0.8% at its level · noisy · several runs
MM-MT-Bench (Mistral, GPT-4o judg… 7.5000 MM-MT-Bench score (1-10) ▅··▃·█·▃▃··● beats 8/9 avg 5.605 MM-MT-Bench s… 0.6% sharp · noisy
VQAv2 (val) 70.7% ▃···█●··▅▃▃▃ beats 4/9 easy · avg 73% 0.3% blunt
Ai2 Molmo human preference Elo (S… 1069.0 ▄▄·▄·▄▄▄·▄·● beats 7/8 avg 1032 0.2% blunt
TextVQA 70.5% ▁▁▁▄●▁▃·▃▃▃█ beats 7/24 easy · avg 75% 0.1% blunt
ChartQA 87.6% ▄·▄▄▄··▄▄▄▄● beats 9/9 easy · avg 78% 0.1% blunt · noisy · several runs
AI2D test 68.9% ▁●▅▇▃▇·▄·█·▃ beats 1/28 easy · avg 82% 0.1% blunt
other tests (10)
test result vs the others in its run difficulty counts why
LiveBench 2024-07-26 59.9% ▁▁▁▂▂▃▂█▆▃▅● beats 61/61 medium · avg 38% 8.9% at its level
MMLU-Redux (ZeroEval, 2,778 quest… 86.0% ▁·▁▁▄▄▄▅█●▂▁ beats 46/53 easy · avg 75% 6.2% typical
ZebraLogic (ZeroEval, 1,000 puzzl… 33.4% ▁▁▃▆█●▁▁·▁▁▁ beats 50/57 hard · avg 15% 4.7% sharp · at its level
GPQA Diamond 54.0% ▁▁▁▁▂▄▆●▅▇█▅ beats 76/211 hard · avg 39% 3.5% sharp · at its level
MMLU-Pro 76.1% ▃▂▅█▃·█▃●··▂ beats 21/23 medium · avg 54% 3.2% at its level
WildBench v2 WB-Score (task macro) 54.6951 WB-Score ▁·▁▁·▃█▄▇▆▅● beats 46/52 avg 43.019 WB-Score 2.2% sharp
KCL-Essay vanilla 33.7% ▁···▁▃▂●▂▃▃▂ beats 12/23 hard · avg 17% 1.4% at its level
EvalPerf correctness (118 tasks, … 88.0% ▂·▂▂▂▆▅▆▆▆▃● beats 26/29 easy · avg 74% 0.8% typical
Judgemark v2 (LLM-as-judge for cr… 78.8% ▃▃··▃▅▅···●▃ beats 9/10 medium · avg 53% 0.7% at its level
KCL-MCQA vanilla 45.1% ▂···█▂▆▆●█▆▂ beats 14/23 very hard · avg 29% 0.6% typical
source results share of the score
WildEval ZeroEval published results 5 19% ▇▇
EvalPlus / EvalPerf published evaluatio… 4 15% ▇▇
Epoch AI 3 9% ▇
AlpacaEval 1.0 and 2.0 leaderboards 1 9% ▇
LiveBench model judgments 1 9% ▇
LiveCodeBench official website results 1 5% ▇
Arena-Hard-Auto v0.1 leaderboard 1 4% ▇
LMArena leaderboard dataset 2 4% ▇
Aider code editing benchmark runs 1 3% ▇
CharXiv leaderboard 2 3% ▇
MMLU-Pro author-released evaluation runs 1 3% ▇
WildBench v2 leaderboard data 1 2% ▇
VHELM: A Holistic Evaluation of Vision … 2 2% ▇
WebApp1K models leaderboard 2 2% ▇
Korean Canonical Legal Benchmark: Towar… 2 2% ▇
Polyrating: A Cost-Effective and Bias-A… 1 2% ▇
Pixtral 12B 5 2% ▇
Microsoft Phi-3.5-vision-instruct model… 5 1% ▇
SciCode leaderboard 1 1% ▇
CapArena: Benchmarking and Analyzing De… 1 1% ▇
Judgemark v2 results 1 1% ▇
Ai2 Molmo-72B-0924 model card: Human Pr… 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 57.7
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · AlpacaEval 1.0 and 2.0 leaderboards ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · CharXiv leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · MMLU-Pro author-released evaluation runs ([Apache-2.0](../../licenses/mmlu-pro-author-runs-Apache-2.0.txt)) · WildBench v2 leaderboard data ([Apache-2.0](https://github.com/allenai/WildBench/blob/d6b8dcaf377d173d031980f97c16e1a82618c03d/LICENSE)) · VHELM: A Holistic Evaluation of Vision Language Models ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities ([CC BY 4.0](../../licenses/cards-final-NOTICE.txt)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Pixtral 12B ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Microsoft Phi-3.5-vision-instruct model card: Benchmarks table ([MIT](https://huggingface.co/microsoft/Phi-3.5-vision-instruct/blob/12b77fb40b63a2c73c68243d3f767aab688a1b2a/LICENSE)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Judgemark v2 results ([MIT](../../licenses/judgemark-v2-MIT.txt)) · Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column ([Apache-2.0](https://huggingface.co/allenai/Molmo-72B-0924/blob/ab586dd9640de059566975b8e5bcd0c6fdc73956/README.md))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/claude-3-5-sonnet-20240620) · [JSON](https://whatmodel.sh/api?format=json&all=1)