[whatmodel.sh](../../README.md) / Claude 3.5 Sonnet (2024-10-22)
# Claude 3.5 Sonnet (2024-10-22)
Anthropic · released 2024-10 · proprietary
**score 60.3** ±0.6 · comparison only: no current run option verified · from 34 benchmark results across 20 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Claude 3.5 Sonnet (2024-10-22) among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▂▆●▄▅▄▄▃▁▁ #251 of 354 8.4% 7
coding ▁▁▁▃▄▆█●▅▄▃▁ #160 of 539 26% 8
chat & writing ▁▁▁▃▅▄▇█●▇▇▃ #168 of 474 13% 5
math ▁▁▃▆█●▅█▃▁▁▁ #288 of 530 16% 5
vision & documents ▁▁▂▃▂▃▅●▆██▄ #116 of 189* 3.3% 2
agent tasks · #251 of 354 · 8.4% of its score
Beats [o1 (2024-12-17)](o1-2024-12-17.md), [o3-mini](o3-mini.md), [DeepSeek R1 (Jan '25)](deepseek-r1.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [Mellum2 12B-A2.5B Instruct](mellum2-12b-a2-5b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
BrowserGym WebArena (AgentLab Gen… 36.2% █··▄·▄█·●··▄ beats 6/7 hard · avg 17% 2.0% at its level
BrowserGym WorkArena-L1 (AgentLab… 56.4% █···█▄▄·▄●·█ beats 7/10 hard · avg 28% 2.0% at its level
BrowserGym WorkArena-L2 (AgentLab… 39.1% ▃▅▃·▅▃▃··●·▃ beats 8/11 very hard · avg 3.6% 1.6% sharp · noisy
BrowserGym MiniWoB (AgentLab Gene… 69.8% ▅····▅▅▃▃·●█ beats 8/11 medium · avg 53% 1.2% blunt · at its level
BFCL v3 56.9% ▁▁▁▁▁▃▅●▅▄▁▁ beats 66/114 medium · avg 51% 0.7% blunt · at its level
Multi-SWE-bench, 7 languages (Mag… 4.7% ▄···▄▄·●▄·▄█ beats 3/7 very hard · avg 2.6% 0.6% blunt · too hard for it
Galileo Agent Leaderboard v1 (too… 80.1% ▃▃▃▃●▅▆▅█▂▅▂ beats 8/31 easy · avg 78% 0.4% typical
coding · #160 of 539 · 26% of its score
Beats [Qwen3.5 Plus](qwen3-5-plus.md), [Qwen3-Coder-Next (80B-A3B)](qwen3-coder-next.md), [Gemini 2.5 Pro](gemini-2-5-pro.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
Aider polyglot (225 exercises, tw… 51.6% ▂·▂▃▅▂▃●▆▄▂▂ beats 27/49 very hard · avg 7.6% 8.1% sharp · at its level
CRUXEval (ZeroEval, 800 problems) 83.9% ▁▂▆▃▆█▆●▂·▁▁ beats 43/47 medium · avg 57% 4.8% sharp
Aider code editing (133 Exercism … 84.2% ▁·▁▂▄▁▆▄█▆▂● beats 62/62 medium · avg 51% 4.4% sharp
SciCode main problems (official l… 4.6% ▃···█··▂·●·▂ beats 13/17 very hard · avg 1.0% 2.5% sharp · too hard for it
LiveCodeBench official (Aug 1, 20… 36.4% ▃▃▅●··▃▅█▅▅▃ beats 4/15 hard · avg 24% 2.2% sharp · at its level
WebApp1K-Duo-React pass@1 (onekq-… 75.0% ▁··▁·▁▁▁▆█▇● beats 38/45 hard · avg 29% 1.9% at its level
HumanEval (Vector Institute, Insp… 94.5% ▄·▄·█·█·●·▄▄ beats 6/9 easy · avg 77% 1.8% sharp
Multi-SWE-bench, 7 languages (Mag… 4.7% ▄···▄▄·●▄·▄█ beats 3/7 very hard · avg 2.6% 0.6% blunt · too hard for it
chat & writing · #168 of 474 · 13% of its score
Beats [o3-mini](o3-mini.md), [INTELLECT-3](intellect-3.md), [MiniMax-M2](minimax-m2.md) here, though they rank higher overall. Trails [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md), [LFM2.5 2.6B](lfm2-5-2-6b.md), [Spark-X2.5 1.7B](spark-x2-5-1-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena-Hard (v0.1, GPT-4-1106 judg… 85.2% ▂▁▁▅█▅▄▃▃▅●▁ beats 58/60 medium · avg 47% 4.1% sharp
Arena-Hard v2.0 hard prompts (sty… 33.0% ▂·▃▅▅·●█▆▂▂▂ beats 9/22 very hard · avg 12% 3.8% sharp · at its level
LMArena (text) 1374.3 ▁▁▁▂▃▃▃▆●▇█▃ beats 216/385 avg 1260 3.2% blunt
IFEval prompt-level strict 85.6% ▄▄··▄·▄··●·▄ beats 4/6 easy · avg 73% 1.3% typical
Arena-Hard v2.0 creative writing … 24.2% ▆▂▂●▂▂▃█▂▃·▂ beats 7/22 hard · avg 19% 0.8% sharp · at its level · noisy
math · #288 of 530 · 16% of its score
Trails [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md), [Phi-2 (2.7B)](phi-2.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 56.9% ▂▂▅▆▄●▆▃▅▂▃▄ beats 50/92 hard · avg 28% 5.0% sharp
MATH Level 5 (ZeroEval, 721 probl… 59.4% ▁▁▁▆█▂▇●▄··▂ beats 32/38 hard · avg 30% 3.9% sharp · at its level
MATH (Vector Institute, Inspect E… 79.4% ▄··██●··▄·▄█ beats 5/9 medium · avg 50% 3.4% sharp
OTIS mock AIME 2024–2025 8.5% ▃▃▄●▃▃▇▇█▃▂▅ beats 49/197 very hard · avg 7.2% 2.6% sharp · at its level · noisy
GSM8K (ZeroEval, zero-shot) 96.7% ▁▄·▁▄█▅▅▅██● beats 43/43 very easy · avg 87% 1.1% sharp · maxed out
vision & documents · #116 of 189 · 3.3% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-5 nano](gpt-5-nano.md) here, though they rank higher overall. Trails [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md), [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Qwen3.5 4B](qwen3-5-4b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena vision (style controlled) 1160.7 ▁▁▂▂▂▂▃●▅▅█▇ beats 55/152 avg 1061 2.1% blunt
Kaleidoscope (in-language multimo… 62.9% █·█···█·█·█● beats 5/5 medium · avg 46% 1.1% at its level
other tests (8)
test result vs the others in its run difficulty counts why
MMLU-Redux (ZeroEval, 2,778 quest… 88.9% ▁·▁▁▄▄▄▅█▄●▁ beats 52/53 easy · avg 75% 7.8% typical
ZebraLogic (ZeroEval, 1,000 puzzl… 36.2% ▁▁▃▆██●▁·▁▁▁ beats 51/57 hard · avg 15% 7.5% sharp · at its level
GPQA Diamond 55.3% ▁▁▁▁▂▄▆●▅▇█▅ beats 77/211 hard · avg 39% 5.4% sharp · at its level
MMLU-Pro 77.6% ▃▂▅█▃·█▃●··▂ beats 22/23 medium · avg 54% 4.3% at its level
GPQA Diamond (Vector Institute, I… 61.0% ▃··▃▃▃▃·●·▃█ beats 5/9 hard · avg 40% 4.2% sharp · at its level
MMLU-Pro (Vector Institute, Inspe… 77.6% ▃····▃▅·▃●·█ beats 5/9 medium · avg 58% 2.4% at its level
HindiGen-v1 3C3H score (Inception… 77.5% ▁▁▁▁·▂·█▆█●▁ beats 29/31 medium · avg 46% 1.5% at its level
Vectara hallucination leaderboard… 95.4% ▁▁▂▃▄▄●▆▃▂▂▁ beats 71/140 very easy · avg 94% 0.1% blunt · maxed out · noisy
source results share of the score
WildEval ZeroEval published results 5 25% ▇▇▇
Epoch AI 3 13% ▇▇
Vector Institute Evaluation Leaderboard 4 12% ▇
Aider polyglot benchmark runs 1 8% ▇
ServiceNow BrowserGym leaderboard 4 7% ▇
LMArena leaderboard dataset 2 5% ▇
Arena-Hard-Auto v2.0-Preview leaderboard 2 5% ▇
Aider code editing benchmark runs 1 4% ▇
MMLU-Pro author-released evaluation runs 1 4% ▇
Arena-Hard-Auto v0.1 leaderboard 1 4% ▇
SciCode leaderboard 1 3% ▇
LiveCodeBench official website results 1 2% ▇
WebApp1K models leaderboard 1 2% ▇
Inception Arabic/Hindi leaderboards 1 1% ▇
OpenCompass CompassAcademic leaderboard 1 1% ▇
Kaleidoscope: In-language Exams for Mas… 1 1% ▇
Berkeley Function Calling Leaderboard r… 1 1% ▇
Multi-SWE-bench experiments 1 1% ▇
Galileo Agent Leaderboard 1 0% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 60.3
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Vector Institute Evaluation Leaderboard ([Apache-2.0](../../licenses/vector-inspect-evals-leaderboard-Apache-2.0.txt)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · ServiceNow BrowserGym leaderboard ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Arena-Hard-Auto v2.0-Preview leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-Apache-2.0.txt)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · MMLU-Pro author-released evaluation runs ([Apache-2.0](../../licenses/mmlu-pro-author-runs-Apache-2.0.txt)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · OpenCompass CompassAcademic leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · Multi-SWE-bench experiments ([Apache-2.0](../../licenses/multi-swe-bench-experiments-Apache-2.0.txt)) · Galileo Agent Leaderboard ([Apache-2.0](../../licenses/galileo-agent-leaderboard-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/claude-3-5-sonnet-20241022) · [JSON](https://whatmodel.sh/api?format=json&all=1)