[whatmodel.sh](../../README.md) / Claude Sonnet 4
# Claude Sonnet 4
Anthropic · released 2025-05 · proprietary
**score 65.0** ±0.6 · comparison only: no current run option verified · from 48 benchmark results across 18 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Claude Sonnet 4 among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▂▆█▄●▄▄▃▁▁ #159 of 354 22% 11
coding ▁▁▁▃▄▆█▇●▄▃▁ #147 of 539 22% 6
chat & writing ▁▁▁▃▅▄▇█▇●▇▃ #143 of 474* 5.3% 2
math ▁▁▃▆██●█▃▁▁▁ #199 of 530 8.9% 5
vision & documents ▁▁▂▃▂▃▅▄●██▄ #103 of 189* 2.5% 1
agent tasks · #159 of 354 · 22% of its score
Beats [o3](o3.md), [Gemini 2.5 Pro](gemini-2-5-pro.md), [DeepSeek V3.2 Exp](deepseek-v3-2-exp.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Large](jamba-1-5-large.md), [GPT-4o (2024-05-13)](gpt-4o-2024-05-13.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
τ²-bench Telecom 46.2% ██··██●···██ beats 4/6 very hard · avg 8.9% 5.6% sharp · at its level · several runs
BrowserGym WorkArena-L2 (AgentLab… 40.4% ▃▅▃·▅▃▃··●·▃ beats 9/11 very hard · avg 3.6% 2.8% sharp · at its level · noisy
BrowserGym WorkArena-L1 (AgentLab… 63.3% █···█▄▄·▄█·● beats 10/10 hard · avg 28% 2.3% at its level
τ-bench Retail (maker) 80.5% ▁····▃·▂▇█▂● beats 26/28 medium · avg 45% 2.2% small run
τ²-bench Retail 80.0% ▅···█·▃····● beats 6/6 medium · avg 40% 2.1% at its level
τ-bench Airline (maker) 60.0% ▁▁▁·▅▄▁█▇▁▁● beats 25/28 hard · avg 26% 1.5% at its level · small run
τ²-bench Airline 60.0% ▄·▄··▄···█▄● beats 6/6 hard · avg 25% 1.4% at its level
BrowserGym MiniWoB (AgentLab Gene… 70.7% ▅····▅▅▃▃·▃● beats 9/11 medium · avg 53% 1.3% blunt · at its level
BFCL V4 (2025-08-26 archive, mode… 70.3% ▄▄····█▄···● beats 5/5 hard · avg 26% 1.2% at its level
Terminal-Bench (pre-2.0 release) 40.7% ▄·▄···▄▄···● beats 5/5 very hard · avg 12% 0.8% at its level
Galileo Agent Leaderboard v2 (act… 55.0% ▅▃·▃▃▅▅█▃█●▅ beats 16/19 hard · avg 26% 0.8% at its level
coding · #147 of 539 · 22% of its score
Beats [Qwen3.5 Plus](qwen3-5-plus.md), [Qwen3-Coder-Next (80B-A3B)](qwen3-coder-next.md), [Gemini 2.5 Pro](gemini-2-5-pro.md) here, though they rank higher overall. Trails [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [Seed-OSS 36B Instruct](seed-oss-36b.md), [Solar Pro 4](solar-pro-4.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Aider polyglot (225 exercises, tw… 61.3% ▂·▂▃▅▂▃█●▄▂▂ beats 36/49 very hard · avg 7.6% 9.5% sharp · at its level
Luau Leetcode fully solved (Torpe… 68.3% ▁▁▁▁·▁▂▃●▃█▇ beats 20/49 hard · avg 17% 3.5% sharp · at its level
LiveCodeBench (version unspecifie… 45.6% █·······●▄▄▄ beats 2/5 hard · avg 21% 2.6% at its level
LiveCodeBench official (Aug 1, 20… 55.9% ▃▃▅▃··●▅█▅▅▃ beats 5/15 hard · avg 24% 2.4% sharp · at its level
MBPP+ 80.2% ▃·····▃█●··▃ beats 5/6 medium · avg 60% 1.9% typical
HumanEval+ 94.5% ▄▄▄·····▄▄·● beats 5/6 easy · avg 66% 1.8% sharp · maxed out
chat & writing · #143 of 474 · 5.3% of its score
Beats [LongCat 2.0 (1.6T-A48B)](longcat-2-0.md), [GPT-5 mini](gpt-5-mini.md), [MiniMax-M2.5](minimax-m2-5.md) here, though they rank higher overall. Trails [LFM2.5 2.6B](lfm2-5-2-6b.md), [Nemotron-Cascade-8B](nemotron-cascade-8b.md), [DeepSeek V3 0324](deepseek-v3-0324.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 1390.9 ▁▁▁▂▃▃▃▆▆●█▃ beats 234/385 avg 1260 3.8% blunt
IFEval (aggregation unspecified) 88.3% █··█·█··●███ beats 3/6 easy · avg 74% 1.5% typical
math · #199 of 530 · 8.9% of its score
Trails [OpenThinker3 7B](openthinker3-7b.md), [Olmo 3 7B Think](olmo-3-7b-think.md), [OpenThinker2 32B](openthinker2-32b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 84.4% ▂▂▅▆▄█▆▃●▂▃▄ beats 71/92 hard · avg 28% 2.9% sharp
OTIS mock AIME 2024–2025 68.9% ▃▃▄▃▃▃●▇█▃▂▅ beats 99/197 very hard · avg 7.2% 2.5% sharp · at its level · noisy
AIME 2025 37.0% ▄●···▄▄··▄·█ beats 1/6 hard · avg 16% 1.8% sharp · at its level · noisy
AIME 2024 47.0% ▁▁▁▂▁▃▁●███▃ beats 20/85 hard · avg 30% 1.0% sharp · at its level · noisy
MATH-500 93.8% ▄·●··█··▄·▄▄ beats 1/6 very easy · avg 87% 0.7% maxed out · noisy
vision & documents · #103 of 189 · 2.5% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-5 mini](gpt-5-mini.md) here, though they rank higher overall. Trails [Qwen3.5 4B](qwen3-5-4b.md), [GPT-4.1 mini](gpt-4-1-mini.md), [Mistral Large 3 (675B-A41B)](mistral-large-3.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena vision (style controlled) 1188.6 ▁▁▂▂▂▂▃▄●▅█▇ beats 66/152 avg 1061 2.5% blunt
other tests (21)
test result vs the others in its run difficulty counts why
GPQA Diamond 79.2% ▁▁▁▁▂▄▆▄▅●█▅ beats 125/211 hard · avg 39% 10% sharp · at its level · several runs
MMLU 91.8% ▄·····█▄▄▄·● beats 6/6 medium · avg 68% 4.1% typical
MMLU-Pro 83.7% █▄·▄···●··▄▄ beats 4/6 medium · avg 58% 3.1% typical
KCL-Essay vanilla 35.5% ▁···▁▃▂█●▃▃▂ beats 14/23 hard · avg 17% 2.9% at its level
EQ-Bench 3 (emotional intelligenc… 1224.3 ▁·▂▁▄▅▅●█▄▁▁ beats 34/62 avg 1034 2.3% typical
AraGen-v3 3C3H score (Inception, … 75.6% ▁▁·▁▂▂▄█▄▅▄● beats 59/66 hard · avg 29% 2.0% at its level
CMMLU 86.5% █····▄··●▄▄▄ beats 3/6 medium · avg 67% 1.6% typical
RobloxQA v2.0 multiple choice (To… 92.3% ▁▁▃▁█●▃▃▆▂▂▂ beats 21/49 easy · avg 77% 1.6% typical
KCL-MCQA vanilla 45.7% ▂···█▂▆▆●█▆▂ beats 15/23 very hard · avg 29% 1.6% at its level
C-Eval 86.6% █····●·▄·▄▄▄ beats 2/6 medium · avg 68% 1.5% typical
HindiGen-v1 3C3H score (Inception… 69.8% ▁▁▁▁·▂·█▆●▇▁ beats 18/31 medium · avg 46% 1.5% typical
COLLIE (constrained text generati… 51.2% ▄···▄█●····█ beats 4/6 hard · avg 30% 1.4% at its level
RPC-Bench overall, text input (Z.… 24.2% ▄··●▄·▄▄█·▄▄ beats 1/8 very hard · avg 12% 1.4% at its level
VitaBench 23.0% █·······▄█·● beats 5/6 very hard · avg 6.5% 1.4% typical
LongSocialBench (full-context, 4-… 52.0% ▆▂·▄█·▂▂·▄●▂ beats 14/16 very hard · avg 36% 1.3% blunt · at its level
DiagnosticIQ industrial maintenan… 62.5% ▁··▃▁▁▁▅●▅▁▅ beats 13/24 hard · avg 51% 0.8% blunt · at its level
ZebraLogic 75.8% █···●··███·█ beats 1/5 medium · avg 51% 0.6% noisy
BeyondAIME 20.5% ●····█·▄▄··▄ beats 0/6 hard · avg 24% 0.5% at its level
DROP 73.1% ▄···▄·●█·▄·▄ beats 2/6 medium · avg 58% 0.2% blunt · at its level · noisy
Vectara hallucination leaderboard… 95.5% ▁▁▂▃▄▄●▆▃▂▂▁ beats 76/140 very easy · avg 94% 0.1% blunt · maxed out · noisy
Vectara hallucination leaderboard… 89.7% ▁▂▁▂▅●▂▃▄▃▁▁ beats 43/98 very easy · avg 90% 0.0% blunt
source results share of the score
Meituan LongCat-Flash-Chat model card 21 36% ▇▇▇▇
Epoch AI 3 11% ▇
Aider polyglot benchmark runs 1 9% ▇
ServiceNow BrowserGym leaderboard 3 6% ▇
LMArena leaderboard dataset 2 6% ▇
Roblox LLM Leaderboard 2 5% ▇
Korean Canonical Legal Benchmark: Towar… 2 4% ▇
Moonshot AI Kimi-K2-Instruct model card 1 4% ▇
τ-bench leaderboard: Anthropic's own su… 2 4% ▇
Inception Arabic/Hindi leaderboards 2 4% ▇
LiveCodeBench official website results 1 2% ▇
EQ-Bench 3 canonical leaderboard results 1 2% ▇
RPC-Bench leaderboard 1 1% ▇
LongSocialBench: Do Long-Context LLMs U… 1 1% ▇
Berkeley Function Calling Leaderboard r… 1 1% ▇
DiagnosticIQ: A Benchmark for LLM-Based… 1 1% ▇
Galileo Agent Leaderboard 1 1% ▇
Vectara hallucination leaderboard 2 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 65.0
Licences: Meituan LongCat-Flash-Chat model card ([MIT](https://huggingface.co/meituan-longcat/LongCat-Flash-Chat/blob/f0b8e8d7380330f450ca1fde4ec56823a514584c/LICENSE)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · ServiceNow BrowserGym leaderboard ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Roblox LLM Leaderboard ([MIT](../../licenses/roblox-llm-leaderboard-MIT.txt)) · Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities ([CC BY 4.0](../../licenses/cards-final-NOTICE.txt)) · Moonshot AI Kimi-K2-Instruct model card ([Modified MIT License (Kimi K2)](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/fd1984e2b7a3350dbf7305fe73a4ede25c14de50/LICENSE)) · τ-bench leaderboard: Anthropic's own submissions ([MIT](https://opensource.org/license/mit)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · EQ-Bench 3 canonical leaderboard results ([MIT](../../licenses/eqbench3-MIT.txt)) · RPC-Bench leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads? ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Galileo Agent Leaderboard ([Apache-2.0](../../licenses/galileo-agent-leaderboard-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/claude-sonnet-4-20250514) · [JSON](https://whatmodel.sh/api?format=json&all=1)