[whatmodel.sh](../../README.md) / Claude 3.5 Haiku
# Claude 3.5 Haiku
Anthropic · released 2024-10 · proprietary
**score 54.4** ±0.6 · comparison only: no current run option verified · from 25 benchmark results across 15 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Claude 3.5 Haiku among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▂●█▄▅▄▄▃▁▁ #290 of 354 4.1% 4
coding ▁▁▁▃▄▆█●▅▄▃▁ #225 of 539 23% 3
chat & writing ▁▁▁▃▅▄▇●▇▇▇▃ #229 of 474* 7.3% 2
math ▁▁▃▆●█▅█▃▁▁▁ #334 of 530 20% 4
vision & documents ▁▁▂▃▂▃●▄▆██▄ #137 of 189* 3.3% 2
web search ▂▁▂▆▅█▅●▁▂▁▁ #12 of 96* 1.3% 1
agent tasks · #290 of 354 · 4.1% of its score
Beats [DeepSeek R1 0528 Qwen3 8B](deepseek-r1-0528-qwen3-8b.md), [Sky-T1 32B Preview](sky-t1-32b-preview.md), [Magistral Small 1.0 (2506)](magistral-small-2506.md) here, though they rank higher overall. Trails [Granite 4.0 1B](granite-4-0-1b.md), [SmolLM3 3B](smollm3-3b.md), [Qwen3-VL 2B Instruct](qwen3-vl-2b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 13.1% █·▃▃·█●▅▅▅▃▅ beats 8/18 very hard · avg 5.5% 1.3% sharp
BFCL V4 (2025-08-26 archive, mode… 43.4% ▄▄····●▄···▄ beats 2/5 hard · avg 26% 1.2% at its level
BFCL v3 52.5% ▁▁▁▁▁▃▅●▅▄▁▁ beats 48/114 medium · avg 51% 0.9% blunt · at its level
Galileo Agent Leaderboard v1 (too… 76.5% ▃▃▃●▆▅▆▅█▂▅▂ beats 6/31 easy · avg 78% 0.7% typical
coding · #225 of 539 · 23% of its score
Beats [Gemini 2.5 Flash](gemini-2-5-flash.md), [Qwen3 30B-A3B Instruct 2507](qwen3-30b-a3b-instruct-2507.md), [Qwen3.5 4B](qwen3-5-4b.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 8B Reasoning 2512](ministral-3-8b-reasoning.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
CRUXEval (ZeroEval, 800 problems) 68.8% ▁▂▆▃▆●▆▃▂·▁▁ beats 26/47 medium · avg 57% 10% sharp · at its level
Aider code editing (133 Exercism … 75.2% ▁·▁▂▄▁▆▄█▆●▂ beats 57/62 medium · avg 51% 7.4% sharp · at its level
Aider polyglot (225 exercises, tw… 28.0% ▂·▂▃▅●▃█▆▄▂▂ beats 15/49 very hard · avg 7.6% 6.1% sharp
chat & writing · #229 of 474 · 7.3% of its score
Beats [Gemma 2 9B IT SimPO](gemma-2-9b-simpo.md), [Granite 4.2 8B](granite-4-2-8b.md), [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 1324.8 ▁▁▁▂▃▃▃●▆▇█▃ beats 162/385 avg 1260 4.1% blunt
Arabic IFEval prompt-level (Incep… 67.1% ▁·▁▁▂▁▄▄▆▄█● beats 35/38 medium · avg 46% 3.2% sharp · at its level
math · #334 of 530 · 20% of its score
Beats [Molmo 2 8B](molmo-2-8b.md), [Llama 3.3 70B Instruct](llama-3-3-70b.md), [Mistral Large 2 (2407)](mistral-large-2407.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [Tulu 2 DPO 13B](tulu-2-dpo-13b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 46.4% ▂▂▅▆▄●▆▃▅▂▃▄ beats 38/92 hard · avg 28% 9.1% sharp · at its level
MATH Level 5 (ZeroEval, 721 probl… 46.5% ▁▁▁▆█▂●▇▄··▂ beats 24/38 hard · avg 30% 5.5% sharp · at its level
GSM8K (ZeroEval, zero-shot) 94.5% ▁▄·▁▄█▅▅▅█●█ beats 34/43 very easy · avg 87% 3.0% sharp · maxed out
OTIS mock AIME 2024–2025 4.3% ▃▃●▃▃▃▇▇█▃▂▅ beats 28/197 very hard · avg 7.2% 2.0% sharp · noisy
vision & documents · #137 of 189 · 3.3% of its score
Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-4o (2024-08-06)](gpt-4o-2024-08-06.md), [GPT-4 Turbo (2024-04-09)](gpt-4-turbo-2024-04-09.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
Arena vision (style controlled) 1127.4 ▁▁▂▂▂▂●▄▅▅█▇ beats 39/152 avg 1061 2.7% blunt
MSEarth-Bench Earth-science figur… 47.6% ●·██····██·█ beats 0/5 hard · avg 48% 0.6% blunt · at its level
web search · #12 of 96 · 1.3% of its score
Beats [GPT-6 Astra](gpt-6-astra.md), [Claude Opus 5.5](claude-opus-5-5.md), [GPT-6.1 Sol](gpt-6-1-sol.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
Finance Agent Benchmark v1 (537 q… 13.1% █·▃▃·█●▅▅▅▃▅ beats 8/18 very hard · avg 5.5% 1.3% sharp
other tests (10)
test result vs the others in its run difficulty counts why
MMLU-Redux (ZeroEval, 2,778 quest… 79.6% ▁·▁▁▄▄▄●█▄▂▁ beats 27/53 easy · avg 75% 15% typical
ZebraLogic (ZeroEval, 1,000 puzzl… 18.7% ▁▁▃▆●█▁▁·▁▁▁ beats 25/57 hard · avg 15% 7.5% sharp · at its level
MMLU-Pro 62.1% ▃▂▅█▃·●▃▅··▂ beats 13/23 medium · avg 54% 6.6% at its level
GPQA Diamond 38.1% ▁▁▁▁▂●▆▄▅▇█▅ beats 35/211 hard · avg 39% 5.8% sharp · at its level
KCL-Essay vanilla 21.2% ▁···▁●▂█▂▃▃▂ beats 3/23 hard · avg 17% 2.3% typical
HindiGen-v1 3C3H score (Inception… 61.3% ▁▁▁▁·▂·█●█▇▁ beats 15/31 medium · avg 46% 2.2% at its level
Judgemark v2 (LLM-as-judge for cr… 77.6% ▃▃··▃▅▅···●▃ beats 8/10 medium · avg 53% 1.4% at its level
KCL-MCQA vanilla 28.0% ▂···●▂▆▆██▆▂ beats 1/23 very hard · avg 29% 1.0% typical
DiagnosticIQ industrial maintenan… 46.9% ▁··●▁▁▁▅█▅▁▅ beats 2/24 hard · avg 51% 0.8% blunt · at its level
Vectara hallucination leaderboard… 95.1% ▁▁▂▃▄▄●▆▃▂▂▁ beats 62/140 very easy · avg 94% 0.2% blunt · maxed out · noisy
source results share of the score
WildEval ZeroEval published results 5 40% ▇▇▇▇▇
Epoch AI 3 17% ▇▇
Aider code editing benchmark runs 1 7% ▇
LMArena leaderboard dataset 2 7% ▇
MMLU-Pro author-released evaluation runs 1 7% ▇
Aider polyglot benchmark runs 1 6% ▇
Inception Arabic/Hindi leaderboards 2 5% ▇
Korean Canonical Legal Benchmark: Towar… 2 3% ▇
Berkeley Function Calling Leaderboard r… 2 2% ▇
Judgemark v2 results 1 1% ▇
Finance Agent Benchmark 1 1% ▇
DiagnosticIQ: A Benchmark for LLM-Based… 1 1% ▇
Galileo Agent Leaderboard 1 1% ▇
MSEarth: A Multimodal Benchmark for Ear… 1 1% ▇
Vectara hallucination leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 54.4
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · MMLU-Pro author-released evaluation runs ([Apache-2.0](../../licenses/mmlu-pro-author-runs-Apache-2.0.txt)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities ([CC BY 4.0](../../licenses/cards-final-NOTICE.txt)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · Judgemark v2 results ([MIT](../../licenses/judgemark-v2-MIT.txt)) · Finance Agent Benchmark ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Galileo Agent Leaderboard ([Apache-2.0](../../licenses/galileo-agent-leaderboard-Apache-2.0.txt)) · MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/claude-3-5-haiku-20241022) · [JSON](https://whatmodel.sh/api?format=json&all=1)