[whatmodel.sh](../../README.md) / Claude 3 Haiku
# Claude 3 Haiku
Anthropic · released 2024-03 · proprietary
**score 49.1** ±0.4 · comparison only: no current run option verified · from 46 benchmark results across 22 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Claude 3 Haiku among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁▁▃▄▆●▇▅▄▃▁ #338 of 539 39% 6
chat & writing ▁▁▁▃▅▄●█▇▇▇▃ #319 of 474 10% 4
math ▁▁▃●██▅█▃▁▁▁ #439 of 530 8.6% 5
vision & documents ▁▁●▃▂▃▅▄▆██▄ #176 of 189 11% 16
coding · #338 of 539 · 39% of its score
Beats [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Solar Open 100B (102B-A12B)](solar-open-100b.md), [Qwen3-VL 30B-A3B Instruct](qwen3-vl-30b-a3b-instruct.md) here, though they rank higher overall. Trails [DeepSeek LLM 67B Chat](deepseek-llm-67b-chat.md), [Yi 1.5 9B Chat](yi-1-5-9b-chat.md), [Mistral NeMo Instruct (2407)](mistral-nemo-instruct-2407.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 76.8% ▁·▁▁▃▄▆▅●▃▁▁ beats 76/99 easy · avg 76% 18% sharp · at its level · several runs
HumanEval+ 68.9% ▁·▁▁▁▃▄▄▄●▃▁ beats 67/99 easy · avg 70% 10% sharp · at its level
CRUXEval (ZeroEval, 800 problems) 54.8% ▁▂▆▃●█▆▃▂·▁▁ beats 16/47 medium · avg 57% 5.1% sharp · at its level
Aider code editing (133 Exercism … 47.4% ▁·▁▂▄●▆▄█▆▂▂ beats 13/62 medium · avg 51% 3.6% sharp · at its level
MBPP+ 68.8% ▁···▁▁▂▅▇█●▂ beats 58/75 easy · avg 66% 2.0% at its level
LiveCodeBench official (Aug 1, 20… 20.2% ●▃▅▃··▃▅█▅▅▃ beats 0/15 hard · avg 24% 0.9% sharp
chat & writing · #319 of 474 · 10% of its score
Beats [Ministral 3 14B Reasoning 2512](ministral-3-14b-reasoning.md), [Ministral 3 14B](ministral-3-14b.md), [OpenThinker2 32B](openthinker2-32b.md) here, though they rank higher overall. Trails [Flan-T5 XXL](flan-t5-xxl.md), [North Micro Vision Instruct (2.4B)](north-micro-vision-instruct.md), [Granite 4.0 H 1B](granite-4-0-h-1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena-Hard (v0.1, GPT-4-1106 judg… 41.5% ▂▁▁▅█▅●▃▃▅▃▁ beats 34/60 medium · avg 47% 4.5% sharp · at its level
LMArena (text) 1261.7 ▁▁▁▂▃▃●▆▆▇█▃ beats 96/385 avg 1260 1.8% blunt
Polyrating rating (bias-aware re-… 1209.0 ▂▁▂▁▃▇█▆▇●▂▂ beats 83/95 avg 1187 1.7% typical
Arabic IFEval prompt-level (Incep… 53.7% ▁·▁▁▂▁▄▄▆▄●▄ beats 29/38 medium · avg 46% 1.5% sharp · at its level
math · #439 of 530 · 8.6% of its score
Trails [OPT-2.7B](opt-2-7b.md), [Gemma 2B (base)](gemma-2b-base.md), [Stable Code 3B](stable-code-3b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 14.9% ▂▂●▆▄█▆▃▅▂▃▄ beats 12/92 hard · avg 28% 3.0% sharp
GSM8K (ZeroEval, zero-shot) 88.8% ▁▄·▁▄█▅●▅███ beats 21/43 very easy · avg 87% 2.5% sharp
MATH Level 5 (ZeroEval, 721 probl… 15.1% ▁▁▁●█▂▇▇▄··▂ beats 3/38 hard · avg 30% 2.0% sharp · at its level
MathVista (testmini) 44.8% ▂····▂···●█▆ beats 2/9 medium · avg 49% 0.8% at its level · small run · several runs
OTIS mock AIME 2024–2025 1.8% ▃●▄▃▃▃▇▇█▃▂▅ beats 13/197 very hard · avg 7.2% 0.3% sharp · too hard for it · noisy
vision & documents · #176 of 189 · 11% of its score
Trails [Qwen3.5 0.8B](qwen3-5-0-8b.md), [Aya Vision 32B](aya-vision-32b.md), [Qwen3-VL 2B Instruct](qwen3-vl-2b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
CharXiv descriptive questions 65.1% ▄█▄▄●·██·▄·█ beats 5/12 easy · avg 69% 2.2% at its level
Arena vision (style controlled) 999.7 ▁▁▂●▂▂▃▄▅▅█▇ beats 10/152 avg 1061 1.2% blunt
CharXiv descriptive questions (te… 64.5% █··●··█··▄·▄ beats 2/6 easy · avg 69% 1.1% at its level
CharXiv (reasoning questions) 31.8% ▂▂▂●▃▂·▂▂··▂ beats 5/13 hard · avg 34% 1.1% at its level
MMMU val (VHELM, zero-shot exact … 48.1% ▄●··▄▄█··▄▄█ beats 1/9 medium · avg 50% 1.1% at its level
MMMU (val) 50.4% ▂·▂······●▄▄ beats 3/9 hard · avg 48% 1.0% at its level · small run · several runs
MathVista testmini (VHELM, zero-s… 35.2% ●·▅···▃▃▃·▃█ beats 0/9 medium · avg 42% 0.9% at its level
MathVista (testmini) 44.8% ▂····▂···●█▆ beats 2/9 medium · avg 49% 0.8% at its level · small run · several runs
MM-MT-Bench (Mistral, GPT-4o judg… 5.4600 MM-MT-Bench score (1-10) ▅··▃·●·▃▃··▅ beats 4/9 avg 5.605 MM-MT-Bench s… 0.6% sharp · noisy
CharXiv reasoning questions (test… 30.5% ▄··▄▄●█····▄ beats 3/6 hard · avg 30% 0.5% at its level
VQAv2 (val) 68.4% ▃···●▃··▅▃▃▃ beats 3/9 easy · avg 73% 0.3% blunt · at its level
PhysBench physical-world understa… 39.4% ▅▃●··▅▃····▃ beats 3/9 hard · avg 42% 0.2% blunt
Ai2 Molmo human preference Elo (S… 999.0 ▄▄·●·▄▄▄·▄·█ beats 2/8 avg 1032 0.2% blunt
ChartQA 69.6% ▄·▄▄▄··▄●▄▄█ beats 5/9 easy · avg 78% 0.1% blunt · noisy · small run · several runs
TextVQA 62.7% ▁●▁▄▅▁▃·▃▃▃█ beats 1/24 easy · avg 75% 0.1% blunt · at its level · small run
AI2D test 60.3% ●▁▅▇▃▇·▄·█·▃ beats 0/28 easy · avg 82% 0.0% blunt · small run
other tests (12)
test result vs the others in its run difficulty counts why
MMLU-Redux (ZeroEval, 2,778 quest… 72.3% ▁·▁▁▄▄●▅█▄▂▁ beats 16/53 easy · avg 75% 8.2% at its level
LiveBench 2024-07-26 35.9% ▁▁▁▂▂▃▂█●▃▅▃ beats 39/61 medium · avg 38% 7.3% at its level
EvoEval (overall) 52.1% ▂▂▃▄▄█▁▄▅●▁▃ beats 36/42 medium · avg 52% 5.3% at its level
BigCodeBench (Instruct) 39.2% ▃▂·▆▆██●▅▃▃▃ beats 23/33 medium · avg 39% 2.1% at its level
ZebraLogic (ZeroEval, 1,000 puzzl… 14.3% ▁▁▃●██▁▁·▁▁▁ beats 20/57 hard · avg 15% 2.1% sharp
BigCodeBench (Complete) 48.6% ▂▁▁▄▂▅▇█●▂▂▂ beats 48/58 medium · avg 48% 2.1% at its level
WildBench v2 WB-Score (task macro) 38.8936 WB-Score ▁·▁▁·▃█▄●▆▅▆ beats 26/52 avg 43.019 WB-Score 2.0% sharp
GPQA Diamond 36.3% ▁▁▁▁▂●▆▄▅▇█▅ beats 30/211 hard · avg 39% 1.7% sharp
HindiGen-v1 3C3H score (Inception… 55.3% ▁▁▁▁·▂·●▆█▇▁ beats 11/31 medium · avg 46% 1.0% at its level
Open PT-LLM Leaderboard average (… 74.2% ▃▁▁▁▁▁▁▁▂█●▁ beats 214/276 medium · avg 72% 0.4% at its level · noisy
ARC-AGI-1 public evaluation set (… 7.1% ▁·●▃▁·█▇▁·▇▃ beats 1/27 very hard · avg 4.0% 0.2% sharp · too hard for it · noisy
ARC-AGI-2 public evaluation set (… 0.0% ●··▂▅▁▃▁▃·▁▃ beats 4/24 very hard · avg 0.3% 0.0% sharp · too hard for it · noisy
source results share of the score
EvalPlus / EvalPerf published evaluatio… 3 23% ▇▇▇
WildEval ZeroEval published results 5 20% ▇▇
Top Leaderboard Ranking = Top Coding Pr… 2 12% ▇
LiveBench model judgments 1 7% ▇
Epoch AI 3 5% ▇
Arena-Hard-Auto v0.1 leaderboard 1 5% ▇
BigCodeBench paper 2 4% ▇
Aider code editing benchmark runs 1 4% ▇
CharXiv leaderboard 2 3% ▇
LMArena leaderboard dataset 2 3% ▇
Inception Arabic/Hindi leaderboards 2 2% ▇
WildBench v2 leaderboard data 1 2% ▇
VHELM: A Holistic Evaluation of Vision … 2 2% ▇
Polyrating: A Cost-Effective and Bias-A… 1 2% ▇
Pixtral 12B 5 2% ▇
CharXiv: Charting Gaps in Realistic Cha… 2 2% ▇
Phi-3 Technical Report 5 1% ▇
LiveCodeBench official website results 1 1% ▇
Open Portuguese LLM Leaderboard 1 0% ▇
PhysBench: Benchmarking and Enhancing V… 1 0% ▇
ARC Prize Foundation model_baseline out… 2 0% ▇
Ai2 Molmo-72B-0924 model card: Human Pr… 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 49.1
Licences: EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Top Leaderboard Ranking = Top Coding Proficiency ([CC BY 4.0](../../licenses/paper-2403.19114v1-CC-BY-4.0.txt)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · BigCodeBench paper ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · CharXiv leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · WildBench v2 leaderboard data ([Apache-2.0](https://github.com/allenai/WildBench/blob/d6b8dcaf377d173d031980f97c16e1a82618c03d/LICENSE)) · VHELM: A Holistic Evaluation of Vision Language Models ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Pixtral 12B ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs ([CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)) · Phi-3 Technical Report ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · ARC Prize Foundation model\_baseline outputs on ARC-AGI-1/2 public evaluation sets ([MIT](../../licenses/arcprize-model-baseline-outputs-MIT.txt)) · Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column ([Apache-2.0](https://huggingface.co/allenai/Molmo-72B-0924/blob/ab586dd9640de059566975b8e5bcd0c6fdc73956/README.md))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/claude-3-haiku-20240307) · [JSON](https://whatmodel.sh/api?format=json&all=1)