[whatmodel.sh](../../README.md) / LLaMA 13B
# LLaMA 13B
Meta · released 2023-02 · 13B params · llama-1-noncommercial-research
**score 32.4** ±0.5 · comparison only: no current run option verified · from 83 benchmark results across 25 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks LLaMA 13B among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁●▃▄▆█▇▅▄▃▁ #517 of 539 8.8% 2
chat & writing ●▁▁▃▅▄▇█▇▇▇▃ #473 of 474 7.0% 3
math ▁▁▃●██▅█▃▁▁▁ #429 of 530 15% 3
coding · #517 of 539 · 8.8% of its score
Beats [Molmo 7B-D (0924)](molmo-7b-d.md), [Tulu 2 7B](tulu-2-7b.md), [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md) here, though they rank higher overall. Trails [Falcon3 1B Instruct](falcon3-1b.md), [Granite 3.0 1B-A400M Instruct](granite-3-0-1b-a400m.md), [Gemma 1.1 2B IT](gemma-1-1-2b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 13.4% ▄··▄····▄▄█● beats 6/6 medium · avg 63% 7.5% sharp · several runs
MBPP (maker) 22.0% ▁●▁▄▅█▆▅▄▁▂▁ beats 4/86 medium · avg 59% 1.3% typical
chat & writing · #473 of 474 · 7.0% of its score
Trails [Gemma 3 270M IT](gemma-3-270m.md), [FastChat-T5 3B](fastchat-t5-3b.md), [SmolLM2 360M Instruct](smollm2-360m.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 975.2 ●▁▁▂▃▃▃▆▆▇█▃ beats 1/385 avg 1260 2.7% blunt
Polyrating rating (bias-aware re-… 853.0 ●▁▂▁▃▇█▆▇▄▂▂ beats 1/95 avg 1187 2.4% typical
MT-Bench (GPT-4 single-answer gra… 2.6100 MT-Bench score (1-10) ●▂▂▆▆▄██▂▄▄▆ beats 0/28 avg 8.539 MT-Bench scor… 1.9% sharp · noisy
math · #429 of 530 · 15% of its score
Beats [GPT-3.5 Turbo (0125)](gpt-3-5-turbo-0125.md), [DBRX Instruct](dbrx-instruct.md), [Command R Plus (04-2024)](command-r-plus-04-2024.md) here, though they rank higher overall. Trails [OPT-2.7B](opt-2-7b.md), [Gemma 2B (base)](gemma-2b-base.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 13.6% ▄▄█▄···▄·●·▄ beats 6/7 easy · avg 65% 13% sharp · several runs
MATH (maker) 3.9% ▃●▂▆▅▃▅▅█▃▃▂ beats 3/30 hard · avg 24% 1.4% too hard for it
MathQA 28.4% ▃·▃▃█▅··▃·●▃ beats 9/10 hard · avg 34% 0.2% blunt · small run
other tests (37)
test result vs the others in its run difficulty counts why
ARC Challenge 43.9% █···▁▁·▃▁▃▁● beats 14/15 medium · avg 74% 15% sharp · several runs
MMLU 46.2% ▄▄·▄···▄▄·●█ beats 6/8 easy · avg 79% 8.4% at its level · several runs
WinoGrande 70.2% ▂····▄·▂▄▄█● beats 14/15 easy · avg 88% 6.7% at its level · several runs
HellaSwag 76.2% ▅█······▃●·▅ beats 6/8 easy · avg 81% 6.1% at its level · several runs
BIG-Bench Hard 37.1% ███··█·●·█·█ beats 4/6 medium · avg 68% 5.8% sharp · several runs
OpenBookQA 30.6% █········▂▂● beats 7/7 hard · avg 40% 3.1% too hard for it · several runs
PIQA 78.8% ▅▃·▃▃▃▅▃▅▃●▃ beats 14/15 easy · avg 87% 2.4% blunt · at its level · several runs
ARC Easy 74.6% ▅▃·▃·▅▅·█▅▃● beats 14/15 easy · avg 87% 2.4% blunt · at its level · several runs
BoolQ 68.5% ▄··▄·█▄··●·▄ beats 6/7 easy · avg 90% 2.3% at its level · several runs
ImageNetVC visual commonsense, te… 58.9% ▂·▄·▂█▄▄█▄●▆ beats 18/21 easy · avg 74% 1.9% blunt · at its level
F2WTQ free-form table QA, 2-shot … 21.4% ▄···●·▄█··▄▄ beats 2/7 medium · avg 56% 1.8% typical
COPA 90.0% ▄··█▄▄···▄·● beats 7/7 very easy · avg 95% 1.7% at its level · several runs
Factual knowledge recall, Premium… 21.3% ▅····▃▅·●█·█ beats 6/13 medium · avg 52% 1.2% typical
TriviaQA (maker) 63.1% ▁▁▂▂▄▄▃▆▅●▅▂ beats 53/80 easy · avg 84% 1.1% at its level
Natural Questions (closed book) (… 28.1% ▂▁▁▁▂▄▄▃▅█●▁ beats 58/68 medium · avg 53% 1.0% typical
CommonsenseQA 66.2% █·····▂▂▃▂●▄ beats 29/38 very easy · avg 90% 1.0% sharp · at its level · noisy · small run
VEC embodied concepts, zero-shot … 52.8% ▁····▁▁·●·▃█ beats 3/11 hard · avg 63% 1.0% typical
Factual knowledge recall, Premium… 41.2% ▃▅▃··▅▃·●▃▃▅ beats 8/13 medium · avg 61% 0.7% at its level
Social IQA (maker) 50.4% ▁·▁██▄▅●▂▃▇▁ beats 41/59 medium · avg 63% 0.7% blunt · at its level · several runs
Winograd Schema Challenge (WSC273) 86.1% ▄▄·▄·▄··█··● beats 6/7 very easy · avg 93% 0.6% blunt · at its level
CoQA 77.0% ▄····▄▄·▄▄·● beats 6/6 very easy · avg 92% 0.6% typical
AQuA-RAT 17.7% ●··▁▁▁··▁▁▁▁ beats 1/27 hard · avg 47% 0.6% sharp · too hard for it · noisy · small…
RACE-high 39.3% ▄···█··▄▄··● beats 5/6 medium · avg 51% 0.6% blunt
SciTab 3-class claim verification… 37.2% █···▄▄·▄●▄·▄ beats 5/7 medium · avg 51% 0.5% blunt · at its level
DROP 13.4% ▄▄···█·▄·▄·● beats 6/6 hard · avg 34% 0.5% blunt
StrategyQA 63.1% █····▂▂▁▂●▂▂ beats 19/25 medium · avg 72% 0.5% small run
12-dataset text classification, 4… 68.1% ▂▂··▂·█···●▂ beats 7/8 easy · avg 79% 0.5% blunt · at its level
Spider dev execution accuracy, ze… 8.7% █·●▄··▄····▄ beats 2/6 medium · avg 54% 0.5% sharp · noisy
This is not a Dataset (negation),… 50.1% ●▁····▃▁·▁·▁ beats 4/14 hard · avg 66% 0.3% sharp · too hard for it · noisy
MenatQA (all factors), exact matc… 28.7% ▂····▂··▂·█● beats 8/8 medium · avg 36% 0.2% blunt
TruthfulQA MC2 39.9% ▄······▄●▄▄█ beats 2/6 medium · avg 56% 0.2% blunt · at its level · several runs
SuperGLUE WSC (prompted) 35.6% ●·······▂·▂▃ beats 4/8 medium · avg 68% 0.1% blunt · noisy
IndoMMLU (zero-shot) 25.3% ▅▃●··▃···▃▃█ beats 3/10 hard · avg 37% 0.1% blunt · too hard for it · noisy
LAMBADA (OpenAI version), last-wo… 0.0% ●·······▂▃▂█ beats 0/10 easy · avg 85% 0.0% blunt · at its level · noisy
WebQuestions (closed book) 0.0% ●···▄··▄▄▄·▄ beats 0/6 very hard · avg 3.5% 0.0% blunt · too hard for it
VEC visual concepts, zero-shot pe… 61.5% ●▅▃█·▃··▃·▃▃ beats 1/11 hard · avg 66% 0.0% blunt · at its level
LogiQA (harness) 26.4% █··········● beats 14/14 very hard · avg 25% 0.0% too hard for it · noisy
source results share of the score
EleutherAI lm-evaluation-harness result… 19 23% ▇▇▇
LLaMA: Open and Efficient Foundation La… 15 17% ▇▇
InstructEval 3 8% ▇
How Far Can Camels Go? Exploring the St… 4 8% ▇
Enhancing Chat Language Models by Scali… 4 6% ▇
Outlier Suppression+: Accurate quantiza… 5 6% ▇
LLM-FP4: 4-Bit Floating-Point Quantized… 6 4% ▇
A Frustratingly Easy Post-Training Quan… 4 4% ▇
Coverage-based Example Selection for In… 1 3% ▇
Examining Inter-Consistency of Large La… 5 3% ▇
LMArena leaderboard dataset 1 3% ▇
Polyrating: A Cost-Effective and Bias-A… 1 2% ▇
Towards a Holistic Evaluation of LLMs o… 2 2% ▇
LMSYS Chatbot Arena leaderboard Space: … 1 2% ▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 2% ▇
Investigating Table-to-Text Generation … 1 2% ▇
Enhancing Computation Efficiency in Lar… 1 2% ▇
Can Language Models Understand Physical… 2 1% ▇
Small Language Models Fine-tuned to Coo… 1 1% ▇
SCITAB: A Challenging Benchmark for Com… 1 1% ▇
Generative Calibration for In-context L… 1 0% ▇
Battle of the Large Language Models: Do… 1 0% ▇
This is not a Dataset: A Large Negation… 1 0% ▇
MenatQA: A New Dataset for Testing the … 1 0% ▇
Large Language Models Only Pass Primary… 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 32.4
Licences: EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · LLaMA: Open and Efficient Foundation Language Models ([CC BY 4.0](../../licenses/paper-2302.13971v1-CC-BY-4.0.txt)) · InstructEval ([MIT / Apache-2.0](https://github.com/declare-lab/instruct-eval/blob/49bbd2ce7e3a5bdc4024ba62a5c9c68c107ee966/LICENSE)) · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources ([CC BY 4.0](../../licenses/paper-2306.04751v2-tulu1-CC-BY-4-0.txt)) · Enhancing Chat Language Models by Scaling High-quality Instructional Conversations ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-183-CC-BY-4-0.txt)) · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-102-CC-BY-4-0.txt)) · LLM-FP4: 4-Bit Floating-Point Quantized Transformers ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-39-CC-BY-4-0.txt)) · A Frustratingly Easy Post-Training Quantization Scheme for LLMs ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-892-CC-BY-4-0.txt)) · Coverage-based Example Selection for In-Context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-930-CC-BY-4-0.txt)) · Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-508-CC-BY-4-0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · LMSYS Chatbot Arena leaderboard Space: MT-Bench column of the published leaderboard table ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios ([CC BY 4.0](../../licenses/acl-2023-emnlp-industry-17-CC-BY-4-0.txt)) · Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-910-CC-BY-4-0.txt)) · Can Language Models Understand Physical Concepts? ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-726-CC-BY-4-0.txt)) · Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-225-CC-BY-4-0.txt)) · SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-483-CC-BY-4-0.txt)) · Generative Calibration for In-context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-152-CC-BY-4-0.txt)) · Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing Comparison ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-750-CC-BY-4-0.txt)) · This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-531-CC-BY-4-0.txt)) · MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-100-CC-BY-4-0.txt)) · Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-760-CC-BY-4-0.txt))
Not counted (1)
source test score why
EleutherAI lm-evaluat… TriviaQA 0.0% failed run: Exactly 0.0% on TriviaQA (0-shot) for a model that scores 56–63% (5-shot) in its own paper, while far smaller BLOOM models score 1–6% in the same harness folder; an exact zero points to an answer-format or scoring failure, not ability.