[whatmodel.sh](../../README.md) / LLaMA 7B
# LLaMA 7B
Meta · released 2023-02 · 6.7B params · llama-1-noncommercial-research
**score 30.3** ±0.5 · comparison only: no current run option verified · from 96 benchmark results across 23 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks LLaMA 7B among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁●▃▄▆█▇▅▄▃▁ #524 of 539 6.4% 2
math ▁▁▃●██▅█▃▁▁▁ #464 of 530 11% 3
coding · #524 of 539 · 6.4% of its score
Beats [Molmo 7B-D (0924)](molmo-7b-d.md), [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Qwen3 0.6B](qwen3-0-6b.md) here, though they rank higher overall. Trails [Falcon3 1B Instruct](falcon3-1b.md), [Gemma 2B IT](gemma-2b.md), [Granite 3.0 1B-A400M Instruct](granite-3-0-1b-a400m.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 10.3% ▄··▄····▄▄●▄ beats 4/6 medium · avg 63% 5.4% sharp · several runs
MBPP (maker) 17.7% ●▁▁▄▅█▆▅▄▁▂▁ beats 2/86 medium · avg 59% 1.0% typical
math · #464 of 530 · 11% of its score
Beats [Yi 34B Chat](yi-34b-chat.md), [Llama 3 8B Instruct](llama-3-8b.md), [Command R (03-2024)](command-r-03-2024.md) here, though they rank higher overall. Trails [GPT-Neo 1.3B](gpt-neo-1-3b.md), [OPT-1.3B](opt-1-3b.md), [OPT-2.7B](opt-2-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 8.0% ▄▄█▄···●·▄·▄ beats 5/7 easy · avg 65% 10% sharp · several runs
MATH (maker) 2.9% ●▅▂▆▅▃▅▅█▃▃▂ beats 1/30 hard · avg 24% 1.0% too hard for it
MathQA 26.8% ▃·▃▃█▅··●·▃▃ beats 8/10 hard · avg 34% 0.2% blunt · small run
other tests (38)
test result vs the others in its run difficulty counts why
ARC Challenge 44.6% ▁·▁··▁▁▇▂█●▃ beats 20/26 easy · avg 82% 24% sharp · several runs
WinoGrande 69.9% ▁····▁··▁▂█● beats 23/26 easy · avg 88% 11% at its level · several runs
HellaSwag 76.2% ▃···▁▇▅█●▃▁▁ beats 21/26 easy · avg 81% 7.4% at its level · several runs
BoolQ 75.0% ▂▃▂▅▆▂▅▆●▂▂▂ beats 22/26 easy · avg 88% 5.2% at its level · several runs
OpenBookQA 34.4% █·▁▁▃·▁▃▅▇●▄ beats 23/26 easy · avg 81% 5.1% at its level · several runs
ARC Easy 72.9% ▁▁·▁▁▄▅▄█●▁▃ beats 22/26 easy · avg 85% 4.5% blunt · at its level · several runs
BIG-Bench Hard 30.9% █●█··█·█·█·█ beats 1/6 medium · avg 68% 4.1% sharp · several runs
MMLU 35.2% ▄▄·▄···●▄·██ beats 3/8 easy · avg 74% 3.8% several runs
PIQA 78.7% ▁·▁··▁▁▃█▅●▂ beats 19/26 easy · avg 87% 3.5% blunt · at its level · several runs
SciQ 94.6% ▁·▁·▁·▃▅▃█●▁ beats 23/26 very easy · avg 97% 1.6% blunt · several runs
ImageNetVC visual commonsense, te… 55.7% ▂·▄·▂█▄▄█●▂▆ beats 16/21 easy · avg 74% 1.6% blunt · at its level
F2WTQ free-form table QA, 2-shot … 19.0% ▄···●·▄█··▄▄ beats 1/7 medium · avg 56% 1.4% typical
Social IQA 44.8% ▁···▁▃█▁●▁·▁ beats 22/26 medium · avg 60% 1.3% blunt · several runs
TriviaQA (maker) 56.3% ▁▁▂▂▄▄▃▆●█▅▂ beats 50/80 easy · avg 84% 1.0% at its level
Factual knowledge recall, Premium… 14.6% ▅····▃●·██·█ beats 3/13 medium · avg 52% 0.9% typical
COPA 84.0% ▄··█▄▄···●·█ beats 5/7 very easy · avg 95% 0.8% at its level
Natural Questions (closed book) (… 22.0% ▂▁▁▁▂▄▄▃▅●▅▁ beats 46/68 medium · avg 53% 0.8% typical
VEC embodied concepts, zero-shot … 55.8% ▁····▁▁·▁·▃● beats 10/11 hard · avg 63% 0.7% typical
Factual knowledge recall, Premium… 35.0% ▃▅▃··▅●·█▃▃▅ beats 6/13 medium · avg 61% 0.6% at its level
CoQA 75.2% ▄····▄▄·▄▄·● beats 5/6 very easy · avg 92% 0.6% typical
DiBiMT lexical-ambiguity translat… 50.2% ██···█●···██ beats 3/5 easy · avg 74% 0.5% blunt · at its level
Winograd Schema Challenge (WSC273) 81.0% ▄▄·▄·▄··●··█ beats 4/7 very easy · avg 93% 0.5% blunt · at its level
LAMBADA (OpenAI version), last-wo… 73.5% ▁··▂▁▁▁▂▆█●▃ beats 22/26 very easy · avg 86% 0.5% blunt · at its level · several runs
RACE-high 39.9% ▄···█··▄▄··● beats 6/6 medium · avg 51% 0.4% blunt
SciTab 3-class claim verification… 32.3% █···●▄·▄▄▄·▄ beats 2/7 medium · avg 51% 0.4% blunt · at its level
DROP 11.3% ▄▄···█·▄·●·▄ beats 5/6 hard · avg 34% 0.4% blunt
Spider dev execution accuracy, ze… 4.1% ●·█▄··▄····▄ beats 0/6 medium · avg 54% 0.3% sharp · noisy
This is not a Dataset (negation),… 50.4% █●····▃▁·▁·▁ beats 8/14 hard · avg 66% 0.2% sharp · too hard for it · noisy
MenatQA (all factors), exact matc… 23.4% ▂····▂··▂·●▄ beats 4/8 medium · avg 36% 0.1% blunt
TruthfulQA MC2 34.1% ●······▄▄▄▄█ beats 0/6 medium · avg 58% 0.1% blunt · at its level
IndoMMLU (zero-shot) 25.3% ▅▃●··▃···▃▃█ beats 4/10 hard · avg 37% 0.0% blunt · too hard for it · noisy
MC-TACO as yes/no plausibility cl… 65.0% █··········● beats 4/4 easy · avg 75% 0.0% blunt · at its level
WebQuestions (closed book) 0.0% ●···▄··▄▄▄·▄ beats 1/6 very hard · avg 3.5% 0.0% blunt · too hard for it
VEC visual concepts, zero-shot pe… 62.6% ▅●▃█·▃··▃·▃▃ beats 3/11 hard · avg 66% 0.0% blunt · at its level
LogiQA (harness) 21.8% ●··········▁ beats 8/14 very hard · avg 25% 0.0% too hard for it · noisy
SVAMP 32.8% █▆▆▅▂▅▂●▂▂▅▃ beats 21/30 easy · avg 82% — not counted yet: no other model in the …
Super-NaturalInstructions English… 11.0% ▄·●······▄▄█ beats 1/5 medium · avg 63% — not counted yet: no other model in the …
Super-NaturalInstructions English… 23.6% ●··█···█·█·█ beats 0/4 easy · avg 71% — not counted yet: no other model in the …
source results share of the score
Stability AI 10 20% ▇▇
EleutherAI lm-evaluation-harness result… 18 18% ▇▇
LLaMA: Open and Efficient Foundation La… 15 14% ▇▇
OLMo: Accelerating the Science of Langu… 8 9% ▇
Textbooks Are All You Need II: phi-1.5 … 8 7% ▇
InstructEval 3 6% ▇
How Far Can Camels Go? Exploring the St… 4 6% ▇
Outlier Suppression+: Accurate quantiza… 5 4% ▇
LLM-FP4: 4-Bit Floating-Point Quantized… 6 3% ▇
A Frustratingly Easy Post-Training Quan… 4 3% ▇
Coverage-based Example Selection for In… 1 2% ▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 2% ▇
Towards a Holistic Evaluation of LLMs o… 2 1% ▇
Investigating Table-to-Text Generation … 1 1% ▇
Can Language Models Understand Physical… 2 1% ▇
Aligning Large Language Models through … 1 1% ▇
Towards Effective Disambiguation for Ma… 1 1% ▇
SCITAB: A Challenging Benchmark for Com… 1 0% ▇
Battle of the Large Language Models: Do… 1 0% ▇
This is not a Dataset: A Large Negation… 1 0% ▇
MenatQA: A New Dataset for Testing the … 1 0% ▇
Large Language Models Only Pass Primary… 1 0% ▇
Do Language Models Have a Common Sense … 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 30.3
Licences: Stability AI ([Apache-2.0](../../licenses/stablelm-original-eval-json-Apache-2.0.txt)) · EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · LLaMA: Open and Efficient Foundation Language Models ([CC BY 4.0](../../licenses/paper-2302.13971v1-CC-BY-4.0.txt)) · OLMo: Accelerating the Science of Language Models ([CC BY 4.0](../../licenses/paper-2402.00838v4-CC-BY-4.0.txt)) · Textbooks Are All You Need II: phi-1.5 technical report ([CC BY 4.0](../../licenses/paper-2309.05463v1-CC-BY-4.0.txt)) · InstructEval ([MIT / Apache-2.0](https://github.com/declare-lab/instruct-eval/blob/49bbd2ce7e3a5bdc4024ba62a5c9c68c107ee966/LICENSE)) · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources ([CC BY 4.0](../../licenses/paper-2306.04751v2-tulu1-CC-BY-4-0.txt)) · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-102-CC-BY-4-0.txt)) · LLM-FP4: 4-Bit Floating-Point Quantized Transformers ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-39-CC-BY-4-0.txt)) · A Frustratingly Easy Post-Training Quantization Scheme for LLMs ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-892-CC-BY-4-0.txt)) · Coverage-based Example Selection for In-Context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-930-CC-BY-4-0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios ([CC BY 4.0](../../licenses/acl-2023-emnlp-industry-17-CC-BY-4-0.txt)) · Can Language Models Understand Physical Concepts? ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-726-CC-BY-4-0.txt)) · Aligning Large Language Models through Synthetic Feedback ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-844-CC-BY-4-0.txt)) · Towards Effective Disambiguation for Machine Translation with Large Language Models ([CC BY 4.0](../../licenses/acl-2023-wmt-1-44-CC-BY-4-0.txt)) · SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-483-CC-BY-4-0.txt)) · Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing Comparison ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-750-CC-BY-4-0.txt)) · This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-531-CC-BY-4-0.txt)) · MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-100-CC-BY-4-0.txt)) · Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-760-CC-BY-4-0.txt)) · Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-418-CC-BY-4-0.txt))
Not counted (12)
source test score why
EleutherAI lm-evaluat… TriviaQA 0.0% failed run: Exactly 0.0% on TriviaQA (0-shot) for a model that scores 56–63% (5-shot) in its own paper, while far smaller BLOOM models score 1–6% in the same harness folder; an exact zero points to an answer-format or scoring failure, not ability.
CRaSh: Clustering OpenBookQA 28.2% only model in this source's run
CRaSh: Clustering ARC Easy 67.3% only model in this source's run
CRaSh: Clustering ARC Challenge 38.2% only model in this source's run
CRaSh: Clustering PIQA 78.3% only model in this source's run
CRaSh: Clustering SciQ 89.7% only model in this source's run
CRaSh: Clustering HellaSwag 56.4% only model in this source's run
Tuna: Instruction Tun… Super-NaturalInstructions Eng… 11.0% too few models have it yet
Tuna: Instruction Tun… Super-NaturalInstructions Eng… 23.6% too few models have it yet
GRACE: Discriminator-… GSM8K 12.9% only model in this source's run
GRACE: Discriminator-… SVAMP 32.8% too few models have it yet
LogiCoT: Logical Chai… MMLU 31.8% only model in this source's run