[whatmodel.sh](../../README.md) / LLaMA 33B
# LLaMA 33B
Meta · released 2023-02 · 32.5B params · llama-1-noncommercial-research
**score 37.5** ±0.5 · comparison only: no current run option verified · from 61 benchmark results across 15 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks LLaMA 33B among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁▁●▄▆█▇▅▄▃▁ #508 of 539 13% 2
math ▁▁▃▆●█▅█▃▁▁▁ #347 of 530 19% 3
coding · #508 of 539 · 13% of its score
Beats [Qwen3.5 2B](qwen3-5-2b.md), [Molmo 7B-D (0924)](molmo-7b-d.md), [Tulu 2 7B](tulu-2-7b.md) here, though they rank higher overall. Trails [Granite 3.0 1B-A400M Instruct](granite-3-0-1b-a400m.md), [Llama 3.2 1B Instruct](llama-3-2-1b.md), [SmolLM2 1.7B Instruct](smollm2-1-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 25.0% █···█·●██··█ beats 2/5 medium · avg 61% 10% sharp · several runs
MBPP 40.9% ▄··▄·●··█··▄ beats 2/5 medium · avg 59% 3.0% at its level · several runs
math · #347 of 530 · 19% of its score
Beats [Molmo 2 8B](molmo-2-8b.md), [DeepSeek-Coder-V2 Instruct (0724)](deepseek-coder-v2-0724.md), [Llama 3.1 70B Instruct](llama-3-1-70b.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [Mistral 7B v0.1 (base)](mistral-7b-v0-1.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 30.5% ▄▄█▄···▄·▄·● beats 7/7 easy · avg 65% 16% sharp · at its level · several runs
MATH 8.4% █·█·●·██···█ beats 2/5 hard · avg 24% 3.1% too hard for it · several runs
MathQA 30.6% ▃·▃▃█▅··▃·▃● beats 10/10 hard · avg 34% 0.2% blunt · small run
other tests (23)
test result vs the others in its run difficulty counts why
ARC Challenge 46.8% █···▁▁·▃▁▃▁● beats 15/15 medium · avg 74% 16% sharp · at its level · several runs
WinoGrande 72.8% ▂····▄·▂▄▄█● beats 15/15 easy · avg 88% 8.0% at its level · several runs
MMLU 56.8% ▄····▄●·▄··█ beats 2/5 easy · avg 82% 7.7% at its level · several runs
HellaSwag 79.2% ▅█······▃▃·● beats 7/8 easy · avg 81% 5.6% at its level · several runs
OpenBookQA 29.4% █········▂●▂ beats 6/7 hard · avg 40% 3.9% too hard for it · several runs
BIG-Bench Hard 49.5% █···▄··▄··●▄ beats 4/5 easy · avg 74% 3.2% sharp · at its level
TriviaQA 68.5% █···██··●·██ beats 3/5 easy · avg 83% 2.6% at its level · several runs
ARC Easy 75.3% ▅▃·▃·▅▅·█▅▃● beats 15/15 easy · avg 87% 2.6% blunt · at its level · several runs
Natural Questions (closed book) 24.1% █·····▄●▄··▄ beats 3/5 medium · avg 41% 2.5% several runs
PIQA 81.0% ▅▃·▃▃▃▅▃▅▃█● beats 15/15 easy · avg 87% 2.4% blunt · at its level · several runs
F2WTQ free-form table QA, 2-shot … 31.8% ▄···█·●█··▄▄ beats 3/7 medium · avg 56% 2.3% at its level
ImageNetVC visual commonsense, te… 61.3% ▂·▄·▂█▄▄█▄▂● beats 19/21 easy · avg 74% 1.9% blunt · at its level
BoolQ 68.4% ▄··▄·█▄··●·▄ beats 5/7 easy · avg 90% 1.7% at its level · several runs
Factual knowledge recall, Premium… 27.3% ▅····▃▅·█●·█ beats 9/13 medium · avg 52% 1.5% at its level
AQuA-RAT 20.0% ●··▁▁▁··▁▁▁▁ beats 0/27 hard · avg 47% 1.3% sharp · too hard for it · noisy · small…
Factual knowledge recall, Premium… 48.9% ▃▅▃··▅▃·█▃●▅ beats 11/13 medium · avg 61% 0.8% at its level
COPA 90.0% ▄··█▄▄···▄·● beats 6/7 very easy · avg 95% 0.8% typical
Spider dev execution accuracy, ze… 4.7% ●·█▄··▄····▄ beats 1/6 medium · avg 54% 0.7% sharp · noisy
Winograd Schema Challenge (WSC273) 86.8% ▄▄·▄·▄··█··● beats 7/7 very easy · avg 93% 0.6% blunt
This is not a Dataset (negation),… 52.4% █▁····●▁·▁·▁ beats 10/14 hard · avg 66% 0.6% sharp · too hard for it · noisy
12-dataset text classification, 4… 72.2% ▂▂··▂·█···▂● beats 8/8 easy · avg 79% 0.5% blunt · at its level
Social IQA (maker) 50.4% ▁·▁██▄●▁▂▃▇▁ beats 35/59 medium · avg 63% 0.5% blunt · at its level
IndoMMLU (zero-shot) 26.5% ▅▃▅··●···▃▃█ beats 5/10 hard · avg 37% 0.1% blunt · noisy
source results share of the score
EleutherAI lm-evaluation-harness result… 11 25% ▇▇▇
Mixtral of Experts 12 21% ▇▇▇
LLaMA: Open and Efficient Foundation La… 15 21% ▇▇▇
How Far Can Camels Go? Exploring the St… 4 11% ▇
Outlier Suppression+: Accurate quantiza… 5 6% ▇
A Frustratingly Easy Post-Training Quan… 4 4% ▇
Towards a Holistic Evaluation of LLMs o… 2 2% ▇
Investigating Table-to-Text Generation … 1 2% ▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 2% ▇
Enhancing Computation Efficiency in Lar… 1 2% ▇
Small Language Models Fine-tuned to Coo… 1 1% ▇
Battle of the Large Language Models: Do… 1 1% ▇
This is not a Dataset: A Large Negation… 1 1% ▇
Generative Calibration for In-context L… 1 0% ▇
Large Language Models Only Pass Primary… 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 37.5
Licences: EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · Mixtral of Experts ([CC BY 4.0](../../licenses/CC-BY-4.0.txt)) · LLaMA: Open and Efficient Foundation Language Models ([CC BY 4.0](../../licenses/paper-2302.13971v1-CC-BY-4.0.txt)) · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources ([CC BY 4.0](../../licenses/paper-2306.04751v2-tulu1-CC-BY-4-0.txt)) · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-102-CC-BY-4-0.txt)) · A Frustratingly Easy Post-Training Quantization Scheme for LLMs ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-892-CC-BY-4-0.txt)) · Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios ([CC BY 4.0](../../licenses/acl-2023-emnlp-industry-17-CC-BY-4-0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-910-CC-BY-4-0.txt)) · Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-225-CC-BY-4-0.txt)) · Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing Comparison ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-750-CC-BY-4-0.txt)) · This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-531-CC-BY-4-0.txt)) · Generative Calibration for In-context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-152-CC-BY-4-0.txt)) · Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-760-CC-BY-4-0.txt))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/llama-33b) · [JSON](https://whatmodel.sh/api?format=json&all=1)