[whatmodel.sh](../../README.md) / OPT-13B (13 billion parameters)
# OPT-13B (13 billion parameters)
Meta · released 2022-05 · 13B params · opt-175b-license-noncommercial
**score 25.9** ±0.8 · #683 of 732 · from 37 benchmark results across 10 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks OPT-13B (13 billion parameters) among every model on that task or test, worst to best.
other tests (22)
test result vs the others in its run difficulty counts why
ARC Challenge 32.9% █··▁·▁▁·▁▂●▂ beats 17/19 easy · avg 81% 33% sharp · several runs
WinoGrande 65.2% ▄····▄▆▂▄▆▆● beats 16/19 easy · avg 88% 17% at its level · several runs
ARC Easy 67.1% ▃·▅▃▃▅·██▃●█ beats 15/19 easy · avg 87% 13% blunt · at its level · several runs
PIQA 76.0% ▂·▄▄·▄▂▆▂▄●▄ beats 15/19 easy · avg 87% 8.8% blunt · at its level · several runs
Story Cloze Test 76.6% ▂▂▄▂·▂·▆·▂●▂ beats 10/14 easy · avg 89% 4.6% blunt · at its level · several runs
LAMBADA (OpenAI version), last-wo… 68.6% ▃··▅▃▃▅▅▅█●█ beats 16/19 very easy · avg 87% 4.1% blunt · at its level · several runs
Natural Questions (closed book) 13.5% ▁·▁·▁▂▁▂·●▂█ beats 10/28 medium · avg 48% 3.9% typical
SciQ 90.8% ▂··▂▃▂▂▆▃▃●▂ beats 17/19 very easy · avg 98% 3.8% blunt · several runs
WebQSP closed-book zero-shot (ans… 28.4% ██···●█····█ beats 2/4 easy · avg 66% 2.3% at its level
VEC embodied concepts, zero-shot … 51.5% ▁····▁●·▁·▃█ beats 2/11 hard · avg 63% 1.8% too hard for it
HellaSwag 52.5% ●█······▃▃·▅ beats 0/8 easy · avg 86% 1.7% at its level
COPA 86.0% ▁·▁▁▁▅▅█●▂▁▁ beats 46/67 very easy · avg 95% 1.6% at its level · small run
12-dataset text classification, 4… 57.7% ▂▂··▂·●···▂▂ beats 5/8 easy · avg 79% 1.3% blunt · at its level
Open PT-LLM Leaderboard average (… 28.2% ●▁▁▁▁▁▁▁▂█▅▁ beats 15/276 medium · avg 72% 0.9% noisy
Mintaka (English) closed-book zer… 26.8% ▄····▄···▄·● beats 3/4 medium · avg 48% 0.8% blunt
TriviaQA 41.0% ▁··▁█▃▃▁▄●▄█ beats 16/28 easy · avg 73% 0.7% at its level · noisy
MenatQA (all factors), exact matc… 21.9% ▂····▂··▂·●▄ beats 3/8 medium · avg 36% 0.4% blunt
SuperGLUE WSC (prompted) 60.6% █·······▂·▂● beats 7/8 medium · avg 68% 0.2% blunt · noisy
HaluEval QA hallucination detecti… 35.3% ●▁▁·▁▁·▁▁▁▁▁ beats 8/27 hard · avg 66% 0.2% too hard for it · noisy
TruthfulQA MC2 34.0% ●▃█▆▁▂▅···▁▁ beats 1/31 medium · avg 57% 0.2% blunt · at its level
VEC visual concepts, zero-shot pe… 68.5% ▅▅▃█·▃··▃·●▃ beats 10/11 hard · avg 66% 0.0% blunt · at its level
LogiQA (harness) 22.7% ●··········▁ beats 2/14 very hard · avg 25% 0.0% too hard for it · noisy
source results share of the score
Pythia: A Suite for Analyzing Large Lan… 6 26% ▇▇▇
GPTQ: Accurate Post-Training Quantizati… 5 24% ▇▇▇
EleutherAI lm-evaluation-harness result… 8 23% ▇▇▇
Outlier Suppression+: Accurate quantiza… 7 15% ▇▇
Hallucinations Leaderboard results 4 5% ▇
Knowledge-Augmented Language Model Prom… 2 3% ▇
Can Language Models Understand Physical… 2 2% ▇
Generative Calibration for In-context L… 1 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
MenatQA: A New Dataset for Testing the … 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 25.9
Licences: Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling ([CC BY-SA 4.0](../../licenses/paper-2304.01373v2-CC-BY-SA-4.0.txt)) · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers ([CC BY 4.0](../../licenses/paper-2210.17323v2-gptq-CC-BY-4-0.txt)) · EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-102-CC-BY-4-0.txt)) · Hallucinations Leaderboard results ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering ([CC BY 4.0](../../licenses/acl-2023-nlrse-1-7-CC-BY-4-0.txt)) · Can Language Models Understand Physical Concepts? ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-726-CC-BY-4-0.txt)) · Generative Calibration for In-context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-152-CC-BY-4-0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-100-CC-BY-4-0.txt))
## How to run it
### Other ways to run it
1. Run it as published (fp16, needs ~25.71 GB, from [huggingface.co/facebook](https://huggingface.co/facebook/opt-13b/tree/e515202d1e7750da62d245fbccb2723b9c1790f5))
```sh
uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='facebook/opt-13b', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/facebook](https://huggingface.co/facebook/opt-13b)
[interactive version](https://whatmodel.sh/model/opt-13b) · [JSON](https://whatmodel.sh/api?format=json&all=1)