[whatmodel.sh](../../README.md) / OPT-30B
# OPT-30B
Meta · released 2022-05 · 30B params · opt-175b-license-noncommercial
**score 28.0** ±0.7 · #676 of 732 · from 48 benchmark results across 11 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks OPT-30B among every model on that task or test, worst to best.
other tests (25)
test result vs the others in its run difficulty counts why
ARC Challenge 34.6% █··▁·▁▁·▁▂▁● beats 18/19 easy · avg 81% 29% sharp · several runs
WinoGrande 68.2% ▄····▄▆▂▄▆▆● beats 18/19 easy · avg 88% 16% at its level · several runs
ARC Easy 70.0% ▃·▅▃▃▅·██▃█● beats 17/19 easy · avg 87% 11% blunt · at its level · several runs
PIQA 77.6% ▂·▄▄·▄▂▆▂▄█● beats 18/19 easy · avg 87% 6.5% blunt · at its level · several runs
Story Cloze Test 77.3% ▂▂▄▂·▂·▆·▂●▂ beats 12/14 easy · avg 89% 4.4% blunt · at its level · several runs
TriviaQA 45.9% ▁▁▂▂▄▄▃▆●█▅▂ beats 43/80 easy · avg 77% 4.0% at its level · small run · several runs
ImageNetVC visual commonsense, te… 53.2% ▂·▄·▂█▄▄●▄▂▆ beats 15/21 easy · avg 74% 3.7% blunt · at its level
MMLU 25.9% ▁●▁▁▁▂▂▅██▆▄ beats 4/214 medium · avg 59% 3.6% too hard for it · small run · several r…
LAMBADA (OpenAI version), last-wo… 71.5% ▃··▅▃▃▅▅▅██● beats 18/19 very easy · avg 87% 3.1% blunt · at its level · several runs
SciQ 91.1% ▂··▂▃▂▂▆▃▃●▂ beats 18/19 very easy · avg 98% 2.7% blunt · several runs
COPA (maker) 84.0% ▁·▁▁▁▅▅●▆▂▁▁ beats 39/67 very easy · avg 95% 2.4% at its level · small run · several runs
BIG-Bench Hard (maker) 28.4% ▁▁▁●▂▄▅▆█▃▃▁ beats 7/155 medium · avg 59% 2.0% sharp · too hard for it · noisy · small…
OpenBookQA (maker) 57.2% ▅▁▁▁▂▂▃█●▁▃▁ beats 87/107 hard · avg 48% 1.9% too hard for it · small run · several r…
RTE (prompted) (maker) 58.1% ▅▃·▃··▅·●▅▃█ beats 6/13 medium · avg 75% 1.6% small run
WiC (prompted) (maker) 50.2% ●·▂▁······▁▁ beats 0/13 medium · avg 71% 1.3% sharp · too hard for it · noisy · small…
HellaSwag 54.3% ●█······▃▃·▅ beats 1/8 easy · avg 86% 1.3% at its level
Natural Questions (closed book) 15.0% ▂▁▁▁▂▄▄▃●█▅▁ beats 35/68 medium · avg 39% 1.1% small run
CommonsenseQA 53.2% █·····▂▂▃●▄▄ beats 24/38 very easy · avg 91% 1.1% sharp · noisy
12-dataset text classification, 4… 57.0% ▂▂··▂·●···▂▂ beats 4/8 easy · avg 79% 1.0% blunt · at its level
BoolQ (maker) 64.0% ▁····▁▂●▆▅▆▄ beats 31/103 easy · avg 84% 0.9% at its level · small run
Winograd Schema Challenge (WSC273… 83.5% █▃·▅▃▃█▅●█▅▃ beats 13/21 easy · avg 92% 0.8% blunt · at its level · small run
Open PT-LLM Leaderboard average (… 39.5% ▃▁▁▁▁●▁▁▂█▅▁ beats 48/276 medium · avg 72% 0.8% noisy
SuperGLUE WSC (prompted) 59.6% █·······▂·●▃ beats 6/8 medium · avg 68% 0.4% blunt · noisy · several runs
CommitmentBank (prompted) (maker) 28.6% ●▃·▂█▂▂▂·▂·▂ beats 0/13 medium · avg 60% 0.2% blunt · at its level · small run
LogiQA (harness) 21.7% ●··········▁ beats 1/14 very hard · avg 25% 0.0% too hard for it · noisy
source results share of the score
Pythia: A Suite for Analyzing Large Lan… 6 22% ▇▇▇
GPTQ: Accurate Post-Training Quantizati… 5 20% ▇▇
EleutherAI lm-evaluation-harness result… 8 19% ▇▇
OPT-IML: Scaling Language Model Instruc… 15 18% ▇▇
Outlier Suppression+: Accurate quantiza… 5 8% ▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 4% ▇
Increasing Probability Mass on Answer C… 3 4% ▇
Recitation-Augmented Language Models 2 3% ▇
Adaptation with Self-Evaluation to Impr… 1 2% ▇
Generative Calibration for In-context L… 1 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
prior: pull toward the middle 0% ▇
weighted average = 28.0
Licences: Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling ([CC BY-SA 4.0](../../licenses/paper-2304.01373v2-CC-BY-SA-4.0.txt)) · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers ([CC BY 4.0](../../licenses/paper-2210.17323v2-gptq-CC-BY-4-0.txt)) · EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization ([CC BY 4.0](../../licenses/paper-2212.12017v3-opt-iml-CC-BY-4-0.txt)) · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-102-CC-BY-4-0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-522-CC-BY-4-0.txt)) · Recitation-Augmented Language Models ([CC BY 4.0](../../licenses/paper-2210.01296v2-recitation-CC-BY-4-0.txt)) · Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-345-CC-BY-4-0.txt)) · Generative Calibration for In-context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-152-CC-BY-4-0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt))
## How to run it
### Other ways to run it
1. Run it as published (fp16, needs ~59.95 GB, from [huggingface.co/facebook](https://huggingface.co/facebook/opt-30b/tree/ceea0a90ac0f6fae7c2c34bcb40477438c152546))
```sh
uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='facebook/opt-30b', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/facebook](https://huggingface.co/facebook/opt-30b)
[interactive version](https://whatmodel.sh/model/opt-30b) · [JSON](https://whatmodel.sh/api?format=json&all=1)