[whatmodel.sh](../../README.md) / OPT-66B
# OPT-66B
Meta · released 2022-05 · 66B params · opt-175b-license-noncommercial
**score 29.4** ±0.7 · #669 of 732 · from 37 benchmark results across 8 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks OPT-66B among every model on that task or test, worst to best.
task vs the others rank counts tests
math ▁●▃▅█▆▄█▄▂▁▁ #424 of 431* 1.3% 1
math · #424 of 431 · 1.3% of its score
Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 0.0% ●▂▂▂▃▄▅▇██▆▄ beats 0/206 medium · avg 64% 1.3% sharp · noisy
other tests (16)
test result vs the others in its run difficulty counts why
ARC Challenge 37.2% █··▁·▁▁·▁▂▁● beats 19/19 easy · avg 81% 39% sharp · several runs
WinoGrande 68.7% ▄····▄▆▂▄▆▆● beats 19/19 easy · avg 88% 16% at its level · several runs
ARC Easy 71.7% ▃·▅▃▃▅·██▃█● beats 19/19 easy · avg 87% 10% blunt · at its level · several runs
PIQA 78.8% ▂·▄▄·▄▂▆▂▄█● beats 19/19 easy · avg 87% 7.5% blunt · at its level · several runs
ImageNetVC visual commonsense, te… 52.8% ▂·▄·▂█▄▄●▄▂▆ beats 14/21 easy · avg 74% 3.8% blunt · at its level
Story Cloze Test 77.3% ▂▂▄▂·▂·▆·▂●▂ beats 13/14 easy · avg 89% 3.6% blunt · at its level · several runs
HellaSwag 56.4% ▅●······▃▃·▅ beats 3/8 very easy · avg 90% 3.5% at its level · several runs
LAMBADA (OpenAI version), last-wo… 73.9% ▃··▅▃▃▅▅▅██● beats 19/19 very easy · avg 87% 3.1% blunt · at its level · several runs
COPA 74.0% ▁·▁▁▁▅▅█●▂▁▁ beats 45/67 easy · avg 92% 2.8% at its level · several runs
SciQ 92.6% ▂··▂▃▂▂▆▃▃█● beats 19/19 very easy · avg 98% 2.6% blunt · several runs
BoolQ 60.7% ▁····▁▂●▆▅▆▄ beats 17/103 medium · avg 78% 2.1% typical
OpenBookQA 46.4% ▅▁▁▁▂▂▃●▃▁▃▁ beats 54/107 easy · avg 77% 2.0% at its level
Natural Questions (closed book) 0.3% ●▁▁▁▂▄▄▃▅█▅▁ beats 1/68 very hard · avg 13% 1.1% too hard for it
Open PT-LLM Leaderboard average (… 35.0% ▃▁▁●▁▁▁▁▂█▅▁ beats 41/276 medium · avg 72% 0.8% noisy
SuperGLUE WSC (prompted) 54.8% █·······●·▂▃ beats 5/8 medium · avg 68% 0.2% blunt · noisy
LogiQA (harness) 22.7% ●··········▁ beats 0/14 very hard · avg 25% 0.0% too hard for it · noisy
source results share of the score
Pythia: A Suite for Analyzing Large Lan… 6 23% ▇▇▇
GPTQ: Accurate Post-Training Quantizati… 5 21% ▇▇▇
EleutherAI lm-evaluation-harness result… 8 20% ▇▇
UPRISE: Universal Prompt Retrieval for … 9 18% ▇▇
Outlier Suppression+: Accurate quantiza… 6 12% ▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 4% ▇
Bridging the Resource Gap: Exploring th… 1 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
prior: pull toward the middle 0% ▇
weighted average = 29.4
Licences: Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling ([CC BY-SA 4.0](../../licenses/paper-2304.01373v2-CC-BY-SA-4.0.txt)) · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers ([CC BY 4.0](../../licenses/paper-2210.17323v2-gptq-CC-BY-4-0.txt)) · EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-758-CC-BY-4-0.txt)) · Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-102-CC-BY-4-0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Bridging the Resource Gap: Exploring the Efficacy of English and Multilingual LLMs for Swedish ([CC BY 4.0](../../licenses/acl-2023-resourceful-1-13-CC-BY-4-0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt))
## How to run it
### Other ways to run it
1. Run it as published (fp16, needs ~132.37 GB, from [huggingface.co/facebook](https://huggingface.co/facebook/opt-66b/tree/7259969061237fe940036d22bea0fd349e4485e9))
```sh
uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='facebook/opt-66b', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/facebook](https://huggingface.co/facebook/opt-66b)
[interactive version](https://whatmodel.sh/model/opt-66b) · [JSON](https://whatmodel.sh/api?format=json&all=1)