[whatmodel.sh](../../README.md) / OPT-6.7B
# OPT-6.7B
Meta · released 2022-05 · 6.7B params · opt-175b-license-noncommercial
**score 23.7** ±0.7 · #693 of 732 · from 40 benchmark results across 11 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks OPT-6.7B among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ●▁▁▃▅▇█▇▅▄▃▁ #467 of 469* 1.8% 1
math ▁▁●▅█▆▄█▄▂▁▁ #413 of 431* 5.2% 1
coding · #467 of 469 · 1.8% of its score
Trails [GPT-Neo 2.7B](gpt-neo-2-7b.md), [Phi-1 (1.3B)](phi-1.md), [Pythia 6.9B](pythia-6-9b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 0.6% ●······▄·▄▄█ beats 0/5 medium · avg 59% 1.8% sharp · too hard for it
math · #413 of 431 · 5.2% of its score
Beats [Command R (03-2024)](command-r-03-2024.md), [Llama 2 13B Chat](llama-2-13b-chat.md), [Flan-T5 Large](flan-t5-large.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 3.0% ●···▄▄····▄▄ beats 0/5 easy · avg 67% 5.2% sharp · too hard for it · several runs
other tests (21)
test result vs the others in its run difficulty counts why
ARC Challenge 34.8% ▁·▁··▁▁●▂█▆▃ beats 6/26 easy · avg 82% 31% sharp · several runs
WinoGrande 65.4% ▁····▁··▁▂●▄ beats 12/26 easy · avg 88% 12% at its level · several runs
ARC Easy 60.1% ▁▁·▁▁●▅▄█▅▁▃ beats 6/26 easy · avg 87% 11% blunt · at its level · several runs
PIQA 76.3% ▁·▁··▁▁▃●▅█▂ beats 13/26 easy · avg 87% 7.3% blunt · at its level · several runs
SciQ 90.1% ▁·▁·▁·▃●▃█▄▁ beats 6/26 very easy · avg 97% 5.5% blunt · several runs
LAMBADA (OpenAI version), last-wo… 67.6% ▁··▂▁▁▁▂●█▅▃ beats 10/26 very easy · avg 87% 4.3% blunt · at its level · several runs
BoolQ 66.0% ▂▃▂▅●▂▅▆█▂▂▂ beats 10/26 easy · avg 88% 3.4% at its level
ImageNetVC visual commonsense, te… 48.5% ▂·▄·▂█▄●█▄▂▆ beats 11/21 easy · avg 74% 3.2% blunt · at its level
HellaSwag 67.2% ▃···▁▇●█▇▃▁▁ beats 9/26 very easy · avg 91% 2.9% at its level
Story Cloze Test 74.6% ▂▂▄▂·▂·▆·●█▂ beats 9/14 easy · avg 89% 2.4% blunt · at its level
BIG-Bench Hard 28.5% ●···▄··▄··▄▄ beats 1/5 easy · avg 74% 2.1% sharp · too hard for it
WebQSP closed-book zero-shot (ans… 29.8% ██···█●····█ beats 3/4 easy · avg 66% 1.6% at its level
OpenBookQA 27.6% █·▁▁▃·●▃▅▇▃▄ beats 10/26 hard · avg 49% 1.1% too hard for it
MMLU 24.6% ●·▄····▄·▄·█ beats 0/5 easy · avg 75% 1.0% noisy
Social IQA 42.6% ▁···▁▃●▁▁▁·▁ beats 18/26 hard · avg 56% 0.9% blunt
COPA 81.0% ▁·▁▁▁▅▅●▆▂▁▁ beats 33/67 very easy · avg 96% 0.7% at its level · small run
Mintaka (English) closed-book zer… 27.2% ▄····▄···▄·● beats 4/4 medium · avg 48% 0.5% blunt
Open PT-LLM Leaderboard average (… 28.6% ●▁▁▁▁▁▁▁▂█▅▁ beats 12/276 medium · avg 72% 0.5% noisy
MenatQA (all factors), exact matc… 16.1% ▂····▂··●·█▄ beats 2/8 medium · avg 36% 0.3% blunt
SuperGLUE WSC (prompted) 42.3% ●·······▂·▂▃ beats 0/8 medium · avg 68% 0.2% blunt · noisy
LogiQA (harness) 23.5% ●··········▁ beats 3/14 very hard · avg 25% 0.0% too hard for it · noisy
source results share of the score
Stability AI 10 34% ▇▇▇▇
Pythia: A Suite for Analyzing Large Lan… 6 18% ▇▇
GPTQ: Accurate Post-Training Quantizati… 5 16% ▇▇
EleutherAI lm-evaluation-harness result… 8 15% ▇▇
How Far Can Camels Go? Exploring the St… 4 7% ▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 3% ▇
INFORM : Information eNtropy based mult… 1 3% ▇
Knowledge-Augmented Language Model Prom… 2 2% ▇
Revisiting Block-based Quantisation: Wh… 1 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
MenatQA: A New Dataset for Testing the … 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 23.7
Licences: Stability AI ([Apache-2.0](../../licenses/stablelm-original-eval-json-Apache-2.0.txt)) · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling ([CC BY-SA 4.0](../../licenses/paper-2304.01373v2-CC-BY-SA-4.0.txt)) · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers ([CC BY 4.0](../../licenses/paper-2210.17323v2-gptq-CC-BY-4-0.txt)) · EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources ([CC BY 4.0](../../licenses/paper-2306.04751v2-tulu1-CC-BY-4-0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · INFORM : Information eNtropy based multi-step reasoning FOR large language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-216-CC-BY-4-0.txt)) · Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering ([CC BY 4.0](../../licenses/acl-2023-nlrse-1-7-CC-BY-4-0.txt)) · Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference? ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-617-CC-BY-4-0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-100-CC-BY-4-0.txt))
## How to run it
### Other ways to run it
1. Run it as published (fp16, needs ~13.32 GB, from [huggingface.co/facebook](https://huggingface.co/facebook/opt-6.7b/tree/a45aa65bbeb77c1558bc99bedc6779195462dab0))
```sh
uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='facebook/opt-6.7b', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/facebook](https://huggingface.co/facebook/opt-6.7b)
[interactive version](https://whatmodel.sh/model/opt-6-7b) · [JSON](https://whatmodel.sh/api?format=json&all=1)