[whatmodel.sh](../../README.md) / OPT-1.3B
# OPT-1.3B
Meta · released 2022-05 · 1.3B params · opt-175b-license-noncommercial
**score 14.2** ±0.8 · #711 of 732 · from 61 benchmark results across 11 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks OPT-1.3B among every model on that task or test, worst to best.
task vs the others rank counts tests
math ▁▁▃●█▆▄█▄▂▁▁ #364 of 431* 0.8% 1
math · #364 of 431 · 0.8% of its score
Beats [GPT-3.5 Turbo (0125)](gpt-3-5-turbo-0125.md), [DBRX Instruct](dbrx-instruct.md), [Command R Plus (04-2024)](command-r-plus-04-2024.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
MathQA 23.3% ▃·▃●█▅··▃·▃▃ beats 2/10 hard · avg 33% 0.8% blunt · too hard for it
other tests (25)
test result vs the others in its run difficulty counts why
ARC Easy 57.0% ▃·▅▃▃▅·●█▃██ beats 9/19 easy · avg 87% 17% blunt · at its level · several runs
ARC Challenge 23.1% ●··▁·▁▁·▁▂▁▂ beats 0/19 easy · avg 81% 17% sharp · too hard for it · several runs
WinoGrande 59.7% ▄····▄▆▂▄●▆█ beats 12/19 easy · avg 88% 12% several runs
PIQA 71.7% ▂·▄▄·▄▂▆●▄█▄ beats 11/19 easy · avg 87% 10% blunt · at its level · several runs
SciQ 84.5% ▂··▂▃▂▂●▃▃█▂ beats 8/19 very easy · avg 98% 6.3% blunt · several runs
LAMBADA (OpenAI version), last-wo… 57.9% ▃··▅▃▃▅▅●███ beats 10/19 very easy · avg 87% 5.6% blunt · at its level · several runs
HellaSwag 41.5% ▃●▃··▅····▃█ beats 1/8 easy · avg 82% 4.9% several runs
Story Cloze Test 70.8% ▂▂▄▂·▂·●·▂█▂ beats 6/14 easy · avg 89% 4.1% blunt · at its level · several runs
COPA 79.0% ▁·▁▁▁▅▅●▆▂▁▁ beats 29/67 very easy · avg 96% 3.9% at its level · several runs
OpenBookQA 24.0% ●······▂·▆·▂ beats 2/8 very hard · avg 29% 3.8% too hard for it · small run · several r…
ImageNetVC visual commonsense, te… 40.7% ▂·▄·▂●▄▄█▄▂▆ beats 4/21 easy · avg 74% 3.7% blunt · at its level
BoolQ 59.6% ●█▄···▄··█▄▄ beats 0/8 easy · avg 88% 2.6% several runs
CommonsenseQA 17.9% ●·····▂▂▃▂▄▄ beats 5/38 easy · avg 83% 1.1% sharp · too hard for it · small run
RTE (prompted) (maker) 54.2% ▅▃·▃··●·▅▅▃█ beats 4/13 medium · avg 75% 1.1% too hard for it · small run
Winograd Schema Challenge (WSC273… 73.3% █▃·●▃▃█▅██▅▃ beats 5/21 easy · avg 92% 1.0% blunt · at its level · small run
MMLU (maker) 25.9% ▁●▁▁▁▂▂▅██▆▄ beats 3/214 medium · avg 59% 0.8% too hard for it · small run
VEC embodied concepts, zero-shot … 51.3% ▁····●▁·▁·▃█ beats 1/11 hard · avg 63% 0.8% too hard for it
BIG-Bench Hard (maker) 27.9% ▁▁●▁▂▄▅▆█▃▃▁ beats 5/155 medium · avg 59% 0.6% sharp · too hard for it · noisy · small…
WiC (prompted) (maker) 50.5% █·●▁······▁▁ beats 9/13 medium · avg 71% 0.4% sharp · too hard for it · noisy · small…
Open PT-LLM Leaderboard average (… 25.6% ●▁▁▁▁▁▁▁▂█▅▁ beats 17/276 medium · avg 72% 0.4% too hard for it · noisy
SuperGLUE WSC (prompted) 38.5% ●·······▂·▂▃ beats 1/8 medium · avg 68% 0.3% blunt · noisy · several runs
CommitmentBank (prompted) (maker) 42.9% ▂▃·▂●▂▂▂·▂·▂ beats 6/13 medium · avg 60% 0.2% blunt · at its level · small run
PubMedQA 57.9% ▂·····▅●▅▆▆█ beats 5/24 medium · avg 73% 0.2% blunt · at its level
VEC visual concepts, zero-shot pe… 67.3% ▅▅▃█·▃··●·▃▃ beats 9/11 hard · avg 66% 0.0% blunt · at its level
LogiQA (harness) 22.3% ●··········▁ beats 5/14 very hard · avg 25% 0.0% too hard for it · noisy
source results share of the score
Pythia: A Suite for Analyzing Large Lan… 6 19% ▇▇
GPTQ: Accurate Post-Training Quantizati… 5 15% ▇▇
EleutherAI lm-evaluation-harness result… 8 15% ▇▇
Preserving Privacy Through Dememorizati… 9 14% ▇▇
OPT-IML: Scaling Language Model Instruc… 15 13% ▇▇
Textbooks Are All You Need II: phi-1.5 … 7 11% ▇
Tunable Soft Prompts are Messengers in … 5 7% ▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 4% ▇
Symbolic Chain-of-Thought Distillation:… 2 2% ▇
Can Language Models Understand Physical… 2 1% ▇
Open Portuguese LLM Leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 14.2
Licences: Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling ([CC BY-SA 4.0](../../licenses/paper-2304.01373v2-CC-BY-SA-4.0.txt)) · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers ([CC BY 4.0](../../licenses/paper-2210.17323v2-gptq-CC-BY-4-0.txt)) · EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-265-CC-BY-4-0.txt)) · OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization ([CC BY 4.0](../../licenses/paper-2212.12017v3-opt-iml-CC-BY-4-0.txt)) · Textbooks Are All You Need II: phi-1.5 technical report ([CC BY 4.0](../../licenses/paper-2309.05463v1-CC-BY-4.0.txt)) · Tunable Soft Prompts are Messengers in Federated Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-976-CC-BY-4-0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Symbolic Chain-of-Thought Distillation: Small Models Can Also \`\`Think'' Step-by-Step ([CC BY 4.0](../../licenses/acl-2023-acl-long-150-CC-BY-4-0.txt)) · Can Language Models Understand Physical Concepts? ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-726-CC-BY-4-0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt))
## How to run it
### Other ways to run it
1. Run it as published (fp16, needs ~2.63 GB, from [huggingface.co/facebook](https://huggingface.co/facebook/opt-1.3b/tree/3f5c25d0bc631cb57ac65913f76e22c2dfb61d62))
```sh
uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text-generation', model='facebook/opt-1.3b')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/facebook](https://huggingface.co/facebook/opt-1.3b)
[interactive version](https://whatmodel.sh/model/opt-1-3b) · [JSON](https://whatmodel.sh/api?format=json&all=1)