[whatmodel.sh](../../README.md) / OPT-2.7B # OPT-2.7B Meta · released 2022-05 · 2.7B params · opt-175b-license-noncommercial **score 18.6** ±0.7 · #704 of 732 · from 40 benchmark results across 9 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks OPT-2.7B among every model on that task or test, worst to best. 
task  vs the others          rank  counts  tests
math  ▁▁▃●█▆▄█▄▂▁▁   #350 of 431*    0.7%      1
math · #350 of 431 · 0.7% of its score Beats [Athene-70B](athene-70b.md), [Gemma 2 9B IT](gemma-2-9b.md), [GPT-3.5 Turbo (0125)](gpt-3-5-turbo-0125.md) here, though they rank higher overall. 
test    result  vs the others in its run  difficulty      counts  why
MathQA   23.8%  ▃·▃▃●▅··▃·▃▃ beats 4/10   hard · avg 33%    0.7%  blunt · too hard for it
other tests (20) 
test                                result  vs the others in its run   difficulty           counts  why
ARC Challenge                        31.3%  ▁·▁··▁●▇▂█▆▃ beats 3/26    easy · avg 82%          26%  sharp · too hard for it · several runs
ARC Easy                             54.3%  ▁▁·▁●▄▅▄█▅▁▃ beats 3/26    easy · avg 87%          15%  blunt · at its level · several runs
WinoGrande                           61.0%  ▁····▁··▁●█▄ beats 5/26    easy · avg 88%          12%  several runs
PIQA                                 73.8%  ▁·▁··▁▁●█▅█▂ beats 4/26    easy · avg 87%         8.6%  blunt · at its level · several runs
SciQ                                 85.8%  ▁·▁·●·▃▅▃█▄▁ beats 2/26    very easy · avg 97%    7.9%  blunt · several runs
HellaSwag                            60.6%  ▃···▁●▅█▇▃▁▁ beats 4/26    very easy · avg 91%    6.7%  at its level · several runs
LAMBADA (OpenAI version), last-wo…   63.6%  ▁··▂▁▁●▂▆█▅▃ beats 5/26    very easy · avg 87%    5.4%  blunt · at its level · several runs
ImageNetVC visual commonsense, te…   45.8%  ▂·▄·▂█●▄█▄▂▆ beats 9/21    easy · avg 74%         3.5%  blunt · at its level
BoolQ                                60.3%  ▂●▂▅▆▂▅▆█▂▂▂ beats 2/26    easy · avg 88%         3.5%  at its level
Story Cloze Test                     71.7%  ▂▂▄▂·▂·●·▂█▂ beats 7/14    easy · avg 89%         2.8%  blunt · at its level
TriviaQA                             21.4%  ▁▁▂▂▄●▃▆▅█▅▂ beats 24/80   easy · avg 77%         2.0%  small run
COPA                                 77.0%  ▁·▁▁▁▅●█▆▂▁▁ beats 26/67   very easy · avg 96%    1.8%  at its level
WebQSP closed-book zero-shot (ans…   19.9%  ●█···██····█ beats 0/4     easy · avg 66%         1.6%  typical
Social IQA                           40.8%  ▁···▁●█▁▁▁·▁ beats 4/26    hard · avg 56%         0.9%  blunt
OpenBookQA                           25.0%  ●·▁▁▃·▁▃▅▇▃▄ beats 1/26    hard · avg 49%         0.8%  too hard for it
Mintaka (English) closed-book zer…   19.8%  ▄····●···▄·█ beats 1/4     medium · avg 48%       0.6%  blunt
Open PT-LLM Leaderboard average (…   27.0%  ●▁▁▁▁▁▁▁▂█▅▁ beats 14/276  medium · avg 72%       0.4%  noisy
PubMedQA                             60.8%  ▂·····▅█●▆▆█ beats 11/24   medium · avg 73%       0.2%  blunt · at its level
SuperGLUE WSC (prompted)             63.5%  █·······▂·▂● beats 8/8     medium · avg 68%       0.2%  blunt · noisy
LogiQA (harness)                     21.0%  ●··········▁ beats 4/14    very hard · avg 25%    0.0%  too hard for it · noisy
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Stability AI                                   10   33% ▇▇▇▇
Pythia: A Suite for Analyzing Large Lan…        6   18% ▇▇
GPTQ: Accurate Post-Training Quantizati…        5   15% ▇▇
EleutherAI lm-evaluation-harness result…        8   15% ▇▇
Preserving Privacy Through Dememorizati…        6   10% ▇
ImageNetVC: Zero- and Few-Shot Visual C…        1    4% ▇
Knowledge-Augmented Language Model Prom…        2    2% ▇
Adaptation with Self-Evaluation to Impr…        1    2% ▇
Open Portuguese LLM Leaderboard                 1    0% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 18.6
Licences: Stability AI ([Apache-2.0](../../licenses/stablelm-original-eval-json-Apache-2.0.txt)) · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling ([CC BY-SA 4.0](../../licenses/paper-2304.01373v2-CC-BY-SA-4.0.txt)) · GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers ([CC BY 4.0](../../licenses/paper-2210.17323v2-gptq-CC-BY-4-0.txt)) · EleutherAI lm-evaluation-harness results folder ([MIT](../../licenses/lm-eval-harness-results-2023-05-MIT.txt)) · Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-265-CC-BY-4-0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering ([CC BY 4.0](../../licenses/acl-2023-nlrse-1-7-CC-BY-4-0.txt)) · Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-345-CC-BY-4-0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt))
Not counted (2) 
source                  test   score  why
Adapting Language Mod…  BoolQ  55.8%  only model in this source's run
Adapting Language Mod…  COPA   66.8%  only model in this source's run
## How to run it ### Other ways to run it 1. Run it as published (fp16, needs ~5.3 GB, from [huggingface.co/facebook](https://huggingface.co/facebook/opt-2.7b/tree/905a4b602cda5c501f1b3a2650a4152680238254)) ```sh uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text-generation', model='facebook/opt-2.7b')('Once upon a time', max_new_tokens=40)[0]['generated_text'])" ``` Model repository and documentation: [huggingface.co/facebook](https://huggingface.co/facebook/opt-2.7b) [interactive version](https://whatmodel.sh/model/opt-2-7b) · [JSON](https://whatmodel.sh/api?format=json&all=1)