[whatmodel.sh](../../README.md) / GPT-Neo 2.7B
# GPT-Neo 2.7B
EleutherAI · released 2021-03 · 2.7B params · mit
**score 17.9** ±0.9 · #705 of 732 · from 47 benchmark results across 14 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks GPT-Neo 2.7B among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁●▁▃▅▇█▇▅▄▃▁ #463 of 469* 20% 2
math ▁▁●▅█▆▄█▄▂▁▁ #401 of 431 4.0% 2
coding · #463 of 469 · 20% of its score
Beats [Molmo 7B-D (0924)](molmo-7b-d.md), [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [ChatGLM-6B](chatglm-6b.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
HumanEval 7.9% ▁·●▁▃▄▆▅█▃▁▁ beats 2/99 easy · avg 76% 11% sharp · too hard for it
HumanEval+ 6.7% ▁·●▁▁▃▄▄▄█▃▁ beats 2/99 easy · avg 70% 8.1% sharp · too hard for it
math · #401 of 431 · 4.0% of its score
Beats [Command R (03-2024)](command-r-03-2024.md), [Mistral 7B Instruct v0.3](mistral-7b-instruct-v0-3.md), [Llama-3-Instruct-8B-SimPO-v0.2](llama-3-instruct-8b-simpo-v0-2.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 1.5% ▂●▂▂▃▄▅▇██▆▄ beats 11/206 easy · avg 68% 2.2% sharp · too hard for it
MathQA 24.1% ▃·▃▃█●··▃·▃▃ beats 7/10 hard · avg 33% 1.8% blunt · too hard for it · several runs
other tests (21)
test result vs the others in its run difficulty counts why
HellaSwag 42.7% ▃▃●··▅····▃█ beats 2/8 easy · avg 82% 15% at its level · several runs
ARC Challenge 27.4% ▅····●▃···▅█ beats 2/8 medium · avg 60% 10% sharp · too hard for it · noisy · sever…
WinoGrande 57.7% ●··█···█▄▄·▄ beats 0/8 easy · avg 86% 9.0% small run · several runs
ARC Easy 61.1% ▅·●▃··▃··▃·█ beats 2/8 easy · avg 87% 5.7% blunt · at its level · several runs
COPA 79.0% ▁·▁▁▁▅▅●▆▂▁▁ beats 28/67 very easy · avg 96% 4.7% at its level · several runs
ImageNetVC visual commonsense, te… 41.6% ▂·▄·▂●▄▄█▄▂▆ beats 6/21 easy · avg 74% 4.6% blunt · at its level
PIQA 72.9% ▄▄··●·▄·▄·██ beats 2/8 easy · avg 88% 4.4% blunt · at its level · several runs
BoolQ 61.8% ▄●▄···▄··█▄▄ beats 2/8 easy · avg 88% 4.3% at its level · several runs
LAMBADA (OpenAI version), last-wo… 62.2% █·▄··●··█··▄ beats 3/7 very easy · avg 88% 4.3% blunt · at its level · several runs
Natural Questions (closed book) 5.7% ▁·▁·▁▂●▂·▅▂█ beats 6/28 medium · avg 48% 3.4% too hard for it · several runs
CoNLL-2003 English NER, few-shot … 40.2% ▃···●·▃▅··▃█ beats 2/9 very easy · avg 85% 2.7% at its level
OpenBookQA 23.2% ●······▂·▆·▂ beats 1/8 easy · avg 77% 2.4% several runs
12-dataset text classification, 4… 51.5% ▂▂··●·█···▂▂ beats 2/8 easy · avg 79% 1.5% blunt · at its level
Social IQA 40.0% ●▄····█▄▄··▄ beats 1/7 medium · avg 60% 1.2% blunt
UD English-GUM UPOS tagging, few-… 70.4% ▄·▄●·······█ beats 2/4 very easy · avg 92% 0.8% blunt
PubMedQA 58.3% ▂·····▅●▅▆▆█ beats 7/24 medium · avg 74% 0.6% blunt · at its level · several runs
TriviaQA 20.1% ▁··▁█▃●▁▄▅▄█ beats 11/28 easy · avg 73% 0.6% noisy
Open PT-LLM Leaderboard average (… 28.0% ●▁▁▁▁▁▁▁▂█▅▁ beats 20/276 medium · avg 72% 0.6% noisy
TruthfulQA MC2 39.9% ▅▃●▆▁▂▅···▁▁ beats 9/31 medium · avg 57% 0.2% blunt · at its level
HaluEval QA hallucination detecti… 43.2% ●▁▁·▁▁·▁▁▁▁▁ beats 13/27 hard · avg 66% 0.1% too hard for it · noisy
TruthfulQA generation, % true (hu… 40.0% ██····█·█●·█ beats 4/5 medium · avg 42% 0.0% blunt · at its level
source results share of the score
EvalPlus / EvalPerf published evaluatio… 2 20% ▇▇
UPRISE: Universal Prompt Retrieval for … 9 17% ▇▇
Textbooks Are All You Need II: phi-1.5 … 8 16% ▇▇
Preserving Privacy Through Dememorizati… 7 14% ▇▇
EleutherAI GPT-J-6B model card (submitt… 4 10% ▇
RedPajama 4 6% ▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 5% ▇
Hallucinations Leaderboard results 4 4% ▇
Prompting Language Models for Linguisti… 2 3% ▇
Coverage-based Example Selection for In… 1 2% ▇
Generative Calibration for In-context L… 1 1% ▇
EleutherAI GPT-Neo 2.7B model card (sub… 2 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
TruthfulQA: Measuring How Models Mimic … 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 17.9
Licences: EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-758-CC-BY-4-0.txt)) · Textbooks Are All You Need II: phi-1.5 technical report ([CC BY 4.0](../../licenses/paper-2309.05463v1-CC-BY-4.0.txt)) · Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-265-CC-BY-4-0.txt)) · EleutherAI GPT-J-6B model card ([Apache-2.0](../../licenses/eleutherai-gpt-j-6b-card-Apache-2.0.txt)) · RedPajama ([CC BY 4.0](../../licenses/redpajama-2024-original-zero-shot-CC-BY-4.0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Hallucinations Leaderboard results ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · Prompting Language Models for Linguistic Structure ([CC BY 4.0](../../licenses/acl-2023-acl-long-367-CC-BY-4-0.txt)) · Coverage-based Example Selection for In-Context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-930-CC-BY-4-0.txt)) · Generative Calibration for In-context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-152-CC-BY-4-0.txt)) · EleutherAI GPT-Neo 2.7B model card ([MIT](../../licenses/eleutherai-gpt-neo-2-7b-card-MIT.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · TruthfulQA: Measuring How Models Mimic Human Falsehoods ([CC BY 4.0](../../licenses/acl-2022-acl-long-229-CC-BY-4-0.txt))
## How to run it
### Other ways to run it
1. Run it as published (fp32, needs ~10.67 GB, from [huggingface.co/EleutherAI](https://huggingface.co/EleutherAI/gpt-neo-2.7B/tree/e24fa291132763e59f4a5422741b424fb5d59056))
```sh
uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text-generation', model='EleutherAI/gpt-neo-2.7B')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/EleutherAI](https://huggingface.co/EleutherAI/gpt-neo-2.7B)
[interactive version](https://whatmodel.sh/model/gpt-neo-2-7b) · [JSON](https://whatmodel.sh/api?format=json&all=1)