[whatmodel.sh](../../README.md) / GPT-Neo 1.3B # GPT-Neo 1.3B EleutherAI · released 2021-03 · 1.3B params · mit **score 11.8** ±1.6 · #715 of 732 · from 20 benchmark results across 9 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks GPT-Neo 1.3B among every model on that task or test, worst to best. 
task  vs the others          rank  counts  tests
math  ▁▁▃●█▆▄█▄▂▁▁   #374 of 431*    2.5%      1
math · #374 of 431 · 2.5% of its score Beats [DBRX Instruct](dbrx-instruct.md), [Command R Plus (04-2024)](command-r-plus-04-2024.md), [Mistral NeMo Instruct (2407)](mistral-nemo-instruct-2407.md) here, though they rank higher overall. 
test            result  vs the others in its run  difficulty      counts  why
MathQA (maker)   24.1%  ▃·▃▃●▅··▃·▃▃ beats 5/10   hard · avg 34%    2.5%  blunt · too hard for it
other tests (21) 
test                                result  vs the others in its run   difficulty           counts  why
HellaSwag (maker)                    48.9%  ▄·●·█··▄▄··▄ beats 1/7     very easy · avg 91%     23%  at its level · several runs
ImageNetVC visual commonsense, te…   40.8%  ▂·▄·▂●▄▄█▄▂▆ beats 5/21    easy · avg 74%          14%  blunt
WinoGrande (maker)                   55.0%  ▄·●·▄▄▄··█·▄ beats 1/7     easy · avg 88%          12%  several runs
CoNLL-2003 English NER, few-shot …   39.1%  ▃···●·▃▅··▃█ beats 1/9     very easy · avg 85%    8.3%  at its level
LAMBADA (OpenAI version), last-wo…   57.2%  █·●··█··█··▄ beats 2/7     very easy · avg 88%    7.2%  blunt · at its level
ARC Easy                             56.2%  █·······●▄·█ beats 2/5     easy · avg 87%         7.0%  blunt · at its level
ARC Challenge                        23.1%  ●·········▂▂ beats 2/5     medium · avg 60%       6.4%  sharp · too hard for it · noisy
Natural Questions (closed book)       4.0%  ▁·▁·▁●▁▂·▅▂█ beats 4/28    medium · avg 48%       5.9%  too hard for it
PIQA (maker)                         71.1%  ●··▃··▃▃·▃·▃ beats 2/7     easy · avg 88%         4.5%  blunt · at its level
UD English-GUM UPOS tagging, few-…   68.5%  ▄·●▄·······█ beats 1/4     very easy · avg 92%    2.8%  blunt · at its level
TriviaQA                             14.2%  ▁··▁█●▃▁▄▅▄█ beats 9/28    easy · avg 73%         1.4%  noisy
PubMedQA (maker)                     54.4%  ▂·····●█▅▆▆█ beats 3/24    medium · avg 74%       1.3%  blunt · at its level
Open PT-LLM Leaderboard average (…   26.7%  ●▁▁▁▁▁▁▁▂█▅▁ beats 22/276  medium · avg 72%       1.2%  too hard for it · noisy
TruthfulQA MC2                       39.6%  ▅▃●▆▁▂▅···▁▁ beats 8/31    medium · avg 57%       0.7%  blunt · at its level
MC-TACO as yes/no plausibility cl…   59.0%  ●··········▅ beats 1/4     easy · avg 75%         0.6%  blunt · at its level
HaluEval QA hallucination detecti…   46.3%  ●▁▁·▁▁·▁▁▁▁▁ beats 15/27   hard · avg 66%         0.2%  too hard for it · noisy
TruthfulQA generation, % true (hu…   37.9%  ██····█·●█·█ beats 3/5     medium · avg 42%       0.1%  blunt · at its level
OpenBookQA                           21.6%  ●▁▁▁▂▂▃█▃▁▃▁ beats 2/107   hard · avg 49%            —  not counted yet: no other model in the …
BoolQ                                62.0%  ▁····▁▂●▆▅▆▄ beats 25/103  easy · avg 88%            —  not counted yet: no other model in the …
COPA                                 69.0%  ▁·▁▁▁●▅█▆▂▁▁ beats 14/67   easy · avg 92%            —  not counted yet: no other model in the …
SciQ                                 86.0%  ▂·▁▂▂▃●▃▇▆█▂ beats 16/51   very easy · avg 97%       —  not counted yet: no other model in the …
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Preserving Privacy Through Dememorizati…        4   31% ▇▇▇▇
EleutherAI GPT-J-6B model card (submitt…        4   30% ▇▇▇▇
ImageNetVC: Zero- and Few-Shot Visual C…        1   14% ▇▇
Prompting Language Models for Linguisti…        2   11% ▇
Hallucinations Leaderboard results              4    8% ▇
EleutherAI GPT-Neo 2.7B model card (sub…        2    4% ▇
Open Portuguese LLM Leaderboard                 1    1% ▇
Do Language Models Have a Common Sense …        1    1% ▇
TruthfulQA: Measuring How Models Mimic …        1    0% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 11.8
Licences: Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-265-CC-BY-4-0.txt)) · EleutherAI GPT-J-6B model card ([Apache-2.0](../../licenses/eleutherai-gpt-j-6b-card-Apache-2.0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Prompting Language Models for Linguistic Structure ([CC BY 4.0](../../licenses/acl-2023-acl-long-367-CC-BY-4-0.txt)) · Hallucinations Leaderboard results ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · EleutherAI GPT-Neo 2.7B model card ([MIT](../../licenses/eleutherai-gpt-neo-2-7b-card-MIT.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-418-CC-BY-4-0.txt)) · TruthfulQA: Measuring How Models Mimic Human Falsehoods ([CC BY 4.0](../../licenses/acl-2022-acl-long-229-CC-BY-4-0.txt))
Not counted (9) 
source                  test           score  why
What Language Model t…  ARC Challenge  23.1%  only model in this source's run
What Language Model t…  ARC Easy       56.2%  only model in this source's run
What Language Model t…  BoolQ          62.0%  too few models have it yet
What Language Model t…  COPA           69.0%  too few models have it yet
What Language Model t…  MathQA         24.1%  only model in this source's run
What Language Model t…  OpenBookQA     21.6%  too few models have it yet
What Language Model t…  SciQ           86.0%  too few models have it yet
What Language Model t…  TriviaQA        5.2%  only model in this source's run
What Language Model t…  WinoGrande     55.1%  only model in this source's run
## How to run it ### Other ways to run it 1. Run it as published (fp32, needs ~5.31 GB, from [huggingface.co/EleutherAI](https://huggingface.co/EleutherAI/gpt-neo-1.3B/tree/dbe59a7f4a88d01d1ba9798d78dbe3fe038792c8)) ```sh uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text-generation', model='EleutherAI/gpt-neo-1.3B')('Once upon a time', max_new_tokens=40)[0]['generated_text'])" ``` Model repository and documentation: [huggingface.co/EleutherAI](https://huggingface.co/EleutherAI/gpt-neo-1.3B) [interactive version](https://whatmodel.sh/model/gpt-neo-1-3b) · [JSON](https://whatmodel.sh/api?format=json&all=1)