[whatmodel.sh](../../README.md) / GPT-J 6B # GPT-J 6B EleutherAI · released 2021-06 · 6.05B params · apache-2.0 **score 24.6** ±0.7 · #689 of 732 · from 44 benchmark results across 12 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks GPT-J 6B among every model on that task or test, worst to best. 
task    vs the others            rank  counts  tests
coding  ▁▁●▃▅▇█▇▅▄▃▁     #458 of 469*     25%      3
math                   not ranked yet       —      1
coding · #458 of 469 · 25% of its score Beats [Molmo 7B-D (0924)](molmo-7b-d.md), [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Qwen3 0.6B](qwen3-0-6b.md) here, though they rank higher overall. Trails [Phi-1 (1.3B)](phi-1.md) here, though they rank lower overall. 
test        result  vs the others in its run  difficulty      counts  why
HumanEval    12.2%  ▁·●▁▃▄▆▅█▃▁▁ beats 3/99   easy · avg 76%     15%  sharp
HumanEval+   11.0%  ▁·▁●▁▃▄▄▄█▃▁ beats 3/99   easy · avg 70%     10%  sharp
MBPP         13.3%  ●▁▁▄▅█▆▅▄▁▂▁ beats 0/86   easy · avg 67%       —  not counted yet: no other model in the …
math · 0.0% of its score 
test   result  vs the others in its run   difficulty      counts  why
GSM8K    2.7%  ▂▂●▂▃▄▅▇██▆▄ beats 21/206  easy · avg 72%       —  not counted yet: no other model in the …
other tests (32) 
test                                result  vs the others in its run   difficulty           counts  why
ARC Challenge                        36.5%  ▁·▁··▁▁▇●█▆▃ beats 11/26   easy · avg 82%          19%  sharp · several runs
WinoGrande                           64.2%  ▁····▁··▁▂●▄ beats 9/26    easy · avg 86%         8.8%  at its level · several runs
HellaSwag                            66.2%  ▃···▁▇●█▇▃▁▁ beats 8/26    very easy · avg 91%    7.6%  at its level · several runs
ARC Easy                             62.2%  ▁▁·▁▁▄●▄█▅▁▃ beats 8/26    easy · avg 85%         3.8%  blunt · at its level · several runs
SciQ                                 91.5%  ▁·▁·▁·▃●▃█▄▁ beats 10/26   very easy · avg 97%    3.6%  blunt · several runs
PIQA                                 75.6%  ▁·▁··▁▁▃●▅█▂ beats 8/26    easy · avg 87%         3.4%  blunt · at its level · several runs
BoolQ                                65.4%  ▂▃▂▅●▂▅▆█▂▂▂ beats 8/26    easy · avg 88%         3.3%  at its level
LAMBADA (OpenAI version), last-wo…   68.2%  ▁··▂▁▁▁▂●█▅▃ beats 11/26   very easy · avg 86%    3.3%  blunt · at its level · several runs
ImageNetVC visual commonsense, te…   49.8%  ▂·▄·▂█▄▄●▄▂▆ beats 12/21   easy · avg 74%         3.2%  blunt · at its level
OpenBookQA                           29.0%  █·▁▁▃·▁●▅▇▃▄ beats 12/26   hard · avg 49%         2.7%  too hard for it · several runs
TriviaQA                             39.3%  ▁··▁█▃▃▁●▅▄█ beats 15/28   easy · avg 68%         2.5%  several runs
Natural Questions (closed book)       9.2%  ▁·▁·▁▂▁●·▅▂█ beats 8/28    medium · avg 48%       2.5%  typical
LogiQA (harness) (maker)             20.9%  ······●····· beats 5/5     very hard · avg 25%    2.1%  too hard for it
CoNLL-2003 English NER, few-shot …   53.0%  ▃···▅·●▅··▃█ beats 3/9     very easy · avg 85%    1.7%  at its level
ANLI Round 1 (prompted) (maker)      32.4%  ●··▂·······▂ beats 3/5     hard · avg 46%         1.0%  sharp · too hard for it · noisy
12-dataset text classification, 4…   57.7%  ▂▂··▂·●···▂▂ beats 6/8     easy · avg 79%         0.9%  blunt · at its level
Social IQA                           41.1%  ▁···▁●█▁▁▁·▁ beats 6/26    hard · avg 56%         0.9%  blunt
ANLI Round 3 (prompted) (maker)      35.5%  ▃······▃●··▃ beats 4/5     hard · avg 43%         0.7%  too hard for it
QA4MRE 2013 (harness) (maker)        37.3%  ▄····▄●▄···▄ beats 3/5     medium · avg 57%       0.6%  blunt
ANLI Round 2 (prompted) (maker)      34.0%  ██·●██·····█ beats 2/5     very hard · avg 43%    0.6%  too hard for it · noisy
PROST (harness, micro accuracy) (…   26.7%  ▃······●···▅ beats 2/5     very hard · avg 35%    0.6%  blunt · too hard for it
Open PT-LLM Leaderboard average (…   35.7%  ▃▁▁●▁▁▁▁▂█▅▁ beats 43/276  medium · avg 72%       0.5%  noisy
UD English-GUM UPOS tagging, few-…   83.1%  ▄·▄▄·······● beats 3/4     very easy · avg 92%    0.4%  blunt
SuperGLUE WSC (prompted) (maker)     36.5%  ●··········▄ beats 3/5     hard · avg 62%         0.4%  blunt
MenatQA (all factors), exact matc…   25.9%  ▂····▂··▂·█● beats 7/8     medium · avg 36%       0.3%  blunt
TruthfulQA MC2                       36.0%  ●▃█▆▁▂▅···▁▁ beats 3/31    medium · avg 57%       0.1%  blunt · at its level
HaluEval QA hallucination detecti…   51.4%  █▁▁·●▁·▁▁▁▁▁ beats 18/27   hard · avg 66%         0.1%  too hard for it · noisy
MC-TACO as yes/no plausibility cl…   59.0%  ●··········▅ beats 2/4     easy · avg 75%         0.1%  blunt · at its level
TruthfulQA generation, % true (hu…   26.8%  ●█····█·██·█ beats 0/5     medium · avg 42%       0.0%  blunt · at its level
CommonsenseQA                        34.5%  █·····●▂▃▂▄▄ beats 15/38   easy · avg 88%            —  not counted yet: no other model in the …
SVAMP                                20.7%  █▆▆▅▂▅●▅▂▂▅▃ beats 20/30   easy · avg 82%            —  not counted yet: no other model in the …
StrategyQA                           47.2%  ●····▂▂▁▂▂▂▂ beats 0/25    medium · avg 70%          —  not counted yet: no other model in the …
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Stability AI                                   10   34% ▇▇▇▇
EvalPlus / EvalPerf published evaluatio…        2   25% ▇▇▇
GPT-NeoX-20B: An Open-Source Autoregres…       16   24% ▇▇▇
EleutherAI GPT-J-6B model card (submitt…        4    6% ▇
Hallucinations Leaderboard results              4    3% ▇
ImageNetVC: Zero- and Few-Shot Visual C…        1    3% ▇
Prompting Language Models for Linguisti…        2    2% ▇
Generative Calibration for In-context L…        1    1% ▇
Open Portuguese LLM Leaderboard                 1    1% ▇
MenatQA: A New Dataset for Testing the …        1    0% ▇
Do Language Models Have a Common Sense …        1    0% ▇
TruthfulQA: Measuring How Models Mimic …        1    0% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 24.6
Licences: Stability AI ([Apache-2.0](../../licenses/stablelm-original-eval-json-Apache-2.0.txt)) · EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · GPT-NeoX-20B: An Open-Source Autoregressive Language Model ([CC BY 4.0](../../licenses/paper-2204.06745v1-CC-BY-4.0.txt)) · EleutherAI GPT-J-6B model card ([Apache-2.0](../../licenses/eleutherai-gpt-j-6b-card-Apache-2.0.txt)) · Hallucinations Leaderboard results ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Prompting Language Models for Linguistic Structure ([CC BY 4.0](../../licenses/acl-2023-acl-long-367-CC-BY-4-0.txt)) · Generative Calibration for In-context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-152-CC-BY-4-0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-100-CC-BY-4-0.txt)) · Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-418-CC-BY-4-0.txt)) · TruthfulQA: Measuring How Models Mimic Human Falsehoods ([CC BY 4.0](../../licenses/acl-2022-acl-long-229-CC-BY-4-0.txt))
Not counted (5) 
source                  test           score  why
ReCode: Robustness Ev…  MBPP           13.3%  too few models have it yet
Democratizing Reasoni…  GSM8K           2.7%  too few models have it yet
Democratizing Reasoni…  SVAMP          20.7%  too few models have it yet
Democratizing Reasoni…  CommonsenseQA  34.5%  too few models have it yet
Democratizing Reasoni…  StrategyQA     47.2%  too few models have it yet
## How to run it ### Other ways to run it 1. Run it as published (fp16, needs ~12.11 GB, from [huggingface.co/EleutherAI](https://huggingface.co/EleutherAI/gpt-j-6b/tree/float16)) ```sh uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='EleutherAI/gpt-j-6b', torch_dtype='auto', device_map='auto', revision='float16')('Once upon a time', max_new_tokens=40)[0]['generated_text'])" ``` Model repository and documentation: [huggingface.co/EleutherAI](https://huggingface.co/EleutherAI/gpt-j-6b) [interactive version](https://whatmodel.sh/model/gpt-j-6b) · [JSON](https://whatmodel.sh/api?format=json&all=1)