[whatmodel.sh](../../README.md) / GPT-3 babbage (API base model) # GPT-3 babbage (API base model) OpenAI · released 2020-06 · proprietary **score 15.4** ±1.2 · comparison only: no current run option verified · from 29 benchmark results across 6 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks GPT-3 babbage (API base model) among every model on that task or test, worst to best. 
task  vs the others          rank  counts  tests
math  ▁●▃▆██▅█▃▁▁▁   #514 of 530*    3.8%      1
math · #514 of 530 · 3.8% of its score Beats [Flan-T5 Large](flan-t5-large.md), [OPT-66B](opt-66b.md), [Flan-T5 Base](flan-t5-base.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall. 
test   result  vs the others in its run  difficulty        counts  why
GSM8K    1.6%  ▅·●█·······▂ beats 3/9    medium · avg 62%    3.8%  sharp · too hard for it · noisy
other tests (23) 
test                                result  vs the others in its run  difficulty           counts  why
HellaSwag                            54.5%  ▄·█·●··▄▄··▄ beats 3/7    very easy · avg 91%     14%  at its level · several runs
WinoGrande                           59.0%  ▄·▄·▄●▄··█·▄ beats 3/7    easy · avg 87%          14%  several runs
LAMBADA (OpenAI version), last-wo…   62.4%  █·▄··●··█··▄ beats 4/7    very easy · avg 88%    8.0%  blunt · at its level · several runs
ARC Challenge                        27.5%  ▄····●···█▄▄ beats 1/5    easy · avg 78%         7.3%  sharp · too hard for it
LogiQA (harness)                     19.8%  ······●····· beats 2/5    very hard · avg 25%    7.0%  too hard for it
CommonsenseQA                        18.7%  ●····▁·····▁ beats 6/9    easy · avg 75%         5.5%  sharp · too hard for it
PIQA                                 75.5%  █··▃··●▃·▃·▃ beats 4/7    easy · avg 88%         5.2%  blunt · at its level · several runs
SciQ                                 86.6%  █●····███··█ beats 1/5    very easy · avg 98%    5.0%  blunt
TriviaQA                             11.5%  █···●·███··█ beats 1/5    easy · avg 68%         4.6%  typical
ARC Easy                             59.8%  ▄···●··█·▄·▄ beats 1/5    easy · avg 87%         4.2%  blunt · at its level
SVAMP                                 4.3%  █·▆▂●▂▂····▂ beats 9/12   easy · avg 70%         4.1%  sharp · too hard for it
DMath (English) few-shot CoT accu…    8.9%  ●·▄······▄·▄ beats 1/4    medium · avg 64%       3.1%  too hard for it
OpenBookQA                           22.4%  ●·······█··▃ beats 0/5    hard · avg 49%         2.7%  too hard for it
QA4MRE 2013 (harness)                37.0%  ▄····▄●▄···▄ beats 2/5    medium · avg 57%       1.8%  blunt
ANLI Round 3 (prompted)              34.0%  ●······▃█··▃ beats 0/5    hard · avg 43%         1.5%  too hard for it
HEAD-QA (English, harness)           27.8%  █··●····█··█ beats 1/3    medium · avg 58%       1.4%  blunt · at its level
PROST (harness, micro accuracy)      27.0%  ▃······●···▅ beats 3/5    very hard · avg 35%    1.2%  blunt · too hard for it
ANLI Round 1 (prompted)              32.6%  ●··▂·······▂ beats 1/5    hard · avg 46%         1.2%  sharp · too hard for it · noisy
StrategyQA                           46.1%  ●········▁▃▁ beats 4/9    hard · avg 62%         1.1%  too hard for it · noisy
ANLI Round 2 (prompted)              30.8%  ●█·███·····█ beats 0/5    very hard · avg 43%    1.1%  too hard for it · noisy
LMentry accuracy (zero-shot)         25.7%  ▂●·▄▄█▂·▂··▂ beats 2/14   medium · avg 65%       0.9%  blunt · at its level
SuperGLUE WSC (prompted)             40.4%  ●··········▄ beats 0/5    hard · avg 62%         0.9%  blunt · too hard for it
AQuA-RAT                             13.8%  ●·····▁·▁··▁ beats 5/9    medium · avg 53%       0.0%  sharp · too hard for it · noisy
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
GPT-NeoX-20B: An Open-Source Autoregres…       17   59% ▇▇▇▇▇▇▇
EleutherAI GPT-J-6B model card                  4   19% ▇▇
Large Language Models Are Reasoning Tea…        5   15% ▇▇
It Ain't Over: A Multi-aspect Diverse M…        1    3% ▇
WinoDict: Probing language models for i…        1    3% ▇
LMentry: A Language Model Benchmark of …        1    1% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 15.4
Licences: GPT-NeoX-20B: An Open-Source Autoregressive Language Model ([CC BY 4.0](../../licenses/paper-2204.06745v1-CC-BY-4.0.txt)) · EleutherAI GPT-J-6B model card ([Apache-2.0](../../licenses/eleutherai-gpt-j-6b-card-Apache-2.0.txt)) · Large Language Models Are Reasoning Teachers ([CC BY 4.0](../../licenses/acl-2023-acl-long-830-CC-BY-4-0.txt)) · It Ain't Over: A Multi-aspect Diverse Math Word Problem Dataset ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-927-CC-BY-4-0.txt)) · WinoDict: Probing language models for in-context word acquisition ([CC BY 4.0](../../licenses/acl-2023-eacl-main-7-CC-BY-4-0.txt)) · LMentry: A Language Model Benchmark of Elementary Language Tasks ([CC BY 4.0](../../licenses/acl-2023-findings-acl-666-CC-BY-4-0.txt)) ## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/gpt-3-babbage) · [JSON](https://whatmodel.sh/api?format=json&all=1)