[whatmodel.sh](../../README.md) / GPT-3 curie (API base model) # GPT-3 curie (API base model) OpenAI · released 2020-06 · proprietary **score 24.7** ±1.0 · comparison only: no current run option verified · from 31 benchmark results across 7 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks GPT-3 curie (API base model) among every model on that task or test, worst to best. 
task  vs the others          rank  counts  tests
math  ▁●▃▆██▅█▃▁▁▁   #506 of 530*    6.9%      1
math · #506 of 530 · 6.9% of its score Beats [Llama 2 13B Chat](llama-2-13b-chat.md), [Flan-T5 Large](flan-t5-large.md), [OPT-66B](opt-66b.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall. 
test   result  vs the others in its run  difficulty        counts  why
GSM8K    2.5%  ▅·▂●·······▂ beats 8/9    medium · avg 62%    6.9%  sharp · too hard for it · noisy
other tests (25) 
test                                result  vs the others in its run  difficulty           counts  why
CommonsenseQA                        24.7%  █····●·····▁ beats 8/9    easy · avg 75%          13%  sharp · too hard for it · several runs
WinoGrande                           65.6%  ▄·▄·▄▄▄··●·▄ beats 6/7    easy · avg 87%          12%  at its level · several runs
ARC Challenge                        33.4%  ▄····▄···●▄▄ beats 2/5    easy · avg 78%          10%  sharp
HellaSwag                            68.5%  ▄·█·█··▄●··▄ beats 6/7    easy · avg 83%          10%  at its level · several runs
SVAMP                                11.3%  █·▆▂▄▂●····▂ beats 11/12  easy · avg 70%         6.8%  sharp
LAMBADA (OpenAI version), last-wo…   69.3%  █·▄··█··●··▄ beats 5/7    very easy · avg 88%    4.4%  blunt · at its level · several runs
LogiQA (harness)                     21.7%  ······●····· beats 1/5    very hard · avg 25%    4.3%  too hard for it
TriviaQA                             19.6%  █···█·█●█··█ beats 3/5    easy · avg 68%         4.2%  typical
DMath (English) few-shot CoT accu…    8.2%  ●·▄······▄·▄ beats 0/4    medium · avg 64%       3.7%  typical
OpenBookQA                           29.0%  ▅·······●··▃ beats 3/5    hard · avg 49%         3.3%  too hard for it
PIQA                                 77.9%  █··▃··▃▃·●·▃ beats 6/7    easy · avg 88%         3.2%  blunt · at its level · several runs
ARC Easy                             68.2%  ▄···▄··●·▄·▄ beats 3/5    easy · avg 87%         2.6%  blunt · at its level
SciQ                                 91.8%  ██····█●█··█ beats 3/5    very easy · avg 98%    2.3%  blunt
ANLI Round 1 (prompted)              32.5%  ●··▂·······▂ beats 0/5    hard · avg 46%         2.1%  sharp · too hard for it · noisy
MMLU                                 27.1%  ▁▁●▁▁▂▂▅██▆▄ beats 9/214  medium · avg 73%       2.0%  typical
ANLI Round 3 (prompted)              35.3%  ▃······●█··▃ beats 1/5    hard · avg 43%         1.5%  too hard for it
QA4MRE 2013 (harness)                37.7%  ▄····▄█●···▄ beats 4/5    medium · avg 57%       1.3%  blunt
ANLI Round 2 (prompted)              33.8%  █●·███·····█ beats 1/5    very hard · avg 43%    1.3%  too hard for it · noisy
PROST (harness, micro accuracy)      28.8%  ▃······█···● beats 4/5    very hard · avg 35%    1.1%  blunt · too hard for it
StrategyQA                           54.7%  █········▁▃● beats 9/9    hard · avg 62%         1.1%  too hard for it · noisy
HEAD-QA (English, harness)           31.7%  █··█····●··█ beats 2/3    medium · avg 58%       1.0%  blunt · at its level
SuperGLUE WSC (prompted)             54.8%  █··········● beats 5/5    hard · avg 62%         0.8%  blunt
LMentry accuracy (zero-shot)         28.0%  ▂●·▄▄█▂·▂··▂ beats 3/14   medium · avg 65%       0.6%  blunt · at its level
AQuA-RAT                             15.8%  ●·····▁·▁··▁ beats 4/9    medium · avg 53%       0.1%  sharp · too hard for it · noisy
Story Cloze Test                     73.4%  ▃▃▃▂●█▅▂▁··▂ beats 19/50  easy · avg 89%            —  not counted yet: no other model in the …
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
GPT-NeoX-20B: An Open-Source Autoregres…       17   50% ▇▇▇▇▇▇
Large Language Models Are Reasoning Tea…        5   26% ▇▇▇
EleutherAI GPT-J-6B model card                  4   13% ▇▇
It Ain't Over: A Multi-aspect Diverse M…        1    4% ▇
Increasing Probability Mass on Answer C…        2    4% ▇
WinoDict: Probing language models for i…        1    2% ▇
LMentry: A Language Model Benchmark of …        1    1% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 24.7
Licences: GPT-NeoX-20B: An Open-Source Autoregressive Language Model ([CC BY 4.0](../../licenses/paper-2204.06745v1-CC-BY-4.0.txt)) · Large Language Models Are Reasoning Teachers ([CC BY 4.0](../../licenses/acl-2023-acl-long-830-CC-BY-4-0.txt)) · EleutherAI GPT-J-6B model card ([Apache-2.0](../../licenses/eleutherai-gpt-j-6b-card-Apache-2.0.txt)) · It Ain't Over: A Multi-aspect Diverse Math Word Problem Dataset ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-927-CC-BY-4-0.txt)) · Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-522-CC-BY-4-0.txt)) · WinoDict: Probing language models for in-context word acquisition ([CC BY 4.0](../../licenses/acl-2023-eacl-main-7-CC-BY-4-0.txt)) · LMentry: A Language Model Benchmark of Elementary Language Tasks ([CC BY 4.0](../../licenses/acl-2023-findings-acl-666-CC-BY-4-0.txt))
Not counted (1) 
source                  test              score  why
Few-shot Learning wit…  Story Cloze Test  73.4%  too few models have it yet
## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/gpt-3-curie) · [JSON](https://whatmodel.sh/api?format=json&all=1)