[whatmodel.sh](../../README.md) / Codex code-davinci-002 (GPT-3.5 base)
# Codex code-davinci-002 (GPT-3.5 base)
OpenAI · released 2022-03 · proprietary
**score 47.2** ±0.4 · comparison only: no current run option verified · from 64 benchmark results across 26 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Codex code-davinci-002 (GPT-3.5 base) among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁▁▃●▆█▇▅▄▃▁ #452 of 539* 1.0% 1
math ▁▁▃▆█●▅█▃▁▁▁ #269 of 530 27% 2
coding · #452 of 539 · 1.0% of its score
Beats [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md), [DeepSeek R1 Distill Qwen 7B](deepseek-r1-distill-qwen-7b.md), [Tulu 2 70B](tulu-2-70b.md) here, though they rank higher overall. Trails [Qwen2.5 1.5B Instruct](qwen2-5-1-5b.md), [WizardLM 13B V1.2](wizardlm-13b-v1-2.md), [Falcon3 3B Instruct](falcon3-3b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 45.9% ▁▁▁▁▂▂▆●█▅▂▁ beats 83/205 easy · avg 69% 1.0% sharp · at its level · small run
math · #269 of 530 · 27% of its score
Beats [MiniMax-M2.1](minimax-m2-1.md), [GPT-4o (2024-08-06)](gpt-4o-2024-08-06.md), [Command A (03-2025)](command-a.md) here, though they rank higher overall. Trails [Gemma 7B (base)](gemma-7b-base.md), [Phi-2 (2.7B)](phi-2.md), [Granite 8B Code Base (4K)](granite-8b-code-base-4k.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 63.1% █·▅▃▃▃·▃·▃▃● beats 12/12 easy · avg 67% 25% sharp · at its level · several runs
MGSM 35.0% ▆·▂·▄▂▂▂▂●▄▆ beats 12/18 medium · avg 48% 2.7% sharp · at its level
other tests (26)
test result vs the others in its run difficulty counts why
AQuA-RAT 45.3% █▁·▁▁·····▂● beats 12/12 medium · avg 48% 17% sharp · at its level · several runs
SVAMP 76.4% ▃▅█···▅·▃▃▃● beats 11/12 easy · avg 82% 14% sharp · several runs
CommonsenseQA 82.3% █······▁·▁▁● beats 12/12 easy · avg 88% 8.3% sharp · several runs
ASDiv 80.4% ▅▃▃▅··▃▃·▃▃● beats 12/12 easy · avg 83% 5.5% sharp · several runs
MAWPS (CoT-paper union) 92.6% ▅▃█···▅·▃▃▃● beats 11/12 very easy · avg 94% 4.2% sharp
BIG-Bench Hard 52.8% ▂▂▂▂·▅·▃▂█●▃ beats 16/18 medium · avg 60% 3.9% sharp · at its level · several runs
MMLU 68.2% ▁···▁··▃▁▅▅● beats 14/18 easy · avg 74% 2.8% at its level
BIG-bench Sports Understanding 71.7% █·····▃·▁·●▁ beats 10/12 medium · avg 81% 2.5% sharp · at its level
BIG-bench Date Understanding 49.0% ▃·▅·▅▃▅·▃▃·● beats 12/12 medium · avg 52% 2.2% at its level · several runs
StrategyQA 67.1% █·····▅▃▃▅▃● beats 11/12 medium · avg 70% 2.0% at its level · several runs
ARC Challenge 83.6% ▄▁▁▂▁▄█▇▃▂●▂ beats 128/140 easy · avg 86% 1.5% sharp
MultiArith 96.2% █·········█● beats 2/2 very easy · avg 95% 1.1% sharp · maxed out · small run · several…
TyDiQA-GoldP (1-shot, EM, 8-langu… 48.1% ▂··▂·▂▄▄·█●█ beats 12/18 medium · avg 61% 1.0% at its level
CoNLL-2003 English NER, few-shot … 82.3% ▃···▅·▃▅··▃● beats 8/9 easy · avg 83% 0.8% small run · several runs
Natural Questions (closed book) 31.4% ▂▁▁▁▂▄▄▃▅█●▁ beats 63/68 medium · avg 39% 0.7% at its level · small run
TriviaQA 81.8% ▁▁▂▂▄▄▃▆▅█▅● beats 77/80 easy · avg 82% 0.5% small run
AddSub 89.4% █·········●█ beats 1/2 very easy · avg 89% 0.5% small run
ARCADE New Tasks pass@1 (notebook… 23.4% █·█········● beats 2/2 hard · avg 23% 0.5% small run
OpenBookQA 71.8% ▅▁▁▁▂▂▃█▃●▃▁ beats 93/107 medium · avg 61% 0.4% at its level · noisy · small run
ACE05-E event argument extraction… 58.5% █·····●····█ beats 1/2 medium · avg 65% 0.4% at its level · small run
WinoGrande 60.7% ▁▁▁▁▂▁▁●▇█▃▁ beats 36/131 medium · avg 73% 0.4% at its level
CSK-PN boolean commonsense QA (10… 84.9% ▃·▃·····▃▃▃● beats 7/7 easy · avg 87% 0.3% blunt · at its level
ODEX English pass@1 (zero-shot) 47.1% ▄▄········▄● beats 4/4 medium · avg 58% 0.3% blunt · at its level
HellaSwag 65.2% ▁▁▁▂▂▂▄●█▇▄▂ beats 53/135 easy · avg 77% 0.3% at its level
Zero-shot relation extraction, va… 39.8% ●····▄····█▄ beats 0/5 medium · avg 50% 0.2% blunt · at its level
ARC Easy 94.0% ▁▃▂▄▆█▅▃▁··● beats 99/100 easy · avg 87% 0.1% blunt
source results share of the score
Chain-of-Thought Prompting Elicits Reas… 9 37% ▇▇▇▇
Self-Consistency Improves Chain of Thou… 10 11% ▇
Scaling Instruction-Finetuned Language … 4 9% ▇
Faithful Chain-of-Thought Reasoning 3 8% ▇
Large Language Models are Better Reason… 4 5% ▇
Batch Prompting: Efficient Inference wi… 3 4% ▇
Coverage-based Example Selection for In… 1 3% ▇
Making Language Models Better Reasoners… 2 3% ▇
Tab-CoT: Zero-shot Tabular Chain of Tho… 3 3% ▇
Automatic Model Selection with Large La… 2 3% ▇
Complexity-Based Prompting for Multi-St… 4 3% ▇
PAL: Program-aided Language Models 1 2% ▇
Explanation Selection Using Unlabeled D… 2 2% ▇
Challenging BIG-Bench Tasks and Whether… 1 1% ▇
Symbolic Chain-of-Thought Distillation:… 2 1% ▇
Recitation-Augmented Language Models 2 1% ▇
StarCoder: may the source be with you! 1 1% ▇
Improving Mathematics Tutoring With A C… 1 1% ▇
Exploring the Curious Case of Code Prom… 2 1% ▇
Natural Language to Code Generation in … 1 0% ▇
Large Language Model Is Not a Good Few-… 1 0% ▇
Code4Struct: Code Generation for Few-Sh… 1 0% ▇
CodeIE: Large Code Generation Models ar… 1 0% ▇
Say What You Mean! Large Language Model… 1 0% ▇
Execution-Based Evaluation for Open-Dom… 1 0% ▇
Aligning Instruction Tasks Unlocks Larg… 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 47.2
Licences: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models ([CC BY 4.0](../../licenses/paper-2201.11903v6-CC-BY-4-0.txt)) · Self-Consistency Improves Chain of Thought Reasoning in Language Models ([CC BY 4.0](../../licenses/paper-2203.11171v4-CC-BY-4-0.txt)) · Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) · Faithful Chain-of-Thought Reasoning ([CC BY 4.0](../../licenses/acl-2023-ijcnlp-main-20-CC-BY-4-0.txt)) · Large Language Models are Better Reasoners with Self-Verification ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-167-CC-BY-4-0.txt)) · Batch Prompting: Efficient Inference with Large Language Model APIs ([CC BY 4.0](../../licenses/acl-2023-emnlp-industry-74-CC-BY-4-0.txt)) · Coverage-based Example Selection for In-Context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-930-CC-BY-4-0.txt)) · Making Language Models Better Reasoners with Step-Aware Verifier ([CC BY 4.0](../../licenses/acl-2023-acl-long-291-CC-BY-4-0.txt)) · Tab-CoT: Zero-shot Tabular Chain of Thought ([CC BY 4.0](../../licenses/acl-2023-findings-acl-651-CC-BY-4-0.txt)) · Automatic Model Selection with Large Language Models for Reasoning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-55-CC-BY-4-0.txt)) · Complexity-Based Prompting for Multi-Step Reasoning ([CC BY 4.0](../../licenses/paper-2210.00720v2-CC-BY-4-0.txt)) · PAL: Program-aided Language Models ([CC0 1.0](../../licenses/paper-2211.10435v2-pal-CC0-1-0.txt)) · Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-41-CC-BY-4-0.txt)) · Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them ([CC BY 4.0](../../licenses/acl-2023-findings-acl-824-bbh-CC-BY-4-0.txt)) · Symbolic Chain-of-Thought Distillation: Small Models Can Also \`\`Think'' Step-by-Step ([CC BY 4.0](../../licenses/acl-2023-acl-long-150-CC-BY-4-0.txt)) · Recitation-Augmented Language Models ([CC BY 4.0](../../licenses/paper-2210.01296v2-recitation-CC-BY-4-0.txt)) · StarCoder: may the source be with you! ([CC BY-SA 4.0](../../licenses/paper-2305.06161v2-starcoder-CC-BY-SA-4-0.txt)) · Improving Mathematics Tutoring With A Code Scratchpad ([CC BY 4.0](../../licenses/acl-2023-bea-1-2-CC-BY-4-0.txt)) · Exploring the Curious Case of Code Prompts ([CC BY 4.0](../../licenses/acl-2023-nlrse-1-2-CC-BY-4-0.txt)) · Natural Language to Code Generation in Interactive Data Science Notebooks ([CC BY 4.0](../../licenses/acl-2023-acl-long-9-CC-BY-4-0.txt)) · Large Language Model Is Not a Good Few-shot Information Extractor ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-710-CC-BY-4-0.txt)) · Code4Struct: Code Generation for Few-Shot Event Structure Prediction ([CC BY 4.0](../../licenses/acl-2023-acl-long-202-CC-BY-4-0.txt)) · CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors ([CC BY 4.0](../../licenses/acl-2023-acl-long-855-CC-BY-4-0.txt)) · Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge ([CC BY 4.0](../../licenses/acl-2023-acl-long-550-CC-BY-4-0.txt)) · Execution-Based Evaluation for Open-Domain Code Generation ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-89-CC-BY-4-0.txt)) · Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors ([CC BY 4.0](../../licenses/acl-2023-findings-acl-50-CC-BY-4-0.txt))
Not counted (4)
source test score why
Batch Prompting: Effi… StrategyQA 73.3% only model in this source's run
Batch Prompting: Effi… AQuA-RAT 46.1% only model in this source's run
From Complex to Simpl… GSM8K 60.3% only model in this source's run
Faithful Chain-of-Tho… StrategyQA 72.5% only model in this source's run