[whatmodel.sh](../../README.md) / Codex code-davinci-001 # Codex code-davinci-001 OpenAI · released 2021-08 · proprietary **score 32.6** ±1.4 · comparison only: no current run option verified · from 9 benchmark results across 4 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at _{● marks Codex code-davinci-001 among every model on that task or test, worst to best.}

task  vs the others          rank  counts  tests
math  ▁▁▃●██▅█▃▁▁▁   #457 of 530*     28%      1

math · #457 of 530 · 28% of its score

Beats [Command R Plus (04-2024)](command-r-plus-04-2024.md), [Phi-3-mini-4k-instruct (June 2024 update)](phi-3-mini-4k-instruct-june-2024.md), [Yi 34B Chat](yi-34b-chat.md) here, though they rank higher overall. Trails [GPT-Neo 1.3B](gpt-neo-1-3b.md), [OPT-1.3B](opt-1-3b.md), [OPT-2.7B](opt-2-7b.md) here, though they rank lower overall.

test   result  vs the others in its run   difficulty      counts  why
GSM8K   14.7%  ▂▂▂▂●▄▅▇██▆▄ beats 41/206  easy · avg 69%     28%  sharp · several runs

other tests (5)

test                                result  vs the others in its run  difficulty        counts  why
SVAMP                                41.2%  █▆▆▅▂▅▂●▂▂▅▃ beats 23/30  easy · avg 84%       34%  sharp · at its level · several runs
AQuA-RAT                             22.4%  █··●▁▁··▁▁▁▁ beats 18/27  medium · avg 56%     16%  sharp · too hard for it · several runs
CommonsenseQA                        46.8%  █·····▂▂●▂▄▄ beats 21/38  easy · avg 87%       14%  sharp · at its level · small run
CoNLL-2003 English NER, few-shot …   61.9%  ▃···▅·▃●··▃█ beats 5/9    easy · avg 84%      5.3%  at its level · small run
ODEX English pass@1 (zero-shot)      33.6%  ▄●········▄█ beats 1/4    medium · avg 58%    3.2%  blunt · at its level

## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md)

source                                    results  share of the score
Faithful Chain-of-Thought Reasoning             3   47% ▇▇▇▇▇▇
Large Language Models are Better Reason…        4   43% ▇▇▇▇▇
CodeIE: Large Code Generation Models ar…        1    5% ▇
Execution-Based Evaluation for Open-Dom…        1    3% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 32.6

_{Licences: Faithful Chain-of-Thought Reasoning ([CC BY 4.0](../../licenses/acl-2023-ijcnlp-main-20-CC-BY-4-0.txt)) · Large Language Models are Better Reasoners with Self-Verification ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-167-CC-BY-4-0.txt)) · CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors ([CC BY 4.0](../../licenses/acl-2023-acl-long-855-CC-BY-4-0.txt)) · Execution-Based Evaluation for Open-Domain Code Generation ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-89-CC-BY-4-0.txt))} ## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/code-davinci-001) · [JSON](https://whatmodel.sh/api?format=json&all=1)