[whatmodel.sh](../../README.md) / Code Tulu 2 34B # Code Tulu 2 34B Allen Institute for AI · released 2023-11 · 34B params · AI2 ImpACT Low-risk; Llama 2 base terms **score 45.2** ±1.6 · #475 of 732 · from 5 benchmark results across 1 source · † no independently run results verified Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at _{● marks Code Tulu 2 34B among every model on that task or test, worst to best.}

task    vs the others          rank  counts  tests
coding  ▁▁▁▃▅●█▇▅▄▃▁   #318 of 469*     15%      1
math    ▁▁▃▅●▆▄█▄▂▁▁   #263 of 431*     21%      1

coding · #318 of 469 · 15% of its score

Beats [Solar Open 100B (102B-A12B)](solar-open-100b.md), [Qwen3-VL 30B-A3B Instruct](qwen3-vl-30b-a3b-instruct.md), [Qwen3 4B Instruct 2507](qwen3-4b-instruct-2507.md) here, though they rank higher overall. Trails [DeepSeek LLM 67B Chat](deepseek-llm-67b-chat.md), [WizardLM 70B V1.0](wizardlm-70b-v1-0.md) here, though they rank lower overall.

test                               result  vs the others in its run  difficulty      counts  why
Tulu 2: CodexEval pass@10 (maker)   82.5%  ▃▃▃·▆▃▃█▃▂●▅ beats 22/25  easy · avg 83%     15%  sharp · at its level

math · #263 of 431 · 21% of its score

Beats [GPT-4o (2024-11-20)](gpt-4o-2024-11-20.md), [GPT-4 Turbo (2024-04-09)](gpt-4-turbo-2024-04-09.md), [Llama 3.1 405B Instruct](llama-3-1-405b.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md) here, though they rank lower overall.

test           result  vs the others in its run  difficulty      counts  why
GSM8K (maker)   54.0%  ▆▂▅█▅●▃▆···▂ beats 17/25  easy · avg 72%     21%  sharp · at its level

other tests (3)

test                                result  vs the others in its run  difficulty        counts  why
BIG-Bench Hard (maker)               64.3%  ▁·····▂█▂●·▁ beats 18/25  easy · avg 74%       41%  sharp · at its level
MMLU (maker)                         53.6%  ▁▁▃▃▄●▄▁█··▁ beats 14/25  medium · avg 73%     18%  at its level
Tulu 2: TyDiQA gold passage, 1-sh…   60.6%  ▂··▁▃▂▂▂█▅●▂ beats 23/25  medium · avg 59%    4.1%  blunt · at its level

## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md)

source                               results  share of the score
Tulu 2 (submitted / maker-reported)        5   99% ▇▇▇▇▇▇▇▇▇▇▇▇
prior: pull toward the middle                   1% ▇
weighted average                              = 45.2

_{Licences: Tulu 2 ([CC BY 4.0](../../licenses/CC-BY-4.0.txt))} ## How to run it ### Other ways to run it 1. Run it as published (BF16, needs ~67.49 GB, from [huggingface.co/allenai](https://huggingface.co/allenai/codetulu-2-34b/tree/20b3389675862e961649b2b031680f20855dcfaf)) ```sh uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='allenai/codetulu-2-34b', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])" ``` Model repository and documentation: [huggingface.co/allenai](https://huggingface.co/allenai/codetulu-2-34b) [interactive version](https://whatmodel.sh/model/code-tulu-2-34b) · [JSON](https://whatmodel.sh/api?format=json&all=1)