[whatmodel.sh](../../README.md) / OLMo 7B Instruct # OLMo 7B Instruct Ai2 · released 2024-02 · 6.89B params · apache-2.0 **score 31.9** ±1.5 · #651 of 732 · from 7 benchmark results across 3 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at _{● marks OLMo 7B Instruct among every model on that task or test, worst to best.}

task            vs the others          rank  counts  tests
coding          █··········●       #1 of 2*     13%      1
chat & writing  ▁▁●▃▆▄▆█▆▇▇▃   #391 of 402*     46%      2
math            █··········●       #1 of 2*     17%      1

coding · #1 of 2 · 13% of its score

test                               result  vs the others in its run  difficulty      counts  why
Tulu 2: CodexEval pass@10 (maker)   27.8%  ▃●▄▁▄▃▃█▃▁▁▄ beats 3/27   easy · avg 78%     13%  sharp · at its level · small run

chat & writing · #391 of 402 · 46% of its score

Beats [Phi-2 (2.7B)](phi-2.md) here, though they rank higher overall. Trails [Gemma 3 270M IT](gemma-3-270m.md), [SmolLM2 360M Instruct](smollm2-360m.md), [Granite 4.0 350M](granite-4-0-350m.md) here, though they rank lower overall.

test                                result  vs the others in its run   difficulty  counts  why
LMArena (text)                      1073.7  ▁▁●▂▃▃▃▆▆▇█▃ beats 13/385  avg 1260       24%  blunt
Polyrating rating (bias-aware re-…   995.0  ▂▁▂▁●▇█▆▇▄▂▂ beats 13/95   avg 1187       22%  typical

math · #1 of 2 · 17% of its score

test           result  vs the others in its run   difficulty        counts  why
GSM8K (maker)   11.0%  ▂▂▂▂●▄▅▇██▆▄ beats 35/206  medium · avg 63%     17%  sharp · small run

other tests (3)

test                                result  vs the others in its run   difficulty        counts  why
BIG-Bench Hard (maker)               35.8%  ▁▁▁▁▂●▅▆█▃▃▁ beats 25/155  medium · avg 68%     14%  sharp · small run
MMLU (maker)                         46.1%  ▁▁▁▁▁▂▂●██▆▄ beats 43/214  medium · avg 71%    7.2%  noisy · small run
Tulu 2: TyDiQA gold passage, 1-sh…   21.7%  ●··▂▂▃▂▂█▅▁▂ beats 0/27    medium · avg 54%    2.3%  blunt · at its level · small run

## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md)

source                                    results  share of the score
OLMo: Accelerating the Science of Langu…        5   53% ▇▇▇▇▇▇
LMArena leaderboard dataset                     1   24% ▇▇▇
Polyrating: A Cost-Effective and Bias-A…        1   22% ▇▇▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 31.9

_{Licences: OLMo: Accelerating the Science of Language Models ([CC BY 4.0](../../licenses/paper-2402.00838v4-CC-BY-4.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/))} ## How to run it ### Other ways to run it 1. Run it as published (BF16, needs ~13.78 GB, from [huggingface.co/allenai](https://huggingface.co/allenai/OLMo-7B-Instruct/blob/5e3ff29c568c175562a78d726c1bd027a0bc6166/README.md)) ```sh pip install ai2-olmo && python -c "from hf_olmo import OLMoForCausalLM, OLMoTokenizerFast; o=OLMoForCausalLM.from_pretrained('allenai/OLMo-7B-Instruct'); t=OLMoTokenizerFast.from_pretrained('allenai/OLMo-7B-Instruct'); p=t.apply_chat_template([{'role': 'user', 'content': 'What is language modeling?'}], tokenize=False, add_generation_prompt=True); i=t.encode(p, add_special_tokens=False, return_tensors='pt'); print(t.batch_decode(o.generate(input_ids=i.to(o.device), max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95), skip_special_tokens=True)[0])" ``` Model repository and documentation: [huggingface.co/allenai](https://huggingface.co/allenai/OLMo-7B-Instruct) [interactive version](https://whatmodel.sh/model/olmo-7b-instruct) · [JSON](https://whatmodel.sh/api?format=json&all=1)