[whatmodel.sh](../../README.md) / OLMo 7B # OLMo 7B Allen Institute for AI · released 2024-02 · 6.89B params · apache-2.0 **score 29.7** ±1.3 · #663 of 732 · from 13 benchmark results across 1 source · † no independently run results verified Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at _{● marks OLMo 7B among every model on that task or test, worst to best.}

task    vs the others      rank  counts  tests
coding  ●··········█   #2 of 2*    8.6%      1
math    ●··········█   #2 of 2*     11%      1

coding · #2 of 2 · 8.6% of its score

test                               result  vs the others in its run  difficulty      counts  why
Tulu 2: CodexEval pass@10 (maker)   21.4%  ●▃▄▁▄▃▃█▃▁▁▄ beats 0/27   easy · avg 78%    8.6%  sharp · small run

math · #2 of 2 · 11% of its score

test           result  vs the others in its run   difficulty        counts  why
GSM8K (maker)    8.5%  ▂▂▂▂●▄▅▇██▆▄ beats 32/206  medium · avg 63%     11%  sharp · small run

other tests (11)

test                                result  vs the others in its run   difficulty           counts  why
ARC Challenge (maker)                48.5%  ▃▃······▃▅▃● beats 7/8     easy · avg 86%          16%  sharp · at its level
WinoGrande (maker)                   67.9%  ▂····▂·▂·▂●█ beats 4/8     easy · avg 86%          12%  at its level
OpenBookQA (maker)                   50.4%  ▄·····▄▄▄█●▄ beats 6/8     easy · avg 81%          10%  at its level
BIG-Bench Hard (maker)               31.7%  ▁▁▁▁●▄▅▆█▃▃▁ beats 16/155  medium · avg 68%       8.7%  sharp · small run
HellaSwag (maker)                    76.4%  ▂·····▂▂·▂·● beats 6/8     very easy · avg 91%    8.4%  at its level
BoolQ (maker)                        73.4%  █····▃··●▃·▃ beats 4/8     easy · avg 88%         5.5%  at its level
ARC Easy (maker)                     65.4%  ▃···▃·▃··●▅█ beats 3/8     easy · avg 85%         5.5%  blunt · at its level
SciQ (maker)                         93.8%  ▂·▂···▂·▂·●▂ beats 5/8     very easy · avg 98%    5.1%  blunt
MMLU (maker)                         28.3%  ▁▁▁●▁▂▂▅██▆▄ beats 11/214  medium · avg 71%       4.4%  noisy · small run
PIQA (maker)                         78.4%  ▄····▄·▄▄▄█● beats 7/8     easy · avg 86%         3.4%  blunt · at its level
Tulu 2: TyDiQA gold passage, 1-sh…   32.3%  ▃··●▂▃▂▂█▅▁▂ beats 4/27    medium · avg 54%       1.5%  blunt · at its level · small run

## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md)

source                                    results  share of the score
OLMo: Accelerating the Science of Langu…       13   99% ▇▇▇▇▇▇▇▇▇▇▇▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 29.7

_{Licences: OLMo: Accelerating the Science of Language Models ([CC BY 4.0](../../licenses/paper-2402.00838v4-CC-BY-4.0.txt))} ## How to run it ### Other ways to run it 1. Run it as published (fp32, needs ~27.55 GB, from [huggingface.co/allenai](https://huggingface.co/allenai/OLMo-7B-hf/tree/11fb3186a2e4f681edea621fa8b4345147a9db6a)) ```sh uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='allenai/OLMo-7B-hf', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])" ``` Model repository and documentation: [huggingface.co/allenai](https://huggingface.co/allenai/OLMo-7B-hf) [interactive version](https://whatmodel.sh/model/olmo-7b) · [JSON](https://whatmodel.sh/api?format=json&all=1)