[whatmodel.sh](../../README.md) / Molmo 7B-D (0924) # Molmo 7B-D (0924) Ai2 · released 2024-09 · 8.02B params · apache-2.0 **score 41.7** ±1.4 · #536 of 732 · from 21 benchmark results across 5 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Molmo 7B-D (0924) among every model on that task or test, worst to best. 
task                vs the others          rank  counts  tests
coding              ▁●▁▃▅▇█▇▅▄▃▁   #464 of 469*    5.2%      1
math                ▁▁●▅█▆▄█▄▂▁▁    #400 of 431     14%      2
vision & documents  ▁▁●▂▁▃▄▂▆▇█▄    #143 of 153     74%     14
coding · #464 of 469 · 5.2% of its score Trails [GPT-Neo 2.7B](gpt-neo-2-7b.md), [Phi-1 (1.3B)](phi-1.md), [Pythia 6.9B](pythia-6-9b.md) here, though they rank lower overall. 
test       result  vs the others in its run  difficulty      counts  why
HumanEval    3.7%  ▁▁●▁▂▂▆▆█▅▂▁ beats 8/205  easy · avg 67%    5.2%  sharp · at its level · noisy
math · #400 of 431 · 14% of its score Beats [Command R (03-2024)](command-r-03-2024.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall. 
test                       result  vs the others in its run  difficulty        counts  why
MathVista (testmini)        12.3%  ●····▂···▂█▆ beats 0/9    medium · avg 59%    8.7%  at its level · small run · several runs
MATH (subset unspecified)   10.2%  ●▄▄▃█▇▅▆▄▃▄▁ beats 0/80   medium · avg 51%    5.5%  at its level
vision & documents · #143 of 153 · 74% of its score Beats [Gemma 3 4B](gemma-3-4b.md), [LLaVA-OneVision Qwen2 72B (ov-sft)](llava-onevision-qwen2-72b-ov.md), [CogVLM2 Llama3 Chat 19B](cogvlm2-llama3-chat-19b.md) here, though they rank higher overall. Trails [Qwen3.5 0.8B](qwen3-5-0-8b.md), [North Micro Vision Instruct (2.4B)](north-micro-vision-instruct.md) here, though they rank lower overall. 
test                                                         result  vs the others in its run  difficulty                counts  why
Arena vision (style controlled)                               993.7  ▁▁▂●▂▂▃▄▅▅█▇ beats 9/152  avg 1061                     14%  blunt
MathVista (testmini)                                          12.3%  ●····▂···▂█▆ beats 0/9    medium · avg 59%            8.7%  at its level · small run · several runs
MMMU (val)                                                    24.3%  ▂·●······█▄▄ beats 1/9    medium · avg 53%            8.6%  at its level · small run · several runs
VQAv2 (val)                                                   57.1%  ●···█▃··▅▃▃▃ beats 0/9    easy · avg 73%              6.8%  blunt · at its level · several runs
Kaleidoscope (in-language multimo…                            32.9%  ●·█···█·█·██ beats 0/5    medium · avg 46%            6.8%  at its level
InfographicVQA (maker)                                        72.6%  ▃·▃▅●▃▃▃▃▃▃▃ beats 6/13   easy · avg 78%              6.4%  at its level · small run
MM-MT-Bench (Mistral, GPT-4o judg…  3.7200 MM-MT-Bench score (1-10)  ●··▃·█·▃▃··▅ beats 1/9    avg 5.605 MM-MT-Bench s…    6.2%  sharp · noisy
CountBench (maker)                                            88.5%  ▂▂▃▂·▂●▃▃·▅█ beats 6/18   very easy · avg 91%         5.8%  small run
ChartQA                                                       27.0%  ▄·●▄▄··▄▄▄▄█ beats 1/9    easy · avg 83%              2.9%  blunt · small run · several runs
DocVQA (maker)                                                92.2%  ▁··▃▁▁●▇█▇▄▅ beats 5/28   very easy · avg 93%         2.7%  blunt · small run
RealWorldQA (maker)                                           70.7%  ▁▃▁▇▃●▇▅█▁▅▅ beats 13/39  easy · avg 73%              2.1%  blunt · at its level · small run
Ai2 Molmo human preference Elo (S…                           1056.0  ▄▄·▄·▄▄▄·●·█ beats 6/8    avg 1032                    2.0%  blunt
AI2D test (maker)                                             93.2%  ▁▁▅▇▃▇·▄·●·▃ beats 24/28  easy · avg 87%              1.1%  blunt · small run
TextVQA (maker)                                               81.7%  ▁▁▁▄▅▁▃·▃●▃█ beats 15/24  easy · avg 79%              1.0%  blunt · at its level · small run
other tests (1) 
test  result  vs the others in its run   difficulty      counts  why
MMLU   61.2%  ▁▁▁▁▁▂▂▅●█▆▄ beats 91/214  easy · avg 75%     14%  at its level
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Pixtral 12B                                     8   42% ▇▇▇▇▇
Molmo2: Open Weights and Data for Visio…       10   35% ▇▇▇▇
LMArena leaderboard dataset                     1   14% ▇▇
Kaleidoscope: In-language Exams for Mas…        1    7% ▇
Ai2 Molmo-72B-0924 model card: Human Pr…        1    2% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 41.7
Licences: Pixtral 12B ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column ([Apache-2.0](https://huggingface.co/allenai/Molmo-72B-0924/blob/ab586dd9640de059566975b8e5bcd0c6fdc73956/README.md)) ## How to run it ### Other ways to run it 1. Run it as published (fp32, needs ~32.08 GB, NVIDIA only, from [github.com/vllm-project](https://github.com/vllm-project/vllm/blob/64cb68384241f046d07ba3797e1b14dbc82a0b2d/docs/models/supported_models.md)) ```sh vllm serve allenai/Molmo-7B-D-0924 --trust-remote-code ``` Model repository and documentation: [huggingface.co/allenai](https://huggingface.co/allenai/Molmo-7B-D-0924) [interactive version](https://whatmodel.sh/model/molmo-7b-d) · [JSON](https://whatmodel.sh/api?format=json&all=1)