[whatmodel.sh](../../README.md) / Molmo 72B (0924) # Molmo 72B (0924) Ai2 · released 2024-09 · 73.3B params · apache-2.0 **score 50.4** ±1.7 · #363 of 732 · from 17 benchmark results across 4 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at _{● marks Molmo 72B (0924) among every model on that task or test, worst to best.}

task                vs the others          rank  counts  tests
math                ▁▁▃▅●▆▄█▄▂▁▁   #311 of 431*     15%      1
vision & documents  ▁▁▂▂●▃▄▂▆▇█▄    #128 of 153     99%     13

math · #311 of 431 · 15% of its score

Beats [Molmo 2 8B](molmo-2-8b.md), [Llama 3.1 Nemotron 70B Instruct](llama-3-1-nemotron-70b.md), [GPT-4 (0613)](gpt-4-0613.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [Mistral 7B v0.1 (base)](mistral-7b-v0-1.md) here, though they rank lower overall.

test                  result  vs the others in its run  difficulty        counts  why
MathVista (testmini)   52.2%  ▂····▂···▂●▆ beats 4/9    medium · avg 59%     15%  at its level · small run · several runs

vision & documents · #128 of 153 · 99% of its score

Beats [Gemma 3 12B](gemma-3-12b.md), [Llama 3.2 90B Vision Instruct](llama-3-2-90b-vision.md), [Nova Pro](nova-pro.md) here, though they rank higher overall. Trails [Qwen3.5 2B](qwen3-5-2b.md), [Qwen3-VL 2B Thinking](qwen3-vl-2b-thinking.md), [Qwen2.5-VL 7B Instruct](qwen2-5-vl-7b.md) here, though they rank lower overall.

test                                                         result  vs the others in its run   difficulty                counts  why
Arena vision (style controlled)                              1044.8  ▁▁▂▂●▂▃▄▅▅█▇ beats 21/152  avg 1061                     20%  blunt
MMMU (val)                                                    52.7%  ▂·▂······●▄▄ beats 5/9     medium · avg 53%             16%  at its level · small run · several runs
MathVista (testmini)                                          52.2%  ▂····▂···▂●▆ beats 4/9     medium · avg 59%             15%  at its level · small run · several runs
MM-MT-Bench (Mistral, GPT-4o judg…  3.5100 MM-MT-Bench score (1-10)  ●··▃·█·▃▃··▅ beats 0/9     avg 5.605 MM-MT-Bench s…    9.2%  sharp · noisy
VQAv2 (val)                                                   75.2%  ▃···█▃··●▃▃▃ beats 5/9     easy · avg 73%              8.8%  blunt · at its level · several runs
InfographicVQA (maker)                                        81.9%  ▃·▃▅█▃▃●▃▃▃▃ beats 9/13    easy · avg 78%              7.9%  small run
CountBench (maker)                                            91.2%  ▂▂▃▂·▂▂●▃·▅█ beats 7/18    very easy · avg 91%         6.6%  small run
ChartQA                                                       75.6%  ▄·▄▄▄··▄▄●▄█ beats 6/9     easy · avg 83%              3.6%  blunt · small run · several runs
DocVQA (maker)                                                93.5%  ▁··▃▁▁▁●█▇▄▅ beats 10/28   very easy · avg 93%         3.2%  blunt · maxed out · small run
Ai2 Molmo human preference Elo (S…                           1077.0  ▄▄·▄·▄▄▄·▄·● beats 8/8     avg 1032                    2.9%  blunt
RealWorldQA (maker)                                           75.2%  ▁▃▁▇▃▇●▅█▁▅▅ beats 20/39   easy · avg 73%              2.8%  blunt · at its level · small run
AI2D test (maker)                                             96.3%  ▁▁▅▇▃▇·▄·█·● beats 28/28   easy · avg 87%              1.5%  blunt · small run
TextVQA (maker)                                               83.1%  ▁▁▁▄▅▁▃·▃▃●█ beats 17/24   easy · avg 79%              1.3%  blunt · small run

## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md)

source                                    results  share of the score
Molmo2: Open Weights and Data for Visio…       10   48% ▇▇▇▇▇▇
Pixtral 12B                                     5   28% ▇▇▇
LMArena leaderboard dataset                     1   20% ▇▇
Ai2 Molmo-72B-0924 model card: Human Pr…        1    3% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 50.4

_{Licences: Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Pixtral 12B ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column ([Apache-2.0](https://huggingface.co/allenai/Molmo-72B-0924/blob/ab586dd9640de059566975b8e5bcd0c6fdc73956/README.md))} ## How to run it ### Other ways to run it 1. Run it as published (fp32, needs ~293.23 GB, NVIDIA only, from [github.com/vllm-project](https://github.com/vllm-project/vllm/blob/64cb68384241f046d07ba3797e1b14dbc82a0b2d/docs/models/supported_models.md)) ```sh vllm serve allenai/Molmo-72B-0924 --trust-remote-code ``` Model repository and documentation: [huggingface.co/allenai](https://huggingface.co/allenai/Molmo-72B-0924) [interactive version](https://whatmodel.sh/model/molmo-72b) · [JSON](https://whatmodel.sh/api?format=json&all=1)