[whatmodel.sh](../../README.md) / Gemini 1.5 Pro (001) # Gemini 1.5 Pro (001) Google · released 2024-05 · proprietary **score 54.0** ±0.7 · comparison only: no current run option verified · from 27 benchmark results across 9 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Gemini 1.5 Pro (001) among every model on that task or test, worst to best. 
task                vs the others          rank  counts  tests
coding              ▁▁▁▃▄▆●▇▅▄▃▁    #274 of 539     15%      3
chat & writing      ▁▁▁▃▅▄▇●▇▇▇▃   #231 of 474*    4.8%      1
math                ▁▁▃▆●█▅█▃▁▁▁    #349 of 530     23%      4
vision & documents  ▁▁▂▃▂●▅▄▆██▄    #143 of 189     16%      5
coding · #274 of 539 · 15% of its score Beats [Qwen3.5 4B](qwen3-5-4b.md), [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Solar Open 100B (102B-A12B)](solar-open-100b.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 3B Reasoning 2512](ministral-3-3b-reasoning.md), [GPT-4 (0613)](gpt-4-0613.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty            counts  why
CRUXEval (ZeroEval, 800 problems)    68.0%  ▁▂▆▃▆●▆▃▂·▁▁ beats 25/47  medium · avg 57%         11%  sharp · at its level
WebApp1K-React pass@1 (onekq-ai)     68.1%  ▃▁▁█▁▄●▃▄▇▁▄ beats 17/31  medium · avg 40%        2.2%  sharp · at its level · noisy
SciCode main problems (official l…    1.5%  ▃···●··▂·▃·▂ beats 3/17   very hard · avg 1.0%    1.8%  sharp · too hard for it
chat & writing · #231 of 474 · 4.8% of its score Beats [Gemma 2 9B IT SimPO](gemma-2-9b-simpo.md), [Granite 4.2 8B](granite-4-2-8b.md), [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md) here, though they rank lower overall. 
test            result  vs the others in its run    difficulty  counts  why
LMArena (text)  1324.1  ▁▁▁▂▃▃▃●▆▇█▃ beats 160/385  avg 1260      4.8%  blunt
math · #349 of 530 · 23% of its score Beats [Molmo 2 8B](molmo-2-8b.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [Mistral 7B v0.1 (base)](mistral-7b-v0-1.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty            counts  why
MATH level 5                         40.7%  ▂▂▅▆●█▆▃▅▂▃▄ beats 33/92   hard · avg 28%           11%  sharp · at its level
MATH Level 5 (ZeroEval, 721 probl…   39.8%  ▁▁▁▆█▂●▇▄··▂ beats 20/38   hard · avg 30%          6.4%  sharp · at its level
GSM8K (ZeroEval, zero-shot)          93.4%  ▁▄·▁▄█▅▅▅●██ beats 29/43   very easy · avg 87%     3.7%  sharp · maxed out
OTIS mock AIME 2024–2025              6.8%  ▃▃▄●▃▃▇▇█▃▂▅ beats 40/197  very hard · avg 7.2%    2.1%  sharp · noisy
vision & documents · #143 of 189 · 16% of its score Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GPT-4 Turbo (2024-04-09)](gpt-4-turbo-2024-04-09.md), [Pixtral Large (2411)](pixtral-large.md) here, though they rank higher overall. Trails [Gemma 3 27B](gemma-3-27b.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty        counts  why
CharXiv descriptive questions        72.0%  ▄█▄▄▄·●█·▄·█ beats 7/12    easy · avg 69%      4.7%  typical
Arena vision (style controlled)     1119.5  ▁▁▂▂▂▂●▄▅▅█▇ beats 35/152  avg 1061            3.2%  blunt
CharXiv (reasoning questions)        43.3%  ▂▂▂█▃▂·●▂··▂ beats 11/13   hard · avg 34%      3.1%  at its level
MMMU val (VHELM, zero-shot exact …   61.9%  ▄▄··▄▄█··●▄█ beats 6/9     medium · avg 50%    2.7%  at its level
MathVista testmini (VHELM, zero-s…   57.6%  ▃·▅···▃▃▃·▃● beats 9/9     medium · avg 42%    2.4%  at its level
other tests (14) 
test                                result  vs the others in its run   difficulty           counts  why
MMLU-Redux (ZeroEval, 2,778 quest…   82.8%  ▁·▁▁▄▄▄▅●▄▂▁ beats 38/53   easy · avg 75%          17%  typical
ZebraLogic (ZeroEval, 1,000 puzzl…   19.4%  ▁▁▃▆●█▁▁·▁▁▁ beats 28/57   hard · avg 15%         8.5%  sharp · at its level
GPQA Diamond                         45.9%  ▁▁▁▁▂▄●▄▅▇█▅ beats 51/211  hard · avg 39%         6.6%  sharp · at its level
AHELM Speech Robust Bench (LibriS…   21.3%  ▂·█··●▄·▂▂·▄ beats 6/13    avg 34%                3.4%  typical
AHELM CoRe-Bench conversational a…   65.9%  ▂·▄··▂▂·●▂█▄ beats 6/13    medium · avg 52%       1.7%  at its level
AHELM LibriSpeech transcription b…    7.1%  ▃···▃·█▃●▃▅█ beats 7/13    avg 11%                0.8%  typical
AHELM VoxCeleb2 same-speaker dete…   52.4%  ▂·▂·▂●▂▄▄··▄ beats 3/13    very hard · avg 54%    0.7%  sharp · noisy
AHELM MUStARD audio sarcasm detec…   56.4%  ▂·······●█▆▆ beats 3/13    very hard · avg 57%    0.6%  blunt
AHELM AIR-Bench Foundation music …   51.1%  ▃●▃▅▃·▃·▃▅▃▃ beats 2/13    medium · avg 57%       0.3%  blunt · at its level
AHELM MELD audio emotion classifi…   46.9%  ▃··▃▃▃·▃▅●█▅ beats 8/13    medium · avg 39%       0.3%  blunt · at its level
Vectara hallucination leaderboard…   90.9%  ▁▁▂▃●▄█▆▃▂▂▁ beats 27/140  very easy · avg 94%    0.3%  blunt · maxed out · noisy
AHELM FLEURS 7-language transcrip…   46.3%  ▄▂···▂▆▂●▂·▂ beats 8/13    avg 83%                0.2%  blunt · noisy
AHELM Multilingual LibriSpeech 7-…    7.3%  ▂▂▂··▂▂·▂▂●▆ beats 7/13    avg 14%                0.0%  blunt · noisy
AHELM VocalSound human-sound reco…   49.2%  █●▄▄·█·█·█▄▄ beats 2/13    easy · avg 67%         0.0%  blunt · at its level · noisy
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
WildEval ZeroEval published results             5   47% ▇▇▇▇▇▇
Epoch AI                                        3   19% ▇▇
AHELM: A Holistic Evaluation of Audio-L…       10    8% ▇
LMArena leaderboard dataset                     2    8% ▇
CharXiv leaderboard                             2    8% ▇
VHELM: A Holistic Evaluation of Vision …        2    5% ▇
WebApp1K models leaderboard                     1    2% ▇
SciCode leaderboard                             1    2% ▇
Vectara hallucination leaderboard               1    0% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 54.0
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · AHELM: A Holistic Evaluation of Audio-Language Models ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · CharXiv leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · VHELM: A Holistic Evaluation of Vision Language Models ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) ## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/gemini-1-5-pro-001) · [JSON](https://whatmodel.sh/api?format=json&all=1)