[whatmodel.sh](../../README.md) / Gemini 1.5 Pro (002) # Gemini 1.5 Pro (002) Google · released 2024-09 · proprietary **score 57.1** ±0.7 · comparison only: no current run option verified · from 26 benchmark results across 11 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Gemini 1.5 Pro (002) among every model on that task or test, worst to best. 
task                vs the others          rank  counts  tests
coding              ▁▁▁▃▄▆●▇▅▄▃▁    #252 of 539     50%      5
chat & writing      ▁▁▁▃▅▄▇█●▇▇▃   #193 of 474*    4.4%      1
math                ▁▁▃▆█●▅█▃▁▁▁   #245 of 530*     12%      2
vision & documents  ▁▁▂▃▂▃▅●▆██▄   #109 of 189*    4.8%      2
coding · #252 of 539 · 50% of its score Beats [Qwen3.5 4B](qwen3-5-4b.md), [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Solar Open 100B (102B-A12B)](solar-open-100b.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 3B Reasoning 2512](ministral-3-3b-reasoning.md), [s1.1 32B](s1-1-32b.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty        counts  why
HumanEval+                           79.3%  ▁·▁▁▁▃▄▄▄█●▁ beats 91/99  easy · avg 70%       14%  sharp
HumanEval                            89.0%  ▁·▁▁▃▄▆▅█●▁▁ beats 95/99  easy · avg 76%       14%  sharp
LiveCodeBench official (Jan 1–Nov…   36.4%  ▁▁·▁▂▄▆●▅▂·▂ beats 21/31  hard · avg 23%       11%  sharp · at its level
Aider code editing (133 Exercism …   65.4%  ▁·▁▂▄▁▆▄●▆▂▂ beats 36/62  medium · avg 51%    7.3%  sharp · at its level
MBPP+                                74.6%  ▁···▁▁▂▅▇█▆● beats 71/75  easy · avg 66%      3.8%  typical
chat & writing · #193 of 474 · 4.4% of its score Beats [MiniMax-M2](minimax-m2.md), [MiniCPM5 2B](minicpm5-2b.md), [Granite 4.2 30B](granite-4-2-30b.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md), [LFM2.5 2.6B](lfm2-5-2-6b.md) here, though they rank lower overall. 
test            result  vs the others in its run    difficulty  counts  why
LMArena (text)  1351.3  ▁▁▁▂▃▃▃▆●▇█▃ beats 194/385  avg 1260      4.4%  blunt
math · #245 of 530 · 12% of its score Beats [MiniMax-M2.1](minimax-m2-1.md), [DeepSeek V3 (Dec '24)](deepseek-v3.md), [GPT-4o (2024-08-06)](gpt-4o-2024-08-06.md) here, though they rank higher overall. Trails [Granite 8B Code Base (4K)](granite-8b-code-base-4k.md), [Mixtral 8x7B v0.1 (base)](mixtral-8x7b-v0-1.md), [Olmo 3 7B Instruct](olmo-3-7b-instruct.md) here, though they rank lower overall. 
test                      result  vs the others in its run   difficulty            counts  why
MATH level 5               70.4%  ▂▂▅▆▄█●▃▅▂▃▄ beats 61/92   hard · avg 28%          9.2%  sharp · at its level
OTIS mock AIME 2024–2025   23.1%  ▃▃▄▃●▃▇▇█▃▂▅ beats 61/197  very hard · avg 7.2%    3.0%  sharp · at its level · noisy
vision & documents · #109 of 189 · 4.8% of its score Beats [DeepSeek V4 Flash Vision (Exp)](deepseek-v4-flash-vision-exp.md), [GLM-4.6V](glm-4-6v.md), [GPT-5 nano](gpt-5-nano.md) here, though they rank higher overall. 
test                                 result  vs the others in its run   difficulty  counts  why
Arena vision (style controlled)      1179.5  ▁▁▂▂▂▂▃▄●▅█▇ beats 61/152  avg 1061      3.0%  blunt
CapArena-Auto detailed image capt…  56.1700  ▃▃▃·▃█·▅▃▃▃● beats 12/12   avg -4.498    1.9%  typical
other tests (16) 
test                                result  vs the others in its run   difficulty           counts  why
GPQA Diamond                         57.2%  ▁▁▁▁▂▄▆●▅▇█▅ beats 86/211  hard · avg 39%         7.2%  sharp · at its level
MMLU-Pro                             70.2%  ▃▂▅█▃·█●▅··▂ beats 18/23   medium · avg 54%       6.7%  at its level
AHELM Speech Robust Bench (LibriS…   20.7%  ▂·█··●▄·▂▂·▄ beats 7/13    avg 34%                3.2%  typical
BRIDGE clinical text, zero-shot a…   43.9%  ▁▁▁▁▁▂▂▃▄▆█● beats 89/97   hard · avg 31%         2.7%  blunt · at its level
LiveBench 2024-08-31                 54.9%  ▄█·▄·▄●·▄··▄ beats 5/7     medium · avg 40%       2.5%  at its level
EvalPerf correctness (118 tasks, …   83.7%  ▂·▂▂▂▆▅▆▆●▃█ beats 21/29   easy · avg 74%         1.8%  typical
AHELM CoRe-Bench conversational a…   79.9%  ▂·▄··▂▂·▄▂█● beats 12/13   medium · avg 52%       1.5%  at its level
AHELM VoxCeleb2 same-speaker dete…   58.5%  ▂·▂·▂█▂▄●··▄ beats 11/13   very hard · avg 54%    0.8%  sharp · noisy
AHELM LibriSpeech transcription b…    5.2%  ▃···▃·█▃▅▃●█ beats 9/13    avg 11%                0.8%  typical
AHELM MUStARD audio sarcasm detec…   57.7%  ▂·······▆●▆▆ beats 5/13    very hard · avg 57%    0.6%  blunt
AHELM AIR-Bench Foundation music …   55.4%  ▃█▃●▃·▃·▃▅▃▃ beats 5/13    medium · avg 57%       0.3%  blunt · at its level
AHELM MELD audio emotion classifi…   51.6%  ▃··▃▃▃·▃▅▅█● beats 13/13   medium · avg 39%       0.2%  blunt · at its level
Vectara hallucination leaderboard…   93.7%  ▁▁▂▃▄●█▆▃▂▂▁ beats 49/140  very easy · avg 94%    0.2%  blunt · maxed out · noisy
AHELM FLEURS 7-language transcrip…   34.2%  ▄▂···▂▆▂█●·▂ beats 12/13   avg 83%                0.2%  blunt · noisy
AHELM Multilingual LibriSpeech 7-…    5.4%  ▂▂▂··▂▂·▂▂█● beats 13/13   avg 14%                0.0%  blunt · noisy
AHELM VocalSound human-sound reco…   52.8%  ██●▄·█·█·█▄▄ beats 4/13    easy · avg 67%         0.0%  blunt · at its level · noisy
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
EvalPlus / EvalPerf published evaluatio…        4   33% ▇▇▇▇
Epoch AI                                        3   19% ▇▇
LiveCodeBench official website results          1   11% ▇
AHELM: A Holistic Evaluation of Audio-L…       10    8% ▇
LMArena leaderboard dataset                     2    7% ▇
Aider code editing benchmark runs               1    7% ▇
MMLU-Pro author-released evaluation runs        1    7% ▇
BRIDGE Medical Leaderboard                      1    3% ▇
LiveBench model judgments                       1    2% ▇
CapArena: Benchmarking and Analyzing De…        1    2% ▇
Vectara hallucination leaderboard               1    0% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 57.1
Licences: EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · AHELM: A Holistic Evaluation of Audio-Language Models ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · MMLU-Pro author-released evaluation runs ([Apache-2.0](../../licenses/mmlu-pro-author-runs-Apache-2.0.txt)) · BRIDGE Medical Leaderboard ([MIT](../../licenses/bridge-medical-leaderboard-MIT.txt)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) ## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/gemini-1-5-pro-002) · [JSON](https://whatmodel.sh/api?format=json&all=1)