[whatmodel.sh](../../README.md) / Gemini 1.5 Pro Experimental (0827) # Gemini 1.5 Pro Experimental (0827) Google · released 2024-08 · proprietary **score 57.2** ±0.8 · comparison only: no current run option verified · from 6 benchmark results across 3 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Gemini 1.5 Pro Experimental (0827) among every model on that task or test, worst to best. 
task    vs the others          rank  counts  tests
coding  ▁▁▁▃▄▆█●▅▄▃▁   #234 of 539*     25%      2
math    ▁▁▃▆█●▅█▃▁▁▁   #240 of 530*    9.3%      1
coding · #234 of 539 · 25% of its score Beats [Gemini 2.5 Flash](gemini-2-5-flash.md), [Qwen3 30B-A3B Instruct 2507](qwen3-30b-a3b-instruct-2507.md), [Qwen3.5 4B](qwen3-5-4b.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [DeepSeek-Coder-V2 Instruct (0724)](deepseek-coder-v2-0724.md), [Ministral 3 8B Reasoning 2512](ministral-3-8b-reasoning.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty        counts  why
CRUXEval (ZeroEval, 800 problems)    79.6%  ▁▂▆▃▆█▆●▂·▁▁ beats 41/47  medium · avg 57%     14%  sharp
Aider code editing (133 Exercism …   66.9%  ▁·▁▂▄▁▆▄●▆▂▂ beats 42/62  medium · avg 51%     11%  sharp · at its level
math · #240 of 530 · 9.3% of its score Beats [MiniMax-M2.1](minimax-m2-1.md), [DeepSeek V3 (Dec '24)](deepseek-v3.md), [Qwen3 235B-A22B (Apr 2025, hybrid)](qwen3-235b-a22b.md) here, though they rank higher overall. Trails [Granite 8B Code Base (4K)](granite-8b-code-base-4k.md), [Mixtral 8x7B v0.1 (base)](mixtral-8x7b-v0-1.md), [Olmo 3 7B Instruct](olmo-3-7b-instruct.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty      counts  why
MATH Level 5 (ZeroEval, 721 probl…   68.1%  ▁▁▁▆█▂▇▇●··▂ beats 36/38  hard · avg 30%    9.3%  sharp · at its level
other tests (3) 
test                                result  vs the others in its run  difficulty        counts  why
LiveBench 2024-07-26                 53.8%  ▁▁▁▂▂▃▂█▆▃▅● beats 57/61  medium · avg 38%     29%  at its level
MMLU-Redux (ZeroEval, 2,778 quest…   86.1%  ▁·▁▁▄▄▄▅█●▂▁ beats 47/53  easy · avg 75%       21%  typical
ZebraLogic (ZeroEval, 1,000 puzzl…   30.5%  ▁▁▃▆█●▁▁·▁▁▁ beats 48/57  hard · avg 15%       15%  sharp · at its level
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                               results  share of the score
WildEval ZeroEval published results        4   59% ▇▇▇▇▇▇▇
LiveBench model judgments                  1   29% ▇▇▇▇
Aider code editing benchmark runs          1   11% ▇
prior: pull toward the middle                   0% ▇
weighted average                              = 57.2
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) ## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/gemini-1-5-pro-exp-0827) · [JSON](https://whatmodel.sh/api?format=json&all=1)