[whatmodel.sh](../../README.md) / Gemma 2 27B IT # Gemma 2 27B IT Google · released 2024-06 · 27.23B params · gemma **score 50.9** ±0.5 · #348 of 732 · from 34 benchmark results across 16 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Gemma 2 27B IT among every model on that task or test, worst to best. 
task            vs the others         rank  counts  tests
coding          ▁▁▁▃▅▇●▇▅▄▃▁   #236 of 469     11%      5
chat & writing  ▁▁▁▃▆▄▆●▆▇▇▃   #243 of 402     19%      6
math            ▁▁▃▅●▆▄█▄▂▁▁   #330 of 431     18%      7
coding · #236 of 469 · 11% of its score Beats [Qwen3.5 4B](qwen3-5-4b.md), [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Solar Open 100B (102B-A12B)](solar-open-100b.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 3B Reasoning 2512](ministral-3-3b-reasoning.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty        counts  why
CRUXEval (ZeroEval, 800 problems)    57.3%  ▁▂▆▃●█▆▃▂·▁▁ beats 17/47  medium · avg 57%    8.3%  sharp · at its level
CodeReviewQA: change localisation…   58.7%  █▂▁▁▂▂▂▃▅▅▅● beats 41/47  hard · avg 33%      1.1%  sharp · at its level · noisy
CodeReviewQA: solution identifica…   65.7%  ▃··▁▂▂▃▂█●▄▂ beats 35/47  medium · avg 51%    1.0%  sharp · at its level · noisy
CodeReviewQA: automated code refi…   46.4%  ▁▁▁▁▁▁▂▂▃█▅● beats 44/47  hard · avg 32%      0.8%  at its level
CodeReviewQA: change type recogni…   74.0%  ▁▁▁······▁▃● beats 28/47  easy · avg 67%      0.0%  blunt · at its level · noisy
chat & writing · #243 of 402 · 19% of its score Beats [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md), [Qwen2.5 Coder 32B Instruct](qwen2-5-coder-32b.md), [Ministral 3 14B Reasoning 2512](ministral-3-14b-reasoning.md) here, though they rank higher overall. Trails [Flan-T5 XXL](flan-t5-xxl.md), [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [Gemma 3 4B](gemma-3-4b.md) here, though they rank lower overall. 
test                                result  vs the others in its run    difficulty        counts  why
Arena-Hard (v0.1, GPT-4-1106 judg…   57.5%  ▂▁▁▅█▅▄●▃▅▃▁ beats 40/60    medium · avg 47%    7.7%  sharp · at its level
LMArena (text)                      1289.5  ▁▁▁▂▃▃▃●▆▇█▃ beats 121/385  avg 1260            3.1%  blunt
Polyrating rating (bias-aware re-…  1232.0  ▂▁▂▁▃▇█▆▇▄●▂ beats 89/95    avg 1187            2.8%  typical
Arabic IFEval prompt-level (Incep…   59.4%  ▁·▁▁▂▁▄▄▆▄●▄ beats 33/38    medium · avg 46%    2.5%  sharp · at its level
AlpacaEval 2 length-controlled       49.0%  ▃▃·▁▁█▇●·▁·▁ beats 19/22    medium · avg 39%    2.0%  sharp · at its level
IFEval prompt-level loose            63.2%  ▁▃▁●▁▄▁▁█▃▃▁ beats 5/22     easy · avg 78%      0.6%  typical
math · #330 of 431 · 18% of its score Beats [GPT-4 (0613)](gpt-4-0613.md), [Athene-70B](athene-70b.md) here, though they rank higher overall. Trails [Stable Code 3B](stable-code-3b.md), [Qwen3 0.6B](qwen3-0-6b.md), [Falcon 40B (base)](falcon-40b.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty            counts  why
MATH level 5                         27.9%  ▂▂▅●▄█▆▃▅▂▃▄ beats 26/92   hard · avg 28%          6.0%  sharp · at its level
MATH Level 5 (ZeroEval, 721 probl…   26.6%  ▁▁▁▆●▂▇▇▄··▂ beats 16/38   hard · avg 30%          3.7%  sharp · at its level
GSM8K (ZeroEval, zero-shot)          90.2%  ▁▄·▁▄█▅▅●███ beats 22/43   very easy · avg 87%     3.4%  sharp
MATH (subset unspecified)            35.1%  ▃●█▂▂▆·····▁ beats 5/22    medium · avg 58%        1.7%  at its level · several runs
GSM8K                                80.7%  ▂▃··▃▂●█▆▃▂▃ beats 8/22    very easy · avg 88%     1.5%  sharp
OTIS mock AIME 2024–2025              1.4%  ▃●▄▃▃▃▇▇█▃▂▅ beats 11/197  very hard · avg 7.2%    0.8%  sharp · noisy
OMEGA (math generalisation)           9.1%  ●···▃▃···▃▃█ beats 0/8     hard · avg 24%          0.6%  at its level
other tests (15) 
test                                          result  vs the others in its run    difficulty           counts  why
MMLU-Redux (ZeroEval, 2,778 quest…             75.7%  ▁·▁▁▄▄●▅█▄▂▁ beats 21/53    easy · avg 75%          13%  at its level
LiveBench 2024-07-26                           41.3%  ▁▁▁▂▂▃▂█▆●▅▃ beats 46/61    medium · avg 38%        13%  at its level
MMLU                                           70.7%  ▁▂·▁·▁▁●·▃▃▅ beats 11/22    easy · avg 75%         4.3%  at its level
ZebraLogic (ZeroEval, 1,000 puzzl…             16.3%  ▁▁▃▆●█▁▁·▁▁▁ beats 21/57    hard · avg 15%         4.2%  sharp
BIG-Bench Hard                                 72.7%  ▁▄·▁▁·▅●·▄▄▁ beats 15/22    medium · avg 72%       3.9%  sharp · at its level
GPQA Diamond                                   36.5%  ▁▁▁▁▂●▆▄▅▇█▅ beats 31/211   hard · avg 39%         3.4%  sharp
WildBench v2 WB-Score (task macro)  48.5402 WB-Score  ▁·▁▁·▃█▄▇▆●▆ beats 39/52    avg 43.019 WB-Score    3.3%  sharp
EvalPerf correctness (118 tasks, …             80.2%  ▂·▂▂▂▆▅▆●▆▃█ beats 18/29    easy · avg 74%         1.7%  typical
BRIDGE clinical text, zero-shot a…             38.2%  ▁▁▁▁▁▂▂▃▄●█▃ beats 61/97    hard · avg 31%         1.7%  blunt · at its level
Open Persian LLM Leaderboard v2 a…             60.8%  ▂▄▂▃█▄▃●▃▅▂▂ beats 54/81    medium · avg 57%       1.4%  blunt · at its level
PopQA                                          33.9%  ▁▁▁▃▃▃▁█●▃·▄ beats 16/21    hard · avg 31%         0.9%  at its level
Open PT-LLM Leaderboard average (…             76.3%  ▃▁▁▁▁▁▁▁▂█●▁ beats 232/276  medium · avg 72%       0.7%  at its level · noisy
EuroEval HellaSwag (English), tes…             83.0%  ▃·▁▁▂▁▂▅█●▄▁ beats 78/100   medium · avg 67%       0.6%  sharp · at its level · noisy
EuroEval Life in the UK (English …             88.2%  ▁▁▁▁▁▁▁▂▄█●▃ beats 111/146  easy · avg 82%         0.4%  noisy
TruthfulQA MC2                                 64.6%  ▃▃▂·▃▃·▅●▂▅▂ beats 14/21    medium · avg 60%       0.1%  blunt · at its level
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
WildEval ZeroEval published results             5   33% ▇▇▇▇
2 OLMo 2 Furious                                8   14% ▇▇
LiveBench model judgments                       1   13% ▇▇
Epoch AI                                        3   10% ▇
Arena-Hard-Auto v0.1 leaderboard                1    8% ▇
WildBench v2 leaderboard data                   1    3% ▇
LMArena leaderboard dataset                     1    3% ▇
CodeReviewQA: The Code Review Comprehen…        4    3% ▇
Polyrating: A Cost-Effective and Bias-A…        1    3% ▇
Inception Arabic/Hindi leaderboards             1    3% ▇
EvalPlus / EvalPerf published evaluatio…        1    2% ▇
BRIDGE Medical Leaderboard                      1    2% ▇
Open Persian LLM Leaderboard v2                 1    1% ▇
allenai/Olmo-3.1-32B-Think model card           2    1% ▇
EuroEval leaderboards                           2    1% ▇
Open Portuguese LLM Leaderboard                 1    1% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 50.9
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · 2 OLMo 2 Furious ([CC BY 4.0](../../licenses/paper-2501.00656v3-CC-BY-4.0.txt)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · WildBench v2 leaderboard data ([Apache-2.0](https://github.com/allenai/WildBench/blob/d6b8dcaf377d173d031980f97c16e1a82618c03d/LICENSE)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · BRIDGE Medical Leaderboard ([MIT](../../licenses/bridge-medical-leaderboard-MIT.txt)) · Open Persian LLM Leaderboard v2 ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · allenai/Olmo-3.1-32B-Think model card ([Apache-2.0](../../licenses/allenai--Olmo-3.1-32B-Think.txt)) · EuroEval leaderboards ([MIT](https://github.com/EuroEval/leaderboards/blob/1a77f1d2f8ba1ba05ab73287fb090d76bf41b2e0/LICENSE)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) ## How to run it ### llama.cpp (recommended) 1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) ```sh # macOS / Linux curl -LsSf https://llama.app/install.sh | sh # Windows irm https://llama.app/install.ps1 | iex ``` 2. Run Gemma 2 27B IT (Q4_K_M, needs ~16.6 GB, from [huggingface.co/lmstudio-community](https://huggingface.co/lmstudio-community/gemma-2-27b-it-GGUF/tree/8f63ea1cfc5ce646813257d8f5d5a1afa82af58f)) ```sh llama cli -hf lmstudio-community/gemma-2-27b-it-GGUF:Q4_K_M ``` 3. Or start a local server and open the address it prints (http://localhost:9931) ```sh llama serve -hf lmstudio-community/gemma-2-27b-it-GGUF:Q4_K_M ``` Other versions: Q8_0 (~28.9 GB) `llama cli -hf lmstudio-community/gemma-2-27b-it-GGUF:Q8_0` ### LM Studio 1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download) 2. Run Gemma 2 27B IT (Q4_K_M, needs ~16.6 GB, from [huggingface.co/lmstudio-community](https://huggingface.co/lmstudio-community/gemma-2-27b-it-GGUF/tree/8f63ea1cfc5ce646813257d8f5d5a1afa82af58f)) ```sh lms get https://huggingface.co/lmstudio-community/gemma-2-27b-it-GGUF@Q4_K_M -y ``` 3. Then run lms ls to see its name, and lms chat followed by that name Other versions: Q8_0 (~28.9 GB) `lms get https://huggingface.co/lmstudio-community/gemma-2-27b-it-GGUF@Q8_0 -y` · 4bit (~15.3 GB, Mac only) `lms get https://huggingface.co/mlx-community/gemma-2-27b-it-4bit -y` · 8bit (~28.9 GB, Mac only) `lms get https://huggingface.co/mlx-community/gemma-2-27b-it-8bit -y` ### Ollama 1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download) ```sh # Linux curl -fsSL https://ollama.com/install.sh | sh ``` 2. Run Gemma 2 27B IT (Q4_0, needs ~16.84 GB, from [ollama.com](https://ollama.com/library/gemma2/tags)) ```sh ollama run gemma2:27b ``` Other versions: Q8_0 (~33.18 GB) `ollama run gemma2:27b-instruct-q8_0` ### MLX (Mac) 1. Install MLX (Mac). Apple silicon Macs only, text only. ```sh # with uv uv tool install mlx-lm # or pip install mlx-lm ``` 2. Run Gemma 2 27B IT (4bit, needs ~15.3 GB, Mac only, from [huggingface.co/mlx-community](https://huggingface.co/mlx-community/gemma-2-27b-it-4bit/tree/a102efcc5341c47791dc07c9f9eece30bdc4bb9d)) ```sh mlx_lm.chat --model mlx-community/gemma-2-27b-it-4bit ``` 3. Or start an OpenAI-compatible server at http://localhost:8080/v1 ```sh mlx_lm.server --model mlx-community/gemma-2-27b-it-4bit ``` Other versions: 8bit (~28.9 GB, Mac only) `mlx_lm.chat --model mlx-community/gemma-2-27b-it-8bit` Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/gemma-2-27b-it) [interactive version](https://whatmodel.sh/model/gemma-2-27b) · [JSON](https://whatmodel.sh/api?format=json&all=1)