[whatmodel.sh](../../README.md) / Gemma 3 12B
# Gemma 3 12B
Google · released 2025-03 · 12.2B params · gemma
**score 52.3** ±0.6 · #331 of 732 · from 56 benchmark results across 16 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Gemma 3 12B among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃●█▄▅▅▄▄▂▁ #262 of 313* 2.4% 2
coding ▁▁▁▃▅●█▇▅▄▃▁ #302 of 469 8.7% 5
chat & writing ▁▁▁▃▆▄▆█●▇▇▃ #175 of 402* 4.2% 2
math ▁▁▃▅█●▄█▄▂▁▁ #208 of 431 15% 6
vision & documents ▁▁▂●▁▃▄▂▆▇█▄ #130 of 153 6.9% 9
agent tasks · #262 of 313 · 2.4% of its score
Beats [DeepSeek R1 0528 Qwen3 8B](deepseek-r1-0528-qwen3-8b.md), [Magistral Small 1.0 (2506)](magistral-small-2506.md), [Llama 4 Scout (109B-A17B)](llama-4-scout.md) here, though they rank higher overall. Trails [Granite 4.0 H 1B](granite-4-0-h-1b.md), [Granite 4.0 1B](granite-4-0-1b.md), [SmolLM3 3B](smollm3-3b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
BFCL V4 (December 2025 archive) 30.4% ▁▁▁▂█●▅▅▆▂▂▂ beats 31/77 hard · avg 31% 1.6% at its level
BFCL v3 50.6% ▁▁▁▁▁▃●█▅▄▁▁ beats 36/114 medium · avg 51% 0.8% blunt · at its level
coding · #302 of 469 · 8.7% of its score
Beats [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Solar Open 100B (102B-A12B)](solar-open-100b.md), [Qwen3-VL 30B-A3B Instruct](qwen3-vl-30b-a3b-instruct.md) here, though they rank higher overall. Trails [DeepSeek LLM 67B Chat](deepseek-llm-67b-chat.md), [WizardLM 70B V1.0](wizardlm-70b-v1-0.md), [Yi 1.5 9B Chat](yi-1-5-9b-chat.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval (maker) 85.4% ▁▁▁▁▂▂▆▆█●▂▁ beats 192/205 easy · avg 83% 2.8% sharp
VisEditBench strict pass rate (1,… 35.0% ▄··●·▄··▄█·▄ beats 1/7 hard · avg 33% 1.8% at its level
MBPP (maker) 73.0% ▁▁▁▄▅█▆▅●▁▂▁ beats 73/86 easy · avg 70% 1.7% at its level
LiveCodeBench (version unspecifie… 24.6% ▁···▁●▁█▇▃▄▁ beats 3/38 hard · avg 23% 1.6% at its level
BIRD-SQL (dev) (maker) 47.9% ▂·······▂▂●▆ beats 4/11 medium · avg 43% 0.7% at its level
chat & writing · #175 of 402 · 4.2% of its score
Beats [MiniCPM5 2B](minicpm5-2b.md), [Granite 4.2 30B](granite-4-2-30b.md), [Seed-OSS 36B Instruct](seed-oss-36b.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 1341.7 ▁▁▁▂▃▃▃▆●▇█▃ beats 179/385 avg 1260 3.6% blunt
IFEval (aggregation unspecified) … 88.9% ▁▂▄▂▂▄▆█●▅▂▁ beats 105/145 very easy · avg 88% 0.6% typical
math · #208 of 431 · 15% of its score
Beats [MiniMax-M2.1](minimax-m2-1.md), [DeepSeek V3 (Dec '24)](deepseek-v3.md), [Qwen3 235B-A22B (Apr 2025, hybrid)](qwen3-235b-a22b.md) here, though they rank higher overall. Trails [Granite 8B Code Base (4K)](granite-8b-code-base-4k.md), [Mixtral 8x7B v0.1 (base)](mixtral-8x7b-v0-1.md), [Olmo 3 7B Instruct](olmo-3-7b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 89.7% █····█··▄·▄● beats 7/7 very easy · avg 86% 5.7% sharp · several runs
MATH (subset unspecified) 83.8% █▄··▄●▄····▄ beats 4/7 easy · avg 80% 4.0% several runs
PolyMath (Marco evaluation) 35.5% ▄··▄██··●··▄ beats 6/7 hard · avg 29% 1.5% at its level
MathVista (testmini) (maker) 62.9% ▁··▁▄▄▆●▄▅█▄ beats 19/48 medium · avg 60% 1.3% at its level · small run
MGSM (Marco evaluation) 84.0% ▄·▄·▄▄▄·▄··● beats 6/7 easy · avg 69% 1.3% at its level
OTIS mock AIME 2024–2025 16.7% ▃▃▄▃●▃▇▇█▃▂▅ beats 56/197 very hard · avg 7.2% 1.2% sharp · noisy
vision & documents · #130 of 153 · 6.9% of its score
Trails [Qwen3.5 2B](qwen3-5-2b.md), [Qwen3-VL 2B Thinking](qwen3-vl-2b-thinking.md), [Pixtral 12B (2409)](pixtral-12b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
VisEditBench strict pass rate (1,… 35.0% ▄··●·▄··▄█·▄ beats 1/7 hard · avg 33% 1.8% at its level
MMMU (val) (maker) 59.6% ▁····▁▁▃●▆▆▅ beats 23/53 medium · avg 57% 1.3% at its level · small run
MathVista (testmini) (maker) 62.9% ▁··▁▄▄▆●▄▅█▄ beats 19/48 medium · avg 60% 1.3% at its level · small run
InfographicVQA (maker) 64.9% ▃·▃●█▃▃▃▃▃▃▃ beats 2/13 easy · avg 65% 0.9% at its level · small run
VQAv2 (val) (maker) 71.6% ▃▅▃●·▃▅·▃·▃▅ beats 6/13 easy · avg 72% 0.5% blunt · at its level · small run
ChartQA (maker) 75.7% ▁··▁▁▁▁▁●▅▅█ beats 9/44 easy · avg 77% 0.4% blunt · small run
DocVQA (maker) 87.1% ▁··●▁▁▁▇█▇▄▅ beats 2/28 very easy · avg 89% 0.3% blunt · small run
TextVQA (maker) 67.7% ▁▁▁●▅▁▃·▃▃▃█ beats 4/24 easy · avg 74% 0.1% blunt · at its level · small run
AI2D test (maker) 84.2% ▁▁▅▇▃●·▄·█·▃ beats 15/28 easy · avg 83% 0.1% blunt · small run
other tests (28)
test result vs the others in its run difficulty counts why
GPQA Diamond 39.5% ▁▁▁▁▂●▆▄▅▇█▅ beats 37/211 hard · avg 37% 14% sharp · several runs
MMLU 76.2% ▄···▄▄·▄●··█ beats 4/7 easy · avg 74% 7.2% at its level · several runs
Medmarks-V mean win rate (19 veri… 0.4379 ▁▄▃▃●▄█▃▅▁▁▁ beats 8/30 avg 0.418 6.7% sharp
MMLU-Pro 55.8% ▄····█·●█·▄▄ beats 3/7 medium · avg 54% 4.4% at its level · several runs
BIG-Bench Hard (maker) 85.7% ▁▁▁▁▂▄▅▆█▃●▁ beats 146/155 easy · avg 79% 3.3% sharp
Medmarks-V weighted mean win rate… 43.8% ▂▃▅▂●▃█▅█▅▃▆ beats 8/34 medium · avg 41% 2.9% blunt · at its level
Polish medical board exams, bias-… 53.3% ▄·▂▂▂··●▂▄·▄ beats 5/13 hard · avg 46% 2.4% at its level
MMLU-ProX-Lite (Marco) 51.3% █··▄··█··●·▄ beats 5/7 medium · avg 42% 2.3% at its level
SimpleQA (maker) 6.3% ▄▄▄●█▆███·▂█ beats 6/32 very hard · avg 6.8% 2.2% typical
Global-MMLU-Lite (maker) 69.5% ▂···▂·●▂▂·██ beats 2/12 medium · avg 64% 2.1% at its level
BRIDGE clinical text, zero-shot a… 37.3% ▁▁▁▁▁▂▂▃▄●█▃ beats 56/97 hard · avg 31% 2.1% blunt · at its level
MIMIC-CDM full information (4 pat… 74.4% ▄·······▄●·▄ beats 2/4 easy · avg 71% 1.7% typical
GlobalMMLU (Marco evaluation) 69.2% ▃··▃▃··▅···● beats 6/7 medium · avg 56% 1.7% at its level
Open Persian LLM Leaderboard v2 a… 61.7% ▂▄▂▃█▄▃●▃▅▂▂ beats 55/81 medium · avg 57% 1.6% blunt · at its level
MIMIC-CDM clinical decision makin… 66.2% █······█··●█ beats 2/3 medium · avg 60% 1.6% at its level
MMMLU (Marco evaluation) 69.4% ▄···▄·█·▄·▄● beats 6/7 medium · avg 59% 1.4% at its level
Big Bench Extra Hard micro avg (m… 16.3% ▄▄·●▄·▄·▄·▄▄ beats 2/8 hard · avg 18% 1.4% small run
AGIEval (Marco suite unspecified) 43.6% ▄··●█▄··▄··█ beats 1/7 medium · avg 46% 1.2% blunt · at its level
FACTS Grounding (maker) 75.8% ▄········▄·● beats 3/3 easy · avg 72% 1.1% typical
RULER 128k (maker) 57.1% ▃··▅●█▅█▇·▃▃ beats 8/33 medium · avg 59% 1.1% at its level · small run
Open PT-LLM Leaderboard average (… 76.4% ▃▁▁▁▁▁▁▁▂█●▁ beats 233/276 medium · avg 72% 0.8% at its level · noisy
EuroEval HellaSwag (English), tes… 80.9% ▃·▁▁▂▁▂▅█●▄▁ beats 75/100 medium · avg 67% 0.7% sharp · at its level · noisy
WMT24++ (chrF) (maker) 51.6% █······█··●█ beats 2/3 medium · avg 52% 0.6% blunt · at its level
EuroEval Life in the UK (English … 85.0% ▁▁▁▁▁▁▁▂▄●▄▃ beats 99/146 easy · avg 82% 0.4% noisy
WMT24++ (maker) 81.6% █···█·●··███ beats 2/5 easy · avg 73% 0.2% blunt · at its level
Vectara hallucination leaderboard… 97.2% ▁▁▂▃▄▄█●▃▂▂▁ beats 110/140 very easy · avg 94% 0.2% blunt · maxed out · noisy
Chess Puzzles (Epoch) 0.0% ●▁▁▁▁▂▃▃▃▂▁▁ beats 28/137 very hard · avg 5.8% 0.2% sharp · too hard for it · noisy
Vectara hallucination leaderboard… 95.6% ▁▂▁▂▅█▂▃▄▃●▁ beats 94/98 very easy · avg 90% 0.0% blunt
source results share of the score
Google DeepMind Gemma 3 270M IT model c… 19 30% ▇▇▇▇
Marco-MoE: Open Multilingual Mixture-of… 11 28% ▇▇▇
Medmarks: A Comprehensive Open-Source L… 2 10% ▇
Gemma Team (submitted / maker-reported) 6 7% ▇
Epoch AI 3 6% ▇
LMArena leaderboard dataset 1 4% ▇
MIMIC Clinical Decision Making leaderbo… 2 3% ▇
Reassessing High-Performing LLMs on Pol… 1 2% ▇
Berkeley Function Calling Leaderboard r… 2 2% ▇
BRIDGE Medical Leaderboard 1 2% ▇
VisEditBench: Can Vision-Language Model… 1 2% ▇
Open Persian LLM Leaderboard v2 1 2% ▇
EuroEval leaderboards 2 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
TranslateGemma Technical Report (submit… 1 0% ▇
Vectara hallucination leaderboard 2 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 52.3
Licences: Google DeepMind Gemma 3 270M IT model card ([CC-BY-4.0](../../licenses/cards-final-NOTICE.txt)) · Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling ([CC BY 4.0](../../licenses/cards-final-NOTICE.txt)) · Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks ([CC BY 4.0](../../licenses/paper-2605.01417v1-CC-BY-4.0.txt)) · Gemma Team ([CC BY 4.0](../../licenses/report-2503.19786v1.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · MIMIC Clinical Decision Making leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance? ([CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · BRIDGE Medical Leaderboard ([MIT](../../licenses/bridge-medical-leaderboard-MIT.txt)) · VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback? ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Open Persian LLM Leaderboard v2 ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · EuroEval leaderboards ([MIT](https://github.com/EuroEval/leaderboards/blob/1a77f1d2f8ba1ba05ab73287fb090d76bf41b2e0/LICENSE)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · TranslateGemma Technical Report ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
### Through an API
api (DeepInfra) · $0.075/M blended · $0.05 in / $0.15 out per 1M tokens · [deepinfra.com](https://deepinfra.com/google/gemma-3-12b-it)
[price source](https://github.com/BerriAI/litellm/blob/c19ce71bcdad234ca87a7af10cee45375ce811a8/model_prices_and_context_window.json): USD per 1M text tokens, standard uncached input and output. Product: Serverless text generation. API model: google/gemma-3-12b-it. Settings: service tier=standard, batch=no, cache=uncached input. Additional/excluded charges: Tools, cache storage, priority service and other modalities excluded.. Currency: USD. · [context source](https://github.com/BerriAI/litellm/blob/c19ce71bcdad234ca87a7af10cee45375ce811a8/model_prices_and_context_window.json): provider maximum input tokens; not an intrinsic model maximum or legacy max\_tokens
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run Gemma 3 12B (Q4_K_M, needs ~7.3 GB, from [huggingface.co/ggml-org](https://huggingface.co/ggml-org/gemma-3-12b-it-GGUF/tree/ec0cbabd8dbff316f659876a50202295c3c4a314))
```sh
llama cli -hf ggml-org/gemma-3-12b-it-GGUF:Q4_K_M
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf ggml-org/gemma-3-12b-it-GGUF:Q4_K_M
```
Other versions: Q8_0 (~12.5 GB) `llama cli -hf ggml-org/gemma-3-12b-it-GGUF:Q8_0`
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run Gemma 3 12B (Q4_K_M, needs ~7.3 GB, from [huggingface.co/ggml-org](https://huggingface.co/ggml-org/gemma-3-12b-it-GGUF/tree/ec0cbabd8dbff316f659876a50202295c3c4a314))
```sh
lms get https://huggingface.co/ggml-org/gemma-3-12b-it-GGUF@Q4_K_M -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
Other versions: Q8_0 (~12.5 GB) `lms get https://huggingface.co/ggml-org/gemma-3-12b-it-GGUF@Q8_0 -y` · 4bit-dwq (~7.2 GB, Mac only) `lms get https://huggingface.co/mlx-community/gemma-3-12b-it-4bit-DWQ -y`
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run Gemma 3 12B (Q4_K_M, needs ~7.82 GB, from [ollama.com](https://ollama.com/library/gemma3/tags))
```sh
ollama run gemma3:12b
```
Other versions: qat (~7.82 GB) `ollama run gemma3:12b-it-qat` · Q8_0 (~15.14 GB) `ollama run gemma3:12b-it-q8_0`
### MLX (Mac)
1. Install MLX (Mac). Apple silicon Macs only, text only.
```sh
# with uv
uv tool install mlx-lm
# or
pip install mlx-lm
```
2. Run Gemma 3 12B (4bit-dwq, needs ~7.2 GB, Mac only, from [huggingface.co/mlx-community](https://huggingface.co/mlx-community/gemma-3-12b-it-4bit-DWQ/tree/f227bac513e1b53066f08c094d6eee4ed89a6e13))
```sh
mlx_lm.chat --model mlx-community/gemma-3-12b-it-4bit-DWQ
```
3. Or start an OpenAI-compatible server at http://localhost:8080/v1
```sh
mlx_lm.server --model mlx-community/gemma-3-12b-it-4bit-DWQ
```
Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/gemma-3-12b-it)
## Published speed measurements
Reference results for the hardware, engine and settings below. These are not verified speeds of the downloads above or predictions for your machine.
hardware precision speed engine and settings
Apple M2 Pro, 32 GB q4_k_m 20.88 tok/s ollama (version not recorded) · l…
Apple M3 Pro, 18 GB q4_k_m 12.49 tok/s ollama (version not recorded) · l…
Apple M3 Pro, 18 GB q4_k_m 13.28 tok/s ollama (version not recorded) · l…
Apple M4, 16 GB qat 11.12 tok/s ollama (version not recorded) · l…
Each speed links to its source, which has the full settings.
## Quantization evidence
How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent. They don't describe every download of that precision.
precision test full → quantized kept
NVFP4 (weights and activation… GSM8K chain-of-thought exact matc… 0.825 → 0.789 95.7%
NVFP4 (weights and activation… MMLU 5-shot (500-example subset; … 0.62 → 0.58 93.4%
W4A16 (RedHatAI gemma-3-12b-i… GSM8K chain-of-thought exact matc… 0.825 → 0.82 99.4%
W4A16 (RedHatAI gemma-3-12b-i… MMLU 5-shot (500-example subset; … 0.62 → 0.59 95.2%
Each kept value links to its source.
[interactive version](https://whatmodel.sh/model/gemma-3-12b) · [JSON](https://whatmodel.sh/api?format=json&all=1)