[whatmodel.sh](../../README.md) / Gemma 3 4B
# Gemma 3 4B
Google · released 2025-03 · 4.3B params · gemma
**score 45.3** ±0.6 · #467 of 732 · from 57 benchmark results across 17 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Gemma 3 4B among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁●▆█▄▅▅▄▄▂▁ #298 of 313* 2.3% 2
coding ▁▁▁▃●▇█▇▅▄▃▁ #371 of 469 11% 5
chat & writing ▁▁▁▃▆▄▆●▆▇▇▃ #227 of 402 7.2% 3
math ▁▁▃▅●▆▄█▄▂▁▁ #268 of 431 11% 6
vision & documents ●▁▂▂▁▃▄▂▆▇█▄ #152 of 153 8.0% 10
agent tasks · #298 of 313 · 2.3% of its score
Beats [DBRX Instruct](dbrx-instruct.md), [QwQ-32B-Preview](qwq-32b-preview.md), [DeepSeek-Coder-V2 Lite Instruct (16B-A2.4B)](deepseek-coder-v2-lite-instruct.md) here, though they rank higher overall. Trails [Granite 4.0 H 350M](granite-4-0-h-350m.md), [ChatGLM3-6B](chatglm3-6b.md), [Qwen3 0.6B](qwen3-0-6b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
BFCL V4 (December 2025 archive) 19.6% ▁▁▁●██▅▅▆▂▂▂ beats 5/77 hard · avg 31% 1.5% typical
BFCL v3 39.6% ▁▁▁▁●▃▅█▅▄▁▁ beats 14/114 medium · avg 51% 0.9% blunt · at its level
coding · #371 of 469 · 11% of its score
Beats [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md), [Qwen3-VL 2B Thinking](qwen3-vl-2b-thinking.md), [DeepSeek R1 Distill Qwen 7B](deepseek-r1-distill-qwen-7b.md) here, though they rank higher overall. Trails [Qwen2.5 1.5B Instruct](qwen2-5-1-5b.md), [Code Tulu 2 13B](code-tulu-2-13b.md), [Phi-3-mini-128k-instruct (Apr 2024)](phi-3-mini-128k-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval (maker) 71.3% ▁▁▁▁▂▂▆▆●▅▂▁ beats 141/205 easy · avg 83% 4.7% sharp · at its level
MBPP (maker) 63.2% ▁▁▁▄▅█▆●▄▁▂▁ beats 58/86 easy · avg 70% 2.2% at its level
VisEditBench strict pass rate (1,… 21.3% ●··█·▄··▄█·▄ beats 0/7 hard · avg 33% 1.8% at its level
LiveCodeBench (version unspecifie… 12.6% ▁···●▁▁█▇▃▄▁ beats 1/38 hard · avg 23% 1.5% typical
BIRD-SQL (dev) (maker) 36.3% ▂·······●▂█▆ beats 1/11 medium · avg 43% 0.7% at its level
chat & writing · #227 of 402 · 7.2% of its score
Beats [Granite 4.2 8B](granite-4-2-8b.md), [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md), [Olmo 3.1 32B Think](olmo-3-1-32b-think.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 1303.2 ▁▁▁▂▃▃▃●▆▇█▃ beats 130/385 avg 1260 4.0% blunt
IFEval mean of four measures 76.8% █··●▄·▄····▄ beats 2/5 easy · avg 84% 2.3% typical
IFEval (aggregation unspecified) … 90.2% ▁▂▄▂▂▄▆█●▅▂▁ beats 118/145 very easy · avg 88% 0.8% typical
math · #268 of 431 · 11% of its score
Beats [GPT-4o (2024-11-20)](gpt-4o-2024-11-20.md), [GPT-4 Turbo (2024-04-09)](gpt-4-turbo-2024-04-09.md), [Molmo 2 8B](molmo-2-8b.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K (maker) 89.2% ▂▂▂▂▃▄▅▇██●▄ beats 176/206 very easy · avg 93% 4.7% sharp
MATH (subset unspecified) (maker) 75.6% ▂▄▄▃█▇●▆▄▃▄▁ beats 48/80 easy · avg 82% 3.0% at its level
MathVista (testmini) (maker) 50.0% ▁··▁▄●▆▇▄▅█▄ beats 8/48 medium · avg 60% 1.5% at its level · small run
MATH-500 73.2% ▃····▃·●···▃ beats 2/5 easy · avg 79% 1.4% at its level
OTIS mock AIME 2024–2025 7.5% ▃▃▄●▃▃▇▇█▃▂▅ beats 43/197 very hard · avg 7.2% 0.3% sharp · too hard for it · noisy
GSM8K loose substring accuracy 89.9% █·█··███···● beats 5/5 easy · avg 83% 0.2% blunt
vision & documents · #152 of 153 · 8.0% of its score
Trails [Qwen3.5 0.8B](qwen3-5-0-8b.md), [Qwen2-VL 2B Instruct](qwen2-vl-2b.md), [Phi-3.5-vision-instruct](phi-3-5-vision-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
VisEditBench strict pass rate (1,… 21.3% ●··█·▄··▄█·▄ beats 0/7 hard · avg 33% 1.8% at its level
MathVista (testmini) (maker) 50.0% ▁··▁▄●▆▇▄▅█▄ beats 8/48 medium · avg 60% 1.5% at its level · small run
MMMU (val) (maker) 48.8% ▁····▁▁●█▆▆▅ beats 7/53 medium · avg 57% 1.4% at its level · small run
InfographicVQA (maker) 50.0% ●·▃▅█▃▃▃▃▃▃▃ beats 0/13 easy · avg 65% 1.1% at its level · small run
VQAv2 (val) (maker) 62.4% ●▅▃█·▃▅·▃·▃▅ beats 0/13 easy · avg 72% 0.6% blunt · at its level · small run
ChartQA (maker) 68.8% ▁··▁▁▁▁●▂▅▅█ beats 5/44 easy · avg 77% 0.5% blunt · at its level · small run
DocVQA (maker) 75.8% ●··▃▁▁▁▇█▇▄▅ beats 0/28 very easy · avg 89% 0.4% blunt · small run
VMMU (Vietnamese multimodal under… 25.4% ●······▃·▅·█ beats 0/7 very hard · avg 36% 0.4% noisy
TextVQA (maker) 57.8% ●▁▁▄▅▁▃·▃▃▃█ beats 0/24 easy · avg 74% 0.2% blunt · at its level · small run
AI2D test (maker) 74.8% ▁▁▅●▃▇·▄·█·▃ beats 7/28 easy · avg 83% 0.2% blunt · small run
other tests (29)
test result vs the others in its run difficulty counts why
GPQA Diamond 23.2% ●▁▁▁▂▄▆▄▅▇█▅ beats 4/211 hard · avg 37% 8.5% sharp · small run · several runs
Medmarks-V mean win rate (19 veri… 0.3214 ●▄▃▃▇▄█▃▅▁▁▁ beats 0/30 avg 0.418 7.5% sharp
MMLU 58.4% ●█···█·····▄ beats 0/5 medium · avg 69% 7.2% at its level · several runs
BIG-Bench Hard (maker) 72.2% ▁▁▁▁▂▄▅▆●▃▃▁ beats 124/155 easy · avg 79% 4.6% sharp · at its level
MMLU-Pro 34.8% ███···●█···█ beats 3/5 medium · avg 42% 3.5% at its level · several runs
Medmarks-V weighted mean win rate… 32.1% ●▃▅▂▆▃█▅█▅▃▆ beats 0/34 medium · avg 41% 3.1% blunt · at its level
EQ-Bench 3 (emotional intelligenc… 1118.2 ▁·▂▁▄●▅▃█▄▁▁ beats 15/62 avg 1034 2.5% typical
Global-MMLU-Lite (maker) 54.5% ▂···●·▂▂▂·██ beats 1/12 medium · avg 64% 2.4% at its level
MIMIC-CDM full information (4 pat… 69.4% ▄·······●█·▄ beats 1/4 easy · avg 71% 2.3% at its level
BRIDGE clinical text, zero-shot a… 28.6% ▁▁▁▁▁▂▂●▄▆█▃ beats 25/97 hard · avg 31% 2.1% blunt · at its level
IFBench (aggregation unspecified) 23.5% █··██●··█··█ beats 3/5 hard · avg 31% 2.0% typical
MIMIC-CDM clinical decision makin… 50.8% █······●··██ beats 1/3 medium · avg 60% 1.9% at its level
Open Persian LLM Leaderboard v2 a… 49.1% ▂▄▂▃●▄▃▆▃▅▂▂ beats 27/81 medium · avg 57% 1.9% blunt · at its level
Multi-IF three-turn average 66.6% ▄··█▄··●···▄ beats 4/5 easy · avg 70% 1.7% at its level
SimpleQA (maker) 4.0% ▄●▄▆█▆███·▂█ beats 3/32 very hard · avg 6.8% 1.6% too hard for it
MMMLU seven languages (Liquid) 50.1% ▄·●····█···▄ beats 2/5 medium · avg 59% 1.6% at its level
FACTS Grounding (maker) 70.1% ▄········●·█ beats 1/3 easy · avg 72% 1.4% at its level
Big Bench Extra Hard micro avg (m… 11.0% ▄●·█▄·▄·▄·▄▄ beats 1/8 hard · avg 18% 1.3% small run
RULER 128k (maker) 46.8% ▃··●▄█▅█▇·▃▃ beats 4/33 medium · avg 59% 1.3% at its level · small run
MGSM six languages, loose (Liquid) 87.3% ▄·▄·█···▄··● beats 5/5 easy · avg 81% 1.1% at its level
Open PT-LLM Leaderboard average (… 67.3% ▃▁▁▁▁▁▁▁▂●▅▁ beats 134/276 medium · avg 72% 1.0% at its level · noisy
EuroEval HellaSwag (English), tes… 57.9% ▃·▁▁▂▁▂●█▇▄▁ beats 30/100 medium · avg 67% 0.8% sharp · at its level · noisy
WMT24++ (chrF) (maker) 46.8% █······●··██ beats 1/3 medium · avg 52% 0.6% blunt · at its level
EuroEval Life in the UK (English … 77.7% ▁▁▁▁▁▁▁▂●█▄▃ beats 59/146 easy · avg 82% 0.6% at its level · noisy
GSM-Plus 68.4% █·····███·█● beats 5/5 easy · avg 71% 0.4% blunt · at its level
Vectara hallucination leaderboard… 96.3% ▁▁▂▃▄▄█●▃▂▂▁ beats 92/140 very easy · avg 94% 0.3% blunt · maxed out · noisy
WMT24++ (maker) 77.2% ●···█·█··███ beats 0/5 easy · avg 73% 0.3% blunt · at its level
Chess Puzzles (Epoch) 0.0% ●▁▁▁▁▂▃▃▃▂▁▁ beats 30/137 very hard · avg 5.8% 0.1% sharp · too hard for it · noisy
Vectara hallucination leaderboard… 93.6% ▁▂▁▂▅█▂▃●▃▁▁ beats 74/98 very easy · avg 90% 0.0% blunt
source results share of the score
Google DeepMind Gemma 3 270M IT model c… 19 38% ▇▇▇▇▇
LFM2 Technical Report 11 19% ▇▇
Medmarks: A Comprehensive Open-Source L… 2 11% ▇
Gemma Team (submitted / maker-reported) 6 8% ▇
MIMIC Clinical Decision Making leaderbo… 2 4% ▇
LMArena leaderboard dataset 1 4% ▇
Epoch AI 3 3% ▇
EQ-Bench 3 canonical leaderboard results 1 2% ▇
Berkeley Function Calling Leaderboard r… 2 2% ▇
BRIDGE Medical Leaderboard 1 2% ▇
Open Persian LLM Leaderboard v2 1 2% ▇
VisEditBench: Can Vision-Language Model… 1 2% ▇
EuroEval leaderboards 2 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
VMMU: A Vietnamese Multitask Multimodal… 1 0% ▇
Vectara hallucination leaderboard 2 0% ▇
TranslateGemma Technical Report (submit… 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 45.3
Licences: Google DeepMind Gemma 3 270M IT model card ([CC-BY-4.0](../../licenses/cards-final-NOTICE.txt)) · LFM2 Technical Report ([CC BY 4.0](../../licenses/cards-gap-NOTICE.txt)) · Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks ([CC BY 4.0](../../licenses/paper-2605.01417v1-CC-BY-4.0.txt)) · Gemma Team ([CC BY 4.0](../../licenses/report-2503.19786v1.txt)) · MIMIC Clinical Decision Making leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · EQ-Bench 3 canonical leaderboard results ([MIT](../../licenses/eqbench3-MIT.txt)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · BRIDGE Medical Leaderboard ([MIT](../../licenses/bridge-medical-leaderboard-MIT.txt)) · Open Persian LLM Leaderboard v2 ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback? ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · EuroEval leaderboards ([MIT](https://github.com/EuroEval/leaderboards/blob/1a77f1d2f8ba1ba05ab73287fb090d76bf41b2e0/LICENSE)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · TranslateGemma Technical Report ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/))
## How to run it
### Through an API
api (DeepInfra) · $0.063/M blended · $0.05 in / $0.1 out per 1M tokens · [deepinfra.com](https://deepinfra.com/google/gemma-3-4b-it)
[price source](https://github.com/BerriAI/litellm/blob/c19ce71bcdad234ca87a7af10cee45375ce811a8/model_prices_and_context_window.json): USD per 1M text tokens, standard uncached input and output. Product: Serverless text generation. API model: google/gemma-3-4b-it. Settings: service tier=standard, batch=no, cache=uncached input. Additional/excluded charges: Tools, cache storage, priority service and other modalities excluded.. Currency: USD. · [context source](https://github.com/BerriAI/litellm/blob/c19ce71bcdad234ca87a7af10cee45375ce811a8/model_prices_and_context_window.json): provider maximum input tokens; not an intrinsic model maximum or legacy max\_tokens
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run Gemma 3 4B (Q4_K_M, needs ~2.5 GB, from [huggingface.co/ggml-org](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF/tree/d0976223747697cb51e056d85c532013931fe52e))
```sh
llama cli -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
```
Other versions: Q8_0 (~4.1 GB) `llama cli -hf ggml-org/gemma-3-4b-it-GGUF:Q8_0`
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run Gemma 3 4B (Q4_K_M, needs ~2.5 GB, from [huggingface.co/ggml-org](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF/tree/d0976223747697cb51e056d85c532013931fe52e))
```sh
lms get https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF@Q4_K_M -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
Other versions: Q8_0 (~4.1 GB) `lms get https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF@Q8_0 -y` · 4bit (~2.2 GB, Mac only) `lms get https://huggingface.co/Outlier-Ai/gemma-3-4b-it-MLX-4bit -y` · 8bit (~4.1 GB, Mac only) `lms get https://huggingface.co/Oscilla/gemma-3-4b-it-mlx-8Bit -y`
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run Gemma 3 4B (Q4_K_M, needs ~3.08 GB, from [ollama.com](https://ollama.com/library/gemma3/tags))
```sh
ollama run gemma3:4b
```
Other versions: qat (~3.08 GB) `ollama run gemma3:4b-it-qat` · Q8_0 (~5.66 GB) `ollama run gemma3:4b-it-q8_0`
### MLX (Mac)
1. Install MLX (Mac). Apple silicon Macs only, text only.
```sh
# with uv
uv tool install mlx-lm
# or
pip install mlx-lm
```
2. Run Gemma 3 4B (4bit, needs ~2.2 GB, Mac only, from [huggingface.co/Outlier-Ai](https://huggingface.co/Outlier-Ai/gemma-3-4b-it-MLX-4bit/tree/d1887210715eb17b65b4e6eed86479bb7d24d744))
```sh
mlx_lm.chat --model Outlier-Ai/gemma-3-4b-it-MLX-4bit
```
3. Or start an OpenAI-compatible server at http://localhost:8080/v1
```sh
mlx_lm.server --model Outlier-Ai/gemma-3-4b-it-MLX-4bit
```
Other versions: 8bit (~4.1 GB, Mac only) `mlx_lm.chat --model Oscilla/gemma-3-4b-it-mlx-8Bit`
Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/gemma-3-4b-it)
## Published speed measurements
Reference results for the hardware, engine and settings below. These are not verified speeds of the downloads above or predictions for your machine.
hardware precision speed engine and settings
NVIDIA DGX Spark (GB10) QAT Q4_0 81.05 tok/s llama.cpp 11fb327bf (7941), CUDA …
NVIDIA DGX Spark (GB10) QAT Q4_0 76.37 tok/s llama.cpp 11fb327bf (7941), CUDA …
NVIDIA DGX Spark (GB10) QAT Q4_0 62.28 tok/s llama.cpp 11fb327bf (7941), CUDA …
Apple M2 Ultra (GPU core co… QAT Q4_0 134.28 tok/s llama.cpp b828e18c7 (7948), Metal…
Apple M2 Ultra (GPU core co… QAT Q4_0 133.11 tok/s llama.cpp b828e18c7 (7948), Metal…
Apple M2 Ultra (GPU core co… QAT Q4_0 113.14 tok/s llama.cpp b828e18c7 (7948), Metal…
NVIDIA DGX Spark (GB10) QAT Q4_0 71.61 tok/s llama.cpp 11fb327bf (7941), CUDA …
NVIDIA DGX Spark (GB10) QAT Q4_0 70.89 tok/s llama.cpp 11fb327bf (7941), CUDA …
Apple M2 Ultra (GPU core co… QAT Q4_0 125.85 tok/s llama.cpp b828e18c7 (7948), Metal…
Apple M2 Ultra (GPU core co… QAT Q4_0 125 tok/s llama.cpp b828e18c7 (7948), Metal…
Intel Alder Lake-P GT2 [Iri… q4_k_m 8.21 tok/s ollama (version not recorded) · l…
Intel Alder Lake-P GT2 [Iri… q4_k_m 7.88 tok/s ollama (version not recorded) · l…
Intel Alder Lake-P GT2 [Iri… q4_k_m 8.93 tok/s ollama (version not recorded) · l…
Intel CoffeeLake-S GT2 [UHD… q4_k_m 5.98 tok/s ollama (version not recorded) · l…
Each speed links to its source, which has the full settings.
[interactive version](https://whatmodel.sh/model/gemma-3-4b) · [JSON](https://whatmodel.sh/api?format=json&all=1)