[whatmodel.sh](../../README.md) / Gemma 7B (base)
# Gemma 7B (base)
Google · released 2024-02 · 8.54B params · gemma
**score 37.1** ±0.5 · #607 of 732 · from 37 benchmark results across 9 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Gemma 7B (base) among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁▁●▅▇█▇▅▄▃▁ #425 of 469 70% 7
math ▁▁▃▅█●▄█▄▂▁▁ #227 of 431 5.6% 2
coding · #425 of 469 · 70% of its score
Beats [DeepSeek R1 Distill Qwen 7B](deepseek-r1-distill-qwen-7b.md), [Qwen3.5 2B](qwen3-5-2b.md), [DeepSeek R1 Distill Llama 8B](deepseek-r1-distill-llama-8b.md) here, though they rank higher overall. Trails [Llama 3.2 1B Instruct](llama-3-2-1b.md), [SmolLM2 1.7B Instruct](smollm2-1-7b.md), [Granite 3B Code Base (2K)](granite-3b-code-base-2k.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 35.4% ▁·▁▁▃●▆▅█▃▁▁ beats 20/99 easy · avg 76% 39% sharp · at its level · several runs
HumanEval+ 28.7% ▁·▁▁▁▃●▄▄█▃▁ beats 17/99 easy · avg 70% 16% sharp · at its level
HumanEvalSynthesize (HumanEvalPac… 34.4% ▁·▂▁▂▆●█▄▄▁▂ beats 12/35 medium · avg 62% 4.5% at its level
MBPP+ 43.4% ▁···▁▁●▅▇█▆▂ beats 9/75 easy · avg 66% 3.4% at its level
MBPP (maker) 44.4% ▁▁▁▄●█▆▅▄▁▂▁ beats 25/86 easy · avg 71% 2.8% at its level · several runs
HumanEvalExplain (HumanEvalPack, … 15.0% ▄▇●▃▄▇▄▁▇▃·▁ beats 9/35 medium · avg 46% 2.4% typical
HumanEvalFix (HumanEvalPack, 6-la… 7.4% ▁·▁▁●██▄█▁▅▅ beats 3/35 medium · avg 51% 2.0% sharp
math · #227 of 431 · 5.6% of its score
Beats [MiniMax-M2.1](minimax-m2-1.md), [DeepSeek V3 (Dec '24)](deepseek-v3.md), [GPT-4o (2024-08-06)](gpt-4o-2024-08-06.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
GSM8K 53.3% ▄··▄···▄··●▄ beats 4/5 easy · avg 78% 3.5% sharp · at its level · small run · seve…
MATH 24.1% ▃·▃····▃···● beats 5/5 medium · avg 49% 2.1% small run · several runs
other tests (22)
test result vs the others in its run difficulty counts why
EvoEval (overall) 23.0% ▂▂▃●▄█▁▄▅▃▁▃ beats 7/42 medium · avg 52% 6.2% typical
CommonsenseQA (maker) 71.3% █·····▂▂▃▂●▄ beats 31/38 very easy · avg 94% 2.2% sharp · at its level · small run
MedQA (USMLE, 4 options) 48.7% ▁▁··▄▁▁··▄●█ beats 10/17 medium · avg 71% 2.1% sharp · at its level
BIG-Bench Hard (maker) 55.1% ▁▁▁▁▂▄▅●█▃▃▁ beats 76/155 easy · avg 79% 1.9% sharp · at its level · small run
ARC Challenge (maker) 53.2% ▄▁▁▂▁▄█●▃▂▃▂ beats 95/140 easy · avg 78% 1.6% sharp · at its level · small run
MMLU (maker) 64.3% ▁▁▁▁▁▂▂▅●█▆▄ beats 102/214 easy · avg 81% 1.2% at its level · small run
MedMCQA 49.3% ▂▂··▄▄▂▂·▄●█ beats 12/17 medium · avg 63% 1.0% at its level
OpenBookQA (maker) 52.8% ▅▁▁▁▂▂▃●▃▁▃▁ beats 78/107 medium · avg 68% 1.0% at its level · small run
WinoGrande (maker) 72.3% ▁▁▁▁▂▁▁▅▇●▃▁ beats 102/131 easy · avg 84% 0.8% at its level · small run
Natural Questions (closed book) (… 23.0% ▂▁▁▁▂▄▄▃▅●▅▁ beats 48/68 medium · avg 40% 0.8% small run
AGIEval English (Gemma report) (m… 41.7% ▄··········● beats 2/2 medium · avg 62% 0.7% at its level · small run
TriviaQA (maker) 63.4% ▁▁▂▂▄▄▃▆▅●▅▂ beats 54/80 easy · avg 81% 0.7% at its level · small run
HellaSwag (maker) 81.2% ▁▁▁▂▂▂▄▆█▇●▂ beats 117/135 easy · avg 89% 0.6% at its level · small run
Open PT-LLM Leaderboard average (… 64.2% ▃▁▁▁▁▁▁▁▂●▅▁ beats 109/276 medium · avg 72% 0.6% at its level · noisy
BoolQ (maker) 83.2% ▁····▁▂█▆▅●▄ beats 84/103 easy · avg 89% 0.6% at its level · small run
EuroEval Life in the UK (English … 84.9% ▁▁▁▁▁▁▁▂▄●▄▃ beats 98/146 easy · avg 82% 0.5% at its level · noisy
EuroEval HellaSwag (English), tes… 33.5% ▃·▁▁●▁▂▅█▇▄▁ beats 17/100 medium · avg 67% 0.4% sharp · noisy
Social IQA (maker) 51.8% ▁·▁██▄●▁▂▃▇▁ beats 38/59 medium · avg 63% 0.4% blunt · at its level · small run
ARC Easy (maker) 81.5% ▁▃▂▄▆█▅●▁··▁ beats 93/100 easy · avg 87% 0.4% blunt · at its level · small run
PIQA (maker) 81.2% ▁▁▁▂▂▃▅█●▃▁▁ beats 101/123 easy · avg 86% 0.2% blunt · at its level · small run
PubMedQA 75.6% ▂·▂▆·▄▂▄▄▂●▂ beats 16/17 medium · avg 75% 0.1% blunt · at its level
TruthfulQA MC2 44.9% ▅▃█●▁▂▅···▁▁ beats 19/31 medium · avg 57% 0.1% blunt · at its level
source results share of the score
EvalPlus / EvalPerf published evaluatio… 3 41% ▇▇▇▇▇
Gemma report (submitted / maker-reporte… 18 20% ▇▇
Top Leaderboard Ranking = Top Coding Pr… 2 18% ▇▇
Granite Code Models: A Family of Open F… 5 11% ▇
CodeGemma: Open Code Models Based on Ge… 2 5% ▇
Open Medical-LLM Leaderboard results 3 3% ▇
EuroEval leaderboards 2 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
Hallucinations Leaderboard results 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 37.1
Licences: EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · Gemma report ([CC BY 4.0](../../licenses/CC-BY-4.0.txt)) · Top Leaderboard Ranking = Top Coding Proficiency ([CC BY 4.0](../../licenses/paper-2403.19114v1-CC-BY-4.0.txt)) · Granite Code Models: A Family of Open Foundation Models for Code Intelligence ([CC BY 4.0](../../licenses/paper-2405.04324v1-CC-BY-4.0.txt)) · CodeGemma: Open Code Models Based on Gemma ([CC BY 4.0](../../licenses/paper-2406.11409v2-CC-BY-4.0.txt)) · Open Medical-LLM Leaderboard results ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · EuroEval leaderboards ([MIT](https://github.com/EuroEval/leaderboards/blob/1a77f1d2f8ba1ba05ab73287fb090d76bf41b2e0/LICENSE)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Hallucinations Leaderboard results ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
## How to run it
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run Gemma 7B (base) (Q4_K_M, needs ~5.3 GB, from [huggingface.co/MaziyarPanahi](https://huggingface.co/MaziyarPanahi/gemma-7b-GGUF/tree/84a72b826b704384e4684ccd0eaa747123e2c004))
```sh
llama cli -hf MaziyarPanahi/gemma-7b-GGUF:Q4_K_M
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf MaziyarPanahi/gemma-7b-GGUF:Q4_K_M
```
Other versions: Q8_0 (~9.1 GB) `llama cli -hf MaziyarPanahi/gemma-7b-GGUF:Q8_0`
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run Gemma 7B (base) (Q4_K_M, needs ~5.3 GB, from [huggingface.co/MaziyarPanahi](https://huggingface.co/MaziyarPanahi/gemma-7b-GGUF/tree/84a72b826b704384e4684ccd0eaa747123e2c004))
```sh
lms get https://huggingface.co/MaziyarPanahi/gemma-7b-GGUF@Q4_K_M -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
Other versions: Q8_0 (~9.1 GB) `lms get https://huggingface.co/MaziyarPanahi/gemma-7b-GGUF@Q8_0 -y`
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run Gemma 7B (base) (Q4_K_M, needs ~5.5 GB, from [ollama.com](https://ollama.com/library/gemma/tags))
```sh
ollama run gemma:7b-text-q4_K_M
```
Other versions: Q8_0 (~9.1 GB) `ollama run gemma:7b-text-q8_0`
Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/gemma-7b)
[interactive version](https://whatmodel.sh/model/gemma-7b-base) · [JSON](https://whatmodel.sh/api?format=json&all=1)