[whatmodel.sh](../../README.md) / Pythia 12B
# Pythia 12B
EleutherAI · released 2023-02 · 11.8B params · apache-2.0
**score 24.7** ±0.9 · #688 of 732 · from 22 benchmark results across 7 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Pythia 12B among every model on that task or test, worst to best.
other tests (16)
test result vs the others in its run difficulty counts why
ARC Challenge 35.1% ▁·▁··▁▁●▂█▆▃ beats 8/26 easy · avg 82% 32% sharp · several runs
WinoGrande 64.0% ▁····▁··▁▂●▄ beats 8/26 easy · avg 86% 13% at its level · several runs
ARC Easy 63.7% ▁▁·▁▁▄▅●█▅▁▃ beats 11/26 easy · avg 87% 9.4% blunt · at its level · several runs
SciQ 90.2% ▁·▁·▁·▃●▃█▄▁ beats 8/26 very easy · avg 97% 6.6% blunt · several runs
PIQA 76.3% ▁·▁··▁▁▃●▅█▂ beats 11/26 easy · avg 87% 6.4% blunt · at its level · several runs
BoolQ 67.3% ▂▃▂▅▆●▅▆█▂▂▂ beats 11/26 easy · avg 88% 5.5% at its level
ImageNetVC visual commonsense, te… 50.0% ▂·▄·▂█▄▄●▄▂▆ beats 13/21 easy · avg 74% 5.2% blunt · at its level
HellaSwag 67.4% ▃···▁▇●█▇▃▁▁ beats 10/26 very easy · avg 91% 4.6% at its level
LAMBADA (OpenAI version), last-wo… 70.6% ▁··▂▁▁▁▂▆●▅▃ beats 15/26 very easy · avg 87% 4.4% blunt · at its level · several runs
F2WTQ free-form table QA, 2-shot … 7.8% ●···█·▄█··▄▄ beats 0/7 medium · avg 56% 3.6% typical
Factual knowledge recall, Premium… 15.3% ▅····▃●·██·█ beats 4/13 medium · avg 52% 2.2% typical
Factual knowledge recall, Premium… 23.2% ▃▅●··▅▃·█▃▃▅ beats 3/13 medium · avg 61% 1.8% typical
OpenBookQA 26.4% █·▁▁●·▁▃▅▇▃▄ beats 9/26 hard · avg 49% 1.8% too hard for it
Social IQA 42.0% ▁···▁▃●▁▁▁·▁ beats 12/26 hard · avg 56% 1.4% blunt
Open PT-LLM Leaderboard average (… 34.0% ▃▁●▁▁▁▁▁▂█▅▁ beats 39/276 medium · avg 72% 0.9% noisy
This is not a Dataset (negation),… 50.1% ●▁····▃▁·▁·▁ beats 3/14 hard · avg 66% 0.3% sharp · too hard for it · noisy
source results share of the score
Stability AI 10 57% ▇▇▇▇▇▇▇
Pythia: A Suite for Analyzing Large Lan… 6 29% ▇▇▇
ImageNetVC: Zero- and Few-Shot Visual C… 1 5% ▇
Towards a Holistic Evaluation of LLMs o… 2 4% ▇
Investigating Table-to-Text Generation … 1 4% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
This is not a Dataset: A Large Negation… 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 24.7
Licences: Stability AI ([Apache-2.0](../../licenses/stablelm-original-eval-json-Apache-2.0.txt)) · Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling ([CC BY-SA 4.0](../../licenses/paper-2304.01373v2-CC-BY-SA-4.0.txt)) · ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-133-CC-BY-4-0.txt)) · Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios ([CC BY 4.0](../../licenses/acl-2023-emnlp-industry-17-CC-BY-4-0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-531-CC-BY-4-0.txt))
## How to run it
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run Pythia 12B (Q4_K_M, needs ~7.6 GB, from [huggingface.co/mradermacher](https://huggingface.co/mradermacher/pythia-12b-GGUF/tree/f0a60e50832ad355a5455d303de403745282e6aa))
```sh
llama cli -hf mradermacher/pythia-12b-GGUF:Q4_K_M
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf mradermacher/pythia-12b-GGUF:Q4_K_M
```
Other versions: Q8_0 (~12.6 GB) `llama cli -hf mradermacher/pythia-12b-GGUF:Q8_0`
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run Pythia 12B (Q4_K_M, needs ~7.6 GB, from [huggingface.co/mradermacher](https://huggingface.co/mradermacher/pythia-12b-GGUF/tree/f0a60e50832ad355a5455d303de403745282e6aa))
```sh
lms get https://huggingface.co/mradermacher/pythia-12b-GGUF@Q4_K_M -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
Other versions: Q8_0 (~12.6 GB) `lms get https://huggingface.co/mradermacher/pythia-12b-GGUF@Q8_0 -y`
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run Pythia 12B (Q4_K_M, needs ~7.6 GB, from [huggingface.co/mradermacher](https://huggingface.co/mradermacher/pythia-12b-GGUF/tree/f0a60e50832ad355a5455d303de403745282e6aa))
```sh
ollama run hf.co/mradermacher/pythia-12b-GGUF:Q4_K_M
```
Other versions: Q8_0 (~12.6 GB) `ollama run hf.co/mradermacher/pythia-12b-GGUF:Q8_0`
### Other ways to run it
1. Run it as published (fp16, needs ~23.84 GB, from [huggingface.co/EleutherAI](https://huggingface.co/EleutherAI/pythia-12b/tree/bb1e3e710cdf6b524461d543cfb5ba773f0a81b6))
```sh
uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='EleutherAI/pythia-12b', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/EleutherAI](https://huggingface.co/EleutherAI/pythia-12b)
[interactive version](https://whatmodel.sh/model/pythia-12b) · [JSON](https://whatmodel.sh/api?format=json&all=1)