[whatmodel.sh](../../README.md) / OpenAssistant SFT-4 Pythia 12B
# OpenAssistant SFT-4 Pythia 12B
LAION (OpenAssistant) · released 2023-04 · 12B params · apache-2.0
**score 25.2** ±1.3 · #685 of 732 · from 11 benchmark results across 7 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks OpenAssistant SFT-4 Pythia 12B among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁▁●▃▅▇█▇▅▄▃▁ #456 of 469* 12% 1
chat & writing ▁●▁▃▆▄▆█▆▇▇▃ #399 of 402 61% 6
coding · #456 of 469 · 12% of its score
Beats [Molmo 7B-D (0924)](molmo-7b-d.md), [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Qwen3 0.6B](qwen3-0-6b.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
HumanEval 9.1% ▄··▄····▄●█▄ beats 3/6 medium · avg 63% 12% sharp · too hard for it
chat & writing · #399 of 402 · 61% of its score
Trails [Gemma 3 270M IT](gemma-3-270m.md), [SmolLM2 360M Instruct](smollm2-360m.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 1023.5 ▁●▁▂▃▃▃▆▆▇█▃ beats 5/385 avg 1260 18% blunt
Polyrating rating (bias-aware re-… 912.0 ▂●▂▁▃▇█▆▇▄▂▂ beats 5/95 avg 1187 17% typical
MT-Bench (GPT-4 single-answer gra… 4.3200 MT-Bench score (1-10) ▆▂▂●▆▄██▂▄▄▆ beats 5/28 avg 8.539 MT-Bench scor… 13% sharp · noisy
AlpacaEval 2 length-controlled 3.3% ▁●▂▃▆▆█▆▄▄▁▁ beats 2/122 medium · avg 35% 9.3% sharp · too hard for it
AlpacaEval 1.0 win rate (vs text-… 26.0% ▂●·▆·▇▃▆█▆▄▂ beats 2/45 very easy · avg 94% 4.9% sharp · at its level · noisy
MixInstruct GPT-Rank (LLM-Blender… 3.9000 mean rank (1-11) █···▄··▄···● beats 4/4 avg 6.709 mean rank (1-… 0.0% blunt · noisy
other tests (4)
test result vs the others in its run difficulty counts why
BIG-Bench Hard 30.0% ●██··█·█·█·█ beats 0/6 medium · avg 68% 12% sharp · too hard for it
MMLU 27.0% ▄▄·●···▄▄·██ beats 2/8 easy · avg 74% 7.0% typical
Static HHH alignment (BIG-bench h… 57.5% █·●········█ beats 1/2 medium · avg 77% 4.4% small run
TruthfulQA MC1 23.3% █·····█····● beats 2/2 medium · avg 50% 2.7% small run
source results share of the score
InstructEval 3 31% ▇▇▇▇
LMArena leaderboard dataset 1 18% ▇▇
Polyrating: A Cost-Effective and Bias-A… 1 17% ▇▇
AlpacaEval 1.0 and 2.0 leaderboards 2 14% ▇▇
LMSYS Chatbot Arena leaderboard Space: … 1 13% ▇▇
Aligning Large Language Models through … 2 7% ▇
LLM-Blender: Ensembling Large Language … 1 0% ▇
prior: pull toward the middle 1% ▇
weighted average = 25.2
Licences: InstructEval ([MIT / Apache-2.0](https://github.com/declare-lab/instruct-eval/blob/49bbd2ce7e3a5bdc4024ba62a5c9c68c107ee966/LICENSE)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · AlpacaEval 1.0 and 2.0 leaderboards ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LMSYS Chatbot Arena leaderboard Space: MT-Bench column of the published leaderboard table ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · Aligning Large Language Models through Synthetic Feedback ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-844-CC-BY-4-0.txt)) · LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/))
## How to run it
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run OpenAssistant SFT-4 Pythia 12B (Q4_K_M, needs ~7.6 GB, from [huggingface.co/ArtyomSubDiv](https://huggingface.co/ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF/tree/460d9ded18c12bd363d82058aef436b10b62666a))
```sh
llama cli -hf ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF:q4_k_m
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF:q4_k_m
```
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run OpenAssistant SFT-4 Pythia 12B (Q4_K_M, needs ~7.6 GB, from [huggingface.co/ArtyomSubDiv](https://huggingface.co/ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF/tree/460d9ded18c12bd363d82058aef436b10b62666a))
```sh
lms get https://huggingface.co/ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF@q4_k_m -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run OpenAssistant SFT-4 Pythia 12B (Q4_K_M, needs ~7.6 GB, from [huggingface.co/ArtyomSubDiv](https://huggingface.co/ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF/tree/460d9ded18c12bd363d82058aef436b10b62666a))
```sh
ollama run hf.co/ArtyomSubDiv/oasst-sft-4-pythia-12b-epoch-3.5-Q4_K_M-GGUF:q4_k_m
```
### Other ways to run it
1. Run it as published (fp16, needs ~23.84 GB, from [huggingface.co/OpenAssistant](https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5/tree/626b8c140cfdedb119dfb78c626cd772283dee33))
```sh
uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5', torch_dtype='auto', device_map='auto')('<|prompter|>What is a meme?<|endoftext|><|assistant|>', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/OpenAssistant](https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5)
[interactive version](https://whatmodel.sh/model/oasst-sft-4-pythia-12b) · [JSON](https://whatmodel.sh/api?format=json&all=1)