[whatmodel.sh](../../README.md) / Llama 3.1 Tulu 3 8B
# Llama 3.1 Tulu 3 8B
Ai2 · released 2024-11 · 8.03B params · llama3.1
**score 45.7** ±0.7 · #457 of 732 · from 26 benchmark results across 4 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Llama 3.1 Tulu 3 8B among every model on that task or test, worst to best.
task vs the others rank counts tests
chat & writing ▁▁▁▃▆●▆█▆▇▇▃ #298 of 402 9.0% 3
math ▁▁▃●█▆▄█▄▂▁▁ #338 of 431 29% 4
chat & writing · #298 of 402 · 9.0% of its score
Beats [Ministral 3 14B Reasoning 2512](ministral-3-14b-reasoning.md), [Gemma 4 E4B](gemma-4-e4b.md), [OpenThinker3 7B](openthinker3-7b.md) here, though they rank higher overall. Trails [EXAONE 3.5 2.4B Instruct](exaone-3-5-2-4b.md), [LFM2 700M](lfm2-700m.md), [Flan-T5 XXL](flan-t5-xxl.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 1220.5 ▁▁▁▂▃●▃▆▆▇█▃ beats 76/385 avg 1260 5.1% blunt
AlpacaEval 2 length-controlled (m… 34.0% ▃▃·▁▁●▇▅·▁·▁ beats 10/22 medium · avg 39% 2.8% sharp · at its level
IFEval prompt-level loose (maker) 82.4% ▁▃▁▃▁▄▁▁●▃▃▁ beats 14/22 easy · avg 78% 1.1% at its level
math · #338 of 431 · 29% of its score
Beats [Athene-70B](athene-70b.md), [Llama 3 70B Instruct](llama-3-70b.md), [OLMo 2 32B Instruct (0325)](olmo-2-32b.md) here, though they rank higher overall. Trails [Gemma 2B (base)](gemma-2b-base.md), [Stable Code 3B](stable-code-3b.md), [Zephyr 7B beta](zephyr-7b-beta.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K (maker) 87.6% ▂▃··▃▂▅█●▃▂▃ beats 17/22 very easy · avg 90% 13% sharp · several runs
MGSM 58.6% ▅▃▃●·▅···▃·▃ beats 6/10 easy · avg 72% 11% sharp · at its level
MATH 46.1% █●██···▄▄▄·▄ beats 2/10 medium · avg 61% 3.7% at its level
MATH (subset unspecified) (maker) 43.7% ▃▃●▂▂▆·····▁ beats 12/22 medium · avg 58% 1.9% at its level
other tests (15)
test result vs the others in its run difficulty counts why
MMLU (maker) 68.2% ▁▂·▁·▁▁●·▃▃▅ beats 6/22 easy · avg 75% 16% at its level · several runs
BIG-Bench Hard (maker) 69.0% ▁▄·▁▁·▅●·▄▄▁ beats 10/22 medium · avg 72% 13% sharp · at its level · several runs
ARC Challenge 79.4% ▂●▂█▂···▄··▂ beats 1/10 very easy · avg 89% 8.7% sharp
OpenBookQA 79.8% ▂··▄●▂·▂···▄ beats 4/10 easy · avg 85% 4.8% at its level
MMLU-Pro 40.9% █▄●·█▄▄▄·▄·▄ beats 3/10 medium · avg 54% 4.5% at its level
Multilingual MMLU (Microsoft) 54.5% █▄▄●█··█···▄ beats 5/10 medium · avg 63% 3.9% at its level
HellaSwag 80.9% ▃▅█·▃▃●···▃▃ beats 8/10 easy · avg 82% 1.9% at its level
WinoGrande 65.4% ▃····▃▃●▃·▅▅ beats 5/10 medium · avg 71% 1.7% at its level
Social IQA 73.4% ▃·▃●▅█·▃···▃ beats 3/10 medium · avg 76% 1.5% blunt · at its level
PopQA (maker) 29.1% ▁▁▁▃▃▃▁●▁▃·▄ beats 13/21 hard · avg 31% 1.3% at its level
BoolQ 79.3% ▃··▃··●█▃·▃▃ beats 3/10 easy · avg 83% 1.3% at its level
GPQA (subset unspecified) 29.9% █··▄▂··●▂··▄ beats 7/10 very hard · avg 32% 1.1% sharp · too hard for it · noisy
EuroEval Life in the UK (English … 83.9% ▁▁▁▁▁▁▁▂▄●▄▃ beats 92/146 easy · avg 82% 0.7% at its level · noisy
PIQA 78.3% ▃·▅·▃●·▃▅··▃ beats 6/10 medium · avg 82% 0.5% blunt · at its level
TruthfulQA MC2 (maker) 55.0% ▃▃▂·●▃·▅█▂▅▂ beats 5/21 medium · avg 64% 0.4% blunt · at its level · several runs
source results share of the score
microsoft/Phi-4-mini-instruct model card 16 67% ▇▇▇▇▇▇▇▇
2 OLMo 2 Furious (submitted / maker-rep… 8 27% ▇▇▇
LMArena leaderboard dataset 1 5% ▇
EuroEval leaderboards 1 1% ▇
prior: pull toward the middle 0% ▇
weighted average = 45.7
Licences: microsoft/Phi-4-mini-instruct model card ([MIT](../../licenses/card-microsoft-phi-4-mini-instruct.txt)) · 2 OLMo 2 Furious ([CC BY 4.0](../../licenses/paper-2501.00656v3-CC-BY-4.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · EuroEval leaderboards ([MIT](https://github.com/EuroEval/leaderboards/blob/1a77f1d2f8ba1ba05ab73287fb090d76bf41b2e0/LICENSE))
## How to run it
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run Llama 3.1 Tulu 3 8B (Q4_K_M, needs ~4.9 GB, from [huggingface.co/lmstudio-community](https://huggingface.co/lmstudio-community/Llama-3.1-Tulu-3-8B-GGUF/tree/7033c16b4f79f8708a27d80bf2ae0c6537253d1b))
```sh
llama cli -hf lmstudio-community/Llama-3.1-Tulu-3-8B-GGUF:Q4_K_M
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf lmstudio-community/Llama-3.1-Tulu-3-8B-GGUF:Q4_K_M
```
Other versions: Q8_0 (~8.5 GB) `llama cli -hf lmstudio-community/Llama-3.1-Tulu-3-8B-GGUF:Q8_0`
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run Llama 3.1 Tulu 3 8B (Q4_K_M, needs ~4.9 GB, from [huggingface.co/lmstudio-community](https://huggingface.co/lmstudio-community/Llama-3.1-Tulu-3-8B-GGUF/tree/7033c16b4f79f8708a27d80bf2ae0c6537253d1b))
```sh
lms get https://huggingface.co/lmstudio-community/Llama-3.1-Tulu-3-8B-GGUF@Q4_K_M -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
Other versions: Q8_0 (~8.5 GB) `lms get https://huggingface.co/lmstudio-community/Llama-3.1-Tulu-3-8B-GGUF@Q8_0 -y` · 4bit (~4.5 GB, Mac only) `lms get https://huggingface.co/mlx-community/Llama-3.1-Tulu-3-8B-4bit -y` · 8bit (~8.5 GB, Mac only) `lms get https://huggingface.co/mlx-community/Llama-3.1-Tulu-3-8B-8bit -y`
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run Llama 3.1 Tulu 3 8B (Q4_K_M, needs ~5.32 GB, from [ollama.com](https://ollama.com/library/tulu3/tags))
```sh
ollama run tulu3:8b-q4_K_M
```
Other versions: Q8_0 (~10.14 GB) `ollama run tulu3:8b-q8_0`
### MLX (Mac)
1. Install MLX (Mac). Apple silicon Macs only, text only.
```sh
# with uv
uv tool install mlx-lm
# or
pip install mlx-lm
```
2. Run Llama 3.1 Tulu 3 8B (4bit, needs ~4.5 GB, Mac only, from [huggingface.co/mlx-community](https://huggingface.co/mlx-community/Llama-3.1-Tulu-3-8B-4bit/tree/ce1fb64f6c7f03cc816f1f86258097e6428ddcc9))
```sh
mlx_lm.chat --model mlx-community/Llama-3.1-Tulu-3-8B-4bit
```
3. Or start an OpenAI-compatible server at http://localhost:8080/v1
```sh
mlx_lm.server --model mlx-community/Llama-3.1-Tulu-3-8B-4bit
```
Other versions: 8bit (~8.5 GB, Mac only) `mlx_lm.chat --model mlx-community/Llama-3.1-Tulu-3-8B-8bit`
Model repository and documentation: [huggingface.co/allenai](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B)
[interactive version](https://whatmodel.sh/model/llama-3-1-tulu-3-8b) · [JSON](https://whatmodel.sh/api?format=json&all=1)