[whatmodel.sh](../../README.md) / Llama 3.1 405B Instruct
# Llama 3.1 405B Instruct
Meta · released 2024-07 · 405B params · llama-3.1
**score 55.8** ±0.5 · #262 of 732 · from 36 benchmark results across 19 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Llama 3.1 405B Instruct among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃▆●▄▅▅▄▄▂▁ #216 of 313 12% 8
coding ▁▁▁▃▅▇●▇▅▄▃▁ #223 of 469 29% 10
chat & writing ▁▁▁▃▆▄▆●▆▇▇▃ #192 of 402 10% 3
math ▁▁▃▅●▆▄█▄▂▁▁ #267 of 431 12% 4
agent tasks · #216 of 313 · 12% of its score
Beats [o1 (2024-12-17)](o1-2024-12-17.md), [o3-mini](o3-mini.md), [DeepSeek R1 (Jan '25)](deepseek-r1.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [Mellum2 12B-A2.5B Instruct](mellum2-12b-a2-5b-instruct.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
SWE-bench Verified 33.4% ▃···▃·▃·▃●·▃ beats 6/7 hard · avg 17% 3.1% sharp · at its level
MASSIVE-Agents AST (10k, zero-sho… 25.6% █····███·█●█ beats 5/6 very hard · avg 15% 2.4% sharp
BrowserGym WorkArena-L1 (AgentLab… 43.3% █···█●▄·▄█·█ beats 4/10 hard · avg 28% 1.6% at its level
BrowserGym WebArena (AgentLab Gen… 24.0% █··●·▄█·▄··▄ beats 2/7 hard · avg 17% 1.5% typical
BrowserGym MiniWoB (AgentLab Gene… 64.6% ▅····▅●▃▃·▃█ beats 4/11 medium · avg 53% 1.1% blunt · at its level
BrowserGym WorkArena-L2 (AgentLab… 7.2% ▃▅▃·●▃▃··█·▃ beats 4/11 very hard · avg 3.6% 0.8% sharp · noisy
τ-bench Airline 26.0% ▁▁●·▅▄▁█▇▁▁▇ beats 2/28 hard · avg 22% 0.7% at its level
τ-bench Retail 29.1% ▁····●·▂▇█▂▆ beats 2/28 hard · avg 27% 0.5% at its level · noisy
coding · #223 of 469 · 29% of its score
Beats [Qwen3.5 4B](qwen3-5-4b.md), [Qwen3-VL 32B Instruct](qwen3-vl-32b-instruct.md), [Solar Open 100B (102B-A12B)](solar-open-100b.md) here, though they rank higher overall. Trails [QwQ-32B-Preview](qwq-32b-preview.md), [Ministral 3 3B Reasoning 2512](ministral-3-3b-reasoning.md), [s1.1 32B](s1-1-32b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LiveCodeBench official (Jan 1–Nov… 37.3% ▁▁·▁▂▄▆●▅▂·▂ beats 22/31 hard · avg 23% 6.5% sharp · at its level
CRUXEval (ZeroEval, 800 problems) 73.5% ▁▂▆▃▆█●▃▂·▁▁ beats 32/47 medium · avg 57% 5.9% sharp · at its level
Aider code editing (133 Exercism … 66.2% ▁·▁▂▄▁▆▄●▆▂▂ beats 39/62 medium · avg 51% 4.8% sharp · at its level
SWE-bench Verified 33.4% ▃···▃·▃·▃●·▃ beats 6/7 hard · avg 17% 3.1% sharp · at its level
Aider polyglot (225 exercises, tw… 13.8% █▄··█·●····▄ beats 5/7 very hard · avg 5.5% 2.8% sharp · noisy
LiveCodeBench v5 (Oct 2024–Feb 20… 29.3% ▄▄·▄·····█●█ beats 5/7 hard · avg 18% 1.8% at its level
SciCode main problems (official l… 1.5% ▃···●··▂·▃·▂ beats 10/17 very hard · avg 1.0% 1.3% sharp · too hard for it
WebApp1K-React pass@1 (onekq-ai) 30.2% ▃▁▁●▁▄▅▃▄▇▁▄ beats 5/31 medium · avg 40% 1.2% sharp · at its level · noisy
MBPP+ 87.0% ██·██···█●██ beats 5/7 easy · avg 78% 1.2% typical
BIRD-SQL (dev) 59.4% ▄··▄·▄·█··▄● beats 6/7 medium · avg 47% 1.0% at its level
chat & writing · #192 of 402 · 10% of its score
Beats [Seed-OSS 36B Instruct](seed-oss-36b.md), [Gemma 2 9B IT SimPO](gemma-2-9b-simpo.md), [Nemotron-Cascade-14B-Thinking](nemotron-cascade-14b-thinking.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
Arena-Hard (v0.1, GPT-4-1106 judg… 69.3% ▂▁▁▅█▅▄▃●▅▃▁ beats 45/60 medium · avg 47% 5.6% sharp · at its level
LMArena (text) 1335.5 ▁▁▁▂▃▃▃●▆▇█▃ beats 171/385 avg 1260 2.7% blunt
Arabic IFEval prompt-level (Incep… 50.7% ▁·▁▁▂▁▄▄▆▄●▄ beats 27/38 medium · avg 46% 2.1% sharp · at its level
math · #267 of 431 · 12% of its score
Beats [GPT-4o (2024-11-20)](gpt-4o-2024-11-20.md), [GPT-4 Turbo (2024-04-09)](gpt-4-turbo-2024-04-09.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MATH level 5 49.8% ▂▂▅▆▄●▆▃▅▂▃▄ beats 42/92 hard · avg 28% 6.0% sharp · at its level
AIME 2024 20.0% ▁▁▁▂▁●▁▅███▃ beats 11/85 very hard · avg 14% 2.2% sharp · small run
GSM8K (ZeroEval, zero-shot) 96.0% ▁▄·▁▄█▅▅▅██● beats 40/43 very easy · avg 87% 1.8% sharp · maxed out
OTIS mock AIME 2024–2025 9.7% ▃▃▄●▃▃▇▇█▃▂▅ beats 50/197 very hard · avg 7.2% 1.6% sharp · noisy
other tests (12)
test result vs the others in its run difficulty counts why
LiveBench 2024-07-26 54.3% ▁▁▁▂▂▃▂█▆▃▅● beats 58/61 medium · avg 38% 12% at its level
MMLU-Redux (ZeroEval, 2,778 quest… 84.4% ▁·▁▁▄▄▄▅●▄▂▁ beats 42/53 easy · avg 75% 9.1% typical
ZebraLogic (ZeroEval, 1,000 puzzl… 25.0% ▁▁▃▆█●▁▁·▁▁▁ beats 38/57 hard · avg 15% 5.5% sharp · at its level
GPQA Diamond 50.9% ▁▁▁▁▂▄●▄▅▇█▅ beats 71/211 hard · avg 39% 4.2% sharp · at its level
EQ-Bench 3 (emotional intelligenc… 1058.4 ▁·▂▁●▅▅▃█▄▁▁ beats 6/62 avg 1034 1.7% typical
RepoQA (32K, threshold 0.8) 90.4% ▄▄▄··▄▄··●·█ beats 5/7 easy · avg 80% 1.5% typical
KINA (Knowledge Index of Noah's A… 29.6% ▁▁▂▁▃▄●▆▄▅▂▃ beats 19/39 very hard · avg 21% 1.4% typical
Polish EQ-Bench v2 (SpeakLeash) 77.2% ▁··▁▁▂▂▆▅▆█● beats 67/70 medium · avg 62% 1.3% at its level
BigCodeBench (Instruct) 46.2% ▄··▄▄···▄▄●▄ beats 6/7 hard · avg 35% 1.3% at its level
Judgemark v2 (LLM-as-judge for cr… 60.1% ▃▃··▃●▅···█▃ beats 4/10 medium · avg 53% 0.9% at its level
DiagnosticIQ industrial maintenan… 56.6% ▁··▃▁▁●▅█▅▁▅ beats 5/24 hard · avg 51% 0.6% blunt · at its level
Open PT-LLM Leaderboard average (… 82.1% ▃▁▁▁▁▁▁▁▂█▅● beats 267/276 medium · avg 72% 0.6% at its level · noisy
source results share of the score
WildEval ZeroEval published results 4 22% ▇▇▇
Cohere 10 16% ▇▇
Epoch AI 3 12% ▇
LiveBench model judgments 1 12% ▇
LiveCodeBench official website results 1 7% ▇
Arena-Hard-Auto v0.1 leaderboard 1 6% ▇
ServiceNow BrowserGym leaderboard 4 5% ▇
Aider code editing benchmark runs 1 5% ▇
LMArena leaderboard dataset 1 3% ▇
MASSIVE-Agents 1 2% ▇
Inception Arabic/Hindi leaderboards 1 2% ▇
EQ-Bench 3 canonical leaderboard results 1 2% ▇
Knowledge Index of Noah's Ark 1 1% ▇
SpeakLeash Polish EQ-Bench leaderboard 1 1% ▇
SciCode leaderboard 1 1% ▇
WebApp1K models leaderboard 1 1% ▇
Judgemark v2 results 1 1% ▇
DiagnosticIQ: A Benchmark for LLM-Based… 1 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
prior: pull toward the middle 0% ▇
weighted average = 55.8
Licences: WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Cohere ([CC BY 4.0](../../licenses/report-2504.00698v2.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · ServiceNow BrowserGym leaderboard ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · MASSIVE-Agents ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Inception Arabic/Hindi leaderboards ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · EQ-Bench 3 canonical leaderboard results ([MIT](../../licenses/eqbench3-MIT.txt)) · Knowledge Index of Noah's Ark ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · SpeakLeash Polish EQ-Bench leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · Judgemark v2 results ([MIT](../../licenses/judgemark-v2-MIT.txt)) · DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt))
## How to run it
### Through an API
api (Amazon Bedrock) · $2.40/M blended · $2.4 in / $2.4 out per 1M tokens · [docs.aws.amazon.com](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-meta-llama-3-1-405b-instruct.html)
[price source](https://aws.amazon.com/bedrock/pricing): USD per 1M tokens, Amazon Bedrock on-demand Llama 3.1 Instruct (405B), standard (not latency-optimized) row, offered in US West (Oregon) and US East (Ohio). Batch .20/.20 and the latency-optimized row excluded. Table cells are rendered from AWS's own price data file for the page (published 2026-09-25T06:55:02Z). Currency: USD. Amazon Bedrock official pricing page, observed 2026-09-25. · [context source](https://github.com/meta-llama/llama-models/blob/0e0b8c519242d5833d8c11bffc1232b77ad7f301/models/llama3_1/MODEL_CARD.md): Meta Llama 3.1 model card (llama-models repo, pinned): 405B context length 128k (stated rounded). Model ceiling, not a promise by any hosted API. Observed 2026-09-25.
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run Llama 3.1 405B Instruct (Q2_K, needs ~151.2 GB, from [huggingface.co/MaziyarPanahi](https://huggingface.co/MaziyarPanahi/Meta-Llama-3.1-405B-Instruct-GGUF/tree/85b9bd67025a4337e9694ec0edaf46437fe6283b))
```sh
llama cli -hf MaziyarPanahi/Meta-Llama-3.1-405B-Instruct-GGUF:Q2_K
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf MaziyarPanahi/Meta-Llama-3.1-405B-Instruct-GGUF:Q2_K
```
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run Llama 3.1 405B Instruct (Q2_K, needs ~151.2 GB, from [huggingface.co/MaziyarPanahi](https://huggingface.co/MaziyarPanahi/Meta-Llama-3.1-405B-Instruct-GGUF/tree/85b9bd67025a4337e9694ec0edaf46437fe6283b))
```sh
lms get https://huggingface.co/MaziyarPanahi/Meta-Llama-3.1-405B-Instruct-GGUF@Q2_K -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
Other versions: 8bit (~431.2 GB, Mac only) `lms get https://huggingface.co/mlx-community/Llama-3.1-405B-Instruct-8bit -y`
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run Llama 3.1 405B Instruct (Q4_K_M, needs ~243.5 GB, from [ollama.com](https://ollama.com/library/llama3.1/tags))
```sh
ollama run llama3.1:405b
```
### MLX (Mac)
1. Install MLX (Mac). Apple silicon Macs only, text only.
```sh
# with uv
uv tool install mlx-lm
# or
pip install mlx-lm
```
2. Run Llama 3.1 405B Instruct (8bit, needs ~431.2 GB, Mac only, from [huggingface.co/mlx-community](https://huggingface.co/mlx-community/Llama-3.1-405B-Instruct-8bit/tree/fcad0c7e14154d0f27e7b9c2dd323f3162eff91a))
```sh
mlx_lm.chat --model mlx-community/Llama-3.1-405B-Instruct-8bit
```
3. Or start an OpenAI-compatible server at http://localhost:8080/v1
```sh
mlx_lm.server --model mlx-community/Llama-3.1-405B-Instruct-8bit
```
Model repository and documentation: [huggingface.co/meta-llama](https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct)
[interactive version](https://whatmodel.sh/model/llama-3-1-405b) · [JSON](https://whatmodel.sh/api?format=json&all=1)