[whatmodel.sh](../../README.md) / DeepSeek R1 (Jan '25) # DeepSeek R1 (Jan '25) DeepSeek · released 2025-01 · 685B params (37B active) · mit **score 63.2** ±0.5 · #168 of 732 · from 64 benchmark results across 21 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks DeepSeek R1 (Jan '25) among every model on that task or test, worst to best. 
task            vs the others         rank  counts  tests
agent tasks     ▁▁▃▆●▄▅▅▄▄▂▁   #232 of 313    4.3%      3
coding          ▁▁▁▃▅▇█●▅▄▃▁   #140 of 469     27%     10
chat & writing  ▁▁▁▃▆▄▆█▆●▇▃   #118 of 402     12%      6
math            ▁▁▃▅█▆●█▄▂▁▁   #153 of 431     16%      7
agent tasks · #232 of 313 · 4.3% of its score Beats [o1 (2024-12-17)](o1-2024-12-17.md), [o3-mini](o3-mini.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty            counts  why
SWE-bench Verified (maker)           49.2%  ▁▁▁▁▁▁▂●█▆▁▁ beats 40/138  hard · avg 18%          3.1%  sharp · at its level · small run · seve…
Multi-SWE-bench, 7 languages (Mag…    5.9%  ▄···▄▄·▄▄·●█ beats 5/7     very hard · avg 2.6%    0.6%  blunt · too hard for it
BFCL v3                              56.9%  ▁▁▁▁▁▃▅●▅▄▁▁ beats 67/114  medium · avg 51%        0.6%  blunt · at its level
coding · #140 of 469 · 27% of its score Beats [Qwen3.5 Plus](qwen3-5-plus.md), [Qwen3-Coder-Next (80B-A3B)](qwen3-coder-next.md), [Gemini 2.5 Pro](gemini-2-5-pro.md) here, though they rank higher overall. Trails [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [Gemma 4 12B](gemma-4-12b.md), [Seed-OSS 36B Instruct](seed-oss-36b.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty            counts  why
Aider polyglot (225 exercises, tw…   56.9%  ▂·▂▃▅▂▃█●▄▂▂ beats 32/49   very hard · avg 7.6%     14%  sharp · at its level · several runs
SWE-bench Verified (maker)           49.2%  ▁▁▁▁▁▁▂●█▆▁▁ beats 40/138  hard · avg 18%          3.1%  sharp · at its level · small run · seve…
SciCode main problems (official l…    4.6%  ▃···█··▂·●·▂ beats 15/17   very hard · avg 1.0%    2.9%  sharp · too hard for it
LiveCodeBench (version unspecifie…   65.9%  ▄····▄▄··▄█● beats 6/6     medium · avg 42%        1.9%  at its level
WebApp1K-Duo-React pass@1 (onekq-…   68.7%  ▁··▁·▁▁▁▆█●█ beats 28/45   hard · avg 29%          1.6%  at its level
HumanEval (Vector Institute, Insp…   95.7%  ▄·▄·█·█·●·▄▄ beats 7/9     easy · avg 77%          1.2%  sharp · maxed out
LiveCodeBench 2024-08–2025-05 (ve…   63.5%  ▃···●··▅··▃▃ beats 2/7     medium · avg 43%        0.6%  at its level · small run
WebApp1K-React pass@1 (onekq-ai)     92.7%  ▃▁▁█▁▄▅▃▄▇●▄ beats 28/31   medium · avg 40%        0.6%  sharp · noisy
Multi-SWE-bench, 7 languages (Mag…    5.9%  ▄···▄▄·▄▄·●█ beats 5/7     very hard · avg 2.6%    0.6%  blunt · too hard for it
LiveCodeBench v6 (Aug 2024–May 20…   63.5%  ▂▅●▃█▃▂····▂ beats 4/17    medium · avg 48%        0.6%  at its level · small run
chat & writing · #118 of 402 · 12% of its score Beats [LongCat 2.0 (1.6T-A48B)](longcat-2-0.md), [Step 3.5 Flash (196B-A11B)](step-3-5-flash.md), [GPT-5 mini](gpt-5-mini.md) here, though they rank higher overall. Trails [Nemotron-Cascade-8B](nemotron-cascade-8b.md), [Mistral Large 3 (675B-A41B)](mistral-large-3.md), [Gemini 2.5 Flash](gemini-2-5-flash.md) here, though they rank lower overall. 
test                                result  vs the others in its run    difficulty           counts  why
Arena-Hard v2.0 hard prompts (sty…   58.0%  ▂·▃▅▅·▃█●▂▂▂ beats 16/22    very hard · avg 12%    3.3%  sharp · at its level
Arena-Hard (v0.1, GPT-4-1106 judg…   92.3%  ▂▁▁▅█▅▃▃▃▅▃● beats 62/62    medium · avg 51%       2.8%  sharp · small run
LMArena (text)                      1398.1  ▁▁▁▂▃▃▃▆▆●█▃ beats 243/385  avg 1260               2.7%  blunt
IFEval prompt-level strict           83.4%  ▄▄··▄·●··█·▄ beats 3/6      easy · avg 73%         1.4%  several runs
AlpacaEval 2 length-controlled (m…   87.6%  ▁▁▃▄▆██▆▃▃▁● beats 155/155  medium · avg 48%       1.1%  sharp · small run
Arena-Hard v2.0 creative writing …   77.0%  ▆▂▂▆▂▂▃█▂●·▂ beats 21/22    hard · avg 19%         0.7%  sharp · at its level · noisy
math · #153 of 431 · 16% of its score Beats [MiMo-V2.5 (310B-A15B)](mimo-v2-5.md), [MiniMax-M2.7](minimax-m2-7.md), [MiniMax-M2.1](minimax-m2-1.md) here, though they rank higher overall. Trails [Nemotron 3 Nano 30B-A3B](nemotron-3-nano-30b-a3b.md), [Qwen3-VL 8B Thinking](qwen3-vl-8b-thinking.md), [Olmo 3 32B Think](olmo-3-32b-think.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty            counts  why
AIME 2025                            70.4%  █··▄··●··▄·▄ beats 3/6     medium · avg 38%        3.6%  sharp · small run · several runs
AIME 2024 (maker)                    79.8%  ▃·▃·▃▃···█·● beats 7/7     medium · avg 51%        3.5%  sharp · small run · several runs
MATH level 5                         93.1%  ▂▂▅▆▄█▆▃▅●▃▄ beats 80/92   hard · avg 28%          2.9%  sharp
MATH (Vector Institute, Inspect E…   92.7%  ▄··██▄··●·▄█ beats 6/9     medium · avg 50%        2.3%  sharp
OTIS mock AIME 2024–2025             53.3%  ▃▃▄▃▃▃●▇█▃▂▅ beats 80/197  very hard · avg 7.2%    2.1%  sharp · at its level · noisy
HMMT 2025 (edition unspecified) (…   41.7%  ▃●▇▅▇▅▁·▁··▁ beats 4/32    hard · avg 21%          0.9%  sharp · at its level · noisy · small run
MATH-500 (maker)                     97.3%  ▄·▄▄·▄▄█···● beats 7/7     very easy · avg 91%     0.3%  maxed out
other tests (23) 
test                                result  vs the others in its run    difficulty            counts  why
GPQA Diamond                         71.7%  ▁▁▁▁▂▄▆▄●▇█▅ beats 109/211  hard · avg 39%           15%  sharp · at its level · several runs
ZebraLogic (ZeroEval, 1,000 puzzl…   78.7%  ▁▁▃▆██▁▁·●▁▁ beats 55/57    hard · avg 15%          6.3%  sharp · at its level
GPQA Diamond (Vector Institute, I…   70.5%  ▃··▃▃▃▃·▃·●█ beats 6/9      hard · avg 40%          3.6%  sharp · at its level
MMLU-Pro (maker)                     84.0%  ▁▁·▁▁▁▃▂▄▅●▁ beats 162/188  medium · avg 59%        2.0%  small run · several runs
SimpleQA (maker)                     30.1%  ▄▄▄▆█▆██●·▂█ beats 27/32    very hard · avg 11%     1.9%  at its level · small run · several runs
MMLU-Pro (Vector Institute, Inspe…   83.8%  ▃····▃▅·▃▅·● beats 8/9      medium · avg 58%        1.9%  typical
BRIDGE clinical text, zero-shot a…   44.3%  ▁▁▁▁▁▂▂▃▄▆█● beats 92/97    hard · avg 31%          1.8%  blunt · at its level
MMLU-Redux (maker)                   92.9%  ▁▁▃▁▂▃▄▅▇▅●▄ beats 51/63    easy · avg 80%          1.7%  small run · several runs
EQ-Bench 3 (emotional intelligenc…  1224.7  ▁·▂▁▄▅▅●█▄▁▁ beats 35/62    avg 1034                1.7%  typical
CPTUB average (SpeakLeash, 0-5 ju…   82.8%  ▁▂▂▂▃█▃▆▅▄●▃ beats 86/95    medium · avg 65%        1.3%  typical
MMLU (maker)                         90.8%  ▁▁▁▁▁▂▂▅██▆● beats 212/214  medium · avg 74%        1.2%  small run
FRAMES (maker)                       82.5%  ●··········█ beats 0/1      medium · avg 64%        1.0%  small run · several runs
LongSocialBench (full-context, 4-…   47.3%  ▆▂·▄█·▂●·▄▄▂ beats 11/16    very hard · avg 36%     0.9%  blunt · at its level
Humanity's Last Exam (maker)          8.5%  ▁▁▁●▆▆▇██▆▁▃ beats 14/111   very hard · avg 3.9%    0.9%  small run
C-Eval (maker)                       91.8%  ▁···▁▁▂▁▂▂▃● beats 29/33    easy · avg 75%          0.8%  small run
CNMO 2024 (maker)                    78.8%  ●··········█ beats 0/1      medium · avg 59%        0.8%  small run · several runs
Judgemark v2 (LLM-as-judge for cr…   77.0%  ▃▃··▃▅▅···●▃ beats 7/10     medium · avg 53%        0.7%  typical
ChineseSimpleQA (maker)              63.7%  ▄··▄●▄··▄··█ beats 2/6      medium · avg 47%        0.6%  at its level · small run
DROP (maker)                         92.2%  ▃▅█▅·▃·▆▄▂▁● beats 44/44    medium · avg 65%        0.2%  blunt · small run
Omni-MATH                            85.0%  █·█·█·██··█● beats 6/6      medium · avg 64%        0.1%  blunt · noisy
Vectara hallucination leaderboard…   85.7%  ▁▁●▃▄▄█▆▃▂▂▁ beats 15/140   very easy · avg 94%     0.1%  blunt · maxed out · noisy
Vectara hallucination leaderboard…   88.7%  ▁▂▁▂●█▂▃▄▃▁▁ beats 30/98    very easy · avg 90%     0.0%  blunt
CLUEWSC (maker)                      92.8%  █··········● beats 1/1                                 —  not counted yet: no other model in the …
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
deepseek-ai/DeepSeek-R1 model card (sub…       18   20% ▇▇
deepseek-ai/DeepSeek-R1-0528 model card…       13   16% ▇▇
Epoch AI                                        3    9% ▇
Vector Institute Evaluation Leaderboard         4    9% ▇
Aider polyglot benchmark runs                   1    7% ▇
WildEval ZeroEval published results             1    6% ▇
Microsoft                                       4    6% ▇
deepseek-ai/DeepSeek-R1-0528-Qwen3-8B m…        4    6% ▇
Arena-Hard-Auto v2.0-Preview leaderboard        2    4% ▇
SciCode leaderboard                             1    3% ▇
LMArena leaderboard dataset                     1    3% ▇
WebApp1K models leaderboard                     2    2% ▇
BRIDGE Medical Leaderboard                      1    2% ▇
EQ-Bench 3 canonical leaderboard results        1    2% ▇
SpeakLeash CPTUB                                1    1% ▇
OpenCompass CompassAcademic leaderboard         1    1% ▇
LongSocialBench: Do Long-Context LLMs U…        1    1% ▇
Judgemark v2 results                            1    1% ▇
Multi-SWE-bench experiments                     1    1% ▇
Berkeley Function Calling Leaderboard r…        1    1% ▇
Vectara hallucination leaderboard               2    0% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 63.2
Licences: deepseek-ai/DeepSeek-R1 model card ([mit](../../licenses/deepseek-ai--DeepSeek-R1.txt)) · deepseek-ai/DeepSeek-R1-0528 model card ([mit](../../licenses/deepseek-ai--DeepSeek-R1-0528.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Vector Institute Evaluation Leaderboard ([Apache-2.0](../../licenses/vector-inspect-evals-leaderboard-Apache-2.0.txt)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Microsoft ([CC BY 4.0](../../licenses/report-2504.21318v1.txt)) · deepseek-ai/DeepSeek-R1-0528-Qwen3-8B model card ([MIT](../../licenses/card-deepseek-ai-deepseek-r1-0528-qwen3-8b.txt)) · Arena-Hard-Auto v2.0-Preview leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-Apache-2.0.txt)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · BRIDGE Medical Leaderboard ([MIT](../../licenses/bridge-medical-leaderboard-MIT.txt)) · EQ-Bench 3 canonical leaderboard results ([MIT](../../licenses/eqbench3-MIT.txt)) · SpeakLeash CPTUB ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · OpenCompass CompassAcademic leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LongSocialBench: Do Long-Context LLMs Understand Online Discussion Threads? ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Judgemark v2 results ([MIT](../../licenses/judgemark-v2-MIT.txt)) · Multi-SWE-bench experiments ([Apache-2.0](../../licenses/multi-swe-bench-experiments-Apache-2.0.txt)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
Not counted (1) 
source                  test     score  why
deepseek-ai/DeepSeek-…  CLUEWSC  92.8%  too few models have it yet
## How to run it ### Through an API api (Together AI) · $4.00/M blended · $3 in / $7 out per 1M tokens · [docs.together.ai](https://docs.together.ai/docs/serverless-models) [price source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/providers/togetherai/models/deepseek-ai/DeepSeek-R1.toml): Standard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ. · [context source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/models/deepseek/deepseek-r1.toml) ### llama.cpp (recommended) 1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) ```sh # macOS / Linux curl -LsSf https://llama.app/install.sh | sh # Windows irm https://llama.app/install.ps1 | iex ``` 2. Run DeepSeek R1 (Jan '25) (Q4_K_M, needs ~411.5 GB, from [huggingface.co/unsloth](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main)) ```sh llama cli -hf unsloth/DeepSeek-R1-GGUF:Q4_K_M ``` 3. Or start a local server and open the address it prints (http://localhost:9931) ```sh llama serve -hf unsloth/DeepSeek-R1-GGUF:Q4_K_M ``` Other versions: ud-q2_k_xl (~206 GB) `llama cli -hf unsloth/DeepSeek-R1-GGUF:UD-Q2_K_XL` · ud-iq1_s (~103.25 GB) `llama cli -hf unsloth/DeepSeek-R1-GGUF:UD-IQ1_S` ### LM Studio 1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download) 2. Run DeepSeek R1 (Jan '25) (Q4_K_M, needs ~411.5 GB, from [huggingface.co/unsloth](https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main)) ```sh lms get https://huggingface.co/unsloth/DeepSeek-R1-GGUF@Q4_K_M -y ``` 3. Then run lms ls to see its name, and lms chat followed by that name Other versions: ud-q2_k_xl (~206 GB) `lms get https://huggingface.co/unsloth/DeepSeek-R1-GGUF@UD-Q2_K_XL -y` · ud-iq1_s (~103.25 GB) `lms get https://huggingface.co/unsloth/DeepSeek-R1-GGUF@UD-IQ1_S -y` · 4bit (~419.5 GB, Mac only) `lms get https://huggingface.co/mlx-community/DeepSeek-R1-4bit -y` ### MLX (Mac) 1. Install MLX (Mac). Apple silicon Macs only, text only. ```sh # with uv uv tool install mlx-lm # or pip install mlx-lm ``` 2. Run DeepSeek R1 (Jan '25) (4bit, needs ~419.5 GB, Mac only, from [huggingface.co/mlx-community](https://huggingface.co/mlx-community/DeepSeek-R1-4bit/tree/5aa58959991bd42ce9c0eb634b1aff00a3410426)) ```sh mlx_lm.chat --model mlx-community/DeepSeek-R1-4bit ``` 3. Or start an OpenAI-compatible server at http://localhost:8080/v1 ```sh mlx_lm.server --model mlx-community/DeepSeek-R1-4bit ``` Model repository and documentation: [huggingface.co/deepseek-ai](https://huggingface.co/deepseek-ai/DeepSeek-R1) ## Quantization evidence How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent. They don't describe every download of that precision. 
precision                       test                                full → quantized    kept
INT4 W4A16 (weights only; GPT…  OpenLLM v1 average of 6 tasks (AR…      0.81 → 0.813  100.3%
INT4 W4A16 (weights only; GPT…  MATH-500 pass@1 (LightEval/open-r…     0.972 → 0.971   99.8%
INT4 W4A16 (weights only; GPT…  GSM8K 5-shot (lm-evaluation-harne…     0.959 → 0.957   99.8%
INT4 W4A16 (weights only; GPT…  MMLU 5-shot (lm-evaluation-harnes…      0.872 → 0.87   99.7%
INT4 W4A16 (weights only; GPT…  Reasoning average of 3 tasks (AIM…       0.83 → 0.82   98.8%
INT4 W4A16 (weights only; GPT…  AIME 2024 pass@1 (LightEval/open-…      0.783 → 0.77   98.3%
INT4 W4A16 (weights only; GPT…  GPQA Diamond pass@1 (LightEval/op…     0.734 → 0.719   98.0%
Each kept value links to its source. [interactive version](https://whatmodel.sh/model/deepseek-r1) · [JSON](https://whatmodel.sh/api?format=json&all=1)