[whatmodel.sh](../../README.md) / DeepSeek V3 (Dec '24) # DeepSeek V3 (Dec '24) DeepSeek · released 2024-12 · 671B params (37B active) · deepseek **score 59.7** ±0.6 · #208 of 732 · from 46 benchmark results across 16 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks DeepSeek V3 (Dec '24) among every model on that task or test, worst to best. 
task            vs the others         rank  counts  tests
agent tasks     ▁▁▃▆●▄▅▅▄▄▂▁   #229 of 313    8.5%      4
coding          ▁▁▁▃▅▇█●▅▄▃▁   #154 of 469     43%     10
chat & writing  ▁▁▁▃▆▄▆█●▇▇▃   #155 of 402     10%      4
math            ▁▁▃▅█●▄█▄▂▁▁   #228 of 431     10%      4
agent tasks · #229 of 313 · 8.5% of its score Beats [o1 (2024-12-17)](o1-2024-12-17.md), [o3-mini](o3-mini.md), [DeepSeek R1 (Jan '25)](deepseek-r1.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md), [Jamba 1.5 Mini](jamba-1-5-mini.md), [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md) here, though they rank lower overall. 
test                result  vs the others in its run   difficulty        counts  why
SWE-bench Verified   45.8%  ▃···▃·▃·▃█·● beats 7/7     hard · avg 17%      6.2%  sharp · at its level · several runs
τ-bench Airline      25.5%  ▁●▁·▅▄▁█▇▁▁▇ beats 1/28    hard · avg 22%      1.0%  at its level
BFCL v3              57.6%  ▁▁▁▁▁▃▅●▅▄▁▁ beats 70/114  medium · avg 51%    0.7%  blunt · at its level
τ-bench Retail       54.8%  ▁····▃·▂●█▂▆ beats 7/28    hard · avg 27%      0.7%  at its level · noisy
coding · #154 of 469 · 43% of its score Beats [Qwen3-Coder-Next (80B-A3B)](qwen3-coder-next.md), [Gemini 2.5 Pro](gemini-2-5-pro.md), [Gemini 3.1 Flash-Lite](gemini-3-1-flash-lite.md) here, though they rank higher overall. Trails [Ministral 3 8B Reasoning 2512](ministral-3-8b-reasoning.md), [Phi-4-reasoning 14B](phi-4-reasoning.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty            counts  why
Aider polyglot (225 exercises, tw…   48.4%  ▂·▂▃▅▂▃●▆▄▂▂ beats 25/49  very hard · avg 7.6%     17%  sharp · at its level · several runs
LiveCodeBench official (Jan 1–Nov…   48.9%  ▁▁·▁▂▄▆█▅●·▂ beats 28/31  hard · avg 23%          8.6%  sharp · at its level
SWE-bench Verified                   45.8%  ▃···▃·▃·▃█·● beats 7/7    hard · avg 17%          6.2%  sharp · at its level · several runs
SciCode main problems (official l…    3.1%  ▃···█··●·▃·▂ beats 12/17  very hard · avg 1.0%    2.5%  sharp · too hard for it
LiveCodeBench v5 (Oct 2024–Feb 20…   33.5%  ▄▄·▄·····█▄● beats 7/7    hard · avg 18%          2.4%  at its level
WebApp1K-Duo-React pass@1 (onekq-…   58.5%  ▁··▁·▁▁▁●█▇█ beats 12/45  hard · avg 29%          2.0%  at its level
MBPP+                                90.0%  ██·██···███● beats 7/7    easy · avg 78%          1.2%  typical
WebApp1K-React pass@1 (onekq-ai)     87.2%  ▃▁▁█▁▄▅▃▄●▁▄ beats 25/31  medium · avg 40%        1.1%  sharp · noisy
BIRD-SQL (dev)                       53.1%  ▄··▄·▄·●··▄█ beats 4/7    medium · avg 47%        1.1%  at its level
LiveCodeBench (version unspecifie…   39.2%  ▁···▁▁▁●▇▃▄▁ beats 8/38   hard · avg 29%          0.9%  at its level · small run
chat & writing · #155 of 402 · 10% of its score Beats [INTELLECT-3](intellect-3.md), [MiniMax-M2](minimax-m2.md), [MiniCPM5 2B](minicpm5-2b.md) here, though they rank higher overall. Trails [LFM2.5 1.2B Thinking](lfm2-5-1-2b-thinking.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md), [LFM2.5 2.6B](lfm2-5-2-6b.md) here, though they rank lower overall. 
test                                result  vs the others in its run    difficulty        counts  why
Arena-Hard (v0.1, GPT-4-1106 judg…   85.5%  ▂▁▁▅█▅▃▃▃▅●▂ beats 59/62    medium · avg 51%    3.7%  sharp · small run
LMArena (text)                      1358.4  ▁▁▁▂▃▃▃▆●▇█▃ beats 202/385  avg 1260            3.4%  blunt
IFEval prompt-level strict           85.8%  ▄▄··▄·▄··●·▄ beats 5/6      easy · avg 73%      2.0%  several runs
AlpacaEval 2 length-controlled (m…   70.0%  ▁▁▃▄▆██▆▃●▁▁ beats 149/155  medium · avg 48%    1.4%  sharp · at its level · small run
math · #228 of 431 · 10% of its score Beats [MiniMax-M2.1](minimax-m2-1.md) here, though they rank higher overall. Trails [Gemma 7B (base)](gemma-7b-base.md), [Phi-2 (2.7B)](phi-2.md), [Granite 8B Code Base (4K)](granite-8b-code-base-4k.md) here, though they rank lower overall. 
test                      result  vs the others in its run   difficulty            counts  why
MATH level 5               64.9%  ▂▂▅▆▄█●▃▅▂▃▄ beats 55/92   hard · avg 28%          5.6%  sharp
OTIS mock AIME 2024–2025   15.8%  ▃▃▄▃●▃▇▇█▃▂▅ beats 53/197  very hard · avg 7.2%    2.7%  sharp · at its level · noisy
AIME 2024 (maker)          39.2%  ▃·●·▃▃···█·▃ beats 1/7     medium · avg 50%        0.7%  sharp · noisy
MATH-500 (maker)           90.2%  ▄·▄●·▄▄█···▄ beats 2/7     very easy · avg 91%     0.5%  maxed out
other tests (21) 
test                                result  vs the others in its run    difficulty           counts  why
GPQA Diamond                         56.5%  ▁▁▁▁▂▄▆●▅▇█▅ beats 83/211   hard · avg 39%         8.4%  sharp · at its level · several runs
ZebraLogic (ZeroEval, 1,000 puzzl…   42.1%  ▁▁▃▆██●▁·▁▁▁ beats 52/57    hard · avg 15%         7.9%  sharp · at its level
KCL-Essay vanilla                    34.3%  ▁···▁▃▂●▂▃▃▂ beats 13/23    hard · avg 17%         2.3%  at its level
CPTUB average (SpeakLeash, 0-5 ju…   80.5%  ▁▂▂▂▃█▃▆▅●▂▃ beats 82/95    medium · avg 65%       1.9%  typical
MMLU (maker)                         88.5%  ▁▁▁▁▁▂▂▅██▆● beats 204/214  medium · avg 74%       1.6%  small run
BigCodeBench (Instruct)              48.6%  ▄··▄▄···▄▄█● beats 7/7      hard · avg 35%         1.6%  at its level
RepoQA (32K, threshold 0.8)          92.2%  ▄▄▄··▄▄··▄·● beats 6/7      easy · avg 80%         1.5%  typical
MMLU-Pro (maker)                     75.9%  ▁▁·▁▁▁▃▂▄●█▁ beats 105/188  medium · avg 58%       1.2%  at its level · small run
EvalPerf correctness (118 tasks, …   88.6%  ▂·▂▂▂▆▅▆▆▆▃● beats 28/29    easy · avg 74%         1.2%  typical
C-Eval (maker)                       86.5%  ▁···▁▁▂▁▂▂●█ beats 16/33    easy · avg 75%         1.1%  small run
KCL-MCQA vanilla                     37.3%  ▂···█▂▆●██▆▂ beats 9/23     very hard · avg 29%    1.1%  typical
SimpleQA (maker)                     24.9%  ▄▄▄▆█▆██●·▂█ beats 24/32    very hard · avg 11%    1.1%  small run
MMLU-Redux (maker)                   89.1%  ▁▁▃▁▂▃▄▅●▅█▄ beats 31/63    easy · avg 80%         1.1%  small run
ChineseSimpleQA (maker)              68.0%  ▄··▄▄●··▄··█ beats 3/6      medium · avg 47%       0.8%  at its level · small run
FRAMES (maker)                       73.3%  ●··········█ beats 0/1      medium · avg 62%       0.6%  small run
CNMO 2024 (maker)                    43.2%  ●··········█ beats 0/1      medium · avg 47%       0.4%  at its level · noisy · small run
DROP (maker)                         91.6%  ▃▅█▅·▃·▆▄▂▁● beats 43/44    medium · avg 65%       0.3%  blunt · at its level · small run
DROP F1, 3-shot (simple-evals)       92.1%  ▅▂▆█▆·▃·▂▅·● beats 23/23    easy · avg 81%         0.2%  blunt · small run
Vectara hallucination leaderboard…   96.1%  ▁▁▂▃▄▄●▆▃▂▂▁ beats 88/140   very easy · avg 94%    0.1%  blunt · maxed out · noisy
Vectara hallucination leaderboard…   93.9%  ▁▂▁▂▅█▂▃●▃▁▁ beats 76/98    very easy · avg 90%    0.0%  blunt
CLUEWSC (maker)                      90.9%  ●··········█ beats 0/1                                —  not counted yet: no other model in the …
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
deepseek-ai/DeepSeek-R1 model card (sub…       17   24% ▇▇▇
Cohere                                          9   19% ▇▇
Epoch AI                                        3   14% ▇▇
LiveCodeBench official website results          1    9% ▇
Aider polyglot benchmark runs                   1    8% ▇
WildEval ZeroEval published results             1    8% ▇
Korean Canonical Legal Benchmark: Towar…        2    3% ▇
LMArena leaderboard dataset                     1    3% ▇
WebApp1K models leaderboard                     2    3% ▇
SciCode leaderboard                             1    2% ▇
SpeakLeash CPTUB                                1    2% ▇
OpenCompass CompassAcademic leaderboard         2    2% ▇
EvalPlus / EvalPerf published evaluatio…        1    1% ▇
deepseek-ai/DeepSeek-V3-0324 model card…        1    1% ▇
Berkeley Function Calling Leaderboard r…        1    1% ▇
Vectara hallucination leaderboard               2    0% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 59.7
Licences: deepseek-ai/DeepSeek-R1 model card ([mit](../../licenses/deepseek-ai--DeepSeek-R1.txt)) · Cohere ([CC BY 4.0](../../licenses/report-2504.00698v2.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · LiveCodeBench official website results ([CC-BY-SA-4.0](../../licenses/lcb-sharealike-NOTICE.txt)) · Aider polyglot benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs’ Legal Reasoning Capabilities ([CC BY 4.0](../../licenses/cards-final-NOTICE.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · WebApp1K models leaderboard ([MIT](../../licenses/webapp1k-leaderboard-MIT.txt)) · SciCode leaderboard ([Apache-2.0](../../licenses/scicode-Apache-2.0.txt)) · SpeakLeash CPTUB ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · OpenCompass CompassAcademic leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · deepseek-ai/DeepSeek-V3-0324 model card ([MIT](../../licenses/deepseek-ai--DeepSeek-V3-0324.txt)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0))
Not counted (1) 
source                  test     score  why
deepseek-ai/DeepSeek-…  CLUEWSC  90.9%  too few models have it yet
## How to run it ### Through an API api (Deep Infra) · $0.46/M blended · $0.32 in / $0.89 out per 1M tokens · [deepinfra.com](https://deepinfra.com/models) [price source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/providers/deepinfra/models/deepseek-ai/DeepSeek-V3.toml): Standard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ. · [context source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/models/deepseek/deepseek-v3.toml) ### llama.cpp (recommended) 1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) ```sh # macOS / Linux curl -LsSf https://llama.app/install.sh | sh # Windows irm https://llama.app/install.ps1 | iex ``` 2. Run DeepSeek V3 (Dec '24) (Q4_K_M, needs ~404.4 GB, from [huggingface.co/unsloth](https://huggingface.co/unsloth/DeepSeek-V3-GGUF/tree/6b9a45d8a30b48660644560df90e5ded5c0cb0e9)) ```sh llama cli -hf unsloth/DeepSeek-V3-GGUF:Q4_K_M ``` 3. Or start a local server and open the address it prints (http://localhost:9931) ```sh llama serve -hf unsloth/DeepSeek-V3-GGUF:Q4_K_M ``` Other versions: Q8_0 (~713.3 GB) `llama cli -hf unsloth/DeepSeek-V3-GGUF:Q8_0` · Q3_K_M (~319.2 GB) `llama cli -hf unsloth/DeepSeek-V3-GGUF:Q3_K_M` ### LM Studio 1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download) 2. Run DeepSeek V3 (Dec '24) (Q4_K_M, needs ~404.4 GB, from [huggingface.co/unsloth](https://huggingface.co/unsloth/DeepSeek-V3-GGUF/tree/6b9a45d8a30b48660644560df90e5ded5c0cb0e9)) ```sh lms get https://huggingface.co/unsloth/DeepSeek-V3-GGUF@Q4_K_M -y ``` 3. Then run lms ls to see its name, and lms chat followed by that name Other versions: Q8_0 (~713.3 GB) `lms get https://huggingface.co/unsloth/DeepSeek-V3-GGUF@Q8_0 -y` · Q3_K_M (~319.2 GB) `lms get https://huggingface.co/unsloth/DeepSeek-V3-GGUF@Q3_K_M -y` · 4bit (~377.6 GB, Mac only) `lms get https://huggingface.co/mlx-community/DeepSeek-V3-4bit -y` ### Ollama 1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download) ```sh # Linux curl -fsSL https://ollama.com/install.sh | sh ``` 2. Run DeepSeek V3 (Dec '24) (Q4_K_M, needs ~403.1 GB, from [ollama.com](https://ollama.com/library/deepseek-v3/tags)) ```sh ollama run deepseek-v3:671b ``` ### MLX (Mac) 1. Install MLX (Mac). Apple silicon Macs only, text only. ```sh # with uv uv tool install mlx-lm # or pip install mlx-lm ``` 2. Run DeepSeek V3 (Dec '24) (4bit, needs ~377.6 GB, Mac only, from [huggingface.co/mlx-community](https://huggingface.co/mlx-community/DeepSeek-V3-4bit/tree/ece24ebacb978c44706ed25ceaf47d94e199f193)) ```sh mlx_lm.chat --model mlx-community/DeepSeek-V3-4bit ``` 3. Or start an OpenAI-compatible server at http://localhost:8080/v1 ```sh mlx_lm.server --model mlx-community/DeepSeek-V3-4bit ``` Model repository and documentation: [huggingface.co/deepseek-ai](https://huggingface.co/deepseek-ai/DeepSeek-V3) [interactive version](https://whatmodel.sh/model/deepseek-v3) · [JSON](https://whatmodel.sh/api?format=json&all=1)