[whatmodel.sh](../../README.md) / Command R Plus (04-2024) # Command R Plus (04-2024) Cohere · released 2024-04 · 104B params · cc-by-nc-4.0 **score 46.0** ±0.5 · #455 of 732 · from 20 benchmark results across 12 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Command R Plus (04-2024) among every model on that task or test, worst to best. 
task            vs the others          rank  counts  tests
agent tasks     ▁▁▃●█▄▅▅▄▄▂▁   #275 of 313*    0.6%      1
coding          ▁▁▁▃▅●█▇▅▄▃▁    #348 of 469     46%      5
chat & writing  ▁▁▁▃▆▄●█▆▇▇▃    #275 of 402     11%      3
math            ▁▁▃●█▆▄█▄▂▁▁   #380 of 431*    8.9%      2
agent tasks · #275 of 313 · 0.6% of its score Beats [Magistral Small 1.0 (2506)](magistral-small-2506.md), [Llama 4 Scout (109B-A17B)](llama-4-scout.md), [Phi-4 14B](phi-4.md) here, though they rank higher overall. Trails [Granite 4.0 H 1B](granite-4-0-h-1b.md), [Granite 4.0 1B](granite-4-0-1b.md), [SmolLM3 3B](smollm3-3b.md) here, though they rank lower overall. 
test     result  vs the others in its run   difficulty        counts  why
BFCL v3   48.0%  ▁▁▁▁▁▃●█▅▄▁▁ beats 31/114  medium · avg 51%    0.6%  blunt · at its level
coding · #348 of 469 · 46% of its score Beats [LFM2.5 2.6B](lfm2-5-2-6b.md), [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md), [Qwen3-VL 2B Thinking](qwen3-vl-2b-thinking.md) here, though they rank higher overall. Trails [Phi-3-mini-128k-instruct (Apr 2024)](phi-3-mini-128k-instruct.md), [DeepSeek LLM 67B Chat](deepseek-llm-67b-chat.md), [WizardLM 70B V1.0](wizardlm-70b-v1-0.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty        counts  why
HumanEval                            64.0%  ▁·▁▁▃▄▆●█▃▁▁ beats 49/99  easy · avg 76%       19%  sharp · at its level
HumanEval+                           56.7%  ▁·▁▁▁▃▄▄●█▃▁ beats 47/99  easy · avg 70%       16%  sharp · at its level
Aider code editing (133 Exercism …   31.6%  ▁·▁●▄▁▆▄█▆▂▂ beats 4/62   medium · avg 51%    5.0%  sharp · at its level
HumanEval (Vector Institute, Insp…   62.2%  ●·▄·█·█·█·▄▄ beats 0/9    easy · avg 77%      3.5%  sharp · at its level
MBPP+                                63.5%  ▁···▁▁▂▅▇●▆▂ beats 46/75  easy · avg 66%      3.0%  at its level
chat & writing · #275 of 402 · 11% of its score Beats [Ministral 3 14B Reasoning 2512](ministral-3-14b-reasoning.md), [Ministral 3 14B](ministral-3-14b.md), [Phi-4 14B](phi-4.md) here, though they rank higher overall. Trails [Flan-T5 XXL](flan-t5-xxl.md), [LFM2 1.2B](lfm2-1-2b.md), [North Micro Vision Instruct (2.4B)](north-micro-vision-instruct.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty        counts  why
Arena-Hard (v0.1, GPT-4-1106 judg…   33.1%  ▂▁▁▅█●▄▃▃▅▃▁ beats 28/60   medium · avg 47%    5.6%  sharp · at its level
LMArena (text)                      1261.7  ▁▁▁▂▃▃●▆▆▇█▃ beats 95/385  avg 1260            2.7%  blunt
Polyrating rating (bias-aware re-…  1214.0  ▂▁▂▁▃▇█▆▇●▂▂ beats 86/95   avg 1187            2.5%  typical
math · #380 of 431 · 8.9% of its score Trails [GPT-Neo 1.3B](gpt-neo-1-3b.md), [OPT-1.3B](opt-1-3b.md), [OPT-2.7B](opt-2-7b.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty           counts  why
GSM8K (ZeroEval, zero-shot)          80.1%  ▁▄·▁▄●▅▅▅███ beats 11/43  very easy · avg 87%    4.7%  sharp
MATH (Vector Institute, Inspect E…   26.3%  ●··██▄··▄·▄█ beats 0/9    medium · avg 50%       4.2%  sharp · at its level
other tests (9) 
test                                          result  vs the others in its run  difficulty           counts  why
MMLU-Redux (ZeroEval, 2,778 quest…             68.6%  ▁·▁▁▄●▄▅█▄▂▁ beats 14/53  easy · avg 75%          13%  at its level
LiveBench 2024-07-26                           32.2%  ▁▁▁▂▂▃▂●▆▃▅▃ beats 32/61  medium · avg 38%        10%  at its level
WildBench v2 WB-Score (task macro)  36.7624 WB-Score  ▁·▁▁·▃█▄●▆▅▆ beats 22/52  avg 43.019 WB-Score    2.8%  sharp
MMLU-Pro (Vector Institute, Inspe…             44.1%  ●····▃▅·▃▅·█ beats 0/9    medium · avg 58%       2.5%  at its level
ZebraLogic (ZeroEval, 1,000 puzzl…             13.9%  ▁▁▃●██▁▁·▁▁▁ beats 19/57  hard · avg 15%         2.1%  sharp
GPQA Diamond (Vector Institute, I…             31.9%  ●··▃▃▃▃·▃·▃█ beats 0/9    hard · avg 40%         1.8%  sharp
Open Persian LLM Leaderboard v2 a…             60.2%  ▂▄▂▃█▄▃●▃▅▂▂ beats 51/81  medium · avg 57%       1.3%  blunt · at its level
RULER 64k                                      84.3%  ▁▁·▁·▂▃▄●▂·▁ beats 12/22  easy · avg 82%         0.7%  typical
RULER 128k                                     63.1%  ▄··▂▄▂●██▄▄▂ beats 6/21   easy · avg 67%         0.2%  at its level · noisy
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
EvalPlus / EvalPerf published evaluatio…        3   37% ▇▇▇▇
WildEval ZeroEval published results             3   20% ▇▇
Vector Institute Evaluation Leaderboard         4   12% ▇
LiveBench model judgments                       1   10% ▇
Arena-Hard-Auto v0.1 leaderboard                1    6% ▇
Aider code editing benchmark runs               1    5% ▇
WildBench v2 leaderboard data                   1    3% ▇
LMArena leaderboard dataset                     1    3% ▇
Polyrating: A Cost-Effective and Bias-A…        1    2% ▇
Open Persian LLM Leaderboard v2                 1    1% ▇
NVIDIA RULER repository README results …        2    1% ▇
Berkeley Function Calling Leaderboard r…        1    1% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 46.0
Licences: EvalPlus / EvalPerf published evaluation runs ([Apache-2.0](../../licenses/evalplus-published-runs-Apache-2.0.txt)) · WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · Vector Institute Evaluation Leaderboard ([Apache-2.0](../../licenses/vector-inspect-evals-leaderboard-Apache-2.0.txt)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · Aider code editing benchmark runs ([Apache-2.0](../../licenses/aider-polyglot-runs-Apache-2.0.txt)) · WildBench v2 leaderboard data ([Apache-2.0](https://github.com/allenai/WildBench/blob/d6b8dcaf377d173d031980f97c16e1a82618c03d/LICENSE)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Open Persian LLM Leaderboard v2 ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · NVIDIA RULER repository README results table ([Apache-2.0](https://github.com/NVIDIA/RULER/blob/c3f5e3b4f87f97e048793bb510a3a6b19a46bf3a/LICENSE)) · Berkeley Function Calling Leaderboard result archive ([Apache-2.0](../../licenses/bfcl-result-archive-v4-Apache-2.0.txt)) ## How to run it ### llama.cpp (recommended) 1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) ```sh # macOS / Linux curl -LsSf https://llama.app/install.sh | sh # Windows irm https://llama.app/install.ps1 | iex ``` 2. Run Command R Plus (04-2024) (Q4_K_M, needs ~62.8 GB, from [huggingface.co/bartowski](https://huggingface.co/bartowski/c4ai-command-r-plus-GGUF/tree/3cbdfe100b0b213db196f164a7d7a8f7326f62dc)) ```sh llama cli -hf bartowski/c4ai-command-r-plus-GGUF:Q4_K_M ``` 3. Or start a local server and open the address it prints (http://localhost:9931) ```sh llama serve -hf bartowski/c4ai-command-r-plus-GGUF:Q4_K_M ``` Other versions: Q2_K (~39.5 GB) `llama cli -hf bartowski/c4ai-command-r-plus-GGUF:Q2_K` ### LM Studio 1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download) 2. Run Command R Plus (04-2024) (Q4_K_M, needs ~62.8 GB, from [huggingface.co/bartowski](https://huggingface.co/bartowski/c4ai-command-r-plus-GGUF/tree/3cbdfe100b0b213db196f164a7d7a8f7326f62dc)) ```sh lms get https://huggingface.co/bartowski/c4ai-command-r-plus-GGUF@Q4_K_M -y ``` 3. Then run lms ls to see its name, and lms chat followed by that name Other versions: Q2_K (~39.5 GB) `lms get https://huggingface.co/bartowski/c4ai-command-r-plus-GGUF@Q2_K -y` · 4bit (~62.9 GB, Mac only) `lms get https://huggingface.co/mlx-community/c4ai-command-r-plus-4bit -y` · 8bit (~113.2 GB, Mac only) `lms get https://huggingface.co/mlx-community/c4ai-command-r-plus-8bit -y` ### Ollama 1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download) ```sh # Linux curl -fsSL https://ollama.com/install.sh | sh ``` 2. Run Command R Plus (04-2024) (Q4_0, needs ~62.9 GB, from [ollama.com](https://ollama.com/library/command-r-plus/tags)) ```sh ollama run command-r-plus:104b-q4_0 ``` Other versions: Q8_0 (~125.3 GB) `ollama run command-r-plus:104b-q8_0` ### MLX (Mac) 1. Install MLX (Mac). Apple silicon Macs only, text only. ```sh # with uv uv tool install mlx-lm # or pip install mlx-lm ``` 2. Run Command R Plus (04-2024) (4bit, needs ~62.9 GB, Mac only, from [huggingface.co/mlx-community](https://huggingface.co/mlx-community/c4ai-command-r-plus-4bit/tree/2d8252fcb544706b46d52dedd73f35aa66d1a90c)) ```sh mlx_lm.chat --model mlx-community/c4ai-command-r-plus-4bit ``` 3. Or start an OpenAI-compatible server at http://localhost:8080/v1 ```sh mlx_lm.server --model mlx-community/c4ai-command-r-plus-4bit ``` Other versions: 8bit (~113.2 GB, Mac only) `mlx_lm.chat --model mlx-community/c4ai-command-r-plus-8bit` Model repository and documentation: [huggingface.co/CohereLabs](https://huggingface.co/CohereLabs/c4ai-command-r-plus) [interactive version](https://whatmodel.sh/model/command-r-plus-04-2024) · [JSON](https://whatmodel.sh/api?format=json&all=1)