[whatmodel.sh](../../README.md) / Phi-3.5-mini-instruct
# Phi-3.5-mini-instruct
Microsoft · released 2024-08 · 3.82B params · mit
**score 44.7** ±0.5 · #484 of 732 · from 38 benchmark results across 10 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Phi-3.5-mini-instruct among every model on that task or test, worst to best.
task vs the others rank counts tests
agent tasks ▁▁▃▆●▄▅▅▄▄▂▁ #238 of 313* 0.4% 1
coding ▁▁▁▃▅●█▇▅▄▃▁ #339 of 469 12% 5
chat & writing ●··········█ #2 of 2* 0.4% 1
math ▁▁▃●█▆▄█▄▂▁▁ #362 of 431 21% 5
agent tasks · #238 of 313 · 0.4% of its score
Beats [o3-mini](o3-mini.md), [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [Nova 2 Lite](nova-2-lite.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
BFCL (version not stated) (maker) 66.1% ▄▆●▂·█▆·▄··▂ beats 5/17 easy · avg 76% 0.4% blunt · at its level · small run
coding · #339 of 469 · 12% of its score
Beats [Qwen3-VL 8B Instruct](qwen3-vl-8b-instruct.md), [Qwen3-VL 4B Instruct](qwen3-vl-4b-instruct.md), [LFM2.5 2.6B](lfm2-5-2-6b.md) here, though they rank higher overall. Trails [Phi-3-mini-128k-instruct (Apr 2024)](phi-3-mini-128k-instruct.md), [DeepSeek LLM 67B Chat](deepseek-llm-67b-chat.md), [WizardLM 70B V1.0](wizardlm-70b-v1-0.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
CRUXEval (ZeroEval, 800 problems) 42.1% ▁▂●▃▆█▆▃▂·▁▁ beats 6/47 medium · avg 57% 7.7% sharp · at its level
HumanEval (maker) 70.1% ▁▁▁▁▂▂▆▆●▅▂▁ beats 133/205 easy · avg 82% 1.7% sharp · at its level · small run
HumanEval+ (maker) 62.8% ▁·▁▁▂▃▄●█▅▄▃ beats 62/145 easy · avg 78% 1.6% sharp · at its level · small run
MBPP (maker) 70.0% ▁▁▁▄▅█▆●▄▁▂▁ beats 69/86 easy · avg 74% 0.9% at its level · small run
MBPP+ (maker) 63.8% ▁··▁▁▃▄●▆▄▁▁ beats 57/120 easy · avg 70% 0.6% at its level · small run
chat & writing · #2 of 2 · 0.4% of its score
test result vs the others in its run difficulty counts why
IFEval (aggregation unspecified) … 50.6% ▁▂●▂▂▄▆██▅▂▁ beats 9/145 easy · avg 74% 0.4% at its level · small run
math · #362 of 431 · 21% of its score
Beats [GPT-3.5 Turbo (0125)](gpt-3-5-turbo-0125.md), [DBRX Instruct](dbrx-instruct.md), [Command R Plus (04-2024)](command-r-plus-04-2024.md) here, though they rank higher overall. Trails [OPT-2.7B](opt-2-7b.md), [Gemma 2B (base)](gemma-2b-base.md), [Stable Code 3B](stable-code-3b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MGSM (maker) 49.6% ▅●▃█·▅···▃·▃ beats 2/10 easy · avg 72% 6.1% sharp · at its level
GSM8K (ZeroEval, zero-shot) 82.0% ▁▄·▁▄●▅▅▅███ beats 13/43 very easy · avg 87% 5.5% sharp
GSM8K (maker) 76.9% ●·█▄▄█···█·▄ beats 1/10 very easy · avg 90% 5.3% sharp
MATH Level 5 (ZeroEval, 721 probl… 18.7% ▁▁▁●█▂▇▇▄··▂ beats 6/38 hard · avg 30% 2.2% sharp
MATH (maker) 49.8% █▄█●···▄▄▄·▄ beats 5/10 medium · avg 61% 2.0% at its level
other tests (26)
test result vs the others in its run difficulty counts why
MMLU-Redux (ZeroEval, 2,778 quest… 67.7% ▁·▁▁▄●▄▅█▄▂▁ beats 12/53 easy · avg 75% 14% at its level
LiveBench 2024-07-26 28.3% ▁▁▁▂▂▃▂●▆▃▅▃ beats 23/61 medium · avg 38% 10% at its level
ARC Challenge (maker) 84.6% ▂▂▂█●···▄··▂ beats 7/10 very easy · avg 89% 5.1% sharp
CLIcK Korean (Daekeun Kim) 43.0% ●▄▄▄█▄▂·▄█·▄ beats 1/23 medium · avg 56% 4.7% at its level
MMLU (maker) 65.5% █▄█●▄▄·▄▄··▄ beats 5/10 easy · avg 75% 4.4% at its level
HAE-RAE 1.0 (Daekeun Kim) 44.2% ●▄▄▂▂█▂▄▂█▂▄ beats 0/22 medium · avg 61% 4.3% at its level
BIG-Bench Hard (maker) 63.1% ██▄·●▄·▄▄··▄ beats 5/10 medium · avg 71% 3.5% sharp · at its level
OpenBookQA (maker) 81.2% ▂··▄●▂·▂···▄ beats 6/10 easy · avg 85% 2.8% at its level
MMLU-Pro (maker) 47.4% █▄▄·█●▄▄·▄·▄ beats 6/10 medium · avg 54% 2.5% at its level
CPTUB average (SpeakLeash, 0-5 ju… 40.2% ▁▂●▂▃█▃▆▅▄▂▃ beats 9/95 medium · avg 65% 2.3% at its level
Multilingual MMLU (Microsoft) (ma… 51.8% █▄●██··█···▄ beats 3/10 medium · avg 63% 2.2% at its level
ZebraLogic (ZeroEval, 1,000 puzzl… 6.4% ▁▁●▆██▁▁·▁▁▁ beats 3/57 hard · avg 15% 1.9% sharp · too hard for it
BRIDGE clinical text, zero-shot a… 25.4% ▁▁▁▁▁▂●▃▄▆█▃ beats 20/97 hard · avg 31% 1.4% blunt · at its level
HellaSwag (maker) 72.2% ▃●█·▃▃▃···▃▃ beats 1/10 easy · avg 82% 1.1% at its level
WinoGrande (maker) 72.2% ▃····▃▃█▃·●▅ beats 8/10 medium · avg 71% 0.9% at its level
Social IQA (maker) 75.1% ▃·▃▅▅●·▃···▃ beats 6/10 medium · avg 76% 0.8% blunt · at its level
BoolQ (maker) 77.7% ▃··▃··●█▃·▃▃ beats 2/10 easy · avg 83% 0.7% at its level
Open PT-LLM Leaderboard average (… 66.8% ▃▁▁▁▁▁▁▁▂●▅▁ beats 131/276 medium · avg 72% 0.7% at its level · noisy
GPQA (subset unspecified) (maker) 26.6% █··▄●··▂▂··▄ beats 6/10 very hard · avg 32% 0.6% sharp · too hard for it · noisy
BigCodeBench (Complete) (maker) 40.4% ▂▁▁▃▂▅●█▅▂▂▂ beats 29/60 medium · avg 49% 0.5% at its level · small run
KMMLU-HARD (Daekeun Kim, zero-sho… 24.0% ●···▁▁▁▂▃·▆▂ beats 4/23 very hard · avg 29% 0.4% sharp · too hard for it
BigCodeBench (Instruct) (maker) 14.3% ▁●··▁▁▂▅█▅▅▅ beats 1/49 hard · avg 34% 0.3% at its level · small run
PIQA (maker) 78.2% ▃·▅·▃●·▃▅··▃ beats 5/10 medium · avg 82% 0.3% blunt · at its level
Vectara hallucination leaderboard… 95.9% ▁▁▂▃▄▄●▆▃▂▂▁ beats 85/140 very easy · avg 94% 0.2% blunt · noisy
LIBRA Mini (Russian long context,… 28.3% ●···▄█·██··▄ beats 0/8 medium · avg 38% 0.1% blunt · at its level
TruthfulQA MC2 (maker) 65.2% ▂·█●▂▄····▂▂ beats 5/10 medium · avg 64% 0.1% blunt · at its level
source results share of the score
microsoft/Phi-4-mini-instruct model car… 16 38% ▇▇▇▇▇
WildEval ZeroEval published results 5 31% ▇▇▇▇
LiveBench model judgments 1 10% ▇
Daekeun Kim Korean LLM benchmark runs 3 9% ▇
Microsoft (submitted / maker-reported) 8 6% ▇
SpeakLeash CPTUB 1 2% ▇
BRIDGE Medical Leaderboard 1 1% ▇
Open Portuguese LLM Leaderboard 1 1% ▇
Vectara hallucination leaderboard 1 0% ▇
ai-forever LIBRA leaderboard 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 44.7
Licences: microsoft/Phi-4-mini-instruct model card ([MIT](../../licenses/card-microsoft-phi-4-mini-instruct.txt)) · WildEval ZeroEval published results ([Apache-2.0](../../licenses/zeroeval-published-results-Apache-2.0.txt)) · LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · Daekeun Kim Korean LLM benchmark runs ([MIT](../../licenses/korean-personal-benchmark-runs-MIT.txt)) · Microsoft ([CC BY 4.0](../../licenses/report-2503.01743v2.txt)) · SpeakLeash CPTUB ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt)) · BRIDGE Medical Leaderboard ([MIT](../../licenses/bridge-medical-leaderboard-MIT.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · ai-forever LIBRA leaderboard ([MIT](../../licenses/hf-leaderboard-spaces-MIT.txt))
## How to run it
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run Phi-3.5-mini-instruct (Q4_K_M, needs ~2.4 GB, from [huggingface.co/bartowski](https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/tree/6d70da17e749a471ccb62ade694486011a75cda3))
```sh
llama cli -hf bartowski/Phi-3.5-mini-instruct-GGUF:Q4_K_M
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf bartowski/Phi-3.5-mini-instruct-GGUF:Q4_K_M
```
Other versions: Q8_0 (~4.1 GB) `llama cli -hf bartowski/Phi-3.5-mini-instruct-GGUF:Q8_0`
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run Phi-3.5-mini-instruct (Q4_K_M, needs ~2.4 GB, from [huggingface.co/bartowski](https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/tree/6d70da17e749a471ccb62ade694486011a75cda3))
```sh
lms get https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF@Q4_K_M -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
Other versions: Q8_0 (~4.1 GB) `lms get https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF@Q8_0 -y` · 4bit (~2.1 GB, Mac only) `lms get https://huggingface.co/mlx-community/Phi-3.5-mini-instruct-4bit -y` · 8bit (~4.1 GB, Mac only) `lms get https://huggingface.co/mlx-community/Phi-3.5-mini-instruct-8bit -y`
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run Phi-3.5-mini-instruct (Q4_K_M, needs ~2.79 GB, from [ollama.com](https://ollama.com/library/phi3.5/tags))
```sh
ollama run phi3.5:3.8b-mini-instruct-q4_K_M
```
Other versions: Q8_0 (~5.08 GB) `ollama run phi3.5:3.8b-mini-instruct-q8_0`
### MLX (Mac)
1. Install MLX (Mac). Apple silicon Macs only, text only.
```sh
# with uv
uv tool install mlx-lm
# or
pip install mlx-lm
```
2. Run Phi-3.5-mini-instruct (4bit, needs ~2.1 GB, Mac only, from [huggingface.co/mlx-community](https://huggingface.co/mlx-community/Phi-3.5-mini-instruct-4bit/tree/7b2052fd882fe017300d4d42a4eb06a27b816af4))
```sh
mlx_lm.chat --model mlx-community/Phi-3.5-mini-instruct-4bit
```
3. Or start an OpenAI-compatible server at http://localhost:8080/v1
```sh
mlx_lm.server --model mlx-community/Phi-3.5-mini-instruct-4bit
```
Other versions: 8bit (~4.1 GB, Mac only) `mlx_lm.chat --model mlx-community/Phi-3.5-mini-instruct-8bit`
Model repository and documentation: [huggingface.co/microsoft](https://huggingface.co/microsoft/Phi-3.5-mini-instruct)
[interactive version](https://whatmodel.sh/model/phi-3-5-mini-instruct) · [JSON](https://whatmodel.sh/api?format=json&all=1)