[whatmodel.sh](../../README.md) / Phi-3.5-vision-instruct # Phi-3.5-vision-instruct Microsoft · released 2024-08 · 4.15B params · mit **score 38.7** ±1.5 · #582 of 732 · from 21 benchmark results across 5 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Phi-3.5-vision-instruct among every model on that task or test, worst to best. 
task                vs the others          rank  counts  tests
chat & writing      ▁▁▁▃▆●▆█▆▇▇▃   #312 of 402*    5.0%      1
math                ▁▁▃▅●▆▄█▄▂▁▁   #324 of 431*    2.8%      1
vision & documents  ●▁▂▂▁▃▄▂▆▇█▄    #150 of 153     71%     15
chat & writing · #312 of 402 · 5.0% of its score Beats [Gemma 4 E4B](gemma-4-e4b.md), [OpenThinker3 7B](openthinker3-7b.md), [Code Llama 70B Instruct](codellama-70b-instruct.md) here, though they rank higher overall. Trails [LFM2 350M](lfm2-350m.md), [Gemma 3 1B](gemma-3-1b.md), [Qwen3 0.6B](qwen3-0-6b.md) here, though they rank lower overall. 
test                              result  vs the others in its run  difficulty      counts  why
IFEval (aggregation unspecified)   54.3%  ●···▄█▄····▄ beats 0/5    easy · avg 78%    5.0%  at its level
math · #324 of 431 · 2.8% of its score Beats [GPT-4 (0613)](gpt-4-0613.md), [Athene-70B](athene-70b.md), [Gemma 2 27B IT](gemma-2-27b.md) here, though they rank higher overall. Trails [Stable Code 3B](stable-code-3b.md), [StarCoder2 7B](starcoder2-7b.md), [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md) here, though they rank lower overall. 
test                          result  vs the others in its run  difficulty        counts  why
MathVista (testmini) (maker)   43.9%  ▁··▁●▄▆▇▄▅█▄ beats 3/48   medium · avg 52%    2.8%  noisy
vision & documents · #150 of 153 · 71% of its score Beats [Gemma 3 4B](gemma-3-4b.md), [Phi-3-vision-128k-instruct](phi-3-vision-128k-instruct.md) here, though they rank higher overall. Trails [Qwen3.5 0.8B](qwen3-5-0-8b.md) here, though they rank lower overall. 
test                                result  vs the others in its run  difficulty        counts  why
Arena vision (style controlled)      919.0  ▁●▂▂▂▂▃▄▅▅█▇ beats 1/152  avg 1061             15%  blunt
CharXiv descriptive questions        63.7%  █·●·······▄█ beats 2/5    easy · avg 78%       14%  at its level
MMStar                               49.5%  ●▄▄█·······▄ beats 0/5    medium · avg 61%    7.8%  at its level
MMMU (val) (maker)                   43.0%  ▁····▁▁●█▆▆▅ beats 4/53   medium · avg 51%    5.4%  typical
MMBench-EN dev v1.1                  73.1%  █·····●··▃·▃ beats 3/5    easy · avg 79%      5.4%  blunt · at its level
HallusionBench                       58.5%  ●▄·▄·····█·▄ beats 0/5    easy · avg 66%      4.2%  blunt · at its level
MathVista (testmini) (maker)         43.9%  ▁··▁●▄▆▇▄▅█▄ beats 3/48   medium · avg 52%    2.8%  noisy
CountBench                           64.5%  ▄··●··█··▄·▄ beats 1/5    easy · avg 84%      2.6%  at its level · noisy
RealWorldQA                          58.0%  ▄···●··▄▄··▄ beats 1/5    easy · avg 68%      2.3%  blunt · at its level
Ai2 Molmo human preference Elo (S…   982.0  ▄●·▄·▄▄▄·▄·█ beats 1/8    avg 1032            2.2%  blunt
PhysBench physical-world understa…   39.8%  ▅▃●··▅▃····▃ beats 5/9    hard · avg 42%      2.1%  blunt
AI2D test                            79.0%  █···▄▄···▄·● beats 5/5    easy · avg 81%      1.9%  blunt · at its level · several runs
OCRBench                             64.2%  ●··███·█···█ beats 0/5    easy · avg 81%      1.7%  blunt · at its level
ChartQA                              82.1%  ▄······▄··█● beats 5/5    easy · avg 79%      1.5%  blunt · at its level · noisy · several …
TextVQA (maker)                      72.0%  ▁▁▁▄▅●▃·▃▃▃█ beats 10/24  easy · avg 75%      1.2%  blunt · at its level
other tests (3) 
test                                result  vs the others in its run  difficulty        counts  why
MMLU                                 35.5%  ●·····▄▄··▄█ beats 0/5    medium · avg 69%     11%  at its level
MMLU-Pro                             28.6%  ●▄·······▄▄▄ beats 0/5    medium · avg 49%    9.0%  typical
Multi-IF (aggregation unspecified)   30.4%  ●·▄··█▄····▄ beats 0/5    medium · avg 57%    4.1%  at its level
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
CohereLabs/North-Micro-Vision-Instruct …       13   68% ▇▇▇▇▇▇▇▇
LMArena leaderboard dataset                     1   15% ▇▇
Microsoft Phi-3.5-vision-instruct model…        5   11% ▇
Ai2 Molmo-72B-0924 model card: Human Pr…        1    2% ▇
PhysBench: Benchmarking and Enhancing V…        1    2% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 38.7
Licences: CohereLabs/North-Micro-Vision-Instruct model card ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Microsoft Phi-3.5-vision-instruct model card: Benchmarks table ([MIT](https://huggingface.co/microsoft/Phi-3.5-vision-instruct/blob/12b77fb40b63a2c73c68243d3f767aab688a1b2a/LICENSE)) · Ai2 Molmo-72B-0924 model card: Human Preference Elo Rating column ([Apache-2.0](https://huggingface.co/allenai/Molmo-72B-0924/blob/ab586dd9640de059566975b8e5bcd0c6fdc73956/README.md)) · PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) ## How to run it ### Other ways to run it 1. Run it as published (BF16, needs ~8.29 GB, NVIDIA only, from [github.com/vllm-project](https://github.com/vllm-project/vllm/blob/64cb68384241f046d07ba3797e1b14dbc82a0b2d/docs/models/supported_models.md)) ```sh vllm serve microsoft/Phi-3.5-vision-instruct --trust-remote-code ``` Model repository and documentation: [huggingface.co/microsoft](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) [interactive version](https://whatmodel.sh/model/phi-3-5-vision-instruct) · [JSON](https://whatmodel.sh/api?format=json&all=1)