[whatmodel.sh](../../README.md) / Qwen3.5 397B-A17B # Qwen3.5 397B-A17B Alibaba · released 2026-02 · 397B params (17B active) · apache-2.0 **score 69.4** ±0.4 · #84 of 732 · from 128 benchmark results across 21 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Qwen3.5 397B-A17B among every model on that task or test, worst to best. 
task                vs the others         rank  counts  tests
agent tasks         ▁▁▃▆█▄▅▅●▄▂▁    #71 of 313     21%     21
coding              ▁▁▁▃▅▇█▇●▄▃▁    #89 of 469     15%      8
chat & writing      ▁▁▁▃▆▄▆█▆▇●▃   #67 of 402*    2.0%      2
math                ▁▁▃▅█▆▄●▄▂▁▁    #89 of 431     20%     14
vision & documents  ▁▁▂▂▁▃▄▂▆▇●▄    #38 of 153    7.4%     21
web search          ▁▁▁●▄█▃▄·▁·▁     #63 of 84    3.6%      5
agent tasks · #71 of 313 · 21% of its score Beats [LongCat 2.0 (1.6T-A48B)](longcat-2-0.md), [GPT-5 Pro](gpt-5-pro.md), [GPT-5](gpt-5.md) here, though they rank higher overall. Trails [MiMo-V2.5 (310B-A15B)](mimo-v2-5.md) here, though they rank lower overall. 
test                                result  vs the others in its run    difficulty            counts  why
SWE-bench Verified                   73.6%  █····█●·████ beats 2/6      hard · avg 21%          4.6%  sharp · at its level · several runs
SWE-bench Multilingual               70.9%  ▄··●▄···▄▄·█ beats 1/6      hard · avg 30%          2.1%  at its level · several runs
Terminal-Bench 2.1                   49.9%  ●··▄█·▄····▄ beats 1/6      very hard · avg 13%     1.4%  sharp · at its level
Humanity's Last Exam with tools      48.3%  ▄····▄·●·▄·▄ beats 3/5      hard · avg 15%          1.3%  at its level · several runs
ContractBench success rate (33 ob…   70.7%  ▆··▆▄▂▄▂▆▆●▄ beats 20/23    very hard · avg 13%     1.3%  sharp · at its level
BrowseComp                           40.5%  ●·▄▄···▄··█▄ beats 0/6      hard · avg 18%          1.1%  at its level · several runs
τ³-bench banking_knowledge (v1.0.…    9.8%  ●▃▂▃·▆▃▂█▅▅▂ beats 0/25     very hard · avg 4.3%    1.0%  typical
τ²-bench aggregate (maker)           86.7%  ▄····█●▄···▄ beats 3/5      medium · avg 45%        0.9%  typical
τ³-bench telecom (tau2-bench ≥ v1…   97.8%  █·▄█▄······● beats 6/6      medium · avg 64%        0.9%  several runs
SWE-bench Pro public set, Qwen-re…   50.9%  ▄······▄●·▄▄ beats 2/5      hard · avg 19%          0.8%  at its level
τ³-bench airline (tau2-bench ≥ v1…   81.5%  ▃······▃·●█▃ beats 2/6      medium · avg 51%        0.7%  typical
τ³-bench                             71.0%  ▄····▄·●·▄·█ beats 3/6      medium · avg 42%        0.7%  at its level
BrowseComp-ZH (maker)                70.3%  ▁▁▁▁▁▃·▁▄▅▄● beats 22/23    hard · avg 21%          0.6%  at its level · small run
τ³-bench retail (tau2-bench ≥ v1.…   84.4%  ▄▄·█··▄·▄··● beats 6/6      medium · avg 48%        0.6%  typical
OSWorld-Verified (maker)             62.2%  █▃·▂▅●▃▂▅▃▆▆ beats 11/27    hard · avg 18%          0.6%  at its level · small run
EEGAgentBench overall (tool-based…   50.1%  ▂·▁▁··▇▇●▂·▁ beats 20/25    hard · avg 28%          0.5%  blunt · at its level
Terminal-Bench 2.0 (maker)           52.5%  ▂▂▅▅██●█▆▂▃▃ beats 22/36    very hard · avg 8.9%    0.4%  at its level · small run
MCPMark (maker)                      46.1%  ▃·····█··▃█● beats 8/8      very hard · avg 11%     0.4%  at its level · small run
BFCL V4 (December 2025 archive) (…   72.9%  ▁▁▁▂▆▆▅▆█▂▄● beats 111/114  medium · avg 39%        0.3%  at its level · small run
Toolathlon (Tool Decathlon) (make…   38.3%  ▂▂▄▂▄▄●▂█▆██ beats 9/27     very hard · avg 10%     0.3%  at its level · small run
WideSearch (maker)                   74.0%  █·▂▆··●···▂▂ beats 8/10     medium · avg 36%        0.3%  at its level · small run
coding · #89 of 469 · 15% of its score Beats [Grok 4.20 (0309 reasoning)](grok-4-20.md), [GPT-5.4 mini](gpt-5-4-mini.md), [GPT-5.1](gpt-5-1.md) here, though they rank higher overall. Trails [Llama 3.1 Nemotron Ultra 253B v1](llama-3-1-nemotron-ultra-253b.md), [GLM-4.7](glm-4-7.md), [GPT-5.4 nano](gpt-5-4-nano.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty        counts  why
SWE-bench Verified                   73.6%  █····█●·████ beats 2/6     hard · avg 21%      4.6%  sharp · at its level · several runs
Arena WebDev (Code Arena)           1400.2  ▁▁▃▂▅●▃▆▅▅▁▂ beats 46/129  avg 981             3.0%  sharp
SWE-bench Multilingual               70.9%  ▄··●▄···▄▄·█ beats 1/6     hard · avg 30%      2.1%  at its level · several runs
LiveCodeBench v6 (Feb–May 2025)      82.6%  ▄··█····●▄·█ beats 3/6     medium · avg 48%    2.0%  several runs
LiveCodeBench v6 (window unspecif…   79.3%  █●···█··████ beats 1/6     medium · avg 46%    0.9%  typical
SciCode subproblems                  48.0%  ▃····▃··●·▃▃ beats 3/6     hard · avg 19%      0.9%  at its level
SciCode (level unspecified)          42.0%  ▄·····▄●▄▄·▄ beats 3/6     hard · avg 16%      0.8%  at its level
SWE-bench Pro public set, Qwen-re…   50.9%  ▄······▄●·▄▄ beats 2/5     hard · avg 19%      0.8%  at its level
chat & writing · #67 of 402 · 2.0% of its score Beats [Claude Haiku 5.5](claude-haiku-5-5.md), [DeepSeek V4 Flash (0731)](deepseek-v4-flash.md), [GPT-5.2](gpt-5-2.md) here, though they rank higher overall. Trails [Nemotron Cascade 2 30B-A3B](nemotron-cascade-2-30b-a3b.md), [Gemma 4 31B](gemma-4-31b.md), [Gemini 2.5 Pro](gemini-2-5-pro.md) here, though they rank lower overall. 
test                                result  vs the others in its run    difficulty      counts  why
LMArena (text)                      1441.4  ▁▁▁▂▃▃▃▆▆▇●▃ beats 306/385  avg 1260          1.7%  blunt
IFEval (aggregation unspecified) …   92.6%  ▁▂▄▂▂▄▆██●▂▁ beats 129/145  easy · avg 79%    0.3%  maxed out · small run
math · #89 of 431 · 20% of its score Beats [Claude Sonnet 4.6](claude-sonnet-4-6.md), [Claude Opus 4.5](claude-opus-4-5.md) here, though they rank higher overall. Trails [Gemini 2.5 Flash](gemini-2-5-flash.md), [Granite 4.2 30B](granite-4-2-30b.md), [Qwen3-Next 80B-A3B Instruct](qwen3-next-80b-a3b-instruct.md) here, though they rank lower overall. 
test                                result  vs the others in its run    difficulty            counts  why
FrontierMath tiers 1–3 v2            29.5%  ▂··▁▁▃●▄█▆▄▃ beats 20/86    very hard · avg 1.2%    7.9%  sharp · at its level
HMMT February 2026 (maker)           87.9%  ▃▃▂▅·████●▃▅ beats 30/35    medium · avg 36%        2.8%  sharp · several runs
AIME 2026                            92.5%  ▄·▄·▄●··█··▄ beats 3/6      medium · avg 53%        1.5%  sharp · maxed out · small run · several…
IMO-AnswerBench                      83.1%  ▄····●·▄▄·█▄ beats 1/6      medium · avg 36%        1.5%  small run · several runs
MathArena ArXivMath January 2026 …   54.9%  ▂··▄▂▄█●▆▄▄█ beats 10/23    hard · avg 23%          1.4%  at its level
MathArena ArXivMath December 2025…   38.2%  ▃·▃●█▁·▄·▁▁▁ beats 4/18     hard · avg 19%          1.0%  at its level
MATH-Vision (maker)                  88.6%  █·····▄▄·▄·● beats 5/5      medium · avg 47%        0.6%  typical
OTIS mock AIME 2024–2025             88.9%  ▃▃▄▃▃▃▇▇●▃▂▅ beats 141/197  very hard · avg 7.2%    0.6%  sharp · noisy
HMMT February 2025 (maker)           94.8%  ▁▁▁▃▁▅▅▃█●▁▄ beats 26/31    easy · avg 74%          0.5%  maxed out · small run
MathVista (testmini) (maker)         90.3%  ▁··▁▄▄▆▇▄▅█● beats 48/48    medium · avg 66%        0.4%  small run
HMMT November 2025 (maker)           92.7%  ▁▁····▁▅▃█●▁ beats 17/23    easy · avg 73%          0.4%  maxed out · small run
PolyMATH (maker)                     73.3%  ▁···▃▁·▇█▄▄● beats 23/23    hard · avg 32%          0.4%  at its level · small run
DynaMath (maker)                     86.3%  ▂·····▂··█▄● beats 9/10     medium · avg 65%        0.3%  small run
We-Math (maker)                      87.9%  █··········● beats 3/3      medium · avg 58%        0.3%  small run
vision & documents · #38 of 153 · 7.4% of its score Beats [MiMo-V2.6-Pro](mimo-v2-6-pro.md), [MiMo-V2.6-Flash (309B-A15B)](mimo-v2-6-flash.md), [DeepSeek V4.1 Flash](deepseek-v4-1-flash.md) here, though they rank higher overall. Trails [Qwen3.6 Flash](qwen3-6-flash.md), [Qwen3.6 Plus](qwen3-6-plus.md) here, though they rank lower overall. 
test                                result  vs the others in its run    difficulty           counts  why
CharXiv (reasoning questions) (ma…   80.8%  ▄▂▄▆▄▄▆··●█▂ beats 15/21    medium · avg 38%       2.1%  typical
Arena vision (style controlled)     1245.8  ▁▁▂▂▂▂▃▄▅▅●▇ beats 102/152  avg 1061               1.1%  blunt
MATH-Vision (maker)                  88.6%  █·····▄▄·▄·● beats 5/5      medium · avg 47%       0.6%  typical
BEAR-Bench (bilingual RU/EN enter…   75.4%  ▃▃·▅·▃·█·█▅● beats 14/14    hard · avg 24%         0.6%  at its level
MMMU (val) (maker)                   85.0%  ▁····▁▁▃█▆▆● beats 53/53    medium · avg 59%       0.4%  small run
MathVista (testmini) (maker)         90.3%  ▁··▁▄▄▆▇▄▅█● beats 48/48    medium · avg 66%       0.4%  small run
MMMU Pro (maker)                     79.0%  ▂▃▁▃▆▇▃▇▇█●▅ beats 48/52    medium · avg 47%       0.4%  at its level · small run
MMStar (maker)                       83.8%  ▃▅▂▃▂▂▆██▆█● beats 35/35    medium · avg 63%       0.3%  small run
SimpleVQA (maker)                    67.1%  ▃·▃·▃·▃·█▅▃● beats 10/10    medium · avg 38%       0.3%  blunt · at its level · small run
CountBench (maker)                   97.2%  ▂▂▃▂·▂▂▃▃·▅● beats 15/18    very easy · avg 89%    0.2%  maxed out · small run
Diagram-MMU diagram question answ…   83.4%  ▄·····▄▄·●·█ beats 3/6      medium · avg 56%       0.2%  blunt
HallusionBench (maker)               71.4%  ▂▁·▂▂▂▃▅▃█▅● beats 33/33    medium · avg 55%       0.2%  blunt · at its level · small run
RealWorldQA (maker)                  83.9%  ▁▃▁▇▃▇▇▅█▁●▅ beats 33/39    easy · avg 72%         0.1%  blunt · small run
CC-OCR (maker)                       82.0%  ▁·▂··▂▂▃▃▂▆● beats 25/26    easy · avg 72%         0.1%  blunt · small run
OCRBench (maker)                     93.1%  ▁·▃▁▆▅▂█▅█▁● beats 37/37    easy · avg 83%         0.1%  blunt · small run
AI2D test (maker)                    93.9%  ▁▁▅▇▃▇·▄·●·▃ beats 26/28    easy · avg 85%         0.1%  blunt · small run
PureDocBench document parsing (Av…   66.7%  ▂·▁·▂▂●█·▁▂▂ beats 7/22     medium · avg 62%       0.0%  blunt · at its level
ScreenSpot-Pro (maker)               65.6%  ▂▃▅▆█●▅▂···▂ beats 16/24    medium · avg 52%          —  not counted yet: no other model in the …
MMBench-EN dev v1.1 (maker)          93.7%  █▄···▂··▄▂▄● beats 14/14    easy · avg 79%            —  not counted yet: no other model in the …
OmniDocBench 1.5 overall (higher …   90.8%  ▂··▂▂▂··▂▂█● beats 11/13    medium · avg 54%          —  not counted yet: no other model in the …
PMC-VQA (maker)                      64.2%  ▃···▃··▃▃·▃● beats 7/7                                —  not counted yet: no other model in the …
web search · #63 of 84 · 3.6% of its score Beats [DeepSeek V4 Flash (Preview, Apr 2026)](deepseek-v4-flash-preview.md), [GPT-5.4 mini](gpt-5-4-mini.md) here, though they rank higher overall. Trails [GPT-4o mini (2024-07-18)](gpt-4o-mini-2024-07-18.md), [Mistral Small 3.1 24B (2503)](mistral-small-3-1.md), [Llama 4 Scout (109B-A17B)](llama-4-scout.md) here, though they rank lower overall. 
test                             result  vs the others in its run  difficulty        counts  why
Humanity's Last Exam with tools   48.3%  ▄····▄·●·▄·▄ beats 3/5    hard · avg 15%      1.3%  at its level · several runs
BrowseComp                        40.5%  ●·▄▄···▄··█▄ beats 0/6    hard · avg 18%      1.1%  at its level · several runs
BrowseComp-ZH (maker)             70.3%  ▁▁▁▁▁▃·▁▄▅▄● beats 22/23  hard · avg 21%      0.6%  at its level · small run
SealQA Seal-0 (maker)             46.9%  ▃····▃·····● beats 3/4    hard · avg 20%      0.3%  at its level · small run
WideSearch (maker)                74.0%  █·▂▆··●···▂▂ beats 8/10   medium · avg 36%    0.3%  at its level · small run
other tests (53) 
test                                result  vs the others in its run    difficulty            counts  why
GPQA Diamond                         85.9%  ▁▁▁▁▂▄▆▄▅●█▅ beats 149/211  hard · avg 44%          6.9%  sharp · several runs
Humanity's Last Exam                 28.5%  ▄··█●··▄·▄·▄ beats 3/6      very hard · avg 5.7%    5.0%  at its level · several runs
GPQA (subset unspecified)            87.1%  ▄█●▄·▄·····▄ beats 3/6      hard · avg 47%          3.6%  sharp · several runs
MMLU-Pro                             88.3%  ▄·····▄▄▄·▄● beats 6/6      medium · avg 53%        3.1%  several runs
LongBench v2                         68.9%  ▂·▁·▂▃▁▄▄▃█● beats 33/33    very hard · avg 34%     1.6%  at its level · several runs
Mystery Game Puzzles (Epoch)         18.0%  █··▂▃●▄█▂▄▂▁ beats 29/77    very hard · avg 9.4%    1.3%  sharp · noisy
Global-MMLU-Lite                     91.9%  ▃······▅··█● beats 6/6      medium · avg 61%        1.3%  typical
KINA (Knowledge Index of Noah's A…   43.0%  ▁▁▂▁▃▄█▆▄▅●▃ beats 35/39    very hard · avg 21%     1.3%  at its level
DSEffi-Bench Pass (1,000 data-sci…   57.6%  ▅··▃▅·█·▃●█▃ beats 10/15    hard · avg 23%          1.2%  at its level
IFBench (aggregation unspecified)    78.8%  ▃······█▃●·▃ beats 5/6      medium · avg 45%        1.2%  several runs
MMLU-ProX                            86.4%  ▄·····▄·▄▄█● beats 6/6      medium · avg 58%        1.1%  several runs
Chess Puzzles (Epoch)                12.0%  █▁▁▁▁●▃▃▃▂▁▁ beats 61/137   very hard · avg 5.8%    1.0%  sharp · noisy
EQ-Bench 3 (emotional intelligenc…  1262.9  ▁·▂▁▄▅▅▃●▄▁▁ beats 38/62    avg 1034                1.0%  typical
MultiChallenge                       63.9%  ▁··········● beats 5/6      hard · avg 28%          1.0%  at its level · several runs
AgentWorldBench (open-ended, over…   54.7%  ▄▄··▄▄·▄●▄██ beats 6/11     hard · avg 26%          0.9%  blunt · at its level
WildClawBench                        34.5%  ▃·▅▅●·▅█▅▃▃▃ beats 7/17     very hard · avg 11%     0.8%  at its level
NL2Repo (maker)                      32.2%  ▄·····█·●▄·▄ beats 3/5      very hard · avg 8.9%    0.7%  at its level
IFBench prompt-level (loose)         78.2%  ▄▄·▄··●▄···█ beats 3/6      medium · avg 47%        0.7%  typical
C-Eval (maker)                       93.0%  ▁···▁▁▂▁▂▂▃● beats 32/33    medium · avg 73%        0.6%  small run
INCLUDE (maker)                      85.6%  ▂····▂▂▄▄▄█● beats 35/36    medium · avg 56%        0.5%  small run
QwenClawBench (Qwen internal) (ma…   51.8%  ●·█··█·····█ beats 0/3      hard · avg 24%          0.5%  at its level
MMLU-Redux (maker)                   94.9%  ▁▁▃▁▂▃▄▅▇▅█● beats 62/63    easy · avg 77%          0.5%  maxed out · small run
MMMLU (maker)                        88.5%  ▂▁▁▂▃▂▄▄▅█●▃ beats 42/46    medium · avg 61%        0.5%  small run
VitaBench (maker)                    49.7%  ▄··▂▄▆▆█▄▄▄● beats 21/21    very hard · avg 13%     0.5%  at its level · small run
Claw-Eval Pass^3 (maker)             48.1%  ●▄▄·▄·▄▄·▄·▄ beats 1/8      hard · avg 25%          0.4%  at its level
SkillsBench 78-task subset (OpenC…   30.0%  █●·········▄ beats 2/3      very hard · avg 11%     0.4%  at its level
SuperGPQA (maker)                    70.4%  ▂▁▂▂▅▆▃▇▇▇█● beats 46/46    medium · avg 37%        0.4%  at its level · small run
Global PIQA (maker)                  89.8%  ▃▃··▃··▃▃▅█● beats 11/11    medium · avg 72%        0.4%  small run
MAXIFE (maker)                       88.2%  ▂·▂·▂···▆·▄● beats 11/11    medium · avg 63%        0.3%  small run
CritPt                                2.4%  ▄····●█···█▄ beats 1/6      very hard · avg 1.1%    0.3%  too hard for it · noisy
VideoMMMU (maker)                    84.7%  ▂···▂···▂▂█● beats 10/10    medium · avg 62%        0.3%  small run
LVBench (maker)                      75.5%  ▇·█▅▅▄●··▃▁▁ beats 24/28    hard · avg 51%          0.3%  at its level · small run
Claw-Eval average score (maker)      70.7%  ▄·▄··█▄▄▄▄●▄ beats 8/9      medium · avg 48%        0.3%  blunt · at its level
DeepPlanning (maker)                 34.3%  ▂·····▂··▄█● beats 9/9      very hard · avg 11%     0.3%  at its level · small run
ERQA (maker)                         67.5%  ▃··▃··█·▃█▅● beats 11/12    medium · avg 36%        0.3%  at its level · small run
MMVU (maker)                         75.4%  ▃···▃···▃▃▅● beats 8/8      medium · avg 50%        0.3%  small run
RefSpatial-Bench (maker)             73.6%  ▃·▃····▃▃▅▅● beats 10/10    medium · avg 45%        0.3%  at its level · small run
MMLongBench-Doc (maker)              61.5%  ▃···▃···▃▅█● beats 8/9      medium · avg 37%        0.3%  at its level · small run
MLVU (maker)                         86.7%  ▂···▂···▂▄▄● beats 9/10     easy · avg 69%          0.3%  small run
NOVA-63 (maker)                      59.1%  ▃▃·▃···▃█▃▃● beats 11/11    medium · avg 35%        0.2%  blunt · at its level · small run
WMT25 Terminology Translation (te…   87.0%  ▂·▂··▂·▄●·█▆ beats 5/12     medium · avg 63%        0.2%  noisy
EmbSpatial-Bench (maker)             84.5%  ▁·····▁·▁·▃● beats 8/10     easy · avg 69%          0.2%  small run
Video-MME (without subtitles) (ma…   83.7%  ▁▃▁▃▁·█▄▄▅·● beats 26/27    medium · avg 67%        0.2%  blunt · small run
WMT24++                              78.5%  █·▃·●▅▃···▃▃ beats 5/10     easy · avg 70%          0.2%  blunt · several runs
MVBench (maker)                      77.6%  ▁··▃▄▁▃▄▄▄█● beats 26/26    medium · avg 66%        0.1%  blunt · at its level · small run
WMT26 General MT (GEMBA-ESA, LLM …   81.6%  ▄·▄▄▄·▄██▄●▄ beats 10/12    medium · avg 65%        0.1%  blunt
AndroidWorld (maker)                 66.8%  ▁·▁▁▂▃█▅●▂·▂ beats 20/24                               —  not counted yet: no other model in the …
Video-MME (with subtitles) (maker)   87.5%  ▁▁▁▁▁·▁▃▁▁▁● beats 15/16                               —  not counted yet: no other model in the …
MedXPertQA MM (maker)                70.0%  ▂···▂·▃··▆▂● beats 13/13                               —  not counted yet: no other model in the …
ZEROBench subquestions (maker)       41.0%  ▄·▄···▄·▄██● beats 8/8                                 —  not counted yet: no other model in the …
SLAKE (maker)                        79.9%  ▄·····▄▄·█●▄ beats 5/7                                 —  not counted yet: no other model in the …
LingoQA (maker)                      81.6%  ▄······▄·█·● beats 4/5                                 —  not counted yet: no other model in the …
SecCodeBench (maker)                 68.3%  █··········● beats 1/1                                 —  not counted yet: no other model in the …
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Qwen/Qwen3.5-397B-A17B model card (subm…       62   25% ▇▇▇
NVIDIA                                         18   19% ▇▇
Epoch AI                                        5   13% ▇▇
Qwen/Qwen3.6-27B model card (submitted …       12   12% ▇
A.X K2 Technical Report                         7    8% ▇
LMArena leaderboard dataset                     3    6% ▇
τ²/τ³-bench leaderboard: Sierra's own e…        4    3% ▇
MathArena ArXivMath / BrokenArXiv / ArX…        2    2% ▇
Qwen (submitted / maker-reported)               1    2% ▇
ContractBench: Can LLM Agents Preserve …        1    1% ▇
Knowledge Index of Noah's Ark                   1    1% ▇
DSEffi-Bench: Demystifying Large Langua…        1    1% ▇
LGAI-EXAONE/K-EXAONE-2.0-750B-A37B mode…        1    1% ▇
EQ-Bench 3 canonical leaderboard results        1    1% ▇
Qwen/Qwen-AgentWorld-35B-A3B model card…        1    1% ▇
WildClawBench: A Benchmark for Real-Wor…        1    1% ▇
BEAR-Bench: A Bilingual Enterprise and …        1    1% ▇
EEGAgentBench: Benchmarking LLM Agents …        1    1% ▇
Cohere                                          3    0% ▇
Diagram-MMU: A Multi-Modal Benchmark fo…        1    0% ▇
How Far Is Document Parsing from Solved…        1    0% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 69.4
Licences: Qwen/Qwen3.5-397B-A17B model card ([apache-2.0](../../licenses/Qwen--Qwen3.5-397B-A17B.txt)) · NVIDIA ([CC BY 4.0](../../licenses/report-2606.15007v1.txt)) · Epoch AI ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Qwen/Qwen3.6-27B model card ([apache-2.0](../../licenses/Qwen--Qwen3.6-27B.txt)) · A.X K2 Technical Report ([CC BY 4.0](http://creativecommons.org/licenses/by/4.0/)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · τ²/τ³-bench leaderboard: Sierra's own evaluation runs ([MIT](https://opensource.org/license/mit)) · MathArena ArXivMath / BrokenArXiv / ArXivLean model-output datasets ([CC BY-SA 4.0](../../licenses/matharena-sharealike-NOTICE.txt)) · ContractBench: Can LLM Agents Preserve Observation Contracts? ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Knowledge Index of Noah's Ark ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation ([CC BY 4.0](../../licenses/paper-2608.30248v1-CC-BY-4.0.txt)) · LGAI-EXAONE/K-EXAONE-2.0-750B-A37B model card ([Apache-2.0](https://huggingface.co/LGAI-EXAONE/K-EXAONE-2.0-750B-A37B/blob/fec0c8dd25fdd20ffd09869d5af5277966d5523d/LICENSE)) · EQ-Bench 3 canonical leaderboard results ([MIT](../../licenses/eqbench3-MIT.txt)) · Qwen/Qwen-AgentWorld-35B-A3B model card ([Apache-2.0](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B/blob/60d2b0434a53d2e62a7c00a489586815d94ebffb/README.md)) · WildClawBench: A Benchmark for Real-World ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Cohere ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/))
Not counted (12) 
source                  test                            score  why
Qwen/Qwen3.5-397B-A17…  AndroidWorld                    66.8%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  CharXiv (reasoning questions)   80.8%  only model in this source's run
Qwen/Qwen3.5-397B-A17…  LingoQA                         81.6%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  MedXPertQA MM                   70.0%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  MMBench-EN dev v1.1             93.7%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  OmniDocBench 1.5 overall (hig…  90.8%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  PMC-VQA                         64.2%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  ScreenSpot-Pro                  65.6%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  SecCodeBench                    68.3%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  SLAKE                           79.9%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  Video-MME (with subtitles)      87.5%  too few models have it yet
Qwen/Qwen3.5-397B-A17…  ZEROBench subquestions          41.0%  too few models have it yet
## How to run it ### Through an API api (Deep Infra) · $1.09/M blended · $0.45 in / $3 out per 1M tokens · [deepinfra.com](https://deepinfra.com/models) [price source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/providers/deepinfra/models/Qwen/Qwen3.5-397B-A17B.toml): Standard base token rate from models.dev. Provider-specific tiers, long-context surcharges, caching and promotions may differ. · [context source](https://github.com/anomalyco/models.dev/blob/a668908cfe7073faa0363699486fc41a3ff52412/models/alibaba/qwen3.5-397b-a17b.toml) ### llama.cpp (recommended) 1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) ```sh # macOS / Linux curl -LsSf https://llama.app/install.sh | sh # Windows irm https://llama.app/install.ps1 | iex ``` 2. Run Qwen3.5 397B-A17B (ud-q4_k_xl, needs ~238.7 GB, from [huggingface.co/unsloth](https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/tree/main)) ```sh llama cli -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL ``` 3. Or start a local server and open the address it prints (http://localhost:9931) ```sh llama serve -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL ``` Other versions: ud-iq2_xxs (~119.6 GB) `llama cli -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-IQ2_XXS` · ud-iq1_m (~60.05 GB) `llama cli -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-IQ1_M` ### LM Studio 1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download) 2. Run Qwen3.5 397B-A17B (ud-q4_k_xl, needs ~238.7 GB, from [huggingface.co/unsloth](https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/tree/main)) ```sh lms get https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF@UD-Q4_K_XL -y ``` 3. Then run lms ls to see its name, and lms chat followed by that name Other versions: ud-iq2_xxs (~119.6 GB) `lms get https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF@UD-IQ2_XXS -y` · ud-iq1_m (~60.05 GB) `lms get https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF@UD-IQ1_M -y` · 4bit (~223.9 GB, Mac only) `lms get https://huggingface.co/lmstudio-community/Qwen3.5-397B-A17B-MLX-4bit -y` · 8bit (~422 GB, Mac only) `lms get https://huggingface.co/lmstudio-community/Qwen3.5-397B-A17B-MLX-8bit -y` ### MLX (Mac) 1. Install MLX (Mac). Apple silicon Macs only, text only. ```sh # with uv uv tool install mlx-lm # or pip install mlx-lm ``` 2. Run Qwen3.5 397B-A17B (4bit, needs ~223.9 GB, Mac only, from [huggingface.co/lmstudio-community](https://huggingface.co/lmstudio-community/Qwen3.5-397B-A17B-MLX-4bit/tree/76a66de5151f25687d96c50254198f1959e25ddd)) ```sh mlx_lm.chat --model lmstudio-community/Qwen3.5-397B-A17B-MLX-4bit ``` 3. Or start an OpenAI-compatible server at http://localhost:8080/v1 ```sh mlx_lm.server --model lmstudio-community/Qwen3.5-397B-A17B-MLX-4bit ``` Other versions: 8bit (~422 GB, Mac only) `mlx_lm.chat --model lmstudio-community/Qwen3.5-397B-A17B-MLX-8bit` Model repository and documentation: [huggingface.co/Qwen](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) ## Quantization evidence How much of the full-precision result a quantized version kept, measured within one study. Rows from the same study share a table and aren't independent. They don't describe every download of that precision. 
precision                       test                                full → quantized    kept
FP8 W8A8 dynamic (E4M3; weigh…  AIME 2025 0-shot (lighteval via l…     0.929 → 0.946  101.8%
FP8 W8A8 dynamic (E4M3; weigh…  IFEval prompt-level strict 0-shot…     0.906 → 0.907  100.1%
FP8 W8A8 dynamic (E4M3; weigh…  MMLU-Pro 0-shot (lm-evaluation-ha…     0.883 → 0.883  100.1%
FP8 W8A8 dynamic (E4M3; weigh…  GSM8K-Platinum 0-shot (lm-evaluat…     0.955 → 0.954   99.8%
FP8 W8A8 dynamic (E4M3; weigh…  LiveCodeBench v6 code generation …     0.838 → 0.836   99.8%
FP8 W8A8 dynamic (E4M3; weigh…  MATH-500 0-shot (lighteval via li…     0.849 → 0.845   99.5%
FP8 W8A8 dynamic (E4M3; weigh…  GPQA Diamond 0-shot (lighteval vi…     0.897 → 0.887   98.9%
Each kept value links to its source. [interactive version](https://whatmodel.sh/model/qwen3-5-397b-a17b) · [JSON](https://whatmodel.sh/api?format=json&all=1)