[whatmodel.sh](../../README.md) / Jamba 1.5 Large # Jamba 1.5 Large AI21 Labs · released 2024-08 · 398B params (94B active) · jamba-open-model-license **score 51.7** ±1.1 · #338 of 732 · from 12 benchmark results across 3 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at _{● marks Jamba 1.5 Large among every model on that task or test, worst to best.}

task            vs the others          rank  counts  tests
agent tasks     ▁▁▃▆█▄●▅▄▄▂▁   #117 of 313*    2.3%      1
coding          █··········●       #1 of 2*    7.5%      1
chat & writing  ▁▁▁▃▆▄▆●▆▇▇▃    #242 of 402     20%      3
math            ▁▁▃▅●▆▄█▄▂▁▁   #274 of 431*     11%      1

agent tasks · #117 of 313 · 2.3% of its score

Beats [o3-pro](o3-pro.md), [Grok 4.3](grok-4-3.md), [Qwen3.6 Plus](qwen3-6-plus.md) here, though they rank higher overall. Trails [SmolLM3 3B](smollm3-3b.md) here, though they rank lower overall.

test                               result  vs the others in its run  difficulty      counts  why
BFCL (version not stated) (maker)   85.5%  ▄▆▄▂·█●·▄··▂ beats 14/17  easy · avg 81%    2.3%  blunt · small run

coding · #1 of 2 · 7.5% of its score

test               result  vs the others in its run    difficulty      counts  why
HumanEval (maker)   71.3%  ▁▁▁▁▂▂▆▆●▅▂▁ beats 138/205  easy · avg 69%    7.5%  sharp · at its level · small run

chat & writing · #242 of 402 · 20% of its score

Beats [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md), [Qwen2.5 Coder 32B Instruct](qwen2-5-coder-32b.md), [Ministral 3 14B Reasoning 2512](ministral-3-14b-reasoning.md) here, though they rank higher overall. Trails [Flan-T5 XXL](flan-t5-xxl.md), [LFM2.5 1.2B Instruct](lfm2-5-1-2b-instruct.md), [Gemma 3 4B](gemma-3-4b.md) here, though they rank lower overall.

test                                result  vs the others in its run    difficulty        counts  why
LMArena (text)                      1289.5  ▁▁▁▂▃▃▃●▆▇█▃ beats 120/385  avg 1260             13%  blunt
Arena-Hard (version/judge unspeci…   65.4%  ▃▃▂▅▃▆●█▅▅▂▃ beats 15/30    medium · avg 55%    4.1%  at its level · small run
IFEval (aggregation unspecified) …   81.5%  ▁▂▄▂▂▄●██▅▂▁ beats 56/145   easy · avg 79%      2.8%  small run

math · #274 of 431 · 11% of its score

Beats [GPT-4 Turbo (2024-04-09)](gpt-4-turbo-2024-04-09.md), [Molmo 2 8B](molmo-2-8b.md), [Llama 3.3 70B Instruct](llama-3-3-70b.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [CodeGemma 7B (v1.0)](codegemma-7b.md) here, though they rank lower overall.

test           result  vs the others in its run    difficulty      counts  why
GSM8K (maker)   87.0%  ▂▂▂▂▃▄▅▇█●▆▄ beats 161/206  easy · avg 82%     11%  sharp · small run

other tests (5)

test                               result  vs the others in its run    difficulty           counts  why
MMLU-Pro                            49.5%  ▃▂▅█●·█▃▅··▂ beats 12/23    medium · avg 54%        27%  at its level · several runs
GPQA (subset unspecified) (maker)   36.9%  ▂▁▂▂▄●▆▃▅█▃▃ beats 32/106   very hard · avg 35%     11%  sharp · small run
BIG-Bench Hard (maker)              65.5%  ▁▁▁▁▂▄▅▆●▃▃▁ beats 98/155   medium · avg 61%        10%  sharp · at its level · small run
MMLU (maker)                        80.0%  ▁▁▁▁▁▂▂▅██●▄ beats 166/214  easy · avg 75%         8.1%  at its level · small run
ARC Challenge (maker)               93.0%  ▄▁▁▂▁▄█▇▃▂▃● beats 139/140  easy · avg 76%         1.9%  sharp · at its level · noisy · small run

## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md)

source                                    results  share of the score
Jamba-1.5: Hybrid Transformer-Mamba Mod…       10   66% ▇▇▇▇▇▇▇▇
MMLU-Pro author-released evaluation runs        1   21% ▇▇▇
LMArena leaderboard dataset                     1   13% ▇▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 51.7

_{Licences: Jamba-1.5: Hybrid Transformer-Mamba Models at Scale ([CC BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/)) · MMLU-Pro author-released evaluation runs ([Apache-2.0](../../licenses/mmlu-pro-author-runs-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/))} ## How to run it ### Other ways to run it 1. Run it as published (BF16 WEIGHTS, EXPERTS_INT8 AT LOAD, needs ~797.11 GB, NVIDIA only, from [huggingface.co/ai21labs](https://huggingface.co/ai21labs/AI21-Jamba-Large-1.5/blob/57d0bbccd09d9f56fb8ad2c3f2353ef61717f906/README.md)) ```sh pip install 'vllm>=0.5.5' && vllm serve ai21labs/AI21-Jamba-Large-1.5 --tensor-parallel-size 8 --max-model-len 225280 --quantization experts_int8 ``` Model repository and documentation: [huggingface.co/ai21labs](https://huggingface.co/ai21labs/AI21-Jamba-Large-1.5) [interactive version](https://whatmodel.sh/model/jamba-1-5-large) · [JSON](https://whatmodel.sh/api?format=json&all=1)