[whatmodel.sh](../../README.md) / Claude Instant 1.1 # Claude Instant 1.1 Anthropic · released 2023-05 · proprietary **score 44.5** ±1.6 · comparison only: no current run option verified · from 6 benchmark results across 2 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at _{● marks Claude Instant 1.1 among every model on that task or test, worst to best.}

task    vs the others          rank  counts  tests
coding  ▁▁▁▃●▆█▇▅▄▃▁   #432 of 539*     20%      1
math    ▁▁▃▆●█▅█▃▁▁▁   #371 of 530*     28%      1

coding · #432 of 539 · 20% of its score

Beats [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md), [Qwen3-VL 2B Thinking](qwen3-vl-2b-thinking.md), [DeepSeek R1 Distill Qwen 7B](deepseek-r1-distill-qwen-7b.md) here, though they rank higher overall. Trails [Qwen2.5 1.5B Instruct](qwen2-5-1-5b.md), [Code Tulu 2 13B](code-tulu-2-13b.md), [Phi-3-mini-128k-instruct (Apr 2024)](phi-3-mini-128k-instruct.md) here, though they rank lower overall.

test               result  vs the others in its run   difficulty      counts  why
HumanEval (maker)   52.8%  ▁▁▁▁▂▂▆●█▅▂▁ beats 99/205  easy · avg 71%     20%  sharp · at its level

math · #371 of 530 · 28% of its score

Beats [Molmo 2 8B](molmo-2-8b.md), [Llama 3.1 Nemotron 70B Instruct](llama-3-1-nemotron-70b.md), [GPT-4 (0613)](gpt-4-0613.md) here, though they rank higher overall. Trails [DeepSeek R1 Distill Qwen 1.5B](deepseek-r1-distill-qwen-1-5b.md), [Gemma 7B (base)](gemma-7b-base.md), [Mistral 7B v0.1 (base)](mistral-7b-v0-1.md) here, though they rank lower overall.

test           result  vs the others in its run    difficulty           counts  why
GSM8K (maker)   80.9%  ▂▂▂▂▃▄▅▇█●▆▄ beats 141/206  very easy · avg 90%     28%  sharp

other tests (4)

test                                result  vs the others in its run    difficulty           counts  why
ARC Challenge (maker)                85.7%  ▄▁▁▂▁▄█▇▃▂●▂ beats 133/140  very easy · avg 90%     21%  sharp · small run
MMLU (maker)                         73.4%  ▁▁▁▁▁▂▂▅█●▆▄ beats 150/214  easy · avg 81%          18%  at its level · small run
TriviaQA (maker)                     78.9%  ▁▁▂▂▄▄▃▆▅█●▂ beats 76/80    easy · avg 85%         9.1%  small run
ConvRe Re2Text, normal relation, …   65.7%  ▂▂▄··●▂█▄··▂ beats 4/12     easy · avg 90%         2.6%  at its level · noisy

## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md)

source                                    results  share of the score
Anthropic system cards and model announ…        7   96% ▇▇▇▇▇▇▇▇▇▇▇▇
An Investigation of LLMs' Inefficacy in…        1    3% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 44.5

_{Licences: An Investigation of LLMs' Inefficacy in Understanding Converse Relations ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-429-CC-BY-4-0.txt))} ## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/claude-instant-1-1) · [JSON](https://whatmodel.sh/api?format=json&all=1)