[whatmodel.sh](../../README.md) / Snowflake Arctic Instruct # Snowflake Arctic Instruct Snowflake · released 2024-04 · 480B params (17B active) · apache-2.0 **score 43.1** ±1.6 · #515 of 732 · from 5 benchmark results across 4 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Snowflake Arctic Instruct among every model on that task or test, worst to best. 
task            vs the others         rank  counts  tests
chat & writing  ▁▁▁▃●▄▆█▆▇▇▃   #326 of 402     96%      3
chat & writing · #326 of 402 · 96% of its score Beats [Gemma 4 E4B](gemma-4-e4b.md), [OpenThinker3 7B](openthinker3-7b.md), [Code Llama 70B Instruct](codellama-70b-instruct.md) here, though they rank higher overall. Trails [LFM2 350M](lfm2-350m.md), [Gemma 3 1B](gemma-3-1b.md), [Qwen3 0.6B](qwen3-0-6b.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty        counts  why
Arena-Hard (v0.1, GPT-4-1106 judg…   17.6%  ▂▁▁●█▅▄▃▃▅▃▁ beats 11/60   medium · avg 47%     44%  sharp
LMArena (text)                      1180.1  ▁▁▁▂●▃▃▆▆▇█▃ beats 56/385  avg 1260             27%  blunt
Polyrating rating (bias-aware re-…  1126.0  ▂▁▂▁▃▇█●▇▄▂▂ beats 61/95   avg 1187             25%  typical
other tests (2) 
test                                result  vs the others in its run    difficulty           counts  why
Vectara hallucination leaderboard…   97.0%  ▁▁▂▃▄▄█●▃▂▂▁ beats 103/140  very easy · avg 94%    2.4%  blunt · noisy
Vectara hallucination leaderboard…   95.7%  ▁▂▁▂▅█▂▃▄▃●▁ beats 95/98    very easy · avg 90%    0.0%  blunt
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Arena-Hard-Auto v0.1 leaderboard                1   44% ▇▇▇▇▇
LMArena leaderboard dataset                     1   27% ▇▇▇
Polyrating: A Cost-Effective and Bias-A…        1   25% ▇▇▇
Vectara hallucination leaderboard               2    2% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 43.1
Licences: Arena-Hard-Auto v0.1 leaderboard ([Apache-2.0](../../licenses/arena-hard-auto-v0-1-Apache-2.0.txt)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) ## How to run it ### Other ways to run it 1. Run it as published (BF16 WEIGHTS, DEEPSPEED FP8 AT LOAD, needs ~957.17 GB, NVIDIA only, from [huggingface.co/Snowflake](https://huggingface.co/Snowflake/snowflake-arctic-instruct/blob/be318cae5aba5291208f27d30991a5150500887d/README.md)) ```sh pip install 'transformers>=4.39.0' 'deepspeed>=0.14.2' accelerate && python -c "import torch; from transformers import AutoModelForCausalLM, AutoTokenizer; from deepspeed.linear.config import QuantizationConfig; r='Snowflake/snowflake-arctic-instruct'; t=AutoTokenizer.from_pretrained(r, trust_remote_code=True); m=AutoModelForCausalLM.from_pretrained(r, trust_remote_code=True, low_cpu_mem_usage=True, device_map='auto', ds_quantization_config=QuantizationConfig(q_bits=8), max_memory={i: '150GiB' for i in range(8)}, torch_dtype=torch.bfloat16); x=t.apply_chat_template([{'role': 'user', 'content': '5x + 35 = 7x - 60 + 10. Solve for x'}], add_generation_prompt=True, return_tensors='pt').to('cuda'); print(t.decode(m.generate(input_ids=x, max_new_tokens=256)[0]))" ``` Model repository and documentation: [huggingface.co/Snowflake](https://huggingface.co/Snowflake/snowflake-arctic-instruct) [interactive version](https://whatmodel.sh/model/snowflake-arctic-instruct) · [JSON](https://whatmodel.sh/api?format=json&all=1)