[whatmodel.sh](../../README.md) / Guanaco 33B # Guanaco 33B University of Washington · released 2023-05 · 33B params · llama-1-noncommercial-research **score 35.3** ±1.4 · #624 of 732 · from 6 benchmark results across 5 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at _{● marks Guanaco 33B among every model on that task or test, worst to best.}

task            vs the others         rank  counts  tests
chat & writing  ▁▁▁●▆▄▆█▆▇▇▃   #378 of 402     94%      5

chat & writing · #378 of 402 · 94% of its score

Beats [Phi-2 (2.7B)](phi-2.md), [Qwen2.5 1.5B Instruct](qwen2-5-1-5b.md), [Nous-Hermes-13b](nous-hermes-13b.md) here, though they rank higher overall. Trails [Gemma 3 270M IT](gemma-3-270m.md), [SmolLM2 360M Instruct](smollm2-360m.md), [Granite 4.0 350M](granite-4-0-350m.md) here, though they rank lower overall.

test                                                      result  vs the others in its run   difficulty                counts  why
AlpacaEval 2 length-controlled                              5.7%  ▁▂●▃▆▆█▆▄▄▁▁ beats 11/122  medium · avg 35%             32%  sharp
LMArena (text)                                            1127.6  ▁▁▁●▃▃▃▆▆▇█▃ beats 25/385  avg 1260                     22%  blunt
Polyrating rating (bias-aware re-…                        1035.0  ▂▁▂▁▃●█▆▇▄▂▂ beats 20/95   avg 1187                     20%  typical
MT-Bench (GPT-4 single-answer gra…  6.5300 MT-Bench score (1-10)  ▆▂▂▆▆▄█●▂▄▄▆ beats 17/28   avg 8.539 MT-Bench scor…     15%  sharp · noisy
AlpacaEval 1.0 win rate (vs text-…                         66.0%  ▂▂·▆·●▃▆█▆▄▂ beats 12/45   very easy · avg 94%         5.2%  sharp · at its level · noisy

other tests (1)

test                                result  vs the others in its run  difficulty        counts  why
Spider dev execution accuracy, ze…   24.4%  █·█▄··●····▄ beats 5/6    medium · avg 54%    4.6%  sharp · noisy

## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md)

source                                    results  share of the score
AlpacaEval 1.0 and 2.0 leaderboards             2   37% ▇▇▇▇
LMArena leaderboard dataset                     1   22% ▇▇▇
Polyrating: A Cost-Effective and Bias-A…        1   20% ▇▇
LMSYS Chatbot Arena leaderboard Space: …        1   15% ▇▇
Battle of the Large Language Models: Do…        1    5% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 35.3

_{Licences: AlpacaEval 1.0 and 2.0 leaderboards ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · LMSYS Chatbot Arena leaderboard Space: MT-Bench column of the published leaderboard table ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT - A Text-to-SQL Parsing Comparison ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-750-CC-BY-4-0.txt))} ## How to run it ### Other ways to run it 1. Run it as published (BF16, needs ~65.06 GB, from [huggingface.co/timdettmers](https://huggingface.co/timdettmers/guanaco-33b-merged/tree/b2e78a916582935b6616d184b22ea5e9e1eb4c34)) ```sh uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text-generation', model='timdettmers/guanaco-33b-merged', torch_dtype='auto', device_map='auto')('Once upon a time', max_new_tokens=40)[0]['generated_text'])" ``` Model repository and documentation: [huggingface.co/timdettmers](https://huggingface.co/timdettmers/guanaco-33b-merged) [interactive version](https://whatmodel.sh/model/guanaco-33b) · [JSON](https://whatmodel.sh/api?format=json&all=1)