[whatmodel.sh](../../README.md) / T0++ (T0pp) 11B # T0++ (T0pp) 11B BigScience · released 2021-10 · 11B params · apache-2.0 **score 22.5** ±2.5 · comparison only: no current run option verified · from 9 benchmark results across 5 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks T0++ (T0pp) 11B among every model on that task or test, worst to best.
other tests (9) 
test                                result  vs the others in its run   difficulty        counts  why
MMLU                                 30.9%  ▁▁▁▁●▂▂▅██▆▄ beats 14/214  medium · avg 74%     33%  small run
ZeroSCROLLS QuALITY (zero-shot ac…   21.4%  ●····▃·▃▃█▅▅ beats 0/10    easy · avg 81%       13%  typical
Super-NaturalInstructions English…   40.3%  ▄·▄······▄●█ beats 3/5     easy · avg 77%       13%  at its level · small run
ZeroSCROLLS Qasper (zero-shot F1)    25.0%  ▂▂··●·▂··▂█▄ beats 2/10    medium · avg 56%     12%  blunt
BIG-Bench Hard                       13.9%  ●▁▁▁▂▄▅▆█▃▃▁ beats 1/155   medium · avg 58%     11%  sharp · too hard for it · small run
ZeroSCROLLS NarrativeQA (zero-sho…   18.7%  ▃··▃·●▃·█▃·▃ beats 2/10    hard · avg 29%      5.9%  blunt
LMentry accuracy (zero-shot)         49.9%  ▂▆·▄▄●▂·▂··▂ beats 11/14   medium · avg 65%    4.1%  blunt · at its level
CREPE event likelihood change (3-…   33.7%  ●█····█····█ beats 0/3     medium · avg 52%    3.8%  blunt
ZeroSCROLLS MuSiQue (zero-shot F1)   35.3%  ▄·█··█●·▄▄▄▄ beats 5/10    medium · avg 43%    2.1%  blunt · at its level
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
ZeroSCROLLS: A Zero-Shot Benchmark for …        4   33% ▇▇▇▇
Model-Generated Pretraining Signals Imp…        1   33% ▇▇▇▇
Unnatural Instructions: Tuning Language…        2   24% ▇▇▇
LMentry: A Language Model Benchmark of …        1    4% ▇
Causal Reasoning of Entities and Events…        1    4% ▇
prior: pull toward the middle                        3% ▇
weighted average                                   = 22.5
Licences: ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-536-CC-BY-4-0.txt)) · Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers ([CC BY 4.0](../../licenses/acl-2023-acl-long-724-CC-BY-4-0.txt)) · Unnatural Instructions: Tuning Language Models with ([CC BY 4.0](../../licenses/acl-2023-acl-long-806-CC-BY-4-0.txt)) · LMentry: A Language Model Benchmark of Elementary Language Tasks ([CC BY 4.0](../../licenses/acl-2023-findings-acl-666-CC-BY-4-0.txt)) · Causal Reasoning of Entities and Events in Procedural Texts ([CC BY 4.0](../../licenses/acl-2023-findings-eacl-31-CC-BY-4-0.txt)) ## How to run it No verified API or local download yet. Model repository and documentation: [huggingface.co/bigscience](https://huggingface.co/bigscience/T0pp) [interactive version](https://whatmodel.sh/model/t0pp) · [JSON](https://whatmodel.sh/api?format=json&all=1)