[whatmodel.sh](../../README.md) / Flan-T5 Small
# Flan-T5 Small
Google · released 2022-10 · 0.077B params · apache-2.0
**score 16.2** ±2.6 · #709 of 732 · from 11 benchmark results across 5 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Flan-T5 Small among every model on that task or test, worst to best.
task vs the others rank counts tests
math ●▁▃▅█▆▄█▄▂▁▁ #427 of 431* 3.9% 1
math · #427 of 431 · 3.9% of its score
Beats [Flan-T5 Base](flan-t5-base.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [BLOOM 560M](bloom-560m.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MGSM (maker) 0.2% ●·▂·▄▂▂▂▂█▄▆ beats 1/18 medium · avg 48% 3.9% sharp · too hard for it
other tests (10)
test result vs the others in its run difficulty counts why
SVAMP 1.7% █·●▂▄▂▂····▂ beats 5/12 easy · avg 70% 20% sharp · too hard for it
MMLU (maker) 28.7% ▁···●··▃▁▅▅█ beats 1/18 easy · avg 74% 20% too hard for it
ZeroSCROLLS Qasper (zero-shot F1) 18.5% ▂●··▂·▂··▂█▄ beats 1/10 medium · avg 56% 11% blunt
ZeroSCROLLS QuALITY (zero-shot ac… 34.6% ▃····●·▃▃█▅▅ beats 1/10 easy · avg 81% 10% typical
TyDiQA-GoldP (1-shot, EM, 8-langu… 1.1% ●··▂·▂▄▄·███ beats 0/18 medium · avg 61% 7.5% too hard for it
POSQA general (yes/no) size quest… 51.0% █···●·····██ beats 1/3 medium · avg 68% 7.3% too hard for it
BIG-Bench Hard (maker) 29.1% ●▂▂▂·▅·▃▂█▃▃ beats 0/18 medium · avg 60% 5.7% sharp · too hard for it
ZeroSCROLLS NarrativeQA (zero-sho… 11.6% ●··▃·█▃·█▃·▃ beats 0/10 hard · avg 29% 5.5% blunt
ConvRe Re2Text, normal relation, … 51.8% ▂●▄··▂▂█▄··▂ beats 1/12 easy · avg 90% 4.8% noisy
ZeroSCROLLS MuSiQue (zero-shot F1) 21.0% ●·█··██·▄▄▄▄ beats 0/10 medium · avg 43% 2.1% blunt
source results share of the score
Scaling Instruction-Finetuned Language … 4 37% ▇▇▇▇
ZeroSCROLLS: A Zero-Shot Benchmark for … 4 28% ▇▇▇
Large Language Models Are Reasoning Tea… 1 20% ▇▇
POSQA: Probe the World Models of LLMs w… 1 7% ▇
An Investigation of LLMs' Inefficacy in… 1 5% ▇
prior: pull toward the middle 3% ▇
weighted average = 16.2
Licences: Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) · ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-536-CC-BY-4-0.txt)) · Large Language Models Are Reasoning Teachers ([CC BY 4.0](../../licenses/acl-2023-acl-long-830-CC-BY-4-0.txt)) · POSQA: Probe the World Models of LLMs with Size Comparisons ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-504-CC-BY-4-0.txt)) · An Investigation of LLMs' Inefficacy in Understanding Converse Relations ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-429-CC-BY-4-0.txt))
## How to run it
### Other ways to run it
1. Run it as published (fp32, needs ~0.31 GB, from [huggingface.co/google](https://huggingface.co/google/flan-t5-small/tree/0fc9ddf78a1e988dac52e2dac162b0ede4fd74ab))
```sh
uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text2text-generation', model='google/flan-t5-small')('Answer the question: what is the capital of France?', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/flan-t5-small)
[interactive version](https://whatmodel.sh/model/flan-t5-small) · [JSON](https://whatmodel.sh/api?format=json&all=1)