[whatmodel.sh](../../README.md) / Flan-T5 XL
# Flan-T5 XL
Google · released 2022-10 · 2.85B params · apache-2.0
**score 37.2** ±0.9 · #606 of 732 · from 22 benchmark results across 14 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Flan-T5 XL among every model on that task or test, worst to best.
task vs the others rank counts tests
math ▁▁●▅█▆▄█▄▂▁▁ #407 of 431 24% 2
math · #407 of 431 · 24% of its score
Beats [Command R (03-2024)](command-r-03-2024.md), [Mistral 7B Instruct v0.3](mistral-7b-instruct-v0-3.md), [Gemma 2 2B IT](gemma-2-2b.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 13.5% ▂▂▂▂●▄▅▇██▆▄ beats 40/206 medium · avg 49% 17% sharp · several runs
MGSM (maker) 1.9% ▆·●·▄▂▂▂▂█▄▆ beats 3/18 medium · avg 48% 7.0% sharp
other tests (16)
test result vs the others in its run difficulty counts why
MMLU (maker) 52.4% ▁···▁··▃▁●▅█ beats 6/18 easy · avg 74% 27% at its level · several runs
BIG-Bench Hard (maker) 41.0% ▂▂▂▂·▅·●▂█▃▃ beats 7/18 medium · avg 68% 24% sharp · several runs
TyDiQA-GoldP (1-shot, EM, 8-langu… 16.6% ▂··▂·▂●▄·███ beats 3/18 medium · avg 61% 4.3% at its level
POSQA general (yes/no) size quest… 58.0% █···█·····█● beats 3/3 medium · avg 68% 3.3% typical
ZeroSCROLLS QuALITY (zero-shot ac… 69.6% ▃····▃·▃▃●▅▅ beats 6/10 easy · avg 81% 2.6% at its level
ZeroSCROLLS Qasper (zero-shot F1) 46.6% ▂▂··▂·▂··●█▄ beats 4/10 medium · avg 56% 2.3% blunt · at its level
SciTab 3-class claim verification… 38.0% █···▄▄·▄▄●·▄ beats 6/7 medium · avg 51% 2.0% blunt · at its level
CSK-PN boolean commonsense QA (10… 81.4% ▃·▃·····▃●▃█ beats 3/7 easy · avg 87% 1.9% blunt · at its level
BIG-Bench Hard, 27 subsets, zero-… 37.1% █····█··●··█ beats 2/3 medium · avg 56% 1.8% blunt · at its level
PopQA 11.3% ▁▂●▃▃█▂█▆▅▁▆ beats 3/40 hard · avg 31% 1.5% typical
HateCheck functional tests, zero-… 77.7% █·····█·●··█ beats 2/3 easy · avg 84% 1.3% blunt · at its level
ZeroSCROLLS NarrativeQA (zero-sho… 21.6% ▃··▃·█●·█▃·▃ beats 5/10 hard · avg 29% 1.1% blunt
ConvRe Re2Text, normal relation, … 91.5% ▂▂▄··▂▂█●··▂ beats 11/12 easy · avg 90% 1.0% at its level · noisy
Zero-shot relation extraction, va… 46.6% █····▄····●▄ beats 4/5 medium · avg 50% 0.9% blunt · at its level
ZeroSCROLLS MuSiQue (zero-shot F1) 42.8% ▄·█··██·▄●▄▄ beats 8/10 medium · avg 43% 0.3% blunt · at its level
FANToM BeliefQ [Choice], short co… 30.5% ●·····▂····▁ beats 8/11 very hard · avg 55% 0.2% sharp · too hard for it · noisy
source results share of the score
Scaling Instruction-Finetuned Language … 4 31% ▇▇▇▇
InstructEval 2 24% ▇▇▇
Specializing Smaller Language Models to… 1 14% ▇▇
Augmentation-Adapted Retriever Improves… 2 9% ▇
ZeroSCROLLS: A Zero-Shot Benchmark for … 4 6% ▇
Chain-of-Thought Hub: A Continuous Effo… 1 4% ▇
POSQA: Probe the World Models of LLMs w… 1 3% ▇
SCITAB: A Challenging Benchmark for Com… 1 2% ▇
Say What You Mean! Large Language Model… 1 2% ▇
The CoT Collection: Improving Zero-shot… 1 2% ▇
InstructSafety: A Unified Framework for… 1 1% ▇
An Investigation of LLMs' Inefficacy in… 1 1% ▇
Aligning Instruction Tasks Unlocks Larg… 1 1% ▇
FANToM: A Benchmark for Stress-testing … 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 37.2
Licences: Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) · InstructEval ([MIT / Apache-2.0](https://github.com/declare-lab/instruct-eval/blob/49bbd2ce7e3a5bdc4024ba62a5c9c68c107ee966/LICENSE)) · Specializing Smaller Language Models towards Multi-Step Reasoning ([CC BY 4.0](../../licenses/paper-2301.12726v1-specializing-CC-BY-4-0.txt)) · Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In ([CC BY 4.0](../../licenses/acl-2023-acl-long-136-CC-BY-4-0.txt)) · ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-536-CC-BY-4-0.txt)) · Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance ([CC BY 4.0](../../licenses/paper-2305.17306v1-CC-BY-4-0.txt)) · POSQA: Probe the World Models of LLMs with Size Comparisons ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-504-CC-BY-4-0.txt)) · SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-483-CC-BY-4-0.txt)) · Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge ([CC BY 4.0](../../licenses/acl-2023-acl-long-550-CC-BY-4-0.txt)) · The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-782-CC-BY-4-0.txt)) · InstructSafety: A Unified Framework for Building Multidimensional and Explainable Safety Detector through Instruction Tuning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-700-CC-BY-4-0.txt)) · An Investigation of LLMs' Inefficacy in Understanding Converse Relations ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-429-CC-BY-4-0.txt)) · Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors ([CC BY 4.0](../../licenses/acl-2023-findings-acl-50-CC-BY-4-0.txt)) · FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-890-CC-BY-4-0.txt))
## How to run it
### Other ways to run it
1. Run it as published (fp32, needs ~11.4 GB, from [huggingface.co/google](https://huggingface.co/google/flan-t5-xl/tree/7d6315df2c2fb742f0f5b556879d730926ca9001))
```sh
uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text2text-generation', model='google/flan-t5-xl', torch_dtype='auto', device_map='auto')('Answer the question: what is the capital of France?', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/flan-t5-xl)
[interactive version](https://whatmodel.sh/model/flan-t5-xl) · [JSON](https://whatmodel.sh/api?format=json&all=1)