[whatmodel.sh](../../README.md) / Flan-T5 XXL
# Flan-T5 XXL
Google · released 2022-10 · 11.3B params · apache-2.0
**score 39.0** ±0.9 · #579 of 732 · from 26 benchmark results across 18 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks Flan-T5 XXL among every model on that task or test, worst to best.
task vs the others rank counts tests
chat & writing ▁▁▁▃▆▄▆●▆▇▇▃ #230 of 402* 0.0% 1
math ▁▁●▅█▆▄█▄▂▁▁ #398 of 431 24% 2
chat & writing · #230 of 402 · 0.0% of its score
Beats [Granite 4.2 8B](granite-4-2-8b.md), [Mistral Small 4 (119B-A6.5B)](mistral-small-4.md), [Olmo 3.1 32B Think](olmo-3-1-32b-think.md) here, though they rank higher overall.
test result vs the others in its run difficulty counts why
MixInstruct GPT-Rank (LLM-Blender… 8.8100 mean rank (1-11) ●···▄··▄···▄ beats 0/4 avg 6.709 mean rank (1-… 0.0% blunt · noisy
math · #398 of 431 · 24% of its score
Beats [Command R (03-2024)](command-r-03-2024.md), [Molmo 7B-D (0924)](molmo-7b-d.md), [Mistral 7B Instruct v0.3](mistral-7b-instruct-v0-3.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K 16.1% ▂▂▂▂▃●▅▇██▆▄ beats 44/206 medium · avg 49% 16% sharp · several runs
MGSM (maker) 4.9% ▆·▂·●▂▂▂▂█▄▆ beats 5/18 medium · avg 48% 7.2% sharp
other tests (18)
test result vs the others in its run difficulty counts why
BIG-Bench Hard (maker) 45.3% ▂▂▂▂·▅·▃●█▃▃ beats 9/18 medium · avg 68% 27% sharp · at its level · several runs
MMLU (maker) 55.1% ▁···▁··▃▁●▅█ beats 8/18 easy · avg 74% 22% at its level · several runs
Factual knowledge recall, Premium… 20.6% ▅····▃▅·●█·█ beats 5/13 medium · avg 52% 4.6% at its level
TyDiQA-GoldP (1-shot, EM, 8-langu… 19.0% ▂··▂·▂●▄·███ beats 4/18 medium · avg 61% 4.0% at its level
Factual knowledge recall, Premium… 20.9% ▃●▃··▅▃·█▃▃▅ beats 1/13 medium · avg 61% 2.5% at its level
ZeroSCROLLS QuALITY (zero-shot ac… 75.2% ▃····▃·▃▃█●▅ beats 7/10 easy · avg 81% 2.3% at its level
ZeroSCROLLS Qasper (zero-shot F1) 48.3% ▂▂··▂·▂··▂●▄ beats 5/10 medium · avg 56% 2.0% blunt · at its level
This is not a Dataset (negation),… 66.1% █▁····▃▁·▁·● beats 14/14 hard · avg 66% 1.9% sharp · noisy
SciTab 3-class claim verification… 34.9% █···▄▄·●▄▄·▄ beats 4/7 medium · avg 51% 1.8% blunt · at its level
BIG-Bench Hard, 27 subsets, zero-… 41.0% █····█··█··● beats 3/3 medium · avg 56% 1.6% blunt · at its level
CSK-PN boolean commonsense QA (10… 83.1% ▃·▃·····▃▃●█ beats 4/7 easy · avg 87% 1.6% blunt · at its level
HateCheck functional tests, zero-… 80.9% █·····█·█··● beats 3/3 easy · avg 84% 1.1% blunt · at its level
ZeroSCROLLS NarrativeQA (zero-sho… 19.3% ▃··▃·●▃·█▃·▃ beats 3/10 hard · avg 29% 1.0% blunt
ConvRe Re2Text, normal relation, … 79.4% ▂▂▄··▂▂●▄··▂ beats 6/12 easy · avg 90% 0.9% at its level · noisy
Zero-shot relation extraction, va… 46.5% █····▄····●▄ beats 3/5 medium · avg 50% 0.8% blunt · at its level
StoryAnalogy multiple-choice stor… 45.4% █·······█··● beats 2/2 hard · avg 42% 0.4% blunt · noisy · small run
ZeroSCROLLS MuSiQue (zero-shot F1) 46.8% ▄·█··██·▄▄●▄ beats 9/10 medium · avg 43% 0.3% blunt · at its level
FANToM BeliefQ [Choice], short co… 27.3% ●·····▂····▁ beats 7/11 very hard · avg 55% 0.2% sharp · too hard for it · noisy
source results share of the score
Scaling Instruction-Finetuned Language … 4 30% ▇▇▇▇
InstructEval 2 22% ▇▇▇
Specializing Smaller Language Models to… 1 13% ▇▇
Towards a Holistic Evaluation of LLMs o… 2 7% ▇
ZeroSCROLLS: A Zero-Shot Benchmark for … 4 6% ▇
Unnatural Instructions: Tuning Language… 1 4% ▇
Increasing Probability Mass on Answer C… 1 4% ▇
Chain-of-Thought Hub: A Continuous Effo… 1 3% ▇
This is not a Dataset: A Large Negation… 1 2% ▇
SCITAB: A Challenging Benchmark for Com… 1 2% ▇
The CoT Collection: Improving Zero-shot… 1 2% ▇
Say What You Mean! Large Language Model… 1 2% ▇
InstructSafety: A Unified Framework for… 1 1% ▇
An Investigation of LLMs' Inefficacy in… 1 1% ▇
Aligning Instruction Tasks Unlocks Larg… 1 1% ▇
StoryAnalogy: Deriving Story-level Anal… 1 0% ▇
FANToM: A Benchmark for Stress-testing … 1 0% ▇
LLM-Blender: Ensembling Large Language … 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 39.0
Licences: Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) · InstructEval ([MIT / Apache-2.0](https://github.com/declare-lab/instruct-eval/blob/49bbd2ce7e3a5bdc4024ba62a5c9c68c107ee966/LICENSE)) · Specializing Smaller Language Models towards Multi-Step Reasoning ([CC BY 4.0](../../licenses/paper-2301.12726v1-specializing-CC-BY-4-0.txt)) · Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-536-CC-BY-4-0.txt)) · Unnatural Instructions: Tuning Language Models with ([CC BY 4.0](../../licenses/acl-2023-acl-long-806-CC-BY-4-0.txt)) · Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-522-CC-BY-4-0.txt)) · Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance ([CC BY 4.0](../../licenses/paper-2305.17306v1-CC-BY-4-0.txt)) · This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-531-CC-BY-4-0.txt)) · SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-483-CC-BY-4-0.txt)) · The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-782-CC-BY-4-0.txt)) · Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge ([CC BY 4.0](../../licenses/acl-2023-acl-long-550-CC-BY-4-0.txt)) · InstructSafety: A Unified Framework for Building Multidimensional and Explainable Safety Detector through Instruction Tuning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-700-CC-BY-4-0.txt)) · An Investigation of LLMs' Inefficacy in Understanding Converse Relations ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-429-CC-BY-4-0.txt)) · Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors ([CC BY 4.0](../../licenses/acl-2023-findings-acl-50-CC-BY-4-0.txt)) · StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-706-CC-BY-4-0.txt)) · FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-890-CC-BY-4-0.txt)) · LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/))
## How to run it
### Other ways to run it
1. Run it as published (fp32, needs ~45.07 GB, from [huggingface.co/google](https://huggingface.co/google/flan-t5-xxl/tree/ae7c9136adc7555eeccc78cdd960dfd60fb346ce))
```sh
uv run --with torch --with 'transformers<5' --with accelerate python -c "from transformers import pipeline; print(pipeline('text2text-generation', model='google/flan-t5-xxl', torch_dtype='auto', device_map='auto')('Answer the question: what is the capital of France?', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/flan-t5-xxl)
[interactive version](https://whatmodel.sh/model/flan-t5-xxl) · [JSON](https://whatmodel.sh/api?format=json&all=1)