[whatmodel.sh](../../README.md) / Flan-T5 Large # Flan-T5 Large Google · released 2022-10 · 0.783B params · apache-2.0 **score 31.6** ±1.1 · #653 of 732 · from 18 benchmark results across 11 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Flan-T5 Large among every model on that task or test, worst to best. 
task  vs the others          rank  counts  tests
math  ▁●▃▅█▆▄█▄▂▁▁   #423 of 431*     19%      2
math · #423 of 431 · 19% of its score Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall. 
test          result  vs the others in its run   difficulty        counts  why
GSM8K           6.9%  ▂▂▂●▃▄▅▇██▆▄ beats 28/206  medium · avg 49%     13%  sharp · too hard for it
MGSM (maker)    0.7%  ●·▂·▄▂▂▂▂█▄▆ beats 2/18    medium · avg 48%    5.4%  sharp · too hard for it
other tests (14) 
test                                result  vs the others in its run  difficulty           counts  why
MMLU (maker)                         45.1%  ▁···▁··▃●▅▅█ beats 4/18   easy · avg 74%          32%  several runs
SVAMP                                 8.3%  █·▆▂▄●▂····▂ beats 10/12  easy · avg 70%          16%  sharp
BIG-Bench Hard (maker)               37.5%  ▂▂▂▂·●·▃▂█▃▃ beats 6/18   medium · avg 60%       6.1%  sharp
TyDiQA-GoldP (1-shot, EM, 8-langu…   12.3%  ▂··▂·●▄▄·███ beats 2/18   medium · avg 61%       4.7%  typical
POSQA general (yes/no) size quest…   56.0%  █···█·····●█ beats 2/3    medium · avg 68%       4.0%  typical
ZeroSCROLLS QuALITY (zero-shot ac…   62.4%  ▃····▃·▃●█▅▅ beats 3/10   easy · avg 81%         3.5%  at its level
ZeroSCROLLS Qasper (zero-shot F1)    33.6%  ▂▂··▂·●··▂█▄ beats 3/10   medium · avg 56%       3.0%  blunt · at its level
SciTab 3-class claim verification…   32.6%  █···▄●·▄▄▄·▄ beats 3/7    medium · avg 51%       2.7%  blunt · at its level
Vectara hallucination leaderboard…   81.7%  ▁●▂▃▄▄█▆▃▂▂▁ beats 6/140  very easy · avg 94%    1.8%  blunt · noisy
HateCheck functional tests, zero-…   70.9%  ●·····█·█··█ beats 0/3    easy · avg 84%         1.8%  blunt · at its level
PopQA                                 7.2%  ●▂▁▃▃█▂█▆▅▁▆ beats 0/40   hard · avg 31%         1.8%  typical
ConvRe Re2Text, normal relation, …   71.5%  ▂▂▄··▂●█▄··▂ beats 5/12   easy · avg 90%         1.5%  at its level · noisy
ZeroSCROLLS NarrativeQA (zero-sho…   20.1%  ▃··▃·●▃·█▃·▃ beats 4/10   hard · avg 29%         1.4%  blunt
ZeroSCROLLS MuSiQue (zero-shot F1)   33.1%  ▄·█··●█·▄▄▄▄ beats 3/10   medium · avg 43%       0.5%  blunt · at its level
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Scaling Instruction-Finetuned Language …        4   30% ▇▇▇▇
Large Language Models Are Reasoning Tea…        1   16% ▇▇
Specializing Smaller Language Models to…        1   13% ▇▇
Model-Generated Pretraining Signals Imp…        1   10% ▇
Augmentation-Adapted Retriever Improves…        2   10% ▇
ZeroSCROLLS: A Zero-Shot Benchmark for …        4    8% ▇
POSQA: Probe the World Models of LLMs w…        1    4% ▇
SCITAB: A Challenging Benchmark for Com…        1    3% ▇
Vectara hallucination leaderboard               1    2% ▇
InstructSafety: A Unified Framework for…        1    2% ▇
An Investigation of LLMs' Inefficacy in…        1    1% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 31.6
Licences: Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) · Large Language Models Are Reasoning Teachers ([CC BY 4.0](../../licenses/acl-2023-acl-long-830-CC-BY-4-0.txt)) · Specializing Smaller Language Models towards Multi-Step Reasoning ([CC BY 4.0](../../licenses/paper-2301.12726v1-specializing-CC-BY-4-0.txt)) · Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers ([CC BY 4.0](../../licenses/acl-2023-acl-long-724-CC-BY-4-0.txt)) · Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In ([CC BY 4.0](../../licenses/acl-2023-acl-long-136-CC-BY-4-0.txt)) · ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-536-CC-BY-4-0.txt)) · POSQA: Probe the World Models of LLMs with Size Comparisons ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-504-CC-BY-4-0.txt)) · SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-483-CC-BY-4-0.txt)) · Vectara hallucination leaderboard ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · InstructSafety: A Unified Framework for Building Multidimensional and Explainable Safety Detector through Instruction Tuning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-700-CC-BY-4-0.txt)) · An Investigation of LLMs' Inefficacy in Understanding Converse Relations ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-429-CC-BY-4-0.txt)) ## How to run it ### Other ways to run it 1. Run it as published (fp32, needs ~3.13 GB, from [huggingface.co/google](https://huggingface.co/google/flan-t5-large/tree/0613663d0d48ea86ba8cb3d7a44f0f65dc596a2a)) ```sh uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text2text-generation', model='google/flan-t5-large')('Answer the question: what is the capital of France?', max_new_tokens=40)[0]['generated_text'])" ``` Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/flan-t5-large) [interactive version](https://whatmodel.sh/model/flan-t5-large) · [JSON](https://whatmodel.sh/api?format=json&all=1)