[whatmodel.sh](../../README.md) / Flan-T5 Base # Flan-T5 Base Google · released 2022-10 · 0.248B params · apache-2.0 **score 24.8** ±1.4 · #686 of 732 · from 16 benchmark results across 9 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks Flan-T5 Base among every model on that task or test, worst to best. 
task  vs the others          rank  counts  tests
math  ●▁▃▅█▆▄█▄▂▁▁   #428 of 431*     15%      2
math · #428 of 431 · 15% of its score Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [BLOOM 560M](bloom-560m.md) here, though they rank lower overall. 
test          result  vs the others in its run   difficulty        counts  why
GSM8K           3.0%  ▂▂●▂▃▄▅▇██▆▄ beats 22/206  medium · avg 49%     11%  sharp · too hard for it
MGSM (maker)    0.4%  ●·▂·▄▂▂▂▂█▄▆ beats 0/18    medium · avg 48%    3.8%  sharp · too hard for it
other tests (12) 
test                                result  vs the others in its run  difficulty        counts  why
MMLU (maker)                         35.9%  ▁···▁··●▁▅▅█ beats 2/18   easy · avg 74%       35%  several runs
SVAMP                                 3.7%  █·▆▂●▂▂····▂ beats 8/12   easy · avg 70%       15%  sharp
BIG-Bench Hard (maker)               31.3%  ▂▂●▂·▅·▃▂█▃▃ beats 2/18   medium · avg 60%    4.8%  sharp · too hard for it
TyDiQA-GoldP (1-shot, EM, 8-langu…    4.1%  ▂··●·▂▄▄·███ beats 1/18   medium · avg 61%    4.8%  typical
ZeroSCROLLS QuALITY (zero-shot ac…   48.4%  ▃····▃·●▃█▅▅ beats 2/10   easy · avg 81%      4.7%  at its level
POSQA general (yes/no) size quest…   50.0%  ●···█·····██ beats 0/3    medium · avg 68%    4.4%  too hard for it
ZeroSCROLLS Qasper (zero-shot F1)    14.2%  ●▂··▂·▂··▂█▄ beats 0/10   medium · avg 56%    4.3%  blunt
SciTab 3-class claim verification…   26.6%  ●···▄▄·▄▄▄·▄ beats 0/7    medium · avg 51%    3.9%  blunt
PopQA                                 8.8%  ▁●▁▃▃█▂█▆▅▁▆ beats 1/40   hard · avg 31%      2.1%  too hard for it
ConvRe Re2Text, normal relation, …   84.6%  ▂▂▄··▂▂●▄··▂ beats 8/12   easy · avg 90%      2.1%  at its level · noisy
ZeroSCROLLS NarrativeQA (zero-sho…   16.5%  ▃··●·█▃·█▃·▃ beats 1/10   hard · avg 29%      2.1%  blunt
ZeroSCROLLS MuSiQue (zero-shot F1)   26.9%  ▄·●··██·▄▄▄▄ beats 1/10   medium · avg 43%    0.7%  blunt · at its level
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Scaling Instruction-Finetuned Language …        4   27% ▇▇▇
Large Language Models Are Reasoning Tea…        1   15% ▇▇
Model-Generated Pretraining Signals Imp…        1   13% ▇▇
ZeroSCROLLS: A Zero-Shot Benchmark for …        4   12% ▇
Augmentation-Adapted Retriever Improves…        2   11% ▇
Specializing Smaller Language Models to…        1   11% ▇
POSQA: Probe the World Models of LLMs w…        1    4% ▇
SCITAB: A Challenging Benchmark for Com…        1    4% ▇
An Investigation of LLMs' Inefficacy in…        1    2% ▇
prior: pull toward the middle                        1% ▇
weighted average                                   = 24.8
Licences: Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) · Large Language Models Are Reasoning Teachers ([CC BY 4.0](../../licenses/acl-2023-acl-long-830-CC-BY-4-0.txt)) · Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers ([CC BY 4.0](../../licenses/acl-2023-acl-long-724-CC-BY-4-0.txt)) · ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-536-CC-BY-4-0.txt)) · Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In ([CC BY 4.0](../../licenses/acl-2023-acl-long-136-CC-BY-4-0.txt)) · Specializing Smaller Language Models towards Multi-Step Reasoning ([CC BY 4.0](../../licenses/paper-2301.12726v1-specializing-CC-BY-4-0.txt)) · POSQA: Probe the World Models of LLMs with Size Comparisons ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-504-CC-BY-4-0.txt)) · SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-483-CC-BY-4-0.txt)) · An Investigation of LLMs' Inefficacy in Understanding Converse Relations ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-429-CC-BY-4-0.txt)) ## How to run it ### Other ways to run it 1. Run it as published (fp32, needs ~0.99 GB, from [huggingface.co/google](https://huggingface.co/google/flan-t5-base/tree/7bcac572ce56db69c1ea7c8af255c5d7c9672fc2)) ```sh uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text2text-generation', model='google/flan-t5-base')('Answer the question: what is the capital of France?', max_new_tokens=40)[0]['generated_text'])" ``` Model repository and documentation: [huggingface.co/google](https://huggingface.co/google/flan-t5-base) [interactive version](https://whatmodel.sh/model/flan-t5-base) · [JSON](https://whatmodel.sh/api?format=json&all=1)