[whatmodel.sh](../../README.md) / U-PaLM 540B # U-PaLM 540B Google · released 2022-10 · 540B params · proprietary **score 47.9** ±0.9 · comparison only: no current run option verified · from 21 benchmark results across 2 sources · † no independently run results verified Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks U-PaLM 540B among every model on that task or test, worst to best. 
task  vs the others         rank  counts  tests
math  ▁▁▃▆█●▅█▃▁▁▁   #223 of 530     26%      2
math · #223 of 530 · 26% of its score Beats [MiniMax-M2.1](minimax-m2-1.md), [DeepSeek V3 (Dec '24)](deepseek-v3.md), [Qwen3 235B-A22B (Apr 2025, hybrid)](qwen3-235b-a22b.md) here, though they rank higher overall. Trails [Mixtral 8x7B v0.1 (base)](mixtral-8x7b-v0-1.md), [Phi-4-mini-reasoning 3.8B](phi-4-mini-reasoning.md) here, though they rank lower overall. 
test           result  vs the others in its run   difficulty        counts  why
MGSM (maker)    49.9%  ▆·▂·▄▂▂▂▂█▄● beats 16/18   medium · avg 48%     18%  sharp · at its level · several runs
GSM8K (maker)   58.5%  ▂▂▂▂▃▄▅●██▆▄ beats 98/206  easy · avg 67%      8.1%  sharp · at its level · small run
other tests (16) 
test                                result  vs the others in its run    difficulty           counts  why
BIG-Bench Hard (maker)               49.2%  ▂▂▂▂·▅·▃▂●▃▃ beats 13/18    medium · avg 60%        17%  sharp · at its level · several runs
MMLU (maker)                         71.5%  ▁···▁··▃▁▅▅● beats 16/18    easy · avg 74%          16%  at its level · several runs
CommonsenseQA (maker)                80.1%  █·····▂▂▃▂▄● beats 36/38    easy · avg 83%         8.3%  sharp · at its level · small run
TyDiQA-GoldP (1-shot, EM, 8-langu…   54.6%  ▂··▂·▂▄▄·█●█ beats 14/18    medium · avg 61%       4.4%  at its level
ARC Challenge (maker)                60.3%  ▄▁▁▂▁▄█▇●▂▃▂ beats 107/140  medium · avg 65%       4.2%  sharp · at its level · small run
Natural Questions (closed book) (…   21.7%  ▂▁▁▁▂▄▄▃▅●▅▁ beats 44/68    hard · avg 29%         3.4%  at its level
WinoGrande (maker)                   82.6%  ▁▁▁▁▂▁▁▅▇█▃● beats 127/131  easy · avg 85%         3.4%  at its level
TriviaQA (maker)                     76.4%  ▁▁▂▂▄▄▃▆▅█●▂ beats 74/80    easy · avg 80%         3.1%  typical
OpenBookQA (maker)                   53.6%  ▅▁▁▁▂▂▃█●▁▃▁ beats 84/107   medium · avg 55%       2.8%  at its level · small run
BoolQ (maker)                        88.8%  ▁····▁▂█▆▅▆● beats 101/103  easy · avg 90%         2.5%  typical
HellaSwag (maker)                    84.1%  ▁▁▁▂▂▂▄▆█▇●▂ beats 123/135  easy · avg 87%         2.5%  typical
COPA (maker)                         96.0%  ▁·▁▁▁▅▅█▆▂●▁ beats 64/67    very easy · avg 95%    1.4%  small run
LAMBADA (OpenAI version), last-wo…   80.5%  ▁·····▁▁▃▃█● beats 86/86    easy · avg 84%         1.1%  blunt
Story Cloze Test (maker)             87.0%  ▃▃▃▂▅█▅▂●··▂ beats 46/50    easy · avg 89%         1.0%  blunt · small run
PIQA (maker)                         84.1%  ▁▁▁▂▂▃▅█▅●▁▁ beats 117/123  easy · avg 85%         1.0%  blunt · at its level
ARC Easy (maker)                     85.9%  ▁▃▂▄▆█▅▃●··▁ beats 97/100   easy · avg 85%         0.9%  blunt · small run
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Transcending Scaling Laws with 0.1% Ext…       17   58% ▇▇▇▇▇▇▇
Scaling Instruction-Finetuned Language …        4   41% ▇▇▇▇▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 47.9
Licences: Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) ## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/u-palm-540b) · [JSON](https://whatmodel.sh/api?format=json&all=1)