[whatmodel.sh](../../README.md) / PaLM 540B # PaLM 540B Google · released 2022-04 · 540.35B params · proprietary **score 46.6** ±0.4 · comparison only: no current run option verified · from 94 benchmark results across 11 sources · † no independently run results verified Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks PaLM 540B among every model on that task or test, worst to best. 
task    vs the others         rank  counts  tests
coding  █·█······█·●      #1 of 4*    2.5%      2
math    ▁▁▃▆█●▅█▃▁▁▁   #271 of 530     21%      3
coding · #1 of 4 · 2.5% of its score 
test               result  vs the others in its run   difficulty        counts  why
HumanEval (maker)   26.2%  ▁▁▁▁▂●▆▆█▅▂▁ beats 39/205  medium · avg 46%    1.8%  sharp · at its level
MBPP (maker)        36.8%  ▁▁▁●▅█▆▅▄▁▂▁ beats 14/86   medium · avg 50%    0.8%  at its level · small run
math · #271 of 530 · 21% of its score Beats [GPT-4o (2024-08-06)](gpt-4o-2024-08-06.md), [Command A (03-2025)](command-a.md), [GPT-4o (2024-05-13)](gpt-4o-2024-05-13.md) here, though they rank higher overall. Trails [Gemma 7B (base)](gemma-7b-base.md), [Phi-2 (2.7B)](phi-2.md), [Granite 8B Code Base (4K)](granite-8b-code-base-4k.md) here, though they rank lower overall. 
test                  result  vs the others in its run  difficulty        counts  why
GSM8K (maker)          56.9%  █·▅▃▃▃·▃·▃▃● beats 11/12  easy · avg 67%       17%  sharp · at its level · several runs
MGSM (maker)           45.9%  ▆·▂·▄▂▂▂▂█●▆ beats 15/18  medium · avg 48%    3.8%  sharp · at its level · several runs
Minerva MATH (maker)    8.8%  ▄··▄··●·▄··█ beats 2/5    hard · avg 33%      0.6%  at its level
other tests (47) 
test                                result  vs the others in its run    difficulty            counts  why
AQuA-RAT (maker)                     35.8%  █▁·▁▁·····●▁ beats 10/12    medium · avg 48%         10%  sharp · several runs
SVAMP (maker)                        79.0%  ▃▅█···▅·▃▃▃● beats 12/12    easy · avg 82%          7.2%  sharp · at its level · several runs
CommonsenseQA (maker)                78.1%  █······▁·▁▁● beats 10/12    easy · avg 88%          6.9%  sharp · several runs
ASDiv (maker)                        73.9%  ▅▃▃▅··▃▃·▃▃● beats 11/12    easy · avg 83%          5.4%  sharp · several runs
BIG-Bench Hard (maker)               49.1%  ▂▂▂▂·▅·▃▂●▃▃ beats 12/18    medium · avg 60%        4.7%  sharp · at its level · several runs
MMLU (maker)                         71.3%  ▁···▁··▃▁▅▅● beats 15/18    easy · avg 74%          4.5%  at its level · several runs
MAWPS (CoT-paper union) (maker)      93.3%  ▅▃█···▅·▃▃▃● beats 12/12    very easy · avg 94%     4.3%  sharp
ARC Challenge (maker)                85.2%  ▄▁▁▂▁▄█▇▃▂●▂ beats 132/140  easy · avg 86%          3.7%  sharp · at its level · several runs
BIG-bench Sports Understanding (m…   80.5%  █·····▃·▁·▄● beats 12/12    medium · avg 81%        2.4%  sharp · at its level
WinoGrande (maker)                   80.1%  ▄····▄··▄·█● beats 6/6      easy · avg 85%          2.4%  at its level · several runs
Natural Questions (closed book) (…   21.2%  ▂▁▁▁▂▄▄▃▅●▅▁ beats 43/68    hard · avg 32%          1.9%  at its level · several runs
TriviaQA (maker)                     76.9%  ▁▁▂▂▄▄▃▆▅█●▂ beats 75/80    easy · avg 80%          1.7%  several runs
BIG-bench Date Understanding (mak…   49.0%  ▃·▅·▅▃▅·▃▃·● beats 11/12    medium · avg 52%        1.7%  at its level
MMLU-STEM (Minerva subset) (maker)   58.7%  ▄······▄▄·▄● beats 4/5      medium · avg 62%        1.4%  sharp · at its level
BoolQ (maker)                        88.0%  ▁····▁▂█▆▅▆● beats 97/103   easy · avg 90%          1.4%  several runs
StrategyQA (maker)                   68.6%  █·····▅▃▃▅▃● beats 12/12    medium · avg 70%        1.4%  at its level · several runs
MedQA USMLE 4-option (few-shot pr…   58.9%  ▄······▄▄▄·● beats 4/5      medium · avg 58%        1.3%  at its level
ANLI Round 1 (prompted) (maker)      48.4%  █▁▃·▃·▄·▃··● beats 16/16    hard · avg 51%          1.2%  sharp · small run
MedMCQA (few-shot prompted) (make…   54.5%  ▄·····▄··█·● beats 4/5      medium · avg 56%        1.1%  at its level
HellaSwag (maker)                    83.4%  ▁▁▁▂▂▂▄▆█▇●▂ beats 121/135  easy · avg 87%          1.0%  several runs
TyDiQA-GoldP (1-shot, EM, 8-langu…   52.9%  ▂··▂·▂▄▄·█●█ beats 13/18    medium · avg 61%        1.0%  at its level
COPA (maker)                         93.0%  ▁·▁▁▁▅▅█▆●▁▁ beats 60/67    very easy · avg 95%     0.8%  small run · several runs
OpenBookQA (maker)                   53.4%  ▅▁▁▁▂▂▃█●▁▃▁ beats 82/107   medium · avg 55%        0.8%  at its level · small run · several runs
ARC Easy (maker)                     95.3%  ▁▃▂▄▆█▅▃▁··● beats 100/100  easy · avg 85%          0.7%  blunt · small run · several runs
WiC (prompted) (maker)               59.1%  █·▂▁······▁● beats 13/13    hard · avg 60%          0.7%  sharp · noisy · small run
ANLI Round 2 (prompted) (maker)      44.2%  █▅▃▅▃▃▅▃█··● beats 16/16    hard · avg 45%          0.6%  small run
Story Cloze Test (maker)             84.6%  ▃▃▃▂▅█▅●▁··▂ beats 44/50    easy · avg 89%          0.6%  blunt · at its level · small run · seve…
OCWCourses (Minerva) (maker)          7.1%  ▄·····▄●··▄▄ beats 2/5      hard · avg 15%          0.6%  typical
PIQA (maker)                         82.3%  ▁▁▁▂▂▃▅█▅●▁▁ beats 108/123  easy · avg 85%          0.5%  blunt · at its level · several runs
RTE (prompted) (maker)               72.9%  ▅▃·▃··▅·▅▅▃● beats 13/13    medium · avg 75%        0.5%  at its level · small run
AddSub (maker)                       91.9%  █·········█● beats 2/2      very easy · avg 89%     0.5%  small run
LAMBADA (OpenAI version), last-wo…   77.9%  ▁·····▁▁▃▃█● beats 84/86    easy · avg 84%          0.4%  blunt · several runs
MultiRC (SuperGLUE, prompted, F1a…   83.5%  ▃·▃██···▅··● beats 10/10    very easy · avg 85%     0.4%  small run
MultiArith (maker)                   94.7%  █·········●█ beats 1/2      very easy · avg 93%     0.3%  sharp · noisy · small run
PubMedQA (few-shot prompted) (mak…   55.0%  ▄·······●▄▄█ beats 1/5      medium · avg 75%        0.3%  at its level · noisy
SQuAD 2.0 (prompted, F1) (maker)     80.8%  ▃▃···▃▃█▃▃▃● beats 10/10    very easy · avg 85%     0.3%  small run
ANLI Round 3 (prompted) (maker)      45.7%  ▂·▆▂██·▄··▂● beats 16/16    hard · avg 44%          0.3%  small run
SuperGLUE WSC (prompted) (maker)     89.1%  █··▁·▃·▄▂▁▅● beats 26/26    medium · avg 82%        0.2%  blunt · at its level · small run
Winograd Schema Challenge (WSC273…   90.1%  █▃·▅▃▃█▅██▅● beats 21/21    easy · avg 92%          0.2%  blunt · small run
CoQA (maker)                         77.6%  ▂···▃▂▂·▆▅▂● beats 16/17    very easy · avg 88%     0.2%  small run
QuAC (prompted, F1) (maker)          45.2%  ▅··▃▃█··▃▅·● beats 10/10    medium · avg 53%        0.2%  blunt · at its level · small run
RACE-middle (maker)                  68.1%  ▄▄····▄█·██● beats 10/10    easy · avg 74%          0.2%  blunt · at its level · small run
RACE-high (maker)                    49.1%  ▂▄·▂▂▂▂▂▆▄▂● beats 17/17    medium · avg 53%        0.2%  blunt · at its level · small run
DROP (maker)                         69.4%  ▃▅█▅·▃·▆●▂▁▃ beats 35/44    medium · avg 62%        0.2%  blunt · at its level · small run
ReCoRD (SuperGLUE, prompted) (mak…   92.9%  ▄·▄···▄███▄● beats 10/10    very easy · avg 95%     0.1%  blunt · maxed out · small run
CommitmentBank (prompted) (maker)    51.8%  ▂▃·▂█▂▂●·▂·▂ beats 11/13    medium · avg 60%        0.1%  blunt · at its level · small run
WebQuestions (closed book) (maker)   10.6%  ▅▅▃▅▃·▃▃▅●▅▃ beats 13/17    very hard · avg 4.1%    0.0%  blunt · too hard for it · small run
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
Chain-of-Thought Prompting Elicits Reas…        9   36% ▇▇▇▇
PaLM: Scaling Language Modeling with Pa…       33   16% ▇▇
Transcending Scaling Laws with 0.1% Ext…       17   13% ▇▇
Self-Consistency Improves Chain of Thou…       10   11% ▇
Scaling Instruction-Finetuned Language …        4    9% ▇
Solving Quantitative Reasoning Problems…        4    6% ▇
Universal Self-Adaptive Prompting (subm…        9    3% ▇
Large Language Models Encode Clinical K…        3    3% ▇
Better Zero-Shot Reasoning with Self-Ad…        3    3% ▇
Challenging BIG-Bench Tasks and Whether…        1    1% ▇
WinoDict: Probing language models for i…        1    1% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 46.6
Licences: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models ([CC BY 4.0](../../licenses/paper-2201.11903v6-CC-BY-4-0.txt)) · PaLM: Scaling Language Modeling with Pathways ([CC BY 4.0](../../licenses/paper-2204.02311v5-CC-BY-4-0.txt)) · Self-Consistency Improves Chain of Thought Reasoning in Language Models ([CC BY 4.0](../../licenses/paper-2203.11171v4-CC-BY-4-0.txt)) · Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) · Solving Quantitative Reasoning Problems with Language Models ([CC BY 4.0](../../licenses/paper-2206.14858v2-CC-BY-4-0.txt)) · Universal Self-Adaptive Prompting ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-461-CC-BY-4-0.txt)) · Large Language Models Encode Clinical Knowledge ([CC BY 4.0](../../licenses/paper-2212.13138v1-CC-BY-4-0.txt)) · Better Zero-Shot Reasoning with Self-Adaptive Prompting ([CC BY 4.0](../../licenses/acl-2023-findings-acl-216-CC-BY-4-0.txt)) · Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them ([CC BY 4.0](../../licenses/acl-2023-findings-acl-824-bbh-CC-BY-4-0.txt)) · WinoDict: Probing language models for in-context word acquisition ([CC BY 4.0](../../licenses/acl-2023-eacl-main-7-CC-BY-4-0.txt))
Not counted (5) 
source                  test            score  why
Transcending Scaling …  GSM8K           54.9%  only model in this source's run
Transcending Scaling …  BIG-Bench Hard  44.8%  only model in this source's run
Transcending Scaling …  StrategyQA      76.4%  only model in this source's run
Transcending Scaling …  CommonsenseQA   76.9%  only model in this source's run
Transcending Scaling …  MGSM            45.9%  only model in this source's run
## How to run it No verified API or local download yet. [interactive version](https://whatmodel.sh/model/palm-540b) · [JSON](https://whatmodel.sh/api?format=json&all=1)