[whatmodel.sh](../../README.md) / PaLM 8B
# PaLM 8B
Google · released 2022-04 · 8.63B params · proprietary
**score 25.7** ±0.7 · comparison only: no current run option verified · from 51 benchmark results across 6 sources · † no independently run results verified
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks PaLM 8B among every model on that task or test, worst to best.
task vs the others rank counts tests
math ▁▁●▆██▅█▃▁▁▁ #503 of 530 9.3% 3
math · #503 of 530 · 9.3% of its score
Beats [Command R (03-2024)](command-r-03-2024.md), [Llama 2 13B Chat](llama-2-13b-chat.md), [Flan-T5 Large](flan-t5-large.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md), [OPT-125M](opt-125m.md), [GPT-2 XL (1.5B)](gpt-2-xl.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
GSM8K (maker) 4.1% █·▅▃●▃·▃·▃▃▅ beats 6/12 easy · avg 67% 8.0% sharp · too hard for it · several runs
MGSM (maker) 3.4% ▆·▂·●▂▂▂▂█▄▆ beats 4/18 medium · avg 48% 0.9% sharp · too hard for it
Minerva MATH (maker) 1.5% ●··▄··▄·▄··█ beats 0/5 hard · avg 33% 0.4% too hard for it
other tests (45)
test result vs the others in its run difficulty counts why
MAWPS (CoT-paper union) (maker) 30.5% ▅▃█···●·▃▃▃▅ beats 6/12 very easy · avg 94% 24% sharp · at its level
SVAMP (maker) 16.8% ▃▅█···●·▃▃▃▅ beats 6/12 easy · avg 82% 11% sharp
ASDiv (maker) 25.2% ▅▃▃▅··▃●·▃▃█ beats 7/12 easy · avg 83% 11% sharp
MMLU (maker) 24.3% ●···▁··▃▁▅▅█ beats 0/18 easy · avg 74% 4.2% several runs
CommonsenseQA (maker) 19.8% ●······▁·▁▁▃ beats 6/12 easy · avg 88% 4.1% sharp
WinoGrande (maker) 69.3% ▄····▄··▄·●█ beats 3/6 easy · avg 85% 2.7% at its level · several runs
BIG-bench Date Understanding (mak… 12.9% ▃·▅·▅▃●·▃▃·█ beats 6/12 medium · avg 52% 2.7% typical
BIG-bench Sports Understanding (m… 55.1% █·····●·▁·▄▁ beats 6/12 medium · avg 81% 2.2% sharp
TriviaQA (maker) 39.5% ▁▁▂▂▄▄▃●▅█▅▂ beats 36/80 easy · avg 81% 2.0% at its level
HellaSwag (maker) 68.7% ▁▁▁▂▂▂▄▆●▇▄▂ beats 65/135 easy · avg 88% 1.5% at its level
BoolQ (maker) 68.3% ▁····▁▂█●▅▆▄ beats 48/103 easy · avg 89% 1.4% at its level
MultiRC (SuperGLUE, prompted, F1a… 47.5% ▃·●██···▅··▃ beats 1/10 very easy · avg 85% 1.4% at its level · small run
COPA (maker) 86.0% ▁·▁▁▁▅▅█●▂▁▁ beats 49/67 very easy · avg 96% 1.2% at its level · small run
BIG-Bench Hard (maker) 30.8% ▂●▂▂·▅·▃▂█▃▃ beats 1/18 medium · avg 60% 1.1% sharp · too hard for it
TyDiQA-GoldP (1-shot, EM, 8-langu… 25.0% ▂··▂·▂▄●·███ beats 6/18 medium · avg 61% 1.1% typical
MMLU-STEM (Minerva subset) (maker) 22.0% ●······▄▄·▄█ beats 0/5 medium · avg 62% 1.1% sharp · too hard for it
AQuA-RAT (maker) 21.7% █▁·●▁·····▂▁ beats 8/12 medium · avg 48% 1.0% sharp · too hard for it
StrategyQA (maker) 55.6% █·····▅▃●▅▃█ beats 6/12 medium · avg 70% 1.0% typical
SQuAD 2.0 (prompted, F1) (maker) 56.4% ▃▃···▃▃●▃▃▃▃ beats 5/10 very easy · avg 85% 1.0% at its level · small run
MedMCQA (few-shot prompted) (make… 26.7% ●·····▄··█·█ beats 0/5 medium · avg 56% 1.0% too hard for it
MedQA USMLE 4-option (few-shot pr… 25.7% ●······▄▄▄·█ beats 0/5 medium · avg 58% 1.0% too hard for it
CoQA (maker) 66.0% ▂···▃▂▂·●▅▂█ beats 8/17 very easy · avg 88% 0.8% at its level · small run
Natural Questions (closed book) (… 8.4% ▂▁▁▁▂▄●▃▅█▅▁ beats 21/68 hard · avg 32% 0.8% too hard for it
RTE (prompted) (maker) 54.2% ▅▃·▃··●·▅▅▃█ beats 5/13 medium · avg 75% 0.8% small run
Winograd Schema Challenge (WSC273… 80.6% █▃·▅▃▃●▅██▅▃ beats 9/21 easy · avg 92% 0.8% blunt · at its level · small run
ARC Challenge (maker) 39.2% ▄▁▁▂▁▄●▇▃▂▃▂ beats 48/140 medium · avg 72% 0.7% sharp · noisy · small run
LAMBADA (OpenAI version), last-wo… 69.5% ▁·····▁▁▃▃●█ beats 52/86 easy · avg 85% 0.7% blunt · at its level
Story Cloze Test (maker) 77.5% ▃▃▃▂▅●▅▂▁··▂ beats 30/50 easy · avg 89% 0.7% blunt · at its level · small run
ANLI Round 1 (prompted) (maker) 34.9% █▁▃·●·▄·▃··▁ beats 10/16 hard · avg 51% 0.7% sharp · too hard for it · small run
ARC Easy (maker) 69.2% ▁▃▂▄▆●▅▃▁··▁ beats 63/100 easy · avg 85% 0.7% blunt · at its level · small run
ANLI Round 2 (prompted) (maker) 35.8% █▅▃▅▃▃●▃█··▃ beats 11/16 hard · avg 45% 0.6% too hard for it · small run
SuperGLUE WSC (prompted) (maker) 78.9% █··▁·▃·▄▂▁●▃ beats 19/26 medium · avg 82% 0.6% blunt · at its level · small run
ReCoRD (SuperGLUE, prompted) (mak… 87.8% ▄·▄···▄●██▄▄ beats 4/10 very easy · avg 95% 0.5% blunt · small run
PIQA (maker) 77.1% ▁▁▁▂▂▃▅●▅▃▁▁ beats 63/123 easy · avg 86% 0.5% blunt · at its level
PubMedQA (few-shot prompted) (mak… 34.0% ●·······▄▄▄█ beats 0/5 medium · avg 75% 0.5% noisy
OCWCourses (Minerva) (maker) 1.5% ●·····▄█··▄▄ beats 0/5 hard · avg 15% 0.5% too hard for it
RACE-middle (maker) 57.9% ▄▄····▄●·███ beats 4/10 easy · avg 74% 0.4% blunt · at its level · small run
DROP (maker) 45.1% ▃▅█▅·●·▆▄▂▁▃ beats 26/44 medium · avg 62% 0.4% blunt · at its level · small run
QuAC (prompted, F1) (maker) 33.1% ▅··▃▃●··▃▅·▃ beats 4/10 medium · avg 53% 0.4% blunt · at its level · small run
WiC (prompted) (maker) 47.0% ●·▂▁······▁▁ beats 6/13 hard · avg 60% 0.3% sharp · too hard for it · noisy · small…
RACE-high (maker) 42.3% ▂▄·▂▂▂▂▂▆●▂█ beats 11/17 medium · avg 53% 0.3% blunt · small run
ANLI Round 3 (prompted) (maker) 34.5% ▂·▆●██·▄··▂▂ beats 4/16 hard · avg 44% 0.3% too hard for it · small run
OpenBookQA (maker) 48.4% ▅▁▁▁▂▂▃●▃▁▃▁ beats 63/107 medium · avg 59% 0.2% noisy · small run
CommitmentBank (prompted) (maker) 41.1% ▂▃·▂●▂▂▂·▂·▂ beats 5/13 medium · avg 60% 0.2% blunt · at its level · small run
WebQuestions (closed book) (maker) 6.7% ▅▅▃▅▃·▃▃●█▅▃ beats 10/17 very hard · avg 4.1% 0.0% blunt · too hard for it · small run
source results share of the score
Chain-of-Thought Prompting Elicits Reas… 9 63% ▇▇▇▇▇▇▇▇
PaLM: Scaling Language Modeling with Pa… 30 23% ▇▇▇
Scaling Instruction-Finetuned Language … 4 6% ▇
Solving Quantitative Reasoning Problems… 4 4% ▇
Large Language Models Encode Clinical K… 3 2% ▇
WinoDict: Probing language models for i… 1 1% ▇
prior: pull toward the middle 0% ▇
weighted average = 25.7
Licences: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models ([CC BY 4.0](../../licenses/paper-2201.11903v6-CC-BY-4-0.txt)) · PaLM: Scaling Language Modeling with Pathways ([CC BY 4.0](../../licenses/paper-2204.02311v5-CC-BY-4-0.txt)) · Scaling Instruction-Finetuned Language Models ([CC BY 4.0](../../licenses/paper-2210.11416v5-CC-BY-4.0.txt)) · Solving Quantitative Reasoning Problems with Language Models ([CC BY 4.0](../../licenses/paper-2206.14858v2-CC-BY-4-0.txt)) · Large Language Models Encode Clinical Knowledge ([CC BY 4.0](../../licenses/paper-2212.13138v1-CC-BY-4-0.txt)) · WinoDict: Probing language models for in-context word acquisition ([CC BY 4.0](../../licenses/acl-2023-eacl-main-7-CC-BY-4-0.txt))
## How to run it
No verified API or local download yet.
[interactive version](https://whatmodel.sh/model/palm-8b) · [JSON](https://whatmodel.sh/api?format=json&all=1)