[whatmodel.sh](../../README.md) / GPT-2 XL (1.5B)
# GPT-2 XL (1.5B)
OpenAI · released 2019-11 · 1.558B params · mit
**score 6.5** ±0.9 · #722 of 732 · from 52 benchmark results across 15 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks GPT-2 XL (1.5B) among every model on that task or test, worst to best.
task vs the others rank counts tests
math ▁▁▃●█▆▄█▄▂▁▁ #384 of 431* 0.7% 1
math · #384 of 431 · 0.7% of its score
Beats [Yi 34B Chat](yi-34b-chat.md), [Llama 3 8B Instruct](llama-3-8b.md), [Command R (03-2024)](command-r-03-2024.md) here, though they rank higher overall. Trails [GPT-Neo 125M](gpt-neo-125m.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
MathQA 23.6% ▃·▃▃●▅··▃·▃▃ beats 3/10 hard · avg 34% 0.7% blunt · too hard for it
other tests (24)
test result vs the others in its run difficulty counts why
ARC Challenge 25.0% ●····▃▃···▅█ beats 0/8 very hard · avg 27% 20% sharp · too hard for it · several runs
HellaSwag 40.0% ●▃▃··▅····▃█ beats 0/8 easy · avg 87% 15% several runs
OpenBookQA 22.4% ●······▂·▆·▂ beats 0/8 very hard · avg 26% 13% too hard for it · several runs
ARC Easy 58.3% ●·▃▃··▃··▃·█ beats 1/8 easy · avg 87% 8.6% blunt · at its level · several runs
Story Cloze Test 67.6% ▃▃▃●▅█▅▂▁··▂ beats 15/50 easy · avg 91% 6.2% blunt · at its level · several runs
PROST physical reasoning (zero-sh… 30.1% █···██·█···● beats 4/4 easy · avg 86% 5.0% too hard for it
COPA 69.0% ▁·▁▁▁▅●█▆▂▁▁ beats 23/67 very easy · avg 97% 4.8% at its level · several runs
PIQA 70.5% ▄●··▄·▄·▄·██ beats 1/8 easy · avg 88% 4.6% blunt · at its level · several runs
COPEN concept identification in c… 37.0% ▃···▃▃·▃·●·█ beats 4/7 easy · avg 81% 3.2% typical
BoolQ 61.8% ▄●▄···▄··█▄▄ beats 1/8 easy · avg 88% 3.1% several runs
WinoGrande 58.3% ●··█···█▄▄·▄ beats 1/8 easy · avg 88% 3.0% several runs
Social IQA 39.4% ●▄····█▄▄··▄ beats 0/7 medium · avg 71% 2.5% blunt · several runs
LAMBADA (OpenAI version), last-wo… 51.2% ●·▄··█··█··▄ beats 0/7 very easy · avg 88% 2.5% blunt · at its level
TriviaQA 11.8% ▁▁▂▂●▄▃▆▅█▅▂ beats 14/80 easy · avg 77% 1.9% small run
12-dataset text classification, 4… 41.8% ▂●··▂·█···▂▂ beats 1/8 easy · avg 79% 1.6% blunt · at its level
SciQ 83.2% ▂·▁▂▂●▆▃▇▆█▂ beats 9/51 very easy · avg 98% 1.2% blunt · at its level · small run
PubMedQA 60.4% ▂·▂●·▄▂▄▄▂█▂ beats 4/17 medium · avg 74% 0.8% blunt · at its level · several runs
MedMCQA 30.9% ▂▂··●▄▂▂·▄██ beats 3/17 medium · avg 63% 0.5% too hard for it
MedQA (USMLE, 4 options) 28.0% ▁▁··●▁▁··▄▃█ beats 4/17 medium · avg 71% 0.4% sharp · too hard for it
COPEN conceptual similarity judgm… 11.7% ▄···█···█●·▄ beats 6/7 hard · avg 22% 0.3% blunt · too hard for it
Open PT-LLM Leaderboard average (… 25.9% ●▁▁▁▁▁▁▁▂█▅▁ beats 23/276 medium · avg 72% 0.3% too hard for it · noisy
TruthfulQA generation, % true (hu… 29.3% █●····█·██·█ beats 1/5 medium · avg 42% 0.0% blunt · at its level
COPEN conceptual property judgmen… 47.1% ●·······▁··▁ beats 3/7 very hard · avg 52% 0.0% blunt · too hard for it · noisy
CommonsenseQA 38.6% █·····▂●▃▂▄▄ beats 18/38 easy · avg 87% 0.0% sharp · too hard for it · noisy · sever…
source results share of the score
Coherence boosting: When your pretraine… 9 25% ▇▇▇
Surface Form Competition 7 20% ▇▇
Textbooks Are All You Need II: phi-1.5 … 8 11% ▇
CASE: Commonsense-Augmented Score with … 4 10% ▇
EleutherAI GPT-J-6B model card 4 9% ▇
Tunable Soft Prompts are Messengers in … 5 8% ▇
PROST 1 5% ▇
COPEN 3 4% ▇
RankGen: Improving Text Generation with… 2 3% ▇
Adaptation with Self-Evaluation to Impr… 1 2% ▇
Generative Calibration for In-context L… 1 2% ▇
Open Medical-LLM Leaderboard results 3 1% ▇
EleutherAI GPT-Neo 2.7B model card 2 1% ▇
Open Portuguese LLM Leaderboard 1 0% ▇
TruthfulQA: Measuring How Models Mimic … 1 0% ▇
prior: pull toward the middle 0% ▇
weighted average = 6.5
Licences: Coherence boosting: When your pretrained language model is not paying enough attention ([CC BY 4.0](../../licenses/acl-2022-acl-long-565-CC-BY-4-0.txt)) · Surface Form Competition ([CC BY 4.0](../../licenses/acl-2021-emnlp-main-564-CC-BY-4.0.txt)) · Textbooks Are All You Need II: phi-1.5 technical report ([CC BY 4.0](../../licenses/paper-2309.05463v1-CC-BY-4.0.txt)) · CASE: Commonsense-Augmented Score with an Expanded Answer Space ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-180-CC-BY-4-0.txt)) · EleutherAI GPT-J-6B model card ([Apache-2.0](../../licenses/eleutherai-gpt-j-6b-card-Apache-2.0.txt)) · Tunable Soft Prompts are Messengers in Federated Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-976-CC-BY-4-0.txt)) · PROST ([CC BY 4.0](../../licenses/_cc-by-4.0-legalcode.txt)) · COPEN ([CC BY 4.0](../../licenses/copen-emnlp2022-zero-shot-CC-BY-4.0.txt)) · RankGen: Improving Text Generation with Large Ranking Models ([CC BY 4.0](../../licenses/acl-2022-emnlp-main-15-CC-BY-4-0.txt)) · Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-345-CC-BY-4-0.txt)) · Generative Calibration for In-context Learning ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-152-CC-BY-4-0.txt)) · Open Medical-LLM Leaderboard results ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · EleutherAI GPT-Neo 2.7B model card ([MIT](../../licenses/eleutherai-gpt-neo-2-7b-card-MIT.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · TruthfulQA: Measuring How Models Mimic Human Falsehoods ([CC BY 4.0](../../licenses/acl-2022-acl-long-229-CC-BY-4-0.txt))
## How to run it
### llama.cpp (recommended)
1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
```sh
# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh
# Windows
irm https://llama.app/install.ps1 | iex
```
2. Run GPT-2 XL (1.5B) (Q4_K_M, needs ~1.2 GB, from [huggingface.co/mradermacher](https://huggingface.co/mradermacher/gpt2-xl-GGUF/tree/a6a1b25a992d7541b185b0cd1227b47311f4eab6))
```sh
llama cli -hf mradermacher/gpt2-xl-GGUF:Q4_K_M
```
3. Or start a local server and open the address it prints (http://localhost:9931)
```sh
llama serve -hf mradermacher/gpt2-xl-GGUF:Q4_K_M
```
Other versions: Q8_0 (~1.7 GB) `llama cli -hf mradermacher/gpt2-xl-GGUF:Q8_0`
### LM Studio
1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download)
2. Run GPT-2 XL (1.5B) (Q4_K_M, needs ~1.2 GB, from [huggingface.co/mradermacher](https://huggingface.co/mradermacher/gpt2-xl-GGUF/tree/a6a1b25a992d7541b185b0cd1227b47311f4eab6))
```sh
lms get https://huggingface.co/mradermacher/gpt2-xl-GGUF@Q4_K_M -y
```
3. Then run lms ls to see its name, and lms chat followed by that name
Other versions: Q8_0 (~1.7 GB) `lms get https://huggingface.co/mradermacher/gpt2-xl-GGUF@Q8_0 -y` · BF16 (~3.1 GB, Mac only) `lms get https://huggingface.co/warshanks/gpt2-xl-bf16 -y`
### Ollama
1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download)
```sh
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
2. Run GPT-2 XL (1.5B) (Q4_K_M, needs ~1.2 GB, from [huggingface.co/mradermacher](https://huggingface.co/mradermacher/gpt2-xl-GGUF/tree/a6a1b25a992d7541b185b0cd1227b47311f4eab6))
```sh
ollama run hf.co/mradermacher/gpt2-xl-GGUF:Q4_K_M
```
Other versions: Q8_0 (~1.7 GB) `ollama run hf.co/mradermacher/gpt2-xl-GGUF:Q8_0`
### MLX (Mac)
1. Install MLX (Mac). Apple silicon Macs only, text only.
```sh
# with uv
uv tool install mlx-lm
# or
pip install mlx-lm
```
2. Run GPT-2 XL (1.5B) (BF16, needs ~3.1 GB, Mac only, from [huggingface.co/warshanks](https://huggingface.co/warshanks/gpt2-xl-bf16/tree/496e535a68bc1775d18c6acdaa66358cdb5ea1aa))
```sh
mlx_lm.chat --model warshanks/gpt2-xl-bf16
```
3. Or start an OpenAI-compatible server at http://localhost:8080/v1
```sh
mlx_lm.server --model warshanks/gpt2-xl-bf16
```
### Other ways to run it
1. Run it as published (fp32, needs ~6.43 GB, from [huggingface.co/openai-community](https://huggingface.co/openai-community/gpt2-xl/tree/15ea56dee5df4983c59b2538573817e1667135e2))
```sh
uv run --with torch --with 'transformers<5' python -c "from transformers import pipeline; print(pipeline('text-generation', model='openai-community/gpt2-xl')('Once upon a time', max_new_tokens=40)[0]['generated_text'])"
```
Model repository and documentation: [huggingface.co/openai-community](https://huggingface.co/openai-community/gpt2-xl)
## Published speed measurements
Reference results for the hardware, engine and settings below. These are not verified speeds of the downloads above or predictions for your machine.
hardware precision speed engine and settings
Apple M2 Pro (16-core GPU),… Q4_K_M 74.96 tok/s llama.cpp · whatmodel bench.py 3.…
Each speed links to its source, which has the full settings.
[interactive version](https://whatmodel.sh/model/gpt-2-xl) · [JSON](https://whatmodel.sh/api?format=json&all=1)