# ⚡ llmspeed

**Feel how fast a local LLM runs before you buy the hardware.**

Pick a GPU or Mac and a model, and watch the answer stream at the real speed, including the wait before the first token.

[![Live site](https://img.shields.io/badge/try_it-llmspeed.dev-e05a3c)](https://llmspeed.dev)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
[![Deploy](https://github.com/nicorobinreal/llmspeed/actions/workflows/deploy.yml/badge.svg)](https://github.com/nicorobinreal/llmspeed/actions/workflows/deploy.yml)

https://github.com/user-attachments/assets/534b4736-6b59-426e-bc0a-e93a196156b9

*RTX 5090 vs Mac Studio M5 Ultra 512GB on GLM-5.3 Flash (Q8). The 5090 can't load it.*

## Features

- **Real-time simulation:** text streams at the predicted tokens/sec, after a realistic time to first token for your prompt length
- **Race mode:** run two setups side by side, e.g. a 5090 against a Mac Studio
- **Memory fit:** see whether a model fits at each quantization, needs partial offload to system RAM, or won't run at all
- **30 GPUs, Macs and CPUs × 15 popular models**, across CUDA, Metal, ROCm, Vulkan and CPU
- **Shareable links** to any setup or race

## How the numbers work

Every estimate is labeled with how much to trust it:

| Label | Meaning |
|---|---|
| **Measured** | Benchmarked on this exact hardware, model and quantization |
| **Derived** | Interpolated from measured benchmarks of similar setups |
| **Projected** | Calculated from hardware specs, e.g. for announced but unreleased hardware |

**Sources.** Benchmarks are collected and verified from [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/), [localscore.ai](https://www.localscore.ai/), llama.cpp discussions and more.

**The model.** When no benchmark exists, llmspeed falls back to first principles:

- **Generation speed** is bound by memory bandwidth: every token reads the active weights once. For mixture-of-experts models only the active parameters count.
- **Prompt processing** is bound by compute (FP16 TFLOPS).
- Both use per-backend efficiency coefficients calibrated against the measured benchmarks, plus a slowdown as context grows.
- **Partial offload** uses a hybrid model: the GPU-resident share streams at GPU bandwidth and the spilled share at an effective system-RAM bandwidth, calibrated from measured offload runs.
- **Memory fit** counts weights, KV cache and runtime overhead against usable memory: about 75% of unified memory on Macs and 93% of VRAM on discrete GPUs.

## Contributing benchmarks

The most valuable contribution is a real measurement. To add one, open a PR that appends a row to [`benchmarks/submissions.csv`](benchmarks/submissions.csv).

1. Run llama.cpp's benchmark tool on your machine:
   ```sh
   llama-bench -m your-model.gguf -p 512 -n 128
   ```
2. Add a row with your results:

   | Column | Example |
   |---|---|
   | `hardware` | `NVIDIA RTX 4090 24GB` |
   | `model` | `Llama 3.1 8B Instruct` |
   | `quant` | `Q4_K_M` |
   | `backend` | `CUDA`, `Metal`, `ROCm`, `Vulkan` or `CPU` |
   | `engine_version` | `llama.cpp b6500` |
   | `gen_tps` | `128.4` (the `tg128` result) |
   | `prefill_tps` | `9850` (the `pp512` result) |
   | `prompt_tokens` | `512` |
   | `source_url` | a link to your post, issue or gist with the full output |
   | `notes` | anything unusual: power limits, flash attention, offload, … |

3. In the PR description, paste the full `llama-bench` output.

Results from other engines (MLX, vLLM, ExLlama, …) are welcome too; just name the engine in `engine_version`. Want a model or GPU added? Open an issue.

## Development

```sh
npm install
npm run dev      # local dev server
npm run build    # production build + prerendered landing pages + sitemap
```

Built with [Vite](https://vite.dev), [Preact](https://preactjs.com) and TypeScript. The site is fully static and deployed to GitHub Pages on every push to `main`. All predictions live in [`data/app-data.json`](data/app-data.json), which is generated by the benchmark pipeline.

## Limitations

- **Single device only.** Multi-GPU setups (tensor or layer split) aren't simulated yet.
- **Estimates, not guarantees.** Drivers, engine versions, flash attention, power limits and thermals can all move real numbers by 10–30%.
- **Projected numbers** for unreleased hardware will be replaced with measurements as they appear.

## Star history

<a href="https://star-history.com/#nicorobinreal/llmspeed&Date">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=nicorobinreal/llmspeed&type=Date&theme=dark" />
    <img alt="Star history chart" src="https://api.star-history.com/svg?repos=nicorobinreal/llmspeed&type=Date" />
  </picture>
</a>

## License

[MIT](LICENSE). Benchmark data credits go to the r/LocalLLaMA and localscore.ai communities and everyone who has shared their numbers.