# Benchmarks

How CAR's published numbers are produced, what each column means, and how to
reproduce or contribute them. The live results table is
**[LEADERBOARD.md](./LEADERBOARD.md)** — read it alongside this methodology, not
on its own.

## What's measured

The leaderboard comes from **`car-bench-run`**, CAR's agentic task suite. It's a
**per-machine, per-model** table, not a single ranking — the same model on
different hardware is a different row, on purpose (see *Hardware normalization*).

Tasks are split into three **tracks**:

| Track | What it exercises |
|-------|-------------------|
| `core` | Deterministic runtime capabilities (state, proposals, verification, tools) — quality should be ~1.0 for any working model; this is a floor/regression check. |
| `agentic` | Multi-step tool-use tasks where the model drives a plan. |
| `judged` | Open-ended tasks scored by an LLM judge against a rubric. |

A model that can't produce valid tool calls for a track is recorded as
**Skipped** (not a 0 score) so a capability gap isn't confused with a quality
failure.

## Reading the columns

| Column | Meaning |
|--------|---------|
| **Quality** | Fraction of attempted tasks passed (per track). `—` when all attempts were skipped. |
| **Decode tok/s** | Sustained generation throughput. **Machine-dependent.** |
| **TTFT p50** | Median time-to-first-token. **Machine-dependent.** |
| **Attempted** / **Skipped** | Task counts — quality is over *attempted*, so always read the two together. |
| **Machine** | Arch / RAM / accelerator / chip. The perf numbers only mean anything next to it. |

## Hardware normalization

**Quality is approximately machine-independent; throughput and latency are not.**
`tok/s` and `TTFT` depend entirely on the chip, memory bandwidth, quantization,
and thermal state of the machine that produced the row — an M4 Max and an M5 Pro
running the same 4-bit model produce different perf and that's expected. That's
why every row carries its machine and there is **no cross-machine "fastest"
ranking**. Compare perf only within the same Machine column; compare quality
across machines freely.

Quantization is part of the model id (e.g. `mlx/qwen3-4b:4bit`,
`qwen/qwen3-4b:q4_k_m`) — a 4-bit and an 8-bit variant of the same base model are
distinct entries, since quantization trades quality for speed/size.

## Reproducing / contributing

The suite runs locally against real models — no hidden harness:

- Run it on your machine and contribute a row via
  **`scripts/bench-contribute.sh`** (details in
  `bench/results/agentic/README.md` in the source repo).
- The public table is generated by `bench-consolidate.py` from the collected
  per-machine result files and published to `LEADERBOARD.md` on each release, so
  it tracks the code it describes.

Contributed rows are attributed to their machine, so adding your hardware widens
the perf picture rather than replacing anyone's numbers.

## Other harnesses

Beyond the agentic suite, CAR is exercised against external harnesses used for
cross-checking (e.g. SWE-bench-style code tasks and an ALE agent-loop harness).
Those are development-facing and not part of the published per-machine
leaderboard; the leaderboard above is the canonical public benchmark surface.

## Caveats

- Numbers reflect the CAR versions and models available at publish time; older
  rows aren't re-run on every release.
- LLM-judged scores (`judged` track) carry the judge's variance — treat small
  quality gaps there as noise, not signal.
- A `—` in Quality means every attempt was skipped (capability gap), not a zero.