# Benchmarks

How BertBrowser's speed is measured, kept, and defended. The tool is `bb-bench`
(`src/crates/bb-bench`); the evidence is `docs/performance.md`, generated from the baselines in
this folder. The C# app's numbers, for comparison, are in the `bertbrowser-csharp-archive` repository (its baselines and
`docs/performance.md`).

## Layout

| Path | What |
|---|---|
| `bench/baselines/<machine-key>.json` | One committed baseline per machine, all tiers merged. Compare runs against the one matching the current machine. |
| `bench/baselines/<machine-key>.real.json` | The same machine measured against its own real index and folders (`--real`). Never compared with a synthetic file. |
| `bench/baselines/ci-windows-latest.json` | The GitHub runner's baseline. CI gates allocations against it. |
| `bench/real.local.json` | Your real-data config. Keep it out of commits. |
| `bench-results/` | Where runs land. Gitignored. |
| `docs/performance.md` | Generated by `report`. Never edit by hand. |
| `%LOCALAPPDATA%\bertbrowser-bench-` | The cached synthetic corpus and real-DB backups. Safe to delete. |

## Three tiers, one schema

| Tier | Measures | How | Unit |
|---|---|---|---|
| A | `bb-core` code paths: search, index writes, planners, hashing, preview readers... | `bb-bench run`: a hand-rolled runner (`src/crates/bb-bench/src/suite.rs`) with a counting global allocator | ns, bytes allocated |
| B | Whole UI scenarios: open a 10k folder, flat view, tiles, search, transfer, preview | The offscreen harness, `tools/ui/bench-*.bbs`, each scenario K times in a fresh process | ms, process memory |
| C | Startup of the real `BertBrowser.exe` to the first frame showing the rows (`listingPainted`), and what it holds once there | N launches against a scratch data dir, marks and memory written by the app | ms, process memory |

A Tier B script is an ordinary harness script with a few extra verbs: `time <name> <verb>[; <verb>...]`
wraps any verb (or chain) under a stopwatch, `repeat <n> <verb>[; <verb>...]` runs one n times, `mem
<name>` reads the process back, and `settle-thumbnails` / `settle-preview` wait for the two things
`settle` deliberately does not. A `#! files=10000 dirs=50` line carries the fixture size into the result.
`bb-bench ui` runs each script K times as separate `bb-harness.exe` processes and takes the median.

Tier C refuses while a real BertBrowser is running: the single-instance claim is per user, so the launch
would be a "second" copy and measure a different path. `--allow-running` measures it anyway and records
`instance=second`; do not commit that as a baseline. It is the one sanctioned launch of
`BertBrowser.exe` - a window appears offscreen for a second per launch.

`--session` comes back to a saved session (two panes, three tabs) over a million-row database instead of
opening a named folder over an empty one, under its own `startup.session.*` ids, and fails if the session did
not come back. `--affinity <hex mask>` restricts the launches to those processors: what parallel start-up buys
depends on the core count, and a 24-thread machine hides it. A/B two builds with `--app`, alternating runs -
another program's load moved these numbers by 20% in an afternoon, so a run against an old baseline says little
about a small change. `pageFaults.*` in the entries' parameters are evidence, never gated.

Every tier writes the `BenchResultSet` JSON in `bb-core/src/benchmarking` (schema v2). Only `bb-bench`
writes it; the harness and the app emit raw samples it aggregates.

## The loop

```powershell
pwsh .\build.ps1 -Config release
cargo build -p bb-bench --release
$bench = "target\release\bb-bench.exe"

& $bench run --filter *fsindex*               # measure what you are working on (glob over the benchmark id, case-blind)
# ...change code, rebuild...
& $bench run --filter *fsindex* --out bench-results\after.json
& $bench compare --baseline bench\baselines\<your-key>.json --current bench-results\after.json
```

`compare` exits 1 on a regression: a median more than 15% slower than the baseline (and past a small
absolute noise floor), or more bytes allocated than the benchmark's tolerance allows. Exit 2 means a
benchmark errored, 64 a usage error. A debug build is refused (`--allow-debug` overrides). `--help` lists
every flag; `--repo <dir>` points any verb at another checkout.

### Recording a baseline

```powershell
& $bench run                      # Tier A, default job - long; this is the number that gets committed
& $bench ui --repeat 5            # Tier B (uses out\release\harness\bb-harness.exe)
& $bench startup --launches 10    # Tier C - the first launch is cold and discarded
& $bench baseline --from bench-results\current.json --from bench-results\ui.json --from bench-results\startup.json
& $bench report
git add bench\baselines docs\performance.md
```

A new machine has no baseline; the first `baseline` creates it. **Re-record deliberately.** When a change
is slower on purpose - a correctness fix, a feature that costs something - re-record in the same commit and
say why in the message. The baseline is a statement, not a log.

To re-record one tier, pass the existing baseline first so the others survive:
`baseline --from bench\baselines\<key>.json --from bench-results\ui.json`.

### CI

The `bench` job in `unstable.yml` runs Tier A with `--ci` (the short job) and `compare --ci`, which
**gates allocations only** - wall-clock on a shared runner is noise and is reported, not judged. The job
summary carries the table; the results are an artifact. On the first run (no baseline committed) it writes
`candidate-baseline.json` into the artifact and passes; commit that as
`bench/baselines/ci-windows-latest.json`. Accepting an allocation change is the same move.

The job runs beside the build, not before it: a red benchmark never withholds an unstable build.

## Allocation tolerance

Pure code allocates exactly the same bytes every run (the counting allocator sees only Rust), and one
extra allocation is the regression the CI gate exists for. Code over SQLite, threads or disk does not have
that property; those benchmarks register a non-zero `tolerance_pct`, which travels in the result. Do not
loosen a pure benchmark to make a run pass.

## Process memory

Three measurements, and they are not interchangeable:

| Where | What | Read it as |
|---|---|---|
| Tier A allocations/op | Bytes one call allocates | Churn. Gated in CI. |
| Tier B `mem <name>` -> `ui.<scenario>.<name>` | The harness process: Rust allocator, process heap, private bytes, working set, peak | What the app *holds* at that point. The OS numbers include the harness and the software rasteriser. |
| Tier C `startup.memory.firstListing` / `.collected` | The real `BertBrowser.exe` reading itself once the first listing is up | The number a user sees in Task Manager. |

**Rust allocator** is the bytes Rust holds, counted at the allocator (`bb_allocator_held`, only under a measurement):
requested and not yet freed, without mimalloc's free pages. Ask for nothing better from mimalloc: its counters want
a stats build, and on Windows `mi_process_info`'s "commit" is the whole process's private bytes. **Process heap** is
what is in use in the Windows heap: the C++ side and the system DLLs, and never Rust. Neither contains the other,
and a leak shows in only one of them, so both are gated. Private bytes contains both, the free pages allocators
keep and everything else, and is too loose to catch a small leak on its own.

`tools/ui/bench-memory.bbs` is the scenario about memory and nothing else: idle, 10,000 rows, 20,000 flat,
after leaving them, then two soaks - 25 navigations, 15 tabs opened and closed - each followed by
`assert-mem allocator|heap under <mb> since <name>`, a budget on *growth* that fails the run on any machine.
That is the leak check.

`compare` puts snapshots in their own table and gates them locally (never in CI): the Rust allocator and the
process heap at 10% over 1 MB, private bytes at 25% over 16 MB. The working set is reported, never judged.
`--memory-threshold <pct>` moves the 10% line; `--no-memory` reports without gating. A snapshot that grew on purpose is
re-recorded like a slowdown: same commit, reason in the message.

## Real data

`--real` runs the read-only subset against your actual index and folders. It never opens the live
database: it takes a SQLite backup once a day and reads that. `bench/real.local.json`:

```json
{
  "dbPath": "C:\\Users\\you\\.bertbrowser\\bertbrowser.db",
  "searchRoot": "C:\\Source",
  "listingFolder": "C:\\Windows\\System32",
  "treeFolder": "C:\\Source",
  "textFolder": "C:\\Source",
  "mediaFolder": "D:\\Photos"
}
```

Results go to `bench-results/current.real.json` and baselines to `<key>.real.json`.

## Noise

- **Compare like with like.** The job (`default` or `short`) is recorded in every results file and
  `compare` says so above the table when they differ; the short job's times are few samples. The
  committed machine baselines are the default job; the CI baseline is the short job, recorded by CI itself.
- Close what you can; plug in; use the High performance power plan (the plan is recorded).
- The index helper tails the USN journal into your database the whole time. It is recorded as
  `indexerRunning`; stop it if you want quiet disks, but a baseline recorded with it running is still a
  fair baseline for a run with it running.
- Tier B renders through WARP into an offscreen bitmap, parked offscreen. It is a regression signal for
  core and layout work, not what a user's GPU sees.
- Tier C skips the post-window side effects (index attach, change-log policy, update check) - the
  change-log push would wipe your real change log from a scratch data dir - so it measures "to the rows on
  screen", not "to idle". `firstListing` is only "nothing is loading", polled from after `windowShown`; the
  number a person sees is `listingPainted`.
- Medians, not means. A single slow sample is the machine.