# Benchmarking vk_gltf_renderer

> **For contributors and agents.** How to measure performance — headless timing and the scripted sequencer. The exact flags, log fields, and comparison rules are owned by the code (`src/benchmarking.cpp`, `utils/benchmark/`); this doc explains the workflow.

Two workflows:

1. **Headless timing (recommended)** — post-warmup render throughput for N frames at 1 or 5 samples per pixel; compare logs between builds.
2. **Scripted sequencer** — multi-step `.cfg` matrix with per-stage GPU profiler stats (optional, heavier).

## Headless timing (simple)

Render **500 frames** (or any count) with path tracing. Set **`--frames` and `--maxFrames` to the same value** so every app frame accumulates samples (default `maxFrames` is already 500).

### One-off CLI

```bash
./vk_gltf_renderer --headless --size 1920 1080 \
  --scenefile shader_ball.gltf \
  --hdrfile std_env.hdr \
  --frames 500 --maxFrames 500 \
  --ptSamples 1 --ptAdaptiveSampling 0 \
  --renderSystem 0 --envSystem 1
```

While rendering, periodic progress lines confirm the run is not stuck (every 50 frames or 5 seconds):

```text
HEADLESS_START frames=500 maxFrames=500 ptSamples=1
HEADLESS_PROGRESS app_frame 50/500 (10%) elapsed_ms=1234.5 ms_per_frame=24.69
...
HEADLESS_SUMMARY frames=500 maxFrames=500 ptSamples=1 effective_spp=500 measured_effective_spp=499 resolution=1920x1080 wall_ms=12320.987 ms_per_frame=24.691 total_wall_ms=12345.678 total_ms_per_frame=24.691 warmup_frames=1 measured_frames=499 throughput_MSps=84.0 spp_per_sec=40.49
BENCHMARK_JSON {"schema":1,"type":"headless_summary",...}
```

- **app_frame** — headless loop index (`--frames`).
- In headless mode, `main()` raises `--maxFrames` to at least `--frames` if you set it lower, so every app frame can accumulate samples during timing runs.
- **wall_ms** — measured post-warmup render time. The first completed frame is excluded so one-time setup such as shader specialization is not charged to throughput.
- **total_wall_ms** — full headless render-loop wall time, including warmup and any synchronous setup.
- **ms_per_frame** — `wall_ms / measured_frames`.
- **effective_spp** — total final-image accumulation: `min(frames, maxFrames) × ptSamples`.
- **measured_effective_spp** — accumulation covered by the measured post-warmup window.
- **throughput_MSps** — measured mega pixel-samples per second (`resolution × measured_effective_spp / wall_s / 10⁶`; higher is faster).
- **spp_per_sec** — measured `measured_effective_spp / wall_s` at this resolution (how fast quality accumulates; higher is faster).

Repeat with `--ptSamples 5` for 5 spp per frame (`effective_spp=2500`).

### Batch helper (1 spp and 5 spp)

```bash
python utils/benchmark/benchmark.py headless --scene resources/shader_ball.gltf --frames 500 --spp 1 5
```

Logs: `utils/benchmark/output/headless_<scene>_spp<N>.log`
CSV: `utils/benchmark/output/headless_results.csv`

Compare two builds (same scene/spp, different executable):

```bash
python utils/benchmark/benchmark.py headless-compare \
  utils/benchmark/output/headless_shader_ball_spp1_baseline.log \
  utils/benchmark/output/headless_shader_ball_spp1_candidate.log
```

(Use one log from build A and one from build B.)

---

## Scripted sequencer (optional)

Scripted benchmarks measure GPU frame time and VRAM usage across scenes, cameras, and renderer settings. Output is designed for regression testing when comparing builds or branches.

## Quick start (sequencer)

Build the sample, then from the project directory:

```bash
# Fast smoke benchmark (one scene)
python utils/benchmark/benchmark.py run quick.cfg --scene resources/shader_ball.gltf --hdr std_env.hdr

# Full matrix (edit utils/benchmark/scenes.example.txt for your asset paths)
python utils/benchmark/benchmark.py run matrix.cfg \
  --scenes-file utils/benchmark/scenes.example.txt \
  --scenes-root . \
  --csv-name benchmark_results.csv
```

Results land in `utils/benchmark/output/` (logs per scene + CSV). See also [utils/benchmark/README.md](../utils/benchmark/README.md).

## How it works

1. **`--benchmark 1`** turns off vsync, hides side panels (keeps a fullscreen viewport with the tonemapped image), and drains the scene load pipeline synchronously each frame. Scripted sequencer runs are **interactive-window** runs (minimal viewport) — they do *not* pass `--headless`; use the headless workflow above for windowless timing.
2. **`ElementSequencer`** steps through a `.cfg` script (`SEQUENCE "name"` blocks).
3. After each sequence, **`ProfilerManager`** logs `ParameterSequence` blocks at log level `eSTATS` (GPU/CPU timer averages).
4. **`benchmarkAdvance()`** records Scene and PathTracer/Rasterizer VRAM stats.
5. When the script finishes, the app closes automatically.

## Benchmark script format

Example — one sequence from `utils/benchmark/quick.cfg` (the file defines several: `Warmup`,
`PT 1spp`, `Rasterizer`; see `quick.cfg` and `matrix.cfg` for the full scripts):

```
SEQUENCE "PT 1spp"
--sequenceframes 256
--sequenceaverages 64
--sequenceresetframes 8
--renderSystem 0
--ptSamples 1
--maxFrames 1
--ptAdaptiveSampling 0
--gltfCamera 0
--updateData
```

| Token | Meaning |
|---|---|
| `SEQUENCE "..."` | Starts a new measured step |
| `--sequenceframes` | Frames to run this step |
| `--sequenceaverages` | Frames averaged for profiler report |
| `--sequenceresetframes` | Warmup frames after parameter changes (0 = measure immediately) |
| Other `--flags` | Any registered CLI parameter (renderer, path tracer, tonemapper, etc.) |

**Path tracer note:** Set `--maxFrames` to match `--ptSamples` when measuring convergence cost. Use `--maxFrames 1` with `--ptSamples 1` for per-frame interactive GPU time.

## Comparing versions

```bash
# Baseline build
python utils/benchmark/benchmark.py run matrix.cfg --scene my_scene.gltf --csv-name baseline.csv

# Candidate build (rebuild executable first)
python utils/benchmark/benchmark.py run matrix.cfg --scene my_scene.gltf --csv-name candidate.csv

python utils/benchmark/benchmark.py compare baseline.csv candidate.csv --output diff.csv --regression-threshold-pct 5
```

`compare` marks **Regression** when candidate GPU time is more than N% slower than baseline **or** when the candidate's Scene VRAM peak exceeds the baseline by more than the VRAM threshold (default 64 MB). Negative delta % means faster. See `compare_csv` in `utils/benchmark/benchmark_results.py` for the exact rules.

## Log parsing

`utils/benchmark/benchmark.py` reads stable `BENCHMARK_JSON` records first, with legacy text parsing as a fallback:

- `BENCHMARK_JSON {"schema":1,"type":"headless_summary",...}`
- `BENCHMARK_JSON {"schema":1,"type":"sequence_memory",...}`
- `ParameterSequence N "name" = { Timer "..."; GPU; avg ...; CPU; avg ...; }`
- `BENCHMARK_ADV N { Memory Scene; ... Memory PathTracer; ... }`

Auto-generated log: `log_<executable>.txt` next to the binary (Logger behavior).

## Tips

- Use fixed resolution (`--size 1920 1080`, set in `utils/benchmark/benchmark_runner.py`) for comparable numbers.
- Disable validation layers for performance runs (`--vvl` off by default in Release).
- Add scenes to `utils/benchmark/scenes.example.txt` (name + relative path per line).
- Multi-camera scenes: add sequences with `--gltfCamera 0`, `--gltfCamera 1`, etc.
- After large setting changes, use `--updateData` or `--resetFrame` (no value; bool triggers) and non-zero `--sequenceresetframes`.