---
name: vlm-benchmark
description: Benchmark an optimization (PR or branch) on a Device Farm device via the vlm-benchmark framework — baseline vs optimized, quality-regression-aware.
argument-hint: "<PR#|PR-url|branch> [--device s25|pixel9|s26|iphone17…] [--model qwen3.5-q8|gemma4-q4|<label>] [--preset full|base|smoke] [--runs 2] [--mmproj both|gpu]"
---

# VLM Benchmark — on-device optimization A/B

Benchmark an optimization change (a PR or branch, in the **addon** or the **qvac-fabric**) on a
specific **AWS Device Farm** device, using the `packages/llm-llamacpp/benchmarks/vlm-benchmark`
framework. It builds the change from source, runs a **baseline vs optimized** A/B on the device
(N runs each), pairs them by CPU fingerprint, and reports speed **and quality** — with
**quality regression as the primary gate**. This codifies the QVAC-21320 / QVAC-21297 methodology.

Scope: Qwen3.5-0.8B VLM, projector `mmproj` CPU-vs-GPU (`mmproj=both`) by default. The **model is
selectable** (`--model`) — the framework already ships multiple VLMs (Qwen3.5 f16/q8, Gemma-4-E2B)
and a new one can be added to `config.cjs`; Qwen3.5-q8 is just the default. Device Farm only (no local
ADB run yet). Thermal / cross-unit variance is handled by CPU-fingerprint matching over multiple runs,
not interleaving.

> **Source safety (read-only target).** This skill **never modifies the PR or branch under test.** It
> only *reads* the target — `gh pr view` / `gh pr diff` / `git diff` — to resolve its ref/commit and
> classify the change. **Every** change (harness/config edits and the optimization itself — the fabric
> overlay `REF` or cherry-picked addon commits) is made **only** on the skill's own `test/<slug>` /
> `test/<slug>-base` refs. It never commits to, pushes to, or otherwise alters the source PR's branch,
> the fabric repo, or `main`.

## Usage

```
/vlm-benchmark #2921
/vlm-benchmark https://github.com/tetherto/qvac/pull/2921
/vlm-benchmark https://github.com/tetherto/qvac-fabric-llm.cpp/pull/172 --model gemma4-q4 --device pixel9
/vlm-benchmark feat/QVAC-21297-opencl-vision-encoder --device s25 --preset full --runs 2
/vlm-benchmark feat/my-opt --mmproj both --preset base
```

## Arguments

| Arg | Required | Default | Meaning |
|-----|----------|---------|---------|
| target | yes | — | `#<n>` / PR URL → PR mode; otherwise a branch name |
| `--device` | no | `s25` | Device Farm phone as a `matrix_mobile` token: `s25` (S25 Ultra / Adreno), `pixel9` (Pixel 9 / Mali), `s26` (S26 Ultra), `iphone16` / `iphone17` / `iphone17pro`. Passed as `-f matrix_mobile=<token>` at dispatch (Step 8) — the workflow resolves it to a **single-device `device_model` filter**. A friendly name ('Pixel 9') maps to its token (`pixel9`). Append `-cpu`/`-gpu` to force the model backend; bare token = the harness picks per `--mmproj`. |
| `--model` | no | `qwen3.5-q8` | Which VLM to benchmark — a `config.cjs` catalog `label` (`qwen3.5-f16`, `qwen3.5-q8`, `gemma4-q4` = Gemma-4-E2B Q4) or a new one the skill adds (Step 6). A friendly value like `gemma4`/`gemma4-e2b` maps to the `gemma4-q4` catalog entry. Default = `MODEL_2` / current `mmprojModel` |
| `--preset` | no | `full` | `full` (5 tasks×5, sharpest — DEFAULT; but **overruns the mobile session for `mmproj=both`** → use only on desktop or `mmproj=gpu`, or raise `mobile_timeout_min` — see Step 8) · `base` (5 tasks×3 = the `cognitive` set with `matrix_samples=3`; **the mobile `mmproj=both` choice**) · `smoke` (textvqa×1, wiring check). Default to `base` on mobile `mmproj=both`. |
| `--runs` | no | `2` | runs per build (2 baseline + 2 optimized = a 4-run CPU-matched set) |
| `--mmproj` | no | `both` | `both` = projector CPU-vs-GPU per run · `gpu` = projector on GPU only |

## Prerequisites & paths

The commands below use placeholders — **resolve each once, up front, then substitute** (don't assume
hard-coded paths):

- **`gh` CLI** authenticated to `tetherto/qvac` (Actions dispatch + read); **`node`** on PATH (for `match-devices.js`).
- **`<qvac>`** — the local qvac **main clone** (`origin` = `tetherto/qvac`). Discover it, don't guess:
  run `git worktree list` from any qvac checkout → the **first** line's path is the main worktree. The
  session's current directory may be a **read-only** worktree — never build or push from it.
- **`<feature-branch>`** — the resolved target branch from Step 1 (the PR head branch, or the branch
  argument). The optimized ref is `test/<slug>`; the baseline ref is `test/<slug>-base`.
- **`<pr-repo>`** — the GitHub repo the PR lives in (resolved in Step 1): `tetherto/qvac-fabric-llm.cpp`
  for a fabric PR, `tetherto/qvac` for an addon PR. Used only to read the PR; Actions dispatch always
  targets `tetherto/qvac`.
- **`<slug>`** — the feature-branch basename (branch `feat/QVAC-21297-opencl-vision-encoder` → slug
  `QVAC-21297-opencl-vision-encoder`).
- **`<worktree>`** — the isolated worktree this run creates in Step 5: `<qvac>/../wt-<slug>` (a sibling
  of the clone — the repo's `wt-<name>` convention). All edits, build, and push happen here.
- **`<base>`** — the branch the worktree is cut from; chosen in Step 4 (`origin/main`, or the qvac
  feature branch related to the target).

## Workflow

### Step 1 — Resolve the target

- PR (`#<n>` or URL): resolve the **repo from the target** — a `qvac-fabric-llm.cpp` PR URL →
  `tetherto/qvac-fabric-llm.cpp` (a **fabric** PR); a `qvac` PR URL or a bare `#<n>` → `tetherto/qvac`
  (an **addon** PR). Then `gh pr view <n> --repo <pr-repo> --json headRefName,commits,files,title,body`
  → the head branch is the optimization. (`<pr-repo>` is only for reading the PR; all Actions dispatch
  in Steps 7–8 targets `tetherto/qvac`, where the workflow lives.)
- Branch: use it directly.
- Parse the flags off the tail (`--device` / `--model` / `--mmproj` / `--preset` / `--runs`); unset
  ones take their defaults, but `--device`/`--model`/`--mmproj` are reconciled against the PR in Step 3
  before anything is built.

### Step 2 — Detect the change scope (from the diff)

Run `gh pr diff <n> --repo <pr-repo>` (PR; `<pr-repo>` from Step 1) or `git diff main...<branch>`
(branch). Classify:

- **Addon change** — touches `packages/*/addon/**` or addon JS/config. The **optimization** is the
  addon commit(s); **baseline** is the pre-optimization state (Steps 5–6 build it as `<base>` +
  harness edits, addon change omitted; the optimized ref adds the addon commits on top).
- **Fabric change** — the optimization is a `qvac-fabric` (`tetherto/qvac-fabric-llm.cpp`) commit (a
  fabric PR/commit, or a branch that bumps the fabric). The A/B uses a **self-contained vcpkg
  overlay-port** whose `REF` is the only delta: **optimized** overlay `REF` = the fabric PR/branch head
  commit; **baseline** overlay `REF` = the pre-change fabric commit (the PR's merge-base with its base,
  or the published registry version's commit). Both refs use the overlay — Step 6 covers creating the
  port and computing its `SHA512` directly (no CI round-trip).
- **Mixed / dependency** — a PR that needs both (addon + a fabric overlay): apply both on the
  optimized build; the baseline strips **only** the optimization under test. Keep the optimization the
  **sole delta** between the two builds (isolation).

### Step 3 — Reconcile the passed options against the PR

Before building, cross-check `--device` / `--model` / `--mmproj` against what the change actually
targets, from the PR title/body/linked ticket + the diff-touched code:
- **Device / backend** — `ggml-opencl` / "OpenCL" / "Adreno" → an Adreno device (token `s25`);
  `ggml-vulkan` / "Vulkan" / "Mali" → a Mali device (token `pixel9`); "Metal" → iOS/desktop. Plus any
  explicit device/SoC mention in the PR (`S25`, `Pixel 9`, `Adreno 830`, `Snapdragon`) → its token.
- **Model** — a Qwen / Gemma / other model or projector mention.
- **mmproj axis** — whether it's a projector (`mmproj`) change vs an LLM change.

Then, per axis:
- **(a)** user did **not** pass it and the PR implies a value → **adopt the PR's value and notify**.
- **(b)** user **passed** it and it **conflicts** with the PR (e.g. `--device pixel9` (Mali/Vulkan)
  on an Adreno/OpenCL-only fix — the change wouldn't even be exercised) → **AskUserQuestion**: proceed
  as passed / switch to the PR's value / cancel.
- **(c)** no PR signal → keep the passed value or the default.

Branch-mode targets with no PR: best-effort from the branch name / diff, else proceed with passed/defaults.

### Step 4 — Choose `<base>`

- `git -C <qvac> fetch origin`
- **Default: `<base>` = `origin/main`.** The optimization is layered as the sole delta (fabric overlay
  `REF` / cherry-picked addon commits per Step 2), so the baseline stays clean regardless of base — this
  is the case for fabric PRs (target in `tetherto/qvac-fabric-llm.cpp`) and standard addon PRs alike.
- **Exception — an existing qvac feature branch the target builds on:** a branch-mode target that names a
  qvac branch, or an optimization explicitly stacked on another unmerged qvac feature branch → set
  `<base>` = that branch so the A/B builds on the same foundation.

The `mmproj=both` benchmark harness may not live on `<base>` yet — until it merges to `main` it lives on
**`test/QVAC-21257-mmproj-gpu-benchmark-changes`** (the harness + addon instrumentation grafted cleanly
onto `main`). Step 5 brings it in once the worktree exists. (Do **not** use the older
`feat/QVAC-21257-mmproj-gpu-config` — it is ~100 commits behind `main` and does not merge cleanly.)

### Step 5 — Create the isolated worktree + bring in the harness

Create the worktree **from `<qvac>`** (never the read-only session checkout or other in-use worktrees):
```
git -C <qvac> worktree add <worktree> -b test/<slug> <base>
```
(drop `-b` if `test/<slug>` already exists.) All later edits, build, and push happen inside `<worktree>`.

**Bring in the `mmproj=both` harness.** Check whether it is already present:
`git -C <worktree> show HEAD:packages/llm-llamacpp/benchmarks/vlm-benchmark/config.cjs` contains
`mmprojGpu`/`mmprojModel` **and** `benchmarks/vlm-benchmark/match-devices.js` exists.
- **Present** (`<base>` already carries it — e.g. once the harness has merged to `main`) → nothing to do.
- **Absent** → merge the grafted harness branch in — it sits directly on `main`, so this is a clean
  merge (no conflicts):
  `git -C <worktree> merge --no-edit origin/test/QVAC-21257-mmproj-gpu-benchmark-changes`
  It brings the `mmproj=both` axis (`mmprojGpu`/`mmprojModel`, `legsFor`, per-row `vision_ms`), the
  **addon `mmproj-use-gpu` key + `VisionEncodeMs` instrumentation** (which `vision_ms` depends on),
  `match-devices.js`, and the `matrix_mmproj_gpu` input + conditional targeted-android prebuild in
  `benchmark-vlm-model-comparison.yml`. (Never merge the old `feat/QVAC-21257-mmproj-gpu-config`: it
  trails `main` by ~100 commits, its addon C++ conflicts with `main`'s `ContextSlideOutcome` refactor,
  and its `matrix-s25`/`android_flagship_models` wiring predates `main`'s `matrix_mobile`/`device_model`.)

**Two refs, one delta.** The A/B is two refs that share **identical** harness config and differ
**only** by the optimization. Build them in this order (Step 6 gives the exact edits):
1. **baseline** = `test/<slug>-base` = `<base>` (+ harness) + the harness/config edits, **no optimization**.
2. **optimized** = `test/<slug>` = the baseline **plus the optimization only** — fabric: point the
   self-contained `packages/llm-llamacpp/vcpkg/ports/qvac-fabric` overlay-port `REF` at the fabric commit with a computed
   `SHA512` (Step 6); addon: cherry-pick / merge the PR/branch's addon commits on top.

(The QVAC-21297 run named these `…-s25-stock` / `…-s25`.)

### Step 6 — Adjust the harness (notify the user, no approval)

Inside the worktree, make the edits and **notify the user of each** (file + what changed) — **no
approval needed**, since it's an isolated branch/worktree:
- **Optimization** — apply the optimization per Step 2 (the sole delta between baseline and optimized):
  - **Fabric** — use a **self-contained overlay-port** at `packages/llm-llamacpp/vcpkg/ports/qvac-fabric/`.
    Copy `portfile.cmake` + `vcpkg.json` **+ `android-vulkan-version.cmake`** from the registry port
    `tetherto/qvac-registry-vcpkg:ports/qvac-fabric/` into that dir — for a **Mali / Android-Vulkan**
    target the port MUST carry `android-vulkan-version.cmake` + its Vulkan-C++-header provisioning (the
    port *owns* the Android `vulkan.hpp`, so the build works even on a fabric commit that predates a
    fabric-side Android Vulkan build fix — this is why the baseline builds cleanly). Add
    `"overlay-ports": ["vcpkg/ports"]` to `packages/llm-llamacpp/vcpkg-configuration.json` — `main` has
    **no** `overlay-ports` key, so add it as the first field (the path is relative to that file's dir →
    `packages/llm-llamacpp/vcpkg/ports`). In the portfile, change the registry's `REF v${VERSION}` to the
    fabric commit and set `SHA512` to the hash **computed directly** (no CI round-trip; see
    `SKILL-fabric-release.md` Phase A). The fabric repo is **private**, so the archive fetch needs a
    token. Use `curl` on the `/archive/<ref>.tar.gz` URL — **not** `gh api …/tarball/<ref>`: the
    `/tarball/` endpoint returns a differently-named top-level dir (`owner-repo-shortsha` vs
    `repo-fullsha`) → a **different `SHA512`** that won't match what vcpkg fetches (and `gh api` can't
    write to a file without a redirect anyway). Commit-SHA `/archive/` tarballs are byte-stable, so the
    computed hash equals vcpkg's fetch (validate once against a `SHA512` already pinned in a known port):
    ```
    curl -fsSL -H "Authorization: Bearer $GH_TOKEN" https://github.com/tetherto/qvac-fabric-llm.cpp/archive/<ref>.tar.gz -o /tmp/f.tgz
    openssl dgst -sha512 /tmp/f.tgz     # or: vcpkg hash /tmp/f.tgz
    ```
    ```cmake
    vcpkg_from_github(REPO tetherto/qvac-fabric-llm.cpp REF <fabric-commit> SHA512 <computed-sha512>)
    ```
    The baseline ref pins the pre-change fabric commit; the optimized ref pins the PR/branch commit —
    the overlay `REF`+`SHA512` is the only difference between them. (Fallback: if the computed hash is
    ever wrong, vcpkg prints the expected hash on the first failed fetch.)
  - **Addon** — cherry-pick / merge the PR/branch's addon commits onto the optimized ref only.
- **Model** (`--model`, as reconciled in Step 3) — where it's set depends on the mmproj axis:
  - `--mmproj both` (mmproj-compare) → the run uses `config.mmprojModel`, read directly by the compare
    path (it does **not** honor `matrix_models`). Set `mmprojModel` in `config.cjs` on the `test/` ref
    (e.g. `mmprojModel: GEMMA4_Q4` for `gemma4-q4` / Gemma-4-E2B). If the model isn't defined yet, add a
    `{label, name, ctx_size, llm: hf(...), mmproj: hf(...)}` literal (match the blob `repo`/`sha`/`file`
    to the registry/HF source) and point `mmprojModel` at it.
  - `--mmproj gpu` (two-models) → pass the model(s) via `-f matrix_models=…` at dispatch (forwarded to
    the phone as `QVAC_VLM_MODELS`); no branch edit.
- **Device / preset / samples / mode are DISPATCH INPUTS — not harness edits.** Main's benchmark forwards
  them to the phone via the `matrix-mobile` job's `device_env` channel (`QVAC_VLM_PRESET` ← `matrix_preset`,
  `QVAC_VLM_SAMPLES` ← `matrix_samples`, `QVAC_VLM_MODE` ← `matrix_mode`, and `device_model`/
  `device_manufacturer` from the `matrix_mobile` token). Pass them as `-f` flags in Step 8 — **do not edit
  the branch for these.** The `--device` token fixes the GPU backend (`s25`→Adreno/OpenCL,
  `pixel9`→Mali/Vulkan) → which drives the Step 9 routing-proof grep and which opts are even relevant.
- **mmproj** (`--mmproj`) — the **one** run knob that IS a test-branch config edit for mobile: set
  `mmprojGpu: 'both'|'gpu'` in `config.cjs` (mobile reads `config.mmprojGpu`; `QVAC_VLM_MMPROJ_GPU` is
  desktop-only and not in `device_env`). `both` also reads `config.mmprojModel` (see the Model bullet).

Commit these harness edits on `test/<slug>` — this commit is the **baseline** state. Branch the baseline
ref at it: `git -C <worktree> branch test/<slug>-base`. Then apply the **optimization** on `test/<slug>`
(fabric: set the overlay-port `REF`+computed `SHA512` to the optimized fabric commit; addon: cherry-pick
/ merge the PR's addon commits) and commit. The two refs now differ by **exactly** the optimization —
nothing else.

### Step 7 — Push the two refs (the one approval gate)

Notify the user of the exact pushes and **ask for approval** — noting they target **separate
`test/<slug>` refs** (never `main`), so it's safe. Push **both** the optimized and baseline
refs from Step 5:
```
git -C <worktree> push -u origin test/<slug>
git -C <worktree> push -u origin test/<slug>-base
```
Push only on approval.

### Step 8 — Run baseline + optimized on Device Farm (`--runs` each)

Dispatch **each ref `--runs` times** — the optimized ref and the baseline ref from Step 5. `<device>`
is the `--device` token (default `s25`); `matrix_desktop=none` forces **zero** desktop cells → the run
is mobile-only, which triggers the targeted android-only prebuild (`prebuild-candidate-mobile`, built
**with the Vulkan SDK** for Mali) and skips the heavy full-desktop prebuild + desktop legs. ⚠️ Use the
literal `none` (any non-empty unknown token works), **not** an empty string — the workflow defaults an
empty/unset `matrix_desktop` to `linux-cpu` (`${MATRIX_DESKTOP:-linux-cpu}`), which would run the full
desktop prebuild and **skip** the targeted mobile build. `matrix_sources=addon@candidate` builds the
addon **from the dispatched ref** so the ref's fabric overlay / addon commits actually take effect:
```
# optimized — dispatch --runs times
gh workflow run benchmark-vlm-model-comparison.yml --repo tetherto/qvac \
  --ref test/<slug> \
  -f matrix_sources=addon@candidate -f matrix_desktop=none \
  -f matrix_mobile=<device> -f matrix_preset=<preset> -f matrix_mode=two-models
# baseline — dispatch --runs times (identical inputs; only --ref differs)
gh workflow run benchmark-vlm-model-comparison.yml --repo tetherto/qvac \
  --ref test/<slug>-base \
  -f matrix_sources=addon@candidate -f matrix_desktop=none \
  -f matrix_mobile=<device> -f matrix_preset=<preset> -f matrix_mode=two-models
```
**`--preset` → dispatch inputs.** `matrix_preset` names a `config.cjs` preset (`smoke`, `cognitive`,
`full`); there is **no `base` preset**, so map the arg:
- `full` → `-f matrix_preset=full`
- `base` → `-f matrix_preset=cognitive -f matrix_samples=3` (the 5 VQA tasks × 3 = 15/cell, no OCR)
- `smoke` → `-f matrix_preset=smoke`

⚠️ **Mobile session budget (critical for `mmproj=both`).** On mobile, `mmproj=both` runs **two** cells
(CPU + GPU) in one Device-Farm session → ~2× the inferences. **`full` × `mmproj=both` overruns the
~30-min per-test ceiling**: the CPU cell finishes but the GPU cell is **cut off mid-run** (`matrix-mobile`
shows `failure` while `matrix-combine` still succeeds on partial data — the GPU cell's quality then covers
only the first 1–2 tasks and is **not comparable**). So for a mobile `mmproj=both` verdict **default to
`base`**, and/or raise the ceiling with **`-f mobile_timeout_min=<min>`** (e.g. `60`; capped by the 120-min
DF/GH job ceiling). `full` on mobile is safe only for `mmproj=gpu` (one cell) or with a raised timeout.

Default `--runs 2` → **4 dispatches total** (`--runs` = dispatches per ref, for CPU-fingerprint
matching across units — not samples/task, which the preset governs). Device / preset / mode reach the
phone via the `matrix-mobile` job's `device_env` (Step 6) — pass them as `-f` inputs, identical for both
refs; only `mmprojGpu` (+ `mmprojModel` for `--mmproj both`) is baked into each ref's `config.cjs`.
Monitor each run to completion (poll `gh run view <id> --repo tetherto/qvac --json status,conclusion,jobs`):
the candidate prebuild (`prebuild-candidate-mobile` on a mobile-only run) is the early build signal;
`matrix-mobile` is the on-device leg (skips if the prebuild fails). **Record which run ID is baseline vs
optimized** — needed in Step 10.

### Step 9 — Retrieve metrics

**The raw `[VLMROW]`/`[VLMSEG]` markers a mobile run needs are in the `console-logs-*-Android`
artifact — NOT in `gh run view --log`** (they print to the on-device Android logcat, which Device Farm
captures as an artifact). For each run, download it and keep the logcat as that run's marker file for
Step 10:
```
gh run download <run-id> --repo tetherto/qvac --name console-logs-<pkg>-<device>-Android --dir <run-dir>
# → <run-dir>/Android/Android_logcat_full.txt holds the [VLMROW]/[VLMSEG] markers
```
Use a distinct `<run-dir>` per build+index (e.g. `opt1/`, `base1/`) so Step 10 can pair them, matching
the run IDs recorded in Step 8. (On **desktop** legs the markers are on native stderr, so there the
`gh run view --log` run log carries them; mobile does not.)

For the ready-made summary, read the **`vlm-matrix-consolidated-<run#>`** artifact (~2 KB markdown — full
quality + speed + samples tables) or the identical `matrix-combine` job-log render:
- **Quality:** per-task `textvqa / vizwiz / gqa / docvqa / ai2d` + **Overall %** (equal-weight mean).
- **Speed:** `mmproj-encode` (from the addon `vision_ms` RuntimeStat), TTFT, wall; small/large
  resolution buckets; samples run / passed / failed.
- Confirm the projector routed to the intended backend from logcat — Adreno OpenCL:
  `Adreno GPU version 830 found keeping OpenCL backend` → `using device GPUOpenCL`; Mali Vulkan:
  `found device description: Mali-G715` → `removing OpenCL backend … rely on Vulkan/cpu only` →
  `using device Vulkan0 (Mali-G715)`.

### Step 10 — Pair + detect quality regression (primary gate)

- Pair baseline↔optimized runs with `benchmarks/vlm-benchmark/match-devices.js` (build-invariant CPU
  `mmproj-encode` fingerprint → same-unit-equivalent), passing the per-run logs saved in Step 9
  (`--shipping` = the baseline runs, `--optimized` = the optimized runs):
  `node benchmarks/vlm-benchmark/match-devices.js --shipping base1/Android/Android_logcat_full.txt [base2/…] --optimized opt1/Android/Android_logcat_full.txt [opt2/…]` (the per-run logcat files downloaded in Step 9).
- **Quality Δ = (optimized − baseline) Overall %**, plus **per-task Δ**. **Flag ANY negative Δ**
  (overall or any task) as a regression — this is the gate.
- With `--mmproj both`, also report the **within-run GPU-vs-CPU** quality Δ per build (the
  projector-backend regression — the QVAC-21297 case: stock GPU −4.1 pp → fixed GPU Δ0).
- `full` (5 tasks×5) gives the sharpest quality signal on **desktop / `mmproj=gpu`**. `base` (3 samples)
  is coarser and can noise-mask small regressions (QVAC-21320 saw the same Adreno bug read as base −4 pp
  vs full −18.8 pp) — **but on mobile `mmproj=both` it is the verdict preset**: `full` overruns the
  Device-Farm per-test window and truncates the GPU cell (Step 8), so a completed `base` beats a
  truncated `full`. Raise `mobile_timeout_min` if you must run `full` on mobile. `smoke` (1 sample) is a
  **wiring check only** — it cannot produce a quality verdict, so on smoke, skip the regression gate +
  the full Step 11 report and just confirm routing / `GPU==CPU` / 0 device-loss.

### Step 11 — Report → write `QVAC-<ticket>-benchmark-results.md`

Write to `QVAC-<ticket>-benchmark-results.md` in the working dir where the prior docs live (repo root,
alongside `QVAC-21320-benchmark-results.md` / `QVAC-21297-benchmark-results.md`). Derive `<ticket>` as
the `QVAC-\d+` match from the PR title (`gh pr view <n> --repo <pr-repo> --json title`) or the branch
name; if there is no ticket, fall back to `<slug>-benchmark-results.md` and say so in the doc.

**Never discard prior results** — layer the newest on top (the shape `QVAC-21320-benchmark-results.md`
already has: newest first, older sections below):
- **Doc absent** → create it in full with the **Write tool**, section-for-section per the template below.
- **Doc present** → **do not overwrite.** Read it, then insert THIS run at the top of the results (below
  the doc's title + goal/context paragraph, above the prior run) in order:
  1. **Summary of the latest result** — the `> **Result:** …` blockquote + the `# Latest results …`
     section (method one-liner, quality gate, CPU-matched table, recommendation) for this run.
  2. **Latest result details** — the `# Detailed results` block for this run.
  3. **Results before that** — the previously-latest content, demoted beneath: rename the prior
     `# Latest results …` heading to a dated/labelled `# Previous results — <date/axis>` and keep it and
     all older sections **intact** below (preserve verbatim; only demote headings / reorder).

Either way the skill does **not** commit/push the doc (see Notes).

Match the two reference docs **section-for-section, in this order** (copy their exact column/wording):

1. `# QVAC-<ticket>: <one-line descriptor> (<device> / <SoC + backend>)`
2. Goal / context paragraph (_italic_) — what the optimization is, the ticket goal, related-ticket links.
3. `> **Result:** …` blockquote — one-paragraph headline verdict (what the A/B proved, **quality first**).
4. `---`
5. **Headline section** `# Latest results — <compared axis> (<device>)`:
   - Method in one line (_italic_): N runs, Device Farm pool, baseline-vs-optimized definitions, model,
     `mmproj` mode, preset (tasks×samples, `n=`/cell), matched-by CPU fingerprint, link to [CI runs](#ci-runs).
   - **Quality** paragraph (the primary gate) — Overall Δ + notable per-task shifts.
   - **Speed** paragraph + the **CPU-matched comparison table**:
     `| Metric | Baseline-CPU | Baseline-GPU | Optimized-CPU | Optimized-GPU | <opt vs base> | <opt vs shipping-CPU> |`
     rows `mmproj-encode (ms)` · `TTFT (ms)` · `wall (ms)` · `quality (Overall %)`; then a short
     column-meaning gloss.
   - **Primary robust metric** paragraph — the within-run CPU/GPU encode ratio (device variance cancels).
   - `## Recommendation` — numbered, actionable verdict.
6. `---`
7. `# Detailed results`:
   - `## Quality — lmms-eval overall %` — per-task table
     `| Build | Config | textvqa | vizwiz | gqa | docvqa | ai2d | Overall % |` + a Δ-callout line
     (**flag ANY negative per-task or overall Δ**; note base-preset granularity if the preset is coarse).
   - `## Per-run summary` — `| Run | Build | CPU enc | GPU enc | GPU TTFT | within-run CPU/GPU | quality GPU vs CPU |`,
     one row per run, run-ID linked.
   - `## Resolution dependence` — small vs large `mmproj-encode` bucket table.
   - `## Caveats` — decode-TPS artifact, peak-RSS not captured, CPU-fingerprint match %.
   - `## Setup & isolation` — model/blobs, device + logcat backend-routing proof, baseline/optimized
     build definitions (overlay `REF` / addon commit), the isolation statement (optimization = sole delta).
   - `## CI runs` — build-validation, smoke, and the N-run matched set, all run-ID linked.
   - `## Investigation` (optional) — headroom / profiler / accuracy-guard notes when relevant.
   - `## Verdict` — closing paragraph.

`QVAC-21320-benchmark-results.md` and `QVAC-21297-benchmark-results.md` are the worked templates —
copy their exact column headers and phrasing.

## Notes

- **Device Farm only** — no local ADB run yet. Thermal / cross-unit variance is handled by
  CPU-fingerprint matching + N runs, not interleaving.
- **Isolation** — the optimization under test must be the ONLY delta between baseline and optimized
  builds. Don't drag in unrelated fabric/addon changes (e.g. keep OpenCL work free of Vulkan opts).
- **Worktree** — created from `<qvac>` (the main clone) at `<worktree>` = `<qvac>/../wt-<slug>` (the
  repo's `wt-<name>` convention; see Prerequisites). Never edit in the read-only session checkout or
  other in-use worktrees.
- **Model support** — `config.cjs` already ships multiple VLMs (Qwen3.5 f16/q8, Gemma-4-E2B); select
  via `--model` → `mmprojModel` (mmproj-compare) / `models` (two-models), and add a new one via `hf()`
  blob literals. No framework code change is needed to benchmark a different model.
- **Approval policy** — harness edits happen in the isolated worktree and are **notify-only**; the
  **only** approval gate is the **push** to `origin` (Step 7), and even that is a separate
  `test/<slug>`, never `main`.
- **Harness source** — until the mmproj benchmark tooling merges to `main`, it lives on
  `test/QVAC-21257-mmproj-gpu-benchmark-changes` (grafted cleanly onto `main`), merged in by Step 5.
  The older `feat/QVAC-21257-mmproj-gpu-config` is NOT usable — it trails `main` by ~100 commits and its
  addon C++ conflicts with `main`'s `ContextSlideOutcome` refactor (the removed `FullWipe`). **End state:**
  once the grafted branch lands on `main`, drop the Step 5 merge and base directly on `origin/main`.
- **Device targeting** — a `-f matrix_mobile=<token>` dispatch input (`s25`/`pixel9`/`s26`/`iphone*`),
  **not** a branch edit. The `context` job resolves each token to `device_model`+`device_manufacturer`
  and the `matrix-mobile` job runs it as a single-device Device Farm filter (one phone per token).
- **What reaches the phone** — the `matrix-mobile` job forwards dispatch inputs to the device via its
  `device_env` (`QVAC_VLM_PRESET` ← `matrix_preset`, `QVAC_VLM_MODE` ← `matrix_mode`, `QVAC_VLM_SAMPLES`
  ← `matrix_samples`, `QVAC_VLM_MODELS` ← `matrix_models`, plus `device_model`). So preset/mode/samples/
  model are `-f` inputs, **not** `config.cjs` edits. The exception is `mmprojGpu` (`QVAC_VLM_MMPROJ_GPU`
  is desktop-only, absent from `device_env`) → set it in `config.cjs` on the `test/` ref.
- **Per-op OpenCL profiling** (`GGML_OPENCL_PROFILING` / `CLPROF`) is out of scope: its per-kernel
  logcat routing is blocked under `GGML_BACKEND_DL` (the backend module's stderr is not captured).
- **Assets it drives:** `benchmarks/vlm-benchmark/{config.cjs,harness.cjs,stage.cjs,aggregate.js,match-devices.js}`,
  `.github/workflows/{benchmark-vlm-model-comparison,reusable-prebuild-targeted,integration-mobile-test-llm-llamacpp}.yml`.
  Worked examples of the output format: `QVAC-21320-benchmark-results.md`, `QVAC-21297-benchmark-results.md`.
- **Writes** the results doc `QVAC-<ticket>-benchmark-results.md` to disk (Step 11) — **creating it if
  absent, otherwise prepending the new run as *Latest* and demoting the prior run to history (never
  discarding prior results)** — but does NOT commit/push it or open a PR; the caller handles version control.