# Skill Recorder — Evals

Repeatable evals for the part of the system with real variance: the multi-turn
**Copilot describer** that turns captured signals into an *overall intent* + an
*ordered list of steps*. Each eval feeds the describer a fixed, synthetic
recording and scores its analysis against a rubric.

## Why fixture-based (not live capture)

The evals are **deterministic and video-less on purpose**. Live capture (driving
real apps, recording the screen) is flaky and slow, and it's not the part we're
trying to measure. By materializing a fixed event stream we isolate the describer
so a run is repeatable and fast (~15–25s per scenario), and a failure points at
the model/instructions, not at capture flakiness. The events are authored to
mirror what the real collectors emit for the same task, so a scenario is a
faithful stand-in for a real recording. Matching **mock pages** live in
`evals/mocks/` for when you *do* want a real end-to-end capture (see below).

## Run

```bash
npm run eval                       # all scenarios
npm run eval -- --only=web-to-spreadsheet
npm run eval -- --judge            # also run the semantic LLM judge
npm run eval -- --keep             # print the temp sessions dir (artifacts kept)
npm run eval -- --model=<model-id> # override the describer model
```

Requires GitHub Copilot CLI to be signed in (same auth the app uses). Exit code
is non-zero if any scenario fails. Full results are written to
`evals/results/<timestamp>.json` (git-ignored).

Under the hood the runner uses Node's TypeScript support
(`--experimental-transform-types`) plus a tiny resolution hook
(`evals/register.mjs` → `evals/hooks.mjs`) that (1) resolves the project's
extensionless imports to `.ts` and (2) swaps the one `electron` import for a
headless stub (`evals/electron-stub.mjs`). No bundler, runs the real app source.

## How a run works

For each scenario the harness:

1. **Materializes** a synthetic session (`session.json` + `events.jsonl`) into an
   isolated temp sessions root (via the `SKILL_RECORDER_SESSIONS_DIR` override, so
   your real sessions are never touched).
2. Runs the **real pipeline** — `processSession()` builds `bundle.json` +
   `description.md` exactly as the app does after Stop.
3. Runs the **real describer** — `new Describer().analyze(id)`, the same agent the
   app uses (reads the timeline/events, pulls frames only if ambiguous, calls
   `submit_analysis`).
4. **Scores** the analysis against the scenario's rubric.

## Scoring

`scoring.ts` is deterministic and LLM-free — the primary pass/fail signal:

- **intent keywords** — the intent sentence names the right subject.
- **step count** — within an expected range (catches over/under-segmentation).
- **expected apps** — the right applications appear.
- **ordered actions** — key actions appear as an ordered subsequence across steps
  (validates the reconstructed order, e.g. *open page → copy → into spreadsheet*).
- **must-mention** — specific copied values/entities are surfaced.
- **forbidden noise** — recorder bracketing (the Skill Recorder app), permission
  dialogs, and tracking-param hops must **not** appear as steps. Scoped to step
  titles/apps + intent, so the agent isn't penalized for *explaining* that it
  correctly ignored noise.

A forbidden-noise hit fails the scenario outright; otherwise pass = ≥80% of checks.

`--judge` adds an optional second opinion: a separate Copilot agent grades
faithfulness 0–5 against the scenario's ground truth (`judge.ts`). Off by default
to keep runs deterministic.

## Scenarios

Business, repeatable knowledge-work patterns (`evals/scenarios/`):

| id | task |
|----|------|
| `web-to-spreadsheet` | Copy pricing figures from a web page into a spreadsheet |
| `invoice-extract` | Extract invoice rows from a web table into a spreadsheet |
| `research-compile` | Research two articles and compile quotes into a note |
| `directory-lookup` | Collect contact details from a directory into a spreadsheet |
| `irrelevant-detour` | Research habit articles with a mid-task off-task recipe detour the intent rules out (Chrome + TextEdit) |
| `expense-report` | Reconcile card charges against receipts and file an expense report (Chrome + Preview + Expensify) |
| `release-notes` | Compile release notes from merged PRs, then version + deploy (Terminal + GitHub + editor) |
| `lead-to-crm` | Qualify inbound leads and enter them into the CRM (Mail + LinkedIn + Salesforce) |
| `windows-deploy` | Deploy a web app to Azure and log the live URL, on Windows (Edge + Windows Terminal/pwsh + Excel) |

The last three are longer, multi-app **business processes** — they loop over several
records, mix a native app with the browser and/or terminal, and end in a submit /
deploy / commit step — stress-testing segmentation and app attribution beyond the
simple copy→paste flows.

`irrelevant-detour` guards a different judgment: a **confident intent must exclude
off-task activity**. Its stream is a clean habit-research flow with a brief hop to a
cooking-recipe page (a different host, so it segments into its own step) that has no
copy and no follow-up. Because the overall intent is unambiguous, the describer must
recognize the recipe detour as irrelevant and drop it — the rubric fails outright if
`recipe`/`allrecipes`/`cookie`/`chocolate` surfaces as a step title/app or in the intent.

Each also exercises the describer's judgment: **pastes are inferred** (a paste
emits no event), **recorder start/stop bracketing is dropped**, **tracking
params are merged**, and **off-task detours the intent rules out are excluded**.

## Add a scenario

Create `evals/scenarios/<id>.ts` exporting a `Scenario`, and add it to
`evals/scenarios/index.ts`. Build the event stream with the helpers in
`scenario.ts` (`recorder`, `visit`, `appActivate`, `clipboard`, `terminal`,
`marker`), and describe a good result in `rubric`. Keep `truth` accurate — it's
what the `--judge` grades against.

```ts
export const myScenario: Scenario = {
  id: "my-task",
  title: "…",
  truth: "What the user actually did, in plain language.",
  build: () => [ recorder(0), ...visit(1500, "Google Chrome", url, title), clipboard(4000, "…"), recorder(8000) ],
  rubric: { intentKeywordsAny: [["…"]], expectedApps: ["chrome"], orderedActions: [["…"]], forbidden: ["skill recorder"] },
};
```

## Builder evals (`evals/builder/`)

A second, smaller harness that guards the **final stage** — the builder that
generalizes an approved analysis into a Scout artifact — rather than the
describer. It exists because of a real regression: when generalizing GitHub work,
the builder preferred driving the **browser (Playwright)** instead of the **`gh`
CLI**, even though Scout runs on the user's own Mac/Windows device where `gh` is
installed and authenticated.

```bash
npm run eval:builder                       # all builder scenarios
npm run eval:builder -- --only=github-issue-triage
npm run eval:builder -- --keep             # print the temp sessions dir
npm run eval:builder -- --model=<model-id> # override the builder model
```

**How it isolates the builder.** Each scenario seeds a **fixed, approved
`Analysis`** (plus a minimal valid `bundle.json`) into a temp sessions dir, then
runs the real `AutomationBuilder.build()` for a chosen `architecture` and
`platform` (macOS or Windows). Seeding a frozen analysis removes describer
variance, so a failure points squarely at the builder's instructions/catalogue.
Only the plan's **steps** (`label` + `prompt`) are scored — the summary and
generalization prose are intentionally excluded, so the builder isn't penalized
for *explaining* which tool it avoided.

**Rubric** (`score.ts`): a scenario passes only if the steps satisfy every
`mustUseAny` group (each group is a set of synonyms; at least one must appear) and
contain **none** of the `forbidden` tokens — all case-insensitive substring
matches over the step `label` + `prompt` text. A forbidden hit fails the scenario
outright.

**Coverage.** Ten scenarios (`scenarios.ts` + `native-tool-scenarios.ts`), spanning
macOS and Windows. Two guard the original **gh-vs-browser** regression directly
(GitHub issue triage · darwin, stale-PR nudge · win32); the other eight mirror the
describer eval set (`evals/scenarios/*`) so the generalization stage is guarded for
every task type. Each rubric encodes the right native capability for its task, in
one of two flavours:

- **Native-tool-wins, browser forbidden** — the task maps to an unambiguous
  first-class CLI/tool, so the browser is a genuine wrong answer. `release-notes`
  and the two GitHub scenarios require `gh` (merged PRs, issues, PRs) and forbid
  `browser_`/`playwright`; `windows-deploy` requires the `az` CLI (+ the `xlsx`
  skill for the log) and forbids the browser. These are the strong "prefer the
  device CLI over the UI" guards.
- **Assert the native path, don't forbid a legitimate browser** — for web-read
  tasks (`web-to-spreadsheet`, `invoice-extract`, `research-compile`) the rubric
  requires `web_fetch` (and the `xlsx`/`docx` skill for the output) but does **not**
  forbid the browser: preferring `web_fetch` while documenting a browser fallback
  for a page that may need a login is exactly what we want, and a pure-browser
  regression is still caught because `web_fetch` would be absent. Genuinely
  browser-driven tasks (`expense-report` → Amex/Expensify, `lead-to-crm` →
  Salesforce/LinkedIn have no CLI/API) don't forbid the browser at all; instead
  they pin the one sub-step that *is* native — reading the local PDF receipts
  (`view`/pdf), and reading the mailbox via `workiq_*` rather than the Mail UI.

This is the suite that drove the catalogue fix in
`electron/skillbuilder/scout-catalog.ts` (prefer first-class device CLIs — above
all `gh` — over the browser, platform-aware for zsh/bash vs PowerShell). When you
add a describer scenario, add the matching builder scenario so the pair stays in
lockstep.

## Skill builder evals (`evals/skillbuilder/`)

A sibling of the automation builder harness that guards the **`SkillBuilder`** —
the stage that turns an approved analysis into a reusable `SKILL.md` plan. Where
the automation harness scores free-text step prompts for native-tool choice, this
one scores the richer **plan structure** the builder now proposes.

```bash
npm run eval:skill                       # all skill scenarios
npm run eval:skill -- --only=price-tracker-skill
npm run eval:skill -- --keep             # print the temp sessions dir
npm run eval:skill -- --model=<model-id> # override the builder model
```

Both builder harnesses share the same seeding (`evals/lib/seed.ts`): a fixed,
approved `Analysis` + a minimal `bundle.json` per scenario, so a failure points at
the builder, not the describer.

**Rubric** (`score.ts`) — beyond the `mustUseAny` / `forbidden` native-tool checks,
each scenario asserts the shape of the proposed `SkillPlan`:

- **fixed values** — `minValues` requires the plan to declare at least that many fixed
  `values`, and the scorer additionally checks that every `{{token}}` a step references
  resolves to a declared value (no stale/unknown tokens survive to the artifact).
- **typed steps** — `minCalculations` / `minActions` require the procedure to be
  split into `calculation` (no side effect) and `action` (changes the world) steps.

**Coverage.** Five scenarios. Two target **Scout**: `price-tracker-skill` (a canonical
page URL → a fixed **value** referenced as `{{…}}`, `web_fetch` + the `xlsx` skill,
calculations then an append) and `github-issue-triage-skill` (the gh-vs-browser case as a
skill —
must use `gh`, forbid the browser, and drive the mutating comment/label actions). Three
target **Cowork** (Microsoft 365 Copilot), whose catalogue has **no browser automation**
— each asserts the right M365 `server/Tool` is reached for while playwright/`click` and
the web hosts are forbidden: `cowork-teams-digest` (read a channel then post via
`m365_teams`), `cowork-outlook-reply` (triage the mailbox then reply via `outlook`), and
`cowork-calendar-schedule` (find a slot then book via `outlook_calendar`).

## Mock pages (`evals/mocks/`)

Static, self-contained HTML fixtures matching the scenarios (`pricing.html`,
`invoices.html`, `directory.html`, `article-habits.html`, `article-focus.html`;
open `index.html` as a launcher). They're **safe** — nothing submits or sends.

Use them for an optional **real** end-to-end capture: open a page in a browser,
copy a value, paste it into TextEdit/Numbers *while the recorder is running*, then
Stop and Analyze. This never performs an irreversible action (no emails, no
messages, no saving over files). The synthetic scenarios reference the same
pages/values, so a live capture should reconstruct the same intent + steps.