# Evaluation

## Host prerequisites

The model-backed runners support native Windows paths and WSL repositories mounted as `/mnt/<drive>/...`. They invoke the Windows DSH installation through `pwsh.exe`; the Windows process environment must resolve `dsh --version` and a drive-qualified `DSH_HOME` or user-profile `.dsh` directory. Native Linux and macOS DSH execution are not supported by these runners.

Both entrypoints validate path conversion, PowerShell startup, the installed DSH command, and the Windows DSH home before creating evaluation artifacts or installing the development plugin. A public Node preload runs a keyless real-worker probe before the DSH version command: it checks persistent bindings, syntax rejection, synchronous and awaited exceptions, and explicit return. The launcher consumes its probe request before creating helpers, so inherited preloads cannot recursively repeat it. The probe owns and cleans its worker scratch directory. A failure uses `PTC-EVAL-PREREQ` and prevents model work.

The actual `process.version`, `process.execPath`, DSH JavaScript entry, and observed release are recorded in manifests; subsequent DSH invocations use that exact Node executable and entry. Do not launch a probe through `npm exec node`, which can resolve a different runtime. The release is evidence, not a behavior selector or compatibility gate. Keep the selected installation and repository DSH dependencies current.

## Configuration Preflight

Model-backed evaluation requires explicit authorization and an explicit provider, model, and credential-variable name. Run it in an isolated DSH home and workspace. Resolve the selected provider from the user's existing DSH configuration, then copy only its validated API, endpoint, model list, and provider-specific routing fields into the isolated settings. Keep the credential variable name unchanged and let the child process resolve its value from the inherited environment; never read, print, serialize, or store credential values.

Generate a fresh routing/session header for providers that require one, and use the same resolved provider configuration for every comparison arm. If the normal profile lacks a required provider field, repair the isolated copy or use the repository's existing isolated audit driver. Do not weaken the provider check or edit the normal profile.

After configuring the selected runner's route, set its configuration-only variable to `1` before invoking the command:

| Command | Configuration-only variable |
| --- | --- |
| `npm run test:expensive` | `DSH_PTC_ACCEPTANCE_CONFIG_ONLY` |
| `npm run test:ab` | `DSH_PTC_AB_CONFIG_ONLY` |

Each runner reads its own configuration-only variable; the names are not interchangeable. When that variable is set to `1`, the runner returns after its isolated configuration checks. Preflight may create local artifacts and isolated configuration; it does not invoke the model. A successful preflight establishes configuration readiness, not model behavior or semantic correctness. Remove the configuration-only setting only when proceeding to the explicitly authorized paid run.

Inspect only redacted session-log and report artifacts, and record model usage without request headers or credential material. Both commands invoke the configured model and consume quota when run without their configuration-only setting.

## Stable ordinary-task A/B fixture

`npm run test:ab` uses a versioned, zero-dependency Node.js fixture as the primary ordinary-task workload. The fixture lives in `fixtures/ab-node-project-v1` and is copied independently into each arm workspace. Both arms receive byte-identical working trees, the same deterministic Git history, and the same deterministic uncommitted dirty state prepared by the runner.

The fixture manifest, `fixtures/ab-node-project-v1/benchmark-manifest.json`, owns:

- `fixtureName` and `fixtureVersion`;
- a content SHA-256 over the fixture tree (excluding the manifest itself);
- the Git identity, commit message, and date used when materializing each arm;
- the deterministic dirty-state entries applied after the initial commit.

The current series is `ab-node-project-v1` / `1.0.0` with content SHA-256 `807580baf64e367bb2dc047c389edf471e13077d54d50ee4b3a143c38f0cebd1`. Reports written by the A/B runner record the fixture path, version, and hash.

Self-hosting acceptance against the active PTC Plus checkout is a separate track: `npm run test:expensive` covers plugin-specific model workflows and integration edges. Comparable ordinary-task reports are grouped only by the fixture version and content hash.

## Protocol

The paired run uses the latest installed DSH release, a configured model, PTC mode's code-only direct-tool projection, and `danger-full-access`. The A/B task set contains ordinary project tasks plus a cheap machine-checkable canary that directly invokes `run_code` once and exercises the transport first. After the canary pair passes, the remaining pairs run with the configured concurrency.

Resolved configurations are identical except for `ptc-plus.disabled`. Agent instructions, skills, auxiliary title generation, and local identity extensions are disabled. The model route, permission mode, neutral persona, fixture bytes, resolved tool surface and initial context unrelated to treatment stay paired within each run. The neutral persona uses only prompt variables every supported Host release registers, so one fixture resolves across the recorded release matrix. When capability management is explicitly enabled in the isolated profile, validated source-backed capability plans must match the resolved capability settings and are treatment context: reports retain their text and usage, while pairing compares the unchanged Host context separately. Workload descriptors must explicitly allow and bound capability plan, navigation and description sources; unknown or malformed source forms fail the audit. Recovered description subsets accumulate while visible, count toward cumulative bounds and may repeat an entry only after its formal surface removal. Statistics should remain disabled for independent manual-baseline comparisons.

The shared overlay selects `tools.mode: ptc` and explicitly projects `sandbox-policy.mode` and `approval.policy`. Before any model work, both runners validate the resolved tools row with the public `ToolRuntime.Config` schema from `@deepseek-ai/dsh-tools`, then check the selected evaluation policy. A successful config dump alone does not validate plugin settings; agreement with a runner-supplied expected string cannot bypass the Host schema. `CONFIG_ONLY` also performs this zero-model schema check. The runners do not rely on child-process environment variables to project these policies across the WSL-to-Windows execution boundary.

Each arm executes in an independently named, opaque scratch tree outside the evaluator artifact tree. Treatment names are confined to evaluator-owned records. Blind packets replace the session workspace in descriptions, source, results, and final answers, including Windows, forward-slash, and JSON-escaped spellings, before the arm map is written.

Machine budgets are per session and are enforced as failures. Machine evidence comes from canonical execution values, workspace postconditions, or runner-owned subprocess results. Free-form final answers are never interpreted with prose or negation matching; their correctness remains `blind-pending` until the blind packet is reviewed. The transport canary requires an actual returned string exactly equal to the fixture package name. A JSON-encoded string containing extra quotation marks, a non-string value, console output, and an absent return do not satisfy that condition. Rendered tool text combines logs and values and cannot establish the return type; neither JSON parsing nor the last output line supplies canonical evidence.

The A/B overlays load the same evaluation-only Cordis observer through the public loader. It observes `tools/result`, the frozen final outcome after DSH policy and cancellation, without altering execution, tool schemas, model messages, or durable Session events. Bounded sidecars outside the model workspace retain session/call identity, source SHA-256, return presence/type, exact bounded strings, error outcome, and a separate log count; the observer does not read provider configuration, headers, or credential variables. Synchronous writes finish before the observer returns because the public notification does not await listeners. The baseline canary reads this canonical `run_code` body; the plugin canary reads its validated journal completion wire. Canonical observations preserve `valueWire` for JSON reports and expose decoded `value` only as a non-enumerable in-memory property, so even mismatching BigInt or cyclic values remain reportable failures. Missing, corrupt, duplicate, mismatched or bounded-out canonical evidence is `incomplete`, cannot pass machine acceptance, and stops the remaining paid matrix. Older presentation-only logs remain incomplete for this oracle; later evidence does not rewrite their recorded outcomes.

The observer is a new profile entry, so each overlay registers it with the public `insert` patch operation. Configuration tests compose the overlay through DSH's public patch composer against a base without the observer; an update targeting an absent entry cannot establish successful loading.

Observer identity comes from the public Agent's `session.id`, including real delegated agents. Canonical sidecars remain under the generated run artifact directory after execution scratch cleanup, so family review can inspect child observations as well as root observations. These files are local evaluation evidence and must not be committed.

For `test-gate`, a successfully completed runner-owned subprocess is valid evidence whether the tested project exits zero or nonzero. The observed exit code is recorded, while blind semantic review decides whether the answer reports it correctly. Failure to start or complete the oracle remains a runner failure.

Acceptance metrics follow durable session-log semantics. `modelRequests` counts `step/start` events, each representing one logical model loop step. `headerEpochs` counts `request/header` events, while `headerChanges` counts the subset whose reason is `change`; initial and equal resume epochs remain distinct epochs without being changes. `historyReplacements` counts events whose `surfaceOp.op` is `replace`. Provider retries can issue multiple adapter attempts within one logical step, and the current durable log does not expose a complete physical-attempt count, so no report field infers one from headers, assistant messages, or usage.

Stable-header acceptance canonicalizes empty fields by DSH rules and compares the ordered complete tool-schema JSON across every header epoch. Current DSH delivers system text through `system/message`, independently of header changes. The logical Session format selects the evidence contract: format 3 introduced formal system messages and retired `header.system`; missing or invalid format evidence cannot authorize a clean result. Absence of system messages in a modern-format log is damage, not a reason to use the historical branch.

For modern logs, the shared system audit validates every request header with public `snapshotSessionEvent`; ordered-surface folding alone does not validate header admission. It then uses the public `foldSurface` and `deriveEventMessage` functions, with the maintained public `imageOffloadProjection`, at request headers and assistant message/attempt boundaries to compare the complete ordered system-text vector consumed by the adapter. Durable message identity, source attribution and text-block segmentation remain audit evidence but do not change equal adapter text. Appending a changed or duplicate prompt fails even when no new header is recorded. A byte-identical formal replacement preserves prompt stability; missing, malformed or uninterpretable surface evidence fails. The leading system message supplies the pi-ai prompt, with its text blocks concatenated without inserted separators; later system messages remain history and are still audited. Historical `header.system` remains bounded header evidence for explicitly recorded pre-format-3 logs; it does not certify a modern ordered surface. Both runners use the same audit for persona, SDK and treatment identification, and the A/B runner also uses it to select sessions.

Any unapproved header change fails with the first differing field. `headerPolicy.allowedTransitions` names the exact epoch and route, configuration, or capability condition; `headerPolicy.historyReplacements` declares an exact replacement count and never permits header drift or a system-message change. Hashes remain diagnostic report data rather than the correctness test.

The shared Host contract recognizes `series` as an unchanged request header beginning a distinct message series, not a new schema or an inferred provider call. Usage evidence supports both historical standalone `assistant/chunk` events and compact streams in `assistant/message` or `assistant/attempt`. An embedded stream's last usage agrees with its message's accounting; formally linked duplicate storage is counted once, conflicting evidence fails, and reported usage from attempts without a surface message remains part of reported token traffic. Logical steps, header epochs and physical attempts remain distinct. Missing usage is not a zero-cost claim. An adapter can report initialized zero usage after a failed request; this does not prove that the provider charged nothing. Reported usage cannot establish complete billing.

Stored logs pass strict public `sessionFormatCatalog` restoration before logical auditing. The original JSONL bytes and physical format are retained separately from restored logical messages and the exact inherited-event cut. Public migrations own historical interpretation; a restoration failure cannot authorize clean evidence. In WSL, Windows logs are restored by the selected Windows Node executable so absolute-path validation uses the Host execution world. Plain `parseEvents` parses JSONL syntax only and does not establish logical provenance. Both paid runners and the Web smoke reader consume restored events. Official `message.isError` outcomes remain errors even without an outer error field.

Runtime-context audits track the binding catalog independently of recovery snapshots. They detect unchanged binding text repeated inside an otherwise changed snapshot, while allowing one reassertion after a formal surface replacement removed its evidence. Model-authored ambient declarations are not plugin catalog injections.

### Focused Program Work

The typed program-surface descriptor owns `expect.programWork`: `inspectionSymbols` limits schema expansion, and each `callLimits` entry declares a namespace/member's `maxCalls` and optional `maxCallsPerArguments`. Counts include unsuccessful calls. The fixed fixture is immutable during this task, so its native read is limited to one observation; the parent passes the observed marker into the isolated child as source data. Tree retrieval is limited to one, and find/inspect counts are bounded without requiring one exact source or cell sequence.

The shared trajectory collector validates the journal and exposes decoded nested arguments as `args` and successful results/completions as `value` for in-memory use. These decoded properties are non-enumerable. JSON reports preserve `argsWire` and `valueWire`, canonical PTC Value Graphs that retain undefined, omitted object properties, explicit null, bigint, cycles, shared references and supported non-JSON types. Readers of persisted reports decode these wire fields with the existing value codec; plain JSON values are not substitutes. Auditing uses structured equality, independent of source spelling or object-key order; missing or corrupt evidence fails. `programWork.observations` reports call counts and `repeatedArgumentCalls` separately from `machineMetrics.repeatedSourceCalls`. A workload needing fresh observations must explicitly allow its counts; these limits never suppress runtime calls or establish effect idempotence.

### Managed Session Continuity Cohort

Four opt-in scenarios are the model-side entry points for capability management, mid-session Host surface changes and delegated sessions. They stay outside the five-workflow default cohort ([ADR 0009](adr/0009-bound-expensive-acceptance-default.md)) and run through the same runner, fixture configuration and audit contract with an explicit selection:

```sh
DSH_PTC_ACCEPTANCE_SCENARIOS=managed-adaptive-surface,managed-session-resume,managed-surface-change,subagent-continuity \
DSH_PTC_ACCEPTANCE_PROVIDER=<provider> \
DSH_PTC_ACCEPTANCE_MODEL=<model> \
DSH_PTC_ACCEPTANCE_API_KEY_ENV=PROVIDER_API_KEY \
npm run test:expensive
```

The managed variants come from the shared overlay: a frozen capability policy with adaptive allocation over the common baseline, one closed member (`tools.write`) and one explicitly Host-disclosed member (`tools.pwsh`); the surface-change phases additionally disable the `tool-fs` module. Every phase is a separate DSH process, and later phases adopt the recorded session through the Host's public `--session-id` resume, so the cohort exercises the same configuration-change and restoration path an operator uses. Each paid phase consumes the configured route; run the cohort only with explicit consumption authorization, in release preparation or a dedicated QA pass, and expect one single-turn, one two-turn, one three-turn and one root-plus-delegated session.

- `managed-adaptive-surface` requires the managed SDK render to declare the Host-disclosed member and the always-callable edit transport while the closed member stays absent from the declaration and the runtime namespace, and the declared member stays callable.
- `managed-session-resume` requires a routine resume to publish no second capability plan, keep the system prompt byte-stable (zero history replacements) and restore the earlier binding without redeclaration.
- `managed-surface-change` adds a Host tool after the plan froze, uses it in a durable cell, then removes it and resumes again: the recorded call must still settle, the binding must survive, and no contraction (`PTC-R002`) may be produced.
- `subagent-continuity` delegates one arithmetic cell to a real subagent and audits the delegated session log — PTC Plus declaration, durable journal, returned product — with the same expectation model.

Runtime-context accounting for this cohort allows `capability-plan`, `capability-navigation`, `capability-descriptions`, `capability-contracts` and `runtime-availability` with explicit per-form bounds, and forbids a capability-plan update after the first publication. Host releases differ in details the scenarios deliberately do not depend on: 0.2.0-rc.2 exposes no `working_directory` tool where 0.2.1-alpha releases do, and delegated-session receipts carry different `kind` values. Assertions use stable facts only (session identity, ordered tool schemas, declared members, journal statuses, decoded completions), so a plugin defect stays distinguishable from a Host version difference: a version difference changes the live surface, not the frozen plan, the binding reconstruction or the delegated session's PTC Plus behavior.

### Guarded Edit Acceptance

Every edit is validated by the canonical editor against the target reconstructed at dispatch from the session log. Both literal and regex deltas accept an optional valid `expected_target_call_seq`; successful edits must preserve the captured target and the editor's exact derived source and description. Scenario-specific literal deltas and source privacy remain separately checked by `editTransports`.

`expect.guardMismatchEdits` lists the one-based edit ordinals intentionally targeting a different call. Such an edit passes only with the structured `{ edited: false, reason }` result and no journal or derived execution metadata. Other edits still require their execution journal. Deterministic tests execute diagnostic-generated EOF closures and reject stale guards through the existing Host tool pipeline; the default paid cohort stays at five workflows.

## Recorded paired observation

One identity-blinded paired experiment used `opencode-go/deepseek-v4-flash`, the same versioned fixture, task prompts and permissions in both arms, and two repetitions of each of nine tasks: 18 sessions per arm.

| Nine tasks, two repetitions | PTC Plus | DSH PTC mode without PTC Plus | Observed difference |
| --- | ---: | ---: | ---: |
| Model requests | 66 | 88 | 25.0% fewer |
| Tool calls | 50 | 79 | 36.7% fewer |
| Token traffic | 729,642 | 942,901 | 22.6% lower |
| Identity-blinded rubric score | 138 / 162 | 118 / 162 | 12.3 percentage points higher |

In the module-syntax task, PTC Plus completed each repetition in one `run_code`. The other arm failed the static-import requirement in both repetitions, using eight tool calls in total. This stochastic observation is not a performance guarantee. Two PTC Plus sessions and five sessions in the other arm exceeded preset machine budgets, so the overall matrix did not pass machine acceptance. Token traffic includes input, cache-read, cache-write and output tokens. These measurements describe that experiment, not acceptance of subsequent plugin changes.

The [2026-10-06 Command Goat / DeepSeek V4.1 Flash report](evaluations/command-goat-deepseek-v4.1-flash-20261006.md) records a separate complete 60-root matrix with four formal children. Both arms completed 24/26 ordinary tasks; PTC Plus's ordinary-family reported traffic was 18.04% higher. Capability probes account for most of its overall completion advantage. The report retains the 6/8 acceptance outcome, the stopped initial pair, exact byte audits, sealed blind scores and cache/billing limits; it does not establish broad savings.

## Reporting

### Binding Workflow

`scripts/binding-workflow-scenario.json` owns the constant-helper authoring, user save, and availability scenario. `scripts/binding-files-workflow-scenario.json` separately covers a file helper with write/delete capabilities and a pre-existing directory and sentinel. `test/binding-workflow.test.js` runs both through public CommandRuntime, AgentLoop, ToolRuntime, Session, and prompt assembly with a deterministic adapter. It checks saving, availability observation, preservation of pre-existing content, and rejection of write/delete probes. The constant workflow also checks ordinary history, stable system/tools, and restart. These fixtures prove the Host and audit contracts, not stochastic model behavior or provider cache gains.

The deterministic authoring test additionally keeps one command turn active across several real REPL cells: an in-memory comparison exposes a wrong result, corrected source passes assertions, and that exact source is submitted as a disabled draft without a store write. It checks the generated task's test scope and final-source review guidance alongside stable system text and tool schemas. This proves the execution path is available, not that an unconstrained model chooses it.

For model-backed authoring evaluation, include both new and edit requests, normal and edge inputs, and a helper whose external integration must remain untested. Inspect submitted source, tool calls and results for tests before acceptance, correct use of feedback, unnecessary discovery, external effects and unsupported completion claims. Compare first-call latency, total time, request count and token usage under the same explicit route; a shorter reasoning block or more calls alone is not a quality criterion. Use concise result evidence when reviewing final answers, not reasoning transcripts as correctness proof.

The packed Client can also be checked in a real isolated DSH Web profile without model requests:

```sh
npm run test:client:web -- --dsh-entry <installed-dsh-cli-js> --binding-workflow
```

Use `--browser-channel <channel>` when testing with an installed Chromium browser. The local adapter drives public command and tool execution; Playwright saves and discards drafts through the UI, checks English and Chinese lifecycle labels, refreshes saved source, and measures settled desktop and narrow layouts including dark mode. Screenshots and measurements are written to `artifacts/client-web-smoke`; the temporary Host and home are cleaned up. `npm run test:client:layout` separately checks focused card, list, and REPL-tab geometry.

For an explicitly authorized model check, use an isolated DSH home and workspace with the selected provider, model, and credential-variable configuration. Run the keyless worker probe with the same resolved Node executable before starting DSH. First send an ordinary message, then run the scenario command, review the generated source, choose Save and enable in the binding review dock above the composer, and send its status question. Export that session log for `auditBindingWorkflow(events, scenario)` in `scripts/acceptance-contract.mjs`; `inspectLog` accepts the same descriptor as `expect.bindingWorkflow`. This interactive workflow remains separate from the five default paid scenarios because the headless single-task runner cannot perform the user save action.

The audit owns completeness: matching command and successful admission, accepted candidate and its successful SDK transcript, a formally linked save before the exact status question, settled calls, and a completed answer must all exist. Results distinguish `incomplete`, `unproved`, and `machine-passed`; all nonpassing results include failures. Final prose always retains `semantic-review-required` and must be checked against actual model-visible facts.

The oracle rejects routine host/storage scanning and checkpoint reflection. `constantExports` optionally proves only nullary synchronous exports returning declared primitive literals, from the exact accepted source, save relation, and the status cell's validated active snapshot and journal fingerprint. A bounded AST proof tracks local values and only marks catch unreachable after proving its try body cannot throw. Unsupported syntax, object coercion, source mismatch, and shadowing remain insufficient evidence; a name allowlist cannot authorize calls. Export metadata alone does not prove a runtime value's type. Reports distinguish `recorded-dispatch`, `potential-source`, and `unreachable-source`; neither potential source nor insufficient evidence claims an effect happened. The next real user message ends the status scope. These rules belong solely to acceptance, not runtime permission or interception.

An effectful follow-up test needs separate task authorization. Its fixture must acquire an exclusive temporary directory with the platform primitive, preserve any pre-existing same-name directory and unrelated sentinel, and clean only its owned directory. For a status-only workflow any write/delete test is a failure, regardless of cleanup success. Historical logs can be audited offline; stochastic model quality and cache improvements remain unmeasured until an explicitly configured model run is performed.

Both runners require an explicit model route and credential-variable name before any host probing or artifact creation. The referenced credential variable must already be available in the inherited environment. These configuration-only examples name it without reading or assigning its value:

```sh
DSH_PTC_ACCEPTANCE_PROVIDER=<provider> \
DSH_PTC_ACCEPTANCE_MODEL=<model> \
DSH_PTC_ACCEPTANCE_API_KEY_ENV=PROVIDER_API_KEY \
DSH_PTC_ACCEPTANCE_CONFIG_ONLY=1 \
npm run test:expensive

DSH_PTC_AB_PROVIDER=<provider> \
DSH_PTC_AB_MODEL=<model> \
DSH_PTC_AB_API_KEY_ENV=PROVIDER_API_KEY \
DSH_PTC_AB_CONFIG_ONLY=1 \
npm run test:ab
```

For the authorized model run, remove the corresponding configuration-only assignment after preflight succeeds. `npm run test:ab` then writes `report.json`, `report.md`, per-session trajectory artifacts, and blind-review packets under `artifacts/ab-trajectories/`. Both commands invoke the configured model and consume quota outside configuration-only mode.

Token traffic is the sum of input, cache-read, cache-write, and output tokens. Model behavior is stochastic, so comparable new results establish a reproducible observation rather than a universal performance claim.