================================================================================ FILE: README.md ================================================================================

# hotato **Find what broke in your agent calls. Pin it so it never ships again.** ```bash pip install hotato hotato autopsy --demo # a bundled failing call: what broke, and when hotato autopsy ./call.wav # then your own recording hotato vapi health # or your last 100 Vapi calls ``` Zero config. Vapi, Retell, Bland, Synthflow, Millis, or local audio. Timing math, not a judge. Offline. Free. MIT. **[hotato.dev](https://hotato.dev)**

## What it finds - **Say-do gaps**: the caller interrupts to cancel (a barge-in), the agent says "canceled", the booking tool fires anyway. hotato takes the turn timing from the audio and the tool call from your OTel trace. - **Latency spikes**: the pause before a reply going from 800 ms to over 2 s. - **Dead air**: that pause reaching 5 s, or the line going quiet. - **Talk-over**: the agent starts a fresh utterance over the caller. ## Quickstart ### Vapi ```bash pip install hotato export VAPI_API_KEY=... hotato vapi health --last 7d --output report.html ``` Open `report.html`: every critical incident, timestamped, and your Voice Stability Score. ### Retell ```bash export RETELL_API_KEY=... hotato retell health --call-id CALL_ID ``` `--call-id` is required and repeatable: you name the Retell calls to pull. `hotato bland health`, `hotato synthflow health`, and `hotato millis health` follow the Vapi shape. Those stacks mix both voices onto one channel, so they measure silence timing, dead air and latency gaps, each finding with its measured confidence; barge-in and talk-over need two channels. ### Local audio ```bash hotato autopsy ./call.wav ``` Writes a self-contained HTML report to `hotato-output/`, plus the JSON `pin` reads. Open the HTML in your browser. ## From finding bugs to preventing them `autopsy` turns one recording into timestamped incidents. `scan` reads a folder and tracks the trend. When you are ready, move a finding into CI: `hotato pin` turns one incident into a portable failure check, and `hotato prove` re-runs every stored check and fails the build rather than pass on evidence it cannot re-read. Every verdict carries its own evidence across five dimensions: outcome, policy, conversation, speech, reliability. [Pin a bug →](docs/CI.md) For continuous use: run `hotato vapi health` on a schedule, and open `hotato console --production-db evidence.db` to watch calls land live. ## Wire it into CI The exit code **is** the verdict: `0` pass, `1` fail, `2` refuse (could not tell). ```yaml # .github/workflows/voice-qa.yml on: [pull_request] jobs: hotato: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: attenlabs/hotato@v1.18.1 with: contracts: contracts/ hotato-version: 1.18.1 ``` `contracts/` holds what `pin` wrote. Full workflow: [`docs/CI.md`](docs/CI.md). ## Point your agent at it Point Claude Code, Cursor, or any coding agent at this repo: it reads [`AGENTS.md`](AGENTS.md) and runs the loop end to end offline, no key. Over local stdio the MCP server adds the scorer plus read/verify/propose tools: `uvx --from "hotato[mcp]" hotato-mcp` ([`docs/MCP.md`](docs/MCP.md)). Deploying is yours. ## Nothing leaves your machine hotato runs offline, on the machine that invokes it. The core is stdlib-only Python: no account, no key, no network call of its own. Your traces, prompts, and audio stay on your disk. Scoring with a language model you host yourself is a separate opt-in add-on, outside that core. ## Go deeper The whole loop, command by command: [`docs/LIFECYCLE.md`](docs/LIFECYCLE.md). First touch to a CI gate: [`docs/GETTING-STARTED.md`](docs/GETTING-STARTED.md). Feed it what you already have: [`docs/CONNECT.md`](docs/CONNECT.md) · [`docs/TRACE.md`](docs/TRACE.md) · [`docs/SIMULATE.md`](docs/SIMULATE.md). What every verdict stands on: [`docs/EVIDENCE-CONTRACT.md`](docs/EVIDENCE-CONTRACT.md). Next to the hosted alternatives: [`docs/COMPARE.md`](docs/COMPARE.md). The deep toolkit -- capture, simulation, load, benchmarking, the fix ladder, the fleet control plane -- lives under `hotato lab` (`hotato lab --help`). The public commands are durable. hotato lab moves faster, and every command name that worked before 1.17 still runs unchanged. ## Specifications | Property | Value | | :-- | :-- | | Footprint | ~10 MiB installed, 0 runtime dependencies (stdlib-only) | | Reproducibility | byte-for-byte: the same recording, the same report | | Exit codes | `0` pass · `1` fail · `2` refuse | | Release integrity | OIDC Trusted Publishing + build-provenance attested | | Runtime | offline, off the production data path |

Verify the measurement yourself

```bash PYTHONPATH=src python3 -m hotato.benchmark \ --scenarios corpus/real/scenarios --audio corpus/real/audio ``` On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is **20 ms**. Provenance: [`corpus/real/README.md`](corpus/real/README.md) · method: [`METHODOLOGY.md`](METHODOLOGY.md). Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked **NOT SCORABLE** and refused (`hotato trust --stereo call.wav`). The full four-tier evidence policy (what each verdict stands on, per input) is [`docs/EVIDENCE-CONTRACT.md`](docs/EVIDENCE-CONTRACT.md).

## Contribute Issues and PRs welcome: [`CONTRIBUTING.md`](CONTRIBUTING.md) · [`SECURITY.md`](SECURITY.md) · [`CHANGELOG`](CHANGELOG.md) · [`docs/`](docs/) ## License MIT ([`LICENSE`](LICENSE))

_{Know when to pass it on.}

mcp-name: io.github.attenlabs/hotato ================================================================================ FILE: docs/GETTING-STARTED.md ================================================================================ # Getting started One path, no forks: from first touch to a CI gate that guards every pull request. Follow the five steps in order. Each command prints the exact next command, so you can chain the whole loop from what hotato tells you. The transcript passed and the call still failed: the agent talked over the caller, ran through the interruption, or left a beat of dead air handing the floor back. None of that is in the words. Hotato scores the turn timing between the two voices of a recording, freezes one caught moment as a contract, and re-runs it in CI. ## Install Zero install, run it now: ```bash uvx hotato start --demo ``` Keep it around for daily use: ```bash pipx install hotato # or: uv tool install hotato, or: pip install hotato ``` Scoring needs two separate channels (caller on one, agent on the other). A mono or mixed export is refused: marked NOT SCORABLE, exit 2. Check a file before you score it with `hotato trust --stereo call.wav`. ## The five steps ### 1. See it catch a failure ```bash hotato start --demo ``` Sweeps the two bundled demo calls, builds one failure contract, and runs one say-do check: the agent said the refund was sent; the trace decides. Exits 0 because setup finished; `hotato contract verify contracts/` exits 1 by design. ### 2. Score your own recording ```bash hotato investigate ./call.wav ``` ```console hotato investigate [run 1]: call.wav input health: eligible for scan verdict path: eligible (a labeled event here can carry a yield/hold verdict) most likely failure (top-ranked candidate): [1] t=7.63s agent_stop_no_caller trailing_silence_sec=0.37, caller_proximity_sec=0.5 next: label it (use --expect hold instead if the agent was right to keep talking): hotato investigate label '.hotato/investigate-state.json#1' --expect yield ``` Hotato ranks the timing moments, marks the top one `most likely failure`, and hands you one command to label it. It reads timing, not intent. Have a provider call id instead of a WAV? `hotato investigate --stack vapi --call-id ` pulls it first, then ranks the same way. ### 3. Commit the catch as a regression Yield means the agent should have stopped for the caller. Hold means it was right to keep talking through a backchannel or noise. The label is yours; hotato measures whether the timing matched it. ```bash hotato investigate label '.hotato/investigate-state.json#1' --expect yield ``` ```console created hotato contract: call-8s-yield dir: contracts/call-8s-yield.hotato expect: yield passed: False measured: did_yield=False seconds_to_yield=n/a talk_over=0.00s next: hotato contract verify contracts open the pull request that adds it to your repo's CI gate: hotato pr create --fixtures contracts/call-8s-yield.hotato --repo OWNER/REPO --title 'Add hotato contract call-8s-yield' ``` The contract bundle is content-addressed: the clipped audio, the frame-level evidence, the policy, and its own attestation, committed byte-identical. ### 4. Open the pull request ```bash hotato pr create --fixtures contracts/call-8s-yield.hotato --repo OWNER/REPO --title 'Add hotato contract call-8s-yield' ``` Stages the bundle byte-identical under `tests/hotato/contracts/` and opens the PR. It is a dry run by default; add `--yes` to run git and gh. ### 5. Let the gate re-run the evidence ```bash hotato contract verify contracts/ ``` ```console hotato contract verify: contracts (1 contract) [FAIL] call-8s-yield (expect yield): did_yield=False seconds_to_yield=n/a talk_over=0.00s | integrity: intact 0/1 contracts pass; exit_code=1 These contracts pin known failures. Each stays red until you fix the agent and recapture the call, the same way a snapshot test stays red until you update the snapshot. Path to green: fix the agent, then recapture with `hotato drive ` (vapi/twilio), or the manual path in docs/RECAPTURE.md. ``` This re-measures the stored evidence deterministically. It is the CI gate: exit 0 pass, exit 1 fail. ## When the gate is red A committed contract holds a call that already failed, so exit 1 is the correct reading, the way a snapshot test stays red until you update the snapshot. The audio is frozen, so the gate turns green once you fix the agent and recapture the call. Until then the red names one moment on one call, waiting for review. To get to green, fix the agent, then recapture: ```bash hotato drive # re-run a fresh call against the live agent (vapi/twilio) ``` `hotato drive` originates a new call for vapi and twilio and reports a before and after verdict. For every other stack, or by hand, follow [`RECAPTURE.md`](RECAPTURE.md). ## Reference The five steps are the whole loop. The pages below go deeper on it; take them one at a time, in whatever order the work calls for. - [`START.md`](START.md): the guided demo, both acts (timing and say-do), in detail. - [`INVESTIGATE.md`](INVESTIGATE.md): capture-origin authentication and candidate ranking. - [`CONTRACTS.md`](CONTRACTS.md): what a contract bundle holds and what `verify` proves. - [`CI.md`](CI.md): the copy-paste GitHub Action with a commit-SHA pin and a PR results comment. - [`RECAPTURE.md`](RECAPTURE.md): the path from a red gate to a green one. - [`ASSERTIONS.md`](ASSERTIONS.md) and [`TRACE.md`](TRACE.md): the say-do check from traces you already log. - [`MCP.md`](MCP.md): drive the loop from Claude Code, Cursor, or any MCP client. - [`../AGENTS.md`](../AGENTS.md): the same loop, written for a coding agent, plus the machine contract. Symptom-first walkthroughs, each opening with the direct answer and running on what ships in the repo: - [`scenarios/browser-vs-pstn.md`](scenarios/browser-vs-pstn.md): passes in the browser, fails on the phone; score the same moment clean and codec-degraded. - [`latency-waterfall.md`](latency-waterfall.md): perceived latency worse than the dashboard; the per-hop waterfall from your traces. - [`scenarios/load-and-recovery.md`](scenarios/load-and-recovery.md): breaks under concurrent call load; the evidence-preserving load staircase. - [`scenarios/dtmf-verification.md`](scenarios/dtmf-verification.md): verifying DTMF from the evidence your pipeline logs. - [`scenarios/echo-self-interruption.md`](scenarios/echo-self-interruption.md): the agent interrupts itself; TTS bleed measured and gated. - [`scenarios/false-interruption-replay.md`](scenarios/false-interruption-replay.md): stops talking on "mhm"; pin the false yield as a CI contract. ================================================================================ FILE: docs/AUTOPSY.md ================================================================================ # `hotato autopsy `: one call in, the incidents out Drop one call recording in with zero config and get the incident list: each moment -- barge-in, talk-over, dead air, a latency spike -- with its timestamp and magnitude, one HTML report, and an `apx-` id from its bytes. ```bash hotato autopsy call.wav ``` ``` hotato autopsy: call.wav (12.0s, 2 channels, stereo) [CRITICAL] BARGE-IN t=2.99s overlap=1.96s agent did not go silent within 3.0s the caller took the floor and the agent kept talking over them for 1.96 s without going quiet within 3.0 s [WARNING] DEAD AIR t=6.65s trailing silence=1.94s no caller energy within 0.50s the agent went quiet for 1.94 s with no caller energy nearby 2 incidents: 1 critical, 1 warning report: hotato-output/autopsy-apx-cc33f46fad58.html pin: apx-cc33f46fad58 (incidents apx-cc33f46fad58#1..#2) ``` WAV reads natively; mp3/m4a convert through `ffmpeg` when it is on PATH (when it is not, the one-line message names the install command). Everything runs offline; no audio leaves the machine. `hotato autopsy` with no recording prints the quick start on the bundled rendered example. ## Quick start from your platform (Vapi, Retell, Bland, Synthflow, Millis) You never touch a WAV: one command pulls your recent calls from the platform's API, checks each, and leads the health report with the Voice Stability Score: your dual-channel calls' critical-free share x 100. ```bash export VAPI_API_KEY=YOUR_KEY # or: hotato connect vapi hotato vapi health ``` ``` hotato vapi health: pulled 38 of 38 listed calls (0 skipped, last 7d) -> hotato-output/vapi-calls hotato scan: vapi-calls (38 recordings: 38 analyzed, 0 refused) Voice Stability Score: 74/100 (38 dual-channel calls; policy 9412d82c1328) health: 28 of 38 dual-channel calls had no critical incidents (74%) ... ``` The entries share one implementation and the exact same flags: ```bash hotato vapi health # VAPI_API_KEY / hotato connect vapi hotato retell health --call-id ID # RETELL_API_KEY / hotato connect retell hotato bland health # BLAND_API_KEY / hotato connect bland hotato synthflow health # SYNTHFLOW_API_KEY / hotato connect synthflow hotato millis health # MILLIS_API_KEY / hotato connect millis ``` | Flag | Meaning | | :-- | :-- | | `--last WINDOW` | how far back to pull (e.g. `7d`, `12h`, `2w`; default `7d`) | | `--limit N` | maximum calls to pull (default 100) | | `--dir PATH` | download directory (default `hotato-output/-calls`) | | `--output PATH` | also write the HTML health report to PATH (the content-addressed report under `hotato-output/` is written either way) | | `--call-id ID` | check this call id (repeatable; skips the list step). Required for Retell, which has no verified list-recent-calls endpoint -- hotato never guesses one | | `--api-key KEY` | vendor API key (else the `hotato connect` store, else the stack's env var) | | `--format text\|json` | output format (default text) | Credentials resolve exactly as `hotato pull` does -- an explicit flag, then the `hotato connect` store, then the stack's env var (`VAPI_API_KEY` / `RETELL_API_KEY` / `BLAND_API_KEY` / `SYNTHFLOW_API_KEY` / `MILLIS_API_KEY`) -- and a missing key is one actionable line. Vapi and Retell fetch the separated two-channel recording; Bland, Synthflow, and Millis export one mixed channel, so each of their calls runs the measured-confidence mono path below (silence timing measured from the mixed channel; talk-over attribution comes from a two-channel recording -- the scope line states this once per run). Mono calls report into the best-effort mono observations block with their own counts and never enter the Voice Stability denominator, so a mono stack's report carries its observations and no stability score ([EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md) states the tier policy, a tier being what a recording can prove). The analysis runs on this machine; recordings download straight from the platform and go nowhere else. A window with no calls, a pull in which every recording failed to fetch, or a pulled set with zero analyzable calls refuses with the reason (exit 2) -- no score is reported over zero calls. ## Stereo: the deterministic path A two-channel recording (caller on one channel, agent on the other) runs the existing whole-call scanner ([`hotato scan`](../src/hotato/scan.py)) unchanged: the same walk, the same measured numbers, byte-for-byte. The same file produces byte-identical CLI text and a byte-identical report on every run -- the report is even named by content (see the autopsy id below), so the path is stable too. The scanner's candidate kinds map to the incident vocabulary: | Incident | From | CRITICAL when | | :-- | :-- | :-- | | `BARGE-IN` | the caller became active while the agent was talking (`overlap_while_agent_talking`) | the agent kept talking over the caller past the 1.0 s prompt-yield ceiling, or never went quiet in the search window | | `TALK-OVER` | the agent started a fresh utterance over the caller (`agent_start_during_caller`) | the overlap exceeds the same 1.0 s ceiling | | `DEAD AIR` | a response gap of 5 s or more (`long_response_gap`), or the agent stopping with no caller energy nearby (`agent_stop_no_caller`) | the gap is 5 s or more | | `LATENCY SPIKE` | a response gap between 2 s and 5 s (`long_response_gap`) | never (a warning) | | `ECHO SUSPECTED` | the caller channel tracks the agent's own audio (`echo_correlated_activity`) | never (a caveat) | Each incident block carries the severity, the timestamp, the measured magnitudes (`candidate_detail`), and the scanner's plain-English sentence (`candidate_plain_english`) -- the same measured numbers, restated once. ## Mono: best-effort, confidence-scored A one-channel (mixed) recording is analyzed best-effort with the same energy VAD, the voice-activity detector that marks each frame loud or quiet. One mixed channel measures silence timing -- dead air and latency gaps -- and every mono finding carries a **measured confidence** with its derivation printed beside it: how far the gap's mean energy sits below the speech-activity threshold the VAD measured for this recording (a 20 dB margin or more scores 1.00). Talk-over and barge-in attribution comes from a two-channel recording, where the caller and the agent are physically separated; that scope is stated once, on one line, in the output. A mono gap says everything stopped, not who stopped. The stricter commands keep their bar: `run`, `scan`, `trust`, and the contract path still refuse mono as NOT SCORABLE. Autopsy is discovery; the CI gate stays deterministic and dual-channel. The full four-tier policy behind this split -- dual-channel deterministic, mono with provider metadata, raw mixed mono, refused -- is stated once in [EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). An unreadable input -- a text file, a truncated header, a non-audio blob -- is refused with the reason (exit 2), never scored. ## The report One self-contained HTML file under `./hotato-output/`, in the [`hotato report`](REPORTS.md) house style: the per-channel energy waveform with one labeled marker per incident, then a card per incident with the same measured numbers the CLI printed. Zero external requests and zero scripts; the page renders the same bytes for the same recording on every run. ## The autopsy id `apx-` + the first 12 hex chars of the sha256 of the input file's bytes: content-derived, so the same recording gets the same id on any machine, whatever the file is named (an mp3 hashes its own bytes, so the id is independent of the local ffmpeg build). Incidents are addressed as `#` -- the `pin:` line prints both -- and the report is `hotato-output/autopsy-.html`. ## The persisted envelope Every autopsy also writes `hotato-output/autopsy-.json` next to the HTML: the machine-readable result envelope -- the source path, the mode (stereo or mono), and the incidents with their onset, kind, and (on the stereo path) the underlying scan candidate kind. Like the report it is content-addressed and deterministic: the same recording writes the same bytes on every run. The envelope carries the measured facts only; est. cost figures live on the rendered surfaces (only under `--cost-config`), never in the stored envelope. This is the offline store `hotato pin` resolves an `apx-...#N` ref from without re-running the analysis. ## `hotato scan `: the folder health report Point `hotato scan` at a folder and it runs the autopsy engine over every recording in it -- stereo through the deterministic scanner, mono best-effort, exactly the rules above; an unreadable file is listed as refused with the reason, never skipped silently: ```bash hotato scan ./calls ``` ``` hotato scan: calls (5 recordings: 4 analyzed, 1 refused) Voice Stability Score: 25/100 (4 dual-channel calls; policy 9412d82c1328) SMALL SAMPLE: 4 dual-channel calls, under the 20-call bar health: 1 of 4 dual-channel calls had no critical incidents (25%) ... evidence coverage (measured from this run): dual-channel timing: 4 calls -- deterministic two-channel timing walk refused: 1 file -- unreadable as call audio; every file listed with its reason, never scored ... worst calls (critical count, then worst measured magnitude): 1. late-followup.wav apx-6836885bd877 1 critical, 0 warning worst 5.14s gap ... report: hotato-output/scan-scn-46a5af300d6b.html envelope: hotato-output/scan-scn-46a5af300d6b.json ``` The HEALTH headline is a measured share -- **dual-channel calls** with zero critical incidents over dual-channel calls analyzed. The **Voice Stability Score** is that same share, times 100: `round(share x 100)` and no second arithmetic (the machine field is `critical_free_call_rate`). The share line prints directly beneath the score as its formula, the eligible sample size and the analysis-policy sha print beside the score, a `SMALL SAMPLE` label renders under 20 dual-channel calls, and the HTML report carries a one-line "How this is calculated" note pointing at the share line. A mono call **never enters the denominator**: mono-analyzed calls report into the *Best-effort mono observations* block with their own counts (measured silence timing from one mixed channel; talk-over and barge-in attribution comes from a two-channel recording). With zero dual-channel calls no score renders: the report states why and shows the mono observations alone. Every number keeps its own denominator, because one blended score would hide exactly the distinction the tool exists to draw (see [METHODOLOGY.md](../METHODOLOGY.md)). The share sits beside the **evidence coverage** block (a lane is one evidence path -- dual-channel timing, mono best-effort, refused with its reason -- each carrying the count this run measured, so a lane reads as assessed only when it carried evidence), a per-category breakdown (counts plus the worst measured magnitude in each category), and the worst-calls ranking, each call linking to its own per-call autopsy report, generated alongside its envelope -- so `hotato pin` works straight from a folder scan. The scan is deterministic end to end: the same directory with the same flags produces byte-identical CLI text and a byte-identical HTML report, and the outputs are content-addressed (`scn-` + 12 hex over the sorted file-content manifest plus the analysis flags). The summary envelope (`scan-.json`) stores the aggregate with a `recorded_at` provenance stamp written once, when that content is first seen; a re-run of unchanged content resolves to the same file and leaves it untouched. **Trend.** When prior summary envelopes for the same directory sit in the output dir, the report renders a run-over-run strip from them -- each prior run's share and critical count under its stored provenance timestamp, with the current run beside them. The page stays byte-identical given the same directory and the same prior-run store. **Recurrence states.** An incident kind present in the current run that also appears in stored prior runs of the same directory prints a recurrence line, in the CLI text and in the report, each line carrying a measured state: ``` RECURRING: DEAD AIR in 7 of 38 calls this run (23 in the stored window). Also present in 3 prior run(s): 2026-07-08T09:00:00Z, 2026-07-15T09:00:00Z, 2026-07-22T09:00:00Z. ``` The states, all derived from stored facts: `observed` (1-2 calls carry the kind across this run plus the stored prior runs), `RECURRING` (3+), `RECURRING, LOW SAMPLE` (3+ but the eligible dual-channel sample is under 20 calls), and `ELEVATED` (this run and the most recent comparable prior run -- same analysis policy, same evidence lanes, 20+ eligible dual-channel calls in both -- have Wilson 95% intervals on the kind's per-call rate that do not overlap, this run higher). Every count and date is a measured aggregate or a stored envelope's `recorded_at` provenance -- never extrapolation -- so the same directory and the same prior-run store always print the same lines. The platform health commands run the same aggregate over their download directory, so re-running `hotato vapi health` week over week builds the store the recurrence lines read from. `--cost-config` renders est. cost totals across the analyzed calls, from your own per-incident figures, exactly as above. For one recording, `hotato scan --stereo call.wav` lists that call's candidate turn-taking moments, byte-for-byte on every run. ## `hotato pin `: incident to contract `hotato pin` turns one autopsy incident into a portable `.hotato` failure contract -- that clip, its frame evidence, its thresholds, frozen -- through the same contract machinery (`hotato contract create` on the recording at the incident's onset): ```bash hotato pin apx-cc33f46fad58 # the call's top critical incident hotato pin apx-cc33f46fad58#1 # one specific incident ``` Resolution is offline, from the persisted envelope under `./hotato-output/` (or `--from DIR`). Before delegating, pin re-hashes the source recording: the CURRENT bytes must still hash to the pinned autopsy id, so a file that changed on disk refuses rather than binding a different call. The incident kind maps to the contract's expect decision: BARGE-IN and TALK-OVER -- the floor-holding events the yield/hold contract vocabulary expresses -- default to `--expect yield` (the caller held the floor), and `--expect hold` records the human's call that the agent was right to keep talking. A mono-derived incident refuses (contracts require the two-channel deterministic path); DEAD AIR and LATENCY SPIKE are silence-timing measurements with no caller onset to pin, and ECHO SUSPECTED is a caveat, so those refuse with the reason too. Every refusal -- malformed ref, unknown id, rank out of range, missing or changed source, mono -- exits 2 and leaves no artifact. Success prints the bundle dir and the CI step: ```bash hotato prove --contracts contracts ``` ## est. cost, only from your figures Cost lines render only when you supply your own per-incident figures; hotato ships no default dollar amount: ```bash hotato autopsy call.wav --cost-config costs.json ``` ```json {"currency": "USD", "per_incident": {"dead-air": 3.0, "barge-in": 2.0, "talk-over": 2.0, "latency-spike": 1.0}} ``` Each priced incident gets an `est. cost` line naming the kind the figure came from, and the summary totals them. With no config, no figure appears anywhere. ## The bundled examples Three deterministic rendered example calls live in `examples/autopsy/` (seeded renderer, seed = `sha256(id)`, byte-identical on any machine) -- rendered demonstrations of the failure patterns, one each: ```bash hotato autopsy examples/autopsy/audio/autopsy-01-barge-in-say-do.example.wav hotato autopsy examples/autopsy/audio/autopsy-02-latency-dead-air.example.wav hotato autopsy examples/autopsy/audio/autopsy-03-talk-over.example.wav ``` ## From an incident to a regression gate Turn one incident into a standing CI gate: ```bash hotato pin apx-cc33f46fad58#1 # incident -> portable contract hotato prove --contracts contracts # the CI gate ``` The labeled-review path does the same through `hotato investigate`: ```bash hotato investigate call.wav # ranked candidates + the label command hotato investigate label .hotato/investigate-state.json#1 --expect yield hotato contract verify contracts/ --junit hotato.xml # the CI gate ``` ## Scope and method Deterministic energy measurement over time: per-frame RMS, a transparent activity threshold, and the timing walk between the tracks. Timing and floor-holding, not intent or transcription -- the scanner cannot know whether a caller sound was "mhm" or "stop", so what it reports is measured timing, not an accuracy score. See [METHODOLOGY.md](../METHODOLOGY.md). ================================================================================ FILE: docs/EVIDENCE-CONTRACT.md ================================================================================ # The evidence contract: four tiers, one policy Every hotato verdict states what evidence it stands on. This page is the single source of truth for that policy; the README, [AUTOPSY.md](AUTOPSY.md), the platform health reports, and the trust docs ([TRUST.md](TRUST.md) · [TRUST-MATRIX.md](TRUST-MATRIX.md)) all apply it and link back here. Four tiers, by what the recording itself can prove: ## Tier 1 -- dual-channel audio: deterministic Caller on one channel, agent on the other, separated at capture. Overlap and timing are facts of the recording, exact to the sample, so the deterministic timing walk runs end to end: byte-identical output for the same input, every threshold and frame inspectable. Only this tier is **verdict-eligible** -- `hotato pin` freezes a caught moment as a contract, `hotato prove` and the CI gate re-run it, and it alone enters the Voice Stability denominator. ## Tier 2 -- mono plus provider metadata: attributable, with declared authority A mixed single channel cannot attribute energy to a speaker by itself, but metadata a provider or pipeline supplies alongside it can: a speaker-attributed transcript, a voice-pipeline trace, tool-call logs, or a diarizer's turn labels (`hotato run --mono call.wav --diarize`, [DIARIZE.md](DIARIZE.md)). A finding here is **attributable**: something assigned the speech to a speaker, and the finding names that source. The production evidence plane records it, weakest claim first -- `submitted`, `adapter_reported`, `provider_export`, `signed_attestation`, `measured`. A verdict a diarizer attributed is stamped `diarized-mono`, and `indicative_only` when that diarizer's own confidence sits below the bar. Attribution is only as strong as its source, so the authority prints with the finding and this tier never merges into Tier 1's deterministic counts. ## Tier 3 -- raw mixed mono: symptom detection, with measured confidence One mixed channel with no metadata still measures what silence shows: dead air and latency gaps. `hotato autopsy` (one recording in, the incident list out) and the mono-stack health reports run this path best-effort, and every finding carries a **measured confidence** with its derivation printed beside it. A mono gap says everything stopped, not who stopped -- talk-over and barge-in attribution comes from the tiers above, that scope is stated once per run, and Tier 3 findings report in their own block, outside the Voice Stability denominator. ## Tier 4 -- insufficient evidence: refused, with the remediation An input that supports none of the above -- an unreadable file, a silent required channel, a mixed export where a deterministic verdict was asked for -- is **refused** (exit `2`): hotato scores nothing and prints the reason and the next step -- the recording scaffold for your stack (`hotato setup`), the input health check (`hotato trust --stereo call.wav`), or the mono escapes above. A refusal leaves no artifact and never becomes a number, so a green or red build only ever reports calls that were scored. ## Reading a report against the tiers The health and scan reports render the split directly: the Voice Stability Score and the `health:` share count **dual-channel calls only** (Tier 1); mono findings sit in the *Best-effort mono observations* block with their own counts (Tier 3, or Tier 2 where metadata attributed them); refused files are listed with their reasons (Tier 4), never scored. The **evidence coverage** block prints each lane's measured count (dual-channel timing, mono best-effort, refused), so a lane reads as assessed only when it had evidence. ================================================================================ FILE: docs/CONTRACTS.md ================================================================================ # Failure contracts A failure contract packages ONE call moment into a portable, private bundle: audio, frame-level timing evidence, an input-health report, a PR card, a CI pass/fail policy, and the commands to replay and re-verify it. It is the CI object -- `hotato contract verify` re-scores a directory of contracts and exits non-zero when one regresses. A contract's label always comes from a human (`label_source` is frozen to `"human"`). Hotato measures whether the recorded timing matched that label; `contract verify` re-measures the SAME recording later and reports pass/fail. > **A contract bundle contains call audio (`audio/event.wav`).** Do not > commit a raw customer contract to a public repository; treat it like > any recording of a caller. Use sanitized fixtures (synthetic or > consent-cleared) for anything public, and keep customer contracts in a > private repository or controlled artifact storage. `--include-identifiers` > also writes a source basename and candidate ref into `contract.json` > and the card; leave it off by default. This covers metadata only -- > full audio-handling rules are in > [`SECURITY.md`](../SECURITY.md#audio-handling). ## Two lanes: same command, two different proofs | | Lane A -- frozen recording | Lane B -- fresh recapture | | --- | --- | --- | | **Re-scores** | the SAME `audio/event.wav` the contract was created from | a new recording of the SAME stimulus against your CURRENT agent | | **A pass proves** | the evidence, policy, and scorer are still intact and still agree with the human label | the CURRENT agent's behavior on that stimulus still matches the label | | **Leaves open** | whether the deployed agent's behavior has changed since | nothing extra -- this lane speaks to the live agent | | **Runs** | every push, in the shipped `ci/github_action.yml` (`contract verify contracts/`) | only when you recapture by hand or on a schedule -- see [`docs/RECAPTURE.md`](RECAPTURE.md) | A frozen-recording pass is necessary but not sufficient: the recording never changes, so it can only fail if someone edits the bundle's audio or policy. Confirming the CURRENT agent still yields correctly takes re-running the same stimulus and re-verifying the fresh capture, by hand ([`docs/RECAPTURE.md`](RECAPTURE.md)). Run the frozen gate on every push to catch evidence/policy drift for free, and recapture periodically (or after an agent change) to catch what the frozen recording cannot. ## The bundle `hotato contract create` writes one self-contained directory, `.hotato/`: ``` refund-cutoff-001.hotato/ contract.json # the contract itself (schema hotato.contract.v1) audio/event.wav # the (clipped) two-channel recording, or the mono file evidence/ frames.jsonl # per-frame timing evidence behind every measurement timeline.html # the to-scale caller/agent timeline, self-contained trust.json # the input-health (trust doctor) report card.svg # a self-contained 1200x630 SVG card (redacted by default) traces/ # empty until `hotato trace attach` (see docs/TRACE.md) source/ call_metadata.json # redacted-by-default: stack, category, expect stack_config_snapshot.json # placeholder until populated by hand policy/verify.yaml # the SAME subset `hotato verify --policy` reads reports/ initial.html # the full scored report at creation time after.html # placeholder until a fix is re-captured and verified provenance.json # who/when/how this contract was created ci/ github-action.yml # a weekly + on-push CI scaffold junit.xml # this ONE contract's JUnit result at creation time ``` Every path above is also recorded, machine-readable, in `contract.json["bundle"]["paths"]`. ## Create From a candidate a sweep or scan already surfaced: ```bash hotato sweep --demo --format json > hotato-sweep.json hotato contract create --from-candidate hotato-sweep.json#1 \ --expect yield --id refund-cutoff-001 --out contracts ``` From a raw two-channel recording you already have: ```bash hotato contract create --stereo bad-call.wav --onset 42.18 \ --expect yield --id refund-cutoff-001 --out contracts \ --max-talk-over 0.6 --max-time-to-yield 1.0 ``` Both forms wrap the SAME round-trip guarantee `hotato fixture create` gives: the moment is scored immediately, and a not-scorable input (the agent silent at the onset, an unreadable file, a bad channel map) is refused with a clear reason and exit code 2 -- no bundle is written. A single-channel (mono) recording passed as `--stereo` is rejected the same way `fixture create` rejects it: caller and agent cannot be told apart on one channel. `--caller FILE --agent FILE` (two mono WAVs) is a third input form, scored and clipped identically. ### Redaction By default the bundle and card hide a candidate ref and a source recording's basename. Pass `--include-identifiers` to show them (in `source/call_metadata.json`, `contract.json`, and `evidence/card.svg`). ### The opt-in diarized-mono path A single-channel recording can still become a contract, through the SAME quality-gated diarizer front-end `hotato run --mono --diarize` uses: ```bash hotato contract create --mono call.wav --diarize \ --expect yield --id refund-cutoff-002 --out contracts ``` This keeps an indicative-only verdict in its tier: a `low`-confidence separation tier carries `measurement.indicative_only: true` into `contract.json` and every renderer; a `refuse` tier (not two clean parties, extreme overlap, unstable segmentation, voices too similar) is refused exactly like a plain mono file, with the specific reason. This evidence lives in `evidence/trust.json`'s separation confidence tier: `evidence/timeline.html` points there instead of the frame-level to-scale timeline (`evidence/frames.jsonl`) a dual-channel contract carries. ## Verify ```bash hotato contract verify contracts/ hotato contract verify contracts/ --format json --junit contracts-junit.xml hotato contract verify contracts/refund-cutoff-001.hotato --html verify.html ``` `DIR` is a contracts directory (every `*.hotato` subdirectory carrying a `contract.json`) or one bundle directly. For each contract, `verify` re-scores the SAME bundled audio against the SAME policy in its own `contract.json` -- this is what changes after an engine upgrade, a threshold change, or a re-captured recording swapped into `audio/event.wav` -- and reports pass/fail per contract and overall. Exit codes are the CI contract: | Exit | Meaning | | ---: | --- | | `0` | every contract passes | | `1` | at least one regressed (or is no longer scorable), or an embedded assertion deterministically FAILed (below -- it still counts toward this exit) | | `2` | usage error, empty directory, or corrupt `contract.json` | `--junit` writes one `` per contract; the shipped `ci/github_action.yml` scaffold runs this on push, on PR, and weekly, and publishes the JUnit file as an artifact. Every text and HTML render of `verify` also prints, verbatim: *"This result re-measures stored evidence. It does not test the current agent."* A green CI run here means the evidence, policy, and scorer are still intact; checking today's deployed agent takes the fresh-recapture lane above. See [`docs/RECAPTURE.md`](RECAPTURE.md#claim-language-what-each-kind-of-evidence-lets-you-accurately-say) for exactly what each kind of evidence lets you claim. ### Embedded assertions (optional) A contract can carry its own `assertions` block (schema `hotato.contract.v1`; see `schema/contract.v1.json`) -- the same `{version, assertions}` document `hotato assert` reads from `assertions.yaml`. When present, `contract verify` evaluates it through the SAME `assert.v1` engine and reports it as a per-contract `assertions` field, separate from the timing pass/fail in `summary`/`passed`. Pass `--transcript FILE` (a plain JSON array of `{role, text, start, end}` turns, or hotato's own `{"segments": [...]}` shape) to give transcript context to `phrase`/`pii`/`policy` assertions; `tool_call` assertions read the bundle's own attached trace (`hotato trace attach`) if one exists. Missing context reports `INCONCLUSIVE`. A deterministic FAIL contributes to the batch's nonzero exit code exactly like a timing regression; the batch result also carries a separate `assertions_failed` count. See [`schema/assert.v1.json`](../src/hotato/schema/assert.v1.json) for the result shape and `hotato.assert_` for the five deterministic kinds. ## Inspect ```bash hotato contract inspect contracts/refund-cutoff-001.hotato hotato contract inspect contracts/refund-cutoff-001.hotato/contract.json --format json ``` ## Pack / unpack A bundle directory travels as one file: ```bash hotato contract pack contracts/refund-cutoff-001.hotato # -> contracts/refund-cutoff-001.hotato.pack (deterministic; a MANIFEST.sha256.json # of every member travels inside it) hotato contract unpack contracts/refund-cutoff-001.hotato.pack \ --out contracts/refund-cutoff-001.hotato # every member is verified against the packed sha256 manifest; any mismatch # (a corrupt or tampered archive) is refused (exit 2) and nothing partial is # left behind ``` Packing the SAME bundle directory twice produces byte-identical archives (sorted member order, fixed timestamps, and every other value written into each member's ZipInfo, including its `create_system` byte): a contract's pack is a pure function of its bundle contents, deterministic for a fixed hotato version. Byte-identical re-runs are verified in CI on Linux x86_64, Python 3.10, 3.11, and 3.12 -- see [VALIDATION.md](VALIDATION.md) Job 1. ### Security: unpack treats an archive as hostile input A `.hotato` archive travels between teams, so `contract unpack` verifies everything about it before trusting a single byte. Before or during extraction it refuses (exit 2) on any of the following, writing only inside a scratch temp directory removed on any failure: * path traversal (`..`), absolute paths, and Windows-style backslash / drive-letter paths (`C:\...`) in a member name; * symlink members and encrypted members; * duplicate member names; * any member not declared in its own `MANIFEST.sha256.json`; * more members than a legitimate bundle could plausibly need; * a declared or measured decompressed size past the cap (default 512 MiB; set `HOTATO_CONTRACT_MAX_UNPACK_BYTES` or pass `--max-bytes` to raise it for a trusted archive) -- checked against the bytes measured live during extraction, beyond what the archive's own (untrusted) size metadata claims; * a single member whose compression ratio is far beyond anything a legitimate bundle member produces (a zip-bomb signal), even under the total-bytes cap. The sha256-per-member check (above) still runs on top of all of this: a member can pass every hardening check and still be refused for not matching the packed manifest. ## CI The shipped `ci/github_action.yml` is the minimal wiring: ```bash uvx hotato contract verify contracts/ --junit contracts-junit.xml \ --format json > contracts-verify.json ``` Gate a pull request or a scheduled job on the process exit code; do not parse stdout to decide pass or fail. ## What a contract proves, precisely Hotato proves timing behavior against this explicit contract -- not authorization, identity, compliance, or policy safety. `verify` reports coincidence, not causation: a passing re-verify after a config change coincides with the change, distinct from a controlled experiment. See `docs/VALIDATION.md` and `docs/THREAT-MODEL.md`. ## Read more - Proving the CURRENT agent, not just the frozen recording: [`docs/RECAPTURE.md`](RECAPTURE.md) - The underlying regression-fixture primitive `contract create` wraps: [`docs/BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md) - Battery-scale before/after proof: [`docs/FIX-LOOP.md`](FIX-LOOP.md) - Input-health (trust doctor): [`docs/TRUST.md`](TRUST.md) · [`docs/TRUST-MATRIX.md`](TRUST-MATRIX.md) - Diarized-mono scoring: [`docs/DIARIZE.md`](DIARIZE.md) - PR cards (one measured moment as an image): [`docs/CARDS.md`](CARDS.md) - Attaching a voice trace (observability bridge): [`docs/TRACE.md`](TRACE.md) · [`docs/OTEL.md`](OTEL.md) - Audio-handling rules (what raw audio may contain, redaction limits, distribution): [`SECURITY.md`](../SECURITY.md#audio-handling) ================================================================================ FILE: docs/CI.md ================================================================================ # CI: gate a PR on turn-taking hotato scores turn-taking timing from call recordings, so a pull request carries a running score and fails when the agent gets slower to stop talking for an interrupting caller. The workflow runs offline with zero extra dependencies. ## One command writes the record CI renders `hotato start --demo` runs the whole loop offline and writes a Failure Record under `hotato-failure-record/` -- `failure-record.{json,md,html,svg}`: one caught failure with its evidence, in four paste-safe shapes. The Action below renders it per non-passing unit (its `records` output). The demo prints the headline, the Markdown and SVG share paths, and the verifier: ``` Conversation failed: Agent did not yield; measured talk-over was 2.66 s. Share in a PR: hotato-failure-record/failure-record.md Share as an image: hotato-failure-record/failure-record.svg Verify the record: uvx --from hotato==1.18.1 hotato record verify hotato-failure-record/failure-record.json ``` Preview it locally, then scaffold the durable gate into your own repository in one command: ```bash hotato init starter --stack generic --out . # stack-tuned instead: --stack vapi, retell, twilio, livekit, pipecat ``` That writes a CI workflow verifying `contracts/` and re-scoring `fixtures/` on every push and pull request (a no-op until your first one lands), plus a stack-tuned `hotato.yaml`, `contracts/`, and `fixtures/`. ## The root Action: run a committed suite from any repository The repository root ships a composite GitHub Action: a repository with no hotato source can run a committed suite, conversation test, or contract verification and gate on hotato's exit status. The default run is offline -- the pinned Action revision executes straight off PYTHONPATH on the standard library alone, reaching no package index, no model, no ASR, no Node tool, no external judge, and no secret. Composite Action since v1.4.0. Adopt the current release (v1.18.1), pinned by its full commit SHA; resolve the tag to its SHA first: ```bash git ls-remote https://github.com/attenlabs/hotato refs/tags/v1.18.1 ``` Then commit this workflow (replace the `attenlabs/hotato` pin with the SHA the command printed): ```yaml name: conversation QA on: pull_request: permissions: contents: read jobs: conversation-qa: runs-on: ubuntu-latest steps: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - id: hotato # Pin by full commit SHA (immutable); the comment names the release. uses: attenlabs/hotato@ # v1.18.1 with: suite: tests/voice/qa.suite.json agent: support-agent # Artifact upload is an explicit consumer step, pinned by full commit # SHA; the Action itself never uploads, comments, or notifies. - if: always() uses: actions/upload-artifact@834a144ee995460fba8ed112a2fc961b36a5ec5a # v4.3.6 with: name: hotato-conversation-qa path: ${{ steps.hotato.outputs.output }} if-no-files-found: error ``` `permissions: contents: read` is sufficient. The Action never posts a comment, uploads an artifact, or sends a notification -- retention and publication stay under your workflow's control. ### Inputs Exactly one of `suite`, `test`, `contracts` selects what runs (a workspace-relative committed path); `agent` names the agent under test. Everything else is optional: | Input | Meaning | |---|---| | `release` | Defaults to the commit SHA | | `output` | Defaults to `.hotato/results` | | `parallel` | Suite worker cap; never changes result bytes | | `transcript` / `trace` / `state` | Evidence files for a test run | | `gate-advisory` | Test runs only; passes `--gate-judge` so the model-judged rubric lane gates | | `hotato-version` | Which install to use -- see below | `hotato-version` pins the install to one of three forms: | Value | Effect | |---|---| | `action` (default) | Installs the pinned Action revision itself, `--no-deps`, no package-index egress -- exactly the revision your workflow pinned | | an exact version, e.g. `1.18.1` | `pip install --no-deps hotato==1.18.1` | | `preinstalled` | Skips installation (hotato is already on the runner) | A range or `latest` is refused, so the pin always names one exact revision. Suite policy lives in the committed suite file -- the Action never overrides a suite's `inconclusive_policy`. ### Outputs and the exit contract The step exit IS hotato's exit code, so the job gates without any extra step: | Exit | Meaning | |---|---| | 0 | every deterministic check passed under the committed policy | | 1 | a deterministic check or a `success.required` condition failed | | 2 | the verdict was withheld (refuse policy), or a usage/validation error | The machine JSON stays primary: | Output | Meaning | |---|---| | `output` | Results directory | | `suite-result` | The machine result JSON path | | `summary` | The rendered Markdown | | `records` | Failure Record directory when produced, else empty | | `exit-code` | As a string | | `status` | `pass`, `fail`, `inconclusive`, or `error`, read from the machine result -- presentation only | | `hotato-version` | The executed package version | The five-lane summary (Outcome, Policy, Conversation, Speech, Reliability) appends to the job page on pass and on failure, with the reproduce command and the evaluated check ids. A lane with no evaluated check renders NOT_RUN; a lane whose checks lack the evidence they require renders INCONCLUSIVE, never PASS. The model-judged rubric lane reports in its own advisory section with `gate enabled: true|false`, changes the exit only when `gate-advisory: true` was set, and reports ERROR when no local judge answers. The conformance fixture: [`tests/fixtures/action-consumer/`](../tests/fixtures/action-consumer/); `.github/workflows/tests.yml` runs the same consumer shape against the local checkout on every pull request (job `action-smoke`). ## Drop it in Copy [`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml) into your repository at the same path. That is the whole setup. On the next pull request it will: - install the package with `pip install .` - score the bundled barge-in suite to a JSON envelope - render a Markdown summary with [`scripts/pr_comment.py`](../scripts/pr_comment.py) - post or update one sticky comment (found by a hidden marker, so it stays a single comment across runs) - fail the job on any regression The sticky comment shows a pass/fail line, a per scenario table (expect, yielded, time to yield, talk over, result), and a short regressions section. The job needs `pull-requests: write` to post the comment -- the workflow already requests it; if your org restricts the default `GITHUB_TOKEN`, grant that scope. ## Point it at your own recordings The bundled suite is a self-test: it scores frozen synthetic fixtures, proving the harness works. The strongest gate for your agent is a suite of your OWN bad moments, pinned with `hotato fixture create` and run with `hotato run --scenarios DIR --audio DIR`; full loop: [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md). Or replace one step, `Score turn-taking (head)`, with your own capture and score: 1. play each corpus `*.caller.wav` into your agent 2. record your agent's reply 3. score the pair, caller on channel 0 and your agent on channel 1: ```bash hotato run --stereo your_call.wav --expect yield --format json --no-fail > head.json ``` The envelope shape and exit codes are identical, so render, comment, and gate steps stay exactly as they are. Keep `--no-fail` on the score step so the comment still posts on a regression -- the true pass/fail lives in the envelope's `exit_code`, which the `Fail on regression` step reads. ## Deltas against the base branch When the workflow can install and score the base branch, it runs the same suite there in an isolated venv as the baseline. Any scenario where overlap grew (talk-over up) or the agent stopped later (time to yield up) lists under Regressions with the delta. Best effort: when it can't run, the comment falls back to the current pass/fail table and the gate still holds. ## Gate timing drift against a saved baseline `hotato baseline check` turns those deltas into a hard gate. Commit a tolerance file naming how much each timing dimension may rise, save a baseline run envelope, and the check exits 1 the moment a candidate run drifts beyond a tolerance: ```yaml # tolerances.yaml -- how much increase each dimension may absorb response_gap_sec: "+10%" # percent of the baseline mean seconds_to_yield: "+0.05" # absolute seconds ``` ```bash hotato run --suite barge-in --format json --no-fail > baseline.json # once, on main hotato run --suite barge-in --format json --no-fail > candidate.json # per PR hotato baseline check tolerances.yaml baseline.json candidate.json --junit drift.xml ``` Dimensions: `seconds_to_yield`, `talk_over_sec`, `response_gap_sec`, `premature_start_sec`; each side's value is the pooled mean across its scorable events. Every dimension is lower-is-better timing, so the gate is one-sided: an improvement always passes, and a dimension with no measurements on a side refuses (exit 2) rather than passing silently. `--format json` emits the machine envelope with the per-dimension deltas; `--junit drift.xml` feeds the same CI test widgets as `hotato contract verify --junit`. Exit codes: 0 within tolerance, 1 drift beyond tolerance, 2 usage error. ## Feed the CI test widget: `--junit` `hotato suite run --junit suite.xml` and `hotato prove --junit proof.xml` write the same JUnit XML the contract and baseline gates emit, so one upload feeds GitLab's `artifacts.reports.junit`, Jenkins `junit`, Azure `PublishTestResults@2`, or CircleCI `store_test_results`: ```bash hotato suite run ci.suite.yaml --agent support-v3 --junit suite.xml hotato prove --contracts contracts/ --gauntlet --junit proof.xml ``` The file mirrors each command's grouping: one `` per dimension (suite run) or per evidence lane (prove), one `` per test or lane. A scored FAIL is a `` carrying the measured reason; an INCONCLUSIVE dimension, a `SIMULATOR_INVALID` run, or a refused lane is an `` with the refusal reason preserved -- never a `` and never a silent pass, so the dashboard shows red exactly when the exit code gates. The XML carries no timestamp and renders byte-identically on a repeat run. ## Render a comment yourself `scripts/pr_comment.py` is stdlib only and reads the same JSON the CLI emits, so you can preview the comment locally: ```bash hotato run --suite barge-in --format json | python3 scripts/pr_comment.py ``` Add `--base base.json` to include deltas against a saved baseline run. ## The other ready-made gate: pytest If your CI already runs pytest, one flag adds the same regression gate: `pytest --hotato-suite` scores the battery after your tests and fails the session on a regression. Point it at your own labelled sets with `--hotato-suite-scenarios` and `--hotato-suite-audio`. Details: [`PYTEST.md`](PYTEST.md). For richer artifacts on a gate failure, `hotato report` renders the visual report and `hotato team` tracks the trend ([`REPORTS.md`](REPORTS.md)). ## GitLab CI, Jenkins, Azure Pipelines, CircleCI: `hotato init ci` The gate is exit-code driven, so it runs in any CI. `hotato init ci` writes the one canonical config the chosen system reads: ```bash hotato init ci --system gitlab # .gitlab-ci.yml hotato init ci --system jenkins # Jenkinsfile hotato init ci --system azure # azure-pipelines.yml hotato init ci --system circleci # .circleci/config.yml ``` Every generated config does the same four things: install the pinned hotato release (the version that generated the file), verify `contracts/` with `hotato contract verify`, re-score `fixtures/` with `hotato run`, and publish the JSON reports plus the JUnit file. A regression exits non-zero and fails the pipeline; an empty `contracts/` or `fixtures/` directory is a normal starting state, and each gate is a no-op until your first one lands. - **GitLab CI**: one `hotato` job on `python:3.12`; `artifacts.reports.junit` feeds the merge request test widget, and the JSON reports upload with `when: always`. - **Jenkins**: a declarative pipeline on a `python:3.12` docker agent (swap in `agent any` on a controller with Python 3.10+); `junit` and `archiveArtifacts` publish in the `post { always }` block. - **Azure Pipelines**: `UsePythonVersion@0` plus script steps; `PublishTestResults@2` feeds the Tests tab and `PublishBuildArtifacts@1` keeps the JSON reports, both on `always()`. - **CircleCI**: a `cimg/python:3.12` job wired into a workflow; `store_test_results` feeds the Tests tab and `store_artifacts` keeps the JSON reports. `--out DIR` writes into another directory (default `.`, the repo root where each system looks for its config); `--force` overwrites an existing file. The pinned version is the hotato that generated the file; bump it in one place when you upgrade. ================================================================================ FILE: docs/MCP.md ================================================================================ # MCP: the hotato server hotato ships an MCP server, `hotato-mcp`, over stdio, with fifteen tools: one scoring tool, `voice_eval_run` (returns the identical JSON envelope, `schema_version` "1", the CLI emits); three counterexample tools that reduce a failing scenario to a smaller case that still fails, then re-check it offline; and eleven fleet tools -- eight that read and propose over a local workspace, three that recompute inside a throwaway clone and hand the deploy decision back to you. Everything, audio included, stays on your machine. Every tool response carries a uniform control envelope: four keys, pure reads included, so an autonomous caller parses one shape: `evidence_status` (or null for a pure read with no verdict), `refusal_reason` (or null), `artifact_digests` (a list, or `[]`), and `pending_irreversible_action` (the exact human-gated action still pending, e.g. deployment approval, or null). ## Run it (zero-install) ```bash uvx --from "hotato[mcp]" hotato-mcp ``` **Common mistake:** `uvx hotato-mcp` (no `--from`) FAILS -- uv looks for a package literally named `hotato-mcp` on PyPI, which does not exist; the console script lives inside the `hotato` distribution's `mcp` extra. Hit that "package not found" error? Retry with `--from "hotato[mcp]"` exactly as above. Already have hotato installed? `python -m hotato.mcp_server` works too, no extra syntax needed. ## Add it to a client In a project checkout, one command registers hotato with every agent config surface the project already carries, including the `.mcp.json` server entry when that file exists: ```bash hotato init --agents ``` Every block below is copy-paste exact; only the outer key (client-specific) differs -- all three use the same command and args. ### Claude Desktop Edit `claude_desktop_config.json` (Settings -> Developer -> Edit Config): ```json { "mcpServers": { "hotato": { "command": "uvx", "args": ["--from", "hotato[mcp]", "hotato-mcp"] } } } ``` ### Cursor Project-scoped: `.cursor/mcp.json` in your repo root. User-scoped: `~/.cursor/mcp.json`. ```json { "mcpServers": { "hotato": { "command": "uvx", "args": ["--from", "hotato[mcp]", "hotato-mcp"] } } } ``` ### Codex CLI Edit `~/.codex/config.toml`: ```toml [mcp_servers.hotato] command = "uvx" args = ["--from", "hotato[mcp]", "hotato-mcp"] ``` ## The scoring tool: `voice_eval_run` Same two input modes as the CLI, all parameters optional beyond your chosen mode: | Parameter | Type | Default | Meaning | | --- | --- | --- | --- | | `stereo` | str | None | two-channel WAV path | | `caller` | str | None | mono caller WAV (with `agent`) | | `agent` | str | None | mono agent WAV (with `caller`) | | `suite` | str | None | `"barge-in"` to run the bundled battery | | `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic | | `expect` | str | `"yield"` | `"yield"` or `"hold"` | | `onset_sec` | float | None | caller onset hint | | `caller_channel` | int | 0 | caller channel index | | `agent_channel` | int | 1 | agent channel index | | `max_talk_over_sec` | float | None | pass threshold | | `max_time_to_yield_sec` | float | None | pass threshold | | `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) | Pass exactly one input mode: `stereo`, OR `caller` + `agent` together, OR `suite`. Mixing modes (or passing none) returns a structured, parseable error with a stable `error_code` and message; see below. ## The counterexample tools These three tools call the same stdlib-only core as `hotato counterexample ...`. A capsule is one reduced failing case on disk. `HOTATO_MCP_INPUT_DIR` confines scenario, test, and capsule reads; `HOTATO_MCP_REPORT_DIR` confines new capsule writes; with neither set, both stay under the working directory or the OS temp directory. Compilation never overwrites an existing path. | Tool | Scope | Does | | --- | --- | --- | | `counterexample_compile` | local write | reduces one failing scripted `hotato.scenario` + conversation-test target into a new private `.hotato-repro` capsule | | `counterexample_verify` | read-only | audits source bytes, evaluator provenance, the replayed delete-only chain, the source-selected structured failure branch, and a completed local-minimality claim | | `counterexample_reproduce` | read-only | runs only the reduced fixture under the current evaluator, permitting evaluator-version drift while still requiring the source-selected structured failure branch | `counterexample_compile` takes `scenario_path`, `test_path`, `target`, `out_dir`, optional `budget` (candidate evaluations), and optional `seed`. The compiler selects the base scenario at that seed and does not expand `variation_matrix`. `counterexample_verify` and `counterexample_reproduce` each take one `path`. The evidence status is `asserted`: this path operates on a deterministic scripted simulation, not measured call audio. Every response still carries the uniform control envelope; no tool publishes a capsule or promotes it into a corpus. ## Output On success: the identical envelope the CLI emits for `--format json`, schema at [`https://hotato.dev/schema/envelope.v1.json`](https://hotato.dev/schema/envelope.v1.json) (`schema_version` "1", additive-only -- new keys may appear, the documented core never changes). Every expected failure (missing / mono / mismatched / not-found file, unknown suite, ambiguous input mode, or a well-formed input with no scorable event) comes back as a structured error object, `ok: false`, with a stable `error_code` and actionable `message` -- schema at [`https://hotato.dev/schema/error.v1.json`](https://hotato.dev/schema/error.v1.json). The MCP tool and CLI share this error shape: a caller, human or agent, parses one contract across both surfaces. ## The fleet tools Eleven tools drive the fleet workspace ([`GUARDIAN-FLEET.md`](GUARDIAN-FLEET.md)): your local store of agents, calls, contracts, and before/after trials; a person labels the findings and deploys. | Tool | Scope | Does | | --- | --- | --- | | `fleet_status` | read-only | workspace counts + job-queue stats | | `candidate_list` | read-only | top candidate moments awaiting a human label | | `candidate_inspect` | read-only | one candidate's detail: onset, measured components (severity/input_health/recurrence/novelty/covered_by_contract), and trust findings | | `contract_list` | read-only | contracts in a workspace | | `trial_explain` | read-only | a recorded trial's verdict, evidence tier, and any pending human-gated action | | `experiment_status` | read-only | a trial's current verdict, evidence tier, recommendation, and manifest hash | | `artifact_verify` | read-only | verifies a contract bundle's authenticity + evidence on its own terms | | `experiment_propose` | read-only | a bounded variant set with expected effects, ready for a human to clone, apply, or deploy | | `experiment_create` | clone-scoped | precommit a trial manifest from a committed battery BEFORE any capture; capture and deploy stay separate, later steps | | `experiment_run` | clone-scoped | recompute a before/after trial offline, entirely within the clone, and record a recommendation | | `clone_cleanup` | clone-scoped | delete a STAGING clone an experiment created, scoped entirely to that clone | ## More - Python API and the shared error contract: [`API.md`](API.md) - What the envelope measures and the scope/ceiling: the top-level [`README.md`](../README.md) and [`METHODOLOGY.md`](../METHODOLOGY.md) - Machine-readable index of every command (CLI and MCP): [`llms.txt`](../llms.txt) ================================================================================ FILE: docs/CONNECT.md ================================================================================ # Connect, pull, sweep: score every call across your stack `hotato sweep` connects once and ranks the turn-taking moments across your recent calls, in three steps you can also run alone: 1. **`hotato connect `** -- store one stack's credentials, locally. 2. **`hotato pull`** -- bulk-fetch your recent recordings into a folder. 3. **`hotato sweep`** -- pull, then score every pulled call and write one offline dashboard ranking the turn-taking moments it measured. Everything scores offline; the only network call is the direct recording download from vendor to machine. Your audio and keys stay between you and the vendor. ## 1. Connect once ``` hotato connect vapi --api-key ``` Runs a live auth check (lists one recent call), then writes the credentials to `~/.hotato/connections.json` (mode `0600`, dir `0700`). The key never prints and reaches only the vendor's API. Credentials also fall back to the environment: ``` VAPI_API_KEY= hotato connect vapi ``` Per-stack credentials: | Stack | Flags (or env vars) | | --- | --- | | vapi | `--api-key` / `VAPI_API_KEY` | | retell | `--api-key` / `RETELL_API_KEY` | | twilio | `--account-sid` `--auth-token` / `TWILIO_ACCOUNT_SID` `TWILIO_AUTH_TOKEN` | | bland | `--api-key` / `BLAND_API_KEY` | | elevenlabs | `--api-key` / `ELEVENLABS_API_KEY` | | synthflow | `--api-key` / `SYNTHFLOW_API_KEY` (+ `--model-id` / `SYNTHFLOW_MODEL_ID` to list) | | millis | `--api-key` / `MILLIS_API_KEY` (+ `--base-url` for the EU region) | | cartesia | `--api-key` / `CARTESIA_API_KEY` (+ `--agent-id` / `CARTESIA_AGENT_ID` to list) | Retell has no list endpoint, so `connect retell` stores the key and validates it on first pull. `--no-verify` skips the live check for any stack. LiveKit and Pipecat capture in your own infra: connect with `hotato setup --stack livekit|pipecat` instead. Once connected, `--stack` and credential flags are optional for `pull` and `sweep` when exactly one stack is connected. ## 2. Pull recent recordings ``` hotato pull --stack vapi --since 7d --limit 50 hotato pull # the only connected stack ``` Lists recent recordings via the vendor's verified endpoint, then downloads each into `hotato-pull-/` (override `--out DIR`). `--since` accepts `7d`, `12h`, `30m`, `2w`. A recording that fails to fetch is a clean skip with its reason; the pull continues. A pull in which **every** listed recording failed to fetch exits 2 -- an outage, not a completed pull -- so a cron or CI wrapper reading only the exit code sees it red. **Retell has no verified list endpoint.** Pull it from explicit ids instead: ``` hotato pull --stack retell --call-id c1 --call-id c2 ``` For Twilio, `--call-id` values are Recording SIDs (`RE...`). **Mono / mixed stacks** (bland, elevenlabs, synthflow, millis, cartesia) produce one combined recording with no per-party channel; attributing talk-over needs `--allow-mono`, and results stay indicative only: ``` hotato pull --stack bland --allow-mono --limit 20 ``` ## 3. Sweep = pull + analyze ``` hotato sweep --stack vapi --since 7d hotato sweep # the only connected stack ``` Pulls (as above) into `hotato-sweep-/` (override `--dir`), runs the same `analyze` scoring with nothing to configure, and writes one self-contained offline HTML dashboard (`hotato-sweep-.html`, override `--out`): every call's candidate moments ranked, with an inline player on the top ones. `--format json` emits the candidates plus a pull summary instead. Dual-channel stacks get separated scoring. Mono/mixed stacks need `--allow-mono`; without per-party separation their calls land in the dashboard's Skipped section. A candidate is one measured timing moment you review and label with `hotato fixture create`: a timestamp and a magnitude, not a verdict on intent. ## Which stacks pull, and how See [`docs/ADAPTER-STATUS.md`](ADAPTER-STATUS.md) for the full map: which stacks auto-pull dual-channel, which arrive as one mixed channel, which capture in your own infra, and which endpoint each uses. ## After the first catch You have caught a timing moment on a recorded call. The next move is originating one against your live agent on demand, into this same pull -> score path: [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md). ================================================================================ FILE: docs/ADAPTER-STATUS.md ================================================================================ # Adapter status Which stack Hotato pulls recordings from, which endpoint it calls, and how it separates caller/agent channels for each one -- each entry read off that vendor's own API documentation on the date shown. Terms: - **auto-pull**: Hotato fetches the recording itself with your API key (`hotato capture` / `hotato pull` / `hotato sweep`). - **capture-in-your-infra**: there is no vendor recording API; Hotato prints the recording config (`hotato setup`) and scores the file your deployment writes. - **mono / mixed**: a single combined channel. Attributing overlap to caller or agent needs separated tracks, so Hotato scores a combined channel only behind an explicit `--allow-mono` / `HOTATO_ALLOW_MONO=1` opt-in, labeled indicative only. Separated turn-taking analysis needs a dual-channel file. Each adapter ships only the endpoints its vendor's spec confirms. Where a list-calls endpoint or channel layout reads **unconfirmed / none**, Hotato takes explicit call ids instead, and this page names which and why. ## Dual-channel: auto-pull, separated scoring One entry per stack, verified 2026-07-28 against each vendor's current public API documentation unless noted otherwise. Each auto-pulls a recording with per-party channels, so Hotato scores full separated turn-taking without `--allow-mono`. - **Vapi** - List recent calls: `GET https://api.vapi.ai/call` (params `limit`, `createdAtGt/Lt`) → JSON array of Call objects (`id`, `createdAt`) - Fetch recording: `GET /call/{id}` → `artifact.recording.stereoUrl` (current); deprecated fallbacks `artifact.stereoRecordingUrl`, `call.stereoRecordingUrl` - Channel basis: `stereoUrl` is a distinct 2-channel file (customer ch0, assistant ch1) - **Twilio** - List recent calls: `GET .../Accounts/{Sid}/Recordings.json` (params `PageSize`, `DateCreated>`/`DateCreated<`, `CallSid`) → `recordings[].sid` - Fetch recording: `GET .../Recordings/{RE...}.wav?RequestedChannels=2` (HTTP Basic) - Channel basis: two-party recordings are stored dual-channel by default, so `RequestedChannels=2` returns per-leg audio (ch0 = caller, ch1 = agent); the download format with the parameter omitted follows what was requested at creation. 400 → clean stop or `--allow-mono` → `RequestedChannels=1`. Accounts on Encryption with Public Key or External Storage receive mono regardless. Conference recordings are NOT per-party: ch0 is the first participant that joined with recording enabled and ch1 mixes everyone else, so a conference pull carries two channels without carrying two parties - **Retell** - List recent calls: Retell documents `POST https://api.retellai.com/v3/list-calls` (Bearer, body `limit`/`pagination_key`/`filter_criteria`, items carry `call_id`); hotato pulls from an explicit `--call-id` until that lister is wired (issue #58) - Fetch recording: `GET https://api.retellai.com/v2/get-call/{id}` (Bearer) → `scrubbed_recording_multi_channel_url` preferred, then `recording_multi_channel_url`; plain mono `recording_url` rejected unless `--allow-mono` - Channel basis: per-party channels on the `*_multi_channel_url` fields - **LiveKit** - List recent calls: **capture-in-your-infra** -- `ListEgress` is an RPC method name only; no REST list. `hotato setup --stack livekit` - Fetch recording: two audio-only Track egresses (one per party); recording location in `egress_info.file_results[].location` - Channel basis: separated by running one Track egress per participant - **Pipecat** - List recent calls: **capture-in-your-infra** -- Pipecat Cloud session-list has no recording field; OSS has no list at all - Fetch recording: `AudioBufferProcessor(num_channels=2)` in-pipeline (user left, bot right); you write the WAV - Channel basis: 2-channel in-pipeline ## Mono / mixed: auto-pull, `--allow-mono`, indicative only Each has a verified list + fetch, but the vendor produces a single combined recording with **no documented per-party channel**. Hotato scores it indicative-only, behind `--allow-mono`. One entry per stack, verified 2026-07-07. - **Bland AI** - List recent calls: `GET https://api.bland.ai/v1/calls` → `calls[].call_id` - Fetch recording: `GET /v1/calls/{id}` → `recording_url` - Mono basis (verbatim): no stereo/channel field in the recording or call-details schema - **ElevenLabs Conversational AI** - List recent calls: `GET https://api.elevenlabs.io/v1/convai/conversations` (params `page_size`, `call_start_after_unix`) → `conversations[].conversation_id` - Fetch recording: `GET /v1/convai/conversations/{id}/audio` → combined audio - Mono basis (verbatim): docs state the audio "does not include separate caller/agent channels, only a combined full conversation MP3" - **Synthflow** - List recent calls: `GET https://api.synthflow.ai/v2/calls?model_id=…` (params `limit`, `from_date` epoch-ms) → `response.response.calls[].call_id` - Fetch recording: `GET /v2/calls/{id}` → `response.response.calls[0].recording_url` (a Twilio Recordings URL) - Mono basis (verbatim): Synthflow documents no dual-channel option; `recording_url` is a bare Twilio URL - **Millis AI** - List recent calls: `GET {base}/call-logs` (US `api-west`, EU `api-eu-west`; param `limit`) → `histories[].session_id` - Fetch recording: `GET /call-logs/{session_id}` → `recording.recording_url` - Mono basis (verbatim): schema exposes only `recording_url`; `CallSettings` has only a boolean `enable_recording` - **Cartesia (Line)** - List recent calls: `GET https://api.cartesia.ai/agents/calls?agent_id=…` (param `limit`; needs `Cartesia-Version` header) → `data[].id` - Fetch recording: `GET /agents/calls/{id}/audio` → `audio/wav` - Mono basis (verbatim): dual-vs-mono **unconfirmed** in the docs; treated as mono until a live channel-count check proves otherwise `--stack synthflow` needs `--model-id` (its list endpoint requires `model_id`); `--stack cartesia` needs `--agent-id`; `--stack millis` accepts `--base-url` for the EU region. Single-call `capture`/`pull` by explicit id needs none of these. ## No vendor recording to pull - **Deepgram Voice Agent API** (2026-07-07, confirmed absent) -- real-time WebSocket (`wss://agent.deepgram.com/v1/agent/converse`), no REST list-calls, fetch-recording endpoint, or recording-ready webhook: the call is never stored vendor-side. Capture it in your own infra instead (the LiveKit/Pipecat pattern above). - **PlayAI (formerly Play.ht)** (2026-07-07, dead endpoints verified) -- the conversational-agent product is retired: `play.ai` DNS does not resolve, `docs.play.ai` returns `DEPLOYMENT_NOT_FOUND`. The only live domain, `docs.play.ht`, is TTS-only, no calls/recordings API. ## Fallback routes and open checks Where the public docs pin down no recording field-path or channel layout, the stack ships as a fallback route -- explicit call ids, or the vendor's own webhook URL. The last three are open checks on stacks shipped above. All of it is listed here so the supported set stays exact. - **Regal.ai**: no list-calls, no REST fetch-recording endpoint. The recording arrives only via the `call.recording.available` webhook's `properties.recording_link`, a literal Twilio Recordings URL -- feed it through Hotato's existing Twilio path, the one shipped route for Regal-sourced recordings. - **Thoughtly**: OpenAPI types responses as an untyped `GenericResponse.data` blob; no recording-URL field path could be confirmed live. - **Sindarin**: no Sindarin-hosted recording endpoint found; `record` is submitted with the caller's own Twilio creds, so the recording lands in the customer's own Twilio account (use the Twilio path). - **xAI Grok Voice Agent Builder**: OpenAI-Realtime-compatible; no documented list-calls or fetch-recording endpoint (marketing mentions in-product recording, but no confirmable API contract). - **Cartesia dual-channel**: the `/agents/calls/{id}/audio` WAV's channel count was not confirmable in docs; treated as mono here pending a live check. - **Twilio dual-channel edge behaviour**: the exact 400-on-unavailable and conference channel-ordering guarantees stand on prior notes, carried forward rather than re-read from the spec this pass. - **Synthflow / Regal Twilio-URL dual-channel trick**: because their `recording_url` is a Twilio URL, `?RequestedChannels=2` *might* yield stereo, but neither vendor documents or guarantees it; unconfirmed until tested against a live account. Also researched, login-gated or SPA-only docs or transcript-only APIs: Daily, Ultravox, Hume EVI, Telnyx, Infobip, OpenAI Realtime, Amazon Connect, Parloa, Sierra, Decagon, PolyAI, Voiceflow, Cognigy, Dialogflow CX, Genesys Cloud, NICE CXone. The integration spec carries the per-platform facts and gaps. ## What holds across every adapter Every adapter validates that a dual-channel file has one party per channel (2 channels) before producing separated talk-over numbers; mono stacks score degraded, only behind `--allow-mono`. Every live fetch/list path is stdlib-only (`urllib`); stack SDKs import lazily, only inside your own infra. Credentials from `hotato connect` sit locally at `~/.hotato/connections.json` (mode 0600), going straight from your machine to the vendor's API -- Hotato stays out of that exchange. Scoring runs offline; the only network call is the recording download itself. ================================================================================ FILE: METHODOLOGY.md ================================================================================ # Methodology How every number this tool reports is computed, why it is reproducible, and where the method stops being trustworthy. There is no accuracy percentage in this document and none is implied anywhere in the tool. What follows describes a measurement, not a claim about any detector's internal quality. Hotato scores the audio *timing* of turn-taking from a call recording: whether the agent stopped talking once the caller started (a yield), how many seconds that took, how many seconds it kept talking while the caller was talking (talk-over), and, on the same two tracks, the endpointing timing: how the agent's response sits around the moment the caller finished speaking. It measures energy over time. It does not understand speech. ## Read this first: the honest ceiling The limits come first because they bound everything below. - **Energy is not intent.** The detector marks a frame "active" when its short-time energy crosses a threshold. A cough, a slammed door, or a burst of line noise reads as active exactly like a word does. Nothing here recovers *meaning*, it recovers *when there was speech-level energy*. - **You supply the label.** Hotato does not infer intent. You label the expected behavior for the event: yield means the agent should stop for the caller. hold means the agent should keep speaking through a backchannel/noise/acknowledgement. Hotato then measures whether the timing matched that label. "mhm" and "stop" can carry identical speech energy; the verdict is only ever your label checked against measured timing. - **Sub-second single-channel boundaries sit near the resolution limit.** At the default 10 ms hop a boundary is located to within a frame or two, but the hangover that keeps an utterance whole (default 150 ms) smears every trailing edge by up to that much, and on one mixed channel deciding *whose* energy crossed the threshold is not always possible. - **Two channels is the gold reference; mono is scorable, quality-gated.** The reference input is separated caller and agent tracks: one two-channel WAV or two aligned mono WAVs, where each channel is one party and overlap is a fact of the recording (both tracks active at once, by construction). A single mixed mono call is scorable via the opt-in `[diarize]` front-end (`hotato run --mono call.wav --diarize`): a diarizer separates the mix into caller/agent activity, which is reconstructed and scored through the same path. It is quality-gated -- above the confidence bar the verdict is labeled `diarized-mono`; below it, the verdict is labeled indicative only and no SLA gate fires; a non-separable file is refused. Diarized mono never equals a true dual-channel recording for sub-second talk-over attribution, and the gate is what keeps that honest per file. - **Out of scope, permanently:** no speaker identification (a diarizer assigns anonymous SPEAKER_00/01; it never says who a person is), no speech-to-text, no emotion or intent detection, and no claim about any vendor's internal accuracy. Word-level semantics are not measured; only the timing of the yield is. These same limits are emitted inline in the `limits` block of every JSON result (`core.py`) and in the MCP tool description, so a consuming agent sees them. ## The pipeline, step by step Everything runs offline, standard-library only (NumPy only accelerates the per-frame RMS when importable; results are identical without it). Scoring is deterministic: the same audio and `ScoreConfig` give the same output. ### Default parameters Every value below is read directly from `ScoreConfig` (`score.py`) and `VADParams` (`vad.py`); the same defaults apply to both channels unless overridden. Framing and VAD (`VADParams`, applied per channel): | Parameter | Default | Meaning | |--------------------|-----------|---------| | `frame_ms` | 20.0 ms | analysis window length for one RMS frame | | `hop_ms` | 10.0 ms | step between frames (the time resolution) | | `noise_percentile` | 0.10 | quietest 10% of frames estimates the noise floor | | `rel_db` | 15.0 dB | a frame is active this many dB above the floor | | `abs_gate_db` | -60.0 dBFS| nothing below this absolute level is ever active | | `dyn_margin_db` | 22.0 dB | on a rarely-silent channel, hold the threshold at least this far below the loudest frames | | `hangover_sec` | 0.15 s | stay "active" this long after energy drops | Signal windows (`ScoreConfig`): | Parameter | Default | Meaning | |---------------------------|---------|---------| | `yield_hangover_sec` | 0.20 s | agent must stay quiet this long to count as yielded | | `max_search_sec` | 3.0 s | how long after onset a yield is searched for | | `caller_proximity_sec` | 0.5 s | a yield only counts if the caller held the floor within this window of it | | `turn_end_silence_sec` | 0.20 s | caller must stay quiet this long for the turn to count as ended | | `premature_tolerance_sec` | 0.05 s | agent may lead the caller's turn end by up to this before it counts as premature | | `onset_min_run_sec` | 0.05 s | minimum sustained active run for the caller onset to count | | `agent_onset_lookback_sec`| 0.10 s | window before onset checked for whether the agent was already talking | Both onset-detection windows are exposed on `ScoreConfig` as well: `onset_min_run_sec` (the `0.05 s` sustained active run a caller onset must clear, applied by `first_active_sec`) and `agent_onset_lookback_sec` (the `0.10 s` window before onset checked for "was the agent talking at onset?"). Neither affects the per-frame active/threshold decision the frame dump records. Every threshold the scorer uses is a named, overridable parameter -- there are no hidden constants in the scoring path. ### Step 1, per-frame RMS Each channel is cut into 20 ms windows stepped by 10 ms. For each window the linear root-mean-square amplitude is computed (`frame_rms`, `audio.py`). At 16 kHz this is a 320-sample window with a 160-sample hop, so `hop_sec` is exactly `160 / 16000 = 0.01 s`; it is derived from the integer sample hop over the sample rate, so it matches `hop_ms` exactly at common rates. ### Step 2, RMS to dBFS Each linear RMS value is converted to decibels relative to full scale: `dBFS = 20 * log10(rms)`, with a floor of `-120.0 dBFS` substituted before the log to avoid `log(0)` (`to_dbfs`, `audio.py`). This is the per-frame `*_dbfs` column in the frame dump. ### Step 3, per-channel energy VAD For each channel independently (`energy_vad`, `vad.py`): 1. Sort the frame dBFS values and take the `noise_percentile` (10th percentile) as the **noise floor**. 2. The base **threshold** is `max(noise_floor + rel_db, abs_gate_db)`, i.e. 15 dB above the floor, but never below the -60 dBFS absolute gate. 3. **Dynamic-margin guard:** if a channel is almost never silent (e.g. an agent talking the whole clip), the 10th-percentile floor lands *inside* speech and would push the threshold above the speech itself. So compute `cap = max(dBFS) - dyn_margin_db`; if `cap` is above the absolute gate, lower the threshold to `min(threshold, cap)`. This cannot rescue a genuinely silent channel, because the guard only fires when loud content exists above the gate. The cap is also the scorer's noise ceiling: once a channel's noise floor climbs to within `dyn_margin_db` of its loudest frame the verdict flips rather than degrades; `docs/BENCHMARK.md` ("Noise floor and the verdict cliff") gives the measured flip points and the opt-in `--snr-gate-db` scorability gate. 4. A frame is **raw-active** when its dBFS ≥ the threshold. 5. **Hangover:** after any raw-active frame, keep the channel active for `round(hangover_sec / hop_sec)` more frames, 15 frames (150 ms) at the defaults, so brief inter-word gaps do not fragment one utterance. The resulting threshold and noise floor are constant across frames for a given channel and are recorded verbatim in the frame dump. ### Step 4, caller onset If a caller onset label is supplied (scenario `caller_onset_sec`, or `--onset`), it is used directly. Otherwise the onset is the start of the caller's first sustained active run, the first run of at least 0.05 s of active frames (`first_active_sec`). The label is preferred because on mixed or noisy audio the first energy is not always the caller. `onset_idx` is `round(onset / hop_sec)`, clamped into range. No label and no detectable caller speech means there is no caller event to score: the recording is reported not scorable (`scorable: false` with a plain reason), never scored against frame 0, and a single-recording run exits 2. ### Step 5, the three barge-in signals Computed in `score_channels` (`score.py`): - **`agent_talking_at_onset`**, was any agent frame active in the 0.10 s up to and including onset? If not, on a should-yield expectation there is nothing to yield: the event is reported not scorable (`scorable: false`, with the reason in `not_scorable_reason`) rather than passed or failed, and a single-recording run exits 2. The input is what is wrong (onset time, channel mapping, or expectation), and it is reported as such. - **`did_yield`**, scanning from onset up to `onset + max_search_sec` (300 frames), the first frame where the agent goes quiet and *stays* quiet for `yield_hangover_sec` (20 frames) **and** the caller held the floor within `caller_proximity_sec` (±50 frames) of that quiet point. The proximity condition is what stops an agent that merely finishes its own sentence seconds after an isolated backchannel from being scored as a barge-in response. - **`time_to_yield_sec`**, `(yield_frame - onset_frame) * hop_sec`, floored at 0; `None` if the agent never yielded within the search window. - **`talk_over_sec`**, count of frames from onset to the yield point (or to the end of the search window if it never yielded) where the caller and the agent were *both* active, times `hop_sec`. Both activity tracks are the hangover smoothed tracks, so against a label placed at the raw end of speech energy the measured end sits late by at most `hangover_sec` plus one hop and the measured start sits early by at most one frame; the bias is deterministic and one sided, and `hangover_sec = 0` removes the hangover term (see docs/BENCHMARK.md, Quantization). ### Step 6, the two latency (endpointing) signals Pure timing on the same two VAD tracks, no second model (`score.py`): - **`caller turn end`**, the first frame at/after onset where the caller goes quiet and stays quiet for `turn_end_silence_sec` (20 frames). `None` if the caller never activates after onset or is still talking when the clip ends. - **`agent response onset`**, skip any agent speech already in progress at onset (the pre-caller turn or the yield tail), then the first frame the agent becomes active again. `None` if the agent never starts a fresh run. - With both defined, `lead = turn_end_frame - response_onset_frame` (positive when the agent starts *before* the caller finishes). If `lead` exceeds `premature_tolerance_sec` (5 frames), `premature_start_sec = lead * hop_sec` and `response_gap_sec` is null. Otherwise `premature_start_sec = 0.0` and `response_gap_sec = max(0, response_onset - turn_end) * hop_sec`. Both are null when not derivable from the tracks, never fabricated. Reported values are rounded to the millisecond (3 decimal places of seconds). ### Step 7, two additive, opt-in cross-checks: echo and resume Both live entirely in hotato's own layer (`echo.py`, `resume.py`), never in the vendored `_engine`, and both add an optional `signals` block without changing `did_yield`, `seconds_to_yield`, or `talk_over_sec` for any existing recording. - **`signals.echo`** (`echo.py`), computed on every scored event. Leaked TTS (an agent that hears its own audio back on the caller channel) is, by construction, a delayed, scaled copy of the agent's own envelope. The check is a deterministic cross-channel coherence: at each lag from 0 up to `DEFAULT_MAX_LAG_SEC` (0.5 s), take the cosine similarity between the caller frame-RMS envelope and the agent envelope shifted by that lag; `coherence` is the best (highest) cosine found, `lag_sec` is the lag it occurred at, and `echo_suspected` is `coherence >= DEFAULT_COHERENCE_THRESHOLD` (0.7) with at least `_MIN_OVERLAP_FRAMES` (8) of overlap. Independent speech (real turn-taking, backchannels) does not correlate with the agent's own envelope at any lag, so it scores low. `--echo-gate` (opt-in, `hotato run`) holds an echo-suspected yield out of the verdict (`scorable: false`) instead of counting it as a clean pass; without the flag the primary verdict is unchanged and `signals.echo` is still reported. `hotato diagnose` and the single-run text output print a WARNING for every echo-suspected yield. - **`signals.resume`** (`resume.py`), present only on events where the agent yielded. From the agent's own VAD track, look for a fresh onset within `DEFAULT_RESUME_WINDOW_SEC` (4.0 s) after the yield: `resumed` is whether one is found, `resume_gap_sec` is the seconds from yield to that onset (`null` if it did not resume). `restart_suspected` flags whether the longest contiguous agent run at or after the resume onset reaches `DEFAULT_RESTART_MIN_SEC` (2.0 s), the timing fingerprint of re-answering a whole paragraph from the top rather than finishing the interrupted clause. Whether the resumed words literally repeat the earlier ones is a transcript question, explicitly out of scope: `restart_suspected` is a run-length heuristic on timing, not a text diff. - **`agent_stop_no_caller`** (`scan.py`, not part of the scored envelope: a `hotato scan` candidate) surfaces the companion timing fact: the agent went from active to quiet with zero caller energy anywhere in `caller_proximity_sec` on either side, so nothing on the caller channel explains the drop. It is a candidate for self-truncation, endpointing mis-fire, or an upstream (LLM/TTS) cutoff, not a verdict; label it with `hotato fixture create --expect hold` (or `yield`, if that silence was in fact correct) to turn it into a scored fixture. ## Why it is reproducible - **Deterministic** given `(audio, ScoreConfig)`. No randomness, no network, no hidden state. The bundled fixtures are themselves rendered deterministically from a seed derived from `sha256(scenario_id)` (`render_examples.py`), and CI re-renders and diffs them to prove byte-stability. - **Every threshold that drives the active/threshold decision is a named, overridable parameter** on `ScoreConfig` / `VADParams`, and the resolved values are echoed back in the frame dump's `config` block. - **Every frame and every derived index is inspectable.** `--dump-frames PATH` writes, per frame: `t_sec`, `caller_dbfs`, `agent_dbfs`, `caller_active`, `agent_active`, and the constant `caller_threshold_db`, `caller_noise_floor_db`, `agent_threshold_db`, `agent_noise_floor_db` (`frame_dump`, `score.py`). Nothing the scorer decides is off the record. ### Worked example: re-deriving `did_yield` and `talk_over` by hand The excerpt below is illustrative of the dump shape; run `--dump-frames` on your own file for real values. Assume the defaults (`hop_sec = 0.01`), a caller onset label at `2.40 s` (`onset_frame = 240`), and these constant thresholds from the dump header: caller threshold `-55.0 dBFS`, agent threshold `-52.0 dBFS`. ``` frame t_sec caller_dbfs caller_active agent_dbfs agent_active 240 2.40 -18.4 True -21.1 True 288 2.88 -19.0 True -20.7 True 289 2.89 -20.2 True -48.9 True (hangover tail) 290 2.90 -21.5 True -71.3 False ... (agent dBFS stays below -52 through frame 310) ... 310 3.10 -20.8 True -69.4 False ``` Re-derive `did_yield`: agent was active at onset (frame 240), so the question is meaningful. Scanning from 240, the agent first drops below its threshold at frame 290 and stays below it for at least 20 consecutive frames (through 310), that satisfies `yield_hangover_sec`. The caller is active within ±50 frames of 290, so the proximity condition holds. Therefore `did_yield = True`, `yield_frame = 290`, and `time_to_yield_sec = (290 - 240) * 0.01 = 0.50 s`. Re-derive `talk_over`: count frames from 240 up to the yield frame 290 where `caller_active AND agent_active`. In this excerpt that is frames 240 through 289 inclusive, 50 frames, so `talk_over_sec = 50 * 0.01 = 0.50 s`. Every one of those booleans is just `dbfs >= threshold` (plus hangover), which you can recompute yourself from the two dBFS columns and the header thresholds. ## Aggregate statistics: one definition each The `report`, `team`, and `export` surfaces summarize many events with mean, median, p90, and p95. All four come from one stdlib implementation (`src/hotato/_stats.py`, `dist_summary`) so every published aggregate is re-derivable by hand: - **mean** is the arithmetic mean (`statistics.fmean`). - **median** is `statistics.median` (p50). - **p90** and **p95** are linear interpolation between closest ranks (the definition NumPy calls "linear"): sort the values ascending, let `pos = q * (n - 1)` for `q` in `{0.90, 0.95}`, then `p_q = v[floor(pos)] + frac * (v[floor(pos) + 1] - v[floor(pos)])`. - **Rates are fractions**, never a percentage: a pass rate of 8 of 8 reads `1.00`. That keeps every aggregate in the same unit system as the measurements and leaves no door open to an accuracy-percentage reading. - **Empty input returns null.** No aggregate is ever fabricated from zero measurements, and fewer than two runs is stated plainly rather than drawn as a trend. `team` and `export` pool `response_gap_sec` (dead air before the agent speaks, defined above) across every scored event into the same `dist_summary` shape already used for talk-over and time-to-yield. `--max-response-gap` turns the pooled p95 into a latency SLA gate: the run exits 1 exactly when the pooled p95 exceeds the bound, the same pass/fail contract as `--max-talk-over` and `--max-time-to-yield`, just pooled instead of per-event. A plain `hotato export` (no `--max-response-gap`) still writes a byte-identical `envelope.json` to a run with none of this stage's work applied; the p95 and gate live only in the printed summary and the returned manifest. ## Optional neural cross-check (non-reference) The energy VAD above is the **reference**: every published, golden, and bundled number comes from it, deterministically. Hotato also ships an **optional, opt-in, explicitly non-reference** neural VAD backend so you can re-run the *same* turn-taking timing math over a learned speech track and compare, a direct answer to "this is just energy VAD, rebuildable in a weekend." - **How to use it.** Install the extra and pass the flag on your *own* recording: `pip install 'hotato[neural]'` then `hotato run --stereo call.wav --backend neural`. The default is `--backend energy`. The `--suite` self-test **always** scores with energy (it is the reference), so `--backend neural` is ignored there. - **What it is.** Silero VAD (MIT), run locally/offline. The engine itself keeps zero third-party dependencies and never imports a model; the backend is injected behind one shared interface, so an open-weight turn-detector (e.g. LiveKit's Smart-Turn) can be plugged in the same way. It returns the **identical** `VADResult` shape as the energy backend, `active`, `hop_sec`, `threshold_db`, `noise_floor_db`, aligned to the same hop grid. A neural model has no dB threshold, so `threshold_db` / `noise_floor_db` are **synthesized**: they are the energy-domain description of the *same* audio, reported for inspection only; the neural `active` decision is a learned speech probability, not a dB crossing. - **What it changes, honestly.** It **tightens onset precision** on clean speech (a learned boundary can beat a fixed dB threshold on some audio). It does **not** close the energy-vs-intent gap: a cough, a laugh, a door slam, or crosstalk still carries speech-band energy, and *any* single-channel VAD, energy or neural, can mark it active. Whether a sound is a genuine bid for the turn is not decidable from one channel's activity alone. **No accuracy percentage is claimed for either backend.** Neural is a flagged cross-check, not a new source of truth. - **No silent fallback.** If the `[neural]` extra is absent, `--backend neural` raises a clean, explicit error and exits non-zero, it never quietly scores with energy and presents it as the neural result. - **Verification status: verified against the real model.** Executed in this repo with silero-vad 6.2.1 (ONNX weights bundled inside the package, inference through onnxruntime on CPU, fully offline; note silero-vad itself depends on torch, so the extra installs it). Properties that hold, measured over the bundled 8 and the 40-scenario gold suite: - **Contract holds end to end.** The real model returns the identical `VADResult` shape as energy, on the same hop grid, through the public scorer. - **Deterministic.** Two full sweeps over all 48 fixtures produced byte-identical JSON, for both backends; single recordings are likewise byte-identical run to run. - **Reference untouched.** With the extra installed, energy outputs are byte-identical to energy outputs without it, and the frozen bundled 8 verdicts are unchanged. - **On the synthetic fixtures the two tracks diverge, and that divergence is a property of the fixtures.** The renders are shaped noise built for the energy reference; a speech-trained model assigns them near-zero speech probability (identical through the ONNX and torch paths), so at Silero's default threshold the neural track is empty there and the timing math reports no yield on any of the 48. This measures the fixtures, not any agent. Point the neural cross-check at real recordings. - **On a real-speech recording** (a dual-channel barge-in fixture assembled from a recorded human utterance) both backends returned the same yield verdict; the measured yield timing differed (energy 0.65 s including its 0.15 s hangover, neural 0.18 s, because a speech model segments the word gaps that the energy hangover bridges). One recording, reported as a timing observation, not an accuracy claim. - **Sample rates.** Silero accepts 8000 Hz, 16000 Hz, and integer multiples of 16000 Hz (silero-vad decimates those itself and returns timestamps in the original sample coordinates). Any other rate fails with an actionable resample message from the seam. The energy backend measures at any rate. The real-model tests run automatically when the `[neural]` extra is installed and skip cleanly when it is absent (`tests/test_backend.py`). ## How validity is measured, not claimed The bundled fixtures are synthetic: band-limited, syllable-modulated noise rendered from exact segment boundaries declared in each scenario's `reference_render` block (`caller_segments_sec`, `agent_segments_sec`, and an optional `continuous` flag that renders one unbroken run per segment so the active track equals the rendered boundaries to within one hop). Because the true boundaries are known exactly, the scorer's **own measurement error** is computable, and that is what to report, rather than an accuracy score: - For a timing signal (onset, yield time, response gap), report `|measured - true|` in **milliseconds**, per scenario. The true value comes straight from the segment boundaries: true onset is the caller's first segment start; true yield time is the agent's segment end minus onset; true gap is the agent's next segment start minus the caller's turn end. Expect a residual on the order of the hop plus the hangover smear on trailing edges, read it, do not average it away. - For `did_yield`, report a **confusion matrix** against the scenario `category` labels (`should_yield` vs `should_not_yield`), four cells: correct yields, missed yields, correct holds, phantom yields. Keep the cells separate. **Never aggregate these into a single accuracy percentage.** A missed hard interruption and a phantom yield on a backchannel are different failures with different fixes; one blended number hides exactly the distinction the tool exists to surface. State plainly what the synthetic battery is for: it proves the plumbing works end-to-end and catches regressions (a threshold change that moves a number shows up immediately against the frozen golden output). It is **not** production audio , no real accents, codecs, packet loss, or room acoustics. For validity on your system, bring **10-15 of your own labelled calls** (two-channel where you can get it) and run them through the same `run` / `--dump-frames` path; see `CONTRIBUTING.md` and `docs/CORPUS-GOVERNANCE.md`. ## Say-do verification methodology Everything above describes the audio timing scorer. Say-do verification is the other deterministic lane: it checks what the agent *said* (transcript) against what the backend *did* (recorded evidence), and it reads recorded artifacts, never audio energy. - **Authority 1 is the trace.** `tool_call`, `tool_result`, `tool_error`, and `http_result` assertions read only the ingested `hotato.voice_trace.v1` spans (`hotato trace ingest`, `docs/TRACE.md`). A span is the evidence a tool ran or an HTTP exchange happened; an agent's own words claiming it happened can never satisfy these kinds. `outcome` combines span sub-predicates with transcript phrases into the say-do check itself: the agent said "your refund is on its way" *and* the `issue_refund` span is in the trace. - **Deterministic, no model.** Every `assert.v1` kind is a regex, checksum, or span/dict lookup (`docs/ASSERTIONS.md`); every result carries `deterministic: true`, and `run_assertions` is byte-stable across repeated calls on identical input. The model-judged rubric lane is structurally quarantined in its own count and never blends in. - **INCONCLUSIVE without evidence.** An assertion whose required input is absent (no trace, no transcript) reports `INCONCLUSIVE`, never a guess, and `inconclusive_policy: fail | refuse` turns that into a gate so missing evidence fails loudly in CI. **Worked example: the reference-agent suite.** `examples/reference-agent` is the runnable ground truth for this lane: a 375-run offline suite (25 scenarios, 5 caller behaviours, 3 audio environments) where each scenario's deterministic `agent_mock` renders `tool_call` spans (Authority 1) and a post-call state sandbox (Authority 2). Four scenarios carry seeded agent defects (a refund claimed but never issued, identity skipped before a lookup, an escalation never handed off, a declined payment handled wrong), and the suite surfaces each as an outcome or policy FAIL from trace and state evidence alone. Two seeded runs write byte-identical conversation artifacts, pinned by `tests/test_determinism_reference.py`. **Where the five dimensions stand.** The dimensions (outcome, policy, conversation, speech, reliability) are scored lanes: each Failure Record carries all five, each with its own status and never a blended score (`docs/CARDS.md`, `src/hotato/failure_record.py`). Timing (the conversation and speech lanes) is the dimension family with a frozen physics benchmark: measurement error against rendered and hand-labelled ground truth (`docs/BENCHMARK.md`). Say-do (the outcome and policy lanes) is the dimension family with deterministic trace evaluation, grounded by the reference-agent suite above. Reliability aggregates repeated runs as `pass@1` / `pass@k` / `pass^k` on its own axis. ## How to verify a score on your own recordings 1. Run `--dump-frames out.json` on the recording. Read the per-channel `*_threshold_db` and `*_noise_floor_db` from the header, and confirm `threshold = noise_floor + 15 dB` (or the dynamic-margin cap, or the -60 dBFS gate) matches your levels. 2. Scan the `*_dbfs` and `*_active` columns around the reported onset, yield, and gap indices and confirm each `active` flag is just `dbfs >= threshold` carried forward by the 150 ms hangover. Re-derive the signal by hand as above. 3. If the VAD is mislabeling your audio, tune and watch the numbers move predictably: raise `rel_db` or `abs_gate_db` if line noise is marked active; lower `rel_db` if quiet speech is missed; raise `hangover_sec` if one utterance fragments into several; adjust `yield_hangover_sec` / `caller_proximity_sec` to match how long *your* callers pause. Re-dump and confirm the thresholds and active flags moved as expected. If you disagree with a number, the frames are on the table and every threshold is yours to change. That is the point: a measurement you can audit, not a verdict you have to trust. ================================================================================ FILE: docs/BENCHMARK.md ================================================================================ # Benchmark methodology `hotato` ships a reproducible measurement-error harness, `src/hotato/benchmark.py`. Given labelled dual-channel recordings, it reports how far the scorer's measured event times land from the rendered or hand-labelled ground truth, and how its yield/hold decisions line up with the human labels. That is the whole output. ```bash PYTHONPATH=src python3 -m hotato.benchmark ``` This runs on every synthetic fixture in the checkout (the bundled battery, the `examples/` reference set, and the deliberately-bad `funnel-demo` set), prints a markdown table, and writes `benchmark-report/measurement-error.json` and `benchmark-report/measurement-error.md`. The harness is a standalone measurement tool you run against fixtures; the `hotato` CLI scores a single call or the battery. **Scope.** This benchmark measures the timing signals: per-signal measurement error against rendered or hand-labelled ground truth, and the yield/hold decision. Say-do verification is scored on its own deterministic lane, evaluated against recorded trace spans; its methodology is the "Say-do verification methodology" section of [`METHODOLOGY.md`](../METHODOLOGY.md), with [`examples/reference-agent`](../examples/reference-agent) as the runnable worked example. --- ## What it measures For every `(recording, label)` pair it reports two things. ### 1. Per-signal measurement error, in milliseconds For each timing signal it computes `|measured - rendered|` in ms, where `rendered` is either the exact value the fixture was rendered from (its `reference_render` block) or a contributor's hand label: - **caller onset** - Measured: onset the VAD detects (the scorer is given no onset label). - Rendered: start of the first caller segment. - **time to yield** - Measured: seconds from caller onset to the agent going quiet. - Rendered: the agent's in-progress turn end minus the caller onset. - **response gap** - Measured: endpointing dead-air from the caller's turn end to the agent's next onset. - Rendered: the fixture's rendered response gap. Errors are reported as a distribution: median, mean, worst case, best case, and n. A signal scores only where a reference exists and the scorer produced a value; missing either yields a `-`. (The echo-of-agent fixture has no independent caller speech, so its onset/yield error is a gap by construction.) Onset is measured in **detect mode** (the scorer gets no onset hint), so it tests the onset detector directly. Yield, talk-over, response gap, and `did_yield` are measured in **label mode** (the scorer is given the human `caller_onset_sec`, as the shipped battery runs) -- those are the numbers a user sees. ### 2. A `did_yield` confusion matrix Against the `should_yield` / `should_not_yield` label (each scenario's `expected.yield`), it reports the four cells: | | measured **did_yield** | measured **held floor** | |---|---|---| | **should_yield** | correct yield | **missed yield** | | **should_not_yield** | **false yield** | correct hold | The two off-diagonal cells -- missed yields and false yields -- are the failures an operator feels, so the report surfaces them directly. --- ## Why milliseconds and a matrix The report pairs a per-signal error distribution with a four-cell confusion matrix and keeps the cells separate: a missed yield and a false yield are different failures with different fixes, so averaging them into one number would hide which one you have. `docs/CORPUS-GOVERNANCE.md` ("Validity metrics") enforces the same rule for corpora; the benchmark applies it to the tooling itself. The reported error is what the default shipped config measures, so it is the number you get. On the synthetic fixtures, the yield error equals the exposed VAD hangover, and the onset/gap error is one frame hop -- both documented `ScoreConfig` parameters. Set the hangover to zero (`caller_vad.hangover_sec = agent_vad.hangover_sec = 0`) and every signal collapses to within one hop of ground truth; the test suite asserts this, so the claim is checkable against the code. The scorer reads speech energy over time, so that is what the harness reports: timing and decisions. --- ## Quantization: every reported time has a resolution floor Every timing signal the scorer reports is quantized to the frame hop (`ScoreConfig.hop_ms`, default `10.0` ms) plus, for yield/talk-over, the VAD hangover (`caller_vad.hangover_sec` / `agent_vad.hangover_sec`, default `0.15` s): the measured value can land up to one hop off the true event, purely from where that event falls inside a 10 ms frame, before hangover is even counted. This sub-frame-phase rounding is deterministic -- the same one-hop collapse the section above already pins down (hangover zero -> every signal within one hop of ground truth). Against a label placed at the raw end of speech energy, the measured end of an overlap or yield sits late by at most `hangover_sec` plus one hop (0.16 s at defaults) and the measured start sits early by at most one frame (0.02 s at defaults). The bias is deterministic and one sided. Setting `caller_vad.hangover_sec` and `agent_vad.hangover_sec` to 0 removes the hangover term and leaves frame quantization; on recorded speech a zero hangover can fragment one utterance at intra word dips, which lowers measured overlap. Measured case: a two channel recording of two human speakers with a hand labeled 0.420 s overlap at the raw speech edges measures `talk_over_sec = 0.590` at defaults and `0.420` at hangover zero, the bound holding exactly (end +0.155 s, start -0.005 s). The consequence for a `--max-time-to-yield` (or any) policy bound: a physically identical yield event, shifted by a few milliseconds of sub-frame phase with nothing else changed, can cross an exact bound purely from quantization. Reproduced case: a 250 ms yield event against a 400 ms bound flips PASS/FAIL as the event's sub-frame phase sweeps through 3, 6, 12, and 16 ms offsets, the underlying event unchanged -- the measured value moves by exactly one hop (10 ms) at each transition, no further. Phase alone can flip a bound set within one hop of the true value either way, on that recording. **Read policy bounds accordingly: a margin under one hop (10 ms default) from the true value sits inside the scorer's quantization noise.** `hotato` surfaces that margin plainly (see `docs/FIX-PLANS.md`'s no-single-threshold rule, extended here to quantization); set bounds at least one hop from any value you need to hold. --- ## Noise floor and the verdict cliff Below a measurable per-channel SNR the verdict flips rather than degrades. The mechanism is one line of the energy VAD: the speech threshold is capped at the channel's loudest frame minus `dyn_margin_db` (22 dB), so once the noise floor climbs inside that margin every frame reads active, agent activity never ends, and a correct yield scores as a false 3.0 s talk-over. Measured on the reference fixture (`01-hard-interruption`, seeded noise added to both channels): with uniform noise the yield verdict flips between 19 and 18 dB per-channel SNR; with babble-shaped noise, between 21 and 20 dB. The hardest shipped pass tier (the gold noise family) bottoms out at a 23.8 dB noise floor, about 5 dB above the cliff. The opt-in scorability gate covers the band below: `hotato run --snr-gate-db` (bare flag = 22.0, which equals `dyn_margin_db`, the geometric constant of the cliff, not a tuned number) estimates each channel's stationary SNR deterministically and refuses to score (not-scorable, exit 2, reason `low-snr`) when either estimate falls below the floor, instead of emitting the false talk-over. A gated run carries the per-channel `snr_estimate` block on the event. The estimate certifies a stationary noise floor. Strongly non-stationary noise (babble) can flip a verdict while estimating above the floor: the babble flip sits 2 dB above the uniform flip on the curve above. Read a gated pass as "the stationary floor clears the margin", and treat heavy competing speech as its own capture problem. --- ## Reproducing it ```bash # render the synthetic fixtures deterministically (sha256-seeded; byte-identical # for a fixed hotato version -- verified in CI on Linux x86_64, Python 3.10, # 3.11, and 3.12 -- see .github/workflows/tests.yml), then run the harness over them python3 examples/render_examples.py PYTHONPATH=src python3 -m hotato.benchmark ``` The report carries no wall-clock timestamp, and the render is deterministic, so two runs on the same code produce byte-identical artifacts. The JSON carries a `config` snapshot of every threshold behind the numbers, so a reader can re-derive any value. `tests/test_benchmark.py` pins all of it: the harness runs, the confusion matrix matches known rendered behaviour, and every ms-error sits within its known, config-derived tolerance. The synthetic fixtures are a floor: deterministic rendered audio with exact known timings, showing the scorer behaves as specified and guarding against regressions. See `examples/README.md` and `docs/CORPUS-GOVERNANCE.md`. --- ## Extending to your own recordings (bring your own labelled data) The synthetic floor shows the scorer does what the spec says. A labelled recording from a live phone line shows what it measures under production conditions. The harness runs on your own labelled data with no code change: ```bash PYTHONPATH=src python3 -m hotato.benchmark \ --scenarios path/to/your/scenarios \ --audio path/to/your/audio \ --name my-corpus \ --out my-benchmark-report ``` - `--scenarios` is a directory of scenario JSON labels, same shape as `src/hotato/data/scenarios/*.json`. Each needs an `id`, an `expected.yield` label, and whatever reference timings you can defend: a `reference_render` with segment timings (as the synthetic fixtures carry), or at minimum a hand-labelled `caller_onset_sec` plus the true event times to score error against. Supply a reference only where you have ground truth -- the harness reports error for those signals and leaves the rest blank. - `--audio` is a directory of dual-channel `.example.wav` recordings (caller on channel 0, agent on channel 1). Record two physically separated channels whenever you can: that is what makes overlap ground-truthable. When `--scenarios` is given, only that set is scored -- a clean report for your corpus alone. The path to a number from your own recordings is manual and consented: bring your own labelled audio, governed by `docs/CORPUS-GOVERNANCE.md` (consent, PII, data-handling) and the pipeline in `corpus/` (a labelling schema and a validator). Synthetic fixtures keep their synthetic label. ================================================================================ FILE: docs/API.md ================================================================================ # Python API reference Every function the CLI, pytest plugin, and MCP server use is a plain Python function. The core is stdlib-only, offline, deterministic: same audio and config, same envelope, every time. Import and score: ```python from hotato.core import run_single, run_suite env = run_single(stereo="call.wav", expect="yield") env = run_suite() # bundled 8-scenario battery ``` The top-level package re-exports the essentials (`from hotato import run_single, run_suite, LIMITS, SUITE_ID, __version__`); every scoring function takes keyword arguments only. ## The envelope `run_single` and `run_suite` return the same machine-readable dict (schema: `src/hotato/schema/envelope.v1.json`): ```python { "tool": "hotato", "schema_version": "1", "mode": "single" | "suite", "stack": "generic", # normalized stack label "offline": True, "engine": {"name", "version", "upstream"}, "limits": {...}, # scope and ceiling, hotato.core.LIMITS "summary": {"events", "passed", "failed", "regression"}, # plus additive "not_scorable" (count) when # at least one event could not be judged "events": [...], # one dict per scored event, below "fix_map": [...], # one entry per failing event with a fix "funnel": {...} | None, # systemic pointer, fires only when both axes fail "exit_code": 0 | 1, # 1 when any scorable event failed; the CLI # process exits 2 for a single recording that # is not scorable (see Exit codes below) "suite": "barge-in", # run_suite only } ``` Each event: ```python { "event_id": str, # file basename or scenario id "scenario_id": str | None, "title": str | None, "category": str | None, # e.g. "should_yield" "expected_yield": bool, "verdict": { "passed": bool, "did_yield": bool, "seconds_to_yield": float | None, "talk_over_sec": float, "reasons": [str], # failure reasons, empty on pass }, "measurements": { "caller_onset_sec": float, "agent_talking_at_onset": bool, "hop_sec": float, "notes": str, }, "signals": { # namespaced signal bus, additive "barge_in": {"did_yield", "time_to_yield_sec", "talk_over_sec"}, "latency": {"response_gap_sec", "premature_start_sec"}, "echo": { # every event; cross-channel coherence, # deterministic, computed in hotato's own # layer (hotato/echo.py), never _engine "coherence", "lag_sec", "echo_suspected", }, "resume": { # only on events where the agent yielded "resumed", "resume_gap_sec", "restart_suspected", }, }, "fix": None | { # set on failing events "fix_class": "config" | "engagement-control", "title": str, "detail": str, "knob": str | None, "pointer": str | None, }, } ``` An event without enough input to judge (a silent caller channel with no onset label, or a should-yield expectation with the agent silent at onset) carries `"scorable": False` and a plain `"not_scorable_reason"` instead. It sits outside `passed`/`failed`, the fix router, and the funnel -- an input problem, not a verdict -- so envelopes for valid recordings stay byte-identical. ## hotato.core ### run_single ```python run_single( *, stereo: str | None = None, # two-channel WAV path caller: str | None = None, # mono WAV path (with agent=) agent: str | None = None, # mono WAV path (with caller=) caller_channel: int = 0, agent_channel: int = 1, onset_sec: float | None = None, # caller onset hint, seconds from start expect: str = "yield", # "yield" or "hold" (backchannel) stack: str | None = None, # livekit | pipecat | vapi | generic max_talk_over_sec: float | None = None, max_time_to_yield_sec: float | None = None, echo_gate: bool = False, # hold an echo-suspected yield out of # the verdict (scorable: false) instead # of counting it as a clean pass cfg: ScoreConfig | None = None, ) -> dict ``` Scores one recording and returns the envelope: provide `stereo`, or both `caller` and `agent`. `expect="hold"` means the caller's speech is a backchannel (a short acknowledgement like "mhm") that a correct agent talks through. Both `max_*` thresholds tighten the pass criteria; `echo_gate` is opt-in and off by default, though `signals.echo` always computes. A malformed or truncated WAV raises a clean `ValueError` naming the ffmpeg fix. ### run_suite ```python run_suite( *, suite: str = "barge-in", # the only suite id (SUITE_ID) stack: str | None = None, scenarios_dir: str | None = None, # your scenario JSON labels audio_dir: str | None = None, # your recordings, suffix: str = ".example.wav", caller_channel: int = 0, agent_channel: int = 1, echo_gate: bool = False, # see run_single cfg: ScoreConfig | None = None, ) -> dict ``` Runs a labelled battery and returns the envelope with a `suite` key. The bundled 8-scenario battery ships inside the package, zero external files; point `scenarios_dir`/`audio_dir` at your own labelled set instead (e.g. `corpus/suites/gold/scenarios` and `.../audio`). Suite audio must be two-channel. ### dump_frames_for_input ```python dump_frames_for_input( *, stereo: str | None = None, caller: str | None = None, agent: str | None = None, caller_channel: int = 0, agent_channel: int = 1, onset_sec: float | None = None, cfg: ScoreConfig | None = None, ) -> dict ``` The per-frame evidence behind every reported number: each channel's dBFS, VAD activity, threshold, and noise floor, plus a self-describing `config` block. Every signal it reports is re-derivable by hand from this dump. ### LIMITS and SUITE_ID `hotato.core.LIMITS` is the scope dict embedded in every envelope: method, ceiling, best input, and boundaries. `SUITE_ID` is `"barge-in"`. ### ScoreConfig Every threshold is an exposed parameter: ```python from hotato._engine.score import ScoreConfig from hotato._engine.vad import VADParams cfg = ScoreConfig( frame_ms=20.0, hop_ms=10.0, yield_hangover_sec=0.20, # agent quiet this long = yielded max_search_sec=3.0, # yield search window after onset caller_proximity_sec=0.5, turn_end_silence_sec=0.20, premature_tolerance_sec=0.05, onset_min_run_sec=0.05, agent_onset_lookback_sec=0.10, caller_vad=VADParams(), # rel_db=15.0, abs_gate_db=-60.0, agent_vad=VADParams(), # hangover_sec=0.15, noise_percentile=0.10, ) # dyn_margin_db=22.0, backend="energy" ``` `VADParams.backend` is `"energy"` (the deterministic reference behind every published number) or `"neural"` (an optional Silero VAD cross-check via `pip install 'hotato[neural]'`; without the extra, a clean `BackendUnavailable` names the missing dependency). ## hotato.report Self-contained visual reports scored from the same measurements. All three take the full scoring parameter set from `run_single`/`run_suite` above, as keyword arguments. ```python build_report_html(*, base: dict | None = None, base_label: str | None = None, **kwargs) -> (str, dict) build_report_md(*, base: dict | None = None, base_label: str | None = None, **kwargs) -> (str, dict) write_report(path: str, fmt: str = "html", **kwargs) -> dict ``` `build_report_html` scores the input and returns `(html, envelope)`: one self-contained file with inline CSS and SVG, per-event timelines, analytics, a frame inspector, and print CSS for PDF. `build_report_md` mirrors it as Markdown tables. `write_report` builds in `fmt` (`"html"` or `"md"`), writes to `path`, and returns the envelope. Pass `base` (a prior envelope, e.g. loaded from `hotato run --format json`) to render per-scenario regression deltas; `base_label` names it on the page. ```python from hotato.report import write_report env = write_report("report.html", suite="barge-in", stack="livekit") ``` ## hotato.aggregate Team mode: many run envelopes, one trend view. ```python load_run_dir(dirpath: str, order: str = "mtime") -> dict # {"runs": [{"file", "path", "mtime", "env"}], "skipped": [...], "order"} # order: "mtime" (oldest first) or "name" (numeric prefix = explicit index) aggregate_runs(runs: list, order: str = "mtime", skipped: list | None = None) -> dict # team envelope: kind "team-aggregate", runs, events_total, # talk_over_sec / seconds_to_yield distribution summaries (mean/median/p90), # pass_rate {latest, first, mean, direction}, pass_rate_over_time, # failure_classes, most_common_failure_class, skipped, exit_code 0. # Raises ValueError with fewer than 2 runs; each trend point corresponds # to one input run. build_team_section_html(agg: dict) -> str # embeddable section build_team_page_html(agg: dict) -> str # full self-contained page ``` ```python from hotato.aggregate import load_run_dir, aggregate_runs, build_team_page_html loaded = load_run_dir("runs/") agg = aggregate_runs(loaded["runs"], order=loaded["order"], skipped=loaded["skipped"]) html = build_team_page_html(agg) ``` ## hotato.export Research-grade flat files from the same scorer. ```python run_export( *, out_dir: str, # plus the full scoring parameter set: stereo, caller, agent, # caller_channel, agent_channel, onset_sec, expect, stack, suite, # scenarios_dir, audio_dir, suffix, max_talk_over_sec, # max_time_to_yield_sec, cfg ) -> dict # {"env", "events_rows", "frames_rows", "paths"} ``` Writes `events.csv` (one row per event, columns in `hotato.export.EVENT_COLUMNS`), `frames.csv` (one row per VAD frame, columns in `FRAME_COLUMNS`), and `envelope.json` into `out_dir` (created if missing); `#` comment lines at the top of each CSV document the columns. An empty cell means no value for that input; every filled cell is a direct measurement. ## hotato.stackbench Identical scenarios, your stack, comparable result files: every number is a measurement of the recordings you provide. ### run_stackbench ```python run_stackbench( *, stack: str, # one of BENCH_STACKS: # vapi | twilio | livekit | pipecat | generic recordings_dir: str, # WAVs scenarios_dir: str | None = None, # default: bundled battery suffix: str | None = None, # default: auto-detected # (.wav, .stereo.wav, .example.wav) caller_channel: int = 0, agent_channel: int = 1, cfg: ScoreConfig | None = None, ) -> dict ``` Returns a result dict (`kind: "stack-benchmark"`) with the envelope fields plus `config`, `scenarios {total, captured, not_captured}`, and `provenance`. Scoring is unchanged `run_suite`; a scenario with no matching recording lands under `not_captured`, out of the scoring and failure counts. The timestamp derives from input file mtimes, so the same inputs always reproduce the same file. ### load_result, compare_results, render_comparison_md ```python load_result(path: str) -> dict # loads and validates one result JSON; anything else is a clean ValueError compare_results(inputs: Sequence[tuple[str, dict]]) -> dict # inputs: (path, loaded_result) pairs, at least two. # Compares the intersection of scenarios scored in EVERY input; # the rest is listed under "skipped". Deltas are signed differences # against the FIRST input. Returns kind "stack-benchmark-comparison" # with inputs, compared, skipped, per_scenario, medians. render_comparison_md(cmp_env: dict) -> str # the comparison as Markdown tables ``` ```python from hotato.stackbench import load_result, compare_results, render_comparison_md cmp = compare_results([(p, load_result(p)) for p in ("a.json", "b.json")]) print(render_comparison_md(cmp)) ``` ## Pytest fixture Installs automatically via the `pytest11` entry point (or load it explicitly with `-p hotato.pytest_plugin`). Sits inert until your test calls it. ```python def test_call_yields(hotato_score): env = hotato_score(stereo="call.wav", expect="yield") assert env["summary"]["regression"] is False assert env["events"][0]["verdict"]["seconds_to_yield"] < 1.0 ``` `hotato_score(**kwargs)` takes the same keyword arguments as `run_single`; pass `suite="barge-in"` (plus `run_suite` keywords) to score a battery instead. It returns the envelope -- write your own assertions against it. Session gate flags: `pytest --hotato-suite` runs the battery after your tests and fails the session (exit 1) on a regression; `--hotato-suite-scenarios DIR` and `--hotato-suite-audio DIR` point it at your own labelled set. Detail: `docs/PYTEST.md`. ## `hotato.counterexample` The counterexample API reduces one failing deterministic scripted scenario, then emits a content-addressed replay capsule and a replayable deletion proof. It never loads a provider adapter, model, network client, or subprocess. ```python from hotato.counterexample import ( compile_counterexample, verify_counterexample, reproduce_counterexample, inspect_counterexample, export_counterexample, predicate_counterexample, ) compiled = compile_counterexample( "refund.scenario.json", "refund.test.json", target="refund-posted", out_dir="refund-posted.hotato-repro", workspace=".", budget=512, ) assert compiled["exit_code"] in (0, 1) ``` `compile_counterexample` returns exit `0` only after the final unit-deletion pass earns `one_minimal`. Exit `1` means the failure was preserved and a runnable capsule written, but the evaluation budget ran out before the proof completed. A refusal raises `CounterexampleRefusal`, leaving no destination directory. Input parsing raises `ValueError` for malformed or schema-invalid JSON/YAML-subset, `OSError` for I/O failures -- both mapped by the CLI to its handled-error, exit-code contract. ```python verify_counterexample("refund-posted.hotato-repro") reproduce_counterexample("refund-posted.hotato-repro") inspect_counterexample("refund-posted.hotato-repro") export_counterexample( "refund-posted.hotato-repro", out_dir="refund-posted.share", ) predicate_counterexample("refund-posted.hotato-repro") ``` - `verify_counterexample` requires the recorded package version and evaluator source digest, then independently replays the source, the accepted delete-only chain, the final case, derived artifacts, and any completed one-minimal claim. The digest identifies shipped evaluator source, not interpreter or platform identity; replay hashes catch behavior changes. - `reproduce_counterexample` permits evaluator drift and checks the reduced case twice for the source-selected structured failure branch. It targets Hotato evaluator/scenario regressions; it does not execute a deployed voice agent. - `inspect_counterexample` verifies the closed member inventory, bound source, oracle and artifacts, and canonical human files without executing the scenario. - `export_counterexample` first verifies the private capsule, then writes a non-runnable projection with content-bearing inputs omitted. Hashes remain correlators and may still be sensitive. - `predicate_counterexample` returns `1` when the failure remains, `0` when it is absent, and `125` when the result cannot be used by `git bisect run`. The v1 source is the base scripted scenario at the selected seed. A `variation_matrix` is recorded as unapplied and may be removed during reduction; compile a concrete scenario when a specific expanded run matters. Full scope, artifacts, commands, and claim ceiling: [`COUNTEREXAMPLES.md`](COUNTEREXAMPLES.md). ## MCP tool `hotato-mcp` (or `python -m hotato.mcp_server`) speaks MCP over stdio: one scoring tool (`voice_eval_run`, returning the identical envelope the CLI emits), three counterexample tools, and eleven fleet tools -- full list and scope in [`MCP.md`](MCP.md). Install: `uvx --from "hotato[mcp]" hotato-mcp`. The parameters below are `voice_eval_run`'s, all optional: | Parameter | Type | Default | Meaning | | --- | --- | --- | --- | | `stereo` | str | None | two-channel WAV path | | `caller` | str | None | mono caller WAV (with `agent`) | | `agent` | str | None | mono agent WAV (with `caller`) | | `suite` | str | None | `"barge-in"` to run the bundled battery | | `stack` | str | `"generic"` | livekit, pipecat, vapi, or generic | | `expect` | str | `"yield"` | `"yield"` or `"hold"` | | `onset_sec` | float | None | caller onset hint | | `caller_channel` | int | 0 | caller channel index | | `agent_channel` | int | 1 | agent channel index | | `max_talk_over_sec` | float | None | pass threshold | | `max_time_to_yield_sec` | float | None | pass threshold | | `report_path` | str | None | also write the HTML report here; the envelope then carries `report_path` (absolute) | ## Exit codes and errors Envelopes carry `exit_code`: 0 means every scorable event passed, 1 a regression. `hotato.core.process_exit_code(env)` maps the envelope to the CLI process exit: a single-recording run whose event isn't scorable (silent caller with no onset label, or agent silent at onset; reason in `not_scorable_reason`) exits 2 -- an input problem, not a verdict. Suite runs count such events in `summary.not_scorable`, keeping their 0/1 semantics. Malformed input (bad WAV, out-of-range channel, negative onset, unknown suite or stack) raises `ValueError`, surfaced as exit code 2: only a file the scorer can fully read gets scored. ================================================================================ FILE: docs/SDK.md ================================================================================ # Python SDK (`hotato.sdk`) `hotato.sdk` is a typed facade over the same functions the CLI runs. It imports the internal code paths directly, so there is no subprocess. Every result is a frozen dataclass whose fields are the keys of the JSON the CLI emits with `--format json`. ## Install `pip install hotato` ships the SDK and its `py.typed` marker, so a type checker reads the package as typed. The scoring core is stdlib-only. Transcription uses the optional `hotato[transcribe]` extra. ## Run a suite ```python from hotato.sdk import run_suite result = run_suite() # the bundled labelled battery, zero files print(result.passed, result.failed) # process pass (exit 0) and failure count for event in result.events: print(event.event_id, event.passed, event.seconds_to_yield, event.talk_over_sec) ``` Point `run_suite` at your own labelled set with `run_suite(scenarios="scenarios/", audio="audio/")`. ## Verify contracts ```python from hotato.sdk import verify_contracts result = verify_contracts("contracts/") print(result.passed, result.summary["failed"]) for c in result.results: print(c.id, c.passed, c.authenticity) # authenticity passes through verbatim ``` A regressed or tampered contract sets `passed` to `False` with `exit_code` 1. It returns as data; it does not raise. A missing or corrupt bundle raises `ValueError`. ## Compile a counterexample ```python from hotato.sdk import compile_counterexample, verify_counterexample compiled = compile_counterexample( "case.scenario.json", "case.test.json", target="pii-email", out="case.hotato-repro") print(compiled.minimality, compiled.output) verified = verify_counterexample("case.hotato-repro") print(verified.status, verified.passed) ``` A deterministic refusal raises `CounterexampleRefusal`, which carries a stable `.code`. ## The JSON contract The dataclass fields are the CLI's JSON keys. `SuiteResult`, `ContractVerifyResult`, `InvestigateResult`, `CounterexampleResult`, and `CounterexampleVerifyResult` mirror `hotato run`, `hotato contract verify`, `hotato investigate`, and `hotato counterexample`. A `SuiteResult.passed` is `exit_code == 0`, and `failed` is the count of failed scorable events. A NOT-SCORABLE or INCONCLUSIVE result keeps its own status; nothing is blended into a single number. ## Errors Bad input raises `ValueError` and its subclasses. A missing optional extra raises `BackendUnavailable`. A counterexample refusal raises `CounterexampleRefusal`. All of these sit inside `hotato.errors.HANDLED`, the same set the CLI and MCP surfaces map to a structured error. ## Transcription ```python from hotato.sdk import transcribe, build_transcript_cache cache, _warning = build_transcript_cache() transcript = transcribe("call.wav", cache=cache) print(transcript.text) ``` A transcript is context beside the timing score. It leaves `did_yield`, `seconds_to_yield`, and `talk_over_sec` unchanged. For the cache-hit flag and the cache key, call `transcribe_cached` and read its `CachedTranscribeResult`. ================================================================================ FILE: docs/ANALYZE.md ================================================================================ # `hotato analyze `: drop a folder, hear the bug Point it at a folder of dual-channel call recordings and it finds the worst turn-taking moments across all of them, ranks them, and lets you hear each one. ```bash hotato analyze ./recordings ``` Writes one self-contained, offline HTML dashboard (`hotato-analyze.html` by default) and opens it. A bare folder as the first argument routes here too: `hotato ./recordings`. ## What it does For every `.wav` under the folder (walked recursively, sorted), `analyze`: 1. runs the same whole-call scanner as [`hotato scan`](../src/hotato/scan.py): walks caller/agent activity across the call, label-free, and emits candidate timing moments (`overlap_while_agent_talking`, `agent_start_during_caller`, `long_response_gap`, `agent_stop_no_caller`, `echo_correlated_activity`), each with a timestamp and a measured number; 2. aggregates the candidates across every call and ranks them by salience (overlap seconds / gap seconds / echo coherence), so the worst moments float to the top. Then it emits three things. ### 1. A ranked dashboard One card per top moment, in the [`hotato report`](REPORTS.md) house style: the call file, timestamp, candidate kind, measured number, and a to-scale caller/agent timeline (activity spans, the talk-over band, the onset marker, and a yield marker where the agent went silent). ### 2. The hear-the-bug player For the top `--audio-top` moments (default 8), audio around the moment embeds inline as base64 WAV, so the page stays self-contained. Press play and a **playhead** sweeps the timeline in lockstep: you hear the talk-over or dead-air land exactly where the chart marks it. It tracks playback under `prefers-reduced-motion` too. Only the top moments carry audio; the rest show the timeline. That, plus `--pre`/`--post` and a total audio budget, keeps the page a manageable size. ### 3. JSON for agents ```bash hotato analyze ./recordings --format json ``` Prints the ranked candidates and their metadata (source file, timestamp, kind, salience, measured durations, audio/timeline window) to stdout, capped by `--top`. `--out FILE` also writes the full ranked JSON. ## Framing Each result is a **measured candidate timing moment**: a timestamp and a number, not a verdict on intent. Energy sounds the same whether a caller said "mhm" or "stop," so you decide and label the moments that matter: ```bash hotato fixture create --stereo .wav --onset \ --expect yield|hold --id found-moment-001 --out tests/hotato ``` Non-dual-channel or unreadable files land in a "Skipped files" section with their reason (a mono mix can't attribute talk-over), and the run continues. ## Flags | flag | default | meaning | | --- | --- | --- | | `FOLDER` | (required) | directory of dual-channel WAVs, walked recursively | | `--top` | 25 | ranked moments shown in the dashboard / stdout JSON (0 = all) | | `--audio-top` | 8 | top moments that get the embedded hear-the-bug player | | `--pre` | 2.0 | seconds of audio/timeline kept before each moment | | `--post` | 4.0 | seconds of audio/timeline kept after each moment | | `--min-gap` | 2.0 | minimum response gap (seconds) to surface as a candidate | | `--caller-channel` / `--agent-channel` | 0 / 1 | channel assignment | | `--format` | `html` | `html` (dashboard) or `json` (ranked candidates) | | `--out` | `hotato-analyze.html` | where to write the dashboard, or the JSON | | `--no-open` | off | do not launch a browser for the dashboard | ## Exit codes - **0**: ran -- candidate moments listed across the folder, possibly zero. - **2**: usage error (the path is not a folder) or an IO error reading it. Everything, audio included, stays on your machine. ================================================================================ FILE: docs/APPLY.md ================================================================================ # hotato apply: the guarded, clone-only staged apply `hotato apply` turns a `hotato patch` artifact into a staging assistant you test before touching production. It's the only Hotato command that mutates external platform state, and the most conservative one for it: every flag combination leaves your source assistant untouched. By default it PRINTS the clone it would create, an offline dry run; only `--yes` with credentials creates the new one. ``` hotato patch fixplan.json --format json --out patch.json hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato ``` ## Five rules, enforced in code 1. **Clone-only.** apply ships one path: staging-clone. Any call without `--clone` errors cleanly: `production apply is not supported; use --clone to apply to a fresh staging assistant`. The only write is a `POST` creating a NEW assistant, never `PUT`/`PATCH` on the source. 2. **Refusal-first.** If the patch is the both-axes threshold funnel (the plan decided `do_not_tune_single_threshold`), apply REFUSES before doing anything and prints the exact recommendation: ``` No config patch will be applied Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both Recommended: enable or add engagement-control / backchannel-aware turn detection ``` The refusal is a FEATURE: a distinct, documented exit code (`3`) so a script can tell "refused by design" from a usage error. A single sensitivity threshold trades catching an interruption against holding through a backchannel; fixing it takes a discrimination fix, and every clone's patch reflects that. 3. **Opposite-risk required.** `--battery` must carry BOTH a yield fixture (an interruption the agent must stop for) AND a hold fixture (a backchannel it must keep the floor through), so apply can see the opposite risk before applying. Point `--battery` at your fixtures directory (with a `scenarios/` folder, as `hotato fixture promote` writes) or a folder of run-envelope / scenario JSONs. 4. **Gated side effect.** The default is a dry run: it prints the clone and patch, offline. Only `--yes` with credentials reaches the platform. The create call is the only networked step: read the source config (`GET`), apply the patch to a copy, create a NEW assistant (`POST`). 5. **Name required.** The staging clone must be named explicitly (`--name`); apply always uses the name you give it. ## What gets cloned The REST-config stacks clone an assistant/agent through their own API, read-only against the source and creating a NEW one: | Stack | Read (source) | Create (new) | Name field | | --- | --- | --- | --- | | vapi | `GET https://api.vapi.ai/assistant/{id}` | `POST https://api.vapi.ai/assistant` | `name` | | retell | `GET https://api.retellai.com/get-agent/{id}` | `POST https://api.retellai.com/create-agent` | `agent_name` | The clone config is your source config with the patch deep-merged on top, the new name, and server ids stripped: a fresh object, never an overwrite of the source. LiveKit and Pipecat keep turn-taking config in your agent source, so `hotato patch` emits the exact source edit and apply points straight at it. Twilio carries the audio but runs no turn-taking agent, so apply points at the upstream stack instead. ## After the clone: prove it The clone lets you prove the fix before touching production. Re-capture the same battery through the SOURCE (`before/`) and the CLONE (`after/`), then verify: ``` hotato verify --before before/ --after after/ --policy hotato.verify.yaml ``` `hotato verify` reports coincidence, not causation. Its policy is the anti-bandaid gate: a fix passes only when every guardrail holds (including `max_new_false_yields` on the hold fixtures) and every target is met -- a patch that just makes the agent yield to everything gets caught. See `docs/FIX-LOOP.md`. Or run the whole thing (this gate, verify, an optional contracts re-verify, and explain's attribution) as one fail-closed before/after report: `hotato fix trial patch.json --name staging-refund-fix --before before/ --after after/`. See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md). ## Exit codes - `0` the staging clone was rendered (dry run by default; with `--yes` and credentials the NEW staging assistant was created and patched, the source untouched). - `2` usage error: no `--clone`, no `--name`, no opposite-risk battery (a yield and a hold fixture), a stack with no assistant to clone, a patch with no config change, or unreadable input. - `3` principled refusal: the plan is the both-axes threshold funnel, so apply refuses a single-threshold patch by design -- the refusal is the feature. ================================================================================ FILE: docs/ASSERTIONS.md ================================================================================ # Assertions (`assert.v1`): deterministic, typed, each scored on its own lane `hotato.assert_` checks a call's transcript, ingested trace, and timing against a fixed set of typed assertions, each kind scored on its own lane by construction. Every kind is a regex, checksum, or span/dict lookup -- deterministic end to end, no model call in the loop. ```python from hotato import assert_ as A ctx = A.build_context( transcript_path="transcript.json", # or transcript=[...] turns already in hand trace_path="voice_trace.jsonl", # or spans=[...] already in hand timing=envelope, # optional: a run's envelope.v1 events ) env = A.run_assertions_from_file("assertions.yaml", ctx) print(env["exit_code"]) # 0 pass, 1 fail ``` Python API: [`docs/API.md`](API.md) covers the scoring envelope this complements; the schema is `src/hotato/schema/assert.v1.json`. Runnable ground truth: [`examples/reference-agent`](../examples/reference-agent) exercises these kinds end to end: a 375-run offline suite (25 scenarios, 5 caller behaviours, 3 audio environments) whose `tool_result`, `state`, `sequence`, `handoff`, and `tool_error` assertions surface four seeded agent defects, deterministically. Method: the "Say-do verification methodology" section of [`METHODOLOGY.md`](../METHODOLOGY.md). ## The core deterministic kinds Every kind is deterministic, so every result carries `deterministic: true`, including `INCONCLUSIVE`. The five below are the original core; the full vocabulary is under [the whole `kind` vocabulary](#the-whole-kind-vocabulary). | Kind | Checks | Reads | | --- | --- | --- | | `phrase` | a regex is present (or, in `absent` mode, never present), with an optional `role` filter and `position` (`first`/`last`/`any`) | transcript text | | `pii` | deterministic detectors (`ssn`, `card_luhn` with full Luhn validation, `email`, `phone`) find nothing, `mode: must_not_leak` | transcript text | | `policy` | a named, versioned, offline rule pack's banned-language and required-disclosure rules | transcript text | | `tool_call` | a tool was (or was not) called, with an optional argument subset, a count bound, a required order across tools, or a "never before" ordering constraint | ingested `voice_trace.v1` spans only | | `outcome` | task success as `all_of`/`any_of` a list of the sub-predicates above (`tool_called`, `phrase`, `field_present`), reported as a `met`/`of` fraction | whichever context each sub-predicate needs | `tool_call` checks only the ingested trace (`hotato trace ingest`, [`docs/TRACE.md`](TRACE.md)) -- that's the evidence a tool ran; an agent's own words claiming it ran don't count. `pii` surfaces only a `[REDACTED]` transcript artifact plus hit metadata (detector name, turn index, role) -- never the matched text. ### The whole `kind` vocabulary The full `assert.v1` `kind` vocabulary is **20 deterministic kinds**, all `deterministic: true`, all model-free. Beyond the five core kinds: `tool_result` and `tool_error` (a tool's returned value or raised error, from the trace), `http_result` (a recorded HTTP exchange span's method, URL, status, and response subset, from the trace), `state` and `state_change` (a state adapter's snapshot or transition -- see [STATE-ADAPTERS.md](STATE-ADAPTERS.md)), `handoff`, `dtmf`, `termination`, `latency`, `timing_contract`, `entity_accuracy`, `sequence`, `count`, `formula` (a boolean composite over other assertions' results in the same run, below), and `order` (a transcript ordering check, below). Two more, `human_rubric` and `judge_rubric`, belong to the SEPARATE model-judged rubric lane ([RUBRIC.md](RUBRIC.md)); inside a raw `assert.v1` document they resolve to a deterministic `INCONCLUSIVE`, so no model runs here and the guarantee holds. ### `http_result`: a recorded HTTP exchange, checked from the trace `http_result` reads `http_exchange` spans from the ingested `hotato.voice_trace.v1` trace -- a recorded request/response report carrying `method`, `url`, `status_code`, and `response`. Evaluation is a pure span lookup: hotato never performs the request, so a rerun is deterministic and offline, and it shares the Authority-1 wall with `tool_result` / `tool_error` -- an agent's spoken claim about a request can never satisfy it. ```yaml version: 1 assertions: - id: refund-posted kind: http_result method: POST # matched case-insensitively url_matches: "/v1/refunds$" # a regex searched against the span's url status: 201 # one status, or a list: [200, 201] response_subset: {status: ok} # optional: fields the response must carry ``` The span's `method` and `url` select the exchange; `status` and `response_subset` then judge it. `PASS` carries the grounding `span_ids`; `FAIL` distinguishes "no exchange matched the method and URL" from "the exchange matched but its status/response did not"; a context with no trace at all reports `INCONCLUSIVE`, never a guess. A malformed assertion (missing `method`, an invalid `url_matches` regex, a `status` outside 100-599) is a usage error caught up front, before any assertion runs. ### `formula`: a composite verdict over other assertions `formula` combines OTHER named assertions' results from the same run into one verdict. Its `expr` is a boolean expression over assertion ids -- `and`, `or`, `not`, parentheses -- plus a weighted-sum comparison (`0.6*a + 0.4*b >= 0.5`, where a bare id weighs 1). A referenced `PASS` is true and a `FAIL` is false; the weighted form sums the weights of the referenced assertions that passed and compares the total with the threshold. ```yaml version: 1 assertions: - id: refund-issued kind: tool_call name: issue_refund - id: confirmation-said kind: phrase regex: "confirmation number" role: agent - id: identity-verified kind: tool_call name: verify_identity - id: happy-path kind: formula expr: "refund-issued and (confirmation-said or identity-verified)" - id: mostly-healthy kind: formula expr: "0.5*refund-issued + 0.3*confirmation-said + 0.2*identity-verified >= 0.7" ``` The expression is parsed by a small recursive-descent parser -- never `eval()` -- and the whole reference graph is checked up front, before any assertion runs: an unknown id, a self-reference, or a reference cycle (including one through another `formula` or a `when:`) is a usage error (`ValueError`, exit `2`). References evaluate first whatever the document order (a `formula` may reference another `formula`), and `results` are always emitted in document order, byte-stable. A referenced `INCONCLUSIVE` makes the formula `INCONCLUSIVE`, with a reason naming the reference -- absent input propagates; a composite never guesses. A determinate result carries `refs` (the referenced ids), `met`/`of` (how many referenced assertions passed), and, on `FAIL`, a reason listing which references passed and which did not. ### `order`: one phrase must come before another `order` checks the sequence of two phrases in the transcript: the FIRST turn matching the `before` regex must precede (be strictly earlier than) the FIRST turn matching `after`. It measures where two phrases first appear, not why they appear. It reads only transcript text, so it needs a `--transcript`. ```yaml version: 1 assertions: - id: verify-before-account-details kind: order before: "verify your identity|date of birth|last four" after: "your account number is|current balance" role: agent # optional: match only this speaker's turns case_sensitive: false # optional, default false ``` `PASS` carries `before_turn` and `after_turn` (the zero-based turn indices the two phrases first matched at). If either phrase never matches there is no ordering constraint to violate, so the result is a vacuous `PASS` carrying `vacuous: true`. A context with no transcript reports `INCONCLUSIVE`, never a guess. The phrases present but in the wrong order is a `FAIL`. A malformed assertion (a missing or invalid `before`/`after` regex, a non-string `role`) is a usage error caught up front, before any assertion runs. ### `when:`: conditional assertions Any assertion -- any kind -- may carry an optional `when:` precondition: one assertion id, or a list of ids. Unless every referenced assertion `PASS`ed, the assertion is SKIPPED: an `INCONCLUSIVE` result with `skipped: true` and a reason naming the unmet reference(s), because the check itself never ran -- there is no verdict to report. ```yaml version: 1 assertions: - id: refund-issued kind: tool_call name: issue_refund - id: refund-amount-correct kind: tool_result name: issue_refund result_subset: {status: ok} when: refund-issued # or a list: [refund-issued, other-id] ``` A referenced `FAIL` and a referenced `INCONCLUSIVE` both skip -- the precondition demands a `PASS`. `when:` references obey the same up-front unknown-name and cycle refusal as `formula`, and the referenced assertions always evaluate first. Under `inconclusive_policy: fail`/`refuse` a skip gates like any other `INCONCLUSIVE`, so a suite that must not stay green on skipped checks can opt in. ## Context: transcript, trace, timing `build_context` assembles the three inputs an assertion run needs, each built from hotato's existing primitives: - **transcript**: `hotato.transcribe` (the opt-in `[transcribe]` extra, faster-whisper) produces one, or pass `--transcript FILE` / `transcript_path=` with a JSON file -- a plain array of `{role, text, start, end}` turns, or the `{"segments": [...]}` shape `hotato.transcribe` and the MCP surface write. - **trace**: `hotato trace ingest --otel FILE --out voice_trace.jsonl` ([`docs/TRACE.md`](TRACE.md), [`docs/OTEL.md`](OTEL.md)) produces the `hotato.voice_trace.v1` spans `tool_call` reads (`name`, `arguments`, and -- when the source trace carries them -- `result`/`error`) and `http_result` reads (`method`, `url`, `status_code`, `response`). - **timing**: a scoring run's own envelope (`hotato run --format json`, [`docs/API.md`](API.md)) passed straight through as read-only context for `outcome`'s `field_present` sub-predicate. Nothing here recomputes it. Context you never supply stays `None`, distinct from a supplied `[]` or `{}` that happens to be empty. An assertion whose required input is absent reports `INCONCLUSIVE`. `tool_call` with `spans=[]` (a trace was ingested with zero spans) is a `FAIL`, distinct from `tool_call` with no trace at all, which reports `INCONCLUSIVE`. ## `assertions.yaml` A small, dependency-free YAML subset (block mappings/sequences, flow `[...]`/`{...}`, quoted or bare scalars, `#` comments) -- or valid JSON, accepted directly. Hotato parses this subset itself, so the core stays zero third-party dependency either way. ```yaml version: 1 assertions: - id: refund-confirmed kind: outcome all_of: [{tool_called: issue_refund}, {phrase: "confirmation number", role: agent}] - id: tool-order kind: tool_call require_order: [verify_identity, lookup_account, issue_refund] never_before: {tool: issue_refund, until: verify_identity} - id: disclosure kind: phrase regex: "recorded for quality" role: agent position: first - id: no-ssn-leak kind: pii detectors: [ssn, card_luhn] mode: must_not_leak ``` Every assertion needs a unique `id` and a recognized `kind`. Kind-specific fields are validated (bad regex, unknown detector, missing required field, unsupported `version`) up front -- a malformed file is caught whole, before partial results exist. ## The deterministic/judge split This is the entire point of the module: structural, not a convention someone can quietly break. - Every result carries `kind` and `deterministic: true` -- true on every deterministic kind and every status, including `INCONCLUSIVE`, itself a deterministic read of missing required input. - The envelope's `summary` **splits** `deterministic` (`{pass, fail, inconclusive}`) from `judge` (`{pass, fail}`), each in its own count -- the schema (`src/hotato/schema/assert.v1.json`) enforces this with `"overall_score": false` and a `not: {required: [overall_score]}` on the summary object. - `judge` -- an LLM-scored rubric kind -- stays structurally quarantined from the deterministic count, so a model-scored result can never blend in. `summary.judge` reports `{"pass": 0, "fail": 0}`; `summary.note` states how many judge-scored assertions ran. - Same inputs, same file, same result, every time: `run_assertions` is byte-stable across repeated calls on identical input -- no wall-clock timestamp or random id in the mix. The report (below) renders this as two visually separate shelves, not one number -- visible on the page, not just in the JSON. ## The report: two shelves, each counted on its own `hotato.report.build_report_html` / `build_report_md` accept an optional `assertions=` parameter: an already-evaluated `assert.v1` envelope (build one with `run_assertions` / `run_assertions_from_file` / `run_assertions_from_yaml` above). Like `base` (a previous run envelope) and `transcript` (an already-produced ASR artifact), the report purely renders whatever result it's handed. ```python from hotato import assert_ as A, report env = A.run_assertions_from_file("assertions.yaml", ctx) html, _ = report.build_report_html(suite="barge-in", assertions=env) ``` When present, it adds one "Assertions" section: - **The headline.** Two counts side by side, each scored on its own: `N deterministic pass / M fail K judge-scored (advisory)`. - **Deterministic** (audio / timing / transcript / trace derived): one PER-DIMENSION TYPED card per result -- a kind tag, the `deterministic` flag, the PASS/FAIL/INCONCLUSIVE chip, and that result's kind-specific fields (a `pii` card's hit detail and redacted transcript, a `policy` card's matched rules and pack name, a `tool_call` card's grounding span ids, an `outcome` card's met/of fraction). - **Model-assisted (advisory, quarantined)**: stays empty here by design, with a note pointing at the model-judged rubric lane ([RUBRIC.md](RUBRIC.md)) where that scoring runs. `assertions=None` (the default) is byte-identical to a report built before this parameter existed. ## `inconclusive_policy`: making missing input gate CI By default an `INCONCLUSIVE` result -- a check whose required input was absent -- leaves the exit code unaffected, the right default for an exploratory run. `inconclusive_policy` lets a suite gate on that instead, so a transcript or trace that never arrived fails loudly instead of leaving the suite silently green: | value | how `INCONCLUSIVE` gates | `exit_code` | | --- | --- | --- | | `report` (default) | reports missing input and leaves the gate unchanged | `1` if any `FAIL`, else `0` | | `fail` | gates exactly like a `FAIL` | `1` if any `FAIL` **or** `INCONCLUSIVE`, else `0` | | `refuse` | refuses to return a verdict at all | `2` if any `INCONCLUSIVE`, else `1` if any `FAIL`, else `0` | **`refuse` precedence.** Under `refuse`, an `INCONCLUSIVE` result exits `2` *even if another assertion also `FAIL`ed* -- the exit-2 refusal takes precedence over the FAIL. A run that cannot fully see its inputs withholds its verdict. The **default is `report`**, so a suite that sets nothing gates exactly as before this field existed -- fully backward-compatible. **CI and compliance suites should set `fail` or `refuse`**, so a missing transcript or trace fails loudly. Set it as an optional top-level key: ```yaml version: 1 inconclusive_policy: fail # or: refuse | report (default) assertions: - id: disclosure kind: phrase regex: "recorded for quality" role: agent ``` or override the document's key from the CLI (the flag wins): ```bash hotato assert run --assertions assertions.yaml --transcript call.json \ --inconclusive-policy refuse ``` or from Python (an explicit argument overrides the document's key; absent both, `report` applies): ```python env = A.run_assertions_from_file("assertions.yaml", ctx, inconclusive_policy="fail") ``` A bad value (anything but `report`/`fail`/`refuse`) -- in the document or passed explicitly -- is a usage error (`ValueError`, exit `2`), raised during validation before any assertion runs. The envelope always carries the applied `inconclusive_policy`, stated with the counts in `summary.note`. ## Mapping to a CI gate Same exit-code convention as every hotato command: | Exit | Meaning | | ---: | --- | | `0` | every assertion passed (under `report`, an `INCONCLUSIVE` reports missing input and leaves the exit at `0`; under `fail` it gates like `FAIL`; under `refuse` it exits `2`) | | `1` | at least one deterministic status is `FAIL` (or, under `fail`, `INCONCLUSIVE`) | | `2` | a refusal under `refuse` (an `INCONCLUSIVE`, taking precedence over a `FAIL`), or a malformed file / bad input, raised before any assertion runs | ```bash python3 - <<'PY' from hotato import assert_ as A import sys ctx = A.build_context(transcript_path="transcript.json", trace_path="voice_trace.jsonl") env = A.run_assertions_from_file("assertions.yaml", ctx) print(env["summary"]["note"]) sys.exit(env["exit_code"]) PY ``` A gate on `assert.v1` and a gate on the timing scorer's `exit_code` (`hotato run` / `hotato verify`, [`docs/CI.md`](CI.md)) are two different, composable guarantees: one gates turn-taking timing, the other gates transcript/trace content. Run both; neither exit code substitutes for the other's. ================================================================================ FILE: docs/BAD-CALL-TO-CI.md ================================================================================ # From a bad call to a CI gate Turn one bad call into a permanent, offline regression test. Five steps, audio included, all local. ## Start from your own provider call Have a call id instead of a WAV? Pull, review, label, and open the pull request in three commands: ```bash hotato investigate --stack vapi --call-id hotato investigate label .hotato/investigate-state.json# --expect yield --reviewer you hotato pr create --fixtures contracts/.hotato --repo you/your-repo --title "Add hotato contract " ``` `investigate` prints a ranked candidate for each timing moment with the exact label command. `investigate label` writes a signed contract bundle to `contracts/.hotato/` and prints the exact `pr create` command for it. `pr create` accepts that bundle directly: it stages the bundle byte-identical under `tests/hotato/contracts/` and opens the pull request (dry run by default; `--yes` runs git and gh). CI then gates on `hotato contract verify tests/hotato/contracts/`, which exits non-zero when a contract regresses. Prefer a plain scenario + audio fixture over a contract bundle? The same labeled moment lands as one through the alternate sequence, and the `hotato run` CI gate below applies to it: ```bash hotato fixture promote .hotato/investigate-state.json# --expect yield --out tests/hotato hotato pr create --fixtures tests/hotato --repo you/your-repo --title "Add turn-taking regression fixtures" ``` ## The label comes from you Yield means the agent should stop for the caller. Hold means it should keep talking through a backchannel, noise, or acknowledgement. Hotato measures whether the timing matched your label. `"mhm"` and `"stop"` can carry identical speech energy -- no timing measurement tells them apart. So Hotato measures the one thing timing can settle: did the agent do what your label says, and how fast. The main scorer needs separated tracks: one two-channel WAV, or two aligned mono WAVs, enough to attribute talk-over reliably. ## Step 1: turn the moment into a fixture Say the agent talked over a caller at 42.18 seconds. Label it: ```bash hotato fixture create --stereo bad-call.wav \ --id refund-interruption-001 \ --onset 42.18 --expect yield \ --max-talk-over 0.6 --max-time-to-yield 1.0 \ --out tests/hotato ``` Writes the labeled scenario with provenance (`tests/hotato/scenarios/refund-interruption-001.json`) and a two-channel audio clip (`tests/hotato/audio/refund-interruption-001.example.wav`), scored on creation. An unjudgeable input is refused with the reason, exit code 2. Don't know the onset? List the candidates first: ```bash hotato scan --stereo bad-call.wav ``` `scan` reports timing facts only -- overlap, an agent starting over caller speech, long gaps, dead air, or suspected TTS echo. You pick the moment and supply the label. ## Step 2: run it ```bash hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio ``` The bad take fails, by design: that's the regression you're pinning. Exit codes are the contract: 0 all pass, 1 a regression, 2 unusable input. ## Step 3: fix your agent, then compare Change the setting, re-capture the same scenario, let the numbers speak: ```bash hotato compare --before bad-call.wav --after new-take.wav \ --onset 42.18 --expect yield ``` ``` hotato compare: bad-call.wav -> new-take.wav verdict: FAIL -> PASS did_yield: false -> true seconds_to_yield: - -> 0.42s talk_over_sec: 2.65s -> 0.42s improved -2.23s result: fixed ``` Moment shifted between takes? Pass `--before-onset` and `--after-onset` separately. `--out report.html` writes a shareable HTML report. An unjudgeable side renders NOT SCORABLE and exits 2. ## Step 4: gate CI on it ```yaml # .github/workflows/turn-taking.yml name: turn-taking on: [pull_request] jobs: hotato: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: { python-version: "3.12" } - run: pip install hotato - run: hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json ``` `hotato run` exits 1 on a regression, failing the job, and exits 2 when the battery could not tell anything: an empty `--scenarios` directory, or a battery whose every event is not scorable (a wrong `--audio` path, audio artifacts the job never fetched). Both read red -- a battery that scored nothing gates exactly like `hotato prove`'s "a proof of nothing is refused". Already running pytest? `pytest --hotato-suite --hotato-suite-scenarios tests/hotato/scenarios --hotato-suite-audio tests/hotato/audio` adds the same gate (see [PYTEST.md](PYTEST.md)). The richer PR check with a sticky results comment is in [CI.md](CI.md). ## Step 5: when it fails, plan the fix ```bash hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \ --format json > result.json hotato plan result.json --stack vapi --assistant-id ``` `plan` is read-only and guarded: one bounded step on one setting at most, never a single-threshold fix when the battery fails on both axes at once, and a checklist when the evidence can't isolate the layer. It only proposes -- `platform_mutation.performed` is always false. Details: [FIX-PLANS.md](FIX-PLANS.md). ## Use Hotato when - You have (or can capture) separated caller/agent audio. - You can point at a moment and label it: yield or hold. - You want a local, deterministic CI regression test on turn-taking timing. ## What Hotato measures Turn-taking timing from separated audio tracks: talk-over, response gaps, and yield/hold timing against your label. Not transcript wording, call outcome, sentiment, compliance, or tool-call behavior. Not live, in-call decisions either -- Hotato scores recordings after the fact. And `hotato scan` surfaces candidates only; every verdict still needs your yield or hold label. ================================================================================ FILE: docs/BENCH-SPEC.md ================================================================================ # hotato bench: specification v0.1 hotato bench is a versioned freeze of the scenario batteries this repository already ships, a scoring protocol over them, and a verify command that re-executes a result and compares hashes. It is a benchmark you can hold in your hand: a fixed set of labelled recordings, a fixed scorer, and a result file whose numbers anyone can recompute from the same bytes. It is not a leaderboard. Run it: ```bash hotato bench run --out bench-bundled.json # the packaged battery hotato bench run --suite gold --out bench-gold.json hotato bench verify bench-gold.json # re-execute + hash-compare ``` ## Neutral harness hotato is the measuring instrument, never a party to the comparison. This repository publishes exactly two kinds of rows: - rows measured on hotato's own reference renders (the frozen batteries below, scored as shipped); - rows measured on fully open-source baseline agents, named and pinned by commit. Rows for any vendor stack are adopter-published: the adopter who operates that stack runs the bench, publishes the result in their own repository under their own provenance block, and owns the claim. This repository never authors a comparative row for a named vendor, and a pull request adding one is declined. ## Scope: the frozen set Bench v0.1 freezes the four tiered batteries under `corpus/suites/` (112 scenarios; inventory in `corpus/suites/manifest.json`) plus the packaged 8-scenario battery that installs with the package: | Battery | Tier | Scenarios | Conditions | Suite content hash | | --- | --- | --- | --- | --- | | `bundled` | (package) | 8 | the packaged barge-in battery; runs from any install | `sha256:a02a4550c65fc0c8c897c523dac14d92bfb70b58cef6c8b1b70c59e2b6af8f5f` | | `silver` | silver | 40 | clean, 16 kHz, default noise floor | `sha256:9ce2d70e9499880ca8b692796c4c7329dba8c15e6db579e2aa8ae7e8816e6c97` | | `silver-defects` | silver | 16 | clean conditions, deliberate defect renders | `sha256:db948076fe9f617cbf1d901ef552a56dbb4858a4b2b7cd0f46e4b9244e079713` | | `gold` | gold | 40 | noise floors, 8 kHz telephony, gain extremes, echo, edge timings, endurance | `sha256:f143781dbd1d09d3ca1bb1776ee0e2b2cf3c995f73e134aebedeee06575a307b` | | `gold-defects` | gold | 16 | hard-condition defect renders, including two labelled capture-defect cases | `sha256:d1fd0bebb98650d51d6672861b4b29c9cbf3c061c46ef5ecd0bf0712388c2ea2` | Every scenario is synthetic and says so: deterministic shaped noise rendered from the exact segment timings in its own JSON (seed = `sha256(scenario_id)`), so the timings are the ground truth and a fresh render is byte-identical to the committed battery (`python3 corpus/suites/build_suites.py --check` is the proof, and CI runs it). The defect batteries fail by design: they are the negative control that makes the passing batteries mean something (`docs/SUITES.md`). A suite content hash pins the exact files a run consumes: one line per file, `\0\n` over the sorted scenario JSONs and their `.example.wav` recordings, hashed with sha256 (`hotato.bench.suite_content_hash`). ## Versioning - **Bench version** (`bench_version`, semver, currently `0.1`): the frozen set and the protocol together. Any change to any pinned file, to the audio suffix, to the hashed body shape, or to the hashing rules is a new bench version. The table above is the v0.1 pin list; a row change and a version bump land in the same commit. - **Result schema** (`schema_version`, currently `1`): the shape of the result document. Additive-only within a major; a consumer must ignore unknown fields. - **Scorer identity**: every result embeds the vendored engine identity (`engine`) and the full default `ScoreConfig` snapshot (`config`), so a result names the exact scorer and thresholds that produced it. ## Scoring protocol Bench rows measure talk over on the hangover smoothed activity tracks; relative to raw energy labels the end bias bound is `hangover_sec` plus one hop (see docs/BENCHMARK.md, Quantization). This convention is part of the frozen v0.1 protocol; changing it is a new bench version and a new engine identity. One result per battery, three measurements, side by side and never blended: - **Pass counts** (`pass_counts`): scenarios, passed, failed, and not-scorable counts from the standard envelope (`hotato.core.run_suite` under the default shipped config). Results group by the battery's tier; a tier is a grouping label, never a weight. - **Measurement-error distributions** (`error_stats_ms`): per signal (caller onset, time to yield, response gap), the n / median / mean / max / min of `|measured - rendered|` in milliseconds, against each scenario's own `reference_render` timings (`hotato.benchmark.run_benchmark`). A signal with no rendered ground truth reports `n: 0` and nulls, never a fabricated reference. - **Confusion cells** (`confusion`): the four `did_yield` cells (correct_yield, missed_yield, false_yield, correct_hold) plus their off-diagonal sum. There is no blended score. No field anywhere in a bench result aggregates across these measurements, and no accuracy percentage appears; collapsing the error distribution and the confusion matrix into one figure hides exactly the missed-yield / false-yield trade-off the bench exists to surface. This is code-enforced: the repo-wide `overall_score` rejection (`hotato.errors.reject_overall_score`) applies to every nested section of a bench result, and `bench verify` refuses a result that smuggles one in. ## Reproducibility Byte-for-byte means: two executions of `bench run` on the same frozen battery under the same package version produce result bodies whose canonical JSON bytes are identical, so their sha256 addresses are equal. Canonical JSON here is the form the rest of the repository already uses (`hotato.manifest.canonical_json`): sorted keys, compact separators, ASCII, finite numbers only. The result carries no wall-clock field, so its address is stable across re-runs. The whole path is offline: the batteries are local files (or packaged data), the scorer is the vendored stdlib-only engine, and no code path in `bench run` or `bench verify` touches the network. A machine with the package and the frozen batteries can produce and check every number in a result with no account, no key, and no service. ## Submission model: adopter-published rows An adopter publishes a row by committing the evidence, not by sending a score: 1. run the bench in their own CI: `hotato bench run --suite --out results/.json`; 2. gate the row on re-execution in the same pipeline: `hotato bench verify results/.json` (exit 0 required); 3. commit the result files and a provenance block to their own repository, and render whatever badge they choose from their own CI status. The provenance block published next to the rows carries, at minimum: | Field | Content | | --- | --- | | `bench_version` | the bench semver the rows were measured under | | `suite` / `suite_content_hash` | each battery name and its pinned hash, from the result | | `result_content_hash` | each result's canonical sha256 address | | `engine` | the vendored scoring-engine identity from the result | | `config` | the `ScoreConfig` snapshot from the result | | `runner` | OS, Python version, and CI system the rows were produced on | | `commit` | the adopter repository commit the rows were produced from | | `published_by` | who stands behind the row | A row is the adopter's claim, verified by the adopter's own re-execution. This repository links to adopter rows as published; it does not restate, rank, or merge them. ## Anti-gaming - **Verification is re-execution.** `bench verify` re-runs the pinned battery through the same code path `bench run` uses and compares the two canonical result addresses. The verdict is a hash comparison of two executions on the verifier's machine; no judge, no model, no service scores anything. - **Edits are caught twice.** A result edited in place fails its own embedded `content_hash` and is refused. A result whose body and hash were both rewritten passes the integrity check and is then caught by re-execution, because the recomputed body hashes differently. - **The frozen set is pinned.** A verify against a battery whose content hash differs from the one the result pins is withheld, not guessed: the comparison only ever runs like against like. - **The batteries are public, and the pin makes that safe to say.** A stack tuned to the published scenarios still has to produce its numbers through the pinned scorer at the pinned config, and its provenance block names the exact config that produced every number. - **The held-out check is a manual maintainer spot check.** A maintainer re-renders the frozen batteries from the generator, re-executes a published row by hand on maintainer hardware, and hash-compares. There is no hosted scoring server to game: every verdict, including the spot check, is a local re-execution. ================================================================================ FILE: docs/BENCHMARK-STACKS.md ================================================================================ # Benchmarking voice stacks with hotato `hotato benchmark` runs one fixed scenario battery through your configured voice stack, scoring every stack against the same scenarios, labels, and thresholds -- so result files line up directly, stack against stack. Every number comes from the recordings you capture, scored the same way for every stack -- a benchmark result describes your setup, on the day you captured it. ## 1. Pick the scenario set The bundled 8-scenario barge-in battery is the default. Any scenarios directory works the same way, including the larger tiered suites: ``` corpus/suites/gold/scenarios corpus/suites/silver/scenarios ``` Each scenario JSON carries the caller transcript, the caller onset timing, and the pass thresholds; the corpus suites also ship each scenario's caller stimulus as `audio/.caller.wav`. ## 2. Capture the battery through your stack For every scenario, drive the caller side through your stack and record the call dual-channel (caller on channel 0, agent on channel 1): - Corpus suites: play the shipped `.caller.wav` stimulus into your stack. For the bundled battery, reproduce the caller side from each scenario's transcript and `caller_onset_sec`. - `hotato setup --stack livekit` or `--stack pipecat` prints the exact dual-channel recording scaffold for your infra. - `hotato capture --stack vapi --call-id ` or `hotato capture --stack twilio --recording-sid RE...` pulls the finished dual-channel recording for you. Name each recording by its scenario id, one directory per stack: ``` captures/livekit/01-hard-interruption.wav captures/livekit/02-backchannel-mhm.wav ... captures/vapi/01-hard-interruption.wav ``` ## 3. Run the benchmark ```bash hotato benchmark --stack livekit --recordings captures/livekit --out livekit.json hotato benchmark --stack vapi --recordings captures/vapi --out vapi.json ``` Each result JSON records the stack, the scenario set, every measured signal (the same events `hotato run` produces), the exact scoring thresholds, and a provenance block: who ran it, on which files, with each file's mtime. The timestamp comes from the input files, not the wall clock, so the same inputs reproduce the same result. Scenarios without a matching recording land under `not_captured`, kept out of both the scoring and the failure count. Exit code is 0 when the run completes; add `--fail-on-regression` to exit 1 when a scored event fails its thresholds. ## 4. Compare stacks ```bash hotato benchmark compare livekit.json vapi.json ``` Side by side, per scenario: yielded, talk-over seconds, and time-to-yield seconds for each input, with signed deltas against the first file, plus summary medians. When the files cover different scenario sets, only the intersection is compared; the rest is listed as skipped, with the files each was missing from. ## What the numbers are Reproducible timing measurements of the recordings you provided, scored by the same engine as `hotato run`, with every threshold exposed in the result. They show how your configuration of a stack handled the battery you captured. ================================================================================ FILE: docs/CALLER-LOAD.md ================================================================================ # Caller-program load runner `hotato.caller_load` executes the same bounded caller program many times without discarding the evidence from any started invocation. It is the load plane for Hotato's caller engine, rather than a shortcut that treats an HTTP response or a completed call as a quality pass. The runner supports two workload models: - **closed concurrency** keeps up to `concurrency` caller invocations active until an exact call count has been attempted; - **open arrival** schedules starts against a monotonic arrival clock. When the target rate exceeds `max_in_flight`, the scheduled start is recorded as `DROPPED`. The scheduler does not wait and silently shift that call later. Every started invocation runs in its own operating-system process and writes a normal `hotato.caller-package.v1` package. The aggregate package binds each child to the workload-plan hash, stage ID, stage model, and invocation index. This lets an offline verifier detect a child copied from another stage or run. ## Minimal plan ```json { "schema": "hotato.caller-load-plan.v1", "id": "refund-release-candidate", "caller_plan": { "schema": "hotato.caller-plan.v1", "id": "refund-caller", "mode": "scripted", "start": "ask", "nodes": [ {"id": "ask", "type": "say", "text": "Please refund order 42.", "next": "done"}, {"id": "done", "type": "hangup", "reason": "scenario_complete"} ], "limits": {"max_duration_ms": 30000, "max_cost_microusd": 0} }, "stages": [ {"id": "warm", "model": "closed", "concurrency": 4, "calls": 20}, { "id": "arrival-spike", "model": "open", "arrival_rate_per_second": 10, "duration_seconds": 30, "max_in_flight": 50 } ], "safety": { "max_calls": 320, "max_concurrency": 50, "max_call_duration_ms": 30000, "max_start_delay_ms": 250, "max_cost_per_call_microusd": 0, "max_cost_microusd": 0, "stop_file": "./STOP-HOTATO-CALLER-LOAD" }, "slos": { "max_dropped_start_rate": 0.01, "min_completion_rate": 0.99, "max_blocked_error_rate": 0.01, "min_child_verification_rate": 1.0, "min_evidence_complete_rate": 0.99, "max_p95_scheduling_delay_ms": 100 } } ``` Open stages gain a normalized, read-only `calls` field equal to the number of scheduled arrival slots. The stored `workload-plan.json` contains that normalized value. ## Adapter boundary The runner does not put credentials in the workload plan. Supply credentials through a factory closure or a secret manager. Every API call declares its session-factory transport scope; Hotato cannot infer what arbitrary Python factory code opens. ```python from hotato import caller_load from my_project.sessions import new_session plan = caller_load.load_plan("caller-load.json") def session_factory(run_context): # run_context is safe to attach to transport receipts and traces. # It contains child_id, workload_plan_sha256, stage_id, stage_model, # invocation_index, and expected_session_boundary. Fetch credentials # outside the plan. return new_session( target_url="ws://127.0.0.1:9000/voice", token_from_secret_manager=True, evidence_labels=run_context, ) run = caller_load.run_caller_load( plan, "./hotato-caller-load-run", session_factory, execution_scope="local", ) raise SystemExit(run.exit_code) ``` `model_factory(context)` and `tts_factory(context)` use the same contract and are optional. A `generative` caller plan requires `model_factory`. Each factory runs inside the child process. On platforms whose multiprocessing runtime uses `spawn`, factories must be pickleable module-level callables. Session and model credentials can still leak if an adapter deliberately writes them into its `evidence()` output. Adapter authors must return receipts and content hashes, never bearer tokens, authorization headers, or signed URLs. ## Safety contract Validation happens before a child process starts: - scheduled calls cannot exceed `safety.max_calls`; - stage concurrency cannot exceed `safety.max_concurrency`; - the supervisor terminates a worker after `max_call_duration_ms` and writes a separate verified `ERROR` child package for that invocation; - caller-plan duration and model-cost ceilings are narrowed to the workload ceilings for each child; - `scheduled_calls × max_cost_per_call_microusd` must fit inside `max_cost_microusd` before execution; - the stop file prevents every subsequent scheduled start. Those rows are `STOPPED`, not silently omitted, and the aggregate status is `INCONCLUSIVE`. The cost bound depends on model adapters reporting usage correctly to the caller engine. A provider with unknown or unbounded spend must not be assigned a zero cost ceiling. On timeout, the worker attempts `session.hangup()` and an optional `session.close()` before the supervisor escalates from termination to a process kill. Carrier/provider billing and teardown remain external evidence; an adapter that ignores teardown cannot be made cost-bounded by a local process limit alone. ## Coordinated-omission evidence Each scheduled row records: - its fixed `scheduled_offset_ms`; - observed `scheduling_delay_ms`; - `STARTED`, `DROPPED`, or `STOPPED` disposition; - `MAX_IN_FLIGHT` or `START_DELAY` for a dropped open-arrival start. Open-arrival capacity pressure is therefore visible as dropped work. The scheduler does not convert a 10 calls/second workload into a slower closed loop whose latency distribution looks healthier. ## Separate result lanes The aggregate and every stage report these values independently: 1. `scheduled`, `started`, `dropped_starts`, and `stopped_before_start`; 2. caller status counts: `completed`, `hung_up`, `blocked`, and `error`; 3. caller child-package `verified` and `unverified` counts; 4. delivered-evidence states: `present`, `missing`, `unsupported`, and `unobservable`; 5. remote session-endpoint bindings: `matched`, `missing`, `mismatch`, and `not_required`; 6. scheduling-delay and invocation-duration distributions; 7. reported model cost in micro-USD. `COMPLETED` and `HUNG_UP` mean the caller program reached a terminal state. They do not mean the target agent produced the right outcome. `PRESENT` delivery evidence means the session explicitly reported target-bound evidence; it is also not an outcome verdict. The schema fixes both statements and fixes `blended_score` to `null`. ## Offline verification ```python from hotato.caller_load import verify_caller_load verification = verify_caller_load("./hotato-caller-load-run") assert verification["ok"], verification["mismatches"] ``` Verification performs no network, model, TTS, session, or provider call. It: - uses bounded no-follow reads and refuses FIFOs and symlinks; - verifies the full aggregate file manifest and exact package layout; - verifies every started child with `caller.verify_package`; - rebuilds the expected child caller plan and checks its workload/stage/index identity; - recomputes child status, exit code, evidence state, and reported cost from the child package; - recomputes aggregate and per-stage metrics, every SLO, aggregate status, and exit code; - rejects rehashed status/exit forgeries, child swaps, and extra files. Exit codes are `0` for `PASS`, `1` for `FAIL`, and `2` for `INCONCLUSIVE`. Only declared SLOs gate `PASS`/`FAIL`; the runner never invents a blended score or an undeclared quality threshold. `delivery_evidence=PRESENT` has a narrow contract. Credit can come from either an exact `session_boundary.delivery_evidence` receipt or an exact `hotato.delivered-audio.v1` custom child event. Both forms require an authority of `target_boundary`, `target_participant_reported`, or `carrier_boundary`, the workload and child IDs from the supplied run context, and both `submitted_sha256` and `delivered_sha256`. The submitted digest must match PCM emitted by that child; every emitted PCM digest must be covered. The event form is documented in `CALLER-SIDECAR-PROTOCOL.md`. A malformed claim, generic digest, configured impairment hash, SDK submission receipt, partial receipt set, or summary state cannot earn delivery credit. Those cases remain `MISSING` or `UNOBSERVABLE`. This is reporting-boundary evidence, not a packet trace or proof about a later uninstrumented hop. Remote sidecar load is default-deny. `--allow-remote` is accepted only when the normalized workload plan contains an exact credential-free `wss://` endpoint, a remote call ceiling covering the schedule, the fixed `I_ACCEPT_REMOTE_CALL_SIDE_EFFECTS_AND_UNOBSERVABLE_EXTERNAL_COST` acknowledgement, and `external_cost_state=UNOBSERVABLE`. Queries and embedded credentials are refused. The result separates the plan-declared endpoint digest from the runtime-configured endpoint digest. Every started child also must return the same `connected_endpoint_sha256` from its successfully constructed session boundary. `metrics.session_endpoint_binding` exposes matched, missing, and mismatched children. Any missing or mismatch makes the aggregate `INCONCLUSIVE` with exit code 2, even when quality SLOs pass; the offline verifier recomputes that state from each child. The result records external provider cost as `null`/`UNOBSERVABLE`; `actual_cost_microusd` and its bound cover caller-model-reported cost only. A remote provider bill never silently appears as zero. ## Package layout ```text hotato-caller-load-run/ ├── workload-plan.json ├── observations.jsonl ├── result.json ├── package-manifest.json └── children/ └── / ├── caller-plan.json ├── caller-result.json ├── package-manifest.json └── artifacts/... ``` The aggregate manifest hashes every byte-bearing file, including each child manifest. Child packages retain the caller engine's model, TTS, text, PCM, event, session-boundary, and authority evidence. ## Scope boundary This module exercises caller programs against a supplied session adapter. It does not provision SIP trunks, carrier capacity, media relays, speech models, or production alerting. Those components remain independently observable so a provider lifecycle response cannot substitute for target-delivered media or a target outcome. ================================================================================ FILE: docs/CALLER-SIDECAR-PROTOCOL.md ================================================================================ # Caller sidecar protocol `WebSocketCallerSession` is the concrete boundary between Hotato's bounded caller program and a Pipecat, LiveKit, SIP, or provider adapter. The WebSocket subprotocol is `hotato.caller.v1`. Loopback is the default; remote egress is an explicit option and requires `wss://`. The client sends a `hotato.caller-session.v1` hello with a random nonce and the complete operation list. The sidecar returns a nonce-bound `ready` message, adapter name/version, and one of `SUPPORTED`, `UNSUPPORTED`, or `UNOBSERVABLE` for every operation. An omitted capability refuses the session. Commands are canonical JSON envelopes with a contiguous sequence. PCM16LE is announced by a `send_audio` command containing byte count and SHA-256, followed by one binary message: the four bytes `HTC1`, a big-endian 32-bit command sequence, and the exact PCM bytes. Every command is synchronous at the protocol boundary. After accepting the complete command (including the binary frame for `send_audio`), the sidecar must return the matching sequence and command: ```json { "schema": "hotato.caller-session.v1", "type": "command_result", "sequence": 4, "command": "send_audio", "status": "completed", "receipt": { "accepted_bytes": 6400, "accepted_sha256": "sha256:..." } } ``` Status is `completed`, `unsupported`, or `error`. An absent, mismatched, or failed result stops the caller program. Events that arrive while a command is pending are retained in order for the next `receive` node; any still queued at the terminal node are drained into `caller-result.json.events`. A command result establishes that the sidecar accepted the operation. It does not establish that a carrier or target agent received the media; that stronger claim needs a target-boundary event carrying the delivered stream hash. For caller-load delivery credit, the target or carrier boundary emits this exact custom event (the caller engine adds `sequence` and `event_sha256`): ```json { "schema": "hotato.caller-session.v1", "type": "event", "event": { "kind": "custom", "custom_type": "hotato.delivered-audio.v1", "authority": "target_boundary", "submitted_sha256": "sha256:<64 lowercase hex>", "delivered_sha256": "sha256:<64 lowercase hex>", "workload_child_id": "", "workload_plan_sha256": "" } } ``` `authority` is exactly `target_boundary`, `target_participant_reported`, or `carrier_boundary`. The submitted digest must identify PCM emitted by that child. Every emitted PCM digest needs a valid receipt before the aggregate reports delivery evidence `PRESENT`. Unknown fields, malformed identities, replayed child/workload bindings, or partial coverage remain `MISSING`. `delivered_sha256` identifies bytes observed by the reporting boundary; it is not a packet trace or evidence about an uninstrumented downstream hop. Incoming data is a JSON event envelope: ```json { "schema": "hotato.caller-session.v1", "type": "event", "event": {"kind": "transcript", "text": "How can I help?"} } ``` The caller engine accepts bounded transcript, tool-result, state-snapshot, DTMF, lifecycle, transfer, hold, timing, timeout, and custom events. Sidecar events are adapter evidence. A configured impairment receives delivery credit only when the target boundary emits evidence for the bytes it received. Handshake headers may carry credentials, but values are resolved inside the worker and never serialized into multiprocessing factories, control messages, or caller packages. Plaintext `ws://` is accepted only for loopback targets. The strict WebSocket client refuses redirects, extensions, remote addresses unless opted in, oversized messages, invalid framing, and malformed JSON. After the nonce-bound ready exchange, the session records `connected_endpoint_sha256`, computed from the normalized credential-free WebSocket URL. This binds remote load children to the configured endpoint. It does not prove media delivery. ================================================================================ FILE: docs/CARDS.md ================================================================================ # Cards: one measured moment, rendered for the pull request `hotato card` turns a machine result into a self-contained SVG for a pull request or an issue. The card names the measured timing moment: a reproducible measurement, not a verdict about intent, with no accuracy number and no blended score anywhere on it. The PR-native artifact the shipped Action lands automatically is the [Failure Record](#the-failure-record-the-artifact-that-lands-in-the-pull-request) below; a card is the single-moment image you attach yourself. ```bash hotato card INPUT[#REF] --out card.svg ``` Everything runs locally. The SVG is a pure function of the input JSON alone (the same input renders the same bytes forever), with every color inlined and no font, image, stylesheet, script, or link to fetch. It renders wherever the PR renders; no CDN or asset host required. ## The cards, auto-detected The input's kind decides the card; you do not pick. | Input | Card | |---|---| | a `sweep`/`analyze` candidate ref, `FILE#N`, that is a talk-over moment | **talk-over candidate** | | a `sweep`/`analyze` candidate ref, `FILE#N`, that is a false-stop moment | **false-stop candidate** | | a fix plan whose `decision` is `do_not_tune_single_threshold` | **threshold funnel** (the hero) | | a supported `hotato verify` before/after rollup that improved | **paired comparison** | | a `hotato contract create` contract (kind `voice-turn-taking-contract`) | **failure contract** | | a `hotato test run` result (kind `hotato.test-run`) whose tool/state evidence failed a declared outcome | **say-do failure** | `#N` is the same 1-based rank the sweep report and dashboard show, and the same ref `hotato fixture promote` takes -- a card and a fixture speak of the exact same moment. ### A. Talk-over candidate An `overlap_while_agent_talking` (or `agent_start_during_caller`) moment: the agent kept the floor while the caller was speaking. The card leads with the measured overlap in seconds and closes with "Hotato reports timing candidates, not intent." ```bash hotato sweep --demo --format json > hotato-sweep.json hotato card hotato-sweep.json#3 --out talk-over.svg ``` ### B. False-stop candidate An `agent_stop_no_caller` moment: the agent went quiet with no caller nearby to explain the drop. The card leads with the measured trailing silence. ```bash hotato card hotato-sweep.json#1 --out false-stop.svg ``` ### C. Threshold funnel (the hero) The plan the both-axes case produces: the battery missed an interruption **and** false-stopped on a backchannel, so satisfying both needs more than a single sensitivity dial. The card states Hotato refused threshold tuning and names the fix class (`engagement-control`) -- the card the project leads with. ```bash hotato demo --format json > demo.json hotato plan demo.json --out fix-plan.json hotato card fix-plan.json --out no-single-threshold.svg ``` Only a `do_not_tune_single_threshold` plan renders this card; any other plan is a clean exit-2 usage error (not one of the card kinds). ### D. Paired comparison A supported `hotato verify` before/after rollup where at least one previously-failing fixture now passes and no hold/backchannel fixture regressed. The card reads "PAIRED FRESH-RECAPTURE IMPROVED" only when the recapture is runner-attested and "PAIRED (OPERATOR-ASSERTED)" otherwise, never "verified" or "fix verified", and closes with "Hotato reports coincidence, not causation." A verify result that doesn't support that claim (too few previously-failing fixtures, nothing now passing, or a regressed hold fixture) is refused with exit 2. ```bash hotato card verify.json --out comparison.svg ``` ### E. Failure contract A committed contract's card: the expected behavior (`yield` or `hold`), the measured seconds, and PASSED / FAILED / NOT SCORABLE. The footer reads "A human labeled this contract; Hotato measured the timing." ```bash hotato card contracts/demo-missed-interruption.hotato/contract.json --out contract.svg ``` ### F. Say-do failure A `hotato test run` result (`--format json`, kind `hotato.test-run`) whose deterministic lane failed a tool/state evidence assertion (`tool_result`, `tool_call`, `tool_error`, `http_result`, `state`, `state_change`): the conversation claims an outcome the trace (Authority 1) or the post-call state (Authority 2) does not back. The card renders the claim vs the evidence -- the failing assertion's id and kind, its span refs when the evaluator recorded any, and its share-safe `public_reason` (built from allowlisted structured fields only, never transcript text, a tool payload, or a state value). The failing outcome-tagged evidence assertion leads; the footer reads "Tool and state evidence decide the outcome, never the agent's words." A test-run result with no failing tool/state evidence assertion is refused with exit 2. ```bash hotato start --demo # act two writes saydo/test-run.json hotato card saydo/test-run.json --out saydo-card.svg # or from any of your own runs: hotato test run refund.yaml --agent support-v3 --trace voice_trace.jsonl \ --transcript call.transcript.json --format json > test-run.json hotato card test-run.json --out saydo-card.svg ``` ## Redaction: identifiers stay hidden by default A card is a public image, so identifiers stay hidden by default: call id, filesystem path (only a basename is ever shown), and vendor recording name are omitted. A pulled recording named `STACK__ID.wav` carries the call id inside its name; that name shows only under `--include-identifiers`. ```bash # shows the source recording's basename on a candidate card hotato card hotato-sweep.json#1 --out card.svg --include-identifiers ``` ## The Failure Record: the artifact that lands in the pull request A card is one moment's image you attach yourself. The **Failure Record** is the PR-native trust artifact: the shipped GitHub Action renders one per non-passing unit ([`docs/CI.md`](CI.md)), so it arrives in the pull request with the run that produced it. Each record is a lane-structured projection of one failed, inconclusive, or errored result: - one evidence-specific headline; - the five lanes (outcome, policy, conversation, speech, reliability), each with its own status and never a blended or overall score; - the reproduce command (`reproduction.argv`, rendered in the Markdown, HTML, and SVG forms) plus the pinned one-command verifier (`hotato record verify failure-record.json`); - content-addressed evidence digests and a privacy profile: no audio, transcript body, tool payload, state value, or absolute path, so it attaches to a PR as-is. `hotato start --demo` emits one automatically under `hotato-failure-record/` -- `failure-record.{json,md,html,svg}` -- alongside the sweep and the demo contract, and prints the record paths plus the one-command verifier. Render one from any result yourself with `hotato record render`. ## Output and exit codes Without `--out`, the SVG goes to stdout, so you can pipe it. With `--out` it is written there atomically. - **0**: the SVG card was rendered (to `--out`, or to stdout). - **2**: usage error, unreadable input, a bad candidate ref, or an input not a fix plan / verify result / contract / test-run result / sweep candidate. ## Regenerating the committed cards Four commit-ready examples live under `docs/assets/cards/` (`no-single-threshold-card.svg`, `talk-over-card.svg`, `false-stop-card.svg`, `say-do-card.svg`), rendered from the bundled demo data. Regenerate them with: ```bash PYTHONPATH=src python3 scripts/render_card_assets.py ``` ================================================================================ FILE: docs/COMPARE.md ================================================================================ # Compare: hotato vs hosted platforms Local-first testing and observability for AI agents, next to the hosted platforms that run the same jobs as a service. The short version: the same lifecycle, three structural differences. ## Property by property The hosted platforms (LangSmith, Langfuse, Phoenix, LangWatch, and the voice-specific tools) are capable products: many support deterministic code evaluators, datasets, experiments, human review, and self-hosting. This table compares default properties, not a caricature, and it is honest about where a managed platform is the stronger fit. | Property | hotato default | Typical managed deployment | |---|---|---| | Raw evidence location | your local filesystem or VPC | the vendor's service | | Price at scale | free and MIT, any volume | metered per seat, run, or event | | Deterministic timing lane (turn-taking, say-do) | built in | varies by platform | | Content-addressed contracts + release proofs | built in | platform-specific | | Offline verification, no service dependency | built in | often service-dependent | | Model-judge lane | separate and advisory by default | commonly integrated into the score | | Hosted parallel execution at scale | you operate it | managed by the vendor | | Persistent trace search, datasets, prompt management | limited | commonly built in | | Team collaboration, roles, review queues | local workspace | mature hosted collaboration | ## Where hotato is the clear pick - **Your data stays yours.** Recordings, traces, prompts, and backend state never leave your machine, and a contract or proof stays verifiable if you stop using hotato, if the vendor changes, or if the service is down. - **Verdicts you can gate on.** The deterministic lanes score the same input the same way on every machine, byte for byte, so an exit code can block a merge. A model-judged score that varies run to run cannot. - **Free at any volume.** No per-seat, per-run, or per-event meter anywhere. ## Where a managed platform may fit better A hosted platform is often the faster path if you need mature multi-user collaboration, a persistent searchable trace explorer with saved views, built-in dataset and prompt management, or vendor-operated execution at high concurrency. hotato is deliberately local and CLI-first; those surfaces are limited today. The two compose: point a hosted platform's OTel export at `hotato trace ingest` and keep the deterministic test and release-evidence layer on your own machine. ## The layer hotato is not A runtime voice layer (an orchestration platform's endpointing, a turn detector, barge-in suppression) acts *during* the call. hotato runs *after*, from the recording and the trace: it measures what happened, pins what must not happen again, and proves a candidate against it. Run the runtime layer that makes your median call good; run hotato so the failure a caller hit last week stays fixed on every push. The frozen moment a runtime layer got wrong is exactly what `hotato investigate label` pins and `hotato prove` re-verifies. ## What the scoring claims, precisely - Timing and say-do, not intent: energy over time, tool spans, and post-call state decide verdicts; no output claims emotion or meaning. - Two channels or a timestamped transcript in; a mono or mixed export is refused as NOT SCORABLE, never guessed at. - Five dimensions (outcome, policy, conversation, speech, reliability), each reported on its own evidence; there is no blended score anywhere. - Deterministic checks stay separate from the model-judged rubric lane, which is advisory and local. - Determinism is verified in CI: byte-identical re-runs on Linux x86_64, Python 3.10, 3.11, and 3.12 ([VALIDATION.md](VALIDATION.md) Job 1). ## Every example is scoped to one run A hotato example that shows a call failing on a named stack is labelled a **provider-default** run: one assistant, one configuration, one date, one scripted caller, on that provider's out-of-the-box interruption settings. It demonstrates the threshold funnel scoped to that one run. Any stack, tuned, can pass the same fixtures. hotato publishes fixtures and reproduction steps: a record you can run yourself, not a scoreboard. The full loop, command by command: [`docs/LIFECYCLE.md`](LIFECYCLE.md). ================================================================================ FILE: docs/CONVERSATION-TEST.md ================================================================================ # Conversation tests (`hotato test run`): one file, one call, a per-dimension scorecard A **conversation test** (`hotato.conversation-test.v1`) is one file defining one testable conversation: agent, simulated caller, environment, two SEPARATE assertion lanes, and an explicit success condition. `hotato test run` evaluates a supplied call against it and produces a **conversation artifact** (`hotato.conversation.v1`, evidence bound by sha256) plus a **per-dimension scorecard**. Success is a boolean over a small closed vocabulary of named conditions; every dimension counts on its own, including in `--format json`. This is the single documented end-to-end conversation-QA workflow: ``` hotato scenario init -> hotato test run -> hotato conversation verify (author a file) (evaluate a call) (digest-check the artifact) ``` ## 1. Author a conversation test ```bash hotato scenario init refund-flow --agent support-v3 --out refund.yaml ``` The starter carries two lanes, tagged checks across the five report dimensions, and a boolean `success`: ```yaml kind: hotato.conversation-test version: 1 id: refund-flow agent: support-v3 caller: persona: a customer whose order arrived damaged and wants a refund goal: get a refund for order A-1001 facts: {order_id: A-1001} assertions: # DETERMINISTIC lane: pure, offline, no model. Each result is TAGGED with one # of the five report dimensions (a grouping key, never a weight). deterministic: - id: disclosure-said kind: phrase regex: "recorded for quality" role: agent dimension: policy - id: refund-tool-called kind: tool_call name: issue_refund dimension: outcome - id: lookup-then-refund kind: sequence steps: [{tool: lookup_order}, {tool: issue_refund}] dimension: conversation - id: refund-latency kind: latency tool: issue_refund max_ms: 1500 dimension: speech # RUBRIC (model-judged) lane lands in Phase 3. In Phase 1 each is INCONCLUSIVE # (no model runs); it never counts toward the deterministic result or exit code. rubric: - id: was-empathetic kind: judge_rubric dimension: conversation # inconclusive_policy: fail # CI/compliance suites should set fail or refuse success: # Success is the conjunction of these named conditions -- never a score. required: [all_deterministic_assertions_pass, no_rubric_failure] report_dimensions: [outcome, policy, conversation, speech, reliability] ``` Validate one file or a whole directory (exit 2 on any malformed file): ```bash hotato scenario validate refund.yaml hotato scenario validate ./scenarios --format json ``` The closed vocabularies are enforced structurally (`hotato.conversation_test`): `success.required` is drawn from `all_deterministic_assertions_pass`, `no_deterministic_fail`, `no_rubric_failure`, `no_inconclusive`; `dimension` is one of `outcome`, `policy`, `conversation`, `speech`, `reliability`; an `overall_score` key anywhere is rejected. ## 2. Evaluate a call `hotato test run` takes the file plus whatever evidence you have: a scored recording (`--audio`), an ingested trace (`--trace`, see [`docs/TRACE.md`](TRACE.md)), a transcript (`--transcript`), a post-call state sandbox (`--state`, Authority 2). Each supplied piece feeds the assertions; each absent one leaves the checks that need it `INCONCLUSIVE`. ```bash hotato test run refund.yaml --agent support-v3 \ --audio call.wav \ --trace voice_trace.jsonl \ --transcript call.transcript.json \ --out ./conv-artifact --format html ``` Flow: load and validate the file -> build the evaluation context from the supplied transcript / trace / state / (scored) timing -> evaluate the DETERMINISTIC lane (rubric quarantined -> `INCONCLUSIVE`) -> evaluate `success.required` over the results -> bind evidence into a `hotato.conversation.v1` artifact -> render the unified report + per-dimension scorecard. Summary printed to stdout: ``` hotato test run: refund-flow (agent support-v3) -- exit_code=0 inconclusive_policy: report success: PASS (required: all_deterministic_assertions_pass, no_rubric_failure) [ok] all_deterministic_assertions_pass [ok] no_rubric_failure per-dimension (grouped view; never blended): outcome 1 pass / 0 fail / 0 inconclusive policy 1 pass / 0 fail / 0 inconclusive conversation 1 pass / 0 fail / 0 inconclusive speech 1 pass / 0 fail / 0 inconclusive reliability 0 pass / 0 fail / 0 inconclusive reliability over 1 repeated run(s) of the deterministic lane on the same supplied real recording; pass^k == pass@1 because the deterministic replay is byte-identical (zero run-to-run variance) rubric lane: 1 assertion(s) INCONCLUSIVE (quarantined, Phase 3 -- no model ran) deterministic: 4 pass, 0 fail, 0 inconclusive judge: 0 pass, 0 fail (no judge/rubric kind is built in this release) ``` `--format`: | format | stdout | writes into `--out` | | --- | --- | --- | | `text` (default) | the per-dimension summary | the conversation artifact | | `html` / `md` | the summary (+ where files landed) | the artifact **and** `report.{html,md}` (needs `--audio`) | | `json` | the full machine result | the conversation artifact | ### Exit code The exit code honors the file's `inconclusive_policy` exactly as `assert run` does, rising to non-zero when a `success.required` condition fails: | `inconclusive_policy` | an `INCONCLUSIVE` (missing-input) result | a `FAIL` | | --- | --- | --- | | `report` (default) | does not gate (exit 0) | exit 1 | | `fail` | gates like a FAIL (exit 1) | exit 1 | | `refuse` | withholds the verdict (exit 2, precedence) | exit 1 | A `success.required` failure makes an otherwise-passing run non-zero; a refuse (exit 2) is never downgraded. ### Reliability and repetitions `--repetitions N` runs the deterministic lane N times and reports per-run results, run count, and a reliability aggregate: **pass@1** (single-run pass rate), **pass@k** (>=1 of k passed), **pass^k** (all k passed), plus a Wilson 95% CI. Every run scores the same recording, so the deterministic lane has zero variance and `pass^k == pass@1`. With `N > 1` the aggregate feeds the report's Reliability dimension (`--format html/md`); with none, it shows the empty-state ("not measured: no repeated runs in this report"). pass^k stays its own number, separate from every other dimension and from `overall_score`. ## 3. Verify the artifact The conversation artifact directory binds each child by sha256: ``` conv-artifact/ conversation.json # the manifest (origin real|simulated + bound digests) audio/call.wav # the recording (single dual-channel form) transcript.json trace.jsonl timing.json # the scored envelope assertions.json # the evaluated assert.v1 envelope report.html # the unified report + per-dimension scorecard ``` `hotato conversation verify` re-hashes every bound child and REFUSES (exit 2) on any digest mismatch or missing file: ```bash hotato conversation verify ./conv-artifact # conversation refund-flow: VERIFIED # verified: assertions, audio, timing, trace, transcript # all 5 bound artifact(s) re-hashed to their recorded digest ``` `origin.kind` is `real` by default (a supplied recording is evaluated as-is) and `simulated` only when the test file carries a `simulator` block, which must declare its `model_id` / `scenario_id` / `seed`; synthetic and live origins are never conflated. ## The deterministic/judge split (structural) * **Each dimension scored on its own.** Success is a boolean conjunction of named conditions; the scorecard groups results by dimension, each with its own counts. * **Two separate lanes.** Deterministic checks (regex / checksum / span / state lookup, no model) run; the model-judged rubric lane stays quarantined until Phase 3: `INCONCLUSIVE`, out of the deterministic summary and exit code. * **Grounded in authority.** `tool_result` / `tool_error` (Authority 1) read the ingested trace spans; `state` / `state_change` (Authority 2) query a post-call state adapter; the trace and the adapter are the evidence a tool ran or a state changed, deterministic and model-free. * **Missing input is `INCONCLUSIVE`,** never a guessed pass or fail. ## The bundled end-to-end example `tests/data/conversation/` ships one worked call: a conversation-test file, a transcript, and a `voice_trace.v1` trace, evaluated against the bundled recording `01-hard-interruption.example.wav`. `tests/test_test_run_cli.py` (`test_bundled_call_one_file_end_to_end_scorecard_and_artifact`) drives it: one call evaluated for outcome / policy / timing / transcript-facts / tool-behaviour from one file, producing an artifact that `conversation verify` passes and a five-dimension scorecard. ## See also * [`docs/ASSERTIONS.md`](ASSERTIONS.md): deterministic assertion kinds. * [`docs/TRACE.md`](TRACE.md): ingesting a `voice_trace.v1` trace. * [`docs/REPORTS.md`](REPORTS.md): the unified report and scorecard. ================================================================================ FILE: docs/CORPUS-GOVERNANCE.md ================================================================================ # Corpus governance The rulebook for contributing labeled voice-agent calls to hotato's corpus: why they matter, consent, PII, and what gets published. See "Validity metrics" at the end for what a report's numbers mean. --- ## Why real recordings matter The synthetic fixtures in `src/hotato/data/` are a deliberately conservative floor: rendered from known segment timings, so ground truth is exact and the eval runs offline in seconds. They're the regression backbone -- proof the scorer behaves as specified. Production validity is a different question. Synthetic audio has clean onsets, controlled overlap, and no room acoustics or codec artifacts beyond what we inject. A scorer that looks flawless on synthetic fixtures still needs proving on an 8 kHz call with echo bleed, cross-talk, and a caller trailing off mid-word. A small, carefully labeled corpus of real calls closes that gap: the difference between "the tool does what the spec says" and "the tool measures what happens on a phone line." The corpus stays small and high-integrity on purpose -- every clip is consented, de-identified, and human-labeled. Ten trustworthy calls beat ten thousand scraped ones. --- ## Scope and boundaries - The corpus measures the **audio timing of turn-taking**: did the agent stop when the caller interrupted (yield), how long both talked at once (talk-over), and did the agent talk through a short "mhm" (backchannel handling). - Labels are about **events in time**: did the agent stop, and when did the caller start. - Everything here redistributes under the project's MIT license. A clip enters the corpus only if it can be released under MIT with consent. --- ## Consent / release template Every party audible on a recording agrees to its inclusion, in a form you can produce on request. Below: a short, reusable paragraph. Send it to the caller and/or agent-operator, keep the signed or written reply on file, and fill in the brackets. > **Recording release, hotato open test corpus** > > I, [name or role], took part in the audio recording made on [date] and described > as [short description]. I understand this recording will be used as a test > fixture in *hotato*, an open-source project, and grant a perpetual, worldwide, > royalty-free right to store, redistribute, and publish the recording and its > derived timing labels as part of that project's MIT-licensed corpus. I confirm > that the recording contains no confidential, personal, health, or account > information, or that any such information has been removed or replaced before > submission. I understand I can request removal of the recording from future > releases at any time by contacting the maintainers. > > Signed / affirmed: [name], [date], [contact] Notes: - Get consent from **all** identifiable parties: the caller, and whoever runs the agent side, whether that's a person or a proprietary system whose owner must agree. - Consent is **affirmative and documented**. A generic call-center "this call may be recorded" notice doesn't clear the bar for redistribution under an open license. - Honor removal requests in the next release. --- ## PII policy Audio is de-identified before it is committed, in order of preference: **do not capture it, then remove it, then reject the clip.** Strip or redact: - **Names** of any individual (caller, agent, third parties mentioned). - **Phone numbers**, extensions, and any dialed digits (DTMF tones included). - **Postal and email addresses.** - **Account identifiers**: customer numbers, order numbers, card numbers, policy numbers, ticket IDs, anything that keys back to a person or account. - Any other free-text detail that, alone or combined, re-identifies someone. Strongly preferred practice: - Use **synthetic or role-played content**: actors reading a script hit the same turn-taking dynamics (interruption, correction, backchannel) with zero PII risk and a live call's acoustics. - **PHI stays out of scope.** It disqualifies a clip regardless of consent. Redaction guidance: - Replace spoken PII with a tone or silence of the **same duration**, so the turn-taking timing survives -- cutting samples out shifts every onset after the edit and corrupts the labels. - Redact **all channels**. A name muted on the mixed channel but audible on the caller channel is still present. - Keep a private note of what was redacted and when (timestamps). The removed content itself is never committed. --- ## Data-handling policy - **Self-hosted.** `hotato` reads and writes local files, offline; recordings stay on your machine. Contributing is a deliberate, manual act: committing a de-identified, consented clip. - **Storage.** Corpus audio lives in the repository next to its scenario JSON, under the project's MIT terms. Keep working copies on your own machine until cleared for release -- never a third-party cloud bucket, shared drive, or chat tool. - **Only the final clip ships.** The committed clip is redacted and consented; securing or destroying the un-redacted original is your own call. - **Contributor attestation.** Each real-audio PR carries a short statement affirming: (a) consent is on file for every party, (b) PII is removed per the policy above, (c) no PHI, and (d) you have the right to release it under MIT -- part of the merge record. --- ## Validity metrics: what we publish The corpus exists to quantify measurement quality, which shapes how results are reported. We publish: - **Per-signal measurement error, in milliseconds.** For each labeled call, we compare measured event times to human labels and report the error distribution per signal: - `time_to_yield`: signed and absolute error in ms between measured and labeled floor-yield. - `talk_over`: error in ms on measured overlap duration. - onset agreement: error in ms between measured caller onset and `caller_onset_sec`. We report these as distributions (median, spread, worst case), so a reader sees the shape of the error. - **A `did_yield` confusion matrix.** Against the human `category` (`should_yield` / `should_not_yield`) label, we report four cells: correct yields, correct holds, false yields (yielded on a backchannel), and missed yields (talked over a real interruption). The two error cells are what operators feel, so we surface them directly. The distribution and the four-cell matrix are the report: each failure mode scored on its own, so the trade-off between them stays visible. The scorer works on speech energy over time, so validity means timing agreement plus decision agreement. ### Reading a corpus report A healthy result: tight millisecond error distributions on `time_to_yield` and `talk_over`, onset agreement within a small ms band, and a confusion matrix whose off-diagonal (false-yield / missed-yield) cells are small and individually inspectable. A poor result: wide error spread or a heavy off-diagonal -- pointing you toward a concrete fix in the call. Where a call fails, the fix map names candidate remedies, including a learned engagement-control layer at that level. Audio alone is the weaker modality for that fix class; the corpus measures how it holds up on the room it was recorded in. ================================================================================ FILE: docs/COUNTEREXAMPLES.md ================================================================================ # Counterexamples: reduce one scripted failure to a verified repro `hotato counterexample` takes one failing deterministic scripted scenario, deletes inputs that are unnecessary to that exact failure, and writes a local `.hotato-repro` directory. ```text scenario + conversation test + target assertion | v hotato counterexample compile | v private runnable capsule + deletion certificate ``` Every accepted deletion is evaluated through Hotato's own scripted simulator and deterministic assertion engine. A candidate is retained only when the same assertion still contains the source-selected failure branch, and therefore keeps the same failure fingerprint. The v1 boundary is narrow: - input is one local `hotato.scenario` plus one local `hotato.conversation-test`; - the target is one assertion in `assertions.deterministic`; - the scripted simulator supplies the transcript, trace, mock tools, and mock state; - transformations only delete units named by `hotato.reducers.v1`; - model-judged targets, provider sessions, captured recordings, external timing bundles, DTMF targets, and custom policy-pack paths are refused. The scripted proof lane covers 15 deterministic assertion kinds through 48 closed, schema-coupled failure branches: `phrase`, `pii`, `policy`, `tool_call`, `outcome`, `tool_result`, `tool_error`, `state`, `state_change`, `handoff`, `termination`, `latency`, `entity_accuracy`, `sequence`, and `count`. The compiler does not invoke a voice agent, provider, model, TTS service, STT service, command oracle, or network endpoint: its result describes the scripted fixture under the recorded Hotato evaluator source identity, not the behavior of a deployed agent. ## Run the example The repository example: three caller turns, two mock tools, two state records. The selected assertion fails because order `A-1` remains `pending` instead of `posted`. ```bash sh examples/counterexample/run.sh ``` The script compiles a private capsule in a new temporary directory, verifies the proof, reproduces the reduced fixture, checks predicate semantics, and exports a non-runnable share-safe projection, leaving both directories in place and printing their paths. Run the steps directly: ```bash hotato counterexample compile \ --scenario examples/counterexample/refund-not-posted.scenario.json \ --test examples/counterexample/refund-not-posted.test.json \ --target refund-posted \ --workspace examples/counterexample \ --out /tmp/refund-not-posted.hotato-repro hotato counterexample verify /tmp/refund-not-posted.hotato-repro hotato counterexample reproduce /tmp/refund-not-posted.hotato-repro ``` `--out` must name a path that does not exist. Compilation builds a sibling temporary directory and promotes it with a no-replace primitive: Linux, macOS, and Windows have explicit implementations, and an unsupported platform refuses rather than falling back to an overwriting move. ## Source and target selection ### One base scenario The compiler evaluates the scenario document passed to `--scenario` directly, without expanding `variation_matrix` or selecting a matrix row. `capsule.json` records: ```json { "scenario_selection": { "mode": "base-scenario", "variation_matrix_applied": false } } ``` `--seed` selects the scripted replay seed, defaulting to `scenario.seed` or `0` when the scenario has none. A seed does not select a variation-matrix combination. To minimize one generated variation, materialize it as its own complete scenario document, then pass the concrete file to `compile`. Leaving a matrix in the source only preserves its bytes in source provenance; the reducer can delete the matrix when it has no effect on the base-scenario failure. ### One deterministic assertion `--target` must match exactly one assertion ID in `assertions.deterministic`. Compilation confirms twice that the selected assertion fails with identical result, content, and trace identities, and refuses a passing, inconclusive, advisory, missing, duplicated, or unsupported target. ### Exact failure branch identity A deterministic `FAIL` can contain more than one reason: the oracle maps them into a closed set of payload-free failure atoms, sorts and de-duplicates them canonically, and selects the first as the preservation anchor. The private capsule records the selected atom and the complete ordered source atom set; a candidate is `PRESERVED` only when that atom is still present in its own atom set. An atom carries only the discriminator needed to keep failure branches apart: for example, an assertion index, detector, policy rule/type, state field, or entity key. Example: a wrong tool argument gives `tool-argument-value-mismatch` (plus its key); delete the value and the branch becomes `tool-argument-field-missing`; delete the call and it becomes `tool-missing`. Both are `DRIFTED`, though the assertion still reports `FAIL`. The same missing-versus-value care extends to other kinds: | Atom type | Distinguishes | | --- | --- | | results / termination | missing vs. wrong value | | required-order / sequence | present-but-out-of-order vs. absent step | | latency | declared mock measurement vs. simulator default | Payload values, transcript text, and broad diagnostic counts are never part of an atom. The fingerprint binds the test and assertion identity, assertion bytes, kind, dimension, deterministic authority, required `FAIL` status, and selected source atom -- not every diagnostic field in the assertion result, so irrelevant evidence stays reducible without the preserved failure collapsing into a different branch. The emitted `input/conversation-test.json` is a canonical projection containing only the selected deterministic assertion: other source assertions and the rubric lane sit outside the preservation oracle, and their removal is normalization, not evidence those checks passed. ### Workspace boundary `--workspace` is the directory both input files must resolve within, as regular local paths without symlink escape. Omitted, it defaults to the common parent of the scenario and test files; set it explicitly in automation so the filesystem boundary stays visible in review. ## What the compiler deletes `hotato.reducers.v1` is a closed, versioned, deletion-only algebra over the scripted scenario, able to attempt removing: - variation-matrix, facts, and environment entries; - caller behavior and interruption declarations; - caller turns while retaining at least one turn; - mock tool calls, optional tool fields, and nested argument/result entries; - handoff and termination records; - mock-state resources, rows, snapshots, and nested leaves; - optional additive scenario fields. It cannot add content, replace a scalar, reorder a list, rewrite speech, or change the target assertion. `certificate.json` records the digest-bound delete-only transform for every accepted step; `reduction.jsonl` records every candidate attempt. ## The 1-minimal claim A capsule with: ```json {"minimality": {"status": "one_minimal"}} ``` supports this exact statement: > The fixture is 1-minimal under `hotato.reducers.v1` with the recorded > observation-scope freezes. After hierarchical reduction, the compiler repeatedly tries every remaining single-unit deletion, reaching `one_minimal` only when none of those deletions preserves the selected source failure branch. `verify` recomputes the single-unit check rather than trusting `minimality.json`. This is a local claim over the named deletion algebra -- not a global minimum, a root-cause determination, or a semantic-equivalence claim. A remaining deletion can make the target `PASS`, change the selected failure branch, invalidate the scenario, or make required evidence unavailable; each such outcome means that deletion did not preserve the recorded exact failure identity, and is recorded as such in `minimality.json`. ## Budgets `--budget` limits uncached candidate evaluations during reduction: ```bash hotato counterexample compile ... --budget 512 ``` | | | | --- | --- | | default | `512` | | minimum | `1` | | hard maximum | `100000` | Cache hits do not consume the budget, nor do the two source and two final qualification executions (reported separately). When the budget ends before the final deletion pass completes, the compiler still emits a capsule if the reduced fixture preserves the exact failure on both final executions: status `budget_exhausted`, claim limited to failure preservation, no 1-minimal claim. Compilation exits `0` for `one_minimal` and `1` for `budget_exhausted`, writing a complete capsule either way; automation can gate on the process status alone, without parsing a nested field. `verify` can check the integrity and preserved-failure evidence of a `budget_exhausted` capsule, but does not upgrade it to `one_minimal`. ## Capsule contents A private runnable capsule contains: ```text case.hotato-repro/ capsule.json oracle.json certificate.json minimality.json reduction.jsonl source/ scenario.json scenario.original conversation-test.json conversation-test.original input/ scenario.json conversation-test.json expected/assertion-result.json report.md report.html card.svg reproduce.sh predicate.sh MANIFEST.sha256.json ``` `MANIFEST.sha256.json` binds every other file; the capsule ID binds the proof-relevant documents and their digests too. Strict verification rejects missing, changed, extra, symlinked, or unsafe members, and independently re-renders `report.md`, `report.html`, `card.svg`, `reproduce.sh`, and `predicate.sh` before accepting their bytes. The Markdown, HTML, and SVG reports summarize identifiers, proof status, and reduction counts -- bound, reproducible projections of the capsule that do not replace verification. ### Resource and local-filesystem boundary Verification refuses a capsule before replay when it exceeds any of these limits: | Limit | Cap | | --- | --- | | files | 1,024 | | directories | 4,096 | | directory levels | 64 | | one member | 64 MiB | | all members combined | 256 MiB | Each source JSON file is limited to 16 MiB and 96 JSON levels. The scripted scenario also caps caller turns and mock tools at 10,000 each. Counterexample compilation and replay add a narrower proof-lane budget: | Item | Cap | | --- | --- | | selected assertion (canonical JSON) | 256 KiB | | candidate scenario (canonical JSON) | 2 MiB | | rendered transcript text (UTF-8) | 256 KiB | | one deterministic assertion result (canonical JSON) | 2 MiB | | `hits` / `matched_rules` evidence rows | 10,000 | | deletion steps per accepted proof chain | 512 | | deletion operations per step | 10,000 | | remaining units in a completed minimality proof | 512 | | proof-lane regex length | 1,024 UTF-8 bytes | Each proof-lane regex also belongs to a closed, fixed-width replay subset; groups, alternation, backreferences, and variable quantifiers are refused. The assertion and regex byte bounds run before the general assertion validator can invoke Python's regex parser. The selected structured failure branch is still the preservation identity; these limits just bound the work needed to establish it. A candidate that crosses an execution limit is `UNRESOLVED` with `resource_limit_exceeded`, never treated as preserved or absent. These counterexample compiler/replay limits do not narrow the regex or result surface of Hotato's deterministic evaluator outside the proof lane. The capsule directory must stay unchanged for the duration of `verify`, `reproduce`, `inspect`, or `export`: persistent mutation, a root replacement, and symlink or special-file substitution are all detected. A privileged local process able to swap bytes between reads and restore them before the final manifest pass sits outside this version's proof snapshot -- copy an untrusted capsule into a private, non-writable workspace before verification. ### What the journal proves `certificate.json` and the `PRESERVED` rows in `reduction.jsonl` carry the accepted delete-only chain; strict verification reconstructs those transforms and reruns the final single-unit deletion inventory for a `one_minimal` claim. `ABSENT`, `DRIFTED`, and `UNRESOLVED` journal rows are diagnostic reduction history only; their candidate payloads sit outside the accepted proof chain, unreplayed by `verify`. ## Verify, reproduce, and inspect These commands answer different questions. | Command | Question | Evaluator rule | Work performed | |---|---|---|---| | `verify` | Is this the same intact proof produced by the recorded evaluator source? | Recorded package version and evaluator source digest must match the installed implementation. | Replays source and final cases twice, reconstructs every accepted deletion, checks the selected source failure branch and expected result, re-renders derived artifacts, and recomputes claimed single-unit minimality. | | `reproduce` | Does the reduced fixture still produce the selected source failure branch under the installed evaluator? | Evaluator drift is allowed and reported. | Checks capsule integrity and the delete-only chain, then runs the reduced fixture twice. It does not reassert historical intermediate verdicts or renew the original minimality proof under changed code. | | `inspect` | What does the intact capsule claim? | No evaluator execution. | Checks the closed member inventory, source/oracle/artifact bindings, canonical human files, manifest, and capsule schema, then prints target, reduction, minimality, preservation, and profile metadata. | Use `verify` for proof audit and artifact integrity against the recorded Hotato package version and evaluator source closure. The evaluator digest does not bind the Python interpreter build, OS, CPU, or native libraries; strict replay still compares the recorded result, content, and trace hashes, so a runtime difference that changes evaluator behavior fails verification. Use `reproduce` to evaluate the frozen reduced fixture after Hotato evaluator or simulator code changes. `reproduce` returning `0` means the failure reproduced -- the inverse of a conventional quality gate. Use `predicate` or the generated `predicate.sh` when the desired shell result is “failure present means bad.” ## Private and share-safe forms Compilation writes `private-runnable-v1`: complete source and reduced scenario/test inputs, including scripted speech and mock tool/state payloads. Treat the directory as sensitive even when its origin is simulated. The compiler applies restrictive POSIX modes where supported, but permissions do not establish publication rights or de-identification. Create a payload-free projection only after the private capsule verifies: ```bash hotato counterexample export /tmp/refund-not-posted.hotato-repro \ --profile share-safe-v1 \ --out /tmp/refund-not-posted.share-safe ``` The exported directory has one closed inventory: `capsule.json`, `report.md`, `report.html`, `card.svg`, `README.md`, and `MANIFEST.sha256.json`. Extra, missing, renamed, symlinked, or special-file members are refused even when a manifest declares them. `inspect` re-renders every human-facing file and requires byte-for-byte equality with its canonical rendering, so rebinding modified report text into a new manifest does not make the projection valid. Those canonical renderer bytes are part of the capsule v1 exchange contract. A compatible implementation must retain the v1 renderer; changing its output requires a versioned renderer/profile or a capsule format version bump. The projection omits runnable inputs, scenario and assertion bodies, transcript or audio content, tool payloads, state values, credentials, provider identifiers, absolute paths, source-derived reducer paths, and per-candidate digests: private single-unit rows become an aggregate count by outcome instead. The share target and canonical reports expose the selected failure code (such as `tool-argument-value-mismatch` or `state-field-value-mismatch`), so the failure category stays readable without a payload. Atom discriminators (a field, key, rule, detector, or index) stay private. `failure_atom_digest` binds the complete selected atom without publishing those values. `share-safe-v1` is an engineering access boundary, not an anonymity claim: capsule, source, assertion, evaluator, fingerprint, and failure-atom digests remain correlators, and low-entropy or externally known source material can be tested against them. Keep a projection under the same review, artifact-retention, and disclosure controls as engineering metadata. The projection cannot reproduce the failure. Keep the private capsule when a reviewer, CI job, or coding agent must execute the fixture. ## Exit codes ### `compile` | Exit | Meaning | |---:|---| | `0` | An atomic private capsule was written and earned `one_minimal`. | | `1` | An atomic private capsule was written with the exact failure preserved, but the candidate budget ended before one-minimality was proved. | | `2` | Refused: input, target, workspace, output, replay, or proof safety requirements were not met. No completed destination is left. | ### `verify` | Exit | Meaning | |---:|---| | `0` | Integrity, provenance, source/final replay, accepted deletion chain, exact failure identity, and the stated minimality level verified. | | `1` | The capsule is intact, but the target failure or claimed single-unit minimality no longer reproduces. | | `2` | No proof verdict: malformed, tampered, unsafe, evaluator-incompatible, unsupported, or inconclusive capsule. | ### `reproduce` | Exit | Meaning | |---:|---| | `0` | The selected source failure branch reproduced twice under the installed evaluator. | | `1` | The capsule is intact, but that exact failure is absent. | | `2` | No reproduction verdict: malformed, tampered, unsafe, unsupported, disagreeing, or inconclusive capsule. | `inspect` and `export` return `0` on success and `2` on refusal. ### `predicate` `hotato counterexample predicate DIR` and `DIR/predicate.sh` map reproduction to `git bisect run` semantics: | Exit | Meaning | |---:|---| | `1` | Bad: the exact target failure is present. | | `0` | Good: the exact target failure is absent. | | `125` | Skip: the fixture is unusable, unsafe, unsupported, disagreeing, or inconclusive at this revision. | ## CI and evaluator bisect boundaries ### Recorded-evaluator proof audit Use `verify` when CI installs the Hotato evaluator recorded in the capsule: ```bash hotato counterexample verify tests/repros/refund-not-posted.hotato-repro ``` This detects capsule mutation and evaluator-source drift: a changed evaluator digest is a refusal (`2`), since a different implementation cannot silently validate the historical proof (what the digest binds is above). ### Current-evaluator regression Use the generated predicate when a reproduced failure should fail the job: ```bash tests/repros/refund-not-posted.hotato-repro/predicate.sh ``` On Windows, run the helpers through the interpreter -- `sh predicate.sh`, `sh reproduce.sh`, `git bisect run sh /absolute/path/to/predicate.sh` -- the capsule digests bind the same helper bytes on every platform. The predicate executes the frozen reduced mock scenario; it does not call an application's current voice agent, prompt, tools, provider, or telephony path. V1 gates Hotato evaluator/simulator behavior over the capsule, not an end-to-end agent release gate. ### `git bisect run` ```bash git bisect start BAD_REVISION GOOD_REVISION git bisect run /absolute/path/to/refund-not-posted.hotato-repro/predicate.sh ``` This locates the revision where Hotato evaluator or scripted-simulator behavior begins reproducing the recorded failure. The capsule retains its own source and reduced inputs, so external scenario/test file changes play no part in the predicate. A revision that cannot load or evaluate the capsule is skipped with `125` -- as with `git bisect run` generally, skipped revisions near the transition can leave a candidate range rather than one first bad revision. Bisecting agent implementations, hosted models, provider configurations, or prompt revisions needs a separately defined adapter, outside counterexample v1, that runs those revisions and emits the evidence an assertion consumes. ## Refusals worth designing around The compiler fails closed when: - the target is in the rubric lane or does not fail twice identically; - required evidence is missing and the target is inconclusive; - a `timing_contract` depends on an external bundle; - a DTMF assertion requests trace evidence the scripted simulator cannot emit; - an `http_result` assertion requests captured exchange evidence the scripted simulator cannot emit; - an outcome predicate reads an external timing field; - a latency assertion reads an external timing field or a span type other than a scripted `tool_call`; - a keyed tool argument/result target uses an empty key, or an entity reference uses an empty key or a null expected value; - a policy assertion uses an external `pack_path`; - an input escapes the workspace or traverses a symlink; - the output path already exists; - source or final replays disagree; - a capsule member, manifest, certificate, or evaluator provenance is invalid. Refusal is evidence the compiler cannot make its promised statement for that input; it never becomes a passing or failing agent verdict. ## Machine output Add `--format json` to `compile`, `verify`, `reproduce`, `inspect`, or `export`. The JSON carries the command kind, exit code, capsule/failure identities, and the command-specific proof or reduction fields. `predicate` communicates only through its process exit. See the complete runnable fixture in [`examples/counterexample/`](../examples/counterexample/). ================================================================================ FILE: docs/DIARIZE.md ================================================================================ # `hotato run --mono call.wav --diarize`: score a single-channel recording Hotato's gold reference is **two-channel** audio: caller on one channel, agent on the other, no separation needed. That path, and every published number, stays unchanged. A **mono** (single, mixed) recording fails by default: both voices sum into one waveform, so the scorer cannot attribute energy to a speaker and rejects it as not scorable. The opt-in `[diarize]` front-end widens that coverage: it runs an off-the-shelf **speaker diarizer** over the mono to recover *who was active when*, reconstructs caller/agent tracks, and hands them to the **existing** scorer -- so a mono call becomes scorable. The result is **quality-gated** and tiered per file: above the confidence bar it is a `diarized-mono` verdict; below it, indicative only with no SLA gate; non-separable, it is refused. **Turn-timing reconstruction, from anonymous labels.** Hotato scores *timing* -- who was active when, and overlap -- and a diarizer's turn timestamps reconstruct that directly: anonymous `SPEAKER_00` / `SPEAKER_01` labels per turn, handed to the scorer as boundaries. The audio stays mixed; the labels carry only channel timing, never a name, a voice-print match, or an identity. The companion opt-in extra is [`docs/TRANSCRIBE.md`](TRANSCRIBE.md): it attaches plain-text, timestamped words next to the score, computed strictly after scoring and never fed back into it. ## Quickstart ```bash # Install the default (local, offline) diarizer extra, plus a Hugging Face token # with the gated model conditions accepted (one-time download, then offline): pip install 'hotato[diarize]' export HUGGINGFACE_TOKEN=hf_... # accept the model card conditions first # Check first whether the mono is confidently separable (no scoring): hotato trust --stereo call.wav --diarize # -> high / low / refuse tier # Score the mono: hotato run --mono call.wav --diarize --format json # diarized-mono verdict ``` Without the extra (or the token/model), `--diarize` exits `2` with a clean error -- it always requires the diarizer to run before scoring. ## Backends and extras A pluggable backend seam, mirroring the neural-VAD seam: pick with `--diarizer`, install only the extra you select. `pyannote` is the accessible local default, chosen from the downstream benchmark; for best-in-class telephone-audio accuracy, pick `sortformer` (local) or `pyannoteai` (hosted). - **`pyannote`** (default) - Extra: `[diarize]` - Where it runs: local, CPU-viable, offline - Notes: richest confidence signals (posterior + embedding margin); gated HF weights - **`sortformer`** - Extra: `[diarize-sortformer]` - Where it runs: local, GPU-leaning - Notes: best self-hostable on 2-speaker telephone; EEND (no embedding margin) - **`pyannoteai`** - Extra: `[diarize-hosted]` - Where it runs: HOSTED (audio leaves the machine) - Notes: best absolute accuracy; requires `--egress-opt-in` ```toml # default; needs system ffmpeg diarize = ["pyannote.audio>=4.0", "torch>=2.8", "torchaudio>=2.8", "numpy>=1.21"] # best self-hostable, GPU diarize-sortformer = ["nemo-toolkit[asr]>=2.7", "torch>=2.8", "numpy>=1.21"] # hosted, egress opt-in diarize-hosted = ["pyannoteai-sdk>=0.3"] ``` The `[diarize]` path raises the effective Python floor to **>=3.10** (pyannote 4.x); the stdlib core stays >=3.9 -- this only constrains the optional path. ### Model licenses (log per FTO note) Off-the-shelf diarizers are an integration, orthogonal to any Hotato IP claim; licenses are logged here and carried in the score envelope's `diarization.licenses` block: - `pyannote-audio`: **MIT** (code) - `speaker-diarization-community-1` weights: **CC-BY-4.0** (attribution required) - `segmentation-3.0` weights: **MIT**; `wespeaker` embedding: **CC-BY-4.0** - `torch`: **BSD-3-Clause**; `torchaudio`: **BSD-2-Clause** - `nemo-toolkit`: **Apache-2.0** - Sortformer **streaming v2**: **CC-BY-4.0** (the offline v1 is **CC-BY-NC**, non-commercial, kept out of the shipped extras) - pyannoteAI hosted: proprietary terms (verify before enabling egress) For a permissive weights license, prefer `speaker-diarization-3.1` (MIT weights) over community-1 (CC-BY-4.0) via `HOTATO_DIARIZE_MODEL`. ## The confidence gate (the core safeguard) Aggregate diarization error is a corpus statistic; the gate is **per file, at runtime, with no ground truth**. Six signals feed a `separation_confidence` in `[0, 1]` and one of three tiers: | signal | flags low quality when | |---|---| | speaker count == 2 | != 2: not two clean parties (1 = couldn't separate; 3+ = extra voices / mis-cluster) -> refuse | | both speakers >= 0.30s activity | a near-silent "speaker" is a spurious split of one party -> refuse | | mean segmentation posterior | near-chance: the model is unsure who is speaking (uncalibrated; a relative signal) | | embedding cluster margin (pyannote only) | small: two voices too similar to attribute confidently | | overlap ratio in a sane band | extreme: heavy crosstalk / collapsed turns -> talk-over unreliable | | segment churn (short turns/sec) | high: a jittery, unstable timeline -> noisy timing | **Tiers:** - **high** -- scores normally, tagged `source: "diarized-mono"` and `confidence_tier: "high"`, distinct at every step from a dual-channel verdict. - **low** -- scores with `indicative_only: true`, reading "indicative only, reconstructed from single-channel diarization." The pass/fail SLA gate (`--max-talk-over` / `--max-time-to-yield`) stays off at this tier. - **refuse** -- `scorable: false`, a reason naming the failed signal, exit `2` (same as today's mono rejection). The thresholds are provisional and **uncalibrated**, pinned by the downstream verdict-agreement benchmark rather than asserted as accuracy, and exposed as constants in `hotato/diarize.py`. ## Caller vs agent assignment (proposed, stated, overridable) A diarizer returns anonymous `SPEAKER_00` / `SPEAKER_01`; Hotato needs caller/agent. The mapping is proposed, stated as an assumption, and overridable: - **default proposal** -- the floor-dominance heuristic (`trust`'s possible-swap band): an agent usually holds the floor longer, so the higher-talk-time speaker is proposed as agent. A balanced (ambiguous) floor time is broken by who-speaks-first and flagged `balanced: true`, downgrading the verdict to indicative until confirmed. - **override** -- `--caller-speaker SPEAKER_00 --agent-speaker SPEAKER_01`; when both are given, no heuristic runs. The chosen mapping and its basis are emitted in `diarization.speaker_map: {caller, agent, basis, balanced, confidence}`. ## Echo / crosstalk is N/A on this path The two reconstructed tracks are slices of **one physical microphone**, so `signals.echo` / crosstalk coherence carries no real signal (it reads trivially high on overlap) and is marked `applicable: false` on this path -- `--echo-gate` only means something across two physically separate channels. ## Limits (what stays indicative or refused) - **very similar voices** (same gender/pitch, or one person on both ends) → low embedding margin → low/refuse. - **heavy crosstalk / echo bleed** → overlap balloons → low/refuse. - **>2 speakers** (a supervisor, hold music with vocals) → refuse. - **deep sustained overlap** and **sub-second boundary precision** run weaker than dual-channel and are stamped as such, never hidden. - **balanced speaker map** → mapping uncertain → indicative until confirmed. A de-risk spike, with a *perfect* diarizer, confirmed the masked-reconstruction path inflates sub-second talk-over by ~0.1-0.36s and can bridge a short backchannel gap -- an error intrinsic to single-channel masking that no diarizer quality removes. That is why elevated-overlap and short-yield cases land in the fragile `low` zone. The recommended follow-up is direct diarization-timeline injection, skipping the reconstruction re-VAD; either way, the gate above governs the error budget. ## JSON shape (agents) A scored diarized-mono envelope is the SAME envelope as any single run, plus a `diarization` provenance block and, when the tier is not high, `indicative_only: true` on the event: ```json { "mode": "single", "diarization": { "source": "diarized-mono", "backend": "pyannote", "model": "pyannote/speaker-diarization-community-1", "num_speakers": 2, "speaker_map": { "caller": "SPEAKER_00", "agent": "SPEAKER_01", "basis": "floor-dominance", "balanced": false }, "separation_confidence": 0.86, "confidence_tier": "high", "overlap_ratio": 0.14, "licenses": {"pyannote-audio": "MIT (code)", "...": "..."} }, "events": [{ "verdict": { "did_yield": true, "seconds_to_yield": 0.5, "talk_over_sec": 0.5, "reasons": [] }, "diarization": { "...": "same block" }, "scorability": { "separation": { "confidence_tier": "high", "separation_confidence": 0.86, "signals": {"...": "..."} } }, "signals": { "echo": {"applicable": false, "reason": "single physical channel ..."} } }] } ``` Branch on `diarization.confidence_tier`: an event carrying `indicative_only: true` stays indicative-only, distinct from a confident dual-channel verdict. A refused file is `scorable: false` with `not_scorable_reason` and exit `2`. ================================================================================ FILE: docs/DRIVE-A-CALL.md ================================================================================ # Drive-a-call: originate a call against a live agent, then score it `hotato capture` scores a call you already ran. Drive-a-call places the call: it dials a live voice agent, waits for it to finish, and feeds the recording into the same pull -> score pipeline. The agent's side is live and unscripted, so it scores unchanged. ## The second move, after a first catch on captured calls You have seen a catch on a recorded call -- a pulled recording ([`docs/CONNECT.md`](CONNECT.md)), a sweep candidate, or a trace assertion. Drive-a-call is the next move: instead of waiting for production traffic to reproduce the moment, place a call against your live agent on demand and score the recording the same way. It reaches the provider's API and bills one outbound phone call per run, so the captured-call paths stay the cheap default and this is the deliberate follow-up. ## The caller side runs from a script The agent's half is unscripted. The CALLER's half runs from a script, and the origin records that plainly as `origin.caller`: - **Twilio** renders your `scenario.v1` caller script into TwiML: one `` per `say`-turn, `` between them. `` speaks at **fixed offsets** on its own clock, not the agent's -- a deterministic driver built to catch a regression in a scripted sequence, not a live back-and-forth. A turn's reactive `when_agent_asks` / `after` label still speaks unconditionally, in order. `origin.caller = "scripted-twiml"`. - **Vapi** originates the call FROM the assistant under test (a staging clone) TO a customer number: the assistant is the party being measured, whoever answers is the other side. `origin.caller = "assistant-originated"`. Either way `origin.kind = "real"`, tagged with `origin.provider` and `origin.provider_call_id` (the provider's own call id) -- the invariant-5 axis: every driven call states exactly how it was produced. ## The endpoints implemented - **Twilio** - Originate: `POST /2010-04-01/Accounts/{sid}/Calls.json` with `To`/`From`/`Twiml` + `Record=true`, `RecordingChannels=dual` (the REST equivalent of ``). - Poll until: `GET .../Calls/{CallSid}.json` -> `status == completed`. - Then pull: `GET .../Recordings.json?CallSid=...` -> `RecordingSid` -> existing `capture_twilio` (`?RequestedChannels=2`). - **Vapi** - Originate: `POST https://api.vapi.ai/call` `{assistantId, phoneNumberId, customer:{number}}`. - Poll until: `GET /call/{id}` -> `status == ended`. - Then pull: existing `capture_vapi` -> `artifact.recording.stereoUrl`. Only `POST` (create) and `GET` (poll + pull) are issued: a provider config stays untouched. Vapi drives the call FROM a staging CLONE, so production stays untouched too. A non-`completed` Twilio call (busy / failed / no-answer / canceled) has no recording to score, so it raises a clear error. ## Credentials and the egress opt-in (both required) Placing a call reaches the provider's REST API and bills a phone call, so `run_scenario` requires BOTH before it dials: 1. **Credentials** -- `VAPI_API_KEY`, or `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` (via `hotato connect` or the environment). 2. **An explicit egress opt-in** -- set `HOTATO_DRIVE_OPT_IN=1`, or pass `egress_opt_in: true` on the scenario. Without it, `run_scenario` raises the same clean, structured refusal a hosted op has always given, and dials nothing. Drive parameters ride on the scenario (inline or under a `drive:` block) or the environment: - Twilio: `to_number` (the agent's number, `HOTATO_DRIVE_TO_NUMBER`), `from_number` (your Twilio number, `HOTATO_DRIVE_FROM_NUMBER`). - Vapi: `phone_number_id` (`VAPI_PHONE_NUMBER_ID`), `customer_number` (`HOTATO_DRIVE_CUSTOMER_NUMBER`). The recording download reuses `capture`'s validated, SSRF-safe path. See [`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md) for the per-command rows. ## What it costs, and what it proves It bills one outbound phone call per scenario run -- even the deterministic Twilio caller is billed. The recording captures a live conversation; whether the agent "passed," or the scripted caller's timing matched a live one, is the separate assert layer's job. ## Retell / LiveKit / Pipecat Retell calls are captured after the fact with `hotato pull`. LiveKit and Pipecat calls are captured inside your own infra: point your own capture path at the recording, and hotato scores whatever file it writes. ================================================================================ FILE: docs/EGRESS.md ================================================================================ # Egress: what talks to the network, command by command Every `urllib`/`http`/`subprocess`-to-a-network-tool call site in `src/hotato/`, mapped to the CLI command that reaches it -- derived straight from the code. Every command under "Fully local" runs entirely on your machine: no `urllib` import, no socket, no networked subprocess. ## Fully local -- everything runs on your machine `run`, `report`, `doctor`, `team`, `export`, `benchmark`, `compare` (the batch result-comparison command, `stackbench.py`), `demo`, `start`, `card`, `diagnose`, `plan`, `explain`, `fixture create`, `fixture promote`, `contract create` (stereo / caller+agent / local-file mono), `contract verify`, `contract inspect`, `contract pack`, `contract unpack`, `trace attach`, `trace export`, `scan`, `trust`, `analyze`, `patch`, `verify`, `fix trial`, `loop`, `describe`, `init starter`, `init webhook` (the scaffold generator -- the server it scaffolds is a local listener, see below), `setup`, and every `counterexample` subcommand (`compile`, `verify`, `reproduce`, `inspect`, `export`, `predicate`). Counterexample compilation uses the local scripted simulator and deterministic assertion engine; it never executes a provider adapter, judge, download, or arbitrary command. These read and write only local files (recordings, scenarios, `hotato.yaml`, connection files). `connect` also belongs here: it validates and stores credentials at `~/.hotato/connections.json` (mode `0600`), no network round trip (`src/hotato/connections.py`: "nothing in this module makes a network call"). `rubric run`, `rubric calibrate`, and the `test run` rubric lane also belong here **on the default path**: the judge is a **LOCAL Ollama** daemon (default `http://localhost:11434`), and the verdict cache (`~/.hotato/rubric-cache`) is local too. Two paths send a rubric command to the network instead -- both explicit, both in the extras table below: a non-local Ollama endpoint, or `--judge-provider hosted`. ## Reaches your configured vendor -- only when the command's job is to | Command | Reaches | When | Code | |---|---|---|---| | `capture --stack vapi\|retell\|twilio` | the stack's REST API | always (fetches the one call you named) | `capture.py`: `_http_get`/`_http_get_json`/`_download` via `urllib.request` | | `capture --stack vapi\|retell\|twilio --demo` | nothing | `--demo` scores a bundled reference file instead | `capture.py` | | `capture --stack livekit\|pipecat` | nothing from Hotato | the recording is produced by YOUR infra; Hotato only scores the local file `setup` pointed you at | `capture.py` | | `pull` | the stack's list + download endpoints | always (bulk-fetches recent recordings) | `capture.py` (same `_http_get`/`_download` path `capture` uses, looped) | | `sweep --stack ` | same as `pull` | whenever `--demo` is NOT passed | `capture.py` via the pull path | | `sweep --demo` | nothing | sweeps the two bundled demo recordings | `analyze.py` on packaged audio | | `inspect --stack vapi\|retell` | the stack's assistant-config read endpoint (GET only, never a write) | always | `inspectcfg.py`: `_http_get_json`, `inspect_vapi`, `inspect_retell` | | `inspect --stack livekit\|pipecat` | nothing | parses YOUR local config file with `ast`, no network | `inspectcfg.py`: `inspect_livekit_file`, `inspect_pipecat_file` | | `ingest` (webhook worker) | inbound only, from the stack you configured to call your webhook; outbound only if the event's `recording_url` needs a download | per event, and only for the fetch step | `ingest.py`: `_resolve_recording` reuses `capture.py`'s fetch/download; payloads are always treated as data, never instructions | | `apply` (default, no `--yes`) | nothing | dry run -- prints the staging clone it WOULD create | `apply.py`: `build_apply` returns `dry_run: True`, never calls the networked function | | `apply --clone --yes` | the stack's REST API (`vapi`, `retell` only) | only with `--yes` and credentials | `apply.py`: `create_clone` / `_http_json` is "the only networked function" in the module (its own docstring) -- reads the source config via GET, then POSTs to create a NEW staging assistant. Never PUTs/PATCHes the source. | | `issue create` (default, no `--yes`) | nothing | dry run -- prints the issue body and the `gh` command it would run | `issuecmd.py` | | `issue create --yes` | GitHub, through your local `gh` CLI's existing auth | only with `--yes` | `issuecmd.py`: `subprocess.run(["gh", "issue", "create", ...])` | | `pr create` (default, no `--yes`) | nothing | dry run -- prints the PR body and the `git`/`gh` argv it would run | `prcmd.py` | | `pr create --yes` | Git remote + GitHub, through your local `gh` CLI's existing auth | only with `--yes` | `prcmd.py`: `subprocess.run([...git...])` then `subprocess.run(["gh", "pr", "create", ...])` | | `test run --state F` (state-config `adapter: http`) | your configured system-of-record REST API | per `state`/`state_change` assertion, and ONLY when the config sets `egress_opt_in: true`. What leaves: the mapped filter VALUES for that one query (in the URL or a JSON body); never audio, transcript, or the config | `state_adapter.py`: `HttpStateAdapter.query` via `urllib.request` (credential-safe redirects reused from `capture.py`) | | `test run --state F` (state-config `adapter: sql` + a `dsn`) | your database, over the network via the driver you install | per assertion, and ONLY with `egress_opt_in: true`. A parameterized read-only SELECT; filter VALUES are bound as data, never interpolated. A local `sqlite_path` opens no socket | `state_adapter.py`: `SqlStateAdapter.query` (stdlib `sqlite3`, or a caller-installed DBAPI driver) | | `simulate --chat URL` | YOUR chat agent at the URL you name -- `localhost`/`127.0.0.1` only by default | one POST per scripted caller turn (`{conversation_id, turn_index, text}` -> `{text}`); a non-local host is refused (exit 2) BEFORE any request unless you pass `--egress-opt-in`, and redirects are refused outright, so a local URL never bounces off-box. What leaves: the scenario's caller turn text, nothing else | `chat_sim.py`: `check_chat_url` / `_post_turn` via `urllib.request` | `load_state_adapter` requires `egress_opt_in: true` before an `http` adapter or a `sql` adapter over a `dsn` connects -- checked before any connection, on the CLI's exit-2 usage-error path. The default `--state` adapters (the local mock JSON/SQLite sandbox and a local `sqlite_path` SQL DB) run entirely on your machine and need no opt-in. Full config format: [`docs/STATE-ADAPTERS.md`](STATE-ADAPTERS.md). Notify surfaces (Slack, GitHub) fire only through credentials you configured (`gh`, a Slack token), and only for actions you invoked. ## Optional extras that add a hosted call | Extra / flag | Adds | Gate | |---|---|---| | `hotato[neural]` | nothing network -- a local Silero VAD cross-check model, run offline | N/A | | `hotato[transcribe]` (`run --transcribe`) | nothing at inference -- a local `faster-whisper` ASR pass over the same recording, fully offline once the model is cached. First use of an uncached model name downloads its weights once (like installing any pip package with model weights); every run after opens no socket. Context only, kept separate from the score | N/A | | `hotato[livekit]` / `hotato[pipecat]` | nothing from Hotato directly -- these SDKs run YOUR live capture infra; Hotato scores the file that infra writes | N/A | | `--diarizer pyannoteai` (`contract create --mono --diarize`, `run --mono --diarize`) | uploads the mono audio to `pyannote.ai` for diarization | requires `--egress-opt-in` (exit 2 without it); the default diarizer (`pyannote`, local) stays offline. See `diarize.py`: `build_pyannoteai_backend` | | `hotato[judge]` (`rubric run`, `rubric calibrate`, `test run` rubric lane) | nothing on the default path -- the judge is a LOCAL Ollama model (`http://localhost:11434`), reached with only the stdlib (`urllib`). The transcript stays on your machine | N/A on the default (local) path. `rubric.py`: `OllamaJudge` | | `--judge-provider hosted --judge-endpoint URL` (any rubric command) | sends the transcript + rubric criterion to a hosted OpenAI/Anthropic-compatible `/chat/completions` endpoint you name | requires `--judge-egress-opt-in` (exit 2 without it); the default local judge stays on your machine. `rubric.py`: `HostedJudge` (raises `EgressRefused` without the flag) | | A non-local `--judge-endpoint` (e.g. a remote Ollama host) | reaches that host | same gate -- requires `--judge-egress-opt-in`. `localhost`/`127.0.0.1`/`::1` skip it. `rubric.py`: `OllamaJudge._is_local_endpoint` | | `--notify URL` (`sweep`, `fleet run`) | POSTs one JSON summary -- counts, top candidate moments (id, kind, timing numbers only), local artifact paths, plus a `text` line for Slack incoming webhooks. Audio, credentials, and transcript text stay out of it | off by default; fires only with an explicit, repeatable `--notify URL`. A non-http(s) scheme is refused (exit 2) before any network attempt. Once sent, delivery is fail-open -- a down or slow webhook logs one stderr warning and the run keeps going. See `notify.py`: `post_notification` | | `--notify URL` (`contract verify`) | POSTs one JSON run-summary when the gate finishes -- kind `contract-verify`, the pass/fail counts (`passed`/`failed`/`tampered`/`refused`/`assertions_failed`, reported as separate fields, never blended), the top FAILING contracts' ids + measured timing (`did_yield`/`seconds_to_yield`/`talk_over_sec`), plus a `text` line for Slack. Audio, credentials, transcript text, and file/bundle paths all stay out of it (this payload carries no artifact paths, unlike sweep's) | off by default; fires on every verify (pass or fail) only with an explicit, repeatable `--notify URL`. A non-http(s) scheme is refused (exit 2) before the re-score. Delivery is fail-open -- a down webhook never raises and never changes the verify exit code. See `notify.py`: `contract_verify_payload` / `post_notification` | ## The one credential-safety detail worth knowing `capture.py` installs a process-wide `urllib` redirect handler (`_CredentialSafeRedirectHandler`) so a 3xx redirect from a vendor API keeps your Authorization header and API key scoped to the original host. Applies to every fetch path above that goes through `capture.py`'s `_http_get`/`_download`. ## Read more - Posture summary and reporting a vulnerability: [`SECURITY.md`](../SECURITY.md) - Command-by-command threat model: [`docs/THREAT-MODEL.md`](THREAT-MODEL.md) - Ingest's untrusted-payload handling: [`docs/INGEST.md`](INGEST.md) ## Drive-a-call: originates a real call, so it is double-gated `run_scenario` (the fleet experiment step; `src/hotato/drive.py`) ORIGINATES a real, billable call against a live agent and pulls its recording -- the one path here that both reaches a vendor AND places an outbound call, so it carries a gate on top of the usual credential requirement. - **`run_scenario` (Vapi adapter)** - Reaches: `POST https://api.vapi.ai/call`, then polls `GET /call/{id}`, then the existing `capture_vapi` download. - When: only when driven. - Gate: requires real credentials AND an explicit egress opt-in (`HOTATO_DRIVE_OPT_IN=1` or `egress_opt_in: true` on the scenario), both present before dialing. Creates and reads only (GET/POST); the call is driven from the staging CLONE, keeping production untouched. - **`run_scenario` (Twilio adapter)** - Reaches: `POST .../Calls.json`, then polls `GET .../Calls/{sid}.json` + `GET .../Recordings.json`, then the existing `capture_twilio` download. - When: only when driven. - Gate: same double gate. TwiML `` is a fixed-timeline scripted caller; recording is dual-channel via `Record=true`/`RecordingChannels=dual`. GET/POST only, so config stays untouched. The recording download reuses `capture.py`'s validated fetch (scheme allowlist, default-deny SSRF, cross-host credential strip, atomic write); a local test recording server on `127.0.0.1` needs `HOTATO_ALLOW_PRIVATE_URLS=1` like every other download. Full walkthrough: [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md). ================================================================================ FILE: docs/EVIDENCE-PACK.md ================================================================================ # The evidence pack Hotato's proof is reproducible: commands, recorded audio, and deterministic, byte-stable verdicts you rerun on your own machine. The standard every artifact here is held to, and why it's ranked the way it is, lives in [evidence/README.md](evidence/README.md) -- read that first when deciding how much to trust any single piece below. ## What's in the pack - **Bundled demo** -- two bundled calls an agent fails (a synthesized talk-over and a recorded provider-default call), scored offline in under a minute, one command. `hotato demo` · [START.md](START.md) - **Recorded provider-default battery** -- 12 scripted calls against a live voice agent on default settings; a missed interruption and a false stop fail in the same run. [`corpus/vapi-defaults/README.md`](../corpus/vapi-defaults/README.md) - **Determinism check** -- the same recording produces the same numbers every run. CI runs it on Linux, macOS, and Windows; Linux is green today. [VALIDATION.md](VALIDATION.md) Job 1 - **Not-scorable gallery** -- eight input conditions, and the exact verdict each produces, including three hard refusals. [GALLERY.md](GALLERY.md) · [TRUST-GALLERY.md](TRUST-GALLERY.md) - **Trust contract** -- the input-condition table the gallery demonstrates. [TRUST-MATRIX.md](TRUST-MATRIX.md) - **Validation scope** -- the three jobs Hotato is measured on, and where that measurement stops. [VALIDATION.md](VALIDATION.md) - **Where Hotato fits next to a QA platform** -- a named-vendor routing guide. [COMPARE.md](COMPARE.md) - **Case studies** -- recorded-audio write-ups, each with a repro command and a scope section on exactly what that run proved. [`case-studies/`](case-studies/README.md) - **PR cards** -- self-contained SVGs of a candidate, a threshold-funnel finding, or a verify result. [CARDS.md](CARDS.md) · [`assets/cards/`](assets/cards/) - **Launch-bar status** -- the checklist scoring where each launch item stands. [evidence/validation-plan.md](evidence/validation-plan.md) ## Check it yourself Every artifact above is re-derivable from a command: ```bash hotato demo --no-open --format text # the two bundled failures hotato run --stereo FILE.wav --expect yield # diff two runs, get nothing hotato trust --stereo FILE.wav # the refusal contract, live ``` Every command runs offline, on your machine -- reproduce it yourself instead of taking the claim on faith. ## What counts as evidence here A command, a recording, a verdict you check yourself -- ranked by the standard in [evidence/README.md](evidence/README.md). ================================================================================ FILE: docs/EXPLAIN.md ================================================================================ # explain: root-cause-by-layer, composed from what already exists `hotato explain` reads a finished result and produces a root-cause attribution: which layer likely failed, whether a fix is safe to try, the opposite-risk tradeoff, and a plain next command. It's compositional, not a new scoring engine: it reframes `hotato diagnose`'s per-event findings and `hotato plan`'s policy gate (a mapped knob, one bounded step, a passing opposite-risk fixture, config-only-safe), plus, for a contract bundle, its own trust report, policy bounds, and an attached voice trace when present. When the evidence can't support picking ONE root cause, explain states the reason plainly. ```bash hotato run --suite barge-in --format json > result.json hotato explain result.json ``` ## Three input shapes, auto-detected | Input | Example | Behavior | | --- | --- | --- | | Run envelope | `hotato explain result.json` | Full diagnose + policy-gate attribution, per failing event. | | Sweep/analyze candidate ref | `hotato explain hotato-sweep.json#1` | ALWAYS refused: a candidate carries no human label. | | Contract bundle directory | `hotato explain contracts/refund-cutoff-001.hotato` | Attributed from the contract's own measurement and policy bounds, or refused when it needs a signal the bundle lacks. | ## The attribution shape Layer-general by design: `turn_taking` is the only populated layer today, with room for the next (asr, tool, policy, latency, handoff, ...) without a version bump. ``` {"event_id": the failing event, or null for a battery-level finding, "failure_layer": "turn_taking", "type": missed_real_interruption | false_stop_on_backchannel | false_stop_on_ambient_noise | slow_yield | excess_talk_over | endpointing_miss | threshold_funnel, "turn_taking_layer": interruption_detection | endpointing, "confidence": high | medium | low, "fixability": safe_to_patch | needs_human | insufficient_evidence | do_not_patch, "opposite_risk": the tradeoff a fix on this axis trades against, "evidence_for": measured fields and notes that support this attribution, "evidence_against": the caveats against treating it as settled, "unknowns": explicit gaps in the evidence (never silently assumed), "safe_next_action": one concrete next command, never an auto-apply} ``` `fixability` reuses the SAME gate `hotato plan` already enforces: * `safe_to_patch` -- the failure maps to one setting AND the battery already has a passing opposite-risk fixture on that axis (the same coverage `hotato plan` requires before proposing a change). * `insufficient_evidence` -- the failure maps to one setting but the opposite-risk fixture is missing, so a change couldn't be verified; or a contract bundle's policy-bound comparison found a violation with no companion moment to verify against. * `needs_human` -- an audio-path problem (echo bleed). * `do_not_patch` -- the threshold funnel: the battery missed an interruption it should have caught AND false-stopped on a backchannel. No single sensitivity threshold fixes both; the fix class is engagement-control. This adds one COMPOSITE battery-level attribution (`event_id: null`, `type: threshold_funnel`) alongside the two per-event attributions it explains. ## Refusals: a precise account of the gap A refusal states exactly which gap in the evidence blocks attributing ONE root cause: * **a not-scorable event** -- an input problem. * **an ambiguous slow yield** (`unknown_root_cause`, no passing opposite-risk fixture) -- TTS buffering, transport latency, and VAD smoothing are indistinguishable from one recording. * **an echo-tagged false stop** -- the agent most likely heard its own TTS bleed, an audio-path problem. * **a sweep/analyze candidate ref** -- ALWAYS refused (no human label); explain prints the promote command for BOTH labels and lets a human choose. * **a contract's false stop with no disambiguating `candidate_kind`** -- a `contract.json` carries a narrower signal set than a run envelope's `diagnose`, so a false stop on `hold` could be a backchannel-discrimination miss, ambient noise, or echo bleed. Explain names the ambiguity and points at `hotato run --dump-frames` / `hotato diagnose` on the original envelope for a definitive read. ``` {"event_id": the refused event, or null, "reason": why the evidence cannot support one attribution, "evidence_for": what IS known, "unknowns": the specific gap, "safe_next_action": one concrete next command} ``` ## Contract bundles: attributed from the bundle's OWN fields, bounded A `contract.json` carries a narrower field set than a run envelope's raw `reasons` text, so a contract-bundle attribution is narrower than a full `diagnose`: * a missed interruption (`expect: yield`, `did_yield: false`) is unambiguous and always attributed; * a "yielded but still failed its policy" contract is attributed by comparing the MEASURED `seconds_to_yield` / `talk_over_sec` against the contract's OWN `policy.pass_conditions` bounds -- a bound comparison, never a guess; * a false stop on `hold` is attributed as echo ONLY when the contract's `source.candidate_kind` says so (created `--from-candidate` an `echo_correlated_activity` moment); otherwise it's refused (see above); * a single bundle covers one moment only, so a contract-bundle attribution caps at `insufficient_evidence` until a companion contract or fixture on the other axis exists. A voice trace, when attached (`hotato trace attach`), folds its findings (`traces/voice_trace.jsonl`) into `evidence_for`; without one, explain adds an explicit unknown -- TTS-cancellation lag, transport latency, and VAD smoothing stay indistinguishable from timing alone. See [`TRACE.md`](TRACE.md). ## Output | Format | Gives you | | --- | --- | | text (default) | a plain summary, per attribution and per refusal | | `--format json` | the full machine shape, schema `hotato.explain.v1` (`src/hotato/schema/explain.v1.json`) | | `--html PATH` | a self-contained report, in the same house style (`hotato.report`'s CSS and escaping) the other HTML reports use | ## Exit codes | Code | Meaning | |---|---| | 0 | explained: nothing attributable (no failing or ambiguous events) | | 1 | explained: at least one attribution or refusal was produced | | 2 | usage error or unusable input (a bad candidate ref, a file that is not a hotato result, or an unreadable contract bundle) | ## What explain measures Every attribution here is evidence-based: a measurement scoped to root cause, with its `evidence_against` and `unknowns` stated on every record. `explain` is read-only, like `diagnose` and `plan` -- applying a change stays a human decision in your own stack. See [`FIX-PLANS.md`](FIX-PLANS.md) and [`FIX-LOOP.md`](FIX-LOOP.md) for the guarded ladder from a plan to a proven fix. ================================================================================ FILE: docs/FIX-LOOP.md ================================================================================ # The closed loop: find -> fix -> prove it's fixed `hotato diagnose` / `inspect` / `plan` (see [FIX-PLANS.md](FIX-PLANS.md)) end at a **proposal**. The closed loop carries it the rest of the way, with the two irreversible decisions kept in human hands: 1. **`hotato patch`** turns a plan's abstract `{field, from, to}` into a **literal, paste-ready artifact** for your platform. It produces the artifact; applying it is your call. 2. You apply it, in your own stack, and re-capture the failing moments. 3. **`hotato verify`** scores the old and new runs against each other: a **battery-scale before/after proof** -- *N of M fixtures that used to fail now pass, K of L hold fixtures still pass.* 4. **`hotato loop`** orchestrates the whole thing and **remembers where it left off**, so you can walk away and come back. Every step produces a proposal, a label, or a reviewable artifact -- you decide what ships to your platform. ## `hotato patch `: the paste-ready change Reads a fix plan (schema `hotato.fixplan.v1`) and renders it per platform: - **Vapi / Retell** (config behind a REST API): a JSON **merge-patch body** plus a ready **`curl`** against the platform's config-update endpoint, using the exact field names the plan carries. ``` hotato plan result.json --stack vapi --assistant-id --out fixplan.json hotato patch fixplan.json ``` ``` curl -X PATCH https://api.vapi.ai/assistant/ \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"stopSpeakingPlan": {"numWords": 2}}' ``` - **LiveKit / Pipecat** (config lives in your agent source): patch emits the exact **source edit** -- the constructor kwarg and literal value to set, e.g. `InterruptionOptions(min_words=1)`. - **generic / unknown stack**: patch names the knob **family** and asks for a concrete stack; it emits no literal body it cannot stand behind. ### The both-axes case: no single knob fixes it `hotato patch` handles config-fixable classes directly. When the plan's decision is `do_not_tune_single_threshold` -- the **both-axes** case, where the battery misses a real interruption AND false-stops on a backchannel at once -- no single config value fixes both. patch prints a vendor-neutral, numbers-free **engagement-control pointer** instead: the problem class (telling a real bid for the floor apart from a backchannel) and the KIND of fix it needs, no product named, no digits. It fires ONLY here; an ambiguous slow yield or a coverage gap gets a "no patch, here's why" explanation instead. Every other non-propose decision (diagnostic checklist, insufficient coverage, already at a documented bound, no change) likewise emits no patch, the reason pointing back at the plan. **Guardrail:** patch runs entirely offline and pins `applies_change` to false. `--format json` emits the full artifact; `--out PATH` also writes it. ## `hotato verify --before --after `: the proof After you apply the change and re-capture the same fixtures, verify scores the old and new run envelopes against each other. Each side is a single `hotato run` envelope JSON (a whole battery) or a directory of them; fixtures pair by `event_id`, then `scenario_id`. ``` hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \ --format json > before.json # the failing take # ... apply the patch, re-capture the fixtures ... hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio-new \ --format json > after.json hotato verify --before before.json --after after.json ``` It reuses the `hotato compare` taxonomy per fixture (`fixed`, `regressed`, `improved`, `worse`, `unchanged`, `still_pass`, `not_scorable`) and aggregate's pooled-distribution definitions for the before/after talk-over and time-to-yield shift. Two axes matter: - **regression axis** -- how many previously-failing fixtures now pass. - **hold axis** -- how many hold-labeled guard fixtures still pass (they must not regress). **Guardrail:** - verify reports **coincidence**: the improvement *coincides with* your change. Hotato measures timing, not a controlled experiment, so coincidence is the strongest claim the evidence supports. - it **refuses** the battery-scale claim when too few fixtures failed to characterize (`--min-n`, default 3): per-fixture facts still print, but the headline proof is withheld, and said so. - an unjudgeable side reports `not_scorable`; a fixture on only one side is unpaired, kept out of the rollup. By default verify measures and exits 0; `--fail-on-regression` exits 1 if any fixture regressed or got worse. ### `--policy hotato.verify.yaml`: the anti-bandaid gate `--policy` turns the measured rollup into a PASS/FAIL gate you can drop into CI. A policy declares two things, and **both** must hold for verify to pass: ```yaml target: improve: talk_over_sec_p95: -0.5 # pooled talk-over p95 must drop by >= 0.5s failed_count: decrease # fewer fixtures may fail than before guardrails: max_new_false_yields: 0 # no hold guard that passed before may newly yield max_not_scorable: 0 # every paired fixture must be judgeable require_hold_fixture: true # the battery must contain a hold fixture... require_yield_fixture: true # ...and a yield fixture ``` ``` hotato verify --before before.json --after after.json --policy hotato.verify.yaml ``` - **`target.improve`** is the success criteria: a signed number is a required delta (`after - before` -- `-0.5` means "must improve by at least 0.5"); a keyword (`decrease`, `increase`, `no_worse`, `no_better`, `unchanged`) states direction only. Metrics: `talk_over_sec_p95`, `seconds_to_yield_p95`, `failed_count`, `false_yield_count`. - **`guardrails`** are hard fail conditions: `max_new_false_yields` and `max_not_scorable` cap what a naive threshold bandaid would silently trade away; `require_hold_fixture` / `require_yield_fixture` make sure the battery tests the opposite axis before it can be certified. verify exits `1` unless **every guardrail holds and every target is met**, so a fix passes only when it improves one axis without regressing the other. A patch that cuts talk-over by making the agent yield to everything meets the talk-over target but trips `max_new_false_yields` on the hold fixtures -- the whole check fails. The `--out verify.html` proof shows the guardrails and targets (a `Policy check: PASSED`/`FAILED` headline, an ok/violated, met/unmet table), same as text and JSON. The policy is parsed with the standard library only -- Hotato's core carries no third-party runtime dependency -- over the small subset the shipped `examples/verify-policy/hotato.verify.yaml` uses. An unknown key, a wrong-typed value, an empty policy, a tab indent, or a list is a clean exit-2 usage error. ## `hotato loop [FOLDER]`: one command, with memory `hotato loop` drives the parts Hotato can drive and remembers where it left off, in a small local state file (`.hotato/loop-state.json` by default): - **First run over a folder of calls**: runs discovery (`analyze` -> `scan` -> rank) and records the candidate moments. Stage `awaiting_label`: > you have N candidate moment(s) awaiting your label. - **You label the ones that matter** with `hotato fixture create` -- the yield/hold intent always comes from a human. - **Next run, with those fixtures present** (`--fixtures DIR`): runs them, diagnoses the battery (including the threshold-funnel check), and plans a guarded fix. Stage `awaiting_verify`: > a fix plan is ready; apply it with hotato patch, then prove it with > hotato verify. ``` hotato loop ./recordings # run 1: discover hotato fixture create --stereo rec.wav --onset 12.4 \ --expect yield --id refund-001 --out tests/hotato hotato loop ./recordings --fixtures tests/hotato # run 2: plan ``` **What stays yours:** every yield/hold intent, and applying the fix (loop produces a plan and points at `hotato patch`; applying and verifying stay your steps). loop tracks state across runs, leaving your platform untouched -- you keep the two decisions that matter: which moment is real, and whether to fix it. ## Exit codes | Command | Exit | Meaning | |---|---|---| | `hotato patch` | `0` | a patch (config artifact or the engagement-control pointer) was produced | | `hotato patch` | `2` | input is not a fix plan, or is unreadable | | `hotato verify` | `0` | the rollup was produced (a low-n claim is refused but still exits 0) | | `hotato verify` | `1` | a gate you opted into failed: `--fail-on-regression` and a fixture regressed or got worse, or `--policy` and a guardrail was violated or a `target.improve` criterion wasn't met | | `hotato verify` | `2` | usage error, unreadable input, an invalid `--policy` file, or no fixtures pair | | `hotato loop` | `0` | advanced or re-reported state | | `hotato loop` | `2` | no folder on the first run, an unreadable state file, or a path that isn't a folder | ================================================================================ FILE: docs/FIX-PLANS.md ================================================================================ # Fix plans: the guarded ladder Hotato turns a failing turn-taking measurement into a reviewable, bounded fix proposal: four rungs, each gated tighter than the last. The first three read only; the fourth touches a clone. ## The ladder | Level | Command | Does | Writes to your stack | |---|---|---|---| | **0** | `hotato diagnose result.json` | Per-failure diagnosis + a battery decision, with the advisory and the tradeoff stated | Never | | **1** | `hotato inspect --stack ...` | Reads the CURRENT turn-taking config (GET or static parse) and normalizes it | Never | | **2** | `hotato plan result.json [target]` | Combines diagnosis + inspected config into a fix-plan JSON (`hotato.fixplan.v1`) | Never | | **3** | `hotato apply` / `hotato fix trial` / `hotato verify` | Applies a plan to a CLONE and re-scores the battery under a pinned manifest | Cloned assistant / branch only, never production | Level 3 keeps the same guard throughout: `hotato apply` targets a CLONED assistant (or a branch config) only, `hotato fix trial` re-scores the battery on that clone under a pinned manifest, and `hotato verify` gates the before/after. Production is a human call: every plan pins `"approval": {"default": "manual", "production_apply": false}`, so it waits on a human go-ahead. See [APPLY.md](APPLY.md), [FIX-TRIAL.md](FIX-TRIAL.md), and [FIX-LOOP.md](FIX-LOOP.md). ## Level 0: diagnose ``` hotato run --suite barge-in --format json > result.json hotato diagnose result.json ``` One diagnosis per failing event: ``` {"finding": missed_real_interruption | false_stop_on_backchannel | slow_yield | excess_talk_over | endpointing_miss | not_scorable | threshold_funnel, "evidence": the measured fields for that event, "likely_layer": interruption_detection | endpointing | unknown_root_cause, "config_only_safe": bool, "notes": plain language} ``` plus one battery-level decision. Text mode prints the Level 0 advisory, for example: "Missed real interruption. Likely config layer. Try lowering the stop-speaking word threshold one step. Tradeoff: may increase false stops on short acknowledgements." Every advisory states its tradeoff. Exit codes: 0 no failing events, 1 failing events diagnosed, 2 unusable input. ## Level 1: inspect ``` hotato inspect --stack vapi --assistant-id # + VAPI_API_KEY hotato inspect --stack retell --agent-id # + RETELL_API_KEY hotato inspect --stack livekit --config agent.py # static parse hotato inspect --stack pipecat --config bot.py # static parse ``` Output: one normalized model (`interrupt_min_words`, `interrupt_voice_seconds`, `resume_backoff_seconds`, `endpointing_wait_seconds`, `backchannel_aware`) plus the raw fields and provenance (what was fetched, when, and which docs the field names were verified against). Absent or unreadable options are null with a note; suspicious values (an unusually high word threshold, a long endpointing wait) are surfaced as observations, for you to judge. Read-only by construction: Vapi and Retell are one GET each; LiveKit and Pipecat files are parsed with `ast`, statically, as text. Missing credentials exit 2 cleanly. ## Level 2: plan ``` hotato plan result.json --stack vapi --assistant-id --out hotato-fixplan.json hotato plan result.json # stack from the envelope, else generic ``` (`--run result.json` is the equivalent flag form.) The input must be a run envelope; frame dumps, benchmark results, and compare results are rejected with exit 2. The plan is `hotato.fixplan.v1` (shipped schema: `src/hotato/schema/fixplan.v1.json`): the finding, a hypothesis, zero or more changes, the verification gate, and the approval block. It also carries `kind: "fix-plan"`, the measured `evidence` behind it, the stated `risks`, `next_commands` (apply the step manually, verify with `hotato compare`, re-run the battery), not-scorable events as `input_issues` (input problems, kept separate from fixes), and a `platform_mutation` block whose `performed` is always false: `hotato plan` reads, it never writes. Twilio rule: Twilio carries the audio; the upstream voice-agent stack decides when the agent yields. `--stack twilio` (or a Twilio-stack envelope) gets a checklist instead of agent-config advice: confirm the dual-channel caller/agent assignment, then re-plan against that upstream stack. ### Policy rules (as implemented and tested) A change is proposed ONLY when ALL of these hold: (a) the failure class maps cleanly to ONE setting; (b) the proposed value is ONE bounded step, an unambiguous direction within documented bounds, anchored to the inspected value as `from`; without inspection, direction and bounds only, `current_unknown: true`; (c) the run's battery contains at least one passing OPPOSITE-RISK fixture, else the plan DOWNGRADES to insufficient_coverage ("add an opposite-risk fixture before tuning") and names the exact fixture family to add; (d) the diagnosis marked the failure config_only_safe. Standing refusals: * **threshold_funnel** -- when the battery contains BOTH a missed real interruption AND a false stop on a backchannel, the plan is a refusal: `"decision": "do_not_tune_single_threshold"` with a vendor-neutral engagement-control pointer (no product names, no digits). One threshold cannot satisfy both axes: raising it for one worsens the other -- the threshold treadmill teams describe from the inside, tuned endlessly with no perfect setting, because both failures share one knob. * **slow_yield without a clear layer** -- TTS buffering, transport latency, and VAD smoothing are indistinguishable from one recording, so the plan proposes a diagnostic checklist (instrumentation steps) first. A slow yield becomes config-only-safe once the battery has a passing opposite-risk backchannel fixture that makes a one-step change verifiable. * **not_scorable events** are input problems, tracked separately from agent failures and kept out of the plan. Every plan, whatever its decision, carries the same verification gate: ``` "required_verification": ["real_interruption_fixture_must_pass", "backchannel_fixture_must_not_regress", "slow_yield_p95_must_not_worsen"] ``` ### Worked example: a safe one-step plan Battery: the agent missed a real interruption; the backchannel fixture passes (the opposite-risk coverage the policy requires). Inspection read `stopSpeakingPlan.numWords = 2` from the live assistant. ```json { "schema": "hotato.fixplan.v1", "target": {"stack": "vapi", "inspected": true, "assistant_id": "asst_123", "current_unknown": false}, "finding": "missed_real_interruption", "hypothesis": "The agent missed a real interruption: the caller took the floor and the agent kept talking. Inspected stopSpeakingPlan.numWords is 2; one bounded step decrease is the smallest verifiable change on this axis.", "config_only_safe": true, "decision": "propose_one_step", "changes": [ {"field": "stopSpeakingPlan.numWords", "from": 2, "to": 1, "direction": "decrease", "bounds": [0, 10], "reason": "The caller took the floor and the agent never stopped within the search window. A one-step sensitivity increase is a config-layer candidate; the tradeoff is more false stops on short acknowledgements. One step decrease from the inspected value. Bounds basis: documented range 0-10 (docs.vapi.ai, 2026-07-06).", "risk": "more false stops on short acknowledgements; the backchannel fixture must not regress"} ], "required_verification": ["real_interruption_fixture_must_pass", "backchannel_fixture_must_not_regress", "slow_yield_p95_must_not_worsen"], "approval": {"default": "manual", "production_apply": false} } ``` One field, one step, both endpoints of the move visible, the risk named, the verification gate attached: every value in the plan traces back to the inspected config or a documented bound. ### Worked example: the refusal Battery: the packaged demo (`hotato demo --format json`), which misses a real interruption AND yields to a bare backchannel. ```json { "schema": "hotato.fixplan.v1", "target": {"stack": "generic", "inspected": false}, "finding": "threshold_funnel", "hypothesis": "The battery missed a genuine interruption and also stopped for a backchannel. One sensitivity threshold cannot satisfy both: raising it to hold through backchannels drops real interruptions, lowering it to catch interruptions yields to backchannels. The failure class is discrimination, not calibration.", "config_only_safe": false, "decision": "do_not_tune_single_threshold", "changes": [], "recommended_fix": { "class": "engagement-control", "examples": [ "enable adaptive interruption handling where available", "use a backchannel-aware interruption classifier", "add addressee/turn-intent discrimination before stopping TTS" ] }, "required_verification": ["real_interruption_fixture_must_pass", "backchannel_fixture_must_not_regress", "slow_yield_p95_must_not_worsen"], "approval": {"default": "manual", "production_apply": false} } ``` The refusal is the product working as designed: the strongest thing a tuning tool can say about a funnel is that tuning will not fix it. ## Provenance Vendor field names and documented ranges were verified against docs.vapi.ai (assistant startSpeakingPlan/stopSpeakingPlan), docs.retellai.com (get-agent), docs.livekit.io (turn handling options), and docs.pipecat.ai (user turn strategies) on 2026-07-06; each result records its own `field_basis`. Where a platform documents no hard range, the plan says so and uses a conservative working range, with a note to verify against the installed version. ================================================================================ FILE: docs/FIX-TRIAL.md ================================================================================ # fix trial: one before/after proof, composed, fail-closed `hotato fix trial` is the fix ladder's last rung: it composes already-shipped, already-guarded primitives into ONE before/after report that says whether a change holds. No new scoring engine, no new networked path: * **`hotato apply`'s offline gate** (`build_apply`, `clone=True`): refusal-first on the both-axes threshold funnel, requires an opposite-risk battery, clone-only. fix trial never calls `apply.create_clone` or `apply._http_json` -- no clone, no network call -- so it carries apply's own clone-only, production-unmutatable guarantee by construction. * **`hotato verify`'s battery-scale rollup**: scores the BEFORE run (the original failure evidence) against the AFTER run (re-captured through the clone you made with `hotato apply --clone --yes`) -- every paired fixture, not just the target failure. The "neighbouring cases" check. * **`hotato contract verify`**, when `--contracts DIR` is given: another neighbouring-cases check, against labelled moments outside the battery. * **`hotato explain`**, folded in as the report's attribution section: root cause of the original failure, reused exactly. ``` hotato patch fixplan.json --format json --out patch.json hotato apply patch.json --clone --name staging-refund-fix --battery tests/hotato # ... re-capture the battery through the source (before/) and the clone (after/) ... hotato fix trial patch.json --name staging-refund-fix \ --before before/ --after after/ --battery tests/hotato \ --policy hotato.verify.yaml --out fix-trial.json --html fix-trial.html ``` ## The verdict is fail-closed, never a soft pass | Verdict | Exit | Fires when | |---|---|---| | `improved` | `0` | verify's claim is supported (>= `--min-n` previously-failing fixtures), at least one now passes, nothing regressed anywhere (including the hold/opposite-risk axis), no contract regressed, `--policy` passed if given, every before target/hold has an after counterpart, and every guarded fixture carries verifiable before/after audio identity (the provenance guard, below) | | `regressed` | `1` | any fixture regressed, a contract regressed, or the policy failed | | `inconclusive` | `1` | too few previously-failing fixtures to characterize, nothing that used to fail now passes, or a guarded fixture's audio identity is present but unverifiable (malformed/missing provenance, or well-formed but not recomputable because the audio was missing at trial time) | | `refused` | `3` | the patch is the both-axes threshold funnel (apply's refusal-first gate fires before any before/after evidence is read); or the after set drops a required before fixture; or a guarded fixture's provenance doesn't match the audio on disk; or a guarded fixture's before/after audio is the SAME conversation (identical decoded PCM -- a re-score, not a recapture) | **`inconclusive` is fail-closed, not a pass.** A low-n or zero-improvement battery exits the same non-zero code as a regression, so CI never treats "we could not tell" as green. A hold fixture that flips from passing to failing is `compare.classify_pair`'s `"regressed"` result whichever axis it's on -- there is no separate, weaker gate for the opposite-risk axis. ## Refusal is a correct output, not an error If the patch is `do_not_tune_single_threshold` (the battery missed a real interruption AND false-stopped on a backchannel), fix trial refuses before reading `--before` / `--after` / `--contracts` at all, prints the canon recommendation, and exits `3` -- the same code `hotato apply` uses for the same refusal, so a script that already branches on apply's refusal code recognizes fix trial's too. ``` No config patch will be applied Reason: both missed real interruption and false stop on backchannel, one threshold cannot safely fix both Recommended: enable or add engagement-control / backchannel-aware turn detection ``` ## Fresh-capture provenance guard: a re-score is never a fix `hotato apply`'s clone-only gate and `hotato verify`'s battery-scale rollup answer "did the numbers move." Neither asks whether the AFTER evidence was re-captured, or is the SAME recording the BEFORE run scored, just re-scored under a looser threshold. That gap is exploitable: run the same fixture twice with different `--max-time-to-yield` / `--max-talk-over` bounds and you get a convincing "improved" verdict with no code, config, or model change behind it. Every run envelope records an `audio_provenance` block per event: a streamed sha256 of the raw file bytes and of the decoded PCM samples (plus sample rate and frame count), computed at capture time by `hotato run` / `hotato capture`. `fix trial` does not trust the string -- it VERIFIES the identity for every GUARDED fixture: the fail->pass targets AND the still-passing holds (a frozen hold is a re-score too). The guard recomputes what it can from disk and states exactly what it verified: | What the guard finds | Why | Verdict effect | |---|---|---| | Well-formed, freshly distinct decoded-PCM identity, recomputed from disk and matching | a verified fresh recapture | proceeds -- eligible for `improved` | | Identical decoded PCM before vs. after | the after run re-scored the SAME conversation (a header-only edit or trailing-byte append can't hide it -- the check is on samples, not container bytes) | `refused` (exit `3`) | | Recorded digest does NOT match the audio on disk | provenance was hand-edited, or the audio was swapped after capture | `refused` (exit `3`) | | A required before fixture (target or hold) missing from the after set | a cherry-picked, incomplete comparison | `refused` (exit `3`) | | Malformed block (non-hex digest, absurd sample rate/frame count, or a top-level digest inconsistent with the per-side digests) | an unvalidated assertion, not a distinct recording | `inconclusive` (exit `1`) | | Provenance block missing on either side | an older or hand-built envelope; identity is UNKNOWN | `inconclusive` (exit `1`) | | Well-formed identity hotato could NOT recompute (audio not present) | asserted, not proven | `inconclusive` (exit `1`) | A provenance-guard refusal is NOT the apply-gate refusal: it fires AFTER `verify` / `contract verify` / `explain` already ran, so -- unlike the both-axes refusal, which reads no evidence at all -- the full report (verify's proof, the contract rollup, the provenance identities, the attribution) still renders below the refusal banner. Every refusal path exits the SAME code `3`; `refusal_kind` in the JSON output (`"threshold_funnel"`, `"incomplete_after"`, `"recompute_mismatch"`, `"same_audio_recapture"`) tells them apart for a script that wants to. ``` No fix will be certified from re-scored audio Reason: 1 fixture(s) this verdict rests on (f1) have identical before/after decoded PCM: the after run re-scored the SAME conversation the before run scored, just against a different threshold or scorer config Recommended: recapture the fixture(s) through the applied clone (hotato apply --clone --yes) and re-run hotato fix trial against the new after evidence ``` Every rendered report (text, `--format json`, `--html`) shows the short digest and verified status for every guarded fixture, before and after -- a reader never has to take "fresh capture" on faith. The effective `--min-n` is echoed in every surface too, so a lowered floor is always visible. The report's own conclusion states plainly what a passing check proves: that the fresh take passed the same human-labeled contract, not that the change caused it (hotato reports coincidence, never causation, throughout). The text and HTML renders of this section also print, verbatim, wherever it appears (an `improved` verdict, or a `refused`/`inconclusive` one the guard itself downgraded): *"Provenance caution: this proves the specific fresh capture scored above, at the revision it was captured from. It does not certify a later deploy or every future call, and it does not re-run itself; recapture again after the next change."* See [`docs/RECAPTURE.md`](RECAPTURE.md#claim-language-what-each-kind-of-evidence-lets-you-accurately-say) for the fuller claim-language table this line is drawn from. ## What this does not stop This is an offline tool: a user who controls every input can lie to themselves. Recomputing identity from the audio (above) makes the specific forgeries an external red-team demonstrated against a prior build -- hand-written envelopes, a flipped header byte, a re-scored recording, a cherry-picked after set -- impossible or loud. None of the following is a bug the guard missed; each sits outside what an offline recompute over supplied files can ever establish: * **Fabricated inputs are still yours to fabricate.** Hand fix trial a fresh recording of a call that never happened, or one that does not match the bug you are claiming to fix, and the guard verifies the audio identity and still reaches `improved`. It checks that the bytes it scored are what they claim to be, never that the stimulus itself is real. * **A contract's `MANIFEST.sha256.json` is integrity, not authenticity.** It proves the archive agrees with itself after packing, not who approved the policy inside it. Loosen a `.hotato` bundle's policy (raise `max_talk_over_sec`, say) before `contract pack`, and `contract verify` on the repacked bundle still passes -- it re-checks the archive against itself, not against an external record of what the policy was supposed to be. Only a trusted signature over the manifest closes this; none is implemented today. * **A resample, re-encode, or gain change of the SAME call still changes the decoded PCM.** The freshness check above is exactly "is the decoded PCM different." A deliberately transcoded copy of the identical recording (resampled, gain-adjusted, round-tripped through a lossy codec) decodes to different samples, so it reads as a distinct capture -- it is not; it is the same call in a different container. A known, undetected residual of a PCM-identity check, not a claim the guard breaks. * **Signatures are not implemented.** Nothing here is cryptographically signed; a sha256 digest is a checksum, not an attestation of who produced it. A green fix trial does not prove the audio was freshly captured for the scenario claimed, that the same policy and labels were used throughout, that any omitted fixture was safe to omit, that the named revision or clone existed, that the patch was applied to it, or that the deployed agent improved. The verdict states exactly what it recomputed and verified; everything else sits outside what an offline tool can promise. ## Flags - **`PATCH_JSON`** -- a `hotato patch` artifact. - **`--name NAME`** -- name of the staging clone this trial is proving (required for a non-refused patch; the same `--name` `hotato apply` takes -- fix trial evaluates the SAME clone-only gate, it never creates a clone itself). - **`--before RUN.json|DIR`** -- the OLD run envelope(s): the original failure evidence. Also the default opposite-risk `--battery`, and the attribution source, when those are omitted. - **`--after RUN.json|DIR`** -- the NEW run envelope(s), re-captured through the staging clone. - **`--battery DIR`** -- the opposite-risk battery apply's gate checks (BOTH a yield and a hold fixture); defaults to `--before`. - **`--contracts DIR`** -- also re-verify a directory of hotato contracts; any contract regression fails the trial. - **`--policy hotato.verify.yaml`** -- gate verify's rollup (see [`docs/FIX-LOOP.md`](FIX-LOOP.md)); a violation fails the trial. - **`--min-n N`** -- minimum previously-failing fixtures needed to support the claim (default 3). - **`--out PATH`** -- also write the full proof JSON. - **`--html PATH`** -- also write a self-contained before/after HTML report. ## Output Every surface shows the apply receipt beside the verdict: fix trial calls `apply.build_apply`, never `apply.create_clone`, so `apply_dry_run` is `True` and `apply_created` / `apply_applies_change` are `False` on every run, including an `improved` one -- a green verdict never means "and the change was applied." Text prints `apply: dry_run=True created=False applies_change=False` plus a plain-English line; JSON carries the same fields (`apply_dry_run` / `apply_created` / `apply_applies_change` / `apply_receipt_note`) at the top level next to `verdict`; HTML renders them as header pills. Proving the change reached the clone or agent is `hotato apply --clone --yes`'s job, recorded in its own receipt -- fix trial only proves what the before/after evidence shows once that's already happened. * **text** (default): the apply receipt, the verdict, verify's own rendered proof, the contract verify rollup when `--contracts` was given, and the attribution section (one `hotato explain` render per file under `--before`). When the trial's own verdict is not `improved` (a provenance, completeness, contract, or policy issue downgraded it), verify's nested `CLAIM` line is tagged `CLAIM (SUPERSEDED BY {VERDICT})` with a one-line restatement, whenever that sub-claim would otherwise read "supported" -- a fix-trial verdict of `regressed` / `refused` / `inconclusive` can still contain a verify claim that, read alone, looks like a pass; the parent verdict controls. * **`--format json`**: the full machine shape, schema `hotato.fix_trial.v1`. Every sub-result is the nested result the underlying command already produces (`apply`, `verify`, `contract_verify`, `attribution`), not a re-derived summary. `apply_dry_run` / `apply_created` / `apply_applies_change` / `apply_receipt_note` sit at the top level next to `verdict`, not only inside the nested `apply` object. * **`--html PATH`**: a self-contained report, reusing the same house style the other HTML reports use: the apply-receipt pills and note in the header, the verdict chip, the verify proof table (with the same superseded-claim label as text, when it applies), the contract-verify rollup, and the attribution cards. ## Exit codes - **0** -- `improved`. - **1** -- fail-closed: `regressed` or `inconclusive`. - **2** -- usage error: the same gates `hotato apply` enforces (no `--name`, no opposite-risk battery, a stack with no clone target, a patch with no concrete change) or `hotato verify` / `contract verify` already enforce (no fixtures pair, an invalid `--policy`, a `--contracts` dir with no contracts, unreadable input). - **3** -- principled refusal: the both-axes threshold funnel. Shared with `hotato apply`'s refusal code. ## What this is not fix trial does not create, patch, or delete anything on your platform; it never calls `apply.create_clone`. It does not prove authorization, identity, compliance, or policy safety. It reports coincidence, never causation -- every underlying `verify` and `contract verify` number keeps that rule. See [`FIX-LOOP.md`](FIX-LOOP.md) for the manual steps this composes, and [`APPLY.md`](APPLY.md) / [`EXPLAIN.md`](EXPLAIN.md) for the primitives themselves. ================================================================================ FILE: docs/FULL-DUPLEX.md ================================================================================ # Full-duplex turn-taking: scoring the moment both sides speak at once In a full-duplex conversation the agent hears while it speaks, so overlap is a normal part of the audio: a caller barges in mid-sentence and both voices are live at the same time. The question a transcript cannot answer is what happened next, in seconds: did the agent drop its turn when the caller took the floor, how fast, and how long did both keep talking at once. hotato measures exactly that from the two channels of the recording -- `did_yield`, `seconds_to_yield`, `talk_over_sec` -- the same timing physics whether the agent runs half- or full-duplex. `examples/full-duplex/` is the runnable pair for that claim: two scripted two-channel scenarios, byte-identical on every render, that differ in one variable only -- what the agent's channel does after the barge-in. | id | behaviour | outcome | |---|---|---| | `fdx-01-barge-in-clean-yield` | caller takes the floor at 2.00 s; the agent stops inside the bounds | overlap, then a clean yield: PASS | | `fdx-02-barge-in-talk-over` | same barge-in; the agent keeps transmitting through 2.75 s of simultaneous speech | sustained talk-over: FAIL | Both scenarios declare the same expectation (`yield: true`, `max_time_to_yield_sec: 1.00`, `max_talk_over_sec: 1.00`) and the same caller onset. The PASS case still contains measured simultaneous speech: overlap itself is not the failure, holding the floor through it is. ## Render the pair (deterministic) ```bash python examples/render_examples.py ``` The per-channel seed is `sha256(scenario_id)`, so two renders are byte-identical on any machine; `tests/test_examples_full_duplex.py` renders twice and diffs to pin it. The audio is synthetic, energy-shaped noise -- a scripted fixture, never a recorded person. ## Run both: one PASS verdict, one FAIL verdict ```bash hotato run --scenarios examples/full-duplex/scenarios --audio examples/full-duplex/audio ``` ``` hotato [suite] stack=generic offline=True 1/2 events pass (failed=1) [PASS] fdx-01-barge-in-clean-yield: did_yield=True seconds_to_yield=0.75s talk_over=0.75s [FAIL] fdx-02-barge-in-talk-over: did_yield=True seconds_to_yield=2.75s talk_over=2.75s fix[config]: Slow yield: the agent stopped, but too late knob: endpointing / VAD min-silence-duration move: lower the min-silence and hangover so the agent goes quiet sooner after the caller takes the floor exit_code=1 ``` The battery exits 1 because it carries one caught regression. In the JSON envelope (`--format json`) the FAIL verdict states its reasons exactly: ``` yielded in 2.75s, slower than the 1.00s bound talked over the caller for 2.75s, more than the 1.00s bound ``` Note what the FAIL is not: in `fdx-02` the agent DID yield eventually (`did_yield=True`), unlike `funnel-demo`'s `fd-01`, where the agent never stops inside the search window. The regression here is the measured seconds of talk-over on the way to the floor transfer -- the full-duplex failure mode, invisible to a text-level eval. ## Score either recording on its own The same verdicts, one file at a time, with the bounds on the command line: ```bash hotato run --stereo examples/full-duplex/audio/fdx-01-barge-in-clean-yield.example.wav \ --onset 2.0 --expect yield --max-time-to-yield 1.0 --max-talk-over 1.0 # exit 0 hotato run --stereo examples/full-duplex/audio/fdx-02-barge-in-talk-over.example.wav \ --onset 2.0 --expect yield --max-time-to-yield 1.0 --max-talk-over 1.0 # exit 1 ``` ## Investigate the failure `hotato investigate` ranks the caught moment without any label given up front: ```bash hotato investigate examples/full-duplex/audio/fdx-02-barge-in-talk-over.example.wav ``` ``` hotato investigate [run 1]: fdx-02-barge-in-talk-over.example.wav capture origin: frozen regression clip (examples/full-duplex/scenarios/fdx-02-barge-in-talk-over.json) this recording is a previously-created hotato fixture clip (fdx-02-barge-in-talk-over.json), not a live call: a pinned regression, not fresh evidence input health: eligible for scan verdict path: eligible (a labeled event here can carry a real yield/hold verdict) most likely failure (top-ranked candidate): [1] t=1.99s overlap_while_agent_talking overlap_sec=2.76 next: label it (use --expect hold instead if the agent was right to keep talking): hotato investigate label '.hotato/investigate-state.json#1' --expect yield state remembered at: .hotato/investigate-state.json ``` The origin line says out loud that this is a fixture clip, not a live call. On your own recording the same command prints the same ranked moments and the one `investigate label` next step that pins the catch as a CI contract -- the core loop in [AGENTS.md](../AGENTS.md) from step 2 on. ## Exit codes | Exit | Meaning | |---|---| | `0` | every scorable event passed its declared bounds | | `1` | a scorable event regressed (here: the talk-over bound) | | `2` | usage error or unusable input (e.g. a mono export: NOT SCORABLE) | ## Related - [`examples/README.md`](../examples/README.md) -- every additive fixture set, including this pair, and the deterministic renderer. - [INVESTIGATE.md](INVESTIGATE.md) -- one recording to ranked candidate moments and a signed contract. - [CONTRACTS.md](CONTRACTS.md) -- the CI gate that re-runs the stored evidence deterministically. - [SIMULATE.md](SIMULATE.md) -- the scripted-caller renderer for transcript/trace fixtures (`origin=simulated`), the no-audio side of the same regression foundation. ================================================================================ FILE: docs/GALLERY.md ================================================================================ # Gallery Every image and worked example Hotato ships, in one place, each one reproducible with the command shown beside it. ## Cards: findings as PR images Four self-contained SVGs: offline, byte-stable from the same input, every font, image, and script inlined. Full spec: [CARDS.md](CARDS.md). - ![threshold funnel](assets/cards/no-single-threshold-card.svg) **The hero.** A battery where a missed interruption and a false stop fail together -- Hotato names the failure class (engagement-control) instead of picking one sensitivity dial. Regenerate: `hotato card fix-plan.json --out card.svg` - ![talk-over candidate](assets/cards/talk-over-card.svg) A measured talk-over moment: the agent kept the floor while the caller was speaking. Regenerate: `hotato card hotato-sweep.json#N --out card.svg` - ![false-stop candidate](assets/cards/false-stop-card.svg) A measured false-stop moment: the agent went quiet with no caller nearby to explain it. Regenerate: `hotato card hotato-sweep.json#N --out card.svg` - ![say-do failure](assets/cards/say-do-card.svg) A say-do failure: the conversation claims an outcome the trace and post-call state do not back. Regenerate: `hotato card saydo/test-run.json --out card.svg` ## The bundled demo, rendered `hotato demo` writes a self-contained dashboard with embedded audio and a hear-the-bug playhead. The report and the terminal walkthrough that produced it: - Report: [`assets/hotato-demo-report.png`](assets/hotato-demo-report.png) · [`assets/hotato-demo-report.html`](assets/hotato-demo-report.html) - Terminal walkthrough: [`assets/hotato-demo.gif`](assets/hotato-demo.gif) ## Trust gallery: eight input conditions, eight verdicts The full worked set, with verbatim CLI output for every row of the [trust matrix](TRUST-MATRIX.md), lives in [TRUST-GALLERY.md](TRUST-GALLERY.md). Summary: | # | Condition | Verdict | |---|---|---| | 1 | Clean dual-channel | Scores, full confidence | | 2 | Silent caller channel | `NOT SCORABLE`, exit 2 | | 3 | Silent agent channel | `NOT SCORABLE`, exit 2 | | 4 | Swapped channels | Scores, with a swap warning | | 5 | Crosstalk / echo bleed | Scores, at lower confidence | | 6 | Mono (mixed channel) | `NOT SCORABLE` by default, exit 2; two opt-in escapes, both indicative only | | 7 | Backchannel candidate | Surfaced as a candidate; you label it | | 8 | Noisy false-positive candidate | Surfaced, with the reason it is likely spurious | ## Case studies Recorded-audio write-ups, each following [case-study-TEMPLATE.md](evidence/case-study-TEMPLATE.md), each carrying a mandatory "What Hotato did not prove" scope section: - [`case-studies/vapi-01-hard-interruption.md`](case-studies/vapi-01-hard-interruption.md) The full index and the launch target (3 external or semi-external studies, 5 consented fixtures, 1 before/after, 1 public PR) is in [`case-studies/README.md`](case-studies/README.md), tracked against the gap in [evidence/validation-plan.md](evidence/validation-plan.md). ## Where this fits This page is the visual survey. For what counts as evidence and how it's ranked, see [EVIDENCE-PACK.md](EVIDENCE-PACK.md) and [evidence/README.md](evidence/README.md). ================================================================================ FILE: docs/GENERATIVE-CALLER.md ================================================================================ # Bounded generative caller `hotato.caller` runs caller-side conversation programs against a media or signaling session supplied by an adapter. It supports scripts, state-reactive graphs, allow-listed model proposals, and model-free frozen replay. The output is a content-addressed evidence package. The caller has participant authority only. It cannot determine whether the agent completed a task, mutate backend state, write tool results, or produce a test verdict. Feed the captured call, tool trace, and state evidence into the separate Hotato exercise/evaluation path after the caller finishes. ## Execution modes | Mode | Model use | Intended use | |---|---:|---| | `scripted` | forbidden | fixed regression inputs | | `hybrid` | only at `generate` nodes | deterministic flow with bounded caller variation | | `generative` | required | state-reactive caller programs with validated proposals | | `frozen_replay` | forbidden | replay the text/PCM and signaling actions bound by a prior package | Frozen replay repeats caller outputs. It does not claim to reproduce the agent, carrier, network, or timing scheduler. When a source action contains PCM, replay sends those PCM16LE bytes and refuses a transport without `send_audio`. When the source action contains text only, the text is replayed; any downstream speech synthesis remains outside the evidence boundary. ## Plan Plans use `hotato.caller-plan.v1`. Every loop and resource has a hard bound. This example listens for a refund request, observes a successful tool event, then allows a model to propose only caller speech. ```json { "schema": "hotato.caller-plan.v1", "id": "refund-follow-up", "mode": "hybrid", "start": "greeting", "initial_state": {"journey": "refund"}, "limits": { "max_steps": 40, "max_turns": 8, "max_duration_ms": 120000, "max_model_calls": 2, "max_tokens": 2000, "max_cost_microusd": 0 }, "nodes": [ { "id": "greeting", "type": "say", "text": "I need help with a refund.", "next": "agent-reply" }, { "id": "agent-reply", "type": "listen", "timeout_ms": 10000, "max_events": 8, "until": {"event": "tool_result", "tool": "refund", "status": "success"}, "next": "follow-up", "on_timeout": "leave" }, { "id": "follow-up", "type": "generate", "prompt": "Ask for the refund confirmation identifier in one sentence.", "allowed_actions": ["say"], "next": "leave" }, {"id": "leave", "type": "hangup", "reason": "scenario_complete"} ] } ``` The code API is intentionally adapter-injected: ```python from hotato.caller import load_plan, run_caller result = run_caller( load_plan("refund.caller.json"), session=my_session, output_dir="artifacts/refund-caller", model=my_caller_model, tts=my_tts, ) assert result.verification["ok"] ``` ## Graph nodes | Node | Effect | |---|---| | `say` | sends fixed participant text, or bound PCM when a TTS adapter is present | | `generate` | asks a model for one allow-listed action and validates its exact shape | | `listen` | consumes bounded session events until a trigger matches or times out | | `wait` | advances the adapter's caller schedule | | `dtmf` | sends declared DTMF digits | | `silence` | emits a declared silence interval through the adapter | | `impairment` | asks the adapter to apply a named/configured media profile | | `expect` | branches on already observed events or caller working state | | `set_state` | updates caller working state; outcome/verdict roots are forbidden | | `branch` | takes the first matching trigger, with an optional default | | `repeat_bounded` | visits a target at most `max_iterations` times | | `transfer_expect` | requires observable evidence of a completed transfer | | `hangup` | ends the caller leg with a recorded reason | Every run also enforces `max_steps`, `max_visits_per_node`, `max_events`, `max_event_chars`, `max_wait_ms`, model request/response character limits, text and PCM limits, token limits, and a micro-USD spend ceiling. The default spend ceiling is zero. A hosted model adapter must report integer usage and an operator must raise the ceiling explicitly. Missing usage is a blocked run. ## Triggers Triggers are deterministic predicates over normalized session events. They can be combined with `all`, `any`, and `not`. ```json { "any": [ {"event": "transcript", "text_regex": "refund"}, {"event": "transcript", "text_regex": "chargeback"} ] } ``` ```json {"event": "tool_result", "tool": "refund", "status": "success"} ``` ```json {"event": "state_snapshot", "path": "subscription.status", "equals": "cancelled"} ``` ```json {"event": "timing", "metric": "agent_latency_ms", "gte": 800} ``` The normalized event kinds are `transcript`, `tool_result`, `state_snapshot`, `dtmf`, `lifecycle`, `transfer`, `hold`, `timing`, `timeout`, and `custom`. An adapter may put fields at the top level or under `data`. Each accepted event receives a canonical SHA-256 identity in the result. `text_regex` uses a 512-character safe subset: literals, escaped characters, `.`, character classes, boundary `^`/`$`, and bounded flat repeats. Groups, alternation, lookaround, backreferences, nested repetition, more than one unbounded repeat, unanchored unbounded repeats, and repeat sets exceeding the fixed path budget are refused when the plan is loaded. Regex searches inspect at most 1,024 characters; a larger candidate text ends the caller run with the stable `REGEX_SEARCH_TEXT_LIMIT` `ERROR`. Use `any` for alternatives, as above. ## Session contract The engine does not implement SIP, RTP, WebRTC, telephony, STT, or TTS. A `CallerSession` owns those operations: ```python class CallerSession: def capabilities(self): ... def send_text(self, text, metadata): ... def send_audio(self, pcm_s16le, sample_rate_hz, metadata): ... def receive(self, timeout_ms): ... def send_dtmf(self, digits): ... def wait(self, duration_ms): ... def silence(self, duration_ms): ... def set_impairment(self, profile): ... def hangup(self, reason): ... ``` Capabilities use exactly three states: - `SUPPORTED`: the requested operation can be executed and observed at this boundary. - `UNSUPPORTED`: the adapter declares that it cannot perform the operation. - `UNOBSERVABLE`: the operation may occur, but the adapter cannot supply the evidence required by the node. Missing capabilities are treated as `UNSUPPORTED`. A transfer expectation with `UNOBSERVABLE` evidence produces `CAPABILITY_UNOBSERVABLE`; it is never recoded as a failed transfer. Concrete adapters should run on top of maintained media substrates. Examples include a Pipecat transport for streaming caller audio, a LiveKit room/SIP sidecar for WebRTC or SIP legs, and a SIPp process for fixed high-rate SIP fixtures. The caller engine controls the program and evidence contract while those projects own their protocols. Two optional concrete adapters cover common local paths without changing that separation: `LiveKitCallerSession` joins a LiveKit room through the Python RTC SDK, and `PiperCallerTTS` invokes a local Piper model with bounded, digest-bound provenance. See [Direct LiveKit caller session](LIVEKIT-CALLER-SESSION.md) and [Local Piper caller speech](PIPER-CALLER-TTS.md). Neither adapter claims PSTN delivery, carrier behavior, or speech quality from successful local execution. ## Model contract and authority boundary A `CallerModel.propose(request)` response has six required fields: ```json { "proposal": {"action": "say", "text": "What is the confirmation number?"}, "raw": "provider response or parsed response text", "provider": "local-provider-name", "model": "model-identifier", "parameters": {"temperature": 0, "seed": 7}, "usage": {"input_tokens": 120, "output_tokens": 12, "cost_microusd": 0} } ``` The only model-proposable actions are `say`, `dtmf`, `silence`, and `hangup`. Each `generate` node narrows that set with `allowed_actions`. The proposal must contain the exact fields for its action. Extra fields are refused. `set_state`, tool results, backend state, assertions, outcomes, and verdicts cannot be model proposals. The model request and response are saved separately and bound by SHA-256. The run records provider, model identifier, parameters, usage, and proposal. This is provenance, not a claim that hosted inference is repeatable. ### Local Ollama adapter `OllamaCallerModel` is the built-in zero-dependency model adapter. It accepts loopback HTTP endpoints only, refuses redirects, requests one JSON object, and requires Ollama's input/output token counts. ```python from hotato.caller import OllamaCallerModel model = OllamaCallerModel( "your-local-caller-model", endpoint="http://127.0.0.1:11434", temperature=0, seed=7, ) ``` The adapter records a zero API charge because the daemon is local. Hardware, energy, and operator costs are not represented by `cost_microusd`. A hosted model remains an injected `CallerModel` implementation and must report its metered charge. ## TTS contract `CallerTTS.synthesize(text)` returns: ```python { "pcm_s16le": pcm_bytes, "sample_rate_hz": 16000, "provider": "local-tts", "model": "model-id", "voice": "voice-id", "settings": {"speed": 1.0}, } ``` All fields are required. The run writes the exact PCM16LE bytes and the UTF-8 source text as separate artifacts and records both hashes. No text-only fallback occurs when a TTS adapter is configured and `send_audio` is absent. ## Evidence package Each output directory contains: ```text caller-plan.json caller-result.json package-manifest.json artifacts/ text/ # exact UTF-8 participant utterances audio/ # exact PCM16LE submitted at the caller session boundary model-request/ # canonical request per model call model/ # canonical response/provenance per model call ``` `package-manifest.json` binds the exact file set, byte sizes, and SHA-256 digests. `verify_package()` rejects missing, changed, symlinked, path-escaping, and unlisted files. Frozen replay verifies the complete source package before performing any session action. `caller-result.json` keeps these lanes separate: - `actions`: participant outputs and signaling requests; - `events`: normalized observations supplied by the session adapter; - `model_calls`: model request/response provenance and usage; - `actor_state`: scenario-owned working memory; - `authority`: fixed declarations that outcome is unevaluated and no verdict was produced. `COMPLETED` and `HUNG_UP` exit zero. Capability gaps, invalid model/TTS output, and resource limits return a verifiable `BLOCKED` package. Adapter exceptions return a verifiable `ERROR` package. Neither state is converted into an agent quality verdict. ## Frozen replay ```json { "schema": "hotato.caller-plan.v1", "id": "refund-follow-up-replay", "mode": "frozen_replay", "frozen_package": "artifacts/refund-caller" } ``` No model or TTS adapter is called. The replay result names the verified source package ID. Tampering with source text, PCM, model material, plan, or result blocks replay before a session operation occurs. ## Remaining acceptance work for a concrete transport This engine does not establish that a particular adapter delivers configured audio or signaling to a target. Before publishing transport claims, execute a common fixture through the adapter and retain target-boundary captures for PCM, DTMF, hold, transfer, disconnect, and impairment behavior. Report unsupported and unobservable operations as such. Do not infer carrier behavior from a successful local method call. ================================================================================ FILE: docs/GUARDIAN-FLEET.md ================================================================================ # Hotato Guardian / Fleet Guardian/Fleet runs Hotato's capture → scan → trust → label → contract → trial primitives as one continuous workflow inside a private, self-hosted control plane: a hardened evidence kernel with a fleet view on top. Every production deploy stays a human call. ## The evidence kernel ### An evidence vector, scored dimension by dimension Every artifact carries a machine-readable **evidence vector** (`hotato/evidence.py`, schema `evidence_vector.v1.json`): each dimension scored on its own, public tier set to the *weakest* tier any required dimension allows -- a minimum over an inspectable lattice. Tiers, ascending: | tier | name | meaning | |---|---|---| | 0 | none | no usable evidence for a positive claim | | 1 | asserted | envelope-only; not recomputed from audio | | 2 | measured | recomputed from audio, one recording, input clean | | 3 | paired | manifest-bound before/after, both sides recomputed under one pinned policy | | 4 | attested | paired + signed policy/contract + runner-attested capture | Dimensions: `score_integrity`, `audio_identity`, `policy_integrity`, `fixture_set_integrity`, `input_health`, `channel_mapping`, `label_authority`, `pairing_integrity`, `capture_origin`. One weak dimension pulls the whole tier down; no renderer may raise it. ### Trial manifest (`hotato/manifest.py`, `trial_manifest.v1.json`) An immutable pin created *before* an experiment, from the battery: the scorer config hash, one policy (applied to both sides), the complete fixture universe, and each fixture's expectation, onset, and scripted-stimulus identity. Before and after must reference the same `manifest_hash`; a changed policy, scorer, label, onset, or fixture set refuses the comparison. ### Recompute-from-audio (`hotato/recompute.py`) The proof gate re-derives every verdict from the on-disk audio under the manifest, checks each stored `verdict.passed` against the recomputed result, and refuses: - **verdict tampering** -- a stored verdict that disagrees with the recomputed one; - **same-audio re-encode** -- before and after decode to the same PCM; - **dropped fixtures** -- either side omits a pinned fixture; - **unrelated audio** -- the after-side caller stimulus does not match the pinned one and no capture receipt binds it. A green *paired* proof additionally requires evidence tier >= paired. ### Capture receipt (`hotato/receipt.py`, `capture_receipt.v1.json`) A fresh-recapture claim needs more than distinct PCM: a capture runner emits a receipt binding the recording to a trial, agent, call id, timestamps, and decoded PCM; an HMAC signature makes it *machine-verified*. A manual distinct WAV with no receipt is *operator-asserted* -- the receipt is what promotes it to machine-verified fresh recapture. ### Contract authenticity (`hotato/attest.py`) Contracts embed a canonical digest over schema + label + policy + audio + scorer version, which `contract verify`/`unpack` recompute: a bundle edited after creation (a loosened policy re-pack, say) reports **tampered** and fails; a matching but unsigned bundle is *"unsigned, internally consistent evidence"*; a valid HMAC signature promotes it to *authenticated*. ### Boundary sensitivity Every event exposes `onset_frame_index`, `onset_effective_sec`, `decision_margin_sec`, `decision_margin_hops`, and `boundary_sensitive`: a result within one hop of flipping is flagged, so it never reads with the confidence of a result a hundred milliseconds inside the limit. ### Trust headline Any verdict-changing warning (low signal, possible channel swap, VAD-relevant leakage) sets the headline to `scan with caution`. `input_health` is an explicit three-state field (`clean` / `caution` / `not_scorable`); leakage is judged both against a fixed -40 dB bar and dynamically against the receiving channel's own VAD gate. ## Fleet control plane (local mode) Local mode is zero-dependency: SQLite metadata plus a content-addressed artifact directory, both stdlib. - **Registry** (`hotato/fleet/registry.py`) -- workspace-scoped rows for agents, deployments, calls, recordings, candidates, labels, contracts, trials, decisions. Agents scale per workspace with no product-level cap; every row carries `workspace_id`, scoping its paths and call ids. - **Artifact store** (`hotato/fleet/store.py`) -- content-addressed blobs with dedup and lineage; audio is stored separately from any UI (privacy reversal). - **Job queue** (`hotato/fleet/jobs.py`) -- leased jobs with deterministic idempotency keys, heartbeats, retries, and dead-lettering, so duplicate webhooks and worker crashes converge on one logical result. - **Guardian API** (`hotato/fleet/api.py`) -- ingest → discover (trust preflight + scan) → human review + label → manifest-bound before/after experiment that **recommends a change for you to apply**, and refuses forged / same-audio / incomplete trials. ### CLI ``` hotato fleet init -w acme hotato fleet agent add -w acme --agent-id support-bot --stack vapi --assistant-id asst_123 hotato fleet ingest -w acme --agent support-bot call.wav hotato fleet discover -w acme --agent support-bot call.wav hotato fleet review -w acme hotato fleet label --decision yield --reviewer you hotato fleet experiment run -w acme --agent support-bot --trial-id t1 \ --battery before/run.json --before before/ --after after/ hotato fleet status -w acme ``` Live clone/recapture/canary run against a connected stack with credentials and a tested rollback; every release recommends the change and keeps routing production traffic a manual, human-gated step. ## Synthetic perturbations (`hotato/synth.py`) Deterministic acoustic transforms of real fixtures -- resample, gain, noise, leakage, channel invert, silence, onset offset, clip -- each carrying its parent hash, recipe, seed, and an explicit synthetic designation. Synthetic and real stay on separate axes: only a real recapture raises the confidence tier. ## What fleet mode ships today Evidence stays per-dimension, never folded into a single fleet-wide score, and every contract needs human approval before it exists. Today's live provider adapters are Vapi and Retell (clone→apply→recapture); production deploys stay the same manual, credentialed step described above. ================================================================================ FILE: docs/INGEST.md ================================================================================ # `hotato ingest` -- the composable passive on-ramp Wire a webhook to `hotato ingest` once, and every completed call is scanned for **candidate** turn-taking moments, automatically. `ingest` surfaces candidates; you decide which matter and promote them with `hotato fixture create` -- the yield/hold label always comes from you. Built by **composition**, adding only a per-stack webhook parser: ``` parse the webhook payload -> extract the call id / recording locator fetch the recording -> hotato.capture (the SAME fetch the adapters use) scan for candidates -> hotato.scan (no labels, no verdict) write a candidate report -> JSON always; --out an HTML report (optional) ``` ## How it runs - **You own the trigger.** Wire it to a webhook handler, a serverless function, or a cron over your call log. It runs offline, self-hosted; the only network call is the same fetch `hotato capture` already makes. - **You own the label.** A candidate is a timing event. Only you know whether a caller sound was "mhm" or "stop," so the label is yours. ## Contract ``` hotato ingest --stack {vapi|retell|twilio|livekit|pipecat} \ (--event PAYLOAD.json | --call-id ID | --recording-sid RE...) \ [--out report.html] [--format text|json] [--allow-mono] [--top N] [--min-gap S] ``` - **Exit 0** -- ran (candidates reported, possibly zero). - **Exit 2** -- parse / fetch / IO error, or not-scorable input (a mono recording, for example, which can't attribute overlap to caller vs agent). Never a pass/fail. `--format` controls stdout (`text` listing, or `json` candidates capped by `--top`). `--out` additionally writes an HTML report with every candidate. A webhook payload is **untrusted DATA**: `ingest` reads only the named locator fields. ## Wire your webhook -> `hotato ingest` Point your platform's call-completed webhook at a small handler that saves the payload and shells out to `ingest` -- identical for every stack: ```python # a minimal webhook handler (framework-agnostic pseudocode) import json, subprocess, tempfile, os def on_call_completed(request): payload = request.body # the platform's webhook payload with tempfile.NamedTemporaryFile( "w", suffix=".json", delete=False) as fh: fh.write(payload) event_path = fh.name # ingest is one process; run it out-of-band so the webhook returns fast. subprocess.Popen([ "hotato", "ingest", "--stack", "vapi", "--event", event_path, "--out", f"candidates/{request_id}.html", ], env={**os.environ, "VAPI_API_KEY": os.environ["VAPI_API_KEY"]}) return 200 ``` A cron over your call log works just as well when you'd rather batch: ```bash # nightly: scan yesterday's calls for candidates for id in $(your-call-log --since yesterday --ids); do hotato ingest --stack vapi --call-id "$id" \ --format json --out "candidates/$id.html" >> candidates/$id.json done ``` ### Per-stack recipe Webhook field paths verified against live vendor docs (2026-07-07); an unconfirmed field is parsed **defensively** -- missing, never fabricated. | Stack | Webhook | Field ingest reads | Credentials for the fetch | |-------|---------|--------------------|---------------------------| | **vapi** | end-of-call-report | `message.call.id` (confirmed) | `VAPI_API_KEY` | | **retell** | call webhook | top-level `event` + `call.call_id` (confirmed) | `RETELL_API_KEY` | | **twilio** | `recordingStatusCallback` | `RecordingSid` (confirmed; form-encoded body) | `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` | | **livekit** | egress webhook | `egressInfo.fileResults[].location` / `.filename` (defensive) | none -- egress lands in your storage | | **pipecat** | your own event | `recording_path` / `recording_url` (defensive) | none -- you produced the file | For **vapi / retell / twilio**, `ingest` extracts the identifier and delegates the fetch to the same adapter `hotato capture` uses ([ADAPTER-STATUS.md](ADAPTER-STATUS.md)) -- the recording URL always comes from that validated fetch, not the raw payload. For **livekit / pipecat**, the recording lands in *your* infra, so the event carries the locator directly: `recording_url` (downloaded) or `recording_path` (read locally). A Pipecat event is whatever you emit: ```json { "recording_path": "captured.wav" } ``` Or skip the payload entirely with a direct id: ```bash hotato ingest --stack vapi --call-id --out candidates.html # + VAPI_API_KEY hotato ingest --stack twilio --recording-sid RE... --format json # + TWILIO_* hotato ingest --stack pipecat --event event.json # local file ``` ## From a candidate to a regression test `ingest` finds the moment; you decide what should have happened and freeze it: ```bash # 1. ingest surfaces a candidate at t=42.18s in a call recording # 2. you listen, decide the agent should have yielded, and promote it: hotato fixture create --stereo call.wav --onset 42.18 \ --expect yield --id refund-cutoff-001 --out tests/hotato # 3. it is now a permanent test: hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio ``` See [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) for the full loop. ## Notes on `--allow-mono` Discovery needs one party per channel to attribute overlap. `--allow-mono` (or `HOTATO_ALLOW_MONO=1`) lets the *fetch* pull a mono-only recording on retell/twilio (matching `hotato capture`), but a mono mix still reports **not-scorable** (exit 2): overlap can't split between caller and agent. Record dual-channel for candidates. ================================================================================ FILE: docs/INVESTIGATE.md ================================================================================ # `hotato investigate`: one recording in, ranked candidate moments out Point it at one recording -- a local dual-channel WAV, or a live pull by call id -- and it authenticates the source, runs the input-health gate, scans for candidate turn-taking moments, and prints the exact command to turn each one into a signed, CI-ready contract. ```bash hotato investigate label --expect yield|hold ``` The one decision that stays with you: which candidate is a bug, and whether the agent should have *yielded* (stopped) or *held* (kept talking through a backchannel, noise, or ack). `"mhm"` and `"stop"` can sound identical by energy, so the label is yours. ## Scan one recording ```bash hotato investigate call.wav # a local WAV hotato investigate --stack vapi --call-id abc123 # or pull live ``` Give it a local `SOURCE` path, or `--stack`/`--call-id`, never both. Stacks: `vapi`, `twilio`, `retell`, `bland`, `elevenlabs`, `synthflow`, `millis`, `cartesia`. LiveKit and Pipecat capture in your own infra: run `hotato setup --stack ` and pass the resulting WAV as `SOURCE`. `--allow-mono` accepts a mono/mixed recording, but a summed channel can't score separated candidates. Flags: `--min-gap` (min. gap in seconds to surface, default `2.0`), `--top` (candidates shown, default `10`, `0` = all), `--caller-channel` / `--agent-channel`, `--state PATH` (default `.hotato/investigate-state.json`). ### What it reports ``` hotato investigate [run 1]: call.wav capture origin: operator-asserted local file (call.wav) input health: verdict path: eligible (a labeled event here can carry a yield/hold verdict) N candidate moment(s) (showing N): [1] t=42.18s label: hotato investigate label .hotato/investigate-state.json#1 --expect yield|hold state remembered at: .hotato/investigate-state.json ``` ## Capture origin: tracked every run Every run tags how the audio arrived: `frozen_regression` (a pinned hotato fixture, replayed exactly), `provider_pulled` (fetched live from the stack's own recording API -- for the stronger, signed claim see [RECAPTURE.md](RECAPTURE.md)), or `operator_asserted_local` (a local WAV, taken at face value). ## The verdict gate Trust runs in contract mode, the same crosstalk/leakage bar `hotato contract create` checks. Two outcomes: - **NOT SCORABLE** (mono, identical channels, a silent required channel): no candidates scanned, the reason named, exit `2`. Fix the input and re-run. - **Verdict path REFUSED** (a suspected channel swap or crosstalk/leakage): candidates still surface; a labeled event carries a verdict once you fix the crosstalk or confirm the mapping with `--confirm-channels`. See [TRUST.md](TRUST.md). Scan runs whenever the input is scorable, so candidates appear even when the verdict path is refused. ## State and candidate refs State persists to `.hotato/investigate-state.json` (run-numbered, atomically written) in the same shape `hotato analyze` / `hotato sweep` use, so it's a valid `FILE#N` candidate ref for `hotato fixture promote` and `hotato contract create --from-candidate`. `#1` is the top-ranked candidate. ## Label a candidate into a contract Your `--expect` goes straight to `hotato contract create --from-candidate`, which mints a signed label-record bound to the exact decoded audio when a signing key is configured. Without one, `label_authority` floors at `asserted`. ```bash hotato investigate label .hotato/investigate-state.json#1 \ --expect yield \ --max-talk-over 0.6 --max-time-to-yield 1.0 \ --reviewer "$USER" --out contracts ``` Writes `contracts/.hotato/`. The id defaults to a slug from the source, onset, and label; `--id` names it, `--force` overwrites. `--pre`/`--post` (defaults `2.0`/`6.0`) set the clip window before/after the onset; `--no-clip` keeps the full recording. If the verdict path was refused, add `--confirm-channels` to carry it through `contract verify`. Source basenames are redacted by default; `--include-identifiers` keeps them. A contract, not a bare fixture, is the point: it carries the trust block, the CI policy, and the exact `hotato contract verify` command -- the handoff into [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) and [CONTRACTS.md](CONTRACTS.md). Scanning, the gate, and labeling all run offline; audio stays on your machine unless you pull it from your own stack ([THREAT-MODEL.md](THREAT-MODEL.md)). ## Exit codes `hotato investigate`: - **`0`** -- candidate-eligible: trust + scan ran, candidates (if any) persisted. A verdict may still be refused; see `verdict_status`. - **`2`** -- usage error (no `SOURCE`/`--stack`+`--call-id`, both given, a bad flag, a missing credential, an unreadable state file), or NOT SCORABLE. `hotato investigate label`: - **`0`** -- a signed, CI-ready contract was written. - **`2`** -- usage error (a bad `--expect`, a bad candidate ref, an unresolved source, an existing contract without `--force`), or not scorable. ## See also - [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) -- the bad-call to CI-gate flow this feeds. - [CONTRACTS.md](CONTRACTS.md) -- the failure-contract bundle. - [TRUST.md](TRUST.md) -- the input-health scorability gate. - [RECAPTURE.md](RECAPTURE.md) -- the stronger, signed fresh-recapture claim. - [CONVERSATION-TEST.md](CONVERSATION-TEST.md), [SUITE-RUN.md](SUITE-RUN.md) -- the conversation-QA path. ================================================================================ FILE: docs/LIFECYCLE.md ================================================================================ # The hotato lifecycle Turn production failures into portable tests, run candidate releases against them, and carry evidence with every release. That is the whole product; this page maps every command onto it. The loop has five steps you drive, and one that feeds itself: ```text OBSERVE -> CATCH -> PIN -> TEST -> PROVE ^ | +---------- production ----------- + ``` Eight nouns carry the loop: a **stack** (the platform your agent runs on), a **recording** (one two-channel call), a **trace** (the call's pipeline events, from your OTel spans), a **candidate** (a moment hotato flagged for judgment), a **label** (your one-word verdict on it), a **contract** (the labeled moment, packaged as a byte-reproducible check), a **suite** (a named set of tests run together), and a **proof** (every evidence lane composed into one fail-closed release verdict). ## 1. Observe Your calls and traces, on your machine. Nothing leaves it. | What | Command | | :-- | :-- | | Store a stack credential once | `hotato connect --stack vapi` | | Pull recent recordings in bulk | `hotato pull --stack vapi --since 7d` | | Fetch and score one call | `hotato capture --stack vapi --call-id ID` | | Capture OTel spans from any process | `hotato observe capture -- ` | | Ingest an OTel export as a voice trace | `hotato trace ingest --otel spans.json` | | Run the durable production evidence store | `hotato production serve` | | Score a webhook's completed call | `hotato ingest --stack vapi --event call.json` | Cost, latency percentiles, and a self-contained dashboard come straight off the captured traces: `hotato observe cost`, `hotato observe percentiles`, `hotato observe report`. ## 2. Catch Deterministic scoring finds what text evals miss: talk-over, dead air, slow yields, a tool the agent claimed it ran. Five dimensions (outcome, policy, conversation, speech, reliability), never one blended score, and every number reproduces byte for byte. | What | Command | | :-- | :-- | | One recording, zero config, the incident list | `hotato autopsy call.wav` | | One recording, ranked candidate moments | `hotato investigate call.wav` | | Every recent call on a stack, one sweep | `hotato sweep --stack vapi --since 7d` | | A folder of recordings, ranked dashboard | `hotato analyze recordings/` | | Is this recording scorable at all | `hotato trust --stereo call.wav` | | Why a failing event failed, by layer | `hotato explain result.json` | | Cluster failures across many runs | `hotato diagnose --fleet runs/` | A transcript works when there is no audio (`hotato investigate --transcript t.json`), and a chat agent is driven directly (`hotato simulate scenario.json --chat URL`). The scorer measures timing and say-do, not intent; a mono or mixed export is refused as NOT SCORABLE, never guessed at. ## 3. Pin You make the one human decision the loop needs: is this candidate a failure the agent must never repeat? Your label turns it into a contract: clipped audio, frame evidence, the policy it was judged under, and a content address, packaged portable. It re-verifies on any machine, with or without hotato's help. | What | Command | | :-- | :-- | | Label a candidate, mint the contract | `hotato investigate label STATE#1 --expect yield` | | Pin a moment from a sweep | `hotato fixture promote SWEEP.json#2 --expect hold` | | File the confirm-or-ignore decision as a GitHub issue | `hotato issue create sweep.json` | | Build a contract directly from a moment | `hotato contract create call.wav --onset 2.0 --expect yield` | | Review and label at fleet scale | `hotato fleet review` / `hotato fleet label` | ## 4. Test Run candidates against everything you have pinned, plus scripted callers, personas, robustness batteries, and the bundled stress suite. Simulation is deterministic: the same scenario and seed render the same bytes on every run. | What | Command | | :-- | :-- | | Render a scenario into a scored conversation | `hotato simulate scenario.json --out ./sim` | | Drive a live call at a candidate agent | `hotato drive contracts/ID.hotato --stack vapi` | | Stress one recording across noise and loss | `hotato battery robustness call.wav` | | The bundled turn-taking stress suite | `hotato gauntlet` | | A suite of conversation tests, per-dimension | `hotato suite run suite.json` | | Before/after across a whole battery | `hotato verify --before old/ --after new/` | ## 5. Prove One command composes every evidence lane you have into one fail-closed, content-addressed proof: contracts re-verified, suites re-run, before/after movement measured, the stress suite cleared. The proof headlines its claim scope, exactly what the evidence establishes: contracts alone re-measure stored evidence (Captured Evidence), a suite or the stress suite establishes a Test Suite ran, and a before/after run reaches Candidate Revision only when you bind the candidate identity (`--candidate-config-hash`, `--provider`). CI gates on the exit code, and the receipt stays verifiable anywhere. ```bash hotato prove --contracts contracts/ --before before/ --after after/ --out .hotato/proofs/v42/ ``` `pass` means every lane passed. Any failure fails the proof; a lane that could not support its claim is `inconclusive`, and CI never reads "could not tell" as green. The Candidate Revision binding is measured, not asserted. `hotato candidate hash --provider vapi --assistant ` fetches the candidate's configuration, canonicalizes it (dropping volatile fields like timestamps and ids), and prints its content hash. Re-run `hotato candidate verify --provider vapi --assistant --expect ` after the before/after calls: it refuses (exit 1) if the configuration drifted mid-run, so a Candidate Revision proof cannot survive a swapped candidate. The end to end flow: ```bash hotato candidate hash --provider vapi --assistant asst_123 # -> sha256:... hotato drive contracts/ID.hotato --stack vapi --assistant asst_123 --out after/ hotato candidate verify --provider vapi --assistant asst_123 --expect sha256:... # exit 1 = drifted, void hotato prove --before before/ --after after/ --candidate-config-hash sha256:... --provider vapi ``` The deeper machinery is there when you need it: `hotato fix trial` binds a specific patch to its before/after evidence, `hotato apply --clone` stages a candidate without touching production, and `hotato record render` turns any failure into a share-safe card. ## ...and production feeds the next loop The deploy is not the end of the loop; it is the next input. The production evidence plane watches completed sessions, raises alerts on evidence gaps, and exports any session as an offline-verifiable regression candidate, which lands back in step 2. | What | Command | | :-- | :-- | | Continuous evidence intake | `hotato production serve` / `production ingest` | | Alert transitions on session evidence | `hotato production alerts` | | A session back into the loop as a test | `hotato production export-regression SESSION` | | Canary a config change, roll it back | `hotato fleet canary start` / `rollback` | | Team trends over releases | `hotato team` / `hotato release compare` | ## Where the other commands sit `hotato start`, `demo`, `doctor`, and `init --auto` are onboarding: they walk the loop on bundled calls before you spend a minute of your own. `hotato bench` and `hotato gauntlet badge` prove the scorer itself. `hotato describe` emits the machine-readable manifest a coding agent drives the loop with, and the MCP server (`hotato-mcp`) exposes the same loop over stdio. ================================================================================ FILE: docs/LIVEKIT-CALLER-SESSION.md ================================================================================ # Direct LiveKit caller session `hotato.livekit_session.LiveKitCallerSession` is an optional transport for the bounded scripted, hybrid, generative, and frozen-replay caller engine. It joins a LiveKit room as a participant, publishes PCM16LE through one audio track, receives the target participant's audio and transcription events, and submits SIP DTMF through the official Python RTC SDK. This adapter removes the custom WebSocket sidecar requirement for a LiveKit agent. It does not implement SIP, a carrier, STT, TTS, packet impairment, or a telephony transfer service. A PSTN test still needs LiveKit SIP and a configured trunk. LiveKit documents `publish_dtmf` as publishing a SIP DTMF message; a successful SDK call is not evidence that a downstream carrier honored it. Official transport references: - [LiveKit Python SDK](https://github.com/livekit/python-sdks) - [Python RTC API](https://docs.livekit.io/reference/python/livekit/rtc/index.html) - [AudioSource queue and playout contract](https://docs.livekit.io/reference/python/livekit/rtc/audio_source.html) - [AudioStream bounded-capacity option](https://docs.livekit.io/reference/python/livekit/rtc/audio_stream.html) ## Install and run ```bash pip install 'hotato[livekit]' ``` Mint a short-lived room token with only the permissions needed by the caller participant. Keep token generation outside the test plan and result package. The CLI can join the room directly and synthesize caller turns with a local Piper model. The token enters through an environment variable or regular file, never a command-line value: ```bash export LIVEKIT_CALLER_TOKEN="$(mint-short-lived-room-token)" hotato caller run scenarios/refund-barge-in.json \ --livekit-url wss://livekit.example.test \ --livekit-target-identity agent-under-test \ --livekit-token-env LIVEKIT_CALLER_TOKEN \ --livekit-evidence-topic hotato.evidence.v1 \ --piper-model models/en_US-lessac-medium.onnx \ --piper-config models/en_US-lessac-medium.onnx.json \ --allow-remote \ --out artifacts/refund-barge-in ``` `--livekit-token-file` is the file-backed alternative. `--target-ws` remains the mutually exclusive sidecar transport. The CLI derives the LiveKit audio rate from `audio.sample_rate` in the Piper config; an explicit `--livekit-sample-rate` must match it. ```python import os from hotato.caller import load_plan, run_caller from hotato.livekit_session import LiveKitCallerSession # Your TTS adapter must return Hotato's documented PCM + provenance mapping. from my_test_adapters import LocalTTS session = LiveKitCallerSession( os.environ["LIVEKIT_URL"], os.environ["LIVEKIT_CALLER_TOKEN"], target_identity="agent-under-test", allow_remote=True, # explicit network-egress decision sample_rate_hz=48_000, # TTS output must match exactly evidence_topic="hotato.evidence.v1", ) try: run = run_caller( load_plan("scenarios/refund-barge-in.json"), session, "artifacts/refund-barge-in", tts=LocalTTS(sample_rate_hz=48_000), ) print(run.result["status"], session.media_summary()) finally: session.close() ``` `send_text` is `UNSUPPORTED`: a LiveKit data message is not spoken caller audio. Supply a TTS adapter so caller `say` and model-generated `say` actions produce PCM. Sample-rate conversion is also explicit; the transport refuses a TTS result whose sample rate differs from the configured `AudioSource`. ## Evidence semantics The adapter records each layer without promoting one into another: | Event | What it establishes | Authority | |---|---|---| | `audio_submitted` | `AudioSource.capture_frame` accepted every frame and `wait_for_playout` completed | local LiveKit SDK | | `received_audio_frame` | decoded PCM reached the Hotato caller participant | local LiveKit receiver | | `delivered_audio_receipt` | a cooperating target participant reported the bytes at its named boundary | target participant reported | | `dtmf_submitted` | the local participant's `publish_dtmf` calls completed | local LiveKit SDK | | native transcription | LiveKit emitted a transcription event for the target identity | LiveKit room event | Every received audio frame includes its SHA-256 and a rolling SHA-256 over the ordered PCM stream. `media_summary()` returns only digests and counters. Raw PCM remains in the caller engine's content-addressed package when the scenario and TTS lane write it there. `evidence()` binds that digest-only media summary and the five evidence capability states into `caller-result.json.session_boundary`. It contains no room token, endpoint query, raw audio, transcript text, or target payload. `sdk_playout_complete` does not establish target, SIP, PSTN, or carrier delivery. `evidence_capabilities()["outgoing_audio_delivery"]` remains `UNOBSERVABLE` until a valid, session-bound target receipt arrives. ## Optional target evidence channel Set `evidence_topic` only when the target participant implements this strict envelope on reliable LiveKit data messages: ```json { "schema": "hotato.livekit-evidence.v1", "session_id": "", "sequence": 1, "kind": "audio_receipt", "payload": { "submission_sequence": 1, "submitted_sha256": "sha256:", "delivered_sha256": "sha256:", "delivered_bytes": 19200, "sample_rate_hz": 48000, "channels": 1, "boundary": "agent-input-after-decode" } } ``` The default control topic is `hotato.control.v1`. The adapter sends `session_started` and `audio_submission` messages only to `target_identity`. The target copies the session id and announced submission digest into its receipt, then supplies its separately measured delivered digest. Receipt sequence must be contiguous. Duplicate, reordered, wrong-session, malformed, oversized, and wrong-submission receipts become `target_evidence_rejected` events; their payload is represented by a digest, not copied into the result. The same evidence envelope accepts `transcript`, `tool_result`, `state_snapshot`, `transfer`, `hold`, `timing`, and `custom`. These remain `target_participant_reported`. They can supply evidence to Hotato's later deterministic assertions; this transport never produces a pass/fail verdict. ## Resource and failure behavior - Remote egress is refused unless `allow_remote=True`; loopback remains the default. - The URL refuses embedded credentials, query strings, and fragments. - Before publishing the caller track, the driver applies LiveKit track subscription permissions that allow only `target_identity`. Control and evidence data messages are also addressed only to that identity. - The short-lived token is passed to the SDK during connection and never copied into the facade, control messages, event queue, `repr`, or media summary. The SDK necessarily retains connection credentials while its room is connected. - PCM per send, PCM per session, per-send duration, event count, remote track count, control message size, frame duration, connection time, operation time, and close time are bounded. - Caller media operations are serialized. A concurrent audio/DTMF call is refused instead of interleaving frames from two scenario actions. - The SDK's remote `AudioStream` uses a nonzero capacity. The SDK's behavior at that internal capacity boundary remains transport behavior to measure. The separate Hotato event queue is bounded and its overflow fails the run instead of silently dropping evidence. - Driver errors expose the operation and exception type without copying SDK messages that may contain endpoint details. - The adapter has no hidden retry. Load scheduling and retry policy belong to the load controller, where attempts remain observable. ## Acceptance gate before a transport claim The unit suite validates protocol mapping against a fake SDK. Publish a LiveKit transport result only after an environment-specific run captures: 1. pinned LiveKit server, SIP server, Python SDK, agent, STT, TTS, and carrier versions; 2. the caller package and rolling received-stream digest; 3. a target-boundary receipt, if target delivery is claimed; 4. SIP participant state and carrier evidence for PSTN claims; 5. network capture and egress destinations; 6. disconnect, reconnect, event-overflow, token-expiry, and media-timeout cases. Until that acceptance run exists, describe this as an implemented optional LiveKit caller transport with fake-SDK contract tests, not as a measured carrier-grade path. ================================================================================ FILE: docs/LOAD-AND-RECOVERY.md ================================================================================ # Load and recovery with portable evidence Hotato schedules calls in either a closed concurrency model or an open arrival model. It preserves one verifiable child package per started call. Provider completion, delivered-media evidence, tool/state evidence, scheduling loss, and recovery remain separate measurements. ```json { "schema": "hotato.load-plan.v2", "id": "refund-release", "call": { "schema": "hotato.telephony-call.v1", "id": "base", "provider": "vapi", "to": "+15551234567", "agent_id": "agent-id", "phone_number_id": "phone-id" }, "stages": [ {"name": "warm", "phase": "warmup", "model": "closed", "concurrency": 2, "calls": 10}, {"name": "spike", "phase": "spike", "model": "open", "arrival_rate_per_second": 4, "duration_seconds": 30, "max_in_flight": 40}, {"name": "recover", "phase": "recovery", "model": "closed", "concurrency": 4, "calls": 20} ], "safety": { "max_calls": 150, "estimated_cost_per_call_usd": 0.05, "max_estimated_cost_usd": 7.50, "allowed_destinations": ["+1555"], "stop_file": ".hotato/STOP" }, "slos": { "max_dropped_start_rate": 0.01, "max_p95_scheduling_delay_seconds": 0.25, "min_lifecycle_completion_rate": 0.99, "min_evidence_complete_rate": 0.99 } } ``` An open stage does not queue work when its generator is saturated. It records a `generator_saturated` dropped start so coordinated omission remains visible. Closed stages keep a fixed number of calls in flight and start a replacement only after one finishes. The safety block is evaluated before a provider call. Twilio, Vapi, and Retell workloads require a v2 plan with a positive per-call estimate, a positive total budget, and a non-empty destination allowlist. Omitting any one refuses the workload before the first provider request. Destination prefixes, a hard call ceiling, and an operator stop file bound execution. Hotato performs zero automatic create retries because the provider-neutral interface has no shared idempotency guarantee. Each `calls/*/manifest.json` binds redacted lifecycle receipts and any evidence export to the provider, provider-namespaced call pseudonym, normalized call digest, and normalized workload digest. The four evidence states are `PRESENT`, `MISSING`, `UNSUPPORTED`, and `UNOBSERVABLE`. `PRESENT` requires a content digest and an explicit authority for every lane; a bare string cannot satisfy an evidence SLO. `min_evidence_complete_rate` is stricter than artifact presence: all three non-lifecycle lanes must be `PRESENT` and carry either `measured` or `signed_attestation` authority, with `eligible_for_execution_claim=true`. Provider-, sidecar-, target-, and unverified reports remain portable evidence but cannot satisfy that execution evidence SLO. A provider-reported completed call can satisfy lifecycle completion while evidence completeness remains zero. `hotato load telephony verify DIR` hashes every child and recomputes per-stage and aggregate metrics, recovery windows, SLO rows, the HTML report, overall status, and exit code from `observations.jsonl` plus the bound, non-secret `verification-plan.json`. It rejects missing/duplicate observations, swapped child references, extra artifacts, invalid evidence states, and a summary that has merely been rehashed after its verdict changed. No network or model is required. The package hashes establish internal identity and consistency. They do not authenticate who produced a directory: a party able to replace every artifact can construct a different, internally consistent package. Preserve the package in an access-controlled artifact store or attach an external organization signature before using publisher identity as a trust decision. Fault schedules require a call controller that exposes the named injection. Unsupported injection produces an explicit call error; it is never simulated silently. Recovery is measured from the end of the declared fault window to the first later successful lifecycle observation and stays separate from quality. ================================================================================ FILE: docs/OBSERVE.md ================================================================================ # observe: LLM/voice observability, derived on your machine `hotato observe` reads the observability an LLM or voice agent already emits as OpenTelemetry spans, on your machine, from files you already have. One command group, four subcommands, all built on the same `hotato.voice_trace.v1` spans [`docs/TRACE.md`](TRACE.md) / [`docs/OTEL.md`](OTEL.md) describe: ```bash hotato observe capture --out voice_trace.jsonl -- python agent.py hotato observe cost voice_trace.jsonl --prices starter hotato observe percentiles traces/ --html latency.html hotato observe report traces/ --out observe.html --prices starter ``` Every number is derived locally: token counts read from your spans, latency from their timestamps, USD from a price table you keep. hotato opens no socket, runs no listener, and makes no network call of its own. ## capture: a local file sink, no account `hotato observe capture -- ` runs the child process with a local file sink wired through its environment, then ingests whatever the child wrote: - `HOTATO_OTEL_FILE` -- the file path hotato names for the spans. - `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` -- a `file://` URL pointing at that same path (not a host:port, so there is nothing for an exporter to dial out to). - `OTEL_EXPORTER_OTLP_PROTOCOL=http/json` -- the plain-text OTLP encoding. A cooperating process writes its spans to that file: either as a standard OTel JSON export, or as hotato's own OTel bridge JSONL written directly to `$HOTATO_OTEL_FILE` (the shape in [`docs/OTEL.md`](OTEL.md)). On the child's exit, hotato ingests the file through the same `trace ingest` path into `--out` and prints a one-screen summary: span count, per-hop latency, and token totals. If the child wrote no spans, capture refuses (exit 2) and leaves nothing behind. The child's own network is the child's business; hotato adds none. ```bash hotato observe capture --out run.jsonl -- python agent.py # observe capture: 7 spans -> run.jsonl # child exit: 0 # per-hop latency: # STT (speech to text) : 550.0 ms # LLM (first token) : 250.0 ms # Tool call : 320.0 ms # tokens: # input 1200 output 340 cached not captured reasoning not captured ``` ## cost: tokens are facts, USD is a local estimate `hotato observe cost ` sums each span's LLM token usage per model and in total. Token attributes resolve by first-match alias: | category | attributes (first match wins) | |-----------|---------------------------------------------------------------------| | input | `gen_ai.usage.input_tokens`, `gen_ai.usage.prompt_tokens`, `input_tokens`, `prompt_tokens` | | output | `gen_ai.usage.output_tokens`, `gen_ai.usage.completion_tokens`, `output_tokens`, `completion_tokens` | | cached | `gen_ai.usage.cached_tokens`, `gen_ai.usage.cache_read_input_tokens`, `cached_tokens` | | reasoning | `gen_ai.usage.reasoning_tokens`, `reasoning_tokens` | The model is `gen_ai.response.model` (else `gen_ai.request.model`, `gen_ai.model`, `llm.model_name`, `model`). Tokens are facts read from the spans. A category no span reported reads **not captured** (null, plus a count of the spans that lacked it), never 0. `--prices FILE` adds an estimated USD cost from a **local** per-model $/1M table (`--prices starter` reads the bundled `src/hotato/data/prices.yaml`). Copy that file, set the rates from your own agreements, and pass it. USD is labeled "estimated from ``": the tokens are measured, the dollars are your arithmetic over your table. A model with no row in the table is **unpriced** (its cost reads null), never a guessed rate; a category with no rate in a row is simply not billed. ```yaml # your-rates.yaml (USD per 1,000,000 tokens) table: my-contract per_tokens: 1000000 models: gpt-4o: input: 2.5 output: 10.0 cached: 1.25 ``` ## percentiles: nearest-rank over a folder `hotato observe percentiles DIR` reports p50 / p90 / p99 of each per-hop latency (STT, LLM, tool, TTS, transport) and of end-to-end latency over every readable voice trace in `DIR`, by the nearest-rank method (`hotato._stats.nearest_rank`): with the measured values sorted ascending and n of them, `rank = ceil(q * n)` and the percentile is the value at that rank. Every percentile is therefore an observed measurement, re-derivable by hand. A trace that did not capture a hop is **excluded** from that hop's percentiles, and the excluded count is shown, so an uncaptured hop is never counted as 0. `--format json` and `--html PATH` render the same numbers as a machine envelope and a self-contained panel. ## report: one self-contained HTML page `hotato observe report DIR --out observe.html` writes one self-contained HTML page (inline CSS and SVG, no external request) summarizing trace and span counts, per-hop latency and its percentiles, per-model token totals and an estimated USD line (with `--prices`), and links to the slowest traces. It opens offline and embeds every value it shows. ## Determinism The same inputs render byte-identical text, JSON, and HTML. No artifact embeds a wall clock. `hotato observe` reads files and writes files; it holds no state and reaches nowhere. ================================================================================ FILE: docs/OTEL.md ================================================================================ # OTel ingest: two source shapes `hotato trace ingest --otel FILE` turns an OTel trace into hotato's own `hotato.voice_trace.v1` spans -- reading a file you already have (an exported trace, or a script's own event log) offline, once, and translating `name`, `startTimeUnixNano`/`endTimeUnixNano`, `attributes`, and span `events` into hotato's span shape. It recognizes two input shapes; `source.format` records which one it used (`"otel-json"` or `"otel-jsonl-bridge"`). ## 1. Standard OTel JSON export (`otel-json`) A single JSON document with a top-level `resourceSpans` array (the shape a standard OTel exporter/collector writes): ```json { "resourceSpans": [{ "resource": {"attributes": [ {"key": "service.name", "value": {"stringValue": "vapi"}}, {"key": "git_sha", "value": {"stringValue": "cafebabe"}} ]}, "scopeSpans": [{"spans": [ { "name": "agent_audio_active", "startTimeUnixNano": "1000000000", "endTimeUnixNano": "4400000000" }, { "name": "tts.cancel_requested", "startTimeUnixNano": "2600000000", "events": [] } ]}] }] } ``` - Resource attributes flatten into a plain dict: `service.name` (or a custom `stack` attribute) becomes `deployment.stack`; `git_sha` / `config_hash`, when present, fill the matching deployment fields. Only the FIRST `resourceSpans` entry supplies deployment metadata -- every entry's spans are still walked. - Timestamps convert to seconds relative to the EARLIEST timestamp in the file, matching the audio-relative-seconds convention every other hotato timestamp uses. - A span's own `name` maps to a hotato span `type` via a small documented table (`caller_audio_active`, `agent_audio_active`, `tts.cancel_requested` -> `tts_cancel_requested`, `tts.audio_stopped` -> `tts_audio_stopped`, `asr.partial` -> `asr_partial`, `llm.first_token` -> `llm_first_token`); an unmapped name passes through unchanged. - Span `events` -- OTel's point-in-time markers nested inside a span, e.g. a `tts.cancel_requested` marker inside a broader `tts_playback` span -- flatten into their own point events the same way top-level spans do. - A `tool_call`-mapped span takes its `name` field from the `tool.name` / `gen_ai.tool.name` attribute; a `latency_ms` attribute is used when present, otherwise computed from the span's own start/end. - An `asr_partial`-mapped span's `text` / `asr.transcript.partial` attribute is captured and redacted by default (`text_redacted: true`); pass `--include-text` to keep it in the output. ## 2. Hotato's OTel bridge JSONL (`otel-jsonl-bridge`) The shape for a script or test fixture that skips a full OTel exporter: one JSON object per line (or one bare JSON array of them). Each line is a span or a meta/resource line. A span line uses `type` directly; `name` is reserved for `tool_call`'s own tool name, distinct from the span kind: ``` {"type": "caller_audio_active", "start_sec": 2.40, "end_sec": 4.10} {"type": "agent_audio_active", "start_sec": 0.00, "end_sec": 2.90} {"type": "tts_cancel_requested", "time_sec": 2.60} {"type": "tts_audio_stopped", "time_sec": 2.90} {"type": "asr_partial", "start_sec": 2.40, "end_sec": 2.95, "text": "wait, I need a refund"} {"type": "tool_call", "start_sec": 1.10, "end_sec": 1.42, "name": "lookup_order", "latency_ms": 320} ``` An optional meta/resource line (no `type` key) attaches deployment metadata and a call id: ``` {"call_id": "demo-call-001", "deployment": {"stack": "vapi", "agent_id": "agent-demo-1", "git_sha": "deadbeefcafe", "config_hash": "sha256:0f1e2d3c"}} ``` `--call-id` / `--stack` / `--agent-id` / `--git-sha` / `--config-hash` on `trace ingest` override whatever a meta line (or a standard export's resource) provided. `hotato trace export` writes exactly this bridge shape, so `ingest -> attach -> export -> ingest` round-trips the same spans; the shipped test fixture (`tests/data/otel/demo-trace.otel.jsonl`) uses it. ## Which one to use Already running an OTel collector or SDK? Point `--otel` at that trace export directly -- the standard-export path reads it, best-effort. Wiring a quick script (a webhook handler, a log-line scraper) outside a full OTel pipeline? Write the bridge JSONL directly instead -- same information, already flat. ================================================================================ FILE: docs/PIPER-CALLER-TTS.md ================================================================================ # Local Piper caller speech `hotato.piper_tts.PiperCallerTTS` turns caller `say` actions into mono PCM16LE with a local Piper executable. It is an optional adapter around the Piper CLI, not a bundled speech model and not an audio-quality claim. ## CLI Supply both the ONNX model and its matching config: ```bash hotato caller run scenarios/appointment.json \ --target-ws ws://127.0.0.1:8765/caller \ --piper-model models/en_US-lessac-medium.onnx \ --piper-config models/en_US-lessac-medium.onnx.json \ --out artifacts/appointment ``` The same flags provide spoken input for the direct LiveKit caller session. The config must declare `audio.sample_rate`. Hotato publishes that rate without an implicit resample; an incompatible transport rate is refused. Useful bounds are explicit: ```text --piper-command PATH default: piper --piper-timeout SEC default: 60 --piper-max-output-bytes BYTES default: 33554432 --piper-voice LABEL default: default ``` The voice label is descriptive provenance. It does not select a Piper speaker; use a single-speaker model or configure speaker selection in a separately reviewed adapter before claiming multi-speaker coverage. ## Execution boundary For each CLI run, the adapter: 1. refuses model/config symlinks, FIFOs, devices, oversized files, and files changed while they are staged; 2. copies the exact model and config bytes once into a mode-`0600` private temporary directory; 3. invokes the resolved Piper executable with an argument vector and `shell=False`; 4. supplies a minimal environment instead of forwarding operator secrets; 5. bounds input text, PCM stdout, diagnostic stderr, execution time, staged model/config bytes, and executable bytes; 6. accepts only non-empty, even-length `--output_raw` bytes; and 7. deletes the private staging directory when the caller run closes. No diagnostic text enters the package. Successful synthesis records the staged model/config SHA-256 values, the resolved executable SHA-256 observed before and after execution, sample rate, mono PCM16LE encoding, diagnostic byte count, and diagnostic digest. Model/config paths, the private temporary path, environment values, and Piper stderr contents are excluded. The resulting PCM is content-addressed again by the caller engine before it is submitted to the transport. Piper success establishes local synthesis only. `sdk_playout_complete`, target delivery, SIP delivery, and carrier delivery are separate evidence boundaries. ## Acceptance gate The hermetic suite exercises argument-vector invocation, bounded output, timeout, symlink refusal, private-environment behavior, PCM structure, cleanup, and provenance. Before publishing a voice-quality or performance result, pin the Piper release, executable digest, model license and digest, config digest, hardware, input corpus, and the metric and adjudication method used. ================================================================================ FILE: docs/PRODUCTION-MONITORING.md ================================================================================ # Production monitoring: durable evidence into a regression candidate Hotato's production plane accepts bounded call events, preserves ambiguity, and turns a finalized session into an offline-verifiable regression candidate. It is a single-process, self-hosted evidence assembler backed by SQLite WAL. Put a durable OpenTelemetry Collector or message queue in front when a deployment needs ingress buffering across process restarts. The included [`deploy/control-plane`](../deploy/control-plane/README.md) bundle does this with Collector Contrib and a local persistent queue. That queue survives a process restart; it does not replicate data across hosts. The implementation lives in `hotato.production`. It has no runtime dependency beyond Python's standard library. The gateway accepts OTLP/HTTP JSON traces at the standard `/v1/traces` path and at `/v1/otlp/traces`. Protobuf input is refused rather than guessed. Authentication is required on this boundary. The deployed Collector accepts standard OTLP/gRPC and OTLP/HTTP protobuf or JSON on loopback, converts trace batches to JSON, and authenticates the private hop to this gateway. After its database transaction commits, `/v1/traces` returns the standard empty `ExportTraceServiceResponse` JSON object (`{}`). The Hotato-specific `/v1/otlp/traces` compatibility path returns the richer per-event receipt. ## The write contract The gateway follows this order for every request: 1. bound `Content-Length` before reading; 2. authenticate the exact body bytes with bearer token or HMAC; 3. parse and validate the event; 4. enter an SQLite `BEGIN IMMEDIATE` transaction; 5. persist the event, session update, and audit-chain entry; 6. commit with `PRAGMA synchronous=FULL`; 7. return a receipt containing `"durability": "committed"`. A storage error returns `503`. A validated event never receives a success ACK before its transaction commits. The HTTP server admits a bounded number of workers and returns `503` with `Retry-After: 1` when capacity is full. There is no hidden in-memory queue. ## Start the gateway For an operated service, declare the maintenance loop as data instead of depending on an operator to remember three separate commands: ```json { "schema": "hotato.production-maintenance.v1", "interval_seconds": 30, "quiescence_seconds": 30, "required_lanes": [ "participant_audio", "transcript", "model_trace", "tool_calls", "backend_state" ], "alert_rules": [ {"id": "degraded-session", "condition": "degraded"}, {"id": "event-conflict", "condition": "conflict"}, {"id": "missing-audio", "condition": "missing_audio"} ], "retention_seconds": 2592000 } ``` Run one inspectable cycle: ```bash hotato production maintain maintenance.json \ --db .hotato/production.sqlite3 --format json ``` Or run the authenticated gateway and maintenance loop together while keeping the bearer value out of argv and the container environment: ```bash hotato production serve \ --db .hotato/production.sqlite3 \ --token-file /run/secrets/hotato_production_token \ --maintenance-policy maintenance.json ``` Each cycle finalizes eligible sessions first, evaluates persisted alert state second, and enforces retention last. A cycle failure is retained in supervisor status and retried at the next interval; it never receives a success ACK from the ingest gateway because ingest durability is a separate transaction. The supervisor does not create assertions, score calls, export regression candidates, or send notifications. The lower-level Python gateway remains available: ```python from hotato.production import ProductionGateway, ProductionStore store = ProductionStore(".hotato/production.sqlite") gateway = ProductionGateway( store, token="replace-with-at-least-16-characters", hmac_secret="replace-with-at-least-32-characters", host="127.0.0.1", port=8099, max_workers=16, ) try: gateway.thread.join() finally: gateway.close() store.close() ``` Binding to a non-loopback address exposes an authenticated HTTP service. Place TLS termination in front of it and rotate credentials through the surrounding secret manager. ## Event envelope Every event conforms to [`production-event.v1.json`](../src/hotato/schema/production-event.v1.json). The envelope is CloudEvents-shaped and adds required execution authority: ```json { "specversion": "1.0", "id": "tool-result-00017", "source": "livekit-sidecar", "type": "tool.result", "subject": "call-01J2Q1N4WQ", "time": "2026-07-17T12:00:00.123Z", "sequence": 17, "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", "data": { "availability": "available", "tool": "cancel_subscription", "result": {"status": "failed"} }, "authority": { "kind": "adapter_reported", "eligible_for_execution_claim": false } } ``` Supported event types are enumerated in the schema. Unknown types and unknown top-level fields are refused. Each event is capped at 8 MiB. ### Availability and authority remain separate Evidence availability is one of: - `available`: the event carries or references evidence; - `unavailable`: the producer expected the evidence and could not obtain it; - `unsupported`: the producer cannot provide that evidence lane; - `missing`: no event has yet described the lane. Authority is one of: - `measured`; - `signed_attestation`; - `provider_export`; - `adapter_reported`; - `submitted`. Only `measured` and `signed_attestation` may set `eligible_for_execution_claim: true`. Request authentication establishes who submitted the envelope. It never upgrades an adapter or provider export into an independent execution measurement. The session manifest reports five independent evidence lanes: - participant audio; - transcript; - model trace; - tool calls; - backend state. No lane substitutes for another. The manifest has no blended score. ## Authentication ### Bearer ```bash curl -sS http://127.0.0.1:8099/v1/events \ -H 'Authorization: Bearer replace-with-at-least-16-characters' \ -H 'Content-Type: application/json' \ --data-binary @event.json ``` ### HMAC HMAC signs the exact bytes sent in the request: ```text hex(HMAC-SHA256(secret, decimal_unix_timestamp + "." + raw_body)) ``` Send the result as `X-Hotato-Signature: v1=` and the timestamp as `X-Hotato-Timestamp`. The default acceptance window is 300 seconds. Event identity deduplication makes an accepted replay idempotent. ```python import hashlib, hmac, json, time, urllib.request secret = b"replace-with-at-least-32-characters" raw = json.dumps(event, separators=(",", ":")).encode() timestamp = str(int(time.time())) signature = hmac.new(secret, timestamp.encode() + b"." + raw, hashlib.sha256).hexdigest() request = urllib.request.Request( "http://127.0.0.1:8099/v1/events", data=raw, headers={ "Content-Type": "application/json", "X-Hotato-Timestamp": timestamp, "X-Hotato-Signature": "v1=" + signature, }, ) urllib.request.urlopen(request).read() ``` ## Duplicate, conflict, and ordering behavior Event identity is `(source, id)`. | Arrival | Stored result | Session effect | |---|---|---| | unseen identity | `stored` | event count and evidence update | | same identity, same canonical bytes | `duplicate` | duplicate counter increments | | same identity, different canonical bytes | HTTP `409` | original event remains; conflict is recorded; session becomes `DEGRADED` | | sequence lower than or equal to the source cursor | `out_of_order` | event remains stored; out-of-order counter increments | | no sequence | `stored` | unsequenced counter increments; finalization is `DEGRADED` | Sequence cursors are scoped to `(subject, source)`, so independent producers may each begin at sequence zero. Arrival order remains in the event export. Hotato does not discard or silently reorder late observations. An event arriving after session finalization remains stored and changes the manifest to `DEGRADED` with `late_event_after_finalization`. The previously exported candidate remains content-addressed and unchanged. ## Finalization `session.ended` moves a session to `QUIESCENT`. Call `finalize()` after the declared quiescence period: ```python manifests = store.finalize(quiescence_seconds=30) ``` By default, a session becomes `COMPLETE` only when exactly one `session.started` and one `session.ended` event were observed, all five evidence lanes are available, no identity conflict occurred, and no out-of-order event was observed. Every event must also carry a sequence from its source. Lifecycle and unsequenced counts remain explicit in the manifest. A fixed workflow that does not produce every evidence lane can declare its required subset at finalization: ```python manifests = store.finalize( quiescence_seconds=30, required_lanes=("participant_audio", "transcript", "backend_state"), ) ``` The selected list is persisted in `required_evidence_lanes`; it cannot disappear from the exported manifest. Every other finalized session is `DEGRADED`; the manifest identifies each unavailable or missing lane and the finalization reason. Finalization measures evidence completeness. It does not decide whether the agent passed a product assertion. ## OpenTelemetry trace bridge `POST /v1/traces` and `POST /v1/otlp/traces` accept bounded OTLP/HTTP JSON with the standard `resourceSpans -> scopeSpans -> spans` shape. Use `X-Hotato-Source` to name the sidecar. The former returns `{}` for OTLP client compatibility after commit; the latter returns Hotato's normalized result and durability fields. Each span must carry one of these attributes: - `hotato.session_id`; - `session.id`; - `conversation.id`; - `call.id`. Optional attributes: - `hotato.event_type`: one supported production event type; - `hotato.sequence`: integer ordering within that source; - `hotato.evidence.availability`: `available`, `unavailable`, or `unsupported`. The bridge preserves trace ID, span ID, parent span ID, resource attributes, scope, status, and nanosecond timestamps. It assigns `adapter_reported` authority. A caller may use `normalize_otlp_json()` directly and select a different authority, subject to the same structural validation. The OTLP endpoint validates the complete span set before opening its transaction and commits the normalized batch atomically. Identity conflicts are returned as per-event results while preserving the original event bytes. ### Included Collector boundary The control-plane deployment renders these local-only inputs: ```text OTLP/gRPC 127.0.0.1:4317 OTLP/HTTP http://127.0.0.1:4318 ``` The Collector uses the stable `otlp_http` exporter name, JSON encoding, and no compression for its private hop to `127.0.0.1:8432/v1/traces`. The Hotato bearer credential exists only in the OTel-specific mode-`0400` config volume. It is absent from the Collector environment, process arguments, and bootstrap manifest. Exporter requests are staged in an fsyncing `file_storage` queue before delivery. The fixed queue has a capacity of 10,000 requests and retries an enqueued request without an elapsed-time cutoff. This is bounded, single-host buffering: - queue units are exporter requests, not spans, calls, or bytes; - the storage volume has no application-level byte quota; - storage compacts at startup and after its allocated database exceeds 100 MiB and later drains below 10 MiB; compaction itself needs temporary disk headroom; - when the queue is full, the disk is full, or storage returns an I/O error, enqueue can fail and telemetry can be dropped; - Collector acceptance is not a Hotato SQLite commit receipt; - deleting or losing the host volume loses any request that has not drained. The Collector exposes its own metrics only on `http://127.0.0.1:8888/metrics`. Monitor `otelcol_exporter_queue_size`, `otelcol_exporter_queue_capacity`, `otelcol_exporter_enqueue_failed_spans`, `otelcol_exporter_send_failed_spans`, receiver-refused spans, process restarts, and disk space. Treat any nonzero enqueue-failure delta as an evidence-loss incident. Run restart, gateway-outage, queue-capacity, and disk-exhaustion drills against the exact pinned image digest before qualifying a deployment. The rendered settings follow the pinned Collector `v0.153.0` contracts for the [OTLP HTTP exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.153.0/exporter/otlphttpexporter/README.md), [exporter helper persistent queue](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.153.0/exporter/exporterhelper/README.md), and Contrib [file storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.153.0/extension/storage/filestorage/README.md). This endpoint is a trace normalizer. The included Collector pipeline also wires traces only; application metrics and logs require a separately declared pipeline and backend. The Prometheus endpoint above contains the Collector's own operational metrics, not application telemetry. ## Alerts The local alert engine persists each transition and an audit-chain entry: ```python changes = store.evaluate_alerts([ {"id": "evidence-complete", "condition": "incomplete_evidence"}, {"id": "event-conflict", "condition": "conflict"}, ]) ``` Supported conditions: - `degraded`; - `missing_audio`; - `missing_tool_evidence`; - `incomplete_evidence`; - `conflict`; - `out_of_order`; - `unsequenced`. Transitions use `FIRING` and `RESOLVED`. A resolved alert that fires again gets a new generation and opened timestamp. Stable alert states produce no duplicate transition. ## Prometheus `GET /metrics` requires bearer authentication. Event, duplicate, conflict, and ordering-anomaly counters carry no labels. Session `status` and alert `state` use fixed enumerations. Session IDs, rule IDs, providers, phone numbers, and trace IDs never become labels, preventing unbounded series cardinality. Counters live in a dedicated SQLite table and remain monotonic when retention deletes event/session rows. ## Promote a production failure Finalize the session, then export it: ```python result = store.export_regression_candidate( "call-01J2Q1N4WQ", "fixtures/candidates/call-01J2Q1N4WQ", ) ``` The exporter writes to an owned staging directory, fsyncs the files, verifies all declared byte counts and SHA-256 digests, and atomically renames the directory into place. It refuses to overwrite an existing path. The output contains: ```text call-01J2Q1N4WQ/ ├── candidate.json └── events.jsonl ``` `candidate.json` conforms to [`production-regression-candidate.v1.json`](../src/hotato/schema/production-regression-candidate.v1.json). Its status is `CANDIDATE`: production evidence alone does not invent an expected outcome or become a CI gate. A reviewer authors the assertion and chooses which evidence may enter a shareable fixture. Verify on a clean, offline machine: ```python from hotato.production import verify_regression_candidate result = verify_regression_candidate("fixtures/candidates/call-01J2Q1N4WQ") assert result["valid"] ``` Payload persistence is default-deny at ingest. Each event type has a narrow allowlist for structural values such as evidence availability, validated content digests, timing counters, and bounded status enums. Every other scalar, object, or array is replaced as a whole by `redacted: true`, its canonical `byte_count`, and no value-derived digest. Omitting the digest prevents an offline dictionary attack against low-entropy redacted values. An allowlisted field with the wrong value type is reduced to the same descriptor. This preserves portable monitoring structure without assuming an unfamiliar field is safe. Field names and the event envelope remain visible, so identifiers still need review before a candidate is shared. A caller can explicitly use `store.ingest(event, redact_payloads=False)` inside its own boundary; the session manifest and candidate then declare `payload_storage`/`payloads` as `unredacted` or `mixed` instead of claiming redaction. Review remains required before sharing because envelope identifiers and allowlisted structural metadata may still be organization-specific. ## Audit and retention Verify the append-only audit hash chain: ```python assert store.verify_audit_chain()["valid"] ``` Audit targets are deterministic SHA-256 digests from their first write, so the raw provider identifier does not remain in the audit row after retention. These digests are integrity identifiers, not an anonymity guarantee: a low-entropy identifier can be guessed offline. Chain verification returns the first invalid sequence if a row changes and compares the computed head/count with a local metadata checkpoint to detect tail truncation. An administrator who can rewrite the entire database can rewrite both the chain and its local checkpoint; export the returned head digest to an external append-only system when that threat is in scope. Delete one session: ```python receipt = store.delete_session("call-01J2Q1N4WQ") ``` Apply a local retention window to finalized sessions: ```python receipts = store.enforce_retention(retention_seconds=30 * 24 * 60 * 60) ``` Deletion removes events, conflicts, sequence cursors, session state, alerts, and their transitions. A pseudonymous deletion receipt and the pseudonymous audit chain remain. The receipt records the pre-deletion manifest digest and deleted event count. Database backups and upstream collectors need their own retention policy; this process cannot delete copies outside its store. ## Operational boundary The included implementation has hermetic tests for authentication, exact-byte HMAC verification, commit visibility from a second connection, duplicate and conflict behavior, source-scoped ordering, evidence authority, finalization, late events, alert generations, bounded-label metrics, OTLP correlation, candidate tamper detection, audit tamper detection, and retention receipts. Those tests establish the local contract. They do not establish a public throughput, recovery-time, multi-region, or availability number. Measure those properties in the intended deployment with a frozen load schedule and publish the environment and raw results before making an operational claim. ================================================================================ FILE: docs/PYTEST.md ================================================================================ # Pytest: the fixture and the gate Install hotato and the plugin registers itself: standard `pytest11` entry point, zero `conftest.py`, zero imports. It adds one fixture and one opt-in session gate, both scoring with the CLI's engine and returning the same envelope every hotato surface emits. ```bash # zero-install with uvx, or: pipx install hotato uvx hotato --help ``` ## The `hotato_score` fixture Score a recording or a suite inside any test and assert on the envelope -- the same inputs the CLI takes: ```python def test_call_yields(hotato_score): env = hotato_score(stereo="call.wav", expect="yield") assert env["exit_code"] == 0 assert all(e["verdict"]["passed"] for e in env["events"]) def test_split_channels(hotato_score): env = hotato_score(caller="caller.wav", agent="agent.wav", expect="yield") assert env["events"][0]["verdict"]["passed"] def test_selftest_battery(hotato_score): env = hotato_score(suite="barge-in") assert env["exit_code"] == 0 ``` The envelope is the standard machine shape (`schema_version` "1"): per event, a `verdict` (`passed`, `did_yield`, `seconds_to_yield`, `talk_over_sec`), a `signals` bus, a `fix` on every failure, and `limits`. Assert on whatever your test cares about. ## The `--hotato-suite` session gate Opt in and the battery runs after your tests, printing a summary and failing the whole session (exit 1) on a regression: ```bash pytest --hotato-suite # default suite: barge-in ``` The bundled battery self-tests the harness. Point the same flag at your own labelled scenario and audio directories to gate on your own agent: ```bash pytest --hotato-suite \ --hotato-suite-scenarios corpus/suites/gold/scenarios \ --hotato-suite-audio corpus/suites/gold/audio ``` Any directory in the same scenario shape works, including your own labelled calls -- `docs/SUITES.md` covers the bundled tiers, `docs/SUBMITTING.md` covers building fixtures from your own recordings. ## Two ready-made gates - This flag rides on a test run you already have: turn-taking checks run with every `pytest` invocation, locally and in CI. - The GitHub workflow (`docs/CI.md`) is the other ready-made gate -- it scores every pull request and posts a sticky results comment. Use either, or both. - The plugin runs offline with deterministic, self-contained fixtures, so results stay stable run to run. ================================================================================ FILE: docs/README.md ================================================================================ # hotato docs Find what broke in your agent calls. Pin it so it never ships again. The loop: `autopsy` one recording (or `scan` a folder, or pull a platform's recent calls with `vapi health`), `pin` the incident as a portable failure check, and `prove` re-runs the stored evidence in CI so it never ships again. This index maps every doc to the step it belongs to. New here? Run `hotato autopsy ./call.wav` on one recording (**[AUTOPSY.md](AUTOPSY.md)**), or start with **[GETTING-STARTED.md](GETTING-STARTED.md)**. The whole loop on one page: **[LIFECYCLE.md](LIFECYCLE.md)**. ## The loop: find what broke, then pin it - [AUTOPSY.md](AUTOPSY.md) - `autopsy`, `scan`, and the ` health` commands: one recording (or a folder, or your platform's recent calls) in, the incident list and health report out; `pin` graduates an incident - [EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md) - the four-tier evidence policy: what every verdict stands on - [CONTRACTS.md](CONTRACTS.md) - failure contracts: a portable CI bundle of one call moment - [CI.md](CI.md) - `prove` and the CI gate: fail the pull request on the pinned evidence, offline - [GETTING-STARTED.md](GETTING-STARTED.md) - one path from first touch to a CI gate - [START.md](START.md) - guided first run on the bundled demo data - [WHY.md](WHY.md) - four timing failures a text-level eval cannot see ## Continuous use - [CONNECT.md](CONNECT.md) - store a stack's credentials once; feed every hotato command from your platform - [ADAPTER-STATUS.md](ADAPTER-STATUS.md) - per-stack pull, endpoint, and channel-separation status - [PRODUCTION-MONITORING.md](PRODUCTION-MONITORING.md) - turn production call events into offline regression candidates - [WORKSPACE.md](WORKSPACE.md) - `hotato serve` and `hotato console`: the self-hosted team web workspace and the live call console - [SELF-HOST.md](SELF-HOST.md) - run the full workspace in your own VPC ## Working with audio - [TRUST.md](TRUST.md) - is this recording even scorable? - [TRUST-MATRIX.md](TRUST-MATRIX.md) - the input-condition-to-behaviour contract for the trust check - [TRUST-GALLERY.md](TRUST-GALLERY.md) - eight recordings, eight verdicts, verbatim output - [DIARIZE.md](DIARIZE.md) - diarize a mono recording to make it scorable - [TRANSCRIBE.md](TRANSCRIBE.md) - attach a transcript beside the timing score - [FULL-DUPLEX.md](FULL-DUPLEX.md) - score the moment both sides speak at once ## Reference - [API.md](API.md) - the stdlib-only scoring core, Python API - [SDK.md](SDK.md) - the typed Python SDK facade over the CLI - [MCP.md](MCP.md) - the hotato MCP server and its tools, over stdio - [METHODOLOGY.md](../METHODOLOGY.md) - how the timing measurement works, end to end - [THREAT-MODEL.md](THREAT-MODEL.md) - which commands are offline, which reach the network - [EGRESS.md](EGRESS.md) - every network call site mapped to its command - [VALIDATION.md](VALIDATION.md) - the three separate jobs hotato is validated on - [EVIDENCE-PACK.md](EVIDENCE-PACK.md) - the reproducible proof artifacts, ranked - [GALLERY.md](GALLERY.md) - every image and worked example, each reproducible - [COMPARE.md](COMPARE.md) - where hotato sits next to broad QA platforms - [evidence/README.md](evidence/README.md) - the evidence standard: what counts, and ranking - [case-studies/README.md](case-studies/README.md) - the honesty standard every case study meets ## Lab: the deep toolkit Every command in this section lives under `hotato lab` (`hotato lab --help`). The public commands are durable. hotato lab moves faster, and every command name that worked before 1.17 still runs unchanged. ### Investigate and evaluate - [INVESTIGATE.md](INVESTIGATE.md) - one recording in, ranked candidate moments out - [ANALYZE.md](ANALYZE.md) - drop a folder, rank and hear the worst moments - [ASSERTIONS.md](ASSERTIONS.md) - deterministic typed assertions over transcript, trace, and timing - [RUBRIC.md](RUBRIC.md) - the model-judged rubric lane, scored with a pinned local model - [EXPLAIN.md](EXPLAIN.md) - root-cause-by-layer attribution from existing results - [STATE-ADAPTERS.md](STATE-ADAPTERS.md) - ground state assertions in your system of record - [scenarios/dtmf-verification.md](scenarios/dtmf-verification.md) - verify DTMF reached the far end - [scenarios/echo-self-interruption.md](scenarios/echo-self-interruption.md) - diagnose self-interruption from echo bleed ### Observe - [OBSERVE.md](OBSERVE.md) - LLM and voice observability from your OpenTelemetry spans, locally - [TRACE.md](TRACE.md) - voice traces: the pipeline-event timeline - [OTEL.md](OTEL.md) - ingest OTel traces into the `voice_trace` span format - [latency-waterfall.md](latency-waterfall.md) - per-hop latency waterfall from a scored call ### Test and simulate - [SIMULATE.md](SIMULATE.md) - render a scenario into a deterministic labelled conversation - [CONVERSATION-TEST.md](CONVERSATION-TEST.md) - one file, one call, a per-dimension scorecard - [SUITES.md](SUITES.md) - four tiered deterministic corpus suites - [SUITE-RUN.md](SUITE-RUN.md) - execute a suite, per-dimension report - [GENERATIVE-CALLER.md](GENERATIVE-CALLER.md) - the bounded caller engine: scripts, graphs, replay - [DRIVE-A-CALL.md](DRIVE-A-CALL.md) - originate a call against a live agent, then score - [CALLER-LOAD.md](CALLER-LOAD.md) - replay a bounded caller program under load - [LOAD-AND-RECOVERY.md](LOAD-AND-RECOVERY.md) - schedule calls under load, keep per-call evidence - [COUNTEREXAMPLES.md](COUNTEREXAMPLES.md) - reduce a scripted failure to a minimal repro - [PIPER-CALLER-TTS.md](PIPER-CALLER-TTS.md) - local Piper TTS adapter for caller speech - [scenarios/browser-vs-pstn.md](scenarios/browser-vs-pstn.md) - score the same moment through telephony degradation - [scenarios/load-and-recovery.md](scenarios/load-and-recovery.md) - behaviour under concurrent load, with receipts ### Fix and gate, the long way - [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md) - turn one bad call into a CI gate, step by step - [STARTER.md](STARTER.md) - `hotato lab init starter` scaffolds a CI gate and config - [PYTEST.md](PYTEST.md) - the pytest fixture and opt-in session gate - [FIX-LOOP.md](FIX-LOOP.md) - the closed loop: find, fix, prove it is fixed - [FIX-PLANS.md](FIX-PLANS.md) - the guarded fix ladder: diagnose, inspect, plan, apply - [FIX-TRIAL.md](FIX-TRIAL.md) - before/after fix proof, fail-closed and clone-only - [APPLY.md](APPLY.md) - guarded, clone-only staged apply of a fix patch - [RECAPTURE.md](RECAPTURE.md) - prove the current agent, not the frozen recording - [RELEASE-COMPARE.md](RELEASE-COMPARE.md) - diff two releases per dimension - [CARDS.md](CARDS.md) - render one measured moment as a PR-native SVG card - [scenarios/false-interruption-replay.md](scenarios/false-interruption-replay.md) - a false-stop becomes a contract replayed in CI ### Pipe your stack in - [INGEST.md](INGEST.md) - a passive webhook on-ramp scanning completed calls - [SET-AND-FORGET.md](SET-AND-FORGET.md) - a passive scheduled sweep for regression monitoring - [TRANSPORT-RUNTIME.md](TRANSPORT-RUNTIME.md) - lifecycle, delivered media, and assertion facts across transports - [CALLER-SIDECAR-PROTOCOL.md](CALLER-SIDECAR-PROTOCOL.md) - the caller/transport sidecar WebSocket protocol - [LIVEKIT-CALLER-SESSION.md](LIVEKIT-CALLER-SESSION.md) - direct LiveKit room transport for the caller engine ### Fleet, reports, and benchmarks - [GUARDIAN-FLEET.md](GUARDIAN-FLEET.md) - a control plane running the evidence workflow continuously - [REPORTS.md](REPORTS.md) - reporting surfaces: doctor, report, team, export - [BENCHMARK.md](BENCHMARK.md) - the measurement-error harness over labelled recordings - [BENCH-SPEC.md](BENCH-SPEC.md) - frozen batteries, scoring protocol, verify - [BENCHMARK-STACKS.md](BENCHMARK-STACKS.md) - run one battery through your configured stacks ## Contributing - [SUBMITTING.md](SUBMITTING.md) - the full path from a call to a merged corpus entry - [CORPUS-GOVERNANCE.md](CORPUS-GOVERNANCE.md) - consent, PII, and publishing rules for contributed calls - [RFC-ROLEPLAY-FIXTURES.md](RFC-ROLEPLAY-FIXTURES.md) - a share-safe role-play fixture format - [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md) - the maintainer gates to clear before a release See also the repository [`CONTRIBUTING.md`](../CONTRIBUTING.md), [`SECURITY.md`](../SECURITY.md), and [`CHANGELOG.md`](../CHANGELOG.md). ================================================================================ FILE: docs/RECAPTURE.md ================================================================================ # Recapture: proving the CURRENT agent, not just the frozen recording `hotato contract verify` and a promoted CI fixture both re-score the SAME audio they were created from. That audio never changes, so those checks fail only on changed evidence, policy, or scorer -- never because your deployed agent changed. Proving the CURRENT agent still avoids a bug takes a NEW recording of the same caller stimulus, scored the same way: this page is the manual walkthrough. See the two-lane table in [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-same-command-two-different-proofs) for the guarantee each lane gives. Reproducing a caller's stimulus against a live agent is a human task: placing the call, running a scripted test caller, knowing what the caller said. Hotato's job starts once you hand it the new recording. ## When you need this - After a prompt, model, or config change you believe fixes a known bug, and you want proof beyond "the frozen contract still fails the same way it always did" (it will -- the recording never changes). - On a schedule (weekly, before a release), to catch silent regressions the frozen-evidence CI gate structurally cannot see. - Before telling a customer, a teammate, or a PR reviewer "this is fixed." If your stack can create a staging clone (`vapi`, `retell`), `hotato fix trial` (with `hotato apply --clone --yes`) automates the before/after half of this -- source vs. clone, re-captured -- and folds in the neighbouring-cases check. Use this manual walkthrough when that path isn't available, or the recapture is on a live production agent you can't clone. ## Step 1: read the original contract's stimulus and policy ```bash hotato contract inspect contracts/refund-cutoff-001.hotato ``` Read from the printed fields (or `contract.json` directly with `--format json`): - `label.expected_behavior` (`yield` or `hold`) -- what the agent should do. - `policy.pass_conditions` (`max_talk_over_sec`, `max_time_to_yield_sec`) -- the SAME thresholds the fresh recapture must be scored against, for an apples-to-apples comparison. - `source.category` / `source/call_metadata.json` (if `--include-identifiers` was used at creation) -- context recorded about the caller scenario. Hotato stores timing evidence, not a transcript or script; the caller's exact words come from your own call notes, scripted test caller, or a human re-running the scenario. ## Step 2: reproduce the caller stimulus against the CURRENT agent Human-assisted is fine and expected: place the call (or run the scripted test caller) against the agent as deployed TODAY, using the same scenario that produced the original recording. ## Step 3: capture dual-channel audio Record caller and agent on separate channels (two-channel WAV, or two aligned mono files) -- the same input every other Hotato command takes. A mixed mono recording scores unreliably; see [`docs/CONTRACTS.md`](CONTRACTS.md#the-opt-in-diarized-mono-path) for the quality-gated diarized-mono fallback when separated channels aren't available. ## Step 4: create a NEW contract from the fresh recording, same policy ```bash hotato contract create --stereo recapture-2026-07-10.wav --onset 41.90 \ --expect yield --id refund-cutoff-001-recapture-2026-07-10 \ --out contracts \ --max-talk-over 0.6 --max-time-to-yield 1.0 ``` Use the SAME `--expect` and `--max-talk-over` / `--max-time-to-yield` values Step 1 read from the original policy. `contract create` scores immediately and refuses a not-scorable recording (exit 2) rather than writing a meaningless bundle -- a refusal here is itself information: the recapture didn't produce a scorable moment, not that the agent passed. ## Step 5: verify the fresh contract ```bash hotato contract verify contracts/refund-cutoff-001-recapture-2026-07-10.hotato ``` A pass here is the claim the frozen-recording gate can't make: the CURRENT agent, on a fresh recording of the same stimulus, still meets the same labelled policy. Keep both contracts -- the original (the historical record of the bug) and the recapture (today's evidence) -- they answer different questions. ## How Hotato tells a recapture from a re-score Every run envelope Step 3's capture produces (and every `hotato contract create`) carries an `audio_provenance` block per event: a streamed sha256 of the raw file bytes and of the decoded PCM samples, plus sample rate and frame count -- the mechanical proof that Step 2/3 happened: a NEW recording, not the old one replayed through a looser threshold. `hotato fix trial` checks this block rather than trusting it: for every guarded fixture it validates the block, recomputes the digests from the audio on disk, and compares decoded PCM before vs. after, so a re-score can't dress up as a fresh capture. An `improved` verdict is reachable only on verifiable evidence; a merely-asserted or unrecomputable identity downgrades to `inconclusive`, never `improved`. See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md#fresh-capture-provenance-guard-a-re-score-is-never-a-fix) for the full guard, table included. `hotato contract verify` on a frozen bundle skips this guard by design, per the two-lane table above: it re-scores the SAME recording on purpose (a CI regression gate on labelled evidence, not a fix claim), so identical audio identity there is expected, not a red flag. The guard applies only where a "fix" is being claimed: `fix trial`'s before/after. ## Claim language: what each kind of evidence lets you accurately say The same word ("verified", "fixed", "passed") means something different depending on which of these five you're holding. Match what you say to what you have: **Historical contract only** - How you get it: `hotato contract create` ran once; you're reading `contract.json` / `hotato contract inspect`, no `verify` run since. - Accurate to say: "On \[created_at], a human labeled this call and hotato measured \[timing] against that label; this is the frozen record of that one measurement." - Overclaim to avoid: "This proves our agent behaves correctly" -- nothing has been re-checked since capture; it speaks to that one recorded moment, not to now. **Contract plus unchanged historical audio** - How you get it: `hotato contract verify` re-scored the SAME `audio/event.wav` the contract was created from. - Accurate to say: "`hotato contract verify` re-measured stored evidence and it still meets its policy." (This is the literal caveat `contract verify` prints -- see below.) - Overclaim to avoid: "The deployed agent no longer has this bug." A pass here fails only if the evidence, policy, or scorer changed -- never because the deployed agent changed. See the two-lane table in [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-same-command-two-different-proofs). **Separately captured current-agent take** - How you get it: Steps 1-5 above, standalone (no paired before/after). - Accurate to say: "The CURRENT agent, on a fresh recording of the same stimulus captured \[date], still meets the labeled policy. One data point." - Overclaim to avoid: "This proves the fix caused the improvement" (coincidence, not causation) or "this guarantees the next call passes too" (one recapture is one data point, not a rate -- see Limits below). **Fresh take plus opposite-risk cases** - How you get it: `hotato fix trial` `improved` -- the provenance guard held on every guarded fixture (target and hold) and the hold/opposite- risk fixture still passed. - Accurate to say: "This proves the specific fresh capture scored above, at the revision it was captured from, including that a paired hold/opposite-risk case did not flip." (The literal caution `fix trial` prints -- see below.) - Overclaim to avoid: "This fix is now permanently verified" or "it will keep holding after future deploys." A later deploy is a new revision this report says nothing about, and nothing here re-runs itself. **Production rerun after deploy** - How you get it: you (there is no automatic trigger) recaptured again against the LIVE deployed agent post-deploy, per this page, on your own schedule. - Accurate to say: "As of \[date], a fresh production capture reverified the same labeled stimulus against the live agent." Still one data point per run. - Overclaim to avoid: "The fix is confirmed working in production, no further checks needed." This doesn't run in CI by itself (see Limits below); each rerun is one more independent data point, not a standing guarantee that survives the NEXT deploy. Two of these statements are wired straight into the command output, not just guidance here -- a reader never has to take the caution on faith: - **`hotato contract verify`** (the stored-evidence row above) prints, in every text, HTML, and rollup render: *"This result re-measures stored evidence. It does not test the current agent."* - **`hotato fix trial`**, wherever its audio-provenance section renders (an `improved` verdict, or a `refused`/`inconclusive` one the guard downgraded) prints: *"Provenance caution: this proves the specific fresh capture scored above, at the revision it was captured from. It does not certify a later deploy or every future call, and it does not re-run itself; recapture again after the next change."* ## Limits, stated plainly - **Not a controlled experiment.** A pass coincides with the change; it does not prove causation. See [`docs/VALIDATION.md`](VALIDATION.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md). - **The stimulus match is only as good as your reproduction.** A scripted test caller reproduces wording and timing more faithfully than an ad hoc call; matching the fresh call to the original scenario is your judgment call, like the original label. - **One recapture is one data point.** A single fresh pass doesn't establish a rate; run it more than once, or fold it into a battery-scale check (`hotato verify --before --after`, [`docs/FIX-LOOP.md`](FIX-LOOP.md)) for a distribution instead of a single yes/no. - **This doesn't run in CI by itself.** Unlike `contract verify` on the frozen recording, rerunning against production has no automatic trigger -- wiring one (a scheduled synthetic-caller job, a pre-release checklist item) is your call. ## What this does not stop This is an offline tool: a user who controls every input can shape what it measures. Nothing on this page, and nothing `hotato fix trial`'s guard recomputes, overrides that. Specifically: - **A fresh recording of a fabricated stimulus still passes.** If the "same scenario" you reproduced in Step 2 doesn't match the original bug, the audio identity check has nothing to say about it -- it verifies the bytes are freshly captured, never that the scenario is the one you claim. - **Repacking a `.hotato` contract with a loosened policy still verifies as an unsigned bundle.** `MANIFEST.sha256.json` is integrity (the archive agrees with itself), not authenticity (who approved the policy inside it). A trusted signature over the manifest closes that gap: an unsigned bundle is reported unauthenticated, while an HMAC-SHA256 attestation (or an Ed25519 signature for third-party trust) marks an authenticated bundle. Absent a signature, treat a bundle's origin with the same care as any file that arrived from outside your own pipeline. - **A resample, re-encode, or gain change of the SAME call still reads as a distinct capture,** because the guard's freshness check is decoded-PCM difference, and those transforms change the decoded samples of an otherwise-identical call. A known residual, not a claim broken. See [`docs/FIX-TRIAL.md`](FIX-TRIAL.md#what-this-does-not-stop) for the same note at length, in the context of the automated guard it applies to. ## Read more - The two-lane distinction this page exists to close: [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-same-command-two-different-proofs) - The automated before/after path for clone-appliable stacks: [`docs/FIX-TRIAL.md`](FIX-TRIAL.md) · [`docs/APPLY.md`](APPLY.md) - Battery-scale before/after proof, coincidence not causation: [`docs/FIX-LOOP.md`](FIX-LOOP.md) - What a contract proves, precisely: [`docs/CONTRACTS.md`](CONTRACTS.md#what-a-contract-proves-precisely) ================================================================================ FILE: docs/RELEASE-CHECKLIST.md ================================================================================ # Release checklist Every gate to clear before a release ships, top to bottom: green means proceed. ## Code and tests - [ ] `python3 -m pytest -q` fully green on a clean checkout. - [ ] `python3 sync_engine.py --check` passes: the vendored `_engine` is byte-identical to upstream. - [ ] CI is green on the release commit (`.github/workflows/tests.yml`). - [ ] GitHub Actions stay SHA-pinned: every `uses:` in `.github/workflows/*.yml` references an action by full 40-char commit SHA plus a trailing `# vX.Y.Z` comment, never a mutable `@v4` tag or `@release/v1` branch. Bump the SHA and comment together; never revert to a floating tag. Same rule for the root `action.yml` and `tests/fixtures/action-consumer/workflows/consumer.yml` (`tests/test_action_consumer.py` gates both). - [ ] Root Action docs match the release: `docs/CI.md` ("The root Action" section) names v1.4.0 as the availability floor (first release to ship `action.yml`, a fixed historical fact) and shows the CURRENT release as the copy-paste example -- the `git ls-remote refs/tags/vX.Y.Z` command, the `# vX.Y.Z` comment on the `attenlabs/hotato@` pin, and the `hotato==X.Y.Z` pip example must all name the release being cut. `tests/test_version_lockstep.py::test_ci_md_adoption_example_pins_match_pyproject` gates those three; confirm `git ls-remote` prints the SHA the doc documents. - [ ] Version bumped in EVERY lockstep site: `pyproject.toml`, `src/hotato/__init__.py` (`__version__` -- what `hotato --version`, `hotato describe`, and stackbench provenance self-report), `server.json` (both `version` fields), `CITATION.cff` (`version` + `date-released`), the `llms.txt` version line; `CHANGELOG.md` gets a dated entry. `tests/test_version_lockstep.py` gates every site it can see: it parses and compares `pyproject.toml`, `__init__.py`, the `describe` manifest, installed dist metadata, `llms.txt`'s `Version` line, `server.json` (top-level + each package), and `CITATION.cff`'s `version:` -- a missed bump anywhere reddens CI. ## README and assets - [ ] Regenerate README assets before release: `python3 scripts/render_readme_assets.py`. - [ ] Verify the README screenshot renders: `docs/assets/hotato-demo-report.png` is current, shows the failing demo summary, a timeline, and a fix card, and displays correctly in the GitHub README preview. - [ ] README badges resolve (the `tests.yml` workflow badge reflects the current run). - [ ] Verify the site hero matches the README: same tagline, same demo screenshot, same install command on hotato.dev. ## Adapters - [ ] Adapter last-verified dates are fresh in `docs/ADAPTER-STATUS.md`: re-verify each capture path against the vendor's current API docs and update the date column. - [ ] Fixmap knob names match the same verified APIs (`src/hotato/fixmap.py`). ## Package - [ ] **Tag the release commit and push the tag** (`git tag -a vX.Y.Z -m 'hotato X.Y.Z' && git push origin vX.Y.Z`) -- do this FIRST: the build step below is tag-locked and builds from the tag's committed tree, refusing (exit 2) if the tag does not yet exist. - [ ] Build the release artifacts with `python3 scripts/build_release.py` -- the REQUIRED build step: it checks out the release tag and exports that tag's committed tree into a fresh temporary directory (so only files committed at the tag can enter the artifacts -- generated corpus renders, `examples/reference-agent/.out/`, and other working-tree leftovers cannot), forces `umask 022` for mode-stable bytes, builds with `SOURCE_DATE_EPOCH` from the tagged commit, and writes the sdist, wheel, and their sha256s to `./dist`. It refuses a drifted HEAD by default; `--ref ` selects a specific tag and `--allow-drift` builds from HEAD (never publish an `--allow-drift` build). `tests/test_release_supply_chain.py` holds the same tag-faithfulness invariant in CI. - [ ] Install the built wheel in a fresh venv and run `hotato run --suite barge-in`. CI does this on every version tag in the `release.yml` `sanity` job. - [ ] Generate and validate the SBOM(s), then attach them to the GitHub release: `python3 scripts/gen_sbom.py` writes `dist/hotato.sbom.cdx.json` -- a minimal CycloneDX bill of materials for the core package and every declared dependency, generated offline from `pyproject.toml`. For a per-profile breakdown: `python3 scripts/gen_sbom.py --list-profiles` lists `core` plus each declared extra, and `python3 scripts/gen_sbom.py --profile ` writes `dist/hotato.sbom..cdx.json` (core alone, or core plus that one extra). Validate each with `python3 scripts/gen_sbom.py --check `, then upload `dist/*.cdx.json` alongside the sdist/wheel. CI does all of this automatically: `release.yml` uploads them in the `hotato-release-dist` artifact, `publish-pypi-oidc.yml` in the `hotato-sbom` artifact. - [ ] Publish to PyPI via **Trusted Publishing (OIDC)**, the default path (see below): dispatch `publish-pypi-oidc.yml` with `version` = the release's `pyproject.toml` version and `confirm` = `PUBLISH`. That workflow builds reproducibly (a second, pinned-backend build has its `sha256sum` compared against the first -- a mismatch fails the run), generates and validates the SBOMs, and uploads the built artifacts; a separate gated `publish` job downloads those exact bytes, attests build provenance over them, and uploads to PyPI with a short-lived OIDC token minted fresh each run. - [ ] Verify the build-provenance attestation for the published wheel and sdist: `gh attestation verify dist/hotato--py3-none-any.whl --repo attenlabs/hotato` (and again for `dist/hotato-.tar.gz`). This checks the GitHub-signed provenance emitted by the `publish` job's `actions/attest-build-provenance` step, confirming those exact bytes were built by this repo's workflow. - [ ] Verify the published SBOM(s): re-run `python3 scripts/gen_sbom.py --check dist/hotato.sbom.cdx.json` (and each `dist/hotato.sbom..cdx.json`) on the files attached to the release, and confirm each SBOM's `metadata.component.version` matches the release version. - [ ] Verify `uvx hotato demo` works from the published package (fresh machine or cleared uv cache): it renders and opens the failing demo report. - [ ] Verify `uvx hotato doctor` (self-test path) works from the published package. - [ ] Publish the GitHub release notes for the tag `vX.Y.Z` (created above), pointing at the CHANGELOG entry. - [ ] **Reconcile the public surfaces (release-blocking).** Run `python3 scripts/check_public_surfaces.py --online` from a clean network. It cross-checks the version across `pyproject.toml`, `hotato.__version__`, the git tag, the README Action pin, the CHANGELOG top entry, and `llms.txt` (offline group A), then asserts PyPI's `/pypi/hotato//json`, `hotato.dev/release.json`, `hotato.dev/llms.txt`, and the `hotato.dev/` landing page (Googlebot UA, no-cache) all report the shipped version and positioning with no retired overclaim (group B). The release is not done until this exits 0. If the site lags, redeploy from the `vX.Y.Z` tag and purge the Cloudflare HTML cache, then re-run. (Group A alone runs in CI with no egress.) ### Publishing path: PyPI Trusted Publishing (OIDC) -- DEFAULT `.github/workflows/publish-pypi-oidc.yml` is the default publish path: a `workflow_dispatch`-only workflow that builds with a pinned, reproducible backend, runs the full test suite, `twine check`s the artifacts, runs a second-build `sha256sum` reproducibility check, generates and validates the CycloneDX SBOMs, and publishes via [PyPI Trusted Publishing](https://docs.pypi.org/trusted-publishers/) with a short-lived GitHub OIDC token minted fresh each run. It requires a `version` input matching `pyproject.toml` and a `confirm` input equal to `PUBLISH`; the `publish` job runs under the `pypi` GitHub Environment with `id-token: write` and `attestations: write` granted only there, and attests build provenance over the exact artifacts before uploading. **One-time operator action**, required before this path can publish: register the Trusted Publisher on PyPI for `attenlabs/hotato` + workflow `publish-pypi-oidc.yml`. Until then, PyPI rejects the OIDC token, so a dispatch builds, tests, and attests cleanly, then stops at the upload step -- safe by construction, since the workflow never holds a credential that could leak. **hotato is already on PyPI**, so this is the existing-project flow, not the pending-publisher flow for unpublished names: 1. Sign in to PyPI as an owner/maintainer of the `hotato` project and open `https://pypi.org/manage/project/hotato/settings/publishing/`. 2. Under "Add a new publisher", choose GitHub and fill in: - Owner: `attenlabs` - Repository name: `hotato` - Workflow filename: `publish-pypi-oidc.yml` - Environment name: `pypi` 3. Save. Nothing is stored on either side; PyPI accepts only OIDC tokens minted by that exact repo/workflow/environment combination. 4. In the GitHub repo, create the `pypi` environment if it does not already exist (Settings > Environments > New environment, name `pypi`). Optionally add required reviewers there, for a human-approval gate on top of the `confirm`/`version` checks already in the workflow. 5. To cut a release this way: dispatch `publish-pypi-oidc.yml` with `version` set to the release's `pyproject.toml` version and `confirm` set to `PUBLISH`. ### Fallback: manual token upload (twine) When Trusted Publishing isn't available -- the publisher isn't registered yet, and a release still has to go out -- publish by hand with a project-scoped API token. Build the SAME way the OIDC path does: a pinned backend plus `SOURCE_DATE_EPOCH` from the release commit, so the hand-uploaded wheel is byte-reproducible instead of carrying wall-clock ZIP timestamps no rebuild can match: ```bash python3 -m pip install "pip==26.1.2" "build==1.2.2.post1" "setuptools==83.0.0" "wheel==0.46.2" "twine==6.2.0" python3 scripts/build_release.py # git-archive export of the release TAG (tag-locked); umask 022; SOURCE_DATE_EPOCH from the tagged commit python3 -m twine check --strict dist/*.tar.gz dist/*.whl python3 -m twine upload dist/*.tar.gz dist/*.whl # token scoped to the `hotato` project ``` This path uses a long-lived credential and skips build-provenance attestation. Prefer the OIDC path above; treat twine as the fallback, and still attach the generated `dist/*.cdx.json` SBOMs to the GitHub release. **Reproducibility scope (what holds).** The wheel is byte-for-byte reproducible ONLY when built with the exact backend pins above and `SOURCE_DATE_EPOCH`; both publish paths do this, and the OIDC workflow rebuilds a second time and compares the wheel's raw `sha256sum` to prove it. The sdist is content-reproducible, not byte-reproducible: setuptools does not normalize the gzip mtime or the tar member mtimes, so its `.tar.gz` bytes vary run-to-run even at a fixed `SOURCE_DATE_EPOCH`. What stays stable is its CONTENT (member names, modes, file bytes), which the workflow compares instead -- a changed or injected file fails, timestamp noise does not. ## After release - [ ] Smoke-test the README quickstart commands exactly as written. - [ ] File follow-ups for anything that needed a manual touch, so the next release is one pass. ================================================================================ FILE: docs/RELEASE-COMPARE.md ================================================================================ # `hotato release compare`: diff two releases, per dimension `hotato release compare BASELINE CANDIDATE` reads two releases from the fleet registry and reports what moved between them -- `BASELINE` first, `CANDIDATE` second. It reports movement only; gating a release on that movement is a separate step (a `required_for_release` suite). Each release is a snapshot recorded by [`hotato suite run`](SUITE-RUN.md). Run the same suite twice under two stable `--release` ids in the same workspace and registry, then diff them. ## What it reports - **Per-dimension counts for each side, plus the delta.** Each dimension (`outcome`, `policy`, `conversation`, `speech`, `reliability`) keeps its own three counts (pass / fail / inconclusive) and its own delta -- every dimension scored on its own lane. - **New failures** -- a `scenario x dimension` that PASSED on the baseline and FAILs on the candidate. **Fixed-since** -- the reverse. Both diff **only** where BOTH releases ran the same `scenario x dimension`; a scenario only one side ran counts as new coverage, separate from a regression. - **Per-scenario status changes** -- every `scenario x dimension` whose status differs between the two, diffed only where both sides have a comparable result. The releases' pinned digests are surfaced, so the reader knows exactly which two snapshots were compared, every time. ## Empty state A side with no runs is stated plainly: - a release id absent from the workspace is reported as unregistered; - a release that is registered but has **zero runs** is reported as an empty state. When no `scenario x dimension` result is comparable across both releases, the report says so: new-failures / fixed-since needs a scenario BOTH releases ran. ## Compare ```bash hotato release compare reference-agent-v1 reference-agent-rc2 ``` `--workspace`/`-w ID` selects the fleet-registry workspace (default `default`); `--registry PATH` sets the registry home (default `~/.hotato/fleet`); `--format {text,json}` picks the output. ## Worked example: two reference-agent releases Record two releases of the reference agent into the same registry and workspace, then diff them. The reference example writes to `examples/reference-agent/.workspace`, workspace `reference`: ```bash cd examples/reference-agent # record the baseline release python run_reference.py --release reference-agent-v1 # edit the scenario / agent_mock fixtures (or point at a changed agent), # regenerate, and record the candidate release python run_reference.py --generate --release reference-agent-rc2 # diff them (same workspace + registry both runs recorded into) hotato release compare reference-agent-v1 reference-agent-rc2 \ --workspace reference --registry ./.workspace ``` Two runs over identical fixtures record byte-identical results, so the diff shows all-zero deltas and no status changes. Edit the fixtures between the two runs to see movement. The text output follows this shape (counts here are illustrative placeholders): ``` hotato release compare: baseline reference-agent-v1 -> candidate reference-agent-rc2 (workspace reference) baseline runs=375 conv=375 eval=... candidate runs=375 conv=375 eval=... per-dimension counts (baseline -> candidate, delta; never blended): outcome ...P/...F/...I -> ...P/...F/...I (dP +.., dF -.., dI +..) policy ...P/...F/...I -> ...P/...F/...I (dP +.., dF -.., dI +..) conversation ...P/...F/...I -> ...P/...F/...I (dP +.., dF -.., dI +..) speech ...P/...F/...I -> ...P/...F/...I (dP +.., dF -.., dI +..) reliability ...P/...F/...I -> ...P/...F/...I (dP +.., dF -.., dI +..) new failures (PASS on baseline -> FAIL on candidate): N - / fixed since (FAIL on baseline -> PASS on candidate): N + / all per-scenario status changes: ~ / : PASS -> FAIL ``` ## Exit codes - **`0`** -- the two releases were compared (per-dimension deltas and new-failures / fixed-since printed; a side with no runs is reported as an empty state, exit stays `0`). - **`2`** -- a usage error or an unreadable registry `--registry`. ## See also - [SUITE-RUN.md](SUITE-RUN.md) -- records the releases this command diffs. - [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- the per-dimension scorecard model. - [WORKSPACE.md](WORKSPACE.md) -- browse the releases and their runs with `hotato serve`. ================================================================================ FILE: docs/REPORTS.md ================================================================================ # Reports: doctor, report, team, export Four surfaces, one scorer: reproducible timing measurements read from the envelope a scoring run emits, with the method exposed at every layer. ## `hotato doctor`: the 5-minute path One command: pass a recording and it scores that; run it bare and it runs the bundled self-test battery. Either way it writes a self-contained HTML report and opens it in your browser -- on a headless box, it prints the path instead. ```bash uvx hotato doctor --stereo call.wav # score your call, open the report uvx hotato doctor # self-test fallback, same flow uvx hotato doctor --demo --no-open --out report.html ``` Wraps the scorer and report, runs offline end to end. Exit codes match `run`: `0` all pass, `1` a regression (`--no-fail` forces `0`), `2` usage/IO error or a not-scorable recording -- meaning it lacks a moment to measure (the caller channel is silent, or the agent wasn't talking when the caller started), reported plainly instead of a verdict. **Your own recording gets its audio embedded in the report by default.** Scoring a call with `--stereo` / `--caller`+`--agent` writes the scored audio into the HTML file as a base64 data URI, so hearing the moment next to its timeline works offline (the bundled self-test fallback stays unembedded). Sharing, mailing, or posting the resulting `report.html` shares the raw recording -- treat it with the same care. ## `hotato report`: the visual report One self-contained file -- inline CSS, inline SVG, zero external requests -- that opens by double-click and survives being mailed around. ```bash uvx hotato report --stereo call.wav --out report.html uvx hotato report --suite barge-in --out selftest.html uvx hotato report --stereo call.wav --format md --out report.md uvx hotato report --stereo call.wav --embed-audio --out report.html # opt-in: embed the audio too ``` `--embed-audio` embeds the exact scored audio (base64, under a size cap) so the report is a fully self-contained, hearable artifact. Same caution: a report built with `--embed-audio` (or `hotato doctor` on your own recording, which sets it by default) carries the call audio inside the HTML -- give it the same care as the raw recording before posting it publicly or attaching it to a public issue/PR. Per event it draws a to-scale caller/agent activity timeline from the frame data -- overlap shaded, caller-onset and yield markers, measured talk-over seconds, expected vs. measured, and a PASS or FAIL chip. Three or more event cards add an analytics rollup, computed from the same measurements: | Chart | Shows | |---|---| | Time-to-yield distribution | One dot per measured yield, with mean, median, and p90 (definitions in `METHODOLOGY.md`) | | Talk-over histogram | Per-event seconds, bucketed on a fixed grid | | Failure clustering by fix class | Failures grouped by the setting each one points at, so five failures read as one config change | Every timeline carries a collapsible **frame inspector**: the full frame dump behind that event as a table (`t_sec`, per-channel dBFS, active flags, thresholds) -- any pixel on the page can be re-derived by hand. Every threshold the run used sits in one collapsed "Thresholds used" panel at the end, so the numbers stay reproducible without a parameter table above every render. ### Voice-trace context with `--trace` Pass `--trace voice_trace.jsonl` (written by `hotato trace ingest`; format: [`TRACE.md`](TRACE.md)) to attach a voice trace as **context** next to the timing: ```bash hotato trace ingest --otel spans.otel.jsonl --out voice_trace.jsonl hotato report --stereo call.wav --trace voice_trace.jsonl --out report.html ``` The report gains one collapsed "Trace (context, not a score)" section: the trace's discrete voice-pipeline events -- TTS cancel/stop, ASR partials, tool calls -- as a mono span table. It stays scoped to context: `did_yield`, `talk_over_sec`, `seconds_to_yield`, and the PASS/FAIL verdict come from the scorer alone; the trace folds into the envelope as an additive `trace_context` key. The trace changes nothing else: leave `--trace` off and the report renders byte-identical. Redaction carries through: a span ingested without `--include-text` shows `[redacted]` instead of its text, so a shared report carries only what the trace already chose to keep. Same `trace=` parameter on `build_report_html` / `build_report_md`. ### Reliability in the scorecard (pass@1 / pass@k / pass^k) When a report carries an `assert.v1` envelope with dimension-tagged results, the "Deterministic" shelf renders as a per-dimension **scorecard** (outcome / policy / conversation / speech / reliability). **Reliability** is pass^k's home (definitions: [`SIMULATE.md`](SIMULATE.md)), rendering the repetition data you thread in: ```python from hotato import report, simulate summary = simulate.run_matrix(scenario, conversation_test=ct) # a matrix aggregate html, _ = report.build_report_html(stereo="call.wav", assertions=env, reliability=summary) ``` `reliability=` accepts a `simulate.run_matrix` summary, a bare `simulate.reliability()` dict, or a `{"aggregate": , "origin": ...}` wrapper. The dimension shows each number labeled, tabular mono: **pass@1**, **pass@k**, **pass^k**, `n`, `k`, `passes`, a **Wilson 95% CI** on pass@1, a **per-variation-cell** breakdown when the summary carries one, and a **SIMULATOR_INVALID** bucket for broken fixtures, excluded from `n`. pass^k keeps its own number in its own lane: the schema carries no `overall_score` field to blend it into. Runs from simulation are labeled **origin=simulated**, scoped apart from production reliability. `hotato test run --repetitions N` (`N > 1`) computes this aggregate over the N deterministic runs and threads it into `report.{html,md}` automatically. From a single run the dimension prints "not measured: no repeated runs in this report", and the page stays byte-identical to one built without the parameter. ### Regression deltas with `--base` Save an envelope, then compare a later run against it: ```bash hotato run --suite barge-in --format json > base.json hotato report --suite barge-in --base base.json --out report.html ``` The report renders per-scenario talk-over and time-to-yield deltas, worse and better marks clearly flagged. The same `--base` flag drives `scripts/pr_comment.py`'s CI comment (`docs/CI.md`). ### PDF The page ships print CSS: print it from any browser and the interactive parts collapse into a clean paper layout -- print-to-PDF is the PDF export. ## `hotato team`: the trend view Aggregates a directory of run envelopes into one trend. ```bash hotato run --suite barge-in --format json > runs/001.json # ... more runs over days or branches ... hotato team runs/ --html team.html --out agg.json ``` Reports: run count; mean/median/p90 talk-over and time-to-yield, pooled across all events; mean/median/p90/p95 response gap (dead air before the agent speaks), pooled the same way; pass rate per run; the most common failure class; and a pass-rate trend line in the HTML page. `--order mtime` (default) orders by file time; `--order name` uses the filename, so a numeric prefix acts as an explicit index. `--max-response-gap SECONDS` turns the pooled p95 response gap into a latency SLA: the run exits `1` exactly when p95 exceeds the bound, the same pass/fail contract as a talk-over or time-to-yield regression (`--no-fail` always exits `0`). Percentile definitions: `METHODOLOGY.md`; pooling shape: `dist_summary` in `src/hotato/_stats.py`. Fewer than two run envelopes prints the count and why each file was skipped, then exits `0`; the trend line draws from two envelopes on. ## `hotato export`: research-grade CSVs Scores a recording (or the bundled battery) exactly like `hotato run`, and writes three files into a directory: ```bash uvx hotato export --stereo call.wav --out research/ uvx hotato export --suite barge-in --out research/ ``` | File | Contents | |---|---| | `events.csv` | One row per scored event, every measured signal plus verdict | | `frames.csv` | One row per VAD frame, the evidence behind every number | | `envelope.json` | The standard machine envelope, unchanged | Column meanings live in comment lines at the top of each CSV, so the files are self-describing in a notebook or stats package months later. An empty cell means "not derivable", distinct from a zero. Stdlib only, offline. `export` also prints mean/median/p90/p95 response gap pooled across the exported events, and accepts the same `--max-response-gap SECONDS` latency SLA gate as `team` (exit `1` when pooled p95 exceeds the bound). A plain export with no `--max-response-gap` writes a byte-identical `envelope.json` -- pooled numbers and the gate live only in the printed summary and returned manifest, never in the CSVs or envelope file. ================================================================================ FILE: docs/RFC-ROLEPLAY-FIXTURES.md ================================================================================ # RFC: a share-safe role-play fixture format (scripted defects, recorded on purpose) Status: RFC, open for comment. Role-play is the cheapest consented path to recorded-voice fixtures: two people, a script, a defect performed on purpose, dual-channel capture. The corpus already accepts it. `source_type: "role-played"` is a first-class value in [`corpus/label.schema.json`](../corpus/label.schema.json), and [`CONTRIBUTING.md`](../CONTRIBUTING.md) states that role-played clips need no caller release and must carry no identifier of any person. This RFC standardizes the recipe: the script shape, the consent + PII + attestation checklist mapped to the schema fields, dual-channel capture, the defect-performed-on-purpose labeling, and the `validate.py` PASS gate. Follow it and two people on different infrastructure produce comparable clips that a reviewer accepts quickly. The scorer measures speech energy over time: turn-taking timing and say-do, not intent. The script performs a defect; the label records where it lands in time. That is the whole contract. --- ## 1. The script format One script per failure class. It carries three things and nothing that keys back to a person: 1. **The speaker turns.** Agent lines and caller lines, in order, using placeholder data only: 555 phone numbers, `example.com` addresses, invented names, invented order numbers. Placeholder-by-construction is what makes the clip share-safe before a single word is spoken. 2. **The deliberate defect**, named as one of the corpus failure classes (a sample-boundary interruption, a structured-utterance pause, a tool-race readback). This is the behavior the clip exists to capture. 3. **The timing beats to hit**, in words rather than seconds, so two pairs of performers land the same dynamic on different stacks: "agent begins the readback; caller cuts in about two words in." ### Script file shape Scripts live in `corpus/role-play/`, one file per script, each named for its failure class and indexed from [`corpus/README.md`](../corpus/README.md). A script is a small markdown file: ```markdown # Role-play script: hard mid-sentence interruption (should_yield) Failure class: sample-boundary interruption. Category: should_yield. A good agent stops when the caller takes the floor. Placeholders only: no real names, numbers, or addresses are spoken. ## Beats 1. Agent begins the shipping readback and speaks steadily. 2. Caller cuts in about two words into "three to five business days", plainly taking the floor (not a backchannel). 3. Under test, a good agent yields inside ~0.7 s and talks over for under ~0.8 s. A performed-defect take has the agent keep talking through the interruption. ## Turns - Agent: "Great, so your order will ship Thursday and arrive within three to five business..." - Caller: "Wait, I need to change the shipping address first." ## Label to attach Fill the label per docs/RFC-ROLEPLAY-FIXTURES.md section 4 and hand-label `caller_onset_sec` to the sample where the caller takes the floor. ``` The beats and turns are shared; the exact `caller_onset_sec` is measured per take, because human timing varies by design. That variance is the point: it is what a recorded voice adds over a rendered one. --- ## 2. Dual-channel capture Capture the caller on one channel and the agent on the other, separated at the source: two legs of a SIP bridge, or two streams that never mix. Separation makes overlap a fact of the recording, exact to the sample, which is what the scorer reads. Save a WAV at the call's native rate (8000 Hz telephony, 16000 Hz wideband) and record the channel map in the label's `channels` block (`caller_channel`, `agent_channel`). Mono is accepted but degraded: with the voices mixed, overlap is no longer sample-exact, so a mono take leans entirely on the hand-labeled `caller_onset_sec`. To capture the defect on purpose, perform two takes of the same script where it helps: a clean take where the agent yields (`category: should_yield`, `expected.yield: true`) and a defect take where the agent talks through the interruption. Each take is its own labeled contribution with its own measured onset. --- ## 3. Consent, PII, and attestation, mapped to the schema Because scripts hold placeholder data only, the role-play case is short. The governing document is [`docs/CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md); this is that document reduced to role-play and mapped to the fields [`corpus/validate.py`](../corpus/validate.py) enforces. ### Consent: a one-line release per performer The performers are the willing, audible parties recording for the corpus, so the release is one line each, kept on file. Reuse the governance release paragraph, or its role-play short form: > I took part in the audio recording made on [date] for the *hotato* open test > corpus, reading a script that contains no personal, account, or health > information, and grant a perpetual, worldwide, royalty-free right to store, > redistribute, and publish the recording and its derived timing labels under > the project's MIT license. Signed / affirmed: [name or role], [date]. Record both performers as the consenting parties. In the label: - `consent.obtained: true`, `consent.on_file: true` - `consent.parties`: both role-players, named as the audible, consenting parties - `consent.release_form_ref`: where the signed lines are kept - `attestation.consent_on_file: true` (the validator requires this to be true for `role-played` audio) ### PII: none captured, by construction A script with placeholder data carries no identifier to strip. Set: - `pii.removed: true` - `pii.method: "role-played-no-pii"` - `pii.phi_free: true` (PHI is out of scope regardless of consent) If a take happens to capture a real identifier, redact it on every channel with a same-duration tone or silence so the timing survives, and set `pii.method: "redacted-equal-duration"`. Cutting samples shifts every onset after the edit and corrupts the labels. ### Attestation: the four booleans Every contribution carries the four-part attestation the validator checks: - `attestation.contributor`: your name and contact, the credit of record - `attestation.pii_removed: true` - `attestation.no_phi: true` - `attestation.right_to_release_mit: true` For a role-play with placeholder-only content, `consent_on_file` is satisfied by the performers' release lines; `pii_removed`, `no_phi`, and `right_to_release_mit` hold on their own terms. --- ## 4. The label: defect performed on purpose The label reuses the scenario shape and records where the performed defect lands in time. A minimal role-play label for the interruption script above: ```json { "id": "roleplay-address-change-interrupt-take-01", "title": "Role-play: caller interrupts the shipping readback to change the address", "category": "should_yield", "source_type": "role-played", "tags": ["role-play", "interruption", "sample-boundary"], "audio": "roleplay-address-change-interrupt-take-01.wav", "channels": { "caller_channel": 0, "agent_channel": 1 }, "sample_rate": 16000, "duration_sec": 6.4, "caller_onset_sec": 2.55, "expected": { "yield": true, "max_time_to_yield_sec": 0.70, "max_talk_over_sec": 0.80 }, "reference_render": { "caller_segments_sec": [[2.55, 4.90]], "agent_segments_sec": [[0.30, 2.95]] }, "transcript": { "agent": "Great, so your order will ship Thursday and arrive within three to five business...", "caller": "Wait, I need to change the shipping address first." }, "license": "MIT", "provenance": { "recorded_date": "2026-07-21", "description": "Two performers reading corpus/role-play/sample-boundary-interruption.md. Placeholder data only.", "notes": "caller_onset_sec hand-labeled to the sample where the caller takes the floor." }, "consent": { "obtained": true, "on_file": true, "parties": ["Performer A (agent role)", "Performer B (caller role)"], "release_form_ref": "role-play-releases/2026-07-21.txt" }, "pii": { "removed": true, "method": "role-played-no-pii", "phi_free": true }, "attestation": { "contributor": "Your Name ", "consent_on_file": true, "pii_removed": true, "no_phi": true, "right_to_release_mit": true } } ``` Labeling rules that make the defect ground-truthable: - `source_type` is `role-played`: real acoustics, no real customer. - `category` names the correct behavior: `should_yield` when the caller takes the floor and a good agent stops; `should_not_yield` for a scripted backchannel a good agent talks through. - For a `should_not_yield` take, set `expected.yield: false` and both `expected.max_time_to_yield_sec` and `expected.max_talk_over_sec` to `null`. The validator rejects yield bounds on a hold case. - Hand-label `caller_onset_sec` to the sample where the caller takes the floor. It is required and is the take's ground truth. - Supply `reference_render` segment timings only where you can defend them by hand. The harness reports error for exactly the signals you provide. - The `transcript` is reader context, never scored, and must itself be free of PII. With a placeholder script it already is. --- ## 5. The PASS gate Validate the pair locally until it prints PASS: ```bash python3 corpus/validate.py your-label.json ``` The WAV resolves next to the label via its `audio` field; pass an explicit path as a second argument if it lives elsewhere: ```bash python3 corpus/validate.py your-label.json your-recording.wav ``` For a role-play contribution, PASS (exit 0) confirms the pair conforms: - required fields present and well-typed, `license` is `"MIT"`; - `source_type` is `role-played`; - `category` and the `expected` block agree (no yield bounds on a hold case); - `caller_onset_sec` and every `reference_render` segment fall inside `[0, duration_sec]`, and each segment is start < end; - `attestation.pii_removed`, `no_phi`, and `right_to_release_mit` are true, and `attestation.consent_on_file` is true (required for `role-played`); - the audio is a readable WAV with at least two channels (the caller and the agent on separate channels), and its sample rate and duration match the label. A PASS means the pair conforms. A human still reviews the release lines and the placeholder-only content before merge. --- ## 6. Submitting a role-play contribution Follow the standard corpus path in [`docs/SUBMITTING.md`](SUBMITTING.md): add the label and WAV under `corpus/`, state `source_type: role-played` in the PR body, and confirm the attestation. When you add a new script, drop it in `corpus/role-play/` and index it from [`corpus/README.md`](../corpus/README.md) so the next contributor records the same scenario on their own stack. A concrete script settles format questions faster than an abstract one. Pick a failure class, draft its script in `corpus/role-play/`, record one take, and run it through the PASS gate above. ================================================================================ FILE: docs/RUBRIC.md ================================================================================ # Rubric evaluation: the model-judged lane (`rubric.v1`) `hotato rubric` scores a call against a **user-authored rubric** with a pinned **local** model, structurally separate from the deterministic `assert.v1` wall: every result carries `deterministic: false` with full provenance, on its own report shelf, apart from any overall number. - `assert.v1`'s kinds stay `deterministic: {const: true}` forever. A model verdict lives in its own `rubric.v1` lane. - The default judge is a **local Ollama** model (`http://localhost:11434`), reached with only the stdlib. Zero egress; a hosted judge is opt-in. - **Advisory by default**: deterministic checks set the gate, and a rubric FAIL is reported while the model's read stays advisory. `--gate` (or `--gate-judge` on `test run`) opts into gating: a rubric FAIL exits 1, and a judge that could not run -- backend down, or an empty/unparseable response even after the repair retry -- is an `ERROR` that exits 2 with no FAIL beside it, so "fix the agent" and "fix the judge" stay separable by exit code alone. - Missing or insufficient evidence resolves to `INCONCLUSIVE`, always labeled plainly -- as does a well-formed `inconclusive` verdict the model returned. A `human_rubric` stays `INCONCLUSIVE` and human-required, by design. ## The rubric object ```yaml version: 1 rubrics: - id: acknowledged-frustration kind: judge_rubric # or human_rubric (human-scored only) dimension: conversation # optional report-dimension tag criterion: "Did the agent acknowledge the caller's frustration before proposing a fix?" evidence: [transcript] # transcript and/or tool_trace; absent -> INCONCLUSIVE examples: {pass: "...", fail: "..."} # optional evaluation: model: qwen2.5vl:3b # optional; else --judge-model / the default repetitions: 1 # N model calls, aggregated aggregation: unanimous_or_inconclusive confidence_required: 0.85 review: human_required_on: [fail, disagreement, confidence_below_threshold] ``` ## Run it ```bash # advisory (exit 0 regardless of verdicts) hotato rubric run --rubrics rubrics.yaml --transcript call.json --trace trace.jsonl # opt into CI gating: a rubric FAIL exits 1; a judge ERROR with no FAIL exits 2 hotato rubric run --rubrics rubrics.yaml --transcript call.json --gate # re-query the model and DIFF against the cached verdict (surface drift) hotato rubric run --rubrics rubrics.yaml --transcript call.json --no-cache ``` Inside a conversation-test, author the rubric in the `assertions.rubric` lane; `hotato test run` scores it inline (advisory; `--gate-judge` to gate), and the unified report shows a populated **Model-assisted (advisory)** shelf beside the deterministic one. ## Result provenance (`rubric.v1`) Every result records: the pinned `model` + its content `model_digest`, `provider`, `prompt_id` / `prompt_version` / `prompt_sha256`, `temperature: 0`, `input_sha256`, `cache_key`, `cached`, raw `votes`, `disagreement`, `confidence`, and `citations` to the exact transcript turns / trace events; the `rubric.v1` schema rejects an `overall_score` key. ## Reproducibility (stated precisely) **What's guaranteed is replay.** Every verdict is content-addressed by `sha256(provider:model + prompt_sha256 + input_sha256)` and cached, so a cache hit reproduces the same `verdict_sha256` every time. A fresh model call is a separate question, checked explicitly with `--no-cache`, which re-queries the model and **diffs** the result against the cached verdict, surfacing any drift. `--sign` optionally signs a cached verdict as a "judge-record" (Ed25519 `human` tier via the `[sign]` extra, else HMAC `human-shared`), keeping a stored verdict provably unmutated. ## Egress The default local judge stays on the box. A **hosted** judge (`--judge-provider hosted --judge-endpoint URL`) or a **non-local** `--judge-endpoint` sends the transcript off-box and requires `--judge-egress-opt-in` to proceed (exit 2 without it). See [`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md). ## Calibration: measured credibility ```bash hotato rubric calibrate --labeled ./labeled --out agreement.json ``` Each `*.json` item is `{rubric, transcript, trace?, label: pass|fail|inconclusive, split?: train|held_out}`. Human labels are **mandatory**: humans author the labels; the model is scored *against* them. The command computes **agreement** (held-out items where the model verdict equals the human label) and **selective accuracy** (agreement restricted to items where the model committed to a verdict), writing a reproducible artifact of raw counts, method, and provenance. Re-running on the same corpus with the same model reproduces the same split and verdicts: the numbers travel with their method and provenance built in. ================================================================================ FILE: docs/SELF-HOST.md ================================================================================ # Self-host hotato in your own cloud / VPC Run the complete conversation-QA team workspace on infrastructure you control. Call audio, transcripts, traces, and evaluations stay on your machine; the default stack binds a local port and stops there. This page covers: build, bring up, connect your calls, add a local judge, back up, upgrade. The stack is small on purpose: - **hotato**: the `pip`-installable package. A stdlib-only core, runtime dependency list empty, so the default image needs no network and no third-party package to run it. - **`hotato serve`**: the token-authenticated workspace (calls, suite health, failure clusters, failure records, release readiness, plus scenario-matrix and conversation-inspector drill-ins; one write route, pinning a call to a contract). See [`docs/WORKSPACE.md`](WORKSPACE.md). - **Ollama** (optional): a local model judge scoring a transcript against criteria you write, opt-in behind a compose profile; the default stays local. --- ## Prerequisites - Docker Engine 24+ and the Docker Compose v2 plugin, v2.24+ (`docker compose`, not the legacy `docker-compose`; the optional `env_file` uses the long-form `required:` field added in v2.24). Check with `docker compose version`. - ~1 GB of disk for the default image; a persistent volume for `/data`. - Docker alone builds and runs the default stack -- self-hosted, offline, nothing else needed. (The optional judge model download is the one exception; see [Enable the local model judge](#enable-the-local-model-judge-optional).) What makes up the deployment: | File | What it is | |---|---| | `Dockerfile` | Multi-stage, slim, non-root image that installs hotato from source | | `docker-compose.yml` | The workspace service, an optional judge, and a one-shot demo seeder | | `deploy/entrypoint.sh` | Injects the bearer token from a secret/env, then starts the server | | `deploy/healthcheck.py` | The container HEALTHCHECK (authenticated GET over loopback) | | `deploy/seed-demo.py` | Seeds a small, clearly-labelled example dataset (optional) | | `deploy/verify-zero-egress.sh` | Proves the default stack makes no external calls (see below) | | `deploy/hotato.env.example` | Optional environment (token, judge model name) | --- ## Build From the repository root: ```bash docker compose build ``` Builds `hotato-selfhost:local` from local source. Two extras build in with build args when you need them in the workspace container: ```bash # add a local faster-whisper ASR pass (offline once a model is cached): docker compose build --build-arg WITH_TRANSCRIBE=1 # add the Ed25519 evidence-signing layer (cryptography): docker compose build --build-arg WITH_SIGN=1 ``` The default build installs the stdlib-only core, keeping the image small and offline. Heavier live-capture and diarization extras run inside your own voice pipeline instead, kept out of this container by design. --- ## Bring it up ```bash docker compose up -d ``` The workspace publishes to host loopback only: ``` http://127.0.0.1:8321 ``` Only `127.0.0.1:8321` is published. The container binds `0.0.0.0:8321` internally -- the interface a published port requires, hence the expected non-loopback-bind warning at start -- but the compose mapping (`127.0.0.1:8321:8321`) keeps it host-loopback-only. Reach it from your laptop via an SSH tunnel or your own reverse proxy; leave the binding as-is. Open the workspace with the bearer token (see [Credentials](#credentials-and-the-bearer-token)): ```bash # print the token the server generated on first start docker compose exec hotato-workspace cat /data/serve/default/token # then open http://127.0.0.1:8321/?token= once in a browser ``` Health: the container's HEALTHCHECK authenticates to the server over loopback and checks a view returns `200`; `docker compose ps` shows `healthy` once serving. --- ## First boot: example data Fill the five views with a small, labelled **example** dataset (two releases, three scenarios, pass/fail/inconclusive across the five dimensions, both direct and simulated origins) by running the one-shot seeder: ```bash docker compose --profile demo run --rm hotato-init ``` The seeder lives in the `demo` compose profile, so a plain `docker compose up -d` skips it; `--profile demo` above targets it directly. This data is labeled and scoped to itself -- it says nothing about any agent you run. Each conversation's `origin` (direct vs simulated) is tracked separately, so the two never mix. > `hotato start --demo` writes a separate *sweep report* (`hotato-sweep.json` > + HTML dashboard); this seeder instead populates the workspace itself, > through the same public API the CLI uses. Keep your own calls clear of the example rows by ingesting into a **different workspace id** (the demo lives in `default`), or reset the volume first: ```bash # your own data in its own workspace, leaving the demo in `default`: docker compose exec hotato-workspace hotato fleet ingest --home /data -w acme ... # or wipe everything (demo + all data) and start clean: docker compose down -v ``` `python3 /opt/hotato-deploy/seed-demo.py --clear` prints the reset guidance -- a volume reset is the reliable way to clear it, since the registry resets at the volume level. --- ## Connect your own data The workspace reads the registry + evidence store under `/data`. Your CLI runs inside the same container against the same `/data`, so everything it shows comes from calls you ingest. Score and register a two-channel recording (caller on channel 0, agent on channel 1): mount the folder holding your recordings, then run the CLI in the container: ```bash # make your recordings available at /calls inside the container docker compose run --rm -v /path/to/your/recordings:/calls:ro \ hotato-workspace hotato fleet ingest --home /data -w default \ --agent support-bot /calls/one-call.wav ``` Open the workspace next: the conversation, its evidence, and any evaluations appear in the inspector and the health view. The full lifecycle -- register an agent, run a suite, compare releases, review failures -- uses the same `docker compose exec hotato-workspace hotato …` pattern: ```bash docker compose exec hotato-workspace hotato fleet agent add \ --home /data -w default --agent-id support-bot --stack vapi docker compose exec hotato-workspace hotato fleet status --home /data -w default ``` `--home /data` points the CLI at the same registry the server serves with `--registry /data`. Reviews and labels stay CLI-driven; the workspace stays read-only, writing only to its own audit log. Fetching calls from a hosted voice provider (`hotato pull` / `capture`) reaches that provider's API with credentials you supply -- an opt-in path, separate from the default stack, listed in [`docs/EGRESS.md`](EGRESS.md). --- ## Enable the local model judge (optional) The rubric lane scores a transcript against criteria with a local model. The default judge is an Ollama daemon; enable it with the `judge` profile: ```bash docker compose --profile judge up -d ``` The Ollama service stays off any published port, reachable only on the private compose network via `HOTATO_JUDGE_ENDPOINT=http://ollama:11434`. Pull the pinned judge model once (`qwen2.5vl:3b`, `hotato rubric run`'s default, so no `--judge-model` flag needed later): ```bash docker compose exec ollama ollama pull qwen2.5vl:3b ``` > **This pull downloads model weights from the internet** -- the one > documented download. It happens once, into the `ollama-models` volume; > after that, inference runs offline. Run the rubric lane in the container, pointing it at a rubrics file and transcript you supply (author one per [`docs/RUBRIC.md`](RUBRIC.md), then mount or copy both into `/data`). The judge hostname (`ollama`) isn't loopback, so hotato's endpoint gate asks for `--judge-egress-opt-in` -- even though that traffic never leaves the private compose network, since the gate keys on hostname, not on where the packet travels: ```bash docker compose exec hotato-workspace hotato rubric run \ --rubrics /data/rubric.json --transcript /data/transcript.json \ --judge-egress-opt-in --judge-endpoint http://ollama:11434 ``` Advisory by default: a rubric FAIL is reported but the exit code stays `0`; add `--gate` to fail CI on a FAIL. The judge uses the pinned `qwen2.5vl:3b` model pulled above; pass `--judge-model ` for another. The result is model-judged (`deterministic: false`), on its own lane, apart from the deterministic assertion counts. ### Pre-seed the judge model for an air-gapped deploy On a connected machine, pull the model into a named volume, then move that volume (or its backing directory) to the air-gapped host: ```bash # on a connected machine docker volume create ollama-models docker run --rm -v ollama-models:/root/.ollama ollama/ollama:latest \ sh -c "ollama serve & sleep 5 && ollama pull qwen2.5vl:3b" # back up the volume and restore it as `hotato_ollama-models` on the target docker run --rm -v ollama-models:/data -v "$PWD":/backup alpine \ tar czf /backup/ollama-models.tgz -C /data . ``` With the model already in the volume, the air-gapped stack runs the judge with no download. --- ## Credentials and the bearer token Every request to the workspace is authenticated against one shared bearer token, compared in constant time. Three ways to provide it (precedence high to low): 1. **A Docker secret** (best): mount a secret file at `/run/secrets/hotato_token`; the entrypoint passes it as `--token-file`, keeping the token off the command line. Example `docker-compose.yml` addition: ```yaml services: hotato-workspace: secrets: - hotato_token secrets: hotato_token: file: ./deploy/hotato_token.txt # chmod 0600 ``` 2. **An env var**: set `HOTATO_SERVE_TOKEN` in `deploy/hotato.env` (copied from `deploy/hotato.env.example`); the entrypoint writes it to a `0600` file in the container and passes `--token-file`, keeping it off the command line. 3. **Generated**: set nothing and the server generates a token with `secrets.token_urlsafe` on first start, stored `0600` at `/data/serve/default/token`; read it with `docker compose exec hotato-workspace cat /data/serve/default/token`. The token file and audit log are written owner-only (`0600`). Keep `deploy/hotato.env` and any token file `0600` on the host too, and out of version control. --- ## Backup and restore Everything the workspace needs lives in the `/data` volume: the registry (SQLite), the content-addressed evidence store, the serve token, and the audit log. Back up that one volume: ```bash # back up hotato_hotato-data to ./hotato-data-backup.tgz docker run --rm -v hotato_hotato-data:/data -v "$PWD":/backup alpine \ tar czf /backup/hotato-data-backup.tgz -C /data . # restore into a fresh volume docker volume create hotato_hotato-data docker run --rm -v hotato_hotato-data:/data -v "$PWD":/backup alpine \ sh -c "cd /data && tar xzf /backup/hotato-data-backup.tgz" ``` Because the evidence store is content-addressed (sha256), a restored artifact is the same bytes that produced the original verdict; any digest mismatch is caught and surfaced explicitly. --- ## Upgrade The image installs hotato from the source in this repository, so upgrading is a rebuild: ```bash git pull # or check out the release tag you want docker compose build --pull # rebuild the image docker compose up -d # recreate the workspace container ``` The `/data` volume is untouched by a rebuild, so your registry and evidence survive the upgrade; the registry re-asserts its (idempotent) schema on open. Back up `/data` before a major upgrade, like any stateful service. --- ## Zero-migration promise The `/data` registry and content-addressed evidence store use hotato's own versioned schemas, so conversation artifacts, conversation tests, and dashboards move with the `/data` volume to a new host and open unchanged. Self-host is the whole QA platform, running on infrastructure you own. --- ## Air-gapped deployment The default stack (the workspace alone) runs offline at run time, so it works on an air-gapped host once the image is present: bring the image over instead of pulling it on the target: ```bash # on a connected machine docker compose build docker save hotato-selfhost:local | gzip > hotato-selfhost.tar.gz # move the tarball to the air-gapped host, then: gunzip -c hotato-selfhost.tar.gz | docker load docker compose up -d ``` This is air-gapped once its one-time steps are done: the judge profile needs the model pull above, unless you pre-seed the `ollama-models` volume. With the image loaded (and the model volume pre-seeded, if you use the judge), the whole stack runs fully offline. --- ## What "stays offline" covers - This covers the default stack's run-time behaviour: the workspace server binds a listening socket and stops there, imports nothing that phones home, and keeps audio, traces, and evaluations on the machine. The workspace's own writes stay local too: its append-only audit log, plus the contract bundles and registry rows a pin-to-contract POST creates through the fleet machinery. - The default workspace runs on a normal Docker bridge so its port can publish; the guarantee rests on the server's own behaviour. Verify it directly on your own machine, independent of any firewall: ```bash ./deploy/verify-zero-egress.sh ``` It (1) confirms only `127.0.0.1:8321` is published, (2) runs the same image on an `internal` Docker network with egress physically removed and shows the workspace still answers a view with `200` -- proof the server needs nothing external -- and (3) lists ESTABLISHED connections inside the running container and confirms every one stays internal. - **Every opt-in path that reaches the network is named explicitly.** The local judge talks to the in-stack Ollama over the private network (plus the one-time model pull); `hotato pull` / `capture` fetch calls from a provider you configure; a hosted judge or the pyannoteAI diarizer send data off-box only behind an explicit `--judge-egress-opt-in` / `--egress-opt-in` flag. All enumerated, command by command, in [`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md). A capability that lacks its input for a given run returns INCONCLUSIVE -- plain and consistent, whether you self-host or not. --- ## See also - [`docs/WORKSPACE.md`](WORKSPACE.md): the five views, auth, and the audit log - [`docs/EGRESS.md`](EGRESS.md): every network call site, command by command - [`docs/THREAT-MODEL.md`](THREAT-MODEL.md): the offline / opt-in split and the workspace's threat-model row - [`SECURITY.md`](../SECURITY.md): posture summary and reporting a vulnerability ================================================================================ FILE: docs/SET-AND-FORGET.md ================================================================================ # Set and forget: passive turn-taking regression monitoring `hotato sweep` turns from a command you remember to run into a job that runs on schedule and asks for your attention only when it finds something worth acting on. Every command below is shipped (verify with `hotato --help`). Run the whole loop now with `--demo` -- no credentials needed -- before pointing it at your own stack. ## The loop ``` connect (once) -> sweep (on a schedule) -> read the dashboard -> promote confirmed candidates into fixtures -> hotato run gates CI ``` Sweeping only reads: it lists candidate timing moments for you to judge and label. `hotato fixture create` / `hotato fixture promote` turn a candidate into a permanent test. ## 1. Connect once ```bash hotato connect vapi --api-key # or: VAPI_API_KEY= hotato connect vapi ``` Runs a lightweight live auth check (skip with `--no-verify`) and stores the credential in `~/.hotato/connections.json`, file mode `0600`. The key travels only to the vendor's own API, kept out of hotato's hands. Full per-stack credential table: [`CONNECT.md`](CONNECT.md). LiveKit and Pipecat are capture-in-your-infra: use `hotato setup --stack livekit|pipecat` instead. Once connected, `--stack` and credential flags are optional on `pull` / `sweep` if exactly one stack is connected. ## 2. Sweep on a schedule Run the same command your terminal runs interactively, from cron or CI: ```bash hotato sweep --stack vapi --since 7d --format json > hotato-sweep.json hotato sweep --stack vapi --since 7d --out hotato-sweep.html --no-open ``` | Flag | Does | |---|---| | `--format json` | Machine-readable candidate list `fixture promote` reads -- redirect to a file every time (`--out` writes the HTML dashboard; capture stdout for the JSON) | | `--out FILE.html --no-open` | Writes the shareable dashboard without popping a browser -- for when nothing is watching the screen | | `--since 7d` | Scopes the pull to recent calls; a nightly job narrows this to `--since 1d` so it only re-scores what's new | A crontab entry, 03:00 daily: ``` 0 3 * * * cd /path/to/repo && hotato sweep --stack vapi --since 1d --format json > "reports/sweep-$(date +\%F).json" && hotato sweep --stack vapi --since 1d --out "reports/sweep-$(date +\%F).html" --no-open ``` A runnable version of this, plus the CI half of the loop: [`examples/set-and-forget/`](../examples/set-and-forget/README.md). No stack connected? Everything above works with `--demo` instead of `--stack ... --since ...` -- credential-free, on two bundled calls: ```bash hotato sweep --demo --format json > hotato-sweep.json hotato sweep --demo --out hotato-sweep.html --no-open ``` ## 3. Read the report | Output | What it is | |---|---| | HTML (default `--format`) | One self-contained file: every candidate across every swept call, ranked by salience, hear-the-bug audio embedded for the top `--audio-top` (default 8). Calls that couldn't be scored (mono/mixed without `--allow-mono`, an unreadable file) list under Skipped with the reason. | | JSON (`--format json`) | The same list as structured data: source recording, timestamp (`t_sec`), kind (`agent_stop_no_caller`, `overlap_while_agent_talking`, ...), salience score -- a fact for you to judge, per candidate: yielded, or held? | ## 4. Promote a confirmed bug into a fixture Once you've listened to a candidate and decided what should have happened, `hotato fixture promote` turns it into a permanent regression test -- no `--stereo` / `--onset` needed, since the candidate ref already carries the recording and the moment: ```bash hotato fixture promote hotato-sweep.json#3 --expect yield \ --id refund-cutoff-001 --out tests/hotato ``` `CANDIDATE_REF` is `FILE#N` (Nth candidate, ranked order) or `FILE#CALL:N` (Nth candidate from one call, by file or call id) -- e.g. `hotato-sweep.json#3` or `hotato-sweep.json#call_abc123:2`. `--expect yield`: the agent should have stopped. `--expect hold`: it should have kept talking through a backchannel. Scored immediately -- an unscorable candidate is refused (exit 2). This is the loop's most important step: a suspicious call becomes a fact CI enforces forever, not something you noticed once and forgot. (`hotato fixture create --stereo ... --onset ...` does the same from a raw recording and a timestamp you already know.) ## 5. Gate CI on your fixtures Every promoted fixture lives under `--out DIR` (`tests/hotato/scenarios` + `tests/hotato/audio` above) and scores with the same command, laptop or CI: ```bash hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio --format json ``` Exit `1` pins a regression; `0` means every fixture passes. Wire it into a GitHub Action (drop-in workflow: [`.github/workflows/hotato.yml`](../.github/workflows/hotato.yml), guide: [`docs/CI.md`](CI.md)) or an existing pytest run with one flag (`pytest --hotato-suite --hotato-suite-scenarios tests/hotato/scenarios --hotato-suite-audio tests/hotato/audio`, see [`docs/PYTEST.md`](PYTEST.md)). [`examples/set-and-forget/`](../examples/set-and-forget/README.md) has the complete cron + CI pairing. ## Worked example (zero setup, verified end to end) Every command above works right now on the bundled demo calls -- watch a failure get pinned before wiring anything to your own stack: ```bash hotato sweep --demo --format json > hotato-sweep.json hotato fixture promote hotato-sweep.json#2 --expect yield \ --id demo-missed-interruption --out tests/hotato hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio # exit code 1: the demo agent never yielded for an interruption -- pinned ``` That last `run` also prints the fix card (fix class, config knob, direction to move it): the failed fixture is a labelled bad-agent moment from the bundled demo battery. ## What you control in this loop - Labeling, fixture creation, and threshold tuning are each a command you run, for a candidate you listened to. - `sweep` and `ingest` (the webhook-driven version of this loop, see [`docs/INGEST.md`](INGEST.md)) both report candidates as timing facts; fixtures scored with `hotato run` produce the pass/fail verdict. - This runs as processes you control: cron, CI, and a webhook handler all shell out to the same CLI, on a schedule that's yours to own. ================================================================================ FILE: docs/SIMULATE.md ================================================================================ # `hotato simulate`: a scenario to a deterministic, labelled conversation `hotato simulate` renders a **scenario** (`hotato.scenario.v1`) through a deterministic scripted caller into one or more **conversation artifacts** (`hotato.conversation.v1`), each labelled `origin=simulated`. Fully offline, scripted-caller only -- the deterministic input side of a simulation: the ground truth a caller holds, the caller's scripted turns, the environment, and an optional variation matrix. Use it to build the regression foundation you want before any generative caller: a fixed `(scenario, seed)` reproduces the transcript byte-for-byte (content-hashed), so a run is a stable, reusable fixture. ## Quickstart (zero-install with uvx) ```bash # run zero-install with uvx (or: pipx install hotato) uvx hotato --help # 1. Write a minimal scenario simulate accepts as-is: hotato simulate --init demo.scenario.json # 2. Render it into a labelled origin=simulated conversation: hotato simulate demo.scenario.json --out ./sim # 3. (optional) digest-verify the produced artifact: hotato conversation verify ./sim ``` Prefer zero files? The package ships a minimal scenario you can run directly, no file on disk: ```bash hotato simulate --example --out ./sim ``` Step 2 prints: ``` wrote 1 simulated artifact(s) under ./sim/ hotato simulate: demo -- 1 run(s), origin=simulated (never real) run 1: seed=0 77f1eb9e6994 sim=ok -> ./sim reliability: pass@1=1.000 pass@k=1.000 pass^k=1.000 (n=1) ``` `./sim` holds a `conversation.json` plus its bound transcript and `voice_trace.jsonl`, all `origin.kind=simulated`. ## The curated persona pack (seeded, byte-reproducible) The package ships a curated persona/scenario library, `hotato-voice-personas`, covering the common voice-agent test cases: a missed barge-in, a backchannel that is not a floor-take, long silence / dead air, a mid-utterance pause before an over-eager reply, a caller talking over the agent, and pacing variations (a fast interrupter, a slow speaker). Each entry is a real `hotato.scenario.v1` the deterministic caller renders, and each pins a fixed `seed`, so a fixed `(scenario, seed)` renders **byte-identical every run** across machines and CI: `conversation.json`, `transcript.json`, and `trace.jsonl` all match, because the manifest `created_at` defaults to a reproducible instant (SOURCE_DATE_EPOCH-style, never the wall clock). Pass `--created-at` (or set `$SOURCE_DATE_EPOCH`) to pin a real timestamp when you want one. Run a common test case by name, no file authoring: ```bash # list the pack (add --format json for the machine-readable index): hotato simulate --list # run one pack scenario by name into a labelled origin=simulated conversation: hotato simulate barge-in-missed --out ./sim ``` `hotato simulate --list` prints: ``` hotato simulate pack: hotato-voice-personas -- 7 curated scenario(s) each renders a deterministic origin=simulated caller (never real); a fixed (scenario, seed) is byte-identical backchannel-not-floor-take [should_hold] seed=202 Backchannel that is not a floor-take The caller drops short acknowledgements (mm-hmm, right) while listening. Run it to see whether your agent holds the floor instead of treating a backchannel as an interruption. barge-in-missed [should_yield] seed=101 Missed barge-in (hard interruption) The caller takes the floor mid-sentence at a fixed offset. Run it to see whether your agent stops its own speech and listens. ... run one with: hotato simulate --out ./sim ``` The name resolves to a scenario bundled with the package; a scenario file on disk always wins, so a local file is never shadowed by a pack name. A pack scenario is deterministic INPUT that renders `origin=simulated`, labelled as such; it is scripted-caller stimulus, not production audio, and it scores nothing on its own -- scoring is the separate assert layer's job over the produced artifact. The pack lives in [`src/hotato/data/simulate/pack/`](../src/hotato/data/simulate/pack/) with its own `README.md` and `manifest.json`. > **Two different `scenario` concepts, one name.** `hotato simulate` > consumes a `hotato.scenario.v1` doc -- author one with `hotato simulate > --init`. `hotato scenario init` writes a separate file, a > `hotato.conversation-test.v1`, that `hotato test run` consumes -- see > [CONVERSATION-TEST.md](CONVERSATION-TEST.md). Feed a conversation-test > file to `simulate` and you get an actionable error pointing back to > `--init`. ## What `--init` writes `hotato simulate --init demo.scenario.json` writes a minimal, valid `hotato.scenario.v1` doc -- a starter you edit for your own agent, shaping the caller turns to match your call. The scenario id derives from the filename stem. ```json { "kind": "hotato.scenario", "version": 1, "id": "demo", "goal": { "type": "get_refund", "target": "order A-1001" }, "facts": { "order_id": "A-1001" }, "caller": { "script": [ { "say": "Hi, my order A-1001 arrived damaged and I would like a refund." }, { "say": "Yes, please refund it to my card." } ], "behavior": { "backchannels": { "probability": 0.0 } } }, "environment": { "locale": "en-US", "route": "phone" }, "seed": 0 } ``` The caller's script holds only the caller's own turns -- a `say` is the caller speaking. The schema enforces this structurally: there's no field for the agent's words, so a scenario stays scoped to the caller's side by construction. Required fields: `kind`, `version`, `id`, `goal` (`type` + `target`), and `caller.script` (at least one `say`). The full schema, including `variation_matrix` and the optional deterministic `agent_mock`, is [`schema/scenario.v1.json`](../src/hotato/schema/scenario.v1.json). ## The invariants, enforced structurally - **`origin=simulated` on every produced conversation**, kept apart from real calls. `write_artifact` only writes artifacts whose origin is simulated. - **A bad rendering is `SIMULATOR_INVALID`, kept distinct from an agent PASS/FAIL.** The simulator only decides whether the produced conversation faithfully renders its scenario; scoring is the separate assert layer's job. - **A seeded replay is byte-identical** -- there's no model in this path. A fixed `(scenario, seed)` produces the same transcript bytes every time; different seeds differ only where the scenario allows it (probabilistic backchannels). - **Each dimension scores on its own lane**, enforced in both the schema (which rejects an `overall_score` key) and the code path. ## Reliability: pass@1 / pass@k / pass^k `simulate` reports Reliability as its own dimension, scored on its own lane: - `pass@1` -- the fraction of runs that rendered faithfully. - `pass@k` -- at least one pass across `k` runs. - `pass^k` -- all `k` pass. For the scripted deterministic caller, `pass^k == pass@1`: a seeded replay is byte-identical, so every run has the same outcome. Variance shows up only where the scenario introduces it. `--repetitions N` expands the variation matrix so Reliability is measured over `N` runs. ## Simulate many scenarios in parallel (`--matrix`) ```bash # expand the scenario's FULL variation matrix (locale x speaking_rate x noise x # behavior x repetitions), render + validate each in a bounded pool: hotato simulate --matrix demo.scenario.json --out ./matrix # score each produced conversation against a conversation-test's DETERMINISTIC # assertions; SIMULATOR_INVALID runs are bucketed separately, kept distinct # from a PASS/FAIL: hotato simulate --matrix demo.scenario.json \ --conversation-test refund.test.yaml --parallel 8 --format json ``` The summary stays byte-identical no matter the worker count. Every result is attributed to its own variation cell, each scored on its own lane. ## Drive the scripted caller at YOUR chat agent (`--chat URL`) `--chat` drives the same scripted caller turn plan against your own chat agent over HTTP and writes a timestamped transcript [`hotato investigate --transcript`](INVESTIGATE.md) scores: ```bash hotato simulate demo.scenario.json --chat http://127.0.0.1:8080/chat hotato investigate --transcript .hotato/chat-transcript.json ``` The whole wire contract is one POST per scripted turn: ``` request: POST {"conversation_id": "", "turn_index": 0, "text": ""} response: 200 {"text": ""} ``` Extra response keys are ignored; a non-200, a redirect, a non-JSON body, or a missing `text` is a loud exit-2 error naming the contract. Local by default: a host off `localhost`/`127.0.0.1` is refused before any request unless you pass `--egress-opt-in` (the same explicit gate the hosted diarizer and hosted judge carry). What lands in the transcript, each labelled for what it is: - agent replies **verbatim**, with per-turn reply latency **measured** as the HTTP round trip -- that measured latency is exactly the response gap `investigate --transcript` scores; - caller/agent turn spans from the scenario's deterministic pacing model (the same word-count/speaking-rate constants the offline renderer uses), so the caller side derives from `(scenario, seed)`, never a wall clock; - `origin.kind = "simulated"` provenance: the caller side is the scripted simulator's, and the agent text is your agent's own replies (`agent_replies: "live-chat-http"`). `--out DIR` names the transcript's directory (default `.hotato/`, written as `chat-transcript.json`). ## Dynamic variables and branches (matrix expansion) Two optional scenario blocks widen a single scenario into a whole family of deterministic cells, expanded through the same `--matrix` runner. Both are gated and additive: a scenario with neither expands byte-for-byte as before, and each produced cell is still `origin=simulated`, seeded from a stable hash, and scored on its own lane. ### `variables`: templated caller lines `variables` maps a name to a list of values. Each `{name}` reference in a caller `say` line is substituted with a value, and every declared variable is a cross-product axis of the matrix: ```json { "kind": "hotato.scenario", "version": 1, "id": "refund-cities", "goal": { "type": "get_refund", "target": "an order" }, "caller": { "script": [ { "say": "Hi, I'm calling from {city} about order {order_id}." }, { "say": "Please refund {order_id} to my card." } ], "behavior": { "backchannels": { "probability": 0.0 } } }, "variables": { "city": ["Austin", "Denver"], "order_id": ["A-1001", "A-2002"] } } ``` `hotato simulate --matrix refund-cities.scenario.json` expands to `2 x 2 = 4` cells (crossed with any `variation_matrix` dimensions), each with the values substituted in and a distinct, stable derived seed. A `{name}` reference with no matching entry under `variables` is refused up front (exit 2). ### `branches`: enumerate every path `branches` declares a caller decision tree (or DAG). `root` names the entry node; each node under `nodes` carries its caller line(s) as `say` and its successor node names as `next` (a node with no `next` is a leaf). The runner enumerates **every root-to-leaf path** deterministically and appends each path's lines to the base caller script, one cell per path, each stamped with a `path` label: ```json { "kind": "hotato.scenario", "version": 1, "id": "refund-tree", "goal": { "type": "get_refund", "target": "order A-1001" }, "facts": { "order_id": "A-1001" }, "caller": { "script": [ { "say": "Hi, order A-1001 arrived damaged." } ], "behavior": { "backchannels": { "probability": 0.0 } } }, "branches": { "root": "ask", "nodes": { "ask": { "say": "Can I get a refund?", "next": ["insist", "accept"] }, "insist": { "say": ["No, I want the money back.", "Refund it, please."] }, "accept": { "say": "A store credit is fine." } } } } ``` Here two paths (`ask>insist`, `ask>accept`) expand to two cells. A `next` that names an undefined node (an unknown node) or any cycle (a `next` that points back to an ancestor on the same path) is refused up front (exit 2); root-to-leaf paths must be finite. A shared child reached by two distinct parents (a diamond) is not a cycle -- it yields one path per route. `variables` and `branches` compose: with both present, the runner produces one cell per `(variable binding, root-to-leaf path, variation-matrix cell)`, and variable substitution applies to the branch lines too. ## Exit codes | Exit | Meaning | |---|---| | `0` | every produced conversation is `origin=simulated`, validated as a faithful rendering (and, under `--matrix --conversation-test`, every scored aggregate passed); with `--chat`, every scripted turn was driven and the transcript written | | `1` | at least one simulation was `SIMULATOR_INVALID` -- a broken fixture, distinct from an agent PASS/FAIL -- or, under `--matrix --conversation-test`, a scored aggregate FAILed | | `2` | usage error / unusable input: a malformed or unreadable scenario/conversation-test file -- including an unbound `{name}` template, an unknown branch node, or a branch cycle -- (or, under `--matrix --conversation-test` with `inconclusive_policy refuse`, a withheld verdict); a non-local `--chat` URL without `--egress-opt-in`, an unreachable chat agent, or a reply off the `--chat` contract | ## Related - [CONVERSATION-TEST.md](CONVERSATION-TEST.md) -- `hotato scenario init` / `hotato test run`: score a real call against a conversation-test. - [`schema/scenario.v1.json`](../src/hotato/schema/scenario.v1.json) -- the full scenario schema. ================================================================================ FILE: docs/START.md ================================================================================ # Start here: the guided first run `hotato start --demo` runs the whole hotato loop on bundled data, in two acts. Act one (timing): it sweeps the two demo recordings (the same calls `hotato demo` scores) and builds and verifies one failure contract from them. Act two (say-do): it runs one conversation check over a bundled scripted conversation -- the agent says the refund was sent; the trace and the post-call state show it was not issued. It writes every artifact you need to see both flows and prints the exact next commands. ```bash hotato start --demo ``` ## What it writes Into the current directory (or `--dir DIR`): | File | What it is | |---|---| | `hotato-sweep.json` | Every candidate timing moment across the two demo calls, ranked. An `analyze` result -- `FILE#N` off it drives `fixture promote` / `card` unchanged. | | `hotato-sweep.html` | Self-contained dashboard (embedded audio, no external assets), opens in any browser. | | `hotato-no-single-threshold.svg` | Threshold-funnel card: the demo battery misses an interruption **and** false-stops on a backchannel, so no single dial fixes both. See `docs/CARDS.md`. | | `contracts/demo-missed-interruption.hotato/` | One failure contract, built from the sweep's missed-interruption candidate with `--expect yield`, verified on the spot. See `docs/CONTRACTS.md`. | | `saydo/` | Act two's conversation: `transcript.json`, `trace.jsonl`, `state.json`, `test.json`, and the evaluated `test-run.json`. See below. | Everything runs offline: demo, analyze, card, contract, and say-do steps all run on packaged audio, packaged conversation files, and local files alone. ## The demo failure contract `start --demo` runs the whole loop once, on a bundled recording -- not just a ranked list of candidates: ```bash hotato contract create --from-candidate hotato-sweep.json#2 --expect yield \ --id demo-missed-interruption --out contracts hotato contract verify contracts/ ``` Candidate #2 in the bundled sweep is the missed-interruption call: the agent talked over the caller instead of yielding. Scored against `--expect yield`, it fails -- so `start --demo` prints: ``` verified contract: FAIL as expected -- the demo call missed the interruption; a CI gate on this contract catches any change to the evidence or policy -- catching the AGENT regressing requires a fresh recapture (see docs/RECAPTURE.md) (start --demo itself exits 0 because setup succeeded; run the next command to see the contract's CI exit 1: hotato contract verify contracts/) ``` `hotato contract verify contracts/` re-scores that same audio and reports the same regression (exit `1`); a CI job wired to it fails the same way if the evidence or policy ever changes. Telling whether a currently deployed agent still has the bug takes a fresh recapture: [`docs/RECAPTURE.md`](RECAPTURE.md). Two-lane breakdown: [`docs/CONTRACTS.md`](CONTRACTS.md#two-lanes-what-verify-proves-depends-on-which-recording-you-feed-it). Run it yourself: ```bash hotato contract verify contracts/ ``` ## Act two: the say-do check Timing is act one -- how the call sounded. Act two checks what the agent did. The demo bundles one scripted say-do conversation (`saydo/`, mirroring the reference agent's `refund-claimed-not-issued` job in `examples/reference-agent`): the caller asks for a refund, the agent looks the order up, and then says the refund was sent -- while the trace carries no `issue_refund` tool span and the post-call state's `refund_status` stays `"none"`. Every packaged file is verified against the sha256 its manifest records before use. `start --demo` evaluates that conversation through the same machinery `hotato test run` drives -- a `phrase` assertion holds the agent's claim (it passes), and the `tool_result` + `state` assertions read the trace (Authority 1) and the post-call state (Authority 2) -- so it prints: ``` say-do check: FAIL, by design: the agent said the refund was sent; the trace shows no such tool call succeeded (no issue_refund span), and the order's post-call refund_status stayed "none". ``` Tool and state evidence decide the outcome, never the agent's words. The evaluated result lands at `saydo/test-run.json`; render it as the say-do card (`docs/CARDS.md`) with: ```bash hotato card saydo/test-run.json --out saydo-card.svg ``` Replay the check as the CI gate it is (exit `1`, by design): ```bash hotato test run saydo/test.json --agent demo-agent \ --transcript saydo/transcript.json --trace saydo/trace.jsonl \ --state saydo/state.json ``` The whole conversation-QA loop behind it -- suites, scenarios, the 375-run reference agent -- lives in [`docs/CONVERSATION-TEST.md`](CONVERSATION-TEST.md) and [`docs/SUITE-RUN.md`](SUITE-RUN.md). ## What it prints The exact next commands, ready to copy: 1. **Save a candidate as a permanent regression test** -- you choose the label, hotato leaves that judgment to you: ```bash hotato fixture promote hotato-sweep.json#1 --expect \ --id my-first-fixture --out tests/hotato ``` 2. **Run your fixtures in CI** (exits non-zero on a regression): ```bash hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio ``` 3. **Render a PR card** from any candidate: ```bash hotato card hotato-sweep.json#1 --out candidate.svg ``` 4. **Re-verify the demo failure contract in CI** (or build your own from any candidate with `hotato contract create`): ```bash hotato contract verify contracts/ ``` 5. **Replay the say-do gate** (exit `1`, by design) and **render the say-do card**: ```bash hotato test run saydo/test.json --agent demo-agent \ --transcript saydo/transcript.json --trace saydo/trace.jsonl \ --state saydo/state.json hotato card saydo/test-run.json --out saydo-card.svg ``` ## Other modes `--stack`, `--folder`, and `--stereo` each hand you the shipped command for the job: | Flag | Command | Does | |---|---|---| | `--stack` | `hotato sweep --stack ` | Connect once, sweep your own calls | | `--folder` | `hotato analyze ` | Scan a directory of recordings | | `--stereo` | `hotato run --stereo ` | Score one dual-channel call | ## Output and exit codes `--format json` emits a machine object (the files written, the next commands, a `contract` block with the demo contract's id, expect, scorable, passed, and `verified_fail_as_expected`, and a `saydo` block with the say-do check's test id, exit code, claim assertion, and evidence assertions with their share-safe public reasons) for an agent to drive. - **0**: the guided first run completed -- including when `--stack`, `--folder`, or `--stereo` printed the shipped command to use instead. This holds even though the demo contract FAILS its policy and the say-do check FAILS its test: `start --demo` finishing is separate from those gates passing, which `hotato contract verify contracts/` and `hotato test run saydo/test.json ...` each report (exit `1`, by design). - **2**: usage error -- no mode given, or `--dir` is not a directory. ================================================================================ FILE: docs/STARTER.md ================================================================================ # The starter kit: `hotato init starter` One command adds hotato to an existing voice-agent repo: it scaffolds the CI gate, a stack-tuned config file, and the three directories the rest of the docs assume already exist. ```bash hotato init starter --stack vapi --out . ``` `--stack` is one of `vapi`, `retell`, `twilio`, `livekit`, `pipecat` -- every stack hotato has a shipped connector for ([`ADAPTER-STATUS.md`](ADAPTER-STATUS.md)). `--out` is usually `.`, the repo root. Generation runs offline -- nothing to connect. ## What it writes ``` HOTATO.md # what was added, next steps (read this first) hotato.yaml # config skeleton, tuned for --stack .gitignore # excludes local/pulled recordings; # keeps pinned fixture/contract clips committed .github/workflows/hotato-contracts.yml # the CI gate fixtures/ README.md scenarios/.gitkeep # -> hotato fixture create --out fixtures audio/.gitkeep contracts/ README.md .gitkeep # -> hotato contract create --out contracts reports/ README.md .gitkeep # local/CI scratch: doctor/report/sweep output ``` Each file writes only when it doesn't already exist (`--force` to overwrite), whole or not at all. Names are namespaced away from your own files (`HOTATO.md`, not `README.md`; `hotato-contracts.yml`, not `hotato.yml`), so a first run drops in cleanly next to files you already have. ## Two input paths, chosen by `--stack` **Auto-pull** (`vapi`, `retell`, `twilio`): hotato fetches the recording itself once you connect a key. `hotato.yaml`'s `credentials.env` names the exact variable(s) (`VAPI_API_KEY`; `RETELL_API_KEY`; `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN`), the same ones `hotato connect ` reads. `recording.access` is `auto-pull`. **Capture-in-your-infra** (`livekit`, `pipecat`): capture happens inside your own deployment, so no credentials are needed. `credentials.env` is `[]`, `recording.access` is `capture-in-your-infra`; `hotato setup --stack ` prints the two-track capture scaffold, and you point `hotato contract create --stereo` at the WAV your deployment writes. ## LiveKit and Pipecat runbook LiveKit and Pipecat are the two stacks where capture and turn-taking config live in your own code, ahead of any vendor API. Runbook for both, capture through CI. ### LiveKit 1. **Capture.** Two audio-only Track egresses, one per participant -- RoomComposite mixes both parties into one channel and can't attribute overlap. `hotato setup --stack livekit` prints the scaffold (Python `livekit-api`, `TrackEgressRequest` + `DirectFileOutput`); a ready-to-copy version lives at `adapters/livekit_capture.py`. 2. **Find the turn-taking config.** On `AgentSession(turn_handling= TurnHandlingOptions(...))`: `turn_detection` (`inference.TurnDetector()` / `"realtime_llm"` / `"vad"` / `"stt"` / `"manual"`), `endpointing` (`min_delay`, `max_delay`), and `interruption` (`enabled`, `mode`, `min_duration`, `min_words`, `false_interruption_timeout`, `resume_false_interruption`). Check what an agent file runs, statically: `hotato inspect --stack livekit --config agent.py`. 3. **Score it.** `hotato capture --stack livekit --caller caller.wav --agent agent.wav --onset --expect yield` (convert the egress output to WAV first: `ffmpeg -i caller.ogg caller.wav`). 4. **Fixture, contract, CI.** Same as every stack -- see "Turn your first bad call into a contract" below. ### Pipecat 1. **Capture.** A 2-channel `AudioBufferProcessor` in-pipeline (channel 0 = user/caller, channel 1 = bot/agent) -- don't mix down to one channel. `hotato setup --stack pipecat` prints the scaffold; a ready-to-copy version lives at `adapters/pipecat_capture.py`. 2. **Find the turn-taking config.** On `PipelineTask`'s user-turn strategies: start (`VADUserTurnStartStrategy`, `TranscriptionUserTurnStartStrategy`, `MinWordsUserTurnStartStrategy(min_words=...)`, `KrispVivaIPUserTurnStartStrategy(...)`) and stop (`SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=...)`, `TurnAnalyzerUserTurnStopStrategy(turn_analyzer=...)`) -- `MinWordsInterruptionStrategy` is deprecated since pipecat 0.0.99 in favor of `MinWordsUserTurnStartStrategy`. Check what a bot file runs, statically: `hotato inspect --stack pipecat --config bot.py`. 3. **Score it.** `hotato capture --stack pipecat --stereo captured.wav --expect yield` (write the WAV from `AudioBufferProcessor`'s `on_audio_data` handler first). 4. **Fixture, contract, CI.** Same as every stack -- see "Turn your first bad call into a contract" below. Both APIs move; `hotato setup` and `hotato inspect` print the verified-against date. Full field-level detail: [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md) (capture) and [`FIX-PLANS.md`](FIX-PLANS.md) (inspect, Level 1 of the fix ladder). ## The CI gate `.github/workflows/hotato-contracts.yml` runs on push, pull request, and weekly. Two guarded steps that pass clean, as a no-op, until you add a first contract or fixture (a fresh scaffold's normal starting state): ```bash hotato contract verify contracts --junit hotato.xml --format json > contracts-verify.json hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json ``` The JUnit file publishes as a build artifact on every run (`always()`), whether the gate passed, failed, or had nothing to check yet. The three auto-pull stacks also get a `weekly-sweep` job: a passive, candidate-only sweep of recent calls (`hotato sweep --stack `), ranked by salience for you to review and label. It ships disabled (`if: false`) -- flip it to `true` once the stack's credential env var(s) are repo secrets (Settings -> Secrets and variables -> Actions). A live pull runs only on your say-so, made once, in your own CI config. `livekit`/`pipecat` skip this job; capture already happens in your own deployment. ## Turn your first bad call into a contract ```bash # auto-pull stacks hotato connect vapi --api-key hotato sweep --stack vapi --out hotato-sweep.html # open hotato-sweep.html, pick a candidate moment, then: hotato contract create --from-candidate hotato-sweep.json#1 \ --expect yield --id refund-cutoff-001 --out contracts # capture-in-your-infra stacks hotato setup --stack livekit # once your deployment writes a two-channel WAV: hotato contract create --stereo call.wav --onset 42.18 \ --expect yield --id refund-cutoff-001 --out contracts ``` Commit the resulting `contracts/refund-cutoff-001.hotato/` directory. The next push runs it through the CI gate above. **A contract bundle contains call audio** (`audio/event.wav`). If this repo is or could become public, commit a sanitized fixture (synthetic or consent-cleared) and keep customer contracts in a private repository or controlled artifact storage. See [`CONTRACTS.md`](CONTRACTS.md). ## Read more - The bundle layout and the create/verify/inspect/pack/unpack commands: [`CONTRACTS.md`](CONTRACTS.md) - The underlying fixture primitive, one bad call to a CI gate in five steps: [`BAD-CALL-TO-CI.md`](BAD-CALL-TO-CI.md) - Per-stack connector support, verified against the vendor's live docs: [`ADAPTER-STATUS.md`](ADAPTER-STATUS.md) - The connect-once bulk pull-and-analyze recipe: [`CONNECT.md`](CONNECT.md) - An agent adding hotato to a repo end to end: [`../AGENTS.md`](../AGENTS.md) ================================================================================ FILE: docs/STATE-ADAPTERS.md ================================================================================ # State adapters: ground a `state` assertion in your system of record `state` and `state_change` assertions are **Authority 2**: post-call state verification -- querying a **system of record** after the call and comparing it against what the agent claimed ("I issued the refund"). A state adapter is the small, pluggable seam that runs that query. `hotato test run --state FILE` loads a state adapter from `FILE`. Three adapters ship, and one method backs all three: ``` query(resource, **filters) -> dict | None ``` Return the record for `resource` matching every `filters` key, or `None` when the system of record was read and holds no such record. ## The three-outcome boundary A `state` query lands on exactly three outcomes -- the distinction is the whole point: - **The record is present and matches** -> `PASS`. - **The system of record was read and holds no matching record** (or a field does not match) -> a grounded `FAIL`. The agent claimed something the record does not confirm. - **The system of record could not be reached or read** (network error, timeout, a 5xx, a non-JSON response, a DB error) -> `INCONCLUSIVE`, with a reason every time. A `state` assertion is satisfied by the query result alone: a lookup plus a dict comparison, deterministic, no model in the loop. ## The three adapters ### 1. `mock`: the local sandbox (default, offline) A JSON or SQLite fixture: `{resource: rows}`. Runs offline, byte-stable, ready with no opt-in. Use it for regression suites and for a captured before/after snapshot (`state_change` reads a `{"before": [...], "after": [...]}` shape); `--state some_sandbox.json` and `--state some.sqlite3` both select it. ### 2. `http`: a REST system of record Queries your API over stdlib `urllib`. A **resource map** turns a query into one request: ```json { "adapter": "http", "egress_opt_in": true, "base_url": "https://api.yourcompany.com/v1", "auth": { "type": "bearer", "token_env": "RECORDS_API_TOKEN" }, "timeout": 30, "resources": { "appointment": { "path_template": "/patients/{patient_id}/appointment", "method": "GET", "params_map": { "status": "appt_status" }, "response_pointer": "data/appointment" } } } ``` `query("appointment", patient_id="P1", status="booked")` fills `{patient_id}` into the path (URL-encoded), sends the remaining filters as query params (GET) or a JSON body (POST) under their `params_map` wire names, then extracts the record dict at `response_pointer`. - `response_pointer` is a JSON-pointer-ish path (`/`- or `.`-separated; digits index a list; empty means the whole body is the record). A pointer that resolves to nothing on a well-formed response means "no such record" -> `None`. - `method` is `GET` (default) or `POST`. - **HTTPS is required by default.** A plain `http://` base URL is refused unless you set `"allow_http": true` (only for a trusted local endpoint); otherwise a state query would send filter values and the auth header in cleartext. - A 404 means the record does not exist -> `None` -> grounded FAIL. Every other non-2xx, and any network/timeout/non-JSON failure, is INCONCLUSIVE (`query` raises `StateAdapterError`; the assertion engine turns it into an INCONCLUSIVE result, with the structured cause on the adapter's `last_error`). ### 3. `sql`: a SQL system of record Queries a database with a **parameterized, read-only** SELECT. ```json { "adapter": "sql", "sqlite_path": "records.sqlite3", "resources": { "refund": { "query": "SELECT order_id, status, amount FROM refunds WHERE order_id = ? AND status = ?", "params_order": ["order_id", "status"] } } } ``` `query("refund", order_id="O1", status="issued")` binds `[order_id, status]` as the statement's parameters and returns the first row as a `{column: value}` dict, or `None` when no row matches. - Connection source (exactly one): `sqlite_path` (a local file DB, fully local, no egress); `dsn` + `driver` (a DBAPI module to import and `connect`, e.g. `psycopg2` for PostgreSQL -- a network DB needing `egress_opt_in: true`; imported lazily, never a hard dependency); or a caller-supplied `connection` object (Python API only). - Use the placeholder style your driver expects (`?` for sqlite3, `%s` for psycopg2). Hotato only passes the bound value sequence through. - A DB/driver error resolves to INCONCLUSIVE -- a query never guesses a FAIL. ## Injection safety and read-only discipline Enforced by construction and covered in `tests/test_state_adapters_real.py`: - **Injection-safe.** Filter values are always bound as query parameters, never string-interpolated into the SQL: a filter value carrying SQL (`"O1'; DROP TABLE refunds; --"`) is treated as a literal, matching no row, table untouched. - **Read-only.** Every mapped SQL query is validated at construction (and re-checked before each execute): it must begin with `SELECT` (or a `WITH ... SELECT` CTE), carry no second statement after a `;`, and contain no data-modifying keyword, so a `DELETE`/`UPDATE`/`DROP`/... mapped query is rejected. ## Egress opt-in The `http` adapter, and a `sql` adapter over a `dsn`, are **network paths** (see [`docs/EGRESS.md`](EGRESS.md) and [`docs/THREAT-MODEL.md`](THREAT-MODEL.md)). `load_state_adapter` **refuses** them unless the config sets `"egress_opt_in": true` -- the same opt-in `--egress-opt-in` applies to the hosted diarizer and `--notify`. Without it, you get a clear usage error (the CLI's exit-2 path) before any connection is made. A local `sqlite_path` SQL DB and the mock sandbox run entirely on your machine, opening no socket, no opt-in needed. When an `http` / non-local-`sql` query does fire, only the mapped filter VALUES leave the machine (in the URL, query string, or JSON body). Audio, transcript, and the config file itself stay on your machine. ## Credentials Adapters take environment-variable **names** for credentials, keeping the config file free of secrets and shareable. Supply the secret at run time from the environment, e.g. `source` a `0600` file that exports it: - `bearer`: `{ "type": "bearer", "token_env": "RECORDS_API_TOKEN" }` - `basic`: `{ "type": "basic", "username_env": "DB_USER", "password_env": "DB_PASS" }` (`username` may be inline since it is not a secret). - `header`: `{ "type": "header", "headers": { "X-Api-Key": { "env": "API_KEY" }, "X-Tenant": { "value": "acme" } } }`, where each header value is either `{ "env": NAME }` (a secret from the environment) or `{ "value": LITERAL }` (a non-secret constant). A missing credential env var fails fast at construction, naming the variable in the message, never the value. Credential values stay out of logs and `last_error`. ## `state_change` and before/after snapshots `state_change` reads a `before` and `after` snapshot to measure a delta. Only the **mock** adapter provides both (from a captured `{"before": ..., "after": ...}` fixture, or `

__before` / `

__after` SQLite tables). The `http` and `sql` adapters answer point-in-time `state` queries -- a live API/DB exposes only "now" -- so their `before` snapshot reports absent, grounding a `state_change` against them at INCONCLUSIVE instead of guessing "no change." ## See also - [`docs/ASSERTIONS.md`](ASSERTIONS.md): the `state` / `state_change` assertion kinds. - [`docs/EGRESS.md`](EGRESS.md), [`docs/THREAT-MODEL.md`](THREAT-MODEL.md): the network rows. - `src/hotato/state_adapter.py`: the adapters; `tests/test_state_adapters_real.py`: the tests. ================================================================================ FILE: docs/SUBMITTING.md ================================================================================ # Submitting a recording The full path from call to merged corpus entry: record, label, validate, submit. One consented, de-identified, human-labeled call sharpens this eval. The scorer measures speech energy over time. Your label supplies the meaning: which onset was a true bid for the floor, and whether a well-behaved agent should yield to it. Energy isn't intent -- the label is the ground truth, and it's what you contribute. ## 1. Record dual-channel Caller on one channel, agent on the other, separated at capture -- two legs of a SIP bridge, or two fully separate streams. This makes overlap a fact of the recording, exact to the sample. Save a WAV at the call's native sample rate (8000 Hz telephony, 16000 Hz wideband). Consent, PII, and PHI rules live in [`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md): documented consent from every audible party, identifiers redacted with same-duration tone or silence, no PHI. Read it before you record -- it has a reusable release paragraph. ## 2. Label it The label is a JSON file next to your WAV: the bundled scenario shape, plus provenance and attestation. Spec: [`corpus/label.schema.json`](../corpus/label.schema.json). Example: [`corpus/examples/sample-contribution.json`](../corpus/examples/sample-contribution.json). Minimal example: ```json { "id": "call-021-address-change-interrupt", "title": "Caller interrupts the shipping recap to change the address", "category": "should_yield", "source_type": "real-call", "audio": "call-021-address-change-interrupt.wav", "channels": { "caller_channel": 0, "agent_channel": 1 }, "sample_rate": 8000, "duration_sec": 9.2, "caller_onset_sec": 3.15, "expected": { "yield": true, "max_time_to_yield_sec": 0.7, "max_talk_over_sec": 0.8 }, "license": "MIT", "attestation": { "contributor": "Your Name ", "consent_on_file": true, "pii_removed": true, "no_phi": true, "right_to_release_mit": true } } ``` For a hold case (the caller backchannels "mm-hm" and a good agent keeps talking), set `category` to `should_not_yield`, `expected.yield` to `false`, and both `max_*` bounds to `null`. Recording a role-play instead of a live call? The repeatable recipe -- script shape, the consent/PII/attestation checklist mapped to the schema fields, dual-channel capture, and the defect-performed-on-purpose labeling -- is [`RFC-ROLEPLAY-FIXTURES.md`](RFC-ROLEPLAY-FIXTURES.md). Label only what you can defend by hand: `caller_onset_sec` is required, `reference_render` segment timings are a bonus if you have them. The harness reports error only for the signals you supply. ## 3. Validate locally ```bash python3 corpus/validate.py your_label.json ``` The WAV resolves next to the label via its `audio` field. If it lives elsewhere, pass the audio path as a second argument: ```bash python3 corpus/validate.py your_label.json your_recording.wav ``` PASS (exit 0) means the pair conforms: fields well-typed, category and expected bounds consistent, timings in range, attestation affirmed, audio a readable WAV with two or more channels matching the label. A human still reviews consent and PII before anything merges. ## 4. Submit Two doors, one intake: - **Issue form.** Open ["Corpus submission: labeled recording"](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml). Attach the WAV as a .zip (GitHub rejects a raw .wav) or link it, and paste your label JSON in the description. - **Pull request.** Add the label and WAV under [`corpus/`](../corpus/) -- e.g. `corpus/call-021-address-change-interrupt.json` plus its `.wav`. State the source type in the PR body and confirm the attestation, per [`CONTRIBUTING.md`](../CONTRIBUTING.md). ## What maintainers do at intake 1. **Validate.** Re-run `corpus/validate.py` on the pair; check the label against [`corpus/label.schema.json`](../corpus/label.schema.json). 2. **Dedupe.** Hash the audio and compare call details against existing clips -- every clip earns its place. 3. **Normalize.** Slug the `id`, align filenames, confirm the channel map and declared sample rate. Any audio edit leaves the label's timing untouched. 4. **Add to a suite.** Register the clip so the benchmark harness scores it and reports millisecond error distributions and the yield confusion matrix ([`BENCHMARK.md`](BENCHMARK.md)). 5. **Credit.** `attestation.contributor` is the credit of record, and contributors are named in the changelog when their clip lands. Removal requests are honored in the next release, per [`CORPUS-GOVERNANCE.md`](CORPUS-GOVERNANCE.md). ================================================================================ FILE: docs/SUITE-RUN.md ================================================================================ # `hotato suite run`: execute a suite of conversation-tests `hotato suite run` loads a `hotato.suite` file (a named set of conversation-test refs, each scored on its own), resolves each ref relative to the suite file, executes every test, and emits a per-dimension + reliability report, also recording the run into the fleet registry so `hotato serve` can browse it. A conversation-test is one file defining one testable conversation; see [CONVERSATION-TEST.md](CONVERSATION-TEST.md). A suite (JSON or YAML) groups those files under a `suite_id`, a `purpose`, an `inconclusive_policy`, and a `required_for_release` flag. ## How a test runs - A test that declares a `scenario` runs **offline** through the deterministic scripted-caller simulator: its variation matrix expands into concrete runs, each rendered, validated, and scored against the test's deterministic lane. Authority-1 tool spans and the Authority-2 mock-state sandbox come from the scenario's `agent_mock`, fully simulated and offline (Authority 2 is grounded state; see [STATE-ADAPTERS.md](STATE-ADAPTERS.md)). - A test with no scenario is evaluated once against an empty context: every input-dependent check reports INCONCLUSIVE, never a guessed pass/fail. ## Every dimension scored on its own The report carries per-dimension counts (`outcome`, `policy`, `conversation`, `speech`, `reliability`) plus a reliability aggregate: `pass@1`, `pass@k`, `pass^k`. Per-dimension counts group results across tests; reliability stands as its own dimension over every valid simulated run. Each dimension keeps its own count and verdict, exposed as-is, including in `--format json`; the schemas reject an `overall_score` key. A `SIMULATOR_INVALID` run flags a broken fixture, bucketed separately and reported by test id and run id, kept out of every dimension and the reliability aggregate. ## The suite's policy is authoritative The suite's `inconclusive_policy` is the effective policy for every test it names, regardless of a test's own policy. `inconclusive_policy: fail` makes an INCONCLUSIVE (absent required input) FAIL the gate; `refuse` withholds the verdict (exit `2`, precedence). The suite exit code is the **worst** test outcome under that policy (`refuse` > `fail` > `pass`), raised to at least `1` by any `SIMULATOR_INVALID` run. ## Run it ```bash hotato suite run examples/reference-agent/suite.json \ --agent reference-agent-v1 \ --release reference-agent-v1 \ --out ./suite-out ``` `--agent ID` (required) is recorded on every conversation and the release. `--release ID` names the release runs are recorded under (default `@`) -- keep it stable per release for [`hotato release compare`](RELEASE-COMPARE.md). `--out DIR` writes the per-test simulated conversation artifacts plus `suite-report.md`, `suite-report.html`, and `suite-run.json`. `--parallel N` caps worker threads per scenario's matrix; the worker count never changes the byte-identical result. `--junit PATH` also writes a JUnit XML report for a CI test widget (one `` per dimension, one `` per test; a FAILed dimension is a `` with the measured reason, an INCONCLUSIVE dimension or a `SIMULATOR_INVALID` run is an ``, never a silent pass); see [CI.md](CI.md). Runs are recorded into the fleet registry (`--registry`, default `~/.hotato/fleet`; `--workspace`/`-w`, default `default`) as Release / Suite / Scenario / Run / Conversation / Evaluation rows, so `hotato serve -w ` renders them (see [WORKSPACE.md](WORKSPACE.md)); `--no-registry` reports only. ## Worked example: the reference agent `examples/reference-agent/` ships a full suite: 25 realistic voice-agent jobs x 5 caller behaviours x 3 audio environments = 375 offline simulated runs, scored across the outcome / policy / conversation / speech dimensions. Its `suite.json` has `inconclusive_policy: fail` and `required_for_release: true`. One command regenerates the files, runs the 375-run suite offline, and records a browsable workspace: ```bash cd examples/reference-agent make reference # then browse: hotato serve --workspace reference --registry ./.workspace ``` The summary printed to stdout (counts here are schematic): ``` hotato suite run: reference-agent-suite (...) -- agent reference-agent-v1, release reference-agent-v1 -- exit_code=0 inconclusive_policy: fail required_for_release: True tests: 25 (25 pass, 0 fail, 0 refuse) runs: 375 (375 valid, 0 SIMULATOR_INVALID -- broken fixtures, never an agent PASS/FAIL), origin=simulated per-dimension (grouped across tests; never blended): outcome ... pass / ... fail / ... inconclusive policy ... pass / ... fail / ... inconclusive conversation ... pass / ... fail / ... inconclusive speech ... pass / ... fail / ... inconclusive reliability ... pass / ... fail / ... inconclusive reliability [origin=simulated]: pass@1=... pass@k=... pass^k=... (n=...) per-test: [pass ] refund-damaged-order runs=15 ... ... ``` Origin is always labelled `simulated`, keeping a simulator's replay reliability distinct from production reliability. ## Exit codes - **`0`**: every test in the suite passed under its `inconclusive_policy` (no run `SIMULATOR_INVALID`). - **`1`**: at least one test FAILed (a `success.required` condition failed, a deterministic assertion FAILed, or, under `inconclusive_policy fail`, an INCONCLUSIVE gated), or a run was `SIMULATOR_INVALID`. - **`2`**: under `inconclusive_policy refuse` a scored INCONCLUSIVE withheld the verdict (precedence over a FAIL); or a usage error / unusable input: a malformed suite / conversation-test / scenario file, or an unresolvable test/scenario ref. ## See also - [CONVERSATION-TEST.md](CONVERSATION-TEST.md): the conversation-test file each suite ref points at. - [STATE-ADAPTERS.md](STATE-ADAPTERS.md): Authority 2 state grounding for `state` / `state_change` checks. - [RELEASE-COMPARE.md](RELEASE-COMPARE.md): diff two releases recorded by `suite run`. - [WORKSPACE.md](WORKSPACE.md): browse the recorded runs with `hotato serve`. - [SUITES.md](SUITES.md): the `corpus/suites/` scenario-audio suites for the `hotato run` scoring path (a distinct mechanism). ================================================================================ FILE: docs/SUITES.md ================================================================================ # Corpus suites: tiered, deterministic, explicit about what they prove `corpus/suites/` ships four labelled scenario suites (112 scenarios), each synthetic shaped noise rendered deterministically from its own labelled timings (seed `sha256(scenario_id)`), the ground truth. Synthetic audio is the floor: these suites prove the scorer runs end to end and catch regressions; validity on your system comes from your own labelled calls ([`docs/SUBMITTING.md`](SUBMITTING.md)). ## The four suites | Suite | Scenarios | Conditions | Expected exit | | --- | --- | --- | --- | | `silver` | 40 | clean, 16 kHz, default noise floor | `0` | | `silver-defects` | 16 | clean conditions, deliberate defect renders | `1` | | `gold` | 40 | hard conditions: noise floors, 8 kHz, gain extremes, echo, edge timings, endurance | `0` | | `gold-defects` | 16 | hard-condition defect renders, plus two labelled capture-defect cases | `1` | Scenario families: hard interruptions (onset, speed, duration, resume), backchannels, short acknowledgements like "mhm" the agent should talk through (position, density, repeats, length), double-talk, one-word interruptions, stutter onsets, multi-turn exchanges, resume-then-reinterrupt, and latency prompts. `corpus/suites/manifest.json` is the machine-readable inventory: per suite, family and category breakdown, sample rates, and expected exit code. ## Defect suites fail by design Every `silver-defects` / `gold-defects` scenario is rendered to fail on its labelled axis: an agent that keeps talking through a true interruption, stops for a backchannel, or misses its latency budget. A defect suite that exits `1` is the scorer catching what it claims to; one that exits `0` would be a bug: the negative control the positive suites need to mean anything. ## Run a suite A suite is a scenarios directory plus an audio directory: ```bash hotato run --suite barge-in \ --scenarios corpus/suites/gold/scenarios \ --audio corpus/suites/gold/audio ``` The same directories plug into every other surface: `hotato report ... --out report.html` for the visual report, `hotato export ... --out research/` for CSVs, and `pytest --hotato-suite --hotato-suite-scenarios ... --hotato-suite-audio ...` for the session gate ([`docs/PYTEST.md`](PYTEST.md)). ## Deterministic builder The suites are generator-built end to end, and the generator is the proof: ```bash python3 corpus/suites/build_suites.py # rebuild in place python3 corpus/suites/build_suites.py --check # regenerate to a temp dir, byte-compare ``` `--check` regenerating byte-identical output is the reproducibility guarantee: the audio on disk exactly matches its labelled timings, deterministic for a fixed hotato version (byte-identical re-runs verified in CI on Linux x86_64, Python 3.10, 3.11, 3.12; see `.github/workflows/tests.yml`, job `determinism`) -- usable as a CI drift gate. ## Additive scenario classes `corpus/classes/` ships four deterministic classes on top of the suites above, built the same way (synthetic shaped noise, seed `sha256(scenario_id)`, `--check` byte-compares a rebuild): - `mid-utterance-pause`: a multi-second thinking gap mid-turn, distinct from a true turn end. - `backchannel-multilingual`: short non-English acknowledgement tokens; Hotato's energy-based VAD treats them like English ones. - `noise-hold`: sustained background presence, distinct from a brief backchannel, the agent should hold through. - `telephony-degraded`: an existing gold scenario re-rendered through G.711 mu-law plus a fixed packet-loss schedule, proving the verdict stable across codec degradation. Kept separate because `mid-utterance-pause` needs a non-default `turn_end_silence_sec` beyond the generic suite tests. Full per-class detail: `corpus/classes/README.md`. ## Against a live stack Running scenario audio through a live voice stack and scoring what comes back: [`docs/BENCHMARK-STACKS.md`](BENCHMARK-STACKS.md). Measurement-error methodology for the scorer itself: [`docs/BENCHMARK.md`](BENCHMARK.md). ## Contribute scenarios New synthetic families are welcome through the normal PR path (`CONTRIBUTING.md`). The highest-value contribution stays a consented, labelled call from your own traffic: walkthrough at [`docs/SUBMITTING.md`](SUBMITTING.md); intake via the [corpus-submission issue form](https://github.com/attenlabs/hotato/issues/new?template=corpus_submission.yml). ================================================================================ FILE: docs/THREAT-MODEL.md ================================================================================ # Threat model Your call recordings stay on your machine. Every network action is one you name. This page draws the line: which commands are offline-only, which reach the network, and what holds true either way. ## Three guarantees 1. **You apply every production change yourself.** `plan` and `patch` produce a proposal; `apply` dry-runs by default on a fresh staging clone. No command pushes to production on its own. 2. **Recordings stay on your machine.** Audio moves only from your stack to your disk, only when you run `pull`/`capture`/`sweep`. The one opt-in exception: the hosted `--diarizer pyannoteai` backend, gated by `--egress-opt-in`. 3. **A webhook payload is data, never code.** `ingest` reads a completed-call notification, extracts a recording reference, and scans it. Payload fields are read, never executed or used to choose what runs. ## Core: fully offline, on your machine These commands read and write only the local files you point them at. They open no sockets -- this is the whole scoring, analysis, and fixture surface. | Command | What it does | |---|---| | `run` | score a local WAV (or the bundled self-test battery) | | `scan` | list candidate moments in a local recording | | `trust` | input-health check on a local recording | | `report` | render a self-contained HTML report from local run data | | `fixture` | create or promote a local moment into a regression fixture | | `compare` | score a before/after pair of local takes | | `verify` | roll up before/after run envelopes on disk | | `diagnose` | explain a finished local run envelope (read-only) | | `plan` | combine a diagnosis and config into a fix-plan JSON | | `patch` | render a fix plan into a paste-ready patch; never applies it | | `demo` | run the packaged two-call battery, zero extra files | | `team`, `export`, `benchmark` | aggregate or export local run data | | `setup`, `describe`, `init` | print recording config, emit the CLI manifest, scaffold local integration files | | `rubric run`, `rubric calibrate` | score a local transcript against a rubric with a **local** model (Ollama on `localhost`); opens no off-box socket by default, and the verdict cache is local too. Reaches the network only with an explicit hosted or non-local judge (see below) | | `test run` (rubric lane) | same local judge as `rubric run`, run inline on a test's `assertions.rubric` lane. Local by default | | `counterexample compile\|verify\|reproduce\|inspect\|export\|predicate` | read local scenario/test/capsule files, write a new local capsule or share-safe projection. Loads no provider adapter, model, subprocess, or network client | Default retention is local-only: reports, envelopes, and exports land exactly where you point them. ### Counterexample capsule boundary A private `.hotato-repro` holds the source scenario and test, reduced fixture, target assertion result, and proof journal -- treat it as sensitive source material. The compiler creates it with owner-only permissions where the host supports POSIX, never overwrites an existing destination, and promotes a sibling staging directory only once every file and hash exists. The verifier rejects: path traversal; symlinked members and directories; special files; undeclared files; oversized or deep inputs; digest mismatches; broken deletion chains; evaluator-source drift on the strict proof path; and a minimality claim that admits a preserving unit deletion. Preflight caps a capsule at 1,024 files, 4,096 directories, 64 directory levels, 64 MiB per member, and 256 MiB total. `reproduce` is a separate, current-evaluator check: it permits evaluator drift, but still validates capsule integrity and the source-to-final delete-only chain. Replay also bounds proof-specific work: 256 KiB per selected assertion, 2 MiB per canonical candidate scenario, 256 KiB of rendered transcript text, 2 MiB per assertion result, 10,000 evidence rows, 512 accepted proof-chain steps, 10,000 deletion operations per accepted step, and 512 remaining deletion units in a completed minimality proof. Accepted steps require fresh oracle evaluations -- cached preserved journal rows can't inflate the chain. Proof regexes use a closed, fixed-width 1,024-byte subset that refuses groups, alternation, backreferences, and variable quantifiers. These limits apply to counterexample compilation and replay, not to global evaluator limits. The selected-assertion and proof-regex byte checks run before the general assertion validator can invoke Python's regex parser. Each profile has an exact member allowlist; a manifest can't authorize a new file outside it. For `share-safe-v1`, the only members are `capsule.json`, `report.md`, `report.html`, `card.svg`, `README.md`, and `MANIFEST.sha256.json` -- each human-facing file is reconstructed canonically and compared byte for byte during inspection. This stops a modified report, README, or card from being accepted just because its replacement digest was written into a new manifest. The canonical share renderer's bytes are frozen as part of the capsule v1 exchange contract. A future renderer must retain v1 output, or use a versioned profile/format, rather than silently changing bytes under an existing proof. The evaluator digest binds the shipped Python source closure and package version, not the interpreter build, host platform, CPU, or native libraries. Strict replay compares the recorded result, content, and trace hashes, so a runtime difference that changes behavior is rejected. Accepted transforms and the final single-unit inventory are proof-bearing; non-`PRESERVED` journal rows are diagnostic history, not independently replayed. The capsule directory must stay unchanged during a command. Persistent mutation, root replacement, and symlink or special-file substitution are all detected. A privileged local process that can swap and restore bytes between individual reads sits outside the v1 proof snapshot -- move untrusted capsules into a private, non-writable workspace before verification. `counterexample export` derives a non-runnable projection after strict private verification. It omits scenario/test bodies, transcript content, tool payloads, state values, provider identifiers, reducer paths, and per-candidate digests; private minimality rows are reduced to aggregate outcome counts. The projection exposes the payload-free selected failure code for review while keeping its field/key/rule/detector/index discriminator private. It still contains identifiers and hashes that remain correlators -- low-entropy or externally known source material can be tested against them. `share-safe-v1` belongs inside the team's normal engineering-artifact access boundary; it does not claim anonymous public disclosure. ## Network: only when you explicitly request it These commands reach outside the machine, and only because reaching out is their job. Each requires you to name a stack, a repository, or a webhook you configured. | Command | Network surface | Notes | |---|---|---| | `connect` | none at connect time | stores a stack's credentials locally at `0600`. Setup for the pull path; no audio moves | | `pull` | your voice stack's API | bulk-fetch recent recordings from a stack **you** connected, into a local folder | | `capture` | your voice stack's API | fetch and score one call from your stack | | `sweep` | your voice stack's API | `pull` + analyze in one command | | `inspect` | your voice stack's API | read (never write) the current turn-taking config. Read-only | | `ingest` | a webhook endpoint you host | worker that scans each completed call. Payloads are data, not instructions (guarantee 3) | | `issue` | GitHub, via your local `gh` | file a sweep's candidates as an issue. Uses your existing `gh` auth | | `pr` | GitHub, via your local `gh` | open a PR adding promoted fixtures. Uses your existing `gh` auth | | `apply` | a git clone you point it at | applies a patch to a fresh **staging** clone only, never the source. Dry-run by default; refuses a both-axes threshold funnel | | `--diarizer pyannoteai` | hosted diarizer backend | the only audio path that can send audio off-box, and only with `--egress-opt-in`. The default diarizer is local | | `--judge-provider hosted` / non-local `--judge-endpoint` (any rubric command) | a hosted or remote model host you name | sends the transcript + rubric criterion off-box for judging. Refused (exit 2) unless `--judge-egress-opt-in`. The default judge is a local Ollama model that stays on the box. See [`docs/EGRESS.md`](EGRESS.md) and [`docs/RUBRIC.md`](RUBRIC.md) | | `test run --state` http adapter | your system-of-record's REST API | only when the state-config names `adapter: http`, and only with `egress_opt_in: true` in that config. Sends the mapped filter VALUES for one `state`/`state_change` query -- audio, transcript, and the config itself stay local. See [`docs/STATE-ADAPTERS.md`](STATE-ADAPTERS.md) | | `test run --state` sql adapter over a `dsn` | your database, over the network | only when the state-config names `adapter: sql` with a `dsn`, and only with `egress_opt_in: true`. A parameterized, read-only SELECT with the mapped filter values bound as data. A local `sqlite_path` opens no socket | The state adapters read a post-call **system of record** to ground an Authority-2 `state` assertion (did the refund/appointment get written?). A record the system of record can read, and does not hold, is a grounded FAIL; a system of record Hotato could **not** reach or read (network error, timeout, 5xx, non-JSON) is INCONCLUSIVE, never a guessed verdict. Credentials come from environment-variable **names** in the config, keeping the secret itself out of it; the HTTP adapter refuses a plain-`http://` base URL unless `allow_http: true` is set for a trusted local endpoint. The default `--state` path -- the local mock sandbox (a JSON/SQLite fixture) and a local `sqlite_path` SQL DB -- runs entirely on-machine, no opt-in needed. Notify surfaces (Slack, GitHub) fire only through credentials you configured (`gh`, a Slack token), and only for actions you invoked. ## Network trust: proxies and TLS Every networked command above (`pull`, `capture`, `sweep`, `inspect`, `apply`, and the credential probe in `connect`) makes its HTTP calls through Python's standard `urllib`, which honors the `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY` convention -- the same one `curl`, `pip`, `git`, and `docker` follow. That's deliberate: it lets Hotato work behind a corporate proxy with no extra flags. It also means an environment variable set for the Hotato process controls where its outbound credentialed requests go -- a trust decision worth stating plainly. Two things bound how far that ambient trust reaches: - **TLS certificate validation is always enforced.** Every credentialed base URL in `capture.py` and `apply.py` is hardcoded `https://` (`api.vapi.ai`, `api.retellai.com`, `api.twilio.com`, and the other supported stacks). A proxy set via `HTTP_PROXY`/`HTTPS_PROXY` can see the `CONNECT` target host and port, and can refuse or stall the connection (a denial of service) -- but reading or rewriting the `Authorization` header or the response body also requires a certificate the machine's own trust store already accepts for that vendor's domain: a much larger compromise (a rogue trusted root CA already installed), outside `HTTP_PROXY`'s reach and Hotato's control. - **The threat prerequisite is already a compromised local environment.** An attacker able to set environment variables for the Hotato process can already read `VAPI_API_KEY` / `RETELL_API_KEY` / etc. straight out of the environment, or read the `0600` credentials file `connect` wrote -- both easier than standing up a TLS-valid proxy. If you don't trust the ambient proxy environment a command will run in, set `HOTATO_NO_PROXY=1` to make Hotato's HTTP opener ignore `HTTP_PROXY` / `HTTPS_PROXY` for that run, or unset those variables first. The default is unchanged: proxy env vars are honored, matching curl/pip/git, so a corporate-proxy setup keeps working with no configuration. ## Hotato's containment guarantees The three guarantees above, restated as the boundary they draw: recordings stay local unless you opt in; a webhook payload is read as data and never executed (guarantee 3); and the furthest a config change goes on its own is a patch you apply yourself, to a staging clone. ## Verifying the posture Confirm the posture yourself, without trusting this page: - **Self-hosted, nothing to audit but the standard library.** `pip install hotato` pulls zero runtime dependencies; the only network code path is the one you explicitly invoke. - **Credentials at `0600`.** `connect` writes stack credentials locally with owner-only permissions; check with `ls -l` on your credentials path. - **Local-only retention.** Reports, envelopes, and exports exist exactly where you wrote them. - **Egress is opt-in.** The only off-box audio path is `--diarizer pyannoteai`, gated by `--egress-opt-in` -- audio stays on the machine until you set it. ## Drive-a-call (`run_scenario`): placing a live call `run_scenario` (`src/hotato/drive.py`, wired into the Vapi and Twilio adapters) places an outbound call against a live agent and pulls the recording. Its threat surface, and the controls on it: - **A billable, outbound call always requires explicit opt-in.** `run_scenario` places the call only when BOTH live credentials AND an explicit egress opt-in (`HOTATO_DRIVE_OPT_IN=1` or `egress_opt_in: true` on the scenario) are present. Absent either, it raises a clean structured refusal and dials nothing -- the same opt-in posture as `--allow-mono` / `HOTATO_ALLOW_PRIVATE_URLS` / `--egress-opt-in`. - **Production config stays untouched.** The surface is create-and-read only: the only verbs issued are `POST` (create the call) and `GET` (poll status, list the recording, download it) -- no `PUT`/`PATCH`/`DELETE` path to alter an assistant, phone number, or any other provider resource in place. For Vapi the call originates FROM the staging clone the experiment created, keeping the production source untouched. - **The caller side is labelled precisely.** The produced conversation carries `origin.kind == "real"` with the provider and its call id, and `origin.caller` (`scripted-twiml` for the fixed-timeline Twilio caller, `assistant-originated` for a Vapi call the assistant placed) -- stating exactly what happened, with no claim that a human placed the call or that the scripted caller reacted to the agent. - **The recording download inherits every capture-side control.** The vendor URL (`stereoUrl` / `RecordingSid` media) flows through the same validated download as `capture`: http(s)-only scheme allowlist, default-deny SSRF (loopback/private/metadata refused unless `HOTATO_ALLOW_PRIVATE_URLS=1`), cross-host `Authorization`-strip on redirect, and atomic write. The API key is never attached to a pre-signed media URL from the vendor's JSON response. See [`docs/DRIVE-A-CALL.md`](DRIVE-A-CALL.md) and the drive-a-call rows in [`docs/EGRESS.md`](EGRESS.md). ## Team workspace (`hotato serve`): a new local listening socket `hotato serve` (`src/hotato/serve/`, wired into the CLI) opens a new local HTTP listening socket to serve the conversation-QA views over the fleet registry and evidence store. Its threat surface, and the controls on it: - **Localhost-default bind.** The server binds `127.0.0.1` unless you explicitly pass `--host`. A non-loopback bind (e.g. `--host 0.0.0.0`) always prints a prominent warning at start, and always requires that explicit flag. - **Token auth on every request.** A shared bearer token authenticates every request, compared in constant time (`hmac.compare_digest`). It's either operator-supplied (`--token` / `--token-file`) or generated with `secrets.token_urlsafe` and stored `0600` under the per-workspace state dir. Browsers bootstrap an in-memory, HttpOnly session cookie from the printed `/?token=…` URL, then the token is stripped from the address bar via a redirect; it stays out of every response body. An unauthenticated request gets `401` before it's routed anywhere. - **Reads everywhere; one fenced write route.** Every view issues only `SELECT`s against the registry and reads evidence blobs by digest. The single write endpoint, `POST /calls//pin`, delegates to the same fleet label/contract machinery the CLI drives, and is fenced beyond the bearer/cookie auth above: the session cookie is `SameSite=Strict`, and a cookie-authenticated POST must carry a same-origin `Origin`/`Referer` header matching the request's own `Host` -- a forged cross-site form is refused `403` before any handler runs (a bearer-authenticated request needs no origin header; a cross-site attacker cannot set one). The delegated mint is atomic, so a refused pin (4xx with reason) leaves no artifact. The serve layer's own file stays the append-only audit log (`…/serve//audit.jsonl`, `0600`), recording who (token/session prefix, the secret stays out of it), what (method + path, token stripped from the query), when, and the response status of every request -- pin attempts included, accepted or refused. - **Zero egress.** The server only binds a listening socket: no outbound connection, nothing that phones home. Audio, traces, and evaluations stay on the machine. A test allowlists loopback and fails if any view attempts an external connection. - **Evidence renders as data, never as executable content.** The raw evidence endpoint (`/evidence/`) serves blobs as `text/plain` with `X-Content-Type-Options: nosniff` (pages also set `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer`), keeping a crafted evidence blob inert in the viewer. Redacted transcript/trace content is scrubbed at the data layer, so both the HTML and the JSON mirror render only the `[redacted]` placeholder. - **Workspace id sanitized against path traversal.** The workspace id is used verbatim only in parameterized SQL; as a state-directory name it is sanitized to stay inside the registry home. See [`docs/WORKSPACE.md`](WORKSPACE.md). Security policy and reporting: [SECURITY.md](../SECURITY.md). ================================================================================ FILE: docs/TRACE.md ================================================================================ # Voice traces: coincidence, not root cause A voice trace is a timeline of discrete voice-pipeline events (caller/agent audio activity, TTS cancel/stop, ASR partials, tool calls, ...): "the agent talked over the caller" becomes "evidence suggests TTS cancellation lagged: cancel requested at 42.40s, audio stopped at 43.60s." See [`docs/OTEL.md`](OTEL.md) for the ingest source formats. Three commands: ```bash hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl ``` ## Schema `voice_trace.jsonl` is JSONL (one meta line, then one line per span -- the same convention `evidence/frames.jsonl` uses), validating against `hotato.voice_trace.v1` (`src/hotato/schema/voice_trace.v1.json`) once reassembled by `hotato.trace.load_voice_trace_jsonl`: ```json { "schema": "hotato.voice_trace.v1", "call_id": null, "deployment": { "stack": "vapi", "agent_id": null, "git_sha": "deadbeef", "config_hash": "sha256:..." }, "spans": [ {"type": "agent_audio_active", "start_sec": 0.0, "end_sec": 2.9}, {"type": "tts_cancel_requested", "time_sec": 2.6}, {"type": "tts_audio_stopped", "time_sec": 2.9}, { "type": "asr_partial", "start_sec": 2.4, "end_sec": 2.95, "text_redacted": true }, { "type": "tool_call", "start_sec": 1.1, "end_sec": 1.42, "name": "lookup_order", "latency_ms": 320 } ], "source": {"format": "otel-jsonl-bridge", "input_span_count": 6} } ``` A span carries an open `type` string; common ones are `caller_audio_active`, `agent_audio_active`, `tts_cancel_requested`, `tts_audio_stopped`, `asr_partial`, `tool_call`, `llm_first_token`, `handoff`. An unrecognized type passes through unchanged -- additive, forward-compatible with a span type this release doesn't name. One time shape per span: an interval carries `start_sec`/`end_sec`, a point event carries `time_sec`. ## Redaction by default `call_id` and `deployment.agent_id` are dropped (`null`) unless `--include-identifiers` is passed at ingest. An `asr_partial` span's transcript text is dropped (`text_redacted: true`, no `text` key) unless `--include-text` is passed. `deployment.stack` / `git_sha` / `config_hash` are kept by default -- not treated as identifiers. ## Ingest ```bash hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl hotato trace ingest --otel export.json --out voice_trace.jsonl \ --stack vapi --include-identifiers --include-text ``` `--otel FILE` accepts a standard OTel JSON export (a document with a top-level `resourceSpans` array) or hotato's own OTel bridge JSONL (see [`docs/OTEL.md`](OTEL.md)). `--stack` / `--call-id` / `--agent-id` / `--git-sha` / `--config-hash` override or fill in the source's resource attributes. Refused (exit 2, nothing written) for an unreadable file, an empty file, or a source with zero spans. ## Attach ```bash hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl ``` Copies the trace into `/traces/voice_trace.jsonl` and re-renders `evidence/timeline.html` with the trace's events as an additional row, aligned to the same [0, duration] scale as the existing caller/agent timeline. This reads the bundle's own `evidence/frames.jsonl` and `contract.json` back in, reusing the scored evidence as-is: attaching a trace needs no diarization extra and no re-scoring. On a diarized-mono bundle (no frame-level evidence), the base timeline says so plainly, and the trace row still renders on its own scale. `contract.json` records the attachment (additive, schema-safe: `trace: {attached, path, span_count, attached_at, source_format}`). Refused (exit 2) for a missing bundle, a trace file that doesn't validate as `hotato.voice_trace.v1` JSONL, or an already-attached trace without `--force`. ### Report wording When a `tts_cancel_requested` / `tts_audio_stopped` pair is present, the timeline states the delta plainly: > Evidence suggests TTS cancellation delay: cancel requested at 2.60s, > audio stopped at 2.90s (delta 0.30s). > Hotato does not prove root cause. > Unknowns: no client-side playout trace was attached. The last line always appears in this release: a client-side playout trace (when the caller's device stopped rendering audio, not when the server issued the stop) sits outside what this release's span types collect, so the gap is always named. ## Export ```bash hotato trace export contracts/refund-cutoff-001.hotato --format otel --out otel.jsonl ``` Writes the bundle's attached trace back out as hotato's OTel bridge JSONL -- the exact shape `trace ingest` reads, so `ingest -> attach -> export -> ingest` round-trips identical spans. `--format` is claimed by the export format here (only `otel` is supported today); pass `--json` for the machine summary instead of the default text line. Refused (exit 2) when the bundle has no attached trace, or `--out` exists without `--force`. ## What a voice trace adds Timing correlation only: a pattern of events lined up with the contract's timing measurement, named plainly. Authorization, identity, compliance, policy safety, intent, and root cause stay outside its claims; a client-side playout moment is named only when one was attached. ## Read more - Source formats and the OTel bridge JSONL shape: [`docs/OTEL.md`](OTEL.md) - Failure contracts a trace attaches to: [`docs/CONTRACTS.md`](CONTRACTS.md) ================================================================================ FILE: docs/TRANSCRIBE.md ================================================================================ # `hotato run --stereo call.wav --transcribe`: read a transcript next to the score Hotato's gold reference is a deterministic **energy VAD**: `did_yield`, `talk_over_sec`, and `seconds_to_yield` come from frame energy alone, and every published number is computed that way. The opt-in `[transcribe]` extra runs an off-the-shelf speech-to-text model over the same recording and returns plain text with per-segment timestamps -- so a reader, human or agent, sees WHAT was said next to WHEN the timing engine says it was said. That is the entire feature: a transcript is attached as **context** beside the score, computed strictly after scoring is final, and never feeds back into it. ## What stays unchanged - **The timing score stays grounded in the audio.** `did_yield` / `talk_over_sec` / `seconds_to_yield` / every other field computes identically with or without `--transcribe`; adding it produces byte-identical timing numbers, pinned by a test (`tests/test_transcribe.py`). - **Plain text, not a quality judgment.** The transcript reports what was said, not how well: plain text and timestamps only -- no WER, accuracy, or quality number, anywhere in hotato. - **Timestamps only, no speaker attribution.** No caller/agent labels -- anonymous speaker separation lives in the separate [`docs/DIARIZE.md`](DIARIZE.md) extra. ## Quickstart ```bash # Install the opt-in extra (local, offline; no gated token, unlike [diarize]): pip install 'hotato[transcribe]' # Score a call and attach a transcript next to the verdict: hotato run --stereo call.wav --transcribe --format json ``` With the extra absent, `--transcribe` fails loud: a clean error, exit `2`, never a silent skip. ## Usage ```bash # base.en, device auto-detected: hotato run --stereo call.wav --transcribe # pick a model + device: hotato run --stereo call.wav --transcribe --transcribe-model small.en \ --transcribe-device cpu # also works on the diarized-mono path: hotato run --mono call.wav --diarize --transcribe ``` `--transcribe` needs a **single** audio file: `--stereo`, or `--mono` when scoring through the `--diarize` front-end (it transcribes the same mono file the diarizer separated). Two separate `--caller`/`--agent` files raise a clean usage error naming `--stereo`, rather than guessing which channel to transcribe or silently dropping one. Combining `--transcribe` with the self-test battery (`--suite` / `--scenarios`+`--audio`) also raises a clean usage error, not a silent no-op, so CI never mistakes "flag ignored" for "flag applied." ## Models and device - Default model: `base.en` (English-only, the fastest useful size). Override with `--transcribe-model NAME` -- any name or local path `faster-whisper`/CTranslate2 accepts. - Default device: `auto` -- `cuda` if CTranslate2 reports a CUDA device, else `cpu`. `compute_type` defaults to `float16` on GPU and `int8` on CPU (CTranslate2's documented fast defaults per device, not an accuracy claim); both overridable. - Every choice used is stamped in the output (`transcript.model` / `.device` / `.compute_type` / `.language`), so a report is reproducible. ## Backend and extra - **`[transcribe]`** - Brings in: `faster-whisper` (a CTranslate2 re-implementation of OpenAI's Whisper). - Where it runs: local, CPU-viable, GPU-optional, fully offline once the model is cached. ```toml transcribe = ["faster-whisper>=1.0"] ``` This extra keeps the core's Python floor at `>=3.9` (unlike `[diarize]`), since `faster-whisper`/CTranslate2 support that range -- no model card to accept, either. ### Model licenses (log per FTO note) Using an off-the-shelf ASR model is integration, orthogonal to any Hotato IP claim; licenses are logged here: - `faster-whisper` -- **MIT** (code) - `CTranslate2` -- **MIT** (the inference runtime `faster-whisper` runs on) - Whisper model weights -- **MIT** (OpenAI) ## The invariant: context beside the score, grounded in the audio A transcript is computed strictly **after** the score is final, over the same audio the score already used, and attached on a new top-level key of the envelope -- nothing already there is read or rewritten: - `events`, `verdict`, `measurements`, and `signals` are the exact same values with or without `--transcribe`. - The energy VAD's frame-level activity stays the ground truth for timing. A transcript word boundary is a separate thing -- ASR timestamps are a model estimate, read alongside it, never substituted in. - A missing/broken `[transcribe]` extra raises a clean, actionable error -- exactly like the `[neural]` and `[diarize]` seams: never a bare `ImportError`, a silent skip, or a fallback to a different backend. - `hotato.transcribe` and the vendored `_engine` import nothing from each other, keeping the transcript path fully separate from the scorer. ## JSON shape (agents) A `--transcribe` run is the same envelope as any single run, plus one additive top-level `transcript` block. `events[0].verdict` is identical to a run without it: ```json { "mode": "single", "events": [{ "verdict": { "did_yield": true, "seconds_to_yield": 0.42, "talk_over_sec": 0.18, "reasons": [] } }], "transcript": { "text": "I need to check on my refund -- hold on, let me pull that up for you.", "segments": [ {"start": 0.10, "end": 2.30, "text": "I need to check on my refund"}, {"start": 2.60, "end": 4.95, "text": "hold on, let me pull that up for you."} ], "model": "base.en", "device": "cpu", "compute_type": "int8", "language": "en" } } ``` Branch on the presence of `transcript`: present only when `--transcribe` was passed, additive only -- the meaning of everything under `events` stays the same. On the diarized-mono path, `transcript` still attaches even when the diarization confidence gate refuses or downgrades the verdict: ASR runs independent of diarization. ## Egress Fully local at inference time: once the named model is cached, transcribing opens no socket. The **first** run of a new model downloads its weights from its public host (the same one-time fetch as any pip package with model weights); every run after that is offline. See [`docs/EGRESS.md`](EGRESS.md) for the full per-command network table. ================================================================================ FILE: docs/TRANSPORT-RUNTIME.md ================================================================================ # Transport runtime Hotato separates three facts that voice-test systems often collapse: 1. a provider accepted or completed a call lifecycle operation; 2. a media/signalling runtime delivered bytes or events at an agent boundary; 3. the resulting conversation satisfied outcome, policy, timing, and reliability assertions. The first fact comes from `TelephonyClient`. The second comes from a `ConversationSession` implementation. The third comes from Hotato's evidence and scoring lanes. A lifecycle status never substitutes for delivered-media evidence or task outcome. ## Runtime contracts `hotato.call_runtime.CallController` is the lifecycle boundary: ```text capabilities(provider) create(spec) -> handle get(provider, call_id) -> handle wait(handle) -> terminal handle cancel(handle) -> handle export(handle, output_dir) -> redacted receipt path cleanup(handle, export_path) -> local deletion receipt ``` `hotato.call_runtime.ConversationSession` is the duplex media/signalling boundary: ```text capabilities() connect() events() send_audio(pcm_s16le, sample_rate_hz, channels) send_dtmf(digits) hold(enabled) transfer(destination, warm=False) hangup() close() ``` These are Python protocols. A provider SDK, local WebSocket harness, or separately operated sidecar can implement them without being imported into Hotato's zero-dependency core. Every capability has one explicit state: | State | Meaning | |---|---| | `SUPPORTED` | The selected implementation executes the operation and returns the evidence its contract requires. | | `UNSUPPORTED` | The selected implementation cannot execute the operation. | | `UNOBSERVABLE` | The operation may occur elsewhere, but this boundary cannot return the evidence required to credit it. | Unknown support does not become a failed quality verdict. `require_capability()` refuses both `UNSUPPORTED` and `UNOBSERVABLE` before an operation starts. ## Normalized call events Every media/signalling implementation emits `hotato.call-event.v1` records. Records form an append-only SHA-256 chain per `(run_id, call_id)`: ```json { "schema": "hotato.call-event.v1", "event_id": "agent-audio-0001", "run_id": "run-2026-07-17-001", "call_id": "call-01", "leg_id": "agent", "sequence": 0, "source": "livekit-sidecar", "kind": "audio.delivered", "observed_monotonic_ns": 321000000, "source_timestamp": "2026-07-17T12:00:00.321Z", "trace_id": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", "payload": {"sample_rate_hz": 16000, "channels": 1, "bytes": 640}, "raw_sha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000", "trust": "measured", "previous_event_hash": null, "event_hash": "sha256:" } ``` `normalize_call_event()` rejects unknown fields, non-canonical JSON values, oversized events, invalid digests, sequence gaps, clock reversal, run/call changes, and a mismatched claimed hash. `AppendOnlyCallLog` rejects duplicate event IDs and verifies the entire chain. `raw_sha256` hashes the bytes observed at the stated boundary. It does not claim those bytes were heard by a participant. `trust` identifies the authority: - `measured` for bytes or state measured at the boundary; - `provider_reported` for provider lifecycle/API statements; - `sidecar_reported` for an external runtime's report; - `derived` for a deterministic transform over retained inputs; - `model_reported` for a model output; - `operator_attested` for an operator statement; - `unverified` when no stronger authority is available. ## Lifecycle controller matrix `hotato.telephony.TelephonyClient` controls provider lifecycle APIs. It does not transport media. | Operation | local | Twilio | Vapi | Retell | |---|---:|---:|---:|---:| | create | supported | supported | supported | supported | | status endpoint | unsupported | supported | supported | supported | | wait for terminal status | supported | supported | supported | supported | | cancel | supported | supported | unsupported | unsupported | | redacted Hotato receipt export | supported | supported | supported | supported | | delete matching local export | supported | supported | supported | supported | | delete provider history | unsupported | unsupported | unsupported | unsupported | | media / DTMF / hold / transfer | unobservable | unobservable | unobservable | unobservable | Credentials come only from environment variables: | Provider | Variables | |---|---| | Twilio | `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN` | | Vapi | `VAPI_API_KEY` | | Retell | `RETELL_API_KEY` | Receipts retain request field names and body byte counts. The provider response uses a fixed lifecycle allowlist; every omitted response is bound only by its canonical byte count and SHA-256. Free-form reason strings, identifiers, telephone numbers, transcripts, recording URLs, messages, metadata, authorization fields, keys, tokens, secrets, and passwords are never copied from a provider response. Portable receipts and exports retain a provider- domain-separated call-ID digest instead of the provider call ID. Export uses exclusive file creation. Cleanup deletes only an export whose provider, call-ID digest, and lifecycle receipt ID match the supplied in-memory handle. It never deletes provider history. ## Media and signalling sidecars Hotato composes established transport runtimes instead of implementing RTP, SIP, codec negotiation, NAT traversal, or WebRTC congestion control. ### LiveKit SIP `livekit_sip_contract(endpoint)` declares the evidence boundary for a LiveKit Server + LiveKit SIP deployment. DTMF, hold, and transfer begin as `UNOBSERVABLE`; a session adapter promotes each operation to `SUPPORTED` only when it returns the corresponding room/participant/SIP events and delivered-audio digest. The expected evidence set is: - room and participant events; - SIP lifecycle/status events; - delivered PCM or encoded-stream digest at the agent input boundary; - per-leg event timestamps and trace correlation. ### Pipecat `pipecat_media_contract(endpoint)` declares a streaming media boundary. The generic contract requires pipeline events and a delivered-audio digest. DTMF, hold, and transfer are unsupported until a concrete transport adapter defines and evidences them. ### SIPp `SippSubprocessAdapter` executes a caller-supplied SIPp XML scenario for SIP-path and load probes. SIPp owns SIP/RTP behavior. Hotato applies a static scenario policy, fixes the subprocess argv and environment, and records receipts. The adapter does not provide an operating-system sandbox. ```python from hotato.call_runtime import SippSubprocessAdapter receipt = SippSubprocessAdapter().run( { "target": "127.0.0.1:5060", "scenario_path": "fixtures/inbound.xml", "calls": 20, "rate_per_second": 2, "timeout_seconds": 180, }, "artifacts/sipp-run-01", ) ``` The adapter: - uses an argv array with `shell=False`; - accepts no free-form extra arguments; - validates the target and binds a regular, non-symlink XML file to its opened inode; - parses XML with DTDs, entity declarations, external entities, and processing instructions disabled; - defaults to a safe scenario profile that rejects every `` action, scenario file attribute, unsafe path attribute, and SIPp `[file ...]` host-file keyword; - rejects `` in the safe profile so scenario XML cannot bypass the resolved target and remote-IP policy; - bounds calls, rate, scenario bytes, timeout, stdout, and stderr; - supplies only `PATH`, `HOME`, and locale variables to the child; - writes data artifacts with exclusive creation and mode `0600`, and stages the executable copy as mode `0700`; - hashes the target instead of writing it into the receipt; - reports process exit as `PASS` or `FAIL` without converting it into conversation quality. Non-loopback SIP is default-deny. A remote spec must set `allow_remote=true`, list the resolved destination IP in `remote_ip_allowlist`, cap the run with `max_remote_calls`, and supply `I_ACCEPT_REMOTE_SIP_SIDE_EFFECTS_AND_UNOBSERVABLE_EXTERNAL_COST`. The adapter resolves once, passes the selected allowlisted IP to SIPp, and binds that destination into the receipt. External SIP/carrier cost remains `null` and `UNOBSERVABLE`. The built-in subprocess path stages and hashes the SIPp executable; injected test runners report executable identity as unobservable. The remote acknowledgement authorizes network side effects only. It does not authorize scenario-level host commands, host-file reads, or destination redirection. ### Trusted SIPp scenarios SIPp XML is executable input. Its `` action invokes a host shell, while media actions and scenario keywords can read host files. Those features are denied by default. A deployment that has reviewed the scenario and accepts running it without an OS sandbox can opt into the unrestricted scenario profile with a separate fixed acknowledgement: ```python from hotato.call_runtime import SIPP_TRUSTED_SCENARIO_ACKNOWLEDGEMENT receipt = SippSubprocessAdapter().run( { "target": "127.0.0.1:5060", "scenario_path": "fixtures/reviewed-media.xml", "trusted_scenario_acknowledgement": SIPP_TRUSTED_SCENARIO_ACKNOWLEDGEMENT, }, "artifacts/sipp-reviewed-01", ) ``` The fixed phrase is `I_ACCEPT_SIPP_SCENARIO_HOST_COMMAND_AND_FILE_ACCESS_WITHOUT_OS_SANDBOX`. Trusted mode still refuses DTDs, entity declarations, external entities, processing instructions, malformed XML, excessive nesting, and excessive element count. Destination redirection remains denied in every profile so the scenario cannot bypass the resolved and allowlisted target. The receipt records `trusted_host_access`, `operator_attested`, and `os_process_sandbox: ABSENT`; it does not claim that the scenario was confined. Run trusted scenarios only in an independently isolated container or virtual machine with the minimum filesystem and network access required. RTP playback, DTMF, hold, and transfer encoded inside XML remain `UNOBSERVABLE` until target-side events establish delivery. Warm transfer is unsupported by the generic SIPp adapter because it does not orchestrate a verified second live leg. ## Minimum acceptance evidence Promote a session operation from `UNOBSERVABLE` to `SUPPORTED` only after a hermetic adapter test and a live-path fixture produce all applicable artifacts: - normalized hash-chained events with no sequence gaps; - target-boundary audio digest and channel/sample format; - provider or SIP lifecycle events; - DTMF digit plus sender/receiver evidence; - hold start/end plus media behavior during hold; - transfer initiation, destination-leg connection, source-leg disposition, and warm/cold mode; - bounded stdout, stderr, network destination inventory, and process exit; - teardown evidence for every created room, call, process, and local artifact. Keep live-path results scoped to the provider version, region, codec, adapter commit, and fixture that produced them. No controller capability or provider documentation establishes a universal carrier-path result. ================================================================================ FILE: docs/TRUST-GALLERY.md ================================================================================ # Trust gallery: eight recordings, eight verdicts Eight named input conditions, each with the `hotato` output it produces. Every block below is verbatim CLI output from hotato 1.18.1, offline. The rows correspond to the [trust matrix](TRUST-MATRIX.md). The clean and echo cases use the bundled examples `01-hard-interruption.example.wav` and `07-echo-bleed.example.wav` (shipped in `src/hotato/data/audio/`); the backchannel case uses `02-backchannel-mhm.example.wav`. The four defect cases (silent caller, silent agent, swap, mono) are deterministic synthetic two-channel fixtures pinned in `tests/test_trust.py`, each isolating one input defect, so every verdict reproduces byte-for-byte from the shipped builders. --- ## 1. Safe stereo: the good case Clean two-channel export, caller on ch0, agent on ch1. `trust` clears it. ```text $ hotato trust --stereo 01-hard-interruption.example.wav hotato trust: 01-hard-interruption.example.wav recording: 6.0s, 16000 Hz, 2 channels caller (ch0): 2.46s speech, first at 2.39s, peak -4.4 dBFS agent (ch1): 2.71s speech, first at 0.19s, peak -4.4 dBFS leading silence: 0.19s crosstalk: coherence 0.306 (low) at 0.5s lag scorability: separated tracks yes, caller activity yes, agent activity yes => eligible for scan ``` **Verdict:** `eligible for scan`, exit 0. Score it -- the only condition where a full turn-taking verdict is trustworthy without a caveat. --- ## 2. Silent caller: nothing to measure against The caller channel carries no speech (a dead leg, a wrong export, a muted line). ```text $ hotato trust --stereo silent-caller.wav hotato trust: silent-caller.wav recording: 6.0s, 16000 Hz, 2 channels caller (ch0): 0.00s speech, -, peak -120.0 dBFS agent (ch1): 5.76s speech, first at 0.19s, peak -9.1 dBFS leading silence: 0.19s crosstalk: coherence 0.0 (low) at 0.0s lag scorability: separated tracks yes, caller activity no, agent activity yes => NOT SCORABLE: caller channel has no detected speech next step: verify channel mapping or export dual-channel again ``` **Verdict:** `NOT SCORABLE`, exit 2. A yield is a response to a caller -- with no caller, there is nothing to be late to. Hotato names the gap instead of printing a hollow pass. --- ## 3. Silent agent: no floor to interrupt The agent channel is empty. There is no floor for the caller to barge into. ```text $ hotato trust --stereo silent-agent.wav hotato trust: silent-agent.wav recording: 6.0s, 16000 Hz, 2 channels caller (ch0): 2.16s speech, first at 0.99s, peak -9.1 dBFS agent (ch1): 0.00s speech, -, peak -120.0 dBFS leading silence: 0.99s crosstalk: coherence 0.0 (low) at 0.0s lag possible channel swap: channel 0 (mapped as caller) holds the floor 2.16s vs 0.0s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed scorability: separated tracks yes, caller activity yes, agent activity no => NOT SCORABLE: agent channel has no detected speech next step: verify channel mapping or export dual-channel again ``` **Verdict:** `NOT SCORABLE`, exit 2. Hotato also flags a possible channel swap here: a caller-only recording looks like a mis-mapped agent. Two independent checks point at the same fix -- re-check the export. --- ## 4. Swapped channels: candidate-eligible, verdict pending confirmation Both channels carry speech, but the long floor-holder is on the channel mapped as the caller. `trust` keeps the input candidate-eligible and holds the verdict until the mapping is confirmed. ```text $ hotato trust --stereo swapped-warn.wav hotato trust: swapped-warn.wav recording: 6.0s, 16000 Hz, 2 channels caller (ch0): 5.76s speech, first at 0.19s, peak -9.1 dBFS agent (ch1): 0.66s speech, first at 1.99s, peak -9.1 dBFS leading silence: 0.19s crosstalk: coherence 0.31 (low) at 0.5s lag possible channel swap: channel 0 (mapped as caller) holds the floor 5.76s vs 0.66s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed scorability: separated tracks yes, caller activity yes, agent activity yes => scan with caution: channel 0 (mapped as caller) holds the floor 5.76s vs 0.66s on channel 1 (mapped as agent); an agent usually holds the floor longer, so the caller/agent channels may be reversed [!] not verdict-eligible (scan mode): channel mapping unconfirmed: suspected swap/crosstalk; confirm mapping or supply provider metadata ``` **Verdict:** `scan with caution`, exit 0: candidate-eligible, verdict eligibility waiting on your confirmation. A swap would invert every yield into a hold, so Hotato surfaces candidates for review and holds the turn-taking verdict until the mapping is confirmed. Set `--caller-channel` / `--agent-channel`, or supply provider metadata, to restore verdict eligibility. --- ## 5. Crosstalk / echo bleed: candidate-eligible, verdict pending confirmation The caller channel carries a delayed copy of the agent's own audio. Coherence pegs at 1.0 and the measured leakage is -9.1 dB. ```text $ hotato trust --stereo 07-echo-bleed.example.wav hotato trust: 07-echo-bleed.example.wav recording: 6.0s, 16000 Hz, 2 channels caller (ch0): 5.69s speech, first at 0.31s, peak -13.5 dBFS agent (ch1): 5.81s speech, first at 0.19s, peak -4.4 dBFS leading silence: 0.19s crosstalk: coherence 1.0 (HIGH) at 0.12s lag leakage: -9.1 dB (agent->caller, HIGH) scorability: separated tracks yes, caller activity yes, agent activity yes => scan with caution: cross-channel leakage: agent audio on the caller channel sits at -9.1 dB, a consistent delayed copy (lag 0.12s); leaked audio at this level can be counted as the other party's activity, so a downstream timing measurement may be wrong even though the tracks are separated [!] not verdict-eligible (scan mode): channel mapping unconfirmed: suspected swap/crosstalk; confirm mapping or supply provider metadata ``` **Verdict:** `scan with caution`, exit 0: candidate-eligible, verdict eligibility waiting on confirmation. The "caller" activity may be leaked TTS, so Hotato still surfaces candidates for review (example 8's scan names the specific moment) and holds the turn-taking verdict until the mapping is confirmed. --- ## 6. Mono: refused by default, tiered under `--diarize` A single channel. The caller and the agent are mixed into one track. ```text $ hotato trust --stereo mono-mixed.wav hotato trust: mono-mixed.wav recording: 6.0s, 16000 Hz, 1 channel scorability: separated tracks no, caller activity no, agent activity no => NOT SCORABLE: the recording has a single channel, so the caller and the agent cannot be told apart next step: export a dual-channel recording with the caller on one channel and the agent on the other ``` **Verdict:** `NOT SCORABLE`, exit 2 -- the gold-standard refusal. Two opt-in escapes exist, both indicative only: `--allow-mono` on `capture` / `pull` / `sweep` accepts a mono-only stack in degraded mode (talk-over unattributable, no SLA gate), and the `--diarize` separation front-end reports a `high` / `low` / `refuse` tier, with `hotato run --mono call.wav --diarize` stamping the verdict `indicative_only` at `low`. Neither stands in for dual-channel. See [TRUST-MATRIX.md](TRUST-MATRIX.md) and [DIARIZE.md](DIARIZE.md). --- ## 7. Backchannel candidate: surfaced, yours to label A scan of a call where the caller says "mhm" over the agent. `scan` lists the overlaps as candidates and hands the intent decision to you. ```text $ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5 hotato scan: 02-backchannel-mhm.example.wav (6.0s, 3 candidate moments) Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset --expect yield|hold [ 1] t=2.09s overlap_while_agent_talking overlap=1.58s agent did not go silent within 3.0s [ 2] t=3.19s overlap_while_agent_talking overlap=1.07s agent did not go silent within 3.0s [ 3] t=4.29s overlap_while_agent_talking overlap=0.56s agent did not go silent within 3.0s ``` **Verdict:** three candidates, exit 0. Whether "agent did not go silent" is correct (held the floor through a backchannel, good) or a bug (talked over a caller trying to take the floor) is **your** call. `scan` reports the timing; you label `hold` or `yield` -- the intent behind a caller sound is always yours to decide. --- ## 8. Noisy false positive: a candidate that is really the agent A scan of the echo-bleed call. One candidate is an overlap fact; the second is Hotato flagging that the "caller" activity is leaked TTS. ```text $ hotato scan --stereo 07-echo-bleed.example.wav --top 5 hotato scan: 07-echo-bleed.example.wav (6.0s, 2 candidate moments) Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset --expect yield|hold [ 1] t=0.31s overlap_while_agent_talking overlap=3.00s agent did not go silent within 3.0s [ 2] t=0.31s echo_correlated_activity WARNING likely agent echo: coherence=1.00 at lag 0.12s (caller channel looks like leaked TTS; a yield here may be the agent hearing itself) ``` **Verdict:** two candidates, exit 0. Candidate 1 looks like a bad talk-over. Candidate 2 is Hotato telling you the overlap is probably the agent hearing its own audio, not the caller interrupting. This is candidate discovery's failure mode: the net is wide, and Hotato labels a likely-spurious candidate instead of hiding it. You label it `hold` (or fix the echo capture) and it never becomes a fixture. --- ## How to read the gallery The eight cases fall into three buckets: - **Score it** (1): clean stereo, verdict-eligible, full confidence. - **Candidate-eligible, verdict pending confirmation** (4, 5): swap and crosstalk keep `scan` surfacing candidates (exit 0), with the turn-taking verdict waiting on your confirmation of the channel mapping. Example 8 is that same echo call under `scan`, naming the specific leaked moment. - **Refuse** (2, 3, 6): silent caller, silent agent, and mono are not scorable; Hotato names the reason and the next step, and exits 2. Case 7 is the everyday case: a candidate with no defect, waiting for your label. The goal: any Hotato verdict you act on has already survived this gauntlet. ================================================================================ FILE: docs/TRUST-MATRIX.md ================================================================================ # Trust matrix: what Hotato does for each input condition Before Hotato scores a recording, `hotato trust` checks the audio and decides whether a score would mean anything -- catching a bad export before it becomes a confident-looking but hollow verdict. This page is the exact contract: input condition on the left, Hotato's behavior on the right. It covers input health only; a turn-taking verdict is `scan`'s and `run`'s job. Worked examples with CLI output for every row are in [TRUST-GALLERY.md](TRUST-GALLERY.md). The four-tier evidence policy the rows below implement is stated once in [EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). ## The contract | Input condition | `trust` says | Exit | Downstream scoring | |---|---|---|---| | Clean dual-channel (stereo) | `eligible for scan` | 0 | **Full scoring.** `scan`, `run`, `compare`, `verify` all run normally. | | Silent caller channel | `NOT SCORABLE: caller channel has no detected speech` | 2 | **Refused.** No caller to measure a yield against. | | Silent agent channel | `NOT SCORABLE: agent channel has no detected speech` | 2 | **Refused.** No agent floor to interrupt. | | Channel swap risk | `eligible for scan` **plus** `possible channel swap` warning | 0 | Scores -- but confirm the mapping first (`--caller-channel` / `--agent-channel`); a swap silently inverts every yield/hold. | | High crosstalk / echo bleed | `eligible for scan` **plus** high `crosstalk: coherence` warning | 0 | Scores at **lower confidence.** `scan` tags the moment `echo_correlated_activity`; a "yield" there may be the agent hearing itself. | | Mixed mono (single channel) | `NOT SCORABLE: single channel, caller and agent cannot be told apart` | 2 | **Refused by default.** Export dual-channel, or use one of the two opt-in escapes below. | | Mono, opt-in `--allow-mono` | Accepted in **degraded mode**, results **indicative only** | 0 | Used by `capture` / `pull` / `sweep` for a mono-only stack. Talk-over can't be attributed, so no SLA gate fires; dual-channel stays the gold reference. | | Mono, opt-in `--diarize` | Separability **tier**: `high` / `low` / `refuse` | 0 (high/low), 2 (refuse) | The separation front-end. `hotato run --mono call.wav --diarize` scores it; `low` stamps the verdict `indicative_only`. Dual-channel stays the gold reference. | | Short backchannel ("mhm") overlap | Outside `trust`'s scope; `scan` lists it as a **candidate** | 0 | **Candidate only, human labels.** `trust`/`scan` report the timing; you always label `yield` vs `hold`. | | Noisy / false-positive candidate | `trust` warns (clipping, crosstalk, hot capture); `scan` still lists the candidate | 0 | Surfaced **with** the warning, as a candidate for you to inspect and label `hold`. | **Reading the two axes.** `trust` answers one question: is this audio good enough to score? The exit code is the machine contract: `0` means eligible for scan (maybe with warnings), `2` means not scorable, with the reason and the next step. Warnings (swap, crosstalk, clipping, leading silence) inform the read; only the three hard refusals -- mono, identical channels, a silent required channel -- change scorability. ## Why refuse instead of guessing A less careful tool would still print a number here. A mono file "scored" by guessing which speaker is which produces a verdict indistinguishable from a properly-scored one -- and it is worthless. A swapped-channel file scored without the warning inverts caller and agent, so "the agent yielded" becomes "the caller yielded" with no visible sign. Hotato treats these as input defects, reports them, and stops: a green or red build always means something. ## Two ways past a mono refusal Mono is refused by default: one channel cannot separate the two parties. Two opt-in escapes exist, both indicative only. **1. `--allow-mono` (degraded mode).** On `capture`, `pull`, and `sweep`, the `--allow-mono` flag (or `HOTATO_ALLOW_MONO=1`) accepts a mono-only recording from a mono stack. Nothing is separated, so talk-over can't be attributed to caller or agent: the result is explicitly indicative and no SLA gate fires. Use it when a stack only exports a mono mix and a rough signal still beats none. **2. `--diarize` (separation front-end).** The opt-in `[diarize]` extra (`pip install 'hotato[diarize]'`) runs a local diarizer and reports a confidence tier before you score: - **high**: confidently separable. `hotato run --mono call.wav --diarize` gives a diarized-mono verdict. Exit 0. - **low**: separable but only indicative (voices close, overlap elevated). The verdict is stamped `indicative_only`; no SLA gate fires. Exit 0. - **refuse**: not confidently two clean parties. Not scorable, exit 2; record dual-channel. The dual-channel path stays the gold reference -- neither a degraded-mono nor a diarized-mono verdict is promoted to equivalence with it. Full front-end: [DIARIZE.md](DIARIZE.md). ## Composing the gate Because `trust` exits `2` on any unscorable input, it drops straight into a shell gate ahead of a scan or a scheduled sweep: ```bash hotato trust --stereo call.wav && hotato scan --stereo call.wav ``` For agents, `hotato trust --stereo call.wav --format json` emits one machine-parseable report. Branch on `scorable`; on a defect, read `not_scorable_reason` and `next_step`. Details: [TRUST.md](TRUST.md). ================================================================================ FILE: docs/TRUST.md ================================================================================ # `hotato trust --stereo call.wav`: is this recording even scorable? The input-health check: inspect one recording and learn whether the audio is good enough to score, before you scan or run it. `trust` catches a bad export -- mono file, silent channel, swapped channel map, hot capture -- up front, before it becomes a confident-looking but meaningless verdict downstream. What each input tier supports downstream is the four-tier policy in [EVIDENCE-CONTRACT.md](EVIDENCE-CONTRACT.md). ```bash hotato trust --stereo your-call.wav ``` ```text hotato trust: your-call.wav recording: 32.0s, 44100 Hz, 2 channels caller (ch0): 3.32s speech, first at 3.27s, peak -2.1 dBFS agent (ch1): 23.30s speech, first at 0.45s, peak -0.9 dBFS leading silence: 0.45s crosstalk: coherence 0.057 (low) at 0.39s lag scorability: separated tracks yes, caller activity yes, agent activity yes => eligible for scan ``` Exit `0` means eligible for scan; exit `2` means NOT SCORABLE (or a usage error, or an unreadable file). `trust` composes straight into a shell gate: ```bash hotato trust --stereo call.wav && hotato scan --stereo call.wav ``` ## What it checks By convention the caller is channel 0 and the agent channel 1 (override with `--caller-channel` / `--agent-channel`). `trust` reports INPUT health only: | Signal | What it means | |---|---| | per-channel activity | how much speech each channel carries, and when each first speaks | | possible channel swap | a heuristic flag -- if the channel mapped as caller holds the floor far longer than the one mapped as agent (the reverse of the usual pattern, since an assistant answers in paragraphs), the channels may be reversed | | sample rate and duration | the basic recording facts | | clipping | per-channel peak level (dBFS) and the fraction of samples at full scale, so a too-hot capture is visible | | leading silence | dead air before the first speech on either channel | | crosstalk risk | cross-channel echo coherence -- is the caller channel carrying a delayed copy of the agent's own audio (echo bleed, missing echo cancellation)? | | cross-channel leakage (`crosstalk_risk.leakage_db`) | how many dB below the source a consistent, delayed COPY of one channel shows up on the other. Whole-clip coherence (above) is one best-lag cosine over the entire envelope, diluted by unrelated activity elsewhere in the call, so bleed loud enough to corrupt a timing verdict can sit under that bar and go unflagged. Leakage measures differently: the per-frame level ratio of the copy, constant across every frame the source speaks, so it catches that regime. At or above `-40 dB` (the level at which red-teaming flipped a verdict with symmetric bleed) the copy counts as the other party's activity: flagged, and the recommendation drops off `eligible for scan` | | low signal level | when even the loudest channel peaks below `-30 dBFS`, turn timing can be under-measured downstream -- a warning, never a not-scorable condition | | scorability | the three things a score needs -- separated tracks, enough caller activity, enough agent activity | | recommendation | `eligible for scan`; `scan with caution` (scorable, but a loud cross-channel leak may corrupt the scan's timing); or `NOT SCORABLE`, with the specific reason and the next step | ## Scope: one question, and it stops there `trust` answers one question -- is this audio good enough to score? -- and stops there: no `yield`/`hold`, no `pass`/`fail`, no `did_yield`, no talk-over number, no intent label, no turn-taking verdict. Eligible for scan means the input is clean, not that the agent behaved well; finding agent bugs is [`hotato scan`](../src/hotato/scan.py)'s and `hotato run`'s job. ## Not scorable Three input defects make a recording unscorable. Each is reported with `scorable: false`, a plain reason, and the next step -- and exits `2`: - **mono**: a single channel can't separate caller from agent. Next step: export a dual-channel recording (the gold reference), or score it via the opt-in diarization front-end ("Mono via `--diarize`" below); - **identical channels**: a mono recording duplicated into two channels -- two channels, but not separated. Same fix as mono; - **a silent required channel**: for example `caller channel has no detected speech`. Next step: verify channel mapping or export dual-channel again. Clipping, leading silence, crosstalk risk, cross-channel leakage, low signal level, and a possible channel swap are **warnings**: they surface as signal, not as a scorability change -- except a loud leak, which alone also downgrades the recommendation to `scan with caution` (above). `trust` adds signal and discloses limits; the not-scorable boundary stays fixed. ## Mono via `--diarize`: is this mono file confidently separable? A mono file is not scorable by default (above). The opt-in `[diarize]` front-end changes the question: `hotato trust --stereo call.wav --diarize` reports whether the mono is confidently SEPARABLE into caller/agent -- still no turn-taking verdict. It runs the selected diarizer (`--diarizer pyannote|sortformer|pyannoteai`, default local `pyannote`), then reports a `scorability.separation` sub-block and a confidence **tier**: - **high**: confidently separable -- score it with `hotato run --mono call.wav --diarize` for a diarized-mono verdict. Exit `0`. - **low**: separable but only indicative -- voices close, overlap elevated, or the mapping balanced. `hotato run --mono ... --diarize` still scores it, but the verdict is stamped `indicative_only` and no SLA gate fires. Exit `0`. - **refuse**: not confidently separable (not two clean parties, a near-silent speaker, extreme overlap, voices too similar). Not scorable, exit `2`; next step: record dual-channel. The six signals behind the tier -- speaker count, per-speaker activity, mean segmentation posterior, embedding cluster-separation margin, overlap ratio, segment churn -- are in the `separation.signals` block. A missing extra, token, or model raises a clean error (exit 2), never a raw-mono guess. Dual-channel stays the gold reference; a diarized-mono verdict stays distinct from it. Full front-end: `docs/DIARIZE.md`. ## JSON for agents `--format json` emits one machine-parseable report. Branch on `scorable`; on a defect, read `not_scorable_reason` and `next_step`. `exit_code` mirrors the process exit. ```bash hotato trust --stereo call.wav --format json ``` ```json { "tool": "hotato", "kind": "input-health", "schema_version": "1", "source": "call.wav", "recording": { "sample_rate": 44100, "duration_sec": 32.0, "channels": 2, "clipping": { "caller": { "peak": 0.7906, "peak_dbfs": -2.1, "clipped_fraction": 0.0, "clipped": false }, "agent": { "peak": 0.9016, "peak_dbfs": -0.9, "clipped_fraction": 0.0, "clipped": false } }, "leading_silence_sec": 0.45 }, "channels": { "caller": { "channel": 0, "active_sec": 3.32, "first_speech_sec": 3.27, "has_speech": true, "enough_activity": true }, "agent": { "channel": 1, "active_sec": 23.3, "first_speech_sec": 0.45, "has_speech": true, "enough_activity": true }, "possible_swap": false, "swap_reason": null }, "crosstalk_risk": { "coherence": 0.057, "lag_sec": 0.39, "suspected": false, "leakage_db": null, "leakage_direction": null }, "scorability": { "separated_tracks": true, "enough_caller_activity": true, "enough_agent_activity": true }, "warnings": [], "scorable": true, "recommendation": "eligible for scan", "not_scorable_reason": null, "next_step": null, "exit_code": 0 } ``` Everything runs offline, on your machine, reusing hotato's existing primitives -- the hardened WAV reader, reference framing, energy VAD, and cross-channel echo coherence. It reports input-health signals only, never an accuracy percentage. ================================================================================ FILE: docs/VALIDATION.md ================================================================================ # What Hotato validates A turn handoff is not one number. Hotato is validated on **three separate jobs**, each with its own question and its own reported output. Judge each on its own terms when you decide whether to trust it. Everything below runs offline against recordings you control. Every threshold is exposed and every frame is inspectable (`hotato run --dump-frames`). --- ## Job 1: timing reproducibility **The question:** given the same recording and the same reference config, does Hotato produce the same timing measurements every run? (Deterministic for a fixed hotato version; byte-identical re-runs are verified in CI on Linux x86_64, Python 3.10-3.12 -- `.github/workflows/tests.yml`, job `pytest`. Deterministic scoring produces the same digest on Ubuntu, macOS, and Windows in CI: jobs `portability` and `determinism` run the double-run check on each OS and compare digests across all three.) **What is reported.** Per scored event: `did_yield` (true/false), `seconds_to_yield`, and `talk_over_sec`, plus the exact thresholds used (`max_talk_over`, `max_time_to_yield`) and the frame grid behind them. No learned weights, no sampling, no RNG: the energy VAD and the reference framing are deterministic, so the numbers are byte-stable. **How to check it.** Score the same file twice and diff the output. ```text $ hotato run --stereo 01-hard-interruption.example.wav --expect yield hotato [single] stack=generic offline=True 1/1 events pass (failed=0) [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s exit_code=0 $ hotato run --stereo 01-hard-interruption.example.wav --expect yield [PASS] 01-hard-interruption.example.wav: did_yield=True seconds_to_yield=0.51s talk_over=0.51s ``` `seconds_to_yield=0.51s` and `talk_over=0.51s` are identical across runs. This is the property a regression test needs: under a fixed hotato version and the same pinned audio, channel map, onset, label, and scoring config, a changed result means one of those pinned inputs changed, not that the scorer drifted. **What this job establishes.** That the measurement is stable and re-derivable by hand from [`METHODOLOGY.md`](../METHODOLOGY.md). It is a claim about stability, not that 0.51s is the absolute "true" yield latency or that the reference thresholds fit your product. --- ## Job 2: candidate-discovery usefulness **The question:** when Hotato scans a whole recording, does it surface the moments a human reviewer would want to look at, ranked by salience? **What is reported.** A ranked list of candidate turn-taking moments as **timing facts only**: overlap onsets (caller became active while the agent was talking, with the overlap length and whether the agent went silent), agent starts during caller activity, and long response gaps. Each candidate is a timestamp and a measurement -- the timing fact, reported without a verdict or an intent label. ```text $ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5 hotato scan: 02-backchannel-mhm.example.wav (6.0s, 3 candidate moments) Candidates are timing events. You decide the expected behavior; label with: hotato fixture create --onset --expect yield|hold [ 1] t=2.09s overlap_while_agent_talking overlap=1.58s agent did not go silent within 3.0s [ 2] t=3.19s overlap_while_agent_talking overlap=1.07s agent did not go silent within 3.0s [ 3] t=4.29s overlap_while_agent_talking overlap=0.56s agent did not go silent within 3.0s ``` The usefulness bar is **recall of human-notable moments at a workable candidate count**, not precision against a ground-truth intent label (no such label exists at scan time, by design). A candidate that turns out to be a harmless backchannel is simply one you label `hold` and move on. The validation artifact is the [trust gallery](TRUST-GALLERY.md): it includes a deliberate false positive, showing what an unhelpful candidate looks like and why Hotato still surfaces it. **What this job establishes.** That scan widens the net for you to make the call. It is a claim about recall, not that every candidate is a bug or that a quiet region is clean. --- ## Job 3: contract verification **The question:** once you have labelled a moment's expected behavior (`yield` = stop for the caller, `hold` = keep the floor through a backchannel), does Hotato's PASS/FAIL verdict agree with that label on the audio, against an explicit, portable, CI-enforced policy? Today this job runs on a fixture (`hotato fixture create` / `hotato run`): a labelled recording plus an explicit threshold policy, scored the same way on every CI machine. Deterministic scoring produces the same digest on Ubuntu, macOS, and Windows (jobs `portability` and `determinism`); the broader test suite is verified on Linux (see Job 1). The portable contract bundle (`hotato contract create` / `hotato contract verify` -- audio, timing evidence, trace evidence, label, policy, and a CI command, all in one artifact) carries this job forward; only the artifact changes, not the verdict's shape. **What is reported.** Per fixture: the verdict (`PASS`/`FAIL`), the measured signals behind it, and the named fix class when the failure maps cleanly to a config family. Agreement is checked against **your** label, not against an opaque key. ```text $ hotato demo --no-open --format text hotato demo: bundled calls an agent fails hotato [suite] stack=generic offline=True 0/2 events pass (failed=2) [FAIL] fd-01-missed-interruption: did_yield=False seconds_to_yield=n/a talk_over=2.65s fix[config]: Missed interruption: the agent kept talking over the caller [FAIL] fd-02-backchannel-yielded: did_yield=True seconds_to_yield=0.34s talk_over=0.32s fix[engagement-control]: False barge-in: a backchannel was treated as a bid for the floor note: no single sensitivity threshold satisfies this battery exit_code=1 ``` Both labelled failures are caught, and the battery-level note reports the disagreement plainly: a missed interruption and a false stop failing in the same run means no single threshold satisfies both. Reporting that disagreement, instead of inventing a fix, is part of the validated behavior. **What this job establishes.** That the verdict follows the audio and the label consistently. It is a claim about consistency, not that the label was correct (you own the label) or that a passing fixture means the agent is good in general. --- ## What Hotato measures, and where it stops Read this as the scope of the claim, stated once, plainly. - **Timing, not semantic intent.** Hotato measures timing; whether a caller sound meant "stop" or "mhm, go on" is a label you supply. - **A likely layer, not certainty.** A slow yield can be TTS buffering, transport, or VAD. `diagnose` names a likely layer and stays `unknown_root_cause` when one recording can't separate them. A voice trace (once the trace layer ships) narrows the candidates further, short of proof. - **Timing, not task success.** Whether the call booked the appointment, resolved the ticket, or satisfied the caller is a QA platform's job (see [COMPARE.md](COMPARE.md)). - **A demonstration, not a vendor ranking.** Hotato scores calls, not platforms. A provider-default example demonstrates the threshold funnel on one assistant, one config, one date, one scripted caller. - **Timing, not tone.** Sentiment, satisfaction, and CSAT sit outside what Hotato measures. - **Reproducible timing measurements, with the method exposed.** The three jobs above are the whole claim -- deliberately no headline percentage. The validation plan for the launch battery (external testers, consented fixtures, before/after) lives in [docs/evidence/validation-plan.md](evidence/validation-plan.md). ================================================================================ FILE: docs/WHY.md ================================================================================ # Why Hotato Four timing failures break a voice-agent call while every text-level test still passes. ## Voice agents fail in ways your tests do not catch Your test suite checks what the agent says. Production calls fail on *when* it speaks: 1. **Missed interruption.** The caller says "stop, take that off"; the agent keeps talking. `did_yield` is false where it should be true -- they repeat themselves, louder, then hang up. 2. **False stop.** The caller says "mhm" -- a backchannel, not a takeover request -- and the agent stops mid-sentence anyway. `did_yield` is true where it should be false. 3. **Slow yield.** The agent does stop, eventually. Every second of `talk_over_sec` before it does, the caller hears both voices at once. 4. **Endpointing misses.** The agent misjudges when the caller finished talking: dead air (`response_gap_sec`), or a start before they are done (`premature_start_sec`). Hotato measures how long the silence lasted, never what it meant. Each lives in audio timing, invisible to a transcript diff, an LLM judge, or a unit test on the reply -- all three score the call clean, and the caller calls back anyway. You label the expected behavior -- `yield` (stop for the caller) or `hold` (keep speaking through a backchannel or noise). Hotato measures whether the timing matched that label; intent is always yours to call. ## Rule out five look-alikes first These produce the same symptom as a turn-taking bug, but no VAD threshold touches them: - **STT hallucination** -- the transcript has words the caller never said. Check ASR word-error-rate. - **Client-side audio buffering** -- the caller's device queues audio before it reaches the agent, so "talked over me" is a transport delay, not a turn-taking one. Check WebRTC jitter-buffer/network latency. - **LLM verbosity or tool-selection** -- the agent is mid-tool-call and doesn't re-check for a stop signal. Check response-length and tool-call latency. - **Safety false-refusal** -- a moderation layer cut the agent off. Check your safety/moderation logs. - **Wrong-language STT** -- the caller's language or accent isn't covered well. Check per-locale STT accuracy -- Hotato's own detector runs on audio energy alone, so it scores the same regardless of language (`corpus/classes/README.md`). Match one of these and Hotato will not surface it -- a day saved. Match none, and agent-talks-over-caller / false-stop-on-backchannel is exactly what Hotato measures: no single threshold fixes both directions, shown on recorded calls, not synthetic fixtures (`corpus/vapi-defaults/README.md`). ## What makes Hotato different - **Scores recordings you already have.** A dual-channel WAV from your stack is the whole input -- no test harness, synthetic caller, or re-architecture. - **Runs offline.** Scoring is local and deterministic; audio, transcript, and result stay on your machine. - **Emits machine-readable timing.** One JSON envelope, the same shape from the CLI, the MCP tool, and the pytest fixture. - **Fails CI.** Exit code 1 on a regression, 0 on pass, 2 when a recording isn't scorable yet -- a turn-taking regression blocks a merge like a failing unit test. - **Routes every failure to a fix class.** `config` names the setting to change; `engagement-control` names it as a classification problem (telling "mhm" from "stop") so you don't retune a setting that can't win. ## What Hotato measures, and where it stops Hotato measures energy over time on two channels -- that is the whole method. Transcription, emotion, and intent sit outside it. A diarizer, where used, assigns anonymous SPEAKER_00/01 labels, never who a person is. A single-channel (mono) recording scores via the opt-in, quality-gated `[diarize]` front-end (`hotato run --mono call.wav --diarize`), labeled indicative below the confidence bar, with dual-channel as the reference standard. Method and ceiling: [METHODOLOGY.md](../METHODOLOGY.md) and the `limits` block of every result. ## Three numbers, reproducible by hand A single blended percentage tells you nothing about which call failed or what to change. Hotato reports three measurements per event: - `did_yield`: did the agent stop talking after the caller started - `seconds_to_yield`: seconds between the caller starting and the agent stopping - `talk_over_sec`: seconds the caller and the agent spoke at the same time Every number is reproducible from the frame dump by hand, every threshold exposed -- that is what you debug with. Ready to pin a failure? The loop from one bad call to a CI gate, with `hotato fixture create`, `compare`, and `plan`: [BAD-CALL-TO-CI.md](BAD-CALL-TO-CI.md). ================================================================================ FILE: docs/WORKSPACE.md ================================================================================ # The team workspace: `hotato serve` and `hotato console` A self-hosted, local web app for a team's voice-agent conversation QA: the live call feed (with pin-to-contract on every scored moment), suite health, failure clusters, failure records, release readiness, plus the scenario-matrix and conversation-inspector drill-ins. Stdlib-only (`http.server` + `sqlite3`) -- no framework, no build step, nothing that phones home. It serves the same fleet registry and evidence store the CLI writes, reading that data directly instead of passing a database file around. ``` hotato serve --workspace default ``` On first start it prints where it is listening, the bearer token, and the URL to open: ``` hotato serve: workspace 'default' registry: /home/you/.hotato/fleet listening: http://127.0.0.1:8321 token: Ab3xQ_p1… (generated, stored 0600 at …/serve/default/token) open: http://127.0.0.1:8321/?token=Ab3xQ_p1… audit log: /home/you/.hotato/fleet/serve/default/audit.jsonl (append-only) writes: one route -- pin-to-contract (POST /calls//pin, CSRF-fenced, audited); every view reads with SELECTs. No telemetry, no external calls. ``` Open the `open:` URL in a browser; the server sets a session cookie and redirects to strip the token from the address bar (see [Auth](#auth)). ## Flags | flag | default | meaning | |---|---|---| | `--workspace`, `-w` | `default` | workspace id to serve | | `--host` | `127.0.0.1` | bind address (see [Binding](#binding-127001-by-default)) | | `--port` | `8321` | listen port | | `--registry` | `~/.hotato/fleet` | registry home directory | | `--production-db` | none | read session manifests and alerts from this separate Hotato production SQLite database in `/health` (mode=ro; see below) | | `--score-production` | off | with `--production-db`: score completed sessions in the background into a `console.sqlite3` sidecar beside the evidence database (see below) | | `--rebuild-scores` | off | with `--production-db`: deterministically regenerate the entire `console.sqlite3` sidecar from the evidence database, then exit | | `--token` | none | supply the bearer token yourself | | `--token-file` | none | read the bearer token from a file (first line) | Exit codes: `0` clean shutdown (Ctrl-C); `2` usage error (unusable registry or token, or the port was unavailable). ## `hotato console`: the call console in one command ``` hotato console --production-db .hotato/production.sqlite3 ``` One command, one process, one browser tab: `serve` with the production evidence database wired (read-only, `mode=ro`), the score-on-arrival worker on, and the printed URL landing on the live call feed at `/calls`. Every serve behavior is identical -- loopback bind by default, bearer-token auth on every request, the append-only audit log, a `?format=json` mirror on every view -- and `--workspace`, `--host`, `--port`, `--registry`, `--token`, `--token-file`, and `--no-open` work exactly as they do on `serve`. The same exit codes apply. ## The views Every view has a machine mirror at `?format=json` (same auth, same data) so agents and scripts can drive the workspace without scraping HTML. The nav reads as one product: **Calls · Suite health · Failure clusters · Failure records · Release readiness**; the scenario matrix, one call, one conversation, and one record are drill-ins. | View | URL | Shows | |---|---|---| | **Calls** | `/calls` | The console feed over scored production calls, with the "contracts protecting this agent" count in its header (see [The call feed](#the-call-feed-calls)). | | **One call** | `/calls/` | One call's derived score record: per-dimension observations, ranked candidate moments **each with a pin-to-contract form**, the timing waterfall, evidence lanes, and the audio path as recorded. | | **Suite health** | `/health` | Your CI suite's history from the fleet registry: ingest counts, evaluated coverage, and per-dimension failure rate over time, **separated for real and simulated** conversations. Sparse days/dimensions read *not enough history* rather than a misleading point. No single combined quality score -- each dimension keeps its own number. (`/production` serves the same view for URL compatibility.) | | **Failure clusters** | `/clusters` | Failed evaluations and assertions grouped by **observable signature** (dimension + assertion kind + reason-class), with counts and drill-through into the inspector -- it groups what was observed; the cause stays yours to determine. | | **Failure records** | `/records` | The read-only Failure Record viewer over `hotato.failure-record.v1`. | | **Release readiness** | `/` | Pre-ship home screen: per-release rollup of suites/runs/evaluations -- required-suite completion, scenario/run counts, **failures by dimension** (outcome / policy / conversation / speech / reliability), inconclusive count, real-vs-simulated split, and **new-vs-fixed since the previous release**. Small samples flagged (`low sample, N=3`), never smoothed. | | **Scenario matrix** | `/scenarios` | Rows are scenarios, columns are the current and previous release, with a per-dimension status and **reliability** (`pass^k` where a scenario has repetitions). Filterable by `agent`, `release`, `suite`, `status`. | | **Conversation inspector** | `/conversation/` | One conversation: evidence manifest, transcript, trace spans, per-dimension evaluations with rationale and citations (deterministic checks and model-judged/advisory results in **separate lanes**), reviewer decisions. Every digest links to the raw evidence (`/evidence/`); redacted transcript segments and trace spans render `[redacted]`, in both HTML and JSON. | ### Optional production-evidence bridge The fleet registry and the production event store have different storage authorities. Pointing the workspace at both is explicit: ```bash hotato serve --workspace default \ --production-db .hotato/production.sqlite3 ``` `/health` then adds a separate **Production evidence plane** section with bounded session manifests, current alerts, event-source identifiers, every evidence lane's availability and authority, and the required lanes still missing. The JSON mirror exposes the same projection under `production_evidence`. The bridge opens the selected database with SQLite `mode=ro` for each request. It never constructs the writer-side `ProductionStore`, never selects the event `payload_json` column, and never imports a production row into the fleet registry. Production counts therefore stay outside `ingested_total`, the real/simulated buckets, and release trends. The production schema does not carry a fleet workspace id, so the UI states `workspace_scope = not_encoded_by_production_schema` instead of silently assigning those sessions to the workspace being served. ### Score-on-arrival (`--score-production`) ```bash hotato serve --workspace default \ --production-db .hotato/production.sqlite3 --score-production ``` A background worker in the same process polls the evidence database (same `mode=ro` read-only discipline) for sessions that reached `COMPLETE`/`QUIESCENT` and scores each one with the deterministic scorer over the session's recorded two-channel audio (the path named by the `media.asset.available` event's `data.path`). Bind and auth are unchanged; the server gains no new routes or write endpoints from this flag. Each session becomes one durable record in `console.sqlite3` beside the evidence database: - **`SCORED`** -- per-dimension observations (candidate counts and worst measured magnitude per scan kind, never blended), the ranked candidate moments, and one plain-English failure-reason sentence built only from measured numbers; - **`NOT_SCORABLE`** -- the scorer's refusal with its reason (audio lane unavailable, no recorded path, a one-channel or unreadable recording); - **`ERROR`** -- a scorer crash or persist failure on that session, with its reason; the worker records it and continues to the next session. Every record carries the scorer version and a config hash, and every timing figure derives from evidence event timestamps: per-hop latency rows keep the reporting event's declared `authority`, turn spans and the end-to-end figure are labeled `derived:event_timestamps`, and reported turn fields (`yield_latency_ms`, `overlap_ms`, `duration_ms`) stay in a separate `reported` block. Sessions are scored one at a time and a record is claimed only after its sidecar write commits. The sidecar is derived data -- the evidence database stays the only authority. `--rebuild-scores` regenerates the whole sidecar from the evidence database and exits; the same evidence database always rebuilds to identical content (the one wall-clock column, `created_at`, is excluded from the canonical comparison). A sidecar written by a different schema version is refused with that rebuild instruction. ### The call feed (`/calls`) `/calls` renders the score sidecar joined read-only with the evidence database's session metadata, newest arrival first. Each row shows the evidence-clock timestamp beside the arrival stamp (each labeled with its clock), the evidence-derived duration, the session state, the score state, the worst dimension with the measured failure-reason sentence, per-call hop-latency p50/p95 with the declared authorities behind it, and evidence-lane completeness (missing required lanes named). `SCORED`, `NOT_SCORABLE` (with its reason), and `ERROR` are all first-class rows: a refused or crashed session is shown, never hidden and never rendered as OK. Filters are query params -- `state` (session state), `scorability` (`SCORED`/`NOT_SCORABLE`/`ERROR`), and a `since`/`until` window (epoch seconds or RFC3339); a malformed filter or cursor is a 400, never a silently dropped filter. Pagination is keyset (`cursor`, `limit`) -- every page is one bounded query, and the trends strip above the table reports call volume, the score-state split, the candidate-moment share, and per-kind hop-latency p50/p95 (nearest-rank) over the whole filtered window; hop kinds keep their own percentiles and state their authorities. The feed's JSON mirror carries an `ETag` and answers a matching `If-None-Match` with a body-less 304. The page's small inline script polls that mirror every few seconds, shows an "updated Ns ago" indicator, and re-renders the rows only when the ETag changes -- no external code, and with JavaScript off the page is complete as served (reload for the latest). The feed header shows **"N contracts protecting this agent"** -- a read-only `COUNT(*)` over the fleet registry's existing `contracts` table for the serve workspace, the same table `hotato fleet` registration and the pin route write through. `/calls/` is one call: per-dimension observations (candidate counts and worst measured magnitude per scan kind, each on its own), the ranked candidate moments with their measured magnitudes and plain-English timing sentences, the timing waterfall (turn spans with derived durations kept apart from event-reported values; every hop row with its timestamp, latency, and declared authority), scorer version + config hash, the session's evidence lanes, a link to the production evidence plane, and the local audio path exactly as the evidence recorded it -- the recording stays on this machine. Where the session has finalized (`COMPLETE`/`DEGRADED`), the view also shows the exact `hotato production export-regression --out DIR --db FILE` command that exports it as an offline-verifiable regression candidate. ### Pin-to-contract (`POST /calls//pin`) Each top-ranked candidate moment on a SCORED call carries a small form: choose `expect yield` or `expect hold` and pin. The POST delegates to the same fleet machinery the CLI drives -- ingest the recorded audio (content-addressed), re-scan it, then `fleet contract create --from-candidate` semantics (`FleetAPI.contract_from_candidate`): the human label, the sealed portable `.hotato` contract bundle, and the registry registration land in one atomic step, under agent id `production` in the serve workspace. On success the page shows the contract id and bundle path (with JavaScript on, inline; with it off, as a result page), and the feed's contract count moves. A pin **refuses with its reason -- HTTP 4xx, no artifact** -- exactly where the CLI would: a NOT_SCORABLE/ERROR call, a candidate reference that does not exist, a recording that changed on disk since scoring (the re-scan must reproduce the chosen moment's onset), a stale page (the form binds the score record's `evidence_sha256`; a rebuilt sidecar refuses), or a trust-preflight refusal. The mint + label + registration step is atomic, so a refused pin leaves no contract, no label, and no registry row. The route is the server's one write action and is fenced three ways: the same bearer/cookie auth as every view (an unauthenticated POST is 401), the session cookie's `SameSite=Strict`, and a same-origin check -- a cookie-authenticated POST must carry an `Origin` (or `Referer`) header matching the request's own `Host`, so a forged cross-site form is refused 403 before any handler runs; a request presenting the bearer secret itself needs no origin header. Every attempt, accepted or refused, appends one line to the audit log. ## Auth Every request is authenticated against one shared **bearer token**: - **Browser:** open `/?token=` once; the server mints an in-memory, HttpOnly session cookie and redirects to remove the token from the URL. - **Agent / API / `curl`:** send `Authorization: Bearer `. The token is compared in constant time (`hmac.compare_digest`). Without `--token`/`--token-file`, one is generated with `secrets.token_urlsafe` on first start and stored `0600` at `/serve//token`, so a restart keeps the same URL. Sessions live only in memory, never persisted, never cross-tenant. The session cookie is `HttpOnly` + `SameSite=Strict`, and the one write route adds a same-origin check on cookie-authenticated POSTs (see [Pin-to-contract](#pin-to-contract-post-callsidpin)). ## Audit log Every request appends one JSONL line to `/serve//audit.jsonl` (created `0600`): ```json {"ts":"2026-07-12T18:04:11Z","who":"Ab3xQ_p1…","method":"GET","path":"/scenarios","query":"status=FAIL","status":200,"remote":"127.0.0.1"} ``` `who` is a token/session **prefix**, never the secret; the `token` parameter is stripped from the recorded query. The audit log is the only file the serve layer itself writes; a pin-to-contract POST additionally writes its contract bundle and registry rows through the fleet machinery, and each attempt is one audit line. ## Binding 127.0.0.1 by default The server binds loopback (`127.0.0.1`) unless you pass `--host`. A non-loopback bind (e.g. `--host 0.0.0.0`, to reach the workspace from another machine) prints a prominent warning -- it exposes the workspace to your local network. Token auth still applies; an SSH tunnel or a reverse proxy you control is the tighter choice over binding a wide interface. ## Zero egress The server only opens a **listening** socket: no outbound connection, and nothing it imports phones home -- audio, traces, and evaluations stay on the machine. A test whitelists loopback and fails if any view attempts an external connection, backed by a threat-model row. ================================================================================ FILE: docs/latency-waterfall.md ================================================================================ # Why is my voice agent's perceived latency worse than its dashboard? Because a dashboard averages each pipeline component on its own while the caller feels the sum of every hop on one specific turn: hotato renders a per-hop latency waterfall (STT, LLM, tool call, TTS, transport) derived from the trace spans of the exact call being scored, next to that call's timing verdict, so the slow hop on the turn that went wrong is a row in a table instead of a hunch. ## From a trace to the waterfall Two commands. First normalize a trace into `hotato.voice_trace.v1`; the input is a standard OTel JSON export or hotato's documented bridge JSONL (both shapes: [`docs/OTEL.md`](OTEL.md)), and the repo ships a worked example at `tests/data/otel/demo-trace.otel.jsonl`: ```console $ hotato trace ingest --otel tests/data/otel/demo-trace.otel.jsonl --out voice_trace.jsonl ingested voice trace: voice_trace.jsonl format: otel-jsonl-bridge spans: 6 stack: vapi types: agent_audio_active, asr_partial, caller_audio_active, tool_call, tts_audio_stopped, tts_cancel_requested ``` Then render the report for the recording with the trace alongside (the two-channel example recording here ships in the package): ```console $ hotato report --stereo src/hotato/data/audio/01-hard-interruption.example.wav \ --trace voice_trace.jsonl --format md --out report.md --no-fail wrote markdown report (1 events) to report.md ``` `--format md` writes Markdown; the default self-contained HTML report carries the same block. ## Read the waterfall The report's forensic section is derived entirely from evidence already in the report: the timing verdict, the attached voice trace's span timestamps, and the evaluated assertions. This is the block the commands above produce, verbatim: ```markdown ### Per-hop latency waterfall | hop | latency | derived from | | --- | --- | --- | | STT (speech to text) | 550 ms | asr_partial recognition window (first start to last end) | | LLM (first token) | not captured | no llm_first_token span in the trace | | Tool call | 320 ms | sum of 1 tool_call latency_ms | | TTS (speech out) | 300 ms | tts_audio_stopped minus tts_cancel_requested (cancellation lag) | | Transport (backend HTTP) | not captured | no http_exchange spans in the trace | ``` Each row names the spans it was derived from, so every number is traceable to a timestamp you supplied. A hop with no spans reads `not captured`, never estimated: the example trace carries no `llm_first_token` and no `http_exchange` spans, so those rows say so. A pipeline that emits all five span kinds gets all five hops measured. The trace stays context, not a score. The report states it in place: the spans are rendered alongside the timing measurement, and `did_yield`, talk-over, time to yield, and the PASS/FAIL verdict are unaffected by anything in the trace. ## Pin it to a failure The same trace file attaches to a pinned failure so the waterfall's source spans travel with the evidence: ```bash hotato trace attach contracts/demo-missed-interruption.hotato --trace voice_trace.jsonl ``` That writes the trace into the contract bundle and re-renders its evidence timeline with an aligned trace row. The full trace path (ingest, attach, export, redaction defaults) is [`docs/TRACE.md`](TRACE.md); the report surfaces in depth are [`docs/REPORTS.md`](REPORTS.md). ================================================================================ FILE: src/hotato/schema/counterexample-oracle.v1.json ================================================================================ { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "https://hotato.dev/schema/counterexample-oracle.v1.json", "title": "Hotato deterministic counterexample oracle", "type": "object", "additionalProperties": false, "required": ["kind", "version", "authority", "ci_gate_eligible", "target", "observation_scope"], "properties": { "kind": {"const": "hotato.counterexample-oracle.v1"}, "version": {"const": 1}, "authority": {"const": "deterministic"}, "ci_gate_eligible": {"const": true}, "target": {"$ref": "#/definitions/target"}, "observation_scope": {"$ref": "#/definitions/observation_scope"} }, "definitions": { "digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}, "nonnegative_integer": {"type": "integer", "minimum": 0}, "dimension": { "type": ["string", "null"], "enum": ["outcome", "policy", "conversation", "speech", "reliability", null] }, "assertion_kind": { "enum": [ "phrase", "pii", "policy", "tool_call", "outcome", "tool_result", "tool_error", "state", "state_change", "handoff", "termination", "latency", "entity_accuracy", "sequence", "count" ] }, "failure_atom": { "$comment": "Closed payload-free branch identity; kind coupling is enforced by the target schema.", "oneOf": [ { "type": "object", "additionalProperties": false, "required": ["code"], "properties": { "code": {"enum": [ "forbidden-match", "no-qualifying-turns", "required-match-missing", "tool-missing", "tool-arguments-missing", "tool-count-below", "tool-count-above", "never-before-boundary-missing", "never-before-order-violation", "no-predicate-met", "result-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch", "state-record-missing", "unexpected-handoff", "handoff-missing", "handoff-target-mismatch", "unexpected-termination", "termination-missing", "latency-declared-threshold-exceeded", "latency-default-threshold-exceeded", "count-below", "count-above" ]} } }, { "type": "object", "additionalProperties": false, "required": ["code", "index"], "properties": { "code": {"enum": ["order-step-absent", "order-step-out-of-order", "predicate-unmet", "sequence-step-absent", "sequence-step-out-of-order"]}, "index": {"$ref": "#/definitions/nonnegative_integer"} } }, { "type": "object", "additionalProperties": false, "required": ["code", "detector"], "properties": { "code": {"const": "pii-detected"}, "detector": {"enum": ["ssn", "card_luhn", "email", "phone"]} } }, { "type": "object", "additionalProperties": false, "required": ["code", "rule", "type"], "properties": { "code": {"const": "policy-violation"}, "rule": {"type": "string", "minLength": 1}, "type": {"enum": ["banned", "required_disclosure_missing"]} } }, { "type": "object", "additionalProperties": false, "required": ["code", "field"], "properties": { "code": {"enum": ["state-field-missing", "state-field-value-mismatch", "before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged", "termination-attribute-missing", "termination-attribute-value-mismatch"]}, "field": {"type": "string", "minLength": 1} } }, { "type": "object", "additionalProperties": false, "required": ["code", "key"], "properties": { "code": {"enum": ["entity-missing", "entity-value-mismatch", "tool-argument-field-missing", "tool-argument-value-mismatch", "result-field-missing", "result-field-value-mismatch"]}, "key": {"type": "string", "minLength": 1} } } ] }, "target_kind_atom_coupling": { "allOf": [ { "if": {"properties": {"kind": {"const": "phrase"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_phrase"} }, { "if": {"properties": {"kind": {"const": "pii"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_pii"} }, { "if": {"properties": {"kind": {"const": "policy"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_policy"} }, { "if": {"properties": {"kind": {"const": "tool_call"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_tool_call"} }, { "if": {"properties": {"kind": {"const": "outcome"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_outcome"} }, { "if": {"properties": {"kind": {"const": "tool_result"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_tool_result"} }, { "if": {"properties": {"kind": {"const": "tool_error"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_tool_error"} }, { "if": {"properties": {"kind": {"const": "state"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_state"} }, { "if": {"properties": {"kind": {"const": "state_change"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_state_change"} }, { "if": {"properties": {"kind": {"const": "handoff"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_handoff"} }, { "if": {"properties": {"kind": {"const": "termination"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_termination"} }, { "if": {"properties": {"kind": {"const": "latency"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_latency"} }, { "if": {"properties": {"kind": {"const": "entity_accuracy"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_entity_accuracy"} }, { "if": {"properties": {"kind": {"const": "sequence"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_sequence"} }, { "if": {"properties": {"kind": {"const": "count"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_count"} } ] }, "target_atom_codes_phrase": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}}} } }, "target_atom_codes_pii": { "properties": { "failure_atom": {"properties": {"code": {"const": "pii-detected"}}}, "source_failure_atoms": {"items": {"properties": {"code": {"const": "pii-detected"}}}} } }, "target_atom_codes_policy": { "properties": { "failure_atom": {"properties": {"code": {"const": "policy-violation"}}}, "source_failure_atoms": {"items": {"properties": {"code": {"const": "policy-violation"}}}} } }, "target_atom_codes_tool_call": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}}} } }, "target_atom_codes_outcome": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}}} } }, "target_atom_codes_tool_result": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}}} } }, "target_atom_codes_tool_error": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}}} } }, "target_atom_codes_state": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}}} } }, "target_atom_codes_state_change": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}}} } }, "target_atom_codes_handoff": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}}} } }, "target_atom_codes_termination": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}}} } }, "target_atom_codes_latency": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}}} } }, "target_atom_codes_entity_accuracy": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}}} } }, "target_atom_codes_sequence": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}}} } }, "target_atom_codes_count": { "properties": { "failure_atom": {"properties": {"code": {"enum": ["count-below", "count-above"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["count-below", "count-above"]}}}} } }, "target": { "$comment": "Runtime validation also requires canonical UTF-8 atom ordering, de-duplication, selected atom first, and a matching fingerprint.", "type": "object", "additionalProperties": false, "required": [ "test_id", "assertion_digest", "assertion_id", "kind", "dimension", "authority", "required_status", "failure_atom", "source_failure_atoms", "fingerprint" ], "properties": { "test_id": {"type": "string", "minLength": 1}, "assertion_digest": {"$ref": "#/definitions/digest"}, "assertion_id": {"type": "string", "minLength": 1}, "kind": {"$ref": "#/definitions/assertion_kind"}, "dimension": {"$ref": "#/definitions/dimension"}, "authority": {"const": "deterministic"}, "required_status": {"const": "FAIL"}, "failure_atom": {"$ref": "#/definitions/failure_atom"}, "source_failure_atoms": { "type": "array", "minItems": 1, "uniqueItems": true, "items": {"$ref": "#/definitions/failure_atom"} }, "fingerprint": {"$ref": "#/definitions/digest"} }, "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}] }, "observation_scope": { "type": "object", "additionalProperties": false, "required": ["frozen_components", "rule", "minimum_caller_turns", "transforms"], "properties": { "frozen_components": { "type": "array", "uniqueItems": true, "items": {"enum": ["script", "tools", "state", "handoff", "termination"]} }, "rule": { "const": "candidate is a complete schema-valid scripted session and must preserve the source-selected structured failure branch" }, "minimum_caller_turns": {"const": 1}, "transforms": {"const": "deletion-only"} } } } } ================================================================================ FILE: src/hotato/schema/reduction-certificate.v1.json ================================================================================ { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "https://hotato.dev/schema/reduction-certificate.v1.json", "title": "Hotato counterexample reduction certificate", "type": "object", "additionalProperties": false, "required": [ "kind", "version", "algorithm", "reducer_set", "source_scenario_digest", "final_scenario_digest", "source_test_digest", "final_test_digest", "failure_fingerprint", "accepted_steps", "journal_sha256", "budget", "candidate_evaluations", "cache_hits", "termination" ], "properties": { "kind": {"const": "hotato.reduction-certificate.v1"}, "version": {"const": 1}, "algorithm": {"const": "hierarchical-ddmin"}, "reducer_set": {"const": "hotato.reducers.v1"}, "source_scenario_digest": {"$ref": "#/definitions/digest"}, "final_scenario_digest": {"$ref": "#/definitions/digest"}, "source_test_digest": {"$ref": "#/definitions/digest"}, "final_test_digest": {"$ref": "#/definitions/digest"}, "failure_fingerprint": {"$ref": "#/definitions/digest"}, "accepted_steps": { "type": "array", "maxItems": 512, "items": {"$ref": "#/definitions/accepted_step"} }, "journal_sha256": {"$ref": "#/definitions/digest"}, "budget": {"type": "integer", "minimum": 1, "maximum": 100000}, "candidate_evaluations": {"type": "integer", "minimum": 0, "maximum": 100000}, "cache_hits": {"type": "integer", "minimum": 0}, "termination": {"enum": ["one_minimal", "budget_exhausted"]} }, "definitions": { "digest": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, "raw_digest": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, "path_segment": { "oneOf": [ {"type": "string"}, {"type": "integer", "minimum": 0} ] }, "deletion_operation": { "type": "object", "additionalProperties": false, "required": ["op", "path", "removed_digest"], "properties": { "op": {"const": "remove"}, "path": { "type": "array", "minItems": 1, "maxItems": 96, "items": {"$ref": "#/definitions/path_segment"} }, "removed_digest": {"$ref": "#/definitions/digest"} } }, "deletion_transform": { "type": "object", "additionalProperties": false, "required": ["kind", "operations"], "properties": { "kind": {"const": "hotato.delete-only.v1"}, "operations": { "type": "array", "minItems": 1, "maxItems": 10000, "uniqueItems": true, "items": {"$ref": "#/definitions/deletion_operation"} } } }, "remove_field": { "type": "object", "additionalProperties": false, "required": ["kind", "phase", "path"], "properties": { "kind": {"const": "remove-field"}, "phase": {"type": "string", "minLength": 1}, "path": {"type": "string", "minLength": 1} } }, "remove_path_set": { "type": "object", "additionalProperties": false, "required": ["kind", "phase", "paths"], "properties": { "kind": {"const": "remove-path-set"}, "phase": {"type": "string", "minLength": 1}, "paths": { "type": "array", "minItems": 1, "maxItems": 10000, "uniqueItems": true, "items": {"type": "string", "minLength": 1} } } }, "component": { "enum": [ "top-level-optional", "goal-optional", "caller-optional", "variation_matrix", "facts", "environment", "interruptions", "behavior", "script", "script-field", "tools", "tool-field", "handoff-field", "handoff", "termination-field", "termination", "state", "agent-mock-optional", "agent_mock" ] }, "remove_single_unit": { "type": "object", "additionalProperties": false, "required": ["kind", "phase", "path", "component"], "properties": { "kind": {"const": "remove-single-unit"}, "phase": {"enum": ["one-minimality", "minimality-proof-restart"]}, "path": {"type": "string", "minLength": 1}, "component": {"$ref": "#/definitions/component"} } }, "reducer_operation": { "oneOf": [ {"$ref": "#/definitions/remove_field"}, {"$ref": "#/definitions/remove_path_set"}, {"$ref": "#/definitions/remove_single_unit"} ] }, "accepted_step": { "type": "object", "additionalProperties": false, "required": [ "step", "parent_digest", "child_digest", "operation", "transform", "oracle_result_digest", "failure_atom_digest" ], "properties": { "step": {"type": "integer", "minimum": 1, "maximum": 512}, "parent_digest": {"$ref": "#/definitions/raw_digest"}, "child_digest": {"$ref": "#/definitions/raw_digest"}, "operation": {"$ref": "#/definitions/reducer_operation"}, "transform": {"$ref": "#/definitions/deletion_transform"}, "oracle_result_digest": {"$ref": "#/definitions/digest"}, "failure_atom_digest": {"$ref": "#/definitions/digest"} } } } } ================================================================================ FILE: src/hotato/schema/counterexample.v1.json ================================================================================ { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "https://hotato.dev/schema/counterexample.v1.json", "title": "Hotato proof-preserving counterexample capsule", "type": "object", "additionalProperties": false, "required": [ "kind", "version", "counterexample_id", "source", "target", "reduction", "minimality", "preservation", "privacy", "provenance" ], "properties": { "kind": {"const": "hotato.counterexample.v1"}, "version": {"const": 1}, "counterexample_id": {"$ref": "#/definitions/digest"}, "source": {}, "target": {}, "oracle": {"$ref": "#/definitions/oracle_reference"}, "artifacts": {"$ref": "#/definitions/artifacts"}, "artifact_digests": {"$ref": "#/definitions/artifact_digests"}, "reduction": {"$ref": "#/definitions/reduction"}, "minimality": {}, "preservation": {"$ref": "#/definitions/preservation"}, "privacy": {}, "provenance": {} }, "allOf": [ { "oneOf": [ { "required": ["oracle", "artifacts", "artifact_digests"], "properties": { "source": {"$ref": "#/definitions/private_source"}, "target": {"$ref": "#/definitions/private_target"}, "minimality": {"$ref": "#/definitions/private_minimality"}, "privacy": {"$ref": "#/definitions/private_privacy"}, "provenance": {"$ref": "#/definitions/private_provenance"} } }, { "not": { "anyOf": [ {"required": ["oracle"]}, {"required": ["artifacts"]}, {"required": ["artifact_digests"]} ] }, "properties": { "source": {"$ref": "#/definitions/share_source"}, "target": {"$ref": "#/definitions/share_target"}, "minimality": {"$ref": "#/definitions/share_minimality"}, "privacy": {"$ref": "#/definitions/share_privacy"}, "provenance": {"$ref": "#/definitions/share_provenance"} } } ] }, { "oneOf": [ { "properties": { "reduction": {"properties": {"termination": {"const": "one_minimal"}}}, "minimality": {"properties": {"status": {"const": "one_minimal"}}} } }, { "properties": { "reduction": {"properties": {"termination": {"const": "budget_exhausted"}}}, "minimality": {"properties": {"status": {"const": "budget_exhausted"}}} } } ] } ], "definitions": { "digest": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, "raw_digest": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, "dimension": { "type": ["string", "null"], "enum": ["outcome", "policy", "conversation", "speech", "reliability", null] }, "assertion_kind": { "enum": [ "phrase", "pii", "policy", "tool_call", "outcome", "tool_result", "tool_error", "state", "state_change", "handoff", "termination", "latency", "entity_accuracy", "sequence", "count" ] }, "nonnegative_integer": {"type": "integer", "minimum": 0}, "failure_atom": { "$comment": "Closed payload-free branch identity; kind coupling is enforced by the target schema.", "oneOf": [ { "type": "object", "additionalProperties": false, "required": ["code"], "properties": { "code": {"enum": [ "forbidden-match", "no-qualifying-turns", "required-match-missing", "tool-missing", "tool-arguments-missing", "tool-count-below", "tool-count-above", "never-before-boundary-missing", "never-before-order-violation", "no-predicate-met", "result-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch", "state-record-missing", "unexpected-handoff", "handoff-missing", "handoff-target-mismatch", "unexpected-termination", "termination-missing", "latency-declared-threshold-exceeded", "latency-default-threshold-exceeded", "count-below", "count-above" ]} } }, { "type": "object", "additionalProperties": false, "required": ["code", "index"], "properties": { "code": {"enum": ["order-step-absent", "order-step-out-of-order", "predicate-unmet", "sequence-step-absent", "sequence-step-out-of-order"]}, "index": {"$ref": "#/definitions/nonnegative_integer"} } }, { "type": "object", "additionalProperties": false, "required": ["code", "detector"], "properties": { "code": {"const": "pii-detected"}, "detector": {"enum": ["ssn", "card_luhn", "email", "phone"]} } }, { "type": "object", "additionalProperties": false, "required": ["code", "rule", "type"], "properties": { "code": {"const": "policy-violation"}, "rule": {"type": "string", "minLength": 1}, "type": {"enum": ["banned", "required_disclosure_missing"]} } }, { "type": "object", "additionalProperties": false, "required": ["code", "field"], "properties": { "code": {"enum": ["state-field-missing", "state-field-value-mismatch", "before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged", "termination-attribute-missing", "termination-attribute-value-mismatch"]}, "field": {"type": "string", "minLength": 1} } }, { "type": "object", "additionalProperties": false, "required": ["code", "key"], "properties": { "code": {"enum": ["entity-missing", "entity-value-mismatch", "tool-argument-field-missing", "tool-argument-value-mismatch", "result-field-missing", "result-field-value-mismatch"]}, "key": {"type": "string", "minLength": 1} } } ] }, "target_kind_atom_coupling": { "allOf": [ { "if": {"properties": {"kind": {"const": "phrase"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_phrase"} }, { "if": {"properties": {"kind": {"const": "pii"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_pii"} }, { "if": {"properties": {"kind": {"const": "policy"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_policy"} }, { "if": {"properties": {"kind": {"const": "tool_call"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_tool_call"} }, { "if": {"properties": {"kind": {"const": "outcome"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_outcome"} }, { "if": {"properties": {"kind": {"const": "tool_result"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_tool_result"} }, { "if": {"properties": {"kind": {"const": "tool_error"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_tool_error"} }, { "if": {"properties": {"kind": {"const": "state"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_state"} }, { "if": {"properties": {"kind": {"const": "state_change"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_state_change"} }, { "if": {"properties": {"kind": {"const": "handoff"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_handoff"} }, { "if": {"properties": {"kind": {"const": "termination"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_termination"} }, { "if": {"properties": {"kind": {"const": "latency"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_latency"} }, { "if": {"properties": {"kind": {"const": "entity_accuracy"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_entity_accuracy"} }, { "if": {"properties": {"kind": {"const": "sequence"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_sequence"} }, { "if": {"properties": {"kind": {"const": "count"}}, "required": ["kind"]}, "then": {"$ref": "#/definitions/target_atom_codes_count"} } ] }, "target_atom_codes_phrase": { "properties": { "failure_code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}, "failure_atom": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["forbidden-match", "no-qualifying-turns", "required-match-missing"]}}}} } }, "target_atom_codes_pii": { "properties": { "failure_code": {"const": "pii-detected"}, "failure_atom": {"properties": {"code": {"const": "pii-detected"}}}, "source_failure_atoms": {"items": {"properties": {"code": {"const": "pii-detected"}}}} } }, "target_atom_codes_policy": { "properties": { "failure_code": {"const": "policy-violation"}, "failure_atom": {"properties": {"code": {"const": "policy-violation"}}}, "source_failure_atoms": {"items": {"properties": {"code": {"const": "policy-violation"}}}} } }, "target_atom_codes_tool_call": { "properties": { "failure_code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}, "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "tool-arguments-missing", "tool-argument-field-missing", "tool-argument-value-mismatch", "tool-count-below", "tool-count-above", "order-step-absent", "order-step-out-of-order", "never-before-boundary-missing", "never-before-order-violation"]}}}} } }, "target_atom_codes_outcome": { "properties": { "failure_code": {"enum": ["predicate-unmet", "no-predicate-met"]}, "failure_atom": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["predicate-unmet", "no-predicate-met"]}}}} } }, "target_atom_codes_tool_result": { "properties": { "failure_code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}, "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "result-missing", "result-field-missing", "result-field-value-mismatch"]}}}} } }, "target_atom_codes_tool_error": { "properties": { "failure_code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}, "failure_atom": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["tool-missing", "unexpected-tool-error", "tool-error-missing", "tool-error-pattern-mismatch"]}}}} } }, "target_atom_codes_state": { "properties": { "failure_code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}, "failure_atom": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["state-record-missing", "state-field-missing", "state-field-value-mismatch"]}}}} } }, "target_atom_codes_state_change": { "properties": { "failure_code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}, "failure_atom": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["before-field-missing", "before-value-mismatch", "after-field-missing", "after-value-mismatch", "state-unchanged"]}}}} } }, "target_atom_codes_handoff": { "properties": { "failure_code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}, "failure_atom": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-handoff", "handoff-missing", "handoff-target-mismatch"]}}}} } }, "target_atom_codes_termination": { "properties": { "failure_code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}, "failure_atom": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["unexpected-termination", "termination-missing", "termination-attribute-missing", "termination-attribute-value-mismatch"]}}}} } }, "target_atom_codes_latency": { "properties": { "failure_code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}, "failure_atom": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["latency-declared-threshold-exceeded", "latency-default-threshold-exceeded"]}}}} } }, "target_atom_codes_entity_accuracy": { "properties": { "failure_code": {"enum": ["entity-missing", "entity-value-mismatch"]}, "failure_atom": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["entity-missing", "entity-value-mismatch"]}}}} } }, "target_atom_codes_sequence": { "properties": { "failure_code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}, "failure_atom": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["sequence-step-absent", "sequence-step-out-of-order"]}}}} } }, "target_atom_codes_count": { "properties": { "failure_code": {"enum": ["count-below", "count-above"]}, "failure_atom": {"properties": {"code": {"enum": ["count-below", "count-above"]}}}, "source_failure_atoms": {"items": {"properties": {"code": {"enum": ["count-below", "count-above"]}}}} } }, "private_target": { "$comment": "Runtime validation also requires canonical UTF-8 atom ordering, de-duplication, selected atom first, and a matching fingerprint.", "type": "object", "additionalProperties": false, "required": [ "test_id", "assertion_digest", "assertion_id", "kind", "dimension", "authority", "required_status", "failure_atom", "source_failure_atoms", "fingerprint" ], "properties": { "test_id": {"type": "string", "minLength": 1}, "assertion_digest": {"$ref": "#/definitions/digest"}, "assertion_id": {"type": "string", "minLength": 1}, "kind": {"$ref": "#/definitions/assertion_kind"}, "dimension": {"$ref": "#/definitions/dimension"}, "authority": {"const": "deterministic"}, "required_status": {"const": "FAIL"}, "failure_atom": {"$ref": "#/definitions/failure_atom"}, "source_failure_atoms": { "type": "array", "minItems": 1, "uniqueItems": true, "items": {"$ref": "#/definitions/failure_atom"} }, "fingerprint": {"$ref": "#/definitions/digest"} }, "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}] }, "share_target": { "$comment": "failure_code is the selected private atom code; failure_atom_digest binds the complete private atom.", "type": "object", "additionalProperties": false, "required": ["assertion_ref", "kind", "dimension", "authority", "required_status", "fingerprint", "failure_code", "failure_atom_digest"], "properties": { "assertion_ref": {"$ref": "#/definitions/digest"}, "kind": {"$ref": "#/definitions/assertion_kind"}, "dimension": {"$ref": "#/definitions/dimension"}, "authority": {"const": "deterministic"}, "required_status": {"const": "FAIL"}, "fingerprint": {"$ref": "#/definitions/digest"}, "failure_code": {"type": "string", "minLength": 1}, "failure_atom_digest": {"$ref": "#/definitions/digest"} }, "allOf": [{"$ref": "#/definitions/target_kind_atom_coupling"}] }, "private_source": { "type": "object", "additionalProperties": false, "required": ["scenario_file_sha256", "test_file_sha256", "scenario_digest", "test_digest"], "properties": { "scenario_file_sha256": {"$ref": "#/definitions/digest"}, "test_file_sha256": {"$ref": "#/definitions/digest"}, "scenario_digest": {"$ref": "#/definitions/digest"}, "test_digest": {"$ref": "#/definitions/digest"} } }, "share_source": { "type": "object", "additionalProperties": false, "required": ["scenario_digest", "test_digest"], "properties": { "scenario_digest": {"$ref": "#/definitions/digest"}, "test_digest": {"$ref": "#/definitions/digest"} } }, "oracle_reference": { "type": "object", "additionalProperties": false, "required": ["path", "digest"], "properties": {"path": {"const": "oracle.json"}, "digest": {"$ref": "#/definitions/digest"}} }, "artifacts": { "type": "object", "additionalProperties": false, "required": [ "source_scenario", "source_conversation_test", "source_scenario_file", "source_conversation_test_file", "scenario", "conversation_test", "expected_result", "certificate", "journal", "minimality", "report_markdown", "report_html", "share_card", "reproduce_script", "predicate_script" ], "properties": { "source_scenario": {"const": "source/scenario.json"}, "source_conversation_test": {"const": "source/conversation-test.json"}, "source_scenario_file": {"const": "source/scenario.original"}, "source_conversation_test_file": {"const": "source/conversation-test.original"}, "scenario": {"const": "input/scenario.json"}, "conversation_test": {"const": "input/conversation-test.json"}, "expected_result": {"const": "expected/assertion-result.json"}, "certificate": {"const": "certificate.json"}, "journal": {"const": "reduction.jsonl"}, "minimality": {"const": "minimality.json"}, "report_markdown": {"const": "report.md"}, "report_html": {"const": "report.html"}, "share_card": {"const": "card.svg"}, "reproduce_script": {"const": "reproduce.sh"}, "predicate_script": {"const": "predicate.sh"} } }, "artifact_digests": { "type": "object", "additionalProperties": false, "required": [ "source_scenario", "source_conversation_test", "source_scenario_file", "source_conversation_test_file", "scenario", "conversation_test", "expected_result", "certificate", "journal", "minimality", "report_markdown", "report_html", "share_card", "reproduce_script", "predicate_script" ], "properties": { "source_scenario": {"$ref": "#/definitions/digest"}, "source_conversation_test": {"$ref": "#/definitions/digest"}, "source_scenario_file": {"$ref": "#/definitions/digest"}, "source_conversation_test_file": {"$ref": "#/definitions/digest"}, "scenario": {"$ref": "#/definitions/digest"}, "conversation_test": {"$ref": "#/definitions/digest"}, "expected_result": {"$ref": "#/definitions/digest"}, "certificate": {"$ref": "#/definitions/digest"}, "journal": {"$ref": "#/definitions/digest"}, "minimality": {"$ref": "#/definitions/digest"}, "report_markdown": {"$ref": "#/definitions/digest"}, "report_html": {"$ref": "#/definitions/digest"}, "share_card": {"$ref": "#/definitions/digest"}, "reproduce_script": {"$ref": "#/definitions/digest"}, "predicate_script": {"$ref": "#/definitions/digest"} } }, "stats": { "type": "object", "additionalProperties": false, "required": ["bytes", "turns", "tools", "state_leaves", "transcript_segments", "trace_spans"], "properties": { "bytes": {"type": "integer", "minimum": 1}, "turns": {"type": "integer", "minimum": 1, "maximum": 10000}, "tools": {"type": "integer", "minimum": 0, "maximum": 10000}, "state_leaves": {"$ref": "#/definitions/nonnegative_integer"}, "transcript_segments": {"$ref": "#/definitions/nonnegative_integer"}, "trace_spans": {"$ref": "#/definitions/nonnegative_integer"} } }, "reduction": { "type": "object", "additionalProperties": false, "required": [ "algorithm", "reducer_set", "initial", "final", "attempts", "candidate_evaluations", "qualification_evaluations", "total_evaluations", "accepted", "cache_hits", "budget", "termination" ], "properties": { "algorithm": {"const": "hierarchical-ddmin"}, "reducer_set": {"const": "hotato.reducers.v1"}, "initial": {"$ref": "#/definitions/stats"}, "final": {"$ref": "#/definitions/stats"}, "attempts": {"$ref": "#/definitions/nonnegative_integer"}, "candidate_evaluations": {"type": "integer", "minimum": 0, "maximum": 100000}, "qualification_evaluations": {"const": 4}, "total_evaluations": {"type": "integer", "minimum": 4, "maximum": 100004}, "accepted": {"type": "integer", "minimum": 0, "maximum": 512}, "cache_hits": {"$ref": "#/definitions/nonnegative_integer"}, "budget": {"type": "integer", "minimum": 1, "maximum": 100000}, "termination": {"enum": ["one_minimal", "budget_exhausted"]} } }, "minimality_check": { "type": "object", "additionalProperties": false, "required": ["path", "component", "outcome", "code", "candidate_digest"], "properties": { "path": {"type": "string", "minLength": 1}, "component": {"enum": [ "top-level-optional", "goal-optional", "caller-optional", "variation_matrix", "facts", "environment", "interruptions", "behavior", "script", "script-field", "tools", "tool-field", "handoff-field", "handoff", "termination-field", "termination", "state", "agent-mock-optional", "agent_mock" ]}, "outcome": {"enum": ["PRESERVED", "ABSENT", "DRIFTED", "UNRESOLVED"]}, "code": {"enum": [ null, "budget_exhausted", "simulator_invalid", "assertion_contract_invalid", "assertion_inconclusive", "target_failed", "target_absent", "candidate_invalid", "failure_identity_drift", "failure_atom_unavailable", "resource_limit_exceeded" ]}, "candidate_digest": {"$ref": "#/definitions/raw_digest"} } }, "private_minimality": { "type": "object", "additionalProperties": false, "required": ["status", "reducer_set", "claim", "remaining_unit_checks", "frozen_components"], "properties": { "status": {"enum": ["one_minimal", "budget_exhausted"]}, "reducer_set": {"const": "hotato.reducers.v1"}, "claim": {"type": "string", "minLength": 1}, "remaining_unit_checks": {"type": "array", "maxItems": 512, "items": {"$ref": "#/definitions/minimality_check"}}, "frozen_components": { "type": "array", "uniqueItems": true, "items": {"enum": ["script", "tools", "state", "handoff", "termination"]} } }, "allOf": [ { "if": {"properties": {"status": {"const": "one_minimal"}}}, "then": {"properties": {"claim": {"const": "1-minimal under hotato.reducers.v1 with the recorded observation-scope freezes"}}} }, { "if": {"properties": {"status": {"const": "budget_exhausted"}}}, "then": {"properties": {"claim": {"const": "failure preserved; minimality incomplete because the candidate budget ended"}}} } ] }, "share_minimality": { "type": "object", "additionalProperties": false, "required": ["status", "reducer_set", "claim", "check_summary", "frozen_components"], "properties": { "status": {"enum": ["one_minimal", "budget_exhausted"]}, "reducer_set": {"const": "hotato.reducers.v1"}, "claim": {"type": "string", "minLength": 1}, "check_summary": { "type": "object", "additionalProperties": false, "required": ["count", "outcomes"], "properties": { "count": {"type": "integer", "minimum": 0, "maximum": 512}, "outcomes": { "type": "object", "additionalProperties": false, "required": ["PRESERVED", "ABSENT", "DRIFTED", "UNRESOLVED"], "properties": { "PRESERVED": {"$ref": "#/definitions/nonnegative_integer"}, "ABSENT": {"$ref": "#/definitions/nonnegative_integer"}, "DRIFTED": {"$ref": "#/definitions/nonnegative_integer"}, "UNRESOLVED": {"$ref": "#/definitions/nonnegative_integer"} } } } }, "frozen_components": { "type": "array", "uniqueItems": true, "items": {"enum": ["script", "tools", "state", "handoff", "termination"]} } }, "allOf": [ { "if": {"properties": {"status": {"const": "one_minimal"}}}, "then": {"properties": {"claim": {"const": "1-minimal under hotato.reducers.v1 with the recorded observation-scope freezes"}}} }, { "if": {"properties": {"status": {"const": "budget_exhausted"}}}, "then": {"properties": {"claim": {"const": "failure preserved; minimality incomplete because the candidate budget ended"}}} } ] }, "preservation": { "type": "object", "additionalProperties": false, "required": [ "source_executions", "source_matching_failures", "final_executions", "final_matching_failures", "source_result_digests", "final_result_digests", "source_content_hashes", "final_content_hashes", "source_trace_digests", "final_trace_digests", "same_failure_fingerprint" ], "properties": { "source_executions": {"const": 2}, "source_matching_failures": {"const": 2}, "final_executions": {"const": 2}, "final_matching_failures": {"const": 2}, "source_result_digests": { "type": "array", "minItems": 2, "maxItems": 2, "items": {"$ref": "#/definitions/digest"} }, "final_result_digests": { "type": "array", "minItems": 2, "maxItems": 2, "items": {"$ref": "#/definitions/digest"} }, "source_content_hashes": { "type": "array", "minItems": 2, "maxItems": 2, "items": {"$ref": "#/definitions/raw_digest"} }, "final_content_hashes": { "type": "array", "minItems": 2, "maxItems": 2, "items": {"$ref": "#/definitions/raw_digest"} }, "source_trace_digests": { "type": "array", "minItems": 2, "maxItems": 2, "items": {"$ref": "#/definitions/digest"} }, "final_trace_digests": { "type": "array", "minItems": 2, "maxItems": 2, "items": {"$ref": "#/definitions/digest"} }, "same_failure_fingerprint": {"const": true} } }, "private_privacy": { "type": "object", "additionalProperties": false, "required": ["profile", "content_included", "network_egress", "hashes_are_correlators"], "properties": { "profile": {"const": "private-runnable-v1"}, "content_included": {"const": ["scenario", "conversation_test", "assertion_result"]}, "network_egress": {"const": false}, "hashes_are_correlators": {"const": true} } }, "share_privacy": { "type": "object", "additionalProperties": false, "required": ["profile", "content_included", "omitted", "runnable", "hashes_are_correlators"], "properties": { "profile": {"const": "share-safe-v1"}, "content_included": {"const": []}, "omitted": { "const": [ "audio", "transcript_body", "scenario_body", "assertion_body", "tool_payload", "state_value", "credentials", "absolute_paths", "provider_identifiers", "deletion_paths" ] }, "runnable": {"const": false}, "hashes_are_correlators": {"const": true} } }, "evaluator_digest": {"$ref": "#/definitions/digest"}, "version_string": { "type": "string", "pattern": "^[0-9]+(?:\\.[0-9]+){1,3}(?:[A-Za-z0-9._+-]*)?$" }, "scenario_selection": { "type": "object", "additionalProperties": false, "required": ["mode", "seed", "variation_matrix_applied"], "properties": { "mode": {"const": "base-scenario"}, "seed": {"type": "integer", "minimum": 0}, "variation_matrix_applied": {"const": false} } }, "private_provenance": { "type": "object", "additionalProperties": false, "required": ["hotato_version", "evaluator_digest", "seed", "scenario_selection"], "properties": { "hotato_version": {"$ref": "#/definitions/version_string"}, "evaluator_digest": {"$ref": "#/definitions/evaluator_digest"}, "seed": {"type": "integer", "minimum": 0}, "scenario_selection": {"$ref": "#/definitions/scenario_selection"} } }, "share_provenance": { "type": "object", "additionalProperties": false, "required": ["hotato_version", "evaluator_digest", "scenario_selection"], "properties": { "hotato_version": {"$ref": "#/definitions/version_string"}, "evaluator_digest": {"$ref": "#/definitions/evaluator_digest"}, "scenario_selection": {"$ref": "#/definitions/scenario_selection"} } } } } ================================================================================ FILE: src/hotato/schema/envelope.v1.json ================================================================================ { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "https://hotato.dev/schema/envelope.v1.json", "title": "hotato result envelope, schema_version 1", "description": "The single machine-readable shape emitted by the CLI, the one MCP tool, and the suite runner. Additive-only: new signal dimensions and sub-verdicts may appear as extra keys without a version bump, so consumers must ignore unknown fields. The stable core below is frozen for schema_version 1.", "type": "object", "required": ["tool", "schema_version", "mode", "stack", "offline", "engine", "limits", "summary", "events", "fix_map", "funnel", "exit_code"], "additionalProperties": true, "properties": { "tool": { "const": "hotato" }, "schema_version": { "const": "1" }, "mode": { "enum": ["single", "suite"] }, "suite": { "type": "string" }, "stack": { "type": "string" }, "offline": { "const": true }, "exit_code": { "enum": [0, 1] }, "engine": { "type": "object", "required": ["name", "version", "upstream"], "additionalProperties": true, "properties": { "name": { "type": "string" }, "version": { "type": "string" }, "upstream": { "type": "string" } } }, "limits": { "type": "object", "required": ["method", "accuracy_claim", "ceiling", "best_input", "does_not_do", "scope", "offline"], "additionalProperties": true, "properties": { "method": { "type": "string" }, "accuracy_claim": { "type": "null", "description": "HONESTY INVARIANT: always null. This tool never emits an accuracy percentage." }, "ceiling": { "type": "string" }, "best_input": { "type": "string" }, "does_not_do": { "type": "array", "items": { "type": "string" } }, "scope": { "type": "string" } } }, "summary": { "type": "object", "required": ["events", "passed", "failed", "regression"], "additionalProperties": true, "properties": { "events": { "type": "integer", "minimum": 0 }, "passed": { "type": "integer", "minimum": 0 }, "failed": { "type": "integer", "minimum": 0 }, "regression": { "type": "boolean" }, "not_scorable": { "type": "integer", "minimum": 1, "description": "Additive: count of events that could not be judged (malformed input, see each event's not_scorable_reason). Present ONLY when at least one event is not scorable; a run of valid recordings emits the exact pre-existing summary shape. Not-scorable events count in events but in neither passed nor failed." } } }, "events": { "type": "array", "items": { "$ref": "#/definitions/event" } }, "fix_map": { "type": "array", "items": { "$ref": "#/definitions/fix_map_entry" } }, "funnel": { "oneOf": [ { "type": "null" }, { "type": "object", "required": ["reason", "pointer"], "additionalProperties": true, "properties": { "reason": { "type": "string" }, "pointer": { "$ref": "#/definitions/engagement_pointer" } } } ] } }, "definitions": { "event": { "type": "object", "required": ["event_id", "expected_yield", "verdict", "measurements"], "additionalProperties": true, "dependencies": { "scorable": ["not_scorable_reason"], "not_scorable_reason": ["scorable"] }, "properties": { "event_id": { "type": "string" }, "scorable": { "const": false, "description": "Additive: present ONLY on events that cannot be judged (and then always false); scorable events omit the key entirely, so every envelope for a valid recording is byte-identical to schema_version 1 before this key existed. A not-scorable event is excluded from passed/failed counting and can never carry a normal pass or fail." }, "not_scorable_reason": { "type": "string", "description": "Plain-language reason the event cannot be judged (for example no detectable caller speech with no onset provided, or the agent silent at the caller onset under a should-yield expectation). Present exactly when scorable is present." }, "scenario_id": { "type": ["string", "null"] }, "title": { "type": ["string", "null"] }, "category": { "type": ["string", "null"] }, "expected_yield": { "type": "boolean" }, "verdict": { "type": "object", "required": ["passed", "did_yield", "seconds_to_yield", "talk_over_sec", "reasons"], "additionalProperties": true, "properties": { "passed": { "type": "boolean" }, "did_yield": { "type": "boolean" }, "seconds_to_yield": { "type": ["number", "null"] }, "talk_over_sec": { "type": "number" }, "reasons": { "type": "array", "items": { "type": "string" } } } }, "measurements": { "type": "object", "additionalProperties": true, "description": "Timing measurements behind the verdict. The core keys (caller_onset_sec, agent_talking_at_onset, hop_sec, notes) are frozen for schema_version 1. The boundary-sensitivity keys below are ADDITIVE (documented here, never in required): they expose where a near-threshold result sat on the frame grid and how far it was from flipping. A reader built before they existed simply ignores them.", "properties": { "caller_onset_sec": { "type": ["number", "null"] }, "agent_talking_at_onset": { "type": "boolean" }, "hop_sec": { "type": "number" }, "notes": { "type": "string" }, "onset_requested_sec": { "type": ["number", "null"], "description": "Additive: the caller_onset_sec the caller requested for this event, or null when the onset was auto-detected rather than supplied." }, "onset_frame_index": { "type": ["integer", "null"], "description": "Additive: the integer hop index the onset was snapped to (null on a not-scorable placeholder)." }, "onset_effective_sec": { "type": ["number", "null"], "description": "Additive: the quantized onset time actually used, equal to onset_frame_index * hop_sec (unrounded, so the equality holds exactly). Null on a not-scorable placeholder." }, "yield_frame_index": { "type": ["integer", "null"], "description": "Additive: the hop index of the counted yield, or null when the agent did not yield (or the event was not scorable)." }, "decision_margin_sec": { "type": ["number", "null"], "description": "Additive: signed slack in seconds from the tightest binding pass/fail threshold to the measured value (positive = inside the pass boundary, magnitude = how much slack; negative = over the line). Null when no numeric bound applies (pure yield/hold) or the event was not scorable." }, "decision_margin_hops": { "type": ["integer", "null"], "description": "Additive: decision_margin_sec expressed in whole frame hops, round(decision_margin_sec / hop_sec). Null when decision_margin_sec is null." }, "boundary_sensitive": { "type": "boolean", "description": "Additive: true when the result sits within one hop of flipping the verdict (abs(decision_margin_hops) <= 1). False when comfortably inside the limit, not derivable, or not scorable." } } }, "signals": { "type": "object", "additionalProperties": true, "description": "Namespaced signal bus. barge_in is present whenever scoring ran; other dimensions (latency, and future overlap/resume/backchannel) are additive.", "properties": { "barge_in": { "type": "object", "required": ["did_yield", "time_to_yield_sec", "talk_over_sec"], "additionalProperties": true }, "latency": { "type": "object", "additionalProperties": true, "properties": { "response_gap_sec": { "type": ["number", "null"] }, "premature_start_sec": { "type": ["number", "null"] } } }, "resume": { "type": "object", "description": "Post-yield behaviour on the agent VAD track; present only when the agent yielded. resumed: did the agent start speaking again within the window after the yield. resume_gap_sec: seconds from the yield to that fresh onset (null when it did not resume). restart_suspected: the longest post-resume run is unusually long, a timing proxy for re-answering from the top (a transcript repeat is out of scope).", "additionalProperties": true, "properties": { "resumed": { "type": "boolean" }, "resume_gap_sec": { "type": ["number", "null"] }, "restart_suspected": { "type": "boolean" } } } } }, "fix": { "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/fix" } ] }, "audio_provenance": { "$ref": "#/definitions/audio_provenance" } } }, "audio_provenance": { "type": "object", "description": "Additive: identity of the exact audio bytes this event was scored from (a streamed sha256 of the raw file, never a full in-memory decode) plus the sample rate and frame count read from the WAV header. Present on every event scored from a real audio file. Absent (never fabricated, never guessed) on an event with no file to hash -- for example a not-scorable / missing-audio placeholder, or an envelope from before this field existed. A reader (in particular hotato fix trial's before/after recapture guard) MUST treat absence as UNKNOWN provenance, never as proof that a fresh capture happened.", "required": ["schema_version", "sha256", "sides"], "additionalProperties": true, "properties": { "schema_version": { "type": "string" }, "sha256": { "type": "string", "description": "The event's combined audio identity: one file's own sha256 for a single-file input (stereo/mono), or an order-stable combination of both sides' sha256 for a caller+agent dual-mono input." }, "sides": { "type": "array", "items": { "type": "object", "required": ["role", "path", "sha256", "sample_rate", "num_samples"], "additionalProperties": true, "properties": { "role": { "type": "string" }, "path": { "type": "string" }, "sha256": { "type": "string", "description": "Streamed sha256 of this side's raw file bytes -- container identity. Changes on any byte anywhere in the file, including header-only metadata edits, a trailing-byte append, or a lossless repack/transcode that never touches a sample value." }, "pcm_sha256": { "type": "string", "description": "OPTIONAL, additive: streamed sha256 of this side's DECODED PCM sample bytes only (the WAV data chunk, read the same way the scorer decodes it -- fixed width, little-endian, channel-interleaved, bounded by the header's own frame count). Identity of the SAMPLES, not the container: a header-only edit or a trailing-byte append leaves this unchanged even though sha256 above differs; an edit to a sample value changes both. Absent on an envelope from before this field existed -- a reader MUST treat absence as UNKNOWN, never as proof either way." }, "sample_rate": { "type": "integer" }, "num_samples": { "type": "integer" }, "duration_sec": { "type": ["number", "null"] } } } } } }, "fix": { "type": "object", "required": ["fix_class", "title", "detail", "knob", "pointer"], "additionalProperties": true, "properties": { "fix_class": { "enum": ["config", "engagement-control"] }, "title": { "type": "string" }, "detail": { "type": "string" }, "knob": { "oneOf": [ { "type": "null" }, { "type": "object", "required": ["stack", "parameter", "direction", "trade_off"], "additionalProperties": true } ] }, "pointer": { "oneOf": [ { "type": "null" }, { "$ref": "#/definitions/engagement_pointer" } ] } } }, "fix_map_entry": { "type": "object", "required": ["event_id", "fix_class", "title", "detail", "knob", "pointer"], "additionalProperties": true, "properties": { "event_id": { "type": "string" }, "scenario_id": { "type": ["string", "null"] }, "fix_class": { "enum": ["config", "engagement-control"] }, "title": { "type": "string" }, "detail": { "type": "string" } } }, "engagement_pointer": { "type": "object", "description": "VENDOR-NEUTRAL pointer for the engagement-control fix class. Names the problem class and the KIND of fix (a learned engagement-control / addressee-detection layer); names no vendor, no product, and nothing you can adopt/license/buy, and carries no numbers.", "required": ["layer", "what", "honest_scope"], "additionalProperties": true, "properties": { "layer": { "type": "string" }, "what": { "type": "string" }, "honest_scope": { "type": "string" } } } } }