--- id: "006" title: Blocking Inbox Watch status: complete amends: specs/001-agent-mail-cli.md blocked_by: [] blocks: [] --- # Blocking Inbox Watch ## Overview The mailbox is pull-based: nothing arrives unless an agent looks. Today an agent that wants to stay reachable must register a timed loop and re-check on a cadence — the pattern JJ's delegation guide sets at 10–15 minutes. That costs one model invocation per tick to usually learn nothing, sets inbox latency to the tick interval, and fails silently when the loop is never registered or is dropped on a cold restart. The recipient goes deaf, the sender cannot tell, and the only symptom is unexplained quiet. This spec adds `watch` — a blocking, non-mutating detector that returns as soon as the agent has mail and otherwise sleeps to a timeout. Waiting moves out of the model and into the process, so an idle agent costs nothing while blocked and is woken the moment a message lands. `watch` also publishes a liveness heartbeat while it waits, which `status` reports, so a sender can see whether a recipient is actually reachable instead of inferring it from silence. The shape is deliberately the one already proven twice in JJ's stack — `mc watch` (Meeting Copilot) and `vm watch` (Voice Tunnel): block, return everything new at once, and report an empty result on timeout rather than an error. Ships as `0.1.7`. > **Amended by [spec 008](008-remove-the-wrong-choices.md) (0.2.0).** Agents overrode the correct 900-second default with 30, because 30 is a natural number of *seconds*. The wait is now `--timeout-minutes` (default 90, floor 5), `--timeout 0` became `--once`, and `--poll` is gone. The TTL was removed in the same release, so the expired-message filter described below no longer exists — there is nothing to expire. Read the shape here as history; spec 001 TC15–TC17 carry the current contract. > > **Also amended by [spec 009](009-rearm-state.md) (0.3.0).** The payload gained `reachable` and `next`, `status` gained `watch_state`, and liveness moved to one row per watch process rather than one per identity. > **Completion rule:** This spec is not complete until all acceptance criteria are verified through the testing approach below, including a live wake — a `watch` blocked on the real second-brain mailbox, woken by a send from a separate process, on the published `0.1.7`. Build-only and CI-only verification are insufficient. The agent must iterate until verification passes. ## Goals - An idle agent can stay reachable without spending a model invocation per interval. - A message reaches a waiting recipient in about one poll interval instead of one tick interval. - A sender can determine whether a recipient is currently watching, rather than inferring it from the absence of replies. - Detection never consumes what it detects: a message seen by `watch` is still there for `read`. ## Requirements ### Functional Requirements - **FR1**: `watch ` blocks until `` has at least one deliverable unread message, then returns all of them at once with a result indicating it did not time out. - **FR2**: When no deliverable message exists within the timeout, `watch` returns an empty result marked as a timeout, on stdout, with exit 0. A timeout is a heartbeat, not an error. - **FR3**: The messages `watch` reports are exactly those `read ` would return in default mode, excluding messages already past their TTL, projected to a fixed envelope field set that excludes the body. - **FR4**: `watch` never mutates mailbox content. It does not mark messages read, does not ack, and does not run the opportunistic expiry purge. - **FR5**: `watch` publishes a liveness record for the watched identity on invocation and refreshes it on a fixed cadence for as long as it stays blocked. - **FR6**: `status` reports, per agent, whether a live watcher is currently present and when that watcher was last seen. - **FR7**: `cleanup` purges liveness records under the same TTL rule that governs messages. - **FR8**: `describe` documents `watch` — its arguments, output fields, and examples — and carries the watch-then-read discipline as an invariant. ### Non-Functional Requirements - **NFR1**: A message committed by a different process while `watch` is blocked is observed within one poll interval. `watch` does not serve a snapshot taken when it started. - **NFR2**: `watch` holds no database transaction while waiting. Only the bounded liveness write opens one, and it commits immediately. - **NFR3**: The liveness refresh cadence is fixed and independent of `--poll`, so a fast poll cannot turn into a write storm. - **NFR4**: The existing output contract holds: one JSON document on stdout, JSON errors on stderr with a non-zero exit, UTF-8 regardless of console code page. - **NFR5**: No new runtime dependencies, and no changes to the CI, release, npm publish, or WinGet workflows. ### Technical Constraints - **TC1**: `--timeout` is a float in `[0, 3600]`, default `900`. `0` performs a single check and returns without blocking. - **TC2**: `--poll` is a float in `[0.05, 60]`, default `1.0`. - **TC3**: The envelope field set is fixed: `id`, `sender`, `subject`, `type`, `created`. `watch` has no `--fields` flag and never returns `body` or `refs`. - **TC4**: Deliverability is the `read` default-mode predicate — unread direct messages addressed to the agent, plus broadcasts with no `read_at` in `broadcast_acks` for that agent, excluding messages the agent itself sent — conjoined with not yet expired under the TC12 rule in spec 001. - **TC5**: Liveness is stored in a `watchers(agent PRIMARY KEY, started_at, last_seen)` table created by the existing first-use initialization. The addition is additive and requires no migration of existing mailboxes. - **TC6**: The liveness refresh interval is 5 seconds. `status` reports `watching` as true when `last_seen` is within 60 seconds. A watcher is never deleted on exit; liveness decays. - **TC7**: `status`'s `output_fields` gains `watching` and `last_seen`, both projectable via `--fields`. - **TC8**: `watch` creates the database and parent directory on first use, like every command except `describe` (spec 001 NFR4). - **TC9**: The fix ships as version `0.1.7` through the existing PyPI, GitHub Release, and npm channels defined by specs 002 and 003. ### Requirement Traceability | Requirement | Acceptance Criteria | |---|---| | FR1 | AC1, AC3, AC6 | | FR2 | AC2, AC4 | | FR3 | AC6, AC7, AC8 | | FR4 | AC9, AC10 | | FR5 | AC11 | | FR6 | AC11, AC12, AC13 | | FR7 | AC14 | | FR8 | AC15 | | NFR1 | AC3 | | NFR2 | AC3, AC17 | | NFR3 | AC11 | | NFR4 | AC2, AC5, AC16 | | NFR5 | AC17 | | TC1 | AC2, AC4, AC5 | | TC2 | AC3, AC5 | | TC3 | AC7 | | TC4 | AC6, AC8 | | TC5 | AC11, AC13 | | TC6 | AC11, AC12 | | TC7 | AC11 | | TC8 | AC13 | | TC9 | AC18, AC19, AC20 | ## Reported Shape Woken by mail: ```json { "agent": "agent-mail:worker", "messages": [ { "id": "9f1c2e34-5678-4abc-9def-0123456789ab", "sender": "second-brain:main", "subject": "Spec 006 handoff", "type": "direct", "created": "2026-07-31T15:04:11.882431-04:00" } ], "count": 1, "timeout": false } ``` Timed out with nothing waiting: ```json { "agent": "agent-mail:worker", "messages": [], "count": 0, "timeout": true } ``` | Field | Meaning | |---|---| | `agent` | The identity watched, echoed back so a backgrounded result is self-identifying | | `messages` | Envelopes of every deliverable unread message, oldest first | | `count` | Length of `messages` | | `timeout` | `true` when the wait expired with nothing to deliver; `false` when mail is being returned | `status` gains two fields per agent: ```json [ { "agent": "agent-mail:worker", "unread": 1, "unacked": 1, "watching": true, "last_seen": "2026-07-31T15:04:09.113470-04:00" } ] ``` ## Key Decisions ### Detection must not consume `read` marks what it returns. `watch` deliberately does not, and the reason is the defect spec 004 was written to fix: a command that commits a read it never delivered destroys the message and leaves no trace. `watch` is designed to be run in the background, where its output may land in a file the agent never actually reads — a dead harness, a closed window, a compacted context. Marking read there would reintroduce the same silent loss one layer up, with a wider window. So detection and consumption are separate commands: `watch` finds out, `read` takes delivery. ### No body, so the loop cannot stall on a full inbox Withholding the body is what keeps `watch → read → watch` the only way to make progress. If `watch` returned everything an agent needed, the agent could act and re-arm without ever calling `read`; the message would still be unread, the next `watch` would return instantly, and the cycle would spin at full speed — a runaway that costs one model invocation per iteration. Returning envelopes only makes `read` structurally necessary, and it also keeps the payload small, which is the point of `--fields` elsewhere. Since the field set is already five short fields, `watch` needs no projection flag of its own. ### The unread flag is the cursor `mc watch` and `vm watch` take a caller-held integer cursor because their append-only logs have no per-reader state. This mailbox does: `messages.read_at` and `broadcast_acks`. So the cursor already exists, per recipient, and `watch` needs no `--since`. That matters beyond convenience — message ids are UUID4, so there is no monotonic value to hand back, and a sequence surrogate would be wrong under this mailbox's TTL. Rows are deleted continuously, and SQLite reuses `rowid` values once the table drains, so a stale cursor would silently skip real messages after a quiet day. Using the read state avoids inventing an ordering the schema does not have. It also means the drain rule those two CLIs need disappears: `read` already returns every unread message up to `--limit`, so there is nothing to drain. ### Expired-but-unpurged messages must not wake a watcher `watch` does not run the opportunistic purge (it polls, and purging per poll is a write storm), so it can see rows that are past their TTL but not yet deleted. Waking on one would be a false wake: the agent is invoked, calls `read`, `read` purges the row and returns nothing, and the agent learns only that it was woken for no reason. Filtering expired rows out of the predicate — without deleting them — costs one clause and removes the whole failure. ### Liveness decays, and is never revoked on exit The heartbeat is what turns "is the recipient reachable?" from an inference into a fact. It is not a registration: nothing is declared, nothing must be torn down, and an identity is still discovered from message traffic exactly as before. Liveness is derived from watch traffic the same way. The record is written on invocation rather than only once blocking begins, so `last_seen` means "when this identity last looked at its inbox" — which stays true during a busy stretch where every `watch` returns immediately and never blocks at all. A watcher never deletes its record on exit, and the 60-second window is deliberately much wider than the 5-second refresh. The gap between `watch` returning mail and the agent re-arming is exactly the interval when the agent is doing the work it was woken for — reporting `watching: false` there would be technically true and practically wrong, and would make a healthy agent flicker. Sixty seconds is twelve missed refreshes: long enough to cover the processing gap, short enough that a killed session, a slept laptop, or a dropped loop reads as unreachable within a minute. A hard kill therefore looks the same as a clean exit, which is the correct outcome — the record answers "was anyone here recently," not "did someone shut down politely." ### `watch` writes only its own liveness, never mailbox content FR4's "never mutates" is about mailbox content: messages, read state, acks. The liveness refresh is a write, and drawing the line here rather than at "read-only" is deliberate — the guarantee agents depend on is that watching cannot destroy or consume mail, not that the process never touches the file. The write is bounded (one row every five seconds), commits immediately, and holds no transaction across a wait, so it does not hold a lock while blocked. On the WSL/Windows shared mailbox, where a concurrent cross-boundary write can be silently lost, the blast radius of a lost heartbeat is one stale timestamp that the next refresh corrects. ### The timeout is the second job the loop was doing The timed loop this replaces did two things: check the inbox, and periodically resume outstanding work. A purely mail-driven wake only does the first, so an agent parked mid-task with no incoming mail would never be woken — the exact stall the loop existed to prevent. The timeout return covers it: an empty, non-error result is a scheduled nudge to resume. That is why `900` is the default rather than the 30 seconds both sibling CLIs use. Fifteen minutes matches the cadence the delegation guide already sets, so adopting `watch` costs no more idle invocations than the loop it replaces while making mail arrive in about a second. Every timeout is one model invocation, so the default is the knob that sets the idle cost of the whole arrangement. ### Waiting is cheap, but waking is the caller's problem `watch` removes the cost of waiting; it does not by itself deliver a message into a model's context. Something must notice the process finished. Where the harness re-invokes an agent when a background process exits, `watch` is run in the background and the exit is the wake. Where it does not, `watch` is run in the foreground and the agent blocks on it, which is correct when the agent is idle but means it cannot work while waiting. The CLI is identical in both; only the caller's discipline differs, which is why the bounded `--timeout` is required rather than an indefinite block — a caller whose harness caps command duration needs to choose a timeout under that cap. ### Spec 001 is amended, not superseded As with specs 004 and 005, spec 001 remains the standing behavioral contract. This spec owns the change and its verification; its implementation tasks fold the resulting rules into spec 001. ## Implementation Tasks - [x] Add the `watch` command: block until deliverable unread mail exists for the agent, then emit the documented result; emit the timeout result when the wait expires. - [x] Restrict `watch` to envelope fields and exclude expired messages from its predicate without deleting them. - [x] Add the liveness table to first-use initialization and refresh the watched identity's record on the fixed cadence while blocked. - [x] Report `watching` and `last_seen` from `status`, projectable via `--fields`. - [x] Purge liveness records in `cleanup` under the message TTL rule. - [x] Validate `--timeout` and `--poll` ranges with the standard JSON error contract. - [x] Record `watch` in the schema, including the watch-then-read discipline in the invariants. - [x] Add regression coverage for every acceptance criterion, including a cross-process wake and a concurrent liveness check. - [x] Amend spec 001: add `watch` to FR-level behavior, the command contracts, the schema table, and the acceptance criteria. - [x] Bump the package version to `0.1.7`; the npm package version is set by the publish workflow. - [x] Add the `0.1.7` CHANGELOG entry. - [x] Release `v0.1.7` after JJ's approval and verify the published artifacts. ## Verification Record - [x] `ruff check src/ tests/` passed. - [x] `pytest` passed: 53 passed, 0 skipped, with the PyInstaller binary and npm wrapper both available. - [x] The full behavior, output-integrity, storage, and watch suites pass with `AGENT_MAIL_RUNNER=binary` (46 passed) and `AGENT_MAIL_RUNNER=npm` (46 passed). - [x] All 19 watch tests and the watch parity test fail against the pre-fix `cli.py` (verified by restoring only that file from git), so they are genuine regression coverage rather than tests written around the implementation. - [x] `python -m build` produced the `0.1.7` wheel and sdist; `twine check` passed on both. - [x] The readiness gate in the cross-process tests initially raced first-use initialization — the mailbox file exists before the `watchers` table does — and was fixed to treat a missing table as not-ready. Caught by the binary runner, whose startup widens the window; the module runner had hidden it. - [x] CI passed on `main` for Python 3.10-3.13 plus lint and build (run `30662022683`). The timing-sensitive cross-process tests did not flake on Linux runners. - [x] Release workflow `30662100972` succeeded; GitHub Release `v0.1.7` carries the wheel, the sdist, and the four platform binaries. - [x] PyPI serves `agent-mail-cli` `0.1.7`; npm publish workflow `30662224711` succeeded and `npm view @juanjofuchs/agent-mail@0.1.7 version` returns `0.1.7`. - [x] `npm-smoke.yml` run `30662282133` passed on Windows x64, Linux x64, macOS x64, and macOS arm64. - [x] AC20 verified against the globally installed `0.1.7` on JJ's machine, on the live second-brain mailbox (`AGENT_MAIL_DB` override, 20 messages). A watcher on `agent-mail:release-check` blocked; `status` reported `watching` true with `last_seen` advancing across the wait; a `send` from a separate process at 16:18:59 woke it with that message's envelope and `timeout` false; the body was withheld; `status` still showed `unread` 1 afterwards; and a following `read` delivered the message with its body, taking `unread` to 0. - [x] Cross-boundary check beyond the AC: `agent-mail watch` invoked **from WSL** against the same shared mailbox returned the same message and refreshed the same liveness row. **Correction (2026-08-01):** this was originally recorded as the *WSL-native* binary, which was wrong. `npm root -g` inside WSL resolves to the Windows prefix and `which agent-mail` is `/mnt/c/.../npm/agent-mail`, so the call ran the Windows binary through interop. What is verified is that a WSL-side caller reaches the shared mailbox correctly; a genuinely Linux-native binary running against `/mnt/d` is **not** verified, and the WSL/Windows lock caveat in the vault's project note still applies to that untested path. ## Acceptance Criteria Each criterion names its validation method. `integration` criteria are automated tests owning their own fixtures; `manual` criteria require live published artifacts and are noted with why they cannot be automated here. ### Blocking and waking - [x] **AC1** (`integration`): With a deliverable unread message already present, `watch ` returns immediately with that message, `count` 1, and `timeout` false. - [x] **AC2** (`integration`): With nothing deliverable, `watch --timeout 1` exits 0 with `messages` empty, `count` 0, and `timeout` true. - [x] **AC3** (`integration`): A `watch` blocked on an empty mailbox is woken by a message sent from a **separate process** and returns it, having observed a commit made after it started waiting. - [x] **AC4** (`integration`): `watch --timeout 0` returns without blocking — the timeout result when nothing is waiting, the message when one is. - [x] **AC5** (`integration`): `--timeout` outside `[0, 3600]` and `--poll` outside `[0.05, 60]` each produce a JSON error on stderr with a non-zero exit. ### What gets reported - [x] **AC6** (`integration`): For a mailbox holding a direct message, an unread broadcast, a self-sent message, and an already-read message, `watch` returns exactly the ids that `read --no-mark-read` returns. - [x] **AC7** (`integration`): Each returned message carries exactly `id`, `sender`, `subject`, `type`, and `created` — no `body`, no `refs`. - [x] **AC8** (`integration`): An unread message already past its TTL does not wake `watch`, which times out instead, and the row still exists afterwards. ### Detection does not consume - [x] **AC9** (`integration`): After `watch` returns a message, that message is still unread — a following `read ` returns it, and `status` unread counts are unchanged across the watch. - [x] **AC10** (`integration`): `watch` does not purge expired messages; a row past its TTL survives a `watch` and is removed only by a later `send`, `read`, `status`, or `cleanup`. ### Liveness - [x] **AC11** (`integration`): While a `watch` is blocked in a separate process, `status --agent ` reports `watching` true with a `last_seen` timestamp that advances across the wait, and `status --fields agent,watching` projects to those keys only. - [x] **AC12** (`integration`): Once the watcher has exited and the liveness window has lapsed, `watching` reads false while `last_seen` retains the final refresh. - [x] **AC13** (`integration`): An agent that has never watched reports `watching` false and `last_seen` null, and `watch` creates the mailbox and its parent directory on first use. - [x] **AC14** (`integration`): `cleanup` deletes liveness records older than the message TTL and leaves fresh ones intact. ### Schema and parity - [x] **AC15** (`integration`): `describe` includes `watch` with its arguments, `output_fields`, and examples; `describe watch` returns that block alone; and the invariants state that `watch` does not mark messages read and must be followed by `read`. - [x] **AC16** (`integration`): The importable module, the PyInstaller binary, and the npm wrapper produce identical `describe` output and identical `watch` results against a shared mailbox. - [x] **AC17** (`integration`): `ruff check src/ tests/` and the full `pytest` suite pass on the existing CI matrix (Python 3.10–3.13 on Linux) with no workflow changes and no new runtime dependencies. ### Released - [x] **AC18** (`manual`): GitHub Release `v0.1.7` carries the six artifacts required by spec 002 AC3, PyPI serves `agent-mail-cli==0.1.7`, and `npm view @juanjofuchs/agent-mail@0.1.7 version` returns `0.1.7`. Verified by the agent against live registries, which CI cannot assert about its own unpublished build. - [x] **AC19** (`integration`): The `npm-smoke.yml` workflow passes for `0.1.7` on Windows x64, Linux x64, macOS x64, and macOS arm64. - [x] **AC20** (`manual`): On JJ's machine, against the live second-brain mailbox, the globally installed `0.1.7` blocks in `watch`, is woken by a real send from another process, reports `watching` true from `status` while blocked, and leaves the message unread for a subsequent `read`. Requires the real installed binary and shared mailbox. ## Testing Approach ### Validation steps 1. `ruff check src/ tests/` and `pytest` pass with the module runner. 2. Blocking behavior is exercised with real subprocesses and short timeouts, never by monkeypatching the clock — the contract being tested is cross-process visibility, which an in-process fake cannot show. 3. Wake latency is asserted as a bound (woken well before the timeout), not as an exact duration, so the suite does not become timing-flaky on CI. 4. The suite re-runs with `AGENT_MAIL_RUNNER=binary` and `AGENT_MAIL_RUNNER=npm` for parity. ### Test cases | Input | Expected output | |---|---| | `watch ` with one unread message | That message, `count` 1, `timeout` false | | `watch --timeout 1` on an empty mailbox | `[]`, `count` 0, `timeout` true, exit 0 | | `watch ` blocked, then a send from another process | Woken with the new message before the timeout | | `watch --timeout 0` on an empty mailbox | Immediate timeout result, no blocking | | `watch --timeout 5000` | JSON error on stderr, non-zero exit | | `watch --poll 0` | JSON error on stderr, non-zero exit | | `watch` on a mailbox with an expired unread message | Times out; the row is still present | | `read ` after `watch ` | Returns the same message — watch consumed nothing | | `status --agent ` during a blocked watch | `watching` true, `last_seen` advancing | | `status --agent ` a minute after the watcher exits | `watching` false, `last_seen` retained | | `cleanup` with a stale liveness record | Record deleted; fresh records untouched | ### Local validation before release ```bash ruff check src/ tests/ pytest python -m build twine check dist/* ``` ### Release validation ```bash gh run watch gh release view v0.1.7 npm view @juanjofuchs/agent-mail@0.1.7 version ``` A same-day `npx` verification needs `npx --min-release-age=0`, per the note in spec 004. ### Human-in-the-Loop Release Protocol 1. **Agent**: Implements, passes local validation, and reports results. 2. **Agent**: Asks JJ before pushing the release tag. 3. **Human**: Approves the release tag. 4. **Agent**: Pushes `v0.1.7`, monitors GitHub Actions, and reports PyPI, GitHub Release, and npm status. 5. **Agent**: Installs the published version and runs the AC20 live-wake check against the real second-brain mailbox. 6. **Agent**: Reports results, or iterates if verification fails. ## Usage Examples The loop, where the harness re-invokes an agent when a background process exits: ```bash # Arm the watch in the background and end the turn. Costs nothing while waiting. agent-mail watch agent-mail:worker --timeout 900 & # Woken with envelopes. Take delivery — watch marked nothing read. agent-mail read agent-mail:worker # Act, ack, then re-arm. Never end a turn without a watch armed. agent-mail ack agent-mail:worker 9f1c2e34-5678-4abc-9def-0123456789ab agent-mail watch agent-mail:worker --timeout 900 & ``` Foreground, where the harness has no such wake — the agent blocks, which is correct when it is idle: ```bash agent-mail watch agent-mail:worker --timeout 300 ``` Checking whether a recipient is reachable before relying on a handoff landing: ```bash agent-mail status --agent agent-mail:worker --fields agent,watching,last_seen ``` One non-blocking check, for an agent that only wants to know: ```bash agent-mail watch agent-mail:worker --timeout 0 ``` ## Out of Scope - Push delivery, notifications, sockets, daemons, or an MCP surface. `watch` is a blocking poll in the calling process. - Waking the model. The CLI returns when there is mail; whether a harness re-invokes an agent on process exit is the caller's concern. - Filtering the wait by sender, subject, or thread. `read --from` covers filtering after the wake. - `--fields` on `watch`, or returning `body` or `refs` — the envelope set is fixed. - A `--since` cursor or any monotonic sequence column. The unread flag is the cursor. - Marking read, acking, or purging from `watch`. - Watching more than one identity per invocation. - Deleting a liveness record on exit, or any explicit register/unregister command. - Configurable liveness refresh or staleness windows. - Cross-machine watching, locking changes, or any change to the WSL/Windows shared-mailbox behavior. - Workflow changes for CI, release, npm publish, or WinGet. ## References - spec 001: [`specs/001-agent-mail-cli.md`](001-agent-mail-cli.md) — the behavioral contract this spec amends. - spec 002: [`specs/002-packaging.md`](002-packaging.md) — PyPI and GitHub Release machinery that ships `0.1.7`. - spec 003: [`specs/003-npm-distribution.md`](003-npm-distribution.md) — npm wrapper and smoke workflow used by AC19 and AC20. - spec 004: [`specs/004-output-integrity.md`](004-output-integrity.md) — the commit-after-output rule that motivates detection-without-consumption, and the `npx --min-release-age=0` note. - spec 005: [`specs/005-storage-transparency.md`](005-storage-transparency.md) — the resolved-mailbox reporting a watcher inherits. - Implementation: [`src/agent_mail/cli.py`](../src/agent_mail/cli.py). - Prior art in the same author's stack: `mc watch` (Meeting Copilot) and `vm watch` (Voice Tunnel) — same block-return-heartbeat contract. - Project context: [PROJECT_UNDERSTANDING.md](../PROJECT_UNDERSTANDING.md).