---
id: "004"
title: Agent Mail Output Integrity
status: complete
amends: specs/001-agent-mail-cli.md
blocked_by: []
blocks: []
---

# Agent Mail Output Integrity

## Overview

Issue [#1](https://github.com/JuanjoFuchs/agent-mail-cli/issues/1) reports two defects that compound into a silent message loss on Windows. First, the CLI writes its JSON payload using the platform's console code page, so any body containing a character outside that code page (`→`, `—`, smart quotes, emoji, CJK) crashes with `UnicodeEncodeError`, exits non-zero, and prints nothing. Second, `read` commits its mark-read update *before* writing that payload, so the crashed read still consumes the message: it disappears from the default unread inbox and from the `status` unread count while never having been displayed. An agent that retries sees `[]`, concludes the inbox is empty, and the 24-hour TTL then erases the evidence.

This spec fixes both. Output encoding becomes UTF-8 unconditionally, on every stream and in every distribution channel. Durability of every mutating command becomes conditional on that command's payload reaching stdout. The behavioral contract in spec 001 is amended to carry both rules; specs 002 and 003 supply the unchanged release machinery that ships the fix as `0.1.5`.

> **Completion rule:** This spec is not complete until all acceptance criteria are verified through the testing approach below, including the original issue #1 repro re-run against a released `0.1.5` on a Windows console with active code page 437. Build-only and CI-only verification are insufficient. The agent must iterate until verification passes.

## Goals

- Any message content SQLite can store round-trips through stdout intact, on any console code page, through any distribution channel.
- No mutating command consumes, alters, or destroys state it failed to report.
- Issue #1 is closable against a published version JJ can reproduce on the machine that reported it.

## Requirements

### Functional Requirements

- **FR1**: All CLI output — success payloads on stdout and error payloads on stderr — is encoded as UTF-8 regardless of the active console code page, the process locale, or the `PYTHONIOENCODING` and `PYTHONUTF8` environment variables.
- **FR2**: FR1 holds identically for every distribution channel: the importable package, the console script, the PyInstaller binary, and the npm wrapper's spawned binary.
- **FR3**: A mutating command's database changes become durable only after that command's result payload has been written and flushed to stdout. If the write or flush fails, no change is committed.
- **FR4**: A command whose payload cannot be written reports the failure as a JSON error on stderr with a non-zero exit code, per spec 001 NFR2.

### Non-Functional Requirements

- **NFR1**: Spec 001 NFR1 is preserved unchanged — success output remains pretty-printed JSON with two-space indentation and unescaped Unicode. The encoding fix does not switch output to ASCII escaping.
- **NFR2**: Both defects have regression coverage that runs deterministically on the existing Linux CI matrix, without a Windows console and without a real broken terminal.
- **NFR3**: No new runtime dependencies. No changes to the CI, release, npm publish, or WinGet workflows.

### Technical Constraints

- **TC1**: UTF-8 is forced unconditionally. The CLI does not consult or honor `PYTHONIOENCODING`, `PYTHONUTF8`, or the active code page when selecting output encoding. The reporter verified those environment routes are not honored through the npm → PyInstaller launch, so the encoding decision must live inside the process.
- **TC2**: Encoding error policy is strict. The CLI must not substitute, escape, or drop characters it cannot encode; an unencodable character surfaces as a failure under FR3 and FR4 rather than as corrupted output.
- **TC3**: Only the character encoding changes. Existing line-ending behavior, indentation, key order, and the absence of a byte-order mark are unchanged.
- **TC4**: FR3 applies to every state change a command reports:

  | Command | Deferred state change |
  |---|---|
  | `send` | The inserted message row |
  | `read` | Mark-read on `messages.read_at` and `broadcast_acks.read_at` |
  | `ack` | `messages.acked_at` and `broadcast_acks.acked_at` |
  | `cleanup` | Deletion of expired messages and their broadcast ack rows |

- **TC5**: The opportunistic expiry purge that runs at the start of `send`, `read`, and `status` is exempt from FR3. It removes only messages already past the TTL defined in spec 001 TC12 — those are unrecoverable by contract and are not part of the command's payload.
- **TC6**: `describe` requires no database and has no deferred state; FR1 still applies to it.
- **TC7**: The fix ships as version `0.1.5` through the existing PyPI, GitHub Release, and npm channels defined by specs 002 and 003.

### Requirement Traceability

| Requirement | Acceptance Criteria |
|---|---|
| FR1 | AC1, AC2, AC3 |
| FR2 | AC10, AC13, AC14 |
| FR3 | AC4, AC5, AC6, AC7, AC9 |
| FR4 | AC8 |
| NFR1 | AC2 |
| NFR2 | AC11 |
| NFR3 | AC11 |
| TC1 | AC1, AC3 |
| TC2 | AC2 |
| TC3 | AC2 |
| TC4 | AC4, AC5, AC6, AC7 |
| TC5 | AC9 |
| TC6 | AC1 |
| TC7 | AC12, AC13, AC14 |

## Key Decisions

### Force UTF-8 rather than escaping to ASCII

The issue offers two viable fixes: reconfigure the streams to UTF-8, or serialize with ASCII escaping. Escaping would stop the crash but contradicts spec 001 NFR1 and degrades every non-Latin body into `\uXXXX` noise for the humans and agents reading it. Forcing UTF-8 preserves the existing contract and fixes stderr at the same time. This is option 1 in the issue, which the reporter also identified as the most faithful.

### Mojibake on a legacy console is accepted

On a cp437 console, correct UTF-8 bytes render as mojibake unless the user runs `chcp 65001`. That is accepted. The consumer of this output is an agent or a pipe, and the JSON contract is defined in bytes, not in what a legacy terminal chooses to draw. Correct data beats lossy display.

### The CLI does not change the console code page

Calling `chcp` or the Windows console API to switch the terminal to UTF-8 would mutate state the CLI does not own and would outlive the process. Out of scope; the user can switch code pages themselves if they want readable rendering.

### Commit-after-output as a general invariant

The reported defect is in `read`, but the shape is general: any command that commits before printing can consume state it never reported. `send` has the same hazard in a different direction — a crashed send leaves a stored message whose id the sender never learned, so the natural retry produces a duplicate. One invariant applied to all four mutating commands is the same size of change as a targeted `read` fix, is easier to state, and closes the class rather than the instance.

### At-least-once delivery is the safe failure direction

Deferring the commit means a process killed between a successful write and the commit re-delivers those messages on the next read. Re-delivery is visible and recoverable; silent consumption is neither. Where the two cannot both be guaranteed, this spec chooses re-delivery.

### The opportunistic purge stays eager

Deferring the TTL purge alongside the payload would add rollback complexity for rows that are already expired by contract and that the next invocation would delete anyway. The purge is not part of any command's reported result, so it is exempt.

### No `status` anomaly detector

The issue comment suggests `status` surface `unread=0, unacked>0` as an anomaly, since that combination was the only surviving symptom. Declined: it detects a condition this spec removes at the source, it would extend spec 001's `status` output contract, and the same combination is a legitimate normal state — a message read but deliberately not acked. Recorded here so the decision is not relitigated.

### Error payload serialization is unchanged

Error payloads stay ASCII-escaped as they are today. That already makes them encodable on any code page, so no change is needed to satisfy FR1; the stderr stream is still forced to UTF-8 so the guarantee does not depend on that serialization choice.

### Spec 001 is amended, not superseded

Spec 001 remains the single standing behavioral contract. This spec owns the change and its verification; its implementation tasks add the resulting rules to spec 001 so a future reader finds them in the contract rather than in a historical fix spec.

## Implementation Tasks

- [x] Force UTF-8 on stdout and stderr before any output is produced, in the path shared by the console script, `python -m agent_mail`, and the PyInstaller binary.
- [x] Defer the commit of every mutating command until after its payload is written and flushed to stdout, leaving the opportunistic purge eager.
- [x] Add regression coverage for the encoding defect, using a non-UTF-8 preferred encoding in the child environment so it runs on Linux CI.
- [x] Add regression coverage for the durability defect for all four mutating commands, using a stdout handle that rejects writes.
- [x] Ensure the test harness decodes child output as UTF-8 rather than the parent's locale encoding.
- [x] Extend `tests/test_parity.py` with a non-ASCII round-trip across module, PyInstaller binary, and npm wrapper.
- [x] Amend spec 001 with the UTF-8 output rule, the commit-after-output rule and its purge exemption, and their acceptance criteria.
- [x] Bump the package version to `0.1.5`; the npm package version is set by the publish workflow.
- [x] Add the `0.1.5` CHANGELOG entry describing both fixes and referencing issue #1.
- [x] Release `v0.1.5` after JJ's approval and verify the published artifacts.
- [x] Close issue #1 referencing the released version.

## Verification Record

- [x] `ruff check src/ tests/` passed.
- [x] `pytest` passed: 25 passed, 0 skipped, with the PyInstaller binary and npm wrapper both available.
- [x] The eight durability and encoding tests fail against the pre-fix `cli.py` (verified by stashing only that file), so they are genuine regression coverage. AC3 passes either way by design — error payloads were already ASCII-escaped.
- [x] Full behavior and integrity suites pass with `AGENT_MAIL_RUNNER=binary` (19 passed) and `AGENT_MAIL_RUNNER=npm` (19 passed).
- [x] Three-way byte-identical parity confirmed for the non-ASCII payload under `PYTHONIOENCODING=cp437`.
- [x] Issue #1 repro re-run locally on `chcp` 437 through `node npm/bin/agent-mail.js` against a locally built binary: `send` then `read` both exit 0, the arrow round-trips, and `status` afterwards reports `unread=0, unacked=1`.
- [x] CI passed on `main` for Python 3.10-3.13 (run `30377583693`).
- [x] Release workflow `30377588756` succeeded; GitHub Release `v0.1.5` carries the wheel, the sdist, and the four platform binaries.
- [x] PyPI `agent-mail-cli==0.1.5` is live via Trusted Publishing.
- [x] npm publish workflow `30377765307` succeeded; `npm view @juanjofuchs/agent-mail@0.1.5 version` returns `0.1.5`.
- [x] `npm-smoke.yml` run `30377818406` passed on Windows x64, Linux x64, macOS x64, and macOS arm64.
- [x] Issue #1 repro re-run against the published `npx -y @juanjofuchs/agent-mail@0.1.5` on `chcp` 437: `send`, `read`, and `status` all exit 0 and the arrow round-trips.
- [x] Issue #1 closed referencing `0.1.5`.

Note for future releases from JJ's machine: a global npmrc sets `min-release-age`, so `npx` refuses to resolve a package published within the last 7 days and fails with `ETARGET ... no matching version found ... with a date before <7 days ago>`. Verify a same-day publish with `npx --min-release-age=0 -y @juanjofuchs/agent-mail@<version> describe`. GitHub-hosted runners are unaffected.

## Acceptance Criteria

Each criterion names its validation method. `integration` criteria are automated tests owning their own fixtures; `manual` criteria require live published artifacts or a real Windows console and are noted with why they cannot be automated here.

### Output encoding

- [x] **AC1** (`integration`): With the CLI process running under a non-UTF-8 preferred encoding (e.g. `PYTHONIOENCODING=cp437`), sending and reading a body containing `→ — “smart” … 😀 日本語 café` exits 0 and returns every character intact. `describe` under the same environment also exits 0.
- [x] **AC2** (`integration`): For that read, stdout bytes decode as UTF-8 and — with line endings normalized — are byte-identical to the UTF-8 encoding of the expected JSON text: no `\uXXXX` escapes, no `U+FFFD` replacement characters, no byte-order mark, two-space indentation preserved.
- [x] **AC3** (`integration`): Under the same environment, an error path that echoes non-ASCII user input (for example an invalid agent identity containing `é`) exits non-zero and writes a UTF-8-decodable JSON object on stderr whose `error` value contains that character.

### Durability of mutating commands

- [x] **AC4** (`integration`): A `read` whose stdout is bound to a handle that rejects writes leaves the targeted message unread — `messages.read_at` is NULL, no `broadcast_acks` row gains a `read_at`, the `status` unread count is unchanged, and a subsequent normal `read` returns the message.
- [x] **AC5** (`integration`): A `send` whose stdout rejects writes inserts no row into `messages`.
- [x] **AC6** (`integration`): An `ack` whose stdout rejects writes leaves `messages.acked_at` and any `broadcast_acks.acked_at` unchanged.
- [x] **AC7** (`integration`): A `cleanup` whose stdout rejects writes deletes no rows from `messages` or `broadcast_acks`, verified by direct database inspection without running another command.
- [x] **AC8** (`integration`): Each of AC4–AC7 exits non-zero and writes a JSON object with an `error` key on stderr.
- [x] **AC9** (`integration`): A `read` whose stdout rejects writes still purges messages already past their TTL, while leaving the unexpired message it failed to report unread.

### Parity across distribution channels

- [x] **AC10** (`integration`): For the AC1 payload under a non-UTF-8 preferred encoding, the importable module, the PyInstaller binary, and the npm wrapper produce identical stdout bytes and identical exit codes.

### Regression suite

- [x] **AC11** (`integration`): `ruff check src/ tests/` and the full `pytest` suite pass on the existing CI matrix (Python 3.10–3.13 on Linux) with no workflow changes and no new runtime dependencies.

### Released fix

- [x] **AC12** (`manual`): GitHub Release `v0.1.5` carries the six artifacts required by spec 002 AC3, PyPI serves `agent-mail-cli==0.1.5`, and `npm view @juanjofuchs/agent-mail@0.1.5 version` returns `0.1.5`. Verified by the agent against live registries, which CI cannot assert about its own unpublished build.
- [x] **AC13** (`integration`): The `npm-smoke.yml` workflow passes for `0.1.5` on Windows x64, Linux x64, macOS x64, and macOS arm64.
- [x] **AC14** (`manual`): On JJ's Windows 11 machine with `chcp` reporting 437, the original issue #1 repro — send a body containing `→`, then `npx -y @juanjofuchs/agent-mail@0.1.5 read <agent>` — exits 0 and prints the body. Requires the reporting machine's real console code page, which no hosted runner reproduces.

## Testing Approach

### Simulating the failure conditions

Both defects need their failure condition reproduced deterministically on Linux CI:

| Condition | How the test creates it |
|---|---|
| Non-UTF-8 console | Run the CLI with `PYTHONIOENCODING` set to a single-byte code page (`cp437`). Pre-fix this reproduces the reported crash on any OS; post-fix it also proves TC1, since the fix must override the variable. |
| Unwritable stdout | Bind the child process's stdout to a handle that rejects writes, so the failure lands on the first write rather than depending on timing. |

Tests decode child output as UTF-8 explicitly. The existing harness captures text using the parent's locale encoding, which would mask the very bytes AC2 asserts.

### Test cases

| Input | Expected output |
|---|---|
| `read` of a body with `→ — "smart" … 😀 日本語 café`, `PYTHONIOENCODING=cp437` | Exit 0; UTF-8 JSON with all characters intact |
| Same read, stdout bytes inspected | No `\uXXXX`, no `U+FFFD`, no BOM |
| `read` with unwritable stdout | Exit non-zero; JSON error on stderr; message still unread and still counted in `status` |
| Repeat `read` afterwards with normal stdout | Message returned, body intact |
| `send` with unwritable stdout | Exit non-zero; zero rows in `messages` |
| `ack` with unwritable stdout | Exit non-zero; `acked_at` unchanged |
| `cleanup` with unwritable stdout | Exit non-zero; no rows deleted |
| `read` with unwritable stdout and one already-expired message present | Expired message purged; unexpired message still unread |
| Invalid identity containing `é`, `PYTHONIOENCODING=cp437` | Exit non-zero; UTF-8-decodable JSON error containing `é` |

### Local validation before release

```bash
ruff check src/ tests/
pytest
python -m build
twine check dist/*
```

### Release validation

```bash
gh run watch
gh release view v0.1.5
npm view @juanjofuchs/agent-mail@0.1.5 version
```

### Human-in-the-Loop Release Protocol

1. **Agent**: Implements, passes local validation, and reports results.
2. **Agent**: Asks JJ before pushing the release tag.
3. **Human**: Approves the release tag.
4. **Agent**: Pushes `v0.1.5`, monitors GitHub Actions, and reports PyPI, GitHub Release, and npm status.
5. **Human**: Runs the AC14 repro on the cp437 Windows console that filed the issue and reports the result.
6. **Agent**: Closes issue #1 referencing `0.1.5`, or iterates if AC14 fails.

## Usage Examples

The repro from issue #1, expected to work after this spec:

```console
$ chcp
Active code page: 437

$ npx -y @juanjofuchs/agent-mail@0.1.5 send --from test:sender --to test:reader \
    --subject "arrow repro" --body "step 1 → step 2"
{
  "id": "d880ff28-...",
  ...
}

$ npx -y @juanjofuchs/agent-mail@0.1.5 read test:reader
[
  {
    "id": "d880ff28-...",
    "body": "step 1 → step 2",
    ...
  }
]
```

Readable rendering on a legacy console remains the user's choice:

```console
$ chcp 65001
```

## Out of Scope

- A `status` anomaly detector for `unread=0, unacked>0` — see Key Decisions.
- Changing error payload serialization to unescaped Unicode.
- Changing the console code page from inside the CLI.
- Input-side encoding: argv, `--body-file`, and `AGENT_MAIL_DB` paths are unchanged from spec 001 TC6.
- Redelivery, queueing, or retry semantics beyond leaving state unchanged on failure.
- Recovering messages already consumed by this bug on JJ's machine — a manual `mail.db` read, not a CLI feature.
- Workflow changes for CI, release, npm publish, or WinGet.
- The open WinGet and clean-machine verification items in spec 002; `0.1.5` does not depend on them.

## References

- Issue: [#1 — Windows: read crashes with UnicodeEncodeError (charmap/cp437) on non-codepage chars in message body](https://github.com/JuanjoFuchs/agent-mail-cli/issues/1).
- spec 001: [`specs/001-agent-mail-cli.md`](001-agent-mail-cli.md) — the behavioral contract this spec amends.
- spec 002: [`specs/002-packaging.md`](002-packaging.md) — PyPI and GitHub Release machinery that ships `0.1.5`.
- spec 003: [`specs/003-npm-distribution.md`](003-npm-distribution.md) — npm wrapper and smoke workflow used by AC13 and AC14.
- Implementation: [`src/agent_mail/cli.py`](../src/agent_mail/cli.py).
- Project context: [PROJECT_UNDERSTANDING.md](../PROJECT_UNDERSTANDING.md).