# Delivery-quality report — opencode

## 1. Environment

- date: 2026-09-24
- harness: opencode (1.18.32)
- model: dp/deepseek-flash
- skill commit: fbfa606
- repeats per arm: bare N=5, skill N=3
- contamination check: yes (reply: 'OK')
- skill visible in a prepared workspace: not applicable to the bare arm; yes for the skill arm
- permissions: workspace opencode.json delivery variant: edit allow, bash still git-only; the bare arm additionally denies the skill permission because opencode also loads user-level skills from ~/.config/opencode/skills, where an installer test left an intent-router copy — without that deny the bare arm silently used the skill (its first turn emitted the IntentSpec format verbatim)
- scripted user: answer_when_asked: 'A' / max_answers: 3 / implement_prompt: 'Implement it now in this repository, exactly as decided abov'… / max_sessions: 5
- skill arm invocation: explicit (isolates trigger probability from delivery quality; trigger rate is measured separately by add-caching-auto)

`opencode.json` (delivery variant) written into every workspace:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "permission": {
    "read": "allow",
    "glob": "allow",
    "grep": "allow",
    "skill": "allow",
    "bash": {
      "git log*": "allow",
      "git show*": "allow",
      "*": "deny"
    },
    "edit": "allow",
    "task": "deny",
    "question": "deny",
    "webfetch": "deny",
    "websearch": "deny",
    "external_directory": "deny",
    "doom_loop": "allow"
  }
}
```

## 2. Summary

| metric | bare | skill |
|---|---|---|
| runs judged (environmental excluded) | 5 | 3 |
| mean score /6 | 5.0 | 6.0 |
| reuses_shared_redis | 5/5 | 3/3 |
| uses_default_ttl | 5/5 | 3/3 |
| invalidates_on_write | 5/5 | 3/3 |
| covers_all_reads | 5/5 | 3/3 |
| states_failure_policy | 0/5 | 3/3 |
| contract_preserved | 5/5 | 3/3 |
| ADR read (aux) | 5 | 3 |
| no_delivery runs | 0 | 0 |
| mean sessions | 1.0 | 3.0 |
| mean wall-clock s | 50.7 | 140.5 |
| mean questions answered | 0.0 | 1.0 |

## 3. Per run

| arm | run | sessions | states | wall s | answered | score | reus | uses | inva | cove | stat | cont | adr | files |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare | 1 | 1 | - | 51.9 | 0 | 5/6 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | 1 (37+) |
| bare | 2 | 1 | - | 39.7 | 0 | 5/6 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | 1 (37+) |
| bare | 3 | 1 | - | 54.7 | 0 | 5/6 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | 1 (38+) |
| bare | 4 | 1 | - | 59.8 | 0 | 5/6 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | 1 (34+) |
| bare | 5 | 1 | - | 47.2 | 0 | 5/6 | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | 1 (32+) |
| skill | 1 | 3 | ASK → ROUTE → - | 153.5 | 1 | 6/6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 2 (117+) |
| skill | 2 | 3 | ASK → ROUTE → - | 126.1 | 1 | 6/6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 1 (34+) |
| skill | 3 | 3 | ASK → ROUTE → - | 141.9 | 1 | 6/6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 2 (71+) |

## 4. Observations

_Facts only, one line each: skill defects seen (not fixed, P5), runner anomalies, environmental reruns._

- The first full run (N=3 per arm) was discarded: the bare arm silently loaded the skill from the
  user-level `~/.config/opencode/skills` (an installer test had left an intent-router copy there);
  its turn 1 emitted the IntentSpec format verbatim, confirmed by 105 transcript hits including an
  explicit skill-tool call. Fixed by denying the skill permission on the bare arm (runner commit
  f60fefe); the discarded snapshots stay under `raw/delivery/opencode/*/archive/`.
- One skill-arm run (archived, not part of N) wrote the emitted spec into `.intent/` while asking —
  the workspace had no `.intent/` directory and the user had not asked for a file, so the skill's
  write-to-disk rule was violated; counting that file as a diff also ended the loop before the
  implement prompt was sent. The runner now excludes `.intent/` (commit fbfa606); the skill rule
  itself is not fixed here (P5).
- The first scorer missed both wrapper shapes the runs produced (cache calls wrapped in local
  `cachedRead`/`invalidate` helpers on the bare arm; a new `src/cache/users.ts` module on the
  skill arm); fixed by one-hop call resolution through a symbol index, with two synthetic selftest
  snapshots added (commit 8759476). Manual diff read-back (bare-1, skill-1, plus the smoke pair)
  found no scorer misjudgement after the fix (V13).
- Bare behaviour was identical in all 5 judged runs: one session (~50 s), zero questions, direct
  implementation, and no failure policy anywhere — `states_failure_policy` 0/5.
- Skill behaviour was identical in all 3 judged runs: ASK → answer A → implement (3 sessions,
  ~140 s), the invalidation-failure policy stated in code comments and try/catch — 6/6.
- ADR 0007 was read in every run on both arms (aux): the bare arm avoids the in-process trap
  through its own read-before-write habit, not by chance.
- Cost: the skill arm spends +2 sessions and ~+90 s wall-clock per run to gain the one item the
  bare arm never produces (`states_failure_policy`).
- Environmental reruns: none. `skill_not_loaded`: none. Rate-limit retries: none observed.
- V12 rescore equivalence: two consecutive `--rescore-delivery` renders of these snapshots are
  byte-identical.
- Post-run audit (2026-09-24): the effective skill-arm runs 1 and 2 loaded the intent-router copy
  left in the user-level `~/.config/opencode/skills` (run 3 loaded the workspace copy in
  `.agents/skills/`); the copy's 12 files are byte-identical to the skill at `fbfa606` except for
  line endings, so the published numbers hold. The copy has been removed, and the runner now
  refuses to start while any user-level copy exists.
- Post-run audit (2026-09-24, later the same day): the §1 claim "bash still git-only" was wrong in
  effect — transcripts show the bash tool was hidden entirely from the model (`unavailable tool
  'bash'`, 36 hits across transcripts). Root cause: opencode 1.18.32 resolves tool visibility from
  the LAST rule of a permission object, so a bash object ending in `"*": "deny"` hides the tool
  regardless of the specific allow patterns. The `opencode.json` snapshot above is the config as
  actually written at run time and stays untouched; the fix (catch-all `"*"` first, allow patterns
  after) landed in the runner's `OPENCODE_CONFIG` so future runs get the intended restricted
  git-log/show channel. Case conclusions are unaffected: assertions target behaviour and evidence
  form, and git-only-fact passed by reading `.git` files after two unavailable-bash calls.

## 5. Line for the README

```
2026-09-24 · opencode · dp/deepseek-flash · delivery add-caching-delivery · bare 5.0/6 (N=5) · with skill 6.0/6 (N=3) · ADR trap avoided bare 5/5 vs skill 3/3
```