# Office Tools Benchmark

_Generated by `bun eval/report.ts` from `eval/results/*.json` on 2026-07-18T04:55:58.571Z. Do not hand-edit — re-run the generator instead._

An 8-card agent-level eval battery, run through real OpenCode + the `opencode-plugin-office`
plugin, scored programmatically against office-core ground truth. Each card asks for one
concrete change to a `.docx`/`.pptx` fixture; scoring checks both that the change landed
(**Task success**) and that everything else in the file is untouched (**Fidelity**).

## Results

| Model | Class | Task success | Fidelity | Median s/card | Notes |
|---|---|---|---|---|---|
| ollama/qwen3.5:397b-cloud | ollama-cloud | 8/8 | 8/8 | 22.26s | clean sweep — full success and fidelity |
| opencode/big-pickle | opencode-hosted | 8/8 | 8/8 | 32.64s | clean sweep — full success and fidelity |
| zai/glm-4.5-flash | api (zai) | 7/8 | 7/8 | 58.61s | 1/8 task miss: pptx-insert |
| ollama/gpt-oss:120b-cloud | ollama-cloud | 6/8 | 7/8 | 21.23s | fidelity breach: docx-replace — comment_refs expected 1, got 0; 2/8 task misses: pptx-image, pptx-create |
| ollama/minimax-m3:cloud | ollama-cloud | 5/8 | 8/8 | 19.18s | 3/8 task misses: docx-create, pptx-insert, pptx-image |
| opencode/nemotron-3-ultra-free | opencode-hosted | 5/8 | 7/8 | 28.54s | 3/8 task misses: docx-replace, docx-create, pptx-create |

Legend: ✓ full pass   ⚠ completed the task but damaged something else (fidelity breach)   ✗ missed the task, nothing else touched   ✗✗ missed the task and damaged something else.

### Per-card matrix

| Model | docx-replace | docx-insert | docx-table | docx-create | pptx-retitle | pptx-insert | pptx-image | pptx-create |
|---|---|---|---|---|---|---|---|---|
| ollama/qwen3.5:397b-cloud | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| opencode/big-pickle | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| zai/glm-4.5-flash | ✓ | ✓ | ✓ | ✓ | ✓ | ✗✗ | ✓ | ✓ |
| ollama/gpt-oss:120b-cloud | ⚠ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| ollama/minimax-m3:cloud | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ |
| opencode/nemotron-3-ultra-free | ✗✗ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ |

## Partial / discontinued

Three local models were discontinued mid-run by author decision — the roster shifted from
local inference to cloud partway through the battery, and these were not re-run. Their partial
results (1-3 of 8 cards) are kept for the record but excluded from the ranked table above: a
1/8 or 3/8 sample isn't comparable to a completed 8/8 run.

| Model | Class | Cards run | Success | Fidelity | Notes |
|---|---|---|---|---|---|
| ollama/llama3.1:8b | local (8B) | 1/8 | 0/1 | 1/1 | 1/8 task miss: docx-replace |
| ollama/qwen2.5-coder:14b | local (14B) | 1/8 | 0/1 | 1/1 | 1/8 task miss: docx-replace |
| ollama/qwen3-coder:30b | local (30B) | 3/8 | 0/3 | 3/3 | 3/8 task misses: docx-replace, docx-insert, docx-table |

## Unavailable

Models the roster intended to include but that never produced a scored run:

- **openai/gpt-5.5** — provider credential expired — 401 on token refresh
- **openai/gpt-5.4-mini** — provider credential expired — 401 on token refresh
- **kimi-k3** — no such upstream model — only k2.x releases exist
- **ollama-cloud (provider entry)** — 401 on every request; superseded by declaring cloud models under the working `ollama` provider instead

## Method

- **Scoring is programmatic office-core ground truth.** Every check reads the resulting
  `.docx`/`.pptx` (and, where needed, runs a Python-side probe such as `docx_probe.py` /
  `pptx_probe.py` for comment references, tracked changes, or image content hashes) and
  compares it against the expected structure. Nothing is scored from the agent's own prose —
  an agent claiming success proves nothing here.
- **Prompts are tool-agnostic.** Cards describe the file, the absolute path, and the concrete
  change; they never name `office_read`/`office_edit`/`office_create`/`office_python`.
  Tool selection is part of what the battery measures, so it can't be handed to the model.
- **`success` and `fidelity` are separate, orthogonal scores.** `success` = the asked-for
  change landed. `fidelity` = everything the prompt didn't ask about is still byte/value
  identical. For the `docx-create` card there is no pre-existing document to damage, so
  `fidelity` is always `true` by construction — including on a run that never produced the
  output file. That's intended: `success` alone captures the miss for a create card, and
  `fidelity` is reserved for collateral damage, which a from-scratch file can't cause.
- **Reproduction gotcha.** opencode treats `provider.<id>.models` in `opencode.json` as an
  **allowlist** for custom OpenAI-compatible providers. A model not declared there fails with
  `ProviderModelNotFoundError`, surfaced to the caller as an opaque "Unexpected server error"
  — if you add a model and it fails immediately with that message, check the allowlist before
  suspecting the provider.
- **Hardware.** MacBook Pro M4 Max, 128 GB unified memory. Every model in the roster below runs
  through a cloud provider (opencode-hosted, ollama-cloud, or a direct API), so this hardware
  bounds the harness (arena I/O, office-core scoring, process orchestration) — it does not
  bound model inference.
- **Variance caveat.** A single run per card per model. Cloud model outputs vary run to run;
  treat single-digit success/fidelity differences between models as noise, not signal.
  Local-model numbers (where present) additionally depend on quantization and would not
  reproduce on different hardware.
- **Refresh policy.** Re-run when the tool surface changes in a way that could invalidate a
  score (new office-core operations, changed fixtures, changed card prompts/checks) — not on a
  calendar. A stale-but-still-valid benchmark is preferable to a fresh one measuring something
  else.
- **Mid-battery re-measurement.** An ollama-cloud usage limit was hit partway through the run
  and lifted by a subscription upgrade. Two fast-fails were re-run with `--force` after the
  upgrade: `ollama/gpt-oss:120b-cloud` `docx-table` (4.98s fail → 44.74s PASS, reproduced
  success and fidelity) and `ollama/minimax-m3:cloud` `docx-create` (5.02s fail → 6.86s
  fail, reproduced). One of these re-measured cards, the gpt-oss docx-table, flipped to a
  published pass — a non-timeout failure that survived re-run was confirmed genuine. Timeout
  failures (300s) were also re-run and confirmed to be quota stalls; the numbers below are
  the post-upgrade values.
- `ollama/gpt-oss:120b-cloud` is the clearest example the battery has produced: it passed `docx-replace`'s text change (the sentence reads correctly) but the fidelity probe caught it destroying the paragraph's comment reference marker in the process (`comment_refs expected 1, got 0`). Task success alone would have scored this a clean pass — exactly the class of damage this battery exists to detect.
- **2026-07-18.** Results measured against the pre-release-prep harness (fixtures/prompts changed in release prep); re-run required before publishing updated numbers.