# Benchmark: how the four DSH compaction strategies actually behave

This is a controlled, reproducible comparison of the compaction strategies that exist in the
DeepSeek Harness ecosystem today. It exists because "zero-latency compaction" is a claim, and a
claim without a measurement is marketing.

Every number below comes from `benchmark/` in this repository and can be regenerated with one
command per arm.

---

## What is measured

| metric | how it is obtained | why it matters |
| --- | --- | --- |
| **blocking latency** | `compaction/end.time − compaction/start.time` from the session event log, per transaction | the base engine appends `compaction/start` *before* it builds the summary input, so this interval is exactly the time the turn is frozen |
| **compactions** | number of `compaction/start` markers | how often the strategy fired |
| **peak / final context** | provider-reported `usage.totalTokens` | whether the strategy actually bought headroom |
| **billed tokens** | summed `usage.input/output` across the session | whether the strategy saved money or spent it |
| **facts kept** | exact constants from `doc-01` reproduced in the final answer | whether compression was lossy on facts the user cares about |
| **wall clock** | process wall time for the whole task | end-to-end user-visible duration |

`blocking latency` is the primary metric. Everything else is context for it.

## The workload

16 generated markdown documents of ~12 KB each (~49 K tokens total) live in `benchmark/fixtures/`.
Each ends with a `## Recorded constants` block holding an exact 8-hex service token, a port, and a
retention period.

The task asks the agent to read `doc-01` … `doc-16` strictly in order, one `read` call per step,
say nothing until all 16 are read, and then report the three constants **from the first
document**. `doc-01` is therefore always inside the span that gets summarised, which is what makes
fidelity measurable at all.

The fixtures are generated by a seeded PRNG and committed, so the task is identical across arms and
across re-runs.

## The four arms

| arm | strategy | what it represents |
| --- | --- | --- |
| **A** | stock `compaction-basic` | the baseline the harness ships with: foreground LLM summary |
| **B** | `tool-result-pruner` + `compaction-basic` | the usual first thing people turn on: prune tool output, then summarise what is left |
| **C** | [`@aiwayds/dsh-dcp`](https://github.com/fan56/dsh-dcp) | the strongest non-LLM alternative: deterministic, rule-based summarization (itself a `BasicCompactionEngine` subclass) |
| **D** | this plugin | LLM summarizer, run in the background before the threshold is reached |

C is the important arm. It is not a strawman: `dsh-dcp` replaces the summarizer with a rule-based
one, which means **it also never blocks on a model**. Any comparison that ignored it would be
dishonest.

All four arms share one profile and one model. Threshold settings are pinned identically
(`contextWindow: 60000`, `thresholdRatio: 0.6`, `headroomTokens: 0`, `retainRatio: 0.1`,
`maxTokens: 4096`), so the trigger point is 36,000 tokens in every arm. Only the compaction stack
differs.

## Results

Three runs per arm, median reported. Full per-run detail and session ids are in
[`benchmark/RESULTS.md`](benchmark/RESULTS.md).

| arm | strategy | compactions | blocking (median) | worst single | wall clock | peak ctx | final ctx | billed in / cached / out | facts kept |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: |
| A | stock `compaction-basic` | 1 | **16.1 s** | 16.1 s | 95.9 s | 35,176 | 30,927 | 150,978 / 378,717 / 1,024 | 3/3 |
| B | `tool-result-pruner` + basic | 1 | **15.3 s** | 15.3 s | 94.9 s | 35,635 | 28,519 | 158,196 / 373,455 / 1,021 | 3/3 |
| C | `dsh-dcp` (zero-LLM) | 2 | **12 ms** | 8 ms | 118.4 s | 36,288 | 32,962 | 199,068 / 446,297 / 529 | 3/3 |
| D | `background-compaction` | 1 | **7 ms** | 7 ms | 90.4 s | 35,811 | 34,122 | 154,249 / 384,333 / 983 | 3/3 |

## What the numbers say

### 1. Foreground summarization is the entire cost, and it is huge

A and B freeze the turn for **15–16 seconds** every time they fire. B is the interesting one: even
though it prunes tool output first and therefore summarises a smaller payload, the saving is only
~5% (16.1 s → 15.3 s). The cost is not the size of the input; it is the round trip to the model.

C and D both eliminate it. **D blocks for 7 ms, A for 16,095 ms: a factor of ~2,300, or 99.96% of
the stall removed.**

### 2. This plugin is *not* the only way to get zero latency — and that is worth stating plainly

C (`dsh-dcp`) is within a few milliseconds of D. On **blocking latency the two are
indistinguishable**; the 7 ms vs 12 ms gap is event-loop scheduling noise, not a design win.

What separates them is *what the summary is made of*:

|  | C — deterministic | D — speculative LLM |
| --- | --- | --- |
| summary is produced by | rules | the model |
| can compress semantics | no — keeps identifiers, drops prose | yes |
| background token cost | zero | one summary call, cache-amortised |
| failure mode | silently drops meaning that no rule preserves | model unavailable / drift → falls back to foreground |

Neither dominates. C is free and predictable; D pays a small token cost for a summary that can
actually reason about what mattered.

### 3. The cost of speculation is about 2% of the token bill

D versus A, same workload, same summarizer:

- billed input **154,249 vs 150,978** (+2.2%)
- cached input **384,333 vs 378,717** (+1.5%)
- output **983 vs 1,024** (−4.0%)

The speculative call is nearly free because it runs against the same prompt prefix that is already
warm in the provider's cache. That is the single largest design constraint in the plugin — the
pinned span exists precisely so the background call and the eventual commit share one cache
prefix — and the measurement confirms it holds in practice.

C's higher billed input (199,068, +32%) is a run-level artifact rather than a summarization cost:
its sessions took more steps and compacted twice. It is not evidence that determinism is expensive.

### 4. Fidelity did not discriminate — and that is a real limitation of this benchmark

All four arms reproduced all three constants in all three runs. **Nothing here proves that LLM
summarization preserves facts better than rule-based summarization.** One compaction of a ~14 K
token span is not enough pressure to separate the strategies on that axis; separating them would
need a workload with many sequential compactions and facts scattered across every one of them.

Do not read the 3/3 column as a victory. It is a floor: no arm is obviously lossy on
identifier-shaped facts at this scale.

### 5. A wasted background call happens when the pruner rewrites the surface

The trace from arm D shows the real failure mode of speculation:

```
[bgc] precompute start  span=8..18  tokens=13820
[bgc] precompute ready  held=1
[bgc] compactIfNeeded   tokens=35019 threshold=36000 held=1
[bgc] dropped held summary invalidated by tool-result pruner
[bgc] precompute start  span=8..54  tokens=17255
[bgc] precompute ready  held=1
[bgc] served held summary saved~17255t
```

The base engine runs `toolResultPruner` before compacting. Pruning rewrites the surface, the pinned
span stops matching, and the held summary is discarded. The plugin detects this, books a miss
against its circuit breaker, and immediately re-precomputes on the pruned surface — which is
correct behaviour, but it means **one background summary per session can be thrown away**. It
happened in 2 of the 3 arm-D runs.

That is the honest price of speculating against a surface that another component is allowed to
rewrite underneath you.

## Reproducing

```bash
node benchmark/gen-fixtures.mjs 16 12000     # regenerate the 16 documents + manifest
node benchmark/run-arm.mjs a b c d           # one run per arm
node benchmark/run-arm.mjs a b c d --rep 2   # repeat for a distribution
node benchmark/report.mjs                    # markdown table + per-run detail
```

`run-arm.mjs` installs each arm's packages around its own run and refuses to record a result if the
composed profile mounts a different set of backends than the arm expects. That gate exists because
a silent double-mount produces a run with **zero compactions and no error at all**, which looks
exactly like a strategy that did nothing.

## Threats to validity

- **One model, one provider.** `gemini / gemini-3.8-flash-high` behind a local proxy. Absolute
  latencies will differ on other stacks; the ratio between foreground and background should not.
- **One task shape.** Sequential reads producing ~49 K tokens. Strategies that shine on tool-heavy
  or code-heavy workloads are not exercised.
- **n = 3.** Enough to see that A/B are an order of magnitude apart from C/D; not enough to
  resolve differences below ~2 s.
- **`dsh-dcp` was pinned to `thresholdRatio: 0.6`** instead of its shipped default of `0.7`. At 0.7
  the run finished without ever compacting, so no comparable data existed. The comparison is of
  *strategies at a matched trigger point*, not of shipped configurations.