# dsh-plugin-rollout-scout

English | [简体中文](README.md)

[![npm](https://img.shields.io/npm/v/dsh-plugin-rollout-scout?color=cb3837&logo=npm)](https://www.npmjs.com/package/dsh-plugin-rollout-scout)
[![license](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
[![dsh](https://img.shields.io/badge/dsh-0.1.1--rc.2-4b8dff)](https://github.com/deepseek-ai/deepseek-harness)
[![stars](https://img.shields.io/github/stars/SpookySandwich/dsh-plugin-rollout-scout?style=flat&label=stars)](https://github.com/SpookySandwich/dsh-plugin-rollout-scout/stargazers)

Providers sometimes roll a new conversation model out gradually, so which one you get is luck of the draw. Rollout Scout opens throwaway conversations on your own account, reads each one's chain-of-thought **as it streams**, and scores how the reasoning is written — cancelling the ones that read like the model you already have, and keeping the ones that don't.

It is a curiosity tool built on phrase heuristics, not an oracle. Everything it does, you could do by hand: start a chat, glance at the reasoning, close the tab.

![the sidebar launcher](https://raw.githubusercontent.com/SpookySandwich/dsh-plugin-rollout-scout/master/assets/console.png)

## How it decides

The signal is in **how each paragraph opens** — not how often a phrase appears overall. A running tally of "Let me" drifts negative with length alone, so a long, perfectly good chain-of-thought eventually accumulates enough of them to look bad. Counting openings keeps the measure per-paragraph.

Only the first 48 characters of a paragraph are ever read. The phrase rarely sits at character zero:

> **The directory is empty. Let me create** a 3D cyberpunk scene.
> **To avoid conflicts, I'll keep** I18n.cs edits under one change.

Two signals are decisive, and they are deliberately **asymmetric**:

| Signal | Effect |
| --- | --- |
| `Let me` opening any paragraph | Old model — cancel the turn immediately |
| `I'll` opening the **whole** chain-of-thought | Rollout model — let it finish and keep it |

`I'll` counts as proof only at the very start, because old-model reasoning happily opens a *middle* paragraph with "I'll create a single HTML file…" and then says "Let me build…" three paragraphs later. Treating every `I'll` as proof produced false positives.

When neither fires, the remaining openings feed a score:

```
bonus      = (even paragraphs ? 1 : 0) + (any stall ? 1 : 0)
confidence = (positive openings + bonus + 1) / (classified openings + bonus + 2)
```

The bonus counts the summariser's shape as evidence in its own right, so a
chain-of-thought with even paragraphs and a stall reads above 50% before a single
opening has been classified. It only moves the number: keeping still needs
`Min. openings` real openings behind it.

A paragraph opening is scored as soon as 48 characters have arrived, even if the model never inserts a newline. `Let me` later in the chain-of-thought **overrides** an earlier keep.

The rollout path often uses a **small model to summarise** the chain-of-thought. That summariser commonly starts `We need to…` (so a first-paragraph `we` is only a negative opening, not a kill), writes **even, essay-sized paragraphs**, and streams in **bursts with stalls** in between. Those three are positive evidence of the new pipeline. The old model is irregular — one blob, or mixed tiny `Let me` lines.

Positive openings are the first-person **singular** planning voice — `I'm`, `I am`, `I've`, `I have`, `I need`, `I think`, `I also`, `I will`, and a leading `For`. Negative openings are `Let me` / `Let's` and **any** first-person plural in the opening (`we`, `we need`, `we will`, `we'll`, …). The rollout model reasons in "I", not "we". The add-one prior keeps a thin sample near 50% instead of swinging to a confident verdict off one word:

| Evidence | Confidence | Verdict |
| --- | --- | --- |
| nothing yet | 50% | keep watching |
| 1 positive, 10 negative | 15% | discard |
| 5 positive, 0 negative | 86% | keep |

A probe is discarded below `discard below` (0.35) and kept above `keep above` (0.7), but only once `min. openings` (4) have been classified. One that opens ten paragraphs without a single positive is given up on, and a chain-of-thought that is **mostly Chinese** (80%+ of its letters) is discarded on sight — quoting a Chinese prompt inside English reasoning does not count.

To leverage latency and throughput characteristics of rollout pipelines, two early-discard filters are available:
- **Maximum TPS limit**: Rollout models often generate at ~40–50 chunks/s, whereas old models stream significantly faster. Once streaming throughput exceeds the limit (default 60 chunks/s), the probe is discarded immediately.
- **Minimum First-Token Latency (TTFT)**: Old models often start streaming very quickly (< 1–2s). Set a minimum TTFT threshold (default 2.0s) to discard probes that respond too fast right on the first token.

The classifier is covered by tests over hand-labelled transcripts, alongside tests for the route guards, the launch loop's failure behaviour, and the rules about deleting sessions:

```bash
npm test
```

## The console

**Rollout Scout** sits at the sidebar foot, beside Settings, and opens the full-frame console described below. It is a `sidebar.footer.action` entry, so it matches the shell's own rows and collapses to a single icon when the sidebar folds to the rail.

**Left** — the probe prompt, model (default V4-Pro / High), concurrency, folder, and the four scoring thresholds. Below them sits the self-check, then the switches: TPS speed limit, first-token latency window, auto-pause on a strong match, discard Chinese reasoning, delete old-model probes from disk. The long explanation of the scoring rules folds away behind **How scoring works** rather than sitting between the controls.

The actions live in a footer under the column, so they stay put while the settings scroll, and are ordered by weight: **Start** / **Pause** on its own, **Force stop** and **Clear finished** beside each other, and **Delete all sessions** as plain text — it should not look like something you press by accident.

Under the scoring thresholds sits a **self-check**: `Self-check 13/13 · known rollout samples kept 5/5`. It runs thirteen hand-labelled chains-of-thought through the same classifier a live probe goes through, under whatever settings are currently in the form — no tokens, no probes. Tighten `Keep above` past what a labelled catch can reach and it turns amber and names the samples you just excluded. It is how you tell *nothing found* apart from *nothing findable*, which otherwise look identical from the console.

**Right** — launched / live / kept / discarded / best score, above a queue in launch order that never jumps. Each row has a score meter, matched phrases, and a preview. Click a card to open the conversation. Putting the mouse on a fading card rescues it; clicking straight into the conversation carries that rescue across the transition until the pointer moves again. To keep a probe for good, press **Keep**.

**Start** asks you to check your notifications first — a run opens a lot of conversations, and DSH Desktop can announce every one of them. Probe prompts are sent as plugin messages so they raise no toast of their own, and the dialog shows whether desktop notifications are on and offers to switch them off. Tick **Do not show this again** to skip it in future.

**Start** becomes **Pause**, which stops launching, cancels the probes already judged as the old model, and lets the undecided ones reach their own verdict. Then **Resume**. **Force stop** aborts everything still in flight.

A probe judged old fades for about 3 seconds (a thin line at the bottom of the card) while the turn is still running, then cancels. Hovering pauses the fade while your mouse is on the card — it resumes when you leave. Only **Keep** keeps it.

**Keep** on any card is a promise about the conversation on disk: it is never faded, swept or deleted, and the promise survives a plugin reload. It does not make a token-spending turn unstoppable — **Force stop** still ends the active turn while preserving both the conversation and its Keep mark. Probes that finish as a confident catch are kept automatically. Click **Kept** again to hand one back to cleanup rules.

Probes are named in the sidebar as you go — `Rollout probe 12` while running, and `★ Rollout catch 12 · 87%` once one is caught, so a hit is obvious in a list full of probes. **Rename** on a kept card lets you name it yourself; naming one keeps it.

**Clear finished** removes completed probes from the list *and* deletes those conversations from disk. **Delete all sessions** wipes every probe in the folder — including ones already cleared from the list — and resets numbering so the next run starts at probe 1.

If probe conversations turn up in the folder that the console is not tracking — left by a plugin reload, an upgrade, or an app restart — a banner offers to **sweep** them. Those are the ones that otherwise sit in the sidebar with no way to remove them, since the shell's own menu offers Archive but not Delete.

Both refuse to touch a probe that is still streaming or draining resources, including a kept one. Pause stops launching but leaves probes in flight, so **Delete all sessions** asks you to **Force stop** first and waits for host teardown before unlinking any log that may still be written to. The probe folder may not be your home directory, a drive root, or anywhere inside `~/.dsh` — deleting is scoped to that folder, and those would put unrelated conversations in its path.

If three probes in a row fail to even start — provider unreachable, folder unwritable — the run stops itself and reports the error instead of relaunching into the same failure forever. **Resume** tries again.

The run lives on the host, so it keeps going when you close the console — the sidebar row carries a status pip on its icon (pulsing while scouting, grey when paused, green on a catch), a live count in the wide column, a green badge with the number caught, and the tried count and best confidence so far on hover. In the rail the pip is the whole signal, which is why it sits on the icon rather than in the label.

## Install

```bash
dsh plugin --profile web add dsh-plugin-rollout-scout
```

Restart DSH afterwards: the host half loads with the server. The interface follows DSH's display language (English / 中文).

Installing from the repository also works and tracks `master` rather than the last release:

```bash
dsh plugin --profile web add github:SpookySandwich/dsh-plugin-rollout-scout
```

`web` is the profile name — use whichever profile you actually run. The standalone harness boots `web`; DSH Desktop uses `desktop`. `~/.dsh/profiles/` lists the ones you have, and the installed bundle ends up in that profile's `package.json`.

`lib/client.js` is a generated bundle but it is **committed**, so a repository install needs no build step. If you edit `plugin.client.js`, run `npm run build` to regenerate it (`npm test` does this too), and commit the result.

## How it works

- The host half serves `/rollout-scout` and creates each probe as a brand-new session with `ctx.agents.create` (no seed), setting model and reasoning effort through `installModelSelection`.
- Probe prompts are sent as plugin-sourced messages, which is what keeps a run from announcing itself once per probe.
- It subscribes to `session/event` scoped to that one agent and reads `reasoning-delta` chunks off `assistant/chunk` — the chain-of-thought as it streams — classifying on every chunk.
- A verdict against calls `agent.cancel` to interrupt the turn; a verdict for lets it run to `turn/end`. While streaming, the last paragraph is withheld because its opening may be half-written; at turn end the complete text is re-classified.
- Probes are created in the folder you choose, which becomes a workspace. Old-model probes can be deleted from disk.
- `/rollout-scout` listens on a local port, so it is guarded like one: writes require an `application/json` content type (which forces a CORS preflight that is never answered) and a cross-origin `Origin` is refused. A page you happen to be visiting cannot make it start or delete anything.

Implementation notes live in [`docs/`](docs/) — [architecture](docs/architecture.md) and the [harness behaviour this depends on](docs/dsh-host-notes.md).

## Compatibility

The launcher takes a `sidebar.footer.action` seat (a list slot, so it sits beside any other footer action rather than displacing one) and the console renders on the frame-wide `shell.overlay` layer. Neither is per-session, so it conflicts with no session plugin. Requires a DSH whose sidebar declares that seat; without it the console has no way in. Part of the same family as [dsh-plugin-smooth-stream](https://github.com/SpookySandwich/dsh-plugin-smooth-stream), [dsh-plugin-no-workspace](https://github.com/SpookySandwich/dsh-plugin-no-workspace) and [dsh-plugin-message-edit](https://github.com/SpookySandwich/dsh-plugin-message-edit).

## A note on cost

Every probe is a real turn against your own quota. Discarded ones are cancelled within a second or two, but a run left going will keep launching until you stop it. Concurrency and the thresholds are yours to tune.

## License

MIT © SpookySandwich