# dsh-llm-verifier

English | [中文](README.md)

> npm package name: **`dsh-llm-as-a-verifier`** (the bare name `dsh-llm-verifier` was already taken on npm; repo and package are unified as `dsh-llm-as-a-verifier`).

> Brings [LLM-as-a-Verifier](https://github.com/llm-as-a-verifier/llm-as-a-verifier) — the unified verification framework — into [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`) as a tool plugin. The agent can now verify its own candidates with **fine-grained probabilistic feedback**: instead of a binary good/bad judgement, the verifier's full logprob distribution over a 20-point score scale is read and its expectation taken.

## What you get

Three model-facing tools, registered automatically after install:

| Tool | Capability | Cost |
|---|---|---|
| `verify_compare` | Score two candidates (code/plan/trajectory) against criteria, returning fine-grained rewards `(scoreA, scoreB)` in [0,1] | 1 verifier call per criterion per evaluation |
| `verify_select` | Best-of-N via the Probabilistic Pivot Tournament: O(Nk) comparisons instead of an O(N²) round-robin | Linear in N |
| `verify_track` | Per-step progress curve scored on the A(0%)..T(100%) scale at each checkpoint | O(K) calls regardless of trajectory length |

**Why finer than LLM-as-a-Judge?** The upstream framework's key ideas, ported verbatim: ① fine-grained scoring granularity (20-letter scale); ② expectation over the **full logprob distribution** of score tokens; ③ reliability via repeated evaluations and criteria decomposition. This plugin ports the scoring extraction, pairwise prompts, pivot tournament, progress tracking, and token accounting, adapted into a Cordis tool plugin for DSH.

## Install

```sh
dsh plugin --profile web add dsh-llm-as-a-verifier
```

Requires dsh ≥ 0.1.0-rc.6 and Node ≥ 18. Restart `dsh web` (or wait for HMR).

## Configure the verifier backend

The verifier model must be an **OpenAI-compatible service returning token-level logprobs**: DeepSeek's hosted API, a local vLLM/SGLang server, OpenAI, etc.

In your profile config (`~/.dsh/profiles/<name>/cordis.patch.yml` or `~/.dsh/cordis.patch.yml`):

```yaml
- id: llm-verifier
  config:
    baseUrl: https://api.deepseek.com   # or vLLM: http://localhost:8000/v1
    apiKey: '${DEEPSEEK_API_KEY}'       # environment variables preferred
    model: deepseek-v4-flash            # omitted: DeepSeek defaults to deepseek-v4-flash, others auto-probe /models
    maxConcurrency: 8
```

**Credential resolution order** (upstream parity): plugin config → `OPENAI_BASE_URL` + `OPENAI_API_KEY` → `DEEPSEEK_API_KEY` (implies the DeepSeek endpoint with thinking enabled). With no credentials configured, tool registration still works and calls fail with `MissingAPIKeyError` only when executed.

```sh
export DEEPSEEK_API_KEY=sk-...   # the simplest setup
```

## Usage

Once installed, just ask the agent in the conversation:

```text
I wrote three candidate implementations. Use verify_select with
"correctness" and "performance" criteria to pick the best one, then use
verify_track to check whether my earlier fix steps made progress.
```

## Configuration reference

| Config | Default | Description |
|---|---|---|
| `model` | DeepSeek: `deepseek-v4-flash`; else auto-probed | Verifier model name |
| `baseUrl` | inferred from credentials | OpenAI-compatible endpoint |
| `apiKey` | inferred from environment | Prefer environment variables |
| `timeoutMs` | `60000` | Per-request timeout (ms) |
| `maxConcurrency` | `8` | Max in-flight verifier calls |
| `deepseek` | inferred from baseUrl | Force the DeepSeek call path (thinking enabled) |
| `prefill` | `true` | Prefill the score tags on non-DeepSeek servers (more reliable letter distribution on vLLM/SGLang) |
| `compare` / `select` / `track` | `true` | Register the corresponding tool |

Tool arguments (`nEvaluations`, `pivots`, `seed`, `groundTruthNote`, ...) mirror the upstream `llm_verifier` Python package; see the [user guide](docs/USER-GUIDE.md) and [SOP](docs/SOP.md).

## Library use

```ts
import { Verifier } from 'dsh-llm-as-a-verifier'

const verifier = new Verifier({ baseUrl: 'http://localhost:8000/v1' })
const { scoreA, scoreB } = await verifier.compare(problem, a, b, { Correctness: '...' })
const result = await verifier.select(problem, candidates, { Correctness: '...' }, { pivots: 2 })
const curve = await verifier.track(problem, steps, { checkpoints: [1, 3] })
```

## Development

```sh
npm ci
npm run check   # typecheck + vitest (76 cases incl. end-to-end against a local mock logprobs server)
npm run build
```

## License & attribution

MIT. The score expectation, pairwise prompts, Probabilistic Pivot Tournament, progress tracking and logprob extraction logic are ported from [llm-as-a-verifier/llm-as-a-verifier](https://github.com/llm-as-a-verifier/llm-as-a-verifier) (MIT); full attribution in [LICENSE](LICENSE).