# Evals Measured checks on model behaviour, kept separate from `__tests__` because they call real models, cost money, and produce rates rather than pass/fail. ## Layers | Layer | Where | Runs | Needs keys | | -------------------- | --------------------- | ------------------------------------------ | ---------- | | Evaluator unit tests | `evals/lib/__tests__` | every PR, via `bun run test` | no | | Experiments | `evals/run-*.ts` | on prompt and model changes, and on demand | yes | The first layer exists because an evaluator is code too. `analyzeRelatedQuestions` is tested against recorded spec blocks so a broken evaluator cannot quietly report a healthy rate. Those tests are part of the normal suite and run on fork PRs, where secrets are unavailable. ## related-questions ```bash bun run eval:related-questions # deployed adaptive model, 1 trial per item bun run eval:related-questions -- --mode quick # the quick search mode instead bun run eval:related-questions -- --trials 3 # 3 generations per item bun run eval:related-questions -- --model openai:some-model # candidate model bun run eval:related-questions -- --items fuji-routes # one item, for debugging bun run eval:related-questions -- --help ``` Answers 22 fixed queries (10 substantive, 12 trivial) against pinned search results and reports: - `emissionRate`: substantive answers that carried a related questions block - `falsePositiveRate`: trivial answers that carried one anyway - `wellFormedRate`: emitted blocks that satisfy the spec contract (one block, three buttons, `submitQuery`, button text identical to the query) Emission and well-formedness are separate metrics on purpose. A model that stops emitting and a model that emits a broken block are different failures with different fixes. ### Search modes `--mode` selects which deployed configuration is replayed. The two differ in more than wording: quick mode runs a different system prompt, a smaller tool set, and its own `providerOptions`, so a result measured on one mode says nothing about the other. The flag defaults to `adaptive`, which is only a default for a bare local run; CI runs both modes as a matrix, because a prompt or model change can move one and leave the other alone. That is the whole reason the modes are separable here. Both modes are gated against the same `thresholds.json` entry. There is no per-mode threshold because nothing yet needs one; add one when a mode's healthy range is genuinely different rather than to make a run go green. ### Why it is built this way - **Search results are fixed.** Live search would put retrieval variance into a measurement about the prompt and the model. - **The task mirrors production.** Same prompt builder, same `providerOptions` from `config/models/cloud.json`, same `toModelOutput` trimming. Reasoning effort has moved emission rates before, so a run without provider options would measure something the product does not do. - **`BRAVE_SEARCH_API_KEY` is pinned by the runner.** The adaptive prompt branches on whether a general search provider is configured, so an unset key would silently evaluate a different prompt. No request reaches Brave; the search tool is a fixture. - **Each trial is its own item.** Emission is probabilistic, so one generation per query measures almost nothing. `--trials` repeats each query while keeping one item equal to one generation in Langfuse. ### What counts as trivial An item is `trivial` only when the fixture holds exactly one unambiguous value answering it and no comparison or procedure is implied. "How tall is Mount Fuji" qualifies. "What is the typical energy density of an LFP pack" does not, because the fixture puts LFP and NMC side by side and the answer naturally becomes a comparison. The distinction matters because the label is set by the shape of the question while the model decides by the shape of its own answer. A loosely trivial question draws out a structured answer, and the block that follows it is arguably correct, which inflates `falsePositiveRate` with disagreement rather than defects. Questions whose answers expand belong in the substantive set. ## source-cards ```bash bun run eval:source-cards # adaptive, 1 trial per item bun run eval:source-cards -- --mode quick --trials 3 ``` Answers 16 fixed queries (8 listing, 8 other) and reports: - `emissionRate`: listing answers ("find", "recommend", "where can I get") that presented the items as linked cards (`Card` with `href` inside a `Grid`) - `misuseRate`: other answers (explanation, steps, comparison, single fact, writing) that carried cards anyway - `wellFormedRate`: emitted blocks with 2 to 8 cards, each with a title `Text` and a unique web `href` - `groundedRate`: cards whose URL is a fixture search result URL, verbatim - `duplicatedRate`: answers that also repeated the cards as a markdown list Fixture URLs use real-looking hosts. With `.example` hosts the adaptive model correctly judged the results to be placeholders and declined to present them, which measured the fixture instead of the prompt. ## draft-edits ```bash bun run eval:draft-edits # synthetic chats, plus local production seeds if present bun run eval:draft-edits -- --trials 3 bun run eval:draft-edits -- --no-synthetic # production seeds only ``` Replays one turn in a chat that already holds a draft. Each item carries the chat so far and goes through the same history preparation as the chat route (draft compaction, then `` attached to the latest user message), with its own mode's prompt, tools, and `providerOptions`. The real `editDraft` tool runs against the rebuilt draft, so a call is recorded exactly as production would apply it. Reports: - `hijackRate`: unrelated requests (a question, a keyword list) where `editDraft` was called at all - `newBlockRate`: requests for a new piece on another topic answered with a new complete draft block and no `editDraft` call - `editSuccessRate`: requests to change the draft that changed it, either through an applied edit or a new complete block - `draftSuggestionRate` / `revisionSuggestionRate`: answers that wrote or changed the draft and ended with follow-up suggestions, and with suggestions under the "Revise" heading the prompt asks for on a draft answer - `unrelatedRevisionRate`: unrelated requests answered with "Revise" suggestions, which belong only to draft answers The committed dataset is synthetic. Production turns, where the hijack was first observed, hold users' own text, so they are built into `evals/_local/draft-edits-seeds.json`, which is gitignored and loaded only when present. Output for production items is never printed, only ids and verdicts. ## Thresholds `thresholds.json` gates CI. A `null` value means the metric is reported but not enforced, which is the state a new metric starts in. `minSamples` is the smallest denominator a threshold may be enforced on, and the sampled rates set it at 50, which the five trials CI runs reach and a local run does not. That is not caution for its own sake: identical code measured 1.000 and 0.833 on `emissionRate` across two runs of thirty substantive samples, so a three-trial run cannot carry a ceiling or a floor. `completionRate` declares no minimum because it counts items that actually failed rather than sampling a probability. `completionRate` guards the measurement itself: the experiment runner drops an item whose task threw, which shrinks the denominator of every other rate. A run that lost more than a tenth of its items is not a valid measurement. Rates are read at the run level, never per item. At three trials an individual query swings by a third from one run to the next, so a single query moving is noise; raise `--trials` before drawing a conclusion about one of them. Whole runs move too: the same code measured 0.28 and 0.36 on `falsePositiveRate` in consecutive runs, which is why CI runs five trials and why a before/after comparison needs more than one run on each side. **Set a baseline before enforcing anything.** Run the eval a few times on the current production model, then commit the numbers. Committing them rather than comparing against the previous Langfuse run is deliberate: a threshold that gets loosened should appear in a diff with an author and a reason. A violation throws `RegressionError`, which the Langfuse experiment GitHub Action reads to fail the job. Be honest about what a single gated run proves. `emissionRate` separates cleanly: the failure it exists to catch is a model swap that empties the block entirely, and fifty samples settle that. `falsePositiveRate` does not. Sampled three times before the trivial-skip fix it read 0.200, 0.278 and 0.361, and twice after it read 0.083 and 0.111, so the distributions overlap at the tails and no ceiling separates them on one run. That gate catches a blowup. Deciding that a change moved the metric still takes several runs on each side. ## Datasets `evals/datasets/*.json` holds hand-written, synthetic data. It is committed, so a prompt change and the data it is measured against show up in the same review. Anything derived from production traces (thumbs-down turns, errored turns) must **not** be committed here. This repository is public. Put that data in a Langfuse managed dataset and fetch it at run time: ```ts const dataset = await langfuse.dataset.get('morphic-thumbs-down-regression') ``` `experiment.run` accepts local items and Langfuse dataset items interchangeably, so the task and the evaluators do not change when the source does. ## Credentials `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_BASE_URL`, plus the API key for the provider under test. The runner loads `.env.local`. Keys are required: the experiment runner uploads item scores through the Langfuse client whether or not spans are exported, so a keyless run fails partway rather than degrading to a local-only run. The CI job skips itself with a warning when they are absent, so a repository without them does not collect a red mark on every PR that touches `evals/`. Experiment traces are tagged by the SDK with the `sdk-experiment` environment, so they can be excluded from production metrics without a separate project. ## Known quirk When items are skipped, the `Input` and `Expected` lines in the printed item list are misaligned: the Langfuse formatter pairs `itemResults[i]` with `originalData[i]` after the failed entries have been filtered out. The scores, the outputs, and the trace links are unaffected, because each result carries its own item. The runner prints a warning whenever this applies.