# Verification

How claims in recommendations are mechanically verified, and when the recommender re-runs after a low pass rate.

## Table of contents

- [Why mechanical verification](#why-mechanical-verification)
- [Claim types](#claim-types)
- [Dispositions](#dispositions)
- [Re-gen trigger and accept criteria](#re-gen-trigger-and-accept-criteria)
- [Verifier implementation](#verifier-implementation)

## Why mechanical verification

The recommender is an LLM. LLMs hallucinate counts, miscount file occurrences, and confuse code snippets between similar-looking files. Mechanical verification — grep + filesystem reads + JSON checks against `signals.json` and `references/docs-library.json` — catches these failures before the customer sees them.

The checker covers explicit structured verifiableClaims, listed files/citations, and selected prose triggers. It does not extract or prove every numeric claim, arbitrary snippet, causal statement or contradiction. A passing disposition means that specific mechanical check passed, not that the recommendation is runtime-safe. Review source and test the change before applying.

## Claim types

The verifier uses selected triggers in these fields and accepts explicit structured claims. Counts, patterns and snippets generally require explicit verifiableClaims rather than automatic prose extraction. Implemented handlers include:

| # | Claim type | Pattern in rec | Verification |
|---|---|---|---|
| 1 | `pattern_count` | "N fetch() calls in file X" | grep/ast-grep in X, exact count match |
| 2 | `pattern_exists` | "uses JSON.parse(JSON.stringify())" | grep, boolean |
| 3 | `pattern_absent` | "no Cache-Control header" | grep, verify absence (with guards — see below) |
| 4 | `file_exists` | "app/not-found.tsx exists" | File boundary, existence and cited-line bounds |
| 7 | `code_snippet` | Code fence labeled "Before:" | substring search in cited file |
| 9 | `repo_count` | "11 unstable_cache usages across 8 files" | grep repo, count distinct files |
| 11 | `citation_in_library` | Any URL in `citations[]` | URL ∈ `references/docs-library.json` |
| 12 | `citation_applies_to_version` | Any URL in `citations[]` | URL's `applicableFrameworks` matches `signals.json.stack.framework@frameworkVersion` |
| 13 | `cache_vary_matches_dynamic_inputs` | CDN cache rec touches route files that read Vercel geolocation | Fails unless the rec varies by a coarse Vercel geolocation header such as `X-Vercel-IP-Country`, `X-Vercel-IP-Country-Region`, or `X-Vercel-IP-City` |
| 13a | `cache_vary_cardinality_safe` | CDN cache rec sets `Vary` on request-specific geography | Fails on high-cardinality `X-Vercel-IP-Latitude` / `X-Vercel-IP-Longitude` / `X-Vercel-IP-Postal-Code` |
| 14 | `next_cached_not_found_causal_support` | Rec claims `notFound()` inside `'use cache'` caused 5xx | Fails unless backed by Next-specific docs or runtime stack evidence |
| 15 | `next_stable_cache_api_for_version` | Next.js 16 cache rec includes code examples | Fails on `unstable_cacheLife` / `unstable_cacheTag` or one-arg `revalidateTag()` |
| 16 | `next_cache_components_runtime_cache_preference` | Next.js rec uses Runtime Cache APIs while `cacheComponents=true` | Fails unless `use cache: remote` is used or Runtime Cache is framed as a fallback |
| 17 | `next_cache_components_route_segment_config` | Next.js 16 rec suggests removed route segment config while `cacheComponents=true` | Fails on `dynamicParams`, `dynamic`, `revalidate`, or `fetchCache` recommendations |
| 17a | `next_route_revalidate_static_prereq` | Rec suggests route-level `export const revalidate` for a Next.js page/layout route | Fails when the route chain contains request-time APIs or common auth helpers that can force dynamic rendering |
| 18 | `next_cache_lifetime_freshness_supported` | Rec lengthens a tagged Cache Components lifetime with `cacheLife()` | Fails unless every affected `cacheTag()` has matching `revalidateTag()` / `updateTag()` evidence |
| 19 | `next_cache_life_cdn_header_semantics` | Rec claims `cacheLife()` emits CDN/Cache-Control headers or that missing `cacheLife()` alone makes a route run per request | Fails unless rewritten to the documented Cache Components lifetime behavior or backed by production header evidence |
| 20 | `next_cache_tag_invalidation_supported` | Cache-lifetime rec claims existing tag invalidation | Fails unless every claimed `cacheTag()` has matching `revalidateTag()` / `updateTag()` evidence |
| 21 | `cache_rec_not_error_dominated_or_acknowledged` | CDN cache rec targets a route with function 5xx metrics | Fails unless the rec excludes or acknowledges error traffic |
| 22 | `cache_control_header_syntax` | CDN cache rec includes `Cache-Control`, `CDN-Cache-Control`, or `Vercel-CDN-Cache-Control` values | Fails on empty directives such as a trailing comma |
| 23 | `cache_policy_positive_or_no_ready_rec` | Cache candidate emits a ready recommendation | Fails unless it names a positive cache policy; no-store-only belongs in no-change/observation output |
| 24 | `cache_404_long_ttl_safety` | CDN cache rec mentions a 404 or not-found branch | Fails unless the rec keeps the 404/not-found branch uncached, short-lived, or explicitly separate |
| 25 | `immutable_dynamic_route_safety` | Dynamic route rec uses browser `immutable` caching | Fails unless the URL is byte-versioned or the directive is scoped to Vercel's CDN |
| 26 | `auth_guard_parallelization_safety` | Parallelization rec touches private/auth/ownership data | Fails if private data can be fetched before the auth or ownership guard |
| 27 | `parallelization_impact_not_overclaimed` | Parallelization rec promises a helper-sized latency drop | Fails unless helper/span timing was measured |
| 28 | `parallelization_not_cpu_bound_work` | Parallelization rec targets CPU or compile work | Fails unless measured wait/I/O time proves there is independent work to overlap |
| 29 | `runtime_error_cause_supported` | Route-error rec names a runtime exception/root cause | Fails unless runtime logs or stack evidence support the cause |
| 30 | `turbo_build_cache_safety` | Rec enables Turbo build caching | Fails when the package build script has migration side effects or Turbo outputs omit framework build output |

Verifier guards:

- **`snippet_in_wrong_file`**: code snippet found, but in a different file from the cited path → disposition `unsupported` (don't fail the rec; the LLM was close, but the source file claim is wrong).
- **`line-number-as-count`**: "filename:42" matched against a `pattern_count` claim → skip; this is a line-number, not a count.
- **`prose-of-absence`**: "no cache headers" without an explicit grep confirmation → `unsupported`; absence claims require evidence.
- **`pattern_count` for abstracted DB calls**: `db.method()` in a file with DB imports + await helpers but literal count 0 → `unsupported` (import-chain resolution is out of scope).

## Dispositions

Each verified claim resolves to one of four states:

| Disposition | Meaning | Counted toward `passRate`? |
|---|---|---|
| `verified` | Specific mechanical check passed | yes (counts as pass) |
| `failed` | Claim contradicts reality | yes (counts as fail) |
| `unsupported` | Claim can't be checked mechanically (see guards above) | no |
| `unverifiable` | Out of scope (e.g., external API behavior, runtime-only) | no |

`passRate = verified / (verified + failed)`. Unsupported and unverifiable don't count either way.

## Re-gen trigger and accept criteria

After verification:

| Condition | Action |
|---|---|
| `passRate < 1 AND verifiableClaimCount >= 1` | Re-run Step 3.3 (the recommender) with `topFailures` injected as feedback |
| Project-config contradiction, cache-safety failure, or framework-semantic failure | Hard re-run. The customer report holds back the original rec until re-gen fixes it or abstains |
| `passRate = 1` (including no verifiable claims; this is not proof of correctness) | Accept the run, proceed to Step 4 |



Re-gen accept criteria:

- `regenPassRate >= originalPassRate` AND
- Rec count not gutted (regen doesn't drop more than 50% of recs) AND
- Findings still cited (no rec orphaning)

If re-gen makes things worse, keep the original output unless the trigger was hard safety (`project_config_contradiction`, `cache_vary_safety`, or `semantic_safety`). Hard-safety failures must not ship to the customer report.

## Verifier implementation

`scripts/verify-and-regen.mjs` invokes `lib/extract-claims.mjs` and `lib/verify-claim.mjs` in-process for each verifiable claim. Pure functions, no network, no LLM — deterministic.

For `citation_in_library` and `citation_applies_to_version`, the script uses `lib/citations.mjs`'s `isKnownUrl()` and `sanitizeCitations()` helpers (already tested). Other handlers use filesystem/string checks; rg is used for invalidation-file discovery. No AST analysis or runtime verification is performed.