# Methodology

See [Semantic WER](semantic-wer.md) for the draft library scorer and its
explicit non-activation boundary.

A leaderboard number on [benchmarks.coval.ai](https://benchmarks.coval.ai) is a
function of four pinned inputs. Reproducing a number requires reproducing all
four. This document describes each one and where it lives.

## 1. Dataset

**STT benchmark.** Two frozen datasets run each cycle, each as its own run:

- `stt-v1` — a 50-utterance subset of
  [LibriSpeech `test-clean`](https://www.openslr.org/12/) (`CC-BY-4.0`):
  studio-quality read English speech, speaker-balanced per ADR-020. The easy
  tier: most providers train on LibriSpeech, so absolute WER runs low.
- `stt-v3` — 897 conversational clips from
  [pipecat-ai/stt-benchmark-data](https://huggingface.co/datasets/pipecat-ai/stt-benchmark-data):
  spontaneous voice-agent speech (fragments, fillers, prompted turns) with
  model-generated reference transcripts. The hard tier.

Each clip is loudness-normalized (RMS target −20 dBFS, peak-guarded) and
selection is deterministic from the frozen sources; per-dataset selection
rules, provenance, and licenses are documented in
`runner/src/coval_bench/datasets/manifests/README.md`. Runs recorded between
2026-07-08 and 2026-07-13 used `stt-v2` (WildASR `fleurs_clean_en`), and runs
before that used `stt-v1` alone.

**TTS benchmark.** Text prompts from the manifest named by `TTS_DATASET_ID`
(default `tts-v1`, a 30-prompt set of short inputs). Source and selection rules
are documented in `runner/src/coval_bench/datasets/manifests/README.md`.

**Per-run sampling.** Each scheduled run draws a random sample of items from
its manifest before any provider is called. Shared executions run every 30
minutes, one per dataset, and take `DATASET_ID` and `DATASET_SAMPLE_SIZE`
from their scheduler trigger; the TTS execution reads `TTS_DATASET_ID` and
`TTS_DATASET_SAMPLE_SIZE`, the latter falling back to `DATASET_SAMPLE_SIZE` and
then to the per-dataset size in `suite.py` (`tts-v2` runs 2 items, other TTS
manifests 10). The dedicated job runs once a day as a single
execution that walks the suite in `runner/src/coval_bench/datasets/suite.py`,
which lists its datasets and the sample size for each. The sample is drawn
once at the start of the run and shared across every model, so all models are
scored on the identical subset within a run (parity); the draw is independent
across runs, so the full manifest is covered over time. The manifests still
carry their complete item sets — sampling only controls how many run each
cycle. A size ≥ the manifest length runs everything.

**SHA pinning.** Every audio file referenced by an STT manifest carries a
`sha256` field. The runner verifies the SHA after fetching from GCS and raises
`DatasetIntegrityError` on mismatch (see `runner/src/coval_bench/datasets/loader.py`).
TTS items are text-only and have no SHA.

**Versioning.** Dataset manifests live at
`runner/src/coval_bench/datasets/manifests/{stt-v1,stt-v3,tts-v1}.json` and
carry a `version` field. Bumping a dataset re-pins by minting a new manifest;
retired manifests (`stt-v2.json`) stay for historical reproducibility.

**Aggregation across datasets.** Each dataset runs as its own run (one run
row per dataset, all on the execution's tick), and a result's dataset is
derived from its parent run. The aggregation layer pools every result in the window regardless of
dataset, so headline stats (e.g. pooled WER) blend all datasets that ran.
`/v2/results` exposes each evaluation's `dataset_id` and takes a `dataset`
filter for per-dataset inspection. It supports metric version selection and
cursor pagination for successful default-variant evaluations. See ADR-023.

The [results API v2](raw-results-api.md) exposes metric evaluations and
their observation's dataset ID, with optional metric components and pagination.

**Rebuilding from scratch.** `coval-build-dataset` applies each dataset's
selection rule, transcodes to the canonical audio format, uploads to GCS, and
writes the manifest with fresh SHAs. See
`runner/src/coval_bench/datasets/manifests/README.md` for per-dataset commands
and exact flags.

**Selection rationale.** See [ADR-020](#adr-references) below for the `stt-v1`
rule, ADR-023 for the multi-dataset mix (ADR-022 for the retired `stt-v2`).

## 2. Provider model versions

Each `(provider, model, voice)` tuple the runner exercises is declared in
`runner/src/coval_bench/runner/config.py` (`DEFAULT_STT_MATRIX` and
`DEFAULT_TTS_MATRIX`). Model identifiers are exact provider strings — not
aliases — wherever the provider exposes a versioned identifier (e.g.
`nova-3`, `flux-general-en`, `gpt-4o-mini-tts`, `aura-2-thalia-en`, `mistv3`).

**Server-side aliases.** Where a provider resolves a name server-side (e.g.
Rime `arcana` → Arcana v3), the resolution is documented inline in
`config.py` next to the matrix entry.

**Disabled entries.** Some entries are intentionally inert (`disabled=True`).
The runner skips them; the API exposes them so that a frontend can grey out
models that are catalogued but not currently producing data. See
[ADR-011](#adr-references) for the policy.

## 3. Normalization pipeline

WER is computed after a deterministic normalization applied to both the
reference transcript and the provider hypothesis: `whisper_normalizer`'s
`EnglishTextNormalizer`, the de facto standard for published WER, wrapped by
`normalize_text` in `runner/src/coval_bench/metrics/wer.py`. It folds spoken
and written forms of numbers, ordinals, dates, currency, and percentages into
a canonical form, expands contractions, drops filler words, maps British
spellings to American, and strips diacritics and punctuation.

**Version constant.** `wer.NORM_VERSION` (currently `"2"`) is bumped on any
behavioural change to the pipeline. The constant is included on every
`WERResult` as `norm_version` while the metric is computed, but is not
persisted with the result row.

**Methodology change (2026-07).** Version `"1"` was a hand-rolled pipeline
that corrupted many number forms (e.g. "thirty six" → `3006`, reported in
[issue #218](https://github.com/coval-ai/benchmarks/issues/218)), putting a
WER floor under providers whose inverse text normalization emits digits while
leaving spelled-out-number providers unaffected. It was replaced wholesale
with `EnglishTextNormalizer` per [ADR-021](#adr-references). WER values
before and after the change are **not comparable**. Because result rows do not
carry a normalization-version marker, historical rows cannot be reliably
segmented by normalization pipeline version.

### Aggregation

The headline WER for a model is corpus-level: the sum of substitutions,
deletions, and insertions across every clip in the window, divided by the sum
of reference words. Each per-clip row carries those four counts, so the figure
is exact rather than an average of ratios. The per-clip distribution (median,
percentiles, spread) is still reported from the per-clip ratios.

The two disagree by design. Reference clips run from 4 to 41 words, so a mean
of per-clip ratios lets one miss on a 4-word clip weigh the same as a perfect
41-word clip, and the empty-reference convention above turns a clip with no
reference into a raw insertion count. Pooling weights every word equally and
absorbs both.

The substitution / insertion / deletion split shown beside the headline is
derived from the same alignment counts: each type's summed count over summed
reference words, so the three reconcile to the total exactly.

**Methodology change (2026-11).** Until this point each clip also stored the
split as percentage points from a separate hand-rolled alignment, and windows
averaged those points. When several alignments tie on edit distance the two
alignments can attribute the same total to different error types, so the
per-type numbers before and after are not strictly comparable even though
the totals are. See [ADR-025](#adr-references).

**Methodology change (2026-09).** WER was previously the mean of per-clip
ratios. Counts are recorded from this point forward and were not backfilled;
until every clip in the 30-day window carried counts (2026-10-09) a window
fell back to the mean. Since then the pooled ratio is the only computation:
a WER row without counts is excluded from every window, bucket, and timeline
point, so no WER aggregate or custom timeline range reaches back before
2026-09-09. Headline numbers before and after are **not comparable**. See
[ADR-024](#adr-references).

## 4. Latency metrics (TTFA)

The TTS latency metric is **TTFA (Time-To-First-Audio)**, reported in
milliseconds. It is *perceived* first-audible latency, the goal being to measure 
what an enqueue-and-play client actually waits before it hears sound:

    TTFA = (first audio chunk arrival − synthesis start)
           + leading silence inside the stream before the first audible sample

The arrival term is wall-clock (`time.monotonic`) from the synthesis trigger to
the first non-empty audio chunk. The leading-silence term is computed from the
assembled PCM by `runner/src/coval_bench/metrics/ttfa.py`
(`first_audible_offset_ms`, an RMS-threshold onset detector). Both terms are
combined in one place — `providers/tts/_common.py:finalize_tts_result` — which
every provider routes through. The offset is best-effort: if it cannot be
computed, TTFA degrades to arrival-only and the audio is still kept.

**Methodology change (2026-05).** TTFA previously measured *network arrival
only* (time to first chunk) and ignored any leading silence a provider
front-loads into the stream. As of this change it is the perceived value above.
This shifts every provider's reported TTFA upward by its leading-silence offset
(near-zero for providers that emit audible audio immediately; several hundred ms
for those that front-load silence), so numbers recorded before the change are
**not comparable** with later ones. There is no schema or metric-name change —
the existing `TTFA` metric simply carries the perceived definition from this
point forward. Historical rows before the change are not source-versioned, so
comparisons must exclude data from before the methodology change.

## 5. Library versions

The DP edit-distance computation is delegated to `jiwer` and text
normalization to `whisper-normalizer`. Both are pinned in `runner/uv.lock`
(currently `jiwer == 4.0.0`, `whisper-normalizer == 0.1.12`). `numpy` is also
locked. Every `uv.lock` change is reviewed against the SPDX license-set
policy before merge.

## 5. Latency measurement convention

TTFT (STT) and TTFA (TTS) measure model inference time. Pre-t0 deployment
setup — TCP, TLS, protocol handshake, optional session-setup RTT — is
excluded for every provider. The rule is applied uniformly across cohorts;
the magnitude of what each cohort excludes varies by deployment architecture,
but the rule is the same.

| Cohort | What is excluded from t0 |
|---|---|
| WS streaming (Deepgram, AssemblyAI, ElevenLabs, Speechmatics, Gradium STT, Cartesia, Together AI, Nari STT, StepFun STT, Cloudflare nova-3, Deepgram aura-2, Rime, Gradium TTS, Hume, StepFun TTS) | TLS + WS upgrade + optional session-setup RTT (~50–200 ms). Handshake naturally completes inside `measure_ttft` / `synthesize` before t0. |
| HTTP TTS (Nari `qwen3-tts-fast`, Cloudflare `aura-2-en`) | TLS + TCP via a shared `httpx2.AsyncClient` pre-warmed once per run (~80–200 ms). |

HTTP-pool warming lives in `runner/src/coval_bench/providers/_http_session.py`.
Providers opt in by overriding `Provider.warmup()` in `providers/base.py`; the
orchestrator invokes warmup on every enabled provider class before the
dataset loop runs and tears down the pool in the run's `finally` block. The
non-persisting probe (`coval-bench probe`) warms the same way, so its numbers
are comparable to a scheduled run's.

Dedicated endpoints warm differently: rather than a connection, what needs
warming is the replica behind it. Baseten scales these deployments on demand,
so the warmup request is also the scale-up trigger. The STT provider streams a
one-second synthetic clip to every configured endpoint and the TTS provider
synthesizes (then deletes) a throwaway phrase, each connecting with a
six-minute handshake budget so a scale-from-zero boot (~3-4 minutes observed)
completes before t0. Measurement traffic keeps the tighter 45-second handshake
cap as a degraded-endpoint tripwire.

### HTTP/2 for the HTTP TTS cohort

The shared pool uses HTTP/2 so that every concurrent request to a host
multiplexes over a single connection. The connection is opened once by
`warmup()`, and the dataset loop — run at a concurrency of 8, with each model
serving one request at a time — reuses it, so TCP+TLS is paid once and
excluded from every TTFA row rather than just the first. `http2=True` and the pool limits are set on `TimedTransport`, not on
`AsyncClient`: httpx2 ignores both when a custom transport is supplied. Every
host in the cohort negotiates HTTP/2; under 8-way concurrency the pool
stays at one socket per host. Requires the `h2` package
(`httpx2[http2]`) — a build without it raises at client construction rather
than downgrading silently.

The protocol is chosen by TLS ALPN at connect time. It can negotiate HTTP/1.1
instead — a TLS-terminating proxy or middlebox, an `HTTPS_PROXY` tunnel, or an
HTTP/1.1-only endpoint. On HTTP/1.1 there is no multiplexing, so concurrent
requests open additional connections that `warmup()` did not warm, and TTFA
reabsorbs TCP+TLS for those rows. `warmup()` logs `http_version` and emits a
`*_prewarm_no_http2` warning on any non-h2 connection, and every HTTP TTS
result row records the negotiated `http_version`, so HTTP/1.1 rows can be
filtered out of analysis rather than silently inflating the published TTFA.

**Follow-ups, not yet in this implementation:**

- *NVCF gRPC hosted cohort (Nvidia Nemotron, Magpie).* These providers ship a
  per-channel warmup probe that excludes TLS + gRPC channel + NVCF GPU
  dispatch (~500–1000 ms) before t0. The probe uses the same
  `Provider.warmup()` hook documented above and will be wired in when
  `nvidia_hosted.py` lands on `main`; the cohort table will gain a third row
  then.
- *Connect-time isolation at the transport layer.* `TimedTransport` records
  `submit-to-headers` time per request, surfaced on each HTTP TTS result row
  as `submit_to_headers_ms` — small and stable values confirm the pool stayed
  warm, while a spike flags a row whose connection reconnected mid-run and
  reabsorbed connect time. It does not isolate TCP + TLS time from server
  processing time. True per-call isolation would require subclassing
  `httpcore.AsyncNetworkBackend` (one layer below httpx's transport).
  Deferred — pool reuse achieves the same end result by construction, and the
  recorded interval is sufficient to detect and filter pool eviction.

### Metric exclusions

A metric is withheld for a model when the provider's wire protocol makes the
number measure something other than the model. It applies per
`(provider, model, metric)`, so a model excluded from one metric still publishes
the others. The orchestrator does not write excluded rows and the API hides
historical ones; the list and its reasons live in
`runner/src/coval_bench/registries/metrics.py`.

The recurring case is an emission cadence. TTFT assumes a provider sends a
partial when it has one; a provider that emits on a fixed interval instead
reports the wait for its next tick. That number can't go below the interval
however fast the model is, and it's a step function rather than a latency
distribution, so its median and tail don't mean what they do elsewhere.
Withholding it is more honest than scoring it. The signature is a first-token
time that barely moves across clips with different audio.

## Rerunning the current methodology

To produce a fresh `(provider, model, voice, metric)` measurement:

1. Clone the current repository.
2. `uv sync` inside `runner/`.
3. Set the relevant provider API key in `.env`.
4. `coval-bench run --kind stt --only <provider>/<model>`
   (or analogous for TTS). The runner downloads the same dataset audio from
   GCS, verifies the SHA, runs against the same model identifier, and applies
   the same normalization pipeline.

The result is comparable to current measurements, but it does not reproduce a
leaderboard cell: leaderboard values aggregate multiple runs over a time
window, and each runner invocation draws a new random dataset sample. Runs
created by current runner versions also do not record a source commit, so
exact source reproduction for those rows is unsupported.

## ADR references

- ADR-011 — Provider scope and the `disabled` flag policy.
- ADR-020 — Dataset selection rule (round-robin by speaker, 2.0–15.0 s window).
- ADR-021 — WER text normalization delegated to `whisper_normalizer`'s
  `EnglishTextNormalizer`.
- ADR-022 — STT dataset swap to WildASR `fleurs_clean_en` (`stt-v2`); LibriSpeech
  `test-clean` (`stt-v1`) retained for historical reproducibility.
- ADR-023 — Multi-dataset benchmarking: one run per dataset, dataset identity
  derived from the parent run (no dataset column on `results`); headline
  aggregates pool across datasets.
- ADR-024 — Corpus-level WER as the headline: per-clip S/I/D and reference-word
  counts persisted as WER components; aggregates report sum(errors) /
  sum(reference words); no backfill, and the interim per-clip mean fallback
  was removed once every window was counted (2026-11).
- ADR-025 — One alignment for WER: jiwer's S/I/D counts are the only stored
  split; the percent-point components from the hand-rolled backtrace are no
  longer written (the v1 contract keeps them optional so stored rows validate),
  and the displayed split is derived from the counts.

ADR rationale is referenced inline in the relevant source files and READMEs
where the decision context is load-bearing.

## Caveats

- The STT corpora are public; LibriSpeech in particular is heavily represented
  in provider training data, which skews `stt-v1` absolute WER low. The
  `stt-v3` reference transcripts are model-generated rather than
  human-verified, so its absolute WER carries a reference-error floor shared
  by all providers. Latency and per-provider drift over time are the more
  informative signals.
- This is a research benchmark. Results reflect a specific dataset and
  methodology and may not generalize to production workloads. See the
  Apache-2.0 `LICENSE` for the warranty disclaimer.