# Preference-ASR

Evaluation code for **Preference-ASR**, a benchmark test set for evaluating ASR
systems on their ability to follow natural-language *preference instructions*
(numbers, disfluencies, entities, casing), scored with a **preference-aware
normalizer** that selectively skips the normalization step the instruction is
testing.

- 📄 **Paper:** *"Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR
  in the Era of Speech LLMs"* — Proc. Interspeech 2026.
- 📦 **Dataset** (audio + manifest, categories, statistics, and per-corpus
  licenses & attribution): **[nvidia/preference-asr-bench](https://huggingface.co/datasets/nvidia/preference-asr-bench)**.

This repo holds the **preference-aware normalizer** and the audio downloader; see
the paper and the Hugging Face dataset for everything else.

## Preference-aware normalizer

Score the model hypothesis against `preference_text`, passing each manifest row
so the normalizer skips the step the instruction targets:

```python
from normalizer import PreferenceAwareNormalizer

norm = PreferenceAwareNormalizer()
ref = norm.normalize_entry(row["preference_text"], row)
hyp = norm.normalize_entry(model_output, row)
# then accumulate WER over (ref, hyp)
```

The normalizer skips: TN/ITN of numbers/symbols for `normalization`, filler
removal for `disfluencies` (keep), and lowercasing for `case` — so WER reflects
whether the model obeyed the instruction. `entities` and `standard` use standard
normalization.

## Getting the audio

The manifest (`data/preference_asr_testset.jsonl`, 2,822 rows) ships here; fetch
the audio for the clips it references from their public sources:

```bash
pip install "datasets>=2.18" huggingface_hub soundfile librosa numpy tqdm
python download_audio.py                              # all corpora
python download_audio.py --datasets librispeech ami   # a subset
```

Clips are written to `data/<corpus>/` at the relative paths the manifest expects,
so run the downloader and your evaluation from the repo root.

## Citation

> N. R. Koluguri, S. Meister, N. Karpov, P. Zelasko, D. Raj, J. Balam, and
> B. Ginsburg, "Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR
> in the Era of Speech LLMs," *Proc. Interspeech 2026*.