# MU-Bench

[![License: Apache 2.0](https://img.shields.io/badge/Code-Apache%202.0-blue.svg)](LICENSE)
[![License: CC BY-NC 4.0](https://img.shields.io/badge/Data-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)
[![CI](https://github.com/sierra-research/mu-bench/actions/workflows/lint.yml/badge.svg)](https://github.com/sierra-research/mu-bench/actions/workflows/lint.yml)
[![Dataset on HF](https://img.shields.io/badge/🤗-Dataset-yellow.svg)](https://huggingface.co/datasets/sierra-research/mu-bench)

Multilingual Utterances Transcription Benchmark — an open benchmark for evaluating speech-to-text providers across multiple locales and metrics.

**[Leaderboard](https://research.sierra.ai/mubench)** · **[Dataset](https://huggingface.co/datasets/sierra-research/mu-bench)** · **[Blog](https://research.sierra.ai/mubench/#paper)**

## Overview

This benchmark compares transcription providers on real customer service phone conversations recorded at 8kHz mono. Human annotators produce word-level ground truth transcripts for each caller utterance.

The dataset covers 5 locales with 4,270 utterances total:

| Locale | Language | Utterances |
|--------|----------|------------|
| en-US | English (US) | 817 |
| es-MX | Spanish (Mexico) | 792 |
| tr-TR | Turkish | 846 |
| vi-VN | Vietnamese | 975 |
| zh-CN | Chinese (Mandarin) | 840 |

## Getting Started

### 1. Install System Dependencies

```bash
brew install ffmpeg   # macOS
# or: apt-get install ffmpeg   # Linux
```

### 2. Download Audio

The audio is hosted as a gated dataset on HuggingFace. You'll need to:

1. [Request access](https://huggingface.co/datasets/sierra-research/mu-bench) to the `sierra-research/mu-bench` dataset
2. Create a [HuggingFace token](https://huggingface.co/settings/tokens) and set it as an environment variable

```bash
export HF_TOKEN=your_token_here
pip install -e ".[tools]"
python scripts/download_audio.py
```

This exports `.wav` files to `audio/<locale>/`.

### 3. View the Leaderboard Locally

```bash
cd leaderboard
npm install
npm run dev
```

### 4. Explore the Manifest

`manifest.json` contains every utterance in the benchmark with its ground truth transcript:

```json
{
  "id": "conv-0-turn-0",
  "locale": "en-US",
  "conversation_id": "conv-0",
  "turn_index": 0,
  "transcript": "Hi. I'm calling to check the status of my card.",
  "audio_path": "audio/en-US/conv-0-turn-0.wav",
  "duration_sec": 2.246
}
```

## Submitting Results

See **[`submissions/SUBMITTING.md`](submissions/SUBMITTING.md)** for the full submission contract. The short version:

1. **Download the audio** from the [HuggingFace dataset](https://huggingface.co/datasets/sierra-research/mu-bench)
2. **Run your model** on all audio files listed in `manifest.json`
3. **Save transcripts** as `.txt` files — one per utterance, plain text, no headers
4. **Create `metadata.yaml`** with your model info
5. **Create `latency.json`** mapping `"<locale>/<utterance_id>"` to API response time in milliseconds
6. **Validate locally** with `python scoring/validate.py submissions/raw/your-model-name --manifest manifest.json`
7. **Open a pull request** adding your submission under `submissions/raw/`

See `submissions/raw/deepgram-nova3/` for a complete example.

### Scoring

Scoring is run by maintainers — submitters do not need to run it locally. When a maintainer comments `/score` on your PR, the CI pipeline runs automatically:

| Metric | Description | Direction |
|--------|-------------|-----------|
| **WER** | Corpus Word Error Rate (sum of word edits / sum of reference words) after LLM normalization, computed against a canonical per-utterance gold normalized once from `manifest.json` so every provider is scored against the same reference. Silent clips with non-`<unintelligible>` empty gold count any hypothesis words as insertion errors (both numerator and denominator receive the hyp word count). Overall is the unweighted mean of the five per-locale corpus WERs. | Lower is better |
| **UER** | Utterance Error Rate — fraction of utterances with at least one meaning-changing error. Overall is the unweighted mean of the five per-locale UERs (same locale-macro convention as WER). | Lower is better |
| **Latency (p95)** | 95th percentile time-to-complete-transcript per utterance (ms). For batch APIs this is request-to-response round-trip; for streaming APIs it is send-to-final-transcript. Streaming submissions additionally surface TTFT as a `+TTFT` annotation. | Lower is better |

The scoring pipeline is pinned for reproducibility: `scoring/llm.py` targets a specific judge model snapshot at `temperature=0` with a fixed `seed`, and each `scores.json` records the model, seed, and prompt SHAs under a top-level `judge` block.

Results are posted as a comment on your PR and added to the leaderboard on merge.

## Repository Structure

```
manifest.json              # Audio file list + ground truth transcripts
pyproject.toml             # Python deps: base + [tools], [scoring], [transcribe] extras
scripts/
  download_audio.py        # Download audio from HuggingFace (requires [tools])
  transcribe.py            # Runs provider APIs to generate transcripts + latency (requires [transcribe])
  latency_stats.py         # Compute p50/p95 latency from latency.json files
submissions/
  SUBMITTING.md            # Full submission contract
  raw/                     # Raw transcripts + latency.json, one directory per provider
  normalized/              # LLM-normalized transcripts (auto-generated by CI)
scoring/
  validate.py              # Submission format validation (run locally)
  normalize.py             # LLM-based transcript normalization
  score.py                 # Metrics computation (corpus WER, sig. WER)
  metrics.py               # Core metric implementations
  prompts.py               # Gitignored — injected from GitHub secret in CI
results/
  leaderboard.json         # Aggregated leaderboard data
  <provider>/scores.json   # Per-provider score breakdowns (includes latency)
leaderboard/               # React/Vite leaderboard web app
.github/workflows/         # CI for validation, scoring, and deployment
```

### Installing dependencies

The project uses `pyproject.toml` with three optional extras:

```bash
pip install -e .                # base: just the validator (pyyaml)
pip install -e ".[tools]"       # + huggingface-hub, requests (for download_audio.py)
pip install -e ".[scoring]"     # + jiwer, openai (maintainers; needs scoring/prompts.py)
pip install -e ".[transcribe]"  # + provider SDKs (for running scripts/transcribe.py)
```

## Citing MU-Bench

If you use MU-Bench in your research, please cite:

```bibtex
@misc{mubench2026,
  title     = {MU-Bench: A Multilingual Transcription Benchmark from Real Phone Calls},
  author    = {Li, Andrea and Ray, Soham},
  year      = {2026},
  url       = {https://github.com/sierra-research/mu-bench},
  note      = {Dataset: https://huggingface.co/datasets/sierra-research/mu-bench}
}
```

## License

This repository uses a dual-license model:

* **Code** (scripts, scoring, leaderboard, workflows, and all other software) is licensed under the [Apache License 2.0](LICENSE).
* **Data** (audio files and transcripts from Hugging Face and manifest.json) is licensed under the [Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/).