# 🧑‍⚖️🎧 ParaPairAudioBench

> **PARAPAIRAUDIOBENCH: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge**

> Jisu Jeon, Seungyeon Jwa, Joosung Lee, Jinhyeon Kim, Woojin Chung, Hwiyeol Jo, Jeonghoon Kim, Jonghyun Choi, Soyoon Kim

> *Interspeech 2026* 

> 📄 Paper: coming soon on arXiv

Prior LALM-as-a-Judge approaches focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. **ParaPairAudioBench** provides **5,175 pairwise audio samples** across five paralinguistic dimensions to enable multi-dimensional, calibration-aware evaluation of LALM judges — revealing a **~20%p gap behind human performance** and severe failures in Tie abstention.

---

## 📊 Dataset Overview

<p align="center">
  <img src="images/dataset_visual.png" width="600" alt="Dataset structure across Attribute, Tie condition, and Transcript axes"/>
</p>

The benchmark is organized along three independent axes:
- **Attribute** — which paralinguistic property is being judged (5 criteria)
- **Tie condition** — whether a clear winner exists (Non-tie) or both samples are equally good/bad (Tie)
- **Transcript** — whether the two audio clips share the same text (acoustic-only focus) or differ (lexical robustness)

### Sample Counts

| Criterion | Total | Non-tie | Tie | Same Transcript | Other Transcript | Source |
|-----------|------:|--------:|----:|----------------:|-----------------:|--------|
| Age | 1,500 | 750 | 750 | — | — | SVC |
| Gender | 1,000 | 500 | 500 | — | — | SVC + LibriTTS |
| Speech Rate | 375 | 375 | 0 | 375 | 0 | EARS |
| Emphasis | 860 | 415 | 445 | 445 | 415 | Expresso |
| Style | 1,440 | 720 | 720 | 362 | 1,078 | Expresso |
| **Total** | **5,175** | **2,760** | **2,415** | | | |

**Non-tie 53.3% / Tie 46.7%** — balanced to test both discrimination ability and hallucination avoidance.

### Evaluation Criteria

| Criterion | Description | Labels | Source |
|-----------|-------------|--------|--------|
| 🧓 Age | Discrimination of speaker age groups | Child · Teen · Adult · MiddleAged · Elderly | [SVC](https://github.com/sonos/svc-demographic-bias-assessment) |
| 🚻 Gender | Consistency in identifying speaker gender | Male · Female | [SVC](https://github.com/sonos/svc-demographic-bias-assessment) + [LibriTTS](https://www.openslr.org/60/) |
| ⏱️ Speech Rate | Relative comparison of speaking speed | slower · faster | [EARS](https://github.com/facebookresearch/ears_dataset) |
| 🔊 Emphasis | Identification of lexical stress and focal prominence | word-level | [Expresso](https://github.com/facebookresearch/textlesslib/tree/main/examples/expresso/dataset) |
| 🎭 Style | Expressiveness of emotions and speaking styles | confused · default · enunciated · happy · laughing · sad · whisper | [Expresso](https://github.com/facebookresearch/textlesslib/tree/main/examples/expresso/dataset) |

---

## 🏆 Benchmark Results

### Overall Performance

<p align="center">
  <img src="images/overall_performance.png" width="720" alt="Per-criterion accuracy of evaluated LALMs vs human and chance level"/>
</p>

Accuracy (%) per criterion for five evaluated LALMs, compared against human performance and chance level. The dashed line marks the target zone above which a model is considered a reliable judge.

| Model | Style | Rate | Emph. | Age | Gender | Avg. |
|-------|------:|-----:|------:|----:|-------:|-----:|
| Human ‡ | 85.7 | 91.0 | 85.7 | 52.7 | 80.7 | 79.2 |
| Gemini 2.5 Flash | **48.5** | **88.9** | **49.7** | **56.5** | **64.2** | **61.5** |
| GPT-4o Audio | 36.4 | 77.6 | 43.8 | 34.9 | 39.3 | 46.4 |
| Kimi-Audio-7B | 45.9 | 76.0 | 42.9 | 27.5 | 58.6 | 50.2 |
| Qwen2.5-Omni-7B | 35.8 | 61.9 | 36.7 | 38.1 | 47.4 | 44.0 |
| SpeechJudge-7B | 32.6 | 48.0 | 32.9 | 25.8 | 29.9 | 33.8 |

‡ Human: n=50 per criterion. 

**Bold**: best model per criterion. Above/below chance assessed via z-test (95% CI).

## Position Bias & Consistency

<p align="center">
  <img src="images/bias_consistency.png" width="720" alt="Position bias (Acc@A vs Acc@B) and consistency analysis per model"/>
</p>

**(a) Position Bias** — gap between Acc@A and Acc@B reveals how much a model's judgment is influenced by audio order rather than content.

**(b) Consistency** — measures whether a model gives the same answer when A and B are swapped. *Consist.* is the raw swap-agreement rate; *Consistent Acc* is the accuracy restricted to only those consistent predictions.

---

## 🚀 Getting Started

### Requirements

- Python 3.9+
- No additional packages required (standard library only)

### Included Benchmarks

The **rate**, **emphasis**, and **style** benchmarks are ready to use:

```
data/benchmarks/pairwise_rate.json
data/benchmarks/pairwise_emphasis.json
data/benchmarks/pairwise_style.json
```

Each file also has a `_swap` variant (A/B positions swapped) for position-bias evaluation.

### Reproducing Age & Gender Benchmarks

The **age** and **gender** benchmarks require the SVC dataset, which is access-controlled.

#### Step 1 — Clone the SVC repository

```bash
git clone https://github.com/sonos/svc-demographic-bias-assessment.git
```

#### Step 2 — Download the SVC dataset

> ⏳ **Note:** Access is manually reviewed and may take **several days**. Submit the request before proceeding with other steps.

Fill in the access request form linked in the SVC repository README. Upon approval you will receive a temporary download URL. Place the downloaded data so the structure matches:

```
svc-demographic-bias-assessment/
└── data/
    └── svc_dataset/
        ├── metadata/
        │   └── test_metadata.json
        └── audios/
            └── test/
                └── audio_<id>.wav
```

#### Step 3 — Build age benchmark

```bash
python scripts/build_age.py --svc_root ./svc-demographic-bias-assessment
```

Output → `output/pairwise_age.json` + `output/pairwise_age_swap.json`

#### Step 4 — Build gender benchmark

```bash
python scripts/build_gender.py --svc_root ./svc-demographic-bias-assessment
```

Output → `output/pairwise_gender.json` + `output/pairwise_gender_swap.json`

> The gender benchmark also incorporates 147 LibriTTS-R pairs already bundled in `data/sources/libritts_gender_pairs.json`.
> If LibriTTS audio is stored elsewhere, pass `--libritts_root /path/to/libritts`.

---

## 📁 Repository Structure

```
ParaPairAudioBench/
├── scripts/
│   ├── build_age.py             ← builds pairwise_age.json from SVC
│   └── build_gender.py          ← builds pairwise_gender.json from SVC + LibriTTS
├── data/
│   ├── benchmarks/              ← ready-to-use (rate, emphasis, style)
│   │   ├── pairwise_rate.json + swap
│   │   ├── pairwise_emphasis.json + swap
│   │   └── pairwise_style.json + swap
│   └── sources/                 ← build-time source IDs (no audio)
│       ├── age_pair_ids.json
│       ├── svc_gender_pair_ids.json
│       └── libritts_gender_pairs.json
├── images/
│   ├── dataset_visual.png
│   ├── overall_performance.png
│   └── bias_consistency.png
└── output/                      ← generated after running build scripts
    ├── pairwise_age.json + swap
    └── pairwise_gender.json + swap
```

---

## 📜 JSON Format

All benchmark files share the same schema:

```json
{
  "id": "pairwise_age_v8_00001",
  "audio_a": "svc-demographic-bias-assessment/data/svc_dataset/audios/test/audio_XXXX.wav",
  "audio_b": "svc-demographic-bias-assessment/data/svc_dataset/audios/test/audio_YYYY.wav",
  "answer": "[[Tie]]",
  "difficulty": "tie",
  "target_label": "Elderly",
  "meta_info": {
    "task": "age",
    "is_tie": true,
    "transcript_match": false,
    "audio_a_transcript": "...",
    "audio_b_transcript": "...",
    "task_specific": { ... }
  }
}
```

- **`answer`**: `[[A]]`, `[[B]]`, or `[[Tie]]`
- **`difficulty`**: `tie` / `easy` for age; `tie` / `non-tie` for gender; criterion-specific for others
- **`target_label`**: the demographic or paralinguistic attribute value to identify
- Audio paths are relative to the directory from which evaluation is run

---

## 🔗 Third-party Datasets

| Dataset | Criteria | License |
|---------|----------|---------|
| [SVC Bias Assessment](https://github.com/sonos/svc-demographic-bias-assessment) | Age, Gender | Non-commercial Academic Research |
| [LibriTTS](https://www.openslr.org/60/) | Gender | CC BY 4.0 |
| [EARS](https://github.com/facebookresearch/ears_dataset) | Speech Rate | CC BY-NC 4.0 |
| [Expresso](https://github.com/facebookresearch/textlesslib/tree/main/examples/expresso/dataset) | Emphasis, Style | CC BY-NC 4.0 |

---

## 📚 Citation

If you use this benchmark, please cite:

```bibtex
@inproceedings{jeon2026parapairaudiobench,
  title     = {{PARAPAIRAUDIOBENCH}: Paralinguistic Pairwise Audio Benchmark for {LALM}-as-a-Judge},
  author    = {Jeon, Jisu and Jwa, Seungyeon and Lee, Joosung and Kim, Jinhyeon and
               Chung, Woojin and Jo, Hwiyeol and Kim, Jeonghoon and
               Choi, Jonghyun and Kim, Soyoon},
  booktitle = {Proceedings of Interspeech},
  year      = {2026},
  note      = {to appear}
}
```

```bibtex
@inproceedings{svc-demographic-bias-assessment,
  title     = {Sonos Voice Control Bias Assessment Dataset: A Methodology
               for Demographic Bias Assessment in Voice Assistants},
  author    = {Chlo{\'e} Sekkat and Fanny Leroy and Salima Mdhaffar and
               Blake Perry Smith and Yannick Est{\`e}ve and
               Joseph Dureau and Alice Coucke},
  booktitle = {Proceedings of LREC-COLING},
  year      = {2024}
}
```