# RAA Dataset and Evaluation

[English](README.md) | [简体中文](README_zh-CN.md)

RAA (Referent Anchoring Accuracy) evaluates whether a full-duplex speech model can anchor its follow-up response to the item the user actually heard after interrupting a list-style answer.

This repository provides:

- 108 synthetic speech evaluation inputs with interruption metadata;
- Baseline inference and P1 played-audio context injection implementations;
- ASR transcription, RAA Judge, and offline data-integrity validation tools;
- Dataset construction scripts and reproduction instructions.

## Repository Structure

```text
raa_bench/
  data/                         # 108 RAA samples and their metadata
  scripts/                      # Minimal inference and evaluation pipeline
  .env.example                  # External service configuration template
  requirements.txt
```

## Quick Start

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r raa_bench/requirements.txt
cp raa_bench/.env.example raa_bench/.env
```

First, run the data validation, which requires neither network access nor API keys:

```bash
python raa_bench/scripts/validate_open_source_data.py
```

See [raa_bench/README.md](raa_bench/README.md) for the complete end-to-end workflow and [raa_bench/data/README.md](raa_bench/data/README.md) for the data format.

## Dataset Size

The dataset contains 108 samples, combining 12 selected scenarios, three interruption delays (12, 16, and 20 seconds), and three interruption intents (`elaborate`, `next`, and `confirm`). Each sample includes a 16 kHz mono `input.wav` file and an `interrupt.json` metadata file.

## External Services

The end-to-end inference, TTS, ASR, and Judge pipeline uses DashScope/Qwen services by default. Users must obtain their own API keys, cover any usage costs, and comply with the applicable service terms. Offline data validation does not call any external service.

## License

The code and data are licensed under the [Apache License 2.0](LICENSE). Copyright 2026 Taobao.