# Interspeech 2026: MSU-Bench *A diagnostic benchmark for evaluating how well Large Audio-Language Models (LALMs) understand **who says what**, and **what happens between speakers**, in real multi-speaker conversations.* [![Demo Page](https://img.shields.io/badge/๐ŸŒ_Demo-Live-00f0ff)](https://aslp-lab.github.io/msu-bench.github.io/) [![Paper](https://img.shields.io/badge/arXiv-2606.22868-b31b1b)](https://arxiv.org/abs/2606.22868) [![Dataset](https://img.shields.io/badge/๐Ÿค—_HuggingFace-Dataset-yellow)](https://huggingface.co/datasets/ASLP-lab/MSU-Benchmark) [![Code License](https://img.shields.io/badge/Code_License-MIT-green)](LICENSE) [![Data License](https://img.shields.io/badge/Data_License-Research_Only-blue)](#license) **Zhaokai Sun**\*, **Shuai Wang**\*, **Zhennan Lin**\*, Chengyou Wang, Dehui Gao, Yuang Cao, Chunjiang He, Pan Zhou, **Lei Xie**\*\* Audio, Speech and Language Processing Group (**ASLP@NPU**), School of Software, **Northwestern Polytechnical University**, China School of Intelligent Science and Technology, **Nanjing University**, China **Shenzhen Loop Area Institute**, China Base Model, **Li Auto**, China \* Equal contribution  ยท  \*\* Corresponding author  ยท  ๐Ÿ“ฎ `zksun@mail.nwpu.edu.cn` MSU-Bench: Understanding Who Said What โ€” and Why โ€” in Multi-Speaker Conversations
--- ## ๐Ÿ“ข News - **2026-07-07**  ยท  QA construction pipeline open-sourced (see [`bench_generation_pipeline/`](bench_generation_pipeline/)). - **2026-07-04**  ยท  Dataset released on ๐Ÿค— [`ASLP-lab/MSU-Benchmark`](https://huggingface.co/datasets/ASLP-lab/MSU-Benchmark). - **2026-06-04**  ยท  MSU-Bench is accepted at **Interspeech 2026**. --- ## Overview Spoken Language Understanding is moving from task-specific pipelines toward **Large Audio-Language Models (LALMs)** that generate natural-language responses over speech. Yet existing speech benchmarks still focus on **single-speaker** audio or isolated subtasks (ASR, speaker verification, emotion recognition, โ€ฆ), leaving **speaker-centric understanding in real multi-speaker conversations** insufficiently evaluated โ€” real conversations involve rapid turn switching, overlaps, interruptions, and speaker-dependent variations in style, emotion, and intent. > **Who is speaking? ยท Whose viewpoint is this? ยท What is the relationship between speakers? ยท How do their emotions and stances evolve across turns?** **MSU-Bench** is a diagnostic benchmark built around these questions โ€” a **5M design** (*multi-tier ยท multi-speaker ยท multilingual ยท multi-scenario ยท multi-task*) organized as a **two-tier** framework of **5 capabilities** and **16 tasks**, with **2,300 human-verified multiple-choice QA instances** sampled from **~731 hours** of Chinese/English telephone, meeting, podcast, and film/TV audio. Diagnostically-designed distractors let us pinpoint *why* a model fails: wrong-speaker attribution, hallucination, or incorrect "unknown" answers. - ๐ŸŒ **Demo:** https://aslp-lab.github.io/msu-bench.github.io/ - ๐Ÿ“„ **Paper (arXiv):** https://arxiv.org/abs/2606.22868 - ๐Ÿค— **Dataset:** https://huggingface.co/datasets/ASLP-lab/MSU-Benchmark - ๐Ÿ’ป **Code:** https://github.com/ASLP-lab/MSU-Bench
Two-tier task hierarchy of MSU-Bench
Figure 1. Two-tier task hierarchy of MSU-Bench, from speaker grounding to multi-speaker reasoning.
--- ## Task Taxonomy MSU-Bench uses a two-tier design that progresses from basic speaker grounding to complex multi-speaker reasoning. It covers **5 capabilities** and **16 representative tasks**.
MSU-Bench capability definitions
Figure 2. Capability definitions and representative tasks in MSU-Bench.
### Tier 1 โ€” Speaker Grounding & Identification Bind speech content, speaker identity, and speaker attributes together โ€” *who is who, who said what, whose attribute is this*. | Capability | Representative Tasks | |---|---| | **Speaker Identification (SID)** | Reverse Speaker Retrieval (RSR) ยท Speaker Retrieval (SR) ยท Speaker-specific Viewpoint Summarization (SVS) ยท Speaker Counting (SC) ยท Speaker Verification (SV) | | **Speaker Attribute Recognition (SAR)** | Accent Identification (AI) ยท Age Recognition (AR) ยท Gender Identification (GI) ยท Emotion Recognition (ER) ยท Speaker Profiling (SP) | ### Tier 2 โ€” Multi-Speaker Dialogue Reasoning Reason about relationships, structure, and context across speakers โ€” *what these utterances mean in the conversation*. | Capability | Representative Tasks | |---|---| | **Dialogue Scene Reasoning (DSR)** | Background Inference (BI) ยท Role/Identity Identification (RII) | | **Dialogue Structure Analysis (DSA)** | Dialogue Act Recognition (DAR) ยท Q&A Structure ID (QASI) | | **Dialogue Contextual Reasoning (DCR)** | Emotion Interaction Reasoning (EIR) ยท Multi-speaker Viewpoint Summarization (MSVS) | ### Speaker-Referencing Schemes To probe robustness to *how* a target speaker is specified โ€” and to disentangle content-understanding errors from speaker-localization errors โ€” each applicable task is instantiated under multiple referencing schemes: | Scheme | Target speaker is specified by | |---|---| | **No Index** | A raw audio snippet of the target speaker (direct acoustic anchor) | | **Time Index** | A time span (e.g. "the person speaking from 10s to 15s") | | **Transcript Index** | A quoted transcript line | | **Speaker Index** | Order of appearance (e.g. "the second speaker") | | **Complex Index** | A combination of time / transcript / speaker cues | This design lets us diagnose whether a failure comes from **content understanding**, **target-speaker localization**, or **higher-order cross-speaker reasoning**. --- ## Data & Construction Pipeline ### Data sources MSU-Bench draws from **eight Chinese/English multi-speaker corpora**, spanning conversational and media-style audio, ~**731 hours in total**: | Domain | Source | Duration | |---|---|---:| | Conversational | [Mandarin Chinese Telephony (MagicHub)](https://magichub.com/datasets/mandarin-chinese-conversational-speech-corpus-telephony/) | 5 h | | Conversational | [English Telephony (MagicHub)](https://magichub.com/datasets/english-conversational-speech-corpus-telephony/) | 5 h | | Conversational | AliMeeting | 12 h | | Conversational | CHiME-6 | 12 h | | Media-style | Wild English Podcast | 66 h | | Media-style | Wild Chinese Podcast | 31 h | | Media-style | Wild Chinese Movie | 400 h | | Media-style | Wild English Movie | 200 h | ### Construction pipeline
MSU-Bench construction pipeline
Figure 3. MSU-Bench construction pipeline: dialogue-quality assessment โ†’ speaker-aware annotation โ†’ speaker-referenced QA generation โ†’ human-in-the-loop quality control.
1. **Dialogue-quality assessment** โ€” Gemini scores candidate segments; only informative and coherent multi-speaker clips are kept. Audio is cut into short (1โ€“2 min) and long (2โ€“5 min) clips. 2. **Speaker-aware annotation** โ€” the Volcano API produces speaker diarization and transcripts; Gemini annotates speaker identity, sound events, and paralinguistic cues. 3. **Speaker-referenced QA generation** โ€” Gemini generates candidate QA from the raw audio + structured annotations + task-specific prompts, across all 16 tasks and 5 referencing schemes. 4. **Human-in-the-loop verification** โ€” eight annotators with audio background verify metadata, revise ambiguous questions, check answer determinacy and format compliance, and remove invalid items. ### Quality assurance | Metric | Tier 1 | Tier 2 | |---|---:|---:| | Initial QA validity (human-judged) | **95%** | **86%** | | Human โ†” ground-truth answer agreement | **98%** | **96%** | The full, reproducible pipeline is open-sourced under [`bench_generation_pipeline/`](bench_generation_pipeline/). --- ## Results We evaluate **9 speech-language models** โ€” 6 open-source (Qwen2.5-Omni, Qwen3-Omni, AudioFlamingo-3, Kimi-Audio, StepAudio2, MiMoAudio) and 3 closed-source Gemini systems (Gemini-2.5-Flash, Gemini-2.5-Pro, Gemini-3-Flash) โ€” under a unified zero-shot A/B/C/D prompting protocol with exact-match accuracy. ### Overall accuracy by tier | Model | Tier 1 (Identification) | Tier 2 (Understanding) | **Avg** | |---|:---:|:---:|:---:| | Qwen2.5-Omni | 0.19 | 0.21 | 0.19 | | AudioFlamingo-3 | 0.40 | 0.38 | 0.39 | | Qwen3-Omni | 0.40 | 0.38 | 0.39 | | Kimi-Audio | 0.41 | 0.47 | 0.43 | | StepAudio2 | 0.44 | 0.46 | 0.44 | | **MiMoAudio** *(best open-source)* | 0.52 | 0.64 | **0.56** | | Gemini-2.5-Flash | 0.64 | 0.77 | 0.69 | | Gemini-2.5-Pro | 0.67 | 0.74 | 0.70 | | **Gemini-3-Flash** *(best overall)* | **0.73** | **0.84** | **0.77** |
Full per-task results
Figure 4. Full per-task results across the 16 MSU-Bench tasks (see paper for details).
### Accuracy under speaker-referencing schemes | Model | Tier 1 (No / Time / Cpx) | Tier 2 (No / Time / Cpx) | |---|:---:|:---:| | Qwen3-Omni | 0.57 / 0.38 / 0.46 | 0.34 / 0.28 / 0.35 | | MiMoAudio | 0.53 / 0.54 / 0.60 | 0.64 / 0.53 / 0.63 | | **Gemini-3-Flash** | **0.71 / 0.64 / 0.84** | **0.83 / 0.76 / 0.92** | ### Diagnostic error-type composition Error rates conditioned on incorrect answers โ€” **WS** = wrong speaker, **HAL** = hallucination, **UNK** = unknown, **INS** = instruction-following failure. | Model | Tier 1 (WS / HAL / UNK / INS) | Tier 2 (WS / HAL / UNK / INS) | |---|:---:|:---:| | Qwen3-Omni | 0.14 / 0.05 / **0.27** / 0.00 | 0.18 / 0.08 / **0.40** / 0.00 | | MiMoAudio | **0.28** / 0.08 / 0.08 / 0.16 | **0.53** / 0.11 / 0.09 / 0.13 | | Gemini-3-Flash | **0.30** / 0.07 / 0.05 / 0.13 | **0.67** / 0.11 / 0.03 / 0.06 | ### Key findings - **Closed-source leads, but nobody solves it.** Gemini-3-Flash reaches **0.77** overall (Tier 1 0.73 / Tier 2 0.84). The best open-source model, MiMoAudio, reaches **0.56**. All models still struggle with complex speaker grounding and multi-speaker reasoning. - **Time Index is the hardest referencing scheme.** Locating a target speaker from a time span remains a bottleneck under fast turn-taking and overlap; multi-cue *Complex Index* helps but exposes weak timeโ†”identity alignment. - **Strong models "blame the wrong speaker".** Weaker models default to *Unknown*, but as capability grows, errors shift toward **wrong-speaker attribution** โ€” for Gemini-3-Flash, wrong-speaker choices account for **0.67** of its Tier-2 errors. This is a subtler and arguably more risky failure mode than answering "unknown". - **General audio ability โ‰  stable speaker-centric understanding.** Omni-style models do not automatically dominate; robust multi-speaker understanding needs dedicated speaker modeling, temporal alignment, and cross-speaker reasoning. --- ## Repository Structure ``` publish-github/ โ”œโ”€โ”€ README.md # this file โ€” project overview โ”œโ”€โ”€ docs/ # ๐ŸŒ interactive demo site (GitHub Pages source) โ”‚ โ”œโ”€โ”€ index.html # single-page demo โ”‚ โ”œโ”€โ”€ data.js / qa_data.js / bench_examples.js โ”‚ โ”œโ”€โ”€ assets/images/ # figures used by the paper & this README โ”‚ โ”œโ”€โ”€ video-example/ # annotated video clips โ”‚ โ”œโ”€โ”€ annotations/ # speaker-segment annotations for the videos โ”‚ โ”œโ”€โ”€ demo_qa/ demo_qa_audio/ # per-video QA + audio snippets โ”‚ โ”œโ”€โ”€ bench_examples/ # 16-task gallery (CN + EN example per task, with audio) โ”‚ โ””โ”€โ”€ .nojekyll โ”œโ”€โ”€ bench_generation_pipeline/ # ๐Ÿ”ง open-sourced QA construction pipeline โ”‚ โ”œโ”€โ”€ 1_annotation_pipeline/ # audio โ†’ speaker/paralinguistic annotation โ”‚ โ”œโ”€โ”€ 2_qa_generation/ # annotation โ†’ QA (per-capability prompts) โ”‚ โ””โ”€โ”€ 3_samples/ # end-to-end CN & EN worked examples โ””โ”€โ”€ bench_evaluation_pipeline/ # ๐Ÿ“Š open-sourced model-evaluation pipeline โ”œโ”€โ”€ step1_prepare_data/ # HF download + CN/EN merge โ†’ language-tagged QA index โ”œโ”€โ”€ step2_run_scoring/ # per-QA scoring (Gemini 2.5 Flash by default; Doubao ref.) โ”œโ”€โ”€ step3_summarize_results/ # per-task / capability / tier / index-scheme / โ”‚ # error-type / language CSVs โ”œโ”€โ”€ configs/ # backend endpoint templates (.env.example) โ””โ”€โ”€ run_all.sh # one-shot: prepare โ†’ score โ†’ summarize ``` --- ## Quick Start ### View the demo online The demo is served via **GitHub Pages** from the `docs/` folder: ๐Ÿ‘‰ **https://aslp-lab.github.io/msu-bench.github.io/** To enable Pages on your own fork: *Settings โ†’ Pages โ†’ Build from branch โ†’ `main` / `docs`*. ### Run the demo locally ```bash cd docs python3 -m http.server 8080 # then open http://localhost:8080 ``` The demo includes: - **Video + annotation explorer** โ€” synchronized speaker segments with metadata (name, gender, age, emotion). - **Interactive QA** โ€” try questions under different speaker-referencing schemes, reveal answers. - **16-task gallery** โ€” one Chinese and one English example per task, each with reference audio and multiple QA variants. ### Reproduce the QA pipeline See [`bench_generation_pipeline/README.md`](bench_generation_pipeline/README.md) for the full audio โ†’ annotation โ†’ QA flow, prompt templates, dependencies, and worked samples. > โš ๏ธ **All API keys are anonymized to `YOUR_*` placeholders โ€” fill in your own before running.** The pipeline uses Volcano ASR (diarization + transcripts) and Gemini (labeling + QA generation). ### Evaluate a model on MSU-Bench See [`bench_evaluation_pipeline/README.md`](bench_evaluation_pipeline/README.md) for the end-to-end evaluation flow. Each stage is a self-contained script: ```bash cd bench_evaluation_pipeline cp configs/gemini.env.example configs/gemini.env # then fill in secrets bash run_all.sh ``` The default backend is **Gemini 2.5 Flash** (any `generateContent`-compatible endpoint works โ€” swap `GEMINI_MODEL` for `gemini-2.5-pro` / `gemini-3-flash-preview` etc.); a Doubao-Seed reference backend is included for completeness. `run_all.sh` will: 1. **Step 1** โ€” pull `ASLP-lab/MSU-Benchmark` from HuggingFace and merge the CN / EN dual-split into a single language-tagged QA index (the two splits share the same audio; only the QA language differs). 2. **Step 2** โ€” call the model on every question, with the paper's speaker-referencing rules (no-index / verification pair / full-audio-only / โ€ฆ) applied automatically. 3. **Step 3** โ€” write six diagnostic CSVs under `work/report/`: - `summary_by_task.csv` โ€” per-task accuracy for the 16 tasks - `summary_by_capability.csv` โ€” per-capability accuracy (SID/SAR/DSR/DSA/DCR) - `summary_by_tier.csv` โ€” Tier-1 / Tier-2 accuracy - `summary_by_language.csv` โ€” CN / EN accuracy per tier - `summary_by_index_scheme.csv` โ€” accuracy & sample-share of each speaker-referencing scheme within each tier (no / time / transcript / speaker / complex) - `summary_by_error_type.csv` โ€” wrong-speaker / hallucination / unknown / instruction-not-follow error composition per tier > Since the outer leaderboard already publishes the aggregate scores, the evaluation pipeline **does not** persist any consolidated leaderboard artefact โ€” the CSVs above are diagnostic-only. --- ## Dataset The benchmark data (audio + annotations + QA) is available on HuggingFace: **https://huggingface.co/datasets/ASLP-lab/MSU-Benchmark** The dataset is published as a **file-tree snapshot** rather than a `datasets`-style tabular split, mirroring the paper's dual-language design: ``` MSU-Benchmark/ โ”œโ”€โ”€ bench_cn/ โ”‚ โ”œโ”€โ”€ QA_cn//QA_short|QA_long//level{1,2}/.json โ”‚ โ”œโ”€โ”€ source_audio//.wav โ”‚ โ”œโ”€โ”€ annotation_json//.json โ”‚ โ””โ”€โ”€ prompts/generation_prompts/level{1,2}/.txt โ””โ”€โ”€ bench_en/ (same structure, English QA over the same audio) ``` Each QA JSON contains a `qa_result` list โ€” one `speaker_meta` entry followed by 4 questions (one per speaker-referencing scheme), each with `question / question_type / options / answer / rationale` fields. `bench_cn` and `bench_en` are two linguistic views of the **same** underlying audio. ```bash # Fetch the full snapshot with huggingface_hub pip install huggingface_hub huggingface-cli download ASLP-lab/MSU-Benchmark --repo-type dataset --local-dir ./MSU-Benchmark ``` See [`bench_evaluation_pipeline/step1_prepare_data/`](bench_evaluation_pipeline/step1_prepare_data/) for a script that flattens this tree into a single language-tagged QA index for evaluation. Because the audio is sourced in part from copyrighted film/TV, telephone, meeting, and podcast material, the dataset is released **for non-commercial academic research use only**. Please refer to the dataset card for full licensing details. --- ## Citation If you find MSU-Bench useful, please cite: ```bibtex @inproceedings{sun2026msubench, title = {{MSU-Bench}: Towards Speaker-Centric Understanding in Conversational Multi-Speaker Scenarios}, author = {Sun, Zhaokai and Wang, Shuai and Lin, Zhennan and Wang, Chengyou and Gao, Dehui and Cao, Yuang and He, Chunjiang and Zhou, Pan and Xie, Lei}, booktitle = {Proc. Interspeech}, year = {2026} } ``` --- ## License - **Code** (demo site + generation pipeline): released under the **MIT License** โ€” see [`LICENSE`](LICENSE). - **Data** (audio, annotations, QA): released **for non-commercial academic research only**, since it is derived from third-party copyrighted media. Do not redistribute the raw media for commercial purposes. > If your intended use differs, please open an issue or contact the authors. --- ## Acknowledgements This research is supported by the **National Natural Science Foundation of China** (Grant No. 62401377). We thank the eight audio-background annotators for their careful human verification of the QA items, and our collaborators at Li Auto for their support. --- ## Contact For questions, collaborations, or dataset access issues, please open a GitHub issue or contact: - Zhaokai Sun โ€” `zksun@mail.nwpu.edu.cn` - Shuai Wang โ€” `shuaiwang@nju.edu.cn` - Lei Xie โ€” `lxie@nwpu.edu.cn` ---
Made by ASLP@NPU ยท Nanjing University ยท Shenzhen Loop Area Institute ยท in collaboration with Li Auto