📢：**Good news! 8000 hours of multi-label Wu dialect data are also available at [⭐WenetSpeech-Wu⭐](https://github.com/ASLP-lab/WenetSpeech-Wu-Repo).** 📢：**Good news! 10,000 hours of multi-label Chuan-Yu speech data are also available at [⭐WenetSpeech-Chuan⭐](https://github.com/ASLP-lab/WenetSpeech-Chuan).** # WenetSpeech-Yue: A Large-scale Cantonese Speech Corpus with Multi-dimensional Annotation

Longhao Li¹*, Zhao Guo¹*, Hongjie Chen², Yuhang Dai¹, Ziyu Zhang¹, Hongfei Xue¹, Tianlun Zuo¹, Chengyou Wang¹, Shuiyuan Wang¹, Xin Xu³, Hui Bu³, Jie Li², Jian Kang², Binbin Zhang⁴, Ruibin Yuan⁵, Ziya Zhou⁵, Wei Xue⁵, Lei Xie¹

¹ Audio, Speech and Language Processing Group (ASLP@NPU), Northwestern Polytechnical University
² Institute of Artificial Intelligence (TeleAI), China Telecom
³ Beijing AISHELL Technology Co., Ltd.
⁴ WeNet Open Source Community
⁵ Hong Kong University of Science and Technology

📑 Paper | 🐙 GitHub | 🤗 HuggingFace
🖥️ HuggingFace Space | 🎤 Demo Page | 💬 Contact Us

This is the official repository 👑 for the WenetSpeech-Yue dataset and the source code for WenetSpeech-Pipe speech data preprocessing pipeline.

## 📢 News and Updates - **2026/01/07**: 🛠️ We fixed several issues in the metadata. Please use the latest metadata for training. - **2025/11/15**: 🚀 We released **Llasa-1B-Yue-Updated**！ You can download the model weights from [WSYue-TTS](https://huggingface.co/ASLP-lab/WSYue-TTS) - **2025/09/08**: 🎉 The WenetSpeechYue dataset, featuring over 21,800 hours of Cantonese speech, is now available! ## Download * The WenetSpeech-Yue dataset is available at [WenetSpeech-Yue](https://huggingface.co/datasets/ASLP-lab/WenetSpeech-Yue). * The WSYue-eval benchmark is available at [WSYue-ASR-eval](https://huggingface.co/datasets/ASLP-lab/WSYue-ASR-eval) for ASR and [WSYue-TTS-eval](https://huggingface.co/datasets/ASLP-lab/WSYue-ASR-eval) for TTS. * The ASR models are available at [WSYue-ASR](https://huggingface.co/ASLP-lab/WSYue-ASR). * The TTS models are available at [WSYue-TTS](https://huggingface.co/ASLP-lab/WSYue-TTS). ## Dataset ### WenetSpeech-Yue Overview * Contains 21,800 hours of large-scale Cantonese speech corpus with rich annotations, the largest open-source resource for Cantonese speech research. * Stores metadata in a single JSON file, including audio path, duration, text confidence, speaker identity, SNR, DNSMOS, age, gender, and character-level timestamps. Additional metadata tags may be added in the future. * Covers ten domains: Storytelling, Entertainment, Drama, Culture, Vlog, Commentary, Education, Podcast, News, and Others.

## Benchmark To address the unique linguistic characteristics of Cantonese, we propose WSYue-eval, a comprehensive benchmark encompassing both Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) tasks. ### ASR Benchmark We introduce WSYue-ASR-eval, a test set developed for Automatic Speech Recognition (ASR) as a key task in speech understanding. It features **multi-round manual annotations** including text transcripts, emotion, age, and gender labels. The set is divided into Short and Long subsets by audio duration to enable comprehensive evaluation across speech lengths. WSYue-ASR-eval also covers diverse real-world Cantonese scenarios, including code-switching and multi-domain conditions. | Set | Duration | Speakers | Hours | |-------|----------|----------|-------| | Short | 0–10s | 2861 | 9.46 | | Long | 10–30s | 838 | 1.97 | ### TTS Benchmark We introduce WSYue-TTS-eval, a zero-shot Cantonese TTS benchmark with two subsets: - Base: Contains 1,000 samples from Common Voice for evaluating real-world performance. - Coverage: Combines manually curated and LLM-generated texts spanning multiple domains (e.g., daily life, news, entertainment, poetry) and incorporates diverse linguistic phenomena including polyphonic characters, tone sandhi, code-switching, proper nouns, and numerals. ## ASR Leaderboard

Model	#Params (M)	In-House		Open-Source					WSYue-eval
Model	#Params (M)	Dialogue	Reading	yue	HK	MDCC	Daily_Use	Commands	Short	Long
w/o LLM
Conformer-Yue⭐	130	16.57	7.82	7.72	11.42	5.73	5.73	8.97	5.05	8.89
Paraformer	220	83.22	51.97	70.16	68.49	47.67	79.31	69.32	73.64	89.00
SenseVoice-small	234	21.08	6.52	8.05	7.34	6.34	5.74	6.65	6.69	9.95
SenseVoice-s-Yue⭐	234	19.19	6.71	6.87	8.68	5.43	5.24	6.93	5.23	8.63
Dolphin-small	372	59.20	7.38	39.69	51.29	26.39	7.21	9.68	32.32	58.20
TeleASR	700	37.18	7.27	7.02	7.88	6.25	8.02	5.98	6.23	11.33
Whisper-medium	769	75.50	68.69	59.44	62.50	62.31	64.41	80.41	80.82	50.96
Whisper-m-Yue⭐	769	18.69	6.86	6.86	11.03	5.49	4.70	8.51	5.05	8.05
FireRedASR-AED-L	1100	73.70	18.72	43.93	43.33	34.53	48.05	49.99	55.37	50.26
Whisper-large-v3	1550	45.09	15.46	12.85	16.36	14.63	17.84	20.70	12.95	26.86
w/ LLM
Qwen2.5-Omni-3B	3000	72.01	7.49	12.59	11.75	38.91	10.59	25.78	67.95	88.46
Kimi-Audio	7000	68.65	24.34	40.90	38.72	30.72	44.29	45.54	50.86	33.49
FireRedASR-LLM-L	8300	73.70	18.72	43.93	43.33	34.53	48.05	49.99	49.87	45.92
Conformer-LLM-Yue⭐	4200	17.22	6.21	6.23	9.52	4.35	4.57	6.98	4.73	7.91

## ASR Inference ### U2pp_Conformer_Yue ``` dir=u2pp_conformer_yue decode_checkpoint=$dir/u2pp_conformer_yue.pt test_set=path/to/test_set test_result_dir=path/to/test_result_dir python wenet/bin/recognize.py \ --gpu 0 \ --modes attention_rescoring \ --config $dir/train.yaml \ --test_data $test_set/data.list \ --checkpoint $decode_checkpoint \ --beam_size 10 \ --batch_size 32 \ --ctc_weight 0.5 \ --result_dir $test_result_dir \ --decoding_chunk_size -1 ``` ### Whisper_Medium_Yue ``` dir=whisper_medium_yue decode_checkpoint=$dir/whisper_medium_yue.pt test_set=path/to/test_set test_result_dir=path/to/test_result_dir python wenet/bin/recognize.py \ --gpu 0 \ --modes attention \ --config $dir/train.yaml \ --test_data $test_set/data.list \ --checkpoint $decode_checkpoint \ --beam_size 10 \ --batch_size 32 \ --blank_penalty 0.0 \ --ctc_weight 0.0 \ --reverse_weight 0.0 \ --result_dir $test_result_dir \ --decoding_chunk_size -1 ``` ### SenseVoice_Small_Yue ``` from funasr import AutoModel model_dir = "sensevoice_small_yue" model = AutoModel( model=model_path, device="cuda:0", ) res = model.generate( wav_path, cache={}, language="yue", use_itn=True, batch_size=64, ) ``` ## TTS Inference ### Install **Clone and install** - Clone the repo ``` sh git clone https://github.com/ASLP-lab/WenetSpeech-Yue.git cd CosyVoice2-Yue ``` - Create Conda env: ``` sh conda create -n cosyvoice python=3.10 conda activate cosyvoice # pynini is required by WeTextProcessing, use conda to install it as it can be executed on all platform. conda install -y -c conda-forge pynini==2.1.5 pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com ``` ### Model download ``` python from huggingface_hub import snapshot_download snapshot_download('ASLP-lab/WSYue-TTS', local_dir='pretrained_models') ``` ### Usage ``` python import sys sys.path.append('third_party/Matcha-TTS') from cosyvoice.cli.cosyvoice import CosyVoice, CosyVoice2 from cosyvoice.utils.file_utils import load_wav import torchaudio import opencc # s2t converter = opencc.OpenCC('s2t.json') cosyvoice_base = CosyVoice2( 'ASLP-lab/Cosyvoice2-Yue', load_jit=False, load_trt=False, load_vllm=False, fp16=False ) cosyvoice_zjg = CosyVoice2( 'ASLP-lab/Cosyvoice2-Yue-ZoengJyutGaai', load_jit=False, load_trt=False, load_vllm=False, fp16=False ) prompt_speech_16k = load_wav('asset/sg_017_090.wav', 16000) text = '收到朋友从远方寄嚟嘅生日礼物，嗰份意外嘅惊喜同埋深深嘅祝福令我心入面充满咗甜蜜嘅快乐，笑容好似花咁绽放。' text = converter.convert(text) for i, j in enumerate(cosyvoice_base.inference_instruct2(text, '用粤语说这句话', prompt_speech_16k, stream=False)): torchaudio.save('base_{}.wav'.format(i), j['tts_speech'], cosyvoice.sample_rate) for i, j in enumerate(cosyvoice_zjg.inference_instruct2(text, '用粤语说这句话', prompt_speech_16k, stream=False)): torchaudio.save('zjg_{}.wav'.format(i), j['tts_speech'], cosyvoice.sample_rate) ``` ## WenetSpeech-Pipe WenetSpeech-Pipe Overview:

### Audio Collection WenetSpeech-Pipe collects large-scale, in-the-wild speech recordings across diverse domains such as storytelling, drama, commentary, vlogs, food, entertainment, news, and education. These long recordings are segmented into short clips with VAD, yielding utterance-level data for transcription and quality evaluation.

### Speaker Attribute Annotation To enrich the dataset with speaker-level metadata for multi-speaker modeling and style-aware synthesis, WenetSpeech-Pipe includes a Speaker Attributes Annotation stage. Using [pyannote](https://github.com/pyannote/pyannote-audio) toolkit for speaker diarization and [Vox-Profile](https://github.com/tiantiaf0627/vox-profile-release) for age and gender estimation, each utterance-level segment is annotated with speaker identity, age, and gender, enabling supervised and style-controllable speech modeling. ### Speech Quality Annotation To support high-fidelity tasks such as TTS and voice conversion, WenetSpeech-Pipe integrates a comprehensive quality assessment stage. Each segment is evaluated by (i) [Brouhaha](https://github.com/marianne-m/brouhaha-vad) for signal-to-noise ratio (SNR), (ii) [DNSMOS](https://github.com/microsoft/DNS-Challenge) for perceptual quality (MOS), and (iii) bandwidth detection for spectral coverage. These complementary measures yield structured annotations with quantitative scores and spectral references. ### Automatic Speech Recognition We select three models with the best performance on Cantonese to perform multi-system labeling: SenseVoice, TeleASR, and Whisper. For each audio file, we obtain the corresponding multi-system transcriptions. ### Text Postprocessing Each ASR transcription system produces outputs in different formats. To standardize these formats, we introduce a text post-processing module, which includes punctuation removal, traditional-to-simplified Chinese conversion, and text normalization. The detailed code can be found in `text_postprocessing.py`.

### Recognizer Output Voting Despite text postprocessing, ASR outputs still vary in lexical choice, segmentation, and phonetic representation. To obtain unified and accurate reference transcriptions, we adopt and extend the ROVER framework for Cantonese. Normalized transcriptions are first aligned using dynamic programming, after which a filtering module removes outlier outputs based on edit distance. Voting then selects the most frequent word at each aligned position, and the average voting frequency is recorded as an utterance-level confidence score. In parallel, we extend the voting mechanism to Cantonese pinyin by introducing a pronunciation-level confidence measure, further reinforcing phoneme consistency. To further enhance transcription accuracy, we leverage [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) for minimal, context-aware refinements of the consensus output. The LLM references all original ASR hypotheses and applies only essential corrections—such as grammar, lexical choice, or named entities—while preserving the integrity of the spoken content.

## Contributors |

| | ---- | ---- | ---- | ---- | ---- | ## Citation Please cite our paper if you find this work useful: ``` @misc{li2025wenetspeechyuelargescalecantonesespeech, title={WenetSpeech-Yue: A Large-scale Cantonese Speech Corpus with Multi-dimensional Annotation}, author={Longhao Li and Zhao Guo and Hongjie Chen and Yuhang Dai and Ziyu Zhang and Hongfei Xue and Tianlun Zuo and Chengyou Wang and Shuiyuan Wang and Jie Li and Xin Xu and Hui Bu and Binbin Zhang and Ruibin Yuan and Ziya Zhou and Wei Xue and Lei Xie}, year={2025}, eprint={2509.03959}, archivePrefix={arXiv}, primaryClass={cs.SD}, url={https://arxiv.org/abs/2509.03959}, } ``` ## Contact If you are interested in leaving a message to our research team, feel free to email lhli@mail.nwpu.edu.cn or gzhao@mail.nwpu.edu.cn.

WeChat Group QR Code
Scan to join our WeChat discussion group