## V2 Audio Dataset & Available Resources
The 6,066 audio WAV files for this dataset, along with the corresponding text annotations for professional descriptions, amateur descriptions, musical segment structure, rhymes, and more, have been uploaded to Hugging Face. Feel free to download them. Scholars and researchers are requested to obtain relevant song audio from legal channels and use it for academic purposes only. Song audio cannot be used for commercial model training without the authorization of the copyright holder.
The dataset is available in two main versions. **Note:** Some data files contain Chinese characters. Please ensure you open them with UTF-8 encoding to prevent garbled text.
- **MuChinV2 6066 (Full Dataset)**
The second, larger release contains 6,066 unique songs and all associated annotations.
- **Audio + Annotations (`.tar.bz2`)**:
- **Github:** [https://github.com/CarlWangChina/MuChin-V2-6066](https://github.com/CarlWangChina/MuChin-V2-6066)
- **Hugging Face:** [huggingface.co/datasets/karl-wang/MuChin-v2-6066](https://huggingface.co/datasets/karl-wang/MuChin-v2-6066)
- **Baidu Netdisk:** [pan.baidu.com/s/1GP5p3Ip_j0MraAcA52gIuQ](https://pan.baidu.com/s/1GP5p3Ip_j0MraAcA52gIuQ) (Password: `68c1`)
- **Database File**:
- **Hugging Face:** [huggingface.co/datasets/karl-wang/MuChin-v2-6066](https://huggingface.co/datasets/karl-wang/MuChin-v2-6066)
- **MuChinV1 1000**
This initial set contains 1,000 audio tracks with detailed text annotations.
- **Github:** [https://github.com/CarlWangChina/MuChin](https://github.com/CarlWangChina/MuChin)
- **Hugging Face:** [huggingface.co/datasets/karl-wang/MuChin1k](https://huggingface.co/datasets/karl-wang/MuChin1k)
- **Baidu Netdisk:** [pan.baidu.com/s/1D4xGQhYUwWbpaHyAS71dfw](https://pan.baidu.com/s/1D4xGQhYUwWbpaHyAS71dfw) (Password: `1234`)
The MuChin dataset was constructed through the custom-built MuChin Music Annotation Platform (MMAP), utilizing a multi-stage, collaborative annotation process that engaged both amateur enthusiasts and professionals to capture diverse linguistic perspectives on music. Each musical piece was subjected to thorough annotations, including descriptive labels, structural segmentation (such as verses and choruses), and rhyme schemes, ensuring high-quality, semantically rich descriptions grounded in colloquial Chinese language. A subset of 1,000 songs has been publicly released as the first open-source benchmark for evaluating multimodal large language models' ability to understand and describe music in line with human perception. Additionally, the full collection of 6,066 songs is available for public use, providing an extensive resource for instruction fine-tuning and alignment training of music-oriented large models.
## Citation
We invite scholars and researchers to employ this resource broadly in their research initiatives. Proper reference to its use in academic publications is appreciated.
If you use the MuChin benchmark or dataset in your research, please cite our paper:
#### BibTeX
```bibtex
@inproceedings{wang2024muchin,
title = {MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music},
author = {Wang, Zihao and Li, Shuyu and Zhang, Tao and Wang, Qi and Yu, Pengfei and Luo, Jinyang and Liu, Yan and Xi, Ming and Zhang, Kejun},
booktitle = {Proceedings of the Thirty-Third International Joint Conference on
Artificial Intelligence},
pages = {7771--7779},
year = {2024},
month = {8},
}
```
# How the amateur-professional divide in musical language biases generative AI
This repository contains the dataset, code, and resources for the paper "How the amateur-professional divide in musical language biases generative AI". Our research leverages generative AI as a novel computational lens to provide a large-scale, quantitative analysis of the cognitive and linguistic divide between experts (professionals) and novices (amateurs) in the domain of music.
The empirical foundation of this research is the **MuChin Dataset**, a large-scale corpus specifically constructed to capture and quantify this descriptive divide.
---
### Key Features & Contributions
- **A Novel Dataset for Cognitive Science**: The MuChin dataset, featuring paired descriptions from both amateur and professional perspectives for thousands of musical pieces.
- **Computational Analysis of the Divide**: Methodologies and code to quantitatively analyze the linguistic and semantic differences between user groups.
- **Probing AI with Human Language**: A framework to test how generative models respond to different human linguistic patterns.
- **AI-driven Interventions**: Code for exploring interventions (e.g., RAG, targeted training) designed to bridge the human-AI communication gap.
---
### Repository Content Overview
This repository maintains a comprehensive structure containing all related scripts and modules used throughout our research. The main components are:
- **/Datasets-for-MuChin-V2/**: Contains data samples, metadata, and detailed documentation about the MuChin dataset. The full audio and database files are hosted externally.
- **/Code-for-Experiment/**: The central hub for our experimental code. It is organized into the following key areas:
- **/Metrics/**: Implementations for evaluating model outputs. This includes our primary **Semantic Similarity** analysis tools used to quantify the descriptive divide, as well as metrics for assessing the downstream impact on AI behaviour, such as **Intent Alignment** (SAA, ARA).
- **/RAG/**: Our implementation of a Retrieval-Augmented Generation system, focusing on a CLAP-based text-to-audio retrieval pipeline to enhance prompt understanding. Key components include data preparation, vector search frontend, and audio processing utilities.
- **/Targeted-Training/**: Contains various scripts and configurations for model training. This includes reference implementations for different architectures and tools like **NVIDIA Apex** for high-performance mixed-precision training.
- **/Code-for-MuChin-AP/**: The complete source code for our custom-built data annotation platform, **MuChin-AP**.
- **/user_manual_munchin_ap/**: Documentation, guidelines, and internal planning documents related to the annotation platform and process.
---
### Installation
1. **Clone the repository:**
```bash
git clone [https://github.com/CarlWangChina/MuChin-V2-6066.git](https://github.com/CarlWangChina/MuChin-V2-6066.git)
cd MuChin-V2-6066
```
2. **Create and activate a virtual environment (recommended):**
```bash
python3 -m venv venv
source venv/bin/activate
```
3. **Install general dependencies:**
```bash
pip install -r requirements.txt
```
4. **Install dependencies for specific modules:**
- **Semantic Analysis**: To use the semantic similarity tools, install the specific libraries required by the `bge-large-zh-v1.5` model:
```bash
pip install -U FlagEmbedding
pip install transformers==4.34.0
```
- **NVIDIA Apex (for advanced training)**: For performance and full functionality, we recommend installing Apex with CUDA and C++ extensions:
```bash
# Navigate to the apex directory
cd Code-for-Experiment/Metrics/music_understanding_model/apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" .
cd ../../../../../.. # Return to root
```
---
### Core Modules & Experiments Guide
This section provides a more detailed look into the key components of our research and how to use them.
#### 1. Semantic Similarity Analysis
This module assesses the semantic similarity between two sets of text labels (e.g., amateur vs. professional).
- **Download the Embedding Model**:
This code uses `bge-large-zh-v1.5` for analyzing Chinese text.
```bash
cd ./Code-for-Experiment/Semantic-Analysis/
mkdir -p models
cd models
git clone [https://huggingface.co/BAAI/bge-large-zh-v1.5](https://huggingface.co/BAAI/bge-large-zh-v1.5)
cd ../../../.. # Return to root
```
- **Usage in Python**:
```python
# Ensure your script is run from the project root
from Code-for-Experiment.Semantic-Analysis.semantic_similarity import MuChindata_Analyzer
analyzer = MuChindata_Analyzer(
'path/to/your/labels-free.xlsx',
'./Code-for-Experiment/Semantic-Analysis/models/bge-large-zh-v1.5'
)
list_1 = [["Nostalgia", "Pop Songs"], ["Slow", "Medium"]]
list_2 = [["Nostalgia", "Style and Genre"], ["Slow", "Slower"]]
similarity_score = analyzer.cal_word_similarity(list_1, list_2)
print(f"Semantic Similarity Score: {similarity_score}")
```
#### 2. Structured Lyric Generation & Evaluation
This module uses LLMs to generate structured lyrics based on a theme and a structural template, then evaluates the output based on structure, alignment, and rhyme.
- **One-Shot Prompting**: The core of the generation is a detailed one-shot prompt that instructs the LLM on how to format the output.
- **Objective Evaluation**: The quality of the generated lyrics is assessed using `glrc_obj_eval.py`. The evaluation considers multiple dimensions:
- **Overall Performance**: Gestalt pattern matching between the required and generated structure.
- **Section Structure**: Correctness of section names, order, and line counts.
- **Word Count Alignment**: How closely the word count of each line matches the template.
- **Rhyme**: Adherence to end-of-line rhyme requirements (`R` markers), based on the "中华新韵" rhyme scheme.
#### 3. RAG and CLAP-based Retrieval
Our Retrieval-Augmented Generation system uses CLAP to bridge the semantic gap.
- **Data Pipeline**: The pipeline in `Code-for-Experiment/RAG/data_prep_muchin_to_clap_vectors/` processes amateur and professional descriptions into 10 different text formats per song, which are then converted into CLAP-based vector embeddings for retrieval.
- **Vector Search Frontend**: The code in `Code-for-Experiment/RAG/clap_retrieval_system/` and `rag_search_frontend/` provides a user interface for the retrieval system. To run it:
```bash
# First, create a user
python Code-for-Experiment/RAG/clap_retrieval_system/adduser.py
# Then, launch the server
python Code-for-Experiment/RAG/clap_retrieval_system/server.py
```
#### 4. Targeted Training & Model Architectures
The repository contains code for training and evaluating various models. A key utility used is **NVIDIA Apex** for high-performance mixed-precision training.
- **Example Training Command (Distributed Training with Apex)**:
A typical command for launching a distributed training job (e.g., for ImageNet, adapted for our models) looks like this:
```bash
python -m torch.distributed.launch --nproc_per_node=2 main_amp.py \
-a resnet50 --b 224 --workers 4 --opt-level O1 ./
```
This demonstrates the use of `torch.distributed.launch` and Apex's automatic mixed precision (`--opt-level O1`).
---
#### Important Usage Notes
1. **Timestamp Discrepancy**: The primary focus of this paper's manual annotation was on descriptive and structural content, not timestamp correction. For the most accurate available timestamps, please use the `raw_lyric` files. The timestamps in `tknz_json` files are incorrect due to manual merging and splitting of lyric lines during annotation.
2. **Duplicate Annotations**: Due to an operational error, 724 songs from the initial `MuChin 1000` release were annotated a second time. We recommend using the annotation results from the `MuChin 1000` directory for these specific songs. A list of these duplicates can be found in `muchin_5790_1000_overlap.jsonl`.
3. **File Compression**: The dataset is packaged as a `.tar.bz2` archive to prevent extraction errors associated with large `.zip` files on some systems.
#### Component Models
To ensure the full reproducibility of our findings, we provide the fine-tuned model weights used in our intervention experiments (e.g., for the RAG system). These are available at our Hugging Face Model repository:
`huggingface.co/karl-wang/ama-prof-divi`
---