# MLC-SLM 2nd Workshop Baseline for Task 2: Multilingual Conversational Speech Understanding

This repository provides the baseline system for **Task 2 (Multilingual Conversational Speech Understanding)** of the 2nd MLC-SLM Challenge.
The system is built on **Qwen2.5-Omni-7B** and fine-tuned with **LoRA** using the [ms-swift](https://swift.readthedocs.io/) framework (Megatron-LM backend).
It evaluates the model's ability to understand acoustic and semantic information in multilingual conversations by answering multiple-choice questions.

## ✨ Key Features of the 2nd MLC-SLM Workshop

- **Extended language coverage**: 3 new languages (Tagalog, Urdu, Turkish) and additional accents (Canadian French, Mexican Spanish, Brazilian Portuguese) on top of the 11 languages from the 1st edition.
- **More challenging tasks**: Task 2 goes beyond speech recognition and diarization, targeting **acoustic and semantic understanding** of conversational speech through multiple-choice questions.
- **Real-world conversational data**: The dataset captures natural human communication including pauses, interruptions, overlaps, and diverse conversational styles.

## 📋 Task Description – Task 2

**Objective**  
Develop a system that answers multiple-choice questions about the content, speaker intentions, emotions, and other acoustic/semantic properties of a given multilingual conversation.  
During evaluation, **no prior/oracle information** (such as pre-segmented utterances or speaker labels) is provided.

**Evaluation Metric**  
Accuracy (ACC) over the multiple-choice questions. No ASR transcript or diarization output is required.

## 🏗 Baseline System

- **Model**: [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) – an omni-modal large language model with native audio understanding.
- **Fine-tuning method**: LoRA (rank=8, alpha=32) on all linear modules, using Megatron-LM via ms-swift.
- **Training data**: Synthetic multiple-choice questions generated by Gemini 2.5-Pro, covering acoustic and semantic understanding of the provided training conversations.  
  *Dev set questions and answers are released publicly to help participants design their own systems.*
- **Inference**: 8-GPU parallel decoding with a retry mechanism and sampling strategy to improve answer extraction.

## 📦 Setup

1. Install the ms-swift framework following the [Megatron-SWIFT Quick Start](https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Quick-start.html).
2. Clone this repository and place the code under the ms-swift examples directory:
   ```bash
   git clone <this-repo-url> MLC-SLM-2nd-Baseline
   cp -r MLC-SLM-2nd-Baseline/examples/mlcslm_2nd /path/to/ms-swift/examples/
   ```

## 🚀 Usage

All commands are orchestrated by `run.sh`. You can run stage by stage.

```bash
cd examples/mlcslm_2nd
bash run.sh
```

**Stages** (configurable via `stage` and `stop_stage` in `run.sh`):
- **Stage 0**: Convert Qwen2.5-Omni-7B to Megatron format
- **Stage 1**: LoRA training with Megatron-LM (8 GPUs, 80 GiB each)
- **Stage 2**: Convert the trained LoRA adapters back to HuggingFace format
- **Stage 3**: Inference on the dev set using `train_infer.py`

You can also run inference separately:
```bash
python train_infer.py \
    --model_path /path/to/train_hf \
    --jsonl_path /path/to/dev.jsonl \
    --save_path /path/to/output.jsonl
```

## 📊 Baseline Results on Dev Set

The following tables report the baseline accuracy on the released development set (300 multiple-choice questions).

### Accuracy by Language

| Language            | Correct | Total | ACC    |
|---------------------|---------|-------|--------|
| English             | 19      | 74    | 0.2568 |
| French              | 4       | 16    | 0.2500 |
| French (Canada)     | 6       | 18    | 0.3333 |
| German              | 2       | 14    | 0.1429 |
| Italian             | 2       | 12    | 0.1667 |
| Japanese            | 9       | 22    | 0.4091 |
| Korean              | 9       | 16    | 0.5625 |
| Portuguese          | 12      | 20    | 0.6000 |
| Portuguese (Brazil) | 1       | 14    | 0.0714 |
| Russian             | 3       | 10    | 0.3000 |
| Spanish             | 2       | 10    | 0.2000 |
| Spanish (Mexico)    | 4       | 10    | 0.4000 |
| Tagalog             | 5       | 14    | 0.3571 |
| Thai                | 7       | 10    | 0.7000 |
| Turkish             | 8       | 14    | 0.5714 |
| Urdu                | 4       | 10    | 0.4000 |
| Vietnamese          | 9       | 16    | 0.5625 |
| **Overall**         | **106** | **300** | **0.3533** |

### Accuracy by Difficulty

| Difficulty | Correct/Total | ACC    |
|------------|---------------|--------|
| Easy       | 44/108        | 0.4074 |
| Medium     | 50/139        | 0.3597 |
| Hard       | 12/53         | 0.2264 |

### Accuracy by Category

| Category            | Correct/Total | ACC    |
|---------------------|---------------|--------|
| Acoustic            | 53/111        | 0.4775 |
| Acoustic-Semantic   | 33/98         | 0.3367 |
| Semantic            | 20/91         | 0.2198 |

### Accuracy by Subtype

| Subtype                               | Correct/Total | ACC    |
|---------------------------------------|---------------|--------|
| non-speech vocal sound                | 11/16         | 0.6875 |
| speaking rate change                  | 9/14          | 0.6429 |
| tone/emotion distinction              | 18/40         | 0.4500 |
| speaker distinction                   | 12/31         | 0.3871 |
| hesitation/pause                      | 18/50         | 0.3600 |
| mismatch literal vs true attitude     | 10/34         | 0.2941 |
| dialogue information extraction       | 15/56         | 0.2679 |
| speaker intention judgment            | 3/15          | 0.2000 |
| cause/motivation understanding        | 2/20          | 0.1000 |

These results highlight that **acoustic-only** questions are easier for the model, while tasks requiring deep semantic reasoning (e.g., cause/motivation) remain very challenging. The baseline performance also varies significantly across languages.

## 📧 Contact

For questions regarding this baseline, please refer to the challenge organizers or open an issue in this repository.

- Bingshen Mu: `bsmu@mail.nwpu.edu.cn`
```