# Audio-LLM-benchmarking-uncertainty # 🎤 Official Implementation of **Interspeech 2025** Paper 📄 **Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning** 🔗 [Read the Paper on arXiv](https://arxiv.org/pdf/2505.13115) 📑 [Poster](https://github.com/iiscleap/Audio-LLM-benchmarking-uncertainty/blob/main/poster/poster_Interspeech2025.pdf) --- ## 📚 Overview This repository accompanies our **Interspeech 2025** paper, introducing the **Temporal Reasoning Evaluation of Audio (TREA)** dataset. ✨ **Key Highlights:** - 🧩 **TREA Dataset** with 3 components: - 🔀 **TREA-O** → *Ordering* questions (200 Qs) - ⏳ **TREA-D** → *Duration* reasoning (200 Qs) - 🔢 **TREA-C** → *Counting* tasks (200 Qs) - 🤖 Benchmarks on multiple **audio-LLMs**. - 🎯 **Uncertainty & Calibration** evaluations via input perturbations. - 🔍 Comparative insights with existing temporal/audio reasoning datasets. --- ## 👀 Dataset Glimpse & Comparisons Here’s a preview of TREA along with comparisons to related benchmarks:

TREA samples

--- ## 📂 TREA Dataset The **TREA dataset** is organized into three task-specific folders: - 🔀 **order** (TREA-O) - 🔢 **count** (TREA-C) - ⏳ **duration** (TREA-D) Each folder contains: - 🎵 **audios/** → audio files for question answering - 📑 **[taskname].csv** → contains audio file details, questions, multiple-choice options, and ground-truth answers - 📝 **[taskname]_metadata.csv** → metadata associated with the respective task. --- ## 📁 Directory Structure The TREA dataset in MCQ setting: ```text TREA_dataset/ ├── order/ │ ├── audios/ – audio clips for ordering questions │ ├── order.csv – Q/A entries (MCQs) with correct answer │ └── order_metadata.csv – metadata for the ordering task │ ├── count/ │ ├── audios/ – audio clips for counting questions │ ├── count.csv – Q/A entries (MCQs) with correct answer │ └── count_metadata.csv – metadata for the counting task │ └── duration/ ├── audios/ – audio clips for duration reasoning ├── duration.csv – Q/A entries (MCQs) with correct answer └── duration_metadata.csv – metadata for the duration task ``` The TREA dataset in short answer in open-ended text setting: ```text TREA_dataset_open_text/ ├── order/ │ ├── audios/ – audio clips for ordering questions │ └──order_nl.csv – open-text correct answer │ ├── count/ │ ├── audios/ – audio clips for counting questions │ └── count_nl.csv – open-text correct answer │ └── duration/ ├── audios/ – audio clips for duration reasoning └── duration_nl.csv – open-text correct answer ``` --- ## 1. Benchmarking ### 1.1 Qwen2-Audio Download model and dependencies: [https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct](https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct) Run benchmarking: ```bash python qwen_benchmarking.py \ --method vanilla \ --task count \ --csv_path TREA_dataset/count/count.csv ``` * `method`: `vanilla`, `cot`, `exp`, or `audio_desc` * `task`: `count`, `duration`, or `order` Each line of the output txt file contains ### 1.2 SALMONN Setup SALMONN: [https://github.com/bytedance/SALMONN](https://github.com/bytedance/SALMONN) Move `salmonn_benchmarking.py` to the SALMONN repo. Resample audio files in `TREA_dataset//audios` to 16kHz and save as `TREA_dataset//_resampled` Run benchmarking: ```bash python salmonn_benchmarking.py \ --method \ --task \ --csv_path \ --wav_folder ``` ### 1.3 Compute Accuracy ```bash python benchmarking_acc.py --model qwen --task count --method vanilla ``` --- ## 2. Generating perturbations ### 2.1 Clone the Repository ```bash git clone https://github.com/iiscleap/Audio-LLM-benchmarking-uncertainty cd Audio-LLM-benchmarking-uncertainty/ ``` ### 2.2 Download Dataset Download the ESC-50 dataset from: [https://github.com/karolpiczak/ESC-50](https://github.com/karolpiczak/ESC-50) Unzip the ESC-50 dataset. ### 2.3 Generate Audio Perturbations (Count Task) ```bash python generate_count_perturb.py \ --input_csv TREA_dataset/count/count_with_metadata.csv \ --output_folder TREA_dataset/count_perturbed \ --num_samples 15 \ --num_samples_per_type 5 ``` * `input_csv`: count task metadata CSV file * `output_folder`: folder to save output CSV and audios * `num_samples`: number of questions randomly sampled * `num_samples_per_type`: number of perturbations per type Generates 5 volume, 5 pitch, and 5 add/delete event perturbations per original sample (15 samples). Total = 225 audio perturbations. Repeat similarly for `duration` and `order` tasks. ### 2.4 Generate Text Perturbations ```bash python generate_text_perturb.py \ --input_csv TREA_dataset/count_perturbed/count_perturbations.csv \ --output_folder TREA_dataset/count_perturbed ``` Output: `reworded_count_perturbations.csv` with 4 text perturbations per original sample. The audio and text perturbations are done for specific tasks in a manner that should not alter the model predictions. A few examples are below:

Perturbations of TREA samples

--- ## 3. Confidence Estimation ### 3.1 Qwen2-Audio ```bash python qwen_uncertainty.py \ --task count \ --csv_path /path/to/your_perturbed_input.csv ``` Output saved to: `confidence_estimation/qwen_count.txt` Each line of the output txt file contains ### 3.2 SALMONN Replace `salmonn.py` in `SALMONN/models` with the provided version. Move `salmonn_uncertainty.py` to SALMONN/ Run: ```bash python salmonn_uncertainty.py \ --task count \ --csv_path /path/to/input.csv \ --wav_folder /path/to/resampled_perturbed_audio_files/ ``` ### 3.3 SALMONN-based Audio Description + LLM Answer Move `desc_llm_uncertainty.py` to SALMONN/ ```bash python desc_llm_uncertainty.py \ --task count \ --csv_path /path/to/input.csv \ --wav_folder /path/to/resampled_perturbed_audio_files/ ``` ### 3.4 ECE and EUE Calculation ```bash python conf_estimation_code.py --task count --model qwen ``` --- ## 📖 Citation If you find this repository useful, please cite our paper: ```bibtex @article{bhattacharya2025benchmarking, title={Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning}, author={Bhattacharya, Debarpan and Kulkarni, Apoorva and Ganapathy, Sriram}, journal={arXiv preprint arXiv:2505.13115}, year={2025} }