# IndicContextEval
### A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages
[](https://arxiv.org/abs/2606.19157)
[](https://huggingface.co/datasets/AI4Bharat/IndicContextEval)
[](https://www.interspeech2026.org/)
[](https://creativecommons.org/licenses/by/4.0/)
Sakshi Joshi · Dhruv Subhash Rathi · Sanskar Singh · Eldho Ittan George · R J Hari · Kaushal Bhogale · Mitesh M. Khapra
**AI4Bharat, IIT Madras** · **Sarvam AI**
📄 [Paper (arXiv)](https://arxiv.org/abs/2606.19157) | 🤗 [Dataset](https://huggingface.co/datasets/AI4Bharat/IndicContextEval) | 🎙️ Accepted at **Interspeech 2026**
---
## Abstract
AudioLLMs enable speech recognition conditioned on textual prompts such as domain descriptions
or entity lists. However, it remains unclear whether these models genuinely utilise such context
or rely on parametric knowledge learned during pre-training. Existing benchmarks cannot answer
this question because they evaluate transcription under fixed prompting conditions and rarely
include explicit contextual inputs. We introduce **IndicContextEval**, a 56-hour multilingual
benchmark of natural speech from **555 speakers** across **8 Indian languages** and **23
professional domains**. We design a **7-level prompting framework** that progressively introduces
contextual signals, including metadata, natural-language descriptions, entity lists in English and
native script, and adversarial prompts with incorrect entities. Evaluating five models reveals
substantial differences in context utilisation behaviour, highlighting the need for explicit
evaluation of contextual grounding in AudioLLMs.
*Index Terms — AudioLLMs, Contextual ASR, Benchmarking*
## 📦 Dataset (audio + benchmark)
Benchmark available here → **🤗 [AI4Bharat/IndicContextEval](https://huggingface.co/datasets/AI4Bharat/IndicContextEval)**
```python
from datasets import load_dataset
ds = load_dataset("SakshiJ/IndicContextEval", split="Hindi") # Bengali, Telugu, ...
print(ds[0]["audio"], ds[0]["l5_prompt"]) # audio decoded automatically
```
## 📁 This repository
```
results/
paper_results.csv # headline tables (reproduce the paper)
/results.jsonl.gz # full per-utterance predictions (gpt4o, gemini_3_flash, gemma_3n, sarvam_audio)
/results_summary.csv # clamped WER + NEER, per language × level
supplementary/ # domain categories, sub-domain descriptions, prompt templates (PDF)
```
Decompress predictions with `gunzip results//results.jsonl.gz`.
> **⏳ Evaluation scripts — coming soon.** The build and evaluation code (prompt construction,
> clamped-WER / NEER scoring) will be released here shortly.
## Prompt taxonomy (L0–L6)
| Level | Context added |
|---|---|
| L0 | None (bare transcription, no language hint) |
| L1 | Target language specified (baseline) |
| L2 | + structured domain metadata |
| L3 | + natural-language audio description |
| L4 | + entity list in **English** script |
| L5 | + the same entity list in **native** script |
| L6 | + entity list from an **unrelated** domain (adversarial) |
Exact templates: [`supplementary/Levels_Prompts.pdf`](supplementary/Levels_Prompts.pdf).
## Transcript variants
Each utterance ships three reference forms (English named entities are transliterated into native
script and also tagged in brackets):
| Field | Example | Use |
|---|---|---|
| `reference_text_with_tags` | `মডেল [Model] নির্বাচন` | native + English tags (original) |
| `reference_text_no_tags` | `মডেল নির্বাচন` | native only — **WER is scored on this** |
| `reference_text_no_brackets` | `মডেল Model নির্বাচন` | native + English inline, no brackets |
## Results (clamped WER %, by prompt level)
| Level | GPT-4o | Gemini 3 Flash | Gemma 3n | Sarvam Audio |
|---|---:|---:|---:|---:|
| L0 | 29.83 | 24.30 | 51.21 | 20.39 |
| L1 | 28.61 | 18.90 | 38.73 | 16.86 |
| L2 | 28.37 | 19.28 | 52.20 | 16.78 |
| L3 | 26.08 | 18.39 | 40.22 | 16.43 |
| L4 | 27.97 | 19.88 | 46.37 | 16.80 |
| L5 | 26.04 | 17.46 | 43.11 | 15.70 |
| L6 | 28.47 | 19.67 | 47.95 | 16.69 |
NEER and per-language breakdowns: [`results/paper_results.csv`](results/paper_results.csv) and the
per-model `results_summary.csv`. Standalone ASR baseline IndicConformer (L1): 18.81% WER / 29.58% NEER.
**Metric note.** WER is normalised with the Indic NLP Library and **clamped per-sample at 200%** of
the reference length (prevents decoder-runaway/repetition loops from dominating the corpus average).
NEER = fraction of reference named entities missing/incorrect.
## Citation
If you use IndicContextEval, please cite:
```bibtex
@inproceedings{joshi2026indiccontexteval,
title = {IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages},
author = {Joshi, Sakshi and Rathi, Dhruv Subhash and Singh, Sanskar and George, Eldho Ittan and Hari, R J and Bhogale, Kaushal and Khapra, Mitesh M.},
booktitle = {Proc. Interspeech},
year = {2026},
note = {arXiv:2606.19157}
}
```
## License
Released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
## Contact
If you have any doubts or questions, please reach out: **sakshijcom@gmail.com**