# SpeechJBB

Code for **SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio
Language Models under Code-Switched Speech** (EMNLP Findings 2026).

Dataset: https://huggingface.co/datasets/McGill-NLP/SpeechJBB

Paper: https://arxiv.org/abs/2606.06037

## Setup

```bash
pip install -r requirements.txt
export OPENAI_API_KEY=...   # code-switching, pseudo-words, judging
export GOOGLE_API_KEY=...   # Gemini inference
```

SALMoNN additionally needs `SALMONN_REPO`, `SALMONN_CKPT`,
`SALMONN_BEATS_PATH`, and `SALMONN_WHISPER_PATH`.

## Pipeline

| Step | Script |
| --- | --- |
| Translate JailbreakBench prompts into DE/ES/FR/IT | `translate.py` |
| Generate EN--X code-switched text | `code_switch.py` |
| Generate X--Y code-switched text | `code_switch_txt.py` |
| Insert pseudo-words (10/30/50%) | `augment_code_switch.py` |
| Synthesize speech with XTTS | `audio_xtts.py` |
| Run the nine LALMs | `query_LLMs.py` |
| Judge responses (Refusal / Deflection / Jailbroken) | `judge.py` |
| Aggregate judged outputs into per-condition rates | `analyze_judge_summary.py` |

## Analyses

| Analysis | Script |
| --- | --- |
| Pseudo-word detection and meaning attribution | `pseudo_words_meaning.py`, `judge_pseudo_words.py` |
| MGSM, FLEURS, FLEURS-SLU comprehension | `judge_comprehension.py`, `audio_xtts_mgsm.py`, `audio_gemini_mgsm.py`, `asr_transcribe_mgsm.py` |
| WER / CER for synthesized audio | `wer_cer.py` |
| Figures | `plots.py`, `plots_gibberish.py`, `plots_mgsm.py` |

## Citation

```bibtex
@inproceedings{ceccatelli2026speechjbb,
  title     = {SpeechJBB: Probing Safety Alignment and Comprehension in Large
               Audio Language Models under Code-Switched Speech},
  author    = {Ceccatelli, Virginia and Jeon, Yejin and Adelani, David Ifeoluwa},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  publisher = {Association for Computational Linguistics}
}
```