# EmoPrefer-Audit

Official implementation of **Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation**.

This repository covers the content-blind probes, counterfactual controls, fine-tuned text and audio-visual judges, ODIN-style diagnostics, the rationale-first judge, and cross-model zero-shot evaluations reported in the paper. Private data, model weights, checkpoints, logs, and precomputed predictions are not distributed.

## Repository layout

```text
.
├── README.md
├── LICENSE
├── requirements-audit.txt
├── requirements-gpu.txt
├── data/
│   └── README.md
├── outputs/                  # generated locally; contents are not distributed
└── scripts/
    ├── shortcut_audit.py
    ├── probe_interventions.py
    ├── train_text_judge.py
    ├── train_omni_judge.py
    ├── train_text_odin.py
    ├── train_omni_odin.py
    ├── generate_star_rationales.py
    ├── train_omni_cot.py
    ├── eval_qwen_vl.py
    ├── eval_llava_video.py
    └── merge_prediction_shards.py
```

## Data

Obtain EmoPrefer and the MER2026 Track 3 media from the official challenge release and accept the applicable data terms. This repository does not redistribute either the annotation tables or media. Place the files under `data/` as described in [`data/README.md`](data/README.md), or pass explicit paths through the command-line options.

The audit requires the CSV variant containing `model1` and `model2`. Neural judges do not receive these two fields; they are used only by the audit oracle and by the explicitly labeled ODIN confound heads.

## Installation

The content-blind audit is CPU-only:

```bash
python -m venv .venv-audit
source .venv-audit/bin/activate
pip install -r requirements-audit.txt
```

For neural judges, first install a PyTorch build compatible with the local CUDA driver, then install the remaining packages:

```bash
pip install -r requirements-gpu.txt
```

The reported runs used Python 3.10, PyTorch 2.5.1 with CUDA 12.1, and the package versions listed in the requirement files. Model identifiers may be replaced by local directories through `--model`.

## Reproduce the shortcut audit

Run the content-blind probes, sliced evaluation, out-of-fold generator prior, and TF--IDF source classifier:

```bash
python scripts/shortcut_audit.py \
  --data_csv data/preference_emopreferv2_with_modelnames.csv \
  --output outputs/shortcut_audit_v2.json
```

Run the generator permutation and test-time identity/length swaps:

```bash
python scripts/probe_interventions.py \
  --data_csv data/preference_emopreferv2_with_modelnames.csv \
  --output outputs/probe_interventions_v2.json
```

Both commands use five folds, fold seed 42, and the feature definitions stated in the paper. `shortcut_audit.py` assigns folds before removing `same` labels. The intervention script operates on the binary evaluation population because its paired outcomes are defined only for `a1`/`a2` decisions.

To evaluate newly generated judge probabilities on the same slices, add `--predictions_dir outputs` to `shortcut_audit.py`. Each fold file must be named `<experiment>_f<fold>_probs.npz` and contain arrays named `name`, `p_a1`, and `gt`.

## Fine-tuned judges

All commands below show one fold. Repeat `--fold 0` through `--fold 4`. Independent folds can run on separate GPUs, for example by assigning a different `CUDA_VISIBLE_DEVICES` value to each process.

Text-only Qwen2.5-7B judge:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/train_text_judge.py \
  --which v2 --fold 0 --model Qwen/Qwen2.5-7B-Instruct
```

Use `--which both` for V1+V2 training. Training is symmetric, and evaluation averages both candidate orders.

Audio-visual Qwen2.5-Omni-7B judge:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/train_omni_judge.py \
  --which v2 --fold 0 \
  --model Qwen/Qwen2.5-Omni-7B \
  --video_dir data/video --max_frames 12 --use_audio_in_video 1
```

The media ablations use `--max_frames 16 --use_audio_in_video 1` and `--max_frames 8 --use_audio_in_video 0`, respectively. If audio is stored separately, use `--audio_dir data/audio --use_audio_in_video 0`; omit `--audio_dir` for the no-audio condition.

## ODIN-style diagnostics

Text control without decorrelation, retaining the auxiliary heads and orthogonality term:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/train_text_odin.py \
  --which v2 --fold 0 --lam 0 --lam_o 1 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --exp text_no_decorrelation_v2_f0
```

Text ODIN diagnostic:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/train_text_odin.py \
  --which v2 --fold 0 --lam 1 --lam_o 1 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --exp text_odin_v2_f0
```

Direct single-head multimodal Bradley--Terry control:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/train_omni_odin.py \
  --which v2 --fold 0 --objective single_head \
  --model Qwen/Qwen2.5-Omni-7B \
  --video_dir data/video --max_frames 12 \
  --exp omni_no_decorrelation_v2_f0
```

Multimodal ODIN diagnostic:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/train_omni_odin.py \
  --which v2 --fold 0 --objective odin --lam 1 --lam_o 1 \
  --model Qwen/Qwen2.5-Omni-7B \
  --video_dir data/video --max_frames 12 \
  --exp omni_odin_v2_f0
```

The ODIN scripts train the summed reward but retain only the decorrelated content head at evaluation. In the multimodal control, `single_head` means a directly trained content head with no auxiliary length or generator heads; it is not the three-head model with a coefficient set to zero.

## Rationale-first judge

Generate rejection-sampled STaR rationales from the train split of each fold:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/generate_star_rationales.py \
  --which v2 --fold 0 --video_dir data/video \
  --max_frames 8 --use_audio_in_video 0 \
  --out outputs/cot_v2_f0.jsonl
```

Then fine-tune and evaluate the generative verifier on the same fold:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/train_omni_cot.py \
  --which v2 --fold 0 --video_dir data/video \
  --max_frames 8 --use_audio_in_video 0 \
  --rationale_glob 'outputs/cot_v2_f0*.jsonl'
```

The JSONL rationale files are generated training artifacts and are not part of this release.

## Zero-shot cross-model judges

Qwen2.5-Omni zero-shot evaluation uses the same pairwise script with `--zero_shot 1`. Sharding avoids duplicating samples across GPUs:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/train_omni_judge.py \
  --which v2 --zero_shot 1 --shard 0 --nshard 8 \
  --model Qwen/Qwen2.5-Omni-7B --video_dir data/video \
  --exp omni_zero_shot
```

The Qwen2.5-VL and LLaVA-NeXT-Video evaluations use:

```bash
CUDA_VISIBLE_DEVICES=0 python scripts/eval_qwen_vl.py \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --data_csv data/preference_emopreferv2.csv \
  --video_dir data/video --shard 0 --nshard 8 --exp qwen_vl_zero_shot

CUDA_VISIBLE_DEVICES=0 python scripts/eval_llava_video.py \
  --model llava-hf/LLaVA-NeXT-Video-7B-hf \
  --data_csv data/preference_emopreferv2.csv \
  --video_dir data/video --shard 0 --nshard 8 --exp llava_zero_shot
```

Repeat each command with shard indices 0--7, then merge the shards:

```bash
python scripts/merge_prediction_shards.py \
  --input_dir outputs --experiment qwen_vl_zero_shot
```

## Paper-to-code map

| Paper component | Script |
|---|---|
| Length, generator, combined, and win-rate probes | `shortcut_audit.py` |
| Length-matched/counter-stereotypical slices and bootstrap CIs | `shortcut_audit.py` |
| TF--IDF generator recovery | `shortcut_audit.py` |
| Generator permutation and identity/length swaps | `probe_interventions.py` |
| Text judges | `train_text_judge.py` |
| Audio-visual judges and Omni zero-shot judge | `train_omni_judge.py` |
| Text ODIN and no-decorrelation control | `train_text_odin.py` |
| Omni ODIN and direct single-head control | `train_omni_odin.py` |
| STaR rationale generation and CoT judge | `generate_star_rationales.py`, `train_omni_cot.py` |
| Qwen2.5-VL and LLaVA-NeXT-Video zero-shot judges | `eval_qwen_vl.py`, `eval_llava_video.py` |

## Outputs and released artifacts

Scripts write generated artifacts to `outputs/` by default. Raw predictions, JSON reports, rationale files, logs, adapters, and checkpoints are intentionally excluded from the submitted archive. They are not required to inspect the method and can all be regenerated from the official data and public model weights. The numerical results to compare against are reported in the paper rather than duplicated as opaque binary files.

## Reproducibility notes

- The shared cross-validation split is shuffled five-fold `KFold` with `random_state=42`.
- `same` examples participate in fold assignment but are excluded from binary training and WAF evaluation, except in the explicitly binary-only intervention script noted above.
- Generator win rates are computed from each training fold only.
- Pairwise neural judges use symmetric training and dual-order inference.
- The reported confidence intervals bootstrap concatenated out-of-fold pairs; they do not represent variation across training seeds.
- Model downloads and data access may require accepting their respective licenses or access conditions.