# Ambivalence / Hesitancy Recognition in Video ### ABAW 2026 · 3rd A/H Challenge · BAH dataset [![arXiv](https://img.shields.io/badge/arXiv-2607.11120-b31b1b.svg)](https://arxiv.org/abs/2607.11120) [![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/release/python-3100/) [![PyTorch](https://img.shields.io/badge/PyTorch-EE4C2C.svg?logo=pytorch&logoColor=white)](https://pytorch.org/) *Given a short interview video, decide: does this person show ambivalence or hesitancy?* *Six models, one honest ensemble, one command.* **[📄 Paper](https://arxiv.org/abs/2607.11120)** · **[⚡ Quick start](#quick-start)** · **[📊 Results](#results)** · **[📁 File map](#file-map)**

--- ## Overview

AMF architecture: three affect-specialised branches (text, visual, audio) feed a reliability-gated fusion module, combined with five other members into an AP-weighted ensemble at a fixed threshold.

_{Text, video, and audio inputs become affect-specialised features. AMF gates weak channels and
predicts A/H. Six members are combined by AP-weighted averaging at a fixed threshold of 0.5.
Features that hurt the model (gaze/brow dynamics, prosody, cue supervision) are excluded.}

**Public test: 0.731 macro-F1**. --- ## Quick start ```bash # 1. environment conda create -n bah python=3.10 -y && conda activate bah pip install -r requirements.txt # + ffmpeg on PATH, 1 GPU # 2. put the BAH data next to this folder (see "Data layout" below) # 3. everything: features → training → ensemble → report → submission bash run_all.sh ``` That is the whole pipeline. Seed 42, no hand-tuning, no hidden steps. Resume anywhere: `bash run_all.sh --step 8`. --- ## How it works ``` ┌──────────────┐ transcript ──► │ text model │──┐ ├──────────────┤ │ face frames ─► │ video model │──┤ ├──────────────┤ │ AP-weighted fixed waveform ────► │ audio model │──┼──► average of 6 ──► threshold ──► A/H? ├──────────────┤ │ probabilities 0.5 all three ───► │ fusion ×2 │──┤ ├──────────────┤ │ affect feats ► │ AMF (ours) │──┘ └──────────────┘ ``` - **Text** RoBERTa fine-tuned on GoEmotions · **Video** VideoMAE on cropped faces · **Audio** HuBERT · **Fusion** orthogonal + gated-attention variants - **AMF** — our member: FER-ViT face statistics + emotion-wav2vec2 audio + **11 readable hesitation markers** (hedges, filled pauses, "but", repeated words, sentiment flips). A learned gate turns weak channels down. - **ASR-erased time** — a signal we introduce: speech recognisers delete fillers and hesitation pauses from the transcript, but the chunk timestamps keep the time those events took. 16 features built from these gaps form the strongest, most independent non-verbal channel we measured (see the paper). - **The key decision is calibration, not architecture.** Tuning ensemble weights and threshold on the 124-video validation split overfits: it scores 0.741 on validation but only **0.690** on test. Weighting members by AP and fixing the threshold at 0.5 gives **0.731** — that one change is worth more than any model change we tried. ## Results | Setup | Macro-F1 | AP | |---|:---:|:---:| | Ensemble, public test (models trained on train split) | **0.731** | 0.875 | | Ensemble, internal 113-video holdout (all-data models) | 0.814 | 0.907 | Verified end-to-end: a from-scratch rerun of `run_all.sh` reproduces the ensemble to ±0.0001 F1 (individual members may drift ~a point across GPUs/sessions; the near-uniform AP weighting absorbs it). Full ablations (conflict-operator study, per-stream probe, calibration study, what-didn't-help) are in the [paper](https://arxiv.org/abs/2607.11120). --- ## Data layout BAH is EULA-protected — request it from the organisers, then place it **next to** this folder: ``` / ├── code/ ← this folder (any name works) │ └── run_all.sh, *.py, models/ └── data/ ├── Videos//Visite_1/*.mp4 ├── cropped-aligned-faces/Videos//Visite_1/