# SpeechEQ

**SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models**

*[Dataset](https://huggingface.co/datasets/SpeechEQ/SpeechEQ)*

SpeechEQ is a benchmark for evaluating emotional intelligence (EQ) in speech-language models. It tests whether models can recognize and respond to **paralinguistic cues** — the *how* of speech (tone, prosody, affect) — not just lexical content. The benchmark is grounded in the validated EQ-i 2.0 psychological framework and covers 15 subscales across 5 dimensions.

---

## Key Results

Three recurring failure modes in current SLMs: **modality shortcut** (over-reliance on text), **affective flattening** (safety alignment suppresses emotional range), **contextual amnesia** (8% accuracy drop over multi-turn context).

---

## Overview

SpeechEQ has two main components:

1. **Data Generation** — Generates realistic multi-turn dialogues across 15 EQ subscales. Each example produces 8 audio files: 3 context sentences, a high/low EQ pair at Turn 4, a bridge sentence, and a high/low EQ pair at Turn 6.

2. **Evaluation** — Tests speech models via a two-round forced-choice MCQ. At each critical turn the model selects the higher-EQ audio response. The primary metric is **SEQ** (Spoken Emotional Quotient), a normalized score (μ=100, σ=15) inspired by Raven's Standard Progressive Matrices.

---

## Using Claude Code

A `skill.md` file is included at the repo root. It gives Claude deep context about the generation and evaluation pipelines so you can run and iterate on them through natural language.

**Setup:** Install [Claude Code](https://claude.ai/code), then open this repo in your terminal and start a session:

```bash
claude
```

Claude will automatically pick up `skill.md` as project context. You can then ask it to run generation or evaluation directly, for example:

```
Generate 10 examples for the Empathy subscale using OpenAI TTS.
Evaluate the dataset at ./hf_dataset with GPT-4o audio.
Run a quick end-to-end test with 2 examples per subscale.
```

Claude handles the correct flags, paths, and command sequencing for you.

---

## Installation

### Quick Setup

```bash
git clone <repository-url>
cd SpeechEQ
bash setup_env.sh
conda activate speecheq
```

The setup script creates a `speecheq` conda environment (Python 3.12), installs all dependencies, and sets up vLLM for on-device model inference.

### Manual Installation

```bash
conda create -n speecheq python=3.12
conda activate speecheq
pip install -r requirements.txt
```

### Environment Variables

Copy `.env.example` to `.env` and fill in your keys:

```bash
cp .env.example .env
```

| Variable | Required for |
|---|---|
| `OPENAI_API_KEY` | Data generation (GPT-4o), OpenAI TTS, GPT-4o Audio evaluation |
| `PORTKEY_API_KEY` | Gemini evaluation via Portkey |
| `HF_TOKEN` | Loading/uploading the dataset from HuggingFace Hub |
| `AZURE_OPENAI_API_KEY` | Azure OpenAI generation (`generate_azure.py`) |

---

## Data Generation

Generate the benchmark dataset from scratch:

```bash
# Generate 60 examples per subscale (900 total)
python generate.py --examples-per-subscale 60

# Single subscale
python generate.py --subscale "Empathy" --examples-per-subscale 60

# Use OpenAI TTS instead of Qwen3-TTS
python generate.py --tts-provider openai

# Azure OpenAI variant
python generate_azure.py
```

**Arguments for `generate.py`:**

| Argument | Default | Description |
|---|---|---|
| `--examples-per-subscale` | 60 | Examples to generate per subscale |
| `--subscale` | *(all)* | Restrict to a single subscale |
| `--output-dir` | `./data` | Output directory |
| `--llm-provider` | `openai` | LLM for text generation (`openai`, `anthropic`, `qwen`) |
| `--tts-provider` | `openai` | TTS provider (`openai`, `qwen3`) |
| `--parallel-workers` | 8 | Concurrent generation workers |
| `--config` | `config.yaml` | Path to config file |

> **Note:** Parallel generation uses timestamp-based IDs. Under high concurrency, occasional ID collisions may cause some examples to be overwritten — the batch summary reports the actual saved count.

### Output Format

```
data/
└── <eq_subscale_slug>/
    └── <dataset_id>/
        ├── metadata.json       # scenario, dialogue, tones, speaker info
        └── audio/
            ├── 01.mp3          # Sentence 1 (context)
            ├── 02.mp3          # Sentence 2 (context)
            ├── 03.mp3          # Sentence 3 (context)
            ├── 04.mp3          # Sentence 4 — HIGH EQ delivery
            ├── 05.mp3          # Sentence 4 — LOW EQ delivery
            ├── 06.mp3          # Sentence 5 (bridge)
            ├── 07.mp3          # Sentence 6 — HIGH EQ delivery
            └── 08.mp3          # Sentence 6 — LOW EQ delivery
```

### Convert to HuggingFace Format

```bash
python convert_to_hf.py --data-dir ./data --output-dir ./hf_dataset

# Upload to HuggingFace Hub
python validate_and_reupload_hf_dataset.py --dataset-path ./hf_dataset
```

---

## Evaluation

Evaluate any supported model on the benchmark. The evaluator runs a two-round forced-choice MCQ per example.

```bash
# GPT-4o Audio (default)
python evaluate.py --dataset-path YOUR_HF_USERNAME/SpeechEQ --from-hub

# Pin a snapshot
python evaluate.py --dataset-path ./hf_dataset --model openai --model-name gpt-4o-audio-preview-2025-06-03

# Gemini via Portkey
python evaluate.py --dataset-path ./hf_dataset --model gemini \
  --model-name "@YOUR_PORTKEY_VIRTUAL_KEY/gemini-2.5-pro"

# On-device Qwen (transformers)
python evaluate.py --dataset-path ./hf_dataset --model qwen \
  --model-name Qwen/Qwen2.5-Omni-7B

# On-device Qwen (vLLM, faster)
python evaluate.py --dataset-path ./hf_dataset --model qwen \
  --model-name Qwen/Qwen2.5-Omni-7B --use-vllm

# MiMo-Audio
python evaluate.py --dataset-path ./hf_dataset --model mimo \
  --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct --use-vllm

# Kimi-Audio
python evaluate.py --dataset-path ./hf_dataset --model kimi \
  --model-name moonshotai/Kimi-Audio-7B-Instruct

# Fun-Audio-Chat
python evaluate.py --dataset-path ./hf_dataset --model funaudio \
  --model-name FunAudioLLM/FunAudioChat-8B

# Text-only baseline (no audio)
python evaluate.py --dataset-path ./hf_dataset --evaluator text --model gemini

# Quick test on 10 examples
python evaluate.py --dataset-path ./hf_dataset --max-examples 10
```

See `run.sh` for ready-to-use evaluation commands with vLLM.

**Key arguments for `evaluate.py`:**

| Argument | Default | Description |
|---|---|---|
| `--dataset-path` | *(required)* | Local HF dataset directory or Hub repo ID |
| `--from-hub` | false | Load from HuggingFace Hub |
| `--model` | `gpt-4o-audio-preview` | Model provider (`openai`, `gemini`, `qwen`, `mimo`, `kimi`, `funaudio`) |
| `--model-name` | *(provider default)* | Override model ID |
| `--evaluator` | `mcq` | Evaluator type (`mcq`, `text`, `ser_llm`, `persona`, `multiturn-factor`) |
| `--output-dir` | `./evaluation_results` | Output directory |
| `--api-parallel-workers` | 8 | Concurrent examples for cloud API |
| `--use-vllm` | false | Use vLLM for on-device inference |
| `--max-examples` | *(all)* | Limit examples (for testing) |
| `--prompt-name` | `base` | Prompt variant to use |
| `--config` | `config.yaml` | Path to config file |

### Evaluation Protocol

For each example:
1. Model listens to sentences 1–3 (context audio)
2. **Turn 4:** selects the higher-EQ delivery from two options → records **Acc₁**
3. Model listens to the selected sentence 4 + sentence 5
4. **Turn 6:** selects the higher-EQ delivery from two options → records **Acc₂**
5. **Acc_traj** = correct on both turns

Results are saved to `evaluation_results/{dataset}_{model}.json` with per-example reasoning, turn accuracy, and per-subscale breakdown.

### Tone Evaluation

A separate task tests whether models can match the *tone* of an utterance:

```bash
python tone_evaluation.py --dataset-path ./hf_dataset --model openai
```

See `TONE_EVALUATION.md` for details.

---

## Configuration

Edit `config.yaml` to customize EQ scales, generation parameters, and model settings. Key sections:

- **`eq_scales`** — All 15 EQ-i 2.0 subscales with descriptions and scenario types
- **`generation`** — Number of sentences, speakers, tone variation positions
- **`models`** — Default model IDs and parameters for each provider
- **`evaluation`** — Validation criteria thresholds

---

## EQ-i 2.0 Framework

The benchmark covers all 15 subscales across 5 dimensions of the EQ-i 2.0 inventory:

| Scale | Subscales |
|---|---|
| Self-Perception | Self-Regard · Self-Actualization · Emotional Self-Awareness |
| Self-Expression | Emotional Expression · Assertiveness · Independence |
| Interpersonal | Interpersonal Relationships · Empathy · Social Responsibility |
| Decision Making | Problem Solving · Reality Testing · Impulse Control |
| Stress Management | Flexibility · Stress Tolerance · Optimism |

---

## Supported Models

| Category | Models |
|---|---|
| **LLM (generation)** | GPT-4o, GPT-4o-mini, Claude 3.x, Qwen-plus/turbo |
| **TTS** | OpenAI `gpt-4o-mini-tts` (11 voices), Qwen3-TTS |
| **Cloud evaluation** | GPT-4o Audio Preview, Gemini 2.5-pro (via Portkey) |
| **On-device evaluation** | Qwen2.5-Omni (3B/7B), Qwen3-Omni (30B), MiMo-Audio-7B, Kimi-Audio-7B, Fun-Audio-Chat-8B |

---

## Project Structure

```
SpeechEQ/
├── generate.py                    # Main data generation entry point
├── generate_azure.py              # Azure OpenAI variant
├── generate_text.py               # Text-only generation (no TTS)
├── generate_audio.py              # Audio generation only
├── evaluate.py                    # Main evaluation entry point
├── evaluate_stat.py               # Statistical evaluation aggregation
├── tone_evaluation.py             # Tone matching evaluation task
├── tone_filter.py                 # Post-TTS tone pair filtering
├── convert_to_hf.py               # Convert local data → HuggingFace format
├── validate_and_reupload_hf_dataset.py  # HF dataset validation & upload
├── check_api_key.py               # Verify API key configuration
├── config.yaml                    # All EQ scales, model, generation settings
├── requirements.txt               # Python dependencies
├── setup_env.sh                   # One-command environment setup
├── run.sh                         # Example evaluation commands
├── .env.example                   # Environment variable template
│
├── src/
│   ├── data_generation/
│   │   ├── generator.py           # Core generation pipeline
│   │   ├── e2e_generator.py       # End-to-end pipeline
│   │   ├── batch_generator.py     # Parallel batch generation
│   │   └── filters.py             # Acoustic distinctiveness filtering
│   ├── evaluation/
│   │   ├── mcq_evaluator.py       # Two-round MCQ evaluator
│   │   ├── batch_processor.py     # Batch evaluation with checkpointing
│   │   ├── text_evaluator.py      # Text-only evaluator
│   │   ├── ser_llm_evaluator.py   # Speech Emotion Recognition evaluator
│   │   ├── persona_evaluator.py   # Persona conditioning evaluator
│   │   ├── multiturn_factor_evaluator.py
│   │   ├── registry.py            # Evaluator registry
│   │   └── process.py             # Result aggregation
│   ├── models/
│   │   ├── llm_providers.py       # LLM abstraction (OpenAI, Anthropic, Qwen)
│   │   ├── tts_providers.py       # TTS abstraction (OpenAI, Qwen3-TTS)
│   │   ├── audio_models.py        # Audio model abstraction
│   │   ├── portkey_client.py      # Gemini via Portkey
│   │   ├── stat_cloud_api.py      # Cloud API helpers
│   │   ├── stat_qwen_vllm.py      # Qwen + vLLM backend
│   │   ├── stat_mimo_omni.py      # MiMo + vLLM
│   │   └── mimo_omni_offline.py   # MiMo offline inference
│   ├── prompts/
│   │   ├── qwen_prompts.py        # MCQ audio evaluation prompts
│   │   ├── text_prompts.py        # Text evaluation prompts
│   │   └── tone_prompts.py        # Tone matching prompts
│   └── utils/
│       ├── dataset_converter.py   # HuggingFace dataset conversion
│       ├── helpers.py             # Shared utilities
│       └── api_retry.py           # API retry logic
│
└── doc/
    ├── GENERATION.md              # Generation pipeline details
    └── EVALUATION.md              # Evaluation protocol details
```

---

## Complete Workflow

```bash
# 1. Generate dataset
python generate.py --examples-per-subscale 60

# 2. Convert to HuggingFace format
python convert_to_hf.py --data-dir ./data --output-dir ./hf_dataset

# 3. (Optional) Upload to HuggingFace Hub
python validate_and_reupload_hf_dataset.py --dataset-path ./hf_dataset

# 4. Evaluate
python evaluate.py --dataset-path ./hf_dataset

# 5. Review results
cat ./evaluation_results/*.json
```

### Quick Test

```bash
python generate.py --examples-per-subscale 2
python convert_to_hf.py --data-dir ./data --output-dir ./hf_dataset
python evaluate.py --dataset-path ./hf_dataset --max-examples 5
```