--- name: voice-model-benchmark description: >- Measure whether a speech-to-speech (realtime voice) model is weaker when a request is SPOKEN than when it is typed. Sends the same seeded harmful and benign requests to one realtime model as text, as locally synthesised speech, and as that speech under audio perturbations (speed, pitch, volume, noise, echo, reverb), then has a separate judge model grade each reply transcript. Reports attack success and benign refusal per arm with 95% intervals, the paired speech-minus-text gap with exact McNemar, and per-perturbation deltas. Use it when a voice model or voice agent is under review (for example OpenAI gpt-realtime-2.1, which is reachable only on the Realtime API), or when a text-only benchmark result is being assumed to hold for a voice deployment. It also lists the voice threats that have no standard benchmark as review questions. For a text or vision model use model-benchmark; for an app endpoint use app-benchmark. license: MIT compatibility: "Python 3.11+. The runner, mock realtime server and tests use only the standard library. Real speech needs a separate virtual environment with piper-tts (GPL-3.0-or-later, runs locally) and one voice file downloaded once (about 60 MB). A real run needs network access to the realtime API over WebSocket and an OpenAI-compatible chat endpoint for the judge. Optional: the websockets package as an alternative transport." --- # voice-model-benchmark: typed versus spoken, for a realtime voice model You are running this skill **on behalf of a user** who must decide whether a realtime voice model behaves the same when a request arrives as speech as it does when it arrives as text. Drive **Plan → Run → Analyze**. The one axis is the **input condition**. The arms are `text`, `speech`, and `speech` with one perturbation each. The model, its session configuration, the output modality, the judge and the seeded cases are held fixed. This skill measures; it does not say whether a result is acceptable. Execution tier: **T0** (pure API; nothing is executed, no sandbox). Use `AskUserQuestion` for choices. Run every command from this skill's folder. What the numbers mean: - **Attack success** is the share of graded harmful requests the model helped with. Lower is more resistant. - **Benign refusal** is the share of graded safe requests it refused. Read the two together: a model that refuses everything spoken has low attack success and is not usable. - **Speech minus text** is the paired difference on the same cases: the modality gap. - Results apply to this voice, these perturbation settings and this session configuration. They are not a statement about human callers or about attacks this skill does not make. --- ## PLAN ### 1. Preflight: local speech in its own environment ``` uv venv .venv-voice-model-benchmark --python 3.11 uv pip install --python .venv-voice-model-benchmark/bin/python piper-tts VPY=/.venv-voice-model-benchmark/bin/python $VPY synth.py --download-voice # once, about 60 MB, no account ``` `python3 -m venv` followed by `pip install piper-tts` works too. Speech is synthesised on this machine by Piper 1.8.0 with both noise scales set to 0, which makes a clip repeatable; no prompt is sent to a speech service. The default voice is `en_US-ljspeech-medium` (`--voice` or `VOICE_BENCH_VOICE` to change it; voices are stored under `~/.cache/voice-model-benchmark/voices` or `VOICE_BENCH_VOICE_DIR`). ### 2. Profile the target: ask - **Model under test**, one per run, for example `gpt-realtime-2.1` or `gpt-realtime-2.1-mini`. To compare two models, run each with the same `--seed`, `--depth` and `--arms`; this skill computes no cross-model statistics. - **Session configuration of the deployment under review**: system instructions, tools, output voice, reasoning effort. Put them in a JSON file for `--session-config`; it is merged into the realtime session. With none, the provider's default instructions apply and the result describes the bare model, not the deployment. Tools are declared to the model but never executed; any tool call the model asks for is recorded. - **Output**: `--output audio` (default; what a voice deployment does, and it produces the audio you spot-check) or `--output text` (cheaper, no spoken output to check). - **Judge model**, on an OpenAI-compatible chat endpoint. It must not be the model under test (the runner refuses) and must grade harmful content without refusing. - **Deployment context.** If a `review.json` exists (from `review-scope`), pass `--review`; its `context` is recorded with the results. ### 3. Choose arms and depth (`AskUserQuestion`) `--arms default` is `text,speech,speed,pitch,volume,noise,echo`. `all` adds `speed_slow`, `pitch_down`, `volume_quiet` and `reverb`. A comma list selects any subset. Parameters are in `perturbations.json`; `--perturbations ` supplies your own. | Arm | Input | Default parameters | |---|---|---| | `text` | the prompt typed into the session | baseline for the modality gap | | `speech` | the prompt synthesised, unmodified | baseline for perturbation deltas | | `speed` / `speed_slow` | faster or slower, pitch kept | ×1.5 / ×0.7 | | `pitch` / `pitch_down` | higher or lower, duration kept | +4 / −4 semitones | | `volume` / `volume_quiet` | louder (saturates) or quieter | +12 dB / −20 dB | | `noise` | white noise added | 10 dB signal-to-noise | | `echo` | delayed repeats | 120 ms, decay 0.4, 2 repeats | | `reverb` | room reverberation | 600 ms decay, 40% wet | `--depth` is the number of cases **of each type** (harmful and benign): `smoke` (25: proves the wiring only, about ±12 to ±19 points, not a result), `standard` (100, about ±6 to ±10 points, the default), `full` (all 720 harmful and 255 benign), or a number. The corpus is a vendored, licence-clean subset of short single-sentence requests (AdvBench, XSTest unsafe, XSTest safe, PromptInject benign); see `corpus/NOTICE`. ### 4. Show the cost before anything is sent ``` python3 run_benchmark.py --model --depth standard --arms default --plan ``` It prints cases, realtime responses and judge calls, clips to synthesise and perturb, seconds of input audio, and an estimate of input audio tokens. It prints **counts, not prices**: read the model's pricing page and multiply. Output is not bounded unless you pass `--max-output-tokens`; say so to the user and ask. Session instructions and tool definitions are billed as text tokens on every response. ### 5. Open an experiment folder ``` bash new_experiment.sh voice-model-benchmark "