{ "cells": [ { "cell_type": "markdown", "id": "fba401c1", "metadata": {}, "source": [ "# Realtime Eval Guide" ] }, { "cell_type": "markdown", "id": "fafa08e6", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "7553621b", "metadata": {}, "source": [ "## Introduction\n", "\n", "Evals are what turn a voice demo into something people can rely on. The gap between “seems fine” and “works every day” is almost always evals.\n", "\n", "This guide shows how to evaluate voice systems by slowly building complexity: start simple (Crawl), add realism (Walk), then test multi-turn (Run). Along the way, you’ll learn to build the three things that make results robust: a dataset, graders, and an eval harness, plus a production flywheel so real failures become new tests.\n", "\n", "Teams that invest in evals can ship to production **5–10× faster** because they can see what’s failing, pinpoint why, and fix it with confidence.\n", "\n", "## Part I: Foundations\n", "\n", "### 1) Why realtime evals are hard\n", "\n", "Realtime is harder than text because you are grading a **streaming interaction** with two outputs: what the **assistant doe**s and **how it sounds**. A response can be “right” and still sound broken.\n", "\n", "#### 1.1 The 2 axes of realtime quality\n", "\n", "Text evals mostly ask if the content is right. Realtime adds a second axis: audio quality. Content and audio can fail independently, so a single score can hide real problems.\n", "\n", "**Most realtime evals can reduce to two independent axes:**\n", "\n", "1. Content quality: Did the assistant understand the user and do the right thing? Correctness, helpfulness, tool choice, tool arguments, and instruction following.\n", "\n", "2. Audio quality: Did the assistant sound acceptable? Naturalness, prosody, pronunciation, stability, and how it behaves under noise and imperfect capture.\n", "\n" ] }, { "cell_type": "markdown", "id": "5d420dfa", "metadata": {}, "source": [ "#### 1.2 Hard to debug\n", "\n", "With the **Responses API**, the mental model is simple: **request in → response out**. With the **Realtime API**, a “turn” is a s**titched pipeline**. That orchestration makes voice apps easy to build, but for evals, you must log stages so you can isolate failures and find root causes." ] }, { "cell_type": "markdown", "id": "89dd169b", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "becc23dc", "metadata": {}, "source": [ "A “turn” is a chain of events (speech start/stop → commit → response.create → audio deltas → done), and failures can happen at any stage. If you treat the system as a black box, you’ll chase “model issues” that are actually turn detection, buffering, or tool-integration issues.\n", "\n", "**Example:**\n", "\n", "- Content is correct but the experience is broken: audio gets chopped during barge-in because the interruption boundary is wrong.\n", "\n", "- Answer is “right” but feels slow: latency came from network quality, turn detection slowness, not the model’s reasoning.\n", "\n", "You can learn more about the various events that the Realtime API triggers [here](https://platform.openai.com/docs/api-reference/realtime-server-events).\n", "\n", "#### 1.3 Transcript ≠ ground truth\n", "\n", "In realtime api, the ground truth for “what the user said” is **the actual audio signal** (what the microphone captured and what the model heard). A transcript is not ground truth, it’s a **model-produced interpretation** of that audio. It can be wrong because it’s constrained by **transcription model errors**." ] }, { "cell_type": "markdown", "id": "7c9550c7", "metadata": {}, "source": [ "**If you treat transcripts as truth, your evals can be misleading:**\n", "\n", "- False fail: ASR drops a digit, but the model heard it and called the tool correctly → your LLM grader marks “wrong.”\n", "\n", "- False pass: transcript looks clean, but audio was clipped and the model guessed → you miss the real problem.\n", "\n", "**Best Practices:**\n", "\n", "- **Improve transcription:** Iterate on transcription [prompts](https://platform.openai.com/docs/guides/speech-to-text#prompting), try different [models](https://platform.openai.com/docs/guides/speech-to-text), try different methods such as [oob transcription](https://cookbook.openai.com/examples/realtime_out_of_band_transcription).\n", "\n", "- **Use transcripts for scale:** run most automated grading on **transcripts + traces**.\n", "\n", "- **Calibrate graders on messy reality:** iterate graders on **production-like, noisy transcripts** (not clean text) so they don’t overreact to ASR errors.\n", "\n", "- **Add an audio audit loop:** spot-check **~1–5%** of sessions end-to-end." ] }, { "cell_type": "markdown", "id": "2ec20772", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "3033c18a", "metadata": {}, "source": [ "## Part II: Strategy\n", "\n", "### 2) Crawl / Walk / Run\n", "\n", "Realtime evals feel overwhelming when teams start at the hardest setting: real audio, multi-turn dialogue and real tools. The fix is to build **complexity in steps**. If your system **cannot crawl, it will not run**. Early evals should be simple enough that failures are diagnosable, repeatable, and cheap to iterate on. You can increase complexity in two independent axes.\n", "\n", "#### 2.1 Isolating input conditions: clean vs production audio\n", "\n", "This axis is about what the model hears. By controlling input audio conditions, you can separate failures in model intelligence from failures in speech perception.\n", "\n", "- **Start with synthetic audio → tests intelligence:**\n", " - Use clean, synthetic repeatable audio (e.g., TTS) when you want to measure the model’s reasoning and decision-making without audio variance muddying the signal → helps isolate intent routing, tool calling, instruction following\n", "\n", "- **Move to noisy, production-like audio → tests audio perception:**\n", " - Once intelligence is stable, introduce audio that resembles production: compression, echo, far-field capture, background noise, hesitations/self-corrections. This tests whether the system still behaves correctly when the input is ambiguous, messy, or partially lost → helps measure mishearing words, robustness to acoustic variations" ] }, { "cell_type": "markdown", "id": "48a6c62b", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "daac735f", "metadata": {}, "source": [ "#### 2.2 Isolating interaction conditions: single-turn vs multi-turn\n", "\n", "This axis is about what you are evaluating: are you evaluating the next turn or the full **conversation**.\n", "\n", "- **Start single-turn → tests core competence:**\n", " - Run one request → one response when you want the cleanest signal on fundamentals: correct intent routing, correct tool choice, valid arguments, and basic instruction following. If the system can’t reliably pick the right tool or produce a valid schema here, evaluating more turns won’t help.\n", "\n", "- **Move to multi-turn → tests robustness:**\n", " - Once single-turn is stable, move to multi-turn where the system must hold goals and constraints across turns, sequence tools correctly, recover from tool failures and handle user corrections. Multi-turn shifts you from turn-level correctness to **episode-level outcomes**: did it complete the goal, how many turns did it take, and did it recover cleanly when something went wrong?" ] }, { "cell_type": "markdown", "id": "4427dd03", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "3b83d021", "metadata": {}, "source": [ "Single-turn tells you *can win the battle*; multi-turn tells you *can win the war*.\n", "\n", "#### 2.3 Eval Quadrants\n", "\n", "Use a 2x2 map for evaluation: **right** = more realistic audio, **up** = more realistic interaction. Start bottom-left, increasing difficulty one axis at a time." ] }, { "cell_type": "markdown", "id": "87a1e0a7", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "deec8bae", "metadata": {}, "source": [ "**Eval modes (increasing complexity):**\n", "\n", "1. Crawl (bottom-left): synthetic audio + single-turn\n", "\n", "2. Walk (move right): real noisy audio + single-turn\n", "\n", "3. Run (move up): synthetic audio + multi-turn simulation \n", "\n", "Top-right (real audio + full multi-turn flow) is manual eval: run end-to-end sessions the way users do in production. Keep it in the loop for the entire project lifecycle." ] }, { "cell_type": "markdown", "id": "cec946d3", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "70590579", "metadata": {}, "source": [ "**Example:**\n", "\n", "User: “Change my reservation to 7pm.”\n", "\n", "- **Crawl:** You feed deterministic TTS for “Change my reservation to 7pm,” then grade only the next assistant turn: it should route to the reservation-update tool and pass the correct time=7pm (or ask one tight clarifying question if a required identifier is missing).\n", "- **Walk:** Record a human-mic version of “Change my reservation to 7pm,” then replay the same utterance with phone-bandwidth compression and light background noise; the system should still hear “7pm” (not “7” or “7:15”) and produce the same correct tool call.\n", "- **Run:** Model simulating a user outputs “Change my reservation to 7pm,” then simulates realistic follow-ups (“It’s under Minhajul for tonight… actually make it 7:30… wait, tomorrow”) plus an injected tool error once; the agent should clarify only what’s missing, keep state consistent, recover cleanly, and end with a single correct update tool call reflecting the final expected outcome.\n", "\n", "You can find reference implementations that you can start from and adapt here [realtime eval start](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals)." ] }, { "cell_type": "markdown", "id": "52bc0ec2", "metadata": {}, "source": [ "## Part III: The three building blocks\n", "\n", "### 4) Data: building a benchmark\n", "\n", "#### 4.1 Start with a “gold” seed set (10–50)\n", "\n", "Cover the flows you cannot afford to fail: core intents, must-work tool calls, escalation and refusal behaviors. Generate quickly, then have humans review for realism and gaps.\n", "\n", "**The goal is to start, not to perfect.**\n", "\n", "#### 4.2 Build for iteration, not just volume\n", "\n", "Eval datasets exist to drive iteration, not to look big. The loop is the product: **run evals → localize failures to a specific behavior → change one thing → re-run → confirm the fix improved without regressions**. A benchmark is “good” if it makes that loop fast, repeatable, and easy to diagnose.\n", "\n", "That requires coverage, not raw count: you need to represent the actual user behaviors and the specific edge cases that cause production failures. Size alone won’t surface fragility; the right coverage will. \n", "\n", "Coverage also has to be balanced. For every behavior, include both positives (the system should do X) and negatives (the system should not do X). Without negatives, you reward shortcuts.\n", "\n", ">**Customer Example:** A team built a voice support bot and optimized hard for the “escalate_to_human” tool call. Their offline score hit 98 percent on escalation. In dogfooding, the bot started escalating for almost everything. The root cause was dataset imbalance. They had many “must escalate” cases and almost no “do not escalate” cases, so the model learned a shortcut: escalate whenever uncertain. \n", "\n", "Finally, you must precisely tag your data to enable fine-grain evaluations. These tags should provide the necessary detail to move from a general observation, like \"score dropped,\" to a specific root cause, such **as \"this intent fails under these audio conditions with this policy boundary.\"**\n", "\n", "Example of tags could be:\n", "\n", "- intent, expected outcome, audio condition, language, and expected tool call. Tagged data enables teams to run fine-grain evaluations leading to faster iteration loops." ] }, { "cell_type": "markdown", "id": "6a591cdb", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "3c7cfb13", "metadata": {}, "source": [ "#### 4.3 Expand from production failures\n", "\n", "Offline evals are how you iterate fast. They are also easy to outgrow. If you keep optimizing against a fixed benchmark, **scores can rise while real quality (reality) stalls** because users do things your dataset does not cover." ] }, { "cell_type": "markdown", "id": "68be4755", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "3b23efdf", "metadata": {}, "source": [ "The operating model is a loop: production expands the benchmark. A new failure shows up, you reproduce it, you label it, and you add it. Over time, your offline suite should grow with the product.\n", "\n", "**A simple way to manage this is three sets:**\n", "\n", "- **Regression suite:** hard cases you already fixed. Run on every prompt, model, and tool change. This is your “do not break” contract.\n", "\n", "- **Rolling discovery set:** fresh failures from production and near misses. This is where you learn what you are missing and what to prioritize next. If they trigger failure modes, promote them to **your offline dataset. Teams usually fill this by:**\n", "\n", " - Running online graders to catch failures directly, and/or\n", "\n", " - Watching proxy metrics (latency, tool error rates, escalation rate, retries) and sampling data when they drift.\n", "- **Holdout set:** a subset of the offline test which stays untouched that you run occasionally to detect benchmark overfitting. If test scores climb while holdout stays flat, you are training for the test." ] }, { "cell_type": "markdown", "id": "785fa85a", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "996ceb4e", "metadata": {}, "source": [ "### 5) Graders\n", "\n", "Graders are your **measurement instruments**. They turn a messy, real-time voice session into **signals you can trust**.\n", "\n", "#### 5.1 Manual review (highest leverage)\n", "\n", "Manual review = listen to real audio + read full traces end-to-end. It’s the fastest way to build product intuition and catch the failures users notice instantly. Automated evals tell you what you can measure. Manual review tells you what you should be measuring.\n", "\n", "**What automation routinely underweights (but users feel immediately):**\n", "\n", "- Turn-taking failures: awkward gaps, double-talk, model cutting the user off.\n", "\n", "- Pacing & prosody: model speech is too fast/slow, rambling, flat, jittery, “robot polite.”\n", "\n", "- Transcript mismatch: ASR lag/drops/normalization → you end up grading the wrong thing.\n", "\n", "- Eval-system bugs: missing coverage in the golden set, mislabeled expectations, graders that are systematically too strict/lenient. \n", "\n", "> **Customer Example:** one large company had execs spend **~3 hours/day** just listening to sessions and scanning traces. They surfaced “hidden” issues, early cutoffs, phantom interruptions, awkward prosody, that would’ve sailed past offline evals.\n", "\n", "#### 5.2 Automated graders\n", "\n", "Humans don’t scale. Without automation, regressions slip through and “improvements” turn into vibes.\n", "\n", "**Use a layered grader stack:**" ] }, { "cell_type": "markdown", "id": "0157c176", "metadata": {}, "source": [ "1. **Deterministic graders** for anything objective and machine-checkable. They’re fast, cheap, and stable, perfect for tight iteration loops and regression gates (tool calling, JSON validity, string and pattern checks).\n", "\n", "2. **LLM graders** help you measure the things that matter but don’t fit neatly into deterministic rules: correctness, instruction following, whether a clarification was appropriate, completeness, and helpfulness.\n", "\n", "3. **Audio graders** because users experience the voice, not the transcript. Audio is still the hardest to judge reliably, so don’t wait for a single perfect scorer, start with simple, measurable checks (silence, overlap, interruption handling) and layer richer rubrics over time." ] }, { "cell_type": "markdown", "id": "28efb845", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "ee3252bf", "metadata": {}, "source": [ "### 6) Eval Harness\n", "\n", "A realtime eval is only as trustworthy as the harness that runs it. A good harness has one job: **make runs comparable**. If the same input can’t be replayed under the same settings and produce similar outcomes, it makes it hard to measure and iterate.\n", "\n", "#### [6.1 Start with single-turn replay (the “Crawl” harness)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/crawl_harness)\n", "\n", "Start here. Single-turn replay gives the fastest, cleanest signal because you can keep almost everything fixed. Keep the exact audio bytes, preprocessing, VAD configuration, codec, and chunking strategy identical across runs.\n", "\n", "In practice, it’s often best to start with voice activity detection (VAD) turned off so you remove one major source of variance. With VAD off, you decide exactly when a user turn ends.\n", "\n", "**A simple single-turn harness looks like:**" ] }, { "cell_type": "markdown", "id": "65abb0c8", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "0bd80a2d", "metadata": {}, "source": [ "**More explicitly (in Realtime API terms):**\n", "\n", "1. Generate or load input audio\n", "\n", " - If the datapoint is text, generate TTS audio.\n", " \n", " - Often, starting with text → TTS → audio is the best first step because it enables much faster iteration. It’s easier to tweak and refine the eval when you can iterate on text quickly.\n", "\n", "2. Stream audio into the input buffer\n", "\n", " - Send audio in fixed-size chunks (for example: consistent frame size per chunk).\n", "\n", " - Important: chunking and timing affect behavior. Pick a standard and stick to it. For example, 20 ms per chunk is a good balance of responsiveness and overhead.\n", "\n", "3. Commit the user audio\n", "\n", " - (Recommended) With VAD off: commit immediately after the last audio chunk.\n", "\n", " - With VAD on: the server detects turns boundaries.\n", "\n", "4. Trigger the assistant response\n", "\n", " - With VAD off: Call response.create to start generation.\n", "\n", " - With VAD on: It is automatic.\n", "\n", "5. Collect outputs\n", "\n", " - Output audio chunks (streaming deltas)\n", "\n", " - Output transcript (if enabled)\n", "\n", " - Tool calls / tool arguments (if any)\n", "\n", " - Final completion event\n", "\n", "6. Grade and persist\n", " - Run graders\n", "\n", " - Save results" ] }, { "cell_type": "markdown", "id": "56bbacd7", "metadata": {}, "source": [ "#### [6.2 Replaying saved audio (the “Walk” harness)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/walk_harness)\n", "\n", "When you move from synthetic TTS to real recordings, the harness changes in one important way: **you are streaming audio buffers from saved realistic audio.**\n", "\n", "**For saved audio, the flow becomes:**" ] }, { "cell_type": "markdown", "id": "4eb150f0", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "ca3f4dbc", "metadata": {}, "source": [ "**How to make the evals realistic in practice:**\n", "\n", "- **Preprocessing must match production**\n", " - Same resampling, normalization, channel handling, noise suppression (if used), and encoding.\n", "\n", " - Store preprocessing config alongside results so you can explain score changes.\n", "\n", "- **Streaming policy must be explicit**\n", "\n", " - If you care about latency: send chunks on a fixed cadence (e.g., “every 20ms, send 20ms of audio”).\n", "\n", " - If you only care about iteration speed: you can stream faster, but keep chunk size constant.\n", "\n", "- **Turn boundaries must be repeatable**\n", "\n", " - Prefer VAD off + manual commit for offline reproducibility.\n", "\n", " - If you must use VAD on (to match production), log VAD settings and track boundary events so you can debug failures.\n", "\n", "#### [6.3 Model-simulated multi-turn (the “Run” harness)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/run_harness)\n", "\n", "Model-simulated multi-turn uses a **user simulator** to generate the next user turn for a full conversation. It can increase coverage of scenarios, but only if episodes stay comparable across runs.\n", "\n", "**Common loop:**" ] }, { "cell_type": "markdown", "id": "ea242990", "metadata": {}, "source": [ "

\n", "\n", "

\n" ] }, { "cell_type": "markdown", "id": "7acb75df", "metadata": {}, "source": [ "**Best practice for simulations:**\n", "- **Pin and version the simulator prompt:** Treat it like code. A small prompt edit can shift behavior more than a model change.\n", "\n", "- **Constrain randomness:** Fix temperature and sampling settings. Use a seed if available. Use deterministic turns where it makes sense (i.e User greetings).\n", "\n", "- **Mock tools deterministically:** Define expected tool output mocks for the scenario and return those exact outputs when the assistant calls tools. This keeps the environment stable and makes runs comparable.\n", "\n", "- **Record the full trajectory:** Store every generated user text turn plus the final audio bytes you streamed. Persist tool calls, tool returns, and timestamps. Simulation is a discovery engine. When it finds a real failure mode, you backfill it into a deterministic scripted episode for the crawl or walk method." ] }, { "cell_type": "markdown", "id": "749dc2cd", "metadata": {}, "source": [ "## Part IV: Case study\n", "\n", "#### 7.1 Customer support voice bot\n", "\n", "**Product goal and constraints**\n", "\n", "Resolve common support requests through tools, quickly and safely. The bot must collect the right details, call the right backend actions, and comply with policy. It must escalate cleanly when it cannot help. It must handle frustrated callers without becoming verbose or brittle.\n", "\n", "**Crawl, Walk, Run plan**\n", "\n", "**Crawl: synthetic + single-turn**\n", "\n", "Focus on routing and policy. Given a short request, the bot should pick the right intent, request missing info, and avoid unsafe actions. Use deterministic synthetic audio so you can rapidly iterate on tool schemas and prompts.\n", "\n", "**Walk: real + single-turn**\n", "\n", "Test understanding under realistic capture. Use synthetic or real recordings in noisy environments and telephony-like quality. This is where order numbers, names, and addresses break on noisy audio. Evaluate whether the bot asks clarifying questions instead of guessing.\n", "\n", "**Run: synthetic + multi-turn simulations**\n", "\n", "Simulate full workflows with simulated users with gpt-realtime and tool mocks: authentication, account lookup, order status, return eligibility, refund, ticket creation, escalation. Add adversarial but realistic patterns: caller changes goal midstream, provides partial info, talks over the assistant, or answers a different question than asked.\n", "\n", "**Manual Review:**\n", "\n", "Run internal call sessions against staging systems. This catches UX failures that graders miss: overlong disclaimers, repetitive questions, poor turn-taking during authentication.\n", "\n", "**Core dataset buckets and useful slices**\n", "\n", "- Top intents: order status, return, refund, cancel, billing issue, password reset, appointment scheduling.\n", "\n", "- Missing and conflicting info: wrong order number, two accounts, caller provides a nickname, caller refuses to authenticate.\n", "\n", "- Policy edges: out-of-window returns, restricted items, partial refunds, subscription cancellation rules.\n", "\n", "- Escalation triggers: the bot should hand off when confidence is low or tools fail.\n", "\n", "- Emotional tone: angry, rushed, confused. The content goal stays the same, but delivery matters.\n", "\n", "**Graders used**\n", "\n", "- Deterministic: tool selection, tool argument validity, policy phrases if required.\n", "\n", "- LLM rubric grader: instruction following, resolution correctness, empathetic tone, whether it avoided hallucinating policy, whether it escalated appropriately, and whether it stayed concise.\n", "\n", "- Audio grader: long silences, interruption handling." ] } ], "metadata": { "kernelspec": { "display_name": "openai", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.9" } }, "nbformat": 4, "nbformat_minor": 5 }