{ "cells": [ { "cell_type": "markdown", "id": "sami-01", "metadata": {}, "source": "# Build a Speaker-Aware Meeting Intelligence Pipeline with Audio Diarization\n\nMany organizations already record important conversations, but a plain transcript often is not enough for reliable follow-up. The missing layer is speaker attribution: knowing who raised a concern, who made a commitment, and where the evidence appears in the recording. A speaker-aware transcript lets you separate customer needs from seller follow-up, keep structured evidence references next to action items, and route sensitive commitments into a review workflow before they land in a CRM, ticketing system, or knowledge base.\n\nThis pattern is useful whenever the downstream workflow depends on who said what:\n\n- Sales discovery and solution consulting: capture customer requirements, seller commitments, decision criteria, blockers, and next steps with evidence.\n- Customer success and account management: turn QBRs, renewal calls, and onboarding sessions into sourced risks, product asks, and follow-up plans.\n- Support escalations and incident reviews: preserve the timeline, reported symptoms, owner commitments, and unresolved questions before creating tickets or postmortems.\n- Recruiting and interview loops: summarize candidate or interviewer feedback while keeping quotes tied to the right speaker.\n- Regulated or high-stakes reviews: add redaction, evidence checks, and human review before storing notes from healthcare, financial services, legal, or compliance-heavy conversations.\n\nFor revenue teams, the impact is usually less about generating another summary and more about reducing leakage between the conversation and the system of record. A pipeline like this can help teams capture CRM-ready next steps faster, identify renewal or expansion risks earlier, preserve evidence behind forecast updates, and coach reps or support teams from sourced examples instead of anecdotal notes. The goal is not to automate judgment away; it is to make handoffs, reviews, and follow-up actions more complete and auditable.\n\nThis notebook shows how to build a production-style, post-call meeting intelligence pipeline with OpenAI audio diarization. You will:\n\n1. Accept a recorded meeting audio file.\n2. Optionally map known speakers using short reference clips.\n3. Call `gpt-4o-transcribe-diarize` with `response_format=\"diarized_json\"`.\n4. Normalize the speaker-labeled segments into JSON and Markdown with stable segment IDs.\n5. Use structured outputs to extract a meeting brief, decisions, risks, explicit questions, suggested follow-ups, action items, evidence references, and a follow-up email draft.\n6. Write reviewable artifacts and a local guardrail report.\n\nThe default cells run without an API key using a synthetic diarized transcript. Real audio calls are opt-in so the notebook is safe to review top-to-bottom.\n" }, { "cell_type": "markdown", "id": "sami-02", "metadata": {}, "source": "## Architecture\n\n![Architecture diagram](images/architecture.svg)\n\n| Layer | Responsibility | Output |\n| --- | --- | --- |\n| Audio intake | Accept a call recording and optional known-speaker clips. | `meeting.wav`, `Agent=agent.wav` |\n| Pipeline runner | Validate inputs, encode references as data URLs, call OpenAI, and write artifacts. | Run metadata and output directory |\n| Diarization | Call `gpt-4o-transcribe-diarize` with `response_format=\"diarized_json\"` and `chunking_strategy=\"auto\"`. | Speaker-labeled segments |\n| Transcript normalization | Convert API output into consistent JSON and Markdown with stable segment IDs. | `transcript_segments.json`, `speaker_labeled_transcript.md` |\n| Meeting intelligence | Extract summary, decisions, actions, risks, explicit questions, suggested follow-ups, quotes, and follow-up email with structured evidence references. | `meeting_intelligence.json`, `meeting_brief.md` |\n| Guardrails and review gate | Redact sensitive fields, verify evidence references, optionally moderate content, and route risky outputs for review. | `guardrail_report.json` |\n\nThis is intentionally request-based. The Realtime API is a better fit for live voice UX, browser capture, or telephony streaming. For durable post-call diarization, this pattern uses the Transcriptions API and then runs structured extraction over the speaker-labeled transcript.\n" }, { "cell_type": "markdown", "id": "sami-03", "metadata": {}, "source": "## Why speaker-aware transcripts matter\n\nThe first version of meeting intelligence is often \"send a transcript to a model and summarize it.\" That works for demos, but it breaks down in customer workflows because it loses who said what. A customer may state a requirement, a seller may make a commitment, and a manager may need the difference to be explicit.\n\nSpeaker-aware diarization gives the rest of the application better structure:\n\n- Action items can include the speaker who committed to them.\n- Risks can quote the exact customer concern.\n- Follow-up email drafts can avoid attributing seller commitments to the customer.\n- QA reviewers can spot-check speaker attribution by segment ID and timestamp.\n- CRM sync jobs can store mechanically verifiable evidence rather than opaque summaries.\n" }, { "cell_type": "markdown", "id": "sami-04", "metadata": {}, "source": "## Security and guardrails\n\nMeeting intelligence should be treated as a sensitive-data workflow, not just a transcription or summarization task. Raw recordings can contain customer names, commercial terms, support details, health or financial information, and internal strategy. Speaker reference clips can also be sensitive because they are tied to a person's voice. Once the pipeline turns that audio into structured outputs, those outputs may flow into CRM records, support tickets, account plans, dashboards, or review queues.\n\nSecurity and guardrails matter most when the output can influence a business process. A sales call summary may capture pricing or contractual commitments. A support escalation may include production-impacting incidents or customer credentials. A recruiting debrief may include candidate feedback. A regulated-industry meeting may contain data that needs retention, access-control, or redaction policies. For these workflows, the safest pattern is to minimize raw audio retention, redact sensitive content where appropriate, require evidence-backed outputs, and route risky or low-confidence outputs through human review before downstream writes.\n\nThe included regex redaction is intentionally illustrative: it masks basic email and phone patterns only. It is not a complete PII or DLP system. For names, addresses, account identifiers, credentials, health data, financial data, or regulated workflows, use a policy-approved PII/DLP detector and keep a human review gate before downstream writes.\n\nFor the structured extraction step, this notebook sets `store=False` on the Responses API call so the generated meeting intelligence response is not stored as application state. `store=False` is a useful request-level control, but it is not the same as enabling Zero Data Retention for an organization or project. If your workflow requires stricter retention guarantees, review OpenAI's [data controls documentation](https://platform.openai.com/docs/models/default-usage-policies-by-endpoint) and confirm the right retention configuration for your use case.\n\n| Risk | Guardrail |\n| --- | --- |\n| Recording or speaker-reference misuse | Require consent and policy approval before recording, diarization, or reference-clip use. Treat speaker references as sensitive biometric-adjacent data. |\n| Over-retention of raw audio | Do not save the raw transcription response by default. Keep raw audio and reference clips only as long as needed. Encrypt and restrict access if retained. |\n| Prompt injection inside transcripts | Treat transcript text as untrusted evidence. Keep instructions in the system message and require the model to use only transcript-backed facts. |\n| Unsupported action items or decisions | Use strict structured outputs and require evidence references that point to real segment IDs and quotes. |\n| Sensitive content in generated notes | Run redaction before summarization where possible, then run post-generation checks on the transcript and brief. |\n| Harmful or policy-sensitive content | Optionally call the Moderation API with `omni-moderation-latest` on transcript text and generated brief text. Moderation detects harmful content; it is not a replacement for privacy review. |\n| Unsafe downstream writes | Do not write directly to CRM, ticketing, or analytics systems from the model output. Put a human review gate in front of medium/high risks, missing evidence, moderation flags, or raw-response retention. |\n| Silent quality drift | Log model versions, prompt versions, schema versions, audio duration, redaction state, moderation state, and reviewer decisions. Sample calls for evals. |\n" }, { "cell_type": "markdown", "id": "sami-05", "metadata": {}, "source": [ "## Prerequisites\n", "\n", "- Python 3.10 or later.\n", "- An OpenAI API key in `OPENAI_API_KEY` for real audio runs.\n", "- A meeting recording in a supported audio format for real audio runs.\n", "- Audio uploads must be 25 MB or smaller. Supported input formats are `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `wav`, and `webm`.\n", "- Optional: up to four short, single-speaker reference clips. The speech-to-text guide recommends 2-10 second references, encoded as data URLs when sent with multipart form data.\n", "\n", "### Run the notebook locally\n", "\n", "From a local clone of the Cookbook repository, create a virtual environment, install Jupyter and the OpenAI SDK, then launch this notebook:\n", "\n", "```bash\n", "git clone https://github.com/openai/openai-cookbook.git\n", "cd openai-cookbook\n", "python3 -m venv .venv\n", "source .venv/bin/activate\n", "python -m pip install jupyter \"openai>=1.93.0\"\n", "export OPENAI_API_KEY=\"your-api-key\"\n", "jupyter notebook examples/audio/speaker_aware_meeting_intelligence/speaker_aware_meeting_intelligence.ipynb\n", "```\n", "\n", "The synthetic demo below uses only the Python standard library and does not call the API. For real audio, you can also install the OpenAI SDK from inside an existing notebook environment:\n", "\n", "```python\n", "%pip install \"openai>=1.93.0\"\n", "```\n" ] }, { "cell_type": "markdown", "id": "sami-30", "metadata": {}, "source": "## Core diarization request\n\nThe core API request is intentionally small:\n\n```python\nclient = OpenAI(timeout=30 * 60)\n\nwith open(\"meeting.wav\", \"rb\") as audio_file:\n stream = client.audio.transcriptions.create(\n model=\"gpt-4o-transcribe-diarize\",\n file=audio_file,\n response_format=\"diarized_json\",\n chunking_strategy=\"auto\",\n stream=True,\n extra_body={\n \"known_speaker_names\": [\"Agent\"],\n \"known_speaker_references\": [to_data_url(Path(\"agent_reference.wav\"))],\n },\n )\n for event in stream:\n if event.type == \"transcript.text.segment\":\n print(event.speaker, event.text, event.start, event.end)\n```\n\nThe important details are:\n\n- Use `response_format=\"diarized_json\"` when you need segment-level speaker metadata.\n- Use `chunking_strategy=\"auto\"` for audio longer than 30 seconds.\n- Use `stream=True` for completed recordings when you want finalized diarized segments as they become available.\n- The Python SDK defaults to a 10-minute read timeout; the helper below uses 30 minutes for longer recordings.\n- Pass known speaker names and references together, in the same order.\n- Keep reference clips short and single-speaker.\n" }, { "cell_type": "markdown", "id": "sami-30a", "metadata": {}, "source": [ "## Diarization vs speaker identification\n", "\n", "Diarization answers \"which voice spoke each segment?\" It separates voices inside one recording, but it does not create a permanent identity profile or remember that `speaker_0` from one call is the same person as `speaker_0` in a later call. Without references, generic labels are still useful because they preserve attribution: the pipeline can distinguish the speaker who raised a requirement from the speaker who made a commitment.\n", "\n", "Known-speaker references add an optional identity hint for the current request:\n", "\n", "| Input | Result |\n", "| --- | --- |\n", "| Meeting audio only | The model separates voices, usually with generic labels such as `speaker_0` and `speaker_1`. |\n", "| Meeting audio plus a named reference clip | Matching segments can use the supplied name; unmatched speakers can remain generic. |\n", "| A later or historical recording | Pass the reference clip again. Labels do not carry across recordings automatically. |\n", "\n", "### Pass a reference clip with the meeting recording\n", "\n", "The meeting recording and the reference clip are separate inputs in one transcription request. Do not concatenate the reference clip onto the meeting audio. The meeting is uploaded as `file=...`; each reference clip is encoded as a data URL and sent through `known_speaker_references` with a name in the same position in `known_speaker_names`.\n", "\n", "The helper below wraps that request shape. For example, this passes a meeting recording plus a separate short clip of an internal rep speaking:\n", "\n", "```python\n", "meeting_audio = Path(\"customer_call.wav\")\n", "known_speakers = [\n", " (\"Internal rep\", Path(\"internal_rep_reference.wav\")),\n", "]\n", "\n", "raw_transcription = transcribe_with_diarization(\n", " audio_file=meeting_audio,\n", " known_speakers=known_speakers,\n", ")\n", "```\n", "\n", "Use a clean, consented 2-10 second clip with one speaker and minimal background noise. For recurring internal speakers, a production application can keep an access-controlled reference registry and attach the appropriate clip on each request. For historical recordings, run the same flow per recording; a reference clip may come from an older consented call if it is clean and single-speaker. Treat references as sensitive data, evaluate match quality on representative audio, and keep human review for high-stakes downstream writes.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "sami-06", "metadata": {}, "outputs": [], "source": "from __future__ import annotations\n\nimport base64\nimport json\nimport mimetypes\nimport os\nimport re\nimport tempfile\nfrom dataclasses import asdict, dataclass\nfrom pathlib import Path\nfrom typing import Any\n\ntry:\n from IPython.display import JSON, Markdown, display\nexcept ImportError: # Makes this cell safe in non-notebook runners.\n JSON = None\n Markdown = None\n\n def display(value):\n print(value)\n\n\ndef show_markdown(text: str) -> None:\n if Markdown:\n display(Markdown(text))\n else:\n print(text)\n\n\ndef show_json(payload: Any, expanded: bool = False) -> None:\n if JSON:\n display(JSON(payload, expanded=expanded))\n else:\n print(json.dumps(payload, indent=2))\n\n\nDEFAULT_TRANSCRIPTION_MODEL = \"gpt-4o-transcribe-diarize\"\nDEFAULT_SUMMARY_MODEL = os.getenv(\"OPENAI_MEETING_INTELLIGENCE_MODEL\", \"gpt-4.1-mini\")\nDEFAULT_MODERATION_MODEL = \"omni-moderation-latest\"\nSUPPORTED_REFERENCE_MIME_PREFIXES = (\"audio/\", \"video/\")\nMAX_AUDIO_UPLOAD_BYTES = 25_000_000\nDEFAULT_TRANSCRIPTION_TIMEOUT_SECONDS = 30 * 60\n\nprint(\"Notebook helpers loaded\")\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-07", "metadata": {}, "outputs": [], "source": "@dataclass(frozen=True)\nclass Segment:\n segment_id: str\n speaker: str\n start: float\n end: float\n text: str\n\n\nDEMO_SEGMENTS = [\n Segment(\n segment_id=\"seg_001\",\n speaker=\"Solutions Engineer\",\n start=0.0,\n end=9.2,\n text=\"Thanks for joining. I would like to understand where your support handoff breaks down today.\",\n ),\n Segment(\n segment_id=\"seg_002\",\n speaker=\"Customer\",\n start=9.3,\n end=22.4,\n text=\"The biggest issue is that escalation notes are inconsistent. Managers spend Monday morning reconstructing what happened from call recordings.\",\n ),\n Segment(\n segment_id=\"seg_003\",\n speaker=\"Solutions Engineer\",\n start=22.5,\n end=38.1,\n text=\"So the priority is reliable call summaries, who committed to what, and enough evidence that the team trusts the handoff.\",\n ),\n Segment(\n segment_id=\"seg_004\",\n speaker=\"Customer\",\n start=38.2,\n end=55.0,\n text=\"Exactly. We also need risks called out, especially compliance-sensitive promises, and we need to push action items into our CRM.\",\n ),\n Segment(\n segment_id=\"seg_005\",\n speaker=\"Solutions Engineer\",\n start=55.1,\n end=70.3,\n text=\"I will send a prototype that includes speaker-aware transcripts, action items with evidence, and a redaction pass before CRM sync.\",\n ),\n]\n\n\nPII_PATTERNS = [\n (re.compile(r\"\\b[\\w.+-]+@[\\w-]+(?:\\.[\\w-]+)+\\b\"), \"[email]\"),\n (re.compile(r\"\\b(?:\\+?1[-.\\s]?)?(?:\\(?\\d{3}\\)?[-.\\s]?)\\d{3}[-.\\s]?\\d{4}\\b\"), \"[phone]\"),\n]\n\nprint(f\"Loaded {len(DEMO_SEGMENTS)} synthetic transcript segments\")\n" }, { "cell_type": "markdown", "id": "sami-08", "metadata": {}, "source": "## Step 1: Define the structured output schema\n\nMeeting intelligence often feeds systems of record. Use strict structured outputs so downstream code gets a stable shape and unsupported fields are rejected rather than silently accepted. This schema also requires structured `evidence_refs`: each extracted item must cite a transcript `segment_id` and a quote from that segment, which lets guardrails verify the grounding mechanically.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-09", "metadata": {}, "outputs": [], "source": "EVIDENCE_REF_SCHEMA: dict[str, Any] = {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"segment_id\": {\"type\": \"string\"},\n \"quote\": {\"type\": \"string\"},\n },\n \"required\": [\"segment_id\", \"quote\"],\n}\n\nEVIDENCE_REFS_SCHEMA: dict[str, Any] = {\n \"type\": \"array\",\n \"items\": EVIDENCE_REF_SCHEMA,\n}\n\nNULLABLE_STRING_SCHEMA: dict[str, Any] = {\"type\": [\"string\", \"null\"]}\n\n\nMEETING_INTELLIGENCE_SCHEMA: dict[str, Any] = {\n \"name\": \"meeting_intelligence\",\n \"strict\": True,\n \"schema\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"summary\": {\"type\": \"string\"},\n \"participants\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"speaker\": {\"type\": \"string\"},\n \"inferred_role\": NULLABLE_STRING_SCHEMA,\n \"evidence_refs\": EVIDENCE_REFS_SCHEMA,\n },\n \"required\": [\"speaker\", \"inferred_role\", \"evidence_refs\"],\n },\n },\n \"customer_context\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"fact\": {\"type\": \"string\"},\n \"evidence_refs\": EVIDENCE_REFS_SCHEMA,\n },\n \"required\": [\"fact\", \"evidence_refs\"],\n },\n },\n \"decisions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"decision\": {\"type\": \"string\"},\n \"speaker_or_group\": NULLABLE_STRING_SCHEMA,\n \"evidence_refs\": EVIDENCE_REFS_SCHEMA,\n },\n \"required\": [\"decision\", \"speaker_or_group\", \"evidence_refs\"],\n },\n },\n \"action_items\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"owner_speaker\": NULLABLE_STRING_SCHEMA,\n \"task\": {\"type\": \"string\"},\n \"due_date_or_trigger\": NULLABLE_STRING_SCHEMA,\n \"evidence_refs\": EVIDENCE_REFS_SCHEMA,\n },\n \"required\": [\"owner_speaker\", \"task\", \"due_date_or_trigger\", \"evidence_refs\"],\n },\n },\n \"risks\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"risk\": {\"type\": \"string\"},\n \"severity\": {\"type\": \"string\", \"enum\": [\"low\", \"medium\", \"high\"]},\n \"evidence_refs\": EVIDENCE_REFS_SCHEMA,\n \"mitigation\": {\"type\": \"string\"},\n },\n \"required\": [\"risk\", \"severity\", \"evidence_refs\", \"mitigation\"],\n },\n },\n \"explicit_questions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"question\": {\"type\": \"string\"},\n \"asked_by_speaker\": {\"type\": \"string\"},\n \"directed_to_speaker\": NULLABLE_STRING_SCHEMA,\n \"evidence_refs\": EVIDENCE_REFS_SCHEMA,\n },\n \"required\": [\"question\", \"asked_by_speaker\", \"directed_to_speaker\", \"evidence_refs\"],\n },\n },\n \"suggested_follow_ups\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"question\": {\"type\": \"string\"},\n \"rationale\": {\"type\": \"string\"},\n \"evidence_refs\": EVIDENCE_REFS_SCHEMA,\n },\n \"required\": [\"question\", \"rationale\", \"evidence_refs\"],\n },\n },\n \"notable_quotes\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"speaker\": {\"type\": \"string\"},\n \"quote\": {\"type\": \"string\"},\n \"timestamp\": {\"type\": \"string\"},\n \"segment_id\": {\"type\": \"string\"},\n },\n \"required\": [\"speaker\", \"quote\", \"timestamp\", \"segment_id\"],\n },\n },\n \"follow_up_email\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"subject\": {\"type\": \"string\"},\n \"body\": {\"type\": \"string\"},\n },\n \"required\": [\"subject\", \"body\"],\n },\n },\n \"required\": [\n \"summary\",\n \"participants\",\n \"customer_context\",\n \"decisions\",\n \"action_items\",\n \"risks\",\n \"explicit_questions\",\n \"suggested_follow_ups\",\n \"notable_quotes\",\n \"follow_up_email\",\n ],\n },\n}\n\nprint(\"Structured output schema ready\")\n" }, { "cell_type": "markdown", "id": "sami-10", "metadata": {}, "source": "## Step 2: Build audio and transcript helpers\n\nKnown-speaker references are optional. Without them, diarization can still separate speakers, but labels may be generic, such as `speaker_0` or `speaker_1`. With references, the API can map segments to the names you provide.\n\nUse short, clean reference clips with one speaker and minimal background noise. Keep reference clips only when you have consent and a clear business need.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-11", "metadata": {}, "outputs": [], "source": "def to_data_url(path) -> str:\n path = Path(path)\n mime_type, _ = mimetypes.guess_type(path)\n if mime_type is None:\n mime_type = \"audio/wav\"\n elif not mime_type.startswith(SUPPORTED_REFERENCE_MIME_PREFIXES):\n raise ValueError(f\"Reference clip must be an audio or video file, got {mime_type}: {path}\")\n encoded = base64.b64encode(path.read_bytes()).decode(\"utf-8\")\n return f\"data:{mime_type};base64,{encoded}\"\n\n\ndef transcribe_with_diarization(\n audio_file: Path,\n known_speakers: list[tuple[str, Path]],\n model: str = DEFAULT_TRANSCRIPTION_MODEL,\n request_timeout_seconds: float = DEFAULT_TRANSCRIPTION_TIMEOUT_SECONDS,\n stream_transcription: bool = True,\n) -> Any:\n if not audio_file.is_file():\n raise FileNotFoundError(f\"Audio file does not exist or is not a regular file: {audio_file}\")\n\n if request_timeout_seconds <= 0:\n raise ValueError(\"request_timeout_seconds must be positive.\")\n\n audio_size_bytes = audio_file.stat().st_size\n if audio_size_bytes > MAX_AUDIO_UPLOAD_BYTES:\n raise ValueError(\n f\"Audio file is {audio_size_bytes:,} bytes; \"\n \"the Audio Transcriptions API accepts uploads up to 25 MB. \"\n \"Compress or split the recording before retrying.\"\n )\n\n from openai import OpenAI\n\n client = OpenAI(timeout=request_timeout_seconds)\n params: dict[str, Any] = {\n \"model\": model,\n \"response_format\": \"diarized_json\",\n \"chunking_strategy\": \"auto\",\n \"stream\": stream_transcription,\n }\n\n if known_speakers:\n if len(known_speakers) > 4:\n raise ValueError(\"gpt-4o-transcribe-diarize accepts up to 4 known speaker references.\")\n params[\"extra_body\"] = {\n \"known_speaker_names\": [name for name, _ in known_speakers],\n \"known_speaker_references\": [to_data_url(path) for _, path in known_speakers],\n }\n\n with audio_file.open(\"rb\") as audio:\n response = client.audio.transcriptions.create(file=audio, **params)\n if stream_transcription:\n return collect_streamed_transcription(response)\n return response\n\n\ndef to_plain(value: Any) -> Any:\n if hasattr(value, \"model_dump\"):\n return value.model_dump()\n if isinstance(value, dict):\n return {key: to_plain(inner) for key, inner in value.items()}\n if isinstance(value, list):\n return [to_plain(item) for item in value]\n return value\n\n\ndef collect_streamed_transcription(events: Any) -> dict[str, Any]:\n segments: list[dict[str, Any]] = []\n full_text = \"\"\n usage: Any = None\n\n for event in events:\n data = to_plain(event)\n if not isinstance(data, dict):\n continue\n\n event_type = data.get(\"type\")\n if event_type == \"transcript.text.segment\":\n segments.append(data)\n elif event_type == \"transcript.text.done\":\n full_text = str(data.get(\"text\") or \"\")\n usage = data.get(\"usage\")\n\n if not segments:\n raise ValueError(\"No diarized transcript segments were emitted by the transcription stream.\")\n return {\"segments\": segments, \"text\": full_text, \"usage\": usage}\n\n\ndef normalize_segments(transcription: Any) -> list[Segment]:\n data = to_plain(transcription)\n raw_segments = data.get(\"segments\", []) if isinstance(data, dict) else []\n segments: list[Segment] = []\n\n for index, item in enumerate(raw_segments):\n if hasattr(item, \"model_dump\"):\n item = item.model_dump()\n if not isinstance(item, dict):\n continue\n\n text = str(item.get(\"text\", \"\")).strip()\n if not text:\n continue\n\n segment_id = str(item.get(\"segment_id\") or item.get(\"id\") or f\"seg_{len(segments) + 1:03d}\")\n segments.append(\n Segment(\n segment_id=segment_id,\n speaker=str(item.get(\"speaker\") or f\"Speaker {index + 1}\"),\n start=float(item.get(\"start\") or 0.0),\n end=float(item.get(\"end\") or 0.0),\n text=text,\n )\n )\n\n if not segments and isinstance(data, dict) and data.get(\"text\"):\n segments.append(Segment(segment_id=\"seg_001\", speaker=\"Speaker 1\", start=0.0, end=0.0, text=str(data[\"text\"])))\n\n if not segments:\n raise ValueError(\"No transcript segments were found in the transcription response.\")\n return segments\n\nprint(\"Audio and transcript helpers ready\")\n" }, { "cell_type": "markdown", "id": "sami-12", "metadata": {}, "source": "## Step 3: Normalize the transcript\n\nThe normalized transcript is the contract between audio processing and meeting intelligence. It helps you rerun summarization without retranscribing audio, inspect attribution quality, and keep raw audio retention short.\n\nEach segment gets a stable `segment_id` such as `seg_005`. Later, the model must cite those IDs in `evidence_refs`, and the guardrail step verifies that each cited quote appears in the referenced segment.\n\nThe regex redaction helper below is intentionally illustrative: it masks basic email and phone patterns only. It is not a complete PII or DLP system; use a policy-approved detector and human review for sensitive or regulated workflows.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-13", "metadata": {}, "outputs": [], "source": "def redact_text(text: str) -> str:\n redacted = text\n for pattern, replacement in PII_PATTERNS:\n redacted = pattern.sub(replacement, redacted)\n return redacted\n\n\ndef redact_segments(segments: list[Segment]) -> list[Segment]:\n return [\n Segment(\n segment_id=segment.segment_id,\n speaker=segment.speaker,\n start=segment.start,\n end=segment.end,\n text=redact_text(segment.text),\n )\n for segment in segments\n ]\n\n\ndef pii_matches(text: str) -> list[str]:\n matches: list[str] = []\n for pattern, replacement in PII_PATTERNS:\n if pattern.search(text):\n matches.append(replacement.strip(\"[]\"))\n return sorted(set(matches))\n\n\ndef format_timestamp(seconds: float) -> str:\n total_ms = max(0, int(round(seconds * 1000)))\n minutes, remainder_ms = divmod(total_ms, 60_000)\n secs, millis = divmod(remainder_ms, 1000)\n return f\"{minutes:02d}:{secs:02d}.{millis:03d}\"\n\n\ndef transcript_as_markdown(segments: list[Segment]) -> str:\n lines = [\"# Speaker-Labeled Transcript\", \"\"]\n for segment in segments:\n start = format_timestamp(segment.start)\n end = format_timestamp(segment.end)\n lines.append(f\"**{segment.segment_id} | {segment.speaker} [{start}-{end}]**: {segment.text}\")\n lines.append(\"\")\n return \"\\n\".join(lines).rstrip() + \"\\n\"\n\n\ndef transcript_for_model(segments: list[Segment]) -> str:\n return \"\\n\".join(\n f\"{segment.segment_id} | {segment.speaker} | {format_timestamp(segment.start)}-{format_timestamp(segment.end)} | {segment.text}\"\n for segment in segments\n )\n\n\nshow_markdown(transcript_as_markdown(DEMO_SEGMENTS))\n" }, { "cell_type": "markdown", "id": "sami-14", "metadata": {}, "source": "## Step 4: Extract structured meeting intelligence\n\nThe model gets a speaker-labeled transcript and must use only that transcript as evidence. The safest default is to produce empty arrays instead of plausible but unsupported CRM notes. The schema also uses required-but-nullable fields, such as `due_date_or_trigger`, `inferred_role`, and `directed_to_speaker`, so unknown values stay `null` instead of being filled with guesses.\n\nFor every extracted fact, action item, risk, question, or recommendation, the model returns `evidence_refs` with a `segment_id` and quote. This gives reviewers a readable source trail and gives code something concrete to validate.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-15", "metadata": {}, "outputs": [], "source": "def response_output_text_or_raise(response: Any) -> str:\n data = to_plain(response)\n status = data.get(\"status\") if isinstance(data, dict) else getattr(response, \"status\", None)\n if status and status != \"completed\":\n details = data.get(\"incomplete_details\") if isinstance(data, dict) else getattr(response, \"incomplete_details\", None)\n raise RuntimeError(f\"Responses API returned status={status!r}; incomplete_details={details!r}\")\n\n refusals: list[str] = []\n if isinstance(data, dict):\n for item in data.get(\"output\", []):\n if not isinstance(item, dict):\n continue\n for content in item.get(\"content\", []):\n if not isinstance(content, dict):\n continue\n if content.get(\"type\") == \"refusal\" or content.get(\"refusal\"):\n refusals.append(str(content.get(\"refusal\") or content.get(\"text\") or content))\n if refusals:\n raise RuntimeError(f\"Responses API returned a refusal: {refusals[0]}\")\n\n content = getattr(response, \"output_text\", None)\n if content is None and isinstance(data, dict):\n content = data.get(\"output_text\")\n content = str(content or \"\").strip()\n if not content:\n raise RuntimeError(\"The model returned an empty response.\")\n return content\n\n\ndef parse_meeting_intelligence_json(content: str) -> dict[str, Any]:\n try:\n parsed = json.loads(content)\n except json.JSONDecodeError as exc:\n raise RuntimeError(f\"Responses API returned invalid JSON: {exc}\") from exc\n if not isinstance(parsed, dict):\n raise RuntimeError(\"Responses API returned JSON, but the top-level value was not an object.\")\n return parsed\n\n\ndef generate_meeting_intelligence(segments: list[Segment], model: str = DEFAULT_SUMMARY_MODEL) -> dict[str, Any]:\n from openai import OpenAI\n\n client = OpenAI()\n transcript = transcript_for_model(segments)\n\n completion = client.responses.create(\n model=model,\n temperature=0,\n store=False,\n input=[\n {\n \"role\": \"system\",\n \"content\": (\n \"You create meeting intelligence from speaker-labeled transcripts. \"\n \"Use only the transcript as evidence. Do not invent names, dates, decisions, \"\n \"commitments, or implementation details. If evidence is missing, leave the relevant array empty. \"\n \"Use null for unknown roles, owners, due dates or triggers, directed-to speakers, or decision owners. \"\n \"Put single-speaker commitments in action_items, not decisions. \"\n \"Only include decisions when the transcript shows an explicit decision or agreement. \"\n \"Put only questions actually asked in explicit_questions. \"\n \"Put inferred next questions in suggested_follow_ups with rationale and evidence_refs. \"\n \"Every extracted item must include evidence_refs with segment_id values copied from the transcript \"\n \"and quote text copied from that same segment. Do not fabricate segment IDs or quotes. \"\n \"Use empty arrays instead of unsupported items. \"\n \"If the follow-up email signer is unknown, end with [Your name].\"\n ),\n },\n {\n \"role\": \"user\",\n \"content\": (\n \"Extract a customer-safe meeting brief from this transcript. \"\n \"Transcript rows use: segment_id | speaker | timestamp range | text.\\n\\n\"\n f\"{transcript}\"\n ),\n },\n ],\n text={\n \"format\": {\n \"type\": \"json_schema\",\n \"name\": MEETING_INTELLIGENCE_SCHEMA[\"name\"],\n \"strict\": MEETING_INTELLIGENCE_SCHEMA[\"strict\"],\n \"schema\": MEETING_INTELLIGENCE_SCHEMA[\"schema\"],\n }\n },\n )\n\n content = response_output_text_or_raise(completion)\n return parse_meeting_intelligence_json(content)\n\n\ndef demo_meeting_intelligence() -> dict[str, Any]:\n return {\n \"summary\": (\n \"The customer needs a dependable post-call handoff process. Their main pain point is \"\n \"inconsistent escalation notes, which forces managers to reconstruct calls manually. \"\n \"The proposed path is a speaker-aware transcript, evidence-backed action items, risk \"\n \"detection, redaction, and CRM sync.\"\n ),\n \"participants\": [\n {\n \"speaker\": \"Solutions Engineer\",\n \"inferred_role\": \"OpenAI technical seller or solution owner\",\n \"evidence_refs\": [\n {\n \"segment_id\": \"seg_001\",\n \"quote\": \"I would like to understand where your support handoff breaks down today.\",\n },\n {\n \"segment_id\": \"seg_005\",\n \"quote\": \"I will send a prototype that includes speaker-aware transcripts, action items with evidence, and a redaction pass before CRM sync.\",\n },\n ],\n },\n {\n \"speaker\": \"Customer\",\n \"inferred_role\": \"Customer stakeholder for support operations\",\n \"evidence_refs\": [\n {\n \"segment_id\": \"seg_002\",\n \"quote\": \"The biggest issue is that escalation notes are inconsistent.\",\n },\n {\n \"segment_id\": \"seg_004\",\n \"quote\": \"we need to push action items into our CRM.\",\n },\n ],\n },\n ],\n \"customer_context\": [\n {\n \"fact\": \"Escalation notes are inconsistent today.\",\n \"evidence_refs\": [\n {\n \"segment_id\": \"seg_002\",\n \"quote\": \"The biggest issue is that escalation notes are inconsistent.\",\n }\n ],\n },\n {\n \"fact\": \"Managers spend time reconstructing calls from recordings.\",\n \"evidence_refs\": [\n {\n \"segment_id\": \"seg_002\",\n \"quote\": \"Managers spend Monday morning reconstructing what happened from call recordings.\",\n }\n ],\n },\n {\n \"fact\": \"The customer wants action items pushed into their CRM.\",\n \"evidence_refs\": [{\"segment_id\": \"seg_004\", \"quote\": \"we need to push action items into our CRM.\"}],\n },\n ],\n \"decisions\": [],\n \"action_items\": [\n {\n \"owner_speaker\": \"Solutions Engineer\",\n \"task\": \"Send a prototype that includes speaker-aware transcripts, action items with evidence, and a redaction pass before CRM sync.\",\n \"due_date_or_trigger\": None,\n \"evidence_refs\": [\n {\n \"segment_id\": \"seg_005\",\n \"quote\": \"I will send a prototype that includes speaker-aware transcripts, action items with evidence, and a redaction pass before CRM sync.\",\n }\n ],\n }\n ],\n \"risks\": [\n {\n \"risk\": \"Compliance-sensitive promises need to be identified in meeting notes.\",\n \"severity\": \"medium\",\n \"evidence_refs\": [\n {\n \"segment_id\": \"seg_004\",\n \"quote\": \"We also need risks called out, especially compliance-sensitive promises\",\n }\n ],\n \"mitigation\": \"Route compliance-sensitive risks to human review before CRM sync.\",\n }\n ],\n \"explicit_questions\": [\n {\n \"question\": \"Where does your support handoff break down today?\",\n \"asked_by_speaker\": \"Solutions Engineer\",\n \"directed_to_speaker\": \"Customer\",\n \"evidence_refs\": [\n {\n \"segment_id\": \"seg_001\",\n \"quote\": \"I would like to understand where your support handoff breaks down today.\",\n }\n ],\n }\n ],\n \"suggested_follow_ups\": [\n {\n \"question\": \"Which CRM object and fields should receive action items?\",\n \"rationale\": \"The customer asked to push action items into their CRM but did not specify the target schema or workflow.\",\n \"evidence_refs\": [{\"segment_id\": \"seg_004\", \"quote\": \"we need to push action items into our CRM.\"}],\n }\n ],\n \"notable_quotes\": [\n {\n \"speaker\": \"Customer\",\n \"quote\": \"Managers spend Monday morning reconstructing what happened from call recordings.\",\n \"timestamp\": \"00:09.300\",\n \"segment_id\": \"seg_002\",\n }\n ],\n \"follow_up_email\": {\n \"subject\": \"Prototype for speaker-aware meeting handoffs\",\n \"body\": (\n \"Hi,\\n\\nThanks for the conversation. I heard that inconsistent escalation notes, \"\n \"evidence-backed action items, compliance-sensitive risk detection, and CRM sync \"\n \"are the core requirements. I will send a prototype with speaker-aware transcripts, \"\n \"action items with evidence, and a redaction pass before CRM sync.\\n\\nBest,\\n[Your name]\"\n ),\n },\n }\n\n\nshow_json(demo_meeting_intelligence(), expanded=False)\n" }, { "cell_type": "markdown", "id": "sami-16", "metadata": {}, "source": "## Step 5: Render a reviewable meeting brief\n\nThe review artifact keeps speaker, segment ID, timestamp, and quote evidence next to decisions, risks, and action items so humans can spot-check before anything is written downstream.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-17", "metadata": {}, "outputs": [], "source": "def clean_markdown_cell(value: Any) -> str:\n if value is None:\n return \"_Not specified._\"\n return str(value).replace(\"|\", \"\\\\|\").replace(\"\\n\", \"
\")\n\n\ndef render_evidence_refs(refs: Any) -> str:\n if not isinstance(refs, list) or not refs:\n return \"_No evidence refs._\"\n\n rendered = []\n for ref in refs:\n if not isinstance(ref, dict):\n continue\n segment_id = clean_markdown_cell(ref.get(\"segment_id\", \"\"))\n quote = clean_markdown_cell(ref.get(\"quote\", \"\"))\n rendered.append(f\"`{segment_id}`: {quote}\")\n return \"
\".join(rendered) if rendered else \"_No evidence refs._\"\n\n\ndef markdown_table(rows: list[dict[str, Any]], columns: list[tuple[str, Any]]) -> str:\n if not rows:\n return \"_None identified._\"\n\n header = \"| \" + \" | \".join(title for title, _ in columns) + \" |\"\n divider = \"| \" + \" | \".join(\"---\" for _ in columns) + \" |\"\n body = []\n for row in rows:\n values = []\n for _, key in columns:\n value = key(row) if callable(key) else row.get(key, \"\")\n values.append(clean_markdown_cell(value))\n body.append(\"| \" + \" | \".join(values) + \" |\")\n return \"\\n\".join([header, divider, *body])\n\n\ndef render_meeting_brief(intelligence: dict[str, Any]) -> str:\n follow_up = intelligence.get(\"follow_up_email\", {})\n evidence_column = (\"Evidence\", lambda row: render_evidence_refs(row.get(\"evidence_refs\", [])))\n lines = [\n \"# Meeting Brief\",\n \"\",\n \"## Summary\",\n \"\",\n str(intelligence.get(\"summary\", \"\")).strip() or \"_No summary generated._\",\n \"\",\n \"## Participants\",\n \"\",\n markdown_table(\n intelligence.get(\"participants\", []),\n [(\"Speaker\", \"speaker\"), (\"Inferred role\", \"inferred_role\"), evidence_column],\n ),\n \"\",\n \"## Customer Context\",\n \"\",\n markdown_table(intelligence.get(\"customer_context\", []), [(\"Fact\", \"fact\"), evidence_column]),\n \"\",\n \"## Decisions\",\n \"\",\n markdown_table(\n intelligence.get(\"decisions\", []),\n [(\"Decision\", \"decision\"), (\"Owner\", \"speaker_or_group\"), evidence_column],\n ),\n \"\",\n \"## Action Items\",\n \"\",\n markdown_table(\n intelligence.get(\"action_items\", []),\n [\n (\"Owner\", \"owner_speaker\"),\n (\"Task\", \"task\"),\n (\"Due date or trigger\", \"due_date_or_trigger\"),\n evidence_column,\n ],\n ),\n \"\",\n \"## Risks\",\n \"\",\n markdown_table(\n intelligence.get(\"risks\", []),\n [(\"Risk\", \"risk\"), (\"Severity\", \"severity\"), evidence_column, (\"Mitigation\", \"mitigation\")],\n ),\n \"\",\n \"## Explicit Questions\",\n \"\",\n markdown_table(\n intelligence.get(\"explicit_questions\", []),\n [\n (\"Question\", \"question\"),\n (\"Asked by\", \"asked_by_speaker\"),\n (\"Directed to\", \"directed_to_speaker\"),\n evidence_column,\n ],\n ),\n \"\",\n \"## Suggested Follow-ups\",\n \"\",\n markdown_table(\n intelligence.get(\"suggested_follow_ups\", []),\n [(\"Question\", \"question\"), (\"Rationale\", \"rationale\"), evidence_column],\n ),\n \"\",\n \"## Notable Quotes\",\n \"\",\n markdown_table(\n intelligence.get(\"notable_quotes\", []),\n [(\"Speaker\", \"speaker\"), (\"Quote\", \"quote\"), (\"Timestamp\", \"timestamp\"), (\"Segment ID\", \"segment_id\")],\n ),\n \"\",\n \"## Follow-up Email Draft\",\n \"\",\n f\"**Subject:** {follow_up.get('subject', '')}\",\n \"\",\n str(follow_up.get(\"body\", \"\")).strip(),\n \"\",\n ]\n return \"\\n\".join(lines).rstrip() + \"\\n\"\n\n\ndemo_brief = render_meeting_brief(demo_meeting_intelligence())\nshow_markdown(demo_brief)\n" }, { "cell_type": "markdown", "id": "sami-18", "metadata": {}, "source": "## Step 6: Add guardrails and write artifacts\n\nThe sample writes a `guardrail_report.json` with local checks for:\n\n- normalized transcript segments;\n- basic email and phone PII patterns;\n- evidence references that point to real segment IDs and matching quotes;\n- medium/high risk outputs;\n- optional moderation flags;\n- raw transcription response storage.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-19", "metadata": {}, "outputs": [], "source": "def summarize_moderation_response(response: Any) -> dict[str, Any]:\n data = to_plain(response)\n summaries: list[dict[str, Any]] = []\n\n for result in data.get(\"results\", []) if isinstance(data, dict) else []:\n categories = result.get(\"categories\", {}) if isinstance(result, dict) else {}\n category_scores = result.get(\"category_scores\", {}) if isinstance(result, dict) else {}\n flagged_categories = sorted(key for key, value in categories.items() if bool(value))\n top_scores = dict(sorted(category_scores.items(), key=lambda item: float(item[1] or 0.0), reverse=True)[:5])\n summaries.append(\n {\n \"flagged\": bool(result.get(\"flagged\")) if isinstance(result, dict) else False,\n \"flagged_categories\": flagged_categories,\n \"top_category_scores\": top_scores,\n }\n )\n\n return {\n \"id\": data.get(\"id\") if isinstance(data, dict) else None,\n \"model\": data.get(\"model\") if isinstance(data, dict) else DEFAULT_MODERATION_MODEL,\n \"flagged\": any(item[\"flagged\"] for item in summaries),\n \"results\": summaries,\n }\n\n\ndef moderate_text(text: str, model: str = DEFAULT_MODERATION_MODEL) -> dict[str, Any]:\n from openai import OpenAI\n\n client = OpenAI()\n response = client.moderations.create(model=model, input=text)\n return summarize_moderation_response(response)\n\n\ndef iter_evidence_refs(value: Any, path: str = \"$\") -> list[tuple[str, Any]]:\n found: list[tuple[str, Any]] = []\n if isinstance(value, dict):\n for key, inner in value.items():\n next_path = f\"{path}.{key}\"\n if key == \"evidence_refs\":\n found.append((next_path, inner))\n else:\n found.extend(iter_evidence_refs(inner, next_path))\n elif isinstance(value, list):\n for index, item in enumerate(value):\n found.extend(iter_evidence_refs(item, f\"{path}[{index}]\"))\n return found\n\n\ndef normalize_for_quote_match(text: str) -> str:\n return re.sub(r\"\\s+\", \" \", text).strip().casefold()\n\n\ndef validate_evidence_refs(intelligence: dict[str, Any], segments: list[Segment]) -> list[dict[str, Any]]:\n segment_by_id = {segment.segment_id: segment for segment in segments}\n problems: list[dict[str, Any]] = []\n\n for path, refs in iter_evidence_refs(intelligence):\n if not isinstance(refs, list) or not refs:\n problems.append({\"path\": path, \"issue\": \"missing_or_empty_evidence_refs\"})\n continue\n\n for index, ref in enumerate(refs):\n ref_path = f\"{path}[{index}]\"\n if not isinstance(ref, dict):\n problems.append({\"path\": ref_path, \"issue\": \"evidence_ref_is_not_an_object\"})\n continue\n\n segment_id = str(ref.get(\"segment_id\", \"\")).strip()\n quote = str(ref.get(\"quote\", \"\")).strip()\n if not segment_id or not quote:\n problems.append({\"path\": ref_path, \"issue\": \"missing_segment_id_or_quote\", \"segment_id\": segment_id})\n continue\n\n segment = segment_by_id.get(segment_id)\n if segment is None:\n problems.append({\"path\": ref_path, \"issue\": \"unknown_segment_id\", \"segment_id\": segment_id})\n continue\n\n if normalize_for_quote_match(quote) not in normalize_for_quote_match(segment.text):\n problems.append(\n {\n \"path\": ref_path,\n \"issue\": \"quote_not_found_in_segment\",\n \"segment_id\": segment_id,\n \"quote\": quote,\n }\n )\n\n for index, quote in enumerate(intelligence.get(\"notable_quotes\", [])):\n if not isinstance(quote, dict):\n continue\n segment_id = str(quote.get(\"segment_id\", \"\")).strip()\n quote_text = str(quote.get(\"quote\", \"\")).strip()\n segment = segment_by_id.get(segment_id)\n if segment is None:\n problems.append({\"path\": f\"$.notable_quotes[{index}]\", \"issue\": \"unknown_segment_id\", \"segment_id\": segment_id})\n elif normalize_for_quote_match(quote_text) not in normalize_for_quote_match(segment.text):\n problems.append(\n {\n \"path\": f\"$.notable_quotes[{index}]\",\n \"issue\": \"quote_not_found_in_segment\",\n \"segment_id\": segment_id,\n \"quote\": quote_text,\n }\n )\n\n return problems\n\n\ndef add_guardrail_check(checks: list[dict[str, Any]], name: str, status: str, detail: str, evidence = None) -> None:\n check: dict[str, Any] = {\"name\": name, \"status\": status, \"detail\": detail}\n if evidence is not None:\n check[\"evidence\"] = evidence\n checks.append(check)\n\n\ndef build_guardrail_report(\n segments: list[Segment],\n intelligence: dict[str, Any],\n meeting_brief: str,\n redaction_enabled: bool,\n raw_saved: bool,\n moderation_results: dict[str, Any],\n) -> dict[str, Any]:\n checks: list[dict[str, Any]] = []\n transcript_text = transcript_for_model(segments)\n\n add_guardrail_check(\n checks,\n \"transcript_segments_present\",\n \"pass\" if segments else \"fail\",\n f\"Found {len(segments)} normalized transcript segments.\",\n )\n\n pii_found = pii_matches(transcript_text + \"\\n\" + meeting_brief)\n pii_detail = (\n \"Basic PII patterns remain after redaction.\"\n if redaction_enabled\n else \"Basic PII patterns were detected; run with redaction or review before storage.\"\n )\n add_guardrail_check(\n checks,\n \"basic_pii_scan\",\n \"review\" if pii_found else \"pass\",\n pii_detail if pii_found else \"No basic email or phone patterns detected.\",\n {\"matches\": pii_found, \"redaction_enabled\": redaction_enabled},\n )\n\n evidence_ref_problems = validate_evidence_refs(intelligence, segments)\n add_guardrail_check(\n checks,\n \"evidence_refs\",\n \"review\" if evidence_ref_problems else \"pass\",\n (\n \"Some evidence references are missing, cite unknown segments, or quote text that is not present in the cited segment.\"\n if evidence_ref_problems\n else \"All evidence references point to real segments with matching quote text.\"\n ),\n {\"problem_count\": len(evidence_ref_problems), \"examples\": evidence_ref_problems[:5]},\n )\n\n risks = intelligence.get(\"risks\", [])\n review_risks = [risk for risk in risks if str(risk.get(\"severity\", \"\")).lower() in {\"medium\", \"high\"}]\n severity_counts = {\n \"low\": sum(1 for risk in risks if str(risk.get(\"severity\", \"\")).lower() == \"low\"),\n \"medium\": sum(1 for risk in risks if str(risk.get(\"severity\", \"\")).lower() == \"medium\"),\n \"high\": sum(1 for risk in risks if str(risk.get(\"severity\", \"\")).lower() == \"high\"),\n }\n add_guardrail_check(\n checks,\n \"risk_outputs\",\n \"review\" if review_risks else \"pass\",\n \"Medium or high risks should be reviewed before downstream writes.\" if review_risks else \"No medium or high risks identified.\",\n {\"severity_counts\": severity_counts},\n )\n\n moderation_flagged = [name for name, result in moderation_results.items() if isinstance(result, dict) and result.get(\"flagged\")]\n if moderation_results:\n add_guardrail_check(\n checks,\n \"moderation\",\n \"review\" if moderation_flagged else \"pass\",\n \"Moderation flagged content that should be reviewed.\" if moderation_flagged else \"Moderation did not flag transcript or brief content.\",\n {\"flagged_artifacts\": moderation_flagged},\n )\n else:\n add_guardrail_check(checks, \"moderation\", \"not_run\", \"Moderation was not requested. Use moderation for content safety classification.\")\n\n add_guardrail_check(\n checks,\n \"raw_response_storage\",\n \"review\" if raw_saved else \"pass\",\n \"Raw transcription response was saved; confirm retention and access controls.\" if raw_saved else \"Raw transcription response was not saved.\",\n )\n\n status = \"review_required\" if any(check[\"status\"] in {\"review\", \"fail\"} for check in checks) else \"pass\"\n if any(check[\"status\"] == \"fail\" for check in checks):\n status = \"fail\"\n\n return {\n \"status\": status,\n \"recommended_next_step\": \"Send artifacts to human review before downstream writes.\" if status != \"pass\" else \"Artifacts passed local guardrail checks.\",\n \"checks\": checks,\n \"moderation\": moderation_results,\n }\n\nprint(\"Guardrail helpers ready\")\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-20", "metadata": {}, "outputs": [], "source": "def write_json(path: Path, payload: Any) -> None:\n path.write_text(json.dumps(payload, indent=2, ensure_ascii=False) + \"\\n\", encoding=\"utf-8\")\n\n\ndef write_artifacts(\n output_dir: Path,\n segments: list[Segment],\n intelligence: dict[str, Any],\n guardrail_report: dict[str, Any],\n raw_payload = None,\n) -> None:\n output_dir.mkdir(parents=True, exist_ok=True)\n write_json(output_dir / \"transcript_segments.json\", [asdict(segment) for segment in segments])\n (output_dir / \"speaker_labeled_transcript.md\").write_text(transcript_as_markdown(segments), encoding=\"utf-8\")\n write_json(output_dir / \"meeting_intelligence.json\", intelligence)\n (output_dir / \"meeting_brief.md\").write_text(render_meeting_brief(intelligence), encoding=\"utf-8\")\n write_json(output_dir / \"guardrail_report.json\", guardrail_report)\n if raw_payload is not None:\n write_json(output_dir / \"raw_transcription_response.json\", to_plain(raw_payload))\n\n\ndef run_pipeline_from_segments(\n segments: list[Segment],\n output_dir: Path,\n intelligence = None,\n redaction_enabled: bool = False,\n moderation_results = None,\n raw_saved: bool = False,\n raw_payload = None,\n) -> dict[str, Any]:\n if redaction_enabled:\n segments = redact_segments(segments)\n if intelligence is None:\n intelligence = generate_meeting_intelligence(segments)\n meeting_brief = render_meeting_brief(intelligence)\n guardrail_report = build_guardrail_report(\n segments=segments,\n intelligence=intelligence,\n meeting_brief=meeting_brief,\n redaction_enabled=redaction_enabled,\n raw_saved=raw_saved,\n moderation_results=moderation_results or {},\n )\n write_artifacts(output_dir, segments, intelligence, guardrail_report, raw_payload=raw_payload if raw_saved else None)\n return {\n \"segments\": segments,\n \"intelligence\": intelligence,\n \"meeting_brief\": meeting_brief,\n \"guardrail_report\": guardrail_report,\n \"output_dir\": output_dir,\n }\n\nprint(\"Artifact helpers ready\")\n" }, { "cell_type": "markdown", "id": "sami-21", "metadata": {}, "source": "## Step 7: Run the deterministic demo fixture\n\nThis section is a deterministic no-network demo, not a model-quality eval. It uses a fixed synthetic diarized transcript and a fixed expected meeting-intelligence object so reviewers can run the notebook without an API key.\n\n### What this fixture checks\n\nThe fixture exercises the same artifact and guardrail path used by real audio: transcript rendering, JSON writing, Markdown brief rendering, PII redaction helpers, evidence-reference validation, nullable fields, and review routing.\n\n### What this fixture does not check\n\nIt does not measure transcription quality, diarization accuracy, or model extraction quality on new meetings. The eval sections below add deterministic scoring and an optional LLM-as-judge pattern for that layer.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-22", "metadata": {}, "outputs": [], "source": "output_dir = Path(tempfile.mkdtemp(prefix=\"meeting-intelligence-demo-\"))\ndemo_run = run_pipeline_from_segments(\n segments=DEMO_SEGMENTS,\n output_dir=output_dir,\n intelligence=demo_meeting_intelligence(),\n)\n\nprint(f\"Wrote meeting intelligence artifacts to {output_dir}\")\nfor artifact_name in [\n \"transcript_segments.json\",\n \"speaker_labeled_transcript.md\",\n \"meeting_intelligence.json\",\n \"meeting_brief.md\",\n \"guardrail_report.json\",\n]:\n artifact_path = output_dir / artifact_name\n print(f\"- {artifact_path} ({artifact_path.stat().st_size} bytes)\")\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-23", "metadata": {}, "outputs": [], "source": "segments = json.loads((output_dir / \"transcript_segments.json\").read_text())\nsegments[:2]\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-24", "metadata": {}, "outputs": [], "source": "show_markdown((output_dir / \"speaker_labeled_transcript.md\").read_text())\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-25", "metadata": {}, "outputs": [], "source": "meeting_intelligence_json = json.loads((output_dir / \"meeting_intelligence.json\").read_text())\nshow_json(meeting_intelligence_json, expanded=False)\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-26", "metadata": {}, "outputs": [], "source": "show_markdown((output_dir / \"meeting_brief.md\").read_text())\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-27", "metadata": {}, "outputs": [], "source": "guardrail_report = json.loads((output_dir / \"guardrail_report.json\").read_text())\nshow_json(guardrail_report, expanded=True)\n" }, { "cell_type": "markdown", "id": "sami-28", "metadata": {}, "source": [ "## Step 8: Run with real audio\n", "\n", "The next cell is intentionally opt-in. Set `RUN_REAL_AUDIO = True`, provide your local audio paths, and make sure `OPENAI_API_KEY` is set.\n", "\n", "The Transcriptions API accepts files up to 25 MB. `chunking_strategy=\"auto\"` segments a valid upload; it does not split an oversized file. For larger meetings, compress to a supported lower-bitrate format or split the recording into bounded files before transcription, then preserve or offset timestamps when combining results.\n", "Long recordings can also outlast the Python SDK default read timeout. The helper streams finalized diarized segments by default and keeps a 30-minute timeout as a backstop. Set `stream_transcription=False` only when you specifically need one non-streamed response; split unusually long recordings when one request is not operationally reliable.\n", "\n", "### How readers supply their files\n", "\n", "This cookbook is notebook-first; there is no separate `.py` command in the published artifact. Put the meeting recording and any optional reference clips somewhere the notebook kernel can read. In local Jupyter, that can be a folder beside the notebook or an absolute path on disk. In a hosted notebook, upload the files into the notebook session first.\n", "\n", "For example, a reader might have:\n", "\n", "```text\n", "audio/\n", " customer_call.mp3\n", " internal_rep_reference.wav\n", "```\n", "\n", "Then point the configuration variables at those files:\n", "\n", "```python\n", "AUDIO_FILE = Path(\"audio/customer_call.mp3\")\n", "KNOWN_SPEAKERS = {\n", " \"Internal rep\": Path(\"audio/internal_rep_reference.wav\"),\n", "}\n", "```\n", "\n", "`AUDIO_FILE` is the original meeting recording. `KNOWN_SPEAKERS` maps the label you want in the output to a separate 2-10 second reference clip; the helper sends the clip with the request rather than appending it to the meeting audio. In a production application, the same helper can receive a temporary file created from an upload or downloaded from object storage.\n", "\n", "For the first production-style run, keep the setup simple:\n", "\n", "- Use one meeting audio file.\n", "- Use `chunking_strategy=\"auto\"` for longer recordings.\n", "- Add known-speaker references only when you have consent and a clear business need.\n", "- Run redaction before storage.\n", "- Run moderation when harmful-content classification is part of your review policy.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "sami-29", "metadata": {}, "outputs": [], "source": "RUN_REAL_AUDIO = False\nAUDIO_FILE = Path(\"/path/to/meeting.wav\")\n# Keep each reference clip separate from AUDIO_FILE. Use a clean, consented 2-10 second sample of one speaker.\nKNOWN_SPEAKERS = {\n # \"Internal rep\": Path(\"/path/to/internal_rep_reference.wav\"),\n # \"Customer\": Path(\"/path/to/customer_reference.wav\"),\n}\n# Demonstration only: masks basic email and phone patterns, not a complete PII/DLP solution.\nREDACT_REAL_AUDIO = True\nMODERATE_REAL_AUDIO = False\nSAVE_RAW_RESPONSE = False\nREAL_OUTPUT_DIR = Path(tempfile.mkdtemp(prefix=\"meeting-intelligence-real-\"))\n\nif RUN_REAL_AUDIO:\n if not os.getenv(\"OPENAI_API_KEY\"):\n raise RuntimeError(\"Set OPENAI_API_KEY before running on real audio.\")\n if not AUDIO_FILE.is_file():\n raise FileNotFoundError(AUDIO_FILE)\n\n known_speakers = [(speaker_name, reference_path) for speaker_name, reference_path in KNOWN_SPEAKERS.items()]\n raw_transcription = transcribe_with_diarization(AUDIO_FILE, known_speakers)\n real_segments = normalize_segments(raw_transcription)\n if REDACT_REAL_AUDIO:\n real_segments = redact_segments(real_segments)\n\n moderation_results: dict[str, Any] = {}\n if MODERATE_REAL_AUDIO:\n moderation_results[\"transcript\"] = moderate_text(transcript_for_model(real_segments))\n\n real_intelligence = generate_meeting_intelligence(real_segments)\n real_brief = render_meeting_brief(real_intelligence)\n if MODERATE_REAL_AUDIO:\n moderation_results[\"meeting_brief\"] = moderate_text(real_brief)\n\n real_report = build_guardrail_report(\n segments=real_segments,\n intelligence=real_intelligence,\n meeting_brief=real_brief,\n redaction_enabled=REDACT_REAL_AUDIO,\n raw_saved=SAVE_RAW_RESPONSE,\n moderation_results=moderation_results,\n )\n write_artifacts(\n REAL_OUTPUT_DIR,\n real_segments,\n real_intelligence,\n real_report,\n raw_payload=raw_transcription if SAVE_RAW_RESPONSE else None,\n )\n print(f\"Wrote real-audio artifacts to {REAL_OUTPUT_DIR}\")\nelse:\n print(\"Skipped real audio run. Set RUN_REAL_AUDIO = True after configuring AUDIO_FILE and OPENAI_API_KEY.\")\n" }, { "cell_type": "markdown", "id": "sami-31", "metadata": {}, "source": "## Step 9: Run deterministic smoke and regression checks\n\nThese checks are deterministic and do not call the API. Treat them as a smoke test and regression suite for the notebook mechanics, not as an eval of model quality.\n\n### Smoke-test checks\n\nThe first checks confirm that the notebook writes all expected artifacts and routes medium-risk outputs to review.\n\n### Regression checks\n\nThe remaining assertions catch regressions in schema nullability, evidence references, unsupported demo claims, response edge cases, redaction, and timestamp formatting.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-32", "metadata": {}, "outputs": [], "source": "expected_files = {\n \"transcript_segments.json\",\n \"speaker_labeled_transcript.md\",\n \"meeting_intelligence.json\",\n \"meeting_brief.md\",\n \"guardrail_report.json\",\n}\nassert expected_files.issubset({path.name for path in output_dir.iterdir()})\nassert guardrail_report[\"status\"] == \"review_required\"\nassert any(check[\"name\"] == \"risk_outputs\" and check[\"status\"] == \"review\" for check in guardrail_report[\"checks\"])\nassert any(check[\"name\"] == \"evidence_refs\" and check[\"status\"] == \"pass\" for check in guardrail_report[\"checks\"])\n\naction_schema = MEETING_INTELLIGENCE_SCHEMA[\"schema\"][\"properties\"][\"action_items\"][\"items\"][\"properties\"]\nparticipant_schema = MEETING_INTELLIGENCE_SCHEMA[\"schema\"][\"properties\"][\"participants\"][\"items\"][\"properties\"]\nquestion_schema = MEETING_INTELLIGENCE_SCHEMA[\"schema\"][\"properties\"][\"explicit_questions\"][\"items\"][\"properties\"]\nassert \"null\" in action_schema[\"due_date_or_trigger\"][\"type\"]\nassert \"null\" in action_schema[\"owner_speaker\"][\"type\"]\nassert \"null\" in participant_schema[\"inferred_role\"][\"type\"]\nassert \"null\" in question_schema[\"directed_to_speaker\"][\"type\"]\n\nassert demo_run[\"segments\"][0].segment_id == \"seg_001\"\nassert demo_run[\"intelligence\"][\"decisions\"] == []\nassert demo_run[\"intelligence\"][\"action_items\"][0][\"due_date_or_trigger\"] is None\nassert demo_run[\"intelligence\"][\"action_items\"][0][\"evidence_refs\"][0][\"segment_id\"] == \"seg_005\"\nassert demo_run[\"intelligence\"][\"explicit_questions\"]\nassert demo_run[\"intelligence\"][\"suggested_follow_ups\"]\nassert validate_evidence_refs(demo_run[\"intelligence\"], demo_run[\"segments\"]) == []\nassert \"structured outputs\" not in demo_run[\"intelligence\"][\"follow_up_email\"][\"body\"].lower()\nassert \"`seg_005`\" in demo_run[\"meeting_brief\"]\nassert \"_Not specified._\" in demo_run[\"meeting_brief\"]\n\nbroken_intelligence = json.loads(json.dumps(demo_run[\"intelligence\"]))\nbroken_intelligence[\"action_items\"][0][\"evidence_refs\"] = [{\"segment_id\": \"seg_999\", \"quote\": \"I will send a prototype\"}]\nassert validate_evidence_refs(broken_intelligence, demo_run[\"segments\"])\n\nbroken_intelligence = json.loads(json.dumps(demo_run[\"intelligence\"]))\nbroken_intelligence[\"action_items\"][0][\"evidence_refs\"] = [{\"segment_id\": \"seg_005\", \"quote\": \"I will send the contract tomorrow\"}]\nassert validate_evidence_refs(broken_intelligence, demo_run[\"segments\"])\n\ntry:\n response_output_text_or_raise({\"status\": \"incomplete\", \"incomplete_details\": {\"reason\": \"max_output_tokens\"}})\n raise AssertionError(\"Expected incomplete response to raise\")\nexcept RuntimeError as exc:\n assert \"incomplete\" in str(exc)\n\ntry:\n response_output_text_or_raise({\"status\": \"completed\", \"output\": [{\"content\": [{\"type\": \"refusal\", \"refusal\": \"Cannot comply.\"}]}]})\n raise AssertionError(\"Expected refusal response to raise\")\nexcept RuntimeError as exc:\n assert \"refusal\" in str(exc).lower()\n\ntry:\n response_output_text_or_raise({\"status\": \"completed\", \"output_text\": \"\"})\n raise AssertionError(\"Expected empty response to raise\")\nexcept RuntimeError as exc:\n assert \"empty\" in str(exc).lower()\n\ntry:\n parse_meeting_intelligence_json(\"not json\")\n raise AssertionError(\"Expected invalid JSON to raise\")\nexcept RuntimeError as exc:\n assert \"invalid JSON\" in str(exc)\n\nredacted = redact_segments([\n Segment(\"seg_test\", \"Customer\", 0.0, 3.0, \"Email me at alex@example.com or call 415-555-0100.\")\n])\nassert redacted[0].segment_id == \"seg_test\"\nassert redacted[0].text == \"Email me at [email] or call [phone].\"\nassert format_timestamp(6000) == \"100:00.000\"\n\nwith tempfile.NamedTemporaryFile(suffix=\".wav\") as oversized_audio:\n oversized_audio.truncate(MAX_AUDIO_UPLOAD_BYTES + 1)\n oversized_audio.flush()\n try:\n transcribe_with_diarization(Path(oversized_audio.name), [])\n raise AssertionError(\"Expected oversized audio to raise\")\n except ValueError as exc:\n assert \"25 MB\" in str(exc)\n\nwith tempfile.NamedTemporaryFile(suffix=\".wav\") as tiny_audio:\n try:\n transcribe_with_diarization(Path(tiny_audio.name), [], request_timeout_seconds=0)\n raise AssertionError(\"Expected invalid timeout to raise\")\n except ValueError as exc:\n assert \"positive\" in str(exc)\n\nstreamed_transcription = collect_streamed_transcription([\n {\"type\": \"transcript.text.segment\", \"id\": \"seg_stream\", \"speaker\": \"A\", \"start\": 0.0, \"end\": 1.0, \"text\": \"Hello\"},\n {\"type\": \"transcript.text.done\", \"text\": \"Hello\", \"usage\": {\"total_tokens\": 1}},\n])\nstreamed_segments = normalize_segments(streamed_transcription)\nassert streamed_segments[0].segment_id == \"seg_stream\"\nassert streamed_segments[0].speaker == \"A\"\nassert streamed_transcription[\"usage\"][\"total_tokens\"] == 1\n\nprint(\"Notebook demo validation passed\")\n" }, { "cell_type": "markdown", "id": "sami-35", "metadata": {}, "source": "## Step 10: Run deterministic evals\n\nThis section runs a small deterministic eval against the labeled demo fixture. It is still not a broad production eval, but it shows how to score extraction quality with reproducible rules before adding model-graded judgments.\n\nThe scorers below measure action-item precision/recall, explicit-question precision/recall, unsupported decisions, nullable unknown fields, and evidence-reference validity. For production, replace `GOLD_EVAL_LABELS` with a larger labeled dataset and run the same scorers across every example.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-36", "metadata": {}, "outputs": [], "source": "GOLD_EVAL_LABELS: dict[str, Any] = {\n \"action_items\": [\n {\n \"owner_speaker\": \"Solutions Engineer\",\n \"task_contains\": [\"send a prototype\", \"speaker-aware transcripts\", \"redaction pass\", \"crm sync\"],\n \"due_date_or_trigger\": None,\n \"evidence_segment_ids\": [\"seg_005\"],\n }\n ],\n \"explicit_questions\": [\n {\n \"question_contains\": [\"support handoff\", \"break down\"],\n \"asked_by_speaker\": \"Solutions Engineer\",\n \"directed_to_speaker\": \"Customer\",\n \"evidence_segment_ids\": [\"seg_001\"],\n }\n ],\n \"decisions\": [],\n}\n\n\ndef normalize_for_eval(text: Any) -> str:\n return re.sub(r\"\\s+\", \" \", str(text or \"\")).strip().casefold()\n\n\ndef evidence_ref_segment_ids(item: dict[str, Any]) -> set[str]:\n return {str(ref.get(\"segment_id\", \"\")) for ref in item.get(\"evidence_refs\", []) if isinstance(ref, dict)}\n\n\ndef contains_all_fragments(text: Any, fragments: list[str]) -> bool:\n normalized = normalize_for_eval(text)\n return all(normalize_for_eval(fragment) in normalized for fragment in fragments)\n\n\ndef action_item_matches_label(item: dict[str, Any], label: dict[str, Any]) -> bool:\n return (\n item.get(\"owner_speaker\") == label.get(\"owner_speaker\")\n and item.get(\"due_date_or_trigger\") == label.get(\"due_date_or_trigger\")\n and contains_all_fragments(item.get(\"task\"), label.get(\"task_contains\", []))\n and set(label.get(\"evidence_segment_ids\", [])).issubset(evidence_ref_segment_ids(item))\n )\n\n\ndef explicit_question_matches_label(item: dict[str, Any], label: dict[str, Any]) -> bool:\n return (\n item.get(\"asked_by_speaker\") == label.get(\"asked_by_speaker\")\n and item.get(\"directed_to_speaker\") == label.get(\"directed_to_speaker\")\n and contains_all_fragments(item.get(\"question\"), label.get(\"question_contains\", []))\n and set(label.get(\"evidence_segment_ids\", [])).issubset(evidence_ref_segment_ids(item))\n )\n\n\ndef precision_recall(predicted: list[dict[str, Any]], labels: list[dict[str, Any]], matcher) -> dict[str, Any]:\n matched_label_indexes: set[int] = set()\n matched_predictions = 0\n\n for item in predicted:\n for index, label in enumerate(labels):\n if index in matched_label_indexes:\n continue\n if matcher(item, label):\n matched_label_indexes.add(index)\n matched_predictions += 1\n break\n\n precision = matched_predictions / len(predicted) if predicted else (1.0 if not labels else 0.0)\n recall = len(matched_label_indexes) / len(labels) if labels else 1.0\n return {\n \"precision\": round(precision, 3),\n \"recall\": round(recall, 3),\n \"matched_predictions\": matched_predictions,\n \"predicted_count\": len(predicted),\n \"label_count\": len(labels),\n }\n\n\ndef evidence_ref_count(intelligence: dict[str, Any]) -> int:\n return sum(len(refs) for _, refs in iter_evidence_refs(intelligence) if isinstance(refs, list)) + len(intelligence.get(\"notable_quotes\", []))\n\n\ndef run_deterministic_evals(\n intelligence: dict[str, Any],\n segments: list[Segment],\n labels: dict[str, Any],\n) -> dict[str, Any]:\n action_item_scores = precision_recall(\n intelligence.get(\"action_items\", []),\n labels.get(\"action_items\", []),\n action_item_matches_label,\n )\n explicit_question_scores = precision_recall(\n intelligence.get(\"explicit_questions\", []),\n labels.get(\"explicit_questions\", []),\n explicit_question_matches_label,\n )\n evidence_problems = validate_evidence_refs(intelligence, segments)\n total_evidence_refs = evidence_ref_count(intelligence)\n valid_evidence_ref_rate = (\n round((total_evidence_refs - len(evidence_problems)) / total_evidence_refs, 3)\n if total_evidence_refs\n else 1.0\n )\n action_items = intelligence.get(\"action_items\", [])\n nullable_due_date_rate = (\n round(sum(1 for item in action_items if item.get(\"due_date_or_trigger\") is None) / len(action_items), 3)\n if action_items\n else 1.0\n )\n unsupported_decision_count = len(intelligence.get(\"decisions\", [])) if not labels.get(\"decisions\") else 0\n\n pass_conditions = [\n action_item_scores[\"precision\"] == 1.0,\n action_item_scores[\"recall\"] == 1.0,\n explicit_question_scores[\"precision\"] == 1.0,\n explicit_question_scores[\"recall\"] == 1.0,\n valid_evidence_ref_rate == 1.0,\n nullable_due_date_rate == 1.0,\n unsupported_decision_count == 0,\n ]\n\n return {\n \"status\": \"pass\" if all(pass_conditions) else \"review_required\",\n \"action_items\": action_item_scores,\n \"explicit_questions\": explicit_question_scores,\n \"valid_evidence_ref_rate\": valid_evidence_ref_rate,\n \"evidence_ref_problem_count\": len(evidence_problems),\n \"unsupported_decision_count\": unsupported_decision_count,\n \"nullable_due_date_rate\": nullable_due_date_rate,\n }\n\n\ndeterministic_eval_report = run_deterministic_evals(\n demo_run[\"intelligence\"],\n demo_run[\"segments\"],\n GOLD_EVAL_LABELS,\n)\nshow_json(deterministic_eval_report, expanded=True)\nassert deterministic_eval_report[\"status\"] == \"pass\"\n" }, { "cell_type": "markdown", "id": "sami-37", "metadata": {}, "source": "## Step 11: Add optional LLM-as-judge evals\n\nLLM-as-judge evals are useful for grading qualities that deterministic scorers cannot fully capture, such as summary usefulness, missing follow-ups, and whether the brief would help a reviewer. Keep this optional because it calls the API and can vary by judge model. Use it alongside deterministic scorers, not instead of them.\n\nThe judge below receives the transcript, the structured output, and a rubric. It returns scores and review findings. Leave `RUN_LLM_JUDGE_EVAL = False` for the default no-network notebook run.\n" }, { "cell_type": "code", "execution_count": null, "id": "sami-38", "metadata": {}, "outputs": [], "source": "RUN_LLM_JUDGE_EVAL = False\nLLM_JUDGE_MODEL = os.getenv(\"OPENAI_MEETING_INTELLIGENCE_JUDGE_MODEL\", DEFAULT_SUMMARY_MODEL)\n\nLLM_JUDGE_SCHEMA: dict[str, Any] = {\n \"name\": \"meeting_intelligence_judge\",\n \"strict\": True,\n \"schema\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"outcome\": {\"type\": \"string\", \"enum\": [\"pass\", \"review\", \"fail\"]},\n \"overall_score\": {\"type\": \"number\"},\n \"scores\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"grounding\": {\"type\": \"number\"},\n \"action_item_correctness\": {\"type\": \"number\"},\n \"completeness\": {\"type\": \"number\"},\n \"safety_review_readiness\": {\"type\": \"number\"},\n },\n \"required\": [\"grounding\", \"action_item_correctness\", \"completeness\", \"safety_review_readiness\"],\n },\n \"findings\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": False,\n \"properties\": {\n \"area\": {\"type\": \"string\"},\n \"severity\": {\"type\": \"string\", \"enum\": [\"low\", \"medium\", \"high\"]},\n \"explanation\": {\"type\": \"string\"},\n },\n \"required\": [\"area\", \"severity\", \"explanation\"],\n },\n },\n },\n \"required\": [\"outcome\", \"overall_score\", \"scores\", \"findings\"],\n },\n}\n\n\ndef run_llm_judge_eval(segments: list[Segment], intelligence: dict[str, Any], model: str = LLM_JUDGE_MODEL) -> dict[str, Any]:\n from openai import OpenAI\n\n client = OpenAI()\n transcript = transcript_for_model(segments)\n completion = client.responses.create(\n model=model,\n temperature=0,\n store=False,\n input=[\n {\n \"role\": \"system\",\n \"content\": (\n \"You are judging a meeting-intelligence extraction. Grade only against the transcript. \"\n \"Penalize unsupported claims, missing major action items, missing customer risks, incorrect speaker attribution, \"\n \"and outputs that are not ready for human review. Return calibrated scores from 0 to 1.\"\n ),\n },\n {\n \"role\": \"user\",\n \"content\": (\n \"Transcript:\\n\"\n f\"{transcript}\\n\\n\"\n \"Meeting intelligence JSON:\\n\"\n f\"{json.dumps(intelligence, indent=2)}\"\n ),\n },\n ],\n text={\n \"format\": {\n \"type\": \"json_schema\",\n \"name\": LLM_JUDGE_SCHEMA[\"name\"],\n \"strict\": LLM_JUDGE_SCHEMA[\"strict\"],\n \"schema\": LLM_JUDGE_SCHEMA[\"schema\"],\n }\n },\n )\n return parse_meeting_intelligence_json(response_output_text_or_raise(completion))\n\n\nif RUN_LLM_JUDGE_EVAL:\n if not os.getenv(\"OPENAI_API_KEY\"):\n raise RuntimeError(\"Set OPENAI_API_KEY before running the LLM judge eval.\")\n llm_judge_report = run_llm_judge_eval(demo_run[\"segments\"], demo_run[\"intelligence\"])\n show_json(llm_judge_report, expanded=True)\nelse:\n print(\"Skipped LLM judge eval. Set RUN_LLM_JUDGE_EVAL = True after configuring OPENAI_API_KEY.\")\n" }, { "cell_type": "markdown", "id": "sami-33", "metadata": {}, "source": "## Production hardening checklist\n\nUse this checklist before turning the sample into a customer workflow:\n\n| Concern | Recommendation |\n| --- | --- |\n| Consent | Make sure call recording, diarization, and known-speaker references are permitted in your product, policy, and region. |\n| Raw audio retention | Store raw audio only as long as needed. Persist normalized transcript segments when possible. |\n| Large recordings | Reject files over 25 MB before upload. Compress or split longer meetings, then preserve timestamp offsets when combining results. |\n| PII and DLP | Treat the included email and phone regexes as illustrative only. Use a policy-approved PII/DLP detector and human review for sensitive or regulated workflows. |\n| Speaker references | Treat reference clips as sensitive data. Store minimally, encrypt at rest, and rotate/delete when no longer needed. |\n| Evidence | Require structured evidence references on decisions, risks, and action items. Validate that each reference points to a real segment ID and quote. |\n| Human review | Route high-risk summaries, compliance promises, pricing claims, or contractual terms for review. |\n| Moderation | Use the Moderation API for harmful-content classification when notes may contain unsafe content. Keep privacy and compliance checks separate. |\n| Retry behavior | Retry transient API errors with backoff. Avoid duplicating downstream CRM writes by using idempotency keys. |\n| Observability | Log model names, prompt versions, schema versions, audio duration, latency, redaction status, and reviewer decisions. |\n| Evaluation | Sample calls weekly. Track speaker attribution accuracy, action-item precision, and unsupported-claim rate. |\n" }, { "cell_type": "markdown", "id": "sami-33-evals", "metadata": {}, "source": "## Evaluation guidance for production\n\nThe deterministic eval above is intentionally small: it proves that the scoring pattern works on a labeled fixture. Production teams should expand it into a representative eval set before writing outputs to downstream systems. Start with a small, consented set of recordings or transcript fixtures, create human-reviewed labels, and keep a holdout set for regression testing when prompts, schemas, or models change.\n\n| Area | Example metrics |\n| --- | --- |\n| Speaker attribution | Speaker-label accuracy, diarization error rate, speaker-turn boundary accuracy, known-speaker match rate. |\n| Transcript grounding | Quote exactness, timestamp correctness, evidence-reference validity, unsupported-claim rate. |\n| Structured extraction | Precision and recall for action items, decisions, risks, explicit questions, suggested follow-ups, and customer requirements. |\n| Safety and privacy | PII redaction recall, moderation flag recall, false-positive review rate, raw-audio retention compliance. |\n| Workflow impact | Time-to-CRM-update, reviewer override rate, follow-up completion rate, renewal or escalation risk detection latency. |\n\nA useful first eval is simple: ask reviewers to mark each extracted action item as correct, partially correct, unsupported, or missing from the output. Track precision for generated items and recall against the human-labeled gold set. For quotes and evidence, prefer exact-match or near-exact-match checks against the transcript segment text so that helpful-sounding but unsupported summaries do not pass unnoticed. LLM-as-judge can help grade usefulness and completeness, but keep deterministic grounding checks in the loop because they are easier to reproduce.\n" }, { "cell_type": "markdown", "id": "sami-34", "metadata": {}, "source": "## Next steps\n\nYou can adapt the same pipeline for:\n\n- Customer success handoffs after quarterly business reviews.\n- Support escalations where accountability and exact quotes matter.\n- Sales discovery calls that feed CRM next steps.\n- Recruiting interview debriefs where each interviewer needs sourced notes.\n- Healthcare or financial-services workflows with stronger review and retention controls.\n\nFor live scenarios, use Realtime for the in-call experience and still run this post-call diarization pipeline when you need durable, evidence-backed meeting intelligence.\n\nUseful docs:\n\n- [Audio and speech guide](https://developers.openai.com/api/docs/guides/audio)\n- [Speech-to-text and speaker diarization](https://developers.openai.com/api/docs/guides/speech-to-text)\n- [Structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs)\n- [Moderation](https://developers.openai.com/api/docs/guides/moderation)\n- [Safety best practices](https://developers.openai.com/api/docs/guides/safety-best-practices)\n- [Realtime guide](https://developers.openai.com/api/docs/guides/realtime)\n" } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10" } }, "nbformat": 4, "nbformat_minor": 5 }