{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c516c861",
   "metadata": {},
   "source": [
    "# Optimizing customer support agents for cost and quality\n",
    "\n",
    "This cookbook demonstrates a repeatable optimization sprint for a tool-using agent: measure a baseline, change one part of the workflow, and check quality before accepting savings. It uses synthetic e-commerce support tickets and a deterministic simulation that runs without API spend. The same measurement loop applies to other agent workflows.\n",
    "\n",
    "By the end, you will have a repeatable pattern for:\n",
    "\n",
    "- Measuring quality, latency, tool use, and total cost on the same evaluation set.\n",
    "- Reducing unnecessary work through prompt and tool controls, model routing, and prompt caching.\n",
    "- Separating customer-facing work from offline follow-up and checking the resulting tradeoffs.\n",
    "\n",
    "The code defaults to dry-run mode. The optional live helpers require `OPENAI_API_KEY` and `RUN_LIVE_API_CALLS=true`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae85ea8b",
   "metadata": {},
   "source": [
    "## Outline\n",
    "\n",
    "1. Define success criteria and a small representative eval set.\n",
    "2. Build the intentionally inefficient baseline support agent.\n",
    "3. Measure baseline cost, tokens, quality, latency, and tool calls.\n",
    "4. Apply prompt, output, tool, and context controls.\n",
    "5. Route simple steps to smaller models.\n",
    "6. Restructure requests for prompt caching.\n",
    "7. Split real-time and follow-up work.\n",
    "8. Add monitoring, evals, and guardrails."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d19f7b31",
   "metadata": {},
   "source": [
    "## Use case and agent setup\n",
    "\n",
    "Our fictional e-commerce assistant handles order status, damaged deliveries, refund eligibility, duplicate charges, and account access. Routine lookups make smaller models worth evaluating; policy-sensitive cases test whether the optimized workflow still escalates correctly.\n",
    "\n",
    "The five mock tools represent an order system (`lookup_order`), customer records (`lookup_customer`), a policy source (`lookup_policy`), refund or replacement cases (`create_refund_case`), and human support (`escalate_to_human`). These are local Python functions, so even the live model examples cannot change a real customer account.\n",
    "\n",
    "The baseline exposes every tool, returns oversized payloads, and uses a full model for every step. It also performs internal QA, analytics tagging, and routing audits before replying. Later rounds keep the business task constant while reducing unnecessary work and moving follow-up processing out of the customer-facing path.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8d42b68",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "**Last verified: September 14, 2026.** The examples use GPT-5.4 models; the model-selection and caching sections also describe considerations for GPT-5.6.\n",
    "\n",
    "| Reference | Details used here |\n",
    "|---|---|\n",
    "| [Responses API](https://developers.openai.com/api/reference/resources/responses/methods/create) | Output limits, reasoning, verbosity, usage, conversation state, and service tiers |\n",
    "| [Function calling](https://developers.openai.com/api/docs/guides/function-calling) | Function schemas, `allowed_tools`, and forwarding reasoning and tool-call items |\n",
    "| [Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching) | Stable prefixes, model-specific caching controls, and token accounting |\n",
    "| [Compaction](https://developers.openai.com/api/docs/guides/compaction) | `context_management` and `compact_threshold` |\n",
    "| [Cost optimization](https://developers.openai.com/api/docs/guides/cost-optimization) | Fewer requests, smaller token budgets, and model selection |\n",
    "| [Batch API](https://developers.openai.com/api/docs/guides/batch) and [flex processing](https://developers.openai.com/api/docs/guides/flex-processing) | Offline processing, the Batch `24h` window, and flex availability tradeoffs |\n",
    "| [GPT-5.4](https://developers.openai.com/api/docs/models/gpt-5.4), [mini](https://developers.openai.com/api/docs/models/gpt-5.4-mini), and [nano](https://developers.openai.com/api/docs/models/gpt-5.4-nano) | Standard text-token prices and supported reasoning settings |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142bcf98",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Use Python 3.10 or later. Clone the Cookbook repository or download this entire [example folder](https://github.com/openai/openai-cookbook/tree/main/examples/agent_optimization), then start the notebook with `examples/agent_optimization` as the working directory so its local imports resolve.\n",
    "\n",
    "Install the dependencies in your notebook's environment:\n",
    "\n",
    "```bash\n",
    "pip install --upgrade openai pandas matplotlib jinja2 ipykernel\n",
    "```\n",
    "\n",
    "The same dependencies are listed in [requirements.txt](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/requirements.txt). `jinja2` is required for the styled pandas tables. The supporting files contain [mock data, tools, and prompts](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/support.py), [simulation and checks](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/simulation.py), [live API helpers](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/live_api.py), [offline answer evaluation](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/evaluation.py), and [scenario scoring](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/scenarios.py).\n",
    "\n",
    "The notebook does not call the API by default. To opt in to the live agent example, set these variables before starting the kernel:\n",
    "\n",
    "```bash\n",
    "export OPENAI_API_KEY=...\n",
    "export RUN_LIVE_API_CALLS=true\n",
    "```\n",
    "\n",
    "The optional answer judge has its own switch, `RUN_LLM_JUDGE=true`, and also requires `OPENAI_API_KEY`. It grades the 50 existing simulated traces (10 tickets × 5 variants) with 50 paid judge requests; it does not run the live agent. Leave both switches unset for a fully offline run.\n",
    "\n",
    "Run the cells from top to bottom. The Batch example writes a local file under `outputs/`; its submission code is displayed for inspection and is not executed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6a1302a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:04.018208Z",
     "iopub.status.busy": "2026-04-30T18:28:04.018043Z",
     "iopub.status.idle": "2026-04-30T18:28:05.055073Z",
     "shell.execute_reply": "2026-04-30T18:28:05.054627Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "RUN_LIVE_API_CALLS = True\nRUN_LLM_JUDGE = True\n"
    }
   ],
   "source": [
    "import json\n",
    "import math\n",
    "import os\n",
    "\n",
    "os.environ.setdefault(\"MPLCONFIGDIR\", \"/tmp/matplotlib\")\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from IPython.display import display\n",
    "from openai import OpenAI\n",
    "\n",
    "RUN_LIVE_API_CALLS = os.environ.get(\"RUN_LIVE_API_CALLS\", \"false\").lower() == \"true\"\n",
    "RUN_LLM_JUDGE = os.environ.get(\"RUN_LLM_JUDGE\", \"false\").lower() == \"true\"\n",
    "if (RUN_LIVE_API_CALLS or RUN_LLM_JUDGE) and not os.environ.get(\"OPENAI_API_KEY\"):\n",
    "    raise RuntimeError(\"Set OPENAI_API_KEY before enabling the live agent or judge.\")\n",
    "client = OpenAI() if RUN_LIVE_API_CALLS else None\n",
    "judge_client = OpenAI() if RUN_LLM_JUDGE else None\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 40)\n",
    "pd.set_option(\"display.width\", 140)\n",
    "print(\"RUN_LIVE_API_CALLS =\", RUN_LIVE_API_CALLS)\n",
    "print(\"RUN_LLM_JUDGE =\", RUN_LLM_JUDGE)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df9d761f",
   "metadata": {},
   "source": [
    "## Success criteria and constraints\n",
    "\n",
    "Accept savings only when the agent still uses the right facts, follows policy, takes the required action, and escalates correctly. A concise response must give the customer the next step without exposing internal data. Compare p50/p95 latency and total cost after those quality checks pass.\n",
    "\n",
    "The eval set is deliberately small. In production, use a stratified sample covering your main intents, risk levels, languages, regions, customer tiers, and edge cases. Keep a holdout set and gate each optimization on quality before comparing savings.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0470ef7",
   "metadata": {},
   "source": [
    "## Simulation contract\n",
    "\n",
    "The default path uses mock data and modeled metrics. It demonstrates the measurement loop; its numbers are not a production benchmark.\n",
    "\n",
    "The harness measures serialized text lengths and compares tool, action, escalation, and response-phrase checks against the fixtures. Token counts are estimated from those lengths. Reasoning tokens, latency, cache hits, and the aggregate quality score follow illustrative formulas; cost applies the verified price table to estimated usage.\n",
    "\n",
    "Routing and optimized actions come from the fixture labels, so this simulation does not measure a model's ability to choose them. Response checks use case-insensitive literal phrases, which can reject valid paraphrases and cannot establish factual correctness. For deployment decisions, replace these traces with real usage, timings, tool results, routing decisions, and calibrated judge or human evaluations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "885b8dcb",
   "metadata": {},
   "source": [
    "## Optimization knobs\n",
    "\n",
    "| Knob | Inefficient baseline | Optimized pattern | Primary metric |\n",
    "|---|---|---|---|\n",
    "| Prompt and output | Broad \"be thorough\" instructions and long answers | Specific task rules, concise response contract, `text.verbosity=\"low\"`, capped output | Output tokens, concision, quality |\n",
    "| Reasoning effort | High reasoning for every ticket | Low for routine work, higher only for high-risk decisions | Reasoning tokens, latency |\n",
    "| Tool surface | All tools exposed for every request | Full stable tool list plus `tool_choice.allowed_tools` per task | Tool calls, cacheability |\n",
    "| Tool schemas | Verbose descriptions and broad payload expectations | Small schemas with only decision-critical arguments | Input tokens |\n",
    "| Tool payloads | Raw CRM, carrier, audit, and appendix blobs | Slim fields needed for the next decision | Tool output tokens |\n",
    "| Model routing | One large model for all steps | Nano for triage/tags, mini for routine resolution, full model for high-risk cases | Cost, latency, escalation accuracy |\n",
    "| Prompt caching | Volatile ticket data mixed into the prefix | Stable instructions, tools, policy framing, and schema first; ticket data last | Cached input tokens, cost |\n",
    "| Workflow split | QA, analytics, summaries, and audits in the customer path | Customer resolution sync; QA/tags/reporting async via background, flex, or Batch | p50 latency, synchronous cost |\n",
    "| Guardrails and evals | Informal spot checks | Deterministic checks plus judge schema for live traces | Regression rate, safety pass rate |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sample-eval-title",
   "metadata": {},
   "source": [
    "## Sample evaluation set\n",
    "\n",
    "This small sample eval set gives the notebook concrete tickets, expected tools, expected actions, escalation labels, and forbidden claims to score each optimization round.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cfac042",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.056372Z",
     "iopub.status.busy": "2026-04-30T18:28:05.056285Z",
     "iopub.status.idle": "2026-04-30T18:28:05.071728Z",
     "shell.execute_reply": "2026-04-30T18:28:05.071309Z"
    }
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "  ticket_id                  intent    risk             difficulty                                     expected_tools  \\\n0     T-001            order_status     low          simple_lookup                                     [lookup_order]   \n1     T-002        damaged_delivery  medium         routine_policy                      [lookup_order, lookup_policy]   \n2     T-003      refund_eligibility  medium         routine_policy  [lookup_order, lookup_policy, create_refund_case]   \n3     T-004           billing_issue  medium       sensitive_policy   [lookup_order, lookup_policy, escalate_to_human]   \n4     T-005          account_access    high       account_security  [lookup_customer, lookup_policy, escalate_to_h...   \n5     T-006          refund_dispute    high  outside_policy_window   [lookup_order, lookup_policy, escalate_to_human]   \n6     T-007  delivered_not_received  medium         routine_policy                      [lookup_order, lookup_policy]   \n7     T-008       high_value_damage    high      high_value_policy   [lookup_order, lookup_policy, escalate_to_human]   \n8     T-009      refund_eligibility     low         routine_policy  [lookup_order, lookup_policy, create_refund_case]   \n9     T-010          account_access    high       account_security  [lookup_customer, lookup_policy, escalate_to_h...   \n\n                        expected_action  must_escalate  \n0                    provide_status_eta          False  \n1  request_photo_then_offer_replacement          False  \n2                      open_refund_case          False  \n3               escalate_billing_review           True  \n4             escalate_account_security           True  \n5                escalate_refund_review           True  \n6            start_delivery_trace_steps          False  \n7            escalate_high_value_damage           True  \n8                      open_refund_case          False  \n9             escalate_account_security           True  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>ticket_id</th>\n      <th>intent</th>\n      <th>risk</th>\n      <th>difficulty</th>\n      <th>expected_tools</th>\n      <th>expected_action</th>\n      <th>must_escalate</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>T-001</td>\n      <td>order_status</td>\n      <td>low</td>\n      <td>simple_lookup</td>\n      <td>[lookup_order]</td>\n      <td>provide_status_eta</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>T-002</td>\n      <td>damaged_delivery</td>\n      <td>medium</td>\n      <td>routine_policy</td>\n      <td>[lookup_order, lookup_policy]</td>\n      <td>request_photo_then_offer_replacement</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>T-003</td>\n      <td>refund_eligibility</td>\n      <td>medium</td>\n      <td>routine_policy</td>\n      <td>[lookup_order, lookup_policy, create_refund_case]</td>\n      <td>open_refund_case</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>T-004</td>\n      <td>billing_issue</td>\n      <td>medium</td>\n      <td>sensitive_policy</td>\n      <td>[lookup_order, lookup_policy, escalate_to_human]</td>\n      <td>escalate_billing_review</td>\n      <td>True</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>T-005</td>\n      <td>account_access</td>\n      <td>high</td>\n      <td>account_security</td>\n      <td>[lookup_customer, lookup_policy, escalate_to_h...</td>\n      <td>escalate_account_security</td>\n      <td>True</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>T-006</td>\n      <td>refund_dispute</td>\n      <td>high</td>\n      <td>outside_policy_window</td>\n      <td>[lookup_order, lookup_policy, escalate_to_human]</td>\n      <td>escalate_refund_review</td>\n      <td>True</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>T-007</td>\n      <td>delivered_not_received</td>\n      <td>medium</td>\n      <td>routine_policy</td>\n      <td>[lookup_order, lookup_policy]</td>\n      <td>start_delivery_trace_steps</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>T-008</td>\n      <td>high_value_damage</td>\n      <td>high</td>\n      <td>high_value_policy</td>\n      <td>[lookup_order, lookup_policy, escalate_to_human]</td>\n      <td>escalate_high_value_damage</td>\n      <td>True</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>T-009</td>\n      <td>refund_eligibility</td>\n      <td>low</td>\n      <td>routine_policy</td>\n      <td>[lookup_order, lookup_policy, create_refund_case]</td>\n      <td>open_refund_case</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>T-010</td>\n      <td>account_access</td>\n      <td>high</td>\n      <td>account_security</td>\n      <td>[lookup_customer, lookup_policy, escalate_to_h...</td>\n      <td>escalate_account_security</td>\n      <td>True</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": null
    }
   ],
   "source": [
    "from support import EVAL_SET\n",
    "\n",
    "pd.DataFrame(EVAL_SET)[\n",
    "    [\n",
    "        \"ticket_id\",\n",
    "        \"intent\",\n",
    "        \"risk\",\n",
    "        \"difficulty\",\n",
    "        \"expected_tools\",\n",
    "        \"expected_action\",\n",
    "        \"must_escalate\",\n",
    "    ]\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e09da4e5",
   "metadata": {},
   "source": [
    "## Support data and tools\n",
    "\n",
    "The five local tool functions in [support.py](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/support.py) stand in for internal systems. The baseline returns oversized payloads to show how tool outputs can dominate input tokens; later rounds return only fields needed for the decision and response.\n",
    "\n",
    "The following cell shows one slim order record. The mock action tools return synthetic results without creating real cases or escalations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cceb46f4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.072804Z",
     "iopub.status.busy": "2026-04-30T18:28:05.072746Z",
     "iopub.status.idle": "2026-04-30T18:28:05.078668Z",
     "shell.execute_reply": "2026-04-30T18:28:05.078319Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "{\n  \"found\": true,\n  \"order_id\": \"O-1001\",\n  \"status\": \"in_transit\",\n  \"carrier\": \"UPS\",\n  \"eta\": \"tomorrow\",\n  \"delivered_days_ago\": null,\n  \"payment_status\": \"paid_once\",\n  \"item_value\": 18.0,\n  \"events\": [\n    \"regional_delay\"\n  ]\n}\n"
    }
   ],
   "source": [
    "from support import lookup_order\n",
    "\n",
    "# Inspect the decision-critical fields returned by the slim payload.\n",
    "print(json.dumps(lookup_order(\"O-1001\", payload=\"slim\"), indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c03cddc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.079609Z",
     "iopub.status.busy": "2026-04-30T18:28:05.079554Z",
     "iopub.status.idle": "2026-04-30T18:28:05.082755Z",
     "shell.execute_reply": "2026-04-30T18:28:05.082425Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Verbose tool schema tokens: 550\nSlim tool schema tokens: 397\n"
    }
   ],
   "source": [
    "from support import SLIM_TOOLS, VERBOSE_TOOLS, allowed_tool_choice\n",
    "\n",
    "print(\"Verbose tool schema tokens:\", math.ceil(len(json.dumps(VERBOSE_TOOLS)) / 4))\n",
    "print(\"Slim tool schema tokens:\", math.ceil(len(json.dumps(SLIM_TOOLS)) / 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa43d461",
   "metadata": {},
   "source": [
    "## Baseline architecture\n",
    "\n",
    "The bad baseline does too much in one synchronous path.\n",
    "\n",
    "```mermaid\n",
    "flowchart LR\n",
    "    A[\"Customer message\"] --> B[\"One general agent on strongest model\"]\n",
    "    B --> C[\"Customer lookup\"]\n",
    "    B --> D[\"Order lookup\"]\n",
    "    B --> E[\"Policy lookup\"]\n",
    "    B --> F[\"Refund or escalation tools\"]\n",
    "    B --> G[\"Customer response\"]\n",
    "    B --> H[\"QA summary\"]\n",
    "    B --> I[\"Analytics tagging\"]\n",
    "    B --> J[\"Routing audit\"]\n",
    "```\n",
    "\n",
    "Broad instructions, high reasoning effort, and unrestricted tools make each request expensive. Large schemas and verbose payloads inflate inputs, while long answers and synchronous QA add work before the customer receives a reply.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f0c2ec0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.083867Z",
     "iopub.status.busy": "2026-04-30T18:28:05.083804Z",
     "iopub.status.idle": "2026-04-30T18:28:05.085893Z",
     "shell.execute_reply": "2026-04-30T18:28:05.085510Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Role: E-commerce support assistant.\n\nGoal: Resolve routine support tickets with the fewest necessary tool calls while preserving policy correctness.\n\nTool rules:\n- Use only tools required for the current decision.\n- Order status: order lookup only.\n- Damaged delivery or refund: order lookup plus the relevant policy.\n- Billing duplicate charge: order lookup plus billing policy, then escalate.\n- Account access with unverified identity: customer lookup plus account policy, then escalate.\n\nResponse rules:\n- Give the customer the outcome and next step.\n- Do not expose internal reasoning, raw tool data, audit notes, or policy text.\n- Keep the customer-facing answer under 120 words unless escalation legally requires more detail.\n"
    }
   ],
   "source": [
    "from support import CONTROLLED_PROMPT\n",
    "\n",
    "print(CONTROLLED_PROMPT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd2233bd",
   "metadata": {},
   "source": [
    "## Metrics helpers\n",
    "\n",
    "The live helper reads `input_tokens`, `output_tokens`, `total_tokens`, `input_tokens_details.cached_tokens`, and `output_tokens_details.reasoning_tokens`. Output-token usage already includes reasoning tokens; do not add them again when calculating cost.\n",
    "\n",
    "The table below shows USD per million text tokens at standard rates, verified September 14, 2026 against the [GPT-5.4](https://developers.openai.com/api/docs/models/gpt-5.4), [mini](https://developers.openai.com/api/docs/models/gpt-5.4-mini), and [nano](https://developers.openai.com/api/docs/models/gpt-5.4-nano) pages. It covers the short GPT-5.4 requests used here. The estimator does not cover long-context premiums, priority pricing, or GPT-5.6 cache-write charges; update it before changing those settings. See the [pricing page](https://developers.openai.com/api/docs/pricing) for current rates.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "754ecec6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.086804Z",
     "iopub.status.busy": "2026-04-30T18:28:05.086749Z",
     "iopub.status.idle": "2026-04-30T18:28:05.089475Z",
     "shell.execute_reply": "2026-04-30T18:28:05.089134Z"
    }
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "              input  cached_input  output\nmodel                                    \ngpt-5.4        2.50         0.250   15.00\ngpt-5.4-mini   0.75         0.075    4.50\ngpt-5.4-nano   0.20         0.020    1.25",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>input</th>\n      <th>cached_input</th>\n      <th>output</th>\n    </tr>\n    <tr>\n      <th>model</th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>gpt-5.4</th>\n      <td>2.50</td>\n      <td>0.250</td>\n      <td>15.00</td>\n    </tr>\n    <tr>\n      <th>gpt-5.4-mini</th>\n      <td>0.75</td>\n      <td>0.075</td>\n      <td>4.50</td>\n    </tr>\n    <tr>\n      <th>gpt-5.4-nano</th>\n      <td>0.20</td>\n      <td>0.020</td>\n      <td>1.25</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": null
    }
   ],
   "source": [
    "from simulation import MODEL_PRICES_USD_PER_1M\n",
    "\n",
    "pd.DataFrame(MODEL_PRICES_USD_PER_1M).T.rename_axis(\"model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24acf5c2",
   "metadata": {},
   "source": [
    "## Dry-run simulation\n",
    "\n",
    "The [simulation helper](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/simulation.py) applies each variant to the same tickets, estimates usage from prompts and payloads, and records the customer response with its quality checks. Missing required phrases and forbidden claims lower quality and fail the demo policy check, even when action and escalation labels match.\n",
    "\n",
    "The optimized variants assume a correct application router and known expected actions. Caching uses a simplified warm-cache assumption and a 1,024-token eligibility threshold, not a measurement of actual cache behavior. The repeated playbook makes the demonstration large enough to exercise that branch; production prompts should contain useful shared context, and cache eligibility depends on request settings.\n",
    "\n",
    "The final variant removes background work from synchronous latency while still counting its tokens and Batch cost. Inspect individual traces before relying on their averages.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56dd8264",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.090482Z",
     "iopub.status.busy": "2026-04-30T18:28:05.090427Z",
     "iopub.status.idle": "2026-04-30T18:28:05.107062Z",
     "shell.execute_reply": "2026-04-30T18:28:05.106771Z"
    }
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "metadata": {},
     "data": {
      "text/plain": "           variant variant_label ticket_id              intent    risk        difficulty    model  routing_tokens  tool_calls  \\\n0  00_bad_baseline  Bad baseline     T-001        order_status     low     simple_lookup  gpt-5.4               0           5   \n1  00_bad_baseline  Bad baseline     T-002    damaged_delivery  medium    routine_policy  gpt-5.4               0           5   \n2  00_bad_baseline  Bad baseline     T-003  refund_eligibility  medium    routine_policy  gpt-5.4               0           5   \n3  00_bad_baseline  Bad baseline     T-004       billing_issue  medium  sensitive_policy  gpt-5.4               0           5   \n4  00_bad_baseline  Bad baseline     T-005      account_access    high  account_security  gpt-5.4               0           5   \n\n                                      expected_tools                                              tools                          action  \\\n0                                       lookup_order  lookup_customer, lookup_order, lookup_policy, ...              provide_status_eta   \n1                        lookup_order, lookup_policy  lookup_customer, lookup_order, lookup_policy, ...  open_replacement_without_photo   \n2    lookup_order, lookup_policy, create_refund_case  lookup_customer, lookup_order, lookup_policy, ...          escalate_refund_review   \n3     lookup_order, lookup_policy, escalate_to_human  lookup_customer, lookup_order, lookup_policy, ...         escalate_billing_review   \n4  lookup_customer, lookup_policy, escalate_to_human  lookup_customer, lookup_order, lookup_policy, ...       escalate_account_security   \n\n                        expected_action  input_tokens  latency_input_tokens  cacheable_prefix_tokens  cached_tokens  output_tokens  \\\n0                    provide_status_eta         12301                 12301                        0              0            612   \n1  request_photo_then_offer_replacement         12273                 12273                        0              0            644   \n2                      open_refund_case         12178                 12178                        0              0            640   \n3               escalate_billing_review         12166                 12166                        0              0            664   \n4             escalate_account_security          7415                  7415                        0              0            697   \n\n   visible_output_tokens  reasoning_tokens  ...  total_tokens  latency_s  sync_cost_usd  background_tokens  background_cost_usd  cost_usd  \\\n0                    167               445  ...         12913       4.88       0.039933                  0                  0.0  0.039933   \n1                    174               470  ...         12917       4.88       0.040343                  0                  0.0  0.040343   \n2                    170               470  ...         12818       4.87       0.040045                  0                  0.0  0.040045   \n3                    169               495  ...         12830       4.87       0.040375                  0                  0.0  0.040375   \n4                    172               525  ...          8112       3.97       0.028993                  0                  0.0  0.028993   \n\n   escalated                                  customer_response missing_required_tools extra_tool_calls  \\\n0      False  I reviewed your message for ticket T-001 and c...                                       4   \n1      False  I reviewed your message for ticket T-002 and c...                                       3   \n2       True  I reviewed your message for ticket T-003 and c...                                       2   \n3       True  I reviewed your message for ticket T-004 and c...                                       2   \n4       True  I reviewed your message for ticket T-005 and c...                                       2   \n\n                                   unnecessary_tools escalation_correct  action_correct  policy_compliant  concise  response_complete  \\\n0  create_refund_case, escalate_to_human, lookup_...               True            True             False    False              False   \n1  create_refund_case, escalate_to_human, lookup_...               True           False             False    False               True   \n2                 escalate_to_human, lookup_customer              False           False             False    False              False   \n3                create_refund_case, lookup_customer               True            True              True    False               True   \n4                   create_refund_case, lookup_order               True            True             False    False              False   \n\n         missing_required_phrases forbidden_claims_absent  forbidden_claims_found quality_score  \n0            in transit, tomorrow                    True                                  0.55  \n1                                                    True                                  0.65  \n2     refund case, within 30 days                    True                                  0.22  \n3                                                    True                                  0.85  \n4  account security, verification                    True                                  0.60  \n\n[5 rows x 41 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>variant</th>\n      <th>variant_label</th>\n      <th>ticket_id</th>\n      <th>intent</th>\n      <th>risk</th>\n      <th>difficulty</th>\n      <th>model</th>\n      <th>routing_tokens</th>\n      <th>tool_calls</th>\n      <th>expected_tools</th>\n      <th>tools</th>\n      <th>action</th>\n      <th>expected_action</th>\n      <th>input_tokens</th>\n      <th>latency_input_tokens</th>\n      <th>cacheable_prefix_tokens</th>\n      <th>cached_tokens</th>\n      <th>output_tokens</th>\n      <th>visible_output_tokens</th>\n      <th>reasoning_tokens</th>\n      <th>...</th>\n      <th>total_tokens</th>\n      <th>latency_s</th>\n      <th>sync_cost_usd</th>\n      <th>background_tokens</th>\n      <th>background_cost_usd</th>\n      <th>cost_usd</th>\n      <th>escalated</th>\n      <th>customer_response</th>\n      <th>missing_required_tools</th>\n      <th>extra_tool_calls</th>\n      <th>unnecessary_tools</th>\n      <th>escalation_correct</th>\n      <th>action_correct</th>\n      <th>policy_compliant</th>\n      <th>concise</th>\n      <th>response_complete</th>\n      <th>missing_required_phrases</th>\n      <th>forbidden_claims_absent</th>\n      <th>forbidden_claims_found</th>\n      <th>quality_score</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>00_bad_baseline</td>\n      <td>Bad baseline</td>\n      <td>T-001</td>\n      <td>order_status</td>\n      <td>low</td>\n      <td>simple_lookup</td>\n      <td>gpt-5.4</td>\n      <td>0</td>\n      <td>5</td>\n      <td>lookup_order</td>\n      <td>lookup_customer, lookup_order, lookup_policy, ...</td>\n      <td>provide_status_eta</td>\n      <td>provide_status_eta</td>\n      <td>12301</td>\n      <td>12301</td>\n      <td>0</td>\n      <td>0</td>\n      <td>612</td>\n      <td>167</td>\n      <td>445</td>\n      <td>...</td>\n      <td>12913</td>\n      <td>4.88</td>\n      <td>0.039933</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>0.039933</td>\n      <td>False</td>\n      <td>I reviewed your message for ticket T-001 and c...</td>\n      <td></td>\n      <td>4</td>\n      <td>create_refund_case, escalate_to_human, lookup_...</td>\n      <td>True</td>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>in transit, tomorrow</td>\n      <td>True</td>\n      <td></td>\n      <td>0.55</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>00_bad_baseline</td>\n      <td>Bad baseline</td>\n      <td>T-002</td>\n      <td>damaged_delivery</td>\n      <td>medium</td>\n      <td>routine_policy</td>\n      <td>gpt-5.4</td>\n      <td>0</td>\n      <td>5</td>\n      <td>lookup_order, lookup_policy</td>\n      <td>lookup_customer, lookup_order, lookup_policy, ...</td>\n      <td>open_replacement_without_photo</td>\n      <td>request_photo_then_offer_replacement</td>\n      <td>12273</td>\n      <td>12273</td>\n      <td>0</td>\n      <td>0</td>\n      <td>644</td>\n      <td>174</td>\n      <td>470</td>\n      <td>...</td>\n      <td>12917</td>\n      <td>4.88</td>\n      <td>0.040343</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>0.040343</td>\n      <td>False</td>\n      <td>I reviewed your message for ticket T-002 and c...</td>\n      <td></td>\n      <td>3</td>\n      <td>create_refund_case, escalate_to_human, lookup_...</td>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>True</td>\n      <td></td>\n      <td>True</td>\n      <td></td>\n      <td>0.65</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>00_bad_baseline</td>\n      <td>Bad baseline</td>\n      <td>T-003</td>\n      <td>refund_eligibility</td>\n      <td>medium</td>\n      <td>routine_policy</td>\n      <td>gpt-5.4</td>\n      <td>0</td>\n      <td>5</td>\n      <td>lookup_order, lookup_policy, create_refund_case</td>\n      <td>lookup_customer, lookup_order, lookup_policy, ...</td>\n      <td>escalate_refund_review</td>\n      <td>open_refund_case</td>\n      <td>12178</td>\n      <td>12178</td>\n      <td>0</td>\n      <td>0</td>\n      <td>640</td>\n      <td>170</td>\n      <td>470</td>\n      <td>...</td>\n      <td>12818</td>\n      <td>4.87</td>\n      <td>0.040045</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>0.040045</td>\n      <td>True</td>\n      <td>I reviewed your message for ticket T-003 and c...</td>\n      <td></td>\n      <td>2</td>\n      <td>escalate_to_human, lookup_customer</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>refund case, within 30 days</td>\n      <td>True</td>\n      <td></td>\n      <td>0.22</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>00_bad_baseline</td>\n      <td>Bad baseline</td>\n      <td>T-004</td>\n      <td>billing_issue</td>\n      <td>medium</td>\n      <td>sensitive_policy</td>\n      <td>gpt-5.4</td>\n      <td>0</td>\n      <td>5</td>\n      <td>lookup_order, lookup_policy, escalate_to_human</td>\n      <td>lookup_customer, lookup_order, lookup_policy, ...</td>\n      <td>escalate_billing_review</td>\n      <td>escalate_billing_review</td>\n      <td>12166</td>\n      <td>12166</td>\n      <td>0</td>\n      <td>0</td>\n      <td>664</td>\n      <td>169</td>\n      <td>495</td>\n      <td>...</td>\n      <td>12830</td>\n      <td>4.87</td>\n      <td>0.040375</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>0.040375</td>\n      <td>True</td>\n      <td>I reviewed your message for ticket T-004 and c...</td>\n      <td></td>\n      <td>2</td>\n      <td>create_refund_case, lookup_customer</td>\n      <td>True</td>\n      <td>True</td>\n      <td>True</td>\n      <td>False</td>\n      <td>True</td>\n      <td></td>\n      <td>True</td>\n      <td></td>\n      <td>0.85</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>00_bad_baseline</td>\n      <td>Bad baseline</td>\n      <td>T-005</td>\n      <td>account_access</td>\n      <td>high</td>\n      <td>account_security</td>\n      <td>gpt-5.4</td>\n      <td>0</td>\n      <td>5</td>\n      <td>lookup_customer, lookup_policy, escalate_to_human</td>\n      <td>lookup_customer, lookup_order, lookup_policy, ...</td>\n      <td>escalate_account_security</td>\n      <td>escalate_account_security</td>\n      <td>7415</td>\n      <td>7415</td>\n      <td>0</td>\n      <td>0</td>\n      <td>697</td>\n      <td>172</td>\n      <td>525</td>\n      <td>...</td>\n      <td>8112</td>\n      <td>3.97</td>\n      <td>0.028993</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>0.028993</td>\n      <td>True</td>\n      <td>I reviewed your message for ticket T-005 and c...</td>\n      <td></td>\n      <td>2</td>\n      <td>create_refund_case, lookup_order</td>\n      <td>True</td>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>account security, verification</td>\n      <td>True</td>\n      <td></td>\n      <td>0.60</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 41 columns</p>\n</div>"
     },
     "execution_count": null
    }
   ],
   "source": [
    "from simulation import CACHE_FRIENDLY_PROMPT, VARIANT_ORDER, simulate_trace\n",
    "\n",
    "traces = pd.DataFrame(\n",
    "    simulate_trace(ticket, variant)\n",
    "    for variant in VARIANT_ORDER\n",
    "    for ticket in EVAL_SET\n",
    ")\n",
    "traces.drop(columns=\"tool_results\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4de8e218",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.108437Z",
     "iopub.status.busy": "2026-04-30T18:28:05.108368Z",
     "iopub.status.idle": "2026-04-30T18:28:05.139846Z",
     "shell.execute_reply": "2026-04-30T18:28:05.139514Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x116b33740>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_01f0b\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_01f0b_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_01f0b_level0_col1\" class=\"col_heading level0 col1\" >mean_quality</th>\n      <th id=\"T_01f0b_level0_col2\" class=\"col_heading level0 col2\" >policy_compliance</th>\n      <th id=\"T_01f0b_level0_col3\" class=\"col_heading level0 col3\" >action_accuracy</th>\n      <th id=\"T_01f0b_level0_col4\" class=\"col_heading level0 col4\" >escalation_accuracy</th>\n      <th id=\"T_01f0b_level0_col5\" class=\"col_heading level0 col5\" >mean_tool_calls</th>\n      <th id=\"T_01f0b_level0_col6\" class=\"col_heading level0 col6\" >mean_extra_tool_calls</th>\n      <th id=\"T_01f0b_level0_col7\" class=\"col_heading level0 col7\" >mean_sync_tokens</th>\n      <th id=\"T_01f0b_level0_col8\" class=\"col_heading level0 col8\" >mean_total_tokens</th>\n      <th id=\"T_01f0b_level0_col9\" class=\"col_heading level0 col9\" >mean_cached_tokens</th>\n      <th id=\"T_01f0b_level0_col10\" class=\"col_heading level0 col10\" >p50_latency_s</th>\n      <th id=\"T_01f0b_level0_col11\" class=\"col_heading level0 col11\" >cost_per_ticket_usd</th>\n      <th id=\"T_01f0b_level0_col12\" class=\"col_heading level0 col12\" >cost_reduction_vs_baseline</th>\n      <th id=\"T_01f0b_level0_col13\" class=\"col_heading level0 col13\" >monthly_cost_at_100k_tickets</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_01f0b_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n      <td id=\"T_01f0b_row0_col0\" class=\"data row0 col0\" >Bad baseline</td>\n      <td id=\"T_01f0b_row0_col1\" class=\"data row0 col1\" >0.51</td>\n      <td id=\"T_01f0b_row0_col2\" class=\"data row0 col2\" >10%</td>\n      <td id=\"T_01f0b_row0_col3\" class=\"data row0 col3\" >60%</td>\n      <td id=\"T_01f0b_row0_col4\" class=\"data row0 col4\" >70%</td>\n      <td id=\"T_01f0b_row0_col5\" class=\"data row0 col5\" >5.0</td>\n      <td id=\"T_01f0b_row0_col6\" class=\"data row0 col6\" >2.4</td>\n      <td id=\"T_01f0b_row0_col7\" class=\"data row0 col7\" >11,935</td>\n      <td id=\"T_01f0b_row0_col8\" class=\"data row0 col8\" >11,935</td>\n      <td id=\"T_01f0b_row0_col9\" class=\"data row0 col9\" >0</td>\n      <td id=\"T_01f0b_row0_col10\" class=\"data row0 col10\" >4.88</td>\n      <td id=\"T_01f0b_row0_col11\" class=\"data row0 col11\" >$0.03813</td>\n      <td id=\"T_01f0b_row0_col12\" class=\"data row0 col12\" >0%</td>\n      <td id=\"T_01f0b_row0_col13\" class=\"data row0 col13\" >$3,813</td>\n    </tr>\n    <tr>\n      <th id=\"T_01f0b_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n      <td id=\"T_01f0b_row1_col0\" class=\"data row1 col0\" >Round 1: controls</td>\n      <td id=\"T_01f0b_row1_col1\" class=\"data row1 col1\" >0.98</td>\n      <td id=\"T_01f0b_row1_col2\" class=\"data row1 col2\" >100%</td>\n      <td id=\"T_01f0b_row1_col3\" class=\"data row1 col3\" >100%</td>\n      <td id=\"T_01f0b_row1_col4\" class=\"data row1 col4\" >100%</td>\n      <td id=\"T_01f0b_row1_col5\" class=\"data row1 col5\" >2.6</td>\n      <td id=\"T_01f0b_row1_col6\" class=\"data row1 col6\" >0.0</td>\n      <td id=\"T_01f0b_row1_col7\" class=\"data row1 col7\" >1,379</td>\n      <td id=\"T_01f0b_row1_col8\" class=\"data row1 col8\" >1,379</td>\n      <td id=\"T_01f0b_row1_col9\" class=\"data row1 col9\" >0</td>\n      <td id=\"T_01f0b_row1_col10\" class=\"data row1 col10\" >2.32</td>\n      <td id=\"T_01f0b_row1_col11\" class=\"data row1 col11\" >$0.00512</td>\n      <td id=\"T_01f0b_row1_col12\" class=\"data row1 col12\" >87%</td>\n      <td id=\"T_01f0b_row1_col13\" class=\"data row1 col13\" >$512</td>\n    </tr>\n    <tr>\n      <th id=\"T_01f0b_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n      <td id=\"T_01f0b_row2_col0\" class=\"data row2 col0\" >Round 2: routing</td>\n      <td id=\"T_01f0b_row2_col1\" class=\"data row2 col1\" >0.98</td>\n      <td id=\"T_01f0b_row2_col2\" class=\"data row2 col2\" >100%</td>\n      <td id=\"T_01f0b_row2_col3\" class=\"data row2 col3\" >100%</td>\n      <td id=\"T_01f0b_row2_col4\" class=\"data row2 col4\" >100%</td>\n      <td id=\"T_01f0b_row2_col5\" class=\"data row2 col5\" >2.6</td>\n      <td id=\"T_01f0b_row2_col6\" class=\"data row2 col6\" >0.0</td>\n      <td id=\"T_01f0b_row2_col7\" class=\"data row2 col7\" >1,485</td>\n      <td id=\"T_01f0b_row2_col8\" class=\"data row2 col8\" >1,485</td>\n      <td id=\"T_01f0b_row2_col9\" class=\"data row2 col9\" >0</td>\n      <td id=\"T_01f0b_row2_col10\" class=\"data row2 col10\" >1.87</td>\n      <td id=\"T_01f0b_row2_col11\" class=\"data row2 col11\" >$0.00302</td>\n      <td id=\"T_01f0b_row2_col12\" class=\"data row2 col12\" >92%</td>\n      <td id=\"T_01f0b_row2_col13\" class=\"data row2 col13\" >$302</td>\n    </tr>\n    <tr>\n      <th id=\"T_01f0b_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n      <td id=\"T_01f0b_row3_col0\" class=\"data row3 col0\" >Round 3: caching</td>\n      <td id=\"T_01f0b_row3_col1\" class=\"data row3 col1\" >0.98</td>\n      <td id=\"T_01f0b_row3_col2\" class=\"data row3 col2\" >100%</td>\n      <td id=\"T_01f0b_row3_col3\" class=\"data row3 col3\" >100%</td>\n      <td id=\"T_01f0b_row3_col4\" class=\"data row3 col4\" >100%</td>\n      <td id=\"T_01f0b_row3_col5\" class=\"data row3 col5\" >2.6</td>\n      <td id=\"T_01f0b_row3_col6\" class=\"data row3 col6\" >0.0</td>\n      <td id=\"T_01f0b_row3_col7\" class=\"data row3 col7\" >2,684</td>\n      <td id=\"T_01f0b_row3_col8\" class=\"data row3 col8\" >2,684</td>\n      <td id=\"T_01f0b_row3_col9\" class=\"data row3 col9\" >1,779</td>\n      <td id=\"T_01f0b_row3_col10\" class=\"data row3 col10\" >1.85</td>\n      <td id=\"T_01f0b_row3_col11\" class=\"data row3 col11\" >$0.00244</td>\n      <td id=\"T_01f0b_row3_col12\" class=\"data row3 col12\" >94%</td>\n      <td id=\"T_01f0b_row3_col13\" class=\"data row3 col13\" >$244</td>\n    </tr>\n    <tr>\n      <th id=\"T_01f0b_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n      <td id=\"T_01f0b_row4_col0\" class=\"data row4 col0\" >Round 4: split workflow</td>\n      <td id=\"T_01f0b_row4_col1\" class=\"data row4 col1\" >0.98</td>\n      <td id=\"T_01f0b_row4_col2\" class=\"data row4 col2\" >100%</td>\n      <td id=\"T_01f0b_row4_col3\" class=\"data row4 col3\" >100%</td>\n      <td id=\"T_01f0b_row4_col4\" class=\"data row4 col4\" >100%</td>\n      <td id=\"T_01f0b_row4_col5\" class=\"data row4 col5\" >2.6</td>\n      <td id=\"T_01f0b_row4_col6\" class=\"data row4 col6\" >0.0</td>\n      <td id=\"T_01f0b_row4_col7\" class=\"data row4 col7\" >2,412</td>\n      <td id=\"T_01f0b_row4_col8\" class=\"data row4 col8\" >2,974</td>\n      <td id=\"T_01f0b_row4_col9\" class=\"data row4 col9\" >1,779</td>\n      <td id=\"T_01f0b_row4_col10\" class=\"data row4 col10\" >1.41</td>\n      <td id=\"T_01f0b_row4_col11\" class=\"data row4 col11\" >$0.00204</td>\n      <td id=\"T_01f0b_row4_col12\" class=\"data row4 col12\" >95%</td>\n      <td id=\"T_01f0b_row4_col13\" class=\"data row4 col13\" >$204</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "summary = (\n",
    "    traces.groupby([\"variant\", \"variant_label\"], sort=False)\n",
    "    .agg(\n",
    "        tickets=(\"ticket_id\", \"count\"),\n",
    "        mean_quality=(\"quality_score\", \"mean\"),\n",
    "        policy_compliance=(\"policy_compliant\", \"mean\"),\n",
    "        action_accuracy=(\"action_correct\", \"mean\"),\n",
    "        escalation_accuracy=(\"escalation_correct\", \"mean\"),\n",
    "        concise_rate=(\"concise\", \"mean\"),\n",
    "        mean_tool_calls=(\"tool_calls\", \"mean\"),\n",
    "        mean_extra_tool_calls=(\"extra_tool_calls\", \"mean\"),\n",
    "        mean_input_tokens=(\"input_tokens\", \"mean\"),\n",
    "        mean_cached_tokens=(\"cached_tokens\", \"mean\"),\n",
    "        mean_output_tokens=(\"output_tokens\", \"mean\"),\n",
    "        mean_reasoning_tokens=(\"reasoning_tokens\", \"mean\"),\n",
    "        mean_sync_tokens=(\"sync_tokens\", \"mean\"),\n",
    "        mean_total_tokens=(\"total_tokens\", \"mean\"),\n",
    "        p50_latency_s=(\"latency_s\", \"median\"),\n",
    "        p95_latency_s=(\"latency_s\", lambda s: s.quantile(0.95)),\n",
    "        sync_cost_per_ticket_usd=(\"sync_cost_usd\", \"mean\"),\n",
    "        background_cost_per_ticket_usd=(\"background_cost_usd\", \"mean\"),\n",
    "        cost_per_ticket_usd=(\"cost_usd\", \"mean\"),\n",
    "    )\n",
    "    .reset_index()\n",
    ")\n",
    "\n",
    "baseline_cost = summary.loc[summary[\"variant\"] == \"00_bad_baseline\", \"cost_per_ticket_usd\"].iloc[0]\n",
    "baseline_tokens = summary.loc[summary[\"variant\"] == \"00_bad_baseline\", \"mean_total_tokens\"].iloc[0]\n",
    "baseline_latency = summary.loc[summary[\"variant\"] == \"00_bad_baseline\", \"p50_latency_s\"].iloc[0]\n",
    "\n",
    "summary[\"cost_reduction_vs_baseline\"] = 1 - summary[\"cost_per_ticket_usd\"] / baseline_cost\n",
    "summary[\"token_reduction_vs_baseline\"] = 1 - summary[\"mean_total_tokens\"] / baseline_tokens\n",
    "summary[\"latency_reduction_vs_baseline\"] = 1 - summary[\"p50_latency_s\"] / baseline_latency\n",
    "summary[\"monthly_cost_at_100k_tickets\"] = summary[\"cost_per_ticket_usd\"] * 100_000\n",
    "\n",
    "summary_view = summary[\n",
    "    [\n",
    "        \"variant_label\",\n",
    "        \"mean_quality\",\n",
    "        \"policy_compliance\",\n",
    "        \"action_accuracy\",\n",
    "        \"escalation_accuracy\",\n",
    "        \"mean_tool_calls\",\n",
    "        \"mean_extra_tool_calls\",\n",
    "        \"mean_sync_tokens\",\n",
    "        \"mean_total_tokens\",\n",
    "        \"mean_cached_tokens\",\n",
    "        \"p50_latency_s\",\n",
    "        \"cost_per_ticket_usd\",\n",
    "        \"cost_reduction_vs_baseline\",\n",
    "        \"monthly_cost_at_100k_tickets\",\n",
    "    ]\n",
    "]\n",
    "\n",
    "display(\n",
    "    summary_view.style.format(\n",
    "        {\n",
    "            \"mean_quality\": \"{:.2f}\",\n",
    "            \"policy_compliance\": \"{:.0%}\",\n",
    "            \"action_accuracy\": \"{:.0%}\",\n",
    "            \"escalation_accuracy\": \"{:.0%}\",\n",
    "            \"mean_tool_calls\": \"{:.1f}\",\n",
    "            \"mean_extra_tool_calls\": \"{:.1f}\",\n",
    "            \"mean_sync_tokens\": \"{:,.0f}\",\n",
    "            \"mean_total_tokens\": \"{:,.0f}\",\n",
    "            \"mean_cached_tokens\": \"{:,.0f}\",\n",
    "            \"p50_latency_s\": \"{:.2f}\",\n",
    "            \"cost_per_ticket_usd\": \"${:.5f}\",\n",
    "            \"cost_reduction_vs_baseline\": \"{:.0%}\",\n",
    "            \"monthly_cost_at_100k_tickets\": \"${:,.0f}\",\n",
    "        }\n",
    "    )\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e995bdd3",
   "metadata": {},
   "source": [
    "## Round-by-round impact\n",
    "\n",
    "Each row compares one round to the previous round. This makes the optimization knobs easier to reason about than a single before/after number.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ed246b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.140857Z",
     "iopub.status.busy": "2026-04-30T18:28:05.140791Z",
     "iopub.status.idle": "2026-04-30T18:28:05.146124Z",
     "shell.execute_reply": "2026-04-30T18:28:05.145767Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x116dded50>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_81e05\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_81e05_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_81e05_level0_col1\" class=\"col_heading level0 col1\" >mean_quality</th>\n      <th id=\"T_81e05_level0_col2\" class=\"col_heading level0 col2\" >policy_compliance</th>\n      <th id=\"T_81e05_level0_col3\" class=\"col_heading level0 col3\" >mean_tool_calls</th>\n      <th id=\"T_81e05_level0_col4\" class=\"col_heading level0 col4\" >mean_extra_tool_calls</th>\n      <th id=\"T_81e05_level0_col5\" class=\"col_heading level0 col5\" >mean_sync_tokens</th>\n      <th id=\"T_81e05_level0_col6\" class=\"col_heading level0 col6\" >mean_total_tokens</th>\n      <th id=\"T_81e05_level0_col7\" class=\"col_heading level0 col7\" >mean_cached_tokens</th>\n      <th id=\"T_81e05_level0_col8\" class=\"col_heading level0 col8\" >p50_latency_s</th>\n      <th id=\"T_81e05_level0_col9\" class=\"col_heading level0 col9\" >cost_per_ticket_usd</th>\n      <th id=\"T_81e05_level0_col10\" class=\"col_heading level0 col10\" >mean_sync_tokens_delta_vs_previous</th>\n      <th id=\"T_81e05_level0_col11\" class=\"col_heading level0 col11\" >mean_total_tokens_delta_vs_previous</th>\n      <th id=\"T_81e05_level0_col12\" class=\"col_heading level0 col12\" >p50_latency_s_delta_vs_previous</th>\n      <th id=\"T_81e05_level0_col13\" class=\"col_heading level0 col13\" >cost_per_ticket_usd_delta_vs_previous</th>\n      <th id=\"T_81e05_level0_col14\" class=\"col_heading level0 col14\" >quality_delta_vs_previous</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_81e05_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n      <td id=\"T_81e05_row0_col0\" class=\"data row0 col0\" >Bad baseline</td>\n      <td id=\"T_81e05_row0_col1\" class=\"data row0 col1\" >0.51</td>\n      <td id=\"T_81e05_row0_col2\" class=\"data row0 col2\" >10%</td>\n      <td id=\"T_81e05_row0_col3\" class=\"data row0 col3\" >5.0</td>\n      <td id=\"T_81e05_row0_col4\" class=\"data row0 col4\" >2.4</td>\n      <td id=\"T_81e05_row0_col5\" class=\"data row0 col5\" >11,935</td>\n      <td id=\"T_81e05_row0_col6\" class=\"data row0 col6\" >11,935</td>\n      <td id=\"T_81e05_row0_col7\" class=\"data row0 col7\" >0</td>\n      <td id=\"T_81e05_row0_col8\" class=\"data row0 col8\" >4.88</td>\n      <td id=\"T_81e05_row0_col9\" class=\"data row0 col9\" >$0.03813</td>\n      <td id=\"T_81e05_row0_col10\" class=\"data row0 col10\" >+nan</td>\n      <td id=\"T_81e05_row0_col11\" class=\"data row0 col11\" >+nan</td>\n      <td id=\"T_81e05_row0_col12\" class=\"data row0 col12\" >+nan</td>\n      <td id=\"T_81e05_row0_col13\" class=\"data row0 col13\" >$+nan</td>\n      <td id=\"T_81e05_row0_col14\" class=\"data row0 col14\" >+nan</td>\n    </tr>\n    <tr>\n      <th id=\"T_81e05_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n      <td id=\"T_81e05_row1_col0\" class=\"data row1 col0\" >Round 1: controls</td>\n      <td id=\"T_81e05_row1_col1\" class=\"data row1 col1\" >0.98</td>\n      <td id=\"T_81e05_row1_col2\" class=\"data row1 col2\" >100%</td>\n      <td id=\"T_81e05_row1_col3\" class=\"data row1 col3\" >2.6</td>\n      <td id=\"T_81e05_row1_col4\" class=\"data row1 col4\" >0.0</td>\n      <td id=\"T_81e05_row1_col5\" class=\"data row1 col5\" >1,379</td>\n      <td id=\"T_81e05_row1_col6\" class=\"data row1 col6\" >1,379</td>\n      <td id=\"T_81e05_row1_col7\" class=\"data row1 col7\" >0</td>\n      <td id=\"T_81e05_row1_col8\" class=\"data row1 col8\" >2.32</td>\n      <td id=\"T_81e05_row1_col9\" class=\"data row1 col9\" >$0.00512</td>\n      <td id=\"T_81e05_row1_col10\" class=\"data row1 col10\" >-10,556</td>\n      <td id=\"T_81e05_row1_col11\" class=\"data row1 col11\" >-10,556</td>\n      <td id=\"T_81e05_row1_col12\" class=\"data row1 col12\" >-2.56</td>\n      <td id=\"T_81e05_row1_col13\" class=\"data row1 col13\" >$-0.03301</td>\n      <td id=\"T_81e05_row1_col14\" class=\"data row1 col14\" >+0.48</td>\n    </tr>\n    <tr>\n      <th id=\"T_81e05_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n      <td id=\"T_81e05_row2_col0\" class=\"data row2 col0\" >Round 2: routing</td>\n      <td id=\"T_81e05_row2_col1\" class=\"data row2 col1\" >0.98</td>\n      <td id=\"T_81e05_row2_col2\" class=\"data row2 col2\" >100%</td>\n      <td id=\"T_81e05_row2_col3\" class=\"data row2 col3\" >2.6</td>\n      <td id=\"T_81e05_row2_col4\" class=\"data row2 col4\" >0.0</td>\n      <td id=\"T_81e05_row2_col5\" class=\"data row2 col5\" >1,485</td>\n      <td id=\"T_81e05_row2_col6\" class=\"data row2 col6\" >1,485</td>\n      <td id=\"T_81e05_row2_col7\" class=\"data row2 col7\" >0</td>\n      <td id=\"T_81e05_row2_col8\" class=\"data row2 col8\" >1.87</td>\n      <td id=\"T_81e05_row2_col9\" class=\"data row2 col9\" >$0.00302</td>\n      <td id=\"T_81e05_row2_col10\" class=\"data row2 col10\" >+106</td>\n      <td id=\"T_81e05_row2_col11\" class=\"data row2 col11\" >+106</td>\n      <td id=\"T_81e05_row2_col12\" class=\"data row2 col12\" >-0.45</td>\n      <td id=\"T_81e05_row2_col13\" class=\"data row2 col13\" >$-0.00210</td>\n      <td id=\"T_81e05_row2_col14\" class=\"data row2 col14\" >+0.00</td>\n    </tr>\n    <tr>\n      <th id=\"T_81e05_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n      <td id=\"T_81e05_row3_col0\" class=\"data row3 col0\" >Round 3: caching</td>\n      <td id=\"T_81e05_row3_col1\" class=\"data row3 col1\" >0.98</td>\n      <td id=\"T_81e05_row3_col2\" class=\"data row3 col2\" >100%</td>\n      <td id=\"T_81e05_row3_col3\" class=\"data row3 col3\" >2.6</td>\n      <td id=\"T_81e05_row3_col4\" class=\"data row3 col4\" >0.0</td>\n      <td id=\"T_81e05_row3_col5\" class=\"data row3 col5\" >2,684</td>\n      <td id=\"T_81e05_row3_col6\" class=\"data row3 col6\" >2,684</td>\n      <td id=\"T_81e05_row3_col7\" class=\"data row3 col7\" >1,779</td>\n      <td id=\"T_81e05_row3_col8\" class=\"data row3 col8\" >1.85</td>\n      <td id=\"T_81e05_row3_col9\" class=\"data row3 col9\" >$0.00244</td>\n      <td id=\"T_81e05_row3_col10\" class=\"data row3 col10\" >+1,199</td>\n      <td id=\"T_81e05_row3_col11\" class=\"data row3 col11\" >+1,199</td>\n      <td id=\"T_81e05_row3_col12\" class=\"data row3 col12\" >-0.02</td>\n      <td id=\"T_81e05_row3_col13\" class=\"data row3 col13\" >$-0.00058</td>\n      <td id=\"T_81e05_row3_col14\" class=\"data row3 col14\" >+0.00</td>\n    </tr>\n    <tr>\n      <th id=\"T_81e05_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n      <td id=\"T_81e05_row4_col0\" class=\"data row4 col0\" >Round 4: split workflow</td>\n      <td id=\"T_81e05_row4_col1\" class=\"data row4 col1\" >0.98</td>\n      <td id=\"T_81e05_row4_col2\" class=\"data row4 col2\" >100%</td>\n      <td id=\"T_81e05_row4_col3\" class=\"data row4 col3\" >2.6</td>\n      <td id=\"T_81e05_row4_col4\" class=\"data row4 col4\" >0.0</td>\n      <td id=\"T_81e05_row4_col5\" class=\"data row4 col5\" >2,412</td>\n      <td id=\"T_81e05_row4_col6\" class=\"data row4 col6\" >2,974</td>\n      <td id=\"T_81e05_row4_col7\" class=\"data row4 col7\" >1,779</td>\n      <td id=\"T_81e05_row4_col8\" class=\"data row4 col8\" >1.41</td>\n      <td id=\"T_81e05_row4_col9\" class=\"data row4 col9\" >$0.00204</td>\n      <td id=\"T_81e05_row4_col10\" class=\"data row4 col10\" >-272</td>\n      <td id=\"T_81e05_row4_col11\" class=\"data row4 col11\" >+290</td>\n      <td id=\"T_81e05_row4_col12\" class=\"data row4 col12\" >-0.44</td>\n      <td id=\"T_81e05_row4_col13\" class=\"data row4 col13\" >$-0.00040</td>\n      <td id=\"T_81e05_row4_col14\" class=\"data row4 col14\" >+0.00</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "round_impact = summary[\n",
    "    [\n",
    "        \"variant_label\",\n",
    "        \"mean_quality\",\n",
    "        \"policy_compliance\",\n",
    "        \"mean_tool_calls\",\n",
    "        \"mean_extra_tool_calls\",\n",
    "        \"mean_sync_tokens\",\n",
    "        \"mean_total_tokens\",\n",
    "        \"mean_cached_tokens\",\n",
    "        \"p50_latency_s\",\n",
    "        \"cost_per_ticket_usd\",\n",
    "    ]\n",
    "].copy()\n",
    "\n",
    "for col in [\"mean_sync_tokens\", \"mean_total_tokens\", \"p50_latency_s\", \"cost_per_ticket_usd\"]:\n",
    "    round_impact[f\"{col}_delta_vs_previous\"] = round_impact[col].diff()\n",
    "\n",
    "round_impact[\"quality_delta_vs_previous\"] = round_impact[\"mean_quality\"].diff()\n",
    "\n",
    "display(\n",
    "    round_impact.style.format(\n",
    "        {\n",
    "            \"mean_quality\": \"{:.2f}\",\n",
    "            \"policy_compliance\": \"{:.0%}\",\n",
    "            \"mean_tool_calls\": \"{:.1f}\",\n",
    "            \"mean_extra_tool_calls\": \"{:.1f}\",\n",
    "            \"mean_sync_tokens\": \"{:,.0f}\",\n",
    "            \"mean_total_tokens\": \"{:,.0f}\",\n",
    "            \"mean_cached_tokens\": \"{:,.0f}\",\n",
    "            \"p50_latency_s\": \"{:.2f}\",\n",
    "            \"cost_per_ticket_usd\": \"${:.5f}\",\n",
    "            \"mean_sync_tokens_delta_vs_previous\": \"{:+,.0f}\",\n",
    "            \"mean_total_tokens_delta_vs_previous\": \"{:+,.0f}\",\n",
    "            \"p50_latency_s_delta_vs_previous\": \"{:+.2f}\",\n",
    "            \"cost_per_ticket_usd_delta_vs_previous\": \"${:+.5f}\",\n",
    "            \"quality_delta_vs_previous\": \"{:+.2f}\",\n",
    "        }\n",
    "    )\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04286bf4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.147058Z",
     "iopub.status.busy": "2026-04-30T18:28:05.146999Z",
     "iopub.status.idle": "2026-04-30T18:28:05.256348Z",
     "shell.execute_reply": "2026-04-30T18:28:05.255961Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<Figure size 1600x400 with 3 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABjMAAAGGCAYAAAA+ZHMbAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAA0I5JREFUeJzs3QmcleP///HPtO97ioQUkRZLVJTsJUqWJJKlSGhBlmTfotBCCdlKkrWESlFSURRKRGgv7fs+zfk/3tf3f5/ffc6cmWY5M+fMnNfz8TiPmbnPfc7cc885576u63Ndn09SIBAIGAAAAAAAAAAAQJwqEOsDAAAAAAAAAAAASA/BDAAAAAAAAAAAENcIZgAAAAAAAAAAgLhGMAMAAAAAAAAAAMQ1ghkAAAAAAAAAACCuEcwAAAAAAAAAAABxjWAGAAAAAAAAAACIawQzAAAAAAAAAABAXCOYAQAAAAAAAAAA4hrBDABRU6xYMevVqxdnNId99NFHlpSUZD/99BPnGgCQI8qVK2d33nlnvjy7devWtUsvvTTWhwEAQL73+eefu77rDz/8kO42AMgoghlICCeccIK7WJ544okR7//ll1/c/bo99dRTuX58SBxew23mzJmxPhQAQB5zzDHHBNsr4bdSpUrlq0kI8XxsOSUR/2YAQM5asWKF3XHHHVarVi13ndFkhWbNmtmIESPs4MGDcXP6mbAHIKMIZiBhlCxZ0hYvXmxz5sxJdd9bb73l7gcAAIhnp512mgUCgVS3nTt3RvX3bN261V5++eWoPicAAMg906ZNs3r16tnChQtt5MiR7tq+ZMkSu/baa+3uu++2Vq1a2d69e3P9X6LVkWq7NG7cONd/N4C8j2AGEoZSChx//PH29ttvh2zfv3+/jR492q666qqYHRsAAAAAAEA0/Pfff26MQ9kppkyZYmeeeaZbmVG5cmXr1q2bjR8/3m3v3bs3JxxAnkIwAwnlhhtusPfffz9k9sGECRNsy5Yt1qlTpzQfpwDI6aefbiVKlLDSpUvbRRddlKpewcknnxxM9VCoUCGrVq2a3XjjjbZ27dqIKQSUH1IzEfTzsccea6+//nqGl1/qcVoeWrFiRTvvvPNs8uTJwftPPfVUdyyRNGrUyOrUqZPlY9FsDu2nVBaVKlWytm3b2m+//ZZqv4w8n/e7Z8yYEdz32WefdfctXbrUzRZRQ6to0aJWu3Ztl/4rOTk5Yt2I559/3qpXr27Fixe35s2bRzymjDznoEGD3HOq4een59P2d999N7htz549dv/991vNmjXd761Ro4Z17tzZVq5caWnRDNfWrVu777W013u9aIlvZo4zks2bN9s555xjVapUCck9qtd7kyZN3Moj3fR6mTVrVshj9b/s0qWL/fzzz9a0aVP39xx99NE2ZMiQVL9H75ezzjrLypcv725nn322ffbZZ+keGwAg9+nz/4wzzrCyZcu6z/kLLrjAvv76a3efrnO6/uzbt88GDx4cvB61bNky3ZoZ2ueBBx6wqVOn2imnnOLaRbqGL1iwwN3/1VdfuXaIrumaRDJ9+vQ0U3967aUjjzzSXYPWr1+f4WMTXTu1SkXXrDJlytjFF1/s0ob67dq1y7p3726HHXaYa7toJuiqVasydR7feOMNdx71eF2br7zySrfS1z8p5rHHHrPjjjvOihQp4q7D119/far2QHb/HwAAZIb6nuojqo+tfmW4c8891/VNX3vttWD/V/vq+qMVHH7qc2u7+uDh27ybrsf169eP2IcMF14zQ/35du3aue817uI9p/rf8+bNc99HWi2qFSe678UXX0zzd6ktoICNxiV0jPratWtXW716dch+GzdutJ49e7r7db40EfbRRx91j/doVUv79u3ddVz7qE3z3HPPhaTr0jHrmNQ2euKJJ9w4RYECBdzzi/riarOoL632ksZu3nvvvUOeMwA+ASAB1K5dO9CoUaPAypUrAwUKFAi8//77wfsuueSSwAUXXBBYsmRJQG+JJ598MuSxvXv3DhQtWjQwePDgwLp16wKrV68O3H777YFixYoF5s+fH/H37d69OzB79uzAySefHDjjjDMCycnJwfv0XPqd7du3D/z555+BTZs2Be6++273u7///vt0/44ZM2YEkpKSAs8++2xg48aNgW3btrltF198cfB3vP766+65Zs2aFfLYefPmue3PP/98lo6lV69egcKFCweeeeaZwIoVKwKbN28OjB8/PtClS5csPZ/21XG3bds2sHjx4sDy5csDEyZMcP+jypUrB0455RR3zFu3bg2MGjUqUKJEiUC7du2Cj//www/d81533XWBF1980Z0PPU+9evUCxx13XMg5z+hzDhw40D3n2rVrQ4514cKFbrse47ntttsCVapUcedf/2/9jrfeeitw3333pfs/1N+o5/ruu+9S3ZfZv/3HH390P//111/ubz7ppJMCS5cuDe732GOPBQoVKhR47rnnAmvWrAn8999/gXvuuSdQpEiRkNdHxYoVAy1atAhcddVVgd9//z2wZcuWwEMPPeR+x5QpU4L76ZgKFiwYeOSRRwLr168P7Nixwz3PZZdd5o4VAJCzjj766MBpp512yP302a3PcF3XdL1We2HatGmBli1bproW9+zZM+JzlC1bNnDHHXeEbNNz6jp//fXXB5YtW+aulxdddFGgWrVqgenTpwc6dOjgrkO63rRq1SpQvnx5d61Iy65du9z1sG7duoFmzZoFDh48mKFju/POOwPFixcPDB06NLBhw4bAqlWrAp07dw6ULFky8NtvvwX307WtQoUKrr2ic/Dtt9+6toeumfo7DkXXeh3HgAED3DVabY1PPvkk0K1bt+A+bdq0CZQuXTrw3nvvuWuhrs1qixx++OHB9kQ0/h8AAGSG2gulSpUKubaGGz58uLs+jR492v3cr18/97P6g366tmm7+qFp0fVYz6e+5iuvvJKq/+sfD4i0LbyP66exnDp16kS8TmuMQn3TtNx4442unaJ+6549e9xYxogRIwJ9+/YN7qPH16hRw7UPJk+e7K7TGh/S2NCYMWPcPmrfqE2h8Z1ffvnFnaM333zTjQupXeRR/11/h/rvgwYNcuflgw8+cGMj48aNc/3zW265JfDPP/+459A509/gP2cA0kcwAwkVzBB1ur3OowZ4NTiri3ekYIY6xAoePProoyHPl5KSEqhfv/4hO8Lq2Os5f/rpp5COqgasd+7cGdy2b98+N6B80003pft8Tz31lAvGHDhwIN2BgXLlygU6duwYsl2dfF0kFZDJ7LHoYq2/48EHH0z3+DLzt2nfMmXKBLZv3x6yXYEiXeB1cffzGlZeEMBr7Nx6660h+2nAQtu//vrrTD9nZoIZJ5xwghu0yaz0ghmZ/dvV0NNrTI0qva7V6PLoOfTaVvAiXJMmTQLNmzcP/qz/j/4X/karGr1HHnlkSBBFjTH9Xv/vAQDkbjBDn8ORbv42iQLSus6qvZKerAQzNCCwd+/e4DZN7ND2WrVquUECz6+//uq2v/POO4f8uyZNmuT29Qci0jo2tam0r66NfprEoGvzlVde6X5WO0D7aZKH39SpU1Odr0jmzJnj9nviiSfS3Oebb75x+2hShZ8mV6i95h1/NP4fAABkhvrlui6mZ+LEie46pgmL2Q1meDQOoUmd0QxmqC2h+9T39WgcQZMJvOt+Wo455phDjrNokoT64ZqQmZabb77ZBWo0ucFPY0X+4/aCGQpY+O3fvz9wxBFHhPTD/b9ffXqNnQA4NNJMIeEo9ZNyQ65Zs8ZGjRrl0gZcfvnlaS5/VN/dW/Lo0bJBpev59ttvg9u0jFCpB6pWrerSJmgfpf2Rv//+O+Tx2u4vOK60BFrG+O+//6Z77A0aNLCUlBTr0KGDS8+k1AbhlPJBf+OHH34YXMqoZaJjxoxxy0iVaiGzx/Lll1+6r0p/dCiZ+du0tFVpu/yUckFLLbW808+raeKlZPBccsklIT8rrYX4f19mnzMj9L8YN26cW4r7119/WTRk9jiV9uvCCy+0q6++2r744guXZsMzceJEt9w1/LUr559/vlve6k9dpdRRSini0VJY5Vf1n0f9zdKxY0f75ptvYlIsDgASXVoFwNVm8X9eK2XRddddZzNnzozYXsgqtX/86Sp0rfDSQihdQvj28Ov//PnzXZpKpWMqWLBgSDql8PZSJN7fGX5903OpDeK1zbxrZps2bVJdA9X2OxRdVw/V9vF+xxVXXBGyXSkiVXDVuz8n/x8AAETyvzkIGdtH19CsePXVV10KRfXp/amhMnI9zwwvtdOwYcOC2zSWs2PHDpfqOT26Bn/wwQculVVax6XxjoYNG7pxi7Tomq6/VekxM9JXD29/qP2jMahI/XOlnlRKMKXNAnBoBDOQcBS4UCdWFz/VwtCFUbkTI/FyR+oCqACFLvIa5NVN9RV27tzpOqTKi6xaA/pZ9St0UVXDwKurceDAgZDnPfzww1P9Lg1Eh+emDKdcz8pp+ccff7jaEMq7rPodkyZNCtnv9ttvd8f15ptvup/feecd2717d8QLfUaOxctjrTogh5KZvy3S823atMkFhMJ527wATVq/zxvQ9/++zD5nRhqCakgpaPTCCy+4QYsjjjjCbr311nRrZhxKZo9TAarChQvbbbfd5l6fkV67ClKEv3a9Ghzbt2/P1P9Ng0QKoKxYscINBun1p0EtFY8DAMQPdayHDh3qakioRpOC1QoYqNZFdoVfLxTA0DUmfLuuTwp6+K8jCmzoeHQN0rGo3aRrrAb4I7WXIvGub7r2hl/fhg8f7q6loq/apjoX4RRIOZSMtH2835XWtdu7bufk/wMAgEhUA1F1ojQZMi3q18lRRx2V6f6wggMqJK6JlqolpWu79lPfNCPX88xQe+Lmm2+2Tz/9NNgOeOWVV1xgoUWLFuk+VjW2NDGhX79+rr6VHqPjVmDBf80/1FhHZvvq4c/nHXePHj2C7RevDaNJHt7vAHBoBDOQcNTpVgBDhZoUFLjpppvS3FfRf/nnn3/cxVkz3dUY0M2bCamVB7qoKoChC6oCH15wRMWcI9GMhay65ZZbXEHqdevWuYCMClG3atXKvvvuu+A+ukgruq9OvY5ZX3UxjXShz8ixeAMB4UWysvu3aaAjXIUKFdzfFs7b5v1PMvP7MvqcGpwX/S/9Iv3dek4FNPQcixYtsnvvvdc++eQTN+CfXoMxGsfpUaEwvd70OyMV9ZZff/01zdeufl9m/28qaqrBmA0bNrhipnqcGl/hATUAQGxpYsPvv/9ua9eudZM3tm3b5toBc+bMydbzpnW9yMh1RIVDNblCEzO0cuFQ7aVIvOubrs2Rrm/eNbhixYrue12v0gpUZLft411H07p2+6/bOfX/AAAgEl1jNGlAGR3SohUJGs/QBLXM9oc1yU390LvuusuNNXirOzJzTc8MBSB0zVdwQmMfGhPR5EIFA9Kja7HaHWoPaOVDr169XD/W+5u9a/6hxjoy21cPH+vw7teEU6/94rVhvP65JqoCODSCGUhICmBs2bLFTjjhBGvcuHG6KyFk7NixGXpef9oF7wKfU5QuSjP9FNDQhc8fzPA6zWpI3HfffW6mhC70WV0+6qVy0uB5TtOM/59//tmWL18esv3jjz8O3p9Tz1mzZk33VQ0jP3/qjnBqPNWpU8c14tQw0qxTzYBJi5eCS+kmsnqcnvLly7uUaVruqoaPVgV5FODSsWX0tZtZaoxplZP3/Ok1kgEAsaMZg0pH+NZbb7kOsz/4rWtSpOtRTstIeymtY8to28wbpJgwYULIdqVJDB+kicT7Pem1fbzrsia1+C1ZssQNmERqs8Tj/wMAkP90797drQR84IEHIqY3nDZtmrtG3nDDDcFU1JntD4dfz9UP1vNmRXr9ZDnmmGPs4osvdoGJl156yU2iSG9iajj1jZWSunfv3nbHHXfYn3/+GZzcoGu+smqklz5a1/S5c+eGrOjIzDiF0nFqZahSXgHIHoIZSEgKYCgAoJUZ6dGsd824f+SRR2zAgAFuGaZWQmgmvpZV3nnnncFZD5rRoAujLoi6wN1///2pLu7Z9fDDD9ujjz7q6nNoZqN+l1YH6EKutFN+qo+h5aIvvviiu1/LMrNK56Fnz57uHKhGhBopCgap8aPUStHUp08f1+hS/RGtAFAqJA0kPPnkky4ntdJ55dRz6qsCXDrHCgApb6UaSv50TP5BEj3HsmXLXINLrwkNZmhVTHpLVPX8WlaqlQzhNSey8rer0af/g4IX+p97jSMdh/4O/b+UVkrHqdeuXvODBw+2Ll26ZPo8PvPMM/bggw+6gMuuXbvcUlqdH/HqwwAAYk9tEF07NKCuz37NGtTqUXXkzz777OB+6tTPnj074kzDnKBBCF0D7777bjdDUu0JTQbwZoL6pXVsjRo1cgM0umYOHDjQpXdUm0gDL7rm6bm967TqSulcaBBGAQyls1K7SNfIQ9FEga5du9rTTz/tUkpqxqbaBaqXpQkj3u/QhA+1E3X91XVbObGVD1uzPPW74/n/AQDIvxQ8Vx1NrQrU9VDXF/Vb1YfTNUir688880zXN/Touqb6jQ899JALzGtfXQMjBRhUE0IT63T9U99QwQBlwFCGiKzQBEFdF1X7Ma2AhoIQuu7r71L9zfBak5EovaMmQGjCoJ5XYymfffaZnXTSScFVmBpnqV69uutP62/S9VzZOdSP1ioO0TnRilL11fUcWmGpdN5qeyiNlWpupEfjRaox8tVXX7n03+qXq02gCaijR4/O8nkDEhHBDOAQ+vfv7y5S6girk6kZ6ddcc43LZ6iOtJe3WZ1bRfc1Y6BJkyauY67ObTQpoKCLoGZP6MKr49Ggt2bkqzaCn1ZhqBPuDTRn5EKfHtUIUboqpYhQYSz9zZpVqGOKJjUifvjhB6tVq5a7oOt8a1BeQSWvIZFTz6nGk2ZWKDWFCnHr/Koho/3CKbCj865GlAIQGszQwIdmoqS3AkY5xYcMGeJ+j2q3KNCkpbLZ+dv1mlADTa8L5Sx9/fXX3XZvcEWzUPX36O/SAIsagFl5bWppr/5WBUI0q0TFXTUwpGAKS2IBIHfMmzcvWGQz/OatDLznnnvcrP+OHTu6z/769eu72Yaq0eDvbGsAQx1ztV38hbhzilJL6fqntosmXGiAQTWnvEF/v/SOTddRzczUc2kwQjNKVVxb6TS0IlX0GKV/1MCKZm5qUEeBCU0C0XUzIzTYo+PQxAJdm3XdU2FT5bv26BjUFlKwX9dtTXDRMel67tURidf/BwAgf1OfUmmHdV3SNUg1ETWOoKC8alt8++23IfVDNeFAE/Q0YU7XbPUhdf2K1OdXAEATEjSJQM/pTX7MyISBtGp8aMKBrrP6/V4xcT9dF73VI4cq/O1RMEb9VU3+VF9WQRx9r2uwlyJTx6+0jwr66Hn1syYLKh2UAhyi8RRd21VzQ2MA2keT/RTk0HhRRlx22WX2/fffu3pi3vHodyrAoXMHIGOSApEq+QDIF15++WU3e1GNAHXyAQAAAABAYtKKC02EVA0Ipar2VifkFQqwaJWFalCpHiqAxMPKDCAf02x9NVK0FBIAAAAAACQurZxQKielxtbqeqVLyiuUCUMrTa6//noCGUACKxTrAwAQfVpwpaWUSgGkPI/MWAAAAAAAAKeddlrEupDxTPWxlCpZYxtefSwAiYmVGUA+owKYqv2g2Qoq+u3ljgaAvEq5erWcXMV3c+Ix2m/x4sVR+90AACD+J399/fXXrnbdli1bMvSYvXv3uhntKtar4r0AckevXr1cvckff/zRRo0a5WpLAUhc1MwAAABxSwXsVdxes7FU3FeF/1QHqHTp0lF5jAr5nX322XbgwAF3U+HD7PxuAAAQ/6l4VbxYhY8XLFhgCxcutLp166b7mJUrV9p5551nhQsXdml6FAi555577PHHH8+14wYAAKzMAAAAcWr9+vV2+eWX20033WQbN260VatWuRUUPXr0iMpjlCO4Y8eObt9o/G4AABD/ihYtal988YW98847GX5Mz549XaFk5esfP368ffTRR/bEE0/Y3Llzc/RYAQBAKNJMAQCAuDRmzBiXBqJv376WlJTkBhF69+5t7733nu3YsSPbj7nlllusffv21qxZs6j8bgAAEP/atm3rVldklFZnqh5h165d3coM0WrN448/3t5///0cPFIAABCOAuBRptzaa9ascSkoNPgBAEBu0yC8BtyPOOIIV0Mnr1Je3JNPPtnNoPSceeaZtn//fpcSQt9n9TGvvvqqLV261OW9VrqJaPzucLQJAADxIL+0C2Llzz//tOTkZKtTp07Idv2seoVp2bdvn7v52wWqwVWxYkXGCgAAMRHIB20CghlRpkBG9erVo/20AABkmvI7H3nkkXn2zG3atMkqVaoUss37WamfsvqYRYsW2UMPPWSzZs0KzrCMxu8OH7RYvXp1qoEPAABiJa+3C2Jl+/bt7mv58uVDtleoUMHWrl2b5uP69etHTQ0AQFxamYfbBAQzoswrCqoXRZkyZaL99AAAZKjTrcB6Xi9UXbBgQbcSws8LFvgLdWf2Mddee63dfPPNboak6mB4AxGaeXn44Ye7wYms/O60Bi1oEwAAYim/tAtiRYXCJTzNpH727oukT58+dvfdd4fU6jrqqKNoF+RRu3/6yba+974d3LIluK1g+fJW7tprrETDhjE9tvyOc8/5T1Q58drfng/aBAQzosxLLaVABsEMAEAs5fV0h2pkhRfW/O+//9zXtGaRZPQxKt6pm3/G5ZVXXmn33HOPq6WRld8dPmjhNRRpEwAA4kFebxfESs2aNd3XZcuW2SmnnBLcrnSVSkmZFqWq9Ker9NAuyHt2z51r+18bYSX0g39V786dbnuZUqWsxBlnxPAI8y/OPec/UeX0az8pD7cJCGYAAIC4dM4559hrr73mgghVq1Z127744gtXjPukk05yP2v1xL///uuCBiVLlszQY3799deQ3/Puu+/a9ddf7/Jee6suMvI8GR20AAAAecvUqVPdisxLLrnEXfsbN25sY8aMscsvvzy4mnPevHn28MMPx/pQkcMCKSm25e2R6e6z6ZVXbf/KVZaUR/PPx/O53/H5F+nuw7nn/Cfqa3/LO6OseMOGCfm5QzADAADEJa2UqFevnrVv396efPJJF7QYMGCA9e/f36WBEm078cQTbeLEidayZcsMPSZavxsAAOQ9CxcutDlz5rh0T/LJJ5/YDz/8YGeeeWaw1tXLL79sW7dudcEMGThwoJ133nnWsWNHq1u3rpvwcPHFF1vr1q1j+rcg5+37Y7Ed3Lw53X0Ce/bY9g8/4t8RA5z72OL8x87BTZvc51OxkxKvRiPBDAAAEJe0SmLKlCn26KOPWs+ePV1aBg0uqN6FRyshateubaVKlcrwY8KVLVvWPYd/qW1WngcAAMQ/1cpS8EI6d+5sK1ascDelk/KCGRdeeKHt3r07+BitzNDKzlGjRtmqVavskUcecYGNvJymAxlzcOvWDO1XtM6JVqhKFU5rFCWvW2f7fv+Dcx8jnP/4P/cHM/j5lN8kBQKBQKwPQoWzVq9e7YphlSjhsoGlsmHDBjewUL58+XSfZ9OmTVatWjUr7M8nlgP7pEX5sTUoouJe1MwAAMQC16L4wP8BABAPuB7FB/4PedPeRb/b+iefOuR+hz38UELOkM5JnHvOf6LKydf+9nwwbh3TxFqLFy+22267zWrUqOFSRIQX2pRXX33VjjvuOJefWvvp64wZM0L2SU5Otq5du1qlSpXsjDPOsCpVqtjo0aNzZB8AAAAAAADkf0VPPMGSihVLd5+CFSu6/RD9c1+wQgXOfYxw/mOHcx/HwYyvv/7aGjRoYN99913E+w8ePGg///yzTZo0ydavX28bN250yz3btGnjvvc8++yz9umnn7rcl9rv+eeftxtuuMEWLFgQ9X0AAAAAAACQ/+39dYEF9u5Nd5/yN1yfkEV4c5rOafkbO6W7D+ee858f8dpPX0w/be+44w7r1q2blS5dOuL9KrA5fPhwl7tSlGbq/vvvd0thfvrpp5DVG126dLHjjz/e/ax81rVq1bLXX3896vsAAAAAAAAgf0veuMk2DR3mvi/WoH6qVQJakVHp7l5W4owzYnSE+Z/Orc4x557zn2h47eejAuC///67+3rkkUe6r//9958rwKWCXH5NmjSxefPmRXUfAAAAAAAA5G+B5GTbOHiwpezcaUWOrWGVe9+jGbe274/FruhuwXLl/peCihUZuTKoW7xhQ859jHD+Y4dznw+CGSrM3b17d7v44outbt26bpsKdYvqXPjp51mzZkV1n0j27dvnbv5CKgAAAAAAAMibto553/Yv+duSSpSwSr16WlLhwm47Rb5jQ0Ejzn3scP459/EkzyT127Nnj1122WVWoEABGzVqVEgqKtm/f3/I/gowKC1VNPeJpF+/fq4KvHerXr16Nv9SAAAAAAAAxMLuuT/aji++dN9XvP02K3TYYfwjACBO5ImVGXv37nVFv9etW2fTpk2zihUrBu+rVq2aJSUl2dq1a0Meo7RRXiqqaO0TSZ8+fezuu+8OWZkRzYBG6z5jLVFN6Nc+1ocAAEDcuHZkW0tk73UaF+tDAAAA+VzyunW2afir7vvSl1xiJRo2jPUhAQDy0soML5CxevVq++abb+ywsIi4ioefdtppNmnSpOA2ra6YOnWqnXPOOVHdJ5KiRYtamTJlQm4AAAAAAADIOwL799uGQYMtsHu3FTnuOCvXgQmWABBvYroyY9u2bW4lhFZcyIoVK2zx4sWuToVuBw8etMsvv9x+/fVX++CDD2zLli3uJocffrhL6ySPP/64C3jUr1/fFex+8cUXrVixYnbbbbcFf1e09gEAAAAAAED+suXd0XZg6TIrULqUVerZw5LSSTkOAEjAlRlaadG2bVvr2rWr1a5d25555hn38/vvvx8s+L106VIrX76820f3ebcpU6YEn6dVq1Y2btw4mzx5snXr1s0FIGbOnGkVKlSI+j4AAAAAAADIP3bNnm07v/rfOFPFO+6wQpX+L705ACB+xDTMrFUXuqWlXLlybqVGRlx66aXulhv7AAAAAAAAIO87sGaNbX5thPu+zOVtrfjJDWJ9SACAvFozAwAAAAAAAIi2lH37bKPqZOzda0Xr1LGyV13JSQaAOEYwAwAAAAAAAAlny1vv2IEVK61A2bJWqfsdllSwYKwPCQCQDoIZAAAAAAAASCg7p39ru6ZPN0tKsko97rSC5cvH+pAAAIdAMAMAAAAAAAAJY//Klbblzbfc92XbXWXFTjop1ocEAMgAghkAAAAAAABICCl799rGgYMssH+/FWtQ38q0vSzWhwQAyCCCGQAAAAAAAMj3AoGAbX59hCWvWWsFK1SwinfcbkkFGBoDgLyCT2wAAAAAAADkezu//sZ2z5ptVqCAVerZ3QqWKRPrQwIAZALBDAAAAAAAAORr+5cutS1vv+O+L9fhGitau3asDwkAkEkEMwAAAAAAAJBvpezebRsHDjZLTrbip51qpS+9JNaHBADIAoIZAAAAAAAAyLd1MjYNf9WS16+3gpUqWcVut1lSUlKsDwsAkAWFsvIgAACA3LRu3TorVaqUlSxZMqqPWb9+vVWoUMEKFQptEiUnJ9uqVatS7V+lShUrXrx4Jo8eAAAAsbJj4iTbM/dHs4IFrVKvHlagVCn+GQCQR7EyAwAAxK3vvvvOatWq5W7ly5e39u3b2+7du7P1mO3bt1vfvn2tcuXKVq9ePRfsuOKKK2zDhg3BfZYtW2Y1atSwpk2b2jnnnBO8zZ07N0f/XgAAAETPviVLbOvo99z35a/vaEVr1eL0AkAeRjADAADEpU2bNlmbNm3syiuvtK1bt9rSpUvtp59+sl69emXrMYsWLXIrLP7++2+3emP58uX2zz//WNeuXVM938yZM11gw7s1b948x/5eAAAARM/BHTts4+CXzA4etOKNzrBSLS7i9AJAHkcwAwAAxKX33nvPDhw4YI8//rgVLFjQqlWrZvfee6+NHDnSdu7cmeXHNGnSxHr06GFly5Z1P1etWtVatGhhixcvTvV8WtHhX7EBAACA+BdISbFNw16xgxs3WqGqVaxi11upkwEA+QDBDAAAEJd+/PFHO/XUU61YsWLBbUr7tG/fPlu4cGG2H7NmzRq3OmPcuHH27rvvWvfu3VM9n57LS1f1xBNP2MGDB6P6NwIAACD6dkz43Pb+/ItZ4cJWqVdPK1CiBKcZAPIBCoADAIC4pBURFStWDNlWqVKl4H3ZfcxNN91kCxYscKmmrr76auvUqVPwvqJFi9qwYcPcPgqMTJkyxS6//HK32kP1NiJRwEQ3f20OAAAA5K69f/xhW8d+4L6vcOMNVuSYY/gXAEA+wcoMAAAQlwoUKGDJyckh25RCShRUyO5jJk+ebGvXrrWVK1e62hoKVniqV69u3bp1C67wuPDCC+3OO++0119/Pc3j7devn0td5d30HAAAAMg9B7dts01DXjJLSbESTZtayfPO5fQDQD5CMAMAAMSlI4880v7777+Qbd7PqoURrcdo+wMPPOBWX2zevDnN46lRo4atWLHCUlJSIt7fp08f27ZtW/CmIAkAAABysU7Gy0Pt4JatVqhaNavQ5WbqZABAPkMwAwAAxKVzzjnHfv7555D0UJMmTXJppOrWrRtcdbFs2TLbs2dPhh8Tqe6FUk0VKlTIpZdKa5/Zs2e7gIZWf0Six5YpUybkBgAAgNyx/dNxtnfhb5ZUtKhVvqunFfDVUAMA5A/UzAAAAHHpyiuvtKeffto6dOhgzzzzjP3777/23HPP2ZNPPukCD/LPP//YiSeeaBMnTrSWLVtm6DGqeVGiRAm74IILXDqoH374wR566CG75ZZbrGTJkm6fRx55xO3fokULK168uH300Uc2atQoe/vtt2N6TgAAAJCaghjbPvrYfV+h881W+MgjOU0AkA8RzAAAAHGpSJEiNnXqVHvwwQfthhtucCsd+vfvb7fffntwn8KFC9vRRx/tAg4Zfcyjjz5qL7/8skstpbRSevzgwYNdAMTz8MMPB/dRyqjjjz/eZsyYYU2bNs3lswAAAID0JG/eYhtfftksEHA1Mkqe3YwTBgD5VFIgEAjE+iDyk+3bt7tZnhr4iEZ6idZ9xlqimtCvfawPAQDypGhfixAf/4drR7ZN6H/Fe53GxfoQACBPol0QH/g/5IzAwYO2/qmnbd8fi63wUUdZlaeesAJFiuTQbwOAvG17PhgroGYGAAAAAAAA8pxtH3zoAhlJxYtbpV49CWQAQD5HMAMAAAAAAAB5yp6ff7bt4z9z31e49RYrfMThsT4kAEAOI5gBAAAAAACAPCN540bbNPQV932pFhdZySaNY31IAIBcQDADAAAAAAAAeUIgOdk2Dh5iKTt3WpFjj7XyHa+L9SEBAHIJwQwAAAAAAADkCVvfG2P7l/xtSSVLWKVePSypcOFYHxIAIJcQzAAAAAAAAEDc2z33R9vx5UT3fcVu3azQYYfF+pAAALmIYAYAAAAAAADi2oH/1tmm4a+670tfeomVaHharA8JAJDLCGYAAAAAAAAgbgX277eNgwdbYPduK3L88VbumvaxPiQAQAwUisUvBQAAAAAAiIXNmzfb+PHj3deGDRta8+bND/mYv//+26ZPn27bt2+3Y4891lq1amVFihTJleOF2ZZR79qBpcusQOlSVqlnd0sqxHAWACSimK7M2LNnj7399tvWuHFjK1eunM2cOTPifm+++aY1aNDAqlatahdeeKH9/PPPMd0HAAAAAADkPX/++aedeOKJ9tZbb9lff/1lV155pXXt2jXdxwwbNszq1q1rX3/9ta1atcoefPBB9/OGDRty7bgT2a5Zs23nlKlmSUlW8Y47rFDFirE+JABAIgYzHnvsMZs2bZrde++9tm3bNktOTk61z5gxY6xbt25un1mzZlnNmjXtvPPOs7Vr18ZkHwAAAAAAkDf16NHD6tev71ZZvPrqq/b555/ba6+95n5Oy+OPP269e/d2YwYvvviizZ0719atW2fvvvturh57IjqwerVtfu11932ZtpdZ8ZMbxPqQAACJGsx49tln7Z133rFGjRqluU+/fv3spptuso4dO7rgwtChQ61o0aL2yiuvxGQfAAAAAACQ92gS5dSpU+3mm2+2AgX+NxyiTBFaZfHRRx+l+bhSpUpZIV9aIz02KSnJSpcunSvHnahS9u2zjYOGWGDfPit6Uh0r2+6qWB8SACCRgxm6+Kdn69attnDhQjv//POD2woWLOhWS3gpqXJzHwAAAAAAkDcprVRKSoodf/zxIdv18+LFi9N83KhRo2zChAnWuXNn69u3r11wwQXWoUMHu+GGG9J8zL59+1x9Df8NmbPlrbftwMqVVqBsWavU/U5L+v8BKABA4orrK8GaNWvc1ypVqoRsP+yww4L35eY+kdBAAQAAAAAg/u3cudN9LVu2bMh21fD07ktrRYcmQO7YscN27drlAhP6WeMBaVHmB/0e71a9evUo/iX5387p023X9G9dnYxKPe60guXKxfqQAABxIK6DGR5v+adHyzsDgUDM9vGjgQIAAAAAQPwrWbKk+xq+SkLBCu++cLqvffv2duedd9oHH3xggwYNsh9//NG+//57e/rpp9P8XX369HGP9W4rV66M8l+Tf+1fscK2vPm2+77s1e2s2EknxfqQAAB5OZihGQhZuS+ztCpCNm7cGLJ9/fr1wftyc59IaKAAAAAAABD/jjvuOJfu+u+//w7ZvmTJEqtdu3bEx/z7779uRUbz5s2D24oXL26nn366zZ8/P83fpfqbZcqUCbnh0FL27LGNgwZbYP9+K9agvpW5rA2nDQCQvWBG+fLls3RfZlWqVMmOPfZY++6770K2z5gxI1g0PDf3iYQGCgAAAAAA8U/jFeeee669/fbbwQwM8+bNswULFtgVV1wR3O/DDz90dTJE4wSqpzlr1qzg/UovpceF195A9uh/svn1EZa8Zq0VrFDBKt5xO3UyAAA5l2Zq7969VqxYsWg+pfXs2dNGjBjhGg779+93aZ3+++8/69q1a0z2AQAAAAAAedPgwYNtzpw51qJFC+vVq5e1atXKOnXq5Ip6e0aPHm1vvPGG+171Lp555hnr3bu320/ZGbQq4+DBg+57RM/OqV/b7tnfK/+3VerZ3QqymgUAEKaQZcKzzz4b8XtJSUmxn376yerWrZvh53vvvffs9ttvD86IuPTSS12NigceeMDdpHv37i71kxoYKrR1zDHH2Lhx49zyUE9u7gMAAAAAAPImjVn88ccf9tFHH9nmzZvtnXfesZYtW4bsc/XVV9uePXuCP993331unODbb7919S/69u1rl112WdQncyay/UuX2pZ3Rrrvy3W4xoqmkfYLAJDYkgLpVbcO07BhQ/dVyylPO+20kPsKFy7sBv8ffPBBq1evXoaeT6sfdu/enWq7GgThjQIdplZ+KDdlWnJzn7SokJhmbqiBE42cmK37jLVENaFf+1gfAgDkSdG+FiE+/g/Xjmyb0P+K9zqNi/UhAECeRLsgPvB/SFvKrl32X5++lrx+vRU/7VSr1PseV9sEAMC1KFsrM7TyQjRrYdKkSZZdRYoUcbeM0IXsUMGF3NwHAAAAAAAAWafJpJuGv+YCGQUrV7aK3W4jkAEAiG7NjGgEMgAAADJK6R+Vmzraj4m0QjQavxsAAACHtmPiJNvz449mBQtapV49rECpUpw2AED0C4BPmzbNrr/+ejvrrLOC24YOHepSKQAAAETDjz/+aA0aNLDy5ctbyZIlrXPnzrZv375sPUYBjH79+tnRRx9tlStXtlKlSrmCnlu2bMn27wYAAEDG7FuyxLaOfs99X/76jla0Zk1OHQAg+sGMDz/80Fq3bm2lS5e22bNnhwwODBgwICtPCQAAEGLr1q2u2ObZZ5/t8kwvXLjQpkyZYr17987WY+bPn+9SGvzwww9u1YX2USrNbt26Zet3AwAAIGMO7thhGwe/ZHbwoJVo3MhKtbiIUwcAyJlgxlNPPWVjx461YcOGhWy//PLLbeTIkVl5SgAAgBBjxoxxwYbnnnvOihUrZscdd5zdd9999sYbb6SZHiojj2natKk9+OCDdvjhh7ufa9SoYZdeeqktWLAgW78bAAAAhxZISbFNw16xgxs3WqGqVazCrbdQJwMAkHPBjL/++svOO++8YLFsT5UqVey///7LylMCAACE0MqJU0891UqUKBHc1rx5c9uzZ09I4CGrj9HqC7Vbpk6d6oIXXbp0ydbvBgAAwKHtmPC57f35F7PCha3SXb2sgK+9BQBAegpZFlSqVMn++ecfq1u3bkgwY/r06S7/NAAAQHZt2LDB1bTw837Wfdl9jFJm/vrrr7Zjxw675ppr7Pbbb8/W71Y9DX9NDaWnAgAAwP/Z+8cftnXsB+77CjfdaEUYQwIA5PTKjBtvvNG6du1qv/32mwtmbNq0yUaPHu1mNKo4JgAAQHapjXHw4MGQbcnJye5rgQIFsv2Y7777zgUc/vzzT1u8eLFdffXV2frdKipetmzZ4K169eqZ+GsBAADyt4PbttmmIS+ZpaRYiWZNreS558T6kAAAiRDMePTRR92qjAYNGriOvlZqdOrUydXMuPfee6N/lAAAIOEcccQRtm7dupBt3s9evYtoPOb444+3hx56yCZMmGBbtmzJ8vP06dPHtm3bFrytXLkyg38pAABAAtTJeHmoHdyy1QofWc0qdL6ZOhkAgNwJZuzfv99effVVW716tX355Zf2+eef24oVK2z48OG2du3arDwlAABAiLPPPtvmz59vmzdvDm776quvrFy5clavXj33syZVbNy40Q4cOJDhxwQCgVRnWkGMggULWuHChTP8POGKFi1qZcqUCbkBAADAbPsnn9rehb9ZUtGiVqlXTytQrBinBQCQO8GMq666yg0aVK1a1S6++GK75JJLrFq1arZq1So75xyWCQIAgOxr166d1ahRw2644Qb7/fff3eSJZ5991q0C9YIOS5YscbUsvv766ww/RqswXnzxRVfIe/ny5TZ27Fi37frrr7dSpUpl+HkAAABwaHsXLrRtH3/ivq/QpbMVPvJIThsAIPeCGZqlqM69f2ajUikokNGwYcOsHQkAAIBPsWLFbOrUqVa8eHFr0aKF3X///da3b1+XzslTqFAhq1ixohUpUiTDj3nggQdcGijVAGvevLkNGzbMHn/8cXv99dcz9bsBAACQvuTNW2zjS0O1NNZKnneulWzWlFMGAMiyQll5kFJLNWvWzHr06GEvvfRSMJDRqFEjGzVqVNaPBgAAwOeoo46yDz74IM1zUqtWLZdmKjOPKV26tAte6JaeQz0PAAAA0hY4eNAV/E7Zvt0KH320lb/xBk4XACD3gxkVKlSwyZMn21lnnWUFChRwqRe8QIbyTQMAAAAAACBxbfvgQ9u3eLElFS9ulXr1sAL/fyUtAAC5GsyQI4880qZMmWJNmza1Cy+80EaOHEkgAwAAAAAAIMHtmf+zbR//mfu+YtdbrPDhh8f6kAAAiRTMKFeuXMTte/bssQkTJrh81Z6tW7dG5+gAAAAAAACQZyRv2GCbhg5z35dq2cJKNG4c60MCACRaMGPEiBE5eyQAAAAAAADIswLJybZx8EuWsmuXFal5rJW/7tpYHxIAIBGDGVdddVXOHgkAAAAAAADyrK2jx9j+v/+2pJIlrFLPnpZUuHCsDwkAkI8UyMqDkpOTbc6cOam2a5vuAwAAAAAAQOLYPXeu7Zg40X1fsVs3K3RY5VgfEgAgn8lSMOPJJ5+0yZMnp9o+adIke/rpp6NxXAAAAAAAAMgDDvy3zja98qr7vnTrS61Ew9NifUgAgHwoS8EM1c+45ZZbUm2/9dZb7c0334zGcQEAAAAAACDOBfbvt42DBltgzx4rWvt4K9f+6lgfEgAgn8pSMGPbtm0WCARSbU9JSbGNGzdG47gAAAAAAAAQ57aMHGUHli2zAqVLW8Ue3S2pUIbLswIAkPPBjCZNmtiAAQNSbX/uueescePGWXlKAACQTwwcONBGjhyZ6fsAAACQt+yaOct2Tv3aLCnJKt55hxWqWDHWhwQAyMeyFC7v16+fNW/e3GbMmGHNmjVzqzS+++47+/PPP2369OnRP0oAAJBnrF692g4ePBjxvhUrVliBAlmaSwEAAIA4cmD1atv8+gj3fZnL21rxBvVjfUgAgHwuS8GMhg0b2rx589zsylmzZllSUpKdfvrpNmbMGKtdu3b0jxIAAMQ9BSrWrFlja9eutf3799sPP/yQKk3llClT7Mknn4zZMQIAACD7Uvbts40DB1tg3z4relIdK3vVlZxWAECOy3IiwxNOOMFeffXV6B4NAADIs4YMGWIvvPBC8OeXXnop5P4iRYpYixYt7NJLL43B0QEAACBatrz5lh1YtcoKlC1rlbrfaUmsvAUA5ALyPAAAgKh45plnbMeOHXb33Xe7Olr63rvt3LnT9u7da5999pkVLlyYMw4AAJBH7Zw+3XZ9O8PVyajU404rWK5crA8JAJAgMrwyo1Ch/+2anJwc/D4t2gcAACQWrbzQzb86AwAAAPnH/hUrbMsbb7nvy17dzoqddFKsDwkAkEAyHMz4/PPPI34PAAAQiepj/PTTT9a0aVNr1qyZrVq1ylJSUuyoo47ihAEAAOQxKXv22MZBgy1w4IAVa1DfylzWJtaHBABIMBkOZrRs2TL4/aRJk2zQoEER9+vVq1fIvgAAIPF06NDBtReKFy/u0kopmKFUU9dcc43Nnz/fCpBXGQAAIM8IBAK2+fURlrxmrRWsUMEq3nE7dTIAAHmjZsbgwYPTvLip+CcAAEhcEydOtHnz5tk///xj1113XXD7CSecYJUqVXIrNgAAAJB37Jz6te2e/b1ZwYJWqWcPK1imTKwPCQCQgDK8MiMjfvzxR6tcuXI0n9IFSD7++GP78ssvbcuWLS41xc0332wNGjQI2e/nn3+24cOH27p166xevXqu+Gj58uVzZB8AAJA2pZbSyowKFSpYUlJSyH3HHXec/fXXX9aiRQtOIQAAQB6wf+lS2/LOSPd9uQ7XWNHax8f6kAAACSpTKzOKFSvmbv7vvZsKfjZq1MgFGqLp4Ycfti5durjAQqdOnWzv3r12+umn2/fffx/c54cffrAmTZq4NBbt2rWz6dOn21lnnWW7d++O+j4AACB9BQsWtB07dkS8b/HixS7IAQAAgPiXsmuXbRw42Cw52Yo3PM1KX9Iq1ocEAEhgmVqZ8dFHH7mvrVu3Dn7vUQDgmGOOsdq1a0f1AMeMGWN33HGH3XXXXe7nyy+/3L777ju3WkOBB3nwwQft4osvtpdfftn9fOmll1q1atXsjTfesO7du0d1HwAAkL5WrVrZRRddZN26dQuuzFDhb9Xb0sSB999/n1MIAAAQ55QpY9Pw1yx5/XorWLmyVbyta6pVtwAAxG0wQ4P7olURjRs3ttxw0kkn2aJFi9xFVBfN9evX23///Wd169Z19+/Zs8dmzJjhAg6esmXL2gUXXOAKjyoIEa19AADAoZ188skuTaOu1aVKlXK3Z5991rZt2+aus1WqVMn0aUxOTrZChQpF/TEHDx50K0nCqd2xa9euVNtV0DzS/gAAAPnNjomTbM+PP5oVKmSVevW0AqVKxfqQAAAJLksFwHMrkCHvvPOOW/VRq1YtO+ecc1y6qUcffdRuvPFGd//KlSvdQET16tVDHnfkkUfasmXLorpPJPv27bPt27eH3AAASHQPPPCAzZ8/3+6991674oor3Nfff//dpYzcv39/hp/n119/dWksldKydOnSbrXmoR5/qMesWbPGrfjUNV7BiRo1arhVI34qXq7HKvBStWrV4G3atGlZOBsAAAB5y74lS2zr6Pfc9+Wv72hFax4b60MCACC6BcBzglJRaODg/vvvt5o1a9pXX31lzz33nFsxoVUb3uCEBiP8SpQoEbwvWvtE0q9fP3v88cej8rcCAJAfDB8+3Nq2beuu07r5ffHFF/bvv/9maMWjJgioUHibNm1s6tSptnz5cvdz0aJF7cUXX8zyY0aPHu1SYyrl1RFHHOFWYF555ZVuH6XG8tPqUO0LAACQKA7u2GEbBw3RElYr0bixlbrowlgfEgAAWV+ZkVuU3kFpKp544ongzE4NkChthepbSLly5dzXzZs3hzx206ZNVr58+ajuE0mfPn1c2gzvphUeAAAkMhX/btmypbsu+n3++ed21VVX2bHHHpvhull6joEDB7pVErr+33ffffbqq6+69JBZfYzaFD179nQrMwoUKOBqfFx22WX26aefppmuCgAAIBEEUlJs07BX7OCmTVaoalWrcGsX6mQAAOJGloIZixcvttygwYi9e/emGvRQOoi1a9e67zUQUalSJfv5559D9tHPDRo0iOo+kWgWZ5kyZUJuAAAksnvuucelh1StLS+AMGHCBGvXrp0NGzbMLrnkkgw9j2p0nXrqqVayZMngtnPPPdd2795tCxYsiNpjZPXq1VaxYsVU2+vUqeNWaR533HHu2FVLAwAAIL/aPuFz2/vzL5ZUuLBVuqunFShRItaHBABA9oIZ4SkjcopSPxx11FE2cuRIV89CVAD8yy+/tCZNmgT3U/5tFRTduHGj+3ny5MkuCHHDDTdEfR8AAJA+rXZQKicF/BXA+OSTT+zqq6+2V155xW666aYMnz5d8ytXrhyyzftZ90XrMTrWWbNm2W233RbcpsLhWhmqullKXaXvtVr0pZdeSvN4qaMFAADysr2//2Hb3h/rvi9/041W5OijY31IAABkP5ihVQzKQZ0b3nvvPZs9e7ZbnXH22We7mZEnnHCCG1Tw6Htt1+2MM85webqfeeYZa9q0adT3AQAAh6ZAxrhx42zdunUuoKE0kTfeeGOmT11KSkrEn5OSkqLymG+++ca6dOni6nE1b948uF11Mh5++GE77LDDXCHxDh06uHoagwcPTreOVtmyZYO36tWrZ+IvBQAAiJ2DW7fZJk3aCASs5NnNrOS55/DvAADkjwLgqhOh2YuaYZnTRTHPOussW7Jkif3555+ufsXRRx+d6ncqlYQKg//+++9u0EQpIapUqZIj+wAAgNTefvttF7wIp0CAVkaoHoVXk0KrM1SjIiMrNH/77beQbd7qisMPPzzbj/n222+tdevW1rdvX1dH41A0mWLIkCEuOKLVJ5HaR1q94dGKDgIaAADEH628VJ0t1cxs2LChm3gR6drup1STEydOtJkzZ7q6mh07dkyzPZIX62RsfPllO7hlqxU+spqVv/km6mQAAPJPMOOBBx5w9SxUu6J48eJWpEiRkPu3bt1q0VS4cGFXwPNQFHzQLTf2AQAA/6dcuXJu5WY4bQuvPVWqVKkMnTqtjFSqSbUr9PwyZcoUV5+qXr16wYGFXbt2ufZIwYIFM/QYmTFjhqvdoTbNQw89lKHjmT9/vlWrVi3NwQ6tRtENAADEr3nz5tk555xjLVq0cCm0NaFh7NixLi1mWlTL84orrrBFixa5laaqxdWyZUuXAlttg7xu28ef2L7fFllS0aJW6a5eVqBYsVgfEgAA0QtmjBgxIisPAwAA+ZRSM+oWTe3bt3cpIDt37mwDBgywf//915599lm3+sGbSKGVmyeeeKKbKalBhYw8RukrFci488477a677rKdO3e67QpSqNi3PPbYY64guAY6FCj56KOPXF2tQYMGRfVvBAAAuatXr1524YUXumu7qK6XJk8qMNGqVauIj3nyySftp59+soULFwazN/Ts2TNfrF7Ys2Chbf/kf6tnK3TpbIXzQXAGAJB/ZSmYcdVVV0X/SAAAAHwURJg6daobdGjUqJFbXaHv/SspFIBQmkgV7M7oY7RyQys6Xn75ZXfz1K5d283W9AYoVEdDARqtRj3++OPdoEe0AzYAACD3KHX1rFmzgoEM0eqMU0891caPHx8xmKE2g2p/qXaWPw21twI0L0vevNk2vTzU1ckodf55VrIZ9UIBAPkwmOH5559/3IxHzWoAAACQPXv2WLNmzVwu6uOOOy54UubMmWP333+/K7p9qLzUnmOPPdY+++yzNO9XkMFbWZHRx2hAQrf0KBe2VnToBgAA8oe///7bBSeUMju87aBanZEsXbrU1dZo0qSJqxGmVFOqidWhQwdXFywt+/btczd/La14Ejh40DYNfslStm+3wsccbeVv6BTrQwIA4JAyNpIQZsOGDXbuuedarVq17KKLLgpuv/jii23atGlZeUoAAJBPKIhx8sknhwQyRCslVC9DNSwAAABiMeFCSpcuHbJdKzlVByMSLwhx33332aRJk1wAQ+kt1c757bff0vxd/fr1s7JlywZvCoDEk21jP7B9f/5pScWLW6VePS0prBYqAAD5Jphxzz33uDzS69atC9mui/tTTz0VrWMDAAB5kFZtVq1aNeJ9Ss+gWZEAAAC5TZMqZOvWrSHbt2zZ4gIakXiBj/r169v777/vxj0UzFDNrscffzzN39WnTx+XqtK7rVy50uLFnnnzbftnE9z3FbveaoXTaLcBAJAvghmTJ0+2l156yQ477LCQ7coz+f3330fr2AAAQB50wgkn2Oeff24HDhxINbNRqzLCV2wAAADkBtXHKliwoP3xxx8h2/VznTp1Ij7mmGOOcUGQU045JWS7VqEuW7Yszd9VtGhRFyDx3+JB8oYNtmnYK+77Ui1bWInGjWJ9SAAA5GwwY8eOHVaiRAn3fVJSUshshsKFC2flKQEAQD5x5ZVXujoWqpvx1ltvuZQMr776qp155pluZcYFF1wQ60MEAAAJSKssVOT79ddft+TkZLdNtbwWL15s7du3D+6n+wcPHuy+V/Dj6quvdhMyVG9D9u/fb99++22qAEe8CyQn28ZBQyxl1y4rUrOmle94XawPCQCAnA9mNG7c2OXD9gczDh48aI899pgbuAAAAImrePHiroZWtWrV7Pbbb3c1tZSiUis4lZYho8W/AQAAom3IkCG2YsUKO/300+2aa66xtm3b2r333usKfHu++OIL+/TTT4M/P/fccy7N9mmnnWY333yz1atXzwoVKmRPP/10nvoHbR39nu3/5x8rULKkVerZw5IKFYr1IQEAkClZunL179/fzj//fJs6daqbmdC9e3f3/erVq23mzJlZeUoAAJCPqMjlxx9/7CY7aJWGZkISxAAAALGmtFGLFi1y6bM3b95sDzzwgEsZ5Xfrrbfavn37gj9XqlTJ5s2b51ZnrF271jp27GjNmzd3qzbyit1z5tqOiZPc9xVv72aFDqsc60MCACB3ghkNGza0n376yQYNGuRmM/zwww929tlnW+/evcmDDQAAgtTJL1u2LGcEAADEjZIlS9oVV1yR5v1KRRVOKbUjbc8LDvz3n20a/qr7vnTr1lb8tFNjfUgAAGRJltcUqnjn0KFDs/pwAACQj7z22mv2wQcfWNeuXV0NLX2fFu3Trl27XD0+AACARBTYv982DhpsgT17rGjt461ce9pgAIAEDGYobcSXX35pf/zxh/u5Tp06Lid2XlpmCQAAouPII490KzerVq3qZjvq+7RoHwAAgMz67bffXBrLrVu32sCBAy0lJcU+++wzu+yyy4L1PBFqy8hRdmDZcitQurRV7NGdOhkAgMQLZvz555/Wpk0bW758udWoUcNtW7p0qfteDQmt2gAAAIlDaRe81AvKQ92lSxerVatWqv10X9GiRWNwhAAAIC8bN26cXXvttXbGGWfY4sWLXTBD9bgU3FB9i/bt28f6EOPOrpmzbOfUr82SkqzinXdYoYoVY31IAABkS4GsPOiWW26xunXr2qpVq9zKDN30vVZnaPACAAAkrrfeessNOKR13/jx43P9mAAAQN52991329ixY93N76abbrJhw4bF7Lji1YHVq23z6yPc92Uub2vFG9SP9SEBABCblRlz5861ZcuWWaVKlYLb9L1qaHgrNQAAAMJt2LDBTjjhBE4MAADIsJ07d7o2ROvWrW39+vUh9x199NEuUwT+T8q+fbZx4GAL7NtnRU+qY2WvupLTAwBI3GCGGgu7du2K2MDQfQAAIPGMGDHCPvroI5eOUqmkpk6dGnL/tm3bbN68efbAAw/E7BgBAEDeU7hwYTtw4EDEcYiff/7ZqlSpEpPjildb3nzLDqxaZQXKlbNK3e+0pAJZSsoBAEDcydIVrXv37taxY0ebP3++K7ilm76//vrr3X0AACDxqLC30lBqtab3vXerV6+eXXrppTZ79mw78cQTY32oAAAgD9EkiZYtW1rv3r1t//79we0zZsywu+66y9q1axfT44snO6dPt13fznB1MhTIKFiuXKwPCQCA2K7MeOihh9zsytNOO80KFfrfUyQnJ7uvqp/Rt2/f4L5bt26N1rECAIA4pmCFbtOmTbOSJUu6Ap0AAADRMHz4cGvVqpXVqlXLjT9UqFDBtmzZYpdffrkLaMBs//IVtuWNt9ypKHt1Oyt2Uh1OCwAgXymU1TQSAAAAkZx77rmcGAAAEFVa9fnTTz/Z119/bQsWLLACBQpY48aNrUmTJpxp1cnYs8c2DhpsgQMHrFiDBlbmsjacFwBAvpOlYMZVV10V/SMBAAAAAAAIs2nTJmvfvr2rx3XhhRe6G/5PIBCwza+PsOS1a61ghQpW8Y5u1MkAAORLVIECAAAAAABxq3jx4jZnzhw3aI/Udk6Zartnf29WsKBV6tnDCpYpw2kCAORLBDMAAAAAAEDcKlGihJ111ln2/vvvx/pQ4s7+f/+1LSNHue/LXXuNFa19fKwPCQCA+EozBQAAkJYnnnjCKleubN26deMkAQCAbNu+fbsVLVrUrr32WlcI/LjjjrNChf5vOKNs2bL23HPPJcSZDqSk2L4/FtvBrVstqVgx2/z222bJyVa84WlWulWrWB8eAAA5imAGAACIqsKFC9uqVas4qwAAICpSUlJc++LKK690P2/dujXk/oMHDybEmd49d65teXukHdy8OWR7gTJlrOJtXS0pKSlmxwYAQNwGM5YvX27vvvuu9e3b1/389NNPW79+/axmzZr28ccfW61ataJ9nAAAII+44oor7LLLLrM77rjDjjjiiFgfDgAAyOPKlStnH330kSUyBTI2vjgo4n0p27fb3t9/txJnnJHrxwUAQNzXzLj77rutXr16wcDGU089ZYMHD7aTTz7Z7rnnnmgfIwAAyEO+//5727lzp5vccPbZZ1vbtm1DbuPHj8/wcy1atMiaN29uRYoUsUqVKrl2RnJycrYes2HDBuvTp487vmLFitmJJ57oUlZE43cDAADkRGoprchIz5Z3Rrn9AADIz7K0MmPatGn2tvIymtmkSZPskksusc6dO7tZmCeccEK0jxEAAOSx2ZMKWqSlVKlSGXoeBUQuuugiu/DCC+2TTz6xpUuX2qWXXupyZKeVFzsjj3n99dfdMaoNc+SRR9qXX35pHTp0cPt06dIly78bAADkrB9++MGGDBlif/31l5tsUL9+fbvvvvvs2GOPzden3tXICEstFe7gpk1uv2In1cm14wIAIE8EM5SHUTkqS5cubZMnT7YLLrjAbS9QoIAFAoFoHyMAAMhDvBUY2TVmzBjbtGmTvfTSS67NUbFiRbv//vvtkUcesccff9ytqsjKYx588MFUabF0vB9++GEwmJGV3w0AAHLOm2++abfccoubbNCyZUvbv3+/ffvtt3bSSSfZN998Y02aNMm3p1/FvqO5HwAACZVmSsGL66+/3g0GKJjRpk0bt33GjBkuHQMAAIBXsDOrRTlnz55tp556qgsmeM477zy3amLBggVRe4ysW7fOrdbI7vMAAIDo06RJ1ewcNWqUTZw40aW67t+/v82ZM8e6d+9ujz32WL4+7QV9bZRo7AcAQEIFM4YOHWrHHXec/fTTTzZy5EiXokFUkOvRRx+N9jECAIA8RoMLZ511lpUsWdIGDhzotk2fPt369euX4edQgKFy5coh2w477LDgfdF6jFZkaELGrbfemq3n2bdvn23fvj3kBgAAsm/Lli22bds2lxYynFJeL168OF+f5qInnmAFK1RId5+CFSu6/QAAyM+ylGZKRTCVbzrcu+++azll3rx5rlZHiRIl7Morr7QqVaqkatwomKIBBhUn12oRpcPKiX0AAEDa1qxZY61atbI77rjDDj/88OD2pk2bWteuXa1jx45WvXr1LK/0kMxcm9N7zMyZM+3GG2+0J554ws4///xs/W4FapSCCgAARJdWSmp1xpIlS+z4448Pue+XX35JNQEhv0kqUMDK39jJNr44KM19yt9wvdsPAID8LFtXur1799p///2X6hZNarB069bNDTCo+KZuKsapgl+e5cuXu8CDVokov7WWmaoYuTfoEM19AABA+saNG2eXXHKJCxDUqFEjuF3Fs08//XSbMmVKhk6hAiHr168P2bZhwwb3tWrVqtl+zPfff++CLnfddZc99NBD2f7dffr0cbNGvdvKlSsz9HcCAID0FS5c2KW6Vq2MESNGuMmOuo4r1ZRWVt588835/hSWOOMMq3R3r1QrNLQiQ9t1PwAA+V2WVmb89ttvdtNNN7kGRKSC39EsAv7qq6/a22+/7X5XnTp13DbV6lAgxXPfffe5GZ5auaGBkjvvvNNOOOEE++CDD+yaa66J6j4AACB9q1evdukoI61i0PV1z549GTqFSlOlVZ9K11SmTBm37euvv3azMzX5wJOcnGwFCxZ0vyujj9EASIsWLdzEBeXdzurv9itatKi7AQCA6BsyZIirjdGrVy/btWtXMAXkM888Y7fffntCnHIFLIo3bGj7/ljsin2rRoZSS7EiAwCQKLK0MuOWW26xo446yr799ltbuHBhqlu0GyzXXnttMJAh5cuXD6at0ADGhAkT3CwNDZBIzZo17eyzz7ZPPvkkqvsAAIBDU/qHWbNmpQpmaNWjinbWr18/Q6exffv2Lq2kUlMpdZVSQj377LPWo0ePYNBAObI1W3Py5MkZfsyPP/7oZnZq4ENpodQG0M1fqDwjzwMAAHJPsWLF3LV4x44dtmLFCpcVQumhldYykShwUeykOlbyrDPdVwIZAIBEkqWVGQsWLLAvv/zSBRVykhopf/zxh91///02adIktzrjiCOOcHUsKlas6PZRI0YzPGvVqhXyWM0I1azLaO6TVrFP3TwU+wQAJLp27dq51Q6qRaFaVJs3b7bBgwfboEGD3HVVtTMyQsXDlZJKqydq167tVkhoQoW/LoWCJVqVUeD/54jOyGOGDx/uZnQ+//zz7ubRakytPs3o8wAAgNyna39Wa28BAIAEDGZoVcbOnTtzPJihfNPy0ksvuUEFL+XDPffcY1OnTrVTTz01uLzUSwHhKVu2bPC+aO0TCcU+AQAIVaJECZs+fbrdfffdLiCgyQK6jl911VU2cODATBXvViDhq6++Svd+rarIzGPeeOMNd8vu7wYAALlHYwGLFi1yfXB/iusrrrjCHnnkETvllFP4dwAAkM9lKZihQpmaqfjaa6+5HJU5pVSpUsFBEaW08qgA+AMPPOAGGLx9vMCHZ+vWrcH7orVPWsU+NVjjX5nBLBEAQKKrVq2ajR071lJSUtxKS9Wa8FZPAAAAZJZqXM6fPz9kmyZI3HbbbfbQQw/ZF198wUkFACCfK5DVRsT48eNdLmmtXChXrlzILVr0XFWrVrXGjRuHbG/SpIktWbIkuEpEwY6//vorZB/9rHQR0dwnEuXN1moO/w0AAPyvHtXff/9tv/zyiy1dutQFNgAAADJLkww18VDjA+Fq1Khhv//+OycVAIAEkKWVGSNGjLDcogKcKrqp5aNeWorvvvvOTjrpJPe98mRfdtllNnLkSDcjQ0VA//zzT7fP+++/H9V9AABAxnzyyScuLeSyZcuC23TtVurIc889l9MIAAAyTJMoVQBc/fNmzZqF3Ddx4kQ3OREAAOR/WQpmKOd1bnn00UftnHPOcasxzjzzTJszZ46b3Tlt2rTgPs8995xr0KimRsOGDW3cuHHWtm1bu/LKK6O+DwAASJ9SQFx99dUuJWSnTp3cSs5Vq1bZK6+8Ypdccon98ccfdvTRR3MaAQBAhmhi45133mnt2rVzKaUaNWpk+/btc7W5+vfvb6NHj+ZMAgCQALIUzPD8888/9u+//7oaFjlFRcbnzp1rn332mS1fvtx69uzpBkJUSNSjGhULFy60Tz/91NatW2dvvvmmtWjRIqTAaLT2AQAA6dM1u2PHjvbUU0+FzKh8+eWXXZpIzaDUKkgAAICMUpFvpbBU3cqdO3e6bUo7pfaFioADAID8L0vBjA0bNrgZl9OnT3c/KwWUXHzxxa6eRrTTR6guhWZgpEeFRTX7Mzf2AQAAh655Fcnhhx/uJioAAABkhlJDP/300/bYY4+5iY6FChVyKz2ZfAgAQOLIUgFw5cCuWLGiW73gp0CGfxYmAABIPFrVqHpTixYtCtn+/fffuzSRZ599dsyODQAA5E1ajaG006pvWatWLZclonPnzq4elzfBEgAA5G9ZWpkxefJk++WXX+ywww4L2X7qqae6gQoAAJC4NNBw4MABq1+/vp1yyilWuXJlW7NmjS1YsMCOPfZY69atW3Dfm266yS677LKYHi8AAIh/vXv3drUtVS9j7dq11qZNG1df85lnnrGUlBSXkhoAAORvWQpm7Nixw0qUKOG+9y/p3LJli5slAQAAEjvN1OWXXx6yrWbNmtasWbNU+5YqVSoXjwwAAORFClaMHTvWBg4c6H4eN26cS289YcIEmzp1qvXt25dgBgAACSBLwYzGjRvbmDFjXPFOL5hx8OBBl7sy0kAFAABIHG3btnU3AACAaNi1a5cr/l28eHH3s9JWXnLJJcEJExs3buREAwCQALIUzOjfv7+df/75bgaEclN2797dfb969WqbOXNm9I8SAAAAAAAkpNKlS1v58uXt9ddfd2ksv/jiC3vuuefcfb///rvVqVMn1ocIAADitQC48lT+9NNPVqVKFTv99NPthx9+cMU8582b5xoWAAAAAAAA0TJo0CDr1auXyxShLBE1atRw2wcPHkyKKQAAEkSWVmZ8+eWX1qpVKxs6dGiq+5599ll74IEHonFsAAAAAAAAdsUVV1jr1q1dyinV5/JqaSiYceKJJ3KGAABIAFlamXHNNdfYrFmzUm3v16+fPfPMM9E4LgAAAAAAgKDChQsHAxlSoEABAhkAACSQLAUzXnrpJTcjYsGCBcFtCmJoVcbkyZOjeXwAACCP2rNnj61fv9727dsX60MBAAAAAACJGMy44YYbrG/fvtaiRQv7559/7Omnn3bFtxTIaNKkSfSPEgAA5Blvv/221atXz0qUKOHqa+mramyNHz8+1ocGAABgP//8s6uzcf3117s0VXv37s3wWfnwww/tqquuso8//pgzCQBAXqiZIffcc49t3LjRDU4cPHjQvvrqK2vUqFF0jw4AAOQpWqWpCQ/Ka921a1eXCkLtBbUTLr/8cnvnnXfcwAEAAEAsTJs2zU3MvPXWW+3MM8+0l19+2T755BO3XWmr0vPnn3/a3Xffbdu3b7eTTz45144ZAABkMpgxaNCgVNs027JkyZLWrFkz+/77791NevXqldGnBQAA+cSWLVvsiSeecDMV27ZtG3Kf2gbDhg2ze++91zp06GCFCmV5PgUAAECW9e7d27VFFMSQVq1a2bHHHusCGlpxkRalzVT90BdffNHuuusu/gMAAMRAhkcSRowYEXF72bJlXe0Mf/0MghkAACSeqVOnulWa4YEMz+23325DhgyxuXPnupmQAAAAuWndunU2f/58e+yxx4Lbjj76aDvjjDPsyy+/TDeYoewUDRo0sHbt2hHMAAAg3oMZv/32W84eCQAAyNP++uuvQwYpdP/ixYsJZgAAgFz377//uq9HHXVUyHb97N0Xiep+TZw40X755ZcM/y6t5NDNo9RUAAAgBgXAAQAAwqmTrhoZ6SlfvjydeQAAEBNecKFEiRIh20uVKpVmEfCVK1e6+hrvvvuulS5dOsO/q1+/fi6ThXerXr16No8eAABkOWH15s2b7bXXXrM//vjDAoGA1alTx13gK1SowFkFACABHTx40JKSktLdR/cnJyfn2jEBAAB4FFTw6nz5bdq0Kc0JGW+99ZYb83jhhRdCxkPef/99W7Jkib3zzjsRH9enTx9XLNw/6YOABgAAMViZ8eOPP1rNmjVt6NChtm3bNtuxY4f7vlatWu4+AACQmFQAvGrVqmneXnnllUw9399//22XXHKJW9FxzDHH2KOPPmopKSnZfsz3339vnTp1sjJlytgVV1yR6jmUaqJYsWKpblOmTMnU8QMAgPhRu3ZtK1KkSEjNT9HP9evXj/gY1dEYNmyYK/7t3YoXL2716tWzK6+8Ms3fVbRoUdfO8N8AAEAMVmZodkHHjh1t4MCBVqjQ/55CsyzvuusuVxRrxowZ2TwsAACQ1yiAUKlSpUPud/bZZ2fo+Xbv3m0XXHCBK8qpHNVLly4NDho8/vjjWX7M8uXLXZvltttuc7Mk9+/fn+p5FPxQKgrV91BhUI8GQAAAQN6k9FKaxKDJFdddd50LSnzyySe2bNkyu/baa4P79e/f33bt2uXaDspCoZtfr1697KSTTrI2bdrE4K8AACBxZSmYodUX48aNCwYy3BMVKuRmPh555JHRPD4AAJBHnHvuue4WLWPHjrW1a9fa66+/7tJCKKiglA1PPvmkPfjgg27GY1Yeo20//PCD23/SpEm2c+fONI9B+2tFBgAAyB8GDx5sF154oVulceyxx9rcuXNtwIABdvLJJwf3mT17tm3dujWmxwkAAKKUZkrFsTRQEE7bdB8AAIBHMxs1ULBu3bpMnZSZM2faKaecEsxvLeeff75bTbFw4cKoPSY9Z511llWsWNGaNGliH3zwAf9UAADyuMMOO8zmz5/val707t3b/vrrL5dhwu/+++9PcxWoqH6o0k8BAIA8EMxQugbliZw6daobHNBNOaS1Lb2ckQAAIH977rnnXHonj75Xna1GjRrZ4Ycf7lJUZpQmSWjAwc/7+b///ovaYyIpUKCA9ejRw7VvfvvtN5d6QukoVAQ0LUpL5bWLvBsAAIg/BQsWtDPPPNMuvfTSiNklNImhefPmaT6+VatWdsIJJ+TwUQIAgKgEM1544QU79dRT7aKLLnIzH3Vr0aKFnXbaafbiiy9m5SkBAEAep3zTb7/9tiuI6c8pXb16dfvuu+9cWoe+fftmOqgQ6edAIBDVx4RT2gkdr3JkKwjTvXt369Kliz377LNpPqZfv37BdpFu+rsBAAAAAEAMgxkq9j1y5EhbtWqVW53x9ddfu++17cCBA1E6NAAAkJeo/oSKb2u2oyjXtNI+Pf/889a0aVMXEFDxb23LiCpVqtiGDRtCtq1fvz54X7Qek1FKX/X333+74uCRqDbHtm3bgreVK1dm6/cBAAAAAIBsFgAvX768m914xBFHuFuk+wAAQGJZs2aNVa5cOfiz6mQosKFUDZ5atWpluHaGHqd81irQ7dXkmjZtmpUsWTJk9Ud2H5NRixYtcimrwld++IuFRypKDgAAAAAAYrQyIy179+61YsWKRfMpAQBAHqGc0z/88EPw58mTJ9sZZ5xhRYoUCW5bvXq1VatWLUPPp1pc5cqVs549e7qVDqq/0b9/f7vtttusePHibh8V7VTbQ7UtMvqYjBgwYICNGTPGNm/ebLt377ZRo0bZ8OHD3eoSAAAAAAAQ5ysz/Hmiw3NGK+XCTz/9ZHXr1o3e0QEAgDzj8ssvt7vuusvat2/vAhvDhg0LKfi9f/9+F1x49dVXM/R8ZcqUcQERBSIqVqzoVlfccMMNrjaFv/2hwtsHDx7M8GNEx7dx48ZgekwFRLSSQ9ukQ4cO9thjj7mgyI4dO9yKkqFDh1rnzp2jcq4AAAAAAEAOBjM++uijiN9L4cKF7ZhjjrE333wzk4cAAADyA6WYmjhxoqsdoQkOt99+u91yyy3B+xVkuOyyy1yqpoyqX7++zZ492wUtIqV3ql27tu3Zsydk9cehHiP//PNPqrSYSUlJIcGOESNGuO+1n/8+AAAAAAAQ58EMDUxIy5YtXZFPAAAAPxX4njVrVsST0rp1a3fLirSCEgoypJXiMq3HSGZqWxDIAAAAAAAgjxYAJ5ABAAAiFQBfv359qu0qAn700Ue7FFAAAAAAAAC5FswAAAAI9+KLL9oLL7yQ5ok5/fTTXeompYECAAAAAADIt8GMDRs22Jw5c+zYY4+1OnXqpLp/4cKFtm7dOnffEUccEfE5orUPAAAI1atXL7vmmmtSnRYV5169erW9/fbbrkj4okWL0kwNBQAAAAAAkKeDGSri2b59e/vuu+/sjjvusEGDBgXv27lzpyso+uuvv9pxxx1nv/zyi/Xt29ceeuihqO8DAAAiU9Fs3dLStm1bO+200+zbb7+1Fi1acBoBAAAAAED+C2Y89dRTVr58eTvppJNS3ffwww/b0qVL7c8//7SKFSvalClT7KKLLnJFSHWL5j4AACBrVJC7UaNG7loLAAAAAACQGQUsC7RyoXfv3qm2a5vui7aZM2faG2+8Ya+99lqq+wKBgI0aNcq6dOniAhBy4YUX2qmnnmojR46M6j4AACB7Nm7c6CYnAAAAAAAA5Hgwo0ePHi4dU7g2bdq4fNnRtHnzZuvYsaMLZnhBBj/l4N60aZM1aNAgZPvJJ58cDKxEa59I9u3bZ9u3bw+5AQCA1KZOnWoTJ060hg0bcnoAAAAAAEDOp5maO3eunXLKKam2a5vui6bOnTvblVdeaRdccEHE+7du3eq+VqhQIWS7Ah/efdHaJ5J+/frZ448/noW/DACA/GXo0KE2ZsyYiAXA16xZYytWrLBHHnnEatasGZPjAwAAAAAACRbMqFKliv3444927rnnhmxXIKNSpUrROjb76KOPbNq0aW5lxueff+62aeWDcm3r50suucSKFCnitu/evTvksfrZuy9a+0TSp08fu/vuu4M/6/iqV6+erb8bAIC8SG2AWrVqpdpesGBBa9mypUvfeOaZZ8bk2AAAAAAAQAIGMzp16mQ33XSTDRkyxBXGVr2JGTNmWPfu3e2GG26I2sEVL17cmjZtam+99VZwm1JBLVy40IYPH26tWrWyo446yg2SrFq1KuSxK1eutBo1arjvo7VPJEWLFnU3AAASXfv27d0NAAAAAAAgLmpmPPzww3b++efb5Zdf7op4KjXTFVdc4WZcKn1EtGjlhVZg+G8KLKg2h74vUKCAFStWzJo3b26ffPJJ8HFKC/X111+7WaASrX0AAAAAAAAAAEAeWZlRuHBhV5D7iSeesF9++cWSkpJc4exq1apZLKhuhVaIdOvWzZo0aeJWbRxzzDGu3ka09wEAAAAAAAAAAHlgZYZHwQutnlC6p9wKZCjYcNJJJ4VsO+OMM1y9DgVVxo8fbxdddJHNnDnTpamK9j4AAAAAAAAAACAPrMyQL774wmbNmmWbN29OdZ9WNOQU1emIpH79+jZs2LB0HxutfQAAAAAAAAAAQJwHMx566CHr37+/NWvWzNXMAAAAAAAAAAAAiKtgxogRI2zy5Ml27rnnRv+IAAAAAAAAAAAAslszIzk52Ro1apSVhwIAAAAAAAAAAOR8MEPppaZMmZKVhwIAAAAAAAAAAOR8mqmaNWvatddeazfeeKPVqlXLkpKSQu7v1atXVp4WAAAAAAAAAAAgOsGMSZMmWY0aNezbb791t3AEMwAAAAAAAAAAQEyDGb/99lvUDgAAAAAAAAAAACDqNTNuv/12++mnn7LyUAAAgAxbtmyZXXnllXb44YfbCSecYP369bNAIJDtx/z8889266232mGHHWbXXHNN1H43AAAAAACIo5UZixYtstNPP93q1atnN998s3Xs2NEqVaoU/aMDAAAJa+/evXbBBRdY3bp1bfr06bZ06VIXeDh48KA99NBDWX7MihUrXPula9eutnbtWtu5c2dUfjcAAAAAAIizlRmqk7FkyRJr3bq1vfDCC1atWjVr166dTZw40VJSUqJ/lAAAIOF88MEHLvDw1ltvWe3ata1ly5bWp08f1/bYv39/lh9TvXp1tzLjtttus5IlS0btdwMAAAAAgDgLZkitWrXs6aeftuXLl9u4cePctssuu8yOOuooN2NxzZo10TxOAACQYL777js75ZRTrHz58sFtF154oW3dujXN+l0ZeUxSUlKO/G4AAAAAABCHwQzPnj17bN26de6m1AvKKf3ZZ59ZjRo17O23347OUQIAgISzevVqq1KlSsg21biQtCZNZOUx0Xqeffv22fbt20NuAAAAAAAgxsGM2bNnW5cuXVxRzL59+9rZZ59tf//9t02dOtUWLFhg77zzjt17771ROkwAAJCIChQIbaoUKvS/cl/pFeLOymOi8TwqEF62bNngTemsAAAAAABADIMZWn3RvHlz27Bhg40ePdrllH7qqafcagxP+/btbcuWLVE6TAAAkGi0EmLjxo0h29avXx+8L1qPidbzqKbGtm3bgreVK1dm+PcBAAAAAIAcCGZ06tTJBTDGjx/vioAXLFgw1T7KR52cnJyVpwcAALBGjRrZ/Pnzbffu3cGz8e2331qJEiWsXr16UXtMtJ6naNGiVqZMmZAbAAAAAACIYTBDMw/9KRaUV/r555+3Tz/9NEqHBQAAEl2HDh2sVKlSds8997igwuLFi61///4uzaWCCrJkyRIrV66cff311xl+TLR+NwAAAAAAyD3/S/6cSa+99pr9+uuvNmzYMNu/f79LOaXi30rH8PTTT1uPHj2if6QAACChKEgxceJEu+WWW1wNCtWsuP76611QwaP2h1I6HThwIMOPkZo1a9qmTZtcoEITNPQ4BS9WrVqVqecBAAAAAABxHMwYNGiQTZgwwX0/ffp0l2bqzz//tFmzZtmtt95KMAMAAETFaaed5tI97du3zwoXLpyqKPfxxx/vanQpEJHRx8jPP/9sKSkpqVJkZuZ3AwAAAACAOA9mLFu2zI488kj3/bRp0+yyyy5zAY0zzjjD1dIAAACIJtWjiEQBBq2iyMxjJDP1LNJ7HgAAAAAAkDuyNMWwRo0a9tlnn9muXbts7NixduGFF7rt//77r7sPAAAAAAAAAAAgpiszHnroIbv22mtdOobTTz/dzjvvPLf9jTfesM6dO0ft4AAAAAAAAKJpypQpNmLECNu8ebM1bNjQ7rvvPitfvnya+69fv96GDx9uc+fOtRIlSrgxEI19KA0lAACI82CGAhlNmzZ1RTJ14VeKKTn33HODqzQAIFG17jPWEtmEfu1jfQgAAABAROPHj7errrrKHn30UTvppJOsf//+NmnSJBeoiBScUMCjcePG1qlTJ7vtttvcz4888oh7zLhx4zjLAADEezBDjjrqKHfzu/TSS6NxTAAAAAAAAFH3wAMP2C233OIyTshZZ51l1apVszFjxriARbjSpUvbokWLrHjx4sFtRxxxhJvIuWTJEjvuuOP4LwEAEM81MwAAAAAAAPISZZdYvHixtWnTJrjtsMMOsyZNmrjUU5FotYY/kCFly5Z1X/fu3ZvDRwwAAKKyMgMAAAAAACCvWL58ufuqlRh++nnFihUZfp5+/fpZrVq1rE6dOmnus2/fPnfzbN++PUvHDAAA/g8rMwAAAAAAQL534MAB97VYsWIh27XyYv/+/Rl6jieeeMLVyxg9enSwfmhaAQ+t4PBu1atXz+bRAwAAghkAAAAAACDfq1Chgvu6adOmkO362bsvPQMGDLBnn33WPvvsMzvjjDPS3bdPnz62bdu24G3lypXZPHoAAECaKQBAXGndZ6wlqgn92sf6EAAAAPKt2rVrW4kSJWzevHnWuHFjty0QCLifb7zxxnQf+8ILL9gjjzziAhkXXHDBIX9X0aJF3Q0AAEQPKzMAAAAAAEC+p+DCtddeay+99JJt2bLFbXvjjTds3bp11qlTp+B+Dz74oHXv3j3486BBg+zhhx92gYwLL7wwJscOAAAIZgAAAAAAgAShFRZHHnmkHXXUUXbCCSfYXXfd5QIaxx9/fHCf33//3RYuXOi+X7p0qdundOnS9vjjj1vTpk2Dt++++y6GfwkAAIknz6SZUjEuFddKr8BWSkqK7d6920qVKpXj+wAAAAAAgLylTJkyNnXqVPv3339t8+bNLqAR3vdX8e7k5GT3fdWqVdMMWtSpUydXjhkAAOSRNFMff/yxNWnSxBXjUgOjefPm9ssvv4TsoxyXffv2tXLlyrn9atSoYV988UWO7AMAAAAAAPK2Y4891ho2bBhxEuOJJ55o9erVc98XL148ZDWG/5aRouEAACBBghkHDx60MWPGuPyUyme5ceNGO+aYY6xFixa2devW4H6DBw+2oUOH2ldffeVWVHTr1s2uuOIK++uvv6K+DwAAAAAAAAAAyF1xHcxQSqmPPvrIGjVqZIULF7aSJUvaM888Y+vXr7e5c+cG91Pxri5duljjxo2tUKFCdt9999kRRxxhr776atT3AQAAAAAAAAAAuSuugxmRLF++3H2tXLmy+7phwwaX61JLPP3OPvtsmzNnTlT3AQAAAAAAAAAAuS9PBTP27t1rPXr0sGbNmtkpp5wSDEL4gxse/ezdF619Itm3b59t37495AYAAAAAAAAAABIwmJGcnGzXXHONbdq0ydXR8CQlJQXra4TvX6BAgajuE0m/fv2sbNmywVv16tWz+ZcCAAAAAAAAAIA8F8xQQKFDhw72yy+/2PTp061atWrB+1TTQtatWxfyGP18+OGHR3WfSPr06WPbtm0L3lauXJnNvxYAAAAAAAAAAOSpYIYCGddee639+OOPLpBx9NFHh9yv1RD169e3qVOnBrdpdcU333zj0lFFc59IihYtamXKlAm5AQAAAAAAAACA6ClkcSwlJcWuv/56+/bbb+2LL76wUqVK2caNG919pUuXdoEE6du3r3Xs2NHOPPNMa9KkiQ0YMMAFQbp16xZ8rmjtAwAAAAAAAAAAcldcr8zYunWrTZkyxa2QaNmypZ1wwgnB24cffhjc7+qrr7YRI0bY4MGD7dxzz3WpnrSiomrVqlHfBwAA5J5Vq1bZddddZzVq1LCTTz7ZBg4cGJXHHGqfZcuWuet/+G3atGlR/fsAAAAAAEA+WJlRoUKF4EqMQ+nUqZO75cY+AAAg5+3bt88uuOACO/bYY23cuHG2dOlSd43ev3+/3X///Vl+TEb20cpM1c2aO3euVa9ePaRtAgAAAAAAcl9cr8wAAACJS6sw//nnHxs1apQ1aNDA2rZtaw888IA999xzduDAgSw/JjPPW7ly5ZCVGUWKFMmVvx0AAAAAAIQimAEAAOLSjBkz7JRTTrGKFSsGt7Vo0cK2bNliv/32W5Yfk5nnvfjii61mzZru61dffZUDfyUAAAAAAMjzaaYAAEDiWr16tVWpUiVkm/fzmjVrXEAiK4/JyD5JSUku9VTXrl2tfPny9tFHH7mAxpgxY1yNrUiUvko3z/bt27P8tyP6rh3ZNmFP63udxsX6EAAAAAAg2whmAACAuBQIBKxQodCmivfzwYMHs/yYjOyjehrvvPNO8P6HH37Y1dZ4/PHH0wxm9OvXz90PAAAAAACijzRTAAAgLqlexcaNG0O2bdiwwX097LDDsvyYjOyjlRnhGjVqZH/++aelpKRE/N19+vSxbdu2BW8rV67M8N8KAAAAAADSx8oMABG17jM2Yc/MhH7tY30IAMzsjDPOcOmd9uzZY8WLFw/WuyhWrJjVq1cvy4/JyvPK33//bRUqVLACBSLPBSlatKi7AQAAAACA6GNlBgAAiEsdOnRwAQateNi/f79L8zRgwAC76aabrGTJksEAQ9WqVW3atGkZfkxG9nn55Zdt4sSJ7n6txJgwYYINGzbMbr311hieEQAAAAAAEhcrM5BvJfLKAmF1AYC8TqsgvvjiC+vcubOVKVPG1bpo3769vfDCC8F9kpOTbd26dcHC2xl5TEb2adGihQt2aLt+R8WKFV09jLvuuiuXzwIAAAAAABCCGQAAIG41btzYFi1a5GpQaDVFeBqn4447ztauXesCFBl9TEafV6moVBBcgZISJUrk4F8JAAAAAAAOhWAGAACIe2XLlo24vWDBgi7NVGYek5l99PwEMgAAAAAAiD1qZgAAAAAAAAAAgLhGMAMAAAAAAAAAAMQ1ghkAAAAAAAAAACCuEcwAAAAAAAAAAABxjWAGAAAAAAAAAACIawQzAAAAAAAAAABAXCsU6wMAAAAAEL+uHdnWEtl7ncbF+hAAAAAAsDIDAAAAAAAAAADEO9JMAQAAAAAAAACAuEYwAwAAAAAAAAAAxDVqZgAAAABADknkmiPUGwEAAEA0sTIDAAAAAAAAAADENVZmAAAAAADiTiKvahFWtgAAAIRiZQYAAAAAAAAAAIhrBDMAAAAAAAAAAEBcI5gBAAAAAAAAAADiGsEMAAAAAAAAAAAQ1whmAAAAAAAAAACAuEYwAwAAAAAAAAAAxDWCGQAAAAAAAAAAIK4RzAAAAAAAAAAAAHGNYEaY8ePH2/nnn29169a1Dh062N9//x2b/wwAALD169db165drV69eta0aVN7/fXXo/KYaO0DAADyntGjR1vz5s3dNf6mm26yVatW5chjAABAdBHM8Pn888/tqquustatW9sbb7xhgUDAmjVrZps3b47yaQcAAIdy4MABu+CCC+yvv/6y4cOH22233Wa9evWyIUOGZOsx0doHAADkPe+8847dfPPNdv3119trr71m69ats3POOcd2794d1ccAAIDoK5QDz5lnPfbYY9axY0c3WCEjR460ww8/3F555RXr27dvrA8PAICE8vHHH9vvv/9uq1evtipVqthZZ51l//zzjz355JN2++23W6FChbL0mGjtAwAA8ma/v0ePHtalSxf385gxY9y1ftSoUW5FZrQeAwAAoo+VGf/fjh07bP78+daiRYvgySlSpIiblTl9+vQcOPUAACA9uv6efPLJbrDA06pVK9u4caMtWrQoy4+J1j4AACBvWbZsmbv5+/1ly5Z1kxbS6vdn5TEAACBnMK3w/1O+S6WVqlq1asgJ0s8LFixI8wTu27fP3Tzbtm1zX7dv3x6Vf9CBfYm7bDW75zCRz51w/jh3vPbynmhdO7zn0XUtL1u5cmXE67J33W7QoEGWHhOtfXK9TbDngCWybF/XEvj8ce44f7z28ibaBdGn67tEusavWLEiao/JjXYBAACJOFZAMOP/S0lJcV8LFy4ccoK0OuPgwYNpnsB+/frZ448/nmp79erVo/ufSkBlB94c60PI0zh/nDtee3lPtN+3WnWomYN5+dpcrFixVNdlSevanJHHRGufcLQJctZHt+Xd13Ksce44f7z28qZov3fzersgVv1+xgoAAPnNjjzcJiCY8f9VqlTJfd20aVPICVI6Ce++SPr06WN33313SENHBcMrVqxoSUlJlpcjdQrIaBZKmTJlYn04eQ7nj/PHay9vyi/vXc2yUOPkiCOOsLxM19/ly5enui5792X1MdHaJ1HaBPnpvRELnDvOH6+9vCk/vXfzS7sgVv3+eBwryE+vz7yGc8+5T1S89vPHuQ/kgzYBwYz/Tzmx9cL4/vvvrU2bNsETNGvWLGvZsmWaJ7Bo0aLu5leuXDnLL/QmoXHE+eP1l/fw3uX85dVZFn6nn366jRs3zvbu3RtcJTFz5kx33a1Xr16WHxOtfRKtTSB8tnDueO3lPbxvOX/5pV0QDccff7x7T6jff+aZZ7ptWpExZ84cu/POO6P2mNxqF/D+jh3OPec+UfHaz/vnvmwebxNQANzntttusxEjRtjixYvdz6+99pr9+++/dsstt8Tq/wMAQMLq0KGDFSxY0J544gk3m/G///6z559/3q677jorXbq020fX6Vq1atl3332X4cdEax8AAJC3KL1U586dbciQIa7ehWao9u/f381SvfHGG4P7devWzV3zM/MYAACQ81iZ4XP//fcHi3qWLFnSDWK8++67ac7ABAAAOady5cpudYQGCoYNG2a7d++2Sy+91AYPHhzcZ//+/fbPP//Yrl27MvyYaO0DAADyHtW4Wr9+vZsMoX5/iRIl7OOPP7ajjjoquM/atWtt69atmXoMAADIeQQzfBS80IDFgAEDXC7Lww8/3AoVSsxTpOWwjz76aKplseD88fqLb7x3OX/5zTnnnGNLly61NWvWWKlSpVItia1Zs6YtWbIkJOfnoR4TzX0SBZ8tnDtee3kP71vOH9J+b2jS4iuvvGLbtm1zbYgCBUKTVgwfPjykIHhGHpObeH/HDueec5+oeO1z7uNFUkBrJAEAAAAAAAAAAOIUNTMAAAAAAAAAAEBcI5gBAAAAAAAAAADiGsEMAAAAAAAAAAAQ1whmAAAAAAAAAACAuEYwAwAAAAAAAAAAxDWCGQDgk5yczPnIpkAgYAcPHuQ8xlBKSooNGDDANm7cyP8BALKINkF0cB5j75dffrHRo0fH+jAAIE/jehZ7/A/yvg0bNlj//v2z9RwEM5An7NmzxzZt2hTrw4g727dvj7hdA5hPPvmkXXvttfbJJ5/k+nHlVX379rWbbrop1oeR51166aU2cODAWB9GQitQoIANGjTIJk+eHOtDAXLEunXr6MxECCTv3Lkz4vmaMGGCde7c2bp27Wr79u3jVZkB//77rx1zzDH2zz//cL6yYeLEiVavXj3bu3cv5zGGvvnmG3v++ef5HyBH6LqiwSnkTl9f4yJPPfWU6+t/9NFHnPZcojZU7969Od8xtHv3bjvxxBNt6tSp/B/ysJUrV9qDDz5omzdvzvJzEMxAXNOLW4PL5cqVs0qVKlmzZs3cCx9ma9assaOOOsrmzJkTcjp+/vlnO/nkk2316tV24403WrVq1Thdh2h8KyrcvHlz1xjUrLXvv/+ec5ZJCqDdd999duaZZ9qCBQtcMG39+vWcx1z0+++/24wZM4I/t2zZ0g0iAfnJ+++/bzVq1LDDDz/cKleubK+//nqsDyludOvWzW6++eZUAQ61o/r06WNNmjRxAx9pBTzwPz/88IO1b9/eOnXq5IJm99xzD6cmCz7++GNr1aqVPfLII7Z48WIG0nPZ/v377d133w0GfS+++GK3OmPt2rW5fSjI5wOLd911l1WoUMEOO+wwa9CggS1atCjWh5Uv6Pqjvv7s2bNDtv/666+ur79ixQq74YYb7Mgjj4zZMSYCtZkef/xxNw6lAfShQ4e6axpy119//WVdunSxCy64wAVOe/XqRSaIPGb8+PHBNsgpp5zixne/+uqrrD9hAIhTe/fuDTRo0CBwxx13BDZt2hRYt25d4MILLwycfPLJgQMHDsT68OLCFVdcEWjUqFEgJSUluO20004LDBgwIKbHlVccPHgw0LRp08AFF1wQ+PbbbwOjR48OHH744YHTTz895JwifRs2bAgcddRRgc6dOwe+//77wMCBAwMlSpRwPyN36DOxVKlSgeLFiwe++eYbt+3DDz8MVKpUyb3Ogfzg9ddfDxx77LGBGTNmuNf1O++8EyhYsGBg4sSJsT60uDBr1qxAUlKSOz8efQ4cffTRgZ07d8b02PKKTz/9NFC6dOnA8OHDA7Nnzw5ce+21AXWXpk6dGutDy1Mefvhh1y7Q+fz6668DTZo0CZQsWTKwatWqWB9awujbt6977V599dWB5ORkt02fBW+++WasDw35hPpK6pvrNbZmzZrA1q1b3Wdm9erVA9u2bYv14eULOrcNGzYM6Zeq79+vX7+YHlei2LNnT6B+/fqBtm3bBmbOnBkYMWJEoEKFCoGWLVvG+tASyvz5813b7Iknngj88MMPgd69e7v27ksvvRTrQ0MGTZkyxbVJTjzxRDeuKzfccEOgU6dOgawimIG49fbbb7uBZs+YMWNc40iNJDWWEAj8+++/gaJFiwZGjhwZPB3qPA4dOtR9/8cffwQGDx4c6Nmzp+tMItRXX33lBt137NgR3Pbrr78GihUr5l5/yJj+/fu7hp7f2LFjAwUKFHCND+QMDU74PwtbtGgRuPjiiwPlypULfPfdd+6+QoUKBebMmcO/AHmegheHHXaY68TI8uXLA+3atQscc8wxBDN81EY65ZRTgkHMYcOGBWrVquUmiOhaN27cuMD9998fePrppwlwRHDqqacGnnnmmVQTR+rWrRscEEb6du/e7dqmamN5FEw74YQTAh07duT05SBN/vIPHOgzs3Hjxu686zPhtttuc5+bQDQoyKvrizfJUBMLateuHWjVqlVg9erVnOQoUFtHE5X8QUhN6lD/XhYvXhwYMmSI6+vrPY/oeu+999znqH8irSaMaCLN559/zunOxbZt+KB3nz59XGDJf91DfNm8eXMwEKt2oNqGbdq0CdSrVy+wcePGwPvvv+/eX1mdREyaKcQtLd+rVauWWxJ99tln24svvmhjx451aYDKli0b68OLC0q1cffdd9sDDzxgu3btctvuv/9+6969u0vNpfOmpanLli1zKWeWLFkS60OOu1RdxYsXt1KlSgW31a9f3+UVV0oOUnFk/Dwq3Yvf1Vdf7VKa9OzZM8r/NXh0brXk2asnpHQehQsXttdee81at25tf/zxh/sfkGoK+cGWLVtc6rojjjjCnnjiCWvYsKH7vNbrXNc3/M9zzz1nf/75p73xxhvuZ6VLOnjwoFvKXbFiRXf/tm3bbMiQIeR9zuD1TO1Pvc6GDx/OyyyDKWKVwtN/HkuWLGn9+vVzbXil8UL06X2vdDOjRo1yP6sPoBRAL730kktDecstt7jPyilTplBvCFHrq6uu0PLly61NmzYu3ZTqtX3xxRfuWo3sU5op1WhQbvkdO3YE+/o61+rrN23a1GbOnOlSTimVHOmPot8mKF++vBUqVCi4TX2vq666yo3BHDhwIMq/ERltmymFpWpEPvroo5y0OKR2oMYhVGdGiyjUDlS75JJLLrEzzjjDLrroIjv99NPdOMb8+fOz9kuiHHwBMk1RuYceeihw6aWXBl555ZVgZO7VV191S9KPPPLIwFtvvRUSsdOsYy+VSqLTTMsjjjjCLSf3/PPPP4FFixaFpJepUqVK4KOPPgokIs2m0OzU1q1bBx544IHgSgytGtDHoFIj+U2ePNlt1774P1u2bAk89thj7r2qGUHe60uzhbTCJXxmxHPPPefOo6LuiD69js8880yXek8zH5YsWeJSTe3fvz8watQoN1vlyiuvDJxxxhmcfuQpn332WeCqq65yqeqWLVvmtunzRquOypYt62YZ+2d9qn3w5JNPxvCI48vjjz/uZjp5aT527doV+PHHH0NWcikdoFJXJCpd/2+++Wb3OlN6Ls9FF10UuOyyy1Ltr9UuzABM7csvv3RpUG666SbX9vTeq0pxGP6eVFtMKzZ1TSKVZ87QquLChQu71eyi17JmbauvpdVF119/vbvfn4oOOBTNqFVao0suuSTw7LPPunamfPHFF4EiRYq4PqauKf7Z63/++Sft/yjRNVzjIVpVmV5fv1q1apzzbKSTev75510fV9curWYVrTBUOiO9nv2Umlp9XD0G0bN06dJAjx493Oz9Tz75JLhd25SeKLzt0KFDB5eF4LfffuPfEId++eWXQPny5V3ZANF1Qqud9bl13XXXufag2iZKH5YVBDOQ49SwURqa8Dy5+jBSfk3lb+3SpUvgwQcfdAOi3vJ+Dc6VKVMmcMstt4Q87r///nMpbX7++eeE7Hy/8MILLk+4P/+1flZqJF0AIlEaKtWCyM/L8HTBi5TTWq8/DU6cf/75rgGulCRKxeNdDJXH+ayzzgpJH/Hxxx+7jrjOqV6jiULn4MUXX0z1OtK5Ul0MLSXXEs9HHnnEvTf1nvUG1StXrhzo2rVrqpzZOo9aDk2dm+yn7VDDWY1mf9ooDVaqxosGJTVQqf+RF+hVfQE1wDV4pP8fEC/mzZsXeOONN1Jt12eNJjeow6KBE3026zNb7QG566673IBy+GeU2g033nhjINFs377dDV7qc1spEv2fF2pb3XPPPREfp1y1Sqek9kR+pWv3U089FfE1NmHChEDVqlXdNUqdKS17V7BHlIZLn5nhE2aOP/54dz277777Aolk4cKFbqJRpPOo86vzovdf8+bN3WDb+vXr3f16H6sDu2LFiuBjvHa92lYffPBBrv4d+fVzVO9htfH9fQJNBlPAQhOYVPtFKX+8/pNSAGkATuk5AL/XXnst5Drivc/VxldKEAUtNeCka/Dtt9/u7ldQQ9caBTn8g+q6NilVNPWsMk/jG7qm69ruT4OsPoCuVX///Xea6ZB0XaO9nzZNjvWu9f7XuAIZ6kepLoZScGoyiJfSSH1jpUhUkMNPn7NqE+g6R32YjNNngyZG6pyH/x808K3AqNqu3bt3d6m8vLaCAnf6Obw9oomq+j8o+IHY0livJg+//PLLgZUrVwa36z2ntp/6cEqLp+81LqT3liYUqU2i8bisIJiBHKfGjQp5+/PkalBdDWoVDdMF238hVgNctR5Es4s1GKcGlDqYml2kgQ29SRKJ3uwK+CggoVlW+tBW4/Gvv/4KXgAU2dQsbH+j595773UDFmqEqkOanykPsDrV3mwhufzyy12BLjVOvOCFZq8raKYCXrJgwQK3AkjnVY141RbR4LsCGjVq1EioGRfe60gXFo8a0ieddJKrx+CPmo8fPz6kJobOl96rd999t7tQaVagGiTTp09372mKpx763GuFi+rghFMjQLVwNFikzqEu+moQ+AeI9BmrvNiaHav3vUeF0TTIS50hxBN10sMD8Ao263NGATlvYE6zEdWJ1PVP1GFU20GDpurQKIitzyt9buXnYH0k06ZNc22CCy64wP39+vz1z4RX3SJ99nqzCdWxUABag/cVK1Z0++bn2fFqR3oDuh7NsNQKC72Gfv/995BczP6aGGpzasBO7Shdz5SLXB0tddIUuE8kOn+a9ei1y72ZdZokojaSBiZEs1jV1vTa+gqo6Zzq/avrv9pXGlS/88473YoYiqcemtr4GnwLp0EAXde1KlttV72f1TfyD3Jq9axe/+ovqc3rDRxpsEGfGf56JoBootc555wTPBnqT5122mnuuqy2vUftel1v9FVUp0GvNfXp1RfQRBpNSPBnDEDGxks0KUwBCb2vda1RrVD/Z69WY/tXDqqvpTa/BuLVVwsPRiGUxgP8g6Y65+pX6Xp06623BrfPnTs3pCaGCn9rBZICHBpU13atglEmB33+Umczc1ke9Nr2TzZRe+y4445z/xutzPZo3EHn15vQpMmUCugNGjTIXR8HDBjgxsP0GI1JeJMpkDPUB9O4mD9w7Q/uqd2soJ/G49Tu+PTTT4P3awW0skdodVnNmjUD3377bfA6o9U1aldmBcEM5Ah9KPkDDkrjo4azf/a7XuwalAtftnfeeee5BpVHS1i1TQMcuggl4tJoBXzUIfQGJLVcXLOx1XDxBu91jtW41ACHqAOuwQxtz88DFh7NRNEMtH379gW36QNXrzF9wIan4NBAuzeTQjPd1WDXvmo4agDDi/Z7y+ISxU8//eTec/736jXXXOPOjWYB+un8NGvWLGTQQ4Mb2leDGmr8ic6pOjeITOdJrz8N7voDkqLXswZuvUJ/orRdeq/7iwHq9a/PAzUUdO6BeKLrlNLOeSvddE1SZ0SdGn/6SH12KCDnp86iOineakxd/zQrVIMl6sBrFp0GThOJ2gKaDehdq0QD7Tp//o6gPp81Y9ajGW6aGOJ1DPM7rVpVwMy7ningpc6WOs1+mrFeunTpwNChQ4OfuwoYa6KDBvJ1DdRj1RnTOfbPlk0EXpoif4Bd1yB1QP3UOdX2H374wf2sgQVd03QOdS7VidVAvK5nCighMrVNe/fu7WYv+gOSHq121/Xea8Pq+q+JS1q17l8Fq89cDcDpOfQ5Cvhp1ZQmEXi8fqO//a+JYvrM00QvP6Ur0yQab1BLn40KVqqvroFhrX5D5j9nNQjoXZ91zdEkJbV1vL6t99nrFfrWgK7+Z7Nnz06Ivn52KTXXu+++G/I5qc9avcY1Sc+vW7duIZMktVpTAXrtq8FY7zWuAfhHH300l/+SvE3tVE2G9D5r9NU7t/6Bcr3u9ZmiVRoeBTAU4ND7QFk39D/Ve0WPDV91g+jQ/0STgDWBSm2K8DEdjR0pyORNnNb7S8FBBTT8kywUAC9evLgbq4jW6lCCGcgRilgrou2fIaBOpaLf3sVWF+BIbwjNlNdjNaCayDTbQh/YohnZ4W96zeBWB1Hn1aMZcf7GZX6mD8rw1GVanqjBdC8yrwaIGiKaiernpeDwz2AX/5JHpeHQYJGXdziRaLBcM32915E6PLoghUfNtcpF72E1pNM6jwoU6f1MLsvIevXq5YIValxrAMgfkBQFI9VAC1/CrMFcPc7/Xl+7dq17vSt4FL58F4glr6PiTwWlAWG9Xr3ZOdK+fXs30BzeKVfg9Oyzzw4kMl3vNCtQASB1otVxCL/Wa0WBP5ipzqICQZMmTQokgkipNtu1a+cCXv7Varom+ZfAewO//hmA3uvWv9pTs5PVxko03qoLf7v8hhtucO0o/8Cnd77Da2LoHPoHLTRBSfVwkJraTLqGK6ira7ra//6ApCj9nlKkhbfH9Lr2Bzi9yTv6DIi0wgOJLdL1Qd8rUObVC1CgTLWqwgdrVbNKA1Lhk8WQOeo/eQElDcyG90tVM0zBSH9aTrWj/CsJkTb1hTRZwU9BCU0Q1cpfUf9KaaXC03IqgK9xgPB0nP7JMxqL0UQ0L7iEjNM1zEvlJVrBqf6uvw8sCjJpvEsrYjxqX3ifUV6KPP2vvP8pokfXAE2M0qowBS20WsZfk090fQhPE6V2n95n4RPUNLld/Zfw7VlFMAM5RjMzzj333FQf+P7Bd1041BkKH3hTNFyztvydyEShgQoNGqsxqWWOouX8mgUTTgM8GrzwNy61fDz8wp0f6bWjWbm6oKkRoUCOIvj64PTXWdGgjzp4CpKFNyA1EK8OoEdBEC+dmdL66HWY32a6qPHrbwCkNWimWZT+ToouVFrN4qWU8KjhrXPlb9xpFqFmXagTrcaFBo+QmgaGFJDwp4BSw06DZV4nxStS72/EiX7Wdi/Nlz8IR2MO8UgdFQXr/DOnlN5HBez9gVPN2vGvOkovcJoI9Jmra7s+ZzUor89vdQb0/tf73U+BIW331ynQCoNESIGg4IQGfTSD2Fu9o0ELrVjRoJtXMN4LrCnVkV+kGYCi51PqFK1C0Od1fkzZmZFrhmbZ+dvlGmjXahad3/DBN72H/W19taOUEk5Be3WKGzVqlHBp4TJKs7D9bS9N0tGAs7/2gIJBkQYCtKpdn6nhIqWvBEQpHPWa82aqK5irVMb+ALD6pGXLlk2VwkX7KF0MaUyz1tdX30jn3lvJppUt4avdvJTJKpjr0Wev+vqJVNMxqzSZ0cv2oeu4VhqpTeXVzPJoYq0m7YVPctAqwvDXvvZRH1cTTtUv9upH4v9kZLW02mf+lZyiFGoK1ofTOJjS2IUH/hXoUJtNYw2sPswZOr8aa/TGwzRmG16TT3XT1EcJHzNTIFCBj5xskxDMQI5RjkdFUv35ivWBrxyvXj5sLxoeXsFeHVEtTc/Pg/IaMA4foNS50hIu5WH0/+26mKpzGN5w0WC7Um8lIm+1gAbCNJPXWx6qDp8/JYkXWFPnL5xmt2jmu0eDHMoj/sADD7gl0/mRGnXhDS8N8mgVlf8ipPekv5OihokuVOHFT9UoVCPb3wBUo08riRQA8ed6RSg1htVI1jnWahj9rPe4Bt68Amca5NUAmhrgfvrs1KClvxEIxDt1VPyzd7zAqWZVefS5oY5meOBU+XVVRyu/0udveG0hXZN0fdPkBn/HW4PPmi2rIst+Wm6vz4W0CoTmd7qmq5OldqXqXGjASOdQ6XfCZwCqnaBZZn6a/KCUE+EdbnXalMYyPxb5VLBG7c7wFSlKK+kvJKvrul5z/jp3/fr1S7WaxUuD5J9JLHrv6jwq1Zk/xQdCabWaUqQuX77ctdX0uaBAkibqeOdNAwT63AzvI2kij7/mGXAoCogrnZnSPnq8ALDX59TrToPu/poCosC6Xm/+SWE4dF9fk+Y0CK62vb9frzSHmvQZnnVAg4nhxaeRMerbakKjxgo0Ucyb9a8JMxpX0ees19dSqr7wYLBe+1ox7P8fajBX7TF9PoenX8b/ZuRrYojq4PrpcyI8ZaLGDzS5wRt/UNtVs/bDH6vzr8CUf/KzJpZoTEJBJQX4kDM0MUjjaOp3qHC7Pp/UjtPkIa92ryZdqO+hzzY/TUBTuzEnEcxAjlJnUml/vA8fBTEUzPAPpqYVDc/v1LHWygINEmt2iwaZ9b2Wl6uB46cPEJ1HFWbzBngU8FHBv/AZrIlCA79qiKjh5+9we7NbtDQ/PLCmmYGJTjMjvcK76oho8EfnUY09pXHxZmh6kXd/0T+l3ApfzYLsNfj0flenRjUzvMCEZjhoZpxXT0DpITR7xT8rXTWJ9DmRiKvXkHd5HRV9DqUXONX7Ijxwmt+pM6D3uQbPFWRXqh59JmjAWJ0EBTz91IFTZ8JfyFcdDQU+EpEG4DVgoWuZzkOkGYCayXeoGYCJRitSlP9b7XXR7Ea109Uu14Cmv4CjAhn+WapqQ+ixPXr0iNnx5zcKZGjFi9oAWt2qa7zOt867N+CsvpTaZ0pL49Vv0T763Bw5cmSM/wLkNbqWaGaz15eKFABW+ikFgHWdQvb6+upfqf+uSRt+2q5BYK1e8wLnWsF27LHHhkz4QMZpjET9Vn2e+gfCNXiu/4s/u4XqtqmdkF8nM+YmTQrV9UhjCro2aVWFxhp0fjXu4AUvvKwt/uuWamt5j0Xs/fL/AxX6vFKwzwscaczIH2RV0E/7eAEOXUfUzvZfR3ICwQxElWYJ+QfXNBgXvlxVH1j64PKWGHnR8PAl//mdZgNoIELFKDWzxcv1rAFjfWiEpzLQ7AJv1YYi2frAUAcyv6VBSo/+Vg3satBLN6Uz0GqB8Byjivzr3Cpy7FFHXY3ERKgncqhzqJnRWvmk9CNKz6UAmV6PCpj581h7kXf/TArVvfE3/pC+tBpjGoBQOjPNWFDjzr+CRQNE+l+onoZHM1r1uaCApgYwFMig0BninQbdwmdtq6Oi129GAqeJNslBM880qKSOnBfw8QaMVaPAT9cypZrRAJOW4OuzWfv9/vvvgUSijpNXiFPXqki1L9KbAcgkh4ALWGjCh1a26rU3Y8YMd5608kcBDa996uVA9s/Q9h6rVUHIXgoOr02rYIY/Xaq3GkPtBW/AWauPvdndem3re7XpEqlPgKzR4KJ/YFfva60A1Gr/9ALAqt2iaw2y39f/8MMPXZven0VA1PdXMFk3va/V79fMaN7XGafVFErVp0C9Vh7pWh+p9oWCFnqN63rnH5BVbQBkj/q4eu1qpbUCGQrM6/+hfqvaFP4anOFZW7zHamIfYj9W0bp1a7d6T9cI/6pafXb5ay5pXE7vHa0aVRCjTp06LuDhTczMKQQzEBVKj6ROtC4KesFr1rBH6VL8+Yp1QW7cuHFI/kd1vvNzSqlIlDpKgR7dwtNpaFAiUvoofVCMGDHCXZD9xdUThTebRR02j2arRlotoIExfz0WfZiGNxoTlWqx6L2qxrW/U63lt+GdF61w0UoX/8BR+BJoBNIcmFADOjwwqbytylOsgVrNvtLFPrzAp/ZRB8gf5ND/RTPolK7Da/QB8UjXc3UK9RpWh12dGa++Q6R8xZECp+FFAPM7dRo0WUEz4sPrDGnAOPzzwF8nQwP4uhYeqh5Sfp5VrNWq6dW+iDQDUDMxWd32P8rBrpmTWgkUPvnB317/8ssvU6Xy1MAnMkYByPDi53rvaza22gxKp6G0qVph5G876HWqmiX+lJNq12p1u1bM5MdaLoh+SjmtuFA7X+/1Rx55JN36guEBYL0+qb8Svb6++leR0kepX6A0ferrsxIm87SaRf1btYs8WtUWqe6LZpqfcsopwYmO6t+Gp0JC1mjyqdpcqlHmp/atxgq9cb9IWVsU9Ah/vyBnvP322yF1k/yTgDXeoH6H0uFpkkV430STLvyP1WoM9Vf03tMq39wIwhLMQLapY6hBY3Vm9KLVjErN0vKWpnvLVf0FrDUoF170J9FoEFkdmEh1CJSDUZ3F8PzZ+N9sFg3qLF68OHg61Als06ZNqsCPUpdoyT5S0yxfdWbCZwhqxYaCjd4FyCs+OX36dE5jFijPpAaJ/JQ+xp9mx5vd4C/wKarz4hWuA/IKdUA0mKxl5uqUq3Oo2Tr6nPY+V7x8xRoc8Siol+j53tUmePbZZ93AR/jKAq3I0ucJQil4ET6rWHl71Q4NX6WizrJWHlCzITUNhmswU6+/SJMf/MFFTXDgtZg1Xr0Wf651BXOVIs1PAWC95/2++OIL9z9KxMlMyJ4pU6a41Tvqm2vgVj9roqF/cKpFixYh9QUVuNDqNX/tS0Svr+/1r7yZzYgeva41+OoNmKdV90WTyjSBxN8nQ3SovX/66aenCmZoXFDpUP1BfS9rS6JNbI4HGzZscKs+/atl9H/QRGFvkpAoQKEgoYKFnvAUmLFAMAPZpiXO3kCnBi0U5dZFWzUN0luuOnPmzIRZNqm/UytUFP3XB7s/x7W3siC8YKfSbunDXh/6iUqBsUiDuUq1498+f/5818FToEONRuXLVodRxT3TWs6fKHQ+NENVM6HVoPByGXqFplVwzk9LoNWg8Bfa1VLcRHmvRptXr8UriqUVanq/hy+71EoNf4FP0Qw5va7V6QTyCjVqvdmG+txQ4EJLxtVx8ZYyR8pXrGtgIhXx00B7u3btgqsIvFWEXg2D8DoE3mCzl1IpUa9nqi+kAXa/SLOKNRisAJpmtKuNqhWdejwrNP9Xc0zvP7Xf+/btGzxnSmei1174ahVNftCkJK89qiBkoqWAiyalYNAKbE+HDh1S1Xn57bffQiaGedT21aAzkBmapOQN2GowSu91BTc0e9p/TQqvL6j6TYncD80MtWuU3kh9fV2n/EGKtPr6GlyPNCsaGadsIP4Js/4Bc62E82imuIJHWs2q94AyOCigpIm1rNDMHrUHtBr7sMMOc7XIvOwBs2fPdquzdc791K/1B/X13qFWSewMHDgwJI2lPsc01ubnrXq+4447QrZrBZlWR3t1fnIbwQxkmgY7/UvD9CGli7PqYmgmvPLjRcq9puWq4W+MRKBBS+W714e7ZrfceeedbrDYv3RUnRp1bvx0jrUMzxsETURafaFVGApS+GlWmgYuNEvNo46gtumcqdjQihUrAolOjTm9trTaInxpbaRC0x4NbqjgXKLXF8kurzHn1WtRQ0DBo/CVGqIUPJrd4J8Z4QWC6eQgnmkVnNL8ea9TpUFRvRd1EBW814CyfxZyeL7iRKv9os9bXdsUYO7fv78bYNISe3+OfK8OQfjKAg1A6Xwmsssvv9ylPgoPsGulQPisYnWsNTtT6XnGjh0bg6ONP2pP6bqvlYDh59BLzxG+otWb/BBehB6Zp/6RV69Fr0kNounaHymdjPoLCi7508fpdR2eVhUIpz6QPyWsBqo0iXDYsGEuiKF2aaRc5gqiH3/88UxgyiSdS9UDU9BIn7E6j3qP+9s+kfr63qxoVr9knV7XaksqbWSkVXCa2OjRCk6NK+gzV+0pb/AWWadzqBWvffr0cRNGwmmSs2rjho8p6L0QKc0acn+s4sD/X7nkrXDWpIlIqy00JqmxNk228KgNE8u0wAQzkGG6SKiBo+X86jR6uezUWdQA8jXXXJNqAPmJJ54Iidr6U0okAkUrNdtanRENZPo74/48pN7KAs1gVcNHS/g1k5AB+YAbFFOdDH+xOi84pgEK/2wKzYL38rIj4GZXqRBq+LnzqIOsoIW/0LR3YSMvbvZoprAGK5V7VZ0cDR5piaYCGf6aQn4aQPLPjADimT4nNACvwWLlfvZWc+l1roFPfW6HD35+/vnnIR1LdUITiT4LNIOpXr16Iavf1L5SZ9w/e00rCzRor3aA6kJ07NjRBY5iNfspXqjQtAaJwtNCeDMAVW/IPzivThcrC/+Prvka0EzLkCFD3EBPeFtKK4M4j1mn964GbrzBAg18KkWvApf6rIxEqebUhgtP/QWkRavTNHCowUWlKfMCGlotoGu1+u/htZc0GdFbfaFrlH+gChmri6H+aI0aNUL6W0qbqRVw9PVznsagIg2YK8WpJpP66fXtT5WD7FHtnfSKpuszSEWhVfPVT5kK/GmMkLtSUlJc7UKtYNL3muCisUj1y9TG1uT1SNQ3iTQpM1YIZiBDNPiggEWk9AaKxKrj46Wv8aggnS4uicybzXbRRReFbFdQR6sz3nnnnZDAh2YQKEWXOpOJtORRjWctW1NgTIO+SrnhzWrXfVrx89RTT4U85qGHHnIDFzpviEznTINmhwp4pFVYFpmnlRRaMaT0Hf5BWw0eqSOpc63BCaXXUeHO5cuXB/fRe15LotNqQADxRA1avY7Di/Sp46Lr27333huyXa91zfxJ9FnFXbt2ddcuf0okUbBCefO9zrjOlwaetUJDaWjC21j5nYISStWhfNYK6vhnXaoei4qle+0Eb7BY51UDSolYCD0jdF50jpRuIy2aoVenTh2X+hDZp0EztbPatm3rVml7g8b63NSEB7UH1PZXLS21axXQ9Lf/lQZIbQXgUNR21IQYvWbCA49aFaz2Z3itSq0OUr0qApVZp8Cvxkj8KwNFExF0/VIxb08i9/WzSxkr1EfSZ6ZWF6kd4E0U9c51+IC5rmO65o0ePTpGR53/KWgXnuYrnCY363+W6JNx4kFycrKbTKUgrD77/YE9TabW6ly9ZzR5XauXtBrD/39TMFCrRuPls4tgBjJEuYaVC89PS8n0QlZnUrn4lRNbjSXNMlLHW7NAIqW2yc/094Yv2+3Xr5+LSIcX81SnRakl/J1xXZQTrUGpJfeKCisnswZ/FeBR8EJRX29QR7n7dA69ot86x3rN9e7dO1WQIxHpvaiBH9UY8S/x1M/qvCioFs5fQF0Dkv6UXcg6zQDSQG74TEo1HpRzXOdaA0l67TZo0MDNxtaMLgV/gbxCM6rU2PXnzdcgqBfY0Ooj3a/PdeXn16ojBfhU5C/RBo91jvzXdS+thGbDhp9TXedee+21kM+NeOkw5Ca1JdU+UntSg28aCNZMMa2+EF3n1OZUqhSPVq/o9aZ6Av4gcaJSe0qDaOHFohUE0qqAcJpk4wWBdI1SekpSTWaflzJOr+fw2gNajV28eHFXR08Fa9u0aeMGRTXxQSu4qUuCzNBAefgsdPWx9L7Wa0+pnnXt0WelrssKrCsFqla7IXN9/fA+vc6p3svhM82VAlmDuP5JH4nY188uDbhqBZsGUZXNYvjw4W4irb/u2uOPP+7GD7w+r2qwaUBW2QdUtxTZDyZpRaw+O/zpj9WeUHAuUm0dZScQ1S9VsC+8PYLYTbQuFpby3vt/adxIbXD9r9WmVptRbRhdW8Jrn8QDghnIEEW0ddHQgIQCG5pdqYEKRcfVYdJFXfkhtV0z6RSB9Q/S53cqKKm0URqY1E0pkLyGjpfKJ7yYp86PPiAefPDBQCLyZudq4CZ8ib2Ke6pQmjfDQo2+Sy65xK3aUGF0LZ8miPG/gS4NGqrxpll9arRpVqqXnkydGM3804zf8EGLRKxfkxv0eahZV6pHEmnVhj4f/MVrNaNLK96YSYy89tmj4IRWXypwp9k8GohXu8ArDqcc0Jr1o88lFboOL9qcn2mwQh0875zoWu9f2eoV2wuf8KHrmj7PtcogETtYahfptaVUXP5aWWoDaIWrVgx4A+waANZnrXL76qagMGkm/5cOSpNBNHiu95/aUv5guQZ8NFgevtJH7YTwnOOIDg3iqP0aHhzS61ppaPyr2DVIpDQPpPpBZik9sd7vCpTr+qOVfmpz6v2uFCIaUFRNPK3Y1uSa++67jzQvmaCBWK0S9Pr6CrJ7s5o16cCbyeync66BXp1rZI4+H72C6epTaXzJT20A/R+82qJqd2ksRm1TreBQ/1eTIZE9mjyiVTAKtKtNofOqr3ptewPgGuwOr7U1adIkFzBF/HnmmWdc30SBwXB33313qvTuixYtimldjPQQzEAINXbUCdKFV0WU/TMPnnzySZfvVWmlFJlTxFv7hq/YSDSa0aJBCUX9NXisAUudP33Qex0XLTGPVMxT0W3N2kokGrTVwJYG39VQ0etJ34dTNNjfcFEjRUvilLrk+++/DyQ6nTsVmtN7UIMXokEgnTN/oEI56pUDUaulNNtS6QoU8Bg/fnwMjz5/UHE5FfBVQ9o/S0WNaNUXijRLRYFOdYaAvECDw5rAoNmbykfsX8GlwJ1mb6utoBV1CpJqADp8xUYi0vVLbQCdEwUsOnfu7NoAM2bMCA58qJ5WeEdPnQdNcIhUmDU/U1tIs1fV+dXgkF5D3rny6Dqn7f4OlVZtqOOl9lciTaBJi1Zj6jxqkoN3TdLsOgV95syZE3yNqZ2gAR+tDFC7X4Pp4SlSkHlaXaWUkjr//tVBSimndlik9v6sWbMiFq8F0qLXjFbxaGKXVvD4C33r9afJX/pcVJtfs6mvu+66VCs2kPkUXgqy672tvr4CRxrwU60Ar62v65je516fzN8P89dywqEprY36t16BaNVn06SZcHqtazWmR+0A1W3T4DurALJP7TFNxrnxxhuDK160TZOZ/StjlYVEbVwF7dQ31v9Aj9MgOGJn9+7dbkK6Ak1eG9Abi9NYkDf5zE+TqSKld49XBDMQTDmjmW3qXGtZuqLdGnBr1qxZusvMlSMv0hshkeiCGd4J1EoNLdNS3lKPCrH5L7iJ2DDRjCDNFtLAutfRVt4+zSRSg9vvvffec7NakTZvxorX0FbnRjODdD79hXcVYFOAQzM19Tr8+uuvOa3ZoIHGc8891wUs1NBWnlYFlrzZ1HotayaclmmGU2F1/S8oeoZ4psF2NX41m1jL+vUZos6KPlvCB5n91HnUPom4ssCj4IUGJ/0znhR8Vkdc6RG99BIaRNYgc/gy70Si1Spanalrk/88aOZfpFWr2o90EWnTBCRvEpLa7srJrlpkulZ5E0i89phWC6szq3a/JpX4Z+Eh87SSWLmm1SbQKiGlnNHqNI8Kf2t7pLRxCmqqLQykR5ME1H/ShDkNqiuwq+uK3sPeLOlI9P6+7LLLOLnZoNTQmqDgpyCl2jv+mjbq56ufhaxPoNG5VvBC1yjvmqXVrMoQ4k+lLLrGHX300ZzuXBpr0IC4VhOqLavghX8FoSY1adKTJkpoohOBjNj68ccf3bWiefPmgaZNm7rJQJqc5lGAVWOV/rTj/vZMeEaZeEUwA8EPIF0ovEa2OkOKwuqFr1keHnWONENeF/Bbb73VdZASrS6G8l37B2o041KBoHCa6aYBT49mauhDQ4GORKR8oZoxqEEe/wenZrhou2ay+mkVhmpp4P9mmyjAo/MSng98ypQprhOt1VN6D3udnfQ6N8g6zURv3bp1cDaWlthq8M2fu1WzUjRzJTyvLpAXqMOo17k/FY1mHWpmogZO/ANy+l4Bvrffftt97iTasn79/Zqx5gXo//jjD9d2Cu/IaSWLts+bNy+4TQNRhyqcmJ89/PDDrk0QXmxaqZCU0sDfidZMMi9dCv5HryXlydd700+BiQsvvNAF2bUiWPvptaf3KKJPgTgNanoBObUNFLxQTmovZ7gmMOjzU/8vICv02ffoo4+6fpPoq1ar673tD/5qAFjXJaXz1UxpreAITyuHzPX1FXDUZ2o4pefTxE+PPm/V19dAIrK2uk35/NUu8J9//T90/b/nnntStSE0WIvo0JieJphoMl74aldt0wQn1b7TZ4zSfkZ6TyD29u3b54JKGjPyaJKr3ldakevRZOxIY5h5CcGMfE7BB83e8HLoZ4RSImlwWXkfVTRJH1zegJwa6ppZpKisilqFR8jzM818V3BCRSfVcNHPoiJUmoGlC62fPkBUK8MvM/+H/EirftToDl9Or5RHmqGqAJmCPUqVoIFgvRbxv8FyzaDUzCrNQvGn2dAAj96jWkYYngtRA0LIPg1Sarml93mn2UFjx44N2UeDSTrn3mCGGhJKzdO9e3f+BYgr6mz7G7OHomublvKrJpbSSGmAzp+DX8FTdWoUzFP6i0SioKU+f08//XQXTBYNImnQUm2kcBrw9F/XNAsxkevlaDBOHa7wWcMajFe+dy8logaGtI8GLShK/T9K8aDgoQY3VTzaH2DUa0+5w3UdEt2n65Par4nUbs9Jek16K2D13tdnQHi7QRNy/MFK9QvUfqC2C8Kp3+MPdB+K+gGaka7rr15/ui5r5a8o7ZSuyfoMVbAj0SYdZocmi2llhSYoqa/vFUfXTOWiRYumeu9qgFefw36J3tfPLtV80fXKq63pUbYLDcb27t3brQLWmILSfMdrLv+8RllZ1J7VJF2NHfpXXSgYqiCdJjN71ObX/4m01fFh7dq1wbolSseu/41W4fppbFdtbq8d7aXA9KcQzmsIZuRzkYrLaSBZs+B104xuP30gqRPupZHQDHq9GfLKUqOcXFWg3Jha+ugtefSoY6jgjxqUfpqVpeJg+D86d8rVHKnOitJNNGjQwAWGlOIs/LWZyFSoL61C8eq0+GcB6xx7Kc0SOa1ZNHkFab0ic0rbET67UuddHUt/4W8FOPS5EP6ZAcTSBx984AbVvZmaur498cQTrp6LOpH+xq8GQ1UsVEv+vUF3pVFRBzI8gJ9oBgwY4NL6eQNIfhpg1uxCb1a2aJWBgvbMkA2lQLzSFYSfR+Vl1uogDf5qEF4zjL0ZyYlOq6mVjzqtQUq1+VW7yaOOqgbWdUsvTRwyTp+VWvkiqp2l/0d4oE3BC62a9Qc41C6j7hvC6fqr+gseLwPCtdde6yYR+KlWkGapezVYNINa/QA9B7JO51GTEJWWK7zdrmuPrkP+FdiiCUteXQdEh5fP31+TwaMMBVodrLECTTCdPXs2pz0K1N5Su179g0hU90VtWm+ChDcRpWXLli5gitibNWuWCzipT6c0t5FWiOvn8MLfWu2Ul9O3EsxIAF5xuZkzZ7qlY/rwUe0CLY3UdjXIPRoE9c8m1CxjXTDU0VTqhESl86YLa1qDklr6qw8Q1SzQjG19MCi6HV7wGwH3OtTrLtFm8GaVXkOHKqirDpDyYWvWitKWqCZDpLzMyDh99vkbyerAeMV6vQLfXqPOoyBcpNnYQLxR7melSlMHRqst27VrF3jkkUdc0F6vba+Gka5tSvfjv/apKKPaBeHFqxOJzodWD/prE4V3xjXQqSJ6Wk2nz2YFPjQgj8jXMAbiMk7X+UgDPeG5kFXnRgF2vQ41e5XAetZptZDSa3jpJTUYoOCk0kdpBqvaaf4aGd7/QQNAwKEomKuZ/wrufvnll27SjAYJb7vtNvdeVmDDo8mI/tR8eg3qsepbMUs967RiVass0lr9p5nr+l8oeKG+vv4/agf4Z7AjOvRZmlY+f0SfgkR6LR8qVbjaaVqldMopp7h6erQpYuvjjz8OTmrxJl4qyK3PMI336vrhp8lqaqsoIJ5fEMxIEJodrBnx+qDyL9tT/lw1fqZPn+5+vu6669xAqGYP6QWvXHiasanCdImcCkE5GVXUKD2aaaU0CNpPy7jSG3zOr/ThqWWfynmZHs0a1IohLoKH5kXR0ysSq3zteu9qSbRmC4cPsiNzNONEM9c1COEFNDQY6S0l17JzFf1W59Ib2ND/Scv8M5MmAIgVpfPTQJwmMGjVhUc5ihXc0GCpt58+f7yZxEqFoQby119/7epAJfJnhM6LBjfS20cr6pR+5qKLLko10JkotBJFHeVDpezR61GvKxyachxrQlJ6VLvmrLPOcm2DRH6vRosmKek9r7RRauuq/apUNN5rW+dZA9De4Jv2URFUzawHMkIrAnR91TXY/1k4depUlwpEwTS5//77XRopXWMUZNNrTO1RDbBr0BFZo9S8WgmfHhVAVtYFDeYqA0N4DUMcmsaYlJ7LXxMjEq28yOv5/PMKBVE1SSm9WpvqDyiQpwlNGhdEbCmjiZm5zyxNTBNlPvEC315aNn9abKUP0zUmP014JZiRj5acp7d0XAPrGnzTEslwGrRQR9sbpFNjXKsKNBtTyye9wbpEpmV36mh7dTL8Em0WjIqYama6ZlJqEEw5+jyKDut1c6iCpsonqtejAh9Inxdd98/K8qiOgwrvIvt0Yfd3AjUIpDoBShGhgTa9ztUo8JZsKuWUXsOqI6COjQLFeXmZJvIXXav0+ZAefX6rIaz2g5/aEv5lyFqBodWZ+hzSqg3lKkbApaPQxIVwOm/heWrzM6XaVA0VDTwoOBZe60rtSHWSly1blu7z3HjjjS6lGW3OQ9P51vXHW0Hlb+v7Cz4ie7wBAm/gQKmkNDFMOcUVzNBr1mvvqv3QtGlT93/RxCal9NLP/udAYtNryEtXmtZn6RFHHOGuv+GTknSt0fVXlOJRNdmUFkY3TVhMbxASGaNrl/r6SgmZ6H397FJKcwXZtGJdbQB/X9VbhXTXXXel+xz5IZ9/XqFJqFoJM2zYsFT3qUYJ4oO/PaH3VKFChVxWGE2k1s+apF69evVUkzA0UV3tc/XjNKaRnxDMyCc0uHyoTqBmHOgirRz7fsrFqVnI3rJKdcI125ACVv9HjUqlmVJuQP/yUw36JNKsAQUf1JnT7H/N+lONgPCih9pHs3cPRdHh9Br1CKRaQaWvHr1H69aty8BPlGjZvlYLeTOFlH5PDXGdcwUqNCNFM7G84lrewJEKr+q1zOwsxJOMdAIVfFY9gkjpamrWrBnyWlcQWykCKb78f/TeV+fPn/9ebYXzzjvPfV4kAr2GNENYg2ma2aeBXbUzv/nmm5DOl9IRHKoIrQLGGrCjQHXGcrsrJZwKom/evNlt03nTamr/LDxk7zNUqy2986lZ8PpZ7231t/RaVZo5pfDyVhmrD6Z27bPPPhuYNGkSq4+RapW/ghXpfcZpcoF/NaRHA1Da7vXhtSJDKzaowRTdFQMKEmnmub+to7TI3qTP/9feuQBbWZV/eCUGEnlJc0ozFQ0tQYU0MRpADU3ES4glSl5GUFAUiVJoQPAyQGDkHSmvGKOCIErekJjEkNQQSYQkNaabQM5kiJFWxn+e5azz/85m73OBw9n77O95Zhg9e2/ObL797bXe9b6/9/dK/ZAjYH+68cYb4/kIkSz5kywU3Snu1Qc2idl4QrYfw4cPj44EWRtwPj+S5VJ+KFiTi6Q4mOjevXvMU9Cp27Vr17gfkC/KWt/RTcZ3km4aYsdqw2JGlcAhkGFgxSqq2S8BNimFSkI8sVES5dnyB1U2iYm61JTYzbRr1y6qDPDAZtEnsZ8XWxkOcCS4ssUwEuoE1/qF1g9JLg4nL774Yq3vGv+PQrpwoHQhDJ9ng0Ltx6B07r3sACfZNihioGzA5x6FJfc7HWocaKZNmxYTFgR0JCpFWgJ4pWJXUVc7MQULuukKRQ6sMRxE8wxrwKOPPlryedYGWu6JC0aNGhXV8hREi3XRVSuIOfh3Z6HlHcW61A/d0NjJpIJEgiQlqn4SlqXAOoqYnu5BXsv3GAVlnmP5pgbRDgXL1G3E8G5m6K1fvz4OoUUEQdGYuE6kPtLQXOwHS8H3lxl4dAZnYW4l563CbixpWnsjEn8kdEkMknAfMWJEPG/5HW8YFB44pzLHJYEVH/du1slBtg/ksRDaFs5sRVTC/lWXNSp5ijPOOCMKUhCqkvPhXIyVtVQGFLWJ9ei4gIkTJ0bhJUIKPjtiQTr4Jk+evDkvWMyoIlACoSAuPBRlQUXEhkKlFajQoeS64IILNlc7HA45eBSDOSIcWBjmVRd44ZKoIKE5bNiweBDNCwwUSrNVEigtiymIEqrV/19phbcyrX8ECX379q11nc4999yYOK8rcZHuPzpfUP7ZUt5w2OTZ+NPcEYK9YvcmwTeKS4KB1O6fDjAkdrnX6WJTOSwtAawosKAoTDZnodBB4EvyJM06Yn3BFqjauzNpycZCK6tCy8JQ9PpiKtYW4ilUh9jN5clHOK2rhbCfMSugGNxTdvd8NOcKpS+dUQxsRIyU7eZhryJWqM+6kDiARDuDa1VoN45FixZtHjt2bK1OjGKQ0GTfR0VMHJAG1bNekuwhLqDzXaQhzJgxI3b4YLVTCs5UiJcokrPOkoTn/KqYpn6wg2IPKlbUbai9EWspsQHXm67C7KxRqRv2okIrUhLrrJPFriPxQLXHms0B9/u4ceNiPEHxrXBf4jrjLoCFcn0wHJqiPXkz47Xmg7zEoEGDagRo5HyKidE4s1BwZcbS8uXLa4SXvPaUU06Jnz22r3nBYkYVMH/+/Hjz0jHADUySvS5oSeJ1tP1xkMKiqr4kaksHpQXzQggOSw2iJOHzxBNPNPt7q0S4H/C3rk/hR6KXeymrwEigEKyvnTovgR3KhtS98thjj8Vrtnr16prXoLQyEdF0cN+SBMp2DDEbCM/IO++8M64FpVTnSXFJIIB1SjYYpGDMASfv97RUNqzHJJmJCTjUUNCoy+InrUkk7FCNkqAj0VfNEPiTTOYQQDdFsb2OuSNcxzzNv6gLkrcNuRY9evSIh6xCiCkokqF2zzMU0eiYogjG4fP999+PnVB0aGbR1q1pITFAV3VWQZzOBKjgmX+Fcr4YkyZNiklQOrhRRZJcTvc0PtT66UtDEoQUyEkm0s2TimKlwDaEfZnkJH8459dVWJePVMucO/lep2HphTTU3kjqh/WyIaJFciusn4VzYIAuzvo6iKV+sItFUJO6KJiZR7xV2N2FyEkqAzrErr766pp7n+4ZOm0RWeD+Qp62lPsL4mI+37Fjx8ZzHnlOIJ5kn6EDLS9YzGjhoNL+zGc+E4NxAnOCaobBFLaXZcGaBsUXszLycEhncWBQJ8OoBgwYENXxpQ4seYdELjZk2I4RRJPUrUs9xMEOv+xicA+ywFbboKHGwAbFNcwmEilcUFHPw3evnNBxhu9t4p577on3NBYo9fnZk7TDN5fPiYSwSEva7zp06BATo6h3aDVGBVqf9RGdBagQmQOTB1Cw0ZLNQYF4iKKzlO7oZZ9n/aSDtS5LRJK8rJulhCEMRs76/eYR7r3CbikSlyTMZfvBDAu+68mek0QCZwMSn9zT9SXTEDawlpKIrvZirzQtzK4iSUUhFwsYkk2sp3XNAiAOxcLwhhtucIh8A4vECDFI8tHFQlGjGv3hKwEKGHS/IIDhPsaSqC7BDHseIodSFt7M36TYLFsHYkgsklORPe13OA1I5UJhiUJ1mk9IF963v/3t+J1i/lwxoXAWLMPatGkTY+48d4dazGjBoCQkMZ+1NSAYx+cRX7y6IBDPS+sYbfi09QK+4ASHbKyl4LrkJaGTvW9YTJkLgBKVVlBUgRTK6mpJxC4pm5jg92QVrtmNNa9k22rZuEiyE2Cg+p8zZ84Wicj6ZmdIcVBIZj0iV65cGQu2BAcwevTo6G/PULqGdKIRrBNgU9RIv0Ok0qHjCJV3FgYzk8RLNmvFoGhNJ0JeeOCBB+JsHBg4cGC0AUw/l1pf8gaFHiz36FxhthrxE5akJH9JQBSDAhoFj2z3GkqxhDHBRx0/2T2F4Y0o7CiujRw5covEJbZv+rU3HmJR1ImoURP8jGgn2UrRmU6BoqH2cMw6oJhRqsNbpBgkGvE2z8IMNsQ1dcWXrKfZ9VNKQ1yfujFQp5PgI+4vhfZGjQchKEpwOlpRlBMX4Q6CTWIpa8nkCJJNthbe08YF277XZW3PKQwhpCR+o+hUaAXOa3EpkOaHLs7Zs2fXsnAnFmEfIF+JsIUuUc4lDeGxxx6LMffFF1+8Oa9YzGjBsBlQvaP6moUkNI/nsY2yIS2P1157bZ1+4CwkqBDztLmme6bQeiep2YslcggEqSjPmzevRnlEq2jh/ZiXQIIKOTZEpRSpBA8Ui7i/CLhRTnNt02DJtKnxGIk2aRx0AJFkw+MzwT1J4Si1NtMNQ9vmdddd16DfyYGo2i34pLrA3znbkZRNpmCzlkcQJ9Sl0KRbDusYEsnFWLp0aex4rWsYeDVCVxoF3WyyjbiIwg/3Wan4KikwSRrj/4udVx4hxiR5TgdrKWsHikMkHejUwHaKbtasgITrTVxVygpN6ga7SLrOEtzLqIhnzpxZ8xgJT657Q1XcdXUrixSD5G9h8pD7qCFzcWRLGnLWnzBhQixUlhJpaG/UeEjCckalWzMLhTru5WKzHFlXSbYmAcSsWbOiqCzPrg3bkvdDpEeuodT1IwfDrDfmwjCvk/2PnFfWsQVBL0X5FStWNOO7l2T/xX5AJ1mCeb3Z7vkFCxbE71Mpi6lC/pDzmMRiRguBqh2WUoWLF3YSxWZk0M7Kc3nxIGxMyyPJSQ7oZ555ZtHnaQFmo83bDANUgXT6ZIMRrgUbXrGDOEpBAhT+y2ZJJTmPhQzaxFFCk3BAZcX9V8wvHArvKTqoCgf6MfAvO+tBGg6Hk06dOtUk32jRpOCWte9gqBndWWnYsUhLhC4CDjUk5bK2idhYcH9nA+V0sGdt4u/kAYrtd9xxx+b27dvHfzfWiRzsSkFXF+3ab7zxxhbPkUQmwdzQImi1kIbQ0jmQhWHxKNSLQSHjqquuqun0ZHZW3iwV8bUnLmdWTc+ePeP3kdiKolkxkUNWOMMhlvs1WSEBQgkKInnppm5KGERLDIt6MTFixIh4BkixLsk2LGnGjBlTxncq1dKJjQis0IaMolqxGRmcGQoTW9I0Z30Sv9jIleoY0N5o66DLgjNvdj+iQEEnTLGZGAj8mM9KroD4gBmS2H5LwyEGRQTJTDuuX7JHLCwqAV2xaW4GkAck/mDfS3BmYMbG2rVr/RiaGXKQ5BiZiZGg85niRdYCm849OmukfixmVDgkPxn6hZoYVSWH7axFEkp6Kq5YqmQ3cA5RJPGmTZu2uZrZ2pZHNl4CobSh8nsK1Yd5g2uHioXrmBLBBC0k6VEUFSYkUAYQUHJvch/m7ZqRsOBwQrKMjooEAQNBRtZeqhRYepXyEZXGQ2DGdx+Fa4KBnzyWEkkE4F26dNl89tln17wmT/Y60rJhncVDG+UV3RZ4RCNcSIcX1mmeGzp0aK2/R4GVffKggw6qeoU3iaTOnTvHmIn/Z29jMCLF91IiBQ7hWMox0yGR3fPytr8l6CTca6+9amyjENWwv1GkKFSNkRwmJiB5T9zakD2w2iDhQEzE9y/ZRWH5wB7UEBsA9jBiU22lmo7BgwfHdS+Ju5iVwWdEF1H2cyP+TVYdnAnq8oAXyYIFHx7nnLspXJDYpSstq2gnWcU8hyycH9iXLaRtn7M+ne+sp1i7pD0qu5fndV/fFtLc1ZRf4t6nKHfEEUdES+/C7gxiBfJUxBEITCzKN47Vq1fHWB8L+TTkmXwVHa98HxpiQ8c83WLCZykPiMo4j2QtME866aRa+SCEVcTTydKSc1t93Wh5xWJGBYNFCsq266+/vqbajbqIjRn1VtqIu3XrtnnfffeNz6Ho7t+/f/yDP3a1d2ZsTctjIikE2HypWqN+yzsE1ChYKVRgJcF9RHKeIIQEGarflAjDh53DeX0DiqoVggm6ADi4FBY5uCe5PvVZUHCgbqhXszQMvvvcq3wOaY3s2LFjtPPI+hCzPtDaSVEDtYvKOKl0OASyZ/Xu3bumZZz7Gyu1rJUK+2FKOHMQYi1iraGzk5+rfV2mKIEtTxYKE+xttHiXAhsp1m6GgdOpWN/ssTyQhtDS9ca8JyyR6B7kIM21IpGUZoyhOOPezHMsRbKG715hFzX7D0X0+mAGGd9xEz5NB53FCLzSkM3Uocl6kO5d4lrEO3xGrBHEBFiEidQH33WSitiVpaIvCV/WAbo0EoifiE2xkaXAiVUfylvO7dnuSmnasz7d73QSkBQkV1LKClgaDoU68lOslfyX+IBkOcU8LPuynxPisjx2aDYV2HVig4ptVKErBN+JrJi5GIh2KSZluz2l/CC2Il5OcDajwIHYOpuTw2mHznHONAwHly2xmFFmCKCZbcHGgJIrO5uAoCgNquN1qONJMGPnw5+kKGCDSEPsWLD4PXVt7HlveUxgncBGgGVC4XCkvEK7PYEIia+s1RGPX3nllXGhxUcbJYZsjvMZuIeyg7xR9uGvXixpyPebJCQbE8PVx48f72XcBvjOcz2zNmioVFCrZwfTk1zjwIP3fYIDJWptvMqzw2pFyg2FTqx6BgwYsMX8nKwNEsp3gmESzKw5DArOKn9I6rM+0aVQamBzNcJ1KDZPjEQGnVqlIKGE/QzJew7pzsv5CK4F1xNrqWwH68KFC+O9R6xFx4b8f8ERkVGWvn37br7kkku2uETE78QMiJEoEJFMTwl22TroqCAuyJ4J6GbD6iR1WxTr0KR7Ewsb5pmZ+JFCUPcPGTIk3h/Z8xFn8axilvMA+wgiGuL8lMQlNiUhSTcBZyn29yS6ke131kewyP5FMhAhk2w75KZYT4kxs3793M98RzhvsZ8pEmsaEDWzZlAEza4z5GqKxamcDdjDyNtQQM0OnJbmhxwa4wKyuVkE51hgIlwvtMBMnynxNp8htvh5m9nXGCxmlBGqpRx68HzEAgIPPA6GhYosAiUOQmzi+KmRzEDxUTismb+XPWjmhca2PHKdUMmRtKegUe2WG40lBX7FPC1RAGQTZrI52pLQKs5BhaIYli8UFWkLxec6G2Sj9OP1qFSci7Ft0G7LtSbA4/tPQTgFANh+kdzNKlbwn8wOVhWpNLA/wXoOlScHQhJtrMXFglgCYBQ7dNGlGIEiXuGhvtq7M0tBF0ESfbDHJwUnFhXZQYgJLCvo0MQqxHk6tWFdZY+j+FsI9xvq47x2aBaD2Cl1ZzLbhsI6P3MNER5lExLYw2HJRWKT2Cuv39emgGvHuok1A2ckzlQpjmUd4HxFIjo7E47XmeCU+uauoKJlP6ErjfMlSUS+24XQrc4+jJ0U+zlnAeL9LOxHeTyrN/dZn+vM9515OBTbPes3LXS6FVrlJLBJROwnTbe3cd5NSn7ESnTB0HGIhVRWzQ+cIzgTMOdN8Wl5Yb2igE1Ogg6b22+/vZYFJlbudVlgSv1YzCgDqDSoslF9yy5A+A9z4MkedICiBQqPbBsq7dEsZB4gG9/yCPysGrs0dF8QIGp1UD8UF5llQ7GRZBgdVBQdr7nmmqgY4gBNO6g0Dagb6LSgS42AjkMhCmEUcNiiJQjwUAZllSokOSzGSSXCnsQ9THCb7TTCxi7rvZ1UPhxish1h+EkTP2SH3ecZisUcHkgkU1hGOIJoBFsPEiIklbPXecWKFZuff/75sr7nlmDzwVor9UOMzx5F/Il4huL7vHnzYicwCaCRI0dqLdNEICQhmcn3u0+fPvFchEKYeS90WSVRQ7Ehm9g2OGRTikEBjOIY+zK2ZOk8xFwb7qPCZC1rJJa82X2FWS3EnXmcH1QJZ/2ZM2cWLTrJtkMClvs7a5Uj2w9cXFKnNX8Q7WFxxxqFwp+CazGhjpQHzmesRxS3iUEQ/pDPZe9ASJ21wJwyZUrN32OvIW5JM9ekfixmVFAbOglPEp+FbacsVFRYE9hF7L///vF36P34EbY8Ni2oKlCsaR3RMEhMFBv6jV0EPrmos4p1ukjjGThwYLTUo9Oi0AKAQC8dMElOEuCRQMqunSripBJhdg737+LFi2vFCnRu8ly2sEwBldemYbU8R0KOGQ+F34s8gyc514kDRFaVySEQyyna7/l/aRgclpnP4hraMKs49n3i9yzch8weo6sqa3EkWw/KYAqXxAXZZA73Kd2YvXr1KjlkE9Vq1iZFJAvFscKOx9///vexaFE4jJUEO/t1gv2ZJDvrZkpeybbhWb8yE+xZqxzZfhDj07ldaC1F1xIDwulCIvaQ8sJ+QWcoMcnUqVNrPUd3H/tCKrJidV1ogZknW+CmwGJGmdvQURNxuJk+fXqsztFeRFDes2fPmq4LknM8hj0Cvnm8hg3Elsna2PLYtGBfUjjEUorDpoQKE7/lYtA6KFsPwVlqlcXXmpZNDpmF0I2R/QzwKD/kkENcK6XiYT/PdsSRYKOziIRoaidPqkNey+sQRGDxw+v4k+ZoyUegbCJpXJhQTt0tFOvt0Gyc3Qp2XVohNYxx48bF729h0jN1aBvDbz18f0kq05UBqLM5UyFiyLJgwYL4eLKOS0M2mVEiUh/cY9wvnIcAcQwJQwrhdGQzd2XNmjXxOe49ujCGDx8eLWbp1EZl6/e8afGsX1kw382OgOaB60w+kAR4IawzdiGVD85f7Bcp3/Pwww/H2IO9IAvP03lx//33x5+JpxGys2/I1mExo8xt6LRFYoHAYKpkRYM9AskLnk+gzGT4FS3R2SG28v/Y8ijl5O67747dGdhJSNPAgCzaaVFRU6jAXxyw0uEgiUqrMKjOWkaQzDTIlpbWEUehjiQ8vqkEvuxt5513XkyUpG4MCnzYqqEyZrithYzi3HrrrbFDC3s6keYEa1gSmsTt0jSwFtIJi6CBvT4ldehoJ8GM1VThZ1DY8YbqUQtVaSj4z3O/MYcJf3OsSkkc0nlNbJqdxYZCHWtIOiS1l90+eNaXPEPhPqvkl/JDjhahOcVtzmqJ4447LorUCqGDj06+BIXwd955p9neb7VhMaMC2tAZCFyo3BgzZkxsT5LGYcujlAu+wxQli3UMSOPBSoeA7b777ivawvmFL3whWnhl185+/fpFj3yRlgqBMJ6qhUU4OjVJypOcl4aD1UynTp3ioUKkuUF9R4HSonrTwKwbvsvFEjm33XZbPDchgkjQXYzwQf9p2VrooiJRxbybwq40hrtSLHN+ZfPiWV/yCklvrKZGjx5d7rcimzdHZx1EK9m4I4GIinMb1qIJLMIQXmjN1nTsGKRsfP7znw9XXnllmDx5cvjzn/8c9t1335rndthhh7DPPvv46TSSk08+OVx11VWhffv2XjtpVj72sY+FBx54IOy5555e+SZgzpw5oXPnzuGcc86peWzTpk2hVatWoU2bNmHKlCnhtNNOC9/4xjdC//79w69+9auwbNmy+LhIS2XixInx3r/33nvDpEmTah7nvgfjgsbBdbvlllvCqlWrEO/EdVqkuTjrrLPCQQcdFL70pS950beR//3vf2HGjBnh2WefrYmzeGzjxo1h1113DYMHDw7Tpk0LPXv2DCNHjgyf/OQn4/nqmmuuCbvvvrvXX7aKXXbZJYwfPz7eX6+88ko44ogjap3VeX7nnXf26jYjnvUlr+y2227hF7/4RTjkkEPK/VYkhHhWu+SSS8Lhhx9ecz02bNgQ94RDDz00XHTRReHiiy+Oe0eXLl3CXXfdFbp16xZzF9I0fIyKRhP9LtkK/vWvf4WDDz443tgPPvhgfGzmzJlhyJAhMWjv06eP11VEcsfChQvj+nf11VeHd955J8yfPz8GAyQoSPYef/zxMRh44YUXQr9+/WLC6MILLzRpIS2eH/7wh2HcuHExAX/ggQeGtWvXhgEDBsQi3uOPPx4TKCIieeOoo44Ke+yxR+jVq1dYsmRJjBNIHJx++unhoYceCosWLQrHHXdcfJ6Y4Fvf+lY45phjyv22pYVD0ezII48M7dq1i8IZIKHIvjxhwoQwcODAcr9FERFpZhClP/LII2Ho0KHh1VdfDU899VT4y1/+EnO7xCM77rhj6NChQ/jc5z4XunbtGsUWZ599do1ATbYdixkVAGpubuw777wzFjLeeuut8NOf/jQWOERE8sqtt94aA4MvfvGLsXBx2GGHheHDh8fk7jPPPBOTvaghKG6ceuqp5X67Ik3CBx98EFVXHTt2DF/72tfCj370o6gKHTt2bGjdurVXWURyyeuvvx7XQTqsSAp8/etfD+vWrQs9evSIxY2jjz46FjbWr18fnnvuuXK/XakiKGJwn7EfI6L5zW9+E2677bZw0kknlfutiYhIGcAxgk5QChgUK0444YQoqjz22GPjuW3UqFHh5ptvjnEL8YvuHU2PxYwKoXv37jExRzs0XRlU8kREZEsVxB/+8Icwe/bs+POwYcPCk08+GVauXGmiV6qGuXPnxqTcN7/5zWiddsABB5T7LYmIVBwULrDpRRWJAnLNmjXR1uvuu++OQjGRpuLMM8+M3ZEkr6644oqw0047eXFFRKRWJx92hHRrDBo0KPz3v/+NwkvEaYjVpWnRq6BCmD59eqzYXXrppRYyRKSqrPQo0rLGFYLCDWuousAJ8T//+U9cHzk80r2GD3aC340N1U033bRd3r9IOejbt29UflLUsJAhItXE4sWL4zwRDvlZVq9eHb761a/GgkRd8Pfee++9KGqgQ+MHP/hBLGQAM/NGjBgRE87EHyJNxY033hjvUWYzWsgQEakOmF2MbeAbb7yxRefFGWecUTMKoK4CBl31dPAx0+fTn/50OP/88+NzCNTZO5iXsXz58u3678gjdmaIiMh2g82b4VgM6KQgwX8TN9xwQ/je974XXnvttehvXYw//elPoXfv3nFeAHMyKGgQJGTBNxsrKjwpRUREpHJhD//HP/4RY4DLLrus5nGECcQCzMFimHcpKFYwR4uZQgzXJEbIQqHj6aefjt1tIiIiIqVgvta8efPCiSeeGB599NFazzGD669//Wt00MHishgUOxBXYiPVv3//aDFVOBdj1qxZ4bTTTov5DGk6LGaIiMh2gwHe77//frj99ttjuyWWOQk6LpYtWxZ9JkVERKS6+ec//xmHY3KoZ2YgIgeGeideeumlODOobdu2ZX2fIiIiUv0wb4vZm6NHj45CCMSTCayt6cTbe++9y/oepTgWM0REZLtBAYMhnQzHIkjA17pUF4aIiIhUL3RiYuewYMGCaA2FipFByiIiIiLNzT777BNWrFgROzTWrl0bfvvb32r730JwZoaIiGxXH8rPfvaz4fLLLw/7779/+O53vxt9JQkaEvyM3cTGjRv9JERERKo8Jth9992jLcNPfvKTGA+8/PLL0Xc6wcygQrsHERERkabiww8/jBaXn/rUp2rmIuEmQVHjrbfeqnkdVlNTp071wlcYFjNERGSrWbp0aRzkPWPGjC2GeQIdGSQuWrduHQsWTzzxRBxofMstt9S85o9//GMc1jl+/Hg/CRERkRYKszCmT58e44I333yzZEwAzLtg3tUJJ5wQ+vTpU2v45qRJk2JnJ79PREREpLGQm2AeBjEFA7oLoWCRZnF26tQpxiU4SRx66KFh0aJFNa+bO3duGDp0aFi8eLEfQgWxY7nfgIiItDx+97vfxSGcr7zySthvv/3C888/Hwdg/fznP681ICupMLGXopjx8Y9/PP7JqhuwnZo8eXIMIkRERKRl8e9//zuqGkkYUKCgkMHMrF/+8pfhK1/5yhYxAR2ZP/7xj6P68e9//3u4//77a1lQXn/99eGOO+7Q6kFEREQaDQWIK664IrRr1y7GHKNGjYq2lpdccskWMQk8/PDD4fHHH4+zPrGcOuuss2peN2TIkBjXYEkllYMzM0REpMFs3rw5WkWReLjqqquigmHHHXcMP/vZz8K5554b5syZE04//fT42nfffTe2bfKaWbNmhTFjxoRjjjkmHHHEETFRMXz4cK+8iIhIC+bZZ58NAwcODB07dgxTpkwJBx54YNiwYUMsYqB4XLJkSc1rL7roovC3v/0tWkshYKCgQUyxatWqsHLlytCmTZuy/ltERESk5UIH6HnnnRfefvvtcNNNN4Vjjz025i8oTiC65HnyE0B+YsKECdH6ktgEUQbxCILN5cuXK7SscLSZEhGRBkPXxbp162LCgeQFhQw455xzajo0Erymffv20QebTo5hw4aFww47LBY3sJRC+SAiIiItl7Zt20bF4qmnnhoLGbDrrruGyy67LLzwwgsxiZCgawM7KWZlMBOD11PQQB15zz33lPFfISIiIi2d3XbbLeYdOnfuHAsZKX+BfdSmTZtqze3s0KFDnIfRt2/fOPi7V69esXPj4IMPjt2lUtnYmSEiIo2CpAOb/Pe///1w7bXXxscoTFDMmDZtWgwIEjy+00471fr7DNrCoxIVp4iIiLRszj///PDUU0+F119/Pey8887xMeKD+fPnh+eee67mdWm2VhJCJH7961+Ho446KrRq1aqZ37mIiIhUE/fee2+44IILYmzRtWvXmi7Sk08+OaxZsybssccedeYq6BTde++9azo4pDKxmCEiIo0GtQJzLlA+fPjhhzFgIEg4/PDDwymnnBLbM1FGiIiISHXD7AtmXtB5OXHixHDzzTdHf2oSAd26dYu2kj169Cj32xQREZEqh45Qihg77LBDLGggrCBX8d5774UuXbpEd4nvfOc78XlpufjpiYhIoxk5cmT0wu7du3c4+uijw/HHHx+9JQcPHhzuuuuumNS47777vLIiIiJVzl577RUtHPCnxk6SxMGTTz4ZHnrooWg12bNnzzBgwICwfv36cr9VERERqWKwlSIeefHFF8OXv/zlOJsLS8tnnnkmzvO68MILY/5i2bJl5X6rsg3YmSEiIlvFgw8+GIdpzZ49O/Tr16/m8Y0bN8bh4FOnTo1/Bg0a5BUWERGpYj744INoH7nnnntGJWQWLKguvfTS2KmxaNGi8IlPfKJs71NERESqH7ov5s6dW2voN7z22mth6NCh4eWXXw5LliyJ87yk5WExQ0REtpru3bvHls2XXnppi1bNDRs2xCGgIiIiUv088sgjcW7W008/HTs2s+BLTZzQunXrsr0/ERERyQcM92bO5+WXXx7Gjx+/xfPvvvtu2GWXXcry3mTbsZghIiJbDe2ZtGsy+JuWTREREckvvXr1CuvWrYvWk4WDvkVERESai+uuuy5MmDAhrFq1KrRv394LX0VYzBARkW2CIVoLFy4Mb775ZmjVqpVXU0REJKe8+uqroXPnzmHGjBmhf//+5X47IiIiklPoCsVG6sQTT4ziS6keLGaIiMg28fbbb4dNmzaF/fbbzyspIiKSc5YuXRqOPPLIcr8NERERyTkrV64MBxxwQGjbtm2534o0IRYzRERERERERERERESkoqk9rVVERERERERERERERKTCsJghIiIiIiIiIiIiIiIVjcUMERERERERERERERGpaCxmiIiIiIiIiIiIiIhIRWMxQ0REREREREREREREKhqLGSIiIiIiIiIiIiIiUtFYzBARERERERERERERkYrGYoaIiIiIiIiIiIiIiFQ0FjNERERERERERERERKSisZghIiIiIiIiIiIiIiIVjcUMERERERERERERERGpaCxmiIiIiIiIiIiIiIhIqGT+D0rVUXNXV478AAAAAElFTkSuQmCC"
     }
    }
   ],
   "source": [
    "plot_df = summary.copy()\n",
    "labels = plot_df[\"variant_label\"].str.replace(\"Round \", \"R\", regex=False)\n",
    "\n",
    "fig, axes = plt.subplots(1, 3, figsize=(16, 4))\n",
    "\n",
    "axes[0].bar(labels, plot_df[\"mean_sync_tokens\"], color=\"#4C78A8\")\n",
    "axes[0].set_title(\"Mean synchronous tokens\")\n",
    "axes[0].set_ylabel(\"sync tokens per ticket\")\n",
    "axes[0].tick_params(axis=\"x\", rotation=30)\n",
    "\n",
    "axes[1].bar(labels, plot_df[\"cost_per_ticket_usd\"], color=\"#59A14F\")\n",
    "axes[1].set_title(\"Estimated cost\")\n",
    "axes[1].set_ylabel(\"USD per ticket\")\n",
    "axes[1].tick_params(axis=\"x\", rotation=30)\n",
    "\n",
    "axes[2].plot(labels, plot_df[\"mean_quality\"], marker=\"o\", color=\"#E15759\")\n",
    "axes[2].set_ylim(0, 1.0)\n",
    "axes[2].set_title(\"Quality score\")\n",
    "axes[2].set_ylabel(\"score\")\n",
    "axes[2].tick_params(axis=\"x\", rotation=30)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8178d42a",
   "metadata": {},
   "source": [
    "## Optional: live Responses API tool loop\n",
    "\n",
    "The implementation in [live_api.py](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/live_api.py) forwards `response.output` before appending function results, preserving reasoning and tool-call items. Follow-up requests retain the configured tool choice, allowing an order and policy lookup followed by a refund call. After `max_tool_rounds` batches, the final request uses `tool_choice=\"none\"` to obtain an answer without executing more tools.\n",
    "\n",
    "The example uses a refund ticket and derives its allowed tools from that same ticket. This router still uses fixture labels; replace it with evaluated application logic for live traffic. An incomplete response or an unexpected final tool call raises an error instead of being reported as a completed answer.\n",
    "\n",
    "For separate conversational turns, `previous_response_id` can carry state. Supply instructions again when they should apply to the next request.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d62e488",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.257431Z",
     "iopub.status.busy": "2026-04-30T18:28:05.257370Z",
     "iopub.status.idle": "2026-04-30T18:28:05.286284Z",
     "shell.execute_reply": "2026-04-30T18:28:05.285885Z"
    }
   },
   "outputs": [],
   "source": [
    "from live_api import (\n",
    "    background_followup_request,\n",
    "    live_config_for_ticket,\n",
    "    run_live_support_ticket,\n",
    ")\n",
    "\n",
    "# Keep the allowed tools tied to the ticket being evaluated.\n",
    "live_ticket = EVAL_SET[2]  # Order/policy lookup, then open a refund case.\n",
    "live_config = live_config_for_ticket(live_ticket, \"01_prompt_tool_context_controls\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4678229f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.287235Z",
     "iopub.status.busy": "2026-04-30T18:28:05.287176Z",
     "iopub.status.idle": "2026-04-30T18:28:05.288967Z",
     "shell.execute_reply": "2026-04-30T18:28:05.288597Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Your refund request is eligible, and I’ve opened a return/refund case for order O-1003.\n\nNext step: please use the return instructions from your order page or confirmation email to send the item back. Once the return is received and processed, your refund will be issued.\n"
    },
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "                            config ticket_id    model  tool_calls  latency_s  estimated_cost_usd  input_tokens  cached_tokens  \\\n0  01_prompt_tool_context_controls     T-003  gpt-5.4           3   8.056909            0.008085          1788              0   \n\n   output_tokens  reasoning_tokens  total_tokens  \n0            241                84          2029  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>config</th>\n      <th>ticket_id</th>\n      <th>model</th>\n      <th>tool_calls</th>\n      <th>latency_s</th>\n      <th>estimated_cost_usd</th>\n      <th>input_tokens</th>\n      <th>cached_tokens</th>\n      <th>output_tokens</th>\n      <th>reasoning_tokens</th>\n      <th>total_tokens</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>01_prompt_tool_context_controls</td>\n      <td>T-003</td>\n      <td>gpt-5.4</td>\n      <td>3</td>\n      <td>8.056909</td>\n      <td>0.008085</td>\n      <td>1788</td>\n      <td>0</td>\n      <td>241</td>\n      <td>84</td>\n      <td>2029</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     }
    }
   ],
   "source": [
    "if RUN_LIVE_API_CALLS:\n",
    "    live_result = run_live_support_ticket(live_ticket, live_config, client=client)\n",
    "    print(live_result[\"response_text\"])\n",
    "    display(pd.DataFrame([{k: v for k, v in live_result.items() if k not in {\"response_text\", \"tool_results\"}}]))\n",
    "else:\n",
    "    print(\"Dry-run mode. Set OPENAI_API_KEY and RUN_LIVE_API_CALLS=true to run a live Responses API ticket.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45580af6",
   "metadata": {},
   "source": [
    "## Optimization round 1: prompt, tool, and context controls\n",
    "\n",
    "Start with concrete response and tool rules. The request below combines low verbosity and reasoning effort with an output cap and an allowed tool subset. The output cap includes both visible and reasoning tokens, so check for incomplete responses when tuning it.\n",
    "\n",
    "The helper also limits tool rounds and returns slim payloads. For long conversations, evaluate compaction or truncation carefully: removing earlier context can discard facts needed for the next decision.\n",
    "\n",
    "This demo restricts tools using known ticket metadata. A production router needs separate evaluation and a fallback for low-confidence routing. If you use prompt optimization, target a specific observed failure and rerun the same evals.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "484bb6e2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.289935Z",
     "iopub.status.busy": "2026-04-30T18:28:05.289879Z",
     "iopub.status.idle": "2026-04-30T18:28:05.291972Z",
     "shell.execute_reply": "2026-04-30T18:28:05.291540Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "{\n  \"model\": \"gpt-5.4\",\n  \"instructions\": \"Role: E-commerce support assistant.\\n\\nGoal: Resolve routine support tickets with the fewest necessary tool calls while preserving policy correctness.\\n\\nTool rules:\\n- Use only tools required for the current decision.\\n- Order status: order lookup only.\\n- Damaged delivery or refund: order lookup plus the relevant policy.\\n- Billing duplicate charge: order lookup plus billing policy, then escalate.\\n- Account access with unverified identity: customer lookup plus account policy, then escalate.\\n\\nResponse rules:\\n- Give the customer the outcome and next step.\\n- Do not expose internal reasoning, raw tool data, audit notes, or policy text.\\n- Keep the customer-facing answer under 120 words unless escalation legally requires more detail.\",\n  \"tools\": [\n    {\n      \"type\": \"function\",\n      \"name\": \"lookup_customer\",\n      \"description\": \"Fetch minimal customer verification and support tier fields.\",\n      \"parameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n          \"customer_id\": {\n            \"type\": \"string\"\n          }\n        },\n        \"required\": [\n          \"customer_id\"\n        ],\n        \"additionalProperties\": false\n      },\n      \"strict\": true\n    },\n    {\n      \"type\": \"function\",\n      \"name\": \"lookup_order\",\n      \"description\": \"Fetch order status, delivery age, payment status, and item value.\",\n      \"parameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n          \"order_id\": {\n            \"type\": \"string\"\n          }\n        },\n        \"required\": [\n          \"order_id\"\n        ],\n        \"additionalProperties\": false\n      },\n      \"strict\": true\n    },\n    {\n      \"type\": \"function\",\n      \"name\": \"lookup_policy\",\n      \"description\": \"Fetch the policy needed for the current support decision.\",\n      \"parameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n          \"topic\": {\n            \"type\": \"string\",\n            \"enum\": [\n              \"shipping\",\n              \"damaged_delivery\",\n              \"refunds\",\n              \"billing\",\n              \"account_access\"\n            ]\n          }\n        },\n        \"required\": [\n          \"topic\"\n        ],\n        \"additionalProperties\": false\n      },\n      \"strict\": true\n    },\n    {\n      \"type\": \"function\",\n      \"name\": \"create_refund_case\",\n      \"description\": \"Open a refund or replacement case only after polic\n...\n"
    }
   ],
   "source": [
    "round1_request_example = {\n",
    "    \"model\": \"gpt-5.4\",\n",
    "    \"instructions\": CONTROLLED_PROMPT,\n",
    "    \"tools\": SLIM_TOOLS,\n",
    "    \"tool_choice\": allowed_tool_choice([\"lookup_order\", \"lookup_policy\"], mode=\"auto\"),\n",
    "    \"reasoning\": {\"effort\": \"low\"},\n",
    "    \"text\": {\"verbosity\": \"low\"},\n",
    "    \"max_output_tokens\": 350,\n",
    "    \"parallel_tool_calls\": True,\n",
    "    \"truncation\": \"auto\",\n",
    "    \"context_management\": [{\"type\": \"compaction\", \"compact_threshold\": 20_000}],\n",
    "    \"input\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"My blender arrived cracked. Order O-1002. Can you replace it?\",\n",
    "        }\n",
    "    ],\n",
    "}\n",
    "\n",
    "print(json.dumps(round1_request_example, indent=2)[:2400] + \"\\n...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2f4cda8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.292887Z",
     "iopub.status.busy": "2026-04-30T18:28:05.292824Z",
     "iopub.status.idle": "2026-04-30T18:28:05.298411Z",
     "shell.execute_reply": "2026-04-30T18:28:05.298110Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d1d040>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_3f362\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_3f362_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_3f362_level0_col1\" class=\"col_heading level0 col1\" >ticket_id</th>\n      <th id=\"T_3f362_level0_col2\" class=\"col_heading level0 col2\" >intent</th>\n      <th id=\"T_3f362_level0_col3\" class=\"col_heading level0 col3\" >tools</th>\n      <th id=\"T_3f362_level0_col4\" class=\"col_heading level0 col4\" >action</th>\n      <th id=\"T_3f362_level0_col5\" class=\"col_heading level0 col5\" >extra_tool_calls</th>\n      <th id=\"T_3f362_level0_col6\" class=\"col_heading level0 col6\" >policy_compliant</th>\n      <th id=\"T_3f362_level0_col7\" class=\"col_heading level0 col7\" >concise</th>\n      <th id=\"T_3f362_level0_col8\" class=\"col_heading level0 col8\" >visible_output_tokens</th>\n      <th id=\"T_3f362_level0_col9\" class=\"col_heading level0 col9\" >total_tokens</th>\n      <th id=\"T_3f362_level0_col10\" class=\"col_heading level0 col10\" >latency_s</th>\n      <th id=\"T_3f362_level0_col11\" class=\"col_heading level0 col11\" >cost_usd</th>\n      <th id=\"T_3f362_level0_col12\" class=\"col_heading level0 col12\" >quality_score</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_3f362_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n      <td id=\"T_3f362_row0_col0\" class=\"data row0 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row0_col1\" class=\"data row0 col1\" >T-001</td>\n      <td id=\"T_3f362_row0_col2\" class=\"data row0 col2\" >order_status</td>\n      <td id=\"T_3f362_row0_col3\" class=\"data row0 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row0_col4\" class=\"data row0 col4\" >provide_status_eta</td>\n      <td id=\"T_3f362_row0_col5\" class=\"data row0 col5\" >4</td>\n      <td id=\"T_3f362_row0_col6\" class=\"data row0 col6\" >False</td>\n      <td id=\"T_3f362_row0_col7\" class=\"data row0 col7\" >False</td>\n      <td id=\"T_3f362_row0_col8\" class=\"data row0 col8\" >167</td>\n      <td id=\"T_3f362_row0_col9\" class=\"data row0 col9\" >12913</td>\n      <td id=\"T_3f362_row0_col10\" class=\"data row0 col10\" >4.88</td>\n      <td id=\"T_3f362_row0_col11\" class=\"data row0 col11\" >$0.03993</td>\n      <td id=\"T_3f362_row0_col12\" class=\"data row0 col12\" >0.55</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n      <td id=\"T_3f362_row1_col0\" class=\"data row1 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row1_col1\" class=\"data row1 col1\" >T-002</td>\n      <td id=\"T_3f362_row1_col2\" class=\"data row1 col2\" >damaged_delivery</td>\n      <td id=\"T_3f362_row1_col3\" class=\"data row1 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row1_col4\" class=\"data row1 col4\" >open_replacement_without_photo</td>\n      <td id=\"T_3f362_row1_col5\" class=\"data row1 col5\" >3</td>\n      <td id=\"T_3f362_row1_col6\" class=\"data row1 col6\" >False</td>\n      <td id=\"T_3f362_row1_col7\" class=\"data row1 col7\" >False</td>\n      <td id=\"T_3f362_row1_col8\" class=\"data row1 col8\" >174</td>\n      <td id=\"T_3f362_row1_col9\" class=\"data row1 col9\" >12917</td>\n      <td id=\"T_3f362_row1_col10\" class=\"data row1 col10\" >4.88</td>\n      <td id=\"T_3f362_row1_col11\" class=\"data row1 col11\" >$0.04034</td>\n      <td id=\"T_3f362_row1_col12\" class=\"data row1 col12\" >0.65</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n      <td id=\"T_3f362_row2_col0\" class=\"data row2 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row2_col1\" class=\"data row2 col1\" >T-003</td>\n      <td id=\"T_3f362_row2_col2\" class=\"data row2 col2\" >refund_eligibility</td>\n      <td id=\"T_3f362_row2_col3\" class=\"data row2 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row2_col4\" class=\"data row2 col4\" >escalate_refund_review</td>\n      <td id=\"T_3f362_row2_col5\" class=\"data row2 col5\" >2</td>\n      <td id=\"T_3f362_row2_col6\" class=\"data row2 col6\" >False</td>\n      <td id=\"T_3f362_row2_col7\" class=\"data row2 col7\" >False</td>\n      <td id=\"T_3f362_row2_col8\" class=\"data row2 col8\" >170</td>\n      <td id=\"T_3f362_row2_col9\" class=\"data row2 col9\" >12818</td>\n      <td id=\"T_3f362_row2_col10\" class=\"data row2 col10\" >4.87</td>\n      <td id=\"T_3f362_row2_col11\" class=\"data row2 col11\" >$0.04004</td>\n      <td id=\"T_3f362_row2_col12\" class=\"data row2 col12\" >0.22</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n      <td id=\"T_3f362_row3_col0\" class=\"data row3 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row3_col1\" class=\"data row3 col1\" >T-004</td>\n      <td id=\"T_3f362_row3_col2\" class=\"data row3 col2\" >billing_issue</td>\n      <td id=\"T_3f362_row3_col3\" class=\"data row3 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row3_col4\" class=\"data row3 col4\" >escalate_billing_review</td>\n      <td id=\"T_3f362_row3_col5\" class=\"data row3 col5\" >2</td>\n      <td id=\"T_3f362_row3_col6\" class=\"data row3 col6\" >True</td>\n      <td id=\"T_3f362_row3_col7\" class=\"data row3 col7\" >False</td>\n      <td id=\"T_3f362_row3_col8\" class=\"data row3 col8\" >169</td>\n      <td id=\"T_3f362_row3_col9\" class=\"data row3 col9\" >12830</td>\n      <td id=\"T_3f362_row3_col10\" class=\"data row3 col10\" >4.87</td>\n      <td id=\"T_3f362_row3_col11\" class=\"data row3 col11\" >$0.04038</td>\n      <td id=\"T_3f362_row3_col12\" class=\"data row3 col12\" >0.85</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n      <td id=\"T_3f362_row4_col0\" class=\"data row4 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row4_col1\" class=\"data row4 col1\" >T-005</td>\n      <td id=\"T_3f362_row4_col2\" class=\"data row4 col2\" >account_access</td>\n      <td id=\"T_3f362_row4_col3\" class=\"data row4 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row4_col4\" class=\"data row4 col4\" >escalate_account_security</td>\n      <td id=\"T_3f362_row4_col5\" class=\"data row4 col5\" >2</td>\n      <td id=\"T_3f362_row4_col6\" class=\"data row4 col6\" >False</td>\n      <td id=\"T_3f362_row4_col7\" class=\"data row4 col7\" >False</td>\n      <td id=\"T_3f362_row4_col8\" class=\"data row4 col8\" >172</td>\n      <td id=\"T_3f362_row4_col9\" class=\"data row4 col9\" >8112</td>\n      <td id=\"T_3f362_row4_col10\" class=\"data row4 col10\" >3.97</td>\n      <td id=\"T_3f362_row4_col11\" class=\"data row4 col11\" >$0.02899</td>\n      <td id=\"T_3f362_row4_col12\" class=\"data row4 col12\" >0.60</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n      <td id=\"T_3f362_row5_col0\" class=\"data row5 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row5_col1\" class=\"data row5 col1\" >T-006</td>\n      <td id=\"T_3f362_row5_col2\" class=\"data row5 col2\" >refund_dispute</td>\n      <td id=\"T_3f362_row5_col3\" class=\"data row5 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row5_col4\" class=\"data row5 col4\" >escalate_refund_review</td>\n      <td id=\"T_3f362_row5_col5\" class=\"data row5 col5\" >2</td>\n      <td id=\"T_3f362_row5_col6\" class=\"data row5 col6\" >False</td>\n      <td id=\"T_3f362_row5_col7\" class=\"data row5 col7\" >False</td>\n      <td id=\"T_3f362_row5_col8\" class=\"data row5 col8\" >168</td>\n      <td id=\"T_3f362_row5_col9\" class=\"data row5 col9\" >12869</td>\n      <td id=\"T_3f362_row5_col10\" class=\"data row5 col10\" >4.88</td>\n      <td id=\"T_3f362_row5_col11\" class=\"data row5 col11\" >$0.04077</td>\n      <td id=\"T_3f362_row5_col12\" class=\"data row5 col12\" >0.60</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n      <td id=\"T_3f362_row6_col0\" class=\"data row6 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row6_col1\" class=\"data row6 col1\" >T-007</td>\n      <td id=\"T_3f362_row6_col2\" class=\"data row6 col2\" >delivered_not_received</td>\n      <td id=\"T_3f362_row6_col3\" class=\"data row6 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row6_col4\" class=\"data row6 col4\" >start_delivery_trace_steps</td>\n      <td id=\"T_3f362_row6_col5\" class=\"data row6 col5\" >3</td>\n      <td id=\"T_3f362_row6_col6\" class=\"data row6 col6\" >False</td>\n      <td id=\"T_3f362_row6_col7\" class=\"data row6 col7\" >False</td>\n      <td id=\"T_3f362_row6_col8\" class=\"data row6 col8\" >172</td>\n      <td id=\"T_3f362_row6_col9\" class=\"data row6 col9\" >12862</td>\n      <td id=\"T_3f362_row6_col10\" class=\"data row6 col10\" >4.87</td>\n      <td id=\"T_3f362_row6_col11\" class=\"data row6 col11\" >$0.04018</td>\n      <td id=\"T_3f362_row6_col12\" class=\"data row6 col12\" >0.57</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n      <td id=\"T_3f362_row7_col0\" class=\"data row7 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row7_col1\" class=\"data row7 col1\" >T-008</td>\n      <td id=\"T_3f362_row7_col2\" class=\"data row7 col2\" >high_value_damage</td>\n      <td id=\"T_3f362_row7_col3\" class=\"data row7 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row7_col4\" class=\"data row7 col4\" >promise_refund_high_value_damage</td>\n      <td id=\"T_3f362_row7_col5\" class=\"data row7 col5\" >2</td>\n      <td id=\"T_3f362_row7_col6\" class=\"data row7 col6\" >False</td>\n      <td id=\"T_3f362_row7_col7\" class=\"data row7 col7\" >False</td>\n      <td id=\"T_3f362_row7_col8\" class=\"data row7 col8\" >174</td>\n      <td id=\"T_3f362_row7_col9\" class=\"data row7 col9\" >12997</td>\n      <td id=\"T_3f362_row7_col10\" class=\"data row7 col10\" >4.91</td>\n      <td id=\"T_3f362_row7_col11\" class=\"data row7 col11\" >$0.04136</td>\n      <td id=\"T_3f362_row7_col12\" class=\"data row7 col12\" >0.22</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n      <td id=\"T_3f362_row8_col0\" class=\"data row8 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row8_col1\" class=\"data row8 col1\" >T-009</td>\n      <td id=\"T_3f362_row8_col2\" class=\"data row8 col2\" >refund_eligibility</td>\n      <td id=\"T_3f362_row8_col3\" class=\"data row8 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row8_col4\" class=\"data row8 col4\" >escalate_refund_review</td>\n      <td id=\"T_3f362_row8_col5\" class=\"data row8 col5\" >2</td>\n      <td id=\"T_3f362_row8_col6\" class=\"data row8 col6\" >False</td>\n      <td id=\"T_3f362_row8_col7\" class=\"data row8 col7\" >False</td>\n      <td id=\"T_3f362_row8_col8\" class=\"data row8 col8\" >169</td>\n      <td id=\"T_3f362_row8_col9\" class=\"data row8 col9\" >12921</td>\n      <td id=\"T_3f362_row8_col10\" class=\"data row8 col10\" >4.88</td>\n      <td id=\"T_3f362_row8_col11\" class=\"data row8 col11\" >$0.04029</td>\n      <td id=\"T_3f362_row8_col12\" class=\"data row8 col12\" >0.22</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n      <td id=\"T_3f362_row9_col0\" class=\"data row9 col0\" >Bad baseline</td>\n      <td id=\"T_3f362_row9_col1\" class=\"data row9 col1\" >T-010</td>\n      <td id=\"T_3f362_row9_col2\" class=\"data row9 col2\" >account_access</td>\n      <td id=\"T_3f362_row9_col3\" class=\"data row9 col3\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_3f362_row9_col4\" class=\"data row9 col4\" >escalate_account_security</td>\n      <td id=\"T_3f362_row9_col5\" class=\"data row9 col5\" >2</td>\n      <td id=\"T_3f362_row9_col6\" class=\"data row9 col6\" >False</td>\n      <td id=\"T_3f362_row9_col7\" class=\"data row9 col7\" >False</td>\n      <td id=\"T_3f362_row9_col8\" class=\"data row9 col8\" >172</td>\n      <td id=\"T_3f362_row9_col9\" class=\"data row9 col9\" >8115</td>\n      <td id=\"T_3f362_row9_col10\" class=\"data row9 col10\" >3.97</td>\n      <td id=\"T_3f362_row9_col11\" class=\"data row9 col11\" >$0.02900</td>\n      <td id=\"T_3f362_row9_col12\" class=\"data row9 col12\" >0.60</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n      <td id=\"T_3f362_row10_col0\" class=\"data row10 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row10_col1\" class=\"data row10 col1\" >T-001</td>\n      <td id=\"T_3f362_row10_col2\" class=\"data row10 col2\" >order_status</td>\n      <td id=\"T_3f362_row10_col3\" class=\"data row10 col3\" >lookup_order</td>\n      <td id=\"T_3f362_row10_col4\" class=\"data row10 col4\" >provide_status_eta</td>\n      <td id=\"T_3f362_row10_col5\" class=\"data row10 col5\" >0</td>\n      <td id=\"T_3f362_row10_col6\" class=\"data row10 col6\" >True</td>\n      <td id=\"T_3f362_row10_col7\" class=\"data row10 col7\" >True</td>\n      <td id=\"T_3f362_row10_col8\" class=\"data row10 col8\" >42</td>\n      <td id=\"T_3f362_row10_col9\" class=\"data row10 col9\" >1252</td>\n      <td id=\"T_3f362_row10_col10\" class=\"data row10 col10\" >2.00</td>\n      <td id=\"T_3f362_row10_col11\" class=\"data row10 col11\" >$0.00418</td>\n      <td id=\"T_3f362_row10_col12\" class=\"data row10 col12\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n      <td id=\"T_3f362_row11_col0\" class=\"data row11 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row11_col1\" class=\"data row11 col1\" >T-002</td>\n      <td id=\"T_3f362_row11_col2\" class=\"data row11 col2\" >damaged_delivery</td>\n      <td id=\"T_3f362_row11_col3\" class=\"data row11 col3\" >lookup_order, lookup_policy</td>\n      <td id=\"T_3f362_row11_col4\" class=\"data row11 col4\" >request_photo_then_offer_replacement</td>\n      <td id=\"T_3f362_row11_col5\" class=\"data row11 col5\" >0</td>\n      <td id=\"T_3f362_row11_col6\" class=\"data row11 col6\" >True</td>\n      <td id=\"T_3f362_row11_col7\" class=\"data row11 col7\" >True</td>\n      <td id=\"T_3f362_row11_col8\" class=\"data row11 col8\" >45</td>\n      <td id=\"T_3f362_row11_col9\" class=\"data row11 col9\" >1354</td>\n      <td id=\"T_3f362_row11_col10\" class=\"data row11 col10\" >2.17</td>\n      <td id=\"T_3f362_row11_col11\" class=\"data row11 col11\" >$0.00482</td>\n      <td id=\"T_3f362_row11_col12\" class=\"data row11 col12\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n      <td id=\"T_3f362_row12_col0\" class=\"data row12 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row12_col1\" class=\"data row12 col1\" >T-003</td>\n      <td id=\"T_3f362_row12_col2\" class=\"data row12 col2\" >refund_eligibility</td>\n      <td id=\"T_3f362_row12_col3\" class=\"data row12 col3\" >lookup_order, lookup_policy, create_refund_case</td>\n      <td id=\"T_3f362_row12_col4\" class=\"data row12 col4\" >open_refund_case</td>\n      <td id=\"T_3f362_row12_col5\" class=\"data row12 col5\" >0</td>\n      <td id=\"T_3f362_row12_col6\" class=\"data row12 col6\" >True</td>\n      <td id=\"T_3f362_row12_col7\" class=\"data row12 col7\" >True</td>\n      <td id=\"T_3f362_row12_col8\" class=\"data row12 col8\" >51</td>\n      <td id=\"T_3f362_row12_col9\" class=\"data row12 col9\" >1393</td>\n      <td id=\"T_3f362_row12_col10\" class=\"data row12 col10\" >2.32</td>\n      <td id=\"T_3f362_row12_col11\" class=\"data row12 col11\" >$0.00512</td>\n      <td id=\"T_3f362_row12_col12\" class=\"data row12 col12\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row13\" class=\"row_heading level0 row13\" >13</th>\n      <td id=\"T_3f362_row13_col0\" class=\"data row13 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row13_col1\" class=\"data row13 col1\" >T-004</td>\n      <td id=\"T_3f362_row13_col2\" class=\"data row13 col2\" >billing_issue</td>\n      <td id=\"T_3f362_row13_col3\" class=\"data row13 col3\" >lookup_order, lookup_policy, escalate_to_human</td>\n      <td id=\"T_3f362_row13_col4\" class=\"data row13 col4\" >escalate_billing_review</td>\n      <td id=\"T_3f362_row13_col5\" class=\"data row13 col5\" >0</td>\n      <td id=\"T_3f362_row13_col6\" class=\"data row13 col6\" >True</td>\n      <td id=\"T_3f362_row13_col7\" class=\"data row13 col7\" >True</td>\n      <td id=\"T_3f362_row13_col8\" class=\"data row13 col8\" >35</td>\n      <td id=\"T_3f362_row13_col9\" class=\"data row13 col9\" >1383</td>\n      <td id=\"T_3f362_row13_col10\" class=\"data row13 col10\" >2.31</td>\n      <td id=\"T_3f362_row13_col11\" class=\"data row13 col11\" >$0.00512</td>\n      <td id=\"T_3f362_row13_col12\" class=\"data row13 col12\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row14\" class=\"row_heading level0 row14\" >14</th>\n      <td id=\"T_3f362_row14_col0\" class=\"data row14 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row14_col1\" class=\"data row14 col1\" >T-005</td>\n      <td id=\"T_3f362_row14_col2\" class=\"data row14 col2\" >account_access</td>\n      <td id=\"T_3f362_row14_col3\" class=\"data row14 col3\" >lookup_customer, lookup_policy, escalate_to_human</td>\n      <td id=\"T_3f362_row14_col4\" class=\"data row14 col4\" >escalate_account_security</td>\n      <td id=\"T_3f362_row14_col5\" class=\"data row14 col5\" >0</td>\n      <td id=\"T_3f362_row14_col6\" class=\"data row14 col6\" >True</td>\n      <td id=\"T_3f362_row14_col7\" class=\"data row14 col7\" >True</td>\n      <td id=\"T_3f362_row14_col8\" class=\"data row14 col8\" >39</td>\n      <td id=\"T_3f362_row14_col9\" class=\"data row14 col9\" >1378</td>\n      <td id=\"T_3f362_row14_col10\" class=\"data row14 col10\" >2.32</td>\n      <td id=\"T_3f362_row14_col11\" class=\"data row14 col11\" >$0.00543</td>\n      <td id=\"T_3f362_row14_col12\" class=\"data row14 col12\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row15\" class=\"row_heading level0 row15\" >15</th>\n      <td id=\"T_3f362_row15_col0\" class=\"data row15 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row15_col1\" class=\"data row15 col1\" >T-006</td>\n      <td id=\"T_3f362_row15_col2\" class=\"data row15 col2\" >refund_dispute</td>\n      <td id=\"T_3f362_row15_col3\" class=\"data row15 col3\" >lookup_order, lookup_policy, escalate_to_human</td>\n      <td id=\"T_3f362_row15_col4\" class=\"data row15 col4\" >escalate_refund_review</td>\n      <td id=\"T_3f362_row15_col5\" class=\"data row15 col5\" >0</td>\n      <td id=\"T_3f362_row15_col6\" class=\"data row15 col6\" >True</td>\n      <td id=\"T_3f362_row15_col7\" class=\"data row15 col7\" >True</td>\n      <td id=\"T_3f362_row15_col8\" class=\"data row15 col8\" >36</td>\n      <td id=\"T_3f362_row15_col9\" class=\"data row15 col9\" >1418</td>\n      <td id=\"T_3f362_row15_col10\" class=\"data row15 col10\" >2.32</td>\n      <td id=\"T_3f362_row15_col11\" class=\"data row15 col11\" >$0.00545</td>\n      <td id=\"T_3f362_row15_col12\" class=\"data row15 col12\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row16\" class=\"row_heading level0 row16\" >16</th>\n      <td id=\"T_3f362_row16_col0\" class=\"data row16 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row16_col1\" class=\"data row16 col1\" >T-007</td>\n      <td id=\"T_3f362_row16_col2\" class=\"data row16 col2\" >delivered_not_received</td>\n      <td id=\"T_3f362_row16_col3\" class=\"data row16 col3\" >lookup_order, lookup_policy</td>\n      <td id=\"T_3f362_row16_col4\" class=\"data row16 col4\" >start_delivery_trace_steps</td>\n      <td id=\"T_3f362_row16_col5\" class=\"data row16 col5\" >0</td>\n      <td id=\"T_3f362_row16_col6\" class=\"data row16 col6\" >True</td>\n      <td id=\"T_3f362_row16_col7\" class=\"data row16 col7\" >True</td>\n      <td id=\"T_3f362_row16_col8\" class=\"data row16 col8\" >41</td>\n      <td id=\"T_3f362_row16_col9\" class=\"data row16 col9\" >1390</td>\n      <td id=\"T_3f362_row16_col10\" class=\"data row16 col10\" >2.17</td>\n      <td id=\"T_3f362_row16_col11\" class=\"data row16 col11\" >$0.00486</td>\n      <td id=\"T_3f362_row16_col12\" class=\"data row16 col12\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row17\" class=\"row_heading level0 row17\" >17</th>\n      <td id=\"T_3f362_row17_col0\" class=\"data row17 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row17_col1\" class=\"data row17 col1\" >T-008</td>\n      <td id=\"T_3f362_row17_col2\" class=\"data row17 col2\" >high_value_damage</td>\n      <td id=\"T_3f362_row17_col3\" class=\"data row17 col3\" >lookup_order, lookup_policy, escalate_to_human</td>\n      <td id=\"T_3f362_row17_col4\" class=\"data row17 col4\" >escalate_high_value_damage</td>\n      <td id=\"T_3f362_row17_col5\" class=\"data row17 col5\" >0</td>\n      <td id=\"T_3f362_row17_col6\" class=\"data row17 col6\" >True</td>\n      <td id=\"T_3f362_row17_col7\" class=\"data row17 col7\" >True</td>\n      <td id=\"T_3f362_row17_col8\" class=\"data row17 col8\" >38</td>\n      <td id=\"T_3f362_row17_col9\" class=\"data row17 col9\" >1445</td>\n      <td id=\"T_3f362_row17_col10\" class=\"data row17 col10\" >2.33</td>\n      <td id=\"T_3f362_row17_col11\" class=\"data row17 col11\" >$0.00568</td>\n      <td id=\"T_3f362_row17_col12\" class=\"data row17 col12\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row18\" class=\"row_heading level0 row18\" >18</th>\n      <td id=\"T_3f362_row18_col0\" class=\"data row18 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row18_col1\" class=\"data row18 col1\" >T-009</td>\n      <td id=\"T_3f362_row18_col2\" class=\"data row18 col2\" >refund_eligibility</td>\n      <td id=\"T_3f362_row18_col3\" class=\"data row18 col3\" >lookup_order, lookup_policy, create_refund_case</td>\n      <td id=\"T_3f362_row18_col4\" class=\"data row18 col4\" >open_refund_case</td>\n      <td id=\"T_3f362_row18_col5\" class=\"data row18 col5\" >0</td>\n      <td id=\"T_3f362_row18_col6\" class=\"data row18 col6\" >True</td>\n      <td id=\"T_3f362_row18_col7\" class=\"data row18 col7\" >True</td>\n      <td id=\"T_3f362_row18_col8\" class=\"data row18 col8\" >51</td>\n      <td id=\"T_3f362_row18_col9\" class=\"data row18 col9\" >1397</td>\n      <td id=\"T_3f362_row18_col10\" class=\"data row18 col10\" >2.32</td>\n      <td id=\"T_3f362_row18_col11\" class=\"data row18 col11\" >$0.00513</td>\n      <td id=\"T_3f362_row18_col12\" class=\"data row18 col12\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_3f362_level0_row19\" class=\"row_heading level0 row19\" >19</th>\n      <td id=\"T_3f362_row19_col0\" class=\"data row19 col0\" >Round 1: controls</td>\n      <td id=\"T_3f362_row19_col1\" class=\"data row19 col1\" >T-010</td>\n      <td id=\"T_3f362_row19_col2\" class=\"data row19 col2\" >account_access</td>\n      <td id=\"T_3f362_row19_col3\" class=\"data row19 col3\" >lookup_customer, lookup_policy, escalate_to_human</td>\n      <td id=\"T_3f362_row19_col4\" class=\"data row19 col4\" >escalate_account_security</td>\n      <td id=\"T_3f362_row19_col5\" class=\"data row19 col5\" >0</td>\n      <td id=\"T_3f362_row19_col6\" class=\"data row19 col6\" >True</td>\n      <td id=\"T_3f362_row19_col7\" class=\"data row19 col7\" >True</td>\n      <td id=\"T_3f362_row19_col8\" class=\"data row19 col8\" >39</td>\n      <td id=\"T_3f362_row19_col9\" class=\"data row19 col9\" >1382</td>\n      <td id=\"T_3f362_row19_col10\" class=\"data row19 col10\" >2.32</td>\n      <td id=\"T_3f362_row19_col11\" class=\"data row19 col11\" >$0.00544</td>\n      <td id=\"T_3f362_row19_col12\" class=\"data row19 col12\" >0.99</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "round1_detail = traces[traces[\"variant\"].isin([\"00_bad_baseline\", \"01_prompt_tool_context_controls\"])]\n",
    "display(\n",
    "    round1_detail[\n",
    "        [\n",
    "            \"variant_label\",\n",
    "            \"ticket_id\",\n",
    "            \"intent\",\n",
    "            \"tools\",\n",
    "            \"action\",\n",
    "            \"extra_tool_calls\",\n",
    "            \"policy_compliant\",\n",
    "            \"concise\",\n",
    "            \"visible_output_tokens\",\n",
    "            \"total_tokens\",\n",
    "            \"latency_s\",\n",
    "            \"cost_usd\",\n",
    "            \"quality_score\",\n",
    "        ]\n",
    "    ].style.format({\"cost_usd\": \"${:.5f}\", \"quality_score\": \"{:.2f}\", \"latency_s\": \"{:.2f}\"})\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9dbf385",
   "metadata": {},
   "source": [
    "## Optimization round 2: model selection\n",
    "\n",
    "Right-size the model to each step instead of choosing one global model. Establish a GPT-5.4 baseline for each workload, and evaluate it against the same labeled tickets, prompts, tools, structured-output schema, and quality criteria.\n",
    "\n",
    "- **Intent classification, extraction, and low-risk routing:** Use `gpt-5.4-nano` for ticket classification, entity extraction, and simple tags. Compare intent accuracy, high-risk false negatives, structured-output reliability, latency, and cost per correctly classified ticket. ([GPT-5.4 nano](https://developers.openai.com/api/docs/models/gpt-5.4-nano))\n",
    "\n",
    "- **Routine support and order workflows:** Use `gpt-5.4-mini` for order status, damaged delivery, straightforward refund-eligibility checks, and other repeatable support tasks that require policy interpretation or tool use. Evaluate resolution correctness, tool-call accuracy, policy compliance, p50/p95 latency, and cost per successfully resolved ticket. ([GPT-5.4 mini](https://developers.openai.com/api/docs/models/gpt-5.4-mini))\n",
    "\n",
    "- **Complex or high-risk cases:** Use `gpt-5.4` for account-access problems, duplicate-charge escalations, refund disputes, and other high-consequence interactions. Preserve deterministic authorization and refund checks, explicit escalation rules, and human review where required. Measure resolution quality, policy adherence, latency, and end-to-end cost. ([GPT-5.4](https://developers.openai.com/api/docs/models/gpt-5.4))\n",
    "\n",
    "The GPT-5.6 family offers newer models that correspond to these same tiers. [GPT-5.6 Luna](https://developers.openai.com/api/docs/models/gpt-5.6-luna) (`gpt-5.6-luna`) maps to the nano tier for classification and high-volume tasks. [GPT-5.6 Terra](https://developers.openai.com/api/docs/models/gpt-5.6-terra) (`gpt-5.6-terra`) maps to the mini tier for routine support workflows. [GPT-5.6 Sol](https://developers.openai.com/api/docs/models/gpt-5.6-sol) (`gpt-5.6-sol`) maps to the full-model tier for complex or high-risk cases. Each can be evaluated against its corresponding GPT-5.4 baseline using the same tickets and quality criteria.\n",
    "\n",
    "For each comparison, begin with the existing reasoning-effort setting and also evaluate one level lower. A newer model can be more economical at the task level if it resolves tickets with fewer retries, unnecessary tool calls, or escalations. Consider fine-tuning only if a selected model explicitly supports it. ([GPT-5.6 migration guidance](https://developers.openai.com/api/docs/guides/latest-model))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59f94fe8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.299343Z",
     "iopub.status.busy": "2026-04-30T18:28:05.299289Z",
     "iopub.status.idle": "2026-04-30T18:28:05.301701Z",
     "shell.execute_reply": "2026-04-30T18:28:05.301345Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "{\n  \"type\": \"json_schema\",\n  \"name\": \"support_triage\",\n  \"strict\": true,\n  \"schema\": {\n    \"type\": \"object\",\n    \"properties\": {\n      \"intent\": {\n        \"type\": \"string\",\n        \"enum\": [\n          \"order_status\",\n          \"damaged_delivery\",\n          \"refund_eligibility\",\n          \"billing_issue\",\n          \"account_access\",\n          \"refund_dispute\",\n          \"delivered_not_received\",\n          \"high_value_damage\"\n        ]\n      },\n      \"risk\": {\n        \"type\": \"string\",\n        \"enum\": [\n          \"low\",\n          \"medium\",\n          \"high\"\n        ]\n      },\n      \"needs_human\": {\n        \"type\": \"boolean\"\n      },\n      \"order_id\": {\n        \"type\": [\n          \"string\",\n          \"null\"\n        ]\n      }\n    },\n    \"required\": [\n      \"intent\",\n      \"risk\",\n      \"needs_human\",\n      \"order_id\"\n    ],\n    \"additionalProperties\": false\n  }\n}\n"
    }
   ],
   "source": [
    "from live_api import TRIAGE_SCHEMA\n",
    "\n",
    "print(json.dumps(TRIAGE_SCHEMA, indent=2))\n",
    "# Optional: from live_api import live_triage_example\n",
    "# live_triage_example(EVAL_SET[0][\"message\"], client=client)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab6bc462",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.302528Z",
     "iopub.status.busy": "2026-04-30T18:28:05.302473Z",
     "iopub.status.idle": "2026-04-30T18:28:05.306606Z",
     "shell.execute_reply": "2026-04-30T18:28:05.306264Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d1cb60>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_1317a\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_1317a_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_1317a_level0_col1\" class=\"col_heading level0 col1\" >ticket_id</th>\n      <th id=\"T_1317a_level0_col2\" class=\"col_heading level0 col2\" >intent</th>\n      <th id=\"T_1317a_level0_col3\" class=\"col_heading level0 col3\" >risk</th>\n      <th id=\"T_1317a_level0_col4\" class=\"col_heading level0 col4\" >model</th>\n      <th id=\"T_1317a_level0_col5\" class=\"col_heading level0 col5\" >routing_tokens</th>\n      <th id=\"T_1317a_level0_col6\" class=\"col_heading level0 col6\" >total_tokens</th>\n      <th id=\"T_1317a_level0_col7\" class=\"col_heading level0 col7\" >sync_cost_usd</th>\n      <th id=\"T_1317a_level0_col8\" class=\"col_heading level0 col8\" >quality_score</th>\n      <th id=\"T_1317a_level0_col9\" class=\"col_heading level0 col9\" >policy_compliant</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_1317a_level0_row0\" class=\"row_heading level0 row0\" >10</th>\n      <td id=\"T_1317a_row0_col0\" class=\"data row0 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row0_col1\" class=\"data row0 col1\" >T-001</td>\n      <td id=\"T_1317a_row0_col2\" class=\"data row0 col2\" >order_status</td>\n      <td id=\"T_1317a_row0_col3\" class=\"data row0 col3\" >low</td>\n      <td id=\"T_1317a_row0_col4\" class=\"data row0 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row0_col5\" class=\"data row0 col5\" >0</td>\n      <td id=\"T_1317a_row0_col6\" class=\"data row0 col6\" >1252</td>\n      <td id=\"T_1317a_row0_col7\" class=\"data row0 col7\" >$0.00418</td>\n      <td id=\"T_1317a_row0_col8\" class=\"data row0 col8\" >0.98</td>\n      <td id=\"T_1317a_row0_col9\" class=\"data row0 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row1\" class=\"row_heading level0 row1\" >11</th>\n      <td id=\"T_1317a_row1_col0\" class=\"data row1 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row1_col1\" class=\"data row1 col1\" >T-002</td>\n      <td id=\"T_1317a_row1_col2\" class=\"data row1 col2\" >damaged_delivery</td>\n      <td id=\"T_1317a_row1_col3\" class=\"data row1 col3\" >medium</td>\n      <td id=\"T_1317a_row1_col4\" class=\"data row1 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row1_col5\" class=\"data row1 col5\" >0</td>\n      <td id=\"T_1317a_row1_col6\" class=\"data row1 col6\" >1354</td>\n      <td id=\"T_1317a_row1_col7\" class=\"data row1 col7\" >$0.00482</td>\n      <td id=\"T_1317a_row1_col8\" class=\"data row1 col8\" >0.98</td>\n      <td id=\"T_1317a_row1_col9\" class=\"data row1 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row2\" class=\"row_heading level0 row2\" >12</th>\n      <td id=\"T_1317a_row2_col0\" class=\"data row2 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row2_col1\" class=\"data row2 col1\" >T-003</td>\n      <td id=\"T_1317a_row2_col2\" class=\"data row2 col2\" >refund_eligibility</td>\n      <td id=\"T_1317a_row2_col3\" class=\"data row2 col3\" >medium</td>\n      <td id=\"T_1317a_row2_col4\" class=\"data row2 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row2_col5\" class=\"data row2 col5\" >0</td>\n      <td id=\"T_1317a_row2_col6\" class=\"data row2 col6\" >1393</td>\n      <td id=\"T_1317a_row2_col7\" class=\"data row2 col7\" >$0.00512</td>\n      <td id=\"T_1317a_row2_col8\" class=\"data row2 col8\" >0.98</td>\n      <td id=\"T_1317a_row2_col9\" class=\"data row2 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row3\" class=\"row_heading level0 row3\" >13</th>\n      <td id=\"T_1317a_row3_col0\" class=\"data row3 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row3_col1\" class=\"data row3 col1\" >T-004</td>\n      <td id=\"T_1317a_row3_col2\" class=\"data row3 col2\" >billing_issue</td>\n      <td id=\"T_1317a_row3_col3\" class=\"data row3 col3\" >medium</td>\n      <td id=\"T_1317a_row3_col4\" class=\"data row3 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row3_col5\" class=\"data row3 col5\" >0</td>\n      <td id=\"T_1317a_row3_col6\" class=\"data row3 col6\" >1383</td>\n      <td id=\"T_1317a_row3_col7\" class=\"data row3 col7\" >$0.00512</td>\n      <td id=\"T_1317a_row3_col8\" class=\"data row3 col8\" >0.98</td>\n      <td id=\"T_1317a_row3_col9\" class=\"data row3 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row4\" class=\"row_heading level0 row4\" >14</th>\n      <td id=\"T_1317a_row4_col0\" class=\"data row4 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row4_col1\" class=\"data row4 col1\" >T-005</td>\n      <td id=\"T_1317a_row4_col2\" class=\"data row4 col2\" >account_access</td>\n      <td id=\"T_1317a_row4_col3\" class=\"data row4 col3\" >high</td>\n      <td id=\"T_1317a_row4_col4\" class=\"data row4 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row4_col5\" class=\"data row4 col5\" >0</td>\n      <td id=\"T_1317a_row4_col6\" class=\"data row4 col6\" >1378</td>\n      <td id=\"T_1317a_row4_col7\" class=\"data row4 col7\" >$0.00543</td>\n      <td id=\"T_1317a_row4_col8\" class=\"data row4 col8\" >0.99</td>\n      <td id=\"T_1317a_row4_col9\" class=\"data row4 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row5\" class=\"row_heading level0 row5\" >15</th>\n      <td id=\"T_1317a_row5_col0\" class=\"data row5 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row5_col1\" class=\"data row5 col1\" >T-006</td>\n      <td id=\"T_1317a_row5_col2\" class=\"data row5 col2\" >refund_dispute</td>\n      <td id=\"T_1317a_row5_col3\" class=\"data row5 col3\" >high</td>\n      <td id=\"T_1317a_row5_col4\" class=\"data row5 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row5_col5\" class=\"data row5 col5\" >0</td>\n      <td id=\"T_1317a_row5_col6\" class=\"data row5 col6\" >1418</td>\n      <td id=\"T_1317a_row5_col7\" class=\"data row5 col7\" >$0.00545</td>\n      <td id=\"T_1317a_row5_col8\" class=\"data row5 col8\" >0.99</td>\n      <td id=\"T_1317a_row5_col9\" class=\"data row5 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row6\" class=\"row_heading level0 row6\" >16</th>\n      <td id=\"T_1317a_row6_col0\" class=\"data row6 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row6_col1\" class=\"data row6 col1\" >T-007</td>\n      <td id=\"T_1317a_row6_col2\" class=\"data row6 col2\" >delivered_not_received</td>\n      <td id=\"T_1317a_row6_col3\" class=\"data row6 col3\" >medium</td>\n      <td id=\"T_1317a_row6_col4\" class=\"data row6 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row6_col5\" class=\"data row6 col5\" >0</td>\n      <td id=\"T_1317a_row6_col6\" class=\"data row6 col6\" >1390</td>\n      <td id=\"T_1317a_row6_col7\" class=\"data row6 col7\" >$0.00486</td>\n      <td id=\"T_1317a_row6_col8\" class=\"data row6 col8\" >0.98</td>\n      <td id=\"T_1317a_row6_col9\" class=\"data row6 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row7\" class=\"row_heading level0 row7\" >17</th>\n      <td id=\"T_1317a_row7_col0\" class=\"data row7 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row7_col1\" class=\"data row7 col1\" >T-008</td>\n      <td id=\"T_1317a_row7_col2\" class=\"data row7 col2\" >high_value_damage</td>\n      <td id=\"T_1317a_row7_col3\" class=\"data row7 col3\" >high</td>\n      <td id=\"T_1317a_row7_col4\" class=\"data row7 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row7_col5\" class=\"data row7 col5\" >0</td>\n      <td id=\"T_1317a_row7_col6\" class=\"data row7 col6\" >1445</td>\n      <td id=\"T_1317a_row7_col7\" class=\"data row7 col7\" >$0.00568</td>\n      <td id=\"T_1317a_row7_col8\" class=\"data row7 col8\" >0.99</td>\n      <td id=\"T_1317a_row7_col9\" class=\"data row7 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row8\" class=\"row_heading level0 row8\" >18</th>\n      <td id=\"T_1317a_row8_col0\" class=\"data row8 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row8_col1\" class=\"data row8 col1\" >T-009</td>\n      <td id=\"T_1317a_row8_col2\" class=\"data row8 col2\" >refund_eligibility</td>\n      <td id=\"T_1317a_row8_col3\" class=\"data row8 col3\" >low</td>\n      <td id=\"T_1317a_row8_col4\" class=\"data row8 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row8_col5\" class=\"data row8 col5\" >0</td>\n      <td id=\"T_1317a_row8_col6\" class=\"data row8 col6\" >1397</td>\n      <td id=\"T_1317a_row8_col7\" class=\"data row8 col7\" >$0.00513</td>\n      <td id=\"T_1317a_row8_col8\" class=\"data row8 col8\" >0.98</td>\n      <td id=\"T_1317a_row8_col9\" class=\"data row8 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row9\" class=\"row_heading level0 row9\" >19</th>\n      <td id=\"T_1317a_row9_col0\" class=\"data row9 col0\" >Round 1: controls</td>\n      <td id=\"T_1317a_row9_col1\" class=\"data row9 col1\" >T-010</td>\n      <td id=\"T_1317a_row9_col2\" class=\"data row9 col2\" >account_access</td>\n      <td id=\"T_1317a_row9_col3\" class=\"data row9 col3\" >high</td>\n      <td id=\"T_1317a_row9_col4\" class=\"data row9 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row9_col5\" class=\"data row9 col5\" >0</td>\n      <td id=\"T_1317a_row9_col6\" class=\"data row9 col6\" >1382</td>\n      <td id=\"T_1317a_row9_col7\" class=\"data row9 col7\" >$0.00544</td>\n      <td id=\"T_1317a_row9_col8\" class=\"data row9 col8\" >0.99</td>\n      <td id=\"T_1317a_row9_col9\" class=\"data row9 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row10\" class=\"row_heading level0 row10\" >20</th>\n      <td id=\"T_1317a_row10_col0\" class=\"data row10 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row10_col1\" class=\"data row10 col1\" >T-001</td>\n      <td id=\"T_1317a_row10_col2\" class=\"data row10 col2\" >order_status</td>\n      <td id=\"T_1317a_row10_col3\" class=\"data row10 col3\" >low</td>\n      <td id=\"T_1317a_row10_col4\" class=\"data row10 col4\" >gpt-5.4-mini</td>\n      <td id=\"T_1317a_row10_col5\" class=\"data row10 col5\" >208</td>\n      <td id=\"T_1317a_row10_col6\" class=\"data row10 col6\" >1353</td>\n      <td id=\"T_1317a_row10_col7\" class=\"data row10 col7\" >$0.00123</td>\n      <td id=\"T_1317a_row10_col8\" class=\"data row10 col8\" >0.98</td>\n      <td id=\"T_1317a_row10_col9\" class=\"data row10 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row11\" class=\"row_heading level0 row11\" >21</th>\n      <td id=\"T_1317a_row11_col0\" class=\"data row11 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row11_col1\" class=\"data row11 col1\" >T-002</td>\n      <td id=\"T_1317a_row11_col2\" class=\"data row11 col2\" >damaged_delivery</td>\n      <td id=\"T_1317a_row11_col3\" class=\"data row11 col3\" >medium</td>\n      <td id=\"T_1317a_row11_col4\" class=\"data row11 col4\" >gpt-5.4-mini</td>\n      <td id=\"T_1317a_row11_col5\" class=\"data row11 col5\" >208</td>\n      <td id=\"T_1317a_row11_col6\" class=\"data row11 col6\" >1451</td>\n      <td id=\"T_1317a_row11_col7\" class=\"data row11 col7\" >$0.00141</td>\n      <td id=\"T_1317a_row11_col8\" class=\"data row11 col8\" >0.98</td>\n      <td id=\"T_1317a_row11_col9\" class=\"data row11 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row12\" class=\"row_heading level0 row12\" >22</th>\n      <td id=\"T_1317a_row12_col0\" class=\"data row12 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row12_col1\" class=\"data row12 col1\" >T-003</td>\n      <td id=\"T_1317a_row12_col2\" class=\"data row12 col2\" >refund_eligibility</td>\n      <td id=\"T_1317a_row12_col3\" class=\"data row12 col3\" >medium</td>\n      <td id=\"T_1317a_row12_col4\" class=\"data row12 col4\" >gpt-5.4-mini</td>\n      <td id=\"T_1317a_row12_col5\" class=\"data row12 col5\" >210</td>\n      <td id=\"T_1317a_row12_col6\" class=\"data row12 col6\" >1490</td>\n      <td id=\"T_1317a_row12_col7\" class=\"data row12 col7\" >$0.00149</td>\n      <td id=\"T_1317a_row12_col8\" class=\"data row12 col8\" >0.98</td>\n      <td id=\"T_1317a_row12_col9\" class=\"data row12 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row13\" class=\"row_heading level0 row13\" >23</th>\n      <td id=\"T_1317a_row13_col0\" class=\"data row13 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row13_col1\" class=\"data row13 col1\" >T-004</td>\n      <td id=\"T_1317a_row13_col2\" class=\"data row13 col2\" >billing_issue</td>\n      <td id=\"T_1317a_row13_col3\" class=\"data row13 col3\" >medium</td>\n      <td id=\"T_1317a_row13_col4\" class=\"data row13 col4\" >gpt-5.4-mini</td>\n      <td id=\"T_1317a_row13_col5\" class=\"data row13 col5\" >207</td>\n      <td id=\"T_1317a_row13_col6\" class=\"data row13 col6\" >1474</td>\n      <td id=\"T_1317a_row13_col7\" class=\"data row13 col7\" >$0.00147</td>\n      <td id=\"T_1317a_row13_col8\" class=\"data row13 col8\" >0.98</td>\n      <td id=\"T_1317a_row13_col9\" class=\"data row13 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row14\" class=\"row_heading level0 row14\" >24</th>\n      <td id=\"T_1317a_row14_col0\" class=\"data row14 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row14_col1\" class=\"data row14 col1\" >T-005</td>\n      <td id=\"T_1317a_row14_col2\" class=\"data row14 col2\" >account_access</td>\n      <td id=\"T_1317a_row14_col3\" class=\"data row14 col3\" >high</td>\n      <td id=\"T_1317a_row14_col4\" class=\"data row14 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row14_col5\" class=\"data row14 col5\" >206</td>\n      <td id=\"T_1317a_row14_col6\" class=\"data row14 col6\" >1490</td>\n      <td id=\"T_1317a_row14_col7\" class=\"data row14 col7\" >$0.00536</td>\n      <td id=\"T_1317a_row14_col8\" class=\"data row14 col8\" >0.99</td>\n      <td id=\"T_1317a_row14_col9\" class=\"data row14 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row15\" class=\"row_heading level0 row15\" >25</th>\n      <td id=\"T_1317a_row15_col0\" class=\"data row15 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row15_col1\" class=\"data row15 col1\" >T-006</td>\n      <td id=\"T_1317a_row15_col2\" class=\"data row15 col2\" >refund_dispute</td>\n      <td id=\"T_1317a_row15_col3\" class=\"data row15 col3\" >high</td>\n      <td id=\"T_1317a_row15_col4\" class=\"data row15 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row15_col5\" class=\"data row15 col5\" >212</td>\n      <td id=\"T_1317a_row15_col6\" class=\"data row15 col6\" >1536</td>\n      <td id=\"T_1317a_row15_col7\" class=\"data row15 col7\" >$0.00537</td>\n      <td id=\"T_1317a_row15_col8\" class=\"data row15 col8\" >0.99</td>\n      <td id=\"T_1317a_row15_col9\" class=\"data row15 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row16\" class=\"row_heading level0 row16\" >26</th>\n      <td id=\"T_1317a_row16_col0\" class=\"data row16 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row16_col1\" class=\"data row16 col1\" >T-007</td>\n      <td id=\"T_1317a_row16_col2\" class=\"data row16 col2\" >delivered_not_received</td>\n      <td id=\"T_1317a_row16_col3\" class=\"data row16 col3\" >medium</td>\n      <td id=\"T_1317a_row16_col4\" class=\"data row16 col4\" >gpt-5.4-mini</td>\n      <td id=\"T_1317a_row16_col5\" class=\"data row16 col5\" >213</td>\n      <td id=\"T_1317a_row16_col6\" class=\"data row16 col6\" >1492</td>\n      <td id=\"T_1317a_row16_col7\" class=\"data row16 col7\" >$0.00142</td>\n      <td id=\"T_1317a_row16_col8\" class=\"data row16 col8\" >0.98</td>\n      <td id=\"T_1317a_row16_col9\" class=\"data row16 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row17\" class=\"row_heading level0 row17\" >27</th>\n      <td id=\"T_1317a_row17_col0\" class=\"data row17 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row17_col1\" class=\"data row17 col1\" >T-008</td>\n      <td id=\"T_1317a_row17_col2\" class=\"data row17 col2\" >high_value_damage</td>\n      <td id=\"T_1317a_row17_col3\" class=\"data row17 col3\" >high</td>\n      <td id=\"T_1317a_row17_col4\" class=\"data row17 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row17_col5\" class=\"data row17 col5\" >218</td>\n      <td id=\"T_1317a_row17_col6\" class=\"data row17 col6\" >1570</td>\n      <td id=\"T_1317a_row17_col7\" class=\"data row17 col7\" >$0.00562</td>\n      <td id=\"T_1317a_row17_col8\" class=\"data row17 col8\" >0.99</td>\n      <td id=\"T_1317a_row17_col9\" class=\"data row17 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row18\" class=\"row_heading level0 row18\" >28</th>\n      <td id=\"T_1317a_row18_col0\" class=\"data row18 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row18_col1\" class=\"data row18 col1\" >T-009</td>\n      <td id=\"T_1317a_row18_col2\" class=\"data row18 col2\" >refund_eligibility</td>\n      <td id=\"T_1317a_row18_col3\" class=\"data row18 col3\" >low</td>\n      <td id=\"T_1317a_row18_col4\" class=\"data row18 col4\" >gpt-5.4-mini</td>\n      <td id=\"T_1317a_row18_col5\" class=\"data row18 col5\" >214</td>\n      <td id=\"T_1317a_row18_col6\" class=\"data row18 col6\" >1498</td>\n      <td id=\"T_1317a_row18_col7\" class=\"data row18 col7\" >$0.00149</td>\n      <td id=\"T_1317a_row18_col8\" class=\"data row18 col8\" >0.98</td>\n      <td id=\"T_1317a_row18_col9\" class=\"data row18 col9\" >True</td>\n    </tr>\n    <tr>\n      <th id=\"T_1317a_level0_row19\" class=\"row_heading level0 row19\" >29</th>\n      <td id=\"T_1317a_row19_col0\" class=\"data row19 col0\" >Round 2: routing</td>\n      <td id=\"T_1317a_row19_col1\" class=\"data row19 col1\" >T-010</td>\n      <td id=\"T_1317a_row19_col2\" class=\"data row19 col2\" >account_access</td>\n      <td id=\"T_1317a_row19_col3\" class=\"data row19 col3\" >high</td>\n      <td id=\"T_1317a_row19_col4\" class=\"data row19 col4\" >gpt-5.4</td>\n      <td id=\"T_1317a_row19_col5\" class=\"data row19 col5\" >209</td>\n      <td id=\"T_1317a_row19_col6\" class=\"data row19 col6\" >1497</td>\n      <td id=\"T_1317a_row19_col7\" class=\"data row19 col7\" >$0.00537</td>\n      <td id=\"T_1317a_row19_col8\" class=\"data row19 col8\" >0.99</td>\n      <td id=\"T_1317a_row19_col9\" class=\"data row19 col9\" >True</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "model_routing_view = traces[traces[\"variant\"].isin([\"01_prompt_tool_context_controls\", \"02_model_routing\"])]\n",
    "display(\n",
    "    model_routing_view[\n",
    "        [\n",
    "            \"variant_label\",\n",
    "            \"ticket_id\",\n",
    "            \"intent\",\n",
    "            \"risk\",\n",
    "            \"model\",\n",
    "            \"routing_tokens\",\n",
    "            \"total_tokens\",\n",
    "            \"sync_cost_usd\",\n",
    "            \"quality_score\",\n",
    "            \"policy_compliant\",\n",
    "        ]\n",
    "    ].style.format({\"sync_cost_usd\": \"${:.5f}\", \"quality_score\": \"{:.2f}\"})\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6ee98b8",
   "metadata": {},
   "source": [
    "## Optimization round 3: prompt caching\n",
    "\n",
    "Every support request includes the same core instructions, policy rules, tool definitions, and response schema. Prompt caching lets the API reuse that shared context across tickets, reducing repeated processing and lowering input-token costs. Customer-specific details, such as order IDs, account information, and retrieved records, should appear after the shared prefix.\n",
    "\n",
    "Prompt caching has evolved between model generations. With `gpt-5.4-mini`, the API automatically identifies repeated prefixes and can reuse the shared support context even when the customer-specific details change. Writing a new prefix does not add a separate cache-write charge. Keep the tool definitions consistent and use `tool_choice.allowed_tools` to control which tools are available without changing the shared tool list.\n",
    "\n",
    "GPT-5.6 introduces two changes: cache writes are billed, and developers can explicitly choose which part of the prompt should be cached. With `gpt-5.6-luna`, `gpt-5.6-terra`, or `gpt-5.6-sol`, the default cache breakpoint is placed after the latest message. If that message changes between tickets, the longest cached prefix may not match. Implicit mode can still reuse earlier eligible message endings, including the initial developer-message block. Because writing content to cache costs 1.25 times the normal input-token price, repeatedly caching those unique messages can increase cost without creating useful reuse.\n",
    "\n",
    "For example, two order-status tickets can share the same support instructions, policy rules, and tools, even though one asks about order `O-1001` and the other asks about order `O-2002`. For GPT-5.6, put the shared playbook in a developer-message `input_text` block and mark its end with `prompt_cache_breakpoint={\"mode\": \"explicit\"}` before the order-specific details. Top-level `instructions` cannot contain a breakpoint. Set `prompt_cache_options` to explicit mode with `ttl=\"30m\"`, and use the same `prompt_cache_key`, such as `support_order_status_v1`, for both requests. With an eligible matching prefix, the first ticket writes the playbook and later tickets can reuse it at the cached-input rate while processing their own order details normally.\n",
    "\n",
    "Compare `cached_tokens` and `cache_write_tokens` alongside latency and cost per resolved ticket. For additional implementation details, see the [prompt caching guide](https://developers.openai.com/api/docs/guides/prompt-caching).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a47d606d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.307551Z",
     "iopub.status.busy": "2026-04-30T18:28:05.307497Z",
     "iopub.status.idle": "2026-04-30T18:28:05.309408Z",
     "shell.execute_reply": "2026-04-30T18:28:05.309160Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "{\n  \"model\": \"gpt-5.4-mini\",\n  \"instructions\": \"Role: E-commerce support assistant.\\nConstraints: Be concise, policy-compliant, and explicit about next steps. Do not disclose internal data.\\nEscalate: duplicate charges, account access without verification, high-value disputes, and refunds outside the window.\\nOutput shape: customer_message, resolution_type, escalate, internal_tags.\\nTool contract: tool definitions are stable across requests; restrict callable tools with tool_choice.allowed_tools.\\nVersion: support-agent-optimization-v1.\\n\\nStable support playbook digest:\\n- Shipping delays: provide status, ETA, and tracking next steps; do not refund solely for short carrier delays.\\n- Delivered-not-received: verify delivery details, ask the customer to check common locations, and start carrier trace steps when appropriate.\\n- Damaged delivery: request photo evidence before offering replacement or refund; high-value damaged items require human review.\\n- Refunds: standard returnable items are eligible within 30 days; outside-window or high-value disputes require human review.\\n- Billing: duplicate-charge reports require billing review; acknowledge and escalate, but do not promise a completed refund.\\n- Account access: when identity is not verified, escalate to account security; do not change credentials or contact information in chat.\\n- Customer messages must be concise, policy-compliant, and explicit about next steps.\\n- Internal notes, raw carrier payloads, CRM audit logs, and policy appendices must never be exposed to the customer.\\nStable support playbook digest:\\n- Shipping delays: provide status, ETA, and tracking next steps; do not refund solely for short carrier delays.\\n- Delivered-not-received: verify delivery details, ask the customer to check common locations, and start carrier trace steps when appropriate.\\n- Damaged delivery: request photo evidence before offering replacement or refund; high-value damaged items require human review.\\n- Refunds: standard returnable items are eligible within 30 days; outside-window or high-value disputes require human review.\\n- Billing: duplicate-charge reports require billing review; acknowledge and escalate, but do not promise a completed refund.\\n- Account access: when identity is not verified, escalate to account security; do not change credentials or contact information in chat.\\n- Customer messages must be\n...\n"
    }
   ],
   "source": [
    "cache_friendly_request = {\n",
    "    \"model\": \"gpt-5.4-mini\",\n",
    "    \"instructions\": CACHE_FRIENDLY_PROMPT,\n",
    "    \"tools\": SLIM_TOOLS,\n",
    "    \"tool_choice\": allowed_tool_choice([\"lookup_order\"], mode=\"auto\"),\n",
    "    \"prompt_cache_key\": \"support_order_status_v1\",\n",
    "    \"reasoning\": {\"effort\": \"low\"},\n",
    "    \"text\": {\"verbosity\": \"low\"},\n",
    "    \"max_output_tokens\": 300,\n",
    "    \"input\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": json.dumps(\n",
    "                {\n",
    "                    \"ticket_id\": \"T-001\",\n",
    "                    \"customer_id\": \"C-100\",\n",
    "                    \"message\": \"Where is order O-1001?\",\n",
    "                    \"order_id\": \"O-1001\",\n",
    "                }\n",
    "            ),\n",
    "        }\n",
    "    ],\n",
    "}\n",
    "\n",
    "print(json.dumps(cache_friendly_request, indent=2)[:2400] + \"\\n...\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fccd3e8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.310188Z",
     "iopub.status.busy": "2026-04-30T18:28:05.310139Z",
     "iopub.status.idle": "2026-04-30T18:28:05.311794Z",
     "shell.execute_reply": "2026-04-30T18:28:05.311395Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\nfrom support import STABLE_SUPPORT_PREFIX\n\nfirst = client.responses.create(\n    model=\"gpt-5.4-mini\",\n    instructions=STABLE_SUPPORT_PREFIX,\n    tools=SLIM_TOOLS,\n    input=\"Customer asks: Where is order O-1001?\",\n    prompt_cache_key=\"support_order_status_v1\",\n)\n\nfollow_up = client.responses.create(\n    model=\"gpt-5.4-mini\",\n    previous_response_id=first.id,\n    instructions=STABLE_SUPPORT_PREFIX,\n    input=\"Customer follow-up: the carrier link is stale. What should I do?\",\n    prompt_cache_key=\"support_order_status_v1\",\n)\n\n"
    }
   ],
   "source": [
    "previous_response_id_example = '''\n",
    "from support import STABLE_SUPPORT_PREFIX\n",
    "\n",
    "first = client.responses.create(\n",
    "    model=\"gpt-5.4-mini\",\n",
    "    instructions=STABLE_SUPPORT_PREFIX,\n",
    "    tools=SLIM_TOOLS,\n",
    "    input=\"Customer asks: Where is order O-1001?\",\n",
    "    prompt_cache_key=\"support_order_status_v1\",\n",
    ")\n",
    "\n",
    "follow_up = client.responses.create(\n",
    "    model=\"gpt-5.4-mini\",\n",
    "    previous_response_id=first.id,\n",
    "    instructions=STABLE_SUPPORT_PREFIX,\n",
    "    input=\"Customer follow-up: the carrier link is stale. What should I do?\",\n",
    "    prompt_cache_key=\"support_order_status_v1\",\n",
    ")\n",
    "'''\n",
    "\n",
    "print(previous_response_id_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ea1590e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.312592Z",
     "iopub.status.busy": "2026-04-30T18:28:05.312542Z",
     "iopub.status.idle": "2026-04-30T18:28:05.316909Z",
     "shell.execute_reply": "2026-04-30T18:28:05.316619Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d1f320>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_e68b0\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_e68b0_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_e68b0_level0_col1\" class=\"col_heading level0 col1\" >ticket_id</th>\n      <th id=\"T_e68b0_level0_col2\" class=\"col_heading level0 col2\" >model</th>\n      <th id=\"T_e68b0_level0_col3\" class=\"col_heading level0 col3\" >input_tokens</th>\n      <th id=\"T_e68b0_level0_col4\" class=\"col_heading level0 col4\" >cacheable_prefix_tokens</th>\n      <th id=\"T_e68b0_level0_col5\" class=\"col_heading level0 col5\" >cached_tokens</th>\n      <th id=\"T_e68b0_level0_col6\" class=\"col_heading level0 col6\" >latency_input_tokens</th>\n      <th id=\"T_e68b0_level0_col7\" class=\"col_heading level0 col7\" >output_tokens</th>\n      <th id=\"T_e68b0_level0_col8\" class=\"col_heading level0 col8\" >cost_usd</th>\n      <th id=\"T_e68b0_level0_col9\" class=\"col_heading level0 col9\" >latency_s</th>\n      <th id=\"T_e68b0_level0_col10\" class=\"col_heading level0 col10\" >quality_score</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_e68b0_level0_row0\" class=\"row_heading level0 row0\" >20</th>\n      <td id=\"T_e68b0_row0_col0\" class=\"data row0 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row0_col1\" class=\"data row0 col1\" >T-001</td>\n      <td id=\"T_e68b0_row0_col2\" class=\"data row0 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row0_col3\" class=\"data row0 col3\" >1068</td>\n      <td id=\"T_e68b0_row0_col4\" class=\"data row0 col4\" >0</td>\n      <td id=\"T_e68b0_row0_col5\" class=\"data row0 col5\" >0</td>\n      <td id=\"T_e68b0_row0_col6\" class=\"data row0 col6\" >1068</td>\n      <td id=\"T_e68b0_row0_col7\" class=\"data row0 col7\" >77</td>\n      <td id=\"T_e68b0_row0_col8\" class=\"data row0 col8\" >$0.00123</td>\n      <td id=\"T_e68b0_row0_col9\" class=\"data row0 col9\" >1.56</td>\n      <td id=\"T_e68b0_row0_col10\" class=\"data row0 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row1\" class=\"row_heading level0 row1\" >21</th>\n      <td id=\"T_e68b0_row1_col0\" class=\"data row1 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row1_col1\" class=\"data row1 col1\" >T-002</td>\n      <td id=\"T_e68b0_row1_col2\" class=\"data row1 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row1_col3\" class=\"data row1 col3\" >1139</td>\n      <td id=\"T_e68b0_row1_col4\" class=\"data row1 col4\" >0</td>\n      <td id=\"T_e68b0_row1_col5\" class=\"data row1 col5\" >0</td>\n      <td id=\"T_e68b0_row1_col6\" class=\"data row1 col6\" >1139</td>\n      <td id=\"T_e68b0_row1_col7\" class=\"data row1 col7\" >104</td>\n      <td id=\"T_e68b0_row1_col8\" class=\"data row1 col8\" >$0.00141</td>\n      <td id=\"T_e68b0_row1_col9\" class=\"data row1 col9\" >1.72</td>\n      <td id=\"T_e68b0_row1_col10\" class=\"data row1 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row2\" class=\"row_heading level0 row2\" >22</th>\n      <td id=\"T_e68b0_row2_col0\" class=\"data row2 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row2_col1\" class=\"data row2 col1\" >T-003</td>\n      <td id=\"T_e68b0_row2_col2\" class=\"data row2 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row2_col3\" class=\"data row2 col3\" >1162</td>\n      <td id=\"T_e68b0_row2_col4\" class=\"data row2 col4\" >0</td>\n      <td id=\"T_e68b0_row2_col5\" class=\"data row2 col5\" >0</td>\n      <td id=\"T_e68b0_row2_col6\" class=\"data row2 col6\" >1162</td>\n      <td id=\"T_e68b0_row2_col7\" class=\"data row2 col7\" >118</td>\n      <td id=\"T_e68b0_row2_col8\" class=\"data row2 col8\" >$0.00149</td>\n      <td id=\"T_e68b0_row2_col9\" class=\"data row2 col9\" >1.87</td>\n      <td id=\"T_e68b0_row2_col10\" class=\"data row2 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row3\" class=\"row_heading level0 row3\" >23</th>\n      <td id=\"T_e68b0_row3_col0\" class=\"data row3 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row3_col1\" class=\"data row3 col1\" >T-004</td>\n      <td id=\"T_e68b0_row3_col2\" class=\"data row3 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row3_col3\" class=\"data row3 col3\" >1150</td>\n      <td id=\"T_e68b0_row3_col4\" class=\"data row3 col4\" >0</td>\n      <td id=\"T_e68b0_row3_col5\" class=\"data row3 col5\" >0</td>\n      <td id=\"T_e68b0_row3_col6\" class=\"data row3 col6\" >1150</td>\n      <td id=\"T_e68b0_row3_col7\" class=\"data row3 col7\" >117</td>\n      <td id=\"T_e68b0_row3_col8\" class=\"data row3 col8\" >$0.00147</td>\n      <td id=\"T_e68b0_row3_col9\" class=\"data row3 col9\" >1.87</td>\n      <td id=\"T_e68b0_row3_col10\" class=\"data row3 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row4\" class=\"row_heading level0 row4\" >24</th>\n      <td id=\"T_e68b0_row4_col0\" class=\"data row4 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row4_col1\" class=\"data row4 col1\" >T-005</td>\n      <td id=\"T_e68b0_row4_col2\" class=\"data row4 col2\" >gpt-5.4</td>\n      <td id=\"T_e68b0_row4_col3\" class=\"data row4 col3\" >1119</td>\n      <td id=\"T_e68b0_row4_col4\" class=\"data row4 col4\" >0</td>\n      <td id=\"T_e68b0_row4_col5\" class=\"data row4 col5\" >0</td>\n      <td id=\"T_e68b0_row4_col6\" class=\"data row4 col6\" >1119</td>\n      <td id=\"T_e68b0_row4_col7\" class=\"data row4 col7\" >165</td>\n      <td id=\"T_e68b0_row4_col8\" class=\"data row4 col8\" >$0.00536</td>\n      <td id=\"T_e68b0_row4_col9\" class=\"data row4 col9\" >2.48</td>\n      <td id=\"T_e68b0_row4_col10\" class=\"data row4 col10\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row5\" class=\"row_heading level0 row5\" >25</th>\n      <td id=\"T_e68b0_row5_col0\" class=\"data row5 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row5_col1\" class=\"data row5 col1\" >T-006</td>\n      <td id=\"T_e68b0_row5_col2\" class=\"data row5 col2\" >gpt-5.4</td>\n      <td id=\"T_e68b0_row5_col3\" class=\"data row5 col3\" >1166</td>\n      <td id=\"T_e68b0_row5_col4\" class=\"data row5 col4\" >0</td>\n      <td id=\"T_e68b0_row5_col5\" class=\"data row5 col5\" >0</td>\n      <td id=\"T_e68b0_row5_col6\" class=\"data row5 col6\" >1166</td>\n      <td id=\"T_e68b0_row5_col7\" class=\"data row5 col7\" >158</td>\n      <td id=\"T_e68b0_row5_col8\" class=\"data row5 col8\" >$0.00537</td>\n      <td id=\"T_e68b0_row5_col9\" class=\"data row5 col9\" >2.49</td>\n      <td id=\"T_e68b0_row5_col10\" class=\"data row5 col10\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row6\" class=\"row_heading level0 row6\" >26</th>\n      <td id=\"T_e68b0_row6_col0\" class=\"data row6 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row6_col1\" class=\"data row6 col1\" >T-007</td>\n      <td id=\"T_e68b0_row6_col2\" class=\"data row6 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row6_col3\" class=\"data row6 col3\" >1179</td>\n      <td id=\"T_e68b0_row6_col4\" class=\"data row6 col4\" >0</td>\n      <td id=\"T_e68b0_row6_col5\" class=\"data row6 col5\" >0</td>\n      <td id=\"T_e68b0_row6_col6\" class=\"data row6 col6\" >1179</td>\n      <td id=\"T_e68b0_row6_col7\" class=\"data row6 col7\" >100</td>\n      <td id=\"T_e68b0_row6_col8\" class=\"data row6 col8\" >$0.00142</td>\n      <td id=\"T_e68b0_row6_col9\" class=\"data row6 col9\" >1.73</td>\n      <td id=\"T_e68b0_row6_col10\" class=\"data row6 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row7\" class=\"row_heading level0 row7\" >27</th>\n      <td id=\"T_e68b0_row7_col0\" class=\"data row7 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row7_col1\" class=\"data row7 col1\" >T-008</td>\n      <td id=\"T_e68b0_row7_col2\" class=\"data row7 col2\" >gpt-5.4</td>\n      <td id=\"T_e68b0_row7_col3\" class=\"data row7 col3\" >1180</td>\n      <td id=\"T_e68b0_row7_col4\" class=\"data row7 col4\" >0</td>\n      <td id=\"T_e68b0_row7_col5\" class=\"data row7 col5\" >0</td>\n      <td id=\"T_e68b0_row7_col6\" class=\"data row7 col6\" >1180</td>\n      <td id=\"T_e68b0_row7_col7\" class=\"data row7 col7\" >172</td>\n      <td id=\"T_e68b0_row7_col8\" class=\"data row7 col8\" >$0.00562</td>\n      <td id=\"T_e68b0_row7_col9\" class=\"data row7 col9\" >2.49</td>\n      <td id=\"T_e68b0_row7_col10\" class=\"data row7 col10\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row8\" class=\"row_heading level0 row8\" >28</th>\n      <td id=\"T_e68b0_row8_col0\" class=\"data row8 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row8_col1\" class=\"data row8 col1\" >T-009</td>\n      <td id=\"T_e68b0_row8_col2\" class=\"data row8 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row8_col3\" class=\"data row8 col3\" >1166</td>\n      <td id=\"T_e68b0_row8_col4\" class=\"data row8 col4\" >0</td>\n      <td id=\"T_e68b0_row8_col5\" class=\"data row8 col5\" >0</td>\n      <td id=\"T_e68b0_row8_col6\" class=\"data row8 col6\" >1166</td>\n      <td id=\"T_e68b0_row8_col7\" class=\"data row8 col7\" >118</td>\n      <td id=\"T_e68b0_row8_col8\" class=\"data row8 col8\" >$0.00149</td>\n      <td id=\"T_e68b0_row8_col9\" class=\"data row8 col9\" >1.87</td>\n      <td id=\"T_e68b0_row8_col10\" class=\"data row8 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row9\" class=\"row_heading level0 row9\" >29</th>\n      <td id=\"T_e68b0_row9_col0\" class=\"data row9 col0\" >Round 2: routing</td>\n      <td id=\"T_e68b0_row9_col1\" class=\"data row9 col1\" >T-010</td>\n      <td id=\"T_e68b0_row9_col2\" class=\"data row9 col2\" >gpt-5.4</td>\n      <td id=\"T_e68b0_row9_col3\" class=\"data row9 col3\" >1123</td>\n      <td id=\"T_e68b0_row9_col4\" class=\"data row9 col4\" >0</td>\n      <td id=\"T_e68b0_row9_col5\" class=\"data row9 col5\" >0</td>\n      <td id=\"T_e68b0_row9_col6\" class=\"data row9 col6\" >1123</td>\n      <td id=\"T_e68b0_row9_col7\" class=\"data row9 col7\" >165</td>\n      <td id=\"T_e68b0_row9_col8\" class=\"data row9 col8\" >$0.00537</td>\n      <td id=\"T_e68b0_row9_col9\" class=\"data row9 col9\" >2.48</td>\n      <td id=\"T_e68b0_row9_col10\" class=\"data row9 col10\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row10\" class=\"row_heading level0 row10\" >30</th>\n      <td id=\"T_e68b0_row10_col0\" class=\"data row10 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row10_col1\" class=\"data row10 col1\" >T-001</td>\n      <td id=\"T_e68b0_row10_col2\" class=\"data row10 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row10_col3\" class=\"data row10 col3\" >2267</td>\n      <td id=\"T_e68b0_row10_col4\" class=\"data row10 col4\" >1779</td>\n      <td id=\"T_e68b0_row10_col5\" class=\"data row10 col5\" >1779</td>\n      <td id=\"T_e68b0_row10_col6\" class=\"data row10 col6\" >933</td>\n      <td id=\"T_e68b0_row10_col7\" class=\"data row10 col7\" >77</td>\n      <td id=\"T_e68b0_row10_col8\" class=\"data row10 col8\" >$0.00093</td>\n      <td id=\"T_e68b0_row10_col9\" class=\"data row10 col9\" >1.54</td>\n      <td id=\"T_e68b0_row10_col10\" class=\"data row10 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row11\" class=\"row_heading level0 row11\" >31</th>\n      <td id=\"T_e68b0_row11_col0\" class=\"data row11 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row11_col1\" class=\"data row11 col1\" >T-002</td>\n      <td id=\"T_e68b0_row11_col2\" class=\"data row11 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row11_col3\" class=\"data row11 col3\" >2338</td>\n      <td id=\"T_e68b0_row11_col4\" class=\"data row11 col4\" >1779</td>\n      <td id=\"T_e68b0_row11_col5\" class=\"data row11 col5\" >1779</td>\n      <td id=\"T_e68b0_row11_col6\" class=\"data row11 col6\" >1004</td>\n      <td id=\"T_e68b0_row11_col7\" class=\"data row11 col7\" >104</td>\n      <td id=\"T_e68b0_row11_col8\" class=\"data row11 col8\" >$0.00111</td>\n      <td id=\"T_e68b0_row11_col9\" class=\"data row11 col9\" >1.70</td>\n      <td id=\"T_e68b0_row11_col10\" class=\"data row11 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row12\" class=\"row_heading level0 row12\" >32</th>\n      <td id=\"T_e68b0_row12_col0\" class=\"data row12 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row12_col1\" class=\"data row12 col1\" >T-003</td>\n      <td id=\"T_e68b0_row12_col2\" class=\"data row12 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row12_col3\" class=\"data row12 col3\" >2361</td>\n      <td id=\"T_e68b0_row12_col4\" class=\"data row12 col4\" >1779</td>\n      <td id=\"T_e68b0_row12_col5\" class=\"data row12 col5\" >1779</td>\n      <td id=\"T_e68b0_row12_col6\" class=\"data row12 col6\" >1027</td>\n      <td id=\"T_e68b0_row12_col7\" class=\"data row12 col7\" >118</td>\n      <td id=\"T_e68b0_row12_col8\" class=\"data row12 col8\" >$0.00119</td>\n      <td id=\"T_e68b0_row12_col9\" class=\"data row12 col9\" >1.85</td>\n      <td id=\"T_e68b0_row12_col10\" class=\"data row12 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row13\" class=\"row_heading level0 row13\" >33</th>\n      <td id=\"T_e68b0_row13_col0\" class=\"data row13 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row13_col1\" class=\"data row13 col1\" >T-004</td>\n      <td id=\"T_e68b0_row13_col2\" class=\"data row13 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row13_col3\" class=\"data row13 col3\" >2349</td>\n      <td id=\"T_e68b0_row13_col4\" class=\"data row13 col4\" >1779</td>\n      <td id=\"T_e68b0_row13_col5\" class=\"data row13 col5\" >1779</td>\n      <td id=\"T_e68b0_row13_col6\" class=\"data row13 col6\" >1015</td>\n      <td id=\"T_e68b0_row13_col7\" class=\"data row13 col7\" >117</td>\n      <td id=\"T_e68b0_row13_col8\" class=\"data row13 col8\" >$0.00117</td>\n      <td id=\"T_e68b0_row13_col9\" class=\"data row13 col9\" >1.84</td>\n      <td id=\"T_e68b0_row13_col10\" class=\"data row13 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row14\" class=\"row_heading level0 row14\" >34</th>\n      <td id=\"T_e68b0_row14_col0\" class=\"data row14 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row14_col1\" class=\"data row14 col1\" >T-005</td>\n      <td id=\"T_e68b0_row14_col2\" class=\"data row14 col2\" >gpt-5.4</td>\n      <td id=\"T_e68b0_row14_col3\" class=\"data row14 col3\" >2318</td>\n      <td id=\"T_e68b0_row14_col4\" class=\"data row14 col4\" >1779</td>\n      <td id=\"T_e68b0_row14_col5\" class=\"data row14 col5\" >1779</td>\n      <td id=\"T_e68b0_row14_col6\" class=\"data row14 col6\" >984</td>\n      <td id=\"T_e68b0_row14_col7\" class=\"data row14 col7\" >165</td>\n      <td id=\"T_e68b0_row14_col8\" class=\"data row14 col8\" >$0.00435</td>\n      <td id=\"T_e68b0_row14_col9\" class=\"data row14 col9\" >2.45</td>\n      <td id=\"T_e68b0_row14_col10\" class=\"data row14 col10\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row15\" class=\"row_heading level0 row15\" >35</th>\n      <td id=\"T_e68b0_row15_col0\" class=\"data row15 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row15_col1\" class=\"data row15 col1\" >T-006</td>\n      <td id=\"T_e68b0_row15_col2\" class=\"data row15 col2\" >gpt-5.4</td>\n      <td id=\"T_e68b0_row15_col3\" class=\"data row15 col3\" >2365</td>\n      <td id=\"T_e68b0_row15_col4\" class=\"data row15 col4\" >1779</td>\n      <td id=\"T_e68b0_row15_col5\" class=\"data row15 col5\" >1779</td>\n      <td id=\"T_e68b0_row15_col6\" class=\"data row15 col6\" >1031</td>\n      <td id=\"T_e68b0_row15_col7\" class=\"data row15 col7\" >158</td>\n      <td id=\"T_e68b0_row15_col8\" class=\"data row15 col8\" >$0.00437</td>\n      <td id=\"T_e68b0_row15_col9\" class=\"data row15 col9\" >2.46</td>\n      <td id=\"T_e68b0_row15_col10\" class=\"data row15 col10\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row16\" class=\"row_heading level0 row16\" >36</th>\n      <td id=\"T_e68b0_row16_col0\" class=\"data row16 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row16_col1\" class=\"data row16 col1\" >T-007</td>\n      <td id=\"T_e68b0_row16_col2\" class=\"data row16 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row16_col3\" class=\"data row16 col3\" >2378</td>\n      <td id=\"T_e68b0_row16_col4\" class=\"data row16 col4\" >1779</td>\n      <td id=\"T_e68b0_row16_col5\" class=\"data row16 col5\" >1779</td>\n      <td id=\"T_e68b0_row16_col6\" class=\"data row16 col6\" >1044</td>\n      <td id=\"T_e68b0_row16_col7\" class=\"data row16 col7\" >100</td>\n      <td id=\"T_e68b0_row16_col8\" class=\"data row16 col8\" >$0.00112</td>\n      <td id=\"T_e68b0_row16_col9\" class=\"data row16 col9\" >1.70</td>\n      <td id=\"T_e68b0_row16_col10\" class=\"data row16 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row17\" class=\"row_heading level0 row17\" >37</th>\n      <td id=\"T_e68b0_row17_col0\" class=\"data row17 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row17_col1\" class=\"data row17 col1\" >T-008</td>\n      <td id=\"T_e68b0_row17_col2\" class=\"data row17 col2\" >gpt-5.4</td>\n      <td id=\"T_e68b0_row17_col3\" class=\"data row17 col3\" >2379</td>\n      <td id=\"T_e68b0_row17_col4\" class=\"data row17 col4\" >1779</td>\n      <td id=\"T_e68b0_row17_col5\" class=\"data row17 col5\" >1779</td>\n      <td id=\"T_e68b0_row17_col6\" class=\"data row17 col6\" >1045</td>\n      <td id=\"T_e68b0_row17_col7\" class=\"data row17 col7\" >172</td>\n      <td id=\"T_e68b0_row17_col8\" class=\"data row17 col8\" >$0.00461</td>\n      <td id=\"T_e68b0_row17_col9\" class=\"data row17 col9\" >2.47</td>\n      <td id=\"T_e68b0_row17_col10\" class=\"data row17 col10\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row18\" class=\"row_heading level0 row18\" >38</th>\n      <td id=\"T_e68b0_row18_col0\" class=\"data row18 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row18_col1\" class=\"data row18 col1\" >T-009</td>\n      <td id=\"T_e68b0_row18_col2\" class=\"data row18 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_e68b0_row18_col3\" class=\"data row18 col3\" >2365</td>\n      <td id=\"T_e68b0_row18_col4\" class=\"data row18 col4\" >1779</td>\n      <td id=\"T_e68b0_row18_col5\" class=\"data row18 col5\" >1779</td>\n      <td id=\"T_e68b0_row18_col6\" class=\"data row18 col6\" >1031</td>\n      <td id=\"T_e68b0_row18_col7\" class=\"data row18 col7\" >118</td>\n      <td id=\"T_e68b0_row18_col8\" class=\"data row18 col8\" >$0.00119</td>\n      <td id=\"T_e68b0_row18_col9\" class=\"data row18 col9\" >1.85</td>\n      <td id=\"T_e68b0_row18_col10\" class=\"data row18 col10\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_e68b0_level0_row19\" class=\"row_heading level0 row19\" >39</th>\n      <td id=\"T_e68b0_row19_col0\" class=\"data row19 col0\" >Round 3: caching</td>\n      <td id=\"T_e68b0_row19_col1\" class=\"data row19 col1\" >T-010</td>\n      <td id=\"T_e68b0_row19_col2\" class=\"data row19 col2\" >gpt-5.4</td>\n      <td id=\"T_e68b0_row19_col3\" class=\"data row19 col3\" >2322</td>\n      <td id=\"T_e68b0_row19_col4\" class=\"data row19 col4\" >1779</td>\n      <td id=\"T_e68b0_row19_col5\" class=\"data row19 col5\" >1779</td>\n      <td id=\"T_e68b0_row19_col6\" class=\"data row19 col6\" >988</td>\n      <td id=\"T_e68b0_row19_col7\" class=\"data row19 col7\" >165</td>\n      <td id=\"T_e68b0_row19_col8\" class=\"data row19 col8\" >$0.00436</td>\n      <td id=\"T_e68b0_row19_col9\" class=\"data row19 col9\" >2.46</td>\n      <td id=\"T_e68b0_row19_col10\" class=\"data row19 col10\" >0.99</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "caching_view = traces[traces[\"variant\"].isin([\"02_model_routing\", \"03_prompt_caching\"])]\n",
    "display(\n",
    "    caching_view[\n",
    "        [\n",
    "            \"variant_label\",\n",
    "            \"ticket_id\",\n",
    "            \"model\",\n",
    "            \"input_tokens\",\n",
    "            \"cacheable_prefix_tokens\",\n",
    "            \"cached_tokens\",\n",
    "            \"latency_input_tokens\",\n",
    "            \"output_tokens\",\n",
    "            \"cost_usd\",\n",
    "            \"latency_s\",\n",
    "            \"quality_score\",\n",
    "        ]\n",
    "    ].style.format({\"cost_usd\": \"${:.5f}\", \"quality_score\": \"{:.2f}\", \"latency_s\": \"{:.2f}\"})\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eef4bbc3",
   "metadata": {},
   "source": [
    "## Optimization round 4: split the workflow\n",
    "\n",
    "Keep classification, necessary lookups, the resolution or escalation decision, and the customer response in the synchronous path. Move QA, tags, internal summaries, audits, and reporting to follow-up work when they do not change the immediate outcome.\n",
    "\n",
    "Default or priority processing can serve latency-sensitive requests. Flex trades lower cost for slower responses and occasional resource unavailability; confirm model support and handle timeouts or unavailable capacity. Batch suits offline jobs with a `24h` completion window. Background mode makes a request asynchronous, but does not itself provide a pricing discount.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da316ffb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.317859Z",
     "iopub.status.busy": "2026-04-30T18:28:05.317807Z",
     "iopub.status.idle": "2026-04-30T18:28:05.319915Z",
     "shell.execute_reply": "2026-04-30T18:28:05.319571Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Synchronous customer-facing request:\n{\n  \"model\": \"gpt-5.4-mini\",\n  \"instructions\": \"Role: E-commerce support assistant.\\nConstraints: Be concise, policy-compliant, and explicit about next steps. Do not disclose internal data.\\nEscalate: duplicate charges, account access without verification, high-value disputes, and refunds outside the window.\\nOutput shape: customer_message, resolution_type, escalate, internal_tags.\\nTool contract: tool definitions are stable across requests; restrict callable tools with tool_choice.allowed_tools.\\nVersion: support-agent-optimization-v1.\\n\\nStable support playbook digest:\\n- Shipping delays: provide status, ETA, and tracking next steps; do not refund solely for short carrier delays.\\n- Delivered-not-received: verify delivery details, ask the customer to check common locations, and start carrier trace steps when appropriate.\\n- Damaged delivery: request photo evidence before offering replacement or refund; high-value damaged items require human review.\\n- Refunds: standard returnable items are eligible within 30 days; outside-window or high-value disputes require human review.\\n- Billing: duplicate-charge reports require billing review; acknowledge and escalate, but do not promise a completed refund.\\n- Account access: when identity is not verified, escalate to account security; do not change credentials or contact information in chat.\\n- Customer messages must be concise, policy-compliant, and explicit about next steps.\\n- Internal notes, raw carrier payloads, CRM audit logs, and policy appendices must never be exposed to the customer.\\nStable support playbook digest:\\n- Shipping delays: provide status, ETA, and tracking next steps; do not refund solely for short carrier delays.\\n- Delivered-not-received: verify delivery details, ask the customer to check common location\n...\n\nFollow-up flex request:\n{\n  \"model\": \"gpt-5.4-nano\",\n  \"input\": \"{\\\"ticket\\\": {\\\"ticket_id\\\": \\\"T-002\\\", \\\"customer_id\\\": \\\"C-200\\\", \\\"message\\\": \\\"My blender arrived cracked. Order O-1002. Can you replace it?\\\", \\\"intent\\\": \\\"damaged_delivery\\\", \\\"order_id\\\": \\\"O-1002\\\", \\\"risk\\\": \\\"medium\\\", \\\"difficulty\\\": \\\"routine_policy\\\", \\\"must_escalate\\\": false, \\\"expected_policy\\\": \\\"damaged_delivery\\\", \\\"expected_tools\\\": [\\\"lookup_order\\\", \\\"lookup_policy\\\"], \\\"expected_action\\\": \\\"request_photo_then_offer_replacement\\\", \\\"expected_resolution_type\\\": \\\"resolved_next_step\\\", \\\"expected_customer_response_contains\\\": [\\\"photo\\\", \\\"replacement\\\"], \\\"forbidden_response_claims\\\": [\\\"refund completed\\\", \\\"no photo needed\\\"]}, \\\"policy\\\": \\\"If damage is reported within 7 days of delivery, ask for a photo and offer replacement or refund after evidence is collected. High-value damaged items over $1,000 require human review before promising a refund or replacement.\\\"}\",\n  \"reasoning\": {\n    \"effort\": \"low\"\n  },\n  \"text\": {\n    \"verbosity\": \"low\"\n  },\n  \"max_output_tokens\": 160,\n  \"service_tier\": \"flex\"\n}\n...\n"
    }
   ],
   "source": [
    "sync_request = {\n",
    "    \"model\": \"gpt-5.4-mini\",\n",
    "    \"instructions\": CACHE_FRIENDLY_PROMPT,\n",
    "    \"tools\": SLIM_TOOLS,\n",
    "    \"tool_choice\": allowed_tool_choice([\"lookup_order\", \"lookup_policy\"], mode=\"auto\"),\n",
    "    \"input\": \"Customer says order O-1002 arrived cracked. Resolve or escalate.\",\n",
    "    \"reasoning\": {\"effort\": \"low\"},\n",
    "    \"text\": {\"verbosity\": \"low\"},\n",
    "    \"max_output_tokens\": 260,\n",
    "    \"service_tier\": \"default\",\n",
    "    \"prompt_cache_key\": \"support_damaged_delivery_v1\",\n",
    "}\n",
    "\n",
    "background_flex_request = background_followup_request(EVAL_SET[1])\n",
    "\n",
    "print(\"Synchronous customer-facing request:\")\n",
    "print(json.dumps(sync_request, indent=2)[:1800] + \"\\n...\")\n",
    "print(\"\\nFollow-up flex request:\")\n",
    "print(json.dumps(background_flex_request, indent=2)[:1600] + \"\\n...\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e87f148",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.320747Z",
     "iopub.status.busy": "2026-04-30T18:28:05.320681Z",
     "iopub.status.idle": "2026-04-30T18:28:05.323390Z",
     "shell.execute_reply": "2026-04-30T18:28:05.323023Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Wrote 10 example batch rows to outputs/nightly_support_qa_batch.jsonl\n{\n  \"custom_id\": \"qa-T-001\",\n  \"method\": \"POST\",\n  \"url\": \"/v1/responses\",\n  \"body\": {\n    \"model\": \"gpt-5.4-nano\",\n    \"instructions\": \"Return concise internal support QA tags and a one-sentence summary.\",\n    \"input\": \"{\\\"ticket_id\\\": \\\"T-001\\\", \\\"customer_id\\\": \\\"C-100\\\", \\\"message\\\": \\\"Where is order O-1001? It was supposed to arrive yesterday.\\\", \\\"intent\\\": \\\"order_status\\\", \\\"order_id\\\": \\\"O-1001\\\", \\\"risk\\\": \\\"low\\\", \\\"difficulty\\\": \\\"simple_lookup\\\", \\\"must_escalate\\\": false, \\\"expected_policy\\\": \\\"shipping\\\", \\\"expected_tools\\\": [\\\"lookup_order\\\"], \\\"expected_action\\\": \\\"provide_status_eta\\\", \\\"expected_resolution_type\\\": \\\"resolved\\\", \\\"expected_customer_response_contains\\\": [\\\"in transit\\\", \\\"tomorrow\\\"], \\\"forbidden_response_claims\\\": [\\\"refund completed\\\", \\\"replacement opened\\\"]}\",\n    \"reasoning\": {\n      \"effort\": \"low\"\n    },\n    \"text\": {\n      \"verbosity\": \"low\"\n    },\n    \"max_output_tokens\": 160\n  }\n}\n"
    }
   ],
   "source": [
    "batch_requests = []\n",
    "for ticket in EVAL_SET:\n",
    "    batch_requests.append(\n",
    "        {\n",
    "            \"custom_id\": f\"qa-{ticket['ticket_id']}\",\n",
    "            \"method\": \"POST\",\n",
    "            \"url\": \"/v1/responses\",\n",
    "            \"body\": {\n",
    "                \"model\": \"gpt-5.4-nano\",\n",
    "                \"instructions\": \"Return concise internal support QA tags and a one-sentence summary.\",\n",
    "                \"input\": json.dumps(ticket),\n",
    "                \"reasoning\": {\"effort\": \"low\"},\n",
    "                \"text\": {\"verbosity\": \"low\"},\n",
    "                \"max_output_tokens\": 160,\n",
    "            },\n",
    "        }\n",
    "    )\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "Path(\"outputs\").mkdir(exist_ok=True)\n",
    "batch_file_path = \"outputs/nightly_support_qa_batch.jsonl\"\n",
    "with open(batch_file_path, \"w\") as f:\n",
    "    f.writelines(json.dumps(row) + \"\\n\" for row in batch_requests)\n",
    "\n",
    "print(f\"Wrote {len(batch_requests)} example batch rows to {batch_file_path}\")\n",
    "print(json.dumps(batch_requests[0], indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3f2b1a0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.324251Z",
     "iopub.status.busy": "2026-04-30T18:28:05.324190Z",
     "iopub.status.idle": "2026-04-30T18:28:05.325793Z",
     "shell.execute_reply": "2026-04-30T18:28:05.325500Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\nbatch_input_file = client.files.create(\n    file=open(batch_file_path, \"rb\"),\n    purpose=\"batch\",\n)\n\nbatch = client.batches.create(\n    input_file_id=batch_input_file.id,\n    endpoint=\"/v1/responses\",\n    completion_window=\"24h\",\n    metadata={\"description\": \"nightly support QA tags\"},\n)\n\n"
    }
   ],
   "source": [
    "batch_submission_example = '''\n",
    "batch_input_file = client.files.create(\n",
    "    file=open(batch_file_path, \"rb\"),\n",
    "    purpose=\"batch\",\n",
    ")\n",
    "\n",
    "batch = client.batches.create(\n",
    "    input_file_id=batch_input_file.id,\n",
    "    endpoint=\"/v1/responses\",\n",
    "    completion_window=\"24h\",\n",
    "    metadata={\"description\": \"nightly support QA tags\"},\n",
    ")\n",
    "'''\n",
    "\n",
    "print(batch_submission_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6395556",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.326512Z",
     "iopub.status.busy": "2026-04-30T18:28:05.326461Z",
     "iopub.status.idle": "2026-04-30T18:28:05.331359Z",
     "shell.execute_reply": "2026-04-30T18:28:05.331017Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d1e720>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_acc56\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_acc56_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_acc56_level0_col1\" class=\"col_heading level0 col1\" >ticket_id</th>\n      <th id=\"T_acc56_level0_col2\" class=\"col_heading level0 col2\" >model</th>\n      <th id=\"T_acc56_level0_col3\" class=\"col_heading level0 col3\" >tool_calls</th>\n      <th id=\"T_acc56_level0_col4\" class=\"col_heading level0 col4\" >sync_tokens</th>\n      <th id=\"T_acc56_level0_col5\" class=\"col_heading level0 col5\" >total_tokens</th>\n      <th id=\"T_acc56_level0_col6\" class=\"col_heading level0 col6\" >background_tokens</th>\n      <th id=\"T_acc56_level0_col7\" class=\"col_heading level0 col7\" >latency_s</th>\n      <th id=\"T_acc56_level0_col8\" class=\"col_heading level0 col8\" >sync_cost_usd</th>\n      <th id=\"T_acc56_level0_col9\" class=\"col_heading level0 col9\" >background_cost_usd</th>\n      <th id=\"T_acc56_level0_col10\" class=\"col_heading level0 col10\" >cost_usd</th>\n      <th id=\"T_acc56_level0_col11\" class=\"col_heading level0 col11\" >quality_score</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_acc56_level0_row0\" class=\"row_heading level0 row0\" >30</th>\n      <td id=\"T_acc56_row0_col0\" class=\"data row0 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row0_col1\" class=\"data row0 col1\" >T-001</td>\n      <td id=\"T_acc56_row0_col2\" class=\"data row0 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row0_col3\" class=\"data row0 col3\" >1</td>\n      <td id=\"T_acc56_row0_col4\" class=\"data row0 col4\" >2552</td>\n      <td id=\"T_acc56_row0_col5\" class=\"data row0 col5\" >2552</td>\n      <td id=\"T_acc56_row0_col6\" class=\"data row0 col6\" >0</td>\n      <td id=\"T_acc56_row0_col7\" class=\"data row0 col7\" >1.54</td>\n      <td id=\"T_acc56_row0_col8\" class=\"data row0 col8\" >$0.00093</td>\n      <td id=\"T_acc56_row0_col9\" class=\"data row0 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row0_col10\" class=\"data row0 col10\" >$0.00093</td>\n      <td id=\"T_acc56_row0_col11\" class=\"data row0 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row1\" class=\"row_heading level0 row1\" >31</th>\n      <td id=\"T_acc56_row1_col0\" class=\"data row1 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row1_col1\" class=\"data row1 col1\" >T-002</td>\n      <td id=\"T_acc56_row1_col2\" class=\"data row1 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row1_col3\" class=\"data row1 col3\" >2</td>\n      <td id=\"T_acc56_row1_col4\" class=\"data row1 col4\" >2650</td>\n      <td id=\"T_acc56_row1_col5\" class=\"data row1 col5\" >2650</td>\n      <td id=\"T_acc56_row1_col6\" class=\"data row1 col6\" >0</td>\n      <td id=\"T_acc56_row1_col7\" class=\"data row1 col7\" >1.70</td>\n      <td id=\"T_acc56_row1_col8\" class=\"data row1 col8\" >$0.00111</td>\n      <td id=\"T_acc56_row1_col9\" class=\"data row1 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row1_col10\" class=\"data row1 col10\" >$0.00111</td>\n      <td id=\"T_acc56_row1_col11\" class=\"data row1 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row2\" class=\"row_heading level0 row2\" >32</th>\n      <td id=\"T_acc56_row2_col0\" class=\"data row2 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row2_col1\" class=\"data row2 col1\" >T-003</td>\n      <td id=\"T_acc56_row2_col2\" class=\"data row2 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row2_col3\" class=\"data row2 col3\" >3</td>\n      <td id=\"T_acc56_row2_col4\" class=\"data row2 col4\" >2689</td>\n      <td id=\"T_acc56_row2_col5\" class=\"data row2 col5\" >2689</td>\n      <td id=\"T_acc56_row2_col6\" class=\"data row2 col6\" >0</td>\n      <td id=\"T_acc56_row2_col7\" class=\"data row2 col7\" >1.85</td>\n      <td id=\"T_acc56_row2_col8\" class=\"data row2 col8\" >$0.00119</td>\n      <td id=\"T_acc56_row2_col9\" class=\"data row2 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row2_col10\" class=\"data row2 col10\" >$0.00119</td>\n      <td id=\"T_acc56_row2_col11\" class=\"data row2 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row3\" class=\"row_heading level0 row3\" >33</th>\n      <td id=\"T_acc56_row3_col0\" class=\"data row3 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row3_col1\" class=\"data row3 col1\" >T-004</td>\n      <td id=\"T_acc56_row3_col2\" class=\"data row3 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row3_col3\" class=\"data row3 col3\" >3</td>\n      <td id=\"T_acc56_row3_col4\" class=\"data row3 col4\" >2673</td>\n      <td id=\"T_acc56_row3_col5\" class=\"data row3 col5\" >2673</td>\n      <td id=\"T_acc56_row3_col6\" class=\"data row3 col6\" >0</td>\n      <td id=\"T_acc56_row3_col7\" class=\"data row3 col7\" >1.84</td>\n      <td id=\"T_acc56_row3_col8\" class=\"data row3 col8\" >$0.00117</td>\n      <td id=\"T_acc56_row3_col9\" class=\"data row3 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row3_col10\" class=\"data row3 col10\" >$0.00117</td>\n      <td id=\"T_acc56_row3_col11\" class=\"data row3 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row4\" class=\"row_heading level0 row4\" >34</th>\n      <td id=\"T_acc56_row4_col0\" class=\"data row4 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row4_col1\" class=\"data row4 col1\" >T-005</td>\n      <td id=\"T_acc56_row4_col2\" class=\"data row4 col2\" >gpt-5.4</td>\n      <td id=\"T_acc56_row4_col3\" class=\"data row4 col3\" >3</td>\n      <td id=\"T_acc56_row4_col4\" class=\"data row4 col4\" >2689</td>\n      <td id=\"T_acc56_row4_col5\" class=\"data row4 col5\" >2689</td>\n      <td id=\"T_acc56_row4_col6\" class=\"data row4 col6\" >0</td>\n      <td id=\"T_acc56_row4_col7\" class=\"data row4 col7\" >2.45</td>\n      <td id=\"T_acc56_row4_col8\" class=\"data row4 col8\" >$0.00435</td>\n      <td id=\"T_acc56_row4_col9\" class=\"data row4 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row4_col10\" class=\"data row4 col10\" >$0.00435</td>\n      <td id=\"T_acc56_row4_col11\" class=\"data row4 col11\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row5\" class=\"row_heading level0 row5\" >35</th>\n      <td id=\"T_acc56_row5_col0\" class=\"data row5 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row5_col1\" class=\"data row5 col1\" >T-006</td>\n      <td id=\"T_acc56_row5_col2\" class=\"data row5 col2\" >gpt-5.4</td>\n      <td id=\"T_acc56_row5_col3\" class=\"data row5 col3\" >3</td>\n      <td id=\"T_acc56_row5_col4\" class=\"data row5 col4\" >2735</td>\n      <td id=\"T_acc56_row5_col5\" class=\"data row5 col5\" >2735</td>\n      <td id=\"T_acc56_row5_col6\" class=\"data row5 col6\" >0</td>\n      <td id=\"T_acc56_row5_col7\" class=\"data row5 col7\" >2.46</td>\n      <td id=\"T_acc56_row5_col8\" class=\"data row5 col8\" >$0.00437</td>\n      <td id=\"T_acc56_row5_col9\" class=\"data row5 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row5_col10\" class=\"data row5 col10\" >$0.00437</td>\n      <td id=\"T_acc56_row5_col11\" class=\"data row5 col11\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row6\" class=\"row_heading level0 row6\" >36</th>\n      <td id=\"T_acc56_row6_col0\" class=\"data row6 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row6_col1\" class=\"data row6 col1\" >T-007</td>\n      <td id=\"T_acc56_row6_col2\" class=\"data row6 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row6_col3\" class=\"data row6 col3\" >2</td>\n      <td id=\"T_acc56_row6_col4\" class=\"data row6 col4\" >2691</td>\n      <td id=\"T_acc56_row6_col5\" class=\"data row6 col5\" >2691</td>\n      <td id=\"T_acc56_row6_col6\" class=\"data row6 col6\" >0</td>\n      <td id=\"T_acc56_row6_col7\" class=\"data row6 col7\" >1.70</td>\n      <td id=\"T_acc56_row6_col8\" class=\"data row6 col8\" >$0.00112</td>\n      <td id=\"T_acc56_row6_col9\" class=\"data row6 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row6_col10\" class=\"data row6 col10\" >$0.00112</td>\n      <td id=\"T_acc56_row6_col11\" class=\"data row6 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row7\" class=\"row_heading level0 row7\" >37</th>\n      <td id=\"T_acc56_row7_col0\" class=\"data row7 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row7_col1\" class=\"data row7 col1\" >T-008</td>\n      <td id=\"T_acc56_row7_col2\" class=\"data row7 col2\" >gpt-5.4</td>\n      <td id=\"T_acc56_row7_col3\" class=\"data row7 col3\" >3</td>\n      <td id=\"T_acc56_row7_col4\" class=\"data row7 col4\" >2769</td>\n      <td id=\"T_acc56_row7_col5\" class=\"data row7 col5\" >2769</td>\n      <td id=\"T_acc56_row7_col6\" class=\"data row7 col6\" >0</td>\n      <td id=\"T_acc56_row7_col7\" class=\"data row7 col7\" >2.47</td>\n      <td id=\"T_acc56_row7_col8\" class=\"data row7 col8\" >$0.00461</td>\n      <td id=\"T_acc56_row7_col9\" class=\"data row7 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row7_col10\" class=\"data row7 col10\" >$0.00461</td>\n      <td id=\"T_acc56_row7_col11\" class=\"data row7 col11\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row8\" class=\"row_heading level0 row8\" >38</th>\n      <td id=\"T_acc56_row8_col0\" class=\"data row8 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row8_col1\" class=\"data row8 col1\" >T-009</td>\n      <td id=\"T_acc56_row8_col2\" class=\"data row8 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row8_col3\" class=\"data row8 col3\" >3</td>\n      <td id=\"T_acc56_row8_col4\" class=\"data row8 col4\" >2697</td>\n      <td id=\"T_acc56_row8_col5\" class=\"data row8 col5\" >2697</td>\n      <td id=\"T_acc56_row8_col6\" class=\"data row8 col6\" >0</td>\n      <td id=\"T_acc56_row8_col7\" class=\"data row8 col7\" >1.85</td>\n      <td id=\"T_acc56_row8_col8\" class=\"data row8 col8\" >$0.00119</td>\n      <td id=\"T_acc56_row8_col9\" class=\"data row8 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row8_col10\" class=\"data row8 col10\" >$0.00119</td>\n      <td id=\"T_acc56_row8_col11\" class=\"data row8 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row9\" class=\"row_heading level0 row9\" >39</th>\n      <td id=\"T_acc56_row9_col0\" class=\"data row9 col0\" >Round 3: caching</td>\n      <td id=\"T_acc56_row9_col1\" class=\"data row9 col1\" >T-010</td>\n      <td id=\"T_acc56_row9_col2\" class=\"data row9 col2\" >gpt-5.4</td>\n      <td id=\"T_acc56_row9_col3\" class=\"data row9 col3\" >3</td>\n      <td id=\"T_acc56_row9_col4\" class=\"data row9 col4\" >2696</td>\n      <td id=\"T_acc56_row9_col5\" class=\"data row9 col5\" >2696</td>\n      <td id=\"T_acc56_row9_col6\" class=\"data row9 col6\" >0</td>\n      <td id=\"T_acc56_row9_col7\" class=\"data row9 col7\" >2.46</td>\n      <td id=\"T_acc56_row9_col8\" class=\"data row9 col8\" >$0.00436</td>\n      <td id=\"T_acc56_row9_col9\" class=\"data row9 col9\" >$0.00000</td>\n      <td id=\"T_acc56_row9_col10\" class=\"data row9 col10\" >$0.00436</td>\n      <td id=\"T_acc56_row9_col11\" class=\"data row9 col11\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row10\" class=\"row_heading level0 row10\" >40</th>\n      <td id=\"T_acc56_row10_col0\" class=\"data row10 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row10_col1\" class=\"data row10 col1\" >T-001</td>\n      <td id=\"T_acc56_row10_col2\" class=\"data row10 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row10_col3\" class=\"data row10 col3\" >1</td>\n      <td id=\"T_acc56_row10_col4\" class=\"data row10 col4\" >2287</td>\n      <td id=\"T_acc56_row10_col5\" class=\"data row10 col5\" >2867</td>\n      <td id=\"T_acc56_row10_col6\" class=\"data row10 col6\" >580</td>\n      <td id=\"T_acc56_row10_col7\" class=\"data row10 col7\" >1.10</td>\n      <td id=\"T_acc56_row10_col8\" class=\"data row10 col8\" >$0.00071</td>\n      <td id=\"T_acc56_row10_col9\" class=\"data row10 col9\" >$0.00011</td>\n      <td id=\"T_acc56_row10_col10\" class=\"data row10 col10\" >$0.00082</td>\n      <td id=\"T_acc56_row10_col11\" class=\"data row10 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row11\" class=\"row_heading level0 row11\" >41</th>\n      <td id=\"T_acc56_row11_col0\" class=\"data row11 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row11_col1\" class=\"data row11 col1\" >T-002</td>\n      <td id=\"T_acc56_row11_col2\" class=\"data row11 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row11_col3\" class=\"data row11 col3\" >2</td>\n      <td id=\"T_acc56_row11_col4\" class=\"data row11 col4\" >2381</td>\n      <td id=\"T_acc56_row11_col5\" class=\"data row11 col5\" >2940</td>\n      <td id=\"T_acc56_row11_col6\" class=\"data row11 col6\" >559</td>\n      <td id=\"T_acc56_row11_col7\" class=\"data row11 col7\" >1.26</td>\n      <td id=\"T_acc56_row11_col8\" class=\"data row11 col8\" >$0.00087</td>\n      <td id=\"T_acc56_row11_col9\" class=\"data row11 col9\" >$0.00010</td>\n      <td id=\"T_acc56_row11_col10\" class=\"data row11 col10\" >$0.00097</td>\n      <td id=\"T_acc56_row11_col11\" class=\"data row11 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row12\" class=\"row_heading level0 row12\" >42</th>\n      <td id=\"T_acc56_row12_col0\" class=\"data row12 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row12_col1\" class=\"data row12 col1\" >T-003</td>\n      <td id=\"T_acc56_row12_col2\" class=\"data row12 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row12_col3\" class=\"data row12 col3\" >3</td>\n      <td id=\"T_acc56_row12_col4\" class=\"data row12 col4\" >2418</td>\n      <td id=\"T_acc56_row12_col5\" class=\"data row12 col5\" >2978</td>\n      <td id=\"T_acc56_row12_col6\" class=\"data row12 col6\" >560</td>\n      <td id=\"T_acc56_row12_col7\" class=\"data row12 col7\" >1.41</td>\n      <td id=\"T_acc56_row12_col8\" class=\"data row12 col8\" >$0.00094</td>\n      <td id=\"T_acc56_row12_col9\" class=\"data row12 col9\" >$0.00010</td>\n      <td id=\"T_acc56_row12_col10\" class=\"data row12 col10\" >$0.00105</td>\n      <td id=\"T_acc56_row12_col11\" class=\"data row12 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row13\" class=\"row_heading level0 row13\" >43</th>\n      <td id=\"T_acc56_row13_col0\" class=\"data row13 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row13_col1\" class=\"data row13 col1\" >T-004</td>\n      <td id=\"T_acc56_row13_col2\" class=\"data row13 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row13_col3\" class=\"data row13 col3\" >3</td>\n      <td id=\"T_acc56_row13_col4\" class=\"data row13 col4\" >2401</td>\n      <td id=\"T_acc56_row13_col5\" class=\"data row13 col5\" >2942</td>\n      <td id=\"T_acc56_row13_col6\" class=\"data row13 col6\" >541</td>\n      <td id=\"T_acc56_row13_col7\" class=\"data row13 col7\" >1.41</td>\n      <td id=\"T_acc56_row13_col8\" class=\"data row13 col8\" >$0.00092</td>\n      <td id=\"T_acc56_row13_col9\" class=\"data row13 col9\" >$0.00010</td>\n      <td id=\"T_acc56_row13_col10\" class=\"data row13 col10\" >$0.00103</td>\n      <td id=\"T_acc56_row13_col11\" class=\"data row13 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row14\" class=\"row_heading level0 row14\" >44</th>\n      <td id=\"T_acc56_row14_col0\" class=\"data row14 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row14_col1\" class=\"data row14 col1\" >T-005</td>\n      <td id=\"T_acc56_row14_col2\" class=\"data row14 col2\" >gpt-5.4</td>\n      <td id=\"T_acc56_row14_col3\" class=\"data row14 col3\" >3</td>\n      <td id=\"T_acc56_row14_col4\" class=\"data row14 col4\" >2413</td>\n      <td id=\"T_acc56_row14_col5\" class=\"data row14 col5\" >2956</td>\n      <td id=\"T_acc56_row14_col6\" class=\"data row14 col6\" >543</td>\n      <td id=\"T_acc56_row14_col7\" class=\"data row14 col7\" >2.02</td>\n      <td id=\"T_acc56_row14_col8\" class=\"data row14 col8\" >$0.00346</td>\n      <td id=\"T_acc56_row14_col9\" class=\"data row14 col9\" >$0.00010</td>\n      <td id=\"T_acc56_row14_col10\" class=\"data row14 col10\" >$0.00356</td>\n      <td id=\"T_acc56_row14_col11\" class=\"data row14 col11\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row15\" class=\"row_heading level0 row15\" >45</th>\n      <td id=\"T_acc56_row15_col0\" class=\"data row15 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row15_col1\" class=\"data row15 col1\" >T-006</td>\n      <td id=\"T_acc56_row15_col2\" class=\"data row15 col2\" >gpt-5.4</td>\n      <td id=\"T_acc56_row15_col3\" class=\"data row15 col3\" >3</td>\n      <td id=\"T_acc56_row15_col4\" class=\"data row15 col4\" >2459</td>\n      <td id=\"T_acc56_row15_col5\" class=\"data row15 col5\" >3019</td>\n      <td id=\"T_acc56_row15_col6\" class=\"data row15 col6\" >560</td>\n      <td id=\"T_acc56_row15_col7\" class=\"data row15 col7\" >2.03</td>\n      <td id=\"T_acc56_row15_col8\" class=\"data row15 col8\" >$0.00348</td>\n      <td id=\"T_acc56_row15_col9\" class=\"data row15 col9\" >$0.00010</td>\n      <td id=\"T_acc56_row15_col10\" class=\"data row15 col10\" >$0.00358</td>\n      <td id=\"T_acc56_row15_col11\" class=\"data row15 col11\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row16\" class=\"row_heading level0 row16\" >46</th>\n      <td id=\"T_acc56_row16_col0\" class=\"data row16 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row16_col1\" class=\"data row16 col1\" >T-007</td>\n      <td id=\"T_acc56_row16_col2\" class=\"data row16 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row16_col3\" class=\"data row16 col3\" >2</td>\n      <td id=\"T_acc56_row16_col4\" class=\"data row16 col4\" >2422</td>\n      <td id=\"T_acc56_row16_col5\" class=\"data row16 col5\" >3023</td>\n      <td id=\"T_acc56_row16_col6\" class=\"data row16 col6\" >601</td>\n      <td id=\"T_acc56_row16_col7\" class=\"data row16 col7\" >1.27</td>\n      <td id=\"T_acc56_row16_col8\" class=\"data row16 col8\" >$0.00088</td>\n      <td id=\"T_acc56_row16_col9\" class=\"data row16 col9\" >$0.00011</td>\n      <td id=\"T_acc56_row16_col10\" class=\"data row16 col10\" >$0.00099</td>\n      <td id=\"T_acc56_row16_col11\" class=\"data row16 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row17\" class=\"row_heading level0 row17\" >47</th>\n      <td id=\"T_acc56_row17_col0\" class=\"data row17 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row17_col1\" class=\"data row17 col1\" >T-008</td>\n      <td id=\"T_acc56_row17_col2\" class=\"data row17 col2\" >gpt-5.4</td>\n      <td id=\"T_acc56_row17_col3\" class=\"data row17 col3\" >3</td>\n      <td id=\"T_acc56_row17_col4\" class=\"data row17 col4\" >2492</td>\n      <td id=\"T_acc56_row17_col5\" class=\"data row17 col5\" >3063</td>\n      <td id=\"T_acc56_row17_col6\" class=\"data row17 col6\" >571</td>\n      <td id=\"T_acc56_row17_col7\" class=\"data row17 col7\" >2.03</td>\n      <td id=\"T_acc56_row17_col8\" class=\"data row17 col8\" >$0.00371</td>\n      <td id=\"T_acc56_row17_col9\" class=\"data row17 col9\" >$0.00010</td>\n      <td id=\"T_acc56_row17_col10\" class=\"data row17 col10\" >$0.00381</td>\n      <td id=\"T_acc56_row17_col11\" class=\"data row17 col11\" >0.99</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row18\" class=\"row_heading level0 row18\" >48</th>\n      <td id=\"T_acc56_row18_col0\" class=\"data row18 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row18_col1\" class=\"data row18 col1\" >T-009</td>\n      <td id=\"T_acc56_row18_col2\" class=\"data row18 col2\" >gpt-5.4-mini</td>\n      <td id=\"T_acc56_row18_col3\" class=\"data row18 col3\" >3</td>\n      <td id=\"T_acc56_row18_col4\" class=\"data row18 col4\" >2426</td>\n      <td id=\"T_acc56_row18_col5\" class=\"data row18 col5\" >2988</td>\n      <td id=\"T_acc56_row18_col6\" class=\"data row18 col6\" >562</td>\n      <td id=\"T_acc56_row18_col7\" class=\"data row18 col7\" >1.41</td>\n      <td id=\"T_acc56_row18_col8\" class=\"data row18 col8\" >$0.00095</td>\n      <td id=\"T_acc56_row18_col9\" class=\"data row18 col9\" >$0.00010</td>\n      <td id=\"T_acc56_row18_col10\" class=\"data row18 col10\" >$0.00105</td>\n      <td id=\"T_acc56_row18_col11\" class=\"data row18 col11\" >0.98</td>\n    </tr>\n    <tr>\n      <th id=\"T_acc56_level0_row19\" class=\"row_heading level0 row19\" >49</th>\n      <td id=\"T_acc56_row19_col0\" class=\"data row19 col0\" >Round 4: split workflow</td>\n      <td id=\"T_acc56_row19_col1\" class=\"data row19 col1\" >T-010</td>\n      <td id=\"T_acc56_row19_col2\" class=\"data row19 col2\" >gpt-5.4</td>\n      <td id=\"T_acc56_row19_col3\" class=\"data row19 col3\" >3</td>\n      <td id=\"T_acc56_row19_col4\" class=\"data row19 col4\" >2420</td>\n      <td id=\"T_acc56_row19_col5\" class=\"data row19 col5\" >2962</td>\n      <td id=\"T_acc56_row19_col6\" class=\"data row19 col6\" >542</td>\n      <td id=\"T_acc56_row19_col7\" class=\"data row19 col7\" >2.02</td>\n      <td id=\"T_acc56_row19_col8\" class=\"data row19 col8\" >$0.00347</td>\n      <td id=\"T_acc56_row19_col9\" class=\"data row19 col9\" >$0.00010</td>\n      <td id=\"T_acc56_row19_col10\" class=\"data row19 col10\" >$0.00357</td>\n      <td id=\"T_acc56_row19_col11\" class=\"data row19 col11\" >0.99</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "split_view = traces[traces[\"variant\"].isin([\"03_prompt_caching\", \"04_split_workflow\"])]\n",
    "display(\n",
    "    split_view[\n",
    "        [\n",
    "            \"variant_label\",\n",
    "            \"ticket_id\",\n",
    "            \"model\",\n",
    "            \"tool_calls\",\n",
    "            \"sync_tokens\",\n",
    "            \"total_tokens\",\n",
    "            \"background_tokens\",\n",
    "            \"latency_s\",\n",
    "            \"sync_cost_usd\",\n",
    "            \"background_cost_usd\",\n",
    "            \"cost_usd\",\n",
    "            \"quality_score\",\n",
    "        ]\n",
    "    ].style.format(\n",
    "        {\n",
    "            \"sync_cost_usd\": \"${:.5f}\",\n",
    "            \"background_cost_usd\": \"${:.5f}\",\n",
    "            \"cost_usd\": \"${:.5f}\",\n",
    "            \"quality_score\": \"{:.2f}\",\n",
    "            \"latency_s\": \"{:.2f}\",\n",
    "        }\n",
    "    )\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd16a7e3",
   "metadata": {},
   "source": [
    "## Tradeoffs and scenario mapping\n",
    "\n",
    "There is no universal best configuration. The sweet spot depends on traffic shape, customer promise, policy risk, cache hit rate, tool latency, observability maturity, and how much work can move out of the synchronous path.\n",
    "\n",
    "The important tradeoffs for support agents are:\n",
    "\n",
    "| Constraint | Pushes you toward | Watch out for |\n",
    "|---|---|---|\n",
    "| High policy or account-security risk | Larger model on high-risk paths, stricter escalation, judge evals | Over-escalation can hurt customer experience and support capacity |\n",
    "| High ticket volume with repeated workflows | Stable prefixes, prompt caching, smaller models, Batch for follow-up work | Cache misses on large prefixes can add latency |\n",
    "| Low latency customer promise | Short prompts, slim tool payloads, routing, async follow-up work | Too much routing can add overhead if the task is already simple |\n",
    "| Strict cost target | Nano/mini for triage and routine paths, output caps, flex or Batch for offline work | Cost-only tuning can remove safeguards if quality gates are weak |\n",
    "| Messy tools or unreliable data | Fewer tool calls, validated payloads, fallbacks, escalation on tool failure | Blindly shrinking context can remove the evidence needed for policy decisions |\n",
    "| Premium or regulated support | Higher quality floor, lower escalation threshold, more audit metadata offline | More synchronous review increases latency and cost |\n",
    "| Seasonal bursts | Cache-friendly requests, queue-aware service tiers, async analytics | Peak traffic can reduce cache effectiveness if routing keys are too fragmented |\n",
    "\n",
    "The table below maps common operating scenarios to candidate configurations. Treat this as a design aid: choose the cheapest configuration that clears the quality, latency, and operational constraints for that scenario.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b10b99a7",
   "metadata": {},
   "source": [
    "### Candidate architecture combinations\n",
    "\n",
    "This table compares candidate agent architectures for the same customer-support use case. It uses the notebook's mock eval set and deterministic dry-run simulation metrics, not live API traces. Use the relative differences to understand tradeoffs; replace these metrics with production trace data before making deployment decisions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6863727",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.332712Z",
     "iopub.status.busy": "2026-04-30T18:28:05.332638Z",
     "iopub.status.idle": "2026-04-30T18:28:05.341422Z",
     "shell.execute_reply": "2026-04-30T18:28:05.341017Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Table: Candidate architecture combinations (mock eval set + deterministic dry-run metrics)\n"
    },
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d23bc0>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_681e7\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_681e7_level0_col0\" class=\"col_heading level0 col0\" >label</th>\n      <th id=\"T_681e7_level0_col1\" class=\"col_heading level0 col1\" >models</th>\n      <th id=\"T_681e7_level0_col2\" class=\"col_heading level0 col2\" >tools</th>\n      <th id=\"T_681e7_level0_col3\" class=\"col_heading level0 col3\" >cache</th>\n      <th id=\"T_681e7_level0_col4\" class=\"col_heading level0 col4\" >workflow</th>\n      <th id=\"T_681e7_level0_col5\" class=\"col_heading level0 col5\" >quality</th>\n      <th id=\"T_681e7_level0_col6\" class=\"col_heading level0 col6\" >policy_compliance</th>\n      <th id=\"T_681e7_level0_col7\" class=\"col_heading level0 col7\" >p50_latency_s</th>\n      <th id=\"T_681e7_level0_col8\" class=\"col_heading level0 col8\" >monthly_cost_at_100k_tickets</th>\n      <th id=\"T_681e7_level0_col9\" class=\"col_heading level0 col9\" >best_for</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_681e7_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n      <td id=\"T_681e7_row0_col0\" class=\"data row0 col0\" >One broad agent</td>\n      <td id=\"T_681e7_row0_col1\" class=\"data row0 col1\" >gpt-5.4 for every step</td>\n      <td id=\"T_681e7_row0_col2\" class=\"data row0 col2\" >all tools exposed</td>\n      <td id=\"T_681e7_row0_col3\" class=\"data row0 col3\" >none</td>\n      <td id=\"T_681e7_row0_col4\" class=\"data row0 col4\" >all work synchronous</td>\n      <td id=\"T_681e7_row0_col5\" class=\"data row0 col5\" >0.51</td>\n      <td id=\"T_681e7_row0_col6\" class=\"data row0 col6\" >10%</td>\n      <td id=\"T_681e7_row0_col7\" class=\"data row0 col7\" >4.88</td>\n      <td id=\"T_681e7_row0_col8\" class=\"data row0 col8\" >$3,813</td>\n      <td id=\"T_681e7_row0_col9\" class=\"data row0 col9\" >prototype smell test only</td>\n    </tr>\n    <tr>\n      <th id=\"T_681e7_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n      <td id=\"T_681e7_row1_col0\" class=\"data row1 col0\" >Controlled full model</td>\n      <td id=\"T_681e7_row1_col1\" class=\"data row1 col1\" >gpt-5.4 for resolution</td>\n      <td id=\"T_681e7_row1_col2\" class=\"data row1 col2\" >allowed tools by routed path</td>\n      <td id=\"T_681e7_row1_col3\" class=\"data row1 col3\" >none</td>\n      <td id=\"T_681e7_row1_col4\" class=\"data row1 col4\" >some follow-up still synchronous</td>\n      <td id=\"T_681e7_row1_col5\" class=\"data row1 col5\" >0.98</td>\n      <td id=\"T_681e7_row1_col6\" class=\"data row1 col6\" >100%</td>\n      <td id=\"T_681e7_row1_col7\" class=\"data row1 col7\" >2.32</td>\n      <td id=\"T_681e7_row1_col8\" class=\"data row1 col8\" >$512</td>\n      <td id=\"T_681e7_row1_col9\" class=\"data row1 col9\" >high-risk launch or low confidence in routing/model mix</td>\n    </tr>\n    <tr>\n      <th id=\"T_681e7_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n      <td id=\"T_681e7_row2_col0\" class=\"data row2 col0\" >Routed, no cache</td>\n      <td id=\"T_681e7_row2_col1\" class=\"data row2 col1\" >nano triage, mini routine, gpt-5.4 high risk</td>\n      <td id=\"T_681e7_row2_col2\" class=\"data row2 col2\" >allowed tools by routed path</td>\n      <td id=\"T_681e7_row2_col3\" class=\"data row2 col3\" >none</td>\n      <td id=\"T_681e7_row2_col4\" class=\"data row2 col4\" >some follow-up still synchronous</td>\n      <td id=\"T_681e7_row2_col5\" class=\"data row2 col5\" >0.98</td>\n      <td id=\"T_681e7_row2_col6\" class=\"data row2 col6\" >100%</td>\n      <td id=\"T_681e7_row2_col7\" class=\"data row2 col7\" >1.87</td>\n      <td id=\"T_681e7_row2_col8\" class=\"data row2 col8\" >$302</td>\n      <td id=\"T_681e7_row2_col9\" class=\"data row2 col9\" >mixed ticket queues with moderate repeat traffic</td>\n    </tr>\n    <tr>\n      <th id=\"T_681e7_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n      <td id=\"T_681e7_row3_col0\" class=\"data row3 col0\" >Routed split, no cache</td>\n      <td id=\"T_681e7_row3_col1\" class=\"data row3 col1\" >nano triage/tags, mini routine, gpt-5.4 high risk</td>\n      <td id=\"T_681e7_row3_col2\" class=\"data row3 col2\" >allowed tools by routed path</td>\n      <td id=\"T_681e7_row3_col3\" class=\"data row3 col3\" >none</td>\n      <td id=\"T_681e7_row3_col4\" class=\"data row3 col4\" >customer path sync, QA/tags/reporting async</td>\n      <td id=\"T_681e7_row3_col5\" class=\"data row3 col5\" >0.98</td>\n      <td id=\"T_681e7_row3_col6\" class=\"data row3 col6\" >100%</td>\n      <td id=\"T_681e7_row3_col7\" class=\"data row3 col7\" >1.57</td>\n      <td id=\"T_681e7_row3_col8\" class=\"data row3 col8\" >$285</td>\n      <td id=\"T_681e7_row3_col9\" class=\"data row3 col9\" >low-repeat queues that still need async follow-up work</td>\n    </tr>\n    <tr>\n      <th id=\"T_681e7_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n      <td id=\"T_681e7_row4_col0\" class=\"data row4 col0\" >Routed + cache</td>\n      <td id=\"T_681e7_row4_col1\" class=\"data row4 col1\" >nano triage, mini routine, gpt-5.4 high risk</td>\n      <td id=\"T_681e7_row4_col2\" class=\"data row4 col2\" >stable full tool list plus allowed_tools</td>\n      <td id=\"T_681e7_row4_col3\" class=\"data row4 col3\" >stable playbook prefix</td>\n      <td id=\"T_681e7_row4_col4\" class=\"data row4 col4\" >some follow-up still synchronous</td>\n      <td id=\"T_681e7_row4_col5\" class=\"data row4 col5\" >0.98</td>\n      <td id=\"T_681e7_row4_col6\" class=\"data row4 col6\" >100%</td>\n      <td id=\"T_681e7_row4_col7\" class=\"data row4 col7\" >1.85</td>\n      <td id=\"T_681e7_row4_col8\" class=\"data row4 col8\" >$244</td>\n      <td id=\"T_681e7_row4_col9\" class=\"data row4 col9\" >high-volume repeated workflows with good cache locality</td>\n    </tr>\n    <tr>\n      <th id=\"T_681e7_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n      <td id=\"T_681e7_row5_col0\" class=\"data row5 col0\" >Balanced split workflow</td>\n      <td id=\"T_681e7_row5_col1\" class=\"data row5 col1\" >nano triage/tags, mini routine, gpt-5.4 high risk</td>\n      <td id=\"T_681e7_row5_col2\" class=\"data row5 col2\" >stable full tool list plus allowed_tools</td>\n      <td id=\"T_681e7_row5_col3\" class=\"data row5 col3\" >stable playbook prefix</td>\n      <td id=\"T_681e7_row5_col4\" class=\"data row5 col4\" >customer path sync, QA/tags/reporting async</td>\n      <td id=\"T_681e7_row5_col5\" class=\"data row5 col5\" >0.98</td>\n      <td id=\"T_681e7_row5_col6\" class=\"data row5 col6\" >100%</td>\n      <td id=\"T_681e7_row5_col7\" class=\"data row5 col7\" >1.41</td>\n      <td id=\"T_681e7_row5_col8\" class=\"data row5 col8\" >$204</td>\n      <td id=\"T_681e7_row5_col9\" class=\"data row5 col9\" >most mature repeated-workflow support deployments</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "from scenarios import (\n",
    "    ARCHITECTURE_OPTIONS,\n",
    "    OPERATING_SCENARIOS,\n",
    "    architecture_metrics,\n",
    "    scenario_fit_score,\n",
    ")\n",
    "\n",
    "architecture_rows = [\n",
    "    {\"architecture\": key, **option, **architecture_metrics(key, summary)}\n",
    "    for key, option in ARCHITECTURE_OPTIONS.items()\n",
    "]\n",
    "architecture_df = pd.DataFrame(architecture_rows)\n",
    "print(\"Table: Candidate architecture combinations (mock eval set + deterministic dry-run metrics)\")\n",
    "display(\n",
    "    architecture_df[\n",
    "        [\n",
    "            \"label\",\n",
    "            \"models\",\n",
    "            \"tools\",\n",
    "            \"cache\",\n",
    "            \"workflow\",\n",
    "            \"quality\",\n",
    "            \"policy_compliance\",\n",
    "            \"p50_latency_s\",\n",
    "            \"monthly_cost_at_100k_tickets\",\n",
    "            \"best_for\",\n",
    "        ]\n",
    "    ].style.format(\n",
    "        {\n",
    "            \"quality\": \"{:.2f}\",\n",
    "            \"policy_compliance\": \"{:.0%}\",\n",
    "            \"p50_latency_s\": \"{:.2f}\",\n",
    "            \"monthly_cost_at_100k_tickets\": \"${:,.0f}\",\n",
    "        }\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc964e13",
   "metadata": {},
   "source": [
    "### Scenario sweet spots\n",
    "\n",
    "This table maps common real-world operating scenarios to the best-scoring architecture combination. The scenario constraints are mocked for demonstration, and the architecture metrics come from the dry-run simulation above. In production, replace the constraints with your support SLAs, budget, policy-risk thresholds, and observed cache hit rates.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e6b44a9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.342594Z",
     "iopub.status.busy": "2026-04-30T18:28:05.342526Z",
     "iopub.status.idle": "2026-04-30T18:28:05.360509Z",
     "shell.execute_reply": "2026-04-30T18:28:05.360159Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Table: Recommended sweet spot by scenario (mock constraints + dry-run architecture metrics)\n"
    },
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117cff9e0>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_fa37e\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_fa37e_level0_col0\" class=\"col_heading level0 col0\" >scenario</th>\n      <th id=\"T_fa37e_level0_col1\" class=\"col_heading level0 col1\" >description</th>\n      <th id=\"T_fa37e_level0_col2\" class=\"col_heading level0 col2\" >label</th>\n      <th id=\"T_fa37e_level0_col3\" class=\"col_heading level0 col3\" >score</th>\n      <th id=\"T_fa37e_level0_col4\" class=\"col_heading level0 col4\" >quality</th>\n      <th id=\"T_fa37e_level0_col5\" class=\"col_heading level0 col5\" >quality_floor</th>\n      <th id=\"T_fa37e_level0_col6\" class=\"col_heading level0 col6\" >policy_compliance</th>\n      <th id=\"T_fa37e_level0_col7\" class=\"col_heading level0 col7\" >policy_floor</th>\n      <th id=\"T_fa37e_level0_col8\" class=\"col_heading level0 col8\" >p50_latency_s</th>\n      <th id=\"T_fa37e_level0_col9\" class=\"col_heading level0 col9\" >p50_latency_target_s</th>\n      <th id=\"T_fa37e_level0_col10\" class=\"col_heading level0 col10\" >monthly_cost_at_100k_tickets</th>\n      <th id=\"T_fa37e_level0_col11\" class=\"col_heading level0 col11\" >monthly_budget_100k_usd</th>\n      <th id=\"T_fa37e_level0_col12\" class=\"col_heading level0 col12\" >cache_locality</th>\n      <th id=\"T_fa37e_level0_col13\" class=\"col_heading level0 col13\" >failed_constraints</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_fa37e_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n      <td id=\"T_fa37e_row0_col0\" class=\"data row0 col0\" >Account and billing sensitive queue</td>\n      <td id=\"T_fa37e_row0_col1\" class=\"data row0 col1\" >Risky account recovery and duplicate-charge workflows dominate.</td>\n      <td id=\"T_fa37e_row0_col2\" class=\"data row0 col2\" >Routed split, no cache</td>\n      <td id=\"T_fa37e_row0_col3\" class=\"data row0 col3\" >13</td>\n      <td id=\"T_fa37e_row0_col4\" class=\"data row0 col4\" >0.98</td>\n      <td id=\"T_fa37e_row0_col5\" class=\"data row0 col5\" >0.98</td>\n      <td id=\"T_fa37e_row0_col6\" class=\"data row0 col6\" >100%</td>\n      <td id=\"T_fa37e_row0_col7\" class=\"data row0 col7\" >100%</td>\n      <td id=\"T_fa37e_row0_col8\" class=\"data row0 col8\" >1.57</td>\n      <td id=\"T_fa37e_row0_col9\" class=\"data row0 col9\" >2.80</td>\n      <td id=\"T_fa37e_row0_col10\" class=\"data row0 col10\" >$285</td>\n      <td id=\"T_fa37e_row0_col11\" class=\"data row0 col11\" >$750</td>\n      <td id=\"T_fa37e_row0_col12\" class=\"data row0 col12\" >medium</td>\n      <td id=\"T_fa37e_row0_col13\" class=\"data row0 col13\" >none</td>\n    </tr>\n    <tr>\n      <th id=\"T_fa37e_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n      <td id=\"T_fa37e_row1_col0\" class=\"data row1 col0\" >Early pilot</td>\n      <td id=\"T_fa37e_row1_col1\" class=\"data row1 col1\" >Low volume, quality learning matters more than unit cost.</td>\n      <td id=\"T_fa37e_row1_col2\" class=\"data row1 col2\" >Routed, no cache</td>\n      <td id=\"T_fa37e_row1_col3\" class=\"data row1 col3\" >13</td>\n      <td id=\"T_fa37e_row1_col4\" class=\"data row1 col4\" >0.98</td>\n      <td id=\"T_fa37e_row1_col5\" class=\"data row1 col5\" >0.94</td>\n      <td id=\"T_fa37e_row1_col6\" class=\"data row1 col6\" >100%</td>\n      <td id=\"T_fa37e_row1_col7\" class=\"data row1 col7\" >98%</td>\n      <td id=\"T_fa37e_row1_col8\" class=\"data row1 col8\" >1.87</td>\n      <td id=\"T_fa37e_row1_col9\" class=\"data row1 col9\" >3.00</td>\n      <td id=\"T_fa37e_row1_col10\" class=\"data row1 col10\" >$302</td>\n      <td id=\"T_fa37e_row1_col11\" class=\"data row1 col11\" >$800</td>\n      <td id=\"T_fa37e_row1_col12\" class=\"data row1 col12\" >low</td>\n      <td id=\"T_fa37e_row1_col13\" class=\"data row1 col13\" >none</td>\n    </tr>\n    <tr>\n      <th id=\"T_fa37e_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n      <td id=\"T_fa37e_row2_col0\" class=\"data row2 col0\" >High-volume routine ecommerce</td>\n      <td id=\"T_fa37e_row2_col1\" class=\"data row2 col1\" >Many repeated order, return, and damage workflows.</td>\n      <td id=\"T_fa37e_row2_col2\" class=\"data row2 col2\" >Balanced split workflow</td>\n      <td id=\"T_fa37e_row2_col3\" class=\"data row2 col3\" >13</td>\n      <td id=\"T_fa37e_row2_col4\" class=\"data row2 col4\" >0.98</td>\n      <td id=\"T_fa37e_row2_col5\" class=\"data row2 col5\" >0.96</td>\n      <td id=\"T_fa37e_row2_col6\" class=\"data row2 col6\" >100%</td>\n      <td id=\"T_fa37e_row2_col7\" class=\"data row2 col7\" >99%</td>\n      <td id=\"T_fa37e_row2_col8\" class=\"data row2 col8\" >1.41</td>\n      <td id=\"T_fa37e_row2_col9\" class=\"data row2 col9\" >2.00</td>\n      <td id=\"T_fa37e_row2_col10\" class=\"data row2 col10\" >$204</td>\n      <td id=\"T_fa37e_row2_col11\" class=\"data row2 col11\" >$300</td>\n      <td id=\"T_fa37e_row2_col12\" class=\"data row2 col12\" >high</td>\n      <td id=\"T_fa37e_row2_col13\" class=\"data row2 col13\" >none</td>\n    </tr>\n    <tr>\n      <th id=\"T_fa37e_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n      <td id=\"T_fa37e_row3_col0\" class=\"data row3 col0\" >Low-repeat long tail</td>\n      <td id=\"T_fa37e_row3_col1\" class=\"data row3 col1\" >Many rare ticket types; cache hit rate is uncertain.</td>\n      <td id=\"T_fa37e_row3_col2\" class=\"data row3 col2\" >Routed split, no cache</td>\n      <td id=\"T_fa37e_row3_col3\" class=\"data row3 col3\" >13</td>\n      <td id=\"T_fa37e_row3_col4\" class=\"data row3 col4\" >0.98</td>\n      <td id=\"T_fa37e_row3_col5\" class=\"data row3 col5\" >0.96</td>\n      <td id=\"T_fa37e_row3_col6\" class=\"data row3 col6\" >100%</td>\n      <td id=\"T_fa37e_row3_col7\" class=\"data row3 col7\" >99%</td>\n      <td id=\"T_fa37e_row3_col8\" class=\"data row3 col8\" >1.57</td>\n      <td id=\"T_fa37e_row3_col9\" class=\"data row3 col9\" >2.50</td>\n      <td id=\"T_fa37e_row3_col10\" class=\"data row3 col10\" >$285</td>\n      <td id=\"T_fa37e_row3_col11\" class=\"data row3 col11\" >$450</td>\n      <td id=\"T_fa37e_row3_col12\" class=\"data row3 col12\" >low</td>\n      <td id=\"T_fa37e_row3_col13\" class=\"data row3 col13\" >none</td>\n    </tr>\n    <tr>\n      <th id=\"T_fa37e_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n      <td id=\"T_fa37e_row4_col0\" class=\"data row4 col0\" >Peak sale burst</td>\n      <td id=\"T_fa37e_row4_col1\" class=\"data row4 col1\" >Latency and cost matter during temporary traffic spikes.</td>\n      <td id=\"T_fa37e_row4_col2\" class=\"data row4 col2\" >Balanced split workflow</td>\n      <td id=\"T_fa37e_row4_col3\" class=\"data row4 col3\" >13</td>\n      <td id=\"T_fa37e_row4_col4\" class=\"data row4 col4\" >0.98</td>\n      <td id=\"T_fa37e_row4_col5\" class=\"data row4 col5\" >0.95</td>\n      <td id=\"T_fa37e_row4_col6\" class=\"data row4 col6\" >100%</td>\n      <td id=\"T_fa37e_row4_col7\" class=\"data row4 col7\" >99%</td>\n      <td id=\"T_fa37e_row4_col8\" class=\"data row4 col8\" >1.41</td>\n      <td id=\"T_fa37e_row4_col9\" class=\"data row4 col9\" >1.80</td>\n      <td id=\"T_fa37e_row4_col10\" class=\"data row4 col10\" >$204</td>\n      <td id=\"T_fa37e_row4_col11\" class=\"data row4 col11\" >$250</td>\n      <td id=\"T_fa37e_row4_col12\" class=\"data row4 col12\" >high</td>\n      <td id=\"T_fa37e_row4_col13\" class=\"data row4 col13\" >none</td>\n    </tr>\n    <tr>\n      <th id=\"T_fa37e_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n      <td id=\"T_fa37e_row5_col0\" class=\"data row5 col0\" >Premium support</td>\n      <td id=\"T_fa37e_row5_col1\" class=\"data row5 col1\" >Higher customer value, lower tolerance for wrong actions.</td>\n      <td id=\"T_fa37e_row5_col2\" class=\"data row5 col2\" >Routed split, no cache</td>\n      <td id=\"T_fa37e_row5_col3\" class=\"data row5 col3\" >13</td>\n      <td id=\"T_fa37e_row5_col4\" class=\"data row5 col4\" >0.98</td>\n      <td id=\"T_fa37e_row5_col5\" class=\"data row5 col5\" >0.98</td>\n      <td id=\"T_fa37e_row5_col6\" class=\"data row5 col6\" >100%</td>\n      <td id=\"T_fa37e_row5_col7\" class=\"data row5 col7\" >100%</td>\n      <td id=\"T_fa37e_row5_col8\" class=\"data row5 col8\" >1.57</td>\n      <td id=\"T_fa37e_row5_col9\" class=\"data row5 col9\" >2.50</td>\n      <td id=\"T_fa37e_row5_col10\" class=\"data row5 col10\" >$285</td>\n      <td id=\"T_fa37e_row5_col11\" class=\"data row5 col11\" >$650</td>\n      <td id=\"T_fa37e_row5_col12\" class=\"data row5 col12\" >medium</td>\n      <td id=\"T_fa37e_row5_col13\" class=\"data row5 col13\" >none</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "fit_rows = [\n",
    "    scenario_fit_score(scenario, option_key, summary)\n",
    "    for scenario in OPERATING_SCENARIOS\n",
    "    for option_key in ARCHITECTURE_OPTIONS\n",
    "]\n",
    "fit_df = pd.DataFrame(fit_rows)\n",
    "\n",
    "best_fit = (\n",
    "    fit_df.sort_values([\"scenario\", \"score\", \"monthly_cost_at_100k_tickets\"], ascending=[True, False, True])\n",
    "    .groupby(\"scenario\", sort=False)\n",
    "    .head(1)\n",
    "    .reset_index(drop=True)\n",
    ")\n",
    "\n",
    "scenario_context = pd.DataFrame(OPERATING_SCENARIOS)[\n",
    "    [\n",
    "        \"scenario\",\n",
    "        \"description\",\n",
    "        \"quality_floor\",\n",
    "        \"policy_floor\",\n",
    "        \"p50_latency_target_s\",\n",
    "        \"monthly_budget_100k_usd\",\n",
    "        \"needs_async\",\n",
    "        \"cache_locality\",\n",
    "    ]\n",
    "]\n",
    "\n",
    "best_fit_view = best_fit.merge(scenario_context, on=\"scenario\")\n",
    "\n",
    "print(\"Table: Recommended sweet spot by scenario (mock constraints + dry-run architecture metrics)\")\n",
    "display(\n",
    "    best_fit_view[\n",
    "        [\n",
    "            \"scenario\",\n",
    "            \"description\",\n",
    "            \"label\",\n",
    "            \"score\",\n",
    "            \"quality\",\n",
    "            \"quality_floor\",\n",
    "            \"policy_compliance\",\n",
    "            \"policy_floor\",\n",
    "            \"p50_latency_s\",\n",
    "            \"p50_latency_target_s\",\n",
    "            \"monthly_cost_at_100k_tickets\",\n",
    "            \"monthly_budget_100k_usd\",\n",
    "            \"cache_locality\",\n",
    "            \"failed_constraints\",\n",
    "        ]\n",
    "    ].style.format(\n",
    "        {\n",
    "            \"quality\": \"{:.2f}\",\n",
    "            \"quality_floor\": \"{:.2f}\",\n",
    "            \"policy_compliance\": \"{:.0%}\",\n",
    "            \"policy_floor\": \"{:.0%}\",\n",
    "            \"p50_latency_s\": \"{:.2f}\",\n",
    "            \"p50_latency_target_s\": \"{:.2f}\",\n",
    "            \"monthly_cost_at_100k_tickets\": \"${:,.0f}\",\n",
    "            \"monthly_budget_100k_usd\": \"${:,.0f}\",\n",
    "        }\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9b5773",
   "metadata": {},
   "source": [
    "### Full combination map for one scenario\n",
    "\n",
    "This table shows all architecture options for one mocked scenario: `Low-repeat long tail`. It is included to make the tradeoff visible rather than hiding everything behind the single best pick. The numbers are still simulated; the point is to show why cache-heavy designs are less attractive when cache locality is low.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ade143a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.361551Z",
     "iopub.status.busy": "2026-04-30T18:28:05.361484Z",
     "iopub.status.idle": "2026-04-30T18:28:05.365384Z",
     "shell.execute_reply": "2026-04-30T18:28:05.364956Z"
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Table: Full architecture ranking for Low-repeat long tail (mock scenario + dry-run metrics)\n"
    },
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d34b30>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_26e63\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_26e63_level0_col0\" class=\"col_heading level0 col0\" >label</th>\n      <th id=\"T_26e63_level0_col1\" class=\"col_heading level0 col1\" >score</th>\n      <th id=\"T_26e63_level0_col2\" class=\"col_heading level0 col2\" >quality</th>\n      <th id=\"T_26e63_level0_col3\" class=\"col_heading level0 col3\" >policy_compliance</th>\n      <th id=\"T_26e63_level0_col4\" class=\"col_heading level0 col4\" >p50_latency_s</th>\n      <th id=\"T_26e63_level0_col5\" class=\"col_heading level0 col5\" >monthly_cost_at_100k_tickets</th>\n      <th id=\"T_26e63_level0_col6\" class=\"col_heading level0 col6\" >failed_constraints</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_26e63_level0_row0\" class=\"row_heading level0 row0\" >27</th>\n      <td id=\"T_26e63_row0_col0\" class=\"data row0 col0\" >Routed split, no cache</td>\n      <td id=\"T_26e63_row0_col1\" class=\"data row0 col1\" >13</td>\n      <td id=\"T_26e63_row0_col2\" class=\"data row0 col2\" >0.98</td>\n      <td id=\"T_26e63_row0_col3\" class=\"data row0 col3\" >100%</td>\n      <td id=\"T_26e63_row0_col4\" class=\"data row0 col4\" >1.57</td>\n      <td id=\"T_26e63_row0_col5\" class=\"data row0 col5\" >$285</td>\n      <td id=\"T_26e63_row0_col6\" class=\"data row0 col6\" >none</td>\n    </tr>\n    <tr>\n      <th id=\"T_26e63_level0_row1\" class=\"row_heading level0 row1\" >26</th>\n      <td id=\"T_26e63_row1_col0\" class=\"data row1 col0\" >Routed, no cache</td>\n      <td id=\"T_26e63_row1_col1\" class=\"data row1 col1\" >10</td>\n      <td id=\"T_26e63_row1_col2\" class=\"data row1 col2\" >0.98</td>\n      <td id=\"T_26e63_row1_col3\" class=\"data row1 col3\" >100%</td>\n      <td id=\"T_26e63_row1_col4\" class=\"data row1 col4\" >1.87</td>\n      <td id=\"T_26e63_row1_col5\" class=\"data row1 col5\" >$302</td>\n      <td id=\"T_26e63_row1_col6\" class=\"data row1 col6\" >async split</td>\n    </tr>\n    <tr>\n      <th id=\"T_26e63_level0_row2\" class=\"row_heading level0 row2\" >29</th>\n      <td id=\"T_26e63_row2_col0\" class=\"data row2 col0\" >Balanced split workflow</td>\n      <td id=\"T_26e63_row2_col1\" class=\"data row2 col1\" >9</td>\n      <td id=\"T_26e63_row2_col2\" class=\"data row2 col2\" >0.98</td>\n      <td id=\"T_26e63_row2_col3\" class=\"data row2 col3\" >100%</td>\n      <td id=\"T_26e63_row2_col4\" class=\"data row2 col4\" >1.41</td>\n      <td id=\"T_26e63_row2_col5\" class=\"data row2 col5\" >$204</td>\n      <td id=\"T_26e63_row2_col6\" class=\"data row2 col6\" >cache locality</td>\n    </tr>\n    <tr>\n      <th id=\"T_26e63_level0_row3\" class=\"row_heading level0 row3\" >28</th>\n      <td id=\"T_26e63_row3_col0\" class=\"data row3 col0\" >Routed + cache</td>\n      <td id=\"T_26e63_row3_col1\" class=\"data row3 col1\" >6</td>\n      <td id=\"T_26e63_row3_col2\" class=\"data row3 col2\" >0.98</td>\n      <td id=\"T_26e63_row3_col3\" class=\"data row3 col3\" >100%</td>\n      <td id=\"T_26e63_row3_col4\" class=\"data row3 col4\" >1.85</td>\n      <td id=\"T_26e63_row3_col5\" class=\"data row3 col5\" >$244</td>\n      <td id=\"T_26e63_row3_col6\" class=\"data row3 col6\" >async split, cache locality</td>\n    </tr>\n    <tr>\n      <th id=\"T_26e63_level0_row4\" class=\"row_heading level0 row4\" >25</th>\n      <td id=\"T_26e63_row4_col0\" class=\"data row4 col0\" >Controlled full model</td>\n      <td id=\"T_26e63_row4_col1\" class=\"data row4 col1\" >5</td>\n      <td id=\"T_26e63_row4_col2\" class=\"data row4 col2\" >0.98</td>\n      <td id=\"T_26e63_row4_col3\" class=\"data row4 col3\" >100%</td>\n      <td id=\"T_26e63_row4_col4\" class=\"data row4 col4\" >2.32</td>\n      <td id=\"T_26e63_row4_col5\" class=\"data row4 col5\" >$512</td>\n      <td id=\"T_26e63_row4_col6\" class=\"data row4 col6\" >budget, async split</td>\n    </tr>\n    <tr>\n      <th id=\"T_26e63_level0_row5\" class=\"row_heading level0 row5\" >24</th>\n      <td id=\"T_26e63_row5_col0\" class=\"data row5 col0\" >One broad agent</td>\n      <td id=\"T_26e63_row5_col1\" class=\"data row5 col1\" >-14</td>\n      <td id=\"T_26e63_row5_col2\" class=\"data row5 col2\" >0.51</td>\n      <td id=\"T_26e63_row5_col3\" class=\"data row5 col3\" >10%</td>\n      <td id=\"T_26e63_row5_col4\" class=\"data row5 col4\" >4.88</td>\n      <td id=\"T_26e63_row5_col5\" class=\"data row5 col5\" >$3,813</td>\n      <td id=\"T_26e63_row5_col6\" class=\"data row5 col6\" >quality, policy, latency, budget, async split</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "# Show the full combination map for one scenario so tradeoffs are visible, not hidden behind the best pick.\n",
    "scenario_to_inspect = \"Low-repeat long tail\"\n",
    "combo_map = fit_df[fit_df[\"scenario\"] == scenario_to_inspect].sort_values(\"score\", ascending=False)\n",
    "\n",
    "print(f\"Table: Full architecture ranking for {scenario_to_inspect} (mock scenario + dry-run metrics)\")\n",
    "display(\n",
    "    combo_map[\n",
    "        [\n",
    "            \"label\",\n",
    "            \"score\",\n",
    "            \"quality\",\n",
    "            \"policy_compliance\",\n",
    "            \"p50_latency_s\",\n",
    "            \"monthly_cost_at_100k_tickets\",\n",
    "            \"failed_constraints\",\n",
    "        ]\n",
    "    ].style.format(\n",
    "        {\n",
    "            \"quality\": \"{:.2f}\",\n",
    "            \"policy_compliance\": \"{:.0%}\",\n",
    "            \"p50_latency_s\": \"{:.2f}\",\n",
    "            \"monthly_cost_at_100k_tickets\": \"${:,.0f}\",\n",
    "        }\n",
    "    )\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b5ac256",
   "metadata": {},
   "source": [
    "In the mock scenarios, repeated workflows favor a shared cache prefix and asynchronous follow-up. Low-repeat queues may favor the routed split without caching, while an early pilot may justify a full model until routing is reliable.\n",
    "\n",
    "Treat these rankings as a design exercise. They include hand-set constraints and scoring bonuses, so a high score is not proof that an architecture meets every requirement. Check `failed_constraints` and enforce quality and policy gates before selecting a production configuration.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9167e9af",
   "metadata": {},
   "source": [
    "## Monitoring, evals, and guardrails\n",
    "\n",
    "Once the optimized workflow is in production, keep a recurring eval loop. The objective is not to minimize tokens in isolation. It is to resolve customer issues correctly, safely, and quickly at the lowest total cost per successful outcome.\n",
    "\n",
    "### Measure task efficiency, not just token efficiency\n",
    "\n",
    "Token counts are useful diagnostics, but they do not tell you whether the customer's problem was solved. A cheaper model that requires repeated attempts, unnecessary tool calls, or human correction can cost more per resolved issue than a stronger model that completes the task correctly on its first attempt.\n",
    "\n",
    "OpenAI's guidance recommends measuring the complete cost of reaching an acceptable outcome, including \"model and tool usage, attempts, completion rate, latency, and human review.\" For customer support, that accepted outcome may be a resolved case. See [How to manage AI investments in the agentic era](https://openai.com/index/managing-ai-investments-in-agentic-era/) and [A scorecard for the AI age](https://openai.com/index/a-scorecard-for-the-ai-age/).\n",
    "\n",
    "A useful operational formula is:\n",
    "\n",
    "`blended cost per verified resolution = total model, tool, infrastructure, retry, human-review, escalation, and rework costs / verified customer issues resolved`\n",
    "\n",
    "The numerator must include spending on unsuccessful attempts, not only the traces that eventually passed. Track autonomous resolutions separately from human-assisted resolutions so an apparent reduction in agent cost does not hide a transfer of work to the support team.\n",
    "\n",
    "For example, a workflow that costs 0.02 USD per ticket and resolves 50% of tickets costs 0.04 USD per successful resolution. A workflow that costs 0.03 USD per ticket and resolves 90% costs approximately 0.033 USD per successful resolution. The second workflow costs more per attempt but less per successful outcome. These figures are illustrative and exclude human-support costs.\n",
    "\n",
    "### Define success before optimizing\n",
    "\n",
    "A successful response uses the right account, order, and policy facts and gives an accurate next step. Required tools must succeed with valid arguments, and promised actions must be completed or clearly pending. Policy and authorization checks determine which cases can be resolved automatically and which require escalation.\n",
    "\n",
    "A policy-required escalation can be a successful handling outcome, but it is not an autonomous resolution. Similarly, opening a case or requesting a photo is not proof that the customer's underlying issue was resolved. Keep these outcomes separate when calculating first-contact resolution and automation rates.\n",
    "\n",
    "### Track the complete support workflow\n",
    "\n",
    "Monitor verified resolutions separately for autonomous and human-assisted cases, including repeat contacts and reopened cases. Pair those outcomes with policy and escalation accuracy, total cost per verified resolution, and customer-facing p50/p95 latency.\n",
    "\n",
    "Use model calls, tool failures, retries, token usage, and routing decisions to explain changes in those outcomes. Segment results by intent, risk, language, region, customer tier, and model route so an average does not conceal a regression.\n",
    "\n",
    "Inspect the full execution trajectory, not only the final answer. OpenAI's [agent evaluation guidance](https://developers.openai.com/api/docs/guides/agent-evals) describes traces that capture model calls, tool calls, guardrails, and handoffs, making it possible to identify unnecessary loops, incorrect actions, and routing failures that a polished response can conceal.\n",
    "\n",
    "Compare workflow variants on the same representative ticket distribution, including difficult and policy-sensitive cases. Treat policy compliance, action correctness, security, and escalation accuracy as hard gates before comparing cost or latency. Refresh the dataset with production failures and rerun evaluations when prompts, models, tools, routing, or policies change. See [Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices).\n",
    "\n",
    "Guardrail failure modes that can look efficient while creating downstream risk: tool timeouts, empty or oversized tool payloads, duplicate tool loops, unsafe account-access actions, skipped required verification, and refund promises made before eligibility or completion is confirmed.\n",
    "\n",
    "**Demo limitation:** This notebook directly models tokens, estimated cost, tool usage, latency, action accuracy, policy compliance, and escalation behavior. True first-contact resolution, reopened cases, retry history, completed downstream outcomes, and human-handling costs require production support-system and trace data. Do not infer those metrics from the dry-run simulation alone.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fda97c92",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.366404Z",
     "iopub.status.busy": "2026-04-30T18:28:05.366344Z",
     "iopub.status.idle": "2026-04-30T18:28:05.376107Z",
     "shell.execute_reply": "2026-04-30T18:28:05.375558Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x11704a150>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_75622\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_75622_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_75622_level0_col1\" class=\"col_heading level0 col1\" >pass_rate</th>\n      <th id=\"T_75622_level0_col2\" class=\"col_heading level0 col2\" >failures</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_75622_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n      <td id=\"T_75622_row0_col0\" class=\"data row0 col0\" >Bad baseline</td>\n      <td id=\"T_75622_row0_col1\" class=\"data row0 col1\" >0%</td>\n      <td id=\"T_75622_row0_col2\" class=\"data row0 col2\" >10</td>\n    </tr>\n    <tr>\n      <th id=\"T_75622_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n      <td id=\"T_75622_row1_col0\" class=\"data row1 col0\" >Round 1: controls</td>\n      <td id=\"T_75622_row1_col1\" class=\"data row1 col1\" >100%</td>\n      <td id=\"T_75622_row1_col2\" class=\"data row1 col2\" >0</td>\n    </tr>\n    <tr>\n      <th id=\"T_75622_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n      <td id=\"T_75622_row2_col0\" class=\"data row2 col0\" >Round 2: routing</td>\n      <td id=\"T_75622_row2_col1\" class=\"data row2 col1\" >100%</td>\n      <td id=\"T_75622_row2_col2\" class=\"data row2 col2\" >0</td>\n    </tr>\n    <tr>\n      <th id=\"T_75622_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n      <td id=\"T_75622_row3_col0\" class=\"data row3 col0\" >Round 3: caching</td>\n      <td id=\"T_75622_row3_col1\" class=\"data row3 col1\" >100%</td>\n      <td id=\"T_75622_row3_col2\" class=\"data row3 col2\" >0</td>\n    </tr>\n    <tr>\n      <th id=\"T_75622_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n      <td id=\"T_75622_row4_col0\" class=\"data row4 col0\" >Round 4: split workflow</td>\n      <td id=\"T_75622_row4_col1\" class=\"data row4 col1\" >100%</td>\n      <td id=\"T_75622_row4_col2\" class=\"data row4 col2\" >0</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    },
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "  variant_label ticket_id                                           failures  passed\n0  Bad baseline     T-001  too_many_unnecessary_tools, missing_required_r...   False\n1  Bad baseline     T-002  too_many_unnecessary_tools, policy_or_action_m...   False\n2  Bad baseline     T-003  missing_required_response_content, policy_or_a...   False\n3  Bad baseline     T-004                           customer_answer_too_long   False\n4  Bad baseline     T-005  missing_required_response_content, policy_or_a...   False\n5  Bad baseline     T-006  missing_required_response_content, policy_or_a...   False\n6  Bad baseline     T-007  too_many_unnecessary_tools, missing_required_r...   False\n7  Bad baseline     T-008  missing_required_response_content, policy_or_a...   False\n8  Bad baseline     T-009  missing_required_response_content, policy_or_a...   False\n9  Bad baseline     T-010  missing_required_response_content, policy_or_a...   False",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>variant_label</th>\n      <th>ticket_id</th>\n      <th>failures</th>\n      <th>passed</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Bad baseline</td>\n      <td>T-001</td>\n      <td>too_many_unnecessary_tools, missing_required_r...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Bad baseline</td>\n      <td>T-002</td>\n      <td>too_many_unnecessary_tools, policy_or_action_m...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Bad baseline</td>\n      <td>T-003</td>\n      <td>missing_required_response_content, policy_or_a...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Bad baseline</td>\n      <td>T-004</td>\n      <td>customer_answer_too_long</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Bad baseline</td>\n      <td>T-005</td>\n      <td>missing_required_response_content, policy_or_a...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Bad baseline</td>\n      <td>T-006</td>\n      <td>missing_required_response_content, policy_or_a...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Bad baseline</td>\n      <td>T-007</td>\n      <td>too_many_unnecessary_tools, missing_required_r...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Bad baseline</td>\n      <td>T-008</td>\n      <td>missing_required_response_content, policy_or_a...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>Bad baseline</td>\n      <td>T-009</td>\n      <td>missing_required_response_content, policy_or_a...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Bad baseline</td>\n      <td>T-010</td>\n      <td>missing_required_response_content, policy_or_a...</td>\n      <td>False</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     }
    }
   ],
   "source": [
    "from simulation import deterministic_guardrail_check\n",
    "\n",
    "guardrail_rows = []\n",
    "for _, row in traces.iterrows():\n",
    "    ticket = next(t for t in EVAL_SET if t[\"ticket_id\"] == row[\"ticket_id\"])\n",
    "    failures = deterministic_guardrail_check(ticket, row.to_dict())\n",
    "    guardrail_rows.append(\n",
    "        {\n",
    "            \"variant_label\": row[\"variant_label\"],\n",
    "            \"ticket_id\": row[\"ticket_id\"],\n",
    "            \"failures\": \", \".join(failures),\n",
    "            \"passed\": not failures,\n",
    "        }\n",
    "    )\n",
    "\n",
    "guardrails = pd.DataFrame(guardrail_rows)\n",
    "guardrail_summary = guardrails.groupby(\"variant_label\", sort=False).agg(pass_rate=(\"passed\", \"mean\"), failures=(\"passed\", lambda s: (~s).sum())).reset_index()\n",
    "\n",
    "display(guardrail_summary.style.format({\"pass_rate\": \"{:.0%}\"}))\n",
    "display(guardrails[~guardrails[\"passed\"]].head(20))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "answer-judge-rubric",
   "metadata": {},
   "source": [
    "### Optional: judge customer-answer completeness and grounding\n",
    "\n",
    "Did the cheaper workflow preserve an accurate, useful answer? This judge checks one question: given the customer ticket, relevant policy, and recorded tool results, does the answer correctly explain the outcome and next step without unsupported claims?\n",
    "\n",
    "The [judge helper](https://github.com/openai/openai-cookbook/blob/bb95430abb908c1edddc38af5911109ab2ce3987/examples/agent_optimization/live_api.py) returns `passed` and a brief `reason`. It accepts equivalent wording: “Your return qualifies under our 30-day policy” need not contain the fixture's exact phrase “within 30 days.” But “Your refund is on its way” should fail when the recorded tool result only confirms that a review case was opened. Tool results are captured when the tools run, rather than reconstructed from expected actions.\n",
    "\n",
    "Set `RUN_LLM_JUDGE=true` and `OPENAI_API_KEY` before running the setup cell. The code below grades the same recorded answers for every optimization round using a fixed `gpt-5.4-mini` judge and rubric. The judge does not see variant names, agent models, costs, or expected action labels. These are real judge calls over **synthetic agent traces**, so the results assess the canned answers, not model performance. For live answers, call `live_judge_response(live_ticket, live_result[\"response_text\"], live_result[\"tool_results\"], client=judge_client)` after opting in.\n",
    "\n",
    "The table places judge pass rate next to the deterministic pass rate. `both_pass_rate` requires both checks to pass; a judge pass never overrides a deterministic failure. Judge and combined pass rates cover successfully graded traces only, so inspect coverage and errors before comparing variants. Skipped, refused, malformed, or incomplete grades remain unavailable. Evaluation cost is reported separately from agent cost and customer latency; `known_judge_cost_usd` uses returned usage, and `judge_cost_unavailable` flags attempts without cost data.\n",
    "\n",
    "Before using these grades as a release gate, label a small sample yourself, including a valid paraphrase, a missing next step, and an unsupported refund promise. Check agreement and revise the rubric when it disagrees. See [evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "answer-judge-comparison",
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d1cda0>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_363cc\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_363cc_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_363cc_level0_col1\" class=\"col_heading level0 col1\" >tickets</th>\n      <th id=\"T_363cc_level0_col2\" class=\"col_heading level0 col2\" >deterministic_pass_rate</th>\n      <th id=\"T_363cc_level0_col3\" class=\"col_heading level0 col3\" >judge_graded</th>\n      <th id=\"T_363cc_level0_col4\" class=\"col_heading level0 col4\" >judge_coverage</th>\n      <th id=\"T_363cc_level0_col5\" class=\"col_heading level0 col5\" >judge_errors</th>\n      <th id=\"T_363cc_level0_col6\" class=\"col_heading level0 col6\" >judge_pass_rate</th>\n      <th id=\"T_363cc_level0_col7\" class=\"col_heading level0 col7\" >both_pass_rate</th>\n      <th id=\"T_363cc_level0_col8\" class=\"col_heading level0 col8\" >known_judge_cost_usd</th>\n      <th id=\"T_363cc_level0_col9\" class=\"col_heading level0 col9\" >judge_cost_unavailable</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_363cc_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n      <td id=\"T_363cc_row0_col0\" class=\"data row0 col0\" >Bad baseline</td>\n      <td id=\"T_363cc_row0_col1\" class=\"data row0 col1\" >10</td>\n      <td id=\"T_363cc_row0_col2\" class=\"data row0 col2\" >0%</td>\n      <td id=\"T_363cc_row0_col3\" class=\"data row0 col3\" >10</td>\n      <td id=\"T_363cc_row0_col4\" class=\"data row0 col4\" >100%</td>\n      <td id=\"T_363cc_row0_col5\" class=\"data row0 col5\" >0</td>\n      <td id=\"T_363cc_row0_col6\" class=\"data row0 col6\" >0%</td>\n      <td id=\"T_363cc_row0_col7\" class=\"data row0 col7\" >0%</td>\n      <td id=\"T_363cc_row0_col8\" class=\"data row0 col8\" >$0.05637</td>\n      <td id=\"T_363cc_row0_col9\" class=\"data row0 col9\" >0</td>\n    </tr>\n    <tr>\n      <th id=\"T_363cc_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n      <td id=\"T_363cc_row1_col0\" class=\"data row1 col0\" >Round 1: controls</td>\n      <td id=\"T_363cc_row1_col1\" class=\"data row1 col1\" >10</td>\n      <td id=\"T_363cc_row1_col2\" class=\"data row1 col2\" >100%</td>\n      <td id=\"T_363cc_row1_col3\" class=\"data row1 col3\" >10</td>\n      <td id=\"T_363cc_row1_col4\" class=\"data row1 col4\" >100%</td>\n      <td id=\"T_363cc_row1_col5\" class=\"data row1 col5\" >0</td>\n      <td id=\"T_363cc_row1_col6\" class=\"data row1 col6\" >80%</td>\n      <td id=\"T_363cc_row1_col7\" class=\"data row1 col7\" >80%</td>\n      <td id=\"T_363cc_row1_col8\" class=\"data row1 col8\" >$0.01252</td>\n      <td id=\"T_363cc_row1_col9\" class=\"data row1 col9\" >0</td>\n    </tr>\n    <tr>\n      <th id=\"T_363cc_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n      <td id=\"T_363cc_row2_col0\" class=\"data row2 col0\" >Round 2: routing</td>\n      <td id=\"T_363cc_row2_col1\" class=\"data row2 col1\" >10</td>\n      <td id=\"T_363cc_row2_col2\" class=\"data row2 col2\" >100%</td>\n      <td id=\"T_363cc_row2_col3\" class=\"data row2 col3\" >10</td>\n      <td id=\"T_363cc_row2_col4\" class=\"data row2 col4\" >100%</td>\n      <td id=\"T_363cc_row2_col5\" class=\"data row2 col5\" >0</td>\n      <td id=\"T_363cc_row2_col6\" class=\"data row2 col6\" >70%</td>\n      <td id=\"T_363cc_row2_col7\" class=\"data row2 col7\" >70%</td>\n      <td id=\"T_363cc_row2_col8\" class=\"data row2 col8\" >$0.01350</td>\n      <td id=\"T_363cc_row2_col9\" class=\"data row2 col9\" >0</td>\n    </tr>\n    <tr>\n      <th id=\"T_363cc_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n      <td id=\"T_363cc_row3_col0\" class=\"data row3 col0\" >Round 3: caching</td>\n      <td id=\"T_363cc_row3_col1\" class=\"data row3 col1\" >10</td>\n      <td id=\"T_363cc_row3_col2\" class=\"data row3 col2\" >100%</td>\n      <td id=\"T_363cc_row3_col3\" class=\"data row3 col3\" >10</td>\n      <td id=\"T_363cc_row3_col4\" class=\"data row3 col4\" >100%</td>\n      <td id=\"T_363cc_row3_col5\" class=\"data row3 col5\" >0</td>\n      <td id=\"T_363cc_row3_col6\" class=\"data row3 col6\" >90%</td>\n      <td id=\"T_363cc_row3_col7\" class=\"data row3 col7\" >90%</td>\n      <td id=\"T_363cc_row3_col8\" class=\"data row3 col8\" >$0.01279</td>\n      <td id=\"T_363cc_row3_col9\" class=\"data row3 col9\" >0</td>\n    </tr>\n    <tr>\n      <th id=\"T_363cc_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n      <td id=\"T_363cc_row4_col0\" class=\"data row4 col0\" >Round 4: split workflow</td>\n      <td id=\"T_363cc_row4_col1\" class=\"data row4 col1\" >10</td>\n      <td id=\"T_363cc_row4_col2\" class=\"data row4 col2\" >100%</td>\n      <td id=\"T_363cc_row4_col3\" class=\"data row4 col3\" >10</td>\n      <td id=\"T_363cc_row4_col4\" class=\"data row4 col4\" >100%</td>\n      <td id=\"T_363cc_row4_col5\" class=\"data row4 col5\" >0</td>\n      <td id=\"T_363cc_row4_col6\" class=\"data row4 col6\" >80%</td>\n      <td id=\"T_363cc_row4_col7\" class=\"data row4 col7\" >80%</td>\n      <td id=\"T_363cc_row4_col8\" class=\"data row4 col8\" >$0.01285</td>\n      <td id=\"T_363cc_row4_col9\" class=\"data row4 col9\" >0</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    },
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "              variant_label ticket_id  deterministic_passed  passed                                             reason\n0              Bad baseline     T-001                 False   False  The answer does not give the customer the actu...\n1              Bad baseline     T-002                 False   False  The reply does not follow the policy: it shoul...\n2              Bad baseline     T-003                 False   False  The answer is not grounded in the evidence: th...\n3              Bad baseline     T-004                 False   False  The reply does not clearly tell the customer t...\n4              Bad baseline     T-005                 False   False  It does not give the customer the needed accou...\n5              Bad baseline     T-006                 False   False  It does not clearly tell the customer that O-1...\n6              Bad baseline     T-007                 False   False  The reply does not give the customer-facing ne...\n7              Bad baseline     T-008                 False   False  The answer is not grounded in the evidence: it...\n8              Bad baseline     T-009                 False   False  The answer is vague and overly internal. It do...\n9              Bad baseline     T-010                 False   False  It follows the policy direction to escalate, b...\n12        Round 1: controls     T-003                  True   False  The case opening is supported, but the answer ...\n19        Round 1: controls     T-010                  True   False  It correctly says identity must be verified be...\n22         Round 2: routing     T-003                  True   False  The answer overstates the outcome: it only sho...\n24         Round 2: routing     T-005                  True   False  The reply correctly says account details can’t...\n29         Round 2: routing     T-010                  True   False  The reply gives the right general guidance, bu...\n39         Round 3: caching     T-010                  True   False  The reply gives the right general guidance, bu...\n44  Round 4: split workflow     T-005                  True   False  The reply is grounded on the identity check an...\n49  Round 4: split workflow     T-010                  True   False  The answer gives the right general guidance, b...",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>variant_label</th>\n      <th>ticket_id</th>\n      <th>deterministic_passed</th>\n      <th>passed</th>\n      <th>reason</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Bad baseline</td>\n      <td>T-001</td>\n      <td>False</td>\n      <td>False</td>\n      <td>The answer does not give the customer the actu...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Bad baseline</td>\n      <td>T-002</td>\n      <td>False</td>\n      <td>False</td>\n      <td>The reply does not follow the policy: it shoul...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Bad baseline</td>\n      <td>T-003</td>\n      <td>False</td>\n      <td>False</td>\n      <td>The answer is not grounded in the evidence: th...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Bad baseline</td>\n      <td>T-004</td>\n      <td>False</td>\n      <td>False</td>\n      <td>The reply does not clearly tell the customer t...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Bad baseline</td>\n      <td>T-005</td>\n      <td>False</td>\n      <td>False</td>\n      <td>It does not give the customer the needed accou...</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Bad baseline</td>\n      <td>T-006</td>\n      <td>False</td>\n      <td>False</td>\n      <td>It does not clearly tell the customer that O-1...</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Bad baseline</td>\n      <td>T-007</td>\n      <td>False</td>\n      <td>False</td>\n      <td>The reply does not give the customer-facing ne...</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Bad baseline</td>\n      <td>T-008</td>\n      <td>False</td>\n      <td>False</td>\n      <td>The answer is not grounded in the evidence: it...</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>Bad baseline</td>\n      <td>T-009</td>\n      <td>False</td>\n      <td>False</td>\n      <td>The answer is vague and overly internal. It do...</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Bad baseline</td>\n      <td>T-010</td>\n      <td>False</td>\n      <td>False</td>\n      <td>It follows the policy direction to escalate, b...</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>Round 1: controls</td>\n      <td>T-003</td>\n      <td>True</td>\n      <td>False</td>\n      <td>The case opening is supported, but the answer ...</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>Round 1: controls</td>\n      <td>T-010</td>\n      <td>True</td>\n      <td>False</td>\n      <td>It correctly says identity must be verified be...</td>\n    </tr>\n    <tr>\n      <th>22</th>\n      <td>Round 2: routing</td>\n      <td>T-003</td>\n      <td>True</td>\n      <td>False</td>\n      <td>The answer overstates the outcome: it only sho...</td>\n    </tr>\n    <tr>\n      <th>24</th>\n      <td>Round 2: routing</td>\n      <td>T-005</td>\n      <td>True</td>\n      <td>False</td>\n      <td>The reply correctly says account details can’t...</td>\n    </tr>\n    <tr>\n      <th>29</th>\n      <td>Round 2: routing</td>\n      <td>T-010</td>\n      <td>True</td>\n      <td>False</td>\n      <td>The reply gives the right general guidance, bu...</td>\n    </tr>\n    <tr>\n      <th>39</th>\n      <td>Round 3: caching</td>\n      <td>T-010</td>\n      <td>True</td>\n      <td>False</td>\n      <td>The reply gives the right general guidance, bu...</td>\n    </tr>\n    <tr>\n      <th>44</th>\n      <td>Round 4: split workflow</td>\n      <td>T-005</td>\n      <td>True</td>\n      <td>False</td>\n      <td>The reply is grounded on the identity check an...</td>\n    </tr>\n    <tr>\n      <th>49</th>\n      <td>Round 4: split workflow</td>\n      <td>T-010</td>\n      <td>True</td>\n      <td>False</td>\n      <td>The answer gives the right general guidance, b...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     }
    }
   ],
   "source": [
    "from evaluation import evaluate_answer_traces, summarize_answer_evals\n",
    "\n",
    "answer_evals = evaluate_answer_traces(\n",
    "    EVAL_SET, traces.to_dict(\"records\"), client=judge_client\n",
    ")\n",
    "answer_eval_summary = summarize_answer_evals(answer_evals)\n",
    "display(answer_eval_summary.drop(columns=\"variant\").style.format(\n",
    "    {\n",
    "        \"deterministic_pass_rate\": \"{:.0%}\",\n",
    "        \"judge_coverage\": \"{:.0%}\",\n",
    "        \"judge_pass_rate\": \"{:.0%}\",\n",
    "        \"both_pass_rate\": \"{:.0%}\",\n",
    "        \"known_judge_cost_usd\": \"${:.5f}\",\n",
    "    },\n",
    "    na_rep=\"Not available\",\n",
    "))\n",
    "if RUN_LLM_JUDGE:\n",
    "    # Inspect failures, errors, and disagreements with the literal phrase checks.\n",
    "    needs_review = answer_evals[\n",
    "        answer_evals[\"judge_status\"].eq(\"error\") | answer_evals[\"passed\"].eq(False)\n",
    "        | answer_evals[\"passed\"].ne(answer_evals[\"deterministic_passed\"])\n",
    "    ]\n",
    "    display(needs_review[[\"variant_label\", \"ticket_id\", \"deterministic_passed\", \"passed\", \"reason\"]])\n",
    "else:\n",
    "    print(\"Judge not run. Set RUN_LLM_JUDGE=true to grade these saved answers.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d76a2916",
   "metadata": {},
   "source": [
    "## Before and after ticket walkthroughs\n",
    "\n",
    "These examples compare the inefficient baseline with the final optimized path.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92610503",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.377119Z",
     "iopub.status.busy": "2026-04-30T18:28:05.377047Z",
     "iopub.status.idle": "2026-04-30T18:28:05.381571Z",
     "shell.execute_reply": "2026-04-30T18:28:05.381203Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d350a0>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_99ce3\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_99ce3_level0_col0\" class=\"col_heading level0 col0\" >ticket_id</th>\n      <th id=\"T_99ce3_level0_col1\" class=\"col_heading level0 col1\" >variant_label</th>\n      <th id=\"T_99ce3_level0_col2\" class=\"col_heading level0 col2\" >intent</th>\n      <th id=\"T_99ce3_level0_col3\" class=\"col_heading level0 col3\" >risk</th>\n      <th id=\"T_99ce3_level0_col4\" class=\"col_heading level0 col4\" >model</th>\n      <th id=\"T_99ce3_level0_col5\" class=\"col_heading level0 col5\" >tools</th>\n      <th id=\"T_99ce3_level0_col6\" class=\"col_heading level0 col6\" >action</th>\n      <th id=\"T_99ce3_level0_col7\" class=\"col_heading level0 col7\" >policy_compliant</th>\n      <th id=\"T_99ce3_level0_col8\" class=\"col_heading level0 col8\" >sync_tokens</th>\n      <th id=\"T_99ce3_level0_col9\" class=\"col_heading level0 col9\" >total_tokens</th>\n      <th id=\"T_99ce3_level0_col10\" class=\"col_heading level0 col10\" >latency_s</th>\n      <th id=\"T_99ce3_level0_col11\" class=\"col_heading level0 col11\" >cost_usd</th>\n      <th id=\"T_99ce3_level0_col12\" class=\"col_heading level0 col12\" >quality_score</th>\n      <th id=\"T_99ce3_level0_col13\" class=\"col_heading level0 col13\" >response_preview</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_99ce3_level0_row0\" class=\"row_heading level0 row0\" >1</th>\n      <td id=\"T_99ce3_row0_col0\" class=\"data row0 col0\" >T-002</td>\n      <td id=\"T_99ce3_row0_col1\" class=\"data row0 col1\" >Bad baseline</td>\n      <td id=\"T_99ce3_row0_col2\" class=\"data row0 col2\" >damaged_delivery</td>\n      <td id=\"T_99ce3_row0_col3\" class=\"data row0 col3\" >medium</td>\n      <td id=\"T_99ce3_row0_col4\" class=\"data row0 col4\" >gpt-5.4</td>\n      <td id=\"T_99ce3_row0_col5\" class=\"data row0 col5\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_99ce3_row0_col6\" class=\"data row0 col6\" >open_replacement_without_photo</td>\n      <td id=\"T_99ce3_row0_col7\" class=\"data row0 col7\" >False</td>\n      <td id=\"T_99ce3_row0_col8\" class=\"data row0 col8\" >12917</td>\n      <td id=\"T_99ce3_row0_col9\" class=\"data row0 col9\" >12917</td>\n      <td id=\"T_99ce3_row0_col10\" class=\"data row0 col10\" >4.88</td>\n      <td id=\"T_99ce3_row0_col11\" class=\"data row0 col11\" >$0.04034</td>\n      <td id=\"T_99ce3_row0_col12\" class=\"data row0 col12\" >0.65</td>\n      <td id=\"T_99ce3_row0_col13\" class=\"data row0 col13\" >I reviewed your message for ticket T-002 and checked the customer profile, order system, policy library, refund workflow, escalation queue, carrier events, billing signals, and internal audit notes. Based on the availabl</td>\n    </tr>\n    <tr>\n      <th id=\"T_99ce3_level0_row1\" class=\"row_heading level0 row1\" >3</th>\n      <td id=\"T_99ce3_row1_col0\" class=\"data row1 col0\" >T-004</td>\n      <td id=\"T_99ce3_row1_col1\" class=\"data row1 col1\" >Bad baseline</td>\n      <td id=\"T_99ce3_row1_col2\" class=\"data row1 col2\" >billing_issue</td>\n      <td id=\"T_99ce3_row1_col3\" class=\"data row1 col3\" >medium</td>\n      <td id=\"T_99ce3_row1_col4\" class=\"data row1 col4\" >gpt-5.4</td>\n      <td id=\"T_99ce3_row1_col5\" class=\"data row1 col5\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_99ce3_row1_col6\" class=\"data row1 col6\" >escalate_billing_review</td>\n      <td id=\"T_99ce3_row1_col7\" class=\"data row1 col7\" >True</td>\n      <td id=\"T_99ce3_row1_col8\" class=\"data row1 col8\" >12830</td>\n      <td id=\"T_99ce3_row1_col9\" class=\"data row1 col9\" >12830</td>\n      <td id=\"T_99ce3_row1_col10\" class=\"data row1 col10\" >4.87</td>\n      <td id=\"T_99ce3_row1_col11\" class=\"data row1 col11\" >$0.04038</td>\n      <td id=\"T_99ce3_row1_col12\" class=\"data row1 col12\" >0.85</td>\n      <td id=\"T_99ce3_row1_col13\" class=\"data row1 col13\" >I reviewed your message for ticket T-004 and checked the customer profile, order system, policy library, refund workflow, escalation queue, carrier events, billing signals, and internal audit notes. Based on the availabl</td>\n    </tr>\n    <tr>\n      <th id=\"T_99ce3_level0_row2\" class=\"row_heading level0 row2\" >7</th>\n      <td id=\"T_99ce3_row2_col0\" class=\"data row2 col0\" >T-008</td>\n      <td id=\"T_99ce3_row2_col1\" class=\"data row2 col1\" >Bad baseline</td>\n      <td id=\"T_99ce3_row2_col2\" class=\"data row2 col2\" >high_value_damage</td>\n      <td id=\"T_99ce3_row2_col3\" class=\"data row2 col3\" >high</td>\n      <td id=\"T_99ce3_row2_col4\" class=\"data row2 col4\" >gpt-5.4</td>\n      <td id=\"T_99ce3_row2_col5\" class=\"data row2 col5\" >lookup_customer, lookup_order, lookup_policy, create_refund_case, escalate_to_human</td>\n      <td id=\"T_99ce3_row2_col6\" class=\"data row2 col6\" >promise_refund_high_value_damage</td>\n      <td id=\"T_99ce3_row2_col7\" class=\"data row2 col7\" >False</td>\n      <td id=\"T_99ce3_row2_col8\" class=\"data row2 col8\" >12997</td>\n      <td id=\"T_99ce3_row2_col9\" class=\"data row2 col9\" >12997</td>\n      <td id=\"T_99ce3_row2_col10\" class=\"data row2 col10\" >4.91</td>\n      <td id=\"T_99ce3_row2_col11\" class=\"data row2 col11\" >$0.04136</td>\n      <td id=\"T_99ce3_row2_col12\" class=\"data row2 col12\" >0.22</td>\n      <td id=\"T_99ce3_row2_col13\" class=\"data row2 col13\" >I reviewed your message for ticket T-008 and checked the customer profile, order system, policy library, refund workflow, escalation queue, carrier events, billing signals, and internal audit notes. Based on the availabl</td>\n    </tr>\n    <tr>\n      <th id=\"T_99ce3_level0_row3\" class=\"row_heading level0 row3\" >41</th>\n      <td id=\"T_99ce3_row3_col0\" class=\"data row3 col0\" >T-002</td>\n      <td id=\"T_99ce3_row3_col1\" class=\"data row3 col1\" >Round 4: split workflow</td>\n      <td id=\"T_99ce3_row3_col2\" class=\"data row3 col2\" >damaged_delivery</td>\n      <td id=\"T_99ce3_row3_col3\" class=\"data row3 col3\" >medium</td>\n      <td id=\"T_99ce3_row3_col4\" class=\"data row3 col4\" >gpt-5.4-mini</td>\n      <td id=\"T_99ce3_row3_col5\" class=\"data row3 col5\" >lookup_order, lookup_policy</td>\n      <td id=\"T_99ce3_row3_col6\" class=\"data row3 col6\" >request_photo_then_offer_replacement</td>\n      <td id=\"T_99ce3_row3_col7\" class=\"data row3 col7\" >True</td>\n      <td id=\"T_99ce3_row3_col8\" class=\"data row3 col8\" >2381</td>\n      <td id=\"T_99ce3_row3_col9\" class=\"data row3 col9\" >2940</td>\n      <td id=\"T_99ce3_row3_col10\" class=\"data row3 col10\" >1.26</td>\n      <td id=\"T_99ce3_row3_col11\" class=\"data row3 col11\" >$0.00097</td>\n      <td id=\"T_99ce3_row3_col12\" class=\"data row3 col12\" >0.98</td>\n      <td id=\"T_99ce3_row3_col13\" class=\"data row3 col13\" >I can help with a replacement. Please reply with a photo of the cracked blender and packaging, and we can start the replacement or refund process under the damaged-delivery policy.</td>\n    </tr>\n    <tr>\n      <th id=\"T_99ce3_level0_row4\" class=\"row_heading level0 row4\" >43</th>\n      <td id=\"T_99ce3_row4_col0\" class=\"data row4 col0\" >T-004</td>\n      <td id=\"T_99ce3_row4_col1\" class=\"data row4 col1\" >Round 4: split workflow</td>\n      <td id=\"T_99ce3_row4_col2\" class=\"data row4 col2\" >billing_issue</td>\n      <td id=\"T_99ce3_row4_col3\" class=\"data row4 col3\" >medium</td>\n      <td id=\"T_99ce3_row4_col4\" class=\"data row4 col4\" >gpt-5.4-mini</td>\n      <td id=\"T_99ce3_row4_col5\" class=\"data row4 col5\" >lookup_order, lookup_policy, escalate_to_human</td>\n      <td id=\"T_99ce3_row4_col6\" class=\"data row4 col6\" >escalate_billing_review</td>\n      <td id=\"T_99ce3_row4_col7\" class=\"data row4 col7\" >True</td>\n      <td id=\"T_99ce3_row4_col8\" class=\"data row4 col8\" >2401</td>\n      <td id=\"T_99ce3_row4_col9\" class=\"data row4 col9\" >2942</td>\n      <td id=\"T_99ce3_row4_col10\" class=\"data row4 col10\" >1.41</td>\n      <td id=\"T_99ce3_row4_col11\" class=\"data row4 col11\" >$0.00103</td>\n      <td id=\"T_99ce3_row4_col12\" class=\"data row4 col12\" >0.98</td>\n      <td id=\"T_99ce3_row4_col13\" class=\"data row4 col13\" >I found a duplicate-charge signal and sent this to billing review. The billing team will verify the charge before any refund is confirmed.</td>\n    </tr>\n    <tr>\n      <th id=\"T_99ce3_level0_row5\" class=\"row_heading level0 row5\" >47</th>\n      <td id=\"T_99ce3_row5_col0\" class=\"data row5 col0\" >T-008</td>\n      <td id=\"T_99ce3_row5_col1\" class=\"data row5 col1\" >Round 4: split workflow</td>\n      <td id=\"T_99ce3_row5_col2\" class=\"data row5 col2\" >high_value_damage</td>\n      <td id=\"T_99ce3_row5_col3\" class=\"data row5 col3\" >high</td>\n      <td id=\"T_99ce3_row5_col4\" class=\"data row5 col4\" >gpt-5.4</td>\n      <td id=\"T_99ce3_row5_col5\" class=\"data row5 col5\" >lookup_order, lookup_policy, escalate_to_human</td>\n      <td id=\"T_99ce3_row5_col6\" class=\"data row5 col6\" >escalate_high_value_damage</td>\n      <td id=\"T_99ce3_row5_col7\" class=\"data row5 col7\" >True</td>\n      <td id=\"T_99ce3_row5_col8\" class=\"data row5 col8\" >2492</td>\n      <td id=\"T_99ce3_row5_col9\" class=\"data row5 col9\" >3063</td>\n      <td id=\"T_99ce3_row5_col10\" class=\"data row5 col10\" >2.03</td>\n      <td id=\"T_99ce3_row5_col11\" class=\"data row5 col11\" >$0.00381</td>\n      <td id=\"T_99ce3_row5_col12\" class=\"data row5 col12\" >0.99</td>\n      <td id=\"T_99ce3_row5_col13\" class=\"data row5 col13\" >I am sorry the item arrived damaged. Because this is a high-value item, I escalated it for human review. Please attach photos of the item and packaging.</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "walkthrough_tickets = [\"T-002\", \"T-004\", \"T-008\"]\n",
    "walkthrough = traces[\n",
    "    traces[\"ticket_id\"].isin(walkthrough_tickets)\n",
    "    & traces[\"variant\"].isin([\"00_bad_baseline\", \"04_split_workflow\"])\n",
    "].copy()\n",
    "walkthrough[\"response_preview\"] = walkthrough[\"customer_response\"].str.replace(\"\\n\", \" \").str.slice(0, 220)\n",
    "\n",
    "display(\n",
    "    walkthrough[\n",
    "        [\n",
    "            \"ticket_id\",\n",
    "            \"variant_label\",\n",
    "            \"intent\",\n",
    "            \"risk\",\n",
    "            \"model\",\n",
    "            \"tools\",\n",
    "            \"action\",\n",
    "            \"policy_compliant\",\n",
    "            \"sync_tokens\",\n",
    "            \"total_tokens\",\n",
    "            \"latency_s\",\n",
    "            \"cost_usd\",\n",
    "            \"quality_score\",\n",
    "            \"response_preview\",\n",
    "        ]\n",
    "    ].style.format({\"cost_usd\": \"${:.5f}\", \"quality_score\": \"{:.2f}\", \"latency_s\": \"{:.2f}\"})\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95b2e8d5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.382498Z",
     "iopub.status.busy": "2026-04-30T18:28:05.382430Z",
     "iopub.status.idle": "2026-04-30T18:28:05.388930Z",
     "shell.execute_reply": "2026-04-30T18:28:05.388660Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d7c200>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_8306c\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_8306c_level0_col0\" class=\"col_heading level0 col0\" colspan=\"5\">delta</th>\n    </tr>\n    <tr>\n      <th class=\"index_name level1\" >variant</th>\n      <th id=\"T_8306c_level1_col0\" class=\"col_heading level1 col0\" >sync_tokens_saved</th>\n      <th id=\"T_8306c_level1_col1\" class=\"col_heading level1 col1\" >total_tokens_saved</th>\n      <th id=\"T_8306c_level1_col2\" class=\"col_heading level1 col2\" >latency_saved_s</th>\n      <th id=\"T_8306c_level1_col3\" class=\"col_heading level1 col3\" >cost_saved_usd</th>\n      <th id=\"T_8306c_level1_col4\" class=\"col_heading level1 col4\" >quality_change</th>\n    </tr>\n    <tr>\n      <th class=\"index_name level0\" >ticket_id</th>\n      <th class=\"blank col0\" >&nbsp;</th>\n      <th class=\"blank col1\" >&nbsp;</th>\n      <th class=\"blank col2\" >&nbsp;</th>\n      <th class=\"blank col3\" >&nbsp;</th>\n      <th class=\"blank col4\" >&nbsp;</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_8306c_level0_row0\" class=\"row_heading level0 row0\" >T-002</th>\n      <td id=\"T_8306c_row0_col0\" class=\"data row0 col0\" >10,536</td>\n      <td id=\"T_8306c_row0_col1\" class=\"data row0 col1\" >9,977</td>\n      <td id=\"T_8306c_row0_col2\" class=\"data row0 col2\" >3.62</td>\n      <td id=\"T_8306c_row0_col3\" class=\"data row0 col3\" >$0.03937</td>\n      <td id=\"T_8306c_row0_col4\" class=\"data row0 col4\" >+0.33</td>\n    </tr>\n    <tr>\n      <th id=\"T_8306c_level0_row1\" class=\"row_heading level0 row1\" >T-004</th>\n      <td id=\"T_8306c_row1_col0\" class=\"data row1 col0\" >10,429</td>\n      <td id=\"T_8306c_row1_col1\" class=\"data row1 col1\" >9,888</td>\n      <td id=\"T_8306c_row1_col2\" class=\"data row1 col2\" >3.46</td>\n      <td id=\"T_8306c_row1_col3\" class=\"data row1 col3\" >$0.03935</td>\n      <td id=\"T_8306c_row1_col4\" class=\"data row1 col4\" >+0.13</td>\n    </tr>\n    <tr>\n      <th id=\"T_8306c_level0_row2\" class=\"row_heading level0 row2\" >T-008</th>\n      <td id=\"T_8306c_row2_col0\" class=\"data row2 col0\" >10,505</td>\n      <td id=\"T_8306c_row2_col1\" class=\"data row2 col1\" >9,934</td>\n      <td id=\"T_8306c_row2_col2\" class=\"data row2 col2\" >2.88</td>\n      <td id=\"T_8306c_row2_col3\" class=\"data row2 col3\" >$0.03754</td>\n      <td id=\"T_8306c_row2_col4\" class=\"data row2 col4\" >+0.77</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "walkthrough_delta = (\n",
    "    walkthrough.pivot(index=\"ticket_id\", columns=\"variant\", values=[\"sync_tokens\", \"total_tokens\", \"latency_s\", \"cost_usd\", \"quality_score\"])\n",
    "    .copy()\n",
    ")\n",
    "\n",
    "walkthrough_delta[(\"delta\", \"sync_tokens_saved\")] = walkthrough_delta[(\"sync_tokens\", \"00_bad_baseline\")] - walkthrough_delta[(\"sync_tokens\", \"04_split_workflow\")]\n",
    "walkthrough_delta[(\"delta\", \"total_tokens_saved\")] = walkthrough_delta[(\"total_tokens\", \"00_bad_baseline\")] - walkthrough_delta[(\"total_tokens\", \"04_split_workflow\")]\n",
    "walkthrough_delta[(\"delta\", \"latency_saved_s\")] = walkthrough_delta[(\"latency_s\", \"00_bad_baseline\")] - walkthrough_delta[(\"latency_s\", \"04_split_workflow\")]\n",
    "walkthrough_delta[(\"delta\", \"cost_saved_usd\")] = walkthrough_delta[(\"cost_usd\", \"00_bad_baseline\")] - walkthrough_delta[(\"cost_usd\", \"04_split_workflow\")]\n",
    "walkthrough_delta[(\"delta\", \"quality_change\")] = walkthrough_delta[(\"quality_score\", \"04_split_workflow\")] - walkthrough_delta[(\"quality_score\", \"00_bad_baseline\")]\n",
    "\n",
    "display(\n",
    "    walkthrough_delta[[\"delta\"]].style.format(\n",
    "        {\n",
    "            (\"delta\", \"sync_tokens_saved\"): \"{:,.0f}\",\n",
    "            (\"delta\", \"total_tokens_saved\"): \"{:,.0f}\",\n",
    "            (\"delta\", \"latency_saved_s\"): \"{:.2f}\",\n",
    "            (\"delta\", \"cost_saved_usd\"): \"${:.5f}\",\n",
    "            (\"delta\", \"quality_change\"): \"{:+.2f}\",\n",
    "        }\n",
    "    )\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "516764fe",
   "metadata": {},
   "source": [
    "## Before and after summary\n",
    "\n",
    "The final row includes both synchronous customer-path cost and the modeled async follow-up cost. `mean_sync_tokens` is the customer-facing path; `mean_total_tokens` also includes background QA/tagging work after the workflow split.\n",
    "\n",
    "The strongest result is not from a single trick. It comes from applying levers in a safe order:\n",
    "\n",
    "establish a baseline -> prompt/output controls -> tool control -> basic context hygiene -> model routing -> caching -> cache-aware context tuning -> split workflow -> processing tier\n",
    "\n",
    "The key engineering habit is to optimize per step, not globally. A routine classifier, a high-risk refund dispute, a customer-facing response, and an offline QA tagger should not have the same model, context, tools, latency target, or service tier.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "199941ac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-04-30T18:28:05.390084Z",
     "iopub.status.busy": "2026-04-30T18:28:05.390019Z",
     "iopub.status.idle": "2026-04-30T18:28:05.393989Z",
     "shell.execute_reply": "2026-04-30T18:28:05.393714Z"
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "metadata": {},
     "data": {
      "text/plain": "<pandas.io.formats.style.Styler at 0x117d7f5f0>",
      "text/html": "<style type=\"text/css\">\n</style>\n<table id=\"T_a4225\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_a4225_level0_col0\" class=\"col_heading level0 col0\" >variant_label</th>\n      <th id=\"T_a4225_level0_col1\" class=\"col_heading level0 col1\" >mean_quality</th>\n      <th id=\"T_a4225_level0_col2\" class=\"col_heading level0 col2\" >policy_compliance</th>\n      <th id=\"T_a4225_level0_col3\" class=\"col_heading level0 col3\" >action_accuracy</th>\n      <th id=\"T_a4225_level0_col4\" class=\"col_heading level0 col4\" >escalation_accuracy</th>\n      <th id=\"T_a4225_level0_col5\" class=\"col_heading level0 col5\" >mean_tool_calls</th>\n      <th id=\"T_a4225_level0_col6\" class=\"col_heading level0 col6\" >mean_extra_tool_calls</th>\n      <th id=\"T_a4225_level0_col7\" class=\"col_heading level0 col7\" >mean_sync_tokens</th>\n      <th id=\"T_a4225_level0_col8\" class=\"col_heading level0 col8\" >mean_total_tokens</th>\n      <th id=\"T_a4225_level0_col9\" class=\"col_heading level0 col9\" >mean_cached_tokens</th>\n      <th id=\"T_a4225_level0_col10\" class=\"col_heading level0 col10\" >p50_latency_s</th>\n      <th id=\"T_a4225_level0_col11\" class=\"col_heading level0 col11\" >sync_cost_per_ticket_usd</th>\n      <th id=\"T_a4225_level0_col12\" class=\"col_heading level0 col12\" >background_cost_per_ticket_usd</th>\n      <th id=\"T_a4225_level0_col13\" class=\"col_heading level0 col13\" >cost_per_ticket_usd</th>\n      <th id=\"T_a4225_level0_col14\" class=\"col_heading level0 col14\" >monthly_cost_at_100k_tickets</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_a4225_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n      <td id=\"T_a4225_row0_col0\" class=\"data row0 col0\" >Bad baseline</td>\n      <td id=\"T_a4225_row0_col1\" class=\"data row0 col1\" >0.51</td>\n      <td id=\"T_a4225_row0_col2\" class=\"data row0 col2\" >10%</td>\n      <td id=\"T_a4225_row0_col3\" class=\"data row0 col3\" >60%</td>\n      <td id=\"T_a4225_row0_col4\" class=\"data row0 col4\" >70%</td>\n      <td id=\"T_a4225_row0_col5\" class=\"data row0 col5\" >5.0</td>\n      <td id=\"T_a4225_row0_col6\" class=\"data row0 col6\" >2.4</td>\n      <td id=\"T_a4225_row0_col7\" class=\"data row0 col7\" >11,935</td>\n      <td id=\"T_a4225_row0_col8\" class=\"data row0 col8\" >11,935</td>\n      <td id=\"T_a4225_row0_col9\" class=\"data row0 col9\" >0</td>\n      <td id=\"T_a4225_row0_col10\" class=\"data row0 col10\" >4.88</td>\n      <td id=\"T_a4225_row0_col11\" class=\"data row0 col11\" >$0.03813</td>\n      <td id=\"T_a4225_row0_col12\" class=\"data row0 col12\" >$0.00000</td>\n      <td id=\"T_a4225_row0_col13\" class=\"data row0 col13\" >$0.03813</td>\n      <td id=\"T_a4225_row0_col14\" class=\"data row0 col14\" >$3,813</td>\n    </tr>\n    <tr>\n      <th id=\"T_a4225_level0_row1\" class=\"row_heading level0 row1\" >4</th>\n      <td id=\"T_a4225_row1_col0\" class=\"data row1 col0\" >Round 4: split workflow</td>\n      <td id=\"T_a4225_row1_col1\" class=\"data row1 col1\" >0.98</td>\n      <td id=\"T_a4225_row1_col2\" class=\"data row1 col2\" >100%</td>\n      <td id=\"T_a4225_row1_col3\" class=\"data row1 col3\" >100%</td>\n      <td id=\"T_a4225_row1_col4\" class=\"data row1 col4\" >100%</td>\n      <td id=\"T_a4225_row1_col5\" class=\"data row1 col5\" >2.6</td>\n      <td id=\"T_a4225_row1_col6\" class=\"data row1 col6\" >0.0</td>\n      <td id=\"T_a4225_row1_col7\" class=\"data row1 col7\" >2,412</td>\n      <td id=\"T_a4225_row1_col8\" class=\"data row1 col8\" >2,974</td>\n      <td id=\"T_a4225_row1_col9\" class=\"data row1 col9\" >1,779</td>\n      <td id=\"T_a4225_row1_col10\" class=\"data row1 col10\" >1.41</td>\n      <td id=\"T_a4225_row1_col11\" class=\"data row1 col11\" >$0.00194</td>\n      <td id=\"T_a4225_row1_col12\" class=\"data row1 col12\" >$0.00010</td>\n      <td id=\"T_a4225_row1_col13\" class=\"data row1 col13\" >$0.00204</td>\n      <td id=\"T_a4225_row1_col14\" class=\"data row1 col14\" >$204</td>\n    </tr>\n  </tbody>\n</table>\n"
     }
    }
   ],
   "source": [
    "before_after = summary[summary[\"variant\"].isin([\"00_bad_baseline\", \"04_split_workflow\"])].copy()\n",
    "display(\n",
    "    before_after[\n",
    "        [\n",
    "            \"variant_label\",\n",
    "            \"mean_quality\",\n",
    "            \"policy_compliance\",\n",
    "            \"action_accuracy\",\n",
    "            \"escalation_accuracy\",\n",
    "            \"mean_tool_calls\",\n",
    "            \"mean_extra_tool_calls\",\n",
    "            \"mean_sync_tokens\",\n",
    "            \"mean_total_tokens\",\n",
    "            \"mean_cached_tokens\",\n",
    "            \"p50_latency_s\",\n",
    "            \"sync_cost_per_ticket_usd\",\n",
    "            \"background_cost_per_ticket_usd\",\n",
    "            \"cost_per_ticket_usd\",\n",
    "            \"monthly_cost_at_100k_tickets\",\n",
    "        ]\n",
    "    ].style.format(\n",
    "        {\n",
    "            \"mean_quality\": \"{:.2f}\",\n",
    "            \"policy_compliance\": \"{:.0%}\",\n",
    "            \"action_accuracy\": \"{:.0%}\",\n",
    "            \"escalation_accuracy\": \"{:.0%}\",\n",
    "            \"mean_tool_calls\": \"{:.1f}\",\n",
    "            \"mean_extra_tool_calls\": \"{:.1f}\",\n",
    "            \"mean_sync_tokens\": \"{:,.0f}\",\n",
    "            \"mean_total_tokens\": \"{:,.0f}\",\n",
    "            \"mean_cached_tokens\": \"{:,.0f}\",\n",
    "            \"p50_latency_s\": \"{:.2f}\",\n",
    "            \"sync_cost_per_ticket_usd\": \"${:.5f}\",\n",
    "            \"background_cost_per_ticket_usd\": \"${:.5f}\",\n",
    "            \"cost_per_ticket_usd\": \"${:.5f}\",\n",
    "            \"monthly_cost_at_100k_tickets\": \"${:,.0f}\",\n",
    "        }\n",
    "    )\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d82d2bba",
   "metadata": {},
   "source": [
    "### Recommended tuning order\n",
    "\n",
    "1. Baseline\n",
    "2. Prompt/output controls\n",
    "3. Tool control\n",
    "4. Context hygiene\n",
    "5. Model routing\n",
    "6. Prompt caching\n",
    "7. Cache-aware context\n",
    "8. Split workflow\n",
    "9. Processing tier\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75d0b8ec",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Cost optimization for support agents works best as a measured sequence of small changes, not as a single model swap or prompt rewrite. Start by building a baseline that exposes where tokens, tool calls, latency, quality failures, and spend are going. Then tighten prompt and output controls, restrict tool use, reduce tool payloads, trim context, route simple work to smaller models, make stable prefixes cache-friendly, and move non-customer-facing work out of the synchronous path.\n",
    "\n",
    "The main principle is to spend capability where it protects quality. A routine order-status question, a structured triage step, a policy-heavy refund dispute, and an offline QA tagger should not use the same model, context, tools, or latency tier. The optimized system should be cheaper because it is more disciplined, not because it blindly removes safeguards.\n",
    "\n",
    "Before shipping changes, validate them with representative evals and trace metrics. Track quality score, policy compliance, action accuracy, escalation accuracy, tool-call count, token usage, cached-token volume, p50 and p95 latency, synchronous cost, async follow-up cost, and total cost per ticket. A configuration is only better if it lowers cost while preserving the support quality bar.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}