{ "cells": [ { "cell_type": "markdown", "id": "e69409d9", "metadata": {}, "source": [ "# Course 5A Lab — Evaluation Literacy, Not Benchmark Reproduction\n", "\n", "This notebook performs transparent arithmetic over a **small set of values transcribed from Section 6 and Table 2** of the Kimi K3 report. It computes within-benchmark gaps, paired Python-tool lifts, a Pareto-frontier miniature, and a local exact-match example.\n", "\n", "> **Evidence boundary:** no model is invoked; no public or private benchmark dataset is present; no paper score is reproduced. Every K3 value below remains **paper reported**." ] }, { "cell_type": "code", "execution_count": 1, "id": "bbf6ffb8", "metadata": { "execution": { "iopub.execute_input": "2026-08-11T03:53:04.844466Z", "iopub.status.busy": "2026-08-11T03:53:04.844152Z", "iopub.status.idle": "2026-08-11T03:53:04.873148Z", "shell.execute_reply": "2026-08-11T03:53:04.871924Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Local analysis helpers imported.\n", "Boundary: supplied-value arithmetic only; zero benchmark runs.\n" ] } ], "source": [ "from pathlib import Path\n", "import subprocess, sys, tempfile\n", "repo = next((p for p in (Path.cwd(), *Path.cwd().parents) if (p / 'src').is_dir()), None)\n", "if repo is None:\n", " repo = (Path('/content') if Path('/content').is_dir() else Path(tempfile.gettempdir())) / 'build-Kimi-K3-architecture'\n", " if not (repo / 'src').is_dir():\n", " subprocess.run(['git', 'clone', '--depth', '1', 'https://github.com/mailtotanvir/build-Kimi-K3-architecture.git', str(repo)], check=True)\n", "sys.path.insert(0, str(repo))\n", "\n", "from src.eval.benchmark_analysis import (\n", " exact_match_rate, gap_to_frontier, paired_tool_lift, pareto_frontier\n", ")\n", "\n", "print('Local analysis helpers imported.')\n", "print('Boundary: supplied-value arithmetic only; zero benchmark runs.')" ] }, { "cell_type": "markdown", "id": "b7e0f8c5", "metadata": {}, "source": [ "## 1. Keep every comparison inside one benchmark\n", "\n", "A signed gap is meaningful only when models share the benchmark and metric. Negative means K3 trails the best reported value in that row." ] }, { "cell_type": "code", "execution_count": 2, "id": "feb1a590", "metadata": { "execution": { "iopub.execute_input": "2026-08-11T03:53:04.875457Z", "iopub.status.busy": "2026-08-11T03:53:04.875264Z", "iopub.status.idle": "2026-08-11T03:53:04.881682Z", "shell.execute_reply": "2026-08-11T03:53:04.880444Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Benchmark K3 Leader Best Gap\n", "GPQA Diamond 93.5 GPT-5.6 Sol 94.1 -0.6\n", "CritPt 23.4 GPT-5.6 Sol 32.3 -8.9\n", "HLE no tools 43.5 Fable 5 53.3 -9.8\n", "SWE-Marathon 42.0 Kimi K3 42.0 +0.0\n", "FrontierSWE 81.2 Fable 5 86.6 -5.4\n", "BrowseComp 91.2 Kimi K3 91.2 +0.0\n", "OSWorld 2.0 58.3 Fable 5 66.1 -7.8\n" ] } ], "source": [ "spotlights = {\n", " 'GPQA Diamond': {'Kimi K3': 93.5, 'Fable 5': 92.6, 'GPT-5.6 Sol': 94.1},\n", " 'CritPt': {'Kimi K3': 23.4, 'Fable 5': 28.6, 'GPT-5.6 Sol': 32.3},\n", " 'HLE no tools': {'Kimi K3': 43.5, 'Fable 5': 53.3, 'GPT-5.6 Sol': 44.5},\n", " 'SWE-Marathon': {'Kimi K3': 42.0, 'Fable 5': 35.0, 'GPT-5.6 Sol': 39.0},\n", " 'FrontierSWE': {'Kimi K3': 81.2, 'Fable 5': 86.6, 'GPT-5.6 Sol': 71.3},\n", " 'BrowseComp': {'Kimi K3': 91.2, 'Fable 5': 88.0, 'GPT-5.6 Sol': 90.4},\n", " 'OSWorld 2.0': {'Kimi K3': 58.3, 'Fable 5': 66.1, 'GPT-5.6 Sol': 62.6},\n", "}\n", "\n", "rows = []\n", "for benchmark, scores in spotlights.items():\n", " result = gap_to_frontier(scores, 'Kimi K3')\n", " rows.append((benchmark, result['score'], result['leader'], result['best'], result['gap']))\n", "\n", "print(f\"{'Benchmark':<20} {'K3':>6} {'Leader':<13} {'Best':>6} {'Gap':>7}\")\n", "for name, score, leader, best, gap in rows:\n", " print(f\"{name:<20} {score:>6.1f} {leader:<13} {best:>6.1f} {gap:>+7.1f}\")" ] }, { "cell_type": "markdown", "id": "f4dbc31f", "metadata": {}, "source": [ "## 2. Paired tool augmentation is an intervention on the evaluated system\n", "\n", "The difference below is observed under the paper's paired no-Python/Python protocols. It does not reveal which tool action caused the improvement." ] }, { "cell_type": "code", "execution_count": 3, "id": "13c43e07", "metadata": { "execution": { "iopub.execute_input": "2026-08-11T03:53:04.883714Z", "iopub.status.busy": "2026-08-11T03:53:04.883532Z", "iopub.status.idle": "2026-08-11T03:53:04.888962Z", "shell.execute_reply": "2026-08-11T03:53:04.887725Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MMMU-Pro 81.6 → 83.4 lift +1.8\n", "CharXiv RQ 84.8 → 91.3 lift +6.5\n", "Math-Vision 94.3 → 97.8 lift +3.5\n", "ZeroBench-main pass@5 23.0 → 41.0 lift +18.0\n" ] } ], "source": [ "tool_pairs = {\n", " 'MMMU-Pro': (81.6, 83.4),\n", " 'CharXiv RQ': (84.8, 91.3),\n", " 'Math-Vision': (94.3, 97.8),\n", " 'ZeroBench-main pass@5': (23.0, 41.0),\n", "}\n", "tool_lifts = {name: paired_tool_lift(*pair) for name, pair in tool_pairs.items()}\n", "for name, (without, with_) in tool_pairs.items():\n", " print(f\"{name:<24} {without:>5.1f} → {with_:>5.1f} lift {tool_lifts[name]:>+5.1f}\")\n", "\n", "assert tool_lifts['ZeroBench-main pass@5'] == 18.0\n", "assert tool_lifts['CharXiv RQ'] == 6.5" ] }, { "cell_type": "code", "execution_count": 4, "id": "f83dde5c", "metadata": { "execution": { "iopub.execute_input": "2026-08-11T03:53:04.891341Z", "iopub.status.busy": "2026-08-11T03:53:04.891152Z", "iopub.status.idle": "2026-08-11T03:53:05.724526Z", "shell.execute_reply": "2026-08-11T03:53:05.723105Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxgAAAFyCAYAAAB2jXxxAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAT9lJREFUeJzt3Qm8zNX/x/GPfV9S2SJbIkuyhFSIskVKZamEVqlEG2mhjbK1Wypb+6pSEhJCWbKUiuxLRZaKSlnn/3if/+M7v5kxM3fu9b3u9no+HuO63/nOdz0z93zO+Zwz2QKBQMAAAAAAwAfZ/dgIAAAAABBgAAAAAPAVPRgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAcB3EydOtGzZstk333wTtnzXrl1Wr149K1iwoM2cOdMtGzRokFvXT4lu01tPxxVq/fr1VrFiRStRooStWLHCLXvzzTetcePGblmePHmsdOnS1q5dO/vqq68so9yPTZs2HZf9aV+6tvH8+uuvbh3v+qamOXPmuGPSTyRu37597h4dy3WLd59T472fqN9//906d+5sxYsXd8dw6aWXWnrkXaOkHk2bNk34ffDee+8lvF+/LF261G699VarWbOmFSpUyH2OXnjhhfbFF1/4tg+kLznT+gAAZA0///yzXXTRRfbbb7/Z559/bg0bNnTLb7jhBmvVqpWv+zqWba5cudJatmxpuXLlsvnz51vlypXd8t27d9u5555rd9xxh5100km2bds2GzlypAs6Zs2aZU2aNLH06uKLL7avv/7aSpUqZemFKp4PP/ywlS9f3s4666y0PhzECDB0jySRCmxy73NqvPcT9eijj9oHH3xg48ePt0qVKlmxYsUsPYq8Rvrc6dChg91+++121VVXBZcXLlzY0jM10CxevNiuu+46q1Wrlv3zzz82ZswYa968uU2aNMmuvfbatD5E+IwAA0CqW7t2rWutOnjwoM2dO9e1YnnKlCnjHn5K6TYXLlxobdq0ca1r6mEJ3cZtt9121PqtW7e2k08+2caNG5euAwwdox6JVCjz589/XI4JSI33fqK+//57F1hcffXVcdc7fPiwHTp0yPVapodr5PVCnnrqqcFGmozg3nvvteHDh4ct02dtnTp17JFHHiHAyIRIkQKQqpQacd5551nOnDldj0BocBGrK16tnW3btrVPPvnEateubfny5bMzzjjD/e6l/Oj3AgUKWP369Y9KxUpJ974CCgVBqnTMmzcvoYqPuvrz5s3rzs3vdCYdT48ePVzLqs5T6VgbNmw46pjbt2/vjlXHcdppp9nNN998VMpXtBQptUjXqFHDvvzyS2vUqJELLNS6KHv37rW7777bKlSoYLlz57ZTTjnF+vTp41odQ2m9G2+80U488USX9qaW1jVr1iSUqnH22We7/+scvTSP0LSqKVOm2DnnnOOOS9dZvV/qhYmkMqVWUK2jdXUuU6dOtZTYuXOn9erVy6pVq+bOR+kzzZo1c+UhkZQrXV8t1/UO9dJLL9npp5/uKqna9htvvGHdu3d35TzytcOGDbMnn3zSPadyr/uka6rgvH///i41r0iRInbZZZfZjh07jjqHt99+2103lRmdg3rjli9fHraO9q3n1q1b5yp5+n/ZsmXtrrvusv379wePxwtK1QPh3SO9VvRa3Tv18Om6q4yojKoHMNH7HO19euTIERs6dKhVrVrVXS/dA7Vuqwc0lFd+lyxZYueff747BqU1PvHEE24bsXjXWb2oq1atCh6TjtV7Tvt/7LHHXPnXMcyePTvhMumd03fffWdXXnmlu1d6D995550uUPnpp5/c+0Sv1z3Wvo5VIvci1H///eeOp2TJkq6MqXEksozEkkj5ikb3MVKOHDmsbt26tnXr1oT2jYyFAANAqlHlTxUB/XHR/1UBSNS3335r9913n/Xr188mT57s/lArNWDgwIH28ssv2+DBg+3111+3PXv2uGDk33//TfFxvv/++24bqgwpJ1gpUPFaNFXZU2XklltusUAg4HKL/Xb99ddb9uzZXWX06aefdukFupZ//vln2FgR/bEfPXq0zZgxwx566CFbtGiRC+h0jElRusU111zjUi0+/fRTV7lWL4YqHEpb6N27t02bNs3dA1WaL7nkEne+op/KW3/11VddxVTpJmpRVa9OUtRqOWHCBPf/Bx54wFXS9FA6iOicFTgp7UOpFeoh+uOPP9z5qxx51BumAEBlQOtoXVXcVLlSRSglefmiMqYgRceoMqv9pnQcwosvvmg33XSTnXnmma4c63xVYY+1vRdeeMEWLFjgfqqcr1692p2PyoMCIKX0qFKqCrJ3vTx6T3Tp0sUFMe+88467N3/99ZergP/4449h66p86H4qOPvoo49ccPnUU0+54EaUTvfZZ5+5/2vf3j168MEHg6lPCixVodd6Ol4F2g0aNHCV6ETuczR6T6m8qfKuCr1SmbR9BY6RgfP27dtdD4TKsNZV2dNnxmuvvRZz+zovHYMaLnRvvWPSsXqeffZZ9zmgFneVfwU7iZZJT8eOHV0qkD5bFITr2vbt29e9Z5SyqPeLyq73+XYsErkXoQYMGOAaK1S+9NDrdR6RDRiRklO+EqGAS8F79erVk/1aZAABAPDZhAkTVAt1jyJFigR27NgRc92BAwe69UKVK1cukC9fvsDPP/8cXLZixQq3XqlSpQL//PNPcPmHH37olk+ZMiXuNuPtW4+KFSsG/v333yRfU6VKleBrdCzz588PJKJbt27uNRs3bkzo2l122WVhyxcsWOCWP/bYY1Ffd+TIkcDBgwcDmzdvdut99NFHR20zdN9NmjRxy2bNmhW2nSFDhgSyZ88eWLJkSdjy9957z63/6aefut+nTZvmfn/mmWfC1nv88cfdcl3beLR9radjC3X48OFA6dKlAzVr1nT/9/z111+B4sWLBxo1ahRc1rBhQ7dMz3kOHToUqFGjRqBMmTLumsjs2bPdvvQzObQtXdPmzZuH3Y9Y29P1DT0nHX/JkiUDDRo0CFtP9yhXrlyunEe+tlatWmHn/fTTT7vll1xySdg2+vTp45bv2bPH/b5ly5ZAzpw5A7fffnvYero2OoaOHTseVRbfeeedsHXbtGnjyrdn586dCd1L71odOHAgULly5UDfvn2TvM/R3qerVq1yv/fq1StsvUWLFrnlAwYMOKr86rlQ1apVC7Rs2TLJ49Xrq1evHrbMuweVKlVy55KSMumd04gRI8K2fdZZZ7nlkydPDi5T2Tr55JMDHTp0SPJ4I49x2LBhyb4XXrmtU6dO8L0hmzZtcuXxhhtuOOo8PMkpX4m6//773T70GY7Mhx4MAKlGLaRqXVZ6jVr+k0MDQtXV71FKlKilLXScgLd88+bNx3Scar1LauYjUYukegneffdd15KnVtPUmJ0oMjdcLbjlypULpmuIUmR69uzp0lvUYqmB6VpHlP6RlBNOOMG1ooZSGppST3T91cLoPZQKEZoW5B1H5HGGDjxNCbW4qkW1a9eurgfHo3SMyy+/3I2TUS+L0rV0H6644gr3XGjahV6rlJporbdJ0cBTtWZ7qW+6phrEn8j1jHYuamVXa3Yo5c9rwoBolLIUet5e+Vardyhv+ZYtW9zP6dOnu/ukdKLQ+6bzUI9UZBnVvVTPSCj1siT6PtK21aKt94DS6HSt9FPjrVJyrULLlJeG5VEapM5X9yGUUnz0XErPId7nge57cstkKPWIhtLx65qH9vDpmimt8ViPN7n3Qu/R0NQ0fWbo8yX0syVScstXUtRz8vjjj7veT/UMIfNhkDeAVKN0ClVUNYhPedFKXVAFMBGRs7roD2a85corTinlyGu7Sg/xcsBj8brzVbFRuoNSLTSzlFK64lGKUWRufjyqPEVbptmsRMfZokULV/HRddbYFuVFa7lSlRJJGYs2q5Rm+VJOd2gFK5SXpqLjUEVGqRlJHXdyeOcX7dg0/kDnp9QUpWjpEWu90G0lSrOCqcKjoE2pOUqVU3nV9U1JpdnbvyYNiKRlGzdu9K3c676JN+YhUmjFWBSkq3IYSuMNEn0fKYdfqThK8VEFU8Gq9qH0p5SmKyZ17yMr4pFlzzuHY0mXjLb/RMtkaMNHtPsV7ZprucYyHYvk3otYny3xPsOSW77iUdqcxoopdVBjjpA5EWAASFXeAFH91B9ijZvwc1C0H/THUTnV3iBbHWfkjCfR6DzU2q18ZL+p5TvaMrV4erPgqEKgoKVbt27BdRQcJCraQHhVqjXwU7n+0XjjU1S5UwumKl+hFb1ox50c3rY0PiSSgindK1WgFFzo/7HWCz3WRCkAVg+ZxrSEUp55KK+S6A2I9kSOEfDOxauchTrW6xTJO1d9x4HXi5WadK3Umq2W88hrULRo0WO+95GTLOieJvd+plTk+yLRMplWknsvYn22RAvY/C5fCi4U+OgzS72FafU9KEh9pEgBSHVKPVKAoYq4uudVMU1vvCBDf/xGjBjhWgWTotZepUd4lX4/KRALpS/0Uwuu930E3h/myOkzx44de0z7VWqHBo+rsqEvRYx8eDMfXXDBBVGPU4NhE+Edd2QLa5UqVVxqnLbjDSgXpUQpPc2bxUe9NRrEqgGyodvwespUQdXMTcmhaxp5PTUbUORMQd410HOhNNA48lzUMhwZgCqtye8vaFQKmwJe3bto902P5Ip1j2JdKw2M/+WXXxLeRiQvXS9ykLZmilIPkgakp4VEy2RaSfReeDRIPfQ89Lmi8hjvu078KF9qDNHnqwblK0WK4CJzS1/NiAAyLc1wpEq80k30x01/5NJbT4b+4GnWH/3UrC86Tv0U5SgrN1u51JrRSrNIqaVbf3A1I4zfNPWu/hhrqktN43j//fe7So5mehLNbKMpdTV1qY5TKRkff/xx8BvSU0rjZVRp0hcIatYb5bSr0q5KsWaqUgqRKvZKz9I6mt9eFS1VMDT7kWaWSYSOXT0lClB0TZXPrnQTPZSiprEdCnaUSqGeAvUsaQYtzZTjGTJkiJttSMGOptVVusmoUaNc747KV3IrMNqfUqM0i5RSTZR7r/Q+TVcaGhQraNCUxtq/Wq7VoqvxAZGzAam8K7DWOWisiGZq0jlomdJtkpNWkhQFPTpWlRONJ9JUqDo29Z5oBjIFZN6X5iVKM3Lp3DTLlCr3KmNqyfamkVaFUeVQZUTf1Kx7FNnzEO8+R6vIK23mueeec9dG4xX0PtNnhsYZqTymBR1LomUyLSR6L0LHbmmaY81upTFyKu/qldMMXKlVvjRmTbORKWVW10+vCaVU07T6rhGkjvT11x1ApqapKvXHWn+kVGl96623LL1RpVS9AMq91/SwOs5nnnnGBRg6XlV4VKFWRUstlwpA9Jzf1Juiynrnzp1dZUaVaB2Hl9utMRIKKDT+Q3+wFayp0qvpSzWIOKVUUdDUkao0KdjSOAFVELVNbd9rvdd9VIu9enpU+Tpw4IAbuKzpblXRSYpafJWGpUqJghVNm6qKjnq71Mul41AFvlOnTu5eaFyJBqGGXmsFAZpOVK/TwGDdK00NquOKHGSbCJVLDdbVtdc5adCs0jgUQEYOYtW90bcpK+9dExhowLSCmsiWXFWYve9WUKVO109BoSrt3gBtv6iCqGNWOdGxqNwoGFLevMaVpISuxT333OOCa21PqS2qzGofKoO6R3///bdLFfSm4U30PkejoF1BifarcQUK5lWZ1X7ipfCktkTLZFpI9F54lEqlXiF9d4bGf2g8mT7bdN1Tq3ypR0Xvz2XLlkWd4ECfM6HfC4OML5umkkrrgwAA/D9V3vSHXxWAlKS1IP1Tq7fStzRJgII4AMhs6MEAACCVaPCspuNUD5Ra4JXvrl4vDRxX7xMAZEYEGAAApBLllSutTmNn9E3hShlSao1Sr/gGYwCZFSlSAAAAAHzDNLUAAAAAfEOAAQAAAMA3BBgAAAAAfEOAgSxNszRrHnBmawYAAPAHAQayNE0VqS9y0k8AAAAcOwIMAAAAAL4hwAAAAADgGwIMAAAAAL4hwAAAAADgGwIMAAAAAL4hwAAAAADgGwIMAAAAAL4hwAAAAADgGwIMAAAAAL4hwAAAAADgGwIMAAAAAL4hwAAAAADgm5z+bQrIwJ4tYpY3rQ8CAABkancHLCugBwMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAABBgAAAAAEh/6MEAAAAA4BsCDAAAAAC+IcBIgTlz5li2bNnszz//tPRg0KBBdtZZZ6X1YQAAAOA4ePHFF61p06ZWuHDhmHXSNWvWWPv27e2kk05y65177rk2e/bsuNsNBAKuXlm6dGnLly+f28cPP/yQegGGV6mO9bjgggsstXXv3j1snyeeeKK1atXKvvvuO8vK7r77bps1a5ZlBocOHbLx48dbixYt7JRTTrGSJUvaeeedZ0899ZT9+++/SZYJPRo2bJgmxw4AAOAXVe4nTpwY9bl9+/a5OvCAAQNivv7iiy929aovvvjCli5d6hqj27Zta9u3b4/5mqFDh9rIkSPt+eeftyVLlrh62EUXXWR//fVX6gQYjRo1sm3bth31GDt2rKvU9erVy1LqwIEDCa+ri+ntW5XqnDlzuouVlRUsWNAFWxndpk2brF69evbMM89Yhw4d7N1337UZM2ZY79693c+aNWvaunXr4pYJPT799NM0OX4AAIDjoU+fPta/f/+Yjaq7du1ydSatc+aZZ1rlypXtiSeecIFJrB4J9V48/fTTdv/997t6WI0aNWzSpEnuNW+88UbqBBi5c+d2UUzo448//rB77rnHRU9XXnllcN0ff/zR2rRp4yq+JUqUsK5du7oTDY3IbrvtNrvzzjtdt40iI5k7d67Vr1/f8uTJY6VKlXIXRZFXKD3n7V+RWL9+/Wzr1q22c+fO4Dq//PKLderUyU444QRX8Vb3kCqvoa3el156qQ0fPtztR+vceuutdvDgweA6+/fvt3vvvdfKli3r9qkbM27cuLBjUTSoCnH+/PldAPbTTz/FvYbefgcPHuyuS9GiRe3hhx9256jrWKxYMStTpoxrwQ+lczz99NPdfipWrGgPPvhg2LFGpkglcn6RvG0oYNQ5a1+6p6Fdbopkda90z4oUKWJNmjSxZcuWHbWdU0891V0zda8pOPCMGjXKXce8efO687/iiiuCz+3du9f1WlxyySW2YsUK69mzp7umelN07NjRpk2b5q5Dy5Ytj+rJCC0Teug6AgAAZFUnnniinXHGGfbKK6/YP//84+qaquOp/lW3bt2or9m4caPr3VB9LLSOpfreV199dXzGYKjiqUqsdvroo48Gl6sFWctUWf3mm2/ss88+s99++81VEkMpIlLvw4IFC9wJKyhQUHL22Wfbt99+a6NHj3YV+sceeyzmMfz999/2+uuv22mnnRZswVeUpXQtBTdffvmlzZ8/3/1frdyhPSXKQVu/fr37qWNRF1RoN9S1115rb731lj377LO2atUqGzNmjNtOKEV4I0aMcOepc7nuuuuSvG7qpvr111/dsakLShVy9cAoGFq0aJGrWOuhoMlTqFAhd2wK3NS6/9JLL7mUoXiSOr9oFOm+88479vHHH7v7poq+AhOPuse6detm8+bNs4ULF7pgQffM6zZ777333HHpfq5du9Y+/PBD1+sgukYKNh555BEXiGn7jRs3Dm77ySeftDp16rjntT0FSQqOtEzHXb16dbvxxhtd0KFrEJm+V7x4cReEaZ0dO3YkeR8AAAAyq2zZstnMmTNt+fLlrh6pxl3V0VT/UgN3NF7qlIKQUPo9XlpV1P0H1B+STEeOHHGVYvUKqKKpgSOehx56yFWUp0+fHlz2888/u1ZxVSxVCVQPxp49e9xJh1bW33//fVeZ10XxWrzVaq11s2fP7iqdr732mrtIoohMldBPPvnEVURFrf/KHwvdjgILXUxVeBWVaTuqlKoCniNHDreOAiDtQ0GFBsVUqVLF3ZgLL7zwqPPXaxXEfP7559a8eXO3TGk5ynVT67p3fJG8/W7YsMHtS6pWreoqxwo45PDhw6534OWXX7bOnTtH3c6wYcPs7bffdpV2UZCic1NAELqfWOcXjbahYE73VL0ookKoc1Lwp56BSDpWBUbqNlN5UMCk4OL777+3XLlyha07efJk69GjhysLKuiRtE/tT91xChI0rkZvBAWMCnLUo6RjU1qcyorKneg6KPArV66ci7zVu6MoXb1LirojaTt6hPacqGzuedSscPTbBgAA4I+7Y1e7leGih0d1StWn1IjtUUbH+eeff1SdVFlFoYGDqvfqCFD2iupNGrCtuuWUKVNcRorqz5HUS6GB4GoID31e9TI1fKuelqo9GEqJ+vrrr+2jjz4KCy5EFTu1mqvS5z1UiRZVeD1KLQqlgOCcc84JBgWik1QvhSqlHl1EVaT1UCCjgKF169a2efPm4P7VEq9KrLd/pcz8999/YftXi7hX+RZdSK/lW9vWc+qJiUfpO6GvF21jy5YtYecfWli0Xy+48KJCr5VftF/1xoS2wqtnQAOdVcnX9lSJ1j7iiXd+sSi1yQsuRPdDwaSX+qXXq3dFQaKCID10f7xjUUqV3gxK41Jh/OCDD4IpbkqtUhCg55Qyp54nBQ/y+++/u4q+ggtRuVKwot4KBXg639Dz0JvIo1Q4BUF6bbt27dwbTwHi1KlTo57jkCFDgseuh4ILAACAtNazZ89gHVcP1ZWV2RG5LBHKmFEDvBqWVZ9WQ7wa7hVoKLMlGq8xObK3QvW/yF6NpPwvJEqQWoyV268KnFJkIqlCqoqeUl4ihUZDBQoUCHtOkVZocOEtk9Dlep1SojzKI1NFUWlDaoHX/rVMFdhIJ598cvD/kS3s2odeK7r4iQjdhneM2oYq6V5vgoSOCYi233jHopZ69WRorIbGH+hcVViUmpXosUVuM1HeOXk/1TOisS4aAKRgQT0ECkK81DOvl0o9P+rd0cB/9bZobI0CPo3XUKStAdvq6VKviaJoBSGhvT7aXmj5CE1NU/pcpUqVYh6zypiOTSla0dx3331u7E9kDwYAAEBaKlasWFidUfVRZbmE1nsT5TXihjZqe7/Hqg9WqFDBBRmqx9WuXTtYJ1M9Llq93rcAQ5VmjTPQKHRVdqNRhKRUp/Lly4d16SSlWrVq7nWhgYa6alQx1XSlsWhdXSxv4K/2ryBINySydyVR6lHQxdcFjZYilRSdd0oKQzQao6IKs7q3PF5vjd/UE6FuMQ3OFvVS6dqqx0I09kLRr8ZdiLrLQgfve28GDdTWQ6lN6r1auXKluy+6LrqeegwcONB15SnC9rrwNH5HAYLGZqiMaQyOep4U0HjlT9dB42Fi2b17tzuuaF1/oqAoWuoUAABARrF9+3b38GbXVF1LdWZloyhIUQOw0tg1dlaNuqqfqTFe6eTK/PConqbsjssuu8zVqTU7lTJv1Imgh/6viX+uuuqqZB1fwilSqkiqIqjxE9dcc03wxLyHN4uTKpVKeenSpYstXrzYjTdQi7UCE+Xsx6LWblUMb7/9dlu9erVLk1ElVK3NodGX8ue9fSqtSusrTUe9JnL11Ve7WY40c5QqxLqQChTuuOOOsFSreBQc6YbomDW2QdtQy7sGQB9vClRU8VevhVK8NOhcqUepQb0IOm/1EujaaVC2xm54XWY6lldffdVdd6Wn6VqH9vZoMLaCAo3B0H3XunpeAZK66XTsChIUIGlWAwVxGuui+6uARHMuiwZxax8KEHUv1LWn12jKNL0JvNkNdN/1HSAKhDQ+Q/dI5UD3X28UAACAzGjMmDGul0Ep6aLGWf2uMRaiupDGTKiu1KxZM5dapYmPVL+uVatWcDvKPNFYZ49mUFWQoXq5XqNxuKrHRxs/G0/CXQxKiVIlT49orcOqRKqSp9Zvtbp7U4oqINBzmsUpspsmlHopNFBa07XqxBV9XX/99fbAAw+EraeL5e1fJ6vIS9+XoMBHFGVpwLT2rwqpZiTStjUYOzk9GprFSmNNdIHVKq6IMN6XmaQWBUp9+/Z10/rqWirq1JgEpRf5TQGErpl6KBQk6qd6LDwaQH/TTTe5AqzroahWFXyPeiTU86CgUMGkeoI0I5XGlOg5DfTWcatXQlHxm2++6caKiIJJFWRF3BowriBHs4/pnindS8FOZP6fxpgoYlewolnNVC40Rkc9WMl9IwAAAKQnc+bMifmc6lNJ1QVVrwqddCmayLme1IuRyLZTZRYpZD6RM1GlBY3bUI+Jer8UyChAUUFXj5Z6PxRgRX5HyLHSGAyNa2EWKQAAkJazSGUmKf4eDMBvGpuhqYvVw6EeKfVc6Ase9X/1TGlyAQAAAKRvyZ5FCkhNSqfTOA4NRPKm1VVqVOQMYwAAAEifSJFClkaKFAAAOG7uJkUKAAAAAJKFMRgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfJPTv00BGVjvPWaFC6f1UQAAAGR49GAAAAAA8A0BBgAAAADfEGAAAAAA8A0BBgAAAADfEGAAAAAA8A0BBgAAAADfEGAAAAAA8A0BBgAAAADfEGAAAAAA8A0BBgAAAADfEGAAAAAA8A0BBgAAAADfEGAAAAAA8E1O/zYFZFy9r51vuXMVSOvDQBwvvtuE6wMAQAZADwYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQaALOXmm2+2SpUqWb58+ezkk0+29u3b2+rVq+O+ZtCgQZYtW7awR8mSJY/bMQMAkJEQYMDp3r27XXrppWm63dQ6BmQ9TZs2tYkTJ0Z9rm7dujZhwgRbtWqVTZ8+3QKBgLVo0cIOHz4cd5vVq1e3bdu2BR8rV65MpaMHACBjI8DIAFTxVotpz549j3quV69e7jmtk4hNmza59VesWHFMx1SzZk274YYboj735ptvWq5cuey3336zZ555JmZFL1Jy1gVS6qabbrLGjRtb+fLlrU6dOvbYY4/Z1q1b3Xsjnpw5c7peC++h3g8AAHA0AowMomzZsvbWW2/Zv//+G1z233//ucr8qaeeetyP5/rrr7d33nnH9u3bd9Rz48ePt7Zt21qJEiWsSJEiVrRo0YS2mZx1AT/8888/rjejQoUK7j0Wz9q1a6106dJu3c6dO9uGDRu4CQAAREGAkUGopVWBxOTJk4PL9H9VimrXrh1c9tlnn9l5553nKuonnniiq+ivX78++LwqR6LXqCdDqSShhg8fbqVKlXKvvfXWW+3gwYNRj6dr1662f/9+e/fdd8OWb9myxb744gsXgERLe3rvvfdc74fy37WPCy+80FXyoq2r7ffu3duKFy9uefPmdee1ZMmS4PNz5sxx5zBr1iyrV6+e5c+f3xo1amQ//fRTCq4wspJRo0ZZwYIF3UPvmZkzZ1ru3Lljrt+gQQN75ZVXXErVSy+9ZNu3b3dlbffu3cf1uAEAyAgIMDKQHj16uNbW0J6C6667LmwdVdbvvPNOVxFXxTt79ux22WWX2ZEjR9zzixcvdj8///xzl0ceGrDMnj3bBSP6OWnSJJeuFCtlScGBBseGHo/od/VctG7d+qjXaH9dunRxx6z8dwUIHTp0cDnw0dx77732/vvvu2NZtmyZnXbaadayZUv7/fffw9a7//77bcSIEfbNN9+4NJbIaxJKQcvevXvDHsj4Bg8eHAwY9Jg3b55LKYxc5rn66qtt+fLlNnfuXKtcubJ17NjR9QjGovJ8+eWXu+BYQfHUqVPdcpVNAAAQLmfE70jH1Gtw3333BcdRLFiwwKVNqaLuUSUo1Lhx41wPwI8//mg1atQI5o0rQIicBeeEE06w559/3nLkyGFVq1a1iy++2AUpN954Y9TjUUW+TZs2LlWkYsWKLlBQQKKeCG0jWoBx6NAhF1SUK1fOLVOFLRoFSqNHj3bb84IVtRyrpVnndM899wTXffzxx61Jkybu//3793fHrcqiej0iDRkyxB5++OE4VxkZkYIJBQmhAYTeCyprnlNOOSUsHU8PBRcNGzZ0Zf+DDz5wAXAiChQo4Mqu0qYAAEA4ejAykJNOOslVntVqqp4C/V/LQqkH4qqrrnIV/sKFCwdTopS6lBTNkhMaGChVaseOHVFbiLU9zbxTpkyZYC+GUqMU/KinJZpatWpZ8+bNXcXsyiuvdAHDH3/8EXVdnYfSs84999zgMg0cr1+/vuv9CHXmmWeGHbN4xx1JAdqePXuCDw3uRcZXrFgx18PlPZSCp8A6clksCo7Vu5Uoraty6JU3AADwPwQYGYx6DdSqryAjWipQu3btXF64Ku+LFi1yDzlw4ECS21YFPpR6SbzUKrUQa+Yp76HBrkq/Um+FjkXrKdDQ7DxqFY5GwYt6IKZNm2bVqlWz5557zqpUqWIbN248al0vbUrHELk8clnocXvPeccdKU+ePC7wCn0g61Bvm3qxli5d6oLkr7/+2vV8KPhQb5xHgbB68zx33323S6dSWdV76oorrnDpdd26dUujMwEAIP0iwMhgWrVq5YIFPTQeIZQCC7WqPvDAA66CdMYZZxzVQ+ANZE1qzv+kWog11kHUW/Hzzz+7sRx6eIO7Y1EAoF4JpSkpB17Ho9SUSNqHnps/f35wmXo0NM5C5wWkhNLmNBZDwYTKmIILpTt99dVXrscjtAdt165dwd9VxpU+pYBYaVcqmwsXLgym+gEAgP9hDEYGo14AL0UocpyD8sg1tuLFF190qRtqodWYhFCqRKm1VjPnKL1JFS7loqeUUrCaNWvmvltAPQlq2Y1FLb8a06HUKh2Hft+5c2fUgEGVvltuucWNtVBwoxm0hg4d6qbFTSqIAULHJYVSz9unn36a5AWK/E4MjXUCAACJoQcjA4qV2qOUJVWElP6hAd19+/a1YcOGha2jnodnn33Wxo4d6ypbmgnqWKnCr54SfTeApoqNd9xffvmlaz0+/fTTXU+LZn+KNuOUPPHEE26grga3a5redevWuWlCFUgBAAAgfcoWiDVHKJAFKI9ePTjd2k+13LkKpPXhII4X3/3/mcIAAED6Rg8GAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN8QYAAAAADwDQEGAAAAAN9kCwQCAf82B2Qse/futSJFitiePXuscOHCaX04AAAAGR49GAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDc5/dsUkHEV+XydWYGCaX0YQJoLtDw9rQ8BAJDB0YMBAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGAAAAAB8Q4ABAAAAwDcEGACAVPH1119bs2bNrECBAla0aFFr2rSp/fvvvzHXHzRokGXLli3sUbJkSe4OAGQwBBiZiP4Yf/jhh2m2/4kTJ7pKBICsQQGD3vexgotWrVpZixYtbPHixbZkyRK77bbbLHv2+H92qlevbtu2bQs+Vq5cmUpHDwBILQQYGcT27dvt9ttvt4oVK1qePHmsbNmy1q5dO5s1a1aq7/vvv/+2SpUq2Z133hm2fNOmTVa4cGF7+eWX3e+dOnWyNWvWHHOFxWu5zJ07t9vvfffdZ/v37z9q3U8++cStX6hQIcufP7+dffbZMSs7AI6vvn37Wu/eva1///4uaKhcubJdccUV7vMrnpw5c7peC+9x8sknH7djBgD4gwAjA1BFvm7duvbFF1/Y0KFDXYveZ599ZhdccIHdeuutqbrvAwcOWMGCBW3ChAn23HPP2bx589zyQCBgPXr0sHPPPdduuOEGtyxfvnxWvHjxY97njTfe6Fou161b5873hRdecKkToXQs7du3t0aNGtmiRYvsu+++s86dO1vPnj3t7rvvPuZjAJByO3bscO9LfR7oPVqiRAlr0qSJzZ8/P8nXrl271kqXLm0VKlRw7+kNGzZwKwAggyHAyAB69erlWvSVZqAWwNNPP921CKpHYeHChWHr7tq1yy677DLXoq8WwylTpgSfO3z4sF1//fXuD7eCgSpVqtgzzzwT9vru3bvbpZdeakOGDHF/5LUvady4setBUVDxzz//uNetWLEi2HsRmSL1008/uWNevXp12PZHjhxp5cuXdwFKLDp2tVyeeuqpdvnll9tFF11kM2bMCD6/detWu+uuu6xPnz42ePBgq1atmp122mlu2bBhw2zEiBGucgMgbXhBgRoG1GCgBpE6depY8+bNXQARS4MGDeyVV16x6dOn20svveR6bhWg7N69+zgePQDgWBFgpHO///67++OsngoNlIwUOebh4Ycfto4dO7oW/TZt2tjVV1/ttiFHjhyxMmXK2DvvvGM//vijPfTQQzZgwAD3eyilXa1atcpmzpzp0pA8qsznypXLrrnmGvc69SKccsopUY9bwYt6XV5//fWw5W+88YZdddVVLvhIxLfffmsLFixw+/W89957dvDgwag9FTfffLPrcXnzzTejbk+pVnv37g17AEiMPgP0/vIe6tFUr2HkMn3WeO9HNUrUrl3bnnrqKfe5MH78+Jjbb926tWtUqFmzpl144YU2depUt3zSpEncIgDIQAgw0jmlCam1v2rVqgmtrx6ILl26uBZ9VQbU26CeD1ElXQGIxiqoF0PBh9aPDDAUyKhnQr0kNWrUCC7PmzevPf30024gucY+KNCIR9tXQOHR+IylS5cm+bpRo0a5iopytc866yzbuXOn3XPPPWHbKVKkiJUqVeqo12rchsapxBoLop4ZvdZ7aCwLgMQomFDPpfeoV6+ePfLII0ct896b6l0MdcYZZ9iWLVsSvtz6LFKwEa/XAwCQ/hBgpHNeKlGiLf5nnnlm2B9nDYBWPrRnzJgxrgKggZOqxCsNIfIPvv6gq6Iezbhx41wKk8aB7NmzJ+6xKH968+bNwTQu9WYoYIisdEQLTFRR0Sw06o257rrrXKtmcq5ZrOPXgHEdt/dQuhWAxBQrVsw1XngPb9xV5DKlQSrFUqmSoRT4lytXLuHLrR5H9aZGa0wAAKRfBBjpnMZRKLjQH9lEhKYSiV7rpSuop0Izu6jCrjENqsQrfUEDuUNFS8WSt99+243p0EBNtf5rW/GoUqCB6F4vhtKWkuq9EG1bFRXlbL/22ms2d+5cF9iEXhMFB7/++utRr9W5KP/bGzsSSb0imvkq9AHAX/rcUa/js88+61Ia1RP74IMPujFZGgfm0ZiM559/Pvi70h71ft+4caMbR6UxZ0pj7NatG7cIADIQAowM0GLYsmVLN5OS0p0i/fnnnwlvS7nRGjCpQePKiVYlfv369Qm99rfffnPjQB577DH3Wg3ofvXVV23atGlJ9kYoMFFvhPalXo3kUMCk8R4PPPCA7du3zy1TpUNTWWowdyT10Gi9a6+9Nln7AeAvTcKgHkM1RNSqVcuN7dK4Lk097dFngiam8Pz8888uxVNjNTp06OB6ItUDmpxeDwBA2suZ1geApGlMggKD+vXru3xnpUEdOnTI/bEePXp0wr0bCii8GVo0BkMBgr78Sv9PigZr6o++910YSrO699577aabbrLvv//e9TpEo0rCLbfc4h7qzYg1KDweDQpXkKHroBZOzS6l6Wv1f40L6dq1qwtEPvroI7eegqDQsSMAUsecOXPiPq/vwNAj3hTcod566y3fjg0AkHbowcgAFAAsW7bMVdA1Fasqz5q6VS2CCjCSM0BTFX59IZ6mg9TUj+rNSIqCEgUz6rUI/RbegQMHulms4qVKKQVJXwio2aDUm5ESasXUNwArqNCX/on2OXnyZNcro2DHm6ZWx6ggAwAAAGkjWyDeFxIAGYSm4lU+twIapW1pIHoilN/tel/eX2pWoGCqHyeQ3gVaRh+/BABAoujBQKYZq/L555+7IEPjPQAAAJA26MFAlkYPBhCOHgwAwLGiBwMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPiGAAMAAACAbwgwAAAAAPgmp3+bAjKuPReeZoULF07rwwAAAMjw6MEAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4Juc/m0KyLiafTbYcuTPY5nNorYPp/UhAACALIYeDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDABJevHFF61p06ZWuHBhy5Ytm/35559JvubQoUP2wAMPWIUKFSxfvnxWsWJFe+SRR+zIkSNccQAAMjECDACOAoiJEydGvRr79u2zVq1a2YABAxK+Wk8++aSNGTPGnn/+eVu1apUNHTrUhg0bZs899xxXHACATIwAI4Pr3r27a1Hu2bPnUc/16tXLPad1krtuStZXBbVPnz5Hrfvhhx+6deMZNGiQW0ePHDlyWNmyZe2GG26wnTt3JnAVkNp0X/v3728NGzZM+DVff/21tW/f3i6++GIrX768XXHFFdaiRQv75ptvUvVYAQBA2iLAyARUGX/rrbfs33//DS7777//7M0337RTTz01xeumZP1jUb16ddu2bZtt2bLFRo8ebR9//LFde+21Udc9fPgwqTbp3HnnnWezZs2yNWvWuN+//fZbmz9/vrVp0yatDw0AAKQiAoxMoE6dOq6yP3ny5OAy/V/BQe3atVO8bkrWPxY5c+a0kiVL2imnnGJt27a13r1724wZM1xwo9SdokWL2ieffGLVqlWzPHny2ObNm+2PP/5wQcgJJ5xg+fPnt9atW9vatWt9PS6kTL9+/axLly5WtWpVy5Urlysv6gnRMgAAkHkRYGQSPXr0sAkTJgR/Hz9+vF133XXHvG5K1veLBgZrQLAGC3vjAIYMGWIvv/yy/fDDD1a8eHGXoqWUmylTpriUnEAg4FrIDx48GHWb+/fvt71794Y9sqrBgwdbwYIFg4958+a5dLjIZSn19ttv22uvvWZvvPGGLVu2zCZNmmTDhw93PwEAQOZFgJFJdO3a1aWfbNq0ybXsL1iwwK655ppjXjcl6/th9erVLk2qfv36VqhQIbdMQcOoUaOsUaNGVqVKFfv1119dYKGA4/zzz7datWrZ66+/br/88osb9xGNApQiRYoEH+qJyaoUTKxYsSL4qFevnpvlKXJZSt1zzz1u3Ebnzp2tZs2arhz17dvX3QMAAJB55UzrA4A/TjrpJDeYVq3DasXX/7XsWNdNyfrxaHyFUpw8mpXIm5lo5cqVrtVc4yvU06BB45oe1ZM7d24788wzg79rZiKlVTVo0CC47MQTT3TBh56L5r777rM777wz+Lt6MLJqkFGsWDH3CO0xUq/Qaaed5sv21eOUPXt4G4YG8DNNLQAAmRsBRiaitKXbbrvN/f+FF17wbd1E19d3JOzZs+eo5frOBD0npUuXdi3jntAKrgID9UioEqr1NM4ilCrAobNRKdiJRstjzVqlbUZuF0nbvn27e6xbty4YDKpnSeNzvHvYvHlzu+yyy4LlpF27dvb444+7dTSAf/ny5TZy5Mjjkl4HAADSDgFGJqLvKThw4ID7f8uWLX1bN9H1NZh32rRpRy1fsmSJCx5EPQ6xWsjVQ5Gc1nP1hGh8xqJFi1zalOzevdvNWnTGGWckvB0kTd9n8fDDDwd/b9y4sfupsTneVMXr16+3Xbt2BdfR9108+OCDbkrjHTt2uKDx5ptvtoceeohLDgBAJkaAkYmo5d9LDdL//Vo30fVVkdSXqt1666120003uR6HmTNn2rhx4+zVV181v1WuXNl9z8KNN95oY8eOdS3qyvnXLFRajuSZM2dO3O8p0SMejdEJpfvx9NNPuwcAAMg6GOSdySgVyUtH8nPdRNbXl6lp1iG1ZOsL1c4++2w3vaweV155paUGtaDXrVvXTWt7zjnnuPSoTz/91E2LCgAAgOMvWyBWIjuQBWiQt2aTqvt2P8uRP/ONzVjU9n9pTQAAAMcDPRgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfJMtEAgE/NsckLHs3bvXihQpYnv27LHChQun9eEAAABkePRgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPANAQYAAAAA3xBgAAAAAPBNTv82BWQ8gUDA/dy7d29aHwoAAICvChUqZNmyZbPjjQADWdru3bvdz7Jly6b1oQAAAPhqz549VrhwYTveCDCQpRUrVsz93LJlixUpUiStDyfdU0+PgrGtW7emyQdWRsK14lpRrngPZhR8XmXea1WoUKE02S8BBrK07Nn/fxiSgouM8EGRXuhacb24VpQr3oMZAZ9XXCvK1fHHIG8AAAAAviHAAAAAAOAbAgxkaXny5LGBAwe6n+B6UbZ4H6Z3fGZxrShXvAczgmwBb55OAAAAADhG9GAAAAAA8A0BBgAAAADfEGAAAAAA8A0BBjK9UaNGWYUKFSxv3rxWt25dmzdvXtz1586d69bT+hUrVrQxY8ZYVjBkyBA7++yz3ZfyFC9e3C699FL76aef4r5mzpw5li1btqMeq1evtsxs0KBBR51zyZIl474mq5ar8uXLRy0jt956q2X1MvXll19au3btrHTp0u4cP/zww7DnNURSZU3P58uXz5o2bWo//PBDktt9//33rVq1am5AuH5+8MEHlpmv1cGDB61fv35Ws2ZNK1CggFvn2muvtV9//TXuNidOnBi1rP3333+WmctV9+7djzrnhg0bJrndrFauJFr50GPYsGFZrlwlFwEGMrW3337b+vTpY/fff78tX77czj//fGvdurX75u5oNm7caG3atHHraf0BAwZY79693QdrZqcKsCp9CxcutJkzZ9qhQ4esRYsW9s8//yT5WgUi27ZtCz4qV65smV316tXDznnlypUx183K5WrJkiVh10llS6688krL6mVK761atWrZ888/H/X5oUOH2siRI93zuo4KYi+66CL766+/Ym7z66+/tk6dOlnXrl3t22+/dT87duxoixYtssx6rfbt22fLli2zBx980P2cPHmyrVmzxi655JKEvoQvtJzpoUaAzFyupFWrVmHn/Omnn8bdZlYsVxJZNsaPH++ChcsvvzzLlatk0yxSQGZVv379QM+ePcOWVa1aNdC/f/+o6997773u+VA333xzoGHDhoGsZseOHZphLjB37tyY68yePdut88cffwSykoEDBwZq1aqV8PqUq/+54447ApUqVQocOXIk6rXKqmVK5/zBBx8Ef9f1KVmyZOCJJ54ILvvvv/8CRYoUCYwZMybmdjp27Bho1apV2LKWLVsGOnfuHMis1yqaxYsXu/U2b94cc50JEya465mZRbtW3bp1C7Rv3z5Z26Fc/T9dt2bNmsW9VlmhXCWCHgxkWgcOHLClS5e6VvhQ+v2rr76K2UoTuX7Lli3tm2++cd3wWcmePXvcz2LFiiW5bu3ata1UqVLWvHlzmz17tmUFa9eudd3qSr/r3LmzbdiwIea6lKv/vSdfe+01u+6661wrYDxZsUxF9npt37497PNIqSlNmjSJ+fkVr6zFe01m/fxSGStatGjc9f7++28rV66clSlTxtq2bet6GLMCpSIqFfb000+3G2+80Xbs2BF3fcqV2W+//WZTp06166+/Psnr+3cWLVehCDCQae3atcsOHz5sJUqUCFuu3/WHOxotj7a+0oW0vaxCDV933nmnnXfeeVajRo2Y66kC+OKLL7pUH6UlVKlSxVUIldeamTVo0MBeeeUVmz59ur300kuu3DRq1Mh2794ddX3K1f9TfvOff/7pcsBjyaplKpL3GZWczy/vdcl9TWajXPf+/fvbVVdd5VJVYqlatarLl58yZYq9+eabLoXl3HPPdY0HmZnShF9//XX74osvbMSIES79rlmzZrZ///6Yr6FcmU2aNMmNUezQoUPc65tVy1WknEctATKZyJZSVZ7jtZ5GWz/a8szstttus++++87mz58fdz1V/vTwnHPOObZ161YbPny4NW7c2DLzH2iPBpbqvCtVquT+ACkwi4ZyZTZu3Dh37dTzE0tWLVN+fX6l9DWZhXqa1aN45MgRN8FHPBrYHDq4WZXAOnXq2HPPPWfPPvusZVYaS+FRA1K9evVca7ta5+NVnrNyuRKNv7j66quTHEuRVctVJHowkGmddNJJliNHjqNa7tQVHNnC59Egymjr58yZ00488UTLCm6//XbX8qK0FHXvJpc+WLNaS41mrlGgEeu8KVdmmzdvts8//9xuuOGGZF/frFimvFnJkvP55b0uua/JTMGFBh4rvUyTCcTrvYgme/bsbia9rFbW1GuoACPeeWflciWafVITT6Tk8yt7Fi1XBBjItHLnzu2mBfVmrfHod6WzRKPW0sj1Z8yY4Vp4cuXKZZmZWqPUc6G0FHWda2xBSijXVH+wshKlFqxatSrmeWflcuWZMGGCy/m++OKLk/3arFim9P5TpS603GgMi2Z7i/X5Fa+sxXtNZgouVIlTIJuSBiF9Bq5YsSLLlTWldqqXMN55Z9VyFdr7qvqEZpxKrkAWLVfMIoVM7a233grkypUrMG7cuMCPP/4Y6NOnT6BAgQKBTZs2uec1m1TXrl2D62/YsCGQP3/+QN++fd36ep1e/9577wUyu1tuucXNfDFnzpzAtm3bgo99+/YF14m8Xk899ZSboWTNmjWB77//3j2vj5X3338/kJnddddd7jqpvCxcuDDQtm3bQKFChShXMRw+fDhw6qmnBvr163fUc1m5TP3111+B5cuXu4fOceTIke7/3sxHmkFK78nJkycHVq5cGejSpUugVKlSgb179wa3oWsXOiveggULAjly5HCvXbVqlfuZM2dOV04z67U6ePBg4JJLLgmUKVMmsGLFirDPr/3798e8VoMGDQp89tlngfXr17tt9ejRw12rRYsWBTLrtdJz+vz66quvAhs3bnSztp1zzjmBU045hXIV5T0oe/bscfWC0aNHR73eWaVcJRcBBjK9F154IVCuXLlA7ty5A3Xq1AmbdlXT9TVp0iRsfVUca9eu7dYvX758zA+VzEYfrtEemnIv1vV68skn3ZSjefPmDZxwwgmB8847LzB16tRAZtepUydX0VPwWbp06UCHDh0CP/zwQ/B5ylW46dOnu7L0008/HXUts3KZ8qbkjXzomnhT1WpKZE1XmydPnkDjxo1doBFK185b3/Puu+8GqlSp4sqnpt3ODMFZvGulinKszy+9Lta1UoOTAl991p988smBFi1auIp3Zr5WajDSeep8VT50/lq+ZcuWsG1Qrv5XTsaOHRvIly9f4M8//4x6vbNKuUqubPonrXtRAAAAAGQOjMEAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAAAA4BsCDAAAAAC+IcAAAByTbNmy2Ycffpihr+KmTZvceaxYscL9PmfOHPf7n3/+GVxH53jaaadZjhw5rE+fPqlyHBMnTrSiRYtmqGvevXt3u/TSS9Ns/w8++KDddNNNaXIfjsWOHTvs5JNPtl9++SVV9wOkBQIMAEiFCpcqgHrkzJnTTj31VLvlllvsjz/+SDfXOj1VUNOjRo0a2bZt26xIkSLBZTfffLNdccUVtnXrVnv00UdTpWLdqVMnW7NmTfD3QYMG2VlnneXLtps2bRosl3ny5LHTTz/dBg8ebIcPH05REJYe/Pbbb/bMM8/YgAEDUvU+JHp9kxN4Fi9e3Lp27WoDBw5MwREC6RsBBgCkglatWrkKqiplL7/8sn388cfWq1evNL/WBw4csMzs4MGDvmwnd+7cVrJkSVehlr///tu1OLds2dJKly5thQoVstSQL18+V/FMLTfeeKMrlz/99JP17t3bHnjgARs+fLhlVOPGjbNzzjnHypcvn6Hug6dHjx72+uuvp6vGB8APBBgAkArUQqwKapkyZaxFixauRXTGjBlh60yYMMHOOOMMy5s3r1WtWtVGjRp1VGvxW2+95VrTtU716tVd6k6ouXPnWv369d3+SpUqZf3797dDhw6Ftaredtttduedd9pJJ51kF110UbAydtlll7l9hFbOFAjVrVvX7a9ixYr28MMPh21v7dq11rhxY/d8tWrVbObMmUlei/fee89q1qzpKm0nnniiXXjhhfbPP/8Enx8/frw7N+8cdLyeLVu2WPv27a1gwYJWuHBh69ixo2u1jmzh1zZ0vNpGIBCwPXv2uLQZVRL1umbNmtm3335riQpNkdL/vYBC29FyXddJkybZRx99FOwViLw33vVUqs2RI0fc72r917r33HNPWM9Ily5djkrN0f91/XXc3j60zLNr1y53D/Pnz2+VK1e2KVOmJHleWlflUvdc17l58+auJ0v3Q9dJ9yry+AsUKGB//fWXVahQwS2rXbt28BqEUqCi+6d7fOutt4YFe6pAX3vttXbCCSe4Y2jdurUrSx7vvKdPn+7eE7rfXpAej94fl1xySdgyr8zroW3qeBRIqVwk93giy9mrr77qrp16tjp37uyui6g3S+9F9aZ490rvYe3n6quvdqlQKv+6T3rfe/S+0P344IMPkrx3QEZCgAEAqWzDhg322WefWa5cuYLLXnrpJbv//vvt8ccft1WrVrlUFeWSq9IaShXRu+66y5YvX+4CDVWmdu/e7Z5T7nabNm3s7LPPdpXQ0aNHuxbdxx57LGwb2qZStRYsWGBjx461JUuWuOWq6KgC5/2uyt0111zjWrZ//PFHt64qWjpGUSW5Q4cObgzCwoULbcyYMdavX7+4567tq/J83XXXufNUJVzb8Cp7OmZVRhUMrFy50lWSNc5BtI5SkH7//XdXeVMws379eheshVq3bp2988479v777wfTdy6++GLbvn27ffrpp7Z06VKrU6eOq0xrW8ml664Wf9E+dE46TgU7XiVYD60XScGYKqG6f6LzUKCnnx5dkyZNmhz1Wp2n7r2CL28foeeu4EPH8N1337lyoIpscs9PlV4FAgoiVGEOrfyKfldamAKsxYsXu2Wff/65O5bJkycH15s9e7a7N/qp8qZyExoMqQL+zTffuOv29ddfu3urYw4NQvbt2+eCFFXiv/zySxdc3n333TGPXZX377//3urVq3fUc16ZX7RokT377LP21FNPuZ7E5BxPJJ2fgrFPPvnEPXQPn3jiCfecAgv1pHg9RHqULVvWvaf1Xpo2bZor/yrvuv+h1EAwb968mPsFMqQAAMBX3bp1C+TIkSNQoECBQN68eVWTdo+RI0cG1ylbtmzgjTfeCHvdo48+GjjnnHPc/zdu3Ohe88QTTwSfP3jwYKBMmTKBJ5980v0+YMCAQJUqVQJHjhwJrvPCCy8EChYsGDh8+LD7vUmTJoGzzjrrqGPUtj/44IOwZeeff35g8ODBYcteffXVQKlSpdz/p0+f7s5r69atweenTZsWdVuepUuXuuc3bdoU9fnSpUsH7r///qjPzZgxw+1vy5YtwWU//PCD297ixYvd7wMHDgzkypUrsGPHjuA6s2bNChQuXDjw33//hW2vUqVKgbFjx0bdl3e9ly9f7n6fPXu2+/2PP/5wv+unftfy0Pvcvn37QFLq1KkTGD58uPv/pZdeGnj88ccDuXPnDuzduzewbds2t91Vq1a55ydMmBAoUqRI8LU6v1q1ah21Tb3mgQceCP7+999/B7Jly+buRywqC3fccYf7v8qH1tVx3HvvvW7ZokWL3PX+5Zdf3O87d+5013bOnDlRr1HodShXrlzg0KFDwWVXXnlloFOnTu7/a9asca9bsGBB8Pldu3YF8uXLF3jnnXeC56111q1bF1aWS5QoEfN8dBx6TWj58M7zjDPOCHtf9OvXzy1LzvFE3of8+fO7e+a55557Ag0aNIh6fT3t2rUL9OjRIxBP3759A02bNo27DpDR0IMBAKngggsucK3pakG9/fbbXe6+fsrOnTvdQOHrr7/epYJ4D/U8qJU0lFpFPWqRVWutWkJFP/W8N05Azj33XDde4Oeffw4ui9bCG41a+h955JGwY/JaZNW6rP1pwLrSvqIdXzS1atVyPQdKBbnyyitdz42Xb64xDb/++qt7PhrtT63AeniUlqXUFe8aSLly5VwKSuh56BooNSb0XDZu3HjU9T0elLKjXgrFBWqpVspXjRo1bP78+a7Fv0SJEi5FLrnOPPPM4P/VA6FeBl3TeJSGp2uhFDf1hqnHyhtkrJZ09Za88sor7nf1JOh+qxcmKXqderY8SpXyjkX3SmW3QYMGwed1b6pUqRJ2H5WqVKlSpajbiObff/91P3UukRo2bBj2vlA5VQqUBrQnejyRlBoVOvYmqeMTTe6gNC6lV91777321VdfRe1F0vsLyExypvUBAEBmpAqfl+qjFA0FHEpp0exDXj6+KtuhlRwJraTF4lWcVGENrUR5y0LX8Y4lETouHaNSmCKpEheawx55LLHofJTapIqVxqA899xzLjVMgVdkqkikaOcXbXnk+ek8VPmLNiYitacejRVgKHVNaWzZs2d3QZJSopRio2ArWnpUIkJT7kTXxCtbsSiNStdfY1U0WD2yvN1www32/PPPu7E8So/SIOSk7nFSxxKt3HjLQ7cdbRuxXite+dE1DA0wk5Lo8fhxvTW2Y/PmzTZ16lSXWqZgWimBoQPrldaWnOMHMgJ6MADgOFArsSoVarFXi/Upp5zixmYoCAl9eANpPRrr4NFga7XOe63dqqiq4h5aYdLvamXV9uNRZSlyelKNU9BYg8hj0sOrGCsvXufgUf56UlQRU8+KgheNRdAMTRrUquNUq/CsWbOivs7bn3p7PMpn1wBuDQSOReeh8RdqpY48j6SCmuTQeSQyxas3DuPpp592wYSuh34qAIo1/iK5+0iUBifrOqhXKFowqx4NXXMFxT/88IN169Yt7Fgkucej+6iyq6DSo3FEmgY23n1Mino7NDBdZSJS6PvG+10DrHXOqXU8se6VggeN+XjttddcGXjxxRfDntc4Eg2cBzITAgwAOE6t2Eoj0WBub1aaIUOGuMGhqthogLNajEeOHBn2uhdeeMFVxlevXu1aPtVaqwHTomlvVflW6pWe14xGCmQ0Y5QCgni8ir0q4l7K0kMPPeTSY3RsqlwqXeTtt992M/CIZn9SGolm31FrvNJ91BoejypxOmcNqFXFVQODlSLmVeS0rxEjRrgKrVJYli1b5no5vP0pDUit7lquQcbatyrk8dK+9DqlxGiAuAauazYfBV46Dx2HX3QNNcBaQZlmdIo1QFiVeqXIqILpzbykoEPnpHsfORtT5D6U2qV0O+1j//79lpo0q5J6sDS5gGY/C02H04xcSufRhAWayUuBXiJUsVdamNLtlBamsqNARkGwlqeUyrjutbYZSe8LvQ90b958801Xpu64445UPR7dK5V3lTfdK/Vu6D2l96UmItB7SoPDQ4MYpUap0UDXGshMCDAA4DhRhUdpUar8KBVFs9poph2NT1ClWf+P7MHQLDVPPvmkG8ugCr0qK14rvCpEmiVJFW8937NnTzeuwwsI4lGlXqlLasn2Wk81TkQVIC3XzFTKY1fAozEOXoVOwY4qucrX1zl4M0zFohZmzQikGXr0xW46Nu1bqSOiFnK16mpsgAKwtm3bBqcL9b4MUJVeVchVmdRUtAp64tHrdF30GgVj2q9mSFLFT71HflEFVQGXgh21UmuWrliUIqfWbS+Y0DmpJV2vi9dqfvnll7uZqvR6ravKcmpTGdL3pXiBrEc9QgoENbuY0quSUxlX8Kzpj3V/Ffyp1033KDLtKLk0+5jGOESmKikQ1RgNlVMF5grCQ7/tOzWORzNeeT0kulcKqNWrcd9997lAWeVRz+t4PXo/a5zL+eefn+L9AulRNo30TuuDAACEU2VYwYZSivz6JmcgEfriN7X2KxXOS4tKr1SFUSCsb9D2vktEQZzeMwpc0zsFQDr2q666Kq0PBfAVPRgAAMCl6yiNR6l7+vK/9B5ceL1VGtMQ+mWQGYVmoNJ3jHiBEZCZEGAAAAAbOnSoa/lXGpnSejIKpQd27drVMhqNadHUtYnM0gVkNKRIAQAAAPANPRgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAfEOAAQAAAMA3BBgAAAAAzC//B/lXxpBaYiVZAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "fig, ax = plt.subplots(figsize=(8, 3.8))\n", "names = list(tool_lifts)\n", "values = [tool_lifts[name] for name in names]\n", "bars = ax.barh(names, values, color=['#36B37E', '#00B8D9', '#6554C0', '#FF8B00'])\n", "ax.set_xlabel('Reported score lift with Python (points)')\n", "ax.set_title('Kimi K3 · paired tool augmentation from Table 2')\n", "ax.bar_label(bars, fmt='+%.1f', padding=4)\n", "ax.spines[['top', 'right']].set_visible(False)\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "82e3d1ae", "metadata": {}, "source": [ "## 3. Pareto efficiency is a relationship, not a ratio\n", "\n", "The fixture below is deliberately illustrative; it is **not** Figure 13 data. A point is excluded only when another point is at least as accurate, no more expensive, and strictly better on one dimension." ] }, { "cell_type": "code", "execution_count": 5, "id": "9c6bcefb", "metadata": { "execution": { "iopub.execute_input": "2026-08-11T03:53:05.727126Z", "iopub.status.busy": "2026-08-11T03:53:05.726860Z", "iopub.status.idle": "2026-08-11T03:53:05.732239Z", "shell.execute_reply": "2026-08-11T03:53:05.730907Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Illustrative frontier: ['efficient-small', 'balanced', 'premium']\n" ] } ], "source": [ "illustrative = [\n", " {'name': 'efficient-small', 'score': 88, 'cost': 1.2},\n", " {'name': 'balanced', 'score': 92, 'cost': 2.1},\n", " {'name': 'dominated', 'score': 90, 'cost': 3.4},\n", " {'name': 'premium', 'score': 95, 'cost': 8.0},\n", "]\n", "frontier = pareto_frontier(illustrative)\n", "print('Illustrative frontier:', [point['name'] for point in frontier])\n", "assert 'dominated' not in [point['name'] for point in frontier]" ] }, { "cell_type": "markdown", "id": "ed3acf91", "metadata": {}, "source": [ "## 4. A local metric miniature is not evidence about K3\n", "\n", "Strict exact match is useful for showing how a metric transforms predictions into a number. It says nothing about model quality when the strings are authored fixtures." ] }, { "cell_type": "code", "execution_count": 6, "id": "130bde53", "metadata": { "execution": { "iopub.execute_input": "2026-08-11T03:53:05.734839Z", "iopub.status.busy": "2026-08-11T03:53:05.734654Z", "iopub.status.idle": "2026-08-11T03:53:05.738971Z", "shell.execute_reply": "2026-08-11T03:53:05.737521Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Authored fixture exact match: 66.7% (2 of 3)\n", "This is metric execution, not a K3 evaluation.\n" ] } ], "source": [ "predictions = ['Paris', '4', 'extra explanation']\n", "references = ['Paris', '4', 'concise answer']\n", "rate = exact_match_rate(predictions, references)\n", "print(f'Authored fixture exact match: {rate:.1f}% (2 of 3)')\n", "print('This is metric execution, not a K3 evaluation.')\n", "assert rate == 100 * 2 / 3" ] }, { "cell_type": "markdown", "id": "556c58f0", "metadata": {}, "source": [ "## What this notebook establishes\n", "\n", "- deterministic arithmetic for within-row gaps;\n", "- exact paired tool lifts from selected paper-reported values;\n", "- the mechanics of Pareto dominance on an explicitly illustrative fixture;\n", "- strict exact-match behavior on authored strings.\n", "\n", "It establishes **no K3 benchmark score, rank, causal training effect, or cost measurement**." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 5 }