{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e9fc3b8b",
   "metadata": {},
   "source": [
    "# Structuring ML projects: diagnose, analyse errors, fix data mismatch\n",
    "\n",
    "A small, real project: train a digit classifier, compare it with a human-level estimate, split the data the right way (train / train-dev / dev / test), find out whether the problem is bias, variance or data mismatch, do an error analysis like the slides, fix the mismatch with synthetic data, and try transfer learning.\n",
    "\n",
    "[Open in Google Colab](https://colab.research.google.com/github/mahmoudyosrimahmoud13/inpace-library/blob/main/courses/dl-3-ml-projects/files/dl3_ml_strategy.ipynb) · part of the inPace course library. Slides © DeepLearning.AI (CC BY-SA 2.0); this notebook: inPace (CC BY-SA 4.0).\n",
    "\n",
    "**Run it top to bottom** (Runtime → Run all in Colab). Each part says what to look for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2a40d80c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-13T17:19:01.433652Z",
     "iopub.status.busy": "2026-11-13T17:19:01.433311Z",
     "iopub.status.idle": "2026-11-13T17:19:04.927706Z",
     "shell.execute_reply": "2026-11-13T17:19:04.926391Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch 2.15.0+cu132 | device: cpu\n"
     ]
    }
   ],
   "source": [
    "# Works in Google Colab as it is (PyTorch is already installed there).\n",
    "# On your own computer:  pip install torch torchvision scikit-learn scipy matplotlib pandas\n",
    "# One framework per notebook: PyTorch and TensorFlow can clash when loaded into the same Python process.\n",
    "import numpy as np, matplotlib.pyplot as plt, torch\n",
    "from torch import nn\n",
    "torch.manual_seed(0); np.random.seed(0)\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"mps\" if torch.backends.mps.is_available() else \"cpu\"\n",
    "print(\"PyTorch\", torch.__version__, \"| device:\", device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d22d75a",
   "metadata": {},
   "source": [
    "## 1. The situation\n",
    "Your **training data** is clean scanned digits. Your **users** take photos with cheap phones: the digits are noisy and a little blurred. The dev and test sets must look like the users' data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "814dc75c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-13T17:19:04.929914Z",
     "iopub.status.busy": "2026-11-13T17:19:04.929624Z",
     "iopub.status.idle": "2026-11-13T17:19:06.457433Z",
     "shell.execute_reply": "2026-11-13T17:19:06.456633Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxsAAADxCAYAAACj6aqwAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAJxdJREFUeJzt3Xl8VeWdx/FvSFgFEQnDFsLWhiJWkEXCIoKCWhw2ZXlpjSE6SLVY0BdOkQGBVpEOIuA2pYwxuFAtzoQAUguB4rAZpGE1LEEJBTQhARKBUJZw5o8xd4xJ4HfIfUggn/dfcPPN8zz33HvPub+ce+4vxPM8TwAAAAAQZFXKewEAAAAArk0UGwAAAACcoNgAAAAA4ATFBgAAAAAnKDYAAAAAOEGxAQAAAMAJig0AAAAATlBsAAAAAHCiQhUb6enpSkhI0MmTJ4M+dm5urhISEpSbm1th1nQ5Nm/erGXLlpX3MgAAAIBL8lVs7N69WwkJCcrPz3eymE8//VRxcXHKyckJ+thTp07V3LlzVbdu3QqzpstRpUoVDR48WFu2bCnvpQAAAAAX5avYSE5OVlxcnI4dO+ZkMVFRUYqNjVXt2rWDOu7+/fv15ptv6vnnn1dISEhQx77SOnbsqP79++vZZ58t76UAAAAAF1WhPkbVq1cvJSQkKDw8PKjjvvnmm6pXr54GDBgQ1HHLS1xcnFatWqUvvviivJcCAAAAlCrMGty8ebNSUlIkSYsWLVK9evUkSffcc48aN26sL774Qp9//rkefPBBeZ6nNWvWKDs7WzExMTp06JCSk5MlSSEhIapVq5ZuuukmtWvXrsgc6enpWr9+vYYOHRo4u/HDcVevXq1Tp06pR48eatKkySXXfeHCBSUkJGjo0KEKCyv57p46dUopKSk6cuSIWrVqpc6dO6tKlUvXYSdPntSGDRuUk5Ojpk2bqnv37qpatWrg59b77fc+3nfffapVq5bi4+M1a9asS64TAAAAKA/mYuPLL79Uenq6JGnjxo2qVauWJOm2225T48aN9ec//1nPPvusWrRooaefflrNmjVTamqqYmJidPToUa1Zs0aS5HmesrKy9Omnnyo6OlrLly9XzZo1pe+ujxg1apR69+4dKDYKx42KitLYsWMVERGhL7/8Unv27NH777+voUOHXnTd27dvV05Ojrp3717iz+fNm6df//rXatCggW655RYdPnxY586d06JFi9SqVatSx50/f77Gjx+vZs2aqW3bttqyZYsuXLigxMREtW/fXpLM99vvfaxWrZo6d+4cKGQAAACACsnz4bXXXvMkeQcPHiz2s5kzZ3qSvP79+3vZ2dme53leRkZGqWNlZGR44eHh3oQJEwK3zZ8/35Pk7d+/v9i4gwcP9nJycjzP87yCggLv3nvv9Zo2ber94x//uOiaC8dMTU0t9rNFixZ5krzJkyd7Fy5cCNy+fft2Ly0trdQ1JSYmepK8F198MXDb2bNnvUGDBnnNmjXz8vPzfd3vy7mPY8aM8UJCQrzTp09f9P4DAAAA5cV8ZsPqoYceClxz0bx58yI/27Ztm/bu3av8/Hx5nqdmzZoF/vJ/KTExMapfv7703TcyPfrooxo+fLjS0tJ06623lvp7WVlZkhT43e+bPn262rRpo6lTpxa5cPynP/3pRdfy4osvqk2bNnruuecCt1WtWlXTp09Xu3bttGTJEo0YMcL3/fZzH+vXry/P83TkyBFFRkZedL0AAABAeQh6sdGxY8dit+3bt09DhgzRwYMH1a1bN4WHhys0NFTHjx83j9uhQ4ci/4+IiJC+uy7iYsXG+fPnJanY9Rrnz5/X1q1bFRMTY7o+4/u/l5qaqq5du+r999+X53nSdx+TKpyr8MJtv/fbz30svDakcE4AAACgogl6sdGgQYNit40ePVonT57U/v37AxeWS1K/fv0C14FcyvXXX1/k/4Vvts+cOXPR3ys8y3Ls2LEiF1ufP39enuepRo0apvkLnTt3ThcuXFBubm6J10zExsYGLgD3e7/93MejR49KpWxvAAAAoCLwVWxcbo+KlJQU/fznPy/yhrugoEBpaWlFvr3JhZtvvln67ixD4b8lqUaNGoqMjPT99bE1a9ZURESEIiMjlZCQcNGsy/u9b98+RUZGqk6dOmUaBwAAAHDFV5+Nwr+i+/n4k777ONAP/5L/hz/8QXl5eb7GuRxdu3ZVjRo1Al/b+32jRo3S+vXr9ec//7nI7adPn1Z2dnapYz7xxBNKTk7W2rVri/3syJEjge3j8n5/9tln6tOnT5nHAQAAAFzxdWajV69eql27tsaPH68HHnhA1apVC/TZuJiJEydq5MiRevDBB9WzZ0+lpqbqyJEjuu+++0osAoKpRo0auv/++5WUlKSXXnqpyM8mTJignTt3auDAgYqNjVWHDh10+PBhLV68WO+8806pH1H69a9/rfT0dN11112KiYlRhw4dlJ+frx07dmjdunVavXq16tWr5+x+b9iwQdnZ2fr5z39+2WMAAAAArvk6s9GoUSOtX79enTt3VkpKitasWRP4K/7NN9+s2NjYEq+BeOSRR7R+/Xo1b95cu3btUu/evZWYmKjevXsX6SERFRWl2NjYQI+Ni40bHh6u2NhYtWjR4pLrHjNmjHbt2qVNmzYVuT0sLEwffPCBVq1apX/6p3/Szp07Vb9+fa1cuVJdunQpdU2hoaF6++23tWHDBkVGRmrHjh369ttvNWDAAO3ZsyfQn8N6v/3ex3feeUdRUVHq27fvJe87AAAAUF5CvMKvU7rGDRgwQGFhYUpMTCzvpZTJN998o9atW+vdd9/VAw88UN7LAQAAAErl68zG1WzWrFm64YYbfF9vUtFs3rxZY8aModAAAABAhVdpzmwAAAAAuLIqzZkNAAAAAFcWxQYAAAAAJyg2AAAAADhBsQEAAADACYoNAAAAAE746iAebLm5uaZc7969TbmRI0ea5x43bpw5WxlkZGSYcoMHDzblOnToYMolJCSYcpWF9XGQpJYtW5pyU6ZMMeWmTp1qnrsy8LM/Wbx4sSm3detWU87SrPRaEOz9jh/WbeznsZgzZ04ZVnR1sB635eM4YN33+HlNVgZ+jhfW91HWXGU5dq9Zs8aUc7GPsr5PvRqO3ZzZAAAAAOAExQYAAAAAJyg2AAAAADhBsQEAAADACYoNAAAAAE5QbAAAAABwgmIDAAAAgBMUGwAAAACcoNgAAAAA4ES5dhC3dqC0dix10cGxsgh2B8rK0l002Pxst7p165py1i6+lYW1y/OCBQuCPre1G21l6ZRs3e9s27Yt6HNbx7S+zuTjWHU17x/9rP3AgQOmHMfuy+NnP0Fn8Mtj3Wfn5eUFfe5p06aZctbXT3m+F+DMBgAAAAAnKDYAAAAAOEGxAQAAAMAJig0AAAAATlBsAAAAAHCCYgMAAACAExQbAAAAAJyg2AAAAADgBMUGAAAAACcoNgAAAAA4EVaekyckJJhyc+bMMeVatGhRxhVVXgsWLDDl3n77bedruRZlZGSYctOmTTOPOXv2bFNu8ODB5jErg8WLF5ty7du3N495ww03mHJbt241j1kZ5ObmBn3MsWPHmnLW4wqKsh63JSk2NtaUs75+Kgvr8cLP/sTP44b/N27cOFNu6tSpppyffZ71Pa31+dKhQwfz3MHGmQ0AAAAATlBsAAAAAHCCYgMAAACAExQbAAAAAJyg2AAAAADgBMUGAAAAACcoNgAAAAA4QbEBAAAAwAmKDQAAAABOlGsH8W3btplydAa/PC66FVs7VVpzleWxddEp2craMbt3797mMa/mjr/W++nnMVuzZo0pV1me71bWrrvW7StJc+fONeWsz2HrGisLP6+L8uxYfDWz7rP9bF/r8936WrMe40eOHGnKVVTBPtb5Gc/Fsaq8cGYDAAAAgBMUGwAAAACcoNgAAAAA4ATFBgAAAAAnKDYAAAAAOEGxAQAAAMAJig0AAAAATlBsAAAAAHCCYgMAAACAEyGe53nBHNBP1+pbb73VlIuNjTXlrB0t/ZgzZ44pVxE7pVrXLklPP/2007WUZtCgQeastatqRWR9LMrrcZCkunXrmrPWLrMV8XXhQkhIiCmXmJhoyg0ePLiMK7q2+Nm3jxs3zpSzHqtcHFcqIuv2sB635ePY3aJFC1POz+viat73lOfxws9xwML6epSkqVOnBnXuisjPe2RrB3HrmNbXmQuc2QAAAADgBMUGAAAAACcoNgAAAAA4QbEBAAAAwAmKDQAAAABOUGwAAAAAcIJiAwAAAIATFBsAAAAAnKDYAAAAAOBEWHkvwCI3N9eU89Op0toB2TqmdbwryUUH1f3795ty1q67frqcWx8LP2NeKS46d95xxx2mnLUra0JCgnluayffq7n7sp/tYe26a+0Ii6L8vH6s+72kpKQyrOja4+J4Eewu7X5ek9bjhZ/3DVeKdU1+Ooi3b9/elLM+Ztb3PJWhK7h8bA/rsVOS8vLyTDnrccXPYzFy5Ehz1oIzGwAAAACcoNgAAAAA4ATFBgAAAAAnKDYAAAAAOEGxAQAAAMAJig0AAAAATlBsAAAAAHCCYgMAAACAExQbAAAAAJyg2AAAAADgRFiwB2zRooU52759+6DOfcMNN5iz1tbyfu5PRdOhQwdz1vpYjBs3Lqhz5+bmmnKStHXrVnO2ohk8eLAp17x5c/OY1m3nZxsHe+6rWVxcnDlrfdysz2EXrwvrc9DPfuNKcbE9Bg0aVIYVVV5+9lG9e/c25azPzTlz5pjnTkhIMOWsx7SKKDY21pzNyMgw5azvjay5q/m4LR/PIz/HC6u6deuactZ9dnnu2zmzAQAAAMAJig0AAAAATlBsAAAAAHCCYgMAAACAExQbAAAAAJyg2AAAAADgBMUGAAAAACcoNgAAAAA4QbEBAAAAwImgdxB30cXb2l20T58+5rmt3WP9dCytaMrzsZg2bZop56cbrbWT59Vs8eLF5uzIkSNNuSFDhpRhRSWbPXt20MesaKZMmWLOWp/vfvZRwWbtwn0l93nWNdWrV888prXrrp/XGv6f9VghH/uouXPnmnJ33HGHee6r+dhtNXXqVHPW2induo+yvs6u9uO2dR9lfS/jp4v3tbSP4swGAAAAACcoNgAAAAA4QbEBAAAAwAmKDQAAAABOUGwAAAAAcIJiAwAAAIATFBsAAAAAnKDYAAAAAOAExQYAAAAAJ0I8z/PKexEAAAAArj2c2QAAAADgBMUGAAAAACcoNgAAAAA4QbEBAAAAwAmKDQAAAABOUGwAAAAAcIJiAwAAAIATFBsAAAAAnKDYAAAAAOAExQYAAAAAJyg2AAAAADhBsQEAAADACYoNAAAAAE4Epdj47LPPFB0drdTU1GAMd03ZuHGjbr/9dh0/fry8l1IhLVy4UNHR0crLyyvvpfj2wQcfaNCgQSooKCjvpQAAAFRIQSk2cnNzlZKSom+//TYYw11SbGyspk+fLkn66KOPFB0drezsbKdzXs48nudp3Lhx+slPfqJ69eo5Xd+V8Je//EXR0dE6ffp00Mb8+uuvlZKSonPnzgVtzCv1nOjfv7/WrVunefPmOZ0HAADganXVfYwqKytL7733nn70ox9JkjIzM5WSkqIzZ844nfdy5vn444+1adMmjR8/3unarpTs7GylpKRU+L/kX6nnxPXXX69Ro0bpt7/9bVCLJQAAgGvFVVdsLF26VGFhYbr33nvLeymX9Oabb6pr165q06ZNeS8FjsTExCgzM1OLFy8u76UAAABUOL6LjczMTD3zzDPq06ePhg8frrVr15aaPXv2rObPn6/7779fvXr10oMPPqhly5YFfr5y5UpFR0crJSWl2O+mpaUpOjpaS5YsKXJ7UlKS7rzzTl1//fV67bXXNGvWLEnSwIEDFR0drejoaH3++efmNUjS+fPntWDBAo0YMUJ9+vTRI488oj/96U/yPE+STPP80LfffqsVK1aof//+xX42Z84c3XHHHcVuX79+vaKjo7V9+3bz2vzcz+9fH7Fo0SINHjxYPXv29DWPxffnWbhwof75n/9Zffv21YwZMy76EazExEQNHDhQd911l1544YUSz0zk5uZq+vTp+tnPfqbevXvr8ccf1+bNmwM/tzxWlxrDz/Zo166dmjVrpkWLFvneTgAAANe6MD/hw4cPq2vXrqpTp44mT56s2rVra+bMmSX+5f7UqVPq16+fvvrqK02aNElt27ZVSkqKhg0bpokTJ2ry5Mnq1q2bvvjiC82bN09du3Yt8vvz58/Xli1b1KNHj8Bt+fn5Sk5O1uzZsyVJgwYN0oEDBzRr1ixNmzZNDRo0kKTAeixrkKRf/epX+vDDD/XCCy+obdu2+vrrr7V48WKlp6fr3/7t3y45T0nWr1+vgoIC3XbbbcV+lpGRUWKBdfz48WLXvlxqbX7uZ+H1ETNnzlROTo4ef/xxJSUlmeexKpxnxowZysvL01NPPaUvv/xSkyZN0vLly7Vq1SpVrVq1yO/8/ve/1zfffKNf/vKX2rt3ryZMmKD9+/frrbfeCmSysrLUvXt3FRQUaNq0aQoPD9d7772n6OhoJSQk6OGHH77kY2UZw+/2iI6O1po1a3xtIwAAgErB8yE2NtarVauWl5mZGbjt/PnzXseOHT1J3l//+tfA7c8884xXtWpVb+fOnUXGeOONN7wqVap46enpnud53qOPPurVrl3bO3HiRCBz5swZLzw83Bs+fHiR3/3v//5vLyQkxDt06FDgttdee82T5B08eLDYeq1rqFOnjjdhwoRiv3/y5EnTPCWZM2eOJ8lLS0sr9rOxY8d61atXL3b70qVLPUne2rVrA7dZ1ma9nzNnzvQkeaNGjQpkzp8/b54nOzvb27hxo1dQUHDR+144z+jRo4vcvnLlSk+S98YbbxTLPvXUU0WykydP9kJDQ4s810aOHOlVrVrV279/f5Fs//79vdq1a3vHjh3zvEs8VtYxLNuj0Pjx4z1Jgd8FAADA//H1MaqkpCT1799fDRs2DNwWGhqqmJiYYtn3339fd9xxh9q1a1fk9uHDh+vChQtavny5JOmxxx7TyZMn9eGHHwYyS5YsUU5Ojh577LEiv7tkyRJ17txZTZs2Na3Xuobw8HAtXbq02Ff3XnfddaZ5SnL06FFJKvO3UFnWZr2fhQr/eq/vHj/rPOHh4YqOjlaVKranzQ+fF3379lXz5s0DZ1NKW5Mk9ejRQwUFBdq7d2/gtqSkJPXr108tWrQokh09erROnjyp5OTkS67JOoaf58SNN94oScrJybnk/AAAAJWJ+WNUp06dUm5urpo3b17sZz+87fTp08rKypIk9ezZM/A5d8/zAv/++uuvJUndu3dX27ZtFR8fHygu4uPjFRkZqb59+wbGLCgo0LJlyzRu3DjTev2sYcGCBYqLi1OnTp3UpEkT9erVS/fff7+GDh2qkJAQ6yYqovBNaX5+/mX9fqFLrc3P/Sz0wzfalnkuR2nPlYMHDxa7PTIyssj/69evL0k6cuSI9N12PH78eIlrb9mypSSVOO73+RnDz/YofIzLUpwCAABci8zFRvXq1VWlSpUSm6/98LZq1aopNDRUXbp0KfXz/k2aNAn8+7HHHtP48eO1e/du1alTRytWrNCkSZOK/AV9w4YNysnJ0aBBg0zr9bOG22+/Xfv27dP27du1bt06ffzxxxo+fLji4uIUHx9vmu+HGjduLH331+5WrVoV+VnNmjV17tw5FRQUBM4sqJS/jF9qbX63tSTVqlXL9zyXIy8vTxEREcVuK2n+sLCSn4qFBVP16tUVGhpaYi+X3NxcyfBm388YfrZHdna2wsLCAteHAAAA4P+Yi42wsDB16tRJGzZskOd5Rf66+8NvpAoNDVWPHj20d+9ede7cudQ3koUeeeQRTZw4UfHx8apbt648z1NcXFyRTFJSklq3bq2bb765yO3VqlWTvjvzUZY1SNItt9yiW265RU8++aRGjBihhQsX6q233lJISEip85QmOjpakrR169ZiF4lHRkbqwoUL+vvf/x74i7q+u6jc79ou535ezMW2gV/r168v8tGu7Oxs7dmzR48++qjvsUJDQ9WpU6fAhfffL9I+/fRTSQp8ycDFnhPWMQpZtkdqaqo6d+5c7KJ3AACAys7XNRvPPvus0tLSNGPGjMBtn3zyiXbu3FksO336dGVkZOiJJ54o8pfkvLw8zZw5s8hn8Rs0aKCBAwfqnXfeUXx8fOCz/d+XlJRU4lmNwjfru3btuqw15OXladKkScrMzAz8/NSpU9q/f79atGgReFN5sXlKEhUVpRYtWpT41cADBgxQtWrV9Lvf/S5w27Jly4qNbV2bn21dEus8fi1btkx79uyRvvtq3l/+8peSpLFjx17WeIXfUDVx4sTAGY+UlBS98sor6t+/vzp06CBd4rGyjOFne5w4cULbtm3TPffcc1n3CQAA4Frmq9gYNmyYZs+erRdeeEGNGjVS69at9Z//+Z+aMGFCsWyPHj2UnJysLVu2qGHDhmrXrp1atmypH/3oRzp69GjgY0aFHnvsMWVlZemrr74qdmF4Wlqa9u3bV2Kxcdddd+nuu+/WkCFD1LFjxyI9FSxruO6663TdddepS5cuatasmW699VY1atRINWrU0EcffWSapzSjRo1SUlJSses2IiIi9Oqrr+qdd95RRESEWrdurcWLFxfrNG5dm99t/UPWefz67W9/q5iYGLVt21bh4eFat26dEhMTFRUVdVnjDRkyRPHx8UpISFB4eLiioqLUs2dP3XffffrjH/8YyF3ssbKM4Wd7/Nd//ZcKCgqKPWcBAAAghXiX0bUtPz9fe/fuVYMGDdS0aVPl5eVp165duummm3T99dcXy2dlZenw4cNq0KCBIiIiSvxL+YULF7Rp0yZJUqdOnYp8JOWll17SrFmzlJWVVeSjL9938OBBZWZmqqCgoMR1WNZw6NAhHTt2TE2bNg1coOx3nu8rvF5j9uzZJb4ZPXHihPbt26fGjRurUaNGys3N1e7du9WuXTvVqVPH99oudT+/+eYbHThw4KIft7LOczEvv/yynn32WWVnZys8PFzp6ek6c+aM2rZtW+zxK21Np06d0o4dOxQVFRX4tqdCFy5c0N69e3X69Gm1atVKdevWLXEdF3usrGNcanv07NlTERER+uCDD3xvJwAAgGvdZRUbV1q3bt3Upk0bJSQklPdSfJsxY4bmzZunvXv3VprP9P+w2LhWrVixQoMGDdKuXbtK/IYrAACAyq5sVxNfIXPnzi321ahXi6efflq9e/fW2bNnK02xUVlERUUpNTWVQgMAAKAUV0Wx8cNvc7qaVK9ePfDNVLi2UGQAAABc3FXxMSpcXSzXhgAAAODaR7EBAAAAwAlfX30LAAAAAFYUGwAAAACcoNgAAAAA4IT56t20tDRTrmHDhubJrU3jxowZY8q98cYb5rmtl6qcPHnSlHvllVdMueeff96UCwbrmiQpOTnZlGvevLkp99xzz5ly6enpppy+6wxuYd3Gv/nNb8xzl+a9994z5Q4fPmzK3Xnnnea5J06caMpZvw2tTZs25rkffvhhU27t2rWm3O23326e+0p5/PHHzdn58+ebci+99JIpd+jQIfPcr7/+ujl7tXr11VfN2Y8//tiUW7FihSl38803m+fu3LmzKdevXz9T7qGHHjLPXVbW51HTpk3NY95///1lWFFx06dPN2eHDh1qyv34xz8uw4rKV2xsrDm7cuVKUy4/P9+US0xMNOX69Oljyl1p1mO39Vg3cOBAU27p0qWmnCTFxcWZcvHx8eYxrf72t7+Zcp06dTLlOLMBAAAAwAmKDQAAAABOUGwAAAAAcIJiAwAAAIATFBsAAAAAnKDYAAAAAOAExQYAAAAAJyg2AAAAADhBsQEAAADAiRDP2Ep7z549pgGPHj1qnrxWrVqm3LvvvmvK+emYbe3k+8wzz5hy1apVM89dEYWEhJhy99xzjynXrFkzUy40NNSUk6QnnnjClGvfvr15zLI6fvy4KWftlD569Gjz3DVq1DDlvvnmG/OYVhkZGaaccfdifv4FQ2pqqin3+9//3jymdXv8/e9/N+W6d+9unnvu3LmmXJ06dcxjlpX1cU9JSTHlkpOTzXNPnjzZlPvwww9NuZdfftk8d+vWrU05a/diP/vH0pw8edKUq127tik3ZcoU89x//OMfTTlrl/YDBw6Y516+fLkp17BhQ/OYZbV582ZTbuHChaZcVFSUee6qVauacjt27DDlxo4da8qdPn3alJOP5+ptt91mHrM0X331lSm3du1aUy4pKcmUy83NNeUkafjw4abcsGHDTDk/j0VERIQ5a8GZDQAAAABOUGwAAAAAcIJiAwAAAIATFBsAAAAAnKDYAAAAAOAExQYAAAAAJyg2AAAAADhBsQEAAADACYoNAAAAAE6EWYNt2rQx5Xbt2mWefP369abcokWLzGNaWTuzbtmyxZSzdvy0dvsNBmu3UkkaMmSIKff666+bctaOsNYO3PLRcd5PJ/myqlevniln7Xg6Y8YM89wjR4405TIzM0251atXm+e2vs7btm1rHvNK6dixoyn3yCOPmMfs2bOnKWd9/X/22Wfmua2PRTC67lpZO8LXr1/flLO+9iXp+eefN+XeeustUy49Pd08d//+/U25YHQGt7J2BreqW7euOWvddlWqBP/vntbO4HFxcabc22+/XcYVST/96U9Nueeee86UmzVrlnnu1NRUU27lypWm3MaNG025lJQUU+5Ks7736NatmykXGxtryr377rumnHx0L7/ppptMuV69epnnXr58uSln3edxZgMAAACAExQbAAAAAJyg2AAAAADgBMUGAAAAACcoNgAAAAA4QbEBAAAAwAmKDQAAAABOUGwAAAAAcIJiAwAAAIAT5g7iH330kSnXpUsX8+TWDuIHDx405QYMGGCe29pV9Y033jDlrB2zg+HQoUOmXOfOnc1jrlq1ypSzdgMNC7M9tYYOHWrKyUeXzKysLFPO2mH2Yvbv32/KnThxwpQ7fPiwee4nnnjClNu9e7cp98UXX5jn7tOnjyn3m9/8xpSzdn0OBmtX47Nnz5rH/NOf/mTK3XXXXaZc8+bNzXPn5+ebctbnasuWLc1zl5X1scjMzDSPae2AvHXrVvOYVtZO8hXRQw89ZMq98sor5jGtx+45c+aYck899ZR57ieffNKUC0ZncKvq1aubclu2bDHl0tLSyrii4u68805T7uGHHw763MeOHTPlbrzxxjLP1bhxY1OuSZMmppz1fdmHH35oyklSzZo1Tbnz58+bx7SyHuOtOLMBAAAAwAmKDQAAAABOUGwAAAAAcIJiAwAAAIATFBsAAAAAnKDYAAAAAOAExQYAAAAAJyg2AAAAADhBsQEAAADACYoNAAAAAE6EWYN9+/Y15WbNmmWe/NZbbzXlrG3g09PTzXMfP37clBs+fLgpd99995nnLquIiIigj7ljxw5T7he/+IUpd+7cOVNu2LBhppwkbd682ZQ7evSoKdewYUPz3KVp2bJlmcf4vjlz5pizNWrUMOXmz59vyp09e9Y89x/+8AdT7sKFC+Yxy2rnzp2mXKNGjUy5H//4x+a5Y2NjTbmCggJT7vPPPzfP/eKLL5pyNWvWNI95paxZs8aUGzFihHnMv/71r6bcvffea8rVq1fPPLf1OHn+/HlTLizMfIgus8aNG5ty1tePJNWtW9eUu/322025m266yTx3z549zdmK5sCBA6bckiVLzGNat8fLL79syk2fPt2UO3z4sCknSZMmTTJny6pJkyam3NKlS025G264wZQ7cuSIKSdJmzZtMuX27dtnyh07dsw894033mjOWnBmAwAAAIATFBsAAAAAnKDYAAAAAOAExQYAAAAAJyg2AAAAADhBsQEAAADACYoNAAAAAE5QbAAAAABwgmIDAAAAgBMhnud5luCvfvUr04DWrqyStGrVKlNu3bp1ppy1g6Mkffvtt6ZccnKyKVe1alVTrlq1aqZcMLz++uvm7AMPPGDKPfzww6Zcu3btTLlXX33VlJPP7pcWwe6QeTELFy405axd0iUpMjLSlFuwYIEp56d7689+9jNTrlatWuYxrxRr192BAweax0xMTDTlrB2z/XSSDwkJMeUOHTpkykVERJjnLo216661w3xubq557k8++cSUsx5//HT8tY7ZqlUrU65ly5bmucvKekw8deqUecyGDRuacgcPHjTlxo0bZ57b+lqbMmVK0Oe+UqZOneoka/HMM8+Yct27dzePOXTo0DKsyI0nn3zSlPv4449NuVGjRpnnvu2220y5u+++25RbuXKlee49e/aYcmPGjDHlOLMBAAAAwAmKDQAAAABOUGwAAAAAcIJiAwAAAIATFBsAAAAAnKDYAAAAAOAExQYAAAAAJyg2AAAAADhBsQEAAADAiTBr0Nrp2U930SZNmphy//M//2PKbdu2zTz33LlzTbn/+I//MOVq165tyv3iF78w5YJh9OjR5uyZM2dMudDQUFPOT9ddqyvZ8TvYevbsacpt2rTJPKZ1G9erV8+U8/PatXYG37hxoynXrVs389yl+dvf/mbKWTuDz58/3zz3smXLTDlrt2/r/kmS/vGPf5hyw4cPN49ZVj/5yU9MuZo1a5pyf/nLX8xzL1682JQ7fvy4eUwr6/O4evXqQZ+7rKwdg8PDw81jFhQUmHLW48+uXbvMc1u7znfs2NE8ZkUTFRVlzs6cOdOU27lzpymXkZFhyrVr186Uk6R///d/N+X+9V//1TxmWcXHx5ty1u7nq1evNs9t7SD+wAMPmHJTpkwxz92vXz9z1oIzGwAAAACcoNgAAAAA4ATFBgAAAAAnKDYAAAAAOEGxAQAAAMAJig0AAAAATlBsAAAAAHCCYgMAAACAExQbAAAAAJyg2AAAAADgRJg1mJOTY8qdP3/ePHmHDh1MuX/5l38x5bKzs81zjxgxwpy1WLt2bVDHuxhry/lp06aZx+zbt68p16hRI1OuRo0aptyiRYtMOUkaNmyYOVvRfPLJJ6ZcZmameczt27ebcq+//rop17x5c/PcVidOnAj6mKXp1KmTKbdr1y5Trm3btua5ly1bZspZ93me55nnHjVqlClXv35985hl1bRpU1Nu7ty5plybNm3Mc1v3PatXrzblkpKSzHPPmzfPnK1orNvNz37C+hqyPl/27NljnnvQoEGmXGpqqinXq1cv89ylWbFihSl39913m3JHjx41zx0ZGWnK7dy505SrXr26KTd48GBTTld4H2W1e/duU27kyJGm3Keffmqeu0uXLqZcTEyMKXfs2DHz3MHGmQ0AAAAATlBsAAAAAHCCYgMAAACAExQbAAAAAJyg2AAAAADgBMUGAAAAACcoNgAAAAA4QbEBAAAAwAmKDQAAAABOhHh+WtUCAAAAgBFnNgAAAAA4QbEBAAAAwAmKDQAAAABOUGwAAAAAcIJiAwAAAIATFBsAAAAAnKDYAAAAAOAExQYAAAAAJyg2AAAAADjxv/4ESV4ECxh7AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x280 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "from scipy.ndimage import gaussian_filter\n",
    "rng = np.random.default_rng(0)\n",
    "d = load_digits(); Xall, yall = d.images / 16.0, d.target\n",
    "def phone_photo(imgs):                            # what users' images look like\n",
    "    return np.clip(gaussian_filter(imgs, sigma=(0, 0.6, 0.6)) + rng.normal(0, 0.25, imgs.shape), 0, 1)\n",
    "idx = rng.permutation(len(Xall))\n",
    "clean, users = idx[:1200], idx[1200:]\n",
    "X_train, y_train = Xall[clean[:1000]], yall[clean[:1000]]           # clean\n",
    "X_tdev, y_tdev = Xall[clean[1000:]], yall[clean[1000:]]             # train-dev: same distribution as train\n",
    "X_users, y_users = phone_photo(Xall[users]), yall[users]\n",
    "X_dev, y_dev, X_test, y_test = X_users[:200], y_users[:200], X_users[200:400], y_users[200:400]\n",
    "X_photos, y_photos = X_users[400:500], y_users[400:500]          # 100 labelled users' photos, for section 6\n",
    "fig, ax = plt.subplots(2, 8, figsize=(10, 2.8))\n",
    "for i in range(8):\n",
    "    ax[0, i].imshow(X_train[i], cmap=\"gray_r\"); ax[1, i].imshow(X_dev[i], cmap=\"gray_r\")\n",
    "    ax[0, i].axis(\"off\"); ax[1, i].axis(\"off\")\n",
    "ax[0, 0].set_title(\"train (clean)\", loc=\"left\"); ax[1, 0].set_title(\"dev/test (users' photos)\", loc=\"left\"); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7bcc3c",
   "metadata": {},
   "source": [
    "## 2. Train a model and measure four errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "13036b3c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-13T17:19:06.459668Z",
     "iopub.status.busy": "2026-11-13T17:19:06.459316Z",
     "iopub.status.idle": "2026-11-13T17:19:08.673726Z",
     "shell.execute_reply": "2026-11-13T17:19:08.673011Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "human level    0.2%\n",
      "train          0.4%\n",
      "train-dev      4.5%\n",
      "dev           35.0%\n",
      "\n",
      "avoidable bias    0.2%\n",
      "variance          4.1%\n",
      "data mismatch    30.5%   <- the biggest gap is the problem to work on\n"
     ]
    }
   ],
   "source": [
    "T = lambda a, f=torch.float32: torch.tensor(a, dtype=f)\n",
    "def train_model(X, y, epochs=300, seed=0):\n",
    "    torch.manual_seed(seed)\n",
    "    m = nn.Sequential(nn.Flatten(), nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 10))\n",
    "    opt = torch.optim.AdamW(m.parameters(), lr=1e-3, weight_decay=1e-2)\n",
    "    Xt, yt = T(X), T(y, torch.long)\n",
    "    for _ in range(epochs):\n",
    "        opt.zero_grad(); nn.functional.cross_entropy(m(Xt), yt).backward(); opt.step()\n",
    "    return m\n",
    "def error(m, X, y):\n",
    "    with torch.no_grad(): return 1 - (m(T(X)).argmax(1) == T(y, torch.long)).float().mean().item()\n",
    "model = train_model(X_train, y_train)\n",
    "human = 0.002  # people misread about 0.2% of clean scans (our estimate of Bayes error for the training data)\n",
    "errs = {\"human level\": human, \"train\": error(model, X_train, y_train), \"train-dev\": error(model, X_tdev, y_tdev),\n",
    "        \"dev\": error(model, X_dev, y_dev)}\n",
    "for k, v in errs.items(): print(f\"{k:<12} {v:6.1%}\")\n",
    "print(f\"\\navoidable bias  {errs['train'] - human:6.1%}\\nvariance        {errs['train-dev'] - errs['train']:6.1%}\"\n",
    "      f\"\\ndata mismatch   {errs['dev'] - errs['train-dev']:6.1%}   <- the biggest gap is the problem to work on\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ab258f5",
   "metadata": {},
   "source": [
    "## 3. Error analysis\n",
    "Look at the dev examples the model gets wrong. Put each into categories (like the spreadsheet in the slides) and count. Here we let two simple measures do the tagging; in a real project you look at ~100 mistakes yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6ed54b31",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-13T17:19:08.676479Z",
     "iopub.status.busy": "2026-11-13T17:19:08.676112Z",
     "iopub.status.idle": "2026-11-13T17:19:09.171348Z",
     "shell.execute_reply": "2026-11-13T17:19:09.169486Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(if noise didn't matter, about 25% of mistakes would land in each column)\n",
      "70 mistakes out of 200\n",
      "very noisy    26% of mistakes\n",
      "faint         16% of mistakes\n",
      "dtype: str\n",
      "\n",
      "most confused pairs:\n",
      " true  predicted\n",
      "8     1            7\n",
      "6     4            7\n",
      "1     4            5\n",
      "5     1            5\n",
      "0     4            4\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2gAAABxCAYAAACgJ4DzAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAJtNJREFUeJzt3XlUVeX+BvAHkEG0UAGxUHFKRTElUzQcUrlO1yFwWqam5pxZcc1MnHK4ampmJWqaOZVmmiYOOOQQ6EWynAMtZRAUzQlBRFB4f3/8giVy8PsCh9zC81nLtWrvx/fde7vP3uc9e/haKKUUiIiIiIiI6ImzfNILQERERERERP+PAzQiIiIiIiKD4ACNiIiIiIjIIDhAIyIiIiIiMggO0IiIiIiIiAyCAzQiIiIiIiKD4ACNiIiIiIjIIDhAIyIiIiIiMggO0IiIiIiIiAyi1JNeAAC4ffs2pkyZgoyMDLz//vuoVq1aodvcvXs3tm3bluf8rl27okOHDoXux4gyMjIQFBSE06dPIyUlBVWqVIGvry9cXV3N2s+JEyewZcsW3LhxA0OHDkWjRo3M2r4RXbt2Dd9//z2ioqLg4OCATp06oUmTJmbt48KFC9izZw+io6Ph6OiIJk2aoG3btmbtw2j8/f1x//79XNObNWuG/v37m62fGzduYMuWLTh58iTc3d3x1ltvma1tIzp37hy++OILk/PM/Zk9e/YsfvjhByQkJKB3795o1aqV2do2quTkZPzwww84f/48AKBWrVro2bMnypYta7Y+4uPjERwcjPPnz6NcuXJo1KgROnbsCAsLC7P1YUQXLlzA9u3bcenSJTg4OOCll14y+3onJSUhKCgI4eHhcHV1xYcffmi2to3uwIEDCAkJQVpaGry8vNCtW7ci2aeioqKwYMECAMDHH3+MMmXKmL0Po4iIiMCBAwcQGxuL559/Hq1bt4anp6dZ+1BKISwsDNu2bUNycjI++OADVK1a1ax9GM28efMQGxuba7qbmxvGjRtX6PYf3kdNqVu3Lt5+++1C95NfhriC9vbbb+OLL75AYGAgrly5YpY2HR0dUbdu3Vx/wsPDERgYiDt37pilH6OJi4tD/fr1MXLkSNy9excVKlTAli1bUKNGDSxdutQsfRw7dgx16tTBoEGDcOLECQQGBmZ/QSnO9u3bh1q1amH16tWoUKEC4uPj8corr2D8+PFm62PYsGHo0aMHIiMj4ezsjKioKHTt2hWvvPIKEhMTzdaP0SxZsgSnT5/O9Xl97rnnzNZHz549Ub9+fRw5cgTLli1DUFCQ2do2qri4OAQGBkIplWvbPvPMM2bpIyoqCg0bNoSvry9OnTqFwMBAnDp1yixtG9nPP/8MNzc3zJ8/HzY2NrC1tcXcuXPh5uaGQ4cOmaWP8ePHo0OHDjh+/DicnJxw+fJl9O3bF40aNUJCQoJZ+jCiGTNmoE6dOti3bx+cnJxw48YN9O3bF02aNMGtW7fM0seoUaNQq1Yt7Nq1C+vXr8d3331nlnaNLjExEe3atcOgQYOQmpoKR0dHrFu3Dm3atDF7X5mZmRg4cCAWL16MwMBApKammr0Po2jVqhUaNmyIkJAQODk54dSpU/Dy8sLgwYPx4MEDs/Sxa9cuVKtWDePGjcMvv/yCwMBA/PXXX2Zp28g2bNiA3bt35zqHubm5maX90qVLmxwv3Lx5E4GBgTh37pxZ+sk39YRt2rRJ2djYqMGDBysAKiwsLF9/PyUlRe3atUsrm5aWppycnJSLi4tKT08v4BIb27BhwxQAdfr06RzT27Rpo2xtbVVSUpJ2WydPnlTnz5/PNf3ixYvq3LlzSiml1q9frwCojRs3mmHpjSs5OVk5OTkpLy+vHPvO119/rQCooKCgfLW3bds2lZaWlmv6qVOnck3btWuXAqACAgIKuPTGZ2trq0aPHl3odtLT09W2bdtMzjt48KC6f/++UkqpMmXKqA4dOhS6P6Pbu3evAqCCg4ML3VZUVJQ6fvx4rul//fWXOnnypFJKqQMHDigA6osvvih0f0bXoEED5eLiopKTk7OnJSUlqYoVKypPT898tbV79251586dXNNNHQ9+/fVXBUANHz68gEtubDExMcrCwkL16dMnx/SffvpJAVCTJ0/WbiszM1Nt2bLF5LzQ0FB17949pZRSNWvWVA0bNizkkhtfZmam8vHxUY0aNVK3b9/OMe/s2bP5auvKlSvq8OHDj83MmTNHOTk5qR49eigA6tq1awVa7qdBzZo11S+//JJj2rp16xQAtWjRony1FRISYnJbnT17VsXFxSmllJo9e7YCoI4ePVrIJTe+xo0bq9atW5ulrc2bN2tnu3btqgBkn9/+aU/0CtqVK1cwcuRITJw4EfXq1StQG59//jm6dOmCrVu3itmtW7fi+vXrGDRoEKytrQvUn9FdvnwZdnZ28PDwyDG9adOmSEtLw40bN7TbGj58OHx8fHDp0qUc06tUqYLatWubbZmfBgcPHsT169cxdOjQHPvOwIEDUbZs2TxvIzMlPj4ePXv2RL9+/ZCRkZFjXoMGDXLls24Vi4mJKdQ6lASrV69Gt27dsGrVqlzzWrdujVKlDHFX91PJ398f7du3x9mzZ3NMd3Z2xosvvvjElutJuXz5Mtzd3XPczvjMM8/A3d0dly9f1m4nMTERvXv3Rvfu3ZGWlpZjnqnjQePGjVGmTJliezxISEiAUirXreNeXl7A39td144dO+Dr64uPP/4417wWLVrA1tbWDEv89Ni9ezd++uknzJ07F88++2yOeXXq1MlXWxMnTkTHjh1x9OhRk/NPnTqFKVOm4LPPPoOTk1OhlvtpEBoammuf7dmzJ6ysrLBnzx7tdtLT09GvXz907NgRSUlJOebVqVMHlStXNtsylzRhYWHw8/PTui0yISEBwcHB8PLyemLntyc6QBs6dCgqVaqECRMmFLiNcePG4bXXXkOfPn2wb9++x2ZXrFgBCwsLDBs2rMD9GV3nzp1x7969HNsiLS0Nu3fvRv369fN1r/L3338PpRR8fHxw7dq1Ilrip8PVq1cBAJUqVcox3dLSEi4uLjh8+DAyMzO12qpcuTK+++47/Pjjjxg2bBiUUo/NZz1LWVyfmcxy8uRJjB8/HuPGjcPy5cvz9WNCliFDhmDYsGEYOnQofvjhhyJZzqfRxo0b8Z///AeTJk3C1q1bC3TLzYoVK1CpUiX4+PgU28FBfnTu3BnHjx/PMWCIj4/H8ePH0blzZ+12ypUrhx9//BH/+9//0KdPH/HfZv/+/UhJSSm2x4MXX3wRrq6uCA4OznFMzbolOT/btkuXLpg0aRI+/PBDLFmypEiW92myefNm2NnZ4eWXX8ZXX32F999/H9OmTUNYWFi+21q4cCE8PDzQqVMn/P777znmpaeno3///vDx8cHrr79uxjUwLlO34yckJCAjIwP29vba7djY2CAoKAhRUVHo0qVLsb4tND8uXbqEiRMnYuzYsfj8888LdA5q3rw5Fi5ciPnz52PmzJmPza5evRoPHjzA8OHDC7HUhfPEfk5etmwZgoODERYW9tirWQEBAbl+RXiUo6MjlFLo3r079u3bl/1L28Pi4+Oxd+9etG3bFjVr1jTLOhjRqFGjkJGRgSFDhqBRo0YoX748QkJC0LhxY3z66aewtPz/MXlMTAzmz58vtufh4YEdO3agQ4cO+Pnnn8323MrTJusFK9HR0Tmmp6Wl4dKlS7h37x6uXbsGFxcXbN++Hbt27RLbrFOnDlauXIlnn30WCxcuzJ6enp6O//znP8jIyMCFCxfwxx9/4Msvv8Qbb7xRBGtmDLa2tnBxcYGDgwNu3bqFWbNmYezYsfj222/RtWvX7NysWbPEX9BLlSoFOzs7vP7669i2bRvat2//D6yBcVWsWBGlSpWCi4sLzp8/j759+6JGjRrYuXNn9g82N2/exJQpU8S26tati40bN8LHxweHDx+Gi4vLP7AGxrR06VI4OzujadOmaN26NfD3ixeGDh2a4+R/8OBBbNq0SWyvXr162Lp1KwYPHoy1a9fmmOfv74+0tDTExsbi5MmT+OSTT+Dv718Ea/Xk2dvb48CBA/D394eHhweaN2+OK1euIDIyEitXroSfn192dtGiRbmu6Jri6OiI0aNHw8HBocQMGEyJiIiAnZ0dGjZsiNq1a6Ndu3Y4ffo0PvroI7z11lsIDAwEHjoHSWrVqoWwsDD861//wuHDh1G9enUAwKRJkxATE4MdO3YU+ToZ2dSpUwEAvXr1yp527NgxfP311+LfbdCgAUJCQuDn54dt27aV6Ls/LC0tUb16dVhbW8PCwgKrVq3C2LFjMW/ePLz33nvZuZUrV+K3334T26tcuTImT56McuXK5fnyj6zvZn369DHruuTLk7iv8sKFC6ps2bLK398/e9q8efNMPoPm4uKiAGj/ef/99032OX36dAVAbdiwocjX70m6fv266tatm6pYsaIaN26cmjVrlmrXrp1ydXXN8ZxUWFhYvrYrAHXo0KFc/ZWUZ9BSU1OVq6urqlOnjrp161b29Kz9CoCKiYlRSik1ceLEfG3XcuXK5egrPT1dffHFF+qTTz5Rw4YNU2XLllV+fn45+i1uHn3WMSUlRTVp0kTZ29urS5cuZU+vX79+vrbtoEGDTPZXUp5Bu3r1ao5npJRS6syZM8re3l55e3tnT4uOjs738WDTpk25+itJz6CdOXNGeXp6Knd3dzVlyhQ1depUVa9ePeXp6al+//337Nynn36a72374MGDHH0tWbJELViwQI0ePVo5ODioDh06qCtXrjyBtf5nrFmzRlWsWFF17txZzZkzR/n7+6sKFSqo119/Pcezeu3atcvXdu3YsaPJ/krKM2gNGzZUAFSPHj1yTJ8yZYoCkP18TnJycr732aVLlyr197N9lpaWKjAwMLv9ESNGFPtn0B6V9Xx6ly5dckzP+s6Unz/x8fG52i9Jz6A9+v0gIyND9e3bVwFQISEh2dP79OmTr+368ssvm+wvJCREAVCjRo0q8nV7nH98SK6UwhtvvIGKFSuKlxgBYPbs2UhJSXls5vbt25g+fTqcnJwwatQok32uWrUKzs7OeO211wq1/EY3fPhw7Nu3D6dPn87+NWvChAkYOHAgevXqhYiICNSoUQPVq1fXem4qPDwc33zzDbp06WLyymRJYWdnh82bN8PPzw8eHh5o27YtLl68iOjoaLRs2RKhoaEoV64c8HcJh0dvhTRlxYoVOHHiBGbMmJFjurW1dY5fdd566y14eXnh3XffxerVq4tg7Z68R69q29vbY9KkSejevTt+/PHH7NfhT5w4Ubz1MTU1FdOmTYONjY3Wr8DFWcWKFXNNq1+/Pvr164fly5fj4sWLqFq1KhwdHbWOB5GRkVi8eDFeeeUVdOrUqYiW2vjS09PRpUsX2Nvb4/jx49nPMgUEBKBx48bo1q0bIiMjYW1tjTZt2mht2++++w6HDx/GRx99BCsrqxzzRo4cmf3f/v7+8PT0xJtvvlksr1AcO3YMAwcOxJgxY/DZZ59lTx8yZAg8PT3x7LPPZt+uOGbMGPGcnpGRgZkzZyIlJQUBAQFFvvxGlvWK+4f3J/z9Ju3p06fjxx9/hK+vL2xtbbX22djYWMyfPx8NGjRA7969kZKSgjfeeAPe3t4mv4uVFFu3bsXw4cPRtGlTrFu3Lse8xo0ba23bnTt3Ijg4GCNHjjR7iaSnzaPfDywtLTFz5kysX78e69evR8uWLQEAb775Jlq0aPHYtpRSmDt3Lq5cuZJ9hfNRWVc4n/jjUP/0iDDrlxlvb281evTo7D/e3t4KgOrZs6d65513tNtLSkpSzZo1U46Ojjl+tXzYvn37Hnt1rbjIzMxUdnZ2qn379rnm7d69WwFQS5Ys0W7v6NGjysHBQbVp00alpqaazJSUK2hZUlNT1Y4dO9TSpUvV5s2bVVJSkurcubNyc3PLVzsLFixQANR///tfrXzz5s2Vs7NzAZf66XTixAmFfL617d69e6p9+/aqbNmy6siRI3nmSsoVtLzMnDlTAVDh4eHaf+fcuXOqUqVKqlGjRioxMdFkpqRcQTt+/LgCoKZMmZJr3rRp0xQAk2+8zMuqVauUhYWFeu+997TyXbt2VdbW1rmutBUHM2bMUAByvRFPKaWaNGmiXF1dtdvK+qXd2tpa7dy5M89cSbmCNmDAAAUTb3nOzMxUpUqVUp06ddJuKy4uTtWoUUPVqlVLJSQkKPXQ56Jz5845vt9l3fkwZMgQNWPGDLOvl5EEBwcrGxsb1bhx4wLf9bJ9+3ZlY2Oj+vXrpzIyMkxmStIVNFPu37+vLCwslK+vb77+3ujRo5WlpaVat26dyflJSUmqTJkyeV5d+yf941fQ8vplJutNgW5ubqhRo4Z2e4sXL0ZERAT27duX55sgDTMaLmJZRSYffRPYw9OynkHTMWbMGNSpUwdBQUGws7Mz45I+vezs7HI8pJ6YmIiff/4ZY8aM0W4jPj4eAQEB+OCDD7R/0b1z506+HjQuDrIePM9PrZNvvvkGISEh2LFjR4m+4ivJ2rb5eWnQ+PHj4eDggD179sDBwaEIl+7pYY5jbWJiIvz9/TF48ODHFkt92J07d2BjY5PrSltxkte2zc85LDg4GN9//z3Wr19foq/4ZmnXrh3Wrl2L6OjoHG96jouLw4MHD/J1rJ02bRru37+P0NDQ7DtGKleubPL73cmTJwEAL7zwQrF+C+H+/fvh6+sLDw8P7N27N/uumvxIT0/H6NGj0bFjR6xatSpf+3tJEhERAaVUvvbZI0eOYMmSJViyZAn69u1rMrNhwwakpKQ80ZeDZHvSI8QseT2DJsnIyFARERF5zk9MTFSlS5dWr776qhmW0vh69OihSpUqlePqQVpamvLx8VHW1tYqOjpau63Lly+rGzduPDZTkq6ghYSEqLt372b/f1pamurdu7eqWrVqvn8pO3PmTK5ply9fznE/dZa1a9cqFOM6aGFhYSo2NjbHtPj4eFWzZk3l5OSkbt68ma/2TG3bR5WUK2hBQUEqJSUlx7Rdu3YpKysr1b1793y1devWLZPPQjyspFxBe/DggXJzc1Ourq7q6tWr2dOvXr2qXF1dVbVq1fJ1dSsiIiLXL+WJiYkma3xu375dWVlZFds6aMeOHVMWFhaqZ8+eObbhrl27lIWFRb6fC9E5HpSUK2h3795Vbm5uqlWrVtl3xWRmZqqhQ4cqS0vLfF1Rv3v3roqKitLKloRn0A4dOqTKlCmjXnrppXyfsx71559/Ztfoy0tJuYIWGRmpTpw4kWPa7du3VZs2bZS1tbXJWpGPIx0PmjVrpsqWLZvr2e0n4al/LYylpSXc3d3znL9u3TqkpqYaYzT8DwgMDERycjJatmyJTp06oXz58ggNDcXdu3fx7bffolq1atptmXptLACkpKRg/PjxAIA///wT+Pt5qoMHDwIA5s2bh9KlS5tlfYwkNjY2+zkIOzs77N+/H88//zwOHDiQ71/K6tevn2ualZUVZs6ciYSEBNSrVw92dnY4ffo0Tp06hbfffhvTpk0z49oYR2pqKjp16oQKFSqgVq1auHnzJvbv3w9XV1fs3r0b5cuXz1d7prYtACxfvjz7l9z09HRERkZmP+vn6+uLdu3amWFtjCUiIgLvvvsuateuDVdXV5w/fx6hoaHo1q1bvp9nLFeuXJ77+bvvvouMjIzsOyE2btyY/Xa9yZMnF7u3PVpZWWHTpk0YNGgQ3N3d0b59eyilsHfvXjz//PNYs2ZNvq5umTqHWVlZYfHixfD390f9+vXxzDPP4OzZswgPD8eAAQNyvPm1OPH09MTSpUvx4YcfwsPDA82aNcPVq1exZ88e+Pn5maxp9jh5HQ82bNiA0NBQAMD169dx+/bt7OOBj49PsXxevXTp0ggKCkL37t3RoEEDeHt748yZM4iMjMSKFSvQtGnTfLWV9Zx7SZeamop///vfSElJQd26dTF58uQc893c3LRqb2WpVauWyekXL17E3Llzgb+f1cTf37ecnZ1hb2+fPa84sbCwwIgRI5CRkYG6desiLS0NISEhUEph06ZNJmtFPk5exwP8/Yz1kSNHMHz48Bz1LZ8UCyUVYfqHnDhxAocOHUKvXr3MejLfuXMnoqOjMXTo0BJVlDLroHvnzh24ubmhWbNmZrtF7t69e/jqq6/ynD98+HDY2NiYpS+juX79On7++WckJyfDw8MDjRs3zr611FyioqJw7Ngx3Lx5E5UqVULz5s3h7Oxs1j6MJjMzE0ePHsXZs2ehlIK7uzuaNm1q1m27c+dOREVFmZzXokULNGrUyGx9GUlycjKOHDmCmJgYlC1bFl5eXvm6jVzH4sWL86wD2L9//wLd6vM0yMzMRHh4OC5cuAD8/TC7l5eXWW9LiouLw2+//YarV6/C2dkZzZs3z/PHs+IkKSkJ4eHhiIuLQ7ly5eDh4YHatWubrf2DBw/izJkzJud5enrC29vbbH0ZTXp6Og4cOIDY2FhUqFABbdq0gaOjY5H1l7Wthw4dWiwfl0hPT8eyZcvynF+xYkX07t270P1cu3YNGzZsMDnP1ta2WD/Gc+bMGZw+fRopKSmoWbMmvL29zf498+TJkwgNDUXnzp3Nfo4sCMMM0IiIiIiIiEo6Pn1IRERERERkEBygERERERERGQQHaERERERERAbBARoREREREZFBcIBGRERERERkEBygERERERERGQQHaERERERERAZRypyNLViwQMz89NNPYsbNzU3MTJgwQcz8+eefYqZdu3ZiZsqUKWJm+vTpYiYv33zzjZi5dOmSmGnbtq2YCQgIEDPNmjUTM3Xq1BEz/fv3FzOhoaFipmXLlmKmoBYtWiRmXF1dxYyfn5+ZlgiYNWuWmOnZs6eYeeGFF8y0RLldu3ZNzOgU19YpRK1T9PdxhdOz7N27V8x8++23YsbchckftXnzZjHTunVrMbNw4UIxk1eh3ocdPXpUzOgcn9544w0xs3r1ajFT1Hbu3Clm6tWrJ2Z0inMvXrxYzMydO1fMdOrUScysX79ezBSUTts6x9EPPvhAzISHh4uZChUqiJnt27eLmebNm4uZvAq0P8ychcsLSmd9J0+eLGamTp0qZmJjY8XMu+++K2aKk+PHj4sZnXOUzmfk3r17YqYoC4avXLlSzAwePFjM6Iwpxo4dq71cj9OmTRsx4+LiImZ0xgsA4O7uLmae/FGDiIiIiIiIAA7QiIiIiIiIjIMDNCIiIiIiIoPgAI2IiIiIiMggOEAjIiIiIiIyCA7QiIiIiIiIDIIDNCIiIiIiIoPgAI2IiIiIiMggLJRS6h/tUKPIa4cOHcRMlSpVxIyVlZWYGTVqlJhp2LChmCmMW7duiRmdotsjRowQMzrFCRMSEsSMjpiYGDGjs/sVtDDwnTt3xEzZsmXFjE5hTt0isB4eHmJGp8inTjFdnaKKRWnkyJFiRqfg9q+//ipm3nnnHTGjs81ee+01MVOqVCkxU9Rmz54tZnT2yeeff17M6BTi1fkcTZgwQcw0adJEzBTU9evXtXInTpwQMzrru3z5cjFTsWJFMRMRESFmdIq1tmrVSsw8aTrn2lOnTomZBg0aiJkHDx6Imfnz54uZzp07i5mipvP9YMOGDWImIyNDzKSlpYmZWbNmiRkdn376qZjx9/cvcPtXrlwRMzt27BAzzs7OYmb16tVipmbNmmLGz89PzDRr1kzM6Oz/RXmu0zk/ffzxx2JmwIABYub9998XMxs3bhQzrVu3FjO///67mAGAV199VczwChoREREREZFBcIBGRERERERkEBygERERERERGQQHaERERERERAbBARoREREREZFBcIBGRERERERkEBygERERERERGQQHaERERERERAZh1ip0OgVlfX19xcyiRYvEjE7BX50C0Pb29mJmwYIFYqYwypcvL2aaNm0qZubMmSNmBg0aJGZ0ijfu379fzERGRooZd3d3MVNQOsVkdTg4OIgZnUKhAGBpaZ7fRHSKUA8ePFjMrFy5skD9f/7552ImPDxczOhsW0dHRzGjs/11ioB+9dVXYkanAHdhHDlyRMwEBASImX79+okZnULJ5iom/9xzz4kZnWKhvXr1EjOm6KwHAPj4+IiZbdu2iRmdwtA6223VqlViJjExUcxs2rRJzOgUjjdl8uTJYmbGjBlixs3NTcz0799fzPz1119iRqcItc7xoE2bNmKmdOnSYqYwdPbtF198UcyEhoaKmXbt2omZixcvipmqVauKmcIUodZha2srZnQKQ3ft2lXM1K5dW8zoHPvT09PFTLly5cRMTEyMmOnYsaOYKSid7706+9rVq1fFzNq1a8VMw4YNxYxOQfKkpCQxo4tX0IiIiIiIiAyCAzQiIiIiIiKD4ACNiIiIiIjIIDhAIyIiIiIiMggO0IiIiIiIiAyCAzQiIiIiIiKD4ACNiIiIiIjIIDhAIyIiIiIiMgjtQtXx8fFi5uWXXxYz+/btEzM6BW5LlZIXXafoZr169cSMTiE8ncLBeYmOjhYzycnJYubSpUtiZtSoUWLm7NmzYub3338XMzoFPKdPny5mpkyZImYK6vXXXxczOoXK4+LitPpbuHChmBkzZoyYeeutt8RMQYtQ69ApZv7hhx+KmfXr14uZrVu3ipmDBw+KmZ07d4oZneKZRU2nEOkvv/wiZu7fvy9mdApD6xR87927t5ipXLmymCloEWodOkVGdXl7e4sZnYLKwcHBYmbEiBFiplu3bmJG5zNSUF5eXmZp586dO2Lmgw8+EDPdu3cXMwMGDBAzOsXVi7oItc6x9ssvvxQzixcvFjP37t0TMwkJCWImICBAzOgUVz927JiYadu2rZjJi87xXmcZXnjhBTFjbW2tvVyPc+7cOTFTpUoVMaPzvbagdLbZ0aNHxUxgYKCY0dlnW7duLWZ0jgeZmZliRqcoNjSP2byCRkREREREZBAcoBERERERERkEB2hEREREREQGwQEaERERERGRQXCARkREREREZBAcoBERERERERkEB2hEREREREQGwQEaERERERGRQWgXqtYpMqrj9OnTYmbkyJFiRqcIq07R019//VXM3LhxQ8wUplB19erVC/x3H6ZTBNnOzk7MLF++XMzoFNNdtmyZmNEp/FeUdAqRVqpUScw4ODho9deyZUsxo1M8vUWLFlr9FZVZs2aZpR2dYpmff/65mImPjxczc+fOFTM6BS2rVasmZgqjVatWZmlHpzDxK6+8ImbWrFkjZmbMmKG9XI+jU9x8zpw5ZukrL5cvXxYzGRkZYub8+fNiRmd/0ylC7OPjI2YOHz4sZl599VUxY0rHjh3FjM45Y9KkSWJGZz1sbW3FTFhYmJg5cuSImClqOuefTz75RMzobLf33ntPzOhsW51/69DQUDETExMjZgpTqFrHSy+9JGaWLFkiZnSOozrfmXQ+o999952YMVdxeVN0tpnO8TEtLU3M9O3bV8zUrFlTzFy4cEHMDB48WMz0799fzOjiFTQiIiIiIiKD4ACNiIiIiIjIIDhAIyIiIiIiMggO0IiIiIiIiAyCAzQiIiIiIiKD4ACNiIiIiIjIIDhAIyIiIiIiMggO0IiIiIiIiAzCQimlzNXYokWLxEyPHj3EjE6ht/r164sZnQK3N2/eFDM6KlSoYJZ28rJu3Toxo1N0u2rVqmJm9erVYkanoGinTp3EjL29vZgpSklJSWImJSVFzOgWKo+LixMzOsVCdYoQT5061Sx9PWmzZ88WMzrF1evWrStmdAp82tjYiBmdgvB50SmCbGVlJWaioqLEzLVr18RMs2bNxMyqVavEzMCBA8WMzufx2WefFTNF7eTJk2JGpxCyzr+Rzv6WnJwsZqZPny5mHBwcxIwpOvvsuXPnxMy8efPETHR0tJhxdnYWMx06dBAzOt8zmjdvLmYKY8iQIWLG19dXzOgUfd60aZOYGTt2rJh5+eWXxYzOMusUs7a2thYzeUlNTRUzpUuXLnD7D9M5jwUEBIiZFStWiJmWLVuKmQcPHogZd3d3MVOUqlevLmYuXbokZsaMGSNmdIqNJyYmihmd47UuXkEjIiIiIiIyCA7QiIiIiIiIDIIDNCIiIiIiIoPgAI2IiIiIiMggOEAjIiIiIiIyCA7QiIiIiIiIDIIDNCIiIiIiIoPgAI2IiIiIiMggSpmzsREjRoiZtLQ0MaNThPWvv/7SXq7HKeoC0+bSokULMfPLL7+IGZ3tVr58eTGjU7xZpwi1TjHXoiwEqlM81cnJSczoFGqF5v4fGRkpZnQKJr700ktay1QQmzdvFjN+fn5i5ptvvhEz/fr1EzNBQUFiZtmyZWJGpwjy8ePHxYynp6eYyYvO8U+nULLOuowcOVJ7uR6nWrVqZmnn1KlTYkbnWFjUdIqj6hzbdD4jo0ePFjN//PGHmNE5HheUzj6rU7z94sWLYkbnnK3z+ejWrZuY0Tn3HDp0SMwUZp/VKUyckJAgZn744Qcx07VrVzFz8+ZNMfPcc8+JGZ398caNG2KmUqVKYiYv5ipCHRwcLGZ0CiGPGjVKzJw5c0bMvPnmm2Jm9+7dYqYoC1XHxsaKmfPnz4sZnWLWISEhYmbSpEliZs+ePWLm3r17YgYAevbsKWZ4BY2IiIiIiMggOEAjIiIiIiIyCA7QiIiIiIiIDIIDNCIiIiIiIoPgAI2IiIiIiMggOEAjIiIiIiIyCA7QiIiIiIiIDIIDNCIiIiIiIoPQLlQ9depUMTNt2jQx4+PjI2Z0Cg/qFL3cuHGjmOnVq5eYMYJdu3aJmStXrogZnUKwixYtEjNubm5iRkdycrJZ2ikonf1IZ111Czq6urqKGZ3i2d27dxczx44dEzOtWrUSM6boFNjNzMwUMwMGDBAzOp/jbdu2iRmdguM626woC4ADwIwZM8RMo0aNxIxOEXCd/VZnfXUKl7du3VrM6OwzBaVTUBiaRYV1CjOvWbNGzCilxMzChQvFTFEWodZx69YtMVOjRg0xk5KSImZ0PqMjRowQMxcuXBAzZ8+eFTO2trZipqjpFIZeuXKlmLl69aqY0Tn2REdHixlvb28xo1OUuTCFqnWOkf379xczOse2uLg4MfPtt9+KmaSkJDGjU0x8/PjxYqagdAqn63yv2rJli5jR+S6ic3x65513xEypUvKQ6fDhw2JGF6+gERERERERGQQHaERERERERAbBARoREREREZFBcIBGRERERERkEBygERERERERGQQHaERERERERAbBARoREREREZFBcIBGRERERERkEBZKp1ImERERERERFTleQSMiIiIiIjIIDtCIiIiIiIgMggM0IiIiIiIig+AAjYiIiIiIyCA4QCMiIiIiIjIIDtCIiIiIiIgMggM0IiIiIiIig+AAjYiIiIiIyCA4QCMiIiIiIjKI/wOQoSdMdxJH+QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1100x160 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "with torch.no_grad(): pred = model(T(X_dev)).argmax(1).numpy()\n",
    "wrong = np.where(pred != y_dev)[0]\n",
    "noise_all = np.array([np.abs(x - gaussian_filter(x, 1)).mean() for x in X_dev])   # how noisy each dev image is\n",
    "ink_all = X_dev.mean(axis=(1, 2))                                                # how much ink it has\n",
    "table = pd.DataFrame({\"true\": y_dev[wrong], \"predicted\": pred[wrong],\n",
    "                      \"very noisy\": noise_all[wrong] > np.quantile(noise_all, 0.75),   # noisiest quarter of ALL dev images\n",
    "                      \"faint\": ink_all[wrong] < np.quantile(ink_all, 0.25)})          # faintest quarter\n",
    "print(\"(if noise didn't matter, about 25% of mistakes would land in each column)\")\n",
    "print(len(wrong), \"mistakes out of\", len(y_dev))\n",
    "print((table[[\"very noisy\", \"faint\"]].mean() * 100).round(0).astype(int).astype(str) + \"% of mistakes\")\n",
    "print(\"\\nmost confused pairs:\\n\", table.groupby([\"true\", \"predicted\"]).size().sort_values(ascending=False).head(5))\n",
    "fig, ax = plt.subplots(1, 10, figsize=(11, 1.6))\n",
    "for a, i in zip(ax, wrong[:10]):\n",
    "    a.imshow(X_dev[i], cmap=\"gray_r\"); a.set_title(f\"{y_dev[i]}→{pred[i]}\"); a.axis(\"off\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2875a102",
   "metadata": {},
   "source": [
    "## 4. A single number, and a satisficing metric\n",
    "Pick one number to optimize (here macro F1) and treat others as limits that only need to be good enough (here: prediction time under 1 ms per image)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "976ea96b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-13T17:19:09.173275Z",
     "iopub.status.busy": "2026-11-13T17:19:09.173060Z",
     "iopub.status.idle": "2026-11-13T17:19:09.212431Z",
     "shell.execute_reply": "2026-11-13T17:19:09.211314Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "macro F1 on dev: 0.644\n",
      "time per image: 0.0008 ms -> OK\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "from sklearn.metrics import f1_score, classification_report\n",
    "print(\"macro F1 on dev:\", round(f1_score(y_dev, pred, average=\"macro\"), 3))\n",
    "t0 = time.perf_counter()\n",
    "with torch.no_grad():\n",
    "    for _ in range(20): model(T(X_dev))\n",
    "ms = (time.perf_counter() - t0) / (20 * len(X_dev)) * 1000\n",
    "print(f\"time per image: {ms:.4f} ms -> {'OK' if ms < 1 else 'too slow'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66ee1e09",
   "metadata": {},
   "source": [
    "## 5. Fix the data mismatch with synthetic data\n",
    "Make training images look like the users' photos (noise and blur), add them to the training set, and measure again. Careful: if you synthesize from a tiny set of noise patterns, the model can overfit to them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8f850f8a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-13T17:19:09.214017Z",
     "iopub.status.busy": "2026-11-13T17:19:09.213794Z",
     "iopub.status.idle": "2026-11-13T17:19:10.060318Z",
     "shell.execute_reply": "2026-11-13T17:19:10.059800Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dev error before 35.0% -> after 22.5%\n",
      "test error (look at it only once, at the end): 20.5%\n"
     ]
    }
   ],
   "source": [
    "X_aug = np.concatenate([X_train, phone_photo(X_train)]); y_aug = np.concatenate([y_train, y_train])\n",
    "model2 = train_model(X_aug, y_aug)\n",
    "print(f\"dev error before {errs['dev']:.1%} -> after {error(model2, X_dev, y_dev):.1%}\")\n",
    "print(f\"test error (look at it only once, at the end): {error(model2, X_test, y_test):.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39640648",
   "metadata": {},
   "source": [
    "## 6. Transfer learning: same task, new kind of data\n",
    "You can label only **100 users' photos**. Train on them from scratch, or pre-train on the 1,000 clean scans and then fine-tune the whole network on the 100 photos?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ed294eab",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-13T17:19:10.062973Z",
     "iopub.status.busy": "2026-11-13T17:19:10.062243Z",
     "iopub.status.idle": "2026-11-13T17:19:10.547511Z",
     "shell.execute_reply": "2026-11-13T17:19:10.546012Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dev error, clean scans only:                 35.0%\n",
      "dev error, 100 photos from scratch:          38.0%\n",
      "dev error, pre-trained + fine-tuned on them: 28.0%\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "scratch = train_model(X_photos, y_photos)\n",
    "tuned = copy.deepcopy(model)                                      # model was pre-trained on the clean scans\n",
    "opt = torch.optim.AdamW(tuned.parameters(), lr=1e-3)            # small learning rate: adjust, don't erase\n",
    "for _ in range(150):\n",
    "    opt.zero_grad(); nn.functional.cross_entropy(tuned(T(X_photos)), T(y_photos, torch.long)).backward(); opt.step()\n",
    "print(f\"dev error, clean scans only:                 {error(model, X_dev, y_dev):.1%}\")\n",
    "print(f\"dev error, 100 photos from scratch:          {error(scratch, X_dev, y_dev):.1%}\")\n",
    "print(f\"dev error, pre-trained + fine-tuned on them: {error(tuned, X_dev, y_dev):.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e671c1a4",
   "metadata": {},
   "source": [
    "## Try this\n",
    "1. Make `phone_photo` much noisier (`0.5`). Which of the four gaps grows?\n",
    "2. Train on half the training data. Which gap grows now?\n",
    "3. In section 5, add only 10% synthetic images. How much of the gain do you keep?\n",
    "4. In section 6, use 20 photos instead of 100. Does pre-training still help? And when does it stop helping?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}