{ "cells": [ { "cell_type": "markdown", "id": "18558450", "metadata": {}, "source": "# Lesson 36: Evaluating a Classifier\n\nEvery classifier so far has been evaluated with a single number: overall accuracy. That number hides a lot. This lesson builds a 4-class classifier on a deliberately imbalanced subset of **CIFAR-10** — real photographs, not synthetic shapes — and shows why accuracy alone can be misleading, using the tools that reveal what's actually going wrong: the **confusion matrix**, and **per-class precision and recall**." }, { "cell_type": "code", "execution_count": null, "id": "57fdea49", "metadata": {}, "outputs": [], "source": [ "import pickle\n", "import tarfile\n", "import urllib.request\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "id": "2733014c", "metadata": {}, "source": [ "## A 4-class, imbalanced dataset\n", "\n", "Four real photo categories from **CIFAR-10** (Krizhevsky, 2009): cat, dog, automobile, and truck. The test set has an even 150 images per class, but the *training* set is deliberately starved of trucks (90 examples, versus 400 each for the other three) — a stand-in for the common real-world situation where some classes are just rarer to collect than others. Cat and dog are included deliberately as a visually similar pair, regardless of how much training data either gets." ] }, { "cell_type": "code", "execution_count": null, "id": "94248cb5", "metadata": {}, "outputs": [], "source": "CIFAR_URL = 'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz'\nCACHE_ROOT = Path.home() / '.cache' / 'cvintro'\nCACHE_DIR = CACHE_ROOT / 'cifar-10-batches-py'\n\ndef ensure_cifar10():\n if CACHE_DIR.exists():\n return\n CACHE_ROOT.mkdir(parents=True, exist_ok=True)\n archive_path = CACHE_ROOT / 'cifar-10-python.tar.gz'\n if not archive_path.exists():\n print('Downloading CIFAR-10 (~163 MB, one-time, cached under ~/.cache/cvintro)...')\n urllib.request.urlretrieve(CIFAR_URL, archive_path)\n print('Extracting...')\n with tarfile.open(archive_path) as tar:\n tar.extractall(CACHE_ROOT)\n\ndef load_cifar_batch(path):\n with open(path, 'rb') as f:\n d = pickle.load(f, encoding='bytes')\n imgs = d[b'data'].reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1).astype(np.float32) / 255.0\n labels = np.array(d[b'labels'], dtype=np.int64)\n return imgs, labels\n\nensure_cifar10()\n\nCIFAR_LABELS = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\nCLASSES = ['cat', 'dog', 'automobile', 'truck']\nCLASS_IDS = {name: CIFAR_LABELS.index(name) for name in CLASSES}\n\ntrain_imgs, train_labels = [], []\nfor i in range(1, 6):\n imgs, labels = load_cifar_batch(CACHE_DIR / f'data_batch_{i}')\n train_imgs.append(imgs); train_labels.append(labels)\ntrain_imgs, train_labels = np.concatenate(train_imgs), np.concatenate(train_labels)\ntest_imgs, test_labels = load_cifar_batch(CACHE_DIR / 'test_batch')\n\nrng = np.random.default_rng(7)\n\ndef subset_for_classes(imgs, labels, per_class_counts):\n out_imgs, out_labels = [], []\n for i, name in enumerate(CLASSES):\n idx = np.where(labels == CLASS_IDS[name])[0]\n idx = rng.permutation(idx)[:per_class_counts[name]]\n out_imgs.append(imgs[idx])\n out_labels.append(np.full(len(idx), i, dtype=np.int64))\n out_imgs, out_labels = np.concatenate(out_imgs), np.concatenate(out_labels)\n perm = rng.permutation(len(out_imgs))\n return out_imgs[perm], out_labels[perm]\n\n# training set: cat/dog/automobile well represented, truck deliberately starved\nX_train, y_train = subset_for_classes(train_imgs, train_labels,\n {'cat': 400, 'dog': 400, 'automobile': 400, 'truck': 90})\n# test set: balanced across all four, so the imbalance is purely a training-time artifact\nX_test, y_test = subset_for_classes(test_imgs, test_labels,\n {'cat': 150, 'dog': 150, 'automobile': 150, 'truck': 150})\n\nprint('training class counts:', dict(zip(CLASSES, (int(c) for c in np.bincount(y_train, minlength=4)))))\nprint('test class counts: ', dict(zip(CLASSES, (int(c) for c in np.bincount(y_test, minlength=4)))))\n\nfig, axes = plt.subplots(1, 4, figsize=(8, 2.2))\nfor ax, name in zip(axes, CLASSES):\n ax.imshow(X_train[y_train == CLASSES.index(name)][0])\n ax.set_title(name, fontsize=9)\n ax.axis('off')\nplt.show()" }, { "cell_type": "code", "execution_count": null, "id": "b2a478d6", "metadata": {}, "outputs": [], "source": [ "class CNN(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.conv = nn.Sequential(\n", " nn.Conv2d(3, 16, 5, padding=2), nn.ReLU(),\n", " nn.MaxPool2d(2),\n", " nn.Conv2d(16, 32, 5, padding=2), nn.ReLU(),\n", " nn.MaxPool2d(2),\n", " nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),\n", " nn.AdaptiveMaxPool2d(1),\n", " )\n", " self.fc = nn.Linear(64, 4)\n", "\n", " def forward(self, x):\n", " return self.fc(self.conv(x).flatten(1))\n", "\n", "torch.manual_seed(0)\n", "model = CNN()\n", "opt = torch.optim.Adam(model.parameters(), lr=0.0007)\n", "Xt = torch.tensor(X_train).permute(0, 3, 1, 2); yt = torch.tensor(y_train)\n", "for _ in range(150):\n", " opt.zero_grad()\n", " loss = F.cross_entropy(model(Xt), yt)\n", " loss.backward()\n", " opt.step()\n", "\n", "Xte = torch.tensor(X_test).permute(0, 3, 1, 2)\n", "with torch.no_grad():\n", " preds = model(Xte).argmax(1).numpy()\n", "\n", "acc = (preds == y_test).mean()\n", "print(f'overall test accuracy: {acc:.1%}')" ] }, { "cell_type": "markdown", "id": "26cf352b", "metadata": {}, "source": [ "Just over 50% overall accuracy, on a 4-class task where chance is 25%, sounds like a reasonable result for a small from-scratch CNN on real photos. It hides something important: the model is not equally good at all four classes.\n", "\n", "## Confusion matrix\n", "\n", "Row `i`, column `j` counts test examples of true class `i` predicted as class `j`. A perfect classifier is diagonal; everything off the diagonal is a specific, nameable mistake." ] }, { "cell_type": "code", "execution_count": null, "id": "38bf8aa6", "metadata": {}, "outputs": [], "source": "cm = np.zeros((4, 4), dtype=int)\nfor t, p in zip(y_test, preds):\n cm[t, p] += 1\n\nprint('confusion matrix (rows=true, cols=predicted):')\nprint(f'{\"\":>12}' + ''.join(f'{s:>12}' for s in CLASSES))\nfor i, s in enumerate(CLASSES):\n print(f'{s:>12}' + ''.join(f'{cm[i, j]:>12}' for j in range(4)))\n\nfig, ax = plt.subplots(figsize=(4.5, 4))\nim = ax.imshow(cm, cmap='Blues')\nax.set_xticks(range(4)); ax.set_xticklabels(CLASSES, rotation=45)\nax.set_yticks(range(4)); ax.set_yticklabels(CLASSES)\nax.set_xlabel('predicted'); ax.set_ylabel('true')\nfor i in range(4):\n for j in range(4):\n ax.text(j, i, cm[i, j], ha='center', va='center',\n color='white' if cm[i, j] > cm.max() / 2 else 'black')\nplt.title('Confusion matrix')\nplt.tight_layout()\nplt.show()" }, { "cell_type": "markdown", "id": "c0nfus10nmatr1xanimcell", "metadata": {}, "source": [ "The animation below fills in the same matrix one cell at a time: green for a hit (on diagonal), red for a miss (off diagonal).\n", "\n", "\"Animated\n" ] }, { "cell_type": "markdown", "id": "f44a0b1d", "metadata": {}, "source": [ "## True positives, false positives, true negatives, false negatives\n", "\n", "Every cell of the confusion matrix above is a specific kind of correctness or mistake, but the standard vocabulary for talking about them is binary: pick one class and ask only \"is it this, or not?\" Collapsing the 4-class matrix down to \"truck vs. everything else\" gives exactly four outcomes:\n", "\n", "- **True positive (TP)**: actually a truck, predicted truck. A hit.\n", "- **False negative (FN)**: actually a truck, predicted something else. A miss — the model let it slip past.\n", "- **False positive (FP)**: actually *not* a truck, predicted truck anyway. A false alarm.\n", "- **True negative (TN)**: actually not a truck, correctly predicted not-truck.\n", "\n", "This is the same 2x2 table underlying every binary classifier's evaluation (a medical test's \"positive/negative\" result, a spam filter's \"spam/not spam\" decision) — a multi-class confusion matrix is just this table computed once per class, with everything off that class's row/column collapsed into \"not this class.\"" ] }, { "cell_type": "code", "execution_count": null, "id": "06300321", "metadata": {}, "outputs": [], "source": [ "cls = CLASSES.index('truck')\n", "tp = cm[cls, cls]\n", "fn = cm[cls, :].sum() - tp # true truck, predicted something else\n", "fp = cm[:, cls].sum() - tp # predicted truck, actually something else\n", "tn = cm.sum() - tp - fn - fp # everything else, correctly not called truck\n", "\n", "print(f'{\"\":>18}{\"predicted truck\":>20}{\"predicted NOT truck\":>24}')\n", "print(f'{\"actually truck\":>18}{tp:>20}{fn:>24}')\n", "print(f'{\"actually NOT truck\":>18}{fp:>20}{tn:>24}')\n", "print()\n", "print(f'TP={tp}, FP={fp}, FN={fn}, TN={tn}, total={tp+fp+fn+tn} (test set size={len(y_test)})')" ] }, { "cell_type": "markdown", "id": "tpfpfntnanimcell", "metadata": {}, "source": [ "The animation below cycles through all four classes, collapsing the full matrix down to each one's TP/FP/FN/TN table — the same collapse performed above for 'truck', generalized.\n", "\n", "\"Animated\n" ] }, { "cell_type": "markdown", "id": "f50b41fd", "metadata": {}, "source": [ "## Precision and recall\n", "\n", "Two numbers per class, computed directly from TP/FP/FN:\n", "- **Recall** = TP / (TP + FN) — of everything that really *was* class `i`, what fraction did the model catch? Low recall means the model misses that class often.\n", "- **Precision** = TP / (TP + FP) — of everything the model *called* class `i`, what fraction actually was? Low precision means the model cries wolf on that class often.\n", "\n", "(A less commonly needed but related pair, built from the other two quadrants: **specificity** = TN / (TN + FP), how well the model avoids false alarms on the negative class, and its complement the **false positive rate** = FP / (FP + TN) = 1 − specificity.)" ] }, { "cell_type": "code", "execution_count": null, "id": "fccace41", "metadata": {}, "outputs": [], "source": [ "print(f'{\"class\":>10} {\"precision\":>10} {\"recall\":>8} {\"support\":>8}')\n", "for i, name in enumerate(CLASSES):\n", " tp = cm[i, i]\n", " fn = cm[i, :].sum() - tp\n", " fp = cm[:, i].sum() - tp\n", " precision = tp / (tp + fp) if (tp + fp) > 0 else float('nan')\n", " recall = tp / (tp + fn) if (tp + fn) > 0 else float('nan')\n", " support = cm[i, :].sum()\n", " print(f'{name:>10} {precision:>10.2f} {recall:>8.2f} {support:>8}')" ] }, { "cell_type": "markdown", "id": "prec1s10nrec4llanimcell", "metadata": {}, "source": [ "The animation below cycles through each class, showing precision and recall as fractions of the same TP/FP/FN counts: a recall bar split into caught (TP) vs. missed (FN), a precision bar split into correct (TP) vs. false alarm (FP).\n", "\n", "\"Animated\n" ] }, { "cell_type": "markdown", "id": "a05739f8", "metadata": {}, "source": [ "For quick reference, since these terms go by several names across various fields:\n", "\n", "| term | formula | also known as |\n", "|---|---|---|\n", "| Precision | TP / (TP + FP) | Positive Predictive Value (PPV) |\n", "| Recall | TP / (TP + FN) | Sensitivity, True Positive Rate (TPR) |\n", "| Specificity | TN / (TN + FP) | True Negative Rate (TNR) |\n", "| False Positive Rate | FP / (TN + FP) = 1 − Specificity | Fallout, False Alarm Rate |\n", "| False Negative Rate | FN / (TP + FN) = 1 − Recall | Miss Rate |" ] }, { "cell_type": "markdown", "id": "2caade78", "metadata": {}, "source": [ "### Bayes' rule\n", "\n", "To see why precision behaves the way it does, it may help to recognize a deeper connection. **Bayes' rule** relates a conditional probability to its reverse: $P(A\\mid B) = P(B\\mid A)\\,P(A)\\,/\\,P(B)$. Bayes' rule connects precision, $P(\\text{actually } i\\mid\\text{predicted } i)$, with sensitivity/specificity, $P(\\text{predicted } i\\mid\\text{actually } i)$:\n", "\n", "$$P(\\text{actually } i\\mid\\text{predicted } i) = \\frac{P(\\text{predicted } i\\mid\\text{actually } i)\\,P(\\text{actually } i)}{P(\\text{predicted } i)}.$$\n", "\n", "The $P(\\text{actually } i)$ term — the **base rate** or **prior**, how common class `i` is to begin with — matters just as much as how good the classifier is.\n", "\n", "The classic illustration: a disease with 1% true prevalence, and a test that's \"99% accurate\" (99% sensitivity, 99% specificity). You test positive — what's the chance you actually have it? Intuition says ≈99%, but the real answer is closer to 50%. The reason: Out of 10,000 people, 100 actually have the disease, and the test catches 99 of them (TP=99). Of the 9,900 healthy people, a 1% false-positive rate flags about 99 more (FP=99) — so among everyone who tests positive, only about half actually has the disease: precision = 99/198 ≈ 50%, nowhere near 99%. A low base rate lets false positives from the enormous healthy majority swamp the true positives from the rare, small minority.\n", "\n", "Truck shows the same effect: it's the rarest class here, and even its respectable 72% precision (table above) is already fighting the same uphill arithmetic that makes precision (also known as \"positive predictive value\") punishing for any rare condition, medical or otherwise." ] }, { "cell_type": "markdown", "id": "12b0845c", "metadata": {}, "source": [ "### Precision-recall curve\n", "\n", "So far, each class's precision and recall came from a single, fixed decision rule: whichever class scores highest wins (hard argmax). But the model doesn't actually output a single verdict — for \"truck vs. everything else,\" it outputs a continuous score (the softmax probability assigned to truck) for every test image. Thresholding that score directly and sweeping the threshold from 0 to 1 traces out a **precision-recall curve**: the complete tradeoff between catching more trucks and crying wolf less often. Hard argmax isn't quite the same decision rule — it depends on how truck's score compares to the *other three* classes, not just whether it clears some fixed number — so the point it actually lands on (marked in red) sits close to, but not exactly on, this single-score curve.\n", "\n", "The raw curve below is jagged: with so few truck examples, a single unlucky (or lucky) false positive can make precision dip and then recover as the threshold relaxes further, which isn't a real change in the underlying tradeoff — just noise from a small test set. The standard fix (used, for instance, to compute \"average precision\") is to replace each point's precision with the *best* precision achieved at that recall or higher, producing a monotonically non-increasing envelope that casts a \"shadow\" leftward over the noisy dips." ] }, { "cell_type": "code", "execution_count": null, "id": "bd693dcf", "metadata": {}, "outputs": [], "source": [ "with torch.no_grad():\n", " probs = F.softmax(model(Xte), dim=1).numpy()\n", "\n", "scores = probs[:, cls] # softmax probability assigned to 'truck'\n", "is_truck = (y_test == cls)\n", "\n", "thresholds = np.linspace(0, 1, 101)\n", "precisions, recalls = [], []\n", "for t in thresholds:\n", " pred_pos = scores >= t\n", " tp_t = (pred_pos & is_truck).sum()\n", " fp_t = (pred_pos & ~is_truck).sum()\n", " fn_t = (~pred_pos & is_truck).sum()\n", " precisions.append(tp_t / (tp_t + fp_t) if tp_t + fp_t > 0 else 1.0)\n", " recalls.append(tp_t / (tp_t + fn_t) if tp_t + fn_t > 0 else 0.0)\n", "\n", "# the raw curve is jagged (precision isn't monotonic as the threshold relaxes --\n", "# one unlucky false positive can make it dip, then recover). The standard fix,\n", "# used e.g. to compute \"average precision\": replace each point's precision with\n", "# the BEST precision achieved at that recall or higher -- a monotonically\n", "# non-increasing envelope that \"casts a shadow\" leftward over the noisy dips.\n", "# Different thresholds can land on the exact same recall (relaxing the\n", "# threshold past a non-truck image changes precision but not recall), so ties\n", "# are first collapsed to their best precision before taking the running max --\n", "# otherwise a later, worse-precision point at the same recall can silently\n", "# undercut an earlier, better one.\n", "order = np.argsort(recalls)\n", "r_sorted = np.array(recalls)[order]\n", "p_sorted = np.array(precisions)[order]\n", "uniq_r, inverse = np.unique(r_sorted, return_inverse=True)\n", "uniq_p = np.zeros_like(uniq_r)\n", "np.maximum.at(uniq_p, inverse, p_sorted)\n", "envelope = np.maximum.accumulate(uniq_p[::-1])[::-1]\n", "\n", "plt.plot(recalls, precisions, marker='.', color='#aac4e0', linewidth=1.2, label='raw (noisy)')\n", "# where='pre': each point's value extends LEFTWARD (toward lower recall) onto\n", "# the gap before it -- matching max_{r'>=r} P(r'), and leaving points that are\n", "# already their own local max untouched, instead of dragging a peak forward\n", "# onto later points that don't need it.\n", "plt.step(uniq_r, envelope, where='pre', color='#1f6fb4', linewidth=2.2,\n", " label='envelope (monotonic)')\n", "plt.scatter([tp / (tp + fn)], [tp / (tp + fp)], color='red', marker='x', s=60, zorder=5,\n", " label='hard-argmax point')\n", "plt.xlabel('recall'); plt.ylabel('precision')\n", "plt.title('truck: precision-recall curve as the decision threshold sweeps')\n", "plt.legend(fontsize=8)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "e85f7a92", "metadata": {}, "source": [ "The animation below makes the mechanism behind that curve concrete: a threshold sweeps across every test image's truck score (sorted left to right), coloring each one TP/FP/FN/TN live as it crosses, while the (recall, precision) point crawls along the same curve shown above.\n", "\n", "\"Animated\n" ] }, { "cell_type": "markdown", "id": "171afc8c", "metadata": {}, "source": [ "Truck — the class starved to 90 training images, versus 400 for each other class — has high precision but low recall: when the model does say \"truck,\" it's usually right, but it fails to recognize the overwhelming majority of actual trucks, defaulting instead to whichever classes it saw plenty of during training (mostly \"automobile,\" the visually closest well-represented class). That is the standard signature of class imbalance, and it is completely invisible in the single overall-accuracy number from before. Cat and dog, by contrast, are both well-represented in training but get confused *with each other* far more than with automobile or truck — a different failure mode entirely, caused by genuine visual similarity between the two animals rather than by lack of data. Automobile ends up with unusually high recall but not-so-great precision, for the same reason truck's recall suffered: it's absorbing the starved truck class's misclassifications, since \"wheeled vehicle\" is an easy fallback guess once the model can't tell trucks apart from automobiles.\n", "\n", "The practical lesson: always inspect the confusion matrix and per-class metrics before trusting a single accuracy figure, especially on a dataset where classes aren't balanced.\n", "\n", "### Exercises\n", "\n", "1. Increase the truck training count from 90 to 400 (matching the other three classes) and rerun. Does truck recall recover? Does the cat/dog confusion also improve, or does it persist — and why would more data not fix a problem caused by genuine visual similarity rather than data scarcity?\n", "2. Try weighting the loss by inverse class frequency (`F.cross_entropy(logits, yt, weight=class_weights)`, where `class_weights[i] = 1 / count(class i)`) instead of collecting more truck images. Does it recover truck recall, and at what cost to the other classes' precision?\n", "3. Compute the **F1 score** (the harmonic mean of precision and recall, `2 * p * r / (p + r)`) for each class. Why might F1 be a better single number to track per-class than accuracy, when accuracy is only meaningful in aggregate?" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 5 }