{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lesson 35: Classic Architectures\n",
"\n",
"Lessons 33-34 built and trained one small CNN. This lesson looks at how CNN *architectures* evolved over roughly two decades, and works through the single biggest architectural idea in that history in detail: the residual connection, and the vanishing-gradient problem it was designed to fix."
]
},
{
"cell_type": "code",
"id": "e4422d59",
"source": "import numpy as np\nimport torch\nimport torch.nn as nn\nimport matplotlib.pyplot as plt",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "2e290d56",
"source": "## A brief tour\n\nA handful of architectures defined the field, each solving a specific problem with the previous one:\n\n- **LeNet-5** (\"Gradient-Based Learning Applied to Document Recognition\", LeCun et al., 1998★) — the template Lesson 33 already implements: conv, pool, conv, pool, then fully-connected layers. Built for digit recognition on 32x32 images.\n- **AlexNet** (Krizhevsky, Sutskever & Hinton, 2012★) — the same template, just much bigger (5 conv layers, ~60M parameters), trained on **ImageNet** (Deng et al., 2009★) using GPUs and ReLU instead of tanh/sigmoid. Its win in the 2012 ImageNet competition (top-5 error dropping from ~26% to ~15%) is usually cited as the event that restarted mainstream interest in neural networks.\n- **VGG** (Simonyan & Zisserman, 2014★) — replaced AlexNet's large 11x11/5x5 filters with a deep stack of small 3x3 convolutions. Two stacked 3x3 convs have the same *receptive field* as one 5x5 conv (Lesson 11's pyramid idea again) but fewer parameters and an extra nonlinearity in between.\n- **GoogLeNet / Inception** (Szegedy et al., 2015★) — went wider instead of just deeper: each \"Inception module\" runs several convolution sizes (1x1, 3x3, 5x5) side by side on the same input and concatenates their outputs, so the network doesn't have to commit to one filter size per layer. 1x1 convolutions are used first to cheaply shrink the channel count, keeping the wider module affordable.\n- **ResNet** (He et al., 2015★) — the subject of the rest of this lesson. Pushed depth from VGG's ~19 layers to 50, 101, even 152, by solving the problem that had made naively stacking more layers *worse*, not better.",
"metadata": {}
},
{
"cell_type": "markdown",
"id": "e36db620",
"source": "## The vanishing-gradient problem\n\nWhy did stacking more layers make plain networks *worse*? Backprop's chain rule multiplies a gradient by every layer's local Jacobian on its way back to the input. If those per-layer factors are consistently smaller than 1 — which happens easily with a saturating activation like sigmoid, whose derivative is at most 0.25 — a deep enough stack multiplies the gradient by a very small number many times over. The gradient reaching early layers shrinks toward zero, and those layers stop learning at all, even though nothing is mathematically wrong with the network.\n\nBuild a 30-layer, sigmoid-activated network of `Linear` layers and track the gradient magnitude at every depth, for two versions: a **plain** stack `x = sigmoid(layer(x))`, and a **residual** stack `x = x + 0.3 * sigmoid(layer(x))` where each layer only has to learn a small *correction* added to its input, rather than replacing it outright.",
"metadata": {}
},
{
"cell_type": "code",
"id": "e5e6a7ba",
"source": "DEPTH = 30\nWIDTH = 32\n\ndef make_layers(seed):\n torch.manual_seed(seed)\n layers = nn.ModuleList([nn.Linear(WIDTH, WIDTH) for _ in range(DEPTH)])\n for layer in layers:\n nn.init.xavier_normal_(layer.weight, gain=nn.init.calculate_gain('sigmoid'))\n return layers\n\nclass PlainDeepNet(nn.Module):\n def __init__(self, seed):\n super().__init__()\n self.layers = make_layers(seed)\n\n def forward(self, x):\n acts = [x]\n for layer in self.layers:\n x = torch.sigmoid(layer(x))\n x.retain_grad()\n acts.append(x)\n return x, acts\n\nclass ResidualDeepNet(nn.Module):\n def __init__(self, seed):\n super().__init__()\n self.layers = make_layers(seed)\n\n def forward(self, x):\n acts = [x]\n for layer in self.layers:\n x = x + 0.3 * torch.sigmoid(layer(x))\n x.retain_grad()\n acts.append(x)\n return x, acts\n\ndef grad_norms(model_cls):\n torch.manual_seed(0)\n x = torch.randn(8, WIDTH, requires_grad=True)\n model = model_cls(seed=1)\n out, acts = model(x)\n out.sum().backward()\n return [a.grad.norm().item() for a in acts if a.grad is not None]\n\nplain_norms = grad_norms(PlainDeepNet)\nres_norms = grad_norms(ResidualDeepNet)\n\nprint(f'plain net: gradient norm at layer 0 / gradient norm at layer {DEPTH-1} = {plain_norms[0] / plain_norms[-1]:.2e}')\nprint(f'residual net: gradient norm at layer 0 / gradient norm at layer {DEPTH-1} = {res_norms[0] / res_norms[-1]:.2e}')\n\nplt.figure(figsize=(6, 4))\nplt.semilogy(plain_norms, label='plain (sigmoid) net')\nplt.semilogy(res_norms, label='residual net')\nplt.xlabel('layer depth (0 = input)')\nplt.ylabel('gradient norm (log scale)')\nplt.title('Gradient magnitude vs. depth')\nplt.legend()\nplt.show()",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "b556f1ea",
"source": "The plain network's gradient shrinks by roughly nineteen orders of magnitude between the last layer and the first — for all practical purposes, the early layers receive no learning signal at all. The residual network's gradient stays essentially flat across all 30 layers.\n\nThe reason is structural, not a matter of tuning: with a residual connection, `x_{l+1} = x_l + F(x_l)`, so `dx_{l+1}/dx_l = I + dF/dx_l`. Backprop multiplies these Jacobians together across layers, but every one of them contains an identity matrix `I` as a direct term. That gives the gradient a path straight back to the input that never gets multiplied by a small sigmoid derivative — an unobstructed shortcut, regardless of how deep the stack gets. The plain network has no such path: every layer's Jacobian is purely `dF/dx_l`, so there's nothing to stop repeated multiplication from driving the product toward zero.",
"metadata": {}
},
{
"cell_type": "markdown",
"id": "a7d53eac",
"source": "## Batch normalization\n\nThe convolutional residual block below uses one more ingredient: **batch normalization** (Ioffe & Szegedy, 2015). The idea is simple — for each channel, subtract that channel's mean and divide by its standard deviation, computed *across the current mini-batch* (over the batch, height, and width dimensions, separately per channel), so every channel's activations always have mean 0 and standard deviation 1 going into the next layer. Two learnable parameters, a per-channel scale `gamma` and shift `beta`, are then applied on top, so the layer can still recover a different mean/scale if that's actually useful — normalization to 0/1 is a *starting point* the network can undo, not a hard constraint.\n\nDeep networks without batch norm are prone to a related but distinct problem from vanishing gradients: as training updates early layers, the *distribution* of activations feeding into later layers keeps shifting (sometimes called \"internal covariate shift\"), so later layers are constantly chasing a moving target. Renormalizing at every layer keeps that distribution stable, which in practice lets much higher learning rates be used and makes deep networks noticeably easier to train — one of the reasons ResNet could push to 50+ layers where earlier architectures struggled past ~20.",
"metadata": {}
},
{
"cell_type": "code",
"id": "bff9bab2",
"source": "C = 4\nx_bn = torch.randn(8, C, 5, 5)\n\nbn = nn.BatchNorm2d(C)\nbn.train()\nout_torch = bn(x_bn)\n\n# from scratch: normalize each channel over (batch, height, width), then scale + shift\nmean = x_bn.mean(dim=(0, 2, 3), keepdim=True)\nvar = x_bn.var(dim=(0, 2, 3), unbiased=False, keepdim=True)\nx_norm = (x_bn - mean) / torch.sqrt(var + bn.eps)\ngamma = bn.weight.view(1, C, 1, 1)\nbeta = bn.bias.view(1, C, 1, 1)\nout_manual = gamma * x_norm + beta\n\nprint(f'max abs diff vs nn.BatchNorm2d: {(out_manual - out_torch).abs().max().item():.2e}')\nprint(f'per-channel mean before: {x_bn.mean(dim=(0, 2, 3)).detach().numpy().round(3)}')\nprint(f'per-channel std before: {x_bn.std(dim=(0, 2, 3), unbiased=False).detach().numpy().round(3)}')\nprint(f'per-channel mean after: {out_torch.mean(dim=(0, 2, 3)).detach().numpy().round(3)}')\nprint(f'per-channel std after: {out_torch.std(dim=(0, 2, 3), unbiased=False).detach().numpy().round(3)}')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "db57c1d7",
"source": "## A convolutional residual block\n\nIn an actual ResNet, `F` is a pair of small convolutions (with batch normalization and ReLU in between), and the shortcut adds the *input feature map* back onto their output, channel-for-channel and pixel-for-pixel:",
"metadata": {}
},
{
"cell_type": "code",
"id": "07774763",
"source": "class ResidualBlock(nn.Module):\n def __init__(self, channels):\n super().__init__()\n self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)\n self.bn1 = nn.BatchNorm2d(channels)\n self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)\n self.bn2 = nn.BatchNorm2d(channels)\n\n def forward(self, x):\n out = torch.relu(self.bn1(self.conv1(x)))\n out = self.bn2(self.conv2(out))\n return torch.relu(x + out) # the shortcut: add the block's input back in\n\nblock = ResidualBlock(channels=16)\nx = torch.randn(4, 16, 20, 20)\ny = block(x)\nprint(f'input shape: {tuple(x.shape)}')\nprint(f'output shape: {tuple(y.shape)} (unchanged — a residual block preserves shape, so blocks can be stacked freely)')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "ea5a9164",
"source": "### Exercise\n\n1. In `PlainDeepNet`/`ResidualDeepNet` above, change the activation from `torch.sigmoid` to `torch.relu` (ReLU's derivative is 1 for any positive input, not capped at 0.25 like sigmoid's) and rerun the gradient-norm comparison. Does the plain network's vanishing problem get better, worse, or stay about the same?\n2. Change the residual net's scale factor from `0.3` to `1.0` (i.e. `x + torch.sigmoid(layer(x))` with no damping) and rerun. Does the gradient ratio change much? What does that suggest about *why* the residual connection works — is it the scale factor, or the `+ x` shortcut itself?\n3. `ResidualBlock` above requires the input and output to have the same number of channels, since they're added directly. Real ResNets sometimes need to change channel count between blocks (e.g. 16 → 32). Sketch (in words, or in code) what the shortcut path would need to do in that case for the addition to still make sense.",
"metadata": {}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}