{ "cells": [ { "cell_type": "markdown", "id": "f0b42049", "metadata": {}, "source": [ "# Lesson 32: Multi-Layer Perceptrons\n", "\n", "Lesson 31's single neuron, trained however carefully, could not beat chance on the ring-inside-a-disk dataset — a single linear projection just isn't expressive enough. This lesson adds one more layer: instead of *one* projection followed by a threshold, use *two* projections with a nonlinearity in between. That's it. That's the entire idea behind every deep network in this course — stack more of these." ] }, { "cell_type": "code", "execution_count": null, "id": "cca3cfad", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import torch\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "id": "70d06988", "metadata": {}, "source": [ "## The unsolvable problem, again" ] }, { "cell_type": "code", "execution_count": null, "id": "4731bcc1", "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng(0)\n", "theta_in = rng.uniform(0, 2 * np.pi, 60)\n", "inner = np.stack([0.5 * np.cos(theta_in), 0.5 * np.sin(theta_in)], axis=1) + rng.normal(0, 0.1, (60, 2))\n", "theta_out = rng.uniform(0, 2 * np.pi, 60)\n", "outer = np.stack([2.0 * np.cos(theta_out), 2.0 * np.sin(theta_out)], axis=1) + rng.normal(0, 0.15, (60, 2))\n", "X = np.vstack([inner, outer])\n", "y = np.concatenate([np.zeros(60), np.ones(60)])\n", "\n", "plt.scatter(*inner.T, s=15, label='inner (y=0)')\n", "plt.scatter(*outer.T, s=15, label='outer (y=1)')\n", "plt.legend(fontsize=8)\n", "plt.gca().set_aspect('equal')\n", "plt.title('No single linear projection separates these (Lessons 30-31)')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "00f80cdb", "metadata": {}, "source": [ "## Two projections, one nonlinearity\n", "\n", "A **multi-layer perceptron (MLP)** chains a *hidden* projection into a new space, a nonlinear **activation** applied elementwise, and then a final linear projection (exactly Lesson 31's neuron) on top of the *transformed* coordinates:\n", "\n", "$$h = \\text{ReLU}(W_1 x + b_1), \\qquad z = w_2^\\top h + b_2, \\qquad p = \\sigma(z)$$\n", "\n", "$\\text{ReLU}(u) = \\max(0, u)$ is the simplest common activation: it's linear everywhere except a single kink at zero. That one kink, applied independently to every unit of $h$, is enough — the hidden layer can bend, fold, and stretch the input space so a *linear* boundary in the transformed space corresponds to a highly nonlinear boundary back in the original coordinates." ] }, { "cell_type": "markdown", "id": "5d26b280", "metadata": {}, "source": [ "### Why the nonlinearity is essential\n", "\n", "Without ReLU — if the hidden layer were left as plain $h = W_1 x + b_1$ — the two layers would collapse into a single linear projection:\n", "\n", "$$z = w_2^\\top(W_1 x + b_1) + b_2 = \\underbrace{(w_2^\\top W_1)}_{w'^\\top} x + \\underbrace{(w_2^\\top b_1 + b_2)}_{b'}$$\n", "\n", "That's exactly Lesson 31's single neuron again, just with the two layers' weights multiplied together into one $w'$ and $b'$. This is the flip side of Lesson 30's observation that matrix multiplication is projection, stacked: composing two matrix multiplications is *still* just one matrix multiplication, so stacking purely linear layers buys nothing, no matter how many you chain. ReLU's kink is what breaks that collapse — because it isn't a linear function, $w_2^\\top \\text{ReLU}(W_1x+b_1)+b_2$ can no longer be rewritten as any single $w'^\\top x + b'$." ] }, { "cell_type": "code", "execution_count": null, "id": "a35bacc4", "metadata": {}, "outputs": [], "source": [ "demo_rng = np.random.default_rng(3)\n", "W1_demo = demo_rng.normal(size=(2, 4))\n", "b1_demo = demo_rng.normal(size=4)\n", "W2_demo = demo_rng.normal(size=(4, 1))\n", "b2_demo = demo_rng.normal(size=1)\n", "\n", "z_two_linear_layers = (X @ W1_demo + b1_demo) @ W2_demo + b2_demo # no nonlinearity in between\n", "\n", "W_combined = W1_demo @ W2_demo # a single (2, 1) matrix\n", "b_combined = b1_demo @ W2_demo + b2_demo # a single offset\n", "z_one_combined_layer = X @ W_combined + b_combined\n", "\n", "print(f'max diff, two LINEAR layers vs. one combined layer: '\n", " f'{np.abs(z_two_linear_layers - z_one_combined_layer).max():.2e}')" ] }, { "cell_type": "markdown", "id": "cddb578c", "metadata": {}, "source": [ "## Backprop through two layers\n", "\n", "The chain rule extends cleanly: propagate the error signal $\\partial L/\\partial z$ from Lesson 31 backward through the output projection to get $\\partial L/\\partial h$, then backward through the ReLU and the hidden projection to get $\\partial L/\\partial W_1, \\partial L/\\partial b_1$. Every step is either \"multiply by a weight matrix's transpose\" or \"multiply elementwise by an activation's derivative\" — this is *all* backpropagation ever is, no matter how many layers are stacked." ] }, { "cell_type": "code", "execution_count": null, "id": "f9bea134", "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " return 1 / (1 + np.exp(-z))\n", "\n", "def relu(z):\n", " return np.maximum(0, z)\n", "\n", "def relu_deriv(z):\n", " return (z > 0).astype(np.float64)\n", "\n", "def forward(X, W1, b1, W2, b2):\n", " z1 = X @ W1 + b1\n", " a1 = relu(z1)\n", " z2 = (a1 @ W2 + b2).ravel()\n", " p = sigmoid(z2)\n", " return p, (z1, a1, z2)\n", "\n", "def backward(X, y, p, cache, W2):\n", " z1, a1, z2 = cache\n", " n = len(y)\n", " grad_z2 = ((p - y) / n).reshape(-1, 1)\n", " grad_W2 = a1.T @ grad_z2\n", " grad_b2 = grad_z2.sum(axis=0)\n", " grad_a1 = grad_z2 @ W2.T\n", " grad_z1 = grad_a1 * relu_deriv(z1)\n", " grad_W1 = X.T @ grad_z1\n", " grad_b1 = grad_z1.sum(axis=0)\n", " return grad_W1, grad_b1, grad_W2, grad_b2" ] }, { "cell_type": "markdown", "id": "42272be4", "metadata": {}, "source": [ "### Sanity check against PyTorch autograd" ] }, { "cell_type": "code", "execution_count": null, "id": "c4d3aa7a", "metadata": {}, "outputs": [], "source": [ "H = 4\n", "init_rng = np.random.default_rng(8)\n", "W1 = init_rng.normal(size=(2, H)) * 0.7\n", "b1 = np.zeros(H)\n", "W2 = init_rng.normal(size=(H, 1)) * 0.7\n", "b2 = np.zeros(1)\n", "\n", "p, cache = forward(X, W1, b1, W2, b2)\n", "grad_W1, grad_b1, grad_W2, grad_b2 = backward(X, y, p, cache, W2)\n", "\n", "X_t = torch.tensor(X)\n", "y_t = torch.tensor(y)\n", "W1_t = torch.tensor(W1, requires_grad=True)\n", "b1_t = torch.tensor(b1, requires_grad=True)\n", "W2_t = torch.tensor(W2, requires_grad=True)\n", "b2_t = torch.tensor(b2, requires_grad=True)\n", "\n", "z1_t = X_t @ W1_t + b1_t\n", "a1_t = torch.relu(z1_t)\n", "z2_t = (a1_t @ W2_t + b2_t).squeeze(-1)\n", "loss_t = torch.nn.functional.binary_cross_entropy_with_logits(z2_t, y_t)\n", "loss_t.backward()\n", "\n", "print(f'W1 max diff: {np.abs(grad_W1 - W1_t.grad.numpy()).max():.2e}')\n", "print(f'W2 max diff: {np.abs(grad_W2 - W2_t.grad.numpy()).max():.2e}')\n", "print(f'b1 max diff: {np.abs(grad_b1 - b1_t.grad.numpy()).max():.2e}')\n", "print(f'b2 max diff: {np.abs(grad_b2 - b2_t.grad.numpy()).max():.2e}')" ] }, { "cell_type": "markdown", "id": "8a366eb8", "metadata": {}, "source": [ "## Training" ] }, { "cell_type": "code", "execution_count": null, "id": "a728c7ab", "metadata": {}, "outputs": [], "source": [ "lr = 0.1\n", "losses = []\n", "for epoch in range(3000):\n", " p, cache = forward(X, W1, b1, W2, b2)\n", " eps = 1e-9\n", " losses.append(-np.mean(y * np.log(p + eps) + (1 - y) * np.log(1 - p + eps)))\n", " grad_W1, grad_b1, grad_W2, grad_b2 = backward(X, y, p, cache, W2)\n", " W1 -= lr * grad_W1; b1 -= lr * grad_b1\n", " W2 -= lr * grad_W2; b2 -= lr * grad_b2\n", "\n", "final_pred = (p > 0.5).astype(float)\n", "print(f'final accuracy: {(final_pred == y).mean():.1%} (single neuron, Lesson 31, managed 50%)')\n", "\n", "plt.plot(losses)\n", "plt.xlabel('epoch'); plt.ylabel('loss')\n", "plt.title('Training loss (2-layer MLP)')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "99d88892", "metadata": {}, "source": [ "## The hidden layer's projections, made visible\n", "\n", "Each hidden unit is itself a Lesson 30-style projection: a direction $w_i$ (one column of $W_1$) and an offset $b_i$ (one entry of $b_1$), together defining a cut $w_i \\cdot x + b_i = 0$ that ReLU zeroes out on one side of. With only 2 input dimensions, all $H$ of these can be drawn directly over the original data." ] }, { "cell_type": "code", "execution_count": null, "id": "53cf0ae9", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(5, 5))\n", "ax.scatter(*X[y == 0].T, s=15, label='inner')\n", "ax.scatter(*X[y == 1].T, s=15, label='outer')\n", "\n", "colors = plt.cm.tab10.colors\n", "for i in range(W1.shape[1]):\n", " w_i, b_i = W1[:, i], b1[i]\n", " # the cut {x : w_i.x + b_i = 0}, drawn through its closest point to the origin\n", " foot = -b_i * w_i / (w_i @ w_i)\n", " perp = np.array([-w_i[1], w_i[0]])\n", " p1, p2 = foot + 5 * perp, foot - 5 * perp\n", " ax.plot([p1[0], p2[0]], [p1[1], p2[1]], color=colors[i], linewidth=1.5, label=f'hidden unit {i}')\n", " ax.arrow(*foot, *(w_i / np.linalg.norm(w_i) * 0.5), head_width=0.1, color=colors[i], length_includes_head=True)\n", "\n", "ax.set_xlim(-3, 3); ax.set_ylim(-3, 3)\n", "ax.set_aspect('equal')\n", "ax.legend(fontsize=7, loc='upper right')\n", "ax.set_title('Each hidden unit is one projection direction $w_i$,\\ncutting the space along $w_i \\\\cdot x + b_i = 0$')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "3c469712", "metadata": {}, "source": [ "On its own, each cut is just a straight line — no single one of them separates the ring from the disk. But the output layer combines all $H$ of their (ReLU'd) results together, and a handful of straight cuts arranged around the ring, combined with the right weights, can approximate a closed curve well enough to enclose it. That combining step is exactly what the rest of this section visualizes." ] }, { "cell_type": "markdown", "id": "8d016c9d", "metadata": {}, "source": [ "## What the hidden layer actually did\n", "\n", "The output layer is *just* a linear projection (Lesson 31's neuron) — but it acts on $h$, the hidden layer's output, not on the original $x$. If the hidden layer did its job, the *transformed* points should already be much easier to separate with a straight line. Since $h$ lives in $\\mathbb{R}^4$ here, we use PCA (Lesson 6) to visualize it in 2D." ] }, { "cell_type": "code", "execution_count": null, "id": "457bc3cf", "metadata": {}, "outputs": [], "source": [ "_, (_, hidden, _) = forward(X, W1, b1, W2, b2)\n", "\n", "hidden_centered = hidden - hidden.mean(axis=0)\n", "cov = np.cov(hidden_centered.T)\n", "eigvals, eigvecs = np.linalg.eigh(cov)\n", "top2_directions = eigvecs[:, -2:]\n", "hidden_2d = hidden_centered @ top2_directions\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(9, 4))\n", "axes[0].scatter(*X[y == 0].T, s=15, label='inner')\n", "axes[0].scatter(*X[y == 1].T, s=15, label='outer')\n", "axes[0].set_aspect('equal')\n", "axes[0].set_title('Original input space')\n", "axes[0].legend(fontsize=7)\n", "\n", "axes[1].scatter(*hidden_2d[y == 0].T, s=15, label='inner')\n", "axes[1].scatter(*hidden_2d[y == 1].T, s=15, label='outer')\n", "axes[1].set_title('Hidden representation\\n(top 2 PCA directions of a 4D space)')\n", "axes[1].legend(fontsize=7)\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "9ce98008", "metadata": {}, "source": [ "Even this lossy 2D snapshot of the full 4D hidden space shows the two classes pulled apart into something close to linearly separable — a big improvement over the 74% ceiling that was the best *any* line could do in the original space (Lesson 30). The actual output neuron works in the full 4D hidden space, where it achieves 100% accuracy exactly. This is the entire mechanism of deep learning in miniature: each layer *reshapes the space* so that the next layer's job gets easier, until the final layer's job is trivial — a single linear projection." ] }, { "cell_type": "markdown", "id": "cdbce6d0", "metadata": {}, "source": [ "## Model capacity: how many hidden units are enough?\n", "\n", "A model's **capacity** is how rich a family of functions it can represent. Here that's set directly by $H$, the hidden layer's width: each hidden unit contributes one straight cut (Lesson 30-style), and the output layer combines these units. Too few units, and the model simply cannot represent the boundary needed by the dataset — no amount of training can find a solution that doesn't exist. Sweep $H$ and retrain from scratch at each width, across several random initializations, to see the ceiling directly." ] }, { "cell_type": "code", "execution_count": null, "id": "ad482f2e", "metadata": {}, "outputs": [], "source": [ "def train_and_eval(H, seed, epochs=3000, lr=0.1):\n", " init_rng = np.random.default_rng(seed)\n", " W1_s = init_rng.normal(size=(2, H)) * 0.7\n", " b1_s = np.zeros(H)\n", " W2_s = init_rng.normal(size=(H, 1)) * 0.7\n", " b2_s = np.zeros(1)\n", " for _ in range(epochs):\n", " p, cache = forward(X, W1_s, b1_s, W2_s, b2_s)\n", " gW1, gb1, gW2, gb2 = backward(X, y, p, cache, W2_s)\n", " W1_s -= lr * gW1; b1_s -= lr * gb1\n", " W2_s -= lr * gW2; b2_s -= lr * gb2\n", " p, _ = forward(X, W1_s, b1_s, W2_s, b2_s)\n", " return ((p > 0.5).astype(float) == y).mean()\n", "\n", "widths = [1, 2, 3, 4, 6, 8, 16]\n", "seeds = range(5)\n", "mean_accs, min_accs, max_accs = [], [], []\n", "for H_test in widths:\n", " accs = [train_and_eval(H_test, seed) for seed in seeds]\n", " mean_accs.append(np.mean(accs)); min_accs.append(np.min(accs)); max_accs.append(np.max(accs))\n", " print(f'H={H_test:>2}: mean acc={np.mean(accs):.1%} (min={np.min(accs):.1%}, max={np.max(accs):.1%}, over {len(seeds)} seeds)')\n", "\n", "plt.figure(figsize=(5.5, 3.5))\n", "plt.plot(widths, mean_accs, '-o', label='mean')\n", "plt.fill_between(widths, min_accs, max_accs, alpha=0.2, label='min-max range')\n", "plt.xlabel('hidden units ($H$)'); plt.ylabel('final accuracy')\n", "plt.title('Accuracy vs. capacity, across 5 random seeds each')\n", "plt.legend(fontsize=8)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "4bedea62", "metadata": {}, "source": [ "$H=1$ caps out well below 100% no matter how long it trains or which seed it starts from — that's not an optimization failure, it's a representational ceiling: one hidden unit is just a single cut, so it inherits Lesson 30's ~74% linear limit almost exactly. $H=2$ is better, but it is still capacity-limited — none of the five seeds above reach 100%. $H=3$ is the actual knife's edge: the capacity to represent a perfect solution is there (one seed above reaches exactly 100%), but whether training finds it depends on *which* seed it starts from (a preview of Lesson 33's optimization landscape problems). \n", "\n", "By $H=4$, every seed reaches 100% reliably: there's enough capacity that the network stops being the bottleneck, and training reliably finds a solution within that capacity. This is the general shape of a capacity curve — a hard floor set by what the model *can* represent, with optimization difficulty riding on top of it near the floor's edge. This same curve is revisited in Lesson 35 from the opposite direction: there, capacity is held generously high and *data* is scarce, which turns \"enough capacity to represent the right answer\" into \"also enough capacity to memorize the wrong one.\"\n", "\n", "Width isn't the only knob. **Depth** — stacking more layers instead of making a layer wider — is the other axis, and it isn't just \"width in disguise\": a deep, narrow network can represent some functions far more compactly than a shallow, wide one needs to (fewer total units for the same expressiveness), but it introduces its own failure mode instead. Lesson 37 picks this up directly: stacking enough layers can make gradients vanish before they ever reach the earliest ones, a problem not encountered by this lesson's shallow 2-layer network." ] }, { "cell_type": "markdown", "id": "3696d165", "metadata": {}, "source": [ "### Exercises\n", "\n", "1. The capacity sweep above used ReLU. Rerun it with `tanh` instead (swap `forward`/`backward`'s activation and its derivative throughout `train_and_eval`, mirroring Exercise 2 below). Does the same $H=1$ ceiling and $H=3$-to-$H=4$ transition appear, or does a smooth activation change where the floor sits?\n", "2. Replace `relu`/`relu_deriv` with `tanh`/its derivative ($1 - \\tanh^2$) throughout, and retrain. Does it still solve the problem? Compare the resulting loss curve's shape to the ReLU version's.\n", "3. Remove the nonlinearity entirely (replace `relu(z1)` with just `z1` in `forward`, and `relu_deriv(z1)` with an array of ones in `backward`). Confirm the network can no longer beat Lesson 31's ~50% ceiling — consistent with the collapse-to-a-single-projection argument above, now demonstrated on the actual training dynamics instead of just the static algebra." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.x" } }, "nbformat": 4, "nbformat_minor": 5 }