{ "cells": [ { "cell_type": "markdown", "id": "47a2dcb6", "metadata": {}, "source": "# Lesson 31: Gradient Descent and Backpropagation\n\nLesson 30 hand-picked a projection direction $w$ (the difference of class means) to separate two classes. That worked, but it required a human to notice a good heuristic. This lesson replaces the human with **gradient descent**: an automatic procedure that *learns* $w$ and $b$ directly from data by repeatedly nudging them to reduce a loss. The recipe — forward pass, loss, backward pass, update — is the entire training loop behind every neural network in this course, no matter how large." }, { "cell_type": "code", "execution_count": null, "id": "19a1b6b9", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import torch\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "id": "gr4d1entd3sc1", "metadata": {}, "source": [ "## Gradient descent: using the slope to find a minimum\n", "\n", "Training a model means minimizing a loss function over its parameters. Before extending the concept to many parameters, let's first see the idea in its simplest form: a 1D function of a single number $x$.\n", "\n", "At any point $x$, the derivative $f'(x)$ is the slope of $f$ at that point: positive means $f$ increases as $x$ increases, negative means it decreases. So taking a small step *opposite* the slope always decreases $f$ (for a small enough step) — that's the entire idea behind gradient descent:\n", "\n", "$$x \\leftarrow x - \\eta\\, f'(x)$$\n", "\n", "where $\\eta$ (the **learning rate**) controls how big a step to take. Repeat this update and $x$ walks downhill, settling near a minimum, where $f'(x) \\approx 0$." ] }, { "cell_type": "markdown", "id": "gr4d1entd3sc3", "metadata": {}, "source": [ "Let $f$ be a parabola. Starting far from the minimum, each step moves against the local slope. The steps shrink on their own as they approach the bottom, since the slope $f'(x)=2(x-3)$ itself shrinks as $x \\to 3$ — gradient descent naturally slows down near a minimum. In the case of a parabola, the local minimum found by gradient descent is also the global minimum, which in this case is at $x=3$." ] }, { "cell_type": "code", "execution_count": null, "id": "gr4d1entd3sc2", "metadata": {}, "outputs": [], "source": [ "def f(x):\n", " return (x - 3) ** 2 + 1\n", "\n", "def fprime(x):\n", " return 2 * (x - 3)\n", "\n", "def gd_path(x0, lr, n_steps=10):\n", " x = x0\n", " path = [x]\n", " for _ in range(n_steps):\n", " x = x - lr * fprime(x)\n", " path.append(x)\n", " return np.array(path)\n", "\n", "path = gd_path(x0=-4.0, lr=0.3)\n", "xs = np.linspace(-5, 8, 200)\n", "\n", "plt.plot(xs, f(xs), color='gray', linewidth=1.5, label='f(x) = (x-3)^2 + 1')\n", "plt.plot(path, f(path), 'o-', color='C1', markersize=5, linewidth=1.2, label='gradient descent steps')\n", "plt.scatter([path[0]], [f(path[0])], color='green', s=70, zorder=3, label='start')\n", "plt.scatter([path[-1]], [f(path[-1])], color='red', s=70, zorder=3, label='after 10 steps')\n", "plt.xlabel('x')\n", "plt.ylabel('f(x)')\n", "plt.title('Gradient descent on a 1D parabola')\n", "plt.legend(fontsize=8)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "gr4d1entd3sc2anim", "metadata": {}, "source": [ "This animation shows the same run one step at a time: the dashed tangent line makes the slope at the current point visible directly, and the arrow is the step taken against it — exactly the update rule above, frame by frame.\n", "\n", "\"Animated" ] }, { "cell_type": "markdown", "id": "gr4d1entd3sc5", "metadata": {}, "source": [ "The step size $\\eta$ matters as much as the direction: too small and progress crawls; too large and successive steps overshoot by more and more each time, diverging instead of converging. Picking a good learning rate is one of the most important practical choices in training any model — a theme Lesson {{lesson:optimization}} returns to in depth." ] }, { "cell_type": "code", "execution_count": null, "id": "gr4d1entd3sc4", "metadata": {}, "outputs": [], "source": [ "fig, axes = plt.subplots(1, 3, figsize=(12, 4))\n", "settings = [(0.05, 'too small: slow'), (0.3, 'good: fast, settles'), (1.05, 'too large: diverges')]\n", "for ax, (lr, label) in zip(axes, settings):\n", " path = gd_path(x0=-4.0, lr=lr, n_steps=10)\n", " ax.plot(xs, f(xs), color='gray', linewidth=1.2)\n", " ax.plot(path, f(path), 'o-', color='C1', markersize=4, linewidth=1)\n", " ax.scatter([path[0]], [f(path[0])], color='green', s=50, zorder=3)\n", " ax.scatter([path[-1]], [f(path[-1])], color='red', s=50, zorder=3)\n", " ax.set_title(f'lr={lr}\\n({label})', fontsize=10)\n", " ax.set_xlabel('x')\n", "axes[0].set_ylabel('f(x)')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "0fb4c305", "metadata": {}, "source": [ "## A new library, and a new data structure: PyTorch and tensors\n", "\n", "Every lesson so far has imported `cv2` for image operations. This lesson imports `torch` instead — the first appearance of PyTorch in the course, and from here on most lessons will lean on it rather than OpenCV. That's because building and training neural networks needs two things OpenCV was never designed for: automatic differentiation (computing gradients of arbitrarily complex functions, used below) and GPU acceleration for the large matrix multiplications that neural networks are built from.\n", "\n", "With PyTorch comes a new word: **tensor**. A tensor is just the general term for a grid of numbers of any dimensionality: a scalar is a 0-dimensional tensor, a vector is 1-dimensional, a matrix is 2-dimensional, and a stack of matrices — a batch of images, say — is a 4-dimensional tensor. Every array used so far in this course — NumPy arrays, images, weight matrices — has really been a tensor all along; PyTorch's `torch.Tensor` is simply NumPy's `ndarray` with two extras bolted on: it can track a `.grad` for automatic differentiation, and it can live on a GPU instead of the CPU for fast, parallel computation. A tensor's `.shape` says exactly what NumPy's `.shape` would say for the same array — the two libraries are close enough that converting between them (`torch.tensor(numpy_array)`, `tensor.numpy()`) is essentially free.\n", "\n", "Image tensors bring two conventions worth calling out up front:\n", "\n", "- **Channel-first.** PyTorch stores color images as `(channel, height, width)`, the opposite of NumPy/OpenCV's `(height, width, channel)` from Lesson {{lesson:images_as_arrays}} — converting between them means moving the channel axis, not just reshaping: `image.transpose(2, 0, 1)` for a NumPy array (HWC→CHW), or `tensor.permute(2, 0, 1)` for a tensor.\n", "- **A mandatory batch dimension.** PyTorch layers and functions almost always expect a batch axis first, even when there's only one example: a single image goes in with shape `(1, channel, height, width)`, not `(channel, height, width)`. Forgetting it is a common source of shape-mismatch errors; `tensor.unsqueeze(0)` inserts a size-1 batch axis at the front, and `tensor.squeeze(0)` removes it again once you're back to handling a single result." ] }, { "cell_type": "markdown", "id": "62b9aea2", "metadata": {}, "source": [ "## A binary classification problem\n", "\n", "We reuse the same two-class dataset from Lesson {{lesson:projection}}, so the learned direction below can be compared directly against the hand-picked one." ] }, { "cell_type": "code", "execution_count": null, "id": "bb7a17b7", "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng(1)\n", "class_a = rng.normal(loc=[-2, -1], scale=0.8, size=(60, 2))\n", "class_b = rng.normal(loc=[2, 1.5], scale=0.8, size=(60, 2))\n", "X = np.vstack([class_a, class_b])\n", "y = np.concatenate([np.zeros(60), np.ones(60)]) # class A = 0, class B = 1\n", "\n", "plt.scatter(*class_a.T, s=15, label='class A (y=0)')\n", "plt.scatter(*class_b.T, s=15, label='class B (y=1)')\n", "plt.legend(fontsize=8)\n", "plt.gca().set_aspect('equal')\n", "plt.title('Same two-class dataset as Lesson {{lesson:projection}}')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "e100244f", "metadata": {}, "source": [ "## From a raw score to a probability\n", "\n", "Recall the projection from Lesson {{lesson:projection}}: a single neuron computes a raw score $z = w^\\top x + b$ from an input $x$, then classifies by that score's sign. The goal of training is to find a $w$ and $b$ that make $z$ come out positive for every point in class B and negative for every point in class A — but instead of hand-picking them the way Lesson {{lesson:projection}} did, we want gradient descent to find them automatically.\n", "\n", "The trouble is that classification accuracy (right or wrong) is flat almost everywhere and jumps discontinuously at the decision boundary — it has no useful gradient to follow, so gradient descent has nothing to climb down. The fix is to stop asking for a hard yes/no, and instead squash the raw score through the **sigmoid** function, which maps any real number to a smooth value between 0 and 1 — read as \"how confident is the model that this point is class B?\"\n", "\n", "$$\\sigma(z) = \\frac{1}{1+e^{-z}}$$\n", "\n", "A large positive $z$ maps close to 1, a large negative $z$ maps close to 0, and $z=0$ maps to exactly 0.5, right on the fence." ] }, { "cell_type": "code", "execution_count": null, "id": "4a497ef8", "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " return 1 / (1 + np.exp(-z))\n", "\n", "zs = np.linspace(-6, 6, 200)\n", "plt.plot(zs, sigmoid(zs))\n", "plt.axhline(0.5, color='gray', linestyle='--', linewidth=1)\n", "plt.axvline(0, color='gray', linestyle='--', linewidth=1)\n", "plt.xlabel('z = w.x + b')\n", "plt.ylabel('sigma(z)')\n", "plt.title('The sigmoid activation')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "19307cf7", "metadata": {}, "source": [ "## Binary cross-entropy: turning probability into a trainable loss\n", "\n", "A probability alone isn't yet something to optimize. Training needs a single number — called **error**, or **loss** — that's small when the model's probabilities are good and large when they're bad, averaged over the whole dataset. **Binary cross-entropy** builds that number by penalizing each prediction according to how confident it was, and whether that confidence was justified: if the true label is 1, the penalty is $-\\log(p)$ (near zero when $p$ is close to 1, exploding toward infinity as $p\\to 0$ — confidently wrong is punished severely); if the true label is 0, it's the mirror image, $-\\log(1-p)$." ] }, { "cell_type": "code", "execution_count": null, "id": "41d9c4c1", "metadata": {}, "outputs": [], "source": [ "p_range = np.linspace(0.001, 0.999, 200)\n", "loss_if_true_1 = -np.log(p_range)\n", "loss_if_true_0 = -np.log(1 - p_range)\n", "\n", "plt.plot(p_range, loss_if_true_1, label='true label = 1: -log(p)')\n", "plt.plot(p_range, loss_if_true_0, label='true label = 0: -log(1-p)')\n", "plt.xlabel('predicted probability p')\n", "plt.ylabel('per-example loss')\n", "plt.legend(fontsize=8)\n", "plt.title('Binary cross-entropy: the penalty for one prediction')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "d93170cb", "metadata": {}, "source": [ "Averaging that per-example penalty over the whole dataset gives the full loss:\n", "\n", "$$L = -\\frac{1}{n}\\sum_i \\big[y_i \\log \\sigma(z_i) + (1-y_i)\\log(1-\\sigma(z_i))\\big]$$\n", "\n", "This is exactly the two curves above, selected by $y_i$ (only one of the two terms is ever nonzero, since $y_i$ is 0 or 1) and averaged over every point. Because both the sigmoid and the logarithm are smooth, $L$ has a well-defined gradient everywhere — the loss decreases smoothly as predictions get closer to being correct, instead of only changing at the decision boundary." ] }, { "cell_type": "markdown", "id": "dc6f5345", "metadata": {}, "source": [ "## Computing the gradient: the chain rule, applied\n", "\n", "The gradient is found by applying the chain rule from calculus to this composed function: $L$ depends on $\\sigma(z)$, which depends on $z=w^\\top x + b$, which depends on $w$ and $b$. Working backward through that chain gives a remarkably clean result (skipping the algebra, which is straightforward but tedious):\n", "\n", "$$\\frac{\\partial L}{\\partial z_i} = \\sigma(z_i) - y_i, \\qquad \\frac{\\partial L}{\\partial w} = \\frac{1}{n}X^\\top(\\sigma(z)-y), \\qquad \\frac{\\partial L}{\\partial b} = \\frac{1}{n}\\sum_i(\\sigma(z_i)-y_i)$$\n", "\n", "In words: the gradient is just the *prediction error*, averaged (weighted by $x$ for $w$'s gradient). Wildly wrong predictions push the weights hard; correct ones barely move them at all. With only one layer here, this chain has just one link; Lesson {{lesson:multilayer_perceptrons}} adds a second one, and that's where applying the chain rule *repeatedly, layer by layer* earns the name **backpropagation**." ] }, { "cell_type": "code", "execution_count": null, "id": "86fec90d", "metadata": {}, "outputs": [], "source": [ "def forward(X, w, b):\n", " z = X @ w + b\n", " return sigmoid(z), z\n", "\n", "def bce_loss(p, y, eps=1e-9):\n", " return -np.mean(y * np.log(p + eps) + (1 - y) * np.log(1 - p + eps))\n", "\n", "def backward(X, p, y):\n", " grad_z = (p - y) / len(y)\n", " grad_w = X.T @ grad_z\n", " grad_b = grad_z.sum()\n", " return grad_w, grad_b" ] }, { "cell_type": "markdown", "id": "e7328979", "metadata": {}, "source": [ "### Sanity check: gradient checking\n", "\n", "Before trusting any hand-derived gradient, it's standard practice to check it numerically: perturb each parameter by a tiny amount and see how much the loss actually changes, then compare to the analytical formula." ] }, { "cell_type": "code", "execution_count": null, "id": "d1d1d4f4", "metadata": {}, "outputs": [], "source": [ "w_test, b_test = np.array([0.5, -0.3]), 0.2\n", "p_test, _ = forward(X, w_test, b_test)\n", "grad_w_analytic, grad_b_analytic = backward(X, p_test, y)\n", "\n", "def loss_at(w, b):\n", " p, _ = forward(X, w, b)\n", " return bce_loss(p, y)\n", "\n", "eps = 1e-5\n", "grad_w_numeric = np.zeros(2)\n", "for i in range(2):\n", " w_plus, w_minus = w_test.copy(), w_test.copy()\n", " w_plus[i] += eps\n", " w_minus[i] -= eps\n", " grad_w_numeric[i] = (loss_at(w_plus, b_test) - loss_at(w_minus, b_test)) / (2 * eps)\n", "grad_b_numeric = (loss_at(w_test, b_test + eps) - loss_at(w_test, b_test - eps)) / (2 * eps)\n", "\n", "print(f'analytic grad_w: {grad_w_analytic}, numeric: {grad_w_numeric}')\n", "print(f'analytic grad_b: {grad_b_analytic:.6f}, numeric: {grad_b_numeric:.6f}')\n", "print(f'max discrepancy: {max(np.abs(grad_w_analytic - grad_w_numeric).max(), abs(grad_b_analytic - grad_b_numeric)):.2e}')" ] }, { "cell_type": "markdown", "id": "cf7afe60", "metadata": {}, "source": [ "### Sanity check: PyTorch autograd\n", "\n", "As a second, independent check, we let PyTorch compute the same gradient automatically via `.backward()`, using its built-in `binary_cross_entropy_with_logits` (which combines the sigmoid and the loss in one numerically stable function)." ] }, { "cell_type": "code", "execution_count": null, "id": "7b6a38e9", "metadata": {}, "outputs": [], "source": [ "X_t = torch.tensor(X, dtype=torch.float64)\n", "y_t = torch.tensor(y, dtype=torch.float64)\n", "w_t = torch.tensor(w_test, dtype=torch.float64, requires_grad=True)\n", "b_t = torch.tensor(b_test, dtype=torch.float64, requires_grad=True)\n", "\n", "z_t = X_t @ w_t + b_t\n", "loss_t = torch.nn.functional.binary_cross_entropy_with_logits(z_t, y_t)\n", "loss_t.backward()\n", "\n", "print(f'our analytic grad_w: {grad_w_analytic}')\n", "print(f'torch autograd grad_w: {w_t.grad.numpy()}')\n", "print(f'max discrepancy: {np.abs(grad_w_analytic - w_t.grad.numpy()).max():.2e}')" ] }, { "cell_type": "markdown", "id": "90ee69aa", "metadata": {}, "source": [ "Two independent checks — numerical finite differences and PyTorch's automatic differentiation — both agree with the hand-derived formula to many decimal places. This kind of double-checking is standard practice whenever you derive a gradient by hand." ] }, { "cell_type": "markdown", "id": "036dcdaa", "metadata": {}, "source": [ "## Training: the full loop\n", "\n", "The same idea extends directly to many parameters at once: instead of a single slope $f'(x)$, the **gradient** is a vector of partial derivatives, one per parameter — exactly $\\partial L/\\partial w$ and $\\partial L/\\partial b$ derived above. The update rule is identical in spirit: $w \\leftarrow w - \\eta\\,\\partial L/\\partial w$, $b \\leftarrow b - \\eta\\,\\partial L/\\partial b$. Initialize $w, b$ randomly (not with a clever heuristic this time), then repeat: forward pass, compute loss, backward pass, take a small step against the gradient.\n", "\n", "Every step below uses the gradient computed over the *entire* training set at once (`n=120` examples) — this is sometimes called full-batch gradient descent, to distinguish it from the mini-batch **stochastic gradient descent (SGD)** real training actually uses, which Lesson {{lesson:optimization}} covers in depth." ] }, { "cell_type": "markdown", "id": "tr41n1ngl00ph4ndw0rk3d", "metadata": {}, "source": [ "### Watching the loop run, one stage at a time\n", "\n", "The animation below plays through exactly this recipe — forward pass, loss, backward pass, update — one labeled stage at a time, on a single tiny example: $x=(2, 1)$, $y=1$, starting from $w=(-0.5, 0.3)$, $b=0$, with $\\eta=0.5$. Unlike the 120-point dataset trained below, these numbers are simple enough to work out by hand.\n", "\n", "\"Animated\n", "\n", "**Working the first step out by hand** is a good way to confirm you've understood the formulas above before trusting them on a full dataset:\n", "\n", "$$z = w^\\top x + b = (-0.5)(2) + (0.3)(1) + 0 = -0.7$$\n", "$$p = \\sigma(z) = \\sigma(-0.7) \\approx 0.332$$\n", "$$L = -\\log(p) \\approx 1.103 \\quad\\text{(since } y=1\\text{, only the } -\\log(p) \\text{ term applies)}$$\n", "$$\\frac{\\partial L}{\\partial z} = p - y \\approx 0.332 - 1 = -0.668$$\n", "$$\\frac{\\partial L}{\\partial w_1} = x_1 \\cdot \\frac{\\partial L}{\\partial z} \\approx -1.336, \\qquad \\frac{\\partial L}{\\partial w_2} = x_2 \\cdot \\frac{\\partial L}{\\partial z} \\approx -0.668, \\qquad \\frac{\\partial L}{\\partial b} \\approx -0.668$$\n", "$$w_1 \\leftarrow -0.5 - 0.5(-1.336) \\approx 0.168, \\qquad w_2 \\leftarrow 0.3 - 0.5(-0.668) \\approx 0.634, \\qquad b \\leftarrow 0 - 0.5(-0.668) \\approx 0.334$$\n", "\n", "That's exactly what the animation shows after its first update — the model started out confidently wrong ($p=0.33$ when $y=1$), and one step already flips the prediction to correct ($p=0.79$) and cuts the loss by about three quarters (from $1.10$ to $0.24$). These latter numbers are left as an exercise for the reader (plug the updated $w, b$ back into the same formulas above).\n\n#### Back to the real dataset\n\nWith the mechanics confirmed by hand, the rest of this section goes back to training on the full 120-point two-blob dataset from Lesson {{lesson:projection}} — same formulas, just applied to every example at once instead of one.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "5353354e", "metadata": {}, "outputs": [], "source": [ "def train(X, y, n_epochs=500, lr=0.5, seed=0):\n", " rng_local = np.random.default_rng(seed)\n", " w = rng_local.normal(size=X.shape[1]) * 0.1\n", " b = 0.0\n", " losses = []\n", " for _ in range(n_epochs):\n", " p, _ = forward(X, w, b)\n", " losses.append(bce_loss(p, y))\n", " grad_w, grad_b = backward(X, p, y)\n", " w -= lr * grad_w\n", " b -= lr * grad_b\n", " return w, b, losses\n", "\n", "w_learned, b_learned, losses = train(X, y)\n", "final_pred = (sigmoid(X @ w_learned + b_learned) > 0.5).astype(float)\n", "print(f'learned w = {np.round(w_learned, 3)}, b = {b_learned:.3f}')\n", "print(f'final accuracy: {(final_pred == y).mean():.1%}')\n", "\n", "plt.plot(losses)\n", "plt.xlabel('epoch')\n", "plt.ylabel('loss')\n", "plt.title('Training loss')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "b3f7187e", "metadata": {}, "source": [ "### Compare to Lesson {{lesson:projection}}'s hand-picked direction" ] }, { "cell_type": "code", "execution_count": null, "id": "cf9c755a", "metadata": {}, "outputs": [], "source": [ "w_handpicked = class_b.mean(axis=0) - class_a.mean(axis=0)\n", "w_handpicked /= np.linalg.norm(w_handpicked)\n", "w_learned_normalized = w_learned / np.linalg.norm(w_learned)\n", "\n", "cos_angle = w_handpicked @ w_learned_normalized\n", "print(f'hand-picked direction (Lesson {{lesson:projection}}): {np.round(w_handpicked, 3)}')\n", "print(f'learned direction (this lesson): {np.round(w_learned_normalized, 3)}')\n", "print(f'angle between them: {np.degrees(np.arccos(np.clip(cos_angle, -1, 1))):.1f} degrees')" ] }, { "cell_type": "markdown", "id": "f6dc937b", "metadata": {}, "source": [ "Gradient descent, starting from nothing but random noise, rediscovers essentially the same direction a human picked by reasoning about class means — reassuring, but also a preview of the limits of a single neuron: it can only ever *rediscover* what a single linear projection is capable of." ] }, { "cell_type": "markdown", "id": "e456fe81", "metadata": {}, "source": [ "## Training on the unsolvable problem\n", "\n", "Lesson {{lesson:projection}} showed that *no* linear projection separates a ring from the disk it surrounds — the best exhaustive search over directions and thresholds found was 74% accuracy. What happens when gradient descent, rather than brute-force search, tries to solve the same problem?" ] }, { "cell_type": "code", "execution_count": null, "id": "9c0de831", "metadata": {}, "outputs": [], "source": [ "theta_in = rng.uniform(0, 2 * np.pi, 60)\n", "inner = np.stack([0.5 * np.cos(theta_in), 0.5 * np.sin(theta_in)], axis=1) + rng.normal(0, 0.1, (60, 2))\n", "theta_out = rng.uniform(0, 2 * np.pi, 60)\n", "outer = np.stack([2.0 * np.cos(theta_out), 2.0 * np.sin(theta_out)], axis=1) + rng.normal(0, 0.15, (60, 2))\n", "X_ring = np.vstack([inner, outer])\n", "y_ring = np.concatenate([np.zeros(60), np.ones(60)])\n", "\n", "w_ring, b_ring, losses_ring = train(X_ring, y_ring, n_epochs=2000)\n", "pred_ring = (sigmoid(X_ring @ w_ring + b_ring) > 0.5).astype(float)\n", "print(f'learned w = {np.round(w_ring, 4)} (norm={np.linalg.norm(w_ring):.4f}), b = {b_ring:.4f}')\n", "print(f'trained single-neuron accuracy: {(pred_ring == y_ring).mean():.1%}')\n", "print(f'(Lesson {{lesson:projection}}\\'s exhaustive-search ceiling for ANY linear separator was 74%)')" ] }, { "cell_type": "markdown", "id": "457257c2", "metadata": {}, "source": [ "This animation shows training on this dataset: watch the loss drop sharply in the first handful of epochs and then flatten out almost immediately, barely moving for the remaining epochs, while the decision boundary settles into a near-arbitrary location.\n", "\n", "\"Animated" ] }, { "cell_type": "markdown", "id": "4d99055a", "metadata": {}, "source": [ "Gradient descent actually does *worse* here than brute-force search — it converges to a near-zero weight vector and roughly chance-level accuracy, instead of finding the small, lopsided arc that let a brute-force search eke out 74%. The dataset is (approximately) symmetric around the origin, so the *average* gradient pull from all the training points nearly cancels out in every direction, and the optimizer settles near $w\\approx 0$ rather than hunting for an asymmetric corner-case solution. Different failure mode, same underlying truth: a single linear projection cannot solve this problem, no matter how it's found. Lesson {{lesson:multilayer_perceptrons}} fixes this — not by using a smarter optimizer, but by giving the model more than one projection to work with." ] }, { "cell_type": "markdown", "id": "02a906bd", "metadata": {}, "source": [ "### Exercises\n", "\n", "1. Increase `lr` in `train` well past a reasonable value (e.g. `lr=20`) on the two-blob dataset. What happens to the loss curve, and how does this relate to the step size overshooting the loss surface's curvature?\n", "2. Retrain on the two-blob dataset with several different random seeds. Does the learned direction always end up close to the hand-picked one from Lesson {{lesson:projection}}, or does it vary a lot? What does that suggest about how many good solutions exist for a well-separated dataset?\n", "3. Make the outer ring asymmetric instead of a full circle (e.g. `theta_out = rng.uniform(0, np.pi, 60)`, so the outer class only occupies half the ring). Retrain the single neuron. Does this genuinely broken symmetry change the final accuracy noticeably compared to the ~50% chance-level result on the full ring, and does that support the explanation above (that the *full* ring's symmetric pull on the gradient, not some fundamental inability to learn, is what stalled training near $w \\approx 0$)? (Note: simply translating both classes together by the same offset would *not* break this symmetry, since the bias term $b$ can absorb any shared translation for free.)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.x" } }, "nbformat": 4, "nbformat_minor": 5 }