{ "cells": [ { "cell_type": "markdown", "id": "39a2def5", "metadata": {}, "source": [ "# 🔢 Machine Learning Homework: Images and Attention\n", "\n", "Welcome to the deep learning worksheet. Here you meet two of the most powerful ideas in modern\n", "machine learning:\n", "\n", "1. The **Convolutional Neural Network (CNN)**, which reads pictures.\n", "2. The **Transformer**, which reads sequences such as a sentence. This is the idea behind the large chat\n", " assistants used today.\n", "\n", "You do not need any physics or any earlier machine learning to do this.\n", "\n", "### How to use this worksheet\n", "* Some cells have blank spaces marked `___` and `# TODO`. Fill them in, then run the cell.\n", "* Every task ends with a short footer:\n", " * ✅ **Check yourself:** what your result should look like.\n", " * 🔬 **Why it matters:** what this idea is used for in normal life.\n", " * 🎉 **Fun fact:** how this idea is used in particle physics.\n", " * 💡 **Go further:** an extra challenge, if you want to push yourself.\n" ] }, { "cell_type": "markdown", "id": "c4ff73a6", "metadata": {}, "source": [ "## Setup\n", "Run this one time. Everything here is already available in Google Colab." ] }, { "cell_type": "code", "execution_count": null, "id": "28d3a04f", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "import torch\n", "from torch import nn, optim\n", "\n", "from sklearn.datasets import load_digits\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import accuracy_score\n", "\n", "print(\"Setup complete\")" ] }, { "cell_type": "markdown", "id": "831908a9", "metadata": {}, "source": [ "## Part A. What is a convolution? 🔎 (about 6 min)\n", "\n", "A CNN reads a picture by sliding a small window over it and looking for a pattern. That sliding\n", "action is called a **convolution**. Let us try the simplest version in one dimension. We slide a\n", "small pattern that finds a change, or an edge, in a list of numbers.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "aa534b2b", "metadata": {}, "outputs": [], "source": [ "signal = np.array([0, 0, 0, 5, 5, 5, 0, 0, 0]) # a flat line with a block in the middle\n", "edge_finder = np.array([1, -1]) # this pattern reacts to a change\n", "\n", "# TODO: slide the edge_finder over the signal using np.convolve\n", "result = np.convolve(signal, edge_finder, mode=\"valid\")\n", "print(\"signal:\", signal)\n", "print(\"result:\", ___)" ] }, { "cell_type": "markdown", "id": "b4c9c817", "metadata": {}, "source": [ "✅ **Check yourself:** the result should be mostly zero, with a spike where the block begins\n", "and another where it ends. The pattern found the two edges. A CNN does this in two dimensions to\n", "find edges and shapes inside a picture.\n", "\n", "🔬 **Why it matters:** the same sliding pattern idea is used to sharpen photos, blur them, and find\n", "outlines. It is the basic building block that lets a computer understand images.\n", "\n", "🎉 **Fun fact:** a convolution is not only a computer trick. The same mathematics describes how a\n", "signal spreads out inside a particle detector, so physicists have used this idea for a very long\n", "time.\n", "\n", "💡 **Go further:** change `edge_finder` to `np.array([1, 1, 1]) / 3`. This one takes an average, so\n", "it smooths the signal instead of finding edges. Run it and see.\n" ] }, { "cell_type": "markdown", "id": "f4417aea", "metadata": {}, "source": [ "## Part B. A CNN that reads handwritten digits 🖼️ (about 12 min)\n", "\n", "Now the real thing. We use a set of small pictures of handwritten digits, from 0 to 9. Each picture\n", "is only 8 by 8 pixels. We build a small CNN and train it to read the digit in each picture.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "cf759cf9", "metadata": {}, "outputs": [], "source": [ "digits = load_digits()\n", "print(\"Number of pictures:\", len(digits.images), \"and each is\", digits.images[0].shape)\n", "\n", "# show the first picture\n", "plt.imshow(digits.images[0], cmap=\"gray\")\n", "plt.title(f\"This digit is a {digits.target[0]}\"); plt.axis(\"off\"); plt.show()\n", "\n", "# Prepare the data: add a colour channel and scale the pixels to the range 0 to 1\n", "X = digits.images[:, None, :, :] / 16.0\n", "y = digits.target\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n", "X_train = torch.tensor(X_train, dtype=torch.float32); y_train = torch.tensor(y_train)\n", "X_test = torch.tensor(X_test, dtype=torch.float32)\n", "print(\"Training pictures:\", X_train.shape[0], \"and test pictures:\", X_test.shape[0])" ] }, { "cell_type": "code", "execution_count": null, "id": "2d59cb8d", "metadata": {}, "outputs": [], "source": [ "torch.manual_seed(0)\n", "\n", "class SmallCNN(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " # TODO: a convolution with 1 input channel, 8 output patterns, and a 3 by 3 window\n", " self.conv = nn.Conv2d(1, ___, kernel_size=3)\n", " self.pool = nn.MaxPool2d(2) # keeps the strongest value in each 2 by 2 area\n", " self.fc = nn.Linear(8 * 3 * 3, 10) # 10 outputs, one per digit\n", "\n", " def forward(self, x):\n", " x = torch.relu(self.conv(x))\n", " x = self.pool(x)\n", " x = x.flatten(1)\n", " return self.fc(x)\n", "\n", "model = SmallCNN()\n", "print(model)" ] }, { "cell_type": "code", "execution_count": null, "id": "ed3a0aad", "metadata": {}, "outputs": [], "source": [ "loss_fn = nn.CrossEntropyLoss()\n", "optimizer = optim.Adam(model.parameters(), lr=0.01)\n", "\n", "# TODO: train for 100 rounds (epochs)\n", "losses = []\n", "for epoch in range(___):\n", " optimizer.zero_grad()\n", " loss = loss_fn(model(X_train), y_train)\n", " loss.backward()\n", " optimizer.step()\n", " losses.append(loss.item())\n", "\n", "plt.plot(losses); plt.xlabel(\"epoch\"); plt.ylabel(\"loss\"); plt.title(\"The loss should go down\"); plt.show()\n", "\n", "acc = accuracy_score(y_test, model(X_test).argmax(1).numpy())\n", "print(\"CNN test accuracy:\", round(acc, 3))" ] }, { "cell_type": "markdown", "id": "2ccaf8c4", "metadata": {}, "source": [ "✅ **Check yourself:** the loss should go down, and the test accuracy should be about **0.97**.\n", "Your CNN can now read handwritten digits it has never seen before.\n", "\n", "🔬 **Why it matters:** CNNs are the reason computers can read handwriting, sort photos, and help\n", "cars see the road. They are one of the biggest success stories in machine learning.\n", "\n", "🎉 **Fun fact:** physicists turn the signals from their detectors into pictures, then use CNNs like\n", "this one to recognise which particle made each picture. Neutrino experiments in particular use CNNs\n", "to read their detector images and tell different particle tracks apart.\n", "\n", "💡 **Go further:** change the number of output patterns in the convolution from 8 to 16 (remember to\n", "change the number 8 in the `Linear` layer to 16 as well). Does the accuracy improve?\n" ] }, { "cell_type": "markdown", "id": "b486d452", "metadata": {}, "source": [ "## Part C. How a computer reads a sentence 🔤 (about 6 min)\n", "\n", "A transformer does not read a sentence as one long string. First it cuts the sentence into small\n", "pieces called **tokens**. The simplest way is to make everything lower case and split on the\n", "spaces.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ac3aa245", "metadata": {}, "outputs": [], "source": [ "def tokenizer(sentence):\n", " # TODO: make the sentence lower case, then split it into a list of words\n", " return sentence.___().split()\n", "\n", "sentence = \"The cat sat on the mat\"\n", "tokens = tokenizer(sentence)\n", "print(\"Tokens:\", tokens)\n", "print(\"Number of tokens:\", len(tokens))" ] }, { "cell_type": "markdown", "id": "122aa779", "metadata": {}, "source": [ "✅ **Check yourself:** you should get a list of 6 tokens:\n", "the, cat, sat, on, the, mat. The word the appears twice, and both times it becomes the same token.\n", "\n", "🔬 **Why it matters:** breaking text into tokens is the first step for every language model,\n", "including the chat assistants that many people use today. The computer works with these tokens\n", "instead of raw letters.\n", "\n", "🎉 **Fun fact:** physicists can read a collision in the same way. Instead of words, the tokens are\n", "the particles that come out, and the model reads them as a sequence, just like a sentence.\n", "\n", "💡 **Go further:** real tokenizers also split off punctuation, so that a full stop becomes its own\n", "token. How would you change the function to separate a full stop from the word before it?\n" ] }, { "cell_type": "markdown", "id": "74b92e0d", "metadata": {}, "source": [ "## Part D. Why word order matters 🔀 (about 5 min)\n", "\n", "Two sentences can have the exact same words but a completely different meaning. The order carries\n", "the meaning. This is why a transformer must pay attention to the position of each token, not only\n", "to the words themselves.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "724c8a18", "metadata": {}, "outputs": [], "source": [ "a = tokenizer(\"dog bites man\")\n", "b = tokenizer(\"man bites dog\")\n", "\n", "print(\"Sentence A tokens:\", a)\n", "print(\"Sentence B tokens:\", b)\n", "\n", "# TODO: check whether the two sentences use the same set of words (ignore the order)\n", "same_words = sorted(a) == sorted(___)\n", "print(\"Same words?\", same_words)\n", "print(\"Same order?\", a == b)" ] }, { "cell_type": "markdown", "id": "beb3e5b8", "metadata": {}, "source": [ "✅ **Check yourself:** the two sentences use the **same words** (True) but **not the same\n", "order** (False). To a human the meaning is very different, so the model must use the position of\n", "each word.\n", "\n", "🔬 **Why it matters:** understanding order is why modern language models can follow grammar and\n", "meaning, instead of just seeing a bag of words. A transformer adds position information to every\n", "token for exactly this reason.\n", "\n", "🎉 **Fun fact:** order and structure matter in physics too. The pattern and position of the\n", "particles that come out of a collision tell physicists what kind of event they are looking at.\n", "\n", "💡 **Go further:** think of two more sentences that use the same words but mean different things.\n" ] }, { "cell_type": "markdown", "id": "785b9bcb", "metadata": {}, "source": [ "## Part E. Self attention, the heart of the transformer 💛 (about 10 min)\n", "\n", "Here is the central idea of the transformer. For every token, the model asks how much it should\n", "**pay attention** to each of the other tokens. It does this with three sets of numbers called\n", "Query (Q), Key (K) and Value (V). We compare Q with K to get attention weights, and then use those\n", "weights to mix the values V. This is called **scaled dot product attention**.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6ef75dd3", "metadata": {}, "outputs": [], "source": [ "def softmax(z):\n", " e = np.exp(z - z.max(axis=-1, keepdims=True))\n", " return e / e.sum(axis=-1, keepdims=True)\n", "\n", "np.random.seed(0)\n", "# Three tokens, each described by 4 numbers.\n", "Q = np.random.randn(3, 4)\n", "K = np.random.randn(3, 4)\n", "V = np.random.randn(3, 4)\n", "d_k = Q.shape[-1] # the size of each token vector, here 4\n", "\n", "# TODO: divide the scores by the square root of d_k (this is the \"scaled\" part)\n", "scores = Q @ K.T / np.sqrt(___)\n", "weights = softmax(scores) # turn scores into attention weights that add up to 1\n", "output = weights @ V # mix the values using the weights\n", "\n", "print(\"Attention weights (each row adds up to 1):\")\n", "print(weights.round(2))\n", "print(\"\\nEach row sums to:\", weights.sum(axis=1).round(3))" ] }, { "cell_type": "markdown", "id": "a598448e", "metadata": {}, "source": [ "✅ **Check yourself:** you should see a 3 by 3 grid of attention weights, and **every row\n", "should add up to 1.0**. Each number says how much one token pays attention to another. You just\n", "built the core of a transformer.\n", "\n", "🔬 **Why it matters:** this attention step is the key idea behind almost every modern language\n", "model. It lets each word look at every other word and decide what is important. The same idea also\n", "works for images, sound, and much more.\n", "\n", "🎉 **Fun fact:** attention is now one of the newest tools in particle physics. Modern taggers read\n", "a collision as a sequence of particles, much like reading a sentence, and use attention to decide\n", "which particles matter most.\n", "\n", "💡 **Go further:** the square root step keeps the numbers from growing too large when `d_k` is big.\n", "Try removing it (divide by 1 instead) and print the weights. They become much more extreme, closer\n", "to only 0 and 1.\n" ] }, { "cell_type": "markdown", "id": "2dceebf5", "metadata": {}, "source": [ "## 🎓 Very well done\n", "\n", "You have now met the two engines of modern deep learning. You built a **CNN** that reads pictures\n", "of digits, and you built the **attention** step that sits at the heart of every transformer. These\n", "same two ideas, only much larger, power photo apps, self driving cars, and the language assistants\n", "that so many people use today.\n", "\n", "The best way to learn more is to change things and see what happens. Every small experiment turns a\n", "fact that you memorised into an understanding that you own.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }