{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "D_a2USyd4giE" }, "source": [ "# **Homework 3 - Convolutional Neural Network**\n", "\n", "This is the example code of homework 3 of the machine learning course by Prof. Hung-yi Lee.\n", "\n", "In this homework, you are required to build a convolutional neural network for image classification, possibly with some advanced training tips.\n", "\n", "\n", "There are three levels here:\n", "\n", "**Easy**: Build a simple convolutional neural network as the baseline. (2 pts)\n", "\n", "**Medium**: Design a better architecture or adopt different data augmentations to improve the performance. (2 pts)\n", "\n", "**Hard**: Utilize provided unlabeled data to obtain better results. (2 pts)" ] }, { "cell_type": "markdown", "metadata": { "id": "VHpJocsDr6iA" }, "source": [ "## **About the Dataset**\n", "\n", "The dataset used here is food-11, a collection of food images in 11 classes.\n", "\n", "For the requirement in the homework, TAs slightly modified the data.\n", "Please DO NOT access the original fully-labeled training data or testing labels.\n", "\n", "Also, the modified dataset is for this course only, and any further distribution or commercial use is forbidden." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "zhzdomRTOKoJ", "outputId": "dadbc075-15af-410d-9051-7a9634a374f2" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.\n", " category=FutureWarning,\n", "Downloading...\n", "From: https://drive.google.com/uc?id=1awF7pZ9Dz7X1jn1_QAiKN-_v56veCEKy\n", "To: /content/food-11.zip\n", "100% 963M/963M [00:03<00:00, 278MB/s]\n", "replace food-11/training/unlabeled/00/5176.jpg? [y]es, [n]o, [A]ll, [N]one, [r]ename: All\n" ] } ], "source": [ "# Download the dataset\n", "# You may choose where to download the data.\n", "\n", "# Google Drive\n", "!gdown --id '1awF7pZ9Dz7X1jn1_QAiKN-_v56veCEKy' --output food-11.zip\n", "\n", "# Dropbox\n", "# !wget https://www.dropbox.com/s/m9q6273jl3djall/food-11.zip -O food-11.zip\n", "\n", "# MEGA\n", "# !sudo apt install megatools\n", "# !megadl \"https://mega.nz/#!zt1TTIhK!ZuMbg5ZjGWzWX1I6nEUbfjMZgCmAgeqJlwDkqdIryfg\"\n", "\n", "# Unzip the dataset.\n", "# This may take some time.\n", "!unzip -q food-11.zip" ] }, { "cell_type": "markdown", "metadata": { "id": "BBVSCWWhp6uq" }, "source": [ "## **Import Packages**\n", "\n", "First, we need to import packages that will be used later.\n", "\n", "In this homework, we highly rely on **torchvision**, a library of PyTorch." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "9sVrKci4PUFW" }, "outputs": [], "source": [ "# Import necessary packages.\n", "import numpy as np\n", "import torch\n", "import torch.nn as nn\n", "import torchvision.transforms as transforms\n", "from PIL import Image\n", "# \"ConcatDataset\" and \"Subset\" are possibly useful when doing semi-supervised learning.\n", "from torch.utils.data import ConcatDataset, DataLoader, Subset\n", "from torchvision.datasets import DatasetFolder\n", "\n", "# This is for the progress bar.\n", "# from tqdm.auto import tqdm # 这样会有bug: AssertionError: can only test a child process\n", "from tqdm import tqdm" ] }, { "cell_type": "markdown", "metadata": { "id": "F0i9ZCPrOVN_" }, "source": [ "## **Dataset, Data Loader, and Transforms**\n", "\n", "Torchvision provides lots of useful utilities for image preprocessing, data wrapping as well as data augmentation.\n", "\n", "Here, since our data are stored in folders by class labels, we can directly apply **torchvision.datasets.DatasetFolder** for wrapping data without much effort.\n", "\n", "Please refer to [PyTorch official website](https://pytorch.org/vision/stable/transforms.html) for details about different transforms." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "gKd2abixQghI" }, "outputs": [], "source": [ "# It is important to do data augmentation in training.\n", "# However, not every augmentation is useful.\n", "# Please think about what kind of augmentation is helpful for food recognition.\n", "train_tfm = transforms.Compose([\n", " # Resize the image into a fixed shape (height = width = 128)\n", " # size (int): 保持长宽比，短边缩放至x\n", " # size (sequence): 绝对缩放\n", " transforms.Resize((128, 128)),\n", " # You may add some transforms here.\n", " # ToTensor() should be the last one of the transforms.\n", " transforms.ToTensor(),\n", "])\n", "\n", "# We don't need augmentations in testing and validation.\n", "# All we need here is to resize the PIL image and transform it into Tensor.\n", "test_tfm = transforms.Compose([\n", " transforms.Resize((128, 128)),\n", " transforms.ToTensor(),\n", "])\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "id": "qz6jeMnkQl0_", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "fc9aeda1-52d3-4b17-80f8-b4ef4842e852" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", " cpuset_checked))\n" ] } ], "source": [ "# Batch size for training, validation, and testing.\n", "# A greater batch size usually gives a more stable gradient.\n", "# But the GPU memory is limited, so please adjust it carefully.\n", "batch_size = 128\n", "\n", "# Construct datasets. 这个很好用嘛，厉害\n", "# The argument \"loader\" tells how torchvision reads the data.\n", "train_set = DatasetFolder(\"food-11/training/labeled\", loader=lambda x: Image.open(x), extensions=\"jpg\", transform=train_tfm)\n", "valid_set = DatasetFolder(\"food-11/validation\", loader=lambda x: Image.open(x), extensions=\"jpg\", transform=test_tfm)\n", "unlabeled_set = DatasetFolder(\"food-11/training/unlabeled\", loader=lambda x: Image.open(x), extensions=\"jpg\", transform=train_tfm)\n", "test_set = DatasetFolder(\"food-11/testing\", loader=lambda x: Image.open(x), extensions=\"jpg\", transform=test_tfm)\n", "\n", "# Construct data loaders.\n", "train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=8, pin_memory=True)\n", "valid_loader = DataLoader(valid_set, batch_size=batch_size, shuffle=True, num_workers=8, pin_memory=True)\n", "test_loader = DataLoader(test_set, batch_size=batch_size, shuffle=False)" ] }, { "cell_type": "markdown", "metadata": { "id": "j9YhZo7POPYG" }, "source": [ "## **Model**\n", "\n", "The basic model here is simply a stack of convolutional layers followed by some fully-connected layers.\n", "\n", "Since there are three channels for a color image (RGB), the input channels of the network must be three.\n", "In each convolutional layer, typically the channels of inputs grow, while the height and width shrink (or remain unchanged, according to some hyperparameters like stride and padding).\n", "\n", "Before fed into fully-connected layers, the feature map must be flattened into a single one-dimensional vector (for each image).\n", "These features are then transformed by the fully-connected layers, and finally, we obtain the \"logits\" for each class.\n", "\n", "### **WARNING -- You Must Know**\n", "You are free to modify the model architecture here for further improvement.\n", "However, if you want to use some well-known architectures such as ResNet50, please make sure **NOT** to load the pre-trained weights.\n", "Using such pre-trained models is considered cheating and therefore you will be punished.\n", "Similarly, it is your responsibility to make sure no pre-trained weights are used if you use **torch.hub** to load any modules.\n", "\n", "For example, if you use ResNet-18 as your model:\n", "\n", "model = torchvision.models.resnet18(pretrained=**False**) → This is fine.\n", "\n", "model = torchvision.models.resnet18(pretrained=**True**) → This is **NOT** allowed." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "id": "Y1c-GwrMQqMl" }, "outputs": [], "source": [ "class Classifier(nn.Module):\n", " def __init__(self):\n", " super(Classifier, self).__init__()\n", " # The arguments for commonly used modules:\n", " # torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)\n", " # torch.nn.MaxPool2d(kernel_size, stride, padding)\n", "\n", " # input image size: [3, 128, 128]\n", " self.cnn_layers = nn.Sequential(\n", " # torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None)\n", " nn.Conv2d(3, 64, 3, 1, 1), # output image size: [64, 128, 128]\n", " # torch.nn.BatchNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, device=None, dtype=None)\n", " nn.BatchNorm2d(64), # 不改变image size\n", " # torch.nn.ReLU(inplace=False)\n", " nn.ReLU(), # 加一个非线性变换\n", " # torch.nn.MaxPool2d(kernel_size, stride=None, padding=0, dilation=1, return_indices=False, ceil_mode=False)\n", " nn.MaxPool2d(2, 2, 0), # output image size: [64, 64, 64]\n", "\n", " nn.Conv2d(64, 128, 3, 1, 1), # output image size: [128, 64, 64]\n", " nn.BatchNorm2d(128),\n", " nn.ReLU(),\n", " nn.MaxPool2d(2, 2, 0), # output image size: [128, 32, 32]\n", "\n", " nn.Conv2d(128, 256, 3, 1, 1), # output image size: [256, 32, 32]\n", " nn.BatchNorm2d(256),\n", " nn.ReLU(),\n", " nn.MaxPool2d(4, 4, 0), # output image size: [256, 8, 8]\n", " )\n", " self.fc_layers = nn.Sequential(\n", " nn.Linear(256 * 8 * 8, 256), # 全连接需要拉直\n", " nn.ReLU(),\n", " nn.Linear(256, 256), # ？\n", " nn.ReLU(),\n", " nn.Linear(256, 11)\n", " )\n", "\n", " def forward(self, x):\n", " # input (x): [batch_size, 3, 128, 128]\n", " # output: [batch_size, 11]\n", "\n", " # Extract features by convolutional layers.\n", " x = self.cnn_layers(x)\n", "\n", " # The extracted feature map must be flatten before going to fully-connected layers.\n", " x = x.flatten(1) # 拉直\n", "\n", " # The features are transformed by fully-connected layers to obtain the final logits. logits: 最终的全连接层的输出\n", " x = self.fc_layers(x)\n", " return x" ] }, { "cell_type": "markdown", "metadata": { "id": "cbm81gwD50fo" }, "source": [ "cnn_layers - First Layer\n", "\n", "![](https://cdn.jsdelivr.net/gh/WanpengXu/myPicGo/img/202208060048059.png)\n", "\n", "1. Conv2d：\n", "\n", "```python\n", "torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None)\n", "```\n", "\n", "卷积\n", "\n", "$$\n", "H_{out} = ⌊\\frac{128+2*1-1*(3-1)-1}{1}+1⌋=128\n", "$$\n", "\n", "$$\n", "W_{out} = ⌊\\frac{128+2*1-1*(3-1)-1}{1}+1⌋=128\n", "$$\n", "\n", "2. BatchNorm2d\n", "\n", "```python\n", "torch.nn.BatchNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, device=None, dtype=None)\n", "```\n", "\n", "batch标准化\n", "\n", "![](https://cdn.jsdelivr.net/gh/WanpengXu/myPicGo/img/202208060104296.png)\n", "\n", "前三步类似概率论中的随机变量$X(𝜇, σ) ∼ N(0, 1)$，ϵ是参数eps，添加到mini-batch中，以保证值的稳定性。\n", "\n", "3. ReLU\n", "\n", "```python\n", "torch.nn.ReLU(inplace=False)\n", "```\n", "\n", "![](https://pytorch.org/docs/stable/_images/ReLU.png)\n", "\n", "4. MaxPool2d\n", "\n", "```python\n", "torch.nn.MaxPool2d(kernel_size, stride=None, padding=0, dilation=1, return_indices=False, ceil_mode=False)\n", "```\n", "\n", "![](https://cdn.jsdelivr.net/gh/WanpengXu/myPicGo/img/202208060116297.png)\n", "\n", "$$\n", "H_{out}=⌊\\frac{128+2*0-1*(2-1)-1}{2}+1⌋=64\n", "$$\n", "\n", "$$\n", "W_{out}=⌊\\frac{128+2*0-1*(2-1)-1}{2}+1⌋=64\n", "$$\n", "\n", "forward\n", "\n", "```python\n", "x = x.flatten(1)\n", "```\n", "or\n", "```python\n", "x = x.view(x.size()[0], -1)\n", "```\n", "\n", "![](https://cdn.jsdelivr.net/gh/WanpengXu/myPicGo/img/202208060133103.png)" ] }, { "cell_type": "markdown", "metadata": { "id": "aEnGbriXORN3" }, "source": [ "## **Training**\n", "\n", "You can finish supervised learning by simply running the provided code without any modification.\n", "\n", "The function \"get_pseudo_labels\" is used for semi-supervised learning.\n", "It is expected to get better performance if you use unlabeled data for semi-supervised learning.\n", "However, you have to implement the function on your own and need to adjust several hyperparameters manually.\n", "\n", "For more details about semi-supervised learning, please refer to [Prof. Lee's slides](https://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2016/Lecture/semi%20(v3).pdf).\n", "\n", "Again, please notice that utilizing external data (or pre-trained model) for training is **prohibited**." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "id": "swlf5EwA-hxA" }, "outputs": [], "source": [ "def get_pseudo_labels(dataset, model, threshold=0.65):\n", " # This functions generates pseudo-labels of a dataset using given model.\n", " # It returns an instance of DatasetFolder containing images whose prediction confidences exceed a given threshold.\n", " # You are NOT allowed to use any models trained on external data for pseudo-labeling.\n", " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "\n", " # Construct a data loader.\n", " data_loader = DataLoader(dataset, batch_size=batch_size, shuffle=False)\n", "\n", " # Make sure the model is in eval mode.\n", " model.eval()\n", " # Define softmax function.\n", " softmax = nn.Softmax(dim=-1)\n", "\n", " # Iterate over the dataset by batches.\n", " for batch in tqdm(data_loader):\n", " img, _ = batch\n", "\n", " # Forward the data\n", " # Using torch.no_grad() accelerates the forward process.\n", " with torch.no_grad():\n", " logits = model(img.to(device))\n", "\n", " # Obtain the probability distributions by applying softmax on logits.\n", " probs = softmax(logits)\n", "\n", " # ---------- TODO ----------\n", " # Filter the data and construct a new dataset.\n", "\n", " # # Turn off the eval mode.\n", " model.train()\n", " return dataset" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "id": "PHaFE-8oQtkC", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "b1f9d2b1-cd1b-42ba-f8f7-05351d25f617" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "\r 0%| | 0/25 [00:00