{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "$$\n", "\\newcommand{\\mat}[1]{\\boldsymbol {#1}}\n", "\\newcommand{\\mattr}[1]{\\boldsymbol {#1}^\\top}\n", "\\newcommand{\\matinv}[1]{\\boldsymbol {#1}^{-1}}\n", "\\newcommand{\\vec}[1]{\\boldsymbol {#1}}\n", "\\newcommand{\\vectr}[1]{\\boldsymbol {#1}^\\top}\n", "\\newcommand{\\rvar}[1]{\\mathrm {#1}}\n", "\\newcommand{\\rvec}[1]{\\boldsymbol{\\mathrm{#1}}}\n", "\\newcommand{\\diag}{\\mathop{\\mathrm {diag}}}\n", "\\newcommand{\\set}[1]{\\mathbb {#1}}\n", "\\newcommand{\\cset}[1]{\\mathcal{#1}}\n", "\\newcommand{\\norm}[1]{\\left\\lVert#1\\right\\rVert}\n", "\\newcommand{\\pderiv}[2]{\\frac{\\partial #1}{\\partial #2}}\n", "\\newcommand{\\bb}[1]{\\boldsymbol{#1}}\n", "$$\n", "\n", "# CS236781: Deep Learning\n", "# Tutorial 7: Transfer Learning and Domain Adaptation" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Introduction\n", "\n", "In this tutorial, we will cover:\n", "\n", "- Transfer learning contexts\n", "- Leveraging pre-trained models for supervised domain adaptation\n", "- Unsupervised domain adaptation using adversarial training" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# Setup\n", "%matplotlib inline\n", "import os\n", "import sys\n", "import torch\n", "import numpy as np\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "plt.rcParams['font.size'] = 20\n", "data_dir = './' #os.path.expanduser('~/.pytorch-datasets')\n", "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Transfer learning" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### The supervised learning context" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "We have a labeled dataset of $N$ labelled samples: $\\left\\{ (\\vec{x}^i,y^i) \\right\\}_{i=1}^N$, where\n", "- $\\vec{x}^i = \\left(x^i_1, \\dots, x^i_D\\right) \\in \\mathcal{X}$ is a **sample** or **feature vector**.\n", "- $y^i \\in \\mathcal{Y}$ is the **label**.\n", "- For classification with $C$ classes, $\\mathcal{Y} = \\{0,\\dots,C-1\\}$, so each $y^i$ is a **class label**.\n", "- Usually we assume each labeled sample $(\\vec{x}^i,y^i)$\n", " is drawn from a joint distribution\n", " $$P_{(\\rvec{X}, \\rvar{Y})}=P_{\\rvec{X}}\\cdot P_{\\rvar{Y}|\\rvec{X}}$$\n", " - We assume some marginal sample distribution $P_{\\rvar{X}}$ exists.\n", " - We want to estimate properties of $P_{\\rvar{Y}|\\rvec{X}}$ from the data." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "So far, we considered mostly the traditional **supervised learning** setting:\n", "\n", "We assumed the **train** and **test** (which is supposed to represent future unseen data)\n", "sets are both sampled from the same **distribution** $P_{(\\rvec{X}, \\rvar{Y})}$ and both labeled.\n", "\n", "We assume this since we wanted to solve one task with one dataset, and we could\n", "therefore split our dataset into such sets.\n", "\n", "What happens when this is not the case?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "In the real world, we often don't have the perfect training set for our problem.\n", "I.e. we may not be able to sample i.i.d. from the underlying distribution.\n", "\n", "What should we do when the supervised learning assumption is invalid?\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Domains, targets and tasks" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Lets start with some definitions to explain the problem.\n", "\n", "- Imagine we have a **feature space**, $\\mathcal{X}$\n", " - For example, $\\mathcal{X}$ is the space of color images of size 32x32, each pixel in the range 0-255" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10^7398\n" ] } ], "source": [ "import math\n", "# size of this \"limited\" feature space\n", "print(f'10^{math.log10(256**(32**2*3)):.0f}')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "- As usual, we have a training set $X=\\{\\vec{x}^{(i)}\\}_{i=1}^{N},\\ \\vec{x}^{(i)}\\in\\cset{X}$.\n", " - For example, CIFAR-10\n", " - This is a small subset of the space of possible images, and even of the space of natural images\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "- There exists some **probability distribution** $P_{X}(\\vec{x})$ over our data.\n", " - Note that we don't know the distribution over $\\cset{X}$.\n", " - For example, if $X$ is CIFAR-10, the probability of an all-black image should be very low\n", " - There may be parts of the feature space not included in our data $X$, but that we may encounter during inference. For example, a picture of a different type of bird, a car at night, etc. So the data distribution during inference could be different." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "- Our **label space**, $\\cset{Y}$ includes the possible labels for sample in our problem.\n", " - For example $\\cset{Y}=\\{0,1\\}$ in binary classification.\n", "- We may have also $Y = \\{y^{(i)}\\}_{i=1}^{N},\\ y^{(i)}\\in\\cset{Y}$, the set of labels for our dataset." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "- We want to learn e.g. a target function $\\hat{y}=f(\\vec{x})$ which predicts a label given an image.\n", " - From the probabilistic perspective, estimate $P(Y=\\hat{y}|X=\\vec{x})$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Finally,\n", "- A learning **domain** $\\cset{D}$, is defined as $\\cset{D}=\\left\\{\\mathcal{X},P_{\\rvec{X}}\\right\\}$.\n", "- A learning **task** $\\cset{T}$ is defined as $\\cset{T}=\\{\\cset{Y},P_{\\rvec{Y}|\\rvec{X}}\\}$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Transfer learning settings" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "**Definition** (Pan & Yang, 2010):\n", "\n", "Given\n", "- A **source** domain $\\cset{D}_S=\\left\\{\\mathcal{X}_S,P(X_S)\\right\\}$ and source learning task\n", " $\\cset{T}_S = \\{\\cset{Y}_S,P(Y_S|X_S)\\}$\n", " \n", "- A **target** domain $\\cset{D}_T=\\left\\{\\mathcal{X}_T,P(X_T)\\right\\}$ and target learning task\n", " $\\cset{T}_T = \\{\\cset{Y}_T,P(Y_T|X_T)\\}$\n", " \n", "(Note slight abuse of notation: we use e.g. $P(X)$ to denote the PDF $P_{\\rvec{X}}(\\vec{x})$.)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "*Transfer learning* deals with estimating the target function $P(Y_T|X_T)$\n", "using *knowledge* of $\\cset{D}_S$ and $\\cset{T}_S$, when\n", "- $\\cset{D}_S \\neq \\cset{D}_T$, and/or\n", "- $\\cset{T}_S \\neq \\cset{T}_T$\n", "\n", "Sometimes also there are other constraints on the target domain, such as little or no labels available." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "When $\\cset{D}_S=\\cset{D}_T$ and $\\cset{T}_S=\\cset{T}_T$ we're in the regular supervised learning setting\n", "we have seen thus far.\n", "For example, splitting CIFAR-10 randomly into a train and test set." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Same domain, different task" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Recall, a learning **task** $\\cset{T}$ is defined as $\\cset{T}=\\{\\cset{Y},P(Y|X)\\}$.\n", "\n", "So there are two cases (not mutually exclusive)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Case 1: The label spaces are different, $\\cset{Y}_S \\neq \\cset{Y}_T$\n", "\n", "For example, target domain has more classes.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Case 2: The target conditional distributions are different, $P(Y_S|X_S)\\neq P(Y_T|X_S)$.\n", "\n", "For example:\n", "$Y_T$ is the true variable which we see at test-time, while $Y_S$ may be what we have in our data, and is different due to sampling bias." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "This may be the case when the class-balance is very different in the source and target distributions, i.e.\n", "we have a different prior $P(Y)$ for the labels between source and target.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Same task, different domain" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Recall, a learning **domain** $\\cset{D}$, is defined as $\\cset{D}=\\left\\{\\mathcal{X},P(X)\\right\\}$.\n", "\n", "Again, two cases." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Case 1: Different feature spaces, $\\cset{X}_S \\neq \\cset{X}_T$.\n", "\n", "For example:\n", "- $\\cset{X}_S$ is a space of grayscale images while $\\cset{X}_T$ is a space of color images\n", "- Documents in different languages." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Case 2: Different feature data distributions, $P(X_S)\\neq P(X_T)$ (sometimes called covariate shift)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "For example:\n", "- Source domain contains hand-drawn images, while target domain contains photographs;\n", "- Documents in the same language about different topics.\n", "- Speech recognition of different speakers." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "\n", "This is a very common scenario, and usually called **domain adaptation**." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Transfer learning is a huge research field.\n", "\n", "
\n", "\n", "In this tutorial we'll see two simple yet common examples." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Part 1: Fine-tuning a pre-trained model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We have trained a model for a source task,\n", "and now we want to use it to speed up training for a different target task.\n", "\n", "In some applications, we may have have much less labeled data in the target domain, making it infeasible to train a deep model from scratch." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Common example: pre-train on ImageNet (1M+ images, 1000 classes), and then classify e.g. medical images.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Why would this work?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "CNNs capture hierarchical features, with deeper layers capturing higher-level, class-specific features.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "What are we looking at? Images generated by optimization to maximally activate various layers (aka. DeepDream objective) of a GoogLeNet trained on the ImageNet data." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "General idea: we can start from a pre-trained model and,\n", "- Keep the parameters in the base layer as-is.\n", "- \"Fine-tune\" the convolutional filters, mainly in the deeper layers.\n", "- Change the classifier head (or completely remove it) to fit our task and train it from scratch." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Can you think of an opposite example? I.e., where we'd want to train the first layers but keep the last layers relatively fixed?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "For our image classification fine-tuning example,\n", "- We'll load a deep CNN pre-trained on ImageNet (1000 classes, 1M+ 224x224 images)\n", "- Using ResNet18 just to reduce download size, you can use something deeper" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Downloading: \"https://download.pytorch.org/models/resnet18-5c106cde.pth\" to C:\\Users\\moshe/.cache\\torch\\hub\\checkpoints\\resnet18-5c106cde.pth\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a9c0ab3afd7647f1873b47b56682aa0a", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0.00/44.7M [00:00) torch.Size([1, 13])\n" ] } ], "source": [ "y0 = resnet18(ds_train[0][0].unsqueeze(dim=0))\n", "print(y0, y0.shape)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Set up optimization to account for the fine-tuning:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "import torch.optim as optim\n", "\n", "# Important nunance 2: Only parameters that track gradients can be passed into the optimizer\n", "params_non_frozen = filter(lambda p: p.requires_grad, resnet18.parameters())\n", "opt = optim.SGD(params_non_frozen, lr=0.05, momentum=0.9)\n", "\n", "# Finetuning usually means we want smaller than usual learning rates and \n", "# decaying them in order to keep improving the weights\n", "lr_sched = optim.lr_scheduler.ReduceLROnPlateau(opt, factor=0.05, patience=5,)\n", "\n", "loss_fn = nn.CrossEntropyLoss()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "And finally, train as usual." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "def train(model, loss_fn, opt, lr_sched, dl_train, dl_test):\n", " # Same as regular classifier traning, just call lr_sched.step(val_loss)\n", " # every epoch to reduce lr if validation loss plateaus (example).\n", " # ...\n", " # ====== YOUR CODE: ======\n", " # :)\n", " # ========================\n", " pass" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Part 2: Unsupervised domain adaptation" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Let's consider a problem with different domains but an identical task:\n", "\n", "- Source domain: MNIST\n", "- Target domain: MNIST-M, a colored and textured version of MNIST\n", "\n", "Task in both cases is the usual 10-class digit classification.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "**Unsupervised** DA setting:\n", "We assume that there are **no available labels** for the target domain.\n", "\n", "Why would a CNN trained on MNIST not generalize to MNIST-M?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Intuition: We need a way to force our CNN to learn features of the digit outline shapes only, and ignore color distributions." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Our approach (based on Ganin et al. 2015): \"Domain-adversarial\" training" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "- Train a classifier for the **label** of images from the **source** domain as a regular CNN feature extractor (green) and FC classifier (blue).\n", "- Train a classifier for the **domain** of an image based on the deep convolutonal features (pink).\n", "- Try to **maximize** the loss of this domain classifier when training the convolutional layers (**confusion loss**).\n", "- Simultaneously, minimize the classification loss on the source domain using the same convolutional features.\n", "- Train the digit classifier with source domain data, and the domain classifier with both domains' data." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Source and target domain data" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "**Note**: for the next block to run, you should manually [download](https://drive.google.com/open?id=0B_tExHiYS-0veklUZHFYT19KYjg) the MNIST-M dataset and unpack it into `data_dir`." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./MNIST\\raw\\train-images-idx3-ubyte.gz\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "455d91c9f828452fbce5aa631503694e", "version_major": 2, "version_minor": 0 }, "text/plain": [ "0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Extracting ./MNIST\\raw\\train-images-idx3-ubyte.gz to ./MNIST\\raw\n", "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./MNIST\\raw\\train-labels-idx1-ubyte.gz\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c0f4b7ae0e4b4f60a7b326e5b7f75ed1", "version_major": 2, "version_minor": 0 }, "text/plain": [ "0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Extracting ./MNIST\\raw\\train-labels-idx1-ubyte.gz to ./MNIST\\raw\n", "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./MNIST\\raw\\t10k-images-idx3-ubyte.gz\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2ab809e296f94439917f380e7fa09fcd", "version_major": 2, "version_minor": 0 }, "text/plain": [ "0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Extracting ./MNIST\\raw\\t10k-images-idx3-ubyte.gz to ./MNIST\\raw\n", "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./MNIST\\raw\\t10k-labels-idx1-ubyte.gz\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "7cdfec88eba743d89831d3babef43296", "version_major": 2, "version_minor": 0 }, "text/plain": [ "0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Extracting ./MNIST\\raw\\t10k-labels-idx1-ubyte.gz to ./MNIST\\raw\n", "Processing...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\moshe\\anaconda3\\envs\\cs3600-tut\\lib\\site-packages\\torchvision\\datasets\\mnist.py:469: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ..\\torch\\csrc\\utils\\tensor_numpy.cpp:141.)\n", " return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Done!\n" ] } ], "source": [ "import torchvision as tv\n", "import torchvision.transforms as tvtf\n", "from tut7.data import MNISTMDataset\n", "from tut7.plot_utils import dataset_first_n\n", "\n", "image_size = 28\n", "batch_size = 4\n", "tf_source = tvtf.Compose([ tvtf.Resize(image_size), tvtf.ToTensor(), tvtf.Normalize(mean=(0.1307,), std=(0.3081,)) ])\n", "tf_target = tvtf.Compose([\n", " tvtf.Resize(image_size), tvtf.ToTensor(), tvtf.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))\n", "])\n", "\n", "ds_source = tv.datasets.MNIST(root=data_dir, train=True, transform=tf_source, download=True)\n", "\n", "# Custom PyTorch Dataset class to load MNIST-M\n", "ds_target = MNISTMDataset(os.path.join(data_dir, 'mnist_m', 'mnist_m_train'),\n", " os.path.join(data_dir, 'mnist_m', 'mnist_m_train_labels.txt'),\n", " transform=tf_target)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdAAAACdCAYAAAAE7CkGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAALAklEQVR4nO3dXYhV9bsH8NmaFKZ2IVaQWjAXjtIbqdFFhRKUUdDY64VU9J4FRUk1o1dRjpUYSC+UGYRmKZSCN1IQUVgRvhRCZRdJjgSBkxdKXoQz+39xDpxzeH6e1n7ce172fD6XX/ba64ezXF9X8/RbtXq93gEANGbCSC8AAMYiBQoACQoUABIUKAAkKFAASFCgAJBwViMfrtVq/p8XKqvX67VGj3GN0aCBer0+o9GDXGc04nT3Mk+gwFh2eKQXwPilQAEgQYECQIICBYAEBQoACQoUABIUKAAkKFAASGhoIwWq27x5c8i6urpCtnDhwuFYDgBN5gkUABIUKAAkKFAASFCgAJBgiKgJSgND3d3dITt48OAwrAagtZYsWRKynTt3huz1118PWU9PT0vWNBI8gQJAggIFgAQFCgAJChQAEgwRNWjp0qUhW7ZsWcjq9fpwLAegpa644oqQbd26NWQTJ04M2Y8//tiKJY0ankABIEGBAkCCAgWABAUKAAmGiBq0cuXKkJUGhkpZX19fS9bE+HTPPfeErLe3N2SXXnppyBYtWhSy3bt3N2VdjF3Tpk0L2bp160I2derU4VjOqOcJFAASFCgAJChQAEhQoACQYIjo//HSSy+F7KqrrgrZkSNHQvbMM8+EbMeOHc1ZGG2ts7MzZA899FDIVqxYEbJJkyZVOsdnn31W6fu2bNkSshMnTlQ6B2PPmjVrQrZ48eJKxx4+fDhkBw4cOOM1jWaeQAEgQYECQIICBYAEBQoACbVGXrtVq9XG1Tu6BgcHQ1b683riiSdCtmHDhpasaSyp1+u1Ro8Zb9dYyf79+0N25ZVXDv9COjo6Dh06FLJVq1aFbNu2bcOxnJJ99Xp9QaMHuc46Oh544IGQvf/++yEr3fM++uijSt936tSp5OpGl9PdyzyBAkCCAgWABAUKAAkKFAASDBH9t82bN4ds2bJlIdu+fXvI7rzzzpasaawzRJRT2tnqoosuGoGVlB0/fjxkO3fuDNl99903HMsxRFTBzJkzQ7Z3796QnX/++SHbs2dPyEq7E508eTK5utHPEBEANJECBYAEBQoACQoUABLG5evMurq6Qtbd3R2y0oBVX19fK5bEOHX//feHbPr06SOwkuqmTZsWsttuuy1kpUGTL7/8siVr4n9MmBCfi5YvXx6yGTNmhKx0z3vttddC1s4DQ43wBAoACQoUABIUKAAkKFAASBiXQ0RTpkwJ2eTJk0NWqzW8kQ6c1jnnnBOy0mvvJk2a1PK17NixI2Q33HBDyEoDQyVTp04N2cUXX9z4wjhj8+bNC1lPT0+lYzdu3BiyTz/99IzX1K48gQJAggIFgAQFCgAJChQAEsblEFFvb2/ISjtwDAwMVMqgilWrVoVsOAaGSq8au+OOO0JWGiwq7TDE6NbZ2Zk+9ueff27iStqfJ1AASFCgAJCgQAEgQYECQELbDxFdf/31IVu6dGnISkNEjz/+eMj6+/srnbf0qqDZs2dXOvaRRx4JWWnAY//+/SE7evRopXPQWjfffHPISsNrZ+Ktt94K2YoVK0I2ODgYstJA05kMDB07dixkv/zyS/r7yLv11ltHegnjhidQAEhQoACQoEABIEGBAkBC2w8RdXV1haw0MFTKSoNAjz76aKXPPfzwwyErDRGVzlt6jVrp+44cORKyZ599NmSlASSapzSUtmnTppBNmJD/9+rXX38dsueeey5k//zzT8jOPvvskD3//PPptZTs2rUrZN9//31Tz0FUuvdcc801Iav6asbbb789ZKXdsrZv3x6y33//PWRDQ0OVzjtWeQIFgAQFCgAJChQAEhQoACTUSkMsp/1wrVb9w6PEnj17QjZ//vyQlXbwKf3iffr06ZU+V/pzLe0cVBpyOvfccyt9X+m8pd1fFi1aFLLh2LGoXq9Xm1z4X0b7NXb55ZeH7KuvvgrZeeedlz7Hn3/+GbK77747ZLt37w7ZtddeG7LSjkWXXXZZcnUdHVu3bg1Zadeu48ePp8/RgH31en1BoweN9uusqu+++y5kV199dfr7qt7LSnp6ekK2du3a9FpGk9PdyzyBAkCCAgWABAUKAAkKFAAS2monotJATtWdiEo7a2zcuDFkpR2BSkq7/3z++eeV1vf0009XOkdpV6Q5c+aErLRTzoYNGyqdg/+rs7MzZGcyMFTy4YcfhuzAgQMhu+WWW0JWumYvuOCC9Fp++umnkPX19YVsmAaGxrXStTdv3rwRWElZ6RV5pfvqb7/9NhzLGRaeQAEgQYECQIICBYAEBQoACW01RDRlypSQTZ48OWSl3Tbee++9kJV2Dtq3b19ydWUHDx4M2fLlyysdOzAwELKVK1eG7LrrrguZIaJ/t3jx4pC98847TT3Htm3bQvbqq6+GrDQwtGXLlpCdyU4yf//9d6Xz9vf3V/o+mqu0o1jpnldSus+UhtVKr82bNWtWyFavXh2ySy65JGQLFy4MmSEiABjnFCgAJChQAEhQoACQ0FZDRKVhiUZe1zbW/PrrryEbb38GrVR6DdiMGTOaeo4HH3wwZB988EHI7rrrrkrfV/VnPTg4WOkcBoZGj6q7oJ08eTJkL7/8csg+/vjj9FpKw0vvvvtu+vvGKk+gAJCgQAEgQYECQIICBYCEthoiKu3CUsomTIj/bii98qu0E9FIKa1v06ZNITt69GjISq9WY3Qo/WxuvPHGpp6jNDC0ZMmSkH3xxRdNPS8j49ChQyE7k4GhCy+8MGSlHYbGI0+gAJCgQAEgQYECQIICBYCEthoiKg3QlLLSbjK9vb0hW7BgQciaPZDT1dUVstLrx0qfK+06s3379pAZIvp39957b8jmzJnT8vM2e2Dojz/+CNm8efNCduLEiaael9Fj7ty5IXv77bdDVtqd6Pjx4yEr7YBUykr3o1OnTp12ne3AEygAJChQAEhQoACQoEABIKGthogOHz4csjfeeCNkpV+eDw0Nheymm24KWWnoo7TbUekX6s3+3Pr160PW19cXMv7dWWfFvwqln8Nosm3btpC98MILITMw1B5KA4Lz588P2cSJE0P22GOPVcpKu7SV7o0lb775Zsg++eSTSseOVZ5AASBBgQJAggIFgAQFCgAJbTVEVLJ69eqQ7d27N2SlnYhKOwKVhnlKSp8bGBgIWWkwoPS50m5Co+l1a7RW6We9bt26kPX39w/HchgBa9euDVlpp5+nnnoqZLNnz650jqr3t9J1Vhpga3eeQAEgQYECQIICBYAEBQoACbWqvzTu6OjoqNVq1T/MuFev1xveymekrrFZs2aF7JtvvgnZzJkzm3reY8eOhWzXrl0he/LJJ0NWevXUOLSvXq/H9w7+i3a+l5VeffjKK6+ErHTNd3Z2huzFF18M2ebNm0NWGn5sF6e7l3kCBYAEBQoACQoUABIUKAAkGCKiZcbSEFFJaXeq0kBF6VVopb9Xf/31V8i6u7tD9u2331ZcIR2GiBgGhogAoIkUKAAkKFAASFCgAJDQ9q8zg6w1a9aEbGhoKGRz584N2Q8//BCy9evXN2dhwKjgCRQAEhQoACQoUABIUKAAkGAnIlpmrO9ExJhgJyJazk5EANBEChQAEhQoACQoUABIUKAAkKBAASBBgQJAggIFgAQFCgAJChQAEhQoACQoUABIUKAAkKBAASDhrAY/P9DR0XG4FQuh7VycPM41RiNcZ7Taaa+xht4HCgD8F/8JFwASFCgAJChQAEhQoACQoEABIEGBAkCCAgWABAUKAAkKFAAS/gNHDuzORSry6AAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdAAAACdCAYAAAAE7CkGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAApC0lEQVR4nO2daZBk2X3V71tzz8rat66uXmam15npntUaWxphO7Ahwg6DLcIY8dWWmMBfEIGAAALCiCCEwiEjIiAIPjgMYRsEshwMtrUMhDSSZtNM90x3z/RW3dXd1VXVVV2ZlXvm2/iAeyDznBplvRkHAXF+3/r0y3z33XffvZV5T56/lSSJEUIIIcT+sP9vN0AIIYT4fxEtoEIIIUQKtIAKIYQQKdACKoQQQqRAC6gQQgiRAi2gQgghRArc/Rzs+26Sy2cGNNvC46hGlmrbwgMd8mLLwhez1xqmGfYznVF/ukPeb8SX8qbgi0f/ERF7w9HalxDRslBj98ixHaLhgXEcD/x7q9oy9WaPNfoDsR0rcbyhl5HOZNdEGXGYWPTA0U7xoWC3kIwTPt5HPMeIXcV+0cbawo6z6IAnfJjDiBg04+0kSaZHe9f/Tcl1kumMN/j2I04VbOzFpFPieMQ3ZPMbexhZW9iwcMgc6uFUH5O5NohC0KIoAs0hN8NjcwW5NnZvbaLa5OKsJEaNvGFMjovZPEjaMsxmt292+yE9dF8LaC6fMT/+8VMDWnZ4sjPGZD1saCmHx+Wz2LnFrAdazsuTc+BxtssuhwwI0rl8jmE3EI9ySDd6OJZMGAXYlhgHp7HwxUmC12uR87K3S+I+OUUPtEIer3e8VAKtmMP70ekOvt/nv/QNbMgIOJ5lKsvZAc0iD3pkkxth431lfxXYZJh4Lh7nkpvIHkL+W2o2URKJDKgwxHGSyfigOQ62j00mZE6ki2Acodjv4diJQjzO8bF9rA/4OjvaHwwOuUd3Xm6ssnf8UUxnPPNPji8NaLkIzxlHOKb6pPPafeynfqcLGl2MclnQcj4+Y2xIheyDSTkHWmZhErROHsfP7fp90Br1OmjlGB+iuUIFtIqD1+aT+S1v4fsVAvJMhtjPNnnuezH2fYfMgw557fAz+cIr1+CY98+95/8IIYQQYk+0gAohhBAp2NdXuMbgV5gW+WrYcun3Q6iRrzNMTN4vJh+zydew7KvjxJDXOvh3Q5RgV/T62L4kJt+HWfh1wdgEfvVZGZ8ArdVqg7a9tQtavV4DbXe3BZrv4dc35Qq2JUnw2ian5kCbm54F7dqVq6BtbN4d+He3j18Rj0JijImHxgrbn4zI91ku2YOxqYbnZV+lsn0U+mUtGZ9sT8yQvSn2VWVCGki+STQOeT/2dS3bi2N9mrCTkL+xbXJeuhNB9r/Yn+wJ62f2dTdr3ofANoNjg92yXgfHsU32ALLku9SEWQB65F4EZJ7xyXnJPetG+LVku47zh9NqgpadHQetksOtIpd8neyEeDN8ss/qkjHgk5vrkSfLcdnzQvwXZK7xyXPlWbjN0Gtgv/Q6g/NqEpLB/aA9e/6PEEIIIfZEC6gQQgiRAi2gQgghRAq0gAohhBAp2JeJyEoSYw39lsb10bSSz5BNdhc3uwu4X20K5Ld3JXIg/xlgB7TyGG6AVybxN1G37+6Adn8Hf/8UJXiOpaUp0J57/gyed6wM2va9Kmg3b66D1u7iRnmxUATNz+L1bt7bBG1l5Saeg/yW7d5OA7S7xOTUGzKTpS4zmyQmGjLlJCOnIRADGvkNKQv6YPagmJjcLPLbS/Z7eWZ28MhvJZmZi3UeO0cYjOaqSZgjh8D6iv14lTWZnpeZtWhyBAvKwGsL9xE78qNwjG3KztCzwsyKPjGtkPdLyG9IbQt/z5sk6PKKA9T65CSei+Mn45AxRdxWYRPbEts4v9kl/N2ma5PfrpLf63fIeV0XNXYd7R7pK/K75FKG/DaUmLp85hQkv1+OYjRxdoaNXh/wmOkTqBBCCJECLaBCCCFECrSACiGEECnQAiqEEEKkYF8mIs+1zPzE4AZwNpvB42xM5llerIB29vQjoOUyaEraWEfTyu3VO6D1umjwKczPgBaS5I+NzQ18P5K28fDDR0E7+shB0C5fw4zrWzdvg5YEaErp9XGzO5PFzf2HjmKSyInTp0Er3RoDbX1zG7R6F01Exsc+6NhoIFjdHDRh9YgxYhQc1zVjlcqAVt2twXE2+duPm2WYCYSEl5PXsiQi7oxhwed4XzOkAEIYjBiUT11Zo5mDaEUVYtCg1VhIFwwnRe2FRfqK+HSMRQwfCXFNpTamMaLEJPVBY17EihaQk5JbZjrEaRKwJJ08zpcRiY/qk/SbmBznOjim8g6egyVtGZymTUCMT2GG9AG5tvuNGmi7xBRa83AuC+qYCDRmo9lorojGyUyI/ZIhz71LjF4hKUDiFQaT2yznHhzzAH0CFUIIIVKgBVQIIYRIgRZQIYQQIgVaQIUQQogU7MtElM245uThwZJc93fRuFPM4+bvqWOLoJ05gyYiK8HXNjtYPqt7HTeOWcpJu4eb+yvXb4KWn/kkaJUCtu/qUNkuY4zZCdHMk8RLoFW3cUN9toRJRKuraEBavXUTtIuXsF8sFzfoV1bRvHThElZZz5ew7NnENPbfuzexLfWh0moBLY31o0ni2IRDiUjUBDOio4QG/RAnS4K3xlgWEVlqDjGVRAG2r2eRFJoETRu01BjrA2KGikjaCnPfMINPSO5ZxEpt0b5nx7EaZ9inDjFyGIPHBQH2VVqiJDbNzmAKTZskT7VI+UKWuBMSt5VHxgUr+eWwknujlnMjZitWasxmcTrk3jrk3rrENNgPcPxUScpSo4frw40WzqHb21ia8eAMmh/bBSyvmIlwXJQsNFIVbNQSkjRlDZU9i1nq2Z+iT6BCCCFECrSACiGEECnQAiqEEEKkQAuoEEIIkYL9JRE5lpkfHzSpsM3pXB43xRfm50GzSMmnza37oH33B2+AtrKKG9HHH8GUoOpwaRpjzI11NO5MOlhqrF0tgHZ3HTfUf+LgGdBKRXy/3c40aLfufxu0KzevgFYoYELTwSOHQLt6Aw1D3/72y6B1+lge7dFZTFTabWC5Hz+LbXn6+MmBf1+89B04ZhTiJDG93mDbWF+6PjE2kESSmJgiaNAPSU6KiTHGIoaCiJh5LB8frYik/4TM8EGSeWwHz8vKQsUh3lfm53JY6TdqXsLXstJqtBwcuzaHlZnCfrZJ2TiHmbpSElvGtP3BNjd62I5qgP1ZDzHCpxuikcUn11DyibmFlSkjSVZkOBqLJSDFxJjGjHPknjE/V3v+IdCufOpX8Byk//zv/SFo9Vf/BLSajc/fOCljuW7jtWUdPG/DYKqa08Z7aUjqm+kOvl/vA5K39AlUCCGESIEWUCGEECIFWkCFEEKIFGgBFUIIIVKwLxNRv983t4dScq7cWofjKpOYzDNRwZSb3QZu6nZIraA7W2j6uXITy5lFzOBx5QZoK7fRgPTMRAO0wMIUjUdPHgPtzGOodbu48bx7fwe0VjUP2tIRTDE6ffIkaAcP4nHXrmG/1Fu4oZ7PY1mgHkkmWVlZAY0xMTaYGuIQA8Uo2JZtMpkho5qNw7S2gyXumLklk0HTBi1TRmJeqFmGlHuKiPnGJoYXllYTkZNEFjEvEQOSTRJieImz0VKbXHLPEjJDxMSVxErEMaMXMyUFATG49NFUUiihqS8toW3MztCUVGNmFHIvmsR40iQpRjYpP9YifpQ8+RhTcrHvisRY5BETETMHsRJ0fdI+j2hhAfu99+SP4Xk7OM8Ur50HbTyPCUNdH8uZOUU8b62Hz5/dQlNXQA1D2FdOSEoRDj1rMhEJIYQQHzFaQIUQQogUaAEVQgghUqAFVAghhEjB/pKI/KyZWR4s8fXa5TU4bucOpgkdPIyb05OLaCxys2j6OH32SdBmFw+B1u9gSZwbt26BtriIJXHcBNN6CkU02kyNY5rQ5t0t0FrEIHV3Dc1LWZLqs7TwKLYvg6aCRgc33sfGceN9aQlLyd25uwnau+9eBq1aRQPX4vwcaJ3GoAkrIaaKUYjjyHRaQ/fRRnNCn5hMWNKPifFvxAwzt5CYl4AkGzE7jkPMHZUsMdKRscMMH4kh10aMDI5LEotIulejg/ewVq+BxsxVDjElWSQpyWLXwUq/EdOU5+H9aLUxAYs9K2mJTGzuR4MmwZiUtgr6eA1ZYm6xclhGkJVfa4Y4lnsWao2IGHJIulXew/mS3Z8+eTZ65BlyfTKWp3HcLlcmQQtK+H7BX/k10MZ3NkALL78CWp+YtXYaaPbsVFFj8Vse6Zeii/23MDN4vfYGmknf/789/0cIIYQQe6IFVAghhEiBFlAhhBAiBVpAhRBCiBTsy0TUbLXNy6+eG9AaXdzsnp5Fk061gxvqr52/BNrOTg20bh/PcXDpELbPwQ3hA0toUKhUsDzW3CyaPkyEphpWvu3cW5gwZCXYlkoJ/165RUxOf/zifwStWEIDxTNPPQ7a1PQMtsVGI8T9Khq9XGKGeeSR46AtkwQkxxt6LUmFGg0LXjtc3swYY1iYkE/SW1gJqLCHZodMFh+FkLx4vFgB7ezpZ0D7pZ/8q6D9wic+BVpCzA62RR5LYshhiUCsr1587b+A9uWvfwG09W1MsWKuKYeUVmOmDZbkxMqohcQgRYasmRyvgLZq0MQ4CpYxJjP0+cFiyU4OK3OHjfNtfHa6OAWYTB7vbTvCubER45y3FaCZxUnQOJnNYHk0L4eNmTz9LGjlpUdAO/253wSti6cwLWKQqheIkYqYwYoZNGxuddEctLGFCWT9Pp4jn8V7OV7A1LfJKTRDFbNDSWhsvD/4vz3/RwghhBB7ogVUCCGESIEWUCGEECIFWkCFEEKIFOzLROT6vplZOjCgHX3sNBxXHkdDTq1aB+17r74O2vYWpvqwTfHdOm6eb9xFM0+3g6aK6TncOA5jLM9TKqFZppT/GGif/HMnQKtXMSXo3JvfAG1z8zpohSL2X6mIG+ClUgW0bAaPK5Zwg75cRm1yegq05UPLoNWI0WtrZ/C+dbqknNAoWMZYQ0aTjI8GCJuZVohrJYxJmhAxIDmkjF6CLzVnjz8N2n/64texLST5JSTl4nilMVLijJT8CknaEysX9rNP/mXQ3rj8fdC++vLvgEbLlJHycvksjjuWxMP6pd/FclTFIjGafITlzKzYGLc72KdhhO3IkFvWI2MqjHC89xNSequIiUUeMRbhK40xEY7RPrnhkY3akTPPg/bxz3wZNL84ge0jz0F22DRojPHIcuKT+WiLGN2aPez7Rge1Vhd7xiWJXJk8jp/xaXJtBZxbhsuXEf/e++gTqBBCCJECLaBCCCFECrSACiGEECnQAiqEEEKkYF8mokIhb558+qkBrdHCskMrN1ZBu3ETE3eCPm4mH3vkIdDK5TF8v+uYmrKxiQakg8tYGuyh4z8JWr2HCT6tGMt2HTmGpcHyeTQ5rd7EtsSmAtqxE5hOs7H2LWxL4zZoO8SoND6ORqClRbyOfAFvfS6PJo2dbUwDOf/W29i+3mBKSi+liShJjIEqYuhXMJZDSmol6PjwiMHAYsYdYiyy6YnJa0kJsZjYQGwXX/z17/5n0K6uXh3pvAkxkJw4iOlUnzj150F7/vGfAe21K1jS7+b6Cp6YxQSRUlGOj89FIYsmmpLB59sh7+d5adOtEMtYxh+a/rwM3u+ImqiwHZ6NppUsKUvXtXGMhjk8b5KQsRfjMxuRgbF46qdAe+6v/WPQimOYGBeRNCGblD0rkJKAeWL265Jn7Ydk3q92MGWpR8oJ+uS5L5JScpOVCp6YJIu1+mhGtYdKaiZ0wvjTY/f8HyGEEELsiRZQIYQQIgVaQIUQQogUaAEVQgghUrAvE1EUxaZVH0wUevc9NDy89vqboHU6uDl94AAx6Rw6BFqxhCaD6n3c/N26VwNtcQlTgo4/9kugXbmMiUC+h5vi4xNYCs0l5pBMBjfZbRe7e3oKTR+nTp8C7Wu//49Ae+8i9v38DPZpuYTnnZvBNCZjoXGhtlUDzfdJckoyeL2WhSkio5IMGSMSYg5KWK0s4jGxqOuHQP6UnB3Hvvzsr/w6noP4Pb753T9G7Qd/AtqFNTRkXbmBZf7aPTTrHV/Gsf3s8U9g+0gXPP/Yz4L2H/77vwPt+hqOsU4H+75ex/ttO9gxY6Skn+fhs1Jr1cg58JlPS+zYpl0eNP5YGRzXvRDNYP2YJCyRAeT5JBXMxz7pEWNRQkqcJaSs49SJT4L26M//bdDylQOgxQGadLLkXjgkiscNcAz45H7nLTSSBaR8XUCe8TEfzUHlCXKOPM7T5TIaItshmi7Z9BAMlZJjZr0H6BOoEEIIkQItoEIIIUQKtIAKIYQQKdACKoQQQqRgXyYiYxITDaUy5Au4Uc42XfukhE2BmIPKY1jKq9FA80CXlEBaXJgHbXIcN97XbmIpp267BJpbxg3rVhvfr1BAY5Ht4Ob5eAWvbWkRN/cPHT0I2luvHQOtuoXxObu1GmjbO9hXvS5ex+QElvs5cBDbNz6F13H12rWhf1+BY0bBsizjuoP9HpEEEVZmi5GQsmfMWMRKoZWLOCZ+6mOY6sPqo7195Rxov/tH/x604yfxvo6RZ6C5gYlQs5UF0J586FlsHnkeLRK9xMxG7G/siPQpKxHXIc98hzw/rkNKyZE39LNoFklLz8RmJRk0ZjV30KjV6WF7HdJ3S3OYZDZDtFYD72O7jfNbQgw+M8d/ArSHf/HzoGUqmJbWD/FehD1i5gmxfW9887dB+wuf/oeguWQA+TbOoTG5txExTU2QeXVmBrU4JqlnGRy3fg5LOPbJa3tDZfg+aK7RJ1AhhBAiBVpAhRBCiBRoARVCCCFSoAVUCCGESMG+TESJsUwytCk8M49pLc88/SRo11ewxFm3g+aWN8+9A9rG2gZotd1d0JYOLIHmuaRMTm8TNNtgOaLqNp73Oy+/BNp45gegNdtoePjpn8GN93FSdickqRwTY2jwmSgtg3Z7DcvGvXPpImhRiBvjj53C0m+zC2hUyRUxrcT1Dg/8+1sv4f0ehSRJTDhcxogk/dik3JVFklBiC80YMTHBGFJ6amUNS3n9+m98BrQv/72vgPbZX/4saKvr2CevXsKx85sv4PvNT6GZK++jySmTQfMau94v/8EXQDu/8jpotot9atskrcYmRguW4EKMJp6H05BNSoY5HhkIKYlsY+pDZcTWtutwnBfjOFuexhSvw0v4LFrERLW7fh80Esxjxs88B9rUp/8BaO4UGoYckiaUIWYeq4fz7x9+8e+A9gsvfBE0lgQW9XGcvfj7/wy0tZVzoGVzOP/GBvvPTlDLRfjsugafgyx5NraISSwaLsNHyhW+3549/0cIIYQQe6IFVAghhEiBFlAhhBAiBVpAhRBCiBTsy0TU6/XN1WuDJpXjDz8Ex50k5bgWFnGz+4dvYimnC++8B1ppDNMnzj6BRqVSCTeir928Cdrlq98GLedj+Zvnnvkx0I7Mo1HpfhUNTf0+moimJqdBazbRuHD+4gXQ6vEToBWtHdAsmuqCG/4BSTpZu4umqakbaEqaXUBDUy43uPHOTCCjEg21N+xhSolFTET5PDMRjdYfCUkbCUh6y9rGnZFeWyljH/3W3/1XoHW7eI6MjePYRKQ/SZkplghUa9ZAW9m4Blqri4k4tBwcubeeT0w/Dj4Dtkv+ZiencIhxw3KY+SsllmWi7OD054zhHDA9XgFtbg4TzzoeKfG2hc9nO0bTysxjz4M28Zl/ClqhiMltrJQXK0nmEz+XTdx5jovvN71wCLSwj86nHim39smf+5ug/fhf/Bug1erYV1/9+1iab3PjLmizHpY98w2mDmUSvLaYlNncHjKoBgHOPw/QJ1AhhBAiBVpAhRBCiBRoARVCCCFSoAVUCCGESMG+TESNRtN85398b0DzHNywPnIYUznGxiqgPXH2LGjjFUz58ElKRTaHKSzFIhqVGq0b2L4jU6CZGJ0M1RqmhjSbeN4SKctmuWh8+tZLmGIUkxSNze0t0Can0ID0zitXQQsjLEf06KnHQduponnp1ioaZF5+GZNynnoS3+/hI8N9n85E5DiOmRgqqxaQUkfNFrY/JglOLkknSmJSTinE19okvaW6i2PipVfQlGaT62fBPGGEok28MvksGlyeOIqly1jlpd/+5r8B7cXXvobnJcYsVsopidB4wUpUseQgh8w4Fq2jhs8FK3GWFsd1zMRQClihjM/2zBImrU3P4PxhBcQMlsfryhGj0sOf/RJopXGcB8tFfK1DTIM2Gd8+OS6TRfPNr37h90Bjz0ZE4pN6JHkp6uF9tGwcBBkP5/g+ud8Bubawh5rbx7GcCTGJyKliGpPZGUqvI6ltD9AnUCGEECIFWkCFEEKIFGgBFUIIIVKgBVQIIYRIwb5MRJZlGc8Z3Ihdv4PpNb0umgzm5mZAm57CtJZKCRMkVu+guWVldR20v/5pTO94/DHcJB6fxA36Th8ND6+/8jugZXO4Ke44uKG+fRsTlUJzG89LUnZ8Yr4o5HHDf20dy7I1apjU8djjaPrJ+rhpb7toCGs1m6jVWYm4wX+n9Xu4rmsmpgbvT2Kwf8N17Lc4IkYg8jei7aDGjEXM2vL2tbdA++XP/SJoxLtEDTmOj21xXTQ7/OrPvQDaE0cxKev6+hXQLtzANrNyVInFzBLYCxExa7FqZnEf7xGUqjPG2KSzmK/ow6RbwfsbyzhDpdosMgY6Ic4ffQevgX0UcWbRlDRWQa11HeeysQhTczIkccf2sE+CiJh5yJiPyPPOjHMhGSshGcvdEOf9zXv3QOvcw7kxaKFxMgjw/bpkPPZdXMb6xBTqk3JrJZL6tVAevDbfqcExD9AnUCGEECIFWkCFEEKIFGgBFUIIIVKgBVQIIYRIwb5MRNlMxjx89MiA1mygoeT82y+DNjuL6R0ffw5LdC0emAWtH2CZpVYDk2jmFzGtJyQb29uk/NjF926C1u4fAe3YcSzVVhlD04eb+S5ofu40aOfOvwNar4t92qieB21mugJat4Ob9vV6FTSaMGMR89cCmr8WFrCcUxAMGktYybBRCKPIVKuD7XVJCSz2/hG5Jou4WxxiqLBIaSfbYrE5RGLeFiLaNppvLGKMYYaxF37+c3gOUs7s9cuvgvbNN/8baSCDGK6YOYb0S0gMXJaFfeqyMmU0tYmkRZHUprSEUWjuV7cHtCYp5xbFaCKqkBJizAzGzFYBuWeFCPvTI+k6G5fPgXbh9T8CzZjRDHEf/9SvgTY2js97zCrQuXhv793C9t06h/Ogdf8iaPfXMFXNJf232ybJQawcXkL6lJjzZiZwzfB77cF2OGikfIA+gQohhBAp0AIqhBBCpEALqBBCCJECLaBCCCFECvZlInIcx4yND5bpqu7gBmunjUkYW5tYBureFqZPlMvYpKlJLA3WPIApGpaHr3WJSWO3jmaBzbuYBmLR2ku4sZ0hpaaefhbTaZotNAe1u9h/K9ewBJtjML3j9Ck0OS0uVkBzSb/YRDtweAm0goeGifFxLN+WDJlmLOY+GYEwCMzGxmC6Vb6AaSEOMRYxmO3EJklPrKSWw0qhkfezWIIPeb+YmGosMp4sYjZi5aNikpzDIqBofg8x89B6awSbvNYZMSXIIc8UM2GFpITWR1nOLI4j02wNPntBB5/PXBHnnkwNj2NmsH4fDS/VBpYbfKf2FdCiV3Cs3LlxHbRr73wPtAwx+HjknjU2boH2lz7/L0HzMzi/WSSM6f4KJl7tvP5V0OYn0FA6R8pTBqUKaKs1TL7bauMcWiIl08YyaM4rF1Ar5PID/3Y+YC7TJ1AhhBAiBVpAhRBCiBRoARVCCCFSoAVUCCGESMG+TET9oG9u3RksydVooCFnfgbThGamsIRYPp8HrRfg7rRLNsWnxjFV4hsv4gb41PxPg7Z2Bw1DbZJw4XpoVHrnApaL2p6vgXb4EBpyJiYroJ04/hBo3d1z2L46mrBcF40L7VYbtGodjQuVCbwfExNomJgsExNFFvt+uLwRM+WMQmKw1BYra+R6aCxiRg5DDD7U4OSw1xKJaCRchiYg2dSCNBoxMfgwjWERA0lCGs1KnLE/sVn5MZYMxdKJQvJ80z5FiV5HWlzXM1PTgyk0Him/Nl8eB60S47QZNHH+aO3UQLM7aHjZ2nwJtGYLjZi7VTQvlTL4LA6XaTPGmBwxA04efwq0ahPn82KI/e7ZeN64j/e2T97PK2GfFnP4PDdd1DbIYEkc9mzg2GMJTTHRwGD5AXOZPoEKIYQQKdACKoQQQqRAC6gQQgiRAi2gQgghRAr2ZSLqdLrm0qVLA1rGwSSHR08+BtrJY8dAC23ceK/XUcuQjfLyGBqQvOQyaM17aBjqd7GEzYHFs6DNzMzh+5GUJZbGNDODZp6Z6QnQHJJE49lYqs0xaKTZur8N2jsXsDzabhPNB66L9y3ro/ngxPGjoD391JOglYqDaSWsZFhaWCpNwMxmpP1s/58l+FjMkEMMSMOJS3tBQod4+g8hT5KtWPPCEE0vPVK+z3HRDBcF2Afs/Ryb9KlL+oWkBCUG3y8kRiWXjBVy2g/ycqTCHjKTdUjfNUmakB3itfokmcch5puZTAU0dm/rIT6zxTE0AjW72L6EGOKyPhpy5k9gOckeubb+DpZ/zDs4J7fbaBiqNvC1nXIFtJk8jvmZLM7xNfJs1BOca7Ms3SrAubuf4BgN4uHSjHsnYOkTqBBCCJECLaBCCCFECrSACiGEECnQAiqEEEKkYF8momKhYD727KMDWhzhDvihJUzhqXdroL11/jxotWoVtKkpNN8ce/gwaEePHAItTtBok3OxJM5EBQ1D09MHQCu00dByvYEpQXfv4oa1a2Mi0NX3MIVkZ+OHoI2VcdPeR8m4pMxXjpTx8bOo9dq4GX/h4rt4DpLQdPbM4wP/jliprRFwbNsUhsqXRWSjPyblvaKYlCQbscwWSyeKSB4Oq1zGDC+sBBJLJ2JxR1954fdAy5OSUm9ceRm0L331N/AUCTaQ9d/It4yYuti1eaQPWGIRS4hhqUOuy6YrNP6MQpwkptsffO3N22tw3HYWn+2FMUzxKjpo8ImI4TBDplyX3Aurh66kArk/FnGrBeT++C4xjZH3C8l4zJDn/d3v/wFoP/zWvwbNsfAk2000Fk0XMPFsrlDBtiyAZNbvbYIW9NBcFcfYp3VSCq09lGwUkvvzAH0CFUIIIVKgBVQIIYRIgRZQIYQQIgVaQIUQQogU7MtElMvnzONDZpE4wk3iZgM3Zlt13Dh2HdycjkmZpatXV0ELyCb74iKal0pFTOCIIjTLvP79fwvaubf/FmjtNqYEjY2NgdZsoIGAlXc6fuwR0E6dwPSfmBhpfGIOeughfL9+l5X7wb+d7m3fAy2K8DosYjRoNAf7JSbXOgqWZRl/yB3V6eG1J8TgEwZ4nBkuTWSMsUmkDfEfUSNUQjTPxnFs0ygdZtzB68hnSKk20mZmbuiS5Bz2jLLKZcyQwkw/1KRCUqBsB/uepwlh+5gJy2El5z4EydA5LA9deR1yrdc28TkxZLhbXRyPBWLomvTxOZ4vFEHLW9ifefIRqEfGnk3ubeMGppb1qmjIccjnrPomlnV0yBggl2uqAY7RO1vYp5kslj2bsfHZcHI4/26S0nQhuUmdCNtSHTIgyUQkhBBCfMRoARVCCCFSoAVUCCGESIEWUCGEECIF+zIRBUFg1u/eHdA8klJx6T0sK9bvYmLIwoFF0BYXMf3n3Xfx/Ta30JR09fod0E6SclweSTRZPohtYREzq7fxHNvbO6CVypgcMzWJCSbzs5iyZNlokOoTl8vOfUxt2ryLJoAkRnNENyAGlB7eozBAw9X6BiY5zc4MXkeU0kQUx7Fpd4bKIpH4H5cYPgxJw2GViGjLyIFxhPchJoYXz8dnICTGHVoejZh0mGHIZQYa+n54nEX+TnbJ2GZlm2iiFDGksOt1iFHJ8bAt7Hl0ifkrSdKNKYZt2yY/ZMJbnpuF4xpbNdB2a2iSDHrYNvIYGyKZuE+Meglef6mIaT1eDkt+BaQ8XJ8Mn3u/+y9Ai1w8sOeQucLB+53JoRmqQ65tt4PzzO0dTHzK+2QAldAw1I/RMBQRQ1PdwuNqxGzUGLpLLJHsAfoEKoQQQqRAC6gQQgiRAi2gQgghRAq0gAohhBAp2JeJKIkT0xsymvRISszG2l3QOh3cTJ6dngJtenYGtDmyuV8npbfevoCltyJSwubY0WXQyqUSaMePYRrI8jKmHW3ewxSNHCkXls9higYrP9bqoEkh7pFyRGRvu7pbA62xS9JpmNmEpL9kczhECnk0SDnuYDknZoQZCcsyzlCCDUsdYuk6EUmNYdCUIGb6IX1kkzJbCSnpF/SxLfQ6iCmJmaFcj7SZEJE0JpYcRdOYyHhibXYd0hZWDo4lOYXERkMSdhwPX8uSjVKTJMYEg+coWGgGGyMpN1Mk/icwxMhCLEPlDJrfZrM4L4x7WB5t3CcpPGRc9Mij1yBGG2aS7JHyY10yMIIEr60XozmoTcZjr9PBc7TwHBVDyjAanHsaNp5j0+Cc1yUGtpaPnRW4g/doOLHq/0SfQIUQQogUaAEVQgghUqAFVAghhEiBFlAhhBAiBfvalXdd10xNTw9ovQ5u1s7Oouln9cYN0Nq9FmgBS+XwRkth2biHKTw2KSE1O4npP9PTqAV93BQPSFpPgZiDcnnUWC0nx0XjQp6UMgpIYkalUgHt8TOPgnZvExOL2m28byxRZ3wC008OLC2A5g5dh0XMNqNgW5bJDZkqOl1sK7s3MUn1cYhJixKPZtKJQzxvN0EtW0ADhGXj/U+IQeO3vvbPQauUsLTTxvYaaLT0EjkHKxfGTD+szTbR2HHMRsbax8rQ9cltKxRwfKYlDiPTqNYGNLuO89EUSfFacDD9J58nyUkOjgufPBcl8jkmT0o9euT+hAG5tz62JUPmmYBkckXEHNQhY36n2wBtN8S5u0/GRUCMZJ0+XtstD82U3S5eR6OHbela2GablI3r+9j3dm7IwEWu4f3/2vN/hBBCCLEnWkCFEEKIFGgBFUIIIVKgBVQIIYRIwb5MRLZtm9xQyZpyCTfUlw/Ng9bpYPmxbhfThG6s3gTtNinR1SZpFvkCpncUi9i+kKRytIk5qN5AU8EFkna0sYFJRBmSLsLSX+bm5lCbxzQml6TTTBHj08Isvt/CfB00VqHHsYkRgvyJ1SQpUBcvvjfw7w65P6MQJwmkVnVJipVDyi55xCjBSnTFxLDASn6x14YkOIiV7bLJo2WTNkcBvuF//cHXsH3EfOLQkmSkPBo2z4QsEYjgkvQfm5VWIzBjkSFl+Vjpuz55HrMk3SstYRCanbXBecVvYjsKxEQ05aPJr0xKiLkR3h+HjCkrQJNcQp73boLvF/vYPpuYiLJFMh+R8eja2L6gh2ae+i5JmWrgmApCYhAjYzQm88waMZluVXFcRMQo6mWxr3xixMxkmCFsMO3IVhKREEII8dGiBVQIIYRIgRZQIYQQIgVaQIUQQogU7MtE1Gq3zRtvvDmgnTlzAo5bPIAmorEybtZeuHgFtMtXroHmeGgeOHnqGGgHDyyCViGb5w4xZLSIMebWnXXQ7qyhoanVRpOL3cANa4eUgWo2b4O2snITtJiUCjp8+ABojzx8FLRCgfQB2RhPSBoP8biY6n00Ja3dGTRSBcFoJhVsBE/nAYgZxSKOJ1bKK4xI6SmS8uKw5Cj2Nycpe9br4jlsVvYswvtqSEkpm5UzIyk0zAxlyHiPmZmF9BXTImLCoqclHiKXGNUCUoYuJFq7nc6Yxsj7GXN2+cig1iYl6DZroBXJc2KTNCVW+i4khpc+KXvmGjTEWcQwtNHC9jk+3p+52QpoCTE+xWQ85n1MI5siRqU6ea4apLwiK31n59mzQdpHjGkdMlasNvZB2cb+yxSISW7ovBZ5vt8/ds//EUIIIcSeaAEVQgghUqAFVAghhEiBFlAhhBAiBRZLLtnzYMvaMsas/tk1R/x/xHKSJNM/+rBBNMbEPtE4E3/W7DnG9rWACiGEEOJ/oa9whRBCiBRoARVCCCFSoAVUCCGESIEWUCGEECIFWkCFEEKIFGgBFUIIIVKgBVQIIYRIgRZQIYQQIgVaQIUQQogU/E9xc9tQEFm4vgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Show a few random images from each dataset\n", "dataset_first_n(ds_source, 3, cmap='gray', random_start=True);\n", "dataset_first_n(ds_target, 3, random_start=True);" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# Dataloaders\n", "dl_source = torch.utils.data.DataLoader(ds_source, batch_size)\n", "dl_target = torch.utils.data.DataLoader(ds_target, batch_size)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Our model will consist of three parts, as in the figure:\n", "- A \"deep\" CNN for image feature extraction (2x Conv, ReLU, MaxPool)\n", "- A digit-classification head (3x FC, ReLU)\n", "- A domain classification head (2x FC, ReLU), with **gradient reversal layer** (GRL).\n", "\n", "\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "What is the gradient reversal layer doing?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "GRL is no-op in forward pass, but applies $-\\lambda$ factor to gradient in the backward pass.\n", "\n", "How can we implement this?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "`autograd.Function` objects are what PyTorch uses to record operation history on tensors.\n", "\n", "They define the functions used for the forward and backprop of any tensor operator." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "from torch.autograd import Function\n", "\n", "class GradientReversalFn(Function):\n", " @staticmethod\n", " def forward(ctx, x, λ):\n", " # Store context for backprop\n", " ctx.λ = λ\n", " \n", " # Forward pass is a no-op\n", " return x\n", "\n", " @staticmethod\n", " def backward(ctx, grad_output):\n", " # grad_output is dL/dx (since our forward's output was x)\n", " \n", " # Backward pass is just to apply -λ to the gradient\n", " # This will become the new dL/dx in the previous parts of the network\n", " output = - ctx.λ * grad_output\n", "\n", " # Must return number of inputs to forward()\n", " return output, None" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Let's see it in action:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "tensor([3., 5., 7., 9.], grad_fn=)" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "w = torch.tensor([1,2,3,4.], requires_grad=True)\n", "t = 2 * w + 1 # What should the gradient w.r.t. w be?\n", "t = GradientReversalFn.apply(t, 0.25)\n", "t" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "tensor([-0.5000, -0.5000, -0.5000, -0.5000])" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "loss = torch.sum(t)\n", "loss.backward(retain_graph=True) # don't discard computation graph during backward, for later vizualization\n", "w.grad" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Does the output make sense?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "As usual, let's have a quick look at the computation graph." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "image/svg+xml": [ "\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "%3\r\n", "\r\n", "\r\n", "1487428673472\r\n", "\r\n", " ()\r\n", "\r\n", "\r\n", "1487428777248\r\n", "\r\n", "SumBackward0\r\n", "\r\n", "\r\n", "1487428777248->1487428673472\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "1487427022912\r\n", "\r\n", "GradientReversalFnBackward\r\n", "\r\n", "\r\n", "1487427022912->1487428777248\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "1487428778736\r\n", "\r\n", "AddBackward0\r\n", "\r\n", "\r\n", "1487428778736->1487427022912\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "1487428777536\r\n", "\r\n", "MulBackward0\r\n", "\r\n", "\r\n", "1487428777536->1487428778736\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "1487428777440\r\n", "\r\n", "AccumulateGrad\r\n", "\r\n", "\r\n", "1487428777440->1487428777536\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "1487428672000\r\n", "\r\n", "w\r\n", " (4)\r\n", "\r\n", "\r\n", "1487428672000->1487428777440\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "\r\n" ], "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import torchviz\n", "torchviz.make_dot(loss, params=dict(w=w))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Now, let's implement the model exactly as in the paper:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "import torch.nn as nn\n", "\n", "class DACNN(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.feature_extractor = nn.Sequential(\n", " nn.Conv2d(3, 64, kernel_size=5, padding=1, stride=1), # (28+2P-F)/S + 1 = 26\n", " nn.BatchNorm2d(64), nn.MaxPool2d(2), nn.ReLU(True), # 26 / 2 = 13\n", " nn.Conv2d(64, 50, kernel_size=5, padding=1, stride=1), # (12+2P-F)/S + 1 = 10\n", " nn.BatchNorm2d(50), nn.MaxPool2d(2), nn.ReLU(True), # 10 / 2 = 5\n", " nn.Dropout2d(),\n", " )\n", " self.num_cnn_features = 50 * 5 * 5 # Assuming 28x28 input\n", " \n", " self.class_classifier = nn.Sequential(\n", " nn.Linear(self.num_cnn_features, 100),\n", " nn.BatchNorm1d(100), nn.ReLU(True),\n", " nn.Linear(100, 100),\n", " nn.BatchNorm1d(100), nn.ReLU(True),\n", " nn.Linear(100, 10),\n", " nn.LogSoftmax(dim=1),\n", " )\n", " \n", " self.domain_classifier = nn.Sequential(\n", " nn.Linear(self.num_cnn_features, 100),\n", " nn.BatchNorm1d(100), nn.ReLU(True),\n", " nn.Linear(100, 2),\n", " nn.LogSoftmax(dim=1),\n", " )\n", " \n", " def forward(self, x, λ=1.0):\n", " # Handle single-channel input by expanding (repeating) the singleton dimension\n", " x = x.expand(x.data.shape[0], 3, image_size, image_size)\n", " \n", " features = self.feature_extractor(x)\n", " features = features.view(-1, self.num_cnn_features)\n", " features_grl = GradientReversalFn.apply(features, λ)\n", " class_pred = self.class_classifier(features) # classify on regular features\n", " domain_pred = self.domain_classifier(features_grl) # classify on features after GRL\n", " return class_pred, domain_pred" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Wait, but why let $\\lambda$ change during training (e.g. every epoch)?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "- In the beginning of training, the domain loss is extremely noisy since the CNN features are not good yet.\n", "- We don't want to backprop domain confusion into the CNN layers in the beginning.\n", "- Therefore, lambda is gradulaly changed from 0 to 1 in the course of training.\n", " $$\n", " \\lambda_p = \\frac{2}{1+\\exp(-10\\cdot p)} -1,\n", " $$\n", " where $p\\in[0,1]$ is the training progress." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "scrolled": true, "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "source domain input: torch.Size([4, 1, 28, 28]) torch.Size([4])\n", "target domain input: torch.Size([4, 3, 28, 28]) torch.Size([4])\n", "yhat0_t_c:\n", " tensor([[-2.8836, -2.5810, -2.4243, -2.8905, -2.3797, -1.8645, -1.8780, -2.1869,\n", " -2.2756, -2.2180],\n", " [-2.4599, -3.1720, -2.8741, -2.2610, -1.7901, -1.7684, -2.3873, -2.5107,\n", " -2.1264, -2.5007],\n", " [-3.1181, -1.6503, -2.5382, -2.0966, -2.4471, -2.5786, -1.9930, -2.8338,\n", " -2.2675, -2.2946],\n", " [-2.6108, -2.6708, -1.8613, -2.0283, -2.4670, -3.1559, -2.5990, -2.2062,\n", " -2.5618, -1.7080]], grad_fn=) torch.Size([4, 10])\n", "yhat0_t_d:\n", " tensor([[-1.0760, -0.4170],\n", " [-0.7493, -0.6400],\n", " [-0.4547, -1.0069],\n", " [-0.4798, -0.9648]], grad_fn=) torch.Size([4, 2])\n" ] } ], "source": [ "model = DACNN()\n", "\n", "x0_s, y0_s = next(iter(dl_source))\n", "x0_t, y0_t = next(iter(dl_target))\n", "\n", "print('source domain input: ', x0_s.shape, y0_s.shape)\n", "print('target domain input: ', x0_t.shape, y0_t.shape)\n", "\n", "# Test that forward pass on both domains:\n", "# get class prediction and domain prediction\n", "yhat0_s_c, yhat0_s_d = model(x0_s)\n", "yhat0_t_c, yhat0_t_d = model(x0_t)\n", "\n", "print('yhat0_t_c:\\n', yhat0_t_c, yhat0_t_c.shape)\n", "print('yhat0_t_d:\\n', yhat0_t_d, yhat0_t_d.shape)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Training" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "import torch.optim as optim\n", "\n", "lr = 1e-3\n", "n_epochs = 1\n", "\n", "# Setup optimizer as usual\n", "model = DACNN()\n", "optimizer = optim.Adam(model.parameters(), lr)\n", "\n", "# Two loss functions this time (can generally be different)\n", "loss_fn_class = torch.nn.NLLLoss()\n", "loss_fn_domain = torch.nn.NLLLoss()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "batch_size = 256\n", "dl_source = torch.utils.data.DataLoader(ds_source, batch_size)\n", "dl_target = torch.utils.data.DataLoader(ds_target, batch_size)\n", "\n", "# We'll train the same number of batches from both datasets\n", "max_batches = min(len(dl_source), len(dl_target))" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "scrolled": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 0001 / 0001\n", "=================\n", "[1/231] class_loss: 2.3654 s_domain_loss: 0.8030 t_domain_loss: 0.6399 λ: 0.000 \n", "[2/231] class_loss: 2.0583 s_domain_loss: 0.7527 t_domain_loss: 0.6639 λ: 0.022 \n", "[3/231] class_loss: 1.9483 s_domain_loss: 0.7060 t_domain_loss: 0.7059 λ: 0.043 \n", "[4/231] class_loss: 1.7993 s_domain_loss: 0.6712 t_domain_loss: 0.7372 λ: 0.065 \n", "[5/231] class_loss: 1.6949 s_domain_loss: 0.6484 t_domain_loss: 0.7506 λ: 0.086 \n", "[6/231] class_loss: 1.5789 s_domain_loss: 0.6304 t_domain_loss: 0.7580 λ: 0.108 \n", "[7/231] class_loss: 1.4256 s_domain_loss: 0.6222 t_domain_loss: 0.7689 λ: 0.129 \n", "[8/231] class_loss: 1.3998 s_domain_loss: 0.6197 t_domain_loss: 0.7686 λ: 0.150 \n", "[9/231] class_loss: 1.2423 s_domain_loss: 0.6287 t_domain_loss: 0.7601 λ: 0.171 \n", "[10/231] class_loss: 1.1961 s_domain_loss: 0.6366 t_domain_loss: 0.7479 λ: 0.192 \n", "[11/231] class_loss: 1.1495 s_domain_loss: 0.6514 t_domain_loss: 0.7371 λ: 0.213 \n", "This is just a demo, stopping...\n" ] } ], "source": [ "for epoch_idx in range(n_epochs):\n", " print(f'Epoch {epoch_idx+1:04d} / {n_epochs:04d}', end='\\n=================\\n')\n", " dl_source_iter = iter(dl_source)\n", " dl_target_iter = iter(dl_target)\n", "\n", " for batch_idx in range(max_batches):\n", " # Calculate training progress and GRL λ\n", " p = float(batch_idx + epoch_idx * max_batches) / (n_epochs * max_batches)\n", " λ = 2. / (1. + np.exp(-10 * p)) - 1\n", "\n", " # === Train on source domain\n", " X_s, y_s = next(dl_source_iter)\n", " y_s_domain = torch.zeros(batch_size, dtype=torch.long) # generate source domain labels: 0\n", "\n", " class_pred, domain_pred = model(X_s, λ)\n", " loss_s_label = loss_fn_class(class_pred, y_s) # source classification loss\n", " loss_s_domain = loss_fn_domain(domain_pred, y_s_domain) # source domain loss (via GRL)\n", "\n", " # === Train on target domain\n", " X_t, _ = next(dl_target_iter) # Note: ignoring target domain class labels!\n", " y_t_domain = torch.ones(batch_size, dtype=torch.long) # generate target domain labels: 1\n", "\n", " _, domain_pred = model(X_t, λ)\n", " loss_t_domain = loss_fn_domain(domain_pred, y_t_domain) # target domain loss (via GRL)\n", " \n", " # === Optimize\n", " loss = loss_t_domain + loss_s_domain + loss_s_label\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", " \n", " print(f'[{batch_idx+1}/{max_batches}] '\n", " f'class_loss: {loss_s_label.item():.4f} ' f's_domain_loss: {loss_s_domain.item():.4f} '\n", " f't_domain_loss: {loss_t_domain.item():.4f} ' f'λ: {λ:.3f} '\n", " )\n", " if batch_idx == 10\n", " print('This is just a demo, stopping...')\n", " break" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Embeddings visualization" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "It's useful to visualize the space of the convolutional features learned by the model.\n", "\n", "Recall, our domain confusion loss was supposed to make images from both domains look the same for the classifier." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The figure shows t-SNE visualizations of the CNN’s activations (a) in case when no adaptation was performed and (b) in case when our adaptation procedure was incorporated into training. Blue points correspond to the source domain examples, while red ones correspond to the target domain.\n", "After adaptation, feature distributions are much more similar for the two domains." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Thanks!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "**Credits**\n", "\n", "This tutorial was written by [Aviv A. Rosenberg](https://avivr.net).
and Moshe Kimhi\n", "To re-use, please provide attribution and link to the original.\n", "\n", "Some images in this tutorial were taken and/or adapted from the following sources:\n", "\n", "- Pan & Yang, 2010, A Survey on Transfer Learning\n", "- C. Olah et al. 2017, Feature Visualization\n", "- Y. Ganin et al. 2015, Domain-Adversarial Training of Neural Networks \n", "- M. Wulfmeier et al., https://arxiv.org/abs/1703.01461v2\n", "- Sebastian Ruder, http://ruder.io/" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.6" }, "rise": { "scroll": true } }, "nbformat": 4, "nbformat_minor": 4 }