{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1czVdIlqnImH"
},
"source": [
"# Data Augmentation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1KD3ZgLs80vY"
},
"source": [
"### Goals\n",
"In this notebook you're going to build a generator that can be used to help create data to train a classifier. There are many cases where this might be useful. If you are interested in any of these topics, you are welcome to explore the linked papers and articles! \n",
"\n",
"- With smaller datasets, GANs can provide useful data augmentation that substantially [improve classifier performance](https://arxiv.org/abs/1711.04340). \n",
"- You have one type of data already labeled and would like to make predictions on [another related dataset for which you have no labels](https://www.nature.com/articles/s41598-019-52737-x). (You'll learn about the techniques for this use case in future notebooks!)\n",
"- You want to protect the privacy of the people who provided their information so you can provide access to a [generator instead of real data](https://www.ahajournals.org/doi/full/10.1161/CIRCOUTCOMES.118.005122). \n",
"- You have [input data with many missing values](https://arxiv.org/abs/1806.02920), where the input dimensions are correlated and you would like to train a model on complete inputs. \n",
"- You would like to be able to identify a real-world abnormal feature in an image [for the purpose of diagnosis](https://link.springer.com/chapter/10.1007/978-3-030-00946-5_11), but have limited access to real examples of the condition. \n",
"\n",
"In this assignment, you're going to be acting as a bug enthusiast — more on that later. \n",
"\n",
"### Learning Objectives\n",
"1. Understand some use cases for data augmentation and why GANs suit this task.\n",
"2. Implement a classifier that takes a mixed dataset of reals/fakes and analyze its accuracy."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "wU8DDM6l9rZb"
},
"source": [
"## Getting Started\n",
"\n",
"### Data Augmentation\n",
"Before you implement GAN-based data augmentation, you should know a bit about data augmentation in general, specifically for image datasets. It is [very common practice](https://arxiv.org/abs/1712.04621) to augment image-based datasets in ways that are appropriate for a given dataset. This may include having your dataloader randomly flipping images across their vertical axis, randomly cropping your image to a particular size, randomly adding a bit of noise or color to an image in ways that are true-to-life. \n",
"\n",
"In general, data augmentation helps to stop your model from overfitting to the data, and allows you to make small datasets many times larger. However, a sufficiently powerful classifier often still overfits to the original examples which is why GANs are particularly useful here. They can generate new images instead of simply modifying existing ones.\n",
"\n",
"### CIFAR\n",
"The [CIFAR-10 and CIFAR-100](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf) datasets are extremely widely used within machine learning -- they contain many thousands of “tiny” 32x32 color images of different classes representing relatively common real-world objects like airplanes and dogs, with 10 classes in CIFAR-10 and 100 classes in CIFAR-100. In CIFAR-100, there are 20 “superclasses” which each contain five classes. For example, the “fish” superclass contains “aquarium fish, flatfish, ray, shark, trout”. For the purposes of this assignment, you’ll be looking at a small subset of these images to simulate a small data regime, with only 40 images of each class for training.\n",
"\n",
"![alt text](CIFAR.png)\n",
"\n",
"### Initializations\n",
"You will begin by importing some useful libraries and packages and defining a visualization function that has been provided. You will also be re-using your conditional generator and functions code from earlier assignments. This will let you control what class of images to augment for your classifier."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "JfkorNJrnmNO"
},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn.functional as F\n",
"import matplotlib.pyplot as plt\n",
"from torch import nn\n",
"from tqdm.auto import tqdm\n",
"from torchvision import transforms\n",
"from torchvision.utils import make_grid\n",
"from torch.utils.data import DataLoader\n",
"torch.manual_seed(0) # Set for our testing purposes, please do not change!\n",
"\n",
"def show_tensor_images(image_tensor, num_images=25, size=(3, 32, 32), nrow=5, show=True):\n",
" '''\n",
" Function for visualizing images: Given a tensor of images, number of images, and\n",
" size per image, plots and prints the images in an uniform grid.\n",
" '''\n",
" image_tensor = (image_tensor + 1) / 2\n",
" image_unflat = image_tensor.detach().cpu()\n",
" image_grid = make_grid(image_unflat[:num_images], nrow=nrow)\n",
" plt.imshow(image_grid.permute(1, 2, 0).squeeze())\n",
" if show:\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "P1A1M6kpnfxw"
},
"source": [
"#### Generator"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "EvO7h0LYnEJZ"
},
"outputs": [],
"source": [
"class Generator(nn.Module):\n",
" '''\n",
" Generator Class\n",
" Values:\n",
" input_dim: the dimension of the input vector, a scalar\n",
" im_chan: the number of channels of the output image, a scalar\n",
" (CIFAR100 is in color (red, green, blue), so 3 is your default)\n",
" hidden_dim: the inner dimension, a scalar\n",
" '''\n",
" def __init__(self, input_dim=10, im_chan=3, hidden_dim=64):\n",
" super(Generator, self).__init__()\n",
" self.input_dim = input_dim\n",
" # Build the neural network\n",
" self.gen = nn.Sequential(\n",
" self.make_gen_block(input_dim, hidden_dim * 4, kernel_size=4),\n",
" self.make_gen_block(hidden_dim * 4, hidden_dim * 2, kernel_size=4, stride=1),\n",
" self.make_gen_block(hidden_dim * 2, hidden_dim, kernel_size=4),\n",
" self.make_gen_block(hidden_dim, im_chan, kernel_size=2, final_layer=True),\n",
" )\n",
"\n",
" def make_gen_block(self, input_channels, output_channels, kernel_size=3, stride=2, final_layer=False):\n",
" '''\n",
" Function to return a sequence of operations corresponding to a generator block of DCGAN;\n",
" a transposed convolution, a batchnorm (except in the final layer), and an activation.\n",
" Parameters:\n",
" input_channels: how many channels the input feature representation has\n",
" output_channels: how many channels the output feature representation should have\n",
" kernel_size: the size of each convolutional filter, equivalent to (kernel_size, kernel_size)\n",
" stride: the stride of the convolution\n",
" final_layer: a boolean, true if it is the final layer and false otherwise \n",
" (affects activation and batchnorm)\n",
" '''\n",
" if not final_layer:\n",
" return nn.Sequential(\n",
" nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),\n",
" nn.BatchNorm2d(output_channels),\n",
" nn.ReLU(inplace=True),\n",
" )\n",
" else:\n",
" return nn.Sequential(\n",
" nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),\n",
" nn.Tanh(),\n",
" )\n",
"\n",
" def forward(self, noise):\n",
" '''\n",
" Function for completing a forward pass of the generator: Given a noise tensor, \n",
" returns generated images.\n",
" Parameters:\n",
" noise: a noise tensor with dimensions (n_samples, input_dim)\n",
" '''\n",
" x = noise.view(len(noise), self.input_dim, 1, 1)\n",
" return self.gen(x)\n",
"\n",
"\n",
"def get_noise(n_samples, input_dim, device='cpu'):\n",
" '''\n",
" Function for creating noise vectors: Given the dimensions (n_samples, input_dim)\n",
" creates a tensor of that shape filled with random numbers from the normal distribution.\n",
" Parameters:\n",
" n_samples: the number of samples to generate, a scalar\n",
" input_dim: the dimension of the input vector, a scalar\n",
" device: the device type\n",
" '''\n",
" return torch.randn(n_samples, input_dim, device=device)\n",
"\n",
"def combine_vectors(x, y):\n",
" '''\n",
" Function for combining two vectors with shapes (n_samples, ?) and (n_samples, ?)\n",
" Parameters:\n",
" x: (n_samples, ?) the first vector. \n",
" In this assignment, this will be the noise vector of shape (n_samples, z_dim), \n",
" but you shouldn't need to know the second dimension's size.\n",
" y: (n_samples, ?) the second vector.\n",
" Once again, in this assignment this will be the one-hot class vector \n",
" with the shape (n_samples, n_classes), but you shouldn't assume this in your code.\n",
" '''\n",
" return torch.cat([x, y], 1)\n",
"\n",
"def get_one_hot_labels(labels, n_classes):\n",
" '''\n",
" Function for combining two vectors with shapes (n_samples, ?) and (n_samples, ?)\n",
" Parameters:\n",
" labels: (n_samples, 1) \n",
" n_classes: a single integer corresponding to the total number of classes in the dataset\n",
" '''\n",
" return F.one_hot(labels, n_classes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "qRk_8azSq3tF"
},
"source": [
"## Training\n",
"Now you can begin training your models.\n",
"First, you will define some new parameters:\n",
"\n",
"* cifar100_shape: the number of pixels in each CIFAR image, which has dimensions 32 x 32 and three channel (for red, green, and blue) so 3 x 32 x 32\n",
"* n_classes: the number of classes in CIFAR100 (e.g. airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "UpfJifVcmMhJ"
},
"outputs": [],
"source": [
"cifar100_shape = (3, 32, 32)\n",
"n_classes = 100"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "gJM9afuu0IuD"
},
"source": [
"And you also include the same parameters from previous assignments:\n",
"\n",
" * criterion: the loss function\n",
" * n_epochs: the number of times you iterate through the entire dataset when training\n",
" * z_dim: the dimension of the noise vector\n",
" * display_step: how often to display/visualize the images\n",
" * batch_size: the number of images per forward/backward pass\n",
" * lr: the learning rate\n",
" * device: the device type"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "sJlx2W71lUCv"
},
"outputs": [],
"source": [
"n_epochs = 10000\n",
"z_dim = 64\n",
"display_step = 500\n",
"batch_size = 64\n",
"lr = 0.0002\n",
"device = 'cuda'"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "jltxAMd00TRE"
},
"source": [
"Then, you want to set your generator's input dimension. Recall that for conditional GANs, the generator's input is the noise vector concatenated with the class vector."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "tuSOzzpwlXl7"
},
"outputs": [],
"source": [
"generator_input_dim = z_dim + n_classes"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ccQZRSYFXsHh"
},
"source": [
"#### Classifier\n",
"\n",
"For the classifier, you will use the same code that you wrote in an earlier assignment (the same as previous code for the discriminator as well since the discriminator is a real/fake classifier)."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "cVPxAjGSfYlX"
},
"outputs": [],
"source": [
"class Classifier(nn.Module):\n",
" '''\n",
" Classifier Class\n",
" Values:\n",
" im_chan: the number of channels of the output image, a scalar\n",
" n_classes: the total number of classes in the dataset, an integer scalar\n",
" hidden_dim: the inner dimension, a scalar\n",
" '''\n",
" def __init__(self, im_chan, n_classes, hidden_dim=32):\n",
" super(Classifier, self).__init__()\n",
" self.disc = nn.Sequential(\n",
" self.make_classifier_block(im_chan, hidden_dim),\n",
" self.make_classifier_block(hidden_dim, hidden_dim * 2),\n",
" self.make_classifier_block(hidden_dim * 2, hidden_dim * 4),\n",
" self.make_classifier_block(hidden_dim * 4, n_classes, final_layer=True),\n",
" )\n",
"\n",
" def make_classifier_block(self, input_channels, output_channels, kernel_size=3, stride=2, final_layer=False):\n",
" '''\n",
" Function to return a sequence of operations corresponding to a classifier block; \n",
" a convolution, a batchnorm (except in the final layer), and an activation (except in the final\n",
" Parameters:\n",
" input_channels: how many channels the input feature representation has\n",
" output_channels: how many channels the output feature representation should have\n",
" kernel_size: the size of each convolutional filter, equivalent to (kernel_size, kernel_size)\n",
" stride: the stride of the convolution\n",
" final_layer: a boolean, true if it is the final layer and false otherwise \n",
" (affects activation and batchnorm)\n",
" '''\n",
" if not final_layer:\n",
" return nn.Sequential(\n",
" nn.Conv2d(input_channels, output_channels, kernel_size, stride),\n",
" nn.BatchNorm2d(output_channels),\n",
" nn.LeakyReLU(0.2, inplace=True),\n",
" )\n",
" else:\n",
" return nn.Sequential(\n",
" nn.Conv2d(input_channels, output_channels, kernel_size, stride),\n",
" )\n",
"\n",
" def forward(self, image):\n",
" '''\n",
" Function for completing a forward pass of the classifier: Given an image tensor, \n",
" returns an n_classes-dimension tensor representing fake/real.\n",
" Parameters:\n",
" image: a flattened image tensor with im_chan channels\n",
" '''\n",
" class_pred = self.disc(image)\n",
" return class_pred.view(len(class_pred), -1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "tYXJTxM9pzZK"
},
"source": [
"#### Pre-training (Optional)\n",
"\n",
"You are provided the code to pre-train the models (GAN and classifier) given to you in this assignment. However, this is intended only for your personal curiosity -- for the assignment to run as intended, you should not use any checkpoints besides the ones given to you."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "UXptQZcwrBrq"
},
"outputs": [],
"source": [
"# This code is here for you to train your own generator or classifier \n",
"# outside the assignment on the full dataset if you'd like -- for the purposes \n",
"# of this assignment, please use the provided checkpoints\n",
"class Discriminator(nn.Module):\n",
" '''\n",
" Discriminator Class\n",
" Values:\n",
" im_chan: the number of channels of the output image, a scalar\n",
" (MNIST is black-and-white, so 1 channel is your default)\n",
" hidden_dim: the inner dimension, a scalar\n",
" '''\n",
" def __init__(self, im_chan=3, hidden_dim=64):\n",
" super(Discriminator, self).__init__()\n",
" self.disc = nn.Sequential(\n",
" self.make_disc_block(im_chan, hidden_dim, stride=1),\n",
" self.make_disc_block(hidden_dim, hidden_dim * 2),\n",
" self.make_disc_block(hidden_dim * 2, hidden_dim * 4),\n",
" self.make_disc_block(hidden_dim * 4, 1, final_layer=True),\n",
" )\n",
"\n",
" def make_disc_block(self, input_channels, output_channels, kernel_size=4, stride=2, final_layer=False):\n",
" '''\n",
" Function to return a sequence of operations corresponding to a discriminator block of the DCGAN; \n",
" a convolution, a batchnorm (except in the final layer), and an activation (except in the final layer).\n",
" Parameters:\n",
" input_channels: how many channels the input feature representation has\n",
" output_channels: how many channels the output feature representation should have\n",
" kernel_size: the size of each convolutional filter, equivalent to (kernel_size, kernel_size)\n",
" stride: the stride of the convolution\n",
" final_layer: a boolean, true if it is the final layer and false otherwise \n",
" (affects activation and batchnorm)\n",
" '''\n",
" if not final_layer:\n",
" return nn.Sequential(\n",
" nn.Conv2d(input_channels, output_channels, kernel_size, stride),\n",
" nn.BatchNorm2d(output_channels),\n",
" nn.LeakyReLU(0.2, inplace=True),\n",
" )\n",
" else:\n",
" return nn.Sequential(\n",
" nn.Conv2d(input_channels, output_channels, kernel_size, stride),\n",
" )\n",
"\n",
" def forward(self, image):\n",
" '''\n",
" Function for completing a forward pass of the discriminator: Given an image tensor, \n",
" returns a 1-dimension tensor representing fake/real.\n",
" Parameters:\n",
" image: a flattened image tensor with dimension (im_chan)\n",
" '''\n",
" disc_pred = self.disc(image)\n",
" return disc_pred.view(len(disc_pred), -1)\n",
"\n",
"def train_generator():\n",
" gen = Generator(generator_input_dim).to(device)\n",
" gen_opt = torch.optim.Adam(gen.parameters(), lr=lr)\n",
" discriminator_input_dim = cifar100_shape[0] + n_classes\n",
" disc = Discriminator(discriminator_input_dim).to(device)\n",
" disc_opt = torch.optim.Adam(disc.parameters(), lr=lr)\n",
"\n",
" def weights_init(m):\n",
" if isinstance(m, nn.Conv2d) or isinstance(m, nn.ConvTranspose2d):\n",
" torch.nn.init.normal_(m.weight, 0.0, 0.02)\n",
" if isinstance(m, nn.BatchNorm2d):\n",
" torch.nn.init.normal_(m.weight, 0.0, 0.02)\n",
" torch.nn.init.constant_(m.bias, 0)\n",
" gen = gen.apply(weights_init)\n",
" disc = disc.apply(weights_init)\n",
"\n",
" criterion = nn.BCEWithLogitsLoss()\n",
" cur_step = 0\n",
" mean_generator_loss = 0\n",
" mean_discriminator_loss = 0\n",
" for epoch in range(n_epochs):\n",
" # Dataloader returns the batches and the labels\n",
" for real, labels in dataloader:\n",
" cur_batch_size = len(real)\n",
" # Flatten the batch of real images from the dataset\n",
" real = real.to(device)\n",
"\n",
" # Convert the labels from the dataloader into one-hot versions of those labels\n",
" one_hot_labels = get_one_hot_labels(labels.to(device), n_classes).float()\n",
"\n",
" image_one_hot_labels = one_hot_labels[:, :, None, None]\n",
" image_one_hot_labels = image_one_hot_labels.repeat(1, 1, cifar100_shape[1], cifar100_shape[2])\n",
"\n",
" ### Update discriminator ###\n",
" # Zero out the discriminator gradients\n",
" disc_opt.zero_grad()\n",
" # Get noise corresponding to the current batch_size \n",
" fake_noise = get_noise(cur_batch_size, z_dim, device=device)\n",
" \n",
" # Combine the vectors of the noise and the one-hot labels for the generator\n",
" noise_and_labels = combine_vectors(fake_noise, one_hot_labels)\n",
" fake = gen(noise_and_labels)\n",
" # Combine the vectors of the images and the one-hot labels for the discriminator\n",
" fake_image_and_labels = combine_vectors(fake.detach(), image_one_hot_labels)\n",
" real_image_and_labels = combine_vectors(real, image_one_hot_labels)\n",
" disc_fake_pred = disc(fake_image_and_labels)\n",
" disc_real_pred = disc(real_image_and_labels)\n",
"\n",
" disc_fake_loss = criterion(disc_fake_pred, torch.zeros_like(disc_fake_pred))\n",
" disc_real_loss = criterion(disc_real_pred, torch.ones_like(disc_real_pred))\n",
" disc_loss = (disc_fake_loss + disc_real_loss) / 2\n",
" disc_loss.backward(retain_graph=True)\n",
" disc_opt.step() \n",
"\n",
" # Keep track of the average discriminator loss\n",
" mean_discriminator_loss += disc_loss.item() / display_step\n",
"\n",
" ### Update generator ###\n",
" # Zero out the generator gradients\n",
" gen_opt.zero_grad()\n",
"\n",
" # Pass the discriminator the combination of the fake images and the one-hot labels\n",
" fake_image_and_labels = combine_vectors(fake, image_one_hot_labels)\n",
"\n",
" disc_fake_pred = disc(fake_image_and_labels)\n",
" gen_loss = criterion(disc_fake_pred, torch.ones_like(disc_fake_pred))\n",
" gen_loss.backward()\n",
" gen_opt.step()\n",
"\n",
" # Keep track of the average generator loss\n",
" mean_generator_loss += gen_loss.item() / display_step\n",
"\n",
" if cur_step % display_step == 0 and cur_step > 0:\n",
" print(f\"Step {cur_step}: Generator loss: {mean_generator_loss}, discriminator loss: {mean_discriminator_loss}\")\n",
" show_tensor_images(fake)\n",
" show_tensor_images(real)\n",
" mean_generator_loss = 0\n",
" mean_discriminator_loss = 0\n",
" cur_step += 1\n",
"\n",
"def train_classifier():\n",
" criterion = nn.CrossEntropyLoss()\n",
" n_epochs = 10\n",
"\n",
" validation_dataloader = DataLoader(\n",
" CIFAR100(\".\", train=False, download=True, transform=transform),\n",
" batch_size=batch_size)\n",
"\n",
" display_step = 10\n",
" batch_size = 512\n",
" lr = 0.0002\n",
" device = 'cuda'\n",
" classifier = Classifier(cifar100_shape[0], n_classes).to(device)\n",
" classifier_opt = torch.optim.Adam(classifier.parameters(), lr=lr)\n",
" cur_step = 0\n",
" for epoch in range(n_epochs):\n",
" for real, labels in tqdm(dataloader):\n",
" cur_batch_size = len(real)\n",
" real = real.to(device)\n",
" labels = labels.to(device)\n",
"\n",
" ### Update classifier ###\n",
" # Get noise corresponding to the current batch_size\n",
" classifier_opt.zero_grad()\n",
" labels_hat = classifier(real.detach())\n",
" classifier_loss = criterion(labels_hat, labels)\n",
" classifier_loss.backward()\n",
" classifier_opt.step()\n",
"\n",
" if cur_step % display_step == 0:\n",
" classifier_val_loss = 0\n",
" classifier_correct = 0\n",
" num_validation = 0\n",
" for val_example, val_label in validation_dataloader:\n",
" cur_batch_size = len(val_example)\n",
" num_validation += cur_batch_size\n",
" val_example = val_example.to(device)\n",
" val_label = val_label.to(device)\n",
" labels_hat = classifier(val_example)\n",
" classifier_val_loss += criterion(labels_hat, val_label) * cur_batch_size\n",
" classifier_correct += (labels_hat.argmax(1) == val_label).float().sum()\n",
"\n",
" print(f\"Step {cur_step}: \"\n",
" f\"Classifier loss: {classifier_val_loss.item() / num_validation}, \"\n",
" f\"classifier accuracy: {classifier_correct.item() / num_validation}\")\n",
" cur_step += 1\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ZYGOiy-xWHOH"
},
"source": [
"## Tuning the Classifier\n",
"After two courses, you've probably had some fun debugging your GANs and have started to consider yourself a bug master. For this assignment, your mastery will be put to the test on some interesting bugs... well, bugs as in insects.\n",
"\n",
"As a bug master, you want a classifier capable of classifying different species of bugs: bees, beetles, butterflies, caterpillar, and more. Luckily, you found a great dataset with a lot of animal species and objects, and you trained your classifier on that.\n",
"\n",
"But the bug classes don't do as well as you would like. Now your plan is to train a GAN on the same data so it can generate new bugs to make your classifier better at distinguishing between all of your favorite bugs!\n",
"\n",
"You will fine-tune your model by augmenting the original real data with fake data and during that process, observe how to increase the accuracy of your classifier with these fake, GAN-generated bugs. After this, you will prove your worth as a bug master."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "oSuAJTuYYr2o"
},
"source": [
"#### Sampling Ratio\n",
"\n",
"Suppose that you've decided that although you have this pre-trained general generator and this general classifier, capable of identifying 100 classes with some accuracy (~17%), what you'd really like is a model that can classify the five different kinds of bugs in the dataset. You'll fine-tune your model by augmenting your data with the generated images. Keep in mind that both the generator and the classifier were trained on the same images: the 40 images per class you painstakingly found so your generator may not be great. This is the caveat with data augmentation, ultimately you are still bound by the real data that you have but you want to try and create more. To make your models even better, you would need to take some more bug photos, label them, and add them to your training set and/or use higher quality photos.\n",
"\n",
"To start, you'll first need to write some code to sample a combination of real and generated images. Given a probability, `p_real`, you'll need to generate a combined tensor where roughly `p_real` of the returned images are sampled from the real images. Note that you should not interpolate the images here: you should choose each image from the real or fake set with a given probability. For example, if your real images are a tensor of `[[1, 2, 3, 4, 5]]` and your fake images are a tensor of `[[-1, -2, -3, -4, -5]]`, and `p_real = 0.2`, two potential random return values are `[[1, -2, 3, -4, -5]]` or `[[-1, 2, -3, -4, -5]]`. \n",
"\n",
"\n",
"Notice that `p_real = 0.2` does not guarantee that exactly 20% of the samples are real, just that when choosing an image for the combined set, there is a 20% probability that that image will be chosen from the real images, and an 80% probability that it will be selected from the fake images.\n",
"\n",
"In addition, we will expect the images to remain in the same order to maintain their alignment with their labels (this applies to the fake images too!). \n",
"\n",
"\n",
"\n",
"\n",
"Optional hints for combine_sample\n",
"\n",
"\n",
"\n",
"1. This code probably shouldn't be much longer than 3 lines\n",
"2. You can index using a set of booleans which have the same length as your tensor\n",
"3. You want to generate an unbiased sample, which you can do (for example) with `torch.rand(length_reals) > p`.\n",
"4. There are many approaches here that will give a correct answer here. You may find [`torch.rand`](https://pytorch.org/docs/stable/generated/torch.rand.html) or [`torch.bernoulli`](https://pytorch.org/docs/master/generated/torch.bernoulli.html) useful. \n",
"5. You don't want to edit an argument in place, so you may find [`cur_tensor.clone()`](https://pytorch.org/docs/stable/tensors.html) useful too, which makes a copy of `cur_tensor`. \n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "16JJ7RlKxrsY"
},
"outputs": [],
"source": [
"# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
"# GRADED FUNCTION: combine_sample\n",
"def combine_sample(real, fake, p_real):\n",
" '''\n",
" Function to take a set of real and fake images of the same length (x)\n",
" and produce a combined tensor with length (x) and sampled at the target probability\n",
" Parameters:\n",
" real: a tensor of real images, length (x)\n",
" fake: a tensor of fake images, length (x)\n",
" p_real: the probability the images are sampled from the real set\n",
" '''\n",
" #### START CODE HERE ####\n",
" make_fake = torch.rand(len(real)) > p_real\n",
" target_images = real.clone()\n",
" target_images[make_fake] = fake[make_fake]\n",
" #### END CODE HERE ####\n",
" return target_images"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "1kDmOc81zJGN"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Success!\n"
]
}
],
"source": [
"n_test_samples = 9999\n",
"test_combination = combine_sample(\n",
" torch.ones(n_test_samples, 1), \n",
" torch.zeros(n_test_samples, 1), \n",
" 0.3\n",
")\n",
"# Check that the shape is right\n",
"assert tuple(test_combination.shape) == (n_test_samples, 1)\n",
"# Check that the ratio is right\n",
"assert torch.abs(test_combination.mean() - 0.3) < 0.05\n",
"# Make sure that no mixing happened\n",
"assert test_combination.median() < 1e-5\n",
"\n",
"test_combination = combine_sample(\n",
" torch.ones(n_test_samples, 10, 10), \n",
" torch.zeros(n_test_samples, 10, 10), \n",
" 0.8\n",
")\n",
"# Check that the shape is right\n",
"assert tuple(test_combination.shape) == (n_test_samples, 10, 10)\n",
"# Make sure that no mixing happened\n",
"assert torch.abs((test_combination.sum([1, 2]).median()) - 100) < 1e-5\n",
"\n",
"test_reals = torch.arange(n_test_samples)[:, None].float()\n",
"test_fakes = torch.zeros(n_test_samples, 1)\n",
"test_saved = (test_reals.clone(), test_fakes.clone())\n",
"test_combination = combine_sample(test_reals, test_fakes, 0.3)\n",
"# Make sure that the sample isn't biased\n",
"assert torch.abs((test_combination.mean() - 1500)) < 100\n",
"# Make sure no inputs were changed\n",
"assert torch.abs(test_saved[0] - test_reals).sum() < 1e-3\n",
"assert torch.abs(test_saved[1] - test_fakes).sum() < 1e-3\n",
"\n",
"test_fakes = torch.arange(n_test_samples)[:, None].float()\n",
"test_combination = combine_sample(test_reals, test_fakes, 0.3)\n",
"# Make sure that the order is maintained\n",
"assert torch.abs(test_combination - test_reals).sum() < 1e-4\n",
"if torch.cuda.is_available():\n",
" # Check that the solution matches the input device\n",
" assert str(combine_sample(\n",
" torch.ones(n_test_samples, 10, 10).cuda(), \n",
" torch.zeros(n_test_samples, 10, 10).cuda(),\n",
" 0.8\n",
" ).device).startswith(\"cuda\")\n",
"print(\"Success!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "LpMGXMYU1a4O"
},
"source": [
"Now you have a challenge: find a `p_real` and a generator image such that your classifier gets an average of a 51% accuracy or higher on the insects, when evaluated with the `eval_augmentation` function. **You'll need to fill in `find_optimal` to find these parameters to solve this part!** Note that if your answer takes a very long time to run, you may need to hard-code the solution it finds. \n",
"\n",
"When you're training a generator, you will often have to look at different checkpoints and choose one that does the best (either empirically or using some evaluation method). Here, you are given four generator checkpoints: `gen_1.pt`, `gen_2.pt`, `gen_3.pt`, `gen_4.pt`. You'll also have some scratch area to write whatever code you'd like to solve this problem, but you must return a `p_real` and an image name of your selected generator checkpoint. You can hard-code/brute-force these numbers if you would like, but you are encouraged to try to solve this problem in a more general way. In practice, you would also want a test set (since it is possible to overfit on a validation set), but for simplicity you can just focus on the validation set."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Fc7mFIVRVT_2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your model had an accuracy of 51.9%\n",
"Success!\n"
]
}
],
"source": [
"# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
"# GRADED FUNCTION: find_optimal\n",
"def find_optimal():\n",
" # In the following section, you can write the code to choose your optimal answer\n",
" # You can even use the eval_augmentation function in your code if you'd like!\n",
" gen_names = [\n",
" \"gen_1.pt\",\n",
" \"gen_2.pt\",\n",
" \"gen_3.pt\",\n",
" \"gen_4.pt\"\n",
" ]\n",
"\n",
" #### START CODE HERE #### \n",
" best_p_real, best_gen_name = 0.6, \"gen_4.pt\"\n",
" #### END CODE HERE ####\n",
" return best_p_real, best_gen_name\n",
"\n",
"def augmented_train(p_real, gen_name):\n",
" gen = Generator(generator_input_dim).to(device)\n",
" gen.load_state_dict(torch.load(gen_name))\n",
"\n",
" classifier = Classifier(cifar100_shape[0], n_classes).to(device)\n",
" classifier.load_state_dict(torch.load(\"class.pt\"))\n",
" criterion = nn.CrossEntropyLoss()\n",
" batch_size = 256\n",
"\n",
" train_set = torch.load(\"insect_train.pt\")\n",
" val_set = torch.load(\"insect_val.pt\")\n",
" dataloader = DataLoader(\n",
" torch.utils.data.TensorDataset(train_set[\"images\"], train_set[\"labels\"]),\n",
" batch_size=batch_size,\n",
" shuffle=True\n",
" )\n",
" validation_dataloader = DataLoader(\n",
" torch.utils.data.TensorDataset(val_set[\"images\"], val_set[\"labels\"]),\n",
" batch_size=batch_size\n",
" )\n",
"\n",
" display_step = 1\n",
" lr = 0.0002\n",
" n_epochs = 20\n",
" classifier_opt = torch.optim.Adam(classifier.parameters(), lr=lr)\n",
" cur_step = 0\n",
" best_score = 0\n",
" for epoch in range(n_epochs):\n",
" for real, labels in dataloader:\n",
" real = real.to(device)\n",
" # Flatten the image\n",
" labels = labels.to(device)\n",
" one_hot_labels = get_one_hot_labels(labels.to(device), n_classes).float()\n",
"\n",
" ### Update classifier ###\n",
" # Get noise corresponding to the current batch_size\n",
" classifier_opt.zero_grad()\n",
" cur_batch_size = len(labels)\n",
" fake_noise = get_noise(cur_batch_size, z_dim, device=device)\n",
" noise_and_labels = combine_vectors(fake_noise, one_hot_labels)\n",
" fake = gen(noise_and_labels)\n",
"\n",
" target_images = combine_sample(real.clone(), fake.clone(), p_real)\n",
" labels_hat = classifier(target_images.detach())\n",
" classifier_loss = criterion(labels_hat, labels)\n",
" classifier_loss.backward()\n",
" classifier_opt.step()\n",
"\n",
" # Calculate the accuracy on the validation set\n",
" if cur_step % display_step == 0 and cur_step > 0:\n",
" classifier_val_loss = 0\n",
" classifier_correct = 0\n",
" num_validation = 0\n",
" with torch.no_grad():\n",
" for val_example, val_label in validation_dataloader:\n",
" cur_batch_size = len(val_example)\n",
" num_validation += cur_batch_size\n",
" val_example = val_example.to(device)\n",
" val_label = val_label.to(device)\n",
" labels_hat = classifier(val_example)\n",
" classifier_val_loss += criterion(labels_hat, val_label) * cur_batch_size\n",
" classifier_correct += (labels_hat.argmax(1) == val_label).float().sum()\n",
" accuracy = classifier_correct.item() / num_validation\n",
" if accuracy > best_score:\n",
" best_score = accuracy\n",
" cur_step += 1\n",
" return best_score\n",
"\n",
"def eval_augmentation(p_real, gen_name, n_test=20):\n",
" total = 0\n",
" for i in range(n_test):\n",
" total += augmented_train(p_real, gen_name)\n",
" return total / n_test\n",
"\n",
"best_p_real, best_gen_name = find_optimal()\n",
"performance = eval_augmentation(best_p_real, best_gen_name)\n",
"print(f\"Your model had an accuracy of {performance:0.1%}\")\n",
"assert performance > 0.512\n",
"print(\"Success!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "mmqeeBjE32ls"
},
"source": [
"You'll likely find that the worst performance is when the generator is performing alone: this corresponds to the case where you might be trying to hide the underlying examples from the classifier. Perhaps you don't want other people to know about your specific bugs!"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "aLRFjtb_HEuP"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f3ec21e2eeae4e6dbcf11405ee893caf",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(FloatProgress(value=0.0, max=21.0), HTML(value='')))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsd0lEQVR4nO3dd3xV9f3H8dcnIey9916ykQgoVbGOqlVwolJUrIoDtXVVq9U6On5atbZua9UqCihWRGvd4mSFFUCGYQdEwgojZH9+f9yDTWmAC+Tm3Ju8n4/HfeSece/5nIz7yfd8z/fzNXdHRERkb0lhByAiIvFJCUJEREqlBCEiIqVSghARkVIpQYiISKmqhB1AWWncuLG3b98+7DBERBLK7NmzN7l7k9K2VZgE0b59e9LS0sIOQ0QkoZjZ6n1t0yUmEREplRKEiIiUSglCRERKpQQhIiKlUoIQEZFSKUGIiEiplCBERKRUShAiEjfyC4uZPHcdM1ZsDjsUoQINlBORxJVbUMTEWWt59vMVrNu2m+Qk4//O6c35qW3CDq1SU4IQkdDszCvklemr+dsXK9m0M48B7Rpw95k9eHnaam6dlM7323MZe0JnzCzsUCslJQgRKXfbcvJ58etVvPDVKrJ3F/Cjzo0Ze0J/BndsiJlxQrem/GrSfB76YBnfb8/jnmE9SU5SkihvMU0QZnYq8BcgGXjO3f9vr+2jgT8B64JVj7v7c2bWD3gKqAsUAb9394mxjFVE/mN3fhFpq7fQtmFN2jasWWb/wWftyOO5L1cwbtpqduUXcdIRzRh7Qif6t23wX/tVrZLEIyP60axedZ75bAUbd+Tylwv7Uz0luUzikOjELEGYWTLwBHAykAnMMrMp7v7NXrtOdPfr9lqXA1zi7t+aWUtgtpm97+7bYhWviMDyrJ28Mn0Nk2avZXtuIQD1aqTQp3W94FGfvq3r07xe9YN633XbdvPsZ8uZMGst+UXF/LR3C8ae0JkjWtTd52uSkoxfn3YEzetW5753vmHUczN47tJU6teseljnKNGLZQtiIJDh7isAzGwCMBzYO0H8D3dfVuL5ejPbCDQBtsUmVJHKq6ComI+++Z5xM1bzVcZmUpKN03q14Kz+LdmQnUd65jbSM7N5+rMVFBU7AE3rVPshYez52rDW/35wr9q0i6emLuefczNxh7P7t+KaoZ3o2KR21PFdNqQDTepU46aJ8znv6Wn84+cDaVW/Rpmdv+xbLBNEK2BtieVMYFAp+51rZscBy4Ab3b3kazCzgUBVYHmsAhWpjDZk5zJ+5hrGz1zDxh15tKpfg1t/0o0RqW1oUqfaD/uNHNQWiNxptGj99h8SRnrmNj5avPGH/Vo3qEHfIGF0alKbKfPX8076eqokJ3HRwLaMOa4jrRvUPKRYz+jTksa1q3HlS2mc8+RXvHjZwP22PqRsmLvH5o3NzgNOdfcrguWLgUElLyeZWSNgp7vnmdlVwAXu/uMS21sAU4FL3X16KccYA4wBaNu27YDVq/dZ1lxEAHfnq4zNjJu+mg8Xf0+xO8d3bcKoQe04oXvTg+4I3pFbwIJ12T8kjPlrs1m3bTcANasmc/Hgdlx+bAea1jm4S1L7smTDdkY/P4tdeYU8e0kqR3dqVCbvW5mZ2Wx3Ty11WwwTxNHAPe7+k2D51wDu/sd97J8MbHH3esFyXSLJ4Q/uPulAx0tNTXVNGCRSuuycAibNyeSV6atZsWkXDWqmMCK1DSMHtaVdo1pleqzNO/NYumEHR7SoS4NSLjsdrvXbdnPp8zNZvTmHh0f05cy+Lcv8GJXJ/hJELC8xzQK6mFkHIncpXQiM3CuwFu7+XbA4DFgcrK8KvAm8FE1yEJHSpWduY9z01UyZv57cgmKObFufR0b05fTeLWJ2R1Cj2tU4pnO1A+94iFrWr8Gkq4/hypfSuH78XDbuyOPyH3WI2fEqs5glCHcvNLPrgPeJ3Ob6vLsvMrP7gDR3nwLcYGbDgEJgCzA6ePkI4DigUXArLMBod58Xq3hFKpItu/L5zeQFvLtgAzVSkjm7f2tGDW5Lz5b1wg6tTNSrmcJLlw/klxPmcf873/D99lxuP7U7SRorUaZidompvOkSk0jEx4u/57Y3FpC9O58bftyFS4e0p271lLDDiomiYufetxfx0rTVDO/Xkj+d15eqVVRi7mCEdYlJRMrRzrxCfvfON0yYtZbuzevw8uUV/06f5CTj3mE9aV6vOg++t5RNO/N4etQA6lTQhFjelCBEKoAZKzZz8+vzWb9tN9cO7cQvTupCtSqVY9SxmXHt0M40q1Od295I58zHvuSyIR04+8hWFbblVF50iUkkgeUWFPHwB0t57suVtG1Yk0dG9GVAu4ZhhxWarzI28cB7S0jPzKZm1WTO6t+KUYPa0aNlxW5JHY5QbnMtb0oQUtksXJfNjRPn8e3GnYwa3JZfn3YEtarpogDA/LX/uXsrr7CYAe0aMGpwW07rFbu7txKVEoRIBVJYVMxTU5fzl4+/pVHtqjx4Xl+O79ok7LDi0racfCbNzuSVGWtYuWkXDWtVZURqG342qC1tGh7aqO6KRglCpIJYnrWTm16bz/y12xjWtyX3De+p4nVRKC52vl6+mZenr+KjxRspdmdo1yaMGtyOod0OfgR5RaIEIZLgioudl6ev5o//Xkz1lGR+d1YvzuijEcSH4rvs3YyfuZYJJWpQjRzUlguOakPj2v87wK+42NmZX0h2TgHbcgrI3l3Att35ka/BcnZOAXmFRVxxbEd6tUqssSZKECIJbP223dw6aT5fZWxmaLcmPHBuH5rVLZvaRpVZQVExH37zPS9PW820FZEqtj/q3Jhih227C8jOiSSB7N0FFO/nY7JqlSTq10ght6CI/KJiHjq/b0Ilb42DEElA7s7rszO5/51vKCp2/nhOby48qo2m3ywjKclJnN67Baf3bkHGxh2Mm76GLzM2UatqMvVqVqVtw5rUr5FCvRop1K+ZQt0aKdSvkUL9mlV/WFevRsoPnd5ZO/K4etxsrnt1Lss27OCXJ3VN+JHdakGIxKGMjTu4882FzFi5haPaN+Dh8/vRtpE6VeNdXmERv3lzIa/PzuQnPZvxyIh+cX9nmVoQIgkit6CIxz/J4JnPl1OzahX+75zejEhtk/D/iVYW1aok8+B5fejWvA5/eHcx5z71Nc9dmnrI82CETQlCJE5MXbqRu99axJotOZxzZCvuPP0IGpXSaSrxzcy44tiOdGlWh+tencPwx7/i6YsHcFT7xBvAqKpWIiH7fnsuY1+dw+gXZlEl2Xj1ykE8MqKfkkOCO75rEyaPHULdGimM/Nt0Js5aE3ZIB00tCJGQFBU7L09bxUMfLCO/qJibT+7KmOM7VpoaSpVBpya1mXztEK4bP4fb3ljA0g07ueP07lRJToz/zZUgREKwIDObO95cwIJ12RzbpTH3D+9F+8ZlO7ObxId6NVN4YfRR/P7dxTz/1Uoysnby2EX9qVcj/gsJKkGIlKMduQU8/MEyXpq2ika1q/HYRf05o08L3bpawVVJTuK3Z/ake/M6/GbyQs5+4iueuzSVjk1qhx3afilBiJQDd+fdBRu49+1FZO3M4+LB7bj5lG4J8V+klJ0LjmpLh8a1uWbcbM564iseH3kkx8VxHa3EuBAmksDWbsnhshdnMfbVOTSpU43J1w7hvuG9lBwqqYEdGvLWdUNoWb8Go1+YyfNfriRex6OpBSESQ8uzdnLOk19TWFTM3Wf04JKj2yVMB6XETusGNXnjmmO46bV53PfONyzdsIP7z+oVd9OlKkGIxMiWXfn8/MVZVEky3hp7rDqh5b/UqlaFp342gEc/WsZfP8kgp6CIv17YL676o5QgRGIgr7CIq15O47vsXMZfOVjJQUqVlGTcdEo3qlZJ4qEPlnFsl8aMSG0Tdlg/iK/2jEgF4O78alI6s1Zt5eHz+zKgXYOwQ5I4d83QzhzdsRG/fWsRGRt3hh3OD5QgRMrYox99y1vz1nPLKV05s2/ilH2W8CQnGY9e2I/qKUlcP34uuQVFYYcEKEGIlKk352byl4+/5bwBrRl7Queww5EE0qxudR46vy+Lv9vOA+8tCTscQAlCpMzMWLGZX01K5+iOjfjD2b3jqrNREsOJRzRj9DHteeGrVXy8+Puww1GCECkLKzft4qpxs2nTsCZPjxoQd7crSuK4/bTuHNGiLrdOSuf77bmhxqLfYpHDtHVXPpe9MJMkM14YfRT1amoAnBy66inJPHZRf3bnF/HLCfMo2t98pzGmBCFyGCK3s85m/bZcnr14AO0a6XZWOXydm9bm3mE9mbZiM09/tjy0OJQgRA6Ru3P7GwuYuWoLfzq/D6kJOCGMxK/zU1tzRp8WPPLhMuas2RpKDEoQIoforx9n8Obcddx0cleG92sVdjhSwZgZfzinNy3qVeeG8XPZnltQ7jEoQYgcgslz1/Hnj5ZxzpGtuP7Hup1VYqNu9RT+elF/vsvO5Y5/Lij3on5KECIHaebKLfxqUjoDOzTkj+fodlaJrSPbNuCmk7vyTvp3vJ6WWa7HVoIQOQirNu3iqpfTaNWgBs+MGqDpQaVcXH18J47p1IjfTinfUhwxTRBmdqqZLTWzDDO7vZTto80sy8zmBY8rSmy71My+DR6XxjJOkWhsy4lUZ3Xg+dFH0aBW1bBDkkoiOcn48wX9qFE1uVxLccQsQZhZMvAEcBrQA7jIzHqUsutEd+8XPJ4LXtsQ+C0wCBgI/NbMVPFMQpNfWMxVL88mc+tunr04lQ6qzirlLFKKo0+5luKIZQtiIJDh7ivcPR+YAAyP8rU/AT509y3uvhX4EDg1RnGK7FdBUTG3v5HOjJVbePC8PgzsoNtZJRw/7t6My4aUXymOWCaIVsDaEsuZwbq9nWtm6WY2ycz2FEKP6rVmNsbM0swsLSsrq6ziFvnB7NVbOfOxL/nn3HXceFJXzuqv21klXLef1p0eLepyy+vzY16KI+xO6reB9u7eh0gr4R8H82J3f9bdU909tUmT+J34WxJPdk4Bd7y5gPOe/prs3QU8PWoAN5yo21klfNWqJPPYyP7kFhTHvBRHLBPEOqDk1Eitg3U/cPfN7p4XLD4HDIj2tSKx4O68OTeTEx+ZyoSZa/j5kA58eNPxnNqruW5nlbjRqUlt7h0e+1IcsZxydBbQxcw6EPlwvxAYWXIHM2vh7t8Fi8OAxcHz94E/lOiYPgX4dQxjFWF51k7umryQr5dvpm+b+rx42UB6taoXdlgipTp/QGu++HYTj3y4jMEdG8Vk5sKYJQh3LzSz64h82CcDz7v7IjO7D0hz9ynADWY2DCgEtgCjg9duMbP7iSQZgPvcfUusYpXKLbegiCenLufpqcuplpLE/Wf1YuTAtiQnqcUg8cvM+P3ZvZi7Ziu3v5HO+788jqQy/p218h66HSupqamelpYWdhiSYL78dhN3vbWQlZt2MbxfS+786RE0rVM97LBEorYgM5vqKUl0aVbnkF5vZrPdPbW0bbG8xCQStzbuyOX3/1rMW/PW075RTV6+fCDHdtGNDpJ4ereO3WVQJQipVIqLnVdmruHB95aQV1DML07swjVDO1E9RSUzRPamBCGVxqL12dz55kLmrd3GMZ0acf9ZvejUpHbYYYnELSUIqfAyNu7kyakZvDVvPfVrpPDnC/pyVr9Wum1V5ACUIKTCWrQ+myc+zeDfCzdQvUoyo49pzw0/7qI5o0WipAQhFc7s1Vt54tMMPlmykTrVqjB2aGcuG9KeRrWrhR2aSEJRgpAKwd35evlmHv8kg2krNtOgZgq3nNKVi49uT70aajGIHAolCElo7s4nSzby2CcZzFu7jaZ1qvGbnx7ByEFtqVlVv94ih0N/QZKQioqdfy/8jic+Xc7i77bTukENfndWL84b0Fq3rIqUESUISSgFRcVMnruOpz5bzoqsXXRqUouHz+/LsH4tSUkOuzixSMWiBCEJY9ryzdzy+nzWbdvNES3q8sTIIzm1V3PVTBKJESUISQjbcwv4xYS51KyazPOjUzmhW1ONYxCJMSUISQgPvreETTvzmDx2CH1a1w87HJFKQRdtJe6lrdrCuOlruGxIByUHkXKkBCFxLb+wmF//cwGt6tfgppO7hh2OSKWiS0wS1575bDnfbtzJC6OPolY1/bqKlCe1ICRuLc/ayWOfZPDTPi04oXvTsMMRqXSUICQuuTt3/HMB1VKS+O2ZPcIOR6RSUoKQuPR6WiYzVm7hjtM1BahIWJQgJO5k7cjj9+8uZmD7hlyQ2ibscEQqLSUIiTv3v/MNu/OL+MM5vUjSKGmR0ChBSFz5dOlGpsxfz7UndKJz0zphhyNSqSlBSNzIyS/kN28upFOTWlwztFPY4YhUerqxXOLGnz9cxrptu3ntqqOpVkUlu0XCphaExIWF67L5+5cruWhgWwZ2aBh2OCJCFAnCzM40MyUSiZnCokg5jUa1q3H7ad3DDkdEAtF88F8AfGtmD5qZ/nqlzL349SoWrMvmnjN7av5okThywATh7qOA/sBy4EUzm2ZmY8xMt5jIYVu7JYeHP1jGid2bcnrv5mGHIyIlRHXpyN23A5OACUAL4GxgjpldH8PYpIJzd+56ayFmcN9ZvTQBkEiciaYPYpiZvQlMBVKAge5+GtAXuDm24UlF9k76d0xdmsUtp3SjVf0aYYcjInuJ5jbXc4E/u/vnJVe6e46ZXR6bsKSiy84p4N63F9GndT0uPaZ92OGISCmiSRD3AN/tWTCzGkAzd1/l7h/HKjCp2P7478VszSngHz8fSLLKaYjEpWj6IF4HikssFwXrRA7J9BWbmTBrLVcc24GeLeuFHY6I7EM0CaKKu+fvWQieV43mzc3sVDNbamYZZnb7fvY718zczFKD5RQz+4eZLTCzxWb262iOJ/Evt6CIO95cQJuGNfjliZpCVCSeRZMgssxs2J4FMxsObDrQi8wsGXgCOA3oAVxkZv8z80twu+wvgBklVp8PVHP33sAA4Cozax9FrBLn/vrxt6zI2sXvz+pNjaoqpyESz6JJEFcDd5jZGjNbC9wGXBXF6wYCGe6+Imh1TACGl7Lf/cADQG6JdQ7UMrMqQA0gH9gexTEljr00bRVPTl3OiNTWHNe1SdjhiMgBRDNQbrm7DybSCjjC3Y9x94wo3rsVsLbEcmaw7gdmdiTQxt3/tddrJwG7iHSOrwEecvctex8gGLCXZmZpWVlZUYQkYRk/cw13v7WIk3s04/dn9w47HBGJQlTVXM3sp0BPoPqewUzuft/hHDio7/QIMLqUzQOJdIa3BBoAX5jZR+6+ouRO7v4s8CxAamqqH048EjtvzM7kjjcXMLRbEx4f2Z+UZJX2EkkEB0wQZvY0UBM4AXgOOA+YGcV7rwNKzhfZOli3Rx2gFzA1SDrNgSlBf8dI4D13LwA2mtlXQCrwXwlC4t/b89dz66T5HNOpEU+PGqAy3iIJJJp/5Y5x90uAre5+L3A0EM3tJ7OALmbWwcyqAhcCU/ZsdPdsd2/s7u3dvT0wHRjm7mlELiv9GMDMagGDgSUHcV4SB95buIFfTpxHaruG/O2SVKqnKDmIJJJoEsSezuMcM2sJFBCpx7Rf7l4IXAe8DywGXnP3RWZ2X8m7ovbhCaC2mS0ikmhecPf0KGKVOPHJku+5fvwc+rSux/OXHUXNqpqbSiTRRPNX+7aZ1Qf+BMwhcofR36J5c3d/F3h3r3V372PfoSWe7yRyq6skoC++zeLqcXPo3rwuL142kNrVlBxEEtF+/3KDjuSP3X0b8IaZvQNUd/fs8ghOEs/0FZu58qU0OjauxUs/H6j5HUQS2H4vMbl7MZHLPXuW85QcZF9mr97Cz1+cRZsGNRl3xSAa1IpqwL2IxKlo+iA+DkphqKKa7NP8tdsY/fwsmtWtzitXDKJx7WphhyQihymaBHEVkeJ8eWa23cx2mJlGNcsPFq3P5pLnZ1K/VgqvXjmIpnWrhx2SiJSBA/YeurumFpV9WrphBxf/fSa1qibz6hWDaVFPE/+IVBTRDJQ7rrT1e08gJJXP8qyd/Oy5GVRJMl69cjBtGtYMOyQRKUPR3H94a4nn1YmUwZhNMJBNKqfVm3cx8m/TAefVK4+mfeNaYYckImUsmktMZ5ZcNrM2wKOxCkjiX+bWHEb+bQb5hcVMGHM0nZvWDjskEYmBQ6malgkcUdaBSGKYvXoLFzwznR25Bbx8+SC6NVcXlUhFFU0fxGNERk9DJKH0IzKiWiqRvMIiHv3oW575bDkt69dg3BWD6NVK04WKVGTR9EGklXheCIx3969iFI/EocXfbefGifNYsmEHFw1sw50/7aHyGSKVQDR/5ZOAXHcvgshUomZW091zYhuahK2o2Hn28xU88uFS6tWoyvOjU/lx92ZhhyUi5SSaBPExcBKwM1iuAXwAHBOroCR8qzfv4qbX5jN79VZO792c353Vm4YqnSFSqUSTIKoH1VWBSKVVM9MN7xWUu/PKjDX84d3FVEky/nJhP4b1bYkqrYhUPtEkiF1mdqS7zwEwswHA7tiGJWH4fnsuv5qUzmfLsji2S2MePK+PRkaLVGLRJIhfAq+b2XrAiEwNekEsg5LyN2X+eu6avJC8wiLuH96TUYPbqdUgUslFM1Bulpl1B7oFq5YGc0VLBbAtJ5/fTF7IO+nf0b9tfR4+vy8dm2jgm4hENw5iLPCKuy8MlhuY2UXu/mTMo5OY+nTpRm6blM6WXfnc+pNuXHVcR6okH8rYSRGpiKL5NLgymFEOAHffClwZs4gk5nILirjjzQVc9sIsGtSsyuSxQxh7QmclBxH5L9H0QSSbmbm7Q2QcBKD7HRPUhuxcxrycxoJ12Vx1XEduPLkr1VOSww5LROJQNAniPWCimT0TLF8F/Dt2IUmszFu7jTEvpbErr5BnL07l5B4a9CYi+xZNgrgNGANcHSynE7mTSRLI5Lnr+NUb6TStU42XLx+iInsickDR3MVUbGYzgE7ACKAx8EasA5OyUVzs/OmDpTw1dTmDOjTkqVEDNCJaRKKyzwRhZl2Bi4LHJmAigLufUD6hyeHakVvAjRPn8dHijYwc1JZ7zuxJ1SrqiBaR6OyvBbEE+AI4w90zAMzsxnKJSg7bms05XPHSLJZn7eK+4T25WAPfROQg7S9BnANcCHxqZu8BE4iMpJY4N235Zq59ZTbFDi/9fCBDOjcOOyQRSUD7vN7g7pPd/UKgO/ApkZIbTc3sKTM7pZzik4M0bvpqLv77DBrWqspbY4coOYjIITvgBWl33+XurwZzU7cG5hK5s0niSEFRMXdNXshvJi/kR10a8+bYIbRvXCvssEQkgR3UtGDBKOpng4fEia278rn2lTlMW7GZMcd15LZTu5OcpKuBInJ4NG9kgvv2+x1c8VIa323L5aHz+3LegNZhhyQiFYQSRAL7ZMn33DB+HtVTkhk/ZjAD2jUIOyQRqUCUIBLUlPnr+cWEufRoUZe/XZJKy/qa2EdEypYSRAJatWkXv34jnQFtG/DS5QOpWVU/RhEpezEdVmtmp5rZUjPLMLPb97PfuWbmZpZaYl0fM5tmZovMbIGZVY9lrIkiv7CYGybMpUpyEn+5qL+Sg4jETMw+XYKy4E8AJwOZwCwzm+Lu3+y1Xx3gF8CMEuuqAOOAi919vpk1AjSLHfDwB0tJz8zm6VFH0kqXlUQkhmLZghgIZLj7CnfPJzISe3gp+90PPADkllh3CpDu7vMB3H2zuxfFMNaE8NmyLJ75fAU/G9SWU3u1CDscEangYpkgWgFrSyxnBut+YGZHAm3c/V97vbYr4Gb2vpnNMbNflXYAMxtjZmlmlpaVlVWWscedrB153PzaPLo2q81dZ/QIOxwRqQRCK+1pZknAI8DNpWyuAvwI+Fnw9WwzO3Hvndz9WXdPdffUJk2axDTeMBUXOze/Pp8duYU8dtGRmgFORMpFLBPEOqBNieXWwbo96gC9gKlmtgoYDEwJOqozgc/dfZO75wDvAkfGMNa49vcvV/L5sizuOqOHJvoRkXITywQxC+hiZh3MrCqRyrBT9mx092x3b+zu7d29PTAdGObuacD7QG8zqxl0WB8PfPO/h6j40jO38eD7S/hJz2b8bFDbsMMRkUokZgnC3QuB64h82C8GXnP3RWZ2n5kNO8BrtxK5/DQLmAfMKaWfosLbmVfI9ePn0qR2NR44t4/mcxCRchXTm+jd/V0il4dKrrt7H/sO3Wt5HJFbXSutuycvZO2WHCaMOZr6NTVNqIiUL80/Gaf+OSeTf85dxw0ndmFgh4ZhhyMilZASRBxauWkXd01eyMD2DbnuhM5hhyMilZQSRJzJLyzmhvGRUhqPXtiPKsn6EYlIOFTIJ8489MFSFqzL5ulRA1ShVURCpX9P48hny7J49vMVjBrcllN7NQ87HBGp5JQg4sSeUhrdmtXhNz9VKQ0RCZ8uMcWB4mLnptfmsSO3kFevHKxSGiISF9SCiAPPfbmCL77dxN1n9qBrM5XSEJH4oAQRsvlrt/Hge0s5tWdzRg5UKQ0RiR9KECHamVfIDRPm0rRONf7v3N4qpSEicUV9ECFxd+58cwFrt+Qw8SqV0hCR+KMWREhembGGt+at58aTunJUe5XSEJH4owQRgvTMbdz39jcM7daEsSqlISJxSgminG3LyeeacXNoUqcafx7Rj6Qk9TuISHxSH0Q5iox3mM/GHbm8fvUxNKilfgcRiV9qQZSjpz5bzidLNnLXGT3o16Z+2OGIiOyXEkQ5+Xr5Jh7+YCln9m3JxYPbhR2OiMgBKUGUg++353LD+Ll0aFyLP56j8Q4ikhjUBxFjhUXFXP/qXHblFfHqlYOpXU3fchFJDPq0irE/fbCUmau28OgF/VRnSUQSii4xxdAHizbwzGcr+NmgtpzVv1XY4YiIHBQliBhZszmHm1+fT+9W9bjrDM3vICKJRwkiBnILirjmldkY8OTPjtT8DiKSkNQHEQP3vv0Ni9Zv57lLUmnTsGbY4YiIHBK1IMrYG7MzGT9zDdcM7cRJPZqFHY6IyCFTgihDSzZs587JCxjUoSE3n9w17HBERA6LEkQZ2ZFbwDXj5lCnegqPjexPlWR9a0UksakPogy4O7e/sYA1W3J45YpBNK1TPeyQREQOm/7NLQMvfr2Kfy34jlt/0o3BHRuFHY6ISJlQgjhMs1dv5ff/WsxJRzRjzLEdww5HRKTMKEEchvzCYm6cOI8W9avz8Pl9NfmPiFQo6oM4DBNnrWHNlhxevOwo6tVMCTscEZEypRbEIdqdX8Rjn2RwVPsGHN+1SdjhiIiUuZgmCDM71cyWmlmGmd2+n/3ONTM3s9S91rc1s51mdkss4zwUL09fxcYdedxySjfN7yAiFVLMEoSZJQNPAKcBPYCLzOx/qtaZWR3gF8CMUt7mEeDfsYrxUO3ILeCpqcs5tktjBumuJRGpoGLZghgIZLj7CnfPByYAw0vZ737gASC35EozOwtYCSyKYYyH5PkvV7E1p4BbTukWdigiIjETywTRClhbYjkzWPcDMzsSaOPu/9prfW3gNuDe/R3AzMaYWZqZpWVlZZVN1AewLSef575YwSk9mtG3Tf1yOaaISBhC66Q2syQil5BuLmXzPcCf3X3n/t7D3Z9191R3T23SpHw6ip/+bAU78wu5Wa0HEangYnmb6zqgTYnl1sG6PeoAvYCpQSdvc2CKmQ0DBgHnmdmDQH2g2Mxy3f3xGMZ7QBt35PLi1ysZ1rcl3Zpr+lARqdhimSBmAV3MrAORxHAhMHLPRnfPBhrvWTazqcAt7p4GHFti/T3AzrCTA8CTny6noMi58SRVahWRii9ml5jcvRC4DngfWAy85u6LzOy+oJWQUNZt282rM9Zw/oDWtG9cK+xwRERiLqYjqd39XeDdvdbdvY99h+5j/T1lHtgh+OtH3wJw/YldQo5ERKR8aCR1FFZu2sWkOZmMHNSWVvVrhB2OiEi5UIKIwp8/XEbV5CTGntA57FBERMqNEsQBLNmwnbfT13PZkPY0qVMt7HBERMqNEsQBPPzBMmpXq8JVx3UKOxQRkXKlBLEf89Zu48NvvmfMsR1VzltEKh0liP146P2lNKxVlct+1CHsUEREyp0SxD5MW76ZLzM2ce3QTtSupnmVRKTyUYIohbvz0AdLaVa3GqMGtws7HBGRUChBlGLq0ixmr97K9T/uQvWU5LDDEREJhRLEXoqLI62HNg1rMCK1zYFfICJSQSlB7OW9RRtYtH47vzyxK1Wr6NsjIpWXPgFLKCp2HvlwGZ2b1uas/q0O/AIRkQpMCaKEyXPXkbFxJzed3JXkJAs7HBGRUClBBPILi3n042X0bFmXU3s2DzscEZHQKUEEXktby9otu7nllG4kqfUgIqIEAZBbUMRjn3zLgHYNGNqtfOa2FhGJd0oQwLjpq/l+ex63nNKNYH5sEZFKr9IniJ15hTw5dTk/6tyYozs1CjscEZG4UemLDOXkFTKoQ0PGHNcx7FBEROJKpU8QTetW56lRA8IOQ0Qk7lT6S0wiIlI6JQgRESmVEoSIiJRKCUJEREqlBCEiIqVSghARkVIpQYiISKmUIEREpFTm7mHHUCbMLAtYfRhv0RjYVEbhJIrKds6V7XxB51xZHM45t3P3UquUVpgEcbjMLM3dU8OOozxVtnOubOcLOufKIlbnrEtMIiJSKiUIEREplRLEfzwbdgAhqGznXNnOF3TOlUVMzll9ECIiUiq1IEREpFRKECIiUqpKlSDM7FQzW2pmGWZ2eynbq5nZxGD7DDNrH0KYZSqKc77JzL4xs3Qz+9jM2oURZ1k60DmX2O9cM3MzS/hbIqM5ZzMbEfysF5nZq+UdY1mL4ne7rZl9amZzg9/v08OIs6yY2fNmttHMFu5ju5nZX4PvR7qZHXnYB3X3SvEAkoHlQEegKjAf6LHXPtcCTwfPLwQmhh13OZzzCUDN4Pk1leGcg/3qAJ8D04HUsOMuh59zF2Au0CBYbhp23OVwzs8C1wTPewCrwo77MM/5OOBIYOE+tp8O/BswYDAw43CPWZlaEAOBDHdf4e75wARg+F77DAf+ETyfBJxoZlaOMZa1A56zu3/q7jnB4nSgdTnHWNai+TkD3A88AOSWZ3AxEs05Xwk84e5bAdx9YznHWNaiOWcH6gbP6wHryzG+MufunwNb9rPLcOAlj5gO1DezFodzzMqUIFoBa0ssZwbrSt3H3QuBbKBRuUQXG9Gcc0mXE/kPJJEd8JyDpncbd/9XeQYWQ9H8nLsCXc3sKzObbmanllt0sRHNOd8DjDKzTOBd4PryCS00B/v3fkBVDiscqTDMbBSQChwfdiyxZGZJwCPA6JBDKW9ViFxmGkqklfi5mfV2921hBhVjFwEvuvvDZnY08LKZ9XL34rADSxSVqQWxDmhTYrl1sK7UfcysCpFm6eZyiS42ojlnzOwk4E5gmLvnlVNssXKgc64D9AKmmtkqItdqpyR4R3U0P+dMYIq7F7j7SmAZkYSRqKI558uB1wDcfRpQnUhRu4oqqr/3g1GZEsQsoIuZdTCzqkQ6oafstc8U4NLg+XnAJx70/iSoA56zmfUHniGSHBL9ujQc4JzdPdvdG7t7e3dvT6TfZZi7p4UTbpmI5nd7MpHWA2bWmMglpxXlGGNZi+ac1wAnApjZEUQSRFa5Rlm+pgCXBHczDQay3f27w3nDSnOJyd0Lzew64H0id0A87+6LzOw+IM3dpwB/J9IMzSDSGXRheBEfvijP+U9AbeD1oD9+jbsPCy3owxTlOVcoUZ7z+8ApZvYNUATc6u4J2zqO8pxvBv5mZjcS6bAencj/8JnZeCJJvnHQr/JbIAXA3Z8m0s9yOpAB5ACXHfYxE/j7JSIiMVSZLjGJiMhBUIIQEZFSKUGIiEiplCBERKRUShAiIlIqJQiJa2ZWZGbzzGyhmb1uZjVDiGGomR2zj22jzSwriHFJcEvloR5namkD9va1XiTWlCAk3u12937u3gvIB66O5kXBSPiyMhQoNUEEJrp7P2AIcKeZtdnPviIJQwlCEskXQGczqxXUxp8Z1PofDj/8Nz/FzD4BPjaz2mb2gpktCOrjnxvsd4qZTTOzOUGrpHawfpWZ3RusX2Bm3S0yJ8jVwI1BK+HYfQUXDDzLAFoE7zcqiHGemT1jZsnB+qfMLM0i8zLcezDfADPbaWZ/Cl77kZkNDFoYK8xsWLBPezP7IjiPOXtaP2aWZGZPBi2dD83sXTM7L9g2wMw+M7PZZva+BVVAzewG+898IRMOJlapAMKuca6HHvt7ADuDr1WAt4jMWfEHYFSwvj6RukK1iBTgywQaBtseAB4t8V4NiNTi+RyoFay7Dbg7eL4KuD54fi3wXPD8HuCWfcQ3Gng8eN4WmEekpMMRwNtASrDtSeCS4Pme+JKBqUCfYHkqpcxNUXI9kRHBpwXP3wQ+IDKati8wL1hfE6gePO9CZGQxRMrHvEvkH8PmwNZgXQrwNdAk2O8CIiOTIVIiu9qe73XYvw96lO+j0pTakIRVw8zmBc+/IFIO5WtgmJndEqyvTuTDGeBDd99TM/8kSpRLcfetZnYGkcljvgpKi1QFppU43j+Dr7OBc6KM8QIzOw7oDlzn7rlmdiIwAJgVHKcGsKfW1QgzG0Mk6bUI4kmP8lj5wHvB8wVAnrsXmNkCoH2wPgV43Mz6ESmr0TVY/yPgdY9UM91gZp8G67sRKWD4YRBrMrCnhk868IqZTSZSz0kqESUIiXe7PXJ9/wcW+RQ7192X7rV+ELDrAO9nRJLIRfvYvqeabRHR/31MdPfrgo7kD8xsSnCcf7j7r/eKsQNwC3BUkLBeJJLgolXg7nvq4xTvidfdi0v0u9wIfE+kVZHEgSdFMmCRux9dyrafEpnJ7Ewi/Su9PTJXilQC6oOQRPQ+cH2QKPZUpC3Nh8DYPQtm1oBI9dYhZtY5WFfLzLru4/V77CBSJny/PFIR9mXgF8DHwHlm1jQ4TkOLzPddl0gSyzazZsBpB3rfQ1AP+C5oKVxMpEUA8BVwbtAX0YyguiuwFGhikTkTMLMUM+tpkbkz2rj7p0QuxdUjUthRKgklCElE9xO5jJJuZouC5dL8DmgQ3CI7HzjB3bOI9BuMN7N0IpeXuh/geG8DZx+okzrwAJEqmmuB3xBpUaQTSVYt3H0+kbmhlwCvEvnQLmtPApcG59yd/7Sq3iDSR/MNMA6YQ6QkdD6RvogHgtfMI3LXVjIwLrh8NRf4q1fsCYZkL6rmKlKJmFltd99pZo2AmcAQd98QdlwSn9QHIVK5vGNm9Yl0zt+v5CD7oxaEiIiUSn0QIiJSKiUIEREplRKEiIiUSglCRERKpQQhIiKl+n+g2pQlv7thwQAAAABJRU5ErkJggg==\n",
"text/plain": [
"