{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UEBilEjLj5wY"
   },
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 536,
     "status": "ok",
     "timestamp": 1524974472601,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "GOzuY8Yvj5wb",
    "outputId": "c19362ce-f87a-4cc2-84cc-8d7b4b9e6007"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Author: Sebastian Raschka\n",
      "\n",
      "Python implementation: CPython\n",
      "Python version       : 3.8.12\n",
      "IPython version      : 8.0.1\n",
      "\n",
      "torch: 1.10.1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rH4XmErYj5wm"
   },
   "source": [
    "# AlexNet CIFAR-10 Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AlexNet [1][2] trained on CIFAR-10 [3].\n",
    "\n",
    "This implementation uses grouped convolutions like in the original AlexNet paper [2]:\n",
    "\n",
    "![](../images/alexnet/alexnet-paper.png)\n",
    "\n",
    "Here, the network is essentially split into two parts to train it on two GPUs with 1.5 Gb RAM each. This was purely done for computational performance reasons (and the video RAM limitation back then). However, there are certain benefits to using grouped convolutions ...\n",
    "\n",
    "\n",
    "**Taking a step back, how do grouped convolutions work?**\n",
    "\n",
    "In a nutshell, you can think of grouped convolutions as convolutional layers that process part of the input independently and merge the results. So, for example, if you consider grouped convolutions with two filter groups, each filter group would process half of the channels. \n",
    "\n",
    "![](../images/alexnet/grouped-convolutions.png)\n",
    "\n",
    "**One of the benefits of grouped convolutions is**, as noted by Yani Ioannou [4], that AlexNet has a slightly improved accuracy when using two filter groups:\n",
    "\n",
    "![](../images/alexnet/alexnet-groups.png)\n",
    "\n",
    "**Another benefit is the reduced parameter size**. \n",
    "\n",
    "Say we have kernels with height 3 and width 3. The inputs have 6 channels, and the output channels are set to 12. Then, we have kernels with 3x3x6 weight parameters with a regular convolution. Since we have 12 output channels, that's 3x3x6x12=648 parameters in total.\n",
    "\n",
    "Now, let's assume we use a grouped convolution with group size 2. We still have a 3x3 kernel height and width. But now, the number of input channels is split by a factor of 2, so each kernel is 3x3x3. The first group produces the first 6 output channels, so we have 3x3x3x6 parameters for the first group. The second group has the same size, so we have (3x3x3x6)x2 = 3x3x3x12 = 324, which is a 2x reduction in parameters compared to the regular convolution.\n",
    "\n",
    "\n",
    "**And how do we do this in PyTorch?**\n",
    "\n",
    "Implementing grouped convolutions in PyTorch is now really straightforward. We just used the `groups` parameter. For example, to implement a grouped convolution with two filter groups we use\n",
    "\n",
    "    torch.nn.Conv2d(..., groups=2)\n",
    "\n",
    "Note that a requirement for this is that the number of input and output channels is divisible by groups (here: 2).\n",
    "\n",
    "### References\n",
    "\n",
    "- [1] L13.7 CNN Architectures & AlexNet (20:17), https://www.youtube.com/watch?v=-IHxe4-09e4\n",
    "- [2] Imagenet classification with deep convolutional neural networks, https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf\n",
    "- [3] https://en.wikipedia.org/wiki/CIFAR-10\n",
    "- [4] https://blog.yani.ai/filter-group-tutorial/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MkoGLH_Tj5wn"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "ORj09gnrj5wp"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import random\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.dataset import Subset\n",
    "\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I6hghKPxj5w0"
   },
   "source": [
    "## Model Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting a random seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I recommend using a function like the following one prior to using dataset loaders and initializing a model if you want to ensure the data is shuffled in the same manner if you rerun this notebook and the model gets the same initial random weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_all_seeds(seed):\n",
    "    os.environ[\"PL_GLOBAL_SEED\"] = str(seed)\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed_all(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting cuDNN and PyTorch algorithmic behavior to deterministic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the `set_all_seeds` function above, I recommend setting the behavior of PyTorch and cuDNN to deterministic (this is particulary relevant when using GPUs). We can also define a function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_deterministic():\n",
    "    if torch.cuda.is_available():\n",
    "        torch.backends.cudnn.benchmark = False\n",
    "        torch.backends.cudnn.deterministic = True\n",
    "    torch.set_deterministic(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23936,
     "status": "ok",
     "timestamp": 1524974497505,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "NnT0sZIwj5wu",
    "outputId": "55aed925-d17e-4c6a-8c71-0d9b3bde5637"
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Hyperparameters\n",
    "RANDOM_SEED = 1\n",
    "LEARNING_RATE = 0.0001\n",
    "BATCH_SIZE = 256\n",
    "NUM_EPOCHS = 40\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10\n",
    "\n",
    "# Other\n",
    "DEVICE = \"cuda:0\"\n",
    "\n",
    "set_all_seeds(RANDOM_SEED)\n",
    "\n",
    "# Deterministic behavior not yet supported by AdaptiveAvgPool2d\n",
    "#set_deterministic()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import utility functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.insert(0, \"..\") # to include ../helper_evaluate.py etc.\n",
    "\n",
    "from helper_evaluate import compute_accuracy\n",
    "from helper_data import get_dataloaders_cifar10\n",
    "from helper_train import train_classifier_simple_v1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "### Set random seed ###\n",
    "set_all_seeds(RANDOM_SEED)\n",
    "\n",
    "##########################\n",
    "### Dataset\n",
    "##########################\n",
    "\n",
    "train_transforms = transforms.Compose([transforms.Resize((70, 70)),\n",
    "                                       transforms.RandomCrop((64, 64)),\n",
    "                                       transforms.ToTensor()])\n",
    "\n",
    "test_transforms = transforms.Compose([transforms.Resize((70, 70)),\n",
    "                                      transforms.CenterCrop((64, 64)),\n",
    "                                      transforms.ToTensor()])\n",
    "\n",
    "\n",
    "train_loader, valid_loader, test_loader = get_dataloaders_cifar10(\n",
    "    batch_size=BATCH_SIZE, \n",
    "    num_workers=2, \n",
    "    train_transforms=train_transforms,\n",
    "    test_transforms=test_transforms,\n",
    "    validation_fraction=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Set:\n",
      "\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n",
      "tensor([0, 2, 3, 5, 4, 8, 9, 6, 9, 7])\n",
      "\n",
      "Validation Set:\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n",
      "tensor([6, 9, 3, 5, 7, 3, 4, 1, 8, 0])\n",
      "\n",
      "Testing Set:\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n",
      "tensor([2, 6, 3, 1, 1, 1, 1, 2, 4, 8])\n"
     ]
    }
   ],
   "source": [
    "# Checking the dataset\n",
    "print('Training Set:\\n')\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    print(labels[:10])\n",
    "    break\n",
    "    \n",
    "# Checking the dataset\n",
    "print('\\nValidation Set:')\n",
    "for images, labels in valid_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    print(labels[:10])\n",
    "    break\n",
    "\n",
    "# Checking the dataset\n",
    "print('\\nTesting Set:')\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    print(labels[:10])\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "class AlexNet(nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes):\n",
    "        super(AlexNet, self).__init__()\n",
    "        self.features = nn.Sequential(\n",
    "            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "            \n",
    "            nn.Conv2d(64, 192, kernel_size=5, padding=2, groups=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "            \n",
    "            nn.Conv2d(192, 384, kernel_size=3, padding=1, groups=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            \n",
    "            nn.Conv2d(384, 256, kernel_size=3, padding=1, groups=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            \n",
    "            nn.Conv2d(256, 256, kernel_size=3, padding=1, groups=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "        )\n",
    "        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(256 * 6 * 6, 4096),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(4096, 4096),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(4096, num_classes)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = self.avgpool(x)\n",
    "        x = x.view(x.size(0), 256 * 6 * 6)\n",
    "        logits = self.classifier(x)\n",
    "        probas = F.softmax(logits, dim=1)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "_lza9t_uj5w1"
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(RANDOM_SEED)\n",
    "\n",
    "model = AlexNet(NUM_CLASSES)\n",
    "model.to(DEVICE)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RAodboScj5w6"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 1547
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2384585,
     "status": "ok",
     "timestamp": 1524976888520,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "Dzh3ROmRj5w7",
    "outputId": "5f8fd8c9-b076-403a-b0b7-fd2d498b48d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/040 | Batch 0000/0175 | Loss: 2.3021\n",
      "Epoch: 001/040 | Batch 0050/0175 | Loss: 2.0457\n",
      "Epoch: 001/040 | Batch 0100/0175 | Loss: 1.8939\n",
      "Epoch: 001/040 | Batch 0150/0175 | Loss: 1.8882\n",
      "***Epoch: 001/040 | Train. Acc.: 32.408% | Loss: 1.743\n",
      "***Epoch: 001/040 | Valid. Acc.: 33.340% | Loss: 1.714\n",
      "Time elapsed: 1.02 min\n",
      "Epoch: 002/040 | Batch 0000/0175 | Loss: 1.7017\n",
      "Epoch: 002/040 | Batch 0050/0175 | Loss: 1.7092\n",
      "Epoch: 002/040 | Batch 0100/0175 | Loss: 1.6315\n",
      "Epoch: 002/040 | Batch 0150/0175 | Loss: 1.5158\n",
      "***Epoch: 002/040 | Train. Acc.: 41.415% | Loss: 1.555\n",
      "***Epoch: 002/040 | Valid. Acc.: 42.140% | Loss: 1.538\n",
      "Time elapsed: 2.06 min\n",
      "Epoch: 003/040 | Batch 0000/0175 | Loss: 1.5034\n",
      "Epoch: 003/040 | Batch 0050/0175 | Loss: 1.5797\n",
      "Epoch: 003/040 | Batch 0100/0175 | Loss: 1.4733\n",
      "Epoch: 003/040 | Batch 0150/0175 | Loss: 1.2810\n",
      "***Epoch: 003/040 | Train. Acc.: 48.246% | Loss: 1.380\n",
      "***Epoch: 003/040 | Valid. Acc.: 49.020% | Loss: 1.373\n",
      "Time elapsed: 3.09 min\n",
      "Epoch: 004/040 | Batch 0000/0175 | Loss: 1.3568\n",
      "Epoch: 004/040 | Batch 0050/0175 | Loss: 1.3999\n",
      "Epoch: 004/040 | Batch 0100/0175 | Loss: 1.4197\n",
      "Epoch: 004/040 | Batch 0150/0175 | Loss: 1.3423\n",
      "***Epoch: 004/040 | Train. Acc.: 52.406% | Loss: 1.289\n",
      "***Epoch: 004/040 | Valid. Acc.: 53.300% | Loss: 1.280\n",
      "Time elapsed: 4.13 min\n",
      "Epoch: 005/040 | Batch 0000/0175 | Loss: 1.2611\n",
      "Epoch: 005/040 | Batch 0050/0175 | Loss: 1.2486\n",
      "Epoch: 005/040 | Batch 0100/0175 | Loss: 1.2251\n",
      "Epoch: 005/040 | Batch 0150/0175 | Loss: 1.2385\n",
      "***Epoch: 005/040 | Train. Acc.: 52.710% | Loss: 1.277\n",
      "***Epoch: 005/040 | Valid. Acc.: 54.220% | Loss: 1.255\n",
      "Time elapsed: 5.17 min\n",
      "Epoch: 006/040 | Batch 0000/0175 | Loss: 1.3014\n",
      "Epoch: 006/040 | Batch 0050/0175 | Loss: 1.2493\n",
      "Epoch: 006/040 | Batch 0100/0175 | Loss: 1.1972\n",
      "Epoch: 006/040 | Batch 0150/0175 | Loss: 1.1217\n",
      "***Epoch: 006/040 | Train. Acc.: 58.699% | Loss: 1.130\n",
      "***Epoch: 006/040 | Valid. Acc.: 58.900% | Loss: 1.133\n",
      "Time elapsed: 6.20 min\n",
      "Epoch: 007/040 | Batch 0000/0175 | Loss: 1.1142\n",
      "Epoch: 007/040 | Batch 0050/0175 | Loss: 1.1751\n",
      "Epoch: 007/040 | Batch 0100/0175 | Loss: 1.1640\n",
      "Epoch: 007/040 | Batch 0150/0175 | Loss: 1.3103\n",
      "***Epoch: 007/040 | Train. Acc.: 59.344% | Loss: 1.130\n",
      "***Epoch: 007/040 | Valid. Acc.: 58.880% | Loss: 1.139\n",
      "Time elapsed: 7.23 min\n",
      "Epoch: 008/040 | Batch 0000/0175 | Loss: 1.0524\n",
      "Epoch: 008/040 | Batch 0050/0175 | Loss: 1.1116\n",
      "Epoch: 008/040 | Batch 0100/0175 | Loss: 1.0474\n",
      "Epoch: 008/040 | Batch 0150/0175 | Loss: 1.0964\n",
      "***Epoch: 008/040 | Train. Acc.: 60.248% | Loss: 1.114\n",
      "***Epoch: 008/040 | Valid. Acc.: 59.220% | Loss: 1.136\n",
      "Time elapsed: 8.28 min\n",
      "Epoch: 009/040 | Batch 0000/0175 | Loss: 1.1965\n",
      "Epoch: 009/040 | Batch 0050/0175 | Loss: 1.0852\n",
      "Epoch: 009/040 | Batch 0100/0175 | Loss: 1.1306\n",
      "Epoch: 009/040 | Batch 0150/0175 | Loss: 1.0086\n",
      "***Epoch: 009/040 | Train. Acc.: 61.824% | Loss: 1.058\n",
      "***Epoch: 009/040 | Valid. Acc.: 60.620% | Loss: 1.076\n",
      "Time elapsed: 9.31 min\n",
      "Epoch: 010/040 | Batch 0000/0175 | Loss: 1.0530\n",
      "Epoch: 010/040 | Batch 0050/0175 | Loss: 1.0641\n",
      "Epoch: 010/040 | Batch 0100/0175 | Loss: 0.9715\n",
      "Epoch: 010/040 | Batch 0150/0175 | Loss: 1.1926\n",
      "***Epoch: 010/040 | Train. Acc.: 64.944% | Loss: 0.977\n",
      "***Epoch: 010/040 | Valid. Acc.: 62.820% | Loss: 1.028\n",
      "Time elapsed: 10.34 min\n",
      "Epoch: 011/040 | Batch 0000/0175 | Loss: 1.0312\n",
      "Epoch: 011/040 | Batch 0050/0175 | Loss: 1.0072\n",
      "Epoch: 011/040 | Batch 0100/0175 | Loss: 0.9125\n",
      "Epoch: 011/040 | Batch 0150/0175 | Loss: 1.0450\n",
      "***Epoch: 011/040 | Train. Acc.: 65.922% | Loss: 0.945\n",
      "***Epoch: 011/040 | Valid. Acc.: 64.360% | Loss: 0.990\n",
      "Time elapsed: 11.37 min\n",
      "Epoch: 012/040 | Batch 0000/0175 | Loss: 1.0078\n",
      "Epoch: 012/040 | Batch 0050/0175 | Loss: 1.0750\n",
      "Epoch: 012/040 | Batch 0100/0175 | Loss: 0.8935\n",
      "Epoch: 012/040 | Batch 0150/0175 | Loss: 0.9567\n",
      "***Epoch: 012/040 | Train. Acc.: 67.286% | Loss: 0.907\n",
      "***Epoch: 012/040 | Valid. Acc.: 64.140% | Loss: 0.983\n",
      "Time elapsed: 12.40 min\n",
      "Epoch: 013/040 | Batch 0000/0175 | Loss: 0.9305\n",
      "Epoch: 013/040 | Batch 0050/0175 | Loss: 0.8980\n",
      "Epoch: 013/040 | Batch 0100/0175 | Loss: 1.0309\n",
      "Epoch: 013/040 | Batch 0150/0175 | Loss: 0.9596\n",
      "***Epoch: 013/040 | Train. Acc.: 69.614% | Loss: 0.853\n",
      "***Epoch: 013/040 | Valid. Acc.: 66.480% | Loss: 0.926\n",
      "Time elapsed: 13.43 min\n",
      "Epoch: 014/040 | Batch 0000/0175 | Loss: 0.8409\n",
      "Epoch: 014/040 | Batch 0050/0175 | Loss: 0.9902\n",
      "Epoch: 014/040 | Batch 0100/0175 | Loss: 0.8945\n",
      "Epoch: 014/040 | Batch 0150/0175 | Loss: 0.9077\n",
      "***Epoch: 014/040 | Train. Acc.: 71.190% | Loss: 0.822\n",
      "***Epoch: 014/040 | Valid. Acc.: 67.460% | Loss: 0.915\n",
      "Time elapsed: 14.45 min\n",
      "Epoch: 015/040 | Batch 0000/0175 | Loss: 0.9422\n",
      "Epoch: 015/040 | Batch 0050/0175 | Loss: 0.8516\n",
      "Epoch: 015/040 | Batch 0100/0175 | Loss: 0.7501\n",
      "Epoch: 015/040 | Batch 0150/0175 | Loss: 0.8076\n",
      "***Epoch: 015/040 | Train. Acc.: 70.270% | Loss: 0.832\n",
      "***Epoch: 015/040 | Valid. Acc.: 66.460% | Loss: 0.936\n",
      "Time elapsed: 15.49 min\n",
      "Epoch: 016/040 | Batch 0000/0175 | Loss: 0.8759\n",
      "Epoch: 016/040 | Batch 0050/0175 | Loss: 0.9273\n",
      "Epoch: 016/040 | Batch 0100/0175 | Loss: 0.8645\n",
      "Epoch: 016/040 | Batch 0150/0175 | Loss: 0.8602\n",
      "***Epoch: 016/040 | Train. Acc.: 69.623% | Loss: 0.859\n",
      "***Epoch: 016/040 | Valid. Acc.: 65.780% | Loss: 0.967\n",
      "Time elapsed: 16.53 min\n",
      "Epoch: 017/040 | Batch 0000/0175 | Loss: 0.9171\n",
      "Epoch: 017/040 | Batch 0050/0175 | Loss: 0.8242\n",
      "Epoch: 017/040 | Batch 0100/0175 | Loss: 0.7830\n",
      "Epoch: 017/040 | Batch 0150/0175 | Loss: 0.9317\n",
      "***Epoch: 017/040 | Train. Acc.: 72.188% | Loss: 0.794\n",
      "***Epoch: 017/040 | Valid. Acc.: 67.400% | Loss: 0.948\n",
      "Time elapsed: 17.56 min\n",
      "Epoch: 018/040 | Batch 0000/0175 | Loss: 0.7613\n",
      "Epoch: 018/040 | Batch 0050/0175 | Loss: 0.8159\n",
      "Epoch: 018/040 | Batch 0100/0175 | Loss: 0.8606\n",
      "Epoch: 018/040 | Batch 0150/0175 | Loss: 0.8943\n",
      "***Epoch: 018/040 | Train. Acc.: 74.721% | Loss: 0.720\n",
      "***Epoch: 018/040 | Valid. Acc.: 69.640% | Loss: 0.866\n",
      "Time elapsed: 18.60 min\n",
      "Epoch: 019/040 | Batch 0000/0175 | Loss: 0.7614\n",
      "Epoch: 019/040 | Batch 0050/0175 | Loss: 0.7849\n",
      "Epoch: 019/040 | Batch 0100/0175 | Loss: 0.8485\n",
      "Epoch: 019/040 | Batch 0150/0175 | Loss: 0.8462\n",
      "***Epoch: 019/040 | Train. Acc.: 75.212% | Loss: 0.704\n",
      "***Epoch: 019/040 | Valid. Acc.: 69.440% | Loss: 0.872\n",
      "Time elapsed: 19.64 min\n",
      "Epoch: 020/040 | Batch 0000/0175 | Loss: 0.6625\n",
      "Epoch: 020/040 | Batch 0050/0175 | Loss: 0.7826\n",
      "Epoch: 020/040 | Batch 0100/0175 | Loss: 0.7387\n",
      "Epoch: 020/040 | Batch 0150/0175 | Loss: 0.7622\n",
      "***Epoch: 020/040 | Train. Acc.: 76.560% | Loss: 0.663\n",
      "***Epoch: 020/040 | Valid. Acc.: 69.820% | Loss: 0.859\n",
      "Time elapsed: 20.68 min\n",
      "Epoch: 021/040 | Batch 0000/0175 | Loss: 0.7006\n",
      "Epoch: 021/040 | Batch 0050/0175 | Loss: 0.6893\n",
      "Epoch: 021/040 | Batch 0100/0175 | Loss: 0.6352\n",
      "Epoch: 021/040 | Batch 0150/0175 | Loss: 0.7598\n",
      "***Epoch: 021/040 | Train. Acc.: 76.339% | Loss: 0.674\n",
      "***Epoch: 021/040 | Valid. Acc.: 70.540% | Loss: 0.859\n",
      "Time elapsed: 21.70 min\n",
      "Epoch: 022/040 | Batch 0000/0175 | Loss: 0.7093\n",
      "Epoch: 022/040 | Batch 0050/0175 | Loss: 0.5893\n",
      "Epoch: 022/040 | Batch 0100/0175 | Loss: 0.6019\n",
      "Epoch: 022/040 | Batch 0150/0175 | Loss: 0.7325\n",
      "***Epoch: 022/040 | Train. Acc.: 78.268% | Loss: 0.623\n",
      "***Epoch: 022/040 | Valid. Acc.: 70.520% | Loss: 0.851\n",
      "Time elapsed: 22.73 min\n",
      "Epoch: 023/040 | Batch 0000/0175 | Loss: 0.6316\n",
      "Epoch: 023/040 | Batch 0050/0175 | Loss: 0.5694\n",
      "Epoch: 023/040 | Batch 0100/0175 | Loss: 0.7315\n",
      "Epoch: 023/040 | Batch 0150/0175 | Loss: 0.6656\n",
      "***Epoch: 023/040 | Train. Acc.: 79.455% | Loss: 0.589\n",
      "***Epoch: 023/040 | Valid. Acc.: 71.440% | Loss: 0.828\n",
      "Time elapsed: 23.77 min\n",
      "Epoch: 024/040 | Batch 0000/0175 | Loss: 0.5285\n",
      "Epoch: 024/040 | Batch 0050/0175 | Loss: 0.6959\n",
      "Epoch: 024/040 | Batch 0100/0175 | Loss: 0.5504\n",
      "Epoch: 024/040 | Batch 0150/0175 | Loss: 0.6831\n",
      "***Epoch: 024/040 | Train. Acc.: 80.174% | Loss: 0.570\n",
      "***Epoch: 024/040 | Valid. Acc.: 70.540% | Loss: 0.830\n",
      "Time elapsed: 24.82 min\n",
      "Epoch: 025/040 | Batch 0000/0175 | Loss: 0.6270\n",
      "Epoch: 025/040 | Batch 0050/0175 | Loss: 0.6128\n",
      "Epoch: 025/040 | Batch 0100/0175 | Loss: 0.5769\n",
      "Epoch: 025/040 | Batch 0150/0175 | Loss: 0.6409\n",
      "***Epoch: 025/040 | Train. Acc.: 80.933% | Loss: 0.547\n",
      "***Epoch: 025/040 | Valid. Acc.: 72.340% | Loss: 0.822\n",
      "Time elapsed: 25.85 min\n",
      "Epoch: 026/040 | Batch 0000/0175 | Loss: 0.5859\n",
      "Epoch: 026/040 | Batch 0050/0175 | Loss: 0.5577\n",
      "Epoch: 026/040 | Batch 0100/0175 | Loss: 0.6651\n",
      "Epoch: 026/040 | Batch 0150/0175 | Loss: 0.5483\n",
      "***Epoch: 026/040 | Train. Acc.: 79.150% | Loss: 0.589\n",
      "***Epoch: 026/040 | Valid. Acc.: 70.240% | Loss: 0.907\n",
      "Time elapsed: 26.90 min\n",
      "Epoch: 027/040 | Batch 0000/0175 | Loss: 0.6005\n",
      "Epoch: 027/040 | Batch 0050/0175 | Loss: 0.5660\n",
      "Epoch: 027/040 | Batch 0100/0175 | Loss: 0.6606\n",
      "Epoch: 027/040 | Batch 0150/0175 | Loss: 0.5047\n",
      "***Epoch: 027/040 | Train. Acc.: 81.045% | Loss: 0.539\n",
      "***Epoch: 027/040 | Valid. Acc.: 71.120% | Loss: 0.864\n",
      "Time elapsed: 27.94 min\n",
      "Epoch: 028/040 | Batch 0000/0175 | Loss: 0.5897\n",
      "Epoch: 028/040 | Batch 0050/0175 | Loss: 0.5210\n",
      "Epoch: 028/040 | Batch 0100/0175 | Loss: 0.5563\n",
      "Epoch: 028/040 | Batch 0150/0175 | Loss: 0.5192\n",
      "***Epoch: 028/040 | Train. Acc.: 83.891% | Loss: 0.464\n",
      "***Epoch: 028/040 | Valid. Acc.: 72.400% | Loss: 0.815\n",
      "Time elapsed: 28.98 min\n",
      "Epoch: 029/040 | Batch 0000/0175 | Loss: 0.5087\n",
      "Epoch: 029/040 | Batch 0050/0175 | Loss: 0.6121\n",
      "Epoch: 029/040 | Batch 0100/0175 | Loss: 0.5465\n",
      "Epoch: 029/040 | Batch 0150/0175 | Loss: 0.4414\n",
      "***Epoch: 029/040 | Train. Acc.: 82.757% | Loss: 0.493\n",
      "***Epoch: 029/040 | Valid. Acc.: 71.080% | Loss: 0.851\n",
      "Time elapsed: 30.01 min\n",
      "Epoch: 030/040 | Batch 0000/0175 | Loss: 0.5460\n",
      "Epoch: 030/040 | Batch 0050/0175 | Loss: 0.5083\n",
      "Epoch: 030/040 | Batch 0100/0175 | Loss: 0.4999\n",
      "Epoch: 030/040 | Batch 0150/0175 | Loss: 0.5453\n",
      "***Epoch: 030/040 | Train. Acc.: 83.397% | Loss: 0.469\n",
      "***Epoch: 030/040 | Valid. Acc.: 71.660% | Loss: 0.869\n",
      "Time elapsed: 31.03 min\n",
      "Epoch: 031/040 | Batch 0000/0175 | Loss: 0.4998\n",
      "Epoch: 031/040 | Batch 0050/0175 | Loss: 0.4808\n",
      "Epoch: 031/040 | Batch 0100/0175 | Loss: 0.4958\n",
      "Epoch: 031/040 | Batch 0150/0175 | Loss: 0.5201\n",
      "***Epoch: 031/040 | Train. Acc.: 84.167% | Loss: 0.447\n",
      "***Epoch: 031/040 | Valid. Acc.: 71.980% | Loss: 0.873\n",
      "Time elapsed: 32.07 min\n",
      "Epoch: 032/040 | Batch 0000/0175 | Loss: 0.3548\n",
      "Epoch: 032/040 | Batch 0050/0175 | Loss: 0.4062\n",
      "Epoch: 032/040 | Batch 0100/0175 | Loss: 0.4292\n",
      "Epoch: 032/040 | Batch 0150/0175 | Loss: 0.4786\n",
      "***Epoch: 032/040 | Train. Acc.: 85.027% | Loss: 0.430\n",
      "***Epoch: 032/040 | Valid. Acc.: 71.920% | Loss: 0.899\n",
      "Time elapsed: 33.10 min\n",
      "Epoch: 033/040 | Batch 0000/0175 | Loss: 0.4133\n",
      "Epoch: 033/040 | Batch 0050/0175 | Loss: 0.3402\n",
      "Epoch: 033/040 | Batch 0100/0175 | Loss: 0.3988\n",
      "Epoch: 033/040 | Batch 0150/0175 | Loss: 0.4555\n",
      "***Epoch: 033/040 | Train. Acc.: 87.281% | Loss: 0.373\n",
      "***Epoch: 033/040 | Valid. Acc.: 72.960% | Loss: 0.846\n",
      "Time elapsed: 34.13 min\n",
      "Epoch: 034/040 | Batch 0000/0175 | Loss: 0.4540\n",
      "Epoch: 034/040 | Batch 0050/0175 | Loss: 0.5704\n",
      "Epoch: 034/040 | Batch 0100/0175 | Loss: 0.5121\n",
      "Epoch: 034/040 | Batch 0150/0175 | Loss: 0.3992\n",
      "***Epoch: 034/040 | Train. Acc.: 88.328% | Loss: 0.340\n",
      "***Epoch: 034/040 | Valid. Acc.: 73.600% | Loss: 0.847\n",
      "Time elapsed: 35.17 min\n",
      "Epoch: 035/040 | Batch 0000/0175 | Loss: 0.3568\n",
      "Epoch: 035/040 | Batch 0050/0175 | Loss: 0.4348\n",
      "Epoch: 035/040 | Batch 0100/0175 | Loss: 0.3955\n",
      "Epoch: 035/040 | Batch 0150/0175 | Loss: 0.4242\n",
      "***Epoch: 035/040 | Train. Acc.: 88.502% | Loss: 0.338\n",
      "***Epoch: 035/040 | Valid. Acc.: 72.520% | Loss: 0.874\n",
      "Time elapsed: 36.20 min\n",
      "Epoch: 036/040 | Batch 0000/0175 | Loss: 0.2860\n",
      "Epoch: 036/040 | Batch 0050/0175 | Loss: 0.3446\n",
      "Epoch: 036/040 | Batch 0100/0175 | Loss: 0.4947\n",
      "Epoch: 036/040 | Batch 0150/0175 | Loss: 0.3412\n",
      "***Epoch: 036/040 | Train. Acc.: 88.482% | Loss: 0.325\n",
      "***Epoch: 036/040 | Valid. Acc.: 72.760% | Loss: 0.905\n",
      "Time elapsed: 37.24 min\n",
      "Epoch: 037/040 | Batch 0000/0175 | Loss: 0.3963\n",
      "Epoch: 037/040 | Batch 0050/0175 | Loss: 0.3388\n",
      "Epoch: 037/040 | Batch 0100/0175 | Loss: 0.3279\n",
      "Epoch: 037/040 | Batch 0150/0175 | Loss: 0.3965\n",
      "***Epoch: 037/040 | Train. Acc.: 90.065% | Loss: 0.294\n",
      "***Epoch: 037/040 | Valid. Acc.: 71.740% | Loss: 0.904\n",
      "Time elapsed: 38.27 min\n",
      "Epoch: 038/040 | Batch 0000/0175 | Loss: 0.3269\n",
      "Epoch: 038/040 | Batch 0050/0175 | Loss: 0.3193\n",
      "Epoch: 038/040 | Batch 0100/0175 | Loss: 0.3548\n",
      "Epoch: 038/040 | Batch 0150/0175 | Loss: 0.3414\n",
      "***Epoch: 038/040 | Train. Acc.: 90.460% | Loss: 0.280\n",
      "***Epoch: 038/040 | Valid. Acc.: 73.400% | Loss: 0.904\n",
      "Time elapsed: 39.30 min\n",
      "Epoch: 039/040 | Batch 0000/0175 | Loss: 0.2794\n",
      "Epoch: 039/040 | Batch 0050/0175 | Loss: 0.2914\n",
      "Epoch: 039/040 | Batch 0100/0175 | Loss: 0.3197\n",
      "Epoch: 039/040 | Batch 0150/0175 | Loss: 0.2898\n",
      "***Epoch: 039/040 | Train. Acc.: 90.545% | Loss: 0.276\n",
      "***Epoch: 039/040 | Valid. Acc.: 73.240% | Loss: 0.922\n",
      "Time elapsed: 40.33 min\n",
      "Epoch: 040/040 | Batch 0000/0175 | Loss: 0.3442\n",
      "Epoch: 040/040 | Batch 0050/0175 | Loss: 0.3222\n",
      "Epoch: 040/040 | Batch 0100/0175 | Loss: 0.4111\n",
      "Epoch: 040/040 | Batch 0150/0175 | Loss: 0.3112\n",
      "***Epoch: 040/040 | Train. Acc.: 89.846% | Loss: 0.292\n",
      "***Epoch: 040/040 | Valid. Acc.: 71.940% | Loss: 0.997\n",
      "Time elapsed: 41.36 min\n",
      "Total Training Time: 41.36 min\n"
     ]
    }
   ],
   "source": [
    "log_dict = train_classifier_simple_v1(num_epochs=NUM_EPOCHS, model=model, \n",
    "                                      optimizer=optimizer, device=DEVICE, \n",
    "                                      train_loader=train_loader, valid_loader=valid_loader, \n",
    "                                      logging_interval=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABMfklEQVR4nO3dd3jTVRfA8e/Nail7byhL9t6g7L1UFBARBQdu1NeFoAhOVBTFhRtRRFBRUUCWDNl7lL2hUEbZ0JVx3z+ShrRN27RNOsj5PE8ekt/KaYGc/O44V2mtEUIIEbwMOR2AEEKInCWJQAghgpwkAiGECHKSCIQQIshJIhBCiCBnyukAMqpEiRI6PDw8p8MQQog8ZdOmTdFa65Le9uW5RBAeHs7GjRtzOgwhhMhTlFJHU9snTUNCCBHkJBEIIUSQk0QghBBBLs/1EQghssZqtRIZGUlcXFxOhyICIDQ0lAoVKmA2m30+RxKBEEEmMjKSggULEh4ejlIqp8MRfqS15ty5c0RGRlKlShWfz5OmISGCTFxcHMWLF5ckcANSSlG8ePEM3+1JIhAiCEkSuHFl5u82aBKBze4gfNRcNh45n9OhCCFErhI0ieCl2TsAuHPKmhyORAihlGLo0KHu1zabjZIlS9KnTx8A5syZw4QJE9K8xsmTJ7nzzjsBmDp1Kk888USGYnjrrbfSPWbYsGH8+uuv6R5XoECBDL13bhM0ieC1W+vldAhCCJf8+fMTERFBbGwsAIsWLaJ8+fLu/f369WPUqFFpXqNcuXI+fUinxpdEECyCJhHksxipXipvZ20hbiQ9e/Zk7ty5AMyYMYPBgwe793l+wx82bBgjR46kTZs2VK1a1f3hf+TIEerVu/4F7/jx4/To0YOaNWsyfvx49/bbbruNpk2bUrduXb788ksARo0aRWxsLI0aNWLIkCEATJs2jQYNGtCwYcMkdysrVqxI8d6p0Vrz/PPPU69ePerXr8/MmTMBiIqKol27djRq1Ih69erx33//YbfbGTZsmPvYSZMmZfp3mVVBNXy0VMEQDpy5itZaOsuEAMb/tZNdJy/79Zp1yhXi1b510z3urrvu4rXXXqNPnz5s376d+++/n//++8/rsVFRUaxcuZI9e/bQr18/d5OQp/Xr1xMREUFYWBjNmzend+/eNGvWjG+//ZZixYoRGxtL8+bNueOOO5gwYQKffPIJW7duBWDnzp28+eabrFq1ihIlSnD+/PkMvXei2bNns3XrVrZt20Z0dDTNmzenXbt2/PTTT3Tv3p0xY8Zgt9uJiYlh69atnDhxgoiICAAuXryY7u8sUILmjgDg2PkYAK7G23I4EiFEgwYNOHLkCDNmzKBXr15pHnvbbbdhMBioU6cOp0+f9npM165dKV68OPny5aN///6sXLkSgMmTJ9OwYUNatWrF8ePH2b9/f4pz//33X+68805KlCgBQLFixTL03olWrlzJ4MGDMRqNlC5dmvbt27NhwwaaN2/Od999x7hx49ixYwcFCxakatWqHDp0iCeffJJ//vmHQoUKpXntQAqqO4KHbqnKq3N2kmBz5HQoQuQKvnxzD6R+/frx3HPPsWzZMs6dO5fqcSEhIe7nWmuvxyS/y1dKsWzZMhYvXsyaNWsICwujQ4cOXsfYp9VK4Mt7p7e/Xbt2rFixgrlz5zJ06FCef/557r33XrZt28aCBQv49NNPmTVrFt9++22a1w+UoLojCDE5f9wEuyQCIXKD+++/n7Fjx1K/fv0sX2vRokWcP3+e2NhY/vjjD9q2bculS5coWrQoYWFh7Nmzh7Vr17qPN5vNWK1WADp37sysWbPcycizaSgj2rVrx8yZM7Hb7Zw9e5YVK1bQokULjh49SqlSpXjooYd44IEH2Lx5M9HR0TgcDu644w5ef/11Nm/enOXfQWYF1R2BxZUI4q2SCITIDSpUqMBTTz3ll2vdfPPNDB06lAMHDnD33XfTrFkz6tevz5QpU2jQoAE1a9akVatW7uNHjBhBgwYNaNKkCdOnT2fMmDG0b98eo9FI48aNmTp1aoZjuP3221mzZg0NGzZEKcW7775LmTJl+P7773nvvfcwm80UKFCAadOmceLECYYPH47D4fw8evvtt/3ye8gMld6tTm7TrFkzndmFaeZuj+Lxnzaz8Jl23FS6oJ8jEyJv2L17N7Vr187pMEQAefs7Vkpt0lo383Z8UDUNJd4RHI6+lsORCCFE7hFUicBkdHYGPfzDphyORAghco/gSgQGmTsghBDJBVUiMMokMiGESCGoEkE+izGnQxBCiFwnqBJBiQIh6R8khBBBJqgSQeKEMiFEzjIaje4CbH379g1InZ0pU6Ywbdo0v1/3RhRUn4xm4/UfN95mz8FIhAhu+fLlY+vWrURERFCsWDE+/fRTv7/HI488wr333uv36/qLzZZ7ap4FVSIweIwaGj07IgcjEUIkat26NSdOnACgQ4cOJE4YjY6OJjw8HHCWpe7fvz89evSgRo0avPDCC+7zCxQowJgxY9yF5RILw40bN46JEye6r/viiy/SokULbrrpJneV05iYGAYOHEiDBg0YNGgQLVu2xNuE1ddee43mzZtTr149RowYgdaa3bt306JFC/cxR44coUGDBgBs2rSJ9u3b07RpU7p3705UVJQ7jtGjR9O+fXs++ugj/vrrL1q2bEnjxo3p0qWLO/azZ8/StWtXmjRpwsMPP0zlypWJjo4G4Mcff6RFixY0atSIhx9+GLs9619qg6rERJhHZ/Gmo7JkpRDMHwWndvj3mmXqQ8+0VxdLZLfbWbJkCQ888EC6x27dupUtW7YQEhJCzZo1efLJJ6lYsSLXrl2jVatWvPnmm7zwwgt89dVXvPzyyynOt9lsrF+/nnnz5jF+/HgWL17MZ599RtGiRdm+fTsRERE0atTI63s/8cQTjB07FoChQ4fy999/07dvXxISEjh06BBVq1Zl5syZDBw4EKvVypNPPsmff/5JyZIlmTlzJmPGjHEXlLt48SLLly8H4MKFC6xduxalFF9//TXvvvsu77//PuPHj6dTp0689NJL/PPPP+51FHbv3s3MmTNZtWoVZrOZxx57jOnTp2f5zieoEoFn05AQIuckLgpz5MgRmjZtSteuXdM9p3PnzhQuXBiAOnXqcPToUSpWrIjFYnEvcdm0aVMWLVrk9fz+/fu7jzly5AjgLBudWOuoXr167m/0yS1dupR3332XmJgYzp8/T926denbty8DBw5k1qxZjBo1ipkzZzJz5kz27t1LRESE+2ey2+2ULVvWfa1Bgwa5n0dGRjJo0CCioqJISEigSpUq7rh+//13AHr06EHRokUBWLJkCZs2baJ58+bu32OpUqXS/d2lJ6gSgad4KUUthM/f3P0tsY/g0qVL9OnTh08//ZSRI0diMpncRdiSl4v2LAdtNBrdbexms9ldQtpze3KJ53se40uttbi4OB577DE2btxIxYoVGTdunDu2QYMGMWDAAPr3749Siho1arBjxw7q1q3LmjXe10fPnz+/+/mTTz7J//73P/r168eyZcsYN25cmnFprbnvvvv8XqAuaL8iR12KY//pKzkdhhBBrXDhwkyePJmJEyditVoJDw9n0yZnCZisrEfsq5tvvplZs2YBsGvXLnbsSNlMlvihX6JECa5evZokrmrVqmE0Gnn99dfd3/Rr1qzJ2bNn3YnAarWyc+dOr+9/6dIl91rN33//vde4Fi5cyIULFwDnXdGvv/7KmTNnAGe57KNHj2b+F+AStIkAoOukFTkdghBBr3HjxjRs2JCff/6Z5557js8//5w2bdq4O0cD6bHHHuPs2bM0aNCAd955hwYNGribnxIVKVKEhx56iPr163Pbbbe5m2USDRo0iB9//JGBAwcCYLFY+PXXX3nxxRdp2LAhjRo1YvXq1V7ff9y4cQwYMIBbbrnFvToawKuvvsrChQtp0qQJ8+fPp2zZshQsWJA6derwxhtv0K1bNxo0aEDXrl3dHdFZEVRlqAHCR81N8vrIhN5ZDUmIPEXKUF9nt9uxWq2EhoZy8OBBOnfuzL59+7BYLDkaV3x8PEajEZPJxJo1a3j00Ufd6yv7IqNlqIO2j0AIIWJiYujYsSNWqxWtNZ9//nmOJwGAY8eOMXDgQBwOBxaLha+++iqg7yeJQAgRtAoWLOh13kBOq1GjBlu2bMm29wtYH4FSqqJSaqlSardSaqdSKsV6dMppslLqgFJqu1KqSaDiEUJcl9eahIXvMvN3G8jOYhvwrNa6NtAKeFwpVSfZMT2BGq7HCODzAMYjhABCQ0M5d+6cJIMbkNaac+fOERoamqHzAtY0pLWOAqJcz68opXYD5YFdHofdCkzTzn+Ra5VSRZRSZV3nBkS1kvk5eFaWqhTBq0KFCkRGRnL27NmcDkUEQGhoKBUqVMjQOdnSR6CUCgcaA+uS7SoPHPd4HenaliQRKKVG4LxjoFKlSlmKZdoDLWk74d8sXUOIvMxsNrtnsAoB2TCPQClVAPgNeFprfTn5bi+npLhf1Vp/qbVuprVuVrJkySzFU75IviSvD0dfY8b6Y1m6phBC5GUBvSNQSplxJoHpWuvZXg6JBCp6vK4AnAxkTMl1nLgMgMEtsnanIYQQeVUgRw0p4Btgt9b6g1QOmwPc6xo91Aq4FMj+AbR2PoQQQrgF8o6gLTAU2KGU2uraNhqoBKC1ngLMA3oBB4AYYHjAotkzF+aMhIeXe92ttXYXrhJCiGASyFFDK/HeB+B5jAYeD1QMSYQVh5joVGuvaw2SB4QQwSh4is6Vruv881QElYuHpdgtDUZCiGAVPIkgpCAUqwqnthNqMqbY7ZC+AyFEkAqeRABQuh6cjuD+m8NT7JI8IIQIVsGVCMo0gPOHGNSgaIpdf249wdkr8TkQlBBC5Kzgqj5apr7zz9MpVwt6/tftADSqWIRGFYswrl/d7IxMCCFyTJDdEbgSwakdPN+9ptdDth6/yNTVR7IvJiGEyGHBlQgKlYN8ReHUDkJMwfWjCyFEaoLr01Ap513BqR0ySkgIIVyCKxGAs8P4zC6U3ZrTkQghRK4QfImgYkuwxVHiyq70jxVCiCAQfImgclsAyl3YkMOBCCFE7hB8iSB/cShdj3IXct+C1UIIkROCLxEAhN9M2cvbsJB6P4Gs5yqECBZBmghuweSIp5E6kOohm45eyMaAhBAi5wRnIqhyC9pgor1xW6qH2B1yRyCECA7BmQhCC6MqtaaTYUuqh+w4cSkbAxJCiJwTnIkA4Kbu1DYcZ2gt76vRvDF3NwfPXs3moIQQIvsFcSLoCcDgIrtTPUT6CYQQwSB4E0GJ6lC8OlXOLkn1kFf+iMjGgIQQImcEbyIAqN2PfCfXUohrXnfH2xyM/2snkRdisjkwIYTIPsGdCGp0BW2nrSH1b/7frTrCMzO3Zl9MQgiRzYI7EVRoAaGF6WLclOZhDg0r90dzMSaBkxdj2R11OZsCFEKIwAuuFcqSM5qgVl9u2/kHo60JxGPxetimoxe455t1lChgIfpqAgCv31aPoa0qZ2e0QggREMF9RwBQrz9G61W6mXeke2hiEgD4YvnBQEYlhBDZRhJBlfYQVoLXq+/N0GkOmXkshLhBSCIwmqDubRQ5voQw4nw+zS5F6YQQNwhJBAD17gBbLONqHPL5lOirCRw4IzOPhRB5nyQCgIqtoFg1BqrUJ5clZ3dounyw3P3aancEIjIhhAg4SQQABgM0GQrH1lBDRWb49KV7z1BjzHy2R170f2xCCBFgkggSNb4XzGE8ZJyb4VOX7TkDwGapTSSEyIMkESTKXxwa3c1txpWU4VyGTlXKWcHUW/ex1e7g1k9XsfpgtB+CFEII/0s3ESilnlBKFc2OYHJc26dQwEOmeZk6fdGu0ym2nbwYy7bjF3nxt+1ZDE4IIQLDlzuCMsAGpdQspVQPlfj190ZUpBLmRoMYbPyXYvhWRsJzbePVB1PeSShu3F+XEOLGkG4i0Fq/DNQAvgGGAfuVUm8ppaoFOLaccfPThKoEhpn+8enwKi/NI95mT3V/YtqUaQdCiNzKpz4C7fzae8r1sAFFgV+VUu8GMLacUbIme4q0Z5hxIQXwrfz0+WsJSV5rrVMkB0kEQojcypc+gpFKqU3Au8AqoL7W+lGgKXBHgOPLEf+VuZdCKoahxsU+HR9rTTqHYOg366n58j/sPCnrHgshcj9f7ghKAP211t211r9ora0AWmsH0Ceg0eWQhi06sNzegAdM83y6K1ix72yS1ysPOEcILYg45dE0JLcEQojcyZc+grFAcdedwZNKqSYe+1Jf8DcPa1W1OO/bBlCMK/zP9GumrzNt7VFu5L51IcSNwZemoVeA74HiOO8OvlNKvezDed8qpc4opbwu/6WU6qCUuqSU2up6jM1o8IF0zx2386u9HUOMS6ioUg4L9cXFGKt7zJDcDwghcitfmobuBpprrV/VWr8KtAKG+HDeVKBHOsf8p7Vu5Hq85sM1s83AZhV53zYAGwbGm75HkblaQrtOOoehSsuQECK38iURHAFCPV6HAOmuyqK1XgGcz1xYucNpivGu7S46GbfyqXkymfle/+C0jf4PTAgh/MiXRBAP7FRKTVVKfQdEAFeVUpOVUpOz+P6tlVLblFLzlVJ1UztIKTVCKbVRKbXx7NmzqR3mdy2rFON7ezc+sd1KL+N6hhjTr05qT2XBGi2NQ0KIXMqXNYt/dz0SLfPTe28GKmutryqlegF/4Jy4loLW+kvgS4BmzZpl2ydqnwZlWXf4PBNtA2mi9vOC6WcW2JsTTeFUz6k22nt5CmkaEkLkVr6MGvoemAFscj1+0lp/n/jI7BtrrS9rra+6ns8DzEqpEpm9XiDc416cXvGKbTiFVQyfWj7K1LU880D4qLm8+uf1PvSr8Ta+XHFQlr8UQuQIX0YNdQD2A58CnwH7lFLtsvrGSqkyiXWLlFItXLFkrOxngHkO/Tyoy/OTrSMtDXtorPZn+FrJ7wi+X3PU/fytebt5a94eFnopWieEEIHmSx/B+0A3rXV7rXU7oDswKb2TlFIzgDVATaVUpFLqAaXUI0qpR1yH3AlEKKW2AZOBu3Qun3X1hm0o0boQz2RhboE3V+JsAGnWLBJCiEDxJRGYtdZ7E19orfcB5vRO0loP1lqX1VqbtdYVtNbfaK2naK2nuPZ/orWuq7VuqLVupbVenfkfI3A2vtzF/TyGUKbY+tLOuIPbDf9l8ErOHBdxImXZiVye/4QQNzhfEsEmpdQ3rglgHZRSX+HsKwgKJQqEJHn9g70rGxw3McH8NU3UPp+vozXM2nicPh+vTLnP9efBM1e5Emfl4yX7Ux19JIQQ/uZLIngE2AmMBJ4Cdrm2BZWCIc4BVvFYeCThGU7o4ky1vMNN6rhP55+7lsALv3pfnCbyQiwAk/89QNM3FvP+on0s2HnKP4ELIUQ60kwESikDsElr/YHWur/W+nat9SStdXw2xZcrLH2uAyte6Oh+fY7CPGR9lgTMfGH+gBAS0jg7ffHW630DCTbnDGbpLxBCZJc0E4Grwug2pVSlbIonV6pSIj9F81uSbDuoyzPS+gRVDKf5wzKWzMw6XrbXuej9nlNXUuxzZK6ihRBCZJgvTUNlcc4sXqKUmpP4CHRguVGVEvmTvF7tqMcye0NqG47xiPGvDF/v4R82pTp34NlftjF11eFMxSmEEBmh0huxopRq72271np5QCJKR7NmzfTGjTlTv8dqd/DtysO8PX+Px1bNx+aP6WHYwB0J49iuM7aC5+AWFZmxPvV+hiMTegOw99QVDpy5Su8GZTMTuhAiyCmlNmmtm3nb58sdQS+t9XLPB9DLvyHmDWajgYfbJ/+gV4yxPsBpivKR+RPCiMvQNdNKAp66f7iCx3/anKFrCyGEL3xJBF29bOvp70DykokDGvLP07dwb2tnCYrL5OeZhMeopM7wtvlr/L36wNI9Z/x6PSGE8JRqIlBKPaqU2oFzZvB2j8dhYEf2hZj73Nm0ArXKFEqybYOuxSTbndxqXM18y0uEqyi/vNe1eBvDp25wv9ZayxwDIYRfpVV99CdgPvA2MMpj+xWtdZ5eZ8Bfki9C+an9Voqryww3LeAXy3g6xE/iGvmy9B7nryUdmlrlJWd108Nv95JlMIUQfpHqHYHW+pLW+ojWejAQCVhxtnkUCPbhpIluqVHS/Xzx/9qhMTDedh8D41+hpLrMaNNPWX6P+z3uBjyN+CFoJncLIQLMl+qjTwCngUXAXNfj7wDHlSd0qVPa/bx6qYLcUsNZRXu9rs0UWx+GmJYw2IfFbNKy/8xVr9sX7TrNK39EcPpyxjqnhRAiOV8WpnkaqKm1zlUlonOjMoWur+j5ru0uaqrjvGaaSgFi+crem5SNSVnzw9qjRF2K4+v7vI4Iy7Tqo+fRq35ZJg9u7NfrCiFyJ19GDR0HUpbMFCmEmo3u5w4MPGV9gu26KmPMP7HA8iKVlP/XG9Ba88m/+zl+PsZv17Q5NHO2nfTb9YQQuZsvieAQsEwp9ZJS6n+Jj0AHlhflsxiTvL5Mfu5NGMU8ewtqGiL5xTKeauqEX9/z5KU4Ji7cl2RkkRBCZIQvieAYzv4BC1DQ4yGAJc+259thzqaZpzpfX3K5QlHnaKFr5OMx69N0jX+XUBL4wfI2NVSk395/d9RlAGIT7ISPmstHi52rp526FIfNLgWLhBDpS7ePQGs9Pvk2pZQvfQtBoVrJAlQrWQCA/CHXfy19G5bj9KU4Zm9x3gHs1xW4L2EUP1te52/LaJ6xPsY8Ryu/xZHYaTxp8T6W7TvDlmMXuadVJd64rb7f3kMIcWNKa0LZSo/nPyTbvT5gEd0gnu9Wk7F96yTZtlVXp2P8B+zWlfnMMpkxph+xYPXL+9k8JpltOXYRgCW7ZUayECJ9aTUNeZbarJdsn8xkSofBoJJ0HieKojgDEl5lmq0rD5nmMcsyngL4r6M3yXtdiuNSTOqJZnfUZXdTEsCqA9Hu9RCEEMEjrUSgU3nu7bXw0LlWqTT3WzEx1jackQlPUF8dZqbldeqpQwGJpd+nKZfGTHTbp6uYtHgfCTYH83ZEMeTrdbzzz55UjxdC3JjSausvopS6HWeyKKKU6u/aroDCAY8sj9r9Wg/MRt9umOY42oAVJls+4e+QlxmcMIY1jrp+jefouet3Gw9+vwGlFO/c0YAwi9HdnPT5soNMWuxcf3nVgehUr/Xp0gPcXL0EDSsW8WuMQoiclVYiWA7083je12PfioBFlMclH0KaqG/DcvzlZWz+HEcbtsdXYaHlBWZY3uQPexs+st3BYe2/dQf2nrrCuDk7WXPIOSdw0a5FNK1c1F28zvPD/+TFWK/X+HPrCd5bsJf3Fux1r5EghLgxpJoItNbDszOQG5HFaKBSsTCe7lKDckXyeU0EAEd0WZrGf8Fsy6vcZlxNT8MGxtju5zf7LWifRvimrfuHKfP2pqMX3M/tHosTeRY2tTs0RoPz7ubXTf4b8iqEyF2y/ikjUmUwKFa80JH+TSrQqmpxjkzoTfVSBbwee4Uwuia8R5f4dzmoyzHR/AV/W8ZQRx0JeJyeScHhkRTm7fBPKW0hRO4miSCbpdd7cEBXoHfCm3xp601dw1HmhYxmoeV5uhmyZ+ZwTILd/dxzBJEjnSVNE9kdmjirPf0DhRC5hiSCXEhj4C3bEFrFfcx+R3luMpzgS8skFlueo5E6kI1xgM3uQGvNukPXl6DYeTL10lMPTdtIrVf+yYbohBD+4ksZ6gFKqYKu5y8rpWYrpZoEPrQbU+tqxX0+9hTF6ZrwHjXjpvKdrTvVDSf5I2Qs40xTMRD48f5frThE9THzef3v3UkmrO2JupLqOf/KsppC5Dm+3BG8orW+opS6GegOfA98Htiwblyv9KnDv8+2Z9WoTiz+XzufzonHwnjbfdwc/xHTbF0ZZlrIodB76GdYTQgJ6V8gk/aedn7g/7j2aMDeQwiR83xJBIkNvr2Bz7XWf+IsQCcywWw0ULVkAcoXyUf1Uhmr3RepSzLWNpzpts6Ac/7BqpCR9DCsJzvn+J26HMfSPWeIs9pxODTfrDxMq7e8L8DjcGgW7jyF9rGPQQiR/XwpHndCKfUF0AV4RykVgvQt+M3MEa0Y9OXaDJ0zxvYAL9uGc4thB+NM3zPF8iEbHDex0xHOG7Z7sPn015p57y3Y635+X+vKfL8m9TuGaWuOMO6vXUwa1JDbG1cIaFxCiMzx5QN9ILAA6KG1vggUA54PZFDBpGXV4pQrHMrrt16fUfzRXY3SPU9jYIWjIT0S3uE161AaqYMMMy1kVchIOhk2+zXGhDTKWf+0/pj7ebt3l6bYf/KSsyrqmcvxLNh5ipX7U5+5LITIGb4kgrLAXK31fqVUB2AAUn3Ur1a/1JmhrcPdr3vUK+PzuQmY+dbekybxXzDL1p4iXONby0S+Mb9Hw2wcYQRwLJ1V0h7+YRP3fLMum6IRQvjKl0TwG2BXSlUHvgGqAD8FNKogF2K6XqZi7xs9fDrnCmG8YHuY9vEfYNVGOhu38GfIWF4x/YAK4Agjq9172//c7alPRjt49mqgwhFCZIIvicChtbYB/YEPtdbP4LxLEH726d1NmHJP0pG5ISYjb/f3fXGZUxSnRvwPtIj7lO2OKjxgms/h0HuYan6Hh4x/kx/vtYT8bdbG4wB8uSJlVdXO7y/P1DXjbXa2Hb+Y5jEzNxxjnaumkhDCN74kAqtSajBwL/C3a5s5cCEFr94NytKjXsoce2ujchm+1hmK0i/hDV62DueaDqGDcRtjzD+xM/QBJps/pqthoz9C9tnMDcdTbDt1KY6X/9hBjw9XcO5qfLrXGDdnF7d+uopj51Jvgnrxtx0Z7nwXItj5kgiGA62BN7XWh5VSVYAfAxuWeKJjdUa61kAOs5j4/bE2mbiK4kd7V+rGf0eNuGn8L+ERYnQIbQ0RfGX5gM/MH1KWwHx7VslqaRyKvpbk9fwdUbR6ewk/rj3GnlNXeGn2jnSvmXg3cDkua6u6xSbYuRZvy9I1hLiR+LJm8S6l1HPATUqpesBerfWEwIcW3J7rXjPJ6/JF8mXpelZMzHa0Y3Z8O4zYedz4J0+afqd1yC4esz7l93UQlu09S/iouanu35KsiSfWamf+jiiuxNkY2Lyi13OyOhPhvQV7aFejJCN+2MSlWKuU0xbCxZcSEx2A/cCnwGfAPqWUb1Nihd+UKhTKrte6p3ucL/0JdoxMtvenX8IbROvCzLC8ycqQkdykUjbfBIrdkfRj/b/90Tw6fTMv/LY91XMSJ6WdvBjLxZiMz6j+dOlBBn25lkux/lknWogbhS9NQ+8D3bTW7bXW7XCWmZiU3klKqW+VUmeUUhGp7FdKqclKqQNKqe1Svyh9YZakN3ANKqRcKK57Xd+Hnu7WlRmaMIqjjlJUUNEsDHmRXy3j+Nw8KaClKyDtldAS7yTWHDxHbELKSqYjfthEh4nLeHPuLjq9vyxQIQoRNHxJBGattXsqqdZ6H751Fk8F0hr72BOo4XqMQOoXZdicJ25Osc3g2yqZbqcoTvuED+keP4Hpts40UfvpadzA3tBhtDbs9FOkKZ25knbn8LFzMQz+ai0vzb5+h+BZpeJijJWv/jvMobPXvJwthMgIXxLBJqXUN0qpDq7HV8Cm9E7SWq8AzqdxyK3ANO20Fue6yDIsNR3v3dmA57vXdA8zDTUn/StUHr20ISYDW17p6tN19+pKjLE9QPX4H1jnqAXADMubfGH+gIeNf/GSaToV1Fk//RRw/lradxyJHcL7Tl+fc5Damgie6yYIITLOl0TwCLATGAk8Bexybcuq8oBno3Ska1sKSqkRSqmNSqmNZ8/678MoLxrQrCKPd6zuHma6a3wPyhQKde/3vCNoXKkIRfNnrD6gAwODEsbSLf4dtjqq0t24kZfMM3jYNJf5llG0VLv98nNkRmqdxTe9PJ9j52I4HJ2xu4N5O6IkiQhBOolAKWUANmmtP9Ba99da3661nqS1Tn/Qd/q8NWJ4/b+utf5Sa91Ma92sZMmSfnjrG4fBoFg7ujMVijpHFRmU4qFbqgDQtlqJTF93n67IbQlv0DhuCoMTxtA7/i0UmpkhrzPSOJvsrHaa6MCZ1Gckj/ljBx0nLvO6b9neM9T2sljOY9M38/6ivV7OECK4pJkItNYOYJtSqlIA3jsS8BwnWAHwvrq7SNfX9zXjuW43kT/ExOhetfnmvmY82qEaAE92qp7p616gEGscddmpw7k5/iMOOsryP/OvvG36OqClK7ZHOldB2xV1mfBRc3lz7q40j/8vWTG7mITr8wTeX7iP2FSWzzzlKoonRDDztejcTqXUEqXUnMSHH957DnCva/RQK+CS1lpWS8+kWmUK8UQn5wQ0pRSda5fGZHT+9T7brSYtqxTL8ntcpCBdEt7jE9utDDYt5TfLOKqoqICslvbOP3uSvP7qv8MZOr/O2AXu5zoH7l6EyEt8KVw/PjMXVkrNADoAJZRSkcCruEYbaa2nAPOAXsABIAbnDGYRIIbkU30zSWNgom0gAE+Y/mRpyLMAfGjrz2Rbfxx+WqrCn2P9I05c9tu1Eg39Zh1rD51j/5u9/H5tIbJbqonAVW20tNZ6ebLt7YAT6V1Yaz04nf0aeNzHOEUWGXz4fC5bOJQon5pKFBNtg1jlqEcHw1aGGRfytGk2A4zLiXBUYamjET/bO2U5Zn9YvOt0mvsTByIl2BzYHZp8luuVX4+fj8FoUBQNszD4q7W8eXs96pZzzt1I3hQlRF6W1sfDh4C3VcpjXPtEHvJKnzrpHtO3YcaK261x1OVt2xBqxk9lZMLjXNAFaW/YxgTz1/xneYotISP40vw+w43zyYnO5YgTl3hwWtrF9eZsO8mZK3H0mvwftccm7VC+5d2ltJnwL6/9vYutxy/yxt85N2JKiEBKKxGEa61TzPfXWm8EwgMWkQiIWmUKed3+vEdNo8w3HinmONrSJ+EtmsR/wVJ7QyoazlJUXaWbcROvmn9gb8gwppgnUVdlrK0/K/ad9vY9JqUHpm5MMSLJc43lGa5V2FKbxyBEXpdWH0FoGvuyVgFN5AiL0ZBk2ckPBzXitsblMRoUE+Y7O2cNChxZ+LyLIZTh1hepYovirC5MHBaeNP3BU6bZ9DBuoKNhK2Ntw5hp70BWUo8vfP3c3nHikk/nShoQN6q07gg2KKUeSr5RKfUAPswsFrnPgmfa0arq9dFD4SXyA0k/9A693Tvdqpzf398i3fc6rMtylTBsmJhku5Pacd/yaMJTbNNVecf8FetCHudB41wC+fGa2W/wh85eZcHOUyl3pHK5Z2dtY/bmyEy9lxC5QVp3BE8DvyulhnD9g78ZYAFuD3BcIgCqlMjP9AdbUW30PMBLXSIvX9B3vdY9yVBMALMx49/kYwllvqMl/yQ0Z7hxAQOMy3jZPJ1bjat42vo4B7XXSeVZsmxfxmehn7kcR6dUVlBbfyRlxZRNR8/z2+ZIftscyeVYKwObV3QXB7TZHcRY7RQKlXWcRO6W6h2B1vq01roNzuGjR1yP8Vrr1lprL1+XRF5gNCjqlnP2FyjXJ3+fBmUxGRQDmlZwH/f3kzfzcu/aKSqeArSuWjzT768x8K29Jz0TJvCTrSP1DUdYEvI8m0NG8ItlHMON8zHjn0Vj0lo3OTXemonScsfna9zPx/21i7fnXZ//8PTMrTQYt5AEm4MVmUhKQmQXXxamWQoszYZYRDZJbDFJnFpQsVgYB95KOh6+XvnC1CuftMz1mpc6UTTMkqSwXeYpRtse4k/7zTxh+p2y6jzNDftobtjHq+YfmG9vzh5HJWwYWepoxF5dETvG9C+bRXO2ZW1y+wXXOgnfrTrM365ENHHhXr5ccYhfH2lNs/DrTXMOh+aJGZu5v22VJNuFyG6+TCgTN5haZQuyK+oyhfNlrMmibGH/jxFYp2uzzlobgFDiud84n1uNq+lp3EBP4wYAnmcWV3UoGxw1+dvemr8crUkI0LLZ6U1kuxRr9en3Nv6v6yUxEktl/xNxincX7GX6gy0xGw38tjmSeTtOsfbQeTb7WCVWiECQRBCE3rq9Pne3qETFYmE+He/LSKKbq5egaeWifLRkf6bjiiOEz+y38Zn9NkpykbqGI4CmrDrPncYVtDbsoqNxG2P0j8yyd+Qj2+3Epjm4LeOW7U27CWfqqiNMWrwv1f1aw6ajF5Jsu+Iqqf31SufQ2SPR1/hw8X7m7ohynSPjkUTOUnntH2GzZs30xo1pTxIS/nXgzBW2HLvIgGbXawQmX4/4jiYVeH9gQ+qPW8CVuMAsDB9CAr0Naxls+pfmBueH8Q+2LsxztGSXozKXKBCQ9/W37nVLs2Dn9RnPRcPMbBnbLQcjEsFAKbVJa93M2z7/FIYRN7TqpQomSQIA3w1rnuR1YmG3/F46l/0lHguzHe0YkPAqH9tuY7ejEkNNi5lheZNtoSM4Eno3dxuXBOz9/cUzCQBciLG6J60JkRPkjkBk2oEzVyicz0LXScuZ/mBL6pYrzMGzVxk9ewfrDqe+OF2TSkXYfOyiX2JopvYwyLiMyobTtDA41xb4096GL229uaALcpqi2dLJ7A+e8zccDs2ZK/GUKZy1pq+526OoXbYgVUvmjbslEThp3RFIIhABUWPMPKx2zdNdajB3exT7z1ylYrF8zBzRmod/2JThYZq+KEgMb5u/po9xrXubXSt+d9zCx7bbOKrL+P09/ckzEUxatI+Pluxn5YsdqVA09b6cXScvc+TcNXrVT7nK6/bIi/T7ZFWKa4vglFYikM5iERCe5ZmHtKxM10nLmXJPU8oVyUe3OqXZceIS7w9oyLO/bPPbe14hjCesI3nTOoTuxg20NURwjVB6G9ZyZ8gKDjrKMs52H4d1GSJ1Kb+9r794fnBXL+X8Bv/srG0UDbPw2ZAmGFLMAIRek/8DvH/QJ15LiPTIHYHIdg6H5uj5GKqUyJ+i0zkQSnGBe00Lud/4D2HKucpqhCOcKF2Mcuoc53QhbjZEMMr2ILPsHQMeT2oKhppS7Wh/oUdNHuuQcqW5xN9fYiKITbCjFISajUl+t9MfbMmQr9exfnRnShXKeHPT6gPRrNgfzaietTJ8rsgdpLNY5CoGg6KKq85Rcm2rZ37WcmrOUJSJtkG0iv+YN6xDAKhnOEIjw0EcKFoa9pCAiXfNXzHM+A85VV4urdFWO0+mvbjOr5uctY5qj/2Hpq8vSrH/+9VHAJL0zdjsDqavO4rNnv4Kc3d/vY4pyw+me5zIm6RpSOQqXw5tRt1XF6R/YCZcpgBf23vztb0XpbnAaYriLLCkCSOenyxvMs48jWdNvxCpS1BZnWFgwitE6KoBiSezrsbbiLPaKVEgxL1t0a5T3OkqEXItwfv6zMlNW3OU1/7eRYLNwfC2VbweE5Ng81pmRNxY5I5A5BpHJvQmf4iJcX29L6Izupe/miUUpynG9Sp7ihhC6Z8wnhesDxGHmXLqHCZs/GEZy/fmCdRXh/z03pmk4c+tJwgfNZf64xbQ7I3FSb7J+/Cl/vqFXC66ZlGnNpv64Nmr1Bm7gF82Hs9s1CKPkFQvcp2wkOv/LI0Ghd01rTlxmchAcWBglr2ju5+gJBf4I2Qs7Y3bucWwg/dsg/jH0ZxG6gAXKcBSRyMCvaZCork7ojxmIju3tX9vmXu/3ZF2JkirPFRq3YSJi/UsTGe5z0C4Gm8jNsFOyYIh6R8sskwSgch1Ej+z+jcpz9g+dWg74V++G96CppWLZmscZylKu/gPqaOO8ohpDi+af+ZFfk5yzG/2W3jNOpRL5Ce7kkKiExdj3c/jbWknAl/GhJy7Go9da0oVdHYmG13Zw56VlYoyqcv7yzl1OU6GvWYTSQQiR80beYt7CGQid3VTDUXCLOx8rYd734s9arEr6jJ/pVEltELRfEReiE11f0bYMbJDV+VJ60jm2ddTXF3CiIOHTHMpzQXuMP7HHcb/2Oyozmjrg+zX5XNkAtvqg+fSrFmU+K3+cpyNod+s47/90YSYnC3Dib/upm8sBpxNdJdire71nvee8m3Jz+S6frCcQc0r8uAtGe9jOXU5LlPvKTJHEoHIUXXKpVxLuVh+Z3XP0l5m1T7aoRpAqomgQYXCzH60DdXHzPdjlM5mo7mOVu7X39l7AtDRsIXnTbNoYjjAPyGjANjsqM5s+y3MsncIWJVUb96atzvdYyYt2kfUJeeHbOJdRPL8EZNg48+tJ9yvPe88LsVYKRhq8jqnIbn9Z67yxtzdmUoEIntJZ7HIdTrWLMXkwY15pstN6R77yyOt3c8PvdWLPx5ri8lo4OXetQMZottSR2N6JbxNg7gvecM6hAhHODXUCd4wf8fukGHMML/BR+ZPeM80BZOfFtxJzVf/HXY/v/WTlV6PSUwCnv7bn7Tiap2xCzjo6h9IruFrC5m4cG8WohS5kdwRiFxHKUW/huV8Ora5x4Iunt9Sk3cyViuZn4OudQGS61izJBaTIUUxuIy4PjS1NxasPGCcT0vDbjoYt+HQCoPStDHu5LQuyo+2Lsx23EIg+xS2RfpewsNb3ad9p70nAnCuq/BCD5lYdiORRCBy3Lt3NqBASOb/Kf7xeFt3e3einvXKsqXNRUZ2rkGYxUio2cicbScZOWNLivO/G96Cb1YeTpEIutQuzeLdGU8OCZj53N6Pz+39CLXGk4CZO4wrGGn8nSaGAzSxHOB5PYs1jjqEkMA0W3fW6ey5g0nNtfikdys6g5PqLsdZaTBuIYOaVeSdOxukuHZ+j79fh0Nz4mIsny07yOu31sVklIaJnCZ/AyLHDWxW0WvRtLR8O6wZz3evCUCjikWoXTZpX4PFZGBcv7oUy28h1OzsvE3rLsNbR+unQxpnKCZv4gjBgYFf7B24JeEj7kp4mR9tnTmhS9DfuJLexvXMDHmdAcZlWX6vrHjip81JXl+NT6MZSzk/+GdtuD6/4KXfdgAw0zXn4HLc9bkJb8zdleT0Rq8t5JZ3lzJj/TGqj5lPQjojnkTgyR2ByJM61SpNp1ql/Xa9VlVTlrYIMRkpXyRfks7SrFrrqMNah3PCXAgJlFSXeMv0Ne+Zv+Rp029sd1SlMNfYpqsx3d6Fk7o4jmz4vrY02cpsESdSL2lx6Ow1GoxbCMCrc3ay7dVunLlyve9h2d4zvD1vj/v1hWtJJ6xdTlZKY9Gu0/RukLEvAsK/5I5ABLWCriaLeuULex2zXr984CaxxWMhUpfkYesz/GTrRHl1jp7GDbQx7uJR01+sDHmKQ6H38I35PToZNmMg931zjrXa2Xj0fJKRR8O+28De074POU1rspu/HYm+xrg5O3HkwNyI3EzuCERQ8nWi0geDGvLP2FMBjSWWUEbbHmS07UH3tnAVxZ3GFTxh+pOGhoN0Nm7hvC7AP/bmbNE1mGdvyTXyBTQuX439cyeF82XfMNm0RJy4RJ+PV/L7Y21oXCnlBMRHftzEnlNXGNS8YormxGAmdwRCpCHMYqJ/4/LZ/r5HdFkm2gYRHvcTreI/4QXrQ0Tp4vQxruU985esC3mcI6F386/lf7xq+p6cqpgKzlIU+9K4Azhy7hqzNhzn7JV4r/tXH4wmfNRcTvqhCW7Z3jMAqXbyX4xxNlN9uHhfhq7rcGjmbDuZI7Oss4PcEQiRjrf616dvw3IMn7ohxb5vhzXj/qmBXR/DhsldA8mEjRaGPYwwzqWiOkM1QxRVDaeook6x1NGI/bo8WxzViSVrS1xmVFoltPecusILv22nUrEwVryQcr2Hn9c7O5g3Hr1A8ehrnLuW4N6ntb4+0zwTbHYHP60/xuAWlTAbDVyIcV47o0OFf90UyQu/bSf6Sjz33+y9UmteJolABJV7W1emXhrF67rULsXi3WfoWLOke1uo2UjHWilXNGtUsQjtapSkQtF8PHBzFcb/5RwdM6xNOFNd9f/9zYaJ1Y56rHbUA8CInVdN07jXtIgOxuurvU23deY/R30O6HIc0OXJ7jpI3hw7H+N1u9GgsDk0doeDIV+vS7Lvh7VHubd1OABRl2I5czmeS7FW2t10/e9n87ELRF6ITTIqLLHPYvq6Y7w6Zye7o64wY/2xTMd+9mp8kj9vNJIIRFB57dZ6ae7/Ymgzoq/GUzTMkmJfrTIF2eOqu/Pn420JL5Efk9HAyhc7AbgTQXayY2SsbTgTbQMooq7R2rCLO43L6WtczRDTEgDWOWrxqnUYB3Q5bLnwv7zJoIgHrPaUzS7zdkQx9s+d7hXWEnn28fT/bDXgHB6c/O4hscT2ol2B7efJ63LfvwohcpDRoCidylKOfz15M6/8EcEdTSvQsGKRFPuXPNue+6duoFf9sgG7I0jNZQpwWRfgmL00M11NSI3VAdoaI7jPuNBdB+mKzsdxXYrjuiQRjnC26BqsdNTPtjhv/yzlOsp219f3hTtTflivO3wegDfmpl9H6flftvGLa6W2xNnPeWwl3hwjiUAIH5mNBibc0SDV/dVKFmD58x3RWjO4RSXuaVWJ3pO91/zx9MsjrRkwZY0/Q8WGiQ26FhtstfjR1pUnTbMpoS5xSeensjpDPcNhuhudfRuRugSvWIdzShdjt67s1ziS2+KlnEWc1TksdvHuMyn2JX6Q745KOq+h08Rl/PlEWwqGXh+tlJgEAA5FXyPB5mCSu1PYt6ax2AQ7e09foV65Ql5nPN+oiUUSgRB+ppTi7f7Ob9l3t6zET+uOsX1cN/ckLHA2bYSPmkvvBmVpFuB1FqIpzKu24Sm2V1Kn+cUyngoqmu8s7wGwxVGdb2w9yafiWWWvx0lKBDS2zDoUfY05204ypGXqieuUlwJ76Xng+w2sPnjOa6mMG5kkAiEC6K3b6/PW7d6bXva83gOz0ZDmqJgBTSsk+abrT8d0aVrGf0oDdYhahmP0MaylnXEHn1g+dh5ghl/t7RhlfTBX9i2M+T0izURg9Vi1La2BR5dirXy78jCPdazG6oPnAPhj6wneubMB/+45naVRS3lF7vvbFeIGt2FMFwB3DaS0dKpVKmCJwEmxXVdju70as+wdCbEmUFVFUVmd5j7jQu40rqCVYRcWbKx01KO2OkZ+YvnW3pPLOozFjiZcpkAA48u85IX0PHkup/DW3N3M3Hicj5bsd29LbAFKHBr8Qo+agQgx1whoIlBK9QA+AozA11rrCcn2dwD+BBILqc/WWr8WyJiEyGne1uH9cmhTRvywKcm2rWO7UsTL6KVAisfCbl2Z3boy/zhacI9jES+afqagiqW/8Xp/xzjDNPfzSF2CM7oI2xzVeNM2JNfcPfT7JGXHdKLEeWH7T19xF8pL4gbtC0hNwP7GlFJG4FOgKxAJbFBKzdFaJx9j95/Wuk+g4hAitygYYuJKKt9Su9Ut436+/PkOVC6e3/16yytdafz6IvfrGQ+1YvBXa69fN9SU5oSurPjR3pUf7V0BjUKjMQCaSuoMFdRZbjespICKpZVhN01MBxhoXMZY63AidUlisbBdVwtIXIk+WOTbDOHUZjX3/Og/r9vBuVJbcpEXYggfNZffHm1N08rFvJzl3cmLsVyLt1GjdEGfz8lOgUzdLYADWutDAEqpn4FbgewfbC1ELjB35C1sOX4h1f0Vi+Xj+PmUZRaK5vd+V9CiSjG61y1D+5tK0OWDFQA8370mxfNbGDV7h3+CdlNo98gbxTFdmmO6tHtiWwFi+Mz8EQ0Mh3jfMsV91l/2VrxpHcJ5CgVk2c7JHs05GfXBon3YUikZkWB3eE0Sqw5EAzBzw3GfE4HWmjYT/gV8r3GV3QKZCMoDnvdckUBLL8e1VkptA04Cz2mtdyY/QCk1AhgBUKlSpQCEKkTgVSoeRqXiYanuz29x/ndUXoY6dqhZkmXJSkUDPOAqd9CvYTnmbDtJkTAzd7WoxF0tKnExJoE/t57k1Tkp/kv53VXCuNf6EqHE09SwjyrqlKufYQF9Q9cSrQuxylGPfY4KmJWNP+xtOaJztvR0eknk6LnrM6Hf/Sfzy3Nmx+8/qwJZdM5bV3vy9LsZqKy1bgh8DPzh7UJa6y+11s201s1Klizp7RAh8ryv7m3Gc91uomKxlFVFpw5v4X5eraSz2ejOphXc23rWczYteZbNLhJm4b424SlWbwMwGwMzEiaOEFY56vOjvStv2u6hd8LbfGvrwQFdnk6GLTxvnsXTptksC3mWDSGPUBzfl9TMq6atOZrTIaQrkHcEkUBFj9cVcH7rd9NaX/Z4Pk8p9ZlSqoTWOjqAcQmRK1UsFsYTnWqkeUzdcoUoVSg0RRNDz/pliRjf3euSn/lDTMTbnMXWFj3Tjq6TVqA1fDioEeWK5GPgF/6dzOZpv67Aa7Z7ATBjo646ghUT9xoXMsi0jD9DXmGPoyKLHU25qAvQyHCQ8uosf9tbs8jRNFsW5cmICzHWVPfN3xHFnlNXeKbrTfy3/yyxCfYkfT/pXvtaAlfjbVQslvSu8dSlOFq9vYTPhjTJ8Ep+vgpkItgA1FBKVQFOAHcBd3seoJQqA5zWWmulVAucdyjnAhiTEHnWppe7EGZJ/b9saus+j+5Vm+d+cRakS2yasmvNbT6U165RqgD7z6S+kH1GWDGxVVcH4EXbCP50tGGUaQadDFvpYnSuJe3QzjuVvsa1HHKU4baE17lM/lSvmZs8Ot253OczXW9i6DfrgZR9Ag6HxmBQRF+N5/2Feymcz0Jsgo3xt9ajw8RlXIq1pjgncVb1zA3H814i0FrblFJPAAtwDh/9Vmu9Uyn1iGv/FOBO4FGllA2IBe7S3haPFUJQvEDKYae+uKNJeXciMBmc37A9/5d566SeOaIVFpOBL1cc8lsiSG61ox79Et5E4aClYQ8OrdijK2LFxCOmv3jK9DvbQx9it6Mi6x21iNLFOaGdM50XOZoSRwhhxFFLHcOKiTO6CKcpSnZUWp21MRKjQfFij1pYTAY+WuwxB8Hjlxs+am6S8xbsPEXP+mV5a95uZm8+4d7epnoJd4G81Qeiufvrdawb3TnVulf+FtABv1rrecC8ZNumeDz/BPgkkDEIEeyUUmx5pSsGpZJMpHLv9/jgvLd1ZaatOUpLL2s4J+evuwWNwb2Oc6JJtgGc1UWoo44Qrk4zyLiMUHW9WeaYoyRbdXX6GVM2a53TBekV/zan8X14Z2bMWH+cGetTzkFI66tsYtNS1MWk5S8e9phDcrerymrLt5Y47w6yYWJz7pj5IYQIKM8hqAVDTTzb9Sb368QKCiPaVeWlnrUY17euT9ecMaIVzd5Y7Nc4PTnnLzgpHBQkhkrqDOVVNI+Z5riTwMvW4VRQZ4nWhalvOMytxtV8bZnI9/buROtCrHXUIQ7n3ZQBR8D7HTwX1klu9O876FGvDHtOXU71mJwgiUCIILNjXPckrxO/cN7VvCJKqVTr8hQJM7uXegQokaypqnA+s7t5w980Bi5TgAhdgAhdlQUJzSnOZc6RbJEhO8y1t2Sy+RMmmr9wbtKKI7oMpdRF8hHPUkdjHrE+jZ30S3xkRvM3006Ov246nmans6fwUXPdfx/L953lUqw1IOtDSyIQIsj5UvMI4LluNXn5jwiv+xJHLCVvEw8clTIJuCx0NKdp/BSqqigaGQ7Q27iOEKwcdZQmhlD6GNcyhUkc1OUxYeOoLs2f9rbZ1in91rw9GTres6lp4JQ1LHimnZ8jkkQgRND76t5mzNp4nColUn4QlvKoi+St6Xvx/9qx6sC5JCOWKhULI9RsYN9p7/0HG1/uEtAmJYBr5GOHrsoOe1V+sHdLsu+0Lspw4z90VZvd20abfmKXrsyXtj4scDQjNyzt6c3e01cCcl1JBEIEuYrFwni2m/fqmi/1qs32E5d4/dZ6bDl+McX+6qUKUr3U9fo5iUMfbXYH7y3cyxfLD6U4J3mTkr/d0aQCv21OvWLr67ahvGcbSCFiiKYwLQ276WtYQxvDTr6wTOJveysm2e7AhpGjujS5NSn4U+6arSGEyFVCzUZ+f6wt9cp7b4ZJjclooG+DcqnunzSooXvxnuSGtw2nVpnMF2dLLLuRljhCOENRHBhY46jLaNuDdE14j49st9PHuJYlIc+zPOR/HAy5h0eMcwgj44vc5CWSCIQQPrFksCyFt+RxX2vnQjK3N67A4Bbe64Z1rVM61SGYK1/smO776kzWkLZiYpJtAE8kPMlHtv6Mtw7lgC7PKPPPbA15iBnmN2ht2EnqNarz7hQoaRoSQvjk9sYVOHIuhtplC1GnbOa+sRf2YX0FhXIvaJ9chaJh1ClbiF1R3odfvn5bPTwWJsuUvx2twXWNH+xd6WzYzBDjEtoZd9DauIt1jlpEOKqQjzgKqViidDE6GzZTUZ3lH0dz1jrqYMPIFZ2Ps7oIsVg4qUtwnkJZCwwwepsI4geSCIQQPrGYDLzYo1amz29UsQj3tEy9evCE/vUZNXsH1UsV4IOBDflw8X5KFwplxvpjSY778cGWNPFYn8FThaL5cPixOIENEwscLVjgaEE+axxDjEt4zjSL5sa9nKcgJZQzIe1whDPH0Ya+htX0Na5NcR2HVqx01OND2x3EY6aaOsk5CrHK4b15LDXGAC2bKYlACBFwsx9rQ5NKRVPdf+itXhgMirtczUUlC4bw7bDm2B2aUT1q0fC1he5ji6WyPgOAQaV+NwGw/82e1BgzPxM/AcQSytf23kyzdyMEK1cIw4SN8ira3an8CsMppi5TgssUULEUIJb8xFHfcIh7jItpF5J0nYjTugjHdSlM2NjsuIkJtsFprttgClDVWEkEQoiA2fZqN0JMhnTnKhhSafIwGhSFw1J+MHapXYrFu8/wXLebGNyiEk1dw1ENyln6Ip/ZSKzVnuI8szHr3aIJmN0f1jZMHNXXK4zGEEqMDiWSUkm6DH5ztOND2x30Ma4lHjOXdRjtDdsoqGKpqM5gxk51dYKEdD6STdI0JITIa9KbBTtxQEM+W3og3ev8+2x7QjySSYWiziqq+SymJMX4DEpRMNTM7td7ZGhy22dDmvDY9M3UKlOQPacCM1b/IgWTlM1Y4GiRZL8BB+kNVU0tYWaVjBoSQuSYO5tW4N/nOqR7XNWSBShfJOWCPYkfiw/d4hwy6jlHYVEqM3C9NS31ql+Wv564mZkjWqcfdIDk5NoLkgiEEHneCz1q8cfjbanpMf+gRumCFPSyRsMPDyT9Jv5ST2cHeP0Khb02Q4FzSGtucNHHGkUZJU1DQog8K3EQjdlooFHFIin2L3+hI9fibew4cYlCoc4P+brlrs9vyGc28nD7amm+x/sDGtKnYVk+Wryfz5YdTPW4TrVK8e+eMxn/IXIBSQRCiBtWsfwWiuW3pFj+MdHEAQ3TPH/hM+24qbRvcybSGs0E8PeTN9Pn45U+XSu7SSIQQgSdvW/0YMW+aLrULpXmcZ5JIL0h/OP61aVSsTA+WLTP637PAn65jfQRCCHynKyuaBtiMtK1TmlUBiZo3d2ycpLXveqX4ccHWgJQrnAoBUJMjOxcI8kxLapcXyWtVKFQwiyZWwPh8Y5pN19llSQCIUSeFYjBlBvGdPG6vXyRfEkWlv9sSFPCSzibnFJLKB8Pbpzk9a7XeqT7/utHd06x7anOzhXl0mt+yixJBEII4aGkn5pwWlUt5l583mLy/lF7U+kCfDusGX883paqrvUgCuUz07b69TWj299UEovJwLi+dfjt0TZ+iS056SMQQuQ5A5pV5Ps1R+lUK3DDOmuX9V4k7ouhTd3JInFBnjbVrn9wTx7cmOV7z/JiT+caD2te6kSo6XqTUJNKRdh87CLgHO2U+DN4zqeY/mAr94S4xJuNYW3TL6+dWSqrbW3ZrVmzZnrjxo05HYYQ4gYWceISFYuF+bQ+8OHoa5Qvki/Vb/3JXYxJoO2Ef7mWYOeOJhV4f6D3kUtL95xh+NQNdK9bmi+GNstQ/N4opTZprb1eSBKBEELkgDUHz9G4UpFU6zDZHZr3F+7lgZurJCmjkVlpJQJpGhJCiBzQ2qM5yRujQfFCFsp+Z4R0FgshRJCTRCCEEEFOEoEQQgQ5SQRCCBHkJBEIIUSQk0QghBBBThKBEEIEOUkEQggR5PLczGKl1FngaCZPLwFE+zGcQMtL8ealWCFvxZuXYoW8FW9eihWyFm9lrXVJbzvyXCLICqXUxtSmWOdGeSnevBQr5K1481KskLfizUuxQuDilaYhIYQIcpIIhBAiyAVbIvgypwPIoLwUb16KFfJWvHkpVshb8ealWCFA8QZVH4EQQoiUgu2OQAghRDKSCIQQIsgFTSJQSvVQSu1VSh1QSo3KoRi+VUqdUUpFeGwrppRapJTa7/qzqMe+l1zx7lVKdffY3lQptcO1b7JSiaua+jXWikqppUqp3UqpnUqpp3J5vKFKqfVKqW2ueMfn5nhd72NUSm1RSv2dB2I94nqfrUqpjbk5XqVUEaXUr0qpPa5/v61zcaw1Xb/TxMdlpdTT2R6v1vqGfwBG4CBQFbAA24A6ORBHO6AJEOGx7V1glOv5KOAd1/M6rjhDgCqu+I2ufeuB1oAC5gM9AxBrWaCJ63lBYJ8rptwarwIKuJ6bgXVAq9war+t9/gf8BPydm/8tuN7nCFAi2bZcGS/wPfCg67kFKJJbY00WtxE4BVTO7ngD9kPlpofrl7PA4/VLwEs5FEs4SRPBXqCs63lZYK+3GIEFrp+jLLDHY/tg4ItsiPtPoGteiBcIAzYDLXNrvEAFYAnQieuJIFfG6rr2EVImglwXL1AIOIxrIExujtVL7N2AVTkRb7A0DZUHjnu8jnRtyw1Ka62jAFx/lnJtTy3m8q7nybcHjFIqHGiM81t2ro3X1dSyFTgDLNJa5+Z4PwReABwe23JrrAAaWKiU2qSUGpGL460KnAW+czW7fa2Uyp9LY03uLmCG63m2xhssicBbW1luHzebWszZ+rMopQoAvwFPa60vp3Wol23ZGq/W2q61boTz23YLpVS9NA7PsXiVUn2AM1rrTb6e4mVbdv9baKu1bgL0BB5XSrVL49icjNeEs/n1c611Y+AazqaV1OSG3y1KKQvQD/glvUO9bMtyvMGSCCKBih6vKwAncyiW5E4rpcoCuP4849qeWsyRrufJt/udUsqMMwlM11rPzu3xJtJaXwSWAT1yabxtgX5KqSPAz0AnpdSPuTRWALTWJ11/ngF+B1rk0ngjgUjX3SDArzgTQ26M1VNPYLPW+rTrdbbGGyyJYANQQylVxZV57wLm5HBMieYA97me34ezLT5x+11KqRClVBWgBrDedZt4RSnVyjUq4F6Pc/zGde1vgN1a6w/yQLwllVJFXM/zAV2APbkxXq31S1rrClrrcJz/Fv/VWt+TG2MFUErlV0oVTHyOsy07IjfGq7U+BRxXStV0beoM7MqNsSYzmOvNQolxZV+8gez8yE0PoBfOkS8HgTE5FMMMIAqw4szgDwDFcXYa7nf9Wczj+DGuePfiMQIAaIbzP+JB4BOSdYz5Kdabcd5abge2uh69cnG8DYAtrngjgLGu7bkyXo/36sD1zuJcGSvOdvdtrsfOxP8/uTjeRsBG17+FP4CiuTVW1/uEAeeAwh7bsjVeKTEhhBBBLliahoQQQqRCEoEQQgQ5SQRCCBHkJBEIIUSQk0QghBBBThKBCFpKqauuP8OVUnf7+dqjk71e7c/rC+FPkgiEcBYCzFAiUEoZ0zkkSSLQWrfJYExCZBtJBELABOAWVz34Z1zF695TSm1QSm1XSj0MoJTqoJxrNPwE7HBt+8NViG1nYjE2pdQEIJ/retNd2xLvPpTr2hGu2vGDPK69TF2voz89Q/XkhcgCU04HIEQuMAp4TmvdB8D1gX5Ja91cKRUCrFJKLXQd2wKop7U+7Hp9v9b6vKusxQal1G9a61FKqSe0swBecv1xznxtCJRwnbPCta8xUBdnjZhVOGsSrfT3DytEcnJHIERK3YB7XSWt1+Gc7l/DtW+9RxIAGKmU2gasxVkMrAZpuxmYoZ2VUk8Dy4HmHteO1Fo7cJb0CPfDzyJEuuSOQIiUFPCk1npBko1KdcBZ1tjzdRegtdY6Rim1DAj14dqpifd4bkf+f4psIncEQsAVnMtxJloAPOoqw41S6iZX1c3kCgMXXEmgFs6lMRNZE89PZgUwyNUPURLn8qXr/fJTCJFJ8o1DCGeVSpuriWcq8BHOZpnNrg7bs8BtXs77B3hEKbUdZyXItR77vgS2K6U2a62HeGz/HefSgttwVnd9QWt9ypVIhMgRUn1UCCGCnDQNCSFEkJNEIIQQQU4SgRBCBDlJBEIIEeQkEQghRJCTRCCEEEFOEoEQQgS5/wNj48uzWl+JJwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss_list = log_dict['train_loss_per_batch']\n",
    "\n",
    "plt.plot(loss_list, label='Minibatch loss')\n",
    "plt.plot(np.convolve(loss_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA4AUlEQVR4nO3deXhU5fXA8e/JvkNCFsKasIQQZA8goIKAioobiIhawQ13RVvXX6vYVmtbaq1ataJVQBQVRZAiIiigoAhhkwBhCSFEQhLClkC2mXl/f9wBA4QwhEwmyZzP88wzM3fmzpy5hDPvnPsuYoxBKaWU9/DxdABKKaXqliZ+pZTyMpr4lVLKy2jiV0opL6OJXymlvIyfpwNwRXR0tElISPB0GEop1aCkpaXtM8bEnLy9QST+hIQEVq9e7ekwlFKqQRGRXVVt11KPUkp5GU38SinlZTTxK6WUl2kQNf6qVFRUkJOTQ2lpqadDaTSCgoJo1aoV/v7+ng5FKeVGDTbx5+TkEB4eTkJCAiLi6XAaPGMMhYWF5OTkkJiY6OlwlFJu1GBLPaWlpTRr1kyTfi0REZo1a6a/oJTyAg028QOa9GuZHk+lvINbSz0i8jBwFyDAFGPMyyISBXwEJABZwA3GmAPujEMppeqaze4gq/AohcVllFTYKa1wUFphP34pqXBQbnOQ0iKCCzpEExzgW2exuS3xi8h5WEm/L1AOLBCR/zm3LTbGvCgiTwJPAk+4Kw53KSwsZOjQoQDs3bsXX19fYmKsAXI//fQTAQEBp9139erVTJs2jVdeeaXa9xgwYAArVqyovaCVUrXO4TD8crCErXlFZOQVsXVvERl5xezIL6bc7nDpNQL9fLiwYzTDOscxpHMsseFBbo3ZnS3+zsCPxpijACKyFLgOuAYY7HzOVGAJDTDxN2vWjHXr1gEwadIkwsLC+N3vfnf8cZvNhp9f1Yc3NTWV1NTUM76HJn2l6q+i0gpemL+Fuet+4Ui5/fj2lk2DSYoL46KkaJJiw2neJIggf1+C/H0I9vclyN/3+LWPD6zOOsDXm/JYtDmPRZvzAejeuimXdI5lWEocneLCa70M687EvxF4XkSaASXAFcBqIM4YkwtgjMkVkdiqdhaRCcAEgDZt2rgxzNozfvx4oqKiWLt2Lb169WLMmDFMnDiRkpISgoODeffdd+nUqRNLlixh8uTJzJs3j0mTJpGdnU1mZibZ2dlMnDiRhx56CICwsDCKi4tZsmQJkyZNIjo6mo0bN9K7d2/ef/99RIT58+fz6KOPEh0dTa9evcjMzGTevHkePhJKNW4rMwv57Sfr2XOwhFG9WtGzTSSdmofTMS6MiKCz6w49sEM0AztE8+xVKWTkFbFok/UFMHnhViYv3MrrN/fiiq7xtRq/2xK/MWaziPwV+BooBtYDtrPY/y3gLYDU1NRq14d87ot0Nu05fA7RniqlRQTPXtXlrPfbunUrixYtwtfXl8OHD7Ns2TL8/PxYtGgRTz/9NJ9++ukp+2zZsoVvv/2WoqIiOnXqxL333ntKX/q1a9eSnp5OixYtGDhwIMuXLyc1NZW7776bZcuWkZiYyNixY2v8eZVSZ1ZaYeelr7cy5btM2kSF8Mk9/endNqpWXltESG4eQXLzCB4Y0pH8w6V8syWfge2ja+X1K3PryV1jzDvAOwAi8gKQA+SJSLyztR8P5Lszhro2evRofH2tkzSHDh1i3LhxbNu2DRGhoqKiyn2uvPJKAgMDCQwMJDY2lry8PFq1anXCc/r27Xt8W48ePcjKyiIsLIx27dod73c/duxY3nrrLTd+OqW8V/qeQzz60Xoy8oq4qV8b/u+KzoQGui+FxkYEcWNf91Q73N2rJ9YYky8ibYCRQH8gERgHvOi8nnOu71OTlrm7hIaGHr/9hz/8gYsvvpjZs2eTlZXF4MGDq9wnMDDw+G1fX19stlN/GFX1HGOq/SGklKoFNruD/yzL5OVFW4kMCeDd2/pwcacqK9QNhrtH7n7qrPFXAPcbYw6IyIvAxyJyB5ANjHZzDB5z6NAhWrZsCcB7771X66+fnJxMZmYmWVlZJCQk8NFHH9X6eyjVGJRW2Nl/pJwDR8s5cKSCA0fLOXi0nP1HKjhcav0S9xHwEUFE8PX59fb32wpYk32QK7vF8+drziMy9PQ99hoKd5d6LqxiWyEw1J3vW188/vjjjBs3jpdeeokhQ4bU+usHBwfz+uuvM3z4cKKjo+nbt2+tv4dSDVVhcRnTf9zF+z9ms6+47LTPCw3wRURwGIPdYTAGHMY4LxAZ4s+/buzB1d1bNJpBjtIQygWpqanm5IVYNm/eTOfOnT0UUf1RXFxMWFgYxhjuv/9+OnbsyCOPPFLj19Pjqhq6rH1HePv7TD5ZnUOZzcGQ5Fh6t40kKjSAyBB/IkMCiAwNIDIkgKYh/vj7nn4Cg2P5saEmfBFJM8ac0ne8wU7SpixTpkxh6tSplJeX07NnT+6++25Ph6SUR6zJPsBbSzP5atNe/H18uK5nS+68MJGOceE1fs2GmvDPRBN/A/fII4+cUwtfqYZuxfZ9/HPRVlZlHSAiyI97B7Vn/IAEYiPcO/q1IdPEr5RqsOas+4VHPlpHfJNgnhmRwpg+rd3axbKx0COklGqQPk3L4bFZ6+mbGMV/x/chJEDTmav0SCmlGpyPV+3mic82MKB9M96+tU+dzmzZGDTo+fiVUt7ng5XZPP7pBi7sGMM74zTp14Qm/hoaPHgwX3311QnbXn75Ze67777TPv9Yl9QrrriCgwcPnvKcSZMmMXny5Grf9/PPP2fTpk3H7z/zzDMsWrToLKNXqmGa/kMWT8/+mYs7xfDWb3oT5K9JvyY08dfQ2LFjmTlz5gnbZs6c6dJEafPnz6dp06Y1et+TE/8f//hHhg0bVqPXUqoheXf5Tv4wJ51hneN4U5P+OdHEX0PXX3898+bNo6zMGhGYlZXFnj17+OCDD0hNTaVLly48++yzVe6bkJDAvn37AHj++efp1KkTw4YNIyMj4/hzpkyZQp8+fejevTujRo3i6NGjrFixgrlz5/LYY4/Ro0cPduzYwfjx45k1axYAixcvpmfPnnTt2pXbb7/9eGwJCQk8++yz9OrVi65du7JlyxZ3Hhqlat3b32Xy3BebuKxLHK/f3ItAP03656JxnNz98knY+3PtvmbzrnD5i6d9uFmzZvTt25cFCxZwzTXXMHPmTMaMGcNTTz1FVFQUdrudoUOHsmHDBrp161bla6SlpTFz5kzWrl2LzWajV69e9O7dG4CRI0dy1113AfD73/+ed955hwcffJCrr76aESNGcP3115/wWqWlpYwfP57FixeTlJTErbfeyhtvvMHEiRMBiI6OZs2aNbz++utMnjyZt99+uxYOklLuY4xh4y+HmbFyFzNX7ebKrvG8fGOPakfaKtfoETwHlcs9x8o8H3/8Mb169aJnz56kp6efUJY52Xfffcd1111HSEgIERERXH311ccf27hxIxdeeCFdu3ZlxowZpKenVxtLRkYGiYmJJCUlATBu3DiWLVt2/PGRI0cC0Lt3b7Kysmr6kZVyu/yiUqYsy2T4y99x1Wvf89naX7i1f1v+pUm/1jSOFn81LXN3uvbaa3n00UdZs2YNJSUlREZGMnnyZFatWkVkZCTjx4+ntLS02tc43ZDw8ePH8/nnn9O9e3fee+89lixZUu3rnGnOpWPTOp9u2melPKnMZmfx5nxmpeWwdGsBdoehV5umPH/deYzo1oImwWe3qpWqnn59noOwsDAGDx7M7bffztixYzl8+DChoaE0adKEvLw8vvzyy2r3v+iii5g9ezYlJSUUFRXxxRdfHH+sqKiI+Ph4KioqmDFjxvHt4eHhFBUVnfJaycnJZGVlsX37dgCmT5/OoEGDaumTKlX7bHYH32/bx1OfbaDv84u5b8YaNu05zN0XtWPxbwfx2X0DublfW036btA4WvweNHbsWEaOHMnMmTNJTk6mZ8+edOnShXbt2jFw4MBq9z22Lm+PHj1o27YtF1746yzWf/rTn+jXrx9t27ala9eux5P9jTfeyF133cUrr7xy/KQuQFBQEO+++y6jR4/GZrPRp08f7rnnHvd8aOVVym0OHvpwLX6+wgNDOpDcPKLGr2WzO/hp537m/ZzLgo172X+knNAAX4alxDGqVysGdojG16dxToxWn+i0zOoEelzVyZ6e/TMfrMwmJMCXo+V2hndpzoNDO9ClRROX9q+wO1iVtZ/5zmS/r7ickABfhnaO48qu8QzuFKNdM91Ep2VWSp21GSt38cHKbO4Z1J57BrXjv8uzeHf5Thak7+WSlDgeGtKRrq1O/QLYvf8oy7YVsGxrASu2F1JUZiPY35chnWMZ0TWewZ1idcStB2niV0pVaXXWfibNTWdQUgyPXdYJXx/h0UuSuOOCRN5bnsU732dy1aY8hibHcu/g9hSV2Vi2tYClWwvILDgCQMumwYzo3oJBSdFclBSjE6nVEw36X8EY02gXSvCEhlD2U3Uj91AJ97y/hpZNg3nlxp4n1N2bBPvz8LCO3H5BAlNXZPH29zu5/s0fAAj08+H8ds24pV9bLkqKoX1MqP4frYcabOIPCgqisLCQZs2a6R9WLTDGUFhYSFCQLl7h7Uor7NwzPY2Schsf3tWPJiFV96oJD/LngSEdGT8wkfkbcmneJIi+iVFar28AGmzib9WqFTk5ORQUFHg6lEYjKCiIVq1aeToM5UHGGP5v9kbW5xzird/0dmnZwrBAP27o07oOolO1pcEmfn9/fxITEz0dhlKNynsrsvh0TQ4Th3Xk0i7NPR2OchMdwKWUAqy1a//8v81c6uytoxovTfxKKXbvP8r9H6yhXXQoL43pgY8OomrUGmypRyll1eTX7j7I/A25bNlbxANDOnB+u2Zn9Rqbcw9z17TV2B2GKbemEqaLlTd6bv0XFpFHgDsBA/wM3AaEAB8BCUAWcIMx5oA741CqMXE4DGt3H+B/G/ayYGMuew6V4u8rNAkO4Oa3V/Lk8GTuvDDRpd5uC9P3MvGjdYQH+fH+nf1IiA6tg0+gPM1tiV9EWgIPASnGmBIR+Ri4EUgBFhtjXhSRJ4EngSfcFYdSjYHdYViTfYD5P+fy5c972Xu4lABfHy5KiuZ3l3ViaOc4fAQe+2QDz8/fzNrdB/jb9d1P23o3xvD6kh38/asMurdqwlu3phIXoV15vYW7f9P5AcEiUoHV0t8DPAUMdj4+FViCJn6lTlFaYWfFjn0sTM9j0eY89hWXE+Dnw6CkGJ7smszQzrGEB53Yx/6NW3ox5btM/roggy17v+c/t5zaJbO0ws7jszYwd/0erunRgr+O6qZ9772MWydpE5GHgeeBEmChMeZmETlojGla6TkHjDGRVew7AZgA0KZNm967du1yW5xK1ReHSipYkpHPwvQ8lmTkc6TcTligHxcnx3JJShwXd4o5JdlX5cfMQh74YA1Hy+38dVQ3rureAoC9h0qZMH01P/9yiMcu68S9g9rrAMhG7HSTtLkt8YtIJPApMAY4CHwCzAJecyXxV1bV7JxKNSY/5xziX4u3siSjAJvDEBMeyCUpcVyaEkf/9s1qtMZs3uFS7puxhrRdB7htYAJXdo3nvhlrOFJm4+Ube3JJSpwbPomqTzwxO+cwYKcxpsAZwGfAACBPROKNMbkiEg/kuzEGpeq1jL1FvPR1Bl+l59E0xJ87LkjksvOa06NV03PuUhkXEcTMCefzwvzNvLs8i3eXZ9EqMpjpdwykU/Mzj8hVjZc7E382cL6IhGCVeoYCq4EjwDjgRef1HDfGoFS9tHPfEV5etJW56/cQGuDHxGEduf2CRCJcKOOcDX9fH569qgu920ayNKOAp67oTFRoQK2+h2p43Jb4jTErRWQWsAawAWuBt4Aw4GMRuQPry2G0u2JQqr755WAJryzaxqw1Ofj7Cndf1J67L2pHpJuT8YhuLRjRrYVb30M1HG7t1WOMeRZ49qTNZVitf6W8hsNheHnxNt5csgOA35zflvsubk9suHahVHVPh+gp5WZHy2088tE6vkrP49oeLXh8eDItmgZ7OizlxTTxK+VGuYdKuOO91WzZe5hnRqRw28AE7T6pPE4Tv1Jusn73Qe6ctpqScjvvjO/DxZ1iPR2SUoAmfqXcYt6GPfz24/XEhAcy485+JLmwoIlSdUUTv1K1yBjDq99s56Wvt5LaNpL//KY3zcICPR2WUifQxK+UC46U2Xjt2+18vGo3oYF+xIYHEhMeSGx4ILERQcSEBRITEcjna39hzro9jOzZkr+M6lqjEbdKuZsmfqWqYYxh7vo9vDB/M3mHy7g0JY4gf18KisrYmlfE8u37OFxqO2Gfxy7rxH2DdQ4cVX9p4lfqNNL3HGLS3HRWZR2ga8smvH5zb3q3PXVaqdIKOwVFZeQXlREW6KfTIah6TxO/Uic5cKScyQsz+PCnbJqGBPDiyK6MTm2N72nmzgny96V1VAito0LqOFKlakYTv1JYI2uzCo/wbUYBryzeRnGZjVv7J/DIsCSahNTu/DlKeZomfuWVCovLWJ9zkHXZB1m7+yDrdx88Xqvv364Zk67uoiUb1Whp4ldew+4wvPjlZhak72X3/hIAfASS4sK5sls8PVo3pUfrSJLiwvTErGrUNPErr2CM4dm5G3n/x2yGJsdyS7+2dG/dlK4tmxB6mnVplWqs9C9eeYVXv9nO+z9mc/egdjx1eWdPh6OUR/l4OgCl3O2Dldm89PVWRvVqxZPDkz0djlIepy1+1agt2LiX33/+Mxd3iuHFUV21du8tyopg5zLYvhiyf4RWqTDwYWjWvm7jOFIIxXshONK6+NeP6bg18atGa2VmIQ/NXEv31k3598298PfVH7iNlsMBezfAjsWw/RvY/SM4bOAfCi17wfqZsHY6pFwDFzwC8d3dH9P6mfC/30J58a/b/IJ+/RIIagohUdDlOjhvFNRho0QTv2qUNuce5s5pq2kdGcx/x/UhJED/1Bul/C2w8g3Y8j84UmBta94V+j8AHYZC6/PBLwCK9sKPb8CqdyB9NrQfChc+Cm0HVp9wjQHjAJ+zmHOp9LCV8H/+GNoMgL53WttKDpx4KT1kfVltmQcr/wPD/2L9MqkDYoypkzc6F6mpqWb16tWeDkM1ELv3H2XUGyvwEeHT+wbQUle7alyMgazvYcWrsO0rqxWdfCV0uATaD4HwuNPvW3IQVr9jfQkcKYBWfaDfPVZiP5wLRc5L5dsA3cfC+fdBdIfqY8tZDZ/eAQezYdCTcNHvqv/ScNhh3QfwzZ+gOA+6joahz0LT1md9WKoiImnGmFO+TTTxq0alsLiM0W/+wL7iMmbdO8B75sE/kAVrpltJLCymdl87fzN89w8IjbUSWUhU7b6+q+w22PS5lfBz10FINPSdAH3ugNDos3utihJY+z6seMVK0sf4BkJ4c4hoAeHx1qVkP2z8FOwVkDQc+t8PCRec+EvB4YDlL8O3z1v7jHob2pzvejxlRfD9y/DDa9b9AQ/CwIkQGHZ2n+skmvhVo3ekzMZNU35ky94iZtzZj9QEDyWourZtkdXKLD0IzTrALZ9BZNtzf93ifCuRrZlm1corjkBQExj8NKTeDr7nWD6zV1hJd/9Oq7Xr6w++AdbFL8BKwn6B1vZdP1it9EPZ1mfs/wB0v/HcT5babZCzCgLDrYQdElV16ac4H1a9bV2OFkLzbtYXQJeR1v3ZE6yTyV2ugxEvQ3DTmsVzcDcsfg5+/gTC4mDIH6DHTWdXaqpEE79q1CrsDu6atpplWwt485beXNqluadDcj+HA76bDN++AHFdrJOW/3sU/EOs5B+XUrPXLT8KP/7baoHaSqHPnTDoCavsseBJK8HFJMNlL1h19DM5Ugg5P0HhDtifaV0O7LSSnLG7HlfbgVZLuONl4OOhE/UVJbDhI/jhddiXAWHNwV5uHafL/wY9b6mdk7S7V8FXT1lfStf/1zr5WwOa+FWjZYzhsVkbmJWWwwvXdeWmfm08HdLZKy6wygkhzaDzVeAfVP3zSw7C7Lth6wLoNsZqZQaEQF46TB8JthK46RNo08/1GBwOK6l98yc4/Askj4Bhz51Y1zbGOpG68P+s8lLScLj0+ROfc3Q/7Fpu1eF3fgf56b8+FtQEotpDVCJEtfv1EhZn1bvt5WAvA5vz2l5u3W7Ssm564rjK4YAd38CPr4OjAq58CaI71u57GAMZX0LSZdriVw3brsIjvLUsk87xEVzTowXhQec+8+XkrzJ47dvtPDS0I49eklQLUdYRhwOylkHae7B5npVAwOrm120M9LoVmp936n57N8JHt8Ch3XDZX6DvXSe2Mg/sgunXweE9cMM0SLq0+jjsNtj+NSz5C+SuhxY9rWSeMPD0+9jKrNLLssnOXwV3gPhan2fvRsCAX7D1xZNwodVaj+nkufMDXkoTv/IoYwwf/JTN8//bTLnNgc1hCAnw5eruLbipXxu6tmxSo8FV03/cxR8+38iNfVrzl5ENZIBWcQGsmwFrplplj+BI6H4T9B5ndTtcMw02z7Vauy17W18A542y6tAbPoG5D1ot5xumnv4EYnEBzBhlJeFrX7fq4Sfbtx3WvQ/rPrQGGUW0gmHPwnnXu15KKcqzfiGsfd+qx7fuayX6hAut2P0Can6c1Dmr88QvIp2Ajyptagc8A0xzbk8AsoAbjDEHqnstTfwNW97hUh6ftYGlWwu4oEM0f7u+G3mHS/nwp2y+WJ9LSYWdLi0iuKlfG67p0ZIwFydNW7Axl3tnrGFocixv3tIbP08O0KootWrfuetBAPGxWsDic+Jl94+/tu7bDIDU26Dz1aeWdo7ut8ouaVOhYLN1crVVKuxcau03+r3quy2C1Xf8o5utuC57wToZWVYMm+ZYiTp7hRVTx0ut2nTHy2qeqIvzITDizCUqVac82uIXEV/gF6AfcD+w3xjzoog8CUQaY56obn9N/A3X3PV7+MPnGymz2Xn6is7c0q8tPpVWsjpcWsGctb8wY2U2W/YWERLgy1XdWnBJShz92zc77cyZP+3czy3vrKRLiwg+uPN8ggM8sKh56SHY9jVs/gK2LzpxhObpBDW1emn0Hm+VPs7EGKtv+JqpsG2h1Rq/5Dmrp4srbGXw2V1Wsk8cBL+kWXE262Al+243QkS8a6+lGhxPJ/5LgWeNMQNFJAMYbIzJFZF4YIkxptr/AZr4G54DR8r5w5yNzNuQS882TfnH6O60izl9n2RjDOt2H+TDn7KZtyGXo+V2Anx96JMYyaCkGAYlxR6fJ39rXhHXv7GC6LBAZt07gKjQOiwnHM6FjPnWaMud31kt99BYSL7COhnadiD4+FmjPY3deX3sYiAgrO7LHw47fPm4NYVAyrXQ6zfQul+dThGgPMPTif+/wBpjzGsictAY07TSYweMMaesYC0iE4AJAG3atOm9a9cut8epasfSrQU89sl6DhwtZ+KwJO6+qN1ZlWHKbHZWZx1g6dYClmYUkJFXBEB8kyAGJcWwdGsBNofhs3sHnP06tzuXWd0IQ2OsQT9hsdZAoKpKLQUZULDFeb3Zuj42kjOqPXQeYSX7lqme6154NozRZO9lPJb4RSQA2AN0McbkuZr4K9MWf8Pxxfo9PDxzLR1jw3lpTHe6tGhyzq+Ze6iEZVsLWJJRwPfb9gEw8+7zz/61V/7HavlWJTDC+jIIbgqHcqwBRcf4h1plmZhkiO1s1cRjOmkSVfXe6RJ/XcxcdTlWa//Y/6Q8EYmvVOrJr4MYVB2Y/3MuEz9aR2pCFO/dVnsTo8U3CWZMnzaM6dOGCruD0gr72XcDXfEqLPw9dLoSLv2T1aI/km/N13KkAI7ss05QluyHDsOsJB+TDLHJVm+XhtCiV8pFdZH4xwIfVro/FxgHvOi8nlMHMSg3W7BxLw99uJaerZvy7nj3zYbp7+tz9tMrf/cPWPxHq7496m3rxGhdz8uuVD3i1maMiIQAlwCfVdr8InCJiGxzPvaiO2NQ7vf1pjwe+GAN3Vo14b3b+9afNWyNgSUvWkm/62gY9Y7rvWGUasTc+j/UGHMUaHbStkLAhQk+VEOweHMe981Io0tLK+m72gff7YyxBhZ99w9rcNQ1r9V42LtSjU09+V+qGqJvM/K59/01dI6PYNrtfYmohekXaoUx8PUfrLp+r3HWPDZao1fqOE38qkaWbS3g7ulpdIwLY/rt/WgSXI+S/oInYeWb0Ocua8ZETfpKnUATvzpr32bkc8/0NNrHhPH+Hf1oEuJi0s9Lt7pUbppjDSA6/15oN7j2ukWWFcP838H6D63Vki57QbtcKlWFMyZ+ERkBzDfGOOogHlWP7T1UygvzNzN3/R6Sm4cz485+RJ5p1KzdBhn/g5Vvwa7vrRkbky6zpu2dfi3EdIbz74GuN1jTCtdUzmpraoL9O2HwU9b88Zr0laqSKy3+G4F/icinwLvGmM1ujknVM+U2B/9dvpNXFm/D5jA8NLQj9w5qX/38OEcKrfllVr0Dh3OgSRu45E/W/DAhUdYcMhs/teYz/+JhWDQJet9mLfrRpKXrwdlt8P1LVu+diBYwfp61LJ5S6rRcGrkrIhFY/fFvAwzwLvChMabIveFZdOSu5yzbWsCkL9LJLDjCsM5xPDMihTbNqmmZ223WIh1p71nztCdeZK0DmzS86l41xsCuFbDyDWuBD/GxFiJJudZaODso4vTvtX+ntRjJ7pVWd80rJtd8yTulGqFzGrlrjDnsbPEHAxOB64DHROQVY8yrtRqpqhdyDhzlz/M2syB9LwnNQnh3fB8uTo6tfieH3UrEG2dBz99Y0wDHdq5+HxFrwY+EgdaKTj9NseaqT59tTXbWdoD1pZE0/NdBV8ZYdfz5j1tfFCPfhm6ja+VzK+UNztjiF5GrgNuB9sB0YKoxJt85OGuzMaYWVnWunrb4686uwiNM+2EX7/+4Cx8RHhjSgTsvTCTQ7wx94B12+Pw+2DAThk2y1n+tqWMLYG9dAFu/siZIA2sq4Y6XWaWjTXOsmTCvexOaNsClFpWqA+fS4h8N/NMYs6zyRmPMURG5vbYCVJ7jcBi+376PqSuy+CYjH18RRnSL57HhybRsGuzKC8Dch6ykf/Hvzy3pA/j6Qdv+1uWS56xfAlsXWl8Eq6ZYUxwPfRYGPqyDspSqAVda/IlArjGm1Hk/GIgzxmS5PzyLtvjdo7jMxqdpOUz9IYvMgiNEhwVwU9823NSvLc2buLiSksMB8yZaJ3IHPQEXP+3WmCkrts4dhEa7932UagTOpcX/CTCg0n27c1ufWopN1bHDpRW8tHArs9JyKC6z0b11U/45pjtXdI0/c0mnMmOsfvNrpsIFj1rdKN0tMMy6KKVqzJXE72eMKT92xxhT7pxjXzVAxhh+9/F6Fm/J5+ruLRg3IIEerZvW5IWsEbKr34EBD8LQZ7TfvFINhCuJv0BErjbGzAUQkWuAfe4NS7nL9B93sXBTHr+/sjN3Xtiu6iflb4Yd31grU4XHQZjzEhxpJXdjrLntV74J/e61+udr0leqwXAl8d8DzBCR1wABdgO3ujUq5Rbpew7x53mbGZIcyx0XJJ76BHsFfP8yLP2rtZbsyXwDrC+AwHDI32TNhTP8L5r0lWpgzpj4jTE7gPNFJAzrZHCdDNpStetImY0HP1hLZKg/f7++G3Jyss7bBJ/fC7nr4LxRVpdMW5m1BGHRXmt1qmLnddFea5DVoCc16SvVALk0gEtErgS6AEHHEoYx5o9ujEvVsmfmpLOz8Agf3Hk+zcICf33AboMVr8CSv1jrzt4wDVKu+fXx6I51H6xSyq1cmaTtTSAEuBh4G7ge+MnNcala9NmaHD5dk8NDQzvSv32ldXEKMqxW/i9pVrK/8iXtJqmUF3ClxT/AGNNNRDYYY54TkX9w4lKKqh7LLCjm959vpG9iFA8N6WBtdNjhh9fgm+chIBSufxfOG+nZQJVSdcaVxF/qvD4qIi2AQqCKM4Oqvimz2Xngg7UE+Pnwrxt74OfrY53A/WwCpH8GySNgxD8h7Axz8CilGhVXEv8XItIU+DuwBmt2zinuDErVjr/M38Km3MO8fWsq8U2CrZO1n9xmzY8/bBIMnKgnZ5XyQtUmfhHxARYbYw4Cn4rIPCDIGHOoLoJTNff1pjzeW5HFbQMTGJYSB+VH4aObrf75l/8d+k3wdIhKKQ+pdjFS56pb/6h0v0yTfv23fPs+fvfJes5rGcGTlydDWRHMuB52fAtXv6ZJXykv58oq1AtFZJSc0vFb1Tcl5XYmzU3n5rdX0iwsgH/f1IvAisMw7VrI/hFGvQ29fuPpMJVSHuZKjf9RIBSwiUgp1uhdY4ypZmkkVdfWZh/gtx+vJ3PfEW4bmMDjlyUTXHEApl5rddu8YRp0HuHpMJVS9YArI3fD6yIQVTPlNgevfrONf3+7neYRQXxwZz8GdIi2RtdOu8aay37sh9BhmKdDVUrVE64M4Lqoqu0nL8xymn2bYg36Og+rN9DtQAbwEZAAZAE3GGMOuBqw+lXG3iIe/Xgd6XsOM6pXK569OoWIIH/IXmktgVicDzfPgsQLPR2qUqoecaXU81il20FAXyANGOLCvv8CFhhjrndO5RwCPI3VU+hFEXkSeBJ44uzC9m4HjpQzY+UuXlm8nfAgP/7zm95c1qU5HCmEOc/C2ukQ0RJunQOtddkEpdSJXCn1XFX5voi0Bv52pv1EJAK4CBjvfJ1yoNw5rfNg59OmAkvQxH9Gxhh+yCxk5k+7WbBxL+V2B5emxPHCyK5Eh/hD2lRY9KzVg2fAQ9ZqWLpgiVKqCi5N0naSHKzSzZm0AwqAd0WkO9avhIexlm3MBTDG5IpIlcNGRWQCMAGgTRvvXUy7oKiMWWk5fLQqm6zCo4QH+XFj39bc2KcNKS0iYO/PMPNRyPkJ2vS35tuJS/F02EqpesyVGv+rWPV5sLp/9gDWu/javYAHjTErReRfWGUdlxhj3gLeAmvNXVf3awyKy2ws376P2Wt+YdHmPGwOQ9+EKB4a2pErusYT5O8LpYdhwVPWYijBkXDtG9B9rI7EVUqdkSst/sqrnNuAD40xy13YLwfIMcasdN6fhZX480Qk3tnajwfyzyriRsgYw6bcwyzdWsDSjALSdh3A5jBEhQZwX79Ibkgso5VjJxR+DbO3Q+EOKNxuTcGQehsM+QOERHn6YyilGghXEv8soNQYYwcQEV8RCTHGHK1uJ2PMXhHZLSKdjDEZwFBgk/MyDnjReT3nnD5BA1Wxcgr2b16k3O6gxAZNjXA1wnW+fgRG+BEc4E9QxUFk7QFY69xJfCEyAZp1gMRB1oIprXp78mMopRogVxL/YmAYUOy8HwwsBAa4sO+DWMs2BgCZwG1Y5aKPReQOIBsYfbZBN3hlRVQs/CO7KpqwySeJ+CaBxEcE0jzcn2A/AeOwpk4OamIthBLV3kr2kW3B19/T0SulGjhXEn+QMeZY0scYUywiIa68uDFmHZBaxUNDXQuvcSr/cQoh9sN8mfhXHh43Fl8frcsrpeqOK3P1HBGRXsfuiEhvoMR9ITVy5UdwLH+VpfZuXHjxcE36Sqk650qLfyLwiYjscd6PB8a4LaLGLu09gsr3MyvscV5pG+npaJRSXsiVAVyrRCQZ6IQ1QdsWY0yF2yNrjCpKsH33MivtXUjpdyk64alSyhPOWOoRkfuBUGPMRmPMz0CYiNzn/tAaoTXT8Duaz78dIxnVq6Wno1FKeSlXavx3OVfgAsA5odpdbouosbKVYb5/mbXSmeAOFxEbEeTpiJRSXsqVxO9TeREWEfEFAtwXUiO19n2kaA+Ty65ldB/vnYJCKeV5rpzc/Qqr3/2bWFM33AN86daoGhtbOXz/TzIDU9ji04shyVVOT6SUUnXClcT/BNZkafdindxdi9WzR7lqw0w4tJvnbU9w3fmtCPBz5YeWUkq5xxkzkHPB9R+xRt6mYg2+2uzmuBoPuw2++wcF4SkstnVjdGprT0eklPJyp23xi0gScCMwFijEWjULY8zFdRNaI/HzJ3Agi1dD/o/urSPp1FxXslRKeVZ1Lf4tWK37q4wxFxhjXgXsdRNWI+Gww3eTKYlKYdr+FG5IbeXpiJRSqtrEPwrYC3wrIlNEZChWjV+5Kn02FG5ndvhNBPn7clX3Fp6OSCmlTp/4jTGzjTFjgGSs5REfAeJE5A0RubSO4mu4HA5Y9ncc0cm8mNWBK86LtxZCV0opD3Pl5O4RY8wMY8wIoBWwjrNYScvr2G2Q8SV8MBoKtpDW9k4Olzn0pK5Sqt44qzV3jTH7gf84L6qyQzmwZhqsmQ5FeyAsDi7+PS9npNAmqpx+ibpCllKqfqjJYuvqGLsNtn8Nae/BtoVgDHQYClf8DZKGs/tQBcu//JbfXpKEj06/rJSqJzTx19ThPfDu5XAgy2rdX/AI9LrVWhrR6ZO0nYjAqN7am0cpVX9o4q+pJX+BQ7/A6PcgecQpSyLaHYZZq3dzYccYWjQN9kyMSilVBU38NVGQAWvfh753Y1KupaC4jN37i9i9v4Ts/UfJ3n+UHQXF7DlUytNXdvZ0tEopdQJN/DWx+I84/EO4M/MiVjyzgNIKxwkPx0UE0iYqhAkXtePSlOYeClIppaqmif9s7V4FW+axOO4Olu2GW/u3pW2zENpEhdA6KoRWkcEE+ft6OkqllDotTfxnwxhY9Cy24GgeyR7ILf3b8sxVKZ6OSimlzorOD3w2tn0Nu5YzM3gsEhjGQ0M7ejoipZQ6a5r4XeVwwOLnKA1rw3N7+nD/xR2ICtWFyJRSDY8mflf9/AnkbeQVxhDbNJzxAxI8HZFSStWIW2v8IpIFFGFN52wzxqSKSBTW3P4JQBZwg3MB9/rLVgbf/pmDTTrzRl53XhqTpCdwlVINVl20+C82xvQwxqQ67z8JLDbGdAQW0xAmfFv9XziYzXNHb6BLy6Zc072lpyNSSqka80Sp5xpgqvP2VOBaD8TgutLDsOzv/NK0D7OLknj68s46745SqkFzd+I3wEIRSRORCc5tccaYXADndWxVO4rIBBFZLSKrCwoK3BxmNX54DY4W8tuD1zEkOY4BHaI9F4tSStUCd/fjH2iM2SMiscDXIrLF1R2NMW8BbwGkpqYadwVYreJ8WPEa6U0v5qe8BL66PNkjYSilVG1ya4vfGLPHeZ0PzAb6AnkiEg/gvM53ZwznZOnfMLZSJhZcxZg+rekYpwulK6UaPrclfhEJFZHwY7eBS4GNwFxgnPNp44A57orhnJQchDVTWR4+nF98W/LIsCRPR6SUUrXCnaWeOGC2iBx7nw+MMQtEZBXwsYjcAWQDo90YQ81lzAd7OX8v6MeEoe2IjQjydERKKVUr3Jb4jTGZQPcqthcCQ931vrUmfTb5PrHsCU3hrgvbeToapZSqNTpytyolBzE7vuXz8j7cM7gDoYE6l51SqvHQxF+VjPmIo4L5jn5c1T3e09EopVSt0sRflfTZ7JVYgtr2ITZca/tKqcZFE//JSg5gdnzLnIo+XKlTMyilGiFN/CfbYpV5vnT0Y3gXXTZRKdX46FnLk5hNn5MnsYQk9CUmPNDT4SilVK3TFn9lJQfgeJmnhaejUUopt9DEX5mWeZRSXkBLPZUcK/OEJfajWZiWeZRSjZO2+I/RMo9Sykto4j/mWJnH9OcyLfMopRoxLfU4mU2fs1diCU/sQ1RogKfDUUopt9EWPxwftDW3og8jtMyjlGrkNPEDbJmPj6OCBaY/l6ZomUcp1bhpqQcw6bPJlVgi2vUlUss8SqlGTlv8zt48cyv6am8epZRX0MS/5X+IsbGQ87lMyzxKKS/g9aUek/45uRJLk3Z9aRLi7+lwlFLK7by7xX9CmUenYFZKeQfvTvyVyjyXpMR5OhqllKoTXl3qMZvmkEsske370SRYyzxKKe/gvS1+WzmOnd+x0NZDe/MopbyK9yb+nFX42kr4ia4M0zKPUsqLeG/i37kUOz5I4oVEBGmZRynlPby2xl+2bQmbHQn06pTg6VCUUqpOub3FLyK+IrJWROY570eJyNciss15HenuGE5RVox/bhorHOfRv12zOn97pZTypLoo9TwMbK50/0lgsTGmI7DYeb9uZf+Aj7Gxwb87yc3D6/ztlVLKk9ya+EWkFXAl8HalzdcAU523pwLXujOGqpjMJZTjR0Bif3x8pK7fXimlPMrdLf6XgccBR6VtccaYXADndWxVO4rIBBFZLSKrCwoKajWoiu1LSLMnkdpRR+sqpbyP2xK/iIwA8o0xaTXZ3xjzljEm1RiTGhMTU3uBHd2Pf0E6yx1dtL6vlPJK7mzxDwSuFpEsYCYwRETeB/JEJB7AeZ3vxhhOtXMZgiE9qCcdYsPq9K2VUqo+cFviN8Y8ZYxpZYxJAG4EvjHG3ALMBcY5nzYOmOOuGKqMK3MpRwgion1fRLS+r5TyPp4YwPUicImIbAMucd6vMxXbv+VHe2f6tdfRukop71QnA7iMMUuAJc7bhcDQunjfUxzKIeDQTlY4buGW9lrfV0p5J++asmHnMgAygnuR0CzEw8EopZRneNWUDSZzCQeIIK5DT63vK6W8lve0+I3BvmMJK+wp9OtQi91DlVKqgfGexF+4Hb8jedp/Xynl9bwn8WcuAWBHWG9aR2l9Xynlvbymxm8yl5JLDG3bd/F0KEop5VHe0eJ32LHvXMZ3ti4M6Bjt6WiUUsqjvCPx792AX9khVjhS6N9OE79Syrt5R+J39t/PadqH5k2CPByMUkp5llfU+B2ZS8k0rUjq0NHToSillMc1/ha/rRyzawXf2bvQX6dpUEopL0j8OavwtZXwgyNF++8rpRTekPh3LsOBD/nN+hATHujpaJRSyuMafeJ3ZC5ho0mkW4e2ng5FKaXqhcad+MuKkZzVfG/XaRqUUuqYxp34s39AjI3lji6cr4lfKaWAxp74dy6lAn+OxPYmMjTA09EopVS90Kj78Ze3vYh/fVdIr/YtPR2KUkrVG4068a/x782/KyqYov33lVLquEZd6vlhRyE+An0TozwdilJK1RuNOvG3aBrE9b1b0STY39OhKKVUvdGoSz1j+rRhTJ82ng5DKaXqlUbd4ldKKXUqTfxKKeVlNPErpZSXcVviF5EgEflJRNaLSLqIPOfcHiUiX4vINud1pLtiUEopdSp3tvjLgCHGmO5AD2C4iJwPPAksNsZ0BBY77yullKojbkv8xlLsvOvvvBjgGmCqc/tU4Fp3xaCUUupUbq3xi4iviKwD8oGvjTErgThjTC6A8zr2NPtOEJHVIrK6oKDAnWEqpZRXcWviN8bYjTE9gFZAXxE57yz2fcsYk2qMSY2JiXFbjEop5W3qZACXMeagiCwBhgN5IhJvjMkVkXisXwPVSktL2yciu07zcDSwr/airVUaW81obDWjsdVcfY7vXGKrcgUqMcbUPJxqiEgMUOFM+sHAQuCvwCCg0Bjzoog8CUQZYx4/h/dZbYxJrZ2oa5fGVjMaW81obDVXn+NzR2zubPHHA1NFxBerpPSxMWaeiPwAfCwidwDZwGg3xqCUUuokbkv8xpgNQM8qthcCQ931vkopparXGEbuvuXpAKqhsdWMxlYzGlvN1ef4aj02t9X4lVJK1U+NocWvlFLqLGjiV0opL9OgE7+IDBeRDBHZ7uwaWm+ISJaI/Cwi60RktYdj+a+I5IvIxkrb6sVkeaeJbZKI/OI8dutE5AoPxdZaRL4Vkc3OiQYfdm73+LGrJjaPH7v6PEFjNbF5/LhVitFXRNaKyDzn/Vo/bg22xu/sJroVuATIAVYBY40xmzwamJOIZAGpxhiPDwoRkYuAYmCaMeY857a/AfsrjaeINMY8UU9imwQUG2Mm13U8J8UWD8QbY9aISDiQhjW31Hg8fOyqie0GPHzsRESAUGNMsYj4A98DDwMj8fxxO11sw6kHf3MAIvIokApEGGNGuOP/akNu8fcFthtjMo0x5cBMrAng1EmMMcuA/SdtrheT5Z0mtnrBGJNrjFnjvF0EbAZaUg+OXTWxeVx9nqCxmtjqBRFpBVwJvF1pc60ft4ac+FsCuyvdz6Ge/OE7GWChiKSJyARPB1MFlybL86AHRGSDsxTk8TUbRCQBa1yKyxMN1pWTYoN6cOzkHCZo9FBsUA+OG/Ay8DjgqLSt1o9bQ078UsW2evPNDQw0xvQCLgfud5Y0lGveANpjreOQC/zDk8GISBjwKTDRGHPYk7GcrIrY6sWxO5cJGt3tNLF5/LiJyAgg3xiT5u73asiJPwdoXel+K2CPh2I5hTFmj/M6H5iNVZqqT/KcdeJj9eIzTpZXV4wxec7/nA5gCh48ds468KfADGPMZ87N9eLYVRVbfTp2zngOAkuoNEEj1I+/ucqx1ZPjNhC42nl+cCYwRETexw3HrSEn/lVARxFJFJEA4EZgrodjAkBEQp0n3BCRUOBSYGP1e9W5ucA45+1xwBwPxnKCY3/kTtfhoWPnPBH4DrDZGPNSpYc8fuxOF1t9OHYiEiMiTZ23g4FhwBbqx3GrMrb6cNyMMU8ZY1oZYxKw8tk3xphbcMdxM8Y02AtwBVbPnh3A/3k6nkpxtQPWOy/pno4N+BDr52sF1i+lO4BmWEtfbnNeR9Wj2KYDPwMbnH/08R6K7QKs8uEGYJ3zckV9OHbVxObxYwd0A9Y6Y9gIPOPcXh+O2+li8/hxOynOwcA8dx23BtudUymlVM005FKPUkqpGtDEr5RSXkYTv1JKeRlN/Eop5WU08SullJfRxK8UICL2SjMzrpNanO1VRBKk0uyjSnmaOxdbV6ohKTHWMH6lGj1t8StVDbHWVfircw73n0Skg3N7WxFZ7JzUa7GItHFujxOR2c753teLyADnS/mKyBTnHPALnaNGlfIITfxKWYJPKvWMqfTYYWNMX+A1rNkTcd6eZozpBswAXnFufwVYaozpDvTCGrkN0BH4tzGmC3AQGOXWT6NUNXTkrlKAiBQbY8Kq2J4FDDHGZDonRdtrjGkmIvuwhvVXOLfnGmOiRaQAaGWMKav0GglY0/92dN5/AvA3xvy5Dj6aUqfQFr9SZ2ZOc/t0z6lKWaXbdvT8mvIgTfxKndmYStc/OG+vwJpBEeBmrCX8wJpE6144vuBHRF0FqZSrtNWhlCXYuSrTMQuMMce6dAaKyEqshtJY57aHgP+KyGNAAXCbc/vDwFsicgdWy/5erNlHlao3tMavVDWcNf5UY8w+T8eiVG3RUo9SSnkZbfErpZSX0Ra/Ukp5GU38SinlZTTxK6WUl9HEr5RSXkYTv1JKeZn/Bwhv6vkx+BP/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), log_dict['train_acc_per_epoch'], label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), log_dict['valid_acc_per_epoch'], label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train ACC: 71.94%\n",
      "Validation ACC: 71.94%\n",
      "Test ACC: 70.93%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    \n",
    "    train_acc = compute_accuracy(model=model,\n",
    "                                 data_loader=test_loader,\n",
    "                                 device=DEVICE)\n",
    "    \n",
    "    test_acc = compute_accuracy(model=model,\n",
    "                                data_loader=test_loader,\n",
    "                                device=DEVICE)\n",
    "    \n",
    "    valid_acc = compute_accuracy(model=model,\n",
    "                                 data_loader=valid_loader,\n",
    "                                 device=DEVICE)\n",
    "    \n",
    "\n",
    "print(f'Train ACC: {valid_acc:.2f}%')\n",
    "print(f'Validation ACC: {valid_acc:.2f}%')\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch      : 1.10.1\n",
      "torchvision: 0.11.2\n",
      "numpy      : 1.22.0\n",
      "pandas     : 1.4.1\n",
      "sys        : 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51) \n",
      "[GCC 9.4.0]\n",
      "matplotlib : 3.3.4\n",
      "PIL        : 9.0.1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%watermark -iv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "convnet-vgg16.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "371px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}