{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reproducing Leslie N. Smith's papers using fastai\n",
    "The following papers by [Leslie N. Smith](https://arxiv.org/search/cs?searchtype=author&query=Smith%2C+L+N) are covered in this notebook :- \n",
    "1. A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay. [paper](https://arxiv.org/abs/1803.09820)\n",
    "2. Super-Convergence: Very Fast Training of Neural Networks Using Learning Rates. [paper](https://arxiv.org/abs/1708.07120)\n",
    "3. Exploring loss function topology with cyclical learning rates. [paper](https://arxiv.org/abs/1702.04283)\n",
    "4. Cyclical Learning Rates for Training Neural Networks. [paper](https://arxiv.org/abs/1506.01186)\n",
    "\n",
    "Although, the main aim is to reproduce the papers but a lot of research has been done since than and thus where needed I would change some things to match the state of the art practices. Most of these things are taught in the fastai courses, namely [Practical Deep Learning for Coders, v3](https://course.fast.ai/). \n",
    "\n",
    "If you are not familiar with [fastai](https://docs.fast.ai/), it is a deep learning library build on top of PyTorch and it contains the implementations of most of the state of the art practices, which keep changing over time. As a result of this you can get state of the art results in most of the tasks by using the defaults of the library.\n",
    "\n",
    "**How this notebook is structured.** I would explain all the concepts discussed in the paper and would provide a walkthrough with a CIFAR-100 example along the way. So there would be explanation of the topic and then the code for that. If you are to use these techniques for your own work, you can follow along the notebook from top-to-bottom. For the implementations of some concepts, I would use the fastai built in functions as fastai provides a callback system that really helps a lot when working on projects in real life. So if you do not know fastai, you can watch the course mentioned above or read the docs, as the docs contains ample examples. For Tensorflow users, as of now I am not aware if some library provides this much functionality as fastai but you can still follow on as the concepts discussed are general, only the implementation is different.\n",
    "\n",
    "If you are learning fastai, this notebook can be a very good tutorial on how to use the vision API in fastai."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents:\n",
    "1. Summary of hyper-parameters\n",
    "2. Hyper-params not discussed\n",
    "3. Things to remember\n",
    "4. Underfitting vs Overfitting\n",
    "5. Deep Dive into Underfitting and Overfitting\n",
    "    1. Underfitting\n",
    "    2. Overfitting\n",
    "6. Choosing Learning Rate\n",
    "    1. Cyclic Learning Rate (CLR) and Learning Rate Test\n",
    "    2. ResNet-56\n",
    "    3. Cyclic Learning Rate\n",
    "    4. Difference from Original paper\n",
    "    5. One-cycle policy summary\n",
    "    6. Learning rate finder test\n",
    "7. Introducing Super-Convergence\n",
    "    1. Testing Linear Interpolation tests\n",
    "    2. How it was found in the first place?\n",
    "    3. Coding Linear INterpolation\n",
    "8. Explanation behind Super-Convergence\n",
    "9. Choosing Momentum\n",
    "    1. Some good values of momentum to test\n",
    "10. Choosing Wight Decay\n",
    "    1. How to set the value\n",
    "11. Train a final classifier model with above param values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fastai imports most of the things that you need like numpy\n",
    "# and much more, so no need to import them again in our work.\n",
    "from fastai import *\n",
    "from fastai.vision import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although deep learning has produced dazzling successes for applications of image, speech and video processing in the past few years, most trainings are done with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyperparameters reamains a black art that requires years of experience to acquire. So I present several efficient ways to set the hyper-parameters that significantly reduce the training time and improves performance. Specifically, we examine the training and validation/test loss for subtle clues of underfitting and overfitting and suggest guidelines for moving toward the optimal balance point.\n",
    "\n",
    "**Note: Should I use hyper-parameter tuning library?** I asked this question on the fastai [forums](https://forums.fast.ai/t/what-is-the-de-facto-library-to-use-as-black-box-for-hyperparameter-tuning/44338) and Jeremy replied, \"I would suggest avoiding black-box hyperparam tuning wherever possible. I've only used it once in my life, and even then it wasen't really a win. Instead, use the techniqeus we've learned to try carefully selected choices.\"\n",
    "\n",
    "And I agree with that. Most of the times there is not much need to use these libraries and as we would soon find that most of the hyper-parameters are linked with each other, so we should tune them accordingly. An argument can be made for Bayesian Optimizations, but I have not used them and I find the techniques discussed here much simpler and safer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of hyper-parameters\n",
    "* **Learning rate** :- Use learning rate finder test and get the maximum value of learning rate that you would use in the 1cycle policy.\n",
    "* **Batch size** :- The largest value that fits on your GPU. You can use batch sizes like 20 that are not powers of two. The performance drop that is discussed when using batch size that are not powers of 2 is true, but we can ignore it if we want.\n",
    "* **Momentum** :- Use cyclic momentum in most of the tasks. When you are using GANs use a constant value of momentum.\n",
    "* **Weight Decay** :- Larger value when using smaller dataset and model. Smaller value when using bigger datasets and models. Use a constant value.\n",
    "\n",
    "### Hyper-params not discussed\n",
    "* Architecture\n",
    "* Regularization\n",
    "* Dataset\n",
    "* Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Things to remember\n",
    "1. Setting hyperparameters is very important. Every dataset would have their own set of hyperparameter values, and setting the right hyper-parameter values should be your only priority I would say initially.\n",
    "2. Regularization vs weight decay. In regularization we subtract something from the loss function, while in weight decay we subtract something from the parameter update step.\n",
    "3. When we use modern architectures like Resnet it is better to use weight decay than L2 regularization.\n",
    "4. You can set all your hyper-parameters in a few epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Underfitting vs Overfitting\n",
    "The basis of this notebook is based on the concept of underfitting vsersus overfitting. Specifically, it consists of examining the training's test/validation loss for clues of underfitting and overfitting in order to strive for the optimal set of hyper-parameters. By observing and understanding the clues available early during training, we can tune our architecture and hyper-parameters with short runs of a few epochs. In particular, by monitoring validation/test loss early in the training, enough information is available to tune the architecture and hyper-parameters and this eliminates the necessity of running complete grid or random searches.\n",
    "\n",
    "One key finding in the paper is that the total regularization needs to be in balance for a given dataset and architecture. And it was found that learning rate, momentum and regularization are tightly coupled and optimal values must be determined together. This means that if you set a learning rate as a large value than other regularizations like momentum must come down, so that the total remains preserved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='Extra/under-overfitting.png' width='500'>\n",
    "<center>Fig 1: The figure shows the optimal capacity that falls between underfitting and underfitting</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='Extra/testLoss1.png' width='600'> | <img src='Extra/overfit.png' width='600'>\n",
    "- | -\n",
    "Fig2(a) See the top-left corner. You can see a similar bump as was shown in Fig1. This is an early sign of overfitting. | Fig2(b)The overfitting can be see around 3500 where our valid loss starts becoming worse\n",
    "\n",
    "<center>Fig 2: Figure showing on how you can detect early signs of overfitting when working on your data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important thing to see here is in some cases like that shown in Fig2(a) where you see that early overfitting but after that your test loss still decreases a little. This can be misleading in some cases where one can get blindsided by reduction in the amoung of test loss. **So draw your losses and see whether you overfit or not.** You should not use a model that overfit early as after that point you can consider your model just learning some examples to reduce test loss, which results in poor generalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Dive into Underfitting and Overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Underfitting\n",
    "Underfitting is when the machine learning model is unable to reduce the error for either the test or training set, which is due to the *under capacity* of the machine learning model i.e. it is now pwerful enough to fit the underlying complexities of the data distributions. **Whenever your valid loss is less than the training loss than it means your model is underfitting.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='Extra/3layerLoss.png' > | <img src='Extra/imagenetTestLoss3.png' >\n",
    "- | -\n",
    "\n",
    "<center>Fig 3: Underfitting is characterized by a continously decreasing test loss, rather than a horizontal plateau.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overfitting\n",
    "Overfitting is when your model is too powerful, such that the model starts fitting the training set too well. Using small learning rates can exhibit some overfitting behaviour (as shown in Fig2(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='Extra/overfitting3.png' width='600'> | <img src='Extra/imagenetResnetOverfitting.png' width='600'>\n",
    "- | -\n",
    "Fig4(a): When using WD=1e-4, the loss reaches a minimum near lr=0.002, then begins to increase displaying overfitting. For WD=4e-3, the loss is stable over a large range of values and attains a lower loss value, indicating that the later WD value is superior to the former | Fig4(b): While the blue curve shows underfiting as the loss keeps decreasing while the red curve shows overfiting where the loss first decreased and then increased.\n",
    "\n",
    "<center>Fig 4: Increasing validation/test loss indicates overfitting. In fig (a) the sharp increase in loss for yellow curve near lr=0.005 is not a sign of overfitting but is caused by instabilities in the training due to high learning rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choosing Learning Rate\n",
    "If the learning rate is too small overfitting can occur. Large learning rates help to regularize the training but if the learning rate is too large, the training will diverge. Hence a grid search of short runs to find learning rates that converge or diverge is possible but there is an easier way.\n",
    "\n",
    "By training with high learning rates we can reach a model that gets 93% accuracy in 70 epochs which is less than 7k iterations (as opposed to the 6rk iterations which made roughly 360 epochs in the original paper of Resnet)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cyclical Learning Rates(CLR) and learning rate test\n",
    "As a motivation for using this approach see the below figure, just how fast we can get the state of the art results is shown. The reasoning for this is discussed in the next topic.\n",
    "\n",
    "<img src='Extra/cifar_LR.jpg'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Sylvain Gugger](https://sgugger.github.io/) has written two very good blog posts explaining this topic, so I would recommend you to read those first.\n",
    "* How Do You Find A Good Learning Rate [link](https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html)\n",
    "* The 1cycle policy [link](https://sgugger.github.io/the-1cycle-policy.html#the-1cycle-policy)\n",
    "\n",
    "Here I just summarize the topic with important details, for a detailed overview refer to the above two links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/kushaj/Desktop/Data/cifar100/test'),\n",
       " PosixPath('/home/kushaj/Desktop/Data/cifar100/models'),\n",
       " PosixPath('/home/kushaj/Desktop/Data/cifar100/train')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load CIFAR-100 data. To downlaod the data refer to this link\n",
    "# https://course.fast.ai/datasets.\n",
    "\n",
    "\n",
    "# 'Path' is a python package which makes working with directory\n",
    "# names a lot easier, to import it use `from pathlib import Path`\n",
    "path = Path('/home/kushaj/Desktop/Data/cifar100/')\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Both test and train folder contains images in the Imagenet format\n",
    "# where every image is placed in a folder with the same name as the\n",
    "# class of the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a databunch\n",
    "src = (ImageList.from_folder(path)\n",
    "                 .split_by_folder(valid='test')\n",
    "                 .label_from_folder())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.491, 0.482, 0.447], [0.247, 0.243, 0.261])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = (src.transform(get_transforms(), size=(32,32))\n",
    "           .databunch(bs=256, val_bs=512, num_workers=8)\n",
    "           .normalize(cifar_stats))\n",
    "        \n",
    "cifar_stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQgAAAEYCAYAAACgIGhkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOy9ebRs91Ue+O2ah1t3evOk9yQ9TZYHeTYxRsY27YYmQFZCd7pDOoxpCA3trGQBzYLVpmNiQkOAACskQNtMxiawAJvEGIzbwgPY2JYHWbKkp6c3T/e+O9U8nv5jf985p0qv6slCr55kfnutu+pW1Rl+53dO/fa39/723hZFEYIECRLkWpK52QMIEiTIc1fCAhEkSJCpEhaIIEGCTJWwQAQJEmSqhAUiSJAgUyUsEEGCBJkqN22BMLNTZvamm3X+IImY2TvN7G0zvo/M7Pg8x/R8lK/EZzogiK9Q+Up8WIPMX8ICESRIkKlysxeIV5rZw2a2aWbvMLMSAJjZ95jZCTPbMLP3mtlBfv7LZvaz6QOY2fvM7C03Y/DPVTGz3wJwC4D3mVnDzH7IzP6LmV0ys20z+0szu3dit91m9udmVjezB8zs6JRjF83sZ8zsjJldNrNfMbPyDb+o55GY2avM7K/MbMvMLprZL5lZIfV9ZGb/wswe53z/GzO7nfvsmNnvaXsze72ZnTOzHzWzdSLDfzK3i4mi6Kb8ATgF4CEARwCsAvgYgLcBeAOAdQAvA1AE8IsA/pL7vArABQAZvt8NoAVg3826jufqH+f3Tan33wmgxjn9eQCfTX33TgB1AF/D738BwEdT30cAjvP/nwfwXt6zGoD3AXj7zb7e58Kf5hzAywG8BkAOwDEAjwB4y8R8vhfAIoB7AXQB/AWA2wAsAXgYwD/jtq8HMADw73lv7gfQBHDXXK7pJk/m96befwOAJwD8OoCfTn2+AKAP4BjfPwLg6/j//w7gv93sB+O5+De5QEx8t8yHdInv3wng3RNzPgRwhO8jAMcBGB/O21PbfhWAJ2/29T4X/qbNOYC3APjD1PsIwGtT7z8N4IdT738WwM/zfy0Q1dT3vwfgx+dxTTfbxDib+v80gIP8O60PoyhqALgK4BA/+g0A38b/vw3Ab934YT6/xcyyZvZTZvaEme3AH2TAEZgkvhec8w34vUjLHgAVAJ8mfN4C8Kf8PAjFzO40sz+hSbcD4N9ifK4B4HLq//Y13i+k3m9GUdRMvddv5YbLzV4gjqT+vwVuPlwAENu/ZlYFsAvAeX702wC+2cxeAuAeAH80n6E+7ySdpvu/APhmOPxdgsNewBGBJL4XZrYANyEuTBxzHf7w3htF0TL/lqIoWkCQtPxHAF8CcEcURYsAfhTjc/3lygp/BxL9Vm643OwF4vvN7LCZrcIn8T0A3gXgO8zsPjMrwlffT0RRdAoAoig6B+Bv4MjhD6Ioat+coT/n5TLcpgXcV9CFI7EKfE4n5RvM7KvpHPs38DlPIzxEUTQC8KsAfs7M9gKAmR0yszffoGt4vkoNwA6AhpndDeD7noVj/oSZFczsdQC+EcB/eRaOeV252QvEuwD8GYCT/HtbFEV/AeDHAfwBgIsAbgfwjyf2+w0AL0IwL2bJ2wH8GM2AVTgsPQ93gP31NbZ/F4D/C25avBzANE/5DwM4AeCvCZ8/COCuZ3foz3v513DUVocvqO/5Wx7vEoBNOGr4Hbjv7kt/y2M+LTE6PZ5XYmZfAzc1jlGrBQnyFSlm9noAvx1F0eGbcf6bjSC+bDGzPID/A8CvhcUhSJAbK8+rBcLM7gGwBeAAPB4fJEiQGyjPSxMjSJAg85HnFYIIEiTIfCU368vbDt0VAcCFNect9ftd38k8pLuYLwIADlVXAACHl/cDAAq5Bb7m42OJiN7ttAAAza7zPrrDHgCgMewAAC51tgEArVEfADAcupthtbrorwtLAICFYsVfKx4ezhT8UtojP16J5+53OvEYGs0mz+2fDUkVqFR8/CuLzsWqlP19LuvH/J0P/drfJob9ZckPf/29EQAcOOLZ1a/9+98KALj7ta8DAAw4Txj466jv85krlgAAUeqWDoc+hxlkAQCFXJnb+JxuXD4DAHj8wU8CAKo1n+O9t9wKACgv7wIA9Np+jlHPI8oZ8+N12kMAQLt5FQBQq/m9eezBj8dj+NynHwQAbG/s+DlKft9yOT/GoN/wfZnNYeY66wd//a/mMue/8rYfjwCgUPQnNJvx+ev3fe66XX/mR6Ph2H69nn+/sbEZf3bu3EUAwKNP+u9lfcuvTYyUUewy8w8sw0vk70nP+oD3LetThFLJx1ar+u+qWvY57LT9Gag3Ew5Vf+DjjHhMy/h8Gs8pRGD8PsvvHzr55DXnOyCIIEGCTJWZCKLddc0x5OqpJSbD1afA1adacCSxVPEVLsr4+9EwCTKI1Z3NumYv5nwbrdh5avyc+Uq33fHVV8iizGOVhv59RGSw1qwDAHa43YD7V4uukoqZOIkORn9LgcigyetrNV27lbI+/lLOtfFoNHN6bojkcn5OrfDxa8bVSY5zm8lzu7KPNVNwrQJPiPV/eb3JMXzfQZ/ow8SD8vtbKfmcrey7BQBQWtwHANi+ug4A6GV9rjPRwF+7W9yf95masjcYxGPo9bktnx7NaI4bD7hPiUiwXJlvYmipzOeQ6nqkS6G2z1DLZ/isy2OXj4jKCskzksv5tpVSkZ80xo4Rjfga+/04B0Of/07P0cpg5HOW529m1PXvC7zn5UJh7Lg2Fszj//Hzg/FXoRce2zKzfZABQQQJEmSqzFSR3S5ZzFyhslyxshOrUUQNVCz64XLUZvVGigUt5FByDVeOagCADDV+RK3Wqvt2i5Efq5Vxn0KVGmb3iue8GFf8cxtXAACbDddmXfourOdjNi3mAJYW/JwlrvCy9dpdP0ePiGJEv8jI5h/hKcoWzsoWdm3Sop0ZRfQrDHxu5YMoEEmMMonfJ6I6zGf9egtFv/4R52jQ8WMInWhOY7uV/oARNV6GY4qGg7Gx6Pno9XyM/V7i9xlw/MORH6NLbTnk3Ob5PBSrVb5P3bA5SIbXKHgcRUTL1Ky5nJDDuImud/lc8hMqUrPr+crniUoi7ePzFPGDIee1P+D94LnzRb+HlQrRzQQK6PK3YhxFlH5ObTT2nV4y42+RyeieYqYEBBEkSJCpMhNBaGWTyA4DV74BfRMtaozza55wWck5gjh4IClKtLrHs1NzBdcY8mJnuKp2th0BbF30JLUifBVt94kgFtw/sGfFM4sLJbm9fUzDAaMhHEs256t5sZDY5KuLqwCAxZprqxERxPm2Z9oOh/RYa7ktzj9JMZ8dn+Nux6+r06IPpuKRhnxxGQDQH0lru6buDVrxsfqcO4NvUyj4a57RJwIBODkVaLT8+40r7oWvElF1dvze5Dgvo2FmbP9c3ud40KMt3enGY+jQV1Qigizl/XVEP4a0o65X0YF5ifwz8gfoEY99DtLINqlL6VNJIwgih1JRz55fa4vXZLxYRTMGA74K6RX9N1FlSKdc9d/RoOf3cUA/XGeg+0pJ+xFG46hXKF/wQ4+2EEQ2IIggQYI8U5mJIOTJlQ06HMm+cekSYazVnbsw6vr7W1f2AgBWa4lH+tAB/6y05Fo8V/bv5IntbXskYXvJOQhbV9y3UG+6J7jA1XlxyTVnhTZrpeZ29e4V369NrkZEjRWluBgZopZW273xA3qNNcoCEVGWV1gqzx9BdIgESrTdu7z+NrV4seTXnav6dcsDHtuWg8RWzmddkw3NfQ1daqxISGBELUkl0277uXbqPveWZ9SHT4nuVdsVGEaEEIpMtRlR6jUa8RiGjMtnJxBfxLnOyJNPfkx3MG7r32jRMyFNK7ZDJqPIAKNFcSRCiIJav5A8X/I9CDnECIJITtE/RTH6vHfZgo9hYcnnqLLIuaKDoM+oRp9z1pefbaSIxFN9ZRnqfkWPYqQkhESkKiQ/TQKCCBIkyFSZiSCKtON7Pdq18mZzNcpRG0jjarVRHHiplhTBWVp2jbewx9l5RWr+HFliiouP7jgGAKhf3QAAbPJVK7fYjWUy8mQjHueVMNSMDu22Rj3RZjvbznq7fNFX5DUuvEUipTKvq0w7OZ+dPw/i3Dr9BRXGvunPqZY4X2Ldtck4ver+E6NfIZ/SCEPen3zR70OfcxJFYjH6PIxiN7u05ohjcLS2uOq8CHn07cI5Pz41o9BAs+EIotVKohgWqywiRXrqQR9ERz4WsgIbnXHG4o2WbGyEE10pMkcfREYRHo5/MIEGFHUCEgRRZkSpyGhEhkRHIQch81yeSLXqx1haJjrkb6JDPkmkZ3vIez9SNIQck2GCILJkucYciYnwhZCEvg8IIkiQIM9YZqrIZdrg/a5r4RFXrCJX1xq11i5qqIOLjg4OHvDaFrWVXfGxCmTIFSq+OpZr9Pgu+Gpbqsh+c1/FsO95EW0hANrm58lzH1L7VchPz5ERp1h+q07ewCDxqDfW3BYf7DiSyNDuLfN6lhc9l6As34PN1x4GEru0F1G7LLpvZdchn9M8HQJD5ggMl5wX0u+J05Cw6jJ9/t9nhKDd5jH8egc8RqQcAM7p1kVyShon/Ti5NT82tevmJb8HxaKPcWW3j6HdcD9SJ5X/ks2Ksad8A+Y4MNJxed3P1e8rD2G+3BPlIsgPI0Qa8TkSYlX+ipFzkCGaKqaINopiVKr+TItlmeM5dG1Z7lsib6i0IP6Ev0b8XsGJrPJW5EOKxN0QOkvNWfzvaGybhE07noORu04YIyCIIEGCTJWZCOIAs/s6Dc/W69M8XGQE4kDNtdstzOK8/dAxAMDBo/6664Aq1SdMuRw9tnmabvkcWZpZeuO5qhbKbr9lqc2HHdc8u4bOg9i84lqtUOV2ZK2JGddt0s6LkryAjHIQWswpYBRmaYFZjAd93IMFt70Hlnio5yXi28dxa+aU5JnnIg2QZzbr0qKPdcRr63cT7d1vOEJqt1xLK56ezbiGGzLiM+KNlS+iXff96ptPcj9qLmY4Kntw35EDAIBl+hW6jGIoAxIAhkREbWaEyg5vdfy+nL5MnwuzUWvzZlJSg8bsxoH4ENTiRECam/hzIY/UL6hUYvSCfomCfFl87cdRH9+3TN9Dbcl/G0IcPW4nf1ORz4SyPYdEB/kSEUqqpotQo/wSEZGomJUxG1oIKfggggQJ8kxlJoK4lZyF7XVnNyrr/JZdjhjuPHIHAGD/kr9fZf0AxehzlUQbiMegFSu2TSdsVPHVxTrLcfXN0G6umWtMdTosiufAlVO+iSxt8Uo58TKvLDtS2KKvodF0rVbZ5SzPXWR+brPVZLaYsDDnJZc3fJZ37aIHWxmx8kJrwacnXBmbWWagZnLJmHPkTORbjjaGnOsh52pEL/lI2oZ1IxZWHaWtHvZ5KeVZc4PIQHyHETsOyEbutNxflGZDShsqw7NI5mGO921p0Z+VAmscLC/Mu82nz0mnq2gK/TTF8SxPzbPGLQSRzlguMBcjRzRboV+sRD5Emyxf/QbKVf9+dZc/j3myfhv05aDF7E4itCwRRZ6IQnyJcjqhgtfRZySpT6RmE3wIIaPsdSJ1AUEECRJkqoQFIkiQIFNlJr64k4VDNq+4iXGJdN/DS15IZP+qh94KBYfuO12G07acxFNKFf+QE6pEYrPCLCIjxVBOabcKJ/FzOSHLWaZsM5ElRyfMiFCqRZpviVArzUJVNGjEsF+u7CZUfejH3t4mRC4zFbkwfydltyvyEkuHqYJJbGko/TjOIaZcg25LOFyskfBEmLyz6U7nEZ2OBTpCLUszRZ43Jl8VGYYb8WR9Es7yGaV9++edltLlU7BbkJz3sUqTT0VODtDckdNS5Kt5yYXL7ux+8kl3yCpB8dABN5sP7Xdzq1wcvw5B9X6UmFN5Fk6Sc7JSmijsIrNZ6dw0p/Ra1G+EVIBK082ErW036SKG/LO81wskIu6imQYAAzqmNy/7Pe403UzRY1Lk2HRf8rnZz3hAEEGCBJkqMxHEHcfuAQB0dnzlql50gszuKunSedfixgIxEUOKxsKx7VTSzrbIHlxxh0w4GY189SvTMWikdyvZJIrTUxXeowbKUmOSPGIslML8IlQG44lXfiF9HpNkKpKvmkq3Hao8mstwMN/UYyCFAxS6kja+XnuCOJ33qeQuhRYzmfHbLW1dZjh73+0vBABklZLN688Y729OqfR0LPPmtJlQ1iVd2lJjFW29rGQ7Otb6TJTb3nSHXL3lr705k9M+9ZkvAAC+8PAjABLkdtft7rCuFO4DANSOeMg+SwehkN0whZZUPKZEB6dSDnRMPcPlihIPPXRdW/RXOXtVeFbnih35nPcej1cjgtizNyEktrnP+kVHEPWGz3OG9PoqSY1yiOYLiRP/WhIQRJAgQabKTAQxYoLQvgNeBr1J232h7BpHK2V5lWEa2laFoYeK2p2kHPfFC+7HGF701a9UdU2y96BTqw8cdr/G6j5fDUvUWpm4bDc15Egl6kjuUdGNPu1gppz3RdnNJ6FWo43XVvo6fQzlsl9nnglko6yIKfMvOdei9pAtLHq7Enzke4gTdjjGLu3/TRaYBZLScUurPqfd9ngjdCGkAgvj1A66z6nGMKcKA/dIjNq54mXyJ4vTNnYYluMYdy0nafIiromGXKW9vb7G6+MxlFOk9PV5yUOPPA4AOH/J500Fmlucz0P7nEZ+6JCTwgoMKerRiIsoIRW+pO9Bvw+hvzyL2i4sMEVhl/uGlpb994N4DliCjuikIHIhyYMqMFOQjyNVtHbA579FslunJXKc79ujzydPpmK64M21JCCIIEGCTJWZy8cm/QQ1enRXWcRW2k020yILw6zu9qhAKeOr8NrVtfhYO0wRvrLuK3XnottGTZY5U+EQJbaIZprjsjqCn7PJlXFzzdPAt656xESKR0lhst+yqTVwRHsrS6RQ2UP/RYk2IDVpn0VLeglLe24ihKByfkMViI3rsftLn2Skyxe8uMu5J127t3bq8bGUy1MgUtqz37XhyrKo1izOGymixNYDKo5C/s2QGqpDn9KQtG5VxxsRmag4STVFkBNpSMQjFS1WmneXkyysNkjZ9POQDUbc1H5Bfpqdhl/jOdrybfqvFkmyM6M/K+VvyU34IBQFE6KIiAyqTA9QCYTKgkoXgMfkvSe6yudJ7yZZUCUDK5zLXiq9/hL9hfqdCOXleLNE974ecpAEBBEkSJCpMnMZ+eyTXwIA1Og9bdHTXKSNvpurkmyrKqmjtUW+7luKj7X/dudMNBpsuddyrVNm7Fhlt0bU3nqN6PfocPuNyx6DP3va25ydPu3IZMBCniv0h+zZ69py1+498RhUpKa21/0e62xI0mMR1lxGFFl6eG+CD+KW3dRQJD70+vJF0GtOH4Xmr8vWAhUmtd1+6x3xsfqck7XL7v8ZdJTeTTt6qMIltEcL5K1QiYtqXlj2+dlsuLa9etaLE7eazou5etnvhYq+FLKpeVN59ZFsYNdgaomwucPngb6k3mC+Pgi12BNilTlPOgpOnvFre+hLjwEARmrXWB73LwAJh0eRAWnpskoRRIo+OGKoMYqRY2ROZfDVYEcIsJpnqYSyksBYYIbtKLfX1cAo8dV16YsYiSck35UKThNZxH6SKRIQRJAgQabKTARxftPLmQ1Ycm6LHvI7DnlU41ZqfdnLKmWe4QpXqCSJQ0W2hFvZ755bxfdHtO16XPFUBz0bMyT92AO+ZsyHrEaxu/e4VuywGbAYbyWu3osLSdk7JS+BZdo3tn0fFW/NsbxbvsBirfn5r5+lvIqDcF5UJm44jiB2Nt3XID6BEtEKpYQZV6NPpUjvd33dfUI9avoB1aUSpQoq46cSdEoF5iEvXLgEAHjocw8BANpsWaiiPEbEUsomfoRahfY2n4ket20xEWlrx5+tnd54MtTcZIJ2oeiQ/C5nL/hv4L994CMAgCdOOnraywiEuAwAsMLoTdyOj4g0yxIGeTJUK2VyEbKODHpkrOp3JOQ3YBsGBUrETK5U/D6JZ7GZaizcV9tDpaNPvMpvGMVl/kO6d5AgQZ6hzEQQjb5rWJXCulx3+3+l6XH1kZrWMO8homcaQzbHScWI1WJNORYjeq+7XD3Fb4jLdQ+0L1c+Fm+pkXe+wNeDxw7yuLSnuZs88pZNmGIb1FZbTdrtRCFGL36GvhStwkqJnadc3HCtvI+lzAYsAKMS8zk2I97e9CjOww8+6J/zXhw6nDQrqiqVmlGmkVoCMBql6nQR0Urz8hMAknL5mZpHpSJ5wFmi7+gtzgnYuupzfOG0NwFuNBkVySd2uZiUPRadURHhHZYEbDEuX+/5OXZVE8Q3D1FDHOVY5DgpA0bN1Ej3Eu38esMZl/msmL3JT2iBHI/aIq+B2nnEiMfSin8vnw8PjS5RYH/YH3sV96VIVJjjfSmQBSkf0tZW4oOoM3ohB0aWSFpRPbUDjPNDJiHUhAQEESRIkKkyE0FcvOya4a7jdwEAqlzdN3ZYin6oRrksRXbJY8ZZavNeM8nMK1Rl36uGt2uZ5habrdAuVkZcscASdVqhqeVloubLjFvT5yAbS31MxTDc3knYgyfpfd8kgogYjVHBlIbi+Rlf0bPX62x6A2QvS59XGPPusciI2ujlmA3bYOHdS+dP8dXzZD7zieRYd9x6GwDgZXceBwDUlNdCP8b+IUuhnXcOxdkNf82x6Ey25kihttdRWn/L73u54gd6bM19Ulev+n0vqJR6Kgu2ySIz9Us+/jrv9yYRRJ0IcsRiNdnMfHWWeAy61yrKotKHikysLLvPYXXZUZW0druTPOOX13weTp/2e1Fmw+kjt3kkrbbkiHVxifwHFrdV8aMmWx32yX5Uub4Os6QzTeW60K/Q9rm9cjnhG9UZFVKzYeWHqOyd2gLmJwrfTJOAIIIECTJVZiKIq+vO0vvSULx5FiDt++p5Yc0Rxi0veAkAwBjVaA/JkttIVtf+Gdfeiruu7HGegg3Hvaki0sl7L//sMM4UZXFb2nHy0KuFmrz8dXLp/+bBz8Zj+OP3fxAAcOvtnqV67Ihr2O0Nll6nL0UMNpVQm6ccWHGt02Ouyfa6z9vpRz4FICnme+qRvwEAXDz7KAAgIi8iShWM3WL0qT10zaMsQqGrqO/7rDVd812s+zxkRj6Xe6uuLfcyl2OZCKK5QUYt7fKtLZ+nRWrE0/UkB2ed9Qn2sjCrSu0LxbVJODBqtHTB23mIGLzK+lVp/yoLM99z9+0AgJfc55muKpF3lYxg3ScgYf3qGKfOOEdnZ8C2iUS9q3u82PO+g/4bEGN1RG3eJFdoa1sohUxJ8iNU42H9vHM0rpJV7PuK68K2fETqdK+hTG6LEJFqV0yTgCCCBAkyVWYiiAE95xvbtD0ZLxdr8c8/5bFh4/L0ype+DABQ282ite1kdW1d9hX34uNunzV2+eq4wqa+BfLSjTaTohwqtT4y2qpdViBiBmGxzJb2bA7b5ZhPnD4FAPjAX/xZPIZPfd61bhe+zZ0vcN/KasFXcrE0szy32tnPU0S9GBE7XXrCoxSfaDmSyNALfZ5a4+yTrkX2FJ0HcZTNjQGgwjj61rpvk2EcXQhJ/o3LV/z7Exf9HFk1rCXKy25S26gBDL3uezk9n9n241xYZ0WuVD5FmZpYFYyazBXoksWqrFszZoa2xjNOb7QsVJSXokbFfu37d7uWf/lLXgwAeMl99wJIWhuObvNokTIrgSTq0CdbNEN/2yOnHwaQZNeKzyA0GIfe1FaS3B2xi7e2ncFqZMB2IvoZ6D9ptxOkPhAii7sosq2EqC10quzf59nThXxgUgYJEuQZykwEIT9Ak/UC5eWXp3mHNnrzI66lL7dcA7/p9fcDAO7Yn8Tkl5n/sHPFvdhXTrt/I2qxnL07ylGujudB9Oq+/TY5Gf0FxoTp7c+XuWJm3da9suX29If+6qMAgE9/4cF4DJv0/H/p8S8CAA4+6PUPXnzvKwAABTLUOgO/zoVaoo3nJaevuLbYU2MD165fP6chjhAMBq61u2y5t0E+QRFJCuoCY/sliKfi15eNIz2uxZe53YuYt6L8jzwRSINRK7VpU52MiOzXZaIDUgOwM0i06tKin1P+iXPnyXydQGdi0tbnXMXrFS9z/9mJE16TskLOwate9VIAwO13uJ9K9RPiRBUVsBglnA/VkoipBRlVUVMNBmZ3VhTRYx4SIyZZ5UVQy6suRIvRDeuz1L5a+qltXjr1JW4ZqEEYr4v1OGgFLDAimVzXtSUgiCBBgkyVmQgiL5uJWX45rkIN1nDoMjZ/htl8rY+7T2KHjT/e+JrXxce673a34W5/udt0m0QQWSo8tX/bvuLa6hJ9Fk3G2Cu73cbef+QFPqZbHJFsNdyX8fCjznD7+Kc/CQD45OcdOVzZSCosaYXfYlXmE497BOCeO31Mew+4XXaqcQpAyo6bo6yR7blSFUeDGafU6sO4y6za5fkgdxiRiN3VABqslfE4mY4HyPBbpQ1co8YqUesUZArLD8Jjyy+gxsLb1PanGb3IciwypdM+CLWN22R9hQvbrFdKs1kpOG3e/+ga1blvpPx3X/e1AIBXvMKfAUUpVshVKDIzU8+OGj7pCqNr1NDsMuogHoMyKcv0OSyyBqj8MyX6HErM0dAMqL5GhwzWHiMUPbI9+0SRMfkHSf3KiAhCPIe9+/z3cviw19ZUzsvoOrVOA4IIEiTIVJmJIPYxDq7oxZGj7lOoMyPvoUfdlm/QF7HD+O1nPu+VglWxGABaX+Na6NUvddvu0Ks9IzTHIHA0ZB47q+O0Djvr7HDVkUeh5jbs2aueXffJD/wxAOBzX/IxPMGoxWXGp3eIclRRB0BsGyr2W+PKvYeVsG673WPebfpanmQNgHmKyiHERa05/jZrOWTz6pfh3+epMdapzevthEdQKDKaRL/Fma5v8/iGozI1eFXdSCEHxfOV1dnmoGRL94kQusrloBbd4VRnK0k+hfgFa8w+laNiNFE5S1pw3hrrIPteHDniVdMyHIeyZLucdzF5s9lxxJCuKCW0p9yKCOMVoSrKbF3wZ3tl0aNElQX2eimP56HkmX804D3dHPizPWBWcoZ1JqNRwvidKEFcJGwAACAASURBVDwW9+JYXvFzVlkJW4hodJ2aJwFBBAkSZKrMRBCyv7ps3X7hvDPDQC9qiR7QiCtflZpDlabPnb8SH+tPP+gsxtaOr4IvfbEjg4MHj3FfjzvnK+yTsd+PdfLcKQDA+//gAwCAT33WfQvr1IIdGrNqzz6ilhvN8h9w0ZR2i7tYUXuoi9fFS+dmHOTGSKevKlfUsNTWmYEGTQ8353jXirMcd+1y23I7xWI8vNd9Krcd8G1On/FszcdO+XVdWnetfpX8/ZhFyPvbH8nWZeNd2bUmO5c9Iuitb3LsS8MkLr9J9mWHWlXMRWmu+D7FKm/azNwYiaRJOb8ECkn1cBU75bULMeh1lHrQVKVp0B+MHUM9QZaX/BlfIIJYqPhrhTVRM5EjAmn3UpaVp7jd9ob7zlod1UBh3Yh+iouhrGiiOlURry0wm5o8oTYbV0fX8bMFBBEkSJCpMhNBXNh2e79AL3dzx/0DnYFy5VkVWsarqaclq1zX9sXHMlbTOU3mX44L8/oFt4tXlt0eK6+4h/fzJz0q8f/+/u/4WNZ8LNJEctxW2CnoKJFInp251jaZabidRDG6rMZcr7tv5Itf/PzYOK+c8YjImVPO9rx07uzUublRImUgr36B/T6G6l1Jw79BuzSX8/l6xSvct/PEySfiYzUvO1K42mHeACs4F6jhe9SOGy120OLcrbD6NTrui+mwN2RHHbqo5qVcVf8wq/ufSTgOPfUtoXYVchiOVCPRtxOdwGabxM+6iP0Yl5RmvoNcIxpvzCuIS0/z+1QNTaGJKEamqkzGPhnsQNcjAthghO7quqOsSsnvpfKKOsylabL6FkwVqcjEJNNSHe2AJPtUtViW6WerldV3laiQOS/DYagHESRIkGcoMxHENll6qlaUZ0UiebPFLTeuur0t1yQrNY+55spJ1+GFBWclHrzVq1u//JXOYJNHd8DI8mNnvNPRu9/7+wCAk+e8RkHcF0L5+hxTteL22713exxb/o9Fnk9sNABY3x7vVnVhzas9/9cPekTk81/w6EuZ3a63iKDmKbcfZv4DKxpt0T/Qq7MWgCpQM69/m5yThx76DABgkOLlb9EfETF3ps56Amst13rrPGaXmi5PtakK4qMe81yYJ9GntuHuKJI4Uc6O98NYXUy6usvIHZAkIW7NKG66ytqbg+sYwzdIpPXlV5HKVA1QoaW4ToWQg3xE6ShZpG3ZY5PclTYzVy9d9OfJss6/Wd3vv5PKgiMH1QAt5tintujvl/Luu1BlqUg+ITIwh6N0pM7HVaQfbVmMSdU+iauj01cXEESQIEGeqYQFIkiQIFNlpolxx/E7ASRQKYrLchNyMpyj9nkD0YFJEGkPkpZgT15xh9lGw2H9Lbc4lH7ZHicpXbjon//RB94HAHjs1Ek/Jx1jRthWYvHbKkOrtSqb4ZBsss4yaCrR1ukk6cMj0ZNjcgtLyDPJ7MR5J12tMhw1GMy3eAkAHGDjnA014SX03bXfiTzLDF1qPs5d8rH3mPhz9nxiFq2v+XfLbBY7YHn/tTabERFeRjxHR+HpdXdm7quy2Syzu3ok5nRptuVGLHPPcw8y4wVgASBPiF7M6ZVt6Msqgy+oznZzc3ZSxuFW2Q58URg5y2uRSSvHo571UYpWbjR/RQ+Q57VHh6BSE4oMPeYqLA5cZjnFMsveldX6gCF8/s7E/e/L1OfnlkmbCX7OFTaJ2rvqv68cTYyBxq8m2CHMGSRIkGcqMxHET779JwEkq2SDzVu3GC777Oe8nNsHPuAkps0ND9dsszx+t/9IfKxS0UM8l6862vjTjzD9lAlEf/KhPwcAfPyzfswuUUqWSSUVNnVZXXbSzyoLo8iZ1Gz72FTcRg7InVZC9xZxRSt9ojTosIlcK3QZXurNufwZAGwxnHmViTkLRV/D9x9y5+6eI05RbzD0tUbHcJVEmMpS4hjO8xjt4Tgb6SBRysFVd/DWGaY8y1TzLrX6bSx2m2G68SMnfE7VUqBCTdeLxqnZnVQmtxK3ymphT8ebstLLRRWN9ff9OUOIOOxK515EJKPyg0W2fMBEyzqh6nSutRmdjKXxZ1aJUSKedZlGP1STXjoZe305ohVqZYlHIvQrl71x0TrRpfz2S7Xknufg57zruN+7VT4PeuZVKj92Vo6CkzJIkCDPUGaHOVnqas8eT2i55agXWFlaGm+Q+/GPfwxAQvxQ49l2K2no0WXhE4UrP/mQk5QusXz4k2ecnCRGsYhTGa7KFSao7Fp0BAFSRutNH+NJhkfP0ZextuGlwEdI2YhxUVGFSkUdJhmGn2vFHgxTqnBOcpHkrALJL+UyU4NJxy1XGL411zZFFQBhEZJ9tDkBoMsw5xJt3QXa/floXFsORq4N91V9rk9c9nt14YojwRrLsjWIFHSc0hLDmSzRJ63U6SZhNxFzYiJUZnzuS1Xa4Xzf7cy3YIxozUb/TIeJemp9qEK/8lspiWvI5y+bsv91TWrau7rivqzeSaLCdZ/Pw0qfJ2FKFHeRxFpsI1lnEuTamqcsNJjw1mHYdIFpCcdvvTUeQyEjH93C2LFj5BCNv0ZRQBBBggR5hjITQajYxKOPOrFjc9NXwBe8wIu2aKWME1ioQRRpKOaS5r1q+94Zuea7ymNt7jgCyNNDXmR7sbKIHfIqs8Dq9o4jjgZ9CzsNP865K44cuvTwjkZPtWWjCU+1CC1LbOqjpC2lit8M6s7uHMu9cekWklDT4lzWNVpu5H6BSuS28s4FIqZmUqr/QK3IfalF6Evosg2eSgp2+PmQXva9Ff9+veH+nFPrmjdqOkZ/BtxeJJzKgui8CfJqNnU/VAqNiJAoxCYat/TnDNoSCjh9WZw/NcgV+WmymbKyytLj1+9FLq59e5wIZZFfqxDEFl/V1EbtIpS1rShRky0SVYh22CBqaflYF/JMklxNfmeKSsg/oSS0BDFMvF4nOS4giCBBgkyVmQiiRu+oim0us9mGSmVJhCDUMn3EdefowePJiZjKurblmu5q01+H1NMrtK3RV6UUpj0zFqxGwlt1Rxw9eYLV6HTKNaSbk+azSm7RuJX4xKYzSkUWIppz6vG1RGNpU7Odf+JLAIC1i84rqbO5kbR5JlUCTXZnh9GYdleFbkVvZjFbKIV5PBqxRAe+Wurt9Mbfi7ugxskLJZ/fciF5rNQqcKtBzgShkXgRcYn42B903Sl5VkVcgl5vnFugZ17X2FUjJ/kesrrGRMeKK6HPKqQ5Ly+5D+/qmj/zVy54NELFjIa58VTzLLki8r/ZgPelwWe9LV+a6N/J0x//p2eYbwWoEyQhNDhbAoIIEiTIVJmJIM6f90YqstPkpb182dl60UTBS2lrab1yJYnP3nXEC8RcZXTh1OUTAICLm76aLtZ8lS2z7T2k5VSMte77bbTdB6FCHteLmqc1UjE33ka9xWS03kAoZHzVnXfqMQCsj1zj5hkT38tCpxUWU91zyx0AgIV93ifgSZbcA30yra3N+Fg5xtcLhfHW71t1RyNKZBtOFkGJtY+aybge2c39pSi7fRVlJRoQdyCbTFxfzD1+VC6N+63kK4p4o4r5+eosRVU0DjEnhaLk0xLCmP3AjWt2pX0fOexNkD9/whPqTj7h6G8XE+vEBdrF0odLjFwZm/Z26vQRdYmyIKaqWKApie/leOGkuBTdxPvr9UoOCCJIkCBTZSaCuO+++wAkUYwrV9zeXU3F2oHEe6uS4IWMH/YSIwsAMOj5yrbJBrHn108BSIq4FOmR3XPAS6cVObRWy2O/Xa6TzT498WxIe52q3WM2VpE5A2K4dZlr0eNGsQ05s17djZVbjzpj8sJFR1YtNRBiXsuA16/Iw3rdUdCxQ0cAAI1U+bHtq466Fqm1VbC0zChTu6WCJGyHRz9Gk1GObeZ1t4gUxARcrdE+j/0GbI/AFPRcKhcjNxRbcLzwSmIL+3bRRI7AvETFXOgqiUvWKzqRY/Soz8I90YTVPuv5y/H3cPiAI4gje/wePXnBuS498kde+tIXAQDuOeTosACf30tszivwEjspVNvmGmNI/GvXfp1EFJnrOH0CgggSJMhUsUk/QpAgQYJIAoIIEiTIVAkLRJAgQaZKWCCCBAkyVcICESRIkKkSFoggQYJMlbBABAkSZKqEBSJIkCBTJSwQQYIEmSphgQgSJMhUCQtEkCBBpkpYIIIECTJVnncLhJl9u5l99GaPI0iQa4mZnTKzN93scTxb8rxbIIIAZtYws9v4/zvN7G38//Vmdu7mji7IV5KEBeJ5KFEULURRdPJmjsHMPmxm330zx/B3VcxsZh2XZ1Nu+AJhZj9sZufNrG5mj5rZG83srWb2e2b2m/z8i2b2itQ+P2JmT/C7h83sH8w4/v9jZh81syW+/04ze8TMNs3sA2Z29EZfY5Cnyjwf4uegvJLP7aaZvcPMSgBgZt9oZp81sy0z+7iZvVg7zHrmaVZ/zMx+zsw2ALx1blcSRdEN+wNwF4CzAA7y/TEAt/MCOwC+AUAWwNsB/HVqv28FcBC+gP1PAJoADvC7bwfwUX73qwA+AKDC774FwAkA98CrZf0YgI/fyGt8lufrOwC8L/X+BIDfS70/C+A+eDHC4/zsnQDexv9fD+Bcavt7AHwYwBaALwL4Jn5+Kz/L8P2vAbiS2u+3Abxlxjh/Et4avQOgAeCX+HkE4PsBPA7gSX52N4A/B7AB4FEA/2PqOEUAPwPgDIDLAH4FQPlm34e/5T08BeAhAEcArAL4GIC3AXgZgCsAXs1n/p9x2+LTfOYHAH6Az/Xc5uhGT9ZxTsqbAORTn78VwAdT718AoD3jOJ8F8M2pyfoEgPcA+AMAhdR27wfwXan3GQAtAEdv9oPzNOfrNv1wARwAcBrA+dR3m/zuugsEgDx8gflRAAUAbwBQB3AXvz8D4OX8/1EAJwHck/rupdcZ64cBfPfEZxEXg1V4EbcqfFH7Dj7YLwOwDuBebv/zAN7L7WsA3gfg7Tf7Pvwt7+EpAN+bev8NAJ4A8B8B/JuJbR8FcP/TfObP3IzruaEmRhRFJwC8Bb4gXDGzd5vZQX59KbVpC0BJsNTM/tcUFNsC8EIAu1PbHwfwzQB+IoqiXurzowB+IbXfBryC36EbcHnPukTuV6jDUcL9cHR03szu5vuPRCpXfH15DYAFAD8VRVEviqIPAfgTAP8zv38AwP1mtp/vf5/vbwWwCOBzz/Ay3h5F0UYURW0A3wjgVBRF74iiaBBF0Wfgi/o/Mi+R/j0A/iW3rwP4twD+8TM873NJzqb+Pw1HBkcB/Cs9m3w+j/C7p/PMp485N7nhdmIURe8C8C4zWwTwnwD8O/iKek2hz+BXAbwRwF9FUTQ0s89ivP7sIwB+GcD7zewNURQ9ys/PAvjJKIp+5wZcyrzkATgSOM7/t+CLw1fx/dOVgwDOTiwop5Eslg8A+CYA5wD8JRwR/FO42fDlLESTkn6QjwJ4NR94SQ7AbwHYA6AC4NOWNPsxAFk8/+VI6v9bAFxA8mz+5OTGT/OZvym1IW8ogjCzu8zsDWZWhD94bQDX675YhU/GGo/xHfDVdEyiKPpdOHz+oJndzo9/BcD/aWb3ct8lM/vWZ+Vi5idaIF7H/x+ALxD348tbIC4AOGJJuyfAH9bzqfO8jud6AO7Xee2XcZ5pD2z687MAHoiiaDn1txBF0ffBTY023NzQd0tRFC08zet7Lsv3m9lhM1uFP6PvgS8A32tmrzaXqpn9D2ZWw9N85m+G3OgoRhHAT8EfhksA9sInbKpEUfQwgJ8F8Fdwx9WL4I6ea237GwD+bwAfMrNjURT9IRyhvNvMduDOoq9/di5lbvIAgK+FO6LOAfgIgP8ewC4AD34Zx/kE3NH1Q2aWN7PXA/j7AN4NAFEUPQ7/gX4bgL+MomgHPt//EE9vgbgM94vMkj8BcKeZ/VOOIW9mrzSze4hQfhXAz5nZXgAws0Nm9uYv4xqfq/IuAH8G9+uchPuIPgU3qX4J7ks6AfctfFnP/NzlZjt1wt81HVQXAbwj9f5TAN6fev90oxj3wn/s2wAeBvAPJs7zu2C0ge9/Bu4DyT2NMX4VgMfgD/t/mBxXaru7APxXuHa8CuBDAO7jdyW43+EkgB246fiDN3v+w1/yF8reBwkSZKoEJmWQIEGmyt9ltluQ64iZNaZ89fVRFH1kroMJclMkmBhBggSZKjMRxMc/+kAEAP2Bt4nXYqIW6RlaKNm4hzibufJ9JpMcXrHuiA1Nc9pGoV5+b+wqOuw5/yliQ1n1GM3lshyDzsFXUxt337/b8zF3u514DN2ud0G1nI+rWPZmtpm8N/WNdIxIjVr9el/56lfO7nD6LMr3vvVnIwCwkY+/VvHrK3IqG9verLfd9HmpVHzsCzVvSJwvJXO+stcjhhfPe9PlMyfP+BcDn8Njtx4DABw6ss/PUfEOtoMBG+lyDnu9bQBABH8tcP7adTXv9f1Wdy37GIqJ0un2vPnykM2WC0VvHLy0xBb3fHRaHb83O80mAOCf/6MfmMucf8ubjkQA8KKD3jC3mov46vP72FUf18PrPpxXHNkFADhW9LntdBO6SD3jEfw2H9btgTcGXm/4Nps7bLzc87moFH1+q0W/H7sL/lqI/PO69/bFBnxehyMf47G9PneDoY+x19mOx7Cc93MtsCdxl7SS3tD37bFTdd7folzwk/y7d3zumvM9c4Ho8wHpD8cXiNHQB5HlAhBl/POE8MIfuyUPimX0MPE7/rDjXfjD7rPTdGboA69y0opl/yHkC37lGS5S6gotIDRkN+kMf9w2SvFuBn6yHhe8IdsmayzI+awlo54/ulpedvJc1vz6a2W1cvaHqscHstnwsde3/QelLtWZXOJWunzZF4aLl0ha5f3Yf2QJAHD0+IofOuPHaPV3AAB7VnzBGPV97q5c8vky+DkKmqfSgO/Z1Z0PXYTkRzMg0XXEhz6b9cVEPKxuxxe87W0/92AwX55Uu+9z0h/5PLf5jOTNL8ayPs48f3gDdvke8pfT53MKAC3+38v7l3q2B3y224Mhj+3nLKs7Oh+zOs+d48+yz+PqJ5LncXct+/2LIj/fxQuteAzqWF8p8lxZLRCZsX2GfG+YzYcLTsogQYJMlZkIQqZEZNE1P88xozc2OaiJ9T6bTQ6foUmAeBu+j3yl69ZdkxhhU7ng+xYK0iiEvVqx+SrkEHGdFcoRKCikNGrE//s0X3rt1viYhGqEjCYnZA5SqToEz2X97IWcz0+v5ZohQxOrvOAmRYFzfMtRZ1BnCglR9cEvejrFNlHGgT2rAIDlXQ5RM3miLWqVbJ7XP/J5GfJerFBj5TLUqhn/vFDwMeWzjiyKJX+1fMq0LPl96bb9/mYyfn39rl9fu0WTid8vVFdnzs+zLY2ea9CrfnosFWmyxiasf17KEE0LcRSJgFIauEv02uM+7T5RH5F4KefbLmV9ThZy/tokCm6TZFzI8PfDk+dG/nmfrxHtsl18BnZyCerq8nqE4nRvhzQbR8QEwwHRTgoBXUsCgggSJMhUmYkgCjQqbTjuvxBSKGT9+wRB0KaiVstkkvUnY+MIIkNU0qPmiLotHpPHoLY32tY5Ord0bCGHkfwieqV/YRTnGqX8IDwnFWWi1WS/F4o8t03sOT8pu1KI56HIeSgXagCA2qJrcyoTlIp+Dw7sc0dWvbkWH2uhJm3tc1GgppHfokzn2N5V90XkMz4Po55rlWbD/UFLNU/4zBFZDUf+/eKyj6lP27pJh3A2U47HMDI/Zoeorbnl97lIwztDpFQqcu7z89VZliGCaI7PUTHycQgMLRZ8nPJxdegh6FuCIOTUlo9O19wf+KtQ4ULeX3PU7i1q/Ww+yzH5a16/Hz2vvC/bvC9HFh0JLhaT67nU9G0GfSIInqtNoJDjb7bd4xj6AUEECRLkGcpMBBFHHqLxdSSOVsg3MR6pjDW1XUN7K1w5pO3Tqbv3Ok+tlCu41suXaItXPBSZzRV5HNlbHBvfDenLyDAUlO35dv1U1nKfYU7Bj6y8zFzp86PxMUY2//XzyL4ax6YyF/Rle9UyFAuuNegYj+eVigEZub4BHD68BwCQG3k0o73t2nv36gEAQKXomn7YIyoTUMw4sup2fU53co4MqhXGznjuiJpvwA/avKejVjLnxjBbm58Ntv0eDIXm8rTPV/z6Srzv85JSgVGhLsc+ol+ND0GNPrAhw8ejvmx8n5tBikeUlf9L/rEYecsvRD8ab96AMFA6vMDnLcftcvod8TEUMjl9fsPHRvQyTAUidto+n8U8ESh/IVFMQQC/z/J689eallgCgggSJMhUmYkgMlMQRMJlmPIa8yAS34V4C/JF9JqulUYdf83Qls7T11AoMV7O7UWAio9IO02v0SSKobLLjVIrfJfqlZ5zRVBGJK5EtC9HN8P5QDm81xHEaOR+gojxeGR9XkZDvzB5/UfUQkIUlXJikO7d5WSeXn3TPyh7hGDfAUYKqGXqDXFHSCrLeH2XXt/PNSR5qVpRdMrHNORJs+SmZMkVGHZTMEaks/5w7NXoayDFBoO2f55bna3Rnm0pM8rSbMon4p/Xiv45v0ap7+Nd6yui4NfaS1U3UaQgisZ/NwWqbfGJOopGcLsCIyZF+sKyfIiFslX4jY8nTp6+DABobDj6vmV/4vPZIncl3/HXVfoRc4wKRoTNKimczcxGbAFBBAkSZKrMRhCxvT/BtuIKlzUxxsQbGGdUphFEzMKkT6DXdHvYqK4VpSiW3OcQcTUeknKcz8lDTDuOa1uOkQetjPEQqQoyhWSFVJQCXLEjaV+NjVohNitvApTotJw2OwJZnvF1aWziHvAe0EbuMYIw7LXjY5UZ0z9GjsT2Vfd+d9tOf86XPHqRrfqcW98/zxKtbG/5uaqRa7ZIXAyyO0dkeyrOn6ExbN0Uu7DhWq7fFiNR94XIj578DNmZXbJb5yWiOdeb4gm4KJohynKJz8KZdSJfUpb7o+QnJHeXqPp6fnQH5VsQ6hjq58ftG22yOTPyg4g27ft1SEc/f9ERYafp9/rg3sPxGMpEc3r+szn68ngd/QneUC47m7kaEESQIEGmymwEQU2bjeTxpM05wZyM+vK086BVD+aPIQj5JZQzIa0j+6vk+yhRSBsW6YvIMbtkwFhwt03fBceiBKxY6eswqRUykxcTcDyPQ7kY8gbn5fcYpmzpOUmWbMUh49M92fMmtEMtzblVbsqIBmo/Zf+XmXyV475rA9c8FaK1PP0+W9ynTG+7UFmOTMvVmkdOaiX/vNlj0lHk90D+hHbDfRWNzWY8hjwTufaseJJTbcGP1SHLcHtzHUBij9cb9VnT86zLbl7Tesa1c2+gyAARKlF0RA5DN2ZJcs6yiY7NZcVE9fdGH5dYjbG25j1udPyaOx3fbkg0pTBaT9E1cka26/RhdPzzbtmPs7CQoOQy/9WwSkWNn4xW+lD0+xhdByUHBBEkSJCpMhNB5PLSytfOxVCUY2ftKgCgQHu/ssDCxKkcDqGNEVdo2f9xFiK/V3r20m7XOGI5CljIds0qI1MsNeVDK62c5x4beWY8JgxqB7Ez9bHGMhhO+F7mIF0y3BZrntUplNPvu9beadH+pK2uESr1vlhajI9VoL3f7blGL1WchVkqueYRY7TMaMYyORb5omuyhVudR9Hb8TnuCNWIFUkkMeQo+h2lHycoRryVym5/Nvbuc1bmxo5HShrb9LnwBrf7SWbiPGSBP4GlPBEEEWqrzXnns8HHFuW8X0eXMKFSSBBqRVqa93BLeRGMZnTIi9jc9vlZJyIQmVHZmqBvp9P2Mem30mn7/IpfpMiVGM8AUCgPx66jz5yL0VA5GOO/5WE2IIggQYI8QwkLRJAgQabK03NSToRC4vekR/fbHj5bWFSVoPHCMQCQoWNwREeM0WzJyYQgztJ2CnsqhTaKKaMuecKqLolWYMKQ9k/4K+nwp4pmxIPyLZRcNuF8xTNuLvXM5fGT3nRsecFDjyMVESGUHdGZJmp5R9WfCCWrxUp8LKWti6BTWXQS1s6Omyl7ltxBt2/R94noOM5yDvt0jl1u+vZLZTcdF6p0NHIi169cBAAs0gFZLCVkrStnvVhNIe/7Lu7e6+fivRgx7laikzmXmflIPuuSoTOyllNiol97p+P3vsE5iFh4qJx3p2uWjt+0iSFnYofYn4fGgMVoNmhSXFyjk5fh4CLN4xGPKUd9L83CQuJMz5P0JPPh8XMb8Ta3HmTSm55xUfFNZolSxkkMu04js4AgggQJMlVmLteiHue4umsF0/tey7V3iStgedG1RBx6TB1LaGQo5MClSWFIhecKVZJyJupcir6tPGcj3Tcnei+dMvI3CsQkaAZQ+pMowjGCyE0gCCGkm1DQt9ERhdqRQpfJN6tMyS5zfnYapId3FZokrdcSrVPnsdbWPVmr2/c5E+GmXPbrXmaRmjY11g5p3Ns9Jzn1qEc2mkp28zmtlB2RFAvucMyxcMzCUhJ2azFBbHvHnZHnz3nnvwMHndyzsssdof2un2t1YXnW9DzrMqJ2L4uoRUci/YHYZJ3JJPHK5yxP52u6llKHIcRmf7y2aa+vmpSsu7nj905OcKWQ5/LjTvSY7KTCTbnx5zHi/g8/kfTBHo78OTm823+LK2R6Dens3hHVPSOa9+zSnwFBBAkSZKrMRBCdbdcMxUVScuNQJYtSsGRblUSafIWaI1JBzGR1iqnPcVm38SSsPrVagSSrpLAM96c/IBL922Sbc7Wm/RwXsdXSnlp0lS4rYKCSeKL5Chlpg7mVsk7Jyi4P7xZJk97acvu/T+r1YtF9E6s517ytLovCZllaLEpsyhHvQ0RiztYWacIEUNsNhZpV8MXn8OqOh0XbIyctDZioVJRPo+zvK0UW8yEBbXubSWHVJNS6sttDq11W4e60/Nji0B289RYf21VHFhVxm+ckA9rgKk3Y4zMitnibxXY0CtB2nQAAIABJREFUl3IL5FQqIPUL6tHp0FcoMRr3JbQIS6TNh4yd9iYetKwQbYwgVCSJRWCKKl5DH2Cq6MuZi0ysY9r83fuZ/MffbLMo35/fs8p1/GwBQQQJEmSqzC4YQxJSl/ZjYYElxpoetTD6AyqrTuqRJkakvhnpdG9qs9y41h4NVSKrO7ZdTJXWWGyiTL5IT/pcXnHVYuPKKIQCAIOByrdzFeVYsjG1uDA27nTJvHnJIu33HDXa9o6/tjokL3VEtVb0YlxL1XcSqrLmVO0Jdq34fWow3X2TBKh8zt/3uX2r41podY/7AzIQndvHVq65J79EtLdrjxeg6VXpk8onKKBHsk8u6+OrkkQXE6LIqcqSNmz5+Za970fj91rJZKrSEgcShFhLJNERYnT7qXQCoWKiELGa6y2WQVRiIhPBlDaQKPHx6F9cppHvdVz9BrK5cfIhADToJ9qoEy1y3wWWAVgl6tMzX87M9rMFBBEkSJCpMhNB1Oh7qG95IdQWKaJFchkqNf++suB25oCaSmniGUsbV0xPptZWkpG03Ihp3QlSIAcjLgRzbY/ASNRq+TaEXmj/qZycb0QbTpRq2nY5Fkwt8HUyWjNPMVJ4u111tVIx1QrH5GPsdFUWzjWFkoc2N1JNVJhiXa74BR866IhAja/W1z1+3l+kt53dr/osoDPsurZf3uX3eadBRKny7lRcfdJ4VaC2UF6Kx9Buu19imw2RFlZ8DPLIbzYYw+e5tzvzjRwp/TlHTVpklCg7EjWZz4wicwLJfL76qWJKpjRvHrMnPwYnSpEDaX4905m44JLKEPh+cSKVyeeg5zca22/sesgH2mZjpR12+Nm11+/JHpVupC+rmJuNEQKCCBIkyFSZnaxFLc9AQVz8Y2nJNUuBnlI14chNaOC01o+LyORl97N9G1l+g8HEKiufhRCCIgsTSELvR6YIxfj3SoMGUo1xJo6REadCLeVUfPQmlJ7LUgsrWLNn196x70c02nM59nWssnDJDll87aTgyrDPdG+2u8uSp9Bjv0yVQD/IBKpWl+iLbML6lnw2jlaaVIkFGuZCNRlGSQYqrmqJH8FUqYSIMIoL3dDnwELCTY673Ut6qc5DZL33ec9LfI5Uok3jjhEEv1eEASkEkTSnYRo3+Q+DOEFxvCy+igHZBBKQ20zsWHF/xEtKHnH+NlK+PvncmqxzvwOWKqx51CtDCNTqibOBmRIQRJAgQabKTAQRF4qlb0GaROpNpca6TPOtqqR2TTboUxFERqxMdfnOqegFy9vHK/WEXRazGnXMcfWeecp24+dN/x8z00bjx4rTvNlRPLoJBWMUAVL38yontU1W5A7bBBTVxXnR502svLTbRCnUS2y2o3wC1SURjwGROok7MmyVfYOrV91/UK8zjVvPA7VQu6XcDfDc9EUgmbcCC5YsscmObt9O01FMRjBNuTrXyQ14tmWU9G4EkJSkz+b0Ot7EJt5+JLZkciylvQ/oRGgx0jEYXhvdTpZmTBpRC3mAY5BvQoVyieDY6jATpRGEv7Zafs+eeNIL8tSW9vBczOXpe6Sq0wtRjCBBgjxDmd16j9zwAcu+lcURp12ptm99cvfFlygxqy9d7i1eJXMqpjleKi7Dwqlxl5DYw3ttZPB00yTG8kEUGVHCRnbCmyy/BzkHKos/T2mTaZjLTxSGZZRiGHu4GZen5l1iHsze3UnU5tJlFgamjavKgFne9k7HIwhnzngGaYX32VgKfWXFkUdbxUfoLyiw0c4Oi72UyA2o1VR4OJn1CosIFWlv7zDzt8V8hEqF8Xj2HKwtzrd5r038p8hBj3Z/ic+MOgL240dDEYZ0WUWXNn0P4j/IB6ENMjGreJyrk6BdMXn9fYms2lrZf3dbzG8RYTONYvS7GHJ8Z895Ho6KQe/b46+HVvw1kwsIIkiQIM9QZiKIhSX6EogE8rR1smRY5rmsKt4+YqmxEVVVLsWosxgZyN4f55nHDEhtz9dJBDGJJK4nygHxMUwyJMfXR/kmFPnI3IRszlJe5fz9tUFWoyakWnNbXo1gI1MdBd9+sZZkUjbb0mSutWvMlKwuM+uWjMpWh1p9298vrbi9unevMy9bLG9XZe5NloMZksNSyAkF+P3Op9iQytJt9f3YbfI3+gPG6QfKEOUzlk+awMxDYr8JNa64DH3SIPPiCUzkIY0GagCUwqh81uq067eVezHUs+ubTXrTJtsu6LeS57FrFb/HB3e51t/DeX74Qn1sPz/HRHsJRitKHNvR3X5PDzFHZnOnce2J0SXN/DZIkCB/p2V2PYiSa5pilrkKQ1WhYSUclaLnkli/4vZOv+1ar1BdSI4VdyAdXy1VKjzqqvLNOBstWcOubacliGI8Ky2aiEyk/9ersuqUKYqYfUmtNpq/D0INceQvyRAhFOnvKbEGQzwNHKPyKNIh9UqVZerZ1k4uoVKVvIiCH6vX8OtuMBp16aIXIa7SH1BmdS8p04gIspcZb/qr3INcqplMi0zP7bjdIVEoNZoQxdW++0OazfnyINSkN2LlLlVeGo5U/FiFm4ma6E/oxtW3kudL83GZDYq7cUNg1X3Qsz/hv4gmXilChWq0q9/bfhYAPrPuiG6zkWRz5nmTV5fcD3j82BEAwOF9fi+zzH06e8FrSDSaSYuCa0lAEEGCBJkqMxGESIhDNWDl5xERxYBaoMioxc66a57Wlnu3y4tJdaBM3vfps12YxRx38SLGm/PETXrirM6JaMZEtmey1o13zknX05ysRREN5ZKmvUauQb/jK3P+JkQx6g0/d7ni9mY+w0xKeb5NUSCyPgdCB6yMVEn8JhHzBgolxs1FiRXq6KmGBD/nMdcueux8zw7HQptXERbtr5qLLdZ6aPV8bLuWEz9Il+zMHrMexScQYzTLik1d5syoBufcRMqbfIwRkZtaOBhflSKyxUpfW+TIFFMFITaJ1DYZoYkBaBRDL3+ZbESNid9XNI409Mg3mclbq7GWQ8nndm0zQc8H6Dd62b3HAQB79/hv8+qmo/vHTnheVZFM1l6qlsS1JCCIIEGCTJXZVa1H8jnQ/sKEG1ZML66yharbtBHb4m1dTGrlqZYEqCkK1FZZRjpKWTHVVM9hPPdi0vM7KdOiG5lUFCPm02dk53P8fB0yDyA3kfU5TxkyM7KQIyeBlaL6nBfVslAdQ81TnggtPegyIxpVIoU2GybXt9373SFiUmUiNTpWnoRybNpCCKwDEj8GrOjc5XPSYaRiNEg845r9PrVfP1INCuXYyPZng51K4reah2QqrLhE/0ufz0iJbQuHRBRNRuha8lvx+a13E5R5/gr5QEJmowmY+5S8onHUO/kM6x6rYtTmDu8X/R9l5j696qW3xPvc98I7AAA1NlhuNBzNX1lz5LC17ffm8GGv4ZHLhIpSQYIEeYYyO4qRGfeuxjwCfq+8dNDTW1t2+6czugwASBqCAWDtRLWDi2hb59i0N8MeGx3G/W2krDlx3mdjiKfUi9DQUho1Q/Zm4qWYQCmqOGVPcXDMTSLOmmogZohmxGoV37/ZUVSACILz2U+1C1yuCYX43NdZM7LFJrvqq5DlPcmzylC15vdkyBoNXfpkItrQo1jz0Ramv6RDQ31zYzO5II6n02UVsgxt97x4Ba4la0t+jHJpvjyIPiMPLUZm1D6wzPlXm0L5qcrMRi5yDtY2ErS0VWcrwrha9UTkLWZSTrKDJ3gSfO30VG2Nbf5YPb5Q8Ptz9x2HAAC3Hj0cj6HFaNYTp0/7GJhXdGnNUaOyOO/be2BsjNMkIIggQYJMldklk2KtTAQRd7eiT0K1G7hdgRmZwx7Zfr1kdVInrDiBUrHdmtucPbL0MkQQ0ixJjPjaPoZpdSJSW8T/5WLWJiMlPMUoDuQPxo6JiY5i85CrG56t2R+pVwKbGXOecpzHAednOHQNMWA/jFHqlnb7QmWMEDAOXyKvoUEkUWZdCGW35lW1il3Looi1JquqVcEcHG1P/9GA9ux2M6mL2WCF7MGQVbmXi9zHj7FDm3h52atWpVmB8xDZ96q1UOJzOSRvQ0zLUl5dynwOu3yu6u2t+FgdaudhzPURF8e/n3xGY3bw5Pv4A9/+2H7X9nccO+jHV70SIuKHTpyIj3lG/IaGj188jy129SqxhkuJtV727l65xqwkEhBEkCBBpsrsqtbKV58wxg3jn8cVmFTDgFWuB42EpaXKuxlqQPXQyDCnPVJHqaFy4ZWVODsfQpyGpyCMOMc+zWUg8lFtv9x4J62RqICqKpydDbBuhLSZuwBq4Xze56FUUE1K2sQj1eRg3ojJu56s+VuscC2iaJkszAx9B2sbzncYxNfNvqnq76DekeRBLK94z45c0f0EG1fZeYvZr23WQWzWk/ve4v+KxmQzZOVSszXrjbFj9MupGqJzkALvtSIxC2W/tibrJeTiXBfydVhpqkP/Tb2d1L4Q+hHPYXHB79neVeciqKbHpbUdbscdGaHKmvJpfL9bb3Efw/1f/TIAwJB+nMsbjloeefIUAODCWuLz2dyWb8pf9qw6F+nYIa9Mdv9r/Fjf/E1vBgBUrlNFPCCIIEGCTJWnhSCUFzBp78fO/glOeYY2bGk56fIsb3yO2noYO3Yn7LK461BfH/Ac42tZ0jhLDLj4G34xnvMBAH1mmeqc6hchrdxXFEOZpvn5dnkCgAV12ja3+zOsIzlgfkOrLW6Cj7XJCIPcJYtLqTlX7gmvQ5mIUU9sTfbkbCrrkLU56VNQN6w6keDSbnb9IurrU+urelGfWYzyugNAjzUk8kV18SKTUixMHnv9iqOZymLikZ+HFDm/Xdb4FHBVDsyQD2pXrGFGeq5ccmZis50gHvl4FhnVedWLnZNw/JDnQTxx+iwAoM6+qmubDR7T5/3wXvcHHDng2+eo3f/moYf9XGQhKzv38hVGpVIoRuM9fszn8c33vxIAsKvq57j7tqMAgDI7vCtqOE0CgggSJMhUCQtEkCBBpsrTC3PGBI9JqqhNvKqEG51+Y8sPTQsdOn5VcdqcTuLnGCppS4U54prqY0NLGu1OpIHztd9LysCLWpyUmKOJQQg+6KooL89xE8KcfYYtVRAmQwJUlzn1TULCLqG76NAjEs20HwDkCyKG+fW0ejKhfJtq1U2MnS13ZhpLpKvdwWWWEKy33am2d8cdXXkl69GU6Muk4H0vlJNkLdRFlRd9ncQ3Pgg9koHOnvPmvdXV2vTJuQGiIrkFPgNDzlWx5NeoZ77CknhtOoXPXdnifgkd8EV33gYAeONr7gMAHD/gDsLzF9y0WF/3bfex3FuLpf7vOuJhzDtv8/YDbT4DT5z1kOUmHbly4MusKfA3c+ctiVn2Zjo03/jalwIAyqSunz1/EUDSdPjhx74EALhECvZL/uG15ycgiCBBgkyV2VTrOJwp0XtpjHEiVaK9qfXTnJe4rPkkXZuNdOJko/EmvD0WHCnk1chkMjVWZ54oaqvW6CknTFwijau+Gghn+D7xc46fY55y+byv6MvUMiLm5JgGnWe4s8hwHPN1MBq547HXT1rviXYt51k+bilIYhOdYEssHFsjGUvzUm+5BlzNudNMRW1VRm6h6uG7TJ5NWlp+r/KlxFFaZdnCEWnbzY6K9PgxltgYSOHqZitBfPOQKjWsnulIbSMV4mZLyHzR579OJ3GVyVz3HL89PtZrXuj/H1z1e9TY9vIHf/2FR/1QpKq/+oWONL7uq1zbL5X92r903hHDFaKTZkuhf/8ttIkc8gwV337EEce3vuk18Rhedd/dAIBNllx49OQZAAlyuNx2tPgXn/wcAOAiiXn/Ysr8BAQRJEiQqTI73TvW5uNJJzE1OfYfTLa0u4buVfELlRef4DdlafcKQehIvZZrxEJZiUcTYdHxWrgYETl0WUqrlyLtyDxX8RrwVYQpIQcllKX6ss9Ntkh6UcHcvSUnnckGzhV8HlQ4ZtglsYzIodFIaM5R5NdeZGuBoRKOWAqQHeFRq/l1l0o+dw0irVtu9zTiAv0gW2yk02fi3cKS+wuKFdrhonSnrqfCNP9oyJAqtWCR0Of43fdosACA9avr06bmhsgKkVq94/MmshLY2qHNAj7dTf++ysbEb/7qVwMA7rzlYHKwvvsKLjAE+tCZCwCA0ySUvZhU6Ve9+AUAgCubjhQefuJJAMA6aek7DGd2iNS6vMdlloC8jT6LY4e8uPAiiVgAcIGI4IsnTvkHROJltk/4As/10BPnACTJW9MkIIggQYJMlafVem8yapGJS9hP7jEezUhL9JT/ZOePl4ZTQ2Clh0dq70dEUGZRmon+Ohgw6aZVZzEU0owxSFbIHJvMqj9djlo5mxsnTOWonbuDhIAyL6lTq4yYplsiKWmB5ftyZTXgZUl70pt7atpryZiXl913UC0zMWrLtbNQWIWFcKsL9Dk03GYecJ5qy+4d32Sq8E7dNZtsYJW3LzBasrHDaAkbKQFAhhCvzHPp2VB0Ro1+FYWZbE93o2WLbQVbLAhTyFLjMqqxUPVn5LZjXsLtlfe9HEBC9Dpz8on4WE+ed8Tw6HlHEA886ASng6Q7713x+7HOhkOPnDwFAHj8jJdH6A2V2k8fw2H6gJYdtdxxqyMHlQA8d8n9VZ957Ml4DCX+jg7tclQxIKp79JSnf5+86PuosFC5MDuQGRBEkCBBpspsqvVkZGCyeU2cFS0bni/XSL2eTOh6yvdK9Fr0la+x7mXQ84x+9OjRlVtANFR5eJtEDvKwy4ZPx6kH/CzLIjXFhfGYe07oRaXnbsL6GXMx6JMZNVlCfUcNZ4gGaMv3SLOtsIR9tZJcryJEG5uusYSg8nkWhFFRHtKNI/h1VxmdyHHu2n2f2/Kiz/Vi2ee+VvPXgSJTvDn9Xgp5sSxhnshBVHulWV9hoeMco1TZ3Hzp7ff/va8GAFxl4tr6uqMARTGO7PVIwR7SzC+ceQwA8DdfcB5BvZFCS3yGL3K+20R1PUaRTlxwZNE66fucI728S8r63Xc4SnnB7ccAAPuJPPJsgNzt+/25sub7jegraqd67x2/y/etMGr0xYvOL9kgl6JPKvbyAttpFsfKOj1FAoIIEiTIVLkOgnAZqWnNZNr3pK9hgpOQRhL2lIKd44xIbVlgrD3HWLqapCpJpifPLkuwx0zJONbPOLZKtKX8CGrzpzL9Opd8LXrfH6g83vyTtVR6LyJDssNGMn1xDLKOciKyF3PG4r+y3YfJPdmkP6M3cE/8nl2ukTI594ZvsRR6iXZoxKa9GfoYhM6WFgscix8vYup2byjNRu3PUyvNGQC6tO0jpjQP1QhYkTEipTyRQ78333TvnQ23/w9xbu456tGJRlsFfj0q8GcPfBQA8MUn3JaXz6KUYo0usCHReTaQWmEZvT65FF980hmVi0SuRw97lOj4IUcp97/G/Rv793l04hJ9Gh//3GcBAE+ccTTQIcrM8fd37+23xmPoNny8D514jJ+oSJCPZbHq+yzxHvX7sxsVBQQRJEiQqfK0ohhxBnVcxnsi/0HRjoy4Dtdujgtcyz8xgUrILy/TF9Hcpld/pHZ/LP3FY+dZVDQaiuPPHIxoXEMBQJkrd5ZIIaMiIBx3nqXYhlyhpc3nKVWOVzkWWyxRX6UNeWiVKdfUCNvbjqTaTZaqS2ngHJWb5rzecbSVK/qxVSrOGMXJkJuQYTm4DtOKOyyd3uX7YcxZ8TH0BtrPj7O4nPh2Wizk2qaPqEevesToRoF+kSF9KfXtpITbPORP/r+PAUhS/eVHuZvFWhpEbo+d9VyG/ft8/o8edk7D42eT1g6nmE/S5TyrTKDK4Cug9qqXea7Ga196LwBglS0S5cu7etX9MucvOFfh8ZOOWr502o+/zNJ/u8lDuXQlGcOZNfrueE/3L7FUIX+jXUbHzl5iEWEb9zNOSkAQQYIEmSo2reFMkCBBggQEESRIkKkSFoggQYJMlbBABAkSZKqEBSJIkCBTJSwQQYIEmSphgQgSJMhUCQtEkCBBpkpYIIIECTJVwgIRJEiQqRIWiCBBgkyVsEAECRJkqtzwBcLM7jKzB82sbmY/eKPPF+TpiZl9n5ldNrOGme0ys8jMjt/scT3fxcxOmdmbbtCxj/E+zS4k+SzKPE70QwA+HEXRS+dwLgCAmb0TwLkoin5sXud8PomZ5QH8ewCviaLoc/zs5g4qyHNS5mFiHAXwxS93p3mukn8HZR+AEp7BfbkREu71c1du6AJhZh8C8LUAfolQ9iVm9ptmtmZmp83sx4x168zs283sY2b2c2a2AeCt/Pw7zewRM9s0sw+Y2VF+btz2ipltm9nnzeyFZvbPAfwTAD/Ec77vRl7jPMTMftjMztNMe9TM3mhm7zSzt6W2eb2ZnUu9P2Vm/5rzsm1m7zGzkpndCeBRbrbFezR5vqUZ9+m0mb2c/38bIe8L+P67zeyP+H/GzH7EzJ4ws6tm9ntmtsrvBJW/y8zOAHjKGJ7n8koze5jP7DvMrAQAZvY9ZnbCzDbM7L1mdpCf/4SZ/SL/z5tZ08x+mu/LZtYxs5WbciVRFN3QPwAfBvDd/P83AfwxgBqAYwAeA/Bd/O7b4TWQfwBu+pQBfAuAEwDu4Wc/BuDj3P7NAD4NYBleluoeAAf43TsBvO1GX9s8/gDcBeAsgIN8fwzA7ZPXCOD1cLNK708B+CSAgwBWATwC4HtTx4gA5FLbRwCOP4379JsA/hX//88AngDwfanv/iX/fwuAvwZwGEARwH8C8LsT5/9NAFUA5Zs9z8/i/ToF4CEARzjvHwPwNgBvALAO4GWcj18E8Jfc5w0AvsD//x7n9BOp7z437b7d8OuZw4R9GMB3A8gC6AJ4Qeq7/w3un9ACcWZi3/frweT7DIAW3Gx5Ax/c1wDITOz3lbRAHAdwBcCbAOSnXeOUBeLbUu9/GsCvTHvQtEA8jfv0XQDey/8f4b19N9+fBvCy1HdvTB3jAIA+fKHX+W+72fN7A+7XKXAh5vtv4A/+1wH8dOrzBc7HMbgy7ADYBeBHAPwogHPc5icA/Idp9+1G/80zzLkbQAH+EElOAziUen92Yp+jAH7BzLbMbAvABhwtHIqi6EMAfgnALwO4bGb/2cwW8RUmURSdgGvjtwK4YmbvFjR9GnIp9X8L/sBdT653nx4A8Doz2w9fTN4D4LVmdgzAEoDPcrujAP4wde8eATCE+z8kk/f7K0XS13UajuIOIjWnURQ1AFyFP8ttAJ8CcD+Ar4HP8ccBvJafPTCfYT9V5rlArMNXzKOpz27B/9/et8VKkmVX7Xhm5PO+763qru7q7uluT8/I9oAZ2zIWgkEIGSSQQEh8WeIHkMAIkPgAiS8YBIgf+EJC8AMaWUgIxAeWRsLyjCzbAoM9Hns80+Oeald3V9frvvLmM558rLUiIrM7s0atrhwMZ0ulrLwZERlxIvLsdfZee22zD1rv1/Xv3jOzv1ZV1X7rX7eqql8xM6uq6l9VVfVjZvZ5M3vTzP7ehuP8gbaqqr5SVdVPG8auMrN/ZmZTM+u1Nrv1KX3d1vvECWtmZn/LAJFvDBPRXzWzX66quuPxe2b2M2v3Lqmqatv9/n/FXmr9/2Uze8B/9Zh6ntc3IAaNx9cMqPgPmdn/5Ps/bWY/bmZff/6n/PG2swmiQmOE/2hmX/Y8b8hg4981s/+wZbd/bWZ/3/O8z5vVwbO/xP9/0fO8n2DKbmqAaGrE+cjMXntOl7JT88Aj+ZLneR3DNc4N1/mbZvZnPM87pDf/25/G932f9+lrZvY3rfFsv7T23gz37sutoPKJ53l//tM4xz8A9jc8z7vDoOw/MKCsr5jZX/E87wu8l//EEGd4l/t8zcx+1sy+VVVVas3S/F5VVU92fQGyXTMpf87wY/6emf2yYdD+3aaNq6r6zwZv+fOe540NwZ+f4ccjM/s3ZnZpgG7nZvYv+Nm/NbPPEd7+l+dwHbu0jpn9U4Nnf2hmp4aH7t+b2TcMa96vGh7CT8uedZ++Zghgfn3DezOzf2lm/9XMvup53o0hYPkTn+I5/t9sXzHck+/x3z+uquq/m9k/NLP/ZGYfGgLNf7m1z68YYhEaw28ZHMIPDD2YOVVrZ86cbTFXi+HMmbON5iYIZ86cbTQ3QThz5myjuQnCmTNnG21rkcxf/0d/sTIzK0tkDwO2hx8MDle2S5doCFrM8fl4jGav88ms2WaBBqZhqCa7aJbqRWycy4awSQcNSQfDY2zfwXf7Ieaywltwe+zf66GBrJVoAjsbs1HqAo1Qj06blPThLXB90hwNb8eXaK8+PUfD2DhBE9Xr2btmZrY3Ah/pyz/3z3dW6njn7/xCZdY0VY0jXHfCJsURxyHwObczxpyzsfK8FXTOeN88UhOCylt5XfAQft2EGa8q7PR5rzo+XlPeswmbGk9yHDecs2Fyhtc0FxXCzLiPz861HpvIejzfSo2f1Ria5z//+Z/dyZj/9J/8E+pEbWZNw+rAD9qnYz7H2+PgqNFud9ipj1UZGxBf4vn3C/5ujE2gua+OWfD+5GzUXJarCQO903eGIZ6B4WhfJ2VmZsvlst7H57Y6X7FS6u8s1LiZP30+Z1/9hV/62PF2CMKZM2cbbXuZbYUZq6THqMp05WPNoh5erKywXZZiJi3KxpN0koTHwGfpEtt6nLqDUHNVxGPSyy2wfaeL/UsPs+V8ARTgefiOTjzie556PUu3PKq8mFfwPNECfb5Ea/qERO1yiu8sbfLRMXnOlq95ca+g9yAFzPf1AbeT66MnCGqumFmubYvVQdHY5JW8JFEcvX1oGNOywuMxpucSMNA4djIgxIjb61zKqnFG8opCMX6NcOS48Vry/KsdkysDP+T31gPO01v1xL63iiCswvuybPvYiNsCDev3UvoaH9kKaLFA//FWUZTutb4y9DSW+j3y91c191zDqyqKqlwFBhVRS6Fno7St5hCEM2fONtpWBFHQ8xQ5pqXFgh4jgveO4658ed7PAAAgAElEQVSZmWUZpqGcM2ZBlOC3pp/hAGUDWZryWPDOVT1bYuMwxKzoB5yxuX8nQqxhmSMGkRc4l/l8jv0CrAXrmZMeK4qac9B6fpHyu6Iuj82ZvpCX0OvGoXluVnBGryf42uuvetba6cjj8nqL1mal1sA5kR8/ywweTt698uiRuIYOeKyK43Edhjw2th8VQFy3qwszM4u5DH/i4T+Py2ZdnlZa6655LB6L4LSOQeyatucJ/jYr/g1briIfIR3fb864qPgclQt+phjEKiqp99VX+fpmIQY+wx9BiXjNGLPIOXhCdGZmnmIlRO9CODVq5KaBYkHB9kWEQxDOnDnbaFunjzzHer+gJ/KpDFZypvQCuOeEr/lCyIHzTtDMxnGXrlxrZcYc0ozxCmU1Ahy7mwBx+Bmj9pzofXkgopsswyy6WGC/FJO3LRc452V2UZ/DIgViuLx5amZm09ljnje23dt7kdcJD+vb3oaReX4WKLOgdWc9lBw3xlx8b3UdW/HvIaPUZmZVQeTAdadCA4mPsR7Sm8y5fab9DIMdcqwj7j9MseVbBcbvR2MIWOUDxId+c3pkZmbXZVKfQ85nw3heFZ+lspA3XYuTlLvFEIqbNQiN6Ekb1MhMMQm8RkRVimeZNddWxxJ4bXnlt99aVa2ilEKIVb8XIQh+7jOjoh9BSbSshyOMGz9fxwW5T6jx9/R3oRS8D/ztGMEhCGfOnG20rQhiNLhtZmYZ3fLNFHwBK7hbroQ55rqCC5w4wgxHR4z/a6nHdZk4B1kOvzWdYF2rWbPTRa7X544pvypgjKLjAw0UBc6tUMS4oPfnDJqVDYKQl+2E2Celi5zx+lLGR2JmRJJ49wgiCrg+5UiEHNuQ0KJ+X0/tikFgXJMaB5h1mH/PA8ZjPCKA3rmZmfWJtt6tcL3vUW+nIAKIeX+P8rGZmX1+Ab2TPx5A0vILPfBI7g+hAfO4Aofle2XzWM0zejDGsUoFSQR0vPW1/44RBNfx5RphQHwIPUf+muetON7LVmKvIlKIGS8z3stcMYfauyu+xu19cSs4Vv5qLCwkWgnq15jH4Xv/owjC95lRCZVZ4XeLJyFI/gxzCMKZM2cbbSuC6CUQ0s0CuJrplJkHrhdLxgdKRmmzJdFBjFmr12/Wogoe5Dn2GQzhbTSL3twgM7JkLKFQ/IMTu0fEkHQY9+hAPa3iLJpzve1rvRcihjGfjetTePCAaIKsy8jnMTSZcg046AG9dJIWBNqRJWQvBpzplYXR3+OIvAhfa3iYZvqqlZcveWEhkdPrHSCHv3DyTWx7hc9/NX/TzMxmHq77osB96y6x32kGZuBP5d81M7Mv+t8wM7M74SMcv4vP75anZmb2e9VpfQ7jqbIYzL97awhhLUvzjLT8p24+vbwRMRR84Hw+C1HMMaQnlhevs21h84yE9NpBD/cgJ1oquW0YieMj/hDjazx2SOQdMi0URfHKdwXMOMj7CzmELR5ERlYlqUhWMp7UcFRWs4bPStU5BOHMmbONthVBaNb0A/HTWQ+hSLRyqSY2GszjrNyaXC1NV2doU6Q8greKIkx5yulq+yxlDUdHmQWcU0wOg09uQ8b1cjFnNN8n76LFLHxwH1H36Rhr8f2DEa+Trx3sGyeKIjcc911ZTM8Vqw6CEWq9Rrze9eBzoXvjNx6hZIzlxAe6+rOnuP4v9n/LzMxmc4zllEjxIkDdytu87D7jRa8Y4kMvp4g5dHO8pin+fquDe3m3RIzqTtaM23spPivoi+pXerK8Rg7M2++YezI6OuP54L2SGXUWgF6/9t707gd7iDOMVAtkZp1Oj/vgmS74jFdEFh6RtQkR+IoHMDvorcYYunzm4wjbRzWKDFfed6zJXE3HQMzv3INu7uUlkHmNZvSb5YU+i+vjEIQzZ8422lYEURn5DsybJ5zRxJSs+Pe4g1m1z9qNgvGAedrMbFbgM0V/xSYT2gi4ti7riC/jG5XyzVrLMhdcKPYAb7VgbcfVBRiWIWMeyaBh9fUTVKHmiw/NzCyK8Z3zCWbd5XLC64VXyKqbLaPzfKzDyxRtJIlXYw9BqPFRVJ2xCMYe8ia9YTmzCXc7GJM/cgTk8OH/vmdmZiGLbe++hGzVT3ZPzMysz5jFlOewFyMGNbuP8Xl3jPcjIq0f/Sx5JIz/9BfNYxXTW5aqXaiRRLX2qmrd3fqs4RmEpvNyNfqhDIKyAvLunT1wPe6+esfMzI6PmoZXikfkRA7LQtfMIBfRiMJECa/ZIy9imorRiveHA8TIBoniIBjn0RAIpdfB/sOwiev4zAoOekAzv/bruOfXY9xs1ZQISaiidJM5BOHMmbONthVBaDIPueZJupghp1MghOUC3lt5X1VWKmK9UtVHJp3q0StVEtKDBIFq5fGl3R5nW3HHfSAB5XMLzpSpeBBkZC6mfE9mZhk0gZDp/BL7VFiX7e1D7yG75rYZjj0vZzyXXcfUzWKOXYdrYC7v61elrz1lO9YI/UJWZmbLEpmi4T6Yj/4I1/+de4jBHBEh3j2Flzkb4d68nrxiZmbfjrE+vya35MEA33E5w3e8zKGNjxgpv2bWat4UwCiWYusIgmvgQkxF/j3fMYJIBsjc6Plc12AQD8LrghMzuvMZbH8AJDFJmkzd4wvEYAIiamXYbub4nRwe4Hcym+MZHRJR+Lz28VJxAqJEoUXGnSIi96GQGtnGh93mno8YqwpZevvB+8g0zef3cU7KGhGldKNWpvFjzCEIZ86cbTQ3QThz5myjbV9i1OWn+sNqya6KnLJQVOvVwIfXKp1VIVe+XIXtlQpVeCo+g5JKK+UZAmPLOaBenCC96bFgLF3OVr5LtFStj/K84cJKEi0MKT5DiK2UYdzBMZYZIHhVNAHOXVkg8kygJQT+rnOMRLkWUcdfzVP5rSClx5xhzjTbRfi6mZm98ecAcU+mKFY7yBCkTYpfNzOzW30QovYHXzAzs9/Of9jMzLK3EJDbewPBzNMzEKSmRxSW4dIiay0TOqquI0W8rF+NrzxHLkWyjeXWz8e8kEtXWyWe1Z+T/rx/gkBudIBl15gB+aD1jJQVns1RF8uPBQsJO1wXJh4+ny8x/pnKB5jG1E+jYErycopnu8N0dTdgIP4S96vPsTuJm5a0HZ7vKZdAZyd4vf8h2rQua244vqP7DDKgQxDOnDnbaNsRhKc0Jemn4VrgRgGPcpU4lacSzmhSKOWcHn1BVFIH2/jKqbsTYpatMpVv3/DvQikI+FQUjMkoeyb6dJriO8dj0Kq9cVO81GXa6OT0FTMz63PmnUbwhCGDQUum6SK/3Rt3NxYROag4SwiiLnfna1gjizWh2ZYP7MaU52Mx1js3f9jMzF6+heBlZ/o/zMxseQ9p36unLFY7QFDz5RGQxKMCXmh5C0LCp/v48n1u9zbH/NGE5LRlgwISBSEZwK68VSQhmruo5V65W5+lQGLJIrdqTbNQBVFJH2OWrUnR9aMW1ZrlAx0G9edTPLuBhHszBidFeOLrgmX0HsWQ9LtS6n46w/4RywwuZgiyZ3OmLosWoa9EMDUSaumKiMjixZp5wPH2tyM2hyCcOXO20bYiiIwzmkphtT6WgKxKjMOE9Og5xV8qysLlzcwWephdaxHNOp9E6ifTe92eRDAkfEuEwLVsynSUBEfyJWbw+VwFZVifzSlpJ7l4nC6+o9/DGnrQxTk99qbcAttePcWMvre/+2KtmNBANFq1CWhiEnqtVl5rj9BCEHEgmjO8yjsTpOimc6T2BiHSnssrpMLe/g14vMNTXPePv4z3RQGE9fvTt8zM7P3ZLTMz27sGgqg8esgF06pV43d6PL+lqMwqMlsrNgsUewh2HIOQOIug2ZoIsAqrlB5Ma9lFoK28LfHPZ/mahYdPL5+uHKsb5XylqC0p6UvGGioWKur3tmSZwJi/pygHujbud3GBZ3w6eVqfQp6DwHV2TFIg42xpju9YpnNrmx/kts0cgnDmzNlG24ogxlciDMFDdLtdvmqtxTV7BoKIGoDkmUQ5W96bM3PUUQmrMglslMIsx3xGhPAQM91iwvVYwrVfTDETkkyUtChLzNqNcKhmxuYcEp53wfObUKRmyUzIdPoEx6R38MKp7driQAhCkmH4ey09J/KaCn0Ug+D+XmvOL3h7tf5/WgI5KNv0RgdIINpDbMFOKLZ6As+1GGD/Jyn2+33vDTMzu55gHI+vkQX5ke7bZmZ2K0A5+WXUxG4eyuv5q8Q4+V0hwTpBVu0WQYjUV0v41QBXVVv05jmex0z6s0xBjL0mS9bv4Vqvr/AcjS+AzBISp/yRRJ4pkkTB5WWqVhASfVZBFY47YzwuJnruMh63pJTB41Zm8PgA9yYheWqxxHcUtRzCahZRhMJN5hCEM2fONtpWBDEhbTmnh605Cj3MiJKuzzg7FZS/r9uVtY6u2TBkQYtk6iUaWpAiXHCKTiekHHM7SeyHanTC/VJmMSqu1/ZHiLgHHrxYp8UkPTwa8HyBHNIb8jUkmceZWjP8fLl7BOHVvABc54Al534dq5EIqd9+qRvTeC0PLPn6nDGCgLf7xjA29wtQzT/3OgqW/uhnGbdhcda0i88vJ4g9LAOsawvyKoo5z/URysgHJxTzCZvWjJ1YZfsq0lo5/foPPlGN/4zioU/bKmUOiDi9tSxGQawzmzEeIw1ebnbTOl+Nt0+vXC3gvTvMICgzd0PEIJ6EZBcleShJBDVNksziTG0JWCSnRjp+izuyoGDM9RXiRinjGklA5N1RCYPaQmzn+jgE4cyZs422XTDGZ7TV1yzJ6ZPJ6zTFbCWvJ3TgL8VyXNTH6lEsw1tTBBHzMWJMYTnDMWt1e86ifs6YA1FMWTemlbCKmuAAJeQUkAmjVis6zuAhhT30nb4vNqa8MyO9u14PW+PJ6qa8RAzipAR14xZtJvl7IYwWF1BQTsKs9Adphet/dw6BmEEfCKHLUIRPxPHwklmPDB/4YkUW4JhMnkLE9uIeYhDHQ8Qo4lHjd2JvVXzXq2XkJTXHDBkRUrt14C5sdo1MjmI7dexB409ehk8BY3+ffBtJuBXN+arNQI+IYZ/Norr02jPyFxYURcoYU1DrhixT/IwZKv5WJLmfkXux0DNfj20jQLtgXOO6wu8oX+A32OPvzFf5PVGOWJybzCEIZ86cbbStCCImqzFkSzWPyCGrJcW0hlckmLMpvfti0pJgH641epHUmGZqimXMJ/Dm+wdYxy7YpDdfkAdBT5orW1G3dsPrkuu5JEZpbdRpRW3Jn++EiPROruA91KYsnSuegevrJMOPHZfnaT65JSPGP8oOWJAHjJsk9OIzw/hUvIXiP3jVxyCIujcN16z02pMFjv3th/D8Tye4z/tkYOYl+CKZageWyFoUWrcTaflkDvoqHW4FnwLe4IjPil8jHdXrlCuvxY5la5s2gxQykqgrgySxxlfZAD6Pc6GxdtZoSW9M7k2PyKHi8yXvnnFMFHtQDC/PVtGjKC4CKTlvZKryFqa2glambkH2Zo9clA7lBo9GlA+QZCGvS+I2m8whCGfOnG20rQhCXli59bwgN2GB/GsjC6fo/4Kfy+O01v8x19CqqSglGIJt5jOK0DAGEZPNly3h5SdjRGWrknEOesOc6zlJhddio1pz+c05qE4hn+E6ZhNKzDGDcnNNPgS9xHB/f9vwPBeTYExBubI7JYQ+/lTxq2Zm5nfg1X8t/CkzM3vocW1cp+0bD6z/SZFMAr4hmwwdB7j+vYgsuyW+8705viOMP8MDseEQkYNPLsvBCRmp3itmZpYOcS5ZG0EwK6VYRE6BVbWs/4EjCDUmltYR/y4EISl7Uw0Qn42MP520xRqtO+wx1dGNlHEjb4GZmkzV0KyhyIkkinw1/iI0yEeh7lOVMwahxrxl2XAxijnGN6KQzfEezuGgh9hcvCaYm7W7PX+MOQThzJmzjbYVQeyNsM7NqI8wmaqCkut8yrlJ/m1JyfnZhNkCvyEhePVaWfoPzPFyDZSQ73ATqrqOJ8hMw4xrK2Ua9oeYETtc74m/Lj6EMe/r5y1eQD1rsg4gVV0DzqGWq2MMZXqze9HaSM1juTT8zBAI4gvz/2ZmZo8evGJmZsd30OzmvEvRVK2hW1kMrfcV11C+vM/6lsMc/IXbEao2K3bzfVq+amZmlyXGOApwfw/YLrEkV2DowQN2+0CaKT1ZFTQoQHoQanuYirlYIwkhCEbkd4wgUnnjtbW4xkqsxojo2Ftgu5zVw0Wr/4DQT1VIJ1Bol1WbuViZvFeUkFNGoViHMZXkGPFWLfwatik/aGUGfR67SxTXY8yhVBbRxKFQa0pXi+HMmbNPaFsRxP4IHuPBI6jR3FCBSDUXiarSUjHrsH2xwOtov2HURQEb3UhLQqKgnKM6lOmeDYBWpHTzwhniADfXiEGoglR5awnpyu9MqR9RZnh94eil+hxqodsSni9gVqNOpfBVjXWqH8D0KT5ASO88ilDnMiO35Dd+B01r/GNcn8oeJP7aFtotyLzzyGuIeZnJEjGkh9/8XTMze/v9r5uZ2d0fQ1bj7IfxWDyeQ0WpaUaLz306nUNWCJ7MwIsI6K1GB2/W53BNJydE5BWrGa9a6LgSgtgtD6KR/PXW/q6qztXzDckvMDIUZ1Ur3sJzFzL1MzybERtQLfi70atXqj8e4061Ehu/mr+RghW9Qhjis1R8VcsIM7MuUX3sidVJMd61KmpPRT6Zq8Vw5szZJ7Tt1ZzXiHKrwrLImUuNWSk2Y60GtSkDzqaDBJ5G/G+zhhUmpp+i1ooWD/pAGC9EiIwf7Q15TPx9yvWcSPxJF65TEuM9IomUGZaYzX0PD+/U5/DoMZSTMsrbP5zDGxesmTdq/6WMvFesZt2l1foO9B7jDJkBO0Jc4OwtnNu7RBgdsu9yrWet8Qg+q3CV4w9XaRE24zg8/hDbHV/A040vcb+vqa1RsjZjb48NW+jp+mNULXbPESc5HeGePYqaqPqY9zkTklCkv9axWK15WK82fN52a0jUW/Mg8HcpRsWEXQn7DsQxnrPgEuP/O48m9bF0DRLF8qlV0uMzOmetxZxePShVYan4kdCxNDPUWEfNe1crfYWi+/ztmJkdkL0Zq1VgfT1qK8G4m5pDB44H4cyZs09oWxHE+Tnq+5fUxlND3C5zrEtWUqbkR4gx1qd+XxI0DK/IW1sDcXZMeKyT4wMeg/p73Ldi3CBi1FVNSNOa/UhOebU62y6mmOEfT99tLoinkBABnan5SQpPmXEG7yfwmPNp4x12ZTy1Wsvh9+ZAQN/s/qSZmUUgPdrFFaPRGeJDZS5diOaWSgG7jo6rzIBaHIM3P2dmZq/2iEb2cP1PJtCJGFHBecJsTkCmXzhFPGjx5AMzMzu/BjJ79QzfdzBsanAGzMtfefCmqttRZF+1JaEi9tX2vPynbZ+9I42M1diHL01HtrurGYgc0wM2dnp68aDe5xEb5c7Z8rAiD6LoEkEQOSxqBMEYEV9DxhJC9l9Udi3ms6+MRJcoesTfztnRXn0O+wOgCZVYiGVZ1HwH6YjgXbh9CnAIwpkzZ5tt6/Qxo+dQkF/xgooxB5+zUoc5124fM+VwxPqBvdFHjrUQe5GzYpd9LoYdRMqvqa+n/ghzci9S1rnPZnh/xYa7Kddv6t9esA9Gr89W7C2V5JDT5t4+lX/JZR+yhuDxFb67J+3N/u6ZlF1pMvJlUSIT9IsXnzczs4QcAy9AjKVfYlznE6oTNc7bRiOgMqkhq72dGHrRPhCU9b5oZmYfPEJ15vIxvuM2EcXxPu5jpPhRTn5IQDXwWxjz4BbubVQ2JxEvlZWh59L1rZYd1NWdYbXbLMbJQPyAVT6NEETkrfI0pNx+mxm+t2416//xYyhInc9YdcmbuGQnZo/P1ZJZjIDXWhaq/WGDYE9K1GRkEsUMiAqOGHO4dQykfjBoxfoYd9LvoVbuUlWqlL2+T6TmEIQzZ8422lYEkdddeOh56PUXKVV26Q0G5CwkHWlWRty/WcPnKTUXxF9gVqLDV6npzIkUljnWc9MxvJXPdbS2y1lRmjMvvd8DKggMxxsTaSzyZqbsUV7qyRMghZwR/5h8jozxjIxk+fFsVQF4FzaIleNm5oFjPyuBBpb04j0RC1hB2A8llviwPtbkKTzakFyQpAdEpAi9KAiZvHuCMTwbgP9QkruyVHSe/JGwj3E8fAOZldtd3Oe8j+civWoeK+lfar0dl1IrJ4RgRiQvV9flu7LHj8DtSYhk1YlNntbEFubzpwrMkufZ9RrEM+Rln1NhPSMayYiipKMqJrIyNjpWtVRcCfsfErkNh9jvbB/n+OItPAujAVB37DfPuCeNSXUuqyt62VC4WkVCtYLWBnMIwpkzZxttO4KgRoP6ZHbIxVeyuBczH6s69Tm8/pKv1sppS9l374AcCc7YRaXshKo9VbqGl5iR2oCfq/tVRnbghAzLrEh4XKzLpteIUSzmDdf8mlHl/hDxCeWPEyKLnGVzGc9/r7/7+bPHNEZWcxfo3fmaV/K01MUgv6QT4JqOqclpZpYuwMIsmWXyC+p0UlErn1LXk5qSEVHZ6OQVMzMrGJuZz1BRm4WsuNUYd5FSuaRuyHyM7c5vmnV5yvH3yS0JpSXBdXhFzxzyeqKy1SVqB/a/vgUOx4AxKyHUug8GFb1ynudclcpzINu81WeCSQyLfXh2ZQrUi1M9NXTNla1yQbJcVZ8YgyzC+707yCbduY17e0i+ifRZPfsoClC2SNqTm5GCq+Z05szZJ7StCGLA6Kg0704ODvie8wrXO+lCPQVZ4xCu1luYmakF4ILeec66+pJVnB3uExGtDEdcX4XSagAnY3IDxDDsMdIbYFadTXAO51Ns1+1/VNV6xDVdn1kW5ZkTsuMGfTFCcbKHR7vPYgy7rLJjf8tC3INaM5GeQRFw0SOVBfCb+pdkiP97Icamkh6h1Kd4H28+QNzi4sG7ZmbWoYrV668hxuBliNnMqcQ15X5XBWIW8wn5Lje4Fzd581hlS/Wl5HVwfV0xzhPSa0bSRNgxgrh/Th4NWcMFkY3W7nm1yvhVlqMsdJ+a7mtCfb5PXgMzIUKFizFjcswwiJVZ5xoUj5kTxVCn5OiQ2Yo9VtdKb6NUJWqrgleMScYLqzXkUNee1FWoDkE4c+bsE5qbIJw5c7bRti4xjtjOXEuFEfG6SCRVKXFOBS8JG0kIqVqklyWDP3WaiH+fE9IJ8qj4KgqwxBgxmFmxeU/I7zjYhxR7zMDjkycoHMqZvkkokDGbNM1vXnwZ6b4uofeY5K2ra6QDFVA7OqWUWr+1PtmR9TpqlydBEorcEKJXpJRXFOlVgc9ajZeZmeWUSws5xjHhvN+hhCCDtXc/h8Y5l/e+he+69w0zM3ttHxTqHlsrjkNQsz/g/lek3i8qpokzQOC03Y6esDtg4C1QQ2ieS1PtTVhtTaHXLixgUZ/PAVT2uJLYbl1AJdo6l3YC662+MwHTl6WeccodihsWM8hfKWCrwq+AhEOmQYekcb/0Ip7xAxL7gkBNb/gMUPioLfoigSWJ+piWGmqUE6jw6/sjTDkE4cyZs422FUEsZ2r8SYGLlIUr9PJKE6r8VOXffqlGOs3hE7b4itlcRzmg+QVSY3nMVClp3AlbpasE+5ANZgd3XjMzs6NjpH5SNiNJ6AnUHDjhtH3+tGmNPiRyiLvY9sF798zM7NEjFNxM5my3Tnr3YX/3svfsudoEmYLVpil5Kbn/VQ+c+x8NDIdsERBJbowFSUqR+kQrp3deMTOz116ASO3dCo1wbj/8ppmZJRGeg7ODd3mOXzIzs3cMiOJJiefAI/EsbImoCOGU3hr1t24VyPc1MWy3VOsuG+5KqEdpY0nT1xJ+Ki4T0UiSia3qdF8Nb/iZFynASUEYkgBzvipIPuBNP9lX8RWeu1tHCBb3EjyvFZGaSrZzfl/aEp7VOAu1BHXTZAZAdTmS2nMIwpkzZ5/UtjfvZUpRwp0Llj/3uHYan7P4hGuvBeMERyzS2hs2aULP19oaM1fSxzaHJy+amdn9D98zM7MPHyIe8Por9GZnKD2WXxTldTqd6g84XpeeQKklEqyi280iMeW23/0u1tq/+51vm1lTehwlWFP7XNtdXe1etLYrkVG+1/wey2tw/ZlXWhOTLMMinTJoN61RYxWmGtVZkOnovlAIRWk/8ypQ2kmGcbn3NmIP0QCo5YfeBLIY9nFv+hmIUjcFKdwSrS0at+rLxWrNq3V0oDTiqnjtrj1Wh6XYEm0JJONWraYQa8QmJKHryJtrDXgtKrWW165LrxnTiikhd8S05Ygxh4MBzuV4nxL13FGpy7KURL/o6RK5be65TqciWsxrYSahDrb9K7cjB5lDEM6cOdtoWxHEzQ2FVOh5O8xO3HC2n83hieYU8NzbgwcekqQ0aWUQUtFLVWKtcDFjEvffQ6mxpLAUze+wEEwS9BOVi5OM0lmT31Lzngu20VM7dDOzJ09RmPP+Q0Tnl1y79VU4FmIGf/0uPKnKbXdpA9LXc8YgFO0viKHy2hNwB8YZBiWuyfebiPbTEkVX5+cUeCFlWt9xdgRiU7eHsTq4i+s98v+YmZnd/+AXzczsJv2emZm9OsJYJvs4Xn+M+z5gwVgqVNB6qjx6S69QgZg8NPepZfBF29+tYEwcrpbAB/TGeg4l7prlqk+nFACLuLygiZlIiKinsm0iABUmEhzakEjihMVXKrZicsi6jBllpaTohB5VsCgxYom/NLJxaqhdrmcrPEnOVSuvH8PSXjGHIJw5c7bRtiKIQO3FyTVQM9LJArOUouP7Q3AWjg8pi85553rWIAi1HQs4m1YGdHI1JvWarfdefekFfBdzxovFnOfA7zrAenfAUtfJFdbJNxO8CmnIw+at9uyjPex7h16jP8RMXmS4zhmzNjcUo0l6/W3D81xsSmqyqNN+vfbFn6VWLqXzmLn2wwkKrsr5h/WxookInmYAAAM5SURBVEMK+t46NTOzZcbybdLWs3PEe/ZzjEsYvmxmZh+Sc/HSl8AH6QyxfeeUUfOQ95nPRyyKPSvs8la7Q48xCJ/ZqFBIiFmZjN4v5Vo53bHkXJmyIZMKC8O1Fo6UylPLSMkq6lolZmtmdrSP52V/RNkDySDw2RU66rLZUz9hHEnNsJVB0SNA7684gjEuUtTCsxSB8Ro/r6xFyFiU50mqAZ+rKFJxjGeZQxDOnDnbaFsRxMnBqsDIgmWrPbIbD/boociwVHMbRX7zljfwGGvoMXtxsI/CL8nXv3gbXm445LpMjDCy0gZD7JfyHN75DiLqKWffxRyxiSnjHmoUcnbrrD4HIRs1VS17yDffLJgZuUADGCEKP2oKcXZls2w1mu8rUM0JXxO/hnbK3LifIBt01m1iEFMTsxXvA7Yh2O9grCPGaconbMbzAhDELMZ+5yy++pHPvY7t2Mx5cvNDZmaWZbj/gUcxIDXibZX5l2JGMlYSim9AZKFaM18szx27rOkVCtU8Na2J8TxWKohS8ZO4DZ44PhjLssX1iSzhsVhOz2e7SBSDYAu+Kb7rRkxKHkNtKBULM0oaKDGlsu7SopX3nt/EyhSzU2yhqjMd2na1YdWzzCEIZ86cbbTtPAgyCmPGGpZkUu7tUZqM7d8PhkADUzIpr8bIFowpj25mFnWBAE5PEVkfseHr7AaeP4gl1Intl0QKM5aHZ4zqTxknmDJbEWn2jTHrLgt8Z4/TrkRxzZrS6Vqow19dT/a57eB4n8fefRZjlhJB0JWKnScKXN1CTS6CWYH3K8RunvgN+7Os2BD5HDGV5RxxmoQZoBG95nFE2fSSbEy6jWWBe/boCZBEFeN+j5dAK0t1w2GJthi01sqxh5JTM2UrtPZl2TQRReAL7ey2cU4pwRe1qGMjolLcA25Xg2FP7e4oldjy3jMMs50/pnQBEYLaD6jsu25UTKHmmAI+sg5/C0m82qjKXxP+7XXVNKoZMyFvxRzKUq0DKWfHZzqKyRt6BmRzCMKZM2cbzft+5a+dOXP2/585BOHMmbON5iYIZ86cbTQ3QThz5myjuQnCmTNnG81NEM6cOdtoboJw5szZRvs/epq2UaZB2S8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch(rows=3, figsize=(4,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating the databunch we have done the following things\n",
    "* Added data augmentation with the following [transforms](https://docs.fast.ai/vision.transform.html#get_transforms) and the size of images is taken as (32,32)\n",
    "* Normalied the images with the CIFAR_STATS\n",
    "* I am using a batch size of 256 for both trian and valid sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resnet-56\n",
    "The reason I am not using the original resnet models is due the two downsampling layers in the starting of the resnet model, namely conv and max_pool. If we use these layers than our image size is reduced to 8x8 after just two layers.\n",
    "\n",
    "So Resnet-56 was also used in the orginal papers and in genral it is a good model for small size images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extra Classes for the head of our model\n",
    "class AdaptiveConcatPool2d(nn.Module):\n",
    "    \"Layer that concats `AdaptiveAvgPool2d` and `AdaptiveMaxPool2d`.\"\n",
    "    def __init__(self, sz=None):\n",
    "        \"Output will be 2*sz or 2 if sz is None\"\n",
    "        super().__init__()\n",
    "        self.output_size = sz or 1\n",
    "        self.ap = nn.AdaptiveAvgPool2d(self.output_size)\n",
    "        self.mp = nn.AdaptiveMaxPool2d(self.output_size)\n",
    "    def forward(self, x): \n",
    "        return torch.cat([self.mp(x), self.ap(x)], 1)\n",
    "\n",
    "class Flatten(nn.Module):\n",
    "    \"Flatten `x` to a single dimension, often used at the end of a model. `full` for rank-1 tensor\"\n",
    "    def __init__(self, full:bool=False):\n",
    "        super().__init__()\n",
    "        self.full = full\n",
    "    def forward(self, x):\n",
    "        return x.view(-1) if self.full else x.view(x.size(0), -1)\n",
    "\n",
    "class BasicBlock(nn.Module):\n",
    "    def __init__(self, c_in, c_out, stride=1):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(c_in, c_out, kernel_size=3, stride=stride, padding=1, bias=False)\n",
    "        self.bn1 = nn.BatchNorm2d(c_out)\n",
    "        self.conv2 = nn.Conv2d(c_out, c_out, kernel_size=3, stride=1, padding=1, bias=False)\n",
    "        self.bn2 = nn.BatchNorm2d(c_out)\n",
    "        \n",
    "        if stride != 1 or c_in != c_out:\n",
    "            self.shortcut = nn.Sequential(\n",
    "                nn.Conv2d(c_in, c_out, kernel_size=1, stride=stride, bias=False),\n",
    "                nn.BatchNorm2d(c_out)\n",
    "            )\n",
    "        \n",
    "    def forward(self, x):\n",
    "        shortcut = self.shortcut(x) if hasattr(self, 'shortcut') else x\n",
    "        out = F.relu(self.bn1(self.conv1(x)))\n",
    "        out = self.bn2(self.conv2(out))\n",
    "        out += shortcut\n",
    "        return F.relu(out)\n",
    "        \n",
    "        \n",
    "class Resnet(nn.Module):\n",
    "    def __init__(self, num_blocks=[9,9,9], num_classes=100):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1, bias=False)\n",
    "        self.bn1 = nn.BatchNorm2d(16)\n",
    "        self.layer1 = self.make_group(16, 16, num_blocks[0], stride=1)\n",
    "        self.layer2 = self.make_group(16, 32, num_blocks[1], stride=2)\n",
    "        self.layer3 = self.make_group(32, 64, num_blocks[2], stride=2)\n",
    "\n",
    "        self.head = nn.Sequential(\n",
    "            AdaptiveConcatPool2d(),\n",
    "            Flatten(),\n",
    "            nn.BatchNorm1d(128),\n",
    "            nn.Dropout(0.25),\n",
    "            nn.Linear(128, 128, bias=True),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.BatchNorm1d(128),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(128, num_classes, bias=True)\n",
    "        )\n",
    "        \n",
    "    def make_group(self, c_in, c_out, num_blocks, stride):\n",
    "        layers = [BasicBlock(c_in, c_out, stride)]\n",
    "        for i in range(num_blocks-1):\n",
    "            layers.append(BasicBlock(c_out, c_out, stride=1))\n",
    "        return nn.Sequential(*layers)\n",
    "    \n",
    "    \n",
    "    def forward(self, x):\n",
    "        out = F.relu(self.bn1(self.conv1(x)))\n",
    "        out = self.layer1(out)\n",
    "        out = self.layer2(out)\n",
    "        out = self.layer3(out)\n",
    "        out = self.head(out)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy], callback_fns=[ShowGraph])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Till now we have done the following:\n",
    "* Fot out data, which is stored in `data` as a databunch\n",
    "* A Resnet model is created\n",
    "* A `learner` object is created which is names `learn`\n",
    "* AdamW is used as the optimization function\n",
    "* CrossEntropyLoss is used as loss function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cyclic Learning Rate\n",
    "The essence of this learning rate policy comes from the observation that increasing the learning rate might have a short term negative effect and yet achieve a longer term beneficial. So we vary our learning rate from a small value to large value than back to small value. This is termed as one complete cycle.\n",
    "\n",
    "<img src='Extra/triangularWindow.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This learning rate policy is taken from the original paper, but that was few years back. Now we use a cosine policy that lookes something like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEGCAYAAABPdROvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXhU5fn/8fedPYGwZWGHBAlL2CHgAmpZVKAK7oBaqdrSRVrR2hZrtX7t8lNr3arWUnG3Iq5gW9ncFVnCDoFA2MMSQoAQSMh6//6Yg01jEoYkM2eSuV/XNVdmzpxz8pmTydxzlud5RFUxxhhj6irE7QDGGGMaNyskxhhj6sUKiTHGmHqxQmKMMaZerJAYY4yplzC3A/hDfHy8JiUluR3DGGMajVWrVh1W1QRv5g2KQpKUlER6errbMYwxptEQkd3ezmuHtowxxtSLFRJjjDH1YoXEGGNMvVghMcYYUy9WSIwxxtSLTwuJiIwVkUwRyRKRmdU8HykibzrPLxeRJGd6nIh8IiInROTpKssMEZENzjJPiYj48jUYY4ypnc8KiYiEAs8A44BUYIqIpFaZ7TbgqKp2Bx4HHnamnwLuA+6uZtV/A6YBKc5tbMOnN8YY4y1ftiMZBmSp6g4AEZkDTAQyKs0zEXjAuf828LSIiKqeBL4Uke6VVygi7YEWqvq18/gV4ErgQx++DuMj5RXKxn35bNp/nLwTxSgQFR5CZFgorWLCSYiNJDE2io6toomOCHU7rjGmBr4sJB2BvZUeZwPn1jSPqpaJSD4QBxyuZZ3ZVdbZsboZRWQanj0XunTpcrbZjQ/lFhTz8tJdvLFiD3knS844vwh0ah1NSmIsvdrFMqRra4Z0bU2rmAg/pDXGnIkvC0l15y6qjqLlzTx1ml9VZwGzANLS0mz0rgBQUaG8vnw3jyzM5ERxGWN6t+Xy/u0Z3KU1iS0iCRGhuKyCU6XlHCss4dDxYg4VFLPnSCFbcwrIOnSCz7fmUlbh+XP2aNuckT0TGdUrkSFdWxMWateOGOMGXxaSbKBzpcedgP01zJMtImFAS+DIGdbZ6QzrNAHo+KlS7npzHUs25zC8exz/N6Ev3RObf2u+8NAQmkeGEd88ku6Jsd96vqiknHXZx0jfdYSvd+Txwlc7+fvnO2gZHc74fu24alAn0rq2JiTErsEwxl98WUhWAikikgzsAyYDN1SZZz4wFfgauBb4WGsZ+1dVD4hIgYicBywHbgb+6ovwpuEcOn6KG59fzs7DJ7n/8lRuGZ5EXS+2i44I5bxucZzXLY7po1IoOFXKl9sOs3DTQd5fs583VuylU+toJg/tzORhXYhvHtnAr8YYU5X4csx2ERkPPAGEAi+o6h9F5EEgXVXni0gU8CowCM+eyORKJ+d3AS2ACOAYcKmqZohIGvASEI3nJPvPais+4Dm0ZZ02uuNAfhFTZi3jUEExz9+cxgXd4332u04Wl7E4I4e3Vu3lq6w8IkJDuHxAe34wohupHVr47Pca0xSJyCpVTfNqXl8WkkBhhcQd+UWlXPfcUvYfO8XLtw5jSNfWfvvdWYcKeOXr3byzKpuTJeWM6d2Wn4/uTv9OrfyWwZjGzApJFVZI/K+0vIKbZ68gffcRXr5lmE/3RGqTX1jKS0t38cJXO8kvKmV0r0R+Pa4XPdp++/yLMea/zqaQ2GUuxif+vDCTr3fk8dDV/V0rIgAtY8K5Y0wKX80cxS8v68mKXUcY+8TnzHxnPYeOn3ItlzFNiRUS0+AWbTrIrM938L3zunLNkE5nXsAPmkeGcfvI7nz2y5FMvSCJd1ZnM/LRT3n+ix2UlVe4Hc+YRs0KiWlQhwpO8et31tO3Ywt+e3lvt+N8S5tmEfzuij4svvNi0pLa8Id/b+byv37Jqt1H3Y5mTKNlhcQ0GFXlN+9u4GRJOU9MGkhkWOB2a5IU34yXbhnKczcNJr+olGufW8of/pXBqdJyt6MZ0+hYITEN5oP1B1iy+RC/uqxntY0JA42IMLZve5bcdTE3nduV57/cyfgnv2DV7traxBpjqrJCYhpEflEpD36QQf9OLblleLLbcc5Ks8gwfn9lX17/wbkUl1Vw3XNf89RH2yivaPpXNBrTEKyQmAbx2KJMjpws5o9X9iO0kXZPMrx7PAvvvIgJAzrw2OKtfG/2cruyyxgvWCEx9bYtp4DXlu/hhnO70K9TS7fj1EvzyDAenzSQR67tz+o9Rxn/1Bd8vjXX7VjGBDQrJKbe/vifzcREhHLXJT3djtIgRITr0zrzwfQRxDWL5OYXVvDowkwq7FCXMdWyQmLqZWnWYT7NzOWO0Sm0ada0xgdJaRvL+7cP5/q0Tjz9SRY/em0VJ4rL3I5lTMCxQmLqTFV5bPFW2reM4nvnd3U7jk9ER4Ty8DX9eeCKVD7ecohrnl3K3iOFbscyJqBYITF19sW2w6TvPspPR3YP6DYj9SUifH94Mi/fMowD+UVMePpLlu3IczuWMQHDCompE1Xl8SVb6dAyiuvTAqMbFF8bkRLPvOkjaNMsgpueX85b6XvPvJAxQcAKiamTz7bmsmbPMaaPSmnSeyNVJcc3473bh3P+OXH88u31PPtpFsHQg7YxtbFCYs6aqvL44q10bBXNtQHSKaM/tYgKZ/bUoUwc2IFHFmTyfx9k2BVdJqj5cqhd00R9ujWXddn5PHR1PyLCgvO7SERYCI9fP5D45pHM/nIneSdLePS6/kG1d2bMaVZIzFl7/osdtGsRxdWDg29vpLKQEOG+y1Np2yKSP/1nC0dPlvD37w2hWaT9W5ngEpxfJ02dZew/zldZeUy9IClo90aqmnbROfzlugF8vSOPqS+soOBUqduRjPEr+yQwZ2X2lzuJDg/lhmFd3I4SUK4Z0om/ThnE2r3HuPmFFRy3YmKCiBUS47VDx08xf90+rk/rRMuYcLfjBJzx/drzzI2D2bgvn5ueX05+oRUTExyskBivvbpsN2UV2ui6ifeny/q047mbhrDlQAFT/rGMoydL3I5kjM9ZITFeKSop57VluxnTuy1J8c3cjhPQRvduy6ybh5CVe4Ip/1jGsUIrJqZps0JivPLemn0cLSzlByNsb8Qb3+mZyOypaezIPcnUF1daZ4+mSbNCYs5IVXlt2W56tYtlWHIbt+M0GhemJHxzzuTWl1ZSVGLjwZumyQqJOaP12flkHDjOjed2QaRxjn7olktS2/LY9QNYuesIP35tFcVlVkxM02OFxJzRGyv2EB0eysRBHd2O0ihNHNiRh67ux2dbc/n5G2soK69wO5IxDcoKialVwalS5q/bz4QBHWgRZZf81tWkoV24//JUFm7K4dfvbLCOHk2TYn05mFrNW7ufwpJyppxrDRDr69YRyRw/VcoTS7bRtkUkvxrby+1IxjQIKySmRqrKP5fvIbV9CwZ0aul2nCbhjtEp5Bwv5tlPt9O2RRRTL0hyO5Ix9WaHtkyNTp9kn2In2RuMiPD7iX24JLUtD3ywif9sOOB2JGPqzaeFRETGikimiGSJyMxqno8UkTed55eLSFKl5+5xpmeKyGWVpt8pIptEZKOIvCEiUb58DcFszkrPSfYrB3ZwO0qTEhYawl+nDGJwl9bMmLPWhu01jZ7PComIhALPAOOAVGCKiKRWme024KiqdgceBx52lk0FJgN9gLHAsyISKiIdgZ8DaaraFwh15jMN7FRpOf9ad4Dx/doTayfZG1xUeCizp6bRJS6GH76SztacArcjGVNnvtwjGQZkqeoOVS0B5gATq8wzEXjZuf82MFo8x1AmAnNUtVhVdwJZzvrAc14nWkTCgBhgvw9fQ9BalJFDQXEZ1wyxS359pVVMBC/fOozo8FBueXEluQXFbkcypk58WUg6AnsrPc52plU7j6qWAflAXE3Lquo+4FFgD3AAyFfVRdX9chGZJiLpIpKem5vbAC8nuLyzKpsOLaM4LznO7ShNWsdW0cyeOpS8k8VMezWdU6XWYNE0Pr4sJNWdna168XxN81Q7XURa49lbSQY6AM1E5KbqfrmqzlLVNFVNS0hIOIvY5tDxU3yxLZerBnckJMROsvtav04teWLSQNbsOcbdb62z8d9No+PLQpINdK70uBPfPgz1zTzOoaqWwJFalh0D7FTVXFUtBd4FLvBJ+iA2b+1+KpSgH0rXn8b2bc/Mcb341/oDPLFkq9txjDkrviwkK4EUEUkWkQg8J8XnV5lnPjDVuX8t8LF6mvzOByY7V3UlAynACjyHtM4TkRjnXMpoYLMPX0PQUVXeWZ3NwM6tOCehudtxgsqPLurG9WmdeOrjLN5dne12HGO85rNC4pzzmA4sxPNhP1dVN4nIgyIywZltNhAnIlnAXcBMZ9lNwFwgA1gA3K6q5aq6HM9J+dXABif/LF+9hmCUceA4Ww4WcM0Q2xvxNxHhD1f24/xuccx8ZwOrdh91O5IxXpFg6PMnLS1N09PT3Y7RKDz4QQavLtvFynvH0Comwu04QelYYQkTnv6KU6XlfPCzEbRtYU2ljP+JyCpVTfNmXmvZbr5RXqHMX7efUb0SrYi4qFVMBLNuHsKJ4jLret40ClZIzDeW7cjj8IliJg60tiNu69WuBX+5bgBr9hzjvvc3Wm/BJqBZITHf+GDdfppFhDKqV6LbUQwwrl97fjaqO3PTs3l12W634xhTIyskBoCSsgo+3HiQS/u0Iyo81O04xnHnmB6M7pXIgx9kWJ9cJmBZITEAfLEtl/yiUq4Y0N7tKKaSkBDh8ckD6RIXw09fX82+Y0VuRzLmW6yQGADmr9tPy+hwRnS3XgACTYuocP5xcxqlZRVMeyWdohI7+W4CixUSQ1FJOYszchjfrx0RYfaWCETnJDTnySkDyThwnPvm2cl3E1jsU8Pw0ZYcCkvKuaK/jTsSyEb1asvPRnbn7VXZzE3fe+YFjPETKySG+Wv3kxAbybndrKffQHfHmB6M6B7PffM2sXFfvttxjAGskAS9E8VlfLo1l+/2a0+o9fQb8EJDhCcnD6RNTAQ/fX01+UWlbkcyxgpJsPt4yyFKyioY38+u1mos4ppH8syNg9h/rIi731pn50uM66yQBLkFGw8Q3zySIV1bux3FnIUhXdtwz/jeLM7IYdbnO9yOY4KcFZIgVlRSzidbchnbt60d1mqEbh2exPh+7XhkYSbLrbGicZEVkiD22dZDFJWWM66vHdZqjESEh6/pT9c2MUx/Yw2HCk65HckEKSskQezDjQdpHRPOuclt3I5i6ig2KpxnbxpMwalSZsxZS7kN02tcYIUkSBWXlfPR5kNcmtqOsFB7GzRmvdq14MEJfVm6PY+/fZrldhwThOwTJEh9ue0wJ4rLGNuvndtRTAO4Lq0TEwZ04PEl21i564jbcUyQsUISpD7ceJDYqDCGnxPvdhTTAESEP17Vl06to7njjTUcKyxxO5IJIlZIglBpeQWLM3IY07ut9a3VhMRGhfPXKYPIPVHML99eb+1LjN/Yp0gQ+np7HvlFpYzra4e1mpr+nVrx67G9WJyRwytf22BYxj+8KiQiMkJEbnHuJ4hIsm9jGV/6cONBYiJCuaiHdRnfFN02IpnRvRL54783W39cxi/OWEhE5HfAr4F7nEnhwGu+DGV8p7xCWZxxkJG9Em0kxCZKRPjzdQNo3Sycn72xhhPFZW5HMk2cN3skVwETgJMAqrofiPVlKOM7K3cd4fCJEsZbI8QmrU2zCJ6cPIjdeSe5f95Gt+OYJs6bQlKinrN2CiAizXwbyfjSgo0HiQwL4Ts97bBWU3detzh+PjqFd1fv451V2W7HMU2YN4Vkroj8HWglIj8ElgDP+zaW8QVVZdGmg1yYkkCzyDC34xg/+NmoFM5NbsN98zayI/eE23FME3XGQqKqjwJvA+8APYH7VfUpXwczDW/T/uPszz/Fpalt3Y5i/MQzfskgIsJCuGPOWkrKKtyOZJogb062P6yqi1X1l6p6t6ouFpGH/RHONKzFGTmIwKjeiW5HMX7UrmUUD13dnw378nliyVa345gmyJtDW5dUM21cQwcxvrc4I4chXVoT3zzS7SjGz8b2bcfkoZ3522fbWWZdzpsGVmMhEZGfiMgGoKeIrK902wms919E0xCyjxaSceA4l9hhraB13+WpdG0Tw11vriW/0IboNQ2ntj2SfwJXAPOdn6dvQ1T1Jj9kMw1oSUYOgBWSINYsMownJw/iUEEx976/wbpQMQ2mxkKiqvmquktVp6jqbqAIzyXAzUWkizcrF5GxIpIpIlkiMrOa5yNF5E3n+eUiklTpuXuc6Zkiclml6a1E5G0R2SIim0Xk/LN4vUFr8eYczkloRreE5m5HMS4a0LkVd17Sg3+tP8B7a/a5Hcc0Ed6cbL9CRLYBO4HPgF3Ah14sFwo8g+d8SiowRURSq8x2G3BUVbsDjwMPO8umApOBPsBY4FlnfQBPAgtUtRcwANh8pizBLr+olOU7jnBJqvWtZeDHF5/DsKQ23D9vE3uPFLodxzQB3pxs/wNwHrBVVZOB0cBXXiw3DMhS1R2qWgLMASZWmWci8LJz/21gtIiIM32Oqhar6k4gCxgmIi2Ai4DZAKpaoqrHvMgS1D7NPERZhdphLQN4Lgl+bNIABJjx5lrKyu2SYFM/3hSSUlXNA0JEJERVPwEGerFcR2BvpcfZzrRq51HVMiAfiKtl2W5ALvCiiKwRkedramkvItNEJF1E0nNzc72I23Qtysghvnkkgzq3cjuKCRCdWsfwh6v6smr3UZ75ZLvbcUwj500hOSYizYHPgddF5EnAm17gpJppVc/u1TRPTdPDgMHA31R1EJ7+v7517gVAVWepapqqpiUkBG93IMVl5XyWmcuY3omEhFS3WU2wmjiwI1cO7MBTH29j9Z6jbscxjZg3hWQiUAjcCSwAtuO5eutMsoHOlR53AvbXNI+IhAEtgSO1LJsNZKvqcmf623gKi6nBsh1HOFFcZoe1TLUevLIv7VpEMWPOWusl2NSZN12knFTVClUtU9WX8ZxAH+vFulcCKSKSLCIReE6ez68yz3xgqnP/WuBjp4PI+cBk56quZCAFWKGqB4G9ItLTWWY0kOFFlqC1OOMg0eGhDO9uQ+qab2sRFc7jkwaSfbSQB+ZvcjuOaaRqa5DYwrkE92kRuVQ8pgM7gOvPtGLnnMd0YCGeK6vmquomEXlQRCY4s80G4kQkC7gL5zCVqm4C5uIpEguA21W13FnmZ3gOsa3Hc67mT2f/soODqrIk4xAX9Yi3sUdMjYYlt+H2kd15e1U2/15/wO04phGSmholicg84CjwNZ5v/q2BCOAOVV3rt4QNIC0tTdPT092O4Xfrs48x4emvePS6AVw7pJPbcUwAKy2v4NrnvmbX4ZMsnHER7VpGuR3JuExEVqlqmjfz1nZoq5uqfl9V/w5MAdKAyxtbEQlmSzJyCBEY1cs6aTS1Cw8N4YlJAykpq+Dut9ZRUWGt3o33aisk33TG4xxW2qmqBb6PZBrKoowc0rq2oU2zCLejmEYgOb4Z912eypdZh3lx6S6345hGpLZCMkBEjju3AqD/6fsictxfAU3d7D1SyJaDBXa1ljkrU4Z1ZnSvRB5esIXMg/a90Xintr62QlW1hXOLVdWwSvdb+DOkOXuLrZNGUwciwkPX9Cc2MowZb66luKz8zAuZoOdNOxLTCC3OyCElsTlJ8dU2/DemRgmxkTx8TX82HzjOY4tsICxzZlZImqBjhSWs2HXE9kZMnY1JbcuUYV2Y9cUOGwjLnJEVkibok8xDlFsnjaaefvvd3nRtE8Mv5q4jv8gGwjI1s0LSBC3OyCExNpIBnayTRlN3zSLDeHzSQA4eP8Xv5m10O44JYN6MR1JQ6eqt07e9IvKeiHTzR0jjvdOdNI7u3dY6aTT1NqhLa342qjvvr93PB+uqdpVnjEeYF/M8hqfDxH/i6ZV3MtAOyAReAL7jq3Dm7C3dnsfJknIutcNapoFMH9mdTzNzufe9DaQltaZ9y2i3I5kA482hrbGq+ndVLVDV46o6Cxivqm/i6TbFBJDFGTnERIRy/jlxbkcxTURYaAiPTxpIWYXyi7nW6t18mzeFpEJErheREOdWucNGe0cFkIoKZUlGDhf3SLBOGk2DOt3qfen2PF74aqfbcUyA8aaQ3Ah8DzgE5Dj3bxKRaDy9+5oAsX5fPocKiu1qLeMTk4d2ZkzvtjyyMJMtB61zC/Nf3oxHskNVr1DVeFVNcO5nqWqRqn7pj5DGO4szDhIaItZJo/EJT6v3frSICmPGHGv1bv7Lm6u2EkTkNyIyS0ReOH3zRzhzdhZn5DA0qTWtYqyTRuMb8c0jeeTa/mw5WMBfrNW7cXhzaGseniFwlwD/rnQzAWR33km25pzgktR2bkcxTdyoXm254dwu/OOLHSzdftjtOCYAeHP5b4yq/trnSUy9nO6k0S77Nf7w2+/25uvtedw9dx0fzriIltHhbkcyLvJmj+RfIjLe50lMvSzalEOvdrF0bhPjdhQTBGIiPK3ecwqKud9avQc9bwrJHXiKSZGNRxKYjpwsIX33EdsbMX41sHMrfj4qhXlr9zNv7T634xgXeXPVVqyqhqhqtI1HEpg+2pxDhcKlfez8iPGv20eew6Aurfjt+xvZf6zI7TjGJTUWEhHp5fwcXN3NfxHNmSzKyKFDyyj6dLD6bvwrzBnrvdxavQe12k623wVMA/5SzXMKjPJJInNWikrK+WJbLtendUbEOmk0/tc1rhn3X57KzHc38MJXO/nBhdaXa7CpsZCo6jTn50j/xTFn68usw5wqreBSu+zXuGjS0M58tOUQjyzIZHj3eHq3t73jYOLVeCQicoGI3CAiN5+++TqY8c6iTQeJjQrj3G5t3I5igpiI8NDV/WgRHc6db67lVKm1eg8m3rRsfxV4FBgBDHVuaT7OZbxQXqF8vOUQI3smEh5qY5QZd8U1j+SRa/s5rd4z3Y5j/MibBolpQKqq2lm0ALN6z1HyTpZwaR+77NcEhlG92nLTeV34xxc7GdkzkQu6x7sdyfiBN19jN+IZyMoEmEWbDhIeKlzcI8HtKMZ8497xqXSLb8Yv3lpHfqGN9R4MvCkk8UCGiCwUkfmnb74OZmqnqizOyOH8c+KJjbLuKUzgiI4I5fFJA8ktKOY+a/UeFLw5tPWAr0OYs5d16AS78grtUksTkAZ0bsUdo1P4y+KtjO6dyMSBHd2OZHyo1kIiIqHAfao6xk95jJcWOZ002iBWJlD95Dvn8EnmIX77/kbSktrQsZWN9d5U1XpoS1XLgUIRaemnPMZLizJyGNCpJW1bRLkdxZhqnR7rvaJC+cXctdbqvQnz5hzJKWCDiMwWkadO37xZuYiMFZFMEckSkZnVPB8pIm86zy8XkaRKz93jTM8UkcuqLBcqImtE5F/e5Ghqco6fYt3eY9a3lgl4XeOa8bsr+rBsxxGe/3KH23GMj3hzjqROA1k5h8WeAS4BsoGVIjJfVTMqzXYbcFRVu4vIZOBhYJKIpAKTgT5AB2CJiPRw9pDA0yPxZiAom88utsNaphG5Lq0TSzbn8OjCrVyYkmCt3psgb3r/fbm6mxfrHgZkOWO+lwBzgIlV5pkInF7X28Bo8XQYNRGYo6rFqroTyHLWh4h0Ar4LPO/NC2yKFmfk0DUuhpTE5m5HMeaMRIT/57R6nzHHWr03Rd60bE8RkbdFJENEdpy+ebHujsDeSo+znWnVzqOqZUA+EHeGZZ8AfgVUnCH3NBFJF5H03NxcL+I2DgWnSvl6ex6Xpra1ThpNoxHXPJI/X9efzJwC/rzQWr03Nd6cI3kR+BtQBowEXgFe9WK56j7lqp5tq2meaqeLyOXAIVVddaZfrqqzVDVNVdMSEppOg71PM3MpKa+wsdlNozOyZyLfO68rs7/cyVdZNtZ7U+JNIYlW1Y8AUdXdqvoA3nUhnw10rvS4E7C/pnlEJAxoCRypZdnhwAQR2YXnUNkoEXnNiyxNxoKNB4lvHsGQrq3djmLMWfvN+N50S2jGL+Zaq/emxKurtkQkBNgmItNF5Cog0YvlVgIpIpIsIhF4Tp5XbRE/H5jq3L8W+Njp02s+MNm5qisZSAFWqOo9qtpJVZOc9X2sqjd5kaVJOFVazieZh7isTztCQ+ywlml8oiNCeXLSIA6fKObe9zdgXfg1Dd4UkhlADPBzYAhwE//98K+Rc85jOrAQzxVWc1V1k4g8KCITnNlmA3EikoVnIK2ZzrKbgLlABrAAuL3SFVtB67OtuRSWlDOub3u3oxhTZ/06tWTGmBT+tf4A89ZWPUhhGiPx9huBiDRT1ZM+zuMTaWlpmp6e7naMepsxZw2fbs1l5b1jrNt406iVVyiT/v41mTkFfHjHhXRqHeN2JFOFiKxSVa+GDPHmqq3zRSQDz14FIjJARJ6tZ0ZzlorLyvlo8yEuTW1rRcQ0eqEhUqnV+zrKrdV7o+bNJ9ITwGVAHoCqrgMu8mUo821fZR2moLjMDmuZJqNzmxh+N6EPy3ce4fkvrNV7Y+bVV1tV3VtlUtCfr/C3DzccJDYyjAu6x7kdxZgGc92QTozt045HF2WSsf+423FMHXlTSPaKyAV42nFEiMjdOIe5jH+UlleweHMOY1LbEhkW6nYcYxqMiPCnq/vRKiaCGW+usVbvjZQ3heTHwO14WpZnAwOBn/oylPlfy3bkcaywlLF9rRGiaXraNIvg0esGsDXnBH/4d8aZFzABx5u+tg6r6o2q2lZVE512Gzf7IZtxfLjxIDERoTakrmmyLu6RwI8u6sZry/bw7/UH3I5jzlJdL/+5q0FTmBqVVyiLNh1kZK9EosLtsJZpuu6+rCeDurRi5jvr2ZNX6HYccxbqWkisWbWfrNx1hMMnShhnh7VMExceGsJTkwchAtPfWE1JWa39spoAUtdCYhd9+8mCjQeJDAthZE9veqUxpnHr3CaGR64dwPrsfB5esMXtOMZLNQ5sJSIFVF8wBLDBl/2gokJZsPEgF/dIoFmkN2OQGdP4je3bjqnne3oJPq9bnA3g1gjUuEeiqrGq2qKaW6yq2qeaH6zac5SDx08xvp81QjTB5Z7xvenToQV3v7WOfceK3I5jzsD62ghg89fuJyo8xMo0OgUAABTJSURBVL6RmaATFR7K0zcMpqy8gjveWENZuZ0vCWRWSAJUWXkF/9lwgNG929phLROUkuOb8aer+5G++yiPL9nqdhxTCyskAWrp9jzyTpZwRf8ObkcxxjUTB3Zk8tDOPPvpdj7f2nSGzG5qrJAEqA/W7Sc2Mozv9LRGiCa4/e6KPqQkNufON9dyMP+U23FMNayQBKDisnIWbDrIpX3aWSNEE/SiI0J59sbBFJWWc/s/rX1JILJCEoA+y8yl4FQZVwywq7WMAeieGMvD1/Rn1e6j/Ok/1mdsoLFCEoDmr9tPm2YRDO8e73YUYwLGFQM6cMvwJF5auot5a/e5HcdUYoUkwBSWlPHR5kOM69vORkI0porfjO9NWtfWzHxnA1tzCtyOYxz2SRVgFmw8SFFpORMG2NVaxlQVHhrCMzcOpllkGD9+bRUFp0rdjmSwQhJw3l29j85tohma1MbtKMYEpLYtonj6hkHszivkV2+vR9W6/nObFZIAciC/iK+2H+bqQZ0ICbEOlo2pyXnd4vj12J58uPEgz3+x0+04Qc8KSQB5b80+VOHqwR3djmJMwPvhhd0Y26cdDy3YwtLth92OE9SskAQIVeXd1fsYmtSarnHN3I5jTMATEf58XX+S45tx++ur2XvEBsNyixWSALE+O5+sQye4enAnt6MY02jERoXzj5vTKK9QfvhKOieLy9yOFJSskASId1dnExEWwnf7WyNEY85Gcnwznr5hMFtzCrhr7loqKuzku79ZIQkAJWUVzF+3n0tT29IiKtztOMY0Ohf1SODe76aycFMOT360ze04Qcf6Jw8AH2/J4WhhKdfYYS1j6uzW4UlsPnCcJz/aRq92sYyzAeH8xvZIAsA/V+ylfcsoLkyxLlGMqSsR4Y9X9WVQl1bcNXcdGfuPux0paFghcdneI4V8sS2XSUM7E2ZdohhTL5Fhofz9piG0jA7nh6+kk3ei2O1IQcGnn1wiMlZEMkUkS0RmVvN8pIi86Ty/XESSKj13jzM9U0Quc6Z1FpFPRGSziGwSkTt8md8f5qzcgwCThnZ2O4oxTUJiiyhm3TyEwyeKmfbqKk6VlrsdqcnzWSERkVDgGWAckApMEZHUKrPdBhxV1e7A48DDzrKpwGSgDzAWeNZZXxnwC1XtDZwH3F7NOhuN0vIK5qZnM6pXIu1bRrsdx5gmo3+nVjx2/UBW7T7K3W+tsyu5fMyXeyTDgCxV3aGqJcAcYGKVeSYCLzv33wZGi4g40+eoarGq7gSygGGqekBVVwOoagGwGWi0zcCXZOSQW1DMDed2cTuKMU3Od/u3Z+a4Xvxr/QEeXZTpdpwmzZeFpCOwt9LjbL79of/NPKpaBuQDcd4s6xwGGwQsr+6Xi8g0EUkXkfTc3MAc6/mfK/bQoWUUF/dIdDuKMU3Sjy7qxpRhXXj20+3MWbHH7ThNli8LSXW9Dlbdv6xpnlqXFZHmwDvADFWt9tIMVZ2lqmmqmpaQEHjjnu/JK+SLbYeZNLQLodZBozE+ISL8fmIfLu6RwL3vb+TzrYH5pbKx82UhyQYqn0HuBOyvaR4RCQNaAkdqW1ZEwvEUkddV9V2fJPeD15bvJjREuH6otR0xxpfCQkN4+oZBpCQ256evr7bLgn3Al4VkJZAiIskiEoHn5Pn8KvPMB6Y6968FPlbP4ALzgcnOVV3JQAqwwjl/MhvYrKqP+TC7T50oLuONFXsY17ednWQ3xg9io8J58ZahxEaFMfXFFezJsw4eG5LPColzzmM6sBDPSfG5qrpJRB4UkQnObLOBOBHJAu4CZjrLbgLmAhnAAuB2VS0HhgPfA0aJyFrnNt5Xr8FX3krfS8GpMn5wYTe3oxgTNNq3jOaVW4dRWl7B915YTm6BtTFpKBIMo4ulpaVpenq62zEAKK9QRj76KQmxkbzzkwvcjmNM0Fm95yg3/mM5yfHNmPOj86x/uxqIyCpVTfNmXmtK7WeLM3LYc6SQH4xIdjuKMUFpcJfWPHuTp7fgaa+kW4PFBmCFxM9mf7mDzm2iubRPO7ejGBO0RvZM5NHrBrBsxxHumLOGsvIKtyM1alZI/Gjd3mOs3HWU71+QbJf8GuOyKwd15P7LPV3P3zV3HeXW+r3OrBt5P3rus+3ERoZxfZpd8mtMILh1RDKnysp5ZEEmEWEhPHJNf0LsS95Zs0LiJxn7j/PhxoP8fFR3Yu3knjEB46ff6U5JWQVPLNlGeGgIf7qqL56WBsZbVkj85MmPthIbFcZtI+ySX2MCzR2jUygpq+DZT7cTESo8MKGPFZOzYIXEDzbuy2fhphxmjEmhZYztjRgTaESEX17Wk9LyCv7xxU7CQkP47Xd7WzHxkhUSP3hiyTZaRIVxq13ya0zAEhF+M743ZRXK7C93cqq0nN9P7GvnTLxghcTHNmTns2RzDndd0sMaPhkT4ESE+y9PJTo8lGc/3U5RSTmPXNvfRi89AyskPvbookxaRodzy/Akt6MYY7wgIvxqbC9iIkJ5dNFWikrLeXLyICLCrJjUxLaMD322NZfPtuYyfaRdqWVMYzN9VAr3XZ7KhxsPMu1VawFfGyskPlJWXsEf/pVB17gYbr6gq9txjDF1cNuIZP50VT8+25rLjc8v5+jJErcjBSQrJD7yyte72XboBPeM60VkWKjbcYwxdXTDuV145obBbNiXzzV/W2pd0FfDCokPHMgv4i+LMrm4RwKXWZ9axjR64/u15/UfnMuRwhKu/ttXrNt7zO1IAcUKSQNTVR6Yv4myCuX3E62FrDFNxdCkNrzzkwuIjghl8qxlLMnIcTtSwLBC0sDmrd3vND7sQZe4GLfjGGMa0DkJzXn3J8NJaducaa+m89xn2wmGMZ3OxApJAzqQX8R98zYypGtrpl1kXaEY0xQlxEYyZ9p5jOvXnoc+3ML0N9ZwsrjM7ViuskLSQErKKpj+zzWUVyh/uW6AdRNvTBMWExHG01MGcc+4Xny44QBXP7uUXYdPuh3LNVZIGsif/rOZVbuP8vA1/UmKb+Z2HGOMj4kIP7r4HF6+dRg5BaeY8PSXfLLlkNuxXGGFpAG8+vUuXlq6i1uGJ3HFgA5uxzHG+NGFKQl8MH0EHVvHcMtLK3nwg4yga7xohaSePtxwgN/N38SY3oncO76323GMMS7o3CaG9356AVPP78oLX+3kyme+YmtOgdux/MYKST3MW7uP6W+sYWDnVjw1ZZB17GZMEIsKD+X/Jvblxe8P5fCJYq7465e8vHRXUFzVZZ98dVBRofz1o23MeHMtaV1b88pt5xITYf1fGmNgZK9EPrzjIs4/J47fzd/E92avaPKt4a2QnKXdeSeZ+uIK/rJ4KxMHdOClW4bRPNKKiDHmvxJiI3nx+0P5/ZV9Wbv3GJc+8RmzPt9OWXmF29F8QoJhtystLU3T09PrvHxFhZJx4DivL9/DO6uziQgN4dfjenHTuV2s5boxplYH8ou47/2NLNl8iF7tYnlgQh/O6xbndqwzEpFVqprm1bxWSKqnqkx7dRW5BcXsOVLIkZMlRISFcPWgjswY04N2LaN8lNYY09SoKgs2HuQP/97MvmNFfLdfe2aO60XnNoHb+8XZFBI7JlMDESG/qJTYqDDG9E7k3OQ4RvZKpE2zCLejGWMaGRFhXL/2jOyVyN8/28HfPsticUYON5zbhZ+OPIfE2Mb9xdT2SIwxxs8O5Bfx1EfbmJueTXiocPP5Sdw2Ipm2LQKnoNihrSqskBhjAtGuwyd5YslW5q/bT2iIMHFgR354YTd6tot1O5oVkqqskBhjAtmevEJmf7mDuenZFJWWk9a1NZOGduby/h2IjnBnYDwrJFVYITHGNAZHT5YwN30vb6bvZUfuSZpHhnFJalvG9W3HRT0SiAr3X1EJmEIiImOBJ4FQ4HlVfajK85HAK8AQIA+YpKq7nOfuAW4DyoGfq+pCb9ZZHSskxpjGRFVZuesob6/ay6KMHI4VlhITEcr53eIY3j2e4d3jSUlsTogPexkPiEIiIqHAVuASIBtYCUxR1YxK8/wU6K+qPxaRycBVqjpJRFKBN4BhQAdgCdDDWazWdVbHCokxprEqLa9g2Y48Fm46yFdZeex0uquPjQyjT8cW9OvYkpTEWDq3iaFzm2jatogivAG6awqUy3+HAVmqusMJNQeYCFT+0J8IPODcfxt4Wjwt/CYCc1S1GNgpIlnO+vBincYY02SEh4ZwYUoCF6YkAJB9tJCl2/NYn32MDfuO8/LXuykp+98W880iQmkZHU6n1jHM/fH5Ps/oy0LSEdhb6XE2cG5N86hqmYjkA3HO9GVVlu3o3D/TOgEQkWnANIAuXbrU7RUYY0yA6dQ6huvTYrg+rTMAZeUV7D92ij1HCtlzpJDcgmLyi0rJLyolPNQ/PW/4spBU9wqqHkeraZ6aple3v1btsTlVnQXMAs+hrZpjGmNM4xUWGkKXuBi6xLnXSt6XnTZmA50rPe4E7K9pHhEJA1oCR2pZ1pt1GmOM8SNfFpKVQIqIJItIBDAZmF9lnvnAVOf+tcDH6jn7Px+YLCKRIpIMpAArvFynMcYYP/LZoS3nnMd0YCGeS3VfUNVNIvIgkK6q84HZwKvOyfQjeAoDznxz8ZxELwNuV9VygOrW6avXYIwx5sysQaIxxphvOZvLf21gK2OMMfVihcQYY0y9WCExxhhTL1ZIjDHG1EtQnGwXkVxgdx0XjwcON2CchmTZ6iaQs0Fg57NsdRPI2aD6fF1VNcGbhYOikNSHiKR7e+WCv1m2ugnkbBDY+Sxb3QRyNqh/Pju0ZYwxpl6skBhjjKkXKyRnNsvtALWwbHUTyNkgsPNZtroJ5GxQz3x2jsQYY0y92B6JMcaYerFCYowxpl6skNRARMaKSKaIZInITJezdBaRT0Rks4hsEpE7nOkPiMg+EVnr3Ma7mHGXiGxwcqQ709qIyGIR2eb8bO1Crp6Vts9aETkuIjPc2nYi8oKIHBKRjZWmVbudxOMp5z24XkQGu5DtzyKyxfn974lIK2d6kogUVdp+z/kyWy35avw7isg9zrbLFJHLXMj2ZqVcu0RkrTPdr9uuls+PhnvfqardqtzwdFG/HegGRADrgFQX87QHBjv3Y4GtQCqe8e7vdnt7Obl2AfFVpj0CzHTuzwQeDoC/60Ggq1vbDrgIGAxsPNN2AsYDH+IZMfQ8YLkL2S4Fwpz7D1fKllR5Phe3XbV/R+f/Yx0QCSQ7/8+h/sxW5fm/APe7se1q+fxosPed7ZFUbxiQpao7VLUEmANMdCuMqh5Q1dXO/QJgM/8dwz6QTQRedu6/DFzpYhaA0cB2Va1rLwf1pqqf4xl7p7KattNE4BX1WAa0EpH2/symqotUtcx5uAzPqKSuqGHb1WQiMEdVi1V1J5CF5//a79lERIDrgTd89ftrU8vnR4O976yQVK8jsLfS42wC5INbRJKAQcByZ9J0Z/fzBTcOHVWiwCIRWSUi05xpbVX1AHjezECia+k8JvO//8yBsu1q2k6B9j68Fc831dOSRWSNiHwmIhe6FYrq/46BtO0uBHJUdVulaa5suyqfHw32vrNCUj2pZprr10mLSHPgHWCGqh4H/gacAwwEDuDZfXbLcFUdDIwDbheRi1zM8i3iGZp5AvCWMymQtl1NAuZ9KCL34hmt9HVn0gGgi6oOAu4C/ikiLVyIVtPfMWC2HTCF//0C48q2q+bzo8ZZq5lW67azQlK9bKBzpcedgP0uZQFARMLxvAleV9V3AVQ1R1XLVbUC+Ac+3HU/E1Xd7/w8BLznZMk5vUvs/DzkVj48BW61quZAYG07at5OAfE+FJGpwOXAjeocRHcOGeU591fhOQfRw9/Zavk7Bsq2CwOuBt48Pc2NbVfd5wcN+L6zQlK9lUCKiCQ732QnA/PdCuMcY50NbFbVxypNr3zc8ipgY9Vl/UFEmolI7On7eE7QbsSzzaY6s00F5rmRz/E/3woDZds5atpO84GbnatozgPyTx+K8BcRGQv8GpigqoWVpieISKhzvxuQAuzwZzbnd9f0d5wPTBaRSBFJdvKt8Hc+YAywRVWzT0/w97ar6fODhnzf+evKgcZ2w3PlwlY83xbudTnLCDy7luuBtc5tPPAqsMGZPh9o71K+bniukFkHbDq9vYA44CNgm/OzjUv5YoA8oGWlaa5sOzzF7ABQiueb3201bSc8hxiecd6DG4A0F7Jl4Tlefvp995wz7zXO33odsBq4wqVtV+PfEbjX2XaZwDh/Z3OmvwT8uMq8ft12tXx+NNj7zrpIMcYYUy92aMsYY0y9WCExxhhTL1ZIjDHG1IsVEmOMMfVihcQYY0y9WCEx5gxE5ITzM0lEbmjgdf+myuOlDbl+Y/zBCokx3ksCzqqQnG54Vov/KSSqesFZZjLGdVZIjPHeQ8CFzhgSd4pIqHjG61jpdBr4IwAR+Y4z/sM/8TToQkTedzq03HS6U0sReQiIdtb3ujPt9N6POOveKJ5xXiZVWvenIvK2eMYJed1puYyIPCQiGU6WR/2+dUzQCnM7gDGNyEw8Y19cDuAUhHxVHSoikcBXIrLImXcY0Fc9XZgD3KqqR0QkGlgpIu+o6kwRma6qA6v5XVfj6YhwABDvLPO589wgoA+e/o++AoaLSAaeLkJ6qaqKMwCVMf5geyTG1N2lePokWounW+44PP0mAayoVEQAfi4i6/CM6dG50nw1GQG8oZ4OCXOAz4ChldadrZ6OCtfiOeR2HDgFPC8iVwOF1azTGJ+wQmJM3QnwM1Ud6NySVfX0HsnJb2YS+Q6ezvvOV9UBwBogyot116S40v1yPCMYluHZC3oHzwBFC87qlRhTD1ZIjPFeAZ6hSk9bCPzE6aIbEenh9H5cVUvgqKoWikgvPMOXnlZ6evkqPgcmOedhEvAM5Vpj77XOWBMtVfU/wAw8h8WM8Qs7R2KM99YDZc4hqpeAJ/EcVlrtnPDOpfrhhBcAPxaR9Xh6ol1W6blZwHoRWa2qN1aa/h5wPp4eYhX4laoedApRdWKBeSIShWdv5s66vURjzp71/muMMaZe7NCWMcaYerFCYowxpl6skBhjjKkXKyTGGGPqxQqJMcaYerFCYowxpl6skBhjjKmX/w8jpha7Dcum5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Skip the code for now\n",
    "learn.fit_one_cycle(1, max_lr=1e-2)\n",
    "learn.recorder.plot_lr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have any question on why is the learning rate policy defined as this or in general to the learning rate policy in fastai, I have answered all the questions in the forums which you can check [here](https://forums.fast.ai/t/shedding-some-light-about-lr-management-in-fastai/43708). There I clearly explain why we use a learning rate policy that looks like the one shown above and what are the reasons behind choosing the deafults. (The main reason is we want to train our model at higher learning rates and then fine-tune them at lower learning rates).\n",
    "\n",
    "For the implementation of this, you can check [this](https://github.com/KushajveerSingh/fastai_without_fastai) where I implement the one-cycle policy in pure pytorch.\n",
    "\n",
    "### Difference from original paper\n",
    "There are some changes that you would notice from the original paper. First, in the papers for most of the cases accuracy is used as the metric for most of the graphs, while I use loss values. The reason loss values are used to compare hyper-parameter value is loss is the actual thing that is being optimized we want to reduce it the most."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-cycle policy summary\n",
    "There are two phases, first learning rate increases from small lr to maximum value and in the second phase it decreased from the maximum value to the minimum value (the minimum value is smaller than the starting value in the first phase). In implementation, you only need to define the maximum value of the learning rate and the minimum values would be calculated appropriately.\n",
    "\n",
    "### Learning rate finder test\n",
    "In order to find the maximum value of leanring rate (max_lr) we using a learning rate test. In this test we start with a small value of learning rate (1e-7) and then increase this value to a maximum of 10 as shown in the figure below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxddX3/8dd7tux7JhCyDUtYFCHIGMGtgKhIFdSiYrXuTWulgK2/uvweP+vSPqr99We1P62aohUpFXygbYNF/YGCaC3LAFkgYYlZSDIhM5nJTGZfP78/7hkYhpnJnSRn7r1z3s/H4z7mLN9zzufmTu5nznc5X0UEZmaWXWWFDsDMzArLicDMLOOcCMzMMs6JwMws45wIzMwyrqLQAUzU4sWLo6amptBhmJmVlIceeuhgRFSPtq/kEkFNTQ11dXWFDsPMrKRI2j3WPlcNmZllnBOBmVnGORGYmWWcE4GZWcalnggklUt6RNKPR9k3TdKtkrZLul9STdrxmJnZ803GHcF1wLYx9n0IOBQRpwF/D3xpEuIxM7NhUk0EkpYDvwvcMEaRK4Ebk+XbgNdKUpoxmZnZ86V9R/AV4C+AwTH2LwP2AEREP9AKLBpZSNI6SXWS6hobG9OK1cysaH31rqf41VPpfP+llggkvQloiIiHxis2yrYXTJAQEesjojYiaqurRx0YZ2Y2ZQ0MBl/9+ZM8sLM5lfOneUfwSuAKSbuAW4BLJP3LiDJ7gRUAkiqAeUA679TMrEQ1tfcwGLBkzrRUzp9aIoiIT0XE8oioAa4GfhER7xlRbAPwvmT5qqSMp0wzMxumoa0HgOqUEsGkP2tI0ueBuojYAHwbuEnSdnJ3AldPdjxmZsWu8dlEMD2V809KIoiIe4B7kuXPDNveDbx9MmIwMytVDW3dQAlWDZmZ2fHRcDjdqiEnAjOzItfY3sPc6RVMryxP5fxOBGZmRa7hcA9L5qbTPgBOBGZmRa+hrZvq2elUC4ETgZlZ0Wts72HJXCcCM7NMiohc1VBKDcXgRGBmVtTaevrp6R9MrccQOBGYmRW1oa6jS1IaTAZOBGZmRS3twWTgRGBmVtQaU37OEDgRmJkVtaFE4KohM7OMamjroaqijLkz0ns0nBOBmVkRa2zLdR1NcxZfJwIzsyLW0NadavsAOBGYmRW1tAeTgROBmVlRa2zvSbWhGNKdvH66pAckbZL0mKTPjVLm/ZIaJW1MXh9OKx4zs1LT0z9AS2df6lVDac5Q1gNcEhHtkiqBX0v6SUTcN6LcrRFxTYpxmJmVpOe6jpZoIkgmoW9PViuTlyemNzPL07OJIMUnj0LKbQSSyiVtBBqAOyPi/lGK/Z6kzZJuk7RijPOsk1Qnqa6xsTHNkM3MikbD0Kji2SXaRgAQEQMRsQZYDqyVdPaIIrcDNRFxDnAXcOMY51kfEbURUVtdXZ1myGZmRaNhKtwRDImIFuAe4LIR25sioidZ/Sfg/MmIx8ysFDS29SDBollVqV4nzV5D1ZLmJ8szgEuBx0eUWTps9QpgW1rxmJmVmsa2bhbNqqKiPN2/2dPsNbQUuFFSObmE84OI+LGkzwN1EbEBuFbSFUA/0Ay8P8V4zMxKSsPhHqpTHkMA6fYa2gycN8r2zwxb/hTwqbRiMDMrZbnBZOm2D4BHFpuZFa3cHYETgZlZJg0OBgd9R2Bmll2HOnvpHwwnAjOzrDpweGiKyvQbi50IzMyK0P7WLgBOmu9EYGaWSfWt3QCcNH9G6tdyIjAzK0L1LV1Ulovq2W4jMDPLpPqWLk6cN52ysvTmKh7iRGBmVoT2t3SzdF761ULgRGBmVpT2tXSxbBLaB8CJwMys6AwMBs8c7p6UHkPgRGBmVnQa23oYGAxXDZmZZdW+ltwYAlcNmZllVH3L0GAyJwIzs0waGlW81G0EZmbZVN/SzZxpFcydXjkp10tzqsrpkh6QtEnSY5I+N0qZaZJulbRd0v2SatKKx8ysVOxr6Zq0aiFI946gB7gkIs4F1gCXSbpgRJkPAYci4jTg74EvpRiPmVlJ2N/aNWnVQpBiIoic9mS1MnnFiGJXAjcmy7cBr5WU/nhqM7MiVt/SPWXuCJBULmkj0ADcGRH3jyiyDNgDEBH9QCuwaJTzrJNUJ6musbExzZDNzAqqq3eA5o7eSes6CikngogYiIg1wHJgraSzRxQZ7a//kXcNRMT6iKiNiNrq6uo0QjUzKwrP9hiaNwWqhoaLiBbgHuCyEbv2AisAJFUA84DmyYjJzKwY1bdM3jwEQ9LsNVQtaX6yPAO4FHh8RLENwPuS5auAX0TEC+4IzMyyon6SRxUDVKR47qXAjZLKySWcH0TEjyV9HqiLiA3At4GbJG0ndydwdYrxmJkVvfrWLiQ4Ye7kVQ2llggiYjNw3ijbPzNsuRt4e1oxmJmVmvqWLqpnT6OqYvLG+3pksZlZEZnsrqPgRGBmVlTqW7smbR6CIU4EZmZFIiKob+nipEmah2CIE4GZWZE41NlHd9+gq4bMzLLquXkIXDVkZpZJkz0hzRAnAjOzIrH30OQPJgMnAjOzorG7qYM50ypYOKtqUq/rRGBmViR2N3eyctFMJvtp/E4EZmZFYndTJzWLZk36dZ0IzMyKQP/AIHuaO1m1aOakXzuvRCDpVZI+kCxXSzo53bDMzLJlf2s3/YNRnIlA0l8CnwA+lWyqBP4lzaDMzLJmV1MHAKuKtGrorcAVQAdARNQDc9IMyswsa3Y1dQIUbRtBbzJZTABImvwozcymuKebOphWUcaSOdMm/dr5JIIfSPoWMF/SHwJ3ATekG5aZWbbsaso1FJeVTW7XUcgjEUTE3wG3AT8EzgA+ExH/cKTjJK2QdLekbZIek3TdKGUuktQqaWPy+sxo5zIzm+qebuosSPsA5DFDmaQvRcQngDtH2TaefuDPI+JhSXOAhyTdGRFbR5T7VUS8acKRm5lNEYODwe7mDl69enFBrp9P1dDrRtn2xiMdFBH7I+LhZLkN2AYsm1h4ZmZTX0NbD919g6xaXJg7gjETgaSPSNoCnCFp87DXTmDzRC4iqYbc/MX3j7L7QkmbJP1E0ovHOH6dpDpJdY2NjRO5tJlZ0duddB2tKcAYAhi/auhfgZ8AfwN8ctj2tohozvcCkmaTa1+4PiIOj9j9MLAqItolXQ78O7B65DkiYj2wHqC2tjbyvbaZWSnYnXQdXbWwyO4IIqI1InZFxLsiYjfQRa4L6WxJK/M5uaRKckng5oj40SjXOBwR7cnyHUClpMJUkpmZFciupg4qyjTpE9IMyWdk8ZslPQXsBH4J7CJ3p3Ck4wR8G9gWEV8eo8yJSTkkrU3iaco7ejOzKWB3cycrFs6korwwj387Yq8h4K+AC4C7IuI8SRcD78rjuFcCfwBskbQx2fZpYCVARHwTuAr4iKR+cnccVyeD18zMMmN3UwcrFxamfQDySwR9EdEkqUxSWUTcLelLRzooIn4NjDsyIiK+Bnwtz1jNzKaciGB3Uyfnr1xQsBjySQQtSYPvvcDNkhrIjREwM7NjdKizj7bu/oINJoP8xhFcCXQCHwN+CvwWeHOaQZmZZcVzTx0t4qqhiOhIFgeBGyWVA1cDN6cZmJlZFjw91HW0GO8IJM2V9ClJX5P0euVcA+wA3jF5IZqZTV27mjqQYMXCGQWLYbw7gpuAQ8B/Ax8G/gdQBVwZERvHOc7MzPK0o7GDk+bNYFpFecFiGC8RnBIRLwGQdANwEFiZPDfIzMyOgycPtHH6CbMLGsN4jcV9QwsRMQDsdBIwMzt++gcG2dHYweknFHbSx/HuCM6VNPRsIAEzknUBERFzU4/OzGwK293cSe/AIKuLNRFEROEqrMzMMuCpA+0ARV01ZGZmKXrqQK62/bQlTgRmZpn0ZEM7KxbOYGZVPg95SI8TgZlZgTx1oI3VSwrbPgBOBGZmBTHUY2h1gdsHIL/J69vITUgzXCtQR25y+h1pBGZmNpXtasr1GDq9CO4I8qmY+jJQT27qSpF7ztCJwBPAd4CL0grOzGyqGmooLvQYAsivauiyiPhWRLQlU0uuBy6PiFuBMR+gLWmFpLslbZP0mKTrRikjSf8gabukzZJeegzvxcysZDzV0I5U+B5DkF8iGJT0jqGJaSQNf+DceLOJ9ZOrOjqL3AxnH5X0ohFl3khusvrVwDrgGxOI3cysZD15oI0VC2Yyo6rwQ7bySQTvJjflZANwIFl+j6QZwDVjHRQR+yPi4WS5DdgGLBtR7Erge5FzHzBf0tKJvw0zs9Ly1IF2VhfB3QDkNx/BDsaeiObX+VxEUg1wHnD/iF3LgD3D1vcm2/bnc14zs1LUNzDIjoPtXHzmkkKHAuTXa6ga+EOgZnj5iPhgPhdIprn8IXB9RBweuXuUQ15Q3SRpHbmqI1auXJnPZc3Mitbupg76BqLgj5YYkk+vof8AfgXcBQxM5OSSKsklgZsj4kejFNkLrBi2vpxcD6XnSRqo1wPU1taO1y5hZlb0nnz2GUOF7zEE+SWCmRHxiYmeWJKAbwPbIuLLYxTbAFwj6Rbg5UBrRLhayMymtKcO5HoMnVpdOncEP5Z0eUTcMcFzv5Jcw/IWSUMzmn0aWAkQEd8E7gAuB7YDncAHJngNM7OS82RDGysXFkePIcgvEVwHfFpSD7nJavKajyAifs3obQDDywTw0TxjNTObEh7ff7hoqoUgj+6jETEnIsoiYkZEzE3WPSmNmdlRaO/pZ8fBDl6ybF6hQ3nWmHcEks6MiMfHGu07NEbAzMzy99i+ViIojUQA/Bm5Lpv/Z5R9AVySSkRmZlPYln2tAJxdCokgItYlPy+evHDMzKa2LftaWTpvOtVzphU6lGflNS2OpFfwwgFl30spJjOzKWvL3taiqhaC/EYW3wScCmzkuQFlATgRmJlNQFt3HzsOdvDW80Y+dq2w8rkjqAVelHT1NDOzo/TovtxTdl6yvLjuCPJ5+uij5CaiMTOzY/Bo0lBcclVDwGJgq6QHgJ6hjRFxRWpRmZlNQZv3tbJs/gwWzS6ehmLILxF8Nu0gzMyy4NF9rZy9rPjG446bCCSVA/8rIi6dpHjMzKakw9197DzYwVXnLy90KC8wbhtBRAwAnZKKq0LLzKzEPFqEA8mG5FM11E3uCaJ3Ah1DGyPi2tSiMjObYrbsLc6GYsgvEfxn8jIzs6O0JWkoXjirqtChvEA+cxbfOBmBmJlNZVv2tXJOkY0fGHLEcQSSVku6TdJWSTuGXpMRnJnZVNDU3sPupk7OWT6/0KGMKp8BZf8MfAPoBy4m92iJm450kKTvSGqQ9OgY+y+S1CppY/L6zEQCNzMrFQ/uagZg7ckLCxzJ6PJJBDMi4ueAImJ3RHyW/B5B/V3gsiOU+VVErElen8/jnGZmJef+nc1MrywryoZiyLPXkKQy4ClJ1wD7gCVHOigi7pVUc2zhmZmVvgd2NvPSlQuoqsjnb+/Jl09U1wMzgWuB84H3AO87Tte/UNImST+R9OKxCklaJ6lOUl1jY+NxurSZWfoOd/exdf/hoq0Wgvx6DT0IICki4gPH8doPA6siol3S5cC/A6vHiGE9sB6gtrbWT0E1s5Lx0K5DRMDamuJNBPn0GrpQ0lZgW7J+rqR/PNYLR8ThiGhPlu8AKiUtPtbzmpkVk/t3NlNRJs5buaDQoYwpn6qhrwBvAJoAImIT8JpjvbCkEyUpWV6bxNJ0rOc1MysmD+xs4pzl85hRVV7oUMaU11SVEbEn+c4eMjBW2SGSvg9cBCyWtBf4S6AyOd83gauAj0jqB7qAqz35jZlNJV29A2ze28qHX31KoUMZVz6JYE8yZ3FIqiLXaLztSAdFxLuOsP9rwNfyitLMrAQ98vQh+geDlxdxQzHkVzX0x8BHgWXAXmAN8CdpBmVmNhXcv7MZCc6vKd72Aciv19BB4N3Dt0m6nlzbgZmZjeGBnc28aOlc5k6vLHQo4zra0Q1/dlyjMDObYnr7B3n46UNFPX5gyNEmAh25iJlZdm3c00JP/2DRtw/A0ScC9+4xMxvH3U80UFEmXnFa8Q+PGrONQFIbo3/hC5iRWkRmZlPA3Y83UFuzoOjbB2CcO4KImBMRc0d5zYmIvMYfmJllUX1LF48/08bFZxzx+ZxFoTgfhWdmVsLufqIBgEvOdCIwM8ukux9vZNn8GZy2ZHahQ8mLE4GZ2XHU3TfAf20/yCVnLmHEo3mKlhOBmdlx9MDOZrr6Brj4zOpCh5I3JwIzs+PoF483MK2ijAtPKf5uo0OcCMzMjqN7nmjgwlMXFfVjp0dyIjAzO052HuxgV1NnyfQWGuJEYGZ2nNy59RmAkhk/MMSJwMzsONmwqZ5zl89jxcKZhQ5lQlJLBJK+I6lB0qNj7Jekf5C0XdJmSS9NKxYzs7Rtb2jn0X2HefO5JxU6lAlL847gu8Bl4+x/I7A6ea0DvpFiLGZmqdqwqR4JJ4LhIuJeoHmcIlcC34uc+4D5kpamFY+ZWVoigts31XPByYs4Ye70QoczYYVsI1gG7Bm2vjfZ9gKS1kmqk1TX2Ng4KcGZmeVry75Wdh7s4Mo1pXc3AIVNBKONvR51noOIWB8RtRFRW11dOqP1zCwbNmysp7JcvPHs0qzUKGQi2AusGLa+HKgvUCxmZkdlYDC4fXM9v3P6EubNLP65B0ZTyESwAXhv0nvoAqA1IvYXMB4zswl7YGczBw73lGy1EIwzQ9mxkvR94CJgsaS9wF8ClQAR8U3gDuByYDvQCXwgrVjMzNLyo4f3MrOqnEvPOqHQoRy11BJBRLzrCPsD+Gha1zczS1tLZy8bNtXztpcuL6lnC43kkcVmZkfpB3V76Okf5L0Xrip0KMfEicDM7CgMDAb/ct/TrK1ZyFlL5xY6nGPiRGBmdhR++WQDTzd38t5XlPbdADgRmJkdle/9926WzJnGG158YqFDOWZOBGZmE7TrYAf3PNHI7798JZXlpf81WvrvwMxskt10324qysTvr11Z6FCOCycCM7MJaO7o5ZYHnubylyxlSQk+YG40TgRmZhPwT7/aQWffAH96yWmFDuW4cSIwM8tTU3sPN/5mF28+5yRWnzCn0OEcN04EZmZ5Wn/vDrr7Brj2tasLHcpx5URgZpaHxrYebvzvXVy5ZhmnLZld6HCOKycCM7M8fOuXv6W3f3BKtQ0McSIwMzuCPc2d3HTfbt5y3jJOqZ5adwPgRGBmdkSfu30r5WXi468/o9ChpMKJwMxsHHduPcBd2w5w3WtXc9L8GYUOJxVOBGZmY+jqHeCzGx5j9ZLZfPBVJxc6nNSkmggkXSbpCUnbJX1ylP3vl9QoaWPy+nCa8ZiZTcTX797OvpYuvvCWs6fEM4XGkuZUleXA14HXkZuo/kFJGyJi64iit0bENWnFYWZ2NJ54po1v3ftb3nbeMi44ZVGhw0lVmiluLbA9InZERC9wC3BlitczMzsuunoH+NPvP8y8GZV8+nfPKnQ4qUszESwD9gxb35tsG+n3JG2WdJukFaOdSNI6SXWS6hobG9OI1czsWV/4z608eaCdL79jDYtnTyt0OKlLMxFolG0xYv12oCYizgHuAm4c7UQRsT4iaiOitrq6+jiHaWb2nJ9s2c+/3v80f/Q7p/Ca07PxfZNmItgLDP8LfzlQP7xARDRFRE+y+k/A+SnGY2Y2rj3NnXzih5s5d8X8KTtmYDRpJoIHgdWSTpZUBVwNbBheQNLSYatXANtSjMfMbEytnX188LsPEsD/vfq8Kd1LaKTUeg1FRL+ka4CfAeXAdyLiMUmfB+oiYgNwraQrgH6gGXh/WvGYmY2lp3+AdTfVsaupgxs/uJaVi2YWOqRJpYiR1fbFrba2Nurq6godhplNEYODwfW3bmTDpnq+8s41vOW80fq0lD5JD0VE7Wj7snPvY2Y2QkTwhf/cyoZN9fyPN5wxZZPAkaRWNWRmVswGBoP/+W9buOXBPXzglTX8yUWnFjqkgnEiMLPM6RsY5M9/sIkNm+r56MWn8vHXn4E0Wo/3bHAiMLNMae3q4/pbHuHuJxr5xGVn8pEM3wkMcSIws8x44pk2/uimOvYe6uKv33o27375qkKHVBScCMwsEzZsqucTt21m9vQKvr/uAl5Ws7DQIRUNJwIzm9Ka2nv47O1buX1TPbWrFvCP734pS+ZOL3RYRcWJwMympIjg9s37+eyGx2jr7uPPXnc6H7no1EyNGM6XE4GZTTmPPH2Iv7njcR7Y1cy5y+fxt1ddwBknzil0WEXLicDMpownnmnjqz9/kju2PMPi2dP4wlvO5l0vW0GF7wLG5URgZiUtIvjNb5tYf+8OfvlkIzOryrn+0tX84atPYdY0f8Xlw/9KZlaSGg5382+P7OO2h/byVEM7i2dP4+OvP513v3wVC2ZVFTq8kuJEYGYlo6Gtm7u2NvCzx57h19sPMjAYnL9qAX/7e+dwxZqTmF5ZXugQS5ITgZkVrZ7+AR7e3cJvfnuQX28/yMY9LUTAyoUzWfeaU7jq/OWcWj270GGWPCcCMysKg4PBnkOdPLrvMBv3HOKRp1vYsq+Vnv5BysvES5bN42OXns4bXnwip58wO9PPBjrenAjMbFJ19PSz91AXu5o62Hmwgx2N7TzV0M4Tz7TR2TsAQFVFGS9ZNo8/uGAVF5yyiLWnLGTu9MoCRz51pZoIJF0GfJXcDGU3RMQXR+yfBnyP3FzFTcA7I2JXmjGZ2fE3MBi0dvXR0tnLoc5eDrb30tTey8H2Hg4c7ubA4dzPfS1dNHf0Pu/YxbOncWr1LN5Ru4Kzls7hrKVzOfPEuVRVuMvnZEktEUgqB74OvI7cRPYPStoQEVuHFfsQcCgiTpN0NfAl4J1pxWQ21UQEg5H7Ih6MYGAwGIhgcDDoH3zuZ/9A0D84SP9g0Dcw+Ox6b39uvbd/kJ7+QXoHBujpG6S7b4Du/tzPrr4BunoH6OwdoLO3n46eATp6+mnv6aetu5/DXX209fSPGeOCmZWcMHc6J86bzjnL57F8wUyWLZhBzaKZ1Cye5b/0i0CadwRrge0RsQNA0i3AlcDwRHAl8Nlk+Tbga5IUKcyf+csnG/mrH289ckGjFCYvPZpfkTGPiHFXR71mPLsNYsQREbnXyGNjWPncz+Hlhm/LfblHPLdtMCJ5Pbd/MNmftqqKMmZUljOzKveaPa2CWdMqWDlrJnOmVzJ3RgVzp1eyYGYl82dWMW9mJdWzp7FodhULZ1UxrcI9eYpdmolgGbBn2Ppe4OVjlUkmu28FFgEHhxeStA5YB7By5cqjCmb2tApWn+DeBfkSJdAQdxQhjnXIyIbHscu9sMzQsc87Rs/9Gw4do2RZKPdTua1D+8uG7SvTc8eWSbl90vPWyyUkUV6WWy8rE+XJ+tCrTKKyXJSXlVFRJirKRUVZGZXlorK8jMryMqoqcsvTKsqpqihjWkUZ0yvLn/1ZXlYCvwt2TNJMBKP99oz8+yWfMkTEemA95CavP5pgzl+1gPNXnX80h5qZTWlptsbsBVYMW18O1I9VRlIFMA9oTjEmMzMbIc1E8CCwWtLJkqqAq4ENI8psAN6XLF8F/CKN9gEzMxtbalVDSZ3/NcDPyHUf/U5EPCbp80BdRGwAvg3cJGk7uTuBq9OKx8zMRpfqOIKIuAO4Y8S2zwxb7gbenmYMZmY2Po/YMDPLOCcCM7OMcyIwM8s4JwIzs4xTqfXWlNQI7D7KwxczYtRyxmT9/YP/Dfz+s/v+V0VE9Wg7Si4RHAtJdRFRW+g4CiXr7x/8b+D3n+33PxZXDZmZZZwTgZlZxmUtEawvdAAFlvX3D/438Pu3F8hUG4GZmb1Q1u4IzMxsBCcCM7OMy0wikHSZpCckbZf0yULHkzZJKyTdLWmbpMckXZdsXyjpTklPJT8XFDrWNEkql/SIpB8n6ydLuj95/7cmj0ifkiTNl3SbpMeT34MLs/T5S/pY8rv/qKTvS5qepc9/IjKRCCSVA18H3gi8CHiXpBcVNqrU9QN/HhFnARcAH03e8yeBn0fEauDnyfpUdh2wbdj6l4C/T97/IeBDBYlqcnwV+GlEnAmcS+7fIROfv6RlwLVAbUScTe5R+FeTrc8/b5lIBMBaYHtE7IiIXuAW4MoCx5SqiNgfEQ8ny23kvgSWkXvfNybFbgTeUpgI0ydpOfC7wA3JuoBLgNuSIlP2/UuaC7yG3JwfRERvRLSQoc+f3GP2ZySzH84E9pORz3+ispIIlgF7hq3vTbZlgqQa4DzgfuCEiNgPuWQBLClcZKn7CvAXwGCyvghoiYj+ZH0q/x6cAjQC/5xUjd0gaRYZ+fwjYh/wd8DT5BJAK/AQ2fn8JyQriUCjbMtEv1lJs4EfAtdHxOFCxzNZJL0JaIiIh4ZvHqXoVP09qABeCnwjIs4DOpii1UCjSdo+rgROBk4CZpGrGh5pqn7+E5KVRLAXWDFsfTlQX6BYJo2kSnJJ4OaI+FGy+YCkpcn+pUBDoeJL2SuBKyTtIlcVeAm5O4T5SVUBTO3fg73A3oi4P1m/jVxiyMrnfymwMyIaI6IP+BHwCrLz+U9IVhLBg8DqpMdAFblGow0FjilVSX34t4FtEfHlYbs2AO9Llt8H/MdkxzYZIuJTEbE8ImrIfd6/iIh3A3cDVyXFpvL7fwbYI+mMZNNrga1k5PMnVyV0gaSZyf+Fofefic9/ojIzsljS5eT+IiwHvhMRf13gkFIl6VXAr4AtPFdH/mly7QQ/AFaS+8/y9ohoLkiQk0TSRcDHI+JNkk4hd4ewEHgEeE9E9BQyvrRIWkOuobwK2AF8gNwff5n4/CV9DngnuR50jwAfJtcmkInPfyIykwjMzGx0WakaMjOzMTgRmJllnBOBmVnGORGYmWWcE4GZWcY5EVjmSGpPftZI+v3jfO5Pj1j/zfE8v1kanAgsy2qACSWC5Em243leIoiIV0wwJrNJ50RgWfZF4NWSNibPri+X9L8lPShps6Q/gtyAtGRuh38lN0APSf8u6aHkeffrkm1fJPe0y42Sbk62Dd19KDn3o5K2SHrnsHPfM2zegJuTkbBI+qKkrUksfzfp/zqWGRVHLmI2ZX2SZMQxQPKF3hoRL5M0DfgvSf8vKbsWODsidibrHzAp7t8AAAGJSURBVIyIZkkzgAcl/TAiPinpmohYM8q13gasITcvwOLkmHuTfecBLyb33Jv/Al4paSvwVuDMiAhJ84/7uzdL+I7A7DmvB94raSO5R3EsAlYn+x4YlgQArpW0CbiP3AMNVzO+VwHfj4iBiDgA/BJ42bBz742IQWAjuSqrw0A3cIOktwGdx/zuzMbgRGD2HAF/GhFrktfJETF0R9DxbKHcs4suBS6MiHPJPbNmeh7nHsvwZ90MABXJM/PXknt67FuAn07onZhNgBOBZVkbMGfY+s+AjySP70bS6clkLiPNAw5FRKekM8lNBTqkb+j4Ee4F3pm0Q1STmz3sgbECS+aRmBcRdwDXk6tWMkuF2wgsyzYD/UkVz3fJzfFbAzycNNg2MvpUhj8F/ljSZuAJctVDQ9YDmyU9nDz2esi/ARcCm8hNhvIXEfFMkkhGMwf4D0nTyd1NfOzo3qLZkfnpo2ZmGeeqITOzjHMiMDPLOCcCM7OMcyIwM8s4JwIzs4xzIjAzyzgnAjOzjPv/VI4fsI+taSEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot_lr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fastai this test can be done using `learn.lr_find`. After running this test we plot a diagram of the loss values for the different values of learning rate and try to find the maximum value of learning rate that we can use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find(start_lr=1e-7, end_lr=10, num_it=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de3zkdX3v8dcn93uyu0n2foG9QEGFhQDF9SAgUkS61lYrHulBpeVoW6H1Qg/VQz14rFXbitrjZYulKqIPQalIVaDilqqwsMu6y22vZBfC7ibZTTbJTJK5fs4f80s2hMlu2OQ3M8m8n4/HPPZ3+c78PvPbyXzm+/3+ft+vuTsiIlK8SvIdgIiI5JcSgYhIkVMiEBEpckoEIiJFTolARKTIKRGIiBS5sjBf3Mz2AQNACki6e9u4/e8B/ipYjQAfdPdtYcYkIiIvF2oiCFzi7ocn2NcOvNHde83sLcAG4IIcxCQiIoFcJIIJufuvx6w+BizJVywiIsUq7ETgwINm5sDX3X3DccpeB/z0RC/Y3NzsK1asmKbwRESKw5YtWw67e0u2fWEngnXufsDMWoGHzGyHuz8yvpCZXUImEbwh24uY2fXA9QDLli1j8+bNYcYsIjLrmNn+ifaFetWQux8I/u0C7gXOH1/GzF4H3A68zd2PTPA6G9y9zd3bWlqyJjQRETlJoSUCM6s1s/qRZeBy4OlxZZYBPwT+yN13hRWLiIhMLMymofnAvWY2cpy73P1nZvYBAHf/GnALMA/4SlDuFZeYiohIuEJLBO7+PHBWlu1fG7P8x8AfhxWDiIicmO4sFhEpckoEIiJFTolARKTIKRGIiMwAt/3HLh7Z1R3KaysRiIgUOHfnyw/vYVN71lutpkyJQESkwMWSaVJpp6YinAs9lQhERApcJJYEoK5SiUBEpChFg0RQq0QgIlKcjtUISkN5fSUCEZECNxhPAaoRiIgUrZEagTqLRUSKVFSdxSIixe1YZ7H6CEREilIklukjUI1ARKRI6fJREZEiF40nqSgrobw0nK9sJQIRkQIXjSWprQinfwCUCERECl40lgqtWQiUCERECl4klgytoxiUCERECl40llSNQESkmEXjahoSESlq0VgytAHnQIlARKTgRWPJ0MYZgpATgZntM7OnzOw3ZrY5y34zsy+Z2R4z225m54QZj4jITBR2Z3F4r3zMJe5+eIJ9bwFWB48LgK8G/4qICJn5ijOdxbO3aehtwLc84zGgycwW5jkmEZGCMZxIk/bwhpeA8BOBAw+a2RYzuz7L/sXAi2PWO4JtIiJCZngJCG/AOQi/aWidux8ws1bgITPb4e6PjNlvWZ7j4zcESeR6gGXLloUTqYhIARodcG6mdha7+4Hg3y7gXuD8cUU6gKVj1pcAB7K8zgZ3b3P3tpaWlrDCFREpOJGQ5yKAEBOBmdWaWf3IMnA58PS4YvcB/yO4eui3gT53PxhWTCIiM000Fu58xRBu09B84F4zGznOXe7+MzP7AIC7fw34CXAlsAcYBN4XYjwiIjNO2HMRQIiJwN2fB87Ksv1rY5Yd+LOwYhARmekiIc9XDPm/fFRERI5jMB5+jUCJQESkgI3OVzxTrxoSEZGpGekjqJmJVw2JiMjURWPhzlcMSgQiIgUt7AHnQIlARKSgDcZTod5MBkoEIiIFLRJLhjq8BCgRiIgUtKiahkREils0lqRGiUBEpHhFQp6vGJQIREQKWjSWUh+BiEgxi8aToQ4vAUoEIiIFa2S+YnUWi4gUqZH5isMcXgKUCEREClYuhqAGJQIRkYKVi/mKQYlARKRgRXMwFwEoEYiIFKyR+YrVNCQiUqSOzVeszmIRkaIUycHE9aBEICJSsKJKBCIixW308lFdNSQiUpwG45nO4hnfR2BmpWa21czuz7JvmZn9Iti/3cyuDDseEZGZIhpLUllWQlmI8xVDbmoENwLPTbDvE8D33X0tcDXwlRzEIyIyI+RivmIIORGY2RLgrcDtExRxoCFYbgQOhBmPiMhMkpmUJtxmIYCwU81twE1A/QT7Pwk8aGYfAmqBy0KOR0RkxojkYC4CCLFGYGZXAV3uvuU4xd4N/Ku7LwGuBL5tZq+IycyuN7PNZra5u7s7pIhFRApLLoaghnCbhtYB681sH/A94FIzu3NcmeuA7wO4+6NAFdA8/oXcfYO7t7l7W0tLS4ghi4gUjsEcTEoDISYCd7/Z3Ze4+woyHcEPu/s144q9ALwJwMx+i0wi0E9+ERFmSWdxNmZ2q5mtD1Y/AvyJmW0Dvgu819091zGJiBSiaCwV+j0EEH5nMQDuvhHYGCzfMmb7s2SakEREZJxoLEnNTO4sFhGRk+fuROOztGlIRERObGS+4hndWSwiIifv2HzF4fcRKBGIiBSgXA1BDUoEIiIFKVeT0oASgYhIQRqtEeiqIRGR4hSN52a+YlAiEBEpSNFYZlIaXT4qIlKk1FksIlLk1FksIlLkRpqGaivURyAiUpSi8dzMVwxKBCIiBSlXQ1CDEoGISEEajOVmUhpQIhARKUiRWEqJQESkmGXmKw6/oxiUCERECtLRoQSN1eU5OZYSgYhIATrYN8SCxqqcHEuJQESkwAzFUxwdTLCwsTonx1MiEBEpMIf6hwFYqBqBiEhxOnh0CEBNQyIixepg30iNQE1DIiJFSU1DIiJF7mDfEHNqyqkqnyX3EZhZqZltNbP7J9j/h2b2rJk9Y2Z3hR2PiEihO3h0mAU5ahYCyMX9yzcCzwEN43eY2WrgZmCdu/eaWWsO4hERKWgH+4ZZlKNmIQi5RmBmS4C3ArdPUORPgP/n7r0A7t4VZjwiIjPBof7hnF0xBOE3Dd0G3ASkJ9i/BlhjZr8ys8fM7IpshczsejPbbGabu7u7w4pVRCTvhhMpeqLxnHUUQ4iJwMyuArrcfctxipUBq4GLgXcDt5tZ0/hC7r7B3dvcva2lpSWUeEVECsGh4NLRXPYRhFkjWAesN7N9wPeAS83sznFlOoAfuXvC3duBnWQSg4hIURq5h2BW9BG4+83uvsTdVwBXAw+7+zXjiv0bcAmAmTWTaSp6PqyYREQK3cG+3N5VDJNMBGa20swqg+WLzeyGbE04k3ytW81sfbD6AHDEzJ4FfgF8zN2PnMzriojMBrm+qxgmf/noD4A2M1sFfAO4D7gLuHIyT3b3jcDGYPmWMdsd+HDwEBEpeof6hmmqKae6Ijc3k8Hkm4bS7p4E3g7c5u5/CSwMLywRkeJ0sG+YBQ25axaCySeChJm9G7gWGLlDODdT54iIFJGDfUM5vXQUJp8I3gdcCHza3dvN7BRg/BVAIiIyRYf6hlnYlLv+AZhkH4G7PwvcAGBmc4B6d/+7MAMTESk2w4kUR6JxFhZi05CZbTSzBjObC2wD7jCzfww3NBGR4tLZP3IzWQEmAqDR3fuB3wfucPdzgcvCC0tEpPiM3kyW46ahySaCMjNbCPwhxzqLRURkGh0bXqIwawS3krn5a6+7P2FmpwK7wwtLRKT4jNQIcn356GQ7i+8G7h6z/jzwB2EFJSJSjA72DdFQVUZtZS6mijlmsp3FS8zsXjPrMrNOM/tBMNeAiIhMk4N9wznvH4DJNw3dQWZYiUXAYuDHwTYREZkmh/pyOyHNiMkmghZ3v8Pdk8HjXwFNDCAiMo3ycVcxTD4RHDaza4KJ6EvN7BpAo4SKiEyTWDLF4Ug8p6OOjphsIng/mUtHDwEHgXeQGXZCRESmQVd/DMj9paMwyUTg7i+4+3p3b3H3Vnf/PTI3l4mIyDQ4Ng9BgSaCCWgOARGRaTIyM1khNw1lY9MWhYhIkTuYp7uKYWqJwKctChGRInfg6BD1VWXU5fhmMjjBncVmNkD2L3wDcl9/ERGZpfZ0RVjZUpeXYx83Ebh7fa4CEREpZrs6I1xyWn5uz5pK05CIiEyD3micw5EYa+bn57e3EoGISJ7t6hwAYPX8/DQNKRGIiOTZrq4IwOytEQRDUmw1swkntDGzd5iZm1lb2PGIiBSaPZ0D1FWW5eVmMshNjeBG4LmJdppZPXADsCkHsYiIFJxdnRFWtdZhlp/bs0JNBMGcBW8Fbj9OsU8BnwOGw4xFRKRQ7e4aYE2e+gcg/BrBbcBNQDrbTjNbCyx19+POg2xm15vZZjPb3N3dHUKYIiL50RONczgSz1v/AISYCMzsKqDL3bdMsL8E+ALwkRO9lrtvcPc2d29radE0CCIyexy7YmgWJgJgHbDezPYB3wMuNbM7x+yvB14DbAzK/DZwnzqMRaSY7B5JBK2zsGnI3W929yXuvgK4GnjY3a8Zs7/P3ZvdfUVQ5jFgvbtvDismEZFCs7srQn0erxiCPNxHYGa3mtn6XB9XRKQQ7eocYNX8/F0xBCcYa2i6uPtGYGOwfMsEZS7ORSwiIoVkd2eEy35rfl5j0J3FIiJ5ciQS40g0nrehJUYoEYiI5MmuzszQEvm8YgiUCERE8mZPV+aKoXzeTAZKBCIiebOrM3PF0IKG/F0xBEoEIiJ5s6tzgNV5vmIIlAhERPJmd1eE1a35nwhSiUBEJA8OR2L0FMAVQ6BEICKSFyNjDOVzsLkRSgQiInnw7IF+AE5foEQgIlKUntjXw7K5NbTm+YohUCIQEck5d+fx9h7OWzE336EASgQiIjm3pytC72CCC05RIhARKUqb2nsAOF+JQESkOD2xr4fW+kqWz6vJdyiAEoGISE65O5ue7+G8U+bm/Y7iEUoEIiI51NE7xKH+4YLpHwAlAhGRnCq0/gFQIhARyakn2ntorC5nTQGMMTRCiUBEJIce39fDeSvmUFJSGP0DoEQgIpIzXf3DtB+OFlSzECgRiIjkzOP7RvoH5uU5kpdTIhARyZEn2nuoqSjlzEUN+Q7lZZQIRERyZFN7D+csm0N5aWF99YYejZmVmtlWM7s/y74Pm9mzZrbdzH5uZsvDjkdEJB+ODsbZ2TlQcP0DkJsawY3AcxPs2wq0ufvrgHuAz+UgHhGRnPv2o/txh0tOa813KK8QaiIwsyXAW4Hbs+1391+4+2Cw+hiwJMx4RETyoTcaZ8Mjz/PmM+bz2iWN+Q7nFcKuEdwG3ASkJ1H2OuCn2XaY2fVmttnMNnd3d09nfCIiofvqf+4lEk/ysd85Ld+hZBVaIjCzq4Aud98yibLXAG3A57Ptd/cN7t7m7m0tLS3THKmISHgO9g3xr7/ex9vXLi6I+YmzKQvxtdcB683sSqAKaDCzO939mrGFzOwy4OPAG909FmI8IiI596Wf78bd+cvL1uQ7lAmFViNw95vdfYm7rwCuBh7OkgTWAl8H1rt7V1ixiIjkw97uCN/f3MF7LljO0rmFMfdANjm/mNXMbjWz9cHq54E64G4z+42Z3ZfreEREwvKPD+6isqyEP790Vb5DOa4wm4ZGuftGYGOwfMuY7Zfl4vgiIrn2xL4e/v2pg9xw6Sqa6yrzHc5xFdbtbSIis0AyleZ//9vTLGqs4gMXr8x3OCekRCAiMs2+s+kFdhwa4BNXnUFNRU4aXqZEiUBEZBodjsT4+wd38oZVzbzlNQvyHc6kKBGIiEyjz/50B0PxFJ9cf0bBTE5/IkoEIiLT5MkXerl7SwfXveEUVhXQVJQnokQgIjINhhMp/vqHTzG/oZIPvWl1vsN5VQq/F0NEZAb4zE+eY8ehAf7lvW3UVc6sr1bVCEREpuiBZw7xzUf3c90bTuHS0+fnO5xXTYlARGQKXjo6xE33bOe1ixu56YrCHF30RJQIREROUjKV5sbvbiWVdr787rVUlpXmO6STMrMaskRECoS785mf7mDz/l6+ePXZrGiuzXdIJ001AhGRk/Dlh/fwjV+2c+2Fy3nb2YvzHc6UKBGIiLxK3/hlO//40C5+/5zF/M3vnpnvcKZMiUBE5FX47uMv8Kn7n+Utr1nA5/7gdZSUzIy7h49HiUBEZJJ+sKWDv773KS4+rYUvXr2WstLZ8RU6O96FiEjIvrNpPx+5exuvXzmPr77nXCrKZs/Xp64aEhE5gW/8sp1P3f8sl57eylfecw5V5TPzMtGJKBGIiEzA3fnKxr18/oGdXPnaBdz2rrWzqiYwQolARCSLwXiST/zb0/zwyZf4/bWL+dw7Xjdr+gTGUyIQERlnT9cAH7zzSfZ0R/iLy1Zzw6WrZ8XVQRNRIhARCaTTzg+3vsQtP3qamopS7rzuAtatas53WKFTIhCRohdLpvjR1gN8/ZG97O2Ocv6KuXz5v69lfkNVvkPLCSUCESlaiVSabz+6n6/95166BmKcuaiBL717LVe+ZsGs7Q/IJvREYGalwGbgJXe/aty+SuBbwLnAEeBd7r4v7JhERJ7Y18Mn7n2anZ0DvH7lPP7hD8/iDauaZ8w8w9MpFzWCG4HngIYs+64Det19lZldDXwWeFcOYhKRInUkEuMzP93BPVs6WNxUzYY/Opc3nzG/KBPAiFATgZktAd4KfBr4cJYibwM+GSzfA/yTmZm7e5hxiUhx+sXOLj529zb6hhJ88OKVfOjSVdRUqIU87DNwG3ATUD/B/sXAiwDunjSzPmAecHhsITO7HrgeYNmyZaEFKyKz03AixWd+8hzffHQ/py+o584/voDTF2RrpChOoSUCM7sK6HL3LWZ28UTFsmx7RW3A3TcAGwDa2tpUWxCRSXF3Htl9mP97/7Ps7orw/nWncNMVp826ISKmKswawTpgvZldCVQBDWZ2p7tfM6ZMB7AU6DCzMqAR6AkxJhEpAoPxJD988iXu+FU7e7ujLGio4lvvP5+L1rTkO7SCFFoicPebgZsBghrBR8clAYD7gGuBR4F3AA+rf0BETlY0luSOX7Xzz//VTt9QgtcubuQL7zqLt7520awcI2i65LyXxMxuBTa7+33AN4Bvm9keMjWBq3Mdj4jMfMOJFHc+tp+vbtzLkWicN53eygcvXsm5y+cU9dVAk5WTRODuG4GNwfItY7YPA+/MRQwiMvt09Q9z1+MvcNemF+gaiLFu1Tw+cvlpnLNsTr5Dm1F03ZSIzCjxZJpN7Uf43uMv8sAzh0imnYvWtHDbu07l9UUwLlAYlAhEpOAdHYzz8I4ufv5cF4/s6mYglqSxupz3rVvBey5Yzorm2nyHOKMpEYhIQTrYN8SDz3TywDOH2NTeQyrttNRXctVZC3nT6fN5w+pmXQY6TZQIRKQgDCdSPLGvh//afZhHdnWz49AAAKta6/jAG0/lzWcs4HWLG2f1vAD5okTwKsWTadoPR9ndNcDz3VH2dkfY2x0hnkxzxWsW8ntnL+LUlrp8hylSsFJpp2tgmI7eIXYc7OeZA5nHzkMDxFNpKkpLaFsxh7+64nQuP3M+K/X3FDolgsBjzx9h6wtHicaSRGJJorEkiVSaZNpJuxNPptl3ZJB9h6Mk08dudVjcVM3K1joSyTRffng3X/r5bl63pJFzls0hlkwxFE8RS6ZprC5n2bwals2tYfncWlbPryvIau3AcIKHnu3k/u0H+fXew9RVljO/oZL5DVUsaKzi1OZaVrbWsaqljsVN1fp1JscVT6bZ3nGUTe09PN7ew97uCIf6hl/2N9RYXc6Zixp477oVXLhyHhecMlfj/+RY0Z/tLft7+PsHdvHo80cAMIO6ijJqK8uoLC+h1IzSEqOstIRTm2v5nTPns2Z+Pata6zi1uY7qimNf5of6hvnxtgP8aNtL/ODJDqrKS6kuL6WyrITewQSHI7HRsuWlxhmLGjlnWRNnLGxgOJmmbzBO72CCo4MJ+oYS9A3FOTqYoL6qjIvWtHDxaa2jVeNILEl7d5TnD0fY3RlhT1eE3V0DHOobZtX8etYubeLspU2cuaiB5rpKGqvLKSkx3J2ugRh7ujI1me6BGP1DCfqHkxyOxNjU3kM8mWZxUzXvPHcpyXSazv4Ynf3DPPlCL0cHE6Pvoa6yjLXLmjhn2RzaVsxhVWsdc2srqCwrvAQ3m43cg5mL6+WTqTQpd0Zu+4wl0+w/kqkZP98dpaN3KPg8JegfSrK/J8pwIg3Amvl1nLt8DoubqlnUVM3iOdWsmV/PosYqXeufZzbTbuRta2vzzZs3v+rnDQwn2H9kkN7BOD3RzBfsL3Z2sXFnN811Ffzpxat4R9sS6ivLQvtQRmNJXuwdpL07yraOPp58oZftHUdH/1AAaipKaaoup7GmIvNvdTkH+4fZ3nEUd5hbW0FFaQmH+odHn1NaYiyfW8Oq1joWNFax89AA2zv6GEqkRsuUGDTVVJBIphmIJUe3m0FDVTkN1WU0VpfTtnwuv3vWItYubcr6a78nGs80h3VFeOZAP5v397LzUD9jfuBRV1nGnNpyIPOLMJZMk0w5TTXltNZX0lpfRUt9JjmNPKorSnGOfamVlZRQXVFCdXkZ1RWlzK2poLWhclK1qMORWNDc0MczB/pxd5bPq2XFvBqWz6sl7U7fYILewQSRWIKmmgrmN1Qxv6GS5rpKaipKqSorHU2cg/EUvYOZz0w8laaqrJTK8hIqy0o4EonzYu8gL/YMceDoEI5TXloSPIzy0hIqykqoCCY56Rs6lugh8/85t7aCObUVxBIpugdidA/EOBKNA5kfDGUlJZSWGqmUk0w7yXSa4USKo4OJ0c9yMp2mrrKM+qpy6qvKcIdoPMlQPMVgPEVZiQUxl1JVXjJaw1vYWEVdZTm9g3EOR2IcicTpH04wnEgTS6QYTqQYTgbLyTSp9MTfFyUGCxuraaopH/1MLW6q4fxT5nLeijnMq6s84f+dhMfMtrh7W9Z9xZII7tt2gBu+u/Vl25pqyvmfF63k2tcvz1tVNJFK09E7RG1FKY015RP+mj4SiWU60XZ3A7CypY5Tm2s5taWOFc01r3heMpVmV2eEXZ0DwZdFnJ7BOKVmo007p7bU0VpfOeXmnYHhBL958Sgv9gzRE818ifVG45gZlWWZL8zSkhJ6ojG6BjKPw5FMTeQ43ytZ1VeWMa+uAjMjFTTbpdJOLJn5coyN+7JaOrea8tISXuwZJJF6dQcbGZIgnkyfoGRGY3U5pSVGIpUOHv6KL84SYzT5OZnEOjB8LDFXlJXQUlc5+h4TyTTJdKaJsqzEKC3JJJjKshLm1BxLIqVmDAwnGBhO0j+cpLQEaivKqKnM1EqTwTmKJdJEY0m6BoY51DdM50CMVNqpLCuhOThuQ1U5VeWZhDGSOMaulwafFzMoLylh6dwaVrbUsmzeKz+HUjiUCMg022zrOBr88ZTTVFPBnJqK0Q+15F467UTiSfoGEwwnUphlmjcMSKScoUQq+EWbpCcapyv4tTzSxFZaYqNNd1VBE1xVeSlNNeWcsaiBMxc20liTqZkkU2kO9g2z/8ggJSXQVF3BnNpyaivL6BtM0Nk/TGd/5rWHgl/CIzWqucFnpammnPKykuCXcppYMsWcmgqWzq1h6dwa6ipf+WMilXYSqUytaKTZcXzijSfTHB2MU1leSkNVeDXSbFJpZziRoqaiVM0zs5wSgYhIkTteItBwfCIiRU6JQESkyCkRiIgUOSUCEZEip0QgIlLklAhERIqcEoGISJFTIhARKXIz7oYyM+sG9mfZ1Qj0HWfb+P0j69nKNAOHTyK8bDFMtsxE8WVbz7Y81diPF9uJ9uvcH9sWVvwne+7Hr4fx2cnnuR+7XIzn/njxjd+/3N1bspZw91nxADYcb9v4/SPr2coAm6crhsmWmSi+E72X8e/jZGOfTPyTjb1Yz32Y8Z/suc/FZyef5z4X8RfyuZ9K/GMfs6lp6Mcn2DZ+/48nUWY6YphsmYniy7aebXmqsU/mNSYb+/htOvcnFta5H78eRvz5PPeTPf7xzORzP5nXOOExZlzTUC6Y2WafYEyOQjeTYwfFn08zOXaY2fHnO/bZVCOYThvyHcAUzOTYQfHn00yOHWZ2/HmNXTUCEZEipxqBiEiRm/WJwMz+xcy6zOzpk3juuWb2lJntMbMv2ZiZO8zsQ2a208yeMbPPTW/Uo8eY9tjN7JNm9pKZ/SZ4XDn9kY/GEMq5D/Z/1MzczJqnL+KXvX4Y5/5TZrY9OO8Pmtmi6Y98NIYw4v+8me0I3sO9ZtY0/ZGHFvs7g7/VtJmF0hY/lbgneL1rzWx38Lh2zPbj/m2clJO9ZGmmPICLgHOAp0/iuY8DFwIG/BR4S7D9EuA/gMpgvXUGxf5J4KMz9dwH+5YCD5C5n6R5psQONIwpcwPwtZl07oHLgbJg+bPAZ2dQ7L8FnAZsBNoKKe4gphXjts0Fng/+nRMszznee5zKY9bXCNz9EaBn7DYzW2lmPzOzLWb2X2Z2+vjnmdlCMn+4j3rm7H8L+L1g9weBv3P3WHCMrhkUe86EGP8XgJuA0Dq4wojd3fvHFK2dgfE/6O4jEyw/BiyZQbE/5+47w4h3qnFP4HeAh9y9x917gYeAK8L62571iWACG4APufu5wEeBr2QpsxjoGLPeEWwDWAP8NzPbZGb/aWbnhRrty001doA/D6r3/2Jmc8ILNaspxW9m64GX3H1b2IFmMeVzb2afNrMXgfcAt4QYazbT8dkZ8X4yv0ZzZTpjz6XJxJ3NYuDFMesj7yWU9/jK2bZnOTOrA14P3D2maa0yW9Es20Z+wZWRqa79NnAe8H0zOzXI0KGZpti/CnwqWP8U8A9k/qhDN9X4zawG+DiZJoqcmqZzj7t/HPi4md0M/DnwN9McalbTFX/wWh8HksB3pjPGiUxn7Ll0vLjN7H3AjcG2VcBPzCwOtLv725n4vYTyHosuEZCpBR1197PHbjSzUmBLsHofmS/MsVXfJcCBYLkD+GHwxf+4maXJjBXSHWbgTEPs7t455nn/DNwfZsDjTDX+lcApwLbgD2sJ8KSZne/uhwo89vHuAv6dHCUCpin+oNPyKuBNYf/wGWO6z32uZI0bwN3vAO4AMLONwHvdfd+YIh3AxWPWl5DpS+ggjPcYRqdJoT2AFYzpwAF+DbwzWDbgrAme9wSZX/0jnTJXBts/ANwaLK8hU4WzGRL7wjFl/hL43kw69+PK7COkzuKQzv3qMWU+BNwzk849cAXwLNASZtxhfm4IsbP4ZONm4s7idjItD3OC5bmTeY8nFXfY/6H5fgDfBQ4CCTLZ9Doyvyp/BmwLPti3TPDcNuBpYOa7o0oAAAOKSURBVC/wTxy7Aa8CuDPY9yRw6QyK/dvAU8B2Mr+iFoYRe1jxjyuzj/CuGgrj3P8g2L6dzPgvi2fSuQf2kPnR85vgEcpVTyHF/vbgtWJAJ/BAocRNlkQQbH9/cM73AO97NX8br/ahO4tFRIpcsV41JCIiASUCEZEip0QgIlLklAhERIqcEoGISJFTIpBZwcwiOT7e7WZ2xjS9VsoyI5I+bWY/PtGonmbWZGZ/Oh3HFgFNTCOzhJlF3L1uGl+vzI8NsBaqsbGb2TeBXe7+6eOUXwHc7+6vyUV8MvupRiCzlpm1mNkPzOyJ4LEu2H6+mf3azLYG/54WbH+vmd1tZj8GHjSzi81so5ndY5lx+L8zMvZ7sL0tWI4Eg8ltM7PHzGx+sH1lsP6Emd06yVrLoxwbYK/OzH5uZk9aZvz5twVl/g5YGdQiPh+U/VhwnO1m9n+m8TRKEVAikNnsi8AX3P084A+A24PtO4CL3H0tmRFA/3bMcy4ErnX3S4P1tcBfAGcApwLrshynFnjM3c8CHgH+ZMzxvxgc/4TjwQRj57yJzB3fAMPA2939HDJzYPxDkIj+F7DX3c9294+Z2eXAauB84GzgXDO76ETHExlRjIPOSfG4DDhjzMiPDWZWDzQC3zSz1WRGbiwf85yH3H3smPKPu3sHgJn9hsxYMr8cd5w4xwbv2wK8OVi+kGNjxd8F/P0EcVaPee0tZMaeh8xYMn8bfKmnydQU5md5/uXBY2uwXkcmMTwywfFEXkaJQGazEuBCdx8au9HMvgz8wt3fHrS3bxyzOzruNWJjllNk/5tJ+LHOtonKHM+Qu59tZo1kEsqfAV8iM2dBC3CuuyfMbB9QleX5BnzG3b/+Ko8rAqhpSGa3B8mM+Q+AmY0MB9wIvBQsvzfE4z9GpkkK4OoTFXb3PjJTWH7UzMrJxNkVJIFLgOVB0QGgfsxTHwDeH4x/j5ktNrPWaXoPUgSUCGS2qDGzjjGPD5P5Um0LOlCfJTN8OMDngM+Y2a+A0hBj+gvgw2b2OLAQ6DvRE9x9K5mRKq8mM/FLm5ltJlM72BGUOQL8Krjc9PPu/iCZpqdHzewp4B5enihEjkuXj4qEJJhRbcjd3cyuBt7t7m870fNEck19BCLhORf4p+BKn6PkaEpQkVdLNQIRkSKnPgIRkSKnRCAiUuSUCEREipwSgYhIkVMiEBEpckoEIiJF7v8Dc3cJCoZ+k5MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We plot the loss vs learning rate\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you need to look at the above figure and select the lr value that you want to use. I would use lr=1e-2. The art of selecting good values comes down to practice, just try the values you are confused about and select the one that gives the best results. There are certain guidelines that you should follow while finding the lr value\n",
    "* Select the one where the loss is small for larger amount of time\n",
    "* When loss begins to increase, select a value that is 10 times less\n",
    "\n",
    "For experimentation, let us see what the resutls are when I use max_lr=1e-2 and max_lr=1e-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.117561</td>\n",
       "      <td>4.697643</td>\n",
       "      <td>0.047400</td>\n",
       "      <td>0.175600</td>\n",
       "      <td>01:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.663519</td>\n",
       "      <td>3.413377</td>\n",
       "      <td>0.173200</td>\n",
       "      <td>0.446900</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.335882</td>\n",
       "      <td>3.095275</td>\n",
       "      <td>0.239100</td>\n",
       "      <td>0.529900</td>\n",
       "      <td>01:48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXxc5X3v8c8zo9E60kga7ZItyZZ3I2RsjMEQjCEJSwJJk7TOJWlCcsNt0lwIXbK0aQJt0jS96ZbblpQ2dLmFtJQkJSELBPDCZoyMd8u7ZFvWvoz2ZTR67h9nbMu2bMswko5G3/frdV7SnHM0ep4X4y9Hz/k95zHWWkRExL08090AERG5NAW1iIjLKahFRFxOQS0i4nIKahERl0uYjDcNZAXtUEo2pwtKEr0ekhI8JHg9JHgNCR5DgteDx4DBkJLoJcFjJqMpciVCJ6C/HRLTIL0IkvzT3SKRWWP79u1t1trc8Y5NSlAPJWez7ovf52NrStnf2E1bzxCNXYO09Q7R1jtEOGIZGXs+kJ+RxPKiAIsL05mX46c8N415OWlkpiZORhNlPJER2PUD2PQt6D4F82+FW78GRVXT3TKRuGeMOX7RY5NRR51UuMAe2beLOdmpFxyz1tI1EKa1Z4ieoRGGR0bZ19DNvlNd7DnVxbG2PiKjZ9uUl57EirmZrCzN4ob5OSwrysAYXX1PqvAgvPlP8PJ3YKATlv0arP8qBOdPd8tE4pYxZru1dtW4xyYjqBcsu9oe3rfrbf1sODLKyY5+atv6ONbax76GLnacDHG8vR8Af1ICC/P93LoknzXzsrmqOJPEBA21T4rBLnjt/8LrfweRYbjmN+HmL0F6wXS3TCTuTHlQr1q1ylZXV8f0PVu6B3n5cBt7TnXx1olOdtd3AZCU4GHF3ExWlwe5rjybxQXpBP1JMf3ds15PM2z5P7D9n8HjgzWfhbUPQkrmdLdMJG7ERVCfr613iOq6DrbVdrKtrp39Dd2cHjGZm53KukW5vHdZAavLs/F5dcUdEx3HYOOfwp7/guRMuPEhuO5/gS9lulsmM1w4HKa+vp7BwcHpbsqkS05OpqSkBJ/Pd87+uAzq83UPhtl5IsSh5h62HmvnlSNtDIZHCaT4uG1JPu9dls+7FuaS7PNOabviUuNueOlP4PDzTnXIui9B1cfAOyn3pmUWqK2tJT09nWAwGNf3oKy1tLe309PTQ3l5+TnHZkVQn29gOMLmQ608v6+JF2qa6R4cIcXn5eaFubx3eT7rF+cTSPFd/o3k4upehRcehvptEKyA9X8ES++BOP6HJpOjpqaGxYsXx3VIn2at5cCBAyxZsuSc/ZcK6ri9BEpJ9HL78gJuX15AODLK1mPtPLevief3NfPLfU0keAzXzw/y3mUFvGdpPnkZydPd5JmnbC18+nk4+At48Y/hvz4BRSvgtodh3rrpbZvMOLMhpOHt9TNur6gvZnTUsrM+xHP7mnhubxN17f0YAwvy/Lx7aT7rF+dxdUkmCRrXvjKjEdj9n84YdtdJJ6hv/ToUXzPdLZMZoKam5oIrzHg2Xn9n5dDHRFhrOdTcyws1zbx6pI2tx9oZtZCenMAN84O8a2EutyzKoyhTN8smLDwI1Y87Ndj97bD0A86QSE7FdLdMXGy6gzoUCvHkk0/yuc997op+7s477+TJJ58kM/PKKqAU1O9AqH+Y1462s+VQK1sOtdLQ5dyBXlWaxcqyLN67rIAVczJnzZ9o78hgN7z+t/Da38LIIFzzcacGO6NoulsmLjTdQV1XV8f73vc+9u7de87+SCSC1xv7AgQFdYxYazna2sfPdjfy0sEW9jd0EY5YijNTuKuykI+sLGFBfvp0N9P9eludq+s3vw8er1POt/YLkJo93S0TF5nuoN6wYQPPPPMMixYtwufz4ff7KSwsZOfOnezfv58PfOADnDx5ksHBQR588EHuv/9+AMrKyqiurqa3t5c77riDG2+8kddee43i4mKeeeYZUlLG/2tcQT1JugbCvLC/mZ/taeTlw62EI5a52amsrQhy88I8blqQQ1pS3N6bfec662Djt5xx7OQMJ6yv+y1IvPAxAzL7jA2uR366j/0N3TF9/6VFGXz9/csuenzsFfWmTZu466672Lt375kSuo6ODrKzsxkYGODaa69l8+bNBIPBc4K6oqKC6upqqqqq+PVf/3XuvvtuPvaxj122v6fNyqqPWAuk+PjQyhI+tLKE9t4hfrqrgVePtvPsrkZ+sO0kiV4Pa+YHuWN5AR+oKiYlUfXa58gqg1/7B1j7gFMh8uIj8MY/ODXYKz4OXpVKinusXr36nDrn7373u/z4xz8G4OTJkxw+fJhgMHjOz5SXl1NV5TzAbOXKldTV1cWsPQrqtyHoT+KTa8v55NpywpFRqus6eelAMy/WtPCVH+3hz395gA+uKOGuykJWzMnEo0e4npW/DP7Hf8Lx150a7Gcfcp4nsv6rsPSD4FG1zWx3qSvfqZKWlnbm+02bNvHCCy/w+uuvk5qayrp168adQZmUdPbRFV6vl4GBgZi1R0H9Dvm8Hq6fH+T6+UH+4M4lvFnXyeOv1PLvW4/z+Ku1FAWSuXmRMzSyfnGeZkaeVno9fOqXcOg55+r66U9B4d84JX3z12vSjEyp9PR0enp6xj3W1dVFVlYWqampHDhwgK1bt05x6xTUMWWMYXV5NqvLs+keDPNiTTM/39PEs7sa+MG2EyQmeFhelMHK0ixuWpDLdfOySUqYxcFtDCy6HRa823l+yMZvwr//GpTdBLc9AiUrp7uFMksEg0HWrl3L8uXLSUlJIT8//8yx22+/ne9973tUVlayaNEi1qxZM+Xt083EKTASGeX1Y+28criNHSdC7KwPMTwySmqil9Xl2XxwRTHrFuVpSvvIEFT/s/Okvv42WPJ+WP81yF043S2TSTbdVR9TTTcTXSjB6+GmBbnctMBZZWdgOMLrx9p46UALmw+18uB/7MRj4Jq5WXxybRk3VeQSSJ2FoZ2QBGt+C1bcC6//Pbz2XTjwM6i6F9Z9GQIl091CkWmhK+ppFhm1VNd18MqRNp7Z2cCJDmdK+5KCDNbMC7KsKIPirBRWl2XPvpuSfW3w8l84q81g4Lr74cbfUQ12HNIVteqoZ4xwZJS3jney9VgHb9S2s/14J0MjowCkJyWwtCiDdYvyuG1JHhV5/tkzQzJ0wqnB3vUDSEp3SvzWfM5ZhFfigoJaQT1jDY1EaAwNsqs+xJt1Hew8GWLvKWciQHFmCrcszuXqkkwq8vwsKwrE/5JkzfvhpW/AwZ9BWh7c/EW45hOQoAWQZzoFtYI6rjSEBth0sJWNB1t49Ugb/cMRwFmSbFWZswDwDfODzMv1k56UEJ/DJSfecGqwT7zmTKRZ/0fOAryqwZ6xFNQK6rg1NBKhITTIwaZuttV28uqRNg42n60FDaT4WFWaRVlOGkF/Igvy0llbESQ1MQ7uIVsLh3/l1GA374WCq+DWh6HiVtVgz0AKalV9xK2kBC/lOWmU56Rx+/JCANp7h3ijtoOG0ACHm3vZfqKT14+1n7nyTvR6uLY8i1sX57OsKIN5uX5y/Ikzb7zbGFj4Hqi4Dfb+EDZ+A574EJTeCLd9Heasnu4WSpzz+/309vbS0NDAAw88wNNPP33BOevWreM73/kOq1aNm78TpqCOM0F/EndeVXjB/v7hEXacCLExWhL4x8/uP3MsPTmBedHAX1YU4M7KQopnyjO4PR6o/IizBNhb/wqbvw3ffzcsugtu/RrkLZ7uFkqcKyoqGjekY0lBPUukJiawtiKHtRU5fBWo7+znaGsfx1p7qW3ro7atjzfrOvnvnQ188+c1LC3MoGpuJleXBKgsyWRBnt/dq94kJMLqz8DVH4Wtjzo12I9e77xe9xXInDPdLRSX+9KXvkRpaemZxQMefvhhjDFs2bKFzs5OwuEw3/jGN7jnnnvO+bmxT94bGBjgvvvuY//+/SxZsiRmz/tQUM9SJVmplGSlcvPC3HP2H2/v46e7Gnj9WDs/3dnAk2+cACDZ52FZUYCqOZmsLM1iYb6f+bkuLBFM8sPNvw+rPgWv/CVs+0dnevq1n4GbfhfSgpd/D5lev/gyNO2J7XsWXAV3/NklT9mwYQNf+MIXzgT1U089xS9/+UseeughMjIyaGtrY82aNdx9990X/dw/+uijpKamsnv3bnbv3s0118RmKboJB7UxxgtUA6este+LyW8X1ykNpvH59Qv4/PoFjI5a6tr72F3fxa76EHvqu/h/W4/z/VdqASgMJHN1SSYLC9JZUpDODRU57pkGnxaE937Teeb15j+DNx6Ft/7tbA12kn+6Wygus2LFClpaWmhoaKC1tZWsrCwKCwt56KGH2LJlCx6Ph1OnTtHc3ExBQcG477FlyxYeeOABACorK6msrIxJ267kivpBoAbIiMlvFtfzeAzzcv3My/XzgRXFgFNpcrCph5rGbrYcaqOmsZvn9zcxasHrMRQGkqksCXBjRS4L8v0szEuf3unwmXPgnr+D6/83vPQnzoOftj0G7/p9WHmfarDd6DJXvpPpwx/+ME8//TRNTU1s2LCBJ554gtbWVrZv347P56OsrGzcR5yONRl/ZU4oqI0xJcBdwDeB34l5K2TGSErwUlmSSWVJJr9x7VwABsMR9jV0sflQG3VtfbxZ18HP9zQBTnFGWTCN3PQkFuWns7AgnUX56SwrypjaFXHyFsOGJ+Dkm05J3y++6KzpeMtX4aqPqAZbAGf44zOf+QxtbW1s3ryZp556iry8PHw+Hxs3buT48eOX/Pl3vetdPPHEE9xyyy3s3buX3bt3x6RdE/2X8tfAFwEtEigXSPZ5WVmazcpS5xkcznqTvZzo6Gd3fRcHm3po6Rniv3ecomdoBODMijg3zA9ybVkWy4sDU/PI1znXwid+CkdfhBcegR/f79x4vPVrsOA9qsGe5ZYtW0ZPTw/FxcUUFhZy77338v73v59Vq1ZRVVXF4sWXriL67Gc/y3333UdlZSVVVVWsXh2bMtHLTngxxrwPuNNa+zljzDrg98YbozbG3A/cDzB37tyVl/s/j8w+1loauwY50NTNa0fa2XiwhaOtfQAkJni4uiTAVcWZLC3KYGlhBgvy/fgms9JkdBT2/ciZlt5ZC3Ovh9sehrlT/7zh2U4TXt7hzERjzLeAjwMjQDLOGPWPrLXjr9qIZibKxLX1DrH9eCfVdR1UH++kprGbwbDzIKpEr4clRRm8e0keaytyWFSQPjmzKiNh50bj5m9DbzMsvANu/SNn2TCZEgrqGE4hv9QV9VgKanm7IqOW2rY+9jd2s7+hm63H2tl5MgQ4oxKl2aksKwpwy+I81i/OIzsthjcDh/vgje/BK38DQ91w9QanBjurNHa/Q8aloNYUcplBvB5DRZ6fijw/d19dBEBz9yA7T4Y40NjDweZuth/v5Gd7GvEYWDPPWfm9siST8tw00pMS3v5d98Q0p9Z65X3wyl851SF7noZrPw03/R74cy//HvK2WWvdV5c/Cd7O85X0UCaZcay17DnVxfP7mvnZnkZq2/rOHPN5DUsKM7ixIoe52ankB5KpLA4Q9Cdd4h0vouuUU4O949/BlwrXfx6u/21IVoVqrNXW1pKenk4wGIzrsLbW0t7eTk9PD+Xl5ecc09PzJG5ZaznR0U9NYzcnOvpp7x1m+/FOdpwMERl1PtseAytLs1i/OJ+bF+YyLzftylaDbz3k1GDX/ARSg04N9qpPOUuHSUyEw2Hq6+svW6McD5KTkykpKcHnO3d+gYJaZp2hkQitPUM0dg3y8qFWXqhpYX+js+iCx0BxVgoVuX5y/EnkpCdRGEimICOZwkAKBYFkgmmJFz7L+9R25znYtVsgMAdu+QOo/A3wzOKV5CVmFNQiQGPXANtqO848jOpYax8dfcO09w0Rjpz77yApwcPaihyWFmawtCiDq+dkUhRIdv4sP/qSE9iNuyB3iVODvegO1WDLO6KgFrmE0VFLe98wTV2DNHYN0NQ9yLHWPjYebKG+c+DMEEpxZgqFgWRy05O4riyTj6TtIO3lP4WOozDnOqcGu/SGae2LzFwKapG3KRwZZdfJEDtPhthV30VrzyANoUFOdPTjMVBZmMYG3xbuCf0bKUOt2AXvwdz6dShYPt1NlxlGQS0SQ9Za3jrRyaaDrew4EeJQcw/dPd180vscn/P9FD/9HM2/na41X6TBU0BFrp+gP5HMVN/UTJOXGUlBLTKJrLXsru9i58kQdfWnWHT0ce4Z/AleIjwZuZW/HfkgbQQAKA2mcv28IPNz/SwsSMday8rSLNKTXfJ4WJk2CmqRKdZYXwubvk3B0aeIeBLZV/pxXs37KNubImw/0UmoP3zm3ASP4dqybNYtyuWq4gBXlQQU3LOQglpkurQdcRbe3fdjSMl2Zj5e+z9pHTQcbe1leGSU1462s+lgCweanBXkE70ecvyJZKT4KA2mkpaUQIrPS1ZqIkF/IkF/EjnRVeVz01XLHS8U1CLTrWGH81jVYxshowRu+QpUbgDv2ac4NHcPcrCph1eOtNHRN0xn3zB17X0MhkcZCEcI9Q8zet4/10CKj6LMFJYUplOanUZpMJV5uWksLwpcWAcurqagFnGLY5ucwG54C3IWOTXYi++aUA326KglNBCmvXeI5u4hDjR1U9fex/H2fo609NLYdXZWX4LHsDA/nZWlWawszeKauVnMyU6J6+nZM52CWsRNrHWmo7/4J9B+GIpXOTXY5Te9o7cdDEeo73QWa6hp7KamsYcdJzrpG44AEExLJJDio2pOJqvKsqmak0lBIJmsVJ8C3AUU1CJuFBmBnU/Apj+DngaouM25wi68Ona/YtRyqLmH7cc72XkyRKg/zM6TnbT1Dp85J5DioywnjZLMFIL+RFaXZ3PN3CwKT8/ElCmhoBZxs/AAbPtHePkvYDAEyz8Et/whBOdPyq+z1nKkpZf9jd209gxR195HbVsfDaFBWnuG6I0ul5bjT6JqToDyHGfNy/RkH8uLAiwqSCcxQWtMxpqCWmQmGAg56ze+/vcwGoZrPgE3fxHSC6asCSORUfY2dLPrZIhd9SF213dR39l/ZtUdcJZNW1qYwYq5maydn8O1ZdlkpLyD54ALoKAWmVl6mmDzn8Nb/wreRFjzWbjhAUjJnLYmdQ+GCfWF2X0qFA3xLnbXh84EeHZaIqvLsrluXjaFgWSq5mRREEietvbORApqkZmo/Shs/Cbs/SEkZzo12Ks/A76U6W4Z4DxK9s3aTg40dXOgqYetx9qp7xw4c7w4M4XEBA9rK4JU5PpZXJjBNXOzNGxyEQpqkZmscRe8+Mdw5AVIL4J1X4aqe8+pwXaLU6EBmrsHeeNYBwebumnvG2bnyRA9g864d2qil3m5aZQG08CCx2OYl5PG4oJ05uX6KQ2mXtmiDnFEQS0SD2pfdp6DfaoaggucldKX3O3652Bba2ntHWLXyS42H2rhWGsfjV2DeD2G4ZFRTnb2czqGjHFuYmYkJ7BmXpCrigOsLM2iNJgW91fiCmqReGEtHHjWqcFuOwhF1zg12PNunu6WvW0DwxEOt/RQ29ZHXVs/DaEBGrsH2XGi88yVeIrPS2EgmcQEDwlew9r5OZTlpOFPSmBudirz8/z4k9z3F8aVUFCLxJvICOz+D9j4Leiuh3m3wG1fh6IV092ymLHWcrill30NXeyp76a5Z5C+oREGwxGq6zoZGTOf3usxzvNQ0nzMz/VTnpNGjj+JtEQvxVmp3LQgx/VDKgpqkXgVHoQ3/wle/g4MdEJWOWSVRbfSMd+XQUrWtDY1lvqHR+jsD9M7OMLx9j62Hutg1FpC/cPsa+imITRwZkYmOEE+PzeNgkAKaYleSrJSWFqUQWZKIleVBMh5O6vUx5iCWiTeDXZB9ePQtAc665ytv/3cc5IDkHleeGeVOuEemAMJiVPe7Mk0MByhezDM7vou9tSH2N/YTVvvMN0DYeo7BxiOnK0Nn5+bRnqyD2stSQleynPSyA8ksyg/nUUF6ZQFU0nwTu4YuYJaZDYa7IbQceg8fja8O+vO7osMnT3XeCCj2Anv8cI8Ldf1Ny2vRDgyypGWXnoGR9h+vJPqug6GI6N4jKF/eITaNmfh49ExNzkDKT4KMpLxegy56UmMWugfGiEvI4nizBSC/iSWFwWonBMg4208T/xSQT2zR99F5OKSM6DgKmc73+go9DaNCfAxYX7kBefYWL7Us8F9fpBnzoXE1EntSqz5vB6WFGYAsLo8G7hwuv5gOMKRll4ONvVwvL2Pjn5nAeRR6zySNsFjSPZ5OdDYwws1LQyPOFfoxsC8nDTSkhIIpiUSjlgGwxEaQgMkJnhISUygNDuVgkAyGSk+MpITyEi5dLArqEVmI48HMoqcbbyV08MDEDpx7pX46TA/thnCfeee788/7yq87Gygpxc6v2+GSfZ5WV4cYHlxYELndw2E2V0fYseJELvrQ/QNRejoG8bn9eDzerhuXpBRa+kdHOFQcw+vHm2jd2iEiQxqaOhDRK6MtdDXFh1CqYPO2jFBftypQrFnx3/xJkZD+/whlTJnf3LGdPTCFUZHLT1DI3QPhJkbTNPQh4jEiDHgz3W2knFyZWQYuk6OGQ+vO7vVVztPCBwrJfvilSoZJa6cgRkrHo8hkOIjoKEPEZlSCYnOI1ov9pjWgc5zx8RPh3njTmdBhdGRs+caLwRKxqlUKXOqVVKy4uom58UoqEVkaqVkOVtR1YXHRiPQ3XDe2HidE+YHfw59reeen5QxzrBKufM6cy4kTH99dCwoqEXEPTxeyJzjbOMtTTbUO37JYdthp1plZHDMyca5WXqxahV/3oy5GldQi8jMkeSH/GXOdr7RUehrGb/k8OhGZ7mzsRJSzr0SPyfISyExbXL7cgUU1CISHzweZzWc9AKYu+bC4+HBszc5zy87rHsFhnvPPT8t78Kbm6fDPKPIufqfIgpqEZkdfMmQs8DZzmct9HdEx8Przg3yk9tg74/Ann12CB6fMwY+XqVKZmnMV+NRUIuIGANpQWcrWXnh8UgYuurHLzlseMupZBkrOfMilSplznNVvFc2xVxBLSJyOV4fZJc723gGu8Z/pkrzXqdaJTJ89lzjcerDz78SvwQFtYjIO5UcgMJKZzvfaAR6GscP8sPPQ2/zZd9eQS0iMpk80Uk7gRIoW3vh8eE+57kqjyy9+Ftc7ncYY5KNMduMMbuMMfuMMY+8o0aLiMhZiWmQt+SSp0zkinoIWG+t7TXG+IBXjDG/sNZujUUbRUTk0i4b1NZ5vN7pAkNfdIv9I/dERGRcE3pIrDHGa4zZCbQAv7LWvjHOOfcbY6qNMdWtra0XvomIiLwtEwpqa23EWlsFlACrjTHLxznnMWvtKmvtqtzc3Fi3U0Rk1rqiZRestSFgE3D7pLRGREQuMJGqj1xjTGb0+xTgNuDAZDdMREQcE6n6KAT+1RjjxQn2p6y1z05us0RE5LSJVH3sBlZMQVtERGQcM29pYBGRWUZBLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlLhvUxpg5xpiNxpgaY8w+Y8yDU9EwERFxJEzgnBHgd621bxlj0oHtxphfWWv3T3LbRESECVxRW2sbrbVvRb/vAWqA4slumIiIOK5ojNoYUwasAN4Y59j9xphqY0x1a2trbFonIiITD2pjjB/4IfAFa233+cettY9Za1dZa1fl5ubGso0iIrPahILaGOPDCeknrLU/mtwmiYjIWBOp+jDA94Eaa+1fTn6TRERkrIlcUa8FPg6sN8bsjG53TnK7REQk6rLledbaVwAzBW0REZFxaGaiiIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuNxlg9oY87gxpsUYs3cqGiQiIueayBX1vwC3T3I7RETkIi4b1NbaLUDHFLRFRETGoTFqERGXi1lQG2PuN8ZUG2OqW1tbY/W2IiKzXsyC2lr7mLV2lbV2VW5ubqzeVkRk1tPQh4iIy02kPO8HwOvAImNMvTHm05PfLBEROS3hcidYaz86FQ0REZHxaehDRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLichMKamPM7caYg8aYI8aYL092o0RE5KzLBrUxxgv8HXAHsBT4qDFm6WQ3TEREHBO5ol4NHLHWHrPWDgP/Adwzuc0SEZHTEiZwTjFwcszreuC6808yxtwP3B99OWSM2fvOm+dqOUDbdDdikqmP8WE29BFmfj9LL3ZgIkFtxtlnL9hh7WPAYwDGmGpr7aoJN28GUh/jg/oYP+K5nxMZ+qgH5ox5XQI0TE5zRETkfBMJ6jeBBcaYcmNMIrAB+MnkNktERE677NCHtXbEGPN54DnACzxurd13mR97LBaNczn1MT6oj/EjbvtprL1guFlERFxEMxNFRP9khXAAAAOUSURBVFxOQS0i4nIxDep4mWpujHncGNMythbcGJNtjPmVMeZw9GtWdL8xxnw32ufdxphrpq/lE2eMmWOM2WiMqTHG7DPGPBjdH2/9TDbGbDPG7Ir285Ho/nJjzBvRfv5n9EY5xpik6Osj0eNl09n+K2GM8Rpjdhhjno2+jqs+GmPqjDF7jDE7jTHV0X1x9Xm9mJgFdZxNNf8X4Pbz9n0ZeNFauwB4MfoanP4uiG73A49OURvfqRHgd621S4A1wG9H/3vFWz+HgPXW2quBKuB2Y8wa4NvAX0X72Ql8Onr+p4FOa20F8FfR82aKB4GaMa/jsY+3WGurxtRLx9vndXzW2phswPXAc2NefwX4Sqzef6o3oAzYO+b1QaAw+n0hcDD6/T8AHx3vvJm0Ac8A747nfgKpwFs4M2vbgITo/jOfXZzqpuuj3ydEzzPT3fYJ9K0EJ6jWA8/iTFSLtz7WATnn7Yvbz+vYLZZDH+NNNS+O4ftPt3xrbSNA9GtedP+M73f0T98VwBvEYT+jQwI7gRbgV8BRIGStHYmeMrYvZ/oZPd4FBKe2xW/LXwNfBEajr4PEXx8t8LwxZnv0kRUQh5/X8UxkCvlETWiqeRya0f02xviBHwJfsNZ2GzNed5xTx9k3I/pprY0AVcaYTODHwJLxTot+nXH9NMa8D2ix1m43xqw7vXucU2dsH6PWWmsbjDF5wK+MMQcuce5M7eO4YnlFHe9TzZuNMYUA0a8t0f0ztt/GGB9OSD9hrf1RdHfc9fM0a20I2IQzJp9pjDl9oTK2L2f6GT0eADqmtqVXbC1wtzGmDufplutxrrDjqY9YaxuiX1tw/oe7mjj+vI4Vy6CO96nmPwE+Ef3+Ezhjuqf3/2b0LvMaoOv0n2JuZpxL5+8DNdbavxxzKN76mRu9ksYYkwLchnPDbSPw4ehp5/fzdP8/DLxko4OcbmWt/Yq1tsRaW4bz7+4la+29xFEfjTFpxpj0098D7wH2Emef14uK8WD/ncAhnDHAP5zuAfh30I8fAI1AGOf/zJ/GGcN7ETgc/ZodPdfgVLscBfYAq6a7/RPs4404fwruBnZGtzvjsJ+VwI5oP/cCX4vunwdsA44A/wUkRfcnR18fiR6fN919uML+rgOejbc+RvuyK7rtO50v8fZ5vdimKeQiIi6nmYkiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuNz/B49tyoNMTnErAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# In fastai there fit_one_cycle function that trains your model using the 1cycle policy\n",
    "# So you don't have to code up anything\n",
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy], callback_fns=[ShowGraph])\n",
    "learn.fit_one_cycle(3, max_lr=1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.334694</td>\n",
       "      <td>4.820587</td>\n",
       "      <td>0.036000</td>\n",
       "      <td>0.156500</td>\n",
       "      <td>01:51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.994016</td>\n",
       "      <td>3.712073</td>\n",
       "      <td>0.109600</td>\n",
       "      <td>0.351700</td>\n",
       "      <td>01:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.565970</td>\n",
       "      <td>3.270659</td>\n",
       "      <td>0.197700</td>\n",
       "      <td>0.490200</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXhcZ2Hv8e87o9FII42k0WixrN3xJm/xIjs22Zw0cLMnbdPUNIE2DXEh0EBonws8t/cWntJSWm6hKYSQ3nIbaEjJQhoIlwSymhDHwVtkybLjfZO1b6N9mff+cUaWbMu2Yms5Gv0+z3MezZw5M/O+yfg3Z97tGGstIiLiXp6pLoCIiJyfglpExOUU1CIiLqegFhFxOQW1iIjLJUzEi2ZlZdmSkpKJeGmZSK1HoasJEvyQkg2BMBh9l4tMhm3btjVaa7NHe2xCgrqkpIStW7dOxEvLRBocgD0vwuZvw/HfQhJQfh+s+TNIy5vq0onENWPMkXM9ptMlGeZNgMV3widegT/9JZReA7/5Z/jWUnj+k1C7a6pLKDIjTcgZtcSBoiucrfkQvPNd2PEf8N5TUHotrPsMzL0BPPqeF5kM+pcm55dZCjf/A3y+Cm74MjS+Dz/6A3h0LWx7Avp7prqEInHPTMQU8mDBAnvHl3/A4tlpfPLaywilJI77e8gUGeiDqudh8784TSGBLFjzAKz+BKRkTXXpZJrq7+/n+PHj9PTE/xd/UlISBQUF+Hy+0/YbY7ZZa8tHe86EBfWah7/HkaYuctOS+OH9a5iTnTru7yNTyFo4tMnpeNz3S0hIgss3wNpPQ/b8qS6dTDOHDh0iGAwSDocxxkx1cSaMtZampiYikQilpaWnPXa+oJ6Qpo8Fs4K8+hfree5TH6K7f5C7HtvMz96rmYi3kqliDMy5Fu55Bj79Liz7Q9j5FHxnNTx5txPiWvBLxqinpyfuQxrAGEM4HP7AvxwmtI368sIMfnj/GooyA/z5Uzt46KkdNER6J/ItZSpkL4DbH4GHq2D9l+DENnjiNvjeNfDej2Gwf6pLKNNAvIf0kIup54R3Ji6enc4zn1zHZ66by8tVtVz19df40k8qONnWPdFvLZMtNRvWfxEeroTb/hkGeuD5jfCtZfDWt6C7dapLKDItTcqoD5/Xw1/+twX8/KGr+L2VBfznb4+x7muvcf3/foNvvLxXoR1vfMmw6k/gwS3wR89A1lx45a/hnxbBL74ALYenuoQip2ltbeXRRx/9wM+7+eabaW2d+BOQCelMLC8vt+ebmfh+XYRN7zfw5vsNvLW/EY8x3LhkFn/yoRLKi0Mz5ifQjHKyAjZ/ByqfBRuFstuc8diFa6a6ZOIC1dXVlJWVTdn7Hz58mFtvvZXKysrT9g8ODuL1esf9/Uar76SP+rhQUI90rLmLH2w+zH/+9hiRngEW5AZZvzCbtCQftyzNoyQrZdzLJ1OovQbefRy2fh962qBgDXzoM7DwVvCM/z8ImR6mOqg3bNjACy+8wIIFC/D5fKSmppKXl8fOnTvZvXs3d955J8eOHaOnp4fPfvazbNy4ERheLqOjo4ObbrqJq666irfffpv8/HxeeOEFkpOTR32/aRfUQ7r6BnhhZw1PvXuUqpp2BqNOuVYWZfC7Kwu4Y/ls0pJ8F3gVmTZ6O2Dnk/DOo05TSEYxrH0QVtwLfg3lnGlGBtdXflbF7pr2cX39RbPT+OvbFp/z8ZFn1G+88Qa33HILlZWVp4bQNTc3k5mZSXd3N6tXr+bNN98kHA6fFtRz585l69atLF++nLvvvpvbb7+de++994L1HXK+oHbNFPJAYgIfXVPER9cUYa3leEs3L1XW8uOtx/if/1XJ13+xh/ULsvnE1XNYXpgx1cWVS+VPhSv+zJkos+fnznjsl74Ab/wdrLoP1myE9PypLqXMUGvWrDltnPMjjzzC888/D8CxY8fYt28f4XD4tOeUlpayfPlyAFatWsXhw4fHrTyuCeqRjDEUZgZ44Jo5fOLqUiqOt/HUu0d5qaqWFytOct2CbD5z/VxWFIbweNSePa15vLDodmc79lsnsN9+xPm75Pedduy8ZVNdSplE5zvznSwpKcNNrm+88QavvPIKmzdvJhAIsH79+lHHQfv9/lO3vV4v3d3jN0jClUE9kjGGywszuLwwg7+6dRFPvH2Yf/31QX7/u5tJT/Zx7fxs7i4vZN1lYbwK7emtcDUUPuE0hbzzGOz4IVT8GEquhg/9Ocz9sBaCkgkRDAaJRCKjPtbW1kYoFCIQCLBnzx7eeeedSS7dNAjqkVL9CXz6url8fF0xv6yq452DTfxydx0/fa+GYFICq4pDzM8NsrIoxLrLwqQnq017WgqVwE1/74zJ3v6EE9o/uhuy5sO6TzuzIH2jd9KIXIxwOMyVV17JkiVLSE5OJjc399RjN954I4899hjLli1jwYIFrF27dtLL55rOxIvV0z/Iq9X1vLW/kR1HWzjY2EnfQJRAopfrFuZweUE6q4ozWZSXRnKiRhVMS4P9zkJQb/8L1FY4C0Gt/oSzpY56QQyZZqZ61Mdkm7adiRcryefllmV53LLMuQJJT/8gu0608fRvj/H2gSZ+XnESgASP4Zr52XxkUS5L8tOZnxskMUE/o6cFrw+W3Q1L/wAOv+W0X7/59/DWN+HyP3TasbMXTHUpRSbMtA/qMyX5vKwuyWR1SSYAjR29/GZ/I5Un2nix4iSv7akHwOc1zM8Nsnh2Gkvy07luQQ6FmYGpLPq0MPQLbEomJRkDpVc7W+M+ZwLNe0/B9h/AvI84zSKl1zrHicSRad/08UFEo5YjzV1Unmijqqadqpo2Kk+00dLVjzFw1dwslhWkszQ/g/KSEFmp/gu/qEtYa2nt6qe2vYfath4ONHTgMYa0ZB/1kR46ewdI9HrxJRjauvtpiPTS3NlHOMVPdtBPqt9L36Cls3eA7v5BWjr72HWijfr2XlKTEkhK8NDZN0ikp5+odb7ocoJJAGSlJuL3eUnyeQn4vMxKTyIvPYnsoJ+89GSyg348BrKC/vEfC9/Z6Eyeefdx6GyAWUudM+zFvwcJWgd9ulDTxzSZ8DJVrLUcbe7iue0nePG9Go42dzEQm2xTmpXC3JxUUhK9rCrJZMnsNOblBkn1T84PEWst7d0DYJwmneMt3bR29dHc2UdrVz8tXX1U1rRzuLGTuvYeegei53wtr8ecmkSU6PWQnOilIJRMc2cfTR199A06zw0keklM8JCe7GNJfjoFGcl09Q3S1TdIqt9LMMlHgtfQ1TdIU0cfFktDpJf+wSjd/VE6ewc42dpNZ9/gqOXICPhI8HjwGEjxJ5ATdL4oluanc9OSPPJDyRc3eqe/B3Y97ZxlN+yBYJ4zFrv8PkgOffDXk0mloFZQfyC9A4NUnmhj6+EWth5p4WhTF63dfdS1O8uzJvk8lOWlsXBWGiXhAHkZyRSGkgmn+LHYU0swB5MSCAUS6R2IkuTznLepYDBqaevup6a1m4rjbeytbWfXiTYONHTS1n3uJUITPIbicIDFs9OZlZ5EbloSs9KSmJXupyScgtdjaOzoIzvVT3rAx2DU0jdKeay19A5E8Xk94zbEsbWrj5aufo40ddLeM8BgNEptWy8nWrsYjDrvGekdoKG9l7pID0eaugDnTD05dlY+LzdIQSiZxbPTWZSXRn5G8oU7hK2F/a86V6A5+Ab4UpzZjms/CZlzxqVuMv4U1ArqS2at5UBDB4cau/jN/kb21kaoqmmjvWdgTM8PJHpJS/Lh93no7HXOND0GIj0DJCZ46B0YpKd/+Gw40ethZXEGl2WnUhwOYDB4PIY5WSlkpiQSCiSSkeIj6E+ImwWs9td3sPVwM0ebu+jsHeBocxeHm7o40dJ96mwfnCGa2UE/2al+isMBlhdlUBJOYfHsNNKTfaf/96jdBZsfhV3PQHQAym6FdX/uXLRXXEVBraCeENZaOvsGOdbcxdHmLjpioT2UE+3d/TR39eNP8NDU0UdzZy/9UYvXGFL8CfQNRMlM8Z06ky0MJZOTlsT83CCzM5IIJMZdP+9F6R+M8n5dhD0nI9RFemiI9NIQ6aU+0su+uggtXcO/OLwew7ycVFYWh1iWn+788skL4u+qH7EQVCsUrHY6HhfeBl79d3aD6RjUqampdHR0UFNTw0MPPcSzzz571jHr16/nG9/4BuXlp+fvjBueN1WMMaT6EyjLS6MsL22qixO3fF4Pi2ens3h2+lmPRaOWk+09HGropPpkO02dfVTVtPGznTX8aMtRwPl1smh2GssL76Ls6j/kyo6Xyd/zfzHP/AlkFMEVn4KVHwN/cJJrJvFi9uzZo4b0eFJQy7Tl8RjyM5LJz0jmqnnDV0CPRp1Fvapq2th5rJUdx1p5eusxuvoGgbkkJfwtvx+oYEPkpyx9+Ut0vPxV3gzeQsfy+1lz+VJKtbTujPSFL3yB4uJiHnzwQQC+/OUvY4xh06ZNtLS00N/fz1e/+lXuuOOO0543cuW97u5u7rvvPnbv3k1ZWdm4rfehoJa44/EYisIBisIBblrqTISKRi01bd1sOdjM+3URTrbN5jsDH6Gkp5ob2p7hxsizRDc9x8/fuIIveG6nPbSY+blBFuYF+Z2FuczLSdUCYJPlF190+hfG06ylzrIE57FhwwY+97nPnQrqp59+mpdeeomHH36YtLQ0GhsbWbt2Lbfffvs5+4a++93vEggEqKiooKKigpUrV45L8RXUMiN4PIaCUICCVWdOaloF3AstR+h889vcvOtJ7hx8m+ruy/n3Q7fwj+8t4h9e2ktGwEd5cSbXLsgmN+gnlJLI7IxkZqcnxU2H7ky3YsUK6uvrqampoaGhgVAoRF5eHg8//DCbNm3C4/Fw4sQJ6urqmDVr1qivsWnTJh566CEAli1bxrJl47Pyo4JaBCBUTNqd/wg3/hVse4KyLY/x9fa/429mXcauwnv5yeDVvH6gjVeq6057Wk7Qz4qiDFaXZLJ2TlhLE4yHC5z5TqS77rqLZ599ltraWjZs2MCTTz5JQ0MD27Ztw+fzUVJSMuoSpyNNxBe3glpkpKR0uPIhWPsp2P0CiW//C6t2fYVVgTD2ij+ldsHHaSKd5s4+Djd1suNoK9uPtvBylRPgCR5nHHhZXhori0OsLMpgSX46szO02t90sGHDBh544AEaGxt58803efrpp8nJycHn8/H6669z5MiR8z7/mmuu4cknn+S6666jsrKSioqKcSmXglpkNF4fLL3LuXjBkbdh87cxm75B3m8eIW/Z3bDu01wzv4yPr3MOP9HazfYjLVSfbKe9p59dJ9r5P78+eGqWayjgIxRIZHlRBssLM8gJJnH1vCxSJmmWq4zN4sWLiUQi5Ofnk5eXxz333MNtt91GeXk5y5cvZ+HChed9/qc+9Snuu+8+li1bxvLly1mzZnwu3qxx1CJj1bgf3vkO7HwKBrph7g3OuiJz1o+6EFR33yC7T7ax81gbhxo7aIj08s7B5lOzTX1ew2XZqZTlpTE/N0iq30thZoDVJZkzLsCn4zjqSzFh46iNMV5gK3DCWnvrJZVSZDrKmgu3fhOu+6vhhaB+eCfkLnEm0Cy567SFoJITvawqzmRVceapfYNRS3NnH/vqI2x6v5G9te1sPtDE8ztOnDom1Z/AXasKuGpuFhkBZ82VJJ/WUp/JxnxGbYz5PFAOpF0oqHVGLTPCQK8zPX3zd6B+N6TOgis2OhfnDWRe+PkjRHr66e4b5P26Dp7bfpyfV5w8NXU+McHD8sIM1pZmcnlhBmV5aeTF2WgTnVGPwxRyY0wB8ATwt8DnFdQiI1gLB151AvvAa+ALwPJ7nA7J8GUX9ZKNHb0cjC3K9e6hJrYcaqbyRBuxJm/SkhIIpSQyLyfIwllB1pRmsrI4NGkrO4636upqFi5cGFdfPudirWXPnj0TEtTPAl8DgsBfjhbUxpiNwEaAoqKiVRfqHRWJS3VVTmBXPO0sBLXwFqcdu2jtJV/QINLTz57aCNUn23m/LkJLZz/Vte0caepiMGrxGAgFErksJ5WPLMrl91YWkJkyPdbkPnToEMFgkHA4HNdhba2lqamJSCRCaWnpaY9dUlAbY24FbrbWPmiMWc85gnoknVHLjBephXf/Fbb+G3S3QP4qpx277I5xXwiqs3eA7Udb2Hq4haPNXafCPMFjKMoMcPW8LNYvzGHdnLBr27r7+/s5fvz4Bccox4OkpCQKCgrw+U6/iMalBvXXgI8BA0ASkAb8xFp777meo6AWienrdC4XtvlRaD4A6UXO2tgrPgZJE7eY157adn72Xg17ayO8tb+Rnv4o/gQPK4tCrC4JsbI4RDjFT1E4QHryOF91Ry7KuC1zqjNqkYsUjcL7v4C3vw1H3wZ/Gqz8OFzxScgonNC37ukfZMuhZt7c28C7h5vYXdN+qq0bYF5OKlfPy+Z3V+SzJD8trpse3ExBLeImJ7Y57dhV/+XcX/y7TrNI/vgs4HMhHb0DVBxvpb17gP31Ed493MI7B5roG4ySE/Rz3YIcrluYw8riDLJS/FqMapLowgEibtR6FLZ8D7Y9AX0RKL7S6XicfyN4Jne9kNauPl6pruf1PfVser+BSK9zIYyMgI95Oc6knFXFIVYWhSgIJeusewIoqEXcrKcdtv8AtjwGbccg8zJY9yBc/keQeOZqfxOvfzDK1sPOdPjqk86oksqatth63pAd9LOqKMTK4gxWFYdYPFsTcsaDglpkOhgcgOoXnHbsmu2QnAmr74fVD0Awd0qLNjAYZW9dhO1HWth+tJVtR5wRJuBcRSc76KcsL40PL8phRVGIrFQ/Gck+NZt8AApqkenEWji62WnH3vPz2AJRzkJQ5C6a6tKdUh/pYfsRZ/XAk209bD/SwonW4SuaJHgMyYlectOSWFaQzpLZ6SwrSGdpQTr+BJ2Bn0lBLTJdNR2Adx6FHU86C0Fd9jtOYF92/SVPoBlv1lr21EbYV99BY6SXxo5euvoGOdrcxa4TbTREegGn2LnBJAozkynKTKEkHKA4K4XScAolWQGCSTNzuKCCWmS662oeXgiqow5yFjuBvfQuSPBPdenGpL69h+1HW9lT286x5m6OtXRxtKmL2vbhSS4+r+G2y2eTn5FMKJDInOwULi/IIDRNZlheCgW1SLwY6IXK55x27PoqSM2FNQ9A+f0feCEot+iOnXUfauzktT11vFxVd2opWHDawD+8KJcPL8pl/YJsMgLxGdoKapF4Yy0cfN0J7AOvQkIyrLgHFtzsLLuamuO6ppGxikYtA1FLe08/++o6eLmqlhcramjs6MPrMawqDnFFaSZZqX5WFYcoy0vDGwedlgpqkXhWt9u5oEHF0zDY5+wLZDkdj7lLIHcx5CyCnDLwTc9LgkWjlveOt/JqdT2vVNexty7CUHT5EzzMyU6lbFaQ8pJMwqmJlIRTKMoMkJw4fTotFdQiM0F3K9TuctbGrqt0VvKrr4Z+ZxgdxuOM0T4zwDOKJ32CzaUajFoaIr1s2tfAvjqnA3PX8TaaOvtOOy4r1U9+KJl5OaksL8xgRVEGhZkB0lzYYamgFpmpolFoOeSEdl2V065dVwXNh4DYv/3EVCewzwzw5IwpLfoHFY1ajrd009rdx+GmLo40dnKitZtDjZ3srYvQ2jXc7r1wVpBr52czPzfIkvx05mSn4PNO7ZeVglpETtfXCfV7Rpx5x87Cu1uGj0krcEJ7ZICH5zrjuqcZay0HGzvZWxvhUGMnv97XwNbDLacuPuwxMCc7lZJwgOxgEsXhAMWZAYrCAWanJ5OW7JvwdnAFtYhcmLUQOem0eY8M8Ia9EI2djXoTIWvBiABfHOu8zJ12nZf9g1GONnex42grR5o6qT7ZTk1rD7XtPTSf0YSS4DFkpfoZiEZJ8SewND+dD12WxeLZzmXRwqn+Sw5yBbWIXLyBPmjaN9x8MrRFaoaPSc4cDu2hAM8um5K1SsZDe08/R5u6ONLURV17D40dvTREeknwemjv7mfrkWbq2ntPHR9MSiCcknhqynxhKEBhZjJdfYMUhAKUzQoyJzuV4nCAvsEonb0D9A9YsoP+Ux2e43IVchGZoRISYyG8+PT9Xc2xJpMR4b39ieHOSwxkzjk7wDNKXN95mZbkXP19SX76qI9baznU2MnBhk5Otvewu6ad9u5+LBZr4XBTFzuOtpDiT6Cu/cRp63+P5DGQ5POScYGLNyioReTiBDKh5CpnGxKNQuvhs8++q3/Gqc5LX4ozVHBkgOcsmlYTdowxzMlOZU526gWP7eob4GBDJ/vrOzjR2o3XY0iPtXkfa+6ip3+Q2vZeNp/v/dT0ISITrq8TGvacEeBndl7mx0afjAjw8DznjH4GUNOHiEytxBTnAr/5q4b3WetcBHjksMG6Kjj4xnDnpccH2QvODvBg3rTrvLwUCmoRmRrGQFqes827YXj/YD807js9wI/8BnY9PXxMcsgJ7ZEBnrPQ+UKIQwpqEXEXry/W8bgI+IPh/d0tsaGDIwJ8x39Af2fsAAOZpbEJO4uHO0BDpa7vvLwQBbWITA/JISi50tmGRKPQeuTsmZfVLzLceRkY7rwcGeDTqPNSQS0i05fH45xFZ5ZC2a3D+/u6zui8rHTCe/sPho8J5g2H9lCAZ813ZeelglpE4k9iAPJXOtsQa52LLtRVDjeh1FXBoU3Dqw56EpywPjPA02ZPaeelglpEZgZjIDjL2eae0XnZtP/0oYNHNsOuZ4aPScoYDu+hAM8pA/+Fx1GPBwW1iMxsXp8TujllzqXNhnS3OMvEDjWd1O2GnT+Cvo7hY0KlZwd4Zil4xncdbAW1iMhokkNQ/CFnGzLUeXlq6nwswPf+P7BR55iE5Fjn5chlYxdDSviii6KgFhEZq5GdlwtvGd4/1Hk5MsD3/sIZPjgkddbZy8ZmzR/TxYkV1CIil+qcnZf1TmiPDPAtvz698zI87+wFr86goBYRmQjGQDDX2eb+zvD+wX5oOnB6gB/bct6XUlCLiEwmr8+Z7p6z8PT9nz/38L/pPa9SRGQGUFCLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFzugkFtjEkyxrxrjHnPGFNljPnKZBRMREQcY5nw0gtcb63tMMb4gLeMMb+w1r4zwWUTERHGENTWWgsMrevni212IgslIiLDxtRGbYzxGmN2AvXAr6y1Z01MN8ZsNMZsNcZsbWhoGO9yiojMWGMKamvtoLV2OVAArDHGLBnlmMetteXW2vLs7OzxLqeIyIz1gUZ9WGtbgTeAGyekNCIicpaxjPrINsZkxG4nAzcAeya6YCIi4hjLqI884AljjBcn2J+21r44scUSEZEhYxn1UQGsmISyiIjIKDQzUUTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLXTCojTGFxpjXjTHVxpgqY8xnJ6NgIiLiSBjDMQPAX1hrtxtjgsA2Y8yvrLW7J7hsIiLCGM6orbUnrbXbY7cjQDWQP9EFExERxwdqozbGlAArgC0TURgRETnbmIPaGJMKPAd8zlrbPsrjG40xW40xWxsaGsazjCIiM9qYgtoY48MJ6SettT8Z7Rhr7ePW2nJrbXl2dvZ4llFEZEYby6gPA/wbUG2t/aeJL5KIiIw0ljPqK4GPAdcbY3bGtpsnuFwiIhJzweF51tq3ADMJZRERkVFoZqKIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i43AWD2hjzfWNMvTGmcjIKJCIipxvLGfW/AzdOcDlEROQcLhjU1tpNQPMklEVEREYxbm3UxpiNxpitxpitDQ0N4/WyIiIz3rgFtbX2cWttubW2PDs7e7xeVkRkxtOoDxERl1NQi4i43FiG5z0FbAYWGGOOG2Pun/hiiYjIkIQLHWCt/ehkFEREREanpg8REZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLjemoDbG3GiM2WuM2W+M+eJEF0pERIZdMKiNMV7gO8BNwCLgo8aYRRNdMBERcYzljHoNsN9ae9Ba2wf8J3DHxBZLRESGJIzhmHzg2Ij7x4ErzjzIGLMR2Bi722uMqbz04rlaFtA41YWYYKpjfJgJdYTpX8/icz0wlqA2o+yzZ+2w9nHgcQBjzFZrbfmYizcNqY7xQXWMH/Fcz7E0fRwHCkfcLwBqJqY4IiJyprEE9W+BecaYUmNMIrAB+OnEFktERIZcsOnDWjtgjPkM8DLgBb5vra26wNMeH4/CuZzqGB9Ux/gRt/U01p7V3CwiIi6imYkiIi6noBYRcblxDep4mWpujPm+MaZ+5FhwY0ymMeZXxph9sb+h2H5jjHkkVucKY8zKqSv52BljCo0xrxtjqo0xVcaYz8b2x1s9k4wx7xpj3ovV8yux/aXGmC2xev441lGOMcYfu78/9njJVJb/gzDGeI0xO4wxL8bux1UdjTGHjanp1OAAAAMVSURBVDG7jDE7jTFbY/vi6vN6LuMW1HE21fzfgRvP2PdF4FVr7Tzg1dh9cOo7L7ZtBL47SWW8VAPAX1hry4C1wKdj/7/irZ69wPXW2suB5cCNxpi1wNeBb8bq2QLcHzv+fqDFWjsX+GbsuOnis0D1iPvxWMfrrLXLR4yXjrfP6+isteOyAeuAl0fc/xLwpfF6/cnegBKgcsT9vUBe7HYesDd2+3vAR0c7bjptwAvAh+O5nkAA2I4zs7YRSIjtP/XZxRndtC52OyF2nJnqso+hbgU4QXU98CLORLV4q+NhIOuMfXH7eR25jWfTx2hTzfPH8fWnWq619iRA7G9ObP+0r3fsp+8KYAtxWM9Yk8BOoB74FXAAaLXWDsQOGVmXU/WMPd4GhCe3xBflW8B/B6Kx+2Hir44W+KUxZltsyQqIw8/raMYyhXysxjTVPA5N63obY1KB54DPWWvbjRmtOs6ho+ybFvW01g4Cy40xGcDzQNloh8X+Trt6GmNuBeqttduMMeuHdo9y6LStY8yV1toaY0wO8CtjzJ7zHDtd6ziq8Tyjjvep5nXGmDyA2N/62P5pW29jjA8npJ+01v4ktjvu6jnEWtsKvIHTJp9hjBk6URlZl1P1jD2eDjRPbkk/sCuB240xh3FWt7we5ww7nuqItbYm9rce5wt3DXH8eR1pPIM63qea/xT449jtP8Zp0x3a//FYL/NaoG3op5ibGefU+d+AamvtP414KN7qmR07k8YYkwzcgNPh9jpwV+ywM+s5VP+7gNdsrJHTray1X7LWFlhrS3D+3b1mrb2HOKqjMSbFGBMcug18BKgkzj6v5zTOjf03A+/jtAH+j6lugL+EejwFnAT6cb6Z78dpw3sV2Bf7mxk71uCMdjkA7ALKp7r8Y6zjVTg/BSuAnbHt5jis5zJgR6yelcD/iu2fA7wL7AeeAfyx/Umx+/tjj8+Z6jp8wPquB16MtzrG6vJebKsaypd4+7yea9MUchERl9PMRBERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURc7v8DuPkmJy5ntEsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy], callback_fns=[ShowGraph])\n",
    "learn.fit_one_cycle(3, max_lr=1e-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see clearly max_lr=1e-2 is better than max_lr=1e-1. Athough this test took around 11 minutes but we easily got the value of lr that we should use. With enough practice you can predict the best value of max_lr from directly the `lr_find` graph but whenever you are confused running some epochs can help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introducing Superconvergence\n",
    "Now we know what is 1cycle policy and how we can use it for our own work in fastai, next I want to discuss **Superconvergence** which is basically what we are getting using the 1cycle policy.\n",
    "\n",
    "<img src='Extra/ResNetCifar10Range1.png' width='60%'>\n",
    "\n",
    "<center>Test accuracy for learning rate range = 0.001 - 1 for ResNet-52, ResNet 56 and ResNet 60</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two noteworthy things to see in this figure. First is the dip in the accuracy around LR=0.1. Second is the consistently high test accuracy over a large span of learning rates (LR=0.25 to 1.0), which is unusual. Seeing these unsual facts, experiments were carried out using cyclic learning rate and the following counterintuitive things results appeared."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='Extra/Cifar10Resnet56TBS1000CLR35SS10k.png' width='700'> | <img src='Extra/Cifar10Resnet56TBS1000LR35CLR1.png' width='700'>\n",
    "- | -\n",
    "(a) Cyclical learning rate between LR=0.1 and LR=0.35 with stepsize=10K | (b) Super-convergence with CLR = 0.1-1.0 versus standard training with initial LR=0.35\n",
    "\n",
    "<center>Fig: Test accuracies for Resnet-56 on CIFAR-10. Note the log scale used for the vertical axis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see an anomly that occurs as the LR increases from 0.1 to 0.35. The training loss increases sharply by four orders of magnitude at a learning rate of approximately 0.255 but training convergence resumes at larger learning rates. In addition, there are divergenct behaviours between test accuracy and loss curves that are not easily explained.In the first cycle, when the learning rate is increasing from 0.13 to 0.18 the test loss increases but the test accuracy also increases. This simultaneous increase in the test loss and the test accuracy also occurs in the second cycle as the learning rate decrease from 0.35 to 0.1 and in various portions of subsequent cycles.\n",
    "\n",
    "Another interesting fact can be see from the second figure. The cyclic learning rate method is able to get 0.93 accuracy after just once cycle and it remains the same for the subsequent cycles, while the standard approach of using a constant learning rate manages to achieve the accuracy close to 0.93 at about 5 times more iterations.\n",
    "\n",
    "### **Superconvergence**\n",
    "Super-convergence refers to this phenomenon where a network is trained to a better final test accuracy compared to tradional training, but with fewer iterations and a much larger learning rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing linear interpolation results\n",
    "In the *Exploring Loss Function Topology with Cyclical Learning Rates* (only 3 pages) an important topic of networkinterpolation is discussed which shows that the solution found by each cycle is different from each other (intuitively you can think the solutions as belonging to different valleys). So we can interpolate solutions from different cycles and we can except better accuracy. So I test this fact now?\n",
    "\n",
    "Interpolation between two values is defined as \n",
    "\n",
    "<img src='Extra/interpolation.png' width='300'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How it was found in the first place?\n",
    "Before getting into code, I would discuss how this fact was found. So interpolations between different cycle weights was tested and the plots for interpolation were drawn for the train and test loss. One of the figure is shown below\n",
    "\n",
    "<img src='Extra/test_loss_normalTrainResNet_10k_to_25k_plot.png' width='600'> | <img src='Extra/test_loss_resNet2_40k_to_60k_plot.png' width='600'>\n",
    "- | -\n",
    "(a) Training and test loss for interpolation between two network states from regular ResNet training. If both the values are for the same minima than this is what we would get, two concave shapes. | Training and test loss for interpolation between two CLR solutions. If there is a peak between two solutions then the two minimas are different solutions.\n",
    "\n",
    "<center>Fig: Results of network interpolation for differently trained models</center>\n",
    "\n",
    "From this plot it was observed that the solution found by each cycle indeed belonged to different minima. Also an addional noteworthy feature, some amount of regularization is possible through interpolating between two solutions. The minima for training loss are at `a=0.0 and 1.0` but the test loss minima are slightly offset towards the center."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coding Linear Interpolation\n",
    "To check the results for the above topic the following things would be done:\n",
    "1. Do some training epochs on the training dataset and get the weights for diferent cycles\n",
    "2. Make a plot of interpolation between different weight values\n",
    "\n",
    "To modify our training loop callbacks would be used. I consifer callbacks to tbe the most important feature of fastai as it allows infinite customization without chaning the training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GetWeights(Callback):\n",
    "    def __init__(self, learn, save_list=[5, 7]):\n",
    "        self.learn = learn\n",
    "        self.save_list = save_list\n",
    "        \n",
    "    def on_train_begin(self, **kwargs):\n",
    "        self.weights = {}\n",
    "        \n",
    "    def on_epoch_end(self, **kwargs):\n",
    "        if kwargs['epoch'] in self.save_list:\n",
    "            weights = learn.model.state_dict()\n",
    "            for k, v in weights.items():\n",
    "                weights[k] = v.cpu()\n",
    "            self.weights[kwargs['epoch']] = weights\n",
    "    \n",
    "    def get_weights(self):\n",
    "        return self.weights\n",
    "        \n",
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy], callback_fns=ShowGraph)\n",
    "getWeights = GetWeights(learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.191520</td>\n",
       "      <td>4.533091</td>\n",
       "      <td>0.053900</td>\n",
       "      <td>0.195400</td>\n",
       "      <td>01:54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.791083</td>\n",
       "      <td>4.001433</td>\n",
       "      <td>0.102000</td>\n",
       "      <td>0.318300</td>\n",
       "      <td>01:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.429356</td>\n",
       "      <td>3.574863</td>\n",
       "      <td>0.161400</td>\n",
       "      <td>0.429000</td>\n",
       "      <td>01:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.125475</td>\n",
       "      <td>3.035374</td>\n",
       "      <td>0.231600</td>\n",
       "      <td>0.533600</td>\n",
       "      <td>01:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.832539</td>\n",
       "      <td>2.619795</td>\n",
       "      <td>0.312300</td>\n",
       "      <td>0.643800</td>\n",
       "      <td>01:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.584690</td>\n",
       "      <td>2.346036</td>\n",
       "      <td>0.374700</td>\n",
       "      <td>0.704900</td>\n",
       "      <td>01:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.407087</td>\n",
       "      <td>2.109318</td>\n",
       "      <td>0.422400</td>\n",
       "      <td>0.756300</td>\n",
       "      <td>01:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.304121</td>\n",
       "      <td>2.082097</td>\n",
       "      <td>0.429800</td>\n",
       "      <td>0.761200</td>\n",
       "      <td>01:48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXwV1f3/8dfJTjaykEBIAknYZN+CsskiiKi4VLFSrdpq1aqttYvf2vrza+2u9ttW64paW5UuSqtWKrgiIAiSgCAa1pCQhUBCVrIv5/fH3GDQAEHuTSbh/Xw88si9M3PvfO6QvJmcOeeMsdYiIiLu5dfVBYiIyPEpqEVEXE5BLSLicgpqERGXU1CLiLhcgC/e1D+0tx0yKJXwYJ+8vYhIj5OZmVlirY1rb51PkjSgdzz1C37FsKQoFk5M4rwRfYmPDPHFrkREegRjTO4x1/miH/XY8RPsJT97juc+cPYbFODHhaMTuOqsAaQPjMYY4/V9ioh0Z8aYTGttervrfBHU6enpNiMjg+YWy7aCCl7KzOOVzYUcrm9iSHw4104ZyEVj+xMVGuT1fYuIdEddFtRt1TQ0sWzLfl7YkMvW/ArCgwO4dfYgrp+WSkigv9drEBHpTlwR1K2stXxcUMHD7+zi7ayDxEUEc/VZA7h2SgoxYTrDFjkdNTY2kp+fT11dXVeX4nMhISEkJSURGBh41HJXBXVbG7IP8eh7e1i9s5iwIH9unjmIG6anEqbeIiKnlb179xIREUFsbGyPvoZlreXQoUNUVVWRmpp61LrjBXWX9qM+Ky2W564/k+XfO5vpQ/rw+7d2MvPBlbyTdaAryxKRTlZXV9fjQxrAGENsbOxJ/+XgigEvwxMiefKadF6+dSrxESHc8NcM/mfpFsqqG3yzw+oSWH4XVB/yzfuLyEnr6SHd6st8TlcEdavxA6L5961TuXlGGv/eVMD5D61h3Z4S7+9o7yr4cDH8aQJsWAzNTd7fh4iIl7gqqAFCAv35yQXDeeW2aYQG+XP10xt48I3tNDa3eG8noy6Hb78PCWNg+Z3w5Nmwd7X33l9EupXy8nIee+yxk37dBRdcQHl5uQ8qOprrgrrVqMTevPbd6VwxMYlHV+7hqqfWk1NS7b0d9B0B1/4Hvvo81B+Gv14EL14L5fu8tw8R6RaOFdTNzc3Hfd3rr79OVFSUr8o6wrVBDRAWHMADC8fy0KJxZO2v4qJH3vfuhUZjYMTF8J0PYfbdsPNNeGQSrPwNNNR4bz8i4mp33XUXe/bsYdy4cUyaNInZs2dz1VVXMXr0aAAuvfRSJk6cyMiRI1m8ePGR16WkpFBSUkJOTg7Dhw/nxhtvZOTIkcybN4/a2lqv1del3fNORn5ZDTc+l0nW/kouHJPALy8ZRbS3+12X58Fb98AnL0PvZJj3SxhxiRPoIuIzWVlZDB8+HID7XvuETwsrvfr+I/pHcu9FI4+5PicnhwULFrBt2zbee+89LrzwQrZt23akC11paSkxMTHU1tYyadIkVq1aRWxsLCkpKWRkZHD48GEGDx5MRkYG48aN46tf/SoXX3wxX//610/4eVu5tnveyUiKDuWV26byw3OH8tYnB7j40fe9/o9JVDJc8Rf4xn8hpDe8dJ3TJHLgE+/uR0Rc7cwzzzyqn/PDDz/M2LFjmTx5Mnl5eezatesLr0lNTWXcuHEATJw4kZycHK/V061GlgQH+PPdOUM4e2gc334+k0sfW8tvvjKayycmeXdHKdPhplWw6S/w7i/hiekw6Vsw6ycQGuPdfYnIUY535ttZwsLCjjx+7733ePvtt/nggw8IDQ1l1qxZ7faDDg4OPvLY39/fq00f3eaMuq1xyVEsu3066QOj+eFLW3jwje00t3i5Ccc/wAnn726C9Oth49Pwp4mw8RloOf4FBhHpXiIiIqiqqmp3XUVFBdHR0YSGhrJ9+3bWr1/fydV106AG6BMezLPfnMSV6ck8unIPFz68hhXbivB6m3toDFz4f3DzGogfAf/9ATw5E3LWenc/ItJlYmNjmTZtGqNGjeLOO+88at38+fNpampizJgx3HPPPUyePLnT6+s2FxOP5+XN+fzp3d1kF1czdVAsf7xynG9uVGCtc6HxzXugMt/pj33uz6G3l5teRE4z7V1c68l67MXE4/nK+CTevGMGv7h0FJv2lXH+Q2v4z5ZC7zeHGAOjLoPvbISZP4bt/3W68616EBp7/qxfItI1ekRQAwT4+3HN5IG89p3pxEeGcPvfN3P54+vYXuTlniEAQaEw+6dw24cweC6s/CU8eiZkveacdYuIeFGPCepWQ/pGsOy70/m/K8ayr7SGBQ+/z8Pv7KK2wQcXAKMHwpXPOyMcA0Phn1+H5y+Fg9u9vy8ROW31uKAG8PczXD4xiXd+MJP5o/rx+7d2ctav3+bpNdnUNfogsNNmOnOHnP8AFG6Gx6c6s/PV+n4OABHp+XpkULeKDgvikasmsPTbUxibHMUv/5vFjAdW8q/MfO/3DvEPgLNudrrzTbgWNjzhzM6X+Vd15xORU9Kjg7pVekoMz11/Ji/ccBaJ0b344UtbuP4vG8kv88F8HmF94KI/ws2roM9QeO12eOoc2LfB+/sSkdPCaRHU4EzWPX1IH/717ance9EI1meXMuOBlTywYjsNTV6cQrVVwlj45nK4/Bk4fBD+PA/+fRNU7vf+vkSk04WHhwNQWFjIwoUL291m1qxZeKOr8mkT1K38/AzfnJbKG3fMYMGY/jz23h4ufXQtW/N90J5sDIxe6HTnO/tHTh/sP02ENb+Hpnrv709EOl3//v1ZunSpT/dx2gV1qwGxoTz8tfEsvmYiB6vqWfjEB/xnS6H3264BgsNhzj1w2wZImwXv3AePngU7lqs7n4hL/PjHPz5qTuqf/exn3HfffcyZM4cJEyYwevRoXn311S+8Licnh1GjRgFQW1vLokWLGDNmDFdeeaXX5vvoVpMy+cK8kf1IT4nhpucyuP3vm3lmTTa3zxnCOWfEe/8ebjFp8LW/we53YMVd8PdFTj/s+b+FPkO8uy+R7mr5XVD0sXffs99oOP+3x91k0aJF3HHHHdx6660AvPjii6xYsYLvf//7REZGUlJSwuTJk7n44ouPmQ2PP/44oaGhbN26la1btzJhwgSvlH/anlG3FRMWxJIbz+IXl4yksq6JG/6awWWPr2P5x/u9P7oRYPAcuGUdnPcbyPsQHpsMb9wNdT4YnCMiHTJ+/HgOHjxIYWEhW7ZsITo6moSEBH76058yZswY5s6dS0FBAQcOHPvmJatXrz4yB/WYMWMYM2aMV2o77c+oWwUH+HPNlBSuSE/mpcx8nlqdzS1LNpEU3YsrJiZz5aRk+vX24vwh/oEw5VYYfYXTFPLBo7D1RZh7L4y9Cvz0f6icpk5w5utLCxcuZOnSpRQVFbFo0SKWLFlCcXExmZmZBAYGkpKS0u4Up2354m7qHU4DY4y/MWazMWaZ16twkZBAf66ZPJCVP5rF41dPYEBMKH94eyezfreSX7+eRdZ+L5/1hsfBJY/Aje9CdAq8ehs8MxfyO29SKxFxLFq0iH/84x8sXbqUhQsXUlFRQXx8PIGBgaxcuZLc3Nzjvn7GjBksWbIEgG3btrF161av1HUyp23fA7K8stduwN/PcP7oBP5242RW3TmL+SP78fSabM5/aA1fW7yejTml3t1h4gS4/g34ypNQUQBPz4GXb4EqL94jUkSOa+TIkVRVVZGYmEhCQgJXX301GRkZpKens2TJEs4444zjvv6WW27h8OHDjBkzhgceeIAzzzzTK3V1aJpTY0wS8FfgV8APrLULjrd9Z09z2lnySmt4/eP9PL5qD5W1jcwZ3pdvTkth6qA+3t1RfRWs/p3THOIfBBO/AVNug96J3t2PiEtomlPvTHP6R+B/gGOODDHG3GSMyTDGZBQXF3e03m4lOSaUm2cO4v0fn8O3zk4jM7eMq57awKLFH7Bud4n3uvYFR8C59znd+YZf5AxHf2gsvHIbFO/0zj5EpNs4YVAbYxYAB621mcfbzlq72Fqbbq1Nj4uL81qBbhQeHMBPLxjOurvO4Z4FI8guruaqpzdw6aNrWZqZT32Tl+b2iB0Elz0J3/vIuR3Ytn8506n+42rIP+4/h4j0ICds+jDG/Aa4BmgCQoBI4N/W2vbvg07Pbfo4lrrGZl7KzOcva/eyp7iaqNBALhufxNzh8Uwd7MVmkeoS2PAkfPgk1FVAytkw/fsw6BxnFKRIN5WVlcUZZ5zhkx4TbmOtZfv27SfV9HFSt+IyxswCfnS6tlGfiLWWNbtKeHTlbjJyy2husYxKjCQ2LJjkmF4M7RvBxWP7ExUadGo7qq9yZuX74BGo2g/9xjiBPeIS8PP3zocR6UR79+4lIiKC2NjYHh3W1loOHTpEVVUVqampR61TUHeBusZmnnl/L+uzD5FzqJoDlfU0NLUQHhzA3OHxzB/Vj3kj+uHndwo/lE31Tt/rtX+EQ7udkY9Tb4exX4NAH9wzUsRHGhsbyc/PP2Ef5Z4gJCSEpKQkAgMDj1rutaDuKAX1FzW3WLL2V/Ls2hxW7TxIyeEGxiVHcf/lYxjWL+LU3rylGbYvg/f/4Ny4ILwvTL7VadcOifTOBxARn1JQu0xzi+XlzQX8YtmnVNY1Mm9EX8YlR3NmagwTBkR9+T/9rIW9q5zAzn4PgnvDpBtg8i0QHu/VzyAi3qWgdqmy6gYeX7WHZVsKKaxw/uQbmxzFnxaNZ0Bs6Km9ecEmp0nk0/84fbHHfx2mfhdiUk/8WhHpdArqbqC8poHXPy7it8uzqG9qYWxSFFMHxzJ1UB/SB0Z/+bbskt2w7iH46O9gm2HkZTD9Dmc2MRFxDQV1N5JfVsMz7+9l1Y5iskuqAUiM6sW1UwZy7ZQUegV9yV4dlYWw/jHIeBYaDsOQeU5PkQFT1LVPxAUU1N2QtZY9xdV8lFfO8+tz2ZJXTlRoIJeOS+TmmWkk9O715d64tgw2Pg3rn4CaEkg+ywnsIedpxj6RLqSg7gEyc0t5dm0Ob35yAGPg7guHc+WkZIIDvuQZdkMNfLQE1j4MFfsgbrjTJDLqcmcKVhHpVArqHmR7USXXP7uRwoo6kmN6cc6weBKiepESG0pyTChD+0YQ6H8SZ8bNjc69HN//Axz8FHonOxcdx18DQad4QVNEOkxB3cNYa3lvZzFPrc7mw72lNLW5C83g+HCum5rCucP7ntyNDqyFXW86N97NWw+hsXDWt2HStyA0xgefQkTaUlD3YPVNzdQ1tpB7qJoN2aX87cN97PVchBzZP5Kpg2KZN7If45OjCOjomXbuB84Z9q43IDAM0r/pDKDRNKsiPqOgPo1Ya9l18DCvbC5g7Z5DfFpYQWOzJa1PGL++bDST02I7/mZF22DtQ86sfcYPxl4JU78HcUN99wFETlMK6tNYVV0jyz8u4pGVu9lXWsP4AVGMTYrittmDiYsI7tiblOXAukdg8/PO/CLDFzg9RRIn+rR2kdOJglqobWjmqTXZvL+rhM15ZQT5+3HTjEF86+xUwoI7eI/jw8XOTQw2PuVMs5o6wwnstNnqiy1yihTUcpTs4sM8+MYOlm8rIjYsiMsnJjFhQBRnD4nrWGjXVULmX5xbhR0ugoSxMONOOGOBAlvkS1JQS7syc8t48I3tbNhbirXQK9CfS8b156cXDicypAN9qZvqYcs/nHbs0j2QOhMueBDihvm+eJEeRkEtx1XX2MyWvHJe+aiQFzPyiI8I5spJyVw/PbVjgd3cBJnPwru/gIZqp1vfzB9rilWRk6Cglg7btK+M/311G58UVmItXDN5IN85ZzB9IzvQJ7u6BN65DzY970yreu4vYMxX1Rwi0gEKajlpm/aV8av/ZpGZWwbA0L7hzD4jnnkj+jFxYPTxX5yfCa//CAo3QfJkpzkkYUwnVC3SfSmo5Uux1vJJYSWvbSnkpcx8SqsbAJhzRjy3zh58/MBuaYGPXoC3f+ZMBJV+Pcy+W6McRY5BQS1eUVnXyENv7+LFjDyq6pqYNSyOH5w7lKF9IwgJPMbkULVlsPI3Tpe+kCiYe68zj4huwityFAW1eFV1fRPPfZDLYyt3U1XfRHhwAD+cN5Rrp6Tgf6wbHBRtg9fvhH3roP94uOB3kNTuz6TIaUlBLT5RVFHHuj0lvLy5gDW7Shid2Jtff2U0o5N6t/8Ca+HjpfDWPVC1H8Z9Heb+DMLjOrNsEVdSUItPWWtZtnU/P1/2KYcO13PtlBSuSE9iREJk+zfqra+C1Q86A2YCw2D2T51Z+vw7OEJSpAdSUEunqKht5Hdv7OCFDblYCwm9Q0iM6sXsM+I5MzWGEQmRR498LN4Jy/8HsldC/Aind0jK9K77ACJdSEEtnWp7USWvb93Phzml1DY0syW/AgA/A0P7RjBneDwXju7PGf0i8DPA9mWw4qfOnWZGXe70v9aUqnKaUVBLl8ovq2HngSo+yqtg/Z5DfJhTCkBIoB/pA2P4xtQU0hODidr0uDMPtl8AzLwTJt8GAUFdXL1I51BQi6vkl9WwIbuUtbtLeG9nMaXVDQT6G+aN7Mft4wMY+tFvMDteh9jBcP79MHhuV5cs4nMKanEtp2mknFc2F7B8WxEVtY0MT4jk0UklpGX8wpns6YwFcN6vIDqlq8sV8RkFtXQLJYfreXlTAc+u3UthRR0j44P5dcJqxuxZjMHCtDucO6UH9urqUkW8TkEt3UpZdQMvrM/lHxvzKCivJdm/lAd6L2VKzXtUBPen+dxfETX+UvxO5m7rIi6noJZuqbG5hU8LK/nPlkLW7TlESlUmdzQ8xTC/fNYylsLJP+PyebPxO9ZoSJFuREEtPcbOwlJ2LPsD5xQ+TaCt5wVzIdnDb6WiJYSm5hbmDu/LZRMS2x9oI+JiCmrpcZoqiij4110M3PcyB2w0TwR/k1ebplBa00hiVC8um5DI9+YMIUDNI9JNKKil58rbCK//EPZvwQ6YyooBP+CFnAjW7j7E8IRIrkxPYtrgPgzpG9HVlYocl4JaeraWZtj0nHN3mbpKOPNG/hv7DX7+dgEHKusBGJscRf/eIcwf1Y9LxmnUo7iPglpODzWl8O4vIePPEBpLy5x7eT/8PN7dUcIHew6x40AVAOeP6scdc4cytG+42rLFNRTUcnop/MiZ7ClvAyROdOa+TpxAQ1MLD6zYztPv7wUgJiyIK9KTuGx8EsP6qWlEupaCWk4/1sLWf8Kb90B1MUy4FubcC2GxFFfVs2xrIf/4MO/IWfbYpN5cMi6Ri8f1p094cBcXL6ejUwpqY0wIsBoIBgKApdbae4/3GgW1uEZdJay6H9Y/DsHhMONOmPgNCHbOoHcfrOL1j4t4ZXMB2SXVAEwdFMtVZw1g3oh+BPobNY9IpzjVoDZAmLX2sDEmEHgf+J61dv2xXqOgFtc5mAUr7oLs9yC4N0y6Hs76NkT0A6ClxbJ6VzGb9pXz6kcF5B6qIcDPYIEZQ/qwcGIyM4fFER6smxuIb3it6cMYE4oT1LdYazccazsFtbhWfiasewiyXnOmUx3zVZh6O8QNO7JJY3MLb3xSxKeFlVTUNvKvTfnUNbaQEhvK09elMzhe7dnifacc1MYYfyATGAw8aq39cTvb3ATcBDBgwICJubm5p1S0iE+VZju3Atu8BJpqYeh8mPpdGDgNPtfUkVdaw+pdxfzhrZ3UNbbw/y4czvmjEugdGthFxUtP5M0z6ijgZeC71tptx9pOZ9TSbVQfgo1PwYeLoeYQ9J8A026H4ReDn/9RmxaU13Lbkk18lFeOMXDRmP7MGBpHXEQwZ6XGABAc4Kc2bflSvNrrwxhzL1Btrf3dsbZRUEu301ADW/4G6x6Bsr3O3NdTvgPjroag0CObNbdYNu8r489r97JqRzHVDc1Hvc34AVF8a3oa547oS1CAhq9Lx53qxcQ4oNFaW26M6QW8CdxvrV12rNcoqKXbaml27uG49mEoyIBeMXDmjXDmTRDW56hND9c3se9QDXuKD7NqZzG7Dx4mv6yWksP1xEcEM35AFJNSYjijXyRNLS1U1jUxJrE3A2NDddYtX3CqQT0G+CvgD/gBL1prf3681yiopduzFvZ94AT2zuUQEALjrnLOsmMHHfNlzS2WVTsP8sL6fWzNL6fkcMMXtokODSQk0J+Lx/XnhmmpxEeG+PKTSDehAS8ip6J4B6z7kzOAprkRhi+Aqd+D5EknfGlpdQPrsw9RXd/EGf0i+biggrV7Svjv1v1HthmREMkPzh3K3BF9ffkpxOUU1CLeUHUAPnwSNj4NdRUwYIrTtW/ofPA7ufbo+qZmPs6vYMW2Iv6zpZCDVfVEhAQwOrE380b0ZeaweFL7hPnog4gbKahFvKm+CjY9D+sfg4o86DPUaRIZcyUEnnwzRmNzC//cmMdf1uWw++DhI8tnD4vj0vGJzB/Vj+AA/+O8g/QECmoRX2huhE9fhbUPQdFWCIuHs26GSTdAr+iTfjtrLYeqG9hWUME7WQd5O+sA+yvqGNk/khvPTmP2sHjCQwLw163HeiQFtYgvWQt7VzkXHve8A4FhziRQU26FqAFf+m1bWiyvfFTA/Su2H5lXOyTQjx+cO5RrJqfQK0hn2T2JglqksxRtcy48blvqBPjIrzgDaBLGfum3bGmxbM4rZ332ITJzy3h3+0H8DKTFhTMpJZqFE5OYMCBaXf66OQW1SGeryHdm7Mv8KzRUQepMJ7AHzfnCEPWTYa3l7ayDvP7xftbuLuFglXOmPSUtlrsvHE58ZDCRIU73P+leFNQiXaW2HDL/AhuegKr90HeUM6fIqMvB/9TnCjlc38S/MvN5YMX2I6MkA/0NoxJ7c+HoBOaN6EdyTC+dbXcDCmqRrtbUAB+/5DSLFGdBZCJMvgUmXAchkaf89iWH63ltSyEA+0preHZtzpF1MWFBzBvRl4vH9mdAbCjRoUGEabpW11FQi7iFtbDrLVj3MOSsgeBI50YGk2+ByP5e201jcwsbskvZkl/Om58eIKuwkobmFgD8DFwwOoEpg2JJHxhDY3MLfSNDiIvQnW26koJaxI0KNjmB/emrYPxh9BVOs0jfEV7fVXFVPVn7KykoryUzt4x3sg5QVtN41DbTBseS0LsXV05KJn2gLk52NgW1iJuV7nUGz2x+ARprYPC5MPE6SJvt3D7MB1paLGt2l7CvtIbIkACWbd1PQVktWUWVWAsTBkTxwMKxBAf4UVrdQFpcGBEhmn/blxTUIt1BTakzPH3Dk1BTAv5BkDrDGaI+7HzoneTzEkoO17NsSyH3r9hBbeNnU7iGBfkzdXAfLhrbn3OH91Ufbh9QUIt0J82Nzsx9O1Y4M/eVZjvL+46GYfNh6PnQf/xJzy9yMvJKa1i7uwR/P0NIoD+vflRARm4Z5Z7mkuAAP6JDg+gfFcIN09OYOyJew9xPkYJapLuyFkp2OYG9YwXkrQfbAuF9Ycg850w7bRYE+X4Cp+r6Jt7dfpB1e0o4dLiB4EB/NuWWUVBeS1xEMBeOTuCmGWn0j+rl81p6IgW1SE9RU+r0Gtm5HHa/A/WVzlzZrU0kQ+dD78ROK6e5xbJy+0GeWpNNZm4Z/n6GG89O47tzBusM+yQpqEV6oqYG2LfusyaSshxnecJYp3lk2HxIGHdKIyFPRn5ZDb9Zvp3/bt1PZEgAP5w3jOSYXkwYEE1UaFCn1NCdKahFejprnRsc7Hgddq6AvA8BCxEJMPQ8J7jTZkKgb5slWoe4/3Z5FnuKqwEICvAjqlcgdY3NjBsQzTemDmRyWiyhQRp005aCWuR0U10Cu96EHcthz7vQcBgCejnt2cM8TSQR/Xy2+6bmFvZX1LG3pJo1u4rZeeAwNQ1NbC+qoqquiYjgABaM7c+8EX1pbG6hxcKMoX1O6/BWUIuczprqIed950x7xwqo2Ocs7z/+syaSfmM6pYmkoqaRlTsOsmJbEWt2HX0X9/DgAMYPiGJ4QiQHK+vIOVQDOGfpVXVNxEUEYy2k9All+pA4hsSHk9onrMdMQKWgFhGHtXDwU+dMe+cKyM8ArDP3SGsTSeqML3WnmpNVVddIZm4ZMWFBlNc08sYnRWTmlrG9qApw5t6eODAaayEqNJADlfXsK62h2DNjYKsz+kVw88w0Lh2X2K1HUyqoRaR9hw+2aSJZCY3Vzo0PBs329CI5D8LjO7WkhqYW/P0MzS2WoIAv9hWva2xm875yNuaUUl3fxFtZB8guriY0yJ8RCZEE+BtiwoKobWgmOjSI+uYWQgP9GdE/klkuvhelglpETqyxzpkoqvVsu7IAMJA48bOBNn1Hdlovko5qbrG8mJHHvzflU1HbSG1jM3mltQAkRfeipqGZ0uqGI9vPHR7PvReNJDkmtKtKbpeCWkROjrVQ9LGnXXs5FG5ylvdO9gxpnw8pZ0OA+2bcs9ZiLfh57i1praX4cD0f7StnS345z7y/l7rGFi6fkMSCsQlMGBBN715dP4+JglpETk1VEex8wwnuPSuhqRaCwiFlutOmnToT4kf4dFi7t+QequaRd3fzUmY+4MxjMmtYPNOH9CEuPBg/P4iPCCG/rIbymkZG9I9kZP/ePr+psIJaRLynsRb2rnZCO3sVlO5xlof2gdSzndBOnQExaa5rJmlrb0k1uw5UsWJbESs+KaKmTQ+U9iT0DmH64D4MT4ikV5A/heW1FFXUYYF9h2oIC/Zn2uA+pMSGkRTTi9iwYPz9DKFB/gQH+NFiwd/PUNfYzIa9pWTmlJJfVktwoD+D4sK4ccYgBbWI+Eh5nhPce1c7d2Ov2u8s7538WWinzoDIhK6t8zgam1vYeaCKA5V1lFU34ucHaX3CafAs376/inV7So4M4mnVJzwYP+P0SjlYVX9k0qq2Wi+MAvQK9KehueXI89AgfwL9/aiobST3/gUKahHpBNbCod2Q/Z4T2nvXQF25s67PUCe402bCwGkQGtOlpX4Z1loOVtVTVFHHsH4RR/XhttZSUF7LnuJqPi2sxGJpbLLsK63Bz0DmvjLGJPYmKMCPKYNimTEkjpiwIIwxrN5ZzMxh8QpqEekCLc3ORcm9q5wz7tx1zs0RMM6cJKkznOAeMKVTZgB0M9xaLCgAAApKSURBVLVRi4g7NDVAQaYT3NmrIH8jtDSCXyAkTXJCO3UGJKZDwOk1kZOCWkTcqaHauUlCtueMe/8WwDqDbgZO+axHSb/R4Nczhoofy/GC+vSdAUVEul5QGAye63yBM9927lpPcK+Ct/7XWR4S1aZHyUzoM8TVPUq8TUEtIu4RGgPDL3K+ACr3f9abJHsVZL3mLI9I+OxsO21mp9xPsispqEXEvSITYOyVzpe1zv0jW4N799uw9Z/OdjFpR3cFDOvTtXV7mdqoRaR7amlxZgJsPdvOXevMuw3OjYBbe5QMnArBEV1bawfoYqKI9HzNjVC4+bPgzvsQmj1TovoHOfeWbP0eENzm63jL2zwOaLPNUcuP9z5ttvc/fgOGLiaKSM/nHwjJZzpfM+50hrrnbYC8jc6ZdlM9NNVBc4Pzvan+s2WNtc7AnNbnTW22aa6HlqZTr8/4Hz/wj0NBLSI9U6Dn1mNps079vZqbnMBuG+5fCPzW5fXHDvwjy9t5n+M4YVAbY5KB54B+QAuw2Fr70Kl/chGRbsI/wPny5ejJ647d3bAjZ9RNwA+ttZuMMRFApjHmLWvtp96qT0REju2Ek8daa/dbazd5HlcBWUCirwsTERHHSc3ybYxJAcYDG9pZd5MxJsMYk1FcXOyd6kREpONBbYwJB/4F3GGtrfz8emvtYmtturU2PS4uzps1ioic1joU1MaYQJyQXmKt/bdvSxIRkbZOGNTGGAM8A2RZa3/v+5JERKStjpxRTwOuAc4xxnzk+brAx3WJiIjHCbvnWWvfB06f+QRFRFzG/fd2FxE5zSmoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLjcCYPaGPNnY8xBY8y2zihIRESO1pEz6r8A831ch4iIHMMJg9pauxoo7YRaRESkHV5rozbG3GSMyTDGZBQXF3vrbUVETnteC2pr7WJrbbq1Nj0uLs5bbysictpTrw8REZdTUIuIuFxHuuf9HfgAGGaMyTfG3OD7skREpFXAiTaw1n6tMwoREZH2qelDRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXK5DQW2MmW+M2WGM2W2MucvXRYmIyGdOGNTGGH/gUeB8YATwNWPMCF8XJiIijo6cUZ8J7LbWZltrG4B/AJf4tiwREWkV0IFtEoG8Ns/zgbM+v5Ex5ibgJs/TemPMtlMvz+f6ACVdXUQHqVbfUK2+oVpP3sBjrehIUJt2ltkvLLB2MbAYwBiTYa1N73B5XaS71Amq1VdUq2+oVu/qSNNHPpDc5nkSUOibckRE5PM6EtQbgSHGmFRjTBCwCPiPb8sSEZFWJ2z6sNY2GWO+A7wB+AN/ttZ+coKXLfZGcZ2gu9QJqtVXVKtvqFYvMtZ+oblZRERcRCMTRURcTkEtIuJyXg1qtw01N8YkG2NWGmOyjDGfGGO+51keY4x5yxizy/M92rPcGGMe9tS/1RgzoZPr9TfGbDbGLPM8TzXGbPDU+U/PxVyMMcGe57s961M6s05PDVHGmKXGmO2e4zvFxcf1+55//23GmL8bY0LccmyNMX82xhxsO+7gyxxHY8x1nu13GWOu68RaH/T8DGw1xrxsjIlqs+4nnlp3GGPOa7Pc5znRXq1t1v3IGGONMX08z7v0uHaItdYrXzgXGvcAaUAQsAUY4a33/5I1JQATPI8jgJ04w+AfAO7yLL8LuN/z+AJgOU7f8cnAhk6u9wfA34BlnucvAos8j58AbvE8vhV4wvN4EfDPLji2fwW+5XkcBES58bjiDNjaC/Rqc0y/4ZZjC8wAJgDb2iw7qeMIxADZnu/RnsfRnVTrPCDA8/j+NrWO8GRAMJDqyQb/zsqJ9mr1LE/G6RiRC/Rxw3Ht0Ofx4oGZArzR5vlPgJ90xYc6To2vAucCO4AEz7IEYIfn8ZPA19psf2S7TqgtCXgHOAdY5vmhKWnzS3Dk+Hp+0KZ4Hgd4tjOdeBwjPeFnPrfcjce1dWRtjOdYLQPOc9OxBVI+F34ndRyBrwFPtll+1Ha+rPVz674CLPE8Pur3v/W4dmZOtFcrsBQYC+TwWVB3+XE90Zc3mz7aG2qe6MX3PyWeP2HHAxuAvtba/QCe7/GezbryM/wR+B+gxfM8Fii31ja1U8uROj3rKzzbd5Y0oBh41tNU87QxJgwXHldrbQHwO2AfsB/nWGXi3mMLJ38c3fK7dz3OmSm4sFZjzMVAgbV2y+dWua7Wz/NmUHdoqHlXMMaEA/8C7rDWVh5v03aW+fwzGGMWAAettZkdrKWrj3UAzp+Vj1trxwPVOH+iH0uX1etp370E58/v/kAYzkyQx6qnq4/t8Ryrti6v2RhzN9AELGld1M5mXVarMSYUuBv43/ZWt7PMFce1lTeD2pVDzY0xgTghvcRa+2/P4gPGmATP+gTgoGd5V32GacDFxpgcnNkJz8E5w44yxrQOSmpby5E6Pet7A6WdUGerfCDfWrvB83wpTnC77bgCzAX2WmuLrbWNwL+Bqbj32MLJH8cu/d3zXGRbAFxtPW0Ex6mpq2odhPOf9RbP71kSsMkY08+FtX6BN4PadUPNjTEGeAbIstb+vs2q/wCtV3Cvw2m7bl1+recq8GSgovVPUF+y1v7EWptkrU3BOW7vWmuvBlYCC49RZ2v9Cz3bd9r/9NbaIiDPGDPMs2gO8CkuO64e+4DJxphQz89Da62uPLbt1NCR4/gGMM8YE+35C2KeZ5nPGWPmAz8GLrbW1nzuMyzy9KJJBYYAH9JFOWGt/dhaG2+tTfH8nuXjdDQowoXHtb0P4M3G+wtwelbsAe7uikb3z9UzHedPla3AR56vC3DaHN8Bdnm+x3i2Nzg3SdgDfAykd0HNs/is10cazg/3buAlINizPMTzfLdnfVoX1DkOyPAc21dwroq78rgC9wHbgW3A8zg9EVxxbIG/47SdN+KExw1f5jjitA/v9nx9sxNr3Y3Tjtv6+/VEm+3v9tS6Azi/zXKf50R7tX5ufQ6fXUzs0uPakS8NIRcRcTmNTBQRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5f4/WqUlO9emMYwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(8, max_lr=1e-2, callbacks=getWeights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = getWeights.get_weights()\n",
    "w1 = weights[5]\n",
    "w2 = weights[7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2.3460364, tensor(0.3747), tensor(0.7049)]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Just to verify that it worked. The loss value is same as above so it worked\n",
    "learn.model.load_state_dict(w1)\n",
    "learn.validate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do interpolation\n",
    "def interpolate(alpha):\n",
    "    w_new = {}\n",
    "    keys = list(w1.keys())\n",
    "    for key in keys:\n",
    "        w_new[key] = alpha*w1[key] + (1 - alpha)*w2[key]\n",
    "    return w_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha_range = np.linspace(start=-0.5, stop=1.5, num=100)\n",
    "train_loss = []\n",
    "val_loss = []\n",
    "for i, alpha in enumerate(alpha_range):\n",
    "    print(f'{i}/{len(alpha_range)} started')\n",
    "    w_new = interpolate(alpha)\n",
    "    learn.model.load_state_dict(w_new)\n",
    "    loss1, _, _ = learn.validate()\n",
    "    loss2, _, _ = learn.validate(data.train_dl)\n",
    "\n",
    "    val_loss.append(loss1)\n",
    "    train_loss.append(loss2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAFlCAYAAADPim3FAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZzNZf/H8ddla7GUopS9UmlX6k6WUtlLC0pKFGlV2kN1l1KpblIU0mIpVChCyFaILLclJBIqhLLLMjPX74/PmZ+5NZgx58z3nO95Px+P85iZc86Mz3eOMW/X8rmc9x4REREROTx5gi5AREREJJEpTImIiIjkgMKUiIiISA4oTImIiIjkgMKUiIiISA4oTImIiIjkQL6g/uBixYr5cuXKBfXHi4iIiGTZnDlzNnrvi2f2WGBhqly5csyePTuoP15EREQky5xzqw70mKb5RERERHJAYUpEREQkBxSmRERERHJAYUpEREQkBxSmRERERHJAYUpEREQkBxSmRERERHJAYUpEREQkBxSmRERERHJAYUpEREQkBxSmRERERHJAYUpEREQS14QJsG5doCUoTImIiEhiGjUK6teHRx4JtAyFKREREUk8X34JN94I554LPXsGWorClIiIiCSW9CB13nkwfjwULRpoOQpTIiIikjhGjrQgdf75cRGkQGFKREREEsWIEdCoEVxwgQWpY48NuiJAYUpEREQSweefQ+PGUKkSjBsXN0EKFKZEREQk3g0dCk2awIUXwtixcRWkQGFKRERE4tmnn8LNN8PFF8fdiFQ6hSkRERGJT0OGwC23QJUqNiJVpEjQFWVKYUpERETiz8cfQ7NmULUqjBkDhQsHXdEBKUyJiIhIfBkwAJo3hxo1YPRoKFQo6IoOSmFKRERE4sf770OLFlCzpjXnLFgw6IoOSWFKRERE4kPv3tCqFdSubc05EyBIgcKUiIiIxIMePeCee6BBA+spddRRQVeUZQpTIiIiEqxu3aBtW7juOhg2DI48MuiKskVhSkRERILz6qvwyCN2TMynn0KBAkFXlG0KUyIiIpL7vIdOneDJJ6FpUxg0CPLnD7qqw6IwJSIiIrnLe3j6afj3v23n3sCBCRukAPIFXYCIiIgkEe/hscega1e46y7o1QvyJPbYTmJXLyIiIokjLc0Wmnftam979074IAUKUyIiIpIbUlPh7ruhZ08bmereHZwLuqqo0DSfiIiIxFZKCrRsCR99BB07wgsvhCZIgcKUiIiIxNKePXZg8dCh0LkzdOgQdEVRpzAlIiIisbFrl/WPGj3aGnO2axd0RTGhMCUiIiLRt2OHdTSfONEWmrdpE3RFMaMwJSIiItG1ZYudsffdd9CvHzRvHnRFMaUwJSIiItGzcSPUrQvz58PgwdCkSdAVxZzClIiIiETH2rVQqxYsXw6ff26jU0ngkH2mnHOlnXOTnHNLnHOLnHMPZfKcK5xzW5xz8yK3Z2NTroiIiMSlVaugRg1YuRLGjEmaIAVZG5lKAR713s91zhUG5jjnxnvvF+/3vG+999dEv0QRERGJa8uWwVVXwdatMH48VKkSdEW56pAjU977td77uZH3twFLgJKxLkxEREQSwMKFUL06/P03TJqUdEEKsnmcjHOuHFAJmJnJw1Wcc/Odc2Occ2cf4PPbOOdmO+dmb9iwIdvFioiISByZORMuvxzy5YNvvoFKlYKuKBBZDlPOuULAUKCd937rfg/PBcp6788H3gI+z+xreO/7eO8re+8rFy9e/HBrFhERkaBNngxXXw1Fi8K330LFikFXFJgshSnnXH4sSH3kvR+2/+Pe+63e++2R90cD+Z1zxaJaqYiIiMSHL7+09gdlyliQKl8+6IoClZXdfA54D1jive96gOeUiDwP59wlka/7ZzQLFRERkTgwZAjccAOcey5MmQInnxx0RYHLym6+qkBzYKFzbl7kvg5AGQDvfS+gMXCvcy4F+Bto6r33MahXREREgtKnD9xzjy04HzkSihQJuqK4cMgw5b2fCrhDPKcH0CNaRYmIiEic6dIFnnrK+kd9+ikcdVTQFcWNbO3mExERkSTjvYWop56CW26B4cMVpPaj42REREQkc6mpcP/90Lu3Te/16AF58wZdVdzRyJSIiIj80549cOutFqTat4e331aQOgCNTImIiMj/2rEDGjeGr76CV1+Fxx8PuqK4pjAlIiIi+2zaBNdcAzNmwLvvQuvWQVcU9xSmRERExKxdC3XqwNKl8Mkn0KhR0BUlBIUpERERgRUroFYt+OMPGDXKjoqRLFGYEhERSXYLFtiI1J49MHEiXHJJ0BUlFO3mExERSWZTp0KNGrZT79tvFaQOg8KUiIhIsvryS5vaO/FEmDYNzjor6IoSksKUiIhIMurfH66/Hs45x0anypYNuqKEpTAlIiKSbLp1gxYt4IorbI1U8eJBV5TQFKZERESSRfo5e488Yk05R42CwoWDrirhaTefiIhIMkhJgTZt4IMPdM5elGlkSkREJOx27oQbbrAg9dxzOmcvyjQyJSIiEmabNsG118L06Rai7r036IpCR2FKREQkrH7/3ZpxLltmx8M0bhx0RVGXlgbO2S0omuYTEREJoyVLoEoVWL0axowJZZDavBkaNLABtyApTImIiITN9OlQtaodDzNlClx5ZdAVRd2PP8K//gVffw0FCgRbi8KUiIhImIwYAVddBcWKWaiqVCnoiqJu1CgLUps2WZusu+4Kth6FKRERkbDo29d27Z17rh0Pc8opQVcUVd7DK6/YevpTT4XZs6F69aCrUpgSERFJfN7DCy/YEE3t2qHsap6aCrfdBu3bw0032Qk4ZcoEXZVRmBIREUlkKSnWhPPZZ+H2222ar1ChoKuKus6d4eOPoVMnGDQIjj466Ir2UZgSERFJVDt3wo03Qp8+0KEDfPgh5M8fdFVRN2kSPP+8jUw9/XSwbRAyoz5TIiIiiWjjRls8NHMm9OwJ990XdEUx8ccf0KwZVKgA77wTf0EKFKZEREQSzy+/QN261kNq6FBbdB5C6eukNm+GcePid/ZSYUpERCSRzJljnSr37LEmS1WrBl1RzLz8sl3iu+/aBsV4pTVTIiIiiWL0aLj8cjjySGt9EOIgNWUK/PvfNsXXqlXQ1RycwpSIiEgi6NsXGjaE00+H776DihWDrigmduyA99+Hpk3htNOgV6/4XCeVkcKUiIhIPPPehmjuuguuvtqGbE46Keiqom7OHOvwcNJJNhJVtCh89hkULhx0ZYemNVMiIiLxas8euPtua3lwxx3Qu3foWh8sWgQtWliYOvJIa8h51102gxnvI1LpFKZERETi0ZYt0LixrcB+9ll47rnESRdZtHChncGcLx/06AG33grHHht0VdmnMCUiIhJvfvsN6teHJUtsAdEddwRdUdTNn2/nMR9xhDXlPP30oCs6fApTIiIi8WT+fGt9sHWr7d6rVSvoiqJu3jwLUkcfbUHqtNOCrihntABdREQkXowbB9Wr2/tTp4YySM2da1N7BQvC5MmJH6RAYUpERCQ+9O1rU3vlysGMGXDeeUFXFHUzZ9qIVJEitinx1FODrig6FKZERESClJZmhxSntz6YOhVKlQq6qqj78kuoWROOO85GpMqXD7qi6FGYEhERCcquXdbi++WXLUyNHGnDNiHTpw9cdx2cfTZMn26Db2GiMCUiIhKEjRttJGrIEHjllVD2kPIennnGWmXVrWuLzU88Meiqok+7+URERHLbsmW2Y2/1ahg8GG6+OeiKom7vXhts69fPOpr36mX9pMIopJclIiISp775Bm64AfLkgQkTQnlYcUqK5cPhw63X6LPPhq7f6P/QNJ+IiEhuGTjQpvaKF7cdeyEMUmlpcOedFqTeeMOOFQxzkAKFKRERkdhLP6y4eXOoVg2++y48fQEy8B7atoUBA+DFF+Ghh4KuKHdomk9ERCSWdu2yRUMff2zHwvTqBQUKBF1VTHToAG+/DU88Ye8nC4UpERGRWNmwAa6/3voBdO4M7duHds7r5ZdtU+I999jbkF5mpg45zeecK+2cm+ScW+KcW+ScO+CgnXPuYudcqnOucXTLFBERSTCLF8O//mXnp3zyiQ3VhDBheA+vvmqXd9tt0LNnKC/zoLIyMpUCPOq9n+ucKwzMcc6N994vzvgk51xeoAswNgZ1ioiIJI7x46FJEzjySDs35ZJLgq4oJnbutPYHH39sl/vBB7ZJMdkc8pK992u993Mj728DlgAlM3lqW2AosD6qFYqIiCSS3r2hXj0oU8YOowtpkPrlF9uMOGiQLTYfPDi8faQOJVuX7ZwrB1QCZu53f0ngBuBK4OKDfH4boA1AmTJlslepiIhIPEtNhUcfhe7dLUwNHhzKo2HABt6aNrVL/vJLO585mWV5MM45VwgbeWrnvd+638NvAE9671MP9jW8932895W995WLFy+e/WpFRETi0dat0LChBamHHoIRI0IZpNLXR9WtCyedBLNnK0hBFkemnHP5sSD1kfd+WCZPqQwMdrbirBhQ3zmX4r3/PGqVioiIxKOVK+Haa2HJEnjnHdvOFkJbtlhnh+HDoXFjWx9VqFDQVcWHQ4YpZwnpPWCJ975rZs/x3pfP8PwPgS8VpEREJPSmT7fWB3v2wFdfWXfzEFq4EBo1ghUroGtXaNcu+XbsHUxWRqaqAs2Bhc65eZH7OgBlALz3vWJUm4iISPwaOBBat4ZSpWzh0JlnBl1RTAwcCG3awDHHwKRJUL160BXFn0OGKe/9VCDL+dN73zInBYmIiMS1tDR4+mnrUnn55TB0KBx/fNBVRd3GjdY76t13oUYNGDIESpQIuqr4lITdIERERA7T9u023/Xyy9Zgady40AWpXbvgtdfgtNPgvffg8cdhwgQFqYNJ0o4QIiIi2bR6te3YW7jQdu21bRuqhUNpadbNoUMHWLUKGjSALl3g7LODriz+KUyJiIgcyowZttD8779h1CjrDRAiK1ZAs2bWY/SCC2xE6qqrgq4qcWiaT0RE5GD697e1UYUKWagKWZD68ku46CJYuhT69YM5cxSkskthSkREJDOpqfDEE9CiBVSrZsM2FSsGXVXUpKbCM89Yi6zy5e085ttvT86z9XJK03wiIiL727rV5r1GjYL77oM33oD8+YOuKmo2brTLGz8e7rwTevSAo44KuqrEpTAlIiKS0c8/20LzpUvh7bfh3nuDriiqfv4ZataE9euhb19o1SroihKfwpSIiEi6CROgSRN7f9w4uPLKYOuJsrVroVYt2LkTpk2ztVKSc5oZFRER8R7eegvq1IGTT4ZZs0IXpDZvtrXz69fD6NEKUtGkMCUiIsltzx47L+XBB6250nffwamnBl1VVP39t81cLlliBxVfcknQFYWLwpSIiCSv9eutD0DfvnZEzPDhULhw0FVFVUoK3HwzTJ1q5+zVqhV0ReGjNVMiIpKc5s61RpwbN9rBczfdFHRFUee9ncU8cqStpQ/hJcYFjUyJiEjyGTQIqla142CmTQttymjf3hpxPv986DYlxhWFKRERSR7pjTibNbOFQ7NmQaVKQVcVE92729l6995rzTkldjTNJyIiyWHTJgtRX30VykacGX3yCTz8MNx4o21SDNF5zHFJYUpERMJv0SJbH7VqFfTpA3fdFXRFMTNpEjRvbifgfPQR5M0bdEXhpzAlIiLhNmyYna9XsKAljapVg64oZubPt8xYoQJ88QUceWTQFSUHrZkSEZFwSk21dgeNGsHZZ8OcOaEOUnPnQr16UKSIzWQWLRp0RclDYUpERMJn82brUtm5sx0+N2UKlCwZdFUxsXw53HKLdTRPSbEgVapU0FUlF4UpEREJl0WLbKfeuHHWXOndd+GII4KuKurWrbN19BUrwogR0LEjLFtmg3CSu7RmSkREwuOzz6BlS+tiPmmSrcIOkR07YPx4C09DhthJOHfdZa0PTjop6OqSl8KUiIgkvtRUG5rp0gWqVLFQdfLJQVcVFZs3W6uDESPg669h92445hho0sSWhJ12WtAVisKUiIgktj//tP5R48bB3Xdbt8qQTOtt2AA1asCPP0L58nDPPbYUrHr10LbISkgKUyIikrj++1/rTLlmjR1W3KpV0BVFzZYtULcurFwJY8faAcVqvhmftABdREQSU79+cNlltoXt229DFaR27oRrr4UFC6xNVu3aClLxTGFKREQSy549to2tZUtbHzVnju3eC4k9e6BxY5g6FQYOtN5REt8UpkREJHH8/jtcfjm8844dWDxuHJxwQtBVRU1qKtx+O4wZA717w803B12RZIXWTImISGKYPNnSxc6dtluvUaOgK4oq7+Hee63lwWuvhfr4wNDRyJSIiMQ37+HVV+Gqq+C44+D770MZpJ54wvqLdugAjz0WdEWSHRqZEhGR+LVlC9xxBwwfbo2V3nvPGnKGzEsvweuvw/33w4svBl2NZJfClIiIxKcffrC2BytWQNeu0K5dKLe09ehhzTebN4c33wzlJYaewpSIiMSfgQOtAWeRInYsTPXqQVcUE/37Q9u2cP318P77kEeLbxKSXjYREYkfu3ZZm+/mzeGii2Du3NAGqWHD4M474eqrYfBgyKfhjYSlMCUiIvFh5Uo7mLh3b3j8cZg4MZSn927caP1FGzWCiy+25WAhOf0maSlMiYhI8EaNggsvhOXLLV28+mrohmrS0mwq78wzbXrvySft4OJChYKuTHJKYUpERIKTkmK9AK65BsqWtW7m118fdFVR98MP1mu0VSuoWNGOFHzlFShYMOjKJBrCFftFRCRxrF0Lt9wCU6ZA69a2le2oo4KuKuomTLCsWLCgjUy1aKGF5mGjMCUiIrlv4kQLUtu324HFt98edEUxMW0aNGwIp51moSpEJ99IBsrGIiKSe9LS4IUXoFatfd3MQxqkZs2yQ4pLlbK1UQpS4aWRKRERyR3r11vLg3HjoFkz27UX0tXXCxZAnTpQrJiNSJ14YtAVSSwpTImISOx9841N6/35J/TqBW3ahLbV948/Wu+oggUtSJUqFXRFEmua5hMRkdhJS7OD52rWtHQxc6Z1Ng9pkBozBq680haYT5gA5csHXZHkBoUpERGJjQ0boH596NgRbrrJ2h6cf37QVcXEypVwww12uUWK2Bqp008PuirJLQpTIiISfZMnwwUX2NveveHjj6Fw4aCrirrdu6FzZzjrLFsK9sortl7qnHOCrkxy0yHDlHOutHNuknNuiXNukXPuoUyec51zboFzbp5zbrZzrlpsyhURkbiWmgrPPw9XXWWLy2fMCO36qIUL4dxz4emnoUEDWyv15JNQoEDQlUluy8oC9BTgUe/9XOdcYWCOc268935xhudMAEZ4771z7jzgE+DMGNQrIiLxas0auO02mDTJ3r7zTmh36337LVx7rS0D++or27knyeuQYcp7vxZYG3l/m3NuCVASWJzhOdszfEpBwEe5ThERiWdjx1rbgx074IMPrM13CEejAEaMgJtvttNvxo2DMmWCrkiClq01U865ckAlYGYmj93gnPsRGAXceYDPbxOZBpy9YcOG7FcrIiLxZc8eeOIJqFvXminNmgUtW4Y2SL3/vi00P+88mDpVQUpMlsOUc64QMBRo573fuv/j3vvh3vszgeuBFzL7Gt77Pt77yt77ysWLFz/cmkVEJB6sWAHVq8Nrr8E991g387POCrqqmPDeFpe3amXN2ydMsIacIpDFMOWcy48FqY+898MO9lzv/TfAqc45/TUTEQmrIUOgUiVYuhQ+/dTWR4XwkGKwEahataB9e+s7OmJEaJeCyWHKym4+B7wHLPHedz3Ac06LPA/n3IVAAeDPaBYqIiJxYMcOuOsuaNoUzj4b5s2Dxo2Driom0kNU9eq2c69bNxg4ULv15J+yspuvKtAcWOicmxe5rwNQBsB73wtoBNzunNsL/A3c7L3XInQRkTCZP99C1NKl8NRT0KkT5M8fdFVRt2ABPProvsOJ//Mfm8U8+uigK5N4lZXdfFOBg64k9N53AbpEqygREYkj3kOPHvDYY3D88TB+vPWRChnvrb9ou3bWxVwhSrJKBx2LiMiBbdwId94JI0daZ8oPPoAQbiDautVmLz/5xHpGDRgQysuUGNFxMiIikrkJE+wsvbFjoXt3C1QhTBhz58KFF8LQofDyyzB6dCgvU2JIYUpERP7Xnj22JqpWLZvvmjEDHnwwlL2jBgyAKlXsjL3Jk+2y8+g3o2STpvlERGSfZcugWTOYPdvO1Ova1c5MCaHRo+GOO+Dyy2167/jjg65IEpXClIiI2Orrfv3ggQfgiCNg2DBr9R1Sc+bATTfZLOYXX6hvlOSMBjNFRJLdpk122Nwdd8DFF1sLhBAHqZUrbS19sWIwapSClOScwpSISDKbNMkOmhs+3M5L+fprKFUq6Kpi5q+/oF49WxY2ZgyUKBF0RRIGClMiIskofZH5VVdZI6UZM+DJJyFv3qAri5ldu+D66+1Iwc8/h4oVg65IwkJrpkREks2PP8Ktt1pPgLvvtu6UIV1kni41FW6/Hb79FgYNgho1gq5IwkQjUyIiycJ76NnTmiqtWmXDM716hT5IeQ/33WfnMb/+up2IIxJNGpkSEUkG69ZZJ/MxY2zR0PvvJ82CoY4doU8faN/eztwTiTaNTImIhN3w4XDOObbYvGdP28KWJEHqtdesq/ndd0PnzkFXI2GlMCUiElbbtkGrVnDjjVC2rK2Ruu++UHYyz0zfvvDEE9b1oWfPpLlsCYCm+UREwmjqVFtxvWqVzW899xwUKBB0VTHxzTfw228WltJvv/5qmxPr1oX+/UO9SVHigMKUiEiY7NkD//43dOkC5ctb0qhaNeiqYmLPHnj4YXj77cwfr1oVPvsstBlS4ojClIhIWPzwAzRvDvPmQevWdq5e4cJBVxUTa9ZAkyYwfTo89phdrvf7bgCnnw759FtOcoH+momIJLrUVAtOTz8Nxxxjh801bBh0VTEzbRo0bgxbt8KQIXbGnkiQtABdRCSRrVgBV1xhK60bNLDRqZAGKe9tSu+KK6w11syZClISHxSmREQSkffWPOm882DBAujXD4YOhRNOCLqymNi5E1q2hPvvh9q1YfZs6/YgEg8UpkREEs3vv9so1N13w6WXwsKFtnMvpHv/f/4ZqlSBAQNsU+LIkXDssUFXJbKP1kyJiCQK7+Gjj6BtW9i9G95804Zq8oT3/8UjR9qa+jx5rNdovXpBVyTyT+H9CRQRCZP166FRI0sWFSvC/PkWqkIapFJT4ZlnbPnXqafCnDkKUhK/wvlTKCISJkOHwtln29DMq6/Ct99ChQpBVxUzu3fDLbfAiy9aA/dp06xllki8UpgSEYlXGzdC06bWByD9OJjHHw91O+8tW2wE6tNP4fXX7UiYI48MuiqRg9OaKRGReDRsGNx7L2zaBC+8YGej5M8fdFUxtW6dBakffrDF5rfdFnRFIlmjMCUiEk82brS1UIMHQ6VKMH68tT8IuWXLoE4dWxo2cqSdqSeSKDTNJyISL9LXRg0dCp06WVfKJAhS06fbOXrbtsHEiQpSkngUpkREgrZ+vbXybtwYSpaEWbNsK1vIp/W8t+4Ol19uRwhOmwaXXBJ0VSLZpzAlIhIU72HQIDjrLDtPr3NnG406//ygK4u57dttx95DD0H9+tb64PTTg65K5PAoTImIBGHtWrjxRmjWzBopzZ0LHTqEfjQKYMkSG4H69FN45RUYPlwdzSWxaQG6iEhu8h4++AAeecQaKr32Gjz8cKjbHWQ0bJidfFOwIHz9NdSsGXRFIjmnkSkRkdzyyy92Sm+rVjaVt2ABPPZYUgQp763faKNGcO65NhCnICVhoTAlIhJrqam20vqcc2DGDHj7bZg0KdRdzDPauxfatLFWWTffbJdesmTQVYlEj6b5RERiadEiaN3aQlTdutC7N5QpE3RVuWbzZtukOGECPP00PP98aI8TlCSmMCUiEgu7d8NLL8HLL0ORIjBwoC02dy7oynLNihVwzTWwfDl8+CG0aBF0RSKxoTAlIhJt06fbaNSSJXDrrdCtGxQvHnRVuWrYMFsa5hyMGwdXXBF0RSKxo8FWEZFo2bIF7rsPqlWDHTtg9GgbkUqiILV7t52G06gRnHYazJ6tICXhpzAlIpJT3tsRMBUrQq9eliZ++MFO7U0iy5fDZZdBjx7W7WHaNDjllKCrEok9TfOJiOTEr7/CAw/AiBHW7uCLL+Dii4OuKuZSUmDNGli92m7LlsF//gP58tm3oGHDoCsUyT0KUyIihyMlBXr2tC1qqanWRKldu9B3MN+0yXbnTZlil51RjRowYEBSbVYUARSmRESyb/ZsuPtu6zxZpw688w6ULx90VTG3ebP1HF2wAB591NZElSljt9KloVChoCsUCYbClIhIVm3daiNRPXvCCSfAkCHQpElStDvYvBlq1bIgNWwYNGgQdEUi8UNhSkTkULyHzz6zaby1a23HXufOcMwxQVeWKzIGqaFDFaRE9qfdfCIiB7N8ue3Ku+kmG42aMcO2qyVRkKpdG+bPtyB1zTVBVyQSfw4ZppxzpZ1zk5xzS5xzi5xzD2XynFudcwsit+nOufNjU66ISC7ZtQuee87O05s+Hbp3h1mz4JJLgq4sV+zcaS2yatSAefNsak9BSiRzWZnmSwEe9d7Pdc4VBuY458Z77xdneM4vwOXe+03OuXpAH+BfMahXRCT2xo61dgfLl0PTptC1K5x0UtBVxZz3trb+vfdg0CBbInbKKfD551C/ftDVicSvQ4Yp7/1aYG3k/W3OuSVASWBxhudMz/ApM4BSUa5TRCT2Vq+2bpPDhsHpp8P48XD11UFXlSt++cW6lv/3v3DUUdb+4M47bWRKBxOLHFy2FqA758oBlYCZB3laK2DM4ZckIpLLdu+20acXXrCPX3oJHnkEjjgi2LpyycqVULOmjUT17g0335w0S8JEoiLLYco5VwgYCrTz3m89wHNqYmGq2gEebwO0ASijrm4iEg/GjbPjX376CW680Q4lTqJ/n1at2hekvv4aLrww6IpEEk+WBm+dc/mxIPWR937YAZ5zHtAXuM57/2dmz/He9/HeV/beVy6eRCLL3MkAACAASURBVAd/ikgcWrnSwlOdOpCWBmPG2Ha1JApSq1dbkNq82WY0FaREDk9WdvM54D1gife+6wGeUwYYBjT33v8U3RJFRKLo77+hUyc7lHjsWJvS++EHqFs36Mpy1a+/WpD66y8LUhddFHRFIokrK9N8VYHmwELn3LzIfR2AMgDe+17As8DxwNuWvUjx3leOfrkiIofJezuM+OGHbbX1TTfB66/bOShJZsECG5TbuNGCVGX9ay2SI1nZzTcVOOhZCd771kDraBUlIhJVixdb9/Lx4+Hss2HiRBuWSTI7dtig3H/+A8cdZwNzSdI2SySmtOFVRMJr82YLUeedZw03u3e3vf8hDVKpqfDjjxaa9jdmjPUfffVVaNkSliyBSy/N9RJFQkln84lI+KSmWufJjh3hzz+hTRtrexDijS9//w3XXgsTJtjHpUrBGWfYbd06a5115pkwZYr1jhKR6FGYEpFwmTzZRqPmz4dq1eDNN6FSpaCriqndu20N1MSJ8Pzz1mRz6VK7DRxojz//PDz5ZNK0zhLJVQpTIhIOP/8Mjz8Ow4dD2bIwZAg0aQLuoEs+E96ePXaZX30F774Lrfdbveo9pKRA/vzB1CeSDLRmSkQS25YtNuRy1lnWgPPFF21B0E03hT5IpaRAs2YwciT07PnPIAX2LVCQEoktjUyJSGJKSYG+feHZZ2HDBmjRwnpGnXxy0JXlitRUaN7c+ox26wb33Rd0RSLJSyNTIpJ4vvoKzj8f7r3XRqRmz4YPP0yaILVmDVx3HQweDF262BIxEQmOwpSIJI6FC61Teb16tlho+HCYNClp2nd7b5nx7LNt195bb8ETTwRdlYgoTIlI/FuzBlq1ggsugJkzbV5r0SK4/vrQr4tKt3o11K8Pd9wB555rXcwfeCDoqkQEtGZKROLZtm3w2mvWsjslxeazOna09t1JpF8/aNvWzmN+6y1bH5VH/xUWiRsKUyISf/butaabzz0Hf/wBTZva4vLy5YOuLFelpUH79ta1/PLL4YMPku5bIJIQFKZEJH54b9vTOnSAZcugenU7nDgJD5DbuRNuv92+HffcYyNS+fQvtkhc0kCxiMSHKVPssLgmTaBAAWueNGVKUgapdevs+MBhw2yG8+23FaRE4pl+PEUkWPPn20jU6NFQsqRN77VoAXnzBl1ZIBYtggYNrHXWsGG2xl5E4ptGpkQkGCtWwK232rl506fDyy/b1N6ddyZtkBo/Hi67zM7S++YbBSmRRKEwJSK5a90629N/xhnWJ+rJJy1YPfUUHHVU0NUF5r33rPVB2bLW/SFJWmeJhIKm+UQkd/z1l21Le+stG3q56y545pmk6Vp+IGlp8PTTNjBXuzZ8+ikUKRJ0VSKSHQpTIhJb27bBG2/A66/b+7fcYi0PKlQIurLA7doFLVvCkCHQpg306KFDiUUSkcKUiMTGzp3wzjvwyiuwcaMtAOrUydp3J7kVK2yz4ocfwrx5NmD32GNJ08xdJHQUpkQkunbtgj59bN5q3TqoVQtefDEpWxxk9P33tjtv5EhYvNjuq1jR+kjdeGOwtYlIzihMiUh07N4N778PnTvD77/DFVfAJ59Y480ktnmznYLTr5/1iqpRw5aLXXstnHpq0NWJSDQoTIlIzuzZY+ecvPSSncZbtSoMGGBdJ5PcV19B69Y2QNexo03lHXts0FWJSLSpNYKIHJ7du6FXLzjtNDvv5OSTLT18+23SB6lt22xBeb16cMwx8N13NtOpICUSTgpTIpI9u3bZwvIKFeDee61r+dix1nizTp2kXkW9caNtXDznHOsb9eSTMGcOXHxx0JWJSCxpmk9EsmbnTltY/tprsGYNVKkCffvaAvMkDlBpaTBxon0rhg+3Wc9//QsGD7ZvkYiEn8KUiBzctm02EvWf/8D69XD55dC/P1x5ZVKHqJ07LUC98Qb88gscd5wN1LVqpe4PIslGYUpEMvfXX9at/M037f3ata1Vd5LvztuyBXr2hG7dbFqvenVbe3/99XDkkUFXJyJBUJgSkf+1Zg107WqLy3fsgIYNoUMHm7tKYlu22Axnjx72fr169m2pVi3oykQkaApTImJ+/tnSwgcfQEqKHfvy1FO2mjrJ/fYb1K1rzTYbNbIQValS0FWJSLxQmBJJdnPmQJcu1oo7Xz648054/HE45ZSgK4sLS5bYJsXNm+Hrr22pmIhIRgpTIsnIe5gwwULU119DkSLwxBPw0ENQokTQ1cWNGTOgQQM7fHjKFI1GiUjmFKZEksnevXbEy+uv2wm7J51kgeruu627pPy/0aOhcWPrRTpunAbqROTA1LRTJBls3WqtDU45BW67zbqXv/uu7el/4gkFqQy2b4dXXrF19xUrwrRpClIicnAamRIJs5Urrb1B374WqK64wnbp1asHefR/qYw2bbJvVffu1gmiYUMYOBAKFw66MhGJdwpTImH03XfW3mDYMGus2aQJPPooVK4cdGVxZ80aC1E9e1p/0muvtUOJk7wThIhkg8KUSFjs3Ws78t54A2bOtFN1H3sMHngASpcOurq4kZoKs2bBqFG2LmruXMubN91kLQ/OOy/oCkUk0ShMiSS6DRugd2878mXNGjjtNOss2aIFFCoUdHVxIy0N2reH99+3zuV58sBll8HLL1vvqAoVgq5QRBJVeMNUaqr9cmndWmc8SDjNnWvzU4MG2YLyOnVsUXnduloPtR/v4b77LHM2bmzhqXZtO09PRCSnwhumpk2Dtm3h+++hX7+kPpBVQmT3bvj0U1vgM2MGFCxo/2F44AE488ygq4tL3ttsZ+/eNjL10ktBVyQiYRPe/77WqAHPPw8DBsCrrwZdjUjOrFplC3pKlYLmzW272Rtv2DknPXooSB3Ec8/ZWvwHH4TOnYOuRkTCKLwjUwDPPGNnQbRvb79srrsu6IpEsi4lBcaMsVYGY8bY6GrDhnD//XamiabyDunVV6FTJ2jVCrp10wC1iMRGuMOUc7ba9Oef4dZbberv/PODrkrk4H77Dd57z3pD/fabdSnv2BHuugvKlAm6uri0bRusX29vt2+3tzNn2uB006Y2xafsKSKxEu4wBXDUUfDFF3DxxdZAZtYsOPHEoKsS+V9798KXX1qIGjPGtp7Vrg1vvgnXXGOHw0mmvvjCAtOuXf987LrroH9/yJs39+sSkeQR/jAF9j/7ESOgWjW44QaYOFE7/CQ+LF1qAapfPxtaOflkeOopm5fSGSaHNGSInY5z4YU2+1m4sN0KFbITcs48U1N7IhJ7yRGmwP61HTDA9kXXrWvNDY8/PuiqJBlt3myHDX/4oXUqz5vXRp9at7a/m/mS58cyJ/r1gzvvhKpVrQGnjn0RkaAcchWBc660c26Sc26Jc26Rc+6hTJ5zpnPuO+fcbufcY7EpNQoaNbLDtmbMgEsugcWLg65IkkVqKowdC82a2Ujp3XfDli3QpYuti/r8cwtUClJZ0qsXtGxp6/DHjFGQEpFgZeVf7hTgUe/9XOdcYWCOc2689z5jEvkLeBC4PhZFRtWtt8Kpp8L118Oll8LgwVC/ftBVSRh5D/Pn24jooEGwdi0ULWpTeC1bwkUXaQ7qMLzxBjz8MDRoAJ99phl7EQneIUemvPdrvfdzI+9vA5YAJfd7znrv/Sxgb0yqjLZLL7WF6KeeaovSu3a1X3wi0bBqlY04nXsuVKpkXcovucSaba5da32hKldWkMqm7dsthz78sA0yDxumICUi8SFbcwrOuXJAJWDm4fxhzrk2QBuAMkFv8S5dGqZOtQaIjz5qU39vvaWdfnJ4/vjD1kENHgzTp9t9l11mRxo1aaL1eTn0/fc2qPzzz9Y2rlMnzYiKSPzIcucV51whYCjQznu/9XD+MO99H+99Ze995eLFix/Ol4iuggVtnqBzZ9tfXbGirWrVKJVkxYYNdhbe1VfbLrwHH7QGRy+9ZL/1p02De+5RkMqB1FR48UXLpXv2wOTJ9u1VkBKReJKlf5Kcc/mxIPWR935YbEvKZXny2DEdN9xgTRFbtoSPP7YVruXLB12dxJs//oDhwy2ET55sv+1PPdX+DjVtCmefHXSFCSclxZaVjR5t7bSOPHLfbcYM2/DYrJkdR3jssUFXKyLyT4cMU845B7wHLPHed419SQGpWBG++cZC1JNPwjnn2HE0Dz4IRx8ddHUSpBUrbOTy889tajgtDU4/3fpBNW5sXfW1/inbvLdvaceOdupT2bJQoIA130y/HX20bcC99dagqxUROTDnDzGl5ZyrBnwLLATSInd3AMoAeO97OedKALOBIpHnbAfOOth0YOXKlf3s2bNzfAEx8euv0Lat/QItUQKeftpGrQoUCLoyyQ1pafDf/1qj188/hwUL7P7zzrNdoI0bW9hWgDpskydbFp050xprdu5sg8P6lopIvHLOzfHeV870sUOFqViJ6zCVbto0m7755hv7b/Nzz1m7ZS3YCJ9t22D8eOv+OHo0rFtnU8DVqlmAuu46dSSPgr17oU0b61daqpSdnXf77fqREpH4pzCVE97bL9mOHWH2bFsf07Yt3HEHFCkSdHVyuLy3EaexY+327bf2m/6YY6BOHWtiVK8exMNGiZDYscM2No4ZYz9OHTva0ZkiIolAYSoavLdpv9dftxGrQoUsUD3wgK2fkfi3Zo2dyzhunN3++MPuP/dcO8alQQPbNqZDhaPur7+swfvMmbYs8a67gq5IRCR7FKaibc4c6N7degrt3WsjGS1a2FSQFqvHjw0bbHHOpEkWopYutfuLFYNataB2bbudfHKgZYbd77/bj8iyZdYI/sYbg65IRCT7FKZiZd066N0b3nvPFq0XLmytmW+7Da64wg6wldyzerVN133zjb1dssTuL1QIatSwg9xq1oQLLrD1UBJzixbZaU2bNtnAbs2aQVckInJ4FKZiLS3NfoEPGGD9h7ZutdGOhg3tVrOmzr2Itj17YN48a0KUflu92h4rUgSqVoXq1e17f9FFmrqLoj/+gHbtbL1To0bWs/SII/Y9npoKX30Fb79t66OKF7ePK1UKrmYRkZxSmMpNf/8NI0fCkCG2sHnHDuu0XqeOnQN49dW2jUmyLjXVpujmzLFNALNn2/u7d9vjpUvbeYvVq9vt3HM1KhgjS5fauvx166xTyJYtll2vucZaG/z8s62JWrnSuoq0aWNN4E86KejKRURyRmEqKLt22ZqdL76wnkVr1tj9p522b8qpZk2dB5jR9u02N7Rggd3mzbOeTzt22ONHHw0XXmjhKf1WsuTBv6ZExdSptiwwb1748kubLZ0wAYYOtXZcf/5pz6tZE+67z56rAUERCQuFqXjgPcyfb4uhJ02CKVNsOhCgXDm4+OJ9t4susvVXYbZpkw1zLFkCP/5ot0WLrNt4+t/JQoWsUeZFF0HlynY74wyNOgXgk0+sH1TZsjZ1t3/LrZQUm2ktVswOExARCRuFqXiUkmIjLlOmwPffw6xZNjcC1ga6fHk46yw76+2ss+x22mmJczjZ7t22jeu33+CXX2z+J+Nt48Z9z82f39pLVKxo4Sn9VrasFooHzHv4z3/g8cdtGdoXX+jcZhFJTgcLU+o7HJR8+faNRKXbsMFC1ezZNkqzeLGtu9q7d99zjjnGRrLKlbOwUbKkTROecMK+23HH2XRYNM/m8N66hG/ebKNKmzfD+vW2Gjn97R9/WID69dd9PZzS5ckDZcpY09MbboAKFSw8nXmmXYtaYMedvXutjVqfPtZss39/7aMQEcmMRqbi3d69NvW1eLG9Xbnyf2/bt2f+eXny2FRh+q1gQRsBynjLl892Iqam7nubmmqL6NNvO3fa2y1b7DkH+rOKFbMgV7KkLQhPv5UqtS/46WzDhLFpkx1BOHEitG8PL76oQUIRSW4amUpk+fPbOqEzzvjnY97bwuz00aH0EaK//rJRpIy3nTstmO3da+Fo61YLTnny2C1v3n1vixa11g5HHbXvduyxdv+xx+67pY+EFSumdUwhsmyZ7c775Rc7Q69Fi6ArEhGJbwpTicw5W6RdqJBNn4nk0JQp1qHcOdupV7160BWJiMQ/hSmRJLd2rS0sHz7cAlSFCtb6QPlcRCRrFKZEkszOnda+a+pU6w81Y4bNGFeoAI89Bk89lTibRkVE4oHClEjIbdxoBwynN45fsmTfXoILL4ROnWyD5VlnRXcDqIhIslCYEgmxCRPs3O116+x4l4susvP00vugnnxy0BWKiCQ+hSmREEpJgX//G15+2Vp5jR6tg4ZFRGJFYUokZFatgmbNYPp0aNUKune3NmMiIhIbClMiIZGSYl3KH33UWogNGgRNmwZdlYhI+IW2p/Gff9pJLb16HbhJuEgi2LvXjnTp2dNGnfaXkgIDBtjpPK1a2bTef/+rICUikltCG6bWrLH/nd97r51o8vDDsHx50FWJZM+UKXDBBXD33XZOXrlydgb000/DzJnw8cd2Fvbtt1vv1s8/t+k99YgSEck9oQ1T555r28CnTYP69aFHD+ujU7++9dURiWfr1tkuvCuusL5QI0bA0qXw+ut2jvUrr8Cll8Ktt9rhw8OG2d/3665TewMRkdyWNAcdr11rUyXvvGPH1zVrZr+QSpfOtRJEDsh7+O036wH1/ffw2muwaxc88YQdNHz00f/7/L/+gq++stGoa67RIcQiIrF2sIOOkyZMpdu+Hbp0sf/hO2e/rJ544p+/rERibetW6NDBput+/PF/1/bVrg1vvQWnnx5cfSIiso/CVCZWrYInn4QhQ2xN1dtvw7XXBlaOJJn1623Kef58m8qrWPF/byeeGHSFIiKS0cHCVNK2RihbFgYPhrZtbWFvw4Y2ndKpE+RL2u+K5IZVq2zk6ddf7YDh+vWDrkhERHIi6VdaVK0K330HbdpYt+g6dWzUQCQWFi2yv3Pr18P48QpSIiJhkPRhCmw3VO/e8MEHtq38oou0408OX1oaLFtmI1B//gm7d9sC8xkzoHp1e/ybbyxUiYhI4lOYyqBlSxulyp8fatSA998PuiJJJNu3W2PNs86yhePlykGxYhbWCxSw8HTccTB1qrXuEBGRcNDqoP1ccIH167nlFusmnZYGrVsHXZXEsxUrrI/Ze+/ZDr2LL7YNDUccYQEr/Qbw4INQokSw9YqISHQpTGWiaFFbGHzDDbaWKl8+G7USSbdxozXK/OQTmDgR8uaFxo0tLF16qRpniogkE4WpAzjiCPtl2bAh3HmnBarbbvvn89I7S+iXZ7ilptoaqMmTrZ3GhAl2X4UK8MwzFrpLlgy6ShERCYLC1EEceaSNUF1zDbRoYaMPt9xiv0SnTYPhw+0stO3brfniffdZCJPE5T2sXm2LxRcssGaaS5fauY67d9tzTjkFHn8cbr4Zzj9fQVpEJNklbdPO7Nixw7awT5tmU3+TJ9s0zxFHQK1aduzH11/bguMXX7TApeM9EsOuXXZ8y3ffWYCaMcPOxQMbjTzlFDjjDDjzTHtbqZLdFKBERJKLOqBHwfbt0KCBdaxu0MBCVZ06ULiwPT5+vHVU/+9/7ZftK69Y0NIv3fiydauF4m+/tfYEs2bBnj32WIUKtt4p/XbOObYLT0RERGEqStLSbBoob94DPz5oEDz9NKxcCZddBs8+a92uFaqCs3UrjBhha53GjoW9e23U6aKLrO9TjRpQpYq1MRAREcmMwlQu273belS9/LIdGXLxxRaqGjRQqMot69fbIvFPP4XRo+01KVUKbrrJpmwvvRQKFgy6ShERSRQKUwHZswf694eXXoJffrFppEKFbA3Wjh02dbhnD5QubU0e029nnGGjWsk0xZSaaqN+hzoXMS3NFoY7ZxsE0m+pqbbuadIkW9O2aJE9v0QJaNLEFotXqaK1bCIicngUpgK2dy98/LFNAebPbyMi6bf8+W3L/U8/2REku3bZ55QoAfffD/fcE87pp7Q0WLjQws+kSbZ+yXv497/tujMLkt9/b4dSz5p14K9bsCBUqwZXXAE1a0LlygeelhUREckqhakEkZYGv/1mi9h79YKvvrJRl9tug3bt4Oyzg64w5+bPh+7dreXEX3/ZfaeeasHn119tTVOFCvD663DttTYCtWEDtG9vHcZLlLDAdcIJNnW3a5fdUlPhwgttSjV//mCvUUREwkdhKkEtXmzBo39/Cwwnn2xh4qST9t2qVYufXYN79liQ2b+WtDQYNQq6dbNRqKOPtm7hV11lIap06X3PHTMGHn0Uliyxx6++Grp0sSnRdu2sQWaRIrl7XSIiIgpTCW7jRujXz8LV2rV2W7fOFlmnpUHFihY0mjeHo446+NfascMaUP70E2zbBldeaf2xsmP6dHj+eRtF27Zt39lze/da762Mge/EE20h+LJltgC8bVs76/C44w789ffuhd69bQTqr78sUL35pl2niIhIEBSmQmr3btut1q0bzJ0Lxx9va6yuugr++MMCV/pt9WoLUL///s+vU7Gi7XCrX99Gug608H3NGuulNXCgjZJVqWIL6gsXtrcFC1q4WrPGAl/62zPPhIcegkaNsjcFt2mTLdxXk0wREQlajsKUc6400B8oAaQBfbz33fd7jgO6A/WBnUBL7/3cg31dhano8d6aUHbrZmuRMr6kBQrYSFHJkrZTsEKFfW8LFIBx46x1wJQpNk1XsKC1Daha1YLVpZfaaNMbb8ALL9hzHnvMjs9RawEREUkWOQ1TJwEnee/nOucKA3OA6733izM8pz7QFgtT/wK6e+//dbCvqzAVG7/8YrcSJexWtGjWRnW2b4eJE20B+LRp1n7Ae2slULQo/PmnLQjv1s0WjIuIiCSTg4WpQx507L1fC6yNvL/NObcEKAkszvC064D+3pLZDOfcsc65kyKfK7mofHm7ZVehQtCwod3AuobPmGHB6scfoWVLqFcvqqWKiIiEwiHDVEbOuXJAJWDmfg+VBH7N8PFvkfsUphJUkSJ2DE7t2kFXIiIiEt+y3A/aOVcIGAq0895v3f/hTD7lH/OHzrk2zrnZzrnZGzZsyF6lIiIiInEoS2HKOZcfC1Ifee+HZfKU34AM3YIoBazZ/0ne+z7e+8re+8rFixc/nHpFRERE4sohw1Rkp957wBLvfdcDPG0EcLszlwJbtF5KREREkkFW1kxVBZoDC51z8yL3dQDKAHjvewGjsZ18y7HWCHdEv1QRERGR+JOV3XxTyXxNVMbneOD+aBUlIiIikiiyvABdRERERP5JYUpEREQkBxSmRERERHJAYUpEREQkBxSmRERERHJAYUpEREQkBxSmRERERHJAYUpEREQkB5z12wzgD3ZuA7AqF/6oYsDGXPhz4pGuPXkl8/Un87VDcl+/rj155cb1l/XeZ3qwcGBhKrc452Z77ysHXUcQdO3Jee2Q3NefzNcOyX39uvbkvHYI/vo1zSciIiKSAwpTIiIiIjmQDGGqT9AFBEjXnryS+fqT+dohua9f1568Ar3+0K+ZEhEREYmlZBiZEhEREYmZhA9TzrnjnHPjnXPLIm+LHuB5qc65eZHbiAz3l3fOzYx8/hDnXIHcqz7nsnL9zrkLnHPfOecWOecWOOduzvDYh865XzJ8by7I3SvIPudcXefcUufccufcU5k8fkTktVweeW3LZXisfeT+pc65OrlZdzRk4dofcc4tjrzOE5xzZTM8lunPQCLJwvW3dM5tyHCdrTM81iLyc7LMOdcidyvPuSxce7cM1/2Tc25zhscS+rV3zr3vnFvvnPvhAI8759ybke/NAufchRkeS/TX/VDXfmvkmhc456Y7587P8NhK59zCyOs+O/eqjp4sXP8VzrktGf5+P5vhsYP+zESV9z6hb8CrwFOR958CuhzgedsPcP8nQNPI+72Ae4O+pmhfP3A6UCHy/snAWuDYyMcfAo2Dvo5sXG9e4GfgFKAAMB84a7/n3Af0irzfFBgSef+syPOPAMpHvk7eoK8pytdeEzg68v696dce+TjTn4FEuWXx+lsCPTL53OOAFZG3RSPvFw36mqJ57fs9vy3wfohe+xrAhcAPB3i8PjAGcMClwMwwvO5ZvPbL0q8JqJd+7ZGPVwLFgr6GGF//FcCXmdyfrZ+ZnN4SfmQKuA7oF3m/H3B9Vj/ROeeAK4HPDufz48Qhr997/5P3flnk/TXAeiDTxmMJ4BJgufd+hfd+DzAY+x5klPF78hlwVeS1vg4Y7L3f7b3/BVge+XqJ4pDX7r2f5L3fGflwBlAql2uMpay89gdSBxjvvf/Le78JGA/UjVGdsZDda78FGJQrleUC7/03wF8Hecp1QH9vZgDHOudOIvFf90Neu/d+euTaIHw/81l57Q8kJ/9eZFsYwtSJ3vu1AJG3JxzgeUc652Y752Y459IDx/HAZu99SuTj34CSsS036rJ6/QA45y7BUvrPGe7uHBki7uacOyJ2pUZFSeDXDB9n9pr9/3Mir+0W7LXOyufGs+zW3wr733q6zH4GEklWr79R5O/zZ8650tn83HiV5fojU7vlgYkZ7k701/5QDvT9SfTXPbv2/5n3wDjn3BznXJuAasoNVZxz851zY5xzZ0fuy9XXPl+svnA0Oee+Bkpk8lDHbHyZMt77Nc65U4CJzrmFwNZMnhd32xujdP1E/qc2AGjhvU+L3N0eWIcFrD7Ak0Cnw6825lwm9+3/mh3oOVn53HiW5fqdc7cBlYHLM9z9j58B7/3PmX1+nMrK9Y8EBnnvdzvn7sFGKK/M4ufGs+zU3xT4zHufmuG+RH/tDyWsP/NZ5pyriYWpahnurhp53U8AxjvnfoyM9ITJXOyYl+3OufrA50AFcvm1T4iRKe/91d77czK5fQH8EQkJ6WFh/QG+xprI2xXAZKASdo7Psc659FBZClgT48vJtmhcv3OuCDAKeDoyDJ7+tddGhsZ3Ax8Q/9NevwGlM3yc2Wv2/8+JvLbHYMPEWfnceJal+p1zV2NBu2HkdQUOUmiW/wAAAmhJREFU+DOQSA55/d77PzNc87vARVn93DiXnfqbst8UXwhe+0M50Pcn0V/3LHHOnQf0Ba7z3v+Zfn+G1309MJz4//c927z3W7332yPvjwbyO+eKkcuvfUKEqUMYAaTv0GgBfLH/E5xzRdOnryLf5KrAYm+r1CYBjQ/2+XEuK9dfAPtB6u+9/3S/x9KDmMPWW2W6YyKOzAIqONuFWQD7xbH/7qSM35PGwMTIaz0CaOpst1957H8v3+dS3dFwyGt3zlUCemNBan2G+zP9Gci1yqMjK9d/UoYPGwJLIu+PBWpHvg9FgdqR+xJFVv7e45w7A1to/V2G+8Lw2h/KCOD2yK6+S4EtkWUPif66H5JzrgwwDGjuvf8pw/0FnXOF09/Hrj3e/33PNudcicjvr/RlLHmAP8niz0zUxGple27dsLUwE4BlkbfHRe6vDPSNvH8ZsBBbzb8QaJXh80/BfqEuBz4Fjgj6mmJw/bcBe4F5GW4XRB6bGPme/AAMBAoFfU1ZuOb6wE/Yuq+Okfs6YQEC4MjIa7k88tqekuFzO0Y+bylQL+hricG1fw38keF1HhG5/4A/A4l0y8L1vwwsilznJODMDJ97Z+TvxHLgjqCvJdrXHvn4OeCV/T4v4V97bKRtbeTfsd+w6ax7gHsijzugZ+R7sxCoHKLX/VDX3hfYlOFnfnbk/lMir/n8yM9Ex6CvJUbX/0CGn/kZwGUZPvcfPzOxuqkDuoiIiEgOhGGaT0RERCQwClMiIiIiOaAwJSIiIpIDClMiIiIiOaAwJSIiIpIDClMiIiIiOaAwJSIiIpIDClMiIiIiOfB/Csj6dNjxyLoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.plot(alpha_range, train_loss, 'b')\n",
    "plt.plot(alpha_range, val_loss, 'r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha_range = np.linspace(start=-10, stop=4, num=50)\n",
    "train_loss2 = []\n",
    "val_loss2 = []\n",
    "for i, alpha in enumerate(alpha_range):\n",
    "    print(f'{i}/{len(alpha_range)} started')\n",
    "    w_new = interpolate(alpha)\n",
    "    learn.model.load_state_dict(w_new)\n",
    "    loss1, _, _ = learn.validate()\n",
    "    loss2, _, _ = learn.validate(data.train_dl)\n",
    "\n",
    "    val_loss2.append(loss1)\n",
    "    train_loss2.append(loss2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAFlCAYAAADPim3FAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZzW4/7H8dfVvhIV0iIRISklZU06RRulCKkoZN/XQ5Tl2FscpG2kPRWtTloUSWmlFZElhWlfpmWW6/fHZ/pNTaWpme/9vee+38/HYx7NzH0734/vGebtuj7fz+W894iIiIjI0ckTdgEiIiIiuZnClIiIiEg2KEyJiIiIZIPClIiIiEg2KEyJiIiIZIPClIiIiEg25AvrwqVKlfIVK1YM6/IiIiIiWbZgwYL13vvSB3sttDBVsWJF5s+fH9blRURERLLMOffroV7TNp+IiIhINihMiYiIiGSDwpSIiIhINihMiYiIiGSDwpSIiIhINihMiYiIiGSDwpSIiIhINihMiYiIiGSDwpSIiIhINihMiYiIiGSDwpSIiIhINihMiYiISO41eTIkJoZagsKUiIiI5E4rVkCLFvDgg6GWoTAlIiIiuc+uXdCmDRQtCm+8EWop+UK9uoiIiMjReOIJ+O47mDgRypQJtRStTImIiEjuMn489Opl23uNG4ddjcKUiIiI5CJr18Ktt0L16vDKK2FXAyhMiYiISG6Rmgpt28LOnTB8OBQsGHZFgHqmREREJLd47TX4/HMYMADOPDPsav6fVqZEREQk+s2ZA88+a0/wdegQdjX7UZgSERGR6LZlC9x4I5QvD717g3NhV7QfbfOJiIhI9PIeOneG33+HL7+EY48Nu6IDKEyJiIhI9Bo40JrNX3oJ6tYNu5qD0jafiIiIRKfvv4d774UrrrAhnVFKYUpERESiz+7d1idVqBAMGgR584Zd0SFpm09ERESiz1NPwaJFMG4clC0bdjX/SCtTIiIiEl0mTYLu3eG++6BZs7CrOSyFKREREYke69bZHKlq1WxIZy6gMCUiIiLRIS0N2rWD7dth2DDrl8oF1DMlIiIi0eHNN2HqVOjTB84+O+xqskwrUyIiIhK+efPg6aehVSvo1Cnsao5IlsOUcy6vc26Rc27CQV4r6Jwb4Zxb5Zyb65yrmJNFioiISAzbutXGIJx8sq1KRdlxMYdzJCtTDwArDvFaR2CT9/50oDvwanYLExERkThxzz2wejUMHQrHHRd2NUcsS2HKOVcOaAL0O8RbrgEGpn8+CrjSuVwWK0VERCTyBg2CwYPh+efh4ovDruaoZHVlqgfwOJB2iNfLAr8DeO9TgC1Aycxvcs7d4Zyb75ybn5iYeBTlioiISMz48Ue4+2647DLrl8qlDhumnHNNgb+99wv+6W0H+Z4/4Bve9/He1/Le1ypduvQRlCkiIiIxZc8e65PKn99WpqL4uJjDycpohIuB5s65xkAh4Bjn3GDvfdt93rMGKA+scc7lA44FNuZ4tSIiIhIb/v1vWLAAxoyB8uXDriZbDrsy5b1/yntfzntfEWgDTM8UpADGAe3TP2+V/p4DVqZERERE+OwzeOMNuOsuaNEi7Gqy7aiHdjrnugHzvffjgP7AIOfcKmxFqk0O1SciIiKx5K+/bMr5OefYkM4YcERhyns/A5iR/nmXfb6/C2idk4WJiIhIjElLs3P3tmyxSeeFC4ddUY7QcTIiIiISGT16wP/+B++9B1Wrhl1NjtFxMiIiIhK8BQvgySetR+rOO8OuJkcpTImIiEiwtm2DNm3gxBOhX79cd1zM4WibT0RERIJ1333w888wfTocf3zY1eQ4rUyJiIhIcIYOhYED4Zln4PLLw64mEApTIiIiEoyff4bOne3MvWefDbuawChMiYiISM5LTrbjYvLmhSFDIF/sdhbF7t+ZiIiIhKdLF/jmG/joIzjllLCrCZRWpkRERCRnTZ0Kr74Kd9wBrVqFXU3gFKZEREQk5yQmwi23QJUq0L172NVEhLb5REREJGd4b8fFbNpkk86LFAm7oohQmBIREZGc0asXTJoEb78N550XdjURo20+ERERyb7Fi+Hxx6FZM7jnnrCriSiFKREREcmeHTvsuJhSpWDAgJg7LuZwtM0nIiIi2fPAA/DDDzBtmgWqOKOVKRERETl6I0ZA//7w9NNwxRVhVxMKhSkRERE5OqtX2yypOnXguedCKWHECPjll1Au/f8UpkREROTIJSfDTTfZ50OHQv78ES9h+nS4+ebQctz/U8+UiIiIHLmuXWHOHBg+HE49NeKXX7XKhqtXqWKTGMKklSkRERE5Mp9/Di+/DLfdBjfcEPHLb9kCzZtDnjwwbhwcc0zES9iPVqZEREQk69avh7Zt4YwzbEhnhKWm2u7ijz/ClClQqVLESziAwpSIiIhkjffQsaMFqgkToGjRiJfw5JM2ZL13b6hXL+KXPyiFKREREcmad9+1fbUePaBGjYhffuBAeOMNG7B+550Rv/whqWdKREREDu+77+CRR6BJE7j//ohffvZsm8Jw5ZXQvXvEL/+PFKZERETknyUl2XExxx0HCQkRPy7mt9+gRQuoUAFGjgxlCsM/0jafiIiIHJr3dlzMypXw2WdQunREL79jB1xzDezaBTNmwPHHR/TyWaIwJSIiIof2/PPQrx889RQ0aBDRS6elQfv2tsM4YQKcdVZEL59lClMiIiJycK++Ct262TypF1+M+OW7dYPRo+HNN+HqqyN++SxTz5SIiIgc6O23bQ7BjTdCnz42ITOCPvrIhqx36AAPPRTRSx8xhSkRERHZX79+9sReixY2jyBv3ohefuFC29676CKbJxXhfvcjpjAlIiIiGYYMsRkEV18Nw4ZF/NG5P/+0hvNSpWDMGChYMKKXPyrqmRIREREzZowtCdWrZ81KEU4yu3bZYtjGjfDVV3DiiRG9/FFTmBIRERE7o6VNG6hd26acFy4c0ct7bwtic+ZYjqtePaKXzxZt84mIiMS7adOgZUuoVg0+/RSKFYt4CW+8AYMG2RN8LVtG/PLZojAlIiISz776Cpo3h8qVYfJkOPbYiJcwYQI88QRcfz0880zEL59tClMiIiLxav58aNwYypWDKVOgZMmIl7BsGdx0k52bHMJJNTlCYUpERCQeffcdNGxoAWraNDjppIiXsGGDLYoVLQpjx0KRIhEvIUeoAV1ERCTerFwJ//qXpZdp02xlKsKSk6FVK/jjD5g5M5QScozClIiISDz5+eeMM/amTYNTTw2ljPvvt4OLBw2CCy8MpYQcozAlIiISL37/Ha68EnbutCRz5pmhlPHuuzbZ/IknoG3bUErIUQpTIiIi8eDPPy1IbdwI06fDueeGUsa0abYq1bQpvPRSKCXkOIUpERGRWLd+vW3trV0Ln30GNWuGUsaqVdC6NVSpYqfWRPjIv8AoTImIiMSyzZuhUSP46SeYONFODw7Bli3QrBnkyWMD1o85JpQyAqEwJSIiEqu2bbMDi5cssdkD9euHUkZqKtx4o61MTZkClSqFUkZgFKZERERiUVKSDXGaNw8++shCVUieeMJOqend285QjjUKUyIiIrFm92474G7mTBg8GFq0CK2UDz6AN9+Ee++FO+8MrYxAKUyJiIjEkuRkaNPGztnr39/OagnJ7NkWoK68Erp3D62MwOk4GRERkViRmgrt2sEnn8Dbb8Ntt4VWym+/2YJYhQowciTki+Hlmxj+WxMREYkjaWlw++0wfDi8+qrtq4Vkxw5r19q1y3Yajz8+tFIiQmFKREQkt/PeJmEmJMBzz8Hjj4dWSloatG9vDxBOmGAzpWKdwpSIiEhu5r2Fp3fegcceszAVoq5dYfRoazoP8QHCiFLPlIiISG7WtSu88Qbcc49t7zkXWikjR0K3bnDrrfDQQ6GVEXEKUyIiIrnVa69ZmLr1VujVK9QgtWABdOhgA9bfey/UUiJOYUpERCQ3+u9/bRpmmzbQt6+d0xKSdevgmmugdGkYMwYKFgytlFCoZ0pERCS36d8f7rvPEsyHH4Z6YvCuXTYCYdMm+OorOPHE0EoJjcKUiIhIbjJ0qI1AaNQIRoyA/PlDK8V7K2XuXGs6r149tFJCpW0+ERGR3OLjj20o5+WXR8V+2uuv22k13brZ6TXxSmFKREQkN/j0U7jhBrjgAhg3DooUCbWc8ePhySetpGeeCbWU0ClMiYiIRLvp023pp2pVC1XFi4daztKlduTf+efDgAHx9eTewShMiYiIRLPZs+1sltNOg88+gxIlQi1n/Xorp1gxOwIw5AWyqHDYMOWcK+Sc+8Y5961zbplzrutB3tPBOZfonFuc/tEpmHJFRETiyIIFNkb85JNh6lQoVSrUcvbsgVatYO1aC1LlyoVaTtTIytN8u4H63vvtzrn8wCzn3Kfe+zmZ3jfCex/eqYoiIiKxZMkSaNgQjjsOpk2Dk04KtZy9x//NnAmDBsGFF4ZaTlQ5bJjy3ntge/qX+dM/fJBFiYiIxLXvv4cGDaBQIeuXKl8+7Ip49114/32bE9q2bdjVRJcs9Uw55/I65xYDfwNTvPdzD/K265xz3znnRjnnDvr/unPuDufcfOfc/MTExGyULSIiEqNWr4Yrr7TPp02DSpXCrSe9jAcegGbN4KWXwq4m+mQpTHnvU7331YFyQG3nXNVMbxkPVPTeVwOmAgMP8b/Tx3tfy3tfq3Tp0tmpW0REJPasWQP160NSEkyZAlWqhF0R06bZg4RVqsCQIaEOW49aR/Q0n/d+MzADuCrT9zd473enf9kXqJkj1YmIiMSLP/+0FamNG+2pvWrVwq6IwYOt/718+aiYyBC1svI0X2nnXIn0zwsDDYCVmd5TZp8vmwMrcrJIERGRmLZmjQWpNWtg0iSoVSvUcryH//wHbrkFLr4YZs2KiratqJWVp/nKAAOdc3mx8DXSez/BOdcNmO+9Hwfc75xrDqQAG4EOQRUsIiISU5YsseWfrVthwgRLLyFKSbEzlHv3tsGcAwaEfmpN1HP2sF7k1apVy8+fPz+Ua4uIiESF6dOhRQubgDlpEpx3Xqjl7NgBN96YcVTMSy9BHo33BsA5t8B7f9AlQ90iERGRMAwdClddZZMvv/469CD1999wxRUwcSK8845t8ylIZY1uk4iISCR5D6++CjffDBddZA1JFSqEWtKPP0Ldunbm3pgxcPfdoZaT62SlZ0pERERyQmqqjRF/911o0wY++CD0hqQ5c6BpUzusePp0qFMn1HJyJa1MiYiIREJSElx3nQWpxx6zoU0hB6lPPrGtvRIlbKdRQeroKEyJiIgELTHRhnGOGwe9esFrr4XekPTOOzaMs1o1C1Knnx5qObmatvlERESC9NNP1mi+Zg2MGmUJJkRpafDUU5bnmjWD4cOhSJFQS8r1FKZERESCMm8eNGlivVJTp4Y+Q2r3bujQwQLUXXfB22/reJicoG0+ERGRIEyYAPXqQdGiMHt26EFq82Zo1MiC1Cuv2DafglTOUJgSERHJae+/D9dcA2edZQ1JZ54Zajm//WZZbvZsO2/viSfs6T3JGQpTIiIiOcV7eOYZ6NzZloFmzICTTgq1pMWL7Sm9NWvgf/+z8VaSsxSmREREcsKePdaQ9NJL0KmTPblXrFioJU2ZApddZtt5s2bZA4WS8xSmREREsmvrVms0//BD6NYN+vSBfOE+4zVwIDRuDBUr2k7jueeGWk5MU5gSERHJjrVr4dJL4fPPYcAAePbZUBuSvIcXX7RFsssvhy+/tOP/JDgajSAiInK0li2Dq6+GTZvshOBGjUItJyXFztXr2xfatoX+/aFAgVBLigtamRIRETkaM2fCJZdAcrJ9HnKQ2r7dHiDs2xeeftp2HBWkIkMrUyIiIkdqxAho1w4qVYJPP7XGpBD99Ze1bC1aBL17w513hlpO3NHKlIiISFZ5D2++CW3aQO3a8NVXoQep77+HunVhxQoYO1ZBKgxamRIREcmK1FR4+GE7qLhVKxg0CAoVCrWkr76C5s1t9MGMGXDBBaGWE7e0MiUiInI4O3fC9ddbkHroIdvmCzlIjR4NV14JJUva6AMFqfAoTImIiPyTDRugQQP4+GN46y37yBPur8+ePaF1a6hRw46IOe20UMuJe9rmExEROZTVq+Gqq+DXX201qnXrUMtJS4PHHrM8d+21MGQIFCkSakmCwpSIiMjBLVhgj8jt2WPnslx6aajl7NoF7dvDyJFw773Qo4f1Skn4tM0nIiKS2aef2vjwQoWsyzvkILVxIzRsaEHq9detdUtBKnooTImIiOyrf39o1gzOOMM6u886K9RyfvkFLr4Y5s6FYcPg0UdDPa1GDkJhSkREBGyG1HPPQadO1nA+cyaUKRNqSYsW2QypP/+Ezz6z8VYSfRSmREREkpOhY0fo1g1uvRXGj4fixUMtafJkuOwyyJ8fZs2yXUeJTgpTIiIS37Zts229hATo0sW2+fLnD7WkAQOs9/2002DOHDjnnFDLkcNQmBIRkfi1bp0t+UydaicEd+0aakOS91ZCx45Qvz588QWcfHJo5UgWaTSCiIjEpxUr4OqrITERxo2Dxo1DLSc5GTp3tlWp9u0t24W8QCZZpJUpERGJP7Nm2SNyO3dao3nIQWrDBmja1IJUly6246gglXsoTImISHwZNcqe1itd2kYf1KoVajlz5tixMDNmRMVOoxwFhSkREYkP3kP37nZgcc2adqhdpUqhltOjh80DzZvXZoN26hRaOZIN6pkSEZHYt20b3HEHDB8OLVrYoXaFC4dWzubNcNttdnbyNdfYtt5xx4VWjmSTVqZERCS2LVliW3kjR8LLL9s2X4hBauFCWxgbPx7efNMClYJU7qYwJSIisSshAWrXhq1bYdo0eOopyBPOrz7voXdvm2i+Z4/1vT/8sPqjYoHClIiIxJ6kJJtkftttcNFFsHgx1KsXWjnbtsHNN8Ndd9n8qEWLrCyJDQpTIiISW1auhAsvhIED4dln7VC7E08MrZy9u4wjRtgu48SJUKpUaOVIANSALiIisWPoUGs0L1wYPv0UGjUKtZyEBLjnHjj2WNtlDHFxTAKklSkREcn9du2yPbSbb4bq1W0fLcQgte8uY926oe8ySsAUpkREJHf76SdrQOrdGx5/HD7/HMqVC62cfXcZu3QJfZdRIkDbfCIiknuNGWNLQHnz2vl6zZqFWs6+u4z/+x80bBhqORIhWpkSEZHcZ88eeOghuO46OPNMG94UYpDad5exRg3b1lOQih8KUyIikrv89htcdpmdxXLffXZoccWKoZWTeZdx+nQoWza0ciQE2uYTEZHcY+JEaNcOkpNtonnr1qGWs+8u4/jx0LRpqOVISLQyJSIi0S8lxaaXN20K5cvDggWhBqk9e+DBB22XsUoVe3hQQSp+aWVKRESi29q10KYNfPmldXf36BHq2Xq//grXXw/ffAMPPACvvQYFCoRWjkQBhSkREYleU6fCTTfBjh0waBC0bRtqORMm2C5jaqqdl3zddaGWI1FC23wiIhJ9UlOha1d7JK50aZg3L9QglZICTz5pDwyecortMipIyV5amRIRkejy9982Y2DqVLjlFnjvPShaNLRy/vgDbrzRdhnvvNN2GQsVCq0ciUIKUyIiEj2+/BJuuAE2bYJ+/ew8FudCK2fKFMt1SUkweLB9LpKZtvlERCR8aWnw6qtwxRVQrBjMmQMdO4YWpFJT4bnn7Hi/E06A+fMVpOTQtDIlIiLh2rAB2re3GVKtW9uK1DHHhFbOX39ZcJo2zcp6551QdxklF1CYEhGR8Myda3MG1q2D//4X7r471G29mTNtCsPmzTBggA3kFDkcbfOJiEjkeQ89e8Kll0KePPDVV3DPPaEFqbQ0+M9/oH59WxSbO1dBSrJOK1MiIhJZW7ZYY/mYMdC8OXzwARx3XGjlbNhgs6MmTbJVqT59oHjx0MqRXEhhSkREImfRIuuL+uUXeOMNePjhULf15syxXca//oJ334XOnUMtR3IpbfOJiEjwvIf334e6dWHXLmtOeuSR0JKL99C9u+0y5ssHs2fDXXcpSMnRUZgSEZFgbd9u08s7d4Z69Wx16uKLQytn82abXv7ww3Y48cKFULNmaOVIDFCYEhGR4CxdChdcAMOHw4svWmNS6dKhlbNgAZx/PowfD2+9ZW1bJUqEVo7ECIUpEREJxsCBULu2TTOfOhX+/W97ci8Eu3bBM89AnTqQnAxffAEPPaRtPckZh/2pds4Vcs5945z71jm3zDnX9SDvKeicG+GcW+Wcm+ucqxhEsSIikgskJdn08g4d4MILYfFim2wekhkzoFo1eOkluOkmK6du3dDKkRiUlf9E2A3U996fB1QHrnLO1cn0no7AJu/96UB34NWcLVNERHKFBQssQCUk2FLQ1Klw0kmhlLJxI3TqZDkuNdXO2Rs4EEqWDKUciWGHDVPebE//Mn/6h8/0tmuAgemfjwKudE6LpyIicWPXLnj6aQtSGzbAp5/CCy9A3rwRL8V7GDECzjrLRlg98QQsWQINGkS8FIkTWdq8ds7ldc4tBv4Gpnjv52Z6S1ngdwDvfQqwBTgg+zvn7nDOzXfOzU9MTMxe5SIiEh2+/hpq1LAR4u3bw/LldkJwCH77DZo1s+GbFSrYAcWvvAJFioRSjsSJLIUp732q9746UA6o7ZyrmuktB1uFyrx6hfe+j/e+lve+VukQn+YQEZEckJRk8wUuvtg+nzwZ+vcP5fG41FQ7nebss61Hqnt3G8hZvXrES5E4dESPVXjvNwMzgKsyvbQGKA/gnMsHHAtszIH6REQkGu3t6u7e3aZdLl0KDRuGUsq331pD+YMPwmWXwbJl9nkIO4wSp7LyNF9p51yJ9M8LAw2AlZneNg5on/55K2C69/6AlSkREcnltm2Du+/OeDpvxgx4551QDrPbuROefNIGbv76KwwbBhMnwimnRLwUiXNZOZuvDDDQOZcXC18jvfcTnHPdgPne+3FAf2CQc24VtiLVJrCKRUQkHJMnwx13wO+/2/beCy+E1ow0daoNVP/pJzsz+fXX4fjjQylF5PBhynv/HVDjIN/vss/nu4DWOVuaiIhEhU2b7By9hAR7RG72bJt+GYING6yUgQOhcmWYPj3UEVYigCagi4jIPxk3Ds45Bz780EYfLFwYSpDyHoYMgSpV7M+nn7ZeKQUpiQZZ2eYTEZF4k5gIDzxgjUjVqsGECXaoXQhWr7Ye98mTbYxV375w7rmhlCJyUFqZEhGRDHsnXp59NowaBd26wbx5oQSplBR4802oWhW++greftv+VJCSaKOVKRERMevWwT33wMcfwwUXwIABlmRCsHChHQWzaJEN4XznHShfPpRSRA5LK1MiIvHOe+uJOuccOwbm9detyTyEILVjBzz6qGW5devgo49g7FgFKYluWpkSEYlnv/8Od95pIeqSS2yC+RlnhFLK5Mk27uCXX2wCw6uvhjJMXeSIaWVKRCQeeQ99+thq1BdfWEPSzJmhBKnERGjbFq66CgoWtDLef19BSnIPhSkRkXjz88/QoIGtSNWuDUuWwL33Qp7I/krw3uZFVakCI0dCly427uCyyyJahki2KUyJiMSLvacBn3suzJ9vMwamTIFTT414KatWwb/+BR06WJhavBi6drWVKZHcRmFKRCQerFxpSz4PPgj16tlpwJ06gXMRLSM5GV55xfLcvHnw3nvw5Zc2iUEkt1KYEhGJZSkp1sldvTqsWAGDBtkAznLlIl7KvHn2lN5TT0HjxrB8uTWcR3h3USTH6UdYRCRWLVliR788+SQ0aWLppW3biK9Gbd9uC2J16liz+ccfw+jRULZsRMsQCYzClIhIrNmzxxqQata00QcffWTp5aSTIl7KpEn2wGCvXrYKtXw5XHttxMsQCZTmTImIxJIFC+C22+C77+Dmm6FHDyhVKuJl/PmnrUbtPZlm1iy46KKIlyESEVqZEhGJBbt2wdNP20nA69fDuHEweHDEg1RqKvTrB2edZdt53brZkTAKUhLLtDIlIpLbzZ4NHTvaE3sdO8Ibb0R84qX38Nln8Pjjtih26aU2E7RKlYiWIRIKrUyJiORWO3bAQw/ZMTA7d1qa6dcv4kFq0SJo2NAmmG/fblt7M2cqSEn8UJgSEcmNPv8cqlWznqi777Yn9/71r4iW8Ouv0K6d9bkvXGilLF8O118f8QcGRUKlbT4Rkdxk9Wp44gl7Qu/0020JKMLnr2zeDC+/bE/ogW3tPfmkztKT+KUwJSKSG2zdCv/5D3TvDnnz2uiDRx+FIkUiVsLu3Tax/IUXYNMmW5Xq1g0qVIhYCSJRSWFKRCSapaZCQgI88wz89ZclmJdfjujEy7Q0O4j46adtYaxhw4yh6iKiMCUiEr0+/9wazL/9Fi6+GMaPt/NYImjmTHjsMTsKplo1mDzZwpSIZFADuohItFm1Clq0gPr1rUFpxAg7DTiCQWr5cmjWzM5EXrcOPvjAmswVpEQOpDAlIhItNm+2Pqizz4apU61HauXKiD4et24d3HEHnHsufPEFvPIK/PADtG9vrVoiciBt84mIhC0lBfr2hS5dYMMGOw7mxRcjepbetm026/ONNyA5Ge67z9q0QjiJRiTXUZgSEQnTZ5/Bww/DsmW2p9a9e0Q7u5OTbc7n88/D33/bItjLL8Npp0WsBJFcT9t8IiJhWLkSmjSBRo3sXL2PP4bp0yMWpLyHTz6x7by774Yzz4Q5c6w9S0FK5MgoTImIRNKGDXD//VC1KsyaZftqy5bBtddGrC9qzhyb89mihV1y7Fh7au/CCyNyeZGYozAlIhIJyck2MrxyZXjnHevyXrUKHnkEChaMSAmrVkHr1lC3Lvz4I7z/vp1C07y5jn8RyQ71TImIBMl7mDjRntL7/ns7P++tt2xlKkISE21q+XvvWW57/nnLcMWKRawEkZimMCUiEpSlS625fMoUa0qaMAEaN47YMlBSEvTsaeMNduyATp0sSEXwIUGRuKBtPhGRnJaYCHfdBeedB/PnW6JZssQaziMQpPaeQHPGGXYETL16dvnevRWkRIKglSkRkZyyeze8/bbtqSUlwb33wnPPwfHHR+Ty3ttxL48/buGpdnXj+U8AABs2SURBVG0YOtSazUUkOFqZEhHJLu9ttME559hBdpdeammmZ8+IBalFi+yol6uvti29ESMyntoTkWApTImIZMeiRXaGXsuWUKiQLQ1NmABVqkTk8r/+Cu3aQc2aVkrPnrBiRURPoBGJewpTIiJH488/oWNHSzFLl9qjcosXR+wk4M2bbTvvzDPho4/giSds9MH990OBAhEpQUTSqWdKRORI7NxpR7785z/WI/Xww3aIXYkSEbn8li3WlvXWWxao2rWzFq3y5SNyeRE5CIUpEZGs8B5GjrQloF9/tYnlr78Op58ekctv2WIzP/eGqObNoVs3e2BQRMKlbT4RkcOZN8+aytu0sRWo6dOt4TwCQWrLFgtNFStCly7WUD5/vh0BoyAlEh0UpkREDmXNGttHq13bGpL69YMFC+CKKwK/9ObNGSHquefg8svt0mPHWpuWiEQPbfOJiGS2bh289ppNufQennrKPooXD/zSmzfbE3ndu9uq1LXX2opUjRqBX1pEjpLClIjIXmvXwquvQp8+djDxLbdYkjn11MAvvXkz9OhhHwpRIrmLwpSIyB9/2AF2fftCSgq0b2/nsJx2WuCXzhyiWrSwEFW9euCXFpEcojAlIvFrzZqMEJWWlhGiKlUK/NKbNmWEqK1bbeZnly5qKhfJjRSmRCT+/P67zYnq399C1K23WoiqWDHwS2/aZP1QPXsqRInECoUpEYkfv/5qIWrAAPv6ttussfyUUwK/9MaNFqJ69bIQdd11FqKqVQv80iISMIUpEYl9v/xiISohwb7u1AmefBIqVAj80ntDVM+esG0btGoFzz6rECUSSxSmRCR2rV4NL78MH3wAefLA7bdbiIrA2SsbNmSsRO0NUV26wLnnBn5pEYkwhSkRiT0//wwvvQQffgh580LnznYMTLlygV96wwY78qVXL9i+HVq3tpUohSiR2KUwJSKxY9UqC1GDBkG+fHD33fD441C2bOCXXr/eQtTbb8OOHRkhqmrVwC8tIiFTmBKR3O/HH+HFF2HIEMifH+6910LUyScHfmmFKBFRmBKR3OuHHzJCVMGCcP/98NhjUKZM4Jdevx7efNNCVFISXH+9hahzzgn80iISZRSmRCT3WbnSQtSwYRaiHnrIQtSJJwZ+6cREC1H//a+FqBtusBB19tmBX1pEopTClIjkHitWwAsvwPDhULgwPPIIPPoonHBC4JdWiBKRQ1GYEpHot2yZhaiRI6FIEVuFevRRKF068EsnJsIbb8A771iIatPGQtRZZwV+aRHJJRSmRCR6LV1qIeqjj6BoURtv8MgjUKpU4Jdes8YGbb77LuzcCTfeCM88oxAlIgdSmBKR6LNkCXTrBqNGQbFiduTLww9DyZKBX3rBAhu2OWKEHdvXpo1ClIj8M4UpEYke335rIWrMGChe3FLMQw/B8ccHetnUVJgwwUYcfPGF5bd777WHA089NdBLi0gMUJgSkfAtWmQh6pNP4JhjrCnpwQcDD1E7dthJMz162LzPChWsybxjRzj22EAvLSIxRGFKRMLhvS0DvfUWjBtn6eW55+CBB+C44wK99B9/2FN5778PmzZB7dq2rdeypQ1OFxE5Eof914ZzrjzwIXASkAb08d73zPSeesBYYHX6t8Z477vlbKkiEhN277bRBj16wOLFtvrUtavtqZUoEeilFy60fqjhw60fqkULa8WqWxecC/TSIhLDsvLfYCnAI977hc654sAC59wU7/3yTO/70nvfNOdLFJGY8Oef0Ls3vPce/P23DWjq0wduvtnGHQQkLS2jH2rmTOuHuucey26VKgV2WRGJI4cNU977dcC69M+3OedWAGWBzGFKRORACxfajIHhw2HPHmjSxPqhrrwy0OWgHTtg4EBbAPvxRyhf3uZFdeqkfigRyVlH1B3gnKsI1ADmHuTlus65b4G1wKPe+2UH+evvAO4AqFChwpHWKiK5RWoqjB1rIeqLL2xG1B13wH33wRlnBHrptWutH6p3b+uHuuACy3EtW9oZyCIiOS3LYco5VwwYDTzovd+a6eWFwCne++3OucbAJ0DlzP8b3vs+QB+AWrVq+aOuWkSi0+bN0L+/pZlffoFTTrHH4267LfB+qEWLMvqhUlIy+qEuukj9UCISrCyFKedcfixIDfHej8n8+r7hyns/yTn3rnOulPd+fc6VKiJR68cfoVcvSEiw/bXLLrMQ1bx5oI/HpaXBxInWDzVjhvVD3XWX9UOddlpglxUR2U9WnuZzQH9ghff+rUO85yTgL++9d87VBvIAG3K0UhGJLt7D1Km2lTdxIhQoYOPCH3gAzj8/0EsnJWX0Q/3wA5QrB6+/bv1QAS+AiYgcICv/yXgxcAuwxDm3OP17TwMVALz3vYFWwF3OuRRgJ9DGe69tPJFYlJQEQ4ZYiFq2DE44weZDde4MJ50U6KXXrrUDh3v3ho0boVYtGDYMrrtO/VAiEp6sPM03C/jHjgPv/X+B/+ZUUSIShdassVN/33/fkkz16jY+vE0bKFgw0EsvXmz9UMOGWT/UtddaP9TFF6sfSkTCp1m/IvLP5s61/bRRo+wpvWuvtdEGl14aaJJJS4NJkyxETZ9uDwSqH0pEopHClIgcKDkZRo+2EDV3rp2Xd//9dvpvwCf/JiXBhx9aiNrbD/Xaa3D77eqHEpHopDAlIhk2bLCp5O+8YwfYnX46vP02tG8PxYsHeul16+yy772X0Q81dCi0aqV+KBGJbgpTImKN5L16waBBsHMnNGhgXd6NG0OePIFe+ttvbRVq6NCMfqiHHoJLLlE/lIjkDgpTIvEqLQ3+9z/bypsyBQoVgltuse28qlUDvfT27TBiBPTta7uIRYvaw4D332+LYSIiuYnClEi82b7dhjT16mVNSSefDC+9ZMe9lCoV6KUXLLBdxGHDYNs2OOssG7jZoQMcd1yglxYRCYzClEi8+OUXO+alXz/YsgVq145IU9KWLXaZvn3tyJfCheH6662hXEe9iEgsUJgSiWUpKfDZZxagxo615NKqlY02qFMnsMt6D3Pm2CrUyJH2hF716tZgftNNeipPRGKLwpRILPrhBzsn78MPbWx4yZLw+ONw991Qvnxgl9240XrY+/a1nvZixeDmm20HsWZNrUKJSGxSmBKJFVu32jJQQgLMng1588LVV9tog6ZN7ey8AHgPX3xhAWrUKNi923YQ+/aFG24IfKKCiEjoFKZEcrO0NEsyCQmWZJKSoEoVePVVezKvTJnALv3339bH3q+fLYQde6wdNHz77XDeeYFdVkQk6ihMieRGv/1mSSYhAVavtuWftm3h1lvhwgsD209LS4Np06wXauxYG5R+ySXw739bK1aRIoFcVkQkqilMieQWO3fCxx9bgJo2zfbX6teHbt2gZctAk8zatXbZ/v0tu5UsaSfLdOoEZ58d2GVFRHIFhSmRaOY9zJtnSWbYMJszULEiPPecHfFSsWJgl05NtZmeffrAxIn2df368PLL0KIFFCwY2KVFRHIVhSmRaPTXX/ZYXEICLF9uw5muu8628erVC/SIl99+sxWoAQNgzRo48UR49FFbhdJ0chGRAylMiUSL5GRbAkpIyFgKqlPHloauv946vAO89Pjx9gTe5Mn2vUaNoGdPaNZMBw2LiPwThSmRsC1dagFq0CBITISTToJHHrEzVs46K9BL//STPY2XkGCLYWXLwjPPQMeOcMopgV5aRCRmKEyJhGHTJuuBSkiA+fNt6adZM7jtNlsSyhfcP5q7d8Mnn9iC1/TpNo6qSRMbaXDVVYFeWkQkJulfmyKRkppqT+ElJNhTebt3Q7Vq0KOHjQkP+JDhlSttG2/gQNiwwXrXX3jB2rDKlg300iIiMU1hSiRoq1bBBx9YilmzBo4/3paBbr0VatQI9IyVjRttluegQTBrlq06XXutXb5Bg0D72EVE4obClEgQtm+3FJOQYBPK8+Sx7bu33oLmzQOdK7BrF0yYAIMHw6RJ1lxepQq89ppNUzjhhMAuLSISlxSmRHKK9/DVVxagRo60QFW5sg1matcu0L201FSYOdMC1OjRdkxfmTJw3302GL16dR0yLCISFIUpkezwHhYssAQzapRt6RUrZif83norXHRRYCnGe1i8GIYMsV72tWvtVJnrrrMWrCuusOZyEREJlsKUyJFKTYWvv7YANWaMTbnMm9fSyzPP2CF1RYsGdvlffoGhQy1ELV9uDwJefbWtQDVtavM9RUQkchSmRLIiOdn20UaPtrkCf/5pfU8NG0LXrtYHdfzxgV1+wwb46CPbxvvqK/veJZdA796W3UqWDOzSIiJyGApTIoeyezdMmWIBatw4ezSuaFFo3NgOFm7SxPbVArJzp00lHzzYzshLTrZDhV9+GW68MdBj+URE5AgoTInsa8cO+PRTC1ATJ8K2bXaMS/PmFqAaNQp0Hy011QZpDhliO4jbtsHJJ8MDD1gf1HnnqZFcRCTaKEyJbN5sswRGj7YloF27oHRpaNPGAlT9+lCgQGCX9x4WLrQANXw4rFsHxxwDrVtbgLr8cjWSi4hEM4UpiU+JiTB2rAWoadNsD61sWZtm2bKlNSQFfK7Kzz9bI/ngwfD999ZI3qSJBaimTaFQoUAvLyIiOURhSuLHH3/YMS6jR9sgzbQ0qFQJHnzQAlTt2oGPBF+/3kZQDR5sDwQCXHaZnWvcqhUcd1yglxcRkQAoTEls+/lnaz4aPRrmzLHvnX02/PvfFqAi0ISUlGSLYEOGwOTJkJICVavCK69YI3mFCoFeXkREAqYwJbFn+fKMALV4sX3v/PPhpZcsQFWpEngJKSm2ezhkiC2Gbd8O5crBww/bNl61aoGXICIiEaIwJbmf97BoUUaAWrnSvn/RRfDmm9CiBZx6akTKmD8/o5H8r7/sQcA2bSxAXXaZDhYWEYlFClOSO6Wl2bbdmDH2sXq1PfJ2+eV2IN2119pMgYDtDVB7c9yPP9qDf02bWoBq3FiN5CIisU5hSnKPXbts/PfHH9vH2rX2CNy//mXHuDRvDqVKBV5GSgrMmpVRxu+/W46rVw8ef9zOxlMjuYhI/FCYkuiVnGzLPtOn28fs2RaoChe2w+hatrQloGOPDbyU3butB2rMGGsmX7/eVpwaNYIXXoBmzQI9TUZERKKYwpREj9RU+PbbjPD05ZfWuQ321N1dd9lhwvXrB3qQ8F7bt9sw9DFjMoahFy9u+a1lS7jqKihWLPAyREQkyilMSXi8tyfvpk+Hzz+HGTNg0yZ7rUoVaNfOwlO9ehHZvgM7UHj8eNu+mzzZVqRKlYIbbsgYhl6wYERKERGRXEJhSiLHe/jpp4zwNH06/P23vXbqqfbUXf36FqAi0Dy+19q18MkntgI1Y4YtkJUvD507W0mXXKLjXERE5NAUpiRYv/+eEZymT7evAcqUscbxveEpAqML9rVqla0+jRmTMcvzzDOtgbxlS6hZUwcKi4hI1ihMSc766y9b3tkbnlatsu+XLGmh6amnLECdcUZE04r3sGRJRoD67jv7/vnnw4svWoA666yIlSMiIjFEYUqyZ9MmmDkzIzwtW2bfP+YYm/l0zz0WnqpWjfjEyrQ0+OabjFFUP/1k+e2SS6B7dxtFVbFiREsSEZEYpDAlR2bbNhuytDc8LVpkyz6FC8Oll8Itt1h4qlED8kX+xys52c4wHjPG+qD2jqKqX9+28K65Bk48MeJliYhIDFOYkn+2cyd8/XVGePrmG+vQLlAA6taF55+3pFK7tn0vpBKnTLEtvHHjYOPG/UdRNWkCJUqEUpqIiMQBhSnZ35Yt1lC0t+/p669tPkDevHDBBfDEExae6taFIkVCK3PrVpg0yVagJk2CHTtsdmfz5hagGjYMtTwREYkjClPxatcuOxB46VLrzF661D5++81edw6qV4d777XG8UsvtT6okKSkwIoVlu3GjoWpU2HPHtuya9vWAlS9eqEtjomISBxTmIp1qanWeb1vYFq61E7kTU219+TPb4+yXXKJNYqfe66tPJUsGUrJaWn2EOC8eXaazLx51pqVlGSvV6xoGa9lS6hTRzOgREQkXApTscJ7+OOP/UPTkiW2nLNrl73HOTjtNAtMrVvbn1WrQuXKFqhCKvu33/YPTgsW2G4jWO9TjRpw++1Qq5btNEZ4qoKIiMg/UpjKjTZuPDA0LV2akUDAJohXrZoxlqBqVTj77NAbidatywhN8+fbR2KivZY/vx3Bd+ONFppq1bKSQ3goUEREJMv0ayqa7dhhZ9dlDk3r1mW8p0QJC0o33ZQRmqpWheOPD6/udBs2ZASmveHpjz/stTx54Jxz7NDgCy6wj3PP1bl3IiKS+yhMRYPkZPjhhwObwX/+2fbBAAoVsmWahg0zAtO559oKVBTseW3dCgsX7r9dt3p1xutnnGEN4nu36qpXh6JFQytXREQkxyhM5RTvrTcpKck+duzI+PxgX2/ZkvE03cqVFqjAuqkrV7ZzTtq1ywhNlSpFTaf1zp2wePH+wen77zNyX8WKFpo6d7Y/a9a0sQUiIiKxKD7CVOagk5Wwc6RfJyVlpImsqlDBglLjxhmh6cwzbRUqSuzZY4tl+27VLV2a8SBgmTK20nTTTfZnzZpQunS4NYuIiERS7Iap776DBg2OPujkyWPN2kWL2p97P4oWhRNO2P/rzK9n9esoaxDascMe/vvuO3uibt48+PZbC1RgbVgXXADNmmVs1518crg1i4iIhC12w1TJknDddUcfeAoUiIpepCDs2ZPRorXvx74tWsWL2yrTAw9kBKeKFWP2loiIiBy12A1TZcvCe++FXUWo0tKsCTxzaPr++/1btM44w4JT+/a223jOOXD66bY4JyIiIv8sdsNUHPEe1q49MDQtX54xNRzg1FMtLDVrlvFA4JlnRt1uo4iISK6iMJXLbNgAy5YdOHpq8+aM95QpY0Hpzjv3n9dZrFh4dYuIiMQqhakotX37/vM6934cbF5nmzb2IODeLbqQjtQTERGJSwpTIduzx3qYMq807TvwsnBhC0mNGu0/5DxK5nWKiIjEtcOGKedceeBD4CQgDejjve+Z6T0O6Ak0BpKADt77hTlfbjDS0qwhOznZwk3mj4N9P7vvTUqyEPXDD5CSYnXkywdVqsCFF0LHjhmhqWLFqJnXKSIiIplkZWUqBXjEe7/QOVccWOCcm+K9X77Pe64GKqd/XAi8l/5naL7/Hjp0yFro2RtmglCggH3kz5/xeYECNpezcmVo0SJjXmflyvaaiIiI5B6HDVPe+3XAuvTPtznnVgBlgX3D1DXAh957D8xxzpVwzpVJ/2tDkT+/zUraN8AcLNQc7fez8tfky6dtOBERkVh3RD1TzrmKQA1gbqaXygK/7/P1mvTvhRamKlWCzz4L6+oiIiISL7I8ltE5VwwYDTzovd+a+eWD/CUHnN/inLvDOTffOTc/MTHxyCoVERERiUJZClPOufxYkBrivR9zkLesAcrv83U5YG3mN3nv+3jva3nva5XWabgiIiISAw4bptKf1OsPrPDev3WIt40D2jlTB9gSZr+UiIiISKRkpWfqYuAWYIlzbnH6954GKgB473sDk7CxCKuw0Qi35nypIiIiItEnK0/zzeLgPVH7vscD9+RUUSIiIiK5RZYb0EVERETkQApTIiIiItmgMCUiIiKSDQpTIiIiItmgMCUiIiKSDQpTIiIiItmgMCUiIiKSDQpTIiIiItngbN5mCBd2LhH4NZSLR49SwPqwi4giuh/70/3Yn+7HgXRP9qf7sT/djwNl556c4r0/6MHCoYUpAefcfO99rbDriBa6H/vT/dif7seBdE/2p/uxP92PAwV1T7TNJyIiIpINClMiIiIi2aAwFa4+YRcQZXQ/9qf7sT/djwPpnuxP92N/uh8HCuSeqGdKREREJBu0MiUiIiKSDQpTAXPOXeWc+945t8o59+RBXi/onBuR/vpc51zFyFcZWVm4Jx2cc4nOucXpH53CqDMSnHMDnHN/O+eWHuJ155zrlX6vvnPOnR/pGiMtC/eknnNuyz4/H10iXWMkOefKO+c+d86tcM4tc849cJD3xM3PSRbvR9z8jDjnCjnnvnHOfZt+P7oe5D1x83smi/cj53/HeO/1EdAHkBf4CagEFAC+Bc7O9J67gd7pn7cBRoRddxTckw7Af8OuNUL34zLgfGDpIV5vDHwKOKAOMDfsmqPgntQDJoRdZwTvRxng/PTPiwM/HOSfmbj5Ocni/Yibn5H0/8+LpX+eH5gL1Mn0nrj5PZPF+5Hjv2O0MhWs2sAq7/3P3vs9wHDgmkzvuQYYmP75KOBK55yLYI2RlpV7Eje8918AG//hLdcAH3ozByjhnCsTmerCkYV7Ele89+u89wvTP98GrADKZnpb3PycZPF+xI30/8+3p3+ZP/0jczN03PyeyeL9yHEKU8EqC/y+z9drOPAf+v9/j/c+BdgClIxIdeHIyj0BuC59u2KUc658ZEqLSlm9X/Gmbvoy/qfOuXPCLiZS0rdnamD/tb2vuPw5+Yf7AXH0M+Kcy+ucWwz8DUzx3h/y5yMefs9k4X5ADv+OUZgK1sGSf+aEnJX3xJKs/P2OByp676sBU8n4L6p4FG8/H1mxEDvW4TzgbeCTkOuJCOdcMWA08KD3fmvmlw/yl8T0z8lh7kdc/Yx471O999WBckBt51zVTG+Jq5+PLNyPHP8dozAVrDXAvom3HLD2UO9xzuUDjiW2tzgOe0+89xu897vTv+wL1IxQbdEoKz9DccV7v3XvMr73fhKQ3zlXKuSyAuWcy48FhyHe+zEHeUtc/Zwc7n7E488IgPd+MzADuCrTS/H2ewY49P0I4neMwlSw5gGVnXOnOucKYI1/4zK9ZxzQPv3zVsB0n94hF6MOe08y9Xo0x3oi4tU4oF3601p1gC3e+3VhFxUm59xJe/s9nHO1sX+PbQi3quCk/732B1Z47986xNvi5uckK/cjnn5GnHOlnXMl0j8vDDQAVmZ6W9z8nsnK/Qjid0y+7P4PyKF571Occ/cCk7Gn2AZ475c557oB873347B/KQxyzq3C/kuhTXgVBy+L9+R+51xzIAW7Jx1CKzhgzrlh2JNHpZxza4DnsIZJvPe9gUnYk1qrgCTg1nAqjZws3JNWwF3OuRRgJ9AmVn8xpLsYuAVYkt4HAvA0UAHi8uckK/cjnn5GygADnXN5sdA40ns/IY5/z2TlfuT47xhNQBcRERHJBm3ziYiIiGSDwpSIiIhINihMiYiIiGSDwpSIiIhINihMiYiIiGSDwpSIiIhINihMiYiIiGSDwpSIiIhINvwfUm2CmzAgp0IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,6))\n",
    "plt.plot(alpha_range, train_loss2, 'b')\n",
    "plt.plot(alpha_range, val_loss2, 'r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Losses went nan for the initial values. Seeing this we cannot combine both of them, but the reason being in the paper SGD was used while here we are using AdamW with cyclic momentum by default so it changes up the situation by a lot. But it is a fun experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explanation behind Super-Convergence\n",
    "As we discussed earlier, one of the indicators of super-convergence are the consistent high value of accuracies with increasing learning rate. Cyclic Learning which allowed super-convergence is indeed a combination of **Curriculm learning** and **simulated annealing**. Also, as the amount of data decreases the gap in performance between the result of standard training and super-convergence increases. Specifically, with a peicewise constant learning rate schedule the training encounters difficulties and diverges along the way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='Extra/maxout_3d.png' width='85%'> | <img src='Extra/maxout_3d_zoom.png' width='85%'>\n",
    "- | -\n",
    "(a) Visualization of how training traverses a loss function topology | (b) A close up of the end of training for figure (a)\n",
    "\n",
    "<center>The 3-D visualizations from Goodfellow et al. [2014]. The z-axis represents the loss potential.</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above figure gives an intuitive understanding of how super-convergence happens. The blue line in the figure represents the trajectory of the training while converging and the x's indicate the location of the solution at each iteration and indicates the progress made during the training.\n",
    "\n",
    "The while loss surface can be divided into 3 phases:-\n",
    "1. In early training, the learning rate must be small in order for the training to make progress in appropriate direction. As you can see in the figure a significant progress is made in those early iterations (the part where we descend the valley)\n",
    "2. Now as the slope decreases so does the amount of progress made per iteration and little improvement occurs over the bulk of the iteration. This is the reason why we increase the learning rate to high values, so that we can quickly move over this region.\n",
    "3. As we approach the bottom of the loss surface (you can think of it as bottom of valley with bumps), so here we need to slow down and get to the bottom of these bumps i.e. why we decrease the value of learning rate to minimum value so that we can fine-tune our final result.\n",
    "\n",
    "### A quick summary\n",
    "Initially when the learning starts there is huge slope and we move down it quickly, now there is a straight out path and we make very less progress through each iterations. At the end we enter a valley and we have to move towards the minima. Cyclic Learning Rate solves it. We start with small learning rate to get over that initial big slope. Then we increase the learning rate to quickly move through the straight path and then we again decrease learning rate to move through the valley."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choosing momentum value\n",
    "Choosing the value of momentum depends on the task at hand. So therea are two options either use cyclic momentum or constant value of momentum. I would directly give you the results here\n",
    "* IF you are training GAN's or any task where you are quickly shifting between different models (like in GANs we shift between generator and discriminator) you should use constant value of momentum. Because you will not have enough time to get the benefits of cyclic momentum in these tasks.\n",
    "* Cyclic momentum should be your default choice for other tasks. And for the momentum values they should be high->low i.e. opposite of learning rate.\n",
    "\n",
    "In implementation we would specify the minimum and maximum value of momentum and move from max->min and then min->max. Below I show the cyclic learning rate and cyclic momentum values side by side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.013027</td>\n",
       "      <td>3.831154</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAucAAAEGCAYAAADG9ztyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3yV9fn/8deVDSEkkMEKEEYIe4ugDEFFnIiodWu1+rU/bZ2to446sNI6Wq1abbXOioqgOBAVmaLIEMIICYEwwgghQBISsq/fH+dg0xjIAXJyn3E9H4/7wTn3ue8776CcXLnP53N9RFUxxhhjjDHGOC/E6QDGGGOMMcYYFyvOjTHGGGOM8RFWnBtjjDHGGOMjrDg3xhhjjDHGR1hxbowxxhhjjI8IczqAkxISEjQlJcXpGMYYc8xWrFixV1UTnc7RlOw92xjjzzx93w7q4jwlJYXly5c7HcMYY46ZiGx1OkNTs/dsY4w/8/R924a1GGOMMcYY4yOsODfGGGOMMcZHWHFujDHGGGOMj7Di3BhjjDHGGB9hxbkxxhhjjDE+wqvFuYhMEJFMEckWkXvreT1SRN5zv75URFJqvXafe3+miJzl3hclIj+IyGoRWScij9Q6vov7Ghvd14zw5vdmjDHGGGNMY/NacS4iocALwNlAb+ByEeld57AbgP2q2h14FpjqPrc3cBnQB5gAvOi+XjkwTlUHAAOBCSIy3H2tqcCzqpoK7Hdf2xhjjDHGGL/hzT7nw4BsVd0MICLTgInA+lrHTAT+6H48Hfi7iIh7/zRVLQdyRCQbGKaq3wEH3ceHuzd1nzMOuML92hvu677knW/NHK+q6hpWbT9Axq4iCkoqCA8NISo8lOiIUNrERtEuNooOcc2IiQp3OqoxJgB8sHw7ew9W0D4uilO7J5DQItLpSMYYP6Gq5BeXs7OwjN2Fh8gvLqeorIrhXeMZ0rmV176uN4vzDsD2Ws9zgZOPdIyqVolIIRDv3v99nXM7wE935FcA3YEXVHWpiCQAB1S1qu7xdYnITcBNAJ06dTrub84cm5LyKl5fsoXXFudQUFLR4PHJrZrRq11L+neIZXi3eAYkxxERZlMkjDHHZtbqnSzauBcAERiXlsTtZ/SgX3Ksw8mMMb5mX0kFSzcXsDRnH+t3FbFhVxFFZVU/O+53Z6X5bXEu9exTD4854rmqWg0MFJE4YKaI9AXyPPhauM9/BXgFYOjQofUeYxrX95sLuPuD1eTuP8RpaYlcMqQjQ1NaER8dQVWNUl5ZQ1FZJXuKy9hVWMbWglI27C4mY1cRX2fkoV9BVHgIp3ZLYELftpzZuw1xzW1KgTGmYW/dcDIHy6vIyS/hy/W7efO7rVzwwmJuHtONO87oYb/0GxPkNuYVM3vtbr5Yu5v1u4oAaBYeSq92MZw/oD1pbWPoENeMtrFRJMVE0bJZGJFhoV7N5M3iPBfoWOt5MrDzCMfkikgYEAvs8+RcVT0gIvNxjUl/GogTkTD33fP6vpZxwNvfb+Whj9fSqXVzPrh5BCeltP6f18NCISo8lNjm4XRs3fxn5x8oreD7zfv4btNevs7Yw9wNewgLEcb2TOLyYR0Z0yOJ0JD6fpczxhiXFpFh9EuOpV9yLDeO7sqUTzN4af4m0nMP8I+rhtgwOmOCzIHSCmas3MF7y7aTmVeMCAzp1Iq7x/dgRLd4+ifHER7q3C/uouqdm8fuYjsLOB3YASwDrlDVdbWOuQXop6o3i8hlwEWqeqmI9AH+g2vcentgLpAKtAYq3YV5M+BLYKqqfioiHwAfquo0EfkHkK6qLx4t49ChQ3X58uWN/a0bt5fmb2LqFxsYm5bI81cMpkXkif0uqKqk5xby+ZpdfLgy1zWONDaKa09J4YqTO9kPWBNURGSFqg51OkdTasz37Okrcrnnw3T6tG/Jf24cfsLvT8YY35e9p5iXF2zm49U7qaiqYUDHOCYP7sCEPm1Jahnl9a/v6fu214pzd4hzgL8CocBrqjpFRB4FlqvqLBGJAt4CBuG6Y35ZrQmkfwCuB6qA21V1toj0xzXZMxRXp5n3VfVR9/FdgWm4CvgfgavcE0qPyIpz73nr+608+NFaLhjQnmcuHUBYI/8GWlFVw9yMPN76fitLNhUQExXG1cM7c+OorrSKtiEvJvBZcX7ivlqfx81vr2B419a8dt1JXv+o2hjjjPTcAzw3N5uvM/KICg/h4iHJXD6sE33aN+3cE58ozn2dFefesXjjXq55bSlj05L4x9VDvP7RUHruAf6xYBOz1+6mRWQYN4/pxvWndqFZhP2gNYHLivPG8eGKXO76YDVXntyJKZP6Neq1jTHO2px/kKe+zOTzNbuJax7ONSNSuO6UFFo7dBPP0/dt+xzPNKodBw5x67sr6Z7UgucuH9QkY7b6J8fx4pVDyNxdzF/mbOAvczJ587st3H9OLy4Y0B5Xp01jjPm5yUOSyXJ/1D2kcysuGpzsdCRjzAkqKqvk2a+yePO7rUSGhfDb01O5cVQXvxn+asW5aTTVNcqd762isqqGV64eSnQTj+FMaxvDv649iWVb9vHoJ+u5bdoq3lu2nUcn9qF7UkyTZjHG+I/fjU9j1bYDPPjRWoZ1aU1yq59PTjfG+D5V5eNVO5nyeQZ7D5Zz+bBO3HFGDxJj/Gt9A+shZRrNq4s3szRnH3+8oA8pCdGO5TgppTUf3XIqj13Yl7U7Cjnnb4t5YV42VdU1jmUyxviusNAQnr50AAC/n55OTU3wDvc0xl/tKjzEdf9exu3vraJ9bBQf33IqT0zq53eFOVhxbhrJtoJSnvkqizN7t+HiIc5/LBwaIlw9vDPf3H0aZ/Zuw1/mZDL5pSVszCt2Opoxxgclt2rOH87tzZJNBXy4MtfpOMYYD6kq01fkMv7ZhfyQs48/nt+bmf/vVPonxzkd7bhZcW5OmKry4MdrCRXh0Yl9fGqMd0KLSF64cjB/v2IQ2/aVct7zi3n3h20E80RoY0z9LjupI0M6t+LJ2RsoLK10Oo4xpgEHy6u4bdoq7v5gNT3bxjD7tlFcd2oXQvx8/RMrzs0Jm5+Zz4KsfO4cn0a72GZOx6nXef3bM+eO0ZyU0pr7ZqzhN+/+SFGZ/fA1xvxXSIjrBsP+0gqe/TrL6TjGmKPI2FXEBc8v5tP0ndw9vgfTbhrh6JDaxmTFuTkh1TXK1C82kBLfnGtGdHY6zlElxUTx5vXD+N1Zacxeu5sL//4t2XsOOh3LGOND+rSP5RcndeKdpVvZWlDidBxjTB2qyrQftnHhC99SXF7FO78azq3jUgNqtXArzs0J+ejHHWzYXczdZ6U5utStp0JChFvGdufdG4dTeKiSSS98y7wNe5yOZYzxIXeckUpYSAhPfWl3z43xJZXVNdw/cy33zljDSSmt+fy3oxjRLd7pWI3O96sp47PKKqt55qss+ifHck7fdk7HOSbDurRm1m9G0im+Ode/sYyX5m+ycejGGACSWkZxw8gufLJ6J5m7bRK5Mb6gsLSSa1/7gXd/2MYtY7vxxvXD/LITiyesODfH7e3vt7LjwCHumdDTLydfdIhrxvSbT+Hcfu2Y+sUG7nhvFeVV1U7HMsb4gBtGdqF5RCgvzc92OooxQW/L3hImvfgty7bs4+lLBvC7s3oG1DCWuqw4N8eltKKKF+ZlMyo1gVO7Jzgd57g1iwjl+csHcff4Hny0aic3vL6cg+VVTscyxjisVXQEV57ciU/Sd7GtoNTpOMYErR9y9nHhi9+yv7SCd341nMk+0K7Z26w4N8flg+W57C+t5LbTU52OcsJEhFvHpfLUJQP4bnMBl7/yPXsPljsdyxi/ICITRCRTRLJF5N56Xu8sInNFJF1E5otIcp3XW4rIDhH5e9Ol9syvRnUlVISXF25yOooxQWluRh5XvbqU1tERfHTLqQzr0trpSE3CinNzzKqqa/jnos0M6dyKoSmB8w/l4iHJ/POaIWzcU8zFLy1h+z67W2bM0YhIKPACcDbQG7hcRHrXOewp4E1V7Q88CvypzuuPAQu8nfV4tGkZxeQhyXywIpc9RWVOxzEmqHy8agf/99YKeraNYfrNp9A5PjDaJHrCinNzzGav3U3u/kPcNLqr01Ea3biebXjnV8PZX1rJRS8tIctWFDXmaIYB2aq6WVUrgGnAxDrH9Abmuh/Pq/26iAwB2gBfNkHW43LzmK5UVdfw6uIcp6MYEzTe/n4rt7+3iiGdW/HOr06mdXSE05GalBXn5pioKi8v3ETXhGjO7NXG6TheMaRzK6bfPAIBLn/le+vWYMyRdQC213qe695X22pgsvvxJCBGROJFJAR4Gvjd0b6AiNwkIstFZHl+fn4jxfZc5/hozu3fnneWbqPE5qMY43X/WLCJBz5ay7i0JN64fhgxUeFOR2pyVpybY/LdpgLW7ijixtFd/bJDi6dS28Qw7abhhIUKV/zTCnRjjqC+N4G6PUnvBsaIyI/AGGAHUAX8P+BzVd3OUajqK6o6VFWHJiYmNkbmY3bdKSkcLK/i41U7Hfn6xgSLl+Zv4snZG7hgQHv+cfUQosJDnY7kCCvOzTF5ZdFmElpEMmlQ3ZtjgadrYgum3TSCsFDh8n9+z4bdRU5HMsbX5AIdaz1PBv6nglXVnap6kaoOAv7g3lcIjABuFZEtuMalXyMiTzZJ6mM0uFMcvdq15K3vt9p6CMZ4yb8WbWbqFxuYOLA9z/5ioF8sbOgtwfudm2O2raCU+Zn5XHlyp6D5bbZLQjTTbhpBRGgIl79iBboxdSwDUkWki4hEAJcBs2ofICIJ7iEsAPcBrwGo6pWq2klVU3DdXX9TVX/W7cUXiAhXDe9Exq4iVm474HQcYwLOG0u28PhnGZzTry1PXzIgoHuYe8KKc+Oxd5dtI0TgsmEdGz44gLgK9OFEhoVy9as/sLWgxOlIxvgEVa0CbgXmABnA+6q6TkQeFZEL3IedBmSKSBauyZ9THAl7gi4c2IEWkWG88/1Wp6MYE1DeWbqVh2et48zebfjbZYMIC+I75ofZ34DxSEVVDR8s3864nm1oF9vM6ThNLiUhmrduGEZVdQ1XvbqUPGurZgwAqvq5qvZQ1W6qOsW97yFVneV+PF1VU93H/EpVf7aIgKq+rqq3NnX2YxEdGcZFgzvwafou9pVUOB3HmIDw0Y87+MPMtYzrmcTfrxgU1ENZarO/BeORL9fvZu/BCq4c3snpKI5JbRPD678cxr6DFVzz6g8cKLUf0MYEk6uGd6ai2nWjwhhzYhZk5XP3B6sZ3rU1L145mMiw4Bgu6wkrzo1H/rN0Gx3imjE61ZluCb5iQMc4/nnNUHL2lvDL15dRWmGt1YwJFj3axDAspTXTlm23iaHGnIBV2w/w67dX0KNNDK9cMzRo5rF5yopz06DN+QdZsqmAK07uFPSTNABO6Z7A81cMYvX2A9z89koqq2ucjmSMaSIXD0kmZ28Jq7bbxFBjjsem/INc//oy4ltE8Pr1J9EyCPuYN8SKc9Ogacu2ExYiXDI02ekoPuOsPm15YlI/Fmbl89DHa+0umjFB4ux+bYkMC+HDlblORzHG7+QVlXHNqz8gwFvXn0xSTJTTkXySV4tzEZkgIpkiki0iP2uRJSKRIvKe+/WlIpJS67X73PszReQs976OIjJPRDJEZJ2I3Fbr+D+KyA4RWeXezvHm9xYsqqprmPnjDsb2TLJ/RHVcNqwTt4ztxrs/bOcfCzY7HccY0wRiosI5q09bPlm9i/KqaqfjGOM3SiuquP71ZRworeD1Xw4jJSHa6Ug+y2vFuYiEAi8AZwO9gctFpHedw24A9qtqd+BZYKr73N64+uX2ASYAL7qvVwXcpaq9gOHALXWu+ayqDnRvn3vrewsm324qIL+4nMmDA3/RoeNx15lpnD+gPVO/2MAnq231QGOCwUWDO1B4qJJ5G/Y4HcUYv1BTo9w+bRUZu4r4+xWD6Zcc63Qkn+bNO+fDgGxV3ayqFcA0YGKdYyYCb7gfTwdOFxFx75+mquWqmgNkA8NUdZeqrgRQ1WJcfXWtavSimStziW0WztieSU5H8UkhIcJfLu7PSSmtuOuD1Szfss/pSMYYLxvZPYHEmEg+XLnD6SjG+IWpX2zgy/V5PHheb6snPODN4rwDULvfVC4/L6R/Osa9mEUhEO/Jue4hMIOApbV23yoi6SLymoi0qi+UiNwkIstFZHl+fv6xfk9B5WB5FXPW5XFe/3bW4ugoosJDeeXqoXSIa8aNby5nW0Gp05GMMV4UFhrChQPbM2/DHut5bkwDpv2wjZcXbubq4Z257pQUp+P4BW8W5/W19ag7a+5Ixxz1XBFpAXwI3K6qh9dTfwnoBgwEdgFP1xdKVV9R1aGqOjQxMbjbAjbki7W7OVRZzUU2pKVBraIjeO26k6iuUW56azkl5dZi0ZhAdtHgZKpqlFmr7O65MUeyJHsvD3y0ltE9Enn4/N64BkeYhnizOM8Faq/zngzUHZT70zEiEgbEAvuOdq6IhOMqzN9R1RmHD1DVPFWtVtUa4J+4htWYEzDzx1w6xzdncKd6P4QwdXRJiObvVwwmK6+YO99fRU2NdXAxJlD1ateSnm1j+CR9l9NRjPFJ2wpK+fU7K90/GwcRZqt/esybf1PLgFQR6SIiEbgmeM6qc8ws4Fr344uBb9TVk24WcJm7m0sXIBX4wT0e/VUgQ1WfqX0hEWlX6+kkYG2jf0dBZFfhIZZsKuDCgR3sN91jMLpHIvef04s56/J4/ptsp+MYY7zovP7tWLF1P7sKDzkdxRifUlpRxU1vLQfgX9cOtV7mx8hrxbl7DPmtwBxcEzffV9V1IvKoiFzgPuxVIF5EsoE7gXvd564D3gfWA18At6hqNXAqcDUwrp6WiX8WkTUikg6MBe7w1vcWDGat2okqTBpkQ1qO1Q0ju3DR4A48+3UWX6zd7XQcY4yXnNPPdU/o8zX279yYw1SV309PJzOvmOcuH0TneGuZeKzCvHlxdzvDz+vse6jW4zLgkiOcOwWYUmffYuofj46qXn2iec1/fbZmFwM6xlkf0uMgIjwxqR+b8ku48/1VdE08lR5tYpyOZYxpZF0TW9CrXUs+X7OLG0Z2cTqOMT7hlYWb+TR9F7+fkMaYHja373jYACDzM9sKSknPLeS8fu0aPtjUy9XBZQjNI8K4+a0VFJdVOh3JGOMFh4e27DxgQ1uMWbQxn6lfbODcfu349ZhuTsfxW1acm5/5bI1rgtPZ/do6nMS/tWkZxQtXDGLrvlLu+TAd13QKY0wg+e/QFpsYaoJb7v5SfvPuj/RoE8OfL+5v89VOgBXn5mc+W7OTQZ3iSG7V3Okofu/krvHcMyGNz9fs5tXFOU7HMcY0si4J0fR2D20xJlhVVNVw639+pLpa+cdVQ4iO9Oqo6YBnxbn5H1v2lrB2RxHn2pCWRnPjqK5M6NOWP83ewA85toKoMYHm3P7tWLntADtsaIsJUk/O3sCq7Qf488X9ba5aI7Di3PyPw0NazrHivNGICH++pD8dWzXj1v+sZE9xmdORjDGN6PDNjNl299wEoS/W7ua1b3O47pQUzrbaoVFYcW7+x2fpuxjcKY72cc2cjhJQWkaF89JVQygqq+T2aauotgWKjAkYKQnR9Gwbw5fr85yOYkyT2lZQyu+mr2ZAciz3n9PL6TgBw4pz85OcvSWs31XEuf3bOx0lIPVq15JHL+jLkk0FvDTfFigyJpCM792G5Vv2sa+kwukoxjSJ8qpqbvnPSgT4+xWDiQizkrKx2N+k+cnste4uLX2tS4u3XDI0mQsGtOfZrzeyfIuNPzcmUIzv05YahbkZdvfcBIcpn2WwZkchT186kI6trYFEY7Li3Pxkzro8BiTH2pAWLxIRpkzqS4e4Ztw2bRUHSu0umzGBoE/7lrSPjbKhLSYofJq+kze/28qNo7pwZu82TscJOFacGwB2F5axevsBxvexu+beFhMVzvOXDyKvqMz6nxsTIESEM3u3YdHGfA5VVDsdxxiv2b6vlPs+XMOgTnH8fkJPp+MEJCvODQBfuT+KPauP/QbcFAZ0jOOeCT2Zsy6Pt7/f6nQcY0wjGN+nLWWVNSzamO90FGO8oqq6htvfWwXAc5cNIjzUykhvsL9VA8CX63bTNSGaboktnI4SNG4Y2YXT0hJ57LMMMnYVOR3HGHOChnVpTcuoMBvaYgLW3+dls2Lrfh6f1NfGmXuRFeeGwkOVfLepgPF92tpyu00oJER46pIBxDYL59b/rKS0osrpSMaYExAeGsK4nknMzcijqrrG6TjGNKoVW/fx3NyNTBrUgYkDOzgdJ6BZcW6Yt2EPVTXKeBvS0uQSWkTy118MZPPeEqZ8luF0HGPMCRrfpy37SytZsXW/01GMaTRFZZXcNm0VHVo149GJfZyOE/CsODd8uX43STGRDEyOczpKUDq1ewI3jurKO0u38c0G+zjcGH82KjWB8FDhm8w9TkcxptE8/PE6dhWW8ddfDCImKtzpOAHPivMgV1ZZzfzMfM7s3YaQEBvS4pS7xvegZ9sYfj99DQUHy52OY4w5TjFR4ZyU0pr5G2xSqAkMH6/awcwfd/Cbcd0Z0rmV03GCghXnQe67TQWUVlRbn1KHRYaF8tfLBlJ0qJL7Zqyx9orG+LGxaUlk5hWz48Ahp6MYc0K27yvlgZlrGdq5FbeO7e50nKBhxXmQm7shj+YRoQzvGu90lKDXs21LfndWGl+uz+OD5blOxzHGIyIyQUQyRSRbRO6t5/XOIjJXRNJFZL6IJLv3DxSR70Rknfu1XzR9eu8Y2zMRcM3nMcZf1dQod32wGoBnfzGQMGub2GTsbzqIqSrzNuQzsnsCUeGhTscxuNorjugazyOfrGNbQanTcYw5KhEJBV4AzgZ6A5eLSO86hz0FvKmq/YFHgT+595cC16hqH2AC8FcRCYiJL90SW9CxdTPm27hz48de+zaHH3L28dD5va1tYhOz4jyIHf7Y9fReSU5HMW4hIcJTlw4gJES44/1VVNfY8Bbj04YB2aq6WVUrgGnAxDrH9Abmuh/PO/y6qmap6kb3453AHiCxSVJ7mYgwNi2Jb7MLKKu01UKN/8neU8yf52RyRq8kLh6S7HScoGPFeRCbm+G6qzM2zYpzX9IhrhmPTezLiq37+ceCTU7HMeZoOgDbaz3Pde+rbTUw2f14EhAjIv8zjk5EhgERQMD8Dz82LYlDldUszdnndBRjjklVdQ13vb+a6IhQnrion61/4gArzoPYNxv20K9DLEkto5yOYuqYOLA95/Vvx7NfZbFuZ6HTcYw5kvp+atf9uOduYIyI/AiMAXYAP624JSLtgLeAX6rqz1buEZGbRGS5iCzPz/efDigjusUTGRZi486N33lp/iZW5xby+IX9SIqx+sAJVpwHqX0lFfy4bT/jetpdc18kIjw2sS9xzSO4+4N0KqpstUHjk3KBjrWeJwM7ax+gqjtV9SJVHQT8wb2vEEBEWgKfAQ+o6vf1fQFVfUVVh6rq0MRE/xn1EhUeyind4m3cufEr63YW8re5Gzl/QHvO7d/O6ThBy4rzILUgaw81ihXnPqxVdARPTOpLxq4iXpiX7XQcY+qzDEgVkS4iEgFcBsyqfYCIJIjI4Z819wGvufdHADNxTRb9oAkzN5mxPZPYUlDK5vyDTkcxpkHlVdXc+d5qWkVH8JitAuoorxbnHrTYihSR99yvLxWRlFqv3efenykiZ7n3dRSReSKS4W6/dVut41uLyFcistH9p3XKP4q5GXtIaBFJvw6xTkcxRzG+T1suHNieF+Zl2/AW43NUtQq4FZgDZADvq+o6EXlURC5wH3YakCkiWUAbYIp7/6XAaOA6EVnl3gY27XfgXYfn88zL9J/hOCZ4/fXrjWTmFTN1cj/imkc4HSeoea0497DF1g3AflXtDjwLTHWf2xvXHZjDLbZedF+vCrhLVXsBw4Fbal3zXmCuqqbi6gzws18GjEtldQ0LsvIZ1zPRVgX1A3+8oA+toiO46/3VNrzF+BxV/VxVe6hqN1Wd4t73kKrOcj+erqqp7mN+parl7v1vq2q4qg6sta1y8ntpbB1bN6d7Ugsb2mJ83oqt+3l5wSZ+MbQj43raooRO8+adc09abE0E3nA/ng6cLq5pwROBaaparqo5QDYwTFV3qepKAFUtxnWnpkM913oDuNBL35ffW7F1P8VlVfYP0E/ENY/giUn92LC7mL/b8BZj/MrYtESWbt5HSXlVwwcb44DSiiru/mA17WKb8cB5vZyOY/CwOBeRkSLyS/fjRBHp4sFpnrTY+ukY98ejhUC8J+e6h8AMApa6d7VR1V3ua+0C6h1M7a8z/xvTNxv2EB4qjExNcDqK8dCZvdtw0aAOvDgvm7U7bHiLMf5ibFoSFdU1fJu91+koxtTrL3Myydlbwl8u6U9MVLjTcQweFOci8jBwD66JPADhwNseXNuTFltHOuao54pIC+BD4HZVLfIgy38v4qcz/xvT3Iw8hneNp0VkmNNRzDF4+Pw+tI6O4O4PbHiLMf5iaEprWkSG2bhz45N+3Laf15ds4erhnTmlm92w8xWe3DmfBFwAlMBPK7nFeHBegy22ah8jImFALLDvaOeKSDiuwvwdVZ1R65g8d7/cw31zbZBfPbYWlLApv8S6tPih2Obh/Oki1/CW57/Z6HQcY4wHIsJCGNEtnkUb81G1FX+N76ioquHeD9fQtmUUv5+Q5nQcU4snxXmFut5RFEBEoj28doMtttzPr3U/vhj4xv21ZgGXubu5dAFSgR/c49FfBTJU9ZmjXOta4GMPcwaVb9wLYlhx7p9O79WGiwZ34MX5m1iTa8NbjPEHo1MTyN1/iK0FpU5HMeYnryzcRGZeMY9N7GvDWXyMJ8X5+yLyMhAnIjcCXwP/augkD1tsvQrEi0g2cCfuDiuqug54H1gPfAHcoqrVwKnA1cC4Wq23znFf60ngTBHZCJzpfm7qWJCVT9eEaDrHe/o7lvE1D5/Xh4QWNrzFGH8xKtU1hHLRRhvaYnzDpvyDPDc3m3P7teOM3tYcwtc0OOhYVZ8SkTOBIiANeEhVv/Lk4qr6OfB5nX0P1XpcBlxyhHOn8N9+uIf3Lab+8T/NYYsAACAASURBVOioagFwuie5glVZZTXfby7gspM6OR3FnIDY5uE8MakfN7yxnBfnZ3P7GT2cjmSMOYrO8c3p2LoZCzfu5eoRKU7HMUGupka5b8YaosJDePiCuh2ujS/wZELoVFX9SlV/p6p3q+pXIjK1KcKZxvVDzj7KKmsY0yM4J8IGktN7tWGie3GirLxip+MYY45CRBiVmsh3mwqorLZPu4yz3lu+nR9y9vGHc3uRFBPldBxTD0+GtZxZz76zGzuI8b6FWflEhIVwctfWTkcxjeCh83rTIjKM309Pp7rGJpoZ48tGpyZwsLyKVdsPOB3FBLE9RWU88XkGw7u25tKhHRs+wTjiiMW5iPxaRNYAaSKSXmvLAdKbLqJpLAuy8jm5S2uaR1gLxUAQ3yKSh8/vw6rtB3hjyRan4xg/JyJDRWSmiKx0v9evERF7r28kI7olECKwKMvGnRvnPDxrHeVVNfzpov64emwYX3S0O+f/Ac7H1QXl/FrbEFW9qgmymUa048AhNu45aENaAszEge05LS2Rv8zJZPs+6wRhTsg7wL+Bybje689z/2kaQWyzcAZ0jGPhRluMyDhjzrrdzF67m9tOT6VLgjWF8GVHLM5VtVBVt6jq5aq6FTiEq51iCxGxGYV+ZqH7bs1oK84DiogwZVI/QgTun7nG+iibE5GvqrNUNUdVtx7enA4VSEalJpKee4ADpRVORzFBpqiskoc+XkvPtjHcNLqr03FMAzyZEHq+uz1hDrAA2ALM9nIu08gWZuXTLjaK1KQWTkcxjaxDXDN+P6EnizbuZcbKHU7HMf7rYRH5l4hcLiIXHd6cDhVIRqcmUKOwZFOB01FMkPnzFxvILy7nycn9CQ/1ZLqhcZIn/4UeB4YDWaraBVe7wm+9mso0qsrqGhZv3MuYHok2xixAXT28M0M6t+Kxz9aTX1zudBzjn34JDAQm8N9hjOc5mijADOgYR0xkmPU7N01q+ZZ9vP39Nq47pQsDO8Y5Hcd4wJPivNLdQzxEREJUdR6uN3DjJ1ZtP0BxeZUNaQlgISHC1Mn9KC2v5o+frHM6jvFPA1R1qKpeq6q/dG/XOx0qkISHhjCiWzwLs/baEDTTJMqrqrnnw3Q6xDXjrvG2Joa/8KQ4PyAiLYCFwDsi8jegyruxTGNamJVPaIhwavcEp6MYL+qeFMNvxnXns/RdfLU+z+k4xv98LyK2IomXjeqRyI4Dh8jZW+J0FBMEXpy3iU35JTw+qS/RkdapzV94UpxPBEqBO4AvgE3YDH6/siArn0Ed44htFu50FONl/zemGz3bxvDAR2soKqt0Oo7xLyOBVSKSaa0UvWd0qusmySLr2mK8LCuvmBfnZzNxYHvGpiU5HcccgwaLc1UtUdUaVa1S1TeAF3CNSTR+YO/BctJzC62FYpCICAth6uT+rok/szc4Hcf4lwlAKjAea6XoNZ3jo+nUurmNOzdeVVOj3DdjDdGRYTx4nn0g5m+OtghRSxG5T0T+LiLjxeVWYDNwadNFNCdisfvujI03Dx4DOsZx/ald+M/SbXy/2bpCGI/pETbTyEalJvDdpgIqqmqcjmIC1DtLt7Ji634ePLc3CS0inY5jjtHR7py/BaQBa4BfAV8ClwATVXViE2QzjWBhVj6toyPo1yHW6SimCd05vgedWjfnvhlrKKusdjqO8Q+fAZ+6/5yL60aMtc31glGpiZRUVPPjtv1ORzEBaFfhIaZ+kcmo1AQuGtzB6TjmOBytOO+qqtep6svA5cBQ4DxVXdU00cyJqqlRFm7MZ1RqAiEh1kIxmDSPCGPKpL7k7C3h+W82Oh3H+AFV7aeq/d1/pgLDgMVO5wpEI7rFExoiNu7cNDpV5cGP1lFVU8OUC/tZ+2Q/dbTi/KfZZKpaDeSoarH3I5nGsn5XEXsPVjA61Ya0BKNRqYlMHpzMyws2k7GryOk4xs+o6krgJKdzBKLYZuEM7Bhn485No5u9djdfZ+Rx55k96BTf3Ok45jgdrTgfICJF7q0Y6H/4sYjYT3o/sCDL9cY/qoe1UAxWD5zbi9hm4dw7Yw3VNTZ82ByZiNxZa7tbRP4DWPXoJaNSE0jfUcj+kgqno5gAUVhayUMfr6Nvh5Zcf2oXp+OYE3DE4lxVQ1W1pXuLUdWwWo9bNmVIc3wWZOXTp31LkmKinI5iHNIqOoKHzu/N6u0HeGPJFqfjGN8WU2uLxDX23OYXecmo1ERU4dtNNrTFNI4/zc5gf2kFT17Un7BQTzplG19l//UCVFFZJSu37rcWioYLBrTntLREnvoyk9z9pU7HMb5rvao+4t6mqOo7WCtFrxmQHEtMVBjfZltxbk7cd5sKmLZsO78a2YW+1gDC71lxHqCWZBdQVaPWQtEgIjx+YV8AHvxorS0bbo7kPg/3mUYQFhrCiK7xLNq41/5NmhNSVlnN/TPX0Kl1c24/o4fTcUwjsOI8QC3cmE+LyDAGd2rldBTjA5JbNeeu8WnMy8znk/RdTscxPkREzhaR54EOIvJcre11oMrheAFtZGoCufsPsbXAPtEyx+/5bzaSs7eEKZP60iwi1Ok4phFYcR6AVJUFmfmc0i2eiDD7T2xcrjslhQHJsTwya51NQjO17QSWA2XAilrbLOAsB3MFvJHdXZP1F9nQFnOcMnYV8fKCzUwenMwo68wWMBqs3A53Z6mzbReRmSLStSlCmmOzKb+EHQcO2ZAW8z9CQ4QnJ/en8FAlUz7PcDqO8RGqulpV3wC6q+obtbYZqmqr5HhRl4RoOsQ1Y7G1VDTHobpGuffDdGKbhfPAub2cjmMakSe3VZ8Bfgd0AJKBu4F/AtOA17wXzRyvwy0UbTKoqatXu5bcNLor01fkstgWQDH/a5iIfCUiWSKyWURyRGSz06ECmYgwsnsCSzYVUFVd43Qc42feWLKF1bmFPHR+b1pFRzgdxzQiT4rzCar6sqoWq2qRqr4CnKOq7wE2oNkHLczKp2tiNB1b2wIE5ud+e3oqXRKiuX/mGg5VVDsdx/iOV3HdjBmJa/GhodgiRF43MjWB4rIq0ncUOh3F+JHc/aU89WUmp6UlcsGA9k7HMY3Mk+K8RkQuFZEQ93ZprdeOOsVcRCaISKaIZIvIvfW8Hiki77lfXyoiKbVeu8+9P1NEzqq1/zUR2SMia+tc648iskNEVrm3czz43gJOWWU1328usLvm5oiiwkN5YlI/tu0r5a9zs5yOY3xHoarOVtU9qlpweGvoJA/e5zuLyFwRSReR+SKSXOu1a0Vko3u7trG/IX9wqnvc+bf2SZbxkKrywEeuEujxC/siIg4nMo3Nk+L8SuBqYA+Q5358lYg0A2490kkiEgq8AJwN9AYuF5HedQ67Adivqt2BZ4Gp7nN7A5cBfYAJwIvu6wG87t5Xn2dVdaB7+9yD7y3gLM3ZR3lVjY03N0c1ols8vxjakX8tymGt3bEzLvNE5C8iMkJEBh/ejnaCh+/zTwFvqmp/4FHgT+5zWwMPAycDw4CHRSToPo1tHR1Bn/YtbVKo8dis1TuZn5nPXePTSG5ln5AHogaLc1XdrKrnq2qCqia6H2er6iFVXXyUU4cB2e7zK3CNUa+72txE4A334+nA6eL6FXAiME1Vy1U1B8h2Xw9VXQjsO6bvMogszMonIiyE4V3inY5ifNz95/SiVfMI7p2RbuNdDbiK5KHAE8DT7u2pBs7x5H2+NzDX/XherdfPAr5S1X3uiadfceQbLwFtZGoCP27bT0m5da40R7e/pIJHP1nPgORYrjslxek4xks86daSKCL3i8gr7iElr4mIJxNBOwDbaz3Pde+r9xhVrQIKgXgPz63Pre6PTl870h0YEblJRJaLyPL8/MCbIb8gK5+Tu7S2XqemQbHNw3nkgj6s3VHEv7/d4nQc4zBVHVvPNq6B0zx5r14NTHY/ngTEiIjH7/OB/p4NMKp7IpXVytKcBkcRmSD3+GcZFB6q5MnJ/QkNseEsgcqTYS0fA7HA18BntbaG1Pd/Td0x6kc6xpNz63oJ6AYMBHbhuuvz84uovqKqQ1V1aGJiYA39yN1fSvaegzbe3HjsnH5tOaNXEk9/lck2WwglqIlIGxF5VURmu5/3FpEbGjqtnn1136vvBsaIyI/AGGAHrsWNPHqfD+T37MOGprQiMiyERTbu3BzF4o17+XBlLjeN7kqvdi2djmO8yJPivLmq3qOq76vqh4c3D87LBTrWep6Ma7GLeo8RkTBcvwTs8/Dc/6Gqeaparao1uFo9DvMgY0BZmOV6Y7fi3HhKRHjswr6EhYTwh4/W2DLiwe11YA5wuPVDFnB7A+c0+F6tqjtV9SJVHQT8wb2v0JNzg0VUeCjDurS29qbmiA5VVHP/zDV0SYjmt6enOh3HeJknxfmnx9n5ZBmQKiJdRCQC1wTPWXWOmQUcnqF/MfCNuqqDWcBl7m4uXYBU4IejfTERaVfr6SRg7ZGODVQLs/JpHxtF96QWTkcxfqRdbDN+PyGNRRv3MvPHHU7HMc5JUNX3gRr4aahhQ702G3yfF5EEETn8s+Y+/rs+xhxgvIi0cg9DHO/eF5RGdk9g456D7C4sczqK8UF//TqLbftKeWJSP6LCbdhqoPOkOL8NV4F+yL06aLGIFDV0kvuN/VZcb7YZwPuquk5EHhWRC9yHvQrEi0g2cCdwr/vcdcD7wHrgC+AWVa0GEJF3ge+ANBHJrfWx659FZI2IpANjgTs8+hsIEJXVNXybvZcxaYnWVskcs6tO7szgTnE89ul6Cg6WOx3HOKPEPRZcAURkOK55QEfk4fv8aUCmiGQBbYAp7nP3AY/hKvCXAY+69wWln1oqWtcWU8faHYX8a3EOvxjakRHdrNlDMJBg/hh76NChunz5cqdjNIofcvZx6cvf8dKVgzm7X7uGTzCmjqy8Ys59bhHn9mvHXy8b5HQc0wARWaGqQxvxeoOB54G+uD55TAQuVtX0xvoaJyqQ3rPrqqlRTpryNaN7JPLsLwY6Hcf4iKrqGia+8C15ReXMvXMMsc3DnY5kToCn79thR7lAT1XdcKQ+t6q68kQCmsa1IGsPoSHCKe67L8Ycqx5tYvj1ad15bu5GJg1OtrkLQUZVV4rIGCAN12TNTFWtdDhW0Ahxv38vzt6LqtonoAaA177NYd3OIl64YrAV5kHkiMU5rmEmN1F/1xMFGmqxZZrQ/Mx8hnRqRWwz+8drjt8tY7vxWfpO7p+xhi/vGE105NHeIkwgcS8odA6Qgutnw3gRQVWfcTRYEBnVPYFPVu8kM6+Ynm2tG0ew21ZQyjNfZXFGryTO6dfW6TimCR1xzLmq3uT+83h635omtKe4jHU7ixiTZnc6zYmJDAvlycn92XHgEM98leV0HNO0PgGuw7XWREytzTSRkamuTz6ta4tRVe6fuYawkBAeu7CvfZISZDy6LSYip/DfuykAqOqbXspkjtGCTNfCHDYMwTSGk1Jac+XJnfj3tzlcMKA9AzrGOR3JNI1kVe3vdIhg1j6uGV0To1m0cS+/GtXV6TjGQTNW7mBx9l4endiHdrHNnI5jmpgnK4S+hWsJ55HASe6t0SYhmRM3PyufxJhI+rS3j0FN47jn7J4kxkRy74w1VFbXOB3HNI3ZIjLe6RDBbmT3BJbmFFBe1VAXSxOo9h4s57HP1jO4UxxXndzZ6TjGAZ60UhwKnKqq/09Vf+PefuvtYMYzVdU1LN64lzE9rIWiaTwto8J5dGJfMnYV8c9Fm52OY5rG98DMY22baxrXyO4JlFXWsHLrAaejGIc89ul6SsqreHJyf0JC7Od6MPKkOF8L2EwEH7U69wCFhyo5zcabm0Z2Vp+2TOjTlr99vZEte0ucjmO872lgBK5VoVuqaoyq2sdxTWx4t3hCQ4TF2flORzEOmJe5h49X7eTXp3WnRxub8hGsPCnOE4D1IjJHRGYd3rwdzHhmfmY+IQKjultxbhrfIxP7EBEWwn0z1hDMayIEiY3AWrX/0I5qGRXOwI5xNik0CJWUV/HAzLV0S4zmlrHdnI5jHOTJhNA/ejuEOX7zM/MZ3KmV9T81XtGmZRT3nd2L+2eu4YPluVx6UkenIxnv2QXMF5HZwE/LxForxaY3snsCz32zkQOlFcQ1j3A6jmkiT3+ZxY4Dh/jg5hFEhoU6Hcc46Kh3zt19bx9U1QV1tybKZ44iv7icNTsKrUuL8arLTurIsJTWTPk8g/zi8oZPMP4qB5gLRGCtFB01KjUBVViyqcDpKKaJrNy2n38vyeHKkztxUkprp+MYhx21OFfVaqBURGKbKI85BguzXGMST0tLcjiJCWQhIcKfJvfjUEU1j3yyzuk4xktU9RFVfQR4Bni61nPTxAZ0jKNFZBiLbGhLUKioquHeD9Np2zKKe8/u6XQc4wM8GXNeBqwRkVdF5LnDm7eDmYbNz8onoUWEtVA0XtctsQW/GdedT9N3MTcjz+k4xgtEpK+I/IirCcA6EVkhIn2czhWMwkNDGN61tU0KDRIvzs8mK+8gj1/Yl5goG6JqPCvOPwMeBBYCK2ptxkHVNcqijfmM7pForZZMk/i/Md1IaxPDAx+t5WB5ldNxTON7BbhTVTuramfgLuCfDmcKWiO7J7B93yG2FZQ6HcV4UVZeMS/My+b8Ae05vVcbp+MYH9Fgca6qb9S3NUU4c2Srcw9woLTShrSYJhMRFsKfJvdjd1EZT83JdDqOaXzRqjrv8BNVnQ9EOxcnuI1Mdc0lWmR3zwNWdY1yz4fpREeG8fD5vZ2OY3yIJyuEporIdBFZLyKbD29NEc4c2eEWiqNTE5yOYoLI4E6tuHZECm98t4UVW/c7Hcc0rs0i8qCIpLi3B3BNEjUO6JYYTbvYKGupGMDe/G4LP247wEPn9SahRaTTcYwP8WRYy7+Bl4AqYCzwJvCWN0OZhi3I3MPAjnHWZss0ubvPSqNdyyjum5FORVWN03FM47keSARmADPdj3/paKIgJiKM7J7Akk0FVNdY6/lAk7u/lL/MyWR0j0QmDergdBzjYzwpzpup6lxAVHWrqv4RGOfdWOZoCg6Wk76j0Ia0GEe0iAzjsQv7kpV3kJcXbHI6jmkkqrpfVX+rqoNVdZCq3qaq9vGIg0amJlB4qJI1OwqdjmIakapy/8y1ADwxqS8iNm/M/C9PFiEqE5EQYKOI3ArsAKwqdNDCjfmoYv3NjWNO79WG8/q34/lvsjm7Xzu6J7VwOpI5Tg2t+KyqFzRVFvO/Tu3uGra4eGM+AzvGOZzGNJaZP+5gYVY+fzy/N8mtmjsdx/ggT+6c3w40B34LDAGuAq71ZihzdPMz84mPjqBfB2s/b5zz8Pl9aBYRyv0z1lBjH7v7sxFAMrAIeAp4us5mHJLQIpJe7Vpav/MAsvdgOY9+up7BneK4ekSK03GMj/KkW8syVT0I7FfVX6rqZFX9vgmymXpU1ygLs6yFonFeYkwkfzi3Fz9s2ce0ZdudjmOOX1vgfqAv8DfgTGCvrQbtG0alJrBy235KK6x9aSB45JP1lJZXM3Vyf0LtZ7g5Ak+6tYwQkfVAhvv5ABF50evJTL1W5x5gf2klp6XZkBbjvEuGJHNKt3j+NDuDvKIyp+OY46Cq1ar6hapeCwwHsoH5IvIbh6MZXP3OK6uVpZv3OR3FnKC5GXl8snont4ztTmqbGKfjGB/mybCWvwJnAQUAqroaGO3NUObI5mbkERoiNt7c+AQR4YlJ/aioquHhj9c5HcccJxGJFJGLgLeBW4DncHVtMQ4b1qU1UeEhzM/c43QUcwKKyip54KO1pLWJ4dendXM6jvFxnhTnqGrdz6yrvZDFeGBuxh6Gdm5lLRSNz0hJiOb2M3rwxbrdzFm32+k45hiJyBvAEmAw8IiqnqSqj6nqDoejGSAqPJThXeNZkGWLEfmzKZ+6Pl18cnI/IsI8Kr1MEPPk/5DtInIKoCISISJ34x7iYprW9n2lbNhdzJm9bYlf41t+NaoLvdq15KGP11JUVul0HHNsrgZ6ALcBS0SkyL0Vi0iRw9kMcFqPRLYUlLJlb4nTUcxxmJ+5h/eWb+em0d0Y1KmV03GMH/CkOL8Z18ecHYBcYCDw/zy5uIhMEJFMEckWkXvreT1SRN5zv75URFJqvXafe3+miJxVa/9rIrJHRNbWuVZrEflKRDa6/wy4fwFzM/IAVxs7Y3xJeGgIUyf3I7+4nKmzNzgdxxwDVQ1R1Rj31rLWFqOqLZ3OZ/hpTQsb2uJ/Cg9Vcu+Ha0hNasHtZ6Q6Hcf4CU+6texV1StVtY2qJqnqVcA1DZ0nIqHAC8DZQG/gchHpXeewG3B1gekOPAtMdZ/bG7gM6ANMAF50Xw/gdfe+uu4F5qpqKjDX/TygfJ2xh66J0XRJiHY6ijE/0z85jutP7cI7S7exbItNXjOmsaQkRJMS39yGtvihxz9dT/7Bcp66ZABR4aENn2AMHo45r8edHhwzDMhW1c2qWgFMAybWOWYi8Ib78XTgdHEtlTURmKaq5aqag6t7wDAAVV0I1PeTv/a13gAuPIbvx+cVlVWyNKeAM+2uufFhd47vQXKrZvx+ejqHKmxqijGN5bS0JL7bXEBZpf278hfzNuzhgxW5/N/orgywRaTMMTje4tyT5pwdgNoTSXPd++o9RlWrgEIg3sNz62qjqrvc19rFEVYxFZGbRGS5iCzPz/efuxALs/KprFYb0mJ8WvOIMP5y8QBy9pYw9Qsb3mJMYxnTI5GyyhqW5tinUv6g8FAl985Ip0ebFtxmw1nMMTre4tyT5QDrK+DrnnekYzw597io6iuqOlRVhyYm+k87wrkZe2jVPJzBney3b+PbRnSL55enpvD6ki0sybaVDY1pDMO7xhMRFsKCTP+5qRTMHvt0PXsPVvDUJQOIDLPhLObYHLE4PzxTv56tGGjvwbVzgY61nicDO490jIiEAbG4hqx4cm5deSLSzn2tdkDAzJypqq5hXuYexqYlERZqLZiM7/v9WT3pmhDN76anU2zdW4w5Yc0iXC0V52cFzI+2gPXNhjymr8jl12O60T/ZbqiZY3fESq+emfu1Z/CHeXDtZUCqiHQRkQhcEzxn1TlmFnCt+/HFwDeqqu79l7m7uXQBUoEfGvh6ta91LfCxBxn9woqt+zlQWmlDWozfaBYRylOXDmBX4SEe/9Q6rwYyD7pydRKReSLyo4iki8g57v3hIvKGiKwRkQwRua/p0/uX03oksjm/hO37Sp2OYo6gsNTVnSWtTQy/Ob2703GMn/LabVj3GPJbgTm4+qK/r6rrRORREbnAfdirQLyIZOOaZHqv+9x1wPvAeuAL4BZVrQYQkXeB74A0EckVkRvc13oSOFNENgJnup8HhLkb9hAeKozukeB0FGM8NrhTK24e0433lm/nmw15TscxXuBhV64HcL3/D8J1k+ZF9/5LgEhV7QcMAf6vdjtd83OnpbmGYs63ri0+65FP11FQYsNZzInx5A74cVPVz4HP6+x7qNbjMlxv0PWdOwWYUs/+y49wfAFw+onk9VVfr89jeNd4YqLCnY5izDG57YxUvtmwh3s+XMOXt7eiVbStbBtgfurKBSAih7tyra91jAKH+6XH8t8higpEu4c0NgMqAFv06Ci6JETTsXUzFmTu4erhnZ2OY+r4ct1uZqzcwW/HdadfcqzTcYwfswHMPm5z/kE27y3hDBvSYvxQZFgoz1w6kAOlFTw0a53TcUzj86Sz1h+Bq0QkF9fNmt+4908HSoBdwDbgKVW1ViRHISKc1iOJJZsKKK+yloq+JL+4nPtmrKF3u5bcOs66s5gTY8W5j/vavSrouJ71doY0xuf1bt+S205P5ZPVO/k0vaF53cbPeNJZ63LgdVVNBs4B3hKREFx33atxNRjoAtwlIl1/9gX8tP2tt5yWlkhpRTXLcvY7HcW4qSr3fJhOcXkVf7tsIBFhVlqZE2P/B/m42Wt306d9Szq2bu50FGOO281jujEgOZYHP1rLnuIyp+OYxuNJZ60bcM0hQlW/A6KABOAK4AtVrVTVPcC3wNC6X8Bf2996y4hu8USEhjAv07q2+Ip3f9jONxv2cO+EnqS2iXE6jgkAVpz7sF2Fh/hx2wHO7tvW6SjGnJCw0BCevnQgpRXV3PvhGlxNmUwA8KQr1zbc84FEpBeu4jzfvX+cuEQDwwFbuaoBzSPCGNEtnm82WHHuC3L2lvDYp+sZ2T2B605JcTqOCRBWnPuwOWt3AzChbzuHkxhz4ronteCeCT35ZsMe3v5+q9NxTCPwsCvXXcCNIrIaeBe4zt0y9wWgBbAWV5H/b1VNb/Jvwg+d0SuJnL0lbMo/6HSUoFZVXcMd760iIiyEpy4ZQEiIJ4unG9Mwr3ZrMSdm9trdpCa1oHtSC6ejGNMofnlqCguy8nn8swxO7hpPD/sI2O950JVrPXBqPecd5AjduszRjevVhgc/XsfcjDy6JdrPB6e8MG8Tq7Yf4PnLB9E2NsrpOCaA2J1zH7X3YDnLtuyzIS0moIgIT10ygBaRYfz23R8pq7SOE8Ycqw5xzejVriVfr7ehLU5Ztf0Az32zkQsHtuf8AZ4smm6M56w491FfrsujRm1Iiwk8iTGR/OWS/mzYXcyfv8h0Oo4xfunMXkks37qP/SUVTkcJOiXlVdzx3iraxETyyMS+TscxAciKcx81e+0uOsc3p1c7+9jfBJ5xPdtwzYjOvPZtDvOt64Qxx+z0Xm2oUZifZf9+mtpDH69ja0EJz/xiILHNbHFA0/isOPdBhaWVfLepgAl92yJiE0xMYLr/nF70aNOCuz9IZ+/BcqfjGONX+nWIJTEmkq8zrDhvSjNW5vLhylxuHZfK8K7xTscxAcqKcx80Z/1uqmqUs21IiwlgUeGh/O2yQRSVVXLP9HRrr2jMMQgJEU7vmcSCzHwqqmqcjhMUNucf5IGP1jIspTW/Hdfd6TgmgFlx7oM+Wb2TTq2bMyA51ukoB0UuzAAAHppJREFUxnhVr3YtuXdCT+Zu2MMbS7Y4HccYv3JGrzYcLK/ih5x9Tkf5/+3deXhU5d3/8fc3G2ENEsIeJEAQAsoisqt1R1xQK4pal6JFrVtt1Wqtffx1ta3V2mpr3QUFFMXH+FPBtVpRgRAIEEgACUKAJGxCWJKQ5H7+mEMb0yQEyMw5ST6v65orM2fO8pkzM3e+c5b7NHml5RXcNnMJcTFR/HnyEGKiVT5J+OjTFTBbi0uZv3YbFw7upkNapFn4/thenN6/E795ZxXL8r/xO45IozG2b0daxkbz3soCv6M0eb9/N5fszbv546WD6da+pd9xpIlTcR4wby/bTKWDC4eoayZpHsyMP00aTFKbFvzw5Ux27TvgdySRRqFlXDTfOS6JuSsKqKzUYWHh8sHKQp6bn8d1Y3pxVlpnv+NIM6DiPGDSszbTv0tbXZxFmpVjWsfx+FXDKNhVwl2vZen4c5F6Gj+oC0XFpSzZqL1O4bBl137ufi2Lgd3acd+E/n7HkWZCxXmAbNyxj8wN32iruTRLw3oew73n9uf9lYU8+1me33FEGoXT+nciNtqYu2KL31GanAMVldwxcyml5ZX89YqhtIiJ9juSNBMqzgMkPWszABecoOJcmqfrx6VwVlpnHno3h8wNO/2OIxJ47eJjGde3I3OzC7THqYH9YW4OC9fv4LcXH0/vpDZ+x5FmRMV5gLyVtZlhPduT3KGV31FEfGFmPHzpYLokxHPbjCV8s09XPxQ5lPGDurBxx36yN+/2O0qT8e7yLTz9rzyuGX0sFw3t7nccaWZUnAdETsFucgqKuXCwtppL85bQKpYnrhxGUXEJP3k1Sye6iRzCmQM6E2UwL1u9tjSEdVv3cPdryxiS3J77zxvgdxxphlScB8TsjHxio40Lh+gXusjg5Pb8/Lw0Pswp4omP1/odRyTQEtu0YGRKInNXqDg/WvvKyrn5pUxio40nrhqm48zFFyrOA6CsvJI3lmzirLTOdGgd53cckUC4ZvSxXDSkG498sJqPcgr9jiMSaOMHdWFN0R7WFhX7HaXRcs5x/xsrWF1UzGOTh9Jd/ZmLT1ScB8BHOYXs2FvGpBOT/Y4iEhhmxu8uOYG0ru24Y+ZS1m3d43ckkcAaP6gLZvBWlnptOVLPfpbHG0s2ceeZ/TilX5LfcaQZU3EeAK9m5NO5XQtOTu3odxSRQGkZF80/rj6R2Jgopk5fTHGJLlAkUpPO7eIZ3TuR9KzN6rXlCHy6eiu/fWcV4wd24dbT+vodR5o5Fec+K9xdwj9zi/jusB7EROvtEKmuxzGtePzKoeRt26sTREXqcOHgbuRt28vyTbv8jtKorN+2l1tnZNKvc1v+dNlgoqLM70jSzIW1GjSz8WaWa2ZrzezeGp5vYWaveM8vMLNeVZ67zxuea2bnHGqeZvaCmeWZ2VLvNiScr62hzMncRKWDScN1SItIbcb06ch95/bnvZWFPPbhGr/jiATSuYO6EhttpC/d7HeURqO45AA3TMsgOsp4+prhtG4R43ckkfAV52YWDTwBnAukAVeYWVq10a4Hdjrn+gKPAr/3pk0DJgMDgfHA38wsuh7zvNs5N8S7LQ3Xa2sozjlmZ2xkRK8OpHRs7XcckUC7flwKl57Yg8c+XMObSzf5HUckcBJaxXJqv068tWwzFdrDdEiVlY47X1lK3ra9PHHVMF1jRAIjnFvORwBrnXPrnHNlwCxgYrVxJgIvevdfA84wM/OGz3LOlTrn8oC13vzqM89GY/7a7azbtpfLT9JWc5FDMTN+e/HxjEjpwN2vLWPx1zv8jiQSOBOHdKNwdykL8/T9OJSH5ubwwaoifnF+GmP66JwvCY5wFufdgY1VHud7w2ocxzlXDuwCEuuY9lDz/I2ZLTOzR82sRU2hzGyqmWWYWcbWrVsP/1U1oBc+zyOxdRznD+7qaw6RxiIuJoonv3ciXRPimTptMRt37PM7kkignDmgM63ioknP0t6lukz/8mue+nQd144+lmtGH+t3HJFvCWdxXtMZFdX3s9U2zuEOB7gP6A+cBHQAflpTKOfcU8654c654UlJ/nWVtGH7Pj7MKeLKkT11kQORw9ChdRzPXXcSByoqmfLCInarBxeRf2sZF83ZaZ15Z3kBpeUVfscJpI9zivifN1dwRv9O/OKCgYR22IsERziL83yg6vEaPYDqZ6n8exwziwESgB11TFvrPJ1zW1xIKfA8oUNgAmvaF+uJNuOqkfrFLnK4+iS14cnvnUjetr3c8nImZeWVfkcSCYyLhnZn1/4DfLiqyO8ogbNi0y5umZFJWrd2/OWKoUSrZxYJoHAW54uAVDNLMbM4Qid4plcbJx241rt/KfCRC3XQmg5M9npzSQFSgYV1zdPMunp/DbgIWBHG13ZU9paW80rGRs49vitdEuL9jiPSKI3p25HfXXI8/1qzjbtfUxeLIgednJpE14R4Zi3aeOiRm5HN3+zn+hcX0b5lLM9de5J6ZpHACtsn0zlXbma3AvOAaOA551y2mf0SyHDOpQPPAtPNbC2hLeaTvWmzzexVYCVQDtzinKsAqGme3iJfNrMkQoe+LAVuCtdrO1pzlmyiuKSc68b08juKSKM2aXgyW/eU8oe5uSS2bsED5w/QLmpp9qKjjEkn9uCvH69l0zf7dRl6YMfeMq55biH7SiuYffNoOrXThjEJrrD+bHTOvQO8U23YL6rcLwEm1TLtb4Df1Gee3vDTjzZvJFRUOp6fn8fx3RMY1rO933FEGr2bT+3D1uJSnpufR6d2Lbjp1D5+RxLx3aThyfzlo7W8lpHPHWem+h3HV3tKy7nu+YVs3LGPF6eMoH+Xdn5HEqmTLkkZYW8v38K6rXu58dTe2sIn0gDMjAfOS+PCwd146N0cZmdoV75IcodWjO2byOzFG5v1IV8lByqYOi2D7M27eeLKYYzqneh3JJFDUnEeQZWVjr9+uIbUTm2YMEjdJ4o0lKgo4+FJgxnXtyP3zlnO28u2+B1JxHeXDU8mf+d+Pv9qu99RfFFeUcntM5fw+VfbeXjSCZyZ1tnvSCL1ouI8gt5dUcCaoj3cdkYqUTpDXKRBxcVE8Y+rT2RocnvumLWEedkFfkcS8dU5A7uQ0DKWV5rh3qSKSsc9ry/jvZWFPHhBGhcP7eF3JJF6U3EeIZWVjr98uIY+Sa0573htNRcJh9YtYnj++ycxqHsCt87I5KOcQr8jifgmPjaai4d2Z96KAoqKS/yOEzEVlY57XlvGnMxN/Pisflw3NsXvSCKHRcV5hMzLLiC3sJjbTk9Vv6oiYdQ2PpYXp4xgQNd23DQ9k09W+3sl4KbOzMabWa6ZrTWze2t4vqeZfWxmS7wrOE+o8twJZvaFmWWb2XIzUxcaDeya0cdSVlHJy19u8DtKRFRUOu5+LYvXM/O588x+3H5G8z4ZVhonFecRUF5RySPvr6Z3x9ZcMLib33FEmryElrFMmzKCvp3aMHVaBv/M1cVYwsHMooEngHOBNOAKM0urNtrPgVedc0MJdZf7N2/aGOAl4Cbn3EDgO4Au99rAeie14Yz+nXjpy68pOdC0rxhaUem4e3YWczI38ZOz+jX7Xmqk8VJxHgGzFm1kTdEe7hnfX1vNRSKkfas4XrphJH2S2vCDaRk6STQ8RgBrnXPrnHNlwCxgYrVxHHCw77oE/nOl6LOBZc65LADn3PaD17OQhjVlXArb95aRvrT6RbqbjvKKSu6ancWcJZu46+x+3KYt5tKIqTgPs137D/Do+6sZmdKBcwbqTHGRSOrQOo6ZU0cxuEd7bpuZyau6YmJD6w5UXan53rCqHgS+Z2b5hK5RcZs3vB/gzGyemWWa2T01LcDMpppZhpllbN2qQ5SOxJg+ifTv0pbn5ucRugh307K/rIIbpy/mjSWbuPuc47j1dBXm0ripOA+zP8zNYee+Mh44P039mov4IKFlLNOvH8m41CTueX0Zz/xrnd+RmpKaGrXq1d8VwAvOuR7ABEJXhY4idBG8ccBV3t+LzeyM/5qZc08554Y754YnJSU1bPpmwsyYMi6FnILiJtet4jf7yvjeswv4KLeIX100iFtO6+t3JJGjpuI8jBZ/vZMZCzfw/bEpDOqe4HcckWarZVw0z1wznAnHd+HXb6/i93NzmvWFWRpQPpBc5XEP/nPYykHXA68COOe+AOKBjt60nzjntjnn9hHaqj4s7ImbqQsHd6NjmziebkI/Trfs2s+kJ79gef4unrhyGFePOtbvSCINQsV5mOwrK+fu2Vl0bRfPj8/q53cckWYvLiaKv14xjCtH9uTv//yKW2Zksr9MhzgfpUVAqpmlmFkcoRM+06uNswE4A8DMBhAqzrcC84ATzKyVd3LoqcDKiCVvZuJjo/n+2BT+mbuVzA07/Y5z1HILivnu3z6nYFcJL04ZwQR1USxNiIrzMPn126vI276Xhy8bTOsWMX7HEREgOsr4zUWD+Pl5A5ibXcDlT31B0e7m0/9zQ3POlQO3Eiq0VxHqlSXbzH5pZhd6o/0E+IGZZQEzgetcyE7gEUIF/lIg0zn3duRfRfNx3ZheJLaO45H3Vvsd5ajMyy7g4r/Np7zSMevGUYzuk+h3JJEGpaoxDOZk5jNjwQZuPKU3Y/p09DuOiFRhZtxwcm+OTWzNHbOWMPGJ+Txz7XAGdtOhZ0fCOfcOoUNSqg77RZX7K4GxtUz7EqHuFCUCWreI4ebv9OHXb6/ii6+2N7qitrLS8deP1vLoB6sZnNyep64+kc7t1DW+ND3act7AlmzYyb1zljO6dyJ3nXOc33FEpBZnpXVm9k2jcQ6++/fPebUZXuJcmp/vjTqWLu3i+dN7uY2q55a9peXcMiOTRz9YzSVDu/PK1FEqzKXJUnHegHIKdnPd84vo0i6ex68cSmy0Vq9IkA3slkD6bWMZmnwM97y2jJ+8msW+snK/Y4mETXxsNLee3peMr3c2mqvn5hYUc9ET85mXXcDPzxvAny4bTHxstN+xRMJG1WMDWfz1Dq546ktaxkbz8g0jSWzTwu9IIlIPndrG89INI7n9jFTmLMln4uPzWVNY7HcskbC5bHgyPTu04nfv5HCgotLvOLVyzjFjwQYufPwzdu47wPTrR3LDyb3VLbE0eSrOj1JlpePFz9dz5dMLSGgZy6ypo0ju0MrvWCJyGKKjjB+f1Y/pU0ayc18ZFzz+GS/Mz1N3i9IkxcVEcf95A8gtLGbaF1/7HadGRcUlTJ2+mJ+9sZwRKR14946TGdtX53BJ86Di/AiVlVfy/spCLv775/xPejajeicy54dj6dWxtd/RROQIjUvtyDu3n8yo3ok8+NZKrnj6SzZs3+d3LJEGd3ZaZ07tl8Sj769m8zf7/Y7zb8450rM2c/ajn/LJ6q3cP2EAL35/BElttTdamg9rTCeENLThw4e7jIyMeo9fWemYOn0xRcUlfFW0h71lFXRv35I7zkxl0ok9tKtNpIlwzjE7I59f/f+VHKis5LbTU/nByb2JiwnO9gwzW+ycG+53jkg63DZb6rZh+z7O+fOnDO91DNOmjPD9f9jX2/fyYHo2H+duZUhyex6eNJi+ndr4mkmkIdW33VZXiochKsrYtb+M9q3iuGRYD07tl8SpxyXpxE+RJsbMuOykZMalduT/vZXNH+fl8npmPg9eMJBT+ukS8tI09Exsxc8m9OeBN7N5acEG366wua+snCc/WceTn3xFXHQUPz9vANeN6UWM/rdKM6Xi/DDNvmmM3xFEJEK6tW/JP64ezse5RTyYns01zy1kXN+O/HR8f47voX7RpfG7auSxfLCqiF+9tZLBPRI4oUf7iC37QEUlsxZt5C8frmFrcSkTh3TjZxMGqItEafb0s1RE5BBOO64T7915Cg+cn0b25l1c8Phn3DR9MVkbv/E7mshRiYoy/nz5EJLatuCm6Ysp2BX+K+aWHKhgxoINnPnIJzzwvytISWzN6zeP4bHJQ1WYi6BjznX8oogcluKSAzz9rzxemJ/H7pJyRvdOZMq4FE47Limiu+F1zLk0pBWbdnH5P76gxzGteOXGUbRvFdfgyygqLmF2Rj7Pz1/Ptj2lnNAjgTvOSOX0/p18P95dJBLq226rOFdDLyJHYE9pOTMXbOCZz9ZRuLuUzu1acNnwZCYO6UbfTm3DvnwV59LQPl+7jeueX0Svjq14ccoIuia0POp5lpZX8NmabczOyOeDVYWUVzpOTu3Izaf2YXSfRBXl0qwEojg3s/HAY0A08Ixz7qFqz7cApgEnAtuBy51z673n7gOuByqA251z8+qap5mlALOADkAmcLVzrqyufGroReRoHaio5KOcImYu3MAnq7fiHKR2asO5g7owLjWJIcntw9LLi4pzCYfPv9rG1GmLiY+N5o+TTuC04zod9jwKd5fwxVfb+SiniI9yithTWk6H1nFcemIPLj8pmT5J6oFFmiffi3MziwZWA2cB+cAi4Arn3Moq4/wQOME5d5OZTQYuds5dbmZpwExgBNAN+ADo501W4zzN7FVgjnNulpk9CWQ55/5eV0Y19CLSkAp2lTB3xRbeXVHAovU7qHQQHxvFsJ7HMLBbO/p3aUe/zm3p2j6eDq3iiIo68q2GKs4lXHILirl95hJyC4s57bgkpoxLYWRK4n/9yNxbWk7B7hLyd+4nZ8tucgqKydr4Deu27QXgmFaxnJ3WhXOP78KYPh0D1RWpiB+CUJyPBh50zp3jPb4PwDn3uyrjzPPG+cLMYoACIAm4t+q4B8fzJvuveQIPAVuBLs658urLro0aehEJl2/2lbEgbwdffLWdxV/vJLewmLLy/1wqPTbaSGzdgjbxMTx62ZDD7v1FxbmEU8mBCp79LI+nPl3Hrv0HiIuJolPbFphByYFK9pdVsKe0/FvTdE2IJ61rO0b27sDo3h1J69aO6KP4ASrS1AShn/PuwMYqj/OBkbWN4xXVu4BEb/iX1abt7t2vaZ6JwDfOufIaxv8WM5sKTAXo2bPn4b0iEZF6at8qjnMGduGcgV0AKK+oZP32vawt2kvh7hI279rPzr1l7Cktp028erWVYImPjeaW0/oyZWwKn63dRsb6HRTuLsHMiI+NIj42mqS2LeiW0JKuCfH069yWY1o3/EmkIs1ROP8j1PRzufpm+trGqW14TfvE6hr/vwc69xTwFIS2wtQ0johIQ4uJjqJvp7YROVlUpKG0jIvmrLTOnJXW2e8oIs1GOA8AyweSqzzuAWyubRzvsJYEYEcd09Y2fBvQ3ptHbcsSEREREQm0cBbni4BUM0sxszhgMpBebZx04Frv/qXARy50EHw6MNnMWni9sKQCC2ubpzfNx9488Ob5Zhhfm4iIiIhIgwvbYS3eMeS3AvMIdXv4nHMu28x+CWQ459KBZ4HpZraW0Bbzyd602V7vKyuBcuAW51wFQE3z9Bb5U2CWmf0aWOLNW0RERESk0dBFiHTmv4g0QuqtRUSkcalvu61OR0VEREREAkLFuYiIiIhIQKg4FxEREREJCBXnIiIiIiIB0axPCDWzrcDXRzBpR0J9qwdBkLKA8tQlSFlAeeoSpCxQc55jnXNJfoTxSxNpsyFYeYKUBZSnLkHKAspTl9qy1KvdbtbF+ZEys4yg9JIQpCygPHUJUhZQnroEKQsEL09jE7T1F6Q8QcoCylOXIGUB5anL0WbRYS0iIiIiIgGh4lxEREREJCBUnB+Zp/wOUEWQsoDy1CVIWUB56hKkLBC8PI1N0NZfkPIEKQsoT12ClAWUpy5HlUXHnIuIiIiIBIS2nIuIiIiIBISKcxERERGRgFBxfhjMbLyZ5ZrZWjO714flJ5vZx2a2ysyyzewOb/iDZrbJzJZ6twkRzLTezJZ7y83whnUws/fNbI3395gI5Diuyutfama7zexHkVw3ZvacmRWZ2Yoqw2pcFxbyF++ztMzMhkUozx/NLMdb5htm1t4b3svM9ldZT09GIEut742Z3eetm1wzO6chs9SR55UqWdab2VJveLjXTW3fa98+O02Jn+222uw6c6jNrl8eX9rsOvL40m4Hqc32lhHedts5p1s9bkA08BXQG4gDsoC0CGfoCgzz7rcFVgNpwIPAXT6tl/VAx2rD/gDc692/F/i9D+9VAXBsJNcNcAowDFhxqHUBTADeBQwYBSyIUJ6zgRjv/u+r5OlVdbwIZanxvfE+01lACyDF+95FhztPtef/BPwiQuumtu+1b5+dpnLzu91Wm31Y75Pa7AC12XXk8aXdDlKb7S0jrO22tpzX3whgrXNunXOuDJgFTIxkAOfcFudcpne/GFgFdI9khnqaCLzo3X8RuCjCyz8D+Mo5dyRXEjxizrlPgR3VBte2LiYC01zIl0B7M+sa7jzOufecc+Xewy+BHg25zMPJUoeJwCznXKlzLg9YS+j7F5E8ZmbAZcDMhlxmHVlq+1779tlpQnxtt9Vm15va7Fry+NVm15anDmFtt4PUZnt5wtpuqzivv+7AxiqP8/GxkTWzXsBQYIE36FZvV8lzkdglWYUD3jOzxWY21RvW2Tm3BUIfYKBTBPMATObbX1K/1g3Uvi6C8HmaQuiX/EEpZrbEzD4xs5MjlKGm98bvdXMyUOicW1NlWETWTbXvdZA/O41FYNaV2uw6qc2unyC02RC8dtu3NhvC026rOK8/q2GYL/1Qmlkb4HXgR8653cDfgT7AEGALod07kTLWOTcMOBe4xcxOieCy/4uZxQEXArO9QX6um7r4+nkys/uBcuBlb9AWoKdzbijwY2CGmbULc4za3hu/v2tX8O1CISLrpobvda2j1jBMfeLWLBDrSm127dRm13PhwWizIZjtti9tNoSv3VZxXn/5QHKVxz2AzZEOYWaxhD4ILzvn5gA45wqdcxXOuUrgaRr4EIC6OOc2e3+LgDe8ZRce3F3j/S2KVB5C/3AynXOFXi7f1o2ntnXh2+fJzK4Fzgeuct7BcN6uyO3e/cWEjhfsF84cdbw3fq6bGOAS4JUqOcO+bmr6XhPAz04j5Pu6Upt9SGqzDyEobba3rEC123612d6yw9Zuqzivv0VAqpmleL/0JwPpkQzgHVf1LLDKOfdIleFVj1u6GFhRfdow5WltZm0P3id04soKQuvlWm+0a4E3I5HH861f0H6tmypqWxfpwDXeGdyjgF0Hd4WFk5mNB34KXOic21dleJKZRXv3ewOpwLowZ6ntvUkHJptZCzNL8bIsDGeWKs4Ecpxz+VVyhnXd1Pa9JmCfnUbK13ZbbXa9qM2uQ5DabG9ZQWu3I95me/MNb7vtwng2a1O7ETrbdjWhX2H3+7D8cYR2gywDlnq3CcB0YLk3PB3oGqE8vQmdnZ0FZB9cJ0Ai8CGwxvvbIUJ5WgHbgYQqwyK2bgj9g9kCHCD0K/n62tYFoV1cT3ifpeXA8AjlWUvouLeDn58nvXG/672HWUAmcEEEstT63gD3e+smFzg3EuvGG/4CcFO1ccO9bmr7Xvv22WlKNz/bbbXZh8yjNvvQeXxps+vI40u7HaQ221tGWNtt8yYSERERERGf6bAWEREREZGAUHEuIiIiIhIQKs5FRERERAJCxbmIiIiISECoOBcRERERCQgV59Ksmdke728vM7uygef9s2qPP2/I+YuINEdqt6WpU3EuEtILOKxG/uBFDurwrUbeOTfmMDOJiEjteqF2W5ogFeciIQ8BJ5vZUjO708yizeyPZrbIzJaZ2Y0AZvYdM/vYzGYQupAAZva/ZrbYzLLNbKo37CGgpTe/l71hB7f2mDfvFWa23MwurzLvf5rZa2aWY2Yve1chw8weMrOVXpaHI752RESCR+22NEkxfgcQCYh7gbucc+cDeI31LufcSWbWAphvZu95444ABjnn8rzHU5xzO8ysJbDIzF53zt1rZrc654bUsKxLgCHAYKCjN82n3nNDgYHAZmA+MNbMVhK6THJ/55wzs/YN/upFRBoftdvSJGnLuUjNzgauMbOlwAJCl+RN9Z5bWKWBB7jdzLKAL4HkKuPVZhww0zlX4ZwrBD4BTqoy73znXCWhywH3AnYDJcAzZnYJsO+oX52ISNOjdluaBBXnIjUz4Dbn3BDvluKcO7gFZu+/RzL7DnAmMNo5NxhYAsTXY961Ka1yvwKIcc6VE9rq8zpwETD3sF6JiEjzoHZbmgQV5yIhxUDbKo/nATebWSyAmfUzs9Y1TJcA7HTO7TOz/sCoKs8dODh9NZ8Cl3vHRyYBpwALawtmZm2ABOfcO8CPCO1aFRFp7tRuS5OkY85FQpYB5d5uzheAxwjtmsz0Tu7ZSmjrR3VzgZvMbBmQS2gX6UFPAcvMLNM5d1WV4W8Ao4EswAH3OOcKvH8SNWkLvGlm8YS23tx5ZC9RRKRJUbstTZI55/zOICIiIiIi6LAWEREREZHAUHEuIiIiIhIQKs5FRERERAJCxbmIiIiISECoOBcRERERCQgV5yIiIiIiAaHiXEREREQkIP4PwPcm25Ssz6YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, Resnet())\n",
    "learn.fit_one_cycle(1)\n",
    "learn.recorder.plot_lr(show_moms=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason behind using this approach is we have to keep the total regularization in check. So if I use higher learning rates which provide regularization on their own, I do not need high values of momentum (as high momentum value would also provide large regularization values) and this would ensure that this results in convergence for a longer range of learning rate and faster convergence. The optimal learning rate is dependent on the momentum and momentum is dependent on the learning rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some good values of momentum to test\n",
    "This is a difficult question to answer. Genrally the default of (0.95 - 0.85) works good, but there are some values that you can test\n",
    "* 0.99 - 0.90\n",
    "* 0.95 - 0.85\n",
    "* 0.99\n",
    "* 0.9\n",
    "* 0.85\n",
    "\n",
    "**Note**:- In practice, we would choose this value in combination with the value of weight decay. But for quick demonstration I show how to choose value of momentum only.\n",
    "\n",
    "I test for the first two cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.074203</td>\n",
       "      <td>4.473627</td>\n",
       "      <td>0.058200</td>\n",
       "      <td>0.209500</td>\n",
       "      <td>01:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.603846</td>\n",
       "      <td>3.422370</td>\n",
       "      <td>0.168400</td>\n",
       "      <td>0.436200</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.306487</td>\n",
       "      <td>3.084147</td>\n",
       "      <td>0.243600</td>\n",
       "      <td>0.533900</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXBcx2Hn8W/P4BgcgwEwuA8SAO8LBEkQpG5KlhSZlig7lr3yFUdWTFt2yrI3qdipVMXObhJvUq5k15usZPmIY69kW9Zh2VpZNnVSskRSIMUDJEiBB0CAIO77Pqb3jzekQBIkQQnHm+HvU/VqZt57M+iWhj80+nX3M9ZaRETEvTxzXQAREbk0BbWIiMspqEVEXE5BLSLicgpqERGXi5mJD/UmBmx2XgG5gQSMmYmfICISXXbv3t1mrc2c7NiMBHVKZi5x9/wz/mAid63O48+uLyHZF4PXo9QWEZmMMabuosdmYhx1eXm5/fZP/h//9vJR9tV3nd1fFExkdNzi9RjmpSeyONvP4uxkFmYlMy+YSGZyPEZNcBG5Chljdltryyc7NiMtaoDblmdz2/Js9tV38frRNobHQlSf7iEuxoPXGGrb+3lsVx1Do6Gz7ylIS+CmxZmsyAuwPC+FJdl+EuK8M1VEEZGIMGNBfcbqwlRWF6ZOeiwUsjR0DnK0tZfatgFeP9rGr/c18ujOkwB4DJRkJrMqP0BFcToVxemUZCSp1S0iV5UZ6/qorKx8T++11gnvQ6d7ONTYw8HGHvbWd9HWNwxARnKcE9pF6VQUB1mS41fft0iEGx0dpaGhgaGhobkuyozz+XwUFBQQGxt7zv456fp4r4wxFKYnUpieyB+tyAGc8D7R1s+uEx3sOtHBzhMdPHegCQC/L4b1RU5re31RGgsyk0lNjJvLKojIFWpoaMDv91NUVBTVfzFba2lvb6ehoYHi4uIpv891QT0ZYwwlmcmUZCZzb8U8ABo6B3ir9t3gfulwy9nzV+ancO2CDK5fmEFFcTq+WPVzi7jZ0NBQ1Ic0OFkWDAZpbW29ovdFRFBPpiAtkYK0RD6ypgCA1t5h9tZ3caSph+3vtPHjP9TyyPbjxMd4WDMvlQ3FQTYtyaS0IFVdJSIuFO0hfcZ7qWfEBvX5Mv3xZ0ea/PktixgcGWfniXZeq2lj14kO/vdLNfyvF2tIT4rjj1Zk87HyQkrzA8R4NTlTRNwtaoL6fAlxXjYtyWLTkiwAOvtH2F7TysuHW3hqzyl+tqsevy+Ga0qCXL8og2sXZLAg8yofUTI6BN448OiXl1xdurq6eOyxx/jSl750Re/bvHkzjz32GKmpk49smy6uG/UxGzr7R/jDsTb+cLSN12raaOgcBCAjOZ7Nq3K4dVn21dm3/fI/QtVTsOELsPoTEJ881yWSq0R1dTXLli2bs59fW1vLnXfeSVVV1Tn7x8fH8XqnPwcmq29EjfqYDWlJcdxZmsedpXkA1LX388axdl6vaeMXb9XzkzfriIvxsL4ojWtKgizO9nPdwgyS4qP8P1dOKdRsg+f+El7877D2M1CxFdLmz3XJRGbUN77xDY4dO0ZZWRmxsbEkJyeTm5vL3r17OXToEB/+8Iepr69naGiIBx98kK1btwJQVFREZWUlfX19fPCDH+T666/njTfeID8/n2eeeYaEhIRpKd9V2aK+lIGRMXae6OD1GqfFfbipF4D4GA83LMqgvCid0vwAa+enRWeL21poeAt2PASHngEsLNkMGx+A+dehVbZkJkxsYf7dbw5yqLFnWj9/eV4K37xrxUWPT2xRv/LKK3zoQx+iqqrq7BC6jo4O0tPTGRwcZP369bz66qsEg8FzgnrhwoVUVlZSVlbGxz/+cbZs2cKnP/3py9b3DLWor0BiXAw3L8ni5nDfds/QKAdP9fC7g028dLiFF6qdYYBJcV5uX5HDtQuClBelUxRMjI7+bWOgsMLZuk/BWz+A3T+Gw89C9irY+EVYeQ/E+ua6pCIzpqKi4pxxzt/97nd5+umnAaivr6empoZgMHjOe4qLiykrKwNg3bp11NbWTlt5FNSXkeKL5ZoFQa5ZEORbW1bQ2T/C3vounq9q4vmDTTz99ikAclJ8znklQTaWBClMT4j84A7kw63fhJv+CvY/Djsfhme+DNu+CeX3Qfn9kJI716WUKHOplu9sSUpKOvv8lVde4YUXXuDNN98kMTGRTZs2TTqDMj4+/uxzr9fL4ODgtJVHQX2F0pLiuHlpFjcvzeLbf7yKo6197DrRwZvH23mtpvVscJcWBLizNJfbl+dQlJF0mU91udgEWPdZWPsncGK7E9jbvwOv/yus+AhseAAK1s11KUXeM7/fT29v76THuru7SUtLIzExkcOHD7Njx45ZLp2C+n3xeEx4qVY/n944H2stR1v6ePWdVp7Y3cA/PneYf3zuMBuK0/lExTzuWJkT2f3axkDJTc7WcRx2fR/2/BQO/BIK1sOGL8Lyu8Ebe/nPEnGRYDDIddddx8qVK0lISCA7O/vssTvuuIOHH36Y0tJSlixZwsaNG2e9fLqYOINOdQ3yq7dP8Yu36jnZMUAgIZYPLMvitmXZ3LA4k+RoGEUy3At7H3Na2R3HwZ8H6++HdfdBUvDy7xdh7ofnzbYrvZiooJ4FoZBlx/F2ntjdwEtHWugaGCXO6+HahUFuXZbNrcuyyQlE+MW5UAiObnNGixx/GWJ8sOpjzmiR7LnvcxR3U1ArqF1lbDxEZV0nLxxqZlt1M3XtAwCUZCZRVpjKXaV53Lg4M7LXI2k57LSw9/0cxgah6AYnsBffAZ4I7vqRGaOgVlC7lrWWY619vFDdwu66TiprO+gcGCUv4OPj6wu5szSPhVkRPDtwoAP2/MTpy+5pgLQiZwLNmk+DLzDXpRMXUVArqCPGyFiIF6qb+dmuk7xW0wbA4uxkbl6aRXEwibXz01iYmYwn0lrb42Nw+Dew42Go3wFxyVD2SefiY3DBXJdOXEBBrQkvESMuxsPmVblsXpVLU/cQz1ed5rdVTfzgtROMh5xfqElxXlbkBbh+UQZ3l+UxPxgBQ/+8Mc4wvhUfgca3ncCu/A/Y9Qgsut0J7AW3aNajyEWoRR0BxsZD1HcOsruukwMNXeyt72JfQzcAZYWp3LY8m5sWZ7I8NyVyWtu9zVD5I6j8IfS3QsaS8GJQ90JcBPzykWmlFrW6PqJSY9cgv9nXyK/3NXIwvC5CRnIc1y/M4MbFmdywKJNMf/xlPsUFxoadFft2PgSn94Ev1ZlYU7EVUgvnunQySyIxqJOTk+nr66OxsZGvfOUrPPHEExecs2nTJr7zne9QXn5u/qrr4yqRl5rAF25awBduWkBL7xCvvdPG9ppWtte08au9jQAszfGzviidzaty2ViS7s4p7THxUPYJpyV9cocT2G/+O7z5b7D0Tme0yLxr1C0irpWXlzdpSE8nBXUUyPL7+Oi6Aj66roBQyHKwsYftNa3sON7Ok3sa+OmOOoqCiaydn8bGkiBr56W57yYJxsD8a5ytqx7e+j7s/k+o/rWz/OrGB2DlR51gF5kBX//615k/f/7Zmwd861vfwhjD9u3b6ezsZHR0lL//+7/n7rvvPud9E1feGxwc5L777uPQoUMsW7Zs2tb7UFBHGY/HsKogwKqCAF++eSFDo+P8Zl8jv61q4pUjrTy1x1mLZH4wkQ8szWZFXgoVxekUpifOccknSC2E2/4b3PR12P8L2Pk9+NUDsO1vofxzzmJQ/uzLf45Ept9+A5oOTO9n5qyCD/6PS55y77338tWvfvVsUD/++OM8//zzfO1rXyMlJYW2tjY2btzIli1bLtrIeeihh0hMTGT//v3s37+ftWvXTkvxFdRRzhfr5WPlhXysvJBQyHK8rZ+dJ9rZdqiZ/7ujjpHxEADZKfFcuyCDj64t4NoFQXdclIxLcoJ53X3ObMcdD8Or/wSv/Qus/GNntEj+9PxDEFmzZg0tLS00NjbS2tpKWloaubm5fO1rX2P79u14PB5OnTpFc3MzOTk5k37G9u3b+cpXvgJAaWkppaWl01K2KQe1McYLVAKnrLV3TstPl1nl8RgWZiWzMCuZT22Yz+h4iGOtfew41s7b9V28WN3M02+fIj81gQ8sy2JVfoBblmYRTJ7j7gZjnOF7C26B9mNOC3vvo05ru3CDE9jLtjjDACXyXablO5PuuecennjiCZqamrj33nt59NFHaW1tZffu3cTGxlJUVDTpEqcTzUSX4pV8sx8EqoGUaS+FzIlYr4elOSkszUnhT4Gh0XF+f6iZJ3c38OTuBn7yZh2xXsOty7LDgR1HaUEqGXMZ3MEFsPmf4Za/gbcfhV3fgyfug5QCqPgzWPtZSEyfu/JJRLv33nv5/Oc/T1tbG6+++iqPP/44WVlZxMbG8vLLL1NXV3fJ99944408+uij3HzzzVRVVbF///5pKdeUgtoYUwB8CPgH4L9Oy08W1/HFetmyOo8tq/MYD1mONPXy5J4GntrTwG+rmgDwGKgoTufaBRmUFaayJMdPdsocLCjlC8A1X3LGXr/zO2e0yAvfglf+CVb/F6eVnRVZw71k7q1YsYLe3l7y8/PJzc3lU5/6FHfddRfl5eWUlZWxdOnSS77/gQce4L777qO0tJSysjIqKiqmpVxTGkdtjHkC+DbgB/7ycl0fGkcdXcbGQ9R1DNDaO8wbx9r5XVUTR5rfXWR9zbxUbliYQUlmMgsykynOTJqbJVybDzqLQe1/HMaGoGSTc1ODRbeDxzP75ZEpi8Rx1O/HtE94McbcCWy21n7JGLOJiwS1MWYrsBVg3rx56y73J4JEtv7hMXYcb+dwUy/PHThN9ekeQhO+SivzU7irNI/yonRW5qcQHzOLq+b1t8OeH8OuH0BvI6SXQMUXnPVFfOq5cyMF9fsP6m8DnwHGAB9OH/VT1trJb6+LWtRXo+GxcU62D3CstY93mvt4vqqJQ6edGZNxMR5K8wMsz0uhIC2B5bkBVuSlkJYUN7OFGh91xmHveBgadkGc31m5b8NWJ7zFNRTU0ziF/FIt6okU1ALQ0jvEnrpOZwnXuk6OtvTROzR29rjfF8Oy3BQ2lgQpKwyQ4oulrDCVGO8MdFOc2u0E9sGnIDTurI298YtQfJNmPbpAdXU1S5cuddckrBlireXw4cMKanGvjv4RDjX2cLCxm4bOQfbWd3Gwsftst0mKL4YbF2cyP5hIdoqPkoxkVhcG8Pum6T6MPafDi0H9CAbaIGu5c0Fy1cchzkWTfq4yJ06cwO/3EwwGozqsrbW0t7fT29tLcXHxOce0KJO4WteAE96dA6O8cqSFV95ppa1vmDNfzZjw+O+K4nRuWpxJeVE6gYT3GdyjQ1D1pDNapOkAJKTBuj+F9Z+HQP77rpNcmdHRURoaGi47Rjka+Hw+CgoKiI099zusoJaIYq3FWmjrG+ZIcy87jrdzsLGHncc7GBwdB5x+740lQW5YmMGaeamszA+8tzu8Wwt1b8CO/wNHngMMLN/ijBYprFC3iMwaBbVEheGxcXbXdvJ2fRfNPUNsf6eV2vA9J+O8HtbOT2V9UTqlBamsLgiQdaXjuzvrnJsZ7PkpDHdD3honsFd8BGJm+MKnXPUU1BK1WnqHePtkF5W1HfzhaDtHmnvP3g0nN+Bjy+o87lqdx5IcP7FTvUg53Af7fuZMVW+vgeRsZyGo8s9BcuYM1kauZgpquWoMjoxzsLGbfQ3d7DzezgvVzYSs0+LOS/WRG0hgfjCRssJUUhPjyEqJpzQ/MPlIk1AIjr3k9GMffQG8cbDyHme0SO7q2a+cRDUFtVy1mrqHeKu2g4ONPTR0DnC6e4h3mnvPGSYYH+OhMN0J7zPbBS3w1necdUX2/gxG+2HetU5gL/mQFoOSaaGgFplgdDxES+8wnf0jnOwYYE9dJ8fb+tlb30VH/wjghPc1C4Lctjybxdl+FmUlk5oYB4Nd8PZPnb7srpMQKISKzzu3D0tIm+OaSSRTUItMgbWWhs5B3q7vYu/JLp7Ze4r2cHAD5KT4WDMvlXXz01hXmMLKvjeIfet7UPc6xCY6txPb8EXIXDKHtZBIpaAWeQ9CIUtj9yA1LX3UNPdSdaqHPSc7aeh0bq8UH+NhdWEqGxNOcXvfr1ja9jtiQiP05N+IZ+MXiVlyO764aZqoI1FPQS0yjVp6hqis66SytpM9Jztp7hmiqWeINNvDJ70v8pmYbWSbLo6Hcnk24S5qCz9MQVYGmf548tMSsBbiY7wkxHnI8vvcdRs0mTMKapEZNjYeon9knOOtfbR39+Gp/jXLTj5Kbt9B+kjk52M38ePxP6LBZl3w3lX5AdbMS2V1QSq5AR8bSoJ43XArNJlVCmqRuVL/Fux8CHvoGQiN0z3vVtpWfI6OzA0MjI5TdaqbPxxtZ39DF/0jzqzLxDgvK/JSWJEXYFV+gJX5ARZkJs3MYlXiGgpqkbnW0whv/QAq/wMGOyB7ZXgxqI9BbALjIcvx1j6ONPdSWdvJgVPdHGrsOWfKfDApjiU5fpblplCckUSKL4bEuBjmpSeSn5Yw9Qk94koKahG3GB2EA790llxtOQiJQecu6+v/DFJyzzn1THgfONXNkaZeWnqHqT7dw9GWPsZC5/67jfEYCtISKMpIYn1ROivzAxQFE8lLVYBHCgW1iNtYC7WvOYF95DnweGH5h53hfYXrL/nW0fEQjV2DDIyM0zM4Sl3HAHXt/dS2OTduONz07m3SvOEAn5eeSFEwiflB53FJjp+CtISoXlI00iioRdys4wTs+r4zkWa4B/LXwcYvwfK7wXvlw/taeoeobRugtr2fk+3hx44Batv66Tnvxg0ZyfHEeT3kpyWwMj9AcUYi/vhYluWlEEyKe28rEsp7oqAWiQTDvc4U9Z0PQ8cx8OfC+vudrpGkjGn5EV0DIxxr7edwUw+HT/fSNTjKyNg47zT3Udvez8Q4iIvxsCTbT2lBgLzUBHJSfOQGfOQEnDVTEuIU4tNJQS0SSUIhZxGonQ85i0J546H0Y86SqzkrZ+zHnrnvZWvfMLVtA5xo62NfQzc1zb10DoxecH5qYixpiXEMjIyRE0hgWY6fGxZlkhPwkZ0ST6Y/fnZvahzhFNQikarlsLMY1L6fw+gAFN3g9GMv+aDTrz1LhkbHOd09xOnuQZq6hzjdPURT9xCdAyMkxHpp7B5kT13X2VEq4MzcnJeeSElmEktyUlia42dJjp+iYJLGiU9CQS0S6QY6YM9PnL7sngZInQ8VW527qiekznXpAOgfHqOufYDmniFaeofYW99Na+8wx9v6qG3rP3tfzLgYD5nJ8aQnxZGWFEdSnJfB0XFyAz5KC1KZH0wkJ8VHcUbSVXWxU0EtEi3Gx+Dws04/9sk3ITYJyj7ptLIzFs516S5qaHScmuY+Djc5wwtb+4bp6B+hs3+E3qExEuO9NHQO0jWhiyXLH8/GkiDL81K4fmEGK/JSojq4FdQi0ahxrxPYVU/C+AgsvA0W3grpJZBe7LS6I+gWYtZa6tqdNcNPtPXz5vF2Kms7ON3t3PA2L+AjLzWB7ICPBZnJLMhMIi0xjpzwBc6U6bpT/RxRUItEs74WqPyRM+uxr+nd/cYDgQInuNOKwwEe3tKKIC4yFoNq6xvmxepmtte00dk/wqmuQeo7Bpg45yfGY8hIjic+1ulWmR9MYlmuM4tzcbafOK+HlIQYV7fIFdQiVwNrob8NOo47W+eJd593HIfBznPP9+dOCPGJQV4MvsDc1GGKhsfGqWsfoHtwlNPdQxxqdLpU4mM8tPUNc7ytn9be4XPeU5CWQHFGEnmBBDL98SzPS+G6hRkEEtzREldQi4gT1B1nwvvEuUHe13zuuYnByVvi6cXOMRe3TM9o6xvm8Olealp6GRoNceBUF6e6hjjdNUhb3zAhCx4D84NJ+H0xrMwPkJoQS1yMx9m8HpLjY7h+UQYFaTP/14eCWkQubbgPOmsnaY2fgO4GYEJOxKe82wJPO68lnpwDHvevLTI6HmJvfRfb32nleFs/HX0jHDjVzdDo+AXrqAAszEomN+AjIzkeX6yX+BjP2XttJsZ5CSTEMjIWIsMfT06Kj1ivh0x/PB7DlLtbLhXUuiuniEB8sjOZZrIJNaNDzv0hzw/x0/ug+jcQendaOjEJE0K86NzWeKBgVsd+X0qs18P6onTWF6VfcCwUsoyMhxgZD9HaO8zvDzazu66T1t4hjrf2Mzw2zvBYiOFR55yL8XoM1loywkMRF2QlkxDrhLoBEuNjWJLtJzUxFr/v0lGsoBaRS4v1QeZiZzvf+Bh0108I8Vrnsf0o1GyD8Qn9xJ5YSJs/eZdK6jzXjFDxeAw+jxdfrJcUXywPbEqe9DxrLSfDFzU7+kfwxXpo6RmmtW+YkbEQTd1DhKylrW+Ytr4R9jd0MTpm6R0axeIMWZyk8T4pdX2IyMwIhaD39LkXNCd2qYz0vXtulIxQuRK9Q6M0dg3RNeCMJb9tRY66PkRklnk8EMh3tuIbzj12/giViSF+6FdRNULlYvy+WJbkTG3EiYJaRGafMZCc6WzzNlx4/GIjVI5uu/gIlckubkbICJXLUVCLiPskpEF+GuSvvfDYxUao1L0B+x9naiNUSsCfEzEhrqAWkchylY1QAQW1iESTKxmhcqZLJQJGqCioReTq4I0Jt6CLgQ+ce+xSI1Tq3rj4CJXzg3yGRqgoqEVELjtCpXXCxc0JIX7w6YuPUEkvvvDi5nscoaKgFhG5FGMgOcvZrmSESs2VjFApuWQRFNQiIu/HlYxQOdMan2yEyiUoqEVEZsqVjFD5uy9f9GMuG9TGGB+wHYgPn/+Etfab77ngIiIyyQiV9xHUwDBwi7W2zxgTC7xujPmttXbHNBRVREQu47JBbZ1Vm86MTYkNb9O/kpOIiExqSit8G2O8xpi9QAuwzVq7c5JzthpjKo0xla2trdNdThGRq9aUgtpaO26tLQMKgApjzAU949baR6y15dba8szMzOkup4jIVeuK7pljre0CXgHumJHSiIjIBS4b1MaYTGNMavh5AnArcHimCyYiIo6pjPrIBf7TGOPFCfbHrbXPzmyxRETkjKmM+tgPrJmFsoiIyCTcf193EZGrnIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMtdNqiNMYXGmJeNMdXGmIPGmAdno2AiIuKImcI5Y8BfWGv3GGP8wG5jzDZr7aEZLpuIiDCFFrW19rS1dk/4eS9QDeTPdMFERMRxRX3UxpgiYA2wc5JjW40xlcaYytbW1ukpnYiITD2ojTHJwJPAV621Pecft9Y+Yq0tt9aWZ2ZmTmcZRUSualMKamNMLE5IP2qtfWpmiyQiIhNNZdSHAX4IVFtr/2XmiyQiIhNNpUV9HfAZ4BZjzN7wtnmGyyUiImGXHZ5nrX0dMLNQFhERmYRmJoqIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLXTaojTE/Msa0GGOqZqNAIiJyrqm0qH8M3DHD5RARkYu4bFBba7cDHbNQFhERmcS09VEbY7YaYyqNMZWtra3T9bEiIle9aQtqa+0j1tpya215ZmbmdH2siMhVT6M+RERcTkEtIuJyUxme9zPgTWCJMabBGHP/zBdLRETOiLncCdbaT8xGQUREZHLq+hARcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXm1JQG2PuMMYcMcYcNcZ8Y6YLJSIi77psUBtjvMC/Ax8ElgOfMMYsn+mCiYiIYyot6grgqLX2uLV2BPg5cPfMFktERM6ImcI5+UD9hNcNwIbzTzLGbAW2hl8OG2Oq3n/xXC0DaJvrQsww1TE6XA11hMiv5/yLHZhKUJtJ9tkLdlj7CPAIgDGm0lpbPuXiRSDVMTqojtEjmus5la6PBqBwwusCoHFmiiMiIuebSlC/BSwyxhQbY+KAe4Ffz2yxRETkjMt2fVhrx4wxfw78DvACP7LWHrzM2x6ZjsK5nOoYHVTH6BG19TTWXtDdLCIiLqKZiSIiLqegFhFxuWkN6miZam6M+ZExpmXiWHBjTLoxZpsxpib8mBbeb4wx3w3Xeb8xZu3clXzqjDGFxpiXjTHVxpiDxpgHw/ujrZ4+Y8wuY8y+cD3/Lry/2BizM1zPX4QvlGOMiQ+/Pho+XjSX5b8SxhivMeZtY8yz4ddRVUdjTK0x5oAxZq8xpjK8L6q+rxczbUEdZVPNfwzccd6+bwAvWmsXAS+GX4NT30XhbSvw0CyV8f0aA/7CWrsM2Ah8Ofz/K9rqOQzcYq1dDZQBdxhjNgL/BPxruJ6dwP3h8+8HOq21C4F/DZ8XKR4Eqie8jsY63mytLZswXjravq+Ts9ZOywZcA/xuwuu/Bv56uj5/tjegCKia8PoIkBt+ngscCT//HvCJyc6LpA14BrgtmusJJAJ7cGbWtgEx4f1nv9FR2sEAAAJXSURBVLs4o5uuCT+PCZ9n5rrsU6hbAU5Q3QI8izNRLdrqWAtknLcvar+vE7fp7PqYbKp5/jR+/lzLttaeBgg/ZoX3R3y9w3/6rgF2EoX1DHcJ7AVagG3AMaDLWjsWPmViXc7WM3y8GwjObonfk/8J/BUQCr8OEn11tMDvjTG7w0tWQBR+XyczlSnkUzWlqeZRKKLrbYxJBp4Evmqt7TFmsuo4p06yLyLqaa0dB8qMManA08CyyU4LP0ZcPY0xdwIt1trdxphNZ3ZPcmrE1jHsOmttozEmC9hmjDl8iXMjtY6Tms4WdbRPNW82xuQChB9bwvsjtt7GmFickH7UWvtUeHfU1fMMa20X8ApOn3yqMeZMQ2ViXc7WM3w8AHTMbkmv2HXAFmNMLc7qlrfgtLCjqY5YaxvDjy04v3AriOLv60TTGdTRPtX818Bnw88/i9One2b/n4SvMm8Eus/8KeZmxmk6/xCottb+y4RD0VbPzHBLGmNMAnArzgW3l4F7wqedX88z9b8HeMmGOzndylr719baAmttEc6/u5estZ8iiupojEkyxvjPPAduB6qIsu/rRU1zZ/9m4B2cPsC/mesO+PdRj58Bp4FRnN/M9+P04b0I1IQf08PnGpzRLseAA0D5XJd/inW8HudPwf3A3vC2OQrrWQq8Ha5nFfC34f0lwC7gKPBLID683xd+fTR8vGSu63CF9d0EPBttdQzXZV94O3gmX6Lt+3qxTVPIRURcTjMTRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXG5/w8Iko5yj1JV4AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy], callback_fns=ShowGraph)\n",
    "learn.fit_one_cycle(3, max_lr=1e-2, moms=(0.99, 0.90))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.091744</td>\n",
       "      <td>4.376647</td>\n",
       "      <td>0.064800</td>\n",
       "      <td>0.228900</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.645397</td>\n",
       "      <td>3.546236</td>\n",
       "      <td>0.142300</td>\n",
       "      <td>0.401900</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.348282</td>\n",
       "      <td>3.130295</td>\n",
       "      <td>0.232100</td>\n",
       "      <td>0.524700</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXAc533m8e+LwQADYDA4BjdAAiApEuAJkhBNHZEpWZYlWYctKV65bMel9Zq19qZ0OLuJXNlN4iqnktR6k5S31s6qyt6kEm28ikRJlmzJlmRStA5KBg+RAAGeAImDuK/Bfcy7f/QABEmQBKUB0Bg+n6quwfT0DN63OHz48u1fv22stYiIiHvFLXYDRETkyhTUIiIup6AWEXE5BbWIiMspqEVEXC5+Pj7Uk5xm49Ny8HriSE2MJys1kXiPwWPMfPw6EZElb//+/Z3W2uzZXpuXoE7PKeDx//k8bf0jvHm0ndBkGABPnCE9ycvaggAVy9K5aWWQ7aVB4uIU4CJyfTPGnLnsa/NRR11ZWWmrqqoAaOwe4v1TXfQNj9M3PE7nwCiHGns53hYiHPnVa/MDrM71s74wjfs2FpAbSMRo9C0i1xFjzH5rbeWsr813UF/O8Ngkv6pppa41RE1LH6c7BmnuHQagIM3H/ZsKuPWGLLYWZ5CcMC8DfxER13BlUM/mWGuId0928s7JTt4+3sFk2OL1GDYVpbN9RZDtK4JsLc4gKcET9TaLiCymJRPUMw2MTlDV0M2+0928f7qL6uY+JsOWBE8c21cGuWttLrfdkM3yYHKUWi0ii2V8fJympiZGRkYWuynzzufzUVRUhNfrvWD/kgzqi4VGxqk608N7Jzv59dE2znQNAZDlT+TO8hwe2VrE1uIMzW2LLEH19fWkpqYSDAZj+u+wtZauri5CoRClpaUXvBYTQT2TtZZTHYO8d6qT/Wd6eONoG0NjkxQHk/ncujw+U5bD1uIM4j0qExdZCmpraykrK4vpkJ5iraWuro7y8vIL9l8pqJfkWTpjDKty/KzK8fMHN5UwODrBa9WtvHyomf/zbj3P7D1NWpKXHWuyuaMshx2rc0hL9l79g0Vk0VwPIQ0fr59LMqgvlpIYzyNbi3hkaxEDoxP89ngHb9W1s7uunZcPteCJM1QWZ3BneS73byogL8232E0WEZmzmJsb8CfGc8+GfH7w+5v48E/vZNe3b+Y/fnoFfcPj/OUva7npr9/iaz/5gJcONjM8NrnYzRURF+jt7eVHP/rRNb/v3nvvpbe3dx5adKElOUf9cTV0DrLrQBMvHGimuXcYf2I8927I45Gty7ixRCciRRZLbW3tJXO2C6mhoYH77ruP6urqC/ZPTk7i8US/HHi2/sbcHPXHVZKVwnfuWsOTd67mg/puXjjQxKuHz/FcVROF6Ul8dm0uX962nDV5qYvdVBFZQE8//TSnTp2ioqICr9eL3+8nPz+fQ4cOcfToUb7whS/Q2NjIyMgITzzxBDt37gSgpKSEqqoqBgYGuOeee7j11lt57733KCws5OWXXyYpKSkq7buuRtSzGRyd4PXqVl6rPsfe452MTYZZnevnU6VBbl4Z5I7yHBLjdYGNyHyaOcL83is1HG3pj+rnry0I8Of3r7vs6zNH1Hv27OHzn/881dXV0yV03d3dZGZmMjw8zI033sjbb79NMBi8IKhXrVpFVVUVFRUVfOlLX+KBBx7gq1/96lX7O0Uj6itISYzn4a1FPLy1iO7BMV4+1Mxbte28dLCZf953hpQED59aEeTTq7O5Z0MeOak6ESkS67Zt23ZBnfMPf/hDXnzxRQAaGxs5ceIEwWDwgveUlpZSUVEBwNatW2loaIhae677oJ4pMyWBx24p5bFbSpkMW357ooO3att552Qnv6lr589/XkNOaiL3byrgM2U5bCnOwOeNodF21ymwFrJWLXZL5Dp2pZHvQklJSZn+ec+ePbz55pu8//77JCcns2PHjlmvoExMTJz+2ePxMDw8HLX2KKgvwxNn2LEmhx1rcgA42tLPe6c6+bC+m398r4GfvFNPYnwc20ozuXllFr93QxbrCgJL+4TkO38HB/8Z8jfBuodg/UOQvnyxWyUy71JTUwmFQrO+1tfXR0ZGBsnJydTV1bFv374Fbp2Ces7WFgRYWxDgP/zeCkIj43xY3827J7t492Qnf/N6HX/zOqzJTeVz63LZtCydyuLMpXeRzY7vQk45VO+CN//c2YpuhPUPw9ovQCB/sVsoMi+CwSC33HIL69evJykpidzc3OnX7r77bv7hH/6BjRs3smbNGrZv377g7bvuTyZGQ3v/CLuPtfMv+85S09JH2EKcgc3LM7h9TTaVJZmU5wdIS1pCwd3TADUvQvUL0HoEMFB8izPKXvsgpGQtdgslhix2ed5Cu9aTiQrqKBscnaC6uY93T3ay53gHh5v6AIiPM2wrzWRDURqrc1JZk5fKqhz/0pjj7jzhjLKrn4fO42A8sOLTzvRI+X2QlLHYLZQlTkGtoF5UnQOjHGnu48P6bvYc6+BU+wBjkVuTeT2G29fk8IXNhdy8Mkh6csIit/YqrIW2GqjZ5Yy0exogzgur7nRG2mvugUTVoMu1U1ArqF1lYjJMQ9cgda0hDp7t5ecftdARGsUYKM8L8OnIQlKbl6W7e/U/a6HloBPYNS9CfzPE+2D155yR9urPgTc6xf4S+xTUCmpXm5gMc+BsL++f6uLdyLKtk2E7vfrfv7+llI1Fae6uJgmHoenDSGi/BIPtkOB3RtjrH4aVd0B84tU/R65bCmoF9ZLSNzzOOyecuu03jrbSPzLB8sxk7l6fx2fX5rI2P0BKoouLdcKT0PCOE9q1P4fhHvClQdn9zvRI6afB4+L2y6JQUCuol6y+4XFeO3KO12taefdkJ+OTzq3Ibr0hi5tXBrltdTarc108Jzw5Dqf3OCci616F0X5IDjpVI+seguKbIW4JnEyVeaegVlDHhP6Rcd472UVVQzev17TS1ONc9bQiO4VtJZlsKc6gJJhCQbqPwvQk902VjI/Aqbeckfax12B8CPx5sO6Lzki76EZwW5tlwSzFoPb7/QwMDNDS0sLjjz/O888/f8kxO3bs4Ac/+AGVlRfmr9b6iFEBn5e71+dx9/o8/ut9a2nvH+GlQ83sO93NL46c42e/a5w+NjMlgU+VZvKp0ky2rwyyOieVuLhFDkGvD8o+72xjg3D8V05oV/0UPvgxpC2HdV9w5rTzNym0ZckoKCiYNaSjSUG9ROUEfOy8bSU7b1vJZNjS0DVIc88wZ7uHOHC2hw9Od/NadSsACZ44khM9bFmeQbY/kWWZSWxZnsG20szFqSxJSHFG0esfgpF+OPZLJ7T3/Qje+yFkroy8/rBzpaTIAviTP/kTiouL+fa3vw3AX/zFX2CMYe/evfT09DA+Ps73v/99HnzwwQveN3PlveHhYR577DGOHj1KeXl51Nb7UFDHAE+cYWW2n5XZfgC+ur0YgMbuIT6o7+ZEW4jW/hFOtA1wpLmPjtAo4Iy81xUEWF+YRmlWCpuK0lmV48ezkKNvXwA2PepsQ91Q+4oT2r/9H7D3v0N2uRPY6x+C4MqFa5csnteejlwNG0V5G+Cev77iIY8++ihPPvnkdFA/99xzvP766zz11FMEAgE6OzvZvn07DzzwwGWnFn/84x+TnJzM4cOHOXz4MFu2bIlK8xXUMWxZZjLLMpMv2T84OsHe4x28WdtO7bl+frzn1PRryQkeVuemsr4wQGVxJluLMyjKWKA57+RM2Pp1Zxtoh6MvOycid3/f2fI3OaG97otaLEqibvPmzbS3t9PS0kJHRwcZGRnk5+fz1FNPsXfvXuLi4mhubqatrY28vLxZP2Pv3r08/vjjAGzcuJGNGzdGpW0K6utQSuS+kvdscBZZmgxb6jsH+aixl8NNvRxrC/HSwRb+Zd9ZAAK+eMryAqzO87MmL0BZXiqrc1Pnd+0Sfw5s+6az9TU59dk1u+CNP3O2om2RdUe0WFTMucrIdz498sgjPP/887S2tvLoo4/y7LPP0tHRwf79+/F6vZSUlMy6xOlM8zGomXNQG2M8QBXQbK29L+otkUXjiTOsyvGzKsfPw1uLACe8j7WG2H+2h9pz/RxvDfHywRZCo2en31eQ5mN1nrNuSTAlgYzkBDYvT2dltj+6X9a0Irj5D52tuz6yWNQueP1peP27UHKrM8rWYlHyCT366KN885vfpLOzk7fffpvnnnuOnJwcvF4vu3fv5syZM1d8/2233cazzz7L7bffTnV1NYcPH45Ku65lRP0EUAsEovKbxdU8cWZ6adcp1lpa+kY41trPsdYBjrX2U9camq7xnpLlT6BiWQZrCwLcvsap9Y7aRTqZpfB733G2juPn1x35xXfgl/8FVuxwRtpl90FSenR+p1w31q1bRygUorCwkPz8fL7yla9w//33U1lZSUVFBWVlZVd8/7e+9S0ee+wxNm7cSEVFBdu2bYtKu+ZUR22MKQL+CfhL4DtXG1Grjvr6MjEZZmQiTGvfCPvPdPPB6W4ON/dxumOAsHUWn7qzPJftK4JsKEpjbX4guqsGTi0WVf2CE9w9DeBJgJWfcea019wDif7o/T6JuqVYR/1JzFcd9d8Dfwxc9jI4Y8xOYCfA8uU60XM9iffE4ffETU+f/LsbnT//zoFRqhq6+bC+h1cOt0yXC3o9hvWFaVQWZ7C1OIN1BWmf7ISlMZC33tk+82fQcsCZGql5EY6/BvFJsPouJ7RvuEuLRcmSc9URtTHmPuBea+23jTE7gP+sEbVcK2strf0jHG7q4+DZXvaf6eajpj7GJpwlX/2J8ZTnp7JleQabl2ewpTj9k99IOByGxg+ckfbRl2CwI7JY1L3O9IgWi3INjag/4SXkxpi/Ar4GTAA+nDnqXdba2e+DjoJa5mZsIkxNSx+150Ica+3nSHMf1c390+t1F2UkRYI7nfJ8p9rkY6/ZPTkBZ95xRtozF4sqv99Zd0SLRS2q2tpaysrK3Lf0wTyw1lJXVzd/a31oRC3zbXRikurmfg6e7eHA2R4OnOmltf98OVRuIJHy/ABblmewoSiNJK+Hldl+MpK9c7/KcmLMWSyqZhfUvgpjIUjOgrUPONMjy2/SYlELrL6+ntTUVILBYEyHtbWWrq4uQqEQpaWlF7ymoJYlra1/hLpWZ9Rddy5EdUsfJ9oHmPnVTU7wsHm5c1PhyhJn+sQ/l0qT8RE4+aYzPXL8dWexqNR8pz57/cNQVKl1RxbA+Pg4TU1NV61RjgU+n4+ioiK83guvQ9DqeRJz+obHqTvXz/D4JA2dg9R3DvK7hh7qWvunby68tsC5uvLGkkxuLMkgJ3CVOe+xQSesq3fBiTdgctRZLGr9F53Qztuo0JZ5o6CW60ZoZJyDZ3upaujmdw09HGzsYWTcmfNeXxggL+BzLq3PSKYg3UduwMeGwrRLp01G+qDul870yKnfQHgisljUw5HFoq5cTytyrRTUct0anwxT09LP28c6qDrTTUdolDNdQwyPT04fk5LgYW1BgHUFac7l8XmpbJwZ3kPdzgnI6l3Q8FuwYchZ61SOrNNiURIdCmqRi7T2jdA9OEZD1yD7TndxtKWfmpb+6QD3egzLMpMpDaZQHEzB543DGMiilzXdu9nQ8yap7ZHveH5FJLS1WJR8fApqkTmYmAxzrm+EQ4291LT009A5SEPXIGe6hqZLBifD5/++5NPFQ77f8WD8PlZPHAcgXLSNuPUPOzdBSJ19hTWR2SioRT4Bay3WQthaJq0lNDLB/jM91HcOcrZ7iNpz/XQ1HuO+uH3cH7+PcnOGMIbOYCWJFb9P2pZHICW42N0Ql1NQi8yzroFRPqzv5sDZHiba6shr/CV3Tr7DyrhzTBLHiZRK6rI+S0veZ1hWkM+20kxyr1aFItcVBbXIAguHLcda+6k++B4JtS9SObCHQtoYtfHsDW/ilcmbOJV5K6uK8ijNSmFVjp/y/AAlwZSFvcOOuIaCWmSxWQvNB5g48jz2yC68Q62MmkTeMVv5t9Ft7J6sYJQEEuPjKM1KIehP4IacVFZkp1CQlsSGojSNwGOcglrETcJhaNznlPtFFoua9KbQlHMH7yfdxm/G19M2GOZk+wCDY04ViifOWXFwTa6f5ZnJ7FiTw9r8wOLfXV6iRkEt4lbTi0W9AEd/DiO904tF2XUP0561jaa+cfYca+eD+m5OdwzQOTAGQGJ8HD6vh+QET+TelslsXp7O5mXpV78KU1xHQS2yFEwtFlX9AtT9YsZiUQ/OWCwqjo7QKHuPd3CsLcTo+CSt/SPUngvR2jcyXUZYkOajPD9A0J9AQnwcYQsbCtPYVJTO6lz/3BewkgWjoBZZasZH4OQbzvTIsddgYthZLGpdZN2Rwq2XrDsyMj5JTYuz3vehxl5Otg/QOzTO4NgEWAiNTgDg88axoTCNsrwA5fkBtpVmUpieRFKCVgxcTApqkaVsdMBZLKrmRTjxa5gcc66AXPeQc0XkHBaLstZypmuIj5qcEP+osZcT7QOERiamjwmmJLAy28+avFRyUhNZlpnM+sIAOQEfqYnxMb38qBsoqEVixdRiUdUvwOndzmJRwVWR0L62xaKstdR3DnLgbC8doVFOtIc40zXE8dbQ9Oh7SpLXQ24gkZyAj/zItIo/MX56W5aZTHEwObr3wrzOKKhFYtH0YlEvQMM7zmJRyVmQU+4sGjX9WOacoLyWjx6b4GT7APWdg7T3j9LWP0JbyHk80zVIW//oJe8xxjnBmZGcQFqSl8qSDLYszyAv4CMrNZGSYAoJ8ZobvxwFtUisC7VB3SvQcgjaa6GjDsYGzr8eKIwE94wQz1oDCcnX/KustQyMTjA8NsnA6AR9w+Oc7R6ioXOIwbEJ2vqdBa8OnOmZLi8EZ258RZafsrxUblmVxaocP/lpPoL+RF3kg4Ja5PoTDkNfoxPY7Ued8G4/Ch3HnRsiAGAgs9QJ7uyy8yEeXAXxH/PelDNMTIZp6Bqic8AZiX/U2Ed95wAfNfXRPTg2fZwnzlCYnsSavFRGxidJ8nq4eWWQtQVplOWnEvB5r/BbYoeCWkQckxPQUz8jvCNb10mwkdFvXDwEb7hoCqUcMkqici/JcNhyvD1EY/cwrf0jtPYN09A1xLHWEEleD92DYzT3Dk8fn5mSQG7AR2aKl2BKIhXL0qlYnk56kpfCjCQS42NjXlxBLSJXNjEKnSfOj7zba6GjFnoazh8T74PsNRfNf5c70ypRrAix1tLWP0rtuX6OtYU42z1Ee/8o7aEROkKjnOs7f1/FOAPLMpNZkZUSOaGZwg05fjJTEgj6nf8VGAyBpHiSvB5XV64oqEXk4xkdgM5jM0bfkRAPnTt/TGLgwqmTqUd/9rw0qaV3mCPNfQyMTHCma5BTnYPUdwzS1DNE/8jEZd/niTMEfPGk+rzkp/kozEgiP83Hymw/K7L9rMhOWdRpFgW1iETXUHdk/nvG9El7DQz3nD/mggqUsvNz4Unp89astv4RGruH6BwYo3twDGOcdcRDIxOERsbpH3Ye67uG6IhUssy8GUR2aiLLMpLITEkg4POSnpxAaVYyJVkp5KT66BseJy/goyDdx0TYRrUc8UpBHR+13yIi14/kTCi+2dmmWAsD7RdOnbTXwqFn560C5WK5Ad81rTI4PhnmbPcQp9oHONUxyKmOAVr7RmjqGaZ7sI/e4XHGJsKXvM8Yp7sBXzzWQlFmMmV5qSQleOgfHiczJYF1BQGKMpIpykgiPy3pE5UmakQtIvPLWqcCZebUyWwVKBklF568jGIFyscVDlvaQ6PUdw7SMTBKwBcfGbUPkxAfR3tohMkwNPU4YR8amSA7NZG2/pELShONgbyAj7K8VFZm+0n1eUlKiCM9KYHiYDJbizPwxns0ohaRRWKMc8l7+nJY/bnz+ycnnJOVM8O7vda5XH7WCpQZAR6lCpSriYsz5KX5yEu7ttUIp+6/2dgzRHPPME09wzR2D3GkuY/3T3cxMn7hKN3nvfJoW0EtIovDEw9Zq5xt7QPn90+MOuWCM8O75QDU7Dp/zGwVKNllkFYU1QqUjyveE8eyzGSWZV46nWOtZWwyzNhEmL7hcY409bH/TA//7Qqfp6kPEVkaLqlAiWyhlvPHLHAFSjTpZKKILH2Jfmd518KtF+4f7oH2mVdg1kLtK3Dgn84fM12BUn7hCHweK1CiSUEtIktbUgYU3+RsU6yFwY5L578P/d/LV6BkTz2WRaUCJZoU1CISe4wBf46zrdhxfv90BcpFa6DU/9bVFSgKahG5flxQgXLX+f3hSeiuv7QG/JIKlFWXzn8vQAWKglpEJM5zDRUoh6DmJSBSiDFVgZJ90UU8UaxAUVCLiFxOfCLkrnO2mcYGoePYhQFevxcO/+z8MQmpF12BGbmMPiX7mgNcQS0icq0SUqBwi7PNNNx70Rrgs1WgBC+d/86+8i3UFNQiItGSlA7LtzvblAsqUGaE+KF/hbHQnD5WQS0iMp+uWIHSdH765HtPXfYjFNQiIovBGEhf5myr7wIuH9S6JbCIiMtdNaiNMT5jzIfGmI+MMTXGmO8tRMNERMQxl6mPUeAOa+2AMcYLvGOMec1au2+e2yYiIswhqK2zvN7UxfHeyBb9JfdERGRWc5qjNsZ4jDGHgHbgDWvtB7Mcs9MYU2WMqero6Ih2O0VErltzCmpr7aS1tgIoArYZY9bPcswz1tpKa21ldrb7134VEVkqrqnqw1rbC+wB7p6X1oiIyCXmUvWRbYxJj/ycBNwJ1M13w0RExDGXqo984J+MMR6cYH/OWvvq/DZLRESmzKXq4zCweQHaIiIis9CViSIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyVw1qY8wyY8xuY0ytMabGGPPEQjRMREQc8XM4ZgL4I2vtAWNMKrDfGPOGtfboPLdNRESYw4jaWnvOWnsg8nMIqAUK57thIiLiuKY5amNMCbAZ+GCW13YaY6qMMVUdHR3RaZ2IiMw9qI0xfuAF4Elrbf/Fr1trn7HWVlprK7Ozs6PZRhGR69qcgtoY48UJ6Wettbvmt0kiIjLTXKo+DPAToNZa+7fz3yQREZlpLiPqW4CvAXcYYw5FtnvnuV0iIhJx1fI8a+07gFmAtoiIyCx0ZaKIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i43FWD2hjzU2NMuzGmeiEaJCIiF5rLiPofgbvnuR0iInIZVw1qa+1eoHsB2iIiIrOI2hy1MWanMabKGFPV0dERrY8VEbnuRS2orbXPWGsrrbWV2dnZ0fpYEZHrnqo+RERcTkEtIuJycynP+1fgfWCNMabJGPON+W+WiIhMib/aAdbaLy9EQ0REZHaa+hARcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXU1CLiLicglpExOUU1CIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i4nIJaRMTlFNQiIi6noBYRcTkFtYiIyymoRURcTkEtIuJyCmoREZdTUIuIuJyCWkTE5RTUIiIup6AWEXE5BbWIiMspqEVEXE5BLSLicgpqERGXm1NQG2PuNsYcM8acNMY8Pd+NEhGR864a1MYYD/C/gHuAtcCXjTFr57thIiLimMuIehtw0lp72lo7BvwMeHB+myUiIlPi53BMIdA443kT8KmLDzLG7AR2Rp6OGmOqP3nzXC0L6FzsRswz9TE2XA99hKXfz+LLvTCXoDaz7LOX7LD2GeAZAGNMlbW2cs7NW4LUx9igPsaOWO7nXKY+moBlM54XAS3z0xwREbnYXIL6d8ANxphSY0wC8Cjw8/ltloiITLnq1Ie1dsIY84fArwAP8FNrbc1V3vZMNBrncupjbFAfY0fM9tNYe8l0s4iIuIiuTBQRcTkFtYiIy0U1qGPlUnNjzE+NMe0za8GNMZnGmDeMMScijxmR/cYY88NInw8bY7YsXsvnzhizzBiz2xhTa4ypMcY8Edkfa/30GWM+NMZ8FOnn9yL7S40xH0T6+f8iJ8oxxiRGnp+MvF6ymO2/FsYYjzHmoDHm1cjzmOqjMabBGHPEGHPIGFMV2RdT39fLiVpQx9il5v8I3H3RvqeBt6y1NwBvRZ6D098bIttO4McL1MZPagL4I2ttObAd+E+RP69Y6+cocIe1dhNQAdxtjNkO/A3wd5F+9gDfiBz/DaDHWrsK+LvIcUvFE0DtjOex2MfbrbUVM+qlY+37OjtrbVQ24CbgVzOefxf4brQ+f6E3oASonvH8GJAf+TkfOBb5+X8DX57tuKW0AS8Dn43lfgLJwAGcK2s7gfjI/unvLk51002Rn+Mjx5nFbvsc+laEE1R3AK/iXKgWa31sALIu2hez39eZWzSnPma71Lwwip+/2HKttecAIo85kf1Lvt+R//puBj4gBvsZmRI4BLQDbwCngF5r7UTkkJl9me5n5PU+ILiwLf5Y/h74YyAceR4k9vpogV8bY/ZHlqyAGPy+zmYul5DP1ZwuNY9BS7rfxhg/8ALwpLW235jZuuMcOsu+JdFPa+0kUGGMSQdeBMpnOyzyuOT6aYy5D2i31u43xuyY2sP5sOkAAAGVSURBVD3LoUu2jxG3WGtbjDE5wBvGmLorHLtU+ziraI6oY/1S8zZjTD5A5LE9sn/J9tsY48UJ6Wettbsiu2Oun1Ostb3AHpw5+XRjzNRAZWZfpvsZeT0N6F7Yll6zW4AHjDENOKtb3oEzwo6lPmKtbYk8tuP8g7uNGP6+zhTNoI71S81/Dnw98vPXceZ0p/b/QeQs83agb+q/Ym5mnKHzT4Baa+3fzngp1vqZHRlJY4xJAu7EOeG2G3gkctjF/Zzq/yPAb2xkktOtrLXftdYWWWtLcP7e/cZa+xViqI/GmBRjTOrUz8BdQDUx9n29rChP9t8LHMeZA/zTxZ6A/wT9+FfgHDCO8y/zN3Dm8N4CTkQeMyPHGpxql1PAEaBysds/xz7eivNfwcPAoch2bwz2cyNwMNLPauDPIvtXAB8CJ4F/AxIj+32R5ycjr69Y7D5cY393AK/GWh8jffkostVM5UusfV8vt+kSchERl9OViSIiLqegFhFxOQW1iIjLKahFRFxOQS0i4nIKahERl1NQi4i43P8Hf3SedFZh0v4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy], callback_fns=ShowGraph)\n",
    "learn.fit_one_cycle(3, max_lr=1e-2, moms=(0.95, 0.85))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The testing took around 11 minutes and I think the value of 0.99-0.90 for momentum is better than 0.95-0.85. This is it. Using this method you can get creative and test out many of your hyperparameter choices without using any external libraries which in most cases do not even work.\n",
    "\n",
    "One of the things that you may be wondering what to do if I have large datasets like Imagenet dataset where every epoch takes hours to run. In that situation I would suggest to take a small smaple of the dataset and adjust your hyper-parameters using those. Generally you don't need to have very large validation set, a subset of the validation set can also work provided that subset is a good representation of the actual validation set. You can become creative when using large datasets where for the training dataset you treat a specific number of batches as a single epoch. For example, if my dataset can be divided into 100 batches and I decide every 20 batches would be treated as a single epoch. So after 100 batches I would have done 5 epochs instead of 1.\n",
    "\n",
    "**Note**:- I did not even need 3 epochs to decide which value is better. If we just compare the resutls from the first two epochs we can clearly make our choice without having to do third epoch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choosing Weight Decay\n",
    "Weight decay is not like momentum and learning rate and the best value should remain constant throughout the training. Since the networks performance is dependent on a proper weight decay value, a grid search is worthwhile and differences are visible early in the training. That is the validation loss early is sufficient for determining a good value.\n",
    "\n",
    "So to set the value of weight decay you should run combined runs using different values of weight decay and momentum and possibly learning rate. Generally what I found good is using the `lr_range` test to get the value of learning rate and then adjust momentum and weight decay accordingly using various combinations.\n",
    "\n",
    "The reason we can avoid finding the values of learning rate, momentum and weight decay simultaneously is all of these hyper-parameters are coupled and if we set a low value for some param we can set higher values of other params, so as to keep the total regularization in check."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to set the value\n",
    "This requires a grid search to determine the proper magnitude but usually does not require more than one significant figure accuracy. Use your knowledge of the dataset and architecture to decide which values to use. For example, a more complex dataset requires less regularization so test smaller weight decay values such as 1e-4, 1e-5, 1e-6. A shallow architecture requires more regularization so test larger weight decay values such as 1e-2, 1e-3, 1e-4. The reason being complex datasets provide regularization on their own and other regularizations should be reduced.\n",
    "\n",
    "So if you guess that 1e-4 should be a good value than test 3e-5, 1e-4, 3e-4. How I chose the value 3? So if you think your weight decay best value lies between 10<sup>-4</sup> and 10<sup>-3</sup>, than you should choose the value 10<sup>-3.5</sup> i.e take average of the exponent. You can keep going in this way.\n",
    "\n",
    "To make testing simpler I would use cyclic momentum as (0.99-0.90) and find the optimal value of weight decay. The reason I do not need to change the momentum value here is that momentum is already changing from 0.99 to 0.9 as we are using cyclic momentum, so in implementation we can first find a good cyclic momentum value and then test out weight decay values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.114937</td>\n",
       "      <td>4.064409</td>\n",
       "      <td>0.061100</td>\n",
       "      <td>0.235800</td>\n",
       "      <td>01:56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.662782</td>\n",
       "      <td>3.437547</td>\n",
       "      <td>0.168800</td>\n",
       "      <td>0.436900</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# WD = 0\n",
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy])\n",
    "learn.fit_one_cycle(2, max_lr=1e-2, moms=(0.99-0.90), wd=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.124258</td>\n",
       "      <td>3.914982</td>\n",
       "      <td>0.085900</td>\n",
       "      <td>0.281900</td>\n",
       "      <td>01:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.668026</td>\n",
       "      <td>3.459805</td>\n",
       "      <td>0.171700</td>\n",
       "      <td>0.436300</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# WD = 1e-4\n",
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy])\n",
    "learn.fit_one_cycle(2, max_lr=1e-2, moms=(0.99-0.90), wd=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.139386</td>\n",
       "      <td>4.032632</td>\n",
       "      <td>0.074400</td>\n",
       "      <td>0.247900</td>\n",
       "      <td>01:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.692349</td>\n",
       "      <td>3.504380</td>\n",
       "      <td>0.165300</td>\n",
       "      <td>0.419900</td>\n",
       "      <td>01:49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# WD = 1e-5\n",
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy])\n",
    "learn.fit_one_cycle(2, max_lr=1e-2, moms=(0.99-0.90), wd=1e-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.130610</td>\n",
       "      <td>4.117643</td>\n",
       "      <td>0.064100</td>\n",
       "      <td>0.235600</td>\n",
       "      <td>01:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.725400</td>\n",
       "      <td>3.531017</td>\n",
       "      <td>0.156800</td>\n",
       "      <td>0.414100</td>\n",
       "      <td>01:48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# WD = 1e-3\n",
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy])\n",
    "learn.fit_one_cycle(2, max_lr=1e-2, moms=(0.99-0.90), wd=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above results is clear that WD=1e-4 is the best. So now I test around 1e-4 with the same rule as explained above. So I take the average of -4 and -3 and I get -3.5, so the multiplier I choose is 10<sup>-3.5</sup> which is approx 3.12\\*10<sup>-3</sup>. The values I test are 3x10<sup>-4</sup> and 3x10<sup>-5</sup>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.128520</td>\n",
       "      <td>4.023188</td>\n",
       "      <td>0.069500</td>\n",
       "      <td>0.250500</td>\n",
       "      <td>01:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.689037</td>\n",
       "      <td>3.464480</td>\n",
       "      <td>0.162000</td>\n",
       "      <td>0.426000</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy])\n",
    "learn.fit_one_cycle(2, max_lr=1e-2, moms=(0.99-0.90), wd=3e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.116028</td>\n",
       "      <td>3.965793</td>\n",
       "      <td>0.077100</td>\n",
       "      <td>0.269000</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.712324</td>\n",
       "      <td>3.490986</td>\n",
       "      <td>0.166200</td>\n",
       "      <td>0.425000</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy])\n",
    "learn.fit_one_cycle(2, max_lr=1e-2, moms=(0.99-0.90), wd=3e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final weight decay value that I choose after seeing the above resuls is 1e-4. In this way you can test different hyperparameter values and see which performs the best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a final classifier model with the above param values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(data, Resnet(), metrics=[accuracy, top_k_accuracy], callback_fns=ShowGraph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>top_k_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.271368</td>\n",
       "      <td>4.100077</td>\n",
       "      <td>0.058000</td>\n",
       "      <td>0.230000</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.935396</td>\n",
       "      <td>4.278841</td>\n",
       "      <td>0.058900</td>\n",
       "      <td>0.215700</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.700406</td>\n",
       "      <td>3.707660</td>\n",
       "      <td>0.128600</td>\n",
       "      <td>0.364100</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.440132</td>\n",
       "      <td>3.624694</td>\n",
       "      <td>0.159000</td>\n",
       "      <td>0.405800</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.107491</td>\n",
       "      <td>3.175907</td>\n",
       "      <td>0.230000</td>\n",
       "      <td>0.519900</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.872592</td>\n",
       "      <td>2.738831</td>\n",
       "      <td>0.286900</td>\n",
       "      <td>0.611200</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.674556</td>\n",
       "      <td>2.516749</td>\n",
       "      <td>0.336100</td>\n",
       "      <td>0.666500</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.477309</td>\n",
       "      <td>2.270287</td>\n",
       "      <td>0.394800</td>\n",
       "      <td>0.717000</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.316169</td>\n",
       "      <td>2.268882</td>\n",
       "      <td>0.390300</td>\n",
       "      <td>0.728300</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.188307</td>\n",
       "      <td>1.973801</td>\n",
       "      <td>0.458700</td>\n",
       "      <td>0.774900</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.041943</td>\n",
       "      <td>1.839805</td>\n",
       "      <td>0.488200</td>\n",
       "      <td>0.798400</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.946529</td>\n",
       "      <td>1.751944</td>\n",
       "      <td>0.512200</td>\n",
       "      <td>0.814900</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.835809</td>\n",
       "      <td>1.701965</td>\n",
       "      <td>0.524300</td>\n",
       "      <td>0.822100</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.792115</td>\n",
       "      <td>1.664993</td>\n",
       "      <td>0.531100</td>\n",
       "      <td>0.830100</td>\n",
       "      <td>01:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.786060</td>\n",
       "      <td>1.660618</td>\n",
       "      <td>0.531600</td>\n",
       "      <td>0.830200</td>\n",
       "      <td>01:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3iUVaLH8e/JpIeENAKE0HvviCAIooCIHREVddUV6yK2Vdf1rl7XtdzVte3KomIDQQVUVAQVKSodpITeQiekkJBAElLO/eMNECANmGQm4fd5nnkyecvMOczwe96c9xRjrUVERLyXj6cLICIipVNQi4h4OQW1iIiXU1CLiHg5BbWIiJfzrYgXDQuPtC2aNamIlxYRqZaWL1+ebK2tVdy+CgnqnKAo5i9YRLB/hby8iEi1Y4zZUdK+Cmv6GPrmryzellJRLy8ict6okKCODQ/iUHYeN45bxLPT13IoO7ci3kZE5LxQIUEdFeLPL3/uz529G/PhggT6vDyHCYt2UFCgUZAiImfKVMQQ8m7dutlly5YBsGZ3On/9Op5Vu9Lo37IW/7qxE+HB/m5/TxGpunJzc9m9ezfZ2dmeLkqFCwwMJC4uDj8/v5O2G2OWW2u7FXdOhQc1gLWWTxbt4Plv1xETGsgzQ9swuF0dt7+viFRN27dvJzQ0lKioKIwxni5OhbHWkpKSQkZGBo0bNz5pX2lBXSn9qI0x3HZhI6bc2wtrLfdOWM6ni3dWxluLSBWQnZ1d7UManCyMioo6478cKnXAS8f64cx9vD/9Wtbir1+t4bOlCmsRcVT3kD7mbOpZ6SMT/X19eOeWrvRpXosnpq7h+zX7KrsIIiJVikeGkAf5u/jvrV3p0iCcRz5fRfyedE8UQ0QEgLS0NP7zn/+c8XlDhgwhLS2tAkp0Mo/N9RHo52LsrV2JCPbj7o+XceBQ9b/bKyLeqaSgzs/PL/W8GTNmEB4eXlHFOs6jkzLFhAby7u3dSDuSy50fLSX9iAbGiEjle/LJJ9m6dSudOnWie/fu9O/fn5tvvpn27dsDcM0119C1a1fatm3LuHHjjp/XqFEjkpOTSUhIoHXr1tx99920bduWgQMHkpWV5bbyeXwyjraxNfn3LZ2555Pl3DthOR/e2Z0AX5eniyUiHvLcN2tZt/eQW1+zTWwYf7uybYn7X3rpJeLj41m5ciVz587liiuuID4+/ngXuvHjxxMZGUlWVhbdu3fn+uuvJyoq6qTX2Lx5M5MmTeLdd99l+PDhTJ06lZEjR7ql/F4xzeklrWrzyrAOLNyWwsOfrSRfIxhFxIN69OhxUj/nN998k44dO9KzZ0927drF5s2bTzuncePGdOrUCYCuXbuSkJDgtvJ4/Ir6mGs7x5GSeZS/f7ee8OB4Xrim3XnTXUdETijtyreyhISEHH8+d+5cfvrpJxYuXEhwcDD9+vUrth90QEDA8ecul6t6NX0U9cc+TUjOPMrYeVuJDvHnkYEtPV0kETkPhIaGkpGRUey+9PR0IiIiCA4OZsOGDSxatKiSS+dlQQ3wxOCWHDx8lDd/3kJEiD939G5c9kkiIucgKiqK3r17065dO4KCgqhdu/bxfYMHD2bs2LF06NCBli1b0rNnz0ovX6XM9XGm8vILuH/iCn5Yl8gbIzpxdad6biydiHib9evX07p1a08Xo9IUV1+Pz/VxpnxdPrx5U2d6Nonkkc9X8ez0taRnqeueiJyfvDKowRkQ8+5t3ejVNIoPFyQwfOxCkjJy3PsmBxNg3isw62nIz3Pva4uIuInXtVEXFRroxyd3XcAvm5O468NljBi3kCn39iIi5Bzms85Oh3Vfw6rJsOO3E9vzsmHIP0E9TUTEy3jtFXVRfZrX4uO7erArNYtr/vMbyZlneGWdnwebf4Ipd8E/W8D0P0FmIlzyDIxZA70fgqXvweL/VkwFRETOgVdfURfVs0kUH9zRnTs/XMqt7y/h0z9eUPaVdeJaWPkprPnCCebAcOg8EjreBPW6nrh6HvAspGyFWU9BZGNoMajC6yMiUl5V4or6mN7Nonnv9m5sTcpk5PuLi58bJPMALPw3jL0I3ukFi8dCvW4w/BN4bBNc8SrEdTu5icPHB64bB3U6wJQ7Yf+ayquUiEgZqlRQg9MMMu7WrmxOLAzrrFzIzYb4aTBxOLzaCmb9BXx84fL/g0c3wU2fQpurwDeg5Bf2D4GbJkNgTfj0RsjYX3mVEpEqp0aNGgDs3buXYcOGFXtMv379OJeuysdUuaAG6Ncyhndu6Yz/3iV898Jwcl9pBlPucK6Ee4+G+xfDqLlwwSgIiSrr5U4Iq+uEdVYaTBoBR49UVBVEpJqIjY1lypQpFfoeVaaN+riDCbDqMwasmsSAgO0csQFMz+5OZothXD/sZmoElXLVXB51O8Cw92HSTfDlKLjhY6dpRESqtSeeeIKGDRty//33A/Dss89ijGH+/PkcPHiQ3Nxc/v73v3P11VefdF5CQgJDhw4lPj6erKws7rjjDtatW0fr1q3dNt9H1QjqggJYOQFWToKdCwADjfvAxU9wuP5AZn63nR/XJjIhaRFvjOhMm9iwc3u/lpfDoH84NxdnPweXPeeWaohIOXz/pPvvE9VpD5e/VOohI0aMYMyYMceD+vPPP2fmzJk8/PDDhIWFkZycTM+ePbnqqqtKnDDunXfeITg4mNWrV7N69Wq6dOniluJXjaA2BhaNhfyjMOB/oP1wCK8PQC3g3duimLcpiQcnruDa//zGM0Pb0DAqmAubROHrOsur4Z73Qcpm+O11iGoGXW51X31ExOt07tyZAwcOsHfvXpKSkoiIiKBu3bo8/PDDzJ8/Hx8fH/bs2UNiYiJ16tQp9jXmz5/P6NGjAejQoQMdOnRwS9mqTlDfPh2Co0ockHJxi1rMebwfD0xcwV+/igfggsaRvHVTZ2LCAs/uPS9/xWlq+XYMRDSExn3PoRIiUi5lXPlWpGHDhjFlyhT279/PiBEjmDhxIklJSSxfvhw/Pz8aNWpU7BSnRVXE9Mzlvtw0xriMMb8bY751eynKIyS6zFGD0TUCmPDHC3h8UEuu6RTLqt1pXPn2r0xespPs3NLXPiuWyw9u+NC5ov7sVkg+fbJwEak+RowYweTJk5kyZQrDhg0jPT2dmJgY/Pz8mDNnDjt27Cj1/L59+zJx4kQA4uPjWb16tVvKdSbtAg8B693yrhXIz+XDA/2b8fqIzky7rzf5BfDktDV0//tPzFiz78xfMLAm3PyZ091v4g1wJNX9hRYRr9C2bVsyMjKoV68edevW5ZZbbmHZsmV069aNiRMn0qpVq1LPv++++8jMzKRDhw688sor9OjRwy3lKtc0p8aYOOAj4AXgEWvt0NKOP9dpTt0p62g+38fv46MFCazanc51Xerx5OWtiAk9w+aQXUvgw6HOiMbbviq9T7aInBFNc+qeaU5fB/4MFJR0gDFmlDFmmTFmWVJSUnnLW+GC/F1c1yWOz+65kGFd45i2Yg8X/GM2t76/mA37z2ABzfo94Jr/OL1OvnkIKmAebxGR4pQZ1MaYocABa+3y0o6z1o6z1naz1narVauW2wroLoF+Lv55Q0e+/dNF9G8Zwy+bkxn8+i+Mmfw7WUfL2X7dfhj0fxpWTYJfXq3YAouIFCpPr4/ewFXGmCFAIBBmjJlgrXXPOuiVrF29moz/Q3fSjhzl5ZkbmbRkJ3M3JXFZ69r89Yo21Az2K/0F+j4OKVvg5+chsgm0u65yCi5SzVlrz4sFrc9mVa0yr6ittU9Za+OstY2AEcDPVTWkiwoP9ufF69rz3m3daFarBl8s303H//2Br1fuKf1EY+Cqt6DBhfDVfbDbO9riRaqywMBAUlJSzirEqhJrLSkpKQQGntk9sjNaM9EY0w94rCrdTCyv+ZuS+J+v40lIceb3GNK+Dn+/pj2RJU2lejgF3rsEjh6GP852+lmLyFnJzc1l9+7dZfZRrg4CAwOJi4vDz+/kv95Lu5nolYvbekp2bj4PfrqCn9YfACA00Jc3b+pM/5YxxZ+QtAnevxTC6sGdsyDwHIeui8h5q8otbuspgX4uxo7syod3dOfrB3oT4OvDXR8u5Y4PlrB2b/rpJ9RqAcM/huRNzux9WndRRCqAgvoUvi4f+rWMoWP9cL68vzdXdYxlzsYkrnjzVx6a/Ds7Ug6ffEKTfnDFa7DlJ5j5pCeKLCLVnIK6FPUjg3l9RGdmjO5Dk+gQvl65l0tfm8c/ZqxnV2qRuaq73g69RsPSd7Xuooi4ndqoz8DOlCM8/dUaftmcTKCfDz0aR/H0kNa0rBPqTMX6+a2wcYaz+IDWXRSRM6CbiW5krWVr0mFe+n4DP61PBKB3syjeHNGZKP88+OByZ6HcO2dBnXYeLq2IVBUK6gpyICOb4WMXHu/S17puGNc29WHEqtsJ9PfD/945EFr8vLUiIkWp10cFiQkNZM5j/fji3gsJDfBl/b5D/OPXNEZkPkxuZgrJ714Lm3+E9N2aG0REzpquqN0o8VA2R47m4zKGrz9/j1H7nyPAFHbZCwiDmNbOo1bhz5g2UMP75kURkcqnpg8PyMnLZ/QHc0nZtpIBUSnc1CiT8IytcGAdZBWZ0zo4+kSAHwvvWq0gKNxzhReRSqeg9pD8Asuz09fyyaId+PoYXri2HcO6xOHKSnYC+8D6kx9HM06cHBp7SoC3dgLcP8RzFRKRCqOg9rAFW5J56ss17Eg5QpNaIVzXuR5DO8SSmZNHu3o1nYOsddqykzYUCfF1kLQR8orMfxDRyGk66X4XNL/MI/UREfdTUHuB3PwCPvhtO5OW7GJ78onRjdd1rseYS1vQICq4+BML8p0Fdo9fea9zZuw7tBuuehs631I5FRCRCqWg9jLLElJZtC2F33emMXuDMwHU4LZ1eO3GjgT7l2OK8JxM+GwkbJsDA1+AXg9WcIlFpKIpqL3Y0oRU/jB+CYeP5hMXEcQr13egV7Posk/My4Fpo2DdV3DRIzDgf8pcpV1EvJf6UXux7o0iWfW3gYy+pBl707K4+b3FvDhjfdkTqPsGwLDx0PUO+PU1+HaM00wiItWOgtoL+Lp8eGRgS3594hIub1eH/87fxuDXf2H+pjIWCfZxwdB/QZ9HYfmHMOVO50pbRKoVNX14mWNd+iYt2UlegaVhVDCD29ZhZM+G1I8s4YYjwIK34YenoUl/uHECBNSovEKLyDlTG3UVlHU0n1dmbWDJ9lQ27s8gr8DSr2UtXhveqeTlwX6fCNMfhNgucMsXEBxZuYUWkbOmoK7iEpIP88jnK1mxM42wwuXBLm5Rq/gVm9d/6zSBRDaGW7+EsNjKL7CInDHdTKziGkWHMO3+3oy7tSuHsvP4wwdLuW/CCnLzC04/uPVQGDkF0vfA+EHOlKsiUqUpqKuQgW3r8Pszl9G6bhgz1+7nyrd+ZWlC6ukHNu4Lf/jGWSF9/CDYt7ryCysibqOgrmIiQvyZMfoiXr+xEztTj3DD2IVMWLSD7NxTuubFdnYWL3AFwIdXwI4FnimwiJwztVFXYQcPH6Xz8z8CEBroy3Wd6/HooJaEBfqdOCh9N3xyLaTthBs+gpaDPVRaESmN2qirqYgQf+KfG8QD/ZuSl2/5aOEObn53EQcPHz1xUM04uGOmM/ve5Jth1WeeK7CInBUFdRVXI8CXxwe1YsUzlzF6QHPi9xziijd/4bvV+04cFBIFt38DjXrDl6Ng0VjPFVhEzpiCupoI8nfxyGUtmP5gb1wuwwOfruDJqatPDEUPCIWbv4BWQ2HmEzDnH1oeTKSKUFBXMx3iwpk1pi9dG0Yweekunv92PelHcp2dfoFOO3XnkTDvZZjxOBQU08VPRLyKgroaCvb3ZdLdPenTPJrxv23nsn/N40BG4eIDLl9nHutef4Kl78K0uyE/17MFFpFSKairKX9fHz6+swdPXd6KAxk5XPjiz/y4LtHZaQwM/Dtc+izET4FJN8HRI54sroiUQkFdjRljuOfipky970KstYz6ZBlPTVvDvvQs54CLHoYr34Cts50ufFlpni2wiBRLQX0e6Nowkllj+tI2NoxJS3Zy4Ys/c/v4JaRk5kDXP8CwD2DvCmdgTMZ+TxdXRE6hAS/nEWsta/ce4vv4fbw7fztB/i7+0KsRowc0x7V9Lky+BWrUgpHTIKqpp4srcl7RgBcBnKaQdvVq8vigVnx2T0+C/Fy8MXsz93yynJyGfZ2+1jkZ8P5A2LPC08UVkUIK6vNU5wYR/PBIX+7o3Yif1ifS8q8zGTHjKIdHzgD/YPhwKGyZ7eliiggK6vNaWKAff7uyLW+M6ERYoC+LtqXS9s2tbLhiKkQ2gU+Hw+rPPV1MkfOeglq4ulM9Vj87iFdv6AjA4Pc387fIV8ivf6HTz3rBWx4uocj5TUEtx13fNY7FfxlAaKAvH/1+kBsyHyWt8VD44a8w62mNYhTxkDKD2hgTaIxZYoxZZYxZa4x5rjIKJp5ROyyQNYVX1xuScui8fgRzal4LC98md8ofIe9o2S8iIm5VnivqHOASa21HoBMw2BjTs2KLJZ52fdc4vnqgN90bR3NH4jBezh2B37qpbPrX5ezad8DTxRM5r5QZ1NaRWfirX+FD066dB1rUDuXzey7k6wcuIrD/Y/zNPECTzBWkjx3EdwtWcig7lz1pWWRka64QkYpUrgEvxhgXsBxoBvzbWvtEMceMAkYBNGjQoOuOHTvcXFTxtMycPLYtmEaLeQ+yvyCc23KfZKetTVigL/93Q0cGta3j6SKKVFmlDXg5o5GJxphw4EvgT9ba+JKO08jE6u3ItkXkfXID+RgeC3iG2WmxAPRsEsm427qdvBSYiJSL20YmWmvTgLmAFt47jwU36UnYAz8TEVaT9wueZeUtvvRqGsWibal0ePYHbn53ER/8tv30BXdF5KyUp9dHrcIraYwxQcClwIaKLph4uejmcNcPENGI8C9v5tMLdvLy9e0J8XexYGsKz32zjlbPzOSLZbs8XVKRKq88V9R1gTnGmNXAUuBHa+23FVssqRLC6sIdM6CBMzDmxtzprP3fwfzy5/7ce3FTAnx9+PPU1Qx+fT6LtqV4urQiVZZmz5Nzl5cD00bBuq/gwgfhsufBx4fDOXk8PmUVM9Y4U6cOaV+Hl6/vQKjasEVOU1obtW9lF0aqId8AGDYeZsbAwrch8wBc/W9CAvz5981dSDl8lBdnbGDqit3M35TMhD9eQLOYGtQI0NdPpDx0RS3uYy388ir8/Dw0vQSGf+ysfl7oq9/3MOazlSed8vigltzdpwn+vprNQM5vbuueV14K6vPc7xNg+mio0x5umeIsRlAo9fBRPl6YwOs/bT6+rWaQH2/d1JnIEH/a1avpgQKLeJ6CWirfplnw+e0QWgduneZMm3qK7Nx83v55C5OX7iI5MweA8GA/hnerz20XNiQuIriySy3iMQpq8YxdS+HTG8DH17myju1U7GHJmTlMXb6bjYkZTFux5/j2T/94Ab2aRVdWaUU8SkEtnpO0CSZcD1mp0PN+6DHqpKaQU1lrmbU2kXsnLAfgyo6x3NO3iZpEpNpTUItnHdoHMx6DDd+CbyB0utnpxlfKArpJGTmM+ex3ftvi9L9uWiuElnVCue/iZrSPU2hL9aOgFu+QtAkWvgWrJkN+LrQeCr0egvrdSzxlZ8oR3p6zmc+X7T6+rXlMDS5pHcO1netROzSQiBD/yii9SIVSUIt3yUiEJf+Fpe9BdrozsrHXaGgxGHxK7qa3Ny2LF7/fwDer9p60/ZYLGjB6QHNqhwVWdMlFKoyCWrxTTib8/gks/Dek74LoFtDrT9DhRmcQTQny8gv4aX0iG/ZnsHhbKgsLh6f3bVGL/47sSpC/q7JqIOI2Cmrxbvm5sPYrWPAG7F8DNWrDBfdAtzshKKLM0xdtS2HEuEXHfx/Svg6v39hZg2ikSlFQS9VgLWybCwvehK0/g18IdL3d6S0SXr/UUwsKLPM3J/GPGevZlOgsSFQrNICRFzSkT4toOtcPxxhTCZUQOTsKaql69q+BBW9B/FQnwNtd57Rj1+1Q6mnWWr5auYfJS3axeHvqSfu6NAjnoUtbcHGLkrsHiniKglqqrvTdsOgdWP4hHM2EJv2h92jnZzmukJdsT+WpaavZmnT4+LbrutTjyo6x9GtRS1fZ4jUU1FL1ZaXBsvGweCxkJjrziPQaDW2vBVfZ06bm5OUTv+cQf5seT/yeQwB0iKvJ1Pt64etjFNjicQpqqT7ycmD1506zSPJGqFnfCeyut5faU6SoAxnZ/HnKauZuTAKc+UX8XT4MaF2bBy9pRr3woIqsgUixFNRS/RQUwOYf4LfXYedCCKsHfR6FzreCb9kDYKy1/POHjSzZnoqvj8/xLn7gdPN74Zp2xEUE6UpbKo2CWqova2H7PPj5Bdi9BGo2gL6POcPUy9EkcuJlLHM3JvH0l2vYm54NQLC/i1lj+lI/UrP4ScVTUEv1Zy1snQ1z/gF7lkNEI+j7Z2fwjOvMVpKZsnw301ftZf6mJIL9XdStGchFzaIZ1K4OreuEaci6VAgFtZw/rHWaROa8APtWOfNgX/wktB8GPmc2YnH17jTGfLaSbUV6jAC0rB3KtV3q0bNJFJ3qh7uz9HIeU1DL+cda2DgD5rwIiWuc4ekXPwFtryt1PpHiZObk8dS0NfRpFs2q3Wl8u3of6Vm5uHwMN/Woz3Vd4ugUF46Pj9qz5ewpqOX8VVAAG75xAjtpPdRqDf2ehNZXnXFgH2OtZW96Ni/OWM+3q/cB0CAymLAgX9KzcunfMoahHWLp0TjSnTWRak5BLVJQAOu+hLkvQfImqN0O+j0Fra4o18CZkqRk5jBtxR6+WL6LfenZZGTnHd/n5zI0iwmlb/NoRvZsqF4kUioFtcgxBfnOsPS5L0HqVqjTAfo/DS0GnVNgH5OelctvW5L5ZtVe5m5MIis3/6T9o/o24S9DWp/z+0j1o6AWOVV+Hqz5HOa9DAcTILaLE9jNBrglsItavuMg43/dzndrnGaSZjE1+PjOHsRqYI0UoaAWKUl+LqyaBPP+D9J3QlwP6P9UuecSORO5+QXc8u5iliQ4k0U9PaQ1A9vWpkFksJpEREEtUqa8o7ByAsx/FQ7tdlad6XIbuPwLA7swSI89Px6sppT9p2+z1jJjUyaPLw7kSK7zf69H40heG96RuAgNrDmfKahFyisvB1Z8DL+8Chn7KuxtbP2efBYzhg+2hLAj9TDZuQUM7xbH0A6xxIQFsCs1i+U7DrJqVxo+PjCgVW0ub1+HujXVXFJdKahFzlReDhzcAVinTzYUeV7CNihhPydvS4yHn593ZgS84F4SOoxm6H9XkZlzosdISZpEh3Bxy1r8ZUhr/FxawaY6UVCLeJsjqTD7f515tmvUJm/gC8yyvUhIPUKQn4u5m5KoGeTHoMI27C+W7WZ78mF+3ZIMQO2wAP52ZVuGtK/r2XqI2yioRbzV7uXw3cPOcPcm/WDIPyG6eYmHH8rO5evf9zD+twS2Jx9mQKsYwoL8qBHgS6CfDzd2r0+zmNBKK764j4JaxJsV5DuLIsx+HnKPQO+HnClb/Uu+uXgoO5dHP1/FT+sTKfpfOMTfxU09GtCvZQy9m0WpN0kVoqAWqQoyD8APz8DqyRDeAC5/BVpeXuopGdm5BPi6mL0+kT1pWbz/63b2FU7T2iQ6hDdv6ky7ejUro/RyjhTUIlVJwq/w3aOQtAFaDoHBL0FEw3Kdmp2bz9KEVH5Ym8iXv+85foMyKsSfyBB/3hjRmTaxYRVZejlLCmqRqiY/Fxb9B+a+DLbAWQyh15/KvdwYQHJmDo99sYq5G5MIDfAlo0ivktZ1w2haK4ToGgG0qRvGsK5xmv3PwxTUIlVV+m6Y+RSsnw5RzZybjU37n9VLLd6WwiuzNhIa6MuBQzlsTMwgv8D5/x9dI4Am0SEMaB3DZW1q0zg6RO3blUxBLVLVbf4JZjwGB7dDu+th4AsQdu5d8/akZTFvYxIz1+5n/qak49uja/jTv2UMcRHB9GoWRfdGmrK1op1TUBtj6gMfA3WAAmCctfaN0s5RUItUgNxsZzHfX15zhrb3/wv0GHXGS42VJD0rl4Tkw7z363Z2pR5h5a60k/bf3acxwf6+tKgdyhUd1H/b3c41qOsCda21K4wxocBy4Bpr7bqSzlFQi1Sg1G0w48+w5UdnXu0rXoUGPd3+NtZaJizawdh529iTlnXSvi4Nwnno0hY0iQ7R4r9u4tamD2PM18Db1tofSzpGQS1SwayFDd/C9086k0h1GgmXPQch0RXydkeO5rE/PRt/Xx+emLqa37akHN/XrWEE797WTYv+niO3BbUxphEwH2hnrT10yr5RwCiABg0adN2xY8fZlldEyuvoYZj3Cix8G/xrQJdboekAZ/Y/v8AKe9udKUeYuGQHi7elHm8i6dM8mk71w2kWU4PB7eqwKzWL6Sv3cF2XOOrUDMTf5XO8Z8mx3NENyxPcEtTGmBrAPOAFa+200o7VFbVIJTuwAX78H9j6MxTkgm8QNOrthHazAc7ivhUUil+v3MPi7al8unhnqcfVDgsg8VDO8d8bRQVzVcdY0rJyyckt4C9XtKZmkF+FlLEqOOegNsb4Ad8Cs6y1r5V1vIJaxENyMmHHb7BlNmydDSlbnO1hcU63vmYDoPHFEOz+XhwHMrJZnnCQqSt2s/tgFv1bxWCtM3py/uYkAn1dbD6Qefz41nXDWL/vxB/mLWrX4MM7zt+Vb871ZqIBPgJSrbVjyvOGCmoRL3Fwh3OVvXU2bJsPOelgfJylx5oNgKaXQL1ubus5cibyCyxfLNtF3fAg/HwM93yynIycPJrF1GB/ejadG4TTODqE0EBfwgL9GNY1jqga5R/wU9Wca1BfBPwCrMHpngfwF2vtjJLOUVCLeKH8PNiz3AntLbNh7wpn1GNATWjS12kmaXpJuYeru9umxAxGT/qdDfszSjymSXQID13anEFt6wDgYwz+vtVjXm4NeBGR0x1Jhe3zCptJfoZDe5ztUc1OhHajiyCgRqUWa8uBDHLyCgjycxHg58LPZViwJYUxn6087Vh/lw8d69ekXngQfi4fruwYS6Cfi8k3DGoAAAnSSURBVB6Nq94AHQW1iJTOWkjedCK0E36FvCzw8XP6aDfqA3XaQe22EN6wwm5MliY7N5/5m5JIyswhfk86Gdl5zF5/gKzc/NOOvbpTLI9e1pLfdx3k29X7uLpTLCH+voQG+tKuXk0C/VyVXv6yKKhF5MzkZsPOhYXt2z9D4lqOLzfmH+oEdu22heHdDmJaQ0DlL1hgrcUYQ/qRXD5amMC0FbsJ8vc96SZlSfo0jyY82J+UzBxeuq4D9SKcm5guH0NBgSUjJ4/QAN9Km6xKQS0i5yYn05l2NTEe9sc7wZ241rk5eUxE48IAb1fk6rsR+FR+G/KKnQeZvGQnvi4f7ru4Ke/M20r7ejXxd/mwYudBJpbSlTAyxB8/lyHxUA4h/i4evqwFf+jViLSsXD5buouIYH/Cg/1oFBWCjw8YDE1qhZzRGpYHDmUzYfFO1u87RK3QAAa0iuHSNnUU1CLiZtZC+i4nsPfHOyGeuLawS+Cxq+8aENPmlKvvNhDo+Tmxs3Pz8TGGfelZfLJwB+/9uv34vo5xNcm3lkNZeexMPVKu12tZO5S4iCDu6tOY+hHBvPXzZnLzLe3q1SQ3v4AVOw6yI+UImTl5pw3JB9jx8lAFtYhUkqNHIGl9kQBfC4lrILvI1Xd4wxNNJjXrQWgshNaB0LoQUssjV+HFsdYyM34/901cQbt6YdzYvQENIoMJ9PVh/b5DuFw+HMrK5b/ztnIou+xV5I8Z2qEu3RtFclXHWGau3U9KZg5/GtBCQS0iHmSt06skcS3sX3Oi6SRls9NFsCgfX6hR+0Rwh9Z1nofFnrwtsKZHbmqWJL/AsikxgzV70vl1czJXd4qlZZ1QflibSESIHwPb1MECwX6uYtu91UYtIt4pPw8OH4CMfXBon/MzYx9k7D95W3ba6ef6BhUf4EWvzIMjISgCfLyvl8epSgvqyh+OJCJyjMvXCdqwWKhXynG5WcUH+LHf9/4Oh2Y4XQpPY5yr7+AoJ7iDoyAosvB50d+L7o8Al/fMO6KgFhHv5xcEkU2cR0msddrBj4X3kRRnUM+RFMhKPfH7ob1O23lWKuSWcqMwoCYER5we4v4hYFzOUHzj47SnH3t+0naX0zRT7Haf0x+lUFCLSPVgDASFO4+YVuU7Jzer+DA/knry74eTIHmj8/zoYY73aqkkCmoROX/5BTm9TmqW1u5SDGsLH/nOzVBbAAVFnhd9nLT92HN7+vbnOpT4dgpqEZEzZUxhj5PK6UboHZ0VRUSkRApqEREvp6AWEfFyCmoRES+noBYR8XIKahERL6egFhHxcgpqEREvp6AWEfFyCmoRES+noBYR8XIKahERL6egFhHxcgpqEREvp6AWEfFyCmoRES+noBYR8XIKahERL6egFhHxcgpqEREvp6AWEfFyCmoRES+noBYR8XIKahERL6egFhHxcmUGtTFmvDHmgDEmvjIKJCIiJyvPFfWHwOAKLoeIiJSgzKC21s4HUiuhLCIiUgy3tVEbY0YZY5YZY5YlJSW562VFRM57bgtqa+04a203a223WrVquetlRUTOe+r1ISLi5RTUIiJerjzd8yYBC4GWxpjdxpi7Kr5YIiJyjG9ZB1hrb6qMgoiISPHU9CEi4uUU1CIiXk5BLSLi5RTUIiJeTkEtIuLlFNQiIl5OQS0i4uUU1CIiXk5BLSLi5RTUIiJeTkEtIuLlFNQiIl5OQS0i4uUU1CIiXk5BLSLi5RTUIiJeTkEtIuLlFNQiIl5OQS0i4uUU1CIiXk5BLSLi5RTUIiJeTkEtIuLlFNQiIl5OQS0i4uUU1CIiXk5BLSLi5RTUIiJeTkEtIuLlFNQiIl5OQS0i4uUU1CIiXk5BLSLi5RTUIiJeTkEtIuLlFNQiIl5OQS0i4uXKFdTGmMHGmI3GmC3GmCcrulAiInJCmUFtjHEB/wYuB9oANxlj2lR0wURExFGeK+oewBZr7TZr7VFgMnB1xRZLRESO8S3HMfWAXUV+3w1ccOpBxphRwKjCX3OMMfHnXjyvFg0ke7oQFUx1rB7OhzpC1a9nw5J2lCeoTTHb7GkbrB0HjAMwxiyz1nYrd/GqINWxelAdq4/qXM/yNH3sBuoX+T0O2FsxxRERkVOVJ6iXAs2NMY2NMf7ACGB6xRZLRESOKbPpw1qbZ4x5EJgFuIDx1tq1ZZw2zh2F83KqY/WgOlYf1baextrTmptFRMSLaGSiiIiXU1CLiHg5twZ1dRpqboxJMMasMcasNMYsK9wWaYz50RizufBnROF2Y4x5s7Deq40xXTxb+pIZY8YbYw4U7ed+NvUyxtxeePxmY8ztnqhLSUqo47PGmD2Fn+dKY8yQIvueKqzjRmPMoCLbvfb7bIypb4yZY4xZb4xZa4x5qHB7tfksS6ljtfosy8Va65YHzo3GrUATwB9YBbRx1+tX9gNIAKJP2fYK8GTh8yeBlwufDwG+x+lz3hNY7Onyl1KvvkAXIP5s6wVEAtsKf0YUPo/wdN3KqOOzwGPFHNum8LsaADQu/A67vP37DNQFuhQ+DwU2Fdal2nyWpdSxWn2W5Xm484r6fBhqfjXwUeHzj4Brimz/2DoWAeHGmLqeKGBZrLXzgdRTNp9pvQYBP1prU621B4EfgcEVX/ryKaGOJbkamGytzbHWbge24HyXvfr7bK3dZ61dUfg8A1iPM4q42nyWpdSxJFXysywPdwZ1cUPNS/tH9XYW+MEYs7xweDxAbWvtPnC+REBM4faqXvczrVdVre+DhX/2jz/WJEA1qKMxphHQGVhMNf0sT6kjVNPPsiTuDOpyDTWvQnpba7vgzBr4gDGmbynHVre6H1NSvapifd8BmgKdgH3Aq4Xbq3QdjTE1gKnAGGvtodIOLWZblahnMXWslp9ladwZ1NVqqLm1dm/hzwPAlzh/PiUea9Io/Hmg8PCqXvczrVeVq6+1NtFam2+tLQDexfk8oQrX0RjjhxNgE6210wo3V6vPsrg6VsfPsizuDOpqM9TcGBNijAk99hwYCMTj1OfYXfHbga8Ln08Hbiu8s94TSD/252cVcab1mgUMNMZEFP7ZObBwm9c65Z7BtTifJzh1HGGMCTDGNAaaA0vw8u+zMcYA7wPrrbWvFdlVbT7LkupY3T7LcnHzXdohOHdmtwJPe/pO6TnUownOneFVwNpjdQGigNnA5sKfkYXbDc7iCluBNUA3T9ehlLpNwvlzMRfnSuOus6kXcCfOzZotwB2erlc56vhJYR1W4/wnrVvk+KcL67gRuLwqfJ+Bi3D+fF8NrCx8DKlOn2UpdaxWn2V5HhpCLiLi5TQyUUTEyymoRUS8nIJaRMTLKahFRLycglpExMspqEVEvJyCWkTEy/0/+BuxL6OYml8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(15, max_lr=1e-2, moms=(0.99-0.9), wd=1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations you made it to the end. You can now set any hyper-parameter value by just visualizing the validation loss for a few epochs and seeing whether our models overfit or not. The content of this notebook has been taken from the four papers by Lesli N. Smith as mentioned in the starting of the notebook and from the fastai courses taught by Jeremy Howard.\n",
    "\n",
    "I regularly write about new research being published either or [Medium](https://medium.com/@kushajreal) or as I first tried on Jupyter notebooks. You can check my homepage where I list all of my content that I publish https://kushajveersingh.github.io/ (I am still updating the website so if the homepage is not working, just give me a few days of time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}