{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The TrainPhase API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*This notebook was prepared by Sylvain Gugger - many thanks!*\n",
    "\n",
    "Here we show how to use a new API in the fastai library, that allows you all the flexibility you might want while training your model.\n",
    "\n",
    "All the examples will run on cifar10, so be sure to change the path to a directory that contains this dataset, with the usual hierarchy (a train and a valid folder, each of them containing ten subdirectories for each class)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.conv_learner import *\n",
    "PATH = Path(\"../data/cifar10/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')\n",
    "stats = (np.array([ 0.4914 ,  0.48216,  0.44653]), np.array([ 0.24703,  0.24349,  0.26159]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will allow us to grab data for a given image size and batch size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(sz,bs):\n",
    "    tfms = tfms_from_stats(stats, sz, aug_tfms=[RandomFlip()], pad=sz//8)\n",
    "    return ImageClassifierData.from_paths(PATH, val_name='test', tfms=tfms, bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "size = 32\n",
    "batch_size = 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_data(size,batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a very simple model that we'll train: a neural net with a hidden layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def SimpleNet(layers):\n",
    "    list_layers=[Flatten()]\n",
    "    for i in range(len(layers)-1):\n",
    "        list_layers.append(nn.Linear(layers[i], layers[i + 1]))\n",
    "        if i < len(layers)-2: list_layers.append(nn.ReLU(inplace=True))\n",
    "        else: list_layers.append(nn.LogSoftmax(dim=0))\n",
    "    return nn.Sequential(*list_layers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.from_model_data(SimpleNet([32*32*3, 40,10]), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use our learner object to give examples of traning.\n",
    "\n",
    "With the new API, you don't use a pre-implemented training schedule but you can design your own with object called TrainingPhase. A training phase is just a class that will record all the parameters you want to apply during this part of the training loop, specifically:\n",
    "- a number of epochs for which these settings will be valid (can be a float)\n",
    "- an optimizer function (SGD, RMSProp, Adam...)\n",
    "- a learning rate (or array of lrs) or a range of learning rates (or array of lrs) if you want to change the lr.\n",
    "- a learning rate decay method (that will explain how you want to change the lr)\n",
    "- a momentum (which will beta1 if you're using Adam), or a range of momentums if you want to change it\n",
    "- a momentum decay method (that will explain how you want to change the momentum, if applicable)\n",
    "- optionally a weight decay (or array of wds)\n",
    "- optionally a beta parameter (which is the RMSProp alpha or the Adam beta2, if you want another vlaue than default)\n",
    "\n",
    "By combining those blocks as you wish, you can implement pretty much any method of training you could think of."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic lr decay"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin with something basic and say you want to train with SGD and momentum, with a learning rate of 1e-2 for 1 epoch then 1e-3 for two epochs. We'll just create a list of two phases for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-2), TrainingPhase(epochs=2, opt_fn=optim.SGD, lr = 1e-3)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we didn't set the momentum parameter because it will default to 0.9. If you don't want any momentum, you'll have to put it to 0.\n",
    "\n",
    "Now that we have created this list of phases, we just have to call fit_opt_sched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "42579b93c5cf45ada3112d3406e991db",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.406514   4.553418   0.1387    \n",
      "    1      3.396738   4.559127   0.1383                                                                                \n",
      "    2      3.421531   4.541069   0.138                                                                                 \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.54107]), 0.138]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to see what we did, we can use learn.sched.plot_lr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuAAAAEKCAYAAABT6eBwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm8HGWV+P/PIRBCSNiSyJIACchikNWIAiKM7IiCChIVBeU3cV8YHQfUUYZRUQcVx4WfKDuMEINocIIiSMRhD0jCJhJRhhCUYDJsIZCE8/2jKtjc3KXvze2u7tuf9+tVr65+uqr7PLnVxaH6qedEZiJJkiSpOdaqOgBJkiSpk5iAS5IkSU1kAi5JkiQ1kQm4JEmS1EQm4JIkSVITmYBLkiRJTWQCLkmSJDWRCbgkSZLURCbgkiRJUhOtXXUAzTB27NicOHFi1WFIUr/dfvvtj2fmuKrjaCbP2ZLaVb3n7I5IwCdOnMicOXOqDkOS+i0iHqo6hmbznC2pXdV7znYIiiRJktREJuCSJElSE5mAS5IkSU1kAi5JkiQ1kQm4JEmS1EQm4JIkSVITmYBLkiRJTWQCLkmSJDWRCbgkSZLURCbgag3XXFMskqTW5zlbWiMdUYpebeCLXyweDzyw2jgkSX3znC2tEa+AS5IkSU1kAi5JkiQ1kQm4JEmS1EQNTcAj4tCIuD8i5kfEyd28vm5EXFa+fktETCzbx0TEdRHxdER8p8s+r4qIu8p9/jMiopF9kCRJkgZTw27CjIhhwHeBg4AFwG0RMTMz763Z7ERgSWa+PCKmAl8FjgWWAf8KvLJcap0FTANuBmYBhwJXNaofapLvf7/qCCRJ9fKcLa2RRs6CsicwPzMfBIiIS4EjgdoE/Ejg1HJ9BvCdiIjMfAb4n4h4ee0bRsTmwAaZeVP5/ELgKAY5Ab/v0Sf5+byFg/mW6sN66wzjhH0mMarqQCRJfdthh6ojkNpaIxPw8cDDNc8XAK/paZvMXBERTwBjgMd7ec8FXd5zfHcbRsQ0iivlbLXVVv0K/I+Lnub7v3mwX/to4BLY/w83s89927H7h99ddTiSpL5ceWXx+KY3VRuH1KYamYB3NzY7B7DNgLbPzLOBswGmTJnS23uu5ohdtuCIXbbozy5aA/csfIKnXvsvbP3QaDABl6TW9/WvF48m4NKANPImzAXAljXPJwBdx3W8uE1ErA1sCCzu4z0n9PGekiRJUstqZAJ+G7BdREyKiOHAVGBml21mAseX60cDv87MHq9WZ+ajwFMR8dpy9pP3AD8b/NAlSZKkxmjYEJRyTPdHgF8Cw4BzM/OeiDgNmJOZM4FzgIsiYj7Fle+pq/aPiD8DGwDDI+Io4OByBpUPAucD61HcfOkMKJIkSWobjRwDTmbOopgqsLbt8zXry4Bjeth3Yg/tc1h9akJJkiSpLTQ0AZfqddIRn+T0t+7M/lUHIknq20UXVR2B1NYsRa/KBcGjG4xj2WbdzigpqWJ1VDXeOiKujYh5ETE7IiZ0eX2DiHika2VjtbEttywWSQNiAq6WcMR917PZL7yfVmo1NVWNDwMmA++IiMldNjsDuDAzdwFOA07v8vq/A79pdKxqossuKxZJA2ICrpZw3O9msdVlF1QdhqTVvVjVODOfB1ZVNa41Gbi2XL+u9vWIeBWwKXB1E2JVs5x1VrFIGhATcElSb7qratx1vNhc4G3l+luA0RExJiLWAr4O/HNfHxIR0yJiTkTMWbRo0SCELUmtywRcktSbeioQfwrYLyJ+B+wHPAKsAD4EzMrMh+lDZp6dmVMyc8q4cePWNGZJamnOgiJJ6k2fVY0zcyHwVoCIGAW8LTOfiIi9gH0j4kPAKIq6Dk9n5mo3ckpSJzEBV+Wiu+trklrFi1WNKa5sTwXeWbtBRIwFFmfmC8ApwLkAmfmumm1OAKaYfEuSCbhaxAePOoUzjtmFA6oORNJL1FnVeH/g9IhI4Hrgw5UFrOaYMaPqCKS2ZgKulrBk5IYs33hM1WFI6kYdVY1nAL1mZJl5PnB+A8JTFcaOrToCqa15E6ZawtF3XcP4nzqnrCS1hfPPLxZJA2ICrpZw9F3XMP5n06sOQ5JUDxNwaY2YgKtldJ3XTJIkaSgyAVflnAVFkiR1EhNwSZIkqYlMwCVJkqQmchpCtYQTjjmVbx27G4dUHYgkqW+zZvW9jaQemYCrJSxbZwQvrDey6jAkSfUY6flaWhMOQVFLOO6O/2bLS8+vOgxJUj2+971ikTQgJuCqXBAc8fvfsvnVV1YdiiSpHtOnF4ukATEBlyRJkprIBFySJElqIhNwSZIkqYlMwCVJkqQmchpCtYSp7/wK333nHryx6kAkSX2bPbvqCKS25hVwVS6i6ggkSZKaxwRcLeEfb/kJ21xwVtVhSJLqccYZxSJpQEzA1RIO+OOtvOz6a6oOQ5JUj5//vFgkDYgJuCRJktREJuCSJElSE5mAS5IkSU3U0AQ8Ig6NiPsjYn5EnNzN6+tGxGXl67dExMSa104p2++PiENq2k+KiHsi4u6I+FFEjGhkH9R4ASxbe11WruufUpLawnrrFYukAWlYAh4Rw4DvAocBk4F3RMTkLpudCCzJzJcD3wS+Wu47GZgK7AQcCnwvIoZFxHjgY8CUzHwlMKzcTm3uhLf/G7d+7+Kqw5Ak1eOqq4pF0oA08gr4nsD8zHwwM58HLgWO7LLNkcAF5foM4ICIiLL90sx8LjP/BMwv3w+K4kHrRcTawEhgYQP7IEmSJA2qRibg44GHa54vKNu63SYzVwBPAGN62jczHwHOAP4XeBR4IjOv7u7DI2JaRMyJiDmLFi0ahO6okT56w4/Y7vvfrDoMSVI9/v3fi0XSgDQyAe+uvmHWuU237RGxMcXV8UnAFsD6EXFcdx+emWdn5pTMnDJu3Lh+hK0q7PPQXMbcckPVYUiS6nHttcUiaUAamYAvALaseT6B1YeLvLhNOaRkQ2BxL/seCPwpMxdl5nLgJ8DeDYlekiRJaoBGJuC3AdtFxKSIGE5xs+TMLtvMBI4v148Gfp2ZWbZPLWdJmQRsB9xKMfTktRExshwrfgBwXwP7oCaI7n7vkCRJGqLWbtQbZ+aKiPgI8EuK2UrOzcx7IuI0YE5mzgTOAS6KiPkUV76nlvveExHTgXuBFcCHM3MlcEtEzADuKNt/B5zdqD5IkiRJg61hCThAZs4CZnVp+3zN+jLgmB72/RLwpW7avwB8YXAjVdWWrLcByzfaqOowJEn1GDOm6giktmYlTLWED77lM8z5+g+qDkNSN+ooqrZ1RFwbEfMiYnZETCjbd4uIm8riafMi4tjmR6+GuPzyYpE0ICbgahldp8iRVL06i6qdAVyYmbsApwGnl+1Lgfdk5qqiamdGhD91Sep4JuBqCZ/+zfm84j9P73tDSc1WT1G1ycCqOemuW/V6Zv4hMx8o1xcCjwHOCzsUnHJKsUgakIaOAZfqE+zxyO/ZePkjVQciaXXdFUZ7TZdt5gJvA74FvAUYHRFjMvNvqzaIiD2B4cAfGxuumuKmm6qOQGprXgGXJPWmnqJqnwL2i4jfAfsBj1DMVFW8QcTmwEXAezPzhW4/xOrFkjqICbgkqTd9FlXLzIWZ+dbM3B34bNn2BEBEbAD8N/C5zLy5pw+xerGkTmICLknqTZ9F1SJibESs+u/JKcC5Zftw4AqKGzR/3MSYJamlmYCrJTw6eizPbrpZ1WFI6iIzVwCriqrdB0xfVVQtIt5cbrY/cH9E/AHYlL/XcHg78HrghIi4s1x2a24P1BATJhSLpAHxJky1hJPe9Cm+NXW3l/zOLak11FFUbQYwo5v9LgYubniAar6L/bNKa8Ir4KpcdHeLlyRJ0hDlFXC1hM9fczavfHgcXPLDqkORJPXlE58oHs88s9o4pDZlAq6WMPmxB9mQx6oOQ5JUjzvvrDoCqa05BEWSJElqoroT8IhYv5GBSJIkSZ2gzwQ8IvaOiHsppp8iInaNiO81PDJJkiRpCKrnCvg3gUOAvwFk5lyKeV2lQRHAg5uM5+mtt6k6FElSPbbfvlgkDUhdN2Fm5sPx0rniVjYmHHWqzxz6Ub41dTcmVh2IJKlvZ59ddQRSW6snAX84IvYGsiwr/DHK4SiSJEmS+qeeBPwDwLeA8cAC4GrgQ40MSp3ny7/4Nrs+MAYuv6TqUKQhKyKmAJ8FtqY4/weQmblLpYGp/UybVjx6JVwakHoS8B0y8121DRGxD3BDY0JSJ9pm8SOMWvuJqsOQhrpLgH8G7gJeqDgWtbM//KHqCKS2Vk8C/m1gjzraJEmtbVFmzqw6CEnqdD0m4BGxF7A3MC4i/qnmpQ2AYY0OTJ2jyw2+khrnCxHxQ+Ba4LlVjZn5k+pCkqTO09sV8OHAqHKb0TXtTwJHNzIoSVJDvBfYEViHvw9BScAEXJKaqMcEPDN/A/wmIs7PzIeaGJM60L0v24Zx249jbNWBSEPbrpm5c9VBaAjYbbeqI5DaWj1jwJdGxH8AOwEjVjVm5hsaFpU6zmkHTuPMY3dj26oDkYa2myNicmbeW3UganNnnll1BFJbq6cS5iXA74FJwL8BfwZua2BM6lBJVh2CNNS9DrgzIu6PiHkRcVdEzKs6KEnqNPVcAR+TmedExMdrhqX8ptGBqbN888ozeNW8jWGWQ1GlBjq06gA0RBx3XPF48cXVxiG1qXoS8OXl46MR8UZgITChcSGp0wSw+VOPs95jy6oORRrq/JlJg2PBgqojkNpaPQn4FyNiQ+CTFPN/bwCc1NCoJEmN8N8USXhQ3NMzCbif4h4fSVKT9JqAR8QwYLvM/DnwBPAPTYlKkjTous6AEhF7AO+vKBxJ6li93oSZmSuBNw/0zSPi0PJmn/kRcXI3r68bEZeVr98SERNrXjulbL8/Ig6pad8oImZExO8j4r6yYJAkqZ8y8w7g1VXHIUmdpp4hKDdGxHeAy4BnVjWWJ+4elVfPvwscBCwAbouImV2mvzoRWJKZL4+IqcBXgWMjYjIwleJn0S2AayJi+/J/CL4F/CIzj46I4cDIejur1nXH+B3ZeMdNGVd1INIQ1qWq8VrAHsCiisJRO9vLa1/SmqgnAd+7fDytpi2BvuYB3xOYn5kPAkTEpcCRQG0CfiRwark+A/hOFHXJjwQuzczngD9FxHxgz4i4B3g9cAJAZj4PPF9HH9TivrbfCWz29l3ZoepApKGttqrxCoox4ZdXFIva2emnVx2B1Nb6TMAzc6DjvscDD9c8XwC8pqdtMnNFRDwBjCnbb+6y73jgWYqrNedFxK7A7cDHM/MZ1LYiqo5A6hj3ZuaPaxsi4hjgxz1sL0lqgHoK8QxUd2lV1ymwetqmp/a1KX4yPSszd6cYErPa2HKAiJgWEXMiYs6iRf7C2urOuuLLvOafp1UdhjTUnVJnm9S7t72tWCQNSD1DUAZqAbBlzfMJFHOId7fNgohYG9gQWNzLvguABZl5S9k+gx4S8Mw8GzgbYMqUKc592+I2fvZJhj/xQtVhSENSRBwGHA6Mj4j/rHlpA4qhKFL//O1vVUcgtbVGXgG/DdguIiaVN0tOBWZ22WYmcHy5fjTw68zMsn1qOUvKJGA74NbM/AvwcESsGip8AC8dUy5JWt1CYA6wjGLo3qplJnBIL/tJkhqgzyvgEfHWbpqfAO7KzMd62q8c0/0R4JfAMODczLwnIk4D5mTmTOAc4KLyJsvFFEk65XbTKZLrFcCHyxlQAD4KXFIm9Q8C762zr2p1/k4hNURmzgXmRsR/ZebyPneQJDVUPUNQTgT2Aq4rn+9PcYPk9hFxWmZe1NOOmTkLmNWl7fM168uAY3rY90vAl7ppvxOYUkfcbe+ZZ55h9uzZPPDAAzz11FOMGDGCl73sZbzuda9j2223BWDJkiX89re/5Y9//CNPP/00I0eOZMyYMey2227svPPODBs2DIBTTz31xfddZ511GDVqFBMmTODVr341W221VRXd0yDxOFE/7BkRpwJbU5z/A8jM3KbSqIYIv4uql8eK6knAXwBekZl/BYiITYGzKGY0uR7oMQHXmpk+fTrLly/nzW9+M5tssgnPPPMMDz30EM8++ywACxcu5IILLmDcuHEcfvjhjB07luXLl7No0SLuuOMONtlkk5d8+d785jez/fbbs2LFCpYsWcLcuXM577zzOPDAA9lnn32q6iZBcMPWuzL6lZs5D/gAdMpxokFxDnASxfCTlX1sq37qqO/iAQdU+/ltrqOOFXWrngR84qrku/QYsH1mLo4If8pskGXLlvHQQw/xnve8h222KS5ObbTRRowfPx6AzOSKK65gzJgxnHjiiUTNXH6bbbYZO++8M8Vw+r8bMWIEo0aNevG9Jk2axOjRo7n22mt5xStewSabbNKk3q3u2/u8g4nH7MpOlUXQnjrtONEaeyIzr+rvThFxKEURtGHADzPzK11e3xo4FxhHMZzwuMxcUL52PPC5ctMvZuYFaxB/y+q47+K//mt1n93mOu5YUbfquQnztxHx84g4vjyR/gy4PiLWB/6vseF1ruHDhzN8+HDuv/9+VqxYfZKCv/zlLyxatIi99977JV/OWj2119prr73ITH7/+9+vccxqPo8T9dN1EfEfEbFXROyxaulth5qqxocBk4F3lNWKa50BXJiZu1AUbTu93HcT4AsUv5juCXwhIjYe3C61Br+LqpfHiqC+K+AfBt4G7EMxXvBC4PJytpKBFulRH9Zaay2OOuoorrzySm6//XY222wzttpqKyZPnsyECRP4WzkF1JgxY17cZ9myZXzjG9948fm+++7Lvvvu2+vnjBw5kvXXX58lS5Y0piN1On/6F9jxt6Phhuv63lgv6rTjRGtsVTG02vto+qpsXE9V48kUQ1uguF/op+X6IcCvMnNxue+vgEOBH61BH1pSx30XDzuseLyq3z+odLyOO1bUrXoqYSbFfNszGh+Oak2ePJntt9+ehx56iAULFjB//nxuvPFGDjjgADbeePWLSOuuuy4f+MAHALjkkktYubK+IZ5df8qqwogVzzFs2TpVh9GWOuk40ZoZYGXjeqoaz6W4UPMt4C3A6IgY08O+4wcQQ6+WPPM8n7nirsF+2wF5YbP9eerxv3Dbn/7KEzf/lqcXz2DCTnuy7sjR/HHeQh68fB7rb1SUxMhMntvktQA8cONV3PzMvfzXQyMBuG3eQu5b7x42uePZ1T7jztsfZuO/rMPPl9zevI51cdKfi0nQvnlxdTG0u045VtrZV962Cxuu15jcpN5pCL8KvIziCviqu+Y3aEhEeom1116bbbfdlm233Zb99tuPmTNnMnv2bN773mL2xccff5zNN98cKH6SWjXOa9Xd0X1ZunQpS5cu7TZRU/vwOFE9ypvovwxskZmHlUNJ9srMc3rbrZu2rv839ingOxFxAsXN+Y9QTCFbz76rYpsGTAP6PXPDiheSPy56ul/7NNRaG8HYjRg+dgdW3H0Dd8+5iS32PJQlzzzPAw8tZPTyETUbF9/Bvy1dwbL/e5ZlZT+WPPM8Dy9eypLhL+3XyueX8diSJ1k5bm1WVNjnZ5cX/+PeUv/u7agDjpV2tvKFxl14qmcIyteAN2XmfQ2LQnUbN24cL7zwAmPHjmXcuHHccMMN7LTTTqy11sBqKt14441EBDvuuOMgR1q/OoayqZ+G4nGiQXE+cB7w2fL5H4DLKGZH6UmfVY0zcyHwVoCIGAW8LTOfiIgFFFPX1u47u7sPWZPqxeNGr8vVJ+3Xn12a5qabhnP11cv5l395E+ec8xjDhj3HtGn7rvZd/N669zB58mT237/ox6lPXMfb3/5qJk9+6XD7a665hhufm8RHPvKOam+s+9lGAC37796Ohuyxom7Vk4D/1eS7+ZYuXcqPf/xjdt99dzbddFOGDx/OwoULueGGG5g0aRIjRozgqKOO4sILL+Scc85h3333fTHpevjhh3nyySdXu0lj2bJlPP3006xcuZLFixczd+5c5s6dy0EHHeSXs015nKifxmbm9Ig4BV4smNbXGKQXqxpTXNmeCryzdoOIGAsszswXgFMoZkSBohDbl2tuvDy4fH3I8buoenmsCOpLwOdExGUUN9U8t6oxM3/SsKjE8OHDmTBhAjfffDOLFy9m5cqVjB49mp133pnXv/71AIwfP573v//9/Pa3v+Wqq67i6aefZu2112bTTTflDW94A3vs8dLJDWbOnAkUwxVWTdR/wgknsPXWWze9f11du+2ejNx1c+cB76dOO060xp4px2YnQES8lqKycY/qrGq8P3B6RCTFEJQPl/sujoh/p0jiAU5bdUPmUNNx38Ujjqg6grbVcceKuhV93VgVEed105yZ+b7GhDT4pkyZknPmzKk6DPXg4cVL2fdr13HGMbty9KsmVB2O1FIi4vbMHJTqv+WUg98GXgncTTFv99GZOW8w3n+weM6W1K7qPWfXMwvKewcnJKl3zrIhNVZm3hER+wE7UNwgeX9mWlBNkpqsxwQ8Ij6dmV+LiG/TzV3rmfmxhkamjnLpf53MNr8aBbffVHUo0pBVFtU5HJhIcf4/OCLIzG/0uqPU1f77F4+zZ1cZhdS2ersCvurGS38HlKSh4UpgGXAX8ELFsUhSx+oxAc/MK8vHC5oXjiSpgSaU5eIlSRWqpxDP9hRFFibWbp+ZvZUuliS1nqsi4uDMvLrqQCSpk9UzDeGPgf8f+CFQX81qSVIruhm4IiLWApZjZWNJqkQ9CfiKzDyr4ZGoo/18x315y+7jeVnVgUhD29eBvYC70mmHtCbe/vaqI5DaWj0J+JUR8SHgCl5aiGdIFlNQNS7e443scvQuvKrqQKSh7QHgbpNvrbEPfajqCKS2Vk8Cfnz5+M81bQlsM/jhqBNFwIjlyxi27NmqQ5GGukeB2RFxFS+9oOI0hOqfpUuLx5Ejq41DalO9JuDlOMHjMvOGJsWjDnX+j09lm9nOAy412J/KZXi5SANz+OHFo/OASwPSawKemS9ExBkUYwYlSW0sM/8NICJGF0/z6YpDkqSOtFYd21wdEW+LiGh4NJKkhomIV0bE74C7gXsi4vaI2KnquCSp09QzBvyfgPWBFRGxDKetkqR2dTbwT5l5HUBE7A/8ANi7yqAkqdP0mYBn5uhmBCJJarj1VyXfAJk5OyLWrzIgSepE9VwBJyI2BrYDRqxqy8zrGxWUOktEMGPnA3n7lC2dB1xqrAcj4l+Bi8rnx1HclCn1zwknVB2B1NbqKUX//wEfByYAdwKvBW4CLEWvQTNj5wPZ84hd2LPqQKSh7X3AvwGXUwwnvB44ocqA1KZMwKU1Us9NmB8HXg08lJn/AOwOLGpoVOo4Gy99guH/Z20nqcG2BbakOPevAxxAkYRL/fP448UiaUDqGYKyLDOXRQQRsW5m/j4idmh4ZOooZ/30dLa5YRQc5DzgUgNdAnyKYhaUFyqORe3s6KOLR+cBlwakngR8QURsBPwU+FVELAEWNjYsSVIDLMrMK6sOQpI6XT2zoLylXD01Iq4DNgR+0dCoJEmN8IWI+CFwLS8tRf+T6kKSpM5T7yworwO2y8zzImIcMB7vnNcgscKT1DTvBXakGP+9aghKAibgktRE9cyC8gVgCrADcB7FiftiYJ869j0U+BYwDPhhZn6ly+vrAhcCrwL+BhybmX8uXzsFOBFYCXwsM39Zs98wYA7wSGYe0WcvJUkAu2bmzlUHIUmdrp4r4G+hmPnkDoDMXBgRfRbnKZPk7wIHAQuA2yJiZmbeW7PZicCSzHx5REwFvgocGxGTganATsAWwDURsX1mriz3+zhwH2A1ziHi4t0P552v2cp5wKXGujkiJnc5D0v998EPVh2B1NbqmYbw+cxMip8p6UfVtD2B+Zn5YGY+D1wKHNllmyOBC8r1GcABERFl+6WZ+Vxm/gmYX74fETEBeCPwwzrjUBv4+Stez8MHvanqMKSh7nXAnRFxf0TMi4i7ImJe1UGpDR17bLFIGpB6roBPj4jvAxtFxD9SFHL4QR37jQcernm+AHhNT9tk5oqIeAIYU7bf3GXf8eX6mcCngT6vwqt9bP7kItb7y0Jgq6pDkYayQ6sOQEPEw+V/3rfcsto4pDZVzywoZ0TEQcCTFOPAP5+Zv6rjvbu7ty7r3Kbb9og4AngsM2+PiP17/fCIacA0gK22Mqlrdd/8+dfZ5pZR8CbnAZcaJTMfqjoGDRHvfnfx6Dzg0oDUNQtKmXDXk3TXWkBRcW2VCaw+f/iqbRZExNoUUxwu7mXfNwNvjojDgRHABhFxcWYe103MZwNnA0yZMqVr4q8WEk6DIkmSOkiPY8Aj4qmIeLKb5amIeLKO974N2C4iJkXEcIqbKmd22WYmcHy5fjTw63K8+UxgakSsGxGTgO2AWzPzlMyckJkTy/f7dXfJtyRJktSqerwCnplrNMa6HNP9EeCXFNMQnpuZ90TEacCczJwJnANcFBHzKa58Ty33vScipgP3AiuAD9fMgCJJkiS1rbqGoAxUZs4CZnVp+3zN+jLgmB72/RLwpV7eezYwezDilCRJkpqloQm4VK8f7PkW3rPXROcBl6R28MlPVh2B1NbqmQdcarhrX/4aFu57YNVhSOpGRBxazh0+PyJO7ub1rSLiuoj4XTm/+OFl+zoRcUE53/h9ZYVjDQVvelOxSBoQE3BVLgi2+dsCRj/0x6pDkdRFTVXjw4DJwDvKasW1PgdMz8zdKe7l+V7ZfgywbmbuDLwKeH9ETGxG3Gqw++8vFkkD4hAUtYQv//I7TLpjFBzjPOBSi3mxqjFARKyqalxbzj6BDcr1Dfn7lLMJrF9OM7se8DxFTQm1u/e/v3h0HnBpQLwCLknqTXdVjcd32eZU4LiIWEBx4/1Hy/YZwDPAo8D/Amdk5uKGRitJbcAEXJLUm3qqGr8DOD8zJwCHU0wvuxbF1fOVwBbAJOCTEbFNtx8SMS0i5kTEnEWLFg1e9JLUgkzA1TqsVyq1onqqGp8ITAfIzJsoKhWPBd4J/CIzl2fmY8ANwJTuPiQzz87MKZk5Zdy4cYPcBUlqLSbgkqTe1FPV+H+BAwAi4hUUCfiisv0NUVgfeC3w+6ZFLkktypswVbkI+PbeU3nfPpOI9t/XAAAN+ElEQVTYtOpgJL1EnVWNPwn8ICJOovgt64TMzIj4LnAecDfFUJbzMnNeNT3RoPrc56qOQGprJuBqCTdM3I037rlz1WFI6kYdVY3vBfbpZr+n6aHasdrcgdZtkNaEQ1DUEib/9UE2+sM9VYchSarHnXcWi6QB8Qq4WsLnrz2bSXNHwbsPqzoUSVJfPvGJ4tF5wKUB8Qq4WojToEiSpKHPBFySJElqIhNwVa67Kh+SJElDlQm4JEmS1ETehKmW8LXXH8/799uGQ6oORJLUty9/ueoIpLZmAq6WcMeEV/D4Lq+sOgxJUj323rvqCKS25hAUtYQ9FtzHmLm3Vx2GJKkeN95YLJIGxCvgagmfvv4CJt27Pkx7a9WhSJL68pnPFI/OAy4NiFfAVT2nQZEkSR3EBFySJElqIhNwSZIkqYlMwCVJkqQm8iZMtYTTDpjGh/5hW46oOhBJUt/OPLPqCKS2ZgKulnDvptuwZPudqg5DklSP3XarOgKprTkERZULgn3+fCeb3fo/VYciSarHNdcUi6QB8Qq4WsJHb7yUiX9YHz72rqpDkST15YtfLB4PPLDaOKQ25RVwSZIkqYlMwCVJkqQmMgGXJEmSmqihCXhEHBoR90fE/Ig4uZvX142Iy8rXb4mIiTWvnVK23x8Rh5RtW0bEdRFxX0TcExEfb2T8kiRJ0mBr2E2YETEM+C5wELAAuC0iZmbmvTWbnQgsycyXR8RU4KvAsRExGZgK7ARsAVwTEdsDK4BPZuYdETEauD0iftXlPdVmIuAzh3yEjx/wco6sOhhJUt++//2qI5DaWiOvgO8JzM/MBzPzeeBSWC2/OhK4oFyfARwQEVG2X5qZz2Xmn4D5wJ6Z+Whm3gGQmU8B9wHjG9gHNcmDYybw5NbbVh2GJKkeO+xQLJIGpJEJ+Hjg4ZrnC1g9WX5xm8xcATwBjKln33K4yu7ALYMYsypywPxbmPA/zikrSW3hyiuLRdKANDIBj27ass5tet03IkYBlwOfyMwnu/3wiGkRMSci5ixatKjOkFWVf7z1Cl7xox9WHYYkqR5f/3qxSBqQRibgC4Ata55PABb2tE1ErA1sCCzubd+IWIci+b4kM3/S04dn5tmZOSUzp4wbN24NuyJJkiQNjkYm4LcB20XEpIgYTnFT5cwu28wEji/XjwZ+nZlZtk8tZ0mZBGwH3FqODz8HuC8zv9HA2CVJkqSGaNgsKJm5IiI+AvwSGAacm5n3RMRpwJzMnEmRTF8UEfMprnxPLfe9JyKmA/dSzHzy4cxcGRGvA94N3BURd5Yf9ZnMnNWofqjxuhtvJEmSNFQ1LAEHKBPjWV3aPl+zvgw4pod9vwR8qUvb/2C+JkmSpDbW0ARcqtdJR3ySTx68PUdXHYgkqW8XXVR1BFJbsxS9WsKjG4xj6aZbVB2GpG7UUdV4q7JK8e8iYl5EHF7z2i4RcVNZvfiuiBjR3OjVEFtuWSySBsQEXC3hiPuuZ+tf/bzqMCR1UVPV+DBgMvCOslpxrc8B0zNzd4p7eb5X7rs2cDHwgczcCdgfWN6k0NVIl11WLJIGxARcLeG4381ih59eXHUYklZXT1XjBDYo1zfk71POHgzMy8y5AJn5t8xc2YSY1WhnnVUskgbEBFyVK2aXlNSi6qlqfCpwXEQsoLjx/qNl+/ZARsQvI+KOiPh0o4OVpHZgAi5J6k09VY3fAZyfmROAwymml12L4kb/1wHvKh/fEhEHdPshVi+W1EFMwCVJvamnqvGJwHSAzLwJGAGMLff9TWY+nplLKa6O79Hdh1i9WFInMQGXJPWmnqrG/wscABARr6BIwBdRFGLbJSJGljdk7kdRYE2SOprzgKslfPCoU/j0ITvwjqoDkfQSdVY1/iTwg4g4iWJ4ygmZmcCSiPgGRRKfwKzM/O9qeqJBNWNG1RFIbc0EXJULYMnIDXluo02qDkVSN+qoanwvsE8P+15MMRWhhpKxY6uOQGprDkFRSzj6rmt4+SyvqEhSWzj//GKRNCAm4GoJRQJ+edVhSJLqYQIurRETcEmSJKmJTMAlSZKkJvImTLWMR/7vWU448/qqw5AaYouN1uPcE15ddRiSpBZgAq7KbTRyHTbbcATPrXiBrceMrDocqSFeNnpE1SFIklqECbgqFxFMvOU3AHx/pAm4JLW8WbP63kZSj0zA1RpMvCWpfXjOltaIN2GqNXzve8UiSWp9nrOlNWICrtYwfXqxSJJan+dsaY2YgEuSJElNZAIuSZIkNZEJuCRJktREJuCSJElSEzkNoVrD7NlVRyBJqpfnbGmNeAVckiRJaiITcEmSJKmJTMAlSZKkJjIBlyRJkprIBFySJElqIhNwSZIkqYlMwCVJkqQmMgGXJEmSmsgEXJIkSWqiyMyqY2i4iFgEPNTP3cYCjzcgnFZkX4cm+zo0bJ2Z46oOopkGeM6GoX0cdGVfhyb72v7qOmd3RAI+EBExJzOnVB1HM9jXocm+qtN00nFgX4cm+9o5HIIiSZIkNZEJuCRJktREJuA9O7vqAJrIvg5N9lWdppOOA/s6NNnXDuEYcEmSJKmJvAIuSZIkNZEJeDci4tCIuD8i5kfEyVXHMxgi4s8RcVdE3BkRc8q2TSLiVxHxQPm4cdkeEfGfZf/nRcQe1Ubfu4g4NyIei4i7a9r63beIOL7c/oGIOL6KvvSlh76eGhGPlH/bOyPi8JrXTin7en9EHFLT3tLHeERsGRHXRcR9EXFPRHy8bB+Sf1etmVY/ngfCc/bQ+G53yjkbPG/3W2a61CzAMOCPwDbAcGAuMLnquAahX38GxnZp+xpwcrl+MvDVcv1w4CoggNcCt1Qdfx99ez2wB3D3QPsGbAI8WD5uXK5vXHXf6uzrqcCnutl2cnn8rgtMKo/rYe1wjAObA3uU66OBP5T9GZJ/V5c1OlZa/ngeYL88Z/fSt3b5bnfKObuM3/N2PxavgK9uT2B+Zj6Ymc8DlwJHVhxToxwJXFCuXwAcVdN+YRZuBjaKiM2rCLAemXk9sLhLc3/7dgjwq8xcnJlLgF8BhzY++v7poa89ORK4NDOfy8w/AfMpju+WP8Yz89HMvKNcfwq4DxjPEP27ao20/PE8iDxnt9l3u1PO2eB5u79MwFc3Hni45vmCsq3dJXB1RNweEdPKtk0z81EovjjAy8r2ofBv0N++tXufP1L+hHfuqp/3GCJ9jYiJwO7ALXTe31V9G6p/Y8/ZhaH63R6y52zwvF0PE/DVRTdtQ2GqmH0ycw/gMODDEfH6XrYdqv8G0HPf2rnPZwHbArsBjwJfL9vbvq8RMQq4HPhEZj7Z26bdtLVVXzVgQ/Vv7Dm7MBS/20P2nA2et+tlAr66BcCWNc8nAAsrimXQZObC8vEx4AqKn7T+uupnyvLxsXLzofBv0N++tW2fM/OvmbkyM18AfkDxt4U272tErENxEr8kM39SNnfM31V1G5J/Y8/ZQ/e7PVTP2eB5uz9MwFd3G7BdREyKiOHAVGBmxTGtkYhYPyJGr1oHDgbupujXqruLjwd+Vq7PBN5T3qH8WuCJVT8ftZH+9u2XwMERsXH5c+DBZVvL6zLW8y0Uf1so+jo1ItaNiEnAdsCttMExHhEBnAPcl5nfqHmpY/6uqlvLH8/95Tl7aH+3h+I5Gzxv91vVd4G24kJxZ+4fKO46/mzV8QxCf7ahuGt6LnDPqj4BY4BrgQfKx03K9gC+W/b/LmBK1X3oo38/ovgZbznF/zmfOJC+Ae+juOllPvDeqvvVj75eVPZlHsUJbfOa7T9b9vV+4LCa9pY+xoHXUfzkOA+4s1wOH6p/V5c1Pl5a+ngeQH88Zw+R73annLPLGD1v92OxEqYkSZLURA5BkSRJkprIBFySJElqIhNwSZIkqYlMwCVJkqQmMgGXJEmSmsgEXB0hIm4sHydGxDsH+b0/091nSZIGxnO2hjqnIVRHiYj9gU9l5hH92GdYZq7s5fWnM3PUYMQnSfo7z9kaqrwCro4QEU+Xq18B9o2IOyPipIgYFhH/ERG3RcS8iHh/uf3+EXFdRPwXRYEAIuKnEXF7RNwTEdPKtq8A65Xvd0ntZ5XVvf4jIu6OiLsi4tia954dETMi4vcRcUlZQYyI+EpE3FvGckYz/40kqVV4ztZQt3bVAUhNdjI1V1PKk/ITmfnqiFgXuCEiri633RN4ZWb+qXz+vsxcHBHrAbdFxOWZeXJEfCQzd+vms94K7AbsCowt97m+fG13YCdgIXADsE9E3EtRlnjHzMyI2GjQey9J7cVztoYkr4Cr0x0MvCci7gRuoSiZu1352q01J3KAj0XEXOBmYMua7XryOuBHmbkyM/8K/AZ4dc17L8jMFyjK9U4EngSWAT+MiLcCS9e4d5I0tHjO1pBgAq5OF8BHM3O3cpmUmauupjzz4kbFOMQDgb0yc1fgd8CIOt67J8/VrK8E1s7MFRRXcC4HjgJ+0a+eSNLQ5zlbQ4IJuDrNU8Domue/BD4YEesARMT2EbF+N/ttCCzJzKURsSPw2prXlq/av4vrgWPLMYvjgNcDt/YUWESMAjbMzFnAJyh+CpWkTuY5W0OSY8DVaeYBK8qfJc8HvkXxU+Id5U01iyiuZHT1C+ADETEPuJ/iJ81VzgbmRcQdmfmumvYrgL2AuUACn87Mv5T/MejOaOBnETGC4krMSQProiQNGZ6zNSQ5DaEkSZLURA5BkSRJkprIBFySJElqIhNwSZIkqYlMwCVJkqQmMgGXJEmSmsgEXJIkSWoiE3BJkiSpiUzAJUmSpCb6f2pOkz3OnJB5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The red dashed line represent the change of phase, and the optimizer name (plus its optional parameters) is indicated, in case you changed it. You can remove that off by using show_text=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuAAAAEKCAYAAABT6eBwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xu4XHV97/H3xyCg3FRIfTQBE2u8RAUvMcVL1UrFQNV4PcZLRaWlraLWWx+oLVp6OGq1Wq3oOSgoIhURtY1tFK2CHq0iQSEQMBpBJYZKLBy8FTHwPX+stWXc7L0zCXvWzN7zfj3PPHvNb/3WzPfHnr3yZc1vfX+pKiRJkiR14w7DDkCSJEkaJybgkiRJUodMwCVJkqQOmYBLkiRJHTIBlyRJkjpkAi5JkiR1yARckiRJ6pAJuCRJktQhE3BJkiSpQ7sNO4AuHHDAAbVkyZJhhyFJO+2iiy76cVUtHHYcXfKcLWmu6vecPRYJ+JIlS1i/fv2ww5CknZbk+8OOoWuesyXNVf2es52CIkmSJHXIBFySJEnqkAm4JEmS1CETcEmSJKlDJuCSJElShwaagCdZlWRTks1Jjpti/x5JPtruvyDJkrZ9/yTnJflZkndPOubhSS5tj3lXkgxyDJIkSdJsGlgCnmQBcDJwBLAceG6S5ZO6HQ1cX1X3Ad4BvKVtvxH4a+C1U7z0e4FjgGXtY9XsRy9JkiQNxiDrgK8ENlfVlQBJzgJWA5f39FkNvLHdPgd4d5JU1c+BLye5T+8LJrkHsG9VfbV9/iHgacCnZzPwb/3nT1i34ZrZfEntwJ67L+CoRy5hrz3GojS9JEkaY4PMdhYBV/c83wL8znR9qmp7khuA/YEfz/CaWya95qKpOiY5huZKOQcddNBOBf6dH/2Mfzxv804do11X1fy872/tw+8vv/twg5EkSRqwQSbgU83Nrl3os0v9q+oU4BSAFStWzPSat/GUQ+7JUw65584cotth49Yb+IN3fZmba6d+TZIkSXPSIG/C3AIc2PN8MbB1uj5JdgP2A67bwWsu3sFrSpIkSSNrkAn4hcCyJEuT7A6sAdZO6rMWOKrdfhbwharpL4NW1TXAT5Mc2lY/eSHwL7MfuiRJkjQYA5uC0s7pPhY4F1gAnFZVG5OcCKyvqrXAqcAZSTbTXPleM3F8ku8B+wK7J3kacHhVXQ78GfBB4E40N1/O6g2YkiRJ0iANtOREVa0D1k1qO6Fn+0bg2dMcu2Sa9vXAg2YvSkmSJKk7roSpkeE9mJIkaRyYgEuSJEkdMgHX0GXK6pKSRkWSVUk2Jdmc5Lgp9t8ryeeTbEhyfpLFk/bvm+SHSd7dXdSSNLpMwCVJ00qyADgZOAJYDjw3yfJJ3d4GfKiqDgZOBN40af/fAl8cdKySNFeYgEuSZrIS2FxVV1bVTcBZwOpJfZYDn2+3z+vdn+ThwN2Bz3YQqyTNCSbgkqSZLAKu7nm+pW3rdQnwzHb76cA+SfZPcgfg74HX7ehNkhyTZH2S9du2bZuFsCVpdJmAS5JmMtVNGpNrFr0WeFySbwKPA34IbAdeCqyrqqvZgao6papWVNWKhQsX3t6YJWmkDbQOuLRzrEMojaAtwIE9zxcDW3s7VNVW4BkASfYGnllVNyR5JPC7SV4K7E2zsNrPquo2N3JK0jgxAZckzeRCYFmSpTRXttcAz+vtkOQA4LqqugU4HjgNoKqe39PnRcAKk29JcgqKRkCsQiiNrKraDhwLnAtcAZxdVRuTnJjkqW23xwObknyb5obLk4YSrCTNEV4BlyTNqKrWAesmtZ3Qs30OcM4OXuODwAcHEJ4kzTleAZckSZI6ZAIuSZIkdcgEXCOjLIIiSZLGgAm4JEmS1CETcA2dVVAkSdI4MQGXJEmSOmQCLkmSJHXIBFySJEnqkAm4JEmS1CETcI0MqxBKkqRxYAIuSZIkdcgEXEMXrEMoSZLGhwm4JEmS1CETcEmSJKlDJuCSJElSh0zAJUmSpA6ZgGtklHUIJUnSGDAB19DFIiiSJGmMmIBLkiRJHTIBlyRJkjpkAi5JkiR1yARckiRJ6pAJuEZGYRkUSZI0/5mAS5IkSR0aaAKeZFWSTUk2Jzluiv17JPlou/+CJEt69h3ftm9K8qSe9lcl2ZjksiQfSbLnIMegwbMKoSRJGicDS8CTLABOBo4AlgPPTbJ8Urejgeur6j7AO4C3tMcuB9YADwRWAe9JsiDJIuAVwIqqehCwoO0nSZIkzQmDvAK+EthcVVdW1U3AWcDqSX1WA6e32+cAhyVJ235WVf2yqq4CNrevB7AbcKckuwF3BrYOcAySJEnSrBpkAr4IuLrn+Za2bco+VbUduAHYf7pjq+qHwNuAHwDXADdU1WenevMkxyRZn2T9tm3bZmE4kiRJ0u03yAR8qqm9k8tcTNdnyvYkd6W5Or4UuCewV5IXTPXmVXVKVa2oqhULFy7cibAlSZKkwRlkAr4FOLDn+WJuO13k133aKSX7AdfNcOzvA1dV1baq+hXwCeBRA4lenSurEEqSpDEwyAT8QmBZkqVJdqe5WXLtpD5rgaPa7WcBX6iqatvXtFVSlgLLgK/TTD05NMmd27nihwFXDHAM6kAsgyJJksbIboN64aranuRY4FyaaiWnVdXGJCcC66tqLXAqcEaSzTRXvte0x25McjZwObAdeFlV3QxckOQc4Btt+zeBUwY1BkmSJGm2DSwBB6iqdcC6SW0n9GzfCDx7mmNPAk6aov0NwBtmN1JJkiSpG66EKUmSJHXIBFySNKM+VjW+V5LPJ9mQ5Pwki9v2hyT5art68YYkz+k+ekkaPSbgGhkWQZFGT5+rGr8N+FBVHQycCLypbf8F8MKqmljV+B+S3KWbyCVpdJmAS5Jm0s+qxsuBz7fb503sr6pvV9V32u2twLWACzNIGnsm4BoB1iGURlg/qxpfAjyz3X46sE+S/Xs7JFkJ7A58d0BxStKcYQIuSZpJP6savxZ4XJJvAo8DfkhTKrZ5geQewBnAi6vqlinfJDkmyfok67dt2zY7kUvSiDIBlyTNZIerGlfV1qp6RlU9FHh923YDQJJ9gX8D/qqqvjbdm1TVKVW1oqpWLFzoLBVJ85sJuCRpJjtc1TjJAUkm/j05Hjitbd8d+CTNDZof6zBmSRppJuCSpGlV1XZgYlXjK4CzJ1Y1TvLUttvjgU1Jvg3cnVsXUfsfwGOBFyW5uH08pNsRSNLoGehKmNLOqLIQoTSK+ljV+BzgnCmO+zDw4YEHKElzjFfAJUmSpA6ZgGvoYhVCSZI0RkzAJUmSpA6ZgEuSJEkdMgGXJEmSOmQCLkmSJHXIBFySJEnqkAm4hs4iKJIkaZz0nYAn2WuQgUiSJEnjYIcJeJJHJbmcZglikhyS5D0Dj0ySJEmah/q5Av4O4EnAfwFU1SXAYwcZlCRJkjRf9TUFpaquntR08wBikSRJkua93froc3WSRwGVZHfgFbTTUaTZVDXsCCRJkgavnwT8T4F3AouALcBngZcOMihJ0uxLsgJ4PXAvmvN/gKqqg4camCSNmX4S8PtV1fN7G5I8GvjKYELSuEksRCh15EzgdcClwC1DjkWSxlY/Cfg/Ag/ro02SNNq2VdXaYQchSeNu2gQ8ySOBRwELk7y6Z9e+wIJBByZJmnVvSPJ+4PPALycaq+oTwwtJksbPTFfAdwf2bvvs09P+E+BZgwxKkjQQLwbuD9yRW6egFGACLkkdmjYBr6ovAl9M8sGq+n6HMUmSBuOQqnrwsIOQpHHXzxzwXyR5K/BAYM+Jxqp6wsCi0lgqrEMoDdjXkiyvqsuHHYgkjbN+FuI5E/gWsBT4G+B7wIUDjEljxhooUmceA1ycZFOSDUkuTbJh2EFJ0rjp5wr4/lV1apJX9kxL+eKgA5MkzbpVww5AktRfAv6r9uc1Sf4A2AosHlxIkqQBcZ6XJI2AfhLw/5lkP+A1NPW/9wVeNdCoJEmD8G80SXho7ulZCmyiucdHktSRGeeAJ1kALKuqG6rqsqr6vap6eL8LOSRZ1c413JzkuCn275Hko+3+C5Is6dl3fNu+KcmTetrvkuScJN9KckVbr1yStANV9eCqOrj9uQxYCXx52HFJ0riZMQGvqpuBp+7KC7fJ+8nAEcBy4LlJlk/qdjRwfVXdB3gH8Jb22OXAGpqrMquA97SvB/BO4DNVdX/gEOCKXYlPo6f8clzqVFV9A3jEsOOQpHHTzxSU/0jybuCjwM8nGtsT90xWApur6kqAJGcBq4He8lergTe22+cA706Stv2sqvolcFWSzcDKJBuBxwIvamO4CbipjzFI0tibtKrxHYCHAduGFI4kja1+EvBHtT9P7GkrYEd1wBcBV/c83wL8znR9qmp7khuA/dv2r006dhHw3zT/WHwgySHARcArq+rnaM6KdQilrvSuarydZk74x4cUiySNrR0m4FX1e7v42lOlVZMnGUzXZ7r23Wiu2Ly8qi5I8k7gOOCvb/PmyTHAMQAHHXTQToQtSfPW5VX1sd6GJM8GPjZNf0nSAPSzEM+u2gIc2PN8MU0Jwyn7JNkN2A+4boZjtwBbquqCtv0cmoT8NqrqlKpaUVUrFi5ceDuHIknzwvF9tkmSBqifKSi76kJgWZKlwA9pbqp83qQ+a4GjgK8CzwK+UFWVZC3wT0neDtwTWAZ8vapuTnJ1kvtV1SbgMH5zTrkkaZIkRwBHAouSvKtn1740U1EkSR0aWALezuk+FjgXWACcVlUbk5wIrG9LGZ4KnNHeZHkdTZJO2+9smuR6O/CytiILwMuBM5PsDlwJvHhQY5CkeWIrsJ6mqtVFPe0/xXUdJKlzO0zAkzxjiuYbgEur6tqZjq2qdcC6SW0n9GzfCDx7mmNPAk6aov1iYMWO4tbcYxlCaTCq6hLgkiT/VFW/2uEBkqSB6ucK+NHAI4Hz2uePp6lQct8kJ1bVGQOKTWMiU95zK2kAViZ5I3AvmvN/gKqqew81KkkaM/0k4LcAD6iqHwEkuTvwXpqSgl8CTMAlaW44lWbKyUXAzTvoK0kakH6qoCyZSL5b1wL3rarrAL/KlKS544aq+nRVXVtV/zXx2NFBSVYl2ZRkc5Ljpth/rySfT7IhyflJFvfsOyrJd9rHUbM9IEmai/q5Av5/k/wrt9aJfSbwpSR7Af9vYJFJkmbbeUneCnwC+OVE40wrGydZAJwMPJGmFOyFSdZWVW8FqrcBH6qq05M8AXgT8IdJ7ga8gea+nQIuao+9frYHJklzST8J+Mtoku5H08wX/BDw8aoqYFcX6ZEkdW9iNeLeG9l3tLLxSmBzVV0JkOQsYDW/WQJ2ObdWUzkP+Od2+0nA59pvTEnyOWAV8JHbMQZJmvP6WQmzaBa8OWfw4WicWQRFGqxdXNl4EXB1z/Mt3JrIT7iE5kLNO4GnA/sk2X+aYxftQgwzuv7nN/GXn7x0tl9W0ph78zMPZr873XEgr91vGcK3AL9FcwV84q75fQcSkSRpINqb6P8XcM+qOiLJcuCRVXXqTIdN0Tb5/5dfC7w7yYtobs7/Ic0aDv0cOxHbMcAxAAcddNBMw7iN7bcU3932s506RpJ25OZbBndpsJ8pKH8HPKWqrhhYFBprsQqh1JUPAh8AXt8+/zbwUZrqKNPZAhzY83wxzcI+v1ZVW4FnACTZG3hmVd2QZAtN6dreY8+f6k2q6hTgFIAVK1bs1L96C/fZg8++6nE7c4gkDVU/VVB+ZPItSfPCAVV1Nk15WapqOzsuR3ghsCzJ0nYF4jXA2t4OSQ5IMvHvyfHAae32ucDhSe6a5K7A4W2bJI21fq6Ar0/yUZqbanrvmv/EwKKSJA3Cz9u52QWQ5FCalY2nVVXbkxxLkzgvAE6rqo1JTgTWV9Vamqvcb0pSNFNQXtYee12Sv6VJ4gFOnLghU5LGWT8J+L7AL2iuXEwomjJWkqS549U0V69/O8lXgIXAs3Z0UFWtA9ZNajuhZ3vaG/Wr6jRuvSIuSaK/Kigv7iIQSdJgVdU3kjwOuB/NDZKbqsoF1SSpY9Mm4En+oqr+Lsk/MsVd61X1ioFGprHTVLyUNCjtojpHAktozv+HJ6Gq3j7UwCRpzMx0BXzixsv1XQQiSRq4TwE3ApfS3ogpSeretAl4VX2q/Xl6d+FIkgZocVUdPOwgJGnc9bMQz31pFllY0tu/qmZauliSNHo+neTwqvrssAORpHHWTxWUjwH/G3g/O64XK0kaXV8DPtnW7P4VrmwsSUPRTwK+vareO/BIJEmD9vfAI4FLy7ueJWlo+lkJ81NJXprkHknuNvEYeGQaO2YD0sB9B7jM5FuShqufK+BHtT9f19NWwL1nPxxJ0gBdA5yf5NP85srGliGUpA7NmIC38wRfUFVf6SgejaFk2BFIY+Oq9rF7+5AkDcGMCXhV3ZLkbTRzBiVJc1hV/Q1Akn2ap/WzIYckSWOpnzngn03yzMTrlJI0lyV5UJJvApcBG5NclOSBw45LksZNP3PAXw3sBWxPciOWrZKkueoU4NVVdR5AkscD7wMeNcygJGnc7DABr6p9ughEkjRwe00k3wBVdX6SvYYZkCSNo36ugJPkrsAyYM+Jtqr60qCC0piyMJo0aFcm+WvgjPb5C2huypQkdaifpej/CHglsBi4GDgU+CrgUvSSNLe8BPgb4OM00wm/BLxomAFJ0jjq5ybMVwKPAL5fVb8HPBTYNtCoNFa8v1fqzG8DB9Kc++8IHEaThEuSOtTPFJQbq+rGJCTZo6q+leR+A49MkjTbzgReS1MF5ZYhxyJJY6ufBHxLkrsA/wx8Lsn1wNbBhiVJGoBtVfWpYQchSeOunyooT28335jkPGA/4DMDjUqSNAhvSPJ+4PP85lL0nxheSJI0fvqtgvIYYFlVfSDJQmAR3jmvWVaWQZEG7cXA/Wnmf09MQSnABFySOtRPFZQ3ACuA+wEfoDlxfxh49GBDkyTNskOq6sHDDkKSxl0/VVCeDjwV+DlAVW0FXJxHs8YaKFJnvpZk+bCDkKRx188UlJuqqpIUgKumSdKc9RjgqCRX0cwBD1BVdfBww5Kk8dLPFfCzk/wf4C5J/hj4d+B9/bx4klVJNiXZnOS4KfbvkeSj7f4Lkizp2Xd8274pyZMmHbcgyTeT/Gs/cUiSAFhFs6rx4cBTgCe3PyVJHeqnCsrbkjwR+AnNPPATqupzOzouyQLgZOCJwBbgwiRrq+rynm5HA9dX1X2SrAHeAjyn/Yp0DfBA4J7Avye5b1Xd3B73SuAKYN9+BypJ466qvj/sGCRJ/V0Bp6o+V1Wvq6rX9pN8t1YCm6vqyqq6CTgLWD2pz2rg9Hb7HOCwNMsirgbOqqpfVtVVwOb29UiyGPgD4P19xiFJkiSNjGmvgCf5KUxZF25izuCOrj4vAq7ueb4F+J3p+lTV9iQ3APu37V+bdOyidvsfgL/AG0HnnbIKoSRJGgPTJuBVdXsT3KmKW0xOsabrM2V7kicD11bVRUkeP+ObJ8cAxwAcdNBBO45WkiRJ6kBfU1B20RbgwJ7ni7ntEva/7pNkN5pVNq+b4dhHA09N8j2aKS1PSPLhqd68qk6pqhVVtWLhwoW3fzQamFiHUJIkjZFBJuAXAsuSLE2yO81NlWsn9VkLHNVuPwv4QlVV276mrZKylOau/a9X1fFVtbiqlrSv94WqesEAxyBJkiTNqr6Wot8V7ZzuY4FzgQXAaVW1McmJwPqqWgucCpyRZDPNle817bEbk5wNXA5sB17WUwFFkiRJmrMGloADVNU6YN2kthN6tm8Enj3NsScBJ83w2ucD589GnJIkSVJXBjkFRZIkSdIkJuAaGVYhlCRJ48AEXEOXKatOShoVSVYl2ZRkc5Ljpth/UJLzknwzyYYkR7btd0xyepJLk1yR5Pjuo5ek0WMCLkmaVpIFwMnAEcBy4LlJlk/q9lfA2VX1UJqb6d/Ttj8b2KOqHgw8HPiTJEu6iFuSRpkJuCRpJiuBzVV1ZVXdRLMGw+pJfQqYWB15P25d86GAvdp1Hu4E3AT8ZPAhS9JoMwGXJM1kEXB1z/MtbVuvNwIvSLKFpvLVy9v2c4CfA9cAPwDeVlXXDTRaSZoDTMAlSTOZ6iaNyfdMPxf4YFUtBo6kWd/hDjRXz28G7gksBV6T5N5TvklyTJL1SdZv27Zt9qKXpBFkAq6RUZZBkUbRFuDAnueLuXWKyYSjgbMBquqrwJ7AAcDzgM9U1a+q6lrgK8CKqd6kqk6pqhVVtWLhwoWzPARJGi0m4JKkmVwILEuyNMnuNDdZrp3U5wfAYQBJHkCTgG9r25+Qxl7AocC3OotckkaUCbiGLlYhlEZWVW0HjgXOBa6gqXayMcmJSZ7adnsN8MdJLgE+AryoqoqmesrewGU0ifwHqmpD54OQpBEz0KXoJUlzX1Wto7m5srfthJ7ty4FHT3Hcz2hKEUqSengFXJIkSeqQCbgkSZLUIRNwSZIkqUMm4BoZdZvSwpIkSfOPCbiGziIokiRpnJiAS5IkSR0yAZckSZI6ZAIuSZIkdcgEXJIkSeqQCbhGRlkERZIkjQETcEmSJKlDJuAaPusQSpKkMWICLkmSJHXIBFySJEnqkAm4JEmS1CETcEmSJKlDJuAaGVYhlCRJ48AEXJIkSeqQCbiGLtYhlCRJY8QEXJIkSeqQCbgkSZLUIRNwSZIkqUMm4BodZR0USZI0/5mAS5IkSR0aaAKeZFWSTUk2Jzluiv17JPlou/+CJEt69h3ftm9K8qS27cAk5yW5IsnGJK8cZPzqRiyCIkmSxsjAEvAkC4CTgSOA5cBzkyyf1O1o4Pqqug/wDuAt7bHLgTXAA4FVwHva19sOvKaqHgAcCrxsiteUJEmSRtYgr4CvBDZX1ZVVdRNwFrB6Up/VwOnt9jnAYUnStp9VVb+sqquAzcDKqrqmqr4BUFU/Ba4AFg1wDJIkSdKsGmQCvgi4uuf5Fm6bLP+6T1VtB24A9u/n2Ha6ykOBC2YxZkmSJGmgBpmATzWzd3KZi+n6zHhskr2BjwN/XlU/mfLNk2OSrE+yftu2bX2GLEmSJA3WIBPwLcCBPc8XA1un65NkN2A/4LqZjk1yR5rk+8yq+sR0b15Vp1TViqpasXDhwts5FHXBIoSSJGkcDDIBvxBYlmRpkt1pbqpcO6nPWuCodvtZwBeqqtr2NW2VlKXAMuDr7fzwU4ErqurtA4xdkiRJGojdBvXCVbU9ybHAucAC4LSq2pjkRGB9Va2lSabPSLKZ5sr3mvbYjUnOBi6nqXzysqq6OcljgD8ELk1ycftWf1lV6wY1Dg2eVQglSdI4GVgCDtAmxusmtZ3Qs30j8Oxpjj0JOGlS25cxX5MkSdIc5kqYkiRJUodMwCVJkqQOmYBLkmaUZFWSTUk2Jzluiv0HJTkvyTeTbEhyZM++g5N8NcnGJJcm2bPb6CVp9Ax0Dri0M8o6hNLISbIAOBl4Ik2J2AuTrK2qy3u6/RVwdlW9N8lymnt/lrTlZT8M/GFVXZJkf+BXHQ9BkkaOV8A1dE11SUkjaiWwuaqurKqbgLOA1ZP6FLBvu70ft675cDiwoaouAaiq/6qqmzuIWZJGmgm4JGkmi4Cre55vadt6vRF4QZItNFe/X9623xeoJOcm+UaSvxh0sJI0F5iAS5JmMtVXVJMnjD0X+GBVLQaOpFnf4Q400xwfAzy//fn0JIdN+SbJMUnWJ1m/bdu22YtekkaQCbgkaSZbgAN7ni/m1ikmE44Gzgaoqq8CewIHtMd+sap+XFW/oLk6/rCp3qSqTqmqFVW1YuHChbM8BEkaLSbgkqSZXAgsS7I0ye40KxavndTnB8BhAEkeQJOAb6NZCfngJHdub8h8HM0Kx5I01qyCopFRlkGRRk5VbU9yLE0yvQA4rao2JjkRWF9Va4HXAO9L8iqa6SkvquYP+vokb6dJ4gtYV1X/NpyRSNLoMAGXJM2oqtbRTB/pbTuhZ/ty4NHTHPthmlKEkqSWU1A0dBYhlCRJ48QEXJIkSeqQCbgkSZLUIRNwSZIkqUMm4JIkSVKHrIKikfHu877LmRf8YNhhSANxz7vcidNfsnLYYUiSRoAJuIbuLne+I3/0mKVsveG/hx2KNDAL995j2CFIkkaECbiGLgl/9eTlww5DkiSpE84BlyRJkjpkAi5JkiR1yARckiRJ6pAJuCRJktQhE3BJkiSpQybgkiRJUodMwCVJkqQOmYBLkiRJHUpVDTuGgUuyDfj+Th52APDjAYQzihzr/ORY54d7VdXCYQfRpV08Z8P8/hxM5ljnJ8c69/V1zh6LBHxXJFlfVSuGHUcXHOv85Fg1bsbpc+BY5yfHOj6cgiJJkiR1yARckiRJ6pAJ+PROGXYAHXKs85Nj1bgZp8+BY52fHOuYcA64JEmS1CGvgEuSJEkdMgGfQpJVSTYl2ZzkuGHHMxuSfC/JpUkuTrK+bbtbks8l+U77865te5K8qx3/hiQPG270M0tyWpJrk1zW07bTY0tyVNv/O0mOGsZYdmSasb4xyQ/b3+3FSY7s2Xd8O9ZNSZ7U0z7Sn/EkByY5L8kVSTYmeWXbPi9/r7p9Rv3zvCs8Z8+Pv+1xOWeD5+2dVlU+eh7AAuC7wL2B3YFLgOXDjmsWxvU94IBJbX8HHNduHwe8pd0+Evg0EOBQ4IJhx7+DsT0WeBhw2a6ODbgbcGX7867t9l2HPbY+x/pG4LVT9F3efn73AJa2n+sFc+EzDtwDeFi7vQ/w7XY88/L36uN2fVZG/vO8i+PynD3D2ObK3/a4nLPb+D1v78TDK+C3tRLYXFVXVtVNwFnA6iHHNCirgdPb7dOBp/W0f6gaXwPukuQewwiwH1X1JeC6Sc07O7YnAZ+rquuq6nrgc8CqwUe/c6YZ63RWA2dV1S+r6ipgM83ne+Q/41V1TVV9o93+KXAFsIh5+nvV7TLyn+dZ5Dl7jv1tj8s5Gzxv7ywT8NtaBFzd83xL2zbXFfDZJBclOaZtu3tVXQPNHw7wW237fPhvsLNjm+tjPrb9Cu+0ia/3mCdjTbIEeChwAeP3e9WOzdffsefsxnz9256352zwvN0PE/DbyhRt86FUzKOr6mHAEcDLkjx2hr7z9b8BTD+2uTxrNmddAAAEX0lEQVTm9wK/DTwEuAb4+7Z9zo81yd7Ax4E/r6qfzNR1irY5NVbtsvn6O/ac3ZiPf9vz9pwNnrf7ZQJ+W1uAA3ueLwa2DimWWVNVW9uf1wKfpPlK60cTX1O2P69tu8+H/wY7O7Y5O+aq+lFV3VxVtwDvo/ndwhwfa5I70pzEz6yqT7TNY/N7Vd/m5e/Yc/b8/duer+ds8Ly9M0zAb+tCYFmSpUl2B9YAa4cc0+2SZK8k+0xsA4cDl9GMa+Lu4qOAf2m31wIvbO9QPhS4YeLrozlkZ8d2LnB4kru2Xwce3raNvElzPZ9O87uFZqxrkuyRZCmwDPg6c+AzniTAqcAVVfX2nl1j83tV30b+87yzPGfP77/t+XjOBs/bO23Yd4GO4oPmztxv09x1/PphxzML47k3zV3TlwAbJ8YE7A98HvhO+/NubXuAk9vxXwqsGPYYdjC+j9B8jfcrmv9zPnpXxga8hOaml83Ai4c9rp0Y6xntWDbQnNDu0dP/9e1YNwFH9LSP9GcceAzNV44bgIvbx5Hz9ffq43Z/Xkb687wL4/GcPU/+tsflnN3G6Hl7Jx6uhClJkiR1yCkokiRJUodMwCVJkqQOmYBLkiRJHTIBlyRJkjpkAi5JkiR1yARcYyHJf7Q/lyR53iy/9l9O9V6SpF3jOVvznWUINVaSPB54bVU9eSeOWVBVN8+w/2dVtfdsxCdJupXnbM1XXgHXWEjys3bzzcDvJrk4yauSLEjy1iQXJtmQ5E/a/o9Pcl6Sf6JZIIAk/5zkoiQbkxzTtr0ZuFP7emf2vle7utdbk1yW5NIkz+l57fOTnJPkW0nObFcQI8mbk1zexvK2Lv8bSdKo8Jyt+W63YQcgdew4eq6mtCflG6rqEUn2AL6S5LNt35XAg6rqqvb5S6rquiR3Ai5M8vGqOi7JsVX1kCne6xnAQ4BDgAPaY77U7nso8EBgK/AV4NFJLqdZlvj+VVVJ7jLro5ekucVztuYlr4Br3B0OvDDJxcAFNEvmLmv3fb3nRA7wiiSXAF8DDuzpN53HAB+pqpur6kfAF4FH9Lz2lqq6hWa53iXAT4AbgfcneQbwi9s9OkmaXzxna14wAde4C/DyqnpI+1haVRNXU37+607NPMTfBx5ZVYcA3wT27OO1p/PLnu2bgd2qajvNFZyPA08DPrNTI5Gk+c9ztuYFE3CNm58C+/Q8Pxf4syR3BEhy3yR7TXHcfsD1VfWLJPcHDu3Z96uJ4yf5EvCcds7iQuCxwNenCyzJ3sB+VbUO+HOar0IlaZx5zta85BxwjZsNwPb2a8kPAu+k+SrxG+1NNdtormRM9hngT5NsADbRfKU54RRgQ5JvVNXze9o/CTwSuAQo4C+q6j/bfwymsg/wL0n2pLkS86pdG6IkzRueszUvWYZQkiRJ6pBTUCRJkqQOmYBLkiRJHTIBlyRJkjpkAi5JkiR1yARckiRJ6pAJuCRJktQhE3BJkiSpQybgkiRJUof+P15M2Ak8+3rIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_text=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the momentums don't change at all so we might want to hide them. This is possible with just a simple option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAF65JREFUeJzt3X+0XWV95/H3x0RAERAhOjaBJg7xR3AtlUbEH3VZsRC0Y9TiMk6dsiozjC0o2lon1FV0WMNaok4df6AdKlRkGANFrWlXKtKi0toaCIpAgqm3oEMKShyYgFp+JHznj7Njz7qce+/J3dm55+S+X2vddfd5zrP3eZ697skne+9nPztVhSRJs/W4uW6AJGm8GSSSpFYMEklSKwaJJKkVg0SS1IpBIklqxSCRJLVikEiSWjFIJEmtLJzrBuwLRx55ZC1dunSumyFJY+PGG2/8cVUtGqbuvAiSpUuXsmnTprluhiSNjSQ/GLaup7YkSa0YJJKkVgwSSVIrBokkqRWDRJLUSqdBkmRVkq1JJpKsHfD+gUmuaN7fmGRpU35Ekq8m+UmST0xa55eS3NKs87Ek6bIPkqTpdRYkSRYAFwKnACuANydZMana6cB9VXUM8BHggqb8QeAPgXcP2PSngDOA5c3Pqr3feknSsLq8j+R4YKKqbgdIsg5YDWzpq7MaeH+zfBXwiSSpqp8Cf5fkmP4NJnk6cGhV/UPz+rPA64C/6qIDH/ub77Fz16NdbFotHLv4ME4+9t/MdTMkNboMksXAnX2vtwEvmqpOVe1MsgM4AvjxNNvcNmmbiwdVTHIGvSMXjj766D1tOwB//PV/4l8e2TWrddWNKnjqIQcaJNII6TJIBl27qFnUmVX9qroIuAhg5cqV021zSlvO86zZqHnvF2/h6s0/nOtmSOrT5cX2bcBRfa+XAHdNVSfJQuAw4N4Ztrlkhm1KkvahLoPkBmB5kmVJDgDWAOsn1VkPnNYsnwpcW1VTHj1U1d3AA0lOaEZr/Sbwpb3fdEnSsDo7tdVc8zgLuBpYAFxSVZuTnAdsqqr1wMXAZUkm6B2JrNm9fpLvA4cCByR5HXBSVW0Bfhv4DPAEehfZO7nQLkkaTqez/1bVBmDDpLJz+5YfBN44xbpLpyjfBDx377VSktSGd7Zr7Ex98lPSXDBINFacx0AaPQaJJKkVg0SS1IpBIklqxSCRJLVikGjsOGhLGi0GicZKBk63JmkuGSSSpFYMEklSKwaJJKkVg0SS1IpBorEzzZMGJM0Bg0Rjxbm2pNFjkEiSWjFIJEmtGCSSpFYMEklSKwaJxo5jtqTRYpBorDhoSxo9BokkqRWDRJLUikEiSWrFIJEktWKQaOw41ZY0WgwSjZU42ZY0cgwSSVIrBokkqRWDRJLUikEiSWrFINHY8QmJ0mgxSCRJrRgkkqRWDBJJUisGiSSplU6DJMmqJFuTTCRZO+D9A5Nc0by/McnSvvfOacq3Jjm5r/xdSTYnuTXJ55Ic1GUfJEnT6yxIkiwALgROAVYAb06yYlK104H7quoY4CPABc26K4A1wLHAKuCTSRYkWQy8A1hZVc8FFjT1NI84ZksaLV0ekRwPTFTV7VX1MLAOWD2pzmrg0mb5KuDE9CZTWg2sq6qHquoOYKLZHsBC4AlJFgJPBO7qsA8aMU61JY2eLoNkMXBn3+ttTdnAOlW1E9gBHDHVulX1z8CHgf8D3A3sqKqvDPrwJGck2ZRk0/bt2/dCdyRJg3QZJIP+7zj5rMRUdQaWJzmc3tHKMuAXgIOTvGXQh1fVRVW1sqpWLlq0aA+aLUnaE10GyTbgqL7XS3jsaaif12lOVR0G3DvNuq8C7qiq7VX1CPAF4CWdtF6SNJQug+QGYHmSZUkOoHdRfP2kOuuB05rlU4Frqzf/xXpgTTOqaxmwHLie3imtE5I8sbmWciJwW4d90Cjyars0UhZ2teGq2pnkLOBqeqOrLqmqzUnOAzZV1XrgYuCyJBP0jkTWNOtuTnIlsAXYCZxZVbuAjUmuAr7VlH8buKirPkiSZtZZkABU1QZgw6Syc/uWHwTeOMW65wPnDyh/H/C+vdtSjYsMvHwmaS55Z7skqRWDRJLUikEiSWrFINHYcdCWNFoMEklSKwaJxopzbUmjxyCRJLVikEiSWjFIJEmtGCQaO73p2CSNCoNEktSKQaKx4qAtafQYJJKkVgwSSVIrBokkqRWDRGPHMVvSaDFIJEmtGCQaK861JY2eoYMkycFdNkSSNJ5mDJIkL0myBbitef28JJ/svGWSpLEwzBHJR4CTgf8LUFXfAV7eZaMkSeNjqFNbVXXnpKJdHbRFGopTbUmjZeEQde5M8hKgkhwAvIPmNJckScMckbwNOBNYDGwDng/8TpeNkqYSh21JI2eYI5JnVdVv9BckeSnwjW6aJEkaJ8MckXx8yDJJ0jw05RFJkhcDLwEWJfndvrcOBRZ03TBJ0niY7tTWAcCTmjqH9JXfD5zaZaOk6ZSzbUkjZcogqaqvA19P8pmq+sE+bJMkaYwMc7H9Z0k+BBwLHLS7sKpe2VmrpCk4ZksaPcNcbL8c+C6wDPivwPeBGzpskyRpjAwTJEdU1cXAI1X19ap6K3BCx+2SJI2JYU5tPdL8vjvJa4C7gCXdNUmSNE6GCZL/luQw4Pfo3T9yKPCuTlslTcO5tqTRMu2prSQLgOVVtaOqbq2qX6mqX6qq9cNsPMmqJFuTTCRZO+D9A5Nc0by/McnSvvfOacq3Jjm5r/zJSa5K8t0ktzX3u0iS5si0QVJVu4DXzmbDTQhdCJwCrADenGTFpGqnA/dV1TH0pqu/oFl3BbCG3kixVcAnm+0BfBT4clU9G3geTiA5vzhsSxo5w1xs//skn0jyy0mO2/0zxHrHAxNVdXtVPQysA1ZPqrMauLRZvgo4Mb1Z+VYD66rqoaq6A5gAjk9yKL1noVwMUFUPV9X/G6ItkqSODHON5CXN7/P6ygqY6T6SxUD/c0y2AS+aqk5V7UyyAziiKf/mpHUXA/8CbAf+NMnzgBuBs6vqp0P0Q5LUgRmDpKp+ZZbbHnQSYvJl0qnqTFW+EDgOeHtVbUzyUWAt8IeP+fDkDOAMgKOPPnoPmi1J2hNDPSFxlrYBR/W9XkJv6PDAOkkWAocB906z7jZgW1VtbMqvohcsj1FVF1XVyqpauWjRopZd0Shx0JY0WroMkhuA5UmWNU9WXANMHu21HjitWT4VuLaqqilf04zqWgYsB66vqh/Se2Ljs5p1TgS2dNgHSdIMhrlGMivNNY+zgKvpTTt/SVVtTnIesKkZQnwxcFmSCXpHImuadTcnuZJeSOwEzmxGkAG8Hbi8Cafbgd/qqg8aPXHYljRyZgySJG8YULwDuKWq7plu3araAGyYVHZu3/KDwBunWPd84PwB5TcBK2dqtyRp3xjmiOR04MXAV5vXr6A3ouqZSc6rqss6apskaQwMEySPAs+pqh8BJHka8Cl6Q3mvAwwSSZrHhrnYvnR3iDTuAZ5ZVffyrxM6SvuOw7akkTLMEcnfJvlL4M+a178OXJfkYMC7yiVpnhsmSM6kFx4vpXej4GeBzzfDdGd7s6I0K3HQljRyhrmzvejd+HdV982RJI2bGa+RJHlDku8l2ZHk/iQPJLl/XzROkjT6hjm19UHg31WV07VLkh5jmFFbPzJENErKYVvSSBnmiGRTkiuAPwce2l1YVV/orFWSpLExTJAcCvwMOKmvrACDRPucg7ak0TPMqC0nRZQkTWnKIEnynqr6YJKPM+Be4qp6R6ctkySNhemOSHZfYN+0LxoiSRpPUwZJVf1F8/vSfdccaWbloC1ppAzzPJJnAu8GlvbXr6pXdtcsSdK4GGbU1p8Bfwx8Gtg1Q12pU861JY2eYYJkZ1V9qvOWSJLG0jB3tv9Fkt9J8vQkT9n903nLJEljYZgjktOa37/fV1bAM/Z+cyRJ42baIEnyOOAtVfWNfdQeaUYO2pJGy7SntqrqUeDD+6gtkqQxNMw1kq8k+fXE8TKae3G2LWnkDHON5HeBg4GdSR6kN29eVdWhnbZMkjQWhpm08ZB90RBJ0nga5oiEJIcDy4GDdpdV1XVdNUqSND6GmSLlPwJnA0uAm4ATgH8AnCJFc6KcbEsaKcNcbD8beCHwg6r6FeAFwPZOWyVJGhvDBMmDVfUgQJIDq+q7wLO6bZY0mGMHpdEzzDWSbUmeTO+Z7dckuQ+4q9tmSZLGxTCjtl7fLL4/yVeBw4Avd9oqSdLYGHbU1suA5VX1p0kWAYuBOzptmSRpLMx4jSTJ+4D/ApzTFD0e+F9dNkqajmO2pNEyzMX21wOvBX4KUFV3Ad6kKEkChguSh6s3cL8AkhzcbZOkqTloSxo9wwTJlUn+J/DkJP8J+GvgT4bZeJJVSbYmmUiydsD7Bya5onl/Y5Klfe+d05RvTXLypPUWJPl2kr8cph2SpO4MM2rrw0l+Fbif3v0j51bVNTOtl2QBcCHwq8A24IYk66tqS1+104H7quqYJGuAC4A3JVkBrAGOBX4B+Oskz6yq3c+MPxu4DXDiSEmaY8MckVBV11TV71fVu4cJkcbxwERV3V5VDwPrgNWT6qwGLm2WrwJObKarXw2sq6qHquoOYKLZHkmWAK8BPj1kOyRJHZryiCTJAwweIDPsNPKLgTv7Xm8DXjRVnaramWQHcERT/s1J6y5ulv8H8B684D9vOdWWNFqmDJK9MH38oOuik/8JmKrOwPIkvwbcU1U3JnnFtB+enAGcAXD00UfP3FpJ0qwMdWprlrYBR/W9XsJjp1b5eZ0kC+ndNX/vNOu+FHhtku/TO1X2yiQD72mpqouqamVVrVy0aFH73mg0ONmWNHK6DJIbgOVJliU5gN7F8/WT6qwHTmuWTwWubYYarwfWNKO6ltF7Fsr1VXVOVS2pqqXN9q6tqrd02AdJ0gyGmiJlNpprHmcBVwMLgEuqanOS84BNVbUeuBi4LMkEvSORNc26m5NcCWwBdgJn9o3YkiSNkM6CBKCqNgAbJpWd27f8IPDGKdY9Hzh/mm1/Dfja3minJGn2ujy1JUmaBwwSSVIrBonGimO2pNFjkEiSWjFIJEmtGCSSpFYMEo2lcsItaWQYJJKkVgwSjRWn2pJGj0EiSWrFIJEktWKQSJJaMUg0lhy0JY0Og0SS1IpBorESZ9uSRo5BIklqxSCRJLVikEiSWjFINJYctCWNDoNEktSKQaKx4lxb0ugxSCRJrRgkkqRWDBJJUisGicaST0iURodBIklqxSDRWHHQljR6DBJJUisGiSSpFYNEktSKQaKx5JgtaXQYJJKkVgwSjRXn2pJGj0EiSWrFIJEktdJpkCRZlWRrkokkawe8f2CSK5r3NyZZ2vfeOU351iQnN2VHJflqktuSbE5ydpftlyTNrLMgSbIAuBA4BVgBvDnJiknVTgfuq6pjgI8AFzTrrgDWAMcCq4BPNtvbCfxeVT0HOAE4c8A2NQ841ZY0Oro8IjkemKiq26vqYWAdsHpSndXApc3yVcCJSdKUr6uqh6rqDmACOL6q7q6qbwFU1QPAbcDiDvsgSZpBl0GyGLiz7/U2HvuP/s/rVNVOYAdwxDDrNqfBXgBs3Itt1oiLw7akkdNlkAz6xk8+ITFVnWnXTfIk4PPAO6vq/oEfnpyRZFOSTdu3bx+yyZKkPdVlkGwDjup7vQS4a6o6SRYChwH3TrduksfTC5HLq+oLU314VV1UVSurauWiRYtadkWSNJUug+QGYHmSZUkOoHfxfP2kOuuB05rlU4Frq/fEovXAmmZU1zJgOXB9c/3kYuC2qvqjDtsuSRrSwq42XFU7k5wFXA0sAC6pqs1JzgM2VdV6eqFwWZIJekcia5p1Nye5EthCb6TWmVW1K8nLgP8A3JLkpuaj/qCqNnTVD42mcrYtaWR0FiQAzT/wGyaVndu3/CDwxinWPR84f1LZ3+GzjSRppHhnuySpFYNEktSKQSJJasUgkSS1YpBoLDnXljQ6DBKNFWdIkUaPQSJJasUgkSS1YpBIkloxSCRJrRgkkqRWDBKNlTjVmjRyDBJJUisGiSSpFYNEktSKQSJJaqXTB1tJXXnNx/6WxzlfijStw594AFe+7cWdf45BorHyquc8lS1338+uRx+d66ZII+/Qgx6/Tz7HINFYWf60Q/j4m18w182Q1MdrJJKkVgwSSVIrBokkqRWDRJLUikEiSWrFIJEktWKQSJJaMUgkSa2kqua6DZ1Lsh34wSxXPxL48V5szjia7/tgvvcf3Acw//bBL1bVomEqzosgaSPJpqpaOdftmEvzfR/M9/6D+wDcB9Px1JYkqRWDRJLUikEys4vmugEjYL7vg/nef3AfgPtgSl4jkSS14hGJJKkVg2QKSVYl2ZpkIsnauW5Pl5J8P8ktSW5Ksqkpe0qSa5J8r/l9eFOeJB9r9svNSY6b29bPTpJLktyT5Na+sj3uc5LTmvrfS3LaXPRltqbYB+9P8s/N38JNSV7d9945zT7YmuTkvvKx/K4kOSrJV5PclmRzkrOb8nn1d7BXVJU/k36ABcA/Ac8ADgC+A6yY63Z12N/vA0dOKvsgsLZZXgtc0Cy/GvgrIMAJwMa5bv8s+/xy4Djg1tn2GXgKcHvz+/Bm+fC57lvLffB+4N0D6q5ovgcHAsua78eCcf6uAE8HjmuWDwH+sennvPo72Bs/HpEMdjwwUVW3V9XDwDpg9Ry3aV9bDVzaLF8KvK6v/LPV803gyUmePhcNbKOqrgPunVS8p30+Gbimqu6tqvuAa4BV3bd+75hiH0xlNbCuqh6qqjuACXrfk7H9rlTV3VX1rWb5AeA2YDHz7O9gbzBIBlsM3Nn3eltTtr8q4CtJbkxyRlP2tKq6G3pfOOCpTfn+vG/2tM/76744qzl1c8nu0zrs5/sgyVLgBcBG/DvYYwbJYBlQtj8Pb3tpVR0HnAKcmeTl09Sdb/sGpu7z/rgvPgX8W+D5wN3Af2/K99t9kORJwOeBd1bV/dNVHVC2X+yDtgySwbYBR/W9XgLcNUdt6VxV3dX8vgf4Ir3TFT/afcqq+X1PU31/3jd72uf9bl9U1Y+qaldVPQr8Cb2/BdhP90GSx9MLkcur6gtN8bz/O9hTBslgNwDLkyxLcgCwBlg/x23qRJKDkxyyexk4CbiVXn93jz45DfhSs7we+M1mBMsJwI7dpwH2A3va56uBk5Ic3pwCOqkpG1uTrne9nt7fAvT2wZokByZZBiwHrmeMvytJAlwM3FZVf9T31rz/O9hjc321f1R/6I3Q+Ed6I1LeO9ft6bCfz6A30uY7wObdfQWOAP4G+F7z+ylNeYALm/1yC7Byrvswy35/jt6pm0fo/Y/y9Nn0GXgrvQvPE8BvzXW/9sI+uKzp4830/uF8el/99zb7YCtwSl/5WH5XgJfROwV1M3BT8/Pq+fZ3sDd+vLNdktSKp7YkSa0YJJKkVgwSSVIrBokkqRWDRJLUikEizSDJ3ze/lyb593t5238w6LOkceLwX2lISV5Bb2bcX9uDdRZU1a5p3v9JVT1pb7RPmisekUgzSPKTZvEDwC83z+l4V5IFST6U5IZmksP/3NR/RfOci/9N78Y1kvx5Mynm5t0TYyb5APCEZnuX939Wc/f0h5Lcmt6zYt7Ut+2vJbkqyXeTXN7coU2SDyTZ0rTlw/tyH2l+WzjXDZDGyFr6jkiaQNhRVS9MciDwjSRfaeoeDzy3elOuA7y1qu5N8gTghiSfr6q1Sc6qqucP+Kw30Js48XnAkc061zXvvQA4lt58Tt8AXppkC70pTZ5dVZXkyXu999IUPCKRZu8kenMv3URv+vEj6M1BBXB9X4gAvCPJd4Bv0pvgbznTexnwuepNoPgj4OvAC/u2va16EyveBCwF7gceBD6d5A3Az1r3ThqSQSLNXoC3V9Xzm59lVbX7iOSnP6/Uu7byKuDFVfU84NvAQUNseyoP9S3vAhZW1U56R0Gfp/cgpi/vUU+kFgwSaXgP0Hsk625XA7/dTEVOkmc2MyhPdhhwX1X9LMmz6T2mdbdHdq8/yXXAm5rrMIvoPRb3+qka1jxT47Cq2gC8k95pMWmf8BqJNLybgZ3NKarPAB+ld1rpW80F7+3862NZ+30ZeFuSm+nNnPvNvvcuAm5O8q2q+o2+8i8CL6Y3K3MB76mqHzZBNMghwJeSHETvaOZds+uitOcc/itJasVTW5KkVgwSSVIrBokkqRWDRJLUikEiSWrFIJEktWKQSJJaMUgkSa38f0Bpvbxqa3C3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_text=False, show_moms=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's complicate things a bit more and say we want to train at constant 1e-2 lr for one epoch, then decrease our learning rate to 1e-3 during one epoch and train for one last epoch at 1e-3.\n",
    "\n",
    "We have currently fours methods of decay (on top of NO that means constant): linear, cosine, exponential or polynomial. Let's have a look at what each of them does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-2), \n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = (1e-2,1e-3), lr_decay=DecayType.LINEAR),\n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f1a5b991a5849b8ace30717b8a61d01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.398595   4.563467   0.142     \n",
      "    1      3.434187   4.567539   0.1435                                                                                \n",
      "    2      3.402653   4.554236   0.1413                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.55424]), 0.1413]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear is simply going from a to b with a line. The formula that gives the learning rate at batch i over n is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_i = start_lr + (end_lr - start_lr) * i/n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(to run the cell above give a value to all the parameters: start_lr, end_lr, i and n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW5x/HPk4QQwmpCAGWRsEqAFjWlimK1oOLGovZKV9vaWqu2tdXbq7ihdam9au2mt7Zal9oqRRGwuCtq3YOCECQaQUpEEQWRHQLP/eMcbIxJGDKZnJk53/fr9XvN5De/c+Y5Z2Z4OMtzjrk7IiIizZUTdQAiIpLZlEhERCQpSiQiIpIUJRIREUmKEomIiCRFiURERJKiRCIiIklRIhERkaQokYiISFLyog6gNXTt2tX79u0bdRgiIhlj3rx5H7h7SSJjY5FI+vbtS0VFRdRhiIhkDDNbnuhY7doSEZGkKJGIiEhSlEhERCQpSiQiIpIUJRIREUmKEomIiCRFiURERJKiRCIiIklRIhERkaQokYhI8h57LGgSS7G4RIqIpNgVVwSPY8dGG4dEQlskIiKSFCUSERFJihKJiIgkRYlERESSooPtIpK8P/4x6ggkQkokIpK8wYOjjkAipF1bklkOPzxokl5mzw6axJK2SEQkedddFzyecEK0cUgktEUiIiJJUSIREZGkKJGIiEhSUppIzGycmVWZWbWZnd/A623N7J7w9RfNrG/YX2xmT5rZBjP7fb1pDjSzheE0vzUzS+UySJqZPj1oIpI2Unaw3cxygT8ARwI1wMtmNsvdF9cZdhqw1t0HmNlk4BrgFGALcDEwLGx13QScDrwAzAHGAQ+majkkzXTtGnUE0pA774w6AolQKs/aGglUu/tSADO7G5gA1E0kE4Cp4fPpwO/NzNx9I/AvMxtQd4ZmtjfQyd2fD/++A5hIihLJ7AUr2emeillLM/WeNY3i9vnse+5ZUYcidfXuHXUEEqFUJpKewIo6f9cAX2xsjLvXmtk6oBj4oIl51tSbZ8+GBprZ6QRbLvTp02dPYwfg59NfY/P2Hc2aVlLj7r/dyrvAu5Mmc1C/4qjDkV3uuSd4POWUaOOQSKQykTR07KL+f+8TGdOs8e5+M3AzQHl5ebM2K+b8ZDSuLZK00uPpLlS9t57zZixkzk9G0zYvN+qQBOCmm4JHJZJYSmUiqQHqbu/2AlY2MqbGzPKAzsCa3cyz127m2WJKu7ZP1aylufJz6du1PW+t3sjNTy3lR2MGRh2RSOyl8qytl4GBZlZqZvnAZGBWvTGzgFPD5ycDT3gTmwDu/i6w3swOCs/W+hYws+VDl3S2V2Ebjhu+N797spq3P9gYdTgisZeyROLutcDZwMPA68A0d680s8vNbHw47Bag2MyqgZ8Bn5wibGZvA9cD3zazGjMrC1/6IfBnoBp4C52xFUuXnFBG29wcLrp/kXY/ikQspdfacvc5BKfo1u27pM7zLcBXGpm2byP9FXz2lGCJiznB16l7YQHnHT2YS2dVMmvBSiaMaPCcCxFpBapsl8xSWBg04BsH7cvnenXmFw8sZt2m7REHFnMqFI01JRLJLDfeGDQgN8e4atJw1mzcxjUPL4k4sJjr2lXFojGmRCKZZdq0oIWG9ezMdw4p5W8v/pt5y5s64U9S6rbbgiaxpEQiGe9nRw5i784FTLlvEdt37Iw6nHhSIok1JRLJeO3b5jF1/FCqVq3nln8tizockdhRIpGscPTQHhxZ1p0bHnuDFWs2RR2OSKwokUjWuGz8UHLMuGSmaktEWpMSiWSWuXOD1oB9urTjZ0cO4smq1Ty46L1WDUskzpRIJKt8e1RfyvbuxGWzK1m/RbUlrWbOnE+KRSV+lEgks1x7bdAakZebw1UnDuf99Vu57pE3WjGwmKtTKCrxo0QimeWBB4LWhBG9u/DNg/bl9uffZsGKj1onrrirUygq8aNEIlnpvKMHU9KhLVNmLKRWtSWpV69QVOJFiUSyUqeCNlxyQhmVKz/m9ueXRx2OSFZTIpGsddzwvTl8cAnXP1LFyo82Rx2OSNZSIpHM0q5d0BJgZvxiwjB2uDN1VmWKAxOJLyUSySwPPhi0BPUuKuTHYwbyyOJVPLp4VQoDE4kvJRLJet8f3Y9B3Ttw6cxFbNxaG3U42amJQlHJfkokkll+8Yug7YE2uTlcNWk4K9dt4YbHVFsi0tKUSCSzPP540PZQed8ivjqyN7c++zaVK9elILCY202hqGQ3JRKJjf8Ztx9d2rVhyoxF7Nipizq2qAQKRSV7KZFIbHQpzOei44ewYMVH/O1F1ZaItBQlEomViSN6csiAYn71UBXvf7wl6nBEsoISiWSW4uKgNZOZccXE4WzdsZPLHljcgoGJxJcSiWSWe+8NWhJKu7bnrMMH8M/X3mVu1fstFFjM7UGhqGQfJRKJpTMO70e/kvZcPHMRm7ftiDqczLeHhaKSXZRIJLNccEHQktQ2L5crJw5nxZrN/O6JN1sgMJH4UiKRzPL880FrAQf3L+akA3px89NLqXpvfYvMM7aaUSgq2SOlicTMxplZlZlVm9n5Dbze1szuCV9/0cz61nntgrC/ysyOrtP/UzOrNLNFZvZ3MytI5TJIdrvwuCF0KMjjwhkL2anakuZrZqGoZIeUJRIzywX+ABwDlAFfNbOyesNOA9a6+wDg18A14bRlwGRgKDAOuNHMcs2sJ/BjoNzdhwG54TiRZilqn8+UY4ZQsXwt0ypWRB2OSEZK5RbJSKDa3Ze6+zbgbmBCvTETgNvD59OBMWZmYf/d7r7V3ZcB1eH8APKAdmaWBxQCK1O4DBIDXynvxcjSIq5+cAkfbNgadTgiGSeViaQnUPe/eDVhX4Nj3L0WWAcUNzatu78DXAv8G3gXWOfuj6QkeklPvXoFrQWZGVdNGsambbVc+c/XW3TeInGQykRiDfTV3wnd2JgG+81sL4KtlVJgH6C9mX2jwTc3O93MKsysYvXq1XsQtqS1v/41aC1sQLeO/OCw/sx49R2erf6gxeef9ZIsFJXMlspEUgP0rvN3Lz67G+qTMeGuqs7AmiamHQssc/fV7r4duA8Y1dCbu/vN7l7u7uUlJSUtsDiS7c7+8gD2LS7kovsXsWW7akv2SAsUikrmSmUieRkYaGalZpZPcFB8Vr0xs4BTw+cnA0+4u4f9k8OzukqBgcBLBLu0DjKzwvBYyhhA+yLi5JxzgpYCBW1y+cWEYSz7YCM3zX0rJe8hko3yUjVjd681s7OBhwnOrrrV3SvN7HKgwt1nAbcAd5pZNcGWyORw2kozmwYsBmqBs9x9B/CimU0HXgn7XwVuTtUySBqaPz+lsz9sUAnjP78PN819i/Ej9qF/SYeUvl/W2FUkevXV0cYhkbBgAyC7lZeXe0VFRdRhSEs4/PDgMYW3dX1//RbGXPcUQ/fpxN+/fxDBxq80qRU+F2ldZjbP3csTGavKdpF6unUs4H/G7ccLS9dw3yvvRB2OSNpTIhFpwNdG9mH/Pl24cs7rrN24LepwRNKaEolklkGDgpZiOTnGVZOGs27zdq5+UOdziDQlZQfbRVLi5tY7t2LI3p343qGl/PHppZx8YG9Glha12ntnnBYuEpXMoi0SkSb8ZOxAenZpx5QZC9lWuzPqcNJXigpFJTMokUhmOf30oLWSwvw8Lp8wlOr3N/CnZ5a22vuKZBIlEsksb7wRtFY0Zkh3jhnWg98+/ibLP9zYqu+dMVJYKCrpT4lEJAGXnjCUNrk5XHT/IuJQe7XH5s9PebGopC8lEpEE9OhcwLlHDeKZNz9g9mvvRh2OSFpJOJGYWftUBiKS7r51cF+G9+zM5bMXs27z9qjDEUkbu00kZjbKzBYTXhzRzD5vZjemPDKRhowYEbQI5OYYV584nDUbt/Krh5ZEEoNIOkqkjuTXwNGEV+519wVmdlhKoxJpzA03RPr2w3p25tRRfbntubc56cBeHNBnr0jjSRutUCQq6SuhXVvuXv9m1rpZg8TWuUcNpnvHAqbct5DtO1RbAgSFoq1YLCrpJZFEssLMRhHcoTDfzM5D9wCRqHzjG0GLUIe2eUwdP5Ql763nL88uizQWkXSQSCI5AziL4D7qNcAI4MxUBiXSqJqaoEXs6KHdGTukG79+9E1q1m6KOpzotXKhqKSXRBLJYHf/urt3d/du7v4NYEiqAxNJZ2bG1PFDAbh0ZqVqSyIoFJX0kUgi+V2CfSKx0muvQn565EAeX/I+D1e+F3U4IpFp9KwtMzsYGAWUmNnP6rzUieDWuSKx951DSpnx6kqmzlrMIQO60rGgTdQhibS6prZI8oEOBMmmY532MXBy6kMTacDBBwctTbTJzeGqScNYtX4L1z2iXTsST41ukbj7U8BTZnabuy9vxZhEGnf11VFH8Bn799mLr3+xD3c8/zYnHdCL4b06Rx1S64uoSFTSg+3uIKGZlQA/B4YCBbv63f3LqQ2t5ZSXl3tFRUXUYUgWW7d5O2Ovf4oenQq4/6xDyM2xqEMSSYqZzXP38kTGJnKw/S5gCVAKXAa8Dbzc7OhEknHSSUFLM53bteGS48tY+M467nj+7ajDEWlViSSSYne/Bdju7k+5+3eBg1Icl0jDPvwwaGno+M/tzWGDSrj24SreXbc56nBaVxoUikp0Ekkkuy5z+q6ZHWdm+wO6QbNIPWbGFROGUbvTuWzW4qjDaV1pUigq0UgkkVxhZp2Bc4HzgD8DP01pVCIZqk9xIT8eM5CHKt/j8ddXRR2OSKtoMpGYWS4w0N3Xufsidz/C3Q9091mtFJ9Ixvn+6H4M7NaBS2ZWsmlbbdThiKRck4nE3XcA41spFpHdGzMmaGksPy+Hq04czjsfbeaGx96MOhyRlEtk19ZzZvZ7MxttZgfsaonM3MzGmVmVmVWb2fkNvN7WzO4JX3/RzPrWee2CsL/KzI6u09/FzKab2RIzez2swJe4uPjioKW5L/Qt4pTy3tzyr2UsXvlx1OGkXpoVikrrSqSO5MkGun13dSThbrE3gCMJrhr8MvBVd19cZ8yZwOfc/QwzmwxMcvdTzKwM+DswEtgHeAwY5O47zOx24Bl3/7OZ5QOF7v5RU7GojkSi8NGmbYy57in6FBdy7xmjyFFtiWSQFq0jCY+L1G+JFCOOBKrdfam7bwPuBibUGzMBuD18Ph0YY2YW9t/t7lvdfRlQDYw0s07AYcAtYWzbdpdEJMscc0zQMkCXwnwuPG4Ir/77I/720r+jDkckZRK6Q2Iz9QTq3lmxJuxrcIy71wLrgOImpu0HrAb+Ymavmtmfzax9asKXtLR5c9AyxKT9ezKqfzHXPLSE99dviTqc1EnTQlFpHalMJA1tx9ffj9bYmMb684ADgJvcfX9gI/CZYy8AZna6mVWYWcXq1asTj1qkBZkZv5g4jK3bd/KLB7L4xqJpXCgqqZfKRFID9K7zdy9gZWNjzCwP6AysaWLaGqDG3V8M+6cTJJbPcPeb3b3c3ctLSkqSXBSR5utf0oEzj+jP7AUrefoN/adGss9uE4mZndhAG2Nm3XYz6cvAQDMrDQ+KTwbq15/MAk4Nn58MPOHB0f9ZwOTwrK5SYCDwkru/R3AP+cHhNGOAmJUQSyb64eH96de1PRfdv4gt23dEHY5Ii2r0MvJ1nAYcDOw6e+tw4AVgkJld7u53NjSRu9ea2dnAwwQ3wrrV3SvN7HKgIixqvAW408yqCbZEJofTVprZNIIkUQucFda0APwIuCtMTkuB7+zpQme6jRs3MnfuXN58803Wr19PQUEB3bp149BDD6V///4ArF27lmeeeYa33nqLDRs2UFhYSHFxMSNGjGD48OHk5gb3Jps6deon823Tpg0dOnSgV69efOELX6BPnz5RLF7Tjj8+6ggatbvP5YpJwzjld49zxpV/5POdt2bX55KmYv1baUWJJJKdwBB3XwVgZt2Bm4AvAk8DDSYSAHefA8yp13dJnedbgK80Mu2VwJUN9M8HEjolLVtNmzaN7du3M378eIqKiti4cSPLly9nc3gQeuXKldx+++2UlJRw7LHH0rVrV7Zv387q1at55ZVXKCoq+tQXf/z48QwaNIja2lrWrl3LggUL+Mtf/sLYsWM55JBDolrMhp13XtQRNGp3n0vfdtvYd9W/mPNmDl+dcirl++2bPZ9LmhaJxvq30ooSSSR9dyWR0PsENR1rzGx7YxNJamzZsoXly5fzrW99i379+gHQpUsXevYMTohzd2bMmEFxcTGnnXYawdnUgR49ejB8+HDq1w4VFBTQoUOHT+ZVWlpKx44defzxxxkyZAhFRUWttHSZK9HP5cSD9+P2tQO4ZeFWjjy4iJwcy47PJQ2LRPVbaT2JHGx/xsweMLNTzexUYCbwdHjarWo4Wll+fj75+flUVVVRW/vZ6zi99957rF69mlGjRn3qh1FXY/11HXzwwbg7S5YsSTrmFnX44UFLM4l+Lkd9+UtMOXYIL729hunzPn213Iz+XNJQ7H8rrSiRLZKzgJOAQwhOy70DuDc8KH5ECmOTBuTk5DBx4kRmz57NvHnz6NGjB3369KGsrIxevXrxYXgKZnFx8SfTbNmyheuvv/6Tv0ePHs3o0aObfJ/CwkLat2/P2rVrU7MgWWZPPpevdO/B9Hk1XDFrAa898BcK2wY/w4z+XHYViT74YLRx1KHfSuvZbSIJE8b0sEkaKCsrY9CgQSxfvpyamhqqq6t57rnnGDNmDHvttddnxrdt25YzzjgDgLvuuosdOxI7a2h3l8+RT0v0c8nJMa6cNJxjbniaNQO/xM8mDs/8zyVNi0T1W2kdiZ7++6aZrTOzj81svZnF4Cp06S0vL4/+/fvzpS99idNOO40DDjiAuXPn0qVLFwA++OCDT8aaGUVFRRQVFX1yBsrubNq0iU2bNjX4Y5PGJfq5DOrekTMO78+cNzawZO1OfS4ppN9K6iVyjORXwHh37+zundy9o7t3SnVgsmdKSkrYuXMnXbt2paSkhGeffZadO3c2e37PPfccZsZ+++3XglHGT1Ofy4++PJA+RYVcNGMRtTsS+6z0uSRPv5WWl8gxklXunsXXdsgsmzZt4h//+Af7778/3bt3Jz8/n5UrV/Lss89SWlpKQUEBEydO5I477uCWW25h9OjRn/xwVqxYwccff/yZA4hbtmxhw4YN7NixgzVr1rBgwQIWLFjAkUcemX5nofzXf0UdQYOa+7mce1gPzvi/RyjMW8HnPzf8U/PMqM8lDcX+t9KKErmM/G+AHsD9wNZd/e5+X2pDaznZdBn52tpannrqKd566y3WrFnDjh076NixI4MHD+awww6jXbt2AKxZs4ZnnnmGpUuXsmHDBvLy8ujevTvDhg3jgAMOIC8v+D9E3SKrvLy8TxVZ7bvvvlEsYkZK5nN5dNkWqrfvxWNXfZtBewe7WzLuc7n22uAxjep89FtJzp5cRj6RRPKXBrrd3b/bnOCikE2JJPY2bQoeCwujjaMFvf/xFsZc/xTDe3bmru99MaFTTkVSbU8SSSJnbcXuEiSSxo49NnicOzfSMFpSt04F/Hzcflx8/yLun/8Ok/bvFXVIInuk0URiZj9391+Z2e/47OXfcfcfpzQykRj5+sg+3DuvhiseeJ0jBnejS2F+1CHtmV1FolmU4CVxTZ21tesAewUwr4EmIi0kJ8e4atJwPtq8nV8+GN8KaclMjW6RuPvs8PH2xsaISMsp26cTpx1ays1PL+WkA3vxhb7xPQtIMksiBYmDzOxmM3vEzJ7Y1VojOJG4OWfsQHp2aceFMxayrbb5tQ0irSmROpJ/AP8H/BnQHXkkWt/+dtQRpFRhfh6XjR/K9+6o4E/PLOWsIwZEHZLIbiWSSGrd/aaURyKSiCxPJABjy7pz9NDu/PbxNznhc/vQpzgDTnVO00JRaR2JXCJltpmdaWZ7m1nRrpbyyEQa8sEHQctyU8cPJS/HuHjmosy4IOCZZwZNYimRRHIq8N/Ac/znjC1V90k0Tj45aFlu787tOPeowTz1xmr+ufDdqMPZvU2b/lMsKrHTZCIxsxzgG+5eWq/1a6X4RGLr1FF9GdazE5fNXsy6zWl+M9Jjj/1PsajETpOJxN13Ate2UiwiUkduWFvy4YatXPtwVdThiDQqkV1bj5jZSaYLAIm0us/16sK3Du7LX19czvwVurO1pKdEEsnPCE4B3qobW4m0vnOPGkS3jm254L6FCd+3RKQ17TaRhDeyynH3fN3YSiL3wx8GLUY6FrRh6glDef3dj/nLs29HHY7IZyRSR4KZ7QUMBAp29bn706kKSqRRp5wSdQSRGDesB1/erxvXP/oGx35ub3p2aRd1SJ8Wg/oeaVwil0j5HvA08DBwWfg4NbVhiTRixYqgxYyZcdn4oQBcOrMy4mga8O1vK5nEWCLHSH4CfAFY7u5HAPsDq1MalUhjvvnNoMVQ76JCzhk7kMdeX8XDle9FHc6nxaRQVBqWSCLZ4u5bAMysrbsvAQanNiwRach3Dy1lvx4duXRmJRu21kYdzn/EpFBUGpZIIqkxsy4E92x/1MxmAitTG5aINKRNbg5XThrOqvVbuP6RN6IORwRI7KytSe7+kbtPBS4GbgEmJjJzMxtnZlVmVm1m5zfwelszuyd8/UUz61vntQvC/iozO7redLlm9qqZPZBIHCLZ5MB99+JrI/tw23PLWPTOuqjDEUloiwQzO9TMvuPuTwHPAz0TmCYX+ANwDFAGfNXMyuoNOw1Y6+4DgF8D14TTlgGTgaHAOODGcH67/IT/3MFRJHZ+Pm4/itq3ZcqMhezYmQEXdZSslshZW5cC/wNcEHa1Af6awLxHAtXuvtTdtwF3AxPqjZkA7LoD43RgTFhBPwG42923uvsyoDqcH2bWCziO4P4oEjfnnhu0mOvcrg0XHz+E12rWcefzb0cdjsRcInUkkwjO1HoFwN1XmlnHBKbrCdQ9T7MG+GJjY9y91szWAcVh/wv1pt21FXQD8HOgyRjM7HTgdIA+ffokEK5khBNOiDqCtDH+8/swfV4N1z7yBuOG7U2PzgW7nyhVYlYkKp+WyK6tbR7cEMEBzKx9gvNu6Npc9bfBGxvTYL+ZHQ+87+7zdvfm7n6zu5e7e3lJScnuo5XMUFUVNMHMuGLiMLbv2MnlD0RcW3LKKbEtFpXEEsk0M/sj0MXMvg88BvwpgelqgN51/u7FZ8/2+mSMmeUBnYE1TUx7CDDezN4m2FX2ZTNLZDebZIsf/CBoAsC+xe350ZcHMGfhezyxZFV0gcS0UFQCiZy1dS3B8Yt7CepHLnH33yUw75eBgWZWamb5BAfPZ9UbM4vgxlkAJwNPhFs/s4DJ4VldpQSXZ3nJ3S9w917u3jec3xPu/o0EYhHJWqcf1p8B3Tpw8f2VbNoWUW1JjAtFJcGzttz9UXf/b3c/z90fTXCaWuBsgkuqvA5Mc/dKM7vczMaHw24Bis2smuAqw+eH01YC04DFwEPAWe6+Y08WTCQu8vNyuHLiMN75aDO/efzNqMORGGr0YLuZreezxzQgOH7hiVwB2N3nAHPq9V1S5/kW4CuNTHslcGUT854LzN1dDCJx8MV+xfxXeS9ueWYZk/bvyX49dIFuaT2NbpHsulx8A02XkRdJQxccM4RO7dpwwX0L2anaEmlFCe3aEkkbF10UNPmMvdrnM+XYIbz674/4+8v/jjociZGE7kcikjbGjo06grR20gE9mT5vBdc8uISjynpQ0rFt67yxikRjTVskklnmzw+aNCioLRnOlu07ueKfi1vvjU84QcWiMaZEIpnlnHOCJo0a0K0DZxzen5nzV/L0G6106yAVisaaEolIFjrz8P6Udm3PxTMXsWV7K5w5r0LRWFMiEclCBW1y+cWEYSz/cBN/eLI66nAkyymRiGSpQwd2ZdL+Pfm/p96i+v31UYcjWUyJRCSLXXjcEArz85gyYxHB1YdEWp4SiWSWq64KmiSka4e2nH/Mfry0bA3/mFcTdTiSpVRHIpll1KioI8g4p5T35t55NVw953XGDulOUfv8ln8TFYnGmrZIJLM891zQJGE5OcaVk4azfkstV81J0R2qx45VsWiMKZFIZpkyJWiyRwb36Mj3D+vH9Hk1PP/Why3/BioUjTUlEpGY+PGXB9K7qB0X3r+QrbUtXFuiQtFYUyIRiYl2+blcPmEYS1dv5I9PLY06HMkiSiQiMXLE4G4c97m9+f2T1Sz7YGPU4UiWUCIRiZlLjy+jbW4OF92/ULUl0iKUSCSz3HBD0KTZunUq4L/HDebZ6g+ZOX9l1OFIFlAdiWSWESOijiArfP2L+3LvvBqu+Odijhjcjc6FbZKboYpEY01bJJJZHnssaJKU3BzjqhOHs3bTdn750JLkZzhqlIpFY0yJRDLLFVcETZI2dJ/OfGdUX/7+0r+peHtNcjNToWisKZGIxNhPjxzEPp0LuHDGIrbv2Nn8GalQNNaUSERirH3bPKaOH0rVqvX8+ZllUYcjGUqJRCTmjhrag6PKuvObx99gxZpNUYcjGUiJRESYOn4ouWZcPFP3LZE9p0QimeWPfwyatKh9urTjp0cOYm7VauYsfC/qcCTDqI5EMsvgwVFHkLW+Paov973yDpfNrmT0oK50KtiD2hIVicZaSrdIzGycmVWZWbWZnd/A623N7J7w9RfNrG+d1y4I+6vM7Oiwr7eZPWlmr5tZpZn9JJXxSxqaPTto0uLycnO4+sThrN6wlesertqziUeMULFojKUskZhZLvAH4BigDPiqmZXVG3YasNbdBwC/Bq4Jpy0DJgNDgXHAjeH8aoFz3X0IcBBwVgPzlGx23XVBk5T4fO8ufOugfbnjheXMX/FR4hOqUDTWUrlFMhKodvel7r4NuBuYUG/MBOD28Pl0YIyZWdh/t7tvdfdlQDUw0t3fdfdXANx9PfA60DOFyyASO+cePZiSDm2Zct9CahOtLVGhaKylMpH0BFbU+buGz/6j/8kYd68F1gHFiUwb7gbbH3ixBWMWib1OBW249IShLH73Y2577u2ow5EMkMpEYg301T+vsLExTU5rZh2Ae4Fz3P3jBt/c7HQzqzCzitWrVycYsogAHDu8B0cMLuG+ruvYAAAKp0lEQVT6R99g5Uebow5H0lwqE0kN0LvO372A+tes/mSMmeUBnYE1TU1rZm0Ikshd7n5fY2/u7je7e7m7l5eUlCS5KCLxYmZcPmEYO925dFZl1OFImktlInkZGGhmpWaWT3DwfFa9MbOAU8PnJwNPeFANNQuYHJ7VVQoMBF4Kj5/cArzu7tenMHZJV3feGTRJud5FhfxkzCAeXbyKRypVWyKNS1kdibvXmtnZwMNALnCru1ea2eVAhbvPIkgKd5pZNcGWyORw2kozmwYsJjhT6yx332FmhwLfBBaa2fzwraa4+5xULYekmd69dz9GWsz3Rpdy/6vvMHVWJYcM6Er7to38k6Ei0VizOFwOoby83CsqKqIOQ1rCPfcEj6ecEm0cMTJv+RpOuul5vndoKRcdr7Pt48LM5rl7eSJjdYkUySw33RQ0aTUH7lvEV0f24dZnl7HonXUND1KhaKwpkYjIbp0/bj+K2udz4YyF7NjZwF4MFYrGmhKJiOxW58I2XHRcGQtq1nHXi8ujDkfSjBKJiCRkwoh9OHRAV/73oSpWfbwl6nAkjSiRiEhCzIwrJg5j646dXD57cdThSBpRIpHMMn160CQSfbu25+wjBvDPhe/yZNX7UYcjaUKJRDJL165Bk8j84Ev96FfSnovvX8TmbTuCThWKxpoSiWSW224LmkSmbV4uV00aTs3azfz2iTeDzt69VSwaY0okklmUSNLCQf2KOfnAXvzp6aVUvbc+KBTdVSwqsaNEIiLNMuXYIXQsyGPKjIW4CkVjTYlERJqlqH0+Fxw7hHnL1/L++q1RhyMRUiIRkWb7yoG9GFlaxL8/3Mi2RO+mKFknZVf/FZHsZ2ZcNWk4H14Fryxfy9cu+GfUIUkdXTu05aULx6b8fZRIJLPM0R0D0s2Abh0o6dGRj7ds56wjBkQdjtRRmN86/8QrkUhmKSyMOgJpQOd/zqQzcK5qfGJJiUQyy403Bo9nnhltHPJpSiCxpoPtklmmTQuapBfV98SaEomIJE+JJNaUSEREJClKJCIikhQlEhERSYrO2pLMMndu1BGISD1KJCKSPBWKxpoSiYgkT4WisaZjJCKSvBtv/E+xqMSOEomIJE+ForGmRCIiIklJaSIxs3FmVmVm1WZ2fgOvtzWze8LXXzSzvnVeuyDsrzKzoxOdp4iItK6UJRIzywX+ABwDlAFfNbOyesNOA9a6+wDg18A14bRlwGRgKDAOuNHMchOcp4iItKJUbpGMBKrdfam7bwPuBibUGzMBuD18Ph0YY2YW9t/t7lvdfRlQHc4vkXmKiEgrSuXpvz2BFXX+rgG+2NgYd681s3VAcdj/Qr1pe4bPdzdPEWltKhSNtVRukVgDfZ7gmD3t/+ybm51uZhVmVrF69eomAxURkeZLZSKpAXrX+bsXsLKxMWaWB3QG1jQxbSLzBMDdb3b3cncvLykpSWIxRESkKalMJC8DA82s1MzyCQ6ez6o3ZhZwavj8ZOAJd/ewf3J4VlcpMBB4KcF5iohIK0rZMZLwmMfZwMNALnCru1ea2eVAhbvPAm4B7jSzaoItkcnhtJVmNg1YDNQCZ7n7DoCG5pmqZRARkd2zYAMgu5WXl3tFRUXUYYiIZAwzm+fu5YmMVWW7iIgkRYlERESSokQiIiJJUSIREZGkxOJgu5mtBpY3c/KuwActGE4mivs6iPvyg9YBxG8d7OvuCRXhxSKRJMPMKhI9cyFbxX0dxH35QesAtA6aol1bIiKSFCUSERFJihLJ7t0cdQBpIO7rIO7LD1oHoHXQKB0jERGRpGiLREREkqJE0og43RvezN42s4VmNt/MKsK+IjN71MzeDB/3CvvNzH4brpfXzOyAaKNvHjO71czeN7NFdfr2eJnN7NRw/JtmdmpD75WuGlkHU83snfC7MN/Mjq3z2gXhOqgys6Pr9Gfkb8XMepvZk2b2uplVmtlPwv5YfQ9ahLur1WsEVxZ+C+gH5AMLgLKo40rh8r4NdK3X9yvg/PD5+cA14fNjgQcJbjJ2EPBi1PE3c5kPAw4AFjV3mYEiYGn4uFf4fK+oly3JdTAVOK+BsWXh76AtUBr+PnIz+bcC7A0cED7vCLwRLmesvgct0bRF0jDdGz5Y3tvD57cDE+v03+GBF4AuZrZ3FAEmw92fJrh1QV17usxHA4+6+xp3Xws8CoxLffQto5F10JgJwN3uvtXdlwHVBL+TjP2tuPu77v5K+Hw98DrBLb1j9T1oCUokDWvofvM9GxmbDRx4xMzmmdnpYV93d38Xgh8c0C3sz+Z1s6fLnK3r4uxw182tu3brkOXrwMz6AvsDL6LvwR5TImlYwveGzxKHuPsBwDHAWWZ2WBNj47ZuoPFlzsZ1cRPQHxgBvAtcF/Zn7Towsw7AvcA57v5xU0Mb6MuKdZAsJZKGJXxv+Gzg7ivDx/eBGQS7K1bt2mUVPr4fDs/mdbOny5x168LdV7n7DnffCfyJ4LsAWboOzKwNQRK5y93vC7tj/z3YU0okDYvNveHNrL2Zddz1HDgKWESwvLvOPjkVmBk+nwV8KzyD5SBg3a7dAFlgT5f5YeAoM9sr3AV0VNiXseod75pE8F2AYB1MNrO2ZlYKDAReIoN/K2ZmBLf7ft3dr6/zUuy/B3ss6qP96doIztB4g+CMlAujjieFy9mP4EybBUDlrmUFioHHgTfDx6Kw34A/hOtlIVAe9TI0c7n/TrDrZjvB/yhPa84yA98lOPBcDXwn6uVqgXVwZ7iMrxH8w7l3nfEXhuugCjimTn9G/laAQwl2Qb0GzA/bsXH7HrREU2W7iIgkRbu2REQkKUokIiKSFCUSERFJihKJiIgkRYlERESSokQishtm9lz42NfMvtbC857S0HuJZBKd/iuSIDM7nODKuMfvwTS57r6jidc3uHuHlohPJCraIhHZDTPbED79JTA6vE/HT80s18z+18xeDi9y+INw/OHhfS7+RlC4hpndH14Us3LXhTHN7JdAu3B+d9V9r7B6+n/NbJEF94o5pc6855rZdDNbYmZ3hRXamNkvzWxxGMu1rbmOJN7yog5AJIOcT50tkjAhrHP3L5hZW+BZM3skHDsSGObBJdcBvuvua8ysHfCymd3r7ueb2dnuPqKB9zqR4MKJnwe6htM8Hb62PzCU4HpOzwKHmNligkua7OfubmZdWnzpRRqhLRKR5juK4NpL8wkuP15McA0qgJfqJBGAH5vZAuAFggv8DaRphwJ/9+ACiquAp4Av1Jl3jQcXVpwP9AU+BrYAfzazE4FNSS+dSIKUSESaz4AfufuIsJW6+64tko2fDAqOrYwFDnb3zwOvAgUJzLsxW+s83wHkuXstwVbQvQQ3Ynpoj5ZEJAlKJCKJW09wS9ZdHgZ+GF6KHDMbFF5Bub7OwFp332Rm+xHcpnWX7bumr+dp4JTwOEwJwW1xX2ossPCeGp3dfQ5wDsFuMZFWoWMkIol7DagNd1HdBvyGYLfSK+EB79X857asdT0EnGFmrxFcOfeFOq/dDLxmZq+4+9fr9M8ADia4KrMDP3f398JE1JCOwEwzKyDYmvlp8xZRZM/p9F8REUmKdm2JiEhSlEhERCQpSiQiIpIUJRIREUmKEomIiCRFiURERJKiRCIiIklRIhERkaT8P4AUXEK+couVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_moms=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-2), \n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = (1e-2,1e-3), lr_decay=DecayType.COSINE),\n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7585e35153145769cf5381380ebeb13",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.388045   4.565604   0.1398    \n",
      "    1      3.406363   4.530048   0.1395                                                                                \n",
      "    2      3.406011   4.558265   0.1462                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.55826]), 0.1462]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cosine is simply going from a to b with a following half a cosine. The formula that gives the learning rate at batch i over n is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_i = end_lr + (start_lr - end_lr)/2 * ( 1 +  np.cos( i * np.pi) / n )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8FPX1//HXISEJN7kkAZQQCVcFVMAIgmJVqKKtotVWRCu2tNaqtfb61fbb1vqtvaq9qi0t3qgKiG1Ff3hX1IpcgoIKGo0IElFBQeQil8D5/TEDxrBJNmw2s5f38/H4PHYz+5nZM7O7HGY+c2bM3REREdlfraIOQERE0psSiYiIJESJREREEqJEIiIiCVEiERGRhCiRiIhIQpRIREQkIUokIiKSECUSERFJSG7UAbSEoqIi79WrV9RhiIikjcWLF7/v7sXx9M2KRNKrVy8qKiqiDkNEJG2Y2ap4++rQloiIJESJREREEqJEIiIiCVEiERGRhCiRiIhIQpRIREQkIUokIiKSECUSERFJiBKJiIgkRIlERBL32GNBk6yUFZdIEZEk+8UvgsexY6ONQyKhPRIREUmIEomIiCREiURERBKiRCIiIgnRYLuIJO5vf4s6AomQEomIJG7AgKgjkAjp0Jakl+OPD5qklvvvD5pkJe2RiEjirr8+eDzttGjjkEhoj0RERBKiRCIiIglRIhERkYQkNZGY2TgzqzSzKjO7Msbr+WY2I3x9gZn1CqcXmtmTZrbZzP5SZ54jzeylcJ4/mZklcx0kxcyaFTQRSRlJG2w3sxzgRuCzQDWwyMxmu/vyWt0mAxvcva+ZTQB+A5wDbAN+AgwOW203AxcB84E5wDjgwWSth6SYoqKoI5BYpk2LOgKJUDLP2hoOVLn7CgAzmw6MB2onkvHA1eHzWcBfzMzcfQvwXzPrW3uBZnYgcIC7Pxf+fQdwBklKJLOXrmH3bv/UtMb2f+ruINXtXnd+26dHrD4Nv163R2PzB32aGGdjcSfjPWOsV9d77wZg7Vnn0r1jAb0K25HTSjulkevZM+oIJELJTCQ9gNW1/q4GRtTXx91rzGwjUAi838Ayq+sss0esjmZ2EcGeC6WlpU2NHYD/mfUiH+/ctV/zSnJMv2sKAOdt6A1Au7wcjutfzOcOP5CTB3WndY6G/SIxY0bweM450cYhkUhmIon130Tfjz771d/dpwBTAMrLyxtaZr0e/PboTy3c/dOLqbtQ3+ddvMHXYwW1b59GltHE/rE0+T33mb/h7RI7jqa+R/B4yNyOAEy/6GjeWr+VJas/5LHl7/Hgy+9yUMcCLj6+DxOHl5KrhNKybr45eFQiyUrJTCTVQO393RJgTT19qs0sF+gIrG9kmSWNLLPZ9Cpql6xFy/4qCL6yR/cu5OjehXypvCe/GD+YJyvX8ten3uCn9y3jrgVv8csvHMaw0s4RByuSHZL537ZFQD8zKzOzPGACMLtOn9nApPD52cATXve/t7W4+zvAJjM7Ojxb6wLgvuYPXdJJq1bGmEO7MfMbI/nr+UeyaVsNX/zrc9z4ZNU+Y1wi0vyStkcSjnlcBjwM5AC3uPsyM7sGqHD32cBUYJqZVRHsiUzYM7+ZrQQOAPLM7AzgpPCMr28CtwFtCAbZdcaWAMFg/rjB3RnVt5Cr/vUSv3u4khfe+pA/nTuEtnm6GpBIslgDOwAZo7y83CsqKqIOQ5rD1q3BY9u2DXZzd26ft5KfP7Ccw0s6MXVSOUXt81sgwCy150Kac+dGGYU0IzNb7O7l8fTViKSkl7ZtG00iEOydXHhMGX87/0gq3/2I8/6+gPVbdrRAgFlKhaJZTYlE0stNNwUtTicN6s7USUex8oMtnP+PBXy4VckkKYqKVCyaxZRIJL3MnBm0JjimbxFTLiinau1mvn5HBdtrVBvU7G67LWiSlZRIJCt8pn8xN5xzBItWbuDKe1/ap/ZFEqREktV0Kotkjc8ffhAr39/CdY+8Ru+idnxrTL+oQxLJCNojkaxy6Ql9OXNoD2547DX++3p9V+IRkaZQIpGsYmZce+Zg+ha354oZL7D2o21RhySS9pRIJL3MnZtwrULbvFxuOm8YW7bv4vLpL7BL1e8iCVEikazUr1sHrhk/iPkr1jP1vyuiDif9zZkTNMlKSiSSXq67LmjN4OwjSzhpYDeue+Q1qtZuapZlZq04C0UlMymRSHp54IGgNYNgvOQw2uXl8L2ZS6nZtbtZlpuVmlgoKplFiUSyWnGHfK4ZP5il1RuZ8owOce23/SgUlcyhRCJZ77QjDuLkQd340+Ovs3r91qjDEUk7SiQiwM9OG0QrM35+//KoQxFJO0okkl7atAlaMzuoUxsuH9OPx155j0eXv9fsyxfJZEokkl4efDBoSfDVY8ro17U9V89exsc7dGFHkXgpkYiE8nJbcc34wbz94ceqLWmqZigUlfSlRCLp5f/+L2hJMrJPIScN7MbNc99g3abtSXsfkUyiRCLp5fHHg5ZEV55yCNtrdvOHx15L6vtklGYsFJX0o0QiUkfv4vacf/TB3L3wLV5/TxXvcWnGQlFJP0okIjFcPqYf7fJy+eWcV6IORSTlKZGIxNClXR6XnNCXJyvXsWjl+qjDEUlpSiSSXgoLg9YCLhzVi+IO+Vz/SGWLvJ9IulIikfRy771BawFt8nK45Pg+zF+xnnlVuptig5JUKCrpQYlEpAHnDi/lwI4FXP/oa7jrBlj1SmKhqKQ+JRJJL1ddFbQWUtA6h0tP6MviVRt46rV1Lfa+IulEiUTSy3PPBa0Ffam8JyWd23CD9krql+RCUUltSU0kZjbOzCrNrMrMrozxer6ZzQhfX2BmvWq9dlU4vdLMTq41/TtmtszMXjazu82sIJnrIJKX24rLT+zHi9UbmVupvZKYWqBQVFJX0hKJmeUANwKnAAOBc81sYJ1uk4EN7t4X+D3wm3DegcAEYBAwDrjJzHLMrAdwOVDu7oOBnLCfSFKdOawHPTq14aa5VVGHIpJykrlHMhyocvcV7r4DmA6Mr9NnPHB7+HwWMMbMLJw+3d23u/ubQFW4PIBcoI2Z5QJtgTVJXAcRAFrntOKi43qzaOUGFr6puhKR2pKZSHoAq2v9XR1Oi9nH3WuAjUBhffO6+9vAdcBbwDvARnd/JCnRS2oqKQlaBM45qidF7fO48UntlYjUlsxEYjGm1R2prK9PzOlm1plgb6UMOAhoZ2bnx3xzs4vMrMLMKtat03HtjPHPfwYtAgWtc/jKMWU89do6Xn57YyQxpKwWLBSV1JPMRFIN9Kz1dwn7Hoba2yc8VNURWN/AvGOBN919nbvvBP4FjIr15u4+xd3L3b28uLi4GVZHBL488mA65Ody89w3og4ltbRgoaiknmQmkkVAPzMrM7M8gkHx2XX6zAYmhc/PBp7w4PzK2cCE8KyuMqAfsJDgkNbRZtY2HEsZA+iqetnkiiuCFpEDClpzwaiDmfPyO6xYtzmyOERSSdISSTjmcRnwMME/9jPdfZmZXWNmp4fdpgKFZlYFfBe4Mpx3GTATWA48BFzq7rvcfQHBoPzzwEth/FOStQ6SgpYsCVqEvnJMGfm5rZjytO6iuFcLF4pKaslN5sLdfQ4wp860n9Z6vg34Yj3zXgtcG2P6z4CfNW+kIvErap/PWcNKuGdxNT84eQCF7fOjDil6LVwkKqlFle0i++Grx5axo2Y30+avijoUkcgpkYjshz7F7TnxkK5Me24V23buijockUgpkUh66d8/aCnga8eW8cGWHdy35O2oQxGJVFLHSESa3ZTUObdiZJ9CDj3wAP7xzJt8qbwnwYmEWSqiIlFJDdojEdlPZsbXji3j9bWbefr1LL/xVYSFohI9JRJJLxddFLQUcdoRB9G1Qz7/eEanAkv2UiKR9PLaa0FLEXm5rZg0qhfPvP4+le9uijqc6ERcKCrRUiIRSdDE4aXk57bi9udWRh1KdFKgUFSio0QikqDO7fI4/YiD+Pfzb7Px451RhyPS4uJOJGbWLpmBiKSzSaN68fHOXdy7uDrqUERaXKOJxMxGmdlywosjmtkRZnZT0iMTiWXIkKClmME9OjKstBPT5q9i927d112ySzx7JL8HTgY+AHD3pcBxyQxKpF5/+EPQUtCkUb148/0tPFOVhacCp1ChqLS8uA5tufvqOpN0TQiROk4ZfCBF7fO5Y97KqENpeVOmpFSxqLSseBLJajMbRXCHwjwz+z66B4hE5fzzg5aC8nJbMXF4T56oXMtbH2yNOhyRFhNPIrkYuJTgPurVwBDgkmQGJVKv6uqgpaiJIw6mlRn/XJBlVwVOsUJRaVnxJJIB7n6eu3dz967ufj5waLIDE0lH3TsWMG5Qd2YsWs3HO7LoCHCKFYpKy4onkfw5zmkiQnBf940f7+T+pWuiDkWkRdR79V8zGwmMAorN7Lu1XjoAyEl2YCLpakRZFwZ068Ad81fypaN6Rh2OSNI1tEeSB7QnSDYdarWPgLOTH5pIDCNHBi2FmRnnHV3Ky29/xIvVH0YdjkjS1btH4u5PAU+Z2W3unmUjh5KyfvWrqCOIyxlDe/CrOa9y14K3OLykU9ThJF8KFolKy4nnxlZbzex3wCCgYM9Edz8xaVGJpLkDClpz+hEHMXvpGn70uUM5oKB11CElV4oWiUrLiGew/U7gVaAM+DmwEliUxJhE6nfWWUFLAxNHlLJ1xy7ue0G34pXMFk8iKXT3qcBOd3/K3b8KHJ3kuERi++CDoKWBw0s6MuigA7hzwVu4Z/j1t1K4UFSSL55Esue62O+Y2efMbCigGzSLNMLMmDiilFff3cQLqzN80D3FC0UlueJJJL8ws47A94DvA/8AvpPUqEQyxPghPWiXl8Od89+KOhSRpGkwkZhZDtDP3Te6+8vufoK7H+nus1soPpG01j4/l/FDe/DAi2vYuFU3vZLM1GAicfddwOktFItI48aMCVoamTi8lO01u/nXCzr0I5kpnkNb88zsL2Y22syG7WnxLNzMxplZpZlVmdmVMV7PN7MZ4esLzKxXrdeuCqdXmtnJtaZ3MrNZZvaqmb0SVuBLtvjJT4KWRgb36MgRJR25K5MH3dOgUFSSJ546klHh4zW1pjnQYB1JeFjsRuCzBFcNXmRms919ea1uk4EN7t7XzCYAvwHOMbOBwASC2pWDgMfMrH+4h/RH4CF3P9vM8oC2cayDSKQmjijlf+59iUUrNzC8rEvU4TS/NCkUleRodI8kHBep2+IpRhwOVLn7CnffAUwHxtfpMx64PXw+CxhjZhZOn+7u2939TaAKGG5mBxDcnXFqGNsOd8/w02HkU045JWhp5rQjDqJDfi53Zdvl5SUrxHWHxP3UA6h9Z8XqcFrMPu5eA2wEChuYtzewDrjVzF4ws3+YWbvkhC8p6eOPg5Zm2ublcuawHsx5+V3Wb9kRdTjNL40KRaX5JTORWIxpdQ8Q19envum5wDDgZncfCmwB9hl7ATCzi8yswswq1q1bF3/UIkkycUQpO2p2c+/iDBx0T6NCUWl+yUwk1UDta2iXAHVv0LC3j5nlAh2B9Q3MWw1Uu/uCcPosgsSyD3ef4u7l7l5eXFyc4KqIJO6Q7gcwrLQTdy/M4EF3yUqNJhIz+0KMNsbMujYy6yKgn5mVhYPiE4C69SezgUnh87OBJzz4hc0GJoRndZUB/YCF7v4uwT3kB4TzjAGWI5ImJo44mBXvb2H+ivVRhyLSbOI5a2syMBJ4Mvz7eGA+0N/MrnH3abFmcvcaM7sMeJjgRli3uPsyM7sGqAiLGqcC08ysimBPZEI47zIzm0mQJGqAS8MztgC+BdwZJqcVwFeautLpbsuWLcydO5fXX3+dTZs2UVBQQNeuXTn22GPp06cPABs2bOCZZ57hjTfeYPPmzbRt25bCwkKGDBnCYYcdRk5OcG+yq6++eu9yW7duTfv27SkpKeGoo46itLQ0itVr2Oc/H3UE9YrnczmmZwGsWsQPr36Ck/p3ypzPJUVl9W+lBcWTSHYDh7r7ewBm1g24GRgBPA3ETCQA7j4HmFNn2k9rPd8GfLGeea8Fro0xfQlQHkfcGWvmzJns3LmT008/nS5durBlyxZWrVrFx+Eg9Jo1a7j99tspLi7m1FNPpaioiJ07d7Ju3Tqef/55unTp8qkv/umnn07//v2pqalhw4YNLF26lFtvvZWxY8dyzDHHRLWasX3/+1FHUK94P5fDC40Xaw7h3Emfp32eZcbnkqJFoln9W2lB8SSSXnuSSGgt0N/d15uZrvnQwrZt28aqVau44IIL6N27NwCdOnWiR4/ghDh359///jeFhYVMnjyZ4GzqQPfu3TnssMP2OT5fUFBA+/bt9y6rrKyMDh068Pjjj3PooYfSpUsG1j00s6Z8Lr+9cALj/vgMT6zcykXH9cmMzyUFi0T1W2k58Qy2P2NmD5jZJDObBNwHPB2edqsajhaWl5dHXl4elZWV1NTU7PP6u+++y7p16xg1atSnfhi11Te9tpEjR+LuvPrqqwnH3KyOPz5oKaYpn8shBx7AkQd35u6Fqz/1D1Vafy4pKOt/Ky0onj2SS4GzgGMITsu9A7g3HBQ/IYmxSQytWrXijDPO4P7772fx4sV0796d0tJSBg4cSElJCR+Ep2AWFhbunWfbtm3ccMMNe/8ePXo0o0ePbvB92rZtS7t27diwYUNyViTDNPVzmTi8lO/evYiLv/+/HFwYlEKl9eeyp0j0wQejjaMW/VZaTqOJJEwYs8ImKWDgwIH079+fVatWUV1dTVVVFfPmzWPMmDF07tx5n/75+flcfPHFANx5553s2rVrnz6x6BTVpmnK5/K5ww/k6tlt2NVjLBeffUT6fy4pWiSq30rLiPf039fNbKOZfWRmm8zso5YITuqXm5tLnz59+MxnPsPkyZMZNmwYc+fOpVOnTgC8//77e/uaGV26dKFLly57z0BpzNatW9m6dWvMH5vUL97PpaB1Dmcd2ZOnVn2M57XT55JE+q0kXzxjJL8FTnf3ju5+gLt3cPcDkh2YNE1xcTG7d++mqKiI4uJinn32WXbv3r3fy5s3bx5mxiGHHNKMUWafhj6X80aUsnOXM6sJle76XBKn30rzi2eM5D13fyXpkUhctm7dyj333MPQoUPp1q0beXl5rFmzhmeffZaysjIKCgo444wzuOOOO5g6dSqjR4/e+8NZvXo1H3300T4DiNu2bWPz5s3s2rWL9evXs3TpUpYuXcpnP/vZ1DsL5UtfijqCmPb3czm8EKb86zHO7r4xvT+XFJT1v5UWZI0d2zOzPwLdgf8A2/dMd/d/JTe05lNeXu4VFRVRh9EsampqeOqpp3jjjTdYv349u3btokOHDgwYMIDjjjuONm3aALB+/XqeeeYZVqxYwebNm8nNzaVbt24MHjyYYcOGkZsb/B+idpFVbm7up4qsDj744ChWMS3t7+dS+d4WHlrxMb+e/DkuHH9i+n4u110XPKZQnY9+K4kxs8XuHlfNXjyJ5NYYk93dv7o/wUUhkxJJ1tu6NXhsmxm3odm2cxcjfvk4x/Yr4saJcd0vTqRFNCWRxHPWVtZdgkRS2KmnBo9z50YaRnMpaJ3DWcNKmDZ/Je9v3k5R+/yoQxJpsnoH283sh+Hjn83sT3Vby4UoktkmjujZ5EH3lJOihaLSMhraI9kzwK5jQiJJ1LdrB4b36sLdC9/iotG9adWq8WpqkVRSbyJx9/vDx9vr6yMizWPiiFKumLGE51Z8wDF9i6IOR6RJ4ilI7G9mU8zsETN7Yk9rieBEssW4wd3p1LY1dy14K+pQRJosnjqSe4C/Av8A4rtegEiyXHhh1BEkxZ5B99vnrWTdpu0Ud9Cgu6SPeBJJjbvfnPRIROKRoYkE4NzhpUz975vMWlzNN4/vE3U4TZOihaLSMuK5RMr9ZnaJmR1oZl32tKRHJhLL++8HLQP17dqe4WXBoPvu3Wl2EcBLLgmaZKV4Eskk4AfAPGBx2HQml0Tj7LODlqHOG1HKW+u38uwbaZYst279pFhUsk6Dh7bMrBVwvrs/20LxiGS1kwd1p3Pb1ty98C1G9yuOOpz4ZVihqDRNg3sk7r4buK6FYhHJensG3R9Z9h5rN22LOhyRuMRzaOsRMzvL4rnnpIgk7NwRpdTsTvNKd8kq8SSS7xKcArxdN7YSSb4+xe0ZUdaF6QtXp9+gu2SlRhNJeCOrVu6epxtbSeS++c2gZbiJ4aD7f6vSbNBdslI8dSSYWWegH1CwZ5q7P52soETqdc45UUfQIsYN/mTQ/bj+aTDonsH1PdK4RhOJmX0N+DZQAiwBjgaeA05MbmgiMaxeHTz27BltHEmWn5vD2UeWcOuzK1m7aRtdOxQ0PlOUlEiyWjxjJN8GjgJWufsJwFBgXVKjEqnPl78ctCxw7vBg0P2eijQYdM/gQlFpXDyJZJu7bwMws3x3fxUYkNywRKR3cXtG9i5Mj0r3DC8UlYbFk0iqzawTwT3bHzWz+4A1yQ1LRCA4Fbh6w8c8o0F3SWHxnLV1prt/6O5XAz8BpgJnxLNwMxtnZpVmVmVmV8Z4Pd/MZoSvLzCzXrVeuyqcXmlmJ9eZL8fMXjCzB+KJQyRdnTyoG13a5XHXglVRhyJSr3j2SDCzY83sK+7+FMFAe4845skBbgROAQYC55rZwDrdJgMb3L0v8HvgN+G8A4EJwCBgHHBTuLw9vs0nd3AUyVj5uTl88cgSHntlLe99pEp3SU3x3NjqZ8D/AFeFk1oD/4xj2cOBKndf4e47gOnA+Dp9xgN77sA4CxgTVtCPB6a7+3Z3fxOoCpeHmZUAnyO4P4pkm+99L2hZZOKIUna7c6dueiUpKp46kjMJztR6HsDd15hZhzjm6wGsrvV3NTCivj7uXmNmG4HCcPr8OvPu2Qv6A/BDoMEYzOwi4CKA0tLSOMKVtHDaaVFH0OIOLmzHCQO6cteCVVx6Qh/yc3Man6mlZUGRqNQvnkNbO9zdAQcws3ZxLjvWtbnqnnpSX5+Y083s88Bad1/c2Ju7+xR3L3f38uLiNCjokvhUVgYty1w4qhfvb97BnJfeiTqU2M45J2uKRWVf8SSSmWb2N6CTmX0deAz4exzzVQO1q8ZK2Pdsr719zCwX6Aisb2DeY4DTzWwlwaGyE80snsNskim+8Y2gZZnR/YroU9yO255dGXUosa1e/UmxqGSdeM7auo5g/OJegvqRn7r7n+NY9iKgn5mVmVkeweD57Dp9ZhPcOAvgbOCJcO9nNjAhPKurjODyLAvd/Sp3L3H3XuHynnD38+OIRSStmRmTRvViafVGXnhrQ9Th7CuLCkVlX3Fda8vdHwUebcqCwzGPy4CHgRzgFndfZmbXABXuPpvgVOJpZlZFsCcyIZx3mZnNBJYDNcCl7r6rKe8vkmm+MKyE3z1UyW3zVjK0tHPU4YjsVW8iMbNN7DumAcH4hcdzBWB3nwPMqTPtp7WebwO+WM+81wLXNrDsucDcxmIQyRTt83P5YnlP7nhuJT8+9VC6HpDi19+SrFHvoa09l4uP0XQZeZGIXDDyYHbpVGBJMXEd2hJJGf/7v1FHEKleRcGpwHcueItLT+hLXm5cNcUiSaVEIull7NioI4jchaN6ccEtC5nz0jucMbTRi0y0jCwrEpVPUyKR9LJkSfA4ZEi0cUTo2L7BqcBT//sm44ccRHAxiIhlYaGofEL7xZJerrgiaFmsVStj8rG9eentjcxfsT7qcAJZWigqASUSkTT0hWE9KGyXx5Sn34g6lECWFopKQIlEJA0VtM5h0qhePFm5jtfe2xR1OJLllEhE0tSXjz6Ygtat+PvTK6IORbKcEolImurcLo8vlffkP0ve1r1KJFJKJJJefvnLoAkAXzu2N7t2O7em6sUcJSvo9F9JL6NGRR1BSiktbMspgw/kzgWruOzEvrTPj+gnneWFotlOeySSXubNC5rsddFxvdm0rYa7o7xsytixKhbNYkokkl5+9KOgyV5H9OzEqD6FTHlmBdt2RnSR7CVLPikWlayjRCKSAS4f0491m7YzfWFEeyUqFM1qSiQiGeDo3oUM79WFvz61gu01unWPtCwlEpEMcfmYfrz70TbuqaiOOhTJMkokIhnimL6FDC3txM1z32BHze6ow5EsokQi6eUPfwia7MPMuHxMP97+8GP+/YL2SqTlqI5E0ksWXz4+Hsf3L+bwko785ckqzhxa0nI3vlKRaFbTHomkl8ceC5rEZGZ897P9Wb3+Y2YsasEzuEaNUrFoFlMikfTyi18ETer1mf7FjCjrwh8fr2LrjpqWeVMVimY1JRKRDGNm/HDcIby/eXvLXYNLhaJZTYlEJAMdeXBnxh7ajb/OfYMNW3ZEHY5kOCUSkQz1g5MHsHlHDX99KkXuoigZS4lEJEMN6N6BM4f24NZ5K1m9fmvU4UgGUyKR9PK3vwVN4vKDkweQY8a1/++VqEORDKY6EkkvAwZEHUFaObBjGy45vg/XP/oa86reZ1TfouS8kYpEs1pS90jMbJyZVZpZlZldGeP1fDObEb6+wMx61XrtqnB6pZmdHE7raWZPmtkrZrbMzL6dzPglBd1/f9Akbl8/rjclndvw8/uXU7MrSZdOGTJExaJZLGmJxMxygBuBU4CBwLlmNrBOt8nABnfvC/we+E0470BgAjAIGAfcFC6vBvieux8KHA1cGmOZksmuvz5oEreC1jn87+cOpfK9TdyZrJtfqVA0qyVzj2Q4UOXuK9x9BzAdGF+nz3jg9vD5LGCMmVk4fbq7b3f3N4EqYLi7v+PuzwO4+ybgFaBHEtdBJCOcPKg7x/Qt5LpHKnnvo23N/wYqFM1qyUwkPYDVtf6uZt9/9Pf2cfcaYCNQGM+84WGwocCCZoxZJCOZGb844zB21OzmZ/ctizocyTDJTCQWY5rH2afBec2sPXAvcIW7fxTzzc0uMrMKM6tYt25dnCGLZK6yonZcMbY/Dy17l4defifqcCSDJDORVAM9a/1dAqypr4+Z5QIdgfUNzWtmrQmSyJ3u/q/63tzdp7h7ubuXFxcXJ7gqIpnha6PLGHjgAfzkvmVs3Loz6nAkQyQzkSwC+plZmZnlEQyez67TZzYwKXx+NvAWPDNYAAAKtklEQVSEu3s4fUJ4VlcZ0A9YGI6fTAVecfcbkhi7pKpp04Im+6V1Tit+e/bhrN+yg5/OfjnqcCRDJK2OxN1rzOwy4GEgB7jF3ZeZ2TVAhbvPJkgK08ysimBPZEI47zIzmwksJzhT61J332VmxwJfBl4ysyXhW/3I3eckaz0kxfTs2XgfadDgHh359ph+3PDoaxw/oJgzh5YkvlAViWY1C3YAMlt5eblXVFREHYY0hxkzgsdzzok2jjS3a7czYcpzvPLOJuZcPprSwrZRhyQpxswWu3t5PH11iRRJLzffHDRJSE4r4/fnDMEMvjX9BbbX7EpsgSoUzWpKJCJZqqRzW3571uEsXf0hP/3PMhI6OqFC0aymRCKSxU457EC+dWJfZlSs5o7nVkUdjqQpJRKRLPedsf0Ze2hXrnlgOU9Wro06HElDSiQiWa5VOF5ySPcOfPOfi6lYuT7qkCTNKJFIepk1K2jSrDoUtOb2rw7noI5t+Mpti3ipemPUIUkaUSKR9FJUFDRpdkXt87lj8nAOKGjNuX+fz/wVH8Q/swpFs5oSiaSX224LmiRFSee23HPxSLp3LOCCWxbywIt1r2pUj549VSyaxZRIJL0okSTdQZ3acM83RnJYj45cdtcL/N8Dy9nZ2A2xZsz4pFhUso4SiYjso3O7PO7++tFcOKoXU//7Jqf9+b+88NaG+mdQoWhWUyIRkZjycltx9emDmPLlI/lw606+cPM8Lr/7BSrf3RR1aJJiknbRRhHJDCcN6s7IPoX85ckqpj23itlL1zCkZydOGdydYQd3pndRO7oQ+yZCkh2USESkUR0KWnPVKYdy8XF9mFGxmvuXruFXD7669/Xp4RleE6/6f1GFKDEUtc9n4Y/HJv19lEgkvczRHQOi1LldHhd/pg8Xf6YPazdt48XVG6nesJWSB9viOJee0DfqEKWWtnkt80+8Eomkl7a63Hmq6NqhgLEDC4I/Hg/2RL6nGp+spEQi6eWmm4LHSy6JNg75NCWQrKaztiS9zJwZNEktqu/JakokIpI4JZKspkQiIiIJUSIREZGEKJGIiEhCdNaWpJe5c6OOQETqUCIRkcSpUDSrKZGISOJUKJrVNEYiIom76aZPikUl6yiRiEjiVCia1ZRIREQkIUlNJGY2zswqzazKzK6M8Xq+mc0IX19gZr1qvXZVOL3SzE6Od5kiItKykpZIzCwHuBE4BRgInGtmA+t0mwxscPe+wO+B34TzDgQmAIOAccBNZpYT5zJFRKQFJXOPZDhQ5e4r3H0HMB0YX6fPeOD28PksYIyZWTh9urtvd/c3gapwefEsU0REWlAyT//tAayu9Xc1MKK+Pu5eY2YbgcJw+vw68/YInze2TBFpaSoUzWrJ3COJdQtnj7NPU6fv++ZmF5lZhZlVrFu3rsFARURk/yUzkVQDPWv9XQKsqa+PmeUCHYH1DcwbzzIBcPcp7l7u7uXFxcUJrIaIiDQkmYlkEdDPzMrMLI9g8Hx2nT6zgUnh87OBJ9zdw+kTwrO6yoB+wMI4lykiIi0oaWMk4ZjHZcDDQA5wi7svM7NrgAp3nw1MBaaZWRXBnsiEcN5lZjYTWA7UAJe6+y6AWMtM1jqIiEjjLNgByGzl5eVeUVERdRgiImnDzBa7e3k8fVXZLiIiCVEiERGRhCiRiIhIQpRIREQkIVkx2G5m64BV+zl7EfB+M4aTjrJ9G2T7+oO2AWTfNjjY3eMqwsuKRJIIM6uI98yFTJXt2yDb1x+0DUDboCE6tCUiIglRIhERkYQokTRuStQBpIBs3wbZvv6gbQDaBvXSGImIiCREeyQiIpIQJZJ6ZNO94c1spZm9ZGZLzKwinNbFzB41s9fDx87hdDOzP4Xb5UUzGxZt9PvHzG4xs7Vm9nKtaU1eZzObFPZ/3cwmxXqvVFXPNrjazN4OvwtLzOzUWq9dFW6DSjM7udb0tPytmFlPM3vSzF4xs2Vm9u1welZ9D5qFu6vVaQRXFn4D6A3kAUuBgVHHlcT1XQkU1Zn2W+DK8PmVwG/C56cCDxLcZOxoYEHU8e/nOh8HDANe3t91BroAK8LHzuHzzlGvW4Lb4Grg+zH6Dgx/B/lAWfj7yEnn3wpwIDAsfN4BeC1cz6z6HjRH0x5JbLo3fLC+t4fPbwfOqDX9Dg/MBzqZ2YFRBJgId3+a4NYFtTV1nU8GHnX39e6+AXgUGJf86JtHPdugPuOB6e6+3d3fBKoIfidp+1tx93fc/fnw+SbgFYJbemfV96A5KJHEFut+8z3q6ZsJHHjEzBab2UXhtG7u/g4EPzigazg9k7dNU9c5U7fFZeGhm1v2HNYhw7eBmfUChgIL0PegyZRIYov73vAZ4hh3HwacAlxqZsc10Dfbtg3Uv86ZuC1uBvoAQ4B3gOvD6Rm7DcysPXAvcIW7f9RQ1xjTMmIbJEqJJLa47w2fCdx9Tfi4Fvg3weGK9/Ycsgof14bdM3nbNHWdM25buPt77r7L3XcDfyf4LkCGbgMza02QRO5093+Fk7P+e9BUSiSxZc294c2snZl12PMcOAl4mWB995x9Mgm4L3w+G7ggPIPlaGDjnsMAGaCp6/wwcJKZdQ4PAZ0UTktbdca7ziT4LkCwDSaYWb6ZlQH9gIWk8W/FzIzgdt+vuPsNtV7K+u9Bk0U92p+qjeAMjdcIzkj5cdTxJHE9exOcabMUWLZnXYFC4HHg9fCxSzjdgBvD7fISUB71Ouznet9NcOhmJ8H/KCfvzzoDXyUYeK4CvhL1ejXDNpgWruOLBP9wHlir/4/DbVAJnFJrelr+VoBjCQ5BvQgsCdup2fY9aI6mynYREUmIDm2JiEhClEhERCQhSiQiIpIQJRIREUmIEomIiCREiUSkEWY2L3zsZWYTm3nZP4r1XiLpRKf/isTJzI4nuDLu55swT46772rg9c3u3r454hOJivZIRBphZpvDp78GRof36fiOmeWY2e/MbFF4kcNvhP2PD+9zcRdB4Rpm9p/wopjL9lwY08x+DbQJl3dn7fcKq6d/Z2YvW3CvmHNqLXuumc0ys1fN7M6wQhsz+7WZLQ9jua4lt5Fkt9yoAxBJI1dSa48kTAgb3f0oM8sHnjWzR8K+w4HBHlxyHeCr7r7ezNoAi8zsXne/0swuc/chMd7rCwQXTjwCKArneTp8bSgwiOB6Ts8Cx5jZcoJLmhzi7m5mnZp97UXqoT0Skf13EsG1l5YQXH68kOAaVAALayURgMvNbCkwn+ACf/1o2LHA3R5cQPE94CngqFrLrvbgwopLgF7AR8A24B9m9gVga8JrJxInJRKR/WfAt9x9SNjK3H3PHsmWvZ2CsZWxwEh3PwJ4ASiIY9n12V7r+S4g191rCPaC7iW4EdNDTVoTkQQokYjEbxPBLVn3eBj4Zngpcsysf3gF5bo6AhvcfauZHUJwm9Y9du6Zv46ngXPCcZhigtviLqwvsPCeGh3dfQ5wBcFhMZEWoTESkfi9CNSEh6huA/5IcFjp+XDAex2f3Ja1toeAi83sRYIr586v9doU4EUze97dz6s1/d/ASIKrMjvwQ3d/N0xEsXQA7jOzAoK9me/s3yqKNJ1O/xURkYTo0JaIiCREiURERBKiRCIiIglRIhERkYQokYiISEKUSEREJCFKJCIikhAlEhERScj/B4qsT3yFM6TdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_moms=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-2), \n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = (1e-2,1e-3), lr_decay=DecayType.EXPONENTIAL),\n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cb2f205d08f0421c9a00afc7cb0f61aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.404826   4.560293   0.1391    \n",
      "    1      3.40894    4.541485   0.1382                                                                                \n",
      "    2      3.411215   4.539713   0.1385                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.53971]), 0.1385]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exponential is multiplying the learning rate by the same thing at each step, this thing being computed to be exactly what's needed to go from our start point to our end point. Here the learning rate on batch i over n is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_i = start_lr * (end_lr/start_lr)**(i/n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VOW9x/HPLwkh7JgQFlkkrBpBAYMLiqIoKFrRyhXa63Zrr7VqW2ttr/a2vdba3tqqVVvxqnWvrVLUFq37guICEhRUQCSCFEQwEGQ1QOB3/zgHTEOWSSaTM8v3/Xo9r5mcec6Z3zkzw49znuc5j7k7IiIiTZUVdQAiIpLalEhERCQuSiQiIhIXJRIREYmLEomIiMRFiUREROKiRCIiInFRIhERkbgokYiISFxyog6gJXTp0sX79u0bdRgiIilj3rx569y9MJa6GZFI+vbtS2lpadRhiIikDDNbEWtdXdoSEZG4KJGIiEhclEhERCQuSiQiIhIXJRIREYmLEomIiMRFiUREROKiRCIiInFRIhERkbgokYhI/F54ISiSkTLiFikikmDXXRc8nnhitHFIJHRGIiIicVEiERGRuCiRiIhIXJRIREQkLmpsF5H43XFH1BFIhJRIRCR+gwdHHYFESJe2JLWMGRMUSS5PPBEUyUg6IxGR+N14Y/D4la9EG4dEQmckIiISFyUSERGJixKJiIjEJaGJxMxONrMlZlZmZlfV8nprM3skfH2OmfUNlxeY2ctmtsXM/lBjncPM7L1wnVvNzBK5D5Jkpk8PiogkjYQ1tptZNnAbcBKwCphrZjPcfVG1ahcCG9x9gJlNAa4HJgOVwE+BIWGp7nbgImA28BRwMvB0ovZDkkyXLlFHILV58MGoI5AIJbLX1uFAmbsvAzCzh4GJQPVEMhG4Jnw+HfiDmZm7bwVeM7MB1TdoZj2Aju7+Zvj3A8AZJCiRPLFgNbvdE7FpaaLeM6bRKa8V/X90WdShSHW9e0cdgUQokYmkJ7Cy2t+rgCPqquPuVWa2ESgA1tWzzVU1ttmztopmdhHBmQt9+vRpbOwA/Gj6u3yxc1eT1pXEePjP91AOdPj2N+naIS/qcGSPRx4JHidPjjYOiUQiE0ltbRc1/3sfS50m1Xf3O4E7AUpKSpp0WvHU90bjOiNJKp1f7MCHazcze1kFpx+6f9ThyB633x48KpFkpEQmklVA9fPdXsDqOuqsMrMcoBNQ0cA2ezWwzWZT1KVdojYtTeRtc8nOMmYvW69EIpIkEtlray4w0MyKzCwXmALMqFFnBnB++HwS8JLXcwrg7p8Cm83syLC31nnA35s/dElWZtAhrxWzP1ofdSgiEkrYGUnY5nEZ8CyQDdzj7gvN7Fqg1N1nAHcDD5pZGcGZyJQ965vZx0BHINfMzgDGhT2+vg3cB7QhaGRXj60M07FNDsvWbWXtpkq6dVQ7iUjUEnqvLXd/iqCLbvVlP6v2vBL4tzrW7VvH8lL27RIsmeKpp6j4ZCPc/Tazl61n4rBa+1qISAvSTRsltbRty0H929AhL0eJJJlokGhGUyKR1DJ1KtnAEUUjmb2svn4Z0qI0UDSj6V5bklqmTYNp0ziyXwHL121lzcbKqCMSgPvuC4pkJCUSSUlH9isAYPYy9d5KCkokGU2JRFLSQT060jEvhzfVDVgkckokkpKys4zDiwqYvVyJRCRqSiSSso7qX8CK9dtY/fkXUYciktGUSCS1zJwZFOCosJ3k9bK67vEpIi1BiURS1oHdO9ClfS6vKZFE76mngiIZSeNIJLXccEPweOWVZGUZRw/owutl69i928nK0mSZkWnbNuoIJEI6I5HU8uSTQQmNHljIui07WLxmU4RBCVOnBkUykhKJpLRjBgQjql9bqstbkQoHikpmUiKRlNa9Ux6DurVXO4lIhJRIJOUdM6CQOcsrqNS0yCKRUCKR1NKmTVCqGT2oCzuqdjP3Y93EUSQK6rUlqeXpfecxO6Ion9zsLGYtXcfogYURBCWS2ZRIJOW1zc1hxAGdmaUG9+iEg0QlM+nSlqSWX/wiKDWMHljI4k83Ub55ewRBiWQ2JRJJLS++GJQaRg8MugHrdikRueGGLweLSsZRIpG0cPD+nchvl8srH5ZHHUpmqjFQVDKLEomkhews47hBhcxc8hm7dnvU4YhkFCUSSRtjBheyYdtOFqz6POpQRDKKEomkloKCoNTiuEGFZBm8/MFnLRyUSGZTIpHU8uijQalF57a5HHbAfrykRNLyahkoKplDiUTSyvEHdmXh6k2s3VQZdSiZ5emnax0sKplBiURSy9VXB6UOxw/uCsDMJTorEWkpSiSSWt58Myh1OLB7B3p0yuPlD9QNuEXVMVBUMkNCE4mZnWxmS8yszMyuquX11mb2SPj6HDPrW+21q8PlS8xsfLXl3zezhWb2vpn9xczyErkPklrMjOMP7MprZevYUbU76nAyRx0DRSUzJCyRmFk2cBtwClAMfM3MimtUuxDY4O4DgN8B14frFgNTgIOBk4GpZpZtZj2B7wIl7j4EyA7riex1wuCubNlepbsBi7SQRJ6RHA6Uufsyd98BPAxMrFFnInB/+Hw6MNbMLFz+sLtvd/flQFm4PQhuNNnGzHKAtsDqBO6DpKBRAwrIzclSN2CRFpLIRNITWFnt71XhslrruHsVsBEoqGtdd/8EuAH4J/ApsNHdn0tI9JKcevUKSj3a5uZwVL8CXli8FneNchdJtEQmEqtlWc1fdV11al1uZvsRnK0UAfsD7czsnFrf3OwiMys1s9LycjW8po0//SkoDTipuBsfr9/G0s+2tEBQUt9AUUl/iUwkq4De1f7uxb6XofbWCS9VdQIq6ln3RGC5u5e7+07gMWBUbW/u7ne6e4m7lxQWarKjTDOuuBsAzy1cE3EkGaKegaKS/hKZSOYCA82syMxyCRrFZ9SoMwM4P3w+CXjJg2sRM4ApYa+uImAg8BbBJa0jzaxt2JYyFlicwH2QZHP55UFpQNeOeQzv05nnFq1tgaBEMlvCEknY5nEZ8CzBP/bT3H2hmV1rZqeH1e4GCsysDLgCuCpcdyEwDVgEPANc6u673H0OQaP828B7Yfx3JmofJAnNnx+UGIwr7s67qzay+vMvEhyUNDRQVNKbZUJjZElJiZeWlkYdhjSHMWOCxximdl1WvoUTbnyFn59+MOeP6pvIqKQRn4ukBjOb5+4lsdTVyHZJW/0K2zOga3ueVTuJSEIpkUhaG39wN+Ysr+DzbTuiDkUkbSmRSGoZNCgoMRpX3J1du50XF2twokii5EQdgEij3Nm4vhWH9OpE9455PLdoDWcdVv9ARolDA4NEJb0pkUhaMzPGHdyNaaUr2bajira5+sonRAyDRCV96dKWpJaLLgpKI5wypAeVO3dr5kSRBFEikdTy4YdBaYTDi/Ip7NCaf7z7aYKCklgHikp60nm+pL3sLGPCkO48PHclW7ZX0b61vvbNLsZBopKedEYiGeG0Q/dne9VuXlysW6aINLeYE4mZtUtkICKJdFif/ejeMY8ndXlLpNk1mEjMbJSZLSK8OaKZHWpmUxMemUhthg0LSiNlZRkThvbglSXlbK7cmYDARDJXLGckvwPGA+sB3H0BcGwigxKp0803B6UJTj2kBzt27eZ53RG4+TVyoKikl5haHd19ZXDX9r12JSYckcQZ0aczPTu34R/vfspXR2gAXbNq5EBRSS+xnJGsNLNRBDMU5prZlWgOEInKOecEpQnMjAlDu/Pq0nI2btPlLZHmEksiuRi4lGAe9VXAMOCSRAYlUqdVq4LSRF85dH927nKefl+N7s2qCQNFJX3EkkgGu/u/u3s3d+/q7ucAByU6MJFEGNqzE/0K2/HYO59EHUp6acJAUUkfsSSS38e4TCTpmRlfHd6Tt5ZXsLJiW9ThiKSFOhOJmR1lZj8ACs3simrlGiC7xSIUaWYTh/UE4O/zdVYi0hzqOyPJBdoT9OzqUK1sAiYlPjSRWhx1VFDi0Du/LYcX5fPYO5+QCVNNiyRand1/3f0V4BUzu8/dV7RgTCJ1+9//bZbNfHV4T6567D3eXbWRQ3t3bpZtZrQmDBKV9BHLOJJtZvZb4GAgb89Cdz8hYVGJJNgpQ3vwsxkLefydT5RImkMTB4lKeoilsf0h4AOgCPg58DEwN4ExidTtrLOCEqdObVpx0kHdmLFgNTt37W6GwEQyVyyJpMDd7wZ2uvsr7v4N4MgExyVSu/Xrg9IMzhzek4qtO3hlSXmzbC+jxTFQVFJfLIlkzxDgT83sVDMbDuj+EpLyjhtcSEG7XP46b2XUoaS+OAeKSmqLJZFcZ2adgB8AVwJ/BL6f0KhEWkCr7CzOOqwXLy7+jM82V0YdjkjKqjeRmFk2MNDdN7r7++5+vLsf5u4zWig+kYSaPLI3Vbudx97WmBKRpqo3kbj7LuD0FopFpGFjxwalmfQvbM/hffN5ZO5KjSkRaaJYLm29YWZ/MLPRZjZiT4ll42Z2spktMbMyM7uqltdbm9kj4etzzKxvtdeuDpcvMbPx1ZZ3NrPpZvaBmS02s/hGp0lq+elPg9KMJo/szfJ1W5mzvKJZt5tRmmGgqKSuWMaRjAofr622zIF6x5GEl8VuA04iuGvwXDOb4e6LqlW7ENjg7gPMbApwPTDZzIqBKQRjV/YHXjCzQeEZ0i3AM+4+ycxygbYx7INInSYM7cE1TyzkkbkrObJfQdThpKZmGigqqanBM5KwXaRmiWUw4uFAmbsvc/cdwMPAxBp1JgL3h8+nA2MtmEFrIvCwu2939+VAGXC4mXUkmJ3x7jC2He7+eSw7KmnilFOC0oza5GZzxrCePPXep5qnRKQJYrm01VQ9ger9KleFy2qt4+5VwEagoJ51+wHlwL1m9o6Z/dHM2iUmfElKX3wRlGY2eWRvtlft5m+6kWPTNNNAUUlNiUwkVsuymq2ZddWpa3kOMAK43d2HA1uBfdpeAMzsIjMrNbPS8nINOJP6DenZiSE9O/LnOf9Uo3tTNONAUUk9iUwkq4De1f7uBayuq46Z5QCdgIp61l0FrHL3OeHy6QSJZR/ufqe7l7h7SWFhYZy7Ipng3CMPYMnazbylRneRRmkwkZjZV2spY82sawOrzgUGmllR2Cg+Bag5/mQGcH74fBLwkgf/HZwBTAl7dRUBA4G33H0NwRzyg8N1xgKLEGkGpx/ak05tWvHAm7rZtUhjxNJr60LgKODl8O8xwGxgkJld6+4P1raSu1eZ2WXAswQTYd3j7gvN7FqgNBzUeDfwoJmVEZyJTAnXXWhm0wiSRBVwadhjC+A7wENhcloG/EdjdzrVbd26lZkzZ7J06VI2b95MXl4eXbt25ZhjjqF///4AbNiwgVmzZvHRRx+xZcsW2rZtS0FBAcOGDWPo0KFkZwdzk11zzTV7t9uqVSvat29Pr169GDlyJH369Ili9+p32mkJ23Sb3Gwmj+zN3a8tZ83GSrp3ymt4pWoy+nNJUvpMWkYsiWQ3cJC7rwUws27A7cARwKtArYkEwN2fAp6qsexn1Z5XAv9Wx7q/BH5Zy/L5QEkMcaetadOmsXPnTk4//XTy8/PZunUrK1as4IuwEXr16tXcf//9FBYWMmHCBLp06cLOnTspLy/n7bffJj8//1+++KeffjqDBg2iqqqKDRs2sGDBAu69915OPPFEjj766Kh2s3ZXXpnQzZ9zxAHcNWsZf56zgivGDW54hWoy+nNpxkGizSmjP5MWFEsi6bsniYQ+Awa5e4WZqa9kC6usrGTFihWcd9559OvXD4DOnTvTs2fQIc7defzxxykoKODCCy8k6E0d6N69O0OHDt2nMTkvL4/27dvv3VZRUREdOnTgxRdf5KCDDiI/P7+F9i56fQracsLgrvz5rX9y6QkDaJ0T26zSGf+5NPMg0eaQ8Z9JC4qlsX2WmT1pZueb2fnA34FXw263GsPRwnJzc8nNzWXJkiVUVVXt8/qaNWsoLy9n1KhR//LDqK6u5dUdddRRuDsffPBB3DE3qzFjgpJA543qy7otO3jm/TUxr5Pxn0sS0mfScmI5I7kUOAs4mqBb7gPAo2Gj+PEJjE1qkZWVxRlnnMETTzzBvHnz6N69O3369KG4uJhevXqxPuyCWVDw5QjtyspKbrrppr1/jx49mtGjR9f7Pm3btqVdu3Zs2LAhMTuSxEYP6EJRl3bc98bHTBxWc+hT7TL+c9kzSPTpp6ONo5qM/0xaUIOJJEwY08MiSaC4uJhBgwaxYsUKVq1aRVlZGW+88QZjx45lv/3226d+69atufjiiwF46KGH2LVr1z51apOp4ymysozzjjqAnz+xiLf/uYERffY9prXJ6M8lAYNEm0NGfyYtKNbuv0vNbKOZbTKzzWa2qSWCk7rl5OTQv39/jjvuOC688EJGjBjBzJkz6dw5mH983bp1e+uaGfn5+eTn5+/tgdKQbdu2sW3btlp/bJng7JLedMzL4a5XlzVqPX0uyUefSeLF0kbyG+B0d+/k7h3dvYO7d0x0YNI4hYWF7N69my5dulBYWMjrr7/O7t1Nn4v8jTfewMw48MADmzHK1NGudQ7nHHkAzyxcw4r1W5u8HX0uyUefSfOLpY1krbsvTngkEpNt27bx17/+leHDh9OtWzdyc3NZvXo1r7/+OkVFReTl5XHGGWfwwAMPcPfddzN69Oi9P5yVK1eyadOmfRoQKysr2bJlC7t27aKiooIFCxawYMECTjrppOTrhXL22S32VheM6ssfZy3n7teWc+3EIfXWzfjPJQnpM2k51tC1PTO7BegO/A3Yvme5uz+W2NCaT0lJiZeWlkYdRrOoqqrilVde4aOPPqKiooJdu3bRoUMHBg8ezLHHHkubNm0AqKioYNasWSxbtowtW7aQk5NDt27dGDJkCCNGjCAnJ/g/RPVBVjk5Of8yyOqAAw6IYheTyo+mL2DGgtW8cdVY8tvl1lkv4z+XG24IHhM8zqcxMv4ziZOZzXP3mMbsxZJI7q1lsbv7N5oSXBTSKZFkvG3bgse2LTMNzYdrNzPud69yxUmD+O7YgS3yniLJoDGJJJZeWxl3CxJJYhMmBI8zZ7bI2w3q1oHjBxdy/xsfc9Gx/chrFVsDrEgmqbOx3cx+FD7+3sxurVlaLkSRaF10bH/Wb93B9Hmrog4lebXAQFFJXvWdkexpYNc1IcloR/bLZ0Sfztw+8yPOLulNbk4iZ18QST11JhJ3fyJ8vL+uOiKZwMz4ztiB/Me9c3n8nVVMHpnZd3oVqanBNhIzGwRcCfStXj/GedtF0sKYQYUc0qsTt738EWeN6EVOts5KRPaIZRzJX4H/A/4IxHa/AJFEueCCSN7WzPjuCQP55gOl/H3+as46rFckcYgko1gSSZW7357wSERiEVEiARh7UFeKe3TkDy+XccbwnmRnNXxn2IzRggNFJfnEcn7+hJldYmY9zCx/T0l4ZCK1WbcuKBEwM747dgDL123lyXdXRxJD0rrkkqBIRorljGTPnOo/rLbMgX7NH45IAyZNCh5baBxJTeOKuzO4WwdueXEppw7tobaSPVp4oKgkl3p/BWaWBZzj7kU1ipKIZKSsLOOKcYNYVr6VR9/WuJK9Jkz4crCoZJx6E4m77wZuaKFYRFLCuOJuDOvdmZtfWErlTvU/EYnlvPw5MzvLYplzUiQDmBk/Onkwn26s5E+zV0QdjkjkYkkkVxB0Ad6uia1EAqP6d2H0wC7c9nIZmyt3Rh2OSKQaTCThRFZZ7p6ria0kct/+dlCSwA/HD2bDtp3cNWt51KGIRCqWXluY2X7AQCBvzzJ3fzVRQYnUafLkqCPY65BenZkwtDt/nLWMc47sQ9cOeQ2vlK4iHN8j0YtlzvZvAq8CzwI/Dx+vSWxYInVYuTIoSeLKcYPZUbWbm577MOpQonXBBUomGSyWNpLvASOBFe5+PDAcKE9oVCJ1OffcoCSJfoXtOX9UXx4pXcn7n2yMOpzoRDhQVKIXSyKpdPdKADNr7e4fAIMTG5ZI6vju2IF0btOKXzy5iIZmHE1bkyZ9OVhUMk4siWSVmXUmmLP9eTP7O6D7Q4iEOrVpxRXjBjNneQXPvL8m6nBEWlwsvbbOdPfP3f0a4KfA3cAZsWzczE42syVmVmZmV9XyemszeyR8fY6Z9a322tXh8iVmNr7Getlm9o6ZPRlLHCKJ9rWRvRncrQO/enqxBilKxonpRkFmdoyZ/Ye7vwK8CfSMYZ1s4DbgFKAY+JqZFdeodiGwwd0HAL8Drg/XLQamAAcDJwNTw+3t8T2+nMFRJHI52Vn85LSDWFnxBXe/pu7Aklli6bX1P8B/AVeHi1oBf4ph24cDZe6+zN13AA8DE2vUmQjsmYFxOjA2HEE/EXjY3be7+3KgLNweZtYLOJVgfhTJND/4QVCS0OiBhZx8cHdufXEp/1y/LepwRFpMLGckZwKnA1sB3H010CGG9XoC1ftprmLfM5m9ddy9CtgIFDSw7s3Aj4Dd9b25mV1kZqVmVlperk5maeMrXwlKkvqf04vJyTJ+NuP9zGp4T6KBotLyYkkkOzz4RTiAmbWLcdu13Zur5i+rrjq1Ljez04DP3H1eQ2/u7ne6e4m7lxQWFjYcraSGJUuCkqR6dGrD908axMwl5ZnV8D55clINFpWWFUsimWZmdwCdzew/gReAu2JYbxXQu9rfvdi3t9feOmaWA3QCKupZ92jgdDP7mOBS2QlmFstlNkkX3/pWUJLYBaP6clCPjlzzxEK2bK+KOpyWkWQDRaVlxdJr6waC9otHCcaP/Mzdfx/DtucCA82syMxyCRrPZ9SoM4MvJ86aBLwUnv3MAKaEvbqKCG7P8pa7X+3uvdy9b7i9l9z9nBhiEWkxOdlZ/OrMIXy2eTs3Ppe8Z0/NKskGikrLiuleW+7+PPB8Yzbs7lVmdhnBLVWygXvcfaGZXQuUuvsMgq7ED5pZGcGZyJRw3YVmNg1YBFQBl7q7+lRKyhjeZz/OOeIA7nvjY047ZH8OO2C/qEMSSRirq0HQzDazb5sGBO0Xnkp3AC4pKfHS0tKow5DmMGZM8BjRVLuNsWV7FeN/9yqtc7J46nujyWuV3fBKqSqFPheJjZnNc/eSWOrWeWlrz+3iaym6jbxIDNq3zuH6sw5h2bqt3PR8ht/UUdJaTJe2RJLGT34SdQSNcszALnz9iD7cNWsZ4w/urktckpaUSCS1nHhi1BE02o8nHMQrS8r54V8XpO8lriQdJCotI6ZbpIgkjfnzg5JC2rfO4TeTgktc//tUmt7ZJ8kHikpiKZFIarn88qCkmKMHdOEbRxdx/5sreGHR2qjDaX5JPlBUEkuJRKSF/Ncpgynu0ZEfTl/A2k2VUYfTvFJgoKgkjhKJSAtpnZPNrV8bTuXO3VwxbT67d2fQvbgkrSmRiLSgAV3b8z9fKeb1svXc8eqyqMMRaRZKJCItbPLI3pw6tAc3PLeEOcvWRx2OSNyUSCS1/OpXQUlhZsavzxrKAfltufTP76Rfe4lkHCUSSS2jRgUlxXXIa8Ud5x7Gth1VXPLQ2+yoqnd6neT3k5+k3GBRaT5KJJJa3ngjKGlgYLcO/GbSIcxbsYFf/mNR1OHE58QTU3KwqDQPjWyX1PLjHwePaXJzwNMO2Z/5//ycP762nCE9O/FvJb0bXikZ7RkkOmxYtHFIJJRIRCL2X6ccyOI1m/jx4+/RJ78tR/QriDqkxtszSDRNErw0ji5tiUSsVXYWU79+GL3z2/KtP81j+bqtUYck0ihKJCJJoFPbVtx7wUgMuPC+uXy+bUfUIYnETIlEJEkcUNCOO84tYeWGbVz8p3lsr9KkoJIalEgktdx8c1DS1OFF+fxm0iHMXlbB9x+Zzy7dRkVSgBrbJbVkQK+gM4f3Yv2WHVz3j8V0avM+vzpzCGYWdVj1S/FBohIfJRJJLS+8EDym+ZiFb47uR8XWHUyd+RH57Vrxw/EHRh1S/dJgkKg0nRKJpJbrrgse0zyRAPxw/GA2bNvBbS9/RKc2rbjo2P5Rh1S3PYNElVAykhKJSJIyM647YyibKqv41VMfkGXGN0f3izqs2qXZQFFpHCUSkSSWnWXcPHkYOFz3j8W4w38em6TJRDKWEolIkmuVncXNU4JOBr8M53xXMpFkokQikgJaZWdxS7VksmPXbi4Z0z/5e3NJRlAikdRyxx1RRxCZnDCZtMo2fvvsEtZv2cFPTj2IrCwlE4mWEomklsGDo44gUjnZWdx09jD2a5fLPa8vZ8O2Hfxm0iG0yo54bHEaDxKVhiX022dmJ5vZEjMrM7Oranm9tZk9Er4+x8z6Vnvt6nD5EjMbHy7rbWYvm9liM1toZt9LZPyShJ54IigZLCvL+Nlpxfxw/GAef+cT/vOBUrZur4o2qGHDMmKwqNQuYYnEzLKB24BTgGLga2ZWXKPahcAGdx8A/A64Ply3GJgCHAycDEwNt1cF/MDdDwKOBC6tZZuSzm68MSgZzsy49PgB/OrMobz6YTmT/u9NVn/+RXQBvfDCl4NFJeMk8ozkcKDM3Ze5+w7gYWBijToTgfvD59OBsRa0Hk4EHnb37e6+HCgDDnf3T939bQB33wwsBnomcB9EktrXj+jDPReMZFXFNibe9jrzV34eTSDXXfflYFHJOIlMJD2BldX+XsW+/+jvrePuVcBGoCCWdcPLYMOBOc0Ys0jKGTO4K49dMoq8VllMvuNNZixYHXVIkmESmUhq60pS81amddWpd10zaw88Clzu7ptqfXOzi8ys1MxKy8vLYwxZJDUN7NaBv11yNIf06sR3//IOv/zHInbu2h11WJIhEplIVgHVJ6DuBdT8r9LeOmaWA3QCKupb18xaESSRh9z9sbre3N3vdPcSdy8pLCyMc1dEkl9B+9b86ZtHcN5RB3DXrOV8/a7ZrN1UGXVYkgESmUjmAgPNrMjMcgkaz2fUqDMDOD98Pgl4yd09XD4l7NVVBAwE3grbT+4GFrv7TQmMXZLVgw8GRWrVOiebaycO4ZYpw1i4ehOn3jqLN8rWRR2WpLmEjSNx9yozuwx4FsgG7nH3hWZ2LVDq7jMIksKDZlZGcCYyJVx3oZlNAxYR9NS61N13mdnZafvAAAAKrElEQVQxwLnAe2Y2P3yrH7v7U4naD0kyvXs3XEeYOKwnxT068u2H3uacu+dw8XH9ufzEQeTmJOj/jhk8UFTAghOA9FZSUuKlpaVRhyHN4ZFHgsfJk6ONI0Vs3V7FL55cxMNzVzKkZ0dunjyMAV07RB2WpAAzm+fuJbHU1VS7klpuvz0oEpN2rXP49VmHcMe5h7H680pOvfU17nt9ObubewpfDRTNaEokIhlg/MHdeeby0YzqX8A1Tyxiyl2zKftsS/O9gQaKZjQlEpEM0bVDHvdcMJLfTDqEJWs2M+GWWdz64lJ2VKmbsMRHiUQkg5gZZ5f05oUrjmP8kO7c9PyHnHrrLN5aXhF1aJLClEhEMlBhh9b8/mvDufeCkWzbsYuz73iTS//8Nqs2bIs6NElBuo28pJbp06OOIK0cf2BXXrjiOO549SP+75WPeGHRWr51bD8uHtOftrn650Fio+6/IgLA6s+/4NdPf8CMBavp1rE13zlhIGeX9I5t7MnK8NZ4GueTNhrT/VeJRFLLffcFjxdcEGUUaa304wp+/fQHlK7YQO/8Nlw+dhBnDO9JtmZizChKJDUokaSRMWOCx5kzo4wi7bk7Mz8s54Znl7Bw9SYGdG3Pd04YwKlDe5BT22yMGiiadjQgUUTiYmYcP7grT1x2DFP/fQQA33t4PsffOJMHZ6+gcueuf11BA0UzmhKJiNQpK8uYMLQHz11+LHecexgF7Vrz07+9zzHXv8RtL5exYeuOqEOUJKBuGSLSoKwsY/zB3RlX3I05yyu4feZH/PbZJdzy4lJOG9qDn1dW0T4vp9aJhCT9KZGISMzMjCP7FXBkvwKWrNnMQ3NW8Njbn3D26o0AfP3qf0QcoVTXpX1r3vrvExP+Pkokklqe0owByWJw9w5cO3EIPzr5QLY92Z7tVbu49PgBUYcl1bTUWCAlEkktbdtGHYHU0L51Du2fDe78+4MuXSKORqKgRCKpZerU4PGSS6KNQ/6VEkhGU68tSS3TpgVFkst99305WFQyjhKJiMRPiSSjKZGIiEhclEhERCQuSiQiIhIX9dqS1KKbNYokHSUSEYmfBopmNCUSEYmfBopmNLWRiEj8pk79crCoZBwlEhGJnwaKZjQlEhERiUtCE4mZnWxmS8yszMyuquX11mb2SPj6HDPrW+21q8PlS8xsfKzbFBGRlpWwRGJm2cBtwClAMfA1MyuuUe1CYIO7DwB+B1wfrlsMTAEOBk4GpppZdozbFBGRFpTIM5LDgTJ3X+buO4CHgYk16kwE7g+fTwfGmpmFyx929+3uvhwoC7cXyzZFRKQFJbL7b09gZbW/VwFH1FXH3avMbCNQEC6fXWPdnuHzhrYpIi1NA0UzWiLPSGqbvtljrNPY5fu+udlFZlZqZqXl5eX1BioiIk2XyESyCuhd7e9ewOq66phZDtAJqKhn3Vi2CYC73+nuJe5eUlhYGMduiIhIfRKZSOYCA82syMxyCRrPZ9SoMwM4P3w+CXjJ3T1cPiXs1VUEDATeinGbIiLSghLWRhK2eVwGPAtkA/e4+0IzuxYodfcZwN3Ag2ZWRnAmMiVcd6GZTQMWAVXApe6+C6C2bSZqH0REpGEWnACkt5KSEi8tLY06DBGRlGFm89y9JJa6GtkuIiJxUSIREZG4KJGIiEhclEhERCQuGdHYbmblwIomrt4FWNeM4aSiTD8Gmb7/oGMAmXcMDnD3mAbhZUQiiYeZlcbacyFdZfoxyPT9Bx0D0DGojy5tiYhIXJRIREQkLkokDbsz6gCSQKYfg0zff9AxAB2DOqmNRERE4qIzEhERiYsSSR0yaW54M/vYzN4zs/lmVhouyzez581safi4X7jczOzW8Li8a2Yjoo2+aczsHjP7zMzer7as0ftsZueH9Zea2fm1vVeyquMYXGNmn4TfhflmNqHaa1eHx2CJmY2vtjwlfytm1tvMXjazxWa20My+Fy7PqO9Bs3B3lRqF4M7CHwH9gFxgAVAcdVwJ3N+PgS41lv0GuCp8fhVwffh8AvA0wSRjRwJzoo6/ift8LDACeL+p+wzkA8vCx/3C5/tFvW9xHoNrgCtrqVsc/g5aA0Xh7yM7lX8rQA9gRPi8A/BhuJ8Z9T1ojqIzktppbvhgf+8Pn98PnFFt+QMemA10NrMeUQQYD3d/lWDqguoau8/jgefdvcLdNwDPAycnPvrmUccxqMtE4GF33+7uy4Eygt9Jyv5W3P1Td387fL4ZWEwwpXdGfQ+agxJJ7Wqbb75nHXXTgQPPmdk8M7soXNbN3T+F4AcHdA2Xp/Oxaew+p+uxuCy8dHPPnss6pPkxMLO+wHBgDvoeNJoSSe1inhs+TRzt7iOAU4BLzezYeupm2rGBuvc5HY/F7UB/YBjwKXBjuDxtj4GZtQceBS539031Va1lWVocg3gpkdQu5rnh04G7rw4fPwMeJ7hcsXbPJavw8bOwejofm8buc9odC3df6+673H03cBfBdwHS9BiYWSuCJPKQuz8WLs7470FjKZHULmPmhjezdmbWYc9zYBzwPsH+7ul9cj7w9/D5DOC8sAfLkcDGPZcB0kBj9/lZYJyZ7RdeAhoXLktZNdq7ziT4LkBwDKaYWWszKwIGAm+Rwr8VMzOC6b4Xu/tN1V7K+O9Bo0Xd2p+shaCHxocEPVL+O+p4Erif/Qh62iwAFu7ZV6AAeBFYGj7mh8sNuC08Lu8BJVHvQxP3+y8El252EvyP8sKm7DPwDYKG5zLgP6Ler2Y4Bg+G+/guwT+cParV/+/wGCwBTqm2PCV/K8AxBJeg3gXmh2VCpn0PmqNoZLuIiMRFl7ZERCQuSiQiIhIXJRIREYmLEomIiMRFiUREROKiRCLSADN7I3zsa2Zfb+Zt/7i29xJJJer+KxIjMxtDcGfc0xqxTra776rn9S3u3r454hOJis5IRBpgZlvCp78GRofzdHzfzLLN7LdmNje8yeG3wvpjwnku/kwwcA0z+1t4U8yFe26MaWa/BtqE23uo+nuFo6d/a2bvWzBXzORq255pZtPN7AMzeygcoY2Z/drMFoWx3NCSx0gyW07UAYikkKuodkYSJoSN7j7SzFoDr5vZc2Hdw4EhHtxyHeAb7l5hZm2AuWb2qLtfZWaXufuwWt7rqwQ3TjwU6BKu82r42nDgYIL7Ob0OHG1miwhuaXKgu7uZdW72vRepg85IRJpuHMG9l+YT3H68gOAeVABvVUsiAN81swXAbIIb/A2kfscAf/HgBoprgVeAkdW2vcqDGyvOB/oCm4BK4I9m9lVgW9x7JxIjJRKRpjPgO+4+LCxF7r7njGTr3kpB28qJwFHufijwDpAXw7brsr3a811AjrtXEZwFPUowEdMzjdoTkTgokYjEbjPBlKx7PAt8O7wVOWY2KLyDck2dgA3uvs3MDiSYpnWPnXvWr+FVYHLYDlNIMC3uW3UFFs6p0cndnwIuJ7gsJtIi1EYiErt3garwEtV9wC0El5XeDhu8y/lyWtbqngEuNrN3Ce6cO7vaa3cC75rZ2+7+79WWPw4cRXBXZgd+5O5rwkRUmw7A380sj+Bs5vtN20WRxlP3XxERiYsubYmISFyUSEREJC5KJCIiEhclEhERiYsSiYiIxEWJRERE4qJEIiIicVEiERGRuPw/2hByVbSxTMsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_moms=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-2), \n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = (1e-2,1e-3), lr_decay=(DecayType.POLYNOMIAL, 2)),\n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "85dd21ee2f0b43aebe4ff2b0f5b8a39f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.385298   4.559372   0.1401    \n",
      "    1      3.410444   4.544588   0.1373                                                                                \n",
      "    2      3.403002   4.568612   0.1398                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.56861]), 0.1398]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this last POLYNOMIAL decay type needs a second argument: it's the value of the power you want in your polynomial decay. The formula that gives your update is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_i = end_lr + (start_lr - end_lr) * (1 - i/n) ** p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where p is this extra argument. Below, we can see the result for p = 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW9//HXJwkJBJAlhB1MQEAWFTEuiAsKClIFW2nF27pU7rVWrbXq9afdrtfb3tbWVtteN1pcqrZKQSu2KG5FrSgQFJRVAkiJbFGQ1QCBz++Pc8AYkjDJZHIyM+/n4/F9zOQ733Pmc2YSPpxzvou5OyIiIvWVEXUAIiKS3JRIREQkLkokIiISFyUSERGJixKJiIjERYlERETiokQiIiJxUSIREZG4KJGIiEhcsqIOoDF06NDBCwoKog5DRCRpzJ8//2N3z4+lbVokkoKCAoqLi6MOQ0QkaZjZmljb6tKWiIjERYlERETiokQiIiJxUSIREZG4KJGIiEhclEhERCQuSiQiIhIXJRIREYmLEomIiMRFiURE4vfyy0GRtJQWU6SISIL95CfB48iR0cYhkdAZiYiIxEWJRERE4qJEIiIicVEiERGRuOhmu4jE78EHo45AIqREIiLx69cv6ggkQrq0Jcll+PCgSNPy3HNBkbSkMxIRid+vfhU8XnBBtHFIJHRGIiIicVEiERGRuCiRiIhIXBKaSMxstJktN7MSM7u1mtdzzOyp8PU5ZlYQ1ueZ2T/MbIeZ/V+VbU4ws/fDbX5rZpbIY5AmZurUoIhIk5Gwm+1mlgncC5wDlALzzGy6uy+p1GwisMXdjzKzCcCdwMVAOfAjYFBYKrsfuAp4G5gBjAaeT9RxSBPToUPUEUh1Hnss6ggkQonstXUSUOLuqwDM7ElgHFA5kYwDbg+fTwX+z8zM3XcC/zSzoyrv0My6AEe4+1vhz38ELiRBieS5hevY756IXUs99Zg+hTbNm9H7luuiDkUq69Ej6ggkQolMJN2AtZV+LgVOrqmNu1eY2VYgD/i4ln2WVtlnt+oamtlVBGcu9OzZs66xA3DL1Pf4bO++em0rifHknx6iDGhx1US6tm0RdThywFNPBY8XXxxtHBKJRCaS6u5dVP3vfSxt6tXe3ScBkwCKiorqdVox47un4zojaVLy/nEES9dv48XFG7hiWGHU4cgB998fPCqRpKVEJpJSoPL5bndgXQ1tSs0sC2gDbD7MPrsfZp8NprBDy0TtWuqrRTNaZGfyghKJSJORyF5b84A+ZlZoZtnABGB6lTbTgcvD5+OBV72WUwB3Xw9sN7NTwt5alwHPNnzo0pS1b5nN3NWb2bxzT9ShiAgJTCTuXgFcB8wElgJT3H2xmd1hZmPDZpOBPDMrAW4EDnYRNrMPgV8DV5hZqZkNCF/6NvAHoARYiXpspZ32udnsd3h5ycaoQxEREjzXlrvPIOiiW7nux5WelwNfrWHbghrqizm0S7CkixkzyHWn2+/mMHPxBr52onoLiURNkzZKcsnNxYBRAzvz+Jw17NhdQasc/RpHToNE05qmSJHkct99cN99jB7UmT0V+5m1fFPUEQkEA0U1WDRtKZFIcpkyBaZM4YQj25HXMpuZi3WfpEl45JGgSFpSIpGklJlhnDOgE/9YtondFRo0GjklkrSmRCJJa9SgzuzYXcHskk+iDkUkrSmRSNI6tXcerXKyeGHRhqhDEUlrSiSStHKyMjn76I68tHQj+/ZrKhuRqCiRSHKZNSsooVEDO7N55x7mfVjbzDoikkjqgC9JbXi/fLKzMpi5eAOn9MqLOpz0NWPG4dtIytIZiSSXu+4KSqhlThZn9OnAzEUb2K/LW9HJzQ2KpCUlEkkuf/tbUCoZc0wX1m0tZ0HppxEFJQcGikp6UiKRpDdyQCeyMzP4+3vrow4lfYUDRSU9KZFI0juieTPO6NuB599fr8tbIhFQIpGU8KVjg8tb767V5S2RxqZEIsmlRYugVDGivy5viURFiUSSy/PPB6WK4PJWPs8v0uUtkcamRCIp4/xju7B+aznvrt0SdSjpp8pAUUkvSiSSXP7nf4JSjRH9O5KdlcHfdHlLpFEpkUhyeeWVoFSjdfNmnNk3n+ff1+DERldloKikFyUSSSnnH9uFDdvKeedfurzVqKoZKCrpQ4lEUsqI/p3Izsrg7+/r8pZIY1EikZTSKieL4X3zmaHBiSKNRolEkkteXlBq8aVju7Bx227m6/KWSKPQNPKSXKZNO2yTg5e33lvPiQXtGyEoqW6QqKQPnZFIymmVk8WIozvyt/fWU7Fvf9ThpIcaBopKelAikeRy221BOYxxg7vy8Y7dvLXqk0YISiS9KZFIcnnrraAcxvB+HWmdk8WzC9Y1QlBS20BRSX0JTSRmNtrMlptZiZndWs3rOWb2VPj6HDMrqPTabWH9cjMbVan+e2a22MwWmdmfzax5Io9BklPzZpmMHtSZFxZtoHzvvqjDSX21DBSV1JewRGJmmcC9wHnAAOASMxtQpdlEYIu7HwXcDdwZbjsAmAAMBEYD95lZppl1A64Hitx9EJAZthM5xLjB3dixu4J/LNsUdSgiKS2RZyQnASXuvsrd9wBPAuOqtBkHPBo+nwqMMDML6590993uvhooCfcHQU+zFmaWBeQCunYh1RraO48OrXJ0eUskwRKZSLoBayv9XBrWVdvG3SuArUBeTdu6+0fAXcC/gPXAVnd/MSHRS9PUvXtQYpCZYVxwXBdeXb6JrZ/tTXBgIukrkYnEqqmrOtS4pjbV1ptZO4KzlUKgK9DSzL5R7ZubXWVmxWZWXFZWVoewpUl7/PGgxOjCwd3YU7GfmYs2JDAoiWWgqKSuRCaSUqBHpZ+7c+hlqINtwktVbYDNtWw7Eljt7mXuvhd4Gji1ujd390nuXuTuRfn5+Q1wOJKMju3ehoK8XJ5d+FHUoaS2adNiGiwqqSmRiWQe0MfMCs0sm+Cm+PQqbaYDl4fPxwOvuruH9RPCXl2FQB9gLsElrVPMLDe8lzICWJrAY5Cm5oYbghIjM2Ps4G7MXvkJm7aVJzAwkfSVsEQS3vO4DphJ8I/9FHdfbGZ3mNnYsNlkIM/MSoAbgVvDbRcDU4AlwAvAte6+z93nENyUfwd4P4x/UqKOQZqgBQuCUgdjj+uKOzynBa8SJ8aBopKaLDgBSG1FRUVeXFwcdRjSEIYPDx7ruKzr+b97g0wznr3utAYPSaj39yJNl5nNd/eiWNpqZLukhXHHdWNh6VZWf7wz6lBEUo4SiaSFC47rihk8805p1KGIpBwlEkkuffsGpY46t2nOaUd1YNo7H2nBK5EGpkQiyWXSpKDUw0VDuvPRp58xZ/XmBg5K6jJQVFKPFraStDFqYGda5WQx7Z1ShvbW4LkGVYdBopJ6dEYiyeWqq4JSDy2yMxlzTGeef389u/ZUNHBgIulLiUSSywcfBKWeLhrSnZ179vGCpkxpWHUcKCqpRYlE0sqJBe3p0b4F09R7q2HVY6CopA4lEkkrGRnGV47vzuyVn7Du08+iDkckJcScSMysZSIDEWksFw3pjjs8864mchRpCIdNJGZ2qpktIZwc0cyOM7P7Eh6ZSHUGDw5KHHrm5XJSQXumzS8lHaYIEkm0WM5I7gZGAZ8AuPtC4IxEBiVSo3vuCUqcLjqhG6s+3sm7az9tgKCkvgNFJTXEdGnL3ddWqdqXgFhEGs2YY7rQvFkG0+brpnuDiGOgqCS/WBLJWjM7lWCFwmwzuxmtASJR+cY3ghKn1s2bMXpgZ6YvXEf5Xv2/SCQesSSSq4FrCdZRLwUGA9ckMiiRGpWWBqUBfO3EHmwvr+D5RVqnJG5xDBSV5BdLIunn7l93907u3tHdvwH0T3RgIok2tFceBXm5/Hlu1Su3UmdxDhSV5BZLIvldjHUiScXM+NqJPZi7ejOrynZEHY5I0qoxkZjZUDO7Ccg3sxsrlduBzEaLUCSBxg/pTmaG8VSxzkpE6qu2M5JsoBXBDMGtK5VtwPjEhyZSjaFDg9JAOh7RnBFHd2Ta/FL27tvfYPsVSSc1TiPv7q8Br5nZI+6+phFjEqnZz37W4LuccFIPXlyykVeWbmT0oC4Nvv+0EOcgUUlusaxHssvMfgkMBJofqHT3sxMWlUgjOqNPPp2PaM6T89YqkdRXAwwSleQVy832J4BlQCHw38CHwLwExiRSs4suCkoDysrM4GtF3XntgzJN5ChSD7Ekkjx3nwzsdffX3P1K4JQExyVSvU8+CUoD+2pRDwD+UqyR7vXSQANFJTnFkkj2ho/rzexLZnY8oMWZJaX0aJ/LaUd1YErxWvbt10SOddaAA0Ul+cSSSH5iZm2Am4CbgT8A30toVCIRmHBiTz769DNeX1EWdSgiSaXWRGJmmUAfd9/q7ovc/Sx3P8HdpzdSfCKN5pwBnejQKpsn3lYnRZG6qDWRuPs+YGwjxSJyeCNGBCUBsrMymHBiT15Ztom1m3cl5D1EUlEsl7Zmm9n/mdnpZjbkQIll52Y22syWm1mJmd1azes5ZvZU+PocMyuo9NptYf1yMxtVqb6tmU01s2VmttTMGm50mjR9P/pRUBLk307uiQF/nvuvhL1HSmrggaKSXGIZR3Jq+HhHpToHah1HEl4Wuxc4h2DW4HlmNt3dl1RqNhHY4u5HmdkE4E7gYjMbAEwgGLvSFXjZzPqGZ0i/AV5w9/Fmlg3kxnAMIjHp2rYFI/t34ql5a/nuyD7kZGk2oJgkYKCoJI/DnpGE90WqllgGI54ElLj7KnffAzwJjKvSZhzwaPh8KjDCzCysf9Ldd7v7aqAEOMnMjiBYnXFyGNsed9cSd+nkvPOCkkCXDj2ST3bu4YVFGxL6PiKpIqYVEuupG1B5JrzSsK7aNu5eAWwF8mrZthdQBjxsZu+a2R/MrGViwpcm6bPPgpJAw3p3oLBDSx57SzfdY5aAgaKSPBKZSKyauqod9GtqU1N9FjAEuN/djwd2AofcewEws6vMrNjMisvK1J1TYpeRYXz95J4Ur9nCknXbog4nOSRooKgkh0QmklKgR6WfuwPrampjZllAG2BzLduWAqXuPiesn0qQWA7h7pPcvcjdi/Lz8+M8FEk3Xz2hB82bZfD4HJ2ViBzOYROJmX2lmjLCzDoeZtN5QB8zKwxvik8Aqo4/mQ5cHj4fD7zq7h7WTwh7dRUCfYC57r6BYA35fuE2I4AliDSwNrnNGHtcV/767kdsK997+A1E0lgsvbYmAkOBf4Q/DwfeBvqa2R3u/lh1G7l7hZldB8wkWAjrIXdfbGZ3AMXhoMbJwGNmVkJwJjIh3HaxmU0hSBIVwLVhjy2A7wBPhMlpFfDNuh50stu5cyezZs1ixYoVbN++nebNm9OxY0dOO+00evfuDcCWLVt44403WLlyJTt27CA3N5e8vDwGDx7MMcccQ2Zm0Bvp9ttvP7jfZs2a0apVK7p3786JJ55Iz549ozi82p1/fqO91aWnFDCluJSn55dyxbDCw7ZP6++lidJ30jhiSST7gf7uvhHAzDoB9wMnA68D1SYSAHefAcyoUvfjSs/Lga/WsO1PgZ9WU78AKIoh7pQ1ZcoU9u7dy9ixY2nfvj07d+5kzZo1fBbehF63bh2PPvoo+fn5jBkzhg4dOrB3717Kysp45513aN++/Rd+8ceOHUvfvn2pqKhgy5YtLFy4kIcffpiRI0cybNiwqA6zejff3GhvdUz3Ngzu0ZZH31rDZUMLyMio7tbd59L6e0nQINF4pfV30ohiSSQFB5JIaBPQ1903m5nO+RtZeXk5a9as4bLLLqNXr14AtG3blm7dgg5x7s4zzzxDXl4eEydOJOhNHejcuTPHHHMMwdXDzzVv3pxWrVod3FdhYSGtW7fmlVdeoX///rRv376Rjq7pufK0Qq7/87v8Y/kmRvTvVGO7tP9eEjhItL7S/jtpRLHcbH/DzP5mZpeb2eXAs8DrYbdbjeFoZNnZ2WRnZ7N8+XIqKioOeX3Dhg2UlZVx6qmnfuEPo7Ka6isbOnQo7s6yZcvijrlBDR8elEZy3qDOdGnTnIfeXF1ru7T/XpogfSeNJ5YzkmuBi4BhBN1y/whMC2+Kn5XA2KQaGRkZXHjhhTz33HPMnz+fzp0707NnTwYMGED37t35JOyCmZeXd3Cb8vJyfv3rXx/8+fTTT+f000+v9X1yc3Np2bIlW7ZsScyBJIlmmRlcNrSAO19YxtL12+jf5Yhq26X993JgkOjzz0cbRyVp/500osMmkjBhTA2LNAEDBgygb9++rFmzhtLSUkpKSpg9ezYjRoygXbt2h7TPycnh6quvBuCJJ55g3759h7SpTtXT+nR1yUk9+O0rK3j4zdX8YvxxNbZL6+8lwYNE6yutv5NGFGv33xVmttXMtpnZdjPTKK2IZWVl0bt3b84880wmTpzIkCFDmDVrFm3btgXg448/PtjWzGjfvj3t27c/2APlcHbt2sWuXbuq/WNLN21zs7nohG78dcE6Pt6xu9a2+l6aHn0niRfLPZJfAGPdvY27H+Hurd29+vN7iUx+fj779++nQ4cO5Ofn8+abb7J///5672/27NmYGUcffXQDRpm8vjmskD0V+3ni7brNCqzvpenRd9LwYrlHstHdlyY8EonJrl27+Mtf/sLxxx9Pp06dyM7OZt26dbz55psUFhbSvHlzLrzwQv74xz8yefJkTj/99IN/OGvXrmXbtm2H3EAsLy9nx44d7Nu3j82bN7Nw4UIWLlzIOeec0/R6oXzta5G8be/8VpzVL5/H3l7D1cN7HTIrcNp/L02QvpPGY4e7tmdmvwE6A38FDp7Xu/vTiQ2t4RQVFXlxcXHUYTSIiooKXnvtNVauXMnmzZvZt28frVu3pl+/fpxxxhm0aNECgM2bN/PGG2+watUqduzYQVZWFp06dWLQoEEMGTKErKzg/xCVB1llZWV9YZDVkUceGcUhNllvrCjj0slzueurxzH+hO5feC3tv5e77goeG3Gcz+Gk/XcSJzOb7+4xjdmLJZE8XE21u/uV9QkuCqmUSNLernDlwtzGX4bG3Rl1z+tkZmQw4/rTYuoaKpKs6pJIYum1lXZTkEgTNmZM8DhrVqO/tZnx76f34pap7/H6io85s68mAxWBWhKJmd3i7r8ws99x6PTvuPv1CY1MpAm6cHA3fv3iBzwwa6USSWUHBolGkOAlerWdkRy4wa5rQiKh7KwMJp5WyE9nLGXB2k8Z3KNt1CGJRK7GROLuz4WPj9bURiQdXXJyT3736goemLWSBy49IepwRCJ32HskZtYXuBkoqNw+xnXbRVJOq5wsLhtawL2zSlhZtoPe+a2iDkkkUrGMI/kL8ADwByC2+QJEEuWKK6KOAIArhhXw+zdWMem1Vdw5/tiowxGJVCyJpMLd7094JCKxaCKJpEOrHL5W1IOn5q3lxnP70umI5lGHFK2IBopK0xDLFCnPmdk1ZtbFzNofKAmPTKQ6H38clCbgP07vRcX+/Tz0z9qnmE8L11wTFElLsZyRHFhT/T8r1TnQq+HDETmM8eODxybQzbRnXi5fOrYrj7+9hm8P703b3OyoQ4pOhANFJXq1npGYWQbwDXcvrFKURESAa8/qzc49+3RWMmbM54NFJe3UmkjcfT9wVyPFIpJ0ju58BOcN6szDb37I1l1aeVrSUyz3SF40s4tMEwuJVOv6EX3YvruCyYdZjlckVcWSSG4k6AK8WwtbiRyqf5cjGDWwEw+/uZqtn+msRNLPYRNJuJBVhrtna2Eridy3vx2UJub6EX3YXl7BwzorkTQUS68tzKwd0Ac42Fne3V9PVFAiNbr44qgjqNbArm04d0AnHvrnaq48rZAjmjeLOqTG1UTG90g0Ylmz/d+B14GZwH+Hj7cnNiyRGqxdG5Qm6PoRfdhWXsEjb34YdSiN74orlEzSWCz3SL4LnAiscfezgOOBsoRGJVKTSy8NShM0qFsbRvbvxB/eWJV+90qa0EBRaXyxJJJydy8HMLMcd18G9EtsWCLJ6XvnBGclv399VdShNK7x4z8fLCppJ5ZEUmpmbQnWbH/JzJ4F1iU2LJHkNLBrGy44riuT/7maTdvLow5HpFHE0mvry+7+qbvfDvwImAxcGMvOzWy0mS03sxIzu7Wa13PM7Knw9TlmVlDptdvC+uVmNqrKdplm9q6Z/S2WOEQa003n9GXvvv3c+2pJ1KGINIpYzkgws9PM7Jvu/hrwFtAthm0ygXuB84ABwCVmNqBKs4nAFnc/CrgbuDPcdgAwARgIjAbuC/d3wHf5fAVHkSaloENLvnZiD/4091+s3bwr6nBEEi6WXlv/Bfw/4LawqhnweAz7PgkocfdV7r4HeBIYV6XNOODACoxTgRHhCPpxwJPuvtvdVwMl4f4ws+7AlwjWR5F0c9NNQWnirj+7Dxlm3P3SB1GHIpJwsYwj+TJBT613ANx9nZm1jmG7bkDlfpqlwMk1tXH3CjPbCuSF9W9X2fbAWdA9wC1ArTGY2VXAVQA9e/aMIVxJChdcEHUEMencpjlXDCtg0uuruOrMXhzdOcXH8DbBQaLSeGK5tLXH3Z1g6njMrGWM+65ubi6PsU219WZ2PrDJ3ecf7s3dfZK7F7l7UX5+/uGjleSwfHlQksC3z+xNq5ws7pqZHPHG5eKLm+xgUUm8WBLJFDN7EGhrZv8BvAz8PobtSoEelX7uzqG9vQ62MbMsoA2wuZZthwFjzexDgktlZ5tZLJfZJFV861tBSQJtc7O5+szevLx0E7NXpvgYiyY8UFQSL5ZeW3cR3L+YRjB+5Mfu/rsY9j0P6GNmhWaWTXDzfHqVNtP5fOGs8cCr4dnPdGBC2KurkGB6lrnufpu7d3f3gnB/r7r7N2KIRSQSE08rpFvbFvzkb0vZt7/qCXkKacIDRSXxYuq15e4vuft/uvvN7v5SjNtUANcRTKmyFJji7ovN7A4zGxs2mwzkmVkJwSzDt4bbLgamAEuAF4Br3X1fXQ5MpClo3iyTW0b3Y8n6bUx7pzTqcEQSosab7Wa2nUPvaUBw/8JjmQHY3WcAM6rU/bjS83LgqzVs+1Pgp7XsexYw63AxiERt7HFdeWT2h/xy5nK+dEwXWubENFeqSNKo8YzkwHTx1RRNIy9SB2bGj84fQNn23Tzw2sqowxFpcPqvkSSXH/4w6gjqZUjPdow9riuTXl/FJSf1pGvbFlGHJNJgYrpHItJkjBwZlCR0y+hgrtOfP78s4kgSIEkGikpiKJFIclmwIChJqHu7XL51Zm+mL1zH7JIU6w58wQVJM1hUGp4SiSSXG24ISpK6ZnhverRvwY+eXcSeiv1Rh9NwkmigqDQ8JRKRRtS8WSZ3jB3EyrKdTP5nCq3vnkQDRaXhKZGINLKzju7IuQM68dtXVvDRp59FHY5I3JRIRCLw4wsG4Dh3PLc46lBE4qZEIhKB7u1yuX5EH2Yu3siryzZGHY5IXJRIJLn87/8GJQX8+2m96NOxFT94ZhHby/dGHY5IvSmRSHI59dSgpIDsrAx+Mf5YNm4r52fJPrbkhz9M2sGiEj8lEkkus2cHJUUc37MdVw4r5E9z/sVbKz+JOpz6S+KBohI/JRJJLt//flBSyE3n9uPIvFxuffo9PtuTpJNcJ/FAUYmfEolIxFpkZ/LzrxzLmk928asXk3RQX5IPFJX4KJGINAFDe+fx9ZN7MvnN1cxdvTnqcETqRIlEpIm4bUx/erbP5XtPLWCbenFJElEiEWkiWuVkcffFg9mwrZzbn9VARUkeWo9Ekss990QdQUIN6dmO6846it+8soKzju7IBcd1jTokkcNSIpHkMnhw1BEk3HfOPorXPijjB8+8zwlHtkuORbBSZJCo1I8ubUlyefnloKSwrMwM7rl4MBX7ne8++S579yXBdPMpNFBU6k6JRJLLT34SlBRX0KElP/vKMcz7cAt3zUyCLsEpNlBU6kaXtkSaqHGDuzHvw808+PoqhhzZjlEDO0cdUs0ODBKdNSvSMCQaOiMRacJ+dP4Aju3ehpv/spA1n+yMOhyRaimRiDRhOVmZ3PtvQ8gw49uPv5O8U6hISlMiEWnierTP5e6Lj2Pphm3cPHUh7h51SCJfoHskklwefDDqCCJx9tGduHX00fzs+WX06diKG0b2jTokkYOUSCS59OsXdQSRueqMXqzYtIN7Xl7BUR1bcf6xTWiwYooPFJXaJfTSlpmNNrPlZlZiZrdW83qOmT0Vvj7HzAoqvXZbWL/czEaFdT3M7B9mttTMFpvZdxMZvzRBzz0XlDRkZvz0y4M4saAdN01ZyIK1n0Yd0ucGD06LwaJSvYQlEjPLBO4FzgMGAJeY2YAqzSYCW9z9KOBu4M5w2wHABGAgMBq4L9xfBXCTu/cHTgGurWafksp+9augpKmcrEwe+MYJdDwihysfmUfJph1RhxRIg4GiUrNEnpGcBJS4+yp33wM8CYyr0mYc8Gj4fCowwswsrH/S3Xe7+2qgBDjJ3de7+zsA7r4dWAp0S+AxiDQ5ea1yeOzKk8kwuPyhuazf+lnUIaXNQFGpXiITSTdgbaWfSzn0H/2Dbdy9AtgK5MWybXgZ7HhgTgPGLJIUCjq05JFvnsTWz/Zy2eS5bNm5J+qQJI0lMpFYNXVV+y3W1KbWbc2sFTANuMHdt1X75mZXmVmxmRWXlZXFGLJI8hjUrQ2/v6yINZt3ccXDc9n6mdYwkWgkMpGUAj0q/dwdWFdTGzPLAtoAm2vb1syaESSRJ9z96Zre3N0nuXuRuxfl5+fHeSgiTdPQ3nnc//UhLFm/jUsnz2HrLiUTaXyJTCTzgD5mVmhm2QQ3z6dXaTMduDx8Ph541YPRVtOBCWGvrkKgDzA3vH8yGVjq7r9OYOzSVD32WFDkoBH9O/HAN05g2frtfH3y23y6S5e5pHElLJGE9zyuA2YS3BSf4u6LzewOMxsbNpsM5JlZCXAjcGu47WJgCrAEeAG41t33AcOAS4GzzWxBWMYk6hikCerRIyjyBSP6d+LBS0/ggw07mDDpbTZuK2/cAB58MG0HiwpYOky3UFRU5MXFxVGHIQ3hqaeCx4svjjaOJur1D8q4+vH5tMvN5tErT+Kojq2iDknivBBIAAAKe0lEQVSSlJnNd/eiWNpqri1JLvffHxSp1hl983nqqqHsrtjH+AdmM3/N5sZ54zQeKCpKJCIp55jubXj628No26IZl/x+Dn8pXnv4jeKV5gNF050SiUgK6pmXy9PXDOPEgnb859T3+PGzi9hTkQRL9kpSUiIRSVHtW2bz6DdP4qozevHHt9Zwye/fZu3mXVGHJSlIiUQkhWVlZvD9Mf353SXH88GG7Zz3mzd4+p1SrWkiDUrTyEtymTo16giS0gXHdWVwj7bcNGUhN05ZyEtLNnL72IF0OqJ51KFJClD3X5E0sm+/M+n1Vdz98gdkZ2Zww8g+XHFqAVmZcV6cWBve0NcYn5RRl+6/SiSSXB55JHi84oooo0h6az7ZyX9NX8ys5WUc1bEVN53Tl1EDO5ORUd00d5KOlEiqUCJJIcOHB4+zZkUZRUpwd15cspFfzlxOyaYdDOx6BN85uw/nDOhEZl0TigaKppy6JBLdIxFJU2bGqIGdGdm/E3999yN+88oKrn58Pt3atuDyU4/kwuO70bF1jPdQDgwSVSJJS0okImkuM8O46ITujBvclZeXbuThNz/kf2cs4+fPL2No7zzGHNOF04/Kp0f7FgTzpop8kRKJiABBV+HRg7owelAXVmzcznML1zF94Tp+8MwiALq1bcEJR7ajX+fW9O3Umm5tW9ChVTZtcpuRHe5DaSY9KZGIyCH6dGrNjef243vn9KVk0w7eWvUJs0s+Yf6aLUxfWHVZIXhy1ScAXHLb35VMmpAOrXKY+4ORCX8fJRJJLjNmRB1BWjEz+nRqTZ9OrblsaAEA28v3smLTDjZtK+fjHXv4dNceuj+fiwPXnXVUpPHKF+VmN84/8Uokklxyc6OOIO21bt6MIT3bfbHylb8DcFOHDhFEJFFTIpHkct99weM110Qbh3yREkha01xbklymTAmKNC2PPPL5YFFJO0okIhI/JZK0pkQiIiJxUSIREZG4KJGIiEhc1GtLkosmaxRpcpRIRCR+Giia1pRIRCR+Giia1nSPRETid999nw8WlbSjRCIi8dNA0bSmRCIiInFJaCIxs9FmttzMSszs1mpezzGzp8LX55hZQaXXbgvrl5vZqFj3KSIijSthicTMMoF7gfOAAcAlZjagSrOJwBZ3Pwq4G7gz3HYAMAEYCIwG7jOzzBj3KSIijSiRZyQnASXuvsrd9wBPAuOqtBkHPBo+nwqMsGAtz3HAk+6+291XAyXh/mLZp4iINKJEdv/tBqyt9HMpcHJNbdy9wsy2Anlh/dtVtu0WPj/cPkWksWmgaFpL5BlJdStueoxt6lp/6JubXWVmxWZWXFZWVmugIiJSf4lMJKVAj0o/dweqLvZ8sI2ZZQFtgM21bBvLPgFw90nuXuTuRfn5+XEchoiI1CaRiWQe0MfMCs0sm+Dm+fQqbaYDl4fPxwOvuruH9RPCXl2FQB9gboz7FBGRRpSweyThPY/rgJlAJvCQuy82szuAYnefDkwGHjOzEoIzkQnhtovNbAqwBKgArnX3fQDV7TNRxyAiIodnwQlAaisqKvLi4uKowxARSRpmNt/di2Jpq5HtIiISFyUSERGJixKJiIjERYlERETikhY3282sDFhTz807AB83YDjJKN0/g3Q/ftBnAOn3GRzp7jENwkuLRBIPMyuOtedCqkr3zyDdjx/0GYA+g9ro0paIiMRFiUREROKiRHJ4k6IOoAlI988g3Y8f9BmAPoMa6R6JiIjERWckIiISFyWSGqTT2vBm9qGZvW9mC8ysOKxrb2YvmdmK8LFdWG9m9tvwc3nPzIZEG339mNlDZrbJzBZVqqvzMZvZ5WH7FWZ2eXXv1VTV8BncbmYfhb8LC8xsTKXXbgs/g+VmNqpSfVL+rZhZDzP7h5ktNbPFZvbdsD6tfg8ahLurVCkEMwuvBHoB2cBCYEDUcSXweD8EOlSp+wVwa/j8VuDO8PkY4HmCRcZOAeZEHX89j/kMYAiwqL7HDLQHVoWP7cLn7aI+tjg/g9uBm6tpOyD8O8gBCsO/j8xk/lsBugBDwuetgQ/C40yr34OGKDojqZ7Whg+O99Hw+aPAhZXq/+iBt4G2ZtYligDj4e6vEyxdUFldj3kU8JK7b3b3LcBLwOjER98wavgMajIOeNLdd7v7aqCE4O8kaf9W3H29u78TPt8OLCVY0jutfg8aghJJ9apbb75bDW1TgQMvmtl8M7sqrOvk7ush+IMDOob1qfzZ1PWYU/WzuC68dPPQgcs6pPhnYGYFwPHAHPR7UGdKJNWLeW34FDHM3YcA5wHXmtkZtbRNt88Gaj7mVPws7gd6A4OB9cCvwvqU/QzMrBUwDbjB3bfV1rSaupT4DOKlRFK9mNeGTwXuvi583AQ8Q3C5YuOBS1bh46aweSp/NnU95pT7LNx9o7vvc/f9wO8JfhcgRT8DM2tGkESecPenw+q0/z2oKyWS6qXN2vBm1tLMWh94DpwLLCI43gO9Ty4Hng2fTwcuC3uwnAJsPXAZIAXU9ZhnAueaWbvwEtC5YV3SqnK/68sEvwsQfAYTzCzHzAqBPsBckvhvxcyMYLnvpe7+60ovpf3vQZ1Ffbe/qRaCHhofEPRI+UHU8STwOHsR9LRZCCw+cKxAHvAKsCJ8bB/WG3Bv+Lm8DxRFfQz1PO4/E1y62UvwP8qJ9Tlm4EqCG88lwDejPq4G+AweC4/xPYJ/OLtUav+D8DNYDpxXqT4p/1aA0wguQb0HLAjLmHT7PWiIopHtIiISF13aEhGRuCiRiIhIXJRIREQkLkokIiISFyUSERGJixKJyGGY2ezwscDM/q2B9/396t5LJJmo+69IjMxsOMHMuOfXYZtMd99Xy+s73L1VQ8QnEhWdkYgchpntCJ/+HDg9XKfje2aWaWa/NLN54SSH3wrbDw/XufgTwcA1zOyv4aSYiw9MjGlmPwdahPt7ovJ7haOnf2lmiyxYK+biSvueZWZTzWyZmT0RjtDGzH5uZkvCWO5qzM9I0ltW1AGIJJFbqXRGEiaEre5+opnlAG+a2Yth25OAQR5MuQ5wpbtvNrMWwDwzm+but5rZde4+uJr3+grBxInHAR3CbV4PXzseGEgwn9ObwDAzW0IwpcnR7u5m1rbBj16kBjojEam/cwnmXlpAMP14HsEcVABzKyURgOvNbCHwNsEEf32o3WnAnz2YQHEj8BpwYqV9l3owseICoADYBpQDfzCzrwC74j46kRgpkYjUnwHfcffBYSl09wNnJDsPNgrurYwEhrr7ccC7QPMY9l2T3ZWe7wOy3L2C4CxoGsFCTC/U6UhE4qBEIhK77QRLsh4wE/h2OBU5ZtY3nEG5qjbAFnffZWZHEyzTesDeA9tX8TpwcXgfJp9gWdy5NQUWrqnRxt1nADcQXBYTaRS6RyISu/eAivAS1SPAbwguK70T3vAu4/NlWSt7AbjazN4jmDn37UqvTQLeM7N33P3rleqfAYYSzMrswC3uviFMRNVpDTxrZs0Jzma+V79DFKk7df8VEZG46NKWiIjERYlERETiokQiIiJxUSIREZG4KJGIiEhclEhERCQuSiQiIhIXJRIREYnL/wfkRG+etOJq2gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_moms=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-2), \n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = (1e-2,1e-3), lr_decay=(DecayType.POLYNOMIAL, 0.5)),\n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce770c4307a741359bb36ae93a6626f3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.624198   4.435553   0.1493    \n",
      "    1      3.594141   4.429299   0.152                                                                                 \n",
      "    2      3.616898   4.439738   0.1514                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.43974]), 0.1514]"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is an example where p = 0.5. In general, the greater p is, the more 'exponential' your curve will look (p=2 wasn't very different from exponential already). p=1 is simply linear (and close values will give something close to a line). Lower values of p (close to 0) will give a curve that stays up a bit longer before going down.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8FdX9//HXJwkhhLBlAZSwKosBFzAoqKitWHexVQtt3aoWrdrW1v6sfvu19Wvb77eLta3WtS5VtCpiVWy1bhS1rgQFFQGNKCWiLJKyypLw+f0xA8aQ5SY3N3Nv5v18PM7j3sycmfuZuQkfZs6cc8zdERERaa2sqAMQEZHMpkQiIiJJUSIREZGkKJGIiEhSlEhERCQpSiQiIpIUJRIREUmKEomIiCRFiURERJKSE3UA7aG4uNgHDRoUdRgiIhlj7ty5q929JJG6sUgkgwYNoqKiIuowREQyhpktTbSubm2JiEhSlEhERCQpSiQiIpIUJRIREUmKEomIiCRFiURERJKiRCIiIklRIhERkaQokYiISFKUSEQkeU8/HRSJpVgMkSIiKfbznwevEydGG4dEQlckIiKSFCUSERFJihKJiIgkRYlERESSosZ2EUnezTdHHYFESIlERJI3fHjUEUiEdGtLMsvhhwdF0sujjwZFYklXJCKSvN/+Nng94YRo45BI6IpERESSokQiIiJJUSIREZGkpDSRmNnRZrbYzCrN7LIG1nc2s/vD9a+Y2aBweZGZ/dPMNpjZH+tts7+ZvRluc62ZWSqPQdLMjBlBEZG0kbLGdjPLBq4HjgSqgDlmNtPd365T7Ryg2t33NLMpwK+AycBm4ApgVFjquhGYCrwMPAYcDTyequOQNFNcHHUE0pBp06KOQCKUyqe2DgAq3X0JgJndB0wC6iaSScCV4fsZwB/NzNx9I/AvM9uz7g7NbDegu7u/FP58F3ASKUokM+cvZ/t2r/P5Tdevf3FUv3rd1VZvbf19N7Vt/bXNb9t4XLtsu8u+mt755/eV+PE3tO/Pn5/6GwcvJQ/eS5dO2ZR+/4L6NSRK/ftHHYFEKJWJpB+wrM7PVcCBjdVx9xozWwsUAaub2GdVvX32a6iimU0luHJhwIABLY0dgB/NeINPt9W2altJjfv+cgsANw2ewE9PGEmnbDXzpYX77w9eJ0+ONg6JRCoTSUP/f/dW1GlVfXe/BbgFoLy8vKl9Nurx703YuXP3z++i/g59l0/wRtc1t63Xq1F3fUvq7hJRC7fdNU5vYt0un5bwvndZ18S5LpvdneqN27j75X/z3sqN3PCNMfTqmlv/w6W93Xhj8KpEEkupTCRVQN3r3VJgeSN1qswsB+gBrGlmn6XN7LPNDCrumqpdS2vldaJ7Xieu+eq+XPbXNznx+n9x6xljGd63W9SRicRWKu8LzAGGmtlgM8sFpgAz69WZCZwZvj8FmOX1/ztah7t/BKw3s3Hh01pnAI+0feiS7r4yppT7p45jy7btfOWGF3jq7RVRhyQSWylLJO5eA1wEPAEsBKa7+wIzu8rMTgyr3QYUmVkl8ANg5yPCZvYBcA1wlplVmVlZuOrbwK1AJfAeemIrtkYP6MXMiw5hj94FTJ1WwfX/rNzltpiIpJ7F4Q+vvLzcKyoqog5D2sKmTcFrfv7ORZu31XLpjDeYOX85J+67O78+ZR/yOmVHFGBM7RhIc/bsKKOQNmRmc929PJG6GrRRMkudBLJDXqds/jBlP4b37cbVTy7m/dUb+dMZ5fTtkRdBgDGlTqKxpmcnJbPccENQ6jEzLvzCntxyejlLVm3gpOtfYMHytREEGFPFxeosGmNKJJJZpk8PSiOOLOvDA+cfhBmcetNLzFqkRvh28ec/B0ViSYlEOpyy3bvz8IUHM7i4K+feWcFdL30QdUgdnxJJrCmRSIfUp3se088bzxdH9OYnjyzgqkffpnZ7x3+wRCQKSiTSYXXtnMPNp5fzzYMHcfsL73PetLls2loTdVgiHY4SiXRo2VnGT08YyZUnlDFr0Qq+evNLrFi3OeqwRDoUJRLJLLNnt6qvwlkHD+ZPZ5SzZNVGTrr+BRZ/vL7NQxOJKyUSiY0j9urD9PPGU7vdOfWmF3llySdRh9RxPPZYUCSWlEgks1x9dVBaaVS/Hvz1goMo7taZ029/lX+89VEbBhdj+fkNdhaVeFAikczyt78FJQmlvfJ58PyDGLl7d759z2tMe3lpGwUXY410FJV4UCKRWOrVNZe/nDuOLw7vzRUPv8Vvn1ysAR+T0UxHUenYlEgktrrkZnPz6fszubw/182q5EcPvkFN7faowxLJOBq0UWItJzuLX568N326d+baWZV8smErf/z6GLrkavRgkUTpikQyS5cuQWlDZsYPvjScn500ilmLV/KNW19m7afb2vQzRDoyJRLJLI8/HpQUOH3cQG74+hje/HAtU255mdUbtqTkc0Q6GiUSkTqO2Xs3bj1zLO+v3sBXb3qJ5f/5NOqQMkMrO4pKx6BEIpnlZz8LSgodNqyEaeccyKr1Wzj1ppd4f/XGlH6eSKZTIpHM8swzQUmxsYMKuXfqOD7dVsupN73Ewo/WpfwzM1qSHUUlsymRiDRiVL8eTD9vPDlZxuSbX+K1f1dHHVL6aoOOopK5lEhEmrBn7wIeOH88hV1zOe3WV3ihcnXUIYmkHSUSkWb0L8xn+vnj6d8rn7P/PIfn3lkVdUgiaUWJRDJLUVFQ2lnvbnncO3UcQ0oKOPeuCmYvXtnuMYikKyUSySwPPhiUCBR2zeXebx3I0N4FTL1rLrMWrYgkjrSUgo6ikjmUSERaoGd+MNjj8L7dOG/aXJ56W8kESGlHUUl/SiSSWS6/PCgR6pHfibvPPZCy3XtwwT1zeWLBx5HGIxI1JRLJLC+9FJSI9ejSiWnnHMCofj248J7XePzNmE+Q1Q4dRSV9pTSRmNnRZrbYzCrN7LIG1nc2s/vD9a+Y2aA66y4Ply82s6PqLP++mS0ws7fM7F4zy0vlMYg0pnteJ+46+wD27d+Ti+59Pd6zLbZTR1FJTylLJGaWDVwPHAOUAV8zs7J61c4Bqt19T+B3wK/CbcuAKcBI4GjgBjPLNrN+wHeBcncfBWSH9UQi0S2vE3eefQD7lvbgO/e+rgZ4iaVUXpEcAFS6+xJ33wrcB0yqV2cScGf4fgZwhJlZuPw+d9/i7u8DleH+IJhDpYuZ5QD5wPIUHoNIswo65/Dnsw9gRN/unH/3a/zrXXValHhJZSLpByyr83NVuKzBOu5eA6wFihrb1t0/BK4G/g18BKx19ydTEr2kp9LSoKSZHbe5hhR35dy75vDq+2uiDkmk3aQykVgDy+pPit1YnQaXm1kvgquVwcDuQFczO63BDzebamYVZlaxapV6IncYd98dlDTUq2sud597IP16duGbd7zK63EamyuijqKSHlKZSKqA/nV+LmXX21A764S3qnoAa5rYdiLwvruvcvdtwF+Bgxr6cHe/xd3L3b28pKSkDQ5HpHnFBZ35y7fGUdytM2fe/ipvfbg26pDaR4QdRSV6qUwkc4ChZjbYzHIJGsVn1qszEzgzfH8KMMvdPVw+JXyqazAwFHiV4JbWODPLD9tSjgAWpvAYJN1cfHFQ0lif7nncc+6BdMvrxOm3vcI7K9ZHHZJISqUskYRtHhcBTxD8Yz/d3ReY2VVmdmJY7TagyMwqgR8Al4XbLgCmA28D/wAudPdad3+FoFH+NeDNMP5bUnUMkobmzQtKmivtlc9fvnUgnbKzOP22V6iq3hR1SKmVBh1FJToWXAB0bOXl5V5RURF1GNIWDj88eM2QaV0Xf7yeU296keKCzjxw/niKCjpHHVJqZNj3Is0zs7nuXp5IXfVsF0mh4X27cftZY1m+9lPOumMOG7bURB2SSJtTIhFJsfJBhdzwjTG8/dE6pt5VwZaa2qhDEmlTSiSSWYYNC0qG+eKIPvz65H148b1P+P7986jd3vFvKUt85EQdgEiL3JK5z1acvH8p1Zu28vO/L6Rn/lv84qRRBA8fdgBp2ElU2o8SiUg7OnfCEFZv2MpNz75Hn255fG/i0KhDahtp2klU2ocSiWSWqVOD1wy+MvnR0cNZuW4zv3v6HQYUdeHLo/W/eclsSiSSWd55J+oIkmZm/PLkffho7WYunfEGfbt3YfweGT68yI5Oor//fbRxSCTU2C4SgdycLG46bX8GFnXlvGkVVK7M8N7vGdJRVFJDiUQkIj3yO3HHWWPJzcnirDvmsGr9lqhDEmmVhBOJmXVNZSAicdS/MJ/bzhzL6g1bOPfOOXy6VX1MJPM0m0jM7CAze5twcEQz29fMbkh5ZCIN2W+/oHQg+/bvybVTRvPGh2u5+P7X2a4+JpJhErki+R1wFPAJgLvPBw5NZVAijfr97ztkg+6XRvblv48r44kFK/j90xn4QEGGdhSVtpHQU1vuvqxexyldf4u0sbMPHsTij9dx7axKhvftznH77BZ1SInL4MexJXmJXJEsM7ODCGYozDWzH6I5QCQqp50WlA7IzPjZSaPYf2AvLnlgXnwmxZKMl0giOR+4kGAe9SpgP+CCVAYl0qiqqqB0UJ1zsrnptP3plZ/L1LsqMudJrqlTP+ssKrGTSCIZ7u7fcPc+7t7b3U8D9kp1YCJxVdKtM386o5w1m7by7bvnsrVme9QhNe+ddzpEZ1FpnUQSyXUJLhORNjKqXw9+c8q+VCyt5iePvBV1OCJNarSx3czGAwcBJWb2gzqrugPZqQ5MJO5O2Hd3Fn28juv/+R6jB/Rk8tgBUYck0qCmntrKBQrCOt3qLF8HnJLKoEQaNX581BG0qx8cOZz5y9ZyxSMLGLl7D0b16xF1SCK7aHbOdjMb6O5L2ymelNCc7ZLJPtmwheOv+xc52cbfLppAj/xOUYe0Kw3a2OG0ZM72RBJJCXApMBLI27Hc3b+YTJDtSYlEMt1r/65m8s0vcejQEv50RjlZWR1kQixJWy1JJIk0tt8DLAIGA/8DfADMaXV0Isk4+eSgxMyYAb244vgynlm0khuffS/qcEQ+J5FEUuTutwHb3P1Zdz8bGJfiuEQa9sknQYmh08cNZNJ+u/PbJxfzr3dXRx3O53XgjqLSvEQSybbw9SMzO87MRgOa0k2knZkZ//eVvdmjpICL75/H6g1p1Fmxg3cUlaYlkkh+bmY9gEuAHwK3At9PaVQi0qD83Byu+/po1m3exiXT52ukYEkLTSYSM8sGhrr7Wnd/y92/4O77u/vMdopPROoZ0bc7Vxy3F8++s4rbX3g/6nBEmk4k7l4LnNhOsYg074gjghJzp40byJFlffjVPxZpcEeJXCK3tl40sz+a2QQzG7OjJLJzMzvazBabWaWZXdbA+s5mdn+4/hUzG1Rn3eXh8sVmdlSd5T3NbIaZLTKzhWEPfImLK64ISsyZGb8+eR+KunbmO/e+zsYtNdEGNH587DqLymcS6UfyzwYWe3P9SMLbYu8ARxKMGjwH+Jq7v12nzgXAPu5+vplNAb7s7pPNrAy4FzgA2B14Ghjm7rVmdifwvLvfama5QL67/6epWNSPRDqql5d8wtf+9DInjynl6lP3jToc6UBa0o+k2Ymt3P0LrYzjAKDS3ZeEQd0HTALerlNnEnBl+H4G8EcLZtCaBNzn7luA982sEjjAzBYQzM54VhjbVmBrK+OTTHTMMcHr449HG0eaGDekiIu+sCfXzapk4l59OHpU36hDkhhK5NZWa/UDltX5uSpc1mAdd68B1gJFTWw7BFgF3GFmr5vZrWbWNTXhS1r69NOgyE7fPWIoo/p158cPvRndI8Ex7SgqgVQmkobGcKh/H62xOo0tzwHGADe6+2hgI7BL2wuAmU01swozq1i1alXiUYtkmE7ZWVzz1f1Yv7mGHz/0Js3drk6JGHcUldQmkiqgf52fS4HljdUxsxygB7CmiW2rgCp3fyVcPoMgsezC3W9x93J3Ly8pKUnyUETS27A+3bjkS8N4YsEKHp73YdThSMw0m0jM7CsNlCPMrHczm84BhprZ4LBRfApQv//JTODM8P0pwCwP/js1E5gSPtU1GBgKvOruHxPMIT883OYIPt/mIhJb504YQvnAXvzkkQV8tFa3/6T9NNvYDpwDjAd2PL11OPAyMMzMrnL3aQ1t5O41ZnYR8ATBRFi3u/sCM7sKqAg7Nd4GTAsb09cQJBvCetMJkkQNcGHYpwXgO8A9YXJaAnyzpQed6TZu3Mjs2bN59913Wb9+PXl5efTu3ZtDDjmEPfbYA4Dq6mqef/553nvvPTZs2EB+fj5FRUXst99+7L333mRnB3OTXXnllTv326lTJwoKCigtLWXs2LEMGJCGEykdf3zUETQq6u8lO8u4+tR9OeYPz3PpjDe46+wDCJ5dia+ov5O4SCSRbAf2cvcVAGbWB7gROBB4DmgwkQC4+2PAY/WW/aTO+83AqY1s+wvgFw0snwck9EhaRzV9+nS2bdvGiSeeSGFhIRs3bmTp0qV8GjZCL1++nDvvvJOSkhKOPfZYiouL2bZtG6tWreK1116jsLDwc7/4J554IsOGDaOmpobq6mrmz5/PHXfcwcSJEzn44IOjOsyG/fCHUUfQqHT4XgYVd+XyY0fwk0cW8NfXPuTk/dtpWLw07SSaDt9JHCSSSAbtSCKhlQR9OtaY2bbGNpLU2Lx5M0uXLuWMM85gyJAhAPTs2ZN+/YIH4tydhx56iKKiIs4555zP/Y+0b9++7L333rs0xubl5VFQULBzX4MHD6Zbt24888wz7LXXXhQWFrbT0WWudPpeTjtwIDPnLefnf3+bw4eXUFTQORWH/Hlp2Ek0nb6Tji6RxvbnzexvZnammZ0JPAI8Fz5222RHQGl7ubm55ObmsnjxYmpqdu3N/PHHH7Nq1SoOOuigRm9rJHK7Y/z48bg7ixYtSjrmNnX44UFJM+n0vWRlBaMEb9hSw8/+Ft8mxHT6Tjq6RK5ILgROBg4meCz3LuDBsFG8tZ0VpZWysrI46aSTePTRR5k7dy59+/ZlwIABlJWVUVpayifhI5hFRUU7t9m8eTPXXHPNzp8nTJjAhAkTmvyc/Px8unbtSnV1dWoOpINJt+9laJ9ufPvwPbn2mXf58phSDhuW4icX07CjaLp9Jx1ZIj3bneAx2xmpD0cSUVZWxrBhw1i6dClVVVVUVlby4osvcsQRR9CrV69d6nfu3Jnzzz8fgHvuuYfa2tpd6jQkkv4IGSzdvpcLv7AHf39jOT9+6E2e/P6h5Ocm8v/GVkrTTqLp9p10VIk+/vuuma01s3Vmtt7M1rVHcNK4nJwc9thjDw477DDOOeccxowZw+zZs+nZsycAq1d/NoOemVFYWEhhYeHOJ1Cas2nTJjZt2tTgH5s0Lp2+l8452fzy5H2oqv6Ua558p3UH1AGk03fSUSXSRvJr4ER37+Hu3d29m7t3T3Vg0jIlJSVs376d4uJiSkpKeOGFF9i+fXur9/fiiy9iZowYMaINo4yfqL+XsYMK+doBA7jjxQ94Z8X6Vn9uRxL1d9IRJXKtu8LdF6Y8EknIpk2beOCBBxg9ejR9+vQhNzeX5cuX88ILLzB48GDy8vI46aSTuOuuu7jtttuYMGHCzj+cZcuWsW7dul0aEDdv3syGDRuora1lzZo1zJ8/n/nz53PkkUem31MoX/1q1BE0KJ2/l0uPGs5jb37ETx9ZwF++dWBs+pak83fS0SQyjPwfgL7Aw8DOEeHc/a+pDa3tdKRh5Gtqanj22Wd57733WLNmDbW1tXTr1o3hw4dz6KGH0qVLFwDWrFnD888/z5IlS9iwYQM5OTn06dOHUaNGMWbMGHJygv9D1O1klZOT87lOVgMHDoziEDNSun8v015eyhUPv8X1Xx/Dcfvs1ibH/DlXXx28plE/n3T/TtJdS4aRTySR3NHAYnf3s1sTXBQ6UiKJvU2bgtf8/GjjyDC1250TrvsX1Zu28swlh6W24V06hLaejyR2Q5BIGjv22OB19uxIw8g02VnGVZNGcspNL3HDP9/jh0cNb34jkQQ1mkjM7FJ3/7WZXceuw7/j7t9NaWQi0qbKBxXy5dH9uOW5JZxaXsrAojacymdHJ1El+Fhq6qmtHQ3sFcDcBoqIZJjLjhlBdpbx6ycWRx2KdCCNXpG4+6Ph653tF46IpFKf7nl869AhXPvMu5x7SDWjB8S374O0nUQ6JA4zs1vM7Ekzm7WjtEdwItL2ph46hOKCXP7v8UWx75EtbSORRzceAG4CbgUSGy9AJFXOOivqCDJeQeccvjdxGFc8/BbPLFzJxLI+UYckGS6RRFLj7jemPBKRRCiRtIkpY/tzx7/e55f/WMThw0vIyU5y1u007Sgq7SOR355HzewCM9vNzAp3lJRHJtKQ1auDIknplJ3FpUePoHLlBh58rSr5HV5wQVAklhK5Itkxp/r/q7PMgSFtH45IM045JXjVY6ZJO2pkH/Yt7cF1syr58uhScnOSuCpRR9FYa/I3x8yygNPcfXC9oiQikuHMjIsnDqOq+tPkr0qOPfazzqISO00mEnffDlzdTrGISDs7fHgJ+/XvyR9nVbK1pvUj4Eq8JXIt+6SZnWxxGTJUJEaCq5KhfPifT3lg7rKow5EMlUgi+QHBI8BbNLGVSMdz2LASRg/oyfWzKtlSoyf8peWaTSThRFZZ7p6ria0kct/+dlCkzZgZ3584jOVrN/Pw6x9GHY5koITGkjazXsBQIG/HMnd/LlVBiTRq8uSoI+iQJgwtZuTu3bn5uSWcun9/srJaeCdb/XtiLZEhUs4FngOeAP4nfL0ytWGJNGLZsqBImzIzzjtsD5as2sjTC1e0fAdnnaVkEmOJtJF8DxgLLHX3LwCjgVUpjUqkMaefHhRpc8eO6kv/wi7c/NySlm+sjqKxlkgi2ezumwHMrLO7LwI0K45IB5OTncW3Jgxh7tJqKj5Y07KNTznls86iEjuJJJIqM+tJMGf7U2b2CLA8tWGJSBRO3b8/vfI7cdOzrbgqkdhK5KmtL7v7f9z9SuAK4DbgpER2bmZHm9liM6s0s8saWN/ZzO4P179iZoPqrLs8XL7YzI6qt122mb1uZn9LJA4RSUyX3GxOHzeQZxatYNmaTVGHIxkiocF1zOwQM/umuz8LvAT0S2CbbOB64BigDPiamZXVq3YOUO3uewK/A34VblsGTAFGAkcDN4T72+F7fDaDo4i0oa8fOJAsM+5+eWnUoUiGSOSprZ8CPwIuDxd1Au5OYN8HAJXuvsTdtwL3AZPq1ZkE7JiBcQZwRNiDfhJwn7tvcff3gcpwf5hZKXAcwfwoEjeXXBIUSZm+PfL4Ulkf7q9YxuZt6qAozUvkiuTLwInARgB3Xw50S2C7fkDd5zSr2PVKZmcdd68B1gJFzWz7e+BSoMmBgcxsqplVmFnFqlV6yKzDOOGEoEhKnT5+IP/ZtI1H5yfYHKqOorGWSCLZ6sF8nA5gZl0T3HdDPZrqz+vZWJ0Gl5vZ8cBKd5/b3Ie7+y3uXu7u5SUlJc1HK5lh8eKgSEqNH1LE0N4FTEv09tbkyeosGmOJJJLpZnYz0NPMvgU8Dfwpge2qgP51fi5l16e9dtYxsxygB7CmiW0PBk40sw8IbpV90cwSuc0mHcV55wVFUsrMOH38QN6oWsubVWub30AdRWMtkae2riZov3iQoP/IT9z9ugT2PQcYamaDzSyXoPF8Zr06M/ls4qxTgFnh1c9MYEr4VNdgguFZXnX3y9291N0Hhfub5e6nJRCLiLTQpP36kZuTldiowOooGmsJjbXl7k8BT7Vkx+5eY2YXEQypkg3c7u4LzOwqoMLdZxI8SjzNzCoJrkSmhNsuMLPpwNtADXChu6vVT6Qd9ejSiaNG9uWRecv58XF70Tknu/mNJJYaTSRmtp5d2zQgaL/wREYAdvfHgMfqLftJnfebgVMb2fYXwC+a2PdsYHZzMYhI6526fymPzl/OMwtXcuzeu0UdjqSpRm9t7RguvoGiYeRFYuLgPYvp2z2PByrU/iGNS+jWlkja+O//jjqCWMnOMr4yph83PfseK9dtpnf3vOY3kthJqGe7SNqYODEo0m6+MqaU7Q5/f/Ojxiupo2isKZFIZpk3LyjSbvbsXcCIvt14rKlEoo6isaZEIpnl4ouDIu3quL13Y84H1Xy8dnPDFdRRNNaUSESkWcfuEzyx1ehViTqKxpoSiYg0a4+S4PZWk+0kEltKJCKSkOP23o25S6tZsa6R21sSW0okIpKQiWV9AJi9eGXEkUi6UT8SySz/+79RRxBbI/p2Y7ceeTyzcCWTxw6IOhxJI0okklkOOijqCGLLzPjiiN489PqHbKmp/fzYW+ooGmu6tSWZ5cUXgyKR+OKI3mzaWsur76/5/Ap1FI01JRLJLP/1X0GRSBy0RzGdc7KYtaheO4k6isaabm2JSMK65GYzdlAhL733yedX7OgkOnt2u8ck0dMViYi0yLghhSz6eD1rNm6NOhRJE0okItIi4/coAuCVJZ80U1PiQolERFpkn9Ke5Odm85ISiYTURiKZ5fe/jzqC2OuUnUX5oEJeViKRkBKJZJb99os6AgEOHFzIb55YTPXGrfTqmquOojGnW1uSWZ5+OigSqdEDegLwxodrgwUHHaTOojGmRCKZ5ec/D4pEau9+PTCD+cv+EyxQR9FY060tEWmxbnmd2KOk4LNEsqOTqPqRxJKuSESkVfYt7cn8qv/g7lGHIhFTIhGRVtm3fw9Wb9jKh//5NOpQJGJKJCLSKiN37w7Aoo/WRxyJRE1tJJJZbr456ggktGfvbgC8s3I9Gvc33pRIJLMMHx51BBLq0aUTfbvn8e6KDeooGnMpvbVlZkeb2WIzqzSzyxpY39nM7g/Xv2Jmg+qsuzxcvtjMjgqX9Tezf5rZQjNbYGbfS2X8koYefTQokhaG9ingnRXrg46i6iwaWylLJGaWDVwPHAOUAV8zs7J61c4Bqt19T+B3wK/CbcuAKcBI4GjghnB/NcAl7r4XMA64sIF9Skf2298GRdLCsD7dqFy5gdonn1JH0RhL5RXJAUCluy9x963AfcCkenUmAXeG72cAR5iZhcvvc/ct7v4+UAkc4O4fufu8dUY1AAAKrklEQVRrAO6+HlgI9EvhMYhIE4b2LmBLzXa2XfUzdRSNsVQmkn7Asjo/V7HrP/o767h7DbAWKEpk2/A22GjglTaMWURaYEBhPgBbttVGHIlEKZWJxBpYVr/nUmN1mtzWzAqAB4GL3X1dgx9uNtXMKsysYtWqVQmGLCIt0X9HIqnZHnEkEqVUJpIqoH+dn0uB5Y3VMbMcoAewpqltzawTQRK5x93/2tiHu/st7l7u7uUlJSVJHoqINGS3HnlkZ5muSGIulYlkDjDUzAabWS5B4/nMenVmAmeG708BZnkw3sJMYEr4VNdgYCjwath+chuw0N2vSWHskq6mTQuKpIWc7Cx265HHZl2RxFrK+pG4e42ZXQQ8AWQDt7v7AjO7Cqhw95kESWGamVUSXIlMCbddYGbTgbcJntS60N1rzewQ4HTgTTObF37Uf7n7Y6k6Dkkz/fs3X0faVWmvLlz31Uu49mtjog5FIpLSDonhP/CP1Vv2kzrvNwOnNrLtL4Bf1Fv2LxpuP5G4uP/+4HXy5GjjkJ1KuuXxZsFu6iwaYxprSzLLjTcGRdJGUddcRs59Vh1FY0xDpIhIUooLcjn6hRlsXz6LrBNOiDociYCuSEQkKYVdOwOwrVbzksSVEomIJKWoIBeAmlo9uRVXSiQikpSirkEi2bZdVyRxpTYSySwzZkQdgdTTLa8T1UCtEklsKZFIZikujjoCqacgL4ezjr+Ey48ZwYlRByOR0K0tySx//nNQJG3k5WTxUfcSqov6Rh2KRESJRDKLEkna6ZSTxfELn6P/k/VHQJK4UCIRkaTkZmdx2uuPMfzhe6IORSKiRCIiScnJCkYtcrW1x5YSiYgkJXtHItlluiGJCz21JSJJMTOyzPiw+lMOu/zvUYcjdRQXdObVH09M+ecokUhmeUwzBqSjISUFfLqthgu/sGfUoUgd+bnt80+8Eolklvz8qCOQBhT/I3hi6xL184klJRLJLDfcELxecEG0ccjnKYHEmhrbJbNMnx4USS/q3xNrSiQikjwlklhTIhERkaQokYiISFKUSEREJCl6aksyy+zZUUcgIvUokYhI8tRRNNaUSEQkeeooGmtqIxGR5N1ww2edRSV2lEhEJHnqKBprSiQiIpKUlCYSMzvazBabWaWZXdbA+s5mdn+4/hUzG1Rn3eXh8sVmdlSi+xQRkfaVskRiZtnA9cAxQBnwNTMrq1ftHKDa3fcEfgf8Kty2DJgCjASOBm4ws+wE9ykiIu0olVckBwCV7r7E3bcC9wGT6tWZBNwZvp8BHGFmFi6/z923uPv7QGW4v0T2KSIi7SiVj//2A5bV+bkKOLCxOu5eY2ZrgaJw+cv1tu0Xvm9unyLS3tRRNNZSeUViDSyrP6lzY3VaunzXDzebamYVZlaxatWqJgMVEZHWS2UiqQL61/m5FFjeWB0zywF6AGua2DaRfQLg7re4e7m7l5eUlCRxGCIi0pRUJpI5wFAzG2xmuQSN5zPr1ZkJnBm+PwWY5e4eLp8SPtU1GBgKvJrgPkVEpB2lrI0kbPO4CHgCyAZud/cFZnYVUOHuM4HbgGlmVklwJTIl3HaBmU0H3gZqgAvdvRagoX2m6hhERKR5FlwAdGzl5eVeUVERdRgiIhnDzOa6e3kiddWzXUREkqJEIiIiSVEiERGRpCiRiIhIUmLR2G5mq4Clrdy8GFjdhuFkorifg7gfP+gcQPzOwUB3T6gTXiwSSTLMrCLRJxc6qrifg7gfP+gcgM5BU3RrS0REkqJEIiIiSVEiad4tUQeQBuJ+DuJ+/KBzADoHjVIbiYiIJEVXJCIikhQlkkbEaW54M/vAzN40s3lmVhEuKzSzp8zs3fC1V7jczOza8Ly8YWZjoo2+dczsdjNbaWZv1VnW4mM2szPD+u+a2ZkNfVa6auQcXGlmH4a/C/PM7Ng66y4Pz8FiMzuqzvKM/Fsxs/5m9k8zW2hmC8zse+HyWP0etAl3V6lXCEYWfg8YAuQC84GyqONK4fF+ABTXW/Zr4LLw/WXAr8L3xwKPE0wyNg54Jer4W3nMhwJjgLdae8xAIbAkfO0Vvu8V9bEleQ6uBH7YQN2y8O+gMzA4/PvIzuS/FWA3YEz4vhvwTnicsfo9aIuiK5KGaW744HjvDN/fCZxUZ/ldHngZ6Glmu0URYDLc/TmCqQvqaukxHwU85e5r3L0aeAo4OvXRt41GzkFjJgH3ufsWd38fqCT4O8nYvxV3/8jdXwvfrwcWEkzpHavfg7agRNKwhuab79dI3Y7AgSfNbK6ZTQ2X9XH3jyD4gwN6h8s78rlp6TF31HNxUXjr5vYdt3Xo4OfAzAYBo4FX0O9BiymRNCzhueE7iIPdfQxwDHChmR3aRN24nRto/Jg74rm4EdgD2A/4CPhtuLzDngMzKwAeBC5293VNVW1gWYc4B8lSImlYwnPDdwTuvjx8XQk8RHC7YsWOW1bh68qwekc+Ny095g53Ltx9hbvXuvt24E8EvwvQQc+BmXUiSCL3uPtfw8Wx/z1oKSWShsVmbngz62pm3Xa8B74EvEVwvDuePjkTeCR8PxM4I3yCZRywdsdtgA6gpcf8BPAlM+sV3gL6UrgsY9Vr7/oywe8CBOdgipl1NrPBwFDgVTL4b8XMjGC674Xufk2dVbH/PWixqFv707UQPKHxDsETKT+OOp4UHucQgidt5gMLdhwrUAQ8A7wbvhaGyw24PjwvbwLlUR9DK4/7XoJbN9sI/kd5TmuOGTiboOG5Evhm1MfVBudgWniMbxD8w7lbnfo/Ds/BYuCYOssz8m8FOITgFtQbwLywHBu334O2KOrZLiIiSdGtLRERSYoSiYiIJEWJREREkqJEIiIiSVEiERGRpCiRiDTDzF4MXweZ2dfbeN//1dBniWQSPf4rkiAzO5xgZNzjW7BNtrvXNrF+g7sXtEV8IlHRFYlIM8xsQ/j2l8CEcJ6O75tZtpn9xszmhIMcnhfWPzyc5+IvBB3XMLOHw0ExF+wYGNPMfgl0Cfd3T93PCntP/8bM3rJgrpjJdfY928xmmNkiM7sn7KGNmf3SzN4OY7m6Pc+RxFtO1AGIZJDLqHNFEiaEte4+1sw6Ay+Y2ZNh3QOAUR4MuQ5wtruvMbMuwBwze9DdLzOzi9x9vwY+6ysEAyfuCxSH2zwXrhsNjCQYz+kF4GAze5tgSJMR7u5m1rPNj16kEboiEWm9LxGMvTSPYPjxIoIxqABerZNEAL5rZvOBlwkG+BtK0w4B7vVgAMUVwLPA2Dr7rvJgYMV5wCBgHbAZuNXMvgJsSvroRBKkRCLSegZ8x933C8tgd99xRbJxZ6WgbWUiMN7d9wVeB/IS2HdjttR5XwvkuHsNwVXQgwQTMf2jRUcikgQlEpHErSeYknWHJ4Bvh0ORY2bDwhGU6+sBVLv7JjMbQTBN6w7bdmxfz3PA5LAdpoRgWtxXGwssnFOjh7s/BlxMcFtMpF2ojUQkcW8ANeEtqj8DfyC4rfRa2OC9is+mZa3rH8D5ZvYGwci5L9dZdwvwhpm95u7fqLP8IWA8wajMDlzq7h+Hiagh3YBHzCyP4Grm+607RJGW0+O/IiKSFN3aEhGRpCiRiIhIUpRIREQkKUokIiKSFCUSERFJihKJiIgkRYlERESSokQiIiJJ+f8Tm1vrsIljmAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_moms=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't specify an end value to the lr, it will be assumed the value you gave is the starting value and the end value is 0. This doesn't work for the EXPONENTIAL type of decay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-2), \n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-2, lr_decay=DecayType.COSINE),\n",
    "          TrainingPhase(epochs=1, opt_fn=optim.SGD, lr = 1e-3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1983a61e45284dc3ab8fc122d990c9ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.399492   4.549183   0.1378    \n",
      "    1      3.388229   4.540058   0.1409                                                                                \n",
      "    2      3.397341   4.548875   0.141                                                                                 \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.54887]), 0.141]"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VdW5//HPQ0IIkwxJBCVgwqiAFTCi4FAUrqKt4vSrSK3Y0lqr1trW9uq97b3W6+2odrhVW26pU1Wg2Fb0OldRKzIEARU0GpEhooCGIoMMIc/vj72jMWQ4yTkn+wzf9+u1Xudkn7X3efY+OXmy99prLXN3RERE2qpD1AGIiEh6UyIREZG4KJGIiEhclEhERCQuSiQiIhIXJRIREYmLEomIiMRFiUREROKiRCIiInHJjTqA9lBYWOglJSVRhyEikjaWLVv2vrsXxVI3KxJJSUkJ5eXlUYchIpI2zGxdrHV1aUtEROKiRCIiInFRIhERkbgokYiISFyUSEREJC5KJCIiEhclEhERiYsSiYiIxEWJRERE4qJEIiLxe+qpoEhWyoohUkQkyW68MXicNCnaOCQSOiMREZG4KJGIiEhclEhERCQuSiQiIhIXNbaLSPx+//uoI5AIKZGISPyGDYs6AomQLm1JepkwISiSWh56KCiSlXRGIiLxu/nm4PHMM6ONQyKhMxIREYmLEomIiMRFiUREROKS1ERiZpPNrMLMKs3s2kZe72Rmc8LXF5tZSbi8wMyeMbMdZvbbBuscbWavhOv8xswsmfsgKWbevKCISMpIWmO7meUAtwL/AlQBS81svruvrldtBrDV3Qeb2VTgZ8AFwG7gh8DIsNR3O3ApsAh4BJgMPJqs/ZAUU1gYdQTSmHvuiToCiVAy79oaC1S6+xoAM5sNTAHqJ5IpwPXh83nAb83M3H0n8A8zG1x/g2Z2CHCQu78Y/nw3cDZJSiTzV26kttY/tayl85+GJ0gNqzdc3w6o0Vid5l9vWKOl9YM6rYyzpbiT8Z6N7NfBD9wPwObzLqRvj3xKCrqS00EnpZHr3z/qCCRCyUwk/YAN9X6uAo5tqo6715jZNqAAeL+ZbVY12Ga/xiqa2aUEZy4MGDCgtbED8K/zXuajffvbtK4kx+z7ZgLwxa0DAeial8NJQ4v43GcO4bQRfemYo2a/SMyZEzxecEG0cUgkkplIGvs30dtQp0313X0mMBOgrKysuW026dFvnfipjbt/ejMNN+oHvIs3+3pjQR1Yp4VttLJ+Y1r9nges3/xxaTyO1r5H8Hj4gh4AzL70ONZX72LFhn/y1OpNPPrqexzaI5/LJgxi2tgB5CqhtK/bbw8elUiyUjITSRVQ/3y3GNjYRJ0qM8sFegDVLWyzuIVtJkxJYddkbVraKj/4lT1uYAHHDSzgC2X9uXHKSJ6p2Mzvnn2L/3hwFfctXs+Pzz2SMQN6RRysSHZI5r9tS4EhZlZqZnnAVGB+gzrzgenh8/OBp73hv7f1uPu7wHYzOy68W+ti4MHEhy7ppEMHY+IRfZj79XH87qKj2b67hv/3uxe59ZnKA9q4RCTxknZGErZ5XAk8DuQAf3T3VWZ2A1Du7vOBWcA9ZlZJcCYytW59M1sLHATkmdnZwKnhHV/fAO4EOhM0suuOLQGCxvzJI/syfnAB1/3lFX7xeAXL1/+T31w4ii55Gg1IJFmsmROAjFFWVubl5eVRhyGJsGtX8NilS7PV3J27Fq7lRw+v5jPFPZk1vYzCbp3aIcAsVTeQ5oIFUUYhCWRmy9y9LJa6apGU9NKlS4tJBIKzk0uOL+X3Fx1NxXsf8sX/XUz1zr3tEGCWUkfRrKZEIunlttuCEqNTR/Rl1vRjWPvBTi76w2L+uUvJJCkKC9VZNIspkUh6mTs3KK1w/OBCZl5cRuXmHXzt7nL21KhvUMLdeWdQJCspkUhW+OzQIm654CiWrt3KtQ+8ckDfF4mTEklW060skjU+/5lDWfv+Tm564g0GFnblmxOHRB2SSEbQGYlklStOHsw5o/txy1Nv8I83mxqJR0RaQ4lEsoqZ8d/njGRwUTeunrOczR/ujjokkbSnRCLpZcGCuPsqdMnL5bYvjmHnnv1cNXs5+9X7XSQuSiSSlYb06c4NU0awaE01s/6xJupw0t8jjwRFspISiaSXm24KSgKcf3Qxpw7vw01PvEHl5u0J2WbWirGjqGQmJRJJLw8/HJQECNpLjqRrXg7fnbuSmv21CdluVmplR1HJLEokktWKunfihikjWVm1jZnP6xJXm7Who6hkDiUSyXpnHnUop43ow2/+/iYbqndFHY5I2lEiEQH+88wRdDDjRw+tjjoUkbSjRCLppXPnoCTYoT07c9XEITz12iaeXL0p4dsXyWRKJJJeHn00KEnwleNLGXJwN66fv4qP9mpgR5FYKZGIhPJyO3DDlJG888+P1LektRLQUVTSlxKJpJf/+q+gJMm4QQWcOrwPty94iy3b9yTtfUQyiRKJpJe//z0oSXTt6Yezp6aWXz31RlLfJ6MksKOopB8lEpEGBhZ146LjDuP+Jet5c5N6vMckgR1FJf0okYg04qqJQ+ial8uPH3kt6lBEUp4SiUgjenfN4/KTB/NMxRaWrq2OOhyRlKZEIumloCAo7eCS8SUUde/EzU9UtMv7iaSrpCYSM5tsZhVmVmlm1zbyeiczmxO+vtjMSuq9dl24vMLMTqu3/NtmtsrMXjWz+80sP5n7ICnmgQeC0g465+Vw+YRBLFpTzcJKzabYrCR1FJX0kLREYmY5wK3A6cBw4EIzG96g2gxgq7sPBn4J/CxcdzgwFRgBTAZuM7McM+sHXAWUuftIICesJ5IUF44dwCE98rn5yTdw1wRYTUpiR1FJfck8IxkLVLr7GnffC8wGpjSoMwW4K3w+D5hoZhYun+3ue9z9baAy3B5ALtDZzHKBLsDGJO6DpJrrrgtKO8nvmMMVJw9m2bqtPPvGlnZ7X5F0ksxE0g/YUO/nqnBZo3XcvQbYBhQ0ta67vwPcBKwH3gW2ufsTSYleUtOLLwalHX2hrD/FvTpzi85KmpbkjqKS2pKZSKyRZQ2/hU3VaXS5mfUiOFspBQ4FuprZRY2+udmlZlZuZuVbtug/SWm7vNwOXHXKEF6u2saCCv0uNaodOopK6kpmIqkC+tf7uZgDL0N9XCe8VNUDqG5m3UnA2+6+xd33AX8Bxjf25u4+093L3L2sqKgoAbsj2eycMf3o17Mzty2ojDoUkZSTzESyFBhiZqVmlkfQKD6/QZ35wPTw+fnA0x5cO5gPTA3v6ioFhgBLCC5pHWdmXcK2lImAeoxJ0nXM6cClJw1k6dqtLHlb/UpE6ktaIgnbPK4EHif4Yz/X3VeZ2Q1mdlZYbRZQYGaVwHeAa8N1VwFzgdXAY8AV7r7f3RcTNMq/BLwSxj8zWfsgKai4OCgRuOCY/hR2y+PWZ3RWIlKfZUPjYVlZmZeXl0cdhmSAW5+p5BePV/DwN09gZL8eUYeTOs47L3hspz4+knxmtszdy2Kpq57tIq3wpXGH0b1TLrcveCvqUFJLO3YUldSjRCLp5eqrgxKRg/I7cvH4w3jk1XdZs2VHZHGIpBIlEkkvK1YEJUJfPr6UTrkdmPmcZlH8WDt3FJXUokQi0kqF3Tpx3phi/rL8HT7YoVkUgUg6ikrqUCIRaYOvnFDK3ppa7lm0LupQRCKnRCLSBoOKunHK4Qdzz4vr2L1vf9ThiERKiUTSy9ChQUkBXz2hlA927uXBFe9EHYpIpHKjDkCkVWamTv/TcYMKOOKQg/jD82/zhbL+BIMtZKmIOolKatAZiUgbmRlfPaGUNzfv4Lk3s3ziqz/9KSiSlZRIJL1cemlQUsSZRx3Kwd078YfndSuwZC8lEkkvb7wRlBSRl9uB6eNLeP7N96l4b3vU4UQn4o6iEi0lEpE4TRs7gE65HbjrxbVRhxKdFOgoKtFRIhGJU6+ueZx11KH89aV32PbRvqjDEWl3MScSM+uazEBE0tn08SV8tG8/DyyrijoUkXbXYiIxs/FmtppwAikzO8rMbkt6ZCKNGTUqKClmZL8ejBnQk3sWraO2NvOnZhCpL5Yzkl8CpwEfALj7SuCkZAYl0qRf/SooKWj6+BLefn8nz1dm4a3AKdRRVNpfTJe23H1Dg0UaE0KkgdNHHkJht07cvXBt1KG0v5kzU6qzqLSvWBLJBjMbD7iZ5ZnZNWiedInKRRcFJQXl5XZg2tj+PF2xmfUf7Io6HJF2E0siuQy4AugHVAGjgMuTGZRIk6qqgpKiph17GB3M+NPiLBsVOMU6ikr7iiWRDHP3L7p7H3c/2N0vAo5IdmAi6ahvj3wmj+jLnKUb+GhvFl0BTrGOotK+Ykkk/xPjMhEhmNd920f7eGjlxqhDEWkXTY7+a2bjgPFAkZl9p95LBwE5yQ5MJF0dW9qbYX26c/eitXzhmP5RhyOSdM2dkeQB3QiSTfd65UPg/OSHJtKIceOCksLMjC8eN4BX3/mQl6v+GXU4IknX5BmJuz8LPGtmd7p7lrUcSsr6yU+ijiAmZ4/ux08eeZ37Fq/nM8U9ow4n+VKwk6i0n1jaSHaZ2S/M7BEze7quxLJxM5tsZhVmVmlm1zbyeiczmxO+vtjMSuq9dl24vMLMTqu3vKeZzTOz183stfASnEhKOSi/I2cddSjzV27kw91ZMP5WCncUleSLJZHcC7wOlAI/AtYCS1taycxygFuB04HhwIVmNrxBtRnAVncfTNCD/mfhusOBqcAIYDJwW7g9gF8Dj7n74cBRqE9LdjnvvKCkgWnHDmDX3v08uFxT8UpmiyWRFLj7LGCfuz/r7l8BjothvbFApbuvcfe9wGxgSoM6U4C7wufzgIkWzFc6BZjt7nvc/W2gEhhrZgcRDM8yC8Dd97q7LkJnkw8+CEoa+ExxD0YcehD3Ll6Pe4aPv5XCHUUl+WJJJHXn5e+a2efMbDQQywTN/YD6Q6tUhcsarePuNcA2oKCZdQcCW4A7zGy5mf1BoxJLqjIzph07gNff287yDRn+/06KdxSV5IolkdxoZj2A7wLXAH8Avh3DetbIsob/ljVVp6nlucAY4HZ3Hw3sBA5oewEws0vNrNzMyrds2RJDuCKJN2VUP7rm5XDvovVRhyKSNM0mkrBdYoi7b3P3V939ZHc/2t3nx7DtKqD+TfTFQMMeWh/XMbNcoAdQ3cy6VUCVuy8Ol88jSCwHcPeZ7l7m7mVFRUUxhCuSeN065TJldD8efnkj23ZlQaO7ZKVmE4m77wfOauO2lwJDzKzUzPIIGs8bJqD5wPTw+fnA0x5cTJ4PTA3v6ioFhgBL3P09gkEkh4XrTARWtzE+SUcTJwYljUwbO4A9NbX8Zbku/UhmarIfST0Lzey3wByCS0kAuPtLza3k7jVmdiXwOEFP+D+6+yozuwEoD89qZgH3mFklwZnI1HDdVWY2lyBJ1ABXhEkN4JvAvWFyWgN8OfbdlbT3wx9GHUGrjezXg6OKe3Df4vVcMr6E4H6SDJPinUQluaylu0nM7JlGFru7n5KckBKvrKzMy8vLow5Dsticpev51wdeYe7XxzG2tHfU4Yi0yMyWuXtZLHVbPCNx95PjD0kkQU4/PXh89NFo42ilM486lBsffo37Fq9TIpGME9MMiSIp46OPgpJmuuTlcs6Yfjzy6ntU79wbdTiJl0YdRSXxlEhE2sm0Ywewt6aWB5ZlYKN7GnUUlcRTIhFpJ4f3PYgxA3py/5Is6OkuWaXFNhIzO7eRxduAV9x9c+JDEslc0449jGv+vJJFa6oZN6gg6nBEEiKW239nAOOAuru3JgCLgKFmdoO735Ok2KQJO3fuZMGCBbz55pts376d/Px8Dj74YE444QQGDRoEwNatW3n++ed566232LFjB126dKGgoIBRo0Zx5JFHkpMTjIF5/fXXf7zdjh070q1bN4qLiznmmGMYMGBAFLvXvM9/PuoImhTL5zK+OB9ft5TvX/80pw7tmTmfS4rK6u9KO4olkdQCR7j7JgAz6wPcDhwLPAcokbSzuXPnsm/fPs466yx69+7Nzp07WbduHR+FjdAbN27krrvuoqioiDPOOIPCwkL27dvHli1beOmll+jdu/enfvHPOusshg4dSk1NDVu3bmXlypXccccdTJo0ieOPPz6q3WzcNddEHUGTYv1cjiowXq45nAunf55ueZYZn0uKdhLN6u9KO4olkZTUJZHQZmCou1ebmcZ8aGe7d+9m3bp1XHzxxQwcOBCAnj170q9fMB6mu/PXv/6VgoICZsyY8anOb3379uXII4884Pp8fn4+3bp1+3hbpaWldO/enb///e8cccQR9O6t21Vb0prP5eeXTGXyr5/n6bW7uPSkQZnxuaRgR1F9V9pPLI3tz5vZw2Y23cymAw8Cz4Wj7mb4kKapJy8vj7y8PCoqKqipqTng9ffee48tW7Ywfvz4JntQx9Kzety4cbg7r7/+etwxJ9SECUFJMa35XA4/5CCOPqwX9y/Z8Kk/VGn9uaSgrP+utKNYzkiuAM4DjicYlfdu4IFwTCx1VmxnHTp04Oyzz+ahhx5i2bJl9O3blwEDBjB8+HCKi4v5ILwFs6Dgk4bc3bt3c8stt3z884knnsiJJ57Y7Pt06dKFrl27snXr1uTsSIZp7ecybewAvnP/Ui675gccVhDMhJDWn0sKdhTVd6X9xNKz3QlG2Z2X/HAkFsOHD2fo0KGsW7eOqqoqKisrWbhwIRMnTqRXr14H1O/UqROXXXYZAPfeey/79+8/oE5jdItq67Tmc/ncZw7h+vmd2d9vEpedf1T6fy4p2klU35X20eKlLTM718zeNLNtZvahmW03sw/bIzhpWm5uLoMGDeKzn/0sM2bMYMyYMSxYsICePXsC8P77739c18zo3bs3vXv3/vgOlJbs2rWLXbt2Nfplk6bF+rnkd8zhvKP78+y6j/C8rvpckkjfleSLpY3k58BZ7t7D3Q9y9+7uflCyA5PWKSoqora2lsLCQoqKinjhhReora1t8/YWLlyImXH44YcnMMrs09znMu3YAezb78xrRU93fS7x03cl8WJpI9nk7q8lPRKJya5du/jzn//M6NGj6dOnD3l5eWzcuJEXXniB0tJS8vPzOfvss7n77ruZNWsWJ5544sdfnA0bNvDhhx8e0IC4e/duduzYwf79+6murmblypWsXLmSf/mXf0m9u1C+8IWoI2hUWz+XI3vDzL88xfl9t6X355KCsv670o5iGUb+10Bf4G/Anrrl7v6X5IaWOJk0jHxNTQ3PPvssb731FtXV1ezfv5/u3bszbNgwTjrpJDp37gxAdXU1zz//PGvWrGHHjh3k5ubSp08fRo4cyZgxY8jNDf6HqN/JKjc391OdrA477LAodjEttfVzqdi0k8fWfMRPZ3yOS6ackr6fy003BY8p1M9H35X4tGYY+VgSyR2NLHZ3/0pbgotCJiWSrLdrV/DYpUu0cSTI7n37OfbHf+eEIYXcOq3RWaNFIpHo+Ug0A6GkjjPOCB4XLIg0jETJ75jDuWP68adF63h/xx4Ku3WKOiSRVmuysd3Mvh8+/o+Z/aZhab8QRTLbF9vQ6J5yUrSjqLSP5s5I6hrYdU1IJIkGH9ydsSW9uX/Jei49cSAdOmTgnO6S0ZpMJO7+UPh4V/uFI5KdLjy2P9+es5IX13zA8YMLow5HpFVi6ZA41MxmmtkTZvZ0XWmP4ESyxekjD6Fnl47ct3h91KGItFos/Uj+DPwO+AMQ23gBIslyySVRR5AU+R1zOG9MMXctXMuW7Xso6q5Gd0kfsSSSGne/PemRiMQiQxMJwIVj+zPrH28zb1kV35gwKOpwWidFO4pK+4hliJSHzOxyMzvEzHrXlaRHJtKY998PSgYafHB3xpYGje61tWk2CODllwdFslIsiWQ68D1gIbAsLDHdyWVmk82swswqzezaRl7vZGZzwtcXm1lJvdeuC5dXmNlpDdbLMbPlZvZwLHFIBjn//KBkqGljB7C+ehcL3/og6lBaZ9euTzqLStZpNpGYWQfgIncvbVAGtrRhM8sBbgVOB4YDF5rZ8AbVZgBb3X0w8EvgZ+G6w4GpwAhgMnBbuL063+KT25NFMsbkkX3p1aUj9y1ZF3UorXPGGZ90FpWs02wicfda4KY2bnssUOnua9x9LzAbmNKgzhSg7vbiecBEC0ZJmwLMdvc97v42UBluDzMrBj5H0PgvklHqGt2fWLWJzdt3Rx2OSExiubT1hJmdZ7HMOflp/YAN9X6uCpc1Wsfda4BtQEEL6/4K+D7Q7LjPZnapmZWbWfmWLVtaGbpIdKaOHUBNbZr3dJesEksi+Q7BLcB7WjmxVWOJp2ELYlN1Gl1uZp8HNrv7spbe3N1nunuZu5cVFRW1HK1Iihh8cDeOLe3N7CUb0q/RXbJSi4kknMiqg7vntXJiqyqgf72fi4GNTdUxs1ygB1DdzLrHA2eZ2VqCS2WnmNmfYohFMsU3vhGUDDft2KDR/YW3MvMONckssfQjwcx6AUOA/Lpl7v5cC6stBYaYWSnwDkHj+bQGdeYT3BX2InA+8LS7u5nNB+4zs1uAQ8P3XuLuLwLXhTFNAK5x94ti2QfJEBdcEHUE7eLjRvfF6zlxSBqcUWdw/x5pWYuJxMy+SnCXVDGwAjiO4A//Kc2t5+41ZnYl8DiQA/zR3VeZ2Q1AubvPB2YB95hZJcGZyNRw3VVmNhdYDdQAV7i7etULbAibzvr3b75emuuUm8P5Rxdzxwtr2bx9Nwd3z295pSgpkWS1WCa2egU4Bljk7qPM7HDgR+6eNv8aamKrDFI3VHmGzEfSnDVbdnDKzc/yvdOGccXJg6MOp3l1nUQLNeBkpmjNxFaxNLbvdvfd4YY7ufvrwLB4AhSRlg0s6sZxA3sze2ka9HTP8I6i0rxYEkmVmfUkmLP9STN7kAMbzUUkCaYdexgbqj/i+Uo1ukvqimWq3XPCp9eb2TMEd1Y9ltSoRASA00b0oXfXPO5fvJ7PDk2DRnfJSrGckWBmJ5jZl939WYKG9oYdC0UkCeoa3Z98bRObPlRPd0lNsUxs9Z/AvxLedgt0BNR3Q6Lx3e8GJYtMGzuAWnfu1aRXkqJi6UdyDjAaeAnA3TeaWfekRiXSlDPPjDqCdldS2JWThx3MfYvXc8XJg+iUm9PySu0tCzqJStNiubS114N7hB3AzLomNySRZlRUBCXLXDK+hPd37OGRV96NOpTGXXBB1nQWlQPFkkjmmtnvgZ5m9jXgKeB/kxuWSBO+/vWgZJkTBhcysKgrd76wNupQGrdhwyedRSXrxDLW1k0EQ7w/QNB/5D/c/X+SHZiIfKJDB+OS8SWsrNrG8vVbow7nQF/6UlAkK8V015a7P+nu33P3a9z9yWQHJSIHOndMMd075XLnwrVRhyLyKU0mkrrh4hspsQ4jLyIJ1K1TLueXFfPIK++yWbcCSwppMpHUDRffSIl1GHkRSbDp40qoqdWtwJJaYhpGXiRl/OAHUUcQqbpbge9dvJ4rTh5MXm5MV6dFkkqJRNLLpElRRxC5S8aXcPEfl/DIK+9y9ugUGWQiyzqJyqcpkUh6WbEieBw1Kto4InTC4EIGFXVl1j/eZsqoQzFrbGbqdpaFHUXlEzovlvRy9dVByWIdOhgzThjIK+9sY9Ga6qjDCWRpR1EJKJGIpKFzx/SjoGseM597K+pQAlnaUVQCSiQiaSi/Yw7Tx5fwTMUW3ti0PepwJMspkYikqS8ddxj5HTvwv8+tiToUyXJKJCJpqlfXPL5Q1p+/rXhHc5VIpJRIJL38+MdBEQC+esJA9tc6d6TqYI6SFXT7r6SX8eOjjiClDCjowukjD+Hexeu48pTBdOsU0Vc6yzuKZjudkUh6WbgwKPKxS08ayPbdNdwf5bApkyaps2gWS2oiMbPJZlZhZpVmdm0jr3cysznh64vNrKTea9eFyyvM7LRwWX8ze8bMXjOzVWb2rWTGLyno3/4tKPKxo/r3ZPygAmY+v4bd+/ZHE8SKFZ90FpWsk7REYmY5wK3A6cBw4EIzG96g2gxgq7sPBn4J/CxcdzgwFRgBTAZuC7dXA3zX3Y8AjgOuaGSbIlnnqolD2LJ9D7OXRHRWoo6iWS2ZZyRjgUp3X+Pue4HZwJQGdaYAd4XP5wETLRjvYQow2933uPvbQCUw1t3fdfe6ueO3A68BKTLYkEh0jhtYwNiS3vzu2TXsqYnorESyVjITST+g/tybVRz4R//jOu5eA2wDCmJZN7wMNhpYnMCYRdLWVROH8N6Hu/lzeVXUoUiWSWYiaWwkOY+xTrPrmlk3gql/r3b3RifZMrNLzazczMq3bNkSY8gi6ev4wQWMHtCT2xe8xd6a2qjDkSySzERSBfSv93MxsLGpOmaWC/QAqptb18w6EiSRe939L029ubvPdPcydy8rKiqKc1ckZfzqV0GRA5gZV50yhHf++RF/Xa6zEmk/yUwkS4EhZlZqZnkEjefzG9SZD0wPn58PPO3uHi6fGt7VVQoMAZaE7SezgNfc/ZYkxi6patSorB5CviUThhVxZL8e/PaZyvY9K1FH0ayWtEQStnlcCTxO0Cg+191XmdkNZnZWWG0WUGBmlcB3gGvDdVcBc4HVwGPAFe6+Hzge+BJwipmtCMsZydoHSUFPPRUUaZSZ8Z1Th7Kh+iPmLG3HO7jGj1dn0SxmwQlAZisrK/Py8vKow5BEmDAheFywIMooUpq7c8HMRazZspPnvj+BLnnt0Nu9rpOokknGMLNl7l4WS131bBfJMGbGv04exvs79rTfGFzqKJrVlEhEMtDRh/Vm0hF9+N2Ct9i6c2/U4UiGUyIRyVDfO20YO/bW8LtnU2QWRclYSiQiGWpY3+6cM7ofdyxcy4bqXVGHIxlMiUTSy+9/HxSJyfdOG0aOGf/9f69FHYpkMM1HIull2LCoI0grh/TozOUTBnHzk2+wsPJ9xg8uTM4bqZNoVtMZiaSXhx4KisTsaycNpLhXZ3700Gpq9iepk6I6imY1JRJJLzffHBSJWX7HHH7wuSOo2LSde5M1+ZU6imY1JRKRLHDaiL4cP7iAm56oYNOHuxP/BjcOoJA9AAALZUlEQVTeGBTJSkokIlnAzLjx7CPZU1PLfz64KupwJMOosV0kS5QWduXqSUP4+WMVPPbqu0weeUjCtr1zbw2799Wy5JV3E7ZNiV9ebgcmHtEn6e+jRCKSRb524kAeXvkuP3xwFeMGFtKjS8eEbHf1xu3sr63lG/e+lJDtSWIUdutE+Q+USEQ+7Z57oo4grXXM6cDPz/8MU259gf+Y/yq/njo6Idutdaeoeyceu/rEhGxPEiO3Q2NzBCbhfdrlXUQSpX//lutIs0b268FVpwzhl0+9wYRhRZwzujjubf5g8pVccEx/vtr3oAREKOlGje2SXubMCYrE5YqTB3FMSS9++LdVrP8g/uFT3urdj20DBiYgMklHSiSSXm6/PSgSl9ycDvzyglGYwTdnL2dPzf64tnfym4sZtOiZBEUn6UaJRCRLFffqws/P+wwrN/yT//jbKto6yZ2787Ulf6XsgTsSHKGkCyUSkSx2+pGH8M1TBjOnfAP3LFrXpm3UZv4kq9ICJRKRLPftSUOZdMTB/Oih1Syo2Nzq9WvDMxlrnxuEJAUpkYhkuQ4djF9eMIrD+3bnsj8to3xtdavWr23jJTHJHEokkl7mzQuKJFT3/I7c9ZWxHNqjM1++cymvvrMt5nWVR0SJRNJLYWFQJOEKu3Xi7hljOSi/IxfOXMSiNR/EtF6tO9/+/Hd55t9vSnKEkqqUSCS93HlnUCQpint14c+XjaNPj3wu/uMSHn55Y4vr1Dq8e1ARH/Xp1w4RSipSIpH0okSSdIf27Myfvz6OI/v14Mr7lvNfD69mXzMTYtW68/nXnmPQ0w+3Y5SSSpKaSMxssplVmFmlmV3byOudzGxO+PpiMyup99p14fIKMzst1m2KSPx6dc3j/q8dxyXjS5j1j7c583/+wfL1Wxut67Vw0fJHOOLB+9o5SkkVSUskZpYD3AqcDgwHLjSz4Q2qzQC2uvtg4JfAz8J1hwNTgRHAZOA2M8uJcZsikgB5uR24/qwR/P5LR/PPXfs49/aFfGv2ct7YtP1T9fbX3f4bRZCSEpI5aONYoNLd1wCY2WxgCrC6Xp0pwPXh83nAb83MwuWz3X0P8LaZVYbbI4ZtikgCnTaiL+MHFfDbZyq558V1PLhiI6P69+T0kX0Zc1gvetUNRa9MkrWSmUj6ARvq/VwFHNtUHXevMbNtQEG4fFGDdeta8lrapogkWPf8jlx3+hFcdtIg5pRv4KGVG/nJo69//PpslEeyWTITSWO/Vw3vOG+qTlPLG7sU1+hd7GZ2KXApwIABA5qOUtLLI49EHUFW69U1j8s+O4jLPjuIzR/u5uWqbVRt3UX/x7rQu2te1OFJRJKZSKqA+pNHFAMN7yWsq1NlZrlAD6C6hXVb2iYA7j4TmAlQVlamLlOZokuXqCOQ0MEH5TNpeH7ww1P/F20wEqlk3rW1FBhiZqVmlkfQeD6/QZ35wPTw+fnA0x4MQTofmBre1VUKDAGWxLhNyWS33RYUSS3qKJrVknZGErZ5XAk8DuQAf3T3VWZ2A1Du7vOBWcA9YWN6NUFiIKw3l6ARvQa4wt33AzS2zWTtg6SguXODx8svjzYO+bS6vj2XXBJlFBIRa+scBOmkrKzMy8vLow5DEmHChOBxwYIoo5CG9LlkHDNb5u5lsdRVz3YREYmLEomIiMRFiUREROKSzNt/RRJP1+BFUo4SiYjETx1Fs5oSiYjETx1Fs5raSEQkfuoomtWUSEQkfnPnftJZVLKOEomIiMRFiUREROKiRCIiInFRIhERkbjo9l8RiZ86imY1nZGIiEhclEhERCQuSiQiIhIXJRIREYmLEomIiMRFiUREROKiRCIiInFRIhERkbgokYiISFzM3aOOIenMbAuwro2rFwLvJzCcdJTtxyDb9x90DCD7jsFh7l4US8WsSCTxMLNydy+LOo4oZfsxyPb9Bx0D0DFoji5tiYhIXJRIREQkLkokLZsZdQApINuPQbbvP+gYgI5Bk9RGIiIicdEZiYiIxEWJpAlmNtnMKsys0syujTqeZDKztWb2ipmtMLPycFlvM3vSzN4MH3uFy83MfhMel5fNbEy00beNmf3RzDab2av1lrV6n81selj/TTObHsW+tFUTx+B6M3sn/F1YYWZn1HvtuvAYVJjZafWWp+V3xcz6m9kzZvaama0ys2+Fy7Pq9yAh3F2lQQFygLeAgUAesBIYHnVcSdzftUBhg2U/B64Nn18L/Cx8fgbwKGDAccDiqONv4z6fBIwBXm3rPgO9gTXhY6/wea+o9y3OY3A9cE0jdYeH34NOQGn4/chJ5+8KcAgwJnzeHXgj3M+s+j1IRNEZSePGApXuvsbd9wKzgSkRx9TepgB3hc/vAs6ut/xuDywCeprZIVEEGA93fw6obrC4tft8GvCku1e7+1bgSWBy8qNPjCaOQVOmALPdfY+7vw1UEnxP0va74u7vuvtL4fPtwGtAP7Ls9yARlEga1w/YUO/nqnBZpnLgCTNbZmaXhsv6uPu7EHzhgIPD5Zl8bFq7z5l6LK4ML938se6yDhl+DMysBBgNLEa/B62mRNI4a2RZJt/edry7jwFOB64ws5OaqZttxwaa3udMPBa3A4OAUcC7wM3h8ow9BmbWDXgAuNrdP2yuaiPLMuIYxEuJpHFVQP96PxcDGyOKJencfWP4uBn4K8Hlik11l6zCx81h9Uw+Nq3d54w7Fu6+yd33u3st8L8EvwuQocfAzDoSJJF73f0v4eKs/z1oLSWSxi0FhphZqZnlAVOB+RHHlBRm1tXMutc9B04FXiXY37q7T6YDD4bP5wMXh3ewHAdsq7sMkAFau8+PA6eaWa/wEtCp4bK01aC96xyC3wUIjsFUM+tkZqXAEGAJafxdMTMDZgGvufst9V7K+t+DVou6tT9VC8EdGm8Q3JHy71HHk8T9HEhwp81KYFXdvgIFwN+BN8PH3uFyA24Nj8srQFnU+9DG/b6f4NLNPoL/KGe0ZZ+BrxA0PFcCX456vxJwDO4J9/Flgj+ch9Sr/+/hMagATq+3PC2/K8AJBJegXgZWhOWMbPs9SERRz3YREYmLLm2JiEhclEhERCQuSiQiIhIXJRIREYmLEomIiMRFiUSkBWa2MHwsMbNpCd72vzX2XiLpRLf/isTIzCYQjIz7+Vask+Pu+5t5fYe7d0tEfCJR0RmJSAvMbEf49KfAieE8Hd82sxwz+4WZLQ0HOfx6WH9COM/FfQQd1zCzv4WDYq6qGxjTzH4KdA63d2/99wp7T//CzF61YK6YC+pte4GZzTOz183s3rCHNmb2UzNbHcZyU3seI8luuVEHIJJGrqXeGUmYELa5+zFm1gl4wcyeCOuOBUZ6MOQ6wFfcvdrMOgNLzewBd7/WzK5091GNvNe5BAMnHgUUhus8F742GhhBMJ7TC8DxZraaYEiTw93dzaxnwvdepAk6IxFpu1MJxl5aQTD8eAHBGFQAS+olEYCrzGwlsIhggL8hNO8E4H4PBlDcBDwLHFNv21UeDKy4AigBPgR2A38ws3OBXXHvnUiMlEhE2s6Ab7r7qLCUunvdGcnOjysFbSuTgHHufhSwHMiPYdtN2VPv+X4g191rCM6CHiCYiOmxVu2JSByUSERit51gStY6jwPfCIcix8yGhiMoN9QD2Oruu8zscIJpWuvsq1u/geeAC8J2mCKCaXGXNBVYOKdGD3d/BLia4LKYSLtQG4lI7F4GasJLVHcCvya4rPRS2OC9hU+mZa3vMeAyM3uZYOTcRfVemwm8bGYvufsX6y3/KzCOYFRmB77v7u+Fiagx3YEHzSyf4Gzm223bRZHW0+2/IiISF13aEhGRuCiRiIhIXJRIREQkLkokIiISFyUSERGJixKJiIjERYlERETiokQiIiJx+f+y21bWB+28VAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_moms=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of those decays are implemented in the file sgdr.py in the function next_val of the class DecayScheduler. Adding your own is as simple as:\n",
    "   - adding it in the DecayType class\n",
    "   - implementing your own formula in the function next_val."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SGDR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The traditional stochastic gradient with restart we used requires three arguments:\n",
    "- the number of cycles\n",
    "- the length of a cycle (cycle_len)\n",
    "- the value by which we should multiply the cycle_len after each cycle (cycle_mult)\n",
    "\n",
    "Also, note there is a warm-up at constant lr of 1/100th of the max value during 1/20th of the first cycle.\n",
    "\n",
    "This can easily be implemented in a function (which you should now try to do by yourself) which creates a list of phases (number of cycles + 1 here). Remember you can have a float number for the epochs in a Training Phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "def phases_sgdr(lr, opt_fn, num_cycle,cycle_len,cycle_mult):\n",
    "    phases = [TrainingPhase(epochs = cycle_len/ 20, opt_fn=opt_fn, lr=lr/100),\n",
    "              TrainingPhase(epochs = cycle_len * 19/20, opt_fn=opt_fn, lr=lr, lr_decay=DecayType.COSINE)]\n",
    "    for i in range(1,num_cycle):\n",
    "        phases.append(TrainingPhase(epochs = cycle_len * (cycle_mult**i), opt_fn=opt_fn, lr=lr, lr_decay=DecayType.COSINE))\n",
    "    return phases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc888fd7f0a246c8aaedd6f99e63c798",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.576923   4.471715   0.1359    \n",
      "    1      3.773803   4.577302   0.1383                                                                                \n",
      "    2      3.521146   4.507113   0.1402                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.50711]), 0.1402]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases_sgdr(1e-2, optim.Adam, 2, 1, 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VfX5wPHPc+/NIosVAoSRAGGEDZGhuGU5QFGUuqhira3WrUX91VpbtajV2jpaXHUjblQEVNyy9wyEJWFL2CsEnt8f92DTmISbcXPueN6v131xc+45J8853HuffLeoKsYYY0xVedwOwBhjTHizRGKMMaZaLJEYY4ypFkskxhhjqsUSiTHGmGqxRGKMMaZaLJEYY4ypFkskxhhjqsUSiTHGmGrxuR1AbWjYsKFmZma6HYYxxoSNOXPm/KiqaYHsGxWJJDMzk9mzZ7sdhjHGhA0RWRfovla1ZYwxploskRhjjKkWSyTGGGOqxRKJMcaYarFEYowxplqCmkhEZJCI5IlIvoiMLuP1OBF503l9hohkOtsbiMgXIrJXRJ4sdUxPEVnkHPMPEZFgXoMxxpiKBS2RiIgXeAoYDOQAvxCRnFK7jQJ2qGob4HFgjLP9IPAH4PYyTv0McC2Q7TwG1Xz0xhhjAhXMcSS9gHxVXQ0gIuOAocDSEvsMBe5znr8NPCkioqr7gG9FpE3JE4pIEyBFVac5P78MnA98EsTr+B+z1xYyY00haUlxdG6WSrv0ZDweKxTVli27D/LW7PV4PR7iYzwkxvpIT42nifNIjo9xO0Rjok4wE0kGsL7EzwVA7/L2UdViEdkFNAB+rOCcBaXOmVHWjiJyLf6SCy1atKhs7OV6eFIeM9cW/jegugkMz23GVSdmkVrHvsSC7YP5G3h0yopyX2+aGk+HJinkNE0hN7M+J2TWo05sVIy7NcY1wfyElfVnulZhnyrtr6pjgbEAubm5FZ2zUo6o0jurPg8N68ycdTuYsGAjT3y+khe/W8ttA9pyee+WVkIJoiNH/f/O/UN/BNh7qJgtuw+ycddBCnbsJ2/zHpZt2s2XK7Zx5Gg+Po/QrXldzujQiLM7NSGzYaKr8RsTiYKZSAqA5iV+bgZsLGefAhHxAalAIeUrcM5T0TmDLsbroVVaEq3Skhie25ylG3fz4MRl3PvBEj5duoUnRnSnfmJsbYcVVerEeomP8VIvMZbm9ev87PUDRUeYva6Q71dt59uVP/LwpDwenpRHhyYpnN+tKRf1bEaDpDgXIjcm8gSz19YsIFtEskQkFhgBTCi1zwRgpPP8ImCqqpZbelDVTcAeEenj9Na6Evig5kOvnJymKbwyqhd/Ob8TM9cUMuzp71jz4z63w4pqCbFeTs5O4/eD2vPh7/rx3egz+MO5OSTEeHjok+X0fWgqv3tjHjPXFFLBW84YE4CglUicNo8bgMmAF3hBVZeIyP3AbFWdADwPvCIi+fhLIiOOHS8ia4EUIFZEzgcGqOpS4DfAf4AE/I3stdbQXhER4fI+LenQJIVfvTybC5/5njev7UN2erLboRn8bVmj+mUxql8WK7bs4fUZP/Du3AI+XLCR3Jb1uP70NpzWLg3rTW5M5Uk0/DWWm5urNTX774XPfE9CjJdXryndb+C/Vm/byyVjpyPA+F/3tXr5GvTMl6sYM2k5y/88iPgYb7XOdaDoCONnr2fs16vZsPMAOU1SuHNQO05tawnFGBGZo6q5gexrI9srKZDE2yotidev6U3xUeXKF2ayY19RLUQWHbTCvhiVkxDrZeSJmXx5x2k8Orwrew8V88sXZ3HF8zNZsnFXjf0eYyKdJZIqCOSP1ez0ZJ4fmcvm3Qf57WtzOXysu5EJOTFeDxf1bMZnt57KvefmsHjjLs7957fc9e5Cdu0/7HZ4xoQ8SyRB1L1FPR66oDPTVm9nzCfL3Q7HHEesz8PV/bL46o7TGXVSFuNnF3DmY1/ywfwN1iBvTAUskQTZhT2bcWXfljz37Rq+XrHN7XBMAFITYvi/c3OYcMNJZNRN4KZx87nyhZls2nXA7dCMCUmWSGrB3Wd3oG16Ere9tYDtew+5HY4JUMemqbz725O477wcZq/dwcDHv+ajhbU+bMmYkGeJpBbEx3h5YkR3du0/zL0fLHE7HFMJXo/wy5OymHjTybRKS+KG1+dxy5vz2X3Q2k6MOcYSSSVVtaa8Q5MUfndGGz5etImpy7fUaEzRxK2miqyGibx9XV9uPiubCQs2MuSf37Js0253gjEmxFgiqUW/PrU12Y2S+MP7S9h3qNjtcMKaG8M8fF4PN5/VlnHX9mF/0REuePo73p1bcPwDjYlwlkhqUazPw0PDOrNh5wGe+Hyl2+GYKjohsz4f33gy3ZrX5dbxC7jnvUUUFVv3bhO9LJHUstzM+gzv2YwXv1vDuu02H1e4SkuO49VRvbnu1Na8NuMHrnxhBjv328BTE50skbjg9oHtiPF6GDPJxpaEM5/Xw+jB7XliRDfmrtvJBU9/z+pte90Oy5haZ4mkkmqisTc9JZ5fn9KaiYs2M2ttRbPmm3AwtFsGr/+qN7sOHOaCp79n2qrtbodkTK2yROKSX52SReOUeP7y8TIbNR0BcjPr8/5vTyItOY6RL8xk4qJNbodkTK2xRFIFNTEzbJ1YHzeflc2C9Tv5Im9rDUQVXaTMxTLd1aJBHd657kS6NEvl+tfn8vqMH9wOyZhaYYnERRf2bEbz+gk8/ulKK5VEiNQ6Mbwyqjentk3j7vcW8dQX+fZ/ayKeJRIXxXg93HhGNos27OKzZVYqiRQJsV6evTKX87s15ZHJeTz0yXJLJiaiWSJx2QXdM8hsUIfHP11hXzYRJMbr4bGLuzGyb0vGfr2aP39kbWEmclkiqaSa/irweT3ceGY2Szft5nMrlUQUj0e4b0hHrjopkxe+W8P9Hy21ZGIikiWSEHBe16Zk1E1g7Ner3Q4l5IXbF7GIcO+5OVx9UhYvfrfWkomJSJZIqqCm+wvFeD2M6pfFzLWFzFm3o4bPHpnCaUl1EeEP53ZgVD9/MvnTh5ZMTGSxRBIiLjmhOakJMYz9epXboZggEBH+7xx/MvnP92v525QVbodkTI2xRBIiEuN8XNm3JVOWbmGVTbMRkY4lkxEnNOfJL/LtjwYTMSyRhJAr+2YS4/Xw3DfWVhKpRIQHLujMOV2a8ODE5YybaYMWTfizRFJZQazbTkuOY1j3DN6du8Fmko1gXo/w+MXdOLVtGne9t8iW7zVhzxJJiLmybyaHio8yfvZ6t0MJSZHSRh3r8/Cvy3uS27Iet7w5n29X/uh2SMZUmSWSKghmj6Gcpin0yqzPK9PXceRohHxrBkEYddoqV0Ksl+dGnkCrhklc9+ocW7rXhC1LJCFo5ImZrC88wJc2mWPES02I4cWrTiAxzstVL85i064DbodkTKVZIglBAzqmk54Sx0vT1rkdiqkFTesm8OIve7H3UDFXvTiLPQcPux2SMZViiSQExXg9XNa7JV+v2GYr7kWJnKYpPH1ZD/K37uU3r87l8BFbA96ED0sklVRbrRYjejUnxiu8ZmtaRI1T2qbx4LDOfJv/I3e/u8hGv5uwEdREIiKDRCRPRPJFZHQZr8eJyJvO6zNEJLPEa3c52/NEZGCJ7beIyBIRWSwib4hIfDCvwS2NkuPpn5POe/M2cKj4iNvhhIxI/2q9OLc5N57RhrfmFPD8t2vcDseYgAQtkYiIF3gKGAzkAL8QkZxSu40CdqhqG+BxYIxzbA4wAugIDAKeFhGviGQANwK5qtoJ8Dr71ara6jF0yQktKNxXxGdLrdG9tJpYpTJU3XxWWwZ3asyDE5fZ6pkmLASzRNILyFfV1apaBIwDhpbaZyjwkvP8beBM8X9DDAXGqeohVV0D5DvnA/ABCSLiA+oAETuaq1+bhmTUTWDcLKveiiYej/C3i7vSrnEKN74+j/yte9wOyZgKBTORZAAlR9UVONvK3EdVi4FdQIPyjlXVDcCjwA/AJmCXqk4JSvQhwOsRhuc249v8H1lfuN/tcEwtqhPr49krexLr83DNS7NtpgMT0oKZSMqqeyhdxV3ePmVuF5F6+EsrWUBTIFFELi/zl4tcKyKzRWT2tm3bKhF2aBme2xyAt+YUuByJqW3N6tXh31f0ZMPOA9zw+jyKrSeXCVHBTCQFQPMSPzfj59VQP+3jVFWlAoUVHHsWsEZVt6nqYeBd4MSyfrmqjlXVXFXNTUtLq4HLOXbeGjtVQDLqJnBydhpvzV5vI92jUG5mfR64wN+T64GJy9wOx5gyBTORzAKyRSRLRGLxN4pPKLXPBGCk8/wiYKr6+zxOAEY4vbqygGxgJv4qrT4iUsdpSzkTiPhP14gTmrNp10G+XhG+JauaEo09Yi/Obc4vT8zkxe/W8sH8DW6HY8zPBC2ROG0eNwCT8X/Zj1fVJSJyv4gMcXZ7HmggIvnArcBo59glwHhgKTAJuF5Vj6jqDPyN8nOBRU78Y4N1DeWp7R5DZ3VIp16dGN6ea9Vbx0Run62y3X12B3Jb1mP0O4tYscUa301oCeo4ElWdqKptVbW1qj7gbLtXVSc4zw+q6nBVbaOqvVR1dYljH3COa6eqn5TY/kdVba+qnVT1ClU9FMxrCAWxPg/ndW3Kp0u3sNumz4hKsT4PT13Wg8Q4H9e9MsfeByak2Mj2MHFB9wyKio/yyaJNbodiXJKeEs9Tl3ZnXeF+bh+/wEa+m5BhiSRMdGtel6yGibw71+rIo1nvVg24++wOTFm6hX99ZStpmtBgiaSS1KVJOkSEYd0zmLGmkIIdNqYkml19UibndmnCI5OX812+LYhl3GeJJIyc390/nvP9edFbKnErkYcSEWHMhV1onZbEjW/MY+vug26HZKKcJZIqcKvHUPP6deiVWZ93522I+vrxCJ5qKyCJcT6evqwH+4uOcNO4+TbGyLjKEkmYuaBHBqu37WNhwS63QzEuy05P5v6hHZm2ejtPTs13OxwTxSyRhJmzOzch1ufhvSiu3jL/dVHPZgzrnsETn69g2qrtbodjopQlkjCTmhDDWR0a8eGCjTb3kkFE+PP5nchsmMhN4+axfW/ED6syIcgSSSWFQtPEkK5N2b6viOmrC90OxYSAxDgfT13ag50HDnPL+AUctfYSU8sskYSh09o1IjHWy4cLInYplnKFQiIPRR2apPDH83L4esU2/v21jS8xtcsSSRW43WMoPsbLgI6NmbRkM0XF0Vm9FckrJFbVpb1acE6XJjw6JY8563a4HY6JIpZIwtS5XZqw68BhG5BmfiIi/HVYZ5rWjefmN+exx+bjMrXEEkmYOjk7jZR4X1RWb5nyJcfH8PdLurNx50H+OGGJ2+GYKGGJJEzF+jwM6tSYKUu3cPDwEbfDMSGkZ8t6/O6MNrw7dwMT7A8NUwsskVRSKDX2nte1KXsPFfNlni14Zf7XDae3oUeLutzz3iI27DzgdjgmwlkiCWN9WzWgQWIsHy2Mnr86QyiPhzSf18PfL+mOKtzypk2hYoLLEkmVhEaPIZ/Xw+DOjfl82Vb2FxW7HY4JMS0a1OFPQzoyc00h//pqldvhmAhmiSTMndulKQcOH+HzZVvdDsWEoGE9Mji3SxMe/3QFC9bvdDscE6EskYS5EzLrk54SZ723TJlEhAfO70yj5DhuGjePfYes5GpqniWSMOf1CIM7NeGrFdvsS8KUKbVODI9d0o11hfv580dL3Q7HRCBLJJUUik2Wgzo15lDxUeu9ZcrVp1UDrj2lFeNmrWfq8i1uh2MijCWSCHBCZn0aJsXyyeJNbocSfKHU/zrM3Nq/Le3Sk/n9O4vYsa/I7XBMBLFEUgWhNs2T1yP0z2nMF8u3RsXgxFC7/+EizuflsUu6snN/EX/4YLHb4ZgIEnAiEZHEYAZiqmdwp8bsKzrCNytt7i1Tvo5NU7npzGw+WrjJRr2bGnPcRCIiJ4rIUmCZ83NXEXk66JGZSunbugGpCTHRUb1lquW6U1vTrXld/vD+YrbsPuh2OCYCBFIieRwYCGwHUNUFwCnBDMpUXozXw1kd0vls6ZaonVreBMbn9fC3i7tyqPgIv39nIWrtTqaaAqraUtX1pTZFfkV8OUL5Qze4U2N2Hyxm2mpbu9tUrHVaEqMHtefLvG2Mm1X6421M5QSSSNaLyImAikisiNyOU80VrUK1rbdfdkMSY71MiuDqrdBN4+Hnyr6ZnNi6AX/+aCk/bN/vdjgmjAWSSK4DrgcygAKgG/DbYAZlqiY+xssZHdKZsmRLRE/SF6qJPNx4PMIjw7viFeH2txZE9HvGBFcgiaSdql6mqumq2khVLwc6BDswUzWDOzVm+74iZq4pdDsUEwYy6ibwxyEdmbm2kBe+XeN2OCZMBZJI/hngNhMCTmuXRnyMx3pvmYBd2COD/jnpPDIlj/yte90Ox4ShchOJiPQVkduANBG5tcTjPsAbyMlFZJCI5IlIvoiMLuP1OBF503l9hohklnjtLmd7nogMLLG9roi8LSLLRWSZiPStxPVGvDqxPk5tm8akxZs5alUVJgAiwgMXdKJOrJc73rYqLlN5FZVIYoEkwAckl3jsBi463olFxAs8BQwGcoBfiEhOqd1GATtUtQ3+bsZjnGNzgBFAR2AQ8LRzPoAngEmq2h7oSpQ3/JdlcKcmbN1ziHnrd7gdigkTjZLj+dOQjsz7YadVcZlK85X3gqp+BXwlIv9R1XVVOHcvIF9VVwOIyDhgKFBy+tGhwH3O87eBJ0VEnO3jVPUQsEZE8oFeIrIE/xiWXzoxFgG1PmlQqE/RcUaHRsR4hSlLttCzZX23w6lRIdz7OuwN6dqUjxZu4tEpeZzRoRGt05LcDsmEiUDaSPaLyCMiMlFEph57BHBcBlCyg3qBs63MfVS1GNgFNKjg2FbANuBFEZknIs+VN3WLiFwrIrNFZPa2bdE1K25KfAx9Wzdk8pLNIT3upaok1DN5mPKvXdKJ+Bgvd7690Kq4TMACSSSvAcuBLOBPwFpgVgDHlfVpL/3OLG+f8rb7gB7AM6raHdgH/KztBUBVx6pqrqrmpqWlBRBuZBmQk87a7ftZaY2nphIapcRz35Ac5qzbwYvfWRWXCUwgiaSBqj4PHFbVr1T1aqBPAMcVAM1L/NwMKD1L3E/7iIgPSAUKKzi2AChQ1RnO9rfxJxZTSv+cdACmLNnsciQm3JzfLYOzOjTikcl5rPlxn9vhmDAQSCI57Py7SUTOEZHu+L/Yj2cWkC0iWSISi7/xfEKpfSYAI53nFwFT1V8XMwEY4fTqygKygZmquhn/SPt2zjFn8r9tLkEXLjVF6SnxdGtelylLbREjUzn+XlydifN5uMMGKpoABJJI/iIiqcBtwO3Ac8AtxzvIafO4AZiMv2fVeFVdIiL3i8gQZ7fngQZOY/qtONVUqroEGI8/SUwCrlfVY/N7/Q54TUQW4h9l/2BAVxqFBnRMZ2HBLjbuPOB2KCbMpKfE88fzOjJ73Q5e+n6t2+GYEFdury34qQtvtqp+hL8h/PTKnFxVJwITS227t8Tzg8Dwco59AHigjO3zgdzKxFHTJEwm6RiQ05iHJ+Xx2bItXNk30+1waoTabFu1ZliPDD5etImHJy/njPaNyGxoSxKZslVYInFKAUMq2seErjaNkmiVlsiUJZFVvRUeaTz8iQgPXtCZGK+HO99eaANcTbkCqdr6XkSeFJGTRaTHsUfQIzM1YkBOY6av3s6u/YePv7MxpTROjefec3OYubaQl6etdTscE6ICSSQn4h9hfj/wN+fxaDCDMjVnQMd0io8qX+RtdTsUE6Yu6tmM09qlMWZSHuu2Wy8u83PHTSSqenoZjzNqI7hQFG519N2a1SUtOY4pS60bsKkaEeGhYZ3xecSquEyZAloh0YQvj0fon5POl3nbOHg4ahe2NNXUJDWB/zu3AzPWFPLajKrMmGQimSWSKgi3GToG5KSzv+gI36/60e1Qqi1cxvFEootzm3NydkMe+mQ56wttRUXzX5ZIokDf1g1IivNFTO+tcEvkkUJE+OuFXfCIMPrdhRE5j5upmgrHkQCIyLAyNu8CFqmqteCGgTifl9PapfHZMv8SvF6PfRObqsmom8BdZ7fnnvcW88bM9Vzau4XbIZkQEEiJZBT+0eyXOY9n8Y9C/05ErghibKYGDejYmB/3FjHvB1ujxFTPpb1acGLrBjw4cRkbbNYEQ2CJ5CjQQVUvVNUL8S9SdQjoDfw+mMGFonAtzZ/WLs2/RonNvWWqSUQYc2EXjqpy17uLrIrLBJRIMlW15LfPVqCtqhby3wkdTYiL9DVKTO1qXr8Ovx/Unq9XbOOtOQVuh2NcFkgi+UZEPhKRkSIyEvgA+NpZUGpncMMLTeHa2DsgJ511Yb5GiaXA0HFFn5b0yqrPnz9ayuZdB90Ox7gokERyPfAf/DPtdgdexj8b7z5VrdQkjsZdkbJGSbhMmhnpPB7h4Qu7cPjIUe55z6q4olkgI9tVVd9W1VtU9Wbnub1jwpCtUWJqWmbDRG4f0I7Pl2/l/fkb3A7HuOS4iUREhonIShHZJSK7RWSPiOyujeBMzbM1SkxNu+qkLHq0qMt9E5aydY9VcUWjQKq2HgaGqGqqqqaoarKqpgQ7sFAV7kWxATmNAfhsmZVKTM3weoSHL+rKgcNH+MP7i62KKwoFkki2qOqyoEdiasWxNUomh3k7iQktbRolcWv/tkxesoWPFm5yOxxTywJJJLNF5E0R+YVTzTWsnNHuUSPcG3sHdmzM9NWF7Nxf5HYolWZ/7Iaua/pl0bVZKn+csITtew+5HY6pRYEkkhRgPzAAOM95nBvMoExwDchJ58hRZeryMJ3hJrzzeMTyeT08Mrwrew8Wc++EJW6HY2rRcefaUtWraiMQU3u6NqtLekock5dsZliPZm6HYyJI2/RkbjyzDY9OWcF5XTYxqFMTt0MytaDcRCIid6rqwyLyT8poY1bVG4MamQkaj0cYkNOYt+as50DRERJivW6HZCLIr09tzSeLN/N/7y+md1YD6iXGuh2SCbKKqraONbDPBuaU8YhKkdIjZWDHxhw8fJRvVm5zOxQTYWK8Hh65qCs79x/mTx9aFVc0KLdEoqofOv++VHvhmNrSu1V9UuJ9TF6yhQEdG7sdjokwOU1TuP70Njzx+UrO6dL0p1kVTGQKZEBiWxEZKyJTRGTqsUdtBBeyIqCxN8br4cwO6Xy+fAvFR466HU7ANOxH8kSP609vQ/vGydzz3iJ27bf5XSNZIL223gLmAf8H3FHiYcLcwI7p7Nx/mJlrC90OpVIiII9HhVifh0eHd2X7viLu/2ip2+GYIAokkRSr6jOqOlNV5xx7BD0yE3SntE0jzueJmCV4TejplJHKdae24p25BXyRF6bdzc1xBZJIPhSR34pIExGpf+wR9MhM0NWJ9XFydhpTbI0SE0Q3nplNdqMk7npnEbsPWhVXJAokkYzEX5X1Pf/tsTU7mEGFskj7uh3YMZ2Nuw6yeIPNw2mCI87n5ZHhXdm65yAPfmyzLUWiChOJiHiAy1U1q9SjVS3FZ4LsrA7peASbe8sEVbfmdfnVya0YN2u9dTmPQBUmElU9CjxaS7GEjUhq7K2XGEuvrPrhk0girUgYRW7p35ZWDRMZ/c4i9h4qdjscU4MCqdqaIiIXilR+gVkRGSQieSKSLyKjy3g9zpkQMl9EZohIZonX7nK254nIwFLHeUVknoh8VNmYzM8N7NiYlVv3snpbeCzBG65LHUe7+BgvjwzvwsZdB/jrJ1bFFUkCSSS34u8CfKgyC1uJiBd4ChgM5AC/EJGcUruNAnaoahvgcWCMc2wOMALoCAwCnnbOd8xN/HfkvammYwMSbeVEE2w9W9bnqhOzeHX6D3y/6ke3wzE1JJCldpNV1aOqsZVc2KoXkK+qq1W1CBgHDC21z1Dg2Mj5t4EznZLPUGCcqh5S1TVAvnM+RKQZcA7wXCAXaI4vo24CnTJSwqd6y4S1Owa2o2WDOox+ZxH7i6yKKxIEUiJBROqJSC8ROeXYI4DDMoD1JX4ucLaVuY+qFgO7gAbHOfbvwJ2AO8OxI7SOfmBOY+b9sJOtu22pVBNcCbFexlzYhR8K9/PwpDy3wzE1IJApUq4BvgYmA39y/r0vgHOXVZNd+mu4vH3K3C4i5wJbAxkQKSLXishsEZm9bZv1EjmegZ2sesvUnj6tGjCyb0temraWWWE2s4L5uUBKJDcBJwDrVPV0oDsQyDdzAdC8xM/NgI3l7SMiPiAVKKzg2JOAISKyFn9V2Rki8mpZv1xVx6pqrqrmpqWlBRBu4KrQ7yDkZTdKIqth6C/BG6EFwqh056D2NKuXwJ1vL+RA0RG3wzHVEEgiOaiqB8Hfy0pVlwPtAjhuFpAtIlkiEou/8XxCqX0m4B/wCHARMFX9Q6wnACOcXl1ZQDYwU1XvUtVmqprpnG+qql4eQCzmOESEATnpTFu1nV0HQnv0cbgvdWz8EuN8jBnWhTU/7uOxT62KK5wFkkgKRKQu8D7wqYh8wM9LFj/jtHncgL8qbBkwXlWXiMj9IjLE2e15oIGI5OPvHTbaOXYJMB5YCkwCrldV+5MlyAZ0bEzxUeVLmxPJ1JIT2zTk0t4teP7bNcz9YYfb4ZgqCmSp3Qucp/eJyBf4q58mBXJyVZ0ITCy17d4Szw8Cw8s59gHggQrO/SXwZSBxmMB0b16XtGT/ErxDu5XuF2FMcNw1uD1fLt/KHW8t4OMbTyY+xlbsDDeB9trqJyJXqepXwDR+3vsqakRyHb3HI/TPSefLvG0cPGwFQFM7kuNjeOjCLqzato+/f7bS7XBMFQTSa+uPwO+Bu5xNMUCZDdwm/A3s2Jj9RUf4Lt8Gi5nac2rbNC7ObcbYr1exYP1Ot8MxlRRIieQCYAiwD0BVNwLJwQwq1EVyU2/fVg1IjveFbO8tm+4+ct1zTg6NkuO5dfx8KxGHmUASSZHTk0oBRCQxuCEZN8X6PJzZvhFTlm7hcIguwRuBva8NkJoQw8MX+au4xkxa7nY4phICSSTjReTfQF0R+RXwGfBscMMybjq7cxN27j/MtFXb3Q7FRJlT2qZxZd+WvPjdWqteDSOBzLX0I4GfAAAWDElEQVT1KP55sN7BP37kXlX9Z7ADM+45pW0aSXE+Ji7a5HYoJgrdNbgDrRomcvtbC0J+TJPxC6jXlqp+qqp3qOrtqvppsIMKZdFQRx8f4+XMDo2YvGRzyFZvmciVEOvl8Uu6sXXPIf74wWK3wzEBKDeRHJsuvoxHQNPIR7JoqKM/u3MTduw/zPTVVr1lal/X5nW54fQ2vD9/Ix8vtJJxqCs3kRybLr6MR6DTyJswdmrbNBJjvSFXvRUFBULjuOGMNnRtlso97y+yWalDXEBVWyb6+Ku30pm8ZAvFIVa9FQUFQgPEeD08dkk3Dh4+wh1vL4yKauVwZYnElOvszk0o3FfE9NU2zbdxR+u0JO4a3IGvVmzj1Rk/uB2OKYclElOu09r5q7c+DrHqLRNdrujTkpOzG/Lgx8tY8+M+t8MxZbBEUknRVLiOj/FyRod0Ji/ZHHLVWyZ6eDzCIxd1Jdbn4eY351tPwhBkiaQKoqmO/pzOjSncV8SMNVa9ZdzTODWeBy7oxIL1O/n7ZyvcDseUYonEVOi0do2oE0LVW9FUIjT/69wuTbk4txlPf7mK71fZqPdQYonEVCg+xssZ7RsxeXHoVG9F4lLHJjD3DelIVsNEbn1zATv2FbkdjnFYIjHHdU7nJmy33lsmBNSJ9fGPEd0p3FfEne9Yl+BQYYnEHNfp7RuRFOdjwoINbodiDJ0yUrlzUDs+XbrFugSHCEsklRSNfwDFx3gZ2LExnyzabOtEmJBw9UlZnNo2jb98tJS8zXvcDifqWSKpgmisox/arSl7DhXzZd5Wt0MxBo9HeHR4V5LjY7jxjXn2B47LLJGYgJzYugENk+J4f95GV+OIxhKhKVtachx/u7greVv28ODEZW6HE9UskZiA+Lwezu3ShKl5W11fIyL6yoOmPKe2TeNXJ2fx8rR1fBIiXdSjkSUSE7Dzu2dQVHyUyYtDcz13E53uGNiebs3rcufbC1lrU6i4whJJJWkUD4nr2iyVlg3q8IH13jIhJNbn4anLeuD1Cr99ba61l7jAEokJmIgwtFsG36/azhZbH8KEkIy6CTx2cVeWbtrN/R8tdTucqGOJpAqiuY5+SNemqMKHC9xpdI/mEqGp2Bnt0/nNaa15fcYPvD/PSs21yRKJqZQ2jZLolJHCBJcSiTEVua1/W3pl1ufu9xaRv9XGl9QWSySm0s7vlsHCgl2s3OLSBzWai4SmQj6vh39e2p2EGC+/eXUu+4uK3Q4pKlgiMZU2tFsGPo/w9pwCt0Mx5mfSU+J5YkR38rft5e53F9l8XLXAEkkl2XvSPxDs9PaNeGfuBltkyISkftkNufWstrw/fyMvfLfW7XAiniUSUyXDezbjx72H+Cpvm9uhGFOm609vw4CcdB6cuMzWLwmyoCYSERkkInkiki8io8t4PU5E3nRenyEimSVeu8vZniciA51tzUXkCxFZJiJLROSmYMZfLquj5/T2jWiYFMtbc9bX6u+1EqEJlMcjPHZJN7IaJnLD6/Mo2LHf7ZAiVtASiYh4gaeAwUAO8AsRySm12yhgh6q2AR4HxjjH5gAjgI7AIOBp53zFwG2q2gHoA1xfxjlNLYjxeji/WwafL9vK9r2H3A7HmDIlxfkYe0VPDh85yq9fmcOBIhusGAzBLJH0AvJVdbWqFgHjgKGl9hkKvOQ8fxs4U/xT6w4FxqnqIVVdA+QDvVR1k6rOBVDVPcAyICOI12AqMDy3OcVHlffn125XYCsQmspolZbEP0Z0Z+mm3dz1ri2GFQzBTCQZQMl6jwJ+/qX/0z6qWgzsAhoEcqxTDdYdmFHWLxeRa0VktojM3rbN6vGDoV3jZLo0S+Wt2evtw2lC2untG3Fbf3/j+/PfrnE7nIgTzERS1h+Opb9tytunwmNFJAl4B7hZVXeX9ctVdayq5qpqblpaWoAhH599X/6v4bnNWb55D0s2lvnfYEzIuP70Ngzu1JgHJy5j6vItbocTUYKZSAqA5iV+bgaUrgP5aR8R8QGpQGFFx4pIDP4k8pqqvhuUyE3AhnRpSpzPw+szbclTE9pEhL9d3JWcpin87vV5LNtkf/zUlGAmkllAtohkiUgs/sbzCaX2mQCMdJ5fBExVfx3JBGCE06srC8gGZjrtJ88Dy1T1sSDGXiGxWvqfpNaJYUjXprw/bwN7Drq7Tokxx1Mn1sdzV55AcnwMo/4zi602+WiNCFoicdo8bgAm428UH6+qS0TkfhEZ4uz2PNBARPKBW4HRzrFLgPHAUmAScL2qHgFOAq4AzhCR+c7j7GBdgwnM5X1asr/oCO/ZRHkmDDROjee5kbns2H+YX70823py1QBfME+uqhOBiaW23Vvi+UFgeDnHPgA8UGrbt1innZDTtXldOmek8ur0dVzRp2XQ17QP9vlN5OuUkco/ftGda1+Zza3j5/PUpT3weOx9VVU2st3UiMv7tGDFlr3MWrvD7VCMCUj/nHTuObsDnyzezJjJy90OJ6xZIjE1YkjXDJLjfbwyfZ3boRgTsFH9sri8Twv+/dVq6xZcDZZITI1IiPVyUc9mTFq8iW17bKS7CQ8iwp+GdGJQx8b8+aOlfDDf2vmqwhJJFVgVfdku692Sw0eUN4LYFdgGPpqa5vUIfx/RjV5Z9bn9rQV8s9IGMFeWJRJTY9o0SuK0dmm8PG0tBw9bTxgTPuJjvDx7ZS6t05K47pU5LCzY6XZIYcUSialRvzq5FT/uLQpqFYGVCE0wpCbE8NLVvaiXGMtVL84if+tet0MKG5ZITI06sXUDOjRJ4blv1lg1lAk76SnxvHx1L0SEy56bztof97kdUliwRFJJ9uVYMRHh2lOyWLl1L1+usLpmE35apSXx2jW9KSo+yqXPTmd9oa1jcjyWSEyNO7dLUxqnxPPs16vdDsWYKmnXOJlXRvVm76FiLn1uOpt2HXA7pJBmiaQKrIq+YjFeD788KZPvV21nwfqabbS08qCpLZ0yUnllVG927jvMpc/OsHm5KmCJxATFZb1bkJoQwz8+X+l2KMZUWdfmdfnP1SewdfdBLv73NDbstJJJWSyRmKBIjo/hmn5ZfL58K4sKdtXoua1EaGpTz5b1eXlUb7bvK+Lif01jjTXA/4wlEhM0I0/KJCXexxNWKjFhrmfLerzxqz4cOHyEi/89jbzNe9wOKaRYIqkkq6MPXEp8DNec3IrPlm1h8YaaLZUYU9s6ZaTy5rV9EOCSsdNs0GIJlkhMUP3SKZX8/bMVbodiTLVlpyfz1nV9SYrzMWLsdL7I2+p2SCHBEkkV2MjqwKXEx/DrU1vz2bKtTF+9vdrns2E8xm0tGyTyzm9OJKthIte8NDuoc8uFC0skJuhG9cuiSWo8D05cxtGjlglM+EtPiefNX/elX5uG3PXuIv42JS+qBytbIjFBFx/j5fYB7VhYsIsPF26s9vlshUQTCpLifDw3MpcRJzTnn1Pz+d0b89hfVOx2WK6wRGJqxQXdM8hpksLDk/JsjWwTMWK8Hh4a1pnfD2rPx4s2Mezp76NyShVLJJUUxaXXavF4hHvPy2HDzgM8+YV1BzaRQ0T4zWmtefGXJ7Bx5wHOe/Jbvl35o9th1SpLJKbW9GnVgAt7NOPfX61mxRbrh28iy2ntGjHhhn6kJ8dz5QszeOKzlRyJkjZBSyRVIDa2usruOacDyfE+7nlvUZUa3tVG8pgQltkwkXd/eyJDu2Xw+Gcr+MWz09kYBdOqWCIxtap+Yix3n92BWWt38NK0tVU6h6VxE8oS43w8fkk3Hru4K0s27GLwE98wafEmt8MKKkskptZd1LMZZ7ZvxEOfLGf55t1uh2NMUAzr0YyPbzyZlg3qcN2rc7nh9bn8uPeQ22EFhSUSU+tEhDEXdSElPoab3phv67ubiJXZ0D948bb+bZmyZAv9H/uK9+dtiLgxJ5ZIKsnq6GtGw6Q4Hh3ehbwte7j7vUUR98Ey5pgYr4ffnZnNxzf2I6thIje/OZ9Ln53Bsk2RUxq3RGJcc1q7Rtx8Vjbvzt3Ac9+scTscY4LKP0/Xifx5aEeWbd7NOf/4hrvfW8T2CKju8rkdQDiygdU158YzslmxZQ8PfbKMjHoJnN25SYX7W8HFhDOvR7iibybndW3K3z9bySvT1/HBvA388qRMrunXinqJsW6HWCVWIjGu8niER4d3pXuLetz4xjw+XbrluMdYIjfhrm6dWO4b0pHJN5/Mae0b8fSXq+g3ZioPT1oelkv6WiIxrqsT6+PFq06gY0Yqv31tDu/MKXA7JGNqRZtGyTx1aQ8m33wKp7dvxDNfreKkMVO58Y15zFlXGDZth0FNJCIySETyRCRfREaX8XqciLzpvD5DRDJLvHaXsz1PRAYGek4TnlLiY3j56l6ckFmf295awF8/WU5R8VG3wzKmVrRNT+bJS3sw9bbTuKJPJl/kbeXCZ6bR//Gv+cfnK1kb4sv7Bq2NRES8wFNAf6AAmCUiE1R1aYndRgE7VLWNiIwAxgCXiEgOMALoCDQFPhORts4xxztnjbng6e84dPh/v8y27y0Kxq8yQGpCDC9d3Ys/TljCv75axVcrtvGX8zvRs2U9t0MzplZkNUzk3vNyuG1AWz6Yv5H352/gsU9X8NinK8hpksIpbdM4JbshPTPrEefzuh3uT4LZ2N4LyFfV1QAiMg4YCpT80h8K3Oc8fxt4UvxzhA8FxqnqIWCNiOQ75yOAc9aYpqkJHCr1V3FGvQSGdG0ajF9n8HeVfPCCzpzerhF3v7eIC5/5npOzG3JRz2ac2jbN7fCMqRWJcT4u7d2CS3u3YOPOA3y0cCOfL9vKc9+s5l9frSLW66F9k2Q6Z6SS0zSFFvXr0LxeHZrWTSDWV/stFsFMJBnA+hI/FwC9y9tHVYtFZBfQwNk+vdSxGc7z452zxjx1WY9gndocR/+cdE5s3YCXpq3l5e/XcdO4+T+9lhJvnQ1N9GhaN4FrT2nNtae0Zu+hYqat2s7stYUsLNjFhAUbeW3G/67QmBDjJTneR1Kcj4ZJcYy/rm/QYwzmJ7KsvjWlW47K26e87WWl2jJbo0TkWuBagBYtWpQfpQlZiXE+fntaG647pTWz1+1g7g87WLV1L+2bpLgdmjGuSIrz0T8nnf456QAcPaps2n2Q9YX7+aFwP5t2HmTvocPsOVjM3kPFJMXVzh9dwfwtBUDzEj83A0ovj3dsnwIR8QGpQOFxjj3eOQFQ1bHAWIDc3Nzw6PpgyuTxCL2y6tMrq77boRgTUjweIaNuAhl1E+jTqoF7cQTx3LOAbBHJEpFY/I3nE0rtMwEY6Ty/CJiq/v5uE4ARTq+uLCAbmBngOY0xxtSioJVInDaPG4DJgBd4QVWXiMj9wGxVnQA8D7ziNKYX4k8MOPuNx9+IXgxcr6pHAMo6Z7CuwRhjzPFJuAx4qY7c3FydPXu222EYY0zYEJE5qpobyL42st0YY0y1WCIxxhhTLZZIjDHGVIslEmOMMdViicQYY0y1REWvLRHZBqyr4uENgR9rMJxwFO33INqvH+weQPTdg5aqGtAEd1GRSKpDRGYH2gUuUkX7PYj26we7B2D3oCJWtWWMMaZaLJEYY4ypFkskxzfW7QBCQLTfg2i/frB7AHYPymVtJMYYY6rFSiTGGGOqxRJJOURkkIjkiUi+iIx2O55gEpG1IrJIROaLyGxnW30R+VREVjr/1nO2i4j8w7kvC0UkLJeRFJEXRGSriCwusa3S1ywiI539V4rIyLJ+V6gq5x7cJyIbnPfCfBE5u8Rrdzn3IE9EBpbYHpafFRFpLiJfiMgyEVkiIjc526PqfVAjVNUepR74p6hfBbQCYoEFQI7bcQXxetcCDUttexgY7TwfDYxxnp8NfIJ/Fcs+wAy346/iNZ8C9AAWV/WagfrAauffes7zem5fWzXvwX3A7WXsm+N8DuKALOfz4Q3nzwrQBOjhPE8GVjjXGVXvg5p4WImkbL2AfFVdrapFwDhgqMsx1bahwEvO85eA80tsf1n9pgN1RaSJGwFWh6p+jX8NnJIqe80DgU9VtVBVdwCfAoOCH33NKOcelGcoME5VD6nqGiAf/+ckbD8rqrpJVec6z/cAy4AMoux9UBMskZQtA1hf4ucCZ1ukUmCKiMxx1roHSFfVTeD/wAGNnO2RfG8qe82Rei9ucKpuXjhWrUOE3wMRyQS6AzOw90GlWSIpm5SxLZK7t52kqj2AwcD1InJKBftG272B8q85Eu/FM0BroBuwCfibsz1i74GIJAHvADer6u6Kdi1jW0Tcg+qyRFK2AqB5iZ+bARtdiiXoVHWj8+9W4D381RVbjlVZOf9udXaP5HtT2WuOuHuhqltU9YiqHgWexf9egAi9ByISgz+JvKaq7zqbo/59UFmWSMo2C8gWkSwRicW/lvwEl2MKChFJFJHkY8+BAcBi/Nd7rPfJSOAD5/kE4EqnB0sfYNexaoAIUNlrngwMEJF6ThXQAGdb2CrV3nUB/vcC+O/BCBGJE5EsIBuYSRh/VkREgOeBZar6WImXov59UGlut/aH6gN/D40V+Huk3ON2PEG8zlb4e9osAJYcu1agAfA5sNL5t76zXYCnnPuyCMh1+xqqeN1v4K+6OYz/L8pRVblm4Gr8Dc/5wFVuX1cN3INXnGtciP+Ls0mJ/e9x7kEeMLjE9rD8rAD98FdBLQTmO4+zo+19UBMPG9lujDGmWqxqyxhjTLVYIjHGGFMtlkiMMcZUiyUSY4wx1WKJxBhjTLVYIjHmOETke+ffTBG5tIbPfXdZv8uYcGLdf40JkIichn9m3HMrcYxXVY9U8PpeVU2qifiMcYuVSIw5DhHZ6zz9K3Cys07HLSLiFZFHRGSWM8nhr539T3PWuXgd/8A1ROR9Z1LMJccmxhSRvwIJzvleK/m7nNHTj4jIYvGvFXNJiXN/KSJvi8hyEXnNGaGNiPxVRJY6sTxam/fIRDef2wEYE0ZGU6JE4iSEXap6gojEAd+JyBRn315AJ/VPuQ5wtaoWikgCMEtE3lHV0SJyg6p2K+N3DcM/cWJXoKFzzNfOa92Bjvjnc/oOOElEluKf0qS9qqqI1K3xqzemHFYiMabqBuCfe2k+/unHG+CfgwpgZokkAnCjiCwApuOf4C+bivUD3lD/BIpbgK+AE0qcu0D9EyvOBzKB3cBB4DkRGQbsr/bVGRMgSyTGVJ0Av1PVbs4jS1WPlUj2/bSTv23lLKCvqnYF5gHxAZy7PIdKPD8C+FS1GH8p6B38CzFNqtSVGFMNlkiMCdwe/EuyHjMZ+I0zFTki0taZQbm0VGCHqu4Xkfb4l2k95vCx40v5GrjEaYdJw78s7szyAnPW1EhV1YnAzfirxYypFdZGYkzgFgLFThXVf4An8FcrzXUavLfx32VZS5oEXCciC/HPnDu9xGtjgYUiMldVLyux/T2gL/5ZmRW4U1U3O4moLMnAByISj780c0vVLtGYyrPuv8YYY6rFqraMMcZUiyUSY4wx1WKJxBhjTLVYIjHGGFMtlkiMMcZUiyUSY4wx1WKJxBhjTLVYIjHGGFMt/w8NfPNSTC3g+AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_text=False,show_moms=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks familiar?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1cycle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Leslie Smith's 1cycle policy states to pick a maximum learning rate with our traditional learning rate finder, choose a factor div by which to divide this learning rate, then do two phases of equal length going linearly from our minimal lr to our maximal lr, then back to the minimum.\n",
    "\n",
    "In parallel, the momentum should begin high (like 0.95) and decrease to a minimum (like 0.85) as the lr grows, then returns to the maximum as the lr decreases.\n",
    "\n",
    "To complete this cycle, allow a bit of time to let the learning rate decrease even more at the end (I chose to go from the minimum_lr to 1/100th of its value linearly since it was the thing that seemed to work the best in my experiments)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's create a function 1cycle that takes the arguments:\n",
    "- cycle_len: the total length of the cycle (in epochs)\n",
    "- lr: the maximum learning rate\n",
    "- div: by how much do we want to divide the maximum lr\n",
    "- pct: what percentage of epochs should be left at the end for the final annealing\n",
    "- max_mom: the maximum momentum\n",
    "- min_mom: the minimum momentum\n",
    "\n",
    "The optim function should be SGD (with momentum)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "def phases_1cycle(cycle_len,lr,div,pct,max_mom,min_mom):\n",
    "    tri_cyc = (1-pct/100) * cycle_len\n",
    "    return [TrainingPhase(epochs=tri_cyc/2, opt_fn=optim.SGD, lr=(lr/div,lr), lr_decay=DecayType.LINEAR, \n",
    "                          momentum=(max_mom,min_mom), momentum_decay=DecayType.LINEAR),\n",
    "           TrainingPhase(epochs=tri_cyc/2, opt_fn=optim.SGD, lr=(lr,lr/div), lr_decay=DecayType.LINEAR, \n",
    "                          momentum=(min_mom,max_mom), momentum_decay=DecayType.LINEAR),\n",
    "           TrainingPhase(epochs=cycle_len-tri_cyc, opt_fn=optim.SGD, lr=(lr/div,lr/(100*div)), lr_decay=DecayType.LINEAR, \n",
    "                          momentum=max_mom)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "197e39621b4742f687786efd3f43cfc3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.604487   4.53111    0.1389    \n",
      "    1      3.739034   4.621771   0.1313                                                                                \n",
      "    2      3.565503   4.512865   0.1425                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.51287]), 0.1425]"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases_1cycle(3, 1e-2, 10, 10, 0.95, 0.85))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuAAAAEKCAYAAABT6eBwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl8VPXVx/HPyU4gYQlhXxIgLCGBoIgKuKICigRQAbtpq9UuVgWXsomALFpBsFXbx2prfdqngsiqCCKC+wYSCEkIhLCFNeyQkI2c548ZbIoBAmTmzkzO+/W6L2bu3Dv5Bs1wMvO754iqYowxxhhjjPGOIKcDGGOMMcYYU5NYAW6MMcYYY4wXWQFujDHGGGOMF1kBbowxxhhjjBdZAW6MMcYYY4wXWQFujDHGGGOMF1kBbowxxhhjjBdZAW6MMcYYY4wXWQFujDHGGGOMF4U4HcAbGjZsqHFxcU7HMMaYC7ZmzZoDqhrrdA5vstdsY4y/quprdo0owOPi4li9erXTMYwx5oKJyHanM3ibvWYbY/xVVV+zbQmKMcYYY4wxXmQFuDHGGGOMMV5kBbgxxhhjjDFeZAW4McYYY4wxXmQFuDHGGGOMMV7k0QJcRPqJSLaI5IjIqEoeDxeR2e7HvxaROPf+GBFZKSInROSlM865XETS3ef8UUTEk9+DMcYYY4wx1cljBbiIBAMvA/2BROBuEUk847D7gMOq2g6YCTzn3l8EPAU8XslT/xl4AEhwb/2qP70xxhhjjDGe4ck+4D2AHFXNBRCRt4BUILPCManABPftucBLIiKqWgB8JiLtKj6hiDQFolX1S/f9N4FBwPse/D5MDbBm+2HKVbkiroHTUYwxF6iguIw/rtjMPT3jaFavltNxjDEesCJrH+t2HvHq13zwurbUDvdMqezJArw5sLPC/TzgyrMdo6plInIUiAEOnOM58854zuaVHSgiD+B6p5xWrVpdaHZTg2w7UMBPX/8aVXjv4d60ia3jdCRjzAU4VFDCm19uZ0t+AX/92eXYykRjAsvaHYe5/83VqII3f7x/enWcXxbglf0V6UUcc1HHq+qrwKsA3bt3P9dzmhqs7FQ5I+akERwkBAcJI2anMffXPQkNtuuTjfEXLRtEMvLm9kxZksWyjL30S2rqdCRjTDUpPVXO6HnpNI6KYPnIa4mKCHU6UrXwZJWRB7SscL8FsPtsx4hICFAXOHSe52xxnuc0pspeWbWFtTuOMHlQElMHJ7Mu7yh/+ijH6VjGmAv0815xdG4WzfiFGRwrKnU6jjGmmrz+2VY27j3OxNTOAVN8g2cL8G+BBBGJF5EwYDiw6IxjFgH3uG/fCXykqmd9t1pV9wDHReQqd/eTnwELqz+6qQnW7TzCiys2M7BrM1JTmnNrclOGXNacl1fm8N2Ow07HM8ZcgJDgIKYNSebAiWKeX5rtdBxjTDXYcbCQWR9uom/nxvTt3MTpONXKYwW4qpYBDwHLgCxgjqpmiMgkERnoPux1IEZEcoCRwPetCkVkG/ACcK+I5FXooPJr4DUgB9iCXYBpLkJhSRkjZqfRKCqcZ1KTvt8/YWBnmkRHMGJ2GgXFZQ4mNMZcqC4t6nFvz3j++fV21my3X6KN8WeqytgF6YQEBTFxYNL5T/AzHl3oqqpLVLW9qrZV1SnufeNVdZH7dpGq3qWq7VS1x+mOKe7H4lS1garWUdUWqprp3r9aVZPcz/nQud4xN+Zspi7JIvdAATPu6krdyP98pBUdEcrMYSnsOFTI5Pcyz/EMxhhf9Ngt7WkaHcGYeemUnip3Oo4x5iItTNvNp5sP8GS/DjSpG+F0nGpnV5qZGmflxv3886sd3N87np7tGv7g8R7xDXjw2rb8+5udLM/c50BCY8zFqh0ewqTUJLL3HefVT3LPf4IxxuccKSzhmXczSWlZjx9f2drpOB5hBbipUQ6eKOaJuevp2CSKx/t2OOtxI29uT2LTaEa9s57848VeTGiMuVQ3JTbm1uQm/HHFZrYdKHA6jjHmAk1dksXRk6VMG5JMcFBgthW1AtzUGKrK6HnpHDtZysxhKUSEBp/12LCQIGYNT+F4cRm/f2c9ttLJGP8y4fbOhIUEMXZBuv38GuNHvtxykDmr8/jltW3o1DTa6TgeYwW4qTHeXp3HB5n7eLxv+yr9ULdvHMXo/h35aON+/u+bHV5IaIypLo2iI/h9v458nnOQ+Wt3OR3HGFMFRaWnGDs/nVYNInmkT4LTcTzKCnBTI+w4WMjExRlc1aYB9/duU+Xz7rk6jmsSGjL53Sxy8094MKExprr9qEcrLm9dn2fezeRQQYnTcYwx5/HKqi3kHihgyuCkc35KHQisADcB7/S0y6AgYcbQFIIuYD1ZUJDw/J1dCQsJYsScddZVwRg/EhQkTBuSzIniMqa8l+V0HGPMOWzed5w/r8phcLfmXJMQ63Qcj7MC3AS8v3y8hTXbD/NMahLN69W64POb1I1wTcnceYSXbEqmMX6lfeMoHry2Le98l8cXOQecjmOMqUR5uTJmfjq1w0MYd1snp+N4hRXgJqCtzzvCrA83M6BLU1JTml3089zWpSlDujXnpZU5rLUpmcb4lYdubEdcTCRj5qdTVHrK6TjGmDO89e1Ovt12mDG3diKmTrjTcbzCCnATsE6WnOLR2Wk0rBPOlEHJiFxaK6MJqTYl0xh/FBEazNTByWw7WGifYhnjY/YfL2La+1lc1aYBd13ewuk4XmMFuAlY097PIje/gBlD/3va5cWKjgjlhaFd2X6okMm2ntQYv9KzXUPuuKwFf/l4C9l7jzsdxxjjNmlxJsVl5UwdfOlvlPkTK8BNQFqVvZ83v9zOL3rF06uSaZcX68o2MTxwbRv+/c0OPrQpmaaGEJF+IpItIjkiMqqSx1uLyAoRWS8iq0SkxRmPR4vILhF5yXupf2jsbZ2IighhzPx0ysutN7gxTlu5cT/vrt/D725oR5vYOk7H8SorwE3AOVRQwhNz19O+cR2e7Hf2aZcXa+TNrj7iv7cpmaYGEJFg4GWgP5AI3C0iiWccNh14U1W7AJOAaWc8/gzwsaeznk+D2mE8NSCRNdsPW29/YxxWWFLGuAUbSGhUhweva+t0HK+zAtwEFFVlzLx0jhSWMGtYN4/0EQ0PCeZF95TMUTYl0wS+HkCOquaqagnwFpB6xjGJwAr37ZUVHxeRy4HGwAdeyHpeg7s1p1e7GJ57fyP7jhU5HceYGmvm8k3sOnKSqUOSCQupeeVozfuOTUCbuyaPpRl7eeyWDiQ289wI2/aNo/h9v46s2Liff3+z02Nfxxgf0Byo+D95nntfReuAO9y3BwNRIhIjIkHADOAJj6esIhFhyqBkSk6VM3FxhtNxjKmRNuw6yuufbeVHV7biirgGTsdxhBXgJmDsPFTIxMWZ9IhvwC+vqfq0y4v1855x9GoXwzPvZrL1QIHHv54xDqnsqqgzP/Z5HLhORNYC1wG7gDLgN8ASVT3vb6ki8oCIrBaR1fn5+Zea+ZziGtbm4T4JLEnfa9dyGONlZafKGT0vnZg64fy+X0en4zjGCnATEE6VKyNmpyHAC0O7EnwB0y4vVlCQMP2uroQGCyNmp1FmUzJNYMoDWla43wLYXfEAVd2tqkNUtRsw1r3vKHA18JCIbMO1TvxnIvJsZV9EVV9V1e6q2j021vNT8B64tg0dGkcxfuEGaytqjBf948vtpO86ytO3J1K31qV3KPNXVoCbgPCXj7ewevthJqZ2pkX9SK993aZ1azF1SDJpO4/w0krrL2wC0rdAgojEi0gYMBxYVPEAEWnoXm4CMBr4G4Cq/lhVW6lqHK53yd9U1R90UXFCaHAQU4cks+dYETM+2OR0HGNqhF1HTjLjg2xu7NiI25KbOh3HUVaAG7+3YddRZi7fxG3JTRnc7cylqZ43oEszBndrzp8+simZJvCoahnwELAMyALmqGqGiEwSkYHuw64HskVkE64LLqc4EvYCXd66Pj+5sjVvfLGV9XlHnI5jTEBTVZ5asAFVmJTauUb1/K6MFeDGrxWVuqZdxtQJY8rgJMd+oCe6p2SOnLOOwhL7ONsEFlVdoqrtVbWtqk5x7xuvqovct+eqaoL7mPtV9Qf9OVX1DVV9yNvZz+eJfh1oWCecUe+k2zIyYzxoSfpePtq4n8duae/VT6p9lRXgxq89+/5GcvafYPpdXakXGeZYjuiIUGYM7cq2gwU2JdMYPxIdEcqk1M5k7jnG3z/f5nQcYwLS0ZOlTFicQVLzaO7tGed0HJ9gBbjxW59syueNL7Zxb884rknw/EVb53NVmxgeuKYN//f1DlZkWWcFY/xF385NuKlTY15YvomdhwqdjmNMwPnD0o0cPFHMs0O6EBJspSdYAW781OGCEh5/ex0Jjeowqr/vtDEaect/pmQeOGFTMo3xByLCpNTOBAk8tXCDDdcyphqt3naIf329g1/0iiepeV2n4/gMK8CN31FVxi5I53BhCTOHpXhk2uXFCg8JZtawFI4VlTHqnXT7h9wYP9GsXi0e79uBVdn5vLt+j9NxjAkIJWWunt/N69VixM3tnY7jU6wAN35n3ne7WJK+lxE3t/fJ36Y7NIniyb4d+DBrH7O/tSmZxviLn10dR9cWdZm4OJOjhaVOxzHG7736yRY27z/BM4M6Uzs8xOk4PsUKcONXdh4q5OlFGfSIa8CD17Z1Os5Z/aJXPL3axTDp3Uy22ZRMY/xCcJAwdUgyhwtLeHapXUxtzKXIzT/BHz/K4bYuTbmxY2On4/gcK8CN3zhVrjw2Zx0AM7w07fJinZ6SGRIkPGpTMo3xG52b1eX+3vH8+5udfLP1kNNxjPFLqsrY+RsIDwni6dsTnY7jk6wAN37j1U9y+WbbISYM7EzLBr7fQ7Rp3VpMHuyakvnyyi1OxzHGVNEjNyXQon4tRs9bT3HZKafjGON35q7J48vcg4zq35FGURFOx/FJVoAbv7Bh11FeWJ5N/6Qm3HGZ96ddXqyBXZuRmtKMP360mbSdNmnPGH8QGRbC5EFJbMkv4C+rcp2OY4xfOXiimClLsujeuj53X9HK6Tg+ywpw4/OKSk8xYnYa9SPDmDo42e/G105KTaJxVDgjZqfZlExj/MT1HRoxsGszXl6Zw5b8E07HMcZvTHkvi4LiMqYNSSbIh5eKOs0KcOPznlu6kc37T/D8XV2pX9u5aZcXq26tUKa7p2ROsSmZxviNpwYkEhEaxJh51lLUmKr4dHM+89bu4lfXtSWhcZTTcXyaRwtwEeknItkikiMioyp5PFxEZrsf/1pE4io8Ntq9P1tE+lbYP0JEMkRkg4j8W0RscVEA+3RzPn//fBv3XN2a69o7P+3yYvVs25BfXtOGf329g4822pRMY/xBbFQ4Y2/rxNdbD/H26jyn4xjj04pKTzFuwQbiG9bmtze0czqOz/NYAS4iwcDLQH8gEbhbRM68FPY+4LCqtgNmAs+5z00EhgOdgX7AKyISLCLNgYeB7qqaBAS7jzMB6Eiha9pl29jajOrfyek4l+yxW9rTsUkUT85N56BNyTTGLwzt3pIe8Q2YsiTLptsacw5/XLGZ7QcLmTI4yacG5PkqT74D3gPIUdVcVS0B3gJSzzgmFfiH+/ZcoI+4FvimAm+parGqbgVy3M8HEALUEpEQIBLY7cHvwTjENe1yAwdPlPDi8G7UCvP/H+bwkGBmDU/h2MlSRtlH2sb4BRFh6uBkTpac4pl3M52OY4xP2rj3GK9+kstdl7egZ9uGTsfxC54swJsDFccA5rn3VXqMqpYBR4GYs52rqruA6cAOYA9wVFU/8Eh646gFabt4b/0en512ebE6NonmyX4dWJ65jzmrbUqmMf6gXaM6/OaGtixM283Hm/KdjmOMTykvV0bPSye6VihjbvX/T6u9xZMFeGWXvp75lt/Zjql0v4jUx/XueDzQDKgtIj+p9IuLPCAiq0VkdX6+vWD6k7zDhYxfkEH31vX51XW+O+3yYv2iVzw928YwcXEm2w/alExj/MGvr29L29jajFuQzskS6w1uzGn/+no7a3cc4akBnfyyUYJTPFmA5wEtK9xvwQ+Xi3x/jHtJSV3g0DnOvQnYqqr5qloKzAN6VvbFVfVVVe2uqt1jY/334r2a5vS0y3JVZg5L8elplxer4pTMETYl0xi/EB4SzNTByew8dJJZKzY5HccYn7D3aBHPLc3mmoSGDErxnxkdvsCTBfi3QIKIxItIGK6LJRedccwi4B737TuBj9S1MHYRMNzdJSUeSAC+wbX05CoRiXSvFe8DWF+3APLap7l8vfUQT/vJtMuL1axeLZ4ZlMR3O47w51U2JdMYf3BlmxiGX9GS1z7dSubuY07HMcZxExZlUHqqnMmDkvxuRofTPFaAu9d0PwQsw1Ukz1HVDBGZJCID3Ye9DsSISA4wEhjlPjcDmANkAkuB36rqKVX9GtfFmt8B6e78r3rqezDelbn7GNM/yKZv58bcdXkLp+N4XGpKcwZ2bcaLKzazzqZkGuMXRvfvRP3IUEbPW8+pcruQ2tRcH2TsZWnGXh65KYHWMbWdjuN3pCZ0YujevbuuXr3a6RjmHIpKTzHwpc84XFjKskevpUENWUd2tLCUfi9+Qq3QYN59uDeRYSFORzI+RkTWqGp3p3N4k6+/Zi9at5uH/72WCbcncm+veKfjGON1J4rLuPmFj6lbK5TFv+tNaLDNdTytqq/Z9jdmfMLzy7LZtO8Ef7izS40pvgHqRoYy466u5B4oYOoSW01ljD+4vUtTrmsfy/PLstl95KTTcYzxuunLstl7rIhpQ5Kt+L5I9rdmHPd5zgFe/2wrP72qNTd0aOR0HK/r2a4h9/eO559f7WDlxv1OxzHGnIeIMHlQEqdUeXpRhtNxjPGqtJ1H+MeX2/jpVa3p1qq+03H8lhXgxlFHC0t5bM462sTWrtH9Qx/v24GOTaJ4Yu56m5JpjB9o2SCSkTe3Z3nmPpZu2Ot0HGO8ovRUOaPnpdM4KoIn+nZwOo5fswLcOGrcwg0cOFHMrGEpATHt8mJFhAYzc5hrSuZom5JpjF/4Ra94EptG8/SiDRwvKnU6jjEe97fPtpK15xgTBnYmKiLU6Th+zQpw45iFabtYvG43j/RJoEuLek7HcVynptE80bcDH2Tu4+3VeU7HMcacR0hwENOGJJN/vJjnl2U7HccYj9p5qJCZH27ilsTG9Etq4nQcv2cFuHHEriMnGbdgA5e1qsevrw+8aZcX677e8VzdJoaJizPYcbDQ6TjGmPPo2rIe9/SM43+/2s53Ow47HccYj1BVxi7YQEhQEBNTOzsdJyBYAW68rrxceXzOOsrLXdMuQ+wK6u8FBQkzhnYlKEgYMcemZBrjDx67pQNNoiMY/U46pfYzawLQonW7+WRTPo/f0p6mdWs5HScgWOVjvO71z7byZe5Bxt+eaM37K9GsXi0mD0pizfbD/OVjm5JpjK+rEx7CM6lJZO87zl8/zXU6jjHV6khhCZMWZ9K1ZT1+enWc03EChhXgxquy9hzj+WXZ3JLYmKHdWzodx2elpjTn9q7NmPXhZtbn2ZRMY3zdTYmN6Z/UhBc/3Mz2gwVOxzGm2kxbspEjJ0t5dkgywUE2br66WAFuvKa47BQjZqcRXSuUaUOSEbEf5HOZnJpEbFQ4j85O42TJKafjGGPOY8LAzoQFBzF2/gbrZGQCwle5B5m9eif3XxNPp6bRTscJKFaAG6+Z8cEmNu49zh/uTCamTrjTcXxe3chQpt/Vldz8Aqa9b1MyjfF1jaMjeLJ/Rz7LOcCCtF1OxzHmkhSXnWLM/HRaNqjFo33aOx0n4FgBbrziyy0H+eunufz4ylbc2LGx03H8Rq92Dbmvdzxvfrmdldk2JdM4Q0T6iUi2iOSIyKhKHm8tIitEZL2IrBKRFu79KSLypYhkuB8b5v303vXjHq24rFU9nnk3i8MFJU7HMeaivbJyC7n5BUwZlFyj53R4ihXgxuOOnizlsTlpxMXUZuxtNXfa5cV6om8HOjSO4sm56zlk/6AbLxORYOBloD+QCNwtIolnHDYdeFNVuwCTgGnu/YXAz1S1M9APmCUiAd30PyhImDakC8dOljJliX1yZfxTzv7jvLIqh0Epzbi2fazTcQKSFeDG455euIF9x4uZOSyFyLAQp+P4ndNTMo8WljJ63npbW2q8rQeQo6q5qloCvAWknnFMIrDCfXvl6cdVdZOqbnbf3g3sBwL+X/MOTaJ48Lo2zF2Txxc5B5yOY8wFKS9XxszbQO3wEMYNOPN3bVNdrAA3HrV43W4WpO3m4RsTSGkZ0G98eVRis2geu6U9yzL28fYam5JpvKo5sLPC/Tz3vorWAXe4bw8GokQkpuIBItIDCANqRG/N392YQFxMJGMXbKCo1C6iNv5jzuqdfLPtEGP6d6KhXa/lMVaAG4/Zc/QkY+en061VPX57g027vFT3X9OGK+MbMHGRTck0XlVZu6IzP4Z5HLhORNYC1wG7gLLvn0CkKfC/wM9VtdJJNSLygIisFpHV+fn51ZPcQRGhwUwZnMzWAwW8vDLH6TjGVMn+40VMXZLFVW0acFf3Fk7HCWhWgBuPKC9XHn97HWXlysyhNu2yOgQHCS8MSyEoSBg5J41T5bYUxXhFHlCxaX8LYHfFA1R1t6oOUdVuwFj3vqMAIhINvAeMU9WvzvZFVPVVVe2uqt1jYwNjlUqvdg0Zcllz/vLxFjbtO+50HGPO65l3sygqLWfKYGsV7GlWFRmP+PsX2/g85yBPDUgkrqFNu6wuzevV4pnUJFbblEzjPd8CCSISLyJhwHBgUcUDRKShiJz+92Q08Df3/jBgPq4LNN/2YmafMe62ROqEhzB6Xjrl9kuz8WErs/ezeN1ufntDO9rG1nE6TsCzAtxUu+y9x3lu6UZu6tSY4VfYtMvqlprSjAFdmjJz+SbS8446HccEOFUtAx4ClgFZwBxVzRCRSSIy0H3Y9UC2iGwCGgNT3PuHAtcC94pImntL8e534KwGtcMYd1sia7Yf5t/f7nA6jjGVKiwpY9z8DbRrVIdfXd/G6Tg1ghXgploVl53i0dlpREeE8Owd9hGWJ4gIUwYl07BOOI/OXmtTMo3HqeoSVW2vqm1VdYp733hVXeS+PVdVE9zH3K+qxe79/1TVUFVNqbClOfm9OGHIZc3p1S6GZ9/fyP5jRU7HMeYHZn24mV1HTjJtSDLhIdbz2xusADfV6oXlm8jac4zn7uhiV097UN3IUGYM7cqW/AKetSmZxvg0EWHyoGSKy8qZuDjT6TjG/JcNu47y+mdbubtHS66Ia+B0nBrDCnBTbb7KPcirn+Ryd49W9Olk0y49rVe7hvyiVzz/+HI7H2/y/64RxgSy+Ia1eaRPAu+l7+GjjfucjmMMAKfKlTHz06kfGcaofjYoz5usADfV4lhRKY/NWUfrBpGMs2mXXvNkvw60b1yHJ95eZ2OvjfFxv7ymDe0b1+GpBRkUFJed/wRjPOwfX2xjfd5Rnr49kbqRoU7HqVGqXICLiLWyMGc1YWEGe48VMXNYCrXDbdqlt0SEBjNrWDcOF5Ywel66Tck0xoeFhQQxbUgyu46c5IXlm5yOY2q4XUdOMv2DbG7oEMuALk2djlPjnLcAF5GeIpKJ6+p3RKSriLzi8WTGb7y3fg/z1u7ioRva0a1Vfafj1DiuKZkdWJqxl7k2JdMYn3Z56wb85KpW/P3zrdbFyDhGVRm/YAOqMCk1yRomOKAq74DPBPoCBwFUdR2utlLGsPdoEWPmp9O1ZT0eurGd03FqrF9e04Ye8Q2YuDiTnYdsSqYxvuzJfh1pWCecUfPWU3aq0sGgxnjU+xv2smLjfkbe3J6WDSKdjlMjVWkJiqruPGOX9T0zlJcrT8xdR0lZObOGpRBq0y4dExwkvDC0KwKMmG1TMo3xZdERoUwY2JmM3cd444ttTscxNcyxolImLMqgc7Noft4rzuk4NVZVKqadItITUBEJE5HHcS9HMTXbP77cxqebDzBuQCfibdql41rUj2RiamebkmmMH+if1ISbOjVixgeb7FMr41V/WLqRAyeKeXZIF0LsjTPHVOVv/lfAb4HmQB6QAvzGk6GM79u87zjPvr+RPh0b8aMerZyOY9wGd2vObcmuKZkbdtn6UvPfRKS7iMwXke9EZL2IpIvIeqdz1UQiwsTUJERg/MINdgG18Yo12w/xz692cG/PeJJb1HU6To1WlQK8g6r+WFUbq2ojVf0JYH3marCSsnIeeSuNOuEhPHtHF7t4w4eICFMGJxFTJ4xHZ6dRVGqrxcx/+Rfwd+AO4HZggPtP44Dm9Wrx+C0dWJmdz3vpe5yOYwJcSVk5o+el07xeLR67pb3TcWq8qhTgf6riPlNDzPxwE5l7jvHsHV2IjbJpl76mXmQYM+5KIWf/CZ59f6PTcYxvyVfVRaq6VVW3n96cDlWT3dMzji4t6jJhUSZHC0udjmMC2F8/zWXTvhNMSu1s7YJ9wFkLcBG5WkQeA2JFZGSFbQIQXJUnF5F+IpItIjkiMqqSx8NFZLb78a9FJK7CY6Pd+7NFpG+F/fVEZK6IbBSRLBG5+gK+X3OJvtl6iL98vIXhV7Tk5kSbdumreic05Oe94njji218YlMyzX88LSKvicjdIjLk9OZ0qJosOEiYOjiZw4UlPLvUfmE2nrH1QAEvrtjMrclNbFK1jzjXO+BhQB0gBIiqsB0D7jzfE4tIMPAy0B9IBO4WkcQzDrsPOKyq7XC1O3zOfW4iMBzoDPQDXnE/H8CLwFJV7Qh0xS4I9ZrjRaWMmJ1GqwaRPDXgzP+Uxtf8vl9HEhrV4XGbkmn+4+e4ruPph2vpyellKMZBSc3rcl/veP79zQ6+3XbI6TgmwKgqY+enEx4SxITbOzsdx7id9TMIVf0Y+FhE3rjIjyh7ADmqmgsgIm8BqUBmhWNSgQnu23OBl8S1oDgVeEtVi4GtIpID9BCRDFw9yO91ZywBrLLwkgmLMtlz9CRv/6qnfXzlByJCg5k1PIVBL3/O2AXpvPyjy2y9vumqqslOhzA/9OhNCby3fg+j56Xz3sO9CQ+p0gfNxpzXvO928cWWg0welESj6Aipyr1PAAAgAElEQVSn4xi3qqwBLxSR50VkiYh8dHqrwnnNgYr9w/Pc+yo9RlXLgKNAzDnObQPkA38XkbXuj1Ir7X8nIg+IyGoRWZ2fbx/BX6r30/fwznd5PHRDOy5vbdMu/UXnZnUZeXMHlqTvZd53u5yOY5z3VSWfRBofEBkWwuTBSeTsP8H/fJzrdBwTIA4VlDD5vUwub13fOpb5mKoU4P8CNgLxwERgG/BtFc6r7K22M/ssne2Ys+0PAS4D/qyq3YAC4AdrywFU9VVV7a6q3WNjY6sQ15zNvmNFjJ6fTtcWdfldnwSn45gL9MC1begR14CnF2VYv2HTG0hzX1tjbQh9zA0dGnF712a89FEOW/JPOB3HBIDJ72VyoriMaUOSCQqyT0B9SVUK8BhVfR0oVdWPVfUXwFVVOC8PaFnhfgtg99mOEZEQoC5w6Bzn5gF5qvq1e/9cXAW58RBV5Ym56ykqPcULNu3SLwUHCTOGdgVg5BybklnD9QMSgFuwNoQ+afyARCJCgxg7P916g5tL8tnmA8z7bhcPXtuW9o2jnI5jzlCVaup0X6Q9InKbiHTDVRCfz7dAgojEi0gYrosqF51xzCLgHvftO4GP1PWKswgY7u6SEo/rH4xvVHUvrsmcHdzn9OG/15Sbavbml9v5ZFM+Y29LpG1sHafjmIvUskEkEwd25ttth/mfT2xKZg2mZ9mMj4iNCmfMrZ34KvcQb6/JczqO8VNFpacYuyCd+Ia1eejGdk7HMZWoypV0k0WkLvAYrv7f0cCI852kqmUi8hCwDFfbwr+paoaITAJWq+oi4HXgf90XWR7CVaTjPm4OruK6DPitqp6eKPI74F/uoj4X11X9xgNy9h9n6pIsbugQy0+utLVj/m7IZc1ZsXEfM5dv4tqEWJKa2xS0Gug9/rPMLwLX0sJsXB2njI8Y2r0l877bxdQlWdzYsREN69i8BXNh/vTRZrYfLOT/7r+SiFC7oNcXybk+4nK3/ntYVWd6L1L16969u65evdrpGH6lpKycIX/+nN1Hilj66DU0irIrpwPB4YIS+s76hOhaobz7u972wuwHRGSNqnb30HNfBjyoqg964vkvlr1mu94A6f/ip9yW3JRZw7s5Hcf4kY17jzHgj5+RmtL8++WHxnuq+pp9ziUo7nedB1ZbKuM3XlyxiQ27jjFtSLIV3wGkfu0wpt/V1aZkGgBU9TvgCqdzmB9q1yiK31zfjgVpu/l0s3XyMlVTXq6MmZdOdK1Qxt7Wyek45hyqsgTlCxF5CZiNq+sI8P0LtwlAq7cd4s+rtjC0ewv6dm7idBxTza5tH8u9PV1TMvt0asQ1CdYlqKYQkZEV7gbhuojdqjsf9Zsb2rJ4/W7Gzt/AskevpVaYfWJlzu1f3+zgux1HeGFoVxrUDnM6jjmHqlyE2RPX+sBJwAz3Nt2ToYxzjheVMmJOGi3qRzLeJmYFrFH9O9LOPSXzSKHNsqpBKk41Dse1JjzV0UTmrMJDgpk6OJkdhwp5ccVmp+MYH7fvWBF/eH8jvdrFMLjbmWNXjK857zvgqnqDN4IY3zBpcSa7Dp/k7V9dTR2bdhmwIkKDmTUshcGvfM7Y+Rt46UfdbEpmzZCpqm9X3CEidwFvn+V447Cr2sQwrHtL/vppLqkpzejUNNrpSMZHTViUQcmpcqYMSrbXcz9gTZ3N95Zu2MPba/L4zfXtuLx1A6fjGA9Lal6XETe35730Pcxfa1Mya4jRVdxnfMjoWztSPzKUUfPSrY+/qdTyzH28v2EvD/dJIK5hpQPCjY+xAtwAsP9YEaPnpZPcvC6P3GTTLmuKB69t65qSuTCDvMM2JTNQiUh/EfkT0FxE/lhhewNXq1fjw+pFhvHUgETW7TzCP7/a7nQc42NOFJcxfuEGOjSO4oFr2zgdx1SRFeAGVeXJd9ZzsvQUM23aZY1yekqmAiPnrLN31wLXbmA1UASsqbAtAvo6mMtU0cCuzbi2fSzPL8tmz9GTTscxPmTGB9nsPVbEtDuS7d9vP3Le/1IiMqSSrY+INPJGQON5//xqO6uy8xlzayfaNbJplzVNywaRTBjYmW+2HuKvn+Y6Hcd4gKquU9V/AO1U9R8VtnmqetjpfOb8RIQpg5IoKy/n6YUZTscxPmLdziO88cU2fnJlay5rVd/pOOYCVOVXpfuA14Afu7e/AiOBz0Xkpx7MZrxgS/4JpizJ4rr2sfz0qtZOxzEOueOy5vRPasKMD7LJ2H3U6TjGc3qIyHIR2SQiuSKyVUTsty4/0bJBJCNuas8HmftYlrHX6TjGYWWnyhk9L51GUeE80a+D03HMBapKAV4OdFLVO1T1DiARKAauBH7vyXDGs0pPlTNidhq1QoN5/s4udtV0DSYiTB2cTP3IMB59K42i0lNORzKe8TrwAtAb1wCe7tggHr/yi97xdGoazdMLMzheVOp0HOOgv32+lcw9x5g4sDPREaFOxzEXqCoFeJyq7qtwfz/QXlUPAfbT78f+uGIz6/OOuqZdRtu0y5qufu0wnr+rK5v3n+C5pTYlM0AdVdX3VXW/qh48vZ3vJBHpJyLZIpIjIqMqeby1iKwQkfUiskpEWlR47B4R2eze7qnub6imCQ0OYtqQZPYdL2L6smyn4xiH7DxUyAvLN3FTp8Y2MM9PVaUA/1RE3nW/iN4DLAQ+EZHawBHPxjOesmb7IV5emcOdl7egX1JTp+MYH3Fd+1juubo1f/98m42/DkwrReR5EblaRC47vZ3rBBEJBl4G+uP6BPRuEUk847DpwJuq2gXX0LZp7nMbAE/j+sS0B/C0iNhC1UuU0rIe91wdx5tfbWftDlvCX9OoKuMWbCBYhEmpne3Taz9VlQL8t8AbQArQDXgT+K2qFtiQHv90oriMEbPX0axeLZ6+/cx/R01NN6p/J9rG1rYpmYHpSlzLTqZS9cnGPYAcVc1V1RLgLX44PTMRWOG+vbLC432B5ap6yH2x53Kg3yV/F4bH+3agSXQEo+elU3qq3Ok4xosWr9/Dx5vyebxvB5rVq+V0HHORzluAq8tcVR2hqo+6b1uvMj/2zOJM8g4XMnNYClG2bsycoVZYMC8O78bBEyWMXbAB+3EPHKp6QyXbjec5rTmws8L9PPe+itYBd7hvDwaiRCSmiueai1AnPISJAzuzce9xXvt0q9NxjJccKSxh0uIMuraoy8+ujnM6jrkEVW1DuFlEjorIMRE5LiLHvBHOVL9lGXuZvXonv7quLVfE2bRLU7nvp2Su38OCNJuSGShEpLGIvC4i77vvJ4rIfec7rZJ9Z/5W9jhwnYisBa4DduEa8FOVc09ne0BEVovI6vx8W/5UFbd0bkK/zk14ccUmth8scDqO8YJn39/I4cJSpg5JJjjIlp74s6osQfkDMFBV66pqtKpGqWq0p4OZ6rf/uGvaZVLzaB69qb3TcYyP+9V1beneuj7jF9iUzADyBrAMaOa+vwl49Dzn5AEtK9xvgWuwz/dUdbeqDlHVbsBY976jVTm3wnO8qqrdVbV7bGxs1b4bw4SBnQkJCmKcfVoV8L7OPchb3+7k/t7xdG5W1+k45hJVpQDfp6pZHk9iPEpV+f3c9RQUlzFrWAphITYty5xbcJAwc1gKCjxmUzIDRUNVnYOrvSyqWgacr+fkt0CCiMSLSBgwHNcEze+JSEMROf2iMhr4m/v2MuAWEanvvvjyFvc+U02a1I3gyX4d+HTzARamVfq7jQkAxWWnGD0/nZYNavHITQlOxzHVoCpV2GoRmS0id1echunxZKZa/evrHazMzmd0/460axTldBzjJ1o2iOTp2xP5eushXrMpmYGgwL02WwFE5CrgnJOX3EX6Q7gK5yxgjqpmiMgkERnoPux6IFtENgGNgSnucw8Bz+Aq4r8FJrn3mWr04ytb061VPZ55N5PDBXbhdCD686ot5OYXMHlQMpFhIU7HMdWgKv8Vo4FCXO9cnKbAPI8kMtUuN/8EU97L4pqEhnbRhrlgd17eghVZ+5n+QTbXJMSS2MxWoPmxkbjevW4rIp8DscCd5ztJVZcAS87YN77C7bnA3LOc+zf+84648YDgIGHakGQG/PEzpi7J4vm7ujodyVSjnP0neGXlFgZ2bcZ17W15VqA4bwGuqj/3RhDjGaenXYaHBjH9rq4E2UUb5gKJCFOHJNN31mEenb2WRQ/1JiI02OlY5iKo6ncich3QAdcFktmqagPVAkDHJtE8cG0bXlm1hSGXteDqtjFORzLVoLxcGTM/nVphwTw1wNoGB5KzLkERkSfdf/5JRP545ua9iOZS/OmjHNblHWXq4GQa27RLc5Ea1A7j+Tu7sGnfCZ636Xt+yz1U51agD65PNX8nIiOdTWWqy8N9EmgdE8nY+ekUlZ5vab/xB2+v2ck3Ww8x5taOxEaFOx3HVKNzrQE/feHlamBNJZvxcd/tOMzLK3MYcllzbk22aZfm0lzfoRE/u7o1r3+2lc9zDjgdx1ycxcC9QAwQVWEzASAiNJgpg5LJPVDAKytznI5jLlH+8WKmvJdFj/gGDO3e8vwnGL9y1iUoqrrY/ec/vBfHVJeC4jJGzE6jSXQEEwZ2djqOCRCj+3fi85wDPDZnHUsfvYZ6kWFORzIXpoV7XLwJUL0TGjKkW3P+/PEWbu/ajITG9vuVv3rm3UyKSsuZOjjZxs0HoKoM4mkvIq+KyAci8tHpzRvhzMWb/F4mOw65pl1G27RLU01qhQUza1g3Dpwotr7D/ul9Ebnl/IcZfzb2tk7UCQ9h9Lx0yq19qF9alb2fRet285sb2tKuUR2n4xgPqEobwreBtcA44IkKm/FRyzP38e9vdvLgtW3pEW/TLk31Sm5Rl0dvSuDd9Xus77D/+QqYLyInbbJx4IqpE86YWzuxevth3vp2p9NxzAUqLClj3IINtI2tza+vb+t0HOMhVSnAy1T1z6r6jaquOb15PJm5KPnHixn1znoSm0Yz8mabdmk841fXteXy1vV5auEGdh056XQcU3UzgKuBSJtsHNjuvLwFV7eJYdr7Wew/VuR0HHMBXvxwM3mHTzJ1cDLhIdZxKlBVpQBfLCK/EZGmItLg9ObxZOaCqSqj3lnP8eIyZg23aZfGc0KCg5g5NIXycuWxOWn2Mbf/2AxsUFs7FPBOtw8tLitn4ruZTscxVZSx+yivfbaV4Ve05Mo21koykFWlQrsH15KTL/hPB5TVngxlLs6/v9nJio37GdWvI+3twhvjYa1iInn69s58lXuI1z6zKZl+Yg+wSkRGi8jI05vToYxnxDesze9uaMd76/ewcuN+p+OY8zhVroyel079yDBG9+/kdBzjYecswEUkCPiJqsafsbXxUj5TRVsPFPDMu5n0bteQe3vGOR3H1BB3dW9B386Nmb5sE1l7bCmxH9gKrADCsDaENcKD17UloVEdxi3YQEFxmdNxzDm8+eU21ucdZfztidSNtOYJge6cBbiqlgPTvZTFXKQy97TLsBCbdmm8S0SYNqQLdSNDefStNBv+4eNUdaKqTgReAGZUuG8CVFhIENOGJLPryElmLt/kdBxzFruPnGT6smyuax/L7V1sbkdNUJUlKB+IyB1yEU0oRaSfiGSLSI6IjKrk8XARme1+/GsRiavw2Gj3/mwR6XvGecEislZE3r3QTIHopZU5pO08wpTBSTSpa9MujXc1qB3GH+7sQva+40y3KZk+TUSSRGQtsAHIEJE1ImKDAgJc97gG/PjKVvzt861s2HXU6TjmDKrK+IUZlCtMHpRkPb9riKoU4CNxtSIsvpC2Ve6Rxy8D/YFE4G4RSTzjsPuAw6raDpgJPOc+NxEYDnQG+gGvuJ/vtEf4z6TOGm3tjsP86aMcBndrzoAuzZyOY2qoGzo04qdXtea1z7byhU3J9GWvAiNVtbWqtgYeA/7qcCbjBU/260hMnXBGz0un7FS503FMBcsy9vJh1j5G3JxAywaRTscxXnLeAtzdpipIVcMusG1VDyBHVXNVtQR4C0g945hU4PSkzblAH/c77anAW6parKpbgRz38yEiLYDbgNeq8g0GssKSMkbOWUeT6AgmptqbWMZZY27tRJvY2jz29jqOFpY6HcdUrraqrjx9R1VXAbWdi2O8pW6tUCbc3pn0XUd544ttTscxbseKShm/MIPEptH8ole803GMF1WpT52I1BeRHiJy7emtCqc1BypOAMhz76v0GFUtA44CMec5dxbwJFDjf4Wf/F4W2w4WMGNoV5t2aRznmpKZQv7xYp5auMHpOKZyuSLylIjEubdxuC7MNDXArclN6NOxES8s30Te4UKn4xjg+aXZHDhRzLQhyYQEW+vgmqQqo+jvBz4BlgET3X9OqMJzV7aI6czes2c7ptL9IjIA2F+VQUAi8oCIrBaR1fn5+edP62dWZO3j/77ewQPXtOEq6xVqfESXFvV4pE8Ci9btZmHaLqfjmB/6BRALvAPMAxoC9zoZyHiPiDBpUBIA4xdmYO3gnbVm+2H++fV27ukZR9eW9ZyOY7ysKr9uPQJcAWxX1RuAbkBVKto8oGWF+y2AM+dWf3+MiIQAdYFD5zi3FzBQRLbhWtJyo4j8s7Ivrqqvqmp3Ve0eGxtbhbj+48CJYn7/zno6NY1m5C027dL4ll9f35bLWtVj3AKbkumD2uJ6bQ0CQoE+uN5gMTVE83q1GHlzez7auJ8l6XudjlNjlZ4qZ8y8dJpER/DYLR2cjmMcUJUCvEhVi8DVtURVNwJV+b/lWyBBROJFJAzXRZWLzjhmEa5BPwB3Ah+5J7QtAoa7u6TEAwnAN6o6WlVbqGqc+/k+UtWfVCFLwHBNu0znWFEZs4al2Jha43NCgoOYOcw1JfPxOetsSqZv+RfwN2AIMMC93e5oIuN19/aMI7l5XSYszuDoSbtewwmvfpJL9r7jPJOaRJ3wEKfjGAdUpQDPE5F6wAJguYgs5IfvZP+Ae033Q7iWrGQBc1Q1Q0QmichA92GvAzEikoOr28oo97kZwBwgE1gK/FZVrcEwMPvbnXyYtY8n+3agQxObn2F8U+uY2oy/PZEvcw/y+me2xNiH5KvqYlXdqqrbT29OhzLeFRLs6g1+8EQxzy3d6HScGmfbgQJeXLGZ/klNuCmxsdNxjEPO+2uXqg5235wgIitxLRNZWpUnV9UlwJIz9o2vcLsIuOss504BppzjuVcBq6qSI1BsO1DApHcz6dUuxq6WNj5vaPeWrMjaz/PLsumd0JBOTavSPMl42NMi8hquaZjFp3eq6jznIhknJDWvyy96xfPaZ1sZ0q053eMaOB2pRlBVxi5IJzw4iAkDrXtZTVbVLii9ReTnqvox8CU/7GZiPKzsVDkj5qQREiQ27dL4BdeUzGSia4UyYrZNyfQRPwdScM1XuN29DXA0kXHMiJvb07xeLUbPS6ekrMY3FvOK+Wt38XnOQZ7s35HG0TY4ryarSheUp4HfA6Pdu0KBSi98NJ7zyqotrN1xhMmDk2lat5bTcYypkpg64fzhzmQ27j3OjA9sSqYP6Oq+OP0eVf25e/uF06GMM2qHhzB5UBKb95/gfz7e4nScgHeooIRn3s3kslb1+HGPVk7HMQ6ryjvgg4GBQAGAqu4GbPGxF63beYQXV2wmNaUZA7vatEvjX27s2JifXNXKNSVzi03JdNhXlUwkNjXYDR0bMaBLU/60Mofc/BNOxwloU97L4nhRGdOGdLFPsU2VCvASd2cSBRARm5rmRYUlZYyYnUbjqHAmpSY5HceYizL21kTiY2rz+Jx11nXBWb2BNBHJFpH1IpIuIuudDmWcNf72RCJCghg7f4P1BveQz3MO8M53eTx4XRtroGCAqhXgc0Tkf4B6IvJL4EPgr56NZU6buiSLrQcLmD60K3Vr2bRL459qhQUzc1gK+48XM96mZDqpH662rrfwn/Xf1oawhmsUFcGo/p34Mvcgc9fkOR0n4BSVnmLs/HTiYiL53Y0JTscxPuK8BbiqTgfm4pqc1gEYr6p/8nQwAys37uefX+3g/t7x9Gzb0Ok4xlySri3r8XCfBBam7WbRuvN2MjUeULH1oLUhNBUNv6IlV8TVZ8qSLA6eKD7/CabKXvooh20HC5kyOJmIUJvdYVyq1AVFVZer6hOq+riqLvd0KAMHTxTzxNz1dGwSxeN9bUqWCQy/ub4t3VrVY9z8dHbblExjfEZQkKtrUUFxGZPfy3I6TsDI3nucv3y8hSGXNadXO3sjzfzHWQtwETkuIscq2Y6LyDFvhqxpVJXR89I5drKUWcNt2qUJHCHBQcwalkJZufL42zYl0xhf0q5RFL++ri3z1+7i0835Tsfxe+Xlypj56URFhDDuNrv22fy3sxbgqhqlqtGVbFGqahM1POjt1Xl8kLmPJ/p2oGMT+6s2gaV1TG3GD0jkiy0H+dvnNiXTGF/ymxva0aZhbcbO38DJEuvdfyn+75sdrNl+mHG3JdKgdpjTcYyPqdISFOM9Ow4WMnFxBle3ieG+3jbt0gSmYVe05KZOjfnD0mw27rUP1IzxFRGhwUwZnMyOQ4X88aPNTsfxW/uOFfHc+xvp1S6GIZfZ7ELzQ1aA+5DT0y6DgoQZQ23apQlcIsKzdyQTXSuER99Ko7jM3mkzxldc3TaGuy5vwV8/ybVfkC/SxMUZlJwqZ8qgZETs33LzQ1aA+5C/fLyFNdsPM3lQEs3q2bRLE9ga1gnnD3d2cU/J3OR0HHMOItLP3Ts8R0RGVfJ4KxFZKSJr3f3Fb3XvDxWRf7j7jWeJyOgfPrvxRWNu7UTdWqGMeiedU3atxgX5MHMfS9L38nCfBOIa2ugUUzkrwH3E+rwjzPpwM7d3bUZqin1cZWqGGzs25kdXtuKvn+by5ZaDTscxlRCRYOBloD+QCNxdyTTNccAcVe0GDAdece+/CwhX1WTgcuBBEYnzRm5zaerXDuOpAYmk7TzCv762TpVVVVBcxviFG2jfuA6/vKaN03GMD7MC3AecLDnFo7PTiI0KZ7JNuzQ1zLjbOhEXU5vH5qTZlEzf1APIUdVcVS0B3gJSzzhGgdNXjNcFdlfYX1tEQoBaQAlgaxr8RGpKM65JaMgflmaz92iR03H8wowPNrHnWBHThnQhLMRKLHN29n+HD5j2fha5+QVMv6srdSNt2qWpWSLDQpg5LIV9x4t52qZk+qLmwM4K9/Pc+yqaAPxERPKAJcDv3PvnAgXAHmAHMF1VD3k0rak2IsKUQcmUlZfz9CL72Tyf9XlHeOOLrfz4ylZc3rq+03GMj7MC3GGrsvfz5pfbua93vDXpNzVWSst6PHxjAgvSdrPYpmT6msquIDtzUfDdwBuq2gK4FfhfEQnC9e75KaAZEA88JiKVfi4vIg+IyGoRWZ2fbz2ofUWrmEge6dOeZRn7+CBjr9NxfFbZqXJGvZNOwzrhPNmvo9NxjB+wAtxBhwpKeGLuejo0juIJm3Zparjf3uCakjl2fjp7jtqUTB+SB7SscL8F/1lictp9wBwAVf0SiAAaAj8ClqpqqaruBz4Hulf2RVT1VVXtrqrdY2Njq/lbMJfi/mvi6dgkivELMzheZMvEKvP3z7eRuecYEwd2JjrCPsk252cFuENUlTHz0jlaWMrMYSlEhNq0S1OzhQQHMXOoTcn0Qd8CCSISLyJhuC6yXHTGMTuAPgAi0glXAZ7v3n+juNQGrgI2ei25qRahwUE8e0cX9h0vso5Fldh5qJAXlm/ipk6N6JfUxOk4xk9YAe6QuWvyWJqxl8duaU9iM5t2aQxAXMPaPDUgkc9zDvL3L7Y5HccAqloGPAQsA7JwdTvJEJFJIjLQfdhjwC9FZB3wb+BeVVVc3VPqABtwFfJ/V9X1Xv8mzCVLaVmPn13Vmn98uY20nUecjuMzVJWnFm5ABCamJlnPb1NlIU4HqIl2Hipk4uJMroxvwP3WpsiY/zL8ipasyNrPc0s30rtdQzo0iXI6Uo2nqktwXVxZcd/4CrczgV6VnHcCVytCEwAe79uBZRn7GD0vnUUP9SI02N7De3f9HlZl5zN+QCLNbX6HuQD20+Nlp8qVEbPTEGDG0K4E27RLY/7L91MyI0J45K21NiXTGB8RFRHKxNTOZO05xuufbXU6juOOFpYycXEGXVrU5Z6ecU7HMX7GCnAv+8vHW1i9/TATUzvTon6k03GM8UkN64Tz3B2uKZkv2JpTY3xG385NuCWxMbM+3MSOg4VOx3HUs0uzOFxYytTByfZmmrlgVoB70YZdR5m5fBO3JTdlcDebdmnMufTp1Ji7e7Ti1U9z+SrXpmQa4ysmpnYmJCiIsQvScS31r3m+2XqIf3+zk/t6x5PUvK7TcYwfsgLcS4pKXdMuY+qEMWWwXahhTFWMu60TrRtE8ticdRyz9mfG+ISmdWvxRN8OfLr5AItqYN/+4rJTjJ63nub1avHoTQlOxzF+ygpwL3n2/Y3k7D/B9Lu6Ui8yzOk4xviF2uGuKZl7jxXx9MIMp+MYY9x+clVrUlrWY9LiTI4Uljgdx6v+siqXLfkFTB6cRGSY9bIwF8cKcC/4ZFM+b3yxjXt7xnFNgg2YMOZCdGtVn4duaMf8tbt4d33Ne7fNGF8UHCRMG5LM0ZOlTF2S5XQcr8nZf4KXV+Zwe9dm3NChkdNxjB+zAtzDDheU8Pjb60hoVIdR/W08rTEX46Eb29G1ZT3Gzt/A3qNFTscxxgCdmkZz/zVtmLM6r0Zcp6GqjJ2fTkRoEOMHJDodx/g5K8A9SFUZuyCdw4UlNu3SmEsQGhzErGEplJSV25RMY3zII30SaNUgkjHz0ykqDeyWoW+vzuPrrYcYc2snYqPCnY5j/JwV4B40f+0ulqTvZcTN7e0qaWMuUbx7SuZnOQd4w6ZkGuMTaoUFM2VwErn5BbyyaovTcTzmwIlipizJokdcA4Z2b+l0HBMArAD3kLzDhTy9MIMecQ148Nq2TscxJiDc3aMlfTo24tmlG4uCDmcAABjUSURBVNm077jTcYwxwDUJsQxKacafV+WQsz8wfy6feTeTwpIypg5JIsh6fptqYAW4B5wqV0bOWYdi0y6NqU6uKZldiAoP4dG30igpK3c6kjEGGDcgkdrhIYyZtyHgloh9vCmfhWm7/7+9O4+vqr7zP/76ZCGssm9lX4WEKGJwR0UsmwvLaEUdB6eO1hm1Ym07IFYZHcRaty46rVWqtVZwFJEqglatiqIsCiQsgbCo7CCyQ4Dw+f1xD05+MYFs956be9/Px+M+cvK9Z/l8uOd++eRsX/7jwq50bdEg7HAkQUS1ADezwWaWb2YFZja2lPczzGxq8P6nZtax2HvjgvZ8MxsUtLUzs/fMbLmZLTWz26MZf2X98cM1zFu7gwmXZ9GuiUa7FKlOzRtERslctmk3j76tUTJF4kGz+hncNbQn89btYOqCr8IOp9ocOFTE3dNz6dy8Hv/RX2ezpfpErQA3s1TgCWAIkAlcbWYlbxu+AfjG3bsCjwG/DJbNBEYBWcBg4MlgfUeAO929J3AWcEsp6wzV0o27eOStfIb0asU/9dFolyLRcHFmS64+ox1/+GA1nybB0xdEaoIrT2/LWZ2bMGnmcrbuSYynFT3+zkq+2nGAB0Zkk5GmBylI9YnmEfAzgAJ3X+Puh4ApwLAS8wwDngumXwYGWGSIyGHAFHcvdPe1QAFwhrtvcvfPANx9D7AciJsq9+DhIu6YuojGdWvxwIhsjXYpEkV3X5JJhyZ1+YlGyRSJC2bGAyOyOXjkKPf9bVnY4VTZ0o27ePrDtVyV046zOjcNOxxJMNEswNsAxc9Dree7xfK387j7EWAX0LQ8ywaXq5wGfFqNMVfJQ7PyWbllL7+68lQa19NolyLRVC8jjUeDUTInaJRMkbjQuXl9bu3fldeXbOK9/K1hh1NpRUedu6bl0rhuOuOGagwPqX7RLMBLO/xb8s6MsuY57rJmVh94BRjj7rtL3bjZTWa2wMwWbNu2rZwhV96cVduZ/NFaRp/dgQu6a7RLkVjo074xt/TvyrTPN/DGkk1hhyMiwM0XdKFbi/rc/Woe+w8dCTucSnl+7joWr9/FLy7NpFFdHVCT6hfNAnw9UPxhmW2BkuNIfzuPmaUBDYEdx1vWzNKJFN8vuPu0sjbu7k+5e4675zRvHt2CeOf+yGiXXZrXY+yQnlHdloj8/267qCuntm3IXa/mapRMkThQKy2FB0Zms2HnAR6rgTdKb9x5gF/Nzuf87s25/NTvhR2OJKhoFuDzgW5m1snMahG5qXJGiXlmAKOD6SuAd93dg/ZRwVNSOgHdgHnB9eHPAMvd/dEoxl5u7s7d0/PYvreQX486jTq1dJOGSCylp6bwWDBK5s9e1iiZIvGgb8cmXH1GeyZ/tI68DbvCDqdC7p2xlCJ3Jg7vpXu5JGqiVoAH13TfCswmcrPkS+6+1MzuM7PLg9meAZqaWQHwE2BssOxS4CVgGTALuMXdi4BzgeuAi8xsUfAaGq0cyuO1RRt5fckmjXYpEqLOzesz/pKefLhqO8/NXRd2OCICjB3Sgyb1ajFuWi5FNeQP41l5m3l72RbuuLi7HiMsUZUWzZW7+0xgZom2e4pNHwSuLGPZicDEEm1zKP368FBs2HmAX7yWR06Hxtx8gZ4PKhKma89sz7srtvLgmys4r2szurXUgBkiYWpYJ517L8vk1r9+zrMfr+OG8zqFHdJx7T54mHtn5NGz9Un8MM5jlZpPI2FW0tGjzp0vLeLoUeexq3prtEuRkEVGycymXkYat2uUTJG4cEl2a/qf3JxH3spnw84DYYdzXA/PzmfrnkImjcwmPVXlkUSX9rBKenrOGj5Zs4N7NdqlSNxo0aA2D47MZtmm3Tz295p385dIojEz7h/eC3e4Z3oekdu84s/CL77h+U++YPTZHendrlHY4UgSUAFeCcs37ebh2SsZlNWSK09vG3Y4IlLMwKxWjOrbjt+/v5p5a3eEHY5I0mvbuC53DuzOOyu28mbe5rDD+Y7DRUe5a1ourU6qzU8HnRx2OJIkVIBX0MHDRYyZsoiGddOZNPIU3SEtEod+cWkm7ZvU5Y6pi9ijUTJFQnf9OR3p1eYkJsxYyq4D8fWd/OOHa8jfsof7hvWifkZUb40T+ZYK8Ap6eHY++Vv28NAVp9BEo12KxKV6GWk8+oPebNp1gAkzav6Q2CI1XVpqCg+OPIXtewt5aNaKsMP51hdf7+PXf1/F4KxWfD+zZdjhSBJRAV4BHxds5+k5a7nurA70P7lF2OGIyHGc3qExt/bvyiufrWdmrkbJFAlbrzYN+ddzO/HCp1+y8IvwLw9zd8a/mkd6agoTLs8KOxxJMirAy2nX/sPc+b+L6dy8HncN1WiXIjXBbQO6fTtK5pbdGiVTJGw/+X532jSqw7hpuaE/qWj6og3MKdjOfw4+mVYNa4caiyQfFeDl9IvX8ti2p5DHr+qt0S5Faoj01BQevao3Bw8X8dP/1SiZImGrl5HG/cOzWLllL099sDq0OHbsO8T9ry/ntPaNuPbMDqHFIclLBXg5vLZoAzMWb+T2Ad04pa0eTyRSk3RpXp/xl2Ty4art/HnuurDDEUl6F/VoySXZrfnNuwWs3b4vlBgemLmc3QcOM2lkNikax0NCoAL8BDbuPMDd0/Po074R/36hRrsUqYn++cz29D+5OZPeXMGqLXvCDkck6d17WSYZaSmMfzU35s8G/7hgOy8vXM9N53emR6uTYrptkWNUgB9HZLTLxd+OdpmmkbFEaiQz45dXnEK9jDTGTNUomRVlZoPNLN/MCsxsbCnvtzez98zsczNbYmZDi713ipnNNbOlZpZrZrrYVmhxUm3GDunBx6u/5pXPNsRsuwcPFzF+eh4dmtblxwO6xWy7IiWpojyOyR+tZe6ar7nnskw6NK0XdjgiUgUtGtRm0shslm7czeMaJbPczCwVeAIYAmQCV5tZZonZ7gZecvfTgFHAk8GyacBfgJvdPQu4EIivh0BLaK7u256cDo2Z+MYyvt5bGJNtPvFe5LKXicOzqZ2u+7kkPCrAy7Bi824empXPwMyW/CCnXdjhiEg1GJTVih/ktOX3769m/rrwH4NWQ5wBFLj7Gnc/BEwBhpWYx4Fj5/IbAhuD6YHAEndfDODuX7t7UQxilhogJcV4YGQ2ewuPMPGN5VHf3sote/iff6xm5GltOK9bs6hvT+R4VICXovBIZLTLk+qkM2lktka7FEkg91yWRdvGGiWzAtoAXxX7fX3QVtwE4J/NbD0wE7gtaO8OuJnNNrPPzOzn0Q5WapbuLRtw8wVdmPb5Buas2h617Rw96oyblkuD2mmMv0SPEpbwqQAvxSsLN7Bi8x4euiKbpvUzwg5HRKpR/Yw0HrvqVDbuPMCVv5/LSt2UeSKlHYEoedfc1cCz7t4WGAo8b2YpQBpwHnBt8HOEmQ0odSNmN5nZAjNbsG3btuqLXuLeLf270qlZPcZPz+Xg4eicIHlx/pcs/OIbxl+Sqf/XJS6oAC/FqL7t+OuNZ3JRDw1LK5KITu/QhGdG92X73kIu++0cnv1obcyfxFCDrAeKX4fXlv+7xOSYG4CXANx9LlAbaBYs+767b3f3/USOjvcpbSPu/pS757h7TvPmzas5BYlntdNTmTiiF198vZ/fvLOq2te/dfdBHnxzBed0aco/9Sl58kYkHCrAS5GSYpzTRdeHiSSy/j1a8Obt53NOl6ZM+Nsyrv/TfLbu0WiZpZgPdDOzTmZWi8hNljNKzPMlMADAzHoSKcC3AbOBU8ysbnBD5gXAsphFLjXGOV2accXpbXnqgzWs2Ly7Wtf9X39bRuGRo0wcoUtKJX6oABeRpNW8QQaTr+/L/cOy+GTN1wx+/EP+vmxL2GHFFXc/AtxKpJheTuRpJ0vN7D4zuzyY7U7gRjNbDLwIXO8R3wCPEiniFwGfufsbsc9CaoLxQ3tyUp10xk3LrbZRa99ZvoU3cjfx44sil7mIxAtLhtOuOTk5vmDBgrDDEJE4VrB1Dz9+cRHLNu3mmjPbc/clPalbKy3ssDCzhe6eE3YcsaQ+O3m9+vl67pi6mPuHZXHd2R2rtK59hUcY+NgH1MtI5fXb+lErTcccJfrK22drbxQRAbq2aMCrt5zDj87vzIvzvuTS384hd/2usMMSSSrDe7ehX7dmPDQrn827qnZJ2KNvr2TDzgM8MCJbxbfEHe2RIiKBjLRUxg3tyQv/dib7C4sY8eRHPPmPAoqq6XS4iByfmfHfw3txqOgoE2YsrfR6ctfv4k8freXaM9uT07FJNUYoUj1UgIuIlHBOl2bMGtOPQVmteGhWPtf88RM27DwQdlgiSaFD03rcfnE3Zi3dzNuVuCfjSNFRxk5bQtP6Gfx8cI8oRChSdSrARURK0ahuLX53zWk8fOWp5G3YxeDHP+C1RRvCDkskKdzYrzM9WjXgntfy2Ft4pELLPvvxOpZu3M2Ey7JoWCc9ShGKVI0KcBGRMpgZV5zelpm396Nbi/rcPmURd0xdxG6NoCkSVempKTwwMpvNuw/y8Oz8ci/31Y79PPLWSgb0aMHQ7FZRjFCkalSAi4icQIem9XjpR2cz5uJuzFi8kSGPf8j8dTvCDkskofVp35jrzurAc3PXsfirnSec392557U8zOC+4b30zG+JayrARUTKIS01hTEXd+elH51Naopx1R/m8shb+RwuOhp2aCIJ62eDTqZlg9qMnZZ7wu/aG7mbeC9/G3cOPJk2jerEKEKRylEBLiJSAad3aMzM2/sxsk9bfvtuAVf8fi5rt+8LOyyRhNSgdjoTLs9i+abdTJ6ztsz5du0/zIQZy8hu05Drz+kYuwBFKkkFuIhIBdXPSOPhK0/lyWv7sG77Pi75zYdMnf8lyTCwmUisDe7Viu9ntuSxv6/kqx37S53nwVkr2LGvkEkjs0lN0aUnEv9UgIuIVNLQ7NbMGtOP3u0a8Z+v5HLzXxbyzb5DYYclknDuG5ZFqhl3T8/7zh+689ft4MV5X3LDeZ3o1aZhSBGKVIwKcBGRKmjdsA5/ueFMxg/tybsrtjLo8Q/4cNW2sMMSSSitG9bhZ4NO5v2V25ixeOO37YVHihg3LZc2jepwx/e7hxihSMWoABcRqaKUFOPG8zsz/ZZzOalOOtc9M4/7X1/GwcNFYYcmkjCuO7sjp7ZrxP2vL2Pn/siZpj+8v4aCrXv57+G9qFsrLeQIRcovqgW4mQ02s3wzKzCzsaW8n2FmU4P3PzWzjsXeGxe055vZoPKuU0QkLFnfa8jrt53H6LM78MyctQx/4iPyN+8JOyyRhJCaYkwakc03+w8zaeYKVm/by+/eLeDSU1rTv0eLsMMTqZCoFeBmlgo8AQwBMoGrzSyzxGw3AN+4e1fgMeCXwbKZwCggCxgMPGlmqeVcp4hIaGqnp/Jfw3rxp+v7sn1vIZf9bg6T56zl6FHdoClSVZnfO4l/69eJqQu+4sY/L6B2egr3XKYyQGqeaB4BPwMocPc17n4ImAIMKzHPMOC5YPplYIBFnpw/DJji7oXuvhYoCNZXnnWKiISuf48WzBpzPv26NuO+15dx/bPzOXREzwwXqaoxA7rTrkkd1mzbx9ghPWnRoHbYIYlUWDQvmGoDfFXs9/XAmWXN4+5HzGwX0DRo/6TEsm2C6ROtEwAzuwm4CaB9+/aVy0BEpAqa1c/g6dE5/OXTL1m9dS+10nTbjUhV1amVypPXnM57+VsZ1bdd2OGIVEo0C/DSHsRZ8hxsWfOU1V7a/16lntd196eApwBycnJ07ldEQmFmXHdWh7DDEEko2W0bkt1WjxyUmiuah2PWA8X/NG0LbCxrHjNLAxoCO46zbHnWKSIiIiISt6JZgM8HuplZJzOrReSmyhkl5pkBjA6mrwDe9cgT9mcAo4KnpHQCugHzyrlOEREREZG4FbVLUIJrum8FZgOpwGR3X2pm9wEL3H0G8AzwvJkVEDnyPSpYdqmZvQQsA44At7h7EUBp64xWDiIiIiIi1S2qT61395nAzBJt9xSbPghcWcayE4GJ5VmniIiIiEhNoVvyRURERERiSAW4iIiIiEgMqQAXEREREYkhFeAiIiIiIjFkkaf+JTYz2wZ8UcHFmgHboxBOPFKuiUm5JoYO7t487CBiqZJ9NiT2flCSck1MyrXmK1efnRQFeGWY2QJ3zwk7jlhQrolJuUqySab9QLkmJuWaPHQJioiIiIhIDKkAFxERERGJIRXgZXsq7ABiSLkmJuUqySaZ9gPlmpiUa5LQNeAiIiIiIjGkI+AiIiIiIjGkArwUZjbYzPLNrMDMxoYdT3Uws3Vmlmtmi8xsQdDWxMzeNrNVwc/GQbuZ2W+C/JeYWZ9woz8+M5tsZlvNLK9YW4VzM7PRwfyrzGx0GLmcSBm5TjCzDcFnu8jMhhZ7b1yQa76ZDSrWHtf7uJm1M7P3zGy5mS01s9uD9oT8XKVq4n1/rgz12Ynx3U6WPhvUb1eYu+tV7AWkAquBzkAtYDGQGXZc1ZDXOqBZibaHgLHB9Fjgl8H0UOBNwICzgE/Djv8EuZ0P9AHyKpsb0ARYE/xsHEw3Dju3cuY6AfhpKfNmBvtvBtAp2K9Ta8I+DrQG+gTTDYCVQT4J+bnqVaV9Je7350rmpT77OLnVlO92svTZQfzqtyvw0hHw7zoDKHD3Ne5+CJgCDAs5pmgZBjwXTD8HDC/W/meP+ARoZGatwwiwPNz9A2BHieaK5jYIeNvdd7j7N8DbwODoR18xZeRalmHAFHcvdPe1QAGR/Tvu93F33+TunwXTe4DlQBsS9HOVKon7/bkaqc+uYd/tZOmzQf12RakA/642wFfFfl8ftNV0DrxlZgvN7KagraW7b4LIFwdoEbQnwr9BRXOr6TnfGpzCm3zs9B4JkquZdQROAz4l+T5XObFE/YzVZ0ck6nc7YftsUL9dHirAv8tKaUuER8Wc6+59gCHALWZ2/nHmTdR/Ayg7t5qc8/8AXYDewCbgkaC9xudqZvWBV4Ax7r77eLOW0lajcpVKS9TPWH12RCJ+txO2zwb12+WlAvy71gPtiv3eFtgYUizVxt03Bj+3Aq8SOaW15dhpyuDn1mD2RPg3qGhuNTZnd9/i7kXufhT4I5HPFmp4rmaWTqQTf8HdpwXNSfO5Srkl5GesPjtxv9uJ2meD+u2KUAH+XfOBbmbWycxqAaOAGSHHVCVmVs/MGhybBgYCeUTyOnZ38WjgtWB6BvAvwR3KZwG7jp0+qkEqmttsYKCZNQ5OBw4M2uJeiWs9RxD5bCGS6ygzyzCzTkA3YB41YB83MwOeAZa7+6PF3kqaz1XKLe7354pSn53Y3+1E7LNB/XaFhX0XaDy+iNyZu5LIXcfjw46nGvLpTOSu6cXA0mM5AU2Bd4BVwc8mQbsBTwT55wI5YedwgvxeJHIa7zCRv5xvqExuwA+J3PRSAPxr2HlVINfng1yWEOnQWhebf3yQaz4wpFh7XO/jwHlETjkuARYFr6GJ+rnqVeX9Ja7350rkoz47Qb7bydJnBzGq367ASyNhioiIiIjEkC5BERERERGJIRXgIiIiIiIxpAJcRERERCSGVICLiIiIiMSQCnARERERkRhSAS5Jwcw+Dn52NLNrqnndd5W2LRERqRz12ZLo9BhCSSpmdiHwU3e/tALLpLp70XHe3+vu9asjPhER+T/qsyVR6Qi4JAUz2xtMPgj0M7NFZnaHmaWa2a/MbL6ZLTGzHwXzX2hm75nZX4kMEICZTTezhWa21MxuCtoeBOoE63uh+LaC0b1+ZWZ5ZpZrZlcVW/c/zOxlM1thZi8EI4hhZg+a2bIglodj+W8kIhIv1GdLoksLOwCRGBtLsaMpQae8y937mlkG8JGZvRXMewbQy93XBr//0N13mFkdYL6ZveLuY83sVnfvXcq2RgK9gVOBZsEyHwTvnQZkARuBj4BzzWwZkWGJe7i7m1mjas9eRKRmUZ8tCUlHwCXZDQT+xcwWAZ8SGTK3W/DevGIdOcCPzWwx8AnQrth8ZTkPeNHdi9x9C/A+0LfYute7+1Eiw/V2BHYDB4GnzWwksL/K2YmIJBb12ZIQVIBLsjPgNnfvHbw6ufuxoyn7vp0pch3ixcDZ7n4q8DlQuxzrLkthsekiIM3djxA5gvMKMByYVaFMREQSn/psSQgqwCXZ7AEaFPt9NvDvZpYOYGbdzaxeKcs1BL5x9/1m1gM4q9h7h48tX8IHwFXBNYvNgfOBeWUFZmb1gYbuPhMYQ+RUqIhIMlOfLQlJ14BLslkCHAlOSz4L/JrIqcTPgptqthE5klHSLOBmM1sC5BM5pXnMU8ASM/vM3a8t1v4qcDawGHDg5+6+OfjPoDQNgNfMrDaRIzF3VC5FEZGEoT5bEpIeQygiIiIiEkO6BEVEREREJIZUgIuIiIiIxJAKcBERERGRGFIBLiIiIiISQyrARURERERiSAW4iIiIiEgMqQAXEREREYkhFeAiIiIiIjH0/wAt6uergY1WyQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_text=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can easily test different types of annealing at the end of the 1cycle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# It supports discriminative learning rates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you unfreeze a pretrained model, you often use differential learning rates, and this works with this new API too. Just pass an array or a list of learning rates instead of a single value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.pretrained(resnet34, data, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()\n",
    "lr = 1e-2\n",
    "lrs = np.array([lr/100,lr/10,lr])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.Adam, lr=(lrs/10,lrs), lr_decay=DecayType.LINEAR),\n",
    "          TrainingPhase(epochs=2, opt_fn=optim.Adam, lr=lrs, lr_decay=DecayType.COSINE)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4d1cca7b9f84707aa86b297f11294e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      1.762334   1.626667   0.3959    \n",
      "    1      1.53574    1.601774   0.4827                                                                                \n",
      "    2      1.496761   1.487644   0.5                                                                                   \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([1.48764]), 0.5]"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is plotted in these cases is the highest learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VOX1wPHvyR4gBAhhXxIgLAFZIyCuqAgIQlWqiFVcfrW2UPcF3OvWulTc2+JWd7DgQgFREURRBMIOCYGwhy2BAAmEkO38/phLm8YEJsvkJpPzeZ55MnPnvXfOvSRzuO977ntFVTHGGGMqKsDtAIwxxtRulkiMMcZUiiUSY4wxlWKJxBhjTKVYIjHGGFMplkiMMcZUiiUSY4wxlWKJxBhjTKVYIjHGGFMpQW4HUB2aNm2qMTExbodhjDG1xooVKw6oarQ3betEIomJiSExMdHtMIwxptYQkR3etrWuLWOMMZViicQYY0ylWCIxxhhTKZZIjDHGVIolEmOMMZXi00QiIsNEJEVEUkVkUinvh4rIdOf9pSIS4yyPEpGFInJURF4tsU4/EVnnrPOyiIgv98EYY8yp+SyRiEgg8BowHIgHrhGR+BLNbgYOqWonYArwjLM8F3gYuKeUTf8NuAWIcx7Dqj56Y4wx3vLldST9gVRV3QogItOA0UBSsTajgcec5zOAV0VEVPUYsFhEOhXfoIi0BBqq6hLn9XvAr4AvfbgfpgbZc/g4M1ekERQYQFhwAPVDgmgeGUZL5xERFux2iMbUOb5MJK2BXcVepwEDymqjqgUicgSIAg6cYptpJbbZurSGInILnjMX2rVrV97YTQ2kqtw/cy0/bC7r1wNaRYbRrWVD4ls1JCGmCWfGNKZeSJ247tYY1/jyL6y0sQutQJsKtVfVqcBUgISEhFNt09QS36Vk8MPmAzw0ohvXDmhPbn4hR08UsD8rlz1Hckk7lEPKvmyS92bx3aYMCotSCQoQerdtxIXdmnFpj5bENK3v9m4Y43d8mUjSgLbFXrcB9pTRJk1EgoBIIPM022xzmm0aP5RfWMQTc5Lo0LQ+158VQ0hQAOEhgTSuH0LbJvV+0f54XiGJOzL5actBFm8+wLPzUnh2XgrdWjbkV71bMaZfG6IahLqwJ8b4H18mkuVAnIjEAruBscC4Em1mAeOBJcAYYIGqlnn2oKp7RSRbRAYCS4HrgVd8EbypWd5fsoOtGcd4a3wCIUGnrxEJDwnk3Lhozo2L5v5hsPvwceat38ectXv485cb+evXmxjaowXXDWzPmTGNseI/YypOTvG9XfmNi1wKvAgEAm+r6lMi8jiQqKqzRCQMeB/og+dMZGyxwfntQEMgBDgMXKKqSSKSAPwTCMczyP7HUyUf8HRt2aSNtdehY3mc/9xCerVtxHs39a/0l/6m/dl8tHQnn65MIyu3gIT2jZkwuBMXdIm2hGKMQ0RWqGqCV219mUhqCksktdujX6zn/Z938OXt59GlRUSVbfd4XiGfJO5i6vdb2X34OPEtG3LfsC6c39kSijHlSSR2Zbup0Tbvz+aDpTsZN6BdlSYR8HR/jR8Uw3f3XsDzv+7F0RMF3PDOcq57axkb9hyp0s8yxp9ZIjE12pNzkqkXEsidF3f22WcEBwYwpl8b5t91Po+MjGf9niOMfGUxkz9dy5GcfJ99rjH+whKJqbEWpqSzaFMGt18UVy0VViFBAdx0TiyL7h3MzWfH8kliGhe98B1frN5NXegCNqaiLJGYGim/sIgnZycRE1WP68+KqdbPjgwP5qGR8cyaeDatG4Vz+7TVXP/2MvYeOV6tcRhTW1giMTXShz/vYEvGMR4cEe9Vua8vdG8Vyad/OJvHLosncfshhk75ntlr7bIlY0qyRGJqnMM5eUyZv5mzO0VxcbdmrsYSGCDccHYsc28/lw7RDZj40SrunL6arFwbOzHmJEskpsZ5cf5msnPzeWhEfI0pw41tWp8Zt57FHRfHMWvNHka9spjkvVluh2VMjWCJxNQoqenZvP/zDsb2b0e3lg3dDud/BAUGcMfFnZl2y0By8gq5/PUf+XRl2ulXNMbPWSIxNcpTc5KpFxzIXUN8V+5bWWfGNGHObefSu20j7vpkDQ9+to68giK3wzLGNZZITI3xXUo6C1MyuO2iOJrW8AkVoyNC+eDmAdx6fkc+XLqT699eyuGcPLfDMsYVlkhMjVBQWMSTc5KJiarH+EExbofjlaDAACYN78pLY3uzcsdhLn/9J7ZmHHU7LGOqnSUSUyN8tGwnqelHeeDSbq6V+1bU6N6t+ei3AzhyPJ/LX/+JJVsOuh2SMdWqdv3FGr90JCefF77ZxKCOUQyJb+52OBWSENOEz/9wNtERoYx/exlz1+11OyRjqo0lEuO6l77dTNbxfB4eWXPKfSuiXVQ9Zt46iJ5tIpnw0Uo+WrrT7ZCMqRaWSIyrtmQc5b0l27n6zJpX7lsRkfWCef/mAZzfOZoHPlvHawtTbZ4u4/cskRhXPT0nmbDgQO6+pOaW+5ZXeEggb1yfwK96t+K5r1L485cbLZkYv+bLW+0ac0rfb8rg243pTB7etcaX+5ZXcGAAL1zVm8jwYKZ+v5WCQuXhkd1qddedMWWxRGJcUVBYxBOzk2jXpB43nB3jdjg+ERAgPDaqOwEBwts/bkNRHqnl40DGlMYSiXHFx8t2sjn9KH//TT9CgwLdDsdnRMSTPPAkE8CSifE7lkhMtTtZ7juwQxOGdq+d5b7lISJOtxa8tXgbqvDoZZZMjP+wRGKq3csLNnPYD8p9y0NEeGhEN8CTTBqEBnHP0C4uR2VM1bBEYqrV1oyjvPvTdq5OaEv3VpFuh1OtTiaTYycKeHVhKg3Dg7jlvI5uh2VMpVkiMdXq6bkny33r5v/GRYSnLj+D7BMFPD13Iw3Dghnbv53bYRlTKZZITLX5YXMG85PTuX9YV6Ij/KvctzwCA4QpV/XmaG4Bkz9bR4OwIEb2bOV2WMZUmF2QaKpFQWERT85Opm2TcG7003Lf8ggJCuDvv+lHQvvG3Dl9NYs3H3A7JGMqzBKJqRbTlu8iZX82DwzvRliw/5b7lkd4SCBvjj+TDk0bcOsHK+zWvabWskRifO7IcU+5b//YJgzr0cLtcGqUyPBg3rnxTOqHBnLjO8vZe+S42yEZU26WSIzPvbpgM4dy8uxCvDK0ahTOOzf05+iJAm58ZznZufluh2RMuVgiMT617cAx/vnTdq7q15YeretWuW95xLdqyOvX9iU1/Si//2Al+YV2D3hTe1giMT719NxkQgIDuHuo/8zu6yvndY7m6SvOYHHqAR74dJ3NGGxqDZ8mEhEZJiIpIpIqIpNKeT9URKY77y8VkZhi7012lqeIyNBiy+8UkQ0isl5EPhaRMF/ug6m4H1MP8E3SfiZc2IlmEfbP5I2rEtpy24Wd+NeKNN5avM3tcIzxis8SiYgEAq8Bw4F44BoRiS/R7GbgkKp2AqYAzzjrxgNjge7AMOB1EQkUkdbAbUCCqvYAAp12poYpLFKemJ1Em8bh3HR2rNvh1Cp3XNyZ4T1a8PTcZBampLsdjjGn5cszkv5AqqpuVdU8YBowukSb0cC7zvMZwEXiGY0dDUxT1ROqug1IdbYHnosow0UkCKgH7PHhPpgKmr58Fxv3ZfPApVbuW14BAcJfr+pFlxYNue2jVaSmZ7sdkjGn5MtE0hrYVex1mrOs1DaqWgAcAaLKWldVdwPPAzuBvcARVf3aJ9GbCsvKzeevX6fQP6YJw63ct0LqhQTxxvX9CAkK4P/eTeRwTp7bIRlTJl8mktLqPEuOHpbVptTlItIYz9lKLNAKqC8ivyn1w0VuEZFEEUnMyMgoR9imsl5bkEpmTl6dmt3XF9o0rsc/ruvH7sPHmfjRKgqsksvUUL5MJGlA22Kv2/DLbqj/tHG6qiKBzFOsezGwTVUzVDUf+BQYVNqHq+pUVU1Q1YTo6Ogq2B3jje0HjvH2j9sY07cNZ7Sxct/KSohpwlOXeyq5npqb7HY4xpTKl4lkORAnIrEiEoJnUHxWiTazgPHO8zHAAvXUPM4CxjpVXbFAHLAMT5fWQBGp54ylXATYX1cN8vTcZIIDA7jX7rVRZa5KaMsNg2J458ftfLF6t9vhGPMLPkskzpjHROArPF/2n6jqBhF5XERGOc3eAqJEJBW4C5jkrLsB+ARIAuYBE1S1UFWX4hmUXwmsc+Kf6qt9MOXzU+oBvk7az4TBnWjW0Mp9q9IDl3YjoX1jJs1cx6b9NvhuahapCxc9JSQkaGJiotth+LXCImXEyz+QnVvAt3efb5VaPrA/K5cRLy+mYVgQn088m4ZhwW6HZPyYiKxQ1QRv2tqV7aZKfJLoKfedfGlXSyI+0rxhGK+N68OOzBzu+WSNXfluagxLJKbSsnLzef6rFBLaN2bEGS3dDsevDegQxQOXduPrpP38fdFWt8MxBrBEYqrAawtTOXgsj0cus3Lf6nDT2TGM7NmS577ayI+pdkMs4z5LJKZSdhw8xjuLt3Nl3zb0bNPI7XDqBBHhmSt70jG6Abd9vIr0rFy3QzJ1nCUSUyl/nruRwADhvmFW7lud6ocG8fq1fcnJK+T2aaspLLLxEuMeSySmwpZsOci8Dfv4wwUdaW7lvtUurnkEj4/uzpKtB3l1Qarb4Zg6zBKJqZCTs/u2bhTOb8/r4HY4ddaYfm24ok9rXvp2E0u2HHQ7HFNHWSIxFTJjxS6S9mYxabiV+7pJRHjiVz2IaVqf26et4uDRE26HZOogSySm3LJz83nuq030a9+YkT2t3Ndt9UODeG1cXw4fz+fOT9ZQZOMlpppZIjHl9vp3Wzhw9ASP2Oy+NUa3lg159LJ4vt+UwT++t+tLTPWyRGLKZVdmDm/9sI0r+ramV1sr961JxvVvx4ieLXn+6xRW7DjkdjimDrFEYsrlz18me8p9h3Z1OxRTgojwlyvOoFWjMO6Yvors3Hy3QzJ1hCUS47WlWw8yd90+fn9BR1pEWrlvTRQRFsyLV/dhz+FcHp21we1wTB1hicR4pbBIeXx2Eq0iw/jtuVbuW5P1a9+YP17YiU9X7mbWmpL3kjOm6lkiMV6ZuSKNDXuyuH94V8JDrNy3pps4uBN92zXiwc/WsfvwcbfDMX7OEok5raMnCnj2qxT6tGvEqF6t3A7HeCEoMIAXr+6DKtw53aZQMb5licSc1usLU63ctxZqF1WPP43qzrJtmfx90Ra3wzF+zBKJOaVdmTm8uXgbl/dpTZ92jd0Ox5TTFX1bM7JnS6Z8s4k1uw67HY7xU5ZIzCn95cuNBAg2u28tJSI89aszaBYRyu3TVnHsRIHbIRk/ZInElGnZtkzmrNvLred3pGVkuNvhmAqKrBfMC1f3ZkdmDk/MTnI7HOOHLJGYUhUVKY/P3kDLyDB+d15Ht8MxlTSwQxS3nNeBact3sWDjfrfDMX7GEokp1cyVaazfncX9w6zc11/cNaQzXZpHcP/MdRw6lud2OMaPWCIxv3DMKfft3dbKff1JaFAgL1zdi8M5eTz8xXq3wzF+xOtEIiL1fRmIqTn+9t0WMrJP8Mhl8QQEWLmvP+neKpLbL4pj9tq9dtW7qTKnTSQiMkhEkoBk53UvEXnd55EZV6QdymHqD1sZ3bsVfa3c1y/den5HerdtxMOfr2d/Vq7b4Rg/4M0ZyRRgKHAQQFXXAOf5MijjnpPlvvcPs9l9/VVQYAB/vaoXJwoKuX/mWlTtqndTOV51banqrhKLCn0Qi3FZ4vZMZq/dy+/O60irRlbu6886Rjdg0rCufJeSwbTlJf+8jSkfbxLJLhEZBKiIhIjIPTjdXMZ/FDmz+7ZoGMbvzrfZfeuC68+KYVDHKJ6YncTOgzluh2NqMW8Sya3ABKA1kAb0Bv7gy6BM9fts1W7Wph3h/uFdqBcS5HY4phoEBAjP/boXgSLc8681NrGjqTBvEkkXVb1WVZurajNV/Q3QzdeBmerjKffdSK+2jRjdq7Xb4Zhq1LpROI+O6s6y7Zm8vXib2+GYWsqbRPKKl8tMLfWPRVvYn+WZ3dfKfeueK/u2Zkh8c577OoXU9KNuh2NqoTITiYicJSJ3A9Eiclexx2OAV5c6i8gwEUkRkVQRmVTK+6EiMt15f6mIxBR7b7KzPEVEhhZb3khEZojIRhFJFpGzyrG/poTdh4/zj++3MqpXK/q1t3LfukhEeOryHtQLCeTeGdbFZcrvVGckIUADIAiIKPbIAsacbsMiEgi8BgwH4oFrRCS+RLObgUOq2glPmfEzzrrxwFigOzAMeN3ZHsBLwDxV7Qr0wgb+K+WZLzcCcP9wK/ety5pFhPGnUd1ZtfOwdXGZcitzVFVVFwGLROSfqrqjAtvuD6Sq6lYAEZkGjAaKTz86GnjMeT4DeFU8d04aDUxT1RPANhFJBfqLyAY817Dc4MSYB9ikQRW0Ykcms9bs4bYLO9Hayn3rvFG9WjF77V6e/zqFC7s1o2N0A7dDMrWEN2MkOSLynIjMFZEFJx9erNcaKF6gnuYsK7WNqhYAR4CoU6zbAcgA3hGRVSLyZllTt4jILSKSKCKJGRkZXoRbtxQVKY//O4nmDUP53fk2u685ee+SHoQFB3LfjLXWxWW85k0i+RDYCMQCfwK2A8u9WK+0UduSv5lltSlreRDQF/ibqvYBjgG/GHsBUNWpqpqgqgnR0dFehFu3fL56N2vSjnDf0K7UD7VyX+PRrGEYj42KZ8WOQ7zzo3VxGe94k0iiVPUtIF9VF6nqTcBAL9ZLA9oWe90GKDlL3H/aiEgQEAlknmLdNCBNVZc6y2fgSSymHHLyCnhm3kZ6tonk8j5W7mv+1696t+bibs147qsUth045nY4phbwJpHkOz/3isgIEemD54v9dJYDcSISKyIheAbPZ5VoMwsY7zwfAyxQz8Q/s4CxTlVXLBAHLFPVfXiutD9539eL+N8xF+OFvy/aauW+pkyeKq4zCA0K4F67UNF4wZtE8qSIRAJ3A/cAbwJ3nm4lZ8xjIvAVnsqqT1R1g4g8LiKjnGZvAVHOYPpdON1UqroB+ARPkpgHTFDVk/N7/RH4UETW4rnK/mmv9tQATrnvoi2M7NmShJgmbodjaqjmDcN49LLuJO44xLs/bXc7HFPDyalm/nRKbm9T1SnVF1LVS0hI0MTERLfDqBFun7aKL9fvY8Hd59OmcT23wzE1mKpy87uJ/LTlAPNuP4+YpnZLorpERFaoaoI3bU95RuKcBYw6VRtTe6zYcYgvVu/hlnM7WBIxpyUiPH35GQQHBnDfjLUUWReXKYM3XVs/icirInKuiPQ9+fB5ZKZKFRUpT8xOIjoilN9fYOW+xjstIsN4ZGQ8y7Zn8t6S7W6HY2oob+o+Bzk/Hy+2TIELqz4c4yuz1uxh9a7DPDemp5X7mnIZ068Nc9bt5Zl5KQzu2oz2UdbFZf7Xac9IVHVwKQ9LIrVITl4Bf/lyI2e0juTKvt4U3BnzXyLCn684g6AAsS4uUyqv7pBoarep329lX1Yuj1xm5b6mYlpGhvPQyG4s3ZbJh0srMmOS8WeWSPzc3iPH+fuiLYzo2ZIzrdzXVMJVCW05N64pf/5yI7sy7Y6K5r8skfi5Z+elUKQwaZjN7msqR0T4y5U9CRBh0qdrOdWlA6ZuOe2oq4hcUcriI8A6VU2v+pBMVVm18xCfrdrNhMEdadvEyn1N5bVuFM7kS7vy4Gfr+XjZLsYNaOd2SKYG8OaM5GY8V7Nf6zzewHMV+o8icp0PYzOVoKo8/p9y305uh2P8yLj+7RjUMYqn5yaz+/Bxt8MxNYA3iaQI6KaqV6rqlXhuUnUCGADc78vgTMXNWrOHVTsPc+/QLjSwcl9ThUSEZ67sSZEqkz9dZ11cxqtEEqOq+4u9Tgc6q2om/53Q0dQgx/MKeebLjXRv1ZAxVu5rfKBtk3rcP6wr32/K4F8r0twOx7jMm0Tyg4jMFpHxIjIe+AL43rmh1GHfhmcq4o0ftrLnSK7N7mt86rqB7ekf24QnZiex70iu2+EYF3mTSCYA/8Qz024f4D08s/EeU9XBPozNVMC+I7n87bstXHpGCwZ0iHI7HOPHAgKEZ6/sSX5hEQ9+Zl1cdZk3V7arqs5Q1TtV9Q7nuf3G1FDPzttIYZEyeXg3t0MxdUBM0/rcc0kXvt2Yzuerd7sdjnHJaROJiFwhIptF5IiIZIlItohkVUdwpnxW7zrMp6t2c/O5sVbua6rNjWfH0rddIx6blUR6tnVx1UXedG09C4xS1UhVbaiqEara0NeBmfJRVR7/9waaNgjlDza7r6lGgQHCs2N6cTy/kIc/X29dXHWQN4lkv6om+zwSUyn/XruXlTsPc+/QzkSEBbsdjqljOjVrwF1DOvPVhv3MXrvX7XBMNfPmAoNEEZkOfI7n+hEAVPVTn0VlyiU3v5C/zE0mvmVDxvRr63Y4po76v3Ni+XLdXh6dtYFBHaOIahDqdkimmnhzRtIQyAEuAS5zHiN9GZQpnze+95T7PjwynkAr9zUuCQoM4Llf9+JobgGPzNrgdjimGp32jERVb6yOQEzF7M/K5fXvtjCsewvO6mjlvsZdnZtHcNtFnXj+601c1nMvw3q0dDskUw3KTCQicp+qPisir+C5I+L/UNXbfBqZ8cqz81I85b6X2uy+pmb43fkd+XL9Ph76fD0DYqNoXD/E7ZCMj52qa+vkAHsisKKUh3HZ2rTDzFyZxo3nxNjtT02NERwYwHNjenE4J58//du6uOqCMs9IVPXfzs93qy8c4y1PuW8STRuEMHGwze5rapb4Vg2ZMLgTL327mRE9WzEkvrnbIRkf8uaCxM4iMlVEvhaRBScf1RGcKducdXtJ3HGIey7pYuW+pkaaMLgTXVtE8OBn6ziSY/O7+jNvqrb+BawCHgLuLfYwLsnNL+TPczfSrWVDfp1g5b6mZgoJCuD5X/fi4LE8Hp+d5HY4xoe8uY6kQFX/5vNIjNfeWryN3YeP89yve1q5r6nRerSO5NbzO/Dawi2M7NWSwV2auR2S8QFvzkj+LSJ/EJGWItLk5MPnkZlSpWfl8trCVIZ2b86gjk3dDseY07rtojjimjVg8sx1ZOVaF5c/8iaRjMfTlfUT/63YSvRlUKZsz32VQn5hEQ9carP7mtohNCiQ537di/TsXJ6eY7Mt+aNTJhIRCQB+o6qxJR4dqik+U8y6tCPMWJnGTWfHWrmvqVV6t23Eb8/twLTlu/hhc4bb4ZgqdspEoqpFwPPVFIs5BVXlidlJNKkXwoQLrdzX1D53DulMh6b1mTRzHUdPFLgdjqlC3nRtfS0iV4pIuUd1RWSYiKSISKqITCrl/VARme68v1REYoq9N9lZniIiQ0usFygiq0Rkdnljqq2+XL+PZdszufuSLjS0cl9TC4UFB/Lcr3uy58hx/vKldXH5E28SyV14SoBPlOfGViISCLwGDAfigWtEJL5Es5uBQ6raCZgCPOOsGw+MBboDw4DXne2ddDv/vfLe7+XmF/L03GS6tojg6jOt3NfUXv3aN+HGQbF88PNOftpywO1wTBXx5la7EaoaoKoh5byxVX8gVVW3qmoeMA0YXaLNaODklfMzgIucM5/RwDRVPaGq24BUZ3uISBtgBPCmNzvoD95avI20Q8d5xGb3NX7g3qFdaB9Vj0kz15GTZ11c/sCbMxJEpLGI9BeR804+vFitNbCr2Os0Z1mpbVS1ADgCRJ1m3ReB+4Aib2Kv7dKzcnl9YSpD4pszqJOV+5raLzwkkGeu7MnOzByenZfidjimCngzRcr/Ad8DXwF/cn4+5sW2S/uvc8lZhMtqU+pyERkJpKvqaSeNFJFbRCRRRBIzMmpvlcjzX6eQZ+W+xs8M7BDF+LPa8+6S7Szfnul2OKaSvDkjuR04E9ihqoOBPoA338xpQPEO/TbAnrLaiEgQEAlknmLds4FRIrIdT1fZhSLyQWkfrqpTVTVBVROio6O9CLfmWb/7CP9akcYNg2KIbWrlvsa/3DesK20ah3PfjLUczyt0OxxTCd4kklxVzQVPlZWqbgS6eLHeciBORGJFJATP4PmsEm1m4bngEWAMsEBV1Vk+1qnqigXigGWqOllV26hqjLO9Bar6Gy9iqXVUlcdnJ9G4XggTL4xzOxxjqlz90CCeuaIn2w4c44VvrIurNvMmkaSJSCM892z/RkS+4JdnFr/gjHlMxNMVlgx8oqobRORxERnlNHsLiBKRVDzVYZOcdTcAnwBJwDxggqrWqf+yzFu/j2XbMrlrSGciw63c1/inQZ2aMm5AO95avI2VOw+5HY6pIPGcAHjZWOR8PN1P85xKrFohISFBExNrz6wuufmFDJmyiHrBQcy57RyCAr2qiTCmVsrOzWfolO8JDwlkzm3nEhYcePqVjM+JyApVTfCmrbdVW+eIyI2qughYwi+rr0wVeufH7ezKPM5DI7tZEjF+LyIsmD9f2ZMtGcd4cf5mt8MxFeBN1dajwP3AZGdRMFDqALepvPRsz+y+F3drxrlxtbNIwJjyOr9zNFcltGHq91tYs+uw2+GYcvLmv7uXA6OAYwCqugeI8GVQddkLX2/iREEhD44oOQmAMf7twRHxNIsI465PVpObX6eGRGs9bxJJnlNJpQAiYnWoPrJhzxGmJ+5i/FlW7mvqnsjwYJ4d4+niembeRrfDMeXgTSL5RET+ATQSkd8C84E3fBtW3aOqPP7vJBqFB/PHi6zc19RN53WO5vqz2vPOj9v5MdXm4qotvJlr63k882DNxHP9yCOq+oqvA6trvtqwn6XbMrnrki5W7mvqtMnDu9GhaX3u+dcajhy3OyrWBl6VBKnqN6p6r6reo6rf+DqouuZEgWd2387NG3CNze5r6rjwkECmXN2b9OwTPPrFerfDMV4oM5GcnC6+lIdX08gb7/3zx+3szMzh4ZHxVu5rDNCrbSMmDu7E56v3MGftXrfDMadR5rfWyeniS3l4O4288UJG9gleWZDKRV2t3NeY4iZe2IlebSJ58PN1pGfluh2OOQX776/LXvhmE7n5hTwwwmb3Naa44MAAXri6N7n5hdw7Yy3lmYXDVC9LJC5K2pPF9OU7uf6sGDpGN3A7HGOn1RyqAAATAklEQVRqnI7RDZg8vBuLNmXwwdKdbodjymCJxCWqyhOzk2gYHsztVu5rTJmuG9iec+Oa8vScZLYdOOZ2OKYUlkhc8nXSfpZsPeiZ3beelfsaU5aAAOG5Mb0ICQrgjumryS+sEzdHrVUskbjgZLlvXLMGjOvfzu1wjKnxWkSG8dTlPViz6zAvzt/kdjimBEskLnj3p+3sOJjDQ1bua4zXRvZsxVUJbXj9uy38tMWueq9J7Fusmh04eoJXvk1lcJdozu9s5b7GlMdjo7oT27Q+d01fw6FjteaWSH7PEkk1e+GbTeTk2+y+xlREvZAgXh7bh8xjedw300qCawpLJNUoeW8W05bt5LqB7enUzMp9jamIHq0juW9YF75J2m8lwTWEJZJqoqo8OSeJiLBg7rjYyn2NqYybzo7l/M7RPDk7iZR92W6HU+dZIqkm85PT+TH1IHdeHEejeiFuh2NMrRYQIDz/615EhAVz28er7EZYLrNEUg3yCop4ak4SnZo14NqB7d0Oxxi/EB0Ryl+v6kXK/myenpvsdjh1miWSavDeku1sP5jDQyO6EWzlvsZUmfM7R/Pbc2N5b8kOvlxnswS7xb7VfOzg0RO89O1mLugSzQVdmrkdjjF+596hXendthH3zVjLdptCxRWWSHxsyvxN5OQV8pDN7muMT4QEBfDatX0JDBT+8OFKGy9xgSUSH0rZl81HS0+W+0a4HY4xfqt1o3BeuKoXSXuzeHx2ktvh1DmWSHzk5Oy+EWE2u68x1eHCrs35/QUd+WjpTj5ftdvtcOoUSyQ+smBjOotTD3DHxXE0rm/lvsZUh7uHdKZ/TBMe+Gwdqel2fUl1sUTiA55y32Q6RNfnN1bua0y1CQoM4JVxfQgPDuT3H6wkJ6/A7ZDqBEskPvD+zzvYeuAYD4+It3JfY6pZ84ZhvDS2D6kZR3ng03U2H1c1sG+5KpZ5LI+X5m/ivM7RXNDFZvc1xg3nxDXlros78/nqPbz943a3w/F7lkiq2JRvNnHMKfcVEbfDMabOmjC4E5fEN+fpucl2/xIf82kiEZFhIpIiIqkiMqmU90NFZLrz/lIRiSn23mRneYqIDHWWtRWRhSKSLCIbROR2X8ZfXin7svlw6Q6uHdCOzs2t3NcYNwUECC9c3ZvYpvWZ+NEq0g7luB2S3/JZIhGRQOA1YDgQD1wjIiVvwnEzcEhVOwFTgGecdeOBsUB3YBjwurO9AuBuVe0GDAQmlLJNV5yc3bdBaBB3XNzZ7XCMMUCD0CCmXteP/MIifvf+Co7n2cWKvuDLM5L+QKqqblXVPGAaMLpEm9HAu87zGcBF4ukPGg1MU9UTqroNSAX6q+peVV0JoKrZQDLQ2of74LWFKen8sPkAt1/cmSZW7mtMjdEhugEvj+1D0t4sJn9qN8PyBV8mktbArmKv0/jll/5/2qhqAXAEiPJmXacbrA+wtLQPF5FbRCRRRBIzMjIqvBPeyC8s4snZyXRoWp/rrNzXmBpncNdm3D3EM/j+1uJtbofjd3yZSEobaS75X4Gy2pxyXRFpAMwE7lDVrNI+XFWnqmqCqiZER/u2eur9JZ5y3wdHdCMkyOoXjKmJJgzuxPAeLXh6bjILNu53Oxy/4stvvTSgbbHXbYA9ZbURkSAgEsg81boiEowniXyoqp/6JPJyOHQsjxfnb+LcuKZc2NVm9zWmphIR/npVL+JbNeSPH60ieW+p/wc1FeDLRLIciBORWBEJwTN4PqtEm1nAeOf5GGCBejowZwFjnaquWCAOWOaMn7wFJKvqCz6M3Wsvzt/E0RMFPDQi3sp9janh6oUE8eb1ZxIRFszN/1xOelau2yH5BZ8lEmfMYyLwFZ5B8U9UdYOIPC4io5xmbwFRIpIK3AVMctbdAHwCJAHzgAmqWgicDVwHXCgiq53Hpb7ah9PZvD+bD5buZNyAdnRpYeW+xtQGLSLDeHN8Aody8vnte4lWyVUFpC5UMCQkJGhiYmKVb3f828tYufMQi+4dbJVaxtQy3yTt55b3ExnWvQWvjetLQID1KBQnIitUNcGbtjYyXEELU9JZtCmD2y+KsyRiTC00JL45D17ajS/X7+OZrza6HU6tFuR2ALWRp9w3idim9bn+rBi3wzHGVNDN58Sy/eAx/rFoK80iwrj5nFi3Q6qVLJFUwIc/72BLxjHevD7Byn2NqcVEhD+N6sGB7DyemJ1E0wYhjO5dI65xrlXsW7CcDufkMWX+Zs7p1JSLulm5rzG1XWCA8OLY3vSPbcI9/1rDD5t9ewGzP7JEUk4vzt9Mdm4+D4202X2N8RdhwYG8cX0CHaMbcOv7K1ibdtjtkGoVSyTlkJqezfs/7+Ca/u3o2qKh2+EYY6pQZHgw797Un8b1Q7jxneWkph91O6RawxJJOTw1J5l6wYHcNcRm9zXGHzVvGMZ7N/VHRLj2zZ/ZfuCY2yHVCpZIvPRdSjoLUzK47aI4ohqEuh2OMcZHOkQ34MP/G0BeQRHj3viZXZl2H5PTsUTihYLCIp6ck0xMVD3GD4pxOxxjjI91aRHB+zcP4OiJAsa9+TN7jxx3O6QazRKJFz5cupPU9KM8cKnN7mtMXdGjdSTv3zyAw8fyGffGUpuX6xTsW/E0POW+mxjUMYoh8c3dDscYU416tW3EP286k/SsXK76xxJ2H7Yzk9JYIjmNl77dTNbxfB4eabP7GlMX9WvfhPduHsDBY3lc9fclbLMB+F+wRHIKqelHeX/JDq4+sx3dWlq5rzF1Vb/2jfn4twM5nl/IVf9YQsq+bLdDqlEskZzC03OTCQsO5O5LrNzXmLquR+tIpt8yEAGunrrELlosxhJJGY4cz2dXZg5/vLATTa3c1xgDxDWP4F+3nkWD0CDGTv2ZhSnpbodUI9j9SE6hoLCIIsUqtYwx/2N/Vi43/XM5G/dl8+SvenBN/3Zuh1Tl7H4kVSQoMMCSiDHmF5o3DGP6787inE5NmfzpOv76dQp14T/lZbFvSWOMqYAGoUG8OT6BsWe25ZUFqfzx41Xk5BW4HZYr7H4kxhhTQcGBAfz5ijNoH1WfZ7/aSGr6Ud64PoG2Teq5HVq1sjMSY4ypBBHh9xd05J0bzmTP4eNc9upiFm8+4HZY1coSiTHGVIELujRj1sRzaB4RxvVvL+Wl+ZspLKob4yaWSIwxporENK3Pp38YxOjerZkyfxPXvPEze+rAtCqWSIwxpgrVDw1iytW9eeGqXmzYfYThL/3AvPV73Q7LpyyRGGOMD1zRtw1zbjuX9lH1uPWDlUz8aCUHjp5wOyyfsERijDE+EtO0PjN/P4i7h3Tm6w37GfLCIj5ftdvvrjmxRGKMMT4UHBjAHy+KY85t5xDbtD53TF/NuDeWkrw3y+3QqowlEmOMqQaeeboG8cTo7iTvy2LEyz/wwGfrOOgH3V0215YxxlSzwzl5vDh/M+//vIOwoABuODuG/zunA43rh7gd2n+UZ64tSyTGGOOS1PRspszfzNx1e6kXHMj4QTHcMCiGZg3D3A7NEklJlkiMMTXZpv3ZvPztZuas20tQgDC8R0vGD2pP33aNXbsza42Z/VdEholIioikisikUt4PFZHpzvtLRSSm2HuTneUpIjLU220aY0xt07l5BK+O68uCuy/guoExLExJ58q/LWHIlO95+dvNbK/ht/f12RmJiAQCm4AhQBqwHLhGVZOKtfkD0FNVbxWRscDlqnq1iMQDHwP9gVbAfODkbQpPuc3S2BmJMaY2OXaigC9W7+Hz1btZti0TgPiWDTmvczTnxTWlX0xjQoMCfRpDec5IfDn7b38gVVW3OkFNA0YDxb/0RwOPOc9nAK+K5zxuNDBNVU8A20Qk1dkeXmzTGGNqtfqhQYwb0I5xA9qx5/BxZq/dw7fJ6bz5w1b+vmgLIYEBdG0ZwRmtI4lv1ZB2TerRtnE9WjUKd+UeSr5MJK2BXcVepwEDymqjqgUicgSIcpb/XGLd1s7z023TGGP8RqtG4dxyXkduOa8jR08UsGTLQRK3Z7I27Qiz1uzhw6U7/6d9eHAgEWFBNAgNommDUD659Syfx+jLRFLaCFHJfrSy2pS1vLRUW2rfnIjcAtwC0K6d/90G0xhT9zQIDWJIfHOGxDcHoKhI2ZuVy67MHHZm5rD3cC5HT+STnVvA0RMFNAitnltO+fJT0oC2xV63AfaU0SZNRIKASCDzNOuebpsAqOpUYCp4xkgqtgvGGFNzBQQIrRuF07pROAM7RLkXhw+3vRyIE5FYEQkBxgKzSrSZBYx3no8BFqhn9H8WMNap6ooF4oBlXm7TGGNMNfLZGYkz5jER+AoIBN5W1Q0i8jiQqKqzgLeA953B9Ew8iQGn3Sd4BtELgAmqWghQ2jZ9tQ/GGGNOzy5INMYY8ws15oJEY4wx/s8SiTHGmEqxRGKMMaZSLJEYY4ypFEskxhhjKqVOVG2JSAawo4KrNwUOVGE4tVFdPwZ1ff/BjgHUvWPQXlWjvWlYJxJJZYhIorclcP6qrh+Dur7/YMcA7BicinVtGWOMqRRLJMYYYyrFEsnpTXU7gBqgrh+Dur7/YMcA7BiUycZIjDHGVIqdkRhjjKkUSyRlEJFhIpIiIqkiMsnteHxJRLaLyDoRWS0iic6yJiLyjYhsdn42dpaLiLzsHJe1ItLX3egrRkTeFpF0EVlfbFm591lExjvtN4vI+NI+q6Yq4xg8JiK7nd+F1SJyabH3JjvHIEVEhhZbXiv/VkSkrYgsFJFkEdkgIrc7y+vU70GVUFV7lHjgmaJ+C9ABCAHWAPFux+XD/d0ONC2x7FlgkvN8EvCM8/xS4Es8d7EcCCx1O/4K7vN5QF9gfUX3GWgCbHV+NnaeN3Z73yp5DB4D7imlbbzzdxAKxDp/H4G1+W8FaAn0dZ5HAJuc/axTvwdV8bAzktL1B1JVdauq5gHTgNEux1TdRgPvOs/fBX5VbPl76vEz0EhEWroRYGWo6vd47oFTXHn3eSjwjapmquoh4BtgmO+jrxplHIOyjAamqeoJVd0GpOL5O6m1fyuquldVVzrPs4FkoDV17PegKlgiKV1rYFex12nOMn+lwNcissK51z1Ac1XdC54/OKCZs9yfj01599lfj8VEp+vm7ZPdOvj5MRCRGKAPsBT7PSg3SySlk1KW+XN529mq2hcYDkwQkfNO0bauHRsoe5/98Vj8DegI9Ab2An91lvvtMRCRBsBM4A5VzTpV01KW+cUxqCxLJKVLA9oWe90G2ONSLD6nqnucn+nAZ3i6K/af7LJyfqY7zf352JR3n/3uWKjqflUtVNUi4A08vwvgp8dARILxJJEPVfVTZ3Gd/z0oL0skpVsOxIlIrIiE4LmX/CyXY/IJEakvIhEnnwOXAOvx7O/J6pPxwBfO81nA9U4Fy0DgyMluAD9Q3n3+CrhERBo7XUCXOMtqrRLjXZfj+V0AzzEYKyKhIhILxAHLqMV/KyIiwFtAsqq+UOytOv97UG5uj/bX1AeeCo1NeCpSHnQ7Hh/uZwc8lTZrgA0n9xWIAr4FNjs/mzjLBXjNOS7rgAS396GC+/0xnq6bfDz/o7y5IvsM3IRn4DkVuNHt/aqCY/C+s49r8XxxtizW/kHnGKQAw4str5V/K8A5eLqg1gKrncelde33oCoedmW7McaYSrGuLWOMMZViicQYY0ylWCIxxhhTKZZIjDHGVIolEmOMMZViicSY0xCRn5yfMSIyroq3/UBpn2VMbWLlv8Z4SUQuwDMz7shyrBOoqoWneP+oqjaoiviMcYudkRhzGiJy1Hn6F+Bc5z4dd4pIoIg8JyLLnUkOf+e0v8C5z8VHeC5cQ0Q+dybF3HByYkwR+QsQ7mzvw+Kf5Vw9/ZyIrBfPvWKuLrbt70RkhohsFJEPnSu0EZG/iEiSE8vz1XmMTN0W5HYAxtQikyh2RuIkhCOqeqaIhAI/isjXTtv+QA/1TLkOcJOqZopIOLBcRGaq6iQRmaiqvUv5rCvwTJzYC2jqrPO9814foDue+Zx+BM4WkSQ8U5p0VVUVkUZVvvfGlMHOSIypuEvwzL20Gs/041F45qACWFYsiQDcJiJrgJ/xTPAXx6mdA3ysngkU9wOLgDOLbTtNPRMrrgZigCwgF3hTRK4Aciq9d8Z4yRKJMRUnwB9VtbfziFXVk2ckx/7TyDO2cjFwlqr2AlYBYV5suywnij0vBIJUtQDPWdBMPDdimleuPTGmEiyRGOO9bDy3ZD3pK+D3zlTkiEhnZwblkiKBQ6qaIyJd8dym9aT8k+uX8D1wtTMOE43ntrjLygrMuadGpKrOBe7A0y1mTLWwMRJjvLcWKHC6qP4JvISnW2mlM+CdwX9vy1rcPOBWEVmLZ+bcn4u9NxVYKyIrVfXaYss/A87CMyuzAvep6j4nEZUmAvhCRMLwnM3cWbFdNKb8rPzXGGNMpVjXljHGmEqxRGKMMaZSLJEYY4ypFEskxhhjKsUSiTHGmEqxRGKMMaZSLJEYY4ypFEskxhhjKuX/Ac4qJ6c+2OXyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_text=False, show_moms=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A customized LR Finder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This API can also be used to run a learning rate finder: just put a very low starting LR and a very large ending one, and choose exponential or linear decay. To stop when the loss go wild, we add the option stop_div= True. This can also be used in a regular fit if you want to stop the training in case the loss suddenly spikes.\n",
    "\n",
    "As you can choose the number of epochs, this is particularly useful when you have a small dataset (you can run more epoxhs to have a clearer curve) or a very large one where a fraction of an epoch is plenty. In general 100-200 values are plenty to have a clear curve, so since we know we have 782 batches on cifar10, we'll run this customized LR Finder on 0.25 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=0.25, opt_fn=optim.SGD, lr=(1e-5,10), lr_decay=DecayType.EXPONENTIAL,momentum=0.9)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.from_model_data(SimpleNet([32*32*3, 40,10]), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be careful to save your model before fitting as the LR Finder will alter it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6a81fb10dc94eac91627af66af8ba44",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 18%|████████████▍                                                          | 137/782 [00:03<00:14, 45.64it/s, loss=17]\n",
      "                                                                                                                       \r"
     ]
    }
   ],
   "source": [
    "learn.save('tmp')\n",
    "learn.fit_opt_sched(phases, stop_div=True)\n",
    "learn.load('tmp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can draw the curve with the usual command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEOCAYAAABmVAtTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8nGW5//HPNZO12Zo2Sbe0TUs3SikFAoilUCqg0lJQVg+giIKoYJWDKEdf/jzoOSiIsomCqKDIQQFBNilrW8pSSFcKpQvd16Rt0qZJM81y/f6Y6RBK0qYlsyXf9+uVV2ee55nnuXJ3Mtfcy3Pf5u6IiIgABBIdgIiIJA8lBRERiVJSEBGRKCUFERGJUlIQEZEoJQUREYlSUhARkSglBRERiVJSEBGRKCUFERGJSov1BcwsCFQAG9x9yj77LgNuATZENt3l7vft73xFRUVeVlYWg0hFRLquuXPnbnX34gMdF/OkAEwDlgD57ez/u7tf3dGTlZWVUVFR0SmBiYh0F2a2piPHxbT5yMxKgcnAfr/9i4hIcoh1n8JtwPVAy36OOdfMFpnZo2Y2MMbxiIjIfsQsKZjZFKDS3efu57CngDJ3Hwu8CDzQzrmuNLMKM6uoqqqKQbQiIgKxrSmMB6aa2WrgYWCSmT3Y+gB33+buocjTPwDHtnUid7/X3cvdvby4+ID9JCIicohilhTc/QZ3L3X3MuAi4GV3v6T1MWbWr9XTqYQ7pEVEJEHiMfroI8zsRqDC3Z8EvmNmU4EmYDtwWbzjERGRD1mqLcdZXl7uGpIqIt1JqKmZGUurGN0vn4G9ehzSOcxsrruXH+g43dEsIpLkdtQ38o2/zmXW8tgPtFFSEBFJcqGm8Kj+zLRgzK+lpCAikuQaGpsByEyL/Ue2koKISJL7sKagpCAi0u2FmiI1hXQ1H4mIdHuhxnBNIUs1BRERiTYfqaYgIiLR5iPVFEREpKFRHc0iIhKhjmYREYnSkFQREYmKjj5STUFERNTRLCIiUaGmFgIGaQGL+bWUFEREklxDYzOZaUHMlBRERLq9UFMLmenx+bhWUhARSXKhxpa49CeAkoKISNILNTXHZS0FUFIQEUl6oaYWstR8JCIiEOlTUE1BRERg7+gj1RRERASNPhIRkVbU0SwiIlEakioiIlHh0UeqKYiICHubj1RTEBERwiuvKSmIiAgQqSmo+UhERNw9cvOaagoiIt1eY7PjHp8FdiAOScHMgmY238ye3s8x55mZm1l5rOMREUkle1dd60qjj6YBS9rbaWZ5wHeAOXGIRUQkpYSawuszd4magpmVApOB+/Zz2M+Am4GGWMYiIpKKPkwKXaOmcBtwPdDS1k4zOxoY6O7tNi2JiHRnDY3h5qOUn/vIzKYAle4+t539AeA3wH924FxXmlmFmVVUVVV1cqQiIskr1Nh1mo/GA1PNbDXwMDDJzB5stT8PGAPMiBzzKeDJtjqb3f1edy939/Li4uIYhiwiklz2djSnfPORu9/g7qXuXgZcBLzs7pe02r/D3YvcvSxyzJvAVHeviFVMIiKppkt1NLfFzG40s6nxvq6ISCqKJoU4DUlNi8dF3H0GMCPy+CftHDMxHrGIiKSS0N6O5q5aUxARkY5riNQUslJ99JGIiHxyH9YUUryjWUREPrku39EsIiId19XuaBYRkU8gep+C+hRERKQr3dEsIiKfUENTMxlpAcwsLtdTUhARSWKhOK7PDEoKIiJJLbwUZ3w6mUFJQUQkqYWamlVTEBGRsFBTS9zuZgYlBRGRpBbuU1DzkYiIEGk+Uk1BRERAo49ERKSVcEezmo9ERIS9Q1JVUxARESJJIU6rroGSgohIUgs1NpOlmoKIiMDemoKSgoiIAA2N6mgWEZEIdTSLiAgATc0tNLW4agoiIgJ7miML7KhPQURE9q66ptFHIiJCqGlvTUHNRyIi3V5DYzMQv/WZQUlBRCRpRWsK6mgWEZFQk2oKIiIS8WGfgpKCiEi3Fx191JU6ms0saGbzzezpNvZdZWbvmNkCM5ttZqNjHY+ISKroqs1H04Al7ex7yN2PdPdxwM3Ar+MQj4hISmho7GIdzWZWCkwG7mtrv7vvbPU0B/BYxiMikkoSUVNIi/H5bwOuB/LaO8DMvg1cC2QAk2Icj4hIyuhSHc1mNgWodPe5+zvO3X/r7ocBPwB+3M65rjSzCjOrqKqqikG0IiLJJxS9ea1rNB+NB6aa2WrgYWCSmT24n+MfBs5pa4e73+vu5e5eXlxc3PmRiogkoQ9vXusCNQV3v8HdS929DLgIeNndL2l9jJkNb/V0MrA8VvGIiKSaRCSFWPcpfIyZ3QhUuPuTwNVmdhrQCFQDX4l3PCIiyap+TzPpQSMt2MWSgrvPAGZEHv+k1fZp8bi+iEgqqm1oJC8rPa7X1B3NIiJJamdDE/lZ8W3QUVIQEUlSqimIiEhUbUMTeaopiIgI7K0pKCmIiAh7awpqPhIREdR8JCIiEc0tzq6QagoiIgLsCjUBaEiqiIiEO5kBNR+JiEi4PwFQ85GIiLROCqopiIh0ex82H6mmICLS7ammICIiUXtrCvnJWFMws2lmlm9hfzSzeWZ2RqyDExHprnYmeU3hcnffCZwBFANfBX4Rs6hERLq5nQ2NZAQDZKXHb31m6HhSsMi/ZwJ/dveFrbaJiEgnS8QUF9DxpDDXzJ4nnBSmm1ke0BK7sEREurdEJYWOXvFrwDhgpbvXm1kvwk1IIiISA4lYYAc6XlM4EVjq7jVmdgnwY2BH7MISEenekr356HdAvZkdBVwPrAH+ErOoRES6uUQssAMdTwpN7u7A2cDt7n47kBe7sEREurdELLADHe9TqDWzG4BLgQlmFgTiH62ISDeR7M1HFwIhwvcrbAYGALfELCoRkW4sUQvsQAeTQiQR/A0oMLMpQIO7q09BRCQGErXADnR8mosLgLeA84ELgDlmdl4sAxMR6a4StcAOdLxP4UfAce5eCWBmxcCLwKOxCkxEpLvaO0NqvCfDg473KQT2JoSIbQfxWhEROQg7dydmLQXoeE3hOTObDvxf5PmFwLOxCUlEpHtL1FoK0MGk4O7fN7NzgfGEJ8K7190fj2lkIiLdVG0o+fsUcPfHgMdiGIuIiNC6ppBkzUdmVgt4W7sAd/f8A10gcqNbBbDB3afss+9a4OtAE1BF+D6INR2MXUSkS0ra5iN374ypLKYBS4C2Esh8oDwy8+o3gZsJ91eIiHRbiVpgB2I8gsjMSoHJwH1t7Xf3V9y9PvL0TaA0lvGIiKSCRE1xAbEfVnob4VlVO7Igz9eAf7e1w8yuNLMKM6uoqqrqzPhERJJOl0wKkekwKt19bgeOvQQop535lNz9Xncvd/fy4uLiTo5URCS5JGqBHTiI0UeHYDww1czOBLKAfDN70N0vaX2QmZ1G+I7pU9w9FMN4RERSQpesKbj7De5e6u5lwEXAy20khKOBe4Cp+9wxLSLSbSVqgR1IwFQVZnajmU2NPL0FyAUeMbMFZvZkvOMREUk2iVpgB2LbfBTl7jOAGZHHP2m1/bR4XF9EJFW4O9vr9lDYIzFJQZPaiYgkka279hBqamFAz+yEXF9JQUQkiWyo2Q3AgMIeCbm+koKISBLZUB1JCgmqKSSmezsBZi6r4vF561m2ZRert9UxvCSX0w7vw6eHFVHWuwe9cjIwMwBCTc3MXFrF3LXV5Gel0zsng145GfTOzaTFnfc27mR5ZS2FPTIY1KsHY0t7MqJPbvT1IiKHakNNeJKHAYVKCjG1dns9b63azvA+eRw7uJB3Nuzg1heWcesLywDokRGkKDeTwpwMVlbuojbURDBgNLe0NR9geKKqulATe3cPK8nlM4eX4A419XvY09RCi0NDYzNbd4Worm9kzIACJh/ZjxF9clm1tY5VW+vYtKOBzTsbyE4PMrwkl/49s9m6K8SWnSF6ZATpm59FMGB8ULWLtdvryU4P0is3g149wkmqd04GPXukU9gjg74FWQmZK0VEOs+G6t3kZaZRkN2FRx8lg0tOGMSlnxr8kW2VOxtYuH4H67bXs666nm279rC9bg+fHdOXs47qz6cP601zi7Otbg/bd+1ha10IHA7vl0+f/Ewam50NNbuZvWIrTy/cyL2zVpKZFqAgO52s9CABM9KDRnFeJiV5WcxeXsVTCzd+JIYeGUH65GexK9TEo3PXR7enB43G5g8TUlrAGFCYTaixhe11e9jT/PGZQwIGg3vnMLQoh9LCbPr3zCYjLUCLh0c0BANGWjBA3/wsBvbKJicjjRZ3stLDMYhI4m2o2Z2wWgJ0o6TQVtNOSX4Wp4/e/4dhejDcttdW+15GmjGkKIchRTlc+qnBNLeEP3jb09Tcwhsrt7F5RwNDi3M5rDiHguz0aGw19XvYsjNEUW64uWpPcwuVO0PsaW5hYGEPMtLCXUDuTt2eZrbtCrGtbg876hvZXreHNdvrWVFZy8qqOt5atZ3aUFOHy2dEn3Bz2meP6MvY0gI1hYkkyPrq3QnrT4BulBTiYX8JASAtGGDC8PbnburZI4OePTKizzPTggzs9fERCGZGbmYauZlpDO6d0+75ahsaaW7x6Ae8uxNqamHTjgbWV9dTv6eZtICxvW4PLy2p5J5ZK7l7xgf0L8hi4qgSxpX2ZMyAAoYU5ZCdoWYpkXjYULOb44f0Stj1lRS6sPbuiOyTn8W4gT0/su3rE4ZSUx9ODv9evJmnFmzkoTlro/tL8jIpyc8kNzONntkZjOibxxH98zmurBe9cjL2vYSIHIKdDY3UNjTRXzUFSQY9e2Rw7rGlnHtsKS0tzuptdby7cSdrttWxdnu4z6U21MTyylqef28zLR7uxygv68UZo8NNT23VbESkYxI9HBWUFKQdgYAxtDiXocW5be7fvaeZ9zbtZObSSp5/bws/f2YJP39mCaP65lHYI4OmlhZK8rM4a2w/Jo4s0agokQ6IJgV1NEuqyc4IcuzgQo4dXMi1Z4xkzbY6nn93CzOXVRFqaiYtEGDOym08s2gTuZlpHFdWyHFDenHa4X0Y0aczVnkV6Xr23s1cqpqCpLrBvXO44uShXHHy0Oi2puYWXv9gG/9evJm3Vm3jlaVV3PzcUk4Y0otLTxzMGaP7RkdUiUg4KWQEAxTlZiYsBiUFiZm0YICTRxRz8ojwiKuq2hCPzVvPg2+u4eqH5tMrJ4NzjxnAMYMKKc7LZGhxrjqtpVvbULOb/j2zCBxgJGMsKSlI3BTnZXLVKYdxxYShvLq8ioffWsefX1vNH15dBUBmWoBrJg3jipOHkpmmPgjpfjZUJ/bGNVBSkAQIBoyJI0uYOLKEHbsb2VC9m8raBv5RsY5fPb+Mx+Zt4PzyUiaOKOHwfnm6kU66jQ01uzl1ZGLXoVdSkIQqyE6nIDud0eQzcWQJM5dV8avpS7n5ufBPdnqQfgVZ9OuZRb+CbPoXZHHC0N6cOLR3QqvYIp2tobGZqtoQA3omdli3koIklVNGFHPKiGK27Gxg5rIqlm6uZdOO3WysaeDV5VVU1oa44+UVDOiZzZSj+kXvutb9EZLqNu1oABI7HBWUFCRJ9cnP4oLygR/b3tDYzPPvbeGRinXc9+qq6Cy2x5f14upJw5gwvEjNTZKSkuHGNVBSkBSTlR5k6lH9mXpUfxoam1m6uZY5q7bx59dW8+U/vcXIPnlMHtuPM4/sy7AS3Q8hqWNFZS0AZUVqPhI5JFnpQY4a2JOjBvbkK58u4/F5G3h07np+/cIyfv3CMj4zqoTvnT6CMQMKEh2qyAHNX1dDn/xM+iZ4GnslBekSMtOCXHT8IC46fhBbdjbwSMU67p21kil3zuaYQT0pL+vFiUN7M3FksZqXJCnNX1vDuIE9E/7+1O2k0uX0yc/i6knDefUHk/jP00dgZtz/2mq+ev/bfOHu15m7ZnuiQxT5iG27QqzdXs/RgwoTHYpqCtJ1FWSnc81nhnPNZ4YTamrmXws28qvpSzn3d28wZWw/fvj5UZQWatSSJN6CdTUAHL3PlPaJoKQg3UJmWpALygcy+ch+3DNrJffM/IAX3tvC+eWljB3Qk8P75TNmQH7Cq+7SPS1YV0MwYBxZmvj+LyUF6VZyMtO49vQRXHjcQG557n0eqVjPg2+GFxM6fXQfbj53LIWaf0nibP7aGkb2yaNHRuI/ktWnIN3SgJ7Z3HbR0bx34+eYcd1Efvj5UcxYWsnnbp/F7OVbEx2edCMtLc7CdTUcPSjxTUegpCDdXDBglBXlcNUph/H4t8aTk5nGJX+cw3cfnk9VbSjR4Uk38EHVLmpDTR9bIjdREl9XEUkSYwYU8Ox3JnD3jA/4/YwPePadzZTkZ9I7N5MJw4q4bHxZQue5l65p/tpIJ3MSjDwCJQWRj8hKD3Lt6SM4e1x//v72OqpqQ2ys2c1vZ6zgD6+u5ILygVwxYSiDemvUknSO+euqyc9KY2hRTqJDAeKQFMwsCFQAG9x9yj77TgZuA8YCF7n7o7GOR6QjDivO5b/OPDz6/IOqXdw7cyUPv72Wv81Zw+Sx/TlzTF+OG9JLtQf5ROatqeGogT2TZtbfeNQUpgFLgPw29q0FLgOui0McIofssOJcfnneWK49YwR/mr2Kh+as5amFGwEoH1zI/zvriKQYTiipZUd9I8sqa5k8tl+iQ4mKaUezmZUCk4H72trv7qvdfRHQEss4RDpLn/wsbjjzcOb95HT++a1P8/3PjmT1tnqm/nY2//X4O2yv25PoECWFzFtbjXv4i0WyiHVN4TbgekDTVUqXkh4McMygQo4ZVMilJw7mtheW88Abq3lq4UamfWY4Xz6xjIw0De6T/atYs51gwBiXJMNRIYY1BTObAlS6+9xOONeVZlZhZhVVVVWdEJ1I58nPSucnZ43muWkTOHpQIT9/Zgmfu20WLy3ZgrsnOjxJYhWrqzmif35S3LS2Vyy/yowHpprZauBhYJKZPXgoJ3L3e9293N3Li4sTu36pSHuG98njga8ex58uKweDrz1QwaV/fIvFG3YkOjRJQnuaWliwrobywb0SHcpHxCwpuPsN7l7q7mXARcDL7n5JrK4nkgzMjEmj+jD9uyfzkymjWbxxB1PunM20h+ezObLcogjAuxt3EGpqobwsefoTIAF3NJvZjWY2NfL4ODNbD5wP3GNm78Y7HpFYSA8GuPykIcz8/ql8a+JhTH93M5PveJXXVmgKDQmrWF0NJFcnM4ClWptneXm5V1RUJDoMkYOyorKWqx6cx8qqXVwxYSjnHVvK8D4af9GdfeOvFSzZVMus60+Ny/XMbK67lx/oOA2PEImDYSV5/Ovb4zl73ADumbWS038zi9N+PZM/v7aKulBTosOTOHN3KlZXJ13TESgpiMRNTmYav7lwHG/e8Bl+dvYR5Gel8d9PvceJN73ET598lzdXbqO5JbVq7nJoFq7fwba6PUnXyQxqPhJJqHlrq/njq6t4YckW9jS1kJ+VxvA+eQwpyuGYQYVMGlVC34LELuQunau5xfnC3a+xsWY3L107kYIe6XG5bkebj5JncKxIN3TMoEKOubiQulATM5dVMXvFVlZW7WLG0ioenbsegLGlBfzH8YM4e9wAsjOCCY5YPqm/vrGaRet3cPtF4+KWEA6GagoiScjdWbZlFy+/X8kT8zewdEst+VlpXP+5UVx8wiAtG5qiNu3YzWm3zuTYsl488NXj4vr/qJqCSAozM0b2zWNk3zyuOmUob6+u5vaXlvHjJxYza1kVv9SyoSnp5ueW0uzOz88ek7SJXR3NIknOzDh+SC/+evkJ/Hjy4byytJIz73iVRetrEh2aHITVW+v414INfPnEsqRej0NJQSRFBALG1ycM5Z/fHE/AjPN+/wYPv7WW+j0a0poKfjfjA9KCAb4+YUiiQ9kvNR+JpJgjSwt46pqTuPqhefzwn+9ww+PvUFqYzfCSPA4rzmHMgAImH9mPtKC+8yWL9dX1PDZvPRefMIiSvOQeTaakIJKCeuVk8JfLj+fl9yt5f3Mty7bUsqJyF6+t2EqoqYU7X17Bf505ilNHliRt23V3cs/MlZjBN045LNGhHJCSgkiKSgsGOOOIvpxxRN/otuYW54X3tvDL597n8vsruKC8lP/5wpGkq9aQMDvqG/l7xTrOPaaU/j2zEx3OASkpiHQhwYDxuTF9mTSqhDteWs5dr6ygsjbEb//jGHIy9eeeCM8u3sSephYuPmFwokPpEH19EOmCMtICXPfZkdz0xSOZtayKs+6azT/eXkdDY3OiQ+t2Hp+/IdLX09Yy9clHSUGkC/vS8YP481ePJyMY4PrHFnHSL1/m1ueXsrFmd6JD6xbWV9fz1qrtnDNuQMr07ag+KdLFnTKimJOHF/H6B9v40+xV3PXKCn77ygo+P6Yf3z51GKP7p8Y32FT0rwUbATh73IAER9JxSgoi3YCZMX5YEeOHFbFuez0PzlnDQ2+u5Zl3NjFheBFlvXMozMlgyth+jNA6D53C3Xli/gbKBxcm9c1q+1LzkUg3M7BXD274/OHM/sEkvnfaCDbtaODpRRu58+XlnHXnbB58cw2pNidaMnp3406WV+7inKNTp5YAqimIdFsFPdKZdtpwpp02HICq2hD/+chCfvzEYt5atZ3fXDiOYCA12sGTTV2oiR88tojczDQmH9kv0eEcFNUURASA4rxM7r/sOK49fQRPLtzI7S8uS3RIKam5xZn28AKWbNrJnV86OuUmLlRNQUSiAgHjmknDWLe9njteXsG4QT2ZNKpPosNKKTc/9z4vLtnCjWcfwamjShIdzkFTTUFEPsLM+Nk5Yziifz7ffXgBr7xfSYuWCe2QOSu3cc+slVx8wiC+fGJZosM5JEoKIvIxWelBfnfxseRlpfPV+9/mtF/P5NG569UBvR8Njc388J/vMLBXNj+afHiiwzlkSgoi0qZBvXvwynUTuf2iceRmpXHdIwv5+gMVVNWGEh1aUrrtxeWs2lrHL744lh4Zqdsyn7qRi0jMZaQFOHvcAM4a25/7X1/NL557n9N/M5MzRvdhwvBiTh5enJTrDMfbB1W7+MOrK7mwfCDjhxUlOpxPRElBRA4oEDAuP2kIJw0v4vYXl/Pc4s38o2I9wYBxfFkvphzVjy8dN4hANx3COv3dzTS3ONeeMSLRoXxiSgoi0mEj+uTx24uPoam5hYXra3j5/UpeeG8LP3p8Ma9/sI1bzz+KrPRgosOMu5lLqxjdL58++cm9gE5HqE9BRA5aWjDAsYN78f3PjmL6d0/mhs+P4plFm/jSH97sdn0OtQ2NzF1TzSkjixMdSqdQUhCRT8TM+MYph/H7S45hyaadnHXnbOatrU50WHHz+gfbaGpxTh6upCAiEvW5Mf147JufJiMtwIX3vMHf5qxJdEhxMXNZFTkZQY4dXJjoUDqFkoKIdJoj+hfw1NUncdKwIn70+GJu+veSLn3jm7szc2kVnx5WREZa1/g4jflvYWZBM5tvZk+3sS/TzP5uZivMbI6ZlcU6HhGJrYIe6dz3leO49FODuWfmSq79xwJCTV1zxbcPqurYULObU0Z0jaYjiE9NYRqwpJ19XwOq3X0Y8Bvgl3GIR0RiLBgwbjz7CL7/2ZE8sWAjl9w3h227ul4H9MxlVQBKCh1lZqXAZOC+dg45G3gg8vhR4DOWKmvWich+mRnfPnUYd37paBat38HUu15jxtJK6kJNiQ7tE2tpcR54fTW/mr6UUX3zGNgrdRbROZBY36dwG3A90N5STgOAdQDu3mRmO4DewNYYxyUicXLWUf0Z1KsHV/ylgsv+/DYBg6MHFfKbC8al1IpkW3Y28PSiTazdVsf8dTUsWr+Dk0cU84svHpno0DpVzJKCmU0BKt19rplNbO+wNrZ9rFfKzK4ErgQYNGhQp8UoIvFx1MCevHzdRCpWb2fummr++uYazv396/zl8uM5vF/yrxH96vIqpj28gO11e8jLTGNQ7x788twjuaB8IF2tccNiNeuhmd0EXAo0AVlAPvBPd7+k1THTgZ+6+xtmlgZsBop9P0GVl5d7RUVFTGIWkfhYvqWWL//pLXaFmvj5OWOYMrZ/UqzytmN3I6u21tHiTkNjM+u372bRhhr+Nmctw0tyues/jmF4SW5KJgIzm+vu5Qc8Lh5T4UZqCte5+5R9tn8bONLdrzKzi4AvuvsF+zuXkoJI17ChZjdfu/9t3t9cy6BePbj61GGcX16asA/cNz7YxjX/N4+tu/Z8ZHswYHzh6AHcePYRKT37aUeTQtx/QzO7Eahw9yeBPwJ/NbMVwHbgonjHIyKJMaBnNs9+ZwLPv7eF381YwfWPLeK5dzdzy3lj6Z2bGbc43J37Xl3FL557n8G9e/Dzc8aQlR4kIxhgQGE2/Xtmkx7sGvcgdERcagqdSTUFka7HPTya53///T75WemcPrqEw4pzKc4LJ4fs9CCnjCwmM63zJ9v77SsruGX6Uj4/pi83nzeWvKyuORV40tYURET2ZWZcNn4IJwztzf8+u4TnFm+mur7xI8f0K8jimxMP46RhRQQDRmNzCxtrGti8s4GduxvZ2dBEVnqAI/oX0K8gixeXbOGphZsIBuDUkSWMH1bEkKIcinMzo1N8/+PtddwyfSlfOHoAt55/VLed+rs11RREJClt2xWiur4RM1i7vZ67X1nB26sPbqK98sGFBMyYu7aa5sh0GxlpAQb0zKZ/zyzeXLmd8cOK+ONXyrt8E5FqCiKS0nrnZkb7Fg4rzmXiiGLmra1mffVumlucYMDom59Fv4Jseuakk5ORxq6GJt7dtIN12+s5cWhR9D6IHfWNzFsXfu367fWsr9nN+urdnHZ4Cb++YFyXTwgHQzUFEZFuoKM1BaVHERGJUlIQEZEoJQUREYlSUhARkSglBRERiVJSEBGRKCUFERGJUlIQEZGolLt5zcyqgDWH+PICYEcMX7e/49rb19HtB3peRGxXrDuUsuuMctvf/ra2d2RbPMsuke+5/e3vjLJLxvfcwbzuUP5e29uXCn+vg939wItJu3u3+QHujeXr9ndce/s6ur0DzyuSrew6o9wOtuw6si2eZZfI91ysyy4Z33OdVXYHu68r/L3u/eluzUdPxfh1+zuuvX0d3X6swsD9AAAHp0lEQVSg57F2KNfrjHLb3/62tndkWzzLLpHvuf3tV9kd2t9re/u6wt8rkILNR9I2M6vwDsxrIh+nsjs0KrdDl8xl191qCl3ZvYkOIIWp7A6Nyu3QJW3ZqaYgIiJRqimIiEiUkoKIiEQpKYiISJSSQjdgZjlmNtfMpiQ6llRiZoeb2e/N7FEz+2ai40klZnaOmf3BzP5lZmckOp5UYmZDzeyPZvZoIq6vpJDEzOxPZlZpZov32f45M1tqZivM7IcdONUPgH/EJsrk1Bll5+5L3P0q4AIgKYcPxkInld0T7n4FcBlwYQzDTSqdVHYr3f1rsY20fRp9lMTM7GRgF/AXdx8T2RYElgGnA+uBt4EvAUHgpn1OcTkwlvAt9VnAVnd/Oj7RJ1ZnlJ27V5rZVOCHwF3u/lC84k+kziq7yOtuBf7m7vPiFH5CdXLZPeru58Ur9r3S4n1B6Th3n2VmZftsPh5Y4e4rAczsYeBsd78J+FjzkJmdCuQAo4HdZvasu7fENPAk0BllFznPk8CTZvYM0C2SQie97wz4BfDv7pIQoPPed4mkpJB6BgDrWj1fD5zQ3sHu/iMAM7uMcE2hyyeE/TiosjOzicAXgUzg2ZhGlvwOquyAa4DTgAIzG+buv49lcEnuYN93vYH/AY42sxsiySNulBRSj7Wx7YBtgO5+f+eHknIOquzcfQYwI1bBpJiDLbs7gDtiF05KOdiy2wZcFbtw9k8dzalnPTCw1fNSYGOCYkk1KrtDp7I7dClVdkoKqedtYLiZDTGzDOAi4MkEx5QqVHaHTmV36FKq7JQUkpiZ/R/wBjDSzNab2dfcvQm4GpgOLAH+4e7vJjLOZKSyO3Qqu0PXFcpOQ1JFRCRKNQUREYlSUhARkSglBRERiVJSEBGRKCUFERGJUlIQEZEoJQWJOTPbFYdrTO3gNOKdec2JZvbpQ3jd0WZ2X+TxZWZ2V+dHd/DMrGzfKZ/bOKbYzJ6LV0wSf0oKkjIiUxC3yd2fdPdfxOCa+5sfbCJw0EkB+C/gzkMKKMHcvQrYZGbjEx2LxIaSgsSVmX3fzN42s0Vm9t+ttj8RWR3uXTO7stX2XWZ2o5nNAU40s9Vm9t9mNs/M3jGzUZHjot+4zex+M7vDzF43s5Vmdl5ke8DM7o5c42kze3bvvn1inGFm/2tmM4FpZnaWmc0xs/lm9qKZ9YlMj3wV8D0zW2BmEyLfoh+L/H5vt/XBaWZ5wFh3X9jGvsFm9lKkbF4ys0GR7YeZ2ZuRc97YVs3LwqvrPWNmC81ssZldGNl+XKQcFprZW2aWF6kRvBopw3lt1XbMLGhmt7T6v/pGq91PABe3+R8sqc/d9aOfmP4AuyL/ngHcS3jWyADwNHByZF+vyL/ZwGKgd+S5Axe0Otdq4JrI428B90UeX0Z4IRyA+4FHItcYTXgue4DzCE+BHQD6AtXAeW3EOwO4u9XzQj68+//rwK2Rxz8Frmt13EPASZHHg4AlbZz7VOCxVs9bx/0U8JXI48uBJyKPnwa+FHl81d7y3Oe85wJ/aPW8AMgAVgLHRbblE54ZuQeQFdk2HKiIPC4DFkceXwn8OPI4E6gAhkSeDwDeSfT7Sj+x+dHU2RJPZ0R+5kee5xL+UJoFfMfMvhDZPjCyfRvQDDy2z3n+Gfl3LuH1DtryhIfXjnjPzPpEtp0EPBLZvtnMXtlPrH9v9bgU+LuZ9SP8QbuqndecBow2i86UnG9mee5e2+qYfkBVO68/sdXv81fg5lbbz4k8fgj4VRuvfQf4lZn9Enja3V81syOBTe7+NoC774RwrQK4y8zGES7fEW2c7wxgbKuaVAHh/5NVQCXQv53fQVKckoLEkwE3ufs9H9kYXszmNOBEd683sxmElw8FaHD35n3OE4r820z77+FQq8e2z78dUdfq8Z3Ar939yUisP23nNQHCv8Pu/Zx3Nx/+bgfS4YnJ3H2ZmR0LnAncZGbPE27maesc3wO2AEdFYm5o4xgjXCOb3sa+LMK/h3RB6lOQeJoOXG5muQBmNsDMSgh/C62OJIRRwKdidP3ZwLmRvoU+hDuKO6IA2BB5/JVW22uBvFbPnyc8GyYAkW/i+1oCDGvnOq8TnlYZwm32syOP3yTcPESr/R9hZv2Bend/kHBN4hjgfaC/mR0XOSYv0nFeQLgG0QJcSnit4H1NB75pZumR146I1DAgXLPY7yglSV1KChI37v484eaPN8zsHeBRwh+qzwFpZrYI+BnhD8FYeIzwgieLgXuAOcCODrzup8AjZvYqsLXV9qeAL+ztaAa+A5RHOmbfo43Vs9z9fcJLVObtuy/y+q9GyuFSYFpk+3eBa83sLcLNT23FfCTwlpktAH4E/Nzd9wAXAnea2ULgBcLf8u8GvmJmbxL+gK9r43z3Ae8B8yLDVO/hw1rZqcAzbbxGugBNnS3dipnluvsuC6+D+xYw3t03xzmG7wG17n5fB4/vAex2dzeziwh3Op8d0yD3H88swgvPVycqBokd9SlId/O0mfUk3GH8s3gnhIjfAecfxPHHEu4YNqCG8MikhDCzYsL9K0oIXZRqCiIiEqU+BRERiVJSEBGRKCUFERGJUlIQEZEoJQUREYlSUhARkaj/D4LlUs+i5KfrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you choose a linear decay, the curve will be plotted without using a log scale for the learning rates. Be careful than we you use a linear scale for the lrs, you get very fast to the high ones so you can't put as wide a range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3a6f798ad2a47b0aa81d16e4189e7c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 19%|█████████████                                                        | 148/782 [00:02<00:12, 51.84it/s, loss=18.1]\n",
      "                                                                                                                       \r"
     ]
    }
   ],
   "source": [
    "phases = [TrainingPhase(epochs=0.25, opt_fn=optim.SGD, lr=(0.001,0.1), lr_decay=DecayType.LINEAR,momentum=0.9)]\n",
    "learn.save('tmp')\n",
    "learn.fit_opt_sched(phases, stop_div=True)\n",
    "learn.load('tmp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH2ZJREFUeJzt3Xl03Gd97/H3d2Y0Gu2yJdnxviXOvmKyB3ITkgNlSUgIgVughdIcDrc3LUspvXAupLSlzeVcCs1tIYUL7bklhGyUJSQhgYSkWe1sjh3vjhNLtjZbGo00M5rle/+YkaM4si1bmv3zOkfH0sxv9Hw1lj7zzPP8fs9j7o6IiFS/QKkLEBGR4lDgi4jUCAW+iEiNUOCLiNQIBb6ISI1Q4IuI1AgFvohIjVDgi4jUCAW+iEiNCJW6gMk6Ozt9+fLlpS5DRKRirFu3bsDdu6ZzbFkF/vLly1m7dm2pyxARqRhmtmu6x2pIR0SkRijwRURqhAJfRKRGKPBFRGqEAl9EpEYo8EVEaoQCX0SkRijwRURK6MGNvXznke1FaUuBLyJSQg9s3MsP//OVorSlwBcRKaFoPE1rQ3EWPVDgi4iUUDSRojVSV5S2Chr4ZvYZM9tgZi+Z2W1mFilkeyIilWY4nqKtocID38wWATcCa9z9NCAIfKhQ7YmIVKJoIkVrpQd+XghoMLMQ0Aj0FLg9EZGKEo2naY1U+Bi+u3cD3wBeBfYAw+7+wMHHmdkNZrbWzNb29/cXqhwRkbKTzTojieoY0pkDXAWsABYCTWb2kYOPc/db3X2Nu6/p6prWGv4iIlUhNp4m61TFkM47gJ3u3u/uKeBu4MICticiUlGi8RRAVZyl8ypwvpk1mpkBlwMvF7A9EZGKEo2nASr/PHx3fwq4E3gWWJ9v69ZCtSciUmmGJ3r4RRrSKejLirt/BfhKIdsQEalU0UT1DOmIiMhhTIzhV/xZOiIicnjRxMQYvgJfRKSqDcdTmEFLfYVP2oqIyOFF4yma60MEAlaU9hT4IiIlUsyVMkGBLyJSMtEirpQJCnwRkZIp5uYnoMAXESkZDemIiNSIaLx4a+GDAl9EpGSKudsVKPBFREoinckyOp7RkI6ISLUbSRR3pUxQ4IuIlMRwkdfRAQW+iEhJFHulTFDgi4iUxOubnyjwRUSq2oEevsbwRUSqm8bwRURqRLE3MAcFvohISUQTKYIBozEcLFqbCnwRkRKYuMrWrDhr4YMCX0SkJKLxNK2R4k3YggJfRKQkooniLpwGCnwRkZKIxou7NDIo8EVESmJoLEVbowJfRKTq9ceSdDXXF7VNBb6ISJEl0xlGEmk6msJFbVeBLyJSZPtGxwHobFEPX0Skqg2M5AJfPXwRkSo3MJoEoENj+CIi1W1gJBf4mrQVEalyg/kx/I5mDemIiFS1gZEkDXVBmuq1tIKISFUbHB0veu8eFPgiIkU3EEvSWeTxe1Dgi4gU3UBsnE718EVEqp96+CIiNSCbdfZpDF9EpPoNx1Nksl5dPXwzO9HMnp/0ETWzPytUeyIilWAgVpqrbAEKdhKou28GzgIwsyDQDdxTqPZERCrBQCy/cFqR19GB4g3pXA5sd/ddRWpPRKQsTfTwi71SJhQv8D8E3FaktkREytbgxJBONfbwzSwMvA+44xD332Bma81sbX9/f6HLEREpqYHYOAGDOY1VGPjAu4Bn3b13qjvd/VZ3X+Pua7q6uopQjohI6QyOJpnbVE8gYEVvuxiB/2E0nCMiAkD/SGmusoUCB76ZNQJXAHcXsh0RkUoxOFqaq2yhwIHv7mPu3uHuw4VsR0SkUgyWaB0d0JW2IiJFNRBLluSiK1Dgi4gUzWgyzdh4pjqHdERE5HV7huMALGyPlKR9Bb6ISJF0DyUAWNjeUJL2FfgiIkXSMzTRw1fgi4hUtZ6hOAGD+SVYRwcU+CIiRdM9FOe41gihYGmiV4EvIlIkPUPxkg3ngAJfRKRoeoYSCnwRkWqXzTp7htXDFxGpegOxJKmMs6hE5+CDAl9EpCi6S3xKJijwRUSKoqfEF12BAl9EpChKfdEVKPBFRIqieyhOc32I1kioZDUo8EVEiiB3Dn4Es+JvbThBgS8iUgQ9JT4lExT4IiJF0TOUYJECX0SkusXHM+wbHVcPX0Sk2vXkNz5RD19EpMqVwymZoMAXESm41wO/dMsqgAJfRKTguocSuY1PWhX4IiJVrWcozvzWCHUl2vhkggJfRKTASr3xyQQFvohIgXUr8EVEql826+wZSpR8whYU+CIiBTUwmmQ8ky35OfigwBcRKagD6+C3KfBFRKpauVx0BQp8EZGCmgh8DemIiFS57qE4TeEgrQ2l2/hkggJfRKSAJs7BL+XGJxOmFfhm9qdm1mo53zezZ83sykIXJyJS6XqGEmUxfg/T7+F/wt2jwJVAF/Bx4O8KVpWISJUol6tsYfqBP/Fe5PeAH7j7C5NuExGRKSRSGQZHx1lUBhddwfQDf52ZPUAu8O83sxYgW7iyREQqXzmdkgkw3WnjPwLOAna4+5iZzSU3rCMiIodw4KKrMgn86fbwLwA2u/uQmX0E+DIwXLiyREQqXzmdgw/TD/x/BsbM7EzgC8Au4N+O9CAzazezO81sk5m9bGYXzKBWEZGKsmvfKMGAlXzjkwnTDfy0uztwFfAtd/8W0DKNx30LuM/dTwLOBF4+tjJFRCrPhp4ox3c1Ew6VxyVP061ixMz+Evgo8EszCwJ1h3uAmbUCbwO+D+Du4+4+NJNiRUQqyYaeKKcubC11GQdMN/CvB5LkzsffCywC/tcRHrMS6Ad+YGbPmdn3zKzp4IPM7AYzW2tma/v7+4+mdhGRstU3kqB/JMmpi9pKXcoB0wr8fMj/O9BmZu8BEu5+pDH8EHAO8M/ufjYwCnxxiu99q7uvcfc1XV1dR1e9iEiZ2tATBai8Hr6ZfRB4GrgO+CDwlJl94AgP2w3sdven8l/fSe4FQESk6m3MB/4pZRT40z0P/0vAW929D8DMuoAHyYX4lNx9r5m9ZmYnuvtm4HJg40wLFhGpBBt6hlk6t5HWyGGnO4tquoEfmAj7vEGm9+7gvwP/bmZhYAe6WEtEakS5TdjC9AP/PjO7H7gt//X1wL1HepC7Pw+sOcbaREQqUjSRYtfgGNe9ZXGpS3mDaQW+u/+5mV0LXERu0bRb3f2eglYmIlKhXj4wYVs+Z+jA9Hv4uPtdwF0FrEVEpCqU4xk6cITAN7MRwKe6C3B3L6+fRkSkDGzcE6WzuZ55ZbKkwoTDBr67T2f5BBERmWTT3ignLyi/+CyPBR5ERKpEJuts7Y1x0nEKfBGRqvbqvjGS6Syr5yvwRUSq2ua9uQnbE9XDFxGpbpv3xjCDE+Yp8EVEqtqW3hGWzW2kIRwsdSlvosAXEZlFm3tHynL8HhT4IiKzJpnOsHNgtCzH70GBLyIya7b3jZLJunr4IiLVbkvvCFCeZ+iAAl9EZNZs7h2hLmgs73jTbq5lQYEvIjJLtuwdYWVnM+FQeUZreVYlIlKBNu0dYXWZDueAAl9EZFb0DMXpHopz5uLyWgN/MgW+iMgseGL7IAAXruoscSWHpsAXEZkFj28fZE5jXVmukjlBgS8iMkPuzpM7BrlgVQeBgJW6nENS4IuIzNCr+8boHopzwcqOUpdyWAp8EZEZejw/fn9BGY/fgwJfRGTGntg+SFdLPau6yvOCqwkKfBGRGXB3Ht8+yIWrOjAr3/F7UOCLiMzIxj1RBmJJLlxV3uP3oMAXEZmRnz3fQyhgXHHKcaUu5YgU+CIixyibdf7j+R7evrqLuU3hUpdzRAp8EZFj9OTOQfZGE1x99qJSlzItCnwRkWP0H8/10BQO8o6T55e6lGlR4IuIHINEKsO96/fwztMWlOWG5VNR4IuIHIOHN/cxkkxz9dkLS13KtCnwRUSOwf0bepnTWFf2yylMpsAXETlKqUyWh17u5bKT5hMKVk6MVk6lIiJl4pmd+4gm0lxxSmVM1k5Q4IuIHKUHNvZSHwrwttXlvVjawRT4IiJHwd359cZeLjmhk8ZwqNTlHBUFvojIUdjQE6V7KM6VFbCUwsEU+CIiR+GBDXsxg8tOnlfqUo5aQd+PmNkrwAiQAdLuvqaQ7YmIFFIyneG2Z17jkhO66GyuL3U5R60YA1D/xd0HitCOiEhB/eKFPfSPJPnGdStKXcox0ZCOiMg0uDvff2wnJ8xr5m0nVNbZORMKHfgOPGBm68zshgK3JSJSME/u2MfGPVE+cfGKst/Z6lAKPaRzkbv3mNk84Ndmtsndfzf5gPwLwQ0AS5cuLXA5IiLH5nuP7mBuU5j3V8hSyFMpaA/f3Xvy//YB9wDnTnHMre6+xt3XdHV1FbIcEZFj8uSOQR7a1McnLlpOpK4yVsacSsEC38yazKxl4nPgSuClQrUnIlII2azz17/cyMK2CJ+8ZGWpy5mRQg7pzAfuyY91hYAfuft9BWxPRGTW3fNcNy91R/mH68+q6N49FDDw3X0HcGahvr+ISKHtHx3n5vs3ceaSdt53ZuWse38oOi1TRGQKmaxz44+fY/9oiq9ddSqBQGWemTNZZa38IyJSJN/89RYe3TrA1685nTMWt5e6nFmhHr6IyEF+9NSr3PLbbVy/ZgkfPrd6ThdXD19EZJJ/engbN9+3mctOmsdNV51a6nJmlQJfRCTv+4/t5Ob7NnPVWQv5xnVnUldB2xdOhwJfRATY0jvC3/9qE1ecMp9vfvCsqpikPVh1vXyJiByDVCbL537yAs2REF+/5vSqDHtQD19EhP/z222s7x7mOx85pyLXuZ8u9fBFpKat3z3MLb/ZxtVnLeSdpy0odTkFpcAXkZqVSGX43B3P09Ec5qb3nVbqcgpOQzoiUrO++eAWtvTG+OHH30pbY12pyyk49fBFpCatfWUft/5uBx8+dymXnlh5G5IfCwW+iNScsfE0n7vjBRa1N/Cld59c6nKKRkM6IlJz/u5Xm3h13xi3/fH5NNfXTgyqhy8iNeXHT7/Kvz2xi49fuILzV3aUupyiUuCLSM24c91u/vKe9bx9dRd/8a4TS11O0dXOexkRqVnj6SzffWQ733xwCxeu6uC7H30L9aHK3r3qWCjwRaRqpTJZHt3az833bWbT3hHec8YCbv7AGRW/VeGxUuCLSNWJj2f4x99s5cfPvMa+0XGOa43wLx9bwxWnzC91aSWlwBeRqvLE9kG+ePeL7Boc412nHcc15yzm7au7CIc0ZanAF5GqEE2k+Pq9m7jt6VdZ1tHIbX98Phesqq2zcI5EgS8iFe+J7YN89ifP0xtNcMPbVvKZd6ymIVyb4/SHo8AXkYqVzmS55bfb+PZDW1ne2cTdn76Is5ZUx4bjhaDAF5GKs7Enyu3PvMov1+9hIDbONWcv4mtXn0ZTDV01eyz07IhIxdi0N8q3H9rKvev3Uh8KcPnJ87j2nMVcdtI8zKpzl6rZpMAXkbKWTGf48dOvcee63azvHqYpHOTGy0/gjy5eQVtD9S9pPJsU+CJStp7YPsiXfrqeHf2jnLaolS+/+2SuOWcxc5vCpS6tIinwRaSsbOsb4UdPvcajW/vZ2hdjydwGfvjxt9bMmvWFpMAXkbLg7vy/J3fx1798GQfOWzGXD527lP967lKdYjlLFPgiUnKvDIzy1Z9v4OHN/bx9dRffuO5MulrqS11W1VHgi0jJJNMZvv3QVv7ldzupCxpfee8p/MEFywkEdMZNISjwRaQktvXFuPG259i4J8o15yzii+88iXmtkVKXVdUU+CJSNAOxJL9av4dHtvTz6NYBGsNBvvexNbyjxlexLBYFvogU3Evdw/zgP1/h5y/0MJ7JsmRuAx9cs4Q/uex45qtXXzQKfBGZddmss2NglOdfG+L2Z17lmVf20xgOcv1bl/DRC5ZxwrxmXRlbAlUR+OlMllBQa12LlIq7s6U3xmPbBnh82wBP79zHSDINwOI5DXz53Sdz3ZolujK2xKoi8M/724doCAdZ0dnE2Uvn8P6zF7Gis6nUZYlUlWzW2dYfY92u/WzeO8L2/hjD8RTusGc4wUAsCcDyjkbee9ZCzl7SzmmL2lg9v4WgzropCxUf+OlMlt8/fxmvDIyyYyDGLb/Zyrcf2spbls3hmnMW8Z7TF9LWqF6FyNEaHktx70t7eGzbAK/tG2PnwCgjiVyvvTEcZFVXM3ObwgQMjp/XzAWrOrjo+E4WtTeUuHI5FHP3wjZgFgTWAt3u/p7DHbtmzRpfu3btjNrbO5zgp893c9e63WztixEMGAvbIyyb28TSjkaWzW1kJJFm454oA7EkdcEAzfUhTlrQwmkL21je0cTiOQ20NdTpXGCpKalMlvXdwzy5Y5Antg/y5I5BUhlnUXsDq+Y1s3RuA2ctmcNbls1h2dxG/X2UCTNb5+5rpnVsEQL/s8AaoLUYgT/B3XmpO8qvN+5l5+AYrw6OsmvfGENjKYIB4/iuZha0R0hlsuwfTbG1b4RU5o3PRWM4SGdzPau6mlh9XAvnr+zg3OVztea2VDR3Z2gsxeh4mv6RJE/t3McT2wdZ+8o+RsczAJw4v4VLTujkqrMWcdqiVk2wlrGjCfyCJpeZLQbeDfwN8NlCtjVF25y+uI3TF7e94fbheIr6UIBI3RvX5hhPZ9nWF+O1/WPs3h9nOJ5iLJlmbzTB9v5RHts2wHcf2YEZdDXXs7C9gYXtERa2NTCnKUxjOEhTOERDOEhbQx3Hz2tmQVuEeCpDz1CCZDqDYTTVB5nfGnlT+yLT5e7Ekmni4xmS6SxzmsI0T9EJ2Tc6zr7RJMPxFMPxFENjKZ59dT8PvdzHnuHEG449fl4z15yzmPNXdnDeyrl0NmtZg2pU6K7qPwBfAFoK3M60HeosgXAowCkLWzllYeuU98fHM6zbtZ91u/bTPTRGz1CCTXtH+M2mPhKp7JSPqQ8FSKanvq+toY75rfUc19bA8V3NHD+vmUw2y/6xFI3hIEvnNrJqXjMrOppm/NY5mc6wfvcwPcMJwsEA9aEA4VCASF2A49oaOK41okm1MtAbTbB+9/CB4caRRJrxTO73J5nKMhBL0j+SZCCWfNPv1eI5DSyd20hbQx3prPPi7iF6o8k3tdFQF+Rtqzv55CUraakP0dpQxznL2pnXonPha0HBAt/M3gP0ufs6M7v0MMfdANwAsHTp0kKVM2MN4SAXn9DJxSd0vuF2d2c8k2UsmWEslWEsmWZwdJytfTFeGRhlblOYRe0N+R69M5JI0zeSpDeaoDeaoHsozm079xFPZaZst7k+xKqupgNvqVsiIVoiIfaPptg9NEYoEODE+S0saI+QSGUZG08zNp5hbDxNIpUlPp5he3/skC88AKGA0d5YR3N9iEDASKayBAIwvyXCvNZ6IqEg4fyLRDgYIOu5F5FgwJjTGKY1/yLq7riD47Q3hOlqrSeWSLOhJ8ru/WOk8uG1oK2BRe25eZJIOEhjXZDGcJB01hmIJYkl07Q11DG3KcycxjDtjXXExzMMxMYJGMxvjTC/NXJgBcVM1umNJtg/Ns5wPEUm6wTNCASMYP4jYIa7MzaeYSSRIppIE0ukSaazZN1JprNE4ymi+d5wNJH/N54mGMi9M5vTmPu/PK4tQnMkREukjvktuXd7c5rCtERC1IdypwenM040kWJsPENDXZC6YIBtfTFe6h4mlkwTDgVIZbIMxsbpHoqzvnuY/pFcQJtBa6TuDd+vLhigq6WelZ1NdLXU09EcpjEcIhwK0BfNdT72DCfY1hcj6875Kzs4fVEbXS31tDXUHfhYeOB3UWpRwcbwzezrwEeBNBABWoG73f0jh3rMbI7hV5Js1tkTzfW+2xvriCXS7No3xpbeEV7qHmbnwCiWD6xYMk00nqKtoY7FcxoZT2fZ0jtCbzRBQzhEU32QhnyANoSD1IeCLOto5LwVHazqaiKVcZLpDOPpLGOpDHuHE+zen5vbGEmkybpTHwqSzmbpjSboG0mSTGUZz2RJZbKMp7MEzIjUBUhlnOF46og/X13QWDynkfpQgKw7e4YSB87RnonWfOj2jSTeNP9yLFoiIdoa6miNvB6QLZEQDsQSaQZiSXqG4vSOJMlkZ+fvpr2xjvktEU5d2Mppi3JDkKcsaNU8kUxbWU3aAuR7+J8v5qStFEc6k2U0mQHL9UwD+Xci+0fH6RtJEKkLcsK8FsKh1y+Mc3eiiTQjiRSJVCb/jiT3jqGzuZ6m+iDReIr9Yyn2jY4zNDZOpC5IV3M9WYe9+XdHfdEEw/EUC9obWDKnkblNYVobQtQFA2SyTjbrZNxJZx3y7zyawrkXiYl3SvWhIIEA1AUC0x4680nvCPZGE/QMxQ+8YE4MwYQCRmtDHQ11QRKpDPFUhhWdTZyxuJ32hjpS2dwLZ50uGJQZKptJW6l+oWCAtsY3h1ZzfYglcxunfIyZHehBH0o5jymbGZG6IJG6IPNaI5yxuP2ov0d9QMMqUnxFCXx3fxh4uBhtiYjI1PR+UkSkRijwRURqhAJfRKRGKPBFRGqEAl9EpEYo8EVEaoQCX0SkRhTlStvpMrN+YFcJS+gEBkrY/tFSvYWlegunkmqF8q53mbt3TefAsgr8UjOztdO9RLkcqN7CUr2FU0m1QuXVeyga0hERqREKfBGRGqHAf6NbS13AUVK9haV6C6eSaoXKq3dKGsMXEakR6uGLiNSImgl8M3unmW02s21m9sUp7q83s9vz9z9lZsvzt3eY2W/NLGZmt5R5rVeY2TozW5//97Iyr/dcM3s+//GCmb2/nOuddP/S/O/D58u5XjNbbmbxSc/xd8q53vx9Z5jZE2a2If97XPCNEWbw/P7+pOf2eTPLmtlZha53RnL7kFb3BxAEtgMrgTDwAnDKQcd8GvhO/vMPAbfnP28CLgY+BdxS5rWeDSzMf34a0F3m9TYCofznC4C+ia/Lsd5J998F3EFuF7dyfn6XAy8VusZZrDcEvAicmf+6AwiWa70HHXM6sKOYz/WxfNRKD/9cYJu773D3ceDHwFUHHXMV8K/5z+8ELjczc/dRd38MSFRArc+5e0/+9g1AxMzqy7jeMXef2Nw2AhRjQumY6wUws6uBHeSe32KYUb0lMJN6rwRedPcXANx90N0zZVzvZB8GbitopbOgVgJ/EfDapK9352+b8ph8CA2T62EU22zVei3wnLsnC1Tnm2rJO6p6zew8M9sArAc+NekFoOzqNbMm4C+Amwpc45S15B3t78MKM3vOzB4xs0sKXSwzq3c14GZ2v5k9a2ZfKPN6J7ueCgj8WtnTdqrezsG9yekcUwwzrtXMTgX+nlyPqdBmVK+7PwWcamYnA/9qZr9y90K+m5pJvTcB33T3WBE70DOpdw+w1N0HzewtwE/N7FR3j852kdOoZTrHhMgNn74VGAMestwG3Q/NbonTqmXax5jZecCYu780m4UVQq308HcDSyZ9vRjoOdQxZhYC2oB9RanuEHXkHVWtZrYYuAf4mLtvL3i1s/TcuvvLwCi5uYdCmkm95wE3m9krwJ8B/8PM/qRc63X3pLsPArj7OnJj1avLtd787Y+4+4C7jwH3AueUcb0TPkQF9O6Bmpm0DZEbd13B6xMzpx50zH/jjRMzPzno/j+kOJO2x1wr0J4//tpKeG7zj5mYtF1G7g+ts1zrPeiYr1KcSduZPL9d5Cc9yU1KdgNzy7jeOcCz5CfzgQeBd5drvfmvA+ReEFYW+ndhVn7eUhdQtB8Ufg/YQq6X86X8bX8FvC//eYTcmRfbgKcn/wcCr5B7RY/l/3NPKcdagS+T6yU/P+ljXrk+t8BHyU1+Pp//Q7+63H8XJn2Pr1KEwJ/h83tt/vl9If/8vrec683f95F8zS8BN1dAvZcCTxajztn40JW2IiI1olbG8EVEap4CX0SkRijwRURqhAJfRKRGKPBFRGqEAl8qhpnFitDG+6ZaMbHAbV5qZhcWs02pTbWytILIAWYW9EMsyuXuPwN+VoA2Q37odYIuJXeNx+Oz3a7IZOrhS0Uysz83s2fM7EUzu2nS7T+13F4AG8zshkm3x8zsr8zsKeACM3vFzG7KL9K13sxOyh/3h5bf98DMfmhm3zazx81sh5l9IH97wMz+Kd/GL8zs3on7DqrxYTP7WzN7BPhTM3tvfj3158zsQTObn19b/VPAZ/Jrql9iZl1mdlf+53vGzC4q5HMptUM9fKk4ZnYlcAK5pW0N+JmZvc3dfwd8wt33mVkD8IyZ3eW59WSayK0N/z/z3wNgwN3PMbNPA58HPjlFcwvILeh1Erme/53ANeTWmj8dmAe8DPzfQ5Tb7u5vz7c5Bzjf3d3MPgl8wd0/Z7mNSWLu/o38cT8it0jbY2a2FLgfOPmYnzCRPAW+VKIr8x/P5b9uJvcC8DvgRnt956wl+dsHgQy5jUsmuzv/7zpyIT6Vn7p7FthoZvPzt10M3JG/fa+Z/fYwtd4+6fPFwO1mtoDcui07D/GYdwCnTFqRs9XMWtx95DDtiByRAl8qkQFfd/fvvuFGs0vJheUF7j5mZg+TWwcFIDHFuP3EXgEZDv23MHk/ATvo3+kYnfT5PwL/291/lq/1q4d4TIDczxA/inZEjkhj+FKJ7gc+YWbNAGa2yMzmkVu2dn8+7E8Czi9Q+48B1+bH8ueTm3SdjjZyK1YC/MGk20eAlklfPwAcWHa57PdJlYqhwJeK4+4PAD8CnjCz9eTG1VuA+4CQmb0IfA14skAl3EVu1dSXgO8CT5HbBelIvgrcYWaPAgOTbv858P6JSVvgRmBNfkJ6I7lJXZEZ02qZIsfAzJo9t/NVB7klcy9y972lrkvkcDSGL3JsfmFm7eQmX7+msJdKoB6+iEiN0Bi+iEiNUOCLiNQIBb6ISI1Q4IuI1AgFvohIjVDgi4jUiP8PPgAJlpPrCCUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can always force the scale to be linear (or log scale) by using the optional argument linear in the plot function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "029a5763006f4ce196413aa0ba22867e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 18%|████████████▋                                                        | 144/782 [00:02<00:11, 55.21it/s, loss=27.2]\n",
      "                                                                                                                       \r"
     ]
    }
   ],
   "source": [
    "phases = [TrainingPhase(epochs=0.25, opt_fn=optim.SGD, lr=(1e-5,10), lr_decay=DecayType.EXPONENTIAL,momentum=0.9)]\n",
    "learn.save('tmp')\n",
    "learn.fit_opt_sched(phases, stop_div=True)\n",
    "learn.load('tmp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEKCAYAAAAW8vJGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW9//HXJwsJZGENOyGALLIISEBZXKpWrXvdoJaqVYtabW17rdVrf97Wbvf2ttXa1t2KaCmoVEVt646KLBJkB9lCQlhCAiEbIfv398cM3gQTMoHMnJnM+/l45JHJnDNn3hzHvHO27zHnHCIiIkfEeB1ARETCi4pBREQaUTGIiEgjKgYREWlExSAiIo2oGEREpBEVg4iINKJiEBGRRlQMIiLSSJzXAVqrR48eLiMjw+sYIiIRZeXKlfudc2mBzBtxxZCRkUFWVpbXMUREIoqZ5QY6r3YliYhIIyoGERFpRMUgIiKNqBhERKQRFYOIiDSiYhARkUZUDCIi0oiKQUQkzDnn+NWbG9m0tzQk76diEBEJc4s2F/LUxztUDCIi4vP4h9vp2zmRS8f2Dcn7qRhERMLY6rxilu8o4qZpg4iPDc2vbBWDiEgYe/Kj7aQkxjFjUnrI3lPFICISpnL2H+Jf6/P51ukDSU4I3ZinKgYRkTD19OJs4mNiuHFqRkjfV8UgIhKG9pdX8VLWLq48tR89UxJD+t4qBhGRMDRnSQ7VdfV858zBIX/voBeDmcWa2Soze+MY81xtZs7MMoOdR0Qk3FVU1zJnWS5fPbkXQ9KSQ/7+odhiuAvY1NxEM0sBvg8sD0EWEZGw9+KKPIorarj1rCGevH9Qi8HM+gMXA08fY7ZfAL8FKoOZRUQkEtTW1fPUxzvIHNiVCQO7epIh2FsMDwP3APVNTTSz8cAA51yzu5lERKLJm+v2srv4sGdbCxDEYjCzS4AC59zKZqbHAA8B/xHAsmaZWZaZZRUWFrZxUhGR8OCc44kPsxmSlsS5I3p6liOYWwxTgcvMLAeYB5xjZi80mJ4CjAYW+ec5HVjY1AFo59yTzrlM51xmWlpaECOLiHjnk20H2Li3lFlnDiYmxjzLEbRicM7d55zr75zLAGYA7zvnZjaYXuKc6+Gcy/DPswy4zDmXFaxMIiLh7ImPtpOWksAV4/t5miPk1zGY2YNmdlmo31dEJJyt313Cx1v3c9PUQSTExXqaJSSDbzjnFgGL/I8faGaes0ORRUQkHD31cTbJCXFcd1roBstrjq58FhHxWF5RBW+s3cs3Jg2gc8d4r+OoGEREvPbM4h0YcNO0QV5HAVQMIiKeOniomvkr8rh8XD/6dO7odRxAxSAi4qkXluVyuKaOWR4MltccFYOIiEcqa+qYvSSHrwxPY3jvFK/jfEHFICLikZdX7uLAoWpPh79oiopBRMQDdfWOpz7OZuyALpw2qJvXcRpRMYiIeODtDfnkHqjgtjMHY+bd8BdNUTGIiISYc47HP9xORvdOnD+qt9dxvkTFICISYst3FLFmVwm3nDGYWA8Hy2uOikFEJMSe/Cib7kkduHpCf6+jNEnFICISQpvzy3j/8wJunJJBYry3g+U1R8UgIhJCT36UTcf4WGaePtDrKM1SMYiIhMjeksO8tno30ycOoGtSB6/jNEvFICISIs9+koMDbg6TwfKao2IQEQmBksM1zF2+k4vH9GFAt05exzkmFYOISAjMXb6T8qrasBosrzkqBhGRIKuqreOvn+zgjKE9GN2vs9dxWqRiEBEJstdW7aGwrIpbzwyvwfKao2IQEQmi+nrHEx9tZ2SfVKae1N3rOAFRMYiIBNF7nxewvfAQt54VfoPlNUfFICISRE9+tJ1+XTpy8Zg+XkcJmIpBRCRIVuYWsSLnIN85YxBxsZHz6zZykoqIRJgnPsymS6d4rp04wOsoraJiEBEJgu2F5byzaR/Xnz6QTh3ivI7TKioGEZEgePrjbDrExnD9lAyvo7SaikFEpI0VlFWyYOVursnsT4/kBK/jtJqKQUSkjc3+JIea+npumRb+w180RcUgItKGDlfX8bflO7lwVG8yeiR5Hee4qBhERNrQa6t3U3K4hm9PDe+htY9FxSAi0kacczy3NJcRvVOYmNHV6zjHTcUgItJGsnIPsmlvKTdMyYiY4S+aomIQEWkjzy3JITUxjsvH9fU6yglRMYiItIF9pZX8e30+12YOiLgL2o6mYhARaQNzl++kzjlmnj7Q6ygnLOjFYGaxZrbKzN5oYtptZrbOzFab2WIzGxnsPCIiba26tp65n+7k7GFpEXuKakOh2GK4C9jUzLS5zrkxzrlxwG+BP4Qgj4hIm/r3hnwKy6oicviLpgS1GMysP3Ax8HRT051zpQ1+TAJcMPOIiATDnCU5DOzeibOGpnkdpU0E+wjJw8A9QEpzM5jZHcCPgA7AOUHOIyLSptbvLiEr9yA/vfhkYmIi9xTVhoK2xWBmlwAFzrmVx5rPOfcX59wQ4CfAT5tZ1iwzyzKzrMLCwiCkFRE5Ps8vzaVjfCzXTIisey4cSzB3JU0FLjOzHGAecI6ZvXCM+ecBVzQ1wTn3pHMu0zmXmZbWPjbVRCTyFVdU8+rq3Vwxvh+dO8V7HafNBK0YnHP3Oef6O+cygBnA+865mQ3nMbOhDX68GNgarDwiIm3txaw8qmrruX5y5J+i2lDIr8IwsweBLOfcQuBOMzsPqAEOAjeEOo+IyPGoq3c8vyyXSYO6cXKfVK/jtKmQFINzbhGwyP/4gQbP3xWK9xcRaWuLNheQV3SYey882esobU5XPouIHIfnlubSOzWR80f18jpKm1MxiIi0UnZhOR9tKeSbp6UTH9v+fo22v3+RiEiQPb8sl/hYY8akdK+jBIWKQUSkFQ5V1fJy1i4uHtOHtJQEr+MEhYpBRKQVXlm1m7Kq2nYzLlJTVAwiIgFyzjFnaQ5j+nVm/IAuXscJGhWDiEiAlmUXsWVfOddPHhjRt+5siYpBRCRAc5bm0LVTPJeOjexbd7ZExSAiEoA9xYd5e+M+pk9MJzE+1us4QaViEBEJwNzlO3HO8c3T2ucpqg2pGEREWlBVW8ffP93JuSf3YkC3Tl7HCToVg4hIC/65bi8HDlVzw+QMr6OEhIpBRKQFzy3JZXBaElNP6u51lJBQMYiIHMOavGJW5xVzw+SMdn2KakMqBhGRY5izNJekDrFceWo/r6OEjIpBRKQZB8qreH3tHq6a0J+UxPZz686WqBhERJoxb0Ue1e3w1p0tUTGIiDShtq6evy3LZepJ3TmpZ4rXcUJKxSAi0oR3NxWwp6SS66PkFNWGVAwiIk2YszSHfl06cu6Inl5HCTkVg4jIUbbuK2PJ9gN88/R04trhrTtbEn3/YhGRFsxZmkuHuBimZw7wOoonVAwiIg2UVtaw4LNdXHpKX7ont89bd7ZExSAi0sA/Vu6iorqOG6ZE1ymqDakYRET86usdc5bmMm5AF07p335v3dkSFYOIiN8n2/eTvf9QVG8tgIpBROQLzy3JpXtSBy4a08frKJ5SMYiIAHlFFbz3+T6+MSmdhLj2fevOlqgYRESAF5bnEmPGdVFw686WqBhEJOodqqpl/oo8zh/Zi75dOnodx3MqBhGJakWHqpn5zHJKD9dw87RBXscJCwEVg5ndZWap5vOMmX1mZucHO5yISDDlFVVw9WNL2LCnlEe/OYHMjG5eRwoLgW4x3OScKwXOB9KAbwP/HbRUIiJBtmFPCVc+toT95VX87ZbTuHB0b68jhY24AOc7cqPTi4BnnXNrLFpufioi7c6SbfuZ9fxKUhPjmHv7FIb2iq77LbQk0C2GlWb2Nr5ieMvMUoD64MUSEQmOhWv2cMOzn9KvS0cWfFel0JRAtxhuBsYB2c65CjPrhm93kohIxHj642x++eYmJg3qxlPXZ9K5Y/Tcx7k1At1imAxsds4Vm9lM4KdASSAvNLNYM1tlZm80Me1HZrbRzNaa2XtmFt3XoYtIUNTXO3715kZ++eYmLhrTmzk3TVIpHEOgxfAYUGFmY4F7gFxgToCvvQvY1My0VUCmc+4U4GXgtwEuU0QkINW19fzwxdU89fEObpg8kD9941QS46P7yuaWBFoMtc45B1wO/NE590egxR1zZtYfuBh4uqnpzrkPnHMV/h+XAf0DzCMi0qKyyhpumr2C11bv4Z4Lh/Ozy0YRG6PzZloS6DGGMjO7D/gWcIaZxQKBbIc9jG8LI5CjOzcD/2pqgpnNAmYBpKfrcnURaVlBWSU3/nUFW/aV8ftrxnLVBP3dGahAtximA1X4rmfIB/oB/3usF5jZJUCBc25lSwv3H7fIbG6ZzrknnXOZzrnMtLS0ACOLSLTKLiznykeXkHPgEE/fkKlSaKWAisFfBn8DOvt/4Vc651o6xjAVuMzMcoB5wDlm9sLRM5nZecD9wGXOuarWhBcROdqqnQe56rElHK6u4+/fOZ2zh/f0OlLECXRIjGuBT4FrgGuB5WZ29bFe45y7zznX3zmXAcwA3nfOzTxqueOBJ/CVQsFx5BcR+cL7n+/juqeWk5IYz4LbpzB2QPTehe1EBHqM4X5g4pFf3maWBryL70yiVjGzB4Es59xCfLuOkoGX/BdS73TOXdbaZYqIvLgij/teWcfIPqn89caJpKUkeB0pYgVaDDFH/UV/gFaMzOqcWwQs8j9+oMHz5wW6DBGRpjjn+PP72/j9O1s4Y2gPHp85gaSEQH+1SVMCXXv/NrO3gL/7f54O/DM4kUREAlNX7/ivhet5YdlOrhzfj/++6hQ6xOluAicq0IPPPwaeBE4BxgJPOud+EsxgbW1zfhm/eGMjvssxRKQ9ePSDbbywbCe3njWY3187VqXQRgLe3nLOLQAWBDFLUM18ZjmFZVXcetZgeqYkeh1HRE5QVk4RD7+3lcvH9eXeC0egAZ/bzjHr1czKzKy0ia8yMysNVci2cEq/zgDE6MMjEvFKKmq4a95q+nXpyC+vGK1SaGPHLAbnXIpzLrWJrxTnXGqoQraFaUN7ABCrD5BIRHPOce8/1rKvtJJHvjGelEQNhtfWomaH3JHxUep0jEEkos39dCf/Wp/Pjy8YzjhdpxAUUVMMR3Yh1derGEQi1ZZ9ZTz4+kbOGNqD75wx2Os47VbUFENlTR2gLQaRSFVZU8edcz8jJTGO3187lhiNkho0UVMM2wrKAThQXu1xEhE5Hr98cyNb9pXz+2vH6czCIIuaYvjqyF4A1GpXkkjE+ff6vbywbCezzhzMWcM0wnKwRU0xHLmNX8nhGo+TiEhr7C4+zD0vr+WU/p25+/zhXseJClFXDKUqBpGIUVtXzw/mraKu3vHIjPG6sjlEomakKW0xiESeP72/jRU5B3l4+jgyeiR5HSdqRE39pqoYRCLKsuwD/On9rVx1an+uGN/P6zhRJWqKITE+lg5xMdqVJBIBDh6q5gfzVjOwexIPXj7K6zhRJ2p2JYFvd5K2GETCm3OOexas5cChKl65YarureCBqNliAEhNjKO0UsUgEs6eX5bLOxv38ZMLRzDaP/ilhFZUFUOv1ER2F1d6HUNEmrFpbym/fHMTXxmexs3TBnkdJ2pFVTEM7N6J3QcPex1DRJpQUV3L9/6+is4d4/nfa8ZqKG0PRVUx9EhOoOhQFXW6+lkk7PzijY1sLyzn4enj6JGc4HWcqBZ1xVDvoLhC4yWJhJM31+7l75/mcftZQ5h6Ug+v40S9qCqGLp181zIU68wkkbCRV1TBvf9Yy/j0Lvzwq8O8jiNEWTEcufq5uELFIBIOaurquWveKnDwyIzxxMdG1a+ksBVVJwh36dQBgPwSnZkkEg4efncLn+0s5k/fGM+Abp28jiN+UVXPg/xjrfz1kx0eJxGJbs45HnpnC3/5YDvTMwdw6di+XkeSBqJqi+HIrqSVuQc9TiISvSpr6rj7pTW8sXYvV53anwev0JAX4SaqigHgzGFplOisJBFPFJRW8p3nV7J2VzH3fm0Et545WNcrhKGoK4b+XTuyKvcgdfWOWN0zViRkNuwp4ZbnsiiuqOHxmRO4YFRvryNJM6LqGAPAqL6plFXVUlCmA9AiofLOxn1c8/hSAF66bbJKIcxFXTH07dIRgD3FGhpDJNicczz+4XZmPZ/F0J7JvHbHVA2MFwGiblfSgK6+YsgrOsyEgR6HEWnHqmvruf+Vdby0chcXn9KH318zlsT4WK9jSQCirhjSuyURG2NsKyj3OopIu1V0qJrbXljJpzuK+P65Q/nBuUOJ0TG9iBF1xdAhLoaM7p3Ysq/M6ygi7dK2gjJump1Ffmklf5wxjsvH6backSbqigFgWK8UPs9XMYi0tY+2FHLH3M9IiItl3qzTOTW9q9eR5DhE3cFn8BVDzoFDHK6u8zqKSLsxZ2kO3569gn5dOvLanVNVChEs6MVgZrFmtsrM3mhi2plm9pmZ1ZrZ1cHOcsTYAZ1xDlbt1BXQIieqtq6eB15bzwOvbeArw9N4+fYp9POf/SeRKRRbDHcBm5qZthO4EZgbghxfmDSoO3ExxuJt+0P5tiLtTsnhGr49ewVzluYy68zBPPGtTJITonIPdbsS1P+CZtYfuBj4FfCjo6c753L889UHM8fRkhPiGDegC0uzD4TybUXalQ+3FHL/K+vIL6nkf64aw/SJ6V5HkjYS7Gp/GLgHSAny+7TaiD4pLFy9x+sYIhGn6FA1v3xjI/9YtZshaUnMv3UyEwbqeEJ7ErRiMLNLgALn3EozO/sElzULmAWQnt42f5VkdE+itLKWvSWH6dNZ+0NFWuKcY+GaPTz4+kZKK2v4/rlDueMrQ0iI00Vr7U0wjzFMBS4zsxxgHnCOmb1wPAtyzj3pnMt0zmWmpaW1SbivjuxFjMHsT3LaZHki7dnu4sPcNHsFd81bzYBunXjje2fwo68OUym0U0HbYnDO3QfcB+DfYrjbOTczWO/XWgO7JzF5SHcdZxA5hrp6xwvLcvntvz/HAQ9cMpIbpmRoZOJ2LuSnD5jZg0CWc26hmU0EXgG6Apea2c+dcyG7a8fgHsm8uno3zjmNCS9ylC37yrh3wVo+21nMmcPS+NUVo3X7zSgRkmJwzi0CFvkfP9Dg+RVA/1BkaEpGjyTKKmspOlRN9+QEr2KIhJWq2joe/WA7jy7aRnJCHA9NH8sV4/rpj6coEtUnHA/q4fvrJ+fAIRWDCL7b3t67YC1bC8q5fFxfHrhkpP7fiEJRXQwZ3ZMA2F54iAkDu3mcRsQ75VW1/O6tzTy3NIc+qYk8e+NEvjKip9exxCNRXQwDuyfROzWRN9bu5drMAV7HEQm5okPV/OOzXfx18Q72llZyw+QM7r5guK5ejnJR/V8/Nsa4bFxfZn+Sw+HqOjp20Kl30v7V1zuWbD/AvBU7eXvDPqrr6hmf3oU/XXeqLlQTIMqLAWDKkO48+VE2n2zbz3kje3kdRyRo9pVW8lJWHvOz8sgrOkznjvFcd1o6MyYNYETvVK/jSRiJ+mKYelIPeiR34MWsPBWDtDu1dfUs2lzIvBU7ef/zAuodTB7cnbvPH84Fo3rrVpvSpKgvhvjYGK6a0J+nP95BYVkVaSk6A0MiX15RBfNX5PHSyjz2lVbRIzmBWWcOYfrEAQzqkeR1PAlzUV8MAOeP7MUTH2azbncx54zQVoNEHuccu4sPszy7iFdW7Wbxtv3EGJw1LI2fX5bOuSf3JD42Ku/LJcdBxQAMSUsGYHvBIc4Z4XEYkQDU1zu2FJSxIucgK3YUsSKniL0llQD069KRH543jGsy+9NXN8yR46BiALp06kCP5A5sKyj3OopIk6pq61i/u4RPdxxkRU4RWTlFlFbWAtAzJYGJg7oxKaMbmRldGdE7VWMZyQlRMfid1DOZz/NLvY4h8oWNe0r557q9fJpTxJq8YqpqffezGpyWxEVj+jAxoxsTM7oxoFtHDVchbUrF4DduQFeeWZxNZU2dztQQz9TVO97/vIBnFmezLLuI2BhjdN9UvnX6QDIzujExo6uGqJCgUzH4jU/vQk2dY01eMacN7u51HIky5VW1vJSVx+wlOeQeqKBfl47850UjmJ6ZTudO8V7HkyijYvCbMqQ7KQlxPLc0R8UgIZNXVMFzS3KYvyKPsqpaJgzsyj0XjOCCUb2I01lE4hEVg19KYjw3TRvEH9/byh/e3sytZw0hSePFSBA451iZe5BnFu/grQ35xJhx0Zg+3DRtEOMGdPE6noiKoaE7zzmJ7P2HeOT9bSz4bDcLbp9C786JXseSdqKu3vHG2j08s3gHa3eV0LljPLeeNYTrJw/UfcclrJhzzusMrZKZmemysrKC+h7Lsg9w0+wVDO2ZzPxbJ+tgtJywg4eq+f68VXy8dT+D05K4aeogrjy1H5066G8zCQ0zW+mcywxkXu3EbMLpg7vz0PRxrNlVwr0L1hJp5SnhZcOeEi7982KWZxfx66+P4d0fnsXM0weqFCRsqRiaccGo3tx9/jBeXb2Hrz+6hCXb9nsdSSLQq6t2c9VjS6itc8y/9XSuOy2dGF18JmFOxXAMd3zlJH5w3lD2lhzm/lfXU1+vLQcJTE1dPT9/fQM/mL+aU/p34fXvTWN8uu51IJFBxXAMZsYPzhvG/RePZMf+Q7z/eYHXkSQCFJZVMfPp5Tz7SQ7fnprB3245TaP2SkTRTs4AfG10b/p16cjv3t7MmcPS6BCnPpWmrc4r5rbnV1J8uJqHpo/l6+P7ex1JpNX0Gy4A8bEx/PyyUXyeX8Y9L69h4x6NqSRfNn/FTq59fClxscaC26eoFCRiqRgCdN7IXsw6czAL1+zhokc+5tf/3ERNXb3XsSQMVNXW8Z+vrOMnC9Zx2uBuvH7nNEb17ex1LJHjpl1JrfCfF53M7WcN4Xdvb+bJj7Kpqqnj55eP9jqWhFhNXT17ig+zs6iC3AMVvLxyF6vzirn97CHcff5wDXktEU/F0Epdkzrwq6+PoWN8LE8v3kFmRjcuHdvX61jSxiqqa8kuPMTOooovCiCvqILcokPsKa6krsEZaqmJcTz6zVO5aEwfDxOLtB0Vw3H6yddGsCqvmB+/vIb+XTvqVMQIVVtXT86BCjbnl7E5v5TP88vYvK+MnUUVNLyusVtSBwZ068T4AV25fGwn0rt3YmA33/deKYm6NkHaFQ2JcQL2l1dx5aNLKK+q5bU7pjKgWyevI8lRnHMUV9RQWF5FYVkVBWWV5JdUsbWgjM35ZWwtKKfafwOcGIOMHkmM6J3C8F6pDO2VTLr/l39qooa+lsjWmiExVAwnKLuwnMv+/Akn90lh3qzJ2r/sgdLKGjbnl/F5fhlb8svYW1LpK4JS3/eaui9/xnulJjC8d6q/BFIY3juFk3oma1wsabdaUwzalXSCBqcl84srRvHD+Wv4wzub+dFXdfAxWJxzbC88xIY9JV8Uweb8MnYXH/5inpSEOPp17UhaSgInpfUgLSWBnikJjb+nJpKsIdVFmqX/O9rAFeP68dGW/fzlg+28tnoPw3qlUFhWxTdPS2fGpHSv47ULJYdr+Omr63l9zR4A4mKMIWnJTBjYletOS+fkPikM751K386Juv+xyAlSMbQBM+N314zlwtG9mbM0h/wS31kr972yjpTEeC4+RWernIhl2Qf4jxfXsK+0krvOHcqFo3szJC1ZV6CLBImKoY3ExhgXjOrNBaN6A1BZU8e3nlnOD+evJikhlrOH9/Q4YeSprq3noXe38PiH28nonsSC26cwVnc4Ewk6/ckVJInxsTx1fSaD05L49uwV/OZfmygorWTdrhJW5h6kVldNH9P2wnKufOwTHlu0nRkTB/DG96apFERCRGclBdnh6jp+8eZG5i7f2ej5Lp3iOWd4T245YzAj+6Z6lC78OOf42/Kd/PLNjXSMj+U3V57ChaN7ex1LJOKF1VlJZhYLZAG7nXOXHDUtAZgDTAAOANOdcznBzhRKHTvE8uuvj+Gi0X3YVlBGny4dqat3vLepgHc25vPGur3816UjuW5SelQdNK2sqWNfaSX5JZXkl1b6H1exfk8Jn+4o4oyhPfjdNWPplap7bouEWiiOMdwFbAKa+rP4ZuCgc+4kM5sB/A8wPQSZQm7a0B5MG9rji58vGtOHokMn84P5q7n/lfWs2FHEb648hY4d2u959CUVNTzzyQ7mLt/J/vKqL03vGB9Ln86JPHDJSG6ckqGriUU8EtRiMLP+wMXAr4AfNTHL5cDP/I9fBv5sZuYibf/WceqW1IHZN07kzx9s46F3t7CzqIKnb5hIt6QOXkdrUyUVNTyzOJtnP8mhrKqW807uybgBXeiVmkiv1ER6d/Z9T02Mi6qtJpFwFewthoeBe4CUZqb3A/IAnHO1ZlYCdAei5gbLMTHG988dytCeydw1fzVXP7aE526a9KXhNWrr6vk8v4yMHkkRc3FWcUU1zyzewWx/IXxtdG++f+5QTu6jYyoi4Sxov2HM7BKgwDm30szObm62Jp770taCmc0CZgGkp7fPC8a+NqYPaSkJ3PxcFhf98WNOHdiVMf0606tzIsuyD/DRlkLKKmvp0imem6cO4oapGWE7fo9zjsc+3M6jH2ynvKqWi8b05nvnqBBEIkXQzkoys98A3wJqgUR8xxj+4Zyb2WCet4CfOeeWmlkckA+kHWtXUqSdldRa2wvLeeLD7azbXcrWfWXU1jt6piRw9vA0JmZ0460N+by7qYCUxDhunjaI284aEnbj+8xZmsMDr23gvJN7cfcFwxjRW4Ug4rWwG0TPv8VwdxNnJd0BjHHO3eY/+Hylc+7aYy2rvRdDQ5U1dRSWVdG/a8dG+97X7y7hkfe28vbGfQzs3olfXTGm0YFtLy3LPsDMp5dz1rA0nro+UweQRcJEa4oh5Be4mdmDZnaZ/8dngO5mtg3fwel7Q50nnCXGxzKgW6cvHZAd3a8zT16fydxbTiPGjJn+K6ybOtMnlHYXH+aOv31GevdOPDRjnEpBJELpArcIV1lTx18+2MbjH27HzDjv5J5cMa4fZw/vGdKxhA5X13HNE0vI3V/Bq3dOZUhacsjeW0RaFlYXuElwJcbH8h/nD+fycf14YVkur6/Zwz/X5dOlUzynD+pObIxR7xzOQVyEpLm0AAAKQElEQVSscdWE/pw9LK1NTwt1znHfP9ayYU8pz9yQqVIQiXDaYmhnaurqWbx1P6+s2s2GPSWYGTEGhlFUUU1hWRUTM7ry4wtGMGlQty9eV11bz7LsA7y1IZ81u4qJNSM+Nsb3FRdDh9gYTunfmQtH92Zoz+RGxfL0x9n88s1N3H3+MO48Z6gX/2wRaUHYHXxuSyqG41ddW8+LWXk88t5WCsqqOGtYGpeO7cvirYW893kBZZW1dOoQy4SBXYmNMWrq6qmpdVTX1VNRXcvWgnKcg0E9kvwjyfairLKWG5/9lAtH9+Yv152qC9REwpSKQY7pcHUdc5bm8NiH2ymuqKFrp3jOO7kXF4zqzbShPZo9/bWgtJK3N+7jrQ35LN1+gNp632dnRO8UFtw+haQIufBOJBqpGCQgpZU15Ow/xMg+qcTFtu5AdUlFDe99vo8VOUV89+yTvnSltoiEFxWDiIg0EtbXMYiISHhTMYiISCMqBhERaUTFICIijagYRESkERWDiIg0omIQEZFGVAwiItJIxF3gZmaFQO5xvrwHkXc/6UjLrLzBFWl5IfIyt9e8A51zaYEsMOKK4USYWVagV/6Fi0jLrLzBFWl5IfIyK692JYmIyFFUDCIi0ki0FcOTXgc4DpGWWXmDK9LyQuRljvq8UXWMQUREWhZtWwwiItKCiC4GM7vQzDab2TYzu7eJ6QlmNt8/fbmZZTSYdp//+c1mdkGgy/Qir5l91cxWmtk6//dzGrxmkX+Zq/1fPcMgb4aZHW6Q6fEGr5ng/3dsM7NHrA3vBXoCeb/ZIOtqM6s3s3H+aUFbvwFmPtPMPjOzWjO7+qhpN5jZVv/XDQ2e93IdN5nXzMaZ2VIz22Bma81seoNps81sR4N1PM7rvP5pdQ0yLWzw/CD/52er//PUweu8ZvaVoz7DlWZ2hX9a69evcy4iv4BYYDswGOgArAFGHjXPd4HH/Y9nAPP9j0f6508ABvmXExvIMj3KOx7o6388Gtjd4DWLgMwwW78ZwPpmlvspMBkw4F/A17zOe9Q8Y4DsYK/fVmTOAE4B5gBXN3i+G5Dt/97V/7hrGKzj5vIOA4b6H/cF9gJd/D/PbjhvOKxf/7TyZpb7IjDD//hx4PZwyHvUZ6MI6HS86zeStxgmAducc9nOuWpgHnD5UfNcDjznf/wycK7/r6fLgXnOuSrn3A5gm395gSwz5Hmdc6ucc3v8z28AEs0soY1ytXne5hZoZn2AVOfcUuf7xM4BrgizvN8A/t5GmVrSYmbnXI5zbi1Qf9RrLwDecc4VOecOAu8AF3q9jpvL65zb4pzb6n+8BygAArrYyou8zfF/Xs7B9/kB3+fJ8/V7lKuBfznnKo43SCQXQz8gr8HPu/zPNTmPc64WKAG6H+O1gSzTi7wNXQWscs5VNXjuWf8m4v9rw90GJ5p3kJmtMrMPzeyMBvPvamGZXuU9YjpfLoZgrN9Gefxasz6O9Rn2ch23yMwm4fuLeHuDp3/l38X0UBv+0XOieRPNLMvMlh3ZLYPv81Ls//wczzKPpa1+/8zgy5/hVq3fSC6Gpv4HPfoUq+bmae3zbeFE8vommo0C/ge4tcH0bzrnxgBn+L++dYI5A8rSwjx7gXTn3HjgR8BcM0sNcJnHqy3W72lAhXNufYPpwVq/LeY5ztd6vY6PvQDfFs3zwLedc0f+6r0PGAFMxLcb5CcnErLh2zXxXGvypjvfFcXXAQ+b2ZA2WOaxtNX6HQO81eDpVq/fSC6GXcCABj/3B/Y0N4+ZxQGd8e17a+61gSzTi7yYWX/gFeB659wXf2k553b7v5cBc/Ftjnqa17+L7oA/10p8fxkO88/fv4Vlhjxvg+lf+ksriOs30Mytfa3X67hZ/j8O3gR+6pxbduR559xe51MFPEtoP8PNOrL71jmXje9Y03h8YxJ18X9+Wr3MFrTF759rgVecczVHnjie9RvJxbACGOo/Q6ADvv+pFx41z0LgyNkaVwPv+/e7LgRmmO8slUHAUHwH7AJZZsjzmlkXfP9D3eec++TIzGYWZ2Y9/I/jgUuA9bSNE8mbZmax/lyD8a3fbOfcXqDMzE7375K5HnjN67z+nDHANfj26+J/LpjrN9DMzXkLON/MuppZV+B84K0wWMdN8s//CjDHOffSUdP6+L8bvv31ofwMN5e365FdLv7PwFRgo//z8gG+zw/4Pk+er98GvnSM7LjW74keSffyC7gI2ILvL9L7/c89CFzmf5wIvITv4PKnwOAGr73f/7rNNDhro6llep0X+ClwCFjd4KsnkASsBNbiOyj9RyA2DPJe5c+zBvgMuLTBMjP9H8ztwJ/xX2QZBp+Hs4FlRy0vqOs3wMwT8f0leQg4AGxo8Nqb/P+Wbfh2zYTDOm4yLzATqDnqMzzOP+19YJ0/8wtAchjkneLPtMb//eYGyxzs//xs83+eErzO65+WAewGYo5aZqvXr658FhGRRiJ5V5KIiASBikFERBpRMYiISCMqBhERaUTFICIijagYpF0xs/IQvMdlTY18GeT3PNvMpoTyPSV6xbU8i0j0MbNY51xdU9OccwtpuwsfG75nnPu/MXiOdjZQDixp6/cVOZq2GKTdMrMfm9kK/+BhP2/w/Kvmu6/FBjOb1eD5cjN70MyWA5PNLMfMfm6+8e/XmdkI/3w3mtmf/Y9nm++eB0vMLNv8Y+SbWYyZPep/jzfM7J921Hj//vkWmdmvzexD4C4zu9R8Y/2vMrN3zayX+e4bcRvwQ/MN5neG/+ryBf5/3wozmxrMdSnRRVsM0i6Z2fn4huKYhG9wsoVmdqZz7iPgJudckZl1BFaY2QLnG9spCd99JB7wLwNgv3PuVDP7LnA3cEsTb9cHmIZvoLKF+IZkvhLflahj8F2lvgn4azNxuzjnzvK/Z1fgdOecM7NbgHucc/9hvpsdlTvnfuefby7wkHNusZml4xsi4+TjXmEiDagYpL063/+1yv9zMr6i+Aj4vpl93f/8AP/zB4A6YMFRy/mH//tKfL/sm/Kq840UutHMevmfmwa85H8+38w+OEbW+Q0e9wfm+8e36QDsaOY15wEj7f9GAU81sxTnG+xP5ISoGKS9MuA3zrknGj1pdja+X6qTnXMVZrYI3xhKAJVNHFc4ct+LOpr//6XhvTHsqO+BONTg8Z+APzjnFvqz/qyZ18Tg+zccbsX7iARExxikvXoLuMnMkgHMrJ/57tfcGTjoL4URwOlBev/FwFX+Yw298B08DkRnfAOhwf+NBAtQBqQ0+Plt4M4jP1gb3idZRMUg7ZJz7m18909Yambr8O33TwH+DcSZ2VrgF8Cy5pdyQhbgGwVzPfAEsBzfHeNa8jPgJTP7GN/Y/0e8Dnz9yMFn4PtApv/A+kZ8B6dF2oRGVxUJEjNLds6Vm1l3fMM0T3XO5XudS6QlOsYgEjxvmO8mSx2AX6gUJFJoi0FERBrRMQYREWlExSAiIo2oGEREpBEVg4iINKJiEBGRRlQMIiLSyP8HvI29b83W8KkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot(linear=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# But there's more!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing that's possible with these blocks and didn't try yet is that you can change the optimizer at each phase. For instance, we can do a 1cycle with SGD and cyclical momentum but then anneal with Adam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.SGD, lr=(1e-3,1e-2), lr_decay=DecayType.LINEAR, \n",
    "                          momentum=(0.95,0.85), momentum_decay=DecayType.LINEAR),\n",
    "           TrainingPhase(epochs=1, opt_fn=optim.SGD, lr=(1e-2,1e-3), lr_decay=DecayType.LINEAR, \n",
    "                          momentum=(0.85,0.95), momentum_decay=DecayType.LINEAR),\n",
    "           TrainingPhase(epochs=1, opt_fn=optim.Adam, lr=1e-3, lr_decay=DecayType.COSINE, momentum=0.9)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "de34aa1e42f6486f9fdd57954e944013",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.59037    4.5196     0.14      \n",
      "    1      3.607808   4.601357   0.133                                                                                 \n",
      "    2      3.543177   4.520047   0.1417                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.52005]), 0.1417]"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuAAAAEKCAYAAABT6eBwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4VGX2wPHvSSchtCTUUEIJJBBABAQVREGaNLGXVdRd1vbT1bVRbAiiq6KubWXtXVcB6QgIiAhKJ5AQCCUk1NACIaS/vz9mwBBSBpiZO5k5n+d5n8zcueXMEN55c+97zxFjDEoppZRSSin38LM6AKWUUkoppXyJDsCVUkoppZRyIx2AK6WUUkop5UY6AFdKKaWUUsqNdACulFJKKaWUG+kAXCmllFJKKTfSAbhSSimllFJupANwpZRSSiml3EgH4EoppZRSSrlRgNUBuENkZKRp1qyZ1WEopdQ5W7169UFjTJTVcbiT9tlKqarK0T7bJwbgzZo1Y9WqVVaHoZRS50xE0qyOwd20z1ZKVVWO9tk6BUUppZRSSik30gG4UkoppZRSbqQDcKWUUkoppdxIB+BKKaWUUkq5kQ7AlVJKKaWUciMdgCullKqQiPQXkRQRSRWRp8p4vamILBSRDSKyWESiS7zWRER+EpFkEUkSkWbujF0ppTyRDsCVUkqVS0T8gXeAAUA8cIuIxJda7VXgM2NMe2AcMLHEa58Brxhj4oCuwAHXR62UUp5NB+BKKaUq0hVINcZsN8bkA98AQ0utEw8stD9edOp1+0A9wBgzH8AYk22MyXFP2Eop5bl0AK6UUqoijYD0Es8z7MtKWg9cZ398LRAuIhFALHBURKaIyFoRecV+Rl0ppXyaDsCV8kULFtiaUpWTMpaZUs8fA64QkbXAFcBuoBBbteUe9te7AM2BEWUeRGSkiKwSkVWZmZlOCl0pL6F9ttfxiVL0SqlSxo+3/ezTx9o4VFWQATQu8Twa2FNyBWPMHmA4gIhUB64zxmSJSAaw1hiz3f7aNKAb8GHpgxhjJgOTATp37lx6gK+Ub9M+2+voGXCllFIVWQm0EpEYEQkCbgaml1xBRCJF5NT3ySjgoxLb1haRKPvzq4AkN8SslFIeTQfgSimlymWMKQQeBOYBycB3xphNIjJORIbYV+sFpIjIFqAeMMG+bRG26ScLRSQR23SW/7r5LSillMfRKShKKaUqZIyZDcwuteyZEo+/B74vZ9v5QHuXBqiUUlWMngFXSimllFLKjfQMuFK+6P33rY5AKaWUo7TP9jo6AFfKF7VubXUESimlHKV9ttfRKSjKM/TqZWvKPWbMsDWllDof2me7l/bZXkfPgCvli157zfZz8GBr41BKKVU57bO9jp4BV0oppZRSyo10AK6UUkoppZQb6QBcKaWUUkopN3LpAFxE+otIioikishTZbweLCLf2l//XUSa2ZdHiMgiEckWkbdLbXOxiCTat/m3iIgr34Nyk++/tzWllFKeT/tspS6Iy27CFBF/4B3gaiADWCki040xSSVWuwc4YoxpKSI3Ay8DNwG5wNNAO3sr6T1gJLACW2W2/sAcV70P5SaRkVZH4Fs+/9zqCJRSVZn22e6lfbbXceUZ8K5AqjFmuzEmH/gGGFpqnaHAp/bH3wO9RUSMMSeMMb9iG4ifJiINgBrGmOXGGAN8Bgxz4XtQbpI+6V0KPvzI6jB8R+PGtqaUFygqNny3Kp2jOflWh+I7PvnE1pRb5DdoRF6DhlaHoZzIlQPwRkB6iecZ9mVlrmOMKQSygIhK9plRyT4BEJGRIrJKRFZlZmaeY+jKnZZuzWT3G++R9tq7VofiO7791taU8gI7Dp7gqR828ObCrVaH4jt0AO42xhg++L+JTH5gotWhKCdy5QC8rLnZ5jzWOa/1jTGTjTGdjTGdo6KiKtilstLJ/CLGTN2IiHAwO49ftx60OiTf8N57tqaUF2hZtzo3d23C58vT2JaZbXU4SjnVz5sP0GnWN1yx8AerQ1FO5MoBeAZQ8hp3NLCnvHVEJACoCRyuZJ/RlexTVSFv/byVXYdzaF0vnJBAf8ZMSyS3oMjqsJRSVcwjfWIJCfRn4uxkq0NRymkKioqZoL/TXsmVA/CVQCsRiRGRIOBmYHqpdaYDd9ofXw/8bJ/bXSZjzF7guIh0s2c/uQP40fmhK3fYvO8Yk3/Zzg0XR1MrNJCYyDDSDuXw1s96GVkpdW6iwoN54MqWLEg+wLJUvZKmvMOXK9LYnnmCAH+/CqcHqKrHZQNw+5zuB4F5QDLwnTFmk4iME5Eh9tU+BCJEJBV4FDidqlBEdgKTgBEikiEi8faX7gM+AFKBbWgGlCqpuNgwekoiNaoFMnpgHAA1qwUy/KJGvL9kO1v2H7c4QqVUVXPXZc2Irl2NF2YmUVSswxVVtWXlFPDGwq1c3jKS8BCXJa1TFnHpv6gxZja2VIEllz1T4nEucEM52zYrZ/kqzk5NqKqYr/7YxZpdR5l0YwdqhwXBbNuvyZhifxalHGDUlET+9/fu+PlpmnellGNCAv15akAbHvxqLd+vTuemLk2sDsl7zZ5d+Trqgry5cCvHThYw5po45DWo+BY5VdVoJUzldgeO5fLy3M1c1jKCay+yJ7EJDYXQUCKqBzN6YByr047w9cpd1gbqzbSIhvJS1yQ04OKmtXll3hay8wqtDsd72fts5RrbM7P5bPlOburSmLgGNXj///7FC3c+b3VYyol0AK7c7vkZSeQVFjNhWAKnC5m++66tAddfHE235nV4ac5mDhzLrWBP6rxFRmohDeWVRISnB8VzMDuP9xanWh2O9yrRZyvnmzhnM8EBfjxydSwAOTVqcax6LYujUs6kA3DlVj9v3s+sxL08dFVLmkWG/fnCd9/ZGrYv0AnXJpBXUMzzM5PK2ZO6IJrDV3mxjo1rMaxjQ/67dAcZR3KsDsc7leizlXP9tu0g85P2c/+VLakbHgJAt19mcPXvesubN9EBuHKbnPxCnp62idh61RnZs0WF67aIqs4DV7Zk1oa9LNp8wE0R+hAdgCsv93j/Ngjwr7kpVoeilMOKig3jZybTqFY17rk85vTy7ktn0HfVXAsjU86mA3DlNq/P38Luoyd58doEggIq/9W7t1dzWkSFMXbaRnLydS6nUspxjWpVY2TP5kxfv4c1u45YHY5SDvlhdQZJe4/x5IA2hAT6Wx2OciEdgCu32Lg7i4+W7eSWrk3o3KyOQ9sEB/gzcXh7dh89yRsLNDe4Uurc3HtFC6LCgxk/M4kKSkwo5RFO5BXyyk8pXNSkFoPbN7A6HOViOgBXLldUbBg9NZHaoUE81b/NOW3bNaYON3dpzIe/7mDTniwXRaiU8kZhwQE83rc1a3YdZcaGvVaHo1SF/rNkG5nH83h6UPyfCQpO05S83kYH4MrlPlu+kw0ZWTwzOJ6aoYFlr7R4sa2VYdSAOGqHBjJqSqIW11DKAiLSX0RSRCRVRJ4q4/WmIrJQRDaIyGIRiS71eg0R2S0ib7svapvrLo4mvkENXp6zmdyCIncf3ntV0Gerc7fn6Ekm/7KdIR0a0qlJbavDUW6gA3DlUnuzTvLqvBSuiI0670tqNUMDeXpQPBsysvhs+U6nxuezZs/WQhrKISLiD7wDDADigVtKVCY+5VXgM2NMe2AcMLHU6y8AS1wda1n8/YSxg+LYffQkH/66w4oQlKrUv+ZuBuCJ/q3LfP3dx99k1D0vuTMk5WI6AFcu9eyPmygyhvHD2pVxSa2EV1+1tXIM6dCQnrFRvDovhT1HT7ogUh+jRTSU47oCqcaY7caYfOAbYGipdeKBhfbHi0q+LiIXA/WAn9wQa5kubRHJ1fH1eHdRKpnH86wKw7tU0mcrx61LP8q0dXv4a48YomuX3S8XhFQjLyjEzZEpV9IBuHKZeZv28VPSfv7RJ5bGdSoZ7M2caWvlEBHGD21HkTE8O32TkyP1QVpEQzmuEZBe4nmGfVlJ64Hr7I+vBcJFJEJE/IDXgMddHmUlRg+MI6+wmEnzNS2hU1TSZyvHGGN4YWYSkdWDua9Xy3LX6zH/OwYvm+bGyJSr6QBcuUR2XiHP/riJNvXDz8hleiGaRITycO9Y5iftZ+7GfU7Zp8/SIhrKcWVduip9M8ZjwBUisha4AtgNFAL3A7ONMelUQkRGisgqEVmVmZl5oTGfJSYyjDu6N+Pblekk7z3m9P0rdT5mJe5lddoRHusbS/XggHLX6/T7AnqtX+y+wJTL6QBcucSr81LYfzyXicMTCPR33q/ZX3vE0KZ+OM9N38Tx3AKn7VcpVa4MoHGJ59HAnpIrGGP2GGOGG2MuAsbYl2UB3YEHRWQntnnid4hImRNZjTGTjTGdjTGdo6KiXPA24OHerahRLZDxszQtobJebkERL83ZTJv64dzQuXGF62oOFO+jA3DldOvTj/Lp8p38pVtTLnLy3dyB/n5MHJ7A/uO5vPbTFqfuWylVppVAKxGJEZEg4GZgeskVRCTSPt0EYBTwEYAx5jZjTBNjTDNsZ8k/M8aclUXFXWqGBvJw71YsSz3Ez1phV1ns42U7yThykqcHxePvp0NsX6MDcOVUhUXFjJqSSN3wYB7vV/bd3GWqVs3WHHBRk9r8pVtTPl2+k3XpR88vUKWUQ4wxhcCDwDwgGfjOGLNJRMaJyBD7ar2AFBHZgu2GywmWBOuA27s1pXlUGBNmJ1NQVGx1OFXXOfTZ6mwHs/N4Z1EqfeLqclnLSKvDURbQAbhyqo+X7SRp7zGeH9KW8JBycn6XZc4cW3PQ4/1aUzc8mFFTEvVLVCkXM8bMNsbEGmNaGGMm2Jc9Y4yZbn/8vTGmlX2dvxpjzko1Yoz5xBjzoLtjLy3Q348xA+PYnnmCL1ekWR1O1XWOfbY606T5W8gtKGLUwDirQ1EW0QG4cpr0wzlMmr+FPnF16de2vkuPFR4SyPND2pK89xgfaW7fc6dFNJQPu6pNXS5rGcEbC7eSlaP3kij32rzvGN/8sYvbuzWlRVR1h7b599P/5ZH73nBxZMqddACunMIYwzM/bkQEnh9aSc7vsrzwgq2dg35t69Mnri6vL9hC+uGcczueUspniQhjr4kn62QB//55q9XhVE3n0Wcr23flhFnJhIcE8o8+rRzeTmeIex8dgCunmJ24j0Upmfyzb2sa1TqPeYELF9raORARnh/aDj8Rxk7bqFkNzoUW0VA+Lq5BDW7q3JjPlu9kx8ETVodT9ZxHn61gcUomS7ce5KHeragVGuTwdlfN/IwbF3/jwsiUu+kAXF2wrJMFPDdjE+0a1eDO7k3deuxGtarxz76tWbIlk5kb9rr12FWaFtFQikf7xhLk78fE2clWh6J8QEFRMeNnJRETGcZfup3bd2XbtUvpnrzCRZEpK+gAXF2wV+Zt5lB2Hi8Nb0+AE3N+O2rEpc1IaFST52ck6XxOpZTD6oaHcP+VLfkpaT+/bTtodTjKy339xy62ZZ5g9MA4ggJ0+OXr9DdAXZDVaUf48vdd3HVZDO0a1bQkBn8/YeLwBA6fyOOluZstiUEpVTXdc3kMjWpVY/zMZIqKdRqbco2snAJen7+F7s0j6BNX97z2ob+d3kUH4Oq8FRQVM3pKIg1qhPDo1bEXtrOICFs7T+0a1eTuy2L4+o9drNx5+MJiUUr5jJBAf54c0Iakvcf4YU2G1eFUHRfYZ/uatxdt5ejJAsYOijv3JAXoTZjeyKUDcBHpLyIpIpIqImdVPxORYBH51v767yLSrMRro+zLU0SkX4nlj4jIJhHZKCJfi0iIK9+DKt9/l24nZf9xxg1tR1hwwIXt7IcfbO0CPHJ1LI1qVWP0lETyCzU3eIW0iIZSpw1u34CLmtTilXkpnMgrtDqcqsEJfbav2HnwBJ/8tpMbLo6mbcPzu1JcEBRMfmCwkyNTVnLZAFxE/IF3gAFAPHCLiMSXWu0e4IgxpiXwOvCyfdt4bOWO2wL9gXdFxF9EGgEPAZ2NMe0Af/t6ys3SDp3gzQVbGdCuPn3i61kdDgBhwQGMG9qWrQeyeX/JNqvD8WxaREOp00SEpwfFk3k8T/sO5XQvzdlMoL8fj/U9h+rQpbz/1Ns8fvdLToxKWc2VZ8C7AqnGmO3GmHzgG2BoqXWGAp/aH38P9BbbtZmhwDfGmDxjzA4g1b4/gACgmogEAKHAHhe+B1UGYwxjp20k0N+PZwe3dc5OR42ytQvUO64eAxPq89aiVE0tppRyWKcmtRncoSGTl25nz9GTVofj+ZzUZ3u7FdsPMXfTPu7v1YK6Nc7/gv35TFtRns2VA/BGQHqJ5xn2ZWWuY4wpBLKAiPK2NcbsBl4FdgF7gSxjzE8uiV6Va/r6PSzdepAn+remfk0nzQBavtzWnODZwW0J9vdjzNREzQ1eHi2iodRZnuzfGmPglXkpVofi+ZzYZ3ur4mLD+FlJNKwZwl97NL+gfV39w2TuWPCZkyJTnsCVA/Cy/lwrPRoqb50yl4tIbWxnx2OAhkCYiNxe5sFFRorIKhFZlZmZeQ5hq4oczcln3IwkOjauxW2XuDfnt6Pq1QjhiQFt+G3bIaas2W11OJ5Ji2godZbo2qH8tUcMU9fuZl36UavDUVXclLW72bj7GE8OaENIoP8F7St24x9cvG2tkyJTnsCVA/AMoHGJ59GcPV3k9Dr2KSU1gcMVbNsH2GGMyTTGFABTgEvLOrgxZrIxprMxpnNUVJQT3o4CmDh7M0dPFjBxeAL+fp57Sey2rk3o1KQW42clcfhEvtXhKKWqiPt6tSSyejDjZybpFTR13nLyC3ll3mY6NK7F4PYNrQ5HeSBXDsBXAq1EJEZEgrDdLDm91DrTgTvtj68Hfja2Hm86cLM9S0oM0Ar4A9vUk24iEmqfK94b0BJmbvL79kN8uyqdv/aIIa5BDavDqZCfnzBxeHuO5xYyYZb+iiilHFM9OIDH+sayKu0IsxP3WR2OqqLeX7Kd/cfyeGZQHH7OOlmlfw96FZcNwO1zuh8E5mEbJH9njNkkIuNEZIh9tQ+BCBFJBR4FnrJvuwn4DkgC5gIPGGOKjDG/Y7tZcw2QaI9/sqveg/pTXmERo6cmEl27Gv/ofYE5v8sSHW1rTtS6fjgjezbnhzUZ/JaqVe6UUo65oXNj2tQPZ+KcZHILiqwOxzO5oM/2FnuzTvL+L9sY1L4BFzet45ydeu4FZ3WeLjB5c8WMMbOB2aWWPVPicS5wQznbTgAmlLH8WeBZ50aqKvOfxdvZlnmCT+7qQrWgC5vLVqYvvnD+PoGHerdi5oa9jJm2kTkP97jgeXheQwtoKFUufz9bWsLbPvidT37byb1XtLA6JM/joj7bG7wyN4ViA0/2b+O0feZUr0mWFDhtf8p6WglTVWp7ZjbvLEplcIeG9Gp9fiV0rRIS6M+Ea9ux4+AJ3l2UanU4nkOLaChVoctaRtInri5v/5zKwew8q8NRVcSGjKNMWbubey6PoXGdUKft95N/vsaY259z2v6U9XQAripkjGHM1I2EBPrx9KA41x3oH/+wNRfo0SqKYR0b8t6SbWzdf9wlx1BKeZ9RA+PILShi0vwtVofieVzYZ1dVxhhemJlEZPUg7u/l3KsmonNQvI4OwFWFvl+dwfLth3hqQBx1w52U87ss69bZmouMHRRPWHAAo6cmUlysd7JoEQ2lKtciqjq3d2vKN3/sImWf/vF+Bhf32VXR3I37WLnzCI9e3ZrwkECn7nvgV//m3rkfOHWfylo6AFflOnwinxdnJ9O5aW1u7tK48g08WGT1YEYPiGPlziN8uyq98g28nRbRUMohD/duRXhIIONnaVpCVb68wiImztlM63rh3NjZ+TenNt26nra7kpy+X2UdHYCrco2flUR2XiEThyc4L42ShW7oHM0lMXWYODuZA8dzrQ5HKVUF1A4L4qHerVi69SCLt2hRN1W2T5btZNfhHMYOiiPA3/lDq6r/DaxK0wG4KtOy1INMWbObv/dsQat64VaH4xQiwovDE8gtKOaFmZobXCnlmL90a0pMZBgTZiVTUFRsdTjKwxzKzuPtn1O5qk1derRyZeE/vQLjTXQArs6SW1DEmKmJNIsI5cGrWrrnoLGxtuZiLaKqc/+VLZixfg+LUw64/HhKqaovKMCPUQPakHogm6//2GV1OJ7BTX12VfD6gi3kFBQxeqDz0g4q76cDcHWWdxalsvNQDhOuTXBf3uzJk23NDe7r1YLmUWGMnbaRk/k+WmRDi2godU6ujq9H9+YRvD5/C1knNR+zO/tsT7Zl/3G++n0Xt1/ShJZ1XXe1OKtOPTJruvLsunI3HYCrM2zdf5z/LNnG8IsacVnLSKvDcYngAH9evDaBjCMneWOhj6YX++ILLaShHCYi/UUkRURSReSpMl5vKiILRWSDiCwWkWj78o4islxENtlfu8n90TuHiDB2UBxHTxbw9s9brQ5HeYgJs5IJCw7g4T6uvRrw9UMv8tyNmrnKm+gAXJ1WXGwYPTWR6sEBjLnGhTm/yzJypK25SbfmEdzYOZoPlu4gac8xtx1XqapGRPyBd4ABQDxwi4jEl1rtVeAzY0x7YBww0b48B7jDGNMW6A+8ISK13BO587VtWJMbLo7mk992knbohNXhWMvNfbYnWpxygCVbMnm4dyvqhAW59Fiid2F6HR2Aq9O+XZXOyp1HGD0wjojqwe49+JYttuZGowfGUataIKOmJlLka7nBtYiGclxXINUYs90Ykw98AwwttU48sND+eNGp140xW4wxW+2P9wAHgCp9Hf2xvq0J9Pdj4uzNVodiLQv6bE9SWFTMhFnJNIsI5Y7uzVx+vMEf/4t/zHzX5cdR7qMDcAXAgeO5TJydTLfmdbj+Yt+YG1wrNIinB8WzPv0oX6xIszoc99IiGspxjYCSyfMz7MtKWg9cZ398LRAuIhElVxCRrkAQsK2sg4jISBFZJSKrMjM9N91f3Roh3HdFC+Zu2seK7YesDkdZ5OuV6Ww9kM1TA+IICnD9UKrhzhRa7U11+XGU++gAXAEwfmYyuQXFTLg2AfGha11DOzakR6tIXpmXwr4szQ2uVBnK6hBKXzJ6DLhCRNYCVwC7gcLTOxBpAHwO3GWMKTOPnzFmsjGmszGmc1SUZ58k/1vP5jSsGcL4WUlaWdcHHcst4PX5W7gkpg792tZz01F953vZVzg8ABeRMFcGoqyzOOUA09fv4YErW9IiqrrV4biViDB+WDsKiop5dvpGq8NRyhNlACVL4UYDe0quYIzZY4wZboy5CBhjX5YFICI1gFnAWGPMCveE7Fohgf480b8NG3cfY8ra3VaHo9zsnZ9TOZKTz9OD4t16wkr/1PMulQ7AReRSEUkCku3PO4iITkTyEifzi3j6x420iArj3l7NrQukY0dbs0DTiDAe7tOKeZv289OmfZbEoJQHWwm0EpEYEQkCbgaml1xBRCJF5NT3ySjgI/vyIGAqths0/+fGmF1uSIeGdGhci1fmbSYnv7DyDbyNhX22lXYdyuHjZTu5rlM07RrVtDocVYU5cgb8daAfcAjAGLMe6OnKoJT7vLlwK+mHT/LitQkEB7gp53dZ3njD1izytx7NaV0vnGenbyI7zwe+TLWIhnKQMaYQeBCYh+1EzHfGmE0iMk5EhthX6wWkiMgWoB4wwb78RmzfFyNEZJ29ecWozc9PeGZQHPuP5fH+ku1Wh+N+FvfZVnlpbjL+fsLj/Vq79bgHGzYlPcI37s/yFQGOrGSMSS91mcVHq5d4l+S9x/jv0u3c1LkxlzSPqHwDLxbo78eLwxO4/j+/8dpPKTw7uK3VIbmWFtBQ58AYMxuYXWrZMyUefw98X8Z2XwBem3D+4qZ1uKZ9A97/ZRs3d21Mg5rVrA5JudAfOw4zO3Efj/SJpV6NELcee+q9z/Dz5gMMc+tRlSs5cgY8XUQuBYyIBInIY9ino6iqq6jYMGpKoi0NnyeUz739dluz0MVNa3PbJU349LedbMg4amksSqmq4an+bSg28Mq8FKtDcS8P6LPdqbjYMH5WEvVrhDCyp/una/pQbgSf4cgA/F7gAWxppzKAjsD9rgxKud5Xv6exLv0oTw+Kp1aoawsIOCQjw9Ys9kT/NkRWD+apHxIpLCozWYN30CIaPklEOovIVBFZY69MmSgiG6yOqyprXCeUuy+LYcqa3b71h7uH9NnuMm3dbjZkZPFE/9ZUC3L/dM1r3xvHqGmvu/24ynUcGYC3NsbcZoypZ4ypa4y5HXBzmUTlTPuP5fKvuSn0aBXJ0I4NrQ7Ho9QICeS5IW1J2nuMj5fttDoc1/HxIho+7EvgY2w5uwcDg+w/1QV44MoWRIQFMX5mMsZorgpvczK/iH/NTaF9dE2GdSydAt89Ivem0eSg7/zB4wscGYC/5eAyVUU8N30T+UXFjB/WzqdyfjtqQLv69G5Tl0nzt5B+OMfqcJRypkxjzHRjzA5jTNqpZnVQVV14SCCP9o3lj52HmbtRMyl5m8m/bGffsVyeHhSPn59+ZyrnKHcALiLdReSfQJSIPFqiPQdYmC5DXYgFSfuZs3EfD/VuRdMITe1eFhFh3LB2iMAzP27UM1rKmzwrIh+IyC0iMvxUszoob3BT58a0rhfOxDmbySvUPAXeYl9WLv9Zso2BCfXp0qyO1eEoL1LRGfAgoDq2TCnhJdox4HrXh6ac7UReIc9O30Rsver8rYeFOb/L0r27rXmIRrWq8ejVsSxKyWR2op7RUl7jLmz38fTHNvXk1DQUdYEC/P0YOyiOXYdz+PS3nVaH43oe1me7yqs/pVBUbHiqv868Vc5VbhpCY8wSYImIfKKXKL3DpPlb2H30JD/c152gAIeLoLrHxIlWR3CWEZc2Y9q63Tw3YxOXt4qkZrVAq0NyHh8soKEA6GCMSbA6CG/Vo1UUV7aO4q2FqVzXKZqI6sFWh+Q6HthnO9vG3Vn8sCaDkT2a0yQi1NJY9jZrTdrhHC62NArlTI6MwnJE5BURmS0iP59qjuxcRPqLSIqIpIrIU2W8Hiwi39pf/11EmpV4bZR9eYqI9CuxvJaIfC8im0UkWUS8/09wJ9i4O4uPl+3gtkuacHFTvYzmiAB/PyZe255D2Xn8a+5mq8PS0V8TAAAgAElEQVRxLh8toqFYISLxVgfhzcZcE0dOQRFvLNhqdSjqAhhjeGFmEnVCg3jgqpZWh8Ose55k0sD7rA5DOZEjA/Avgc1ADPA8sBNbaeIKiYg/8A4wAIgHbimj478HOGKMaYmt4ubL9m3jsZU7bovtUum79v0BvAnMNca0ATqgOckrVVhUzKgpiURUD+aJ/h6Q87ss111nax4mIbomIy6N4cvfd7E67bDV4Sh1oS4H1tlPbGgaQhdoWTec2y5pwld/7GLr/uNWh+M6HtpnO8u8Tfv5fcdhHrk6lhohXnT1U3kMRwbgEcaYD4ECY8wSY8zdQDcHtusKpBpjthtj8oFvgKGl1hkKfGp//D3QW2xpOYYC3xhj8owxO4BUoKuI1MBW1vhDAGNMvjHGhxKvnp9Pl6eRuDuLZwfHe+40ikOHbM0D/bNvLA1rhjBqSiL5hV6SG9zHimio0/oDrYC+aBpCl/lHn1hCg/yZMNuLzw95cJ99ofIKi5g4J5nYetW5uUtjq8MB4MY3RjHufy9ZHYZyIkcG4AX2n3tF5BoRuQiIdmC7RkB6iecZ9mVlrmOMKQSygIgKtm0OZAIfi8ha+938msqjAnuOnuS1n1K4snUU1yQ0sDqcKiksOIBxQ9uxZX82/1263epwnMPHimio00w5TTlRnbAgHu7disUpmSzZkml1OOocfb48jbRDOYy5Jp4Af8+4X6rGof3UPXbQ6jCUEznymzVeRGoC/wQeAz4AHnFgu7KSZZbu6Mtbp7zlAUAn4D1jzEXACeCsueUAIjJSRFaJyKrMTN/sAI0xPPPjJoyBcUM15/eF6BNfjwHt6vPvhVvZefCE1eEodb5mATPtPxcC24E5lkbkpf7SvSlNI0IZPzPJu6vqepnDJ/J5c+FWroiN4orYKKvDOU2/vb1PhQNw+7zrVsaYLGPMRmPMlcaYi40x0x3YdwZQ8tpNNLCnvHVEJACoCRyuYNsMIMMY87t9+ffYBuRnMcZMNsZ0NsZ0jorynP9E7jRv0z4WJO/nkatb0biOtXdwe4PnhrQl0N+PsdM0N7iqmowxCcaY9vafrbBNFfzV6ri8UXCAP6MGxLH1QDbfrEyvfAPlEd5csIWc/CLGXqNpB5VrVTgAN8YUAUPOc98rgVYiEiMiQdhuqiw9cJ8O3Gl/fD3ws7GNbKYDN9uzpMRgm7P4hzFmH5AuIq3t2/QGks4zPq92PLeAZ6dvIq5BDe6+LMbqcCrXu7etebB6NUJ4on9rfk09yLR1u60OR6kLZoxZA3SxOg5v1a9tPS6JqcPr87dwLLeg8g2qkirQZ5+r1APH+eL3XdzStTGt6oVbHY7ycuXmAS/hNxF5G/gW25QP4HTHXS5jTKGIPAjMw1Y58yNjzCYRGQessp9F/xD4XERSsZ35vtm+7SYR+Q7b4LoQeMD+xwDA/wFf2gf127EVllClvDovhQPH83j/L509Zg5bhZ5+2uoIHHLbJU2ZsmY3L8xMpldsXWqHBVkd0vnxgQIa6mwi8miJp37YriD65hw9NxARnh4Uz+C3f+WdRamMGuBFZ1WrSJ99LibMSiY0yJ9H+sRaHcpZ0lt3IDUzm85WB6KcxpEB+KX2n+NKLDPAVZVtaIyZDcwuteyZEo9zgRvK2XYCMKGM5etAfwcrsnbXET5bkcad3ZvRsXEtq8PxKv5+wsThCQx+61denJ3MKzd0sDqk8+MDRTRUmUqe1ivENhf8B4ti8QntGtXkuk7RfPzrTm7r2tTygi6qbL9syWRRSiajB7bxyAJK8//yD2Zu2MPdVgeinKbSAbgx5kp3BKKco8Ce87teeAj/7Ot5f8WXa8AA2885nn8/WFyDGvy1R3P+s2QbwztF071FhNUhKeWoJGPM/0ouEJEbgP+Vs75ygsf7tWbWhr28NDeZd2/zklqGVajPrkxhUTETZiXTpE4od17azOpwlI+oAnMT1Ln46NcdbN53nOeGtCW8KhUPOHnS1qqIh3u3onGdaoyZmkhuQVHlG3gaLy+ioco1ysFlyonq1Qjh3itaMDtxHyt3eklBryrWZ1fku1UZpOw/zqgBbQgO8K98Awvc/PIjvPrNuMpXVFWGDsC9SPrhHF5fsIWr4+vRv119q8PxatWC/JkwLIHtB0/w3uJtVodz7ry4iIY6m4gMEJG3gEYi8u8S7RNsU1GUi/2tZwz1a4Twwswkios1i5KnOJ5bwKT5KXRtVsejvzdDjx+l5kkvrqzqg3QA7iWMMTz940b8RXh+SFurw/EJPWOjGNqxIe8t3kbqgWyrw1GqInuAVUAusLpEmw70szAunxEaFMAT/VuzISOLH9drFiVP8e7ibRzMzmfsoDitlaHcqtIBuIgML6P1FpG67ghQOWbmhr0sTsnkn31b07BWNavD8RlPD4qnWpA/o6cm6lktJ8vJyeG5555j586dVodS5Rlj1htjPgVaGmM+LdGmGGOOWB2frxjWsRHto2vyr7kpnMyvglPXvEz64Rw+/HUHwzs1on205ycs8PTyE9pnnxtHsqDcA3QHFtmf9wJWALEiMs4Y87mLYvN5J06cYPHixWzdupXjx48TEhJC3bp1ufzyy2nRogUAR44cYe6CRTzx8XzC/Ao4FJnMJ6mRdOzYkYSEBPz9bfPZnnvuudP7DQwMpHr16kRHR9OlSxeaNGlixds706BBVkfg8Oe9dOlStm3bRnZ2NqGhoVxaBD+uCOLbDvW5pZst57rHf94W2bt3L5MnTyY6Opp77rnH6nB8UVcReQ5oiq3/F8AYY5pbGpWXqKwP8fMTHrq8Ibe+8Am3PryQTg2rERoaSkREhPbZ5+l8++2IiAjm7AlGTDiP97OVFvH4z9wC2me7jiMD8GIgzhizH0BE6gHvAZcAvwA6AHeR7777joKCAoYMGUKdOnU4ceIEaWlpnLTf+LJnzx4+/fRTlqbnYaI78sFDfWkZWY3MzEzWrFlDnTp1zug0hgwZQmxsLIWFhRw5coT169fz8ccf06dPHy677DKr3qbNY49Ze3wc/7yjoqIYOHAgkZGRFBQUsH//AVZP+h/jvl3G1e0aEmlPYeXRn7dFBTTWrFlDly5dWL9+PZmZmfhqlVoLfQg8gm36iZ6CdTJH+pCVc74lvrZhvTTnxb8MoHaIv/bZF+B8++0l61NZ+L8p3H/rEBrU/POqsad+5js6dCN57zG3V83SPtt1HBmANzs1+LY7AMQaYw6LiJeV9vIcubm5pKWlcccdd9C8ue3kVK1atWjUqBFgm/M9depUjpsQNoUncG+P5lzeznb2tX79+iQkJJxVLj0kJITq1auf3ldMTAzh4eEsXLiQuLg46tSp48Z36Fkc/bwjIiK45557zpgrWL9+fb56JYb+b/zCCzOTePPmiwAP/7wtKKJRUFBAYmIid911FwUFBaxdu5a+ffuefn337t3MnDmTzMxMIiMjueqqM0sNFBcXM2PGDHbs2EF2djY1atTg4osv5tJLLz397zFt2jRycnJo0qQJK1asoKCggC5dutC7d2+WLFnCypUrERG6devG5Zdf7tb37yGyjDHnnDdORPoDb2IrqvaBMealUq83BT4CorAVVbvdGJNhf+1OYKx91fH2qTBe51z6kE//9hBXv/4LH648yKs3dNA++zydb79dXGz4KnUbLS4byJPXnzmo9tTP/Jeb7mXKmgwecOMxtc92LUduwlwqIjNF5E57R/oj8IuIhAFHXRue7woKCiIoKIiUlBQKC89OUrBv3z727T/A4qO1iK4dyiNXn53z25EbSrp3744xhs2bNzsl7vPWq5etWcSRzzszM/OMjqOklnXDuf/KVvy4bg9LtpRfWNBjPm8LJCUlUbNmTerVq0eHDh1Yv349RUW2k7D5+fl89dVX1K5dm5EjR9KnTx9++umnM7Y3xlCjRg1uuOEGHnzwQXr37s3SpUtZu3btGeulpaVx9OhRRowYwaBBg1i2bBlffvklhYWF3H333fTq1YsFCxawZ88et713D7JIRF4Rke4i0ulUq2gDEfEH3gEGAPHALSISX2q1V4HPjDHtsRVtm2jftg7wLLYrpl2BZ0WktnPfkmc4lz6kaWQYd13WjB/WZLBxd9bpdbTPPjfn22/P2LCH9elHebxfG8KCK0/X6zGfuZtpn+1ajpwBfwC4DrgM23zBz4AfjO1PdS3S4yJ+fn4MGzaMGTNmsHr1aurXr0+TJk2Ij48nOjqaQ4cOsTrtCLvCW/LpXW0JCw4gNzeXSZMmnd5Hjx496NGjR4XHCQ0NJSwsjCNHfPs+LEc+b4CIiD+L7pT+vLt2v5TmkWGMnZZIz6LiMo/jMZ+3BUU01qxZQ4cOtsqhTZs2JTAwkJSUFOLj40lMTKSoqIhhw4YRFBRE3bp16dmzJ1OmTDm9vb+/P1de+WeXU6tWLfbu3cvGjRvp1OnPMWRwcDADBw7Ez8+PyMhIli9fzvHjx7n99tsB27/hr7/+ys6dO2nYsKGb3r3HuMT+s2Q14coqG3cFUo0x2wFE5BtgKJBUYp14bFNbwHa/0DT7437AfGPMYfu284H+wNcX8B480rn2IQ9c1ZJvV2znlgee5OYujRER7bPP0fn020eOneD/nnyG0CB/UsLXsTSnZ5X4zG97/l76HMuF59yXtEj7bNdypBKmAb63N+VG8fHxxMbGkpaWRkZGBqmpqfz222/07t2bHIL5fcchBt1Yl95x9QDbL/G9994LwJdffnn6L9XKlL7s6asq+rxr1z77pF3pzztAYMK1Cdzy3xX8nnGY28o5jkd83m4uoHH48GHS09O5/vrrAduZvoSEBNasWUN8fDyZmZnUq1ePoKCg09tER0eftZ9Vq1axZs0ajh49SmFhIUVFRdSqdWb2gqioKPz8/ry4FxYWRkhIyBnrhIWFceLECWe+xSrhPCsbNwLSSzzP4M+B/CnrsZ2oeRO4FggXkYhytm10HjFUCefSh9QICeTRAW0Zk3Wc2F4d2LdmgfbZ5+Fc++3P/9iNaXM1/76zMynL5lSZzzwwP5fA/Dzu+2K1W46Xm53Fxvm/076wBZ9vtx0zY38wM976gdjLTrJrw2/kHD1J2neJZ2yTuGEPW6cmUiPK9sfPge1JHNy5mbyc45jiIkxxMUGh1VnuZ9vnjlXbKMg9wdav/jwrvmXtAQKCgthQ4r0mJR5kwd61NN4e7I63f9pL17WnZjXXFDWsdAAuIsOBl4G62M6An7prvoZLIlJnCAgIoEWLFrRo0YIrrriC6dOns2jRItaFXIS/CHdd/Odf9iJyen7aqTvpK5OTk0NOTk6ZHZUvKuvzXrx4MXfddRcABw8epEGDBkDZn3f3FhHccHE0Hy47wvbMbEpfp/fVz3vNmjUUFxfz+uuvn1526gstKyurvM3OsHHjRubOnUvfvn1p3LgxwcHB/PHHH2ddFi79uy8iZS6z+gvVCvab6F8EGhpjBtinknQ3xnxY0WZlLCv94T0GvC0iI7DdnL8bW4EfR7Y9FdtIYCRQpTNOnEsfcmvXJny+PI23f9vHdSGOleXw1T6kIo5+5geO5fKfX7ZzTedW9O3UktQVVed7skZIIFknC9mW6Z6aE4e2rOVodi6Lf/ikxFLbf93ixvvIOnqSvON5BJaIp+DECY6cyGfX4RNUI5vsvds5sPFXImK7ENK0Ln7+gWTtSibrwK7T7+PAsZMU5RfgX2I/B47n4R8EpsSyzOw8sgNPku+m939KkQvTCzsyBeVfwGBjTLLLolAOi4qKImlPFqurFzKgS2s2r19Fz0s6nXHG71z89ttviAht2rRxcqTeISoqiuLiYiIjI4mKimLZsmW0bdu2ws979MA4vnjXj9cXbGFgjy74+f05BvHFz7u4uJh169bRp08fYmPPvFdhypQprFu3jqioKNatW0d+fv7ps+AZGRlnrLtr1y4aNWpE165dTy/Ty/Dn7BPgY2CM/fkW4Fts2VHKkwE0LvE8Glthn9OMMXuA4QAiUh24zhiTJSIZ2FLXltx2cVkHMcZMBiYDdO7c2Wv+OqqoDwnw92PMNXGM+Hglv+cfon1C5fvzxT7kXJX3mb/6UwoFRcU8NeDcPjtP+MzrhAVRJyyInx65wuXHKi4uZtKklXS77a9l9tlxcdWp3q838+fP59EHup/uszds2MCUwmaMGNGNZs2aMXv2CfbHXX76DyGAr7/OZv/+IP7xD9v7mDbtCDk5Odx665/v66uvdhMaGsqwYX8u++CDrTRp0oS+fV3//t3FkVHbfh18u19OTg6ffvopGzZsYP/+/Rw5coRNmzYxf9ESlmf606l5XcY/fBdHjhzhww8/ZPPmzRw6dOh0Oqtjx46ddUNPbm4u2dnZZGVlsWPHDqZNm8ayZcvo06eP9XfT33ijrVmkvM972bJlxMTEEBISwrBhwxz6vGuHBdEzNoqk9EN8uCjJMz9vN9qyZQs5OTl06tSJunXrntHatWvH2rVradeuHX5+fvz4448cOHCAbdu2sXTp0jP2ExERwd69e9m6dSuHDh1iyZIlWvDh3EUaY77Dll4WY0whlacjXAm0EpEYEQkCbsZWQfM0EYkUkVPfJ6OwZUQBmAf0FZHa9psv+9qXeZ3z7UPa1hHaBR9i4YadHM8980ZC7bMrdi6f+Quvvc2XPy3nxna1CDM5VfN70k20z3YPR86ArxKRb7HdVJN3aqExZkr5m6gLFRQURHR0NCtWrODw4cMUFRURHh5OYk5N/Ju3ZuLwBBrXr8Hf//53li5dypw5c8jOziYgIIB69epx1VVXnXGTA8D06bbvzICAgNMFBkaMGEHTpk2teItnuv9+Sw9f3uedkJBAz549AWjUqJHDn3eb+uEkrU1kzLiVbOvZkvoRtTzr83ZjEY21a9cSExNDaGjoWa+1bduWBQsWkJGRwa233sqsWbN4//33iYyMpE+fPnz99Z/36nXu3Jl9+/bxww8/ABAXF8ell1561h31qkIn7HOzDYCIdAMqnANkjCkUkQexDZz9gY+MMZtEZBywyhgzHdtZ7okiYrBNQXnAvu1hEXkB2yAeYNypGzK9zYX0IZfWqsna6AT+yIlkcIl9ap9dMUc/85EjRzLs6Q9hVxIk7+ODnUuq3vek9tleRyqbBykiH5ex2Bhj7nZNSM7XuXNns2rVKqvDuGArth/i5skruK9XC57s72WXH3NybD/L+A9fVe08eIJ+b/xC77i6vHvbxVaHo6ooEVltjOlc+ZoO7asT8BbQDtiILW/39caYDc7Yv7N4S599LsZOS+TrP9KZ948etKwbbnU4latCffZPm/Yx8vPVjBvalju6N7M6HOXlHO2zHcmCcldl6yjXyyssYvTURBrXqcZDV7WyOhznGzjQ9nPxYkvDcKZmkWE81LsVr8xLYWHy/tPZapSyijFmjYhcAbTGdoNkijFGC6p5gEf6xPLj2j28OHszH41wd73D81BF+uz8wmImztlMy7rVubVr1b25V3mfcgfgIvKEMeZfIvIWZdy1box5yKWRqTO8t3gb2zNP8NndXakW5Nid28p6f+vRnB/X7eaZHzfRrXkEYcGOzPpyg1MFNDz8y1M5l72ozkCgGbb+v689I8ykCjdULhdRPZgHr2rJxDmbWbo1kx6ttOS3M3y+Io0dB0/w8YguBPifX7ICj6B9ttep6Lfx1I2Xq4DVZTTlJqkHsnl30TaGdmxIz1jtlKuSoAA/Jg5PYPfRk0yav8XqcJSaAYwAIoDwEk15gBGXNaNxnWqMn5ns0vRnvuJoTj7/XriVHq0i6dVavzuVZyn3dJwxZob956fuC0eVZoxhzNREQgL9GHtN6azSqiq4uGkdbrukCR8v28Gwjo1IiK5pdUjKd0Xby8UrDxQc4M+oAXHc/+Uavl2Zzq2X6JSJC/HGgq0czy1g7DXxZ2U7UcpqlV6PEZFYEZksIj+JyM+nmjuCU/C/1Rn8vuMwowfGERXu3gpQynme6N+GiOrBjJq6gcJyytQr5QZzRKSv1UGo8g1oV5+uzeowaX4Kx3N1ev752paZzRcr0ri5axNa19eLPMrzODIh6n/AWmAs8HiJplzsUHYeL85OpmuzOtzYuXHlG1RlI0bYmpeqWS2QZwfHs3H3MT75bafV4SjftQKYKiInReSYiBwXkWNWB6X+JCKMHRTHwex83l28zepwyufhffbE2cmEBPrz6NWxla+slAUcuSOs0BjznssjUWcZPyuZE3mFvDi83RnVFL2SB3fkznJNQgN+aJ3BpPlbGJDQgEa1qlkXjMUFNJRlXgO6A4mmshy0yjLto2sx/KJGfPjrDm7t2oTGdTww1Z8H99nLUg+yIPkAT/ZvQ2R1L7lyrH2213HkDPgMEblfRBqISJ1TzeWR+bhftx5k6trd3HdFi6qRE/ZCHTxoa15MRBg3tB3GwLM/bsTS8c/993tEIQ3ldluBjTr49nyP92+Nn8DLczdbHUrZPLTPLio2vDAzieja1bjrsmZWh+M82md7HUcG4Hdim3LyG39mQHGoQoKI9BeRFBFJFZGnyng9WES+tb/+u4g0K/HaKPvyFBHpV2o7fxFZKyIzHYmjqsktKGLMtERiIsO4/8qWVofjHtdfb2ternGdUB69OpYFyQeYu3GfdYHk5PxZSEP5kr3AYnv/+uipZnVQ6mwNalZjZM8WzNywl9VpHlg81EP77P+tSmfzvuM8NaANIYFelLJX+2yvU+EAXET8gNuNMTGlWvPKdmzPN/sOMACIB24RkdJpPO4BjhhjWgKvAy/bt40HbgbaAv2Bd+37O+Vh/kyT6HXe+nkraYdymDCsnXd1IAqAuy5rRtuGNXh2+iaOWXWT1cCBfxbSUL5kB7AQCELTEHq8e69oTr0awYybmUyxpiWsVHZeIa/+tIXOTWtzTUIDq8NxLu2zvU6FA3BjTDHw6nnuuyuQaozZbozJB74BhpZaZyhwKs3h90BvseUKGgp8Y4zJM8bsAFLt+0NEooFrgA/OMy6PtmX/cd5fsp3rOkVzactIq8NRLhDgb8sNfjA7j1fmplgdjvIhxpjnjTHPA5OA10o8Vx4oNCiAx/u1YX36UWZs2GN1OB7vvcWpHMzOY+wgTTuoPJ8jU1B+EpHr5Nx/mxsB6SWeZ9iXlbmOMaYQyMJWIKKibd8AngAqzOUmIiNFZJWIrMrMzDzH0K1RXGwYNSWR8JAAxlwTZ3U4yoXaR9fiju7N+OL3NFanHbE6HOUjRKSdiKwFNgKbRGS1iLS1Oi5VvuEXNaJdoxq8PGczJ/OLrA7HY2UcyeG/S3cwrGNDOjauZXU4SlXKkQH4o9hSEeadY9qqsgbspa+hlbdOmctFZBBwwBhTaSVOY8xkY0xnY0znqKiqUQHr65W7WJ12hDHXxFMnLMjqcJSLPdavNfVrhDB6SiIFmhtcucdk4FFjTFNjTFPgn8B/LY5JVcDPTxh7TTx7snL58NftVofjsf41NwXBVnNBqaqg0gG4MSbcGONnjAkyxtSwP6/hwL4zgJLJq6OB0tfQTq8jIgFATeBwBdteBgwRkZ3YprRcJSJfOBCLxztwPJeX5myme/MIrutU+kKBD7jvPlvzIdWDA3h+SFtS9h/ng6U7rA5H+YYwY8yiU0+MMYuBMOvCUY7o1jyCfm3r8e7ibRw4lmt1ODYe1Gev2XWE6ev3MLJncxpamd5VqXPgSB5wRKQ20AoIObXMGPNLJZutBFqJSAywG9tNlbeWWmc6tiwry4HrgZ+NMUZEpgNficgkoKH92H8YY5YDo+wx9QIeM8bc7sh78HTjZiSRV1jMhGvb+ebctZtusjoCS/RtW59+bevx5sItXJPQgCYRbsr368E5fJVLbReRp4HP7c9vx3ZjpvJwowbE8fPmJbz20xZevr691eF4TJ9tjC3tYN3wYO69ooXV4biO9tlex5FS9H8FfgHmAc/bfz5X2Xb2Od0P2tdPBr4zxmwSkXEiMsS+2odAhIikYpvq8pR9203Ad0ASMBd4wBjjtZPfFqUcYOaGvTx4ZUuaR1W3OhxrpKfbmg96fkg7Avz8GDMt0X25wT28ip1ymbuBKOAHYAoQCYywMiDlmGaRYYy4tBnfrU5n054sq8PxmD57xoa9rN11lMf6tSYs2KFzilWT9tlex5E54A8DXYA0Y8yVwEWAQ3c1GmNmG2NijTEtjDET7MueMcZMtz/ONcbcYIxpaYzpaozZXmLbCfbtWhtj5pSx78XGmEGOxOHJcvILGTt1Iy3rVufvV1Sa3dF7/eUvtuaD6tcM4fF+rVm69SDT17sp04GHFtFQLtcC2/Q+PyAQ6I3tBIuqAh68qhW1qgUyfmaytYW8wCP67NyCIl6es5n4BjW4rlO0pbG4nPbZXseRAXiuMSYXbIVzjDGbgdauDct3vLlgK7uPnuTFaxMIDtCc377q9m5N6di4FuNmJHE0J9/1B/TQIhrK5b4EPgKGA4PsbbClESmH1awWyCNXx7J8+yEWJB+wOhzLffjrDnYfPcnYQXH4+3n51E3ts72OIwPwDBGpBUwD5ovIj5x9M6U6D5v2ZPHBrzu4uUtjusbUsTocZSF/P+HFaxM4erKAibM9tPS08gaZxpgZxpgdxpi0U83qoJTjbunahBZRYbw4O5n8Qt/NnnTgeC7vLkrl6vh6XNpCa2aoqseRLCjXGmOOGmOeA57GNm97mKsD83ZFxYbRUzdSOzSQpwZo2iQF8Q1r8NfLY/h2VTq/bz9kdTjKOz0rIh+IyC0iMvxUszoo5bhAfz/GXhPPjoMn+GKF7/7t9Pr8LeQXFTN6oNbMUFWTI2fAEZHLReQuY8wSbBlLfDBPnnN9sSKN9elHeXpQPLVCNee3snm4Tyuia1dj9NRE8gq99r5jZZ27gI5Af2xTTwZjm4aiqpBeraPo0SqSNxdudc+UNQ+TvPcY365M547uzYiJ1CyaqmpyJAvKs8CT2NP/Ybtxxytyb1tlX1Yur8xLoUerSM5MyYQAACAASURBVIZ0aGh1OJ7hn/+0NR8XGhTA+GHt2JZ5gv8s1qIbyuk62AuU3WmMucve7rY6KHVuRGzFeY7nFvDGgq3WBGFRn22MYfysJGpUC+Shq1q5/fhKOYsjOXuuxZb5ZA2AMWaPiIS7NCov99z0TRQUFTNhWIJv5vwuy2C9D+yUXq3rMrhDQ95ZlMqgDg1o4YrUlB5SQEO53QoRiTfGJFkdiLowreuHc3PXJnyxIo2/dG/qmn6iIhb12T9vPsCy1EM8NziemqGBlsRgCe2zvY4jU1DyjS3fkQEQEb3ecwHmJ+1n7qZ9PNynlfuKrlQFKSm2pgB4ZlA8IYF+jJnqotzgN93kMYU0lFtdDqwTkRQR2SAiiSKyweqg1Pl5pE8sIYH+TJyd7P6DW9BnFxQVM2F2Ms2jwritW1O3Htty2md7HUcG4N+JyPtALRH5G7AA+K9rw/JO2XmFPPPjRtrUD+dvPXw453dZ/v53W1MARIUHM2pgHCu2H+Z/qzOcfwAPKaKh3K4/tsrCfflz/nelpzJFpL990J4qIk+V8XoTEVkkImvtA/uB9uWBIvKpfaCfLCKjzt67Ol9R4cE8cGVLFiQfYFmqm3NEW9Bnf7kije2ZJxgzMI5Af4duYfMe2md7HUeyoLwKfI+tclpr4BljzFuuDswbTfppC/uO5TLh2gTf6zzUObupc2O6NKvNi7OTOZSd59yde0ARDeV+JVMPOpqGUET8gXeAAUA8/H97dx4fVX3ucfzzZCchCxDWJECASAggCBFBULEqIqJYW6/aa2tvvaW12tZebYtKW2txaa3aTXvl3lqq7dVaRYuCUKrgAiq7LGELqwGUAGENhCy/+8c52BgSSMgsycz3/Xqd15w5c86Z58fMPDw5y+/HjWZWUGe1yXijHZ8D3AA86S+/Dkh0zg0EhgLfMLOegWuR/MfInmS3a8PPXiuiuibMg/ME0YHySn71xkZG9cnkc/mdwh1O6ClnR5xGVYHOubnOue875+5yzs0NdlCRaFXJAaYt3MK/n9edoT3ahTscaQVi/L7Bj1RUMWVmGE4xi3iGAcXOuc3OuePA88CEOus4IM2fT+dfY0U4IMXM4oA2wHHgYPBDjh5J8bFMuiKfdR8f4m9LIvcI6a/f2MjBo5Xce2U/3TslEaHBAtzMDpnZwXqmQ2amBNoEVdU1TJq+ksy2ifxgrPr8lsbL65zKNy/qzcvLd/DuRg1DLGGRBdSu7Eo4uSva+4CbzKwEmAV821/+InAE2AVsB37pnNsX1Gij0JUDuzK0Rzt++Y8NHK6oCnc4Abe59DDPvLeV68/NoV/XtNOuL9IaNFiAO+dSnXNp9Uypzjn9Appg2sKtrNl5kPuu7k9aUhTdtS0BcdvFfcjNTOHeV1ZxrFJ9g0vI1Xe4se61DjcC05xz2cA44Fkzi8E7el4NdANygTvNrN4bYMxsopktMbMlpaWlgYs+CpgZPxpfwJ7DFfx+fnG4wwm4h15fR2JcDN+77KxwhyISMLoQOch27D/KY3M38Ln8TlwxoEu4w2m5Jk/2JjlJUnwsD1wzgG17y/ntm2Hq81eiWQmQU+t5Nv+6xOSEW4AXAJxz7wFJQCbwJWC2c67SObcbWAAU1vcmzrmpfh/lhR07dgxwEyLf4JwMrhncjf95ZwslZeXBf8MQ5eyFm/Ywt+gTvnVxHzqlJgX9/URCRQV4EDnn+PErq3EO7p/QX9etncqll3qT1Ov8Ppl8YUg2T721mfUfH2r+DjXwkTTeYiDPzHLNLAHvJssZddbZDlwCYGb98ArwUn/558yTAgwH1oUs8ijz/bH5GPCL2SHoHjAEObu6xjHltbVkZbThllG5QX2vFk85O+KoAA+i2as/5o11u7lzzFlkt1Of36e0YoU3SYPuvbIfqUlx3PPyKmqa29vBVVdp8CNpFOdcFXA7MAdYi9fbyRozu9/MrvZXuxP4upl9CDwHfNUfP+IJoC2wGq+Q/6NzTv2OB0lWRhsmXtiLGR/uZNn2suC+WQhy9ktLSyjadZAfXpFPUnxsUN+rxVPOjjgWlEE+WpjCwkK3ZMmSkL7nwWOVXProW3RMTeTvt40kTt0Ontro0d7j/PnhjKLFe2lpCXf+7UOmXDOAm5ozEMWJATT69g1MYBI0ZrbUOVfvZRuRKhw5O1Icqahi9C/nk92uDdNvPT94Z16DnLND1o7WQjm71WhszlZVGCSPzF7PnsMVPHTtQBXfEjDXDsni/N4d+Pnsdew+eOzMd6SBj0QiUkpiHN8f05fl2/fz6spd4Q7njP33W5soPVTBj8YXqPgG5ewIpMowCJZtL+PPH2zj5vN7cnZ2RrjDkQhiZjzw+YFUVNXw09eKwh2OiLRAXxiaTUHXNH7++rpW2XPSzv1Hmfr2Zq4e1I0h3TVuhkQmFeABVlldwz3TV9ElLYk7x+hUkQRebmYK3764DzNX7mLeut3hDkdEWpjYGGPy+H7s2H+UP7y7JdzhNNkvZnv36f5grP4PlcilAjzA/vedLaz7+BA/vbo/bRPjwh2ORKhvXNSbvE5tmfzKasqPR97AGyLSPOf3zuSygs48Oa+Y0kMV4Q6n0VZ8tJ9XVuzkPy/IVecFEtFUgAfQ9r3l/PqNDVzevzNj+qvP7yZ58EFvkkZJiIvhwWsHsmP/UR6fuyHc4YhIC3TPuH5UVNXw2NwgdEsYhJztnONnrxWR2TaRW0f3Cei+RVoaHaINEOcck/++mriYGO67un+4w2l9zj8/3BG0Ouf2bM+Nw7rz9IKtTBicxYCs9MZvrEGPRCJebmYKXxnRk2kLt/CVET0DO4x7EHL2zFW7WLqtjIevHagzyHUpZ0ccHQEPkBkf7uTtDaXcNeYsuqa3CXc4rc/Chd4kTTJpbD7tkhO45+VVVDelb3ANfCQSFb57SR5pbeKZMrOIgHY7HOCcfayymodfX0d+l1SuK8w5/QbRRjk74gS1ADezsWa23syKzWxSPa8nmtlf/dc/MLOetV6721++3swu95flmNk8M1trZmvM7LvBjL+xDpRX8rPXihiUnc6XR/QMdzit0z33eJM0SXpyPD+5qoCVJQf408Ktjd9QAx+JRIX05Hi+e0keC4r38mYgb9oOcM7+44KtlJQd5UfjC4iNUbeDJ1HOjjhBK8DNLBZvFLQrgALgRjMrqLPaLUCZc64P8Djwc3/bArzhjvsDY4En/f1VAXc65/rhDWl8Wz37DLmHZ6+lrLySB68dqMQhITf+7K5cdFZHHv3HenbuP9q4je64w5tEJOLdNLwHvTqm8MCstVRW14Q7nJPsOVzBE/OKubRfJ0b2yQx3OC2TcnbECeYR8GFAsXNus3PuOPA8MKHOOhOAP/nzLwKXmNfj/gTgeedchXNuC1AMDHPO7XLOLQNwzh3CGxY5K4htOK1FW/bx3KKPuGVULv27NeEaXJEAMTOmXDOAauf4yYw14Q5HRFqY+NgY7h3Xj82lR/jL+9vCHc5JHpu7gWOV1dw9rl+4QxEJmWAW4FnAR7Wel3BysfzpOs65KuAA0KEx2/qXq5wDfBDAmJvkeFUN97y8iqyMNtxxaV64whAhp30y37v0LOYWfcLs1R+HOxwRaWE+l9+JkX068Ks3NnKgvDLc4Xxq3ccHeX7Rdm4a3oPeHduGOxyRkAlmAV7ftRh17wBpaJ1TbmtmbYGXgDuccwfrfXOziWa2xMyWlJaWNjLkpnnqrU0U7z7MlGsGkJygO7YlvL42Kpd+XdO4b8YaDh1rOf/Bikj4mRmTryzgwNFKfvPmxnCHA3i9hz0wcy2pSfE6iCVRJ5gFeAlQ+1bmbGBnQ+uYWRyQDuw71bZmFo9XfP/FOTe9oTd3zk11zhU65wo7duzYzKacbHPpYX47r5grz+7KxfmdAr7/qPOrX3mTnLH42BgeunYgnxw6xi/nBKHfXxFp1fp1TeP6whyeeW8rW/Ycad7OApCz568v5Z2Ne/jOJXlkJCc0Lx6RViaYBfhiIM/Mcs0sAe+myhl11pkB3OzPfxF403n9JM0AbvB7SckF8oBF/vXhfwDWOuceC2Lsp+ScY/Irq0mMi+En48N+D2hkGDzYm6RZBudkcPOInjzz/jaWby9reEUNfCQSlf5rzFkkxMbw0Ky1zdtRM3N2ZXUNU2YWkZuZwpeH92heLNFAOTviBK0A96/pvh2Yg3ez5AvOuTVmdr+ZXe2v9gegg5kVA/8FTPK3XQO8ABQBs4HbnHPVwEjgy8DnzGyFP40LVhsaMn3ZDhZu2ssPx+bTKS0p1G8fmf75T2+SZrtzzFl0Tk3i7umrGu7x4PzzNfiRSBTqlJrEty7uwz+KPmHhpj1nvqNm5uznFm1nU+kR7hnXj4Q4DUlyWsrZEccC2jF/C1VYWOiWLFkSkH3tO3KcSx6dT6+ObfnbN0YQo24HA2P0aO9x/vxwRhEx5qz5mG88u5S7r8jnGxf1PnmFEwNoKKG3eGa21DlXGO44QimQOVtOdqyymksefYv0NvG8+u1RZ9Z9bjNy9oHySkb/ch75XdL4v6+fh3dyW05JObvVaGzO1p+dTfTgrLUcOlbFg58fqOJbWqzL+3fhsoLOPP7PDXy0r/zkFTTwkUjUSoqP5YdX5FO06yAvLSsJ+fv/bt5G9h+tZPL4fiq+G0s5O+KoAG+ChZv28OLSEiZe2Iu+XVLDHY7IKf306v7EmjH5ldWBHYJaRFq9q87uyjndM3hkznqOVFSF7H237jnCtIVbuW5otsbOkKimAryRjlVWc+/Lq+nRIZnvXKLukqTl65bRhrsu78tbG0p5deWucIcjIi2ImfGj8QWUHqrgqbc2hex9H359HfGxMdw1pm/I3lOkJVIB3khPzt/Elj1HmHLNAJLiY8MdjkijfGVETwZlp3P/q2ta1OAbIhJ+Q7q346pB3Zj6zmZ27j8a9Pd7f/NeZq/5mFsv6q0ODCTqqQBvhOLdh/j9/GKuGdyNC/IC36e4AE895U0SULExxoPXDqSsvJKHZzez2zERiTg/HNsX5+AXs9c1bcMm5uyaGseUmUV0S0/i6xf2amKUIpFHwzeeRk2N457pq0lOiGOy+vwOnr46HRks/bulc8uoXKa+vZlrh2Rzbs/2GvRIRADIbpfMf16QyxPzNvHVkbkMzslo3IZNzNnTl+9g9Y6D/PqGwTqLfCaUsyOOjoCfxt+WfsSirfu4Z1w+mW0Twx1O5Hr1VW+SoLjj0jyyMtpwz/RVHK+q0cBHIvKpW0f3IbNtIlNeK2r8DdtNyNnlx6t4ZM46BuVkcNXZ3ZoRaRRTzo44KsBPYc/hCh6ctY5hue35t8KccIcT2R591JskKJIT4phyzQA27j7s3XClgY9ExNc2MY67xpzFkm1lzFr1ceM2akLOfuqtzXxysIIfj++n7nvPlHJ2xFEBfgpTXiui/LjX57f6KpXW7uL8Tlx5dld+O6+Yoz/5KUyZEu6QpJUws7Fmtt7Mis1sUj2vdzezeWa23MxW1h6h2MzONrP3zGyNma0yM9191wJdV5hDfpdUHnp9LccqqwO2310HjvLU25u48uyuDO3RPmD7jTpTpihnRxgV4A14Z2Mpr6zYya2j+9CnU9twhyMSED+5qoDEuBi27DmCegaXxjCzWOAJ4AqgALjRzOreEDMZeME5dw5wA/Ckv20c8Gfgm865/sBoQN3xtECxMV63hCVlR/njgq0B2+8js9dT42DS2PyA7VMkEugmzHocPe71+d0rM4Vvja5nGG+RVqpTahKTrsjn4NOVlJQdpXjdbuJjY4iPNeLjYkhJiCMjOZ70NvG6UUpOGAYUO+c2A5jZ88AEoKjWOg5I8+fTgZ3+/BhgpXPuQwDn3N6QRCxnZGSfTC7t14kn5hVzXWF2s+97Wlmyn+nLd3Dr6N7ktE8OUJQikUEFeD1eXr6D7fvKee7rw1WESMS58dzurE2KZ0dZOf8xbXGD6yXFx5DRJoGOqYl0TU+iW0YbuqYn0TWjDT3aJ9OrYwqpSfEhjFzCJAv4qNbzEuC8OuvcB/zDzL4NpACX+svPApyZzQE6As87535R35uY2URgIkD37t0DFrw0zd3j+nH542/z2NwNPPj5gWe8H+ccP3utiMy2CTqQJVIPFeD1uHFYDnmd23rdtUloPPtsuCOIGjExRr+uaRytrOKV20ZSWV1DZVUNFdU1HKmo4sDRSvaXV/qPx/nkYAVb9x5h4aa9HK4zZHWXtCR6d0qhd8e25HVqS/+sdAq6pukP18hS3w0wda9guhGY5px71MxGAM+a2QC8/2NGAecC5cAbZrbUOffGSTt0biowFaCwsFBXSIVJ745tuWl4D555bys3j+hJ3y6p9a94mpw9e/XHLN5axoOfH6g/1EXqoQK8Hmam4jvUctTLTCjFTH2KFGh8n7++g8cq2bn/KNv2lrOp9DDFuw+zqfQILy/bwSG/OI+NMfp0bEv/rDQGZqVzdnYGA7PSSYjTLSetVAlQ+weazb8uMTnhFmAsgHPuPf9Gy0x/27ecc3sAzGwWMAQ4qQCXluO7l+Tx8vIdTJlZxDNfG1Z/JwSnyNkVVdU89Po6+nZO5d8Ks4MYaRTRQHURRwW4tAx//av3eP314Y0jWpzhwEdpSfGkdYknv0vaZ5Y759h54Bhrdhxg9Y4DrN55kHc27mH6sh0AJMbFMCgng8Ie7Ti3Z3uG9GhHehsdFWslFgN5ZpYL7MC7yfJLddbZDlwCTDOzfkASUArMAX5gZsnAceAi4PFQBS5npl1KAt+5JI+fvVbE/PWlXJzf6eSVTpGzpy3YyvZ95Tx7yzDiYvWHd0BosLqIowJcWobf/957VAEeGicG0LjqqoDszszIymhDVkYbxvTv8unyTw4eY/n2/SzZuo/F28qY+vZmnpy/CTPo2zmV4b06MKpPJuf1aq/T1C2Uc67KzG7HK6Zjgaedc2vM7H5giXNuBnAn8D9m9j28y1O+6rwRXcrM7DG8It4Bs5xzM8PTEmmKLw/vwZ/f38aUmUWMysskvm4h3UDO3nu4gt+9Wczn8jtxQV7HEEUbBQKcsyX8VICLRKMTA2gEOZl3Tkti7IAujB3gFeXlx6tY8dF+lmwtY9GWfTy3aDvTFm4lNsY4OzudUX0yOb93JkN6ZJAYp+vIWwrn3CxgVp1lP641XwSMbGDbP+N1RSitSEJcDHdfkc/EZ5fy3KLtfGVEz0Zt9/g/N1BeWc0949TtYECFKGdL6KgAF5GQSU6I4/zeXpENcKyymmXby1hYvJcFm/bw5PxN/PbNYpLiYzi3Z3tG9clkZJ9MCrqmaQQ9kRC7rKAzI3p14PG5G5gwOOu0l41t+OQQ//fBdr48vAd9OjVw86aIACrARSSMkuJjPy3I76IvB49V8sHmfSwo3sOC4j089Po6ADqkJHB+n0xG9enAqLyOZGW0CXPkIpHPzJg8vh/jf/suv3tzI/deWXf8pc96YOZaUhLj+O6lZ4UoQpHWSwW4iLQYaUnxXFbQmcsKOgPeNeTvbvSK8XeK9/Dqh17nG70yUxjZJ5NReZmM6N2BNF0/LhIU/bulc93QbKYt3MpNw3vQo0NKvevNX7+btzaUMvnKfrRPSQhxlCKtjwpwaRlefDHcEUgL1DktiS8MzeYLQ7NxzrHhk8O8W7yHdzeW8tKyEp59fxsxBoNyMrigTyaj8joyOCdDXR6KBNBdY/ry2spdPDRrHf/95aHewlo5u6q6hgdmrqVnh+RGXysuEu1UgEvLkJkZ7giiSysc+MjM6Nsllb5dUrllVC7Hq2pYvr3MK8iL9/C7ecX85s1iUhJiOc/vXWVUXiZ5ndrW34+xiDRKp7Qkbr2oN4/O3cD7m/cyvFeHz+Ts5xZ/xMbdh/nvm4bqj99gaYU5W05NBbi0DNOmeY9f/Wo4o4geETDwUUJcDOf16sB5vTpw55i+HDhayXub9vJucSkLivfy5rrdAHROS/QuV/GnTmlJYY5cpPX5+oW9eG7RdqbMLGLGbaOIeeZPABy84d95fO4Gzsttz+X9O4c5yggWATlbPksFuLQMKsBDKwIHPkpvE/+ZLg9Lysp5d6N3dHzeut2fDgrUo0MyQ7q3Y0j3DM7p3o78LqkaLETkNJLiY/nB2Hzu+OsKpi/fwRf9nP1Ep/MoKz/Oj8YX6ExTMEVgzo52QS3AzWws8Gu8wRv+1zn3cJ3XE4FngKHAXuB659xW/7W78YY3rga+45yb05h9ikgjRMHAR9ntkrlhWHduGNadmhpH0a6DLNy0h6XbvMtWXl7uFeTJCbGcnZ3OoJwMCrqm0a9rGr0yU1SUi9Rx9aBu/HHhVh6Zs47PO0dlleOPC7byhSHZDMhKD3d4kS0Kcna0CVoBbmaxwBPAZUAJsNjMZvgDNpxwC1DmnOtjZjcAPweuN7MCvOGO+wPdgH+a2Yl+jU63TxGRz4iJMQZkpX9aJDjnKCk7yrLtZSzbVsay7ft5+t0tVFY7wLu85azObenXJY28zm3p0SGF3MwUurdPJileAwRJdIqJMX48vh9f+P177Nx/jKPHq4iNMb5/uYZJF2mqYB4BHwYUO+c2A5jZ88AEoHaxPAG4z59/EfideeewJgDPO+cqgC1mVuzvj0bsU0TklMyMnPbJ5LRPZsLgLACOV9WwqfQwa3cd9KdDvLluN39bWvKZbbumJ5HTLpmOaYl0Sk2kU2oSnVIT6dA2gdSkOFIS42jrT0nxscTFGLExptPzEhGG9mjPlWd3ZedzR3HO8c2LetNZ91WINFkwC/As4KNaz0uA8xpaxzlXZWYHgA7+8vfrbJvlz59unyIiTZYQF0M//xKU2g6UV7J17xFv2lPOtr1HKCk7StHOg8w/eIwjx6sbtf9eHVN4887RQYhcJLQmjc1n593eb2bihb3CHY5IqxTMAry+wz2ukes0tLy+izLr7tPbsdlEYCJA9+7dG45SWoZZs8IdgUi90pPjGZScwaCcjHpfP1JRxe5DFew7UsHhimqOVFRx+FgVhyuqOFpZTXWNo6rGnXYYb5HWIqd9Mjtee42UlATaJOiSLJEzEcwCvASo3W9ONrCzgXVKzCwOSAf2nWbb0+0TAOfcVGAqQGFhYb1FurQgycnhjiC6aOCjgElJjCM3MY7czPpHCBSJRMMH6sBWSClnR5xg3ua/GMgzs1wzS8C7qXJGnXVmADf7818E3nTOOX/5DWaWaGa5QB6wqJH7lNboySe9SUIjM1ODH4nImVPODi3l7IgTtCPg/jXdtwNz8LoMfNo5t8bM7geWOOdmAH8AnvVvstyHV1Djr/cC3s2VVcBtzrlqgPr2Gaw2SAi98IL3+K1vhTeOaKF+10WkOZSzQ0s5O+IEtR9w59wsYFadZT+uNX8MuK6BbR8AHmjMPkWkiZTMRURaD+XsiKORJkREREREQkgFuIiIiIhICKkAFxEREREJoaBeAy7SaPPnhzsCERFpLOVskWZRAS4SjTTwkYhI66GcHXFUgItEIw18JCLSeihnRxxdAy4SjTSIhohI66GcHXFUgItEoxde+NdAGiKnYWZjzWy9mRWb2aR6Xu9uZvPMbLmZrTSzcfW8ftjM7gpd1CIRRDk74qgAFxGRBplZLPAEcAVQANxoZgV1VpsMvOCcOwdvROO6h+oeB14PdqwiIq2FCnARETmVYUCxc26zc+448Dwwoc46Dkjz59OBnSdeMLNrgM3AmhDEKiLSKqgAFxGRU8kCPqr1vMRfVtt9wE1mVgLMAr4NYGYpwA+BnwY/TBGR1kMFuIiInIrVs8zVeX4jMM05lw2MA541sxi8wvtx59zh076J2UQzW2JmS0pLS5sdtIhIS6ZuCEWikQbRkMYrAXJqPc+m1iUmvluAsQDOuffMLAnIBM4DvmhmvwAygBozO+ac+13dN3HOTQWmAhQWFtYt8EWim3J2xFEBLiIip7IYyDOzXGAH3k2WX6qzznbgEmCamfUDkoBS59wFJ1Yws/uAw/UV3yIi0UaXoIiISIOcc1XA7cAcYC1ebydrzOx+M7vaX+1O4Otm9iHwHPBV55yOYouINEBHwEVE5JScc7Pwbq6svezHteaLgJGn2cd9QQlORKQV0hFwEREREZEQUgEuIiIiIhJCKsBFREREREJIBbiIiIiISAipABcRERERCSEV4CIiIiIiIWTR0FWrmZUC25q4WSawJwjhtERqa2RSWyNDD+dcx3AHEUpnmLMhsr8HdamtkUltbf0albOjogA/E2a2xDlXGO44QkFtjUxqq0SbaPoeqK2RSW2NHroERUREREQkhFSAi4iIiIiEkArwhk0NdwAhpLZGJrVVok00fQ/U1siktkYJXQMuIiIiIhJCOgIuIiIiIhJCKsDrYWZjzWy9mRWb2aRwxxMIZrbVzFaZ2QozW+Iva29mc81so//Yzl9uZvYbv/0rzWxIeKM/NTN72sx2m9nqWsua3DYzu9lff6OZ3RyOtpxOA229z8x2+J/tCjMbV+u1u/22rjezy2stb9HfcTPLMbN5ZrbWzNaY2Xf95RH5uUrztPTv85lQzo6M33a05GxQ3m4y55ymWhMQC2wCegEJwIdAQbjjCkC7tgKZdZb9Apjkz08Cfu7PjwNeBwwYDnwQ7vhP07YLgSHA6jNtG9Ae2Ow/tvPn24W7bY1s633AXfWsW+B/fxOBXP97HdsavuNAV2CIP58KbPDbE5Gfq6ZmfVda/Pf5DNulnH2KtrWW33a05Gw/fuXtJkw6An6yYUCxc26zc+448DwwIcwxBcsE4E/+/J+Aa2otf8Z53gcyzKxrOAJsDOfc28C+Ooub2rbLgbnOuX3OuTJgLjA2+NE3TQNtbcgE4HnnXIVzbgtQjPf9bvHfcefcLufcMn/+ELAWyCJCP1dplhb/fQ4g5exW9tuOlpwNyttNpQL8ZFnAR7Wel/jLWjsH/MPMlprZRH9ZZ+fcLvB+OEAnf3kk/Bs0tW2tvc23+6fwnj5xeo8IaauZwJklUAAABK9JREFU9QTOAT4g+j5XOb1I/YyVsz2R+tuO2JwNytuNoQL8ZFbPskjoKmakc24IcAVwm5ldeIp1I/XfABpuW2tu8++B3sBgYBfwqL+81bfVzNoCLwF3OOcOnmrVepa1qrbKGYvUz1g52xOJv+2IzdmgvN1YKsBPVgLk1HqeDewMUywB45zb6T/uBl7GO6X1yYnTlP7jbn/1SPg3aGrbWm2bnXOfOOeqnXM1wP/gfbbQyttqZvF4Sfwvzrnp/uKo+Vyl0SLyM1bOjtzfdqTmbFDebgoV4CdbDOSZWa6ZJQA3ADPCHFOzmFmKmaWemAfGAKvx2nXi7uKbgb/78zOAr/h3KA8HDpw4fdSKNLVtc4AxZtbOPx04xl/W4tW51vPzeJ8teG29wcwSzSwXyAMW0Qq+42ZmwB+Atc65x2q9FDWfqzRai/8+N5VydmT/tiMxZ4PydpOF+y7Qljjh3Zm7Ae+u43vDHU8A2tML767pD4E1J9oEdADeADb6j+395QY84bd/FVAY7jacpn3P4Z3Gq8T7y/mWM2kb8DW8m16Kgf8Id7ua0NZn/basxEtoXWutf6/f1vXAFbWWt+jvODAK75TjSmCFP42L1M9VU7O/Ly36+3wG7VHOjpDfdrTkbD9G5e0mTBoJU0REREQkhHQJioiIiIhICKkAFxEREREJIRXgIiIiIiIhpAJcRERERCSEVICLiIiIiISQCnCJCma20H/saWZfCvC+76nvvURE5MwoZ0ukUzeEElXMbDRwl3NufBO2iXXOVZ/i9cPOubaBiE9ERP5FOVsilY6AS1Qws8P+7MPABWa2wsy+Z2axZvaImS02s5Vm9g1//dFmNs/M/g9vgADM7BUzW2pma8xsor/sYaCNv7+/1H4vf3SvR8xstZmtMrPra+17vpm9aGbrzOwv/ghimNnDZlbkx/LLUP4biYi0FMrZEuniwh2ASIhNotbRFD8pH3DOnWtmicACM/uHv+4wYIBzbov//GvOuX1m1gZYbGYvOecmmdntzrnB9bzXtcBgYBCQ6W/ztv/aOUB/YCewABhpZkV4wxLnO+ecmWUEvPUiIq2LcrZEJB0Bl2g3BviKma0APsAbMjfPf21RrUQO8B0z+xB4H8iptV5DRgHPOeeqnXOfAG8B59bad4lzrgZvuN6ewEHgGPC/ZnYtUN7s1omIRBblbIkIKsAl2hnwbefcYH/Kdc6dOJpy5NOVvOsQLwVGOOcGAcuBpEbsuyEVteargTjnXBXeEZyXgGuA2U1qiYhI5FPOloigAlyizSEgtdbzOcCtZhYPYGZnmVlKPdulA2XOuXIzyweG13qt8sT2dbwNXO9fs9gRuBBY1FBgZtYWSHfOzQLuwDsVKiISzZSzJSLpGnCJNiuBKv+05DTg13inEpf5N9WU4h3JqGs28E0zWwmsxzulecJUYKWZLXPO/Xut5S8DI4APAQf8wDn3sf+fQX1Sgb+bWRLekZjvnVkTRUQihnK2RCR1QygiIiIiEkK6BEVEREREJIRUgIuIiIiIhJAKcBERERGREFIBLiIiIiISQirARURERERCSAW4iIiIiEgIqQAXEREREQkhFeAiIiIiIiH0/1NMTRuuRLkbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, you can even change your data during the training. This is only applicable for a full CNN that can works with any size, but you could decide to train it for a bit with a smaller size before increasing it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ConvBN(n_in, n_out, stride):\n",
    "    return nn.Sequential(nn.Conv2d(n_in,n_out,3,stride=stride,padding=1), nn.BatchNorm2d(n_out))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ShallowConvNet():\n",
    "    listlayers = [ConvBN(3,64,20), nn.ReLU(inplace=True), ConvBN(64,128,2), nn.ReLU(inplace=True),\n",
    "                  nn.AdaptiveAvgPool2d(1), Flatten(), nn.Linear(128,10), nn.LogSoftmax(dim=0)]\n",
    "    return nn.Sequential(*listlayers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's grab the data for two different sizes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1 = get_data(28,batch_size)\n",
    "data2 = get_data(32,batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And create a learner object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.from_model_data(ShallowConvNet(), data1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's say we want to phases of cos anneal with Adam, but want to train on the size 28 for the first, then on the size 32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [TrainingPhase(epochs=1, opt_fn=optim.Adam, lr=1e-2, lr_decay=DecayType.COSINE),\n",
    "          TrainingPhase(epochs=2, opt_fn=optim.Adam, lr=1e-2, lr_decay=DecayType.COSINE)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's as simple as passing a list of data in the arguments of fir_opt_sched. One thing to pay attention to, this list must have the same size as phases, so if the same data object should be used through multiple phases, repeat it as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd8d1bb710fa4a818ae9d6fd42047bf5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      3.701185   4.411829   0.156     \n",
      "    1      3.638553   4.40993    0.1406                                                                                \n",
      "    2      3.605458   4.432413   0.1452                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.43241]), 0.1452]"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit_opt_sched(phases, data_list=[data1,data2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VFX6wPHvO5MGIQkhCQkkgYROCAIhhGIHFERdpCjFrmsXy64F3dW1rPvT1bWLimKXJiAiKqAiiIUSkkgvgVBCh4QaIIXz+2MGDTEhQyaTm5m8n+e5T2bunHvnvfeZyTv3nHvOEWMMSimlVFXZrA5AKaWUd9NEopRSyi2aSJRSSrlFE4lSSim3aCJRSinlFk0kSiml3KKJRCmllFs0kSillHKLJhKllFJu8bM6gJoQGRlpEhISrA5DKaW8xtKlS/caY6JcKVsnEklCQgLp6elWh6GUUl5DRDa7WlartpRSSrlFE4lSSim3aCJRSinlFk0kSiml3KKJRCmllFs0kSillHKLJhKllFJu0USilFLKLZpIlFJKuUUTifIu333nWJRStUadGCJF+ZB//9vxt29fa+NQSv1Or0iUUkq5RROJUkopt2giUUop5RaPJhIR6S8ia0UkW0RGl/N6oIhMcr6+SEQSnOsjROQHETksIq+X2aariCx3bvOqiIgnj0EppdTpeayxXUTswBvARUAusEREZhhjVpUqdjOQb4xpJSLDgeeAYcAx4DEg2bmU9iZwK7AQ+BroD3zjqeNQtczbb1sdgVKqDE/etZUGZBtjNgKIyERgIFA6kQwEnnA+ngK8LiJijDkC/CQirUrvUESaAKHGmF+dzz8CrsBDieTFb9cR6GcjIjiApKahtG8Sir9dawOttDOmOROXbMFv23oC/ezUC7AT2SCQmLAgYkKDaBwSiM2mF6lK1SRPJpJYYGup57lA94rKGGOKReQAEAHsPc0+c8vsM7a8giJyK44rF5o1a3amsWOM4f2fczh0rPj3dQ3r+9MvKYbreyWQ1DT0jPep3Jc15iOWL83l+1ZlP0oO9QPstI4OoV10CMlxYXRPbESrqAaaXJTyIE8mkvK+uaYKZapU3hgzFhgLkJqaerp9lv9GIix/oh/Hi0vYdeA4y7bt57tVu/hy2XYmpW/loqRoHr8sifhG9c9018oNHSeNo2FeAS998hgCFBSWsOfQcXYeOMaOg8fYsPswa3ce4tvVu5iU7vgdE17fn+6JEVyUFE3vdo0JDw6w9iCU8jGeTCS5QHyp53HA9grK5IqIHxAG5FWyz7hK9lmtAv3sNIuoT7OI+lx2VlMOFBTxwS+bGPvjBi56aT4P9mvHTWcnoG3+NSvAbiPI305IkD/RoUEkx4ad8roxhq15R1mUs4/FOXksWL+XWSt3YrcJaQmNuKJLUy49qykNArVPrlLu8uS3aAnQWkQSgW3AcGBkmTIzgOuBX4GhwFxjTIVXD8aYHSJySER6AIuA64DXPBF8RcLq+3Nv39Zc1S2Ox6av4OmZq1iSk8fzV55FSJB/TYaiTkNEfv8BcGVqPMYYlm87wOyVO/lm+U4enrqcJ79cxYCOTRiR1oyuzcOtDlkpryWn+b/t/s5FBgAvA3bgPWPMMyLyFJBujJkhIkHAx0AXHFciw0s1zm8CQoEAYD9wsTFmlYikAh8A9XA0so86XfIBR9VWenp6tR+fMYZ3F+Tw7Kw1tIkO4aOb0ogKCaz291F/2Na5O1vzCui8IYsgf3uV9mGMIWPLfj5L38qXv23nSGEJXZuHc8u5LbgoKRq7tqcohYgsNcakulTWk4mktvBUIjlp/ro93P7xUqJDA/nkr92JC9d2E0+pjkRS2pHjxXyWvpVxP+ewNe8oiZHB3NOnFX/pFKsJRdVpZ5JI9F7WanB+myg++Wt38o4Ucs27i9hz6LjVIfms7x99gfsv+3u17S840I8bzk5k3gMXMubqFOr527l/0m9c8sqPzFqxk7rwQ0spd2kiqSZdm4fz/o1p7Dp4nOvfW8zBY0VWh+STDjduwo7QqGrfr90mDOjYhJmjzuGNkSkUnzDc/slShrz5Cyu2Haj291PKl2giqUZdm4fz1rVdWbfrEKPGZ1JyQn/NVrdWc7/istU/emz/Nptw6VlNmHPfeTw3pCNb8gq4/PWfePTz5eQfKfTY+yrlzTSRVLPz20Tx5MAOzF+3hxfmrLU6HJ/TYcZ4rsn82uPv42e3MaxbM+Y+cAE39kpk0pKtXPDCPKYszdXqLqXK0ETiAVd3b87I7s14c94Gvlq2w+pwlBtCg/x5/PIkvr7nXFo3bsADn/3GzR+ms/PAMatDU6rW0ETiIU9c3oEuzRoyetoytu0/anU4yk1tY0KYdFtPHr8siV827OWil+YzOX2rXp0ohSYSjwnws/HKsC6cOGG4f2KWtpdUMysGErDbhJvOSWTWvefRPiaUh6Ys4+4JmXpjharzNJF4ULOI+jw1MJnFm/J4a/4Gq8NR1SQhMpiJt/bgof5tmbViJ5e+uoDMLflWh6WUZTSReNjglFguPasJr3y3nuzdh6wOx+vNeuI17rjiEavDwGYT7rygFZNv68mJE3DlW78y9scNWtWl6iRNJB4mIjxxeQfqBdh5ZNpyTmgVl1uOhTUiv35Y5QVrSNfm4Xx977lclBTNf75ew90TMikoLK58Q6V8iCaSGhAVEsg/Lm3Pkk35jF+8xepwvFq7WVMZuvw7q8M4RVg9f8ZcncIjl7Tjm+U7GDzmF7bmFVgdllI1RhNJDbmyaxy9Wkbw3DdrdAgVN7SbPa3WJRJwXHnedn5L3r8xje37j3L56z/x0/qK5mdTyrdoIqkhIsK/r0jmWHEJL8zWjoruknLnOLPe+W2imHH3OTQOCeT69xczaYlegSrfp4mkBrWIasANvRKYvHSrjt/kwxIig5l6Ry/ObhXJw1OX8785a7URXvk0TSQ1bFSf1jSqH8CTX67Ufy4+LCTIn3HXpzIsNZ7X5mbz98m/UVh8wuqwlPIITSQ1LDTInwf6tWXJpnxm6vApPs3fbuPZIR35+0VtmJa5jRveX8wh7byofJAmEgtclRpPu5gQXpizlqIS/ZV6Jr78zzvccOUTVofhMhFhVJ/WvHhVJxbn5HHNu4t0FGHlczSRWMBuEx7s15bN+wr4LD3X6nC8SnFQPY75B1kyRIo7BqfE8dY1XVm98xDDxy5k90Ed9FH5Dk0kFundrjFdm4fzyvfrOFZUYnU4XiP5i0+5JuMrq8Ookr5J0XxwYze25hdw5du/al8T5TM0kVhExHFVsuvgcT7+dbPV4XiN1vO/4bI1C6wOo8p6tYzkk792J/9IIVe9/Ssb9xy2OiSl3KaJxEI9WkRwXpsoxszL1kbYOiSlWTiTbutJYfEJRryzkJy9R6wOSSm3aCKx2IMXtyW/oIiP9KrEJb5yx3T7JqGMv6UHRSWGEWMXskmTifJimkgs1jEujAvbRjHupxwd7K+OaRsTwvhbunO8uIQR7yxkyz5tM1HeSRNJLXB379bkHSlk/CIdTsNVXnbTVoXaxYTy6V97cLTIkUy0AV55I00ktUDX5uH0bBHBOws26h1clfj8xU8YPvJZq8OoVklNQ/nk5u4cPl7MiHcW6nzwyutoIqklRvVuxa6Dx5myVPuV1EXJsWF8fHMa+wuKuHacdlpU3kUTSS3Rs2UEXZo15M15G7S3+2l0njyOWxZNszoMjzgrriHvXJfK5rwCbvhgCYePa5uZ8g4eTSQi0l9E1opItoiMLuf1QBGZ5Hx9kYgklHrtEef6tSLSr9T6+0VkpYisEJEJIhLkyWOoKSLCXRe0Ytv+o3y9XMfgqkjiwh/os2Gx1WF4TM+WEbwxMoUV2w5w28fpHC/Wqk5V+3kskYiIHXgDuARIAkaISFKZYjcD+caYVsBLwHPObZOA4UAHoD8wRkTsIhIL3AOkGmOSAbuznE/o3a4xLaKCeXdBjo4MXAnxtjFSzsBFSdH8d8hZ/Jy9j3snZFGsV6iqlvPkFUkakG2M2WiMKQQmAgPLlBkIfOh8PAXoI47/EAOBicaY48aYHCDbuT8AP6CeiPgB9YHtHjyGGmWzCTefk8jybQdYlJNndTjKQkO6xvH4ZUnMWrmTf05foT8sVK3myUQSC2wt9TzXua7cMsaYYuAAEFHRtsaYbcALwBZgB3DAGDOnvDcXkVtFJF1E0vfs2VMNh1MzhqTE0Sg4gHcXbLQ6FGWxm85JZFTvVkxcspXX5mZbHY5SFfJkIimv7qHsz6qKypS7XkTCcVytJAJNgWARuaa8NzfGjDXGpBpjUqOios4gbGsF+du5pkdzvlu9mw06DtOfFAcGcswv0OowaszfLmrD4JRYXvx2nd7Rp2otTyaSXCC+1PM4/lwN9XsZZ1VVGJB3mm37AjnGmD3GmCJgGtDLI9Fb6NoezQnws/HeTzlWh1LrfPGfcdxw1ZNWh1FjRIRnB5/F2a0iGD11GT9n77U6JKX+xJOJZAnQWkQSRSQAR6P4jDJlZgDXOx8PBeYaR2XwDGC4866uRKA1sBhHlVYPEanvbEvpA6z24DFYIiokkEGdY5myNJc87U9QLt9tav+zAD8bb17TlZZRDbj946Ws2XnQ6pCUOoXHEomzzeNuYDaOf/aTjTErReQpEfmLs9g4IEJEsoG/AaOd264EJgOrgFnAXcaYEmPMIhyN8hnAcmf8Yz11DFa6+dxEjhefYNKSrZUXrkPSPnmDUT9PsDqMGhca5M/7N3ajfqCdG99for3fVa0ideFukNTUVJOenm51GGds+Nhf2Zp3lB8fuhC7rS79Bq9Ybqfu5OYXkLZpGbY6eE5WbT/IVW//Snyj+ky5vSfBgX5Wh6R8lIgsNcakulJWe7bXYtf1TGDb/qP8sGa31aGoWiKpaShvXJ3C2p0HuX9SFidO+P4PQVX7aSKpxS5KiiY6NJCPFupcJeoP57eJ4rHLkpizahcvzFlrdThKaSKpzfztNkamNefHdXt0Fj11iht6JTAirRlj5m1geuY2q8NRdZwmklpuRPd4/O3CJ3pVAsDR0Ibk1wvFh0dIcYmI8ORfOtA9sREPTV1G5pZ8q0NSdZgmklqucUgQ/ZOb8Fn6Vo4W6gB+X/3rNe4Y9KjVYdQKAX423rqmKzGhQdzy0VK27z9qdUiqjtJE4gWu69mcg8eK+SJLqzDUqcKDAxh3fSrHikq45aN0na5ZWUITiRdIbR5Ou5gQPlmk1Vtnj/sfD83/wOowapXW0SG8NqILq3cc5MEpy3SAR1XjNJF4ARFhZPdmrNh2kBXbDlgdjqViVmWRsm2N1WHUOhe2a8yD/drx1bIdvKMDfqoaponESwzsHEuQv40Ji7dYHUqt4MvzkVTV7ee3YEDHGJ79Zo2OyaVqlCYSLxFWz58BHZvwRdZ2rQdX5RIR/ju0Ey2jGnD3+Axy8wusDknVEZpIvMiItGYcPl7MzGU6Fa8qX4NAP96+tivFJYbbP1nKsSK90095niYSL5LaPJyWUcFMrMPVW4cjo9kREml1GLVai6gGvDy8Myu2HdTZFVWN0ETiRUSE4d2akbFlP+t2HbI6HEt8M/oF7r/8AavDqPX6tI/m3j6tmbI0VzuzKo/TROJlBqfE4m8XbXRXlbq3T2v6tGvMk1+uIn1TntXhKB+micTLRDQI5OIOMXyeua1O1n9f8OYzPP6dT05BU+1sNuHFYZ2JC6/HXeMz2Hv4uNUhKR+licQLjejWjP0FRcxeudPqUGpc1IY1JO3WfhKuCqvnz5iru7K/oIj7JmZRosPOKw/QROKFerWMIL5RPSYu1tkTVeWSmoby1MAO/JS9l1e/X291OMoHuZxIRCTYk4Eo19lswpVd4/l14z625mlfAVW5q1LjGZISx6tz17Ng/R6rw1E+ptJEIiK9RGQVjnnXEZFOIjLG45Gp0xqcEgvA53VuLgqtmqkKEeHfVyTTpnEI903M0jnfVbVy5YrkJaAfsA/AGPMbcJ4ng1KViwuvT6+WEUxZmlun+gnkxyaQ0yjW6jC8Ur0AO29cncKxohLuHp9BUckJq0NSPsKlqi1jTNnK+Lp3u1AtNLRrHFvyCliyqe5MavTdfU/z6CWjrA7Da7Vq3ID/G3IW6ZvzeX62TtOrqocriWSriPQCjIgEiMgDOKu5lLX6J8cQHGBnylJtdFeu+0unplzbozljf9zInDp455+qfq4kktuBu4BYIBfoDNzpyaCUa+oH+DGgYxO+Wrajzgzk2Pflx/jPN69ZHYbX++dl7ekYG8YDn/2mN2wot7mSSNoaY642xkQbYxobY64B2ns6MOWaoV3jOFJYwqwVdeOXZfi2TSTm1bUbDKpfoJ+dMVenAHDX+AwKi7W9RFWdK4mkvJ9/+pOwluiW0IhmjeozZWmu1aEoLxPfqD7/HdqJZbkHeH62Thamqs6vohdEpCfQC4gSkb+VeikUsHs6MOUam00YkhLHy9+vIze/gLjw+laHpLxI/+QYruvZnHcW5NCzZQS920VbHZLyQqe7IgkAGuBINiGlloPAUFd2LiL9RWStiGSLyOhyXg8UkUnO1xeJSEKp1x5xrl8rIv1KrW8oIlNEZI2IrHYmvDptcEosxsDnGVrlo87cowPa075JKA98tkz7l6gqqTCRGGPmG2OeBHoYY54stbxojKl0nAURsQNvAJcAScAIEUkqU+xmIN8Y0wpHf5XnnNsmAcOBDkB/YIxzfwCvALOMMe2ATugdZMQ3qk/PFhFMyfD9PiW7W7RndXQLq8PwKUH+dl4f2YVjRSXcNylTx+NSZ8yVNpICEXleRL4WkbknFxe2SwOyjTEbjTGFwERgYJkyA4EPnY+nAH3EMRn3QGCiMea4MSYHyAbSRCQUR2fIcQDGmEJjzH4XYvF5Q7vGsXlfAembfbtPybw7HuXpvrdaHYbPaRnVgKcGJrNwYx6vz822OhzlZVxJJJ8Ca4BE4ElgE7DEhe1igdIdHHKd68otY4wpBg4AEafZtgWwB3hfRDJF5F0dA8zhko7OPiXpvt3o7uMXXJYakhLLoC6xvPL9OhZt3Gd1OMqLuJJIIowx44AiZ3XXTUAPF7aTctaV/TdQUZmK1vsBKcCbxpguwBHgT20vACJyq4iki0j6nj2+P0hd/QA/+ic34evlO3x6npIBzz3Ii1++YHUYPklEePqKZJpHBHPvxCzyjhRaHZLyEq4kkiLn3x0icqmIdAHiXNguF4gv9TwO2F5RGRHxA8KAvNNsmwvkGmMWOddPwZFY/sQYM9YYk2qMSY2KinIhXO83OCWWQ8eL+X71bqtD8ZiQvTuJOaS/lj2lQaAfr43oQt6RQh787Defb3NT1cOVRPJvEQkD/g48ALwL3O/CdkuA1iKSKCIBOBrPZ5QpMwO43vl4KDDXOD65M4Dhzru6EoHWwGJjzE4cQ7a0dW7TB1jlQix1Qo8WEUSHBvJ5pm9XbynPSo4N45EB7fh+zW7e+3mT1eEoL1BhPxL4/c6r1saYmTjaLy50dcfGmGIRuRuYjaPfyXvGmJUi8hSQboyZgaPR/GMRycZxJTLcue1KEZmMI0kUA3cZY07W14wCPnUmp43Aja4frm+z24QrOscy7qcc8o4U0ig4wOqQlJe6oVcCP2fv49lvVpOW0IiOcWFWh6RqMans0lVEfjDGuJxAaqPU1FSTnp5udRg1Ys3Og/R/eQFPDezAdT0TrA6n2m05K43tB47RY/Myq0PxeflHChnw6gIC/GzMHHUOIUH+VoekapCILDXGpLpS1pWqrV9E5HUROVdEUk4ubsaoPKRdTCjtYkKY5qOdE7e370xWbDurw6gTwoMDeGV4F7bmFfDY9BXaXqIq5Eoi6YWjY+BTwP+ci942U4sNTokla+t+cvYesTqUarfgpr/xwgU3WB1GnZGW2Ih7+7RhetZ2pvrojxPlvkoTiTHmwnKW3jURnKqav3SKRaQuTsOrPOHu3q3ontiIx79YwcY9h60OR9VCLs2QqLxLTFgQZ7eMZHrmNp+rjhj49D28Pu0Zq8OoU+w24eXhnQnwszFqQibHi323n5KqGk0kPmpQl1i25BWQscW3hkwJOrif8KOHrA6jzmkSVo/nh3Zi5faDPPeNTtGrTqWJxEf1S44hyN/ms43uquZdlBTNDb0SeO/nHOau2WV1OKoWqTSRiMjgcpY+ItK4JgJUZ6agoIAnnniCvTty6dchhpnLdujsd6ra3HN+M/yypjBq7Bx2HdQh55WDK1ckN+PozX61c3kH+Bvws4hc68HYlNOOHTt48sknGTdu3Bltd0WXWA4cLeKHtb47ZIpy35l8voL87Qzo2IRjRSe4b2KWDjmvANcSyQmgvTFmiDFmCI65RY4D3YGHPRmccsjIyKBbt27s3r2bMxmA8txWkUQ2CGS6D929tblzD35N6GR1GD7lTD9fjYIDuL9va37duI+35m+ogQhVbXfaIVKcEowxpStEdwNtjDF5IlJU0UaqehQVFbF8+XJuvPFGioqKyMzM5OKLL/799W3btjFz5kz27NlDZGQkvXv/cWe2n93GZR1jGPvJZJptn0dJ4VFCQ0Pp2rUrvXr1wjH1C0yfPp2CggKaNWvGwoULKSoqolu3bvTp04f58+ezZMkSRIQePXpwzjnn1PQpOMWvI+/knQUbuc/SKHxHVT9fl57VhLUF9fjfnDUcy16I/cheDh8+7PWfL1U1riSSBSIyE/jM+XwI8KNzHhCdVMrDVq1aRVhYGNHR0XTq1InPPvuMPn36YLfbKSwsZPz48TRv3pxBgwZx8OBBZs2adcr2V3RuylsT6xGadC5Xn9OGbdu28eWXX1KvXj1SUv4YoGDz5s2EhoZyww03sGPHDqZNm8bOnTuJiYnhpptuIicnh5kzZ9KiRQuaNm1a06fhd1qRUr2q+vkSEZ4ZlEzm5r28v3gXnz08hCaRDb3+86WqxpWqrbuAD4DOQBfgIxyDKB7x9jG4vEFGRgadOjmqcpo3b46/vz9r1zpuv1y+fDklJSVcccUVNG7cmFatWnHeeeedsn2nZuEkd+vF/NwSGjZsSIcOHUhNTWXFihWnlAsMDGTAgAFERkbSsWNHmjRpwqFDh+jbty8RERGkpqbSsGFDNm3aVCPHXZErH7uVcRMftzQGX+LO5ys0yJ/XRqZyLKo9ry/KIywszOs/X6pqKr0icQ7rPsW5qBqUl5fH1q1bGTp0KOD4FdixY0cyMjJISkpiz549REdHExDwxyi/cXGnThUjIqTUz2Pc9Lk8uvN7GgTYKClxJJXSoqKisNn++F0RHBxMUFDQKWWCg4M5csTaYVf8jh8jsFgnXKoO1fH56tIsnKHNC3nn4/fJXziF5CYNvPrzpaqm0kQiIoOB54DGOGYuFBz5JdTDsdV5GRkZnDhxgpdeeun3dSd7qh84cMClfaxYsQKzNYuAxokEd+jO7X3as3jxYtasWXNKObvdfspzESl3na/1lK/LquvzFbx7OWldu7D4SCD3Dz6HfZtW6+erjnGljeS/wOXGmNWeDkb94cSJE2RlZdG3b1/atGlzymvTpk0jKyuLqKgosrKyKCws/P1XY27uqZNabdmyhaRWCZzXrB3fbznOo+Hh5Of7Vm93deaq8/MVFxfHh0NHcMnLC/jnN5sZGakzWNY1rrSR7NIkUvPWrVtHQUEBKSkpNG7c+JQlOTmZzMxMkpOTsdlsfPHFF+zevZsNGzawYMGCU/YTERHBjh076BFRyPotO3h/ytdaD62q/fN1YOdWHr8ont+W/MKHc5ZYdFTKKq4kknQRmSQiI0r3bvd4ZHVcZmYmiYmJ1K9f/0+vdejQgf3795Obm8vIkSPJy8vj7bffZs6cOfTt2/eUsqmpqXTo0IEDq3/i6PI5zMnaSK9evWrqMKpddtoF/NAqzeowvF51f76mTp3K6nnT6duyAatKYli942BNHYqqBVyZIfH9clYbY8xNngmp+tWlGRJP565PM1i4cR8LH+2Dv907h1l79ps1vPdzDuv+fYnVoahyFBafYOhbv7B5XwFf33susQ3rWR2SqqJqnSHRGHNjOYvXJBH1h8Epsew7UsiP61zvHa/UmQjws/Hq8C4Ul5zgvomZFJfoOG91QYWJREQecv59TUReLbvUXIiqupzXJoqI4ACvHhF45MPX8fHHOjJPbZYQGcy/ByWzZFM+r83NtjocVQNOd9fWyQZ2rRPyEf52G5d3asr4xVs4cLSIsHr+VoekfNSgLnEsWL+X1+aup2fLCHq0iLA6JOVBFV6RGGO+dP79sLyl5kJU1WlwSiyFxSf4evkOq0OpEu1l4D2eHphM84hg7puYRf4R7UTqy1yZj6SNiIwVkTkiMvfkUhPBqerXMTaMVo0b8LkXV28p7xAc6MdrI7qw78hxHpyyTDsb+jBXbt35DMgE/gk8WGpRXkhEGNQllsWb8tiaV2B1OMrHJceG8XD/dny3ehcfL9xsdTjKQ1xJJMXGmDeNMYuNMUtPLh6PTHnMFV1iEYHPvXCekjXn9mdW+3OtDkOdgZvPSeTCtlH8+6vVrNqu/Ut8kSuJ5EsRuVNEmohIo5OLxyNTHhPbsB49EiOYlpHrddUNGZeNYHzqZVaHoc6AiPDClZ1oWM+fURMyKCgstjokVc1cSSTX46jK+gVY6lz0Ti4vNzgllk37CsjY4l1TyvgdPUpQkc4V7m0iGgTy8rDObNx7hCdnrLI6HFXNTptIRMQGXGOMSSyztHBl5yLSX0TWiki2iIwu5/VA5/Ar2SKySEQSSr32iHP9WhHpV2Y7u4hkOifcUlVwSccmBPnb+Dwzt/LCtchVT9zO2In/sjoMVQW9WkVy5wUtmZS+lS9/2251OKoanTaRGGNOAC9UZcciYgfeAC7BMc/7CBFJKlPsZiDfGNMKeAnHcPU4yw0HOgD9gTHO/Z10L3/0c1FV0CDQj34dYvjytx0cLy6xOpwzIlYHoKrsvr5t6NKsIY9OW643e/gQV6q25ojIEDk5AbPr0oBsY8xGY0whMBEYWKbMQOBkn5QpQB/n+wwEJhpjjhtjcoBs5/4QkTjgUuDdM4xHlTGoSywHjhbxwxodMkXVDH+7YwgVBEZNyKRIh1DxCa4kkr/huAX4uIjCtaB3AAAZ2UlEQVQcFJFDIuLKrRexwNZSz3Od68otY4wpBg4AEZVs+zLwEKCfQDed0yqSqJBApmV4V/WW8m7xjerz7OCzyNq6nxe/XWd1OKoauDJoY4gxxmaMCTDGhDqfuzI7YnlXMGVvEaqoTLnrReQyYLcrtx+LyK0iki4i6Xv26C/u8vjZbQzs1JQf1u7WnseqRl16VhNGpMXz1vwN/LR+r9XhKDe5NJa4iISLSJqInHdycWGzXCC+1PM4oGwL2+9lRMQPCAPyTrPt2cBfRGQTjqqy3iLySXlvbowZa4xJNcakRkVFuRBu3TQ4JY6iEsPMZd7R+LmszxV83qlv5QVVrff4ZR1oGdWA+ydnsffwcavDUW5wZYiUvwI/ArOBJ51/n3Bh30uA1iKSKCIBOBrPZ5QpMwPH7cUAQ4G5xtGxYQYw3HlXVyLQGlhsjHnEGBNnjElw7m+uMeYaF2JRFUhqGkq7mBCmesmQKcsvGsT0ThdbHYaqBvUC7Lw+sgsHjhbxwGe/ceKEd/VpUn9w5YrkXqAbsNkYcyHQBai0rsjZ5nE3jsSzGphsjFkpIk+JyF+cxcYBESKSjaMtZrRz25XAZGAVMAu4yxjjXbcWeZHBKbFkbd3Pxj2HrQ6lUvUO5NOw4IDVYahq0i4mlMcubc+8tXt47+ccq8NRVeTKDIlLjDHdRCQL6G6MOS4iWcaYzjUTovt0hsTT23XwGD3/73vuurAVf7+4rdXhnNbms9LYdeAYaZuXWR2KqibGGG7/ZClz1+xm2h1n0zEuzOqQFNU8QyKQKyINgenAtyLyBX9u61BeLDo0iLNbRfJ55jatXlA1TkR4bshZRDYIZNSEDA4f1yFUvI0rd20NMsbsN8Y8ATyGozrqCk8HpmrW4JRYcvOPsmRTntWhnJ7mOZ/UsH4Arwzvwpa8Ah6fvsLqcNQZcvWurXNE5EZjzHzgV/7cH0R5uX4dYggOsDNlqRf0KdGu7T4pLbER9/RpzbTMbdq3ycu4ctfWv4CHgUecq/yBcm+5Vd6rfoAfl57VhK+W79CqBWWZUb1bk5bYiMemryBn7xGrw1EucuWKZBDwF+AIgDFmOxDiyaCUNYZ1i6egsISvl9XeaXiXDhjGxK6XWh2G8hC7TXhleGf8/WzcMyGTwmIdwMIbuJJICp19OwyAiAR7NiRllZRm4bSICmZy+tbKC1tk9XmXMKvD+VaHoTyoSVg9/jvkLJZvO8B/Z62xOhzlAlcSyWQReRtoKCK3AN8B73g2LGUFEWFYajzpm/PJ3l07+5SE7NlBzAEd8sbXXdwhhut6Nufdn3L4fvUuq8NRlXDlrq0XcIzMOxVoCzxujHnN04EpawxKicVuEz5bWjuvSgb+7xGe/eJ5q8NQNeDRAe3p0DSU+ydlsWWfDjlfm7l015Yx5ltjzIPGmAeMMd96OihlncYhQVzYtjFTl27TIb6VpYL87bx1TVdEhNs/WcqxIh3coraqMJGcHC6+nMXVYeSVlxrWLZ69h48zb61WISlrxTeqz8vDOrN650H+OX0FlY3EoaxRYSI5OVx8OYurw8grL3VB2ygiGwTW6kZ3VXdc2K4xo3q3ZsrSXCYs1s9kbeRS1ZaqW/ztNoZ0jWXumt3sPnTM6nCU4t4+rTmvTRRPzFjJb1v3Wx2OKkMTiSrXlV3jKTlh+LyWDS//66Dr+bDHYKvDUDXMbhNeGdaZqJBA7vw0gzydiK1W0USiytWqcQO6Ng9ncvrWWlUvvT7tQua37Wl1GMoC4cEBvHVNV/YcPs69EzMp0QFGaw1NJKpCw1Lj2bDnCOmb860O5XcRuTkk7NV68rqqY1wYT/2lAwvW7+Xl73S+99pCE4mq0GWdmhAS6Mf4RVusDuV3A954kn999arVYSgLDU9rxlWpcbw2N5s5K3daHY5CE4k6jfoBfgxOieWrZTu0TlrVKk8NTKZTXBj3T8pi7c5DVodT52kiUac1sntzCktOMKWW9nRXdVOQv523r02lfqAft3yUzv4C/aFjJU0k6rTaxoTQLSGc8Yu26OyJqlaJCQvi7Wu7svPAMe4en0mxjsRgGU0kqlLX9GjOpn0F/Lxhr9WhKHWKlGbhPDMomZ+y9/LM16utDqfO0kSiKtU/OYZGwQF8utD6RvcFV93G2HNHWB2GqkWuTI3nprMTef/nTToag0U0kahKBfrZubJrHN+u3sXOA9b2dM/p3JOFLVIsjUHVPo8OaMc5rSL55+crWFqLblevKzSRKJeM7N6MkhOGSUus/cUXvXE1bXdusDQGVfv42W28PrILTRoGcdvHS9m+/6jVIdUpmkiUS5pHBHNu60gmLN5iaaNmv3ef5eHZb1n2/qr2alg/gHeuS+V4UQk3fbCEQ8eKrA6pztBEolx2TY/m7Dx4jDmrdMY6VTu1iQ5hzDUprN99WO/kqkGaSJTL+raPJr5RPd77KcfqUJSq0Lmto/j3FcnMX7eHf81YWavGivNVmkiUy+w24YZeiaRvztehvFWtNiKtGbed34JPF21hnP7w8TiPJhIR6S8ia0UkW0RGl/N6oIhMcr6+SEQSSr32iHP9WhHp51wXLyI/iMhqEVkpIvd6Mn71Z1elxtEg0I/3f9Yvp6rdHu7XjkuSY3jm69XMWqFjcnmSxxKJiNiBN4BLgCRghIgklSl2M5BvjGkFvAQ859w2CRgOdAD6A2Oc+ysG/m6MaQ/0AO4qZ5/Kg0KC/LkyNY6Zy3aw62DN3wr8/TX38kqfG2v8fZX3sdmEl4Z1plNcQ+6blKlX0R7kySuSNCDbGLPRGFMITAQGlikzEPjQ+XgK0EdExLl+ojHmuDEmB8gG0owxO4wxGQDGmEPAaiDWg8egynFDrwRKjOHjXzfX+Htvbd+FZfH620G5JsjfzjvXpRLZIJCbPlhCzt4jVofkkzyZSGKB0p0OcvnzP/3fyxhjioEDQIQr2zqrwboAi6oxZuWC5hHB9G0fzaeLNnOsqKRG3zt+dSadtq6q0fdU3i0qJJCPbkrDANeOW2TJlbSv82QikXLWlb19oqIyp91WRBoAU4H7jDEHy31zkVtFJF1E0vfs2eNiyMpVN52dSH5BEdMza3Yq3j6fvMI9379fo++pvF+LqAZ8cGM38o8Ucv17izlwVPuYVCdPJpJcIL7U8zhge0VlRMQPCAPyTretiPjjSCKfGmOmVfTmxpixxphUY0xqVFSUm4eiyurRohFJTUIZu2CjTnmqvMJZcQ15+9pUNuw5zC0fptf41bQv82QiWQK0FpFEEQnA0Xg+o0yZGcD1zsdDgbnGcdP3DGC4866uRKA1sNjZfjIOWG2MedGDsatKiAh3XNCSjXuOMFtnqVNe4pzWkbx4VWeWbM5j1ATtsFhdPJZInG0edwOzcTSKTzbGrBSRp0TkL85i44AIEckG/gaMdm67EpgMrAJmAXcZY0qAs4Frgd4ikuVcBnjqGNTpDejYhISI+oyZl62dvpTXuLxTU564vAPfrtrFI9OW6zw71cDPkzs3xnwNfF1m3eOlHh8Drqxg22eAZ8qs+4ny20+UBew24fbzWzJ62nIWrN/LeW1qqgpRPwLKPdf3SiDvSCGvfL+eQH8bTw9MxlHhoapCe7YrtwxKiSU6NJAx87Jr5P1m3Tya5/vfXiPvpXzbfX1bc9v5Lfhk4Raenrlar6rdoIlEuSXQz84t57Zg4cY8MrZ4fh6InS3asbZJS4+/j/J9IsLo/u248ewE3vs5h+dmrdVkUkWaSJTbRqQ1o2F9f16f6/mrkhZZv9J9Q4bH30fVDSLC45clcXX3Zrw1fwMvf7fe6pC8kkfbSFTdEBzoxy3ntuD52WvJ2JJPSrNwj73XuZPfpv3hQpz3ZSjlNhHh6YHJFBaf4JXv12O3CaN6t9I2kzOgVySqWtzQK4FGwQG8OGedx99Lv9+qutlswrNDzmJwSiwvfruO52drNdeZ0ESiqkVwoB93XtCSn7L38uuGfVaHo9QZs9uEF4Z2YmT3ZoyZt4Env1ylycRFmkhUtbmmR3OiQwN58Vv9Nae8k80mPHNFMjedncgHv2zi0c+X68gNLtBEoqpNkL+du3u3ZsmmfOav0/HNlHcSER67rD13X9iKCYu38vfJWRRpD/jT0kSiqtWw1HjiG9Xj2W/WeOSX3Jd3Ps7Tl99T7ftVqjQR4YF+bXmwX1umZ23nrx+mc+R4sdVh1VqaSFS1CvCzMbp/e9bsPMTk9K2Vb3CG9sYmsjkyvvKCSlWDuy5sxbODO/JT9l6Gj13InkPHrQ6pVtJEoqrdgI4xdEsI539z1nLoWPUO19128TzOX7uwWvep1OkMT2vGO9d1JXv3YQa/+bNOjlUOTSSq2okI/7w0ib2HCxkzb0O17rvXFx9y7S9Tq3WfSlWmd7toJtzagyPHSxjy5i+kb8qzOqRaRROJ8ohO8Q0Z3CWWcQty2KS/4JQP6BzfkGl39CKsnj8j3lnIpCVbrA6p1tBEojzm4UvaEeBn45/TV+jtwMonJEQGM/3Os+nRIoKHpy7niRkr9Y4uNJEoD4oODeKh/m35KXsv07NqdkpepTwlrL4/79/QjVvOdfQ1uf69xeQfKbQ6LEtpIlEedXX35nSOb8jTM1eTV8e/bMp3+Nlt/OPSJP53ZSfSN+dz6asL6nS7iSYS5VF2m/B/gzty8GgRT3250u39Tbnv/3hsyEPVEJlS7hvSNY5pd/TC38/GsLELGTMvu07OuKiJRHlc+yah3HVhK6ZnbWfGb9vd2tfByBh2hTWupsiUcl9ybBgzR51D/+QY/jtrLTd+sKTO9TfRRKJqxKjeregc35B/fL6cbfuPVnk/yT/N4uLl86ovMKWqQUiQP6+P6MIzg5L5deM++r38I18t22F1WDVGE4mqEX52Gy8P60zJCcPfJmVRXMU7XdJmTWLokpnVHJ1S7hMRru7enK9GnUN8eD3uGp/B3eMz6kRDvCYSVWMSIoN5emAyi3LyePabNVaHo5RHtI4OYeodvXjg4jbMXrmTi176kemZ23z6FnhNJKpGDekaxw29Enj3pxw+z8y1OhylPMLPbuPu3q354q5ziG0YxH2TshjxzkLW7zpkdWgeoYlE1bh/XNqe7omNGD11OUvq8C2TyvclNQ1l2p1n88ygZFbvOMQlryzg6ZmrfK66SxOJqnH+dhtjrk4htmE9bvpgCau2H7Q6JKU8xm5ztJ3M/fv5DEmJ4/2fczjvvz/w+tz1FBT6xtD04sv1dielpqaa9PR0q8NQZeTmF3DlW79SVGKYcEt3WkeHVLrNfz76ke9X7eb7Z4fWQIRKVb91uw7x/Oy1fLtqF5ENArntvBYMT4snJMjf6tBOISJLjTGprpTVKxJlmbjw+nx8cxoicOXbv5KxJb/SbY6EhLM/OKwGolPKM9pEh/DOdalMvaMXbaIb8MzXq+n17Fyem7WG3YeOWR1elWgiUZZq1TiEqbc7RlS9+p1FfFHJmFxd5n7B5Zlzaig6pTyna/Nwxt/Sgy/uOpvzWkfx9vwN9Pq/udzxyVLmrd3tVXPFe7RqS0T6A68AduBdY8yzZV4PBD4CugL7gGHGmE3O1x4BbgZKgHuMMbNd2Wd5tGqr9ttz6Dh3fZrB4k15jEhrxiMD2hFazqX+xuRu5B8ppGvObxZEqZTnbNp7hE8XbWZqxjbyjhTSNCyIAR2bcEnHGLrEh2OzSY3GcyZVWx5LJCJiB9YBFwG5wBJghDFmVakydwJnGWNuF5HhwCBjzDARSQImAGlAU+A7oI1zs9PuszyaSLxDcckJXpizjrd/3EBkg0Ae6teWK7rE4m//48JZE4nydYXFJ/hu9S4+S9/KT9l7KSoxRIUEckGbKHq2jKBnywiahNXzeBxnkkj8PBhHGpBtjNnoDGoiMBAo/U9/IPCE8/EU4HUREef6icaY40COiGQ794cL+1Reys9uY/Ql7bgkOYbHvljBg1OW8dK36xjaNY6LkmJoG1N5Y7xS3i7Az8aAjk0Y0LEJB48V8cOa3cxeuZM5q3bx2VJH36vYhvVo3ySUpCYhtIkJIS68Pk3DgohsEFjjVy7g2UQSC2wt9TwX6F5RGWNMsYgcACKc6xeW2TbW+biyfSov1ym+IdPvPJt563bz7oIcXv8hm1fnZuNnEz45eAy7TZv2VN0QGuTPwM6xDOwcy4kThjU7D7Fw4z4yt+5n9Y6DzF2zi9JNKX42oUGQH/X97dQLsNM4JIgJt/bweJyeTCTlpcWy9WgVlalofXn/QcqtmxORW4FbAZo1a1ZxlKpWstmE3u2i6d0umr2Hj/Nz9l7W7DxExPRAggPtVoenVI2z2YSkpqEkNQ39fd2xohI27jnC9v1H2XHgKDsOHOPw8WIKCks4WlhC/YCa+a54MpHkAvGlnscBZccQP1kmV0T8gDAgr5JtK9snAMaYscBYcLSRVO0QVG0Q2SDQ8asM4LwfrQ5HqVojyN/+p+RiBU/WESwBWotIoogEAMOBGWXKzACudz4eCsw1jtb/GcBwEQkUkUSgNbDYxX0qX1a/vmNRStUaHrsicbZ53A3MxnGr7nvGmJUi8hSQboyZAYwDPnY2pufhSAw4y03G0YheDNxljCkBKG+fnjoGVQuNGeP4e+ed1sahlPqdDpGivMsFFzj+zptnZRRK+TwdIkUppVSN0USilFLKLZpIlFJKuUUTiVJKKbd4sh+JUtVPG9mVqnX0ikQppZRbNJEopZRyiyYSpZRSbtFEopRSyi2aSJRSSrlFE4lSSim3aCJRSinlFk0kSiml3KKJRCmllFvqxDDyIrIH2FzFzSOBvdUYjjeq6+egrh8/6DmAuncOmhtjolwpWCcSiTtEJN3VMfl9VV0/B3X9+EHPAeg5OB2t2lJKKeUWTSRKKaXcoomkcmOtDqAWqOvnoK4fP+g5AD0HFdI2EqWUUm7RKxKllFJu0URSARHpLyJrRSRbREZbHY8nicgmEVkuIlkiku5c10hEvhWR9c6/4c71IiKvOs/LMhFJsTb6qhGR90Rkt4isKLXujI9ZRK53ll8vItdbcSxVVcE5eEJEtjk/C1kiMqDUa484z8FaEelXar1XfldEJF5EfhCR1SKyUkTuda6vU5+DamGM0aXMAtiBDUALIAD4DUiyOi4PHu8mILLMuv8Co52PRwPPOR8PAL4BBOgBLLI6/ioe83lACrCiqscMNAI2Ov+GOx+HW31sbp6DJ4AHyimb5PweBAKJzu+H3Zu/K0ATIMX5OARY5zzOOvU5qI5Fr0jKlwZkG2M2GmMKgYnAQItjqmkDgQ+djz8Erii1/iPjsBBoKCJNrAjQHcaYH4G8MqvP9Jj7Ad8aY/KMMfnAt0B/z0dfPSo4BxUZCEw0xhw3xuQA2Ti+J177XTHG7DDGZDgfHwJWA7HUsc9BddBEUr5YYGup57nOdb7KAHNEZKmI3OpcF22M2QGOLxzQ2Lnel8/NmR6zr56Lu51VN++drNbBx8+BiCQAXYBF6OfgjGkiKZ+Us86Xb2872xiTAlwC3CUi552mbF07N1DxMfviuXgTaAl0BnYA/3Ou99lzICINgKnAfcaYg6crWs46nzgH7tJEUr5cIL7U8zhgu0WxeJwxZrvz727gcxzVFbtOVlk5/+52Fvflc3Omx+xz58IYs8sYU2KMOQG8g+OzAD56DkTEH0cS+dQYM825us5/Ds6UJpLyLQFai0iiiAQAw4EZFsfkESISLCIhJx8DFwMrcBzvybtPrge+cD6eAVznvIOlB3DgZDWADzjTY54NXCwi4c4qoIud67xWmfauQTg+C+A4B8NFJFBEEoHWwGK8+LsiIgKMA1YbY14s9VKd/xycMatb+2vrguMOjXU47kj5h9XxePA4W+C40+Y3YOXJYwUigO+B9c6/jZzrBXjDeV6WA6lWH0MVj3sCjqqbIhy/KG+uyjEDN+FoeM4GbrT6uKrhHHzsPMZlOP5xNilV/h/Oc7AWuKTUeq/8rgDn4KiCWgZkOZcBde1zUB2L9mxXSinlFq3aUkop5RZNJEoppdyiiUQppZRbNJEopZRyiyYSpZRSbtFEolQlROQX598EERlZzft+tLz3Usqb6O2/SrlIRC7AMTLuZWewjd0YU3Ka1w8bYxpUR3xKWUWvSJSqhIgcdj58FjjXOU/H/SJiF5HnRWSJc5DD25zlL3DOczEeR8c1RGS6c1DMlScHxhSRZ4F6zv19Wvq9nL2nnxeRFeKYK2ZYqX3PE5EpIrJGRD519tBGRJ4VkVXOWF6oyXOk6jY/qwNQyouMptQViTMhHDDGdBORQOBnEZnjLJsGJBvHkOsANxlj8kSkHrBERKYaY0aLyN3GmM7lvNdgHAMndgIindv86HytC9ABx3hOPwNni8gqHEOatDPGGBFpWO1Hr1QF9IpEqaq7GMfYS1k4hh+PwDEGFcDiUkkE4B4R+Q1YiGOAv9ac3jnABOMYQHEXMB/oVmrfucYxsGIWkAAcBI4B74rIYKDA7aNTykWaSJSqOgFGGWM6O5dEY8zJK5IjvxdytK30BXoaYzoBmUCQC/uuyPFSj0sAP2NMMY6roKk4JmKadUZHopQbNJEo5bpDOKZkPWk2cIdzKHJEpI1zBOWywoB8Y0yBiLTDMU3rSUUnty/jR2CYsx0mCse0uIsrCsw5p0aYMeZr4D4c1WJK1QhtI1HKdcuAYmcV1QfAKziqlTKcDd57+GNa1tJmAbeLyDIcI+cuLPXaWGCZiGQYY64utf5zoCeOUZkN8JAxZqczEZUnBPhCRIJwXM3cX7VDVOrM6e2/Siml3KJVW0oppdyiiUQppZRbNJEopZRyiyYSpZRSbtFEopRSyi2aSJRSSrlFE4lSSim3aCJRSinllv8H/JOXUQBYG0MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr(show_moms=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "141px",
    "width": "251px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}