{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## The 1cycle policy" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [], "source": [ "from fastai.gen_doc.nbdoc import *\n", "from fastai.vision import *\n", "from fastai.callbacks import *" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What is 1cycle?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This Callback allows us to easily train a network using Leslie Smith's 1cycle policy. To learn more about the 1cycle technique for training neural networks check out [Leslie Smith's paper](https://arxiv.org/pdf/1803.09820.pdf) and for a more graphical and intuitive explanation check out [Sylvain Gugger's post](https://sgugger.github.io/the-1cycle-policy.html).\n", "\n", "To use our 1cycle policy we will need an [optimum learning rate](https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html). We can find this learning rate by using a learning rate finder which can be called by using [`lr_finder`](/callbacks.lr_finder.html#callbacks.lr_finder). It will do a mock training by going over a large range of learning rates, then plot them against the losses. We will pick a value a bit before the minimum, where the loss still improves. Our graph would look something like this:\n", "\n", "![onecycle_finder](imgs/onecycle_finder.png)\n", "\n", "Here anything between `3x10^-2` and `10^-2` is a good idea.\n", "\n", "Next we will apply the 1cycle policy with the chosen learning rate as the maximum learning rate. The original 1cycle policy has three steps:\n", "\n", " 1. We progressively increase our learning rate from lr_max/div_factor to lr_max and at the same time we progressively decrease our momentum from mom_max to mom_min.\n", " 2. We do the exact opposite: we progressively decrease our learning rate from lr_max to lr_max/div_factor and at the same time we progressively increase our momentum from mom_min to mom_max.\n", " 3. We further decrease our learning rate from lr_max/div_factor to lr_max/(div_factor x 100) and we keep momentum steady at mom_max.\n", " \n", "This gives the following form:\n", "\n", "\"1cycle\n", "\n", "Unpublished work has shown even better results by using only two phases: the same phase 1, followed by a second phase where we do a cosine annealing from lr_max to 0. The momentum goes from mom_min to mom_max by following the symmetric cosine (see graph a bit below)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basic Training" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The one cycle policy allows to train very quickly, a phenomenon termed [_superconvergence_](https://arxiv.org/abs/1708.07120). To see this in practice, we will first train a CNN and see how our results compare when we use the [`OneCycleScheduler`](/callbacks.one_cycle.html#OneCycleScheduler) with [`fit_one_cycle`](/train.html#fit_one_cycle)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "path = untar_data(URLs.MNIST_SAMPLE)\n", "data = ImageDataBunch.from_folder(path)\n", "model = simple_cnn((3,16,16,2))\n", "learn = Learner(data, model, metrics=[accuracy])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First lets find the optimum learning rate for our comparison by doing an LR range test." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n" ] } ], "source": [ "learn.lr_find()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl4XNWV9/vvqtI8S5ZsNBkbWwI84Ek2g0kCBIIhgCF0DCYkIaTxvZ2mE+CGvqFzO28aLgkJ3SFASAjtNxDSDCGEwSQQmxBGGwfLeJYxnrE8ypIsa1YN6/2jjk1ZllyyraNTKq3P89Tjql276vwkS1q1zz5nH1FVjDHGmGPxeR3AGGNM/LNiYYwxJiYrFsYYY2KyYmGMMSYmKxbGGGNismJhjDEmJisWxhhjYrJiYYwxJiYrFsYYY2JK8jpAfyksLNRRo0Z5HcMYYwaV5cuX71fVolj9EqZYjBo1iurqaq9jGGPMoCIi2/vSz3ZDGWOMicmKhTHGmJisWBhjjInJioUxxpiYrFgYY4yJyYqFMcaYmKxYGGOMiSlhzrPoT6GwsrOxnU11zWze10pLZ5Bkv5Ds95Hs95Ga7CM92U96sp/UZB9+n49kn+D3CW1dIRrbumho7aK5I4gCAohAdloypXlplOSlU5ybTkFmCn6f9JghHFYOtAfY39JJQ2sXqpDkF3wigNLeFaY9EKI9ECLJJ6Ql+0hL9pOW7CfF7yM1yUeKc0vy+Ujx+0g6/DUIIj1v1xhjejLki0VTW4AH39jI3uYO6g52sq+5g91NHXQGwwOy/Zy0JPIzU0hP9tMVCtMZCNMRCNHUHiAYdu/66H6fkJbkoyArhYLMVAozU8hJTyYz1U9mShJZqUmckptGWX4GZfnp5GYk0xEI0RkI0xkMISKHi5KI0NIZ5GB7gIMdAQQhLyOZ3PRk8jNTyEzxH7M4hcNKWJVgWOkMhGkLBGnrCuEXoTA7NebrjTHuG/LFwueD3y/7hOE5aRRlpzKxLI8vjE9jTFEmY4qyGDs8i5y0ZALhMMGQEgiF6QyGae8K0REM0REIEwqHCYSUUFjJSPGTn5FCfkYK2WlJ+HyCqhJWaO4IUNvYzq4DkVtDW4Cmti4OtAdo7QyRmuwjLSkyWslLT6YwK5Wi7FSGZaaAQDgMwXAYETk8sklL9hEMKx2BSJaOYIiuYPjTWyhMMBSmK6QEQ2GCYaUrGCYYDtPeFaahtZP61i52NXWwYW8zbV0hWjuD/Vos/T4hKzWJnPQkUpP8tHeFaO0K0tYZoisUezvpyX4KsyPf05y0ZHLSk8jLSKEkN43S/HRKnFFaapL/8GgqPdlPapIPXy8jN2PM8RFV9z69DqSqqiq15T76T2cwxO4DHew80E5tYxsH24OHd3WlJvtRVTqdghRWJTstyflDnowqHGjr4kBbgMa2yO645o4ABzuCdAZDZKQkkZniJz0liZQkHz4Bvwg+n1MEU/xkpPgJhpT9LZ3UNXeyv6WTA+0BDrYHaGoP0NgWoKG1K+bXkZbsIzMliWFZKQzLTGVYVqToHBr55GWkUJqXzshhGZySk9brbkFjEpWILFfVqlj9hvzIwvQsNcnPqMJMRhVmeh2lV+1dIXY1tbOzsZ2m9sDhkVRnIETHodFfIERzZ5D6lk7qW7pYu7PpcNHpvpcv2S+U5qVz6rBMTh2WwciCDHLTk0l1RikFmSlMG5lvoxUzJFmxMINWeoqfMUVZjCnKOu7XhsNKc2eQxtYuahvb2dHYxicNbXxS38b2hlY+/KSR5o7gUa8bXZjJN2aO4tqpZWSm2q+PGTpsN5QxPVBVmtoDNHdE5m86gyE27m3h8cVbWVXbRHZaEl+cWMyM0QVMH1VAWX66TcKbQamvu6FcLRYiMgt4EPAD81X1vh76zAF+CCiwSlVvcNp/AnzR6XaPqv7+WNuyYmEGgqry4ScHeHzxVt7+uO7w6KM4N41ZE07hS1PKmFCaY4XDDBqez1mIiB94BLgEqAWWicgCVa2J6lMB3AXMVNVGERnutH8RmApMBlKBt0XkNVU96FZeY/pCRJh2aj7TTs0nFFY27GmmensDizft56mln/D44m2MHZ7Fl6eVMffskeSkJXsd2Zh+4eZO1xnAJlXdAiAizwKzgZqoPrcAj6hqI4Cq7nPaxwFvq2oQCIrIKmAW8JyLeY05Ln6fMK4kh3ElOXzt3FE0tQX485rdvPBhLT9+7SN+8eYmbjpvFN+YOZqCzBSv4xpzUtxc7qMU2BH1uNZpi1YJVIrIYhFZ6uy2AlgFXCYiGSJSCFwIlLuY1ZiTlpuRzA1nj+T5fzqPV249n5ljCnn4b5s4/yd/4xd/20iwD+eUGBOv3BxZ9LTTtvsESRJQAVwAlAHvisgEVV0kItOBJUAd8D5w1KEpIjIPmAcwcuTI/ktuzEmaWJbLo1+dxsa9zTzw14/5z0Uf8+aGOn5+3WTKCzK8jmfMcXNzZFHLkaOBMmBXD31eVtWAqm4FNhApHqjqvao6WVUvIVJ4NnbfgKo+pqpVqlpVVBTzeuPGDLiKEdn88ivTePD6yXy8p5nLHnyXPy6vJVGOQjRDh5vFYhlQISKjRSQFuB5Y0K3PS0R2MeHsbqoEtoiIX0SGOe1nAWcBi1zMaoyrZk8u5bXbPsO44hz+nz+s4tanV3CgLfYZ6MbEC9eKhTM5fSuwEFgPPKeq60TkbhG5yum2EKgXkRrgTeBOVa0HkonskqoBHgNudN7PmEGrLD+DZ+adw52Xns7CdXu49Ofv8N7G/V7HMqZP7KQ8YzywdmcT33l2BZvrWrmuqpzrZ5QzuTzPzs8wAy4uTsobSFYszGDT3hXiJ3/5iGc++ITOYJjTCjO5ekopN80cZednmAFjxcKYQeJgR4C/rNnDCytqWbqlgc9UFPLbb8ywBQvNgOhrsbDLqhrjsZy0ZOZML+fZeedy7zUTeHfjfv773S1exzLmCFYsjIkjN8wYyWUTTuH+hRtYueOA13GMOcyKhTFxRES470tnMSInjW8/s4LmjoDXkYwBrFgYE3dyM5J58PrJ7DzQzvdfXGsn8Jm4YMXCmDhUNaqA2y+uYMGqXdz32kdWMIzn7FJfxsSpf75wLHsPdvLrd7bg9wl3Xnq6nYdhPGPFwpg4JSL8x1XjCanyy7c2k+QT7vjC6V7HMkOUFQtj4pjPJ/z/sycQCikP/W0TbV0h7px1OqlJfq+jmSHGioUxcc7nE378pYmkJPmY/95W3tlYx/3/MIlJ5XleRzNDiE1wGzMI+HzCPVdP4PGbpnOwPcg1v1zMj19bT2cw5HU0M0RYsTBmELnwjOEsuuOzzKkq59dvb+HmJ5bR0mkLMhv3WbEwZpDJSUvmvmvP4mdzJrF0SwPXP/Y++1s6vY5lEpwVC2MGqS9NLWP+16rYtK+Ff/jVEnY0tHkdySQwKxbGDGIXnjGcp/7xHBrbAlz36/dp67JdUsYdViyMGeSmnZrP/K9Xsaupg/nvbvU6jklQrhYLEZklIhtEZJOIfK+XPnNEpEZE1onI01HtP3Xa1ovIQ2KnrhrTq+mjCrh0/Ah+/fZmm78wrnCtWIiIH3gEuAwYB8wVkXHd+lQAdwEzVXU8cJvTfh4wEzgLmABMBz7nVlZjEsG/zjqDjmCYh9/Y6HUUk4DcHFnMADap6hZV7QKeBWZ363ML8IiqNgKo6j6nXYE0IAVIBZKBvS5mNWbQG1OUxXXTy3nq75+wbX+r13FMgnGzWJQCO6Ie1zpt0SqBShFZLCJLRWQWgKq+D7wJ7HZuC1V1vYtZjUkIt32+gmS/j/sXbfA6ikkwbhaLnuYYuq+znARUABcAc4H5IpInImOBM4EyIgXmIhH57FEbEJknItUiUl1XV9ev4Y0ZjIbnpHHLZ0/jz6t325X2TL9ys1jUAuVRj8uAXT30eVlVA6q6FdhApHhcAyxV1RZVbQFeA87pvgFVfUxVq1S1qqioyJUvwpjBZt5nT6MgM4VH39rsdRSTQNwsFsuAChEZLSIpwPXAgm59XgIuBBCRQiK7pbYAnwCfE5EkEUkmMrltu6GM6YOs1CSumVLKGx/tpbG1y+s4JkG4VixUNQjcCiwk8of+OVVdJyJ3i8hVTreFQL2I1BCZo7hTVeuB54HNwBpgFbBKVV9xK6sxiebaqWUEQsorq7sP5o05MZIol2usqqrS6upqr2MYEzcue/BdUvzCy7ee73UUE8dEZLmqVsXqZ2dwG5Ogrp1ayqraJjbubfY6ikkAViyMSVCzJ5fi9wnPf1jrdRSTAKxYGJOgirJTuaCyiJdW7CQUTozdzcY7ViyMSWDXTitj78FO3tu03+soZpCzYmFMAvv8mcPJTU/mj8ttV5Q5OVYsjElgqUl+rpxUzMJ1ezjYEfA6jhnErFgYk+C+NLWMzmCYRetsLU5z4qxYGJPgppTnUZafzoJVdoKeOXFWLIxJcCLClZNKWLxpP/V2YSRzgqxYGDMEXHlWCaGw8traPV5HMYOUFQtjhoAzi7MZU5TJK7YrypwgKxbGDAGHdkV9sK2BPU0dXscxg5AVC2OGiCvOKkEV/rxmt9dRzCBkxcKYIWLs8CzGFefYrihzQqxYGDOEXDW5hJU7DrCjoc3rKGaQsWJhzBDyxYnFAHZRJHPcrFgYM4SUF2QwdWQeL3y4k7CtRGuOgxULY4aYr507ik37Wli4zs65MH3narEQkVkiskFENonI93rpM0dEakRknYg87bRdKCIro24dInK1m1mNGSqunFTCaYWZPPjGRhtdmD5zrViIiB94BLgMGAfMFZFx3fpUAHcBM1V1PHAbgKq+qaqTVXUycBHQBixyK6sxQ4nfJ9x60Vg+2tPMohpbXND0jZsjixnAJlXdoqpdwLPA7G59bgEeUdVGAFXd18P7/APwmqra4RvG9JOrJpUwalgGD72xEVUbXZjY3CwWpcCOqMe1Tlu0SqBSRBaLyFIRmdXD+1wPPNPTBkRknohUi0h1XV1dv4Q2ZihI8vu49aIKanYf5K/re/qMZsyR3CwW0kNb948wSUAFcAEwF5gvInmH30CkGJgILOxpA6r6mKpWqWpVUVFRv4Q2Zqi4enIJpw7L4ME3PrbRhYnJzWJRC5RHPS4Duh/cXQu8rKoBVd0KbCBSPA6ZA7yoqnaJL2P6WZLfxz9fOJa1Ow/y5gYbXZhjc7NYLAMqRGS0iKQQ2Z20oFufl4ALAUSkkMhuqS1Rz8+ll11QxpiTd82UUoqyU3lumV2j2xyba8VCVYPArUR2Ia0HnlPVdSJyt4hc5XRbCNSLSA3wJnCnqtYDiMgoIiOTt93KaMxQl+z3ccVZxfxtwz6a2m0Ab3rn6nkWqvqqqlaq6hhVvddp+4GqLnDuq6reoarjVHWiqj4b9dptqlqqqmE3Mxoz1M2eXEpXMGwn6ZljsjO4jRniJpXlcuqwDBastPWiTO+sWBgzxIkIsyeVsGTzfvYdtAsjmZ5ZsTDGcNXkEsIKf1ptF0YyPbNiYYxh7PBsxpfksMAujGR6YcXCGANElgBZueMA2+tbvY5i4pAVC2MMEFmNFrCJbtMjKxbGGABK8tKZMbqAl1butOU/zFGsWBhjDvvixGI217VS29judRQTZ6xYGGMOmzIyso7n2p1NHicx8caKhTHmsMoR2fh9wtpdVizMkaxYGGMOS0v2UzE8i3W7DnodxcQZKxbGmCNMKM1l7c4mm+Q2R7BiYYw5woSSHPa3dLGvudPrKCaOWLEwxhxhQmkuYJPc5khWLIwxRzizOAcRWLvT5i3Mp6xYGGOOkJmaxGmFmXZElDmCFQtjzFEmlOayznZDmSiuFgsRmSUiG0Rkk4h8r5c+c0SkRkTWicjTUe0jRWSRiKx3nh/lZlZjzKfGl+Swq6mDhtYur6OYOOFasRARP/AIcBkwDpgrIuO69akA7gJmqup44Laop58E7lfVM4EZwD63shpjjjShJDLJvc52RRmHmyOLGcAmVd2iql3As8Dsbn1uAR5R1UYAVd0H4BSVJFV93WlvUdU2F7MaY6KMLzl0RJRNcpsIN4tFKbAj6nGt0xatEqgUkcUislREZkW1HxCRF0RkhYjc74xUjDEDIDcjmfKCdJvkNoe5WSykh7bup4QmARXABcBcYL6I5DntnwG+C0wHTgNuOmoDIvNEpFpEquvq6vovuTGGCSU2yW0+1adiISJjRCTVuX+BiHzb+aN+LLVAedTjMqD7VVVqgZdVNaCqW4ENRIpHLbDC2YUVBF4CpnbfgKo+pqpVqlpVVFTUly/FGNNHE0pz2VbfxsGOgNdRTBzo68jij0BIRMYC/xsYDTx97JewDKgQkdEikgJcDyzo1ucl4EIAESkksvtpi/PafBE5VAEuAmr6mNUY0w/Gl+QAUGOLChr6XizCzif8a4Cfq+rtQPGxXuD0vxVYCKwHnlPVdSJyt4hc5XRbCNSLSA3wJnCnqtaraojILqg3RGQNkV1a/328X5wx5sSNP3xElBULE5kb6IuAiMwFvg5c6bQlx3qRqr4KvNqt7QdR9xW4w7l1f+3rwFl9zGeM6WdF2amMyEm1NaIM0PeRxTeAc4F7VXWriIwG/se9WMaYeDDt1Hz+vqXelis3fSsWqlqjqt9W1WdEJB/IVtX7XM5mjPHYzLGF7GrqYOv+Vq+jGI/19Wiot0QkR0QKgFXA4yLyM3ejGWO8dv7YQgAWb9rvcRLTm10H2lnxSaPr2+nrbqhcVT0IfAl4XFWnARe7F8sYEw9GFmRQlp/Oe1Ys4k4orDyxeCuX/Oxt7nx+NeGwu7sK+zrBnSQixcAc4Psu5jHGxBER4fyxhfx5zW5CYcXv6+lcWzPQPtpzkLteWMOKTw7wmYpCfnTNRHwu/9/0dWRxN5HDXDer6jIROQ3Y6F4sY0y8mDm2kOaOIGvsqCjPdQZD/GzRBq546D2217fx8+sm8+TNMygvyHB9230aWajqH4A/RD3eAlzrVihjTPw4b8wwIDJvMbk81sINxi0rPmnkX59fzcZ9LVwzpZR/v2IcBZkpA7b9vk5wl4nIiyKyT0T2isgfRaTM7XDGGO8Ny0plXHEO7220eQsvdARC3PvnGq791RJaOoM8ftN0Hrhu8oAWCuj7nMXjRJb3+LLz+Ean7RI3Qhlj4sv5FYU8sXgb7V0h0lNsAeiBsnx7I3f+YRVb9rdyw9kjueuyM8hOi3k+tCv6OmdRpKqPq2rQuT0B2Mp9xgwRM8cW0hUK88G2Bq+jDAkdgRA/fnU9X350CZ3BME/949n86JqJnhUK6PvIYr+I3Ag84zyeC9S7E8kYE2+mj8onxe9j8ab9fK7SPie6ae3OJm7//Uo27mth7oyR/Nvl3o0movW1WNwM/AJ4gMg1KZYQWQLEGDMEZKQkMfXUPJu3cFEwFOZXb23mwTc2Miwrhd/ePCOuCnNfj4b6BLgquk1EbgN+7kYoY0z8OX9sIf+56GPqWzoZlpXqdZyEsm1/K7c/t5IVnxzgykkl3DN7PHkZAzuBHcvJXCnvqJVijTGJ6/yKyKfcNz7a53GSxKGqPLdsB5c/9C6b97Xw0NwpPDx3StwVCuj7bqie2Kmcxgwhk8pyGTs8i6eWbmdOVXnsF5hjamzt4t9eXMNra/dwzmkF/GzOZEry0r2O1auTGVnYmsXGDCEiwlfPOZVVtU2s2nHA6ziDWl1zJ1f/cjF/Xb+Xuy47g6f/8Zy4LhQQo1iISLOIHOzh1gyUDFBGY0yc+NLUUjJT/Dz5/navowxarZ1Bbn5iGXsPdvDMLefwf31ujOvrOvWHYxYLVc1W1ZwebtmqGnMXlojMEpENIrJJRL7XS585IlIjIutE5Omo9pCIrHRu3a/dbYzxQHZaMtdMLeWV1btoaO3yOs6gEwiF+aenPqRm90EeuWEqVaMKvI7UZyezG+qYRMQPPAJcBowD5orIuG59KoC7gJmqOh64LerpdlWd7NyOOBLLGOOdr54ziq5gmOeqd3gdZVBRVf7fP67mnY/ruPfqCXz+zBFeRzourhULYAawSVW3qGoX8Cwwu1ufW4BHVLURQFXtMAtj4tzpp2QzY3QB/7N0OyGXr6GQKFo6g9z++5W88OFObr+4kutnjPQ60nFzs1iUAtEfPWqdtmiVQKWILBaRpSIyK+q5NBGpdtqvdjGnMeY4fe3cU6ltbOetDfb5Lpa1O5u48uH3WLBqF3dcUsm3Pz/W60gn5GQOnY2lpxmb7h9DkoAK4AKgDHhXRCao6gFgpKrucq6d8TcRWaOqm4/YgMg8YB7AyJGDr1IbM1hdOv4Uhmen8uT72wfd7pSB9Lv3t3HPn9aTn5nM07ecwzmnDfM60glzc2RRC0QfjF0G7Oqhz8uqGlDVrcAGIsUDVd3l/LsFeAuY0n0DqvqYqlapalVRUfycFm9Mokv2+7huejnvbKxjX3OH13HiUkNrFz9YsI4Zowt47TufHdSFAtwtFsuAChEZLSIpwPVA96OaXgIuBBCRQiK7pbaISL6IpEa1zwRqXMxqjDlOl08sRhVer9nrdZS4VNfciSrMnTFywK894QbXioWqBoFbiVyOdT3wnKquE5G7ReTQ0U0LgXoRqQHeBO5U1XrgTKBaRFY57fepqhULY+LIGadkM2pYBn9Zu8frKHGpvrUTgPxM71eM7Q9uzlmgqq8Cr3Zr+0HUfSWyxtQd3fosASa6mc0Yc3JEhFkTipn/7hYOtHXF5XpGXmpsDQAkxKgC3N0NZYxJcJdNOIVgWG1XVA8a2iInLRYkSBG1YmGMOWFnleVSmpduu6J60Oic4Z5vIwtjzFAnIlw6/hTe3bif5o6A13HiSkNrF9lpSST7E+PPbGJ8FcYYz1w28RS6QmH+Zte5OEJDa1fCzFeAFQtjzEmaNjKfouxU2xXVTWNbF/kJMl8BViyMMSfJ5xMuHT+CtzbU0d4V8jpO3Gho7WKYjSyMMeZTl00opj0Q4u2P67yOEjcaW7sSZnIbrFgYY/rB2aMLKMhM4ZVV3Vf0GZpUlXqbszDGmCMl+X3MnlzCopo9dlEkoD0QojMYtjkLY4zp7rrp5QRCyksrdnodxXOHCmZBgiz1AVYsjDH95IxTcphUlstz1TuIrOQzdH261Eeqx0n6jxULY0y/mTO9nI/2NLO6tsnrKJ46vNSHjSyMMeZoV04qIS3Zx++H+PW5Gw6tOGtzFsYYc7SctGQun1DMKyt3DelzLhoSbMVZsGJhjOlnc6aX09wZ5NU1u72O4pnG1i78PiEnzXZDGWNMj84eXcCoYRk8N4R3RTW0dZGfkYzPJ15H6TdWLIwx/UpE+HJVOX/f2sCOhjav43iioSWx1oUCKxbGGBdcfOYIAJZta/A4iTca2hJrqQ9wuViIyCwR2SAim0Tke730mSMiNSKyTkSe7vZcjojsFJFfuJnTGNO/xg7PIjPFz6odB7yO4onG1q6EuULeIa5dg1tE/MAjwCVALbBMRBaoak1UnwrgLmCmqjaKyPBub3MP8LZbGY0x7vD7hIlluawcqsWirYuCrMQqFm6OLGYAm1R1i6p2Ac8Cs7v1uQV4RFUbAVT18NVTRGQaMAJY5GJGY4xLJpXnUbP7IJ3BoXUIbTisNLYFEm5k4WaxKAWiD4eoddqiVQKVIrJYRJaKyCwAEfEB/wXceawNiMg8EakWkeq6Olsa2Zh4MqU8j0BIqdl10OsoA+pgR4BQWG3O4jj0dMxY9wVjkoAK4AJgLjBfRPKAbwGvquoxj71T1cdUtUpVq4qKivohsjGmv0wqzwMYcruiEnERQXBxzoLISKI86nEZ0H2x+1pgqaoGgK0isoFI8TgX+IyIfAvIAlJEpEVVe5wkN8bEn+LcdEbkpA65Se5GZ10oO3S275YBFSIyWkRSgOuBBd36vARcCCAihUR2S21R1a+o6khVHQV8F3jSCoUxg8+ksrwhOLKILPUxLIFWnAUXi4WqBoFbgYXAeuA5VV0nIneLyFVOt4VAvYjUAG8Cd6pqvVuZjDEDa/LIPLbVt3GgbehcEKnR2Q2Vb7uh+k5VXwVe7db2g6j7Ctzh3Hp7jyeAJ9xJaIxx0+SyT+ctLji9+5Hxian+8JyF7YYyxpg+mViWiwis2jF0rm/R2NZFapKP9GS/11H6lRULY4xrstOSGVuUxcodjV5HGTANrV0My0xBJHEWEQQrFsYYl00uj0xyD5VLrTa2Jt66UGDFwhjjsknleTS2BdjR0O51lAFR39qVcPMVYMXCGOOyyc7JeSuGyK6oxrbEW54crFgYY1x2+inZpCb5hswkd4ONLIwx5vgl+31MLM0dEpPcgVCY5o6gFQtjjDkR40py+HhvS8JPch9e6sOKhTHGHL+KEdm0dAbZ1dThdRRXHV5E0OYsjDHm+J0+IhuAj/c2e5zEXQ0JutQHWLEwxgyAyhFZAHy8J7GLRWOCLiIIViyMMQMgLyOFouxUPt7b4nUUVzW02cjCGGNOSuWILDbuS+yRRUNLYl7LAqxYGGMGSOWIbDbubSEcTtwjohrbushOSyLZn3h/WhPvKzLGxKXKEdm0B0LUNibmsh/tXSG27G9NyHMswOXrWRhjzCGHJ7n3NjNyWIbHafpHOKy8vn4vf1q9mzfW76WtK8Sl40d4HcsVViyMMQNi7HDn8Nl9zVw8bvD/Qe0IhLj99yt5be0eCjJTuHpKKVdMLObs04Z5Hc0VrhYLEZkFPAj4gfmqel8PfeYAPwQUWKWqN4jIqcALzuuSgYdV9VE3sxpj3JWbnkxxbhobE+CIqKa2ALc8Wc0H2xr4t8vP4OaZo0lKwHmKaK4VCxHxA48AlwC1wDIRWaCqNVF9KoC7gJmq2igih667uBs4T1U7RSQLWOu8dpdbeY0x7qsYkc2GQX6uxc4D7dz0mw/YXt/GQ3OncNWkEq8jDQg3S+EMYJOqblHVLuBZYHa3PrcAj6hqI4Cq7nP+7VLVTqdPqss5jTEDpHJ4FpvrWggN0iOidh1o58u/WsKepg6euHn6kCkU4O4f4VJgR9TjWqctWiVQKSKLRWSps9sKABEpF5HVznv8xEYVxgx+lSOy6QyG+aShzesox62pLcDXf/MBzR1Bnpl3DueNKfQ60oBys1j0dAHa7h8nkoAK4AJgLjBfRPIAVHWHqp7Qod+IAAAPqklEQVQFjAW+LiJHzYiJyDwRqRaR6rq6un4Nb4zpfxVRR0QNJh2BELc8Wc22+lZ+/dVpTCjN9TrSgHOzWNQC5VGPy4Duo4Na4GVVDajqVmADkeJxmDOiWAd8pvsGVPUxVa1S1aqioqJ+DW+M6X8VzoKCGwdRsQiFldueXckH2xr42ZzJnDd2aI0oDnGzWCwDKkRktIikANcDC7r1eQm4EEBEConsltoiImUiku605wMziRQSY8wglpWaRGleOhsG0RFR9/yphr+s28O/XzGOK4fQHEV3rhULVQ0CtwILgfXAc6q6TkTuFpGrnG4LgXoRqQHeBO5U1XrgTODvIrIKeBv4T1Vd41ZWY8zAqRyRNWhGFr9dso0nlmzjm+eP5pvnj/Y6jqdcPc9CVV8FXu3W9oOo+wrc4dyi+7wOnOVmNmOMNypHZLN4Uz3BUDiuz01486N9/Mcr67j4zBH82+Vneh3Hc/H7P2WMSUiVI7LpCoXZVh+/R0St332QW5/+kDOLc3ho7mT8vp6O1xlarFgYYwZUpTPJ/eZH+wiGwh6nOdq+5g6++cQystOS+d9fn05Giq2KBFYsjDEDrGJEFuUF6dz76nrO/tEbfP/FNSzf3uB1LCByiOy8J5fT2BZg/terOCU3zetIccOKhTFmQKUl+3n99s/xq69M5Zwxw3jhw51c+6v3eeHDWk9zqSp3vbCGlTsO8MB1k4bkuRTHYuMrY8yAS0v2c9nEYi6bWExrZ5B//G01dz6/mryMZC46w5sVaR99ewsvrtjJd79QyawJxZ5kiGc2sjDGeCozNYnHvjaNM4uz+dZTH1K9beB3Sb1es5efLvyIKyeV8M8Xjh3w7Q8GViyMMZ7LTkvmiW/MoCQ3nZufWMZHew4O2LY37m3mtmdXMLE0l/v/4SxE7MinnlixMMbEhcKsVJ785gzSU/zc+vQKAgNwpFRTe4B5v1tOekoSj321irRkv+vbHKysWBhj4kZZfgb3zJ7Apn0tPLV0u6vbCoWV7zy7gtrGNh69caod+RSDFQtjTFy5ZNwIzh9byAN/3Uhja5dr2/mvRRt4a0Md/+vK8VSNKnBtO4nCioUxJq6ICP9+xTiaOwI88NeP+/y6T+rbmPdkNfUtnTH7/nn1bn751mbmzijnK2ePPJm4Q4YVC2NM3Dn9lGxuPOdU/mfp9j5fhvXllTtZVLOXx97Zcsx+63cf5Lt/WMXUkXn88KrxNqHdR1YsjDFx6faLK8lOS+buP60jsubosS3evB+AJ9/f3uvo4kBbF/N+V012WhKP3jiN1CSb0O4rKxbGmLiUn5nC7RdXsHhTPe9t2n/Mvh2BEB9uP8BFZwynIxhi/ntbj+oTDIX5l2dWsLepk0e/Oo3hOTahfTysWBhj4tYNZ59KXkYyzy8/9lIg1dsa6QqF+eo5p3LFWSU8uWTbUZPjP124gXc37ufu2eOZOjLfzdgJyYqFMSZupST5+OLEYhau20NLZ7DXfks27yfJJ0wfXcC3LxpLWyDE/PcicxedwRB3vbCax97Zwo3njOT6GTahfSKsWBhj4tqXppbSEQizcO2eXvss2VzPpPI8slKTqBiRzeUTi/ntku3U7DrInEff55kPdvCtC8bwH1dNGMDkicXVYiEis0Rkg4hsEpHv9dJnjojUiMg6EXnaaZssIu87batF5Do3cxpj4tfUkfmUF6Tz4oqdPT5/sCPA6toDnDdm2OG2b19UQUtnkCsefpfNda08euM0/nXWGXYRo5Pg2qqzIuIHHgEuAWqBZSKyQFVrovpUAHcBM1W1UUSGO0+1AV9T1Y0iUgIsF5GFqnrArbzGmPgkIlwzuZSH39zE3oMdjOg2Mf3BlgbCCueNKTzcdvop2VxXVc6anU08fMMUxhRlDXTshOPmyGIGsElVt6hqF/AsMLtbn1uAR1S1EUBV9zn/fqyqG537u4B9QJGLWY0xcezqKaWowoKVu456bvHm/aQm+ZgyMu+I9vuuncir3/mMFYp+4maxKAV2RD2uddqiVQKVIrJYRJaKyKzubyIiM4AUYLNrSY0xce20oiwmlefxQg+7ot7fXM/0UQVHLQJoJ9v1LzeLRU//U93PrEkCKoALgLnAfBE5/PFARIqB3wHfUNWjlqAUkXkiUi0i1XV1df0W3BgTf66ZXML63QePWL58f0snH+1p5tyo+QrjDjeLRS1QHvW4DOg+hqwFXlbVgKpuBTYQKR6ISA7wZ+D/U9WlPW1AVR9T1SpVrSoqsr1UxiSyKyeV4PfJERPd72+uB2Dm2MLeXmb6iZuXVV0GVIjIaGAncD1wQ7c+LxEZUTwhIoVEdkttEZEU4EXgSVX9g4sZjTGDxLCsVD5XWcSTS7ZT39LFZRNO4a0NdWSnJjGhJMfreAnPtWKhqkERuRVYCPiB36jqOhG5G6hW1QXOc18QkRogBNypqvUiciPwWWCYiNzkvOVNqrrSrbzGmPj3wyvH88BfP2bhuj2Hz+q++MwRJPntlDG3SV8W6BoMqqqqtLq62usYxpgB0BUMs2Tzft7aUMeVk4qZdqpdj+JEichyVa2K1c/N3VDGGOOKlCQfF5w+nAtOHx67s+kXNnYzxhgTkxULY4wxMVmxMMYYE5MVC2OMMTFZsTDGGBOTFQtjjDExWbEwxhgTkxULY4wxMSXMGdwiUgccAJp6eDq3W/uxHh+631NbIbD/OKN131Zfnz+RzNH3TybzsXId6/lYbfGYuad2+/mIbaj8fAzGzD21H+txharmxkyiqglzAx7rS/uxHh+630tbdX9lciNzT/lPJPOJ5o7VFo+Z7efDfj4SLfPJ/Hwc65Zou6Fe6WP7sR6/coy2/swU6/kTyRx9/2Qy9+X1PT0fqy0eM/fUbj8fsQ2Vn4/BmLmn9r7+fPQqYXZDDQQRqdY+LLgVTyzzwBmMuS3zwBiMmbtLtJGF2x7zOsAJsMwDZzDmtswDYzBmPoKNLIwxxsRkIwtjjDExDcliISK/EZF9IrL2BF47TUTWiMgmEXlIRCTquX8RkQ0isk5Eftq/qd3JLSI/FJGdIrLSuV0e75mjnv+uiKhzSd5+49L3+R4RWe18jxeJSEl/ZnYx9/0i8pGT/UURyRsEmb/s/A6GRaTf5glOJmsv7/d1Edno3L4e1X7Mn3vPnMjhXIP9RuSSrVOBtSfw2g+AcwEBXgMuc9ovBP4KpDqPhw+S3D8EvjuYvtfOc+VELsu7HSiM98xATlSfbwOPDobvNfAFIMm5/xPgJ4Mg85nA6cBbQJXXWZ0co7q1FQBbnH/znfv5x/q6vL4NyZGFqr4DNES3icgYEfmLiCwXkXdF5IzurxORYiK/9O9r5H/1SeBq5+l/Au5T1U5nG/sGSW5XuZj5AeBfgX6fdHMjs6oejOqaOYhyL1LVoNN1KVA2CDKvV9UN/ZnzZLL24lLgdVVtUNVG4HVglpe/q7EMyWLRi8eAf1HVacB3gV/20KcUqI16XOu0AVQCnxGRv4vI2yIy3dW0nzrZ3AC3OrsZfiMi+e5FPeykMovIVcBOVV3ldtAoJ/19FpF7RWQH8BXgBy5mjdYfPx+H3Ezkk67b+jOz2/qStSelwI6ox4fyx8vXdRS7BjcgIlnAecAfonYPpvbUtYe2Q58Qk4gMJ88BpgPPichpzqcDV/RT7l8B9ziP7wH+i8gfBVecbGYRyQC+T2T3yIDop+8zqvp94PsichdwK/C/+jnqkWH6KbfzXt8HgsBT/ZnxqCD9mNltx8oqIt8AvuO0jQVeFZEuYKuqXkPv+T3/unpjxSLCBxxQ1cnRjSLiB5Y7DxcQ+cMaPQwvA3Y592uBF5zi8IGIhImsB1MXz7lVdW/U6/4b+JOLeeHkM48BRgOrnF/QMuBDEZmhqnviNHN3TwN/xuViQT/ldiZfrwA+7+aHH0d/f6/d1GNWAFV9HHgcQETeAm5S1W1RXWqBC6IelxGZ26jF+6+rZ15Pmnh1A0YRNVEFLAG+7NwXYFIvr1tGZPRwaPLpcqf9/wbudu5XEhliyiDIXRzV53bg2XjP3K3PNvp5gtul73NFVJ9/AZ4fJD/Xs4AaoMiNvG7+fNDPE9wnmpXeJ7i3Etkbke/cL+jrz70XN88DePJFwzPAbiBApJJ/k8in1b8Aq5xfjh/08toqYC2wGfgFn57YmAL8j/Pch8BFgyT374A1wGoin9iK4z1ztz7b6P+jodz4Pv/RaV9NZC2e0kHy87GJyAeflc6tX4/icinzNc57dQJ7gYVeZqWHYuG03+x8fzcB3zien3svbnYGtzHGmJjsaChjjDExWbEwxhgTkxULY4wxMVmxMMYYE5MVC2OMMTFZsTAJTURaBnh780VkXD+9V0giq9SuFZFXYq34KiJ5IvKt/ti2Md3ZobMmoYlIi6pm9eP7JemnC+u5Kjq7iPwW+FhV7z1G/1HAn1R1wkDkM0OLjSzMkCMiRSLyRxFZ5txmOu0zRGSJiKxw/j3dab9JRP4gIq8Ai0TkAhF5S0Sel8i1Hp46dM0Bp73Kud/iLB64SkSWisgIp32M83iZiNzdx9HP+3y6kGKWiLwhIh9K5LoHs50+9wFjnNHI/U7fO53trBaR/+jHb6MZYqxYmKHoQeABVZ0OXAvMd9o/Aj6rqlOIrAr7o6jXnAt8XVUvch5PAW4DxgGnATN72E4msFRVJwHvALdEbf9BZ/sx1/1x1kX6PJEz7AE6gGtUdSqR66j8l1OsvgdsVtXJqnqniHwBqABmAJOBaSLy2VjbM6YntpCgGYouBsZFrRSaIyLZQC7wWxGpILLSZ3LUa15X1ehrGXygqrUAIrKSyJpB73XbThefLsy4HLjEuX8un16j4GngP3vJmR713suJXPMAImsG/cj5wx8mMuIY0cPrv+DcVjiPs4gUj3d62Z4xvbJiYYYiH3CuqrZHN4rIw8CbqnqNs///rainW7u9R2fU/RA9/y4F9NNJwd76HEu7qk4WkVwiReefgYeIXA+jCJimqgER2Qak9fB6AX6sqr8+zu0acxTbDWWGokVEricBgIgcWmI6F9jp3L/Jxe0vJbL7C+D6WJ1VtYnIpVi/KyLJRHLucwrFhcCpTtdmIDvqpQuBm53rLiAipSIyvJ++BjPEWLEwiS5DRGqjbncQ+cNb5Uz61hBZXh7gp8CPRWQx4Hcx023AHSLyAVAMNMV6gaquILKy6fVELkBUJSLVREYZHzl96oHFzqG296vqIiK7ud4XkTXA8xxZTIzpMzt01pgB5lztr11VVUSuB+aq6uxYrzPGSzZnYczAmwb8wjmC6QAuXsbWmP5iIwtjjDEx2ZyFMcaYmKxYGGOMicmKhTHGmJisWBhjjInJioUxxpiYrFgYY4yJ6f8AUuG5qR6de3sAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "learn.recorder.plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here 5e-2 looks like a good value, a tenth of the minimum of the curve. That's going to be the highest learning rate in 1cycle so let's try a constant training at that value." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "Total time: 00:05

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracy
10.1008880.0589690.978901
20.0530190.0361120.989696
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn.fit(2, 5e-2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also see what happens when we train at a lower learning rate" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "Total time: 00:05

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracy
10.1148850.0840580.968597
20.0665230.0457440.982826
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "model = simple_cnn((3,16,16,2))\n", "learn = Learner(data, model, metrics=[accuracy])\n", "learn.fit(2, 5e-3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training with the 1cycle policy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now to do the same thing with 1cycle, we use [`fit_one_cycle`](/train.html#fit_one_cycle)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "Total time: 00:05

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracy
10.0967210.0405240.986752
20.0342540.0228850.990677
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "model = simple_cnn((3,16,16,2))\n", "learn = Learner(data, model, metrics=[accuracy])\n", "learn.fit_one_cycle(2, 5e-2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This gets the best of both world and we can see how we get a far better accuracy and a far lower loss in the same number of epochs. It's possible to get to the same amazing results with training at constant learning rates, that we progressively diminish, but it will take a far longer time.\n", "\n", "Here is the schedule of the lrs (left) and momentum (right) that the new 1cycle policy uses." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAAD8CAYAAACIA0mdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd8lfX5//HXlT3IIAtIQghkAIEwAyiyxAWoUBGsq44626pt1SruqnVbtUNt3XsgomJFcIIsgTASRgYhQEgCZC+yk8/vDw7+8o0gAZLcZ1zPxyOPnnOfz+G8Q024zn0+93WJMQallFJKKaXUsblZHUAppZRSSilHocWzUkoppZRSHaTFs1JKKaWUUh2kxbNSSimllFIdpMWzUkoppZRSHaTFs1JKKaWUUh2kxbNSSimllFIdpMWzUkoppZRSHaTFs1JKKaWUUh3kYXWA9sLCwkxsbKzVMZRS6oRs2LChxBgTbnWO7qS/t5VSjupEfmfbXfEcGxtLamqq1TGUUuqEiMgeqzN0N/29rZRyVCfyO1u3bSillFJKKdVBWjwrpZRSSinVQVo8K6WUUkop1UFaPCullFJKKdVBWjwrpZRSSinVQR0qnkVkmohkiUiOiMw7wuPeIvKh7fG1IhJrOx4rInUistn29Z/Oja+UUkoppVT3OWarOhFxB54HzgLygfUissgYs73NsmuAcmNMvIhcDDwB/Nr22E5jzIhOzq2UUkoppVS360if57FAjjEmF0BEPgBmAW2L51nAX223FwD/FhHpxJyqi+0srmFTXgUHquoxxuDu5kZPP0+C/byICPSmf6g/Pf29rI6plHICmfur+HLLfnoF+jAsOoghkYHoPxlKqY5obmllX2U9e8tqOVBdT019M9UNzfh7eXDl+NhuydCR4jkK2Nvmfj4w7mhrjDHNIlIJhNoe6y8im4Aq4F5jzIr2LyAi1wPXA8TExBzXN6BOnDGGJVv38+LynaTnVx5zfbCfJ4m9AhgV05ORMcGM6x9CsJ8W1Eqp45Oxr4p/frcDYw7d7xfqxx+mxDN7VBQe7nopjlLqEGMMuSUHWb2zlC35FaTnV7KjqIaWVvOztXHh/nZVPB/pdED71Edbsw+IMcaUisho4FMRGWKMqfo/C415CXgJICUl5ed/I6rTFVc3cOv8zazYUUJcuD8PnJ/ExIQwonv64SZCc2srFbVNlB1sZH9lPbtLD5JbcpBthVW8ujKXphaDm8Dofj05fVAEM4b2ITbM3+pvSynlAC4YGc15wyI5UFXP6p2lvPPjHu74OJ331+fxj1+PJCbUz+qISimLtLQafswt5Yst+1ieVUxBRR0AIf5eJEcFMXVQBP1C/ejb04/eQT4E+nrSw9sDb4/ue+PdkeI5H+jb5n40UHiUNfki4gEEAWXGGAM0ABhjNojITiAR0DmuFtqSX8m1b62nsq6JB2cO4bJxMT872+OFG35eHkQG+zI0Kuj/PFbf1MKWgkpWZBfzbWYRTy7J4sklWYzu15PZo6KYNSKKHt52N/ldKWVHPN3diO7px0UpfswdHc1nmwu577OtnP/vlbx2VQqj+4VYHVEp1Y12Ftfw3to8FqUVUlzdgJ+XOxPiw7hxShyTEsKICfGzm+1dYswvn+i1FcPZwBlAAbAeuNQYs63Nmj8AycaYG20XDM42xlwkIuEcKqJbRGQAsMK2ruxor5eSkmJSU7W27iob88q58tV1BPp68vIVKSRFBp70n1lYUceitEI+3pDPjqIaArw9uGhMX648NVbPICmXIyIbjDEpVufoTp31ezuvtJYrX1/Hvso6XrtqDOPjwjohnVLKXhlj+C6ziDdW72bFjhI83YWpgyKYOTyKqYMi8PVy7/IMJ/I7+5jFs+0PngE8B7gDrxljHhGRh4BUY8wiEfEB3gZGAmXAxcaYXBG5EHgIaAZagAeMMZ//0mtp8dx1coqqueCF1YT4e/HedacQFezbqX++MYa0/EpeX7WLL9L30WoMs0ZE8cczEnRLh3IZWjyfnJKaBi556Uf2V9bz4Q2ndsobfKWUfTHG8E1GEc99k822wir6BPlw2bgYfj0mhvAA727N0mXFc3fS4rlrlNY0MOv5VdQ3tfLJ78fTN6Rrzwjvr6zn1ZW5vP3jHppaDBeOiuL2swcSEejTpa+rlNW0eD55hRV1XPjiagC+uGUiIdrpRymn8UN2MU8syWRbYRX9Qv24eWoCs0ZE4mnRxcIn8jtbL2t2Aa2thj/PT6OouoFXr0zp8sIZoHeQD/ecm8QPd5zOlafG8ummQk5/ehn/Wb6ThuaWLn99pZTjigz25aXfpFBa08ifPtxM6xGurFdKOZa80lqueyuVK15bR3V9M0/PHc63t05mzuhoywrnE+VYadUJeWlFLj9kF3P/eUkM7xvcra8dEeDD/ecn8dWfJ3FqXCiPf5nJtOdW8EN2cbfmUEo5luToIB6YmcQP2cW8sXq31XGUUieovqmFZ77K4sxnl7Mqp4Q7pw3i61snMWd0tMO2pnTM1KrDdhbX8MxX2ZwzpBeXjbOuh3ZsmD+vXDmGN64egwBXvLaOeR+nU13fZFkmpZR9u3RsDFMHRfDk0kx2lRy0Oo5S6jil7a3gvH+t5J/f5TBjaG++v30Kv5sSh7dH118I2JW0eHZira2GuxZuwcfTjYd/NdQuWrxMGRjB4j9O5MbJccxP3cs5z/7Ayh0lVsdSStkhEeGx2cl4ubtxzydbsLdrdJRSR9bQ3MJTSzOZ/eJqDjY08+Zvx/LcxSPp5STXPWnx7MTmp+5l3a4y7p4xmIgA+/kP1sfTnXnTB7Hgd+Px8XLn8lfX8rf/baeppdXqaEopO9Mr0IfbzxnI6p2lLN12wOo4Sqlj2F1ykAueX83z3+/kwlFRLP3zJCYnhlsdq1Np8eykKmubeOzLTMb1D+HXY/oe+wkWGBXTk8W3TOSKU/vxyspdXPTfNT9NElJKqcMuHRvDwF4BPLJ4O/VNesGxUvbqi/R9nPevlRRW1vHyFSk8OWc4gT6eVsfqdFo8O6kXluVQVd/EA+cPsYvtGkfj4+nOQ7OG8u9LR7LjQA3n/nMF32Xq2SWl1P/n4e7G/ecnsbesjldX7rI6jlKqnYbmFu7/bCt/eG8jib168MUtEzkrqZfVsbqMFs9OKL+8ltdX72b2yGiHGTBw3rBIPr95ApFBvlzzZiovLtup+xuVUj85LT6Ms5J68Z9lO6mobbQ6jlLKprSmgctfWctba/Zw3cT+fHjDqZ0+hM3eaPHshJ75KhuA285OtDjJ8ekf5s/C34/n3OQ+PLEkk9s+StOe0Eqpn9x2diLVDc28pmeflbILWfurmfX8KtLzK/nnJSO559wkh+vZfCKc/zt0MTlF1XyyuYCrT4sl0gHf+fl4uvOvS0bypzMTWLixgMteXktJTYPVsZRSdmBQ70BmJPfmtVW79eyzUhb7LvMAs19YRWNzKx/ecCozh0daHanbaPHsZF74fic+Hu7cMCnO6ignTET405mJPH/pKLYWVjLnxdXsLau1OpZSDkNEpolIlojkiMi8IzzeT0S+FZF0EVkmItHtHg8UkQIR+Xf3pe6YW85IoKahWfc+K2Wh+ev3cu2bqcSG+fPZTacxopsHsFlNi2cnklday2dphVw6LoYQfy+r45y0c4f14d1rT6G8tonZL64mY1+V1ZGUsnsi4g48D0wHkoBLRCSp3bKngbeMMcOAh4DH2j3+MLC8q7OeiMNnn19ftZsqHbKkVLcyxvDCshzu+DidCQnhzL/hVPoEOd6n3CdLi2cn8p8fduIuwvWTBlgdpdOM7teTj248FXcRLvrvGtbmllodSSl7NxbIMcbkGmMagQ+AWe3WJAHf2m5/3/ZxERkN9AK+6oasJ+TGyXHUNDQzf/1eq6Mo5TJaWw0P/y+DJ5dkMXN4JK9ckYK/t4fVsSyhxbOTKKqqZ0FqPnNSop1mgs9hib0C+Pj34wkP8OaK19axPLvY6khK2bMooG1VmW871lYacKHt9gVAgIiEiogb8HfgL12e8iQMiw5mbP8QXl+1m2YdrqRUl2ttNcxbmM5rq3Zx1fhYnvv1CLw8XLeEdN3v3Mm8/eMemlpbucGJzjq3FRXsy4IbxxMX3oPr3krl+6wiqyMpZa+O1Ni9fd/H24HJIrIJmAwUAM3A74HFxphjntIVketFJFVEUouLu/8N7bUT+lNQUadTB5XqYi2thjs+Tmd+aj63nJHAA+cn4eZmv/MjuoMWz06gvqmF99bmccagXvQL9bc6TpcJ8ffivevGkdirBze8tYFvM/QfTaWOIB9oO1Y0Gihsu8AYU2iMmW2MGQncYztWCZwK3CQiuzm0L/oKEXn8SC9ijHnJGJNijEkJD+/+0btnDO5Fv1A/XlmZ2+2vrZSraGk1/GVBGgs25POnMxO49axEux681l20eHYCn6cVUnqwkatPi7U6SpcL9vPi3WtOYXCfAG58ZwNfb9cCWql21gMJItJfRLyAi4FFbReISJhtiwbAXcBrAMaYy4wxMcaYWA6dnX7LGPOzbh32wN1NuHp8LJvyKkjbW2F1HKWcTkur4faP0li4sYBbz0rkT2c61uyIrqTFs4MzxvD6qt0M7BXA+LhQq+N0iyA/T96+dhxDIoP4w7sbWbmjxOpIStkNY0wzcBOwFMgA5htjtonIQyIy07ZsCpAlItkcujjwEUvCnqTZo6Px9XTnvbV5VkdRyqm0thr+8lEan2wq4PazE7nljASrI9kVLZ4d3Prd5WzfV8VVp8W61EcpgT6evHn1WAaE+3PdW6ls2FNudSSl7IYxZrExJtEYE2eMecR27H5jzCLb7QXGmATbmmuNMT+bRGSMecMYc1N3Zz8egT6ezBweyaK0Qm1bp1QnMcbw4OfbWLipgNvOSuSmqVo4t6fFs4P7YF0eAd4e/GpE+4vpnV+QnydvXzOO3kE+XP36OrYXah9opVzNpeNiqGtq4bPNhcderJQ6pme/2cGba/Zw/aQB3DQ13uo4dkmLZwdWWdfEF1v2MWtkJL5e7lbHsUR4gDfvXDuOHt4eXPHaWnKLa6yOpJTqRsOigxgSGch7a/Mwpn1TEaXU8Xht5S7++e0OLkqJ5q7pg1zqE+3jocWzA1u0uYCG5lYuHhNjdRRLRQX78va14zAGLn9lLfsr662OpJTqJiLCpeNiyNhXRVp+pdVxlHJYH2/I56H/bWfakN48ekGyFs6/QItnB/Zh6l6S+gQyNCrI6iiWiwvvwZu/HUtlXRNXv7GemoZmqyMppbrJzOGReHu4sXBjvtVRlHJIy7KKuOPjdE6LD+Ufl4zAw13Lw1+ifzsOamtBJVsLqrh4bN9jL3YRQ6OCeOHy0WQfqOb3726kSSePKeUSAnw8OXtIbxalFdLYrD/3Sh2P7YVV/OHdjQzsFcB/f5OCt4drbgM9Hlo8O6gP1+/F28ONWcNd70LBXzI5MZxHLxjKD9nF3PfpVt0DqZSLmD0yioraJpbp9FGlOmx/ZT2/fWM9AT6evHbVGHp4e1gdySFo8eyA6pta+HRzAdOH9ibIz9PqOHbn12NiuHlqPB+s38vz3+dYHUcp1Q0mJoQR1sOLTzYVWB1FKYdQ09DM1W+sp7q+ideuGkPvIB+rIzkMLZ4d0FfbD1Bd38xFKbpl42huPSuR2SOjePqrbD7brP+YKuXsPNzdmDk8im8ziqis1Z7PSv2S5pZWbnpvI9kHqnn+slEkRQZaHcmhaPHsgD7bVECfIB9OGeAaEwVPhIjw+IXDGBsbwh0L0knP1/G9Sjm72aOiaGxp5X9btOezUr/kb19ksCyrmIdmDWHKwAir4zgcLZ4dTPnBRpZnFzNzeCRubtpG5pd4ebjx4uWjCOvhzfVvbaCoSlvYKeXMhkQGktirB59s1E+blDqa+al7eWP1bq4+LZbLxvWzOo5D0uLZwXyxZR/NrYaZIyKtjuIQQnt48/IVKVTVN3H92xuob2qxOpJSqouICBeMjCZ1Tzl5pbVWx1HK7mzKK+feT7ZyWnwo98wYbHUch9Wh4llEpolIlojkiMi8IzzuLSIf2h5fKyKx7R6PEZEaEbm9c2K7rkWbC0mI6EFSH92f1FFJkYE8c9FwNu+t4O5PtmgHDqWc2HnD+gCweOs+i5MoZV8OVNVzw9sb6BXkzb8vGaW9nE/CMf/mRMQdeB6YDiQBl4hIUrtl1wDlxph44FngiXaPPwt8efJxXVt+eS3rdpcxa0SkTv45TtOG9uHPZyaycGMBr67cZXUcpVQX6Rvix/DoIBZv0eJZqcMamlu48Z0N1DQ08/IVKfT097I6kkPryNuOsUCOMSbXGNMIfADMardmFvCm7fYC4AyxVXci8isgF9jWOZFd16K0QxfBzBqhvZ1PxM1T45k2pDePfZnJ2txSq+MopbrIjOQ+pOdXsrdMt24oZYzhvk+3simvgr/PHc6g3vrJ9cnqSPEcBextcz/fduyIa4wxzUAlECoi/sCdwIMnH1Ut2lzI6H496RviZ3UUh+TmJjw1dxj9Qvy46f1NegGhUk5qRvKhrRtf6NlnpXhnbR7zU/O5ZWo8020/G+rkdKR4PtL+gPabRo+25kHgWWNMzS++gMj1IpIqIqnFxcUdiOR6dhyoJnN/NTOH64WCJyPAx5MXLx9NTX0zN723SUd4K+WE+ob4MUy3bihFen4FD3++ndMHhvOnMxOtjuM0OlI85wNtp3FEA+2baP60RkQ8gCCgDBgHPCkiu4E/AXeLyE3tX8AY85IxJsUYkxIeHn7c34QrWLxlPyIwPbm31VEc3sDeATw2O5l1u8t4ckmm1XGUUl3gXN26oVxcZW0Tv393I+EB3jxz0Qhtb9uJOlI8rwcSRKS/iHgBFwOL2q1ZBFxpuz0H+M4cMtEYE2uMiQWeAx41xvy7k7K7lC+37mNMvxAiAnR8Zmf41cgorji1Hy+v2KVnp5RyQoe3bujPt3JFra2G2z7azIGqev596Ui9QLCTHbN4tu1hvglYCmQA840x20TkIRGZaVv2Kof2OOcAtwI/a2enTtzO4hoy91frWedOds+5gxnRN5g7FqSzs/gXdxYppRyMbt1QruylFbl8k1HE3TMGMzKmp9VxnE6HmvwZYxYbYxKNMXHGmEdsx+43xiyy3a43xsw1xsQbY8YaY3KP8Gf81RjzdOfGdw1Ltu4HYNpQLZ47k7eHOy9cNgovDzf+8O5GHaCilJOZPrQPafmVFFTUWR1FqW6zNreUp5ZmcW5yH64aH2t1HKekHbIdwJdb9zEyJpg+Qb5WR3E6kcG+/H3ucDL3V/PIFxlWx1FKdaKzh/QC4JvtByxOolT3KK5u4Ob3NxET4sfjFybrTIguosWzncsrrWVrQRUzhmp7ma5y+qAIrpvYn7d/3MMSnUqmlNOIC+/BgHB/vtbiWbmA1lbDrfM3U1nXxPOXjiLAx9PqSE5Li2c796WtmNMtG13rL+cMYlh0EHcsSCe/XK/OV8pZnJXUix9zS6msa7I6ilJd6rVVu1ixo4R7z0siKVIHoXQlLZ7t3OKt+xkWHaSDUbqYl4cb/7pkJK0G/vjBZpq1/7NSTuHspN40txqWZRVZHUWpLrO1oJInlmRyVlIvLh8XY3Ucp6fFsx3bX1lP2t4KzhmiZ527Q79Qfx6dncyGPeU8980Oq+MopTrByL7BhPXw5ivduqGcVG1jM7d8sIkQfy+euHCY7nPuBlo827FvMg79sj87qZfFSVzHzOGRXJQSzfPLcliVU2J1HKXUSXJzE84cHMHyrGIamrWjjnI+Dy7azq6Sgzz76xGEaD/nbqHFsx37JuMAsaF+xEf0sDqKS/nrzCEMCPPn1vmbqahttDqOUuoknZXUi5qGZn7MLbM6ilKd6ov0fXyYupffTY5jfFyY1XFchhbPdqqmoZnVOaWcObiXfgTTzfy8PPjHxSMprWnknk+2YoyxOpJS6iScFh+Gr6c7X2/fb3UUpTpNQUUddy1MZ3jfYP58VqLVcVyKFs92akV2MY0trZypWzYsMTQqiD+flcgXW/bxyaYCq+MopU6Cj6c7kxLD+GZ7kb4ZVk6hpdXwpw820WrgnxePwNNdy7nupH/bdurr7QcI9vMkpZ+O1bTKjZPjGBPbkwc+26bt65RycGcM7sX+qnoy9lVbHUWpk/af5TtZv7uch2YNoV+ov9VxXI4Wz3aouaWV77KKmDowAg99N2kZdzfhmYtGYIBb56fR0qpnrJRyVFMSwwH4XlvWKQe3vbCK577J5txhfbhgZJTVcVySVmZ2KHVPORW1TZylWzYs1zfEj7/OHMK6XWW89EOu1XGU6hARmSYiWSKSIyLzjvB4PxH5VkTSRWSZiETbjo8QkTUiss322K+7P33XiAj0YWhUoPZ7Vg6tobmFW+dvJtjPi7/NGqrXRFlEi2c79M32A3i5uzHRdqZEWevCUVFMH9qbZ77OYmtBpdVxlPpFIuIOPA9MB5KAS0Qkqd2yp4G3jDHDgIeAx2zHa4ErjDFDgGnAcyIS3D3Ju97UgRFs2FOuXXSUw3r26x1k7q/miQuT6alt6SyjxbOdMcbwdcYBxseH0sPbw+o4ChARHr0gmZ5+Xvz5w83UN2mvWGXXxgI5xphcY0wj8AEwq92aJOBb2+3vDz9ujMk2xuyw3S4EigCneRc/ZVAErQZ+2KE93JXjWb+7jP/+sJNLxvZl6iD9ZNpKWjzbmZyiGvaU1nLmYP3BsCc9/b14au5wdhTV8PTSLKvjKPVLooC9be7n2461lQZcaLt9ARAgIqFtF4jIWMAL2NlFObvd8OhgQvy9WJapWzeUYznY0Mxt89OI7unLPee2/yBJdTctnu3M17apglo825/JieFcNi6GV1ftInW3DltQdutImyDbX+16OzBZRDYBk4ECoPmnP0CkD/A2cLUxpvWILyJyvYikikhqcXFx5yTvYu5uwuTEcJZlF+sFwMqhPLI4g73ltfx97gj9VNoOaPFsZ5ZlFjMkMpDeQT5WR1FHcNeMwUQF+3L7R2nUNer2DWWX8oG+be5HA4VtFxhjCo0xs40xI4F7bMcqAUQkEPgCuNcY8+PRXsQY85IxJsUYkxIe7jg7O6YMDKfsYCPp+RVWR1GqQ77PKuK9tXlcN3EAY/uHWB1HocWzXamqb2JDXjlTBjrOP0Supoe3B0/OGcbu0lqeXJppdRyljmQ9kCAi/UXEC7gYWNR2gYiEicjh3/93Aa/ZjnsBn3DoYsKPujFzt5mcGI6bwPdZjnG2XLm2itpG7lyQTmKvHtyqUwTthhbPdmR1TgktrYbJiRFWR1G/YHxcGFec2o83Vu9mbW6p1XGU+j+MMc3ATcBSIAOYb4zZJiIPichM27IpQJaIZAO9gEdsxy8CJgFXichm29eI7v0OulawnxejYnpqyzrlEO7/bBtlBxt55qIR+Hi6Wx1H2WjxbEeWZxcT4O3ByBin6QzltO6cNoi+Pf34y4J0ahubj/0EpbqRMWaxMSbRGBNnjHnEdux+Y8wi2+0FxpgE25prjTENtuPvGGM8jTEj2nxttvJ76QqnD4ogPb+Soup6q6ModVRfbdvPorRCbp6awNCoIKvjqDa0eLYTxhiWZRVzWnyYzqh3AP7eHjw1Zxh5ZbU88aVu31DKkRzeGrdMt24oO1VZ28S9n25lcJ9Afn96nNVxVDtapdmJHUU17Kus1/3ODmTcgFCuGh/Lm2v2sGanbt9QylEk9QkkIsCbH7K1eFb26eEvtlN6sJGn5gzTE2p2SP8fsRPLbWdAJulUQYdyx7SBxIb68ZcFaRxs0O0bSjkCEWFiQjgrbdeZKGVPlmUVsWBDPjdOHqDbNeyUFs92Ynl2MYm9ehAZ7Gt1FHUc/Lw8eGrucAoq6njsywyr4yilOmhSYhgVtU1sLai0OopSP6mub+LuhVuIj+jBzVMTrI6jjkKLZztQ29jMul1lTNazzg5pTGwIvz2tP+/8mMeP2n1DKYcwIT4MQLduKLvy+JeZ7K+q56k5w7S7hh3T4tkOrNlZSmNLq7aoc2C3nZ1I3xBf7lq4hfomHZ6ilL0L7eHN0KhAVuwosTqKUgCs3lnCu2vzuGZCf0bG9LQ6jvoFWjzbgeXZxfh6ujOmv/6wOCo/Lw8enz2MXSUHee6bHVbHUUp1wKSEcDbmlVNd32R1FOXiahubmffxFmJD/bj1rIFWx1HHoMWzHVieXcz4uFC8PfQjGkd2WnwYF6VE8/KKXN1HqZQDmJQYTnOrYbV2y1EWe2ppFnlltTw5Zzi+XloL2Dstni22u+Qge0prmawt6pzCPTOSCPH34o4F6TS1tFodRyn1C0bF9MTfy50VO3Tfs7JO6u4y3li9mytP7cfY/iFWx1EdoMWzxQ6PiNWLBZ1DkJ8nD88awvZ9Vby8ItfqOEqpX+Dl4capcaH8kK37npU16ptauGNBOpFBvtwxbZDVcVQHafFsseXZxfQP86dfqL/VUVQnmTa0D9OH9ua5b3aws7jG6jhKqV8wMSGcvLJa9pQetDqKckH/+HYHuSUHeeLCYfh7e1gdR3VQh4pnEZkmIlkikiMi847wuLeIfGh7fK2IxNqOjxWRzbavNBG5oHPjO7b6phbW5JbqWWcn9OCsIfh4uHHXx1to1SEMStmtw4OptGWd6m7bC6t46YdcLkqJZkJCmNVx1HE4ZvEsIu7A88B0IAm4RESS2i27Big3xsQDzwJP2I5vBVKMMSOAacB/RUTfWtms311GfVOrFs9OKCLAh3vPS2Ld7jLeXZdndRyl1FHEhvoR3dOX5bp1Q3WjllbDXZ9soaefJ3fPGGx1HHWcOnLmeSyQY4zJNcY0Ah8As9qtmQW8abu9ADhDRMQYU2uMOTyz2AfQU3BtLM8qxsvDjXED9AIBZzR3dDQT4sN44stMCivqrI6jlDoCEWFSYjhrdpboRb6q27y9Zjdpeyu4//whBPt5WR1HHaeOFM9RwN429/Ntx464xlYsVwKhACIyTkS2AVuAG9sU0z8RketFJFVEUouLXeejs2XZxYzrH4Kfl56Md0YiwqMXJNPSarj3060Yo+8dlbJHkxLCOdjYwsY95VZHUS6gsKKOp5ZmMTkxnPOH9bE6jjoBHSme5QjH2lcyd0Y+AAAgAElEQVQBR11jjFlrjBkCjAHuEhGfny005iVjTIoxJiU83DW2MOSX15JTVKNbNpxcTKgft52dyHeZRSxKK7Q6jlLqCMbHh+LuJjptUHU5Ywz3f7aVVgN/+9VQRI5UPil715HiOR/o2+Z+NNC+CvhpjW1PcxBQ1naBMSYDOAgMPdGwzuRwa6Qp2t/Z6V19Wn+G9w3mwc+3U36w0eo4Sql2An08Gdk3mB+037PqYku27uebjCJuPSuRviF+VsdRJ6gjxfN6IEFE+ouIF3AxsKjdmkXAlbbbc4DvjDHG9hwPABHpBwwEdndKcge3PLuIqGBf4sJ7WB1FdTF3N+Hx2clU1TXx6OIMq+MopY5gYkI4Wwoq9Q2u6jKVdU08sGgbQyIDufq0WKvjqJNwzOLZtkf5JmApkAHMN8ZsE5GHRGSmbdmrQKiI5AC3Aofb2U0A0kRkM/AJ8HtjjMt/LtbY3MqqnFImJYbrRzYuYnCfQK6bNICPNuSzeqfL/wgoZXcmJIRhDKzSn0/VRZ5ckklJTQOPzx6Gh7uO2XBkHbpSzRizGFjc7tj9bW7XA3OP8Ly3gbdPMqPT2ZhXTk1Ds+53djF/PCOBL9L3cc8nW/nyjxPx8XS3OpJSymZ4dBABPh6s3FHCecMirY6jnEzq7jLeXZvHNRP6kxwdZHUcdZL0rY8FlmcX4+EmnBYfanUU1Y18PN155IKh7Co5yAvf51gdRynVhoe7G+PjQlmxo0Q746hO1djcyl0LtxAV7MutZyVaHUd1Ai2eLbA8q5jR/XoS4ONpdRTVzSYmhHPByCheXL6THQeqrY6jlGpjYkI4BRV17CrRUd2q8/x3+U52FNXwt18N1RHcTkKL525WVFXP9n1VTNYuGy7r3nMH4+/twV0LdXS3UvZkom1EsrasU51lZ3EN//ouh/OG9eH0QRFWx1GdRIvnbrY8+1ArJN3v7LpCe3hz94zBpO4p54P1e4/9BKVUt+gX6k9MiJ8Wz6pTGGO4e+EWfDzduP/8JKvjqE6kxXM3W55dTHiAN0l9Aq2Ooiw0d3Q0pwwI4bEvMyiqqrc6jlLKZkJCGD/mluqobnXSPkrNZ+2uMu6eMZiIgJ/Nh1MOTIvnbtTSalixo4TJ2qLO5R0e3d3Q3MqD/9tudRyllM2khDBqGprZvLfC6ijKgRVXN/DI4gzGxoZwUUrfYz9BORQtnrtRWn4FlXVNumVDATAgvAc3nR7PF+n7+C7zgNVxlFLAqXFhuAmsyNZpg+rEPfy/7dQ1tvDo7KG4uenJMmejxXM3WpZVjJvAhPgwq6MoO3Hj5DgSInpw36fbONjQbHUcpVxekK8nw/sGsyJH9z2rE7Msq4hFaYX8/vQ44iMCrI6juoAWz91oeXYxw/sG09Pfy+ooyk54ebjx6OxkCirqePbrbKvjKKWAifFhpO2toLK2yeooysHUNjZz76dbiQv353dT4qyOo7qIFs/dpOxgI+n5FbplQ/3MmNgQLh0Xw2urdrG1oNLqOEq5vImJ4bQaWJOrZ5/V8Xnumx3kl9fx2OxheHvoFFlnpcVzN1mxoxhjYMpA7fOofu7OaYMI7eHNvIXpNOtV/uokicg0EckSkRwRmXeEx/uJyLciki4iy0Qkus1jV4rIDtvXld2b3D6M6BtMD28PftCWdeo4bC2o5JUVuVwyNoax/UOsjqO6kBbP3WR5djE9/TxJjtKZ9urngnw9eeD8JLYWVPHG6t1Wx1EOTETcgeeB6UAScImItG8y+zTwljFmGPAQ8JjtuSHAA8A4YCzwgIj07K7s9sLT3Y1TBoSyUotn1UHNLYdGcIf4ezNv2iCr46gupsVzN2htNfyQXczEhHDc9apbdRTnJvdh6qAI/v5VNvnltVbHUY5rLJBjjMk1xjQCHwCz2q1JAr613f6+zePnAF8bY8qMMeXA18C0bshsdyYmhJFXVsueUh3VrY7tjdW72VJQyV9nJhHk52l1HNXFtHjuBtv3VVFS06j7ndUvEhEemjUEgPs/24YxOrpbnZAooO3oynzbsbbSgAttty8AAkQktIPPdQk6qlt1VH55LX//KpupgyI4N7mP1XFUN9DiuRscHsk9SYtndQzRPf247exEvsssYvGW/VbHUY7pSB9vtX8ndjswWUQ2AZOBAqC5g8899CIi14tIqoikFhc7X0/k/mH+RAX7smKH831vqvMYY7j/s22IwEOzhugANBehxXM3WJ5VzNCoQMIDvK2OohzAVeNjGRoVyF8/30ZlnbbKUsctH2g70iwaKGy7wBhTaIyZbYwZCdxjO1bZkee2+TNeMsakGGNSwsOd78SAiDAxIYzVO0v1Il51VF9s2cd3mUXcdvZAonv6WR1HdRMtnrtYZV0TG/LKdcuG6jAPdzcenz2M0poGnliSaXUc5XjWAwki0l9EvICLgUVtF4hImIgc/v1/F/Ca7fZS4GwR6Wm7UPBs2zGXNCEhjOr6ZtLytYWk+rnK2ib+umg7yVFBXDU+1uo4qhtp8dzFVueU0NJqmJyoLepUxw2NCuKaCf15b20e63eXWR1HORBjTDNwE4eK3gxgvjFmm4g8JCIzbcumAFkikg30Ah6xPbcMeJhDBfh64CHbMZd0WlwYImjXDXVEjy/JoLy2kcdmJ2szABejxXMXW55dTIC3ByNjgq2OohzMn89KJCrYl3kfp9PQ3GJ1HOVAjDGLjTGJxpg4Y8zhwvh+Y8wi2+0FxpgE25prjTENbZ77mjEm3vb1ulXfgz3o6e9FclSQ7ntWP7M2t5T31+3lmgn9GaotaF2OFs9dyBjD8uxiJiSE4emuf9Xq+Ph5efC3C4ays/ggLy7baXUcpVzSxIQwNu2toLperz9QhzQ0t3DXJ1uI7unLn85MsDqOsoBWdF0o+0AN+yrrdb+zOmGnD4xg5vBIXvh+JzlF1VbHUcrlTIgPp6XVsGZnqdVRlJ144fud5BYf5G+/Goqfl4fVcZQFtHjuQsuziwBtUadOzn3nJeHr5c7dC7fS2qq9n5XqTqP6BePn5c7KHN33rCCnqJoXluUwa0QkUwbqtUyuSovnLrQ8u5jEXj2IDPa1OopyYOEB3twzYzDrdpfxYereYz9BKdVpvD3cGdc/RIelKFpbDXct3IK/twf3ndd+4r1yJVo8d5GDDc2s36Ut6lTnmJsSzSkDQnh0cQZFVfVWx1HKpUxMCGdXyUH2ltVaHUVZ6P31eazfXc49MwYT1kPnNrgyLZ67yJqdpTS2tGqLOtUpRIRHL0imobmVB/+33eo4SrmUw6O6deuG6zpQVc/jizM5dUAoc0ZHWx1HWUyL5y6yLLsIPy93xvTvaXUU5SQGhPfg5tPj+SJ9H99mHLA6jlIuIz6iB70DfbTfswt78PNtNLS08ujsZB3BrbR47grGGL7PLGZ8XBjeHu5Wx1FO5IbJcSRE9OC+T7dysKHZ6jhKuQQRYUJCGCttQ6+Ua/l6+wEWb9nPH89IoH+Yv9VxlB3Q4rkL7CyuoaCijikDdb+z6lxeHm48fmEyhZX1/P2rbKvjKOUyJiaEUVnXxNYCHdXtSmoamrn/s60M7BXA9ZMGWB1H2QktnrvAsqxD06i0eFZdYXS/EC4/JYY3Vu8ibW+F1XGUcgmnxR/a96zTBl3L00uz2F9Vz2MXJuuwM/UT/S+hCyzLKiYhogfRPf2sjqKc1B3TBhEe4M28hVtoamm1Oo5STi+shzdDIgO1ZZ0L2ZRXzptrdnPFKf0YFaPXL6n/r0PFs4hME5EsEckRkXlHeNxbRD60Pb5WRGJtx88SkQ0issX2v1M7N779OdjQzLpdZXrWWXWpQB9PHpw5lIx9Vby2cpfVcZRyCRMSwtiYV67XG7iAppZW7lq4hV4BPtx+zkCr4yg7c8ziWUTcgeeB6UAScImItO8Ofg1QboyJB54FnrAdLwHON8YkA1cCb3dWcHu12tai7nSdPKS62LShvTk7qRfPfpNNXqn2n1Wqq01KCKepxbB2l47qdnYvr8glc381D80aQoCPp9VxlJ3pyJnnsUCOMSbXGNMIfADMardmFvCm7fYC4AwREWPMJmNMoe34NsBHRJy6s/iyrCL8vdxJiQ2xOopyAQ/OGoKHmxv3fLoFY7QLgFJdaXS/nnh7uPFDtm7dcGa7Sw7yj292MH1ob84e0tvqOMoOdaR4jgLazgTOtx074hpjTDNQCYS2W3MhsMkY03BiUe2fMYZlWcWMjw/Dy0O3k6uu1yfIlzumDWTFjhI+3VxgdRylnJqPpzvjBoTqsBQnZozhnk+34OXuxl9nDrE6jrJTHanwjtQNvP0prl9cIyJDOLSV44YjvoDI9SKSKiKpxcWOeyVzTtGhFnW6ZUN1p8vG9WNkTDAP/y+DsoONVsdRyqlNjA8jp6iGfZV1VkdRXeDjjQWsyinlzumD6BXoY3UcZac6UjznA33b3I8GCo+2RkQ8gCCgzHY/GvgEuMIYs/NIL2CMeckYk2KMSQkPd9wL7bRFnbKCu5vw2Oxkquqa+NsXOrpbqa40MfFwyzo9++xsiqrrefh/20np15NLx8ZYHUfZsY4Uz+uBBBHpLyJewMXAonZrFnHogkCAOcB3xhgjIsHAF8BdxphVnRXaXi3LLiKxVw8ig32tjqJczKDegdwweQALNxawPNtxP71Ryt4N7BVAeIC3Fs9O6K+LtlHX1MITc4bh5qYjuNXRHbN4tu1hvglYCmQA840x20TkIRGZaVv2KhAqIjnArcDhdnY3AfHAfSKy2fbllHsaDreo0y0byio3T00gLtyfuxduoUZbaSnVJUSEifFhrMopoVVHdTuNJVv3/TSCOy68h9VxlJ3r0FVtxpjFxphEY0ycMeYR27H7jTGLbLfrjTFzjTHxxpixxphc2/G/GWP8jTEj2nwVdd23Y51VOSU0tRgm65YNZREfT3eenDOcwso6nvgy0+o4SjmtiYlhlB1sZPu+KqujqE5QUdvIvZ9uY0hkoI7gVh2iLSE6ybLs4kMt6vppizplndH9enL1+P68/eMefszVXrRKdYX/P6pbt244g4f/l0FFbSNPzhmmI7hVh+h/JZ3AGMPyrGImJGiLOmW9289JJCbEj3kfp1PX2GJ1HKWcTkSAD4N6B7Bih15f4OiWZxfz8cZ8bpwcx5DIIKvjKAehlV4n2GFrUTdF9zsrO+Dn5cHjFyazu7SWZ77OsjqOUk5pYkIYqbvL9Q2qA6tpaObuhVuIC/fnpqnxVsdRDkSL507wbcahbdzaok7Zi/FxYVw2LoZXV+5iU1651XGUcjoTE8JpbGnVUd0O7MklmRRW1vHknOH4eLpbHUc5EC2eO8E3GQcYGhVInyBtUafsx7zpg+gd6MMdC9JpaNazY0p1prH9Q/DycGOl7nt2SOt2lfHWmj1cNT6W0f16Wh1HORgtnk9SaU0DG/PKOXNwL6ujKPV/BPh48sjsZHYU1fDv73KsjqOUU/HxdGdsbIheNOiA6ptauPPjdPqG+PKXcwZaHUc5IC2eT9J3mUUYgxbPyi6dPjCCC0dF88KynWwrrLQ6jlJOZUJCGFkHqimqqrc6ijoOz36Tza6Sgzw+exh+Xh5Wx1EOSIvnk/RtRhG9A30YEhlodRSljui+8wYT4u/FHQvSaWpptTqOUk5jYoK2rHM0G/PKefmHXC4e0/enloNKHS8tnk9CfVMLP+wo5ozBEYjoKE9ln4L9vHh41lC2FVbx3+U7rY6jlNMY3DuQUH8vVuZo8ewI6hpbuH1+Gn2CfLnn3MFWx1EOTIvnk/Bjbim1jS26ZUPZvWlDe3PusD7849sdZO7XqWhKdQY3N2FCQhgrdpRgjI7qtndPLc0it+QgT84ZRoCPp9VxlAPT4vkkfJtRhK+nO6fGhVodRaljenjWUIJ8vfjzh2k0Nuv2DaU6w4T4MEpqGsjcX211FPUL1uaW8vrqXfzmlH66XUOdNC2eT5Axhm8zDjAxIUz7QyqHEOLvxeOzk8nYV8U/v91hdRzVhURkmohkiUiOiMw7wuMxIvK9iGwSkXQRmWE77ikib4rIFhHJEJG7uj+9Y5mYcKi/v04btF8HG5q5fUHaocmr0wdZHUc5AS2eT9D2fVUUVtbrlg3lUM5M6sXc0dG8sCxHh6c4KRFxB54HpgNJwCUiktRu2b3AfGPMSOBi4AXb8bmAtzEmGRgN3CAisd2R21H1DvIhIaKHXjRoxx77MoP88jqenjscf2/trqFOnhbPJ+ib7UWIwOmDdCS3ciz3nZ9EnyBfbpufpqOFndNYIMcYk2uMaQQ+AGa1W2OAwy2CgoDCNsf9RcQD8AUaAd0kfwyTEsNZu6uM2sZmq6OodlbsKOadH/O45rT+jIkNsTqOchJaPJ+gbzIOMKJvMOEB3lZHUeq4BPp48tScYYcunFmaaXUc1fmigL1t7ufbjrX1V+ByEckHFgM3244vAA4C+4A84GljTFmXpnUCZwyKoLG5lVU5OqrbnlTVN3HHgnTiwv25XYehqE6kxfMJKKioY0tBJWcl6ZYN5ZjGx4dx1fhYXl+1m9U79eNmJ3OkvpntW0FcArxhjIkGZgBvi4gbh85atwCRQH/gNhEZcMQXEbleRFJFJLW42LX3+6bEhhDg7cG3GQesjqLaePjz7RyoqufpucP12iTVqbR4PgFLtu4HYPrQPhYnUerE3TltEAPC/PnLR+lU1zdZHUd1nnygb5v70fz/bRmHXQPMBzDGrAF8gDDgUmCJMabJGFMErAJSjvQixpiXjDEpxpiU8PDwTv4WHIuXhxuTEsP5LrOI1lZtWWcPvt5+gI825PO7KXGMjOlpdRzlZLR4PgFLt+5nUO8A+of5Wx1FqRPm6+XO0xcNZ19lHQ//b7vVcVTnWQ8kiEh/EfHi0AWBi9qtyQPOABCRwRwqnottx6fKIf7AKYDu7emAqYMiKKpuYFuhbhG3WlF1PXd+nE5Sn0BuOSPB6jjKCWnxfJyKqutZv6eMc4b0tjqKUidtVExPbpwcx/zUfL7erh85OwNjTDNwE7AUyOBQV41tIvKQiMy0LbsNuE5E0oD3gavMoSkfzwM9gK0cKsJfN8akd/s34YBOHxSByKHrYZR1jDH85aN0DjY0889LRuDtods1VOfTni3H6evtBzAGpidr8aycw5/OTGR5djF3LEhjyZ8m0SvQx+pI6iQZYxZz6ELAtsfub3N7O3DaEZ5Xw6F2deo4hfh7MSqmJ99lFvHnsxKtjuOy3ly9m+XZxTw0awjxEQFWx1FOSs88H6clW/fTP8yfgb30h1I5By8PN/5x8Ujqmlq4bX6a7tlU6gRNHRTBloJKDlTVWx3FJWUfqObRLzM5fWA4vzmln9VxlBPT4vk4VNQ2smZnKecM6Y3IkS5oV8oxxUf04IHzh7Ayp4RXV+6yOo5SDunw0KzvM4ssTuJ6GppbuOX9TQR4e/DknOH6b7TqUlo8H4dvMopobjVMH6pbNpTzuXhMX84Z0osnl2aytaDS6jhKOZzEXj2ICvblmwwtnrvb00uzyNxfzZNzhun8BdXltHg+Dku27iMyyIdh0UFWR1Gq04kIj88eRqi/N7e8v0mnpSl1nESEMwZHsCqnhPomnd7ZXVbllPDyil1cfkoMZwzW+Quq62nx3EGVtU0szy5mRnIf/ThIOa2e/l48c9FwdpUe1PZ1Sp2AqYMiqGtqYU2uThvsDmUHG7ltfhpx4f7cMyPJ6jjKRWjx3EFLtu2jqcUwc0Sk1VGU6lLj48O4YVIc76/by5db9lkdRymHcsqAUPy93Plqm7as62qtrYbb5m+m7GAj/7h4JL5e2pZOdQ8tnjvo87R99Av1IzlKt2wo53frWYkkRwUxb+EWCirqrI6jlMPw8XRnyqAIvt5+gBbtXNOlXlmZy/dZxdx33mCG6r/Nqhtp8dwBRdX1rN5ZwszhkbplQ7kELw83/nnJSFpaDTe9t5HG5larIynlMM4Z0puSmgY25pVbHcVpbcwr58klWcxI7s3l2pZOdTMtnjvgyy37aTVw/nDdsqFcR/8wfx6/MJlNeRU8uUQnNCvVUacPDMfL3Y0lW/dbHcUpVdY2cfN7m+gT7MNjs4fpSS3V7bR47oBFaYUM6h1Aog5GUS7mvGGRXHFqP15ZuYul27QQUKojAnw8mZAQxtJt+zk09Vx1FmMMty9Io6i6nn9fMoogX0+rIykXpMXzMeSX17JhT7medVYu655zB5McFcTtH6WRV1prdRylHMK0Ib3JL69jW2GV1VGcyhurd/P19gPcOW0Qw/sGWx1HuagOFc8iMk1EskQkR0TmHeFxbxH50Pb4WhGJtR0PFZHvRaRGRP7dudG7x+dph7oNnD9Mi2flmrw93HnhslEA/OG9jTQ0a/9apY7lzKReuAn6iU0n2phXzqOLMzhzcC+umdDf6jjKhR2zeBYRd+B5YDqQBFwiIu2bKV4DlBtj4oFngSdsx+uB+4DbOy1xNzLGsHBjPqNigokJ9bM6jlKW6Rvix9Nzh7OloJJHv8iwOo5Sdi/E34tx/UN133MnKa5u4HfvbKBPkC9Pz9V9zspaHTnzPBbIMcbkGmMagQ+AWe3WzALetN1eAJwhImKMOWiMWcmhItrhbN5bwY6iGuam9LU6ilKWO2dIb66Z0J831+zh000FVsdRyu5NG9qbHUU15BTVWB3FoTW1tHLTexuprGviP5ePJtjPy+pIysV1pHiOAva2uZ9vO3bENcaYZqASCO1oCBG5XkRSRSS1uLi4o0/rch9tyMfH043zhvWxOopSdmHe9EGM7R/CnR+ns7Wg0uo4Stm1s4ccGhWtw4ZOzuNfZrJ2VxmPzU4mKTLQ6jhKdah4PtJnI+0vH+7ImqMyxrxkjEkxxqSEh4d39Gldqr6phc/TCpk+tA8BPno1r1IAnu5uvHDZKEL9vbjh7Q2U1jRYHUkpu9UnyJexsSEsSivUrhsn6LPNBby6chdXjY/lgpHRVsdRCuhY8ZwPtN23EA0UHm2NiHgAQUBZZwS0ytJt+6mub2buaP1hVaqtsB7e/Oc3oymuaeCm9zbR1KIDVJQ6mvNHRLKjqIbM/dVWR3E4GfuquPPjdMbE9uSecwdbHUepn3SkeF4PJIhIfxHxAi4GFrVbswi40nZ7DvCdcfC32R+l5hPd05dTBnR494lSLmNYdDCPXZDMmtxSHl2sFxAqdTQzhvbG3U1YlNb+nJP6JRW1jdz4zgYCfTx5/tJReLprZ11lP475X6NtD/NNwFIgA5hvjNkmIg+JyEzbsleBUBHJAW4FfmpnJyK7gWeAq0Qk/widOuxOfnktq3aWMGd0NG5uekWvUkdy4ehorj4tltdX7Wbhxnyr4yhll0J7eDMhPozPdetGhzW1tPK7dzayr6KeFy8fRUSgj9WRlPo/PDqyyBizGFjc7tj9bW7XA3OP8tzYk8hniY9S8zEGLhylWzaU+iV3zxhMxr4q5i3cQmyYP6NielodSSm7M3N4JLd9lMamvRX6M3IMxhju/2wba3JL+fvc4YzuF2J1JKV+Rj8HaaeppZX31+UxOTGcviHa21mpX+Lp7sbzl46id6AP17+Vyt4ynUCoVHtnD+mFt4cbizbr1o1jeX3Vbt5fl8fvpsRxoV5zpOyUFs/tLN22n6LqBq4c38/qKEo5hNAe3rx21Rgam1v57RvrqaxrsjqSUnYlwMeTqYMi+GLLPlpadevG0XyfVcTfvtjO2Um9+MvZA62Oo9RRafHczltr9tA3xJfJiRFWR1HKYcRH9OA/l49mV8lB/vDuRu3AoVQ7M4dHUlzdwKqcEquj2KWs/dXc/N4mBvUO5Nlfj9DrjZRd0+K5jcz9VazbVcbl4/rhrj+4Sh2X8fFhPDo7mZU5Jdz/2Va9OEqpNqYOjiDYz5OPNujFte0VVtRx1evr8PVy55UrU/D37tDlWEpZRovnNt5cvQdvDzcu0nHcSp2Qi1L68vspcby/bi8vLNtpdRyl7Ia3hzu/GhHF0m37qahttDqO3aisbeKq19dRU9/Mm1ePJTLY1+pISh2TFs82xdUNfLwxn9mjoujp72V1HKUc1u1nD2TWiEieWprFB+vyrI6jlN24KKUvjc2tfLqpwOoodqG+qYXr3k5lV8lB/vub0Tp6WzkMLZ5t3li9i6aWVq6bOMDqKEo5NDc34ak5w5mcGM7dn2xhydZ9VkdSyi4kRQaSHBXE/FTdutHSavjzh5tZt6uMv180gvHxYVZHUqrDtHgGahqaeXvNHqYN6c2A8B5Wx1HK4Xl5uPHi5aMY0TeYW97fzOqdepGUUgAXpUSzfV8VWwsqrY5iGWMMDyzaypdb93PvuYOZOTzS6khKHRctnoH31+ZRVd/MDZPjrI6ilNPw8/LgtavGEBvmx/VvbXDpYkGpw2YOj8LLw435qXutjmIJYwyPLs7gnR/zuGHyAK7VT3uVA3L54rmusYWXV+RyyoAQRvQNtjqOUk4l2M+Lt347jiBfT37z6loy9lVZHUkpSwX5eTJ9aG8+3VRAfVOL1XG63TNfZ/Pyil1cNT6WedMGWR1HqRPi8sXzW2t2U1TdwK1naUN2pbpC7yAf3r12HN4e7lz2yloy92sB3dVEZJqIZIlIjojMO8LjMSLyvYhsEpF0EZnR5rFhIrJGRLaJyBYR8ene9M7v4jExVNU3u9zEwX9/t4N/fZfDxWP6cv95SYhoS1jlmFy6eK6qb+LF5TuZnBjO2P4hVsdRymnFhvnzwfWn4OkuXPryWrL2V1sdyWmJiDvwPDAdSAIuEZGkdsvuBeYbY0YCFwMv2J7rAbwD3GiMGQJMAXRkZCc7ZUAIg3oH8Prq3S7TD/2VFbk8/VU2F4yM4pELknUIinJoLl08v7JiFxW1TdyuY0CV6nKHCuhT8XATLn35R7IPaAHdRcYCOcaYXGNMI/ABMKvdGgMc7gsWBBw+BXo2kG6MSQMwxpQaY1xvb0EXExGuHB9Lxv9r797Do6rvPLmDFe0AABFzSURBVI6/v+QeAoRLgEAChIAQFUQMt4W1iFUEWxFrBYurbal4g22f1lpZuz6uT61226XYp9UuoqViG2+1wiotlfWGViDhIrdwExBCAoRbCEICSX77x5zUNJsMU8jMCTOf1/PMk3POnHl+n/wyfPnNOb9zpuw4hbuP+h0n7J5+9xN+9GYxEwd156c3D9aXkMkFL2YHz/srqnh2+U4mXNqdQVkd/I4jEhNyurSlYMZI4rwB9OZSTeEIg55Aw6vRSrxtDT0C3GZmJcASYJa3/SLAmdlSM1tjZg+EO2ysunFITzqkJLDgr7v8jhI2zjnm/GUrP/nzFr58WQ+enHo58XExO+yQKBKz7+IfvbmZM3WO2RPy/I4iElNyM9IomDGShLg2TJn3EUW7j/gdKdo0dViv8dyAW4EFzrksYCKw0MzaAPHAGGCa93OymV3dZCNmM8ysyMyKysvLWy59jEhJjGPq8GyWbjrAvmOn/I7T4pxzPPZmMb94ewe35Gcxd8oQEjRwligRk+/kD3cc4o31Zdw7NpdenVP9jiMSc3Iz0nj1nn8iIy2J255dyTtbDvodKZqUANkN1rP4fFpGvenAywDOuY+AZKCL99r3nHOHnHMnCRyVHtpUI865ec65fOdcfkZGRgv/CrHh9lF9MOCZ93f6HaVF1dTW8W9/3MD8D3Zxx6jePHGTpmpIdIm5wXN1TS0PL9pIr06p3K37Oov4pmd6Ci/fPYp+XdO48/kifWVxyykE+ptZjpklErggcHGjffYAVwOYWR6BwXM5sBQYbGap3sWDXwA2Ryx5jOmZnsKNl/ekYNUeyiur/Y7TIj6rruHO54soWLWXmVf145EbLtHFgRJ1Ym7wPOcv2/ik/DMenXQJyQlxfscRiWld0pIouHMk+X068p2X1jF32baYuftAuDjnaoCZBAbCxQTuqrHJzB41sxu83b4H3GlmHwMFwNddwFFgDoEB+DpgjXPuzcj/FrHjnrG5nK6t49kPLvy5zwcrq5g6bwXvbSvnx5MHcf/4AbodnUSleL8DRNJHnxxm3vKdfG1EL8YO6Op3HBEB2iUn8Pw3RzD7tQ3MXbad3Yc+44mvDNaH2/PgnFtCYMpFw20PN1jeDIxu5rUvELhdnURAbkYaEwdl8sKKT7nnC7l0SE3wO9I52bq/kum/LeTwidPMvyOfcQO7+R1JJGxi5shzWcUpZhWsoU/ntjw0URcJirQmifFt+NlXB/P98QN4fV0p0+avjJrT2CJnM/OqfpyoruGZ5Rfm3OclG8qY/NSHnK6p48UZIzVwlqgXE4Pnk6druHvhak6drmXev1xB26SYOuAuckEwM+67qh9PTRvKptIKrv/Fcgp1Jw6JAXmZ7fnyZT2Y/8FO9ldU+R0nZLV1jp8u3cK9v1vDwO7t+J9ZY7gsO93vWCJhF/WD5+qaWu5auJoN+yr4+ZQh9O/Wzu9IIhLExEGZvH7faNomxTN13gqeeX+n5kFL1Htg/ABq6xxz3trqd5SQHDpRzTcXFPKrdz7h1uHZFMwYSbf2+iZ3iQ1RPXg+XnWGb/ymkOXbD/HEVwZz7SXd/Y4kIiEY2L09i2aO5pq8bjy2pJi7Fq7m8AlN45Dold0pldtH9eGV1SVs2d+6vzzovW3lXDd3OSt2HubHkwfx+E2DSYrXNQoSO6J28Ly+5Bg3/upDVu06wpxbLuOW/Oyzv0hEWo32yQk8fdtQfnh9Hu9uLWf83OUs23zA71giYTNrXD/aJcXz8KJN1NW1vrMt1TW1/OiNzdzx3Co6t01k8cwxfG1EL79jiURcVA2enXMUlx3n+698zOSn/sqp07UsnD6Cm4Zm+R1NRM6BmfGtf+7Lopmj6ZKWyLeeL+IHr66nsuqM39FEWlx6aiIPXZ/Hql1HKCjc43ecv7Nq1xEmPLmc+R/s4vZRvVk0czQDumsapMSmqLhy7tkPdvHm+lJKjp7iYGU1SfFtuH1Ub77zxYvokHJh3vZHRD6XlxmYxvHksu38+r1PeHvrQX54fR43XNZD95GVqHJLfjaL1pXy+JItjBvYlcwOKb7mOV51hif+tIXfr9xDz/QUFnxjmG71KjEvKgbPtXV1pCTGMaZfF4bndOKLF3ejS1qS37FEpAUlxcfxwHUDGX9Jd/590Ua+/eI6Clbt4dFJl3KRLgSWKGFmPHHTYK6d+x4/+MMGFnx9mC/f0FdTW8crq0uY89Y2Dp+o5ltjcvjutReRmhgVwwaR82Kt7Sr2/Px8V1RU5HcMEWnFauscBav28NOlWzlRXcNXhvZk1rj+ZHdK9TsaZrbaOZfvd45IUt1ueS+s+JQfvr6R711zEbOu7h+xdp1zvLu1nMf/VMy2AycY2iudR264hMFZugWdRKdzqdn6CCkiF5y4NsZtI3sz4dLu/PKdHfxuxR7+uHYfU4f14t6rcn0/1S1yvqaN6EXR7iPMWbaNAd3bhf1uUfWD5qfe3UHh7qP06ZzK09OGct2l3TU1SqQRHXkWkQte6bFT/PKdHbxcuBcI3Ct6+pgcX76wQUeepaWcPF3Drc+sZEvZcRZOH8HwnE4t3sap07W8sb6U5z7cTXHZcXp0SObusblMHdaLxPiouqeASJPOpWaHNHg2s+uAJ4E4YL5z7olGzycBzwNXAIeBKc653d5zs4HpQC3wr865pcHaUhEWkXO198hJFvx1Ny8V7uVEdQ2X9GjPTUOzuOGyHmS0i8x1EBo8S0s6fKKar/73R5QeO8XT067gqoHnf7FeXZ1jXckxFq3dx2tr91FZVUO/rmncdWVfJg3pqUGzxJSwDJ7NLA7YBlwDlACFwK3Ouc0N9rkXGOycu9vMpgKTnXNTzOxioAAYDvQAlgEXOedqm2tPRVhEzldl1RleW7OPP6wpYX1JBXFtjPzeHRk3sCvjBnalX9e0sJ2K1uBZWlp5ZTVf/80qisuOM2tcf2aN60d83D82wD1RXUPhriO8v72cP2/cT1lFFYlxbZgwqDtfG96L4TmdND1DYlK4Bs+jgEecc+O99dkAzrnHG+yz1NvnIzOLB/YDGcCDDfdtuF9z7akIi0hL2n6gkkXrSllWfIAt+ysBSE9N4PLsdIZkdyS3a1tyurQlu1Mq7ZLiz3sAocGzhMOJ6hoefn0jr63dR98ubbl7bC7XXtyN9NTEv9vvTG0dB45XUVZRxbYDlRSXHWfjvuNs3FdBTZ0jMb4NV/bvwsRBmVyd1023c5WYF64LBnsCexuslwAjmtvHOVdjZhVAZ2/7ikav7fmPBBQROR/9u7Xj/vEDuH/8AEqPnWL59nLWfHqMtXuP8u62choeP0iIMzqkJNI+OZ4vDc7ku9cO8C+4SANpSfHMmTKECYMy+dnSrTzw6noeNMhol0RKQhxVZ+qoqqml4tSZv3tPt0uOJ697e+76Ql/+KbcLV/TuSHKCvkpb5HyEMnhu6jBM48PVze0TymsxsxnADIBevfRVnyISHj3SU5gyrBdThgXqzKnTtew+/Bm7Dn1GydGTHD15hmMnT1NZVUO3Dsk+pxX5/665uBtXD+zK+n0VvLe1nJKjJ6muqSM5oQ3JCXF0TE2kR3oy3donk5uRRlbHFE3HEGlhoQyeS4DsButZQGkz+5R40zY6AEdCfC3OuXnAPAic/gs1vIjI+UhJjCMvsz15me39jiISsjZtjCHZ6Qzx4W4yIgKhXHFQCPQ3sxwzSwSmAosb7bMYuMNbvhl42wUmUy8GpppZkpnlAP2BVS0TXUREREQkss565NmbwzwTWErgVnXPOec2mdmjQJFzbjHwLLDQzHYQOOI81XvtJjN7GdgM1AD3BbvThoiIiIhIaxbSNww655YASxpte7jBchXw1WZe+xjw2HlkFBERERFpFXQndBERERGREGnwLCIiIiISIg2eRURERERCpMGziIiIiEiINHgWEREREQmROde6vpPEzMqBT8/hpV2AQy0c53woT3DKE5zyBNea8/R2zmX4GSbSzrFut+a/YWugPMEpT3DKE9x51exWN3g+V2ZW5JzL9ztHPeUJTnmCU57glOfC19r6THmCU57glCe4aMujaRsiIiIiIiHS4FlEREREJETRNHie53eARpQnOOUJTnmCU54LX2vrM+UJTnmCU57goipP1Mx5FhEREREJt2g68iwiIiIiElZRMXg2s+vMbKuZ7TCzB33KsNvMNpjZOjMr8rZ1MrO3zGy797NjGNt/zswOmtnGBtuabN8CfuH113ozGxqhPI+Y2T6vj9aZ2cQGz8328mw1s/EtnCXbzN4xs2Iz22Rm3/a2+9I/QfL41T/JZrbKzD728vyHtz3HzFZ6/fOSmSV625O89R3e830ilGeBme1q0D9DvO1hfz977cSZ2Voze8Nb96V/ooFqtmr2WbKoZgfPo5odWq7w1Wzn3AX9AOKAT4C+QCLwMXCxDzl2A10abftP4EFv+UHgJ2Fs/0pgKLDxbO0DE4E/AQaMBFZGKM8jwP1N7Hux93dLAnK8v2dcC2bJBIZ6y+2AbV6bvvRPkDx+9Y8Bad5yArDS+71fBqZ6238N3OMt3wv82lueCrzUwv3TXJ4FwM1N7B/297PXzneB3wNveOu+9M+F/kA1u74t1ezms6hmB8+jmh1arrDV7Gg48jwc2OGc2+mcOw28CEzyOVO9ScBvveXfAjeGqyHn3PvAkRDbnwQ87wJWAOlmlhmBPM2ZBLzonKt2zu0CdhD4u7ZUljLn3BpvuRIoBnriU/8EydOccPePc86d8FYTvIcDxgGvetsb9099v70KXG1mFoE8zQn7+9nMsoDrgfneuuFT/0QB1WxUs8+SRTU7eB7V7LMId82OhsFzT2Bvg/USgr+pw8UBfzGz1WY2w9vWzTlXBoF/fEDXCGdqrn0/+2ymd5rmOfv8lGjE8ninYy4n8MnY9/5plAd86h/v9NY64CDwFoEjJcecczVNtPm3PN7zFUDncOZxztX3z2Ne//zczJIa52kia0uZCzwA1HnrnfGxfy5wqtnN870mNUE1u/k8oJrdZJ5or9nRMHhu6tOBH7cQGe2cGwpMAO4zsyt9yBAqv/rsaSAXGAKUAf8VyTxmlgb8AfiOc+54sF19yuNb/zjnap1zQ4AsAkdI8oK0GfE8ZnYpMBsYCAwDOgE/iEQeM/sScNA5t7rh5iBttpaa1Fq1lv5RzT471ezgeVSzm8kT7TU7GgbPJUB2g/UsoDTSIZxzpd7Pg8AfCbyZD9SfivB+HoxwrOba96XPnHMHvH9gdcAzfH4aK+x5zCyBQNH7nXPuNW+zb/3TVB4/+6eec+4Y8C6BeWjpZhbfRJt/y+M934HQT/eea57rvFOnzjlXDfyGyPXPaOAGM9tNYIrBOAJHNXzvnwuUanbzVLM9qtmhUc1uUthrdjQMnguB/t5VlIkEJnsvjmQAM2trZu3ql4FrgY1ejju83e4AFkUyV5D2FwO3e1e8jgQq6k+FhVOjOU2TCfRRfZ6p3hWvOUB/YFULtmvAs0Cxc25Og6d86Z/m8vjYPxlmlu4tpwBfJDCn7x3gZm+3xv1T3283A28751ryqEFTebY0+E/TCMxVa9g/Yft7OedmO+eynHN9CNSXt51z0/Cpf6KAanbzVLNRzQ4hj2p2EBGp2S4MVzhG+kHgys1tBOb8PORD+30JXFn7MbCpPgOBOTP/C2z3fnYKY4YCAqeNzhD4FDW9ufYJnKL4lddfG4D8COVZ6LW33nuzZjbY/yEvz1ZgQgtnGUPgFMx6YJ33mOhX/wTJ41f/DAbWeu1uBB5u8L5eReBil1eAJG97sre+w3u+b4TyvO31z0bgBT6/ujvs7+cG2cby+ZXbvvRPNDxQzW6uRqpmO9XsEPKoZoeebSxhqNn6hkERERERkRBFw7QNEREREZGI0OBZRERERCREGjyLiIiIiIRIg2cRERERkRBp8CwiIiIiEiINnkVEREREQqTBs4iIiIhIiDR4FhEREREJ0f8BHD8VPH+5BRUAAAAASUVORK5CYII=\n", "text/plain": [ "

" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "learn.recorder.plot_lr(show_moms=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class OneCycleScheduler[source]

\n", "\n", "> OneCycleScheduler(**`learn`**:[`Learner`](/basic_train.html#Learner), **`lr_max`**:`float`, **`moms`**:`Floats`=***`(0.95, 0.85)`***, **`div_factor`**:`float`=***`25.0`***, **`pct_start`**:`float`=***`0.3`***, **`final_div`**:`float`=***`None`***, **`tot_epochs`**:`int`=***`None`***, **`start_epoch`**:`int`=***`1`***) :: [`LearnerCallback`](/basic_train.html#LearnerCallback)\n", "\n", "Manage 1-Cycle style training as outlined in Leslie Smith's [paper](https://arxiv.org/pdf/1803.09820.pdf). \n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(OneCycleScheduler)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a [`Callback`](/callback.html#Callback) that handles the hyperparameters settings following the 1cycle policy for `learn`. `lr_max` should be picked with the [`lr_find`](/train.html#lr_find) test. In phase 1, the learning rates goes from `lr_max/div_factor` to `lr_max` linearly while the momentum goes from `moms[0]` to `moms[1]` linearly. In phase 2, the learning rates follows a cosine annealing from `lr_max` to 0, as the momentum goes from `moms[1]` to `moms[0]` with the same annealing." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

steps[source]

\n", "\n", "> steps(**\\*`steps_cfg`**:`StartOptEnd`)\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(OneCycleScheduler.steps, doc_string=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Build the [`Stepper`](/callback.html#Stepper) for the [`Callback`](/callback.html#Callback) according to `steps_cfg`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Callback methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_begin[source]

\n", "\n", "> on_train_begin(**`n_epochs`**:`int`, **\\*\\*`kwargs`**:`Any`)\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(OneCycleScheduler.on_train_begin, doc_string=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Initiate the parameters of a training for `n_epochs`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_batch_end[source]

\n", "\n", "> on_batch_end(**`train`**, **\\*\\*`kwargs`**:`Any`)\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(OneCycleScheduler.on_batch_end, doc_string=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Prepares the hyperparameters for the next batch." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Undocumented Methods - Methods moved below this line will intentionally be hidden" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## New Methods - Please document or move to the undocumented section" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_epoch_end[source]

\n", "\n", "> on_epoch_end(**`epoch`**, **\\*\\*`kwargs`**:`Any`)\n", "\n", "Tell Learner to stop if the cycle is finished. \n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(OneCycleScheduler.on_epoch_end)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "jekyll": { "keywords": "fastai", "summary": "Implementation of the 1cycle policy", "title": "callbacks.one_cycle" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 2 }