{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# SGDR - Stochastic Gradient Descent with Warm Restarts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial we are going to be looking at the `SGDR` or as referred to in the `timm` library - the `cosine` scheduler in little more detail with all the supporting hyperparams. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The SGDR schedule as mentioned in the [paper](https://arxiv.org/abs/1608.03983) looks like: " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"SGDR\"" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "#hide\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from timm.scheduler.cosine_lr import CosineLRScheduler\n", "from nbdev.showdoc import show_doc" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

class CosineLRScheduler[source]

\n", "\n", "> CosineLRScheduler(**`optimizer`**:`Optimizer`, **`t_initial`**:`int`, **`t_mul`**:`float`=*`1.0`*, **`lr_min`**:`float`=*`0.0`*, **`decay_rate`**:`float`=*`1.0`*, **`warmup_t`**=*`0`*, **`warmup_lr_init`**=*`0`*, **`warmup_prefix`**=*`False`*, **`cycle_limit`**=*`0`*, **`t_in_epochs`**=*`True`*, **`noise_range_t`**=*`None`*, **`noise_pct`**=*`0.67`*, **`noise_std`**=*`1.0`*, **`noise_seed`**=*`42`*, **`initialize`**=*`True`*) :: `Scheduler`\n", "\n", "Cosine decay with restarts.\n", "This is described in the paper https://arxiv.org/abs/1608.03983.\n", "\n", "Inspiration from\n", "https://github.com/allenai/allennlp/blob/master/allennlp/training/learning_rate_schedulers/cosine.py" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(CosineLRScheduler)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `CosineLRScheduler` as shown above accepts an `optimizer` and also some hyperparams which we will look into in detail below. We will first see how we can train models using the `cosine` LR scheduler by first using `timm` training docs and then look at how we can use this scheduler as standalone scheduler for our custom training scripts. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using `cosine` scheduler with `timm` training script" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To train models using the `cosine` scheduler we simply update the training script args passed by passing in `--sched cosine` parameter alongside the necessary hyperparams. In this section we will also look at how each of the hyperparams update the `cosine` scheduler. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> NOTE: In the paper this scheduler is referred to as `SGDR` but in `timm` this is referred to as `cosine` scheduler. They are both one and the same with minor implementation difference. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The training command to use `cosine` scheduler looks something like: \n", "\n", "```python \n", "python train.py ../imagenette2-320/ --sched cosine\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Args" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This way we start to use the `cosine` scheduler with all the defaults. Let's now look at the associated hyperparams and how that updates the annealing schedule. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `optimizer`" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "This is the `optimizer` that will be used for the training process. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "hidden": true }, "outputs": [], "source": [ "from timm import create_model \n", "from timm.optim import create_optimizer\n", "from types import SimpleNamespace" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "hidden": true }, "outputs": [], "source": [ "model = create_model('resnet34')\n", "\n", "args = SimpleNamespace()\n", "args.weight_decay = 0\n", "args.lr = 1e-4\n", "args.opt = 'adam' \n", "args.momentum = 0.9\n", "\n", "optimizer = create_optimizer(args, model)" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "This `optimizer` object created using `create_optimizer` is what get's passed to the `optimizer` argument. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `t_initial` " ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "The initial number of epochs. Example, 50, 100 etc. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `t_mul`" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "Defaults to 1.0. Updates the `SGDR` schedule annealing. " ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "\"SGDR_params\"" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "As shown in the image below, here **T0** is the `t_initial` hyperparameter and **Tmult** is the `t_mul` hyperparameter. One can see how updating these parameters updates the scheduler. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `lr_min` " ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "Defaults to `1e-5`. The minimum learning rate to use during the scheduling. The learning rate does not ever go below this value. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `decay_rate`" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "When `decay_rate` > 0 and <1., at every restart the learning rate is decayed by new learning rate which equals `lr * decay_rate`. So if `decay_rate=0.5`, then in that case, the new learning rate becomes half the initial `lr`. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "hidden": true }, "outputs": [], "source": [ "from matplotlib import pyplot as plt\n", "\n", "def get_lr_per_epoch(scheduler, num_epoch):\n", " lr_per_epoch = []\n", " for epoch in range(num_epoch):\n", " lr_per_epoch.append(scheduler.get_epoch_values(epoch))\n", " return lr_per_epoch" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "#### `decay_rate=1.` or no decay" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "hidden": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxY0lEQVR4nO3dd3xUVdrA8d8zk0aAJEAKIQQCJKGETkRERDEgoAisbW0v6K6LWBDrLrrvvpZdd9m1rL2AroJrWXQtWAGxICBqKAKhJYQWuiChE2bmvH/MRWMIyaRM7pTn+/nwSebOPTPP0Tt55rnn3HPFGINSSinlC4fdASillAoemjSUUkr5TJOGUkopn2nSUEop5TNNGkoppXwWYXcA/paYmGgyMjLsDkMppYLK4sWLfzDGJFXcHvJJIyMjg/z8fLvDUEqpoCIimyrbrqenlFJK+UyThlJKKZ9p0lBKKeUzTRpKKaV8pklDKaWUz3xKGiIyTETWikiRiEyq5HkRkSes55eLSO/q2opIcxGZIyKF1s9m1vYWIvK5iBwUkacqvE8fEVlhvdYTIiK177pSSqmaqjZpiIgTeBoYDnQBrhCRLhV2Gw5kWf/GAc/60HYSMNcYkwXMtR4DHAX+BNxZSTjPWq9/4r2G+dRLpZRS9cKX6zT6AkXGmGIAEXkDGAWsKrfPKGC68a6zvkhEEkQkFcioou0o4Byr/TTgC+APxphDwHwRySwfhPV6ccaYr63H04HRwMc167JvXl6wgUNlbuIaRRLfKJJW8TF0To2jcXTIX9qiGogxhle/2czhMhdxMd7jrF1SYzokNSHSqWeOVWDy5S9gGrCl3OMS4HQf9kmrpm2KMWY7gDFmu4gk+xBHSSXvcRIRGYe3IqFNmzbVvGzlXvt2M+t2HqzwupDRojG92zTjvJwUBmYl0SjKWavXV2rH/qP877srT9oeFeGgY0pT+ndowXk5LemVnoDDoWdiVWDwJWlUdrRWvHPTqfbxpa2vfH4tY8wUYApAbm5urd5v9m1nc/S4m/1Hj7P/yHE27TlMwbb9rNxaypxVO/jvkhJiIh3kdU7h2v4Z9GnbDB1iUTVR5vIA8OfRXcnrlMyPh8so2nWQgm37WV6yjxfnb+D5ecUkN41mdK80xpzRltbNYm2OWoU7X5JGCZBe7nFrYJuP+0RV0XaniKRaVUYqsMuHOFpXE0e9iol0EhPpJLlpDJnJTcnrnALAcbeHb4r3MqtgB+8t28qHy7fTo3U84wZ24PxuLTV5KJ+4PN7vM3ExEbRKaESrhEbktIpnVE9vAV165Difr9nFRyu28+L8Dbw4fwPDclpywzkd6JoWb2foKoz5cuL0OyBLRNqJSBRwOTCzwj4zgTHWLKp+QKl16qmqtjOBsdbvY4H3qgrCer0DItLPmjU1pro2/hLpdDAgK5E/j+7K13fn8cCoHPYfdXHTa0v41TMLWbxprx1hqSDjtpKG8xSnnuIbRTK6VxpTxuQy7/eDuG5AO+YV7mbEk/O5fcYydpQebchwlQJ8SBrGGBdwMzALWA3MMMYUiMh4ERlv7fYRUAwUAVOBG6tqa7WZDAwRkUJgiPUYABHZCDwKXCMiJeVmXN0AvGC9z3r8NAheE42jIxhzRgZzbz+bhy/twfbSI1z87NdMeH0pew4eszs8FcBOJI0IH8Yr0hIacff5nVkw6VzGn92BD77fzjkPf85TnxXicnv8HapSPxHvhKfQlZubaxpyldvDZS6e+7KY575YT9OYCP56UTeG5rRssPdXwWPl1lJGPDmfqWNyGdIlpUZtt+w9zN8+Xs1HK3bQo3U8j1zWk8zkJn6KVIUjEVlsjMmtuF3n9dWz2KgIbh+SzfsTBtAyPobrX1nMHTO+53CZy+7QVIBx1aDSqCi9eSzPXNWHp6/szea9h7ngia+YtnAjof4lUNlPk4afdGzZlHduPJNbzs3knaUlXPTMQjbvOWx3WCqAuD3e00qnGtPwxQXdU5l120DOzEzk3pkF3DHje44ed9dXiEqdRJOGH0VFOLj9vI68dG1ftu07wsin5zO/8Ae7w1IBwuWufaVRXnLTGF4Yk8vtQ7J5e+lWLnluIVv3HamPEJU6iSaNBnB2dhLvTxhAStMYxvzrG974drPdIakAUN3sqZpwOIRb8rJ4YUwuG384zKin5rNya2mdX1epijRpNJC2LRrz9o39OSsriUlvr+CZL4r0/HOY+2lMw1l/1/UM7pLCOzf2J9Lp4Iopi1hUvKfeXlsp0KTRoBpHRzB1TC4je7TiH5+s5cEPV+PxaOIIVz9XGvX7McxKacp/b+hPclw0Y/71LbMLdtTr66vwpkmjgUVFOHjs1z25pn8GL8zfwH3vF2jFEabqMnuqOq0SGvHW+P50To3jhleX8MlKTRyqfmjSsIHDIdx7YRfGDWzP9K838eCHqzVxhKH6mD1VlWaNo3j1utPp3jqeCa8v4bM1O/3yPiq8aNKwiYhw9/BOP1UcD89eq4kjzPiz0jihSXQEL1/bl04t4xj/7yXMW7fbb++lwoMmDRuJeCuOK/q24enP1zNlXrHdIakGVJ+zp6oS3yiSV37blw5JTRj3Sj5LN//o1/dToU2Ths1EhAdHd2VE91T+9vEaZn7v14V7VQD5+ToN/38ME2KjeOW3fUmJi+G30/LZ+MMhv7+nCk2aNAKAwyE8clkP+rZrzp0zvtdpkmHip0qjHqfcViWxSTQvX9sXgGte+lYX1FS1okkjQERHOJn6P7m0aRHLuOn5FO06YHdIys8aYkyjonaJjXlhbC7bS4/y22n5uuSIqjFNGgEkPjaSl689jagIJ9dNy6f0yHG7Q1J+5O/ZU6fSu00znriiF9+X7OPut1foBAxVI5o0AkzrZrE8d3Vvtu47wi2vL/3pFIYKPXZUGicMzWnJ7YOzeWfpVl6cv6HB318FL00aASg3ozn3j+zKl+t289CstXaHo/ykoWZPncpNgzIZltOSv360WhfSVD7TpBGgrjy9DVed3obnvlzPB8t1RlUo+rnSsOdjeGICRlZyU25+fQlb9urS/ap6mjQC2L0X5tC7TQKT/rtCp0iGILsrDfCuhzZlTB/cbsPNry+lzKW3jlVV06QRwKIiHDx5ZW8inMJNry3RmS4hpr7up1FXbVs05h+XdOf7Lfv4xydrbI1FBT5NGgEuLaERj1zag4Jt+3nww9V2h6PqkdvjQcR7mshuw7ulMvaMtrwwfwNzVukaVerUNGkEgbzOKYwb2J5XFm3ioxXb7Q5H1ROXx9heZZR3zwWd6ZoWx51vfq93/lOnpEkjSNw1tCM90hO4++0V7Cg9anc4qh64PcbW8YyKoiOcPH1lb1xuD3fMWKb3elGV0qQRJCKd3vtwlLk83Pnm9/qBDgHeSiOwPoJtWzTm3pE5LCreywvzdQFNdbLAOmJVldolNuZPI7owv+gHXl640e5wVB0FWqVxwqV9WjM0J4WHZ61j9fb9doejAowmjSBzRd908jolM/mTNazbqetTBTOXxxNQYxoniAh/u6g78bGR3PrGMp21p35Bk0aQERH+fkl3mkZHcPuMZbjcOq8+WAVqpQHQvHEU/7ikO2t3HuCxTwvtDkcFEE0aQSixSTR/Gd2VlVv387zeuCloudyBNXuqokEdk7n8tHSmzFvP91v22R2OChCaNILU8G6pnN+tJY9/WkihnqYKSm6PabB7adTWPRd0JrlpDHe99T3HXHqaSmnSCGoPjOpK42gnd721XFfDDUKBOHuqoriYSP52UTfW7TzI058V2R2OCgCBfcSqKiU2iea+kTks27KPf+ny1kEnkMc0yhvUKZmLeqfxzBfrKdhWanc4ymaaNILcyB6tGNw5hUfmrNVVSoNMoM6eqsy9I3JIiI3i7rdXaFUb5jRpBDkR4YFROThF+L/3Vupd2IJIsFQa4L2r5P9d2IXlJaVM/3qj3eEoG2nSCAGtEhpxx3kd+Xztbj7UtamCRqCtPVWdC7uncnZ2Eg/PWsv2Ul2bKlxp0ggRY/tn0C0tnvvfX6X3Fg8SwVRpgLeq/cvorriN4d73CuwOR9lEk0aIcDqEv13UjT0Hj+k9EYKE9zqN4PoIpjePZWJeNrNX7WRWwQ67w1E28OmIFZFhIrJWRIpEZFIlz4uIPGE9v1xEelfXVkSai8gcESm0fjYr99zd1v5rRWRoue1XiMgK6z0+EZHE2nc99HRNi2ds/wxe+3Yzy0v22R2OqkawVRonXHdWOzqmNOWB91dxpEyv3Qg31SYNEXECTwPDgS7AFSLSpcJuw4Es69844Fkf2k4C5hpjsoC51mOs5y8HcoBhwDMi4hSRCOBxYJAxpjuwHLi5lv0OWbcNyaZF42j+770CXQk3wLk8HiIC/OK+ykQ6HTwwKoet+47wzBd67Ua48aXS6AsUGWOKjTFlwBvAqAr7jAKmG69FQIKIpFbTdhQwzfp9GjC63PY3jDHHjDEbgCLrdcT611hEBIgDttW4xyEuLiaSe87vxLIt+3hz8Ra7w1FVCNZKA+D09i0Y1bMVz39ZrPevDzO+JI00oPxfnxJrmy/7VNU2xRizHcD6mVzVaxljjgM3ACvwJosuwIuVBSwi40QkX0Tyd+/e7UMXQ8uveqVxWkYz/v7JWvYdLrM7HHUKwTZ7qqJ7zu9MpFN44INVdoeiGpAvSaOyo7rieY9T7eNLW5/eT0Qi8SaNXkArvKen7q7sBYwxU4wxucaY3KSkpGreLvSICPeP7Mq+w2U8Mnud3eGoUwjmSgMgJS6GWwdn89maXXyq9xUPG74kjRIgvdzj1px8WuhU+1TVdqd1Cgvr565qXqsngDFmvfFewTYD6O9D/GGpS6s4/qdfW179ZpPeSCdABcPaU9W55swMMpOb8OcPV+mChmHClyP2OyBLRNqJSBTeQeqZFfaZCYyxZlH1A0qtU05VtZ0JjLV+Hwu8V2775SISLSLt8A6ufwtsBbqIyInSYQiwuob9DSu3DckmrlEkf/5glV4pHoDcHoMjiCsN8A6K/2lEFzbtOczLCzbaHY5qANUmDWOMC+8spVl4/0jPMMYUiMh4ERlv7fYRUIx30HoqcGNVba02k4EhIlKINwFMttoU4K0iVgGfADcZY9zGmG3A/cA8EVmOt/L4a926H9oSYqO4fUg2C9fvYbaePgg4wbT2VFXOzk4ir1MyT35WxO4Dx+wOR/mZhPo30NzcXJOfn293GLZxuT2c/8RXHD3uYc7tA4mOcNodkrL0/9tc+mcm8vClPewOpc6Kdx9k6GPzuLh3ayZf3N3ucFQ9EJHFxpjcituD+4SqqlaEdfpg897D/Gv+RrvDUeUE++yp8tonNWHsGRn8J38LK7fq8umhTJNGGDgrK4nBnVN4+vMifjiopw8CRbDPnqpoQl4WzWOjdAwtxGnSCBP3nN+Jo8fdPPapTsENFG4TOpUGQHyjSG4dks03G/by6epd1TdQQUmTRphon9SEq05vw+vfbqFol95TPBC43QZnkE+5rejy09LpkNSYv328muNuj93hKD8IrSNWVemWvCxiI51M/lhXwQ0ELo8JyrWnqhLpdHD38M4U7z7EG99utjsc5QeaNMJIiybR3Dgok09X72Lh+h/sDifshdqYxgl5nZPp1745//y0kP1H9d4uoUaTRpi59swM0hIa8dePVusquDYLles0KhIR/nh+F/YeKuO5L9bbHY6qZ5o0wkxMpJM7h2azcut+3l+uiwTbxeMxeAwhWWkAdGsdz+ierXhx/gZ2lB61OxxVjzRphKFRPdLonBrHw7PXUubSwUo7uK0pqaFYaZxwx3kd8RijM/ZCjCaNMORwCH8Y1pEte4/w2jeb7A4nLLmtU4OhNnuqvPTmsVzdry0z8nXGXigJ3SNWVens7CTOaN+CJz8r4uAxl93hhB2XJ/QrDYCbB2USGxXBQ7PW2h2KqieaNMKUiDBpeCf2HCpj6rxiu8MJO273iUojtJNGiybRXD+wPbMKdrJ40492h6PqgSaNMNYjPYELuqUy9atiXZ20gbk83rGkULtOozK/PasdiU2i+fvHa3R5kRCgSSPM3XFeNsdcHp7+vMjuUMLKz2MaoZ80YqMimDg4i2837uWLdeF3++VQo0kjzLVPasKlfVrz2jebKfnxsN3hhI1wGdM44de56aQ3b8RDn6zV64OCnCYNxcTBWSDw+KeFdocSNsJh9lR5UREObh+Szart+/lo5Xa7w1F1EB5HrKpSanwjxvRry3+XlOjUyAYSbpUGwMgeaXRMacqjs9fh0sUMg5YmDQXADed0oFGkk0dm64VYDcFtDYSHw5jGCU6HcMd52RT/cIi3FpfYHY6qJU0aCvBOjbzurPZ8vHIHK0r0zmv+Fo6VBsCQLin0apPA43MLOXrcbXc4qhY0aaifXHdWOxJiI3l0jl6I5W+uMLlOoyIR4a7zOrK99KgunR6kNGmonzSNieT6gR34fO1uFm/aa3c4Ie3EQHg4XKdRUf/MRPq1b85Tn6/nSJlWG8FGk4b6hbH925LYJErHNvzMFWazpyq647yO/HDwGNO/3mh3KKqGwvOIVacUGxXBjedksnD9HhYW6Y2a/MUdpmMaJ5yW0Zyzs5N47sv1HNAbNQUVTRrqJFee3oaWcTE8MmedLvvgJ64wnD1V0R3nZfPj4eO8tGCj3aGoGtCkoU4SE+lkQl4mizf9qMs++Em4VxoA3VsncF6XFKZ+VUzpYa02goUmDVWpS/ukk5bQiMe02vALVxitPVWV24Zkc+Coixfn60rLwUKThqpUVISDW/Iy+b6klM/W7LI7nJBzYmn0iDAdCD+hc2oc53dryb8WbGTf4TK7w1E+CO8jVlXpot6tadM8ln9+qtVGfdNK42cT87I5VOZi6ldabQQDTRrqlCKdDiacm8nKrfuZs2qn3eGElHC+TqOiji2bMqJ7K15asJG9h7TaCHSaNFSVftUrjYwWsfzz00Jd0roe6eypX5qYl8XR426en7fe7lBUNTRpqCpFOB1MHJzF6u37mVWww+5wQobOnvqlzOQmjOqZxvSFm/jhoN5FMpBp0lDVGtkjjfaJjXl8rlYb9UXHNE424dxMjrncTNF71gc0TRqqWk6HcEteFmt2HNBqo578XGnoR/CE9klNGN0zjelfb9RqI4DpEat8cmGPVlpt1COtNCp387mZlLk8Wm0EME0ayiflq41PtNqoM7d15zod0/il8tXG7gNabQQin5KGiAwTkbUiUiQikyp5XkTkCev55SLSu7q2ItJcROaISKH1s1m55+629l8rIkPLbY8SkSkisk5E1ojIxbXvuqqpC3u0on1SYx7XmVR19lOloVNuT/JztaEzqQJRtUlDRJzA08BwoAtwhYh0qbDbcCDL+jcOeNaHtpOAucaYLGCu9Rjr+cuBHGAY8Iz1OgB/BHYZY7Kt1/uyFn1WteR0CBPzsli78wAfr9Rqoy509tSpnag2Xlm0SauNAORLpdEXKDLGFBtjyoA3gFEV9hkFTDdei4AEEUmtpu0oYJr1+zRgdLntbxhjjhljNgBF1usA/Ab4G4AxxmOM0bW7G9iI7t5q4wkd26gTHdOomlYbgcuXpJEGbCn3uMTa5ss+VbVNMcZsB7B+Jlf1WiKSYD3+s4gsEZE3RSSlsoBFZJyI5ItI/u7dukprfXI6hFvO9VYbOpOq9nT2VNXaJ3mv23hlkV63EWh8OWIr+ypU8Svmqfbxpa2v7xcBtAYWGGN6A18DD1f2AsaYKcaYXGNMblJSUjVvp2pKZ1LV3YlKQwuNUztRbUzVmVQBxZekUQKkl3vcGtjm4z5Vtd1pncLC+nliKdVTtdkDHAbesba/CfRGNTinQ7j53EzW7DjAbF2TqlbcHg8RDkFEs8apdEhqwoU9WjH9603s0WojYPiSNL4DskSknYhE4R2knllhn5nAGGsWVT+g1DrlVFXbmcBY6/exwHvltl8uItEi0g7v4Pq3xrvM6vvAOdZ+ecCqmnVX1ZeRPVqR0SKWJ+YW6gq4teDyGB3P8MGEczM56nIz9asNdoeiLNUmDWOMC7gZmAWsBmYYYwpEZLyIjLd2+wgoxjtoPRW4saq2VpvJwBARKQSGWI+xnp+BNyF8AtxkjHFbbf4A3Cciy4H/Ae6oQ99VHUQ4HUw4N4tV23UF3Npwu43OnPJBZnJTLuzeiulf6wq4gUJC/Vtibm6uyc/PtzuMkORyexj86Jc0jo7ggwkD9FRLDdw3s4C3l5Sw/L6h1e8c5op2HWDIP+dxw9kd+P2wTnaHEzZEZLExJrfidp26oWotwungpkGZFGzbr3f3qyG3xxDh1I+fLzKTm3JBt1SmLdS7+wUCPWpVnYzulUZ680Y8rmMbNaJjGjUz4dwsDpW5eXG+jm3YTZOGqpNIp4ObB2WyvKSUL9bqNTG+OjF7SvmmY8umnN+tJS8v2Ejp4eN2hxPWNGmoOvtVr9akJTTiMa02fKaVRs1NODeLA8dcvLhAqw07adJQdRYV4R3b+H7LPr5cp9WGL9wenT1VU51T4xiak8JLCzZQekSrDbto0lD14pI+rWkVH6PXbfhIK43auSUviwNHXby8YKPdoYQtTRqqXkRFOLhhUCZLNu9jfpGuI1kdj8foulO1kNMqnsGdU3hxfjH7j2q1YQc9alW9uSy3NanxMTz+qVYb1dFKo/Ym5mWx/6iL6Qs32h1KWNKkoepNdISTG87pQP6mH/l6/R67wwlobk0atdatdTx5nZJ5Yf4GDh5z2R1O2NGkoerVZbnppMRF89jcQrtDCWhaadTNLXlZ7Dt8nGlabTQ4TRqqXsVEOhl/dge+3bBXq40q6HUaddMjPYFzOibxwlfFHNJqo0Fp0lD17oq+bUhqGs3jc9fZHUrAcrm10qiriXlZ/Hj4ONO/3mR3KGFFk4aqdyeqjUXFe/mmWKuNynjXntKkURe92jRjYHYSU7XaaFCaNJRfXHV6GxKbRPO4jm1UyjumoR+/upqYl8XeQ2X8e5FWGw1Fj1rlF95qoz0L1+/hu4177Q4n4OgV4fWjT9tmnJWVyJR5xRwu02qjIWjSUH5z1eltSWwSxeOfarVRkc6eqj8T87LYc6iMVxdttjuUsKBJQ/lNoygn4wa2Z37RD+RrtfELOnuq/uRmNOfMzBY8P289R8rc1TdQdaJJQ/nV1f3a0qJxlI5tVKCVRv2amJfNDwfLePUbHdvwN00ayq9ioyK4/uz2fFWo1UZ5OqZRv/q281Ybz32p1Ya/adJQfqfVxsm812nox68+abXRMPSoVX6n1cbJtNKof1ptNAxNGqpBXN3PO5PqMZ1JBVhjGnpxX727dbC32tDrNvxHk4ZqELFREVw/sAPzi37Q6zbQ2VP+clpGcwZkJvL8vPV63YafaNJQDeaqfm2sakPXpNLZU/4zcXCWVht+pElDNZjYqAjGn92BBUV7wn5NKh3T8J/TMppzVlYiz3+pa1L5gyYN1aCuOr0tSU2j+WeYVxu69pR/3To4mz2HynQFXD/Qo1Y1qEZRTm48x7sC7sL14Xsvca00/KtP22ac0zGJKfPW69396pkmDdXgrujbxnt3vznheS9xY4ze7rUB3Do4mx/17n71TpOGanAxkU5uGpTJtxv3sqAo/MY23B5votRKw796pieQ1ymZKfOK2X/0uN3hhAxNGsoWvz4tndT4GB6dszbsqg2XlTT0Og3/u21INqVHjvPS/I12hxIyNGkoW0RHOLn53EyWbN7HF+t22x1Og9JKo+F0TYvnvC4pvPBVMfsOl9kdTkjQpKFsc2mfdNKbN+KR2eFVbfxUaejsqQZx+3nZHCxzMfWrYrtDCQl61CrbREU4mJiXzcqt+5lVsNPucBqMVhoNq1PLOEZ0b8VLCzbyw8FjdocT9DRpKFuN7tmK9kmNeXTO2p/+mIY6l8cDoLOnGtCtg7M4etzNc1+stzuUoKdJQ9kqwungtsHZrNt5kA+Wb7M7nAahlUbD65DUhIt6t+aVRZvYUXrU7nCCmk9JQ0SGichaESkSkUmVPC8i8oT1/HIR6V1dWxFpLiJzRKTQ+tms3HN3W/uvFZGhlbzfTBFZWfPuqkB0QbdUOrVsyj/nrOO422N3OH7ncp8Y09Ck0ZAm5mXh9hie/ExXWq6LapOGiDiBp4HhQBfgChHpUmG34UCW9W8c8KwPbScBc40xWcBc6zHW85cDOcAw4BnrdU7EcxFwsDadVYHJ4RBuH5LNxj2H+e/iErvD8bufKg2dctug0pvH8uvT0vnPd1vYvOew3eEELV8qjb5AkTGm2BhTBrwBjKqwzyhguvFaBCSISGo1bUcB06zfpwGjy21/wxhzzBizASiyXgcRaQLcDvyl5l1VgWxIlxR6tUng8bmFHD0e2jfQ0dlT9rklL4sIp4T92md14ctRmwZsKfe4xNrmyz5VtU0xxmwHsH4m+/B+fwYeAar8miAi40QkX0Tyd+8Or2sAgpWIcNfQjmwvPRryS1rrmIZ9UuJiGNs/g3eXbWXtjgN2hxOUfEkalR3ZFae5nGofX9r69H4i0hPINMa8U017jDFTjDG5xpjcpKSk6nZXAaJ/h0QGZCbyzBehvciczp6y1/iBHWgSFcHDs9faHUpQ8iVplADp5R63BipOcznVPlW13WmdwsL6uaua1zoD6CMiG4H5QLaIfOFD/CqI3DW0I3sPlfHiVxvsDsVvtNKwV7PGUYwb2J45q3aydPOPdocTdHxJGt8BWSLSTkSi8A5Sz6ywz0xgjDWLqh9Qap1yqqrtTGCs9ftY4L1y2y8XkWgRaYd3cP1bY8yzxphWxpgMYACwzhhzTi36rAJYj/QEhuW0ZOpXxewJ0Quxfh7T0KRhl98MaEdikyj+/smasFqNoD5UmzSMMS7gZmAWsBqYYYwpEJHxIjLe2u0joBjvoPVU4Maq2lptJgNDRKQQGGI9xnp+BrAK+AS4yRgT2iOj6hfuHJrN4TIXT31eZHcofvFzpaED4XZpHB3BzYMyWVS8ly/DbO2zuorwZSdjzEd4E0P5bc+V+90AN/na1tq+B8g7RZsHgQeriGcj0NWH0FUQykxuymW56fx70SZ+c2Y70pvH2h1SvdLrNALDlae35cUFG5j88RoGZiXh0P8fPtGvOiog3To4G4cIj4TgYKVepxEYoiIc3HleR9bsOMB732+1O5ygoUlDBaSW8TH8ZkA73l22jZVbS+0Op17p7KnAcWH3VuS0iuPhWes45tKz4L7QpKEC1vizOxDfKJK/f7LG7lDqlc6eChwOhzBpeCe27jvCK1+H9vVB9UWThgpY8Y0iuXlQJl8V/sBXhaEzWKmzpwLLWVlJDMhM5KnPiyg9oreFrY4mDRXQxvRvS+tmjXjww9Uhs3S6zp4KPHef34nSI8d5JkRn7NUnPWpVQIuOcPKHYZ1Ys+MAby8JjcUMtdIIPDmt4rm4d2teWrCRLXt1McOqaNJQAW9E91R6pifw8Oy1HC4L/uVF3NZAuI5pBJY7zsvG4YB/zAq9GXv1SZOGCngiwh8v6MzO/cd4IQSWF9HrNAJTanwjfndWe97/fpsuL1IFTRoqKJyW0ZxhOS157sv17Nof3Hde0+s0Atf1Z3cgsUkUD364WpcXOQVNGipoTBreieNuT9CfPtAxjcDVJDqC24d0JH/Tj3ywfLvd4QQkTRoqaGQkNubaM9vx1uISlpfsszucWvMYnT0VyH59WjqdU+OY/PGakL8hWG3oUauCys3nZtKicRQPvL8qaE8f6JhGYHM6hP8b0YWt+44wZV6x3eEEHE0aKqjExURy51Dv6YP3g/T0gV4RHvjO6NCCYTktefaL9ewoDe4xtPqmSUMFncty0+mSGsfkj1ZzpCz4Th/omEZwuOf8zriNCbllbOpKk4YKOk6HcO+FXdhWepRnvwi+K3j1Oo3g0KZFLL87qx3vLN1K/sa9docTMDRpqKB0evsWjOzRiufmFbNpzyG7w6kRrTSCx02DMkmNj+FP7xXgcnvsDicgaNJQQeuPF3Qm0iHc//4qu0OpEbfH4HQIIpo0Al1sVAR/GtGF1dv38+o3m+0OJyBo0lBBKyUuhlsHZ/PZml18umqn3eH4zGUlDRUchndtyYDMRB6evZYfQvS+9TWhSUMFtWvOzCAruQn3f1AQNHPq3R6j4xlBRES4b2QOR4+7+fvHOiiuSUMFtUing/tH5bBl7xGeDpJlrV1urTSCTWZyE34zoB1vLi7h2w3hPSiuSUMFvf4dErmoVxrPfbmewp0H7A6nWm6PR5NGEJqYl0VaQiPueWcFZa7wHRTXpKFCwh8v6Ezj6AjueWcFngC/WZNLT08FpdioCP4yuitFuw4yZd56u8OxjSYNFRJaNInmnvM7893GH5mRv8XucKrk1oHwoDWoUzIXdE/lic+K2PBDcE31ri+aNFTIuLRPa/q2a85fP1rN7gOBO8vFW2noRy9Y3TuiC9FOB//77oqgXf+sLvTIVSFDRPjrr7px9LiH+2YW2B3OKWmlEdyS42L4w/BOLCjaw5v5oXEL4prQpKFCSmZyEyYOzuLDFdv5eEVgLmioYxrB78q+bTi9XXP+/OGqsFvQUJOGCjnjBrana1ocf3pvJXsPldkdzkl09lTwcziEv1/cneNuD/e8E16nqTRpqJAT6XTw0CU9KD1ynAfeD7zTVHqdRmjISGzMXUM78dmaXby7bKvd4TQYTRoqJHVOjeOmQZm8u2wbswt22B3OL7g9Ru8PHiKu6Z9Bn7bNuG/mKnYG+b3rfaVJQ4WsG8/JpEtqHHe/vSKgZlN5157Sj14ocDqEhy7pzjGXm7veWh4Wp6n0yFUhKyrCwWOX9+TAMRd/+G/gfKB17anQ0j6pCX88vzPz1u1m+teb7A7H7zRpqJCWndKUScO8551f+zYwlrZ26UB4yLm6X1vO6ZjEXz9aTdGuwF/Kpi40aaiQd03/DAZkJvKXD1ZTvPug3eFopRGCRIR/XNyd2Cgnt/5nWUivTaVJQ4U8h0N4+NIeREc6mPD6UtuXUNf7aYSm5LgYJl/cnZVb9/PQrNBdQl2ThgoLLeNjePiSHhRs289fP1ptayxaaYSuoTktGXNGW6Z+tYE5QXRjsJrwKWmIyDARWSsiRSIyqZLnRUSesJ5fLiK9q2srIs1FZI6IFFo/m5V77m5r/7UiMtTaFisiH4rIGhEpEJHJdeu6CjeDu6Twu7PaMf3rTXxk49Xi3us09PtaqLrn/M50TYvjzje/p+THw3aHU++qPXJFxAk8DQwHugBXiEiXCrsNB7Ksf+OAZ31oOwmYa4zJAuZaj7GevxzIAYYBz1ivA/CwMaYT0As4U0SG16bTKnz9flgneqYn8Ie3lrNpjz2rlGqlEdpiIp08dUVv3B7DhNeXctwdWuMbvnzd6QsUGWOKjTFlwBvAqAr7jAKmG69FQIKIpFbTdhQwzfp9GjC63PY3jDHHjDEbgCKgrzHmsDHmcwDrtZYArWveZRXOIp0OnrqyFw6HcP0rizlc5mrwGFweD069uC+kZSQ2ZvLF3Vi6eR9/+WCV3eHUK1+SRhpQ/gYFJdY2X/apqm2KMWY7gPUz2df3E5EE4EK8FcpJRGSciOSLSP7u3bur6psKQ62bxfLEFb1Yt/MAd73Z8NdvaKURHkZ0b8XvzmrHtK83MeO7wL7HS034kjQqO7orfspOtY8vbWv0fiISAbwOPGGMKa7sBYwxU4wxucaY3KSkpGreToWjs7OT+MOwTny4YjvPfNGwd2HT2VPh4w/DOnFWViL/++5Klmz+0e5w6oUvSaMESC/3uDWwzcd9qmq70zqFhfVzl4/vNwUoNMY85kPsSp3SuIHtGdmjFQ/PXstnaxpupotWGuEjwungySt60TI+hvGvLA6JZdR9SRrfAVki0k5EovAOUs+ssM9MYIw1i6ofUGqdcqqq7UxgrPX7WOC9ctsvF5FoEWmHd3D9WwAR+QsQD9xa864q9Usi3uWtc1rFMeG1pazcWtog76trT4WXhNgopo7J5dAxF9e+/B0Hjh63O6Q6qfbINca4gJuBWcBqYIYxpkBExovIeGu3j4BivIPWU4Ebq2prtZkMDBGRQmCI9Rjr+RnAKuAT4CZjjFtEWgN/xDsLa4mILBOR6+r6H0CFt0ZRTl4cexoJsVFc+/J3bNnr/ymSWmmEn44tm/LM1X1Yt/MAN766JKhnVEmgLOLmL7m5uSY/P9/uMFSAK9x5gIufXUhyXAxvjT+DhNgov71X9/tmcVHv1tw3Msdv76EC04z8Lfz+reVc0qc1D13SHZHA/fIgIouNMbkVt2uNrBSQldKUKWNy2bznML+dls+hY/6biquVRvi6LDediXlZvLW4hMmfrAmYlZdrQpOGUpZ+7Vvw+OU9WbZlH7+d9h1HyvyzRpXLY/Q6jTB26+Asrjq9Dc9/WcxjnxbaHU6NadJQqpzh3VJ59LIefLNhL+NeyffL4oZaaYQ3EeHPo7pyaZ/WPD63kKc/L7I7pBqJsDsApQLNqJ5pHHN5+P1by7n+lcU8d3UfGkU5q2/oA2OMzp5SOBzC5Iu7c9zt4aFZawG48ZwOAT3GcYIeuUpV4rLcdCZf1I15hbu5+sVv2He4rF5e12OdwtZKQzmtJftH9WzFQ7PW8uCHq/F4An+MQ5OGUqdwed82PH1lb1aUlPLr5xfVy4VZLo93qqVeEa7Ae/HfPy/rydgz2vLC/A3c+db3AT8dV5OGUlU4v1sqL117GiU/HuaiZxbU+QJAt/VNUisNdYLDIdw3MofbBmfz9pKt/Obl7+qtsvUHTRpKVePMzET+c/0ZAFz87ELeXbq11q/lspKGVhqqPBFh4uAs/n5xN74p3svIpxawZsd+u8OqlCYNpXzQNS2emRMG0CM9gVv/s4wH3l/FMVfNZ1a53VppqFP79WlteH1cP44ed3PRMwt5b1ntv6D4iyYNpXyU2CSaV687nWv6Z/CvBRsY+eQCCrbV7HTVT5WGUz96qnJ92jbj/QkD6Jwax8Q3lnHza0v48VDgnK7SI1epGoh0OrhvZA4vXXMaPx4uY9RTC3hibqHP13PomIbyRUpcDP8Z14+7hnZkVsEOzntsHrMKdgTEFeSaNJSqhUGdkpl160CGdW3Jo3PWMeSfX/Lh8u3Vfqh19pTyVYTTwU2DMnn3pjNp0TiK619ZzFUvfFPj6ra+adJQqpaaNY7iqSt78+/fnk7jqAhuem0JFz+7kNkFO36qKCrSSkPVVE6reN6fMID7R+awavt+Rjw5n9v/s4xV2+wZKNcrwpWqowFZiXx4y1m8mb+FJ+YWMu6VxbRtEcvYMzIY0T2V5LiYn/Z16+wpVQuRTgdj+2cwumcaT31eyL8XbebtpVs5o30LxpzRlnM6JtfbqgXV0aXRlapHLreHTwp28OL8DSzdvA8R6JWewJAuLenVJoFIp4OLn13I01f25oLuqXaHq4JU6eHjvP7dZqYt3Mj20qPERDoYmJXEuZ2S6ZoWT3ZKU6Ii6nYi6VRLo2vSUMpP1u44wKyCHcwq2EFBhVMJz13dm2FdNWmounG5PXyzYS+zCnYwu2AnO/Z7Vy2IdApZyU15/Xf9iI+NrNVra9JQyka7DhylYNt+Vm3bz9Z9R7jzvI40b+y/Gz2p8OPxGDbuOUTBtv0UbNtP8e6DPP8/fWq9CKImDaWUUj7TO/cppZSqM00aSimlfKZJQymllM80aSillPKZJg2llFI+06ShlFLKZ5o0lFJK+UyThlJKKZ+F/MV9IrIb2FTL5onAD/UYTjAIxz5DePY7HPsM4dnv2vS5rTEmqeLGkE8adSEi+ZVdERnKwrHPEJ79Dsc+Q3j2uz77rKenlFJK+UyThlJKKZ9p0qjaFLsDsEE49hnCs9/h2GcIz37XW591TEMppZTPtNJQSinlM00aSimlfKZJoxIiMkxE1opIkYhMsjsefxGRdBH5XERWi0iBiEy0tjcXkTkiUmj9bGZ3rPVNRJwislREPrAeh0OfE0TkLRFZY/0/PyPU+y0it1nH9koReV1EYkKxzyLyLxHZJSIry207ZT9F5G7r79taERlak/fSpFGBiDiBp4HhQBfgChHpYm9UfuMC7jDGdAb6ATdZfZ0EzDXGZAFzrcehZiKwutzjcOjz48AnxphOQA+8/Q/ZfotIGnALkGuM6Qo4gcsJzT6/DAyrsK3Sflqf8cuBHKvNM9bfPZ9o0jhZX6DIGFNsjCkD3gBG2RyTXxhjthtjlli/H8D7RyQNb3+nWbtNA0bbEqCfiEhr4ALghXKbQ73PccBA4EUAY0yZMWYfId5vIAJoJCIRQCywjRDsszFmHrC3wuZT9XMU8IYx5pgxZgNQhPfvnk80aZwsDdhS7nGJtS2kiUgG0Av4BkgxxmwHb2IBkm0MzR8eA34PeMptC/U+twd2Ay9Zp+VeEJHGhHC/jTFbgYeBzcB2oNQYM5sQ7nMFp+pnnf7GadI4mVSyLaTnJYtIE+C/wK3GmP12x+NPIjIC2GWMWWx3LA0sAugNPGuM6QUcIjROy5ySdQ5/FNAOaAU0FpGr7Y0qINTpb5wmjZOVAOnlHrfGW9KGJBGJxJswXjXGvG1t3ikiqdbzqcAuu+LzgzOBkSKyEe+px3NF5N+Edp/Be1yXGGO+sR6/hTeJhHK/BwMbjDG7jTHHgbeB/oR2n8s7VT/r9DdOk8bJvgOyRKSdiEThHTCaaXNMfiEigvcc92pjzKPlnpoJjLV+Hwu819Cx+Ysx5m5jTGtjTAbe/7efGWOuJoT7DGCM2QFsEZGO1qY8YBWh3e/NQD8RibWO9Ty843ah3OfyTtXPmcDlIhItIu2ALOBbX19UrwivhIicj/e8txP4lzHmQXsj8g8RGQB8Bazg5/P79+Ad15gBtMH7wbvUGFNxkC3oicg5wJ3GmBEi0oIQ77OI9MQ7+B8FFAPX4v3iGLL9FpH7gV/jnSm4FLgOaEKI9VlEXgfOwbsE+k7gXuBdTtFPEfkj8Bu8/11uNcZ87PN7adJQSinlKz09pZRSymeaNJRSSvlMk4ZSSimfadJQSinlM00aSimlfKZJQymllM80aSillPLZ/wP2smbAX+gL3QAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch, decay_rate=1., lr_min=1e-5)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*2)\n", "\n", "plt.plot([i for i in range(num_epoch*2)], lr_per_epoch);" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "#### `decay_rate=0.5` " ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "hidden": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsZElEQVR4nO3dd3wVVf7/8dcnnQSSACmEhECAEAggCKEjgoqAZSO6KtjAsgiia3fR/fl117KLq+4ii6KoCOoqgjUWUEFRqhB6J6EIoYRe05Pz++MObDakXEKSueXzfDx43Ny5c+Z+DlzuOzNnZo4YY1BKKaWc4WN3AUoppdyHhoZSSimnaWgopZRymoaGUkopp2loKKWUcpqf3QXUtoiICNOiRQu7y1BKKbeyYsWKQ8aYyLLLPT40WrRoQXp6ut1lKKWUWxGR38pbroenlFJKOU1DQymllNM0NJRSSjlNQ0MppZTTNDSUUko5zanQEJHBIrJFRDJFZFw5r4uITLReXysiXapqKyKNROQHEcmwHhtayxuLyE8ickpEJpV5n64iss7a1kQRkep3XSml1PmqMjRExBd4DRgCJAPDRSS5zGpDgETrzyhgshNtxwHzjDGJwDzrOUAe8DTwWDnlTLa2f+a9BjvVS6WUUjXCmes0ugOZxpjtACIyA0gFNpZaJxV4zzjus75URMJFJAZoUUnbVKC/1X46MB/4kzHmNLBQRFqXLsLaXqgxZon1/D3gOmD2+XXZOdMW7eB0QTENgvwIDfInJiyItk1CCQv2r423U0opt+BMaMQCu0s9zwJ6OLFObBVto40x+wCMMftEJMqJOrLKeY9ziMgoHHskxMfHV7HZ8n24bBdbs0+ds7xpWBCd48MZkBTFgLZRRNQPrNb2lVLKHTkTGuWNG5SduamidZxp6yynt2WMmQJMAUhJSanW+33/8KXkFRZzMq+IE3mF7D6Sw+b9J9m07wRLtx/m23X7EYFuLRpxW8/mDG7fhAA/Pa9AKeXZnAmNLKBZqedxwF4n1wmopG22iMRYexkxwAEn6oiroo4aFeTvS5C/L5ENAmkVWZ/+SY6dIWMMG/aeYO6mbD5ftYc/frSKiPqB3NYznrv6JhAapIewlFKeyZlfjZcDiSKSICIBwDAgrcw6acAd1llUPYHj1qGnytqmASOsn0cAX1ZWhLW9kyLS0zpr6o6q2tQWEaFDbBgPXdGGnx7tz7Q7u9EpLowJczO49B8/8faC7eQVFttRmlJK1SpxZo5wEbkKmAD4AlONMS+IyGgAY8wb1pf4JBxnM+UAdxpj0itqay1vDMwE4oFdwI3GmCPWazuBUBx7KseAK40xG0UkBZgG1MMxAP6AqaIDKSkppq5uWLh+z3FenLOZBRmHiA2vx/NDOzAgqaqhGqWUcj0issIYk3LOcmdCw53VZWicsSjzEM+kbSDzwCmuvziWp69JpmFIQJ3WoJRSF6Ki0NCR21rQp3UE3/yxL3+8rDVpa/Yy8F8/syDjoN1lKaXUBdPQqCWBfr48cmUSXz3Ql0YhAdwxdRn/+mErxSWevWenlPJsGhq1rF1MKF+M7cPQi2N5dV4GI6Yu41hOgd1lKaVUtWho1IHgAD9eubET46/vyLIdR7h+8mJ2Hc6xuyyllDpvGhp1REQY1j2eD+7pweFTBQx9fRGrdh21uyyllDovGhp1rHtCIz67rzchgX4Mm7KUn7ZUdU2jUkq5Dg0NG7SKrM9n9/WmdVR97n1vBT9szLa7JKWUcoqGhk0i6gfy4T09aRfTgDEfrGD2un12l6SUUlXS0LBRWLA/79/Tg07Nwrn/o1UaHEopl6ehYbPQIH+m39WdTnFhPDhjNQszDtldklJKVUhDwwXUD/Tj3ZHdaRkZwqj30/WsKqWUy9LQcBFhwf68d1d3IhsEMvLd5WRkn7S7JKWUOoeGhguJCg3ig7t7EODnw8h3l3PwZL7dJSml1P/Q0HAxzRoF886IFA6fzmfU++k6L4dSyqVoaLigi+LCmXDzxazefYxHZ62hRG9yqJRyERoaLmpwhyaMG9yWb9bu49V5GXaXo5RSgIaGSxvVryU3dInj1XkZzNukV40rpeynoeHCRIQXhnagQ2woD328mh2HTttdklLKy2louLggf1/euK0rfj7C6PdXcDq/yO6SlFJeTEPDDcQ1DGbi8IvJOHCSpz5fh6fP666Ucl0aGm7iksRIHr6iDV+u3sus9Cy7y1FKeSkNDTdy34DW9G7VmP9LW69XjCulbKGh4UZ8fYQJN3emfqAf93+4Si/8U0rVOQ0NNxMVGsQ/b+rMluyTPPv1RrvLUUp5GQ0NN9SvTST39mvJh7/u4sfNev2GUqruaGi4qUeubEPbJg144pN1HD6lNzZUStUNDQ03Fejny4RhnTmRW8iTn+lpuEqpuqGh4cbaNgnl8UFJfL8xW0/DVUrVCQ0NN3d33wR6JDTiua83su94rt3lKKU8nIaGm/PxEV76fSeKSoweplJK1ToNDQ8Q3ziYJwYnMX/LQT5ZoYeplFK1R0PDQ4zo1YLuLRyHqbJP5NldjlLKQ2loeAgfH+HF319EflEJf/58vR6mUkrVCqdCQ0QGi8gWEckUkXHlvC4iMtF6fa2IdKmqrYg0EpEfRCTDemxY6rUnrfW3iMigUsuHi8g66z3miEhE9bvueRIiQnj0yjbM3ZTNnPX77S5HKeWBqgwNEfEFXgOGAMnAcBFJLrPaECDR+jMKmOxE23HAPGNMIjDPeo71+jCgPTAYeF1EfEXED3gVGGCMuQhYC9xfzX57rLv6JJAcE8ozaRs4kVdodzlKKQ/jzJ5GdyDTGLPdGFMAzABSy6yTCrxnHJYC4SISU0XbVGC69fN04LpSy2cYY/KNMTuATGs7Yv0JEREBQoG9591jD+fn68P4Gzpy6FQ+/5iz2e5ylFIexpnQiAV2l3qeZS1zZp3K2kYbY/YBWI9RlW3LGFMIjAHW4QiLZOCd8goWkVEiki4i6QcPHnSii57lorhwRvZO4IOlu0jfecTucpRSHsSZ0JBylpUdZa1oHWfaOvV+IuKPIzQuBpriODz1ZHkbMMZMMcakGGNSIiMjq3g7z/TolW2IDa/HU5+vo7C4xO5ylFIewpnQyAKalXoex7mHhSpap7K22dYhLKzHA1VsqzOAMWabcZwaNBPo7UT9Xikk0I+//K49W7NP8e6iHXaXo5TyEM6ExnIgUUQSRCQAxyB1Wpl10oA7rLOoegLHrUNOlbVNA0ZYP48Aviy1fJiIBIpIAo7B9WXAHiBZRM7sOgwENp1nf73KwORormgXxYS5GXqLEaVUjagyNIwxRTjOUvoOx5f0TGPMBhEZLSKjrdW+BbbjGLR+C7ivsrZWm/HAQBHJwBEA4602G3DsRWwE5gBjjTHFxpi9wF+BX0RkLY49j79dWPc93zPXtqe4xPCcTtiklKoB4ukXgaWkpJj09HS7y7DVpB8zePn7rUy/qzuXtvHOMR6l1PkRkRXGmJSyy/WKcC/wh34taRkRwjNfrie/SOcVV0pVn4aGFwj08+Uvv2vPzsM5vLNQB8WVUtWnoeEl+rWJZGByNJN+zGT/cb2hoVKqejQ0vMjTVydTVGIYP1tPOlNKVY+GhheJbxzMvf1a8sXqvXqluFKqWjQ0vMyY/q2ICQvimbQNFJd49plzSqmap6HhZYID/HjyqnZs2HuCT3WWP6XUedLQ8ELXXhRDl/hw/vHdFk7lF9ldjlLKjWhoeCER4f+ubc+hU/lMnp9pdzlKKTeioeGlOjcLZ+jFsby1YAe7j+TYXY5Syk1oaHixJwYn4SMwXidrUko5SUPDi8WE1ePefq34Zu0+Vvymp+AqpaqmoeHl7r20JVENAnn+m014+s0rlVIXTkPDywUH+PHYlUms2nWMb9ftt7scpZSL09BQ3NA1jrZNGvDinM16F1ylVKU0NBS+PsJTV7Vj15Ec3l/ym93lKKVcmIaGAhx3we3XJpJ//5jJsZwCu8tRSrkoDQ111lNXteVEXiGvz99mdylKKReloaHOatsklBu6xDFt0U6yjuoFf0qpc2loqP/xyMA2iMA/v99qdylKKRekoaH+R9PwetzZJ4HPV+9hw97jdpejlHIxGhrqHGP6tyKsnj/jZ+vtRZRS/0tDQ50jrJ4/9w9ozYKMQyzMOGR3OUopF6Khocp1W8/mxIbX48U5mynRGf6UUhYNDVWuIH9fHh7YhnV7jjN7vd5eRCnloKGhKjT04liSohvw8vdbKCwusbscpZQL0NBQFfL1ER4flMSOQ6eZmb7b7nKUUi5AQ0NV6vJ2UaQ0b8irczPILdCbGSrl7TQ0VKVEhHFD2nLgZD5TF+2wuxyllM00NFSVUlo04vK2Ubz58zaO5xTaXY5SykYaGsopjw1K4mR+EW/8ojczVMqbaWgop7SLCSW1U1PeXbSDAyfy7C5HKWUTp0JDRAaLyBYRyRSRceW8LiIy0Xp9rYh0qaqtiDQSkR9EJMN6bFjqtSet9beIyKBSywNEZIqIbBWRzSJyQ/W7rs7XwwPbUFRsmPhjht2lKKVsUmVoiIgv8BowBEgGhotIcpnVhgCJ1p9RwGQn2o4D5hljEoF51nOs14cB7YHBwOvWdgD+DBwwxrSxtvdzNfqsqql54xCGd49nxrLd7Dx02u5ylFI2cGZPozuQaYzZbowpAGYAqWXWSQXeMw5LgXARiamibSow3fp5OnBdqeUzjDH5xpgdQKa1HYC7gL8DGGNKjDF6Y6Q69sBlrfHzFSbM1VunK+WNnAmNWKD0lV1Z1jJn1qmsbbQxZh+A9RhV2bZEJNx6/pyIrBSRWSIS7UT9qgZFhQZxZ58Evlyzl837T9hdjlKqjjkTGlLOsrJ3sKtoHWfaOvt+fkAcsMgY0wVYArxc7gZERolIuoikHzx4sIq3U+fr3n4tqR/ox8vf6d6GUt7GmdDIApqVeh4H7HVyncraZluHsLAeD1SxrcNADvC5tXwW0IVyGGOmGGNSjDEpkZGRVfVPnafw4ADu7deSuZuyWbnrqN3lKKXqkDOhsRxIFJEEEQnAMUidVmadNOAO6yyqnsBx65BTZW3TgBHWzyOAL0stHyYigSKSgGNwfZkxxgBfAf2t9S4HNp5fd1VNubNPAhH1A3hpzhYc/zRKKW9QZWgYY4qA+4HvgE3ATGPMBhEZLSKjrdW+BbbjGLR+C7ivsrZWm/HAQBHJAAZaz7Fen4kjEOYAY40xZ2569CfgLyKyFrgdePQC+q4uQEigH2MHtGbJ9sMszNTzEZTyFuLpvyWmpKSY9PR0u8vwSPlFxVz28s9E1A/gi7F9EClvOEop5Y5EZIUxJqXscr0iXFVboJ8vD16eyJqs43y/MdvucpRSdUBDQ12Q67vE0jIihH9+v5VinRZWKY+noaEuiJ+vD49c2YYt2Sf5ak3Zk+qUUp5GQ0NdsKs6xJAcE8o/f9iq08Iq5eE0NNQF8/ERHhvUhl1HcnRaWKU8nIaGqhEDkqLo2rwh/56XSV6hTgurlKfS0FA1QkR4fFAS+0/k8cHS3+wuRylVSzQ0VI3p2bIxlyRG8NpPmZzM02lhlfJEGhqqRj0+KImjOYVMXbjT7lKUUrVAQ0PVqIviwhnUPpq3Fmzn6OkCu8tRStUwDQ1V4x69MonTBUW88fM2u0tRStUwDQ1V49pEN2Bo51imLd5J9ok8u8tRStUgDQ1VKx66og3FJYZ//5hhdylKqRqkoaFqRXzjYIZ3j2fGst38dvi03eUopWqIn90FKM/1wGWtmbViNxPmZvCvmzvbXY5Lmr1uH8dyCwn08yHQz5fwYH8i6gcSUT+ARiEBert55XI0NFStiQoNYkTvFkz5ZTujL21FUpMGdpfkUvYfz2PMf1ZW+HpwgC8JESEkRITQLiaUi+LCuCg2nLBg/zqsUqn/paGhatWYS1vx4dJdvPTdFt4ecc58Ll7tVL7jAsjnUttzaZsocguLOZpTwKFT+Rw8mc+uIzlsP3iaNVnH+HrtvrPt2kTXp0/rCC5JjKBHQmNCAvW/sao7+mlTtSo8OIBR/Vryyg9bWfHbUbo2b2h3SS4jt8BxR+Do0CDiGwdXuu7xnELW7jnGmt3H+HXHET78dRfvLtpJoJ8P/dpEMqRDE65IjiY0SPdCVO3S0FC17q6+CUxfspOXvtvMR3/oqcfpLXlFjhs71gvwrXLdsGB/LkmM5JLESO4H8gqLSd95lLmbspmzfj8/bMwm0M+HQe2bcFNKM3q3aoyPj/49q5qnoaFqXUigH/cPaM1fvtrIgoxD9GsTaXdJLiG3wBEaQf5Vh0ZZQf6+9E2MoG9iBP93TTKrs47xxao9fLFqD2lr9hLXsB4jerXgpm7NCKunex+q5ugpt6pODO8RT1zDevzju82U6LSwAGdvIV+vGqFRmo+P0CW+Ic+mdmDZn69g4vCLaRpejxe+3USvv8/jmS/Xk3U0pyZKVkpDQ9WNQD9fHr6iDev3nGD2+v12l+MS8oocYxpB/jX33zDI35ffdWrKzHt78fUDfRnSIYYPl+2i/0vz+dMna/WaGXXBNDRUnbnu4ljaRNfn5e+36LSwQN4FHJ5yRofYMF65qRPzHx/ArT3i+Xz1Hi575Wf+/Pk6DujtXVQ1aWioOuPrIzw+qC07Dp1mVnqW3eXY7sxAeG2Fxhmx4fX4a2oHFj4xgNt6xPPx8t1c+tJ8Xv5uC6fzi2r1vZXn0dBQdeqKdlGkNG/IhLlbzw4Ee6sz/b/QMQ1nRYUG8dfUDsx95FKuSI5m0k+ZXPbKfL5cvQdjdJxJOUdDQ9UpEeFPQ9py4GQ+7y7eYXc5tsorPDOmUTehcUaLiBD+PfxiPh3Tm6gGQTw4YzU3vrGEzftP1Gkdyj1paKg6161FIy5vG8Xk+ds4luO9EzXlFhYT4OuDr03XU3Rt3pAvx/bhxRs6sv3Qaa6ZuJCXv9ty9qwupcqjoaFs8fjgJE7lFzF5vvdO1JRXWExgDZ45VR0+PsLN3eKZ+8il/K5zUyb9lMmQVxewfOcRW+tSrktDQ9mibZNQhl4cy7uLd7LnWK7d5dgir7C4zsYzqtIoJIB/3tSZD+7uQVFJCTe9uYQXvtmoex3qHBoayjaPXpkEwD+/32pzJfbIKyyu8/GMqvRNjGDOg/24pXs8by3YwdUTF7Au67jdZSkXoqGhbBMbXo+RvVvw2aosNu3zvkHYXBfa0ygtJNCPF4Z25P27u3M6v5jrJy/izZ+36ZX8CtDQUDYb2781oUH+jJ+92e5S6lxeYUmNXg1e0y5JjGTOQ5dwedto/j57M7dP/VUvClQaGspeYcH+jB3Qip+3HmRR5iG7y6lTuS54eKqs8OAAJt/WhfHXd2Tlb8e4auICFnvZv5P6X06FhogMFpEtIpIpIuPKeV1EZKL1+loR6VJVWxFpJCI/iEiG9diw1GtPWutvEZFB5bxfmoisP//uKld0R68WxIbX4++zN3nVIZB8NwgNcFxbM6x7PGn39yE8OIBb3/mVV+dmUOxF/1bqv6oMDRHxBV4DhgDJwHARSS6z2hAg0fozCpjsRNtxwDxjTCIwz3qO9fowoD0wGHjd2s6Zeq4HTlWns8o1Bfn78tggx80Mv1yzx+5y6oyrjmlUJDG6AWn392Fo51j+NXcrd05b7tXX2XgrZ/Y0ugOZxpjtxpgCYAaQWmadVOA947AUCBeRmCrapgLTrZ+nA9eVWj7DGJNvjNkBZFrbQUTqA48Az59/V5UrS+0US4fYUF6a4z0Xl7n6mEZ5ggP8eOWmTvxtaEeWbjvMtZMWsnGv953E4M2c+cTGArtLPc+yljmzTmVto40x+wCsxygn3u854BWg0skBRGSUiKSLSPrBgwcrW1W5CB8f4amr2rH3eB5TF3nH7UXcYUyjPCLCLT3imXFvTwqLDNdPXsSXq71nD9HbORMa5d3joOzBzIrWcaatU+8nIp2B1saYz6tojzFmijEmxRiTEhmps8S5i96tIriiXRSv/7SNw6fy7S6n1rnidRrno0t8Q756oC8XxYbz4IzVvKQTbHkFZ0IjC2hW6nkcsNfJdSprm20dwsJ6PFDFtnoBXUVkJ7AQaCMi852oX7mRcUPakVtYzKvzMuwupda5e2gARDYI5IN7ejC8ezNe+2kbo95fwSm93bpHcyY0lgOJIpIgIgE4BqnTyqyTBtxhnUXVEzhuHXKqrG0aMML6eQTwZanlw0QkUEQScAyuLzPGTDbGNDXGtAD6AluNMf2r0WflwlpH1eeW7vH859ddZB44aXc5taa4xFBYbNxqILwiAX4+/G1oR/5ybTI/bTnA7ycv9tpbw3iDKkPDGFME3A98B2wCZhpjNojIaBEZba32LbAdx6D1W8B9lbW12owHBopIBjDQeo71+kxgIzAHGGuM8Y6RUQXAQ1ckEuzvy/PfbLK7lFpzZrDf3QbCKyIijOyTwLQ7u7HnaC6pkxaxZvcxu8tStUA8ffKVlJQUk56ebncZ6jy99ct2Xvh2E9Pu7Eb/pKiqG7iZQ6fySXl+Ls+mtueOXi3sLqdGZWSf5M5pyzl0Kp8JN3dmcIcYu0tS1SAiK4wxKWWXe8avOcrj3NG7Oc0bB/P8N5so8sD5xM/uafi5/+GpshKjG/DF2D60iwllzH9W8s5C7zgbzltoaCiXFOjny1NXtSPzwCk+XLbL7nJq3NnQCPC80ACIqB/IR3/oyZXJ0Tz39Uae/WqjnlnlITQ0lMu6MjmaXi0b868ftnrclcdnp3r189z/gkH+vrx+a1dG9m7B1EU7uP+jlV5z4aYn89xPrHJ7IsLT1yRzPLeQCXM96xTcXOvLs56H7mmc4esjPHNtMv/v6nZ8u24/I6Yu40Reod1lqQugoaFcWnLTUG7t0Zz3l/7G5v2ec7uK/5495dmhAY7wv+eSlrw6rDMrdx3lpjeWkK23WHdbGhrK5T0ysA0Ngvz4a9pGPOVsv9wCa0/DC0LjjNTOsUwd2Y3dR3K4/vXFbD+o9x11RxoayuU1DAng0SuTWLL9MLPX77e7nBqRV2SNaXjIdRrOuiQxkhmjepFXWMyNbyxh/R6dStbdeNcnVrmtW7rH0y4mlBe+2XT2t3R3llfgPYenyuoYF8as0b0I8vdl2JSlLN6mkzq5Ew0N5RZ8fYRnU9uz51guk35y/0HxvCLvDQ2AlpH1+XRMb2LCghj57nJ+2Jhtd0nKSRoaym10a9GI67vEMuWX7WQecO/j4d44plFWk7AgZo3uRbuYUEZ/sILPVmbZXZJygoaGcitPDmlHkL8vz6Std+tB8bPXaXhxaIBjDvL/3NODHgmNeGTmGqZ5yVwq7kxDQ7mVyAaBPDEoiUWZh/lq7T67y6m23MJiAnx98PUpb/oY71I/0I+pI7txZXI0f/lqI5N+zHDrXwg8nYaGcju39GhOx9gwnv96Iyfd9EKxvMJiAr3szKnKOK4e78LQi2N5+futjJ+zWYPDRemnVrkdXx/hues6cPBUPi9/t8XucqrFEyZgqml+vj68cmMnbusZz5s/b+f/fbFe71flgjQ0lFvq3CycEb1a8N7S31i566jd5Zy3vMJirx4Er4iPj/BcagdGX9qK//y6i8dmrfHIuxy7Mw0N5bYeG5REk9Agnvx0HYVu9sWSW1jsdRf2OUtEGDekLY8PSuKzVXt44KNVFBS517+vJ9NPrXJb9QP9eDa1A1uyT/LWgu12l3Ne8gpLdE+jCmMHtOb/rklm9vr9jHo/Xe+Q6yI0NJRbG5gczeD2TXh1bgY7Dp22uxyn5RYWE6ihUaW7+iYw/vqO/Lz1ICPfXcap/CK7S/J6GhrK7f01tT0Bfj786dO1bjNwmq9jGk4b1j2eCTd3ZvnOo9z+zq8cz3XPM+Y8hYaGcnvRoUE8fU0yy3Yc4f2lv9ldjlPyCkt0TOM8pHaO5bVburB+z3FueWspR0571qRc7kQ/tcoj3Ng1jn5tInlxzmZ2H8mxu5wq5eqexnkb3KEJb92RQuaBU9z85hIO6JwcttDQUB5BRBh/fUd8RPjTp2td/sIwvU6jevonRTHtzu7sOZbLjW8uIeuo6/+C4Gk0NJTHaBpejz9f3Y7F2w7zgYsfpsrV0Ki2Xq0a88E9PThyuoCb3ljiVidAeAINDeVRhnVrRv+kSF74dhPbXHhmuPzCEg2NC9AlviEf/aEneUUl3PTmErZmn7S7JK+hoaE8iojwjxsuIsjfl4c/Xu2SF/0VlxgKivU6jQvVITaMj0f1RICb39RZAOuKhobyOFGhQfx9aEfWZh3n3z9m2l3OOc5cpKZnT124xOgGzBrdi+AAP4ZPWUr6ziN2l+Tx9FOrPNKQjjHc0CWO137KZMVvrnVvqlwrNOoF6J5GTWjeOIRZo3sR0SCQ299ZxoKMg3aX5NE0NJTHeuZ3ycSEBfHHj1ZxPMd1Lgg7u6fhp6FRU5qG12Pmvb1o3jiYu6el892G/XaX5LE0NJTHCg3yZ9ItXcg+kedSp+GeDQ3d06hRkQ0C+XhUL5KbhnLff1bq9LG1RENDebTOzcJ5YnASczbsd5nTcM9O9eqn//1qWliwPx+Umj52+uKddpfkcfRTqzzePX1b0j8pkue+2cSGvfafYZN7diBc9zRqw5npYwcmR/NM2gb+PU+nj61JGhrK4/n4CK/c2ImGwf6M+WAlx3LsvW9Rng6E17ogf18m39qF67vE8soPW3n+m01uczNLV6ehobxC4/qBvH5rV/Ydz+XBGasptvELJLdAB8Lrgp+vDy//vhMje7fgnYU7eOyTNS553Y67cSo0RGSwiGwRkUwRGVfO6yIiE63X14pIl6raikgjEflBRDKsx4alXnvSWn+LiAyylgWLyDcisllENojI+AvruvI2XZs35Jlr2/Pz1oO8OnerbXXkWbPQ1QvQ39lqm4+P8My1yTw6sA2frdzD6PdX6GROF6jKT62I+AKvAUOAZGC4iCSXWW0IkGj9GQVMdqLtOGCeMSYRmGc9x3p9GNAeGAy8bm0H4GVjTFvgYqCPiAypTqeV97q1Rzw3pcQx8cdM207LzLP2NAJ1T6NOiAgPXJ7Ic9d14MctB7jt7V9tP0Tpzpz5Vac7kGmM2W6MKQBmAKll1kkF3jMOS4FwEYmpom0qMN36eTpwXanlM4wx+caYHUAm0N0Yk2OM+QnA2tZKIO78u6y8mYjwbGoHOsWF8fDHq2259URekY5p2OH2ns2ZNLwLa7OOc+MbS9h7LNfuktySM6ERC+wu9TzLWubMOpW1jTbG7AOwHqOcfT8RCQeuxbGHcg4RGSUi6SKSfvCgXh2q/leQvy9v3ZFCeD1/7pmezv7jdTsvw9kxDT17qs5dfVEM0+7qxv7jedwwebHe6LAanAkNKWdZ2VHEitZxpu15vZ+I+AEfARONMdvL24AxZooxJsUYkxIZGVnF2ylvFBUaxDsju3Eyr5C7py/ndB3OPa3Xadird6sIPr63F8UlhhsmL2bJtsN2l+RWnPnUZgHNSj2PA/Y6uU5lbbOtQ1hYjwecfL8pQIYxZoITtStVoXYxoUy6pQub9p3g/g9X1tmZNXlFxfj7Cn6+Ghp2SW4ayqdjehMdGsSIqctIW1P2K01VxJlP7XIgUUQSRCQAxyB1Wpl10oA7rLOoegLHrUNOlbVNA0ZYP48Aviy1fJiIBIpIAo7B9WUAIvI8EAY8dP5dVepcA9pG8fx1Hflpy0Eem7WmTs7lzy3QCZhcQbNGwXw6ujed48P540ermDx/m14E6AS/qlYwxhSJyP3Ad4AvMNUYs0FERluvvwF8C1yFY9A6B7izsrbWpscDM0XkbmAXcKPVZoOIzAQ2AkXAWGNMsYjEAX8GNgMrRQRgkjHm7Rr4e1Be7JYe8RzLLeAfc7YQVs+fv/6uPdbnq1bkF2louIqwYH/ev7s7j81ay4tzNrPz0GmeH9oBf90LrFCVoQFgjPkWRzCUXvZGqZ8NMNbZttbyw8DlFbR5AXihzLIsyh/vUOqCjbm0FcdyCpnyy3bqB/rx+KCkWguO3IJinYDJhQT6+fLqzZ1JaBzMxB8z2X00h8m3diUs2N/u0lySxqlSOE7FfXJIW4Z3j+f1+dsYP3tzrR2qyCss0QmYXIyPj/DIlUm8cmMnlu88wtDJi3Tu8QroJ1cpi4jwwnUduK1nPG/+sp1nv95YK8GRW6h7Gq7qhq5xfHB3D47lFJI6aaFO6FQODQ2lSvHxEZ5L7cBdfRJ4d9FOnvp8PUU1fFZVXmExgRoaLqtHy8Z8ObYPTcPrMfLd5UxduEMHyEvR0FCqDBHh6WvacV//Vny0bBej3l9Ro9dx5Omehstr1iiYT8b05vK2UTz79UYenbnm7EWZ3k5DQ6lyiAhPDG7Lc9d1YP6WA9z05hKyT9TMleM6puEe6gf68cZtXXlkYBs+X72HGyYvZveRHLvLsp1+cpWqxO09m/POiG7sOHSa1EmLSN955IK3qWMa7sPHR/jj5YlMHdGNrKM5XDtpIT9uzra7LFtpaChVhQFto/hkdG8C/X0YNmUpb/2y/YKOcecV6nUa7mZA2yi+eqAvTcPqcde0dMbP3lzjY13uQkNDKSckNw3lqwf6cnm7KF74dhN/eG8FB05W73BVroaGW2reOITP7uvNLT3ieePnbQx/ayl7vPBOuRoaSjkpNMifN27rytPXJPNLxkGueOVnZqXvPu+9jvzCEg0NNxXk78vfhnbk1WGd2bj3BEMm/MI3a/fZXVad0tBQ6jyICHf3TWD2g5eQ1KQBj3+yltve+ZUNe52bl6O4xFBQrAPh7i61cyzfPngJCZH1GfvhSp74ZA2n6vBOyXbST65S1dAqsj4fj+rFc6ntWb/nBFdPXMiDM1ax63DlZ9ecmWpUB8LdX/PGIXwyuhdjB7TikxVZDJ7wi1fcZl1DQ6lq8vERbu/Vgl+eGMCY/q34bsN+BrwynzEfrGDxtkPlHrbKLdQJmDyJv68Pjw9qy6zRvfDzEYa/tZS/pG2o0/lZ6ppTNyxUSlUsrJ4/fxrclpG9WzB10Q5mLt/N7PX7aRkZwpAOTbisbTSdm4Xj6yO6p+GhujZvxOwH+/HinM1MW7yTHzZm8/zQDgxIiqq6sZsRT788PiUlxaSnp9tdhvIieYXFfL12H7PSd5P+21GKSwwNg/3pEBtGdGgQn6zI4tVhnUntXHbWZOUJlu88wpOfrSPzwCmu7dSUp69uR1RokN1lnTcRWWGMSTlnuYaGUrXneE4h87ceYGHGITbuO0FG9ikKikuYeW8vuic0srs8VUvyi4p58+ftTPoxE39fxwWCd/ZJIMCNpvjV0FDKBRQVl3DkdAGRDQJrdaIn5Rp+O3ya577eyNxNB0iICGHckLZcmRztFv/2FYWG+8SeUh7Az9eHqNAgt/jSUBeueeMQ3h7RjXfv7IYI3Pv+Cm6YvJjlNXA7GrtoaCilVC0bkBTF9w/14+/Xd2TPsVxufGMJt7/zq1uGhx6eUkqpOpRbUMz0JTt5e8F2Dp0qoGfLRtx7aSsuTYzEx8d19kB1TEMppVxIbkExHy3bxZu/bCP7RD4tI0MY2bsF13eJo36g/VdDaGgopZQLKigqYfb6fUxduIM1WccJDvDlqo4x3Ng1ju4JjWwb/9LQUEopF2aMYdXuY8xcvpuv1+7jVH4RseH1GNS+CUM6NqFrfMM6PXyloaGUUm4ip6CIOev3883afSzIOERBcQmNQgLo3aoxfVtH0LNlY5o3Dq7VvRANDaWUckMn8wr5cfMBft56kEWZh8g+kQ9AeLA/HWPD6BAbRsuIEFpGhtCsUTCNQwLxrYE9Eg0NpZRyc8YYMg+cIv23o6zNOsbq3cfJyD5JUcl/v8d9BBqFBBBRP5BPxvSu9qB6RaFh/xC9Ukopp4gIidENSIxuwPDu8YDjLgNZR3PZceg0u4/mcOhkPgdPFXDkdD7BtXBjTA0NpZRyY36+PrSICKFFREidvJ9eEa6UUsppGhpKKaWcpqGhlFLKaRoaSimlnKahoZRSymkaGkoppZymoaGUUsppGhpKKaWc5vG3ERGRg8Bv1WweARyqwXLcgTf2Gbyz397YZ/DOflenz82NMZFlF3p8aFwIEUkv794rnswb+wze2W9v7DN4Z79rss96eEoppZTTNDSUUko5TUOjclPsLsAG3thn8M5+e2OfwTv7XWN91jENpZRSTtM9DaWUUk7T0FBKKeU0DY1yiMhgEdkiIpkiMs7uemqLiDQTkZ9EZJOIbBCRB63ljUTkBxHJsB4b2l1rTRMRXxFZJSJfW8+9oc/hIvKJiGy2/s17eXq/ReRh67O9XkQ+EpEgT+yziEwVkQMisr7Usgr7KSJPWt9vW0Rk0Pm8l4ZGGSLiC7wGDAGSgeEikmxvVbWmCHjUGNMO6AmMtfo6DphnjEkE5lnPPc2DwKZSz72hz68Cc4wxbYFOOPrvsf0WkVjgj0CKMaYD4AsMwzP7PA0YXGZZuf20/o8PA9pbbV63vvecoqFxru5ApjFmuzGmAJgBpNpcU60wxuwzxqy0fj6J40skFkd/p1urTQeus6XAWiIiccDVwNulFnt6n0OBfsA7AMaYAmPMMTy83zimtK4nIn5AMLAXD+yzMeYX4EiZxRX1MxWYYYzJN8bsADJxfO85RUPjXLHA7lLPs6xlHk1EWgAXA78C0caYfeAIFiDKxtJqwwTgCaCk1DJP73NL4CDwrnVY7m0RCcGD+22M2QO8DOwC9gHHjTHf48F9LqOifl7Qd5yGxrmknGUefV6yiNQHPgUeMsacsLue2iQi1wAHjDEr7K6ljvkBXYDJxpiLgdN4xmGZClnH8FOBBKApECIit9lblUu4oO84DY1zZQHNSj2Pw7FL65FExB9HYPzHGPOZtThbRGKs12OAA3bVVwv6AL8TkZ04Dj1eJiIf4Nl9BsfnOssY86v1/BMcIeLJ/b4C2GGMOWiMKQQ+A3rj2X0uraJ+XtB3nIbGuZYDiSKSICIBOAaM0myuqVaIiOA4xr3JGPPPUi+lASOsn0cAX9Z1bbXFGPOkMSbOGNMCx7/tj8aY2/DgPgMYY/YDu0UkyVp0ObARz+73LqCniARbn/XLcYzbeXKfS6uon2nAMBEJFJEEIBFY5uxG9YrwcojIVTiOe/sCU40xL9hbUe0Qkb7AAmAd/z2+/xSOcY2ZQDyO/3g3GmPKDrK5PRHpDzxmjLlGRBrj4X0Wkc44Bv8DgO3AnTh+cfTYfovIX4GbcZwpuAq4B6iPh/VZRD4C+uO4BXo28AzwBRX0U0T+DNyF4+/lIWPMbKffS0NDKaWUs/TwlFJKKadpaCillHKahoZSSimnaWgopZRymoaGUkopp2loKKWUcpqGhlJKKaf9f0LoyjA7uv9lAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch, decay_rate=0.5, lr_min=1e-5)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*2)\n", "\n", "plt.plot([i for i in range(num_epoch*2)], lr_per_epoch);" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `warmup_t` " ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "Defines the number of warmup epochs. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `warmup_lr_init` " ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "The initial learning rate during warmup. " ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "hidden": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2JUlEQVR4nO3dd3gVZdrH8e99ThohCSmkAKEECCAQpIQOMYIgKgK2FStiQUVcC7qi67rq7rsvq+gLWFAsiH3RXTXrWiiKgsGlWGgSCIiQQBolBdLzvH+cQ4whJCd1TpL7c125zpnyzNwDmh8zz8wzYoxBKaWUcoXN6gKUUko1HxoaSimlXKahoZRSymUaGkoppVymoaGUUsplHlYX0Njat29vunXrZnUZSinVrGzZsiXLGBNaeX6LD41u3bqxefNmq8tQSqlmRUR+qWq+Xp5SSinlMg0NpZRSLtPQUEop5bIW36ehlGoYxcXFpKSkUFBQYHUpqgH5+PgQGRmJp6enS+traCilXJKSkoK/vz/dunVDRKwuRzUAYwxHjhwhJSWFqKgol9q4dHlKRCaJSJKIJIvIvCqWi4gsdi7fKiKDa2orIsEiskpE9jg/g5zzQ0TkSxHJE5FnK+1niIhsc25rseh/uUo1mYKCAkJCQjQwWhARISQkpFZnjzWGhojYgeeAC4C+wFUi0rfSahcA0c6fWcASF9rOA9YYY6KBNc5pgALgT8B9VZSzxLn9U/ua5NJRKqUahAZGy1Pbv1NXLk8NA5KNMfucO3gXmArsrLDOVOB14xhn/VsRCRSRDkC3atpOBeKd7ZcDa4EHjDEngPUi0rPSgXUAAowxG5zTrwPTgE9rdcSu2v5PKMwFuzfYPcHD+9fvnr7gGwK+weATCDa9n0Ap1Tq4EhqdgIMVplOA4S6s06mGtuHGmMMAxpjDIhLmQh0pVezjNCIyC8cZCV26dKlhs2ewMwGO7a95PbGBTztniIRAYJff/rQJAv3XmVL1ds8999C1a1fuvvtuAM4//3w6d+7Myy+/DMDcuXPp1KkTPXv2ZOfOncybN48PP/yQXr160bev4wJHfHw8CxYsIDY21qrDaPZcCY2qfuNVfnPTmdZxpa2rXN6WMWYpsBQgNja2bvv73XIoK4PSQigtgpIix2dpERTlwcmjkH/0t595GXD4Bygp/HU73v4Q2BVCe0N4XwjrB/4RGiRK1dKoUaN47733uPvuuykrKyMrK4ucnJzy5YmJiSxcuJDhw4czZcoUAD788EMmT55cHhpNqbS0FLvd3uT7bWyuXFdJATpXmI4EDrm4TnVt052XnE5despwoY7IGupoWDYbeLZxnEn4hUK7ThAcBREx0P0c6HcJDL0J4u6HSf8Ll78CMz+Dq/8BFz4Jo+6EHuc6trPrP7DmL/DOdHjjEvjsIfj+TUjbDmWljXoYSrUEo0ePJjExEYAdO3bQv39//P39OXbsGIWFhfz0008MGjSI1157jTlz5pCYmEhCQgL3338/AwcOZO/evQC89957DBs2jF69erFu3brT9jN79mwSEhIAuOSSS7jxxhsBeOWVV3j44YcBmDZtGkOGDKFfv34sXbq0vK2fnx+PPPIIw4cPZ8OGDfj5+fHAAw8wZMgQzjvvPDZu3Eh8fDzdu3cv38epek+ZPHkya9euLd/e3LlzGTx4MOPHjyczM7OB/1Rrz5UzjU1AtIhEAanAdODqSuskAHOcfRbDgWznJafMatomADOA+c7Pj6orwrm9XBEZAfwXuB54xoX6m5bN5jiT8I+AzsN+nV9WCkd/howdkL4TMnbCL984lnn7Q6chjvUjhzkCSik39ti/d7DzUE7NK9ZC344B/Pnifmdc3rFjRzw8PDhw4ACJiYmMHDmS1NRUNmzYQLt27RgwYABeXl7l648aNYopU6YwefJkLr/88vL5JSUlbNy4kU8++YTHHnuM1atX/2Y/cXFxrFu3jilTppCamsrhw4cBWL9+PdOnTwfg1VdfJTg4mPz8fIYOHcpll11GSEgIJ06coH///jz++OMAnDhxgvj4eP7+979zySWX8PDDD7Nq1Sp27tzJjBkzys+IzuTEiRMMHjyYp556iscff5zHHnuMZ599tto2ja3G0DDGlIjIHOBzwA68aozZISK3OZe/AHwCXAgkAyeBmdW1dW56PrBCRG4CDgBXnNqniOwHAgAvEZkGTDTG7ARuB14D2uDoAG+cTvDGYLND+56On75THfMKsiH1Ozi4EVI2wb61AKRKOD959afL0IuIPmsQoh3tSgG/nm0kJiZy7733kpqaSmJiIu3atWPUqFEubePSSy8FYMiQIezfv/+05WPHjmXhwoXs3LmTvn37cuzYMQ4fPsyGDRtYvHgxAIsXL+aDDz4A4ODBg+zZs4eQkBDsdjuXXXZZ+ba8vLyYNMlxk2dMTAze3t54enoSExNT5b4rs9lsXHnllQBce+215bVbyaWH+4wxn+AIhorzXqjw3QB3uNrWOf8IMP4MbbqdYf5moL8rNTcLPu0cl696nAvGwLGf4eAm/pvwL7oXf0jBvn+xzjOcoi5xRI2YTPfoGA0Q5RaqOyNoTKNGjSIxMZFt27bRv39/OnfuzFNPPUVAQED5ZaSaeHt7A2C32ykpKTlteadOnTh27BifffYZcXFxHD16lBUrVuDn54e/vz9r165l9erVbNiwAV9fX+Lj48ufc/Dx8flNP4anp2f5La02m6183zabrXzfHh4elJWVlbep7pkJd7jlWX8DuQsRCO7O0Z6XcO+J61k7bCnZsXdR6NOe0L3vc+KtGXz9t4tY89aTpB8+YHW1Slli9OjRfPzxxwQHB2O32wkODub48eNs2LCBkSNHnra+v78/ubm5td7PyJEjWbhwIXFxcYwdO5YFCxYwduxYALKzswkKCsLX15ddu3bx7bff1uuYunXrxg8//EBZWRkHDx5k48aN5cvKysp4//33AXj77bcZM2ZMvfbVEHQYETezLTUbgGFndWNUj1hgJkcy09j+zb8pTlpDWNJbpCW9zY+BAwkechkDR52Ph4tjxijV3MXExJCVlcXVV1/9m3l5eXm0b9/+tPWnT5/OLbfcwuLFi8t/+bpi7NixrFy5kp49e9K1a1eOHj1aHhqTJk3ihRdeYMCAAfTu3ZsRI0bU65hGjx5NVFQUMTEx9O/fn8GDywfUoG3btuzYsYMhQ4bQrl07/vGPf9RrXw1BHFeWWq7Y2FjTnF7C9OwXe1iwcjdbH51IgM/pYZD6y152rn0Hv/0raVuaQ549iJNRE+g77io6Rro2doxSdfHTTz9x1llnWV1Gq+Ln50deXl6j76eqv1sR2WKMOe2BFj3TcDPbUrPp3r5tlYEB0KlrDzrNeJji4j/w/fpPOfHdvwhLfo/M5PfYHjKcLufeSJ8Bw6psq5RS9aWh4Wa2pWQT2y24xvU8Pb0Ydu5UOHcqh1J+ZseqNwj65VOK3p/Fmk97EzDiOoaMuQBbC3y4SKnWoinOMmpLO8LdSFZeIYeyCxgQ2a5W7TpGRjFh5iOcdd/nHBlwCz6FmXiveZj1/3sR6xKWUViY30gVK6VaGw0NN3KqEzymU+1C4xQ/vwDOvfwOhj/4OXmjH6TU5k27zYvY8vcL+fqjVygqLKx5I0opVQ0NDTeyLSUbEehXx9A4xcPTk1HnX0n8vA8ou+BJ8r2CCdzyDBv/fhHrP36dkuLiBqpYKdXaaGi4ka0pjk5wP++G6WoSm42BIycw7g/vUTJxPkWe/gRsfJoN8y/im0/fprRUx7xSStWOhoYb2ZZ6nAGRgQ2+XbHZGDxmEuc+8E+Kxv+VUpsP/hueYP38KXyfuLLB96dUY7jnnntYuHBh+fT555/PzTffXD49d+5cnn76aRISEpg/fz7gGOV2585fX/0THx9PQ92C/7e//a1BttPcaGi4iYycAtJzCulfz0tT1RGbjdhzJhM37wPyz3kEe1kR9s/+wKqnZvDznu2Ntl+lGsKpIUSA8qHRd+zYUb48MTGR0aNHM2XKFObNc7wItHJoNKTGDo2qhjhxBxoabuJUJ3ht75yqC5vdzvDxlzL0DwkcibmZwNzd5LxxPStf/ANHs9Ibff9K1UVjDo1eUFDAzJkziYmJYdCgQXz55ZfAmYctnzdvHvn5+QwcOJBrrrnmN3WuWLGCe++9F4BFixbRvXt3APbu3Vs+DMjjjz/O0KFD6d+/P7NmzeLUQ9bx8fE89NBDnHPOOSxatIj4+Hjuuece4uLiOOuss9i0aROXXnop0dHR5cO079+/n/79fx2Sb8GCBTz66KPl27v77rsZNWoU/fv3/80QJXWlz2m4ia0p2dgE+nYIaLJ9enu34dwr5nBs3JVs+ufThKd8zr5nvubHvlcx5tLb8fT0qnkjqnVKfAay9jTsNttHO95BcwaNOTT6c889B8C2bdvYtWsXEydOZPfu3WesZf78+Tz77LP88MMPpy2Li4vjySefBGDdunWEhISQmprK+vXry4cimTNnDo888ggA1113HR9//DEXX3wxAMePH+err74C4N///jdeXl58/fXXLFq0iKlTp7JlyxaCg4Pp0aMH99xzT41/rCdOnCAxMZGvv/6aG2+8ke3b63dVQc803MT21Gx6hPrRtoE6wWsjKCSUibP+l3bXv8WxgLMI2bGMxCcuYfuW9U1ei1LVqTg0+siRIxk5cmT5dH2GRl+/fj3XXXcdAH369KFr167VhkZ1IiIiyMvLIzc3l4MHD3L11Vfz9ddfs27duvLQ+PLLLxk+fDgxMTF88cUXv7nMdmoo9FNOvXMjJiaGfv360aFDB7y9venevTsHDx6kJldddRXgCLOcnByOHz9ep+M6Rc803IAxhq2p2YyNPn3AtabUredZdJu7jE1rE/D4eiFlH83h82/PZdj0hwgK0RdDqQqqOSNo1N020tDoZxqDrzbDllc0cuRIli1bRu/evRk7diyvvvoqGzZs4KmnnqKgoIDZs2ezefNmOnfuzKOPPvqb7bZt27bKeisOrX5quqSkpMYaKw+nXt/h1fVMww2k5xSSmVvIgEbsBK+NofFTGDT3I9KjLiU8/Sv2PDOVdQnLKNNbdJXFGmto9Li4ON566y0Adu/ezYEDB+jdu3e1w5Z7enpSfIZnnuLi4liwYAFxcXHlfSTe3t60a9eu/Jd6+/btycvLq9Xou1UJDw8nIyODI0eOUFhYyMcff/yb5adGxl2/fj3t2rWjXbv6/Z7R0HAD5U+CN8LttnXl29afCTMfod11r5PbpjPtNi/iyyevZH/yT1aXplqxU0OjVxyOPCYmhnbt2p1xaPQnn3ySQYMGlXeEV2X27NmUlpYSExPDlVdeyWuvvYa3t/dvhi2/7777fjNs+axZsxgwYMBpHeHgGFr94MGDxMXFYbfb6dy5c3kneGBgILfccgsxMTFMmzaNoUOH1uePBE9Pz/L3kk+ePJk+ffr8ZnlQUBCjRo3itttu45VXXqnXvkCHRncLT69M4tkvk9nx2CTaeLnfAIOmrIxvPn0Lr01L8DBFHO97HXGXztb3eLQyOjR68xMfH8+CBQuIjT1thPPfqM3Q6Hqm4Qa2pmbTK9zfLQMDHM93jLnoOqJmv09W4CDa71jGugW/02c7lGqFNDQsZoxhe2p2nQcpbEqhYR2ZcNeLnBzzR3wLMzn+xg188fZTOpaVUm5q7dq1NZ5l1JaGhsUOZxeQlVdETBM81NcQxGZjxMQr6HnH+2SGxNJ+1xuse/IKPetoJVr65ezWqLZ/pxoaFtuaUr/h0K0SEhrBxLteIP+cP+NblMWxN2/gqw9exFS49U+1LD4+Phw5ckSDowUxxnDkyBF8fHxcbqPPaVhse2o2HjbhrCZ8ErwhDR9/CZkxw/n+9QeJ+H4Jq/clMui6v9M+LMLq0lQDi4yMJCUlhczMTKtLUQ3Ix8eHyMhIl9fX0LDY1tRsosP98fF0z05wV4SGdWTCvcv46sOXCPnxJZKfv4Jfxj3AkLjJVpemGpCnpydRUVFWl6EsppenLGSMYVvKcbd5qK8+xGYj/tJbCbr2NQo8A/Fc/TArX/wD+SdPWF2aUqoBaWhYKPV4PsdOFjebTnBXREX3Z+R9K0jrOoWI1JVs/L/f6QOBSrUgGhoW2tZMO8Fr4u3dhok3PU7ppCfwLs7hyBs3kPjZ21aXpZRqABoaFtqamo2nXejTwd/qUhrFoFET6X7bOxzzjcIv8QlWPn+vXq5SqpnT0LDQ9tRsekf44+3RfDvBaxIWEUn8vW+Q0eMKwtO+1MtVSjVzGhoWMcawNaV5PAleXx6enpw344+UnT8fn+JsjrxxAxs+f9fqspRSdaChYZHD2QVk5xc36Zv6rDZo9PlE3foOx3270fab+ax8+Y8UFxdZXZZSqhZcCg0RmSQiSSKSLCLzqlguIrLYuXyriAyuqa2IBIvIKhHZ4/wMqrDsQef6SSJyfoX5V4nINuc+PhMRa99aVA9J6Y4x/ntHtJ7QAAjr0Jm4e98grcvFRBz4D189fT1ZGWlWl6WUclGNoSEiduA54AKgL3CViPSttNoFQLTzZxawxIW284A1xphoYI1zGufy6UA/YBLwvIjYRcQDWASca4wZAGwFfn3jezOTlOYMjfCW2QleHU9PLybe/BfyRj1A8Mm97F4ynZ0/bLC6LKWUC1w50xgGJBtj9hljioB3gamV1pkKvG4cvgUCRaRDDW2nAsud35cD0yrMf9cYU2iM+RlIdm5HnD9txfG+wgDgUK2P2E0kpeUSEeBDO9/W+06KUZOuwv/KpZTZPCj61xy+/ugVHbtKKTfnSmh0Aiq+vTzFOc+VdaprG26MOQzg/AyrblvGmGLgdmAbjrDoC1T5GioRmSUim0Vks7uOk5OUlkvviNZ3llFZdN9B9J/zDzID+hG45RlWvXg/RYWFVpellDoDV0KjqreQVx7m8kzruNLWpf2JiCeO0BgEdMRxeerBqjZgjFlqjIk1xsSGhobWsLumV1JaRnJmHn00NAAIDAph3D3LSO9xORGH17Bu4fUczUq3uiylVBVcCY0UoHOF6UhOvyx0pnWqa5vuvISF8zOjhm0NBDDG7DWOsZlXAKNcqN/t7D9ygqKSMnq1wv6MM7Hb7UyY8TB5ox8k6ORedj5/Ncm7frS6LKVUJa6ExiYgWkSiRMQLRyd1QqV1EoDrnXdRjQCynZecqmubAMxwfp8BfFRh/nQR8RaRKByd6xuBVKCviJw6dZgANMunxJLS8gD08lQVRp1/Jb6XPY/NlJL9ziw2ra38n5pSyko1Do1ujCkRkTnA54AdeNUYs0NEbnMufwH4BLgQR6f1SWBmdW2dm54PrBCRm4ADwBXONjtEZAWwEygB7jDGlAKHROQx4GsRKQZ+AW5ogD+DJpeUloNNoGeYn9WluKU+A4aREfYmO5bdSfAXf2bN4WTGXXk3YtPHipSymrT0t3DFxsaazZs3W13Gb9z6xmb2ZOTxxdx4q0txawX5J/nqpfvpkPUNh0PHcM4tT+Lj08bqspRqFURkizHmtBeM6z/dLJCUlqud4C7waePLxDnPkNXnOjpkruebRTM4dsQ974ZTqrXQ0GhiJ4tK+OXoSe0Ed5HYbIy7ei4n4/5E0Ml9bH/+Gg78nGR1WUq1WhoaTSw5Iw9j0DONWhpx3mX4TFuMV+lJ0pbPZPvmdVaXpFSrpKHRxHaltc4xpxpC30GjiJy5nAKPAIoT7tGRcpWygIZGE0tKy8XH00aXYF+rS2mWOnXtwYA73uJI2560/WY+a956UoceUaoJaWg0sd3puUSH+WO3VfXgu3JFYFAIcXcvJy1sLKFJb7HqhfsoKS62uiylWgUNjSa2S8ecahBe3t5MmL2IjJ6/IyLtC75cPIuTJ3KtLkupFk9DowkdPVFEZm5hqxwOvTGIzcZ51z/E8SF3Epb9A98unqFjVinVyDQ0mlD5OzT0TKNBxU29iaJxjxFYcIAdS64j5Zc9VpekVIulodGEktJyAA2NxjA0fgo+0xbjXZpHyms3krRto9UlKdUiaWg0oaT0XAJ9PQnz97a6lBap76BRhF/3KiXiTd77d/L9N59bXZJSLY6GRhNKSsuld7g/jhcPqsbQtUcfet+6nFyvcOTzh9iw8j2rS1KqRdHQaCLGGHan5+mlqSYQGt6JwXcuJ6ttNL7r/8ZXH7xodUlKtRgaGk0k5Vg+eYUlGhpNJCAgiDG/f5X0wEEEfb+E1W/M14cAlWoAGhpNZHe6484pHXOq6fi08WXcnS9yODyesD3vsuqlhygtLbW6LKWaNQ2NJnJqzKlofUajSXl4ejLx9qdJ7zaNiNTPWP3cHIoKC60uS6lmS0OjiSSl5dIpsA0BPp5Wl9LqiM3GhBsfJavfTDpkbeCrZ27Wp8eVqiMNjSayOz2XXuH6elcrjbvyLrKH3k1YznY2PHMjOTnHrC5JqWZHQ6MJFJeWsTczT4dDdwNjL76B/LiHCTm5jy3P6rAjStWWhkYT2Jd5guJSo53gbmLEeZdROvF/aFd4iO0vzCDj8EGrS1Kq2dDQaAJJzjun9BWv7mPImEl4Tn4K3+Lj7H5ppo5XpZSLNDSaQFJaDnab0COsrdWlqApihp6D/+WL8SzL5+BrN7MvaZvVJSnl9jQ0mkBSWi7d27fF28NudSmqkt4xwwi7xvHEeNY7t7J722aLK1LKvWloNIGk9Fx6aX+G24qK7k+XG16lyOZL9j/vZOf3iVaXpJTb0tBoZHmFJRw8mk8f7c9wa5269qDnTa+Sbw8g/8N72LpxrdUlKeWWNDQa2anhQ/RMw/1FdOrCWbcuI88rhJL/3Md36z+zuiSl3I6GRiPbnaZjTjUnoWEdGXDbcrK9O2Jb+Uc2fvmR1SUp5VY0NBrZrrRc2nja6Rzka3UpykVBIaEMnv0aR3y74f3lY/pODqUq0NBoZElpjuFDbDZ98VJz0i4wmOFzlpHZthe+6//GN5+8aXVJSrkFDY1Gtjs9V9+h0Uz5+QUw+s5XyPTvi9+3T7EuYZnVJSllOZdCQ0QmiUiSiCSLyLwqlouILHYu3yoig2tqKyLBIrJKRPY4P4MqLHvQuX6SiJxfYb6XiCwVkd0isktELqv7oTe+zNxCjpwo0jGnmrE2vm0Zc+fLZLQbQLvNi/jqg6VWl6SUpWoMDRGxA88BFwB9gatEpG+l1S4Aop0/s4AlLrSdB6wxxkQDa5zTOJdPB/oBk4DnndsB+COQYYzp5dzeV3U45iZz6s6p3nq7bbPm49OGuDkvkBY4mKDvn+fL95+zuiSlLOPKmcYwINkYs88YUwS8C0yttM5U4HXj8C0QKCIdamg7FVju/L4cmFZh/rvGmEJjzM9AsnM7ADcC/wtgjCkzxmTV7nCb1qkXL+nlqebP27sN8XcsIS14GCFbX+KLfyyyuiSlLOFKaHQCKg4DmuKc58o61bUNN8YcBnB+hlW3LREJdE7/RUS+E5H3RCS8qoJFZJaIbBaRzZmZmS4cYuNISsshpK0Xof7eltWgGo6Xtzfn3vEsh9uPpP2OZax5e4HVJSnV5FwJjapu+zEuruNKW1f35wFEAt8YYwYDG4Aq/681xiw1xsQaY2JDQ0Nr2F3jSUrP05FtWxhPTy/G376Yw2FjCd31JqvfnG91SUo1KVdCIwXoXGE6Ejjk4jrVtU13XsLC+ZlRw7aOACeBD5zz3wMG46bKygx79M6pFsnD05Pzbl/I4fB4wna/y+rX/4YpK7O6LKWahCuhsQmIFpEoEfHC0UmdUGmdBOB6511UI4Bs5yWn6tomADOc32cAH1WYP11EvEUkCkfn+kZjjAH+DcQ71xsP7Kzd4Tadg8dOcrKoVJ8Eb6HsdjsTbnuKtIhxhCWvYPUbGhyqdagxNIwxJcAc4HPgJ2CFMWaHiNwmIrc5V/sE2Iej0/olYHZ1bZ1t5gMTRGQPMME5jXP5ChyB8BlwhzGm1NnmAeBREdkKXAfMrcexN6qkNB1zqqWz2e2cd+uTHO44gfC977N6+V81OFSLJ45/wLdcsbGxZvPmpn9HwjNr9vDUqt1sf+x8/Lw9mnz/qumUlZay+uUHiUhdSVrXKUyY+Shi0+dmVfMmIluMMbGV5+t/2Y1kV3ounYPbaGC0Aja7nQm3zCct8gIifklg1at/1jMO1WJpaDSS3Wm5+lBfKyI2GxNu/h/SulxExIF/s+qVP2lwqBZJQ6MRFJaUsi/rhN451cqIzcaEG/9CWpeLiTj4Hw0O1SJpaDSCvRknKC0zOuZUK+QIjscqBMfDGhyqRdHQaASnxpzS221bp/Lg6DqFiIOfsOrlP2pwqBZDQ6MR7ErLxdMuRLVva3UpyiJiszFh5qOkd5tGRMqnGhyqxdDQaAS703PpEeqHp13/eFszsdk474ZHfg2Olx7S4FDNnv5WawRJaTp8iHIoD46oaUSkfsaql+ZRVlpac0Ol3JSGRgPLKSgm9Xi+DlSoyonNxnkzHiE96lIiUley+uUHNThUs6Wh0cD2aCe4qoIjOB7W4FDNnoZGA9MXL6kz0eBQLYGGRgNLSsvFz9uDToFtrC5FuSENDtXcaWg0sKS0XHqF+yFS1buklKo6OPSuKtVcaGg0IGMMSem5+iS4qlHl4Fj10jwNDtUsaGg0oIzcQo6fLKZ3uJ/VpahmQINDNUcaGg0oqbwTXM80lGs0OFRzo6HRgJL0zilVB78GxzQNDuX2NDQa0K60XEL9vQlu62V1KaqZ+fUBQA0O5d40NBrQ7vRcfahP1ZkGh2oONDQaSGmZYXe6vq1P1Y8Gh3J3GhoN5MDRkxSWlNFLzzRUPVUVHPoAoHIXGhoNJCktB9Axp1TDqDzI4aqX9Mlx5R40NBrIrrRcRCA6TENDNYzyu6q6X0aHQ3rGodyDhkYD2Z2eS9dgX9p42a0uRbUgYrNx3vV/dAbHKg0OZTkNjQayS1+8pBpJeXD0uFyDQ1lOQ6MBFBSXsj/rhN45pRqN2Gycd91DZPS4whEcSx/Q4FCW0NBoAMkZeZQZHT5ENS6x2Rh/3YNk9LyCDodXs/rF+zU4VJPT0GgAOnyIaipiszH+2gfJ6Pk7ItK+YNULcynV4FBNSEOjASSl5+LlYaNbiK/VpahWwNHH8RAZvabTIX0ta5bcrcGhmoyGRgNISsulZ6gfHnb941RN57xr55HZ+xoiMtax+vm7KCkutrok1Qrob7kGkJSmY04pa4y/5n4yz7qeDpnr+WKJBodqfC6FhohMEpEkEUkWkXlVLBcRWexcvlVEBtfUVkSCRWSViOxxfgZVWPagc/0kETm/iv0liMj22h9uw8s+WUxaToEOH6IsM/6qe8nqN5OIrES+eP5OiouLrC5JtWA1hoaI2IHngAuAvsBVItK30moXANHOn1nAEhfazgPWGGOigTXOaZzLpwP9gEnA887tnKrnUiCvLgfbGJLStRNcWW/clXdxJOZmIo58y5fPzaGosNDqklQL5cqZxjAg2RizzxhTBLwLTK20zlTgdePwLRAoIh1qaDsVWO78vhyYVmH+u8aYQmPMz0CyczuIiB9wL/DX2h9q49Axp5S7OPeKORwdMIuIoxtZ+9ztFBbmW12SaoFcCY1OwMEK0ynOea6sU13bcGPMYQDnZ5gL+/sL8BRwsrqCRWSWiGwWkc2ZmZnVrVpvSem5+Pt4EBHg06j7UcoV8ZfP5tig2UQc/46vnrmNgvxq/1dRqtZcCQ2pYp5xcR1X2rq0PxEZCPQ0xnxQQ3uMMUuNMbHGmNjQ0NCaVq+XU53gIlWVrVTTO+eSWWTH3kV4zlbWP3sL+SdPWF2SakFcCY0UoHOF6UjgkIvrVNc23XkJC+dnRg3bGgkMEZH9wHqgl4isdaH+RmOM0TGnlFsaO2UmeSPmEpq7k2+evYkTeTlWl6RaCFdCYxMQLSJRIuKFo5M6odI6CcD1zruoRgDZzktO1bVNAGY4v88APqowf7qIeItIFI7O9Y3GmCXGmI7GmG7AGGC3MSa+DsfcYNJyCsgtKNExp5RbGn3htZwc/QCheUl8++xN5OYct7ok1QLUGBrGmBJgDvA58BOwwhizQ0RuE5HbnKt9AuzD0Wn9EjC7urbONvOBCSKyB5jgnMa5fAWwE/gMuMMY45aPu+4qHz5Ex5xS7mnk+dPJj3uYkJN72fTcTWQfP2p1SaqZE2Nq6mJo3mJjY83mzZsbZdsvfLWX+Z/u4sdHJtLO17NR9qFUQ9i0NgGPLx7nuE9HBt76CkEhjdvXp5o/EdlijImtPF+fCK+H3Wm5RAT4aGAotzc0fgpm4l9pV3iIrUtmkJmeanVJqpnS0KgH7QRXzcngMZOwX/QEfsVH+GnpjaSlHrC6JNUMaWjUUUlpGcmZeRoaqlk5e9g42kx7Gt+SbJJfmUnqL3utLkk1MxoadbT/yEmKSsr0zinV7PQdNJqAy5/FqyyfX167if3JP1ldkmpGNDTqSF+8pJqzXjGxhFy1BBulpL01i+SdP1hdkmomNDTqKCktB5tAzzA/q0tRqk569D6bDte+RJl4cHTFbHZ+n2h1SaoZ0NCoo6T0XLq1b4uPp73mlZVyU1179KHbja9SaPcj/8N7+GHDGqtLUm5OQ6OO9MVLqqXoGBlFn1tfI9crFPPpA2xaW3nAB6V+paFRByeLSvjl6El6aSe4aiFCwzoycPZyjrXpgtcXj5L42dtWl6TclIZGHSRn5GGMvkNDtSyBQSEMu/M1Mv1645f4BF99sNTqkpQb0tCoAx1zSrVUfn4BjPn9q6QFDibo++dZ8/YCTFmZ1WUpN6KhUQe703Lx8bTRJdjX6lKUanA+Pm04984XOBw2ltBdb7Lq1T9RVuqWY4YqC2ho1EFSei7RYf7YbfriJdUyeXp6MeH2haR1uZiIA/9h1ZK7KS4usros5QY0NOpAx5xSrYHNbmfCjY+R1ec6OmSsY+1ifQug0tCotaMnisjMLdThQ1SrIDYb466ey/EhdxKWvZXExTPIPqbv5GjNNDRqSYcPUa1R3NSbKDjnTwSf3M8Pz19HRlqK1SUpi2ho1FJSmuNdy3q7rWptho+/FNuFf6dtURZ7ll7P/uQdNTdSLY6GRi0lpecR6OtJqL+31aUo1eTOHjGegN89j92UkPHmLWzfst7qklQT09CopaS0HHqH+yOid06p1qlXvyF0vmEZBXZ/ihLuZuMXH1pdkmpCGhq1YIxhd7q+eEmpTl17EDP7TY77dMV77WP69HgroqFRC6nH88krLNHQUAoICgll5F2vkx44iKDvn2fVa3/Rp8dbAQ2NWjh155R2givl0Ma3LePufJHDHc4jfN8/WfX8XRQVFlpdlmpEGhq1cGrMqWh9RkOpch6enky89Qkye19DRMY6vl40g+PHjlhdlmokGhq1sDs9l06BbQjw8bS6FKXcithsjL/mfvJGPUBIXjJbn7uGlF/2WF2WagQaGrWQpMOHKFWtUZOuwuPip/ApySF12Ux9hWwLpKHhouLSMvZm5umLl5SqQczQcwi/fhlFdl8KPvw9367+p9UlqQakoeGin7NOUFxqtBNcKRd07d6b/rPf4libKHy//gtr3nla76xqITQ0XLRLx5xSqlaCQkIZfffrpLUfRehPr7Pq+bsoKMi3uixVTxoaLtqdlovdJnQPbWt1KUo1Gz4+bZgw59nyO6u+WXgtmRmHrC5L1YOGhot2peXSvX1bvD3sVpeiVLNy6s6q/HP+TGD+Qfa8cA27d2yxuixVRy6FhohMEpEkEUkWkXlVLBcRWexcvlVEBtfUVkSCRWSViOxxfgZVWPagc/0kETnfOc9XRP4jIrtEZIeIzK/foddOUnqOXppSqh6Gj7+EgCtfBCBvxW3aQd5M1RgaImIHngMuAPoCV4lI30qrXQBEO39mAUtcaDsPWGOMiQbWOKdxLp8O9AMmAc87twOwwBjTBxgEjBaRC+py0LV1orCEg0fz9cVLStVTdN9B9LrtHY616Yrv139h9Rvz9f3jzYwrZxrDgGRjzD5jTBHwLjC10jpTgdeNw7dAoIh0qKHtVGC58/tyYFqF+e8aYwqNMT8DycAwY8xJY8yXAM5tfQdE1v6Qa293unaCK9VQ2odFMOaeNzgcdg5he95l9aJbyMk5ZnVZykWuhEYn4GCF6RTnPFfWqa5tuDHmMIDzM8zV/YlIIHAxjjOURqdv61OqYXl7t2Hi7P/j6Nm3En78B35cNJ2f92y3uizlAldCo6oXRxgX13Glba32JyIewDvAYmPMvio3IDJLRDaLyObMzMwadlezpPRcfL3sdA7yrfe2lFIOYrMRf9nt2C7+PzzLTnL0zZv475p/WV2WqoEroZECdK4wHQlUvmfuTOtU1zbdeQkL52eGi/tbCuwxxiw8U8HGmKXGmFhjTGxoaOiZj8xFSWm5RIf7Y7Ppi5eUamgxQ8+h+6y3Od4mkjZfPc6qZY9SUlxsdVnqDFwJjU1AtIhEiYgXjk7qhErrJADXO++iGgFkOy85Vdc2AZjh/D4D+KjC/Oki4i0iUTg61zcCiMhfgXbA3bU/1LpLSsuld7hfU+5SqVYlrENnxtzzJoc7TiT85w9Zu/AGjmalW12WqkKNoWGMKQHmAJ8DPwErjDE7ROQ2EbnNudonwD4cndYvAbOra+tsMx+YICJ7gAnOaZzLVwA7gc+AO4wxpSISCfwRx11Y34nIDyJyc33/AGqSlVfIkRNF9I4IaOxdKdWqeXu34fzbniBn2L0E5yWx67kr2b55ndVlqUrEmJq6GJq32NhYs3nz5jq3/yY5i2te/i9v3Tyc0T3bN2BlSqkz2b3jO9L/NY+A4iyyzrqWc393Fza7PljblERkizEmtvJ8fSK8BqfunNLRbZVqOr36DWbgXStIaz+c0J9eZ/X/3aiXq9yEhkYNktJyCWnrRai/t9WlKNWq+AcEMnHOcxwbPIf2uTvY9dx0tm9Zb3VZrZ6GRg12peuLl5SyithsnDPtZvx+t5RSmyelH/2eNW89Sak+RW4ZDY1qlJUZ9qTn6qUppSz26+WqkYQmvcXaJ6eTeqDKx7RUI9PQqEbKsXxOFpXqi5eUcgP+AYGc//vnyBv1AO3yD3L4latJ/OxtfblTE9PQqMautBxAhw9Ryp2MmnQVHW56m2zfLvglPsGqZ+eQffyo1WW1Ghoa1Tg1UGG0Xp5Syq106tKd+PveIbPPtYRlfcuOxZfx48YvrC6rVdDQqMautFw6B7fBz9vD6lKUUpXY7XbGX30fPpe/QIl4IR/fy8oX53EiL8fq0lo0DY1qOIYP0bMMpdxZnwHDiL33fdK6XEx46ud8//Ql/PhtkwyA3SppaJxBYUkpP2ed0P4MpZoB37b+TLz5L3hMewYjHsgnc/n8hfvJzTludWktjobGGezLPEFJmdExp5RqRvoNHsPQ+z4gvds0Ig6tZtvCS/hu/WdWl9WiaGicQfmLl/TylFLNik8bXybc+Chely2h2OaDx8p5fL7odjLSUqwurUXQ0DiDpPRcPO1C99C2VpeilKqDs84ewYj7/0Vm72sIPbqJA0suZ+0/l+i7OupJQ+MMktJy6RHqh6dd/4iUaq68vdsw/pr7CbvxHY769yb4xxdZ9+Rl7Nq60erSmi39jXgGyRl5OnyIUi1EZNdoJty7jJNxf8K7OIfC92/l8xfu59iR+r8OurXR92mcQUFxKScKSwjx09FtlWpJcnOOs2HFU4Qf+A9FNm/y+l7NqKk34+3dxurS3MqZ3qehoaGUapV+3rOd5IQnCc/+kRzPUDxHzWbYuVMRm16AAX0Jk1JK/UZUdH8mzF1O6aQnKLV50earx/jiyena31EDDQ2lVKs2aNRExsxLIDv2LtoWHKbo/VmsXHQrP+/ZbnVpbkkvTymllFNeXg4bPnieoOQP8TSFpLUfRZ+L7qRrjz5Wl9bktE9DKaVcdOxoFpsTlhD888d4mCLSwsbS96I5dI7qZXVpTUZDQymlaulIZhrf/fsFQn75D3ZTSnrYaLqPu4mefQdaXVqj09BQSqk6ysw4xPcJSwg5uBJPU0h6QAyho6/n7OHjW+zdVhoaSilVT9nHjrL5s9fw3f0hbUtzOOrdGc/BVzF03GV4ebesZ7o0NJRSqoEUFuazadV7lP7wLkFFhzhhDyCv6wT6nns1nbr2sLq8BqGhoZRSDcyUlfFd4kqObFxB+PHvEQwZAf3xH3Qpg8Zc1KzPPjQ0lFKqEaUfPsC2Ne/SZt9n+Jcc5aTNn9wu4+k6dDLR/QY3u74PDQ2llGoCpaWl/LBhFUc2vU/Yse+wUUa2ZxgFXeKIGjqZHn0GNIsA0dBQSqkmdvzYEbat/zeFSasJzdmJUMZxrwiKusTR6exx9Oo/FLvdbnWZVdLQUEopCx3NSmf7Nx9TlLSa0LxdCIYCmy/Hgs6mbY+R9IodR1hEpNVlltPQUEopN3HsaBa7Nn9Jzp5vCMj6jralOY753p0obB9D2y4D6NxnCJ27Rlt2KateoSEik4BFgB142Rgzv9JycS6/EDgJ3GCM+a66tiISDPwD6AbsB35njDnmXPYgcBNQCvzeGPO5c/4Q4DWgDfAJcJep4QA0NJRS7syUlbF39zYO/LiW0gMbCcpLxssUAlBg8yXbPxpbRH/aRZ5FWJdoOkZ2x8PTs9HrqnNoiIgd2A1MAFKATcBVxpidFda5ELgTR2gMBxYZY4ZX11ZEngCOGmPmi8g8IMgY84CI9AXeAYYBHYHVQC9jTKmIbATuAr7FERqLjTGfVle/hoZSqjkpLS1lf/J2DiV9R37KNryP7SKw8DCC43d1GXayvcIp9ItEgrrgExxJ28Bw/ILDCAwJJzgkvEFC5Uyh4eFC22FAsjFmn3ND7wJTgZ0V1pkKvO78V/+3IhIoIh1wnEWcqe1UIN7ZfjmwFnjAOf9dY0wh8LOIJAPDRGQ/EGCM2eDc1uvANKDa0FBKqebEbrfTo/fZ9Oh9dvm8nJxjpOzbxdHUZPIzfsYc/wWfvAMEHN2MbW8ZAAVAGnAYId/mR4FnO4be9TZt/QIatD5XQqMTcLDCdAqOs4ma1ulUQ9twY8xhAGPMYREJq7Ctb6vYVrHze+X5pxGRWcAsgC5dulRzaEop5f4CAoLoO3AkDBz5m/nFxUVkpqeSfSSd3KPp5GdnUpybRemJI0j+cXx9/Rq8FldCQ6qYV/ma1pnWcaWtq/tzeVvGmKXAUnBcnqphf0op1Sx5enrRMTKKjpFRTbZPV7rlU4DOFaYjgUMurlNd23TnJSycnxkubCuyivlKKaWaiCuhsQmIFpEoEfECpgMJldZJAK4XhxFAtvPSU3VtE4AZzu8zgI8qzJ8uIt4iEgVEAxud28sVkRHOu7Wur9BGKaVUE6jx8pQxpkRE5gCf47ht9lVjzA4Ruc25/AUcdzJdCCTjuOV2ZnVtnZueD6wQkZuAA8AVzjY7RGQFjs7yEuAOY0yps83t/HrL7adoJ7hSSjUpfbhPKaXUac50y637j5qllFLKbWhoKKWUcpmGhlJKKZdpaCillHJZi+8IF5FM4Jc6Nm8PZDVgOc2FHnfrosfdurh63F2NMaGVZ7b40KgPEdlc1d0DLZ0ed+uix9261Pe49fKUUkopl2loKKWUcpmGRvWWWl2ARfS4Wxc97talXsetfRpKKaVcpmcaSimlXKahoZRSymUaGlUQkUkikiQiyc73l7dYIvKqiGSIyPYK84JFZJWI7HF+BllZY2MQkc4i8qWI/CQiO0TkLuf8Fn3sIuIjIhtF5EfncT/mnN+ijxtAROwi8r2IfOycbvHHDCAi+0Vkm4j8ICKbnfPqfOwaGpWIiB14DrgA6AtcJSJ9ra2qUb0GTKo0bx6wxhgTDaxxTrc0JcBcY8xZwAjgDuffc0s/9kJgnDHmbGAgMMn5DpyWftwAdwE/VZhuDcd8yrnGmIEVns+o87FraJxuGJBsjNlnjCkC3gWmWlxTozHGfA0crTR7KrDc+X05MK0pa2oKxpjDxpjvnN9zcfwy6UQLP3bjkOec9HT+GFr4cYtIJHAR8HKF2S36mGtQ52PX0DhdJ+BghekU57zWJNz5pkScn2EW19OoRKQbMAj4L63g2J2XaX7A8YrlVcaY1nDcC4E/AGUV5rX0Yz7FACtFZIuIzHLOq/Ox1/jmvlZIqpin9yW3UCLiB/wTuNsYk+N4k3DL5nwT5kARCQQ+EJH+FpfUqERkMpBhjNkiIvEWl2OF0caYQyISBqwSkV312ZieaZwuBehcYToSOGRRLVZJF5EOAM7PDIvraRQi4okjMN4yxvzLObtVHDuAMeY4sBZHn1ZLPu7RwBQR2Y/jcvM4EXmTln3M5Ywxh5yfGcAHOC7B1/nYNTROtwmIFpEoEfECpgMJFtfU1BKAGc7vM4CPLKylUYjjlOIV4CdjzNMVFrXoYxeRUOcZBiLSBjgP2EULPm5jzIPGmEhjTDcc/z9/YYy5lhZ8zKeISFsR8T/1HZgIbKcex65PhFdBRC7EcQ3UDrxqjPkfaytqPCLyDhCPY7jkdODPwIfACqALcAC4whhTubO8WRORMcA6YBu/Xud+CEe/Ros9dhEZgKPj047jH40rjDGPi0gILfi4T3FenrrPGDO5NRyziHTHcXYBju6It40x/1OfY9fQUEop5TK9PKWUUsplGhpKKaVcpqGhlFLKZRoaSimlXKahoZRSymUaGkoppVymoaGUUspl/w/KFNmUd6LnogAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch, warmup_t=5, warmup_lr_init=1e-5)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"With warmup\");\n", "\n", "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"Without warmup\", alpha=0.8);\n", "\n", "plt.legend();" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "As we can see by setting up `warmup_t` and `warmup_lr_init`, the `cosine` scheduler first starts with a value of `warmup_lr_init`, then gradually progresses up to the `initial_lr` set in the optimizer which is 1e-4. It takes `warmup_t` number of epochs to go from `warmup_lr_init` to `initial_lr`. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `warmup_prefix`" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "Defaults to `False`. If set to `True`, then every new epoch number equals `epoch = epoch - warmup_t`." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "hidden": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABIRElEQVR4nO3dd3hUZfbA8e/JpBESCKGGJPTQQXoHcVG6gIICimABFMHe0F1ddXXXn6ILKIqINFdEXFEQsQDSCb1KTeihJQQChEBIeX9/3CEbMSQTyNxJwvk8zzyTuXPvPe8NYc7c+773PWKMQSmllHKFl6cboJRSqvDQpKGUUsplmjSUUkq5TJOGUkopl2nSUEop5TJvTzfA3cqUKWOqVKni6WYopVShsnHjxlPGmLJXLy/ySaNKlSps2LDB081QSqlCRUQOZbdcL08ppZRymSYNpZRSLtOkoZRSymVFvk9DqcIuNTWV2NhYLl265OmmqCLI39+f8PBwfHx8XFpfk4ZSBVxsbCxBQUFUqVIFEfF0c1QRYowhISGB2NhYqlat6tI2Ll2eEpGuIrJHRGJEZHQ274uIjHe+v01EmuS2rYiEiMhCEYl2PpdyLi8tIktEJElEProqTlMR2e7c13jR/0HqJnDp0iVKly6tCUPlOxGhdOnSeTqLzTVpiIgDmAB0A+oCA0Wk7lWrdQMinY/hwCcubDsaWGyMiQQWO18DXAJeBZ7PpjmfOPd/JVZXl45SqUJOE4Zyl7z+bblyeaoFEGOM2e8MMAvoDezMsk5vYIax5llfIyLBIhIKVMlh295AR+f204GlwEvGmAvAShGpcdWBhQIljDFRztczgD7AT3k64sLg4Eo4uAq8/cDb/4/PfkEQWA4Cy0NAGXDoFUallH1c+cQJA45keR0LtHRhnbBcti1vjDkOYIw5LiLlXGhHbDYx/kREhmOdkVCpUqVcdlsA/fg8xO9yYUWB4mWsBBJUAUrXgDKRUKam9QgsD/oNVSmVj1xJGtl96lxduela67iyratc3pcxZhIwCaBZs2aFq8pUWgqc2gvtnoEOL1iv0y45n1Mg5RwkxUHSCefzSev5bCwcioLUC//bl18JK3lUbAThza1HSDVNJCpPnnnmGSpXrszTTz8NQJcuXYiIiGDy5MkAPPfcc4SFhVGjRg127tzJ6NGj+f7776lZsyZ161pXozt27MiYMWNo1qzZDbfnn//8J6+88soN76coeeGFF1iwYAHdu3enevXqBAQEMHjwYLfEciVpxAIRWV6HA8dcXMc3h21Pikio8ywjFIhzoR3hubSj8DsVDSYdytcH3+LWw1XGwLljVtK58ojbDVtnwXrrPzjFQiC8mZVAqrSznh2uDbVTN6c2bdrwzTff8PTTT5ORkcGpU6c4d+5c5vurV69m7NixtGzZkl69egHw/fff07Nnz8ykkZ/cnTTS0tLw9i5Yl33T09NxOBzXfP/TTz8lPj4ePz8/t7fFld/MeiBSRKoCR4EBwH1XrTMPGOXss2gJnHUmg/gctp0HDAHecT7PzakRzv2dF5FWwFpgMPChC+0vXOKcXUXl6+V9WxEoGWY9qt/2v+UZ6RC/G2I3QOx66zl6IWCss5GqHaBGJ6jeCUpVzpfDUO7xxg872HnsXO4r5kHdiiX4+53X/ntr27YtzzzzDAA7duygfv36HD9+nDNnzhAQEMCuXbto3Lgx06ZNY8OGDdx3333MmzePZcuW8dZbb/Htt98C8M033/D444+TmJjI559/Tvv27bl06RIjRoxgw4YNeHt788EHH3Dbbbdl7uujj6wBlD179uT555/n559/5uLFizRq1Ih69erx5ZdfZrZz9uzZrFmzhg8++IBx48Yxbtw49u/fz759+xgyZAgrV67kzTff5IcffuDixYu0adOGTz/9FBGhY8eOtGnThlWrVtGrVy9++OEHGjduzMaNG4mPj2fGjBn861//Yvv27fTv35+33nqLgwcP0rNnT37//XcAxowZQ1JSEq+//jodO3akUaNGrFu3jnPnzjFlyhRatGiR7e/39ddfZ9++fRw9epQjR47w4osvMmzYMJYuXcobb7xBaGgoW7ZsYfv27YwePZqlS5eSkpLCyJEjefTRR+nVqxcXLlygZcuWvPzyy+zatYvAwECefvppWrduzXvvvUfHjh15+eWX8fLy4u23376hv5dck4YxJk1ERgG/AA5gijFmh4g85nx/IrAA6A7EAMnAQzlt69z1O8BsEXkEOAzccyWmiBwESgC+ItIH6GyM2QmMAKYBxbA6wIteJ/jJHWSIN//emE6XhmepV7HEjY+c8XJYSah8PWg6xFp2MREOroCYRRDzG+yeby0vXQNqdoV6d0FYU72UpahYsSLe3t4cPnyY1atX07p1a44ePUpUVBQlS5akYcOG+Pr6Zq7fpk0bevXqRc+ePenXr1/m8rS0NNatW8eCBQt44403WLRoERMmTABg+/bt7N69m86dO7N3795rtuWdd97ho48+YsuWLX96r0OHDrz33nsArFixgtKlS3P06FFWrlxJ+/btARg1ahSvvfYaAA888ADz58/nzjvvBCAxMZFly5YB8MMPP+Dr68vy5csZN24cvXv3ZuPGjYSEhFC9evXMJJqTCxcusHr1apYvX87DDz+cmVyys23bNtasWcOFCxdo3LgxPXr0AGDdunX8/vvvVK1alUmTJlGyZEnWr19PSkoKbdu2pXPnzsybN4/AwMDM38nrr78OgLe3N9OmTaNfv36MHz+en3/+mbVr1+ba7ty4dA5mjFmAlRiyLpuY5WcDjHR1W+fyBKDTNbapco3lG4D6rrS50IrbxX7C+HDZIT5cdoiqZYrTs2EoPRqGUqt8UP4NvSwWDHXutB7GWJfF9i22ksjaTyHqIwiuZCWPendD6C2aQAqAnM4I3Klt27asXr2a1atX8+yzz3L06FFWr15NyZIladOmjUv7uPvuuwFo2rQpBw8eBGDlypU88cQTANSuXZvKlSvnmDRyUqFCBZKSkjh//jxHjhzhvvvuY/ny5axYsSIz9pIlS3j33XdJTk7m9OnT1KtXLzNp9O/f/w/7u3KprUGDBtSrV4/Q0FAAqlWrxpEjRwgODs6xPQMHDgSsZHbu3DkSExOvuU3v3r0pVqwYxYoV47bbbmPdunUEBwfTokWLzJvufv31V7Zt28Z///tfAM6ePUt0dHSON+XVq1ePBx54gDvvvJOoqKg/JPfrVbAu3CnST+7g97RKPN6xOhEhAczfdowJS2L48LcYapQLpNctFbmnWTihJYvlX1ARKFvTerQaYZ2F7P4RdsyBqAmwapzVgV6/HzQepJewbkJt2rRh9erVbN++nfr16xMREcH7779PiRIlePjhh13ax5Xr7Q6Hg7S0NMC6Izk73t7eZGRkZL529eaz1q1bM3XqVGrVqkX79u2ZMmUKUVFRvP/++1y6dInHH3+cDRs2EBERweuvv/6H/RYv/sf+wyvt9fLy+kNfgZeXV2a/R05tvPoLXk5f+K61btY2GWP48MMP6dKlS46/g6tt376d4OBgTp48maftrkUnLCxILp3FcS6WPRmVaBdZhoEtKvHl0Fas++vt/KNPfUoX9+WDhXtp+85vPDJtPYt2niQtPSP3/eZVsWBofD8M+haej4Y7x1tnHSvGwLhb4Iu7YedcSE/N/9iqQGrbti3z588nJCQEh8NBSEgIiYmJREVF0bp16z+tHxQUxPnz53Pdb4cOHTL7Jfbu3cvhw4epVasWVapUYcuWLWRkZHDkyBHWrVuXuY2Pjw+pqdn/7XXo0IExY8bQoUMHGjduzJIlS/Dz86NkyZKZH+plypQhKSkp8xv79SpfvjxxcXEkJCSQkpLC/Pnz//D+119/DVhnUyVLlqRkyZLX3NfcuXO5dOkSCQkJLF26lObNm/9pnS5duvDJJ59kHvvevXu5cOHCn9bLas6cOSQkJLB8+XKefPJJEhMT83iUf6ZnGgVJ3G4A9phwRoT97w+sTKAfD7SqzAOtKnM4IZmvNxxm9oZYFs/YQIUS/tzbLJx7m0cQXiog/9sUEGL1gzQdYg3r3fwf2DQDZg+G4uWg0X3QZDCUrp7/sVWB0aBBA06dOsV99933h2VJSUmUKVPmT+sPGDCAYcOGMX78+Bw/nB9//HEee+wxGjRokHkN3s/Pj7Zt21K1alUaNGhA/fr1adIkc2Yihg8fTsOGDWnSpMkfOsIB2rdvz5EjR+jQoQMOh4OIiAhq164NQHBwMMOGDaNBgwZUqVIl2w/mvPDx8eG1116jZcuWVK1aNTPOFaVKlaJNmzaZHeE5adGiBT169ODw4cO8+uqrVKxY8U+X6YYOHcrBgwdp0qQJxhjKli3L999/f819njp1itGjR7N48WIiIiIYNWoUTz31FNOnT7/uYwaQa50eFhXNmjUzhaZy34YpMP8ZBgZ8xlcv3pvjqqnpGfy2O46v1h1m2d54BOhWP5RhHarRKCLYve3MSLf6PjZOg72/WEOEa3WHNk9Apdba95HPdu3aRZ06dTzdDJUHebkv5fXXXycwMJDnn89u5iR7ZPc3JiIbjTF/OgA90yhITu7kAsUoG14j11V9HF50qVeBLvUqEHsmmf+sOcyXaw/x4/bjtKgSwrAO1ehUuxxeXm74APdyQM0u1uPcMdgw1boPZM8Ca8RVmyetDnava48rV0oVTnqmUYCkTu7GtsPxbL5jNkPbV8vz9kkpaXy9/ghTVh7gaOJFqpUtzrD21bi7SRh+3m7+AL+cDFtnwuqP4MwBKFUFWo20Os593XDZ7CaiZxpFw9SpUxk3btwflrVt2zZz2LEn5eVMQ5NGQWEMqf+qwjfJTaj+8GRaVit93btKS8/gx+3H+WzFfn4/eo7Qkv6M+ksN7mkaga+3m8c+ZKRbI69Wj7duJAwsD+2ehaYPgo+/e2MXUZo0lLvlJWno6KmC4vwJfC4nstdEUC/s2qMsXOHt8KJ3ozB+GNWOLx5pQYWS/vz1u9/5y/tLmb3+iHtGXF3h5YC6vWDoInhwgTX31c8vwfjG1iWstBT3xVZKuZ0mjYLCOX3IuRKRBPrlT1eTiNA+sixzRrRh2kPNCSnuy4vfbuP2D5YxZ1Ms6RluPsus0hYenA9DfrCG7P74HHzYFDZO1+G6ShVSmjQKCmfSCAhvkO+7FhE61irH3JFtmTy4GQG+3jw7eyvdx61g+d74fI/3J1U7wMM/w6A5Vi2QH56Ej1vB7gXW3ehKqUJDk0YBcTF2O3EmmKqV3Xe3tYhwe93yzH+iHR/f34SLqekMnrKOB6euI/pk7jdi3WBwa1LEoYth4CwQL5g1EKbfCce3uje2UirfaNIoIFKP72B3RgQNw2+sP8MVXl5C9wahLHy2A3/rUYeNh87QddwK/vb9dhKS3NznIAK1usGI1dB9DJzcAZ/eCt+PhHPH3RtbXZdnnnmGsWPHZr7u0qULQ4cOzXz93HPP8cEHHzBv3jzeeecdwJoafefO/xX37NixI4ViQIoHpKSkcPvtt9OoUSO+/vprhg4d+offXUGjSaMgyEgn4Gw0e00EdUNL2BbWz9vB0PbVWPbCbQxqWYmv1h2h43tLmbR8H6nu7CwHq4ZHi2Hw5GbrpsDts+HDJrDsXe0sL2CuzDsFZNbT2LFjR+b7q1evpm3btvTq1YvRo0cDf04adkpPT/dI3JxcmWsrO5s3byY1NZUtW7bQv39/Jk+e7JY6JPlFk0ZBcPoA3hkpJAbWoHg+dYLnRUhxX97oXZ9fnu5A86oh/HPBbnqOX8n6g6fdH7xYMHT+B4xcB5F3wJK34ePWsG+J+2MXRj+Nhqk98vfx0+gcQ16Z4Rb+V08jKCiIM2fOkJKS8od6GqNGjWL16tXMmzePF154gUaNGrFv3z7AqqfRokULatasyYoVK/4U5/HHH2fevHkA3HXXXZkTIX7++ef87W9/A6BPnz40bdqUevXqMWnSpMxtAwMDM6f0iIqKIjAwkJdeeommTZty++23s27dOjp27Ei1atUyY1xp7xU9e/Zk6dKlmft77rnnaNKkCZ06dSI+/tp9fx07duTpp5+mTZs21K9fP3OerNdff53hw4fTuXNnBg8eTHx8PH379qV58+Y0b96cVatWERcXx6BBg9iyZUvm7+rKWdmhQ4eIjIzk1KlTZGRk0L59e3799dcc/63soEmjADBx1rc2r1DPTHt9RY1ygUx5sDmTHmhKUkoa90yM4oVvtnL6wmX3Bw+pCvfOsDrLMfBFH/jvw3D+hPtjqxxlV0/jyofzhg0brllP47333mPLli1Ur27NS3alnsbYsWN54403/hSnQ4cOmcnk6NGjmWcqWethTJkyhY0bN7JhwwbGjx9PQkICYNWuqF+/PmvXrqVdu3ZcuHCBjh07snHjRoKCgvjb3/7GwoUL+e677zLraeTkwoULNGnShE2bNnHrrbdm296r11+9ejUff/zxH2b93bhxI3PnzmXmzJk89dRTPPPMM6xfv55vv/2WoUOHUq5cOSZPnkz79u3/8LsCqFy5Mi+99BKPPfYY77//PnXr1qVz5865tt3ddBqRAiDp8HaKG6Fs1YaebgoAnetVoF1kGcYvjmHyiv0s3HWS0V1rc2+zCPdMS5JVjU4wIgpWjYUVH1gVBv/yN2g+VKclAej2jkfCuqueRlbt27dn7Nix7Ny5k7p163LmzBmOHz9OVFQU48ePB2D8+PF89913ABw5coTo6GhKly6Nw+Ggb9++mfvy9fWla9eugDWxop+fHz4+PjRo0CDb2Ffz8vLKrK8xaNCgzLZfS3a1M8CqyVGsmFXGYNGiRX+4ZHfu3LlcZwIeOnQo33zzDRMnTsy28JQn6JlGAZAcu41Dphx1K4d6uimZAny9Gd2tNgueak/NckGMnrOdez6NIiYuyf3Bffyh42h4PMqqZ/7Ti/DZbXBiu/tjq2xdXU+jVatWREVFZfZnuCK7ehpZhYWFcebMGX7++Wc6dOhA+/btmT17NoGBgQQFBbF06VIWLVpEVFQUW7dupXHjxpnTnfv7+/+hhraPj09mTYqs9TCu1MKAvNXsyK34mSv1MDIyMoiKimLLli1s2bKFo0ePEhQUlON+k5OTiY2NBSApyYb/ey7QpFEA+JzaxV5TydZOcFfVLB/E14+24r1+DYmJS6L7+BVMXLbPvXeVX1G6unW5qt8Ua2LESR1hyb8gzYbLZeoP3FVP42qtW7dm7NixmUljzJgxmZemzp49S6lSpQgICGD37t2sWbPmho4pp5odGRkZmVO6z5w5k3bt2uW4L1dqZ3Tu3Dmz5jng0pnDSy+9xP3338+bb77JsGHDXDkst9Ok4WmpFwm+FMupgGoU8y2Yl19EhHuaRbDw2Q7cVqss7/y0m76frGavu+/tsIJD/b5WR3m9u2HZO9ZZx7Et7o+tMl2pp9GqVas/LCtZsuQ162m89957NG7cOLMj3BXt27cnLS2NGjVq0KRJE06fPp2ZNLp27UpaWhoNGzbk1Vdf/UNbrkfWmh3PP//8H2p2FC9enB07dtC0aVN+++23XPtBrtTOeOyxx/j888+zXWf8+PGZfUB169Zl4sSJ2a53xbJly1i/fn1m4vD19WXq1Kl5P9B8phMWepg5thmZ1JEvwt/ggaFPe7o5uTLGMH/bcf4+bwdJl9J46vZIHu1QDW+HTd8/di+A+c/AhXho9wzc+iJ4++W+XSGmExbaLzAw0OXLQXmpnVFQ6YSFhUjiQetu6KDKt3i4Ja4REe68pSK/PtOBO+qW571f9nDXx6vdf0f5FbW7w8g10LC/VX7201v1jnKlbKRJw8MSD24lxfhQObK+p5uSJ2UC/ZhwfxM+ub8JRxMv0vPDlcyIOogtZ67FSsFdn8B938ClRPisE6wcCxk29LOom0J2ZxkjR46kUaNGf3hMnTqVpUuXFuqzjLzSIbeednIn+0xF6oSFeLol16Vbg1CaVinFC99s47W5O1iyO453+91C2SAbLhnV7GxNR/LDk7Do71YJ2rsmQslw98e2mTEm1xE8yr0KQrEkd8jrFz090/CwkknRHPOvhr9PwewEd0W5IH+mPdScN3rVY/W+BLqOXc7iXSftCR4QAvd+Ab0+gqOb4JM28Psce2LbxN/fn4SEBHvO4tRNxRhDQkIC/v6uF0jTMw0PMsmnCUlP4HKZWp5uyg0TEYa0qULr6qV58qvNPDJ9A4NaVeKv3eu6f1SYCDR5ACq3gTnD4b8PQfSv0O1d8C94w5jzKjw8nNjY2BynslDqevn7+xMe7vrZuSYND4rfv5lygL8bamh4Ss3yQcwd1ZYxv+zhsxUHWH/gDBPub0KNcoHuD166ulW3Y/l71uPIOrhnGoQWjDvtr5ePjw9Vq1b1dDOUAvTylEfFx2wGoEKNJrmsWbj4eTv4a4+6zHi4BaeSUuj10Uq+2xxrT3CHD9z2Cjz4I6RehMm3w4YpWuxJqXyiScODLh/bwTkTQPUaNT3dFLfoULMsC55qT/2wkjzz9VZe+u82Ll62adrqym3gsRVQtb11X8e3j8Clc/bEVqoI06ThQQGJezjiUwU/n6J7lbB8CX9mDm3JqNtqMHvjEfpMWGXP/FUAxctYw3I7/R12fG9NQ3J8mz2xlSqiNGl4iMnIIPTyAc6XiPR0U9zO2+HF811qMe2hFsQ7L1fN3XLUnuBeXtD+WXhwPqQmW5erNk6zJ7ZSRZAmDQ85eXQ/JUhGynu2hoadbq1ZlgVPtqd+xZI8NWsLb/yww/0VAq+o3AYeWwlV2sIPT8G8JyD12rOaKqWy51LSEJGuIrJHRGJE5E9lvsQy3vn+NhFpktu2IhIiIgtFJNr5XCrLey87198jIl2yLB8oItudMX4WkT/PlFZInIzZBBSe6UPyS4WS/nw5rCUPt63K1FUHuX/yWuLP21TetXgZuP+/0P552DQDpnaDszZ10CtVROSaNETEAUwAugF1gYEicnUB225ApPMxHPjEhW1HA4uNMZHAYudrnO8PAOoBXYGPRcQhIt7AOOA2Y0xDYBvwv1qNhczFI1ZtiLCaTT3cEvv5OLx47c66jO3fiG2xifT8cAWbDp+xJ7iXAzq9Cv2/hFPR1txVB5bbE1upIsCVM40WQIwxZr8x5jIwC+h91Tq9gRnGsgYIFpHQXLbtDUx3/jwd6JNl+SxjTIox5gAQ49yPOB/FxZpPoQRwLM9HXEB4n9pFHCGUDCnr6aZ4TJ/GYXw7og2+3l70/zSKL9cesu+u5zo9Ydhv1h3lM/rA6o90WK5SLnAlaYQBR7K8jnUuc2WdnLYtb4w5DuB8LpfTvowxqcAIYDtWsqgLZDtxvYgMF5ENIrKhoN5FW+pCDCf8q+e+YhFXr2JJfhjVjtbVy/DX737n5TnbuZxmUz9H2ZpW4qjdHX79qzUs93KyPbGVKqRcSRrZzZJ29Veya63jyrYuxRMRH6yk0RioiHV56uXsdmCMmWSMaWaMaVa2bMH7Jp+WepmItCMkBxfN+zPyKjjAl6kPNmfkbdWZtf4IgyavJSHJpn4OvyBr7qpOr1lzVk3tZlUJVEply5WkEQtEZHkdzp8vC11rnZy2Pem8hIXzOS6XfTUCMMbsM9Y1jNmAaxXtC5ij+37HV9JwVLh5Rk7lxuElvNClNuMGNGJrbCK9PlrFruM23YwnAu2fg4FfQUIMTLoNYjfaE1upQsaVpLEeiBSRqiLii9VJPe+qdeYBg52jqFoBZ52XnHLadh4wxPnzEGBuluUDRMRPRKpida6vA44CdUXkyqnDHcCuPB5vgZCwfwsApao28mg7CqLejcKY/Whr0jIy6PvJan7dccK+4LW6wSMLwdvXOuPY9o19sZUqJHJNGsaYNKxRSr9gfUjPNsbsEJHHROQx52oLgP1YndafAY/ntK1zm3eAO0QkGisBvOPcZgfWWcRO4GdgpDEm3RhzDHgDWC4i27DOPP55Y4fvGZeP/U66EcIjb67htq66JSKYeaPaEVkukEf/s5EJS2Ls6yAvXxeGLYHwZjBnKCx+U4s7KZWF1gj3gM3vdaf0xYNUem2np5tSoF1KTefF/25j3tZj9GlUkXf6NrSv7kjaZVjwnHU/R+2ecPck8C1uT2ylCgCtEV6AlEvex6kAHTmVG38fB+MGNOKFLrX4fssxHvh8LacvXLYnuLcv3Dkeuv4f7FlgXa46b+OlMqUKKE0aNktOOktoxklSQmp7uimFgogw8rYaTLivCVtjz3L3x6s4cOqCXcGh1WMwcBacirFqkZ/ckft2ShVhmjRsdjR6C15i8A+r7+mmFCo9Goby1bBWnLuUxl0fr2LdgdP2Ba/ZxSruZNLh8y5WLXKlblKaNGyWeGALAGWqNfZsQwqhppVL8f3jbQkp7sugyWvtmykXrOp/QxdDqSrw5b2wYap9sZUqQDRp2Cz95E4uGl8qVr16+i7likqlA5gzog2NKwXz1KwtfLg42r6RVSXD4OGfoEYnmP80LHxNR1apm44mDZsFJu4h1rsSDu+iW3jJ3YIDfPnikZbc3TiM9xfuZfS320mza4p1vyAY8BU0HwqrxllTj6TZdPe6UgWAfnLZrELKAQ4Et/J0Mwo9X28v3r/3FsJLFWP8bzHEJ6Xw0X2NCfC14U/a4Q3dx0BwJets40I8DPgS/Eu6P7ZSHqZnGjY6E3+cMiSSXqaOp5tSJIgIz3auxdt31WfpnjgGfmbjnFUi0PYpuPszOLwGpuicVermoEnDRsf2WvMZFY9o6OGWFC33t6zMxEFN2X38HH0/Wc2hBJuG5AI0vBfu/wYSD8PkOyCuUM5so5TLNGnY6PzhbQBUiGySy5oqrzrXq8DMYa1IvJhK309Wsy020b7g1W+DhxZARipM6QKHVtsXWymbadKwkVf8ThIJpEyFSp5uSpHUtHIpvh3RBn8fBwMmrWHZXhtrqYQ2tCY7LF7OKuq08+o5PZUqGjRp2KjkuWiO+VZFvPTX7i7VywYyZ0QbqpQuztDp65m31cZ+hlKV4ZFfIfQW+GYIbJxmX2ylbKKfXjYxGRmEpx7kfIlITzelyCtXwp9Zj7aicaVSPDVrMzOiDtoXPCAEBn8P1TvBD0/Big+0jKwqUjRp2OT44WiKyyUor4WX7FDC34cZD7egU+3yvDZ3B/9euNe+mwB9i1sFnRrcA4vfgF//pjcBqiJDk4ZN4mM2AVCyso6csou/j4OJg5rQr2k44xZH89rcHaRn2JQ4HD5w1yRo8ShEfQRzH4f0VHtiK+VGenOfTZKPbgegYs2mHm7JzcXb4cV7/RoSUtyXScv3cyb5Mh/c2whfbxu+L3l5Qbf/g+JlYMnbcPEM9JsKvgHuj62Um+iZhk18Tu3mBGUpEVza00256YgIr3Svw8vdajN/23Eemb6e5MtpdgWHW1+EHu/D3l/gP3fDpbP2xFbKDTRp2KT0hX2cLFbV0824qT16a3X+r28DVsWcYvDn6zh3ycbLRc2HQr8pELsept8JFxLsi61UPtKkYYPUyymEpR8hOVgLL3la/+aV+HBgE7bGJjJw0hr7ph0BqH+3Ndlh/B6rEqBOO6IKIU0aNoiN2YavpOMTqiOnCoIeDUOZNLgZMXFJ3PtpFCfOXrIveM3OMOhbOHcUpnSF0wfsi61UPtCkYYMEZ+GlUlUbebQd6n9uq1WOGQ+34OS5FPpNtHm+qirtYMg8SDlnJQ6dr0oVIpo0bJB67HfSjBfhkTrctiBpWa00M4e15EJKGvdMjGLPifP2BQ9rCg8usH6e2g2ObrQvtlI3QJOGDYqd2UOsIww/fx1qWdA0DA9m9qOtAeg/KYrtsTaObCpf16oE6BcE03vDoSj7Yit1nTRp2KDcxf0kBFT3dDPUNUSWD+K/j7Uh0M+b+yavYeOhM/YFD6kGD/8CQRWs4bj7ltgXW6nroEnDzZLOnaGiOcnl0jpyqiCrVDqA2Y+2pkygHw98vpaofTYOiS1R0ZpavVRVmNnfup9DqQJKk4abxe61pg/xD2vg4Zao3FQMLsbXj7YivFQxHpy6jqV74uwLHlgOHpwP5erArPth51z7YiuVB5o03OzcIavwUrkajT3cEuWKckH+zBremhrlAhk2YwO/7DhhX/CAEGtUVVgT+OYh2DbbvthKuUiThptlnNhBsvEjtLJeniosQor7MnNYK+qHleTxLzfZW5PDvyQMmgOV28Cc4bBxun2xlXKBJg03Czq7l1ifyng5HJ5uisqDksV8+OKRljStbNXk+O/GWPuC+wVadcdrdIIfnoS1k+yLrVQuNGm4WejlAyQG1vB0M9R1CPTzZvpDLWhbvQwv/Hcrs9Ydti+4TzEYMBNq9YCfXoCoCfbFVioHLiUNEekqIntEJEZERmfzvojIeOf720SkSW7bikiIiCwUkWjnc6ks773sXH+PiHTJstxXRCaJyF4R2S0ifa//0N3v1IkjhHCOjHJ1Pd0UdZ2K+TqYPKQZt9Ysy+g52/nCziqA3n5w73So2xt+eQVW/tu+2EpdQ65JQ0QcwASgG1AXGCgiV38KdgMinY/hwCcubDsaWGyMiQQWO1/jfH8AUA/oCnzs3A/AX4E4Y0xN5/6WXccx2+ZEtDVyqni43glemPn7OPj0gabcXqccr87dwecrbZwvyuEDfadA/X6w6HVY9q59sZXKhitnGi2AGGPMfmPMZWAW0PuqdXoDM4xlDRAsIqG5bNsbuNLLNx3ok2X5LGNMijHmABDj3A/Aw8C/AIwxGcaYU3k7XHslHbFGToXWbJLLmqqg8/N28PH9TelarwL/mL+TT5ftsy+4wxvungQNB1jFnH57S+uOK49xJWmEAUeyvI51LnNlnZy2LW+MOQ7gfC6X075EJNj5+h8isklEvhGR8tk1WESGi8gGEdkQHx/vwiG6h1fcTk5TgjIVIjzWBpV/fL29+PC+xvRsGMq/ftrNR79F2xfcywF9PobGD8Dy92DR3zVxKI9wJWlINsuu/mu91jqubOtqPG8gHFhljGkCRAFjstuBMWaSMaaZMaZZ2bJlcwnnPsFJMRz31cJLRYmPw4ux/RtxV+Mwxvy6l7GL9toX3MsBd46HZg/DqnHwy181cSjbuVIjPBbI+lU5HLh64Pq11vHNYduTIhJqjDnuvJR15fbba+0rAUgGvnMu/wZ4xIX2e0RGejrhqYfYVq6Xp5ui8pm3w4sx99yCw0sYuyiajAzDM3fURCS77zv5zMsLenwAXj6wZgKYDOj6L6usrFI2cOVMYz0QKSJVRcQXq5N63lXrzAMGO0dRtQLOOi855bTtPGCI8+chwNwsyweIiJ+IVMXqXF9njDHAD0BH53qdgJ15O1z7HD+0mwBJwauCFl4qihxewrt9G9K/WQTjf4thzK97MHZ96xeBbv8HrR6HtZ/ATy/qGYeyTa5nGsaYNBEZBfwCOIApxpgdIvKY8/2JwAKgO1andTLwUE7bOnf9DjBbRB4BDgP3OLfZISKzsRJCGjDSGJPu3OYl4AsRGQvEX4lTEMXFbCYMKFFZR04VVV5ewr/uboCXlzBhyT7SM+ClrrXsOeMQgS7/BPGCqI8gIx26j7HORJRyI1cuT2GMWYCVGLIum5jlZwOMdHVb5/IErLOF7LZ5G3g7m+WHgA6utNnTLh3dDkC4jpwq0ry8hLf71MfhBROX7SM9I4NXutexL3F0fgu8vGHVWDDp0OPfmjiUW7mUNFTe+Sbs5piUp2KJUrmvrAo1Ly/hH73r4yXCZysOkJ4Br/a0MXHc/rrVSb7ifauPo+c4TRzKbTRpuEnp5H3EFatGRU83RNlCRHijVz0cXsKUVQfIMIa/31nXvsTxl1dBHLD8XStx3PmhJg7lFpo03CDlUjLh6Uc5Xirbq2+qiBIRXutZFy+RzLvG7U0cf7X6OJa9Yw1S76WJQ+U/TRpuEBu9jeqSgU/F+p5uirKZiPC3HnUQYLLdiQPgtpetBLL0X4BxJg6dYVnlH00abnDmwBYAylTTwks3IxHhrz3qIAKfrTiAMYbXe9WzL3F0HA0ILP2ndamq9wRNHCrfaNJwg9TjO7hsHIRV1xKvNysRyRxFNWn5fgzwhq2J4yXrjGPJ29Y9HH0+1sSh8oUmDTcISNzDUUcEVX39PN0U5UEiwsvdaiPAp8v3Ywy82dvGxHHri4DAkres15o4VD7QpOEG5S/uJ7bELeisU0pEGN2tNgh8umw/BsObverj5WVX4njBms3tt7cAA30+0cShbogmjXx2LjGBCsRzoHQtTzdFFRAiwuiutRGEic4p1W1NHB1eAAR++4f1WhOHugGaNPLZsb2bKAEU08JLKgsR4aWu1hcJzySO561nTRzqBmnSyGdnD20FoHwNHTml/kgThyoKNGnkt5M7SDLFqBAR6emWqAJIE4cq7DRp5LPAc9HE+lShtt6Jq64hu8Txj9717RtVpYlD3QBNGvnIZGQQdvkAe0L+4ummqAJOE4cqrDRp5KNTJw5TliRM2TqebooqBDRxqMJIk0Y+OhG9ibJAUKVbPN0UVUgUjMRhnPdxoIlD5UqTRj66cGQbABW18JLKgyuJw2D4dNl+wO7E8YL1rIlDuUCTRj5yxO/iFMGUKRvq6aaoQubKDYCAJg5VoGnSyEfBSTEc96tGGU83RBVKmjhUYaBJI5+kp6URkXaILWX6ebopqhDTxKEKOk0a+eTYwV1ESCpeFep6uimqkMsucdg/VxWaOFS2NGnkk1P7NhEBlKrSyNNNUUXA1ZMcGmOdcXhkkkNj4K6JmjgUoEkj31w6+jsZRgivqXNOqfxxZVSVCHyydB8GeMvWxPG8Vchp8ZtY06pPBId+ZNzs9C8gn/id3s0xrwqEFw/ydFNUESIivNilFgJ8vNQ643i7j42Jo/1z1vPiN51nHJ9q4rjJ6b9+PimTvI/4gOqEe7ohqsgREV7oYp1xTFhi3QBof+IQWPyG9VoTx01N/+XzwaXkJMLSj3GsVFdPN0UVUSLC851rIQgfLYkBDG/3aWBj4njWulS16HXAwF2TNHHcpPRfPR/ERm+lhhh8KjbwdFNUESYiPNe5JiLw4W8xZGTAv+62MXG0ewYQWPR3MBlw92RNHDch/RfPB2cObAGgTHXtBFfuJSI8e0dNBBj/WwzpxvB/fRvisC1xPA3iBQtftRJH38/B4WNPbFUgaNLIB+kndpBifAirpvdoKPcTEZ7tXAsvL2HsomgyMgzv3XOLfYmj7ZPW8NtfXoGMdOg3Fbx97YmtPE6TRj4ISNxLrHcE1X30P46yz9O318QhwvsL95JhDGPuuQVvh03Fv1qPtM44fh4N3zwI90zTxHGT0KSRDypc2sehks2o7umGqJvOE50i8fIS3vtlD+kG/n2vjYmj1Qjw8oYFz8PsB+DeGeDtZ09s5TEu/XWJSFcR2SMiMSIyOpv3RUTGO9/fJiJNcttWREJEZKGIRDufS2V572Xn+ntEpEs28eaJyO95P9z8d/Z0POU4TXrp2p5uirpJjbytBi93q80PW4/x1KwtpKZn2Be8xTDo8QHs/Rm+HgSpl+yLrTwi16QhIg5gAtANqAsMFJGrL953AyKdj+HAJy5sOxpYbIyJBBY7X+N8fwBQD+gKfOzcz5X23A0kXc/BusPRvRsBKBbR0MMtUTezR2+tzt961OHH7cd5YuZmLqfZmDiaPwI9x0L0rzDrPki9aF9sZTtXzjRaADHGmP3GmMvALKD3Vev0BmYYyxogWERCc9m2NzDd+fN0oE+W5bOMMSnGmANAjHM/iEgg8CzwVt4P1T3OH9oKQIVILbykPGto+2q81rMuP+84weNfbiQlLd2+4M0egl4fwb7fYOa9cPmCfbGVrVxJGmHAkSyvY53LXFknp23LG2OOAzify7kQ7x/A+0ByTg0WkeEiskFENsTHx+e06o2L28k5AihXsap74yjlgofbVeUfveuxaFccw2ds5FKqjYmjyQPWxIYHV8KX90DKeftiK9u4kjSyG8dnXFzHlW1diicijYAaxpjvctkeY8wkY0wzY0yzsmXL5rb6DSlxLpqjPlURL5s6H5XKxQOtq/DO3Q1YHh3P0OkbuHjZxsRxywC4+zM4vAb+0xcunbMvtrKFK590sUBEltfhwDEX18lp25POS1g4n+Ny2VdroKmIHARWAjVFZKkL7Xcbk5FBWOoBzpWI9GQzlPqTAS0q8V6/W1i17xQPTVvHhZQ0+4I36Af9psDRjfBFH7h4xr7Yyu1cSRrrgUgRqSoivlid1POuWmceMNg5iqoVcNZ5ySmnbecBQ5w/DwHmZlk+QET8RKQqVuf6OmPMJ8aYisaYKkA7YK8xpuN1HHO+iTt2gBIkQzm9qU8VPP2ahjO2fyPWHTjNkCnrOH8p1b7g9fpYQ3CPb4MZvSH5tH2xlVvlmjSMMWnAKOAXYBcw2xizQ0QeE5HHnKstAPZjdVp/Bjye07bObd4B7hCRaOAO52uc788GdgI/AyONMTaeX7vuRPQmAIIq3+LhliiVvd6NwvhwYBO2HEnkgc/XcfaijYmjdg8YMBPidsP0OyHJzf2LyhZiTG5dDIVbs2bNzIYNG9yy7zUzXqXV/vGcfTKGkiHu7TtR6kb8uuMEo2ZuJrJ8IF880pKQ4jbevb3vN/jqPgiOgMHzoESofbHVdRORjcaYZlcv197bG+BI2E0cIZowVIHXuV4FJg1uSkxcEv0/jSLunI034VX/Cwz6Fs4dg6ndIPGwfbFVvtOkcQNCkmI44a+Th6jCoWOtckx9qDlHEy/Sf9IajiXaeBNelbYweC5cPA1TukHCPvtiq3ylSeM6paVeJjztCMnBNT3dFKVc1qZ6Gb54pAWnzqdwz8QoDifkeMtT/gpvBkPmQ9pFmNrd6utQhY4mjet0dP9O/CQVR4V6nm6KUnnStHIIM4e14sLlNO75dDUxcTbOyhPaEB5cABiY1t0aXaUKFU0a1+nUvs0AlKrayLMNUeo6NAgvyazhrUjPMAyYFMXOYzbehFeuNjz0E3gXg+k94ch6+2KrG6ZJ4zqlHttOuhHCI3W4rSqcalcowdePtsbH4cWASVFsPGTjTXilq8PDP0FAaes+jv3L7Iutbogmjevkd2YPRx0V8Q8I9HRTlLpu1csG8s1jrQkp7ssDn69lZfQp+4IHV4KHfoZSla25qnYvsC+2um6aNK5T2eR9nArQkVOq8AsvFcDsx1pTKSSAh6et59cdJ+wLHlQeHvwRKtS36nFsm21fbHVdNGlch+Sks1TMOEFKiBZeUkVDuSB/Zg1vRZ2KJRjx5Sa+2xxrX/CAEGs4buU2MGc4rJ9sX2yVZ5o0rsPR6K14icE/rL6nm6JUvgkO8OXLoS1pUSWEZ77eyhdRB+0L7hcE938DNbvAj8/Big/si63yRJPGdThzcAsAZapr4SVVtAT6eTP1oebcXqccr87dwUe/RWPbVEM+xaD/f6B+X1j8Bix8DYr4NEeFkSaN65BxYicXjS8Vq9TxdFOUynf+Pg4+GdSUuxqHMebXvbw5fycZGTZ9eDt8rHoczR6BVeNg3ihIt3Fad5Urb083oDAqnriHWO9KRHrrr08VTT4OL96/5xZKBfgyZdUBEpNTebdfQ3wcNnzP9HJAj/eheBlY9n9wMRH6fg4+/u6PrXKlZxrXITRlP4lBWnhJFW1eXsKrPevwQpdafLf5KMNn2FgFUARuewW6vQu758OX/bQKYAGhSSOPzsQfpwyJpJfRkVOq6BMRRt5Wg7fvqs/SvfEM+nwtZ5NtrMnR8lG4ezIcjoJpPSApLvdtlFtp0sijY3utwkvFIxp6uCVK2ef+lpWZcF8Ttsee5d5Pozhp59TqDe+BgbPgVDRM6QJnDtoXW/2JJo08On94KwChkU093BKl7NW9QShTH2pO7Jlk7v54NTFx5+0LHnkHDJlnlY39vDMc32pfbPUHmjTySOJ3kUggpStEeLopStmubY0yfP1oa1LSMuj7SRTrD9pY+zuiBTz8C3j5WFOr7/vNvtgqkyaNPCp5LpqjvlURL/3VqZtT/bCSzBnRhpDivgyavJaff7dx2pFytWHoQihVxZqvauvX9sVWgCaNPDEZGYSnHiSphI6cUje3SqUD+HZEG+qElmDElxvtvXu8REV4aAFUag3fDYeV/9abAG2kSSMPThyJJlAuQnktvKRUSHFfvhrWik61rbvH3/15t313j/uXtOqO1+8Hi16Hn16EDJuGA9/kNGnkwckYq/BSycpaQ0MpgGK+DiYOasrAFhF8vHQfz83eyuW0DHuCe/tZd4+3HgXrJsHswXDZxvK1NylNGnlwMdYqTVmxps45pdQV3g4v/nlXA567oyZzNh9l8JS1JCZftie4lxd0eRu6vgO7f7QqAeq9HG6lSSMPfBL2cIKylAgu7emmKFWgiAhPdIpkbP9GbDqUyN2frOZQwgX7GtBqhDXZ4cmd8FkniNttX+ybjCaNPCidFMPJYtU83QylCqw+jcP44pEWnL5wmbs+Xm1vCdk6Pa0O8vQU616O/Uvti30T0aThotTLKYSlHyE5uJanm6JUgdayWmnmjGhDCX9vBn62hh+3HbcveFgTGLrIGmH1n76w+T/2xb5JaNJw0dF92/GVdHxCdeSUUrmpVjaQOY+3pWFYSUbO3MTHS2PsG1kVXAke+QWqtIe5I2Hxm5BhU+f8TUCThotO7bdGToVUa+zhlihVOIQU9+U/Q1ty5y0VeffnPTw3eyuXUm0aFutf0qoE2GQIrHgfvhkMl23sYynCNGm4KPXYDtKMF2E1Gni6KUoVGv4+DsYPaMSzzpFVAz9bQ9x5myY7dPjAneOgy7+skVWfd4HEw/bELsI0abjI/8weYh1h+PkHeLopShUqIsKTnSL55P4m7D5+nt4freL3o2ftCg6tH4f7voHEQ/DZX+DwWntiF1EuJQ0R6Soie0QkRkRGZ/O+iMh45/vbRKRJbtuKSIiILBSRaOdzqSzvvexcf4+IdHEuCxCRH0Vkt4jsEJF3buzQ86b8xX0kFK9hZ0ilipRuDUL574jWCNBv4mp7O8gjb4ehi8EvyLqXY/OX9sUuYnJNGiLiACYA3YC6wEARqXvVat2ASOdjOPCJC9uOBhYbYyKBxc7XON8fANQDugIfO/cDMMYYUxtoDLQVkW7Xc9B5deF8IhXNSS6H6MgppW5EvYolmTuqHXVDSzBy5ib+vXCvffXHy9a0Ekel1jD3cfjlr1p//Dq4cqbRAogxxuw3xlwGZgG9r1qnNzDDWNYAwSISmsu2vYHpzp+nA32yLJ9ljEkxxhwAYoAWxphkY8wSAOe+NgHheT/kvIvdYxVe8g/T/gylblTZID++Gt6Kvk3CGbc4muFfbOTcJZuqAQaEWHNWNR8GUR/Bl33hQoI9sYsIV5JGGHAky+tY5zJX1slp2/LGmOMAzudyrsYTkWDgTqwzFLc7e8gq+FKuho6cUio/+Hk7GHNPQ/5+Z12W7Imjz0eriD5pU1Enhw/0GAO9PoRDq2FSRzi2xZ7YRYArSUOyWXb1+eS11nFl2zzFExFv4CtgvDFmf7Y7EBkuIhtEZEN8fHwu4XKXcXInycaP0MpaF1yp/CIiPNS2KjOHtuTcpVR6T1jFgu029nM0GQwP/wwm3Soju2WmfbELMVeSRiyQtUxdOHDMxXVy2vak8xIWzucrs4zlFm8SEG2MGXutBhtjJhljmhljmpUtW/baR+aiwLN7ifWpjJfDkfvKSqk8aVmtNPOfaE+tCkE8/uUm/vXTLtLSbboZL6wpDF8G4c3h+xHw4/OQZtNki4WUK0ljPRApIlVFxBerk3reVevMAwY7R1G1As46LznltO08YIjz5yHA3CzLB4iIn4hUxepcXwcgIm8BJYGn836o169iyn4SA3XklFLuUqGkP7OGt+L+lpX4dNl+Hpy6noSkFHuCB5aFB76HNk/A+s+s0VXnrv5erK7INWkYY9KAUcAvwC5gtjFmh4g8JiKPOVdbAOzH6rT+DHg8p22d27wD3CEi0cAdztc4358N7AR+BkYaY9JFJBz4K9YorE0iskVEht7oLyA3CSdjCeEcGeWuHjCmlMpPft4O3r6rAe/2bci6g6fpPn4F6w7YVIPc4Q2d34J+U+DEdpjYHmJs6TItdMS2+WA8pFmzZmbDhg3Xvf3vK+ZSf/Fgfu80g/rtrx40ppRyh9+PnmXUzE0cOXORZ++oyYhbq+PllV13pxvE74HZQyB+N7R/Djq+bCWVm4yIbDTGNLt6ud4RnoukI9sBqBCphZeUskv9sJL88EQ7utWvwHu/7OGhaTZeripbC4b9Bo0HwYoxMKM3nLOxg76A06SRC6/4nZymBGUqROS+slIq3wT5+/DhwMa81ac+UfsT6DF+JesP2nS5yjcAen8EfSbCsU0wsR3s+82e2AWcJo1cBJ+P5pifFl5SyhNEhEGtKjNnRBv8fbwYMGkNHy6OJt2uu8gbDYThS6F4Wfjiblj0xk0/ukqTRg4y0tMJTz1EUolITzdFqZvalctVPRqE8v7CvfT/NIojp5PtCX7lclWTB2DlB/D5HXAq2p7YBZAmjRwcP7SXAEnBq4IWXlLK04L8fRg/sDFj+zdiz4nzdB+3gu82x9pT3Mk3wLqDvP9/rNlyP+0AG6ZAER9IlB1NGjmIi9kIQMnKt3i4JUqpK/o0DmPBU+2pHRrEM19v5clZWzh70aa5q+rcCSOiIKIlzH8GZt0HF07ZE7uA0KSRg0vHfgcgrKbOOaVUQRIREsCs4a15vnNNftp+nG5jl7N6n00f3iVCYdAcq7hTzCL4uDXs/dWe2AWAJo0c+Cbs5piUJ7BEqdxXVkrZyuEljPpLJN+OaIOfj4P7PlvLq9//zoUUG6Y79/KyijsNWwLFy8DMe+D7x+HiGffH9jBNGjkofSGGuGI6ckqpguyWiGB+fLIdD7etyn/WHqLzv5ezKsams44K9a3RVe2fg62zYEIr2POTPbE9RJPGNaRcSiYs/RgXS2nhJaUKugBfb167sy7fPNoaP28v7p+8lpfnbOe8HXU6vP2g02swbLFVr+OrATBnOCTbdE+JzTRpXMPRmO34SDo+Fet7uilKKRc1qxLCgqfaM7xDNb5ef5gu/17O0j1xuW+YHyo2tmbMvfUl+P1bmNASdv1gT2wbadK4htP7NwNQumojzzZEKZUn/j4OXuleh29HtCHAz5sHp65n5MxNnDx3yf3BvX3htlesvo7A8vD1IJg5ABIPuz+2TTRpXEPqiR1cNg7CazT0dFOUUtehcaVS/PhkO569oyYLd56k0/vLmLrqgD21OkIbwvAlcMebcGAZfNQCVv67SNxNrknjGgLO7OGoIwIfXz9PN0UpdZ38vB082SmShc90oGnlUrzxw056T1jFliOJ7g/u8IG2T8HIdVCjEyx6HT5tDwdXuj+2G2nSuIYylw6SULy6p5uhlMoHlUsXZ9pDzZlwXxNOJaVw18ereOW77Zy+YMM3/+AIGPAlDPwaLifDtB7w3WNw/oT7Y7uB1tO4hksXL5B8PpGQcmFuaJVSylPOX0rl3wujmR51kABfB0/+JZLBbSrj521DOefLybD8PVj9ITh8od0z0HqkNU1JAXOtehqaNJRSN6Xok+f554JdLNkTT6WQAF7uVpuu9SsgYkOxp4R9sOjv1uiqEmHQ6e/Q4B7rpsECQpOGUkplY/neeN7+cRd7Tp6neZVS/K1HXW6JCLYn+MFV8MsrcHwLVGwCXf4JlVvbEzsXmjSUUuoa0jMMszcc4f1f93Aq6TLdG1Tg6dtrUrN8kPuDZ2TA9tlWrY7zx6BmN7jtZQj17ESpmjSUUioXSSlpTFq+nykrD3Dhchp3NqzIU7dHUr1soPuDX06GNR9b/R2XEqF2T+uej/KeKc2gSUMppVx05sJlPluxn2mrD3IpNZ0+jcN48i+RVClT3P3BL52FNZ9A1ARIOQf17oJbR0O52u6PnYUmDaWUyqNTSSlMWr6fGVEHSU039L6lIkPbV6NuxRLuD37xjJU41nwCly9AvT7Q5kkIa+L+2GjS8HQzlFKFWNz5S3y6bD9frTtM8uV02keWYXiHarSrUcb9o60uJEDUh7D+c+vMo3I7aPMERHZ262grTRpKKXWDzian8uW6Q0xddZD48ynUrhDE8A7V6NmwIr7ebh4ue+kcbP4Coj6Gc7FQpia0HgUN+4OPf76H06ShlFL5JCUtnblbjvHZ8v1ExyVRNsiPe5qGM6B5JSqVdvONeumpsON7WD0OTmyHgNJwy0Bo+iCUicy3MJo0lFIqnxljWLo3ni/XHOK33XFkGGgfWYaBLSpxe53y7j37MMaaDHH957BnAWSkWZeumj5o1TK/wbMPTRpKKeVGx89eZPb6WGZvOMLRxIuULu5L36bh3NmwIvXDSri37+P8SdjyJWyaAWcOQLFS1tnHrS9aP18HTRpKKWWD9AzD8uh4vlp7mN92x5GWYahSOoAeDUPp0aAidUKD3JdAMjLg4HLYOA0ORcFTW6/7jEOThlJK2Swx+TK/7DjB/G3HWb0vgfQMQ7WyxenZIJTbapejYXgwDi83JZC0FKsU7XXSpKGUUh6UkJTCzztOMH/rcdYeSCDDQHCAD21rlOHWmmW5tWZZypfI/1FQ10uThlJKFRBnLlxmRcwplu+NZ9neeOLPpwBQq3wQLaqG0LhSMI0rlaJK6QB7Zt3Nxg0lDRHpCowDHMBkY8w7V70vzve7A8nAg8aYTTltKyIhwNdAFeAgcK8x5ozzvZeBR4B04EljzC/O5U2BaUAxYAHwlMnlADRpKKUKMmMMu0+cZ9neeFZEx7PlcCIXLqcD1plI4wgrgdQNLUH1coFElCqGt8P9U6hfd9IQEQewF7gDiAXWAwONMTuzrNMdeAIrabQExhljWua0rYi8C5w2xrwjIqOBUsaYl0SkLvAV0AKoCCwCahpj0kVkHfAUsAYraYw3xvyUU/s1aSilCpP0DEN03Hm2HE5k8+FENh85Q3RcElc+qn0cQpXSxaleNpDq5YpTKSSAckH+lA3yo2yQH6WL++ZLUrlW0vB2YdsWQIwxZr9zR7OA3sDOLOv0BmY4v/WvEZFgEQnFOou41ra9gY7O7acDS4GXnMtnGWNSgAMiEgO0EJGDQAljTJRzXzOAPkCOSUMppQoTh5dQu0IJalcowYAWlQA4dymVmLgk9sUlsS/+Avvik9gbd55Fu06SlvHHL/4iEBLgS9kgP74d0Ybifq58zLvOlb2FAUeyvI7FOpvIbZ2wXLYtb4w5DmCMOS4i5bLsa002+0p1/nz18j8RkeHAcIBKlSrlcGhKKVXwlfD3oUmlUjSp9Md7LlLTMzh57hLx51OIO59C/JVHUgoJSSkE+OZ/CVtXkkZ2vTBXX9O61jqubOtqPJf3ZYyZBEwC6/JULvGUUqpQ8nF4EV4qgPBS9tUYd+XCVywQkeV1OHDMxXVy2vak8xIWzuc4F/YVnks7lFJKuZErSWM9ECkiVUXEFxgAzLtqnXnAYLG0As46Lz3ltO08YIjz5yHA3CzLB4iIn4hUBSKBdc79nReRVs7RWoOzbKOUUsoGuV6eMsakicgo4BesYbNTjDE7ROQx5/sTsUYydQdisIbcPpTTts5dvwPMFpFHgMPAPc5tdojIbKzO8jRgpDEm3bnNCP435PYntBNcKaVspTf3KaWU+pNrDbl1/x0iSimligxNGkoppVymSUMppZTLNGkopZRyWZHvCBeReODQdW5eBjiVj80pLPS4by563DcXV4+7sjGm7NULi3zSuBEisiG70QNFnR73zUWP++Zyo8etl6eUUkq5TJOGUkopl2nSyNkkTzfAQ/S4by563DeXGzpu7dNQSinlMj3TUEop5TJNGkoppVymSSMbItJVRPaISIyzfnmRJSJTRCRORH7PsixERBaKSLTzuVRO+yiMRCRCRJaIyC4R2SEiTzmXF+ljFxF/EVknIludx/2Gc3mRPm4AEXGIyGYRme98XeSPGUBEDorIdhHZIiIbnMuu+9g1aVxFRBzABKAbUBcYKCJ1Pdsqt5oGdL1q2WhgsTEmEljsfF3UpAHPGWPqAK2Akc5/56J+7CnAX4wxtwCNgK7OGjhF/bgBngJ2ZXl9MxzzFbcZYxpluT/juo9dk8aftQBijDH7jTGXgVlAbw+3yW2MMcuB01ct7g1Md/48HehjZ5vsYIw5bozZ5Pz5PNaHSRhF/NiNJcn50sf5MBTx4xaRcKAHMDnL4iJ9zLm47mPXpPFnYcCRLK9jnctuJuWdlRJxPpfzcHvcSkSqAI2BtdwEx+68TLMFq8TyQmPMzXDcY4EXgYwsy4r6MV9hgF9FZKOIDHcuu+5jz7Vy301Islmm45KLKBEJBL4FnjbGnLMqCRdtzkqYjUQkGPhOROp7uEluJSI9gThjzEYR6ejh5nhCW2PMMREpBywUkd03sjM90/izWCAiy+tw4JiH2uIpJ0UkFMD5HOfh9riFiPhgJYwvjTFznItvimMHMMYkAkux+rSK8nG3BXqJyEGsy81/EZH/ULSPOZMx5pjzOQ74DusS/HUfuyaNP1sPRIpIVRHxBQYA8zzcJrvNA4Y4fx4CzPVgW9xCrFOKz4FdxpgPsrxVpI9dRMo6zzAQkWLA7cBuivBxG2NeNsaEG2OqYP1//s0YM4gifMxXiEhxEQm68jPQGfidGzh2vSM8GyLSHesaqAOYYox527Mtch8R+QroiDVd8kng78D3wGygEnAYuMcYc3VneaEmIu2AFcB2/ned+xWsfo0ie+wi0hCr49OB9aVxtjHmTREpTRE+7iucl6eeN8b0vBmOWUSqYZ1dgNUdMdMY8/aNHLsmDaWUUi7Ty1NKKaVcpklDKaWUyzRpKKWUcpkmDaWUUi7TpKGUUsplmjSUUkq5TJOGUkopl/0/eoSoXPfI9lQAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch, warmup_t=5, warmup_lr_init=1e-5)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"Without warmup_prefix\");\n", "\n", "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch, warmup_t=5, warmup_lr_init=1e-5, warmup_prefix=True)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"With warmup_prefix\");\n", "\n", "plt.legend();" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "In the example above we can see how the `warmup_prefix` updates the LR annealing schedule. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `cycle_limit`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The number of maximum restarts in SGDR. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "#### `cycle_limit=1`" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "hidden": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAiX0lEQVR4nO3deXwV9b3/8dcnO1tYAwTCpoQlLCJEpGpbFSmLS6iWFu9tpbS9XK24dLvita239ba1dlGpiqWtFrtIaeuCXoUiKLYqxQCKsgQiIIQdLfsa+Pz+OEN/acgyQJLJOef9fDx4nDMz3++cz5clb74zc2bM3REREQkjJeoCREQkfig0REQkNIWGiIiEptAQEZHQFBoiIhJaWtQF1Ld27dp59+7doy5DRCSuLFmyZJe751Ren/Ch0b17d4qLi6MuQ0QkrpjZ+1Wt1+EpEREJTaEhIiKhKTRERCQ0hYaIiISm0BARkdBChYaZjTKzEjMrNbMpVWw3M5sabF9uZoNr62tmbcxsnpmtDV5bB+vbmtnLZrbfzB6q9DlDzOydYF9TzczOfOgiInK6ag0NM0sFHgZGAwXA9WZWUKnZaCA/+DUJmBai7xRgvrvnA/ODZYDDwLeAr1dRzrRg/yc/a1SoUYqISJ0I8z2NoUCpu68DMLOZQBGwskKbIuAJj91nfZGZtTKzXKB7DX2LgEuD/jOAV4A73P0A8Dcz61mxiGB/2e7+RrD8BDAWePH0hhzOr19bz4Gjx8lukk7LJul0aplF39xsmmUm/FdbRESqFeYnYGdgU4XlMuDCEG0619K3g7tvBXD3rWbWPkQdZVV8xinMbBKxGQldu3atZbdV+/3ijazZvr/SfqF722YM7tqaT/TrwMfyc2iSkXpG+xcRiUdhQqOq8waVn9xUXZswfcMKvS93nw5MBygsLDyjz/vLVz7O4WPH2Xv4GHsPHeP9Dw6yYsteVmzZw0urtvPnpWVkpacwvG8HJl7UnSHdWqNTLCKS6MKERhnQpcJyHrAlZJuMGvpuN7PcYJaRC+wIUUdeLXXUqaz0VLLSU2nfIoue7VswvG8HAI4dP8Hi9R8yd8U2nn1rC/+3fCvn5bXkPz52DmP655KSovAQkcQU5uqpN4F8M+thZhnAeGB2pTazgRuCq6iGAXuCQ0819Z0NTAjeTwCeramIYH/7zGxYcNXUDbX1qS/pqSlc3LMd3y3qzxt3Xs49Rf3Yd7icyb9fxienvU7xhg+jKEtEpN5ZmGeEm9kY4AEgFXjM3b9nZjcCuPujwQ/xh4hdzXQQmOjuxdX1Dda3BWYBXYGNwDh3/zDYtgHIJjZT2Q18wt1Xmlkh8GugCbET4Ld4LQMoLCz0hrhh4YkTztPLNnPf3NVs33uEqwbm8j/X9KNd88x6/2wRkbpmZkvcvfCU9WFCI541VGicdPBoOT9fuI5pC9+jRWYa3/vkAEb179hgny8iUheqCw19I7yONc1I4ysjevH8LZeQ2yqLG3+7hK/+4S0OHCmPujQRkbOm0KgnvTq04OkvX8ytw/N55q3NXPvI67z/wYGoyxIROSsKjXqUnprCV0f0YsYXhrJt72Gueeg1Xl2zM+qyRETOmEKjAXw0P4fnJl9CbsssPv/4Yp5cvDHqkkREzohCo4F0bduUp758ER/vlcOdT73Dwy+XkugXIYhI4lFoNKCmGWlMv6GQT57fmR/NLeG7z6/kxAkFh4jED919r4Glp6bwk3Hn0bppBo+9tp5jx09wT1F/3YJEROKCQiMCKSnGt67qS0ZaCo8ufI/01BS+fVWBgkNEGj2FRkTMjDtG9eZI+XEef20DGWkpTBnVR8EhIo2aQiNCZsa3ryrg2PET/HzhOrKz0rn5sp61dxQRiYhCI2Jmxnev6c++w+X8aG4JnVs1Yez5VT4mREQkcgqNRiAlxbjvUwPZsfcI3/jT27RvkclFPdtFXZaIyCl0yW0jkZmWyqOfG8I57Zrzn79ZQsm2fVGXJCJyCoVGI9KySTqPT7yAJhmp/McTxew+eDTqkkRE/oVCo5Hp1KoJj35uCNv2HOaWJ5dRfvxE1CWJiPyTQqMRGty1Nf87tj9/XbuLH85ZHXU5IiL/pBPhjdSnL+jCii17+MVf19O/c0uKBumKKhGJnmYajdg3ryrggu6tufOpd3hv5/6oyxERUWg0ZumpKUy9/nyy0lO5+XdLOXzseNQliUiSU2g0crktm/DTT5/H6m37+M5zK6IuR0SSnEIjDlzauz03XXouTy7exLNvbY66HBFJYgqNOPG1Eb0Y0q0133zmXTbvPhR1OSKSpBQacSItNYX7Pz2IEyecr816Sw9vEpFIKDTiSNe2Tbn7mn4sWvchv/zbuqjLEZEkpNCIM+OG5DGyXwd+PHcNK7fsjbocEUkyCo04Y2b84NqBtGyazldnvcXRct1mREQajkIjDrVplsH3PzmA1dv28cgrpVGXIyJJRKERp0YUdKBoUCceWlDKqq06TCUiDUOhEcf+5+p+tGqazjf+9DbHdDdcEWkACo041rpZBvcU9efdzXuZ/qquphKR+qfQiHOjB+QyZkBHHpy/lg27DkRdjogkuFChYWajzKzEzErNbEoV283Mpgbbl5vZ4Nr6mlkbM5tnZmuD19YVtt0ZtC8xs5EV1l9vZu8EnzHHzPQgbeDuq/uRmZrCXc+8g7u+9Cci9afW0DCzVOBhYDRQAFxvZgWVmo0G8oNfk4BpIfpOAea7ez4wP1gm2D4e6AeMAh4xs1QzSwMeBC5z94HAcmDyGY47oXTIzuK/RvfhtdIPeEb3phKRehRmpjEUKHX3de5+FJgJFFVqUwQ84TGLgFZmlltL3yJgRvB+BjC2wvqZ7n7E3dcDpcF+LPjVzMwMyAa2nPaIE9S/D+3KoC6tuOf5VfzjgJ4tLiL1I0xodAY2VVguC9aFaVNT3w7uvhUgeG1f077c/RhwE/AOsbAoAH5VVcFmNsnMis2seOfOnSGGGP9SUowfXDuAvYeO8YMXV0VdjogkqDChYVWsq3zgvLo2YfqG+jwzSycWGucDnYgdnrqzqh24+3R3L3T3wpycnFo+LnH0zc3mi5f0YFZxGUve/0fU5YhIAgoTGmVAlwrLeZx6WKi6NjX13R4cwiJ43VHLvgYBuPt7HjvbOwu4KET9SeXW4fl0zM7i7tnvclx3whWROhYmNN4E8s2sh5llEDtJPbtSm9nADcFVVMOAPcEhp5r6zgYmBO8nAM9WWD/ezDLNrAexk+uLgc1AgZmdnDqMAHQcppJmmWncdWVf3t28l98v3hh1OSKSYNJqa+Du5WY2GZgLpAKPufsKM7sx2P4o8AIwhthJ64PAxJr6Bru+F5hlZl8ENgLjgj4rzGwWsBIoB2529+PAFjP7DvCqmR0D3gc+Xwe/BwnnqoG5PLl4Iz+as5ox/TvStnlm1CWJSIKwRL+uv7Cw0IuLi6Muo8GV7tjHqAf+ynWD8/jhpwZGXY6IxBkzW+LuhZXX6xvhCapn+xZ84ZIezFqyiXfK9kRdjogkCIVGApt8eU/aNM3gO8+t0DfFRaROKDQSWHZWOt8Y2Zvi9//B88u3Rl2OiCQAhUaCG1fYhYLcbO59cTWHjh6PuhwRiXMKjQSXmmLcfXUBm3cf0u3TReSsKTSSwIXntOXKAblMW1jKtj2Hoy5HROKYQiNJTBndhxMn4KfzSqIuRUTimEIjSXRp05QJF3Xjj0vK9ExxETljCo0kMvmyfLKz0vn+C6t0Ca6InBGFRhJp2TSdW4fn89e1u1i4JjluGS8idUuhkWQ+N6wb3do25fsvrNJdcEXktCk0kkxGWgr/NbIPa7bv589Ly6IuR0TijEIjCY0Z0JHzurTi/nlrOHxMX/gTkfAUGknIzJgyqg9b9xxmxusboi5HROKIQiNJfeTctlzaO4eHXy5lz8FjUZcjInFCoZHE7hjVh31HynlkYWnUpYhInFBoJLG+udl88vzOPP7aBrbuORR1OSISBxQaSe4rV/TC3Zk6f23UpYhIHFBoJLkubZry7xd2Y1ZxGet3HYi6HBFp5BQaws2X9SQzLYWfzlsTdSki0sgpNIScFpl84eIePPf2FlZs0fPERaR6Cg0B4D8+dg4tm6Tz47m6dbqIVE+hIQC0bJLOjR8/l5dLdlK84cOoyxGRRkqhIf/0+Yu60655Jj/5i85tiEjVFBryT00yUpl82bm8se4DXi/dFXU5ItIIKTTkX1x/YVc6tczix38p0YOaROQUCg35F5lpqUy+PJ+lG3fzSoke1CQi/0qhIacYV5hH1zZNNdsQkVMoNOQU6akp3DY8nxVb9jLn3W1RlyMijYhCQ6pUNKgT5+Q044GX1nJCj4UVkUCo0DCzUWZWYmalZjaliu1mZlOD7cvNbHBtfc2sjZnNM7O1wWvrCtvuDNqXmNnICuszzGy6ma0xs9Vmdt2ZD11qkhbMNkq27+NFzTZEJFBraJhZKvAwMBooAK43s4JKzUYD+cGvScC0EH2nAPPdPR+YHywTbB8P9ANGAY8E+wG4C9jh7r2C/S08gzFLSFcN7ER+++Y88NIajmu2ISKEm2kMBUrdfZ27HwVmAkWV2hQBT3jMIqCVmeXW0rcImBG8nwGMrbB+prsfcff1QGmwH4AvAD8AcPcT7q4vE9Sj1BTj9it6sXbHfp5fviXqckSkEQgTGp2BTRWWy4J1YdrU1LeDu28FCF7b17QvM2sVLN9jZkvN7I9m1qGqgs1skpkVm1nxzp26bPRsjO7fkT4dW/DgS2spP34i6nJEJGJhQsOqWFf5WEV1bcL0Dft5aUAe8Jq7DwbeAH5c1Q7cfbq7F7p7YU5OTi0fJzVJCWYb63YdYPbbmm2IJLswoVEGdKmwnAdU/ulRXZua+m4PDmERvO6oZV8fAAeBp4P1fwQGI/VuZL8OFORm87MFpZptiCS5MKHxJpBvZj3MLIPYSerZldrMBm4IrqIaBuwJDjnV1Hc2MCF4PwF4tsL68WaWaWY9iJ1cX+yxb5k9B1watBsOrDy94cqZMDNuuyKf9ZptiCS9tNoauHu5mU0G5gKpwGPuvsLMbgy2Pwq8AIwhdtL6IDCxpr7Bru8FZpnZF4GNwLigzwozm0UsEMqBm939eNDnDuA3ZvYAsPPk50j9+0RBbLYxdf5arjmvE2mp+oqPSDKyRL9NRGFhoRcXF0ddRkKYu2Ib//mbJfxk3HlcNyQv6nJEpB6Z2RJ3L6y8Xv9dlNBOzjZ+tkBXUokkK4WGhGZm3H5FPhs+OMgzb+nchkgyUmjIaRkRzDYe0mxDJCkpNOS0nLySasMHB3UllUgSUmjIaftEQQf66nsbIklJoSGnzcy4bXhP1u86wHO6J5VIUlFoyBn5REHsnlQ/m1+qO+CKJBGFhpyRlBTjtuH5rNt1gOd0bkMkaSg05IyN7BebbUxdsFazDZEkodCQM5aSYtxyeT7rdh7Q8zZEkoRCQ87K6P4d6dWhOT9bUKpniYskAYWGnJWTs43SHft54d2tUZcjIvVMoSFnbcyAXM7NacbU+Ws12xBJcAoNOWupKcatw/NZs30/c1Zsi7ocEalHCg2pE1cN7MQ5mm2IJDyFhtSJ1BTjlst7snrbPv6ycnvU5YhIPVFoSJ25emAnerSLzTYS/eFeIslKoSF1Ji01hZsv68nKrXuZp9mGSEJSaEidGjuoE93aNuVBzTZEEpJCQ+rUydnGii17mb9qR9TliEgdU2hInfvk+Z3p0qYJUxdotiGSaBQaUufSU1OYfFlPlpft4eUSzTZEEolCQ+rFtYPzyGvdhAdf0mxDJJEoNKRenJxtvF22h1dKdkZdjojUEYWG1JtrB+fRuVUTHtCVVCIJQ6Eh9SYjLYXJl/fk7U27WbhGsw2RRKDQkHp13cnZhs5tiCQEhYbUq4y02Pc23tJsQyQhKDSk3n1qSGy2cb9mGyJxT6Eh9a7iuY1XNNsQiWsKDWkQOrchkhhChYaZjTKzEjMrNbMpVWw3M5sabF9uZoNr62tmbcxsnpmtDV5bV9h2Z9C+xMxGVvF5s83s3dMfrkQlIy2FW07ONvS9DZG4VWtomFkq8DAwGigArjezgkrNRgP5wa9JwLQQfacA8909H5gfLBNsHw/0A0YBjwT7OVnPtcD+MxmsROu6IbFvid//0hrNNkTiVJiZxlCg1N3XuftRYCZQVKlNEfCExywCWplZbi19i4AZwfsZwNgK62e6+xF3Xw+UBvvBzJoDXwX+9/SHKlFLT43NNpaX7WHBat2TSiQehQmNzsCmCstlwbowbWrq28HdtwIEr+1DfN49wE+AgzUVbGaTzKzYzIp37tShkMbk2sF5dG3TlJ/O02xDJB6FCQ2rYl3lf+3VtQnTN9TnmdkgoKe7P11Lf9x9ursXunthTk5Obc2lAaWnpnDr8HxWbNmrZ4mLxKEwoVEGdKmwnAdsCdmmpr7bg0NYBK8nj1dU1+cjwBAz2wD8DehlZq+EqF8ambGDYs8Sv3/eGk6c0GxDJJ6ECY03gXwz62FmGcROUs+u1GY2cENwFdUwYE9wyKmmvrOBCcH7CcCzFdaPN7NMM+tB7OT6Ynef5u6d3L07cAmwxt0vPYMxS8TSUlO4bXg+q7ftY86KbVGXIyKnodbQcPdyYDIwF1gFzHL3FWZ2o5ndGDR7AVhH7KT1L4Av19Q36HMvMMLM1gIjgmWC7bOAlcAc4GZ3P14HY5VG5OrzOnFuTjMeeGkNxzXbEIkblugnIwsLC724uDjqMqQKzy/fwuTfL+PB8YMoGlT52goRiZKZLXH3wsrr9Y1wicyY/rn06diC++etofz4iajLEZEQFBoSmZQU42uf6M2GDw7y1NLNUZcjIiEoNCRSV/Rtz3l5LXlw/lqOlOvUlUhjp9CQSJnFZhubdx9i1pubau8gIpFSaEjkPprfjqHd2/CzBaUcOqrZhkhjptCQyJkZXx/Zmx37jjDjjQ1RlyMiNVBoSKMwtEcbPt4rh2mvvMfew8eiLkdEqqHQkEbjGyN7s+fQMX7x6rqoSxGRaig0pNHo37klVw7M5Vd/W8/OfUeiLkdEqqDQkEblayN6caT8BI+8Uhp1KSJSBYWGNCrn5DTnU4Pz+N2ijWz6sMbHpohIBBQa0ujcPiIfM7h/3pqoSxGRShQa0ujktmzCxIt78PRbm1m5ZW/U5YhIBQoNaZRu+vi5ZGel88M5q6MuRUQqUGhIo9SyaTo3X3YuC9fs5PXSXVGXIyIBhYY0Wjd8pDudWmZx75zVeiysSCOh0JBGKys9la+M6MXysj08/87WqMsRERQa0shdOziPvrnZ/PDF1Rw+ppsZikRNoSGNWmqK8c0r+7J59yFmvL4h6nJEkp5CQxq9i3u247LeOTz0cikfHjgadTkiSU2hIXHhzjF9OXCknKnz10ZdikhSU2hIXOjVoQWfuaArv130Pu/t3B91OSJJS6EhceOrI3qRlZ7K9/5vVdSliCQthYbEjZwWmdxyeU8WrN7BwjU7oy5HJCkpNCSufP7i7nRr25R7nl/JseMnoi5HJOkoNCSuZKal8s0rCyjdsZ/fLXo/6nJEko5CQ+LOFX3bc0nPdtz/0lr+oUtwRRqUQkPijpnx7asL2H+knPvmlkRdjkhSUWhIXOrVoQWfv6g7M9/cyPKy3VGXI5I0FBoSt26/Ip92zTP51rMrdBdckQYSKjTMbJSZlZhZqZlNqWK7mdnUYPtyMxtcW18za2Nm88xsbfDausK2O4P2JWY2MljX1Mz+z8xWm9kKM7v37IYu8a5FVjr/PaYPb2/azaziTVGXI5IUag0NM0sFHgZGAwXA9WZWUKnZaCA/+DUJmBai7xRgvrvnA/ODZYLt44F+wCjgkWA/AD929z7A+cDFZjb6TAYtiWPsoM4M7d6GH85ZrZPiIg0gzExjKFDq7uvc/SgwEyiq1KYIeMJjFgGtzCy3lr5FwIzg/QxgbIX1M939iLuvB0qBoe5+0N1fBgj2tRTIO/0hSyIxM747th97D5dz74t6NKxIfQsTGp2BinP/smBdmDY19e3g7lsBgtf2YT/PzFoBVxOboZzCzCaZWbGZFe/cqW8OJ7o+HbP50kd78IfiTfx93QdRlyOS0MKEhlWxrvJZx+rahOl7Wp9nZmnAk8BUd19X1Q7cfbq7F7p7YU5OTi0fJ4ngtuH55LVuwn8//Q5HyvWwJpH6EiY0yoAuFZbzgC0h29TUd3twCIvgdUfIz5sOrHX3B0LULkmiaUYa94ztz3s7D/DzhVX+X0JE6kCY0HgTyDezHmaWQewk9exKbWYDNwRXUQ0D9gSHnGrqOxuYELyfADxbYf14M8s0sx7ETq4vBjCz/wVaAref/lAl0V3Wuz1XDczloQWlun26SD2pNTTcvRyYDMwFVgGz3H2Fmd1oZjcGzV4A1hE7af0L4Ms19Q363AuMMLO1wIhgmWD7LGAlMAe42d2Pm1kecBexq7CWmtlbZvals/0NkMTy7asLyEpPYcqfl+u7GyL1wNwT+x9WYWGhFxcXR12GNKA/LSnj6398m7uvLmDixT2iLkckLpnZEncvrLxe3wiXhHPd4M5c2juH++aU8P4HB6IuRyShKDQk4ZgZP7h2AGkpxh06TCVSpxQakpByWzbhriv7smjdh/xGz90QqTMKDUlYn7mgC5f2zuH7L6yidIeuphKpCwoNSVhmxn3XDaRpRiq3/2EZR8v1eFiRs6XQkITWPjuLH1w7kHc37+XB+WuiLkck7ik0JOGN6t+RcUPymPbKe7y54cOoyxGJawoNSQp3X9OPLm2acuuTy3QLdZGzoNCQpNA8M42Hrh/MB/uP8rU/vq3LcEXOkEJDksaAvJbcdWVfFqzewS/+qpsaipwJhYYklRs+0o3R/Tty39wSlryv8xsip0uhIUnFzPjhpwaS17oJN/12Kdv3Ho66JJG4otCQpJOdlc7PPzeE/UfKufG3S/TQJpHToNCQpNSnYzY/GXceyzbu5lvPvEui3+1ZpK4oNCRpjR6Qyy2X92RWcRm/fn1D1OWIxIW0qAsQidJXrujF6m37+O7zK+nUqgkj+3WMuiSRRk0zDUlqKSnG1PHnc15eK259chlLN/4j6pJEGjWFhiS9Jhmp/GpCIR1bZvGlGcWs36UHN4lUR6EhArRtnsmMiUMB+Owv/87m3YcirkikcVJoiAS6t2vGjIlD2Xv4GP/2i0Vs26PvcIhUptAQqWBAXktmfGEou/Yd4d9+uYgd+xQcIhUpNEQqGdy1NY9PHMrW3Ye5fvoituhQlcg/KTREqjC0Rxt+PfECduw9wqemva7HxYoEFBoi1bjwnLY8OWkYR4+fYNyjr/P2pt1RlyQSOYWGSA36d27Jn268iOZZaXxm+hvMfntL1CWJREqhIVKL7u2a8dRNFzOgc0tufXIZ339hFeXHT0RdlkgkFBoiIeS0yOR3XxrG54Z1Y/qr65jw+GK27tEJckk+Cg2RkDLSUrhnbH/uu24gS9/fzcj7X+WZZZt1h1xJKgoNkdP06Qu68MJtHyW/Qwtu/8Nb3PTbpZT942DUZYk0CIWGyBno0a4Zs/7zI9wxqg+vrNnB5T9ZyH1zVrPv8LGoSxOpV5boU+vCwkIvLi6OugxJYFt2H+JHc0t4etlm2jbL4LPDuvHZYd3IaZEZdWkiZ8zMlrh7YeX1oWYaZjbKzErMrNTMplSx3cxsarB9uZkNrq2vmbUxs3lmtjZ4bV1h251B+xIzG1lh/RAzeyfYNtXM7HR+E0TqQ6dWTbj/M4N49uaLGdSlFQ/OX8vF9y7gq7Pe4pWSHRwt15VWkjhqnWmYWSqwBhgBlAFvAte7+8oKbcYAtwBjgAuBB939wpr6mtl9wIfufm8QJq3d/Q4zKwCeBIYCnYCXgF7uftzMFgO3AYuAF4Cp7v5iTfVrpiENbd3O/Tz+2gaeWlrGgaPHaZGZxsd753B+19b065RN39xssrPS0P95pDGrbqYR5sl9Q4FSd18X7GgmUASsrNCmCHjCYwm0yMxamVku0L2GvkXApUH/GcArwB3B+pnufgRYb2alwFAz2wBku/sbwb6eAMYCNYaGSEM7J6c594ztz11X9uX193Yx993tvFyyg+eXb/1nm4zUFLKbpJGdlU5qisJD6sfzt15CZlpqne4zTGh0BjZVWC4jNpuorU3nWvp2cPetAO6+1czaV9jXoir2dSx4X3n9KcxsEjAJoGvXrjUMTaT+ZKWncnmfDlzepwMAO/YdZuWWvazZvo8PDxxj7+Fj7D10jBMJfl5RomPU/X9IwoRGVZ9a+W95dW3C9A37eaH35e7TgekQOzxVy+eJNIj2LbJo3zuLS3u3r72xSCMV5kR4GdClwnIeUPkGPNW1qanv9uAQFsHrjhD7yqulDhERqUdhQuNNIN/MephZBjAemF2pzWzghuAqqmHAnuDQU019ZwMTgvcTgGcrrB9vZplm1gPIBxYH+9tnZsOCq6ZuqNBHREQaQK2Hp9y93MwmA3OBVOAxd19hZjcG2x8ldiXTGKAUOAhMrKlvsOt7gVlm9kVgIzAu6LPCzGYRO1leDtzs7seDPjcBvwaaEDsBrpPgIiINSF/uExGRU5zVl/tERERAoSEiIqdBoSEiIqEpNEREJLSEPxFuZjuB98+weztgVx2WEw+SccyQnONOxjFDco77TMbczd1zKq9M+NA4G2ZWXNXVA4ksGccMyTnuZBwzJOe463LMOjwlIiKhKTRERCQ0hUbNpkddQASSccyQnONOxjFDco67zsascxoiIhKaZhoiIhKaQkNEREJTaFTBzEaZWYmZlQbPL09IZtbFzF42s1VmtsLMbgvWtzGzeWa2NnhtHXWtdc3MUs1smZk9Hywnw5hbmdmfzGx18Gf+kUQft5l9Jfi7/a6ZPWlmWYk4ZjN7zMx2mNm7FdZVO04zuzP4+VZiZiNP57MUGpWYWSrwMDAaKACuN7OCaKuqN+XA19y9LzAMuDkY6xRgvrvnA/OD5URzG7CqwnIyjPlBYI679wHOIzb+hB23mXUGbgUK3b0/scczjCcxx/xrYFSldVWOM/g3Ph7oF/R5JPi5F4pC41RDgVJ3X+fuR4GZQFHENdULd9/q7kuD9/uI/RDpTGy8M4JmM4CxkRRYT8wsD7gS+GWF1Yk+5mzgY8CvANz9qLvvJsHHTeyZQU3MLA1oSuxpnwk3Znd/Ffiw0urqxlkEzHT3I+6+nthzkIaG/SyFxqk6A5sqLJcF6xKamXUHzgf+DnQInpRI8JpoD7V+APgv4ESFdYk+5nOAncDjwWG5X5pZMxJ43O6+GfgxsYe8bSX2RNG/kMBjrqS6cZ7VzziFxqmsinUJfV2ymTUH/gzc7u57o66nPpnZVcAOd18SdS0NLA0YDExz9/OBAyTGYZlqBcfwi4AeQCegmZl9NtqqGoWz+hmn0DhVGdClwnIesSltQjKzdGKB8Tt3fypYvd3McoPtucCOqOqrBxcD15jZBmKHHi83s9+S2GOG2N/rMnf/e7D8J2IhksjjvgJY7+473f0Y8BRwEYk95oqqG+dZ/YxTaJzqTSDfzHqYWQaxE0azI66pXpiZETvGvcrdf1ph02xgQvB+AvBsQ9dWX9z9TnfPc/fuxP5sF7j7Z0ngMQO4+zZgk5n1DlYNB1aS2OPeCAwzs6bB3/XhxM7bJfKYK6punLOB8WaWaWY9gHxgcdid6hvhVTCzMcSOe6cCj7n796KtqH6Y2SXAX4F3+P/H9/+b2HmNWUBXYv/wxrl75ZNscc/MLgW+7u5XmVlbEnzMZjaI2Mn/DGAdMJHYfxwTdtxm9h3gM8SuFFwGfAloToKN2cyeBC4ldgv07cDdwDNUM04zuwv4ArHfl9vd/cXQn6XQEBGRsHR4SkREQlNoiIhIaAoNEREJTaEhIiKhKTRERCQ0hYaIiISm0BARkdD+H/nWtVfgFDYpAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch, cycle_limit=1)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*2)\n", "\n", "plt.plot([i for i in range(num_epoch*2)], lr_per_epoch);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### `cycle_limit=2`" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzeUlEQVR4nO3deXxU1fn48c8zk40QkgBZSEIgQBL2PSIiohiQRQSq1YL6hdpapCJiXb6i/vrVLra0aqu4gxtYl6J1QUUBcUFAREAEAgRC2MK+SNgJMzm/P+aiaQjJZJncWZ7368UrmTv33HmO3uTJc8+554oxBqWUUsobDrsDUEopFTg0aSillPKaJg2llFJe06ShlFLKa5o0lFJKeS3M7gB8LSEhwWRkZNgdhlJKBZQVK1YcMMYklt8e9EkjIyOD5cuX2x2GUkoFFBHZVtF2vTyllFLKa5o0lFJKeU2ThlJKKa9p0lBKKeU1TRpKKaW85lXSEJHBIpIvIgUiMrmC90VEplrvrxaRHlW1FZEmIjJfRDZZXxtb25uKyOcickxEnir3OT1FZI11rKkiIjXvulJKqeqqMmmIiBN4GhgCdABGi0iHcrsNAbKsf+OAZ71oOxlYYIzJAhZYrwFOAb8H7q4gnGet45/9rMFe9VIppVSd8OY+jV5AgTGmEEBE3gRGAOvK7DMCmGk866wvFZF4EUkBMippOwK4zGo/A/gCuNcYcxxYJCKZZYOwjhdrjPnaej0TGAl8XL0ue+eVxVs4XuImtkE4cQ3CSY2Lon1KLA0jg/7WFlVPjDG89s12TpS4iI3ynGetEhvSJjGGcKdeOVb+yZvfgGnAjjKvi4ALvdgnrYq2ycaY3QDGmN0ikuRFHEUVfMY5RGQcnoqEFi1aVHHYir2+bDsb9x4rd1zIaNqQHi0ac0XHZPplJdIgwlmj4yu158gp/t97a8/ZHhHmoG1yI/q0acoVHZvRPT0eh0OvxCr/4E3SqOhsLf/kpvPt401bb3l9LGPMNGAaQE5OTo0+b97vLuXUGTdHTp3hyMkzbDt4grxdR1i7s5j56/bwn5VFRIU7yG2fzE19MujZsjE6xKKqo8RVCsCfRnYit10SP5wooWDfMfJ2HWF10WFeXLSF5xcWktQokpHd0xhzUUuaN462OWoV6rxJGkVAepnXzYFdXu4TUUnbvSKSYlUZKcA+L+JoXkUcdSoq3ElUuJOkRlFkJjUit30yAGfcpXxTeIi5eXt4f9VOPlq9m67N4xjXrw1DOzfT5KG84ir1/D0TGxVGanwDUuMb0DE1jhHdPAV08ckzfL5hH3PW7ObFRVt4cdEWBndsxm8va0OntDg7Q1chzJsLp98CWSLSSkQigFHA7HL7zAbGWLOoegPF1qWnytrOBsZa348F3q8sCOt4R0WktzVrakxVbXwl3Omgb1YCfxrZia/vy+WPIzpy5JSLCa+v5GfPLGHFtkN2hKUCjNtKGs7zXHqKaxDOyO5pTBuTw8L/7c/NfVuxcNN+hj25iDtnrWJP8an6DFcpwIukYYxxAbcBc4H1wCxjTJ6IjBeR8dZuc4BCoACYDtxaWVurzRRgoIhsAgZarwEQka3AP4BfikhRmRlXvwVesD5nMz4aBK+OhpFhjLkogwV3Xsqj13Zld/FJrnn2aya+8R0Hj522Ozzlx84mjTAvxivS4htw39D2LJ58OeMvbcOH3+/mskc/56nPNuFyl/o6VKV+JJ4JT8ErJyfH1OcqtydKXDz3ZSHPfbGZRlFh/OXqzgzq2KzePl8FjrU7ixn25CKmj8lhYIfkarXdcegEf/14PXPW7KFr8zgeu64bmUkxPopUhSIRWWGMySm/Xef11bHoiDDuHJjNBxP70iwuilteXcFds77nRInL7tCUn3FVo9IoL71JNM/c0JOnr+/B9kMnuHLqV8xYspVg/yNQ2U+Tho+0bdaId2+9mNsvz+Td74q4+pklbD94wu6wlB9xl3ouK51vTMMbV3ZJYe7v+nFxZgIPzs7jrlnfc+qMu65CVOocmjR8KCLMwZ1XtOXlm3qx6/BJhj+9iEWbDtgdlvITLnfNK42ykhpF8cKYHO4cmM073+3k588tYefhk3URolLn0KRRDy7NTuSDiX1JbhTFmJe+4c1l2+0OSfmBqmZPVYfDIdyem8ULY3LYeuAEI55axNqdxbU+rlLladKoJy2bNuSdW/twSVYik99ZwzNfFOj15xD345iGs+7u6xnQIZl3b+1DuNPB6GlLWVp4sM6OrRRo0qhXDSPDmD4mh+FdU/n7J/k8/NF6Sks1cYSqnyqNuv0xzEpuxH9+24ek2EjGvLSMeXl76vT4KrRp0qhnEWEOHv9FN37ZJ4MXFm3hoQ/ytOIIUbWZPVWV1PgGvD2+D+1TYvntayv5ZK0mDlU3NGnYwOEQHryqA+P6tWbm19t4+KP1mjhCUF3MnqpM44YRvHbzhXRpHsfEN1by2Ya9PvkcFVo0adhERLhvSLsfK45H5+Vr4ggxvqw0zoqJDOOVm3rRrlks4/+1koUb9/vss1Ro0KRhIxFPxTG6Vwue/nwz0xYW2h2Sqkd1OXuqMnENwnn1171okxjDuFeX8932H3z6eSq4adKwmYjw8MhODOuSwl8/3sDs7326cK/yIz/dp+H7H8P46Ahe/XUvkmOj+PWM5Ww9cNznn6mCkyYNP+BwCI9d15VerZpw96zvdZpkiPix0qjDKbeVSYiJ5JWbegHwy5eX6YKaqkY0afiJyDAn0/8nhxZNoxk3czkF+47aHZLysfoY0yivVUJDXhibw+7iU/x6xnJdckRVmyYNPxIXHc4rN11ARJiTm2csp/jkGbtDUj7k69lT59OjRWOmju7O90WHue+dNToBQ1WLJg0/07xxNM/d2IOdh09y+xvf/XgJQwUfOyqNswZ1bMadA7J597udvLhoS71/vgpcmjT8UE5GE/4wvBNfbtzPI3Pz7Q5H+Uh9zZ46nwn9MxncsRl/mbNeF9JUXtOk4aeuv7AFN1zYgue+3MyHq3VGVTD6qdKw58fw7ASMrKRG3PbGSnYc0qX7VdU0afixB6/qSI8W8Uz+zxqdIhmE7K40wLMe2rQxPXG7Dbe98R0lLn10rKqcJg0/FhHm4MnrexDmFCa8vlJnugSZunqeRm21bNqQv/+8C9/vOMzfP9lgayzK/2nS8HNp8Q147Nqu5O06wsMfrbc7HFWH3KWliHguE9ltSOcUxl7UkhcWbWH+Ol2jSp2fJo0AkNs+mXH9WvPq0m3MWbPb7nBUHXGVGturjLLuv7I9ndJiufut7/XJf+q8NGkEiHsGtaVrejz3vbOGPcWn7A5H1QF3qbF1PKO8yDAnT1/fA5e7lLtmrdJnvagKadIIEOFOz3M4Slyl3P3W9/oDHQQ8lYZ//Qi2bNqQB4d3ZGnhIV5YpAtoqnP51xmrKtUqoSG/H9aBRQUHeGXJVrvDUbXkb5XGWdf2bM6gjsk8Oncj63cfsTsc5Wc0aQSY0b3SyW2XxJRPNrBxr65PFchcpaV+NaZxlojw16u7EBcdzh1vrtJZe+q/aNIIMCLC337ehUaRYdw5axUut86rD1T+WmkANGkYwd9/3oX8vUd5/NNNdoej/IgmjQCUEBPJn0d2Yu3OIzyvD24KWC63f82eKq9/2yRGXZDOtIWb+X7HYbvDUX5Ck0aAGtI5haGdm/HEp5vYpJepApK71NTbszRq6v4r25PUKIp73v6e0y69TKU0aQS0P47oRMNIJ/e8vVpXww1A/jh7qrzYqHD+enVnNu49xtOfFdgdjvID/n3GqkolxETy0PCOrNpxmJd0eeuA489jGmX1b5fE1T3SeOaLzeTtKrY7HGUzTRoBbnjXVAa0T+ax+fm6SmmA8dfZUxV5cFhH4qMjuO+dNVrVhjivkoaIDBaRfBEpEJHJFbwvIjLVen+1iPSoqq2INBGR+SKyyfrauMx791n754vIoDLbR4vIGuszPhGRhJp3PTiICH8c0RGnCP/3/lp9ClsACZRKAzxPlfy/qzqwuqiYmV9vtTscZaMqk4aIOIGngSFAB2C0iHQot9sQIMv6Nw541ou2k4EFxpgsYIH1Guv9UUBHYDDwjIg4RSQMeALob4zpAqwGbqthv4NKanwD7rqiLZ/n7+cjXZsqYPjb2lNVuapLCpdmJ/Lo3Hx2F+vaVKHKm0qjF1BgjCk0xpQAbwIjyu0zAphpPJYC8SKSUkXbEcAM6/sZwMgy2980xpw2xmwBCqzjiPWvoYgIEAvo04ksY/tk0Dktjj98sE6fLR4gAqnSAE9V++eRnXAbw4Pv59kdjrKJN0kjDdhR5nWRtc2bfSprm2yM2Q1gfU2q7FjGmDPAb4E1eJJFB+DFigIWkXEislxElu/fv9+LLgY+p0P469WdOXjstD4TIUB47tMIrGHF9CbRTMrNZt66vczN22N3OMoG3pyxFf0pVP7C+fn28aatV58nIuF4kkZ3IBXP5an7KjqAMWaaMSbHGJOTmJhYxccFj05pcYztk8Hry7azuuiw3eGoKgRapXHWzZe0om1yI/74wTpOlui9G6HGm6RRBKSXed2ccy8LnW+fytrutS5hYX3dV8WxugEYYzYbz2jvLKCPF/GHlN8NzKZpw0j+7/08XQnXz7lKSwnz85v7KhLudPDHER3Zefgkz3yh926EGm+SxrdAloi0EpEIPIPUs8vtMxsYY82i6g0UW5ecKms7GxhrfT8WeL/M9lEiEikirfAMri8DdgIdRORs6TAQ0EfZlRMbFc79Q9uxasdh3lqxo+oGyjaBWmkAXNi6KSO6pfL8l4X6/PoQU2XSMMa48MxSmovnl/QsY0yeiIwXkfHWbnOAQjyD1tOBWytra7WZAgwUkU14EsAUq00enipiHfAJMMEY4zbG7AL+ACwUkdV4Ko+/1K77weln3dO4IKMxf/skn8MnSuwOR51HoM2eKu/+oe0Jdwp//HCd3aGoeiTBPq8/JyfHLF++3O4w6t26XUcY9uRX3HBhS/40spPd4agKDH58IS2bRvP8/+TYHUqNTV9YyMNz1vPCmBwGdEi2OxxVh0RkhTHmnJMzsKZuKK91SI3lf3q35LVvtumDdPxUIKw9VZVfXpxBZlIMf/ponS5oGCIC+4xVlfrdwGxiG4Tzpw/X6Z3ifshdanAE8OUp8AyK/35YB7YdPMEri7faHY6qB5o0glh8dAR3DsxmyeaDzFu31+5wVDmBtPZUZS7NTiS3XRJPflbA/qOn7Q5H+ZgmjSB3fa8WZCfH8PBH6/XygZ9xuwN39lR5D1zZntMuN4/Ny7c7FOVjmjSCXJh1+WD7oRO8tGir3eGoMgJ99lRZrRNjGHtRBv9evoO1O3X59GCmSSMEXJKVyID2yTz9eQEHjunlA38RyPdpVGRibhZNoiN0DC3IadIIEfcPbcepM24e/3Sj3aEoi9sET6UBENcgnDsGZvPNlkN8un5f1Q1UQNKkESJaJ8Zww4UteGPZDgr26TPF/YFnTCO4fgRHXZBOm8SG/PXj9Zxxl9odjvKB4DpjVaVuz80iOtzJlI91FVx/4Co1Abn2VGXCnQ7uG9Kewv3HeXPZdrvDUT6gSSOENI2J5Nb+mXy6fh9LNh+wO5yQF2xjGmfltk+id+sm/PPTTRw5pc92CTaaNELMTRdnkBbfgL/MWa+r4NosWO7TKE9EeGBoBw4dL+G5LzbbHY6qY5o0QkxUuJO7B2WzducRPlitDz60S2mpodQQlJUGQOfmcYzslsqLi7awp/iU3eGoOqRJIwSN6JpG+5RYHp2XT4lLByvt4LampAZjpXHWXVe0pdQYnbEXZDRphCCHQ7h3cFt2HDrJ699sszuckOS2Lg0G2+ypstKbRHNj75bMWq4z9oJJ8J6xqlKXZidyUeumPPlZAcdOu+wOJ+S4SoO/0gC4rX8m0RFhPDJXlxcJFpo0QpSIMHlIOw4eL2H6wkK7wwk5bvfZSiO4k0bTmEhu6deauXl7WbHtB7vDUXVAk0YI65oez5WdU5j+VaGuTlrPXKWesaRgu0+jIr++pBUJMZH87eMNurxIENCkEeLuuiKb065Snv68wO5QQspPYxrBnzSiI8KYNCCLZVsP8cXG/XaHo2pJk0aIa50Yw7U9m/P6N9sp+uGE3eGEjFAZ0zjrFznppDdpwCOf5Ov9QQFOk4Zi0oAsEHji0012hxIyQmH2VFkRYQ7uHJjNut1HmLN2t93hqFoIjTNWVSolrgFjerfkPyuLdGpkPQm1SgNgeNc02iY34h/zNuLSxQwDliYNBcBvL2tDg3Anj83TG7Hqg9saCA+FMY2znA7hriuyKTxwnLdXFNkdjqohTRoK8EyNvPmS1ny8dg9rivTJa74WipUGwMAOyXRvEc8TCzZx6ow+fjgQadJQP7r5klbER4fzj/l6I5avuULkPo3yRIR7rmjL7uJTunR6gNKkoX7UKCqcW/q14fP8/azYdsjucILa2YHwULhPo7w+mQn0bt2Epz7fzMkSrTYCjSYN9V/G9mlJQkyEjm34mCvEZk+Vd9cVbTlw7DQzv95qdyiqmkLzjFXnFR0Rxq2XZbJk80GWFOiDmnzFHaJjGmddkNGES7MTee7LzRzVBzUFFE0a6hzXX9iCZrFRPDZ/oy774COuEJw9Vd5dV2Tzw4kzvLx4q92hqGrQpKHOERXuZGJuJiu2/aDLPvhIqFcaAF2ax3NFh2Smf1VI8QmtNgKFJg1VoWt7ppMW34DHtdrwCVcIrT1Vmd8NzOboKRcvLtKVlgOFV0lDRAaLSL6IFIjI5AreFxGZar2/WkR6VNVWRJqIyHwR2WR9bVzmvfus/fNFZFCZ7REiMk1ENorIBhG5puZdV5WJCHNwe24m3xcV89mGfXaHE3TOLo0eFqID4We1T4llaOdmvLR4K4dPlNgdjvJClWesiDiBp4EhQAdgtIh0KLfbECDL+jcOeNaLtpOBBcaYLGCB9Rrr/VFAR2Aw8Ix1HIAHgH3GmGzreF/WoM/KS1f3aE6LJtH881OtNuqaVho/mZSbzfESF9O/0mojEHjzZ04voMAYU2iMKQHeBEaU22cEMNN4LAXiRSSlirYjgBnW9zOAkWW2v2mMOW2M2QIUWMcB+BXwVwBjTKkxRqf3+FC408HEyzNZu/MI89fttTucoBLK92mU17ZZI4Z1SeXlxVs5dFyrDX/nTdJIA3aUeV1kbfNmn8raJhtjdgNYX5MqO5aIxFuv/yQiK0XkLRFJrihgERknIstFZPn+/TqQWxs/655GRtNo/vnpJl3Sug7p7Kn/Nik3i1Nn3Dy/cLPdoagqeJM0Kjqry//2ON8+3rT19vPCgObAYmNMD+Br4NGKDmCMmWaMyTHG5CQmJlbxcaoyYU4HkwZksX73Eebm7bE7nKChs6f+W2ZSDCO6pTFzyTYOHNOnSPozb5JGEZBe5nVzYJeX+1TWdq91CQvr69nR1vO1OQicAN61tr8F9ED53PCuabROaMgTC7TaqCs6pnGuiZdnctrlZpo+s96veZM0vgWyRKSViETgGaSeXW6f2cAYaxZVb6DYuuRUWdvZwFjr+7HA+2W2jxKRSBFphWdwfZnxjMR+AFxm7ZcLrKted1VNOB3C7blZbNhzVKuNOvJTpRHas6fKap0Yw8huacz8eqtWG36syjPWGOMCbgPmAuuBWcaYPBEZLyLjrd3mAIV4Bq2nA7dW1tZqMwUYKCKbgIHWa6z3Z+FJCJ8AE4wxZ1c1uxd4SERWA/8D3FWLvqtquKprqlYbdUgrjYrddnkmJa5SrTb8WJg3Oxlj5uBJDGW3PVfmewNM8Lattf0gnmqhojYPAw9XsH0b0M+bmFXdOltt3PHvVXySt4ehnVPsDimgua0n1+mYxn8rW2385pLWJDaKtDskVY7WxsprV3VNpXViQ57QmVS19mOloVNuz/FTtaEzqfyRJg3lNadDmJSbRf7eo3y8Vsc2akNnT53f2Wrj1aXb2H9Uxzb8jSYNVS3Duniqjak6tlErOqZROa02/JcmDVUtTodw++WeakNnUtWczp6qXOtEz30bry7V+zb8jZ6xqtp0JlXtna00tNA4v7PVxnSdSeVXNGmoanM6hNsuz2TDnqPM0zWpasRdWkqYQxDRrHE+bRJjuKprKjO/3sZBrTb8hiYNVSPDu6aS0TSaqQs26Qq4NeAqNTqe4YWJl2dyyuVm+ldb7A5FWTRpqBoJczqYeHkW63brCrg14XYbnTnlhcykRlzVJZWZX+sKuP5Ck4aqsRHdPNXGE1ptVJtWGt67PTeTk2fcvKDP2/ALmjRUjYU5HUzon0neriP6dL9qcpcawpz64+eNzKRGXNk5hRlL9Ol+/kDPWlUrI7unkd6kgVYb1aSVRvVMvDyL4yVuXlykYxt206ShaiXc6eC2/pmsLirmi3x94JW3zs6eUt5p26wRQzs345XFWyk+ccbucEKaJg1Vaz/r3py0+AY8rtWG17TSqL6Jl2dx9LSLFxdrtWEnTRqq1iLCPGMb3+84zJcbtdrwhrtUZ09VV/uUWAZ1TOblxVsoPqnVhl00aag68fOezUmNi9L7NryklUbN3J6bxdFTLl5ZvNXuUEKWJg1VJyLCHPy2fyYrtx9mUcEBu8Pxe6WlRtedqoGOqXEMaJ/Mi4sKOXJKqw076Fmr6sx1Oc1JiYviiU+12qiKVho1Nyk3iyOnXMxcstXuUEKSJg1VZyLDnPz2sjYs3/YDX28+aHc4fs2tSaPGOjePI7ddEi8s2sKx0y67wwk5mjRUnbouJ53k2EgeX7DJ7lD8mlYatXN7bhaHT5xhhlYb9U6ThqpTUeFOxl/ahmVbDmm1UQm9T6N2uqbHc1nbRF74qpDjWm3UK00aqs6N7tWCxEaRPLFgo92h+C2XWyuN2pqUm8UPJ84w8+ttdocSUjRpqDp3ttpYWniIbwq12qiIZ+0pTRq10b1FY/plJzJdq416pUlD+cQNF7YgISaSJ3Rso0KeMQ398autSblZHDpewr+WarVRX/SsVT7hqTZas2TzQb7desjucPyO3hFeN3q2bMwlWQlMW1jIiRKtNuqDJg3lMzdc2JKEmAie+FSrjfJ09lTdmZSbxcHjJby2dLvdoYQETRrKZxpEOBnXrzWLCg6wXKuN/6Kzp+pOTkYTLs5syvMLN3OyxG13OEFPk4byqRt7t6Rpwwgd2yhHK426NSk3mwPHSnjtGx3b8DVNGsqnoiPCuOXS1ny1SauNsnRMo271auWpNp77UqsNX9OkoXxOq41zee7T0B+/uqTVRv3Qs1b5nFYb59JKo+5ptVE/NGmoenFjb89Mqsd1JhVgjWnozX117o4BnmpD79vwHa+ShogMFpF8ESkQkckVvC8iMtV6f7WI9KiqrYg0EZH5IrLJ+tq4zHv3Wfvni8igCj5vtoisrX53lV2iI8K4pV8bFhUc0Ps20NlTvnJBRhP6Zibw/MLNet+Gj1SZNETECTwNDAE6AKNFpEO53YYAWda/ccCzXrSdDCwwxmQBC6zXWO+PAjoCg4FnrOOcjedq4FhNOqvsdUPvFla1oWtS6ewp35k0IEurDR/yptLoBRQYYwqNMSXAm8CIcvuMAGYaj6VAvIikVNF2BDDD+n4GMLLM9jeNMaeNMVuAAus4iEgMcCfw5+p3VdktOiKM8Ze2YXHBwZBfk0rHNHzngowmXJKVwPNf6ppUvuBN0kgDdpR5XWRt82afytomG2N2A1hfk7z4vD8BjwEnKgtYRMaJyHIRWb5///7KdlX17IYLW5LYKJJ/hni1oWtP+dYdA7I5eLxEV8D1AW/O2or+HCr/LM/z7eNNW68+T0S6AZnGmHeraI8xZpoxJscYk5OYmFjV7qoeNYhwcutlnhVwl2wO3WeJa6XhWz1bNuaytolMW7hZn+5Xx7xJGkVAepnXzYFdXu5TWdu91iUsrK/7qjjWRUBPEdkKLAKyReQLL+JXfmZ0rxaep/vND81niRtj9HGv9eCOAdn8oE/3q3PeJI1vgSwRaSUiEXgGqWeX22c2MMaaRdUbKLYuOVXWdjYw1vp+LPB+me2jRCRSRFrhGVxfZox51hiTaozJAPoCG40xl9Wgz8pmUeFOJvTPZNnWQywuCL2xDXepJ1FqpeFb3dLjyW2XxLSFhRw5dcbucIJGlUnDGOMCbgPmAuuBWcaYPBEZLyLjrd3mAIV4Bq2nA7dW1tZqMwUYKCKbgIHWa6z3ZwHrgE+ACcYYvVMnyPzignRS4qL4x/z8kKs2XFbS0Ps0fO93A7MpPnmGlxdttTuUoBHmzU7GmDl4EkPZbc+V+d4AE7xta20/COSep83DwMOVxLMV6ORF6MpPRYY5ue3yTB54dy1fbNxP/7ZJVTcKElpp1J9OaXFc0SGZF74qZGyflsRHR9gdUsDT6RvKNtf2TCe9SQMemxda1caPlYbOnqoXd16RzbESF9O/KrQ7lKCgZ62yTUSYg0m52azdeYS5eXvtDqfeaKVRv9o1i2VYl1ReXryVA8dO2x1OwNOkoWw1slsqrRMb8o/5+T/+Mg12rtJSAJ09VY/uGJDFqTNunvtis92hBDxNGspWYU4HvxuQzca9x/hwdfmZ3MFJK4361yYxhqt7NOfVpdvYU3zK7nACmiYNZbsrO6fQrlkj/jl/I2fcpXaH43Mu99kxDU0a9WlSbhbuUsOTn+lKy7WhSUPZzuEQ7hyYzdaDJ/jPiiK7w/G5HysNnXJbr9KbRPOLC9L597c72H6w0pWIVCU0aSi/MLBDMt1bxPPEgk2cOhPct+Xo7Cn73J6bRZhTQn7ts9rQs1b5BRHhnkFt2V18KuiXtNYxDfskx0Yxtk8G763aSf6eo3aHE5A0aSi/0adNAn0zE3jmi+BeZE5nT9lrfL82xESE8ei8fLtDCUiaNJRfuWdQWw4dL+HFr7bYHYrPaKVhr8YNIxjXrzXz1+3lu+0/2B1OwNGkofxK1/R4BndsxvSvCjkYpDdi/TSmoUnDLr/q24qEmAj+9smGkFqNoC5o0lB+5+5B2ZwocfHU5wV2h+ITP1Ua+uNnl4aRYdzWP5OlhYf4cqM+qK069KxVficzqRHX5aTzr6Xb2HEo+KZG6n0a/uH6C1uS3qQBUz7eQGmIrEZQFzRpKL90x4BsHCI8FoSDlXqfhn+ICHNw9xVt2bDnKO9/v9PucAKGJg3ll5rFRfGrvq14b9Uu1u4stjucOqWzp/zHVV1S6Zgay6NzN3LaFdz3B9UVTRrKb42/tA1xDcL52ycb7A6lTunsKf/hcAiTh7Rj5+GTvPp1cN8fVFc0aSi/FdcgnNv6Z/LVpgN8tSl4Bit19pR/uSQrkb6ZCTz1eQHFJ/WxsFXRpKH82pg+LWneuAEPf7Q+aJZO19lT/ue+oe0oPnmGZ4J0xl5d0rNW+bXIMCf3Dm7Hhj1HeWdlcCxmqJWG/+mYGsc1PZrz8uKtQTljry5p0lB+b1iXFLqlx/PovHxOlAT+8iJuayBcxzT8y11XZONwwN/nBt+MvbqkSUP5PRHhgSvbs/fIaV4IguVF9D4N/5QS14DfXNKaD77fpcuLVEKThgoIF2Q0YXDHZjz35Wb2HQnsJ6/pfRr+65ZL25AQE8HDH63X5UXOQ5OGChiTh7TjjLs04C8f6JiG/4qJDOPOgW1Zvu0HPly92+5w/JImDRUwMhIactPFrXh7RRGriw7bHU6NlRqdPeXPfnFBOu1TYpny8YagfyBYTehZqwLKbZdn0rRhBH/8YF3AXj7QMQ3/5nQI/zesAzsPn2TawkK7w/E7mjRUQImNCufuQZ7LBx8E6OUDvSPc/13UpimDOzbj2S82s6c4sMfQ6pomDRVwrstJp0NKLFPmrOdkSeBdPtAxjcBw/9D2uI0JumVsakuThgo4Tofw4FUd2FV8ime/CLw7ePU+jcDQomk0v7mkFe9+t5PlWw/ZHY7f0KShAtKFrZsyvGsqzy0sZNvB43aHUy1aaQSOCf0zSYmL4vfv5+Fyl9odjl/QpKEC1gNXtifcIfzhg3V2h1It7lKD0yGIaNLwd9ERYfx+WAfW7z7Ca99stzscv+BV0hCRwSKSLyIFIjK5gvdFRKZa768WkR5VtRWRJiIyX0Q2WV8bl3nvPmv/fBEZZG2LFpGPRGSDiOSJyJTadV0FuuTYKO4YkM1nG/bx6bq9dofjNZeVNFRgGNKpGX0zE3h0Xj4HgvS59dVRZdIQESfwNDAE6ACMFpEO5XYbAmRZ/8YBz3rRdjKwwBiTBSywXmO9PwroCAwGnrGOA/CoMaYd0B24WESG1KTTKnj88uIMspJi+MOHeQEzp95danQ8I4CICA8N78ipM27+9rEOintTafQCCowxhcaYEuBNYES5fUYAM43HUiBeRFKqaDsCmGF9PwMYWWb7m8aY08aYLUAB0MsYc8IY8zmAdayVQPPqd1kFk3Cngz+M6MiOQyd5OkCWtXa5tdIINJlJMfyqbyveWlHEsi2hPSjuTdJIA3aUeV1kbfNmn8raJhtjdgNYX5O8/TwRiQeuwlOhqBDXp00CV3dP47kvN7Np71G7w6mSu7RUk0YAmpSbRVp8A+5/dw0lrtAdFPcmaVR0dpe/Ffd8+3jTtlqfJyJhwBvAVGNMhbdrisg4EVkuIsv37w+eJ76p83vgyvY0jAzj/nfXUOrnD2ty6eWpgBQdEcafR3aiYN8xpi3cbHc4tvEmaRQB6WVeNwd2eblPZW33WpewsL7u8/LzpgGbjDGPny9gY8w0Y0yOMSYnMTHx/D1TQaNpTCT3D23Pt1t/YNbyHVU3sJFbB8IDVv92SVzZJYWpnxWw5UBgTfWuK94kjW+BLBFpJSIReAapZ5fbZzYwxppF1Rsoti45VdZ2NjDW+n4s8H6Z7aNEJFJEWuEZXF8GICJ/BuKAO6rfVRXsru3ZnF6tmvCXOevZf9R/Z7l4Kg2d7R6oHhzWgUing//33pqAXf+sNqo8c40xLuA2YC6wHphljMkTkfEiMt7abQ5QiGfQejpwa2VtrTZTgIEisgkYaL3Gen8WsA74BJhgjHGLSHPgATyzsFaKyCoRubm2/wFU8BAR/vKzzpw6U8pDs/OqbmATrTQCW1JsFPcOacfigoO8tTw4HkFcHWHe7GSMmYMnMZTd9lyZ7w0wwdu21vaDQO552jwMPFxuWxEVj3co9aPMpBgmDcjikbn5DFuzmyGdU+wO6Rw6phH4ru/Vgg++38WfPlpHv+xEmsVF2R1SvdEaWQWdcf1a0yktlt+/v5ZDx0vsDuccOnsq8Dkcwt+u6cIZdyn3vxtal6k0aaigE+508MjPu1J88gx//MD/LlPpfRrBISOhIfcMasdnG/bx3qqddodTbzRpqKDUPiWWCf0zeW/VLubl7bE7nP/iLjX6fPAg8cs+GfRs2ZiHZq9jb4A/u95bmjRU0Lr1skw6pMRy3ztr/Go2lWftKf3RCwZOh/DIz7tw2uXmnrdXh8RlKj1zVdCKCHPw+KhuHD3t4t7/+M8PtK49FVxaJ8bwwND2LNy4n5lfb7M7HJ/TpKGCWnZyIyYP9lx3fn2Zfyxt7dKB8KBzY++WXNY2kb/MWU/BPv9fyqY2NGmooPfLPhn0zUzgzx+up3D/MbvD0UojCIkIf7+mC9ERTu7496qgXptKk4YKeg6H8Oi1XYkMdzDxje9sX0Jdn6cRnJJio5hyTRfW7jzCI3ODdwl1TRoqJDSLi+LRn3clb9cR/jJnva2xaKURvAZ1bMaYi1oy/astzA+gB4NVhyYNFTIGdEjmN5e0YubX25izZrdtcXju09AfvWB1/9D2dEqL5e63vqfohxN2h1Pn9MxVIeV/B7ejW3o89769mm0H7VmlVCuN4BYV7uSp0T1wlxomvvEdZ9zBNb6hSUOFlHCng6eu747DIdzy6gpOlLjqPQZXaSlOvbkvqGUkNGTKNZ35bvth/vzhOrvDqVOaNFTIad44mqmju7Nx71Hueav+79/QSiM0DOuSym8uacWMr7cx61v/fsZLdWjSUCHp0uxE7h3cjo/W7OaZL+r3KWw6eyp03Du4HZdkJfD/3lvLyu0/2B1OndCkoULWuH6tGd41lUfn5fPZhvqb6aKVRugIczp4cnR3msVFMf7VFewpDvz1qTRpqJAl4lneumNqLBNf/461O4vr5XN17anQEh8dwfQxORw/7eKmV77l6KkzdodUK3rmqpDWIMLJi2MvID46gpte+ZYdh3w/RVIrjdDTtlkjnrmxJxv3HuXW11YG9IwqTRoq5CXHRvHKTRdw+oybm175lsMnfPvgJpdb154KRZdmJ/LXqzvz1aYD3PdO4D64SZOGUkBWciOmjclh+8ET/HrGco6f9t1UXK00Qtd1OelMys3i7RVFTPlkQ0AmDk0aSll6t27KE6O6sWrHYX4941tOlvhmjSpXqdH7NELYHQOyuOHCFjz/ZSGPf7rJ7nCqTZOGUmUM6ZzCP67ryjdbDjHu1eU+WdxQK43QJiL8aUQnru3ZnCcWbOLpzwvsDqlawuwOQCl/M6JbGqddpfzv26u55dUVPHdjTxpEOOvk2MYYnT2lcDiEKdd04Yy7lEfm5gNw62VtEPH/Pyb0zFWqAtflpDPl6s4s3LSfG1/8ps4Gx0utS9haaSintWT/iG6pPDI3n4c/Wk9pqf+PcWjSUOo8RvVqwdPX92BNUTG/eH5pndyY5Sr1TLXU2VMKPDf//fO6boy9qCUvLNrC3W9/7/fTcTVpKFWJoZ1TePmmCyj64QRXP7O41jcAuq2/JLXSUGc5HMJDwzvyuwHZvLNyJ7+qh2nftaFJQ6kqXJyZwL9vuQiAa55dwnvf7azxsVxW0tBKQ5UlIkwakMXfrunMN4WHGP7UYjbsOWJ3WBXSpKGUFzqlxTF7Yl+6psdzx79X8ccP1nHaVf2ZVW63Vhrq/H5xQQveGNebU2fcXP3MEt5fVfM/UHxFk4ZSXkqIieS1my/kl30yeGnxFoY/uZi8XdW7XPVjpeHUHz1VsZ4tG/PBxL60T4ll0puruO31lfxw3H8uV+mZq1Q1hDsdPDS8Iy//8gJ+OFHCiKcWM3XBJq/v59AxDeWN5Ngo/j2uN/cMasvcvD1c8fhC5ubt8Ys7yDVpKFUD/dslMfeOfgzu1Ix/zN/IwH9+yUerd1f5Q62zp5S3wpwOJvTP5L0JF9O0YQS3vLqCG174ptrVbV3TpKFUDTVuGMFT1/fgX7++kIYRYUx4fSXXPLuEeXl7fqwoytNKQ1VXx9Q4PpjYlz8M78i63UcY9uQi7vz3Ktbtsmeg3KukISKDRSRfRApEZHIF74uITLXeXy0iPapqKyJNRGS+iGyyvjYu89591v75IjKozPaeIrLGem+qBMLtkyro9c1K4KPbL2HK1Z3ZU3yKca+u4PLHvuClRVvYd+S/7+1w6+wpVQPhTgdj+2Tw5d39ublvKz5eu4ehU79i9LSlfLxmt8/WSauIVFVOi4gT2AgMBIqAb4HRxph1ZfYZCkwEhgIXAk8YYy6srK2I/B04ZIyZYiWTxsaYe0WkA/AG0AtIBT4Fso0xbhFZBkwClgJzgKnGmI8riz8nJ8csX768ev9VlKohl7uUT/L28OKiLXy3/TAi0D09noEdmtG9RTzhTgfXPLuEp6/vwZVdUuwOVwWo4hNneOPb7cxYspXdxaeICnfQLyuRy9sl0SktjuzkRkSE1e5CkoisMMbklN/uzdpTvYACY0yhdaA3gRHAujL7jABmGk8GWioi8SKSAmRU0nYEcJnVfgbwBXCvtf1NY8xpYIuIFAC9RGQrEGuM+do61kxgJFBp0lCqPoU5HQzrksqwLqnk7znK3Lw9zM3bw98+2fBf++nkKVUbcdHhjL+0DTf3bcU3Ww4xN28P8/L2Mm+d57HF4U4hK6kRb/ymN3HR4XX62d4kjTRgR5nXRXiqiar2SauibbIxZjeAMWa3iCSVOdbSCo51xvq+/PZziMg4YBxAixYtKumaUr7Ttlkj2jZrxO25Wew7eoq8XUdYt+sIOw+fpFerpnaHp4JAmNPBxZkJXJyZwENXdWTrwePk7TpC3q4jFO4/RmyDul+T1psjVnTxtfw1rfPt401bbz/P62MZY6YB08BzeaqKz1PK55IaRZHUNor+bZOq3lmpGnA4hNaJMbROjOGqrqm++xwv9ikC0su8bg7s8nKfytrutS5hYX3d58WxmlcRh1JKKR/yJml8C2SJSCsRiQBGAbPL7TMbGGPNouoNFFuXniprOxsYa30/Fni/zPZRIhIpIq2ALGCZdbyjItLbmjU1pkwbpZRS9aDKy1PGGJeI3AbMBZzAS8aYPBEZb73/HJ6ZTEOBAuAEcFNlba1DTwFmicivge3AtVabPBGZhWew3AVMMMacnU/2W+AVoAGeAXAdBFdKqXpU5ZTbQKdTbpVSqvrON+VWJ/4ppZTymiYNpZRSXtOkoZRSymuaNJRSSnkt6AfCRWQ/sK2GzROAA3UYTiAIxT5DaPY7FPsModnvmvS5pTEmsfzGoE8atSEiyyuaPRDMQrHPEJr9DsU+Q2j2uy77rJenlFJKeU2ThlJKKa9p0qjcNLsDsEEo9hlCs9+h2GcIzX7XWZ91TEMppZTXtNJQSinlNU0aSimlvKZJowIiMlhE8kWkwHp+eVASkXQR+VxE1otInohMsrY3EZH5IrLJ+trY7ljrmog4ReQ7EfnQeh0KfY4XkbdFZIP1//yiYO+3iPzOOrfXisgbIhIVjH0WkZdEZJ+IrC2z7bz9FJH7rN9v+SIyqDqfpUmjHBFxAk8DQ4AOwGgR6WBvVD7jAu4yxrQHegMTrL5OBhYYY7KABdbrYDMJWF/mdSj0+QngE2NMO6Arnv4Hbb9FJA24HcgxxnTC83iGUQRnn18BBpfbVmE/rZ/xUUBHq80z1u89r2jSOFcvoMAYU2iMKQHeBEbYHJNPGGN2G2NWWt8fxfNLJA1Pf2dYu80ARtoSoI+ISHPgSuCFMpuDvc+xQD/gRQBjTIkx5jBB3m88zwxqICJhQDSep30GXZ+NMQuBQ+U2n6+fI4A3jTGnjTFb8DwHqZe3n6VJ41xpwI4yr4usbUFNRDKA7sA3QLL1pESsr8H2YOvHgf8FSstsC/Y+twb2Ay9bl+VeEJGGBHG/jTE7gUfxPORtN54nis4jiPtczvn6WavfcZo0ziUVbAvqeckiEgP8B7jDGHPE7nh8SUSGAfuMMSvsjqWehQE9gGeNMd2B4wTHZZnzsq7hjwBaAalAQxG50d6o/EKtfsdp0jhXEZBe5nVzPCVtUBKRcDwJ4zVjzDvW5r0ikmK9nwLssys+H7gYGC4iW/FcerxcRP5FcPcZPOd1kTHmG+v123iSSDD3ewCwxRiz3xhzBngH6ENw97ms8/WzVr/jNGmc61sgS0RaiUgEngGj2TbH5BMiIniuca83xvyjzFuzgbHW92OB9+s7Nl8xxtxnjGlujMnA8//2M2PMjQRxnwGMMXuAHSLS1tqUC6wjuPu9HegtItHWuZ6LZ9wumPtc1vn6ORsYJSKRItIKyAKWeXtQvSO8AiIyFM91byfwkjHmYXsj8g0R6Qt8Bazhp+v79+MZ15gFtMDzg3etMab8IFvAE5HLgLuNMcNEpClB3mcR6YZn8D8CKARuwvOHY9D2W0T+APwCz0zB74CbgRiCrM8i8gZwGZ4l0PcCDwLvcZ5+isgDwK/w/He5wxjzsdefpUlDKaWUt/TylFJKKa9p0lBKKeU1TRpKKaW8pklDKaWU1zRpKKWU8pomDaWUUl7TpKGUUspr/x9OVz4vBzmF1gAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch, cycle_limit=2)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*2)\n", "\n", "plt.plot([i for i in range(num_epoch*2)], lr_per_epoch);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `t_in_epochs`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If set to False, the learning rates returned for epoch `t` are `None`." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[None, None, None, None, None]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "num_epoch = 50\n", "scheduler = CosineLRScheduler(optimizer, t_initial=num_epoch, t_in_epochs=False)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "\n", "lr_per_epoch[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `noise_range_t`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Add noise to learning rate scheduler. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `noise_pct`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The amount of noise to be added. Defaults to 0.67." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `noise_std`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Noise standard deviation. Defaults to 1.0." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `noise_seed`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Noise seed to use. Defaults to 42." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `initialize`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If set to True, then, the an attributes `initial_lr` is set to each param group. Defaults to `True`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> NOTE: we leave it up to the reader to try and experiment how the LR annealing scheduling get's updated by adding in noise hyperparams. " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }