{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# TanhLRScheduler" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial we are going to be looking at the `TanhLRScheduler` in the `timm` library in little more detail with all the supporting hyperparameters. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"Tanh\"" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "#hide\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from timm.scheduler.tanh_lr import TanhLRScheduler\n", "from nbdev.showdoc import show_doc" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "

class TanhLRScheduler[source]

\n", "\n", "> TanhLRScheduler(**`optimizer`**:`Optimizer`, **`t_initial`**:`int`, **`lb`**:`float`=*`-6.0`*, **`ub`**:`float`=*`4.0`*, **`t_mul`**:`float`=*`1.0`*, **`lr_min`**:`float`=*`0.0`*, **`decay_rate`**:`float`=*`1.0`*, **`warmup_t`**=*`0`*, **`warmup_lr_init`**=*`0`*, **`warmup_prefix`**=*`False`*, **`cycle_limit`**=*`0`*, **`t_in_epochs`**=*`True`*, **`noise_range_t`**=*`None`*, **`noise_pct`**=*`0.67`*, **`noise_std`**=*`1.0`*, **`noise_seed`**=*`42`*, **`initialize`**=*`True`*) :: `Scheduler`\n", "\n", "Hyberbolic-Tangent decay with restarts.\n", "This is described in the paper https://arxiv.org/abs/1806.01593" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(TanhLRScheduler)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `TanhLRScheduler` as shown above accepts an `optimizer` and also some hyperparams which we will look into in detail below. We will first see how we can train models using the `TanhLRScheduler` by first using `timm` training script and then look at how we can use this scheduler as standalone scheduler for our custom training scripts. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using `cosine` scheduler with `timm` training script" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To train models using the `TanhLRScheduler` we simply update the training script args passed by passing in `--sched tanh` parameter alongside the necessary hyperparams. In the next section we will also look at how each of the hyperparams update the `TanhLRScheduler`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The training command to use `TanhLRScheduler` looks something like: \n", "\n", "```python \n", "python train.py ../imagenette2-320/ --sched tanh\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Args" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now look at the associated hyperparams and how that updates the annealing schedule. " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### `optimizer`" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "This is the `optimizer` that will be used for the training process. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "hidden": true }, "outputs": [], "source": [ "from timm import create_model \n", "from timm.optim import create_optimizer\n", "from types import SimpleNamespace" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "hidden": true }, "outputs": [], "source": [ "model = create_model('resnet34')\n", "\n", "args = SimpleNamespace()\n", "args.weight_decay = 0\n", "args.lr = 1e-4\n", "args.opt = 'adam' \n", "args.momentum = 0.9\n", "\n", "optimizer = create_optimizer(args, model)" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "This `optimizer` object created using `create_optimizer` is what get's passed to the `optimizer` argument. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `t_initial` " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The number of epochs to schedule the hyperparameter for. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `lb`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The lower bound denoted by `L` in the paper." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `ub`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The upper bound as denoted by `U` in the paper." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "NOTE: `L` and `U` indicate the lower and upper bounds of the interval `[L, U]` for the function `tanh(x)`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `t_mul`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Defaults to 1.0. Similar to `SGDR` it updates schedule by increasing the time for annealing. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from matplotlib import pyplot as plt\n", "\n", "def get_lr_per_epoch(scheduler, num_epoch):\n", " lr_per_epoch = []\n", " for epoch in range(num_epoch):\n", " lr_per_epoch.append(scheduler.get_epoch_values(epoch))\n", " return lr_per_epoch" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### `t_mul=1.`" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2DElEQVR4nO29fXxc5Xnn/b1mpNHodSTZerVG2MbCYMB2HGMgoYkJIRiSYp6mSUzTB5pNliU1u8l20wSWbnbz2bKl2zSbsE1g2TZPoE1CSJosfhoSAhRIE95sDLYxxrYM2JZtJPlFsmTrdc69f5wz8lgeSUe2Ruft+n4++ozmnHPPuUaXzv27X67rvsUYg6IoiqK4Iea1AYqiKEpwUNFQFEVRXKOioSiKorhGRUNRFEVxjYqGoiiK4poirw0oNHPnzjXz58/32gxFUZRA8corrxw2xtSNPx560Zg/fz6bNm3y2gxFUZRAISJ78x3X4SlFURTFNSoaiqIoimtUNBRFURTXqGgoiqIorlHRUBRFUVzjSjREZI2I7BSRdhG5M895EZH7nPNbRWTFVGVFpFZEnhSR3c5rjXN8jog8IyL9IvI34+7zXhHZ5nzWfSIiZ//VFUVRlOkypWiISBz4NnA9sAS4WUSWjLvseqDN+bkNuN9F2TuBp40xbcDTznuAQeA/AV/KY879zudn77XG1bdUFEVRZgQ3eRqrgHZjzFsAIvIIsBZ4I+eatcDDxl5n/UURqRaRJmD+JGXXAqud8g8BzwJfMcacAH4jIotyjXA+r8oY84Lz/mHgJuAX0/vK7njo+Xc40j+EiCACQvYV+9Xp5OQ7Fxs7J6euz753zieL4ySLY5QWxyktjlNSHKcsEWduRQlzyhPEYtqJysfWjh6eeqMTTvvbnumbvMed92BfExvnFxGhpOiUT5LFcUoTMZLFceaUlzC3IkFRXEd08/HO4RP89NUDwJl/7/HPTEwm94uA7Ztcv8RjJBNxkkUxShOOb4rj1JQnqKsoIVGkfpkt3IjGPGB/zvsO4HIX18ybomyDMeYQgDHmkIjUu7CjI889zkBEbsPukdDa2jrFx+bn+y/tZVdn/1mVPVfiMWFuRYKGqiTn11WwuLGSlefVsCxdTXHEK637nm7nqR2dntxbBOaUJ6ivTLKgrpzFDZW897wa3nteDcniuCc2+YWHX9jLd3/7tmf3rykrpr4yyXlzyljcWMnydDWrFtRSmSz2zKaw4kY08jV5x+/cNNE1bsq6xfVnGWMeBB4EWLly5Vnd71f//oPZz8IY+0bGGOcVDPZxxr3PXmcZ27rxxw1gWYbBEYvB0QwDwxkGRjIMjmQ4MZThcP8QXX2DdB0f4t3jg7yw5wg/c1pw5Yk4H7m4kU+sbOHKhXOI4pTOqGWxrCXFY3dcNaVv8vklew0GrLH3jl+MYWjEYnAk6xOLgZEMA8OjHO4fpuv4IF19Q3QeH2RrRw8/33oIgEQ8xgcX1/HJlWmuXlwXyd7IqGVRU1bMq1/9yDn7JXvOyvHL8Kjtl1M+sX109MQwXcftZ6bz+BDt3f08taMTy9iNr8sX1PLJlWnWXNIYeWGfKdyIRgeQznnfAhx0eU1ikrKdItLk9DKagC4XdrRMYceMk+0iO+8Kfbu8HD0xzMtvH+G5Xd3809ZD/OzVAyxPV/OfPraE955X44lNXpGxTM7QoLe+6RscYdPeY/zLrsNs2HKQJ9/oZGFdOXffcBEfurA+UqKesczpw7Ie+mVwJMOr+3r4bbvtly/+6DXqHy/hT69bzMdXtOjQ7znipkm0EWgTkQUikgDWARvGXbMBuMWJoroC6HWGniYruwG41fn9VuCxyYxwPq9PRK5woqZumapMWKgtT7Dmkib+4veWsvHuD/OXH7+Ugz0DfPz+5/kvG7YzmrG8NnHWME4L0g9UJou5enE9X/3dJbxw14e4/9N20OBnH9rE+h9s5uTwqMcWzh6WwTeVcbI4zpXnz+FL1y3m2S+t5vufu5zm6lL+9Cdb+dSDL3C4f8hrEwPNlKJhjBkF7gCeAHYAjxpjtovI7SJyu3PZ48BbQDvwv4E/nqysU+Ze4FoR2Q1c67wHQETeAb4B/JGIdOREXH0e+FvnPnso0CS4n0kWx/nUZa0886XV/NH75vO959/hM9/bSO/AiNemzQp2i9ZrK86kOB7j+kubeOKLH+DLaxbzy9ff5ffvf4GDPQNemzYrWD71SywmvH/RXH76+ffxV7+/lG0Heln7N79lx6HjXpsWWMRkB+ZDysqVK02YV7n90cZ9/Nn/eZ3l6Wq+/7krQh9F8sn/9QIC/OjfXOm1KZPyzM4u/u0PXmVuRYLH1l9FqizcE7Jf+vEWnm8/zPN3XeO1KZOyraOXzz28kaFRiw3rr6J1TpnXJvkWEXnFGLNy/PFw1zAR4FOXtfKNTy5n4zvH+OpjrxP2RoAxxjfDU5Nx9eJ6vveZyzjQM8D6H2wO/RCiZYxvhqcm49KWFD+67UqMgX/98Cb6h6IzhDhTqGiEgN9d1swdVy/ikY37+cHL+7w2p6DkTrj6nZXza7nnpkv5Tfth/uqJnV6bU1CsAPll/txyvvPpFbR39/OlR7d4bU7gUNEICX9y7QVctWguf/mLNzl6YthrcwpGxkcTrm745GVp1l2W5m9/8za7O/u8NqdgZHwUoOCG9y+ay3/4yAX8cvu7PLNzqsBNJRcVjZAQiwn/+XeXcGI4wzef2uW1OQXDGEM8OHUTAF9ecyFliTh//vMdXptSMCzjz4nwyfjcVQuZP6eMe36+g5GQDx/OJCoaIaKtoZJPX97K91/aF9pWbZCGp7LUlif4wjVtPLerO7St2iANT2VJFMX4jzdcRHtXPz94KdzDujOJikbI+OKHL6A8EecbT4azt5GxgjHhOp5brpzP/DllfP2JnaEMVshYwQhQGM+1Sxq4cuEcvvX0boZGM16bEwhUNEJGbXmCdataefKNTrqOD3ptzoxjDMQD1qIFu1X7ud9ZyPaDx9na0eu1OTOOZQhcTwPs7PXPrz6foyeGeWK7N2uaBQ0VjRCy7rI0o5bhx690TH1xwMgYQyyg/7VrlzdTloiHcijECrBfrlo0l3RtKT94aa/XpgSCgLpZmYyFdRVcuXAOP3x5H5YVrqGQII6dZ6lMFnPjsmY2bDlI32C4MvgtYwLZAwQ7iGTdZa28+NZR3ur2ZmXrIKGiEVJuvryVjmMD/Kb9sNemzChWQJL7JuLmVa0MjGR47LWCr7U5qwR1rinLJ1a2UBQTHtm4f+qLI46KRki57uIGassTPLopXA9BxgS3pwGwtCXFkqaq0PnFCrhf6iuTXLukgZ+80kEmZL3zmUZFI6SUFMX5yJIGntvVHaoYdMsK5oRrFhHho0ub2NrRG6pABcsKZoBCLh9d2sTRE8O8tr/Ha1N8jYpGiFm9uI6+wVE27z3mtSkzRhCTyMazenEdAM/u6vbYkpkjYwwB1wx+Z1EdMYFnQ5pLM1OoaISY9y+aS1FMwlU5BTQfIJclTVXUV5bw3M7w+MUKgV9SZcWsaK3h2RD5pRCoaISYymQxK+fX8Myb4Wk5+Wmzn7NFRFi9uI5/2d0dmtVvgx6gkOXqC+vZdqCXrr7wDB3ONCoaIWf14nrefLePd3vD8RCEYXgK7KXTjw+O8mpIxs8zhlBsb/vBC+yhw1/vClfU4UyiohFysuPnz+0KR28jYwU3HyCX97fZQ4dh6QVaVvAWkszHxc1V1FWW6LzGJKhohJzFDZU0pZKhGacNymY/U1GVLOa959XwXEjmm8IyPCUirL6gjl/v6tbQ2wlQ0Qg5IsIVC+eweV84IqiCnBE+nisWzmHHoeOcHA7+7nEZy4RieApsvxwfHNXs8AlQ0YgAy1pSdB4f4lDvgNemnDOZkLRoAZanq7GMvW910AnyMiLjWZauBgjNfNNMo6IRAbIPwZYQPARBXU01H0tbUgBs6ejx1pAZwArYzn2TsXBuOZUlRaF4XgqBikYEuKipiuK48Nr+ELRorXBETwHMqSghXVvKlpD4JSRaTiwmLE2nQiHmhUBFIwIki+MsaaoKRcspTMNTAMtaqkOxbEVYJsKzLE9X8+ahPgZHdGOm8ahoRIRl6Wq2HegNdESIMQYTouEpsCunAz0DgU8my4RoTgNsMR+1DNsPHvfaFN+hohERlrVU0z8U7IiQrN6FSTSy801bAz5EZVnhSO7LsjxE84AzjYpGRAhDREi2lxQP0X/tJc0p4jEJ/Pi5PTzltRUzR31VkqZUMhRDhzNNiNysTMbCueVUJoMdEWIZWzTCkNyXpTQRZ3FDZeArpzAsJDme5enqwIt5IVDRiAixmLC0JcW2A8EdBhkTjRANgwAsS6fY2tGLMcGdb7JMeJL7sixtqWbvkZP0ngzX1rzniopGhLigoZL2rv7AVk7ZOY0wTbiC7ZfegREO9w97bcpZY5nw+WVxYwUA7d19HlviL1Q0IsSi+gpODmc4GNAVb7NzGiGrm1hUb1dOu7uCWzllQpQ/k2VRXSUA7V3BDR4pBCoaEaKt3n4IdncGs3KyxibCw1U7Zf0S5MrJssKxkGQu82pKSRbH2N0ZXL8UAleiISJrRGSniLSLyJ15zouI3Oec3yoiK6YqKyK1IvKkiOx2Xmtyzt3lXL9TRK7LOX6ziGxz7vFLEZl79l89emRbtEGtnLJzGmETjYaqEipLigLrFwjX2lNZ4jHh/LoKdgfYL4VgStEQkTjwbeB6YAlws4gsGXfZ9UCb83MbcL+LsncCTxtj2oCnnfc459cBFwNrgO+ISFxEioBvAVcbY5YCW4E7zvJ7R5La8gRzyhOBrZwyJjs8Fa7KSUQ4v74i0C3aTEiWrB/PovqKwD4vhcJNT2MV0G6MecsYMww8Aqwdd81a4GFj8yJQLSJNU5RdCzzk/P4QcFPO8UeMMUPGmLeBdudzxPkpF7vWqAIOTvsbR5xF9cFtOVnOzqhha9ECtAXYL2D7JmxRbWD75UDPACeGgr98/UzhRjTmAftz3nc4x9xcM1nZBmPMIQDntX6yzzLGjACfB7Zhi8US4O/yGSwit4nIJhHZ1N0djk1uZopsyymIEVSnhqc8NqQAtDVUcLh/iJ6TwYygCltyX5ZFznzTngCvpDDTuHFzvubD+BpnomvclHV1PxEpxhaN9wDN2MNTd+X7AGPMg8aYlcaYlXV1dVPcLlq01VfQOzBCd/+Q16ZMm1PRU+Fr0QZ9viljwrM5Vi5jkW0BHjqcadyIRgeQznnfwpnDQhNdM1nZTmcIC+c1uynvRGWWAxhj9hi7mfwo8D4X9is5ZFtO7QF8CMZ6GiGsnMYi2wIoGmFcSDLLeXPKKI4L7drTGMONaGwE2kRkgYgksCepN4y7ZgNwixNFdQXQ6ww5TVZ2A3Cr8/utwGM5x9eJSImILMCeXH8ZOAAsEZFs1+FaYMc0v2/kaWvIJiwF7yEYS+4L4YTrvGo7vDOIPY0w+6U4HmPB3HLtaeRQNNUFxphREbkDeAKIA981xmwXkdud8w8AjwM3YE9anwQ+M1lZ56PvBR4Vkc8C+4BPOGW2i8ijwBvAKLDeGJMBDorI14Bfi8gIsBf4oxn4G0SK+ko7vDOID0FYk/vAXuYlqOGdWb+EUDMAe4jqDV0ifYwpRQPAGPM4tjDkHnsg53cDrHdb1jl+BLhmgjL3APfkOf4A8MCZJRS3iAiLGioCmX0c1jyNLG31Fbz89lGvzZg2YVxIMpdF9ZX88vV3GRzJkCyOe22O54Qw3kGZioVzK9h75KTXZkybMM9pACysq+Bg72DgdosLu1/OryvHMtBxLHjPTCFQ0Ygg6dpS3j0evMopzNFTYPsFoOPYgMeWTI9Tw1Ph9EtLTRkA+46qaICKRiRprS3DGDjQE6zKaSy5L6TDIK21duW0P2CV09iOiiH3y74A9s4LgYpGBAlu5RTuCdd01i8BGwaxQj4RPrciQWlxnP0B6wEWChWNCBJU0ciEfMK1rqKEZHEscC3aTMgDFESEdG2pDk85qGhEkLrKEkqKYoF7CEzIJ1xFhHRNWeD8EtYdFXNprS0LXCOrUKhoRBC75RS8yinjzGmEvnIK2DCIFQG/pB3RCOKabTONikZEsVtOwaqcxqJ0QvxfG8TKKcwLSWZJ15RxYjjD0RPBXFByJgmxm5XJSNeUBq5yCvvwFNii0T80Ss/JEa9NcU3YQ24hZx4wYL3AQqCiEVHStWX0Ba1yCvlEONhiDsHKCYjCnEY2si1IfikUKhoRpTWA4Z2RaNHOCV7lFOYFC7NkEy91MlxFI7IEsXIyUaicagIs5iH2S1miiLkVJSoaqGhElnQAl0YI+2qqAOUlRcwpTwSqcgp70mUWzdWwUdGIKKcqp+BM7GUiMHYOBC4cOuwLFmZpDZhfCoWKRoRJByxhyYQ88zhL0MKhozA8BbZfDvUOMpJNGIooKhoRJmgt2igk94E9DHKgZ4DRgFROUUjuA3tIN2MZDvUMem2Kp6hoRJjm6iTv9g6OLTjndzIRSCIDaK4uJWMZuvuHvDbFFVFI7gPbLwAHe4PTCywEIXezMhnNqVKGMxZHApLlaiIyp9GcciqngLRoozLX1FSdBOCQioYSVZpSwXoIopCnAcGrnKyI+CVoYl4oVDQiTFPAHoKsaIR9Ijzrl6CMnUchuQ+gNBGnuqw4MGJeKFQ0IkzQWrTZ5L6QN2ipShZRnogHZuz81Da8HhsyCzSlSgMj5oVCRSPCzClPkCiKcag3GA9B2Df7ySIiNFUHp3KKwkKSWZpTSQ4G5HkpFCoaEUZEaEolORiQvcKjkkQG9nxTUHqAURFzsHvnQfFLoVDRiDh25RSMlpM1NgwS/sqpOVUamBZtJkJ+aUqV0nNyhIHhjNemeIaKRsRpTpVyKCA9jahMhIPdoj3cP8TwqP8T/KKwkGSWZmceMCjzTYVARSPiNFUn6ewbGquQ/cxYlE4EWrTNqVKMgc7j/u9tjIl5BPySjWx7NyC9wEKgohFxmlJO9nGf/7OPs3MaEoH/2mxkWxDmm7JzGhHQjJxcDf/7pVBE4PFTJiNI3e0otmiDMN8UlYUkARpSJUAw/FIoVDQiTpASyaKSRAZBE3P7NQp+KSmKM7eiJNIRVCoaEad5rEXr/4fAitAwSFmiiFRpcSDEPBORTZiyNFcnA7OKQiFwJRoiskZEdopIu4jcmee8iMh9zvmtIrJiqrIiUisiT4rIbue1JufcXc71O0XkupzjCRF5UER2icibIvLxs//qCkBVaRFliXggHoIoDU9BcHI1orKQZJbGqmD4pVBMKRoiEge+DVwPLAFuFpEl4y67Hmhzfm4D7ndR9k7gaWNMG/C08x7n/DrgYmAN8B3ncwDuBrqMMRc4n/fcWXxnJYdsgl8QHgIrQmPngJN4GSAxj4hfmgOUrV8I3PQ0VgHtxpi3jDHDwCPA2nHXrAUeNjYvAtUi0jRF2bXAQ87vDwE35Rx/xBgzZIx5G2h3PgfgXwF/AWCMsYwxh6f3dZV8NFcHI5EsSsl9gL2USADEPCqrD2dpSiXpGxqlb3DEa1M8wY1ozAP257zvcI65uWaysg3GmEMAzmv9ZJ8lItXO+/8qIptF5Mci0pDPYBG5TUQ2icim7u5uF18x2jRWJQOR4JcxJjKtWbDXOToWgOzjbHJf2Ld7zdJUHZzItkLgRjTy/SeMzwSb6Bo3Zd3erwhoAX5rjFkBvAB8Pd8HGGMeNMasNMasrKurm+J2SlPKzj72+/ailonOfAZAoxOk4PcEv0yE1gSDU/vQRDXBz41odADpnPctwEGX10xWttMZwsJ57Zris44AJ4GfOcd/DKxAOWcaUkksg++3F7UsE4nIqSyNVU7l5HPRsCIWPRUUvxQKN6KxEWgTkQUiksCepN4w7poNwC1OFNUVQK8z5DRZ2Q3Arc7vtwKP5RxfJyIlIrIAe3L9ZWOHaPz/wGrnumuAN6b3dZV8NFTaD0HncZ+LRsSGpxqq7EQyv/c0xnbui4hv6iptv3T53C+FomiqC4wxoyJyB/AEEAe+a4zZLiK3O+cfAB4HbsCetD4JfGayss5H3ws8KiKfBfYBn3DKbBeRR7EFYRRYb4zJDup+Bfh7Efkm0J29j3JuNOZ2t9NTXOwhGSs6k61g9wDB/6IRtYnwZHGcmrLiyPY0phQNAGPM49jCkHvsgZzfDbDebVnn+BHs3kK+MvcA9+Q5vhf4gBubFfc0VAWjcrKMicwQCEBliZ1D826v33uA9mtU5jTAfmb87pdCoRnhCnPKExTHxfctp6gNT4kIjVXJQIg5QCxCtUljyv9+KRQRcrMyEbGYUF+ZpNPn0SAZy0RmCCRLQ1XS92IeteEpsOcB/e6XQqGioQBQX1VCZ5+/HwLLmMhMtmZpqCrxfYs2SgtJZmkISJh6IVDRUAA7jNDvceeWFa1xc7Arp67jQ2PrO/kRK2JrT4H9vJgAhKkXAhUNBbCHQfwecpuJ2EQ42JXTcMbi6Ilhr02ZkFPDUx4bMos0Ovtq+L2hVQhUNBTAntjrHxqlf2jUa1MmxLKiNzwVhESyqC0kCVBfGYyIw0KgoqEAOZWTj1tOUYuegmDkamQz9aOykCSMy22KGCoaCmBPhIO/s1wzJlrj5pCbQ+PfoUMrgn6pLbPD1Dv7/OuXQqGioQABGQaxojenUV9Zgoi/W7QZYyIXoBCUMPVCoKKhADndbT+LRgSHp4rjMeaU+zvs1p5r8tqK2acxFc1cjQi6WslHWaKIymSRr1tOUUzuAztSx8+Vk728S/T80lDlb78UChUNZYxGn2cfR7Vy8nsOTSaC+TPghKn72C+FQkVDGcPvuRqWidb6Rlnqq5J0+XjC1TLR2uckS2NVkhPDGV+HqReCCD6CykQ0+HxxvIwVvQlXsCunoyeGGRr157avUZxrguiG3apoKGM0pkro6hsay/D1G1FcewpORbZ1+bQXmLGiKRpRTfBT0VDGaKxKkrEMR3y6nk5U5zQafB7ZZg9PRc8v2tNQIk+9z3M1ojo8ld321a+VUxQXkoRg5DYVAhUNZYxGn2cfR3UivNHnOytmIjqnUZqIU5Us8vUqCoUggo+gMhF+T/CzIpqnkSotpqQo5lvRyK49FUWCsEnWTKOioYwxt6KEeEx8G3se1RatiDjZx37tAUbTL4Cv/VIoVDSUMeIxoa7Cv1muUVwYL4ufE8kyJppzGuBvvxQKFQ3lNPy8vWgUFyzM4udhkKgm94E939Td798w9UKgoqGchp8T/KKaDwDQ6Ii5H7d9tSLsl4aUv8PUC4GKhnIajSn/rnMU1TwNsMV8aNSid2DEa1POIKoLSQI0VDrh0D5taBUCFQ3lNBqqkhwfHGVg2H9LVkRZNPwc2RbluaYoJvipaCin4eeEpWgPT/m3cop09JTPc2gKgYqGchoNPq6cjCGSa09B7rav/vNLxormmmAAc5wwdT82sgqFioZyGo0pZ6/wPv89BBkT3eip7B7ufszWtyLsl2yYuh/9UihUNJTT8HNPI6prTwGUFMWpLU/4skVrRXCP8FwaUv6NOCwEKhrKaVQmiylPxH1ZORlDJFdTzeLXRLIoR0+BHQ7tx0ZWoVDRUM7Ary0neyLcayu8o9Gne1JHdSHJLH7fJnmmceVqEVkjIjtFpF1E7sxzXkTkPuf8VhFZMVVZEakVkSdFZLfzWpNz7i7n+p0icl2e+20Qkden/3UVNzRU+jNXI8pROuDfxMsoJ/eB3cjqGxzl5HA0tn2dUjREJA58G7geWALcLCJLxl12PdDm/NwG3O+i7J3A08aYNuBp5z3O+XXAxcAa4DvO52Tt+T2g/2y+rOKOxpQ/9wqP6mY/WRqqkhzuH2YkY3ltymlkIpw/A3YjC/wZpFAI3PQ0VgHtxpi3jDHDwCPA2nHXrAUeNjYvAtUi0jRF2bXAQ87vDwE35Rx/xBgzZIx5G2h3PgcRqQD+BPjz6X9VxS0NVUm6+gaxfLaeTpQnwuFUIpnfehtRTu6DU3451DvgsSWzgxvRmAfsz3nf4Rxzc81kZRuMMYcAnNd6F/f7r8BfAycnM1hEbhORTSKyqbu7e7JLlTw0pZKMZAyHT/ir5WQZIj0M4tfs46gPT/nVL4XCjWjk+28Y3wSd6Bo3ZV3dT0SWA4uMMT+bojzGmAeNMSuNMSvr6uqmulwZR1O25dTjr4cgypv9ADSnSgE46LPKKcp5GnDKL4d85pdC4UY0OoB0zvsW4KDLayYr2+kMYeG8dk3xWVcC7xWRd4DfABeIyLMu7FemSXN19iHwV3c7E/F8gKbqrJj7zC8RD7ktTcSpLivmoM/8UijciMZGoE1EFohIAnuSesO4azYAtzhRVFcAvc6Q02RlNwC3Or/fCjyWc3ydiJSIyALsyfWXjTH3G2OajTHzgauAXcaY1WfxnZUpyPY0DvqtpxHx6KmqZDEVJUW+a9FG3S8ATalS3/mlUBRNdYExZlRE7gCeAOLAd40x20Xkduf8A8DjwA3Yk9Yngc9MVtb56HuBR0Xks8A+4BNOme0i8ijwBjAKrDfG+G/J1RBTW56gpCjmu56GZUU7uQ9sQfdbizbqE+EAzakkB3zml0IxpWgAGGMexxaG3GMP5PxugPVuyzrHjwDXTFDmHuCeSex5B7jEhenKWSAiduXks5aTvUe411Z4S1O1/1q0VoQXLMzSVJ1k095jXpsxK0T8EVQmoilV6rux86ivcQR2i9ZvPUB7rslrK7ylKVVK78BIJBL8VDSUvDRVJ33VojXGRH7tKbArp8P9wwyN+mfENsqbY2VprvbnPGAhUNFQ8tKcKqXz+CCjPsk+zjiJhpGfcHUqp85e/+TQWFZ09znJ0pTyZ8RhIVDRUPLSVJ3EMtDV54/KKZucHnXROJWr4Z/KKeqZ+pCTq6E9DSWqNPus5WQZWzUiXjedytXwiV/AGZ6KeE3S4Gxe5icxLxQRd7UyEU0+G6MdG56KuGqM9TR84hfQOQ2wN8maW1GiPQ0luvhtjDbb04h65ZTNPvaLX0DzNLI0Vye1p6FEl6pkEeWJuG9atJYzHx/1CVfIhkP7wy+Q3RxL/dKU8lfEYaFQ0VDyIiJOIpk/Wk7ZnkbU8wHAztXwU+KlFfG1p7Jkc5uM8deWAjONioYyIX5qOWWyw1PaonVyaPwh5qCr3GZprk5yYjjD8cFwJ/ipaCgT0pwq9dHwlM5pZGlKldJzcoSBYX8k+GV0wULAv6tDzzQqGsqENFUnOdw/5IvsY83TOEU2+9gvC+Rpcp9N01hkmz/8UihUNJQJaakpA+DAMe8fgrHhKa2bxvzScWzSDSxnDR2esknX2KLR4YPnpZCoaCgT0lprV077ffAQ6PDUKfzkF9DNsbLUVZZQUhRj/1F/iHmhUNFQJiRbOe3zwUMwFj2lTVrqKvxTOWUXktThKTviMF1b5ovnpZCoaCgTUl9ZQsInlVNGexpjxGJCS00p+45475fsXJP6xaa1tox9R/3RAywUKhrKhGQrJz+IhqUht6fRWlvGfh/Maejqw6fTWltGx9GToc7VUNFQJqXVJ93tsegpbdECjl+OeF856fIup9NSU0rf0Cg9J0e8NqVgqGgok5Ku8YdonBqe8tgQn5CuLaNvaJTeAW8rJ0uj2k7DT/OAhUJFQ5mU1toy+gZH6fW45TQmGlo7AbZogPeVkw5PnU7rnGxkm4qGElH8UjkZnXA9jbGwW48nXbPDhlHfhjdLusYfz0shUdFQJiVdaycsed1yyoyF3Hpqhm/wi5hbli4kmUt5SRFzyhOei3kh0UdQmRTfVE464XoaFSVF1JYnPPdLRvNnzqCltswXEYeFQkVDmZSqZDE1ZcWeV06aEX4m6doyz5cSObUNr/oli18iDguFioYyJa0+aDnphOuZ+KFyym6OpX45RWttKQd7BhjNWF6bUhBUNJQp8UN3WzOPzyRdU8qBY95WTmPDU+qXMdI1ZYxaxjd70cw0KhrKlLTWlnGgZ2Cste8Fmg9wJq213ldO2WFD1YxTnIpsC+cQlYqGMiXn1ZYxkjGeLpGuw1Nnks0JeOfICc9s0IUkzyTrl7c99EshUdFQpmRRfQUA7d19ntmga0+dyZhfuvo9s0HF/EyaU6WUFsc99UshUdFQpiRbOe3u9O4h0JDbM6mrKCFVWuxp5aTJfWcSiwmL6iuiLRoiskZEdopIu4jcmee8iMh9zvmtIrJiqrIiUisiT4rIbue1JufcXc71O0XkOudYmYj8XETeFJHtInLvuX11xS3VZQnqKks8btHarzrhegoRoa2+gt2eioZOhOejLcqiISJx4NvA9cAS4GYRWTLusuuBNufnNuB+F2XvBJ42xrQBTzvvcc6vAy4G1gDfcT4H4OvGmAuB9wDvF5Hrz+ZLK9NnUZ0/KqeY9o1Pw+sW7anhKc9M8CXn11dwqHeQvsHwrXbrxtWrgHZjzFvGmGHgEWDtuGvWAg8bmxeBahFpmqLsWuAh5/eHgJtyjj9ijBkyxrwNtAOrjDEnjTHPADiftRlomf5XVs6Gtga7cvJqKW5N7svPovoKjp4Y5kj/kCf31+S+/LT5YL6pULgRjXnA/pz3Hc4xN9dMVrbBGHMIwHmtd3s/EakGfhe7h3IGInKbiGwSkU3d3d2TfTfFJW31FfQPjdJ53JvKSZeryE9bQyXgXeVk6bBhXrz2SyFxIxr5/hvGNzcnusZN2WndT0SKgB8C9xlj3sr3AcaYB40xK40xK+vq6qa4neKG87OT4V3eRFBpcl9+TkW2eSQaKuZ5SdeUkojHIisaHUA6530LcNDlNZOV7XSGsHBeu1ze70FgtzHmmy5sV2aItnq75eRVBJWlmzDlpTmVpDwR98wvGaPJffkoisdYWFfu6TxgoXAjGhuBNhFZICIJ7EnqDeOu2QDc4kRRXQH0OkNOk5XdANzq/H4r8FjO8XUiUiIiC7An118GEJE/B1LAF6f/VZVzYW5FguqyYs9atJoPkB8RO7xzj1c9DfXLhHgdpFAophQNY8wocAfwBLADeNQYs11EbheR253LHgfewp60/t/AH09W1ilzL3CtiOwGrnXe45x/FHgD+CWw3hiTEZEW4G7sKKzNIvKaiHzuXP8AijtEhEV1FbR71dPQPI0JOb++wrseoA4bTsii+gr2HzvJwHDGa1NmlCI3FxljHscWhtxjD+T8boD1bss6x48A10xQ5h7gnnHHOsg/36HMEm0NFfzy9Xc9ubdmhE9MW30lP918gOODI1Qli2f13hmNapuQtvpKjIE93f1cMi/ltTkzhkZXK645v66CYydHOOxBeGe2RatROmfi5XIiOhE+MX5Y5qUQqGgorrm42W4tbTvQO+v3zuhE+IRc3FwFwOse+EVXH56YhXXllBTFPHleComKhuKaS1tSiMCW/T2zfm8dnpqYplSSusoSXtvXM+v3HhNz9csZFMdjXDovxWsePC+FREVDcU1FSREX1Fd68hCMReno8NQZiAjL09W81tEz6/fWtacmZ1m6mtcP9DISol38VDSUabEsnWLL/p5ZX04ko1E6k7I8Xc1b3SfoHZjdtY6yGeHql/wsS1czNGqx813vthWYaVQ0lGmxLF3NsZMj7D86uxsyjSX36X9sXpa1VAOwrWN2x88zupDkpCx3/LLFg15goVBXK9MiWznN9lCIRulMzqUtdpDCa/uPzep9jfplUtK1pdSWJzyZbyoUKhrKtFjcWElJUWzWJ8Mzmtw3KanSYhbWlfPa/lnuaejw1KSICMtaUtrTUKJLNiJktkVDl0afmuXpal6b5fkmFfOpWZauZndXP/1Do16bMiOoaCjTZlm6mm2zHBEyltynwyATsjxdzeH+IQ72Ds7aPXV4amqWp6sxBraGpLehoqFMm+VORMhsJpNpct/ULE9XA7DpnaOzdk/1y9Rk5wE3vTO7802FQkVDmTbvXzQXEXhu1+xtcGUZg4juEDcZFzenqCkr5rmds+cXXXtqamrKEyxtSc3q81JIVDSUaVNbnmBZSzXPzmLlZBmjFdMUxGPCBy6o47ld3WNzQIUmO32iGeGTs/qCOl7dd4yek8Nem3LOqGgoZ8XVi+vZ0tEza3tTZyzNOnbD1YvrOXJieNbWO8poRrgrPri4HsvAr3cf9tqUc0ZFQzkrVi+uwxj4l1l6CCxjNIHMBR+4oA4RZq0XmNGkS1csT1dTXVbMszu7pr7Y56irlbPi0nkp5pQneGaWHgLL0uEpN9SWJ1jaUs2zu2bHL0ZDbl0RjwkfaKvj17M4dFgoVDSUsyIWEz54gf0QZGbhIcgYo0MgLrl6cR2v7e/h6InCj59ndCFJ16xeXMfh/mG2HzzutSnnhIqGctasvrCeYydH2DgLIZ7G6GSrW65eXI8x8NSOzoLfK6MT4a7JDh0+sd2b3S9nChUN5az58EX1VJYU8aON+wt+r4xlNBfAJUtbUiysK58VvxjdhMk1cytK+OAFdfz4lf2MBnipdBUN5awpSxRx03vm8fNthwoeSpgxRrOOXSIi/MGqVl7Ze6zgS3KPDU+pb1zxB6ta6Tw+xDOzGK4+06hoKOfEzataGR61+OnmAwW9j9E8jWnxeytaSMRj/PDlfQW9T3Y6S33jjg9dWE99ZUnB/VJIVDSUc2JJcxXL0tX84OV9BV0oL6PRU9OitjzBmksa+enmDgaGMwW7j6XRU9OiKB7jU5eleXZnFwd6ZndPmplCRUM5Zz69qpX2rv6C5mxkLB0CmS5/cHkrxwdH+cfNHQW7hw5PTZ9PXZbGAA89/47XppwVKhrKOXPj8mZaakr5b4/vKFj4rdHkvmlz+YJaLptfwzef2kXfYGG2gbV0InzatNSU8f+8Zx7f++077D960mtzpo0+hso5kyyOc9f1F/Hmu308uqkwETsZndOYNiLCn310CYf7h/n2M3sKcg/L0oUkz4YvX3ch8ZjwF7/Y4bUp00ZFQ5kRbri0kcvm1/DXv9rJ8QK0ajOWJvedDcvS1fzeinl89zdvs/fIiRn/fE26PDsaU0lu/+D5PL7tXV5864jX5kwLFQ1lRhARvvqxizl2coQ/+dGWGV8qQZP7zp4vX3chiaIY63+wecYnxS31y1lz2wcW0lJTyp/86DW6+2Zn4c+ZQEVDmTEubUnx1Y8t4akdnXz9Vztn9LM1ue/saUwl+da65Ww/eJw//cmWGY1ys9QvZ01pIs4Df/hejp4c5t/8/SaGRgsX5TaTqGgoM8otV57HzavSfOfZPdz/7J4Zq6B0TuPcuOaiBr583YX809ZDfPWx7TOWkazDhufGJfNS/PUnlrN5Xw/rv/8qJ4f9v4+4ioYyo4gIX7vxEj66tIm//OWbfOnHWxkcOfcWlCb3nTu3f3Aht31gIX//4l4+872NM5LFbxnN0ThXPrq0ia/deDH//GYnv3//C3Qc83dElYqGMuMkimL8zc3v4YsfbuMfN3dwzV8/x4YtB89pniNj6TIi54qI8B9vuIi//PilvPjWEa7++rM89Pw7jJxDr8Pe50T9cq7c+r75/N0fXcb+oye59hu/5ptP7fJtr8OVaIjIGhHZKSLtInJnnvMiIvc557eKyIqpyopIrYg8KSK7ndeanHN3OdfvFJHrco6/V0S2OefuE43z8y0iwhc/fAE//NdXkCot5t/98FVWf/1Z/ufTu9n5bt+0h610wnXm+NRlrTy2/ioubKziP2/Yzvvu/Wf+4hc72LK/Z9p5NpauCTZjXL24nse/8Dt86KJ6vvnUbi7/b09z98+28dJbRxge9c8ChzLVwysicWAXcC3QAWwEbjbGvJFzzQ3AvwVuAC4HvmWMuXyysiLy34Gjxph7HTGpMcZ8RUSWAD8EVgHNwFPABcaYjIi8DHwBeBF4HLjPGPOLyexfuXKl2bRp0/T+KsqMkrEMP992iEde3sfze+zwwrkVCS6dl2JxYxXp2lLqK5PUV5ZQX1VCqrSYZFH8NJH4f//uJfqHRvnZH7/fq68ROowxPLerm++/tI9/frOLjGWoTBaxrKWaxY2VzJ9TRl1lkvqqEuorS6guS1BaHD9NJO7+2Tae2P4um/7sWg+/SfjYvO8Y//DCXh5//RCDIxbJ4hhLW6q5sLGSBXPL7efF8Uttue2XovjMDhyJyCvGmJXjjxe5KLsKaDfGvOV80CPAWuCNnGvWAg8bW4FeFJFqEWkC5k9Sdi2w2in/EPAs8BXn+CPGmCHgbRFpB1aJyDtAlTHmBeezHgZuAiYVDcV74jHhxmXN3LismYM9A/ym/TAv7jnCG4eO85v2w4xk8jdcEkUxkkUxShNxek6OcOm81CxbHm5EhNWL61m9uJ4j/UP8ds8RXthzmNcPHOcfXtzL0ASt20Q8RklxjNLiOCeGRikvcVONKNNhRWsNK1pr+Nrai3lhzxGe33OErR09/HTzAfqH8g9bFcWE0uI4yUScpOOfDXdcRbI4PqO2ufH2PCA3zbcDuzcx1TXzpijbYIw5BGCMOSQi9Tmf9WKezxpxfh9//AxE5DbgNoDW1tZJvpoy2zRXl/LJlWk+uTINwEjG4nD/EF3Hh+jqG6Lz+CD9Q6MMDGcYHM0wNGIxMJxhYCTDh5c0eGx9eJlTUTIm7GD3Do/02z7p6huk6/gQvQMjDI5YDIxkGMz5WdpS7a3xIaYyWcxHLm7kIxc3Anbv8OiJ4bFnpatviJ6Tw3n8Yj83xTPc+wB3opFvwHJ803Cia9yUdXs/159ljHkQeBDs4akp7qd4SHE8RlOqlKZUqdemKDnEY0J9VZL6qiSgPTy/ICLMqShhTkUJFzVVeWKDGxnqANI571uAgy6vmaxspzOEhfPa5eKzWqawQ1EURSkgbkRjI9AmIgtEJAGsAzaMu2YDcIsTRXUF0OsMPU1WdgNwq/P7rcBjOcfXiUiJiCwA2oCXnc/rE5ErnKipW3LKKIqiKLPAlMNTxphREbkDeAKIA981xmwXkdud8w9gRzLdALQDJ4HPTFbW+eh7gUdF5LPAPuATTpntIvIo9mT5KLDeGJPNDvs88D2gFHsCXCfBFUVRZpEpQ26DjobcKoqiTJ+JQm41I1xRFEVxjYqGoiiK4hoVDUVRFMU1KhqKoiiKa0I/ES4i3cDesyw+Fzg8g+YUArXx3PG7faA2zhRqo3vOM8bUjT8YetE4F0RkU77oAT+hNp47frcP1MaZQm08d3R4SlEURXGNioaiKIriGhWNyXnQawNcoDaeO363D9TGmUJtPEd0TkNRFEVxjfY0FEVRFNeoaCiKoiiuUdHIg4isEZGdItLu7F/uOSKSFpFnRGSHiGwXkS84x2tF5EkR2e281vjA1riIvCoi/+RHG53tiH8iIm86f88rfWjjv3f8/LqI/FBEkl7bKCLfFZEuEXk959iENonIXc4ztFNErvPQxr9yfL1VRH4mItVe2ZjPvpxzXxIRIyJzvbLPDSoa4xCROPBt4HpgCXCziCzx1irAXib+PxhjLgKuANY7dt0JPG2MaQOedt57zReAHTnv/Wbjt4BfGmMuBJZh2+obG0VkHvDvgJXGmEuwtxVY5wMbvwesGXcsr03O/+Y64GKnzHecZ8sLG58ELjHGLAV2AXd5aGM++xCRNHAt9jYR2WNe/Q0nRUXjTFYB7caYt4wxw8AjwFqPbcIYc8gYs9n5vQ+7opuHbdtDzmUPATd5YqCDiLQAHwX+Nuewb2wUkSrgA8DfARhjho0xPfjIRocioFREioAy7F0qPbXRGPNr4Oi4wxPZtBZ4xBgzZIx5G3uvnVVe2GiM+ZUxZtR5+yKndgCddRsn+BsC/A/gy5y+hbUnf8OpUNE4k3nA/pz3Hc4x3yAi84H3AC8BDc6uhjiv9R6aBvBN7H9+K+eYn2xcCHQD/58zhPa3IlLuJxuNMQeAr2O3Og9h74T5Kz/ZmMNENvn1OfpXnNq8zRc2isiNwAFjzJZxp3xh33hUNM5E8hzzTVyyiFQA/wh80Rhz3Gt7chGRjwFdxphXvLZlEoqAFcD9xpj3ACfwfrjsNJx5gbXAAqAZKBeRP/TWqmnju+dIRO7GHub9fvZQnstm1UYRKQPuBr6a73SeY57XRSoaZ9IBpHPet2APDXiOiBRjC8b3jTE/dQ53ikiTc74J6PLKPuD9wI0i8g72sN6HROQf8JeNHUCHMeYl5/1PsEXETzZ+GHjbGNNtjBkBfgq8z2c2ZpnIJl89RyJyK/Ax4NPmVHKaH2w8H7txsMV5blqAzSLS6BP7zkBF40w2Am0iskBEEtgTURs8tgkREexx+B3GmG/knNoA3Or8fivw2GzblsUYc5cxpsUYMx/77/bPxpg/xF82vgvsF5HFzqFrsPej942N2MNSV4hImeP3a7DnsPxkY5aJbNoArBOREhFZALQBL3tgHyKyBvgKcKMx5mTOKc9tNMZsM8bUG2PmO89NB7DC+T/13L68GGP0Z9wPcAN2lMUe4G6v7XFsugq7a7oVeM35uQGYgx21stt5rfXaVsfe1cA/Ob/7ykZgObDJ+Vv+H6DGhzZ+DXgTeB34e6DEaxuBH2LPsYxgV26fncwm7GGXPcBO4HoPbWzHnhvIPjcPeGVjPvvGnX8HmOvl33CqH11GRFEURXGNDk8piqIorlHRUBRFUVyjoqEoiqK4RkVDURRFcY2KhqIoiuIaFQ1FURTFNSoaiqIoimv+L/cCl2pDlXKcAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, t_mul=1.)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*3)\n", "plt.plot([i for i in range(num_epoch*3)], lr_per_epoch, label=\"With `t_mul=1.`\");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### `t_mul=2.`" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvkElEQVR4nO3deXhc9Xno8e87o30dyVosWZIXLNvYjmUbYewAgSQsNgVMFlKTpDgJKaWFliS3t4GbNr25Kb3cNE0TWgIhS2NaGuM0SXEJCRCHJYTNBmxjGy/yKnmR5EWyvGibee8fcySPZS1HtqRzZub9PI+embP8zrwjz/jVbz2iqhhjjDFuBLwOwBhjTPywpGGMMcY1SxrGGGNcs6RhjDHGNUsaxhhjXEvxOoDRVlRUpJMmTfI6DGOMiStvvfXWYVUt7rs/4ZPGpEmTWLdunddhGGNMXBGRvf3tt+YpY4wxrlnSMMYY45olDWOMMa5Z0jDGGOOaJQ1jjDGuuUoaIrJYRLaJSJ2I3NfPcRGRh5zjG0Vk/lBlRaRQRJ4XkR3OY4Gzf5yIvCAiJ0TkX/q8ziUi8q5zrYdERM7/rRtjjBmuIZOGiASBh4ElwEzgNhGZ2ee0JUC183Mn8IiLsvcBa1S1GljjbAO0A38D/GU/4TziXL/ntRa7epfGGGNGhJt5GguAOlXdBSAiK4GlwJaYc5YCj2t0nfXXRSQkImXApEHKLgWudsqvAF4EvqyqJ4FXRGRqbBDO9fJU9TVn+3HgFuBXw3vL7qx4dQ9HTnQgIoiA0PNI9NGp5PR3LNB7TM6c37PtHM9IDZKRGiAzNUhmapD01CBZaUGKctIZl51GIGCVqP5sbGjhN1saoxs9v2fO2kSI/bfpc6xP5bTn3+/s8uder2+dVvp5bel7LOZ6gYCQEhBSAgFSgkJqMBDdDvazLxAgLSVAdnqQ7LQUstNTSEuxlmTjD26SxgSgPma7AbjMxTkThihbqqoHAVT1oIiUuIijoZ/XOIeI3Em0RkJVVdUQl+3fE2/sZXvjifMqe6GCAaEoJ43SvAwuKs5h+vhcaicWUFMZIjWY3P95PLSmjt+81+h1GGMuNShkp6c4SSRITnoKhdnpFOemMS47naKcNCoKsqgszKKyMJOstISft2s84uaT1d+fvH3v3DTQOW7KuuX6Wqr6GPAYQG1t7Xm93nNfvKrnWqhGX0hVnUdQovvps91zXkSj0fXdr0AkorR3RWjvDnO6M8zprjDtXWFOdoQ5fKKDprZ2mo53cOh4O6/tPMIv3tkPQHZakOtmjefW2goWTRl3zl/NyaA7EqGmIp+n7rmid1/PjcR6/z1i9p953nPs7HOJOd73WOx1zt4+s+GmjGp0uyuidIcjdIWV7kiE7rDSPcC+ju4wpzrCnOzs5mRHNyc7w9HHjuhjW0cXDcdOsb6+haMnO6KftxhFOWlMH5/LrPJ8ZpXnMas8jylFOVaDNRfMTdJoACpjtiuAAy7PSRukbKOIlDm1jDKgyUUcFUPEMeJ6mpScrdF+uX4dPdnJm7uP8NL2Zp7eeJBfvLOfuZUh/ubGmVwyscCTmLwSjmg/TUxnNwfFHBmboDwWiShHTnbScOwU9cdOU3/0FPuOnOK9Q8f58e/30BmOAFCUk85V04q5enoxV1YXEcpK8zhyE4/cJI21QLWITAb2A8uAT/Y5ZzVwj9NncRnQ6iSD5kHKrgaWAw86j08NFoRzvTYRWQi8AdwO/LOL+ONeYXYai2eXsXh2GX970yyeWr+ff3xuOx975FU+8/5J/PUfXExKkjRbqUab78wZgYBQnJtOcW4686rO/iOiKxyhrukE7za08ru6w6zZ2sjP3m4gIHD51CI+fkkF188aT0Zq0KPoTbwZMmmoareI3AM8CwSBH6nqZhG5yzn+KPAMcANQB5wCPjtYWefSDwKrROQOYB9wa89risgeIA9IE5FbgOtUdQvwp8CPgUyiHeCj0gnuZxmpQf7w0ipunFPOPzy7jR+/uoedzSf4l0/OJz8z1evwRl04oljOcC81GODisjwuLsvjE5dWEo4oGxpa+O17Tfzinf3cu3I9RTlpLF80idsXTSI/K/E/Q+bCiPZt3E0wtbW1msir3D65dh9//V+bmFsZ4onPL0z4UTaf+N5rCPDknyzyOpS4F4kor+48wg9e2cWL25rJz0zlzz80lT9aNJH0FKt5JDsReUtVa/vuT+z/YZLAH15axbc+MZe1e47x1ac2keh/BKiqNU+NkEBAuKK6iB9/dgHP/MWV1FSG+LtfvscfPPQK7+w75nV4xqcsaSSAm2rKueeDU1m5tp7/eHOf1+GMqmjzlCWNkTazPI/HP7eAf/3spZzq6OZjj7zKt57bRrjvsCyT9CxpJIgvXTuNK6YW8f9+tZWjJzu9DmfUhBUbNjqKPji9hF9/8QN8dH4FD/22jjtWrKX1VJfXYRkfsaSRIAIB4W9vmsnJzjDf/s12r8MZNapK0HLGqMrLSOWbt9bwwEdm8/u6w3zs0VdpPN7udVjGJyxpJJDq0lw+dVkVT7yxjx2NbV6HMyqseWrsfOqyiTz+ucs42HKaT3zvNRqOnfI6JOMDljQSzBeumUZ2WpBvPZ+YtY1wRK15agwtumgc//b5yzh2spNPfv8NDp/o8Dok4zFLGgmmMDuNZQuqeH5LI00J2KSgCkGraYyp+VUFrPjcApra2rljxTpOd4a9Dsl4yJJGAlp2aSXdEeWnbzUMfXKcCasSsE/tmJtXVcB3ls1jY0MLX1q1PuGHdpuB2dcvAU0pzmHRlHH85M19RBJsyGTE+jQ8c/2s8dy/ZAa/2nSIx1/b63U4xiOWNBLUbZdV0XDsNL+rO+x1KCMqYpP7PPXHV07hQzNKeOCX77Fpf6vX4RgPWNJIUNfPKqUwO41V6+qHPjmOhNVqGl4SEb55aw0F2ancu/IdOrqtfyPZWNJIUOkpQa6bWcrL25vpcpbGTgSRCJY0PFaYncY3Pl7DzuaTPPriLq/DMWPMkkYCu3p6MW3t3by9N3HWEYqorXLrB1dNK+bGOWU8/GIdu5q9ucOl8YYljQR2+dQiUgLCi9ubvQ5lxIQj1qfhF1+9cSbpKQG++tRmG02VRCxpJLDcjFRqJxXwwtahbooYPyK29pRvlORl8KVrp/FK3WFe3pFYAy7MwCxpJLirp5ew9VAbh1oTY6KfNU/5y6cum0hFQSbf+PXWhBvebfpnSSPBXT29GIAXtyVGbSMcUZsR7iNpKQG+dO00Nh84zi/fPeh1OGYMWNJIcNNLcynLz+ClBOnXiKitPeU3S+dOYHppLt96frvdfyMJWNJIcCLCwinjeGvvsYTorLQZ4f4TDAj3XlPN7sMneW7zIa/DMaPMkkYSqKnIp6mtg0MJsIBh2GaE+9L1s8YzcVwW33t5V0L8cWIGZkkjCdRUhgDYUN/iaRwjIaI2uc+PggHh81dMZn19C2v3JM68IHMuSxpJ4OKyPFKDwvr6+F8rKNo85XUUpj8fv6SSwuw0Hnt5p9ehmFFkSSMJZKQGmVmWlxA1DWue8q/MtCCfXjiR37zXRP1Ru8tforKkkSRqKkO8u781rke3qCpqzVO+tuzSSgICT65NrIUyzRmWNJJETUWIEx3d7IzjdYJ68p0lDf8qD2Vy1bRifvpWPd0JtFCmOcOSRpKYWxUCYH0cN1H11JKC9qn1tWULqmg83sEL2xJjbpA5m339ksTkcdnkZqTEdb9GxBnKaZP7/O1DM0oozk1n5Zv7vA7FjAJLGkkiEBDmVOTzbhzfba03aVjzlK+lBgPcekkFL2xroqkt/ucGmbNZ0kgi00pzqWs6EbcLy/WEbWtP+d9H5k0govDMRluPKtFY0kgiU0tyONUZ5mCczgzv6dOwnOF/1aW5zBify+oNB7wOxYwwSxpJpLokF4AdjW0eR3J+Ir0d4ZY14sHNc8t5e1+LzdlIMK6ShogsFpFtIlInIvf1c1xE5CHn+EYRmT9UWREpFJHnRWSH81gQc+x+5/xtInJ9zP7bRORd5zV+LSJF5//Wk091SQ4AdU3xOey2p0/DkkZ8uGlOOQBPWxNVQhkyaYhIEHgYWALMBG4TkZl9TlsCVDs/dwKPuCh7H7BGVauBNc42zvFlwCxgMfBdEQmKSArwHeCDqjoH2Ajcc57vOykVZKcxLjstbpNGWHuapyxpxIPKwizmVYWsiSrBuKlpLADqVHWXqnYCK4Glfc5ZCjyuUa8DIREpG6LsUmCF83wFcEvM/pWq2qGqu4E65zri/GRL9H+NPMA+jcM0tSSHHXGaNCLOXDHrCI8fN80p572Dx9l9+KTXoZgR4iZpTABi1wRocPa5OWewsqWqehDAeSwZ7Fqq2gX8KfAu0WQxE/hhfwGLyJ0isk5E1jU32wSjWFNLcqhrOhGXy1efaZ7yOBDj2nWzSgH4zZZGjyMxI8XN16+/P+v6/o8z0Dluyrp6PRFJJZo05gHlRJun7u/vAqr6mKrWqmptcXHxEC+XXKpLcmg93UXziQ6vQxm2M6OnrKYRLyoKsri4LI/nLWkkDDdJowGojNmu4NxmoYHOGaxso9OEhfPYcxPrgcrMBVDVnRr9M3kV8H4X8ZsY1aXREVR1jfHXRNVb07CkEVeunVnKur1HOXqy0+tQzAhwkzTWAtUiMllE0oh2Uq/uc85q4HZnFNVCoNVpchqs7GpgufN8OfBUzP5lIpIuIpOJdq6/CewHZopIT9XhWuC9Yb7fpDe1ZwRVHC5c2Du5z0ZPxZVrLy4lovDbrU1Dn2x8L2WoE1S1W0TuAZ4FgsCPVHWziNzlHH8UeAa4gWin9Sngs4OVdS79ILBKRO4A9gG3OmU2i8gqYAvQDdytqmHggIh8DXhZRLqAvcBnRuB3kFRKctPJzUhhRxzWNGxyX3yaPSGP8XkZ/GZLIx+/pMLrcMwFGjJpAKjqM0QTQ+y+R2OeK3C327LO/iPAhwco8wDwQD/7HwUePbeEcUtEnBFU8TfBz+ZpxCcR4ZqZJfz87f20d4XJSA16HZK5ADYOJQlNKcphz+H4m6VrfRrx68MzSjnVGWbtnqNeh2IukCWNJFRVmEVjWzvtXWGvQxkWGz0Vvy6bUkhaMMDvdhz2OhRzgSxpJKHKwkxUYX/Laa9DGZbeyX3WPBV3stJSuHRyAS9vt3lT8c6SRhKqKswCYF+cLSRnk/vi25XVxWw91EZjnK6ybKLs65eEepJGQ5wlDVt7Kr59oDo6Wt5qG/HNkkYSKs5NJz0lEHc1DbWO8Lg2Y3wuRTnp1q8R5yxpJCERobIwK+6SRtjp07DbvcanQEC4srqIV+oOx+3dI40ljaRVVZhF/dH46gjvGT0VsE9t3PrAtCKOnuxky8HjXodizpN9/ZJUNGmciqvVbq15Kv4tmhK9b9rru454HIk5X5Y0klRFQSZtHd20nOryOhTXejrCAzbkNm6Nz89gclE2r+20pBGvLGkkqZ4RVPXH4qdfo7d5ymoacW3hlELe3H2099/TxBdLGkmqalz8zdVQW+U2ISycMo62jm62HLB+jXhkSSNJVRbEX9I4U9PwOBBzQRZOGQfAa7ts6G08sqSRpLLTUxiXnUZ9PCUNteapRFCal8GUomxe32WLF8YjSxpJrDLOht2qLY2eMC6bMo61u4/S3TP5xsQNSxpJLN4m+NnkvsSx6CKnX8Pma8QdSxpJrDyUwaHW9riZnRu2BQsTxqWTCgBYu+eYx5GY4bKvXxIrz8+kMxzhyMlOr0NxRa1PI2GU5WcyIZTJW3utXyPeWNJIYmX5GQAcbI2Pfg2bp5FYLp1UwNo9x+JqVQJjSSOplYcyATjQEh/3N+hJGtYRnhhqJxXS3NYRV4MxjCWNpBZvNY2eP0itopEYLp1UCGD3DY8zljSSWGF2GmkpAQ62xklNw4bcJpTqkhzyMlJYZ/0accWSRhITEcryMzgQJ/cKj9gqtwklEBAumVjAOhtBFVcsaSS5svyMuKlp9AwNttu9Jo7aSYXsaDpBy6n4GMFnLGkkvfL8TA7GSU3DOsITT+3E6HyNt/dZbSNeWNJIcmWhDBrbOuJimeqeEK15KnG8ryKfYEBYv6/F61CMS5Y0klxZfibhiNLU5v8mqp4+DbFPbcLISkthemku79S3eB2Kccm+fkmuPBQddhsPczV6m6esppFQ5laFWF/fEjfL2SQ7SxpJriw/OsEvHuZqROwmTAlpbmWItvZudh0+6XUoxgVLGkmu3Ekah+JgBFVv85TljIQyrzIEwHproooLrpKGiCwWkW0iUici9/VzXETkIef4RhGZP1RZESkUkedFZIfzWBBz7H7n/G0icn3M/jQReUxEtovIVhH52Pm/dQOQl5lCVlrQmqeMZy4qziE3PYX19TaCKh4MmTREJAg8DCwBZgK3icjMPqctAaqdnzuBR1yUvQ9Yo6rVwBpnG+f4MmAWsBj4rnMdgK8ATao6zbneS+fxnk2Mngl+8dE8ZUNuE1EgIMypzOcdG0EVF9zUNBYAdaq6S1U7gZXA0j7nLAUe16jXgZCIlA1Rdimwwnm+ArglZv9KVe1Q1d1AnXMdgM8B/xdAVSOqajcZHgHloUwOxEPzlE3uS1jzKgvYeqiN051hr0MxQ3CTNCYA9THbDc4+N+cMVrZUVQ8COI8lg11LRELO9tdF5G0R+amIlPYXsIjcKSLrRGRdc3Ozi7eY3MryM+Jigl9Y1WoZCWpuZYhwRNl0oNXrUMwQ3CSN/r6lfcfGDXSOm7JuXy8FqAB+r6rzgdeAb/Z3AVV9TFVrVbW2uLh4iJcz4/MyOHyiw/f3a46o9WckqrlVIQCb5BcH3CSNBqAyZrsCOODynMHKNjpNWDiPTUNc6whwCviFs/+nwHzMBSvNzyCi0Hyiw+tQBhWJqI2cSlBFOelUFGTyjnWG+56bpLEWqBaRySKSRrSTenWfc1YDtzujqBYCrU6T02BlVwPLnefLgadi9i8TkXQRmUy0c/1Njd7e67+Bq53zPgxsGd7bNf0Znxed4Of3YbfhiDVPJbK5lSGracSBlKFOUNVuEbkHeBYIAj9S1c0icpdz/FHgGeAGop3Wp4DPDlbWufSDwCoRuQPYB9zqlNksIquIJoRu4G5V7ekd+zLwbyLybaC553XMhSl1kkbjcZ/XNNRu9ZrI5laGeHrjQZqOt1PifCaN/wyZNABU9RmiiSF236MxzxW4221ZZ/8RorWF/so8ADzQz/69wAfcxGzcO5M0/F3TiKhiFY3ENa8qOlXrnfoWrp813uNozEBsRrhhXHYaqUHhUBwkDWueSlyzyvNIDYrN1/A5SxqGQEAoyc2gMQ76NKx5KnFlpAa5uCzPZob7nCUNA0BpXnpc1DQCVtNIaHMrQ7zb0BoX93dJVpY0DBDt1/B9n0bE5mkkurmVIU52htnR1OZ1KGYAljQM0JM0/D16Kmwd4Qlvbs+Kt9av4VuWNAwA4/MzONHRzYmObq9DGVAkYs1TiW7SuGzyMlLY0GDLifiVJQ0DxMcEPxs9lfgCAWFORYiNDS1eh2IGYEnDAPExVyNsk/uSQk1lPlsPtdHeZSve+pElDQNER0+Bv5NGJGJ9GslgTkV0xdvNB457HYrphyUNA0T7NABfD7u15qnkUFMRAmCD3f7VlyxpGACy0lLIzUjx9QQ/m9yXHMbnZ1Cal279Gj5lScP0Gp+X4fuahiWN5BDtDLcRVH5kScP0Gp+fwSEfz9WIqN0fPFnMrQyx6/BJWk93eR2K6cOShulVkptBk49rGmHrCE8acyryAXjXahu+Y0nD9Bqfn05TW4dv1/2xtaeSx5wJIQA2WL+G71jSML3G52UQjihHfHrbV+vTSB75WalMLsq2EVQ+ZEnD9OqZ4OfXzvBwRG3BwiQypyLfOsN9yJKG6VXq86VEIgoB+8QmjZqKEIeOt/t6wmkysq+g6dUzwa+xzafNUzZPI6nUVEY7w62Jyl8saZheRTnpBAPi2wl+YZsRnlRmlecTDIg1UfmMJQ3TKxgQinP8ewe/iC1YmFQyUoNML821EVQ+Y0nDnKU037938LMFC5NPTWWIDfUtqPpzGHgysqRhzlKam+7bjvBwxJqnkk1NRT7H27vZc+SU16EYhyUNc5bxfq5p2DyNpDPHWfHWFi/0D0sa5iyleRkcb+/mdKf/boBjSSP5TCvNISM1wIZ66wz3C0sa5izjfTzBz5qnkk9KMMDs8nzrDPcRSxrmLL03Y/Jhv4YqtvZUEqqpDLH5QCtd4YjXoRgsaZg+/Hzb17Da6KlkNKcin/auCNsb27wOxWBJw/TRs5SIL5OGrT2VlGp6O8OtX8MPLGmYs+RmpJKdFvRln4YqiCWNpDNxXBb5mak2gsonLGmYc/h1gl+0I9zrKMxYExHmVOSz3kZQ+YKrr6CILBaRbSJSJyL39XNcROQh5/hGEZk/VFkRKRSR50Vkh/NYEHPsfuf8bSJyfT+vt1pENg3/7Ro3xudl+LIjPGJrTyWtuZUhtje2+XIoeLIZMmmISBB4GFgCzARuE5GZfU5bAlQ7P3cCj7goex+wRlWrgTXONs7xZcAsYDHwXec6PfF8FDhxPm/WuFOal0GjD+8VHlG15qkkNaciRDiibDlotQ2vualpLADqVHWXqnYCK4Glfc5ZCjyuUa8DIREpG6LsUmCF83wFcEvM/pWq2qGqu4E65zqISA7wJeDvhv9WjVvRpNFOxGe3fbWO8OTVs0z6O/tavA3EuEoaE4D6mO0GZ5+bcwYrW6qqBwGcxxIXr/d14B+BQReiEZE7RWSdiKxrbm4e7FTTj/JQBt0R5fBJf9U2Ioo1TyWpktwMJoQyWW/31vCcm6TR37e075+gA53jpqyr1xORucBUVf3FEOVR1cdUtVZVa4uLi4c63fRRlp8JwMEWf/VrRCKKVTSS19yqkNU0fMBN0mgAKmO2K4ADLs8ZrGyj04SF89g0xLUWAZeIyB7gFWCaiLzoIn4zTGXOrPCDrac9juRsYbXmqWQ2v6qA/S2nfTmyL5m4SRprgWoRmSwiaUQ7qVf3OWc1cLszimoh0Oo0OQ1WdjWw3Hm+HHgqZv8yEUkXkclEO9ffVNVHVLVcVScBVwDbVfXq83jPZgjloWhN44Dfaho2eiqpzasKAdav4bWUoU5Q1W4RuQd4FggCP1LVzSJyl3P8UeAZ4AaindangM8OVta59IPAKhG5A9gH3OqU2Swiq4AtQDdwt6raOLsxVJCVSnpKwHc1jUjEJvcls1nleaQFA7yz7xiLZ4/3OpykNWTSAFDVZ4gmhth9j8Y8V+But2Wd/UeADw9Q5gHggUHi2QPMdhG6OQ8iQll+Bgd8Nlcjeo9wr6MwXklPCTKzPM9qGh6zr6DpV1l+JgdbfFbTsD6NpDe/qoCN+1tsxVsPWdIw/SoLZXDQRzUNVbW1pwzzqkK0d0XYetBWvPWKJQ3Tr/L8TBqPt9Ptk7/ows5EQ+sIT249neFv7zvmbSBJzJKG6VdZKIOIQlObPyb49UxOt6SR3CaEMinNS+etvZY0vGJJw/SrvGeCn09GUEU0mjWsdSq5iQi1kwpZt+eo16EkLUsapl9loegEP7/M1ehtnrKskfRqJxZwoLWd/T4bqJEsLGmYfpX5tKYRsKSR9C6dVAhgtQ2PWNIw/crLSCE7LeibmkbE6Y8PWJ9G0psxPpfstKD1a3jEkobpl4hQFsr0XU0jaDkj6aUEA8yrKmDtHksaXrCkYQZUlu+fuRrhnuYpq2kYoHZSAVsPHed4e5fXoSQdSxpmQOX5mT5qnrI+DXPGpZMKUbXFC71gScMMqCyUweETHXR0e79epM3TMLHmVoYIBoS1u60zfKxZ0jADqizIAmD/Me/7NXqbpyxnGCA7PYXZE/J5Y/cRr0NJOpY0zIAqC6NJY9/RQe+uOyasecr0tWjKONbXt3C60/uacDKxpGEGVOUkjXof1DR6R09ZVcM4Fk4ppCusNvR2jFnSMAMqyU0nLSVAvQ9qGmGraZg+aicVEgwIr++yJqqxZEnDDCgQECoKMtl3xPukEbEht6aPnPQU5lTk85oljTFlScMMqqowi/pjfkga0Udbe8rEWjhlHBvqWzjV2e11KEnDkoYZVFVhFvuOnEKdv/S9cqZ5ytMwjM8smjKO7oiyzmaHjxlLGmZQlQVZtHV003ra25m3vUnDsoaJccnEAlICYk1UY8iShhlUz7Db+qPejqBSa54y/chOT2FeVYhXdhz2OpSkYUnDDKrKJ3M1zqw95WkYxoc+UF3MpgOtHDnhj7tMJjr7CppBVRZG76vhdWe4Dbk1A7lyWjGq8Eqd1TbGgiUNM6jcjFQKslI9r2mo3YTJDOB9E/IJZaXy8nZLGmPBkoYZUlVhlucT/Hpv92od4aaPYEC4fGoRv9vR7Pkov2RgScMMqdIHSaNnnobVNEx/rqoupqmtg22NbV6HkvAsaZghVRZm0XDsdO9f+16I2Cq3ZhBXTisC4OXtzR5HkvgsaZghTRqXRXdEPV0i3ZqnzGDK8jOZVprDb7c2eR1KwrOkYYY0tSQHgLpm76r+tvaUGco1F5eyds8xWk51eh1KQrOkYYY0tTgXgB2NJzyLIWKjp8wQrp1ZSjiivLjNmqhGk6ukISKLRWSbiNSJyH39HBcRecg5vlFE5g9VVkQKReR5EdnhPBbEHLvfOX+biFzv7MsSkV+KyFYR2SwiD17YWzdu5WelUpybzo4m75JGOBJ9tBnhZiA1FSGKc9N5fkuj16EktCGThogEgYeBJcBM4DYRmdnntCVAtfNzJ/CIi7L3AWtUtRpY42zjHF8GzAIWA991rgPwTVWdAcwDLheRJefzps3wVZfkUOdh0ojYjHAzhEBAuObiEl7a3uyL+9onKjdfwQVAnaruUtVOYCWwtM85S4HHNep1ICQiZUOUXQqscJ6vAG6J2b9SVTtUdTdQByxQ1VOq+gKAc623gYrhv2VzPqY6ScOrcfB2u1fjxrUzSznR0c3ru456HUrCcpM0JgD1MdsNzj435wxWtlRVDwI4jyVuX09EQsBNRGso5xCRO0VknYisa2629s2RUF2Sw4mObg4db/fk9cN2u1fjwvsvKiIzNchzmw95HUrCcpM0+vuW9v1zc6Bz3JQd1uuJSArwE+AhVd3V3wVU9TFVrVXV2uLi4iFezrgxtSTaGe5VE5VN7jNuZKQG+dCMEp7dfIjuno4wM6LcJI0GoDJmuwI44PKcwco2Ok1YOI89A6yHer3HgB2q+m0XsZsR0jPs1qsRVBG7CZNx6aaaMg6f6LR7bIwSN0ljLVAtIpNFJI1oJ/XqPuesBm53RlEtBFqdJqfByq4GljvPlwNPxexfJiLpIjKZaOf6mwAi8ndAPvCF4b9VcyGKctIIZaV6NoLKJvcZt66eXkJOegqr1/f929aMhCGThqp2A/cAzwLvAatUdbOI3CUidzmnPQPsItpp/X3gzwYr65R5ELhWRHYA1zrbOMdXAVuAXwN3q2pYRCqArxAdhfW2iKwXkc9f6C/AuCMiVJfksNOz5inrCDfuZKQGuW5WKb/efMhGUY2CFDcnqeozRBND7L5HY54rcLfbss7+I8CHByjzAPBAn30N9N/fYcbI1JIcfrXpEKqKjPF/3jYj3AzHzTXl/Pzt/by0rZnrZo33OpyEYqPejWtTS3JpOdXF4RNjv0yDTe4zw3H51CIKs9N4ypqoRpwlDeParPI8ADbtbx3z17ZVbs1wpAYD3FxTzvNbGjl60taiGkmWNIxr75uQT0BgfX3LmL+2NU+Z4bptQRWd4Qg/f7vB61ASiiUN41p2egrVJblsaGgZ89fuGXJrzVPGrenjc5lXFeInb+6zO/qNIEsaZljmVobYUN8y5l/CsE3uM+fhtkur2Nl8knV7j3kdSsKwpGGGpaYyxLFTXewb49u/9k7us0+sGYYba8rISU/hP97Y53UoCcO+gmZYairzgbHv14jY2lPmPGSlpfCx+RN4euMBDrV6s25aorGkYYZlWmkuGakBNtSP7QiqsE3uM+fpjiumEI4oP351j9ehJARLGmZYUoMBZpfnj3lnuC2Nbs5X1bgslswu44k39nKio9vrcOKeJQ0zbDWVITbtb6VrDFcR7Vnl1pqnzPm48wNTaGvvZuWb1rdxoSxpmGGbVxWiozvCu2M4yS9sq9yaC1BTGWLhlEIee3kXpzttPaoLYUnDDNvlFxUhAi9tG7sbXEVUEWHM17wyieNL106nqa2DFa/t8TqUuGZJwwxbQXYacytDvLitaeiTR0hE1Sb2mQuyYHIhH5pRwiMv7qT1dJfX4cQtSxrmvHxwegkb97dy+ETHmLxeOGKd4ObC/c/rp3O8vYvvvbTT61DiliUNc16unl6MKry8fWyaqCKqNrHPXLCLy/K4Ze4EfvDKbnYfPul1OHHJvobmvMwuz6coJ40Xx6hfIxJRq2mYEXH/khmkBwP8zX9tsjWpzoMlDXNeAgHhA9OKeXlHc+/IptEUtj4NM0JK8jL4qyUzeKXusN1v4zxY0jDn7YPTS2g51cUbu4+M+mup2rLoZuR8akEVcytDfO2/N9vyIsNkScOct2suLiU3I4Un19aP+muFI2pzNMyICQSEf/xEDR3dEe5d+c6Y1JYThSUNc94y04J8ZN4EfvXuIY6N8t3Rwqo2G9yMqIuKc/g/S2fzxu6jPLRmh9fhxA1LGuaCfPKy6N3RfjbKd0dTtY5wM/I+fkkFH50/ge+s2cEvNx70Opy4YEnDXJAZ4/PG5O5oYRs9ZUbJ33/kfdROLOCLq9azbs9Rr8PxPUsa5oLdtiB6d7SXdxwetdcIR2yxQjM6MlKDfP/2WiaEMrljxTo2enA743hiScNcsKVzy6kqzOLvf/ke3aO08q3a5D4zigqy03j8cwvIy0zhU99/w2ocg7Cvoblg6SlB/tcNM9jW2MbKURpJFbY+DTPKKguzWPUniyjOTefTP3yDp9bv9zokX7KkYUbE9bPGc9nkQr71/HZaT438YnDhiE3uM6OvLD+TVXctYs6EEPeuXM/Xn95CR7ctpR7LkoYZESLCV2+ayfHTXXzhyZEf926T+8xYKcpJ54k/vozliybyw1d2c9M/v2L9HDEsaZgRM6s8n/998yxe2NbMN57dOqLXtsl9ZiylBgN8bels/vUzl9J6uotbHv49X/7PjTQet9njKV4HYBLLpxdOZOuh43zvpV3kpqdw9wenjsiNk6xPw3jhgzNKeO6LV/HQmh08/toentqwn1svqeTzV05m4rhsr8PzhCUNM+L+9qZZnGjv5pvPbaeu6QR//9H3kZV2YR81tRnhxiP5man8zY0zWb5oEv/ywg6eXFvPv7+xl8svKuJjl0zgQ9NLyc9K9TrMMWNJw4y41GCAf/rDuVSX5vIPz27j9V1H+avF01k6d8J5/8dvk/uM16rGZfGNj9fwl9dN54k39vGztxv44pMbCAjMryrgqmnFXDW9mIvL8kgNJm7Lv7iZxSsii4HvAEHgB6r6YJ/j4hy/ATgFfEZV3x6srIgUAk8Ck4A9wCdU9Zhz7H7gDiAM/IWqPuvsvwT4MZAJPAPcq0O8gdraWl23bt2Q79GMjrV7jvL1p7ewsaGVCaFMPnZJBdfNLGVmWd6wOrZv/9GbtJ7u4qm7Lx/FaI1xLxJR3qk/xovbmnlpezMbG1oBSEsJML00l1nlecwqz2NyUQ6VhZmUhzLjKpmIyFuqWnvO/qGShogEge3AtUADsBa4TVW3xJxzA/DnRJPGZcB3VPWywcqKyDeAo6r6oIjcBxSo6pdFZCbwE2ABUA78BpimqmEReRO4F3idaNJ4SFV/NVj8ljS8F4kov958iJVr6/ndjmZUoSArldkT8pkxPpeKgixK89Ipzs2gJDed/KxUMlKCpAaltz/kj374Bic6uvnFn1nSMP50+EQHr+48wqb9rWw+0Mqm/cfPuhd5QKJDeotz0xmXnUZBdlrvY2FWGrkZKWSkBclMjf5kpQXJSA2SmRZ9nhYMEAzIiPQRujFQ0nDTPLUAqFPVXc6FVgJLgS0x5ywFHnf+6n9dREIiUka0FjFQ2aXA1U75FcCLwJed/StVtQPYLSJ1wAIR2QPkqeprzrUeB24BBk0axnuBgHDD+8q44X1lNB1v5/c7D/PaziNsOXicx1/bS0d3/7PIgwEhIyVAZlqQ46e7mVORP8aRG+NeUU46N9eUc3NNORDthzvY2s7eI6eoP3qK+mPRxyMnOzl0vJ0tB49z5GQnnQN8/gcSDAjBgJBy1mOgdzv22H//+RVkpAZH9H26SRoTgNhpvg1EaxNDnTNhiLKlqnoQQFUPikhJzLVe7+daXc7zvvvPISJ3AncCVFVVDfLWzFgrycvgI/Mq+Mi8CiDaV3H0ZCdNbe00tXXQdLydtvZu2rvCtHdFON0Vpr0rzOnOMNfMLPU4emPcExHKQ9FmqUUXjev3HFXldFeYIyc6OdnZzenOcO9n/lRn9HPf3hXd19EVIaxKOKJ0R5zHsBKORM5sn/UYGZXBI26SRn+v2rdNa6Bz3JR1+3qur6WqjwGPQbR5aojXMx4KBoTi3HSKc9OZ5XUwxowxESErLYWswvgZk+SmV6YBqIzZrgD63lh3oHMGK9voNGHhPDa5uFbFEHEYY4wZRW6SxlqgWkQmi0gasAxY3eec1cDtErUQaHWangYruxpY7jxfDjwVs3+ZiKSLyGSgGnjTuV6biCx0RmvdHlPGGGPMGBiyTqSq3SJyD/As0WGzP1LVzSJyl3P8UaIjmW4A6ogOuf3sYGWdSz8IrBKRO4B9wK1Omc0isopoZ3k3cLeq9qwY9qecGXL7K6wT3BhjxpSreRrxzIbcGmPM8A005DZ+ZpoYY4zxnCUNY4wxrlnSMMYY45olDWOMMa4lfEe4iDQDe8+zeBFweATDGQ0W44Xze3xgMY4Ui9G9iapa3HdnwieNCyEi6/obPeAnFuOF83t8YDGOFIvxwlnzlDHGGNcsaRhjjHHNksbgHvM6ABcsxgvn9/jAYhwpFuMFsj4NY4wxrllNwxhjjGuWNIwxxrhmSaMfIrJYRLaJSJ1z/3LPiUiliLwgIu+JyGYRudfZXygiz4vIDuexwAexBkXkHRF52o8xOrcj/k8R2er8Phf5MMYvOv/Om0TkJyKS4XWMIvIjEWkSkU0x+waMSUTud75D20Tkeg9j/Afn33qjiPxCREJexdhffDHH/lJEVESKvIrPDUsafYhIEHgYWALMBG4TkZneRgVEl4n/H6p6MbAQuNuJ6z5gjapWA2ucba/dC7wXs+23GL8D/FpVZwA1RGP1TYwiMgH4C6BWVWcTva3AMh/E+GNgcZ99/cbkfDaXAbOcMt91vltexPg8MFtV5wDbgfs9jLG/+BCRSuBaoreJ6Nnn1e9wUJY0zrUAqFPVXaraCawElnocE6p6UFXfdp63Ef2PbgLR2FY4p60AbvEkQIeIVAB/APwgZrdvYhSRPOADwA8BVLVTVVvwUYyOFCBTRFKALKJ3qfQ0RlV9GTjaZ/dAMS0FVqpqh6ruJnqvnQVexKiqz6lqt7P5OmfuADrmMQ7wOwT4J+CvOPsW1p78DodiSeNcE4D6mO0GZ59viMgkYB7wBlDq3NUQ57HEw9AAvk30wx+J2eenGKcAzcC/Ok1oPxCRbD/FqKr7gW8S/avzINE7YT7npxhjDBSTX79Hn+PMzdt8EaOI3AzsV9UNfQ75Ir6+LGmcS/rZ55txySKSA/wM+IKqHvc6nlgiciPQpKpveR3LIFKA+cAjqjoPOIn3zWVncfoFlgKTgXIgW0Q+7W1Uw+a775GIfIVoM+8TPbv6OW1MYxSRLOArwFf7O9zPPs//L7Kkca4GoDJmu4Jo04DnRCSVaMJ4QlV/7uxuFJEy53gZ0ORVfMDlwM0isodos96HROTf8VeMDUCDqr7hbP8n0STipxivAXararOqdgE/B97vsxh7DBSTr75HIrIcuBH4lJ6ZnOaHGC8i+sfBBud7UwG8LSLjfRLfOSxpnGstUC0ik0UkjWhH1GqPY0JEhGg7/Huq+q2YQ6uB5c7z5cBTYx1bD1W9X1UrVHUS0d/bb1X10/grxkNAvYhMd3Z9mOj96H0TI9FmqYUikuX8u3+YaB+Wn2LsMVBMq4FlIpIuIpOBauBND+JDRBYDXwZuVtVTMYc8j1FV31XVElWd5HxvGoD5zufU8/j6par20+cHuIHoKIudwFe8jseJ6QqiVdONwHrn5wZgHNFRKzucx0KvY3XivRp42nnuqxiBucA653f5X0CBD2P8GrAV2AT8G5DudYzAT4j2sXQR/c/tjsFiItrsshPYBizxMMY6on0DPd+bR72Ksb/4+hzfAxR5+Tsc6seWETHGGOOaNU8ZY4xxzZKGMcYY1yxpGGOMcc2ShjHGGNcsaRhjjHHNkoYxxhjXLGkYY4xx7f8DCoULkI8aY+UAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, t_mul=2.)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*3)\n", "plt.plot([i for i in range(num_epoch*3)], lr_per_epoch, label=\"With `t_mul=1.`\");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see in the example, the second schedule that starts at epoch 50 takes twice as long for the learning to be decreased to the minimum value. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `lr_min` " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Defaults to `1e-5`. The minimum learning rate to use during the scheduling. The learning rate does not ever go below this value. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `decay_rate`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When `decay_rate` > 0 and <1., at every restart the learning rate is decayed by new learning rate which equals `lr * decay_rate`. So if `decay_rate=0.5`, then in that case, the new learning rate becomes half the initial `lr`. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### `decay_rate=1.` or no decay" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuU0lEQVR4nO3de5RcVZ3o8e+vqvpdnU76nXQn9CPdCZ2OCDRJAJeiCElADerohLkOAZ2LzERH5ylclzoPmWHN9XoV5bFQEdAZc1EZyQwIakZGEfJEhDxI0um8Oo9OJ52kO93pR1X97h91EtqmHyfprjpVp36ftbKq6tTZVXvDrv6dvffvnCOqijHGGONGwOsKGGOMSR8WNIwxxrhmQcMYY4xrFjSMMca4ZkHDGGOMayGvK5BopaWlWlNT43U1jDEmrWzZsuW4qpaN3O77oFFTU8PmzZu9roYxxqQVEdk/2nabnjLGGOOaBQ1jjDGuWdAwxhjjmgUNY4wxrlnQMMYY45qroCEiy0Rkp4i0isjdo7wvInK/8/5rInLFRGVFpFhEfi4iu53HGc72EhH5pYicEZFvjvieK0Xkdeez7hcRufimG2OMuVATBg0RCQIPAMuBJuBWEWkasdtyoMH5dyfwkIuydwPrVLUBWOe8BugHvgD89SjVecj5/HPftcxVK40xxkwJN+dpLAJaVbUNQETWACuA7cP2WQE8ofHrrK8XkekiMhOoGafsCuA6p/zjwAvA51S1F3hRROYOr4TzedNU9WXn9RPALcBPL6zJ7jz2m7109Q4Or4CrcmPtJQLivBt/7jzKm9sCIgScRxEhKBAMCMFAgFBACAaEUFDIDgbICgbIDgXIzQqSlxUkLztIYW6IgpwQBdlBbBCW+lSV76/fT2fPwJsbXfx/G2+PifrZ7/ex+GMwIAQC8mYfCwhZ5/uYkBMKkpsVICcUJJwTIpwbIpwTIicUsH6WgdwEjSrg4LDX7cBiF/tUTVC2QlWPAKjqEREpd1GP9lG+4y1E5E7iIxLmzJkzwceO7t82HmD3sTPE63dRH+GZUECYnp/FjPxsSsM5zJqeR9X0XKqL85lXUUhDRZj8bN+f15nyjnb384WntwHxP+zp1s9yQgGKC7KZkZ9N+bRz/SyPmpIC5lWGuaSkgKygLZv6jZu/HKMdSozs3mPt46asW64/S1UfAR4BaGlpuajv+9lfvOtiio1KVc//QdBzr4n/kVCGvacQUyWqisYgqko0psRUicSUSDTGUFQZisYYisYYiMToH4rSPxSjbzDCmYEIZ/ojdPcP0dU7xMneQY719PPSnuN0dPcTG/Zfora0gMW1xVxdX8I19aWUFeZMWXuNO4ORGABf/ehlfOiK6kl/npt+du55TCEaU1TjzyOxGLFY/DESVSKxGIMRZTAaY2AoSn8kxtnB6Pl+1tMfofvsEF29g3T1DnK0u5/X2k//3ug8Oxjg0pmFXF1fytX1JSyqKSYvOzjpdhpvuQka7cDsYa+rgcMu98kep2yHiMx0RhkzgWMu6jH8lzVaPVKSOFMBw7YkvQ5D0RjtJ8+yq6OHXUd7+F37KZ55/QhrNh1EBN4xt5Q/uLKapQsqyc2yH3YyRJwoHgxMTX9IhX7WNxihrbOXXR097Ozo4ZX9J/n2r9t4+L/3kJ8d5OaFM/mDK6tZVFtsU1tpyk3Q2AQ0iEgtcAhYCfzRiH3WAp9y1iwWA6edYNA5Ttm1wCrgPufx6fEq4Xxej4gsATYAtwHfcFF/A2QFA9SWFlBbWsDSBZVA/Ehz2+HT/Hx7B0+9cojPrHmVorwsVr+7ntuurrHgkWDRKQ4aqSA/O0RzVRHNVUXnt/UORNi0r4tnXz/CM68d4Ydb2mkoD3P38vm8Z365BY80I27uES4iNwFfA4LAo6p6r4jcBaCqDzupr98kns3UB9yhqpvHKutsLwGeBOYAB4CPqGqX894+YBrxkcop4EZV3S4iLcBjQB7xBfBP6wQNaGlpUbtg4cRiMWV92wke+XUbL+zspHpGHn+7bD7vf9tM+1EnyPbD3dx0/695+GNXsKx5ptfVSYq+wQjPvn6UB3/ZStvxXq6uK+HzN1/6e0HGpAYR2aKqLW/Z7iZopDMLGhfuN63HufeZHWw/0s0HL6/inz+00EYdCbD10Gne940X+dZtLdzQVOF1dZJqKBrjBxsP8LVf7Kb77BBfen8TH1tyiR2gpJCxgoalNpi3uHZuKf/56XfwVzc08pNXD/GRh1/m8KmzXlfLd86taYR8ND3lVlYwwG1X1/DLv76OdzaW8YWnt3HPU68zEIl6XTUzAQsaZlSBgPDp6xv41h+3sPd4Lx/45ovsONLtdbV8JRqLZ0/5aU3jQhXlZfGt21pY/e561mw6yMe+vYHegYjX1TLjsKBhxvXepgp+svpaQoEAn3hsE8e6+72ukm9Eopk70hguGBD+Zul87r/1crbsP8ln1rx6PknApB4LGmZCc8vDfOf2Fk6dHeITj2+mb9COBKeCH7OnJuMDl83iS+9fwC92dPBPz+7wujpmDBY0jCsLZhXxjVsvZ9vh03zWjgSnxPk1jaAFjXNWXVPD7dfU8J0X9/K99aPebdR4zIKGce36Syv4wvua+Nn2Dr796zavq5P23hxp2M9wuC+8r4nr55fzd2u3sfXQaa+rY0aw3mouyB3X1vLeS8v5+rrdHDltGVWTkcnZU+MJBoSvfvTtTM/L4otPbyVmo9qUYkHDXLAvvX8B0Zjy5Wds3nkyLHtqbEX5Wdy9fD6vHDjFj15pn7iASRoLGuaCzS7O58+um8szrx3hxd3Hva5O2rKRxvg+fEU1V14yg/t++gan+4a8ro5xWNAwF+WT76pjTnE+X1y79fzVWs2Fseyp8QUCwj+sWMCpvkG+8rOdXlfHOCxomIuSmxXk7z7QRFtnL/+2wbJcLsab52nYz3AsC2YVcdvVNXx/w35anfvbGG9ZbzUX7T3zK2i5ZAbffWmfpeBehPMjDUu5Hden3zOXrGCAx17a63VVDBY0zCTdfm0N+0/08cLOiW6HYkayNQ13SsI5rLhsFj/ecojTZ21tw2sWNMykLF1QSeW0XL77m31eVyXtWPaUe7dfW8PZoShPbjo48c4moSxomEnJCgb446sv4cXW4+zu6PG6OmnFRhruLZhVxKLaYh5/2aZCvWZBw0zarYvmkBMK8N2X9nldlbRi2VMX5o5ramg/eZZf7OjwuioZzYKGmbTigmxueXsVT73Sbvn0F+DNkYb9DN24oamCqul5PGZToZ6y3mqmxKpraugfivHDLTbn7JaNNC5MyJkKfbntBLtsKtQzFjTMlGiaNY2FVUX852tHvK5K2rD7aVy4D19RjQg8Y/3MMxY0zJRZ1lzJqwdPcfS03ajJjWgshkj8zGfjTllhDlddUszz2456XZWMZUHDTJmlCyoB+Nl2+0G7EYmpjTIuwtLmSt442sO+471eVyUjWdAwU2ZueZj6sgKe22pBw41oTG094yLc2FQBYKMNj1jQMFNqWXMlG/Z2cbJ30OuqpLz4SMN+ghdqdnE+zVXTLGh4xHqsmVJLF1QSjanl0rtgI42Lt7SpklcOnKKj29bPks2ChplSC6uKmFWUy/PbLGhMJBKL2ZrGRVrWfG79zPpZslnQMFNKRLhxQSW/2t1J70DE6+qkNBtpXLy55WHqSgt43tbPks6Chplyy5orGYzEeGFnp9dVSWmRqGVPXSwRYWlzJS+3neBUn62fJZMFDTPlrqopZkZ+FutsXWNc0ZjavTQm4camCqIx5b932cFJMlnQMFMuGBCW1JWwYW+X11VJaZY9NTlvq55OYU6I9W3Wz5LJeqxJiMW1xRw6dZb2k31eVyVl2ZrG5AQDQkvNDDbuPeF1VTKKBQ2TEIvrSgDYYEeBY7LsqclbXFfCns5eOnsGvK5KxnAVNERkmYjsFJFWEbl7lPdFRO533n9NRK6YqKyIFIvIz0Vkt/M4Y9h79zj77xSRpcO23yoirzvf8ZyIlF58000izasopCgviw12FDgmG2lM3qLaYgA22lRo0kwYNEQkCDwALAeagFtFpGnEbsuBBuffncBDLsreDaxT1QZgnfMa5/2VwAJgGfCgiARFJAR8HXi3qr4NeA341EW22yRYICBcVVNsP+Zx2LWnJm9hVRH52UGbokoiNyONRUCrqrap6iCwBlgxYp8VwBMatx6YLiIzJyi7Anjcef44cMuw7WtUdUBV9wKtzueI869ARASYBhy+4BabpFlSV8y+E3121u4YbKQxeVnBAFdeMsOSLpLITdCoAobfWafd2eZmn/HKVqjqEQDnsXy8z1LVIeBPgdeJB4sm4DujVVhE7hSRzSKyubPT0vG8srg2vq6xvs2OAkcTP0/DlhUna3FtMW8c7bHrnSWJmx472qHQyDu7j7WPm7Kuvk9EsogHjcuBWcSnp+4Z7QNU9RFVbVHVlrKysgm+ziTKpTMLCeeEbIpqDDbSmBqLnIOTTfusnyWDm6DRDswe9rqat04LjbXPeGU7nCksnMdjE3zW2wFUdY+qKvAkcI2L+huPhIIBWmps6mAskViMkJ3cN2mXzS4iJxSwfpYkboLGJqBBRGpFJJv4IvXaEfusBW5zsqiWAKedKafxyq4FVjnPVwFPD9u+UkRyRKSW+OL6RuAQ0CQi54YONwA7LrC9JskW15bQeuwMx89YSuRINtKYGjmhIJfPmW6ZekkyYdBQ1QjxLKXnif+RflJVt4nIXSJyl7Pbs0Ab8UXrbwF/Nl5Zp8x9wA0ispt4ALjPKbON+ChiO/AcsFpVo6p6GPh74Fci8hrxkcc/Ta75JtEW11lK5Fgse2rqLK4tYfvhbrr7h7yuiu+F3Oykqs8SDwzDtz087LkCq92WdbafAK4fo8y9wL2jbH8YePitJUyqWlhVRF5WkA1tJ7hp4Uyvq5NSbKQxdRbXFvN1hc37unjP/Aqvq+NrlrphEiorGOCy2UW82n7a66qkHLv21NS5fM4MAgKvHrR+lmjWY03CLawqYseRboaiMa+rklJspDF18rKDzC0Ps/WQBY1Es6BhEq65qojBSIzdHWe8rkpKsWtPTa3mWUW8bkEj4SxomIRrrioCYOth+0EPF43aSGMqNVcV0dkzwDG7AkFCWdAwCVdbUkBBdtCmDkaIxNTO05hCC6vjByc22kgsCxom4QIBYcGsIgsaI9iaxtRqmjkNEdh6qNvrqviaBQ2TFM1VRWw/0k3EFsPPs+ypqVWQE6KutMBGGglmPdYkxcLqafQPxdjT2et1VVKGjTSm3sIqG9EmmgUNkxTNs5zFcPtBn2fZU1OvuaqIo939die/BLKgYZKirixMXlbQpg6GicaUgAWNKWWZeolnQcMkRTAgNM2aZiONYezaU1OvadY0ALbaFQgSxoKGSZqFzmJ4NDbRLVX8LxZTVLE1jSk2LTeL2tICG2kkkAUNkzTNVUX0DUbZe9zODI9qPHDaSGPqNVcVWdptAlnQMEnTXBWfOrB1Dc6PtoKWcjvlmmdN49Cps3TZ7V8TwnqsSZq5ZWFyswJ2FEh8PQNspJEIC6ssUy+RLGiYpAkFA8yvtMVwiF93CmxNIxEWWAZVQlnQMEl16cxCdnX0oJrZi+GRWPzMeLv21NQrystiVlEuu472eF0VX7KgYZKqobyQk31DHD+T2fPNb65pWNBIhIaKQnbZpfgTwoKGSap5lYUA7OrI7KNAW9NIrHmVhbR2nrFrnSWABQ2TVA0VYQB2ZvjUgWVPJVZDeZjBSIz9XX1eV8V3rMeapCoL5zAjP4vdxzI7aNhII7HOjWh3Z/iINhEsaJikEhEaKwptpOEshNuaRmLMLQ8jAjuP2rrGVLOgYZKusaKQ3R1nMjqDykYaiZWfHWL2jHx2ZfiINhEsaJika6wspGcgwpHTmXsv54idp5FwjRWFlnabABY0TNLNq4jPN+/M4Pnmcwvhdp5G4syrDLP3eC+DEcugmkoWNEzSNToZVJm8SBmx7KmEa6woJBJT9h63u0VOJeuxJumm52dTXpiT0YuUUVvTSLhGG9EmhAUN44nGisKMPsEvYtlTCVdXVkAwILauMcUsaBhPNFYUsvtYD7EMvSGTjTQSLycUpKYkP6MPThLBgobxxLzKMP1DMQ6ezMwzdiN27amkmFeZ2SPaRHAVNERkmYjsFJFWEbl7lPdFRO533n9NRK6YqKyIFIvIz0Vkt/M4Y9h79zj77xSRpcO2Z4vIIyKyS0TeEJEPX3zTjZcaKs5dgyoz1zXOXRo9ZAvhCdVQXsj+rj76h6JeV8U3JuyxIhIEHgCWA03ArSLSNGK35UCD8+9O4CEXZe8G1qlqA7DOeY3z/kpgAbAMeND5HIDPA8dUtdH5vP++iDabFNBQHs+gytSjQBtpJMe8ykJUofVYZh6cJIKbw5xFQKuqtqnqILAGWDFinxXAExq3HpguIjMnKLsCeNx5/jhwy7Dta1R1QFX3Aq3O5wB8HPhnAFWNqerxC2uuSRWFuVlUTc/L2MuJ2HkayXE+gypD+1kiuAkaVcDBYa/bnW1u9hmvbIWqHgFwHsvH+ywRme68/kcReUVEfigiFaNVWETuFJHNIrK5s7PTRRONFxorwhk80rDsqWSoKcknOxiwy4lMITdBY7RePTLlZax93JR1+30hoBr4japeAbwMfGW0D1DVR1S1RVVbysrKJvg645W55fEzdqMZmEFl2VPJEQoGqCnNZ88xO8FvqrgJGu3A7GGvq4HDLvcZr2yHM4WF83hsgs86AfQB/+5s/yFwBSZtzS0PMxCJcejkWa+rknS2ppE8c8vD7Om0NY2p4iZobAIaRKRWRLKJL1KvHbHPWuA2J4tqCXDamXIar+xaYJXzfBXw9LDtK0UkR0RqiS+ub9T4JVH/A7jO2e96YPuFNdekkvqy+GJ4Jv6g3xxpWPZUotWXhTnQ1cdAxDKopkJooh1UNSIinwKeB4LAo6q6TUTuct5/GHgWuIn4onUfcMd4ZZ2Pvg94UkQ+ARwAPuKU2SYiTxIPCBFgtaqe+7/9OeB7IvI1oPPc95j0NDxovHt++QR7+4uNNJKnvixMNKYcONF3PtXbXLwJgwaAqj5LPDAM3/bwsOcKrHZb1tl+gvhoYbQy9wL3jrJ9P/BON3U2qW9GQTYlBdkZmQ4Zde5dbWsaiTfXSe9uPXbGgsYUsLGx8VR9WWbON58faVjKbcLVlhYAmTkNmggWNIyn6ssL2NOZeZktlj2VPAU5IWYV5WZkP0sECxrGU/VlYbp6B+nqHfS6KkllaxrJVV8ezshp0ESwoGE8VV+emRlUlj2VXOemQTP5vvRTxXqs8dTccxlUGXYUeG6kYQON5KgvD9M3GOVod+bel36qWNAwnqqankdOKJBxUwfRWIxQQBCxqJEM5w5OMq2fJYIFDeOpQECoy8AMqkhMbT0jierLnQwqCxqTZkHDeK6+LPMyqKJRtcypJCoL51CYG8q4fpYIFjSM5+aWhzl4MrNulGMjjeQSEbsG1RSxoGE8V18WRhX2Hs+co8BoTAkF7eeXTPVllnY7FazXGs9l4oULbaSRfPVlYY71DNDdP+R1VdKaBQ3jubqyAkTIqHsenMueMslz7hpUbbauMSkWNIzncrOCVM/Io9VGGiaB6sviGVQ2RTU5FjRMSqgvC2dUOmQ0ZtlTyTanOJ+soGTUNGgiWNAwKWFuWZi242eIZcitX22kkXyhYICakgIbaUySBQ2TEurLw/QPxTh0KjNu/Ro/T8N+fsmWqZfin0rWa01KyLQMKhtpeKO+vIADJ/oYcm6CZS6cBQ2TEuaev9ptZmS2RGMxQnYDpqSbWx4mElP2n+jzuippy4KGSQnFBdnMyM/KmPlmG2l4o94uXDhpFjRMysik+WbLnvJGXYZNgyaCBQ2TMurLwrRlyI/ZRhreCOeEqJyWa0FjEixomJQxtzzM8TODnOrz/61f4yMN+/l5YW55Zp0TNNWs15qUcf6eBxlwFBiJKQEbaXji3KX47davF8eChkkZ59NuM+AaVHbtKe/Ul4c5MxDhWM+A11VJSxY0TMqonpFPdiiQEdegikRtTcMrduvXybGgYVJGMCDUlRZkxHyzZU95p77cMqgmw4KGSSmZknYbVRtpeKW8MIdwTigjDk4SwYKGSSn15WEOdPn/1q820vCOiFBfHs6IadBEsKBhUkp9WQExxfeXeYivadjPzyv1ZQUZkXCRCNZrTUrJlAsX2kjDW/VlYY5293NmIOJ1VdKOBQ2TUjLl2kCRmBK0CxZ65vwFMn3ezxLBVdAQkWUislNEWkXk7lHeFxG533n/NRG5YqKyIlIsIj8Xkd3O44xh793j7L9TRJaO8n1rRWTrhTfXpLq87CBV0/MyYKRh52l4KVNGtIkwYdAQkSDwALAcaAJuFZGmEbstBxqcf3cCD7koezewTlUbgHXOa5z3VwILgGXAg87nnKvPhwD7P+1j9eX+z6Cya09565KSfEIBu/XrxXAz0lgEtKpqm6oOAmuAFSP2WQE8oXHrgekiMnOCsiuAx53njwO3DNu+RlUHVHUv0Op8DiISBv4S+PKFN9Wki3OLlH6+9autaXgrKxhgTkk+uzssaFwoN0GjCjg47HW7s83NPuOVrVDVIwDOY7mL7/tH4P8A46bWiMidIrJZRDZ3dnaOt6tJQY0VhZwdivr61q/xkYYtKXqpsbzQ92tnieCm1452ODTyEHCsfdyUdfV9IvJ2YK6q/vsE5VHVR1S1RVVbysrKJtrdpJjGikIAdh7t8bgmiWMjDe81Vhay70Sv788JmmpugkY7MHvY62rgsMt9xivb4Uxh4Twem+CzrgauFJF9wItAo4i84KL+Js00VMQXKXcd82fQUFWitqbhucaKMDG1xfAL5SZobAIaRKRWRLKJL1KvHbHPWuA2J4tqCXDamXIar+xaYJXzfBXw9LDtK0UkR0RqiS+ub1TVh1R1lqrWAO8AdqnqdRfRZpPipuVmMasol10+HWlEnbUaG2l4a54zot3V4c9+liihiXZQ1YiIfAp4HggCj6rqNhG5y3n/YeBZ4Cbii9Z9wB3jlXU++j7gSRH5BHAA+IhTZpuIPAlsByLAalW18WOGaawsZKdPFykjTtCw8zS8VVNaQFZQ2HnUn/0sUSYMGgCq+izxwDB828PDniuw2m1ZZ/sJ4PoxytwL3DtOffYBzS6qbtJUY0UhL+05QSQaIxT014KxjTRSQ1YwQF1pmN020rgg/vo1Gt9orChkMBJjf5f/rkF1fqRh2VOei49oLWhcCOu1JiU1OovhfjwKtJFG6mgsD9N+8iy9dg0q1yxomJQ0tzyMCL6cb47EYgCWPZUCGivji+G77XwN1yxomJSUnx1iTnG+L9NubaSROiyD6sJZ0DApq6G80Jdpt5HouTUNCxpem12cT04o4Mt+ligWNEzKmlcZZu/xXgYjMa+rMqXOjzQs5dZzwYDQUBFml01PuWZBw6SsxopCIjFl73F/3WHNsqdSS2OFP0e0iWK91qSs89eg8tl8s61ppJbGikKOdvdz+uyQ11VJCxY0TMqqKysgGBDfpd1a9lRqObcY7rd+ligWNEzKygkFqSnJ993Vbm2kkVrOXSDTbyPaRLGgYVLavMpC3+XQv7mmYUEjFVRNz6MgO2g3ZHLJgoZJaY0V8XsenB30zzUr3xxp2M8vFYgIjZWFvHG02+uqpAXrtSalNc2chirs8NEP2s7TSD1NM6ex/XA38WuvmvFY0DAprbmqCICth057XJOpY+dppJ7mqiK6+yMc8OEFMqeaBQ2T0mYW5VJSkM3r7f4JGpY9lXoWOgcnr/vo4CRRLGiYlCYiNFcVsfWwf6anLHsq9TRWFJIVFLYe8k8/SxQLGiblNVdNY3dHD/1D/lgMt+yp1JMdCjCvstBX06CJYkHDpLyFVUVEYsobPjlfw7KnUtPCqiJeP3TaFsMnYL3WpDy/LYbbSCM1NVcVcfrsEO0nz3pdlZRmQcOkvKrpeUzPz/JN0Ig6C+G2ppFammf56+AkUSxomJQnIuenDvzAztNITfMqCwkFxDf9LFEsaJi00FxVxK6OHgYi6b8YbudppKbcrCCNFYW+ytRLBAsaJi00zypiKKrs8sE9w21NI3U1V01jqy2Gj8uChkkLfjr5yrKnUtfCqiK6egc5fLrf66qkLOu1Ji3MLs5jWm6IrYfTP2jYSCN1LfBZpl4iWNAwaeH8meE++DFb9lTqapo5jWBAfNHPEsWChkkbC6uKeONID4ORmNdVmRQbaaSu3KwgDeVhX0yDJooFDZM2FlYXMRiNsf1Ieme3RKN27alUtrCqiFcPniIWs8Xw0VjQMGljUW0xABv3nvC4JpNjI43Utqi2mFN9Q767Y+RUsaBh0kZ5YS51pQVsaOvyuiqTEo0pwYAgYkEjFS2pKwFgQ5ofnCSKq6AhIstEZKeItIrI3aO8LyJyv/P+ayJyxURlRaRYRH4uIrudxxnD3rvH2X+niCx1tuWLyDMi8oaIbBOR+ybXdJOOFtcVs3Ff1/m01XQUcYKGSU3VM/KYVZTLhr3pfXCSKBMGDREJAg8Ay4Em4FYRaRqx23Kgwfl3J/CQi7J3A+tUtQFY57zGeX8lsABYBjzofA7AV1R1PnA5cK2ILL+YRpv0tai2mJ7+SFrfzzkai9l6RgoTERbVFrOhrctO8huFm5HGIqBVVdtUdRBYA6wYsc8K4AmNWw9MF5GZE5RdATzuPH8cuGXY9jWqOqCqe4FWYJGq9qnqLwGcz3oFqL7wJpt0trjWmTpI4ykqG2mkvsV1JRw/M0Db8V6vq5Jy3ASNKuDgsNftzjY3+4xXtkJVjwA4j+Vuv09EpgPvJz5CeQsRuVNENovI5s7OzvHaZtLMrOl5zC7OY2MaTx1EY2ojjRS3+HzSRfr2s0RxEzRG690jx2xj7eOm7AV9n4iEgB8A96tq22gfoKqPqGqLqraUlZVN8HUm3SyqKWHjvvSdOoiPNCwHJZXVlhZQGs5hQ5stho/kpue2A7OHva4GDrvcZ7yyHc4UFs7jMZff9wiwW1W/5qLuxocW1xXT1TuYtimR0agStJiR0kSExXXFbNibvgcnieKm624CGkSkVkSyiS9Srx2xz1rgNieLaglw2plyGq/sWmCV83wV8PSw7StFJEdEaokvrm8EEJEvA0XAZy+8qcYvlpxb10jTqYNITO1ihWlgSW0xR0732538Rpiw56pqBPgU8DywA3hSVbeJyF0icpez27NAG/FF628BfzZeWafMfcANIrIbuMF5jfP+k8B24DlgtapGRaQa+DzxLKxXRORVEfmTyf4HMOlndnEeM4ty03bqIBqL2UJ4GljsnK+xPk37WaKE3Oykqs8SDwzDtz087LkCq92WdbafAK4fo8y9wL0jtrUz+nqHyTDnUiJf2nMCVU27k+QithCeFuaWhZmRn8WGvV18pGX2xAUyhI2RTVpaXFtCZ096pkTG1FJu00EgED84Wd92wtY1hrGgYdLSOxtLAfjF9g6Pa3LhIlELGuniXY3ltJ88y86OHq+rkjIsaJi0VD0jn4VVRTy37ajXVblg0Zja/cHTxA1NFYjAc1vTr58ligUNk7aWNVfy2wOn6OhOr1tz2nka6aOsMIerLinm+W3pN6JNFOu5Jm0tXVABwM/SbLRhZ4SnlxsXVLDjSDf7T6Tf+lkiWNAwaWtueSH1ZQVpN0UVsZTbtLJ0QSUAz6dZP0sUCxomrS1dUMn6ti5O9Q16XRXXbKSRXmYX57Ng1jSbonJY0DBpbVlzJdGY8osdxybeOUXYVW7Tz7IFlWzZf5JjabZ+lggWNExaW1hVxKyi3LTKbrGRRvpZ2uxMUaVhivdUs6Bh0pqIcOOCSn69u5PegYjX1XElfp6G/fTSSUN5mLrSgrRLukgE67km7S1vrmQgEkub0YaNNNKPiLCsuZKX9pzg6OnMnqKyoGHS3qLaYurLCnjspX1pcbmHSCxG0E7uSzsrr5pDTJXvr9/vdVU8ZUHDpD0R4fZra3n90GleOXDS6+pMyEYa6WlOST7Xz6/g3zYeoH8o6nV1PGNBw/jChy6vojA3xKO/2ed1VSZk2VPp645ra+jqHWTt70behy5zWNAwvlCQE2LlVbN5butRjpxO7Zvm2EgjfV1TX0JjRZjHfpMeU6GJYEHD+MZtV9egqnzv5dSec7ZrT6UvEeH2a2rZfqSbjWl658jJsp5rfGN2cT7vvbSCH6T4nLONNNLbBy+voigvi8de2ud1VTxhQcP4yu3X1nCyb4h/3XDA66qMKRK1a0+ls7zsICsXzeb5bUfZeTTz7rNhQcP4ytV1JbyzsYyv/XwXx3pSM5/eRhrp75PvrGdaXhZffHprxq1tWNAwviIi/P0HFjAQiXHfs294XZ1RRWJq52mkueKCbP526Xw27O3KuEwqCxrGd2pLC/if76zlqd8eYkPbCa+r8xY20vCHP7xqNm+rLuLLz+ygp3/I6+okjQUN40ur3z2Xqul5fPHpbQxFY15X5zxVtewpnwgGhH9c0czxMwN87Re7va5O0ljPNb6Unx3iC+9rYmdHD/evS50fdMyZ/raRhj9cNns6K6+aw2Mv7ePlPak3qk0ECxrGt5YuqOCjLdV8479aeeqVdq+rA8SvOwVY9pSP3HPTfOpKC7jr+1vY03nG6+oknAUN41siwr0fXMg19SV87sevpcT6RtQZathIwz+m5Wbx6O1XEQoIH39sE1296XMXyYthQcP4WlYwwEP/40rmFOfzye9vofWYt3n1ESdo2EjDX2YX5/OtVS0cPd3PJ7+3mb7B9Li3y8WwoGF8ryg/i+/evohQQPjggy/xyze8uzVsNGojDb+6Ys4MvvrRt7Nl/0k+9OBLHOzq87pKCWFBw2SEOSX5/GT1tcyekc/HH9/Egy+0enJS1vmRRtB+en5089tm8t07FnH41Fk+8M0Xean1uNdVmnLWc03GqJ6Rz4//9BpuXjiTf3luJx/7zga2H+5Oah1sTcP/3tVYxtpPvYPScA5//OhG/m7tNl+tc1jQMBklLzvIN269nH9YsYBth7u5+Ru/5m9++LukTSVY9lRmqCkt4N9XX8vKq2bzvfX7ede//JKHXtjji5MAxe/XTWlpadHNmzd7XQ2Tgk73DfHAC6089pt9DEZjXFNfwh9cWc2NCyoJ54QS8p37T/Tyrv/9Al/96GV86IrqhHyHSS27O3q476dvsO6NY+RlBVneXMmHr6xmcW0xoRSephSRLaraMnK7q1+GiCwDvg4EgW+r6n0j3hfn/ZuAPuB2VX1lvLIiUgz8P6AG2Ad8VFVPOu/dA3wCiAJ/rqrPO9uvBB4D8oBngc+o36OeSZii/Cz+102Xcvs1NfxoSzs/2tLOXz75O0KB17hs9nSurivhypoZzK8spHJaLvFuPjmWPZV5GioK+c7tV/HqwVM8ufkg//HqYZ767SEKc0Isqi3m6voSLps9ncbyQorys7yu7oQmHGmISBDYBdwAtAObgFtVdfuwfW4CPk08aCwGvq6qi8crKyL/AnSp6n0icjcwQ1U/JyJNwA+ARcAs4BdAo6pGRWQj8BlgPfGgcb+q/nS8+ttIw7gViymb9nXxwq5OXt5zgtfaT50/g7swN0RdWZiq6bnMKsqjsiiXGfnZFBdkMz0/i3BOiHBuiPzsELlZAbKDgVGDzK6OHm78v7/igT+6gpvfNjPJLTSpoH8oyrodx3ix9Tgv7znOvhNvTo1WTsvlkpJ8Zk3PY9b0XMoLc5lRkE1xfjbT8kLxfpYTIj8nRG4okNCRymRGGouAVlVtcz5oDbAC2D5snxXAE85R/3oRmS4iM4mPIsYquwK4zin/OPAC8Dln+xpVHQD2ikgrsEhE9gHTVPVl57OeAG4Bxg0axrgVCAiL60pYXFcCQE//EDuO9LCzo4ddR3vYe7yXN4728F9vHKN/aPzrWQUEckJBskMBsoIBsoJCKCg4Sxqk8KyESbDcrCA3v23m+YOGo6f72XGk+3w/O3iyj417uzja3X8+cWIsoYCQEwqQda6fBYRQMEAoIAQDwn98+h3kZgWntP5ugkYVcHDY63bio4mJ9qmaoGyFqh4BUNUjIlI+7LPWj/JZQ87zkdvfQkTuBO4EmDNnzjhNM2ZshblZLKotZlFt8e9tV1W6z0Y42TdIV98gp/uGODMQ4cxAhN6BCP1DUfqHYvQPRYnElMFojKFIjGgsfrHCJXUltNQUj/GtJtNUFuVSWZTLu+eX/972SDTGqbNDnOob5MSZQXr6432sZyDC2cHI+T7WPxQjEosxFI0xGFFizkUxo7HE3OzLTdAY7VtHhr+x9nFT1u33uf4sVX0EeATi01MTfJ8xF0REKMrPoig/ixoKvK6O8alQMEBpOIfScA5zyyfeP1ncDJLbgdnDXlcDI+86MtY+45XtcKawcB7PnaY73mdVj7LdGGNMkrgJGpuABhGpFZFsYCWwdsQ+a4HbJG4JcNqZehqv7FpglfN8FfD0sO0rRSRHRGqBBmCj83k9IrLEyda6bVgZY4wxSTDh9JSqRkTkU8DzxNNmH1XVbSJyl/P+w8QzmW4CWomn3N4xXlnno+8DnhSRTwAHgI84ZbaJyJPEF8sjwGpVjTpl/pQ3U25/ii2CG2NMUtnJfcYYY95irJRbS/wzxhjjmgUNY4wxrlnQMMYY45oFDWOMMa75fiFcRDqB/RdZvBTw311UxpeJbYbMbHcmthkys90X0+ZLVLVs5EbfB43JEJHNo2UP+Fkmthkys92Z2GbIzHZPZZttesoYY4xrFjSMMca4ZkFjfI94XQEPZGKbITPbnYlthsxs95S12dY0jDHGuGYjDWOMMa5Z0DDGGOOaBY1RiMgyEdkpIq3O/ct9SURmi8gvRWSHiGwTkc8424tF5Ocistt5nOF1XaeaiARF5Lci8p/O60xo83QR+ZGIvOH8P7/a7+0Wkb9w+vZWEfmBiOT6sc0i8qiIHBORrcO2jdlOEbnH+fu2U0SWXsh3WdAYQUSCwAPAcqAJuFVEmrytVcJEgL9S1UuBJcBqp613A+tUtQFY57z2m88AO4a9zoQ2fx14TlXnA5cRb79v2y0iVcCfAy2q2kz89gwr8WebHwOWjdg2ajud3/hKYIFT5kHn754rFjTeahHQqqptqjoIrAFWeFynhFDVI6r6ivO8h/gfkSri7X3c2e1x4BZPKpggIlIN3Ax8e9hmv7d5GvBO4DsAqjqoqqfwebuJ3zMoT0RCQD7xu336rs2q+iuga8Tmsdq5AlijqgOqupf4fZAWuf0uCxpvVQUcHPa63dnmayJSA1wObAAqnDsl4jym0B2Kp8TXgL8FYsO2+b3NdUAn8F1nWu7bIlKAj9utqoeArxC/ydsR4ncU/Rk+bvMIY7VzUn/jLGi8lYyyzdd5ySISBn4MfFZVu72uTyKJyPuAY6q6xeu6JFkIuAJ4SFUvB3rxx7TMmJw5/BVALTALKBCRj3lbq5Qwqb9xFjTeqh2YPex1NfEhrS+JSBbxgPGvqvqUs7lDRGY6788EjnlVvwS4FviAiOwjPvX4HhH5Pv5uM8T7dbuqbnBe/4h4EPFzu98L7FXVTlUdAp4CrsHfbR5urHZO6m+cBY232gQ0iEitiGQTXzBa63GdEkJEhPgc9w5V/eqwt9YCq5znq4Cnk123RFHVe1S1WlVriP+//S9V/Rg+bjOAqh4FDorIPGfT9cB2/N3uA8ASEcl3+vr1xNft/Nzm4cZq51pgpYjkiEgt0ABsdPuhdkb4KETkJuLz3kHgUVW919saJYaIvAP4NfA6b87v/y/i6xpPAnOI//A+oqojF9nSnohcB/y1qr5PRErweZtF5O3EF/+zgTbgDuIHjr5tt4j8PfCHxDMFfwv8CRDGZ20WkR8A1xG/BHoH8CXgJ4zRThH5PPBx4v9dPquqP3X9XRY0jDHGuGXTU8YYY1yzoGGMMcY1CxrGGGNcs6BhjDHGNQsaxhhjXLOgYYwxxjULGsYYY1z7/1Mm/nAXrkZ3AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, decay_rate=1.)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*2)\n", "\n", "plt.plot([i for i in range(num_epoch*2)], lr_per_epoch);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### `decay_rate=0.5` " ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqXklEQVR4nO3de3xV5Zno8d+zd+53Qq4kIAECkuANI+Ct1doKWC1Oqy3MmUpt+6FOqdO5nsHTaTs9Uz9j50zPaW2t1mmt2LFSe5WqrVq0Wm9AtAoGBAJBCIQQ7peQy977OX/sBaYxJCuQ7LXX2s/388ln73V5135eeJNnr/d911qiqhhjjDFuhLwOwBhjjH9Y0jDGGOOaJQ1jjDGuWdIwxhjjmiUNY4wxrqV5HcBoKykp0YkTJ3odhjHG+Mprr722T1VL+68PfNKYOHEijY2NXodhjDG+IiLvDLTeuqeMMca4ZknDGGOMa5Y0jDHGuGZJwxhjjGuWNIwxxrjmKmmIyDwR2SQizSKybIDtIiJ3O9vXicjMocqKSLGIPCMiW5zXMc76sSLynIgcE5Hv9vuci0VkvXOsu0VEzrzqxhhjhmvIpCEiYeAeYD5QBywSkbp+u80Hap2fJcC9LsouA1apai2wylkG6AK+DPzjAOHc6xz/5GfNc1VLY4wxI8LNdRqzgGZV3QYgIiuABcCGPvssAB7S+H3WXxWRIhGpBCYOUnYBcJVTfjnwB+CfVfU48KKITOkbhHO8AlV9xVl+CLgR+O3wquzOgy+1cOB4T98AXJU73V4iIM7W+HvnVd5dFxIh5LyKCGGBcEgIh0KkhYRwSEgLCxnhEOnhEBlpIbLSw2Snh8nOCJOflUZuZhq5GWHsJMwYMxrcJI0qYGef5VZgtot9qoYoW66qbQCq2iYiZS7iaB3gM95DRJYQPyNhwoQJQxx2YD9Zs4Mte48Rj++MDuGZtJBQlJPOmJwMSvIyGVeUTVVRFtXFOUwrz6e2PI+cjMBf12mMGQVu/nIM9JW1/5/R0+3jpqxbro+lqvcD9wM0NDSc0ec9/XfvP5NiA1LVU4lHTy4TT0ZKn20KMVWiqmgMoqpEY0pMlUhMiURj9EaV3miM3miM7kiMrt4oXb0xOnsiHOuOcKwrwpGuXg4c7+Xg8R72Hu3i5a37aD/SRazPv0RNSS6za4q5dPJYLptcQml+5ojV1xgTXG6SRiswvs9yNbDb5T4Zg5RtF5FK5yyjEtjrIo7qIeJISiLSr3cr8V1HvdEYrQdPsLn9KJv3HOXN1sM8sb6NFWt3IgJXTCnhpourmVtfQVZ6OOHxGWP8wU3SWAvUikgNsAtYCPxlv31WAl9wxixmA4edZNAxSNmVwGLgLuf1scGCcI53VETmAKuBW4DvuIjfAOnhEDUludSU5DK3vgKAaExp2n2Y329o5xev7+KLK96gMDudpVdP5pZLJ1ryMMa8h7h5RriIXAd8CwgDD6jqnSJyG4Cq3udMff0u8dlMncCtqtp4urLO+rHAo8AEYAdws6oecLZtBwqIn6kcAq5V1Q0i0gA8CGQTHwC/XYeoQENDg9oNC4cWiymvbtvP/X/cxh82dVBVlM3/nDeNj1wwzgbVjUlBIvKaqja8Z72bpOFnljSG76Xmfdz5xEY2tB3hLy6q4t8/ep6ddRiTYk6XNOyKcPMel08p4fHbr+AfPjSVX7+xi5vue5ldh054HZYxJglY0jADCoWE26+p5b8+2cD2fZ185DsvsrHtiNdhGWM8ZknDDOqDdeX8eunlpIdDfObBtew90uV1SMYYD1nSMEOaUpbHDz/VwKETvXxmeSOdPRGvQzLGeMSShnGlflwh31l0EU27D/PFFW8QjQV7AoUxZmCWNIxr10wv5yvX1/HMhnZ+8MdtXodjjPGAJQ0zLJ+6vIYPTi/n26u20HbYZlQZk2osaZhh++oNdURjytef2Oh1KMaYBLOkYYZtfHEOS6+ewhPr2nhxyz6vwzHGJJAlDXNGlrxvEueMzeErK9+iJxLzOhxjTIJY0jBnJCs9zL9+pJ5tHcd5ePU7XodjjEkQSxrmjF09rYyGc8bwo5e22xRcY1KEJQ1zVm69vIYdBzp57u2hHodijAkCSxrmrFxbX05lYRYPvrzd61CMMQlgScOclfRwiL+acw4vNu9jc/tRr8MxxowySxrmrC2aNYHMtJCdbRiTAixpmLNWnJvBjRdW8cvXWznU2eN1OMaYUWRJw4yIT10+ka7eGD9rbPU6FGPMKLKkYUbE9MoCzq8u5PH1bV6HYowZRZY0zIiZW1/BmzsP2Y0MjQkwSxpmxMytrwDg6aZ2jyMxxowWSxpmxEwpy2NKWR5PNe3xOhRjzCixpGFG1Nz6cla3HODgcZtFZUwQWdIwI2pefSXRmPL7jdZFZUwQWdIwI2pGVQFVRdnWRWVMQFnSMCNKRLi2vpwXtuzjWHfE63CMMSPMkoYZcfPqK+iJxHh+U4fXoRhjRpglDTPiGiYWU5ybwSob1zAmcCxpmBEXDglzJhWzuuWA16EYY0aYJQ0zKmbXjGXXoRPsPNDpdSjGmBFkScOMilk1xQCssbMNYwLFVdIQkXkisklEmkVk2QDbRUTudravE5GZQ5UVkWIReUZEtjivY/psu8PZf5OIzO2zfpGIrHc+43ciUnLmVTejaVp5PkU56axu2e91KMaYETRk0hCRMHAPMB+oAxaJSF2/3eYDtc7PEuBeF2WXAatUtRZY5SzjbF8I1APzgO+JSFhE0oBvA1er6vnAOuALZ1hvM8pCIeGSiTauYUzQuDnTmAU0q+o2Ve0BVgAL+u2zAHhI414FikSkcoiyC4DlzvvlwI191q9Q1W5VbQGaneOI85MrIgIUALuHXWOTMLNrinlnfyd7Dnd5HYoxZoS4SRpVwM4+y63OOjf7DFa2XFXbAJzXssGOpaq9wF8D64knizrghwMFLCJLRKRRRBo7OuxaAa/MrhkLYF1UxgSIm6QhA6xTl/u4Kevq80QknXjSuAgYR7x76o6BDqCq96tqg6o2lJaWDvFxZrTUjSsgPzPNuqiMCRA3SaMVGN9nuZr3dgudbp/ByrY7XVg4r3uHONaFAKq6VVUVeBS4zEX8xiPhkNAwcQyrt9mZhjFB4SZprAVqRaRGRDKID1Kv7LfPSuAWZxbVHOCw0+U0WNmVwGLn/WLgsT7rF4pIpojUEB9cXwPsAupE5OSpw4eAjcOsr0mwWTVj2dpxnI6j3V6HYowZAWlD7aCqERH5AvAUEAYeUNUmEbnN2X4f8CRwHfFB607g1sHKOoe+C3hURD4D7ABudso0icijwAYgAixV1SiwW0S+BrwgIr3AO8CnRuDfwIyi2ZPi12us3X6A686r9DgaY8zZknhPT3A1NDRoY2Oj12GkrN5ojAu+9jQ3X1zN1xbM8DocY4xLIvKaqjb0X29XhJtRlR4OcUF1EW/sPOR1KMaYEWBJw4y6GVUFbNxzlN5ozOtQjDFnyZKGGXUzqgrpicTY0n7M61CMMWfJkoYZdedVFQLw1q7DHkdijDlbljTMqJs4Npe8zDTe2m1Jwxi/s6RhRl0oJNSNK2C9nWkY43uWNExCnFdVyMa2I0RsMNwYX7OkYRJiRlUBXb0xtnYc9zoUY8xZsKRhEuLkYLh1URnjb5Y0TELUlOSRkxG2GVTG+JwlDZMQ4ZBQV1lgScMYn7OkYRJmRlUhTbuPEI0F+35nxgSZJQ2TMDOqCjnRG2Vbh10ZboxfWdIwCXPqynC7yM8Y37KkYRJmcmkuWekh1rce8ToUY8wZsqRhEiYtHGJ6ZYGdaRjjY5Y0TEKdW5HPlvajBP3hX8YElSUNk1BTy/M52NlLxzF7ZrgxfmRJwyTUtPJ8ADbvsRlUxviRJQ2TULUnk0b7UY8jMcacCUsaJqFK8jIozs2wpGGMT1nSMAklItSW5bHJkoYxvmRJwyTctIp8trQfsxlUxviQJQ2TcFPL8znWHWH34S6vQzHGDJMlDZNwU20w3BjfsqRhEm5qeR4Am/dY0jDGbyxpmIQrysmgvCDTBsON8SFLGsYTU8vjg+HGGH+xpGE8MbU8ny17j9oDmYzxGUsaxhPTyvPp6o2x80Cn16EYY4bBVdIQkXkisklEmkVk2QDbRUTudravE5GZQ5UVkWIReUZEtjivY/psu8PZf5OIzO2zPkNE7heRzSLytoh87MyrbrxUe3Iw3MY1jPGVIZOGiISBe4D5QB2wSETq+u02H6h1fpYA97oouwxYpaq1wCpnGWf7QqAemAd8zzkOwJeAvao61Tne82dQZ5ME7B5UxviTmzONWUCzqm5T1R5gBbCg3z4LgIc07lWgSEQqhyi7AFjuvF8O3Nhn/QpV7VbVFqDZOQ7Ap4F/B1DVmKruG151TbLIy0yjqiibzTYYboyvuEkaVcDOPsutzjo3+wxWtlxV2wCc17LBjiUiRc7yv4nI6yLyMxEpHyhgEVkiIo0i0tjR0eGiisYL0yry7UzDGJ9xkzRkgHX9p7ycbh83Zd1+XhpQDbykqjOBV4D/HOgAqnq/qjaoakNpaekQH2e8MqUsj237jtsMKmN8xE3SaAXG91muBna73Gewsu1OFxbO694hjrUf6AR+5az/GTAT41uTS3PpicRoPWgzqIzxCzdJYy1QKyI1IpJBfJB6Zb99VgK3OLOo5gCHnS6nwcquBBY77xcDj/VZv1BEMkWkhvjg+hqN3xL1N8BVzn7XABuGV12TTKaUxWdQbe2wcQ1j/CJtqB1UNSIiXwCeAsLAA6raJCK3OdvvA54EriM+aN0J3DpYWefQdwGPishngB3AzU6ZJhF5lHhCiABLVTXqlPln4Mci8i2g4+TnGH+aVOIkjb3H+cC5HgdjjHFFgv5Mg4aGBm1sbPQ6DHMaF//bM3xwejnfuOl8r0MxxvQhIq+pakP/9XZFuPHU5LI8654yxkcsaRhPTS61pGGMn1jSMJ6aUpbHwc5eDhzv8ToUY4wLljSMpyaX5gLQvNfONozxA0saxlOTS23arTF+YknDeKqqKJus9BBb7UzDGF+wpGE8FQoJk0ryaLYzDWN8wZKG8ZxNuzXGPyxpGM9NKc2j9eAJunqjQ+9sjPGUJQ3juclluajCto7jXodijBmCJQ3jOZtBZYx/WNIwnqspyUXEkoYxfmBJw3guKz3M+DE5doGfMT4w5K3RjUmEyaW5KZc0YjHlzic3su9Y91kdZ6BHXQKIyJ/vIyffCyLxxZA470UISXw5JPGp0GERwqH4T1o4RFpISA+HyEgLkREWstLDZGeEyU4Pk5uZRl5mGgVZ6RTlppOfmfZnn2+Cw5KGSQpTyvJ4eet+YjElFEqNPza7Dp3ghy+2UJKXQV7mmf0qnu7BBn2feKDoqeV3XxV1lmOqxJ+4G3+NxpRYTImqEo3FfyLDfCRvRjjE2LwMyguyqB6TzfjiHCaV5FI3roDasnwy0qyTw68saZikMLk0j+5IjF2HTjC+OMfrcBKiOxKfYvzVG+q54YJxHkczONV44ohElZ5IjO5olO7eGCd6o3T2ROnsjnCkK8LRrl4Odfay73g3+472sOfICdbvOszv3tpzKvGkh4W6ygKurC3lytoSZp4zhvSwJRG/sKRhksJk59GvzR3HUiZpdPXGAMj0wbduESE9LKSHITsjDKQPq3w0pryz/zgb2o7QtPsIa1sOcO/zW/nuc80U5aRz08xqFs2ecGomnUleljRMUphyctrt3mNcPa3M42gSozviJI30sMeRjL5wSJhUmsek0jyuPz9+VnX4RC+vbN3Hb95s48GXt/ODF1u4YkoJy+afy4yqQo8jNqdjScMkhTG5GRTnZqTUtNuT3VN+ONMYDYXZ6cybUcm8GZXsPdrFzxpb+eGLLdzw3RdZNGsC/3jtNIpzM7wO0/STmq3VJKXJpbls3Zs6V4WfPNOwQWEoy89i6dVTeO4fruJTl03kp2t38oFv/oEXt+zzOjTTj7VWkzRS7dGv3T4a00iUwpx0vnpDPb/74pWU52ex+Edr+PEr270Oy/RhrdUkjSlleew/3sPBFHn067vdU8Ef0xiu2vJ8fvH5y7h6WilffqyJf/n1eqLDnPZrRoclDZM0Uu0eVKcGwu1MY0B5mWl8/5MNfO59k/jvV3fw9Sc2eB2SwQbCTRLpmzQaJhZ7HM3oe3f2lCWN0wmHhDuum05vVHngpRbOKc7hU5fXeB1WSrOkYZJG1ZhsMtNCKXM7ke5e655y60sfns7Og53878c3UD0mhw/WlXsdUsqyrzgmaYRDQk1JLltT5LkaPVHrnnIrHBK+vfBCZlQVcvsjf2JL+1GvQ0pZ1lpNUkmlR7/a7KnhyclI4weLG8hKD7Hsl+uJ2cC4J6y1mqQypTSPnQc6U+LRr92RGBlpIbsb7DCU5WfxLx+u47V3DvLw6ne8DiclWdIwSWVyWR4xhe37g99F1R2J2lnGGfjozCqurC3hG7/bRNvhE16Hk3KsxZqkMrk0FyAlrgzvjsRsEPwMiAh33ngekViMrzzWhKp1UyWSJQ2TVCaV5CFCSsyg6u6N2ZnGGZowNoe//9BUntnQzrNv7/U6nJTiqsWKyDwR2SQizSKybIDtIiJ3O9vXicjMocqKSLGIPCMiW5zXMX223eHsv0lE5g7weStF5K3hV9cku+yMMFVF2SkxGN4dido1Gmfh05fXMKE4h2+v2mJnGwk0ZIsVkTBwDzAfqAMWiUhdv93mA7XOzxLgXhdllwGrVLUWWOUs42xfCNQD84DvOcc5Gc9HgeD/RUlhqXIPKuueOjtp4RCfv2oy61oP8/zmDq/DSRluvubMAppVdZuq9gArgAX99lkAPKRxrwJFIlI5RNkFwHLn/XLgxj7rV6hqt6q2AM3OcRCRPODvga8Pv6rGL6Y4026DPqUynjTsTONsfHRmNVVF2dxtZxsJ46bFVgE7+yy3Ouvc7DNY2XJVbQNwXk8+eWewMv8GfBPoHCxgEVkiIo0i0tjRYd9A/GZqeR5dvTFaDwZ7Zkx3r82eOlsZaSFue/8kXt9xiJe37vc6nJTgpsUONIm8f0o/3T5uyrr6PBG5EJiiqr8aojyqer+qNqhqQ2lp6VC7myRTW54PwKaAX/XbHYmlxFP7RtvNDeMpL8jk7lVbvA4lJbhJGq3A+D7L1cBul/sMVrbd6cLCeT05BeJ0ZS4FLhaR7cCLwFQR+YOL+I3P1DrPC9+cCknDzjTOWlZ6mM+9bzKrWw6wpuWA1+EEnpsWuxaoFZEaEckgPki9st8+K4FbnFlUc4DDTpfTYGVXAoud94uBx/qsXygimSJSQ3xwfY2q3quq41R1InAFsFlVrzqDOpskl5+VTlVRdgokDeueGimLZk2gICuNh+yBTaNuyLvcqmpERL4APAWEgQdUtUlEbnO23wc8CVxHfNC6E7h1sLLOoe8CHhWRzwA7gJudMk0i8iiwAYgAS1U1+PeUMH9mankem/YEPGn02uypkZKdEeajM6v5yeodHDjeY88WH0Wubo2uqk8STwx9193X570CS92WddbvB645TZk7gTsHiWc7MMNF6Manplbk81LzfiLRGGnhYH4bj49pBLNuXlg0awIPvrydX77eymevnOR1OIFlLdYkpall+fREY2zfP+hEOV+z7qmRNa0in5kTivjJmh02/XYUWYs1SWlaRXwGVZDHNezivpG3cNYEtnUcZ+32g16HEliWNExSmlIWvwdVUJOGqtJjs6dG3PXnV5KfmcaKNTu8DiWwrMWapJSVHuac4pzAJg17PvjoyMlIY8FF43hifRuHO3u9DieQrMWapDW1PJ/N7cG8B9WppGHdUyNu4SUT6I7E+PUbu7wOJZAsaZikNbU8n5Z9x+mOBG/G9ck6WffUyJtRVci5Ffn85s3+1yCbkWAt1iStqRX5RGNKy77gPZDJng8+uq4/v5LGdw6y+1Cw71/mBWuxJmlNO3kPqgBe5PfumIZ1T42G688fB8AT69o8jiR4LGmYpFVTkktaSAI5GG7dU6NrYkkuM6oKeHyddVGNNGuxJmllpIWoKckN5GD4yTONDEsao+aG88fxZuthdgT4AlEvWIs1SW1qRX4wzzRsTGPUffj8SgB+Y2cbI8parElqU8vy2XGgk86eiNehjKh3u6dsTGO0VI/J4aIJRTxu4xojypKGSWr14wpQhY1tR7wOZUS9e52G/QqOpuvPH8fGtiMp8cz5RLEWa5LajKpCANa3HvY4kpF1Mmlk2RXho+rD51UiAo+/aWcbI8VarElq5QWZlORl8tbugJ1p9Fr3VCJUFGZx8YQxPNW0x+tQAsOShklqIsJ5VQW8tSuYZxrWPTX6rq0vZ0PbEXYesFlUI8FarEl6M6oK2bL3GF29wbmdSI/deyphrq2rAODpDe0eRxIMljRM0ptRVUg0poEaDLe73CbOxJJcppXnWxfVCLEWa5LeycHwIHVRnZxymxHQR9kmm7n15TRuP8D+Y91eh+J71mJN0htXmEVxbgbrA5U0YmSEQ4RC4nUoKeHa+gpiCqs27vU6FN+zpGGSnogwo6qQ9bsC1D3Va0/tS6T6cQVUFWXz9Abrojpb1mqNL8wYV8CW9qOBGQzvjkRtPCOBRIRr68t5Ycs+jncH6+4CiWat1vjCeVWFRGIamNukd0diNnMqwa6tq6AnEuP5zR1eh+JrljSML5y6Mjwg4xrxpGG/fol0ycQxjMlJt1lUZ8larfGF6jHZFGan07Q7IEmjN2q3RU+wtHCID04v59m39566TsYMn7Va4wvxK8MLg3WmYU/tS7hr6ys42hXh1W37vQ7FtyxpGN+YUVXIpj1HT13j4Gfdkah1T3ngytoScjLCNovqLFirNb5xQXUhvVGlKQA3L7QxDW9kpYd5/9RSnm5qJxZTr8PxJWu1xjcuqSkGYPW2Ax5Hcvbi12lY95QX5tZXsPdoN2+0HvI6FF+ypGF8oyQvk8mluaxp8X9/tF2n4Z2rp5WRFhKbRXWGXLVaEZknIptEpFlElg2wXUTkbmf7OhGZOVRZESkWkWdEZIvzOqbPtjuc/TeJyFxnXY6IPCEib4tIk4jcdXZVN340e9JYGrcfJOrzrgXrnvJOYU46l04ey9NN7aj6ux15YchWKyJh4B5gPlAHLBKRun67zQdqnZ8lwL0uyi4DVqlqLbDKWcbZvhCoB+YB33OOA/CfqnoucBFwuYjMP5NKG/+aXVPM0e4IG3w+rmEX93nr2voKWvYdp3mvPQZ2uNx81ZkFNKvqNlXtAVYAC/rtswB4SONeBYpEpHKIsguA5c775cCNfdavUNVuVW0BmoFZqtqpqs8BOMd6HagefpWNn82uGQvAap93UXX32uwpL11bVw5gXVRnwE2rrQJ29lludda52WewsuWq2gbgvJa5/TwRKQJuIH6G8h4iskREGkWksaPDbhkQJBWFWZwzNofVLf4eDI9fp2FJwyvlBVlcNKGIJ9db0hguN612oHs39+8IPN0+bsoO6/NEJA14BLhbVbcNdABVvV9VG1S1obS0dIiPM34zu6aYtdsP+HbKpKpa91QS+PB5lWxoO8K2DuuiGg43SaMVGN9nuRrY7XKfwcq2O11YOK8nb3Q/1OfdD2xR1W+5iN0E0OyasRzq7GXzXn/evLAnas8HTwYfPr8SgMfXtXkcib+4abVrgVoRqRGRDOKD1Cv77bMSuMWZRTUHOOx0OQ1WdiWw2Hm/GHisz/qFIpIpIjXEB9fXAIjI14FC4G+HX1UTFLN8fr3GqUe9WtLwVGVhNrMmFvP4uv7fgc1ghmy1qhoBvgA8BWwEHlXVJhG5TURuc3Z7EthGfND6v4DPD1bWKXMX8CER2QJ8yFnG2f4osAH4HbBUVaMiUg18ifgsrNdF5A0R+ezZ/gMY/xlfnENVUbZvB8O7e08+H9y6p7x2/QWVbG4/Fphb7idCmpudVPVJ4omh77r7+rxXYKnbss76/cA1pylzJ3Bnv3WtDDzeYVLQ7JpiXtjSgaoi4q9mcfLeWXam4b35Myr515VNPL5uN9Mqpnkdji9YqzW+NKummH3Henw5z966p5JHaX4mcyaN5fF1bXahn0vWao0vvX9afFbc0xvaPY5k+E51T1nSSArXnz+Oln3HA3EjzESwVmt8qbIwmwvGF/G0Dy/Oerd7ysY0ksG8GRWEQ2KzqFyypGF8a159BW+2HmbXoRNehzIs1j2VXIpzM7iytoTH3tjl+3uaJYK1WuNbc+vjt4Lw29nGqaRhV4QnjU80jKftcBfPb9479M4pzlqt8a1JpXlMLc/z3f2DunuteyrZXDO9nJK8DB5Zs3PonVOcJQ3ja/PqK1jTcoD9x7q9DsU1655KPhlpIW66eDzPvr2X9iNdXoeT1KzVGl+7tr6CmMLvN/pnFtW7ScPONJLJwkvGE40pP2u0s43BWNIwvlY/roDqMdk81eSfpNFjYxpJaWJJLpdNHstPG3f69maYiWCt1viaiDC3voIXt+zjaFev1+G4YleEJ6+Fsyaw88AJXtq6z+tQkpa1WuN782dU0BON8eR6f8yzt+6p5DW3vpwxOen8ZPUOr0NJWpY0jO9dfM4YppXn86OXtvviVhAnrwjPsDONpJOZFubjl4znqaY9tOw77nU4SclarfE9EeHWyyfy9p6jvniiX3ckSnpYCIf8daPFVPHZKyaRHg5xz3PNXoeSlCxpmEBYcGEVRTnp/OilFq9DGZI9tS+5leZn8pezJ/CrP+1i54FOr8NJOpY0TCBkZ4RZNGsCz2xoT/pf9O5I1AbBk9xt759MOCR87w9bvQ4l6VjLNYHxyTnnICL8+NV3vA5lUN29MUsaSa68IItPNIzn56/t9N29zUabtVwTGOOKsplXX8GKNTvo7Il4Hc5pdUdi9tQ+H7jtqskAfP95O9voy5KGCZRbL5/Ika4Iy19O3rMN657yh6qibG66eDyPrNnB5nZ7HOxJ1nJNoDRMLOaD08v5zrNbaDucnN0K8YFw+9Xzg3+aO438rHSW/WKdXSXusJZrAuerN9QRjSlff2Kj16EMKD6mYd1TflCcm8GXr5/O6zsO8d+rk/fsNZEsaZjAGV+cw9Krp/DEujZe3JJ8t4PojkTtvlM+cuOFVbxvainf+O3b7LZBcUsaJpiWvG8S54zN4Ssr3zp1g8BkYd1T/iIi3HnjDGIK//Lrt3xx14HRZC3XBFJWeph/vaGebR3H+Y/fve11OH/GLu7zn/HFOfzT3Gk8+/Ze/t8zm70Ox1OWNExgXX1uGYsvPYcfvNjCw0nUH22zp/zp1ssn8vGGau5+tjmln7mR5nUAxoymL19fxzsHOvnKY01Uj8nh/VNLvQ4pPhBuYxq+IyLc+RfnsevQCe745XqqirK5bEqJ12ElnLVcE2hp4RDf/cuZ1JblsfTh13lr12GvQ7LuKR9LD4f43v+4mJqSXD7349eScqLFaLOkYQIvLzONH916CflZadx83yueP3fDuqf8rTA7neWfnsW4omwW/2gNP35lu9chJZS1XJMSKguzeWzp5UyvzOfzD7/O/3nqbU8u1lJVmz0VAOOKsvnF5y/j6mmlfPmxJr70q/Wc6Il6HVZCWMs1KaOsIItHlsxh4SXjuee5rdx038u89s7BhMbQG1VUsXtPBUBeZhrf/2QDn3v/JB5evYNrvvkHfvPm7sBPybWkYVJKZlqYf//oeXzz5gtoPXiCj937Mp9/+LWE3VvIng8eLOGQcMf86Tz6uUspysng9kf+xCe+/yq/39BONKC3HbHZUybliAgfu7ia+edV8F8vtPD9F7by5Po9XFBdyE0N47luRgVj8zJH5bNPPh/cHvUaLLNqivnN7Vfw07U7+dbvN/PZhxoZV5jFJy6ZwLX15ZxbkY9IMJ7UKG5OpURkHvBtIAz8QFXv6rddnO3XAZ3Ap1T19cHKikgx8FNgIrAd+LiqHnS23QF8BogCf6OqTznrLwYeBLKBJ4Ev6hAVaGho0MbGxiHraFLX/mPd/OpPu/j5a628vSd+xnFuRT6XTh7LxeeM4dyKfM4Zm0t6+Oz/0O86dILL73qWb3zsPD5xyYSzPp5JPr3RGKs2tvPw6h380ZldVZafyZW1pVw4oYi6ygKmV+aTk5Hc39lF5DVVbXjP+qGShoiEgc3Ah4BWYC2wSFU39NnnOuB24kljNvBtVZ09WFkR+Q/ggKreJSLLgDGq+s8iUgc8AswCxgG/B6aqalRE1gBfBF4lnjTuVtXfDha/JQ3jlqrStPsIz2/u4JWt+1m7/cC7ZwbhEBPG5lBVlM24omwqCrIozk2nKCeDMTkZ5GaGyc9KIycjjaz0MNnpYTLTQoT6PQd8W8cxPvDN5/nWJy7kxouqvKimSaA9h7t4YUsHL2zu4KXmfRzs7AVABCoKsqgek834MTmUFWRRkpfB2LwMCrPTyc9KJz8rjdyT7Skj3p7SQpKwM5bTJQ03qW4W0Kyq25wDrQAWABv67LMAeMj51v+qiBSJSCXxs4jTlV0AXOWUXw78AfhnZ/0KVe0GWkSkGZglItuBAlV9xTnWQ8CNwKBJwxi3RIQZVYXMqCpk6dVT6I5E2dJ+jM3tR9nUfpSWjuO0He6iafdh9h3rcXXMtJCQHg6RHhbSwqFTg6Q2ppEaKgqz+HjDeD7eMB5VZffhLjbsPsKG3UfYcaCTnQc7eXXbfjqOddMbddPrE/8Ckx4OkRYW0kJCOCSERQg570MiiEBIhMdvv4KsEZ504SZpVAF9r5lvJX42MdQ+VUOULVfVNgBVbRORsj7HenWAY/U67/uvfw8RWQIsAZgwwboAzJnJTAufSiL99UZjHOrs5WBnDweP93C8J8Kx7ijHuyN09Ubp6o3R1RslEovRG1V6IjGiMSUSU9LDwpxJYz2okfGSiFBVlE1VUTYfqiv/s22qytHuCPuOdnP4RC/HuiMc7Yqcak+dPVF6IjF6ovGf3ogSjcXojSnRqBJVJRaLv6pCzHkNjcJZiZukMdCn9k+Jp9vHTVm3n+f6WKp6P3A/xLunhvg8Y4YtPRyiND+T0vzRGTA3qUVEKMhKpyAr3etQhuTmHLkVGN9nuRrY7XKfwcq2O11YOK97XRyreog4jDHGjCI3SWMtUCsiNSKSASwEVvbbZyVwi8TNAQ47XU+DlV0JLHbeLwYe67N+oYhkikgNUAuscY53VETmOLO1bulTxhhjTAIM2T2lqhER+QLwFPFpsw+oapOI3OZsv4/4TKbrgGbiU25vHaysc+i7gEdF5DPADuBmp0yTiDxKfLA8AixV1ZPX5/817065/S02CG6MMQnl6joNP7Mpt8YYM3ynm3Jr8/6MMca4ZknDGGOMa5Y0jDHGuGZJwxhjjGuBHwgXkQ7gnTMsXgKk2vMcU7HOkJr1TsU6Q2rW+0zqfI6qlvZfGfikcTZEpHGg2QNBlop1htSsdyrWGVKz3iNZZ+ueMsYY45olDWOMMa5Z0hjc/V4H4IFUrDOkZr1Tsc6QmvUesTrbmIYxxhjX7EzDGGOMa5Y0jDHGuGZJYwAiMk9ENolIs/P88kASkfEi8pyIbBSRJhH5orO+WESeEZEtzusYr2MdaSISFpE/icjjznIq1LlIRH4uIm87/+eXBr3eIvJ3Ttt+S0QeEZGsINZZRB4Qkb0i8lafdaetp4jc4fx92yQic4fzWZY0+hGRMHAPMB+oAxaJSJ23UY2aCPAPqjodmAMsdeq6DFilqrXAKmc5aL4IbOyznAp1/jbwO1U9F7iAeP0DW28RqQL+BmhQ1RnEH8+wkGDW+UFgXr91A9bT+R1fCNQ7Zb7n/N1zxZLGe80CmlV1m6r2ACuABR7HNCpUtU1VX3feHyX+R6SKeH2XO7stB270JMBRIiLVwIeBH/RZHfQ6FwDvA34IoKo9qnqIgNeb+DODskUkDcgh/rTPwNVZVV8ADvRbfbp6LgBWqGq3qrYQfw7SLLefZUnjvaqAnX2WW511gSYiE4GLgNVAufOkRJzXMg9DGw3fAv4nEOuzLuh1ngR0AD9yuuV+ICK5BLjeqroL+E/iD3lrI/5E0acJcJ37OV09z+pvnCWN95IB1gV6XrKI5AG/AP5WVY94Hc9oEpHrgb2q+prXsSRYGjATuFdVLwKOE4xumdNy+vAXADXAOCBXRP7K26iSwln9jbOk8V6twPg+y9XET2kDSUTSiSeMh1X1l87qdhGpdLZXAnu9im8UXA58RES2E+96/ICI/DfBrjPE23Wrqq52ln9OPIkEud4fBFpUtUNVe4FfApcR7Dr3dbp6ntXfOEsa77UWqBWRGhHJID5gtNLjmEaFiAjxPu6Nqvp/+2xaCSx23i8GHkt0bKNFVe9Q1WpVnUj8//ZZVf0rAlxnAFXdA+wUkWnOqmuADQS73juAOSKS47T1a4iP2wW5zn2drp4rgYUikikiNUAtsMbtQe2K8AGIyHXE+73DwAOqeqe3EY0OEbkC+COwnnf79/8X8XGNR4EJxH/xblbV/oNsviciVwH/qKrXi8hYAl5nEbmQ+OB/BrANuJX4F8fA1ltEvgZ8gvhMwT8BnwXyCFidReQR4Crit0BvB74K/JrT1FNEvgR8mvi/y9+q6m9df5YlDWOMMW5Z95QxxhjXLGkYY4xxzZKGMcYY1yxpGGOMcc2ShjHGGNcsaRhjjHHNkoYxxhjX/j88aV2bspE6TwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, decay_rate=0.5)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*2)\n", "\n", "plt.plot([i for i in range(num_epoch*2)], lr_per_epoch);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `warmup_t` " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Defines the number of warmup epochs. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `warmup_lr_init` " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The initial learning rate during warmup. " ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0LUlEQVR4nO3deXxU9b3/8ddnJjOTnSwkAdnRALJVILIKolZFi0BdKtoq17aXWvX+XHtLl9uqt/rz/oq96K3VeqtVbxcEe23RWq1sCgIiWGRNAiJCgMlGQiZ7Zub7+2MOMUBIJuuZTD7PR/OYOcv3nM8BmzfnfM/5HjHGoJRSSoXDYXcBSimleg4NDaWUUmHT0FBKKRU2DQ2llFJh09BQSikVthi7C+hqffv2NUOHDrW7DKWU6lG2b99eYozJOHN+1IfG0KFD2bZtm91lKKVUjyIinzc3Xy9PKaWUCpuGhlJKqbBpaCillApb1PdpKKU6R0NDAwUFBdTW1tpdiupEsbGxDBw4EJfLFdb6GhpKqbAUFBSQlJTE0KFDERG7y1GdwBhDaWkpBQUFDBs2LKw2YV2eEpE5IpInIgdEZEkzy0VEnraW7xSRia21FZE0EXlXRPZbn6nW/HQRWScilSLyyzP2M0lEdlnbelr0v1yluk1tbS3p6ekaGFFEREhPT2/T2WOroSEiTuAZ4BpgNHCLiIw+Y7VrgGzrZzHwbBhtlwBrjDHZwBprGqAW+DfgoWbKedba/ql9zQnrKJVSnUIDI/q09e80nMtTk4EDxpiD1g6WA/OBvU3WmQ+8YkLjrG8RkRQR6Q8MbaHtfGC21f5lYD3wfWNMFbBRRC4448D6A8nGmM3W9CvAAuBvbTricO3+E9SUn3t5p/2f5+ztHCyp4mBxZZhNvsj9s/7yrWmxGpgzlos4mqwDNE6L9T9pnBea7UCsn9A8B+KwfsSBiBNxOkPfnTE4HILDEYPDGfpxxsTgcLpwOGNwud3ExLiJcXtwuTy43G7cbg8eTywOpzO8PzqlVLcLJzQGAEeaTBcAU8JYZ0ArbbOMMccBjDHHRSQzjDoKmtnHWURkMaEzEgYPHtzKZs9h7yoob/bZFujid5D4iyrJbAh06T66mwH8TaZbOhn2Swx+ceMXD36Hm4DDQ9DpJhCTgHHHgysBhycRhycBZ2wSnqR0EpLTSUhNp09qFimp6cSE2amneo7777+fIUOGcN999wFw9dVXM2jQIH7zm98A8OCDDzJgwAAuuOAC9u7dy5IlS/jzn//MiBEjGD06dIFj9uzZLF26lJycHLsOo8cLJzSa+yf1mb81z7VOOG3DFfa2jDHPA88D5OTktG9/X3u5Xc06qs4f4NqfvMN3Lh3O964eddZyEwx+8b1JeJkzlhtrjglan1jzG1c0GGMwJhj6bNI2GAyCCW3DGIMJBjDGEDTBULtgkGDQEDQBTCBIIOjHBA3BoJ9gMEAwEMQEAgRNkGCggYDfTzBw6ufUdAMBfwNBf33oJ9CA8TcQbKjF+GsxDXXgr8P465BALeKvxdVQQUyNl5hgDZ5gDTGm4bQ/myrr5yhCrSOBancaDXFZkNQPd+oAEvsOJK3fEM4bfD6xcfEd+4tS3W769OmsXLmS++67j2AwSElJCRUVFY3LN23axLJly5gyZQrz5s0D4M9//jNz585tDI3uFAgEcEbhWXM4oVEADGoyPRA4FuY67hbaFopIf+ssoz9QFEYdA1upo8c7WFyFP2gY2S+52eXiaHI5qrkVovA/0nOpr6uj0ldORXkJFWUlVJ8soc5XQn1lGcGqUhxVRXiqCkgq30HM4VDAVAJ5COXu/tQmD8OVNYLUQRcy6ILx9M3sZ+8BqRbNmDGD+++/H4A9e/YwduxYjh8/TllZGfHx8ezbt48JEybw0ksvsW3bNm699VZWrVrFe++9x89+9jP+9Kc/AbBy5UruuusuysvLeeGFF5g5c+Zp+7nrrruYM2cO8+bN46tf/Sqpqam8+OKLvPDCC3z22Wf87Gc/Y8GCBRw5coTa2lruvfdeFi9eDEBiYiIPPPAA77zzDk8++SRz5szh7rvvZvXq1aSmpvL444/zr//6rxw+fJhly5Yxb968xnp/+cvQfT9z587loYceYvbs2SQmJvKd73yHdevWkZqayvLly8nIOGs4qG4VTmh8BGSLyDDgKLAQuPWMdVYB91h9FlOAk1YYFLfQdhWwCHjC+vxLS0VY2/OJyFTgQ+B24L/CqL9HyfP6ABjVL8nmSiKf2+MhzZNFWt+sFtczwSClJUUUHz9EeeERqrz7CZYcIKl8L4klH8Ce0L8+8mLSqOp7EcnZ0xg56TJS0/p2z4H0QI+8sYe9xypaX7ENRp+XzE+vG3PO5eeddx4xMTEcPnyYTZs2MW3aNI4ePcrmzZvp06cP48ePx+12N64/ffp05s2bx9y5c7nxxhsb5/v9frZu3cpbb73FI488wurVq0/bz6xZs9iwYQPz5s3j6NGjHD9+HICNGzeycOFCAF588UXS0tKoqanh4osv5oYbbiA9PZ2qqirGjh3Lo48+CkBVVRWzZ8/mP/7jP/jqV7/Kj3/8Y95991327t3LokWLGs+IzqWqqoqJEyfy5JNP8uijj/LII480hotdWg0NY4xfRO4B3gGcwIvGmD0icqe1/DngLeBa4ABQDdzRUltr008AK0TkW8Bh4KZT+xSRQ0Ay4BaRBcBVxpi9wHeBl4A4Qh3gXdMJbqNcrw+XUxjWN8HuUqKGOBz0zexnnUlMPW1ZeVkpn+//hBOf76Ph6CekFm3G413L4Q2PsyNuKIHzcug/7lJGXzT9tLM8ZY8ZM2awadMmNm3axAMPPMDRo0fZtGkTffr0Yfr06WFt4/rrrwdg0qRJHDp06KzlM2fOZNmyZezdu5fRo0dTVlbG8ePH2bx5M08//TQATz/9NK+//joAR44cYf/+/aSnp+N0Ornhhhsat+V2u5kzJ3ST57hx4/B4PLhcLsaNG9fsvs/kcDi4+eabAfjGN77RWLudwnq4zxjzFqFgaDrvuSbfDXB3uG2t+aXAFedoM/Qc87cBY8OpuafKL/RxfkYiLqf+guoOKanppEy+HCZfDoC/oYH83R9xbM/7SME2Mj79E4FPV7Lxr1n4R87loisWkppu7+WBSNDSGUFXmj59Ops2bWLXrl2MHTuWQYMG8eSTT5KcnMw3v/nNsLbh8XgAcDqd+P3+s5YPGDCAsrIy3n77bWbNmsWJEydYsWIFiYmJJCUlsX79elavXs3mzZuJj49n9uzZjc85xMbGntaP4XK5Gu9qdDgcjft2OByN+46JiQn1I1paemYiEm551t9MESbP62OkXpqyTYzLxegJ0/nyN5ZwxZLXGPrAanxTH6LelUz67hc49NQ1/P3pu9i5df1pNyWo7jFjxgzefPNN0tLScDqdpKWlUV5ezubNm5k2bdpZ6yclJeHz+dq8n2nTprFs2TJmzZrFzJkzWbp0aWPfx8mTJ0lNTSU+Pp7c3Fy2bNnSoWMaOnQoO3bsIBgMcuTIEbZu3dq4LBgM8tprrwHwhz/8gUsuuaRD++oMGhoRpKK2gaPlNYzI0tCIFH1S0phx7Te4YslKEr/+PxQPupqUEzvgzfvY8Pi1bPn7Sg2PbjRu3DhKSkqYOnXqafP69OlD375n90EtXLiQn//850yYMIFPP/007P3MnDkTv9/PBRdcwMSJEzlx4kRjaMyZMwe/38/48eP5t3/7t9NqaY8ZM2YwbNgwxo0bx0MPPcTEiY0DapCQkMCePXuYNGkSa9eu5Sc/+UmH9tUZxHTxMwd2y8nJMT3lJUzbPz/BDc9u5oVFOVxxYcudu8o+tTXVbF/3OvX/eJX0usMUJ4xg0NzvM2LMJLtL61L79u3jwgsvtLuMXiUxMZHKyhYe9O0kzf3dish2Y8xZD7TomUYEybXunNLLU5EtNi6eGdd+nVlLXudkzr0k1Byn5tVv8/dnH6SkyGt3eUp1KQ2NCJLn9ZHoiWFASpzdpagwOJ1OZs67g9EPrKJwyHVkHl/HoWcWsO7Vp6ivq7O7PBUFuuMso600NCJIntfHiKzEiLhDQoUvOTmVq771KKmL/kBZ8oWk7/ktG/7zVoqLou7ZU6U0NCKFMYa8Qt85nwRXkW/I+aO48sHfUj3zR6TUHGH/c18nf892u8tSqlNpaESIIl8d5dUNjMxKtLsU1UFTr7yJ5Jt/jUGoXHEnW/6+0u6SlOo0GhoRIq+xE1zPNKJB9ugJjPruHyiLG0r8xsdY/cpjBAPRNXKx6p00NCJEnt45FXXSM/pxyf2v4O13GZkHVrL66Tupquzc8Zp6k/vvv59ly5Y1Tl999dV8+9vfbpx+8MEH+cUvfsGqVat44okngNAot3v3fvHqn9mzZ9NZt+A//vjjnbKdnkZDI0Lken1kJHlIS3C3vrLqMTyeOK6880lOjF9MVtk2PvyvO6iprrK7rB7p1BAiQOPQ6Hv27GlcvmnTJmbMmMG8efNYsiT0ItAzQ6MzdXVoNDfESSTQ0IgQeYUVOrJtlBKHg9k33kXd7J+QXnOQDb++l4BeqmqzU4MVwhdDoyclJVFWVkZdXd1pQ6Pfc889bNq0iVWrVvG9732Piy66qPGJ8JUrVzJ58mRGjBjBhg0bgNB4T3fccQfjxo1jwoQJrFu3DqBxW6fMnTuX9evXs2TJEmpqarjooov4+te/flqdK1as4IEHHgDgqaeeYvjw4QB8+umnjcOAPProo1x88cWMHTuWxYsXN74bZ/bs2fzwhz/k0ksv5amnnmL27Nncf//9zJo1iwsvvJCPPvqI66+/nuzsbH784x8DcOjQIcaO/WJIvqVLl/Lwww83bu++++5j+vTpjB079rQhStorrAELVdcKBA37Cyu5beoQu0tRXWjy5V9lfZmXfp/8mjW//QlXffsxu0tqv03/BSX7O3ebfbNh+r+cc3FXDo3+zDPPALBr1y5yc3O56qqryM/PP2ctTzzxBL/85S/ZsWPHWctmzZrFz3/+cwA2bNhAeno6R48eZePGjY1Dkdxzzz2NQ4LcdtttvPnmm1x33XUAlJeX89577wHwxhtv4Ha7ef/993nqqaeYP38+27dvJy0tjfPPP7/x/SItqaqqYtOmTbz//vt885vfZPfu3a22aYmeaUSAz0urqPMHGaFnGlFv9g3fxTv4Ovod/ivrXnvG7nJ6nKZDo0+bNo1p06Y1TndkaPSNGzdy2223ATBq1CiGDBnSYmi0pF+/flRWVuLz+Thy5Ai33nor77//Phs2bGgMjXXr1jFlyhTGjRvH2rVrT7vMdmoo9FNOvXNj3LhxjBkzhv79++PxeBg+fDhHjhyhNbfccgsQCrOKigrKy8vbdVyn6JlGBNAXL/UuX77jYVb/VyFZO3/D1rTzmHz5V+0uqe1aOCPo0t120dDo5xqDry3Dljc1bdo0fvvb3zJy5EhmzpzJiy++yObNm3nyySepra3lrrvuYtu2bQwaNIiHH374tO0mJJz+Lp2mw6mf+n5q2u/3t1rjmQ8Ld/ThYT3TiAB5hT5EIDtTQ6M3cDidzPrOMkrihxPz3mPs3bHZ7pJ6jK4aGn3WrFn8/ve/ByA/P5/Dhw8zcuTIFoctd7lcNDQ0nHN7S5cuZdasWY19JB6Phz59+jT+Uu/bty+VlZWNQ5+3V1ZWFkVFRZSWllJXV8ebb7552vJXX30VCJ1N9enThz59+nRofxoaESDP62NIWjxx7t7zfu/eLjYungnf/hVVMan4/vw9Cj7v5P6BKNVVQ6PfddddBAIBxo0bx80338xLL72Ex+NpcdjyxYsXM378+LM6wiE0tPqRI0eYNWsWTqeTQYMGNXaCp6Sk8M///M+MGzeOBQsWcPHFF3fkjwSXy8VPfvITpkyZwty5cxk1atRpy1NTU5k+fTp33nknL7zwQof2BTo0ekS4fOl6srMS+fVtZ41CrKLc5wfzKHzlDmpjkrn4gT8RFx+5r/nVodF7ntmzZ7N06VJyclr+3aJDo/cgtQ0BDpVW6ZPgvdSQ4SPxzPl3Uuq9bHz153aXo1SrNDRsdqCokqCBkfq2vl7rS1OvwNv/CjI/W6UDHKpOtX79+lbPMtpKQ8Nm+uIlBTDllh9R7Uzi2J8foaGh3u5yzinaL2f3Rm39O9XQsFmetwJ3jIOh6fF2l6Js1CclDefM+0mrO8z7r/3S7nKaFRsbS2lpqQZHFDHGUFpaSmxsbNht9DkNm+UVVnJBRiIxTs3v3m7y5Qv4+8636Jv7Bz4/+BWGDB9pd0mnGThwIAUFBRQXF9tdiupEsbGxDBw4MOz1NTRsluetYMb5Z98qqHqnCbf8lM+evYn9K3/KoId+j8MZObdhu1wuhg0bZncZymb6z1sblVfXU1hRp/0ZqlFG1gDqJi0msyqXjW++bHc5Sp1FQ8NGp4YP0TGnVFOXfOV2ipJGE//x8xQdb31sIaW6k4aGjfIKdcwpdTZxOBj1tUdwEGDH8ocxTcYVUspuGho2yvX6SI6NoV9y+HcuqN5h4JBsykd/g35l2/nkwzV2l6NUIw0NG+V7fYzsl9ThUSdVdJqxYDGVMamUvve8nm2oiKGhYRNjDHmFPu0EV+fk8cTRMO4WMqr3s3PrOrvLUQrQ0LDNsZO1+Gr9OuaUatHUa2+j0plK8Xv/bXcpSgFhhoaIzBGRPBE5ICJLmlkuIvK0tXyniExsra2IpInIuyKy3/pMbbLsB9b6eSJydZP5t4jILmsfb4tIj33AIf/U8CE65pRqgccTR/3Ym8msymXn1vV2l6NU66EhIk7gGeAaYDRwi4iMPmO1a4Bs62cx8GwYbZcAa4wx2cAaaxpr+UJgDDAH+JWIOEUkBngKuMwYMx7YCXzxxvceJldDQ4Vp6lcWUeXsQ9H6X9tdilJhnWlMBg4YYw4aY+qB5cD8M9aZD7xiQrYAKSLSv5W284FTTy+9DCxoMn+5MabOGPMZcMDajlg/CRLqOU4GjrX5iCNEnreC/n1i6RPvsrsUFeFiY+OoHfM1Miv3sXvbBrvLUb1cOKExAGj6hFGBNS+cdVpqm2WMOQ5gfWa2tC1jTAPwXWAXobAYDTT7GioRWSwi20RkW6SOk5NXWMkIPctQYZr2lTuodiZzfN1zdpeierlwQqO5+0HPHObyXOuE0zas/YmIi1BoTADOI3R56gfNbcAY87wxJscYk5ORkdHK7rpfQyDIp0WV+lCfCltsXDw1o79Glm8Pez7eaHc5qhcLJzQKgEFNpgdy9mWhc63TUttC6xIW1mdRK9u6CMAY86kJjc28ApgeRv0R51BJFfWBoN5uq9pkyrWLqHYkcWyt9m0o+4QTGh8B2SIyTETchDqpV52xzirgdusuqqnASeuSU0ttVwGLrO+LgL80mb9QRDwiMoxQ5/pW4CgwWkROnTpcCexr4/FGhFPDh+jlKdUW8QlJVF94I1kVu9i7Y7Pd5aheqtXQMMb4Cd2l9A6hX9IrjDF7ROROEbnTWu0t4CChTuv/Bu5qqa3V5gngShHZTygAnrDa7CF0FrEXeBu42xgTMMYcAx4B3heRnYTOPB7v2OHbI8/rw+kQLshMtLsU1cNM+cod1DgSObpG+zaUPSTa38KVk5Njtm3bZncZp/nnV7ZxsLiSNQ/OtrsU1QOt+eMvyNj3Cu4bn2fU+Ml2l6OilIhsN8ac9YJxfSLcBvk6fIjqgCnXfZt6ieXz939vdymqF9LQ6GbV9X4On6hmZJYOH6LaJzExmdL+s8go/oCTZSfsLkf1Mhoa3Sy/sBJj0DMN1SHDZ91CjPHzj7Ur7C5F9TIaGt2sccwpDQ3VAdmjJ1AaNwzJfUOHTVfdSkOjm+V6fcS6HAxOi7e7FNXDxYyZR2rdUfZ9ssXuUlQvoqHRzfIKKxiRlYTToS9eUh0z8fIbqJdYjnzwqt2lqF5EQ6Ob5Xl1zCnVORK0Q1zZQEOjG5VW1lFSWadjTqlOM2zmQu0QV91KQ6Mb5WknuOpkI8ZMpEQ7xFU30tDoRqfGnNIXL6nO5NIOcdWNNDS6UZ7XR2q8i4wkj92lqCgS6hD3aIe46hYaGt0o1xsaPiT04kGlOod2iKvupKHRTYJBw/5Cn16aUl1i2EzrCfF1K+0uRUU5DY1ucrS8hqr6ACP76ZhTqvM1dojvW6Ud4qpLaWh0k1y9c0p1Me0QV91BQ6Ob5De+rU9fvKS6xqkO8YLNr9ldiopiGhrdJNfrY0BKHEmxLrtLUVEqITGZExmTSSnaQl1djd3lqCilodFN8rwV+iS46nJ9v3QtscFqdm1ZY3cpKkppaHSDen+Qg8VVjNDQUF1s3NQrqHXEU7rzb3aXoqKUhkY3OFhSiT9o9ExDdTmXy01Zvxn0LdlGTXWV3eWoKKSh0Q10zCnVnfpNvAaXqeOTTW/bXYqKQhoa3SDP6yPGIQzvq3dOqa43dtKlVDmTqdj9jt2lqCikodEN8rw+hmck4I7RP27V9ZxOJ77zZtK37GN8FeV2l6OijP4W6wahMaf0SXDVfQbmfIUY42fnB2/ZXYqKMhoaXcxX28DR8hpG6kN9qhtd+KWpVMSkU71XL1GpzqWh0cXyCysB9ExDdStxOKgZNJOMk7soKy22uxwVRTQ0utipO6f0dlvV3YZMnoeDILs2vmF3KSqKaGh0sfxCH/FuJwNS4uwuRfUy2RdexElXFvV579pdiooiGhpdLNdbwYisJBwOffGS6l7icFA79DIyKnMpLjxqdzkqSmhodCFjDHlen16aUrbJnnYdgmHPxlV2l6KiRFihISJzRCRPRA6IyJJmlouIPG0t3ykiE1trKyJpIvKuiOy3PlObLPuBtX6eiFzdZL5bRJ4XkXwRyRWRG9p/6F2vuLKOsuoGRujb+pRNhl4whlLPYAL7dQBD1TlaDQ0RcQLPANcAo4FbRGT0GatdA2RbP4uBZ8NouwRYY4zJBtZY01jLFwJjgDnAr6ztAPwIKDLGjLC29147jrnbaCe4igTB4bPJqD7A0cMH7S5FRYFwzjQmAweMMQeNMfXAcmD+GevMB14xIVuAFBHp30rb+cDL1veXgQVN5i83xtQZYz4DDljbAfgm8H8BjDFBY0xJ2w63e+mYUyoSjJqxAIC8zXoXleq4cEJjAHCkyXSBNS+cdVpqm2WMOQ5gfWa2tC0RSbGm/11EPhaRlSKS1VzBIrJYRLaJyLbiYvvuUc/z+uib6CY90WNbDUoNGDyckrjzkU/1EpXquHBCo7nbfkyY64TTNtz9xQADgQ+MMROBzcDS5jZgjHneGJNjjMnJyMhoZXddJ6/Qp2cZKjIMn0V67WG9RKU6LJzQKAAGNZkeCBwLc52W2hZal7CwPota2VYpUA28bs1fCUwkQgWChvxCHyOz9ElwZb/sydcAkP+hvpxJdUw4ofERkC0iw0TETaiT+sz791YBt1t3UU0FTlqXnFpquwpYZH1fBPylyfyFIuIRkWGEOte3GmMM8AYw21rvCmBv2w63+xw5UU1tQ5CR/XTMKWW/QcNGcMIzkODB9+0uRfVwMa2tYIzxi8g9wDuAE3jRGLNHRO60lj8HvAVcS6jTuhq4o6W21qafAFaIyLeAw8BNVps9IrKCUCD4gbuNMQGrzfeB/xGRZUDxqf1EotzGTnA901CRwT/4EjL2v0pJkZe+mf3sLkf1UK2GBoAx5i1CwdB03nNNvhvg7nDbWvNLCZ0tNNfmMeCxZuZ/DswKp2a75Xl9iMAIHd1WRYihk+ZQvX85+7b8jZnzIvbfWyrC6RPhXSS/0MfgtHji3WHlslJd7vxR46lwZVCTv97uUlQPpqHRRU6NOaVUpBCHg+rzppFZsZuKijK7y1E9lIZGF6htCHCotFqfBFcRZ+DEq3EQYPfmv9tdiuqhNDS6wKfFlQSCRp/RUBFn1PipVDmTqcxda3cpqofS0OgCjcOH6OUpFWEcTicVWVNIP7GD2ppqu8tRPZCGRhfI8/pwOx0M7ZtgdylKnSVz/JW4TB27turZhmo7DY0ukFfo4/zMRFxO/eNVkWfMpEupc8RxYvdqu0tRPZD+VusCeV4fI/X5DBWh3B4PJ/rmkFq8FX9Dg93lqB5GQ6OTnaxu4PjJWn0SXEW01NGXExusZu/2DXaXonoYDY1Oll+kL15SkW/slC/jFxfene/aXYrqYTQ0OtmpMadGaGioCBafkERJypdIPL6FYCDQegOlLBoanSzPW0FSbAzn9Ym1uxSlWpQw6nISA2Xk79ludymqB9HQ6GT53kpGZiUh0ty7pJSKHGOnziGIgyPb37a7FNWDaGh0ImMMud4KfRJc9Qh9UtMoSR5NbMEHmGDQ7nJUD6Gh0Ym8FbVU1Po1NFSP4brgUvo0FHLo04h9n5mKMBoanUiHD1E9zYVTQ6+BPbhVXwOrwqOh0YkaQ0PPNFQPkdlvIMXxF+D4fKPdpageQkOjE+V5fWQle0iJd9tdilJhk2EzSa/9nKOHD9pdiuoBNDQ6UV6hT58EVz1O9uTQJap8vUSlwqCh0Un8gSD7iyr1SXDV4wwaNoITnoEEP33f7lJUD6Ch0UkOlVZT7w/qK15Vj+QffAl9q/IpKfLaXYqKcBoanSS/UMecUj3X0ElzEAz7tuglKtUyDY1Okuv14RC4IFOHRFc9z/mjxlPhyqAmf73dpagIp6HRSfK8FQxNTyDW5bS7FKXaTBwOqs+bRmbFbioqyuwuR0UwDY1Okl9Yqc9nqB5t4MSrcRBgz+Z37C5FRTANjU5QUx/gUGmVhobq0UaNn0qVMxlf7jq7S1ERTEOjE+wv8mGMDh+iejaH00lF1lTST/yD2ppqu8tREUpDoxPo8CEqWmSO/zIuU8+urWvtLkVFKA2NTpDn9eGJcTAkPcHuUpTqkDGTLqXWEc+JXfoaWNU8DY1OkFfoIzsrEadDX7ykeja3x0NZ30mklnxEQ0O93eWoCKSh0QnyvD5GZumYUyo6pI6+gthgNXu3b7C7FBWBwgoNEZkjInkickBEljSzXETkaWv5ThGZ2FpbEUkTkXdFZL/1mdpk2Q+s9fNE5Opm9rdKRHa3/XA7X1lVPUW+On0SXEWNcVOvxC8uCneutrsUFYFaDQ0RcQLPANcAo4FbRGT0GatdA2RbP4uBZ8NouwRYY4zJBtZY01jLFwJjgDnAr6ztnKrneqCyPQfbFXKtTvARGhoqSsTFJ1CSchGJxzcTDATsLkdFmHDONCYDB4wxB40x9cByYP4Z68wHXjEhW4AUEenfStv5wMvW95eBBU3mLzfG1BljPgMOWNtBRBKBB4Cftf1Qu4aOOaWiUcKoy0gMlJO780O7S1ERJpzQGAAcaTJdYM0LZ52W2mYZY44DWJ+ZYezv34EngRZvIheRxSKyTUS2FRcXt7Rqh+V6faTEu8hM8nTpfpTqTuNmXEOAGAq2/9XuUlSECSc0mrslyIS5Tjhtw9qfiFwEXGCMeb2V9hhjnjfG5BhjcjIyMlpbvUPyvBWMyEpCRO+cUtEjOTmV4tSLSDq6kYBeolJNhBMaBcCgJtMDgWNhrtNS20LrEhbWZ1Er25oGTBKRQ8BGYISIrA+j/i5jjCG/UF+8pKJT8pirSQicZI/eRaWaCCc0PgKyRWSYiLgJdVKvOmOdVcDt1l1UU4GT1iWnltquAhZZ3xcBf2kyf6GIeERkGKHO9a3GmGeNMecZY4YClwD5xpjZ7TjmTnO0vIbKOr8+Ca6i0vgZc2gQN8c/1ktU6gsxra1gjPGLyD3AO4ATeNEYs0dE7rSWPwe8BVxLqNO6GrijpbbWpp8AVojIt4DDwE1Wmz0isgLYC/iBu40xEXl+3Dh8iI45paJQfEISpek5pHg30dBQj8vltrskFQFaDQ0AY8xbhIKh6bznmnw3wN3htrXmlwJXnKPNY8BjLdRzCBgbRuldKq9Qb7dV0S11/Bw8azex68O1TLxkjt3lqAigT4R3QJ7Xx4CUOJJjXXaXolSXGD/tSuoccRTv0NfAqhANjQ7I8/oYkaWvd1XRy+OJ40TGVNKLP6S2tsbuclQE0NBop4ZAkE+LKxnZT8ecUtEtc8I1uE0tOzf/3e5SVATQ0Ginz0qqaAgYvd1WRb1xF19GjSORsl1v212KigAaGu3UOOaU3jmlolyMy8XJ/tPpW7qd6iqf3eUom2lotFO+14fTIZyfqS9eUtGv/8Sv4DL17Nh41o2QqpfR0GinXK+P4X0T8MQ4W19ZqR5uzMRLqHSmULnnHbtLUTbT0GinvMIKfT5D9RoOp5PKATPJLP+Ek2Un7C5H2UhDox2q6vwcOVHDKO3PUL3I4MnX4SDArg/etLsUZSMNjXY49Q4NHXNK9SYjx+ZQ4cqgJvddu0tRNtLQaIfGMac0NFQvIg4HNYMvJbNiD6XFXrvLUTbR0GiHvEIf8W4ng1Lj7S5FqW41fOp1CEF2bfhL6yurqKSh0Q55Xh/ZWUk4HPriJdW7DM8eR2nsYNj3JiYYtLscZQMNjXbI8/oYqWNOqV5IHA6cY+aTVneE3F1b7S5H2UBDo41KKusorarXMadUrzXhshtoEA9HPnjV7lKUDTQ02uhUJ7iOOaV6q6TkFEr7XUJ64QdUVJTZXY7qZhoabaRjTikFQy65GZep5x9rVtpdiupmGhptlO/1kZ7gJiPJY3cpStlmxJgcSmMHY/a+oR3ivYyGRhvlFvr0+QzV64nDgWP0PNLqjpC3e5vd5ahupKHRBsGgYX+hTy9NKQVMvPxGGsTD4Y1/tLsU1Y00NNqgoKyG6vqAdoIrhXaI91YaGm2Q660AdPgQpU4ZPMPqEF/7mt2lqG6iodEGp263zdbLU0oBoUEMT3gGE9yjHeK9hYZGG+QV+hiUFkeiJ8buUpSKCOJwIGPmkV53WDvEewkNjTYIDR+iT4Ir1VSoQ9zN4Y3L7S5FdQMNjTDV+QMcLKliZD8dc0qpppKSUyjNuoT0wo34KsrtLkd1MQ2NMB0sriIQNDrmlFLNGHzJQlymno/X6hPi0U5DI0w65pRS5zZybA6lsUOQXa/hb2iwuxzVhTQ0wpTr9eFyCsP6JthdilIRRxwOEqb+EykNhWzV8aiimoZGmPILfZyfkYjLqX9kSjUn59J5nPAMxL/tFQKBgN3lqC4S1m9AEZkjInkickBEljSzXETkaWv5ThGZ2FpbEUkTkXdFZL/1mdpk2Q+s9fNE5GprXryI/FVEckVkj4g80bFDb5s8r445pVRLHE4ncVO/SUq9lw9X68N+0arV0BARJ/AMcA0wGrhFREafsdo1QLb1sxh4Noy2S4A1xphsYI01jbV8ITAGmAP8ytoOwFJjzChgAjBDRK5pz0G3VUVtA0fLa3TMKaVacfHs+ZR5zqPho5f0bCNKhXOmMRk4YIw5aIypB5YD889YZz7wignZAqSISP9W2s4HXra+vwwsaDJ/uTGmzhjzGXAAmGyMqTbGrAOwtvUxMLDth9x2+wu1E1ypcDicTjwX30Fq/XE+Wvu/dpejukA4oTEAONJkusCaF846LbXNMsYcB7A+M8Pdn4ikANcROkM5i4gsFpFtIrKtuLi4pWMLy6kXL+nlKaVad/EV11Pm7k/d1t8S1LONqBNOaEgz80yY64TTtk37E5EY4I/A08aYg81twBjzvDEmxxiTk5GR0cruWpfn9ZHoiWFASlyHt6VUtHM6nbgu/idS647x0fq/2F2O6mThhEYBMKjJ9EDgWJjrtNS20LqEhfVZFOb+ngf2G2OWhVF7p8jz+hiRlYhIc3mmlDrTlC/fSLm7HzVb9Gwj2oQTGh8B2SIyTETchDqpV52xzirgdusuqqnASeuSU0ttVwGLrO+LgL80mb9QRDwiMoxQ5/pWABH5GdAHuK/th9o+xhjyCn36JLhSbeB0OonJuZ20uiNse+8Nu8tRnajV0DDG+IF7gHeAfcAKY8weEblTRO60VnsLOEio0/q/gbtaamu1eQK4UkT2A1da01jLVwB7gbeBu40xAREZCPyI0F1YH4vIDhH5dkf/AFpT5KujvLqBkVk65pRSbTH5ipsod2VRvflFHTY9ioQ1xrcx5i1CwdB03nNNvhvg7nDbWvNLgSvO0eYx4LEz5hXQfH9Hl8pr7ATXMw2l2iLG5cI56TZStixl24a3uPjSuXaXpDqBPt7cijy9c0qpdpty5c2cdGVQufF5HZMqSmhotCLX6yMjyUNagtvuUpTqcWJcLtzTv0t63WHe/99f2V2O6gQaGq3IL/TpQ31KdcDkyxbgTZ1Eyt7/4chn+XaXozpIQ6MFgaAhv9DHSB0+RKl2E4eDixY+TJAY8l57WDvFezgNjRZ8XlpFnT/ICD3TUKpDMvsPonriYjJ9e9n411fsLkd1gIZGC/J1zCmlOs0lcxdRlDCK2O2/prjwqN3lqHbS0GhBrteHCGRnamgo1VEOp5MRX3sEp2ngH398xO5yVDtpaLQgz+tjSFo8cW5n6ysrpVo1eNhIykbdSr8TW9m69nW7y1HtoKHRgtDwIXqWoVRnmnXjPZzwDCaw4T85WXbC7nJUG2lonENtQ4BDJVX6JLhSnczlcnPegp8SH6jkwz/+zO5yVBtpaJzDgaJKgga93VapLjBizCSKhs2jn3ctH7z1O7vLUW2goXEOOnyIUl3r0lt/QGHSGBK2/IJPtjT7PjUVgTQ0ziGv0Ic7xsHQ9Hi7S1EqKrk9HqZ855ec9PSn4W8/4mDeLrtLUmHQ0DiHXK+PCzISiXHqH5FSXSU5OZWR//QsDQ4P3lf/D0XeArtLUq3Q34jnUHiyVh/qU6ob9BswmPQb/xN3oJq9L9xFZWWF3SWpFmhonMPb983k8evH2V2GUr3CiDETcVz5CKl1BWx67l90GPUIpqFxDiJCrEsf6lOqu0y8ZA4VOf9Cv4pPWPPCD3VgwwiloaGUihgz591B4fAb6H/sXd597kHq6mrsLkmdQUNDKRVRrrjthxRdcBP9vOvY+J+3U1rstbsk1YSGhlIqojicTr58+4+ovuRHpNZ8Ru6zt7J/7z/sLktZNDSUUhFp6lU3kXjjc4gxVLz6HT5cowMcRgINDaVUxBoxLofsO3/HydhBxL33CKt/9wTBQMDusno1DQ2lVETLyBrA9Pt/x/HMWWTmL2ft0q/r5SobaWgopSJebGwcV921jJM595JUc4Tq5d/knee+x4mSQrtL63U0NJRSPYI4HMycdwej7n2DwsFzyTq2hk9/uYB1K5+hvq7O7vJ6DQ0NpVSP0ic1jau+/e+k3P4/lCdmk77rv9ny8/lsWf0nGhrq7S4v6okxxu4aulROTo7Ztm2b3WUopbqACQb5+IO/UbX+aVIaCqlyJlM59CpGX3YrAwYPt7u8Hk1Ethtjcs6ar6GhlOrpAoEAOza9TelHfyKrPNRJXpQ8lj4Tr2fCzK/gcrltrrDn0dBQSvUK3qOH2b3uD8QffIdEfxm1jnjKUr9E/PApZE+8nH4DBttdYo+goaGU6lX8DQ3s2PQOpbtXk1Syg8RAOQBl7v7U9ZtI+ohpDB45gb4Z/RGHdu+eSUNDKdVrmWCQz/bv4fOd7+H//EPSffuIMX4Aqp1JVCYMwaRdQOKAUfQbPpb+g84nNjbO5qrt1aHQEJE5wFOAE/iNMeaJM5aLtfxaoBr4J2PMxy21FZE04FVgKHAI+Joxpsxa9gPgW0AA+D/GmHes+ZOAl4A44C3gXtPKAWhoKKXOVFtTTf7ujzhxeC913nzcJz8lpfYYTvyN61Q5k6n2ZOKPz8LRpz+xqQOIT80koU8GyWkZpKZlEhefYONRdK12h4aIOIF84EqgAPgIuMUYs7fJOtcC/0IoNKYATxljprTUVkT+H3DCGPOEiCwBUo0x3xeR0cAfgcnAecBqYIQxJiAiW4F7gS2EQuNpY8zfWqpfQ0MpFY6GhnoOH9xH4Wd7qCk9SuDkMRyVXmLriklqOIFw9vs9GsRDTUwS9TFJBGPiCMYkYNwJiDsBhycBpycBhzsOpyuWGHccMZ44XJ44XJ5YYmI8xLg9OF1uXG43LrcHl8tDjMtFTIyLGGcMTqfTtktn5wqNmDDaTgYOGGMOWhtaDswH9jZZZz7wivWv/i0ikiIi/QmdRZyr7XxgttX+ZWA98H1r/nJjTB3wmYgcACaLyCEg2Riz2drWK8ACoMXQUEqpcLhcbs4f+SXOH/mls5b5GxooKizgZGkh1SdLqKk4QZ2vhEBVGaamDEe9D0dDNZ76Y7h81biCtbiDtQjN/6Pcb/2EI4iDoDgJ4sCIA4MQxAkiBHGc9mkQTJPvUx76306/zBZOaAwAjjSZLiB0NtHaOgNaaZtljDkOYIw5LiKZTba1pZltNVjfz5x/FhFZDCwGGDxY75RQSnVMjMvFeQOHcd7AYWG3CQYC1NRUUVtTTW1NNfV11TTU1lJfV0NDXTV+fwOBhjqC/nqC/gYC/nqMvx4TaCAYDEAwEPoMNEAwgAkGwATBBEPfMUgwgDFBMKEoMcaAMUAQMQaHdP5ZSjihIc3MOzM+z7VOOG3D3V/Y2zLGPA88D6HLU63sTymlOp3D6SQhMZmExGS7S+lU4cRQATCoyfRA4FiY67TUttC6hIX1WRTGtga2UodSSqkuFE5ofARki8gwEXEDC4FVZ6yzCrhdQqYCJ61LTy21XQUssr4vAv7SZP5CEfGIyDAgG9hqbc8nIlOtu7Vub9JGKaVUN2j18pQxxi8i9wDvELpt9kVjzB4RudNa/hyhO5muBQ4QuuX2jpbaWpt+AlghIt8CDgM3WW32iMgKQp3lfuBuY8ypt658ly9uuf0b2gmulFLdSh/uU0opdZZz3XKrz84rpZQKm4aGUkqpsGloKKWUCpuGhlJKqbBFfUe4iBQDn7ezeV+gpBPL6Sn0uHsXPe7eJdzjHmKMyThzZtSHRkeIyLbm7h6IdnrcvYsed+/S0ePWy1NKKaXCpqGhlFIqbBoaLXve7gJsosfdu+hx9y4dOm7t01BKKRU2PdNQSikVNg0NpZRSYdPQaIaIzBGRPBE5YL2/PGqJyIsiUiQiu5vMSxORd0Vkv/WZameNXUFEBonIOhHZJyJ7RORea35UH7uIxIrIVhH5xDruR6z5UX3cACLiFJF/iMib1nTUHzOAiBwSkV0iskNEtlnz2n3sGhpnEBEn8AxwDTAauEVERttbVZd6CZhzxrwlwBpjTDawxpqONn7gQWPMhcBU4G7r7znaj70OuNwY8yXgImCO9Q6caD9ugHuBfU2me8Mxn3KZMeaiJs9ntPvYNTTONhk4YIw5aIypB5YD822uqcsYY94HTpwxez7wsvX9ZWBBd9bUHYwxx40xH1vffYR+mQwgyo/dhFRaky7rxxDlxy0iA4GvAL9pMjuqj7kV7T52DY2zDQCONJkusOb1JlnWmxKxPjNtrqdLichQYALwIb3g2K3LNDsIvWL5XWNMbzjuZcC/AsEm86L9mE8xwN9FZLuILLbmtfvYW31zXy8kzczT+5KjlIgkAn8C7jPGVITeJBzdrDdhXiQiKcDrIjLW5pK6lIjMBYqMMdtFZLbN5dhhhjHmmIhkAu+KSG5HNqZnGmcrAAY1mR4IHLOpFrsUikh/AOuzyOZ6uoSIuAgFxu+NMf9rze4Vxw5gjCkH1hPq04rm454BzBORQ4QuN18uIr8juo+5kTHmmPVZBLxO6BJ8u49dQ+NsHwHZIjJMRNzAQmCVzTV1t1XAIuv7IuAvNtbSJSR0SvECsM8Y84smi6L62EUkwzrDQETigC8DuUTxcRtjfmCMGWiMGUro/89rjTHfIIqP+RQRSRCRpFPfgauA3XTg2PWJ8GaIyLWEroE6gReNMY/ZW1HXEZE/ArMJDZdcCPwU+DOwAhgMHAZuMsac2Vneo4nIJcAGYBdfXOf+IaF+jag9dhEZT6jj00noH40rjDGPikg6UXzcp1iXpx4yxsztDccsIsMJnV1AqDviD8aYxzpy7BoaSimlwqaXp5RSSoVNQ0MppVTYNDSUUkqFTUNDKaVU2DQ0lFJKhU1DQymlVNg0NJRSSoXt/wO/H6rksOE5tQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, warmup_t=5, warmup_lr_init=1e-5)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"With warmup\");\n", "\n", "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"Without warmup\", alpha=0.8);\n", "\n", "plt.legend();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see by setting up `warmup_t` and `warmup_lr_init`, the `cosine` scheduler first starts with a value of `warmup_lr_init`, then gradually progresses up to the `initial_lr` set in the optimizer which is 1e-4. It takes `warmup_t` number of epochs to go from `warmup_lr_init` to `initial_lr`. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `warmup_prefix`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Defaults to `False`. If set to `True`, then every new epoch number equals `epoch = epoch - warmup_t`." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABBy0lEQVR4nO3deXxV1bn4/89zTuYRMpCRKRDGMMgsCOIIKoMVbwtYtbbqtWpva6tXbXu96reD/Vb7U6rVOuu3dbatONQBFAVBGWRMGBLGJISQhCEDGc9Zvz/2SQyBJIeQ7J3heb9e57XP2Xuv/awdwnmy91p7LTHGoJRSSvnD5XQFlFJKdR2aNJRSSvlNk4ZSSim/adJQSinlN00aSiml/BbgdAU6WlxcnBkwYIDT1VBKqS5lw4YNxcaY+Kbru33SGDBgAOvXr3e6Gkop1aWIyP7TrdfbU0oppfymSUMppZTfNGkopZTymyYNpZRSftOkoZRSym9+JQ0RmS0iO0UkR0TuOc12EZElvu1bRGRca2VFJEZEPhGRbN+yt299rIh8JiLlIvJ4kzjjRWSr71hLRETafupKKaXOVKtJQ0TcwBPAZcAIYJGIjGiy22VAuu91M/CkH2XvAZYbY9KB5b7PAFXA/wB3nqY6T/qOXx9rtl9nqZRSql3485zGJCDHGLMHQEReA+YDWY32mQ+8bKxx1r8SkV4ikgQMaKHsfGCmr/xLwArgbmNMBbBKRAY3roTveFHGmDW+zy8DVwL/PrNT7vyyVr9Padby5ndodIElNLrYkpMWp9u96ZaGbdLkc/0663Oj/USsl+/AItY+gsv33lq66pcuwSW+l0sa1rvdLtyuxi8hwO3C7XbjEpfvoC6rFg1BXM283OCqX7qtda4A670rAFyB1tId4FsGffsKCAZ3ILiDrfcudwv/Mkopf5JGCpDb6HMeMNmPfVJaKZtgjCkAMMYUiEgfP+qRd5oYpxCRm7GuSOjXr18rh+18wpbdwwjvAbzm1G97l+j8Jx3KHQyBoRAYdvIyOBJCoiA4yvc+2nofFgvhcRAeb73CYq3kpFQ35c9v9+n+Tm36zdXcPv6U9ZffxzLGPA08DTBhwoQu9S1bXXWCVE8ea1Jv4NybHj1le+NJsxrPn2WabDdN9jG+NSeVMdZ6a/ltWa+xDlC/zWus0l6vF4y13Wu8eI3B6wWPt/69F48x1mevtb/H66XO46XOa6jzGjweL7UeL3UeD7V1Huo8XmrqrHU1dR5qauuorqujusZDda2H6to6qurqqKqp5UR1HSeqajlRU8uJ6hpq6jy48eLGi8u3dOPFJYbeIS76hAeQFBlAQkQACeFu4sPd9Al3Ex8mBOEBTzV4aqCuxnpfVw21lb7XCWtZVwU15VB1DI4dgOpSqC6ztjcnNAaiUqBXP+jd31r26ge9+kPMQAgK9/fXQalOx5+kkQf0bfQ5FTjo5z5BLZQtFJEk31VGEnDYj3qktlKPLi8/Zwtp4iUwOeO02xu3/Z/+tlPP6RtQU+elrKqWoydqKC6voaS8hpKKaorLaygur6bgWCWbj1aSl3uCqlpvQzmXwKD4CIYnRTEiOYoRSVEMT4oiPjLY/+CeWqgqhRMlUFFkvU4UQ0UxlB+G43lwZA/s+axJghGIHQyJoyAxAxJHW+8jElq6j6hUp+FP0lgHpIvIQCAfWAgsbrLPUuB2X5vFZOC4LxkUtVB2KXA98JBv+U5LlfAdr0xEpgBfA9cBf/aj/l3KkT2bSAPi0sY6XZVOLyjARWxEMLERwQxu4eamMYbi8hryjp4g92glOYVlZBWUsn7fEZZu/vbvjpReoZw3OI7pQ+KYNiiO3uFBzR/UHQjhsdYrfkhLwa3Ecmw/HN0Pxbvg0FbIXw+Z//h2v8gkSJsJgy6ylhGnjBOnVKfQatIwxtSJyO3AR4AbeN4Ykykit/i2PwV8AFwO5AAngBtaKus79EPAGyLyI+AA8B/1MUVkHxAFBInIlcClxpgs4MfAi0AoVgN4t2sEry3IpMa4SRk0yumqdBsiQnxkMPGRwZzTr/dJ246dqCGroJSsg6Vs2H+UD7YV8Pr6XERgdEo009PjmTk0nvH9e9OmHt4ivjaPOEgZf/K2ymNQmGklkdyvYNeHsPlVa1viaBh0IQy+GPpPsxr6leoEpPE98u5owoQJpiuNcrv5D5cQVX2YgfdtdroqPVKdx8vmvOOszC5iZXYxm3KP4fEa0uLCWTSpHwvGpxLT0hXI2fB6oGAz7F4Ouz+D3K/BW2e1hYy/HsZ+HyITOia2Uk2IyAZjzIRT1mvS6FwO3T+YvKgxTPj5205XRQHHK2v5JKuQ19YeYP3+owS5XczKSGTxpH5MSYtp29WHv6rLYNdHsOFF2LfS6i489HIY/wNIu0CvPlSHai5paN/ATqT0WAmJFLE3dpjTVVE+0aGBXD0+lavHp7LzUBmvrj3AP77J493NB0mLD+eOi4cwZ3RSxySP4EgYdbX1Ks6Bb16ETa/A9qXQewDM+G8Ys0iTh7KV/rZ1Igd3bQAgNFXbMzqjoYmR3D9vJGt/dTGP/McYggPc/OTVjXz3r2vYln+8Y4PHDYZLfwM/3w4LnrOeB3nnVnj2Qshd27GxlWpEk0Yncnz/FgASBp/jcE1US0IC3SwYn8p7PzmPh64axZ6iCuY+vop73t5CUVl1xwYPCLauPH60DK56BsoOwXOXwNs3wfH8jo2tFJo0OpfCTMpNKIl9052uifKD2yUsnNSPz+6ayY3nDeStDXlc+PAKnv5iNzV13tYPcDZcLhj9Xbh9PUy/E7LegccnwOd/tB5KVKqDaNLoRCJLs8kLHIDoPeouJSokkF9dMYKP7pjBxIEx/O6DHXzv6TUcLqvq+ODBEXDR/8Dta63uuZ/9xrryOJbbelml2kC/nToJ4/WSXLOX41F6ldFVDYqP4PkfTOTxxeewo6CM+Y9/2fFtHfV6D4Dv/T9Y9Lr1EOEzF8CBr+yJrXoUTRqdRPGhA/SiHBPfdNR51dXMGZ3MWz8+F5cIVz+1mnc32zjazdDZcOMyq+fVi3Ng49/si616BE0ancQhX8+pyH6jHa6Jag8jk6N55/ZpZCRH85NXN/LIxzvxem16Jip+KNz0KQyYBu/cBh/+Ejx19sRW3Z4mjU6iIm8rAMlDx7eyp+oq4iKC+ftNk/nuhFT+/GkOP/77BiqqbfryDu0N17wNk38MXz0Br3zXGrZEqbOkSaOTcBdtp4je9I5LdLoqqh0FB7j5w4LR3DdnBJ9kFXLNs19TWeOxJ7g7AC57COb9GfZ+AS9cDlU2tbGobkuTRifRuzybQ8EDna6G6gAiwg/PG8hfrhnH5rxj/PS1jXjsulUFMO46WPw6FO+EN66zhnVXqo00aXQCnro6UusOUNFrqNNVUR1odkYS980ZwcdZhfzm/azWC7SnwRdZVxx7VsC7Pz15Ni6lzoCOPdUJHNybSV+pxZU40umqqA52w7SB5B2t5LlVe0ntHcaPzrPx6nLsYmv2wRW/t2YSnHmPfbFVt6FJoxMo2r2RvkDvAWOcroqywa8uH07+0Up+834WKb1CmJ2RZF/w8+8+OXGMbTqfmlIt09tTnUBN/ja8RkgdomNO9QQul/DowrGM7duLn762iW8OHLUvuAjMeRQGng9Lf2LdrlLqDGjS6ASCjuwk35VEaHik01VRNgkJdPPsdRNIjA7hxpfWs7+kwr7gAUHW0+NxQ+D1a6HQ5vYV1aVp0ugE4k/kUByW5nQ1lM1iI4J54QcT8RrDDS+us68rLkBINFzzJgSGWc9waFdc5SdNGg6rOlFOsreAqhideKknSouP4InF49hTVMGjy3fZGzw6FRa+AqX58Mn/2htbdVmaNByWl70JtxiCkjOcropyyLTBcSyc2JdnV+61b4DDeqnjYcqtsOEF2LfK3tiqS9Kk4bCjezcBEJemjeA92b2XDyc2PIj/fmsLtZ4OnoujqQt+ZY2Su/S/dC4O1SpNGg7zFGyj2gSSkqaj2/Zk0aGBPDg/g6yCUp5Zucfe4EFhVo+qI7vh8z/YG1t1OZo0HBZ+fBe5Af0ICAxyuirKYbMzEpk9MpFHl2Wzp6jc3uCDLoCx34cvl0DBZntjqy5Fk4bDEqr2cjRikNPVUJ3Eg/NHEhLg4t5/bLVvKPV6s34DYbHW8xs6lLpqhiYNBx0vKaQPR/DEDXe6KqqT6BMVwq+uGM7Xe4/w2jqbp2wN7Q2X/9G60vjqCXtjqy5Dk4aD8ndaEy+FpurES+pb353Ql6mDYvn9B9s5dNyGecYbGzEfhl4Bn/0OSnbbG1t1CZo0HFSWuwWAxHTtOaW+JSL8/qpR1Hq9/M872zB2jkgrAlc8DO4gHQ1XnZYmDScdzqKUcPok6zwa6mT9Y8O54+IhfJJVyKqcYnuDRyXDJQ/AvpWQ9Y69sVWnp0nDQdGl2eQFDkRc+s+gTvWDaQNIig7hsWXZ9l5tAIy7HmLT4fP/C16bnxtRnZp+WznEeL2k1O6lLGqw01VRnVRwgJtbZw5i/f6jrN5dYm9wlxtm3AWHM2HHe/bGVp2aJg2HFObtJpJKSNCJl1TzvjuxL4lRITy2PNv+4BkLIGaQdbWhbRvKx6+kISKzRWSniOSIyCnTfYlliW/7FhEZ11pZEYkRkU9EJNu37N1o272+/XeKyKxG6xeJyFZfjA9FJK7tp+6swznfABDZTydeUs0LDnBzy/lprN17hDV2X224A2DGnVC4FXZ+YG9s1Wm1mjRExA08AVwGjAAWiUjTMS8uA9J9r5uBJ/0oew+w3BiTDiz3fca3fSEwEpgN/EVE3CISADwGXGCMGQ1sAW5v43k7riJvGwApQ8a1sqfq6RZO6kefyGCWOHG1Meq71rhUn/9BrzYU4N+VxiQgxxizxxhTA7wGzG+yz3zgZWP5CuglIkmtlJ0PvOR7/xJwZaP1rxljqo0xe4Ec33HE9woXEQGigINnfMadRGBxFoXEEh0T73RVVCcXEujmP88fxJo9Jazde8Te4O4AmH6n9cDfro/sja06JX+SRgrQ+NHUPN86f/ZpqWyCMaYAwLfs09KxjDG1wI+BrVjJYgTw3OkqLCI3i8h6EVlfVFTkxynaL6Yih0MhOvGS8s81k/sRFxHMY3bPuQEwZiH06g+fP6RXG8qvpCGnWdf0N6e5ffwp61c8EQnEShrnAMlYt6fuPd0BjDFPG2MmGGMmxMd3vr/ka2uqSa3Lo7LXEKerorqIkECrbePLnBLW77P7aiMQpv8CDm6EnGX2xladjj9JIw/o2+hzKqfeFmpun5bKFvpuYeFbHm7lWGMBjDG7jdVp/Q1gqh/173QO7t5GkNThTtKJl5T/Fk/uR2x4kDM9qcYsgui+sEKvNno6f5LGOiBdRAaKSBBWI/XSJvssBa7z9aKaAhz33XJqqexS4Hrf++uBdxqtXygiwSIyEKtxfS2QD4wQkfpLh0uA7Wd4vp1C8d6NAPQeONbZiqguJSwogJtnpLEyu5hvDhy1N3hAEEz/OeSvh92f2htbdSqtJg1jTB1WL6WPsL6k3zDGZIrILSJyi2+3D4A9WI3WzwC3tlTWV+Yh4BIRycZKAA/5ymRiXUVkAR8CtxljPMaYg8ADwBcisgXryuN3Z3f6zqg5mEmdcZGart1t1Zn5/pT+xIQHOdOTauw1EJWqPal6OLF9eAKbTZgwwaxfv97papxk4/+9jJiqA/S/L7P1nZVq4i8rcvi/H+7kndumMaZvL3uDr30GPrgTrnsH0mbaG1vZSkQ2GGMmNF2vT4Q7oE/lborDdOIl1TbXnTuAiOAAXl6z3/7g466DsDgreageSZOGzU6UHyfFFFITM8zpqqguKiI4gPljk3lvy0GOn6i1N3hAMJxzDez8N5QW2BtbdQqaNGyWt8tqBA9OGeVwTVRXtnhyP6rrvPxzY579wcddD8YDG/9mf2zlOE0aNju+dxMA8YPGOloP1bWNTI5mTGo0r67NtX/Y9NhBMPB8+OZl8Hrsja0cp0nDZp7CLCpNEMkDdF5wdXYWTerHzsIyvjlwzP7g438Axw/A7s/sj60cpUnDZhHHd5IX2B93QIDTVVFd3NwxyUQEB/DK1wfsDz5sjtUgvuEF+2MrR2nSsFlS9V6ORejES+rshTvaIB6kDeI9lCYNGx0pzCOW43ji9daUah+LJmmDuLKXJg0bHcy2Jl4K7zva4Zqo7iIjJZrR2iCubKRJw0YVuVsASBqsEy+p9rNYG8SVjTRp2EgOb+cokcQm9m19Z6X8NHdMMuFBbm0QV7bQpGGj6LJsDgYNRFz6Y1ftJzw4gPnnpGiDuLKFfnvZxOvx0Ld2H+VR6U5XRXVDi30N4v/alG9/8PoG8U3aIN4TaNKwyaED2YRJNSSMdLoqqhuqbxB/5esDzjWIb9AG8Z5Ak4ZNDudYPaeiB+gcGqpj6BPiyg6aNGxSlb8VgJQh2nNKdYx5vgbxN9bl2h+8vkF848v2x1a20qRhk8CSHRQQT2R0jNNVUd1UeHAAs0Ym8u9tBVTX2XybKCAIMq6CXR9BdZm9sZWtNGnYJLYih8JQnXhJday5Y5Iprapj5a5i+4NnLIC6Ktjxgf2xlW00adigprqKFE8+lb2HOF0V1c2dlx5Hr7BA3t1y0P7gqZMgui9se9v+2Mo2mjRskJ+zmUDxEJiU4XRVVDcX6HZxWUYSn2QVUllj8y0qlwtGfgd2L4cTR+yNrWyjScMGJXs2ARCbNtbReqieYe6YJE7UePh0x2H7g2csAG8dbH/X/tjKFpo0bFBXsI1a4yZlsHa3VR1v8sBY4iODeXezA7eoksZAzCC9RdWNadKwQcjRXeS7UwgKDnG6KqoHcLuEK0Yl8enOw5RV2TysiIh1tbFvJZQV2htb2UKThg0SKndTEq4TLyn7zB2TTE2dl0+yHPjizlgAxgtZ79gfW3U4TRodrOz4EZIooiZ2qNNVUT3IuH69SOkVylInblH1GQZ9Ruotqm5Kk0YHy99lDR8SmqITLyn7iAhzxiSxKruYIxU19lcg4yrI/QqOOfB0uupQmjQ62PF9mwHooxMvKZvNHZ1Mndfw4bZD9gfPuMpaZv7T/tiqQ2nS6GiHszhhgknsp0OiK3uNTI4iLS7cmV5UMWmQPE5vUXVDmjQ6WMTxXeQGDsDldjtdFdXDWLeokvlqbwmHS6vsr0DGAijYBCW77Y+tOowmjQ5kvF6Sa/ZyPFKvMpQz5o1Jwhh4f6sDs+qN/A4gsO0f9sdWHcavpCEis0Vkp4jkiMg9p9kuIrLEt32LiIxrrayIxIjIJyKS7Vv2brTtXt/+O0VkVqP1QSLytIjsEpEdIrKg7afe8UoKc+lNGd744U5XRfVQg/tEMjwpyplbVNEp0H+q3qLqZlpNGiLiBp4ALgNGAItEZEST3S4D0n2vm4En/Sh7D7DcGJMOLPd9xrd9ITASmA38xXccgF8Bh40xQ3zH+7wN52ybgmyr51REP+05pZwzd0wS3xw4Ru6RE/YHz7gKirZDYZb9sVWH8OdKYxKQY4zZY4ypAV4D5jfZZz7wsrF8BfQSkaRWys4HXvK9fwm4stH614wx1caYvUCO7zgAPwR+D2CM8RpjHBj/2X8VuVsASB4y3uGaqJ5s7uhkwKFbVMPng7j1aqMb8SdppACNO1vn+db5s09LZROMMQUAvmWflo4lIr18n/+PiHwjIm+KSMLpKiwiN4vIehFZX1RU5Mcpdgx30XaK6UVMn6Y/LqXs0zcmjLF9ezlziyoiHtLOt5KG3XOXqw7hT9KQ06xr+q/f3D7+lPU3XgCQCnxpjBkHrAEePt0BjDFPG2MmGGMmxMfHtxKu4/Qqz+FQ8ADH4itV77KMRDIPljpzi2r4PDi6Fw7rLaruwJ+kkQf0bfQ5FWj6J0tz+7RUttB3Cwvfsn4c5+bKlAAngPqnhd4EOu0Tc566OlJr91MercOHKOfNGpkIwEeZDjzoN+wKQHS49G7Cn6SxDkgXkYEiEoTVSL20yT5Lget8vaimAMd9t5xaKrsUuN73/nrgnUbrF4pIsIgMxGpcX2uMMcC7wEzffhcBnfZPl4J92wmVGlyJI52uilIMiAtnWGIkH2c6MIBhRB/oNwW2v2d/bNXuWk0axpg64HbgI2A78IYxJlNEbhGRW3y7fQDswWq0fga4taWyvjIPAZeISDZwie8zvu1vYCWED4HbjDH1U5DdDdwvIluAa4FfnMW5d6jDuzcB0GuAzqGhOodZIxNZt/8IRWXV9gcfPhcKt8KRvfbHVu1KTDdvnJowYYJZv3697XHXPP/fTN7/NFV37ScsItr2+Eo1lXWwlMuXrOT3V41i0aR+9gY/uh8eGw2X/gam/sTe2KpNRGSDMWZC0/X6RHgHCT6yg4OuBE0YqtMYnhRJv5gwZwYw7N0fEkdru0Y3oEmjg8Se2E1RaJrT1VCqgYgwa2QCq3cXU2r3jH5g3aLKXQtlDiQt1W40aXSAqsoKUjwHqYoZ5nRVlDrJ7IxEaj2Gz3Ycbn3n9jZ8LmBgx/v2x1btRpNGB8jP3kSAeAlKznC6Kkqd5Jy+vYmPDHam6238MIgZBDu0F1VXpkmjAxzZa028FJt2jsM1UepkLpdwyYgEVuwsoqrW03qB9iQCw+fA3i+g8qi9sVW70aTRATwF26gxAaSk6TMaqvOZPTKREzUeVmY7MHTb8HngrYNdH9sfW7ULTRodIOzYLnID+hIYFOx0VZQ6xZS0WCJDApy5RZU8DiKTYHvT54NVV6FJowMkVO3hSPhgp6uh1GkFBbi4eHgCy7YXUufx2hvc5YJhcyBnOdQ4MA6WOmuaNNrZ8SNFJFBCXZxOvKQ6r1kjEzh2opa1e4/YH3z4HKirhN3L7Y+tzpomjXZ2MHsDAGGpoxyuiVLNmzEknpBAlzO3qPpPg9DeOhZVF6VJo52V7rcmXkpI77QD8CpFWFAAM9Lj+SizEK/X5qGE3IEw5DLY9W/wOPCQoTormjTaW2EmpYSRkKJPg6vObXZGIodKq9iSf9z+4MPnQtVx2LfS/tjqrGjSaGdRpdnkBw5AXPqjVZ3bRcMSCHCJM2NRDboAAsN1LKouSL/Z2pHxekmp3Udp1BCnq6JUq6LDAjl3UCwfZx7C9tGuA0Mh/WJrSBGvzT241FnRpNGODh/cSxQV0Ed7Tqmu4dKRiewpriDncLn9wYfNhfJCyLd/6gLVdpo02lFh9jcARPbTiZdU13DpiAQAZ25RDbkUXIH6oF8Xo0mjHZ3I2wpAytDxDtdEKf8kRIUwrl8vPspyIGmEREPa+VbX224+GVx3okmjHbmLt3OYGKJj+jhdFaX8NmtkItvyS8k94sAT2sPmwNG9UJjZ+r6qU9Ck0Y5iynM4FKJdbVXXMmtkIgAfZxXaH3zYFYDocOldiCaNdlJXW0NqXS4neg11uipKnZEBceEMS4zkIyfaNSL6QL8p2vW2C9Gk0U7y92QRLLW4EnU4dNX1zBqZyLr9Rygqq7Y/+PC5ULgNjuyxP7Y6Y5o02knJno0AxAzUnlOq65k1MhFjYNl2J25RzbGWOhZVl6BJo51UH9yGxwip6WOdropSZ2x4UiT9YsKc6Xrbuz8kjtZbVF2EJo12EnJkB/muZELCIpyuilJnTESYNTKB1buLKa1yYBDB4XMhby2UOZC01BnRpNFO4k7soThskNPVUKrNZmckUusxfLbjsP3Bh8+1ltqLqtPTpNEOKivKSPEWUB07zOmqKNVm5/TtTXxksDNzbMQPg5hB2q7RBWjSaAd5uzbiEkNwcobTVVGqzVwu4dIRCXy2o4iqWo+9wUWsq419K6HyqL2x1RnRpNEOju7bDED8oLHOVkSpszRrZCKVtR5WZhfbH3z4XPDWwa6P7I+t/KZJox14D2VSaYJIHqjPaKiubUpaLFEhAc70okoeB5HJ2ouqk9Ok0Q7Cj+0kL6Af7oAAp6ui1FkJCnBx0fAElu8opNZj8zwXLpc1rEjOcqipsDe28psmjXaQWL2XYxGDna6GUu1i1shEjp2oZe3eI/YHHz4X6iqtxKE6Jb+ShojMFpGdIpIjIvecZruIyBLf9i0iMq61siISIyKfiEi2b9m70bZ7ffvvFJFZp4m3VES2nfnptr9jxYeI5yieeJ14SXUP5w+JJyTQ5Uwvqv7TILS3dr3txFpNGiLiBp4ALgNGAItEZEST3S4D0n2vm4En/Sh7D7DcGJMOLPd9xrd9ITASmA38xXec+vpcBTgwzdjp5e/cAEBY39EO10Sp9hEa5Ob8IfF8lHkIr9fmeS7cATD0ctj5IdTV2Btb+cWfK41JQI4xZo8xpgZ4DZjfZJ/5wMvG8hXQS0SSWik7H3jJ9/4l4MpG618zxlQbY/YCOb7jICIRwM+B35z5qXaM8twtACSlj2tlT6W6jlkjEyksrWZj7jH7gw+bA9XHYe8X9sdWrfInaaQAuY0+5/nW+bNPS2UTjDEFAL5l/cxFLZX5P8AjQIuzxYjIzSKyXkTWFxUVtbTr2TucxTEiiEvs17FxlLLRxSMSCHK7eG/LQfuDD7oQgqMg85/2x1at8idpyGnWNb1mbW4ff8r6FU9ExgKDjTGt/iYZY542xkwwxkyIj49vbfezEl2aTX7QQMSlfQpU9xEVEsjMofG8v6UAj923qAJDrKuN7e9CnQNDtasW+fNNlwf0bfQ5FWj650dz+7RUttB3Cwvfsn7Am+bKnAuMF5F9wCpgiIis8KP+HcZ4vaTW7qM8Kt3JaijVIeaOSeZwWbUzvagyFli3qHKW2R9btcifpLEOSBeRgSIShNVIvbTJPkuB63y9qKYAx323nFoquxS43vf+euCdRusXikiwiAzEalxfa4x50hiTbIwZAJwH7DLGzGzDObebQ7nZREglJOhDfar7uWh4H0ID3bzrxC2qtPMhNAa2vW1/bNWiVpOGMaYOuB34CNgOvGGMyRSRW0TkFt9uHwB7sBqtnwFubamsr8xDwCUikg1c4vuMb/sbQBbwIXCbMcbmgXD8U5hjTbwU1V8nXlLdT1hQABePSODfWwvsf9DPHQgj5sPOf+uDfp2MX48wG2M+wEoMjdc91ei9AW7zt6xvfQlwUTNlfgv8toX67AMcHx2wMm8rAClDtOeU6p7mjk7i3c0H+TKnmJlD+7ReoD1lLIANL8CuD633qlPQ1tuzEFiyg0PEE9Ur1umqKNUhzh8aT2RIAO9uLrA/eP+pEJEI2/5hf2zVLE0aZyG2PIdDoWlOV0OpDhMc4Gb2yEQ+zjxk/3DpLjeM/A5kfwxVx+2NrZqlSaONamuqSfHkUtlrqNNVUapDzR2TTFl1HZ/v6uBnnk4nYwF4amDH+/bHVqelSaON8ndvJUg8BCZpzynVvU0dFEtMeBDvbnagF1XqBIjup72oOhFNGm1UvMfqOdV74DkO10SpjhXgdnH5qESWbz/MiZo6e4OLQMZVsPszqCixN7Y6LU0abVR7MJM64yI1fZTTVVGqw80dnUxlrYdPsgrtDz7qajAe2P5O6/uqDqdJo41Cju4kz51CcEiY01VRqsNNHBBDQlSwM72oEjIgboj2ouokNGm0UULlborDdeIl1TO4XMKc0cl8vuswx0/U2htcxGoQ37cKSh1IWuokmjTaoKLsGMmmkNoY7Tmleo65Y5Kp9Rg+ynJgcqaRVwEGsv5lf2x1Ek0abZC38xsAQlK0PUP1HGNSo+kXE+ZML6r4IZA4SntRdQKaNNrg+P7NAPQZrMOHqJ5DRJg7JonVu0soLndgyPKMBZC3Do7usz+2aqBJow28hVmcMMEk9dfbU6pnmTsmGY/X8P4WB9oWRl5lLbe+ZX9s1UCTRhtEHN9FXmB/XG536zsr1Y0MTYhkZHIUr649gDVOqY1694f+58HG/wdem0fdVQ00abRBcvUejkVozynV84gIiyf3Y8ehMjY5MX/4hBus21N7V9gfWwGaNM5YSWEeMZTi7TPC6aoo5Yh5Y5IJC3Lz6toD9gcfPteanGnDi/bHVoAmjTNWsGsDABF9RztcE6WcERkSyLwxyby7uYDSKpuf2QgIhrGLrQEMyxx4Ol1p0jhT5blbAEhM155TqudaPLkflbUe3tmYb3/w8T8Abx1s+rv9sZUmjTPlKtrOEaKIS+zrdFWUcsyolGhGJkfx968daBCPS7caxL95SRvEHaBJ4wz1KsvmYLBOvKR6NhFh0SSrQXxzngMTJGmDuGM0aZwBr8dDau1+yqPSna6KUo6bP9ZqEH/l6/32Bx82RxvEHaJJ4wwU7N9FmFTjStSJl5RytEE8MOTbBvHyw/bG7uE0aZyBwzlWz6no/mMcrolSncOiSb4G8U0OjEdV3yC+8W/2x+7BNGmcgar8rQCkDNHZ+pQCGJ0azYikKF7RBvEeQ5PGGQg6spODkkBEVG+nq6JUp1D/hPj2glJnGsTH/0AbxG2mSeMMxFbkUBg6yOlqKNWpzB+bTGigm1e/1ifEewJNGn6qrjpBqiefqt5DnK6KUp1KfYP40s0HKdMG8W5Pk4af8nO2EiBeApMznK6KUp1O/RPi/9IG8W5Pk4afjuzZCEBcmjaCK9XU6NRoMlKieH7VXuo8NjdKx6XDgOmw9mmorbI3dg+kScNPtQWZ1Bg3KYN0ilelmhIRbr9gMHuLK3h3iwNXGzPuhLICa64N1aE0afgp7NhO8t19CQwKdroqSnVKl45IZFhiJH/+NAeP1+butwPPh75TYNX/B3UOTEXbg/iVNERktojsFJEcEbnnNNtFRJb4tm8RkXGtlRWRGBH5RESyfcvejbbd69t/p4jM8q0LE5H3RWSHiGSKyENnd+pnJqFyDyU68ZJSzXK5hP+6KJ09RRW8Z/fVhgjMvBtK87Vto4O1mjRExA08AVwGjAAWiUjTGYguA9J9r5uBJ/0oew+w3BiTDiz3fca3fSEwEpgN/MV3HICHjTHDgHOAaSJyWVtO+kyVHishkSJqY3VOcKVaMntkIkMSIpy52ki7AFIn+q42auyN3YP4c6UxCcgxxuwxxtQArwHzm+wzH3jZWL4CeolIUitl5wMv+d6/BFzZaP1rxphqY8xeIAeYZIw5YYz5DMB3rG+A1DM/5TN3cNc3AISm6sRLSrXE5RJ+cmE6OYfL+fe2AnuDi8D598DxXNj8ir2xexB/kkYKkNvoc55vnT/7tFQ2wRhTAOBb9vE3noj0AuZiXaGcQkRuFpH1IrK+qKiopXPzy/H9m60KD9aeU0q15vJRSQzuE8GS5dl47b7aGHwRpIyHlY+Ax+ZnRnoIf5KGnGZd09+E5vbxp+wZxRORAOBVYIkxZs/pDmCMedoYM8EYMyE+Pr6VcH4ozKTchJLYV4dEV6o1bpfwkwsHs6uwnA8zD9kbvP5q49gB2PyqvbF7CH+SRh7QeJq6VKBpK1dz+7RUttB3Cwvfsv5xztbiPQ1kG2Me9aPu7SKiNJu8wAGISzubKeWPOaOTSYsPd+ZqI/0SSD4HvnhYrzY6gD/fguuAdBEZKCJBWI3US5vssxS4zteLagpw3HfLqaWyS4Hrfe+vB95ptH6hiASLyECsxvW1ACLyGyAa+NmZn2rbGK+XlJq9HNeJl5TyW/3Vxo5DZXycVWhvcBE4/244th+2vGFv7B6g1aRhjKkDbgc+ArYDbxhjMkXkFhG5xbfbB8AerEbrZ4BbWyrrK/MQcImIZAOX+D7j2/4GkAV8CNxmjPGISCrwK6xeWN+IyCYRufFsfwCtKT50gF6UY+KHd3QopbqVuaOTGRhnXW3YPmz6kNmQOBpWPgyeOntjd3MB/uxkjPkAKzE0XvdUo/cGuM3fsr71JcBFzZT5LfDbJuvyOH17R4c6lP0N8UBkP514SakzEeB2cdsFg7nzzc0s236YS0Yk2Be8/mrj9Wtg21swZqF9sbs5vUnfiorcLQAkDxnXyp5KqaauHJtM/9gw/vTJLvvHpBp2BSSMgs//oGNStSNNGq1wF22nmF70jk9yuipKdTkBbhd3zx7G9oJSnlu1197gInDJA3BkD3zxR3tjd2OaNFrRqzyHguA0p6uhVJd1WUYil4xI4E+f7GJfcYW9wQdfBKMXwpePwqFt9sbupjRptMBTV0ffuv1U9NLhQ5RqKxHh/8zPIMjt4pf/3Gp/o/js30NIL1j6E/B67I3dDfnVEN5THdybSV+pxZXYdKgtdTq1tbXk5eVRVaX3j9WpXvhOMkdP1LJh8zbCg8/sqyckJITU1FQCAwPPPHBYDFz2B3j7R/DVkzD19jM/hmqgSaMFxXs20RfoPWCs01XpEvLy8oiMjGTAgAGI2N7RTXVyxhj2FFdQVethcEIkgW7/bnQYYygpKSEvL4+BAwe2LXjGAtj6Jnz6G6uBPKaNx1F6e6olVfnb8BohdYiOOeWPqqoqYmNjNWGo0xIRUnuF4jVw8FjlGZWLjY09uytYEbjiT+AKgPd+BnbfIutGNGm0ILhkBwddiYSGRzpdlS5DE4ZqSXCgm4SoYI5X1nK80v/hy9vl9yo6BS7+X9izAjb9/eyP10Np0mhBXOVuisIGOV0NpbqVuIhgQgPd5B+rsv/ZjQk/gn7nwke/hDKbhzfpJjRpNKPqRDkpnoNUxQxzuipKdSsuEVJ7h+LxGA4dt7nThMsFc5dAbSX8+y57Y3cTmjSakZe9CbcYgpIynK6K8tMdd9zBo48+2vB51qxZ3Hjjt8OT/eIXv+BPf/oTS5cu5aGHrNmC//Wvf5GVldWwz8yZM1m/fn271Od3v/tduxynO7nrrrsYOXIk9/3qXj548yVeevlljlTYPMte/BA4/78h6x34+q/2xu4GtPdUM47utSZeihukjeBt8cC7mWQdLG3XY45IjuJ/545sdvvUqVN58803+dnPfobX66W4uJjS0m/rsHr1ah599FEmT57MvHnzACtpzJkzhxEj2r9b9e9+9zt++ctftvtx69XV1REQ0Ln+C3s8Htxud7Pb//rXv1JUVERwcDBeY9hXXEH+0UqC3EJESBu607bVeT+H/I3w77shOtXqUaX8olcazfAcyqTaBJKSps9odBXTpk1j9erVAGRmZpKRkUFkZCRHjx6lurqa7du3c8455/Diiy9y++23s3r1apYuXcpdd93F2LFj2b17NwBvvvkmkyZNYsiQIaxcuRKweobdcMMNjBo1inPOOYfPPvsMoOFY9ebMmcOKFSu45557qKysZOzYsVxzzTUn1fONN97g5z//OQCPPfYYaWnWiAO7d+/mvPPOA+DBBx9k4sSJZGRkcPPNNzc8EDdz5kx++ctfcv755/PYY48xc+ZM7rjjDmbMmMHw4cNZt24dV111Fenp6fz6178GYN++fWRkfHvF/PDDD3P//fc3HO9nP/sZU6dOJSMjg7Vr1zb7873//vu59tprufDCC0lPT+eZZ54BYMWKFVxwwQUsXryYUaNG4fF4uOuuu5g4cSKjR4/mr3+1/pqfN28eFRUVTJ48mddff50HH3iAt158Erd4mTJlMh8vsybivPfee/nVr37l/z98W7jcsOBZSBkHb/0I8jZ0bLxupHP9mdKJhB3bSV5AXwYFBjldlS6ppSuCjpKcnExAQAAHDhxg9erVnHvuueTn57NmzRqio6MZPXo0QUHf/ntOnTqVefPmMWfOHK6++uqG9XV1daxdu5YPPviABx54gGXLlvHEE08AsHXrVnbs2MGll17Krl27mq3LQw89xOOPP86mTZtO2TZjxgz++EdrLKSVK1cSGxtLfn4+q1atYvr06QDcfvvt3HfffQBce+21vPfee8ydOxeAY8eO8fnnnwPw7rvvEhQUxBdffMFjjz3G/Pnz2bBhAzExMQwaNIg77rij1Z9bRUUFq1ev5osvvuCHP/wh27Y1P9zGli1b+Oqrr6ioqOCcc87hiiusv9DXrl3Ltm3bGDhwIE8//TTR0dGsW7eO6upqpk2bxqWXXsrSpUuJiIho+Jncf//9uEQYnBDN7x59kh/fdB2P/3kJH374IV9//XWr9T5rQWGw6HV49iJ49Xtw4zLoPaDj43ZxeqXRjMiaIo5EDHa6GuoM1V9t1CeNc889t+Hz1KlT/TrGVVddBcD48ePZt28fAKtWreLaa68FYNiwYfTv37/FpNGSxMREysvLKSsrIzc3l8WLF/PFF1+wcuXKhqTx2WefMXnyZEaNGsWnn35KZmZmQ/nvfe97Jx2v/lbbqFGjGDlyJElJSQQHB5OWlkZubm6r9Vm0aBFgJbPS0lKOHTvW7L7z588nNDSUuLg4LrjggoYrk0mTJjU8ePfxxx/z8ssvM3bsWCZPnkxJSQnZ2dnNHjMowMUlU8czZ8H3uPLK+Tzz7HMnJfcOFREP33/bmuHvb1fDiSP2xO3CNGk0Y8CvNzHqP19wuhrqDE2dOpXVq1ezdetWMjIymDJlCmvWrGH16tVMmzbNr2MEBwcD4Ha7qauzJvBpbrykgIAAvN5vu436+wDaueeeywsvvMDQoUOZPn06K1euZM2aNUybNo2qqipuvfVW3nrrLbZu3cpNN9100nHDw8NPW1+Xy9Xwvv5zfbtHS3Vs+gxES89ENLdv4zoZY/jzn//Mpk2b2LRpE3v37uXSSy9t8ecRGhRA/p6dREZFsy3ngL3jU8Wlw6JXrZn+Xv8+1FXbF7sL0qTRDHG5CAmLcLoa6gxNmzaN9957j5iYGNxuNzExMRw7dow1a9Zw7rnnnrJ/ZGQkZWVlrR53xowZ/P3v1gNhu3bt4sCBAwwdOpQBAwawadMmvF4vubm5J7UJBAYGUlt7+jmqZ8yYwcMPP8yMGTMa2kiCg4OJjo5u+FKPi4ujvLyct956qy0/igYJCQkcPnyYkpISqquree+9907a/vrrrwPW1VR0dDTR0dHNHuudd96hqqqKkpISVqxYwcSJE0/ZZ9asWTz55JMN575r1y4qKloe3fYf//gHpceO8v7Hy/nfe+8ka3+BvYmj/1S48knY/yX861bw2vz8SBeibRqqWxk1ahTFxcUsXrz4pHXl5eXExcWdsv/ChQu56aabWLJkSYtfzrfeeiu33HILo0aNIiAggBdffJHg4GCmTZvGwIEDGTVqFBkZGYwb9+1kXTfffDOjR49m3LhxDQmn3vTp08nNzWXGjBm43W769u3LsGHWM0G9evXipptuYtSoUQwYMOC0X8xnIjAwkPvuu4/JkyczcODAhjj1evfuzdSpUyktLeX5559v8ViTJk3iiiuu4MCBA/zP//wPycnJp9ymu/HGG9m3bx/jxo3DGEN8fDz/+te/mj1mcXEx99xzD8uXL6dv377c9J8/5t67fsFf/vocyb1Dcdk1ysCoq+HYAVj+AGBg3uNWu4c6idg+TLHNJkyYYNqr371q2fbt2xk+XOdS70pmzpzJww8/zIQJE1rd9/777yciIoI777yzQ+tkjKGwtJrDZVWEBwXQLzaMQLfLnt8vY+DLx2DZ/ZA0Bha+Yg0/0gOJyAZjzCm/GHp7SinVqYgIidEh9IsJo7LWw+7D5VTW1NkVHM77GSx6DUpy4JkLIHedPbG7CL09pVQPtmLFilPWvfDCCzz22GMnrZs2bVpDt2O79AoLIijAxf6SE+wuqqCmxsYJlIbOtrrgvroQXrzcGnpk7CL74ndientKtRu9PaU6Qq3Hy/6SE+zN2UXmiQj+68J0XC6b2jlOHIE3roN9K2HqT+DiB6wHA3sAvT2llOqSAt0u0uLCCQty8+iybBY+8xWZB4/bEzwsBq79J0y8EVb/GZ67BPJ69h+hmjSUUp2eyyXEhAfx+6tGkV1Yxtw/r+Lef2ylpNyGZyrcgXDFI3DVM3A8z3qC/J+3QGlBx8fuhDRpKKW6jEWT+rHizgv4wdSBvLk+l5kPr+DZlXuoqbPhuYrR34WfbIDz7oBtb8Ofx8PKR6DW5uHdHaZJQynVpUSHBXLf3BF8+LPpnNOvN795fzuzH/uC97cUUNvRkzoFR8LF98NtX8OgC2D5g/DEJNjyRo95klyThuo2Ott8Gt1NdXU1F198MWPHjuX111/nxhtvPOlnZ7fBfSJ56YaJPP+DCRgDt73yDVMf+pQ/frSD3CMnOjZ4TBos/Dtc9w4EhcM/boI/DYePfw3FOR0b22Ha5VZ1jH/fA4e2tu8xE0fBZQ81u7mzzafRmtbmnnBCS3N0bNy4kdra2oZRapsOnOgEEeHCYQmcP6QPn+86zCtfH+DJFbv5y4rdnDc4jmsm9+Oi4QkEujvo7+O0mXDLl7B3Bax/Ab560mowHzAdxv8Ahs+FgOBWDtK16JWG6jY6cj6Nxm699VaWLl0KwHe+8x1++MMfAvDcc881zGFx5ZVXMn78eEaOHMnTTz/dUDYiIqJhSI81a9YQERHB3Xffzfjx47n44otZu3YtM2fOJC0trSFGc3N21B/vF7/4BePGjeOiiy6iqKio2Z9Pc3Nn3H///dx8881ceumlXHfddRQVFbFgwQImTpzIxIkT+fLLLzl8+DDf//732bRpU8PPqv6qbP/+/aSnp1NcXIzX62X69Ol8/PHHbfo3bCu3y0oez14/kVV3X8hPL0on53A5t/ztGyb+dhm3vfINr687wMFjle0f3OWCQRfC9/4f3JEFF91nDUfy9o/gj+nWIIjrn4ej+9s/tgP0SkN1jBauCDpKR86n0diMGTNYuXIl8+bNIz8/n4ICqxfNqlWrWLhwIQDPP/88MTExVFZWMnHiRBYsWEBsbCwVFRVkZGTw4IMPAtZcFjNnzuQPf/gD3/nOd/j1r3/NJ598QlZWFtdff33DFVFzKioqGDduHI888ggPPvggDzzwAI8//niL+59u7owNGzawatUqQkNDWbx4MXfccQfnnXceBw4cYNasWWzfvp1nn32Whx9++JQBD/v378/dd9/NLbfcwuTJkxkxYkSro9p2pOReofzs4iHcfsFgVuws4sPMQ6zMLuL9Lda/0+A+EUxPj+O8wXGMSokmPjK4xZF9z0hkAkz/BUy7A/Z8Cpn/gt2fwfZ3re0xg6wEkzYTks+BqGTrKfQuRJOG6lYaz6fx85//nPz8fFavXk10dPRZzafR2PTp03n00UfJyspixIgRHD16lIKCAtasWcOSJUsAWLJkCf/85z8ByM3NJTs7m9jYWNxuNwsWLGg4VlBQELNnzwasgRWDg4MJDAxk1KhRp43dlMvlarhN9P3vf7+h7s1pbu6MefPmERoaCsCyZctOaqsoLS1tdSTgG2+8kTfffJOnnnrqtBNPOSHA7eLiEQlcPCIBYwy7CstZmV3E57uKeOXrA7zw5T4A4iKCGJ4UxYikKEYkRzE8KYp+MWGEBJ7FrUOXCwZfbL2MgeJs2P2p9dr0d1hnzXpIaIx127X+lZBhtZd04oES/UoaIjIbeAxwA88aYx5qsl182y8HTgA/MMZ801JZEYkBXgcGAPuA7xpjjvq23Qv8CPAA/2WM+ci3fjzwIhAKfAD81HT3R9rVGWk6n0bfvn155JFHiIqKariN1JrTzafRWEpKCkePHuXDDz9kxowZHDlyhDfeeIOIiAgiIyNZsWIFy5YtY82aNYSFhTFz5syG4c5DQkJOascIDAxs+Cu38XwY9XNhwJnN2dHaX8z+zIfh9XpZs2ZNQxLxx4kTJ8jLywOgvLycyMhIv8vaQUQYmhjJ0MRIbpyeRlWth025x9heUErWwVKyCkp54ct91DTqfRUfGUzf3qGk9g6jb4y1TIgKJjY8mNiIIGLDgwkN8iOxiED8EOs15Rarl9XBjVab36Et1nLtM+Bp1PsqvA/06me9eve3lpHJEB4P4bHWMii8+ZgdqNWkISJu4AngEiAPWCciS40xjbtNXAak+16TgSeBya2UvQdYbox5SETu8X2+W0RGAAuBkUAysExEhhhjPL7j3gx8hZU0ZgP/Ptsfguo+pk2bxiOPPEJaWtpJ82lkZmY2zGndmL/zaTR17rnn8uijj/Lpp59SUlLC1Vdf3XCL6/jx4/Tu3ZuwsDB27NjBV199dVbnNGDAAP7yl7/g9XrJz88/ac4Or9fLW2+9xcKFC3nllVca5hhvzuuvv84FF1zQ4twZl156KY8//jh33XUXQEM7RkvuvvturrnmGvr3789NN910yi2sziYk0M2UtFimpMU2rKv1eNldVM72glJyj1SSd/QEuUcq2Zh7lPe3FuDxnvr3aViQm9iIIHqHBRERHEBkSACRIYFEBAcQFRJAREgAoYFuggPdhNa/gtyEBA4hKHk4Qf0WEugWglxeQsv2ElK8nYDS/bhLc3EdP4Ac3Ihsfxe8p5mXJTAMwuOsq5XgSAiJhuAo3/so6/3k/2z3hnh/rjQmATnGmD0AIvIaMB9onDTmAy/7/ur/SkR6iUgS1lVEc2XnAzN95V8CVgB3+9a/ZoypBvaKSA4wSUT2AVHGmDW+Y70MXIkmDdVIR82n0VR9Y+/gwYPp378/R44caZiqdfbs2Tz11FOMHj2aoUOHMmXKlLM6p5bm7AgPDyczM5Px48cTHR3dMKFSc/yZO2PJkiXcdtttjB49mrq6OmbMmMFTTz3V7DE///xz1q1bx5dffonb7ebtt9/mhRde4IYbbmjbCTsk0O1iWGIUwxKjTtlW5/FyqLSKorJqjlTUUFJeQ3FFNSXlNZSUV3Osspbyqjr2FZ+grKqWsuo6yqvrOPP7IJFAhu9lCXIZkt1HSZDjxLrKiJdSYjlOTF0pMaWl9CotI9wUEcEBwjlBuDlBGJW48VI1/iZC2rkRotUBC0XkamC2MeZG3+drgcnGmNsb7fMe8JAxZpXv83KsBDCgubIicswY06vRMY4aY3qLyOPAV8aYv/nWP4eVGPb5YlzsWz8duNsYM+c0db4Z64qEfv36jd+/v3v0WujsdMBC+0VERFBeXu7Xvmcyd0Zn1NV+v7xeQ2Wtx3rVeKiq9VBV621YV1PnpabOS63HWtb4lnVeL3VeQ53H+JbWZ4/v5TX1SyuGxxiMseYh8Zr69V6CvJU8tGgqQQFt6yTb3ICF/uSg090kbZppmtvHn7L+xvP7WMaYp4GnwRrltpV4SinV7lwuITw4gPDg7tXfyJ+zyQP6NvqcChz0c5+gFsoWikiSMabAdyvrcCvHyvO9b6keSvUop7vKuO222/jyyy9PWvfTn/70tHNnKHWm/Eka64B0ERkI5GM1Ui9uss9S4HZfm8Vk4LgvGRS1UHYpcD3wkG/5TqP1r4jIn7AawtOBtcYYj4iUicgU4GvgOuDPbTlp1XGMMe3X5121id2TJdlBO0l2Hq0mDWNMnYjcDnyE1W32eWNMpojc4tv+FFZPpsuBHKwutze0VNZ36IeAN0TkR8AB4D98ZTJF5A2sxvI64DZfzymAH/Ntl9t/o43gnUpISAglJSXExsZq4lDtxhhDSUkJISEhTldFoTP3qXZUW1tLXl5ei88RKNUWISEhpKamEhgY6HRVeoyzaQhXyi+BgYEMHDjQ6WoopTqQDliolFLKb5o0lFJK+U2ThlJKKb91+4ZwX7fftj4SHgcUt2N1ugo9755Fz7tn8fe8+xtj4puu7PZJ42yIyPrT9R7o7vS8exY9757lbM9bb08ppZTymyYNpZRSftOk0bKnW9+lW9Lz7ln0vHuWszpvbdNQSinlN73SUEop5TdNGkoppfymSeM0RGS2iOwUkRzf/OXdlog8LyKHRWRbo3UxIvKJiGT7lr2drGNHEJG+IvKZiGwXkUwR+alvfbc+dxEJEZG1IrLZd94P+NZ36/MGEBG3iGz0zTTaI84ZQET2ichWEdkkIut969p87po0mhARN/AEcBkwAlgkIiOcrVWHehGY3WTdPcByY0w6sNz3ubupA35hjBkOTAFu8/07d/dzrwYuNMaMAcYCs31z1HT38wb4KbC90eeecM71LjDGjG30fEabz12TxqkmATnGmD3GmBrgNWC+w3XqMMaYL4AjTVbPB17yvX8JuNLOOtnBGFNgjPnG974M68skhW5+7sZSP91foO9l6ObnLSKpwBXAs41Wd+tzbkWbz12TxqlSgNxGn/N863qSBGNMAVhfrkAfh+vToURkAHAO1oyQ3f7cfbdpNmFNsfyJMaYnnPejwH8D3kbruvs51zPAxyKyQURu9q1r87nrfBqnOt2Uc9ovuZsSkQjgbeBnxpjSnjDjoG8mzLEi0gv4p4hkOFylDiUic4DDxpgNIjLT4eo4YZox5qCI9AE+EZEdZ3MwvdI4VR7Qt9HnVOCgQ3VxSqGIJAH4locdrk+HEJFArITxd2PMP3yre8S5AxhjjgErsNq0uvN5TwPmicg+rNvNF4rI3+je59zAGHPQtzwM/BPrFnybz12TxqnWAekiMlBEgoCFwFKH62S3pcD1vvfXA+84WJcOIdYlxXPAdmPMnxpt6tbnLiLxvisMRCQUuBjYQTc+b2PMvcaYVGPMAKz/z58aY75PNz7neiISLiKR9e+BS4FtnMW56xPhpyEil2PdA3UDzxtjfutsjTqOiLwKzMQaLrkQ+F/gX8AbQD/gAPAfxpimjeVdmoicB6wEtvLtfe5fYrVrdNtzF5HRWA2fbqw/Gt8wxjwoIrF04/Ou57s9dacxZk5POGcRScO6ugCrOeIVY8xvz+bcNWkopZTym96eUkop5TdNGkoppfymSUMppZTfNGkopZTymyYNpZRSftOkoZRSym+aNJRSSvnt/wcQVi9r+RyPPAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, warmup_t=5, warmup_lr_init=1e-5)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"Without warmup_prefix\");\n", "\n", "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, warmup_t=5, warmup_lr_init=1e-5, warmup_prefix=True)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"With warmup_prefix\");\n", "\n", "plt.legend();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the example above we can see how the `warmup_prefix` updates the LR annealing schedule. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `cycle_limit`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The number of maximum restarts in `TanhLRScheduler`. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### `cycle_limit=1`" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAf1ElEQVR4nO3de3RV533m8e+jI8RF4iaQBEjYCBAkgJ3YVsEJWR23Tgp2LridpIPb1MRJl8etnbiddhp72mknabPqmdXpStw49nIaJ7jThiG9xDQ3x6XJNHFisMjFNmBABhsEGISxuSN0+c0fZ7vREkLagKR9ztHzWUvraO/9vvv8Xi56tPe7z96KCMzMzNIoy7oAMzMrHg4NMzNLzaFhZmapOTTMzCw1h4aZmaVWnnUBw2369OkxZ86crMswMysqW7ZsORIRNX3Xl3xozJkzh5aWlqzLMDMrKpJe7m+9T0+ZmVlqDg0zM0vNoWFmZqk5NMzMLDWHhpmZpZYqNCStlLRDUquke/vZLkkPJNuflXTtYH0lVUt6UtKu5HVqsn6apO9IOinps33e5zpJzyX7ekCSLn3oZmZ2sQYNDUk54EHgJmARcKukRX2a3QQ0JV93AA+l6HsvsDEimoCNyTLAWeC/A7/fTzkPJft/471WphqlmZkNiTSf01gKtEbEbgBJ64BVwLZebVYBj0X+PutPS5oiaSYwZ4C+q4Abkv5rge8CH4+IU8D3Jc3vXUSyv0kR8cNk+THgFuCbFzfkdL701B6OnjrXu4BU/S7USgIlW/PfJ6/62boyibLkVRI5Qa5M5MrKKC8TuTJRnhMVuTLG5MqoKC9j3Jgc48fkGF+RY+K4cirHllNZkcMHYWY2HNKERj2wr9dyG7AsRZv6QfrWRcRBgIg4KKk2RR1t/bzHeSTdQf6IhCuuuGKQ3fbv7zbvZdfhk+Tru6RdZKa8TEyZMIapEyqYXjWWWVPGUz9lHA3VE1hYN5GmuiomVJT85zrNbBik+cnR36+sfX+MXqhNmr5ppd5XRDwCPALQ3Nx8Se/37d/9D5fSrV8R8e/BE28skw+joNe2gJ4IuiOIHuiOoLsn6Imgqyfo6u6hszvo7O6hs7uHjq4eznZ2c7azh9PnujjZ0cXJs10cP9vJ0VOdvHbqHIdPnOUHLx7h0PGz9PT6k2icXsmyxmreNm8ab583nZqJY4dsvGZWutKERhswu9dyA3AgZZuKAfoekjQzOcqYCRxOUUfDIHUUJEl9zm6N/Kmjzu4e2l47w85DJ9j5ygl+2naMrz93kHXP7EOCd8yfzvuva2DF4hmMG5Mb8frMrDikCY1ngCZJjcB+YDXwa33abADuTuYslgHHkjBoH6DvBmANcH/y+vhARST7OyHpemATcBvwVynqN2BMrozG6ZU0Tq9kxeIZAHT3BFsPHONfth3iH360n3vW/YTJ48fw2zfMY83b5zg8zOw8SvOMcEk3A58GcsCjEfEpSXcCRMTDyaWvnyV/NdNp4PaIaLlQ32T9NGA9cAWwF/hARBxNtr0ETCJ/pPI68EsRsU1SM/AlYDz5CfCPxiADaG5uDt+wcHA9PcHTu1/lke/t5rs72qmfMp4/WLmQ971llifVzUYhSVsiovm89WlCo5g5NC7eU61H+NTXt7Pt4HF++Zp6/vxXrvJRh9koc6HQ8CfC7TzL50/nax99B7/3rgV89Sf7ef/DP2D/62eyLsvMCoBDw/pVViY+emMTn/+NZl46cpr3/dX32X7weNZlmVnGHBo2oHcuquOrdy1nTK6MD3/pGQ4dP5t1SWaWIYeGDWp+bRVf+FAzx8508pG1z3D6XFfWJZlZRhwalsriWZP57K9dw7YDx7ln3U/o7intCyjMrH8ODUvtF99Uxx+/ZxFPbjvE57+3O+tyzCwDDg27KB9a3sg731zHAxt3ccBXVJmNOg4Nu2h/8t5FdPcEn/r69qxLMbMR5tCwiza7egJ3/cJ8vv7cQb63qz3rcsxsBDk07JLc8fNzuXLaBP7k8a10dHVnXY6ZjRCHhl2ScWNy/I/3LWb3kVP83aa9WZdjZiPEoWGX7BcW1tJ85VS++NRLvgTXbJRwaNhluX15I3uPnuY7Lwz2OBQzKwUODbssv7S4jpmTx/HFH+zJuhQzGwEODbssY3Jl/MbbruSp1lfZeehE1uWY2TBzaNhlu/XnrmBseRlffOqlrEsxs2Hm0LDLNrWygl++pp5/+nEbr58+l3U5ZjaMHBo2JD60fA5nO3v4Sktb1qWY2TByaNiQeNOMSVzdMJmvPXcw61LMbBg5NGzIrFg8g5/ue52Dx3wjQ7NS5dCwIbNyyQwAvr31UMaVmNlwcWjYkJlXU8X82iq+9fwrWZdiZsPEoWFDauXiGWx+6ShHT/kqKrNS5NCwIbVi8Qy6e4J/2e5TVGalyKFhQ2pJ/STqp4znCZ+iMitJDg0bUpJYsXgG32s9wsmOrqzLMbMh5tCwIbdicR3nunr47g7f+das1Dg0bMg1z6mmurKCjdsdGmalxqFhQy5XJq6fW82m3a8S4YczmZUSh4YNi2WN0zhw7Cxtr/nT4WalxKFhw2LZ3GoANu05mnElZjaUUoWGpJWSdkhqlXRvP9sl6YFk+7OSrh2sr6RqSU9K2pW8Tu217b6k/Q5JK3qtv1XSc8l7fEvS9Esfug2nBbUTmTJhDJt2v5p1KWY2hAYNDUk54EHgJmARcKukRX2a3QQ0JV93AA+l6HsvsDEimoCNyTLJ9tXAYmAl8DlJOUnlwGeAX4iIq4Fngbsvcdw2zMrKxM/NqfaRhlmJSXOksRRojYjdEXEOWAes6tNmFfBY5D0NTJE0c5C+q4C1yfdrgVt6rV8XER0RsQdoTfaj5KtSkoBJwIGLHrGNmGWN1ew9etp3vTUrIWlCox7Y12u5LVmXps1Afesi4iBA8lo70L4iohP4LeA58mGxCPhCfwVLukNSi6SW9vb2FEO04XD93GkAbPbRhlnJSBMa6mdd3+soL9QmTd9U7ydpDPnQuAaYRf701H397SAiHomI5ohorqmpGeTtbLi8eeYkJo4t5+ndDg2zUpEmNNqA2b2WGzj/tNCF2gzU91ByCovk9Y1Pgl2oz1sBIuLFyF/8vx54e4r6LSO5MtE8Zyqb9ngy3KxUpAmNZ4AmSY2SKshPUm/o02YDcFtyFdX1wLHklNNAfTcAa5Lv1wCP91q/WtJYSY3kJ9c3A/uBRZLeOHR4F7D9IsdrI2zZ3Gnsbj9F+4mOrEsxsyFQPliDiOiSdDfwBJADHo2IrZLuTLY/DHwDuJn8pPVp4PaB+ia7vh9YL+kjwF7gA0mfrZLWA9uALuCuiOgGDkj6BPBvkjqBl4EPDcGfgQ2jZY35z2ts3nOUd189M+NqzOxyqdRv89Dc3BwtLS1ZlzFqdXb38JZPfJv3X9fAJ1ctybocM0tJ0paIaO673p8It2E1JlfGWxqm8NN9r2ddipkNAYeGDburGiaz/ZUTdHb3ZF2KmV0mh4YNuyX1kznX1cPOQyeyLsXMLpNDw4bdklmTANi6/3jGlZjZ5XJo2LCbM62SqrHlPLf/WNalmNllcmjYsCsrE4tnTXJomJUAh4aNiCX1k9l+8Dhdngw3K2oODRsRV9VPpqOrh9b2k1mXYmaXwaFhI2JJfX4y/Lk2n6IyK2YODRsRjdOrmFCRY+sBX0FlVswcGjYicp4MNysJDg0bMYtnTWbbgeN095T2/c7MSplDw0bMVfWTOdPZzW5PhpsVLYeGjZirGiYD+BSVWRFzaNiImTu9knFjyhwaZkXMoWEjpjxXxqKZk3wPKrMi5tCwEbVwxiR2Hj5BqT/8y6xUOTRsRC2oq+L1051+ZrhZkXJo2IhaWDcRgJ2HfAWVWTFyaNiIWjAjHxo7/EAms6Lk0LARNb1qLNWVFex8xaFhVowcGjbiFtRVsfOwQ8OsGDk0bMQtrJvIzld8BZVZMXJo2IhrqpvIqXPd7H/9TNalmNlFcmjYiFuYTIbv8hVUZkXHoWEjbkGtr6AyK1YODRtxkyeMoW7SWHY6NMyKjkPDMrGgbqJDw6wIOTQsEwvrJrLr0Ek/kMmsyDg0LBML6ibS0dXDvqOnsy7FzC5CqtCQtFLSDkmtku7tZ7skPZBsf1bStYP1lVQt6UlJu5LXqb223Ze03yFpRa/1FZIekbRT0guS/uOlD92y5NuJmBWnQUNDUg54ELgJWATcKmlRn2Y3AU3J1x3AQyn63gtsjIgmYGOyTLJ9NbAYWAl8LtkPwB8ChyNiQbK//3cJY7YC0FRbBeDbiZgVmTRHGkuB1ojYHRHngHXAqj5tVgGPRd7TwBRJMwfpuwpYm3y/Fril1/p1EdEREXuA1mQ/AB8G/hwgInoi4sjFDdcKReXYchqmjmfnYX9Ww6yYpAmNemBfr+W2ZF2aNgP1rYuIgwDJa+1A+5I0JVn+U0k/kvQVSXX9FSzpDkktklra29tTDNGy8MbtRMyseKQJDfWzru8lLxdqk6Zv2vcrBxqApyLiWuCHwF/0t4OIeCQimiOiuaamZpC3s6zMr61iz5FTvoLKrIikCY02YHav5QbgQMo2A/U9lJzCInk9PMi+XgVOA/+UrP8KcC1WtObVVHGuu4e213wFlVmxSBMazwBNkholVZCfpN7Qp80G4LbkKqrrgWPJKaeB+m4A1iTfrwEe77V+taSxkhrJT65vjvwtUf8ZuCFpdyOw7eKGa4VkXjIZ3up5DbOiUT5Yg4joknQ38ASQAx6NiK2S7ky2Pwx8A7iZ/KT1aeD2gfomu74fWC/pI8Be4ANJn62S1pMPhC7grojoTvp8HPgbSZ8G2t94HytO82oqAXix/SQ3vrnf6SkzKzAq9WcaNDc3R0tLS9Zl2AU0/9mT3PimOv7n+6/OuhQz60XSloho7rvenwi3TM2rqaK13aenzIqFQ8MyNa+2itbDJ/0UP7Mi4dCwTM2rqeLYmU6OnjqXdSlmloJDwzI131dQmRUVh4Zl6mdXUJ3KuBIzS8OhYZmaNXk848fkeNGT4WZFwaFhmSorE3NrKn16yqxIODQsc/NqqnykYVYkHBqWufm1Vex//QxnznUP3tjMMuXQsMzNq6kiAnYf8dGGWaFzaFjm5tX6CiqzYuHQsMzNmVZJmeBFT4abFTyHhmVu3Jgcs6sn+B5UZkXAoWEFYV5NlY80zIqAQ8MKgh/9alYcHBpWEObVVNLR1cP+185kXYqZDcChYQVhXk3+xoX+kJ9ZYXNoWEFwaJgVB4eGFYSplRVMq6zwPajMCpxDwwqG70FlVvgcGlYw3nj0q5kVLoeGFYx5NZW8dtqPfjUrZA4NKxjzaj0ZblboHBpWMObX+HnhZoXOoWEFo37KeMaWl/l2ImYFzKFhBSP/6FdfQWVWyBwaVlDm11b5brdmBcyhYQVlXk0lba+d4WynH/1qVogcGlZQ3nj0654jfoqfWSFyaFhBmV/rK6jMCplDwwpK4/RKJH9Ww6xQpQoNSSsl7ZDUKunefrZL0gPJ9mclXTtYX0nVkp6UtCt5ndpr231J+x2SVvTzfhskPX/xw7VCN25Mjoap43mx3aenzArRoKEhKQc8CNwELAJulbSoT7ObgKbk6w7goRR97wU2RkQTsDFZJtm+GlgMrAQ+l+znjXp+BfCvoSVsfo3vQWVWqNIcaSwFWiNid0ScA9YBq/q0WQU8FnlPA1MkzRyk7ypgbfL9WuCWXuvXRURHROwBWpP9IKkK+C/An138UK1YzKupYnf7ST/61awApQmNemBfr+W2ZF2aNgP1rYuIgwDJa22K9/tT4H8DpwcqWNIdkloktbS3tw/U1ArQgrqJdHT1sO/ogH/NZpaBNKGhftb1/RXwQm3S9E31fpLeCsyPiH8apD8R8UhENEdEc01NzWDNrcAsmDERgJ2HTmRciZn1lSY02oDZvZYbgAMp2wzU91ByCovk9fAg+3obcJ2kl4DvAwskfTdF/VZkmpLLbh0aZoUnTWg8AzRJapRUQX6SekOfNhuA25KrqK4HjiWnnAbquwFYk3y/Bni81/rVksZKaiQ/ub45Ih6KiFkRMQd4B7AzIm64hDFbgascW07D1PHsOOTJcLNCUz5Yg4joknQ38ASQAx6NiK2S7ky2Pwx8A7iZ/KT1aeD2gfomu74fWC/pI8Be4ANJn62S1gPbgC7grojwPSVGmQV1E9nlIw2zgqOI0r5Cpbm5OVpaWrIuwy7S/d98gS98fzfbPrmSMTl/BtVspEnaEhHNfdf7f6MVpIUzqujsDl5+1R/yMyskDg0rSE21+SuodrzieQ2zQuLQsII0v7aKMsEOz2uYFRSHhhWkcWNyzJlW6clwswLj0LCC1VRX5SMNswLj0LCCtbBuIi+/etpP8TMrIA4NK1hNdRPp7gl2+zbpZgXDoWEFa6HvQWVWcBwaVrDmTKtkTE4ODbMC4tCwglVRXkbj9EqHhlkBcWhYQVtQN5GdvnGhWcFwaFhBW1g3kb1HT3OqoyvrUswMh4YVuEWzJgGw7eDxjCsxM3BoWIG7qn4yAM/vP5ZxJWYGDg0rcLWTxlE7cSzPOTTMCoJDwwrekvrJPtIwKxAODSt4S+on03r4JKfPeTLcLGsODSt4V9VPpidg+0F/XsMsaw4NK3hL6vNXUPkUlVn2HBpW8GZMGsf0qgpPhpsVAIeGFTxJngw3KxAODSsKS2ZNZtfhk362hlnGHBpWFJbUT6a7J9juT4abZcqhYUXhqgZ/MtysEDg0rCjMmjyOqRPG8Px+H2mYZcmhYUXhjclwX0Flli2HhhWNq+ons/PQCU+Gm2XIoWFF4+qGKXT1BFsP+GjDLCsODSsaSxurAXh699GMKzEbvRwaVjSqKytYUFfFpj0ODbOspAoNSSsl7ZDUKunefrZL0gPJ9mclXTtYX0nVkp6UtCt5ndpr231J+x2SViTrJkj6uqQXJG2VdP/lDd2K0bLGaWx56Shd3T1Zl2I2Kg0aGpJywIPATcAi4FZJi/o0uwloSr7uAB5K0fdeYGNENAEbk2WS7auBxcBK4HPJfgD+IiLeBFwDLJd006UM2orX0sZqTp3rZusBX3prloU0RxpLgdaI2B0R54B1wKo+bVYBj0Xe08AUSTMH6bsKWJt8vxa4pdf6dRHRERF7gFZgaUScjojvACT7+hHQcPFDtmK2bG5+XmPTnlczrsRsdEoTGvXAvl7Lbcm6NG0G6lsXEQcBktfatO8naQrwXvJHKOeRdIekFkkt7e3tA43NikztxHHMnV7JZs9rmGUiTWion3WRsk2avhf1fpLKgS8DD0TE7v52EBGPRERzRDTX1NQM8nZWbJbNrWbznqN09wz2T8nMhlqa0GgDZvdabgAOpGwzUN9DySksktfDKd/vEWBXRHw6Re1WgpY2VnP8bBcvvOJ5DbORliY0ngGaJDVKqiA/Sb2hT5sNwG3JVVTXA8eSU04D9d0ArEm+XwM83mv9akljJTWSn1zfDCDpz4DJwO9c/FCtVCxrnAbgU1RmGRg0NCKiC7gbeALYDqyPiK2S7pR0Z9LsG8Bu8pPWnwd+e6C+SZ/7gXdJ2gW8K1km2b4e2AZ8C7grIrolNQB/SP4qrB9J+omk37zcPwArPrOmjGd29Xg2+UN+ZiNOEaV9Xri5uTlaWlqyLsOG2O+t/ynf2XGYLX/0TqT+psHM7HJI2hIRzX3X+xPhVpSWza3m6Klz7Dh0IutSzEYVh4YVpRsW1CDBk1sPZV2K2aji0LCiVDtpHNdeMZUntr2SdSlmo4pDw4rWisV1PL//OPuOns66FLNRw6FhRWvF4hkAPLHVRxtmI8WhYUXrymmVvHnmJIeG2QhyaFhRW7G4jpaXX6P9REfWpZiNCg4NK2orl8wgAp7c5quozEaCQ8OK2sK6iVw5bQLf8ikqsxHh0LCiJomVi2fwwxePcOxMZ9blmJU8h4YVvZuumklnd/DPP+1782UzG2oODSt6b2mYzOJZk1j7g5co9XupmWXNoWFFTxK3L29k1+GTPNXqx8CaDSeHhpWE91w9k2mVFXzpB3uyLsWspDk0rCSMG5Pj15ddwcYXDvPyq6eyLsesZDk0rGT8+vVXkpNY+4OXsy7FrGQ5NKxk1E0ax7uvnslXWvZxsqMr63LMSpJDw0rK7csbOdHRxaPf99yG2XBwaFhJeevsKbz7qpk8+J1W3zLdbBg4NKzk/NF73kyuTHzya9uyLsWs5Dg0rOTMnDyej93YxJPbDvGvL/hGhmZDyaFhJenDyxuZV1PJ/9iwjbOd3VmXY1YyHBpWkirKy/jkqiXsPXqaP/3aNt9exGyIODSsZC2fP53//PNz+dtNe/mCr6YyGxLlWRdgNpw+vvJN7D16mk99YztXVE/gl5LnipvZpfGRhpW0sjLxl7/6Vq5umMI9637ClpePZl2SWVFzaFjJG1+R4/O3XUftpLHc+sgmvtKyL+uSzIqWQ8NGhdqJ4/jqby+nec5U/uvfP8sn/nkrXd09WZdlVnQcGjZqTK2s4LEPL+XDyxv54lMv8d7PPsX3dx3JuiyzouLQsFGlPFfGH793EQ9/8FpOdnTywS9sYs2jm/nx3td8Wa5ZCir1/yjNzc3R0tKSdRlWgDq6uvmbH77MAxt3cfxsF/Nrq3j/dQ28+6qZzK6ekHV5ZpmStCUims9bnyY0JK0EPgPkgL+OiPv7bFey/WbgNPChiPjRQH0lVQP/F5gDvAT8akS8lmy7D/gI0A18LCKeSNZfB3wJGA98A7gnBhmAQ8MGc+JsJ19/9iBf2dLGlpdfA2B29XjeNncazVdWs3DGRJrqqphQ4SvUbfS45NCQlAN2Au8C2oBngFsjYluvNjcDHyUfGsuAz0TEsoH6SvpfwNGIuF/SvcDUiPi4pEXAl4GlwCzgX4AFEdEtaTNwD/A0+dB4ICK+OVD9Dg27GHuOnOK7Ow7zwxdfZdOeoxw70/nv22ZNHsesKeOZNWU8MyaPY+qECqorxzBlQgVVY8upGltO5dgcY8tzjBuTY+yYMipy+a+yMmU4KrOLd6HQSPOr01KgNSJ2JztaB6wCet9CdBXwWPJb/9OSpkiaSf4o4kJ9VwE3JP3XAt8FPp6sXxcRHcAeSa3AUkkvAZMi4ofJvh4DbgEGDA2zi9E4vZLG6Y3cvryRnp5g79HT7Dh0gp2vnGDPkVMcOHaGn7a9zhNbz9LRlf7qq1yZyJWJ8uQ1HyxljC0vo0wOFBseX/vYOxhbnhvSfaYJjXqg94XtbeSPJgZrUz9I37qIOAgQEQcl1fba19P97Ksz+b7v+vNIugO4A+CKK64YYGhmF1ZWJuZMr2TO9EpW9PNJ8jPnujl6+hyvnTrHqY4uTp3r4mRHN2c7u+no7OZsZw+dPT2c6+qhs7uHrp6gpyfo7A46unro6Oqmo7OHoLTnFS07Yuh/IUkTGv29a99/5Rdqk6Zv2vdLva+IeAR4BPKnpwZ5P7NLMr4iR33FeOqnjM+6FLMRk+aS2zZgdq/lBuBAyjYD9T2UnMIieT2cYl8Ng9RhZmbDKE1oPAM0SWqUVAGsBjb0abMBuE151wPHklNPA/XdAKxJvl8DPN5r/WpJYyU1Ak3A5mR/JyRdn1ytdVuvPmZmNgIGPT0VEV2S7gaeIH/Z7KMRsVXSncn2h8lfyXQz0Er+ktvbB+qb7Pp+YL2kjwB7gQ8kfbZKWk9+srwLuCsi3niKzm/xs0tuv4knwc3MRpQ/3GdmZue50CW3vo2ImZml5tAwM7PUHBpmZpaaQ8PMzFIr+YlwSe3Ay5fYfTow2h64MBrHDKNz3KNxzDA6x30pY74yImr6riz50Lgcklr6u3qglI3GMcPoHPdoHDOMznEP5Zh9esrMzFJzaJiZWWoOjYE9knUBGRiNY4bROe7ROGYYneMesjF7TsPMzFLzkYaZmaXm0DAzs9QcGv2QtFLSDkmtyfPLS5Kk2ZK+I2m7pK2S7knWV0t6UtKu5HVq1rUONUk5ST+W9LVkeTSMeYqkv5f0QvJ3/rZSH7ek303+bT8v6cuSxpXimCU9KumwpOd7rbvgOCXdl/x82yFpxcW8l0OjD0k54EHgJmARcKukRdlWNWy6gN+LiDcD1wN3JWO9F9gYEU3AxmS51NwDbO+1PBrG/BngWxHxJuAt5MdfsuOWVA98DGiOiCXkH8+wmtIc85eAlX3W9TvO5P/4amBx0udzyc+9VBwa51sKtEbE7og4B6wDVmVc07CIiIMR8aPk+xPkf4jUkx/v2qTZWuCWTAocJpIagHcDf91rdamPeRLw88AXACLiXES8TomPm/wzg8ZLKgcmkH/aZ8mNOSL+DTjaZ/WFxrkKWBcRHRGxh/xzkJamfS+HxvnqgX29ltuSdSVN0hzgGmATUJc8KZHktTbD0obDp4E/AHp6rSv1Mc8F2oEvJqfl/lpSJSU87ojYD/wF+Ye8HST/RNFvU8Jj7uNC47ysn3EOjfOpn3UlfV2ypCrgH4DfiYjjWdcznCS9BzgcEVuyrmWElQPXAg9FxDXAKUrjtMwFJefwVwGNwCygUtIHs62qIFzWzziHxvnagNm9lhvIH9KWJEljyAfG30bEPyarD0mamWyfCRzOqr5hsBx4n6SXyJ96/EVJ/4fSHjPk/123RcSmZPnvyYdIKY/7ncCeiGiPiE7gH4G3U9pj7u1C47ysn3EOjfM9AzRJapRUQX7CaEPGNQ0LSSJ/jnt7RPxlr00bgDXJ92uAx0e6tuESEfdFRENEzCH/d/uvEfFBSnjMABHxCrBP0sJk1Y3ANkp73HuB6yVNSP6t30h+3q6Ux9zbhca5AVgtaaykRqAJ2Jx2p/5EeD8k3Uz+vHcOeDQiPpVtRcND0juA7wHP8bPz+/+N/LzGeuAK8v/xPhARfSfZip6kG4Dfj4j3SJpGiY9Z0lvJT/5XALuB28n/4liy45b0CeA/kb9S8MfAbwJVlNiYJX0ZuIH8LdAPAX8CfJULjFPSHwIfJv/n8jsR8c3U7+XQMDOztHx6yszMUnNomJlZag4NMzNLzaFhZmapOTTMzCw1h4aZmaXm0DAzs9T+P7h4GCHz+y7SAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, cycle_limit=1)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*2)\n", "\n", "plt.plot([i for i in range(num_epoch*2)], lr_per_epoch);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### `cycle_limit=2`" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuU0lEQVR4nO3de5RcVZ3o8e+vqvpdnU76nXQn9CPdCZ2OCDRJAJeiCElADerohLkOAZ2LzERH5ylclzoPmWHN9XoV5bFQEdAZc1EZyQwIakZGEfJEhDxI0um8Oo9OJ52kO93pR1X97h91EtqmHyfprjpVp36ftbKq6tTZVXvDrv6dvffvnCOqijHGGONGwOsKGGOMSR8WNIwxxrhmQcMYY4xrFjSMMca4ZkHDGGOMayGvK5BopaWlWlNT43U1jDEmrWzZsuW4qpaN3O77oFFTU8PmzZu9roYxxqQVEdk/2nabnjLGGOOaBQ1jjDGuWdAwxhjjmgUNY4wxrlnQMMYY45qroCEiy0Rkp4i0isjdo7wvInK/8/5rInLFRGVFpFhEfi4iu53HGc72EhH5pYicEZFvjvieK0Xkdeez7hcRufimG2OMuVATBg0RCQIPAMuBJuBWEWkasdtyoMH5dyfwkIuydwPrVLUBWOe8BugHvgD89SjVecj5/HPftcxVK40xxkwJN+dpLAJaVbUNQETWACuA7cP2WQE8ofHrrK8XkekiMhOoGafsCuA6p/zjwAvA51S1F3hRROYOr4TzedNU9WXn9RPALcBPL6zJ7jz2m7109Q4Or4CrcmPtJQLivBt/7jzKm9sCIgScRxEhKBAMCMFAgFBACAaEUFDIDgbICgbIDgXIzQqSlxUkLztIYW6IgpwQBdlBbBCW+lSV76/fT2fPwJsbXfx/G2+PifrZ7/ex+GMwIAQC8mYfCwhZ5/uYkBMKkpsVICcUJJwTIpwbIpwTIicUsH6WgdwEjSrg4LDX7cBiF/tUTVC2QlWPAKjqEREpd1GP9lG+4y1E5E7iIxLmzJkzwceO7t82HmD3sTPE63dRH+GZUECYnp/FjPxsSsM5zJqeR9X0XKqL85lXUUhDRZj8bN+f15nyjnb384WntwHxP+zp1s9yQgGKC7KZkZ9N+bRz/SyPmpIC5lWGuaSkgKygLZv6jZu/HKMdSozs3mPt46asW64/S1UfAR4BaGlpuajv+9lfvOtiio1KVc//QdBzr4n/kVCGvacQUyWqisYgqko0psRUicSUSDTGUFQZisYYisYYiMToH4rSPxSjbzDCmYEIZ/ojdPcP0dU7xMneQY719PPSnuN0dPcTG/Zfora0gMW1xVxdX8I19aWUFeZMWXuNO4ORGABf/ehlfOiK6kl/npt+du55TCEaU1TjzyOxGLFY/DESVSKxGIMRZTAaY2AoSn8kxtnB6Pl+1tMfofvsEF29g3T1DnK0u5/X2k//3ug8Oxjg0pmFXF1fytX1JSyqKSYvOzjpdhpvuQka7cDsYa+rgcMu98kep2yHiMx0RhkzgWMu6jH8lzVaPVKSOFMBw7YkvQ5D0RjtJ8+yq6OHXUd7+F37KZ55/QhrNh1EBN4xt5Q/uLKapQsqyc2yH3YyRJwoHgxMTX9IhX7WNxihrbOXXR097Ozo4ZX9J/n2r9t4+L/3kJ8d5OaFM/mDK6tZVFtsU1tpyk3Q2AQ0iEgtcAhYCfzRiH3WAp9y1iwWA6edYNA5Ttm1wCrgPufx6fEq4Xxej4gsATYAtwHfcFF/A2QFA9SWFlBbWsDSBZVA/Ehz2+HT/Hx7B0+9cojPrHmVorwsVr+7ntuurrHgkWDRKQ4aqSA/O0RzVRHNVUXnt/UORNi0r4tnXz/CM68d4Ydb2mkoD3P38vm8Z365BY80I27uES4iNwFfA4LAo6p6r4jcBaCqDzupr98kns3UB9yhqpvHKutsLwGeBOYAB4CPqGqX894+YBrxkcop4EZV3S4iLcBjQB7xBfBP6wQNaGlpUbtg4cRiMWV92wke+XUbL+zspHpGHn+7bD7vf9tM+1EnyPbD3dx0/695+GNXsKx5ptfVSYq+wQjPvn6UB3/ZStvxXq6uK+HzN1/6e0HGpAYR2aKqLW/Z7iZopDMLGhfuN63HufeZHWw/0s0HL6/inz+00EYdCbD10Gne940X+dZtLdzQVOF1dZJqKBrjBxsP8LVf7Kb77BBfen8TH1tyiR2gpJCxgoalNpi3uHZuKf/56XfwVzc08pNXD/GRh1/m8KmzXlfLd86taYR8ND3lVlYwwG1X1/DLv76OdzaW8YWnt3HPU68zEIl6XTUzAQsaZlSBgPDp6xv41h+3sPd4Lx/45ovsONLtdbV8JRqLZ0/5aU3jQhXlZfGt21pY/e561mw6yMe+vYHegYjX1TLjsKBhxvXepgp+svpaQoEAn3hsE8e6+72ukm9Eopk70hguGBD+Zul87r/1crbsP8ln1rx6PknApB4LGmZCc8vDfOf2Fk6dHeITj2+mb9COBKeCH7OnJuMDl83iS+9fwC92dPBPz+7wujpmDBY0jCsLZhXxjVsvZ9vh03zWjgSnxPk1jaAFjXNWXVPD7dfU8J0X9/K99aPebdR4zIKGce36Syv4wvua+Nn2Dr796zavq5P23hxp2M9wuC+8r4nr55fzd2u3sfXQaa+rY0aw3mouyB3X1vLeS8v5+rrdHDltGVWTkcnZU+MJBoSvfvTtTM/L4otPbyVmo9qUYkHDXLAvvX8B0Zjy5Wds3nkyLHtqbEX5Wdy9fD6vHDjFj15pn7iASRoLGuaCzS7O58+um8szrx3hxd3Hva5O2rKRxvg+fEU1V14yg/t++gan+4a8ro5xWNAwF+WT76pjTnE+X1y79fzVWs2Fseyp8QUCwj+sWMCpvkG+8rOdXlfHOCxomIuSmxXk7z7QRFtnL/+2wbJcLsab52nYz3AsC2YVcdvVNXx/w35anfvbGG9ZbzUX7T3zK2i5ZAbffWmfpeBehPMjDUu5Hden3zOXrGCAx17a63VVDBY0zCTdfm0N+0/08cLOiW6HYkayNQ13SsI5rLhsFj/ecojTZ21tw2sWNMykLF1QSeW0XL77m31eVyXtWPaUe7dfW8PZoShPbjo48c4moSxomEnJCgb446sv4cXW4+zu6PG6OmnFRhruLZhVxKLaYh5/2aZCvWZBw0zarYvmkBMK8N2X9nldlbRi2VMX5o5ramg/eZZf7OjwuioZzYKGmbTigmxueXsVT73Sbvn0F+DNkYb9DN24oamCqul5PGZToZ6y3mqmxKpraugfivHDLTbn7JaNNC5MyJkKfbntBLtsKtQzFjTMlGiaNY2FVUX852tHvK5K2rD7aVy4D19RjQg8Y/3MMxY0zJRZ1lzJqwdPcfS03ajJjWgshkj8zGfjTllhDlddUszz2456XZWMZUHDTJmlCyoB+Nl2+0G7EYmpjTIuwtLmSt442sO+471eVyUjWdAwU2ZueZj6sgKe22pBw41oTG094yLc2FQBYKMNj1jQMFNqWXMlG/Z2cbJ30OuqpLz4SMN+ghdqdnE+zVXTLGh4xHqsmVJLF1QSjanl0rtgI42Lt7SpklcOnKKj29bPks2ChplSC6uKmFWUy/PbLGhMJBKL2ZrGRVrWfG79zPpZslnQMFNKRLhxQSW/2t1J70DE6+qkNBtpXLy55WHqSgt43tbPks6Chplyy5orGYzEeGFnp9dVSWmRqGVPXSwRYWlzJS+3neBUn62fJZMFDTPlrqopZkZ+FutsXWNc0ZjavTQm4camCqIx5b932cFJMlnQMFMuGBCW1JWwYW+X11VJaZY9NTlvq55OYU6I9W3Wz5LJeqxJiMW1xRw6dZb2k31eVyVl2ZrG5AQDQkvNDDbuPeF1VTKKBQ2TEIvrSgDYYEeBY7LsqclbXFfCns5eOnsGvK5KxnAVNERkmYjsFJFWEbl7lPdFRO533n9NRK6YqKyIFIvIz0Vkt/M4Y9h79zj77xSRpcO23yoirzvf8ZyIlF58000izasopCgviw12FDgmG2lM3qLaYgA22lRo0kwYNEQkCDwALAeagFtFpGnEbsuBBuffncBDLsreDaxT1QZgnfMa5/2VwAJgGfCgiARFJAR8HXi3qr4NeA341EW22yRYICBcVVNsP+Zx2LWnJm9hVRH52UGbokoiNyONRUCrqrap6iCwBlgxYp8VwBMatx6YLiIzJyi7Anjcef44cMuw7WtUdUBV9wKtzueI869ARASYBhy+4BabpFlSV8y+E3121u4YbKQxeVnBAFdeMsOSLpLITdCoAobfWafd2eZmn/HKVqjqEQDnsXy8z1LVIeBPgdeJB4sm4DujVVhE7hSRzSKyubPT0vG8srg2vq6xvs2OAkcTP0/DlhUna3FtMW8c7bHrnSWJmx472qHQyDu7j7WPm7Kuvk9EsogHjcuBWcSnp+4Z7QNU9RFVbVHVlrKysgm+ziTKpTMLCeeEbIpqDDbSmBqLnIOTTfusnyWDm6DRDswe9rqat04LjbXPeGU7nCksnMdjE3zW2wFUdY+qKvAkcI2L+huPhIIBWmps6mAskViMkJ3cN2mXzS4iJxSwfpYkboLGJqBBRGpFJJv4IvXaEfusBW5zsqiWAKedKafxyq4FVjnPVwFPD9u+UkRyRKSW+OL6RuAQ0CQi54YONwA7LrC9JskW15bQeuwMx89YSuRINtKYGjmhIJfPmW6ZekkyYdBQ1QjxLKXnif+RflJVt4nIXSJyl7Pbs0Ab8UXrbwF/Nl5Zp8x9wA0ispt4ALjPKbON+ChiO/AcsFpVo6p6GPh74Fci8hrxkcc/Ta75JtEW11lK5Fgse2rqLK4tYfvhbrr7h7yuiu+F3Oykqs8SDwzDtz087LkCq92WdbafAK4fo8y9wL2jbH8YePitJUyqWlhVRF5WkA1tJ7hp4Uyvq5NSbKQxdRbXFvN1hc37unjP/Aqvq+NrlrphEiorGOCy2UW82n7a66qkHLv21NS5fM4MAgKvHrR+lmjWY03CLawqYseRboaiMa+rklJspDF18rKDzC0Ps/WQBY1Es6BhEq65qojBSIzdHWe8rkpKsWtPTa3mWUW8bkEj4SxomIRrrioCYOth+0EPF43aSGMqNVcV0dkzwDG7AkFCWdAwCVdbUkBBdtCmDkaIxNTO05hCC6vjByc22kgsCxom4QIBYcGsIgsaI9iaxtRqmjkNEdh6qNvrqviaBQ2TFM1VRWw/0k3EFsPPs+ypqVWQE6KutMBGGglmPdYkxcLqafQPxdjT2et1VVKGjTSm3sIqG9EmmgUNkxTNs5zFcPtBn2fZU1OvuaqIo939die/BLKgYZKirixMXlbQpg6GicaUgAWNKWWZeolnQcMkRTAgNM2aZiONYezaU1OvadY0ALbaFQgSxoKGSZqFzmJ4NDbRLVX8LxZTVLE1jSk2LTeL2tICG2kkkAUNkzTNVUX0DUbZe9zODI9qPHDaSGPqNVcVWdptAlnQMEnTXBWfOrB1Dc6PtoKWcjvlmmdN49Cps3TZ7V8TwnqsSZq5ZWFyswJ2FEh8PQNspJEIC6ssUy+RLGiYpAkFA8yvtMVwiF93CmxNIxEWWAZVQlnQMEl16cxCdnX0oJrZi+GRWPzMeLv21NQrystiVlEuu472eF0VX7KgYZKqobyQk31DHD+T2fPNb65pWNBIhIaKQnbZpfgTwoKGSap5lYUA7OrI7KNAW9NIrHmVhbR2nrFrnSWABQ2TVA0VYQB2ZvjUgWVPJVZDeZjBSIz9XX1eV8V3rMeapCoL5zAjP4vdxzI7aNhII7HOjWh3Z/iINhEsaJikEhEaKwptpOEshNuaRmLMLQ8jAjuP2rrGVLOgYZKusaKQ3R1nMjqDykYaiZWfHWL2jHx2ZfiINhEsaJika6wspGcgwpHTmXsv54idp5FwjRWFlnabABY0TNLNq4jPN+/M4Pnmcwvhdp5G4syrDLP3eC+DEcugmkoWNEzSNToZVJm8SBmx7KmEa6woJBJT9h63u0VOJeuxJumm52dTXpiT0YuUUVvTSLhGG9EmhAUN44nGisKMPsEvYtlTCVdXVkAwILauMcUsaBhPNFYUsvtYD7EMvSGTjTQSLycUpKYkP6MPThLBgobxxLzKMP1DMQ6ezMwzdiN27amkmFeZ2SPaRHAVNERkmYjsFJFWEbl7lPdFRO533n9NRK6YqKyIFIvIz0Vkt/M4Y9h79zj77xSRpcO2Z4vIIyKyS0TeEJEPX3zTjZcaKs5dgyoz1zXOXRo9ZAvhCdVQXsj+rj76h6JeV8U3JuyxIhIEHgCWA03ArSLSNGK35UCD8+9O4CEXZe8G1qlqA7DOeY3z/kpgAbAMeND5HIDPA8dUtdH5vP++iDabFNBQHs+gytSjQBtpJMe8ykJUofVYZh6cJIKbw5xFQKuqtqnqILAGWDFinxXAExq3HpguIjMnKLsCeNx5/jhwy7Dta1R1QFX3Aq3O5wB8HPhnAFWNqerxC2uuSRWFuVlUTc/L2MuJ2HkayXE+gypD+1kiuAkaVcDBYa/bnW1u9hmvbIWqHgFwHsvH+ywRme68/kcReUVEfigiFaNVWETuFJHNIrK5s7PTRRONFxorwhk80rDsqWSoKcknOxiwy4lMITdBY7RePTLlZax93JR1+30hoBr4japeAbwMfGW0D1DVR1S1RVVbysrKJvg645W55fEzdqMZmEFl2VPJEQoGqCnNZ88xO8FvqrgJGu3A7GGvq4HDLvcZr2yHM4WF83hsgs86AfQB/+5s/yFwBSZtzS0PMxCJcejkWa+rknS2ppE8c8vD7Om0NY2p4iZobAIaRKRWRLKJL1KvHbHPWuA2J4tqCXDamXIar+xaYJXzfBXw9LDtK0UkR0RqiS+ub9T4JVH/A7jO2e96YPuFNdekkvqy+GJ4Jv6g3xxpWPZUotWXhTnQ1cdAxDKopkJooh1UNSIinwKeB4LAo6q6TUTuct5/GHgWuIn4onUfcMd4ZZ2Pvg94UkQ+ARwAPuKU2SYiTxIPCBFgtaqe+7/9OeB7IvI1oPPc95j0NDxovHt++QR7+4uNNJKnvixMNKYcONF3PtXbXLwJgwaAqj5LPDAM3/bwsOcKrHZb1tl+gvhoYbQy9wL3jrJ9P/BON3U2qW9GQTYlBdkZmQ4Zde5dbWsaiTfXSe9uPXbGgsYUsLGx8VR9WWbON58faVjKbcLVlhYAmTkNmggWNIyn6ssL2NOZeZktlj2VPAU5IWYV5WZkP0sECxrGU/VlYbp6B+nqHfS6KkllaxrJVV8ezshp0ESwoGE8VV+emRlUlj2VXOemQTP5vvRTxXqs8dTccxlUGXYUeG6kYQON5KgvD9M3GOVod+bel36qWNAwnqqankdOKJBxUwfRWIxQQBCxqJEM5w5OMq2fJYIFDeOpQECoy8AMqkhMbT0jierLnQwqCxqTZkHDeK6+LPMyqKJRtcypJCoL51CYG8q4fpYIFjSM5+aWhzl4MrNulGMjjeQSEbsG1RSxoGE8V18WRhX2Hs+co8BoTAkF7eeXTPVllnY7FazXGs9l4oULbaSRfPVlYY71DNDdP+R1VdKaBQ3jubqyAkTIqHsenMueMslz7hpUbbauMSkWNIzncrOCVM/Io9VGGiaB6sviGVQ2RTU5FjRMSqgvC2dUOmQ0ZtlTyTanOJ+soGTUNGgiWNAwKWFuWZi242eIZcitX22kkXyhYICakgIbaUySBQ2TEurLw/QPxTh0KjNu/Ro/T8N+fsmWqZfin0rWa01KyLQMKhtpeKO+vIADJ/oYcm6CZS6cBQ2TEuaev9ptZmS2RGMxQnYDpqSbWx4mElP2n+jzuippy4KGSQnFBdnMyM/KmPlmG2l4o94uXDhpFjRMysik+WbLnvJGXYZNgyaCBQ2TMurLwrRlyI/ZRhreCOeEqJyWa0FjEixomJQxtzzM8TODnOrz/61f4yMN+/l5YW55Zp0TNNWs15qUcf6eBxlwFBiJKQEbaXji3KX47davF8eChkkZ59NuM+AaVHbtKe/Ul4c5MxDhWM+A11VJSxY0TMqonpFPdiiQEdegikRtTcMrduvXybGgYVJGMCDUlRZkxHyzZU95p77cMqgmw4KGSSmZknYbVRtpeKW8MIdwTigjDk4SwYKGSSn15WEOdPn/1q820vCOiFBfHs6IadBEsKBhUkp9WQExxfeXeYivadjPzyv1ZQUZkXCRCNZrTUrJlAsX2kjDW/VlYY5293NmIOJ1VdKOBQ2TUjLl2kCRmBK0CxZ65vwFMn3ezxLBVdAQkWUislNEWkXk7lHeFxG533n/NRG5YqKyIlIsIj8Xkd3O44xh793j7L9TRJaO8n1rRWTrhTfXpLq87CBV0/MyYKRh52l4KVNGtIkwYdAQkSDwALAcaAJuFZGmEbstBxqcf3cCD7koezewTlUbgHXOa5z3VwILgGXAg87nnKvPhwD7P+1j9eX+z6Cya09565KSfEIBu/XrxXAz0lgEtKpqm6oOAmuAFSP2WQE8oXHrgekiMnOCsiuAx53njwO3DNu+RlUHVHUv0Op8DiISBv4S+PKFN9Wki3OLlH6+9autaXgrKxhgTkk+uzssaFwoN0GjCjg47HW7s83NPuOVrVDVIwDOY7mL7/tH4P8A46bWiMidIrJZRDZ3dnaOt6tJQY0VhZwdivr61q/xkYYtKXqpsbzQ92tnieCm1452ODTyEHCsfdyUdfV9IvJ2YK6q/vsE5VHVR1S1RVVbysrKJtrdpJjGikIAdh7t8bgmiWMjDe81Vhay70Sv788JmmpugkY7MHvY62rgsMt9xivb4Uxh4Twem+CzrgauFJF9wItAo4i84KL+Js00VMQXKXcd82fQUFWitqbhucaKMDG1xfAL5SZobAIaRKRWRLKJL1KvHbHPWuA2J4tqCXDamXIar+xaYJXzfBXw9LDtK0UkR0RqiS+ub1TVh1R1lqrWAO8AdqnqdRfRZpPipuVmMasol10+HWlEnbUaG2l4a54zot3V4c9+liihiXZQ1YiIfAp4HggCj6rqNhG5y3n/YeBZ4Cbii9Z9wB3jlXU++j7gSRH5BHAA+IhTZpuIPAlsByLAalW18WOGaawsZKdPFykjTtCw8zS8VVNaQFZQ2HnUn/0sUSYMGgCq+izxwDB828PDniuw2m1ZZ/sJ4PoxytwL3DtOffYBzS6qbtJUY0UhL+05QSQaIxT014KxjTRSQ1YwQF1pmN020rgg/vo1Gt9orChkMBJjf5f/rkF1fqRh2VOei49oLWhcCOu1JiU1OovhfjwKtJFG6mgsD9N+8iy9dg0q1yxomJQ0tzyMCL6cb47EYgCWPZUCGivji+G77XwN1yxomJSUnx1iTnG+L9NubaSROiyD6sJZ0DApq6G80Jdpt5HouTUNCxpem12cT04o4Mt+ligWNEzKmlcZZu/xXgYjMa+rMqXOjzQs5dZzwYDQUBFml01PuWZBw6SsxopCIjFl73F/3WHNsqdSS2OFP0e0iWK91qSs89eg8tl8s61ppJbGikKOdvdz+uyQ11VJCxY0TMqqKysgGBDfpd1a9lRqObcY7rd+ligWNEzKygkFqSnJ993Vbm2kkVrOXSDTbyPaRLGgYVLavMpC3+XQv7mmYUEjFVRNz6MgO2g3ZHLJgoZJaY0V8XsenB30zzUr3xxp2M8vFYgIjZWFvHG02+uqpAXrtSalNc2chirs8NEP2s7TSD1NM6ex/XA38WuvmvFY0DAprbmqCICth057XJOpY+dppJ7mqiK6+yMc8OEFMqeaBQ2T0mYW5VJSkM3r7f4JGpY9lXoWOgcnr/vo4CRRLGiYlCYiNFcVsfWwf6anLHsq9TRWFJIVFLYe8k8/SxQLGiblNVdNY3dHD/1D/lgMt+yp1JMdCjCvstBX06CJYkHDpLyFVUVEYsobPjlfw7KnUtPCqiJeP3TaFsMnYL3WpDy/LYbbSCM1NVcVcfrsEO0nz3pdlZRmQcOkvKrpeUzPz/JN0Ig6C+G2ppFammf56+AkUSxomJQnIuenDvzAztNITfMqCwkFxDf9LFEsaJi00FxVxK6OHgYi6b8YbudppKbcrCCNFYW+ytRLBAsaJi00zypiKKrs8sE9w21NI3U1V01jqy2Gj8uChkkLfjr5yrKnUtfCqiK6egc5fLrf66qkLOu1Ji3MLs5jWm6IrYfTP2jYSCN1LfBZpl4iWNAwaeH8meE++DFb9lTqapo5jWBAfNHPEsWChkkbC6uKeONID4ORmNdVmRQbaaSu3KwgDeVhX0yDJooFDZM2FlYXMRiNsf1Ieme3RKN27alUtrCqiFcPniIWs8Xw0VjQMGljUW0xABv3nvC4JpNjI43Utqi2mFN9Q767Y+RUsaBh0kZ5YS51pQVsaOvyuiqTEo0pwYAgYkEjFS2pKwFgQ5ofnCSKq6AhIstEZKeItIrI3aO8LyJyv/P+ayJyxURlRaRYRH4uIrudxxnD3rvH2X+niCx1tuWLyDMi8oaIbBOR+ybXdJOOFtcVs3Ff1/m01XQUcYKGSU3VM/KYVZTLhr3pfXCSKBMGDREJAg8Ay4Em4FYRaRqx23Kgwfl3J/CQi7J3A+tUtQFY57zGeX8lsABYBjzofA7AV1R1PnA5cK2ILL+YRpv0tai2mJ7+SFrfzzkai9l6RgoTERbVFrOhrctO8huFm5HGIqBVVdtUdRBYA6wYsc8K4AmNWw9MF5GZE5RdATzuPH8cuGXY9jWqOqCqe4FWYJGq9qnqLwGcz3oFqL7wJpt0trjWmTpI4ykqG2mkvsV1JRw/M0Db8V6vq5Jy3ASNKuDgsNftzjY3+4xXtkJVjwA4j+Vuv09EpgPvJz5CeQsRuVNENovI5s7OzvHaZtLMrOl5zC7OY2MaTx1EY2ojjRS3+HzSRfr2s0RxEzRG690jx2xj7eOm7AV9n4iEgB8A96tq22gfoKqPqGqLqraUlZVN8HUm3SyqKWHjvvSdOoiPNCwHJZXVlhZQGs5hQ5stho/kpue2A7OHva4GDrvcZ7yyHc4UFs7jMZff9wiwW1W/5qLuxocW1xXT1TuYtimR0agStJiR0kSExXXFbNibvgcnieKm624CGkSkVkSyiS9Srx2xz1rgNieLaglw2plyGq/sWmCV83wV8PSw7StFJEdEaokvrm8EEJEvA0XAZy+8qcYvlpxb10jTqYNITO1ihWlgSW0xR0732538Rpiw56pqBPgU8DywA3hSVbeJyF0icpez27NAG/FF628BfzZeWafMfcANIrIbuMF5jfP+k8B24DlgtapGRaQa+DzxLKxXRORVEfmTyf4HMOlndnEeM4ty03bqIBqL2UJ4GljsnK+xPk37WaKE3Oykqs8SDwzDtz087LkCq92WdbafAK4fo8y9wL0jtrUz+nqHyTDnUiJf2nMCVU27k+QithCeFuaWhZmRn8WGvV18pGX2xAUyhI2RTVpaXFtCZ096pkTG1FJu00EgED84Wd92wtY1hrGgYdLSOxtLAfjF9g6Pa3LhIlELGuniXY3ltJ88y86OHq+rkjIsaJi0VD0jn4VVRTy37ajXVblg0Zja/cHTxA1NFYjAc1vTr58ligUNk7aWNVfy2wOn6OhOr1tz2nka6aOsMIerLinm+W3pN6JNFOu5Jm0tXVABwM/SbLRhZ4SnlxsXVLDjSDf7T6Tf+lkiWNAwaWtueSH1ZQVpN0UVsZTbtLJ0QSUAz6dZP0sUCxomrS1dUMn6ti5O9Q16XRXXbKSRXmYX57Ng1jSbonJY0DBpbVlzJdGY8osdxybeOUXYVW7Tz7IFlWzZf5JjabZ+lggWNExaW1hVxKyi3LTKbrGRRvpZ2uxMUaVhivdUs6Bh0pqIcOOCSn69u5PegYjX1XElfp6G/fTSSUN5mLrSgrRLukgE67km7S1vrmQgEkub0YaNNNKPiLCsuZKX9pzg6OnMnqKyoGHS3qLaYurLCnjspX1pcbmHSCxG0E7uSzsrr5pDTJXvr9/vdVU8ZUHDpD0R4fZra3n90GleOXDS6+pMyEYa6WlOST7Xz6/g3zYeoH8o6nV1PGNBw/jChy6vojA3xKO/2ed1VSZk2VPp645ra+jqHWTt70behy5zWNAwvlCQE2LlVbN5butRjpxO7Zvm2EgjfV1TX0JjRZjHfpMeU6GJYEHD+MZtV9egqnzv5dSec7ZrT6UvEeH2a2rZfqSbjWl658jJsp5rfGN2cT7vvbSCH6T4nLONNNLbBy+voigvi8de2ud1VTxhQcP4yu3X1nCyb4h/3XDA66qMKRK1a0+ls7zsICsXzeb5bUfZeTTz7rNhQcP4ytV1JbyzsYyv/XwXx3pSM5/eRhrp75PvrGdaXhZffHprxq1tWNAwviIi/P0HFjAQiXHfs294XZ1RRWJq52mkueKCbP526Xw27O3KuEwqCxrGd2pLC/if76zlqd8eYkPbCa+r8xY20vCHP7xqNm+rLuLLz+ygp3/I6+okjQUN40ur3z2Xqul5fPHpbQxFY15X5zxVtewpnwgGhH9c0czxMwN87Re7va5O0ljPNb6Unx3iC+9rYmdHD/evS50fdMyZ/raRhj9cNns6K6+aw2Mv7ePlPak3qk0ECxrGt5YuqOCjLdV8479aeeqVdq+rA8SvOwVY9pSP3HPTfOpKC7jr+1vY03nG6+oknAUN41siwr0fXMg19SV87sevpcT6RtQZathIwz+m5Wbx6O1XEQoIH39sE1296XMXyYthQcP4WlYwwEP/40rmFOfzye9vofWYt3n1ESdo2EjDX2YX5/OtVS0cPd3PJ7+3mb7B9Li3y8WwoGF8ryg/i+/evohQQPjggy/xyze8uzVsNGojDb+6Ys4MvvrRt7Nl/0k+9OBLHOzq87pKCWFBw2SEOSX5/GT1tcyekc/HH9/Egy+0enJS1vmRRtB+en5089tm8t07FnH41Fk+8M0Xean1uNdVmnLWc03GqJ6Rz4//9BpuXjiTf3luJx/7zga2H+5Oah1sTcP/3tVYxtpPvYPScA5//OhG/m7tNl+tc1jQMBklLzvIN269nH9YsYBth7u5+Ru/5m9++LukTSVY9lRmqCkt4N9XX8vKq2bzvfX7ede//JKHXtjji5MAxe/XTWlpadHNmzd7XQ2Tgk73DfHAC6089pt9DEZjXFNfwh9cWc2NCyoJ54QS8p37T/Tyrv/9Al/96GV86IrqhHyHSS27O3q476dvsO6NY+RlBVneXMmHr6xmcW0xoRSephSRLaraMnK7q1+GiCwDvg4EgW+r6n0j3hfn/ZuAPuB2VX1lvLIiUgz8P6AG2Ad8VFVPOu/dA3wCiAJ/rqrPO9uvBB4D8oBngc+o36OeSZii/Cz+102Xcvs1NfxoSzs/2tLOXz75O0KB17hs9nSurivhypoZzK8spHJaLvFuPjmWPZV5GioK+c7tV/HqwVM8ufkg//HqYZ767SEKc0Isqi3m6voSLps9ncbyQorys7yu7oQmHGmISBDYBdwAtAObgFtVdfuwfW4CPk08aCwGvq6qi8crKyL/AnSp6n0icjcwQ1U/JyJNwA+ARcAs4BdAo6pGRWQj8BlgPfGgcb+q/nS8+ttIw7gViymb9nXxwq5OXt5zgtfaT50/g7swN0RdWZiq6bnMKsqjsiiXGfnZFBdkMz0/i3BOiHBuiPzsELlZAbKDgVGDzK6OHm78v7/igT+6gpvfNjPJLTSpoH8oyrodx3ix9Tgv7znOvhNvTo1WTsvlkpJ8Zk3PY9b0XMoLc5lRkE1xfjbT8kLxfpYTIj8nRG4okNCRymRGGouAVlVtcz5oDbAC2D5snxXAE85R/3oRmS4iM4mPIsYquwK4zin/OPAC8Dln+xpVHQD2ikgrsEhE9gHTVPVl57OeAG4Bxg0axrgVCAiL60pYXFcCQE//EDuO9LCzo4ddR3vYe7yXN4728F9vHKN/aPzrWQUEckJBskMBsoIBsoJCKCg4Sxqk8KyESbDcrCA3v23m+YOGo6f72XGk+3w/O3iyj417uzja3X8+cWIsoYCQEwqQda6fBYRQMEAoIAQDwn98+h3kZgWntP5ugkYVcHDY63bio4mJ9qmaoGyFqh4BUNUjIlI+7LPWj/JZQ87zkdvfQkTuBO4EmDNnzjhNM2ZshblZLKotZlFt8e9tV1W6z0Y42TdIV98gp/uGODMQ4cxAhN6BCP1DUfqHYvQPRYnElMFojKFIjGgsfrHCJXUltNQUj/GtJtNUFuVSWZTLu+eX/972SDTGqbNDnOob5MSZQXr6432sZyDC2cHI+T7WPxQjEosxFI0xGFFizkUxo7HE3OzLTdAY7VtHhr+x9nFT1u33uf4sVX0EeATi01MTfJ8xF0REKMrPoig/ixoKvK6O8alQMEBpOIfScA5zyyfeP1ncDJLbgdnDXlcDI+86MtY+45XtcKawcB7PnaY73mdVj7LdGGNMkrgJGpuABhGpFZFsYCWwdsQ+a4HbJG4JcNqZehqv7FpglfN8FfD0sO0rRSRHRGqBBmCj83k9IrLEyda6bVgZY4wxSTDh9JSqRkTkU8DzxNNmH1XVbSJyl/P+w8QzmW4CWomn3N4xXlnno+8DnhSRTwAHgI84ZbaJyJPEF8sjwGpVjTpl/pQ3U25/ii2CG2NMUtnJfcYYY95irJRbS/wzxhjjmgUNY4wxrlnQMMYY45oFDWOMMa75fiFcRDqB/RdZvBTw311UxpeJbYbMbHcmthkys90X0+ZLVLVs5EbfB43JEJHNo2UP+Fkmthkys92Z2GbIzHZPZZttesoYY4xrFjSMMca4ZkFjfI94XQEPZGKbITPbnYlthsxs95S12dY0jDHGuGYjDWOMMa5Z0DDGGOOaBY1RiMgyEdkpIq3O/ct9SURmi8gvRWSHiGwTkc8424tF5Ocistt5nOF1XaeaiARF5Lci8p/O60xo83QR+ZGIvOH8P7/a7+0Wkb9w+vZWEfmBiOT6sc0i8qiIHBORrcO2jdlOEbnH+fu2U0SWXsh3WdAYQUSCwAPAcqAJuFVEmrytVcJEgL9S1UuBJcBqp613A+tUtQFY57z2m88AO4a9zoQ2fx14TlXnA5cRb79v2y0iVcCfAy2q2kz89gwr8WebHwOWjdg2ajud3/hKYIFT5kHn754rFjTeahHQqqptqjoIrAFWeFynhFDVI6r6ivO8h/gfkSri7X3c2e1x4BZPKpggIlIN3Ax8e9hmv7d5GvBO4DsAqjqoqqfwebuJ3zMoT0RCQD7xu336rs2q+iuga8Tmsdq5AlijqgOqupf4fZAWuf0uCxpvVQUcHPa63dnmayJSA1wObAAqnDsl4jym0B2Kp8TXgL8FYsO2+b3NdUAn8F1nWu7bIlKAj9utqoeArxC/ydsR4ncU/Rk+bvMIY7VzUn/jLGi8lYyyzdd5ySISBn4MfFZVu72uTyKJyPuAY6q6xeu6JFkIuAJ4SFUvB3rxx7TMmJw5/BVALTALKBCRj3lbq5Qwqb9xFjTeqh2YPex1NfEhrS+JSBbxgPGvqvqUs7lDRGY6788EjnlVvwS4FviAiOwjPvX4HhH5Pv5uM8T7dbuqbnBe/4h4EPFzu98L7FXVTlUdAp4CrsHfbR5urHZO6m+cBY232gQ0iEitiGQTXzBa63GdEkJEhPgc9w5V/eqwt9YCq5znq4Cnk123RFHVe1S1WlVriP+//S9V/Rg+bjOAqh4FDorIPGfT9cB2/N3uA8ASEcl3+vr1xNft/Nzm4cZq51pgpYjkiEgt0ABsdPuhdkb4KETkJuLz3kHgUVW919saJYaIvAP4NfA6b87v/y/i6xpPAnOI//A+oqojF9nSnohcB/y1qr5PRErweZtF5O3EF/+zgTbgDuIHjr5tt4j8PfCHxDMFfwv8CRDGZ20WkR8A1xG/BHoH8CXgJ4zRThH5PPBx4v9dPquqP3X9XRY0jDHGuGXTU8YYY1yzoGGMMcY1CxrGGGNcs6BhjDHGNQsaxhhjXLOgYYwxxjULGsYYY1z7/1Mm/nAXrkZ3AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, cycle_limit=2)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch*2)\n", "\n", "plt.plot([i for i in range(num_epoch*2)], lr_per_epoch);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `t_in_epochs`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If set to False, the learning rates returned for epoch `t` are `None`." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[None, None, None, None, None]" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "num_epoch = 50\n", "scheduler = TanhLRScheduler(optimizer, t_initial=num_epoch, t_in_epochs=False)\n", "lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n", "\n", "lr_per_epoch[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `noise_range_t`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Add noise to learning rate scheduler. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `noise_pct`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The amount of noise to be added. Defaults to 0.67." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `noise_std`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Noise standard deviation. Defaults to 1.0." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `noise_seed`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Noise seed to use. Defaults to 42." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `initialize`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If set to True, then, the an attributes `initial_lr` is set to each param group. Defaults to `True`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> NOTE: we leave it up to the reader to try and experiment how the LR annealing scheduling get's updated by adding in noise hyperparams. " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }