{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# StepLRScheduler"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this tutorial we are going to be looking at the `StepLRScheduler` in the `timm` library."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The schedule looks something like:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"StepLR\"](\"images/StepLR.png\")
"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"#hide\n",
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from timm.scheduler.step_lr import StepLRScheduler\n",
"from nbdev.showdoc import show_doc"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"\n",
"\n",
"> StepLRScheduler(**`optimizer`**:`Optimizer`, **`decay_t`**:`float`, **`decay_rate`**:`float`=*`1.0`*, **`warmup_t`**=*`0`*, **`warmup_lr_init`**=*`0`*, **`t_in_epochs`**=*`True`*, **`noise_range_t`**=*`None`*, **`noise_pct`**=*`0.67`*, **`noise_std`**=*`1.0`*, **`noise_seed`**=*`42`*, **`initialize`**=*`True`*) :: `Scheduler`\n",
"\n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_doc(StepLRScheduler)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `StepLRScheduler` as shown above accepts an `optimizer` and also some hyperparams which we will look into in detail below. We will first see how we can train models using the `StepLRScheduler` by first using `timm` training docs and then look at how we can use this scheduler as standalone scheduler for our custom training scripts. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using `StepLRScheduler` scheduler with `timm` training script"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To train models using the `StepLRScheduler` we simply update the training script args passed by passing in `--sched step` parameter alongside the necessary hyperparams. In this section we will also look at how each of the hyperparams update the `cosine` scheduler. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The training command to use `cosine` scheduler looks something like: \n",
"\n",
"```python \n",
"python train.py ../imagenette2-320/ --sched step\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Args"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This way we start to use the `StepLRScheduler` with all the defaults. Let's now look at the associated hyperparams and how that updates the annealing schedule. "
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### `optimizer`"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"This is the `optimizer` that will be used for the training process. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"from timm import create_model \n",
"from timm.optim import create_optimizer\n",
"from types import SimpleNamespace"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"model = create_model('resnet34')\n",
"\n",
"args = SimpleNamespace()\n",
"args.weight_decay = 0\n",
"args.lr = 1e-4\n",
"args.opt = 'adam' \n",
"args.momentum = 0.9\n",
"\n",
"optimizer = create_optimizer(args, model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"This `optimizer` object created using `create_optimizer` is what get's passed to the `optimizer` argument. "
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### `decay_t`"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The number of epochs after which to decay the learning rate where the new learning rate value equals `lr * decay_rate`. "
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### `decay_rate`"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"When `decay_rate` > 0 and <1., at every restart the learning rate is decayed by new learning rate which equals `lr * decay_rate`. So if `decay_rate=0.5`, then in that case, the new learning rate becomes half the initial `lr`. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"hidden": true
},
"outputs": [],
"source": [
"from matplotlib import pyplot as plt\n",
"\n",
"def get_lr_per_epoch(scheduler, num_epoch):\n",
" lr_per_epoch = []\n",
" for epoch in range(num_epoch):\n",
" lr_per_epoch.append(scheduler.get_epoch_values(epoch))\n",
" return lr_per_epoch"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### `decay_rate=1.` or no decay"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"By setting `decay_t` = 5 and `decay_rate` = 1., we are telling the schedule to reduce the learning rate by decay_rate where new lr `lr * decay_rate` every 5 epochs. \n",
"\n",
"But since, `decay_rate=1.`, the new learning rate equals the old learning rate hence, we get a constant line. "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAD4CAYAAADLhBA1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAASqklEQVR4nO3cf6xf913f8ecLG5dqNOSXHWVxglNxETiItulVMOqkFYKCnUGvJQg4UhUvimYlNBts7YYLq4BOSFkrjRE1TchKFIdfqceA3KJUWTCp1k5Nm+uVJdjFzVUIieUoNtCFtlGbuXvvj+/H7d3l5n7Pvfdj3+Xe50O6Oud8fp3P52vFL5/P+d6kqpAkaaW+ZbUnIElaGwwUSVIXBookqQsDRZLUhYEiSepi42pPYLVcfPHFtW3bttWehiS9phw+fPivq2rzQnXrNlC2bdvGzMzMak9Dkl5TkvzVq9W55SVJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6mJQoCTZmeRYktkk+xeoT5I7W/2TSa4e1zfJhUkeTfJ0O17Qyi9K8liSLyf50Lz7vDXJU22sO5NkXv1PJqkkk0v9ICRJKzM2UJJsAO4CdgHbgRuTbJ/XbBcw0X72AXcP6LsfOFRVE8Chdg3wVeB9wHsWmM7dbfwz99o5Z55vAP4F8Jlxa5Ik9TfkCeUaYLaqnqmqV4AHgal5baaAB2rkceD8JJeO6TsFHGjnB4DdAFX1lar6FKNg+YY23nlV9emqKuCBM32afwd8YH4/SdK5MSRQLgOen3N9vJUNabNY30uq6gWAdtwyYB7HFxoryVuAy6vqjxcbIMm+JDNJZk6dOjXmdpKkpRgSKFmgrAa2GdJ3qAXHSvItwK8B7x43QFXdW1WTVTW5efPmZU5DkrSQIYFyHLh8zvVW4MTANov1fbFtY53Zzjo5YB5bFxjrDcD3AZ9I8iywA5j2xbwknVtDAuUJYCLJlUk2AXuA6XltpoGb2re9dgAvtW2sxfpOA3vb+V7gocUm0cb7UpId7dtdNwEPVdVLVXVxVW2rqm3A48A7qmpmwNokSZ1sHNegqk4nuR14BNgA3FdVR5Lc2urvAR4GrgdmgZeBmxfr24a+AziY5BbgOeCGM/dsTxrnAZuS7Aauq6qjwG3A/cDrgY+3H0nS/wcy+sLU+jM5OVkzMz7ESNJSJDlcVQu+UvA35SVJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6mJQoCTZmeRYktkk+xeoT5I7W/2TSa4e1zfJhUkeTfJ0O17Qyi9K8liSLyf50Lz7vDXJU22sO5Oklf+rJEfbvQ8l+c7lfiCSpOUZGyhJNgB3AbuA7cCNSbbPa7YLmGg/+4C7B/TdDxyqqgngULsG+CrwPuA9C0zn7jb+mXvtbOWfAyar6vuB3wc+MG5dkqS+hjyhXAPMVtUzVfUK8CAwNa/NFPBAjTwOnJ/k0jF9p4AD7fwAsBugqr5SVZ9iFCzf0MY7r6o+XVUFPDCnz2NV9XJr+jiwddDqJUndDAmUy4Dn51wfb2VD2izW95KqegGgHbcMmMfxMfMAuAX4+JixJEmdbRzQJguU1cA2Q/oONXasJO8EJoF/vOAAyT5GW2ZcccUVy5yGJGkhQ55QjgOXz7neCpwY2Gaxvi+2bawz21knB8xj7lbW/zOPJD8C/CLwjqr62kIDVNW9VTVZVZObN28ecztJ0lIMCZQngIkkVybZBOwBpue1mQZuat/22gG81LaxFus7Dext53uBhxabRBvvS0l2tG933XSmT5K3AL/BKEzGBZMk6SwYu+VVVaeT3A48AmwA7quqI0lubfX3AA8D1wOzwMvAzYv1bUPfARxMcgvwHHDDmXsmeRY4D9iUZDdwXVUdBW4D7gdez+g9yZl3JR8Evh34z+2bxM9V1TuW8XlIkpYpoy9MrT+Tk5M1MzOz2tOQpNeUJIeranKhOn9TXpLUhYEiSerCQJEkdWGgSJK6MFAkSV0YKJKkLgwUSVIXBookqQsDRZLUhYEiSerCQJEkdWGgSJK6MFAkSV0YKJKkLgwUSVIXBookqQsDRZLUhYEiSerCQJEkdWGgSJK6MFAkSV0YKJKkLgwUSVIXBookqQsDRZLUhYEiSerCQJEkdWGgSJK6MFAkSV0YKJKkLgwUSVIXgwIlyc4kx5LMJtm/QH2S3Nnqn0xy9bi+SS5M8miSp9vxglZ+UZLHknw5yYfm3eetSZ5qY92ZJK38dUk+2so/k2TbMj8PSdIyjQ2UJBuAu4BdwHbgxiTb5zXbBUy0n33A3QP67gcOVdUEcKhdA3wVeB/wngWmc3cb/8y9drbyW4AvVtV3Ab8G/Ptx65Ik9bVxQJtrgNmqegYgyYPAFHB0Tpsp4IGqKuDxJOcnuRTYtkjfKeDtrf8B4BPAz1fVV4BPJfmuuZNo451XVZ9u1w8Au4GPt7F+uTX9feBDSdLm09WvfOwIR0/8Xe9hJemc2f4Pz+OXfvyq7uMO2fK6DHh+zvXxVjakzWJ9L6mqFwDaccuAeRx/lbG+cZ+qOg28BFw0f4Ak+5LMJJk5derUmNtJkpZiyBNKFiib/y//V2szpO9Qi4016D5VdS9wL8Dk5OSy5nE2Ul2S1oIhTyjHgcvnXG8FTgxss1jfF9s21pntrJMD5rH1Vcb6xn2SbAS+A/jbMeNJkjoaEihPABNJrkyyCdgDTM9rMw3c1L7ttQN4qW1jLdZ3GtjbzvcCDy02iTbel5LsaN/uumlOn7lj/STwp2fj/Ykk6dWN3fKqqtNJbgceATYA91XVkSS3tvp7gIeB64FZ4GXg5sX6tqHvAA4muQV4DrjhzD2TPAucB2xKshu4rqqOArcB9wOvZ/Qy/uOty28Cv5VkltGTyZ7lfBiSpOXLev2H/OTkZM3MzKz2NCTpNSXJ4aqaXKjO35SXJHVhoEiSujBQJEldGCiSpC4MFElSFwaKJKkLA0WS1IWBIknqwkCRJHVhoEiSujBQJEldGCiSpC4MFElSFwaKJKkLA0WS1IWBIknqwkCRJHVhoEiSujBQJEldGCiSpC4MFElSFwaKJKkLA0WS1IWBIknqwkCRJHVhoEiSujBQJEldGCiSpC4MFElSFwaKJKmLQYGSZGeSY0lmk+xfoD5J7mz1Tya5elzfJBcmeTTJ0+14wZy697b2x5L86Jzyn27jH0nygTnlVyR5LMnnWv31y/kwJEnLNzZQkmwA7gJ2AduBG5Nsn9dsFzDRfvYBdw/oux84VFUTwKF2TavfA1wF7AQ+nGRDkouADwLXVtVVwCVJrm1j/VvgYFW9pfX98FI/CEnSygx5QrkGmK2qZ6rqFeBBYGpemynggRp5HDg/yaVj+k4BB9r5AWD3nPIHq+prVfWXwGwb543AF6rqVGv3J8BPtPMCzmvn3wGcGLAuSVJHQwLlMuD5OdfHW9mQNov1vaSqXgBoxy1jxpoFvifJtiQbGQXQ5a3NLwPvTHIceBj45wPWJUnqaEigZIGyGthmSN9B96uqLwK3AR8FPgk8C5xu9TcC91fVVuB64LeS/L21JdmXZCbJzKlTp+ZXS5JWYEigHOebTwIAW/n7W0qv1maxvi+2bTHa8eS4+1XVx6rqB6rqB4FjwNOtzS3Awdbm08C3ARfPX0hV3VtVk1U1uXnz5jHLliQtxZBAeQKYSHJlkk2MXnpPz2szDdzUvu21A3ipbWMt1nca2NvO9wIPzSnfk+R1Sa5k9KL/swBJtrTjBcDPAB9pfZ4Drm1138soUHwEkaRzaOO4BlV1OsntwCPABuC+qjqS5NZWfw+j9xbXM3rP8TJw82J929B3AAeT3MIoEG5ofY4kOQgcZbSl9a6q+nrr8+tJ3tTO319VX2jn7wb+U5J/yWhL7Z9W1bitNUlSR1mvf+9OTk7WzMzMak9Dkl5TkhyuqsmF6vxNeUlSFwaKJKkLA0WS1IWBIknqwkCRJHVhoEiSujBQJEldGCiSpC4MFElSFwaKJKkLA0WS1IWBIknqwkCRJHVhoEiSujBQJEldGCiSpC4MFElSFwaKJKkLA0WS1IWBIknqwkCRJHVhoEiSujBQJEldGCiSpC4MFElSFwaKJKkLA0WS1IWBIknqwkCRJHVhoEiSujBQJEldDAqUJDuTHEsym2T/AvVJcmerfzLJ1eP6JrkwyaNJnm7HC+bUvbe1P5bkR+eU/3Qb/0iSD8ybw08lOdrqfnepH4QkaWXGBkqSDcBdwC5gO3Bjku3zmu0CJtrPPuDuAX33A4eqagI41K5p9XuAq4CdwIeTbEhyEfBB4Nqqugq4JMm1rc8E8F7gba3u55b+UUiSVmLIE8o1wGxVPVNVrwAPAlPz2kwBD9TI48D5SS4d03cKONDODwC755Q/WFVfq6q/BGbbOG8EvlBVp1q7PwF+op3/M+CuqvoiQFWdHLZ8SVIvQwLlMuD5OdfHW9mQNov1vaSqXgBoxy1jxpoFvifJtiQbGQXQ5a3NdwPfneS/J3k8yc6FFpJkX5KZJDOnTp1aqIkkaZmGBEoWKKuBbYb0HXS/9vRxG/BR4JPAs8DpVr+R0Xbb24EbgY8kOX+BQe6tqsmqmty8efOYaUiSlmJIoBznm08CAFuBEwPbLNb3xbYtRjue2aZ61T5V9bGq+oGq+kHgGPD0nD4PVdX/bttkxxgFjCTpHBkSKE8AE0muTLKJ0Qvz6XltpoGb2re9dgAvtW2sxfpOA3vb+V7goTnle5K8LsmVjILhswBJtrTjBcDPAB9pff4I+KFWdzGjLbBnhn0EkqQeNo5rUFWnk9wOPAJsAO6rqiNJbm319wAPA9czes/xMnDzYn3b0HcAB5PcAjwH3ND6HElyEDjKaEvrXVX19dbn15O8qZ2/v6q+0M4fAa5LchT4OvCvq+pvlveRSJKWI1XjXmmsTZOTkzUzM7Pa05Ck15Qkh6tqcqE6f1NektSFgSJJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpCwNFktSFgSJJ6sJAkSR1YaBIkrowUCRJXRgokqQuDBRJUhcGiiSpi1TVas9hVSQ5BfzVMrtfDPx1x+m8VqzXdcP6XbvrXl+GrPs7q2rzQhXrNlBWIslMVU2u9jzOtfW6bli/a3fd68tK1+2WlySpCwNFktSFgbI89672BFbJel03rN+1u+71ZUXr9h2KJKkLn1AkSV0YKJKkLgyUJUqyM8mxJLNJ9q/2fM6WJPclOZnkz+eUXZjk0SRPt+MFqznHsyHJ5UkeS/L5JEeS/GwrX9NrT/JtST6b5H+2df9KK1/T6z4jyYYkn0vyx+16za87ybNJnkryZ0lmWtmK1m2gLEGSDcBdwC5gO3Bjku2rO6uz5n5g57yy/cChqpoADrXrteY08O6q+l5gB/Cu9me81tf+NeCHq+pNwJuBnUl2sPbXfcbPAp+fc71e1v1DVfXmOb97sqJ1GyhLcw0wW1XPVNUrwIPA1CrP6ayoqv8G/O284ingQDs/AOw+l3M6F6rqhar6H+38S4z+krmMNb72Gvlyu/zW9lOs8XUDJNkK/BPgI3OK1/y6X8WK1m2gLM1lwPNzro+3svXikqp6AUZ/8QJbVnk+Z1WSbcBbgM+wDtbetn3+DDgJPFpV62LdwH8E/g3wf+aUrYd1F/BfkxxOsq+VrWjdGztPcK3LAmV+73oNSvLtwH8Bfq6q/i5Z6I9+bamqrwNvTnI+8IdJvm+Vp3TWJfkx4GRVHU7y9lWezrn2tqo6kWQL8GiSv1jpgD6hLM1x4PI511uBE6s0l9XwYpJLAdrx5CrP56xI8q2MwuR3quoPWvG6WDtAVf0v4BOM3qGt9XW/DXhHkmcZbWH/cJLfZu2vm6o60Y4ngT9ktKW/onUbKEvzBDCR5Mokm4A9wPQqz+lcmgb2tvO9wEOrOJezIqNHkd8EPl9V/2FO1Zpee5LN7cmEJK8HfgT4C9b4uqvqvVW1taq2Mfrv+U+r6p2s8XUn+QdJ3nDmHLgO+HNWuG5/U36JklzPaM91A3BfVf3q6s7o7Ejye8DbGf3vrF8Efgn4I+AgcAXwHHBDVc1/cf+aluQfAZ8EnuKbe+q/wOg9yppde5LvZ/QSdgOjf2gerKr3J7mINbzuudqW13uq6sfW+rqTvJHRUwmMXn38blX96krXbaBIkrpwy0uS1IWBIknqwkCRJHVhoEiSujBQJEldGCiSpC4MFElSF/8XsGgHjRt4k5kAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"num_epoch = 50\n",
"scheduler = StepLRScheduler(optimizer, decay_t = 5, decay_rate=1.)\n",
"lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n",
"\n",
"plt.plot([i for i in range(num_epoch)], lr_per_epoch);"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### `decay_rate=0.5` "
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"By setting `decay_t` = 5 and `decay_rate` = 0.5, we are telling the schedule to reduce the learning rate by decay_rate where new lr `lr * decay_rate` every 5 epochs, that is halve the learning rate after every 5 epochs. \n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAb6UlEQVR4nO3df5TV9X3n8eeLe4eZIXBhQCTAYIE4SYMxMTohk5o9q7FpwG2KTWMWz+6RWvdwTDBJdzenwe3Zs6dtkrpJmiZ2/bE0dYPbrSw9u67TxsQQmmyarATHaDWoyARURhBQlB/yc+C9f9zvwM0wM/c7yOX7vXNfj3M4997v9/v5fj+fRH3x+Xy+3+9HEYGZmVka47KugJmZ1Q+HhpmZpebQMDOz1BwaZmaWmkPDzMxSK2ZdgVq74IILYu7cuVlXw8ysrjz22GOvRMT0wdvHfGjMnTuXnp6erKthZlZXJL0w1HYPT5mZWWoODTMzS82hYWZmqTk0zMwsNYeGmZmllio0JC2StFlSr6SVQ+yXpDuS/U9KurxaWUlTJa2TtCX5bEu2T5P0A0kHJf2XQde5QtJTybnukKSzb7qZmY1W1dCQVADuBBYDC4AbJC0YdNhioCP5sxy4O0XZlcD6iOgA1ie/AY4A/xH43BDVuTs5/8C1FqVqpZmZnRNpntNYCPRGxFYASWuAJcDTFccsAe6L8nvWN0iaImkmMHeEskuAq5Lyq4EfAp+PiDeAH0u6uLISyflKEfFI8vs+4DrgO6Nrcjrf+sk29r5xrBanPkUSH7+inTlTJ9T0OmZm50qa0JgNbK/43Qe8P8Uxs6uUnREROwEiYqekC1PUo2+Ia5xB0nLKPRIuuuiiKqcd2t9sfJEtuw+eVdm0IuD4iZP8waJfrel1zMzOlTShMdS8weCVm4Y7Jk3ZtFKfKyJWAasAOjs7z+p63/u3//xsio1K5xfWse/w8Zpfx8zsXEkzEd4HzKn43Q7sSHnMSGV3JUNOA0NPu1PUo71KPepKqaXJoWFmdSVNaDwKdEiaJ2k8sBToHnRMN3BjchdVF7AvGXoaqWw3sCz5vgx4cKRKJOc7IKkruWvqxmpl8m5SaxP7j/RnXQ0zs9SqDk9FRL+kW4GHgQJwb0RsknRLsv8e4CHgWqAXOATcNFLZ5NS3A2sl3Qy8CFw/cE1JzwMlYLyk64DfiIingU8C3wJaKU+A12QS/HwptRTZ756GmdWRVG+5jYiHKAdD5bZ7Kr4HsCJt2WT7q8A1w5SZO8z2HuBdaepcD0qtTbz0+uGsq2FmlpqfCM9QqaWJ/Yc9PGVm9cOhkaFSa5H9Rzw8ZWb1w6GRoVJLE8f6T3Lk+Imsq2JmlopDI0Ol1iYA9zbMrG44NDJUainfh+B5DTOrFw6NDLmnYWb1xqGRoVJLEhp+VsPM6oRDI0OTW5PhKT8VbmZ1wqGRIfc0zKzeODQy5DkNM6s3Do0MtTQVGF8c57unzKxuODQyVmppck/DzOqGQyNjpVa/6dbM6odDI2NeiMnM6olDI2MlL8RkZnXEoZGxUkuRA+5pmFmdcGhkrNzTcGiYWX1waGRsYCGm8uKHZmb55tDIWKm1yLETJznafzLrqpiZVeXQyJhfJWJm9cShkTG/SsTM6olDI2MDCzHt86tEzKwOODQy5p6GmdUTh0bGPKdhZvXEoZGxkhdiMrM64tDImHsaZlZPHBoZO7Wmhuc0zKwOODRyYOCpcDOzvHNo5MDk1qJ7GmZWFxwaOVBqbfKchpnVBYdGDpSHpxwaZpZ/qUJD0iJJmyX1Slo5xH5JuiPZ/6Sky6uVlTRV0jpJW5LPtop9tyXHb5b0kYrtN0h6KrnGdyVdcPZNzw8vxGRm9aJqaEgqAHcCi4EFwA2SFgw6bDHQkfxZDtydouxKYH1EdADrk98k+5cClwCLgLskFSQVgW8AV0fEu4EngVvPst25UmrxOuFmVh/S9DQWAr0RsTUijgFrgCWDjlkC3BdlG4ApkmZWKbsEWJ18Xw1cV7F9TUQcjYhtQG9yHiV/3iJJQAnYMeoW59DAQkxeU8PM8i5NaMwGtlf87ku2pTlmpLIzImInQPJ54UjniojjwCeBpyiHxQLgr4aqsKTlknok9ezZsydFE7NVamni+IngyHGvqWFm+ZYmNDTEtsF/JR7umDRlU11PUhPl0HgvMIvy8NRtQ50gIlZFRGdEdE6fPr3K5bJ3+lUiHqIys3xLExp9wJyK3+2cOSw03DEjld2VDGGRfO6ucq7LACLiF1Eex1kL/FqK+ueeXyViZvUiTWg8CnRImidpPOVJ6u5Bx3QDNyZ3UXUB+5Ihp5HKdgPLku/LgAcrti+V1CxpHuXJ9Y3AS8ACSQNdhw8Dz4yyvbnk16ObWb0oVjsgIvol3Qo8DBSAeyNik6Rbkv33AA8B11KetD4E3DRS2eTUtwNrJd0MvAhcn5TZJGkt8DTQD6yIiBPADkl/BPxI0nHgBeB3z8H/BpkbWIjJrxIxs7yrGhoAEfEQ5WCo3HZPxfcAVqQtm2x/FbhmmDJfBL44xPZ7gHvOLFHf3NMws3rhJ8JzwHMaZlYvHBo5MKnFCzGZWX1waORAS1OB5uI49zTMLPccGjkx8FS4mVmeOTRyovz+KQ9PmVm+OTRyYrJ7GmZWBxwaOeGFmMysHjg0cqLU0sQ+h4aZ5ZxDIydKrUXfcmtmuefQyImBJV+9poaZ5ZlDIydKrU30nwwOHz+RdVXMzIbl0MiJ068S8RCVmeWXQyMnvBCTmdUDh0ZO+KWFZlYPHBo54dejm1k9cGjkhBdiMrN64NDICfc0zKweODRy4tSaGp7TMLMcc2jkRHOxQEvTOD8Vbma55tDIkYGnws3M8sqhkSNeiMnM8s6hkSNeiMnM8s6hkSNeiMnM8s6hkSNeiMnM8s6hkSNeiMnM8s6hkSMDCzF5TQ0zyyuHRo6UWpo4cTI4dMxraphZPjk0csSvEjGzvHNo5IgXYjKzvHNo5IgXYjKzvEsVGpIWSdosqVfSyiH2S9Idyf4nJV1eraykqZLWSdqSfLZV7LstOX6zpI9UbB8vaZWk5yQ9K+l3zr7p+eOFmMws76qGhqQCcCewGFgA3CBpwaDDFgMdyZ/lwN0pyq4E1kdEB7A++U2yfylwCbAIuCs5D8AfArsj4u3J+f7vWbQ5tzynYWZ5l6ansRDojYitEXEMWAMsGXTMEuC+KNsATJE0s0rZJcDq5Ptq4LqK7Wsi4mhEbAN6k/MA/B7wpwARcTIiXhldc/PNCzGZWd6lCY3ZwPaK333JtjTHjFR2RkTsBEg+LxzpXJKmJL//RNLPJP2tpBlDVVjSckk9knr27NmToon5MMnDU2aWc2lCQ0NsG/z02XDHpCmb9npFoB34SURcDjwCfHWoE0TEqojojIjO6dOnV7lcfowvjqO1qeDhKTPLrTSh0QfMqfjdDuxIecxIZXclQ1gkn7urnOtV4BDwQLL9b4HLGWNKrX7TrZnlV5rQeBTokDRP0njKk9Tdg47pBm5M7qLqAvYlQ04jle0GliXflwEPVmxfKqlZ0jzKk+sbo/xujb8DrkqOuwZ4enTNzb9Si990a2b5Vax2QET0S7oVeBgoAPdGxCZJtyT77wEeAq6lPGl9CLhppLLJqW8H1kq6GXgRuD4ps0nSWsqB0A+siIiB92p8Hvjvkr4O7Bm4zljihZjMLM801l+O19nZGT09PVlXI7Wb/ttGXjl4jL/79AezroqZNTBJj0VE5+DtfiI8Z7wQk5nlmUMjZ7wQk5nlmUMjZ8oT4V5Tw8zyyaGRM6XWIidOBm94TQ0zyyGHRs74pYVmlmcOjZzxSwvNLM8cGjnjhZjMLM8cGjlzaiEmD0+ZWQ45NHLmVE/Dw1NmlkMOjZw5NafhnoaZ5ZBDI2cmDSzEdMRzGmaWPw6NnGkqjGPC+IJ7GmaWSw6NHPLr0c0sr6q+Gt3Ov1JrkX/avo87f9Bb0+t89N2zuGjahJpew8zGFodGDl0yazIPPP4SX3l4c02v0/faIf70Y++u6TXMbGxxaOTQ1z7xHv7z79T2P+bX/9dH2L73cE2vYWZjj0MjhyQxvqiaXmNOWys/f2lfTa9hZmOPJ8IbVHvbBF56/TAnT/oV7GaWnkOjQbW3tXL8RLD7wNGsq2JmdcSh0aDa21qB8mS4mVlaDo0G1d5WvtV2u0PDzEbBodGgTvU0fAeVmY2CQ6NBtTQVmD6pmb7XHBpmlp5Do4G1t7XS97qHp8wsPYdGA2tvm+CehpmNikOjgbW3tbLj9cOc8LMaZpaSQ6OBnX5W40jWVTGzOuHQaGADt916iMrM0nJoNDA/4Gdmo+XQaGCzp5RDw2+7NbO0HBoN7PSzGu5pmFk6qUJD0iJJmyX1Slo5xH5JuiPZ/6Sky6uVlTRV0jpJW5LPtop9tyXHb5b0kSGu1y3p56Nvrg02p63VcxpmllrV0JBUAO4EFgMLgBskLRh02GKgI/mzHLg7RdmVwPqI6ADWJ79J9i8FLgEWAXcl5xmoz8eAg2fTWDuTn9Uws9FI09NYCPRGxNaIOAasAZYMOmYJcF+UbQCmSJpZpewSYHXyfTVwXcX2NRFxNCK2Ab3JeZA0Efh3wBdG31Qbip/VMLPRSBMas4HtFb/7km1pjhmp7IyI2AmQfF6Y4np/AvwZMOIgvKTlknok9ezZs2ekQxtee9sE+k8Gu/b7WQ0zqy5NaAy17ujgv5YOd0yasqmuJ+ky4OKIeKBKeSJiVUR0RkTn9OnTqx3e0E7fdushKjOrLk1o9AFzKn63AztSHjNS2V3JEBbJ5+4q5/oAcIWk54EfA2+X9MMU9bcR+FkNMxuNNKHxKNAhaZ6k8ZQnqbsHHdMN3JjcRdUF7EuGnEYq2w0sS74vAx6s2L5UUrOkeZQn1zdGxN0RMSsi5gIfBJ6LiKvOos1WYdYU9zTMLL1itQMiol/SrcDDQAG4NyI2Sbol2X8P8BBwLeVJ60PATSOVTU59O7BW0s3Ai8D1SZlNktYCTwP9wIqIOHGuGmy/rKWpwIWTmtm+1z0NM6tOEWP7rpnOzs7o6enJuhq59rG7fkJzscD9y7uyroqZ5YSkxyKic/B2PxFuzJk6wYsxmVkqDg2jva2Vna8fof/EyayrYmY559Cw089qHDiadVXMLOccGnb6tltPhptZFQ4N82JMZpaaQ8OYNaUFcGiYWXUODaO5WGBGyetqmFl1Dg0DykNU2x0aZlaFQ8OA8mS4h6fMrBqHhgEwp20CO/f5WQ0zG5lDw4ByT+PEyeBlr6thZiNwaBjg227NLB2HhgFejMnM0nFoGAAzp7QgeTEmMxuZQ8OA5FmNSS3uaZjZiBwadkr5tlv3NMxseA4NO6W9rZXte93TMLPhOTTslPa2Cby8389qmNnwHBp2ypyp5Wc1du7zsxpmNjSHhp3iZzXMrBqHhp1y+lkNT4ab2dAcGnbKzMmtybMa7mmY2dAcGnbK+OI43lrysxpmNrxi1hWwfJnTNoEHHu/j20/tqOl1Pv2hDlZcfXFNr2Fm555Dw37J5z7yDtY/s6um1/j+M7t48ImXHBpmdcihYb9k4bypLJw3tabXmDyhiS9/dzOvHDzKBROba3otMzu3PKdh513X/GkA/HTr3oxrYmaj5dCw8+7S2ZN5y/gCG7a+mnVVzGyUHBp23jUVxtE5d6pDw6wOOTQsE13zp7Fl90FeOXg066qY2Sg4NCwTXfPLk+3ubZjVl1ShIWmRpM2SeiWtHGK/JN2R7H9S0uXVykqaKmmdpC3JZ1vFvtuS4zdL+kiybYKkb0t6VtImSbe/uaZbljyvYVafqoaGpAJwJ7AYWADcIGnBoMMWAx3Jn+XA3SnKrgTWR0QHsD75TbJ/KXAJsAi4KzkPwFcj4leB9wJXSlp8No227BUL43jfvKls8B1UZnUlTU9jIdAbEVsj4hiwBlgy6JglwH1RtgGYImlmlbJLgNXJ99XAdRXb10TE0YjYBvQCCyPiUET8ACA518+A9tE32fKia/40encfZM8Bz2uY1Ys0oTEb2F7xuy/ZluaYkcrOiIidAMnnhWmvJ2kK8FHKPZQzSFouqUdSz549e0Zqm2Xo1PMa2zxEZVYv0oSGhtgWKY9JU3ZU15NUBO4H7oiIrUOdICJWRURnRHROnz69yuUsK++aVWJic5FHfuHQMKsXaUKjD5hT8bsdGPw2u+GOGansrmQIi+Rzd8rrrQK2RMTXU9TdcqxYGMf75rZ5MtysjqQJjUeBDknzJI2nPEndPeiYbuDG5C6qLmBfMuQ0UtluYFnyfRnwYMX2pZKaJc2jPLm+EUDSF4DJwO+PvqmWR13zp/GLPW+w+4CXmDWrB1VDIyL6gVuBh4FngLURsUnSLZJuSQ57CNhKedL6L4FPjVQ2KXM78GFJW4APJ79J9q8Fnga+C6yIiBOS2oE/pHwX1s8kPSHp37zZ/wEsW34PlVl9UUS1KYb61tnZGT09PVlXw4bRf+Ik7/3jdXz0sll86bcvzbo6ZpaQ9FhEdA7e7ifCLVOnn9fwvIZZPXBoWOa65k9l65432L3f8xpmeefQsMwNzGts2OZ5DbO8c2hY5i6ZNZlJfl7DrC44NCxzhXFi4byp/NTzGma559CwXOiaP42tr7zBLs9rmOWaQ8Ny4dS8hnsbZrlWzLoCZgALZpWY1FLke5t2Me+Ct9T0WvOnT2Ris//RNzsb/jfHcqEwTvza26bx7ad28u2ndtb0Wh+YP437l3fV9BpmY5VDw3LjS799KZ/onFP9wDdhw9ZX+ct/3MYjv3iVD7xtWk2vZTYWOTQsN6ZNbOaad86o6TWuvPgCHnxiB3/+/efomt+FNNSb+M1sOJ4It4bS0lTgU1e9jY3b9vKIJ93NRs2hYQ1n6cKLmFFq5uvrtjDWX9hpdq45NKzhlHsbF7Px+b1+Ct1slBwa1pD+5fvm8NZSC3/+/efc2zAbBYeGNaSWpgKfuvptPPr8a/w/9zbMUnNoWMM61dtY596GWVoODWtYzcUCK65+Gz0vvMZPet3bMEvDoWEN7RPvm8PMyS183XMbZqk4NKyhNRcLfOrqi+l54TV+3PtK1tUxyz0/EW4N7xOd7dz9g14+ff/jzJjUUtNr/bOOC7jt2ndSGOcn0a0+OTSs4TUXC3zpY5eyZuP2ml7njWP9fPPH23jt0HG+/PF3OzisLjk0zICr3nEhV73jwppf5xvf31J+NoTgKx9/j4PD6o5Dw+w8+uyvdyDB19Y9BwFfud7BYfXFoWF2nn3mmg4E/Nm65wAHh9UXh4ZZBj59TbnH8dXvPUcAX3VwWJ1waJhl5NYPdSCJrzy8mZdeO8zsttaaXm/+BW/hxg/MZfKEpppex8Y2h4ZZhlZcfTHNxXH89YYXeHn/kZpdJwgeePwlVv1oKzddOZff++A8pkwYX7Pr2dilsf4UbGdnZ/T09GRdDbPMPbNzP3es38J3fv4yE5uL3HTlXG52eNgwJD0WEZ1nbHdomDWWZ3bu5y/+YQsPPVUOj9+6bBYTm2s76NA2YTzvnz+VS2dPpqngF1HUg+FCI9U/KZIWAd8ACsA3I+L2QfuV7L8WOAT8bkT8bKSykqYC/xOYCzwPfCIiXkv23QbcDJwAPhMRDyfbrwC+BbQCDwGfjbGeembn2DtnlrjrX13Bsy/v5y/W9/Lg4y9xssb/Fh0+fgKACeMLXPErbXTNn0bX/KlcOnsK44sOkXpStachqQA8B3wY6AMeBW6IiKcrjrkW+DTl0Hg/8I2IeP9IZSV9GdgbEbdLWgm0RcTnJS0A7gcWArOA7wNvj4gTkjYCnwU2UA6NOyLiOyPV3z0Ns+y9evAoG7ftZcPWV9mwdS+bdx04b9dubSowu62V9lN/JtDe1spbSy2Mq/Eda8VxYlJLE6WWIqXWprrqZb2ZnsZCoDcitiYnWgMsAZ6uOGYJcF/yt/4NkqZImkm5FzFc2SXAVUn51cAPgc8n29dExFFgm6ReYKGk54FSRDySnOs+4DpgxNAws+xNm9jM4ktnsvjSmcDpEHn25QPUeqjgjaP9vPTaYfpeP8QT21/n9UPHa3zF4bU2FSi1FpnYXGScan+L9d9/5oM0Fwvn9JxpQmM2UPlSnj7KvYlqx8yuUnZGROwEiIidkgbe4TCbck9i8LmOJ98Hbz+DpOXAcoCLLrpohKaZWRYGh8j5dDAJkV37j3CyxqPb/SeCA0ePs/9wP/sPH2f/kfL3g0f7iZrHJYhzH0xpQmOoqw5u7XDHpCmb9nqpzxURq4BVUB6eqnI9M2sgE5uLvOOtk3jHWydlXZW6lGaArQ+YU/G7HdiR8piRyu5KhrBIPnenOFd7lXqYmVkNpQmNR4EOSfMkjQeWAt2DjukGblRZF7AvGXoaqWw3sCz5vgx4sGL7UknNkuYBHcDG5HwHJHUld2vdWFHGzMzOg6rDUxHRL+lW4GHKt83eGxGbJN2S7L+H8p1M1wK9lG+5vWmkssmpbwfWSroZeBG4PimzSdJaypPl/cCKiDiRlPkkp2+5/Q6eBDczO6/8cJ+ZmZ1huFtu6+emYTMzy5xDw8zMUnNomJlZag4NMzNLbcxPhEvaA7xwlsUvAF45h9WpF253Y3G7G0vadv9KREwfvHHMh8abIalnqLsHxjq3u7G43Y3lzbbbw1NmZpaaQ8PMzFJzaIxsVdYVyIjb3Vjc7sbyptrtOQ0zM0vNPQ0zM0vNoWFmZqk5NIYgaZGkzZJ6k/XLxyxJ90raLennFdumSlonaUvy2ZZlHWtB0hxJP5D0jKRNkj6bbB/TbZfUImmjpH9K2v1HyfYx3W4ASQVJj0v6++T3mG8zgKTnJT0l6QlJPcm2s267Q2MQSQXgTmAxsAC4QdKCbGtVU98CFg3athJYHxEdwPrk91jTD/z7iHgn0AWsSP5/HuttPwp8KCLeA1wGLErWwBnr7Qb4LPBMxe9GaPOAqyPisornM8667Q6NMy0EeiNia0QcA9YASzKuU81ExI+AvYM2LwFWJ99XA9edzzqdDxGxMyJ+lnw/QPk/JrMZ422PsoPJz6bkTzDG2y2pHfgXwDcrNo/pNldx1m13aJxpNrC94ndfsq2RzEhWSiT5vDDj+tSUpLnAe4Gf0gBtT4ZpnqC8xPK6iGiEdn8d+APgZMW2sd7mAQF8T9JjkpYn28667VVX7mtAGmKb70seoyRNBP4X8PsRsb+8kvDYlqyEeZmkKcADkt6VcZVqStJvArsj4jFJV2VcnSxcGRE7JF0IrJP07Js5mXsaZ+oD5lT8bgd2ZFSXrOySNBMg+dydcX1qQlIT5cD4HxHxv5PNDdF2gIh4Hfgh5TmtsdzuK4HfkvQ85eHmD0n6a8Z2m0+JiB3J527gAcpD8GfddofGmR4FOiTNkzQeWAp0Z1yn860bWJZ8XwY8mGFdakLlLsVfAc9ExNcqdo3ptkuanvQwkNQK/DrwLGO43RFxW0S0R8Rcyv8+/0NE/GvGcJsHSHqLpEkD34HfAH7Om2i7nwgfgqRrKY+BFoB7I+KL2daodiTdD1xF+XXJu4D/BPwfYC1wEfAicH1EDJ4sr2uSPgj8I/AUp8e5/wPleY0x23ZJ76Y88Vmg/JfGtRHxx5KmMYbbPSAZnvpcRPxmI7RZ0nzKvQsoT0f8TUR88c203aFhZmapeXjKzMxSc2iYmVlqDg0zM0vNoWFmZqk5NMzMLDWHhpmZpebQMDOz1P4/tMcbsPg1kiAAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"num_epoch = 50\n",
"scheduler = StepLRScheduler(optimizer, decay_t = 5, decay_rate=.5)\n",
"lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n",
"\n",
"plt.plot([i for i in range(num_epoch)], lr_per_epoch);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `warmup_t` "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Defines the number of warmup epochs. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `warmup_lr_init` "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The initial learning rate during warmup. "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsKElEQVR4nO3deXhV1b3/8fc3M0MIhBkSTFBQA0GGKJNgrEPRq2C1tWgHWlt5rHpbre1POqutvfYRe9Wr1ceqVe+1tdjeKnqtFVEEChbBWhFQBEQS5jEHQhJykvX74+zEGEiyz5TDOXxez5MnOfvstdd3BT3frLX2Xsucc4iIiPiRlugAREQkeShpiIiIb0oaIiLim5KGiIj4pqQhIiK+ZSQ6gHjr06ePKyoqSnQYIiJJZdWqVXucc31bH0/5pFFUVMTKlSsTHYaISFIxs4+PdVzDUyIi4puShoiI+KakISIivqX8nIaIxEZ9fT2VlZXU1tYmOhSJoZycHAoKCsjMzPR1vpKGiPhSWVlJbm4uRUVFmFmiw5EYcM6xd+9eKisrKS4u9lXG1/CUmU0zsw/MbIOZzTnG+2Zm93vvv2tmYzsqa2b5ZrbAzD70vvfyjvc2s9fN7JCZPdCqnnFmttq71v2m/3JFOk1tbS29e/dWwkghZkbv3r3D6j12mDTMLB14ELgIKAGuMrOSVqddBAzzvmYDD/koOwdY6JwbBiz0XgPUAj8BvneMcB7yrt9U1zRfrRSRmFDCSD3h/pv6GZ46C9jgnNvkVfAMMANY2+KcGcBTLrTO+ptm1tPMBgJF7ZSdAZR75Z8EFgG3OueqgaVmdkqrhg0EejjnlnuvnwIuA/4aVov9eu/PUHPA9+mH6oKs2LyPxsYwlpq3NE4/50oGDxkafnwiIgngJ2kMBipavK4Exvs4Z3AHZfs757YDOOe2m1k/H3FUHqOOo5jZbEI9EoYMGdLBZduwdj4cOOazLcdUe+gIvQ/UhFWF4VhTV83gb/483OhETjg333wzJ510EjfddBMAn/3sZyksLOTRRx8F4JZbbmHw4MGccsoprF27ljlz5vDcc88xfPhwSkpCAxzl5eXMnTuXsrKyRDUj6flJGsfqu7T+c7qtc/yU9cv3tZxzjwCPAJSVlUVW35VPhnX6069+yH++up4Nd15ERrq/O5kX3XkpBCo7PlFEmDRpEs8++yw33XQTjY2N7Nmzh0Ag0Pz+smXLuPfeexk/fjzTp08H4LnnnuOSSy5pThqdqaGhgfT09E6vN978fLpVAoUtXhcA23ye017Znd6QU9PQ0y4fcRR0EEfCBGrr6ZaV7jthABzpNpCcw9vjGJVI6pg8eTLLli0DYM2aNYwcOZLc3Fz2799PXV0d69atY8yYMTzxxBPceOONLFu2jPnz5/P973+f0aNHs3HjRgCeffZZzjrrLIYPH86SJUuOquf6669n/vz5AHzuc5/jmmuuAeCxxx7jxz/+MQCXXXYZ48aNY8SIETzyyCPNZbt3785Pf/pTxo8fz/Lly+nevTu33nor48aN4/zzz2fFihWUl5czdOjQ5jqa4m1yySWXsGjRoubr3XLLLYwdO5bzzjuP3bt3x/i3Gj4/PY23gGFmVgxsBWYCV7c6Zz5wozdnMR6o8oacdrdTdj4wC7jL+/58e0F41ztoZhOAfwBfBf7LR/ydIlBTT48u/u5zbmJ5g8jd93bK/kUiqev2F9awdlug4xPDUDKoBz+7dESb7w8aNIiMjAy2bNnCsmXLmDhxIlu3bmX58uXk5eUxatQosrKyms+fNGkS06dP55JLLuHzn/988/FgMMiKFSt46aWXuP3223n11Vc/Vc/UqVNZsmQJ06dPZ+vWrWzfHvrDbunSpcycOROAxx9/nPz8fGpqajjzzDO54oor6N27N9XV1YwcOZI77rgDgOrqasrLy/nVr37F5z73OX784x+zYMEC1q5dy6xZs5p7RG2prq5m7Nix3HPPPdxxxx3cfvvtPPDAA+2WibcO/yx2zgWBG4G/AeuAec65NWZ2nZld5532ErAJ2AD8Fri+vbJembuAC8zsQ+AC7zUAZrYZ+DXwNTOrbHHH1beAR716NhKvSfAIBGrr6ZETXtLI6X0S6QTZsdX/3InIiaypt9GUNCZOnNj8etKkSb6ucfnllwMwbtw4Nm/efNT7U6ZMYcmSJaxdu5aSkhL69+/P9u3bWb58eXMd999/P2eccQYTJkygoqKCDz/8EID09HSuuOKK5mtlZWUxbVroJs/S0lLOOeccMjMzKS0tPWbdraWlpfHFL34RgC9/+cssXbrUVxvjydfDfc65lwglhpbHHm7xswNu8FvWO74XOK+NMkVtHF8JjPQTc2cL1ATp0SW8ZyXzBoQeptm99SPdQSVJpb0eQTxNmjSJZcuWsXr1akaOHElhYSH33HMPPXr0aB5G6kh2djYQ+oAPBoNHvT948GD279/Pyy+/zNSpU9m3bx/z5s2je/fu5ObmsmjRIl599VWWL19O165dKS8vb37OIScn51OjBpmZmc23tKalpTXXnZaW1lx3RkYGjY2NzWXae2bieLjlWWtPxUgkPY3+haFEUbXzo3iEJJJyJk+ezIsvvkh+fj7p6enk5+dz4MABli9fzsSJE486Pzc3l4MHD4Zdz8SJE7n33nuZOnUqU6ZMYe7cuUyZMgWAqqoqevXqRdeuXXn//fd58803o2pTUVER77zzDo2NjVRUVLBixYrm9xobG/nTn/4EwO9//3vOPvvsqOqKBSWNGAnUhj+n0bffYIKWQd3eLXGKSiS1lJaWsmfPHiZMmPCpY3l5efTp0+eo82fOnMndd9/NmDFjmifC/ZgyZQrBYJBTTjmFsWPHsm/fvuakMW3aNILBIKNGjeInP/nJp2KJxOTJkykuLqa0tJTvfe97jB3bvKAG3bp1Y82aNYwbN47XXnuNn/70p1HVFQsWGllKXWVlZa4zNmE64/ZXuGz0IG6fEd7o2Rt3Xkxd14FcePNjcYpMJDbWrVvH6aefnugwTijdu3fn0KFDca/nWP+2ZrbKOXfUAy3qacRAY6PjYAQ9DYC6rgPJrtZttyKSHJQ0YqD6SJBGR9hzGgCWV0hu/S4aGxriEJmIJLPO6GWES0kjBgK1obsg8iLoaWT3LiTDBdm1Y2uswxIRiTkljRgI1NQDhH3LLUBe/yIAdm7dEMuQRETiQkkjBqqakkYEw1N9C0K33QZ26AE/ETn+KWnEwCc9jfCTRv+BQ2ggg9o9ShoicvxT0oiBpjmNSHoa6enpBLL64qo0pyHSnptvvpl77723+fVnP/tZvvnNbza/vuWWW/j1r3/N/Pnzueuu0KpEzz33HGvXfrL1T3l5ObG6Bf+Xv/xlTK6TbJQ0YiCaOQ0I3XabpdVuRdrVtIQI0Lw0+po1a5rfX7ZsGZMnT2b69OnMmRPaCLR10oileCeNYy1xcjxQ0oiBQG0oaXTPjixp0GMwPY7sxLVYf0ZEPi2eS6PX1tby9a9/ndLSUsaMGcPrr78OtL1s+Zw5c6ipqWH06NF86Utf+lSc8+bN47vf/S4A9913H0OHhuYtN27c2LwMyB133MGZZ57JyJEjmT17Nk0PWZeXl/PDH/6Qc845h/vuu4/y8nJuvvlmpk6dyumnn85bb73F5ZdfzrBhw5qXad+8eTMjR37yUPHcuXO57bbbmq930003MWnSJEaOHPmpJUoiFeGnnLQUqAnSPTsjrL00WsruXUjGlnp279pGvwEFHRcQSbRl/wV7PoztNfsMg0n/3ubb8Vwa/cEHHwRg9erVvP/++1x44YWsX7++zVjuuusuHnjgAd55552j3ps6dSp33303AEuWLKF3795s3bqVpUuXNi9FcuONNzYvCfKVr3yFF198kUsvvRSAAwcO8MYbbwDwwgsvkJWVxeLFi7nvvvuYMWMGq1atIj8/n5NPPpmbb765w19rdXU1y5YtY/HixVxzzTW89957HZZpj3oaMRBarDDy/NvDu+12R4X/tXFETkTxWhp96dKlfOUrXwHgtNNO46STTmo3abRnwIABHDp0iIMHD1JRUcHVV1/N4sWLWbJkSXPSeP311xk/fjylpaW89tprnxpma1oKvUnTnhulpaWMGDGCgQMHkp2dzdChQ6moqKAjV111FRBKZoFAgAMHDkTUribqacRAJBswtdSv4BT2AlU7PgLOiVlcInHTTo8grtXGaWn0ttbgC2fZ8pYmTpzI7373O0499VSmTJnC448/zvLly7nnnnuora3l+uuvZ+XKlRQWFnLbbbd96rrdunU7Zrwtl1Zveh0MBjuMsfVy6tEur66eRgxEsix6S/0HFtJIOrV7O/6rQeREFq+l0adOncrTTz8NwPr169myZQunnnpqu8uWZ2ZmUl9f3+b15s6dy9SpU5vnSLKzs8nLy2v+UO/Tpw+HDh1qXvo8Uv3792fXrl3s3buXuro6XnzxxU+9/8c//hEI9aby8vLIy8uLqj71NGIgUBNkUM8uEZfPyMzkYGYfXFVlDKMSST1NS6NfffXVnzp26NChNpdGv/baa7n//vvb/XC+/vrrue666ygtLSUjI4MnnniC7OzsTy1bPnLkyE8tWz579mxGjRrF2LFjmxNOkylTplBRUcHUqVNJT0+nsLCQ0047DYCePXty7bXXUlpaSlFREWeeeWZUv5PMzMzmfcmLi4ub62nSq1cvJk2aRCAQ4PHHH4+qLtDS6DFx9q9e46zifH595eiIr7Hgnllk1u6n/EfzYxeYSAxpafTkU15ezty5cykrO2qF80/R0uidrKomuuEpANejgFzddisixzkNT0WpsdFxqC4Y1UQ4QFbvIWRW1LFnzw769hsUo+hE5ES2aNGimF9TPY0oHawL4hxR3XIL0KPfSQDsrNgUi7BE4iLVh7NPROH+myppRCmaxQpb6js49NTogR0fRR2TSDzk5OSwd+9eJY4U4pxj79695OTk+C6j4akoNS0hEu2cxoCCIvaRRu2eLbEISyTmCgoKqKysZPfu3YkORWIoJyeHggL/K1EoaUQpUOOtcBvhYoVNMjOzCGT2puGAbruV41NmZibFxcWJDkMSTMNTUYpVTwOgpssgsqq3RX0dEZF4UdKIUtOcRiT7gx+lxyBy63TbrYgcv5Q0ohTNBkytZeUXkuVq2b9PY8YicnxS0ohSU0+je5S33ALk9g/ddqvVbkXkeKWkEaVAbT25ORmkp0W3ciRAn0EnA7B/u267FZHjk5JGlAI1wZgMTQEMLCjGYdTs+Tgm1xMRiTUljShVRbmXRktZ2dkczOxNw4GtMbmeiEisKWlEKdpd+1qr7TJAt92KyHFLSSNK0e7a11pj7iC61e2M2fVERGJJSSNKB2tjN6cBkJk/hJzGw+zftydm1xQRiRUljSiFehqxG57K7VcEwI6KDTG7pohIrPhKGmY2zcw+MLMNZjbnGO+bmd3vvf+umY3tqKyZ5ZvZAjP70Pveq8V7P/DO/8DMPtvi+FVmttqr42UzO3p/x07U0Og4WBfbnkafwaG1fXTbrYgcjzpMGmaWDjwIXASUAFeZWUmr0y4Chnlfs4GHfJSdAyx0zg0DFnqv8d6fCYwApgG/MbN0M8sA7gPOdc6NAt4Fboyw3TFxqOlp8BjOaQwoGIrDOKzbbkXkOORnXOUsYINzbhOAmT0DzADWtjhnBvCUCy20/6aZ9TSzgUBRO2VnAOVe+SeBRcCt3vFnnHN1wEdmtsGLYSVgQDcz2wv0ABI6hvPJYoWxG57KyenCoYxeZG5ezCuPVMXsusdSfPaVDCsZE9c6RCS1+Pm0GwxUtHhdCYz3cc7gDsr2d85tB3DObTezfi2u9WbraznnlpvZt4DVQDXwIXDDsQI2s9mEejwMGTLERxMjUxWjDZhaqx44kbyti8jZtjim120pp/EwmxYeZFjJA3GrQ0RSj5+kcaz1MVpv3dXWOX7K+qrPzDKBbwFjgE3AfwE/AH5x1MnOPQI8AlBWVha3bcaaehoxWeG2hQuvvTOm1zuWBffMIjugITARCY+fifBKoLDF6wKg9dNnbZ3TXtmd3hAW3vddHVxrNIBzbqM3DDYPmOQj/rhp3oAphhPhnSb/ZHrWbaO+/kiiIxGRJOInabwFDDOzYjPLIjRJPb/VOfOBr3p3UU0Aqryhp/bKzgdmeT/PAp5vcXymmWWbWTGhyfUVwFagxMz6euddAKwLs70x9cn+4Mm3AWK3gcNIo4GKTR8kOhQRSSIdfto554JmdiPwNyAdeNw5t8bMrvPefxh4CbiY0MT0YeDr7ZX1Ln0XMM/MvgFsAb7glVljZvMITZYHgRuccw3ANjO7HVhsZvXAx8DXYvA7iFjzRHiMh6c6Q7+iEg4vgx2b1zL01NJEhyMiScLXn8jOuZcIJYaWxx5u8bOjjUnpY5X1ju8FzmujzJ3AUQP7Xp0PH10iMQI19ZhB96zk62kMKT6NtaRRvX19okMRkSSiJ8KjEKgNkpudQVoM9tLobFnZ2VRlD8Tt25ToUEQkiShpRCHWixV2ttrcIrod0h1UIuKfkkYUQsuiJ2/SyOh7MrnBfQQC+xMdiogkCSWNKARqgkl551STnoOHA1CxYU0HZ4qIhChpRCHZexqDTg7dNbVny/sJjkREkoWSRhSSfU5jwMBC6tK6ULdTd1CJiD9KGlEI1AZjvoRIZ7K0NAI5BaQf2JzoUEQkSShpRCjY0MihGO+lkQjBnsX0qKnANTYmOhQRSQJKGhE62LyXRvJOhAPkDBhGdmMNO7ZXdHyyiJzwlDQi9MleGsnd0+hdeDoA2za9l+BIRCQZKGlEqHmF2ySe0wAoPDm0keKBSi1cKCIdU9KIUDx27UuEvJ75HMroRXDPxkSHIiJJQEkjQoE47dqXCNXdisgJbE50GCKSBJQ0IpTMy6IfpfdQ8uq2c6SuLtGRiMhxTkkjQp/s2pfcw1MA3QcOI41GKjbryXARaZ+SRoQCtfWkGXRLwr00WutXFLqDatfmhG6EKCJJQEkjQoGaenJzMpNyL43WCotOo5F0Dm3TciIi0j4ljQgl+xIiLWVlZ3MgeyDs0x1UItI+JY0IhRYrTP6hqSZ1PU6iW/WWRIchIsc5JY0IJfuy6K1l9DmZ7sH9VO3fl+hQROQ4pqQRoaqa1EoavQpCk+EVG7Uhk4i0TUkjQsm+a19rg4aOAGBvhe6gEpG2KWlEKNWGp/oPLKAurQu1OzckOhQROY4paUSgvqGRw0caUuNpcI+lpRHoUkjGgU2JDkVEjmNKGhFo3ksjBZ4Gb6mh51Dyaiq1IZOItElJIwKptFhhS9n9TyHL1bJ928eJDkVEjlNKGhFIlQ2YWusz5DQAtm/SHVQicmxKGhFIlQ2YWis8JXQHVZU2ZBKRNqTWoHwnaepppMoyIk169OhFIKM3fPx3Xv9TTlzrOmXcBRQWD49rHSISe0oaEfhkTiP1fn3VvUsZuHMRvBvfu6jWVbxN4c2PxbUOEYm91PvU6wSpOqcBcOG3fk0gcCCudbz53z8hb9+7uMZGLE0jpCLJREkjAlU19aSnGV2z0hMdSsxZWhp5PfPjWkdO4Ri67v472yo3M3jI0LjWJSKxpT/zIhCoCdIjJwOz5N9LIxEGDR8HwJZ1KxIciYiES0kjAoHa+pS7c6ozFQ8bSb1lc/DjfyU6FBEJk5JGBAIptsJtZ8vIzGR/t2Iy9mpPcpFk4ytpmNk0M/vAzDaY2ZxjvG9mdr/3/rtmNrajsmaWb2YLzOxD73uvFu/9wDv/AzP7bIvjWWb2iJmtN7P3zeyKyJseuUBtaq1wmwiu3wjyaz6mtuZwokMRkTB0mDTMLB14ELgIKAGuMrOSVqddBAzzvmYDD/koOwdY6JwbBiz0XuO9PxMYAUwDfuNdB+BHwC7n3HDvem9E0OaoqacRvZ7FY0ijkQ1r3050KCISBj89jbOADc65Tc65I8AzwIxW58wAnnIhbwI9zWxgB2VnAE96Pz8JXNbi+DPOuTrn3EfABu86ANcA/wHgnGt0zu0Jr7mxkWrLoifC0JIyAHZtWJXgSEQkHH6SxmCgosXrSu+Yn3PaK9vfObcdwPver71rmVlP7/XPzextM3vWzPofK2Azm21mK81s5e7du300MTyptgFTIvTuO4BAZl+C27XOlUgy8ZM0jnVfqfN5jp+yfuvLAAqAvzvnxgLLgbnHuoBz7hHnXJlzrqxv374dVBeeI8FGauobUm4JkUSo7nkquVUfaCl2kSTiJ2lUAoUtXhcA23ye017Znd4QFt73XR1cay9wGPiLd/xZYCyd7GBtai6Lngg5BaV0a6hix/aKjk8WkeOCn6TxFjDMzIrNLIvQJPX8VufMB77q3UU1AajyhpzaKzsfmOX9PAt4vsXxmWaWbWbFhCbXVzjnHPACUO6ddx6wNrzmRi/QvAGTkka0Bp4Sesjv4zV6yE8kWXQ4MO+cC5rZjcDfgHTgcefcGjO7znv/YeAl4GJCk9aHga+3V9a79F3APDP7BrAF+IJXZo2ZzSOUEILADc65Bq/MrcB/m9m9wO6mejpTKi9W2NmKTxvFassksOVdvH9+ETnO+frkc869RCgxtDz2cIufHXCD37Le8b2EegvHKnMncOcxjn8MTPUTc7xU1aTuYoWdLTMzi/1dh5Kxu9M7jCISIT0RHqaA5jRiqrHf6fSq+Zi6uppEhyIiPihphKl51z71NGIi76TRpBNk49p/JjoUEfFBSSNMn/Q0NKcRC8UjQ89t7tygpCGSDJQ0whSoqScjzeiSmXp7aSRC336DCGT0oX7b6kSHIiI+KGmEqWlZdO2lETuHew6ne9UHiQ5DRHxQ0ghT0wZMEjtZg0vpHtzPzu1bEh2KiHRASSNMgdp6LSESYwOGhR7y27x2ZYIjEZGOKGmEKVCjXftibeipZxC0DKo2v5PoUESkA0oaYQrUBnW7bYxlZWezv0sR6XrIT+S4p6QRplBPQ3MasdbQt4ReNZs5UleX6FBEpB1KGmGq0q59cZFXNJoMF2TTB/9KdCgi0g4ljTDU1jdQF2zUnEYcFJecCcCOD7WTn8jxTOMsYTjYvCy6fm2x1m9gIRsyehHcuJh/LBwQ17qKS8roN7Cw4xNF5Cj69AuDFiuMr4P5pQzctRjeuD2u9bz7z5Gc/73/iWsdIqlKSSMMAS2LHlfnfPNXbP14fVzr2PjaU/Td/gaHqw/StVtuXOsSSUVKGmFo3rVPd0/FRU5OF04+9Yy41nFw3zTSti/k/beXMnbKRXGtSyQVaSI8DOppJL/TRk8maBns+WBpokMRSUpKGmHQnEbyy+nSlb25JeTs0F1aIpFQ0ghD0wZMWnsquWWcNJ6eR3awfevHiQ5FJOkoaYQhUFtPVnoa2Rn6tSWzotHnALBh1WsJjkQk+ejTLwxV3hIi2ksjuRWdXMKh9F7UfPRmokMRSTpKGmEIaAmRlGBpaRzqO5qe+1cTrK9PdDgiSUVJIwyB2iC5ms9ICT2GTSKn8TDr12gPD5FwKGmEIdTT0DMaqeDUcefiMLa990aiQxFJKkoaYWjaH1ySX6/8PuztUoxVqqchEg4ljTCE9gdX0kgVjYPL6HN4I1UH9iU6FJGkoaQRhlBPQ8NTqWLAiKkYjvdXaYhKxC8lDZ9q6xs4EmxUTyOFnDrqLI5YDgfW/z3RoYgkDSUNn7SESOrJzMxiX89Suu3+J66xMdHhiCQFJQ2ftIRIasouHk+P4F62fBTfJdlFUoWShk9VzSvcak4jlZwy9jMAfPTO6wmORCQ5KGn4pOGp1DR4yFAOZPanfrOWFBHxQ0nDJ+2lkbpq+48lP7COurqaRIcictxT0vBJu/alrvxTJ5PpjvDBP5cnOhSR456Shk/qaaSu08dNpZF0dq5bnOhQRI57vv5sNrNpwH1AOvCoc+6uVu+b9/7FwGHga865t9sra2b5wB+BImAzcKVzbr/33g+AbwANwLedc39rVd98YKhzbmT4TY5MoLaerIw0cjLTO6tK6STduvdgT/dh9Nr8V974xYq41hUsPpfzvvT9uNYhEk8dJg0zSwceBC4AKoG3zGy+c25ti9MuAoZ5X+OBh4DxHZSdAyx0zt1lZnO817eaWQkwExgBDAJeNbPhzrkGL57LgUMxaHtYtIRIaus79Vp2/mNeXOvIPlhBz/V/oubw9XTp2i2udYnEi5+exlnABufcJgAzewaYAbRMGjOAp5xzDnjTzHqa2UBCvYi2ys4Ayr3yTwKLgFu948845+qAj8xsgxfDcjPrDnwXmA3E9//wVrSESGo7Y8J5MOG8uNbxrxWvYS9+l38te5kJ518R17pE4sXPnMZgoKLF60rvmJ9z2ivb3zm3HcD73s9HfT8H7iE0BNYmM5ttZivNbOXu3bvbO9U3bcAk0Ro57hyq0/MIrH450aGIRMxP0jjW3qbO5zl+yvqqz8xGA6c45/7SQXmcc48458qcc2V9+/bt6HRfArVBPaMhUUlPT+fg4LPpc+CfBAL7Ex2OSET8JI1KoLDF6wJgm89z2iu70xvCwvu+q4NrTQTGmdlmYCkw3MwW+Yg/Jg5qAyaJgYKyS8hwQVYv/b9EhyISET9J4y1gmJkVm1kWoUnq+a3OmQ981UImAFXekFN7ZecDs7yfZwHPtzg+08yyzayY0OT6CufcQ865Qc65IuBsYL1zrjyCNkekqqZe605J1E4fNZ5ARh+q1y5IdCgiEenwT2fnXNDMbgT+Rui22cedc2vM7Drv/YeBlwjdbruB0HzD19sr6136LmCemX0D2AJ8wSuzxszmEZosDwI3NN05lSjOOe3aJzFhaWnUDDmH/pv+wv69u+nVOzbDpyKdxdd4i3PuJUKJoeWxh1v87IAb/Jb1ju8Fjnm7inPuTuDOduLZDHTaMxq19Y3UNzhNhEtMFJ91KYc3/Zl3lzzPOZd9M9HhiIRFT4T78MlihZrTkOidfNooDmQNoP6DVxMdikjYlDR80BIiEkuWlsaRos/Qt/oDdm2v6LiAyHFEScMHLYsusTZs4nQMx5qlre8pETm+KWn40LRrn265lVg56eTT2JszBLdhYaJDEQmLkoYP6mlIPLiTz6NPzSYqP/4w0aGI+Kak4YPmNCQeTps8HYD1f9cQlSQPJQ0fmjZgytXwlMTQoIJidnc9hfSPXkt0KCK+KWn4UFVTT7b20pA4SB9+Ab3qtrLpg9WJDkXEFyUNHwJaQkTiZMTZl+IwNv1DQ1SSHDTe4oOWEJF46dtvEO/klpC/8XkW/2JJXOuqHXAmF37z53GtQ1KfkoYPoV379KuS+Bh8/vVULPmfjjcNiELW4R303/IilR9/lYKThsWvIkl5+iT0IVBbT363rESHISmqZMxkSsZMjmsdu3ZUsu0301m74CkK1NuQKGhOwwft2ifJrt+AAnb2mUh+xQIOHQokOhxJYkoaPoR27VOnTJJb4dQvk+VqWbXgD4kORZKYkkYHnHPqaUhKOH3UePZ2KSbtvf+lsSGhW9RIElPS6EBNfQPBRqe7pyTpWVoa2WOuJK9+J+8s186BEhkljQ58slihkoYkv7JzL6c6vQd7/6EhKomMkkYHqmq0AZOkjqzsbKpPvoT+Vf/i443vJzocSUJKGh1oXuFWPQ1JEaMv/CoNZPDBwicTHYokISWNDjSvcKs5DUkRffoNYFe/SeRvfY1AYH+iw5Eko6TRgaaehtaeklRSfM5XyXJ1rHrl94kORZKMkkYHtGufpKLhpWXs7jqMzHXP0aDbbyUMShodaBqeytWchqSYbuOupEf9bv659K+JDkWSiP587kCgtp4umelkZSi/SmoZVz6Dt5b/hrq//5bXd22Oa10Dhpdx+hkT4lqHdA4ljQ4EarSEiKSmzMwsjpR8gd7vPgKrH41rXTXvPcGufn+h38DCuNYj8adPww4EarWEiKSu8s9fT/W0L9PY2Bi3OvbsrKTqv2fxznP3cuG37olbPdI5lDQ6oA2YJNV1694jrtfP7dGTV4ZMo/+Wl9i8YQ1Fp4yIa30SXxqo74A2YBKJXtll36besln/wn8mOhSJkpJGB6pq1NMQiVZ+n/5UDb+CAftXsvaff090OBIFJY0OaE5DJDYmXnYd1ek92P7Kfbg4zqFIfClptKN5Lw3dPSUSta7dcqkfPYu+1etZteSlRIcjEVLSaEf1kQYanZYQEYmViRd/harM/lQv+Y2eRE9SShrtaF6sUMNTIjGRmZlF1qTr6HVkG8v/+nSiw5EIKGm0o3lZdPU0RGLmrHNnsKfLUNJW/Y7amsOJDkfCpKTRDu3aJxJ7lpZGn/O/Q/eG/Syb/9tEhyNh8jXDa2bTgPuAdOBR59xdrd437/2LgcPA15xzb7dX1szygT8CRcBm4Ern3H7vvR8A3wAagG875/5mZl2BZ4GTveMvOOfmRNxyHwLatU8kLkadeQ4LFp9BvzVPsuK2P8a1rkD3oZx942/JyekS13pOFB1+GppZOvAgcAFQCbxlZvOdc2tbnHYRMMz7Gg88BIzvoOwcYKFz7i4zm+O9vtXMSoCZwAhgEPCqmQ336pnrnHvdzLKAhWZ2kXMubkt0atc+kfg54+pf8K+/PoZrOBK/SuprGLDjdZb84W4u+PpP41fPCcTPn9BnARucc5sAzOwZYAbQMmnMAJ5yzjngTTPraWYDCfUi2io7Ayj3yj8JLAJu9Y4/45yrAz4ysw3AWc655cDrAM65I2b2NlAQYbt90a59IvHTb2AhF1xzW9zreeWhW+j/0V9Y+84FlIyeGPf6Up2fOY3BQEWL15XeMT/ntFe2v3NuO4D3vZ/f+sysJ3ApsPBYAZvZbDNbaWYrd+/e3V7b2lXlzWnkahkRkaQ18Us/4VBGL/a8eLsm3mPAT9KwYxxzPs/xUzas+swsA/gDcH9TD+aok517xDlX5pwr69u3bwfVtS1QW0/XrHQy03W/gEiyyu3Rk+4X/oieR3aw+On/SHQ4Sc/Pp2El0HIR/AJgm89z2iu70xvCwvu+y2d9jwAfOufu9RF7VAI1WkJEJBWcMeE8dgyeRv8tL/LeyiWJDiep+UkabwHDzKzYm4CeCcxvdc584KsWMgGo8oac2is7H5jl/TwLeL7F8Zlmlm1mxYQm11cAmNkvgDzgpvCbGr7QsugamhJJBZO/9EMOZvZh/1/v4HD1wUSHk7Q6TBrOuSBwI/A3YB0wzzm3xsyuM7PrvNNeAjYBG4DfAte3V9YrcxdwgZl9SOjuqru8MmuAeYQmy18GbnDONZhZAfAjoAR428zeMbNvRvsLaE9oWXT1NERSQbfuPcib9mN61O9h6dO/THQ4SctCNzylrrKyMrdy5cqIyv7b/UsY0COHx752ZoyjEpFEeeXRnzBgywtw8a8ZNeEziQ7nuGVmq5xzZa2Pa+ylHYHaeob3z010GCISQ1OunsNb96wg6+WfsWDpE3Gty+UVcu7XfkZmZlZc6+lMShrt0K59IqmnS9duDLj8l2z5v7tJr6+OY02N9Kn4P1577AgXzv4VlpYad2HqE7ENjY2Og9ofXCQlDR8xjuEjnol7PQue/AUDN/6JRX8u4twv3BD3+jpDaqS+OKg+EqTRaQkREYnceV/+ATvyzyJ/9aOsfOPFRIcTE0oabajSYoUiEqW09HTOvvYe9uWcBK/9nA3v/yvRIUVNSaMNWhZdRGKha7dcRlzzAEfSurJz3s3s3tX62ejkoqTRBm3AJCKx0m9AAX0+P5echkOsfuzfk3oNLI29tEFbvYpILA0fMY4V5T+i9+u3sfjhb9O95IK41zn+/M+Tnp4e02sqabQhUOsNT2lOQ0Ri5KxzZ/Dars0MWPM7+HtkDx2Ho/6c6aSnx3bzKX0itkE9DRGJh8988Tvs2Po56mrjP0SVFYeHCpU02tA0p6G9NEQk1gYMHpLoECKmifA2BGqCdM/OIEN7aYiINNMnYhsCtfVaQkREpBUljTYEarSEiIhIa0oabQj1NJQ0RERaUtJoQ1VNULfbioi0ok/FNkwc2ptBPXMSHYaIyHFFSaMNP720JNEhiIgcdzQ8JSIivilpiIiIb0oaIiLim5KGiIj4pqQhIiK+KWmIiIhvShoiIuKbkoaIiPhmzrlExxBXZrYb+DjC4n2APTEMJ1mo3ScWtfvE4rfdJznn+rY+mPJJIxpmttI5V5boODqb2n1iUbtPLNG2W8NTIiLim5KGiIj4pqTRvkcSHUCCqN0nFrX7xBJVuzWnISIivqmnISIivilpiIiIb0oax2Bm08zsAzPbYGZzEh1PPJnZ42a2y8zea3Es38wWmNmH3vdeiYwxHsys0MxeN7N1ZrbGzL7jHU/ptptZjpmtMLN/ee2+3Tue0u0GMLN0M/unmb3ovU75NgOY2WYzW21m75jZSu9YxG1X0mjFzNKBB4GLgBLgKjNL5W38ngCmtTo2B1jonBsGLPRep5ogcItz7nRgAnCD9++c6m2vAz7jnDsDGA1MM7MJpH67Ab4DrGvx+kRoc5NznXOjWzyfEXHblTSOdhawwTm3yTl3BHgGmJHgmOLGObcY2Nfq8AzgSe/nJ4HLOjOmzuCc2+6ce9v7+SChD5PBpHjbXcgh72Wm9+VI8XabWQHwb8CjLQ6ndJs7EHHblTSONhioaPG60jt2IunvnNsOoQ9XoF+C44krMysCxgD/4ARouzdM8w6wC1jgnDsR2n0v8P+AxhbHUr3NTRzwipmtMrPZ3rGI254RhwCTnR3jmO5LTlFm1h34M3CTcy5gdqx//tTinGsARptZT+AvZjYywSHFlZldAuxyzq0ys/IEh5MIk51z28ysH7DAzN6P5mLqaRytEihs8boA2JagWBJlp5kNBPC+70pwPHFhZpmEEsbTzrn/9Q6fEG0HcM4dABYRmtNK5XZPBqab2WZCw82fMbP/IbXb3Mw5t837vgv4C6Eh+IjbrqRxtLeAYWZWbGZZwExgfoJj6mzzgVnez7OA5xMYS1xYqEvxGLDOOffrFm+ldNvNrK/Xw8DMugDnA++Twu12zv3AOVfgnCsi9P/za865L5PCbW5iZt3MLLfpZ+BC4D2iaLueCD8GM7uY0BhoOvC4c+7OxEYUP2b2B6Cc0HLJO4GfAc8B84AhwBbgC8651pPlSc3MzgaWAKv5ZJz7h4TmNVK27WY2itDEZzqhPxrnOefuMLPepHC7m3jDU99zzl1yIrTZzIYS6l1AaDri9865O6Npu5KGiIj4puEpERHxTUlDRER8U9IQERHflDRERMQ3JQ0REfFNSUNERHxT0hAREd/+P3BMuxk6KHizAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"num_epoch = 50\n",
"scheduler = StepLRScheduler(optimizer, decay_t=5, warmup_t=2, warmup_lr_init=1e-5, decay_rate=0.8)\n",
"lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n",
"plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"With warmup\");\n",
"\n",
"num_epoch = 50\n",
"scheduler = StepLRScheduler(optimizer, decay_t=5, decay_rate=0.8)\n",
"lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n",
"plt.plot([i for i in range(num_epoch)], lr_per_epoch, label=\"Without warmup\", alpha=0.8);\n",
"\n",
"plt.legend();"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see by setting up `warmup_t` and `warmup_lr_init`, the `cosine` scheduler first starts with a value of `warmup_lr_init`, then gradually progresses up to the `initial_lr` set in the optimizer which is 1e-4. It takes `warmup_t` number of epochs to go from `warmup_lr_init` to `initial_lr`. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `noise_range_t`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The upper and lower limit of noise. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `noise_pct`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Percentage of noise to add. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `noise_std`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Noise standard deviation. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `noise_seed`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Seed to use to add random noise."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `t_in_epochs`\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If set to False, the learning rates returned for epoch `t` are `None`."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[None, None, None, None, None]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"num_epoch = 50\n",
"scheduler = StepLRScheduler(optimizer, decay_t=5, t_in_epochs=False)\n",
"lr_per_epoch = get_lr_per_epoch(scheduler, num_epoch)\n",
"\n",
"lr_per_epoch[:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `initialize`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If True, then inside each param group of the `optimizer` a new field is set called `initial_{field_name}` where `field_name` refers to the field in param group that we are scheduling. Typically `field_name='lr'`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> NOTE: we leave it up to the reader to try and experiment how the LR annealing scheduling get's updated by adding in noise hyperparams. From here on, this should be a fairly simple exercise. :)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 4
}