{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PlateauLRScheduler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we are going to be looking at the `PlateauLRScheduler` in the `timm` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from timm.scheduler.plateau_lr import PlateauLRScheduler\n",
    "from nbdev.showdoc import show_doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"PlateauLRScheduler\" class=\"doc_header\"><code>class</code> <code>PlateauLRScheduler</code><a href=\"https://github.com/rwightman/pytorch-image-models/tree/master/timm/scheduler/plateau_lr.py#L12\" class=\"source_link\" style=\"float:right\">[source]</a></h2>\n",
       "\n",
       "> <code>PlateauLRScheduler</code>(**`optimizer`**, **`decay_rate`**=*`0.1`*, **`patience_t`**=*`10`*, **`verbose`**=*`True`*, **`threshold`**=*`0.0001`*, **`cooldown_t`**=*`0`*, **`warmup_t`**=*`0`*, **`warmup_lr_init`**=*`0`*, **`lr_min`**=*`0`*, **`mode`**=*`'max'`*, **`noise_range_t`**=*`None`*, **`noise_type`**=*`'normal'`*, **`noise_pct`**=*`0.67`*, **`noise_std`**=*`1.0`*, **`noise_seed`**=*`None`*, **`initialize`**=*`True`*) :: `Scheduler`\n",
       "\n",
       "Decay the LR by a factor every time the validation loss plateaus."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(PlateauLRScheduler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `PlateauLRScheduler` as shown above accepts an `optimizer` and also some hyperparams which we will look into in detail below. We will first see how we can train models using the `PlateauLRScheduler` by first using `timm` training docs and then look at how we can use this scheduler as standalone scheduler for our custom training scripts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `PlateauLRScheduler` scheduler with `timm` training script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train models using the `PlateauLRScheduler` we simply update the training script args passed by passing in `--sched plateau` parameter alongside the necessary hyperparams. In this section we will also look at how each of the hyperparams update the `plateau` scheduler. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training command to use `cosine` scheduler looks something like: \n",
    "\n",
    "```python \n",
    "python train.py ../imagenette2-320/ --sched plateau\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `PlateauLRScheduler` by default tracks the `eval-metric` which is by default `top-1` in the `timm` training script. If the performance plateaus, then the new learning learning after a certain number of epochs (by default 10) is set to `lr * decay_rate`. This scheduler underneath uses PyTorch's [ReduceLROnPlateau](https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#ReduceLROnPlateau). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Args"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All arguments passed to this scheduler are the same as PyTorch's `ReduceLROnPlateau` except they are renamed as follows: \n",
    "\n",
    "| TIMM      | PyTorch |\n",
    "| ----------- | ----------- |\n",
    "| patience_t      | patience       |\n",
    "| decay_rate   | factor        |\n",
    "| verbose      | verbose       |\n",
    "| threshold   | threshold        |\n",
    "| cooldown_t   | cooldown        |\n",
    "| mode   | mode        |\n",
    "| lr_min   | min_lr        |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The functionality is very similar to [ReduceLROnPlateau](https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#ReduceLROnPlateau) except the addition of Noise."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}