{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Learning Rate Finder" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [], "source": [ "%matplotlib inline\n", "from fastai.gen_doc.nbdoc import *\n", "from fastai.vision import *\n", "from fastai.callbacks import *" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Learning rate finder plots lr vs loss relationship for a [`Learner`](/basic_train.html#Learner). The idea is to reduce the amount of guesswork on picking a good starting learning rate.\n", "\n", "**Overview:** \n", "1. First run lr_find `learn.lr_find()`\n", "2. Plot the learning rate vs loss `learn.recorder.plot()`\n", "3. Pick a learning rate before it diverges then start training\n", "\n", "**Technical Details:** (first [described]('https://arxiv.org/abs/1506.01186') by Leslie Smith) \n", ">Train [`Learner`](/basic_train.html#Learner) over a few iterations. Start with a very low `start_lr` and change it at each mini-batch until it reaches a very high `end_lr`. [`Recorder`](/basic_train.html#Recorder) will record the loss at each iteration. Plot those losses against the learning rate to find the optimal value before it diverges." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Choosing a good learning rate" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For a more intuitive explanation, please check out [Sylvain Gugger's post](https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "path = untar_data(URLs.MNIST_SAMPLE)\n", "data = ImageDataBunch.from_folder(path)\n", "def simple_learner(): return Learner(data, simple_cnn((3,16,16,2)), metrics=[accuracy])\n", "learn = simple_learner()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First we run this command to launch the search:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

lr_find[source]

\n", "\n", "> lr_find(**`learn`**:[`Learner`](/basic_train.html#Learner), **`start_lr`**:`Floats`=***`1e-07`***, **`end_lr`**:`Floats`=***`10`***, **`num_it`**:`int`=***`100`***, **`stop_div`**:`bool`=***`True`***, **`wd`**:`float`=***`None`***)\n", "\n", "Explore lr from `start_lr` to `end_lr` over `num_it` iterations in `learn`. If `stop_div`, stops when loss diverges. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(Learner.lr_find)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n" ] } ], "source": [ "learn.lr_find(stop_div=False, num_it=200)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then we plot the loss versus the learning rates. We're interested in finding a good order of magnitude of learning rate, so we plot with a log scale. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Min numerical gradient: 7.59E-03\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEKCAYAAADXdbjqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8XXWd//HX52Zt2nRPtzQ0pXShdKWhgggWhVK2ll1wGVBHfqCIC+rAjIKCOIAKowMMUxFHR5FBVCxaKRUo+9IUaKGl0HSjG3QjXbPfz++Pe9rehjRNm3ty7r15Px+P88g937Pcd27afHLO95zvMXdHRESkPWJRBxARkcynYiIiIu2mYiIiIu2mYiIiIu2mYiIiIu2mYiIiIu2mYiIiIu2mYiIiIu2mYiIiIu2WG3WAVOnbt6+Xl5dHHUNEJKMsWLBgs7uXtHc/oRYTM5sG/AzIAe5z91ubLb8TOCWYLQL6uXvPYNllwHeDZT9091+39l7l5eVUVlamMr6ISNYzs9Wp2E9oxcTMcoC7gdOAtcB8M5vl7kv2rOPu30ha/6vAxOB1b+BGoAJwYEGw7Qdh5RURkcMXZp/JZKDK3Ve4ez3wIDCjlfUvBX4fvD4dmOvuW4MCMheYFmJWERFphzCLSSmwJml+bdD2IWY2BBgKPHmo24qISPTS5WquS4CH3b3pUDYysyvMrNLMKjdt2hRSNBEROZgwi8k6oCxpfnDQ1pJL2HeKq83buvtMd69w94qSknZfjCAiIocpzGIyHxhuZkPNLJ9EwZjVfCUzGwX0Al5Map4DTDWzXmbWC5gatImISBoK7Woud280s6tJFIEc4H53X2xmNwGV7r6nsFwCPOhJj3x0961mdjOJggRwk7tvDSuriIi0j2XLY3srKiq8PfeZ1DY0Ub27ATNoaIqzu76JXXWNe782NDlN7sTjTtydpuBr3KEp7njQ1uTsfZ0TM4oLczGMusYmahviNMTjeLANQEFujC75ORTmJaYuwZSfGyM3x8iNGTkxIy8nRl5OjMK8GIW5OeTlJg4q3R0H9v4YHRzHnaA9sbymvoltNQ1sr21gd13rXVMOie8tHnzPwff0ofUc6hqbaIw7OWbEYkbMDGtl39bawqT97hGLQUFuDgW5scTXvNi+17mxYH7P8hi5OenSDSiSGcxsgbtXtHc/WXMH/OHauL2WU+94mu21jVFHkRTIjVlQZPYVmJaKULfCXHp0yaN7YV7ia5d98z2L8ulZlEevony65OdE/S2JZIROX0x6FuVz3sRS+nUvpFdRPo6TF4tRVJBD1/xcivJzKMrPpSAvRswgZokjhVjwl3iOWaI9aEv8hZ5YrzHu7KhtwMz2/mLLy4kltg3+Qq9rjFPT0ERtMNXUx6ltbKKuIU5jPE5T3GmMJ4509hzd1DY0Ud8YxwwM2/vXvgUvDIJliTYzKMzL2fuLsyg/h9hBDhH2fA97vtc9+2t+fFKYl0NuzPYdqcUPvE//0NbNlvu+I5c9xzdN7tQ3xqlrbKKuMU5dw77XtQ172oKve9ZrSHq933pxdtc3snVXnJ0bG9le28D2mgbircTqmp9Dv+6FlBQXUFJcQL/ga2nPLowe2J3yvl3J09GQiIpJfm6MH8wYE9r+e3TJa3V5bk6MrgWd/scQmXjc2VXfyPbaRrbtbmBbTQPVu+uprmlg6656Nu+sY+OOOjZtr2PJ+u3M217Lrvp9pwnNoH9xIccM6s7YwT0o79OVSUN6Uda7KMLvSqTj6beYdGqxmFFcmEdxYR6lPbu0aZtddY2s+WA3i9dt592tu1nzwW4WrqnmiaUb965zXHkvzp1YylljB9KzKD+s+CJpQx3wIilS29DE6i27+cdb7/Pn19ZRtXEn+TkxPjGqH1/42FAmD+0ddUSRD0lVB7yKiUgI3J3F67fzp1fX8ZfX17FlVz2nje7PzTPGMKBHYdTxRPZSMWlGxUTSVW1DE798biV3PVlFUX4Od35qAieP0IgNkh5SVUx0GYpIyArzcvjKKUfx6FdPpE+3fC771SvcMfedvfcaiWQDFRORDnJUv2Ie+cqJXHDsYH7+xDK+/LsFNDS1ci21SAZRMRHpQEX5ufzkovF896yjmbP4fb7+4Os0qqBIFtClwSIR+OeTjsQdbpn9Fj2L8vjhuWP23nQqkolUTEQi8qWTj2TLrnrufXo5g3sVcdWUYVFHEjlsKiYiEfrO6SNZV13D7XOWcvTAYqaM7Bd1JJHDoj4TkQjFYsbtF4xjZP9ivvbg66zZujvqSCKHRcVEJGJd8nP4789NIu7OVb9bQG3DIT29WiQtqJiIpIEhfbpy58UTeHPddr73yJstPj9GJJ2pmIikiVNH9+fqU47iDwvWMmfx+1HHETkkKiYiaeRrpw5nZP9ibv7rEnbX64FtkjlUTETSSF5OjJvPHcO66hrufqoq6jgibaZiIpJmJg/tzfkTS5n5zAqWb9oZdRyRNlExEUlD1595NIW5OXx/1mJ1xktGCLWYmNk0M3vbzKrM7LoDrHOxmS0xs8Vm9kBSe5OZvR5Ms8LMKZJuSooLuHbqCJ5dtpnH3nwv6jgiBxVaMTGzHOBu4AxgNHCpmY1uts5w4HrgRHc/Bvh60uIad58QTNPDyimSrj57/BCO6teNO//xDnENVy9pLswjk8lAlbuvcPd64EFgRrN1vgTc7e4fALj7RkQEgNycGF/9xFG88/5O5izW0YmktzCLSSmwJml+bdCWbAQwwsyeN7OXzGxa0rJCM6sM2s8NMadI2jp73CCO7NuVnz2xTEcnktai7oDPBYYDU4BLgV+YWc9g2ZDgUZKfBv7DzD40pKqZXREUnMpNmzZ1VGaRDpMTM66cMoyl7+3g5ZVbo44jckBhFpN1QFnS/OCgLdlaYJa7N7j7SuAdEsUFd18XfF0BzAMmNn8Dd5/p7hXuXlFSomdqS3Y6Z9wguubn8OfX1kYdReSAwiwm84HhZjbUzPKBS4DmV2U9QuKoBDPrS+K01woz62VmBUntJwJLQswqkra65Odw5tiBzH7jPWrqNQikpKfQiom7NwJXA3OAt4CH3H2xmd1kZnuuzpoDbDGzJcBTwLfdfQtwNFBpZguD9lvdXcVEOq0LJg1mZ10jjy9RR7ykJ8uWG6IqKiq8srIy6hgioYjHnZNuf4ph/brxmy9MjjqOZBEzWxD0T7dL1B3wItIGsZgxY8Ignq/azNZd9VHHEfkQFRORDHH2uEE0xV13xEtaUjERyRBHDyzmyL5d+dsb66OOIvIhKiYiGcLMOGvcQF5cvoXNO+uijiOyHxUTkQxy1riBxB2d6pK0o2IikkFG9i9mWElX/rZoQ9RRRPajYiKSQcyMs8cN4uWVW9i4ozbqOCJ7qZiIZBid6pJ0pGIikmFG9C9mRP9u/FWnuiSNqJiIZKCzxg5i/qqtbNyuU12SHlRMRDLQ6WP64w7/eEvPk5P0oGIikoFG9i/miN5FzNXAj5ImVExEMpCZcdro/jy/fAu76hqjjiOiYiKSqU4b3Z/6xjjPvKOnjEr0VExEMlTFkF70LMpj7pL3o44iomIikqlyc2JMGVHC0+9sIh7PjucSSeZSMRHJYCePKGHLrnqWbNgedRTp5FRMRDLYScNLAHha/SYSMRUTkQxWUlzA6IHd1QkvkVMxEclwJ48oYcHqD9ipS4QlQiomIhnu5BF9aYw7L1RtjjqKdGKhFhMzm2Zmb5tZlZldd4B1LjazJWa22MweSGq/zMyWBdNlYeYUyWQVQ3rTrSCXp97W0CoSndywdmxmOcDdwGnAWmC+mc1y9yVJ6wwHrgdOdPcPzKxf0N4buBGoABxYEGz7QVh5RTJVfm6Mj48o4Ym3NhKPO7GYRR1JOqEwj0wmA1XuvsLd64EHgRnN1vkScPeeIuHue/60Oh2Y6+5bg2VzgWkhZhXJaJ88uh8bd9Tx5vptUUeRTirMYlIKrEmaXxu0JRsBjDCz583sJTObdgjbikjglJH9iBn8Q3fDS0Si7oDPBYYDU4BLgV+YWc+2bmxmV5hZpZlVbtqkSyOl8+rVNZ+KIb01JL1EJsxisg4oS5ofHLQlWwvMcvcGd18JvEOiuLRlW9x9prtXuHtFSUlJSsOLZJpPHt2PJRu2s2FbTdRRpBMKs5jMB4ab2VAzywcuAWY1W+cREkclmFlfEqe9VgBzgKlm1svMegFTgzYROYCPj0z8QfXsMl0iLB0vtGLi7o3A1SSKwFvAQ+6+2MxuMrPpwWpzgC1mtgR4Cvi2u29x963AzSQK0nzgpqBNRA5gZP9i+hUX6G54iURolwYDuPtsYHazthuSXjvwzWBqvu39wP1h5hPJJmbGScNLeGLp+zTFnRxdIiwdKOoOeBFJoZNH9KV6dwNvrtMlwtKxVExEssiJR/UF4NllOtUlHUvFRCSL9O1WwJjS7szVJcLSwVRMRLLM2eMGsXBNNSs374o6inQiKiYiWWbGhEGYwSOvfejWLJHQqJiIZJmBPbpwwpF9eOT1dSQumBQJn4qJSBY6b2Ipq7fs5tV3q6OOIp2EiolIFpo2ZgAFuTGd6pIOo2IikoWKC/OYeswAHl20nvrGeNRxpBNQMRHJUudNHET17gae1vAq0gFUTESy1EnDS+jTNZ8/v7Y26ijSCaiYiGSpvJwY54wfxD/e2siuusao40iWUzERyWKnHt2f+sY4L6/cEnUUyXIqJiJZ7Limrdwy97848dhhEItB9+7w5S/D8uVRR5Mso2Iikq3+/ncKJk3kUwvnUFCzC9xhxw647z4YNw7+/veoE0oWUTERyUbLl8OFF8Lu3eQ2NesvaWiA3bsTy3WEIimiYiKSjX7600TRaE1DA9x5Z8fkkaynYiKSjX7727YVk//9347JI1lPxUQkG+3cmdr1RA5CxUQkG3Xrltr1RA5CxUQkG332s5CX1/o6eXnwuc91TB7JeqEWEzObZmZvm1mVmV3XwvLLzWyTmb0eTP+ctKwpqX1WmDlFss6117atmHzjGx2TR7Jeblg7NrMc4G7gNGAtMN/MZrn7kmar/p+7X93CLmrcfUJY+USy2rBh8PDDict/Gxr264z3vDwsLy+xfNiwCENKNgnzyGQyUOXuK9y9HngQmBHi+4lIsjPOgEWL4IoroHt33Iwd+UXs+KfPJ9rPOCPqhJJFwiwmpcCapPm1QVtzF5jZIjN72MzKktoLzazSzF4ys3NDzCmSvYYNg7vugm3bWP7edsZ+4yEe+eL1OiKRlIu6A/5RoNzdxwFzgV8nLRvi7hXAp4H/MLMP/es3syuCglO5aZOe2SDSmmElXRnerxt/Xbgh6iiShcIsJuuA5CONwUHbXu6+xd3rgtn7gElJy9YFX1cA84CJzd/A3We6e4W7V5SUlKQ2vUiWMTPOGT+I+au3smFbTdRxJMuEWUzmA8PNbKiZ5QOXAPtdlWVmA5NmpwNvBe29zKwgeN0XOBFo3nEvIofo7HEDcYe/LdLRiaRWaMXE3RuBq4E5JIrEQ+6+2MxuMrPpwWrXmNliM1sIXANcHrQfDVQG7U8Bt7ZwFZiIHKIjS7pxzKDuPKpiIikW2qXBAO4+G5jdrO2GpNfXA9e3sN0LwNgws4l0VjMmDOJHs5ey9L3tjBrQPeo4kiXadGRiZsOSTjtNMbNrzKxnuNFEJAwXV5RRmBfj1y+sijqKZJG2nub6I9BkZkcBM0l0rD8QWioRCU3PonzOm1jKn19bxwe76qOOI1mircUkHvSBnAf8p7t/Gxh4kG1EJE1d9tFyahviPPDKu1FHkSzR1mLSYGaXApcBfw3aDjLwj4ikq1EDuvOJUf2468kqVm3eFXUcyQJtLSafB04AbnH3lWY2FNBTdUQy2C3njSE3x7j2DwtpinvUcSTDtamYuPsSd7/G3X9vZr2AYne/LeRsIhKigT26cNOMY1iw+gP+/Nq6g28g0oq2Xs01z8y6m1lv4FXgF2Z2R7jRRCRs504oZWT/Yu57dgXuOjqRw9fW01w93H07cD7wG3f/CHBqeLFEpCOYGV/82FCWvreDF5ZviTqOZLC2FpPcYOiTi9nXAS8iWWD6hEH07ZbPfc+uiDqKZLC2FpObSAyLstzd55vZkcCy8GKJSEcpzMvh0slHMO+dTWzcXht1HMlQbe2A/4O7j3P3q4L5Fe5+QbjRRKSjnDN+EO4wZ/F7UUeRDNXWDvjBZvZnM9sYTH80s8FhhxORjjGifzFH9evG7DdUTOTwtPU0169IDB8/KJgeDdpEJEucOWYAL6/cwuaddQdfWaSZthaTEnf/lbs3BtP/AHoalUgWOWPsQOI61SWHqa3FZIuZfdbMcoLps4CuIxTJIqMGFDNqQDH3PLWcnXWNVG3cyTPv6HHY0jZtLSZfIHFZ8HvABuBC9j3ISkSygJlxy3ljWb+thqt+u4AZdz3H5b96hYVrqqOOJhmgrVdzrXb36e5e4u793P1cQFdziWSZSUN68c8fG8qzyzZT1ruIkuICrvvTGzQ0xaOOJmmuPY/t/WbKUohI2rh26khuv3Acf7jyBH4wfQxvbdjOb15cHXUsSXPtKSaWshQikjYK83K4uKKM4sI8po0ZQMWQXvz+lXc1dpe0qj3FRP+yRDqB844tpWrjThav3x51FEljrRYTM9thZttbmHaQuN9ERLLcWWMHkpdj/OV1DVMvB9ZqMXH3Ynfv3sJU7O65B9u5mU0zs7fNrMrMrmth+eVmtsnMXg+mf05adpmZLQumyw7v2xOR9upZlM/HR/TjL6+vp1Ed8XIA7TnN1SozywHuBs4ARgOXmtnoFlb9P3efEEz3Bdv2Bm4EPgJMBm4MHsolIhG4cNJgNu6o48TbnuT2x5by3jYNCCn7C62YkCgCVcGgkPXAg8CMNm57OjDX3be6+wfAXGBaSDlF5CBOP6Y/9352EmMG9eDep5dz0u1P8ujC9VHHkjQSZjEpBdYkza8N2pq7wMwWmdnDZlZ2iNuKSAcwM6aNGcAvLz+Oed86hSF9uvILPf9EkoRZTNriUaDc3ceROPr49aFsbGZXmFmlmVVu2qRhH0Q6whF9iri4YjCL1m5j1eZdUceRNBFmMVkHlCXNDw7a9nL3Le6+Z4jS+4BJbd022H6mu1e4e0VJicadFOkoZ41LXMz510U61SUJYRaT+cBwMxtqZvnAJSSGsd8reBTwHtOBt4LXc4CpZtYr6HifGrSJSBoo7dmF48p78ejCDVFHkTQRWjFx90bgahJF4C3gIXdfbGY3mdn0YLVrzGyxmS0EriEYPNLdtwI3kyhI84GbgjYRSRPnjB/E2+/v4OUVGkA8Ssve38H66pqoY2DZMkRCRUWFV1ZWRh1DpNPYVdfI1DufoSAvxuxrTqIwLyfqSJ1OfWOc6Xc9R2PcefzrJxOLHfooV2a2wN0r2psl6g54EclQXQty+ffzx7Ji0y6++8ibrN6izviOdtdTVSx9bwfXnzHqsApJKqmYiMhhO3lECZ8/sZyHF6zl4z+ex/++pNGFO8ri9du456kqzptYyieP7h91HBUTEWmfG885hme/cwrHlffiZ/9YRm1DU9SRsl5T3Ln+T2/QsyiPG85uaWCRjqdiIiLtVta7iG+cNoLNO+v446tro46T9X79wioWrd3GDeccQ6+u+VHHAVRMRCRFTjiyD+PLevLfT6+gpl5HJ2FZV13DTx5/mykjSzhn3MCDb9BBVExEJCXMjK+fOpw1H+zmvHue193xIXB3bnjkTdzh5hljMEufZxSqmIhIypwysh/3X34c722v5eL/fpGN2zW6cCr9/c33eGLpRq6dOoKy3kVRx9mPiomIpNQpI/vx4BXHs6O2kS//7lXqG/UMlFTYVtPAjbMWM6a0O5d/tDzqOB+iYiIiKTdqQHduu3Aclas/YOYzy6OOkxVue2wpW3bWcev548jNSb9f3emXSESywvTxg/jEqH7c//wqdci304LVW3ng5Xf54seGMqa0R9RxWqRiIiKhuWrKMLbuquehyjUHX1la1BR3vvvIYgb1KOQbp42IOs4BqZiISGiOK+9NxZBezHxmBQ16fvxheeDl1by1YTvfPXs0Rfm5Ucc5IBUTEQnVVVOGsa66Rs8+OQxbdtbx4zlvc+JRfThjzICo47RKxUREQnXKyH6M7F/Mf81bTjyeHaOUd5SfPP42u+ub+P45x6TVPSUtUTERkVDFYsaVU47knfd38uTSjVHHyRgL11Tz4Pw1fP7Ecob3L446zkGpmIhI6M4eN4jSnl2477kVUUfJCPG4c8OsxfTtVsA1nxwedZw2UTERkdDl5cT41HFlvLxya1o8FTDdPbxgLQvXVPOvZ46iuDAv6jhtomIiIh1i+vhBuKOO+IPYVtPAbY8t5bjyXpw7oTTqOG2mYiIiHaK8b1fGl/Vk1kIVk9Y8t2wzW3bV8+3TR6V9p3syFRMR6TDTxw/izXXbqdq4M+ooaWvPacCRA9K/0z2ZiomIdJhzxg0kZvAnPUDrgNZV19CtIJfuhel7g2JLQi0mZjbNzN42syozu66V9S4wMzezimC+3MxqzOz1YLo3zJwi0jH6dS/kE6P681DlWt0RfwDrq2sY1LMwo05xQYjFxMxygLuBM4DRwKVm9qGHFZtZMfA14OVmi5a7+4RgujKsnCLSsT7zkSPYvLOOuUvejzpKWlq/rYZBPbtEHeOQhXlkMhmocvcV7l4PPAjMaGG9m4HbAD1FR6QTOHlECaU9u/DAy+9GHSUtra+uZWAPFZNkpUDyUKFrg7a9zOxYoMzd/9bC9kPN7DUze9rMTgoxp4h0oJyY8anjyniuarMe7dtMbUMTW3fVU9qzMOoohyyyDngziwF3ANe2sHgDcIS7TwS+CTxgZt1b2McVZlZpZpWbNm0KN7CIpMynjisjJ2b8fr6OTpLtuZJLp7n2tw4oS5ofHLTtUQyMAeaZ2SrgeGCWmVW4e527bwFw9wXAcuBDA/m7+0x3r3D3ipKSkpC+DRFJtf7dC/nkqH48XLmWukY9OGuP9dWJs/0qJvubDww3s6Fmlg9cAszas9Ddt7l7X3cvd/dy4CVgurtXmllJ0IGPmR0JDAc0qI9IFvn0R45gy6565ixWR/wee45MSjOwmIR2IbO7N5rZ1cAcIAe4390Xm9lNQKW7z2pl85OBm8ysAYgDV7r71rCyikjHO3l4CWW9u3Db35dyVEk3Fq2tZsmG7Xz3rNHk53bOW+DWb6vBLHHklmlCvSvG3WcDs5u13XCAdackvf4j8Mcws4lItGIx465Lj+VLv6nkzJ8/u7e9a0Eu/zJtVITJorO+uoaSbgUZWUwzL7GIZI3xZT159Ksf41MVZdzzmWO55Lgy7n16OS+v2BJ1tEisr67NyP4SUDERkYj1717IbReO48yxA/ne2aMZ3KsLtz62NOpYkVhfXZOR/SWgYiIiaaRrQS6fO34Ir71bzYpNnWswSHdnw7ZaBvTIvP4SUDERkTQzY0JpMBjkuoOvnEV21TdR09BEv+KCqKMcFhUTEUkr/bsX8rHhJfz5tXXE4x51nA6zaUcdAH27qZiIiKTEBceWsq66hpdXdp47AjbvTBSTEh2ZiIikxtTRA+hWkNupnnuyWUcmIiKp1SU/hzPHDmD2Gxuoqe8cw61sCo5M+hbnR5zk8KiYiEhaOv/Yweyqb2LO4veijtIhNu+oI2bQp6uOTEREUmZyeW9Ke3bhj53kVNemnfX07ppPTiyznrC4h4qJiKSlWMy4YNJgnqvazOot2f/ck8076zK2vwRUTEQkjX3mI0eQGzN+9fyqqKOEbtOOuoy9kgtUTEQkjfXvXsg54wfxUOUattU0RB0nVDoyEREJ0Rc/NpTd9U388tnsfaSRuwfFJDOv5AIVExFJc8cM6sFZ4wby8yeruP2xpVl5V/zOukZqG+IZfZor1OeZiIikwn98agLdC3O5Z95ySooL+PyJQ6OOlFKbd9YDmXvDIujIREQyQF5OjB+dN5aTR5Tw08ffYcO2mqgjpdSeoVRUTEREQmZm/HDGGBqa4tz4l8W4Z8/prj2DPGbyaS4VExHJGEf0KeLaqSN4fMn7/OeTVVHHSRkdmYiIdLAvnXQk5x9byh1z3+EvryeeeVJT38SarbsjTnb49gyl0rurruYSEekQZsat54+jYkgvvvfIm7y3rZbL7n+FKT+Zx91PVdGUIVd7bdlZR019E7UNTTxbtZl+xYUZO5QKhFxMzGyamb1tZlVmdl0r611gZm5mFUlt1wfbvW1mp4eZU0QyS35ujFsvGEdtQ5xz7nqOV1ZtZWJZT348521OveNpfvncyrQuKuura/jET5/mY7c9yad/8RKvvVvN9WeOijpWu4RWTMwsB7gbOAMYDVxqZqNbWK8Y+BrwclLbaOAS4BhgGnBPsD8REQCO6teNq6YMY9OOOi6uGMwfrjyBez5zLH265nPzX5fw+1fejTpii+Jx59qHFtLYFGfkgGJefbeaH0w/hhkTSqOO1i5h3mcyGahy9xUAZvYgMANY0my9m4HbgG8ntc0AHnT3OmClmVUF+3sxxLwikmG+cspRjOhfzCeP7oeZcebYgZwxZgAX3vsi//nkMi6cNJjCvPT5O9Td+enct3lxxRZuv2AcFx9XRvXuenoWZW5fyR5hnuYqBdYkza8N2vYys2OBMnf/26FuKyKSnxvjrHED9ysYZsa3po7k/e11/Pal1RGm219jU5zvPvImdz+1nIsrBnNRxWCArCgkEOEd8GYWA+4ALm/HPq4ArgA44ogjUhNMRDLeCcP6cNLwvtz1VBXnTSylT7cCahuaOuQo5fmqzWzYVsvEI3oyf+VW1m+rZXJ5b+56ahkvrdjKlR8fxr9MG4lZ5na2tyTMYrIOKEuaHxy07VEMjAHmBR/qAGCWmU1vw7YAuPtMYCZARUVF+va2iUiH+97Zoznr58/yw7+9Rf/uhdz/3EoevuoExg3uGer7fufhRayr/vAd+oV5Me64eDznHzs41PePSpjFZD4w3MyGkigElwCf3rPQ3bcBfffMm9k84FvuXmlmNcADZnYHMAgYDrwSYlYRyTIj+hdz5ceH7b250QxmPrOCuz59bGjvWdfYxPptNVw0aTDjy3oyoawnZb2LeHH5Zkb0L+bIkm5ErSTAAAAM8klEQVShvXfUQism7t5oZlcDc4Ac4H53X2xmNwGV7j6rlW0Xm9lDJDrrG4GvuHtTWFlFJDt95ZSjWLh2G5OO6MXOugbuf34V66trGNSzSyjvt/aDGtwTp9mSj0CmjRkYyvulk1D7TNx9NjC7WdsNB1h3SrP5W4BbQgsnIlmvMC+H33xhMgBrP9jNL59byX3PruR7Zx8dSp/Fu1sSd+EP6VOU8n2nO90BLyKdwuBeRcyYUMr9z6/kkpkvsfS97Sl/j3eDIV3KequYiIhkrVsvGMsPph9D1cadnH/PC8xZ/F5K9796y26K8nMoyeABGw+XiomIdBoFuTlc9tFyZn/tJIb368aVv13AC8s3p2z/727dxRG9i7Lust+2UDERkU6nf/dCfn/F8ZT36cp3Hl7EzrrGlOz33a27O+UpLlAxEZFOqig/lx9fOI511TXc/OiSdj9sy915d+tuhqiYiIh0LhXlvbnq48P4v8o1/OLZFe3a16YdddQ2xDmiE17JBREOpyIikg6+NXUkq7fu5kezl/K3RRsoKS7gE6P6c9bYgfQoymvzflYHV3Id0UmPTFRMRKRTi8WMn140nn7FBSzftItlG3fyj7c2cteTy/i//3dCm/tA9t1j0jXMuGlLxUREOr3CvBxuPOcYINH3MX/VB3zpN5Vc+ouXeOj/nXDQO+YXrN7Kz55YRtf8HEpDurs+3anPREQkiZkxeWhvfvvFj1C9u4Ev/+5V6hvj+63zxtptXPP713j6nU3c/thSLrr3ReLu/M8XJpOf2zl/rXbO71pE5CDGDu7BTy4ax+trqvnR7Lf2W/ZQ5RpmLVzPZfe/wj3zlnPRpDIe+/rJHFfeO6K00dNpLhGRA5g2ZiCXf7Sc/3lhFRdXlDF6UHcAFq2tpmJILz53whD6divgxKP6HmRP2U9HJiIirfjGqSMoys/hl8+tBBLDzC/ZsJ1JQ3oxY0KpCklAxUREpBU9ivK4uKKMWQvXsXF7LUs37KChyRlfFu5DtjKNiomIyEF8/sRyGuPOr19cxaK11QCMG9wj2lBpRn0mIiIHMaRPV84cO5D7n1vFpCG96Nstv9NeAnwgOjIREWmD688YheM8V7WZcYN7dsqRgVujYiIi0gaDexXxlSlHATB+sPpLmtNpLhGRNvrSyUeyo66R848tjTpK2lExERFpo8K8HP71zKOjjpGWdJpLRETaLdRiYmbTzOxtM6sys+taWH6lmb1hZq+b2XNmNjpoLzezmqD9dTO7N8ycIiLSPqGd5jKzHOBu4DRgLTDfzGa5+5Kk1R5w93uD9acDdwDTgmXL3X1CWPlERCR1wjwymQxUufsKd68HHgRmJK/g7tuTZrsC7XtupoiIRCLMYlIKrEmaXxu07cfMvmJmy4HbgWuSFg01s9fM7GkzOynEnCIi0k6Rd8C7+93uPgz4F+C7QfMG4Ah3nwh8E3jAzLo339bMrjCzSjOr3LRpU8eFFhGR/YRZTNYBZUnzg4O2A3kQOBfA3evcfUvwegGwHBjRfAN3n+nuFe5eUVJSkrLgIiJyaMIsJvOB4WY21MzygUuAWckrmNnwpNmzgGVBe0nQgY+ZHQkMB1aEmFVERNohtKu53L3RzK4G5gA5wP3uvtjMbgIq3X0WcLWZnQo0AB8AlwWbnwzcZGYNQBy40t23tvZ+CxYs2Gxmqw8hYg9g2yEsa96WPN/S6+ZtecDmQ8iX6oytZUtu63uIOVvLeDg5W8t2uBkPljMVGVvKm8rPUj/vQ5ONP+/DydmWjEMOIfOBuXunnICZh7KseVvyfEuvm7eRKKCRZWwtW7Ovh5SztYyp/iwPN2NH/LzD/iz189bPO6rPsq1T5B3wEXr0EJc1b3v0IK8PtPxQpDJj8vzB8h6Kg22Xys/ycDMebNtUZEx+rZ936236eR98uwMti+KzbBMLqpWEzMwq3b0i6hwHkwk5MyEjZEZOZUydTMgZZsbOfGTS0WZGHaCNMiFnJmSEzMipjKmTCTlDy6gjExERaTcdmYiISLupmBwGM7vfzDaa2ZuHse2kYKTkKjP7uSU9+9PMvmpmS81ssZndno45zez7ZrYuaUTnM9MtY9Lya83MzaxvumU0s5vNbFHwGT5uZoPakzHEnD8O/k0uMrM/m1m7HjEYUsaLgv8zcTM77P6A9mQ7wP4uM7NlwXRZUnur/27TJOMtZrbGzHa2eWdhXSaWzROJ+2COBd48jG1fAY4HDPg7cEbQfgrwD6AgmO+Xpjm/D3wrnT/LYFkZiXucVgN90y0j0D1pnWuAe9PxswSmArnB69uA29Iw49HASGAeUNHR2YL3LW/W1pvEjda9gV7B614H+3ebRhmPBwYCO9v6HjoyOQzu/gyw302UZjbMzB4zswVm9qyZjWq+nZkNJPFL5CVP/MR+QzCEDHAVcKu71wXvsTFNc6ZUiBnvBL5DCkaiDiOjhzBidkg5H3f3xmDVl0gMi5RuGd9y97fbk6s92Q7gdGCuu2919w+AucC09v7f6oiMwfu85O4b2poLdJorlWYCX3X3ScC3gHtaWKeUxOjJeySPpDwCOMnMXrbESMnHpWlOSIxcsCg45O6VbhnNbAawzt0XhpAtJRmDnLeY2RrgM8AN6ZozyRdI/CWdaqnMGEW2lhxo1PQwvo9UZzwsegZ8CphZN+CjwB+STn8WHOJuckkcbh4PHAc8ZGZHBn+9pFPO/wJuJvGX9M3AT0n8kkmLjGZWBPwridMzoUjR54i7/xvwb2Z2PXA1cGPKQpK6nMG+/g1oBH6XmnR795uyjKnWWjYz+zzwtaDtKGC2mdUDK939vM6YUcUkNWJAtTd7MqQlBqtcEMzOIvGLOPk0QfJIymuBPwXF4xUzi5MYRyeVY+u3O6e7v5+03S+Av6YwXyoyDgOGAguD/1yDgVfNbLK7v5cmGZv7HTCbFBeTVOU0s8uBs4FPpvKPm1RmDEmL2QDc/VfArwDMbB5wubuvSlplHTAlaX4wiX6LdaT2+wgj4+E5lE4cTft1XpWT1AkGvABcFLw2YPwBtmve+XZm0H4lcFPwegSJw09Lw5wDk9b5BvBgumVsts4q2tkBH9LnODxpna8CD6fpv8tpwBKgJBX5wvx5084O+MPNxoE7t1eS6NjuFbzu3dZ/t1FnTFqnzR3wKfnH0dkm4PckHuDVQOKI4osk/hp+DFgY/Oe74QDbVgBvknhGy13su3E0H/htsOxV4BNpmvN/gTeARST+YhyYbhmbrbOK9l/NFcbn+MegfRGJcZNK0/TnXUXiD5vXg6ldV52FlPG8YF91wPvAnI7MRgu/qIP2LwSfXxXw+UP5d5sGGW8P9h8Pvn7/YNl0B7yIiLSbruYSEZF2UzEREZF2UzEREZF2UzEREZF2UzEREZF2UzGRrHZIo56m5v3uM7PRKdpXkyVGFX7TzB61g4zYa2Y9zezLqXhvkUOlS4Mlq5nZTnfvlsL95fq+gQ9DlZzdzH4NvOPut7SyfjnwV3cf0xH5RJLpyEQ6HTMrMbM/mtn8YDoxaJ9sZi+a2Wtm9oKZjQzaLzezWWb2JPCEmU0xs3lm9rAlnvXxO7O9z9aYZ8EzNcxsZzCY40Ize8nM+gftw4L5N8zsh208enqRfQNZdjOzJ8zs1WAfM4J1bgWGBUczPw7W/XbwPS4ysx+k8GMU2Y+KiXRGPwPudPfjgAuA+4L2pcBJ7j6RxCi+P0ra5ljgQnf/eDA/Efg6MBo4EjixhffpCrzk7uOBZ4AvJb3/z9x9LPuPINuiYJyqT5IYcQCgFjjP3Y8l8RycnwbF7DpgubtPcPdvm9lUYDgwGZgATDKzkw/2fiKHQwM9Smd0KjA6aZTV7sHoqz2AX5vZcBKjIuclbTPX3ZOfI/GKu68FMLPXSYyX9Fyz96ln30CYC4DTgtcnsO8ZFg8APzlAzi7BvkuBt0g8bwIS4y/9KCgM8WB5/xa2nxpMrwXz3UgUl2cO8H4ih03FRDqjGHC8u9cmN5rZXcBT7n5e0P8wL2nxrmb7qEt63UTL/5cafF+n5IHWaU2Nu08IhtWfA3wF+DmJ55+UAJPcvcHMVgGFLWxvwL+7+38f4vuKHDKd5pLO6HESI/UCYGZ7hu/uwb7hwC8P8f1fInF6DeCSg63s7rtJPNr3WjPLJZFzY1BITgGGBKvuAIqTNp0DfCE46sLMSs2sX4q+B5H9qJhItisys7VJ0zdJ/GKuCDqll5AY/h8SI6X+u5m9RrhH7V8Hvmlmi0g8tGjbwTZw99dIjDB8KYnnn1SY2RvAP5Ho68HdtwDPB5cS/9jdHydxGu3FYN2H2b/YiKSMLg0W6WDBaasad3czuwS41N1nHGw7kXSmPhORjjcJuCu4AquaFD72WCQqOjIREZF2U5+JiIi0m4qJiIi0m4qJiIi0m4qJiIi0m4qJiIi0m4qJiIi02/8HHyREps84cv0AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "learn.recorder.plot()\n", "min_grad_lr = learn.recorder.min_grad_lr" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we choose a value that is approximately in the middle of the sharpest downward slope. In this case, training with 3e-2 looks like it should work well:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "Total time: 00:06

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracy
10.0859230.0576100.978901
20.0546140.0307950.991168
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "simple_learner().fit(2, 1e-2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Don't just pick the minimum value from the plot!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "Total time: 00:06

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracy
11.3462250.6931470.495584
20.7061180.6931470.495584
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn = simple_learner()\n", "simple_learner().fit(2, 1e-0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Picking a value before the downward slope results in slow training:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "Total time: 00:06

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracy
10.1674990.1425230.946025
20.1282270.1094660.960255
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn = simple_learner()\n", "simple_learner().fit(2, 1e-3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Suggest LR\n", "The red dot on the graph is the point with the minimum numerical gradient. We can use that point as a first guess for an LR" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "Total time: 00:06

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracy
10.0954970.0633200.978410
20.0480790.0418370.983317
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learn = simple_learner()\n", "simple_learner().fit(2, min_grad_lr)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class LRFinder[source]

\n", "\n", "> LRFinder(**`learn`**:[`Learner`](/basic_train.html#Learner), **`start_lr`**:`float`=***`1e-07`***, **`end_lr`**:`float`=***`10`***, **`num_it`**:`int`=***`100`***, **`stop_div`**:`bool`=***`True`***) :: [`LearnerCallback`](/basic_train.html#LearnerCallback)\n", "\n", "Causes `learn` to go on a mock training from `start_lr` to `end_lr` for `num_it` iterations. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(LRFinder)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Callback methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_begin[source]

\n", "\n", "> on_train_begin(**`pbar`**, **\\*\\*`kwargs`**:`Any`)\n", "\n", "Initialize optimizer and learner hyperparameters. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(LRFinder.on_train_begin)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_batch_end[source]

\n", "\n", "> on_batch_end(**`iteration`**:`int`, **`smooth_loss`**:`TensorOrNumber`, **\\*\\*`kwargs`**:`Any`)\n", "\n", "Determine if loss has runaway and we should stop. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(LRFinder.on_batch_end)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_epoch_end[source]

\n", "\n", "> on_epoch_end(**\\*\\*`kwargs`**:`Any`)\n", "\n", "Tell Learner if we need to stop. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(LRFinder.on_epoch_end)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

on_train_end[source]

\n", "\n", "> on_train_end(**\\*\\*`kwargs`**:`Any`)\n", "\n", "Cleanup learn model weights disturbed during LRFind exploration. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(LRFinder.on_train_end)" ] } ], "metadata": { "jekyll": { "keywords": "fastai", "summary": "Implementation of the LR Range test from Leslie Smith", "title": "callbacks.lr_finder" }, "kernelspec": { "display_name": "Python 3.7 fasta.ai1 DEV", "language": "python", "name": "fastai1_dev" } }, "nbformat": 4, "nbformat_minor": 2 }