{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Tracking Callbacks" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "hide_input": true }, "outputs": [], "source": [ "from fastai.gen_doc.nbdoc import *\n", "from fastai.callbacks.general_sched import * \n", "from fastai import *\n", "from fastai.vision import *" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This module regroups the callbacks that track one of the metrics computed at the end of each epoch to take some decision about training. To show examples of use, we'll use our sample of MNIST and a simple cnn model." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "path = untar_data(URLs.MNIST_SAMPLE)\n", "data = ImageDataBunch.from_folder(path)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class TerminateOnNaNCallback[source]

\n", "\n", "> TerminateOnNaNCallback() :: [`Callback`](/callback.html#Callback)\n", "\n", "A [`Callback`](/callback.html#Callback) that terminates training if loss is NaN. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(TerminateOnNaNCallback)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, training diverges and the loss goes to nan. In that case, there's no point continuing, so this callback stops the training." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total time: 00:04\n", "epoch train_loss valid_loss accuracy\n", "1 nan nan 0.495584 (00:02)\n", "2 nan nan 0.495584 (00:02)\n", "\n" ] } ], "source": [ "model = simple_cnn((3,16,16,2))\n", "learn = Learner(data, model, metrics=[accuracy])\n", "learn.fit(2,1e4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using it prevents that situation to happen." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " 0.00% [0/2 00:00<00:00]\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "
epochtrain_lossvalid_lossaccuracy
\n", "\n", "\n", "
\n", " \n", " \n", " Interrupted\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch/Batch (0/5): Invalid loss, terminating training.\n" ] } ], "source": [ "model = simple_cnn((3,16,16,2))\n", "learn = Learner(data, model, metrics=[accuracy], callbacks=[TerminateOnNaNCallback()])\n", "learn.fit(2,1e4)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class EarlyStoppingCallback[source]

\n", "\n", "> EarlyStoppingCallback(`learn`:[`Learner`](/basic_train.html#Learner), `monitor`:`str`=`'val_loss'`, `mode`:`str`=`'auto'`, `min_delta`:`int`=`0`, `patience`:`int`=`0`) :: [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback)\n", "\n", "A [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback) that terminates training when monitored quantity stops improving. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(EarlyStoppingCallback)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This callback tracks the quantity in `monitor` during the training of `learn`. `mode` can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will stop training after `patience` epochs if the quantity hasn't improved by `min_delta`. " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " 6.00% [3/50 00:07<01:55]\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "
epochtrain_lossvalid_lossaccuracy
10.6927830.6929650.517664
20.6927280.6929650.517664
30.6927240.6929650.517664
\n", "\n", "\n", "
\n", " \n", " \n", " 100.00% [16/16 00:00<00:00]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 4: early stopping\n" ] } ], "source": [ "model = simple_cnn((3,16,16,2))\n", "learn = Learner(data, model, metrics=[accuracy], \n", " callback_fns=[partial(EarlyStoppingCallback, monitor='accuracy', min_delta=0.01, patience=3)])\n", "learn.fit(50,1e-42)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class SaveModelCallback[source]

\n", "\n", "> SaveModelCallback(`learn`:[`Learner`](/basic_train.html#Learner), `monitor`:`str`=`'val_loss'`, `mode`:`str`=`'auto'`, `every`:`str`=`'improvement'`, `name`:`str`=`'bestmodel'`) :: [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback)\n", "\n", "A [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback) that saves the model when monitored quantity is best. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(SaveModelCallback)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This callback tracks the quantity in `monitor` during the training of `learn`. `mode` can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will save the model in `name` whenever determined by `every` ('improvement' or 'epoch'). Loads the best model at the end of training is `every='improvement'`." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class ReduceLROnPlateauCallback[source]

\n", "\n", "> ReduceLROnPlateauCallback(`learn`:[`Learner`](/basic_train.html#Learner), `monitor`:`str`=`'val_loss'`, `mode`:`str`=`'auto'`, `patience`:`int`=`0`, `factor`:`float`=`0.2`, `min_delta`:`int`=`0`) :: [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback)\n", "\n", "A [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback) that reduces learning rate when a metric has stopped improving. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(ReduceLROnPlateauCallback)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This callback tracks the quantity in `monitor` during the training of `learn`. `mode` can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will reduce the learning rate by `factor` after `patience` epochs if the quantity hasn't improved by `min_delta`. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "hide_input": true }, "outputs": [ { "data": { "text/markdown": [ "

class TrackerCallback[source]

\n", "\n", "> TrackerCallback(`learn`:[`Learner`](/basic_train.html#Learner), `monitor`:`str`=`'val_loss'`, `mode`:`str`=`'auto'`) :: [`LearnerCallback`](/basic_train.html#LearnerCallback)\n", "\n", "A [`LearnerCallback`](/basic_train.html#LearnerCallback) that keeps track of the best value in `monitor`. " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_doc(TrackerCallback)" ] } ], "metadata": { "jekyll": { "keywords": "fastai", "summary": "Callbacks that take decisions depending on the evolution of metrics during training", "title": "callbacks.tracker" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 2 }