{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dogs vs Cats | The quick way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WARNING : Put this file in the /courses/dl1 folder of your Fastai folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Original file : https://github.com/fastai/fastai/blob/master/courses/dl1/lesson1.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Setup the Fastai library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 1) The wrong way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Put these at the top of every notebook, to get automatic reloading and inline plotting\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# This file contains all the main external libs we'll use\n",
    "from fastai.imports import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from fastai.transforms import *\n",
    "from fastai.conv_learner import *\n",
    "from fastai.model import *\n",
    "from fastai.dataset import *\n",
    "from fastai.sgdr import *\n",
    "from fastai.plots import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 2) The quick way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from fastai.conv_learner import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the path to data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is available at http://files.fast.ai/data/dogscats.zip. You can download it directly on your server by running the following line in your terminal. wget http://files.fast.ai/data/dogscats.zip. You should put the data in a subdirectory of this notebook's directory, called data/. Note that this data is already available in Crestle and the Paperspace fast.ai template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put the path to your dogscats folder that corresponds to your installation\n",
    "PATH = \"data/dogscats/\"\n",
    "\n",
    "# Size of images when they are used by GPU\n",
    "sz=224"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note : If needed, script to create training and validation sets based on choose RATIO\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(from Welton Rodrigo Torres Nascimento) https://github.com/weltonrodrigo/mag-fofa/blob/master/scripts/create_sets.sh"
   ]
  },  
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.is_available(), torch.backends.cudnn.enabled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Check your path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "os.listdir(PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "os.listdir(f'{PATH}valid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "files = os.listdir(f'{PATH}valid/cats')[:5]\n",
    "files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "img = plt.imread(f'{PATH}valid/cats/{files[0]}')\n",
    "plt.imshow(img);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Easy steps to train a world-class image classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Define your model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fastai notes : Enable data augmentation, and precompute=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the below if you need to reset your precomputed activations\n",
    "shutil.rmtree(f'{PATH}tmp', ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the lesson1.ipynb, Jeremy Howard uses resnet34.\n",
    "\n",
    "After completing this exercice with resnet34, test resnet50, resnet101 and resnet152. Check if the validation accuracy changes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# our model\n",
    "# If you want to know the list of the pretrained models under pytorch : http://pytorch.org/docs/master/torchvision/models.html\n",
    "\n",
    "arch=resnet34\n",
    "# arch=resnet50\n",
    "# arch=resnet101\n",
    "# arch=resnet152"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# our data transformation\n",
    "tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE of the data object using PATH, the test set, tfms and bs=64 (even if it is the batch default value) (1 line)\n",
    "# data = ...\n",
    "# CODE HERE of the learn object using the pretrained model, the data object and precompute=True (1 line)\n",
    "# learn = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all details about layers of your model\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the layers structure of your model\n",
    "learn.get_layer_groups()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Find the learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fastai notes : Use lr_find() to find highest learning rate where loss is still clearly improving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# CODE HERE of the function to search for the best learning rate (1 line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE of the function to plot the loss versus the learning rate (1 line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Train quickly the last layer of your model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fastai notes : Train last layer from precomputed activations for 1-2 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# CODE HERE for training the learn model with learning rate 1e-2 and 1 epoch (1 line)\n",
    "# learn.fit(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Train the last layer of your model with data augmentation and SGDR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fastai notes : Train last layer with data augmentation (i.e. precompute=False) for 2-3 epochs with cycle_len=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE for setting the learn model to precompute=False (1 line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# CODE HERE for training the learn model with learning rate 1e-2, 3 cycles and one learning rate restart by epoch (1 line)\n",
    "# learn.fit(...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE for saving your model (the values of the parameters/weights) to 224_lastlayer (1 line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5) Allow the layers of the pretrained model to be trained"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fastai notes : Unfreeze all layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE for loading your model (the values of the parameters/weights) from 224_lastlayer (1 line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE for unfreezing the first layers of the model (1 line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6) Get one learning rate by group of layers (diferential learning rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fastai notes : Set earlier layers to 3x-10x lower learning rate than next higher layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# CODE HERE of the function to search for the best learning rate (1 line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE of the function to plot the loss versus the learning rate (1 line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE for setting the lr variable to an array of 3 learning rates (1 line)\n",
    "# lr = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# CODE HERE for training the learn model with the diferential learning rate, 3 cycles, one learning rate restart by epoch\n",
    "# and double the length of a cycle after each cycle (1 line)\n",
    "# learn.fit(...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CODE HERE for saving your model (the values of the parameters/weights) to 224_all (1 line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7) Get the validation accuracy by using TTA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the log of the prediction for the validation set using TTA()\n",
    "# log_preds,y = ...\n",
    "\n",
    "# For each image in the validation set, the TTA() provides with 5 predictions for each class : we need to get the mean of them\n",
    "# To get the probabilities, do not forget to take the exp of the logs\n",
    "# probs = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the accuracy of the validation set\n",
    "accuracy_np(probs, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8) Get the predictions on the test set by using TTA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# It is possible to use TTA on the test set because the test set has been passed to learn\n",
    "log_preds,y = learn.TTA(is_test=True)\n",
    "probs = np.mean(np.exp(log_preds), axis=0)\n",
    "preds = np.argmax(probs, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9) Get the prediction on a specific image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the path to a specific image of the test set (for example : the number 16)\n",
    "files = os.listdir(f'{PATH}/test/')\n",
    "img = open_image(f'{PATH}/test/{files[15]}')\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the transformation to be applied on the image and then get the prediction\n",
    "trn_tfms, val_tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)\n",
    "im = val_tfms(img)\n",
    "pred = learn.predict_array(im[None])\n",
    "result = np.argmax(pred, axis=1)\n",
    "print(f'The class is : {data.classes[result[0]]}')"
   ]
  },
    {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the image after transformation by tfms_from_model()\n",
    "print(im.shape)\n",
    "im_show = np.swapaxes(im,0,2)\n",
    "im_show = np.swapaxes(im_show,0,1)\n",
    "print(im_show.shape)\n",
    "plt.imshow(im_show)\n"
   ]
  },
    {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the scatter diagram of the predictions (probabilities) for the image\n",
    "# nb_classes : the number of classes\n",
    "matplotlib.pyplot.scatter(list(range(nb_classes), np.exp(pred))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing results | Confusion matrix "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = np.argmax(probs, axis=1)\n",
    "probs = probs[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A common way to analyze the result of a classification model is to use a [confusion matrix](http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/). Scikit-learn has a convenient function we can use for this purpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(y, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can just print out the confusion matrix, or we can show a graphical view (which is mainly useful for dependents with a larger number of categories)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Increase the display size of the confusion matrix\n",
    "matplotlib.rcParams['figure.figsize'] = [30,20]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_confusion_matrix(cm, data.classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing results: looking at pictures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 1) Statistics on dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "fn = PATH+data.trn_ds.fnames[0]; fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "img = PIL.Image.open(fn); img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "img.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "size_d = {k: PIL.Image.open(PATH+k).size for k in data.trn_ds.fnames}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "row_sz, col_sz = zip(*size_d.values())\n",
    "row_sz = np.array(row_sz)\n",
    "col_sz = np.array(col_sz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "row_sz[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plt.hist(row_sz);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plt.hist(row_sz[row_sz<1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "len(col_sz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plt.hist(col_sz);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plt.hist(col_sz[col_sz<1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "len(data.trn_ds),len(data.test_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "len(data.classes),data.classes[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Visualize the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as looking at the overall metrics, it's also a good idea to look at examples of each of:\n",
    "1. A few correct labels at random\n",
    "2. A few incorrect labels at random\n",
    "3. The most correct labels of each class (ie those with highest probability that are correct)\n",
    "4. The most incorrect labels of each class (ie those with highest probability that are incorrect)\n",
    "5. The most uncertain labels (ie those with probability closest to 0.5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the label for a val data\n",
    "data.val_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from here we know that 'cats' is label 0 and 'dogs' is label 1.\n",
    "data.classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this gives prediction for validation set. Predictions are in log scale\n",
    "log_preds = learn.predict()\n",
    "log_preds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_preds[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = np.argmax(log_preds, axis=1)  # from log probabilities to 0 or 1\n",
    "probs = np.exp(log_preds[:,1])        # pr(dog)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], 4, replace=False)\n",
    "def rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_val_with_title(idxs, title):\n",
    "    imgs = np.stack([data.val_ds[x][0] for x in idxs])\n",
    "    title_probs = [probs[x] for x in idxs]\n",
    "    print(title)\n",
    "    return plots(data.val_ds.denorm(imgs), rows=1, titles=title_probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plots(ims, figsize=(12,6), rows=1, titles=None):\n",
    "    f = plt.figure(figsize=figsize)\n",
    "    for i in range(len(ims)):\n",
    "        sp = f.add_subplot(rows, len(ims)//rows, i+1)\n",
    "        sp.axis('Off')\n",
    "        if titles is not None: sp.set_title(titles[i], fontsize=16)\n",
    "        plt.imshow(ims[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_img_id(ds, idx): return np.array(PIL.Image.open(PATH+ds.fnames[idx]))\n",
    "\n",
    "def plot_val_with_title(idxs, title):\n",
    "    imgs = [load_img_id(data.val_ds,x) for x in idxs]\n",
    "    title_probs = [probs[x] for x in idxs]\n",
    "    print(title)\n",
    "    return plots(imgs, rows=1, titles=title_probs, figsize=(16,8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. A few correct labels at random\n",
    "plot_val_with_title(rand_by_correct(True), \"Correctly classified\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. A few incorrect labels at random\n",
    "plot_val_with_title(rand_by_correct(False), \"Incorrectly classified\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def most_by_mask(mask, mult):\n",
    "    idxs = np.where(mask)[0]\n",
    "    return idxs[np.argsort(mult * probs[idxs])[:4]]\n",
    "\n",
    "def most_by_correct(y, is_correct): \n",
    "    mult = -1 if (y==1)==is_correct else 1\n",
    "    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_val_with_title(most_by_correct(0, True), \"Most correct cats\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_val_with_title(most_by_correct(1, True), \"Most correct dogs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_val_with_title(most_by_correct(0, False), \"Most incorrect cats\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_val_with_title(most_by_correct(1, False), \"Most incorrect dogs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "most_uncertain = np.argsort(np.abs(probs -0.5))[:4]\n",
    "plot_val_with_title(most_uncertain, \"Most uncertain predictions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annex : get automatically the best learning rate number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_loss_change(sched, sma=1, n_skip=20, y_lim=(-0.01,0.01)):\n",
    "    \"\"\"\n",
    "    Plots rate of change of the loss function.\n",
    "    Parameters:\n",
    "        sched - learning rate scheduler, an instance of LR_Finder class.\n",
    "        sma - number of batches for simple moving average to smooth out the curve.\n",
    "        n_skip - number of batches to skip on the left.\n",
    "        y_lim - limits for the y axis.\n",
    "    \"\"\"\n",
    "    derivatives = [0] * (sma + 1)\n",
    "    for i in range(1 + sma, len(learn.sched.lrs)):\n",
    "        derivative = (learn.sched.losses[i] - learn.sched.losses[i - sma]) / sma\n",
    "        derivatives.append(derivative)\n",
    "        \n",
    "    plt.ylabel(\"d/loss\")\n",
    "    plt.xlabel(\"learning rate (log scale)\")\n",
    "    plt.plot(learn.sched.lrs[n_skip:], derivatives[n_skip:])\n",
    "    plt.xscale('log')\n",
    "    plt.ylim(y_lim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# search for best learning rate\n",
    "lrf=learn.lr_find()\n",
    "#learn.sched.plot_lr()\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_loss_change(learn.sched, sma=20)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (fastai)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "266px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}