{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GrFpZXNQB9FW"
      },
      "source": [
        "#### Copyright 2018 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "43_9Kh8LCDPK"
      },
      "outputs": [],
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YHK6DyunSbs4"
      },
      "source": [
        "# Cat vs. Dog Image Classification\n",
        "## Exercise 2: Reducing Overfitting\n",
        "**_Estimated completion time: 30 minutes_**\n",
        "\n",
        "In this notebook we will build on the model we created in Exercise 1 to classify cats vs. dogs, and improve accuracy by employing a couple strategies to reduce overfitting: **data augmentation** and **dropout**. \n",
        "\n",
        "We will follow these steps:\n",
        "\n",
        "1. Explore how data augmentation works by making random transformations to training images.\n",
        "2. Add data augmentation to our data preprocessing.\n",
        "3. Add dropout to the convnet.\n",
        "4. Retrain the model and evaluate loss and accuracy. \n",
        "\n",
        "Let's get started!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "E3sSwzshfSpE"
      },
      "source": [
        "## Exploring Data Augmentation\n",
        "\n",
        "Let's get familiar with the concept of **data augmentation**, an essential way to fight overfitting for computer vision models.\n",
        "\n",
        "In order to make the most of our few training examples, we will \"augment\" them via a number of random transformations, so that at training time, **our model will never see the exact same picture twice**. This helps prevent overfitting and helps the model generalize better.\n",
        "\n",
        "This can be done by configuring a number of random transformations to be performed on the images read by our `ImageDataGenerator` instance. Let's get started with an example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "XK-IN_zNgLlT"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "datagen = ImageDataGenerator(\n",
        "      rotation_range=40,\n",
        "      width_shift_range=0.2,\n",
        "      height_shift_range=0.2,\n",
        "      shear_range=0.2,\n",
        "      zoom_range=0.2,\n",
        "      horizontal_flip=True,\n",
        "      fill_mode='nearest')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ijUDyVZtSgz3"
      },
      "source": [
        "These are just a few of the options available (for more, see the [Keras documentation](https://keras.io/preprocessing/image/). Let's quickly go over what we just wrote:\n",
        "\n",
        "- `rotation_range` is a value in degrees (0–180), a range within which to randomly rotate pictures.\n",
        "- `width_shift` and `height_shift` are ranges (as a fraction of total width or height) within which to randomly translate pictures vertically or horizontally.\n",
        "- `shear_range` is for randomly applying shearing transformations.\n",
        "- `zoom_range` is for randomly zooming inside pictures.\n",
        "- `horizontal_flip` is for randomly flipping half of the images horizontally. This is relevant when there are no assumptions of horizontal assymmetry (e.g. real-world pictures).\n",
        "- `fill_mode` is the strategy used for filling in newly created pixels, which can appear after a rotation or a width/height shift.\n",
        "\n",
        "Let's take a look at our augmented images. First let's set up our example files, as in Exercise 1.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "grzOIOhoY366"
      },
      "source": [
        "**NOTE:** The 2,000 images used in this exercise are excerpted from the [\"Dogs vs. Cats\" dataset](https://www.kaggle.com/c/dogs-vs-cats/data) available on Kaggle, which contains 25,000 images. Here, we use a subset of the full dataset to decrease training time for educational purposes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "dhztKtUSFMXp"
      },
      "outputs": [],
      "source": [
        "!wget --no-check-certificate \\\n",
        "   https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip -O \\\n",
        "   /tmp/cats_and_dogs_filtered.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "LWkSRoJRfvGL"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import zipfile\n",
        "\n",
        "local_zip = '/tmp/cats_and_dogs_filtered.zip'\n",
        "zip_ref = zipfile.ZipFile(local_zip, 'r')\n",
        "zip_ref.extractall('/tmp')\n",
        "zip_ref.close()\n",
        "  \n",
        "base_dir = '/tmp/cats_and_dogs_filtered'\n",
        "train_dir = os.path.join(base_dir, 'train')\n",
        "validation_dir = os.path.join(base_dir, 'validation')\n",
        "\n",
        "# Directory with our training cat pictures\n",
        "train_cats_dir = os.path.join(train_dir, 'cats')\n",
        "\n",
        "# Directory with our training dog pictures\n",
        "train_dogs_dir = os.path.join(train_dir, 'dogs')\n",
        "\n",
        "# Directory with our validation cat pictures\n",
        "validation_cats_dir = os.path.join(validation_dir, 'cats')\n",
        "\n",
        "# Directory with our validation dog pictures\n",
        "validation_dogs_dir = os.path.join(validation_dir, 'dogs')\n",
        "\n",
        "train_cat_fnames = os.listdir(train_cats_dir)\n",
        "train_dog_fnames = os.listdir(train_dogs_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "02r1oXaegECk"
      },
      "source": [
        "Next, let's apply the `datagen` transformations to a cat image from the training set to produce five random variants. Rerun the cell a few times to see fresh batches of random variants."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "ap-nt8Byfaov"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.image as mpimg\n",
        "\n",
        "from tensorflow.keras.preprocessing.image import array_to_img, img_to_array, load_img\n",
        "\n",
        "img_path = os.path.join(train_cats_dir, train_cat_fnames[2])\n",
        "img = load_img(img_path, target_size=(150, 150))  # this is a PIL image\n",
        "x = img_to_array(img)  # Numpy array with shape (150, 150, 3)\n",
        "x = x.reshape((1,) + x.shape)  # Numpy array with shape (1, 150, 150, 3)\n",
        "\n",
        "# The .flow() command below generates batches of randomly transformed images\n",
        "# It will loop indefinitely, so we need to `break` the loop at some point!\n",
        "i = 0\n",
        "for batch in datagen.flow(x, batch_size=1):\n",
        "  plt.figure(i)\n",
        "  imgplot = plt.imshow(array_to_img(batch[0]))\n",
        "  i += 1\n",
        "  if i % 5 == 0:\n",
        "    break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "eywLKduLmYPY"
      },
      "source": [
        "## Add Data Augmentation to the Preprocessing Step\n",
        "\n",
        "Now let's add our data-augmentation transformations from [**Exploring Data Augmentation**](#scrollTo=E3sSwzshfSpE) to our data preprocessing configuration:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "e8HgwcAbmdcu"
      },
      "outputs": [],
      "source": [
        "# Adding rescale, rotation_range, width_shift_range, height_shift_range,\n",
        "# shear_range, zoom_range, and horizontal flip to our ImageDataGenerator\n",
        "train_datagen = ImageDataGenerator(\n",
        "    rescale=1./255,\n",
        "    rotation_range=40,\n",
        "    width_shift_range=0.2,\n",
        "    height_shift_range=0.2,\n",
        "    shear_range=0.2,\n",
        "    zoom_range=0.2,\n",
        "    horizontal_flip=True,)\n",
        "\n",
        "# Note that the validation data should not be augmented!\n",
        "val_datagen = ImageDataGenerator(rescale=1./255)\n",
        "\n",
        "# Flow training images in batches of 32 using train_datagen generator\n",
        "train_generator = train_datagen.flow_from_directory(\n",
        "        train_dir,  # This is the source directory for training images\n",
        "        target_size=(150, 150),  # All images will be resized to 150x150\n",
        "        batch_size=20,\n",
        "        # Since we use binary_crossentropy loss, we need binary labels\n",
        "        class_mode='binary')\n",
        "\n",
        "# Flow validation images in batches of 32 using val_datagen generator\n",
        "validation_generator = val_datagen.flow_from_directory(\n",
        "        validation_dir,\n",
        "        target_size=(150, 150),\n",
        "        batch_size=20,\n",
        "        class_mode='binary')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "K-3PrfDwDJjB"
      },
      "source": [
        "If we train a new network using this data augmentation configuration, our network will never see the same input twice. However the inputs that it sees are still heavily intercorrelated, so this might not be quite enough to completely get rid of overfitting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lYguAfH3gyv6"
      },
      "source": [
        "## Adding Dropout\n",
        "\n",
        "Another popular strategy for fighting overfitting is to use **dropout**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VtDl3oEZo_7Z"
      },
      "source": [
        "**TIP:** To learn more about dropout, see [Training Neural Networks](https://developers.google.com/machine-learning/crash-course/training-neural-networks/video-lecture) in [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bi3c0YtwpRUr"
      },
      "source": [
        "Let's reconfigure our convnet architecture from Exercise 1 to add some dropout, right before the final classification layer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "SVC4FgxiDje6"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras.optimizers import RMSprop\n",
        "\n",
        "# Our input feature map is 150x150x3: 150x150 for the image pixels, and 3 for\n",
        "# the three color channels: R, G, and B\n",
        "img_input = layers.Input(shape=(150, 150, 3))\n",
        "\n",
        "# First convolution extracts 16 filters that are 3x3\n",
        "# Convolution is followed by max-pooling layer with a 2x2 window\n",
        "x = layers.Conv2D(16, 3, activation='relu')(img_input)\n",
        "x = layers.MaxPooling2D(2)(x)\n",
        "\n",
        "# Second convolution extracts 32 filters that are 3x3\n",
        "# Convolution is followed by max-pooling layer with a 2x2 window\n",
        "x = layers.Conv2D(32, 3, activation='relu')(x)\n",
        "x = layers.MaxPooling2D(2)(x)\n",
        "\n",
        "# Third convolution extracts 64 filters that are 3x3\n",
        "# Convolution is followed by max-pooling layer with a 2x2 window\n",
        "x = layers.Convolution2D(64, 3, activation='relu')(x)\n",
        "x = layers.MaxPooling2D(2)(x)\n",
        "\n",
        "# Flatten feature map to a 1-dim tensor\n",
        "x = layers.Flatten()(x)\n",
        "\n",
        "# Create a fully connected layer with ReLU activation and 512 hidden units\n",
        "x = layers.Dense(512, activation='relu')(x)\n",
        "\n",
        "# Add a dropout rate of 0.5\n",
        "x = layers.Dropout(0.5)(x)\n",
        "\n",
        "# Create output layer with a single node and sigmoid activation\n",
        "output = layers.Dense(1, activation='sigmoid')(x)\n",
        "\n",
        "# Configure and compile the model\n",
        "model = Model(img_input, output)\n",
        "model.compile(loss='binary_crossentropy',\n",
        "              optimizer=RMSprop(lr=0.001),\n",
        "              metrics=['acc'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jKSgmOt5itEF"
      },
      "source": [
        "## Retrain the Model\n",
        "\n",
        "With data augmentation and dropout in place, let's retrain our convnet model. This time, let's train on all 2,000 images available, for 30 epochs, and validate on all 1,000 validation images. (This may take a few minutes to run.) See if you can write the code yourself:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "VWr-MDk4ksJr"
      },
      "outputs": [],
      "source": [
        "# WRITE CODE TO TRAIN THE MODEL ON ALL 2000 IMAGES FOR 30 EPOCHS, AND VALIDATE \n",
        "# ON ALL 1,000 VALIDATION IMAGES"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OpFqg-R1g9n6"
      },
      "source": [
        "### Solution\n",
        "\n",
        "Click below for the solution."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "SdW6geEVi2S8"
      },
      "outputs": [],
      "source": [
        "history = model.fit_generator(\n",
        "      train_generator,\n",
        "      steps_per_epoch=100,\n",
        "      epochs=30,\n",
        "      validation_data=validation_generator,\n",
        "      validation_steps=50,\n",
        "      verbose=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1LTWMLV6SUvP"
      },
      "source": [
        "Note that with data augmentation in place, the 2,000 training images are randomly transformed each time a new training epoch runs, which means that the model will never see the same image twice during training."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IZqvC9UJlWc2"
      },
      "source": [
        "## Evaluate the Results\n",
        "\n",
        "Let's evaluate the results of model training with data augmentation and dropout:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "NKCjHegASXaA"
      },
      "outputs": [],
      "source": [
        "# Retrieve a list of accuracy results on training and validation data\n",
        "# sets for each training epoch\n",
        "acc = history.history['acc']\n",
        "val_acc = history.history['val_acc']\n",
        "\n",
        "# Retrieve a list of list results on training and validation data\n",
        "# sets for each training epoch\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "# Get number of epochs\n",
        "epochs = range(len(acc))\n",
        "\n",
        "# Plot training and validation accuracy per epoch\n",
        "plt.plot(epochs, acc)\n",
        "plt.plot(epochs, val_acc)\n",
        "plt.title('Training and validation accuracy')\n",
        "\n",
        "plt.figure()\n",
        "\n",
        "# Plot training and validation loss per epoch\n",
        "plt.plot(epochs, loss)\n",
        "plt.plot(epochs, val_loss)\n",
        "plt.title('Training and validation loss')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ej-s7-_eShij"
      },
      "source": [
        "Much better! We are no longer overfitting, and we have gained ~3 validation accuracy percentage points (see the green line in the top chart). In fact, judging by our training profile, we could keep fitting our model for 30+ more epochs and we could probably get to ~80%!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Q4mticgLs5Yf"
      },
      "source": [
        "## Clean Up\n",
        "\n",
        "Before running the next exercise, run the following cell to terminate the kernel and free memory resources:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "colab_type": "code",
        "id": "Pjaok2GqtBtI"
      },
      "outputs": [],
      "source": [
        "import os, signal\n",
        "os.kill(os.getpid(), signal.SIGKILL)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "GrFpZXNQB9FW",
        "OpFqg-R1g9n6"
      ],
      "default_view": {},
      "name": "image_classification_part2.ipynb",
      "provenance": [],
      "version": "0.3.2",
      "views": {}
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}