{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "qfja9tMaB1ZH" }, "source": [ "#### Copyright 2018 Google LLC." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "TkNQqjZaB4_u" }, "outputs": [], "source": [ "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "YHK6DyunSbs4" }, "source": [ "# Cat vs. Dog Image Classification\n", "## Exercise 1: Building a Convnet from Scratch\n", "**_Estimated completion time: 20 minutes_**\n", "\n", "In this exercise, we will build a classifier model from scratch that is able to distinguish dogs from cats. We will follow these steps:\n", "\n", "1. Explore the example data\n", "2. Build a small convnet from scratch to solve our classification problem\n", "3. Evaluate training and validation accuracy\n", "\n", "Let's go!" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "UY6KJV6z6l7_" }, "source": [ "## Explore the Example Data" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "-L7r2zdl64Hg" }, "source": [ "Let's start by downloading our example data, a .zip of 2,000 JPG pictures of cats and dogs, and extracting it locally in `/tmp`." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "keyquuYMWPMa" }, "source": [ "**NOTE:** The 2,000 images used in this exercise are excerpted from the [\"Dogs vs. Cats\" dataset](https://www.kaggle.com/c/dogs-vs-cats/data) available on Kaggle, which contains 25,000 images. Here, we use a subset of the full dataset to decrease training time for educational purposes." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "RXZT2UsyIVe_" }, "outputs": [], "source": [ "!wget --no-check-certificate \\\n", " https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip \\\n", " -O /tmp/cats_and_dogs_filtered.zip" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "PLy3pthUS0D2" }, "outputs": [], "source": [ "import os\n", "import zipfile\n", "\n", "local_zip = '/tmp/cats_and_dogs_filtered.zip'\n", "zip_ref = zipfile.ZipFile(local_zip, 'r')\n", "zip_ref.extractall('/tmp')\n", "zip_ref.close()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "o-qUPyfO7Qr8" }, "source": [ "The contents of the .zip are extracted to the base directory `/tmp/cats_and_dogs_filtered`, which contains `train` and `validation` subdirectories for the training and validation datasets (see the [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/validation/check-your-intuition) for a refresher on training, validation, and test sets), which in turn each contain `cats` and `dogs` subdirectories. Let's define each of these directories:" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "MLZKVtE0dSfk" }, "outputs": [], "source": [ "base_dir = '/tmp/cats_and_dogs_filtered'\n", "train_dir = os.path.join(base_dir, 'train')\n", "validation_dir = os.path.join(base_dir, 'validation')\n", "\n", "# Directory with our training cat pictures\n", "train_cats_dir = os.path.join(train_dir, 'cats')\n", "\n", "# Directory with our training dog pictures\n", "train_dogs_dir = os.path.join(train_dir, 'dogs')\n", "\n", "# Directory with our validation cat pictures\n", "validation_cats_dir = os.path.join(validation_dir, 'cats')\n", "\n", "# Directory with our validation dog pictures\n", "validation_dogs_dir = os.path.join(validation_dir, 'dogs')" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "LuBYtA_Zd8_T" }, "source": [ "Now, let's see what the filenames look like in the `cats` and `dogs` `train` directories (file naming conventions are the same in the `validation` directory):" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "4PIP1rkmeAYS" }, "outputs": [], "source": [ "train_cat_fnames = os.listdir(train_cats_dir)\n", "print(train_cat_fnames[:10])\n", "\n", "train_dog_fnames = os.listdir(train_dogs_dir)\n", "train_dog_fnames.sort()\n", "print(train_dog_fnames[:10])" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "HlqN5KbafhLI" }, "source": [ "Let's find out the total number of cat and dog images in the `train` and `validation` directories:" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "H4XHh2xSfgie" }, "outputs": [], "source": [ "print('total training cat images:', len(os.listdir(train_cats_dir)))\n", "print('total training dog images:', len(os.listdir(train_dogs_dir)))\n", "print('total validation cat images:', len(os.listdir(validation_cats_dir)))\n", "print('total validation dog images:', len(os.listdir(validation_dogs_dir)))" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "C3WZABE9eX-8" }, "source": [ "For both cats and dogs, we have 1,000 training images and 500 test images.\n", "\n", "Now let's take a look at a few pictures to get a better sense of what the cat and dog datasets look like. First, configure the matplot parameters:" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "b2_Q0-_5UAv-" }, "outputs": [], "source": [ "%matplotlib inline\n", "\n", "import matplotlib.pyplot as plt\n", "import matplotlib.image as mpimg\n", "\n", "# Parameters for our graph; we'll output images in a 4x4 configuration\n", "nrows = 4\n", "ncols = 4\n", "\n", "# Index for iterating over images\n", "pic_index = 0" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "xTvHzGCxXkqp" }, "source": [ "Now, display a batch of 8 cat and 8 dog pictures. You can rerun the cell to see a fresh batch each time:" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "Wpr8GxjOU8in" }, "outputs": [], "source": [ "# Set up matplotlib fig, and size it to fit 4x4 pics\n", "fig = plt.gcf()\n", "fig.set_size_inches(ncols * 4, nrows * 4)\n", "\n", "pic_index += 8\n", "next_cat_pix = [os.path.join(train_cats_dir, fname) \n", " for fname in train_cat_fnames[pic_index-8:pic_index]]\n", "next_dog_pix = [os.path.join(train_dogs_dir, fname) \n", " for fname in train_dog_fnames[pic_index-8:pic_index]]\n", "\n", "for i, img_path in enumerate(next_cat_pix+next_dog_pix):\n", " # Set up subplot; subplot indices start at 1\n", " sp = plt.subplot(nrows, ncols, i + 1)\n", " sp.axis('Off') # Don't show axes (or gridlines)\n", "\n", " img = mpimg.imread(img_path)\n", " plt.imshow(img)\n", "\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "5oqBkNBJmtUv" }, "source": [ "## Building a Small Convnet from Scratch to Get to 72% Accuracy\n", "\n", "The images that will go into our convnet are 150x150 color images (in the next section on Data Preprocessing, we'll add handling to resize all the images to 150x150 before feeding them into the neural network).\n", "\n", "Let's code up the architecture. We will stack 3 {convolution + relu + maxpooling} modules. Our convolutions operate on 3x3 windows and our maxpooling layers operate on 2x2 windows. Our first convolution extracts 16 filters, the following one extracts 32 filters, and the last one extracts 64 filters.\n", "\n", "**NOTE**: This is a configuration that is widely used and known to work well for image classification. Also, since we have relatively few training examples (1,000), using just three convolutional modules keeps the model small, which lowers the risk of overfitting (which we'll explore in more depth in Exercise 2.)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "IYVLSfbsbf9K" }, "outputs": [], "source": [ "from tensorflow.keras import layers\n", "from tensorflow.keras import Model" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "ugysGIPxnaYo" }, "outputs": [], "source": [ "# Our input feature map is 150x150x3: 150x150 for the image pixels, and 3 for\n", "# the three color channels: R, G, and B\n", "img_input = layers.Input(shape=(150, 150, 3))\n", "\n", "# First convolution extracts 16 filters that are 3x3\n", "# Convolution is followed by max-pooling layer with a 2x2 window\n", "x = layers.Conv2D(16, 3, activation='relu')(img_input)\n", "x = layers.MaxPooling2D(2)(x)\n", "\n", "# Second convolution extracts 32 filters that are 3x3\n", "# Convolution is followed by max-pooling layer with a 2x2 window\n", "x = layers.Conv2D(32, 3, activation='relu')(x)\n", "x = layers.MaxPooling2D(2)(x)\n", "\n", "# Third convolution extracts 64 filters that are 3x3\n", "# Convolution is followed by max-pooling layer with a 2x2 window\n", "x = layers.Conv2D(64, 3, activation='relu')(x)\n", "x = layers.MaxPooling2D(2)(x)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "fWO_RJpbsmVD" }, "source": [ "On top of it we stick two fully-connected layers. Because we are facing a two-class classification problem, i.e. a *binary classification problem*, we will end our network with a [*sigmoid* activation](https://wikipedia.org/wiki/Sigmoid_function), so that the output of our network will be a single scalar between 0 and 1, encoding the probability that the current image is class 1 (as opposed to class 0)." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "3v88_ZTAslvR" }, "outputs": [], "source": [ "# Flatten feature map to a 1-dim tensor so we can add fully connected layers\n", "x = layers.Flatten()(x)\n", "\n", "# Create a fully connected layer with ReLU activation and 512 hidden units\n", "x = layers.Dense(512, activation='relu')(x)\n", "\n", "# Create output layer with a single node and sigmoid activation\n", "output = layers.Dense(1, activation='sigmoid')(x)\n", "\n", "# Create model:\n", "# input = input feature map\n", "# output = input feature map + stacked convolution/maxpooling layers + fully \n", "# connected layer + sigmoid output layer\n", "model = Model(img_input, output)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "s9EaFDP5srBa" }, "source": [ "Let's summarize the model architecture:" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "7ZKj8392nbgP" }, "outputs": [], "source": [ "model.summary()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "DmtkTn06pKxF" }, "source": [ "The \"output shape\" column shows how the size of your feature map evolves in each successive layer. The convolution layers reduce the size of the feature maps by a bit due to padding, and each pooling layer halves the feature map." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "PEkKSpZlvJXA" }, "source": [ "Next, we'll configure the specifications for model training. We will train our model with the `binary_crossentropy` loss, because it's a binary classification problem and our final activation is a sigmoid. (For a refresher on loss metrics, see the [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course/descending-into-ml/video-lecture).) We will use the `rmsprop` optimizer with a learning rate of `0.001`. During training, we will want to monitor classification accuracy.\n", "\n", "**NOTE**: In this case, using the [RMSprop optimization algorithm](https://wikipedia.org/wiki/Stochastic_gradient_descent#RMSProp) is preferable to [stochastic gradient descent](https://developers.google.com/machine-learning/glossary/#SGD) (SGD), because RMSprop automates learning-rate tuning for us. (Other optimizers, such as [Adam](https://wikipedia.org/wiki/Stochastic_gradient_descent#Adam) and [Adagrad](https://developers.google.com/machine-learning/glossary/#AdaGrad), also automatically adapt the learning rate during training, and would work equally well here.)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "8DHWhFP_uhq3" }, "outputs": [], "source": [ "from tensorflow.keras.optimizers import RMSprop\n", "\n", "model.compile(loss='binary_crossentropy',\n", " optimizer=RMSprop(lr=0.001),\n", " metrics=['acc'])" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Sn9m9D3UimHM" }, "source": [ "### Data Preprocessing\n", "\n", "Let's set up data generators that will read pictures in our source folders, convert them to `float32` tensors, and feed them (with their labels) to our network. We'll have one generator for the training images and one for the validation images. Our generators will yield batches of 20 images of size 150x150 and their labels (binary).\n", "\n", "As you may already know, data that goes into neural networks should usually be normalized in some way to make it more amenable to processing by the network. (It is uncommon to feed raw pixels into a convnet.) In our case, we will preprocess our images by normalizing the pixel values to be in the `[0, 1]` range (originally all values are in the `[0, 255]` range).\n", "\n", "In Keras this can be done via the `keras.preprocessing.image.ImageDataGenerator` class using the `rescale` parameter. This `ImageDataGenerator` class allows you to instantiate generators of augmented image batches (and their labels) via `.flow(data, labels)` or `.flow_from_directory(directory)`. These generators can then be used with the Keras model methods that accept data generators as inputs: `fit_generator`, `evaluate_generator`, and `predict_generator`." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "ClebU9NJg99G" }, "outputs": [], "source": [ "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n", "\n", "# All images will be rescaled by 1./255\n", "train_datagen = ImageDataGenerator(rescale=1./255)\n", "val_datagen = ImageDataGenerator(rescale=1./255)\n", "\n", "# Flow training images in batches of 20 using train_datagen generator\n", "train_generator = train_datagen.flow_from_directory(\n", " train_dir, # This is the source directory for training images\n", " target_size=(150, 150), # All images will be resized to 150x150\n", " batch_size=20,\n", " # Since we use binary_crossentropy loss, we need binary labels\n", " class_mode='binary')\n", "\n", "# Flow validation images in batches of 20 using val_datagen generator\n", "validation_generator = val_datagen.flow_from_directory(\n", " validation_dir,\n", " target_size=(150, 150),\n", " batch_size=20,\n", " class_mode='binary')" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "mu3Jdwkjwax4" }, "source": [ "### Training\n", "Let's train on all 2,000 images available, for 15 epochs, and validate on all 1,000 validation images. (This may take a few minutes to run.)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "Fb1_lgobv81m" }, "outputs": [], "source": [ "history = model.fit_generator(\n", " train_generator,\n", " steps_per_epoch=100, # 2000 images = batch_size * steps\n", " epochs=15,\n", " validation_data=validation_generator,\n", " validation_steps=50, # 1000 images = batch_size * steps\n", " verbose=2)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "-8EHQyWGDvWz" }, "source": [ "### Visualizing Intermediate Representations\n", "\n", "To get a feel for what kind of features our convnet has learned, one fun thing to do is to visualize how an input gets transformed as it goes through the convnet.\n", "\n", "Let's pick a random cat or dog image from the training set, and then generate a figure where each row is the output of a layer, and each image in the row is a specific filter in that output feature map. Rerun this cell to generate intermediate representations for a variety of training images." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "-5tES8rXFjux" }, "outputs": [], "source": [ "import numpy as np\n", "import random\n", "from tensorflow.keras.preprocessing.image import img_to_array, load_img\n", "\n", "# Let's define a new Model that will take an image as input, and will output\n", "# intermediate representations for all layers in the previous model after\n", "# the first.\n", "successive_outputs = [layer.output for layer in model.layers[1:]]\n", "visualization_model = Model(img_input, successive_outputs)\n", "\n", "# Let's prepare a random input image of a cat or dog from the training set.\n", "cat_img_files = [os.path.join(train_cats_dir, f) for f in train_cat_fnames]\n", "dog_img_files = [os.path.join(train_dogs_dir, f) for f in train_dog_fnames]\n", "img_path = random.choice(cat_img_files + dog_img_files)\n", "\n", "img = load_img(img_path, target_size=(150, 150)) # this is a PIL image\n", "x = img_to_array(img) # Numpy array with shape (150, 150, 3)\n", "x = x.reshape((1,) + x.shape) # Numpy array with shape (1, 150, 150, 3)\n", "\n", "# Rescale by 1/255\n", "x /= 255\n", "\n", "# Let's run our image through our network, thus obtaining all\n", "# intermediate representations for this image.\n", "successive_feature_maps = visualization_model.predict(x)\n", "\n", "# These are the names of the layers, so can have them as part of our plot\n", "layer_names = [layer.name for layer in model.layers]\n", "\n", "# Now let's display our representations\n", "for layer_name, feature_map in zip(layer_names, successive_feature_maps):\n", " if len(feature_map.shape) == 4:\n", " # Just do this for the conv / maxpool layers, not the fully-connected layers\n", " n_features = feature_map.shape[-1] # number of features in feature map\n", " # The feature map has shape (1, size, size, n_features)\n", " size = feature_map.shape[1]\n", " # We will tile our images in this matrix\n", " display_grid = np.zeros((size, size * n_features))\n", " for i in range(n_features):\n", " # Postprocess the feature to make it visually palatable\n", " x = feature_map[0, :, :, i]\n", " x -= x.mean()\n", " x /= x.std()\n", " x *= 64\n", " x += 128\n", " x = np.clip(x, 0, 255).astype('uint8')\n", " # We'll tile each filter into this big horizontal grid\n", " display_grid[:, i * size : (i + 1) * size] = x\n", " # Display the grid\n", " scale = 20. / n_features\n", " plt.figure(figsize=(scale * n_features, scale))\n", " plt.title(layer_name)\n", " plt.grid(False)\n", " plt.imshow(display_grid, aspect='auto', cmap='viridis')" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "tuqK2arJL0wo" }, "source": [ "As you can see we go from the raw pixels of the images to increasingly abstract and compact representations. The representations downstream start highlighting what the network pays attention to, and they show fewer and fewer features being \"activated\"; most are set to zero. This is called \"sparsity.\" Representation sparsity is a key feature of deep learning.\n", "\n", "\n", "These representations carry increasingly less information about the original pixels of the image, but increasingly refined information about the class of the image. You can think of a convnet (or a deep network in general) as an information distillation pipeline." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Q5Vulban4ZrD" }, "source": [ "### Evaluating Accuracy and Loss for the Model\n", "\n", "Let's plot the training/validation accuracy and loss as collected during training:" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "0oj0gTIy4k60" }, "outputs": [], "source": [ "# Retrieve a list of accuracy results on training and validation data\n", "# sets for each training epoch\n", "acc = history.history['acc']\n", "val_acc = history.history['val_acc']\n", "\n", "# Retrieve a list of list results on training and validation data\n", "# sets for each training epoch\n", "loss = history.history['loss']\n", "val_loss = history.history['val_loss']\n", "\n", "# Get number of epochs\n", "epochs = range(len(acc))\n", "\n", "# Plot training and validation accuracy per epoch\n", "plt.plot(epochs, acc)\n", "plt.plot(epochs, val_acc)\n", "plt.title('Training and validation accuracy')\n", "\n", "plt.figure()\n", "\n", "# Plot training and validation loss per epoch\n", "plt.plot(epochs, loss)\n", "plt.plot(epochs, val_loss)\n", "plt.title('Training and validation loss')" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "DgmSjUST4qoS" }, "source": [ "As you can see, we are **overfitting** like it's getting out of fashion. Our training accuracy (in blue) gets close to 100% (!) while our validation accuracy (in green) stalls as 70%. Our validation loss reaches its minimum after only five epochs.\n", "\n", "Since we have a relatively small number of training examples (2000), overfitting should be our number one concern. Overfitting happens when a model exposed to too few examples learns patterns that do not generalize to new data, i.e. when the model starts using irrelevant features for making predictions. For instance, if you, as a human, only see three images of people who are lumberjacks, and three images of people who are sailors, and among them the only person wearing a cap is a lumberjack, you might start thinking that wearing a cap is a sign of being a lumberjack as opposed to a sailor. You would then make a pretty lousy lumberjack/sailor classifier.\n", "\n", "Overfitting is the central problem in machine learning: given that we are fitting the parameters of our model to a given dataset, how can we make sure that the representations learned by the model will be applicable to data never seen before? How do we avoid learning things that are specific to the training data?\n", "\n", "In the next exercise, we'll look at ways to prevent overfitting in the cat vs. dog classification model." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "j4IBgYCYooGD" }, "source": [ "## Clean Up\n", "\n", "Before running the next exercise, run the following cell to terminate the kernel and free memory resources:" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "651IgjLyo-Jx" }, "outputs": [], "source": [ "import os, signal\n", "os.kill(os.getpid(), signal.SIGKILL)" ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [ "qfja9tMaB1ZH" ], "name": "image_classification_part1.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }