{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Convolutional Neural Networks: Step by Step\n", "\n", "Welcome to Course 4's first assignment! In this assignment, you will implement convolutional (CONV) and pooling (POOL) layers in numpy, including both forward propagation and (optionally) backward propagation. \n", "\n", "**Notation**:\n", "- Superscript $[l]$ denotes an object of the $l^{th}$ layer. \n", " - Example: $a^{[4]}$ is the $4^{th}$ layer activation. $W^{[5]}$ and $b^{[5]}$ are the $5^{th}$ layer parameters.\n", "\n", "\n", "- Superscript $(i)$ denotes an object from the $i^{th}$ example. \n", " - Example: $x^{(i)}$ is the $i^{th}$ training example input.\n", " \n", " \n", "- Subscript $i$ denotes the $i^{th}$ entry of a vector.\n", " - Example: $a^{[l]}_i$ denotes the $i^{th}$ entry of the activations in layer $l$, assuming this is a fully connected (FC) layer.\n", " \n", " \n", "- $n_H$, $n_W$ and $n_C$ denote respectively the height, width and number of channels of a given layer. If you want to reference a specific layer $l$, you can also write $n_H^{[l]}$, $n_W^{[l]}$, $n_C^{[l]}$. \n", "- $n_{H_{prev}}$, $n_{W_{prev}}$ and $n_{C_{prev}}$ denote respectively the height, width and number of channels of the previous layer. If referencing a specific layer $l$, this could also be denoted $n_H^{[l-1]}$, $n_W^{[l-1]}$, $n_C^{[l-1]}$. \n", "\n", "We assume that you are already familiar with `numpy` and/or have completed the previous courses of the specialization. Let's get started!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Updates\n", "\n", "#### If you were working on the notebook before this update...\n", "* The current notebook is version \"v2a\".\n", "* You can find your original work saved in the notebook with the previous version name (\"v2\") \n", "* To view the file directory, go to the menu \"File->Open\", and this will open a new tab that shows the file directory.\n", "\n", "#### List of updates\n", "* clarified example used for padding function. Updated starter code for padding function.\n", "* `conv_forward` has additional hints to help students if they're stuck.\n", "* `conv_forward` places code for `vert_start` and `vert_end` within the `for h in range(...)` loop; to avoid redundant calculations. Similarly updated `horiz_start` and `horiz_end`. **Thanks to our mentor Kevin Brown for pointing this out.**\n", "* `conv_forward` breaks down the `Z[i, h, w, c]` single line calculation into 3 lines, for clarity.\n", "* `conv_forward` test case checks that students don't accidentally use n_H_prev instead of n_H, use n_W_prev instead of n_W, and don't accidentally swap n_H with n_W\n", "* `pool_forward` properly nests calculations of `vert_start`, `vert_end`, `horiz_start`, and `horiz_end` to avoid redundant calculations.\n", "* `pool_forward' has two new test cases that check for a correct implementation of stride (the height and width of the previous layer's activations should be large enough relative to the filter dimensions so that a stride can take place). \n", "* `conv_backward`: initialize `Z` and `cache` variables within unit test, to make it independent of unit testing that occurs in the `conv_forward` section of the assignment.\n", "* **Many thanks to our course mentor, Paul Mielke, for proposing these test cases.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1 - Packages\n", "\n", "Let's first import all the packages that you will need during this assignment. \n", "- [numpy](www.numpy.org) is the fundamental package for scientific computing with Python.\n", "- [matplotlib](http://matplotlib.org) is a library to plot graphs in Python.\n", "- np.random.seed(1) is used to keep all the random function calls consistent. It will help us grade your work." ] }, { "cell_type": "code", "execution_count": 74, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The autoreload extension is already loaded. To reload it, use:\n", " %reload_ext autoreload\n" ] } ], "source": [ "import numpy as np\n", "import h5py\n", "import matplotlib.pyplot as plt\n", "\n", "%matplotlib inline\n", "plt.rcParams['figure.figsize'] = (5.0, 4.0) # set default size of plots\n", "plt.rcParams['image.interpolation'] = 'nearest'\n", "plt.rcParams['image.cmap'] = 'gray'\n", "\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "np.random.seed(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2 - Outline of the Assignment\n", "\n", "You will be implementing the building blocks of a convolutional neural network! Each function you will implement will have detailed instructions that will walk you through the steps needed:\n", "\n", "- Convolution functions, including:\n", " - Zero Padding\n", " - Convolve window \n", " - Convolution forward\n", " - Convolution backward (optional)\n", "- Pooling functions, including:\n", " - Pooling forward\n", " - Create mask \n", " - Distribute value\n", " - Pooling backward (optional)\n", " \n", "This notebook will ask you to implement these functions from scratch in `numpy`. In the next notebook, you will use the TensorFlow equivalents of these functions to build the following model:\n", "\n", "\n", "\n", "**Note** that for every forward function, there is its corresponding backward equivalent. Hence, at every step of your forward module you will store some parameters in a cache. These parameters are used to compute gradients during backpropagation. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3 - Convolutional Neural Networks\n", "\n", "Although programming frameworks make convolutions easy to use, they remain one of the hardest concepts to understand in Deep Learning. A convolution layer transforms an input volume into an output volume of different size, as shown below. \n", "\n", "\n", "\n", "In this part, you will build every step of the convolution layer. You will first implement two helper functions: one for zero padding and the other for computing the convolution function itself. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.1 - Zero-Padding\n", "\n", "Zero-padding adds zeros around the border of an image:\n", "\n", "\n", "
\n", " **Z**\n", " | \n", "\n", " -6.99908945068\n", " | \n", "
\n", "\n", " | \n", "\n", " | \n", "\n", " | \n", " |
\n", " **dA_mean**\n", " | \n", "\n", " 1.45243777754\n", " | \n", "
\n", " **dW_mean**\n", " | \n", "\n", " 1.72699145831\n", " | \n", "
\n", " **db_mean**\n", " | \n", "\n", " 7.83923256462\n", " | \n", "
\n", "\n", "**x =**\n", " | \n", "\n", "\n",
"\n",
"[[ 1.62434536 -0.61175641 -0.52817175] \n", " [-1.07296862 0.86540763 -2.3015387 ]]\n", "\n", " | \n",
"
\n", "**mask =**\n", " | \n", "\n",
"[[ True False False] \n", " [False False False]]\n", " | \n",
"
\n", "distributed_value =\n", " | \n", "\n",
"[[ 0.5 0.5]\n",
" \n", "[ 0.5 0.5]]\n", " | \n",
"
\n", "\n", "**mean of dA =**\n", " | \n", "\n", "\n", "\n", "0.145713902729\n", "\n", " | \n", "
\n", "**dA_prev[1,1] =** \n", " | \n", "\n",
"[[ 0. 0. ] \n", " [ 5.05844394 -1.68282702] \n", " [ 0. 0. ]]\n", " | \n",
"
\n", "\n", "**mean of dA =**\n", " | \n", "\n", "\n", "\n", "0.145713902729\n", "\n", " | \n", "
\n", "**dA_prev[1,1] =** \n", " | \n", "\n",
"[[ 0.08485462 0.2787552 ] \n", " [ 1.26461098 -0.25749373] \n", " [ 1.17975636 -0.53624893]]\n", " | \n",
"