{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Credits: Forked from [deep-learning-keras-tensorflow](https://github.com/leriomaggio/deep-learning-keras-tensorflow) by Valerio Maggio" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Convolution Nets for MNIST" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Deep Learning models can take quite a bit of time to run, particularly if GPU isn't used. \n", "\n", "In the interest of time, you could sample a subset of observations (e.g. $1000$) that are a particular number of your choice (e.g. $6$) and $1000$ observations that aren't that particular number (i.e. $\\neq 6$). \n", "\n", "We will build a model using that and see how it performs on the test dataset" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using Theano backend.\n", "Using gpu device 0: GeForce GTX 760 (CNMeM is enabled with initial size: 90.0% of memory, cuDNN 4007)\n" ] } ], "source": [ "#Import the required libraries\n", "import numpy as np\n", "np.random.seed(1338)\n", "\n", "from keras.datasets import mnist" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "from keras.models import Sequential" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "from keras.layers.core import Dense, Dropout, Activation, Flatten" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "from keras.layers.convolutional import Convolution2D\n", "from keras.layers.pooling import MaxPooling2D" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": true, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "from keras.utils import np_utils\n", "from keras.optimizers import SGD" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Loading Data" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": true, "slideshow": { "slide_type": "-" } }, "outputs": [], "source": [ "path_to_dataset = \"euroscipy_2016_dl-keras/data/mnist.pkl.gz\"\n", "\n", "#Load the training and testing data\n", "(X_train, y_train), (X_test, y_test) = mnist.load_data(path_to_dataset)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": true, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "X_test_orig = X_test" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Data Preparation" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": true, "slideshow": { "slide_type": "-" } }, "outputs": [], "source": [ "img_rows, img_cols = 28, 28\n", "\n", "X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)\n", "X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)\n", "\n", "X_train = X_train.astype('float32')\n", "X_test = X_test.astype('float32')\n", "\n", "X_train /= 255\n", "X_test /= 255" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# Seed for reproducibilty\n", "np.random.seed(1338)\n", "\n", "# Test data\n", "X_test = X_test.copy()\n", "Y = y_test.copy()\n", "\n", "# Converting the output to binary classification(Six=1,Not Six=0)\n", "Y_test = Y == 6\n", "Y_test = Y_test.astype(int)\n", "\n", "# Selecting the 5918 examples where the output is 6\n", "X_six = X_train[y_train == 6].copy()\n", "Y_six = y_train[y_train == 6].copy()\n", "\n", "# Selecting the examples where the output is not 6\n", "X_not_six = X_train[y_train != 6].copy()\n", "Y_not_six = y_train[y_train != 6].copy()\n", "\n", "# Selecting 6000 random examples from the data that \n", "# only contains the data where the output is not 6\n", "random_rows = np.random.randint(0,X_six.shape[0],6000)\n", "X_not_six = X_not_six[random_rows]\n", "Y_not_six = Y_not_six[random_rows]" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# Appending the data with output as 6 and data with output as <> 6\n", "X_train = np.append(X_six,X_not_six)\n", "\n", "# Reshaping the appended data to appropraite form\n", "X_train = X_train.reshape(X_six.shape[0] + X_not_six.shape[0], \n", " 1, img_rows, img_cols)\n", "\n", "# Appending the labels and converting the labels to \n", "# binary classification(Six=1,Not Six=0)\n", "Y_labels = np.append(Y_six,Y_not_six)\n", "Y_train = Y_labels == 6 \n", "Y_train = Y_train.astype(int)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(11918, 1, 28, 28) (11918,) (10000, 1, 28, 28) (10000, 2)\n" ] } ], "source": [ "print(X_train.shape, Y_labels.shape, X_test.shape, Y_test.shape)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": true, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Converting the classes to its binary categorical form\n", "nb_classes = 2\n", "Y_train = np_utils.to_categorical(Y_train, nb_classes)\n", "Y_test = np_utils.to_categorical(Y_test, nb_classes)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# A simple CNN" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "#Initializing the values for the convolution neural network\n", "nb_epoch = 2\n", "batch_size = 128\n", "# number of convolutional filters to use\n", "nb_filters = 32\n", "# size of pooling area for max pooling\n", "nb_pool = 2\n", "# convolution kernel size\n", "nb_conv = 3\n", "\n", "sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Step 1: Model Definition" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model = Sequential()\n", "\n", "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n", " border_mode='valid',\n", " input_shape=(1, img_rows, img_cols)))\n", "model.add(Activation('relu'))\n", "\n", "model.add(Flatten())\n", "model.add(Dense(nb_classes))\n", "model.add(Activation('softmax'))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Step 2: Compile" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": true }, "outputs": [], "source": [ "model.compile(loss='categorical_crossentropy',\n", " optimizer='sgd',\n", " metrics=['accuracy'])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Step 3: Fit" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 11918 samples, validate on 10000 samples\n", "Epoch 1/2\n", "11918/11918 [==============================] - 0s - loss: 0.2890 - acc: 0.9326 - val_loss: 0.1251 - val_acc: 0.9722\n", "Epoch 2/2\n", "11918/11918 [==============================] - 0s - loss: 0.1341 - acc: 0.9612 - val_loss: 0.1298 - val_acc: 0.9599\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.fit(X_train, Y_train, batch_size=batch_size, \n", " nb_epoch=nb_epoch,verbose=1,\n", " validation_data=(X_test, Y_test))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Step 4: Evaluate" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test score: 0.129807630396\n", "Test accuracy: 0.9599\n" ] } ], "source": [ "# Evaluating the model on the test data \n", "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n", "print('Test score:', score)\n", "print('Test accuracy:', accuracy)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Let's plot our model Predictions!" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": true, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA50AAABVCAYAAADKf4AOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXm4VmW5/z+PgMIJA4UABYckPZoYaJSWeEmWph3MMicQ\nTpAoTimCx711b3NhbGWTECAKKhYdUBQnNAtSK0xIJcejRqeccjghDj9NTAXx+f1x32tP7I17eOf3\n+7mufT3rfdaz1nvfe71reu4pxBgRQgghhBBCCCGywTb5FkAIIYQQQgghROmil04hhBBCCCGEEFkj\nKy+d3bp1WxdCiG3969at27psyJNtpK/0lc7St5iQvtJXOkvfYkL6St9S0rnc9E0J2YjpDCHE9uw3\nhECMMWRcoCwjfVu9XVnp69uWlc7StziQvq3erqz09W3LSmfpWxxI31ZvV1b6+rZFp3O56Zsi91oh\nhBBCCCGEEFlDL51CCCGEEEIIIbJGzl46V6xYwd57781ee+1FbW1trr42b5xyyin07duXL3zhC/kW\nJWeU2zGWvqVNuekL5aez9C1tpG9po+es0j/G0rfEiDFm/M92W8/mzZvjwIED44svvhg3btwYBw8e\nHNeuXRub4ttlRaZs/jXVN8YYH3jggfj444/H/fbbb4t1pahva45xuelbSjpL39LWt7U6S9/i+NM1\nS/qWur4x6jlL16zi1bnc9E3/cmLpXLNmDXvuuSe77bYbXbp04aSTTuLOO+/MxVfnjWHDhrHDDjvk\nW4ycUW7HWPpK31Kj3HSWvtK3lCg3fUHPWaV+jKVv6embk5fOV199lV122aXu84ABA3j11Vdz8dUi\nR5TbMZa+0rfUKDedpa/0LSXKTd9ypNyOsfQtPX1z8tLppuRGhFCU2X5FC5TbMZa+0rfUKDedpa/0\nLSXKTd9ypNyOsfQtPX1z8tI5YMAAXnrppbrPr7zyCjvvvHMuvlrkiHI7xtJX+pYa5aaz9JW+pUS5\n6VuOlNsxlr4lqG8uAmQ/+uijuuDYDz/8MA4ePDj++c9/Lpng2Kb6przwwgtx0KBBza4rNX1bc4zL\nTd9S0ln6lra+rdVZ+hbHn65Z0rfU9U3Rc5auWcWoc7npm/51zsqbbBM6derE3LlzOeKII/j44485\n5ZRT2GeffZodG0KypX25QIgxabWde9SoUaxcuZI333yTXXfdlSlTpjBu3LgtxpWKvq09xuWmL5SG\nztK3tPUFncPSt55S0Fn6lra+oOcsXbPqKQWdy0Hf4G/cGSWEENuzX/NdviTj8mSKlv6R0rd1lJu+\nvi3lpLP0LSykryF9W0+56Sx9Cwvpa0jf1lOMOpebvik5iekUQgghhBBCCFGe6KVTCCGEEEIIIUTW\nyElMp2g7J8TPArALLwNwxcsXA5DsWj/mG/EAAI56bwUAG7pflUMJhRBCCCGKld0BiBMsDrT2Guut\neBzC/oXrvihEsSJLpxBCCCGEEEKIrCFLZ4ERX5sCQG2TMNxab7s16FsdHgPg3aF9AAikQclJ1uQr\nCPolAExdZ/+ke+NyAO4PD+VLogxhlusH4pS6Y1txrq0Js8vk2AohSoyjYdgXm1+16mEAkni7tc/4\nne50YFWSdclEC5yfWHvFTGv3nsTotdcBsGj2aQCMOffaRpss/q9TfZskBwJ2lN0BeCBeCcBMf97q\n4mvPH/Jj4KOcSyVEtqiOG+kULgcgmW994fTcW/OzYuns2rXrayEE2vq33Xa9siFO1pG+0lc6S99i\nQvpKX+ksfYsJ6St9S0nnctM3JSslUxp9QQHXkmkrram10xF942tTqO3b/LqKI6y95TcjOP6JuwGo\n3b/xmL/HGQDMC/9srwiN5cmyvu3mWZudmfE5mzOZ/JCLcFDSod22tpZS1nTunQAw+416Md739s74\nWwAeDH/I6Ffm9Bgfmdh3PmNfGV5uxyzb/MSsIEB7rL4F+5u+OwHgkhEm3j11x/sJoP3nc270HW/f\nNXcXAGafZZaQicEDpNptnf+GNXcPs3bEPO9/rcUtCvb4ZomC1PfWBIBHv/d5AA44aC21Dzc/tOJQ\na2++39oXG6yrbKEkQEHqnEVyo+9w+65VXwPgNj/l0gqBrwJrPmEPe3v7BWCvJ1ycIUmbJcmFviOj\nJccYGk4BYNJS3++vfbcLk47svk3o97wl2dP3aGummefF9RWjAHgpLAHqPQkrXoOf9RkJwCnhp96b\n3n/aTiEc4zh/CsnpjfumpF+ZJJn9LpVMEUIIIYQQQgiRDxTTWQh8JwFgXt8pdV0V37U23GEzEZX3\nLPeOh4FvAxDHmkWhdqGt+ip/tP0wKKvi5psrB1rsSJ2WHbRw5p/vAxB/aJNDc0o0ad5py2cD8GYH\nrjq/nfBVvnb8gwBs06sU/lEnALDxZDv2Nd57RPg6AA9yEx2xdGafo/krZuGcd7b1nHu8xXpNZKd2\n7tMsnHHQIQCsHGG9X+vts7JvJO3cb7YYDsDKWAHAobPXECZe7etatsoWPd0TAJa+a9aD7YJ54Dx7\nnN3H7vVh9zbdrgG192dJNtEmHowTAahpYp/4i7dVR8Dh3/YPQ5ts/IE1NcPrt/n9ENvR1w7yc/ah\nJHPCZoCjuQuAf/jnMX7N4oQkL/KIbNKXU+JmAOb1tB/vlZW25m1vUwtnGtM7sy+AWT+fi78HYGCY\nkH1Rs0hTKydA9SUXATA12TZncsjSKYQQQgghhBAiaxSWpfOJBID4vM2S/f27n+Ef7AzAV857wsYs\n9LFvJzkVLatYSBSbljW0cL7jK5Mthp8R3wVgYZNZyTE1t7a4TUkwNAFg12CKd/Y6pbQqErPwOCza\nrNtvF+wBwJxTWx77x/Vm+RqeZuq98EhbMS3JmnyZY0cArpluqXjf7MCeTuIm1o/fzT/58eexDuwx\nz6yyuLeaYY27KzdYO6X7szkWqLWcDED8+V7UWok7DoimRHArbXtZ6Bkl0wze18anbSEkHdpvpuny\nxiQANu7fA6iX13Joe7AiS3MtVu4w4xjPu4WzLaQ5Cvh5y2Mq+7ddpJzRM4Hv2OK1Px8DwKnnL7aO\n7azpdekrALzVr38BWucbMCihV5jSqKvKEuITzjRLZXUS4Z4VvrZpgK7ZiaqjPbPsEC5npa+J/+3x\n+3s94j2/zJzcHeDz/BmAzf558VHpzTfJhzj54xsJALE6UDvcutI8Eg3jGwHCv7vV+u27YJmbvb9z\ns49am1UxO8Kw+Dk+H+yCc2ULYyosMTMzm3kGWxbWZUewAmBqyJ2FM0WWTiGEEEIIIYQQWaOgLJ3x\neI9r+lva87r/wVQ3Zx3sa4b/qJ1f4m7Z4SyftVmWtHNHGWREAsC5RM6943+8c2aLw68+YTJQX7uz\nbFhgzVNDrK2++FFfkeRDmg5zRLBYqDmtGDvHsxofy1EArPSA1jAi1mU+LVjmngPAzLPN0tkpWixy\neyzUX2U1M+9IP+3nbTFaOncHIC6yf0LSZG23ZQVel3X+ngB1Vk6AI8J/+9Iv2r/fQQmvuCdD1V+t\nqzL8X/v3lxUslviZ3jZnm8bhNryZvr9hXwC6dX/Se27PjWgZxyzatdGuUhW/uBLGJrbKDQCeY5gD\nzKGBx97yz+7SMGnHy5h11YX24Wy7uVfe45aR/kVy7rqXzZo/2TWnd5jCjQtt1Xpva5psMumyAYD5\neZzhSTMfumswAF8JZ/moV7MhbdsYCkvdmSD9DX/1Ncue3TrvArONpVaTWfE0+gaLkazZy4fc6nVa\njysES+cFrHDLboUftDFVSf7EySn+IDHrDAA2XmLX2pnD6+MZUxrHN8Ir0cp09F/4FrVu5U/etpP8\ng54FaOkclACwJIQW/U0ejgsBCOFfAFwZ7ZlyY7g+y8KVLwX10hkm2IPWMZMtgPfOX4zkmO/b8s/4\nAQA79LKo9ZpLbZuqnaGmhWcSvwfy/U9Z+9P3AN9uUTwOgDGhkJLuJJ84YmTclcVNHtYr/AW8svq2\nzItUQMSfu+KftaZ66l35E6YDxN/YDS/55iePPcHbz9ozPrU+IVPrDwnTng4tlhYoCIYmPHO2HbfP\nH2hdIfgdaws3rU9m2VWjtjIdU0QcNxaA5JpxjbrTaxajk1xK0wYsFuDRCeYWfO/pMCP+3VaFn7V/\nt/6A8MbagNetpluf1BG7NdMyuWNZNNf227cyaTK7u7WPYC8Zl8dFANwWUue1AnjZ2Crmuh7H2ltD\n6jrMigYOagsSAIbiEyRvJb7C217ptOj7FOzkSUt4crpHH7Tf+b/8JeWeBsf8Ij9ZP3zJ2qs/ZQ/y\nnYOVVjjnAuuvmQ7/+p3vtr9NQgyO9vL6ZCiA38GA+sXjYz8AKjtQmmti2Ino9VNqPBPRX79nycb2\n4pR27zdTPBAPYXUmQnLGJ9ZObNCXzsCsSzLwBVlgtP1GZ0y0f0B6Np+3I2yz1M/jvzTeZI+zngHg\nv17bHYCnxnUn9Xz/YMGOFBx+L4kLTMfag+pfoCf57zJ0dl1Dw2sU/DD45MhJC1h3k21/g4+I37Vr\nQLijgJ+3tsLmeCGdwuWN+qrjRiC3brZyrxVCCCGEEEIIkTUKytLJ+QkAd56fdiTcOdaWelHhfY2L\nu1ZXroQrhje/vxetOXPuCwA8wx6knnlX8kNf+n0Hhc4R7gb8rRDq5sjHd7I2fFzgrngdJgGgZrbN\nNFWlPtYvFIlrVkOqEzjX9EgD9bs1M2wfT8qyx68esI5VvvnlluL6Mw1mrA6INr36WLgv09J2mI9/\nE7jBvHLYdxf/nT6ctGNPxwKQnD2l2f9XsXHlLWYxfKPJjPs57tV2bsdy8WSNN6OZ3OcE8zhJJkBl\neKLD+x38lKXfmR9gpFtbqkNhWTiNhB6h8UGrGukLx1tTe2z9uvSMvKazJZu5jTe8p6WUFvnGbBjx\nT8cAUPsl6w3R6+GE5rxpkk/4XFw8Gpeyg1s2b2xyflbdaO0tI0ewTTjRPjRJ9hXdVB+m2fUu/ihw\ns3tYnegJekaGrwDwJM/7Vh1wSe8gsWeocw0+htR76Fcd2ueqtWmSP7tH7/nNVzq0v0wyrP9jrPbl\nPhe5l0bVJ3tpPBiXAXBQX7NWL19gv5G1C+rHpOnDlsQfAzAjfNRheTOCW/9edje51N10knv9h2tj\nXVKhpjx/tpns41nmMjgT2Da6xbrAkrsBdH/IQvJq3dvkfeo9iI5bax4nLcvtWZNuSrg9Wim7zsHO\nzcRfHp6IdhEYEkZlTugc0NTKmS9k6RRCCCGEEEIIkTUKy9LZKjwQvbJBQPr5K7e+ySOJbTkUqnw2\nujrskHHJskn1MWbhahgBsuojzztfgLNNmWTHj1zr9NeaFqle3dzoQiUB4I3LA3M2Nz/iHA8vvu6p\n0RxRF/eYNBozddplALyznc1aLfwQpoZDALg62vzl3SG1puSxML2XP7qv1xRGu3PCmFuTdu/uxmiB\nyy8FqPAyC5XjVmxli8Lm7EstUUHin9MYmfDjwvZa2OwnYZ21eRxwTXvi0vYB4LJoFs7/cMvPcgq8\nCPd9sNrjtqpmWRsmeozPks8BMCAeyMv7WyxkjRuBr/RzPk7ubdvMKLQEQxZwPSPafTWN4TzZ1+6S\nWvUoPG+KjmPH7YL4HAD3hrWk9qnve3tNtERIIRxkHaMeA1ooZ/Qbb93SEnaOdYkQa17InNQdpmti\n7cIpdaGI1We4abuDls5DnrSELFMLqp6ZxdH+6x2ocItzZYtx6DvCSZYA771fevyjq5LeddLfxlFP\nAZ6fIfH8Ipf3vBiAGRTG9fzj+0346e51VGGXXUtECFsv6TP/3wCYeXp91w8vTs27W9kuT7yOHdyr\nG/Sd6pfZ00LrS5CdGXYHIKYeLLdYO3jc35odL1qHLJ1CCCGEEEIIIbJGEVo624LFTb081GZ5FgCh\nwmd2bknyI1IbiY9b3MDNPsv2InBS/AwAu4c0bVrbs4AWE7M7WakNj74g7FsYs4dtwgwc3PDGlqvO\nScMMxniNiHAD8FQLO0oA+PYHywE4NhxFWmTn8GC5bu/u7v+fDUlHJO4QcSdPxQ5U37XSOkN74qcT\nAA7zOLrbgfBaER7/hpyekFzSuBj7AenCyiTX0nSI2oMgzj/TPnisdVgdWxw/Y4KNnXSp6Z9a1Jb7\n+orXoLJvNiTNEAPqb5phSNPfoc2ivxKerbN2dW6aoXrPdKEAspY2oOvbZuncFL4FQIVnCA8vfOwj\npjSzVYmwcDQAw/0a8xhwQQ9bte1RaZbLxAc3l0fg095OAuDE2+2fF/vb/u49Hdb4iE3enhXNDFod\n8uetccH7dkxrAlR5Bl52TfImT7Y5LG4HWNz4lzekMadNR1n8YnWcyg7+e0izaV/ggYFz3rSSX3sE\nX7Nf/bkRH7LlmW4QZ6y3Czssfrt5Oi5iuuuZ3mfC2FZYOL28SvR/UprnteI6mHzq1rbLF/b80/Wg\nLa9V3Q5rfyb0E5cuBOCAMLadcomGyNIphBBCCCGEECJrlLSlM4mWIfAXPstzNDBlfP7kaRs2a/rC\n/jZr86L3ngzssv16/5TkWKY8MD7hCz7jONqPXfWIxfmTJ4NUTLY2jGm75e7+YLUCV14AtdMzK1fH\n8Oig/etnG9e0y8JpzIoW33eDn8OTjoYzK5N2768QOGLenfXT58790eqnFVQIVDP0OfpdADb2cEv2\nO1DrsT5pHNy0rSjxvo+tbdJfMcTaMDBSyNe1uCnUyf70oQMBGMSYLca90EIN3rkTPOvj6YXlnXJv\nj8OBBmHyv/Z2nxK2cKZ47OX2Dbq6uFXr0CVmifzPJeaF8oOHl9QPcmsw/puec4dd0E/2p6qaZmL3\n/Synz07v+lLSXqk7TO3X7bsfBsLMIvceaQVprffbgFoqvfeZRmMeiJYfYXWot2hP8lSv4dKmVu8t\nz40lBx7jS3d2XOAMse+Y5+vCjA/czhf+kmxlC7NwXh3tGjXTL+dprcvwaoFeo+dbTd2GsacAFXOh\nsmcGg6nvTxcmeVsSlcNzhiydQgghhBBCCCGyRmlaOs9OANjZLWSpXXBoZYRpSV5EaivxRxZUUntp\n4/4H4jEln622IROvu5zbPVHawLk+R7Kg9RnICo33GyyHGZ71siOzhh823ifAO29vC0CPzhe1f7/t\nZncAZnoWv/EfdWFyB64y566/1vbnnzctAnq2f3+FwC2bj6/T5yhvDwpX+VKBW5buTgDYlkfs89Qv\ncm2VWfrWB/NAGOtD74qjt9j8tP0tgHnaE42toese9yC6Ar+2hS6xzpK77yivsbh7Yu3d1ry/e+Ba\nt56lxzfNtX5YsKzF7O1Wk61aHHJH/9A4VvFmSy7MhdFiOi8/3G9E9yU5lCpHHHczAMM8jP6t/aDW\nDSPDgx3Bf/jQtJ7l9sC7NM/8JhbObsAkj5kMu75jC+uSjsmcQQ68CLgsybcYWWe3O15veWW/BIBP\nh/rr71fjYABCOMt7klZ/1yRLQ8Hk2de1QcJ8kwBwfbT6k2+HJY3WnufW/8lJYXlppJwxwe+qTSyd\nYV1mLbMzU6PpXI/lPjtju845U8O2Of/OknzpnHGlJatYP9c+pwWdq0fNy5NEbWBFAsDCIxs/fKZl\nIkKYQKknDmrIT1++qO5G3/2R9G6e5Ema9rPo9eMAeCtD7pOHRnP7mhPqy1ekbY+eG30pycyXtQlz\n0JuUVr0YuQk41j+0pTyE+VLX9m18Hhza47fAHzoiYP6YmAAwr3O9Tgfu4gsvF/jL5hb4a1T1Lzmt\neqD32SRKdTqkud96V2tSV9yqsT40PO49v8ikkJln7/+pKy9R689kaVmIzl7yaDZQkRZd/4G9XMZ9\nPRmH+6+uWWvlG74cvpdtiVtF+l9PryEvetsjWJH72Vi7TxzGf7xp5TTG9bKb0rXP+BP2cU12equ3\ngyKFPZmyFoCwXzoJeBRXR3tWqJju/xmvTrZiyKEA9GE9B1TYdjWfEN7w5XgAIaQFHArBFc/TybyV\nXylyjpcv2gR0qrsCGff8w9xql/s169TYlV51Zctan/Tr8/zZFnZMe/KfMOy9BdvAYps8uvlD64uj\ntzwfN/3SkyA1cadN+b83XamwnEJkd+xtcFPTFVOTjOy/6eS+aB9yrxVCCCGEEEIIkTVKzNJpPkGT\nTrUZm4XeG/5QLEHyJxB/1zg9dcqSsR6gPq5crJw/BOC1XadQZZPLVA9L8idOBxl90G1AexJ2p3gp\n6mrLXnFbZ/ud3NBgxFhPEnDuhnZ/SQaw2X/cG29mX4gXuZvSqJZLaST7VgBw3uafAvDpGT7rWtF4\n3GY6ZVDWHDPCmvdn1Xc989IetlDgCYQyxSvvW3Xyxa5vGOq/iYVJfgRqM7czdJbJ/MBEU+K+JiMG\nx8MIYZx/SgCYvcrKLHwUzF38S3s8bau7P5XX0kYpn41WbmBdWNrs+nSW/7Gwih/To9G6aUzccgMA\nt/yOBfp4xYJtel3S/NiCYnldYfgz3XpPxZaj1scZtjC98QXXKwcxNtq1sDr8mvqiQPlnQLTnpBo/\nB6u+lfnvWD/Y0jJdm3b0aHFo7vBbRxdgc5NH38NfWAXUFyrbcfEHbdy5lVpZ4e65g9MSNAXwc+/e\n9UbiqpMAqB1mfTO3koux8lb3zjjEk8V5CasB49KyI0kWpOw4F9xlro3Z8iVIvUCaWoBF25ClUwgh\nhBBCCCFE1igpS+cF8S9Agxk8zxM97ptJfgRqI7Pi7Lpi6SkVlqndYzmhXOI5T4kWMLUgwOiVn7HO\nMrEGNceyaNHxfw/3AI0tnBV7WxtGuNXoiiR3grVAONhnS18N3Nbf+mZc1vIBPNnbTT4bPaeZcgMA\na8K6DEmYe94ZbkH7M6lPMDPo+Od8KcmDRDnk6QSAhW4JSK1BafKdosJjcw9xi+dz5+4EwKMMBeDo\ncCZNr9MTw2EAxKfM9lNrIZ38I/Zkp9CCpTCHjA0H2oIn1Ykv+rnqeW/m/M3a9sQ1LQS6mIGbidHc\nMWaFD9slZ6EwOvbn1tC8S8nX11n/x+EnuRSpcBiR8FJoHDMYTvR70y1J7uVpBWEPK53yRqd9AVg8\nBs6Ilihmnpcng+a9AADiBZ8C4C6P7w0/KiTvurWEYW5yTRIATrikcez80pu/DyfZupXfs2tB+ixa\n4Z5mkxdmsOxIEfPldKEY710FgCydQgghhBBCCCGyRmlYOm9NAOjvJVI8NxvhlkKabfpkeoRraRpJ\nEPbyqeaCyHqXO8ZjdVJ+C6zhwPwKk0fib3zG2EsYNBsTOt/b4Un2BWotzyYAhP6XwDdsmStaHj55\niI9xC+c/ogU+3hjeaTJybYYEzCU2y9woa+3RvuDXrlLnuX3NGpgm4f/SBz7f2TXJizwZwS2eAydO\naLKiOW8Uz5C6vWWWnoZZvfv1fYf2ZXfONP+0xn+PoWkwWuo90Rvinz3vwCdkbW1ImlHyp09bGadZ\nhRDs1g72iMcD8J9hEGuarKvyc7q63y25FapQGJEAsOHWTszyLNVVnpyi+rj2ZzPoOBZvyeKtZVA2\nK2bvzXa/iWf1YH2YbMuzrA0T7/GxnoJ6WWLr/xi4zs+F/42W5blgSz+5pXNpssWKuqWHgv2y09jF\n11d6/adQ4JnFs8TNx44F6vOsXBYtP0fBHuMWSOZD0qScTHW0+1EuS6fI0imEEEIIIYQQImuUgKXz\nBGK1zbym9RyHpqEW3ZN8CJRZBnkB2qdP3sqgNH/ia956urG6inJA7z0BeOD1Lza7h/f5N46oiy16\nqtkxueSLPZ8EzNJ5Qm1aWj3Jlzgdxy14jWKiFieNhmw8y7PFNTDuJd+0thstE4YXuNUgLSg/pPWb\nzOEcAHp7fcA6hibwSJIJqXLHX+y4vr93fdf0u7yidJnEKffobLG4HspI9+8Ub83dDrG7eaykGaZr\nu8MF8Q4Aphfyb+H8pG5xfLwSgD2nW4bxtCThkGjpar98tN0/3llms+fzOm9ROa/48Hi3692banWD\nVWmMV9itODyrXqm1Z4GDP2Fc67FnlI2eknp6V7goLen4VCH8T9wM6aaqTcfCL1dbtuZQV0w2feax\n8zNcdUmdJXPy6MZ7eyKeBdRnqr2LegvnjNC4/mdxkXhreqW/6z5HvdtkfWESfmq/tXU0zuz/XLyG\ngaGpN0rr+eevrR3rnyvDv9q9L1HUL512oYtn7UvNVdZT5Q+1oV8hXOgyw7SnP/lJ5ALPZH3Hjpae\n5FvvWXr22d0bDHrDmtVb2d3Z0fyD5ubz4ccfbuZdUcjFxNvO0X8y953Dwwl1fdNG2z86faGc521z\nL5gtvXTuE4eV5ItLWrx7i8fVYnvhBA7491WNPp8M7BXO8U83bDG+1Bgcj2S+P6CNTztXJPkSJ8/Y\ntFM3T/p1K4FnUzet9L61Lsm9WG3g+qNswmSal7V6y/t/F6wMTFzkk2fNPF38btBXsi5fNrhsyXkA\nrL6pvi+dQDki8ePmrosFT2UCwPDD/PMjUJ/KrRXXo4Ns+zUP2n/gSxPtuNf0rB+yzZn+P5matF/O\nDBOO/RiAePw2daVDYs1xtu7rLu+C+vGXHWPH/ML/slCPmT63uyJNsOP/v3BpLDpXy+b4a9wFgNtd\nvy985A62nZN8iNN2ViYAfG7DegCqu/cB4OmwDo60dW257zwdFwEwz/8f6XM2vZ7toKA5xq9LyemF\n8Uwt91ohhBBCCCGEEFmjeC2d/SYBUHPV5LquMNVnqzygvdgYuxZq92n7dtN7pUtu4WywLnV9amop\nG+2euKP6XF/Xt+TUH/hS0nYhMsSNP/kOAM970pmq8VBdWfxJlO4OloDihu1gYTuqBYy1SgN82pMG\nhR/6bz3cRiG4Q2eatHh3KRRifvSJQ4D6s2rP8cCC+1oaXnI8MeordUkY/hkH2EKddT518jvc2yRX\nYuUXn5VfEhexfxgDWEISgLBHak4r0KRZK6ysyqS37ePMno1X145p/PnTwBl+iQqh0nsfy5p4mcV+\nn73DEQCsb7BmJ3cnLnYrV809ECfsZR9GfvL4vw03i8lSP4fT9DqjvP0w7kZ1uC2jMmYGkzvc8g7x\n4B4AzKyyNTOqGrsLbQK6uNWz6dPHMXE3209wv8vfJVmQNccMSnitSZmbvhvS8mT5TALVdjb0thJ7\nZ7ildl5QoWDYAAAGLUlEQVTnTYxYbsm97m6tV9i0hJB653gZt20OLk4PytjPlE7yK0YdsnQKIYQQ\nQgghhMgaRWjpTACIXeqTBz3uvteEmuY3KRLCPpcwOvYHoCf/r9kxP3mvonGsZgNGxD0AmE990HTl\nnp4u3EtYpIxJcw3xUoPexmNyi8WHjPRZ1PRITrhuFixo/n9RXCQA9DgvsvRyi519PrS+uvCRH/wW\ngAfDHxrtr1TZmf8DYKN//lz+ROkAfpJVNeneBeqTfpUX/3Kfiy5vmKfKxiqzOKybb+1OdcnMyoPb\nwsHcOtSWa/bwzoNOtPahJB8itQKzwG7b0ywhcWU/AN78uq1d4DmiKjxx1sVrLyTUpeQvFgunlena\n2MMsnNObVG6qGgkhVPinIovxcsKlZrmJewRqrvHOa1oe35TUe+q73tZ4gqnrwxsUtvfNTMLqhwAY\nFM3ucr7X8xoYLPfCi/F7bKZTo63GftYs/JPrLNtLsy9qjuj+0Ous9ufK1Lvogxd3bHF8QfNBAkCP\nzvb0EPcMHOy5NL4RTwNgYtip8Tb9bJsuT1vpqP/tHbjDV23/kWeSKjaPhq3Eck5J49DzoJMsnUII\nIYQQQgghskbRWTpnxDMBqGngm33bd9Oc1knO5ck0i8OrW10/dytFtSvr/icNUzonHRUpR3i0zKHW\nVFnCOEIYAC1YfYuSaQknTPOyNQutFEw80A7cHI/nPeevEP7gM1HpGVp3bFNLZ2lzfLBY4zRh3Hlx\nmS2Ex/MjULuwTKV//5XFmBBeB2Ddj3qwldO4pLk7/A2AH2OWzSejlW8YEv7qI5I8SJVPfkF4xM71\nqX6Sb1hpFpbuXS/Om1Stw/JtpyWbum+wUhJ3f2qE9adxfWHBlpsWOissE3zNkfaxaWx5GBxhSZJT\nkTLOsASAwHvE734KgJo7tjLeOSrajeoH/AyAyWGFrQhvZFzE7GH5L9LiAGNJk2n4hbnZ2L8kyzLl\nj69/6r663/gkSz/B5CFJvsTJEAkA4W/vEP9k9xvCtQBEf2XYb9EaAJZ7KaSlva3/DmCSP2qE7Rc1\n2l+xsjleyNTU4ySPmbZl6RRCCCGEEEIIkTWKx9Lp9Rv38xmJNXkURWQDi3ELl/pM46VpfyHHhnSQ\nsQkAoYnZ69y9oNhn1TpKXy9t2W9V/mIPOo7Fh+weXgTguTgQgGO5gXKxWAOE/pF4nV23Lx5/IQBT\nw7kAVIa7fFSSB8kKhQSAKg8Ru6ur1RNkb+vnL0muBWoXG7pbwezhHOU9RWjhdOL19TkjGtInFml8\n11aZTrij9a4X1XVWwBVZkUbknmVjRtVl6T1x6UJbCC/kS5wMM5PwpQds8SQr0BpPtx/xuPBlAJb7\neb29b3HayEVM3j/N2pvkSM4M49bMKVu13uceWTqFEEIIIYQQQmSNorF0nvYTqz655orG/VVDoPqJ\nPAgkhMgaYVg6857kU4wMMR2AgSHNKl0+Vk4Arkjqrfmnpp3z8iVNwRJO+D0A8bNfsw6Pa+fI/MhT\nzkRLFs4m/zze213CVb5U/LWjhUh5b8E2sPjjfIuRRbwu9k3WhpuaWPa3sAIm2RaobCmal86mVKVe\nLoufRD8QIYQQxc1KAMIL/kB0ZJI3Scqdb79pvs5f9FILI+M9tiLoZVOUHv0+Wse7F/QBIFz8fe9N\n8iaPKF3kXiuEEEIIIYQQImsUjaXz2pCWzbBZ4OrF6Zrb8yGOEEIIIUqQu8Mz1tYl4VidR2mEyC4b\nul/VIKFhkk9RRIkjS6cQQgghhBBCiKwRYoz5lkEIIYQQQgghRIkiS6cQQgghhBBCiKyhl04hhBBC\nCCGEEFlDL51CCCGEEEIIIbKGXjqFEEIIIYQQQmQNvXQKIYQQQgghhMgaeukUQgghhBBCCJE19NIp\nhBBCCCGEECJr6KVTCCGEEEIIIUTW0EunEEIIIYQQQoisoZdOIYQQQgghhBBZQy+dQgghhBBCCCGy\nhl46hRBCCCGEEEJkDb10CiGEEEIIIYTIGnrpFEIIIYQQQgiRNfTSKYQQQgghhBAia+ilUwghhBBC\nCCFE1tBLpxBCCCGEEEKIrKGXTiGEEEIIIYQQWUMvnUIIIYQQQgghsoZeOoUQQgghhBBCZI3/D4vq\nWbumY8f6AAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "slice = 15\n", "predicted = model.predict(X_test[:slice]).argmax(-1)\n", "\n", "plt.figure(figsize=(16,8))\n", "for i in range(slice):\n", " plt.subplot(1, slice, i+1)\n", " plt.imshow(X_test_orig[i], interpolation='nearest')\n", " plt.text(0, 0, predicted[i], color='black', \n", " bbox=dict(facecolor='white', alpha=1))\n", " plt.axis('off')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Adding more Dense Layers" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "collapsed": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "model = Sequential()\n", "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n", " border_mode='valid',\n", " input_shape=(1, img_rows, img_cols)))\n", "model.add(Activation('relu'))\n", "\n", "model.add(Flatten())\n", "model.add(Dense(128))\n", "model.add(Activation('relu'))\n", "\n", "model.add(Dense(nb_classes))\n", "model.add(Activation('softmax'))" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 11918 samples, validate on 10000 samples\n", "Epoch 1/2\n", "11918/11918 [==============================] - 0s - loss: 0.3044 - acc: 0.9379 - val_loss: 0.1469 - val_acc: 0.9625\n", "Epoch 2/2\n", "11918/11918 [==============================] - 0s - loss: 0.1189 - acc: 0.9640 - val_loss: 0.1058 - val_acc: 0.9655\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.compile(loss='categorical_crossentropy',\n", " optimizer='sgd',\n", " metrics=['accuracy'])\n", "\n", "model.fit(X_train, Y_train, batch_size=batch_size, \n", " nb_epoch=nb_epoch,verbose=1,\n", " validation_data=(X_test, Y_test))" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test score: 0.105762729073\n", "Test accuracy: 0.9655\n" ] } ], "source": [ "#Evaluating the model on the test data \n", "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n", "print('Test score:', score)\n", "print('Test accuracy:', accuracy)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Adding Dropout" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "collapsed": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "model = Sequential()\n", "\n", "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n", " border_mode='valid',\n", " input_shape=(1, img_rows, img_cols)))\n", "model.add(Activation('relu'))\n", "\n", "model.add(Flatten())\n", "model.add(Dense(128))\n", "model.add(Activation('relu'))\n", "model.add(Dropout(0.5))\n", "model.add(Dense(nb_classes))\n", "model.add(Activation('softmax'))" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 11918 samples, validate on 10000 samples\n", "Epoch 1/2\n", "11918/11918 [==============================] - 0s - loss: 0.3128 - acc: 0.9097 - val_loss: 0.1438 - val_acc: 0.9624\n", "Epoch 2/2\n", "11918/11918 [==============================] - 0s - loss: 0.1362 - acc: 0.9580 - val_loss: 0.1145 - val_acc: 0.9628\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.compile(loss='categorical_crossentropy',\n", " optimizer='sgd',\n", " metrics=['accuracy'])\n", "\n", "model.fit(X_train, Y_train, batch_size=batch_size, \n", " nb_epoch=nb_epoch,verbose=1,\n", " validation_data=(X_test, Y_test))" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test score: 0.11448907243\n", "Test accuracy: 0.9628\n" ] } ], "source": [ "#Evaluating the model on the test data \n", "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n", "print('Test score:', score)\n", "print('Test accuracy:', accuracy)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Adding more Convolution Layers" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "collapsed": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "model = Sequential()\n", "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n", " border_mode='valid',\n", " input_shape=(1, img_rows, img_cols)))\n", "model.add(Activation('relu'))\n", "model.add(Convolution2D(nb_filters, nb_conv, nb_conv))\n", "model.add(Activation('relu'))\n", "model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n", "model.add(Dropout(0.25))\n", " \n", "model.add(Flatten())\n", "model.add(Dense(128))\n", "model.add(Activation('relu'))\n", "model.add(Dropout(0.5))\n", "model.add(Dense(nb_classes))\n", "model.add(Activation('softmax'))" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 11918 samples, validate on 10000 samples\n", "Epoch 1/2\n", "11918/11918 [==============================] - 1s - loss: 0.4707 - acc: 0.8288 - val_loss: 0.2307 - val_acc: 0.9399\n", "Epoch 2/2\n", "11918/11918 [==============================] - 1s - loss: 0.1882 - acc: 0.9383 - val_loss: 0.1195 - val_acc: 0.9621\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.compile(loss='categorical_crossentropy',\n", " optimizer='sgd',\n", " metrics=['accuracy'])\n", "\n", "model.fit(X_train, Y_train, batch_size=batch_size, \n", " nb_epoch=nb_epoch,verbose=1,\n", " validation_data=(X_test, Y_test))" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test score: 0.11954063682\n", "Test accuracy: 0.9621\n" ] } ], "source": [ "#Evaluating the model on the test data \n", "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n", "print('Test score:', score)\n", "print('Test accuracy:', accuracy)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Exercise\n", "\n", "The above code has been written as a function. \n", "\n", "Change some of the **hyperparameters** and see what happens. " ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "collapsed": true, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "# Function for constructing the convolution neural network\n", "# Feel free to add parameters, if you want\n", "\n", "def build_model():\n", " \"\"\"\"\"\"\n", " model = Sequential()\n", " model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n", " border_mode='valid',\n", " input_shape=(1, img_rows, img_cols)))\n", " model.add(Activation('relu'))\n", " model.add(Convolution2D(nb_filters, nb_conv, nb_conv))\n", " model.add(Activation('relu'))\n", " model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n", " model.add(Dropout(0.25))\n", " \n", " model.add(Flatten())\n", " model.add(Dense(128))\n", " model.add(Activation('relu'))\n", " model.add(Dropout(0.5))\n", " model.add(Dense(nb_classes))\n", " model.add(Activation('softmax'))\n", " \n", " model.compile(loss='categorical_crossentropy',\n", " optimizer='sgd',\n", " metrics=['accuracy'])\n", "\n", " model.fit(X_train, Y_train, batch_size=batch_size, \n", " nb_epoch=nb_epoch,verbose=1,\n", " validation_data=(X_test, Y_test))\n", " \n", "\n", " #Evaluating the model on the test data \n", " score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n", " print('Test score:', score)\n", " print('Test accuracy:', accuracy)" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 11918 samples, validate on 10000 samples\n", "Epoch 1/2\n", "11918/11918 [==============================] - 1s - loss: 0.5634 - acc: 0.7860 - val_loss: 0.3574 - val_acc: 0.9363\n", "Epoch 2/2\n", "11918/11918 [==============================] - 1s - loss: 0.2372 - acc: 0.9292 - val_loss: 0.2253 - val_acc: 0.9190\n", "Test score: 0.225333989978\n", "Test accuracy: 0.919\n", "1 loop, best of 1: 5.45 s per loop\n" ] } ], "source": [ "#Timing how long it takes to build the model and test it.\n", "%timeit -n1 -r1 build_model()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Batch Normalisation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## How to BatchNorm in Keras\n", "\n", "```python\n", "from keras.layers.normalization import BatchNormalization\n", "\n", "BatchNormalization(epsilon=1e-06, mode=0, \n", " axis=-1, momentum=0.99, \n", " weights=None, beta_init='zero', \n", " gamma_init='one')\n", "```" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "collapsed": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# Try to add a new BatchNormalization layer to the Model \n", "# (after the Dropout layer)" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.4.3" } }, "nbformat": 4, "nbformat_minor": 0 }