{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 1 : Neural networks on MNIST database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) The MNIST database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem addressed by MNIST data is a supervised learning problem, and more precisely a classification problem. The objective is to build a model that can recognize numbers from images.\n",
    "The training datasets contains 60000 example of images (arrays 28\\*28) which makes 784 features. The testing datasets contains 10000 examples.\n",
    "The evaluation criteria used is the test error rate (ie the rate of missclassified images).\n",
    "\n",
    "So far, the best performance reported for a fully connected neural networks is 0.35%. Better results have been reached with convolutional neural networks: 0.23 %. This is the best performance so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/home/Anaconda/anaconda3/envs/tensorflow/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "import numpy as np\n",
    "\n",
    "# Load data through keras library\n",
    "(X_train, Y_train), (X_test, Y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28)\n",
      "(10000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an overview of the dataset (first 20 images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5ba7715128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "w = 10\n",
    "h = 10\n",
    "fig = plt.figure(figsize=(8, 8))\n",
    "columns = 4\n",
    "rows = 5\n",
    "for i in range(0, columns*rows):\n",
    "    img = np.random.randint(10, size=(h,w))\n",
    "    fig.add_subplot(rows, columns, i+1)\n",
    "    plt.imshow(X_train[i])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can check the corresponding labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9],\n",
       "      dtype=uint8)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_train[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape training and testing data\n",
    "X_train = X_train.reshape(X_train.shape[0], 28*28)\n",
    "X_test = X_test.reshape(X_test.shape[0], 28*28)\n",
    "\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "X_train /= 255\n",
    "X_test /= 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape labels\n",
    "from keras.utils import np_utils\n",
    "Y_train = np_utils.to_categorical(Y_train, 10)\n",
    "Y_test = np_utils.to_categorical(Y_test, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the dataset is ready to process. We'll use keras library to learn and test neural networks with diferent sets of parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import modules\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, Activation, Input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the parameters chosen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters\n",
    "activation = 'relu'  # activation function\n",
    "optimizer = 'adam'  # optimization\n",
    "loss = 'categorical_crossentropy' # Cost function\n",
    "batch_size = 30 # Batch size \n",
    "epochs = 5 # Epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize model\n",
    "model = Sequential()\n",
    "\n",
    "# Define NN structure\n",
    "inputs = Input(shape=(784,))\n",
    "x = Dense(200, activation=activation)(inputs)    # First hidden layer with 200 nodes\n",
    "x = Dense(200, activation=activation)(x)         # Second hidden layer with 200 nodes\n",
    "predictions = Dense(10, activation='softmax')(x)\n",
    "\n",
    "model = Model(inputs=inputs, outputs=predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer, loss=loss,metrics=['accuracy'])   # optimization parameters + cost function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 19s 324us/step - loss: 0.2083 - acc: 0.9375\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 19s 311us/step - loss: 0.0897 - acc: 0.9720\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 19s 320us/step - loss: 0.0610 - acc: 0.9801\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 19s 309us/step - loss: 0.0459 - acc: 0.9852\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 18s 298us/step - loss: 0.0368 - acc: 0.9880\n",
      "Training time : 94.06952619552612 sec\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "t = time.time()\n",
    "# Train model\n",
    "model.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size)\n",
    "t = time.time() - t\n",
    "print(\"Training time : \" + str(t) + \" sec\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 3s 321us/step\n",
      "[0.07905915418734595, 0.978799996137619]\n",
      "Accuracy : 0.978799996137619\n",
      "Test error rate : 0.021200003862380967\n"
     ]
    }
   ],
   "source": [
    "# Testing the model\n",
    "score = model.evaluate(X_test, Y_test, batch_size=10)\n",
    "print(score)\n",
    "print(\"Accuracy : \" + str(score[1]))\n",
    "print(\"Test error rate : \" + str(1-score[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this set of parameters, we find a test error rate approximately equal to 2% which is better than the 3% needed in the assignement. Objective completed !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}