{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning from Scratch With Python\n",
    "\n",
    "This post describes the steps involved in deep learning, with a practical example, written in Python.  The code creates a multi layer neural network with logistic regression. This system will be trained with a set of images of handwritten digits and then be tested with a different set of images to see if it can identify the digits.\n",
    "\n",
    "This is not a complete guide to Machine Learning or Deep Learning, but just one example, offered without rigorous proof of the equations used. If you're looking for a course on the subject, [Andrew Ng's Machine Learning course on Coursera](https://www.coursera.org/learn/machine-learning) would be a very good and popular choice.\n",
    "\n",
    "The 4 layers of this example neural network are:\n",
    " \n",
    "1. Input layer of 256 inputs\n",
    "1. Hidden layer of 30 nodes\n",
    "1. Hidden layer of 20 nodes\n",
    "1. Output layer of 10 outputs\n",
    "\n",
    "\"nodes\" and \"layers\" are merely names used to conceptualize the mathematics.  In the code, the numbers of layers and nodes simply translate to the sizes of the weights arrays. These arrays store the \"learned\" information of the system.\n",
    "\n",
    "There are some great machine learning/deep learning libraries for Python, such as [scikit-learn](http://scikit-learn.org/stable/), [TensorFlow](https://www.tensorflow.org) and [Theano](http://deeplearning.net/software/theano/).  To better illustrate deep learning, we are not going to use these libraries, but instead code the system from scratch. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only Python libraries needed are numpy for matrix manipulation, matplotlib for plotting the data, random to initiate the weights, and urllib to pull down the data from a URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy import genfromtxt\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import random\n",
    "import urllib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoid Function\n",
    "\n",
    "The sigmoid function outputs a value between zero and one from an input between $-\\infty$ and $\\infty$. It is used to take an input and turn it into an activation value for a \"neuron\". or node.\n",
    "\n",
    "$ h_\\theta (x) = g(\\theta^T x) = g(z) = \\dfrac{1}{1 + e^{-z}} = \\dfrac{1}{1 + e^{-\\theta^T x}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1.0 /(1.0 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Partial Derivative of the Sigmoid Function\n",
    "\n",
    "Later, we use gradient decent to make corrections to the weights. This is how the system learns. \"_Gradient_ decent\" suggests differentials, and indeed, we will need the partial differential of the sigmoid function:\n",
    "\n",
    "$ \\dfrac{dg(z)}{dz} = g(z) (1 - g(z))$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoidDerivative(z):\n",
    "  return sigmoid(z) * (1 - sigmoid(z));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The system uses \"weights\" to determine the influence of each activation value on the output of each neuron.  It is these weights which are altered during the learning phases.  To start with, these will be initialized to random values.  If we initialized the weights to zero, the system would not be able to form suitable differentials between the neurons.\n",
    "\n",
    "This function creates a 2D array of random weights, $w$ where $ -1 \\le w_{ij} \\le 1 $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def initWeights(numInputs,numOutputs):\n",
    "    return 2 * np.random.random((numInputs,numOutputs)) - 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"Bias terms\" are used to add extra weighing to the inputs.  Think of these as the intercept point of the equations.\n",
    "\n",
    "This function prepends a column of ones to an array for adding bias terms to inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def addOnes(arr):\n",
    "    ones = np.ones((arr.shape[0],1))\n",
    "    return  np.insert(arr,[0],ones,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function prepends a column of zeros to an array. This is to remove the influence of the bias term weights later on for regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def addZeros(arr):\n",
    "    zeros = np.zeros((arr.shape[0],1))\n",
    "    return  np.insert(arr,[0],zeros,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our system is designed to predict boolean outputs of either zero or one.  However, the sigmoid function outputs a number somewhere between zero and one. This output is the probability that Y is one, so we simply return one if the sigmoid function output is greater than half, otherwise we return zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convertOutputs(outputs):\n",
    "    return (outputs > 0.5).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction\n",
    "\n",
    "The \"predict\" function uses the weighted terms and inputs to predict outputs for each layer in the neural network.\n",
    "\n",
    "For each layer:\n",
    "\n",
    "$ h_\\theta (x) = g(\\theta X)$\n",
    "\n",
    "where:\n",
    "\n",
    "* $g$ is the sigmoid function\n",
    "* $h_\\theta$ is the hypothesis function (output) for each layer\n",
    "* $x$ are the inputs on each neuron\n",
    "* $\\theta$ are the weights\n",
    "\n",
    "Function outputs:\n",
    "\n",
    "* list of activations per layer (for every sample)\n",
    "* weighted outputs per layer (for every sample)\n",
    "* last layer outputs (per sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# X is an array of inputs per sample\n",
    "# thetas is a list of 2D weights for each layer\n",
    "def predict(X,thetas):\n",
    "    \n",
    "    activations       = []  # net inputs on each neuron per layer\n",
    "    weighted_outputs  = [] # weighted outputs from each neuron per layer\n",
    "    \n",
    "    a = X # First set of activations are the inputs\n",
    "    \n",
    "    for theta in thetas:\n",
    "        a_bias = addOnes(a)      # add bias terms to activations\n",
    "        activations.append(a_bias)\n",
    "        \n",
    "        z = a_bias.dot(theta.T)  # add weights to each activation per neuron\n",
    "        weighted_outputs.append(z)\n",
    "        \n",
    "        a = sigmoid(z)           # h(x) out of each neuron in the layer (also the activations for next layer)\n",
    "    \n",
    "\n",
    "    outputs = a # final outputs\n",
    "    \n",
    "    return (activations,weighted_outputs,outputs)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost Function for Regularized Logistic Regression\n",
    "\n",
    "The learning algorithm seeks to minimize this cost function.\n",
    "\n",
    "$ J(\\theta) = - \\dfrac{1}{m} \\sum\\limits_{i=1}^{m} \\sum\\limits_{j=1}^{J} [t_i \\ log(h_\\theta(x_{ij})) + (1 - t_{ij}) \\ log(1-h_\\theta(x_{ij}))] + \\dfrac{\\lambda}{2m} \\sum\\limits_{k=1}^{K} \\theta_{k}^2 $\n",
    "\n",
    "Where:\n",
    "\n",
    "* $\\theta$ are the weight terms\n",
    "* $m$ is the number of training examples\n",
    "* $J$ is the number of output nodes\n",
    "* $K$ is the number of weight terms across all nodes (not including the bias values)\n",
    "* $\\lambda$ is the regularization parameter\n",
    "* $t$ is the target output\n",
    "* $h(\\theta)$ is the predicted output, or hypothesis function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# X is a 2D array of inputs for each sample\n",
    "# ideal is a 2D array of ideal values for each sample\n",
    "# lambd is the regularizaion parameter\n",
    "# thetas is a list of 2d arrays of theta, starting at the \"input -> first layer\" theta\n",
    "def cost(X,ideal,lambd,thetas):\n",
    "    \n",
    "    (activations,weighted_outputs,predicted) = predict(X,thetas)\n",
    "\n",
    "    m = ideal.shape[0] # Numer of samples\n",
    "    \n",
    "    J = (1/m) *  sum(sum(  (-ideal * np.log(predicted) - (1 - ideal) * np.log(1-predicted)) ));\n",
    "        \n",
    "    # Sum all the theta values together, except the regularized terms\n",
    "    sumThetas = 0\n",
    "    for theta in thetas:\n",
    "        sumThetas += sum(sum( theta[:,1:] * theta[:,1:] ))\n",
    "        \n",
    "    # Regularised cost function \n",
    "    r = (lambd/(2*m)) * (sumThetas);\n",
    "    \n",
    "    return J + r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backpropagation\n",
    "\n",
    "The _grads_ function returns the gradients of the cost function by \"backpropagation\". First the forward pass is run (using the \"predict\" function). This computes all the activation terms, for all nodes in the network.\n",
    "\n",
    "Then the gradients of the error terms are calculated and summed over all the training samples. The code shows different calculations for output layers (where the target output value is known), and the hidden layers (where the target value has to propagate down from the output layer).\n",
    "\n",
    "The gradient of the cost function (the return value of this function) is then calculated as:\n",
    "\n",
    "$\\dfrac {\\partial}{\\partial \\theta}  J(\\theta) = \\dfrac{1}{m} \\Delta $ for the output layer\n",
    "\n",
    "$\\dfrac {\\partial}{\\partial \\theta}  J(\\theta) = \\dfrac{1}{m} \\Delta  + \\dfrac{\\lambda}{m} \\theta$ for the hidden layers\n",
    "\n",
    "for each neuron, where:\n",
    "\n",
    "* $m$ is the number of data samples\n",
    "* $\\lambda$ is the regularization parameter\n",
    "* $\\Delta$ are the accumulated gradients (for all test samples)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def grads(X,Y,lambd,thetas):\n",
    "    \n",
    "    m = X.shape[0]\n",
    "    Xb =  addOnes(X)\n",
    "    deltas = []\n",
    "    \n",
    "    # Init the deltas\n",
    "    for theta in thetas:\n",
    "        deltas.append(np.zeros(theta.shape))\n",
    "    \n",
    "    # Make predictions\n",
    "    (activations,weighted_outputs,outputs) = predict(X,thetas)\n",
    "    \n",
    "    # Backpropogation\n",
    "    for t in range(0,m): # for every data row\n",
    "\n",
    "        # Output layer\n",
    "        deltaOut = np.array([(outputs[t] - Y[t,:])])\n",
    "        deltas[-1] = deltas[-1] + deltaOut.T.dot(np.array([ activations[-1][t] ]))\n",
    "        \n",
    "        # Work backwards through hidden layers\n",
    "        dLast = deltaOut\n",
    "        for layer in range(len(deltas) -2,-1,-1):\n",
    "            \n",
    "            a = np.array([ activations[layer][t] ])\n",
    "            z = np.array([ weighted_outputs[layer][t] ])    \n",
    "            d = (thetas[layer + 1].T.dot(dLast.T))[1:].T * sigmoidDerivative(z);\n",
    "            deltas[layer] = deltas[layer] + d.T.dot(a)\n",
    "            \n",
    "            dLast = d\n",
    "    \n",
    "    return [ (1/m)*deltas[i] + (lambd/m) * addZeros(thetas[i][:,1:]) for i in range(0,len(deltas)) ]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Data\n",
    "\n",
    "The data source is a selection of [1593 handwritten digits](https://archive.ics.uci.edu/ml/machine-learning-databases/semeion/semeion.names).  The data consists of rows of digit data with each row containing a 16x16 pixel image (the first 256 columns), followed by 10 digits of ones and zeros, representing the Y value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Y values are stored as ten columns of data. \"1\" in the first column signifies zero, a \"1\" in the second signifies two and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ansToDigit(y):\n",
    "    ans = (y > 0.5).astype(int)\n",
    "    for i in range(10):\n",
    "        if ans[i] == 1: return i\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is extracted from the given URL. It is then split into Dev and Test data. The Dev data being used for training, while the Test data is used to check the results of the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X data: (1593, 256)\n",
      "Y data: (1593, 10)\n"
     ]
    }
   ],
   "source": [
    "# data description:\n",
    "# https://archive.ics.uci.edu/ml/machine-learning-databases/semeion/semeion.names\n",
    "dataSource = (\n",
    "  'https://archive.ics.uci.edu/ml/machine-learning-databases/semeion/semeion.data'\n",
    ")\n",
    "data = np.loadtxt(urllib.request.urlopen(dataSource).readlines())\n",
    "X = data[:,0:256].astype(int)\n",
    "Y = data[:,256:].astype(int)\n",
    "print(\"X data:\",X.shape)\n",
    "print(\"Y data:\",Y.shape)\n",
    "\n",
    "\n",
    "# Shuffle the values\n",
    "XShuffled = []\n",
    "YShuffled = []\n",
    "indexes = np.random.permutation(len(Y))\n",
    "\n",
    "for i in indexes:\n",
    "    XShuffled.append(X[i])\n",
    "    YShuffled.append(Y[i])\n",
    "\n",
    "# Use most for dev and 100 for test\n",
    "XDev  = np.array(XShuffled[0:-100])\n",
    "YDev  = np.array(YShuffled[0:-100])\n",
    "XTest = np.array(XShuffled[-100:])\n",
    "YTest = np.array(YShuffled[-100:])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### System Initialization\n",
    "\n",
    "Initial values of $\\theta$ (the weights) are set. The input layer is of a known size (the 256 pixels of the images), as is the output layer (10 outputs representing the output number).  However the number of hidden layers and the nodes per hidden layer can be of any size we choose. One hidden layer is usually acceptable for such a case, however two hidden layers are used here, by way of an example. The only representation of these sizes in the code is in the size of these weights arrays (theta1, theta2, etc), which can be changed as required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Seed weights with random numbers\n",
    "np.random.seed(1)\n",
    "hiddenLayer1Size = 30\n",
    "hiddenLayer2Size = 20\n",
    "\n",
    "theta1   = initWeights(hiddenLayer1Size,XDev.shape[1]+1)   # between input and hidden layer\n",
    "theta2   = initWeights(hiddenLayer2Size,hiddenLayer1Size+1) # between hidden layer and output\n",
    "theta3   = initWeights(Y.shape[1],hiddenLayer2Size+1) # between hidden layer and output\n",
    "thetas   = [ theta1,theta2,theta3 ]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Learning Phase\n",
    "\n",
    "The code below puts the training (Dev data) dataset through the learning algorithm, updates the weights, then tests its performance using the testing dataset.  It loops through this process 500 times, which is enough to bring the performance of the tests to an acceptable level.\n",
    "\n",
    "The weights are updated by simply subtracting the error gradients.  More complex algorithms are available which can improve the efficiency here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".................................................."
     ]
    }
   ],
   "source": [
    "# Try out the cost function\n",
    "\n",
    "lambd = 0.1\n",
    "devCosts = []\n",
    "\n",
    "testAccuracies = []\n",
    "\n",
    "for i in range(0,500):\n",
    "    J = cost(XDev,YDev,lambd,thetas)\n",
    "    (theta1Grad,theta2Grad,theta3Grad) = grads(XDev,YDev,lambd,thetas)\n",
    "    if i % 10 == 0: print('.', end=\"\")\n",
    "    \n",
    "    theta1 = theta1 - theta1Grad\n",
    "    theta2 = theta2 - theta2Grad\n",
    "    theta3 = theta3 - theta3Grad\n",
    "    \n",
    "    thetas = [ theta1,theta2,theta3]\n",
    "    devCosts.append(J)\n",
    "    \n",
    "    \n",
    "    # Run test against test data\n",
    "    (activations,weighted_outputs,outputs) = predict(XTest,thetas)\n",
    "    answers = convertOutputs(outputs)\n",
    "    right = wrong = 0\n",
    "    \n",
    "    for k in range(0,len(outputs)):\n",
    "        if np.array_equal(np.array(YTest[k]),np.array(answers[k])): right +=1\n",
    "        else: wrong +=1\n",
    "    testAccuracies.append(right/(right+wrong))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Plot of the cost against the number of iterations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGVRJREFUeJzt3Xt4XNV97vHvb2Y0ukuWJfl+kTF2XGOCTQ2xY0i4lEAI\nAdokT6EphzTpoSVwIG16gXOetKft0+b0OW0uza0QAvQkOZCGEOCQNCEBYooNDjI2YGNj8B3fJPki\nybqOZtb5Y7aEMB7vsfForxm9n+eZZ+/Ze0v+LTO8Xlraa21zziEiIsUjFnUBIiJychTcIiJFRsEt\nIlJkFNwiIkVGwS0iUmQU3CIiRUbBLSJSZBTcIiJFRsEtIlJkEoX4pk1NTa6lpaUQ31pEpCStXbu2\nwznXnM+1BQnulpYWWltbC/GtRURKkpntzPdaDZWIiBQZBbeISJFRcIuIFBkFt4hIkVFwi4gUGQW3\niEiRUXCLiBQZr4L7a0++zsot7VGXISLiNa+C+5u/2sqqNzqiLkNExGteBbcZZDJ6eLGIyIl4Fdwx\nMxTbIiIn5lVwG5Bxim4RkRPxK7gNlNsiIifmWXAbTsktInJCXgV3zNAYt4hICK+C28w0xi0iEsKr\n4I5pjFtEJJRXwQ2GbuMWETkxr4I7ZqBRbhGRE/MquLMzJ6OuQkTEb14Fd3bmpHrcIiIn4lVwZ2dO\nRl2FiIjf/ApuM91VIiISwrPgRjMnRURC5BXcZvYnZrbRzDaY2QNmVlGQYrQ6oIhIqNDgNrPpwG3A\nUufcIiAOXFeIYsy0OqCISJh8h0oSQKWZJYAqYG9BitEYt4hIqNDgds7tAf4J2AXsAzqdc08Uohit\nxy0iEi6foZIG4BpgDjANqDaz3z/OdTeZWauZtba3n9oDf02rA4qIhMpnqOS3gO3OuXbnXAp4GHj/\nsRc55+52zi11zi1tbm4+pWK0HreISLh8gnsXsMzMqszMgEuBTQUpRqsDioiEymeMew3wEPAi8Erw\nNXcXohhD63GLiIRJ5HORc+6vgb8ucC165qSISB48mzmp9bhFRMJ4Fdxaj1tEJJxXwZ2dORl1FSIi\nfvMquGO6HVBEJJRXwa31uEVEwvkV3FodUEQklGfBrfW4RUTCeBXcWh1QRCScV8Gt1QFFRMJ5Fdwx\n05R3EZEwXgU3mvIuIhLKq+DW6oAiIuE8C27D6YZAEZET8iq4NeVdRCScV8GtKe8iIuG8Cm5Qj1tE\nJIxXwR3TlHcRkVBeBbemvIuIhPMquDXlXUQknFfBrSnvIiLh/Apu9bhFREJ5FtzqcYuIhPEquLMP\nCxYRkRPxKrgNrQ4oIhLGq+COxbTIlIhIGK+CWz1uEZFwfgW3oZmTIiIhPAtu3Q4oIhLGq+COacq7\niEgor4I7O3My6ipERPzmVXDrCTgiIuG8Cm4MMpmoixAR8ZtXwR0zTZ0UEQnjVXBrdUARkXBeBbfW\n4xYRCedVcGt1QBGRcJ4Ft545KSISxrPg1gQcEZEweQW3mU0ws4fMbLOZbTKz5QUpxrQ6oIhImESe\n130V+Jlz7uNmlgSqClGMVgcUEQkXGtxmVg98APgUgHNuEBgsRDExrQ4oIhIqn6GSOUA7cJ+ZrTOz\ne8ys+tiLzOwmM2s1s9b29vZTKsbMyGixEhGRE8onuBPAucC3nHNLgB7gjmMvcs7d7Zxb6pxb2tzc\nfErFaD1uEZFw+QT3m8Cbzrk1wfuHyAb5aWdoAo6ISJjQ4HbO7Qd2m9l7gkOXAq8WpBjdDigiEirf\nu0r+G/D94I6SbcAfFKKY7MzJQnxnEZHSkVdwO+fWA0sLXIvW4xYRyYNXMydRj1tEJJRXwR3TbSUi\nIqG8Cm6txy0iEs6r4I5pdUARkVBeBbfW4xYRCedZcGsCjohIGL+CO9hqEo6ISG5eBffwU96V2yIi\nuXkV3EFua5xbROQEvAruWBDcim0Rkdy8Cm4LutzqcYuI5OZZcGe3ym0Rkdz8Cm70y0kRkTBeBfdb\nY9xKbhGRXLwK7rfuKom2DhERn3kV3G/dx63kFhHJxavgHqYet4hIbl4Fd8x0I7eISBivglszJ0VE\nwnkV3CNj3BHXISLiM6+CWz1uEZFwngW3JuCIiITxK7iDrW4HFBHJzavg1hi3iEg4r4JbY9wiIuG8\nCu6YVgcUEQnlVXAPrw6oHreISG5+Bbd63CIioTwLbt0OKCISxqvg1nrcIiLhvApurcctIhLOq+Ae\nvo87reQWEcnJq+CuKIsD0J9KR1yJiIi/vAru6mQCgD4Ft4hITl4Fd1V5tsfdMzAUcSUiIv7yKriH\ne9y9g+pxi4jk4lVwVyXV4xYRCZN3cJtZ3MzWmdnjhSpmOLjV4xYRye1kety3A5sKVQhAdXl2qKRn\nUD1uEZFc8gpuM5sBfAS4p5DFlCdixAz61OMWEckp3x73V4C/ADIFrAUzozqZoGdAwS0ikktocJvZ\nVUCbc25tyHU3mVmrmbW2t7efckFV5XF6NVQiIpJTPj3uFcDVZrYDeBC4xMy+d+xFzrm7nXNLnXNL\nm5ubT7mg6mSCHg2ViIjkFBrczrk7nXMznHMtwHXAU8653y9UQZXJOL26HVBEJCev7uOGbI/7qIJb\nRCSnkwpu59yvnHNXFaoYgPqqMjr7UoX8I0REipp3Pe6mmiQHewajLkNExFveBXdjdTmHegbJaE1u\nEZHj8i+4a5KkM07DJSIiOXgY3OUAdBwdiLgSERE/eRfcTdVJADqOapxbROR4vAtu9bhFRE7Mu+Ce\n3lAJwM6DPRFXIiLiJ++Cu6Y8wcyJlWza3x11KSIiXvIuuAEWTKlj876uqMsQEfGSl8F99vR6tnX0\nsPdIX9SliIh4x8vg/u0l0wG4+5ltEVciIuIfL4N75sQqPvm+Wdy/egffVniLiLxNIuoCcvmrq87i\ncE+Kv//pJtqPDnDHFQuIxSzqskREIudtcCcTMf7l+iVMrE5y9zPb2N7Rw5d/dzE15d6WLCIyJrwc\nKhkWjxl/e81Z/M+PLuSpzW187Jur2X2oN+qyREQi5XVwQ/YBwp9aMYd/+4Pz2dfZx9Vff5bnth6M\nuiwRkch4H9zDLpjXxKO3XsDE6iQ3fGcN331+Z9QliYhEomiCG2BOUzU/vmUFF85r4guPbOC///gV\nBocyUZclIjKmiiq4AeoqyrjnxvO4+aK5/N81u/jkPc9rQSoRGVeKLrgh+0vLv7xiAV+9bjGv7Onk\n6q89y4Y9nVGXJSIyJooyuIdds3g6D/3x+3HAx/91NY+9tDfqkkRECq6ogxtg0fR6Hrv1As6eXs9t\nD6zjH3+2mbSeVykiJazogxugubac7//hMq4/fxbf+tVW/uv/aaWrX8+sFJHSVBLBDdmZll/8nbP5\nu2sX8cyWdq79xiq2tR+NuiwRkdOuZIJ72A3LZvO9P3wfR3pTXPONVfzqtbaoSxIROa1KLrgBlp3R\nyKO3rGBGQxWfvv8F7lq5Fec07i0ipaEkgxuyS8P+6OblfHjRVL74H5v53A/W059KR12WiMi7VrLB\nDVCVTPD131vCn31oPo+9tJdP/OtzeqqOiBS9kg5uyC5Sdesl8/j2DUvZ3tHD1V9/ltYdh6IuS0Tk\nlJV8cA/7rYWTeeSW91NbUcb1336eB369K+qSREROybgJboAzJ9XyyGdXsHxuE3c+/ApfeGQDqbQW\nqRKR4jKughugvqqM+z51Hn/0gTP47vM7+eQ9azioRapEpIiMu+CG7CJVd175G3zldxfz0u4jXP31\nVbzyphapEpHiMC6De9i1S4JFqpzjY99azXef26H7vUXEe+M6uAHOnlHPT267kBVnNvKFRzdy6wPr\n6NY6JyLisXEf3AAN1Um+c+N5/OUVC/jZhv189GvPsnGvhk5ExE8K7kAsZtx80VwevGkZ/akMv/3N\n1dy3ajsZLRErIp4JDW4zm2lmT5vZq2a20cxuH4vConJey0R+ctsFXHhmE3/z/17lhnvXsK9Tsy1F\nxB/59LiHgM875xYCy4BbzGxhYcuKVmNNOffcuJQv/s7ZrNt1hMu//AyPrt8TdVkiIkAewe2c2+ec\nezHY7wY2AdMLXVjUzIzrz5/FT2+7kLmTarj9wfXc/L21tHX1R12aiIxzJzXGbWYtwBJgTSGK8VFL\nUzU//KPl/Pnl7+HJzW1c+qWVPPjrXbptUEQik3dwm1kN8CPgc865ruOcv8nMWs2stb29/XTWGLlE\nPMYtF5/Jz26/kIVT67jj4Ve47u7n9YQdEYmE5dNzNLMy4HHg5865L4Vdv3TpUtfa2noayvNPJuP4\n99bd/MNPN9GXSvPpC+Zw68VnUltRFnVpIlLEzGytc25pPtfmc1eJAd8BNuUT2qUuFjOuO38Wv/z8\nB7l28XTuWrmNS/55JT9s3a1bB0VkTOQzVLICuAG4xMzWB68rC1yX9ybVVvC/P3FO8Ii0Sv78oZe5\n9purtNa3iBRcXkMlJ6uUh0qOJ5NxPPrSHv7Xf2zmQNcAly6YxJ9d/h5+Y2pd1KWJSJE4maESBfdp\n1Ds4xH2rdnDXyq10Dwxx9TnT+NPL5jO7sTrq0kTEcwruiHX2prjrma3ct2oHqXSGj507g5svmktL\nkwJcRI5Pwe2Jtu5+vvHUGzzwwm6G0hk+8t5pfPaiuRpCEZF3UHB7pq27n3uf3cH3nt/J0YEhLl0w\nic9ePJffnD0x6tJExBMKbk919qb4t+d2cO+q7RzpTXHOzAl8ekULH140lWRCCzWKjGcKbs/1DAzx\n0No3uX/1DrZ39DCptpwbls3m9943i8aa8qjLE5EIKLiLRCbjWLmlnXtXbec/X+8gmYhx1dlTue78\nWZzX0kB27pOIjAcnE9yJQhcjucVixsULJnHxgkm80dbN/at38Mi6vTy8bg9nNFdz3Xkz+di5M9QL\nF5G3UY/bM72DQ/zk5X384IXdtO48TFncuGzhZD7xmzO5YF4TZXGNhYuUIg2VlIjXD3Tz4Au7efjF\nNzncm2JidZKPnD2Va5dM49xZGkoRKSUK7hIzMJTmmS0dPLJ+D7989QADQxlmNFRy9TnT+Og501gw\npVYhLlLkFNwlrLs/xRMbD/DI+j2seqODjIPZjVV8aOFkLj9rCufOaiAWU4iLFBsF9zjR3j3AE6/u\n5+cbD/Dc1g5SaUdTTTmXLZzM5WdNZvncRsoT8ajLFJE8KLjHoa7+FE9vbuOJjQd4+rU2egfTVJbF\nWT63kQ/Ob+aD85u1VoqIx3Q74DhUV1HGNYunc83i6fSn0qze2sHK19pZuaWdpza3AdkhlQ/Ob+YD\n85o5/4yJ1OmpPSJFST3ucWBHRw/PvN7OytfaWb31IH2pNDGDs6bVs+yMiSw7o5GlLROpr1SQi0RF\nQyWS08BQmrU7D/P8tkOs2XaQdbuOMJjOEDNYOK2OZXOyIX7urAlMqquIulyRcUPBLXnrT6VZt+sI\nz287yJrtB3lx1xEGhzIATJ9QyeJZEzh3VgNLZk3grGl1+mWnSIFojFvyVhH8AnP53EYg2yPfuLeL\ndbuO8OKuw6zfdYSfvLwPgGQ8xsJpdbx3Rj2LptWzcFod8yfXamVDkTGmHreEOtDVz7pdR1i36zDr\ndh1h495OegbTAJTFjfmTazlrWh1nTatn0fQ6Fkypo7pcfQKRk6GhEimoTMax81AvG/Z0snFvFxv3\nZreHegZHrpk5sZL5k2qZN7mW90ypYd6kWs6cVENFmYZaRI5HQyVSULGYMaepmjlN1Xz0nGkAOOfY\n39XPhj1dbNrXxZYD3bx+4CjPvN5OKp3tHMQMZjdWM29SDfMn1zJvcg0tjdW0NFXrjhaRk6DgltPC\nzJhaX8nU+kouWzh55HgqnWFHRw9bDhxly4HukdeTm9tIZ976aa+xOsmcpmyIz2mq5oxgv6Wxmsqk\neukio2moRCIxMJRm18Fetnf0sL2jhx0He9jWnt1v6x5427VT6yuY2VDFjIbK7Gtidn9mQxVT6ytI\naKlbKQEaKhHvlSfizJucHQM/1tGBIXYEYb69vYcdB3vZfbiXNdsP8cj6PkZ11InHjCl1FcycWMmM\nhipmNlQxbUIFU+srmVJfwZT6Cmr0i1IpMfpEi3dqyhMsml7Poun17zg3OJRhf2c/uw/38ubhXnYf\n6stuD/fxn6+3c6Br4B1fU1ueGAnxqfUVTKmvZErd8H4FU+oqqK8s06qKUjQU3FJUkokYsxqrmNVY\nddzz/ak0B7r62dfZz/7OfvZ3Zbf7OvvY39nPlgPdtHUPcOwIYSJmNNWU01SbpLmmnKaacppr37lt\nrimnrjKh9c8lUgpuKSkVZXFmN1YzuzH3SoipdIb27gH2dfZzIAj2jqMDtHcPZLdHB9i0r5uOowMM\nZd75O6BkPEZTTZKG6iQTq5M0VGW3E6rKcr7XbZByOim4Zdwpi8eYNqGSaRMqT3hdJuPo7EvRfnSA\nju5soLcH247uQQ73DnKoZ5Bdh3o53DNIV/9Qzu9VlYzTUJWkobqMhqok9ZVl1FeWUTe8rSijrjIx\naj97vLYioeeMyjsouEVyiMWMhupsz3r+cX6JeqxUOsOR3hRHgkDPBnuKw72DHO4Z5NDINsWbh/vo\n6kvR2Zc6bq9+tKpk/LjhXluRoLo8+6qtSFCdTFBTkaCmPPsaOV6eoKosrjH8EqLgFjlNyuKx7Dh4\nbXneX+Ocoy+VpqtviM6+FF39qZFA7+pL0dU/NLI/fH7vkX429XVzdGCIowNDb7sfPhczqE4mqC6P\nvy3YR+9XJeNUJuPBNhv2lcPHyuJUJRNvnQ/OlSdiGu+PgIJbJEJmRlUyQVUye+fLyXLOMTCUyYZ4\n/9BImPcMHLuf5mj/O4/v6ukd2e8dTDMQrAyZr5gxEuiVZfG3h3/Z8PEYFWVxKsqyQT+8LS+LUzHq\n/bHXVJTFKE9ktxVlcZLxmH5qCCi4RYqYmY0EXlNN/j39XNKZ7E8AvYND9A9m6E1lA71vME3vYPZ4\n32A6uOat432jrhs+d6inj77BIfpTGfqH0vSnsv8wvJs5f8lEjIrh0C+LUZGIUx5sh0M/OfyKj9pP\nxCiPH3su/rb3Ob82/s5jiZhF+pOGgltERsRjNjJ8UgjOOQbTGfpTGQaG0gykMiOBPno7fL4/x/mR\nrx1669q+VJrDvYMMDmUYTGey2+A1ELw/XcwYCfLyUaE+qbaCf//j5aftz8lFwS0iY8bMKE/Egwdy\njO3CYs45UunsPxypUeE+MBzwo8M+nc597tivHXWuaozW1VFwi8i4YGYkE5Z98Me7H1WKlG4QFREp\nMnkFt5ldYWavmdkbZnZHoYsSEZHcQoPbzOLAN4APAwuB681sYaELExGR48unx30+8IZzbptzbhB4\nELimsGWJiEgu+QT3dGD3qPdvBsdERCQCp+2Xk2Z2k5m1mllre3v76fq2IiJyjHyCew8wc9T7GcGx\nt3HO3e2cW+qcW9rc3Hy66hMRkWPkE9wvAPPMbI6ZJYHrgMcKW5aIiOSS18OCzexK4CtAHLjXOff3\nIde3AztPsaYmoOMUv7ZYqc3jg9o8Ppxqm2c75/IarijIU97fDTNrzfdJx6VCbR4f1ObxYSzarJmT\nIiJFRsEtIlJkfAzuu6MuIAJq8/igNo8PBW+zd2PcIiJyYj72uEVE5AS8Ce5SXYHQzO41szYz2zDq\n2EQz+4WZvR5sG4LjZmb/EvwdvGxm50ZX+akzs5lm9rSZvWpmG83s9uB4ybbbzCrM7Ndm9lLQ5r8J\njs8xszVB234QzIXAzMqD928E51uirP/dMLO4ma0zs8eD9yXdZjPbYWavmNl6M2sNjo3pZ9uL4C7x\nFQjvB6445tgdwJPOuXnAk8F7yLZ/XvC6CfjWGNV4ug0Bn3fOLQSWAbcE/z1Lud0DwCXOuXOAxcAV\nZrYM+Efgy865M4HDwGeC6z8DHA6Ofzm4rljdDmwa9X48tPli59ziUbf9je1n2zkX+QtYDvx81Ps7\ngTujrus0tq8F2DDq/WvA1GB/KvBasH8XcP3xrivmF/AocNl4aTdQBbwIvI/sRIxEcHzkcw78HFge\n7CeC6yzq2k+hrTPIBtUlwOOAjYM27wCajjk2pp9tL3rcjL8VCCc75/YF+/uBycF+yf09BD8OLwHW\nUOLtDoYM1gNtwC+ArcAR59xQcMnodo20OTjfCTSObcWnxVeAvwCGn8TbSOm32QFPmNlaM7spODam\nn209czJizjlnZiV5a4+Z1QA/Aj7nnOsys5Fzpdhu51waWGxmE4AfAwsiLqmgzOwqoM05t9bMLoq6\nnjF0gXNuj5lNAn5hZptHnxyLz7YvPe68ViAsIQfMbCpAsG0LjpfM34OZlZEN7e875x4ODpd8uwGc\nc0eAp8kOE0wws+EO0uh2jbQ5OF8PHBzjUt+tFcDVZraD7ANWLgG+Smm3GefcnmDbRvYf6PMZ48+2\nL8E93lYgfAy4Mdi/kewY8PDx/xL8JnoZ0Dnqx6+iYdmu9XeATc65L406VbLtNrPmoKeNmVWSHdPf\nRDbAPx5cdmybh/8uPg485YJB0GLhnLvTOTfDOddC9v/Zp5xzn6SE22xm1WZWO7wPfAjYwFh/tqMe\n6B81aH8lsIXsuOD/iLqe09iuB4B9QIrs+NZnyI7rPQm8DvwSmBhca2TvrtkKvAIsjbr+U2zzBWTH\nAV8G1gevK0u53cB7gXVBmzcAfxUcPwP4NfAG8EOgPDheEbx/Izh/RtRteJftvwh4vNTbHLTtpeC1\ncTirxvqzrZmTIiJFxpehEhERyZOCW0SkyCi4RUSKjIJbRKTIKLhFRIqMgltEpMgouEVEioyCW0Sk\nyPx/4ku7JrQmYacAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bf48d8438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(devCosts)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Plot of the accurancy on the test data against the number of iterations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGiFJREFUeJzt3X901fWd5/HnOwkkQPiVEH4mIaARRFTUFEntVivqoqdH\np9vpVJ2etjNtaXfGrTN1Z1aPezxduzPbTmfbsVvPTLF13XbcsVZ3Ro5lyija6lhQUEAEBEL4kQCS\n5CYhP8nP9/5xL8wFCblJ7r3fe795Pc7J4X6/93vvfX/g8uLD5/v9fj7m7oiISLjkBF2AiIgkn8Jd\nRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSOEuIhJCCncRkRBSuIuIhFBeUB88a9Ysr6ioCOrjRUSy\n0ttvv93k7iXDHRdYuFdUVLBt27agPl5EJCuZ2ZFEjtOwjIhICCncRURCSOEuIhJCCncRkRBSuIuI\nhJDCXUQkhBTuIiIhFNh17iIimcTd+dmWIzS196T8s1ZfPoery2ak9DMU7iIiwO7jbTzywm4AzFL7\nWbOnFWRGuJvZGuAxIBf4sbt/+7zny4H/A8yIHfOgu29Icq0iksVO9w3Q1t0XdBlDeuX9BgC2PLSa\nudMLAq5m7IYNdzPLBR4HbgXqga1mtt7d98Qd9l+BZ939b81sGbABqEhBvSKShfoHBrnxu69ysi31\nQx5jsXjWlFAEOyTWc18J1Lh7LYCZPQPcBcSHuwPTYo+nA8eTWaSIZLfdx9s42dbD56sXsmTu1KDL\nGdKKFA+VpFMi4b4AqIvbrgeuP++YbwL/Ymb/CZgC3JKU6kQkJdyd//7LvdQ1d6Xl8461dgNw382X\nMntqOHrGmS5ZJ1TvAZ5y9/9pZtXAz8xsubsPxh9kZmuBtQDl5eVJ+mgRGamahg5+8q+HKJ05icL8\n9FxX8dmqMgV7GiXyp3oMKIvbLo3ti/clYA2Au282swJgFtAQf5C7rwPWAVRVVfkoaxYZl3bWtXIq\nSSckX9vfCMDTX76ehcVTkvKeklkSCfetQKWZLSIa6ncD9553zFFgNfCUmV0OFACNySxUZDzb90E7\ndz3+RlLfc2HxZMqLJif1PSVzDBvu7t5vZvcBG4le5viku+82s0eBbe6+HngAeMLM/pToydUvurt6\n5iJJ8vqBaF/pyS9WMX3ShKS8Z9nMyViqL+iWwCQ02Ba7Zn3DefseiXu8B7ghuaWJhJO785Wfvs3+\nk+0Jv6a5s5eFxZO5eemcFFYmYaI7VEXS7HCki5f3nmRlRRELZk5K+HW3LVOwS+IU7iIptqU2wrv1\nrWe39xxvA+B/fPpKLikpDKosCTmFu0gKDQ46f/z0O0Q6e8/ZXzm7kMWzdJWKpI7CXeQiBgadnv6B\nUb++pqGDSGcvf/Gp5fzOigVn9xdMyNXJTEkphbvIENydOx57nX0jOPE5lBsvK2FKmm4WEgGFu8iQ\nDjV1su9kO3etmM+yedOGf8EQ5s+YROlMXU8u6aVwl4x3JNLJD1+pYWBw6Fsnyosnc//qylEPdRyN\ndPHDVw/QP/Bvn3FmPpT7V1eyWCc+Jcso3CXjPf3mUZ5/p37Iywa7ewdo2t7Lf7imlPLi0fWQn37r\nCM+9/eHPWL10Not04lOykMJdMsbBxg5auz48d8pr+xupWljEs1+rvuDrDpxs59bvv8Zz79Rz42Ul\no/rs1/Y3cW35TJ77jx8d1etFMo3CXTLC0UgXt3zvNww1acU3br1syNdeOruQudMK+MGmA/xg04FR\n1/Ant1SO+rUimUbhLhnh9ZpG3OF7v3c1xYX55zyXa0ZVxcwhX2tmPPvVag5FOkf9+cN9hki2UbhL\n4PafbOfhf3yP2VPz+dQ1C0Z1UrS8ePKox9tFwign6AJE1u+Irsr49TFc7SIi51LPXQLj7vxy1wk2\n7v6AFWUz+NyqhUGXJBIa6rlLYLYdaeG+/7udAw0drF46O+hyREJFPXcJzBs1TZjBKw/cRIXGy0WS\nSuEuSefu3PnDN9h9/NRFjxt0uGL+NN0kJJICCndJukNNnew6doo1V8ylcs7Fb9u/aYmGY0RSQeEu\no7btcDPrdx7/0P7DkS4A/nzNEs3JIhIQhbuM2l9t3Mf2oy0UXmAq2+rFxRpuEQmQwl0S5u4cbe6i\np3+QvoFBdhxt5Q9vWMRDd1wedGkich6FuyTstQNNfOHJt87ZV31JcUDViMjFKNwlYb/e10DBhBz+\n+jNXYxiTJ+aOehZGEUkthbsk5I2aJv73G4f52KWz+ORV84MuR0SGoTtUJSE/3XwYgD+4oSLIMkQk\nQeq5y4dsO9x8dom5M7bUNvOZ60pZffmcgKoSkZFQuMs5Wjp7+ey6LRdcr1Q3HIlkD4W7nOPNQxEG\nBp3H772WpfOmnt0/MTeH0iHWMBWRzKNwl7OORrr42t+/Q35eDrcum8PEPJ2SEclW+tsrZ7209yQA\nf/bvlyjYRbKceu6Cu/OzLUd4dmsd5UWT+fK/Wxx0SSIyRgp3oaahg0de2E1ejvFHN10SdDkikgQK\nd2FLbQSATQ/cyMJiTfYlEgYaWBU210aYP72A8iKthiQSFgr3cc7d2VLbzKpLijGzoMsRkSTRsMw4\ntv1oC99/+QDNnb1UL9bsjiJhop77OPbCjuP8tqaJVYuLNK2ASMio5z6O1bd0censQp5ZWx10KSKS\nZOq5j2N1zd2U6SSqSCgp3Mcpd6e+pUvzxYiEVELDMma2BngMyAV+7O7fvsAxvwd8E3Bgp7vfm8Q6\nZYT6Bwa594k3qW/puuDzDnT2DlA6Uz13kTAaNtzNLBd4HLgVqAe2mtl6d98Td0wl8BBwg7u3mJnm\nhg3Y8dbTvHW4mesXFQ15/fqEvBzuuHJumisTkXRIpOe+Eqhx91oAM3sGuAvYE3fMV4DH3b0FwN0b\nkl2ojExdrMd+/y2VfPSSWQFXIyLplsiY+wKgLm67PrYv3mXAZWb2hpltiQ3jfIiZrTWzbWa2rbGx\ncXQVS0LODMeUadhFZFxK1gnVPKASuAm4B3jCzGacf5C7r3P3KnevKikpSdJHy4UcONlBjsG86QVB\nlyIiAUgk3I8BZXHbpbF98eqB9e7e5+6HgP1Ew14CsKv+FD/+10PMmz6JvFxdECUyHiXyN38rUGlm\ni8xsInA3sP68Y/6JaK8dM5tFdJimNol1yghsej+66Ma3fueKgCsRkaAMG+7u3g/cB2wE9gLPuvtu\nM3vUzO6MHbYRiJjZHuBV4M/cPZKqomVoe4638TcvH2D5gmncvFRTCoiMVwld5+7uG4AN5+17JO6x\nA9+I/UiAnvrtIQB+//qFAVciIkHSgGzIbK6NcNuyOdyzsjzoUkQkQAr3EGnq6KGuuZuVi4qCLkVE\nAqZwD5HG9h4AFszQfDEi453CPUQiHb0AFBfmB1yJiARN4R4ikc5oz71oysSAKxGRoCncQ+RMz31W\nocJdZLxTuIdIpLOH3BxjWsGEoEsRkYAp3EMk0tFL0ZSJ5ORY0KWISMAU7iHS2N5DscbbRQSFe2gM\nDjrb61pZNn9a0KWISAZQuIfEdza+T3NnL9WLi4MuRUQygMI9BNpP9/Gj30Qn4bxpiVY4FBGFeyhs\nO9wCwNNfvp6SqbqBSUQU7qGwuTbCxNwcrls4M+hSRCRDKNxDYPPBCCvKZ1AwITfoUkQkQyQ0n7tk\nBnfnT3++gyPNXefsf+/4Kb5+s1Y1FJF/o557Fmls7+Gfdhyns6efwvy8sz83L5nNp65ZEHR5IpJB\n1HPPInUt3QA8ePtSLaEnIhelnnsWqW+JDseUzZwccCUikukU7lmkPtZzXzBTi3GIyMUp3LPIkUgn\nxVMmMnmiRtNE5OIU7lnknaOtXFk6PegyRCQLKNyzxDNvHaWmoYNVmjtGRBKgcM8SP9h0AIBbl+kq\nGREZnsI9C7g7kc5evvrxxVxSUhh0OSKSBRTuWaCzd4Ce/kEtfC0iCVO4Z4FIRw8AxYWa8VFEEqNw\nzwKRzl4AigvVcxeRxCjcs0CkIxbuGpYRkQTpbpgMNjDovH6gkTdqmgANy4hI4hTuGeylPR/wtb9/\nB4BJE3LVcxeRhCncM9jrB5qYMjGXn3+1mpKp+VqMQ0QSpnDPYFtqI3xkURHLF2jKAREZGZ1QzVAN\nbac52NhJtaYbEJFRULhnqM21EQCqL1G4i8jIKdwz1K76U+Tn5XDFfA3JiMjIKdwzVF1LF2VFk8nN\nsaBLEZEspHDPUPUt3ZRqxSURGSWFe4aqb+nWWqkiMmoJhbuZrTGzfWZWY2YPXuS4T5uZm1lV8koc\nf9pO93Gqu089dxEZtWHD3cxygceB24FlwD1mtuwCx00F7gfeTHaR4817x04BUDlHc7eLyOgk0nNf\nCdS4e6279wLPAHdd4LhvAd8BTiexvnFpy8EIOQZVFUVBlyIiWSqRcF8A1MVt18f2nWVm1wJl7v7L\ni72Rma01s21mtq2xsXHExY4Xm2sjXLlgOtMKJgRdiohkqTGfUDWzHOB7wAPDHevu69y9yt2rSkpK\nxvrRodTdO8COulYthC0iY5LI3DLHgLK47dLYvjOmAsuBX5sZwFxgvZnd6e7bklVomG0/2sJ/ef5d\n+gecnv5B+gacVbozVUTGIJFw3wpUmtkioqF+N3DvmSfd/RQw68y2mf0a+M8K9sS9sOM4RyJd3HbF\nXABuXjqbjyrcRWQMhg13d+83s/uAjUAu8KS77zazR4Ft7r4+1UWG3eaDEVYuKuJ/3XNN0KWISEgk\nNOWvu28ANpy375Ehjr1p7GWNH5GOHvadbOfOFfODLkVEQkR3qAbsjYPR2R91AlVEkknhHqCDjR18\n/R+2UzAhh6tKNfujiCSPwj1Ar77fAMC37lrOhFz9UYhI8ihRArT5YIRFs6bwmaqy4Q8WERkBhXtA\nBgadtw41a6xdRFJCC2QHoKu3n5f3NtDe069l9EQkJRTuAXjw+V2s33mc3Bxj1WJNDiYiyadwT7PB\nQee1A418YkkJD9y2hNlTC4IuSURCSGPuabbvZDutXX188qr5LF+gyx9FJDUU7mlW19wFwJK5UwOu\nRETCTOGeZm2n+wGYPklztYtI6ijc0+xUdx+AFuIQkZRSuKdZWyzcCwt0LltEUkfhnmZtp/uYmp9H\nbo4FXYqIhJjCPc3auvuZpvF2EUkxhXuatZ3uY6qGZEQkxRTuadbW3acrZUQk5RTuaTQ46Ow72a5h\nGRFJOYV7Gj3xei2tXX3MKpwYdCkiEnIK9zR6dV8DeTnGN25dEnQpIhJyOrOXQl29/Tzw7E5au6LX\ntr99pIUvfrSCkqn5AVcmImGnnnsKvVET4Z/f+4COnn4GBp2PVBTx6etKgy5LRMYB9dxTaEtthIl5\nOfzia9UUTMgNuhwRGUfUc0+h7UdbWFE2Q8EuImmncE+ho83dLJ41JegyRGQcUrinyOm+AZo6eiid\nOSnoUkRkHFK4p0h9S3RRjtKZkwOuRETGI4V7itS1dANQVqSeu4ikn8I9Rd4/0Q5ARbHG3EUk/RTu\nKeDu/Gr3B1TOLqS4UDcsiUj6KdxT4Du/2sfOulaqLykOuhQRGacU7imwae9Jpubn8fXVlUGXIiLj\nlMI9yZo6ejjQ0MEffeJSZmlIRkQConBPsjMnUq8umx5wJSIyninck+zM9e1lur5dRAKkcE+yupYu\ncnOMedMLgi5FRMYxhXuS1bd0M296AXm5+q0VkeAogZKsrrlLQzIiEjiFe5LVt3RrsjARCVxC4W5m\na8xsn5nVmNmDF3j+G2a2x8zeNbNNZrYw+aVmvtN9AzS092iyMBEJ3LDhbma5wOPA7cAy4B4zW3be\nYduBKne/CngO+KtkF5oNjrVqsjARyQyJ9NxXAjXuXuvuvcAzwF3xB7j7q+7eFdvcAozLhULrYzNB\nqucuIkFLJNwXAHVx2/WxfUP5EvDPYykqWx2JdALquYtI8JK6QLaZfQ6oAm4c4vm1wFqA8vLyZH50\nRth2uIWSqfnMnaZr3EUkWIn03I8BZXHbpbF95zCzW4CHgTvdvedCb+Tu69y9yt2rSkpKRlNvxnJ3\nttRGqF5cjJkFXY6IjHOJhPtWoNLMFpnZROBuYH38AWZ2DfAjosHekPwyM1+ks5eG9h6uKZ8RdCki\nIsOHu7v3A/cBG4G9wLPuvtvMHjWzO2OHfRcoBH5hZjvMbP0Qbxdadc3R88nlRTqZKiLBS2jM3d03\nABvO2/dI3ONbklxX1tGVMiKSSXSHapLUxWaDXKC7U0UkAyjck6SuuZuZkydQmJ/UC5BEREZF4Z4k\nO+taWTp3WtBliIgACvek+PW+BvacaNOC2CKSMRTuSfDg87sAuHnp7IArERGJUriPUU//AB+0neYP\nb1jE8gVaN1VEMoPCfYxOtJ4GYNl8jbeLSOZQuI/RmUsgtUCHiGQShfsYuDt/ueF9AMp0Z6qIZBCF\n+xjsPt7G3hNtzJg8QTNBikhGUbiPwZbaCAC/uv/j5OZoJkgRyRwK9zHYUhth0awpzJ2uXruIZBaF\n+ygNDDpvHmpm1eKioEsREfkQTYQyCr892MR3N+6j/XQ/qxbrrlQRyTzquY/Ci++eYPfxNm5bNodP\n6K5UEclA6rmPQn1LN0vnTmXd56uCLkVE5ILUcx+F+pYu3bQkIhlN4T5Cg4NOfUs3ZVpxSUQymMJ9\nBHbWtfKRv3iZ3v5B9dxFJKMp3Efg3fpWIp29/MENFdxx5bygyxERGZJOqI5AU0cvAA/fcTl5ufp3\nUUQylxJqBJo7e5kxeYKCXUQynlJqBCKdPRRPmRh0GSIiw1K4j0Cko5fiKflBlyEiMiyF+whEOnsp\nLlTPXUQyn8J9BCIdPRRpWEZEsoDCPUEfnDpNS1cfFcVTgi5FRGRYCvcE/d1vDgJQfYlmgRSRzKdw\nT8D+k+089dvDmMHl86YFXY6IyLAU7gmobewA4MkvfETL6YlIVlC4J6C+pRuAa8tnBlyJiEhiFO4J\nqGvuYmp+HtMmabYGEckOSqs4p/sGePrNo5zuGzhn/+baCKVFkzHTkIyIZAeFe5zX9jfyrRf3XPC5\nz1cvTHM1IiKjp3CPc7S5C4C3Hl7NjEnn3qw0IVe9dhHJHgr3OPUt3RTm51FSmK8hGBHJajqhGufM\n2qgKdhHJduq5Axt2nWDDrhNsO9JC1cKioMsRERkzhTvw1xv30djRQ8nUfG5fPjfockRExiyhcDez\nNcBjQC7wY3f/9nnP5wM/Ba4DIsBn3f1wcktNvr0n2jjW0k1tUycP33E5X/n44qBLEhFJimHD3cxy\ngceBW4F6YKuZrXf3+GsGvwS0uPulZnY38B3gs6koOFn2n2zn9sdeP7v9scpZAVYjIpJcifTcVwI1\n7l4LYGbPAHcB8eF+F/DN2OPngB+ambm7J7HWpHr9QBMAf/e56yidOUkTgolIqCQS7guAurjteuD6\noY5x934zOwUUA03JKDLes1vreOL12jG/z8m20ywsnswajbGLSAil9YSqma0F1gKUl5eP6j1mTJ5A\n5ZzCMddSOaeQNcvnjfl9REQyUSLhfgwoi9suje270DH1ZpYHTCd6YvUc7r4OWAdQVVU1qiGb266Y\ny21XqLctInIxidzEtBWoNLNFZjYRuBtYf94x64EvxB7/LvBKJo+3i4iE3bA999gY+n3ARqKXQj7p\n7rvN7FFgm7uvB34C/MzMaoBmov8AiIhIQBIac3f3DcCG8/Y9Evf4NPCZ5JYmIiKjpbllRERCSOEu\nIhJCCncRkRBSuIuIhJDCXUQkhCyoy9HNrBE4MsqXzyIFUxtkOLV5fFCbx4extHmhu5cMd1Bg4T4W\nZrbN3auCriOd1ObxQW0eH9LRZg3LiIiEkMJdRCSEsjXc1wVdQADU5vFBbR4fUt7mrBxzFxGRi8vW\nnruIiFxE1oW7ma0xs31mVmNmDwZdT7KY2ZNm1mBm78XtKzKzl8zsQOzXmbH9ZmY/iP0evGtm1wZX\n+eiZWZmZvWpme8xst5ndH9sf2nabWYGZvWVmO2Nt/m+x/YvM7M1Y234em14bM8uPbdfEnq8Isv7R\nMrNcM9tuZi/GtkPdXgAzO2xmu8xsh5lti+1L23c7q8I9brHu24FlwD1mtizYqpLmKWDNefseBDa5\neyWwKbYN0fZXxn7WAn+bphqTrR94wN2XAauAP479eYa53T3Aze5+NbACWGNmq4guKv99d78UaCG6\n6DzELT4PfD92XDa6H9gbtx329p7xCXdfEXfZY/q+2+6eNT9ANbAxbvsh4KGg60pi+yqA9+K29wHz\nYo/nAftij38E3HOh47L5B3gBuHW8tBuYDLxDdE3iJiAvtv/s95zoOgrVscd5seMs6NpH2M7SWJDd\nDLwIWJjbG9fuw8Cs8/al7budVT13LrxY94KAakmHOe5+Ivb4A2BO7HHofh9i//2+BniTkLc7NkSx\nA2gAXgIOAq3u3h87JL5d5yw+D5xZfD6b/A3w58BgbLuYcLf3DAf+xczejq0fDWn8bqd1gWwZPXd3\nMwvlpU1mVgg8D/yJu7eZ2dnnwthudx8AVpjZDOAfgaUBl5QyZvZJoMHd3zazm4KuJ80+5u7HzGw2\n8JKZvR//ZKq/29nWc09kse4wOWlm8wBivzbE9ofm98HMJhAN9qfd/f/Fdoe+3QDu3gq8SnRYYkZs\ncXk4t11n23yxxecz2A3AnWZ2GHiG6NDMY4S3vWe5+7HYrw1E/xFfSRq/29kW7oks1h0m8QuPf4Ho\nmPSZ/Z+PnWFfBZyK+69e1rBoF/0nwF53/17cU6Ftt5mVxHrsmNkkoucY9hIN+d+NHXZ+m7N28Xl3\nf8jdS929gujf11fc/fcJaXvPMLMpZjb1zGPgNuA90vndDvqkwyhOUtwB7Cc6Tvlw0PUksV3/AJwA\n+oiOt32J6FjjJuAA8DJQFDvWiF41dBDYBVQFXf8o2/wxouOS7wI7Yj93hLndwFXA9lib3wMeie1f\nDLwF1AC/APJj+wti2zWx5xcH3YYxtP0m4MXx0N5Y+3bGfnafyap0frd1h6qISAhl27CMiIgkQOEu\nIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQkhhbuISAj9f7Nj94nmlwRUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bf524d400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(testAccuracies)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction\n",
    "\n",
    "Now the weights are set, these values can be used to make predictions. Here, 5 samples are selected from the Test data set and are run through the prediction algorithm.  Note that the prediction of new data is relatively quick.  The hard work has already been done in setting the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual:   6\n",
      "Guess:    0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADWZJREFUeJzt3WusZfVZx/HvT65CEYZSKbcIGCTBpgUywbY22IgCRcLU\npC8gVqE0IY2iYGrIVBLb+Kq1Wq9NG2yxqAQaKVjSgMNI2xgTmRbG4d7CgMhtuFgM1DYWaB9f7DXm\nzOGcmcPea63ZZ/7fT3Jy1t5r7b2e+e/zm3XZe+0nVYWk9vzY7i5A0u5h+KVGGX6pUYZfapThlxpl\n+KVGGX6pUYZfapThlxq195gr2zf71f4cOOYqR/Ezb/3+7i5Bc+6hew4YZT3/y/d4uX6QlSw7avj3\n50B+LmeMucpRbNiwZXeXoDl31pEnj7KeTXX7ipd1t19q1EzhT3J2km8n2ZpkfV9FSRre1OFPshfw\naeA9wEnABUlO6qswScOaZct/GrC1qh6tqpeB64F1/ZQlaWizhP8o4IkFt5/s7pO0Cgx+tj/JJcAl\nAPszztsdknZtli3/U8AxC24f3d23g6q6qqrWVtXafdhvhtVJ6tMs4f8mcEKS45LsC5wP3NxPWZKG\nNvVuf1W9muRSYAOwF3B1Vd3fW2WSBjXTMX9V3QLc0lMtkkbkJ/ykRhl+qVGjXtgzpg1Pe7GN5sdY\nf4+nnbXyK0zd8kuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CjDLzVqj72w\nZ082VvcXvdaedMGYW36pUYZfapThlxo1S7uuY5J8LckDSe5PclmfhUka1iwn/F4FPlxVm5McBNyV\nZGNVPdBTbZIGNPWWv6q2VdXmbvq7wIPYrktaNXp5qy/JscApwKYl5tmuS5pDM5/wS/IG4EvA5VX1\n0uL5tuuS5tNM4U+yD5PgX1tVN/ZTkqQxzHK2P8DngQer6lP9lSRpDLNs+X8e+HXgF5Ns6X7O6aku\nSQObpVHnvwLpsRZJI/ITflKjVsVVffN+JZVX2bVj2td6Hv+G3fJLjTL8UqMMv9Qowy81yvBLjTL8\nUqMMv9Qowy81yvBLjTL8UqMMv9Qowy81KlU12srWvm3/+saGY0Zb3zS8SEer2aa6nZfqhRVdau+W\nX2qU4ZcaZfilRvXx1d17Jfn3JF/poyBJ4+hjy38Zk249klaRWb+3/2jgV4DP9VOOpLHMuuX/M+AK\n4Ec91CJpRLM07TgXeK6q7trFcpckuTPJnc9/54fTrk5Sz2Zt2nFekseA65k07/j7xQst7NX3pjfu\nNcPqJPVplhbdH6mqo6vqWOB84KtV9f7eKpM0KN/nlxrVS9OOqvo68PU+nkvSONzyS40y/FKjDL/U\nKMMvNcrwS40y/FKjDL/UKMMvNcrwS40y/FKjDL/UKMMvNcrwS40y/FKjDL/UqF6u559H9tyTds4t\nv9Qowy81atamHYckuSHJt5I8mOQdfRUmaVizHvP/OfBPVfW+JPsCB/RQk6QRTB3+JAcDpwMXAVTV\ny8DL/ZQlaWiz7PYfBzwP/E3XpfdzSQ7sqS5JA5sl/HsDpwKfqapTgO8B6xcvZLsuaT7NEv4ngSer\nalN3+wYm/xnswHZd0nyapV3XM8ATSU7s7joDeKCXqiQNbtaz/b8NXNud6X8U+MDsJUkaw0zhr6ot\nwNqeapE0Ij/hJzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK\n8EuNMvxSowy/1CjDLzXK8EuNmrVd1+8muT/JfUmuS7J/X4VJGtbU4U9yFPA7wNqqeguwF3B+X4VJ\nGtasu/17Az+eZG8mffqenr0kSWOY5Xv7nwL+GHgc2Aa8WFW39VWYpGHNstu/BljHpGffkcCBSd6/\nxHK265Lm0Cy7/b8E/EdVPV9VrwA3Au9cvJDtuqT5NEv4HwfenuSAJGHSruvBfsqSNLRZjvk3MWnO\nuRm4t3uuq3qqS9LAZm3X9VHgoz3VImlEfsJPapThlxo1a4vuubXh6S1TPe6sI08ebV3a0TRjr+m5\n5ZcaZfilRhl+qVGGX2qU4ZcaZfilRhl+qVGGX2qU4ZcaZfilRhl+qVGGX2rUHnthj1afMS+Q8iIi\nt/xSswy/1Khdhj/J1UmeS3LfgvsOTbIxycPd7zXDlimpbyvZ8n8BOHvRfeuB26vqBOD27rakVWSX\n4a+qfwFeWHT3OuCabvoa4L091yVpYNMe8x9eVdu66WeAw3uqR9JIZj7hV1UF1HLzbdclzadpw/9s\nkiMAut/PLbeg7bqk+TRt+G8GLuymLwS+3E85ksaykrf6rgP+DTgxyZNJPgh8HPjlJA8zadj58WHL\nlNS3XX68t6ouWGbWGT3XImlEfsJPapThlxo16lV9D91zwNy3w/LKst3HtmfjcssvNcrwS40y/FKj\nDL/UKMMvNcrwS40y/FKjDL/UKMMvNcrwS40y/FKjDL/UqFXRrmveLwbSazn+888tv9Qowy81yvBL\njZq2V98nk3wryT1JbkpyyLBlSurbtL36NgJvqaq3Ag8BH+m5LkkDm6pXX1XdVlWvdjfvAI4eoDZJ\nA+rjmP9i4NblZi5s1/UKP+hhdZL6MFP4k1wJvApcu9wyC9t17cN+s6xOUo+m/pBPkouAc4Ezumad\nklaRqcKf5GzgCuAXqur7/ZYkaQzT9ur7K+AgYGOSLUk+O3Cdkno2ba++zw9Qi6QR+Qk/qVGr4qo+\naWdsezYdt/xSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSo/bY\nq/rGvNJr2r50Y/azm3Y87Lm353LLLzXK8EuNmqpd14J5H05SSQ4bpjxJQ5m2XRdJjgHOBB7vuSZJ\nI5iqXVfnT5l8fbff2S+tQlMd8ydZBzxVVXevYFnbdUlz6HW/1ZfkAOD3mezy71JVXQVcBfATOdS9\nBGlOTLPl/2ngOODuJI8x6dC7Ocmb+yxM0rBe95a/qu4FfnL77e4/gLVV9V891iVpYNO265K0yk3b\nrmvh/GN7q0bSaPyEn9SoPfbCnjGthotmvEBHi7nllxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZf\napThlxpl+KVGGX6pUYZfapThlxqVqvG+Vi/J88B/LjP7MGAevg3IOnZkHTua9zp+qqretJInGDX8\nO5Pkzqpaax3WYR3j1OFuv9Qowy81ap7Cf9XuLqBjHTuyjh3tMXXMzTG/pHHN05Zf0ohGDX+Ss5N8\nO8nWJOuXmL9fki928zclOXaAGo5J8rUkDyS5P8llSyzz7iQvJtnS/fxB33UsWNdjSe7t1nPnEvOT\n5C+6Mbknyak9r//EBf/OLUleSnL5omUGG4+lWsAnOTTJxiQPd7/XLPPYC7tlHk5y4QB1fDLJt7px\nvynJIcs8dqevYQ91fCzJUwvG/5xlHrvTfL1GVY3yA+wFPAIcD+wL3A2ctGiZ3wQ+202fD3xxgDqO\nAE7tpg8CHlqijncDXxlpXB4DDtvJ/HOAW4EAbwc2DfwaPcPkveJRxgM4HTgVuG/BfX8ErO+m1wOf\nWOJxhwKPdr/XdNNreq7jTGDvbvoTS9Wxktewhzo+BvzeCl67neZr8c+YW/7TgK1V9WhVvQxcD6xb\ntMw64Jpu+gbgjCTps4iq2lZVm7vp7wIPAkf1uY6erQP+tibuAA5JcsRA6zoDeKSqlvsgVu9q6Rbw\nC/8OrgHeu8RDzwI2VtULVfXfwEbg7D7rqKrbqurV7uYdTPpSDmqZ8ViJleRrB2OG/yjgiQW3n+S1\nofv/ZbpBfxF441AFdYcVpwCblpj9jiR3J7k1yc8OVQNQwG1J7kpyyRLzVzJufTkfuG6ZeWONB8Dh\nVbWtm34GOHyJZcYcF4CLmeyBLWVXr2EfLu0OP65e5jDodY9Hsyf8krwB+BJweVW9tGj2Zia7vm8D\n/hL4xwFLeVdVnQq8B/itJKcPuK5lJdkXOA/4hyVmjzkeO6jJPu1ufUsqyZXAq8C1yywy9Gv4GSbd\nsU8GtgF/0seTjhn+p4BjFtw+urtvyWWS7A0cDHyn70KS7MMk+NdW1Y2L51fVS1X1P930LcA+SQ7r\nu47u+Z/qfj8H3MRk922hlYxbH94DbK6qZ5eocbTx6Dy7/dCm+/3cEsuMMi5JLgLOBX6t+4/oNVbw\nGs6kqp6tqh9W1Y+Av17m+V/3eIwZ/m8CJyQ5rtvKnA/cvGiZm4HtZ23fB3x1uQGfVncO4fPAg1X1\nqWWWefP2cw1JTmMyTkP8J3RgkoO2TzM5wXTfosVuBn6jO+v/duDFBbvEfbqAZXb5xxqPBRb+HVwI\nfHmJZTYAZyZZ0+0Gn9nd15skZwNXAOdV1feXWWYlr+GsdSw8x/Oryzz/SvK1oz7OUL6OM5nnMDm7\n/ghwZXffHzIZXID9mex2bgW+ARw/QA3vYrIbeQ+wpfs5B/gQ8KFumUuB+5mcMb0DeOdA43F8t467\nu/VtH5OFtQT4dDdm9wJrB6jjQCZhPnjBfaOMB5P/cLYBrzA5Tv0gk/M8twMPA/8MHNotuxb43ILH\nXtz9rWwFPjBAHVuZHEdv/zvZ/k7UkcAtO3sNe67j77rX/h4mgT5icR3L5WtnP37CT2pUsyf8pNYZ\nfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGXGvV/ORTu1UuAsnsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bf48e3390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------\n",
      "Actual:   6\n",
      "Guess:    6\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADY9JREFUeJzt3W2sZVV9x/HvrzPgFKQyiEUEUqBREmpQyISiNdZ0Whgp\ncWziizG1BTEhprXVxoaMJammr7S29tFoqFJpS9TUh0oMdpiipmlSRmE6DE8KA6XKODwUGrCSAqP/\nvjh7zJ3LvcOdc/bec+9d309yc/c5e52z/7PP/O7ae5+zzkpVIak9P3GkC5B0ZBh+qVGGX2qU4Zca\nZfilRhl+qVGGX2qU4ZcaZfilRq0dc2NH5wW1jmPH3KSOkFec89SRLmHFu2f3MYf9mP/jBzxTT2cp\nbUcN/zqO5eezccxN6gjZtm3XkS5hxbvoZa8+7MfsqJuW3NbDfqlRM4U/yaYk306yJ8nWvoqSNLyp\nw59kDfBR4I3A2cBbk5zdV2GShjVLz38+sKeq7q+qZ4DPAJv7KUvS0GYJ/ynAd+fcfrC7T9IKMPjV\n/iRXAFcArOPw37qQNIxZev69wGlzbp/a3XeQqrq6qjZU1YajeMEMm5PUp1nC/03g5UnOSHI0sAW4\nvp+yJA1t6sP+qtqf5F3ANmANcE1V3dlbZZIGNdM5f1XdANzQUy2SRuQn/KRGGX6pUaMO7FE/tn3P\nQTMtmOZ1Pv+ipY+mtOeXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRhl9q\nlAN7euBAm+eaZrYZze6eemzJbe35pUYZfqlRhl9q1CzTdZ2W5GtJ7kpyZ5J391mYpGHNcsFvP/De\nqtqZ5Djg1iTbq+qunmqTNKCpe/6q2ldVO7vl7wN343Rd0orRy1t9SU4HzgV2LLDO6bqkZWjmC35J\nXgh8HnhPVT05f73TdUnL00zhT3IUk+BfV1Vf6KckSWOY5Wp/gE8Cd1fVR/orSdIYZun5fwH4DeCX\nkuzqfi7uqS5JA5tlos5/A9JjLZJG5Cf8pEY5qm+e1TpCz1F2ms+eX2qU4ZcaZfilRhl+qVGGX2qU\n4ZcaZfilRhl+qVGGX2qU4ZcaZfilRhl+qVGrdmDPah2gAw7SUT/s+aVGGX6pUYZfalQfX929Jsl/\nJPlyHwVJGkcfPf+7mczWI2kFmfV7+08FfhX4RD/lSBrLrD3/nwNXAj/qoRZJI5pl0o5LgEeq6tbn\naXdFkluS3PIsT0+7OUk9m3XSjjcleQD4DJPJO/5hfiPn6pOWp1mm6H5fVZ1aVacDW4CvVtXbeqtM\n0qB8n19qVC+f7a+qrwNf7+O5JI3Dnl9q1IoY1bdaR+g5Ok9Hkj2/1CjDLzXK8EuNMvxSowy/1CjD\nLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1KgVMapvuXN0nlYie36pUYZfatSsk3Ycn+RzSb6V\n5O4kr+mrMEnDmvWc/y+Af66qtyQ5Gjimh5okjWDq8Cd5EfB64DKAqnoGeKafsiQNbZbD/jOAR4G/\n7Wbp/USSY3uqS9LAZgn/WuA84GNVdS7wA2Dr/EZO1yUtT7OE/0Hgwara0d3+HJM/Bgdxui5peZpl\nuq6HgO8mOau7ayNwVy9VSRrcrFf7fwe4rrvSfz/w9tlLkjSGmcJfVbuADT3VImlEfsJPatSoA3te\ncc5TbNu2vKfecpCOWmHPLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/\n1CjDLzXK6brm2fa95T3qcDVzROW47PmlRhl+qVGzTtf1e0nuTHJHkk8nWddXYZKGNXX4k5wC/C6w\noapeCawBtvRVmKRhzXrYvxb4ySRrmczT973ZS5I0hlm+t38v8CfAd4B9wBNVdWNfhUka1iyH/euB\nzUzm7HsZcGySty3Q7sfTdT362A+nr1RSr2Y57P9l4D+r6tGqehb4AvDa+Y3mTtf1khevmWFzkvo0\nS/i/A1yQ5JgkYTJd1939lCVpaLOc8+9gMjnnTuD27rmu7qkuSQObdbqu9wPv76kWSSPyE35Sowy/\n1KhVO6pv2hFiY47qWwk1avWy55caZfilRhl+qVGGX2qU4ZcaZfilRhl+qVGGX2qU4ZcaZfilRhl+\nqVGGX2pUqmq0jW141br6xrbTRtveWFbzNFOrdRDRan3NdtRNPFmPZylt7fmlRhl+qVHPG/4k1yR5\nJMkdc+47Icn2JPd2v9cPW6akvi2l5/8UsGnefVuBm6rq5cBN3W1JK8jzhr+q/hV4fN7dm4Fru+Vr\ngTf3XJekgU17zn9SVe3rlh8CTuqpHkkjmfmCX03eK1z0/UKn65KWp2nD/3CSkwG6348s1tDpuqTl\nadrwXw9c2i1fCnypn3IkjWUpb/V9Gvh34KwkDyZ5B/BB4FeS3Mtkws4PDlumpL497/f2V9VbF1m1\nsedaJI3IT/hJjTL8UqNW7XRd01qto72m5ZRiq5c9v9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8\nUqMMv9Qowy81yvBLjTL8UqMc2KNBTDMgyMFA47Lnlxpl+KVGGX6pUdPO1ffhJN9KsjvJF5McP2yZ\nkvo27Vx924FXVtU5wD3A+3quS9LAppqrr6purKr93c2bgVMHqE3SgPo4578c+MpiK52uS1qeZgp/\nkquA/cB1i7Vxui5peZr6Qz5JLgMuATZ2k3VKWkGmCn+STcCVwC9W1VP9liRpDNPO1ffXwHHA9iS7\nknx84Dol9Wzaufo+OUAtkkbkJ/ykRo06qu+e3ccs+9Fe02zLKb6eyxF6y589v9Qowy81yvBLjTL8\nUqMMv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9SoFTFX32ocCaj+OKpyOvb8UqMMv9So\nqabrmrPuvUkqyYnDlCdpKNNO10WS04ALge/0XJOkEUw1XVfnz5h8fbff2S+tQFOd8yfZDOytqtuW\n0PbH03U9y9PTbE7SAA77rb4kxwB/wOSQ/3lV1dXA1QA/lRM8SpCWiWl6/p8FzgBuS/IAkxl6dyZ5\naZ+FSRrWYff8VXU78NMHbnd/ADZU1X/3WJekgU07XZekFW7a6brmrj+9t2okjcZP+EmNWhEDe6Yx\n5mAPB/b0wwE647Lnlxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZfapTh\nlxqVqvG+Vi/Jo8B/LbL6RGA5fBuQdRzMOg623Ov4map6yVKeYNTwH0qSW6pqg3VYh3WMU4eH/VKj\nDL/UqOUU/quPdAEd6ziYdRxs1dSxbM75JY1rOfX8kkY0aviTbEry7SR7kmxdYP0Lkny2W78jyekD\n1HBakq8luSvJnUnevUCbNyR5Ismu7ucP+65jzrYeSHJ7t51bFlifJH/Z7ZPdSc7reftnzfl37kry\nZJL3zGsz2P5YaAr4JCck2Z7k3u73+kUee2nX5t4klw5Qx4eTfKvb719Mcvwijz3ka9hDHR9IsnfO\n/r94kcceMl/PUVWj/ABrgPuAM4GjgduAs+e1+S3g493yFuCzA9RxMnBet3wccM8CdbwB+PJI++UB\n4MRDrL8Y+AoQ4AJgx8Cv0UNM3iseZX8ArwfOA+6Yc98fA1u75a3AhxZ43AnA/d3v9d3y+p7ruBBY\n2y1/aKE6lvIa9lDHB4DfX8Jrd8h8zf8Zs+c/H9hTVfdX1TPAZ4DN89psBq7tlj8HbEySPouoqn1V\ntbNb/j5wN3BKn9vo2Wbg72riZuD4JCcPtK2NwH1VtdgHsXpXC08BP/f/wbXAmxd46EXA9qp6vKr+\nB9gObOqzjqq6sar2dzdvZjIv5aAW2R9LsZR8HWTM8J8CfHfO7Qd5buh+3Kbb6U8ALx6qoO604lxg\nxwKrX5PktiRfSfJzQ9UAFHBjkluTXLHA+qXst75sAT69yLqx9gfASVW1r1t+CDhpgTZj7heAy5kc\ngS3k+V7DPryrO/24ZpHToMPeH81e8EvyQuDzwHuq6sl5q3cyOfR9FfBXwD8NWMrrquo84I3Abyd5\n/YDbWlSSo4E3Af+4wOox98dBanJMe0TfkkpyFbAfuG6RJkO/hh9jMjv2q4F9wJ/28aRjhn8vcNqc\n26d29y3YJsla4EXAY30XkuQoJsG/rqq+MH99VT1ZVf/bLd8AHJXkxL7r6J5/b/f7EeCLTA7f5lrK\nfuvDG4GdVfXwAjWOtj86Dx84tel+P7JAm1H2S5LLgEuAX+/+ED3HEl7DmVTVw1X1w6r6EfA3izz/\nYe+PMcP/TeDlSc7oepktwPXz2lwPHLhq+xbgq4vt8Gl11xA+CdxdVR9ZpM1LD1xrSHI+k/00xB+h\nY5Mcd2CZyQWmO+Y1ux74ze6q/wXAE3MOifv0VhY55B9rf8wx9//BpcCXFmizDbgwyfruMPjC7r7e\nJNkEXAm8qaqeWqTNUl7DWeuYe43n1xZ5/qXk62B9XKE8jCuZFzO5un4fcFV33x8x2bkA65gcdu4B\nvgGcOUANr2NyGLkb2NX9XAy8E3hn1+ZdwJ1MrpjeDLx2oP1xZreN27rtHdgnc2sJ8NFun90ObBig\njmOZhPlFc+4bZX8w+YOzD3iWyXnqO5hc57kJuBf4F+CEru0G4BNzHnt5939lD/D2AerYw+Q8+sD/\nkwPvRL0MuOFQr2HPdfx999rvZhLok+fXsVi+DvXjJ/ykRjV7wU9qneGXGmX4pUYZfqlRhl9qlOGX\nGmX4pUYZfqlR/w9B1vcC3mIdhQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bf51fea20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------\n",
      "Actual:   3\n",
      "Guess:    3\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADXZJREFUeJzt3WusZeVdx/HvT65CKZdSKbcIGEqCDQKZUFobbES5SZia\n9MUQq1CakEZRMDWESmIbX7VW67Vpg4CiEmikYEkDDiNtY0xkLIzDdSgMiFw6XAQDtaQF2r8v9prm\nzOGc4czea605Z57vJzk5a+/17L3+Z+39O+uy17OfVBWS2vMTO7sASTuH4ZcaZfilRhl+qVGGX2qU\n4ZcaZfilRhl+qVGGX2rU7mMubM/sVXuz7yjLevcJr46yHK1Mj9y3z84uYRDf53u8Vj/IUtqOGv69\n2Zf35vRRlrV27cZRlqOV6czDTtzZJQxifd255Lbu9kuNmin8Sc5K8u0km5Nc0VdRkoY3dfiT7AZ8\nATgbOB44P8nxfRUmaVizbPlPATZX1eNV9RpwI7C6n7IkDW2W8B8OPDXn9tPdfZJWgMHP9ie5GLgY\nYG92zY9XpJVoli3/M8CRc24f0d23jaq6qqpWVdWqPdhrhsVJ6tMs4f8WcGySo5PsCawBbu2nLElD\nm3q3v6reSHIJsBbYDbi2qh7srTJJg5rpmL+qbgNu66kWSSPyCj+pUYZfatSoHXumtfY7y7uTzq7a\nSWRsY77OY7+nluN7xC2/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxS\no1JVoy3s7TmoxhqxR9qe5d5ZDKbrDLS+7uSVemlJw3W55ZcaZfilRhl+qVGzDNd1ZJJvJHkoyYNJ\nLu2zMEnDmuWbfN4APlFVG5LsB9yTZF1VPdRTbZIGNPWWv6q2VNWGbvq7wCYcrktaMXr5Dr8kRwEn\nAesXmOdwXdIyNPMJvyRvA74CXFZVr8yf73Bd0vI0U/iT7MEk+NdX1c39lCRpDLOc7Q9wDbCpqj7f\nX0mSxjDLlv/ngV8HfjHJxu7nnJ7qkjSwWQbq/DdgSdcQS1p+vMJPapThlxpl+KVGGX6pUYZfapTh\nlxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZfapThlxrVy3f4STvTrjr01tDc8kuNMvxSowy/1Kg+\nvrp7tyT/meRrfRQkaRx9bPkvZTJaj6QVZNbv7T8C+BXg6n7KkTSWWbf8fwZcDvyoh1okjWiWQTvO\nBZ6vqnveot3FSe5Ocvfr/GDaxUnq2ayDdpyX5AngRiaDd/zD/EaO1SctT7MM0f3Jqjqiqo4C1gBf\nr6qP9FaZpEH5Ob/UqF6u7a+qbwLf7OO5JI3DLb/UKHv1aRAroafdNJZj77xpueWXGmX4pUYZfqlR\nhl9qlOGXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGXGmWvvh7sqj3YVopdqafdmNzyS40y\n/FKjZh2044AkNyV5OMmmJO/rqzBJw5r1mP/PgX+uqg8n2RPYp4eaJI1g6vAn2R84DbgQoKpeA17r\npyxJQ5tlt/9o4AXgb7pReq9Osm9PdUka2Czh3x04GfhiVZ0EfA+4Yn4jh+uSlqdZwv808HRVre9u\n38Tkn8E2HK5LWp5mGa7rWeCpJMd1d50OPNRLVZIGN+vZ/t8Gru/O9D8OfHT2kiSNYabwV9VGYFVP\ntUgakVf4SY1aER177Diz89hpZtflll9qlOGXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGX\nGmX4pUYZfqlRhl9q1Iro1aedZ1ftUWlvRbf8UrMMv9SoWYfr+t0kDyZ5IMkNSfbuqzBJw5o6/EkO\nB34HWFVV7wF2A9b0VZikYc2627878JNJdmcyTt93Zi9J0hhm+d7+Z4A/Bp4EtgAvV9UdfRUmaViz\n7PYfCKxmMmbfYcC+ST6yQDuH65KWoVl2+38J+K+qeqGqXgduBt4/v5HDdUnL0yzhfxI4Nck+ScJk\nuK5N/ZQlaWizHPOvZzI45wbg/u65ruqpLkkDm3W4rk8Bn+qpFkkj8go/qVGGX2rUiujVZw+sNozZ\ng3Al9FYc+n3vll9qlOGXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRK6Jj\nj9qwEjpwrYQOQUvlll9qlOGXGvWW4U9ybZLnkzww576DkqxL8mj3+8Bhy5TUt6Vs+f8WOGvefVcA\nd1bVscCd3W1JK8hbhr+q/hV4ad7dq4HruunrgA/1XJekgU17zH9IVW3ppp8FDumpHkkjmfmEX1UV\nUIvNd7guaXmaNvzPJTkUoPv9/GINHa5LWp6mDf+twAXd9AXAV/spR9JYlvJR3w3AvwPHJXk6yceA\nzwC/nORRJgN2fmbYMiX17S0v762q8xeZdXrPtUgakVf4SY0y/FKjMvmkbhxvz0H13ni0oPaM1Rvw\nlDOf4u57v5+ltHXLLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/1CiH\n65J2gMN1SVrxDL/UKMMvNWrasfo+l+ThJPcluSXJAcOWKalv047Vtw54T1WdADwCfLLnuiQNbKqx\n+qrqjqp6o7t5F3DEALVJGlAfx/wXAbcvNtPhuqTlaabwJ7kSeAO4frE2DtclLU9TX+ST5ELgXOD0\nGvMrgCX1YqrwJzkLuBz4hap6td+SJI1h2rH6/grYD1iXZGOSLw1cp6SeTTtW3zUD1CJpRF7hJzXK\nXn1q0kronXfmYSfu8GMeqReX3NYtv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81\nyvBLjTL8UqMMv9SoUXv1vfuEV1m7dvn3ppL6Nk0PvaG55ZcaZfilRk01XNeceZ9IUkkOHqY8SUOZ\ndrgukhwJnAE82XNNkkYw1XBdnT9l8vXdfme/tAJNdcyfZDXwTFXdu4S2Px6u64UXfzjN4iQNYIfD\nn2Qf4PeBP1hK+7nDdb3zHbvt6OIkDWSaLf/PAEcD9yZ5gskIvRuSvKvPwiQNa4cv8qmq+4Gf2nq7\n+wewqqr+p8e6JA1s2uG6JK1w0w7XNXf+Ub1VI2k0XuEnNcrhujSI5diRRdtyyy81yvBLjTL8UqMM\nv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81KlXjfflukheA/15k9sHAcvg2IOvY\nlnVsa7nX8dNV9c6lPMGo4d+eJHdX1SrrsA7rGKcOd/ulRhl+qVHLKfxX7ewCOtaxLevY1i5Tx7I5\n5pc0ruW05Zc0olHDn+SsJN9OsjnJFQvM3yvJl7v565McNUANRyb5RpKHkjyY5NIF2nwwyctJNnY/\nSxqabMp6nkhyf7ecuxeYnyR/0a2T+5Kc3PPyj5vzd25M8kqSy+a1GWx9LDQEfJKDkqxL8mj3+8BF\nHntB1+bRJBcMUMfnkjzcrfdbkhywyGO3+xr2UMenkzwzZ/2fs8hjt5uvN6mqUX6A3YDHgGOAPYF7\ngePntflN4Evd9BrgywPUcShwcje9H/DIAnV8EPjaSOvlCeDg7cw/B7gdCHAqsH7g1+hZJp8Vj7I+\ngNOAk4EH5tz3R8AV3fQVwGcXeNxBwOPd7wO76QN7ruMMYPdu+rML1bGU17CHOj4N/N4SXrvt5mv+\nz5hb/lOAzVX1eFW9BtwIrJ7XZjVwXTd9E3B6kvRZRFVtqaoN3fR3gU3A4X0uo2ergb+ribuAA5Ic\nOtCyTgceq6rFLsTqXS08BPzc98F1wIcWeOiZwLqqeqmq/hdYB5zVZx1VdUdVvdHdvIvJuJSDWmR9\nLMVS8rWNMcN/OPDUnNtP8+bQ/bhNt9JfBt4xVEHdYcVJwPoFZr8vyb1Jbk/ys0PVABRwR5J7kly8\nwPylrLe+rAFuWGTeWOsD4JCq2tJNPwscskCbMdcLwEVM9sAW8lavYR8u6Q4/rl3kMGiH10ezJ/yS\nvA34CnBZVb0yb/YGJru+Pwf8JfBPA5bygao6GTgb+K0kpw24rEUl2RM4D/jHBWaPuT62UZN92p36\nkVSSK4E3gOsXaTL0a/hFJqNjnwhsAf6kjycdM/zPAEfOuX1Ed9+CbZLsDuwPvNh3IUn2YBL866vq\n5vnzq+qVqvq/bvo2YI8kB/ddR/f8z3S/nwduYbL7NtdS1lsfzgY2VNVzC9Q42vroPLf10Kb7/fwC\nbUZZL0kuBM4Ffq37R/QmS3gNZ1JVz1XVD6vqR8BfL/L8O7w+xgz/t4BjkxzdbWXWALfOa3MrsPWs\n7YeBry+2wqfVnUO4BthUVZ9fpM27tp5rSHIKk/U0xD+hfZPst3WayQmmB+Y1uxX4je6s/6nAy3N2\nift0Povs8o+1PuaY+z64APjqAm3WAmckObDbDT6ju683Sc4CLgfOq6pXF2mzlNdw1jrmnuP51UWe\nfyn52lYfZyh34EzmOUzOrj8GXNnd94dMVi7A3kx2OzcD/wEcM0ANH2CyG3kfsLH7OQf4OPDxrs0l\nwINMzpjeBbx/oPVxTLeMe7vlbV0nc2sJ8IVund0PrBqgjn2ZhHn/OfeNsj6Y/MPZArzO5Dj1Y0zO\n89wJPAr8C3BQ13YVcPWcx17UvVc2Ax8doI7NTI6jt75Ptn4SdRhw2/Zew57r+Pvutb+PSaAPnV/H\nYvna3o9X+EmNavaEn9Q6wy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqP+H6NB8yovjH6XAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bf50dfc88>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------\n",
      "Actual:   2\n",
      "Guess:    2\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADYJJREFUeJzt3X2sZPVdx/H3RxZYochDqZSHjYChJNjQQjaU1gYbV+mC\nhK1J/4BYhdKENIqCqSFUEtv4V2u1PjZtEFBUAo0ULGnAZaVtjIlsC+uyPCyFBRFYlgeLYWtJeWi/\n/jFn693Lvbt3Z86ZvXd/71dyc2fm/GbOd8/s554zZ+Y331QVktrzE3u7AEl7h+GXGmX4pUYZfqlR\nhl9qlOGXGmX4pUYZfqlRhl9q1LJpruyAHFjLOXiP7/eOU18ZoBotNo9uOmhvl7Dk/YDv81q9moWM\nnWr4l3Mw78mqPb7f2rUbB6hGi80Hj3n33i5hyVtfdy94rIf9UqMmCn+S1Um+k2RLkqv6KkrS8MYO\nf5L9gC8A5wCnABcmOaWvwiQNa5I9/xnAlqp6oqpeA24G1vRTlqShTRL+Y4GnZ1x/prtN0hIw+Nn+\nJJcClwIsx7dypMVikj3/VmDFjOvHdbftpKquqaqVVbVyfw6cYHWS+jRJ+L8NnJTkhCQHABcAt/dT\nlqShjX3YX1VvJLkMWAvsB1xfVQ/1VpmkQU30mr+q7gDu6KkWSVPkJ/ykRhl+qVGZ5vf2r3zX8vrW\n2hW7H9gDJ4nsbO2z++7kKJ/r/7e+7mZ7vbSgWX3u+aVGGX6pUYZfapThlxpl+KVGGX6pUYZfapTh\nlxpl+KVGGX6pUYZfapThlxo11Y49j246yEkYe8m4231fnhDUOvf8UqMMv9Qowy81apJ2XSuSfCPJ\nw0keSnJ5n4VJGtYkJ/zeAD5RVRuSHALcl2RdVT3cU22SBjT2nr+qtlXVhu7y94DN2K5LWjJ6easv\nyfHAacD6OZbZrktahCY+4ZfkLcBXgCuqavvs5bbrkhanicKfZH9Gwb+xqm7tpyRJ0zDJ2f4A1wGb\nq+rz/ZUkaRom2fP/PPDrwC8m2dj9nNtTXZIGNkmjzn8DFtQcQNLi4yf8pEZNdVafNIRxZh46u9Q9\nv9Qswy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqOc2KMlz0k643HPLzXK\n8EuNMvxSo/r46u79kvxHkq/1UZCk6ehjz385o249kpaQSb+3/zjgV4Br+ylH0rRMuuf/M+BK4Ec9\n1CJpiiZp2nEe8EJV3bebcZcmuTfJva/z6rirk9SzSZt2nJ/kSeBmRs07/mH2IHv1SYvTJC26P1lV\nx1XV8cAFwNer6iO9VSZpUL7PLzWql8/2V9U3gW/28ViSpsM9v9QoZ/U1YpyWVtq3ueeXGmX4pUYZ\nfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGXGuWsPi0a9tybLvf8UqMMv9So\nSZt2HJbkliSPJNmc5L19FSZpWJO+5v9z4J+r6sNJDgAO6qEmSVMwdviTHAqcBVwMUFWvAa/1U5ak\noU1y2H8C8CLwN12X3muTHNxTXZIGNkn4lwGnA1+sqtOA7wNXzR5kuy5pcZok/M8Az1TV+u76LYz+\nGOzEdl3S4jRJu67ngKeTnNzdtAp4uJeqJA1u0rP9vw3c2J3pfwL46OQlSZqGicJfVRuBlT3VImmK\n/ISf1Cgn9uxF+3ILLSfpLH7u+aVGGX6pUYZfapThlxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZf\napThlxpl+KVGOatvln15pt04nJ2373LPLzXK8EuNmrRd1+8meSjJg0luSrK8r8IkDWvs8Cc5Fvgd\nYGVVvRPYD7igr8IkDWvSw/5lwE8mWcaoT9+zk5ckaRom+d7+rcAfA08B24CXq+quvgqTNKxJDvsP\nB9Yw6tl3DHBwko/MMc52XdIiNMlh/y8B/1lVL1bV68CtwPtmD7Jdl7Q4TRL+p4AzkxyUJIzadW3u\npyxJQ5vkNf96Rs05NwAPdI91TU91SRrYpO26PgV8qqdaJE2Rn/CTGmX4pUYtiVl9++pMO2fMaW9y\nzy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9SoqU7secepr7B27b43\nSccJOlqK3PNLjTL8UqN2G/4k1yd5IcmDM247Ism6JI91vw8ftkxJfVvInv9vgdWzbrsKuLuqTgLu\n7q5LWkJ2G/6q+lfgpVk3rwFu6C7fAHyo57okDWzc1/xHVdW27vJzwFE91SNpSiY+4VdVBdR8y2e2\n63rxuz+cdHWSejJu+J9PcjRA9/uF+QbObNf1trfuN+bqJPVt3PDfDlzUXb4I+Go/5UialoW81XcT\n8O/AyUmeSfIx4DPALyd5jFHDzs8MW6akvu32471VdeE8i1b1XIukKfITflKjDL/UqCXRrmsczrST\nds09v9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqP22Yk9a58dry2Y\nE4LUCvf8UqMMv9Qowy81atxefZ9L8kiSTUluS3LYsGVK6tu4vfrWAe+sqlOBR4FP9lyXpIGN1auv\nqu6qqje6q/cAxw1Qm6QB9fGa/xLgzvkW2q5LWpwmCn+Sq4E3gBvnG2O7LmlxGvtDPkkuBs4DVnXN\nOiUtIWOFP8lq4ErgF6rqlX5LkjQN4/bq+yvgEGBdko1JvjRwnZJ6Nm6vvusGqEXSFPkJP6lRmea5\nup/KEfWe7Hlz33Fn6GnvcXbk3rG+7mZ7vZSFjHXPLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/\n1CjDLzXK8EuNMvxSowy/1CjDLzVqSczqmyZnEGopO+ODT3Pv/T9wVp+k+Rl+qVFjteuasewTSSrJ\nkcOUJ2ko47brIskK4GzgqZ5rkjQFY7Xr6vwpo6/v9jv7pSVorNf8SdYAW6vq/gWM/XG7rtd5dZzV\nSRrAHjftSHIQ8PuMDvl3q6quAa6B0Vt9e7o+ScMYZ8//s8AJwP1JnmTUoXdDkrf3WZikYe3xnr+q\nHgB+esf17g/Ayqr67x7rkjSwcdt1SVrixm3XNXP58b1VI2lq/ISf1KixWnTvy2wztTMnOu273PNL\njTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjZpqu64kLwL/Nc/i\nI4HF8G1A1rEz69jZYq/jZ6rqbQt5gKmGf1eS3FtVK63DOqxjOnV42C81yvBLjVpM4b9mbxfQsY6d\nWcfO9pk6Fs1rfknTtZj2/JKmaKrhT7I6yXeSbEly1RzLD0zy5W75+iTHD1DDiiTfSPJwkoeSXD7H\nmA8keTnJxu7nD/quY8a6nkzyQLeee+dYniR/0W2TTUlO73n9J8/4d25Msj3JFbPGDLY95moBn+SI\nJOuSPNb9Pnye+17UjXksyUUD1PG5JI902/22JIfNc99dPoc91PHpJFtnbP9z57nvLvP1JlU1lR9g\nP+Bx4ETgAOB+4JRZY34T+FJ3+QLgywPUcTRwenf5EODROer4APC1KW2XJ4Ejd7H8XOBOIMCZwPqB\nn6PnGL1XPJXtAZwFnA48OOO2PwKu6i5fBXx2jvsdATzR/T68u3x4z3WcDSzrLn92rjoW8hz2UMen\ngd9bwHO3y3zN/pnmnv8MYEtVPVFVrwE3A2tmjVkD3NBdvgVYlSR9FlFV26pqQ3f5e8Bm4Ng+19Gz\nNcDf1cg9wGFJjh5oXauAx6tqvg9i9a7mbgE/8//BDcCH5rjrB4F1VfVSVf0PsA5Y3WcdVXVXVb3R\nXb2HUV/KQc2zPRZiIfnayTTDfyzw9Izrz/Dm0P14TLfRXwbeOlRB3cuK04D1cyx+b5L7k9yZ5OeG\nqgEo4K4k9yW5dI7lC9lufbkAuGmeZdPaHgBHVdW27vJzwFFzjJnmdgG4hNER2Fx29xz24bLu5cf1\n87wM2uPt0ewJvyRvAb4CXFFV22ct3sDo0PddwF8C/zRgKe+vqtOBc4DfSnLWgOuaV5IDgPOBf5xj\n8TS3x05qdEy7V9+SSnI18AZw4zxDhn4Ov8ioO/a7gW3An/TxoNMM/1ZgxYzrx3W3zTkmyTLgUOC7\nfReSZH9Gwb+xqm6dvbyqtlfV/3aX7wD2T3Jk33V0j7+1+/0CcBujw7eZFrLd+nAOsKGqnp+jxqlt\nj87zO17adL9fmGPMVLZLkouB84Bf6/4QvckCnsOJVNXzVfXDqvoR8NfzPP4eb49phv/bwElJTuj2\nMhcAt88aczuw46zth4Gvz7fBx9WdQ7gO2FxVn59nzNt3nGtIcgaj7TTEH6GDkxyy4zKjE0wPzhp2\nO/Ab3Vn/M4GXZxwS9+lC5jnkn9b2mGHm/4OLgK/OMWYtcHaSw7vD4LO723qTZDVwJXB+Vb0yz5iF\nPIeT1jHzHM+vzvP4C8nXzvo4Q7kHZzLPZXR2/XHg6u62P2S0cQGWMzrs3AJ8CzhxgBrez+gwchOw\nsfs5F/g48PFuzGXAQ4zOmN4DvG+g7XFit477u/Xt2CYzawnwhW6bPQCsHKCOgxmF+dAZt01lezD6\ng7MNeJ3R69SPMTrPczfwGPAvwBHd2JXAtTPue0n3f2UL8NEB6tjC6HX0jv8nO96JOga4Y1fPYc91\n/H333G9iFOijZ9cxX7529eMn/KRGNXvCT2qd4ZcaZfilRhl+qVGGX2qU4ZcaZfilRhl+qVH/Byj8\n8HAS4gblAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bf507f6a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------\n",
      "Actual:   5\n",
      "Guess:    5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADV9JREFUeJzt3WusZfVZx/HvT24jlHIplXKLgEESbFogE0prg42j3CRM\nTfpiiFUoTUijKJgaQiWxja9aq/XatEFAUQk0UrCkAYeRtjEmMi2Mw53CgMhtuBQMVEnk0scXe01z\n5nDOzGHvtdacw//7SU722nv9z17PrD2/sy57r/2kqpDUnp/Y1QVI2jUMv9Qowy81yvBLjTL8UqMM\nv9Qowy81yvBLjTL8UqN2H3Nhe2avWsU+Yy5SnZ993yu7ugSN4LEnXuMHL76RpYwdNfyr2IcPZM2Y\ni1Rn/frNu7oEjeCk055Y8lh3+6VGzRT+JKcn+X6SLUku7asoScObOvxJdgO+DJwBHAeck+S4vgqT\nNKxZtvwnAVuq6tGqehW4DljbT1mShjZL+A8D5p5deLJ7TNIKMPjZ/iQXABcArGLvoRcnaYlm2fI/\nBRwx5/7h3WPbqarLq2p1Va3eg71mWJykPs0S/u8BxyQ5KsmewDrgpn7KkjS0qXf7q+r1JBcC64Hd\ngKuq6r7eKpM0qJmO+avqZuDmnmqRNCI/4Sc1yvBLjRr1wh5tb/3TXmzTh9MOPX5Xl7BsPFQvLHms\nW36pUYZfapThlxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZfapThlxpl+KVGpapGW9jq96+q764/\nYucDe+DFHmrRxrqNl+vFJbXrcssvNcrwS40y/FKjZmnXdUSSbye5P8l9SS7qszBJw5rlm3xeBz5d\nVZuS7AvcmWRDVd3fU22SBjT1lr+qtlbVpm76h8AD2K5LWjF6OeZPciRwArBxgXkXJLkjyR3Pv/BG\nH4uT1IOZw5/kHcDXgYur6uX58+e263r3u3abdXGSejJT+JPswST411TVDf2UJGkMs5ztD3Al8EBV\nfam/kiSNYZYt/88Dvw78YpLN3c+ZPdUlaWCzNOr8N2BJnyGWtPz4CT+pUYZfapThlxpl+KVGGX6p\nUYZfapThlxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZfalQfX929W5L/\nSPLNPgqSNI4+tvwXMenWI2kFmfV7+w8HfgW4op9yJI1l1i3/nwGXAD/qoRZJI5qlacdZwHNVdedO\nxtmrT1qGZm3acXaSx4DrmDTv+If5g+zVJy1Ps7To/kxVHV5VRwLrgG9V1cd7q0zSoHyfX2rU1O26\n5qqq7wDf6eO5JI3DLb/UKMMvNcrwS40y/FKjDL/UKMMvNcrwS40y/FKjDL/UKMMvNcrwS40y/FKj\nDL/UKMMvNcrwS43q5Xr+5Wj905t3dQmDOe3Q43d1CXobcMsvNcrwS42atWnH/kmuT/JgkgeSfLCv\nwiQNa9Zj/j8H/rmqPpZkT2DvHmqSNIKpw59kP+AU4DyAqnoVeLWfsiQNbZbd/qOA54G/6br0XpFk\nn57qkjSwWcK/O3Ai8JWqOgH4X+DS+YNs1yUtT7OE/0ngyara2N2/nskfg+3YrktanmZp1/UM8ESS\nY7uH1gD391KVpMHNerb/t4FrujP9jwKfmL0kSWOYKfxVtRlY3VMtkkbkJ/ykRqWqRlvYO3NgfSBr\nRlveWFbCRUReDNSGjXUbL9eLWcpYt/xSowy/1CjDLzXK8EuNMvxSowy/1CjDLzXK8EuNMvxSowy/\n1CjDLzXK8EuNMvxSo9627brGNO0Vc2NeDTjtsrwa8O3LLb/UKMMvNWrWdl2/m+S+JPcmuTbJqr4K\nkzSsqcOf5DDgd4DVVfVeYDdgXV+FSRrWrLv9uwM/mWR3Jn36np69JEljmOV7+58C/hh4HNgKvFRV\nt/ZVmKRhzbLbfwCwlknPvkOBfZJ8fIFxP27X9Rr/N32lkno1y27/LwH/WVXPV9VrwA3Ah+YPmtuu\naw/2mmFxkvo0S/gfB05OsneSMGnX9UA/ZUka2izH/BuZNOfcBNzTPdflPdUlaWCztuv6LPDZnmqR\nNCI/4Sc1yvBLjfKqvl3IqwG1K7nllxpl+KVGGX6pUYZfapThlxpl+KVGGX6pUYZfapThlxpl+KVG\nGX6pUYZfapQX9qxA01w0M+bFQFoZ3PJLjTL8UqN2Gv4kVyV5Lsm9cx47MMmGJA93twcMW6akvi1l\ny/+3wOnzHrsUuK2qjgFu6+5LWkF2Gv6q+lfgxXkPrwWu7qavBj7ac12SBjbtMf/BVbW1m34GOLin\neiSNZOYTflVVQC0233Zd0vI0bfifTXIIQHf73GIDbdclLU/Thv8m4Nxu+lzgG/2UI2ksS3mr71rg\n34FjkzyZ5JPA54FfTvIwk4adnx+2TEl92+nHe6vqnEVmrem5Fkkj8hN+UqMMv9Qor+pbgbxCT31w\nyy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMMv9Qowy81yvBLjTL8UqMM\nv9SoaXv1fTHJg0nuTnJjkv2HLVNS36bt1bcBeG9VvQ94CPhMz3VJGthUvfqq6taqer27eztw+AC1\nSRpQH8f85wO3LDbTdl3S8jRT+JNcBrwOXLPYGNt1ScvT1N/em+Q84CxgTdesU9IKMlX4k5wOXAL8\nQlW90m9JksYwba++vwL2BTYk2ZzkqwPXKaln0/bqu3KAWiSNyE/4SY1aEe26xmxPddqhx4+2rJXQ\ndmvM9aFxueWXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGXGmX4pUYZfqlRhl9qlOGXGrUi\nruob00q40m4aXp2n+dzyS40y/FKjpmrXNWfep5NUkoOGKU/SUKZt10WSI4BTgcd7rknSCKZq19X5\nUyZf3+139ksr0FTH/EnWAk9V1V1LGGu7LmkZestv9SXZG/h9Jrv8O1VVlwOXA7wzB7qXIC0T02z5\nfwY4CrgryWNMOvRuSvKePguTNKy3vOWvqnuAn9p2v/sDsLqqftBjXZIGNm27Lkkr3LTtuubOP7K3\naiSNxk/4SY1aERf22EJre16koz645ZcaZfilRhl+qVGGX2qU4ZcaZfilRhl+qVGGX2qU4ZcaZfil\nRhl+qVGGX2qU4ZcalarxvlYvyfPAfy0y+yBgOXwbkHVszzq2t9zr+OmqevdSnmDU8O9IkjuqarV1\nWId1jFOHu/1Sowy/1KjlFP7Ld3UBHevYnnVs721Tx7I55pc0ruW05Zc0olHDn+T0JN9PsiXJpQvM\n3yvJ17r5G5McOUANRyT5dpL7k9yX5KIFxnwkyUtJNnc/f9B3HXOW9ViSe7rl3LHA/CT5i26d3J3k\nxJ6Xf+ycf+fmJC8nuXjemMHWx0It4JMcmGRDkoe72wMW+d1zuzEPJzl3gDq+mOTBbr3fmGT/RX53\nh69hD3V8LslTc9b/mYv87g7z9SZVNcoPsBvwCHA0sCdwF3DcvDG/CXy1m14HfG2AOg4BTuym9wUe\nWqCOjwDfHGm9PAYctIP5ZwK3AAFOBjYO/Bo9w+S94lHWB3AKcCJw75zH/gi4tJu+FPjCAr93IPBo\nd3tAN31Az3WcCuzeTX9hoTqW8hr2UMfngN9bwmu3w3zN/xlzy38SsKWqHq2qV4HrgLXzxqwFru6m\nrwfWJEmfRVTV1qra1E3/EHgAOKzPZfRsLfB3NXE7sH+SQwZa1hrgkapa7INYvauFW8DP/X9wNfDR\nBX71NGBDVb1YVf8NbABO77OOqrq1ql7v7t7OpC/loBZZH0uxlHxtZ8zwHwY8Mef+k7w5dD8e0630\nl4B3DVVQd1hxArBxgdkfTHJXkluS/NxQNQAF3JrkziQXLDB/KeutL+uAaxeZN9b6ADi4qrZ2088A\nBy8wZsz1AnA+kz2whezsNezDhd3hx1WLHAa95fXR7Am/JO8Avg5cXFUvz5u9icmu7/uBvwT+acBS\nPlxVJwJnAL+V5JQBl7WoJHsCZwP/uMDsMdfHdmqyT7tL35JKchnwOnDNIkOGfg2/wqQ79vHAVuBP\n+njSMcP/FHDEnPuHd48tOCbJ7sB+wAt9F5JkDybBv6aqbpg/v6perqr/6aZvBvZIclDfdXTP/1R3\n+xxwI5Pdt7mWst76cAawqaqeXaDG0dZH59lthzbd7XMLjBllvSQ5DzgL+LXuD9GbLOE1nElVPVtV\nb1TVj4C/XuT53/L6GDP83wOOSXJUt5VZB9w0b8xNwLazth8DvrXYCp9Wdw7hSuCBqvrSImPes+1c\nQ5KTmKynIf4I7ZNk323TTE4w3Ttv2E3Ab3Rn/U8GXpqzS9ync1hkl3+s9THH3P8H5wLfWGDMeuDU\nJAd0u8Gndo/1JsnpwCXA2VX1yiJjlvIazlrH3HM8v7rI8y8lX9vr4wzlWziTeSaTs+uPAJd1j/0h\nk5ULsIrJbucW4LvA0QPU8GEmu5F3A5u7nzOBTwGf6sZcCNzH5Izp7cCHBlofR3fLuKtb3rZ1MreW\nAF/u1tk9wOoB6tiHSZj3m/PYKOuDyR+crcBrTI5TP8nkPM9twMPAvwAHdmNXA1fM+d3zu/8rW4BP\nDFDHFibH0dv+n2x7J+pQ4OYdvYY91/H33Wt/N5NAHzK/jsXytaMfP+EnNarZE35S6wy/1CjDLzXK\n8EuNMvxSowy/1CjDLzXK8EuN+n87busQ37MqDwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bf4feff98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------\n",
      "Right: 4\n",
      "Wrong: 1\n",
      "Accuracy: 80%\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "m = XTest.shape[0]\n",
    "\n",
    "right = 0\n",
    "wrong = 0\n",
    "\n",
    "for t in range(5):\n",
    "    \n",
    "    (activations,weighted_outputs,outputs) = predict(np.array([ XTest[t] ]),thetas)\n",
    "    \n",
    "    yDigit   = ansToDigit(YTest[t])\n",
    "    ansDigit = ansToDigit(outputs[0])\n",
    "    print(\"Actual:  \",yDigit)\n",
    "    print(\"Guess:   \", ansDigit)\n",
    "\n",
    "    np.array([ XTest[t] ])\n",
    "    \n",
    "    plt.imshow(np.array([ XTest[t] ]).reshape(16,16))\n",
    "    plt.show()\n",
    "    print(\"-------------------------------\")\n",
    "    \n",
    "    if yDigit == ansDigit: right +=1\n",
    "    else: wrong +=1\n",
    "\n",
    "print(\"Right:\",right)\n",
    "print(\"Wrong:\",wrong)\n",
    "print(\"Accuracy: %i%%\" % (right*100/(right+wrong)) )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "Deep learning code need not be overly long.  Here we have shown an example which is small, yet makes reasonable predictions on hand written digits.\n",
    "\n",
    "The algorithms used are not specifically for handwriting recognition, so this same code could easily be used for other Deep Learning applications.\n",
    "\n",
    "For real world applications it may be prudent to use existing libraries which;\n",
    "\n",
    "* are be better optimized for the given hardware\n",
    "* have been more thoroughly tested\n",
    "* contain more complex algorithms for faster gradient decent\n",
    "\n",
    "## The Code\n",
    "\n",
    "The complete code and description can be downloaded in the form of a [Jupyter notebook](https://nbviewer.jupyter.org/github/douglas-gibbons/douglas-gibbons.github.io/tree/master/notebooks/Deep%20Learning%20from%20Scratch%20With%20Python.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}