{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Linear Regression\n",
    "To get familiar with automatic differentiation, we start by learning a simple linear regression model using Stochastic Gradient Descent (SGD).\n",
    "\n",
    "Recall that given a dataset $\\{(x_i, y_i)\\}_{i=0}^N$, with $x_i, y_i \\in \\mathbb{R}$, the objective of linear regression is to find two scalars $w$ and $b$ such that $y = w\\cdot x + b$ fits the dataset. In this tutorial we will learn $w$ and $b$ using SGD and a Mean Square Error (MSE) loss:\n",
    "\n",
    "$$\\mathcal{l} = \\frac{1}{N} \\sum_{i=0}^N (w\\cdot x_i + b - y_i)^2$$\n",
    "\n",
    "Starting from random values, parameters $w$ and $b$ will be updated at each iteration via the following rule:\n",
    "\n",
    "$$w_t = w_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial w}$$\n",
    "<br>\n",
    "$$b_t = b_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial b}$$\n",
    "\n",
    "where $\\eta$ is the learning rate.\n",
    "\n",
    "**NOTE:** Recall that **linear regression** is indeed a **simple neuron** with a linear activation function!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Placeholders and variables\n",
    "To implement and run this simple model, we will use the [Keras backend module](http://keras.io/backend/), which provides an abstraction over Theano and Tensorflow, two popular tensor manipulation libraries that provide automatic differentiation.\n",
    "\n",
    "First of all, we define the necessary variables and placeholders for our computational graph. Variables maintain state across executions of the computational graph, while placeholders are ways to feed the graph with external data.\n",
    "\n",
    "For the linear regression example, we need three variables: `w`, `b`, and the learning rate for SGD, `lr`. Two placeholders `x` and `target` are created to store $x_i$ and $y_i$ values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras.backend as K\n",
    "import numpy as np\n",
    "\n",
    "# Placeholders and variables\n",
    "x = K.placeholder()\n",
    "target = K.placeholder()\n",
    "lr = K.variable(0.1)\n",
    "w = K.variable(np.random.rand())\n",
    "b = K.variable(np.random.rand())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Model definition\n",
    "Now we can define the $y = w\\cdot x + b$ relation as well as the MSE loss in the computational graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Define model and loss\n",
    "y = w * x + b\n",
    "loss = K.mean(K.square(y-target))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Then, given the gradient of MSE wrt to `w` and `b`, we can define how we update the parameters via SGD:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "grads = K.gradients(loss, [w,b])\n",
    "updates = [(w, w-lr*grads[0]), (b, b-lr*grads[1])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The whole model can be encapsulated in a `function`, which takes as input `x` and `target`, returns the current loss value and updates its parameter according to `updates`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Training\n",
    "Training is now just a matter of calling the `function` we have just defined. Each time `train` is called, indeed, `w` and `b` will be updated using the SGD rule.\n",
    "\n",
    "Having generated some random training data, we will feed the `train` function for several epochs and observe the values of `w`, `b`, and loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 0.045, w, b: [0.48, 0.38]\n",
      "Loss: 0.010, w, b: [0.64, 0.41]\n",
      "Loss: 0.006, w, b: [0.72, 0.37]\n",
      "Loss: 0.003, w, b: [0.78, 0.34]\n",
      "Loss: 0.002, w, b: [0.82, 0.31]\n",
      "Loss: 0.001, w, b: [0.86, 0.30]\n",
      "Loss: 0.001, w, b: [0.88, 0.28]\n",
      "Loss: 0.000, w, b: [0.90, 0.27]\n",
      "Loss: 0.000, w, b: [0.92, 0.26]\n",
      "Loss: 0.000, w, b: [0.93, 0.26]\n"
     ]
    }
   ],
   "source": [
    "# Generate data\n",
    "np_x = np.random.rand(1000)\n",
    "np_target = 0.96*np_x + 0.24\n",
    "\n",
    "# Training\n",
    "loss_history = []\n",
    "for epoch in range(200):\n",
    "    current_loss = train([np_x, np_target])[0]\n",
    "    loss_history.append(current_loss)\n",
    "    if epoch % 20 == 0:\n",
    "        print(\"Loss: %.03f, w, b: [%.02f, %.02f]\" % (current_loss, K.eval(w), K.eval(b)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can also plot the loss history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x115b81fd0>]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEACAYAAABYq7oeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+4lHWd//HnC/CgIKaEZKGgCWJqpVJo1tYYrmC1e7RS\noa2lr1Rb/mi/a+2Cu7Yedq/d0uuqtK9LfSv6hmQXV7H5i/xBhrPlZnJKgjSQ0w9YEFASTQQEPLy/\nf3zuA+PMHGbO4ZyZM+e8Htd1X3PPfX/mM58Zhnmdz+ee+/4oIjAzMys0qN4NMDOzvsfhYGZmJRwO\nZmZWwuFgZmYlHA5mZlbC4WBmZiWqCgdJ0yStkbRW0uwy+5skLZLUJukRSWOL9o+VtF3StQXb1kla\nKWmFpOWH/lLMzKynVAwHSYOAW4GpwOnADEmnFhWbBWyLiAnAzcBNRfu/BNxbtG0fkIuIsyJicnca\nb2ZmvaOansNkoC0i1kfEXmAR0FxUphlYkK0vBqZ07JDUDPwOeKLoMary+c3MrMaq+XIeA2wouL8x\n21a2TES0A89LGilpGPAPwFxSGBQK4AFJrZI+3p3Gm5lZ7xhSRZniL3VIX+wHK6OszFzgyxGxU1Jx\nufMiYoukY4EfSVodEQ9X2W4zM+tF1YTDRqDwAPPxwKaiMhuAE4BNkgYDR0XEc5LOAT4g6SbgGKBd\n0q6ImBcRWwAiYqukO0jDVyXhIMkXfzIz64aIKPfHfVWqGVZqBcZLGiepCZgO3F1U5h5gZrZ+KbAs\na9g7I+L1EfF60oHqf4+IeZKGSToSQNJw4ELg8c4aEBFeemC54YYb6t6G/rT4/fT72ZeXQ1Wx5xAR\n7ZKuBpaSwmR+RKyWNBdojYglwHxgoaQ24FlSgBzMa4A7sl7BEOD2iFh6KC/EzMx6TjXDSkTE/cDE\nom03FKzvBi6rUMfcgvU/AGd2qaVmZlYz/inpAJLL5erdhH7F72fP8vvZt6gnxqZ6k6To6200M+tr\nJBG9fEDazMwGGIeDmZmVcDiYmVkJh4OZmZVwOJiZWQmHg5mZlXA4mJlZCYeDmZmVcDiYmVkJh4OZ\nmZVwOJiZWQmHg5mZlXA4mJlZCYeDmZmVqCocJE2TtEbSWkmzy+xvkrRIUpukRySNLdo/VtJ2SddW\nW6eZmdVPxXCQNAi4FZgKnA7MkHRqUbFZwLaImECaK/qmov1fAu7tYp1mZlYn1fQcJgNtEbE+IvYC\ni4DmojLNwIJsfTEwpWOHpGbgd8ATXazTzMzqpJpwGANsKLi/MdtWtkxEtAPPSxopaRjwD8BcQOXK\nH6ROMzOrkyFVlCk3zVzxvJ3FZZSVmQt8OSJ2SjpY+XJ17tfS0rJ/PZfLea5ZM7Mi+XyefD7fY/VV\nnENa0rlAS0RMy+7PASIibiwoc19W5lFJg4HNETFa0k+A47NixwDtwD8Dj1Wqs6BuzyFtZtZFhzqH\ndDU9h1ZgvKRxwGZgOjCjqMw9wEzgUeBSYBlARLyzoKE3ANsjYl4WIJXqNDOzOqkYDhHRLulqYCnp\nGMX8iFgtaS7QGhFLgPnAQkltwLOkL/su13mIr8XMzHpIxWGlevOwkplZ1x3qsJLPkDYzsxIOBzMz\nK9EQ4dDeXu8WmJkNLA0RDrt317sFZmYDi8PBzMxKOBzMzKyEw8HMzEo4HMzMrITDwczMSjgczMys\nhMPBzMxKOBzMzKxEQ4TDnj31boGZ2cDSEOHgnoOZWW05HMzMrITDwczMSlQVDpKmSVojaa2k2WX2\nN0laJKlN0iOSxmbb3yppRcFyccFj1klamW1ffrDndziYmdVWxWlCJQ0CbgWmAJuAVkl3RcSagmKz\ngG0RMUHS5cBNpKlCfw1Mioh9ko4DVkq6OyL2AfuAXEQ8V6kNDgczs9qqpucwGWiLiPURsRdYBDQX\nlWkGFmTri0lBQkS8lAUBwBGkQOigKp/f4WBmVmPVfDmPATYU3N+YbStbJiLagecljQSQNFnS48BK\n4JMFYRHAA5JaJX38YA1wOJiZ1VbFYSXSX/jFokIZdZSJiOXAGZImArdJui8i9gDnRcQWSccCP5K0\nOiIeLteABx5o2R8QuVyOXC5XRbPNzAaOfD5PPp/vsfoUUfw9X1RAOhdoiYhp2f05QETEjQVl7svK\nPCppMLA5IkaXqWsZ8NmIeKxo+w3A9oj4UpnHxJw5wec/341XZ2Y2QEkiIsr9cV+VaoaVWoHxksZJ\naiIdaL67qMw9wMxs/VJgWda4E7OwQNI44BRgnaRhko7Mtg8HLgQe76wBHlYyM6utisNKEdEu6Wpg\nKSlM5kfEaklzgdaIWALMBxZKagOeJQUIwDuAOZL2kA5Gfyoitkk6CbhDUmRtuD0ilnbWBoeDmVlt\nVRxWqjdJMWtW8M1v1rslZmaNoxbDSnXnnoOZWW05HMzMrITDwczMSjREOHg+BzOz2mqIcHDPwcys\nthwOZmZWwuFgZmYlHA5mZlbC4WBmZiUaIhx27ap3C8zMBpaGCIcdO+rdAjOzgaUhwuHFF+vdAjOz\ngaUhwiHCJ8KZmdVSQ4TDkUe692BmVksNEQ4jRjgczMxqqSHCwT0HM7PaqiocJE2TtEbSWkmzy+xv\nkrRIUpukRySNzba/VdKKguXiauss5HAwM6utiuEgaRBwKzAVOB2YIenUomKzgG0RMQG4Gbgp2/5r\nYFJEnAVcBPxfSYOqrHM/h4OZWW1V03OYDLRFxPqI2AssApqLyjQDC7L1xcAUgIh4KSL2ZduPIM0j\nXW2d+zkczMxqq5pwGANsKLi/MdtWtkxEtAPPSxoJIGmypMeBlcAns7Cops79HA5mZrU1pIoy5Sao\njgpl1FEmIpYDZ0iaCNwm6b4q69zvySdb2L4d1q6FXC5HLperotlmZgNHPp8nn8/3WH3VhMNGYGzB\n/eOBTUVlNgAnAJskDQaOiojnCgtExJOSdgBnVFnnfu96VwtjxsBnPlNFa83MBqDiP5znzp17SPVV\nM6zUCoyXNE5SEzAduLuozD3AzGz9UmAZgKQTs7BA0jjgFGBdlXXu52ElM7PaqthziIh2SVcDS0lh\nMj8iVkuaC7RGxBJgPrBQUhvwLOnLHuAdwBxJe0gHoz8VEdsAytXZWRtGjIBnnun2azQzsy5SRKdD\n/X2CpJg3L1i1Cr761Xq3xsysMUgiIsod362Kz5A2M7MSDgczMyvhcDAzsxIOBzMzK+FwMDOzEg4H\nMzMr4XAwM7MSDRMO27fXuxVmZgNHQ4TD4YfDyy/D3r31bomZ2cDQEOEgpd7Djh31bomZ2cDQEOEA\nPu5gZlZLDgczMyvhcDAzsxIOBzMzK+FwMDOzElWFg6RpktZIWitpdpn9TZIWSWqT9Iiksdn2CyT9\nQtJKSa2Szi94zENZnSskPSZp1MHa4HAwM6udijPBSRoE3ApMIc3z3CrprohYU1BsFrAtIiZIuhy4\niTQb3FbgfRGxRdLpwAOk+aI7zIiIFdU01OFgZlY71fQcJgNtEbE+IvYCi4DmojLNwIJsfTEpSIiI\nlRGxJVt/Ahgq6bAuPj/gcDAzq6VqvpzHABsK7m/MtpUtExHtwPOSRhYWkPRBYEUWMB2+lQ0pXV+p\nESNGOBzMzGqlmnAoNwdp8cTTxWVUWCYbUvo88ImCMh+KiDcDfwb8maQPH6wRvr6SmVntVDzmQOop\njC24fzzp2EOhDcAJwCZJg4GjIuI5AEnHAz8APhIR6zoeEBGbs9sdkr5LGr76TrkGtLS08MtfwlNP\nwUUX5cjlctW8NjOzASOfz5PP53usPkUUdwKKCqQv+ydJxxE2A8tJB5JXF5S5EjgjIq6UNB24OCKm\nSzoayANzI+KOojqPjohns2MQ3wV+FBFfL/P8ERHccQcsWAB33nmoL9nMrP+TRESUG/mpSsWeQ0S0\nS7oaWEoahpofEaslzQVaI2IJMB9YKKkNeJb0SyWAq4CTgc9J+mfSUNOFwE7gAUlDgMHAg8A3DtaO\nY4+FrVu78xLNzKyrKvYc6q2j57BmDTQ3w5NP1rtFZmZ936H2HBrmDGn3HMzMaqdheg779sHQobBz\nJxx2WOXHmZkNZAOm5zBoEBxzDGzbVu+WmJn1fw0TDuChJTOzWnE4mJlZiYYKh1Gj4I9/rHcrzMz6\nv4YKB/cczMxqw+FgZmYlGiocPKxkZlYbDRUO7jmYmdWGw8HMzEo0VDh4WMnMrDYaKhzcczAzq42G\nubYSwO7dabrQ3btB3b5iiJlZ/zdgrq0E6cJ7hx8OL7xQ75aYmfVvDRUO4KElM7NaqCocJE2TtEbS\nWkmzy+xvkrRIUpukRySNzbZfIOkXklZKapV0fsFjzpa0Kqvz5mobPHo0PP10taXNzKw7KoaDpEHA\nrcBU4HRghqRTi4rNArZFxATgZuCmbPtW4H0R8Wbgo8DCgsd8FfhYRJwCnCJpajUNPvFEWLeumpJm\nZtZd1fQcJgNtEbE+IvYCi4DmojLNwIJsfTEwBSAiVkbElmz9CWCopMMkHQeMiIjl2WNuAy6upsEn\nnwy/+101Jc3MrLuqCYcxwIaC+xuzbWXLREQ78LykkYUFJH0QWJEFzJisnoPVWdbrXw+//301Jc3M\nrLuqCYdyP4Uq/v1rcRkVlpF0OvB54BNdqLMsh4OZWe8bUkWZjcDYgvvHA5uKymwATgA2SRoMHBUR\nzwFIOh74AfCRiFhXUOcJFercr6WlZf/6G96Q4/e/z1XRbDOzgSOfz5PP53usvoonwWVf9k+SjiNs\nBpYDMyJidUGZK4EzIuJKSdOBiyNiuqSjgTwwNyLuKKr3UeAaoBX4IfCViLi/zPNHYRvb22H4cHju\nOTjiiO68ZDOz/q/XT4LLjiFcDSwFngAWRcRqSXMlvS8rNh8YJakN+N/AnGz7VcDJwOckrZD0mKRR\n2b4rs8etJR3wLgmGcgYPhrFj/YslM7Pe1FCXz+hw0UVw9dXw3vfWqVFmZn3cgLp8RgcflDYz610N\nGw4+18HMrPc0bDi452Bm1nsaNhzcczAz6z0NeUB65840K9y2bekS3mZm9koD8oD0sGEwcSKsXFnv\nlpiZ9U8NGQ4A55wDjz5a71aYmfVPDRsOkyfD8uWVy5mZWdc5HMzMrERDHpCGdI2lo4+G9eth5Mgy\nDzQzG8AG5AFpSNdYmjQJfvGLerfEzKz/adhwAB+UNjPrLQ0dDuefD/dXdS1XMzPrioY95gCweze8\n5jWwZg0cd1yNG2Zm1ocN2GMOAEOHwrRpcPfd9W6JmVn/0tDhAHDJJXDnnfVuhZlZ/1JVOEiaJmmN\npLWSZpfZ3yRpkaQ2SY9IGpttHylpmaTtkr5S9JiHsjqLZ4jrkosugocfhhde6M6jzcysnIrhIGkQ\ncCswFTgdmCHp1KJis4BtETEBuBm4Kdv+EnA98JlOqp8REWdFxNkR8cfuvICjjoJcDr73ve482szM\nyqmm5zCZNMfz+ojYCywCmovKNAMLsvXFwBSAiNgZET8Ddh/C81d0zTVwyy3Qx4+tm5k1jGq+nMcA\nGwrub8y2lS0TEe3A85KqOW/5W9mQ0vXVNLYzF1wA+/bBQw8dSi1mZtZhSBVlyv0Uqvhv9OIyKlOm\n2IciYrOk4cAPJH04Ir5TrmBLS8v+9VwuRy6Xe+WTCT79abj5Znj3uys8q5lZP5TP58nn8z1WX8Xz\nHCSdC7RExLTs/hwgIuLGgjL3ZWUelTQY2BwRowv2zwQmRcSnO3mOTvcf7DyHQjt3wvjxcNdd8Na3\nVixuZtav1eI8h1ZgvKRxkpqA6UDxmQX3ADOz9UuBZeXaun9FGizp1dn6YcD7gMe72PZXGDYM/uVf\n4LOf9bEHM7NDVdUZ0pKmAbeQwmR+RHxB0lygNSKWSBoKLATOAp4FpkfEuuyxfwBGAE3A88CFwP8A\nPyENaw0GHgSuLddFqLbnAOlKrWeeCf/6r3DxxVU9xMysXzrUnkNDXz6jnB//GK64Alatgle9qhcb\nZmbWhzkcyvjEJ9Lt17/eCw0yM2sADocyXngB3vhGmDcP3vveXmqYmVkfNqAvvNeZo46C22+HWbNg\nw4bK5c3M7JX6ZTgAvOMd8Hd/B5dfni7tbWZm1euXw0od9u2Dyy6DI46A225LJ8uZmQ0EHlY6iEGD\nUiisXZt+3mpmZtWp5vIZDW3YsHTW9LnnpjOoP/SherfIzKzv6/fhAGkK0SVL0nWXxoyBd72r3i0y\nM+vb+vWwUqEzzoBFi+DSS6G1td6tMTPr2wZMOEDqOcyfD3/xF/D4IV3JycysfxtQ4QApGL78ZZg6\nFdra6t0aM7O+aUAccyg2Ywa8+CJMmZKuxTRhQr1bZGbWtwzIcAD4+MfT7fnnp4CYOLG+7TEz60sG\nbDhACojDDkvHIn70IzjttHq3yMysbxjQ4QDw0Y/CkCFpiGnJEpg0qd4tMjOrvwEfDgAf/jAMHw4X\nXQTf/S5ccEG9W2RmVl9V/VpJ0jRJayStlTS7zP4mSYsktUl6RNLYbPtIScskbZf0laLHnC1pVVbn\nzT3zcrrvkktg8eJ0BvX3vlfv1piZ1VfFcJA0CLgVmAqcDsyQdGpRsVnAtoiYANwM3JRtfwm4HvhM\nmaq/CnwsIk4BTpE0tXsvoee8853w4INw7bXwH/9R79aYmdVPNT2HyUBbRKyPiL3AIqC5qEwzsCBb\nXwxMAYiInRHxM+AVF82WdBwwIiKWZ5tuA/rErM9vehP89Kdwyy3wuc9BH79orZlZr6gmHMYAhVPm\nbMy2lS0TEe3A85JGVqhzY4U66+akk+Dhh2HpUvirv4Jdu+rdIjOz2qrmgHS564EX/z1dXEZlynS1\nzv1aWlr2r+dyOXK53EGq7hmjR0M+D1dckS7Ud9dd8NrX9vrTmpl1Sz6fJ5/P91h9FSf7kXQu0BIR\n07L7c4CIiBsLytyXlXlU0mBgc0SMLtg/E5gUEZ/O7h8HPBQRb8juTwfeFRGfKvP83Z7spydEwL/9\nG3z963DnnXD22XVriplZ1Wox2U8rMF7SOElNwHTg7qIy9wAzs/VLgWXl2tqxEhFbgBckTZYk4K+B\nu7ra+FqQ4PrrD1yPafHierfIzKz3VTVNqKRpwC2kMJkfEV+QNBdojYglkoYCC4GzgGeB6RGxLnvs\nH4ARQBPwPHBhRKyRNAn4NnA4cG9E/G0nz13XnkOhFSvg4oth+vTUmxjis0TMrI861J5Dv55Dujf8\n8Y/ppLmdO9P8EK97Xb1bZGZWynNI19ioUXDvvfDnfw5veQssKzeAZmbW4NxzOAQPPggf+QhcdRVc\ndx0MHlzvFpmZJR5WqrOnnkqX3AC47TYYN66+7TEzAw8r1d2YMWlo6b3vTcNMCxf6rGoza3zuOfSg\nFSvSwerTT4evfQ1GHuwccTOzXuSeQx9y1lnwi1+kXzC9+c1wzz31bpGZWfe459BLHnoozTT3lrfA\nV76SLsdhZlYr7jn0UeefD6tWpQPUb3wjLFjgYxFm1jjcc6iBxx6DWbPg2GPTPBETJtS7RWbW37nn\n0ADOPhuWL4cLL4S3vQ3mzIEXX6x3q8zMOudwqJHDDoPPfhZ+/WvYtAlOPTXNV93gnSIz66c8rFQn\n//3fcM01MHw4fPGLMHlyvVtkZv2Jh5Ua1NvfDq2tMHMmXHIJXHYZtLXVu1VmZonDoY4GD4aPfSyF\nwplnpuMRV10FTz9d75aZ2UDncOgDhg2Df/xHWLMGmprgtNPSQeutW+vdMjMbqBwOfcioUWnGuRUr\n4E9/SgetZ892SJhZ7VUVDpKmSVojaa2k2WX2N0laJKlN0iOSxhbsuy7bvlrShQXb10laKWmFpOU9\n83L6h7Fj4atfhV/9Kv3kdeJE+Pu/T79yMjOrhYrhIGkQcCswFTgdmCHp1KJis4BtETEBuBm4KXvs\nacBlwBuAi4B52ZzRAPuAXEScFRH+rU4ZJ5yQTppbtQp274YzzoArroAnnqh3y8ysv6um5zAZaIuI\n9RGxF1gENBeVaQYWZOuLgXdn638JLIqIl7M5pduy+gBU5fMPeMcfn67P1NYGJ58MF1yQLhH+0EM+\nT8LMekc1X85jgA0F9zdm28qWiYh24E+SRpZ57FMFjw3gAUmtkj7ejbYPOK9+NfzTP8Ef/pB+/nrl\nlfCmN8G8efDCC/VunZn1J0OqKFPuJIriv1c7K3Owx54XEVskHQv8SNLqiHi4XANaWlr2r+dyOXK5\nXKU292uHH55+AjtrVuo9zJsH118Pl18On/pUCgwzG1jy+Tz5fL7H6qt4hrSkc4GWiJiW3Z8DRETc\nWFDmvqzMo5IGA5sjYnRxWUn3AzdExKNFz3EDsD0ivlTm+fvlGdI97amn4JvfhG98A048ET75SXj/\n+9PPZM1s4KnFGdKtwHhJ4yQ1AdOBu4vK3APMzNYvBZZl63cD07NfM50EjAeWSxom6cjsBQwHLgQe\n7+6LsDRd6Q03pCGna6+F73wnbbviCsjnYd++erfQzBpJVddWkjQNuIUUJvMj4guS5gKtEbFE0lBg\nIXAW8CwwPTsAjaTrSL9m2gv8bUQszYLiDtIQ0xDg9oj4QifP7Z5DN23enC7ut2BBOm/iIx9Jy8SJ\n9W6ZmfW2Q+05+MJ7A8TKlXDbbXD77eknsh/8IHzgAzB+fL1bZma9weFgXfLyy/Bf/wWLF8MPfpDm\nu/7gB9PiHoVZ/+FwsG5rb0+XDl+8GP7zP+GYY6C5Gd7zHjj33HRhQDNrTA4H6xH79sHPfw5LlsAP\nf5h+/TR1ajrZburUdI6FmTUOh4P1io0b4d57U1A89FC6dMcFF8CUKalXMXRovVtoZgfjcLBe99JL\n8NOfwo9/DMuWwerVae6JKVPg3e9Oc2R7CMqsb3E4WM0991w6qL1sWQqMjRtTb+K889IMd+ecAyNG\n1LuVZgObw8HqbutWeOSRdHD7Zz9L81FMmHAgLM47D8aNA3X7Y2pmXeVwsD5nzx547LEUFB2BsXcv\nTJqUhqAmTUrLiSc6MMx6i8PBGsKmTSkwfvnLA8uuXQfC4swz00HviRPTVKlmdmgcDtawtmw5EBir\nVsHjj8O6dXDSSSkoCpeTT/ZBb7OucDhYv/LSS/Dkk2m2u8cfP7Bs3pxC45RT0vGMwtvXvtbDU2bF\nHA42IOzcCb/9bZoNb+3aV97u2JGCYsKEdK2ok05KB8BPPDHNx3344fVuvVntORxswPvTnw6ExW9/\nC+vXp+Gp9ethwwYYOTIFRUdgjBuXljFj0rWlRo1yz8P6H4eD2UG0t6chqY7A6AiN9evTQfJNm+DF\nF9PQ1Otel5aO0ChcP+44eNWrHCLWOBwOZofopZdSgDz11IHA6Fg6tm3ZksqNGgWjR8Oxx6alY734\n9tWvTmEyqJrptMx6QU3CIZvs52YOTPZzY9H+JuA2YBLwR+DyiPifbN91wBXAy2ST/VRTZ0HdDgfr\nE3bvTif8bd0KzzzT+e0zz8C2balHMmJEutrtyJHptmMpd//oo+Goo9JjjjoqHStxT8W6q9fDQdIg\nYC0wBdhEmjZ0ekSsKSjzKeCNEXGlpMuBSyJiuqTTgNuBtwLHAw8CEwBVqrOgbodDD8nn8+RyuXo3\no9+o9H62t6fjIdu2pUuOPPdc5+sdy/btaXnhhfT4jqDouO1s/cgjYfjwNGd44VK87Ygj+m5vxp/P\nnnWo4TCkijKTgbaIWJ894SKgGSj8Im8GbsjWFwP/J1v/S2BRRLwMrJPUltWnKuq0Hub/fD2r0vs5\neHDqEYwc2b369+w5EBQHu920Kd3u3Fm67Njxyvu7dqUr6pYLko7l8MPTMnRoWsqtV9pfuN7UBIcd\n9sqlXED589m3VBMOY4ANBfc3kr7gy5aJiHZJf5I0Mtv+SEG5p7JtqqJOswGtqSkdu+jJuTQiUkB0\nFiI7dqThs9270zGWwttdu1Lvpnh/ubKF63v2pMunFC6DBh0IiiFD0u2uXfDtb5cGSTXLkCEpjMst\nB9vXE+U7lkGD0jDgoEGl6z19v9y+jqWnVBMO5Z6ueJynszKdbS/XsfXYkVkvkw70EOolIg2ZFQfG\njTfCNdekqWyL91VaXn451dnZ0rF/z56DlysuX+2yb9+BJaL8elfvd/WxHf++3/9+mh++B/6h4qAL\ncC5wf8H9OcDsojL3Aedk64OBZ8qVBe4HzqmmzoJ94cWLFy9eur5U+n4/2FJNz6EVGC9pHLAZmA7M\nKCpzDzATeBS4FFiWbb8buF3Sl0nDSeOB5aSeQ6U6Ib06/17DzKzGKoZDdgzhamApB352ulrSXKA1\nIpYA84GF2QHnZ0lf9kTEbyR9D/gNsBe4MvvpUdk6e+H1mZlZN/T5k+DMzKz2+ugvntNJcpLWSFor\naXa929OIJK2TtFLSCknLs23HSFoq6UlJD0h6Vb3b2VdJmi/paUmrCrZ1+v5J+oqkNkm/knRmfVrd\nN3XyXt4gaaOkx7JlWsG+67L3crWkC+vT6r5L0vGSlkn6jaRfS/p0tr3HPp99MhyyE+9uBaYCpwMz\nJJ1a31Y1pH1ALiLOioiOnwrPAR6MiImkY0PX1a11fd//I30GC5V9/yRdBJwcEROAvwG+VsuGNoBy\n7yXAlyLi7Gy5H0DSG4DLgDcAFwHzJJ8rXuRl4NqIOA14G3BV9h3ZY5/PPhkOFJx4FxF7gY6T5Kxr\nROm/cTOwIFtfAFxc0xY1kIh4GHiuaHPx+9dcsP227HGPAq+S9JpatLMRdPJeQvmfuzeTnTwbEeuA\njpNnLRMRWyLiV9n6i8Bq0lUoeuzz2VfDodyJd2Pq1JZGFsADklolfSzb9pqIeBrSBww4tm6ta0yj\ni96/0dn24s9sxwmfdnBXZcMc3ywYAvF72QWSTgTOBH5O6f/vbn8++2o4VHPinVV2XkS8BXgP6T/h\nn+H3sbf4M9t180hDHWcCW4AvZtv9XlZJ0pGkSxb9bdaD6Ox96vJ72lfDYSMwtuD+8aQL9FkXZH85\nEBFbgTtJXfOnO7qTko4DnqlfCxtSZ+/fRuCEgnL+zFYQEVsLrqr5DQ4MHfm9rIKkIaRgWBgRd2Wb\ne+zz2VfDYf+Jd9nlwKeTTqizKkkalv1VgaThwIXAr0nv40ezYjOBu8pWYB3EK//qKnz/PsqB9+9u\n4K8BJJ3F4w4NAAAA60lEQVQLPN/Rvbf9XvFeZl9eHd4PPJ6t3w1Ml9Qk6SQOnDxrr/Qt4DcRcUvB\nth77fPbZ8xyyn7XdwoGT5L5Q5yY1lOw/1R2kruMQ4PaI+EJ2QcTvkf6K+B/g0oh4vn4t7bskfRfI\nAa8GniZdefhO4PuUef8k3QpMA3YA/ysiHqtDs/ukTt7L80lj5fuAdcDfdHxhZfPAzCKdPLt/HhhL\nJL0d+AnpD76Oy2X8IylEy/7/7urns8+Gg5mZ1U9fHVYyM7M6cjiYmVkJh4OZmZVwOJiZWQmHg5mZ\nlXA4mJlZCYeDmZmVcDiYmVmJ/w856gDqst6XTAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10487f908>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot loss history\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(loss_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Your Turn\n",
    "\n",
    "Please switch to the **Theano** backend and re-run the notebook.\n",
    "\n",
    "You _should_ see no difference in the execution!\n",
    "\n",
    "**Reminder**: please keep in mind that you *can* execute shell commands from a notebook (pre-pending a `!` sign).\n",
    "Thus:\n",
    "\n",
    "```shell\n",
    "    !cat ~/.keras/keras.json\n",
    "```\n",
    "should show you the content of your keras configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to re-implement the Logistic Regression Model using the `keras.backend` APIs.\n",
    "\n",
    "The following code will look like very similar to what we would write in Theano or Tensorflow - with the *only difference* that it may run on both the two backends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from kaggle_data import load_data, preprocess_data, preprocess_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9 classes\n",
      "93 dims\n"
     ]
    }
   ],
   "source": [
    "X_train, labels = load_data('data/kaggle_ottogroup/train.csv', train=True)\n",
    "X_train, scaler = preprocess_data(X_train)\n",
    "Y_train, encoder = preprocess_labels(labels)\n",
    "\n",
    "X_test, ids = load_data('data/kaggle_ottogroup/test.csv', train=False)\n",
    "\n",
    "X_test, _ = preprocess_data(X_test, scaler)\n",
    "\n",
    "nb_classes = Y_train.shape[1]\n",
    "print(nb_classes, 'classes')\n",
    "\n",
    "dims = X_train.shape[1]\n",
    "print(dims, 'dims')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "feats = dims\n",
    "training_steps = 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = K.placeholder(dtype=\"float\", shape=X_train.shape) \n",
    "target = K.placeholder(dtype=\"float\", shape=Y_train.shape)\n",
    "\n",
    "# Set model weights\n",
    "W = K.variable(np.random.rand(dims, nb_classes))\n",
    "b = K.variable(np.random.rand(nb_classes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Define model and loss\n",
    "y = K.dot(x, W) + b\n",
    "loss = K.categorical_crossentropy(y, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "activation = K.softmax(y) # Softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Minimize error using cross entropy\n",
    "cross_entropy = K.categorical_crossentropy(activation, target)\n",
    "loss = K.mean(-K.sum(cross_entropy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grads = K.gradients(loss, [W,b])\n",
    "updates = [(W, W-lr*grads[0]), (b, b-lr*grads[1])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: -218494.890625\n",
      "Loss: -989694.875\n"
     ]
    }
   ],
   "source": [
    "# Training\n",
    "loss_history = []\n",
    "for epoch in range(training_steps):\n",
    "    current_loss = train([X_train, Y_train])[0]\n",
    "    loss_history.append(current_loss)\n",
    "    if epoch % 20 == 0:\n",
    "        print(\"Loss: {}\".format(current_loss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAEPCAYAAAATXoCrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcFNW9///Xe1g0YBg2QQSEEYgajLtoFnXQqIiPiNeI\nwSBBXGLikvw0RjDEMIQkXkmMRr3e3K9ixBU1XhWjUVCcLNcFjMENQYyCMIgim4Bhm/n8/uiasWl6\nNpbp0nk/H496UP2pU+ecrmn60+dUdbUiAjMzs7QqKnQHzMzM6uJEZWZmqeZEZWZmqeZEZWZmqeZE\nZWZmqeZEZWZmqZbKRCVpoqQ3JM2W9KCkdlnbrpQ0P9l+QlZ8kKS5kt6UNDor3lvS85LmSbpXUssk\n3lrSlKSu5yTtVV8bZmbW9FKZqIBpQP+IOAiYD1wJIOmLwBnAfsBJwM3KKAJuAk4E+gNnSto3qesa\n4NqI2AdYBZybxM8FVkREP+B6YGJdbezk52tmZrVIZaKKiKcioip5+DzQI1k/BZgSEZsjYgGZJDYg\nWeZHxMKI2ARMAYYk+xwLPJisTwZOTdaHJI8B/piUq6sNMzMrgFQmqhznAI8n692BRVnbKpJYbnwx\n0F1SJ2BlVtJbnJTdoq6IqARWS+pYRxtmZlYALQvVsKTpQNfsEBDA2Ih4NCkzFtgUEfdmlckV5E+4\nkZTP3af6nlG11VVb3MzMCqBgiSoijq9ru6SRwGA+mZKDzIioZ9bjHsASMsllr9x4RHwoqb2komRU\nVV0+u64lkloAxRGxUlJtbeT2z8nLzGwbRESjzvuncupP0iDgCuCUiNiQtWkqMCy5Yq8E6AvMBGYB\nfSX1ktQaGAY8kuwzAxiarI/Mik9NHpNsn1FPG1uJCC8RjBs3ruB9SMviY+Fj4WNR97ItUpmogBuB\n3YDpkl6SdDNARMwB7gfmkDlvdWFkVAIXk7la8HUyF0PMTeoaA1wm6U2gIzApiU8COkuaD/x/Sbla\n28jXybPOGs877yzcsc/czMy2ULCpv7pE5pLx2rZdDVydJ/4EsE+e+DvAEXniG8hcht7gNnLdfffl\nPP/8OKZPv4SSkl71FTczs22Q1hHVp0Rb/vWv8Vx11e2F7khBlZaWFroLqeFj8Qkfi0/4WGwfbeuc\nYXOXuZgic+wGDhzHjBnjC9wjM7P0k0Q08mKKVE79fbqsY889PTCtT+/evVm40OfzzJqLXr16sWDB\ngh1Sl0dU2ygzolpLnz4+R9UQyaeoQnfDzJpIbf/nt2VE5aHAdhg+/DdOUmZmO5lHVNtIUm1XrVse\nHlGZNS8eUZmZWbPhRGW2E9xzzz0MGjRom/bdf//9+etf/7qDe5R+gwcP5s477yx0N+p14okncu+9\n99ZfsJFld6QRI0bw85//vMnb3Vk89beNPPXXOGme+ispKWHSpEkce+yx9RfewUaNGkXPnj23+01l\n4cKFlJSUsNtuuwHQuXNnLrjgAkaPHl3Pnp9tgwcP5m9/+xuSWL9+PZLYZZddADjrrLO4+eabC9zD\nnWPEiBH069ePn/3sZwXrw46c+vPl6VZQ77yzkKuuup2Kiiq6dy9iwoSzG31xyo6o47NAEqtXr0YS\n//jHPzjmmGM47LDDOO6443ZoO5WVlbRo0WKH1rmzPP744zXrDflQ8Gl6bs2Jp/6sYN55ZyHHH38j\nd999OeXl47n77ss5/vgbG3X/xB1RR11uueUW+vXrR+fOnTn11FN57733arZNmzaNfffdlw4dOnDR\nRRdRWlrKbbfdBsDkyZM56qijaspeeumldO3alfbt23PQQQcxZ84cbrnlFu6++24mTpxIu3btGDIk\n81ufJSUlzJiRuUdyVVUVv/rVr+jbty/FxcUcfvjhVFRU1Nrf6k+whx56KP3792f27Nk129577z1O\nP/10unTpQp8+fbjxxhtrtq1fv56RI0fSsWNH+vfvz69//Wt69vzkRwRKSkqYOHEiBx54ILvtthtV\nVVV11jdr1iwOP/xwiouL6datG5dffjkAGzZsYMSIEXTu3JkOHTpwxBFHsGzZMgAGDhxYc/wigl/8\n4hf07t2bPfbYg7PPPpuPPvoIyIwei4qKuOOOO+jVqxddunThV7/6VYP/ptmefvppSkpKuPrqq+nW\nrRvf/e53WbFiBSeffDJdunShU6dOnHLKKSxZ8skPKBx11FHccccdAEyaNInS0lIuu+wyOnToQN++\nfZk+ffo2lX377bc56qijKC4uZtCgQVx44YWMGjWqzn5PmDCBzp0706dPH+67774tyixfvpzBgwfT\nrl07vvrVr27xPcZLLrmEnj170r59e4444giee+65mm0vvPAChx56aM3fLntU/n//9398+ctfpkOH\nDhxyyCH87W9/25bD3niFvpPup3XJHDprqHzHa/jwsoC1AZG1rI3hw8saXO+OqKN3797x9NNPbxV/\n+umno3PnzjF79uzYuHFjXHLJJXH00UdHRMSyZcuiXbt28fDDD0dlZWX87ne/i9atW8ekSZMiIuL2\n22+Po446KiIinnzyyTjssMPio48+ioiIuXPnxtKlSyMi4uyzz46rrrqq1v5MnDgxDjjggJg/f35E\nRLzyyiuxYsWKrfq6YMGCKCoqis2bN0dExHPPPRdt27aNhx9+OCIiqqqq4tBDD41f/OIXsXnz5njn\nnXeiT58+MW3atIiIGD16dJSWlsbq1aujoqIiDjjggOjZs+cWfTr44IOjoqIi1q9fX299X/7yl+Ou\nu+6KiIh169bFCy+8EBER//M//xOnnHJKTR0vvfRSrFmzJiIiSktLa47fpEmTol+/frFgwYJYt25d\nnHbaaTFixIia5yopvvvd78aGDRvi5Zdfjl122SXmzp1b598537F+6qmnomXLlvHTn/40Nm3aFOvX\nr49ly5bFww8/HBs2bIg1a9bEN7/5zRg6dGjNPl/72tdi8uTJERFx6623RuvWreP222+PqqqquPHG\nG7c4bo0pO2DAgLjyyitj06ZN8de//jU+//nPx6hRo/I+l+p+jx49OjZu3BgzZsyINm3axL/+9a+I\niDjrrLNi9913j5deeik2b94c3/rWt2qOX0TEXXfdFatWrYrKysqYOHFidO/ePTZu3BgREYcffnhM\nmTIlIiLWrl0bM2fOjIiIRYsWRadOnWL69OkRkXldd+7cOe/rMSL///mseKPebz2isoKpqKgC2uZE\n27JkSVW+4jutjtrcc889nHvuuRx44IG0atWKq6++mueff553332XP//5z+y///4MGTKEoqIifvCD\nH9C1a9e89bRq1Yo1a9YwZ84cIoJ99tmn1rK5Jk2axC9/+Uv69u0LwJe+9CU6dOiQt2xEsPvuu9Om\nTRu++tWvcuGFF9aM0mbNmsWHH37I2LFjadGiBb179+a8885jypQpADzwwAOMHTuWdu3aseeee/KD\nH/xgq/p/+MMfsueee7LLLrvUW1+rVq146623WL58OW3atGHAgAE18eXLl/Pmm28iiYMPPrjmvFru\nsb/sssvo1asXbdq04eqrr2bKlClUVWX+rpIoKyujdevWHHDAARx44IG8/PLLDTqmuVq1akVZWRkt\nW7Zkl112oXPnzgwZMoTWrVuz2267MWbMGP7yl7/Uun+fPn0YOXIkkhg5ciSLFy9mxYoVDSpbUVHB\nihUrePvtt3nllVcYN24cLVu25KijjuLkk0+us98tWrSgrKyMVq1aMXDgQAYNGsQDDzxQs/3000/n\n4IMPpkWLFgwfPnyL0fXw4cMpLi6mqKiIyy+/nI8++oi33noLgNatWzN//nxWrFhB27ZtOfzwwwG4\n4447GDJkCF//+tcBOOGEEzjwwAN54oknGnagt4MTlRVM9+5FwLqcaONuSbUj6qjNkiVL6NXrk3Nd\nbdu2pWPHjlRUVLBkyZItpsYAevTokbeegQMHcvHFF3PRRRexxx578L3vfY+1a9c2qA+LFi1i7733\nblBZSSxfvpx169bxm9/8hvLycjZv3gxkpssqKiro2LEjHTt2pEOHDlx99dV88MEHNc81u/+5zy33\n+dVX32233ca8efPYd999OeKII3jssceAzEn+E088kWHDhtGjRw9Gjx5NZWXlVm3lHvtevXqxefNm\n3n///ZpYdrJv06ZNg49prq5du25xXurjjz/mvPPOo1evXrRv357jjjuODz/8sNb999hjjy36AdTa\nl9yyEcHatWt577336NSpU82FHpD/b5CtU6dO7LrrrjWPe/XqtcUUZW5b2X2aOHEi++23Hx06dKBj\nx458/PHHNc/xD3/4A6+//jr77LMPRx55JH/+85+BzN/8nnvu2eJv/sILL2zR5s7iRGUFM2HC2fTp\nM45PEs06+vQZx4QJZzdpHbXZc889t5jXX7duHcuXL6d79+5069aNRYsWbVF+8eLFtdZ18cUX8+KL\nL/L6668zb948fv3rXwOZ5FKXnj178q9//avBfY4IJHHppZeyyy671FzV1rNnT/bee29WrFjBihUr\nWLlyJatXr+bRRx+tea7Z/X/33Xe3qju7r/XV16dPH+655x6WLVvGFVdcwemnn86///1vWrZsyVVX\nXcXrr7/Os88+y5/+9KeaczjZco/9woULadWqVYNHoo2R+zeYOHEiCxcu5MUXX2TVqlU15wt3pm7d\nurF8+XI2btxYE8t9feVavnw5GzZ88ruy7777LnvuuWe9bT3zzDNcd911PPTQQ6xcuZKVK1fStm3b\nmvOb/fr1495772XZsmVcdtllfPOb32Tjxo307NmTc845Z4u/+Zo1a/jRj360jc+64ZyorGBKSnox\nffolDB/+GwYOHLdNt6TaEXUAbNy4kQ0bNtQslZWVfPvb3+YPf/gDr7zyChs2bOAnP/kJRx55JHvt\ntRcnn3wyr732GlOnTqWyspKbbrppi0/72V588UVmzpzJ5s2b+dznPseuu+5a8wm+a9euvP3227X2\n67zzzuOqq66qmZZ59dVXWblyZd6y1W801caMGcM111zDxo0bGTBgAO3atWPixImsX7+eyspKXn/9\ndV588UUAhg4dytVXX82qVauoqKjgv/7rv+o8XvXVd/fdd9d8Qi8uLkYSLVq0oLy8nNdee42qqip2\n2203WrVqRcuWW198fOaZZ3LdddexYMEC1q5dy9ixYxk2bBhFRUV5n+uOtGbNGtq0aUNxcTHLly9n\n/Pid/8sIe++9N1/60pcYP348mzZt4u9//3vNKLQ2lZWVlJWVsWnTJsrLy3niiScYOnRonftAZrTX\nqlUrOnbsyMaNGxk3bhwff/xxzfa77rqL5cuXA9CuXTuKioooKipixIgRPPTQQzz11FNUVVWxfv16\nysvLWbp06fY9+QZworKCKinpxV13ZX4m5a67xm3TZeU7oo6TTz6ZNm3a8LnPfY42bdowfvx4jj32\nWCZMmMBpp51G9+7deeedd2rOwXTq1IkHHniAH//4x3Tu3Jm5c+dy2GGHbTF1U+2jjz7i/PPPp2PH\njpSUlNC5c+eaq+DOPfdcXn/9dTp27Mhpp50GbPkJ/7LLLuOMM87ghBNOoLi4mPPOO49///vfeZ9D\n7sjg5JNPpmPHjtxyyy0UFRXx6KOPMnv2bEpKSujSpQvnn39+zZV0P/vZz+jevTslJSWccMIJDB06\ndIvnklt3ffU98cQT9O/fn3bt2nHppZdy33330bp1a5YuXcrpp59OcXEx/fv3Z+DAgQwfPnyrNs45\n5xxGjBjB0UcfTZ8+fWjTpg033HBDrf2pb2Ta0DKQOearVq2iU6dOfO1rX9vqXFF99WRvb0zZe++9\nl7/85S907tyZCRMmMGzYsLyvp2o9e/akbdu2dOvWjVGjRjFp0qSaaeK62h08eDDHHXcc/fr1Y++9\n96Z9+/Z069atZvvjjz/OfvvtR3FxMVdccQX3338/LVu2pFevXjz00ENMmDCB3Xffnd69e/Pb3/62\n5rzhzuQv/G4jf+G3cdL8hd8dISLo0aMH99xzD8ccc0yhu7Pdfv/733PffffxzDPPFLorzVb1xRBj\nx47datvTTz/N+eefX+dovNB8rz+zFJg2bRqrV69mw4YN/PKXvwTgyCOPLHCvts3SpUt59tlniQjm\nzZvHtddeWzPCs6Yxa9YsFixYQETw+OOP89hjj9VctdncpTJRSfq5pJcl/VPSE5L2yNp2g6T5kmZL\nOigrPlLSm5LmSfpOVvwQSa8k267PineQNC0p/6Sk4vraMMv23HPP0adPH7p06cJjjz3GI488UudU\nTZpt3LiRCy64gHbt2vH1r3+d//iP/+D73/9+obvVrCxZsoSjjz6adu3a8aMf/Yhbb72V/fffv9Dd\nSoVUTv1J2i0i1ibrlwBfjIjvSxoMXBQRJ0s6AvhdRBwpqQPwInAIIOAfwCERsVrSC8AlETFT0uPJ\nPk9KugZYHhETJY0GOkTEGEknARfntpGnj576a4TP+tSfmW3pMz/1V52kEm2B6rN1pwB3JGVeAIol\ndQVOBKZFxOqIWAVMAwYlI7HPR8TMZP87gFOT9SHA5GR9cvK4Op6vDTMzK4DU3pRW0i+A7wCrgIFJ\nuDuQ/eWCxUksN16RFV+cpzxA14h4HyAilkrqUksb1XXlv/bYzMx2qoKNqCRNT84dVS+vJv9+AyAi\nfhoRewF3A5dU75ZbDRB54tQTr7Nr27CPmZntJAUbUUXE8Q0sei/wJ6CMzIgo+74iPYAlSbw0J/5M\nHeUBlkrqGhHvJ1OEHyTxuvbZQllZWc16aWkppaWl+YoZmdu7NPR7LGb26Vd9C6zy8nLKy8u3q660\nXkzRNyLeStYvAY6KiDNyLqY4Erg+z8UURcn6oRGxqvpiCmAW8BhwQ0Q8kVxMsSIirpE0BmifXEyR\nt408ffTFFGZmjfRZ+uHE/5T0BTIXUSwEvgcQEY9LGizpLTI3dxuVxFdKmkAmQQUwPrmoAuBC4HZg\nV+DxiKi+1e81wP2SzgHeBYbW1YaZmRVGKkdUnwYeUZmZNd5n5vJ0MzOzak5UZmaWak5UZmaWak5U\nZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaW\nak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWaqlOVJIul1QlqWNW7AZJ8yXNlnRQ\nVnykpDclzZP0naz4IZJeSbZdnxXvIGlaUv5JScX1tWFmZk0vtYlKUg/g68DCrNhJQJ+I6AdcAPw+\niXcAfgYcDhwBjMtKPP8NnBcRXwC+IOnEJD4GeCoi9gFmAFfW1YaZmRVGahMVcB3w45zYEOAOgIh4\nASiW1BU4EZgWEasjYhUwDRgkaQ/g8xExM9n/DuDUrLomJ+uTk8d1tWFmZgWQykQl6RvAooh4NWdT\nd2BR1uPFSSw3XpEVX5ynPEDXiHgfICKWAl1qaaMiax8zM2tiLQvVsKTpQPZIRUAAPwV+Ahyfb7c8\njyNPnHridXatofuUlZXVrJeWllJaWlpP1WZmzUt5eTnl5eXbVUfBElVE5EtESNof6A28LElAD+Al\nSQPIjIh6ZhXvASxJ4qU58WfqKA+wVFLXiHg/mSL8IInXtc8WshOVmZltLfdD/Pjx4xtdR+qm/iLi\ntYjYIyL2jogSMonj4Ij4AJgKfAdA0pHAqmT67kngeEnFyYUVxwNPJlN6H0kakCS97wCPJE1NBc5O\n1s/Oiedrw8zMCqBgI6pGqJnCi4jHJQ2W9BawDhiVxFdKmgC8mJQfn1xUAXAhcDuwK/B4RDyRxK8B\n7pd0DvAuMLSuNszMrDAUUd8pG8tHUvjYmZk1jiQiIt+1ALVK3dSfmZlZNicqMzNLNScqMzNLNScq\nMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNL\nNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLNScqMzNLtVQmKknjJC2W9FKyDMradqWk+ZLe\nkHRCVnyQpLmS3pQ0OiveW9LzkuZJuldSyyTeWtKUpK7nJO1VXxtmZtb0UpmoEr+NiEOS5QkASfsB\nZwD7AScBNyujCLgJOBHoD5wpad+knmuAayNiH2AVcG4SPxdYERH9gOuBiUkbX8zXxs5/umZmlk+a\nE1W+5DAEmBIRmyNiATAfGJAs8yNiYURsAqYkZQGOBR5M1icDp2bVNTlZ/2NSDuCUWtowM7MCSHOi\nukjSbEm3SipOYt2BRVllKpJYbnwx0F1SJ2BlRFRlx3PriohKYLWkjnW0YWZmBdCyUA1Lmg50zQ4B\nAYwFbgZ+HhEh6RfAtcB55B9lBfkTbiTlc/eJrPZq2ydffCtlZWU166WlpZSWluYrZmbWbJWXl1Ne\nXr5ddRQsUUXE8Q0segvwaLK+GOiZta0HsIRMctkrNx4RH0pqL6koGVVVl8+ua4mkFkBxRKyUVFsb\nW8lOVGZmtrXcD/Hjx49vdB2pnPqTtEfWw9OA15L1qcCw5Iq9EqAvMBOYBfSV1EtSa2AY8Eiyzwxg\naLI+Mis+NXlMsn1GPW2YmVkBFGxEVY+Jkg4CqoAFwAUAETFH0v3AHGATcGFEBFAp6WJgGpnkOyki\n5iZ1jQGmSJoA/BOYlMQnAXdKmg8sJ5Pc6mrDzMwKQH4P3jaSnL/MzBpJEhHRqK/8pHLqz8zMrJoT\nlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZ\npZoTlZmZpZoTlZmZpZoTlZmZpZoTlZmZpVqDEpWkoQ2JmZmZ7WgN+uFESS9FxCH1xZoT/3CimVnj\nbcsPJ9b5U/SSTgIGA90l3ZC1qR2wufFdNDMza5w6ExWwBHgROAX4R1Z8DXDpzuqUmZlZtTrPUUXE\nyxExGegbEZOT9anAWxGxcmd2TNIlkuZKelXSf2bFr5Q0X9Ibkk7Iig9Kyr8paXRWvLek5yXNk3Sv\npJZJvLWkKUldz0naq742zMys6TX0qr/pktpJ6gi8BNwi6bqd1SlJpcA3gP0j4kvAb5L4fsAZwH7A\nScDNyigCbgJOBPoDZ0raN6nuGuDaiNgHWAWcm8TPBVZERD/gemBi0sYX87Wxs56rmZnVraGJqjgi\nPgJOA+6IiCOA43Zet/g+8J8RsRkgIj5M4kOAKRGxOSIWAPOBAckyPyIWRsQmYEpSFuBY4MFkfTJw\nalZdk5P1PyblIDPNma8NMzMrgIYmqpaSupEZafxpJ/an2heAo5Mpu2ckHZrEuwOLsspVJLHc+GIy\nF4B0AlZGRFV2PLeuiKgEVicjxtraMDOzAqjvYopqPweeBP4vImZJ2pvMSGObSZoOdM0OAQH8NOlX\n+4g4UtLhwAPA3kmZXEH+hBtJ+dx9qq8pr62u2uJbKSsrq1kvLS2ltLQ0XzEzs2arvLyc8vLy7aqj\nQYkqIh4gkyyqH78NfHN7Go6I42vbJul7wP8m5WZJqkxGR4uBvbKK9iBzZaLyxSPiQ0ntJRUlo6rq\n8iR19QSWSGpBZnpzpaTqeG4bW8lOVGZmtrXcD/Hjx49vdB0NvTNFD0kPSfpA0vuSHpTUo9GtNdzD\nJOfAJH0BaB0Ry8lccfit5Iq9EqAvMBOYBfSV1EtSa2AY8EhS1wyg+i4aI7PiU5PHJNtnZMWH5WnD\nzMwKoKFTf38A7uGTN/yzklito6Lt9AfgNkmvAhuA7wBExBxJ9wNzgE3AhcntISolXQxMI5N8J0XE\n3KSuMcAUSROAfwKTkvgk4E5J84HlZJJbXW2YmVkBNPQWSrMj4qD6Ys2Jb6FkZtZ423ILpYZe9feh\npLMktUiWs8iMQszMzHaqhiaqc8hcmr4UeA84HRi1szplZmZWraHnqCYAI6tvm5R83+g3ZBKYmZnZ\nTtPQEdUB2ff2i4gVwME7p0tmZmafaGiiKpLUofpBMqJq6GjMzMxsmzU02VwLPCvpj2Tu0nAG8Mud\n1iszM7NEgy5Ph5q7ih9L5i4QT0fEnJ3ZsbTz5elmZo23LZenNzhR2ZacqMzMGm9nfo/KzMysIJyo\nzMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws1ZyozMws\n1VKZqCRNkfRSsrwj6aWsbVdKmi/pDUknZMUHSZor6U1Jo7PivSU9L2mepHsltUzirZN25kt6TtJe\n9bVhZmZNL5WJKiKGRcQhEXEI8CDwvwCS9iPzEyP7AScBNyujCLgJOBHoD5wpad+kumuAayNiH2AV\ncG4SPxdYERH9gOuBiUkbX8zXxs5+zmZmll8qE1WOM4B7kvUhwJSI2BwRC4D5wIBkmR8RCyNiEzAl\nKQuZnyZ5MFmfDJyaVdfkZP2PSTmAU2ppw8zMCiDViUrSUcDSiHg7CXUHFmUVqUhiufHFQHdJnYCV\nEVGVHc+tKyIqgdXJLxfX1oaZmRVAwX5OXtJ0oGt2iMyvB4+NiEeT2JnAvTllcgX5E24k5XP3qf4R\nqdrqqi2+lbKyspr10tJSSktL8xUzM2u2ysvLKS8v3646CpaoIuL4urZLagGcBhySFV4M9Mx63ANY\nQia57JUbj4gPJbWXVJSMqqrLZ9e1JGmrOCJWSqqtja1kJyozM9ta7of48ePHN7qONE/9HQ+8ERHZ\nSWIqMCy5Yq8E6AvMBGYBfSX1ktQaGAY8kuwzAxiarI/Mik9NHpNsn1FPG2ZmVgAFG1E1wLfYctqP\niJgj6X5gDrAJuDD5PfhKSRcD08gk30kRMTfZbQwwRdIE4J/ApCQ+CbhT0nxgOZnkVlcbZmZWAPJ7\n8LaR5PxlZtZIkoiIRn3lJ81Tf2ZmZk5UZmaWbk5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaW\nak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5UZmaWak5U\nZmaWak5UZmaWak5UZmaWaqlMVJIOlPScpH9Kminp8KxtN0iaL2m2pIOy4iMlvSlpnqTvZMUPkfRK\nsu36rHisIpIdAAAMQElEQVQHSdOS8k9KKq6vDTMza3qpTFTARGBcRBwMjEseI2kw0Cci+gEXAL9P\n4h2AnwGHA0cA47ISz38D50XEF4AvSDoxiY8BnoqIfYAZwJVJXSfla8PMzAojrYmqCqhONO2BimT9\nFOAOgIh4ASiW1BU4EZgWEasjYhUwDRgkaQ/g8xExM9n/DuDUZH0IMDlZn5w8ro7na8PMzAqgZaE7\nUItLgSclXQsI+EoS7w4syiq3OInlxiuy4ovzlAfoGhHvA0TEUkldammjuq73t/M5mZnZNihYopI0\nHcgeqQgIYCzwdeCHEfGwpNOB24DjkzLk2Sc3Tj3xOrvW0H3Kyspq1ktLSyktLa2najOz5qW8vJzy\n8vLtqkMR9b1vNz1JqyKife5jSb8HnomI+5L4XOAYYCBQGhHfS+K/B54B/pKU3y+JDwOOiYjvS3oj\n2ef9ZIrwmYjYr7Y2qkdfWX2KNB47M7M0k0RE5BsQ1Cqt56gqJB0DIOk4YH4Snwp8J4kfCaxKEsiT\nwPGSipMLK44HnoyIpcBHkgZIUrLvI1l1nZ2sn50Tz9eGmZkVQFrPUZ0P3CCpBbAe+C5ARDwuabCk\nt4B1wKgkvlLSBOBFMtN045OLKgAuBG4HdgUej4gnkvg1wP2SzgHeBYbW1YaZmRVGKqf+Pg089Wdm\n1nifpak/MzMzwInKzMxSzonKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxS\nzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnKzMxSzYnK\nzMxSLZWJStIBkp6V9LKkRyTtlrXtSknzJb0h6YSs+CBJcyW9KWl0Vry3pOclzZN0r6SWSby1pClJ\nXc9J2qu+NszMrOmlMlEBtwJXRMSBwEPAFQCSvgicAewHnATcrIwi4CbgRKA/cKakfZO6rgGujYh9\ngFXAuUn8XGBFRPQDrgcm1tXGTn6+ZmZWi7Qmqi9ExN+T9aeAbybrpwBTImJzRCwA5gMDkmV+RCyM\niE3AFGBIss+xwIPJ+mTg1GR9SPIY4I9JubraMDOzAkhronpN0jeS9TOAHsl6d2BRVrmKJJYbXwx0\nl9QJWBkRVdnx3LoiohJYLaljHW2YmVkBtCxUw5KmA12zQ0AAY4FzgBsl/QyYCmzMKpMryJ9wIymf\nu0/UU1dt8a2UlZXVrJeWllJaWpqvmJlZs1VeXk55efl21VGwRBURx9dT5EQASf2Ak5PYYqBnVpke\nwBIyyWWv3HhEfCipvaSiZFRVXT67riWSWgDFEbFSUm1tbCU7UZmZ2dZyP8SPHz++0XWkcupP0u7J\nv0XAT4HfJ5umAsOSK/ZKgL7ATGAW0FdSL0mtgWHAI8k+M4ChyfrIrPjU5DHJ9hn1tGFmZgVQsBFV\nPc6UdBGZKbf/jYjbASJijqT7gTnAJuDCiAigUtLFwDQyyXdSRMxN6hoDTJE0AfgnMCmJTwLulDQf\nWE4mudXVhpmZFYD8HrxtJDl/mZk1kiQiolFf+Unl1J+ZmVk1JyozM0s1JyozM0s1JyozM0s1Jyoz\nM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1JyozM0s1\nJyozM0s1JyozM0s1JyozM0u1tP4U/WfOO+8s5Kqrbqeiooru3YuYMOFsSkp67fB9mrKttPevKdtK\ne/+asq20968p23L/dpCIKMgCnA68BlQCh+RsuxKYD7wBnJAVHwTMBd4ERmfFewPPA/OAe4GWSbw1\nMCWp6zlgr21tI0//o6HefntB9Onzo4C1ARGwNvr0+VG8/faCHbpPU7aV9v75WPhYFLot9y+/5L2z\ncfmisTvsqAXYB+gHzMhOVMB+wD/JjPZ6A28BIjNN+RbQC2gFzAb2Tfa5DxiarP83cEGy/n3g5mT9\nW8CUZP2LjW0jT/8b9EeJiBg+vCzrjxo1f9zhw8t26D5N2daW+zyTuv75WPhYFLqttB+Lpjx+2bYl\nURXsHFVEzIuI+UmCyDaETELZHBELyIx6BiTL/IhYGBGbyIyUhiT7HAs8mKxPBk7Nqmtysv7HpBzA\nKdvQxjarqKgC2uZE27JkSdUO3acp29pyn/LU9a8p2/KxqG2f8tT1rynbSvuxaMrjt73SeDFFd2BR\n1uOKJJYbXwx0l9QJWBkRVdnx3LoiohJYLaljY9vY7ifUvQhYlxNdx5571n74t2Wfpmwr7f1ryrbS\n3r+mbCvt/WvKtty/HaixQ7DGLMB04JWs5dXk329klXmGLaf+bgK+nfX4VuA/yJzT+n9Z8bOA3wGd\nyYyCquM9gJeT9deAPbO2zQc6NLaNWp5bg4a5EemfP97+fcalrn8+Fj4WhW4r7cfC56gal8xyE9UY\ntrxQ4gngCOBI4Il85YBlQFGyfiTw5+x9k/UWwAfb2kaefocXL168eGn80tg8kZbL07PPU00F7pZ0\nHZlpt77ATDLTlH0l9QLeA4YlC2QuyBhK5qKKkcAjWXWNBF5Its/YhjbOzNfhiMg9t2ZmZjtBwRKV\npFOBG8lM3f1J0uyIOCki5ki6H5gDbAIuTIaLlZIuBqaRSSiTImJuUt0YYIqkCWSu5puUxCcBd0qa\nDywnSWyNbOONnXwozMysDkqmsczMzFIpjVf9pZ6kQZLmSnpT0uhC96eQJC2Q9LKkf0qaWej+NCVJ\nkyS9L+mVrFgHSdMkzZP0pKTiQvaxqdRyLMZJWizppWQZVMg+NhVJPSTNkDRH0quSfpDEm91rI8+x\nuCSJN+q14RFVI0kqInPXiuOAJcAsYFjWNGSzIult4NCIWFnovjQ1SV8D1gJ3RMQBSewaYHlETEw+\nxHSIiDGF7GdTqOVYjAPWRMRvC9q5JiZpD2CPiJgtaTfgH2S+jzmKZvbaqONYfItGvDY8omq8nfKl\n4E+x6jt6NDsR8XcgN0Fnf8k8+8vnn2m1HAvY+gv9n3kRsTQiZifra8ncpq0HzfC1UcuxqP5uaoNf\nG83yDWY77ZQvBX+KBfCkpFmSzi90Z1KgS0S8D5n/pMDuBe5PoV0kabakW5vDVFcuSb2Bg8jci7Rr\nc35tZB2LF5JQg18bTlSNl+9TQHOeP/1KRBwGDCbzwvtaoTtkqXEz0CciDgKWAs1tCnA3Mrdu+2Ey\nmmi27xN5jkWjXhtOVI23GNgr63EPMueqmqXkkyERsQx4iMzUaHP2vqSuUDM//0GB+1MwEbEsPjkJ\nfgtweCH705QktSTzxnxnRFR/r7NZvjbyHYvGvjacqBpvFsmXgiW1JvPdrKkF7lNBSGqTfFJCUlvg\nBDK3rWpOxNZfWD87Wc/+8nlzsMWxSN6Mq51G83pt3AbMiYjfZcWa62tjq2PR2NeGr/rbBsmllL/j\nky8F/2eBu1QQkkrIjKKCzJfH725Ox0LSPUAp0Al4HxgHPAw8APQE3iXz8zOrCtXHplLLsRhI5pxE\nFbCAzM/vvF+gLjYZSV8F/krm3qbVtw36CZm739xPM3pt1HEsvk0jXhtOVGZmlmqe+jMzs1RzojIz\ns1RzojIzs1RzojIzs1RzojIzs1RzojIzs1RzojJrhiQdI+nRQvfDrCGcqMyaL3+J0j4VnKjMUkzS\ncEkvJD8u99+SiiStkfRbSa9Jmi6pU1L2IEnPJXekfrD6jtSS+iTlZkt6MbmjCMDnJT0g6Q1Jdxbs\nSZrVw4nKLKUk7UvmB+a+EhGHkLndzHCgDTAzIvYnc3uacckuk4EfJ3ekfi0rfjdwYxL/CvBeEj8I\n+AHwRaCPpK/s/Gdl1ngtC90BM6vVccAhwCxJAnYlcx+9KjL3jAO4C3hQUjugOPkBQ8gkrfuTmwZ3\nj4ipABGxESBTHTMj4r3k8WygN/BsEzwvs0ZxojJLLwGTI2LsFkHpqpxykVU+Xx212ZC1XonfDyyl\nPPVnll5PA6dL2h1AUgdJewEtgNOTMsOBv0fER8CK5G7VACOAv0TEGmCRpCFJHa0lfa5Jn4XZdvIn\nKLOUiog3JP0UmCapCNgIXAysAwYkI6v3yZzHgsxvHP1PkojeBkYl8RHA/5P086SOofma23nPxGz7\n+Gc+zD5lJK2JiM8Xuh9mTcVTf2afPv50ac2KR1RmZpZqHlGZmVmqOVGZmVmqOVGZmVmqOVGZmVmq\nOVGZmVmqOVGZmVmq/f8TBfwcp3ZQPwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1044473c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plotting\n",
    "plt.plot(range(len(loss_history)), loss_history, 'o', label='Logistic Regression Training phase')\n",
    "plt.ylabel('cost')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}