{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use our most advanced technologies as metaphors for the brain: The industrial revolution inspired descriptions of the brain as mechanical. The telephone inspired descriptions of the brain as a telephone switchboard. The computer inspired descriptions of the brain as a computer. Recently, we have reached a point where our most advanced technologies - such as AI (e.g., [Alpha Go](https://en.wikipedia.org/wiki/AlphaGo)), and our current understanding of the brain inform each other in an awesome synergy. Neural networks exemplify this synergy. Neural networks offer a relatively advanced description of the brain and are the software underlying some of our most advanced technology. As our understanding of the brain increases, neural networks become more sophisticated. As our understanding of neural networks increases, our understanding of the brain becomes more sophisticated. \n",
    "\n",
    "With the recent success of neural networks, I thought it would be useful to write a few posts describing the basics of neural networks. \n",
    "\n",
    "First, what are [neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network) - neural networks are a family of machine learning algorithms that can learn data's underlying structure. Neural networks are composed of many *neurons* that perform simple computations. By performing many simple computations, neural networks can answer even the most complicated problems. \n",
    "\n",
    "Lets get started.\n",
    "\n",
    "As usual, I will post this code as a jupyter notebook on [my github](https://github.com/dvatterott/jupyter_notebooks). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np #import important libraries. \n",
    "from matplotlib import pyplot as plt\n",
    "import pandas as pd\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When talking about neural networks, it's nice to visualize the network with a figure. For drawing the neural networks, I forked a [repository from miloharper](https://github.com/miloharper/visualise-neural-network) and made some changes so that this repository could be imported into python and so that I could label the network. [Here](https://github.com/dvatterott/visualise_neural_network) is my forked repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcwAAAGCCAYAAABtifOaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYVHX/PvB7YHAHFUkFUXPHUFQGFTQVFXBlSk0oH1ds\nAfrW06/VejItl/ZFyyXxyUC/omQuYIJLkjsKIyKaC5aCBgSCuygzcH5/FHw1RQeYOZ85M/frurye\nJ0bmc8N1PPec95xzRiVJEoiIiOjB7EQHICIiUgIWJhERkRFYmEREREZgYRIRERmBhUlERGQEFiYR\nEZER1A96UKVS8ZoTIiKyOZIkqf75tQcW5t/fZJ40REREFkiluqcrAXAkS0REZBQWJhERkRFYmERE\nREZgYRIRERmBhUlERGQEFiYREZERWJhERERGYGESEREZgYVJRERkBBYmERGREViYRERERmBhEhER\nGYGFSUREZAQWJhERkRFYmEREREZ46OdhEpE8bt68iVu3bkGv16OsrAxqtRpqtRoNGzZE3bp1Rccj\nsnksTCKZlJaW4vjx49DpdDhx4gTy8vKQm5uLvLw85OXlQa/Xo379+lCr1bC3t4fBYIDBYMCNGzfg\n6OgIV1dXuLq6ws3NDW5ubujevTs0Gg06deoEOzsOi4jMTSVJUtUPqlTSgx4noqoVFhZiy5YtOHDg\nAHQ6HY4fP4727dtDo9HA09MTrVq1uqsEnZyc7vtJ7+Xl5SguLq4s19zcXFy4cAEZGRnQ6XQoKipC\nr169oNFoMHDgQAQGBqJhw4YCfmIi66BSqSBJ0j3/GFmYRCYiSRJOnjyJ+Ph4JCQkIDMzEwEBARg0\naBA0Gg169uxpliIrKirC4cOHkZaWhp07d+LgwYMYMGAAtFotgoOD4ebmZvI1iawZC5PITIqKirBi\nxQpERUXh5s2b0Gq10Gq18Pf3F/Le45UrV5CUlIT4+HgkJibCw8MD4eHhCAkJQb169WTPQ6Q0LEwi\nE0tNTcWiRYuwceNGaLVaREREwNfX975jVVH0ej0SExOxZMkS6HQ6TJ06FeHh4Wjfvr3oaEQWq6rC\n5JkCRNUgSRKSkpLQp08fhISE4LHHHsOZM2cQExMDPz8/iypLAHBwcIBWq0ViYiL2798PSZLQt29f\naLVaZGRkiI5HpCg8wiQyUkpKCmbMmIH8/HzMnTsXY8aMgb29vehY1VZSUoKoqCjMnz8fAQEBmDNn\nDtq1ayc6FpHF4BEmUQ2dPHkSY8eOxfjx4zFx4kQcO3YMTz31lCLLEgDq16+Pl19+GVlZWejUqRN8\nfHzw8ssvo6CgQHQ0IovGwiSqgl6vx9y5czFgwAD4+fnh9OnTePbZZ6FWW8fly46Ojpg1axZOnDgB\nlUqF7t27IzY2FpwqEd0fR7JE95GZmYmpU6fCxcUFUVFRaNOmjehIZpeamoqpU6eiS5cuWLJkCVq0\naCE6EpEQHMkSGcFgMGDevHkYMmQIIiIikJSUZBNlCQC9e/eGTqdDly5d4OXlhdjYWNGRiCwKjzCJ\n/lZUVISQkBCoVCp89913NlOU95OamopJkyZhyJAhWLBgARwcHERHIpINjzCJHuD48ePo27cvNBoN\ntm7datNlCfx1tHno0CHk5OQgKCgIFy9eFB2JSDgWJtm8+Ph4DB48GLNmzcInn3yi2LNfTc3JyQmb\nNm2Cr68v+vTpg6NHj4qORCQUR7Jk0z7//HN8+eWXWL9+Pfr06SM6jsVavXo1/v3vfyM6OhojR44U\nHYfIrHhrPKI7SJKEDz74ALGxsdixYwfc3d1FR7J4KSkpeOKJJ7B06VKMGTNGdBwis6mqMK3jgjKi\napAkCTNnzsSmTZuwa9cuXj5hJF9fXyQmJmLkyJEwGAwYP3686EhEsmJhks2ZO3cuNm7ciOTkZDzy\nyCOi4yiKt7c3tm7dimHDhqFu3brQarWiIxHJhif9kE1ZuHAhVq1ahR07drAsa6hHjx7YvHkznn32\nWSQnJ4uOQyQbvodJNmPr1q2YOnUqUlJS0LZtW9FxFG/nzp2YMGEC9u/fz48LI6vC6zDJpp0+fRqT\nJ09GXFwcy9JEhgwZgpkzZ0Kr1eLatWui4xCZHY8wyepduXIFffv2xauvvornn39edByrIkkSXnjh\nBRQUFGD9+vWws+NrcFI+XlZCNqm8vBzBwcFo164dvvnmG9FxrFJpaSmGDh2KwYMH44MPPhAdh6jW\neFkJ2aSlS5eiqKgIGzduFB3FatWpUwfr1q1Dz549MWLECPj5+YmORGQWPMIkq3X27Fn07t0be/fu\nhYeHh+g4Vm/dunV49913kZ6ejvr164uOQ1RjHMmSTSkvL8fQoUMxcuRIvPHGG6Lj2IzQ0FC0adMG\nn376qegoRDXGwiSbsnjxYqxcuRJ79+7lzdRlVFhYCC8vL6xfv56jWVIsFibZjLy8PHTv3p2jWEHW\nrVuHWbNmISMjA2o1T5Mg5eF1mGQz5syZgylTprAsBRk3bhxcXFwQExMjOgqRSfEIk6zKmTNn4Ovr\ni1OnTqFZs2ai49isAwcOIDQ0FKdOneIJQKQ4PMIkmzBz5ky88sorLEvB/Pz84O3tjcWLF4uOQmQy\nPMIkq3H48GGMHj0aWVlZaNiwoeg4Nu/XX3+Fv78/srKy0LhxY9FxiIzGI0yyevPmzcOMGTNYlhbi\nsccew4gRI3iUSVaDR5hkFS5cuAAvLy9kZ2fD0dFRdBz62+HDhzFmzBj8/vvvvLyHFINHmGTVoqKi\nMGHCBJalhfH29oarqyu2bNkiOgpRrfEIkxRPr9ejbdu22L59Ozw9PUXHoX+IiYlBbGwsEhMTRUch\nMgqPMMlqbdy4EZ07d2ZZWqiQkBCkpaXht99+Ex2FqFZYmKR43333HV544QXRMagK9erVw+TJkxEd\nHS06ClGtcCRLinbt2jW0atUKf/zxB9+/tGD79u3Diy++iCNHjoiOQvRQHMmSVdq2bRv8/PxYlhbO\n19cXubm5yM7OFh2FqMZYmKRoCQkJ0Gq1omPQQ9jb22PUqFFISEgQHYWoxliYpFhlZWX46aefEBwc\nLDoKGUGr1SI+Pl50DKIaY2GSYh06dAhubm5o06aN6ChkhMDAQBw4cAA3btwQHYWoRliYpFiHDh1C\nv379RMcgIzVq1AgeHh7IyMgQHYWoRliYpFg6nQ4ajUZ0DKoGjUaDtLQ00TGIaoSFSYql0+ng4+Mj\nOgZVg4+PD3Q6negYRDXCwiRFun79Os6dO8e7+yiMRqNhYZJisTBJkTIyMuDp6QkHBwfRUagaPD09\ncfbsWZ74Q4rEwiRFOnPmDLp06SI6BlVTnTp10KZNG5w7d050FKJqY2GSIuXl5cHNzU10DKoBNzc3\n5OXliY5BVG0sTFKk3NxcuLq6io5BNeDq6orc3FzRMYiqjYVJipSXl2eRhfn999/Dy8sLDRs2hJub\nGyIjI3HlyhWjvrddu3bYuXOnybKY+vlMxdXVlUeYpEgsTFIkSxzJfv7553j77bfx+eef4+rVq0hJ\nSUF2djYCAwNhMBhEx7MYHMmSUrEwSZEKCgrQvHlz0TEqXbt2DbNnz8Y333yDwMBA2Nvbo02bNoiL\ni0N2djZWrlyJadOm4b333qv8nl27dqF169YAgMmTJyMnJwfBwcFwcnLCZ599huzsbNjZ2SEqKgqt\nWrVCq1at8Pnnn1d+f3Wfz1I0b94cBQUFomMQVZtadACimigtLUXdunVFx6i0f/9+3L59G2PGjLnr\n6w0bNsSIESOwfft21KtX757vU6n++si9mJgY7NmzB9999x0GDx4MAJUfhfXLL7/gt99+w5kzZzBk\nyBD06tULQ4YMuW+OBz2fpahbty70er3oGETVxiNMUiSDwQC12nJe7128eBEuLi6ws7v3n5SrqyuK\nioqMep77fWD77NmzUa9ePXTr1g3Tpk1DbGys0bks8QPg1Wo1C5MUiYVJimRpReDi4oKLFy+ivLz8\nnsfy8vLg4uJSo+dVqVRwd3ev/O+2bdsq/gxTlUp1398TkaVjYZIiqdVqlJWViY5Ryc/PD3Xr1sX6\n9evv+vr169eRmJiIoUOHokGDBrh582blY/888aVinHonSZJw/vz5yv/OycmpPNmpYcOG1X4+S2Aw\nGHiHJlIkFiYpkqWN9ZycnPDee+/hpZdewtatW2EwGHDu3DmEhoaiTZs2mDRpEnr27IktW7bg0qVL\nyM/Px4IFC+56jpYtW+L333+/57nnzJmDkpISHD9+HCtWrMDTTz8NADV+PtEsbZxOZCwWJimSk5OT\n0dc3yuWNN97A/Pnz8frrr6Nx48bw8/ND27ZtsWPHDjg4OGDSpEnw8vLCo48+iuHDh1cWX4UZM2Zg\nzpw5cHZ2xhdffFH59UGDBqFjx44IDAzEm2++iaFDhwJAjZ9PtMuXL8PJyUl0DKJqUz3ovSCVSiVZ\n2ntFRAAwatQoREREYPTo0aKjmE12djbat28PvV5/35OJlGr27NmQJAnvv/++6ChE96VSqSBJ0j3v\naVjPv0KyKbZyezVrfMHK2xqSUrEwSZFs5W4xlnriTm1Y4l2aiIzBwiRFsoUjzLZt26KsrMyqxrEA\njzBJuazrXyLZjNatW1feCYeUQ5Ik5OTk3HVtKZFSsDBJkXr06IH09HSrfI/Pml24cAF2dnZo2bKl\n6ChE1cbCJEVyd3eHJEn4448/REehatDpdNBoNFb53ixZPxYmKZJKpYJGo4FOpxMdhapBp9PBx8dH\ndAyiGmFhkmKxMJWn4giTSIlYmKRYPj4+SE1NFR2DjCRJEtLS0liYpFgsTFKsgQMHYv/+/SgpKREd\nhYyQnp6Opk2b8gxZUiwWJimWs7MzevXqhZ9//ll0FDJCfHw8tFqt6BhENcbCJEXTarWIj48XHYOM\nwMIkpePN10nRsrKyMGjQoMrr+8gynT9/Hr169UJ+fj4/2ossHm++TlapU6dOaNy4MdLS0kRHoQeI\nj4/HyJEjWZakaCxMUrynn34a0dHRomPQA0RHRyM0NFR0DKJa4UiWFC83NxfdunXDuXPn+MHEFig1\nNRUhISE4c+YM7O3tRccheiiOZMlqubm5YejQoVi1apXoKHQfS5YsQXh4OMuSFI9HmGQVkpOT8dJL\nLyEzM5P3KbUgxcXF6NChA06fPo1HHnlEdBwio/AIk6yav78/ysrKsHv3btFR6A4rVqzA6NGjWZZk\nFXiESVYjKioKa9aswY4dO3iUaQFu3LiBTp06ISEhgbfDI0XhESZZvalTp+L8+fPYsWOH6CgEYOHC\nhXj88cdZlmQ1eIRJVuWHH37ARx99hNTUVN7IQKDi4mJ07twZ+/fvR+fOnUXHIaoWHmGSTRg3bhxU\nKhXWrVsnOopN++ijjzBu3DiWJVkVHmGS1dmxYwciIiKQmZmJevXqiY5jc7Kzs+Ht7Y3MzEy4ubmJ\njkNUbTzCJJsREBAALy8vfPDBB6Kj2BxJkvDcc8/htddeY1mS1eERJlmlP//8E15eXti8eTN69+4t\nOo7NiIqKwrJly3DgwAHeN5YUq6ojTBYmWa3Y2FjMnTsXOp2Oo1kZ5OTkQKPRIDk5Gd26dRMdh6jG\nWJhkcyRJwrhx4+Dh4YH58+eLjmPVJEnCsGHD4O/vj3feeUd0HKJaYWGSTfrzzz/Rs2dPxMTEIDAw\nUHQcq/XZZ58hLi4O+/fv5yiWFI+FSTZr165dCAkJwb59+9CxY0fRcaxOYmIipk+fjoMHD6J169ai\n4xDVGguTbNrSpUuxcOFCpKSk8CPATOjkyZMYOHAgNm7ciH79+omOQ2QSLEyyeZGRkcjJycGmTZv4\nUVMmcOnSJfj6+uKtt95CWFiY6DhEJsPCJJun1+sRFBSEHj164Msvv+QN2mvh1q1bGD16NLp164av\nvvpKdBwik2JhEuGvo6IhQ4Zg5MiRmDdvnug4ilRaWoqxY8eiUaNGWLVqFU/yIatTVWFySyeb0rRp\nU2zfvh3+/v5Qq9WYPXs2jzSrobS0FKGhoXBwcMDKlStZlmRTeGs8sjkuLi7YuXMnNmzYgLfffhuc\nohjn1q1bGDNmDFQqFdauXQsHBwfRkYhkxcIkm9S8eXMkJydjx44deOGFF1BaWio6kkW7ePEihg0b\nBicnJ6xduxZ16tQRHYlIdixMslnNmjVDcnIyCgoKEBAQgIKCAtGRLNLRo0fRp08f9OvXD6tWreKR\nJdksFibZNEdHR6xfvx6DBg1Cnz59cOTIEdGRLMqGDRswdOhQzJs3Dx9++CEvxyGbxrNkif4WFxeH\nF198EV9//TWefvpp0XGEKisrw9y5c7F8+XJs2LABPj4+oiMRyYaXlRAZIT09Hc888wx69OiBhQsX\nokWLFqIjye748eOYPn06GjZsiFWrVsHV1VV0JCJZ8QOkiYzQq1cvpKeno3HjxnB3d8eaNWtER5JN\nWVkZZsyYgZ49e2LChAnYvn07y5LoDixMon+oV68efv/9dzz33HOYNWsWQkNDkZubKzqWWR09ehT9\n+/dHamoqAgMD8ccff8DOjrsHojvxXwTRP3z77be4du0aFi5ciCNHjqB9+/bo3r073n77bVy+fFl0\nPJM6e/YsJk2ahKCgIEybNg3bt2/H999/j+joaKSkpIiOR2RRWJhEdzh37hxmzpyJFStWQK1Wo379\n+vjwww+RkZGBwsJCdO7cGZ9++ilKSkpER62VgoIC/Pvf/4aPjw86dOiArKwsvPDCC7Czs0Pz5s2x\ncOFCTJs2Dbdu3RIdlchisDCJ/iZJEp599lm8/vrreOyxx+56zN3dHcuXL8euXbuQkpKCTp06Yf78\n+fjzzz8Fpa2ZU6dO4ZVXXqn8+U6cOIHZs2fD0dHxrr8XEhKCbt26YdasWSJiElkkFibR3ypGsa+9\n9lqVf6dr16748ccfkZCQgLNnz8LDwwMTJkzAnj17LPYWewaDAevXr0dAQAAGDRqEhg0b4vDhw1iw\nYAGaN29e5fctWrSIo1miO/CyEiL8NYrt3bs3du3adc/R5YNcvnwZ0dHRWLx4MerUqYPx48dDq9Wi\nR48eQm/qbjAYsHfvXsTHxyMuLg7t2rVDZGQkxo4di7p16xr9PHFxcZg1axbS09NRr149MyYmshy8\nDpOoCpIkITAwEIGBgXjrrbdq/Bx79uzBxo0bER8fj9LSUgQHB0Or1aJ///5o1KiRiVPfq6ioCD//\n/DPi4+ORmJiIdu3aQavVYuzYsejWrVuNn3f8+PFo3749Pv74YxOmJbJcLEyiKixduhQrVqzAvn37\nTPJxVZIk4cSJE0hISEBCQgIOHz6MRx99FBqNBj4+PtBoNPD09ESTJk1qdBRaXl6OwsJCZGRkIC0t\nDTqdDjqdDpcuXUL//v2h1WoxevRouLu71/pnAf46QcjLywsbN26Er6+vSZ6TyJKxMInuo6aj2Ooo\nLS3F8ePHK4tNp9Ph5MmTMBgMcHV1haurK9zc3NCyZUvUr18farUa9vb2MBgMMBgMuH79OvLy8pCb\nm4u8vDzk5+fDyckJ3bp1u6uEO3bsaLZrJzmaJVvCwiT6B1OMYmvjfkV4+/Zt6PV6lJWVQa1WQ61W\no0GDBnBzc4ObmxtcXV3RsmVLIaXF0SzZChYm0T+YehRr7TiaJVvBwiS6gxyjWGvE0SzZAhYm0d9E\nj2KVjqNZsnYsTKK/cRRbOxzNkrVjYRKBo1hT4WiWrBkLk2weR7GmxdEsWSsWJtk8jmJNi6NZslYs\nTLJpHMWaB0ezZI1YmGSzOIo1L45mydqwMMlmcRRrXhzNkrVhYZJN4ihWHhzNkjVhYZLN4ShWXhzN\nkrVgYZLN4ShWXhzNkrVgYZJN4ShWDI5myRqwMMlmcBQrFkezpHQsTLIZHMWKxdEsKR0Lk2wCR7GW\ngaNZUjIWJlk9jmItC0ezpFQsTLJ6HMVaFo5mSalYmGTVOIq1TBzNkhKxMMlqcRRr2TiaJaVhYZLV\n4ijWsnE0S0rDwiSrxFGsMnA0S0rCwiSrw1GssnA0S0rBwiSrw1GssnA0S0rBwiSrwlGsMnE0S0rA\nwiSrwVGssnE0S5aOhUlWg6NYZeNoliwdC5OsAkex1oGjWbJkLExSPI5irQtHs2SpWJikeBzFWheO\nZslSsTBJ0TiKtU4czZIlYmGSYnEUa904miVLw8IkxeIo1rpxNEuWhoVJisRRrG3gaJYsCQuTFIej\nWNvC0SxZChYmKQ5HsbaFo1myFCxMUhSOYm0TR7NkCViYpBgcxdo2jmZJNBYmKQZHsbaNo1kSjYVJ\nisBRLAEczZJYLEyyeBzF0p04miVRWJhk8TiKpTtxNEuisDDJonEUS/fD0SyJwMIki8VRLD0IR7Mk\nt6oK005EGLJuERERmDdvnlF/d9q0aRg9ejSuXbuG1157zczJSIkaNmyIRYsWISUl5aF/d9q0aXjv\nvfdkSEW2iIVJ+OijjzBy5Mi7vtapUyeMGjXqrq917twZcXFxD32+JUuW4D//+Y9Ra1+/fh3JyclY\nsWLFfd+3tLOzw++///7Q5/nll19gZ2eHTz/91Kh1yXxMvT19//33+O677zBt2jTcunWrVtketj1F\nR0dDrVbDyckJTk5O6NixI5YuXVqrNcl6sDAJAwcOxIEDB1Axfs/Pz4fBYEB6evpdX/vtt98wcOBA\nk60rSRL27dsHPz+/Kt+3VKnumYrcV0xMDJo1a4aYmBiT5aOaMcf2FBISgm7dumHWrFm1ymbM9tSv\nXz9cvXoVV69exbp16/Dmm28iIyOjVuuSdWBhEnr37o3S0lIcOXIEALBnzx4MHjwYXbp0uetrHTp0\nQMuWLQEAJ0+eRFBQEJo1a4auXbvihx9+qHy+f47FPvnkE7i5ucHd3R3//e9/K1/lf/vtt9Dr9ejS\npQtGjx4NJycn+Pn54ezZswCAQYMGQZIkeHl5wcnJ6a417nTz5k2sW7cOixYtQlZWFg4fPmyW3xMZ\nx1zb06JFixAdHY3/+Z//ue/2VKG4uLhW29Odevbsia5du+LEiRMm+/2QcrEwCQ4ODujbty92794N\nANi9ezcGDhyIxx9//J6vAX8VVFBQECZOnIiLFy9izZo1iIyMxMmTJ+957qSkJHz11VfYuXMnzpw5\ng19++QUqlQoXLlzAzJkz8fjjj+OHH37A+++/j8uXL6NDhw6V49xdu3YBADIzM3H16lWMHz/+vvl/\n/PFHODo6Yvz48QgKCkJ0dLTJf0dkPHNtT82bN0dYWBi+/fZbJCYm3rU93Wnt2rW12p7ulJqaiqys\nLPj4+NTul0JWgYVJAP569V2xM9uzZw8GDBhw1w5uz549GDRoEABg8+bNaNeuHSZPngyVSoUePXpg\n3Lhx933F/sMPP2DatGnw8PBAvXr1MHv2bADA22+/jddffx1NmjTBmDFjoNFoYGdnh3/961+VRyEV\nHnamdkxMDJ5++mmoVCpMmDABa9asQVlZWW1/JVQL5tqe/vzzT3Ts2BGrV6++a3u6U223pwMHDsDZ\n2RlOTk7w9fXFpEmT0LFjx5r8GsjKsDAJwF/vO+3duxeXLl3CxYsX0aFDB/Tr1w/79+/HpUuXcOzY\nscojguzsbKSkpMDZ2RnOzs5o2rQpVq9ejT///POe583NzUXr1q0r/7t169aQJAnXr1+vPCu2YiwH\nAA0aNMD169eNzn3hwgUkJydjwoQJAACtVouSkhL89NNPNfo9kGmYc3sKCwtDdHQ0UlJSKrenO9Vm\newIAPz8/FBcX4+rVq8jPz8exY8eMPomNrBsLkwD8tZO4fPkyoqKi0L9/fwCAo6Mj3NzcEBUVhVat\nWqFt27YA/io9f39/FBcXo7i4GJcuXcLVq1fxzTff3PO8rq6uuHDhQuV/79+/H5Ik4ZNPPjHJ3Xxi\nYmIgSRKCg4Ph6uqKDh064Pbt2xzLCmbO7enKlStYuHAhpk2bhqysLKNPDKuJRx55BOPGjUNCQoLZ\n1iDlYGESAKBevXrw8fHBF198gQEDBlR+vX///vjiiy/uOptx9OjROH36NFatWgWDwQC9Xo+0tDSc\nOnXqnucNCQnBihUrcPLkSdy4cQMTJkyASqVCp06djMrVsmXLB14GEBMTg9mzZ+PIkSPIyMhARkYG\n1q1bh59++gmXLl2qxm+ATMnc25OXlxc8PDzw9NNPVyvXw7Yn4O6RbVFRETZs2IBu3bpVax2yTixM\nqjRo0CAUFhbi8ccfr/zagAEDUFhYWPl+EwA0atQI27Ztw5o1a+Dm5gY3NzfMmDEDt2/fvuc5hw8f\njpdffhmDBw+Gu7s76tatC5VKhbp16xqVafbs2Zg8eTKcnZ2xbt26ux47ePAgcnJyEBkZiebNm1f+\nCQ4ORqdOnRAbG1vD3wSZgrm3p4MHDyI7OxsATLI9VUhJSam8DtPT0xMtWrTAwoULq/Ojk5XirfFI\nFhX3il2xYgXGjBmD27dvw86Or9eodr788ku8+uqruHHjBho0aCA6DlkJ3hqPhNmwYQPCwsIQGRmJ\nqKgoaLValiXV2MaNG1FaWopLly7hl19+QatWrfD++++LjkU2gHstMrt3330Xu3fvxqJFi+Dg4IDF\nixeLjkQK9u2336J58+bo1KkTHBwckJSUVHnWLJE5cSRLZsWP7SI58GPAyJT48V4kO35sF8mJHwNG\npsLCJNktXboUK1aswL59+0xyzSXRgxQUFMDLywsbN26Er6+v6DikYCxMkhVHsSQCR7NkCixMkg1H\nsSQSR7NUWyxMkg1HsSQSR7NUWyxMkgVHsWQJOJql2mBhktlxFEuWhKNZqikWJpkdR7FkSTiapZpi\nYZJZcRRLloijWaoJFiaZDUexZMk4mqXqYmGS2XAUS5aMo1mqLhYmmQVHsaQEHM1SdbAwyeQ4iiUl\n4WiWjMXCJJPjKJaUhKNZMhYLk0yKo1hSIo5myRgsTDIZjmJJyTiapYdhYZLJcBRLSsbRLD0MC5NM\ngqNYsgYczdKDsDCp1jiKJWvC0SxVhYVJtcZRLFkTjmapKixMqhWOYskacTRL98PCpBrjKJasGUez\n9E8sTKoxjmLJmnE0S//EwqQa4SiWbAFHs3QnFiZVG0exZEs4mqUKLEyqNo5iyZZwNEsVWJhULRzF\nki3iaJbH1DMOAAAgAElEQVQAFiZVA0exZMs4miUWJhmNo1iyZRzNEguTjMJRLBFHs7aOhUkPxVEs\n0f/haNZ2sTDpoTiKJfo/HM3aLhYmPRBHsUT34mjWNrEwqUocxRJVjaNZ28PCpCpxFEtUNY5mbQ8L\nk+6Lo1iih+No1rawMOkeHMUSGY+jWdvBwqR7cBRLZDyOZm0HC5PuwlEsUfVxNGsbWJhUiaNYoprj\naNb6sTCpEkexRDXH0az1Y2ESAI5iiUyBo1nrxsIkjmKJTIijWevFwiSOYolMiKNZ68XCtHEcxRKZ\nHkez1omFacM4iiUyH45mrQ8L04ZxFEtkPhzNWh8Wpo3iKJbI/DiatS4sTBvEUSyRfDiatR4sTBvE\nUSyRfDiatR4sTBvDUSyR/DiatQ4szBooLy9HVlYWMjMzkZubi9zcXOTl5VX+uXHjBgwGAwwGA+zt\n7aFWq1GvXj20bNkSrq6ucHV1hZubG1xdXdG1a1d4enqiTp06Zs/NUSyROHKPZq9fv46MjAycOXMG\neXl5d+2nCgoKUFpaCoPBAEmSoFaroVar4eTkVLlvqvjf1q1bo0ePHnB3d4dKdU9X2BQWphHOnz+P\n3bt3Q6fTQafTIT09HS4uLvDy8oK7u/s9G1ijRo2gVqthb2+P8vJyGAwGlJSUID8/v3KjrSja48eP\n4/fff4enpyc0Gg00Gg369euHxx57zOQbJ0exROKYczRbVlaGQ4cO4dChQ5X7qXPnzsHT0xNdunSp\n3DdV7KeaN2+OunXrVu4HysrKoNfrceXKlbvKNTc3F9nZ2UhPT4ckSZX7KB8fHwwcOBDOzs4m/Tks\nHQvzPsrLy3H48GHEx8cjISEBFy5cwKBBg+Dj4wONRgNvb280a9bMZOvduHEDR44cqdzQd+3aBTs7\nOwQHB0Or1WLgwIFwcHCo1RocxRKJZ8rR7LVr17Bt2zbEx8djy5YtcHNzQ79+/SoLzdPTs9b7jQqS\nJOGPP/6o3EelpqZi37590Gg0lfupjh07mmQtS8bCvMOxY8fw7bffYv369XB0dIRWq4VWq4Wfnx/s\n7e1lyyFJEo4dO1ZZ2KdOncLIkSPx/PPPY+DAgdU+8uQolshy1GY0q9frsXHjRnz33XfYt28f/Pz8\noNVqERwcjDZt2pghbdVKSkrw888/V+6nmjRpgqeffhrPPfcc3NzcZM0il6oKE5IkVfnnr4etw+3b\nt6U1a9ZIAwcOlNzc3KT33ntPOnXqlOhYd8nNzZW+/vpr6bHHHpMee+wx6euvv5YuX75s9PcvWbJE\n6tOnj6TX682YkoiM8eeff0otWrSQDhw4YPT3nD9/XnrvvfckV1dXyd/fX1q9erV05coVM6asnrKy\nMungwYNSZGSk1LRpU+mpp56Sdu7cKZWXl4uOZlJ/d9+9nXi/L0pWVJhXrlyRZs+eLbVs2VIaPHiw\n9MMPP0ilpaWiYz1QeXm59Msvv0ghISFSkyZNpPDwcCknJ+eB33P27FnJxcVFOn78uEwpiehh1q5d\nK3l4eEglJSUP/Hs6nU4aO3as1LRpU+nFF1+Ujh07JlPCmrty5Yq0aNEiydPTU/Lw8JCWLVtm8ftW\nY9lcYZaUlEhffPGF1Lx5c2ny5MmKLZLc3FxpxowZkrOzs/Taa69JFy9evOfvlJeXS0OHDpU++ugj\nAQmJ6EGeeuop6c0337zvY1lZWVJoaKjk6uoqLViwQLp69arM6Wqv4gX+kCFDpM6dO0txcXGKP+K0\nmcI0GAzSihUrpDZt2kjBwcFSZmam6EgmkZubK4WHh0vNmjWT5s6dK12/fr3yMY5iiSzX/Uaz//z3\nfO3aNYEJTaO8vFzatm2b5O3tLWk0Gmn79u2iI9WYTRRmZmampNFopP79+0t79uwRHccsTp8+LYWG\nhkqtW7eWEhMTOYolUoCK0eyNGzekRYsWSc2aNZNeffVVqbCwUHQ0kysrK5PWrl0rdezYURo7dqyU\nn58vOlK1WXVhlpaWSnPnzpVcXFyk5cuXK34cYIwdO3ZIbdq0kVxdXaXZs2eLjkNEDzFixAipTZs2\nkq+vr3TixAnRccyupKREmjFjhtSiRQspNjZWUfvlqgpT8ZeVZGZmYurUqXBxccHy5cvRunVr0ZFk\nU1xcjODgYOTk5CAqKgrDhw8XHYmI/qG8vBxLlizBzJkzERAQgNjYWFkvXxMtNTUVU6dORZcuXbBk\nyRK0aNFCdKSHquqyEjsRYUxl+fLlGDJkCCIiIpCUlGRTZQkAzs7O2LdvH1asWIHw8HC89tprMBgM\nomMR0d+uXLmC4OBgrFy5Evv27UNcXJxNlSUA9O7dGzqdDh4eHujZsyd27dolOlKNKfIIU6/X47XX\nXsO2bduwadMmdOnSRXQk4YqLixEaGgp7e3vExsaiadOmoiMR2bTTp09Dq9UiICAAX375pcnuxqNk\n27dvx8SJE/HBBx/ghRdeEB2nSlZzhFlUVIThw4cjKysLKSkpLMu/OTs7IzExER4eHujbty9Onjwp\nOhKRzdq2bRsGDBiAV199Fd988w3L8m+BgYHYu3cvFixYgMjISOj1etGRqkVRhXny5En06dMH3t7e\n2Lx5M5o0aSI6kkVRq9X46quvMGPGDAwcOBBbt24VHYnI5ixcuBBTpkzBunXr8Pzzz4uOY3E6deqE\nAwcOICcnB0FBQbh06ZLoSEZTzEj26NGjGD58OObOnYuwsDDRcSze3r17MW7cOCxbtgxPPPGE6DhE\nVk+SJMydOxerVq3Ctm3b0LZtW9GRLFpZWRlee+017Nq1C9u3b4eLi4voSJUUffP19PR0jBgxAgsW\nLEBoaKjoOIqh0+kwatQofPPNN3jqqadExyGyWpIkYebMmdi4cSN27NiBli1bio6kCJIk4d1338Wm\nTZuwc+dONG/eXHQkAFUXpsV/WOKxY8cwcuRILF68GGPHjhUdR1E0Gg22bduGoKAgODg48EiTyEzm\nzJmDTZs2ITk5GY888ojoOIqhUqkwb948qNVqBAQEIDk52aQfqWhqFl2YZ86cwbBhw/DVV1+xLGvI\ny8sLP/30E0aOHIl69eph2LBhoiMRWZXPP/8cq1evxq5du1iWNTR79mzcvn0bw4YNQ3JyMhwdHUVH\nui+LHcleuXIFvr6++Pe//43w8HAhGazJ3r17MXbsWOzevRseHh6i4xBZhfj4eERGRiIlJQXu7u6i\n4yiaJEl4/vnnUVhYiPXr18POTtw5qYp6D7OsrAxarRbt2rXDN998I/v61uq7777Dxx9/jJSUFF6n\nSVRLx48fh7+/PzZv3oy+ffuKjmMVSktLMXToUPj7+2POnDnCcijqOsz//Oc/KCkpwZdffik6ilUJ\nCwvDiBEj8Mwzz6CsrEx0HCLFKioqglarxRdffMGyNKE6dergxx9/xMqVKxEXFyc6zj0s7ghz9erV\nePfdd5GammrRb/4qlcFgwIgRI9CjRw989tlnouMQKY7BYMCwYcPg7e2NTz/9VHQcq3TkyBEEBgZi\n27Zt6NWrl+zrK2Ike/bsWfTp0wc7d+5E9+7dZVvX1hQXF8Pb2xtLly7lDduJqmnevHlITk7G1q1b\nbe6+sHJas2YNZs+ejfT0dNSvX1/WtS2+MMvLyxEQEIARI0bgjTfekGVNW7Zjxw6EhYUhMzMTjRs3\nFh2HSBEyMzMxZMgQ6HQ6tGnTRnQcqxcaGoo2bdrIfiRv8YW5ePFirFy5Env37uWrNpmEh4fDYDBg\n+fLloqMQWTy9Xg9fX19ERETg2WefFR3HJhQWFsLLywvr16+Hn5+fbOtadGGePXsWvXv3xt69e3nJ\ng4yuXbuG7t27czRLZIS5c+diz549SEpKgkp1z76UzGTdunV49913ZR3NWmxhSpKEoKAgBAUFcRQr\nQMVo9sSJE2jYsKHoOEQW6eTJkxgwYABHsYKEhoaiffv2+PDDD2VZz2ILMykpCf/v//0/ZGZmQq22\n6BsPWa3Q0FD06NED77zzjugoRBZp7Nix8PPz44t6QXJzc9G9e3dkZGTIcoMIiyzM8vJyeHt74733\n3uOt7wTKysqCn58fTp06xUt5iP4hJSUF48ePx+nTp2U/W5P+z9tvv43CwkJZzrmwyMKMjY3FV199\nhZSUFL4nIFh4eDgcHR15XRnRHSRJwuDBgzFx4kSe6CPY5cuX0alTJ+zZs8fs57pYXGGWlpaia9eu\nWL58OQYPHmyWNch4FSOPI0eOoHXr1qLjEFmEpKQkvPLKKzh27BjfMrIAn3zyCQ4ePIgff/zRrOtY\n3K3xYmJi0KFDB5alhXBzc8Nzzz0n25vqRErw3nvvYe7cuSxLC/HSSy/h4MGDSE9PF7K+kCNMSZLg\n7e2Njz/+GEFBQSZ/fqqZ3NxceHp6Ijs7G05OTqLjEAmVmpqK0NBQZGVl8dpwCzJ//nycPXsWUVFR\nZlvDoo4wU1JScP36dQQEBIhYnqrg5uaGwMBArFy5UnQUIuEWL16M8PBwlqWFmT59OtatW4fLly/L\nvraQwly8eDEiIiKEft4Z3V9kZCQWL14MUZ+DSmQJioqKsHHjRoSFhYmOQv/QokULjBgxAtHR0bKv\nLXtjFRYWIiEhAVOnTpV7aTLCoEGDIEkSdu/eLToKkTDff/89goOD4eLiIjoK3UdERISQF/ayF+bK\nlSvx5JNPwtnZWe6lyQgqlQoRERFYtmyZ6ChEQkiShGXLliEiIkJ0FKrC448/DgcHB+zZs0fWdWUv\nzI0bNyIkJETuZakannrqKSQmJkKv14uOQiS7kydP4ubNm/D19RUdhaqgUqkQEhKCjRs3yrqurIVZ\nVFSEjIwMDBkyRM5lqZpcXV0rLxAmsjUJCQnQarW8mYqF02q1iI+Pl3UsK2thbtmyBUOHDkW9evXk\nXJZqoGJjJLI18fHx0Gq1omPQQ/To0QOlpaU4efKkbGvKWpjcEJUjODhY9ldvRKIVFhYiMzMT/v7+\noqPQQ6hUqsr9lFxkK0y9Xo/t27dj1KhRci1JtdC9e3eUl5fj119/FR2FSDZbtmxBQEAA6tatKzoK\nGUGr1SIhIUG29WQrzGPHjsHd3R2PPPKIXEtSLahUKgwcOBAHDhwQHYVINgcOHMCgQYNExyAj9e/f\nH+np6bKdoChbYep0Omg0GrmWIxPQaDTQ6XSiYxDJhvspZWnUqBHatm2L48ePy7KebIWZlpbGDVFh\nWJhkS0pLS3H8+HH07NlTdBSqBo1Gg7S0NFnWkvUI08fHR67lyAR69uyJ48eP83pMsgnHjx9H+/bt\n0bBhQ9FRqBp8fHxke2EvS2GWlpbi119/5Ss3hWnUqBEeffRR2cYdRCJxHKtMck7CZCnMs2fPwtXV\nFQ0aNJBjOTIhT09PnDhxQnQMIrM7ceIEPD09RcegavL09JTtWkxZCjMvLw9ubm5yLEUm5ubmhry8\nPNExiMyO+yllatKkCfR6Pa5fv272tWQpzNzcXG6ICuXm5obc3FzRMYjMjvspZVKpVLK9sJftCNPV\n1VWOpYzSrl077Ny50+zrvP/++5g8ebJRf9ff3x/Ozs4Wd4KNq6srjzDJJnA/VbVHH30UDRo0gJOT\nE5o1a4bg4GD88ccfZs9mLLn2UxzJWoDs7Gzs3bsXdnZ2Fnf/Vo5kyVZwP1U1lUqFn376CVevXkVe\nXh6aN2+Ol156SXSsSnJNwmQpzPz8fLRs2VKOpaotOjoaAwYMwBtvvAFnZ2d06NABSUlJlY8PHjwY\n77zzDvr27YvGjRtjzJgxuHz5MgBg165daN269V3PV/GqcOvWrZg/fz7Wrl0LR0dH9OrVq8oMMTEx\n8PPzw9SpU/H999+b5eesqZYtW7IwyerdvHkTer0eTk5OoqPclyXspyruK12nTh089dRTFnXbzJYt\nWyI/P9/s68hSmLdv30b9+vXlWKpGDh06hK5du6KoqAhvvPEGpk+fftfjK1euxPfff4/8/HzY29vf\n9cqqqo8AGjZsGN555x2Ehobi2rVrSE9Pr3L9mJgYTJw4ERMmTMDWrVtRWFhomh/MBOrXr4/bt2+L\njkFkVrdu3UL9+vUt+iO9RO+nKty8eRNr166Fn59f7X4gE5JrPyVLYer1eqjVajmWqpG2bdsiLCwM\nKpUKU6ZMQV5eHgoKCiofnzRpErp27Yr69etjzpw5+OGHH0z2KR579+5FTk4OQkJC4O3tjY4dO2L1\n6tUmeW5TUKvVMBgMomMQmZWl76MAsfspAHjyySfh7OyMJk2aYMeOHXj99ddN9ty1pVarZTn/Q5bC\nLCsrg52drJ8kVi13josrjoTvPEX5znFG27ZtodfrcfHiRZOsHRMTg6CgIDRt2hQA8MwzzyA6Otok\nz20KdnZ2KCsrEx2DyKzKyspgb28vOsYDidxPAcCmTZtQXFyM27dv4+uvv8bAgQPvKmyR7O3tZdlP\nydJiarVa0Tvd8+fPV/7/7OxsODg4wMXFBQ0bNsTNmzcrHysrK7trnPqw8c6tW7cQFxeHXbt2wdXV\nFa6urvjqq6+QkZGBzMxM0/8gNVBWVmbxr7yJassaJinm2k9VqDhaValUGDNmDOzt7bF3714Tpa8d\ng8Egy35KtsJU8sa4atUqnDx5Ejdv3sSsWbMwfvx4qFQqdO7cGbdu3UJiYiIMBgPmzp2L0tLSyu9r\n0aIFzp07V+VYZMOGDVCr1Thx4gQyMjKQkZGBEydOYMCAARZzlCnXhkgkktL3UYD59lP3s2nTJly+\nfBldu3Y1x49SbVZVmA0bNpTlLgzGetgrqn8+PmnSJEyZMgVubm4oLS3FggULAABOTk5YvHgxpk+f\nDnd3dzg6OsLd3b3y+8aPHw9JktCsWbP73ng+JiYGYWFhaNWqFZo3b17558UXX8Tq1atRXl5ugp+2\ndq5fv85bGpLVa9iwIW7cuGER/+YqWMp+qkJwcDCcnJzQuHFjzJw5EzExMRZTmHLtp1QPelWhUqkk\nU7xp/Pbbb8PR0RHvvPNOrZ9LboMHD8akSZMQFhYmOooQSUlJ+PLLL7F161bRUYjMqlmzZjh16hRc\nXFxER6k2W99PjRkzBhMnTsS4ceNM8nwqlQqSJN3zikWWI0xXV1feXk2hcnNzLeruJ0Tmwv2Ucsl1\nW0PZClOpF79b8nVZcrC024URmQv3U8ol135KlrM5lHwDbznu5WjJcnNz4eHhIToGkdlxP6VM5eXl\nyM/Pl6UwZTnCdHNzw4ULF+RYikzsjz/+4P01ySZwP6VMhYWFcHJyQt26dc2+liyF2bZtW1y5cgXF\nxcVyLEcmlJGRge7du4uOQWR23bt3x9GjR0XHoGrKyMhAt27dZFlLlsK0s7NDr169oNPp5FiOTKSo\nqAjFxcXo2LGj6ChEZqfRaJCWliY6BlVTWloaNBqNLGvJdr86Hx8fFqbC6HQ69OrVy6Jva0hkKp06\ndUJRURGKiopER6Fq0Ol0D7x+1JRk2xNqNBoWpsLodDrZXrkRiVYxCTt8+LDoKFQNcu6nZC1MjjuU\nhYVJtob7KWW5ePEiLl26JNvbRrIVZqdOnVBSUoLff/9driWpFgwGA3755RcMGDBAdBQi2QwYMMCm\nL9FQmp07d6J///6yvW0kW2Ha2dlh9OjRSEhIkGtJqoUDBw6gTZs293xSO5E1CwwMxMGDB3HlyhXR\nUcgI8fHx0Gq1sq0n69kcWq0W8fHxci5JNST3hkhkCRo1aoQBAwYgKSlJdBR6CL1ej8TERIwePVq2\nNWUtzICAAKSmpuLSpUtyLks1EB8fj+DgYNExiGTHF/bKsG/fPrRr1+6uT14xN1kLs0GDBvD39+er\nNwt36tQpXL9+Hd7e3qKjEMlu9OjRSEpKgl6vFx2FHiAhIUH2KZjsF9iNGzcOq1atkntZqobVq1dj\n3LhxNn9DZ7JNrVq1QpcuXZCYmCg6ClXBYDBg7dq1GDt2rKzryl6Y48ePx8GDB3H27Fm5lyYj6PV6\nREVFITw8XHQUImHCw8OxZMkS0TGoCvHx8WjXrp1st8SrIHthNmjQAFOmTMG3334r99JkhI0bN6JL\nly547LHHREchEiYkJARpaWn47bffREeh+1i8eDEiIyNlX1clSVLVD6pU0oMer6msrCz0798f58+f\nl+UO82S8wYMHIzIyEuPHjxcdhUioN998E5Ik4dNPPxUdhe5w6tQpDBo0CNnZ2WbrD5VKBUmS7nlP\nSshNQjt16oRevXph3bp1IpanKvz66684deoUnnzySdFRiIQLDw/H999/j5KSEtFR6A5Lly7F9OnT\nhRxsCbur9ksvvYSPP/4YZWVloiLQP3z00UcIDw+Hg4OD6ChEwrVv3x5+fn6IiooSHYX+VlBQgJUr\nV+KFF14Qsr6QkSwASJKE/v37IzIyEhMnTjTLGmS8zMxMBAQEICsrC05OTqLjEFmEjIwMDBs2DFlZ\nWXB0dBQdx+b9+9//hiRJWLhwoVnXqWokK6wwAWD37t2YMmUKTp48yfcyBQsODsbQoUPxyiuviI5C\nZFEmTpyITp06YdasWaKj2LSzZ8/Cx8cHJ06cQPPmzc26lkUWJgCMGjUKw4YNw8svv2zWdahqe/fu\nxcSJE3Hq1Cm+cCH6h99//x29e/eWZUdNVZs8eTLat2+P2bNnm30tiy3MipHH6dOnOQoUQJIkDBgw\nAM899xymTJkiOg6RRXrppZdgZ2eHBQsWiI5ik44ePYqgoCDZesJiCxMAnnvuOajVal4oLEBUVBSW\nLVuGlJQU2Nvbi45DZJEKCgrQvXt3bN68Gb179xYdx6YYDAb0798fYWFhsp3sY9GFeeXKFXTv3h0r\nVqzA0KFDzb4e/SUnJwcajQbJycmy3zGDSGliY2Mxd+5cHD58mG9dyOiTTz7B1q1bsX37dtk+99Ki\nCxMAkpKSEBERgaNHj/JsNBlIkoRhw4bB398f77zzjug4RBZPkiSMGzcOHh4emD9/vug4NuHEiRMY\nOHAgUlNT8eijj8q2rsUXJgBMnz4dderU4WhWBlFRUfj222+RkpICtVotOg6RIuTn56NHjx4czcqg\nYhQ7depUREREyLq2IgqzYjS7aNEifhajGZ06dQqPP/44R7FENRAbG4v3338fhw4d4omKZvT+++9j\n9+7dso5iKyiiMAEgJSUFWq0Wv/zyC28AbgaXL19G37598dZbbyEsLEx0HCJFioyMRE5ODjZt2sST\n5cxgw4YNePnll3Ho0CG4urrKvr5iChMAoqOjMWfOHBw6dAjOzs6yr2+tysrKMGrUKHTp0oWnxxPV\ngl6vR2BgIPz8/PDhhx+KjmNVMjMzMXToUGzZsgU+Pj5CMljUzdcfZsqUKXjiiScQGhoKg8EgOo7V\neOutt1BWVobPP/9cdBQiRXNwcMAPP/yANWvWIDY2VnQcq3Hx4kU88cQTWLBggbCyfBCLPMIE/nrD\nd/To0ejQoQO++eYbqFT3lD1Vw3//+198+OGHPGonMqGjR49i6NChSEhIgK+vr+g4inbr1i0MGzYM\n/fr1E37UrqiRbIXLly9jyJAhGDZsGObPn8/SrKE1a9bg1Vdfxc6dO+Hh4SE6DpFV2bJlC6ZNm4bE\nxER4e3uLjqNIpaWlGDNmDBwdHfG///u/wt8XVtRItkKTJk2wbds2bN68GbNnz4bI8laqH3/8Ea+8\n8gq2bdvGsiQyg5EjR2LJkiUYOXIkMjIyRMdRnNLSUoSGhqJu3bpYuXKl8LJ8EIu/AM/FxQU7duxA\nQEAASktLeaRZDbGxsXj11VeRlJTEy0eIzGjs2LEoKytDUFAQfvrpJ4t8/80S3bp1C0899RQcHByw\nZs0ai/8sXos+wqzQokULJCcnY+vWrfif//kfnghkhKVLl+L111/H9u3b0bNnT9FxiKze+PHjsWzZ\nMowcORLJycmi41i8S5cuYfTo0WjUqBHi4uJQp04d0ZEeShGFCfx1pLlz50789ttvGDFiBIqLi0VH\nskh6vR6RkZFYuHAhdu3axSNLIhk98cQTWLt2LZ555hksWrSIbyNV4dSpU/D19UW3bt2watUqiz+y\nrKCYwgT+ek9z8+bN6NGjB/r27Ytff/1VdCSLcvHiRQQFBSEnJwcpKSno2LGj6EhENmfw4MHYv38/\nlixZgvDwcJSWloqOZFESExMxYMAAvPXWW/jqq68UdWtORRUmAKjVanz22Wd49913MWjQICQkJIiO\nZBGOHj2KPn36wNfXF5s2beItu4gEat++PQ4cOID8/HwEBASgoKBAdCThJEnCZ599hunTp2PDhg2K\nvNOY4gqzwpQpU5CQkIAXX3wRERERuHbtmuhIQpSVleGTTz7B0KFDMXfuXHz44YcWfZYZka1wdHTE\nhg0b4O/vjx49euDHH38UHUmYnJwcDB8+HHFxcTh48CD69+8vOlKNKLYwAcDX1xeZmZnQ6/Xw8vLC\nzz//LDqSrE6cOIH+/ftj69atSE1NxYQJE0RHIqI72NnZ4YMPPsD69evxn//8B6GhoSgsLBQdSzaS\nJCEqKgoajQb+/v7Yv38/WrduLTpWjSm6MAGgcePGWL58OZYsWYJp06YhIiICV69eFR3LrAwGAz75\n5BMMHDgQU6ZMwfbt22X9rDgiqh4/Pz+kp6ejTZs28PLywrp166z+hKDs7GwMHz4cy5YtQ3JyMt5+\n+21FvV95P4ovzArDhw9HZmYmDAYDOnfujIULF+L27duiY5mUJEn48ccf0a1bN2zfvh2HDh1CRESE\n7B99Q0TVV79+fXz66adYv349Zs2ahcGDByMlJUV0LJMrLi7Gm2++CW9vbwwaNAgHDhywmrP1rWpP\n27hxY0RFRWHr1q3YunUrPDw8sHLlSpSVlYmOVms7d+5E3759MW/ePCxYsADbtm1Du3btRMciomry\n8/NDRkYGJk+ejPHjx2PMmDFWccb/jRs38OGHH6JLly64evUqMjMz8c477yj+qPIukiRV+eevh5Vr\n165dkp+fn9S9e3dp5cqVUklJiehI1VJWViYlJiZKAQEBUocOHaTY2FiprKxMdCwiMpGbN29Kn376\nqfPJnwwAAAPVSURBVPTII49IkydPlnQ6nehI1VZUVCR99tlnkpubmxQSEiKdOnVKdKRa+7v77ulE\ni775uilIkoQtW7Zg4cKFSE9PR1hYGF544QWLPjorKirCihUrsHTpUjg5OeHFF1/E5MmTFXNxLxFV\nz+XLl7FkyRIsXboUrq6uiIyMREhICOrVqyc6WpXS0tKwePFibNiwAaNHj8Yrr7wCjUYjOpZJKPLT\nSkwtKysLS5cuRXR0NPr27Yt//etfGDFiBJo2bSo6GkpKSrBz507ExcUhPj4eWq0WkZGR6NOnD++d\nS2QjysrKsGXLFixevBhpaWmYPHkyxo4dC19fX4u4XOz8+fOIj49HdHQ0CgsLER4ejrCwMDzyyCOi\no5kUC/MOJSUliIuLw/r165GcnAwfHx9otVoEBwejQ4cOsuUoKCjATz/9hPj4eOzcuRM9e/bEk08+\niUmTJsHFxUW2HERkec6cOYOYmBjEx8cjNzcXo0aNglarRWBgIBo1aiRLBkmSkJ6ejvj4eMTHxyMn\nJwcjR45EaGgohg8fbhElbg4szCrcvHkTP//8M+Lj45GQkID69etDo9FAo9HAx8cH3t7eaNasWa3X\nuXHjBo4cOYK0tDTodDrodDr88ccfCAoKglarxYgRI0yyDhFZn+zsbCQkJCA+Ph4HDhyAh4cHfHx8\nKvdVnp6etb55uSRJuHDhQuX+SafTIS0tDU2aNMETTzwBrVYLPz8/6zqJpwosTCOUl5fjzJkzd20s\n6enpcHJyQqtWreDm5gZXV9fK/23UqBHUajXs7e1RXl4Og8GAkpIS5OfnIzc3F3l5ecjNzUVubi7y\n8/Ph6elZWcQVGznflySi6rhx4wYyMjLuevF99uxZtGnTpnLfVLGfat68OerWrQu1Wg2VSgWDwQCD\nwYDLly9X7p8q/jcnJwd2dnZ3FbFGo4G7u7voH1l2LMwaKi8vR05OTmXx5eXlVW5gN2/erNwA7e3t\n4eDggLp166Jly5Z3bbSurq5o164dy5GIzOLGjRs4d+7cXSWYl5eHgoIC6PV66PV6lJeXw8HBAWq1\nGk5OTvccALi7u6Nly5Y8ZwIsTCIiIqNUVZhWdeMCIiIic2FhEhERGYGFSUREZAQWJhERkRFYmERE\nREZgYRIRERmBhUlERGQEFiYREZERWJhERERGYGESEREZgYVJRERkBBYmERGREViYRERERmBhEhER\nGYGFSUREZAQWJhERkRFYmEREREZQP+wvqFT3fOg0ERGRzVFJkiQ6AxERkcXjSJaIiMgILEwiIiIj\nsDCJiIiMwMIkIiIyAguTiIjICP8f1I1TGKUD1doAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faba174c810>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from visualise_neural_network import NeuralNetwork\n",
    "\n",
    "network = NeuralNetwork() #create neural network object\n",
    "network.add_layer(2,['Input A','Input B'],['Weight A','Weight B']) #create the input layer which has two neurons.\n",
    "#Each input neuron has a single line extending to the next layer up\n",
    "network.add_layer(1,['Output']) #create output layer - a single output neuron\n",
    "network.draw() #draw the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above is our neural network. It has two input neurons and a single output neuron. In this example, I'll give the network an input of [0 1]. This means Input A will receive an input value of 0 and Input B will have an input value of 1. \n",
    "\n",
    "The input is the input unit's *activity.* This activity is sent to the Output unit, but the activity changes when traveling to the Output unit. The *weights* between the input and output units change the activity. A large positive weight between the input and output units causes the input unit to send a large positive (excitatory) signal. A large negative weight between the input and output units causes the input unit to send a large negative (inhibitory) signal. A weight near zero means the input unit does not influence the output unit. \n",
    "\n",
    "In order to know the Output unit's activity, we need to know its input. I will refer to the output unit's input as $net_{Output}$. Here is how we can calculate $net_{Output}$\n",
    "\n",
    "$net_{Output} = Input_A * Weight_A + Input_B * Weight_B$\n",
    "\n",
    "a more general way of writing this is \n",
    "\n",
    "$net = \\displaystyle\\sum_{i=1}^{Inputs}Input_i * Weight_i$\n",
    "\n",
    "Let's pretend the inputs are [0 1] and the Weights are [0.25 0.5]. Here is the input to the output neuron - \n",
    "\n",
    "$net_{Output} = 0 * 0.25 + 1 * 0.5$\n",
    "\n",
    "Thus, the input to the output neuron is 0.5. A quick way of programming this is through the function numpy.dot which finds the [dot product](https://en.wikipedia.org/wiki/Dot_product) of two vectors (or matrices). This might sound a little scary, but in this case its just multiplying the items by each other and then summing everything up - like we did above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n"
     ]
    }
   ],
   "source": [
    "Inputs = np.array([0, 1])\n",
    "Weights = np.array([0.25, 0.5])\n",
    "\n",
    "net_Output = np.dot(Inputs,Weights)\n",
    "print net_Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All this is good, but we haven't actually calculated the output unit's activity we have only calculated its input. What makes neural networks able to solve complex problems is they include a non-linearity when translating the input into activity. In this case we will translate the input into activity by putting the input through a [logistic function](https://en.wikipedia.org/wiki/Logistic_function). \n",
    "\n",
    "$Logistic = \\frac{1}{1+e^{-x}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def logistic(x): #each neuron has a logistic activation function\n",
    "    return 1.0/(1+np.exp(-x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take a look at a logistic function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEPCAYAAACgFqixAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH11JREFUeJzt3XucVVX9//HXBxC1AgU1LBBElCQVFBBRM0cpHbFUNBXs\nq2HeUlHEUrxkUIqA9bU0TaRv8lM0sTQQ+3pB0VFMuYmAIQMDEhdBCC+gfuX++f2xztg4zOXAnH32\n2fu8n4/Hfsycc/Yc3oeZOZ9Za+21lrk7IiJS3BrFHUBEROKnYiAiIioGIiKiYiAiIqgYiIgIKgYi\nIkLExcDM/mRmq81sbh3n3G1mFWY228wOjzKPiIjULOqWwRjg5NoeNLNTgA7ufhBwGTAq4jwiIlKD\nSIuBu78KfFjHKacDD2XOnQbsYWatoswkIiLbi3vMoDWwvMrtdzP3iYhIHsVdDEREpAA0ifnffxfY\nr8rtNpn7tmNmWkRJRGQnuLvVd04+WgaWOWoyEbgAwMx6Ah+5++ransjdU3sMGTIk9gx6fXp9xfba\nsn19mzY5y5Y506Y5Eyc6o0c7t93mDBzonHeec9JJTteuTrt2TrNmTuPGTsuWzgEHhPsvuii+15et\nSFsGZvZnoATYy8yWAUOApoC7+2h3f9rMepvZIuBT4MIo84iI1OTDD2HxYliyBJYuDcfy5eFYsQI+\n+AC++lX42tdg332hVatwu1076N4d9t47HHvtBS1bQrNm0ChhnfCRFgN3Py+LcwZEmUFEBGDTJqio\ngLffhvnzYeHCcMydC3feCR06QPv24Q2+Qwc44QRo0yYcrVpB48Zxv4JoxT1mIBklJSVxR4iUXl9y\nJfG1ffABzJoVjjlzwhv+okXhjb5Tp3B897tw5ZWwZk0Jp50GVm+verrZjvQpxcnMPClZRSR/tm4N\nb/j/+AdMmwZTp8KaNXD44dC1KxxxBHTuHArAbrvFnTb/zAzPYgBZxUBEEmXbNpg9G154AV56CV57\nLXTlHHss9OwZjoMPTl6ffVRUDEQkNdauhWefhWeegeeeg332gV69wnHccWHwVmqmYiAiibZ8OTzx\nBIwfH1oCJ54IvXtDaSnst1/9Xy+BioGIJM7atfDYYzBuXLjq5/TToU8f+M53YPfd406XTCoGIpII\nW7eGrp8HHgjjAL17w3nnwUknQdOmcadLPhUDESloa9aEAjBqVJjAddFFcO65sOeecSdLl2yLgeYZ\niEhezZ8fJnk9/jiceWb42L173KlExUBE8mLmTLj11jAP4IorwuzfffaJO5VUUjEQkUjNnAlDhoSJ\nYYMHh8FhDQYXHk3LEJFIVFSEMYDTToNTTw0LwV11lQpBoVIxEJGcWrcOrr0Wjj4aunQJReGKK2DX\nXeNOJnVRMRCRnNi2DcaMCUtBfPxxmCdw003w5S/HnUyyoTEDEWmwBQvg0kthwwZ46ildHZREahmI\nyE7bsgVuvz0sEnfWWWHROBWCZFLLQER2ysKFcMEFYVevWbOgbdu4E0lDqGUgIjvEHe6/H445Bs4/\nPywloUKQfGoZiEjW1q+HSy6B8nJ49dUwWCzpoJaBiGRlzhzo1i1s+D51qgpB2qhlICL1+stfwn7B\nd90VVhSV9FExEJFabdsGt9wCjzwCkyaF/YQlnVQMRKRGn30WrhZ67z2YMUOLyqWdxgxEZDtr14b9\nhZs0CRvOqBCkn4qBiHzB0qXhstHjjw/dQ1pTqDioGIjI58rL4bjjYMAAGD4cGukdomhozEBEAHjz\nzbD/8PDh0L9/3Gkk31QMRIRZs+CUU+APfwhrDEnxUTEQKXKVLYJRo6BPn7jTSFzUIyhSxObODS2C\ne+9VISh2KgYiRWrRIigthbvvVteQqBiIFKWVK+Gkk+CXv4Rzzok7jRQCFQORIvPhh3DyyWFnsksu\niTuNFApz97gzZMXMPClZRQrVxo2hEHTtCv/932AWdyKJmpnh7vV+p1UMRIqEe9iMZsOGsAqpJpQV\nh2yLgS4tFSkSQ4aEQeOXXlIhkO1F/iNhZqVmVm5mC81scA2PNzeziWY228zeMrP+UWcSKTaPPgoP\nPwwTJ8Luu8edRgpRpN1EZtYIWAj0AlYCM4C+7l5e5ZwbgebufqOZ7Q0sAFq5+5Zqz6VuIpGd8MYb\n4RLSyZOhc+e400i+ZdtNFHXLoAdQ4e5L3X0zMA44vdo5DjTLfN4MeL96IRCRnbN6dZhMNmqUCoHU\nLepi0BpYXuX2isx9Vd0DfNPMVgJzgIERZxIpCps3ww9+EBad06QyqU8hDCCfDLzp7ieaWQfgeTPr\n7O6fVD9x6NChn39eUlJCSUlJ3kKKJM2NN8JXvgJVfm2kCJSVlVFWVrbDXxf1mEFPYKi7l2Zu3wC4\nu4+scs7fgeHu/o/M7cnAYHefWe25NGYgkqXx4+Gaa8JqpHvtFXcaiVOhjBnMAA40s3Zm1hToC0ys\nds5S4DsAZtYK6Ai8E3EukdRavBguuyzMJVAhkGxF2k3k7lvNbAAwiVB4/uTu883ssvCwjwZuA/6f\nmc3NfNn17v5BlLlE0mrTprDW0C23wFFHxZ1GkkQzkEVS5LrroKIidBNpqQkBzUAWKTqTJsG4cWGz\nGhUC2VEqBiIpsGYNXHghjB0Le+8ddxpJInUTiSScO5xxBnTqBCNGxJ1GCo26iUSKxIMPwr/+Fa4e\nEtlZahmIJNiyZdCtm9YdktoVyjwDEYnItm1hnODaa1UIpOFUDEQSatQo+PTTcDmpSEOpm0gkgZYt\nC1tXTpkSBo5FaqNuIpGUcg/LTQwapEIguaNiIJIwY8fCqlVw/fVxJ5E0UTeRSIKsWQOHHQbPPBO6\niUTqk203kYqBSIKcfz60agW/+U3cSSQpNOlMJGUmT4ZXXoF58+JOImmkMQORBNiwAS6/HO65J+xe\nJpJrKgYiCTBiBBx6KHz/+3EnkbTSmIFIgVu0CHr2DEtT77df3GkkaTTPQCQlBg0Ks4xVCCRKGkAW\nKWB//zssXAhPPBF3Ekk7FQORArVhAwwcCPfdB02bxp1G0k7dRCIF6je/gS5d4KST4k4ixUADyCIF\n6N13w7LUM2dC+/Zxp5Ek0wxkkQQ7/3xo2xaGDYs7iSSdZiCLJNTUqfDii7BgQdxJpJhozECkgGzb\nBtdcA7ffrpnGkl8qBiIF5NFHYevW0E0kkk8aMxApEJ99BgcfDA8/DMcdF3caSQvNQBZJmLvvhm7d\nVAgkHmoZiBSAf/87bGH52mvQsWPcaSRNdGmpSIJcdRWYhdaBSC6pGIgkREUFHH00lJfD3nvHnUbS\nRmMGIglx883ws5+pEEi81DIQidH06XDmmWFl0i99Ke40kkZqGYgUOHcYPBiGDlUhkPipGIjE5Lnn\n4L33oH//uJOIqBiIxGLbNrjhhrDsRBOtECYFIPJiYGalZlZuZgvNbHAt55SY2Ztm9k8zeynqTCJx\nGzcOdtsNzjgj7iQiQaQDyGbWCFgI9AJWAjOAvu5eXuWcPYDXgJPc/V0z29vd19bwXBpAllTYvDlM\nMPvjH+GEE+JOI2lXKAPIPYAKd1/q7puBccDp1c45D3jC3d8FqKkQiKTJAw+EDWtUCKSQRN1b2RpY\nXuX2CkKBqKojsEume+grwN3uPjbiXCKx+OwzuPVWGD8+7iQiX1QIQ1dNgK7AicCXgdfN7HV3XxRv\nLJHcu/deOOooOPLIuJOIfFFWxcDMjgH2r3q+uz+UxZe+C7StcrtN5r6qVgBr3X0DsMHMXgG6ANsV\ng6FDh37+eUlJCSUlJdnEFykI69fDHXdAWVncSSTNysrKKNuJH7J6B5DNbCzQAZgNbM3c7e5+db1P\nbtYYWEAYQF4FTAf6ufv8KuccDPweKAV2BaYB57r729WeSwPIkmi33hpmGo9VJ6jkUS73QO4OfHNn\n3ondfauZDQAmEQar/+Tu883ssvCwj3b3cjN7DphLKDajqxcCkaT74AO4666wv7FIIcqmZfBX4Gp3\nX5WfSLXmUMtAEuvmm2HNmnA5qUg+5bJlsDfwtplNBzZW3unupzUgn0jRWLMGRo2CN9+MO4lI7bIp\nBkOjDiGSZiNHQr9+0LZt/eeKxCWrGchm1gqovBhuuruviTRVzRnUTSSJs2oVHHII/POf8PWvx51G\nilHOZiCb2TmEq4DOBs4BppnZDxoeUST9RowIq5KqEEihy2YAeQ7w3crWgJntA7zg7l3ykK9qDrUM\nJFFWrIDOnWH+fGjVKu40UqxyuTZRo2rdQu9n+XUiRW34cLj4YhUCSYZsBpCfzcwDeDRz+1zg6egi\niSTfsmVhmery8vrPFSkE2Q4gnwUcm7k5xd3zvsyWuokkSX7yE2jRIrQOROKUbTdRpPsZ5JKKgSTF\n0qXQtWtYemKvveJOI8WuwZPOzOxVd/+WmX0MVH0XNsJSEs1zkFMkdYYNCy0DFQJJErUMRHJoyZKw\nPPXChdCyZdxpRHI7z2C7NRZruk9EQqvg8stVCCR5srma6JCqN8ysCdAtmjgiybVkSdjBrKIi7iQi\nO67WloGZ3ZgZL+hsZuszx8fAauDJvCUUSYhhw+DKK9UqkGTKZgbycHe/MU956sqhMQMpWO+8Az16\naKxACk9OLy01sxbAQcBulfe5+ysNSriDVAykkF18cVh/6Fe/ijuJyBflbD8DM7sYGEjYv3g20BN4\nnbCBvUjRe+cdmDAhtApEkiqbNYYGEpavXuruJwBHAB9FmkokQYYNgyuuUPeQJFs2VxNtcPcNZoaZ\n7ZrZs/gbkScTSYDKVoGuIJKky6YYrDCzPYEJwPNm9iGwNNpYIsmgK4gkLXZoBrKZHQ/sATzr7psi\nS1Xzv60BZCko77wTZhtXVKgYSOHK2dVEZnY3MM7dX8tVuJ2hYiCF5qKLoHVrXUEkhS1nVxMBbwA/\nz4wTjCcUhpkNDSiSZBorkLTJupvIzFoCZwF9gbbuflCUwWr499UykIKhVoEkRS5bBpUOBA4G2gHz\ndzaYSNItXgxPPqlWgaRLNmMGdwB9gMXAY8B4d8/7PAO1DKRQXHghtG0Lv/xl3ElE6pfLlsFi4Gh3\nX9vwWCLJtmgRPPVU+CiSJrW2DMzs4MwEs641Pe7usyJNtn0etQwkdv37Q/v2MGRI3ElEstPgS0vN\nbLS7X2pmL9XwsLt7XtcmUjGQuFVUwDHHhFbBHnvEnUYkO7mcZ7Cbu2+o776oqRhI3C64AA46CG65\nJe4kItnLZTGY5e5d67svaioGEqfycvj2t0OroHnzuNOIZK/BA8hmti/QGtjdzI4AKp+sOfClnKQU\nSYhf/QoGDVIhkPSqa8zgR0B/oDswg/8Ug/XAg+7+t3wErJJHLQOJxbx5cOKJoVXQrFncaUR2TC67\nic5y9ydylmwnqRhIXM45JyxId911cScR2XHZFoNsNrfpllnCuvKJW5jZbQ1KJ5IQc+fClClh8xqR\nNMumGJxSdcaxu38I9I4ukkjh+MUv4Prr4ctfjjuJSLSyKQaNzWzXyhtmtjuwax3nf4GZlZpZuZkt\nNLPBdZx3pJltNrMzs31ukSjNmAEzZ8Lll8edRCR62SxH8Qgw2czGEAaR+wMPZvPkZtYIuAfoBawE\nZpjZk+5eXsN5I4Dnso8uEq1bboGf/xx22y3uJCLRq7cYuPtIM5sDfAdwwht2uyyfvwdQ4e5LAcxs\nHHA6UF7tvKuAx4Ejs3xekUhNmQILFsCPfxx3EpH8yKabCGA1oRCcDZxI9ktYtwaWV7m9InPf58zs\n68AZ7n4f/7l8VSQ27qFVMGQING0adxqR/Khr0llHoF/mWEtYvtrc/YQcZ/gdUHUsQQVBYvX887Bq\nFfzXf8WdRCR/6uomKgemAN9z90UAZjZoB5//XaBtldttMvdV1R0YZ2YG7A2cYmab3X1i9ScbOnTo\n55+XlJRQUlKyg3FE6uYON90Et90GTXZk6yeRAlFWVkZZWdkOf11dM5DPIGxxeSzwLDAO+B93b5/1\nk5s1BhYQBpBXAdOBfu5eYzdTZpD6qZpmN2vSmeTDE0/AsGHhKqJG2XaiihSwBk86c/cJ7t6XsNXl\nS8A1wFfN7D4zOymbEO6+FRgATALmAePcfb6ZXWZml9b0Jdk8r0gUtmwJVw/dfrsKgRSfepej+MLJ\nZi0Ig8jnunuvyFLV/G+rZSCRGjMmHC+/DKaRK0mJnK1NVChUDCRKGzbAN74BjzwC3/pW3GlEcieX\naxOJpN5990GXLioEUrzUMpCit25d2MHsxRfh0EPjTiOSW2oZiGTpjjvge99TIZDippaBFLWVK+Gw\nw2D2bNhvv7jTiOSeBpBFsnDppbDnnqF1IJJGDd4DWSTt5s2DCRPCgnQixU5jBlK0rr8ebr4ZWrSI\nO4lI/NQykKL0wguhRTB+fNxJRAqDWgZSdLZuhZ/9DEaO1BLVIpVUDKToPPRQ2NP4TG2wKvI5XU0k\nReXjj+Hgg0P3UI8ecacRiZ4mnYnUYMQI6NVLhUCkOrUMpGgsWQJHHglz5kDr1vWfL5IGahmIVHP9\n9TBwoAqBSE10aakUhZdfhunTw+CxiGxPLQNJvc2bYcAAuPNO2H33uNOIFCYVA0m9e++FfffVpaQi\nddEAsqTae++FpalffTVcUipSbLRqqQjwox9Bq1ZalVSKl1YtlaL3yisweTLMnx93EpHCpzEDSaVN\nm+AnP4G774ZmzeJOI1L4VAwklX79a+jQAfr0iTuJSDJozEBSZ/FiOOooeOMNaNcu7jQi8dIMZClK\n7nD55TB4sAqByI5QMZBUefBBWLsWrrkm7iQiyaJuIkmNVaugSxeYNAkOPzzuNCKFQfMMpOicdVaY\nWDZsWNxJRAqH5hlIUXn8cXj7bXjkkbiTiCSTWgaSeKtXh+6h8ePh6KPjTiNSWNRNJEXBPcwl6NQJ\nhg+PO41I4VE3kRSFhx4KO5g99ljcSUSSTS0DSaxly6B7d3j++dBNJCLb06QzSbWtW+H88+Haa1UI\nRHJBxUASafhwaNwYrrsu7iQi6aAxA0mc11+He+4Jaw81bhx3GpF0iLxlYGalZlZuZgvNbHANj59n\nZnMyx6tmdljUmSS5PvoIfvhDuP9+aN067jQi6RHpALKZNQIWAr2AlcAMoK+7l1c5pycw393XmVkp\nMNTde9bwXBpALnLuYR/jNm3g97+PO41IMhTKpaU9gAp3X5oJNQ44Hfi8GLj71CrnTwX0957U6M47\nYeVKGDcu7iQi6RN1MWgNLK9yewWhQNTmYuCZSBNJIr36atjHePp02HXXuNOIpE/BDCCb2QnAhcC3\najtn6NChn39eUlJCSUlJ5LkkfqtWQb9+MGaM9igQqU9ZWRllZWU7/HVRjxn0JIwBlGZu3wC4u4+s\ndl5n4Amg1N0X1/JcGjMoQhs3wgknQGkp/OIXcacRSZ6CWJvIzBoDCwgDyKuA6UA/d59f5Zy2wGTg\n/GrjB9WfS8WgyLjDJZeEK4j+8hdopFkxIjusIAaQ3X2rmQ0AJhEuY/2Tu883s8vCwz4auAVoCfzB\nzAzY7O51jStIkbjnnjBG8NprKgQiUdPaRFKQnn4aLrooFIL27eNOI5JcBdEyENkZb74JP/oRPPWU\nCoFIvqjxLQVl+XL4/vdh1Cjoud3UQxGJioqBFIz33w9XDQ0aFPYzFpH80ZiBFIRPPoFevaCkBEaO\nrPd0EclSQVxamksqBum1cSN873thQtkf/whW74+tiGRLxUASYdMmOPts2GWXsOZQE13SIJJT2ulM\nCt7mzXDuuaEl8Oc/qxCIxEm/fhKLzZuhb9+wfeXjj0PTpnEnEiluahlI3n32GfTpA1u2wF//qkIg\nUghUDCSv1q+HU06BPfcMLQItRy1SGFQMJG9Wrw6Xj3bqBA89FAaNRaQwqBhIXpSXw9FHw6mnwh/+\noIXnRAqNBpAlcq+8Ei4fHTkS+vePO42I1ER/n0mk7r8ffvADePhhFQKRQqaWgURi0ya4+urQKvjH\nP+Cgg+JOJCJ1UctAcm7pUjj+eHjvPZg6VYVAJAlUDCSnJk6EHj3CqqN/+xs0bx53IhHJhrqJJCc+\n/RQGDw4b0kyYEK4cEpHkUMtAGmzqVDjiCFi3DubMUSEQSSK1DGSnffopDBkCY8eGzevPPjvuRCKy\ns9QykJ3y3HNw2GGwciW89ZYKgUjSqWUgO2TJEvjpT2H2bLj33rDOkIgkn1oGkpV16+Dmm6F7d+jW\nDd5+W4VAJE1UDKROGzfCb38LHTuGLqHZs0NR2G23uJOJSC6pm0hqtGFD2I945MhwpdDkyXDooXGn\nEpGoqBjIF3z0EYweDXfdFbqEJkwIH0Uk3dRNJAAsWgSDBsEBB8DcufC//wtPPqlCIFIsVAyK2ObN\n4Q2/tBSOOSZsNjNnTlhh9PDD404nIvmkbqIi4x7mBTz4IDzyCBx4IFx2WegO0qCwSPFSMSgS5eVh\n8/lx4+CTT+CHPwzLS3fsGHcyESkE5u5xZ8iKmXlSshaCLVtg2jT4+99h/PhQAM48E/r1g549wSzu\nhCKSD2aGu9f7G69ikCJLloRLQF94AZ5/HvbbD3r3hj59wkCwCoBI8VExSLlt22DBgrCL2CuvwJQp\n8H//B716haO0FFq3jjuliMRNxSBF3MPuYbNmwRtvwMyZMH06tGgRrgI67rhwdOqkv/5F5ItUDBLI\nHVavhvnzw9o/8+aFK3/eegu+9KWwJlDlcdRR8NWvxp1YRApdwRQDMysFfkeY0/Andx9Zwzl3A6cA\nnwL93X12Deekohhs2gTLl8O//hWOd96BxYvDpK+FC6Fp0/AX/je/GT527hyWit5nn7iTi0gSFUQx\nMLNGwEKgF7ASmAH0dffyKuecAgxw91PN7CjgLnfvWcNzFXQx2LAB/v1vWLMmbAS/ejWsWhWOlSth\nxYpQBN5/H77+ddh/f2jXDjp0CMe6dWX07VtCy5Zxv5JolJWVUVJSEneMyKT59aX5tUH6X1+2xSDq\neQY9gAp3X5oJNQ44HSivcs7pwEMA7j7NzPYws1buvjribLVauzb8pb5+fVi6ufL46CP44AP48MPw\n8f33w7F2bSgG++wTjq99DVq1Ch+/8Q0oKYE2bcLVPa1aQZMa/teHDi2jZcuSfL/UvEn7L1yaX1+a\nXxuk//VlK+pi0BpYXuX2CkKBqOucdzP3xVYMXn4Zfv1raN48HHvuCXvsEY5OnaBlyzB4u9de/zn2\n2EODtyKSXJqBXIOzzgqHiEixiHrMoCcw1N1LM7dvALzqILKZjQJecvfHMrfLgeOrdxOZWeEOGIiI\nFLBCGDOYARxoZu2AVUBfoF+1cyYCVwKPZYrHRzWNF2TzYkREZOdEWgzcfauZDQAm8Z9LS+eb2WXh\nYR/t7k+bWW8zW0S4tPTCKDOJiMj2EjPpTEREopO4zW3M7Cozm29mb5nZiLjzRMHMfmpm28wsVbMO\nzOyOzPdutpk9YWbN487UUGZWamblZrbQzAbHnSeXzKyNmb1oZvMyv29Xx50p18yskZnNMrOJcWeJ\nQuZS/b9mfu/mZeZy1ShRxcDMSoDvA4e5+2HAb+JNlHtm1gb4LrA07iwRmAQc4u6HAxXAjTHnaZDM\npMp7gJOBQ4B+ZnZwvKlyagtwrbsfAhwNXJmy1wcwEHg77hARugt42t07AV2A+bWdmKhiAFwOjHD3\nLQDuvjbmPFH4LXBd3CGi4O4vuPu2zM2pQJs48+TA55Mq3X0zUDmpMhXc/b3KpWHc/RPCG0lq1sLN\n/OHVG/ifuLNEIdPyPs7dxwC4+xZ3X1/b+UkrBh2Bb5vZVDN7ycxStV27mZ0GLHf3t+LOkgc/Bp6J\nO0QD1TSpMjVvllWZ2f7A4cC0eJPkVOUfXmkdOG0PrDWzMZmusNFmtnttJxfcpDMzex5oVfUuwjfr\n54S8Ldy9p5kdCfwFOCD/KXdePa/vJkIXUdXHEqWO13ezuz+VOedmYLO7/zmGiLKDzOwrwOPAwEwL\nIfHM7FRgtbvPznQ/J+53LQtNgK7Ale4+08x+B9wADKnt5ILi7t+t7TEz+wnwt8x5MzKDrHu5+/t5\nC9hAtb0+MzsU2B+YY2ZG6EJ5w8x6uPuaPEZskLq+fwBm1p/QND8xL4Gi9S7QtsrtNpn7UsPMmhAK\nwVh3fzLuPDl0LHCamfUGdgeamdlD7n5BzLlyaQWhp2Fm5vbjQK0XOSStm2gCmTcRM+sI7JKkQlAX\nd/+nu+/r7ge4e3vCN/KIJBWC+mSWM78OOM3dN8adJwc+n1RpZk0JkyrTdlXKA8Db7n5X3EFyyd1v\ncve27n4A4fv2YsoKAZnJu8sz75UQVo+udbC84FoG9RgDPGBmbwEbgVR986px0td0/T3QFHg+NH6Y\n6u5XxBtp59U2qTLmWDljZscCPwTeMrM3CT+TN7n7s/Emkx1wNfCIme0CvEMdk3o16UxERBLXTSQi\nIhFQMRARERUDERFRMRAREVQMREQEFQMREUHFQKRGZvZxBM/Zzsyq7/QnUhBUDERqFsUEnPbAeRE8\nr0iDqRiI1MHMjs+skFu5QcjYKo8tMbORZjY3s5LuAZn7x5jZmVXOq2xlDAe+lVlBcmB+X4lI3VQM\nROp3OGFa/zeBDmZ2TJXHPnT3zsC9hI1EalLZyrgBmOLuXdO21o8kn4qBSP2mu/sqD2u3zCasLltp\nXObjo0DPfAcTyRUVA5H6VV1hdStfXODRa/h8C5nfrcxy5E0jTSeSAyoGIjXLdsXYczMf+wKvZz7/\nF1C5C9/pwC6Zzz8GmuUinEiuqRiI1Ky2q4mq39/CzOYAVwGDMvf9ETg+s+xzT+DTzP1zgW1m9qYG\nkKXQaAlrkZ1kZkuAbu7+QdxZRBpKLQORnae/pCQ11DIQERG1DERERMVARERQMRAREVQMREQEFQMR\nEUHFQEREgP8P52lYK5SX7kIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faba1541210>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-5,5,0.1) #create vector of numbers between -5 and 5\n",
    "plt.plot(x,logistic(x))\n",
    "plt.ylabel('Activation')\n",
    "plt.xlabel('Input');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, the logistic used here transforms negative values into values near 0 and positive values into values near 1. Thus, when a unit receives a negative input it has activity near zero and when a unit receives a postitive input it has activity near 1. The most important aspect of this activation function is that its non-linear - it's not a straight line. \n",
    "\n",
    "Now lets see the activity of our output neuron. Remember, the net input is 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.622459331202\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEPCAYAAACgFqixAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH6NJREFUeJzt3XucVVXdx/HPDxQlAwU1TBBElMQLKCCiZo6SNKKpaCpo\nGuYtFUUsJTUCUwTMLM0L0lOkaGJpIPZ4AdFRTLmJgOEMDEjchfACyiP33/PHOkPjMJcDc/bss8/5\nvl+v/ZrZZ/Ycv8dhzm/WWnutZe6OiIjkt3pxBxARkfipGIiIiIqBiIioGIiICCoGIiKCioGIiBBx\nMTCzP5rZKjObU801D5pZqZnNMrNjoswjIiKVi7plMAr4XlVfNLMzgDbufhhwDTAi4jwiIlKJSIuB\nu78FfFrNJecAT6SunQrsbWbNoswkIiI7invMoDmwtNz58tRjIiJSh+IuBiIikgV2i/m/vxw4qNx5\ni9RjOzAzLaIkIrIL3N1quqYuWgaWOiozHrgMwMy6Ap+5+6qqnsjdc/YYNGhQ7Bn0+vT68u21pfv6\nNm1ylixxpk51xo93Ro507r7b6dfPufhip3t3p2NHp1Urp1Ejp359p2lT55BDwuNXXBHf60tXpC0D\nM/sLUADsa2ZLgEFAA8DdfaS7v2hmPcxsAbAeuDzKPCIilfn0U1i4EBYtgsWLw7F0aTiWLYNPPoFv\nfAO++U044ABo1iyct2oFnTvDfvuFY999oWlTaNQI6iWsEz7SYuDuF6dxTd8oM4iIAGzaBKWl8MEH\nUFwM8+eHY84cuP9+aNMGWrcOb/Bt2sCpp0KLFuFo1gzq14/7FUQr7jEDSSkoKIg7QqT0+pIria/t\nk09g5sxwzJ4d3vAXLAhv9O3aheP00+H662H16gLOPhusxl713GY706cUJzPzpGQVkbqzdWt4w//n\nP2HqVJgyBVavhmOOgY4d4dhjoX37UAD23DPutHXPzPA0BpBVDEQkUbZtg1mz4NVX4fXX4e23Q1fO\nSSdB167hOPzw5PXZR0XFQERyxpo18PLL8NJL8MorsP/+0K1bOE4+OQzeSuVUDEQk0ZYuheeeg7Fj\nQ0vgtNOgRw8oLISDDqr5+yVQMRCRxFmzBp55BsaMCXf9nHMO9OwJ3/0uNGwYd7pkUjEQkUTYujV0\n/fzpT2EcoEcPuPhi6N4dGjSIO13yqRiISFZbvToUgBEjwgSuK66Aiy6CffaJO1luSbcYaJ6BiNSp\n4uIwyevZZ+G888LHzp3jTiUqBiJSJ2bMgLvuCvMArrsuzP7df/+4U0kZFQMRidSMGTBoUJgYNmBA\nGBzWYHD20bQMEYlEaWkYAzj7bDjzzLAQ3A03qBBkKxUDEcmotWvh5pvhhBOgQ4dQFK67DvbYI+5k\nUh0VAxHJiG3bYNSosBTE55+HeQK33w577RV3MkmHxgxEpNbmzYOrr4YNG+CFF3R3UBKpZSAiu2zL\nFrjnnrBI3Pnnh0XjVAiSSS0DEdkl8+fDZZeFXb1mzoSWLeNOJLWhloGI7BR3eOwxOPFEuPTSsJSE\nCkHyqWUgImlbtw6uugpKSuCtt8JgseQGtQxEJC2zZ0OnTmHD9ylTVAhyjVoGIlKjv/417Bf8wANh\nRVHJPSoGIlKlbdtg4EB46imYMCHsJyy5ScVARCr15ZfhbqGPPoLp07WoXK7TmIGI7GDNmrC/8G67\nhQ1nVAhyn4qBiHzF4sXhttFTTgndQ1pTKD+oGIjIdiUlcPLJ0LcvDB0K9fQOkTc0ZiAiALz3Xth/\neOhQ6NMn7jRS11QMRISZM+GMM+CRR8IaQ5J/VAxE8lxZi2DECOjZM+40Ehf1CIrksTlzQovg4YdV\nCPKdioFInlqwAAoL4cEH1TUkKgYieWnFCujeHe68Ey68MO40kg1UDETyzKefwve+F3Ymu+qquNNI\ntjB3jztDWszMk5JVJFtt3BgKQceO8JvfgFnciSRqZoa71/iTVjEQyRPuYTOaDRvCKqSaUJYf0i0G\nurVUJE8MGhQGjV9/XYVAdhT5PwkzKzSzEjObb2YDKvl6YzMbb2azzOx9M+sTdSaRfPP00/DkkzB+\nPDRsGHcayUaRdhOZWT1gPtANWAFMB3q5e0m5a24DGrv7bWa2HzAPaObuWyo8l7qJRHbBu++GW0gn\nTYL27eNOI3Ut3W6iqFsGXYBSd1/s7puBMcA5Fa5xoFHq80bAxxULgYjsmlWrwmSyESNUCKR6UReD\n5sDScufLUo+V9xBwhJmtAGYD/SLOJJIXNm+GH/wgLDqnSWVSk2wYQP4e8J67n2ZmbYCJZtbe3b+o\neOHgwYO3f15QUEBBQUGdhRRJmttug69/Hcr92kgeKCoqoqioaKe/L+oxg67AYHcvTJ3/HHB3H17u\nmn8AQ939n6nzScAAd59R4bk0ZiCSprFj4aabwmqk++4bdxqJU7aMGUwHDjWzVmbWAOgFjK9wzWLg\nuwBm1gxoC3wYcS6RnLVwIVxzTZhLoEIg6Yq0m8jdt5pZX2ACofD80d2Lzeya8GUfCdwN/NnM5qS+\n7VZ3/yTKXCK5atOmsNbQwIFw/PFxp5Ek0QxkkRxyyy1QWhq6ibTUhIBmIIvknQkTYMyYsFmNCoHs\nLBUDkRywejVcfjmMHg377Rd3GkkidROJJJw7nHsutGsHw4bFnUayjbqJRPLE44/Dv/8d7h4S2VVq\nGYgk2JIl0KmT1h2SqmXLPAMRici2bWGc4OabVQik9lQMRBJqxAhYvz7cTipSW+omEkmgJUvC1pWT\nJ4eBY5GqqJtIJEe5h+Um+vdXIZDMUTEQSZjRo2HlSrj11riTSC5RN5FIgqxeDUcfDS+9FLqJRGqS\nbjeRioFIglx6KTRrBvfdF3cSSQpNOhPJMZMmwZtvwty5cSeRXKQxA5EE2LABrr0WHnoo7F4mkmkq\nBiIJMGwYHHUUfP/7cSeRXKUxA5Est2ABdO0alqY+6KC400jSaJ6BSI7o3z/MMlYhkChpAFkki/3j\nHzB/Pjz3XNxJJNepGIhkqQ0boF8/ePRRaNAg7jSS69RNJJKl7rsPOnSA7t3jTiL5QAPIIllo+fKw\nLPWMGdC6ddxpJMk0A1kkwS69FFq2hCFD4k4iSacZyCIJNWUKvPYazJsXdxLJJxozEMki27bBTTfB\nPfdoprHULRUDkSzy9NOwdWvoJhKpSxozEMkSX34Jhx8OTz4JJ58cdxrJFZqBLJIwDz4InTqpEEg8\n1DIQyQL/+U/YwvLtt6Ft27jTSC7RraUiCXLDDWAWWgcimaRiIJIQpaVwwglQUgL77Rd3Gsk1GjMQ\nSYg77oCf/UyFQOKlloFIjKZNg/POCyuTfu1rcaeRXKSWgUgWW7xoEYMv+SF3dz+VU1v8kP+sWhR3\nJMlzahmI1LHFixbx+9NP586FC9kLWA8MatOGGyZOpJVWpZMMU8tAJEv9eeDA7YUAYC/gzoUL+fPA\ngXHGkjwXeTEws0IzKzGz+WY2oIprCszsPTP7l5m9HnUmkThtW758eyEosxewbcWKOOKIABGvWmpm\n9YCHgG7ACmC6mT3v7iXlrtkbeBjo7u7LzUz3VEhu+2Zz1sNXCsJ6oN6BB8YUSCT6lkEXoNTdF7v7\nZmAMcE6Fay4GnnP35QDuvibiTCKx2uOIu7iqYRvWp87Lxgz63HVXnLEkz0W9n0FzYGm582WEAlFe\nW2D3VPfQ14EH3X10xLlEYvHll/DIiNY88peJ3PfsQLatWEG9Aw/khrvu0uCxxCobNrfZDegInEZo\nOb9jZu+4+4J4Y4lk3sMPw/HHw9nntubsc5+MO47IdmkVAzM7ETi4/PXu/kQa37ocaFnuvEXqsfKW\nAWvcfQOwwczeBDoAOxSDwYMHb/+8oKCAgoKCdOKLZIV16+Dee6GoKO4kksuKiooo2oV/ZDXOMzCz\n0UAbYBawNfWwu/uNNT65WX1gHmEAeSUwDejt7sXlrjkc+D1QCOwBTAUucvcPKjyX5hlIot11V5hp\nPFqdoFKHMrkHcmfgiF15J3b3rWbWF5hAGKz+o7sXm9k14cs+0t1LzOwVYA6h2IysWAhEku6TT+CB\nB8L+xiLZKJ2Wwd+AG919Zd1EqjKHWgaSWHfcAatXwx/+EHcSyTeZbBnsB3xgZtOAjWUPuvvZtcgn\nkjdWr4YRI+C99+JOIlK1dIrB4KhDiOSy4cOhd29o2bLma0XiktZCdWbWDDgudTrN3VdHmqryDOom\nksRZuRKOPBL+9S/QBGOJQ8YWqjOzCwl3AV0AXAhMNbMf1D6iSO4bNgz69FEhkOyXzgDybOD0staA\nme0PvOruHeogX/kcahlIoixbBu3bQ3ExNGsWdxrJV5lcwrpehW6hj9P8PpG8NnQoXHmlCoEkQzoD\nyC+n5gE8nTq/CHgxukgiybdkCYwZEza5F0mCdAeQzwdOSp1OdvexkaaqPIO6iSQxfvITaNIktA5E\n4pRuN5G2vRTJsMWLoWPHsPTEvvvGnUbyXa0nnZnZW+7+bTP7HCj/LmyEpSQaZyCnSM4ZMiS0DFQI\nJEnUMhDJoEWL4LjjQqugadO404hkdp7BDmssVvaYiIRWwbXXqhBI8qRzN9GR5U/MbDegUzRxRJJr\n0SIYOxZKS+NOIrLzqmwZmNltqfGC9ma2LnV8DqwCnq+zhCIJMWQIXH+9WgWSTOnMQB7q7rfVUZ7q\ncmjMQLLWhx9Cly4aK5Dsk9FbS82sCXAYsGfZY+7+Zq0S7iQVA8lmV14Z1h/61a/iTiLyVRnbz8DM\nrgT6EfYvngV0Bd4hbGAvkvc+/BDGjQutApGkSmeNoX6E5asXu/upwLHAZ5GmEkmQIUPguuvUPSTJ\nls7dRBvcfYOZYWZ7pPYs/lbkyUQSoKxVoDuIJOnSKQbLzGwfYBww0cw+BRZHG0skGXQHkeSKnZqB\nbGanAHsDL7v7pshSVf7f1gCyZJUPPwyzjUtLVQwke2XsbiIzexAY4+5vZyrcrlAxkGxzxRXQvLnu\nIJLslrG7iYB3gV+kxgnGEgrDjNoGFEkyjRVIrkm7m8jMmgLnA72Alu5+WJTBKvnvq2UgWUOtAkmK\nTLYMyhwKHA60Aop3NZhI0i1cCM8/r1aB5JZ0xgzuBXoCC4FngLHuXufzDNQykGxx+eXQsiXceWfc\nSURqlsmWwULgBHdfU/tYIsm2YAG88EL4KJJLqmwZmNnhqQlmHSv7urvPjDTZjnnUMpDY9ekDrVvD\noEFxJxFJT61vLTWzke5+tZm9XsmX3d3rdG0iFQOJW2kpnHhiaBXsvXfcaUTSk8l5Bnu6+4aaHoua\nioHE7bLL4LDDYODAuJOIpC+TxWCmu3es6bGoqRhInEpK4DvfCa2Cxo3jTiOSvloPIJvZAUBzoKGZ\nHQuUPVlj4GsZSSmSEL/6FfTvr0Iguau6MYMfAX2AzsB0/lsM1gGPu/vf6yJguTxqGUgs5s6F004L\nrYJGjeJOI7JzMtlNdL67P5exZLtIxUDicuGFYUG6W26JO4nIzku3GKSzuU2n1BLWZU/cxMzurlU6\nkYSYMwcmTw6b14jksnSKwRnlZxy7+6dAj+giiWSPX/4Sbr0V9tor7iQi0UqnGNQ3sz3KTsysIbBH\nNdd/hZkVmlmJmc03swHVXHecmW02s/PSfW6RKE2fDjNmwLXXxp1EJHrpLEfxFDDJzEYRBpH7AI+n\n8+RmVg94COgGrACmm9nz7l5SyXXDgFfSjy4SrYED4Re/gD33jDuJSPRqLAbuPtzMZgPfBZzwht0q\nzefvApS6+2IAMxsDnAOUVLjuBuBZ4Lg0n1ckUpMnw7x58OMfx51EpG6k000EsIpQCC4ATiP9Jayb\nA0vLnS9LPbadmR0InOvuj/Lf21dFYuMeWgWDBkGDBnGnEakb1U06awv0Th1rCMtXm7ufmuEMvwPK\njyWoIEisJk6ElSvhhz+MO4lI3amum6gEmAyc5e4LAMys/04+/3KgZbnzFqnHyusMjDEzA/YDzjCz\nze4+vuKTDR48ePvnBQUFFBQU7GQckeq5w+23w913w247s/WTSJYoKiqiqKhop7+vuhnI5xK2uDwJ\neBkYA/yPu7dO+8nN6gPzCAPIK4FpQG93r7SbKTVI/UJls5s16UzqwnPPwZAh4S6ieul2oopksVpP\nOnP3ce7ei7DV5evATcA3zOxRM+ueTgh33wr0BSYAc4Ex7l5sZteY2dWVfUs6zysShS1bwt1D99yj\nQiD5p8blKL5ysVkTwiDyRe7eLbJUlf+31TKQSI0aFY433gDTyJXkiIytTZQtVAwkShs2wLe+BU89\nBd/+dtxpRDInk2sTieS8Rx+FDh1UCCR/qWUgeW/t2rCD2WuvwVFHxZ1GJLPUMhBJ0733wllnqRBI\nflPLQPLaihVw9NEwaxYcdFDcaUQyTwPIImm4+mrYZ5/QOhDJRbXeA1kk182dC+PGhQXpRPKdxgwk\nb916K9xxBzRpEncSkfipZSB56dVXQ4tg7Ni4k4hkB7UMJO9s3Qo/+xkMH64lqkXKqBhI3nniibCn\n8XnaYFVkO91NJHnl88/h8MND91CXLnGnEYmeJp2JVGLYMOjWTYVApCK1DCRvLFoExx0Hs2dD8+Y1\nXy+SC9QyEKng1luhXz8VApHK6NZSyQtvvAHTpoXBYxHZkVoGkvM2b4a+feH++6Fhw7jTiGQnFQPJ\neQ8/DAccoFtJRaqjAWTJaR99FJamfuutcEupSL7RqqUiwI9+BM2aaVVSyV9atVTy3ptvwqRJUFwc\ndxKR7KcxA8lJmzbBT34CDz4IjRrFnUYk+6kYSE769a+hTRvo2TPuJCLJoDEDyTkLF8Lxx8O770Kr\nVnGnEYmXZiBLXnKHa6+FAQNUCER2hoqB5JTHH4c1a+Cmm+JOIpIs6iaSnLFyJXToABMmwDHHxJ1G\nJDtonoHknfPPDxPLhgyJO4lI9tA8A8krzz4LH3wATz0VdxKRZFLLQBJv1arQPTR2LJxwQtxpRLKL\nuokkL7iHuQTt2sHQoXGnEck+6iaSvPDEE2EHs2eeiTuJSLKpZSCJtWQJdO4MEyeGbiIR2ZEmnUlO\n27oVLr0Ubr5ZhUAkE1QMJJGGDoX69eGWW+JOIpIbNGYgifPOO/DQQ2Htofr1404jkhsibxmYWaGZ\nlZjZfDMbUMnXLzaz2anjLTM7OupMklyffQaXXAKPPQbNm8edRiR3RDqAbGb1gPlAN2AFMB3o5e4l\n5a7pChS7+1ozKwQGu3vXSp5LA8h5zj3sY9yiBfz+93GnEUmGbLm1tAtQ6u6LU6HGAOcA24uBu08p\nd/0UQH/vSaXuvx9WrIAxY+JOIpJ7oi4GzYGl5c6XEQpEVa4EXoo0kSTSW2+FfYynTYM99og7jUju\nyZoBZDM7Fbgc+HZV1wwePHj75wUFBRQUFESeS+K3ciX07g2jRmmPApGaFBUVUVRUtNPfF/WYQVfC\nGEBh6vzngLv78ArXtQeeAwrdfWEVz6Uxgzy0cSOceioUFsIvfxl3GpHkyYq1icysPjCPMIC8EpgG\n9Hb34nLXtAQmAZdWGD+o+FwqBnnGHa66KtxB9Ne/Qj3NihHZaVkxgOzuW82sLzCBcBvrH9292Myu\nCV/2kcBAoCnwiJkZsNndqxtXkDzx0ENhjODtt1UIRKKmtYkkK734IlxxRSgErVvHnUYkubKiZSCy\nK957D370I3jhBRUCkbqixrdklaVL4fvfhxEjoOsOUw9FJCoqBpI1Pv443DXUv3/Yz1hE6o7GDCQr\nfPEFdOsGBQUwfHiNl4tImrLi1tJMUjHIXRs3wllnhQllf/gDWI3/bEUkXSoGkgibNsEFF8Duu4c1\nh3bTLQ0iGaWdziTrbd4MF10UWgJ/+YsKgUic9Osnsdi8GXr1CttXPvssNGgQdyKR/KaWgdS5L7+E\nnj1hyxb4299UCESygYqB1Kl16+CMM2CffUKLQMtRi2QHFQOpM6tWhdtH27WDJ54Ig8Yikh1UDKRO\nlJTACSfAmWfCI49o4TmRbKMBZIncm2+G20eHD4c+feJOIyKV0d9nEqnHHoMf/ACefFKFQCSbqWUg\nkdi0CW68MbQK/vlPOOywuBOJSHXUMpCMW7wYTjkFPvoIpkxRIRBJAhUDyajx46FLl7Dq6N//Do0b\nx51IRNKhbiLJiPXrYcCAsCHNuHHhziERSQ61DKTWpkyBY4+FtWth9mwVApEkUstAdtn69TBoEIwe\nHTavv+CCuBOJyK5Sy0B2ySuvwNFHw4oV8P77KgQiSaeWgeyURYvgpz+FWbPg4YfDOkMiknxqGUha\n1q6FO+6Azp2hUyf44AMVApFcomIg1dq4EX77W2jbNnQJzZoVisKee8adTEQySd1EUqkNG8J+xMOH\nhzuFJk2Co46KO5WIREXFQL7is89g5Eh44IHQJTRuXPgoIrlN3UQCwIIF0L8/HHIIzJkD//u/8Pzz\nKgQi+ULFII9t3hze8AsL4cQTw2Yzs2eHFUaPOSbudCJSl9RNlGfcw7yAxx+Hp56CQw+Fa64J3UEa\nFBbJXyoGeaKkJGw+P2YMfPEFXHJJWF66bdu4k4lINjB3jztDWszMk5I1G2zZAlOnwj/+AWPHhgJw\n3nnQuzd07QpmcScUkbpgZrh7jb/xKgY5ZNGicAvoq6/CxIlw0EHQowf07BkGglUARPKPikGO27YN\n5s0Lu4i9+SZMngz/93/QrVs4CguhefO4U4pI3FQMcoh72D1s5kx4912YMQOmTYMmTcJdQCefHI52\n7fTXv4h8lYpBArnDqlVQXBzW/pk7N9z58/778LWvhTWByo7jj4dvfCPuxCKS7bKmGJhZIfA7wpyG\nP7r78EqueRA4A1gP9HH3WZVckxPFYNMmWLoU/v3vcHz4ISxcGCZ9zZ8PDRqEv/CPOCJ8bN8+LBW9\n//5xJxeRJMqKYmBm9YD5QDdgBTAd6OXuJeWuOQPo6+5nmtnxwAPu3rWS58rqYrBhA/znP7B6ddgI\nftUqWLkyHCtWwLJloQh8/DEceCAcfDC0agVt2oRj7doievUqoGnTuF9JNIqKiigoKIg7RmRy+fXl\n8muD3H996RaDqOcZdAFK3X1xKtQY4BygpNw15wBPALj7VDPb28yaufuqiLNVac2a8Jf6unVh6eay\n47PP4JNP4NNPw8ePPw7HmjWhGOy/fzi++U1o1ix8/Na3oKAAWrQId/c0awa7VfJ/ffDgIpo2Lajr\nl1pncv0XLpdfXy6/Nsj915euqItBc2BpufNlhAJR3TXLU4/FVgzeeAN+/Wto3Dgc++wDe+8djnbt\noGnTMHi7777/PfbeW4O3IpJcmoFcifPPD4eISL6IesygKzDY3QtT5z8HvPwgspmNAF5392dS5yXA\nKRW7icwsewcMRESyWDaMGUwHDjWzVsBKoBfQu8I144HrgWdSxeOzysYL0nkxIiKyayItBu6+1cz6\nAhP4762lxWZ2Tfiyj3T3F82sh5ktINxaenmUmUREZEeJmXQmIiLRSdzmNmZ2g5kVm9n7ZjYs7jxR\nMLOfmtk2M8upWQdmdm/qZzfLzJ4zs8ZxZ6otMys0sxIzm29mA+LOk0lm1sLMXjOzuanftxvjzpRp\nZlbPzGaa2fi4s0Qhdav+31K/d3NTc7kqlahiYGYFwPeBo939aOC+eBNlnpm1AE4HFsedJQITgCPd\n/RigFLgt5jy1kppU+RDwPeBIoLeZHR5vqozaAtzs7kcCJwDX59jrA+gHfBB3iAg9ALzo7u2ADkBx\nVRcmqhgA1wLD3H0LgLuviTlPFH4L3BJ3iCi4+6vuvi11OgVoEWeeDNg+qdLdNwNlkypzgrt/VLY0\njLt/QXgjyZm1cFN/ePUA/ifuLFFItbxPdvdRAO6+xd3XVXV90opBW+A7ZjbFzF43s5zart3MzgaW\nuvv7cWepAz8GXoo7RC1VNqkyZ94syzOzg4FjgKnxJsmosj+8cnXgtDWwxsxGpbrCRppZw6ouzrpJ\nZ2Y2EWhW/iHCD+sXhLxN3L2rmR0H/BU4pO5T7roaXt/thC6i8l9LlGpe3x3u/kLqmjuAze7+lxgi\nyk4ys68DzwL9Ui2ExDOzM4FV7j4r1f2cuN+1NOwGdASud/cZZvY74OfAoKouzirufnpVXzOznwB/\nT103PTXIuq+7f1xnAWupqtdnZkcBBwOzzcwIXSjvmlkXd19dhxFrpbqfH4CZ9SE0zU+rk0DRWg60\nLHfeIvVYzjCz3QiFYLS7Px93ngw6CTjbzHoADYFGZvaEu18Wc65MWkboaZiROn8WqPImh6R1E40j\n9SZiZm2B3ZNUCKrj7v9y9wPc/RB3b034QR6bpEJQk9Ry5rcAZ7v7xrjzZMD2SZVm1oAwqTLX7kr5\nE/CBuz8Qd5BMcvfb3b2lux9C+Lm9lmOFgNTk3aWp90oIq0dXOViedS2DGowC/mRm7wMbgZz64VXg\n5F7T9fdAA2BiaPwwxd2vizfSrqtqUmXMsTLGzE4CLgHeN7P3CP8mb3f3l+NNJjvhRuApM9sd+JBq\nJvVq0pmIiCSum0hERCKgYiAiIioGIiKiYiAiIqgYiIgIKgYiIoKKgUilzOzzCJ6zlZlV3OlPJCuo\nGIhULooJOK2BiyN4XpFaUzEQqYaZnZJaIbdsg5DR5b62yMyGm9mc1Eq6h6QeH2Vm55W7rqyVMRT4\ndmoFyX51+0pEqqdiIFKzYwjT+o8A2pjZieW+9qm7twceJmwkUpmyVsbPgcnu3jHX1vqR5FMxEKnZ\nNHdf6WHtllmE1WXLjEl9fBroWtfBRDJFxUCkZuVXWN3KVxd49Eo+30Lqdyu1HHmDSNOJZICKgUjl\n0l0x9qLUx17AO6nP/w2U7cJ3DrB76vPPgUaZCCeSaSoGIpWr6m6iio83MbPZwA1A/9RjfwBOSS37\n3BVYn3p8DrDNzN7TALJkGy1hLbKLzGwR0MndP4k7i0htqWUgsuv0l5TkDLUMRERELQMREVExEBER\nVAxERAQVAxERQcVARERQMRAREeD/AQ6AcJemZmA4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fab9dc6ea90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Output_neuron = logistic(net_Output)\n",
    "print Output_neuron\n",
    "plt.plot(x,logistic(x));\n",
    "plt.ylabel('Activation')\n",
    "plt.xlabel('Input')\n",
    "plt.plot(net_Output,Output_neuron,'ro');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The activity of our output neuron is depicted as the red dot.\n",
    "\n",
    "So far I've described how to find a unit's activity, but I haven't described how to find the weights of connections between units. In the example above, I chose the weights to be 0.25 and 0.5, but I can't arbitrarily decide weights unless I already know the solution to the problem. If I want the network to find a solution for me, I need the network to find the weights itself. \n",
    "\n",
    "In order to find the weights of connections between neurons, I will use an algorithm called [backpropogation](https://en.wikipedia.org/wiki/Backpropagation). In backpropogation, we have the neural network guess the answer to a problem and adjust the weights so that this guess gets closer and closer to the correct answer. Backpropogation is the method by which we reduce the distance between guesses and the correct answer. After many iterations of guesses by the neural network and weight adjustments through backpropogation, the network can learn an answer to a problem.\n",
    "\n",
    "Lets say we want our neural network to give an answer of 0 when the left input unit is active and an answer of 1 when the right unit is active. In this case the inputs I will use are [1,0] and [0,1]. The corresponding correct answers will be [0] and [1]. \n",
    "\n",
    "Lets see how close our network is to the correct answer. I am using the weights from above ([0.25, 0.5]). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.56217650088579807, 0.62245933120185459]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEPCAYAAABsj5JaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHFpJREFUeJzt3X10VfWd7/HPNzwUgYAJT5EAAYMPl2mVijeCoAm1IA7c\n2o6OlQl1sHWuD+gSFrel3oqBYqu2cr1qnbpsqcggMlalZZXxgWkNioqACt6hIAIhtgGjaUUeRAvJ\n9/5xDocQc5KTX3IeQt6vtc7KOXv/zt7f7HVyPtn7t/f+mbsLAICWykp3AQCA9okAAQAEIUAAAEEI\nEABAEAIEABCEAAEABEl7gJjZIjOrNrO348wvNrN9ZvZm9HF7qmsEAHxe53QXIOlRSQ9KWtJEm5fc\n/WspqgcAkIC074G4+1pJHzXTzFJRCwAgcWkPkASNMbNNZrbKzEakuxgAQGYcwmrOG5KGuPsnZnaZ\npN9IOjPNNQFAh5fxAeLuB+s9f9bM/tXMct39rw3bmhk39gKAFnL3oG6CTDmEZYrTz2FmA+o9L5Jk\njYXHMe7Oow0eZWVlaa/hZHqwPdmemfpojbTvgZjZMkklkvqY2XuSyiR1leTu/oikK83sRklHJB2W\n9M101QoAOC7tAeLu/9TM/IckPZSicgAACcqUQ1jIMCUlJeku4aTC9mxbbM/MYK09BpZJzMxPpt8H\nAJLNzOSBnehpP4QFoGMZOnSoKisr011Gh1NQUKDdu3e36TLZAwGQUtH/eNNdRocTb7u3Zg+EPhAA\nQBACBAAQhAABAAQhQAAAQQgQAGhg+fLlGj16tHr27Km8vDyNGTNGP//5z9NdVsYhQABkjIqKSk2b\nNl/jx5dp2rT5qqho+em+rV3GwoULNWvWLM2ZM0fV1dV6//339fDDD+vVV1/VkSNHWlzPSS3dN/Jq\n45uCOYDMFu/vdNeu3V5YONulgy65Swe9sHC279q1O+Flt3YZH3/8sffo0cNXrFgRt01JSYkvWrQo\n9nrx4sU+bty42OutW7f6hAkTPDc3188++2x/8sknY/NWrVrlI0aM8OzsbB80aJAvXLjQ3d1ramp8\nypQpfuqpp3pubq5ffPHFsffs2bPHr7jiCu/Xr5+ffvrp/sADD8TmrV+/3s8//3zv1auX5+Xl+ezZ\ns+PWHW+7R6eHfeeGvjETHwQIkPni/Z2Wls6r98XvsQAoLZ2X8LJbu4znnnvOu3Tp4rW1tXHbNBYg\nF110kbu7Hzp0yAcPHuyPPfaY19XV+aZNm7xv376+detWd3c/7bTT/JVXXnF393379vlbb73l7u63\n3Xab33jjjV5bW+tHjx71tWvXurt7XV2djxo1yu+8804/evSoV1RUeGFhob/wwgvu7j5mzBhfunRp\nbN2vv/563LqTESAcwgKQEaqq6iT1aDC1h/bsqUvZMmpqatS3b19lZR3/ahw7dqxycnLUo0cPvfzy\ny02+/3e/+52GDRuma665Rmamc889V1dccYV+/etfS5K6du2qLVu26MCBA+rdu7dGjhwpSerSpYv2\n7t2riooKderUSWPHjpUkbdiwQTU1NfrBD36gTp06aejQobruuuu0fPny2Pt27Nihv/zlL+revbuK\niooS+j3bCgECICPk52dJOtRg6iENHJj411Rrl9GnTx/V1NSoru544Lzyyiv66KOP1KdPn2NHOuKq\nrKzUunXrlJubq9zcXOXk5GjZsmWqrq6WJD399NNatWqVCgoKNH78eK1bt06S9L3vfU+FhYWaOHGi\nhg8frnvuuSe2vKqqqhOWd9ddd+mDDz6QJP3qV7/SO++8o7PPPlsXXHCBVq1aldDv2WZCd10y8SEO\nYQEZL97faSb0gezbt8979uzpzzzzzOfmDRo0yNesWeOTJ0/2Bx98MDb97rvvjh3CeuKJJ3zixInN\nrufo0aN+3333+eDBgz83b8uWLd6/f3//wx/+4K+99pqfeeaZCdX+1FNPebdu3fyTTz5pdH687S4O\nYQFo74YNK9Dq1beotPRejR9fptLSe7V69S0aNqwgZcvo3bu37rjjDt100016+umndfDgQbm7Nm3a\npE8++USSNHLkSD3zzDM6fPiwduzYoUWLFsXeP2XKFG3fvl1Lly7V0aNHdeTIEW3cuFHbtm3TkSNH\ntGzZMu3fv1+dOnVSdna2OnXqJElatWqVdu7cKUnKzs5W586dlZWVpaKiImVnZ+snP/mJPv30U9XW\n1mrLli3auHGjJOnxxx9XTU1NrHYzO+HwW9KFJk8mPsQeCJDx2sPf6bJly7yoqMh79Ojh/fv399Gj\nR/svf/lLP3LkiNfU1PjEiRO9V69ePm7cOJ8/f35sD8Tdffv27T558mTv16+f9+3b1y+55BLfvHmz\n/+1vf/NJkyZ5bm6u9+7d24uKivzVV191d/f77rvPhw4d6j179vTBgwf7j370o9jy9u7d61OnTvW8\nvDzPzc31MWPG+O9//3t3d582bZr379/fs7Oz/Ytf/KKvXLky7u8Ub7urFXsg3I0XQEpxN9704G68\nAICMQYAAAIIQIACAIAQIACAIAQIACEKAAACCECAAgCAECAAgCAECAAhCgABAI0pKSpSbm8sohE0g\nQABkjMqKCs2fNk1l48dr/rRpqqyoSM8yKiu1du1aZWVlaeXKlS1+fyrU1tamuwRupgggteL9ne7e\ntctnFxb6wehQggcln11Y6Lt37Up42W2xDHf3H/7whz5u3DifPXu2T5kyJTZ9+vTpPmPGDJ88ebJn\nZ2f76NGjfVe9Zc+cOdP79+/vvXr18nPOOce3bNniFRUVfuqpp8baXHfddd6/f//Y629961t+//33\nu3tkSN3vfOc7ftppp/mgQYP89ttv97q6OnePjHw4duxYnzVrlvfp08fnzp3rO3bs8OLiYu/du7f3\n69fPr7766ri/U7ztLoa0JUCA9iLe3+m80tLYF7/XC4B5paUJL7stluHuPnz4cH/44Yf9jTfe8C5d\nuvgHH3zg7pEA6du3r2/cuNFra2u9tLTUp06d6u7uzz//vJ9//vm+f/9+d3fftm2bv//+++7uXlBQ\n4G+++aa7u5911lleWFjo27Ztc3f3IUOG+ObNm93d/etf/7rfeOONfvjwYf/www/9ggsu8EceecTd\nIwHSuXNnf+ihh7y2ttYPHz7sU6dO9R//+Mfu7v7ZZ5/FhsttTDIChENYADJCXVVVI4PRSnV79qR0\nGWvXrtV7772nq666Suedd56GDx+uZcuWxeZ/4xvf0KhRo5SVlaXS0lJt2rRJUmR42QMHDuiPf/yj\n3F1nnXWWBgwYIEm6+OKLtWbNmtjIhFdeeaXWrFmj3bt368CBAzrnnHNUXV2tZ599Vvfdd5+6deum\nvn37aubMmXriiSdi687Pz9dNN92krKwsdevWTV26dImNWti1a1ddeOGFCf+ebYEAAZARsvLzGxmM\nVsoaODCly1iyZIkmTpyonJwcSdLUqVP12GOPxebn5eXFnnfv3l0HDx6UJI0fP14333yzZsyYoQED\nBuiGG26IzSsuLtaLL76ol156ScXFxSopKVF5ebnWrFmjiy66SJL03nvv6ciRIzrttNNiw9fecMMN\nsQGjJGnw4MEn1PrTn/5UdXV1Kioq0pe+9CU9+uijCf+ebSJ01yUTH+IQFpDx4v2dZkIfyOHDh713\n796enZ3teXl5npeX5zk5OZ6VleWbN2/26dOn+9y5c2Pty8vLGx2W9sMPP/SSkhK/44473N393Xff\n9ZycHJ8xY4Y//vjjvn//fh8yZIhfe+21vnDhQnePDBzVvXv3WJ9HQ4sXLz5h4KqG1q5d6926dfOd\nO3c2Oj/edheHsAC0dwXDhumW1at1b2mpysaP172lpbpl9WoVDBuWsmWsWLFCnTt31tatW7V582Zt\n3rxZ27Zt00UXXaQlS5Y0+d6NGzdq/fr1Onr0qE455RR169YtNrzs8OHDdcopp2jp0qUqLi5Wdna2\nBgwYoGeeeUbFxcWSIns2EydO1KxZs3TgwAG5u3bt2qWXXnop7jqfeuopVVVVSZJOPfVUZWVlpXRI\n284pWxMANKNg2DCVLV2atmUsWbJE3/72t5Wfn3/C9BkzZujWW2/VV7/61bjv3b9/v2bNmqWKigp1\n69ZNl156qb773e/G5hcXF2v9+vWxZRcXF2v79u0677zzTlj/nDlzNGLECB08eFCnn3665syZE3ed\nGzZs0MyZM7V//34NGDBADzzwgIYOHRr0u4dgSFsAKcWQtunBkLYAgIxBgAAAghAgAIAgBAgAIAgB\nAgAIQoAAAIJwHQiAlCooKJBZ0FmjaIWCgoI2XybXgQBAB8Z1IACAlCNAAABB0h4gZrbIzKrN7O0m\n2jxgZu+a2SYzG5nK+gAAjUt7gEh6VNKl8Waa2WWSCt39DEnXS3o4VYUByCxtMd452k7az8Jy97Vm\n1tTpAZdLWhJt+7qZ9TazAe5enZoKAWSCyooKPThhgubv3KkeigwUVbZuXYtv+Y62kwl7IM3Jl/Sn\neq+rotMAdCCL586NhYcUGap2/s6dWjx3bjrL6tDSvgfS1ubNmxd7XlJSopKSkrTVAqDttMV455DK\ny8tVXl7eJstqDwFSJan+QMCDotMaVT9AAJw8jo13Xj9EWjreOT7/j/X8+fODl5Uph7As+mjMSknX\nSJKZjZa0j/4PoOOZvmCBygoLdSj6+pCkssJCTV+wIJ1ldWhpvxLdzJZJKpHUR1K1pDJJXRUZ6P2R\naJufSZqkyGfmWnd/M86yuBIdOIlVVlRo8dy5qtuzR1kDB2r6ggV0oLdSa65ET3uAtCUCBABahluZ\nAABSjgABAAQhQAAAQQgQAEAQAgQAEIQAAQAEIUAAAEEIEABAEAIEABCEAAEABCFAAABBCBAAQBAC\nBAAQhAABAAQhQAAAQQgQAEAQAgQAEIQAAQAEIUAAAEEIEABAEAIEABCEAAEABCFAAABBCBAAQBAC\nBAAQhAABAAQhQAAAQQgQAEAQAgQAEKRzIo3M7EJJQ+u3d/clSaoJANAONBsgZvZvkgolbZJUG53s\nkggQAOjAEtkDOV/SCHf3ZBcDAGg/EukD+S9JeckuBADQviSyB9JX0h/NbL2kz45NdPevJa0qAEDG\nSyRA5iW7CABA+2OJdG2Y2QBJ/z36cr27f5DUqgKZGV01wEmsoqJSc+cuVlVVnfLzs7RgwXQNG1aQ\n7rLaNTOTu1vQe5v7wjWzqyT9VFK5JJN0kaTvuvtTIStMJgIEOHlVVFRqwoQHtXPnfEk9JB1SYWGZ\nVq++hRBphWQHyGZJE47tdZhZP0n/6e7nhqwwmQgQ4OQ1bdp8Pf74/1IkPI45pNLSe7V0aVm6ymr3\nWhMgiZyFldXgkNVfEnwfALSZqqo6nRgektRDe/bUpaMcKLFO9OfM7HlJT0Rff1PSfySvJAD4vPz8\nLEmH1HAPZOBA/p9Nl0Q70a+QNDb68mV3X5HUqgJxCAs4edEHkhxJ7QNpTwiQ1uMsF2SyY5/PPXvq\nNHAgn8+2kJQAMbO17j7OzA4ocu+r2CxJ7u69QlaYTARI6/AfHtDxsAcSRYC0Dme5AB1PUs/Cit6N\nt9lpaP84ywVASyRy+sLf1X9hZp0ljUpOOUin42e51MdZLgAaF/ebwcxui/Z/nGNm+6OPA5KqJf02\nZRUiZRYsmK7CwjIdD5FIH8iCBdPTVhOAzJXIleh3ufttSSvAbJKk/6tImC1y93sazC9WJLB2RSc9\n4+53xlkWfSCtxFkuQMeS9E50M8uRdIakbsemuftLIStssNwsSdslXSJpj6QNkq5292312hRLmp3I\n7eMJEABomdYESCJD2l4n6VZJgxQZ1na0pNckfSVkhQ0USXrX3Suj61ou6XJJ2xq0C/rlAADJk0jv\n6K2K3Mq90t3HS/qypH1ttP58SX+q9/rP0WkNjTGzTWa2ysxGtNG6AQCtkMi9sD5190/NTGb2BXff\nZmZnJb2y496QNMTdPzGzyyT9RtKZ8RrPmzcv9rykpEQlJSXJrg8A2o3y8nKVl5e3ybIS6URfIela\nSTMVOWz1kaQu7v73rV652WhJ89x9UvT19xW5yv2eJt5TIWmUu/+1kXn0gQBAC6TsSvRoh3ZvSc+5\n+99CVthgeZ0kvaNIJ/peSeslTXX3rfXaDHD36ujzIklPuvvQOMsjQACgBZLdif6ApOXu/qq7rwlZ\nSTzuXmtmN0t6QcdP491qZtdHZvsjkq40sxslHZF0WJHbyQMA0iyRQ1j/rMiX9lmSVigSJhtTUFuL\nsQcCAC2TkkNYZpYr6QpJVyvSqX1GyAqTiQABgJZJ9pC2xwyXdLakAn3+Og0AQAeTyCGsn0j6hqSd\nkv5d0gp3b6vrQNoUeyAA0DJJ7URXJDjGuHtNyAoAACenpkYkPDt60eB5jc139zeTWlkA9kAAoGWS\nNaTtI+7+P83sxUZmu7u3xb2w2hQBAgAtk9SzsMysm7t/2ty0TECAAEDLJPssrFcTnAYA6EDidqKb\nWZ4id8Y9xcy+rOO3VO8lqXsKagMAZLCmzsK6VNJ0RcYBWajjAbJf0v9OblkAgEyXSB/IFe7+dIrq\naRX6QACgZZLdBzLKzE6tt7IcM2t0THIAQMeRSIBcVv/Kc3f/SFKrxwIBALRviQRIJzP7wrEXZnaK\npC800R4A0AEkciuTxyX93sweVaQjfbqkx5JZFAAg8yV0O3czmyTpq5JckbOw8tx9RpJrazE60QGg\nZVJxO/dqRcLjHxUZF31r080BACe7pi4kPFPS1OijRpFbuZu7j09RbQCADNbUzRTrJL0s6TvuviM6\nbZe7n57C+lqEQ1gA0DLJOoT1D5L2SnrRzH5hZpfo+NXoAIAOLpEr0XtIulyRQ1lfkbREkVEJX0h+\neS3DHggAtExSb+feYEU5inSkf9PdLwlZYTIRIADQMikLkExHgABAy6TiNF4AAE5AgAAAghAgAIAg\nBAgAIAgBAgAIQoAAAIIQIACAIAQIACAIAQIACEKAAACCECAAgCAECAAgCAECAAhCgAAAghAgAIAg\nBAgAIAgBAgAIQoAAAIIQIACAIAQIACAIAQIACEKAAACCECAAgCAECAAgSNoDxMwmmdk2M9tuZnPi\ntHnAzN41s01mNjLVNQIAPq9zOlduZlmSfibpEkl7JG0ws9+6+7Z6bS6TVOjuZ5jZBZIeljQ6LQV3\nAJUVFVo8d67qqqqUlZ+v6QsWqGDYsHSXBSADpTVAJBVJetfdKyXJzJZLulzStnptLpe0RJLc/XUz\n621mA9y9OuXVnuQqKyr04IQJmr9zp3pIOiSpbN063bJ6NSEC4HPSfQgrX9Kf6r3+c3RaU22qGmmD\nNrB47txYeEhSD0nzd+7U4rlz01kWgAyV7j2QNjdv3rzY85KSEpWUlKStlvamrqoqFh7H9JBUt2dP\nOsoBkATl5eUqLy9vk2WlO0CqJA2p93pQdFrDNoObaRNTP0DQMln5+ToknRAihyRlDRyYpooAtLWG\n/1jPnz8/eFnpPoS1QdJwMysws66Srpa0skGblZKukSQzGy1pH/0fyTF9wQKVFRbqUPT1IUllhYWa\nvmBBOssCkKHM3dNbgNkkSfcrEmaL3P1uM7tekrv7I9E2P5M0SZHvtGvd/c04y/J0/z7tXewsrD17\nlDVwIGdhASc5M5O7W9B7T6YvXAIEAFqmNQGS7kNYAIB2igABAAQhQAAAQQgQAEAQAgQAEIQAAQAE\nIUAAAEEIEABAEAIEABCEAAEABCFAAABBCBAAQBACBAAQhAABAAQhQAAAQQgQAEAQAgQAEIQAAQAE\nIUAAAEEIEABAEAIEABCEAAEABCFAAABBCBAAQBACBAAQhAABAAQhQAAAQQgQAEAQAgQAEIQAAQAE\nIUAAAEEIEABAEAIEABCEAAEABCFAAABBCBAAQBACBAAQhAABAAQhQAAAQQgQAEAQAgQAEIQAAQAE\nIUAAAEEIEABAEAIEABCEAAEABOmcrhWbWY6kf5dUIGm3pKvc/eNG2u2W9LGkOklH3L0ohWUCAOJI\n5x7I9yX9p7ufJekPkm6L065OUom7f5nwSJ3y8vJ0l3BSYXu2LbZnZkhngFwu6bHo88ckfT1OOxOH\n2lKOP9C2xfZsW2zPzJDOL+b+7l4tSe7+vqT+cdq5pNVmtsHM/iVl1QEAmpTUPhAzWy1pQP1JigTC\n7Y009ziLGevue82snyJBstXd17ZxqQCAFjL3eN/bSV6x2VZF+jaqzSxP0ovu/t+aeU+ZpAPu/n/i\nzE/PLwMA7Zi7W8j70nYWlqSVkqZLukfSP0v6bcMGZtZdUpa7HzSzHpImSpofb4GhGwEA0HLp3APJ\nlfSkpMGSKhU5jXefmZ0m6RfuPsXMhklaocjhrc6SHnf3u9NSMADgBGkLEABA+9ZuT481sxwze8HM\n3jGz582sd5x2u81ss5m9ZWbrU11npjOzSWa2zcy2m9mcOG0eMLN3zWyTmY1MdY3tSXPb08yKzWyf\nmb0ZfTR2QgkkmdkiM6s2s7ebaMNnM0HNbc+Qz2a7DRBxIWKrmVmWpJ9JulTS30maamZnN2hzmaRC\ndz9D0vWSHk55oe1EItsz6iV3Py/6uDOlRbYvjyqyLRvFZ7PFmtyeUS36bLbnAOFCxNYrkvSuu1e6\n+xFJyxXZrvVdLmmJJLn765J6m9kAoTGJbE8p8plEM6Kn63/URBM+my2QwPaUWvjZbM9frFyI2Hr5\nkv5U7/Wfo9OaalPVSBtEJLI9JWlM9JDLKjMbkZrSTkp8Ntteiz6b6TyNt1lciIiT0BuShrj7J9FD\nML+RdGaaawKkgM9mRgeIu0+INy/aGTSg3oWIH8RZxt7ozw/NbIUihxkIkIgqSUPqvR4UndawzeBm\n2iCi2e3p7gfrPX/WzP7VzHLd/a8pqvFkwmezDYV8NtvzIaxjFyJKTVyIaGY9o8+PXYj4X6kqsB3Y\nIGm4mRWYWVdJVyuyXetbKekaSTKz0ZL2HTt0iM9pdnvWP0ZvZkWKnEpPeMRnin9cns9my8XdniGf\nzYzeA2nGPZKeNLNvK3ohoiTVvxBRkcNfK6K3ODl2IeIL6So407h7rZndLOkFRf6ZWOTuW83s+shs\nf8Td/8PM/t7Mdkg6JOnadNacyRLZnpKuNLMbJR2RdFjSN9NXcWYzs2WSSiT1MbP3JJVJ6io+m0Ga\n254K+GxyISEAIEh7PoQFAEgjAgQAEIQAAQAEIUAAAEEIEABAEAIEABCEAAGaYGYHkrDMAjObmkC7\nZdG2t5oZ14sg4xAgQNOScaHUMEn/lEC7oe5eKalY0ktJqANoFQIESEB0sJ0XzezXZrbVzP6t3rwK\nM7vHzN42s3Vmdnp0+qNm9g/12h3bm7lL0rjooD23NrKupWa2RdJZZvaWIrfgWRW96wKQMdrzrUyA\nVBspaYSk9yW9YmYXuvur0Xkfufs5ZvYtSfdL+h+NvP/Y3sz3Jc129681thJ3n2ZmVyhyY8anJN3r\n7hzCQsZhDwRI3Hp33+uR+/9skjS03rzl0Z9PSBrdBusaJeltSedGfwIZhz0QIHGf1XteqxP/fryR\n50cV/SfNzEyRG9c1KToOw48V6SeZrMhAaQfN7Cvufkl46UDbYw8EaFqiQ3weO8R0taTXos93Szo/\n+vxySV2izw9Iym5sIe7+rCJ7H//P3c+VtEXSSMIDmYgAAZoW7yyshtNzzGyzpFskzYpO+4Wk4mhH\n+GhFbjkuRQ5J1ZnZW411okv6sqTNZtZFUmd3P9RIGyDtuJ070EpmViFpFANDoaNhDwRoPf4LQ4fE\nHggAIAh7IACAIAQIACAIAQIACEKAAACCECAAgCAECAAgyP8HH7CgBWlnaZYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faba1595e50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Inputs = [[1,0],[0,1]]\n",
    "Answers = [0,1,]\n",
    "\n",
    "Guesses = [logistic(np.dot(x,Weights)) for x in Inputs] #loop through inputs and find logistic(sum(input*weights))\n",
    "plt.plot(Guesses,'bo')\n",
    "plt.plot(Answers,'ro')\n",
    "plt.axis([-0.5,1.5,-0.5,1.5])\n",
    "plt.ylabel('Activation')\n",
    "plt.xlabel('Input #')\n",
    "plt.legend(['Guesses','Answers']);\n",
    "print Guesses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The guesses are in blue and the answers are in red. As you can tell, the guesses and the answers look almost nothing alike. Our network likes to guess around 0.6 while the correct answer is 0 in the first example and 1 in the second. \n",
    "\n",
    "Lets look at how backpropogation reduces the distance between our guesses and the correct answers. \n",
    "\n",
    "First, we want to know how the amount of error changes with an adjustment to a given weight. We can write this as \n",
    "\n",
    "$\\partial Error \\over \\partial Weight_{Input_{1}\\to.Output}$\n",
    "\n",
    "This change in error with changes in the weights has a number of different sub components. \n",
    "\n",
    "* Changes in error with changes in the output unit's activity: $\\partial Error \\over \\partial Output$\n",
    "* Changes in the output unit's activity with changes in this unit's input: $\\partial Output \\over \\partial net_{Output}$\n",
    "* Changes in the output unit's input with changes in the weight: $\\partial net_{Output} \\over \\partial Weight_{Input_{1}\\to.Output}$\n",
    "\n",
    "Through the [chain rule](https://en.wikipedia.org/wiki/Chain_rule) we know \n",
    "\n",
    "$\\frac{\\partial Error}{\\partial Weight_{Input_{1}\\to.Output}} = \\frac{\\partial Error}{\\partial Output} * \\frac{\\partial Output}{\\partial net_{Output}} * \\frac{\\partial net_{Output}}{\\partial Weight_{Input_{1}\\to.Output}}$\n",
    "\n",
    "This might look scary, but with a little thought it should make sense: (starting with the final term and moving left) When we change the weight of a connection to a unit, we change the input to that unit. When we change the input to a unit, we change its activity (written Output above). When we change a units activity, we change the amount of error.\n",
    "\n",
    "Let's break this down using our example. During this portion, I am going to gloss over some details about how exactly to derive the partial derivatives. [Wikipedia has a more complete derivation](https://en.wikipedia.org/wiki/Delta_rule).  \n",
    "\n",
    "In the first example, the input is [1,0] and the correct answer is [0]. Our network's guess in this example was about 0.56.\n",
    "\n",
    "$\\frac{\\partial Error}{\\partial Output} = -(target-Output) = -(0-0.56)$ \n",
    "\n",
    "$\\frac{\\partial Output}{\\partial net_{Output}} = Output(1-Output) = 0.56*(1-0.56)$ - Please note that this is specific to our example with a logistic activation function\n",
    "\n",
    "$\\frac{\\partial net_{Output}}{\\partial Weight_{Input_{1}\\to.Output}} = Input_{1} = 1$\n",
    "\n",
    "to summarize (the numbers used here are approximate)\n",
    "\n",
    "$\\frac{\\partial Error}{\\partial Weight_{Input_{1}\\to.Output}} = -(target-Output) * Output(1-Output) * Input_{1} = -(0-0.56) * 0.56(1-0.56) * 1 = 0.14$\n",
    "\n",
    "This is the direction we want to move in, but taking large steps in this direction can prevent us from finding the optimal weights. For this reason, we reduce our step size. We will reduce our step size with a parameter called the *learning rate* ($\\alpha$). $\\alpha$ is bound between 0 and 1. \n",
    "\n",
    "Here is how we can write our change in weights\n",
    "\n",
    "$\\Delta Weight_{Input_{1}\\to.Output} = \\alpha * \\frac{\\partial Error}{\\partial Weight_{Input_{1}\\to.Output}}$\n",
    "\n",
    "This is known as the [delta rule](https://en.wikipedia.org/wiki/Delta_rule). \n",
    "\n",
    "We will set $\\alpha$ to be 0.5. Here is how we will calculate the new $Weight_{Input_{1}\\to.Output}$.\n",
    "\n",
    "$Weight_{Input_{1}\\to.Output}^{\\prime} = Weight_{Input_{1}\\to.Output} - 0.5 * 0.14 = 0.25 - 0.5 * 0.14 = 0.18$\n",
    "\n",
    "Thus, $Weight_{Input_{1}\\to.Output}$ is shrinking which will move the output towards 0. Below I write the code to implement our backpropogation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "alpha = 0.5\n",
    "\n",
    "def delta_Output(target,Output):\n",
    "    return -(target-Output)*Output*(1-Output) #find the amount of error and derivative of activation function\n",
    "\n",
    "def update_weights(alpha,delta,unit_input):\n",
    "    return alpha*np.outer(delta,unit_input) #multiply delta output by all the inputs and then multiply these by the learning rate "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above I use the [outer product](https://en.wikipedia.org/wiki/Outer_product) of our delta function and the input in order to spread the weight changes to all lines connecting to the output unit.\n",
    "\n",
    "Okay, hopefully you made it through that. I promise thats as bad as it gets. Now that we've gotten through the nasty stuff, lets use backpropogation to find an answer to our problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def network_guess(Input,Weights):\n",
    "    return logistic(np.dot(Input,Weights.T)) #input by weights then through a logistic\n",
    "\n",
    "def back_prop(Input,Output,target,Weights):\n",
    "    delta = delta_Output(target,Output) #find delta portion\n",
    "    delta_weight = update_weights(alpha,delta,Input) #find amount to update weights\n",
    "    Weights = np.atleast_2d(Weights) #convert weights to array\n",
    "    Weights += -delta_weight #update weights\n",
    "    return Weights\n",
    "\n",
    "from random import choice, seed\n",
    "seed(1) #seed random number generator so that these results can be replicated\n",
    "\n",
    "Weights = np.array([0.25, 0.5])\n",
    "\n",
    "Error = []\n",
    "while True:\n",
    "    \n",
    "    Trial_Type = choice([0,1]) #generate random number to choose between the two inputs\n",
    "    \n",
    "    Input = np.atleast_2d(Inputs[Trial_Type]) #choose input and convert to array\n",
    "    Answer = Answers[Trial_Type] #get the correct answer\n",
    "    \n",
    "    Output = network_guess(Input,Weights) #compute the networks guess\n",
    "    Weights = back_prop(Input,Output,Answer,Weights) #change the weights based on the error\n",
    "    \n",
    "    Error.append(abs(Output-Answer)) #record error\n",
    "    \n",
    "    if len(Error) > 6 and np.mean(Error[-5:]) < 0.05: break #tell the code to stop iterating when mean error is < 0.05 in the last 5 guesses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems our code has found an answer, so lets see how the amount of error changed as the code progressed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEPCAYAAABRHfM8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUXFW59/Hv0x2SQAKZmTpkTkBQhmgCyNQEJYH3SgBR\nBgFFRZbCBVERVLxkeVXUl3tBQcVAhCsOiAYhKJNc6PAiQyIJMiUkIRAyB8hE5pB+3j+eKrp6Ph36\nVPXp+n3WqtXnnDp1etcR+5e999l7m7sjIiLSmopSF0BERLJBgSEiIokoMEREJBEFhoiIJKLAEBGR\nRBQYIiKSSOqBYWYTzGyumc0zsyubOafazGab2Ytm9ljaZRIRkbazNMdhmFkFMA84AVgGzATOcve5\nBef0Ap4ETnT3pWbW393fSq1QIiKyU9KuYYwF5rv7InffDtwJTGxwzjnAVHdfCqCwEBHpmNIOjCpg\nccH+ktyxQqOAvmb2mJnNNLPzUi6TiIjshC6lLgBRhtHAOKAH8JSZPeXuC0pbLBERKZR2YCwFBhXs\nD8wdK7QEeMvdtwBbzOxx4BCgXmCYmSa9EhHZCe5u7XGdtJukZgIjzGywmXUFzgKmNTjnXuBoM6s0\ns92Aw4E5TV3M3fVy55prril5GTrKS/dC90L3ouVXe0q1huHuO8zsEuBhIpymuPscM7so3vbJ7j7X\nzB4Cngd2AJPd/eU0yyUiIm2Xeh+Guz8I7N/g2K8a7F8HXJd2WUREZOdppHcGVVdXl7oIHYbuRR3d\nizq6F+lIdeBeezIzz0pZRUQ6CjPDM9LpLSIinYQCQ0REElFgiIhIIgoMERFJRIEhIiKJKDBERCQR\nBYaIiCSiwBARkUQUGCIikogCQ0REElFgiIhIIgoMERFJRIEhIiKJZCownnii1CUQESlfmZreHJyM\nFFdEpEPQ9OYiIlJ0CgwREUlEgSEiIokoMEREJJHMBUZtbalLICJSnjIXGBs2lLoEIiLlKXOBsX59\nqUsgIlKeFBgiIpKIAkNERBLJXGC8806pSyAiUp4yFxiqYYiIlEbqgWFmE8xsrpnNM7Mrm3j/ODNb\na2azcq+rW7qenpISESmNLmle3MwqgJuAE4BlwEwzu9fd5zY49XF3PyXJNTUOQ0SkNNKuYYwF5rv7\nInffDtwJTGzivMQzKSowRERKI+3AqAIWF+wvyR1r6Egze87M/mZmB7Z0QU1vLiJSGqk2SSX0LDDI\n3TeZ2UnAPcCo5k5WDUNEpDTSDoylwKCC/YG5Y+9x9w0F2w+Y2S/MrK+7r258uUlMnQpLlsC4cdVU\nV1enUWYRkcyqqamhpqYmlWunuuKemVUCrxCd3suBGcDZ7j6n4Jy93H1lbnsscJe7D2niWg5R1i98\nAW69NbVii4h0Gu254l6qNQx332FmlwAPE/0lU9x9jpldFG/7ZOAMM/sysB3YDJzZ2nVnzkyz1CIi\n0pTMrekNMGQIvPZaacsjIpIFZb+mt6YHEREpPgWGiIgkksnA2Lat1CUQESk/mQwMEREpPgWGiIgk\nosAQEZFEFBgiIpKIAkNERBLJbGDcfnupSyAiUl4yGxhTppS6BCIi5SWzgSEiIsWlwBARkUQUGCIi\nkkgmZ6vNq60Fa5c5GEVEOqeyn602T1Oci4gUT6YDY/36UpdARKR8ZDow1q0rdQlERMpHpgKjsrL+\nvgJDRKR4FBgiIpJIpgKjokFpN28uTTlERMpRpgKjYQ1DRESKJ9OBkZEhJCIinUKmA6O2tjTlEBEp\nR5kKjIZ9GH/9q2oZIiLFkqnAaFjDuP9+WLKkNGURESk3mQ4MEREpHgWGiIgkkqnAaNiHISIixZOp\nP8FN1TAOOQQ2bix+WUREyk3qgWFmE8xsrpnNM7MrWzhvjJltN7PTmzunqcBYs0bTnIuIFEOqgWFm\nFcBNwHjgIOBsMzugmfN+BDzU0vWa68NYs+b9llRERFqTdg1jLDDf3Re5+3bgTmBiE+f9O/BnYFVL\nF1NgiIiUTtqBUQUsLthfkjv2HjPbFzjV3X8JtLiMoAJDRKR0upS6AMANQGHfRrOhsWrVpIK96twL\nNmxo/0KJiGRRTU0NNTU1qVzbPMW5NczsCGCSu0/I7V8FuLv/uOCchflNoD+wEfiSu09rcC0/7DBn\n9uzGv+emm+Dii9P5DiIiWWZmuHuLrTdJpV3DmAmMMLPBwHLgLODswhPcfVh+28xuA+5rGBZ5Goch\nIlI6qQaGu+8ws0uAh4n+kinuPsfMLoq3fXLDj7R0veYCQxMQioikL9UmqfZkZj52rDNjRuP3unaF\nKVPg3HOLXy4RkY6sPZukMtXI01wNY9s2ePXV4pZFRKTcZCowrIWMXLmyeOUQESlHmQqMljq9FRgi\nIunKVGC8+27z761qcYy4iIi8X5kKjJZGdKuGISKSrkwFxltvNf+eAkNEJF2Zeqy2lWEabN4M3bsX\nqUAiIhlQto/VXnZZy++rliEikp5MBcbxx7f8vgJDRCQ9mQqMlsZhgJ6UEhFJU6YCo6VxGAMGqIYh\nIpKmTAVGr17Nv7fPPgoMEZE0ZSow9tmn+ff23VeBISKSpkwFxt57N/+eahgiIunKVGD07Nn8e/vu\nCytWFK8sIiLlJlOBAc0vljRoECxbVtyyiIiUk8wFRnP22w+WLNHqeyIiaek0gdGrV6y819IEhSIi\nsvMyGRif/WzjYxUVMHBg1DJERKT9ZTIwmhqPUVmpwBARSVMmA6N378bHVMMQEUlXJgPj29+GT3yi\n/jHVMERE0tVqYJhZpZldV4zCJNWtGwwfXv9YvoaxdGlpyiQi0tm1GhjuvgM4ughlaZOGzVJqkhIR\nSVeiFffM7JdAFfAnYGP+uLvfnV7RGpXBC8u6dWv91fVefhl27IAzz4SXXipWqUREOrb2XHGvS8Lz\nugNvA+MKjjlQtMBoqFs3GDYMFi6M/YqKmE9KNQwRkXQkCgx3vyDtguyMwgWVKivjcdsdO2D9ethj\nj9KVS0SkM0r0lJSZDTSzv5jZqtxrqpkNTLtwrdlrr7rtiooIkCFD4PXXS1UiEZHOK+ljtbcB04B9\nc6/7csdaZWYTzGyumc0zsyubeP8UM/uXmc02sxlmdlTSwu+3X912ZWX8LGymEhGR9pM0MAa4+23u\n/m7udTswoLUPmVkFcBMwHjgIONvMDmhw2iPufoi7HwZ8Abg1aeEvKGgoyy/fqsAQEUlH0sB428zO\nzY3JqDSzc4lO8NaMBea7+yJ33w7cCUwsPMHdNxXs9gRqE5aJ8ePh/vtjWzUMEZF0JQ2MzwOfBlYA\ny4EzgCQd4VXA4oL9Jblj9ZjZqWY2h2jq+nzCMgF1NQvVMERE0tXqU1JmVgmc7u6npFUId78HuMfM\njga+D3y8qfMmTZr03nZ1dTXV1dX06xf7qmGIiEBNTQ01NTWpXDvpwL0Z7j62zRc3OwKY5O4TcvtX\nAe7uP27hM68CY9x9dYPj3lRZ16yBvn3hzTehf3/YtCn2N22qq3WIiJSr9hy4l/RP6j/M7CYzO8bM\nRudfCT43ExhhZoPNrCtwFvG01XvMbHjB9miga8OwaEmfPvEzP+p7t90iMLRcq4hI+0o60vvQ3M/v\nFRxz6o/8bsTdd5jZJcDDRDhNcfc5ZnZRvO2TgU+a2fnANmAz0VfSJg0rHsOGwauvxtxSIiLSPlpt\nkso9GnuGu99VnCI1W44mm6SacsYZ0SSVf4JKRKRcFbVJyt1rgW+2xy8rlq98BZ56qtSlEBHpXJL2\nYTxiZt8ws/3MrG/+lWrJ3odjj4W1a+Hhh0tdEhGRziPpU1KvNXHY3X1Y+xep2TIkbpKK8+NnGz4i\nItLptGeTVKLA6Ah2NjBeeQVGjUqpUCIiHVzR+jDM7JsF259q8N4P26MAafnDH+Ln/vuXthwiIp1F\nizUMM5vl7qMbbje1n7a21jDc6wbubd8OXZI+QCwi0okU8ykpa2a7qf0OpXBxJa3CJyLy/rUWGN7M\ndlP7HdasWfB2krl1RUSkWa01Se0ANhK1iV2B/FTkBnR3911SL2FdWdrUJAUxcO/CC2OakCOPhCef\nTKlwIiIdlJ6SaoPPfx5uy60NuHAhDB3azgUTEenASjH5YGaNGFG3PX586cohIpJ1nT4wDjusbnvH\nDvjtb+Hee0tXHhGRrOr0TVIrVsA++8T2iBGwYEFsZ+Rri4i8L2qSaoO9967brqjQqG8RkZ3V6QMD\n4JFH4mdlJXTrFturVpWuPCIiWVQWgVFVFT8Ll2wdXbQx6iIinUNZBEa+Gaqiom4E+NKlpSuPiEgW\nlUVgVFTA1KnQr1/946+9ps5vEZGkyiIwAA48EF5/vf6xYcPgoYdKUhwRkcwpm8DYf3/YsKHxRIRP\nP12a8oiIZE3ZBIZZLN26enX942vXwvTppSmTiEiWlE1gABx3XONjs2fDJz8JtbXFL4+ISJaUVWAc\ne2zjY++8E1OfP/988csjIpIlZRUYBx8MAwbUP7ZmTfx89NHil0dEJEvKKjAqKmD+/PrH1qyBXXZR\nYIiItKasAgOgVy94/PG6/XXrYOTIeOR21iz1ZYiINKfsAgPgmGOgb9+6/f7945HbD38Y7rijdOUS\nEenIyjIwAI4/vm67Xz/YuDG2n302ahkaAS4iUl/qgWFmE8xsrpnNM7Mrm3j/HDP7V+71hJl9KO0y\nAVx+ed12voYBERbHHw8nn1yMUoiIZEeXNC9uZhXATcAJwDJgppnd6+5zC05bCBzr7uvMbAJwC3BE\nmuUCOOqoeJT24IOhTx/YujWOu9f1cZhFgFi7LD0iIpJtadcwxgLz3X2Ru28H7gQmFp7g7k+7+7rc\n7tNAVcples9BB8XPRYvqQmHLlvrnvPZasUojItKxpR0YVcDigv0ltBwIXwQeSLVEBSoqoqYxcGBd\nYOT7MvKGD4cXXihWiUREOq5Um6TawsyOBy4Ajm7unEmTJr23XV1dTXV19fv+vU88ET9vvjnComFg\nAMydCz17xuy2aqISkY6spqaGmpqaVK5tnuLjQGZ2BDDJ3Sfk9q8C3N1/3OC8g4GpwAR3f7WZa3ma\nZT31VLj3Xhg3rvEgvrvuikWYDj005p469NDUiiEi0q7MDHdvl3/mpt0kNRMYYWaDzawrcBYwrfAE\nMxtEhMV5zYVFMVx9dTRRNVXDMINt22L7sMPgt78tbtlERDqCVAPD3XcAlwAPAy8Bd7r7HDO7yMy+\nlDvtu0Bf4BdmNtvMZqRZpuZ85CPwxhuwYEHj9woDA+C885oOFhGRzizVJqn2lHaTVN5JJ8GDD9Y/\n9uc/x8jwcePqjt12G3zuc6kXR0TkfclSk1TmfPnLjY9t316/hgGxUt/XvqYR4SJSPhQYDTQ1wnvr\n1saBsXAhXH89PPVUccolIlJqCowGunRpvO73li2NA2Ndbqjh9dcXp1wiIqWmwGhCVYOhhVu31k0d\nkrduHey5JzzzTKwTPnGimqdEpHNTYDRj9epY6xuarmGsXx9zUG3ZAs89B9OmxWO5mzYVv6wiIsWg\nwGhGnz5w440xwnvduqabpHr3hs2b6x+fMqV4ZRQRKSYFRgv22QemT4df/xrefrv+e5s2RWA0nKzw\n0kvh1luLV0YRkWJRYLRi9Gj4+MfhZz9r/F7PnjG31Lvv1j9+4YXwj38Up3wiIsWiwEjg+9+HNWsa\nH+/WDXbdtW7xpUITJzY+JiKSZQqMBAYNirmmGurePV75R2wLvf02DB2aftlERIpFgZHQ1VfD1Kn1\nj+VrGGvXNv2Z11+Hyy5LvWgiIkWhwGiD00+Pkd133BH7774bgdFUc1Xez34Wj9yKiGSdAqONjjgC\nzj031sdwjyaplgLjxBNjfqrVq4tXRhGRNGi22p3kHq9LL4Xf/a75Zqkf/ABWrYI334zzRESKSbPV\ndgBmMbL7Jz+J8RotnTdpEvz+97F9442wY0fRiiki0m4UGO/TbrvB3/8e2/36NX7fDHr1qtu/9FK4\n+OLGI8dFRDo6BUY7qKqKsRjPPVd/kSWIwLAGlcFf/QqGDYPTTouah8JDRLJAfRgpqK2FRx+Fiy6K\nQDj88Mah0dDkyTFCXESkPbVnH4YCo0imT4fq6niqquH8U3mf+hR88YvxZJWISHtQYGTUqlXxCO4B\nB7R8XrduMHMmfOhDxSmXiHReCoyMe/ddOO88uPPOls/7z/+McDnyyJhufbfdilM+Eek8FBidxB/+\nAOec0/p5++0HixfDiy/CQQelXy4R6Tw0DqOTOPvsGPx32211q/s1Za+94ucHPwjf/Gbj5WJFRIpB\nNYwO5J//jA7vD3wAnnyy7vjQofDaa3X7J5wQa3R84xtQWVn8copIdqhJqpNzj/6NfHNVjx6wcWPj\n80aOjEdxx42LDvKuXYtbThHp+BQYZWLLluj4vvFGeOed+u916dJ4pT+ABx+Ej31MNQ8RCQqMMrN5\nMyxcGLWMV16B88+HU0+Fe+5p+vxjjoFFi2KZWPfoNBeR8qTAKHO1tbBpU4wiv+IKWL++6fOuuy76\nOQCeeSZqHcOGxSO6IlIeMvWUlJlNMLO5ZjbPzK5s4v39zexJM9tiZl9LuzydQUUF9OwJX/oSvPUW\nPPQQ9O1b/5wBA+CJJ+r2P/pR+MhH4rwPfxjmzIlai4hIUqnWMMysApgHnAAsA2YCZ7n73IJz+gOD\ngVOBNe7+381cSzWMBObNg6uugjFj4Hvfa34akt13r+sXOf10+O534eCDI4xEpPPIUg1jLDDf3Re5\n+3bgTmBi4Qnu/pa7Pws00YUrbTVqFNx9N3zrW/C//xv9GddeW/+cffet34l+991w2GFw4IHRwb5s\nWTR7iYgUSjswqoDFBftLcsekCD76UXj88ahxLFkSYdC7N5x1VtPn77FHrNdRVRXnXXwxzJ4dHeci\nImqAKBNVVXDJJTH54XXXxdNWd9wBw4fXnXPqqXXbhx8Ov/gFjB4dzVRXXx1LzC5dGp3s69YV/zuI\nSGl1Sfn6S4FBBfsDc8d2yqRJk97brq6uprq6emcvVdbMoulq1Cg499y64+5w0knwt7/B+PERFA8/\nHO9dd11MSXLaafHE1bJl0Xn+1a/C4MHx9FWV6o4iJVdTU0NNTU0q106707sSeIXo9F4OzADOdvc5\nTZx7DbDB3f+rmWup07tEamvh5Zfh2WejlpFfkrZQ797x+va3YwzIGWdEn4hGn4uUVqbGYZjZBOCn\nRPPXFHf/kZldBLi7TzazvYB/ArsDtcAG4EB339DgOgqMDmLt2mjaqqyEX/4yBhZWVcXEiBUVMRfW\nSy/VnX/eedGpfsop0ZR16KF6GkukWDIVGO1FgZENM2bA3nvDCy/EU1rXX9/8uWeeGVO8/+pX0K8f\nDBoEY8dG01e3bq0vaysirVNgSKa4w4YNMcDwzTdjfqzly+O9MWNidcG83/8ebrgh+kgOPjgGG37i\nE9FfogARaTsFhnQK27bFaPRly6KpasaM6HB/993oK/m3f4vzevaMwKmqggsuiD6V3r1jksUDDoh1\n0hUmIk1TYEintWMHrF4dU5vMnRvL0prB9Onw6qtRS3nqKfjMZ+CxxyJsIPpN9t4b+veHK6+MfpLv\nfAeOOw4GDozZfYcMiaYu9Z9IOVFgiBB9HfPnx6j1adNidt4ZM6IGss8+8f7KlfU/s8susH07/Pzn\nMTAR4OST4dOfjs77kSOjH6VnT9VapHNQYIi0YMeOWP+8R48YMzJ9Otx3H6xYUXeOWd0I9m7d6pa9\n7d495t8aMiQeDb7nHjjiCLj1Vpg6NcacLF4cU67ssw+sWhWf79VLNRfpmBQYIu/DunUwa1Y0eb34\nIlx2WXS033IL/PWvcPvtMd/WfxdMg1ldDTU1MX1Kfjr5a6+NObvyhg6NIFmwIPpfvv71WHZ3992j\n9jJ4cNRcevQo4peVsqfAECmC2tp4LVsWa6y/+GKMPznsMPiP/4gQWLu27vzJk2PKeYBDDoEjj4Sb\nb65/zV13jaB5++2omUybFk+GDR8eTWRjxkRgHXVUTMvSpUvMCbZ+fdSK+veP0BJJSoEh0kFs3BjB\nsX077LVXTKWy994xwv03v4layRtv1HW433ADHH10PPWVd9ppUSP5whfa9rvz/TGHHBLhMnRolGfI\nkJhs8qCDYvqXJUtg//0jbLZujaY0NZ+VDwWGSIZt3x7jUJYsgddfj6e/5s+Px4lnz46msvPPj8Bp\nb/kBkWPGxEj973wHJkyI/WOOiTnB+vWLAKqtjbLmnzIzixqSZIsCQ6QMuEeIvPpq9Lt06xY1iGee\niY73WbPi1bt3jEd55JHWr3nLLTEIcvTo2O/aNcbDAFx+ecsj8/M+8IEIkNWr4dhj4Zprotmue/d4\nSm3IkOivefnleOJsxIgIx+7dI4R69Kh7XFrSp8AQkWZt3hwhsGZN1GB6945ay9tvx3QsfftG01hl\nZayXcvfdMXble9+LUGrNyJFRI4Los3n+eXj66frn7Lln1E7WrIkpYk44oelr7bdfhN/w4REkkybF\nWJtddokmulmzYoDmoEFxnW3boklt333j+4wcWffEW5e0597OKAWGiKSutjb+EK9bF2vH77pr3R/n\nVaviia+qquikX7Aggqp//3iEecCAWENl3LjYfvPNZL+zsjL+8G/dWv/R5/vug3POqb9SZI8eMXXM\n9On1y/yDH8T4myuuiJrOqFHR1HbSSfEU3Nat0d90wgnRl7NxY/T/9OkTMyyvXRu1ohdfjGPLl0d/\n0BtvxAMH3bpF6GaFAkNEMmfz5roQyo/mf/nlCIahQ2Nxrpkzoz9l0aKYTn/BghhXs2ZN1DKWL4cH\nHojay4MPwuc+F+NrPvjB+ANfVRXXgVgE7Jln2lbG/HU2bIiw2G23CMxdd43y5913XzyosHFjBCfE\nJJq33hqzDORncu4IFBgiIgXco7lq5cqoIWzeHE1dW7ZE89iKFfHabbcImxEjoomra1e46654jPmB\nByIYli2rC4jCPp68vn1joOahh0aYTZsWxwsHgELMizZ4cNROSkmBISKSstraCI0uXaLTfvfdoza0\nxx4xDc2aNfF6552oIc2cGef16RNB9cYbESo//GFpv4cCQ0REEmnPwNDwHRERSUSBISIiiSgwREQk\nEQWGiIgkosAQEZFEFBgiIpKIAkNERBJRYIiISCIKDBERSUSBISIiiSgwREQkEQWGiIgkknpgmNkE\nM5trZvPM7MpmzvmZmc03s+fM7NC0yyQiIm2XamCYWQVwEzAeOAg428wOaHDOScBwdx8JXATcnGaZ\nOoOamppSF6HD0L2oo3tRR/ciHWnXMMYC8919kbtvB+4EJjY4ZyLwGwB3fwboZWZ7pVyuTNP/Gero\nXtTRvaije5GOtAOjClhcsL8kd6ylc5Y2cY6IiJSYOr1FRCSRVFfcM7MjgEnuPiG3fxXg7v7jgnNu\nBh5z9z/m9ucCx7n7ygbX0nJ7IiI7ob1W3OvSHhdpwUxghJkNBpYDZwFnNzhnGnAx8MdcwKxtGBbQ\nfl9YRER2TqqB4e47zOwS4GGi+WuKu88xs4vibZ/s7veb2clmtgDYCFyQZplERGTnpNokJSIinUcm\nOr2TDP7rLMxsoJk9amYvmdkLZnZp7ngfM3vYzF4xs4fMrFfBZ76VG/g4x8xOLF3p02FmFWY2y8ym\n5fbL8l6YWS8z+1Puu71kZoeX8b243MxeNLPnzex3Zta1XO6FmU0xs5Vm9nzBsTZ/dzMbnbt/88zs\nhkS/3N079IsItQXAYGAX4DnggFKXK8XvuzdwaG67J/AKcADwY+CbueNXAj/KbR8IzCaaF4fk7pWV\n+nu08z25HPgtMC23X5b3ArgduCC33QXoVY73AtgXWAh0ze3/EfhsudwL4GjgUOD5gmNt/u7AM8CY\n3Pb9wPjWfncWahhJBv91Gu6+wt2fy21vAOYAA4nv/D+50/4HODW3fQpwp7u/6+6vA/OJe9YpmNlA\n4GTg1oLDZXcvzGwP4Bh3vw0g9x3XUYb3IqcS6GFmXYBdifFbZXEv3P0JYE2Dw2367ma2N7C7u8/M\nnfebgs80KwuBkWTwX6dkZkOIf0k8DezluafH3H0FsGfutM4+8PF64AqgsLOtHO/FUOAtM7st1zw3\n2cx2owzvhbsvA/4LeIP4Xuvc/RHK8F4U2LON372K+Fual+jvahYCoyyZWU/gz8BluZpGw6cTOv3T\nCmb2f4CVuRpXS49Vd/p7QTQpjAZ+7u6jiScKr6I8/7voTfyLejDRPNXDzD5DGd6LFqTy3bMQGEuB\nQQX7A3PHOq1cNfvPwB3ufm/u8Mr8HFu56uSq3PGlwH4FH+9M9+co4BQzWwj8ARhnZncAK8rwXiwB\nFrv7P3P7U4kAKcf/Lj4GLHT31e6+A/gL8FHK817ktfW779Q9yUJgvDf4z8y6EoP/ppW4TGn7NfCy\nu/+04Ng04HO57c8C9xYcPyv3lMhQYAQwo1gFTZO7f9vdB7n7MOJ/90fd/TzgPsrvXqwEFpvZqNyh\nE4CXKMP/LoimqCPMrLuZGXEvXqa87oVRv9bdpu+ea7ZaZ2Zjc/fw/ILPNK/UPf4JnwqYQDwtNB+4\nqtTlSfm7HgXsIJ4Gmw3Myn3/vsAjufvwMNC74DPfIp5+mAOcWOrvkNJ9OY66p6TK8l4AhxD/gHoO\nuJt4Sqpc78U1ue/1PNHJu0u53Avg98AyYCsRnhcAfdr63YEPAy/k/q7+NMnv1sA9ERFJJAtNUiIi\n0gEoMEREJBEFhoiIJKLAEBGRRBQYIiKSiAJDREQSUWBIp2Zm7+R+Djazhqs9vt9rf6vB/hPtfP1R\nubmjzMyebM9ri+wMBYZ0dvmBRkOBc9ryQTOrbOWUb9f7Re5Ht+X6CRwDPA58iBhgJVJSCgwpF9cC\nR+dmer0styjTT8zsGTN7zswuBDCz48zscTO7l5h6AzP7i5nNtFjQ6ou5Y9cCu+aud0fu2Dv5X2Zm\n/zd3/r/M7NMF136sYBGkO5oqqJkdbWazgZ8A3wD+Bow3s6xPZyEZp5He0qmZ2Xp338PMjgO+7u6n\n5I5fCAxw9x/m5ij7B3AGscjMX4GD3P2N3Lm93X2tmXUnpuY41t3X5K/dxO/6JPAldx9vZnvmPjOW\nWAjrHmJH1edJAAABfUlEQVRRmxW53/kNd2+yucnM/uHuR5nZr4GfuPvc9r9DIsmphiHl6kTg/Ny/\n5J8h5iEamXtvRj4scr5qZs8R65IMLDivOUcRs+vi7quAGmBMwbWXe/xL7TkioBrJrXWxNbc7kpjv\nR6SkupS6ACIlYsC/u/vf6x2MmsjGBvvjgMPdfauZPQZ0L7hG0t+Vt7VgewdN/H8w1xx2ANDLzP5F\nrPsw08yudfc/JfydIu1ONQzp7PJ/rN8Bdi84/hDwldzaI5jZyNy/6hvqBazJhcUBwBEF723Lf77B\n7/p/wJm5fpIBROd14v4Hd58I3AJ8GbgUuNndRysspNQUGNLZ5TvpngdqzWy2mV3m7rcQayjMMrMX\ngJuJdaIbehDYxcxeAn4IPFXw3mTg+YLOawdw97/kft+/iCmnr8g1TTVXtqYcAzyR+zm99a8pkj51\neouISCKqYYiISCIKDBERSUSBISIiiSgwREQkEQWGiIgkosAQEZFEFBgiIpKIAkNERBL5/5RBRiy3\nPsDHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fab9db4c510>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Error_vec = np.array(Error)[:,0]\n",
    "plt.plot(Error_vec)\n",
    "plt.ylabel('Error')\n",
    "plt.xlabel('Iteration #');\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the while loop excecuted about 1000 iterations before converging. As you can see the error decreases. Quickly at first then slowly as the weights zone in on the correct answer. lets see how our guesses compare to the correct answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([ 0.05420561]), array([ 0.95020512])]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEPCAYAAABsj5JaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHMpJREFUeJzt3Xt0VPW99/HPN1yKYICEWyRAiMHLQ1ul4hNB0IRaEI+e\nao8eKyfURuvzeEGXslgt9akxpLFVWz0utbYuWxQpIseqeFjleOG0BsQbooI9FERCiG1AlFbkItqQ\nfJ8/ZhhjzCQzO3NL8n6tNYuZvX+z93f2msyHvX9775+5uwAAiFdWugsAAHRNBAgAIBACBAAQCAEC\nAAiEAAEABEKAAAACSXuAmNlCM9ttZm9FmV9iZnvN7I3w46ZU1wgA+KLe6S5A0kOS7pW0uJ02a9z9\nmymqBwAQg7Tvgbj7WkkfdtDMUlELACB2aQ+QGE02sw1mttLMxqe7GABAZhzC6sjrksa4+8dmdo6k\npyQdn+aaAKDHy/gAcfcDLZ4/bWa/NLNcd/9767Zmxo29ACBO7h6omyBTDmGZovRzmNmIFs+LJVlb\n4XGEu/NIwKOysjLtNXSnB9uT7Zmpj85I+x6ImS2VVCppiJm9K6lSUl9J7u4PSLrIzK6W1CjpkKRv\np6tWAMBn0h4g7v5vHcy/T9J9KSoHABCjTDmEhQxTWlqa7hK6FbZnYrE9M4N19hhYJjEz706fBwCS\nzczkATvR034IC0DPMnbsWNXX16e7jB6noKBAO3bsSOgy2QMBkFLh//Gmu4weJ9p278weCH0gAIBA\nCBAAQCAECAAgEAIEABAIAQIArSxbtkyTJk3S0Ucfrby8PE2ePFm/+tWv0l1WxiFAAGSMurp6zZ5d\npWnTKjV7dpXq6uI/3bezy7jzzjs1d+5czZ8/X7t379Z7772n+++/Xy+99JIaGxvjrqdbS/eNvBJ8\nUzAHkNmi/Z1u377Di4rmuXTAJXfpgBcVzfPt23fEvOzOLuOjjz7yAQMG+PLly6O2KS0t9YULF0Ze\nL1q0yKdOnRp5vXnzZp8+fbrn5ub6iSee6I899lhk3sqVK338+PGenZ3to0aN8jvvvNPd3ffs2ePn\nnXeeDx482HNzc/3MM8+MvGfnzp1+4YUX+rBhw/zYY4/1e+65JzJv3bp1fuqpp/rAgQM9Ly/P582b\nF7XuaNs9PD3Yb27QN2bigwABMl+0v9OysgUtfvg9EgBlZQtiXnZnl/HMM894nz59vKmpKWqbtgLk\njDPOcHf3gwcP+ujRo/3hhx/25uZm37Bhgw8dOtQ3b97s7u7HHHOMv/jii+7uvnfvXn/zzTfd3f3G\nG2/0q6++2puamvzw4cO+du1ad3dvbm72iRMn+i233OKHDx/2uro6Lyoq8ueee87d3SdPnuxLliyJ\nrPvVV1+NWncyAoRDWAAyQkNDs6QBraYO0M6dzSlbxp49ezR06FBlZX320zhlyhTl5ORowIABeuGF\nF9p9/+9//3sVFhbq0ksvlZnp5JNP1oUXXqjf/e53kqS+fftq06ZN2r9/vwYNGqQJEyZIkvr06aNd\nu3aprq5OvXr10pQpUyRJr732mvbs2aMf/ehH6tWrl8aOHasrrrhCy5Yti7xv27Zt+tvf/qb+/fur\nuLg4ps+ZKAQIgIyQn58l6WCrqQc1cmTsP1OdXcaQIUO0Z88eNTd/FjgvvviiPvzwQw0ZMuTIkY6o\n6uvr9corryg3N1e5ubnKycnR0qVLtXv3bknSE088oZUrV6qgoEDTpk3TK6+8Ikn6wQ9+oKKiIs2Y\nMUPjxo3T7bffHlleQ0PD55Z366236v3335ckPfjgg3r77bd14okn6rTTTtPKlStj+pwJE3TXJRMf\n4hAWkPGi/Z1mQh/I3r17/eijj/Ynn3zyC/NGjRrlq1ev9nPPPdfvvffeyPTbbrstcgjr0Ucf9Rkz\nZnS4nsOHD/tdd93lo0eP/sK8TZs2+fDhw/2Pf/yjv/zyy3788cfHVPvjjz/u/fr1848//rjN+dG2\nuziEBaCrKyws0KpV16ms7A5Nm1apsrI7tGrVdSosLEjZMgYNGqSbb75Z11xzjZ544gkdOHBA7q4N\nGzbo448/liRNmDBBTz75pA4dOqRt27Zp4cKFkfefd9552rp1q5YsWaLDhw+rsbFR69ev15YtW9TY\n2KilS5dq37596tWrl7Kzs9WrVy9J0sqVK1VbWytJys7OVu/evZWVlaXi4mJlZ2frZz/7mT755BM1\nNTVp06ZNWr9+vSTpkUce0Z49eyK1m9nnDr8lXdDkycSH2AMBMl5X+DtdunSpFxcX+4ABA3z48OE+\nadIk/81vfuONjY2+Z88enzFjhg8cONCnTp3qVVVVkT0Qd/etW7f6ueee68OGDfOhQ4f6WWed5Rs3\nbvR//OMfPnPmTM/NzfVBgwZ5cXGxv/TSS+7uftddd/nYsWP96KOP9tGjR/tPfvKTyPJ27drls2bN\n8ry8PM/NzfXJkyf7H/7wB3d3nz17tg8fPtyzs7P9K1/5iq9YsSLqZ4q23dWJPRDuxgsgpbgbb3pw\nN14AQMYgQAAAgRAgAIBACBAAQCAECAAgEAIEABAIAQIACIQAAQAEQoAAAAIhQACgDaWlpcrNzWUU\nwnYQIAAyRn1dnapmz1bltGmqmj1b9XV16VlGfb3Wrl2rrKwsrVixIu73p0JTU1O6S+BmigBSK9rf\n6Y7t231eUZEfCA8leEDyeUVFvmP79piXnYhluLv/+Mc/9qlTp/q8efP8vPPOi0wvLy/3OXPm+Lnn\nnuvZ2dk+adIk395i2TfccIMPHz7cBw4c6CeddJJv2rTJ6+rqfPDgwZE2V1xxhQ8fPjzy+jvf+Y7f\nfffd7h4aUvd73/ueH3PMMT5q1Ci/6aabvLm52d1DIx9OmTLF586d60OGDPGKigrftm2bl5SU+KBB\ng3zYsGF+ySWXRP1M0ba7GNKWAAG6imh/pwvKyiI//N4iABaUlcW87EQsw9193Lhxfv/99/vrr7/u\nffr08ffff9/dQwEydOhQX79+vTc1NXlZWZnPmjXL3d2fffZZP/XUU33fvn3u7r5lyxZ/77333N29\noKDA33jjDXd3P+GEE7yoqMi3bNni7u5jxozxjRs3urv7BRdc4FdffbUfOnTIP/jgAz/ttNP8gQce\ncPdQgPTu3dvvu+8+b2pq8kOHDvmsWbP8pz/9qbu7f/rpp5HhctuSjADhEBaAjNDc0NDGYLRS886d\nKV3G2rVr9e677+riiy/WKaeconHjxmnp0qWR+d/61rc0ceJEZWVlqaysTBs2bJAUGl52//79+vOf\n/yx31wknnKARI0ZIks4880ytXr06MjLhRRddpNWrV2vHjh3av3+/TjrpJO3evVtPP/207rrrLvXr\n109Dhw7VDTfcoEcffTSy7vz8fF1zzTXKyspSv3791KdPn8iohX379tXpp58e8+dMBAIEQEbIys9v\nYzBaKWvkyJQuY/HixZoxY4ZycnIkSbNmzdLDDz8cmZ+Xlxd53r9/fx04cECSNG3aNF177bWaM2eO\nRowYoauuuioyr6SkRM8//7zWrFmjkpISlZaWqqamRqtXr9YZZ5whSXr33XfV2NioY445JjJ87VVX\nXRUZMEqSRo8e/blaf/7zn6u5uVnFxcX66le/qoceeijmz5kQQXddMvEhDmEBGS/a32km9IEcOnTI\nBw0a5NnZ2Z6Xl+d5eXmek5PjWVlZvnHjRi8vL/eKiopI+5qamjaHpf3ggw+8tLTUb775Znd3f+ed\ndzwnJ8fnzJnjjzzyiO/bt8/HjBnjl112md95553uHho4qn///pE+j9YWLVr0uYGrWlu7dq3369fP\na2tr25wfbbuLQ1gAurqCwkJdt2qV7igrU+W0abqjrEzXrVqlgsLClC1j+fLl6t27tzZv3qyNGzdq\n48aN2rJli8444wwtXry43feuX79e69at0+HDh3XUUUepX79+keFlx40bp6OOOkpLlixRSUmJsrOz\nNWLECD355JMqKSmRFNqzmTFjhubOnav9+/fL3bV9+3atWbMm6joff/xxNTQ0SJIGDx6srKyslA5p\n2ztlawKADhQUFqpyyZK0LWPx4sW6/PLLlZ+f/7npc+bM0fXXX69vfOMbUd+7b98+zZ07V3V1derX\nr5/OPvtsff/734/MLykp0bp16yLLLikp0datW3XKKad8bv3z58/X+PHjdeDAAR177LGaP39+1HW+\n9tpruuGGG7Rv3z6NGDFC99xzj8aOHRvoswfBkLYAUoohbdODIW0BABmDAAEABEKAAAACIUAAAIEQ\nIACAQAgQAEAgXAcCIKUKCgpkFuisUXRCQUFBwpfJdSAA0INxHQgAIOUIEABAIGkPEDNbaGa7zeyt\ndtrcY2bvmNkGM5uQyvoAAG1Le4BIekjS2dFmmtk5korc/ThJV0q6P1WFAcgsiRjvHImT9rOw3H2t\nmbV3esD5khaH275qZoPMbIS7705NhQAyQX1dne6dPl1VtbUaoNBAUZWvvBL3Ld+ROJmwB9KRfEl/\nafG6ITwNQA+yqKIiEh5SaKjaqtpaLaqoSGdZPVra90ASbcGCBZHnpaWlKi0tTVstABInEeOdQ6qp\nqVFNTU1CltUVAqRBUsuBgEeFp7WpZYAA6D6OjHfeMkTiHe8cX/yPdVVVVeBlZcohLAs/2rJC0qWS\nZGaTJO2l/wPoecqrq1VZVKSD4dcHJVUWFam8ujqdZfVoab8S3cyWSiqVNETSbkmVkvoqNND7A+E2\nv5A0U6HvzGXu/kaUZXElOtCNrV2zVj/+7jX60t79+nRwtm5++JeaeubUdJfVpXXmSvS0B0giESBA\n91VXV6/p0+9VbW2VFD4Pq6ioUqtWXafCwsTf56mn4FYmALq9iopFLcJDkgaotrZKFRWL0lhVz0aA\nAOgSGhqapTbOw9q5szkd5UAECIAuIj8/S4p0oR9xUCNH8jOWLmx5AF1CdXW5iooqpRbnYRUVVaq6\nujxtNfV0dKID6DLq6upVUbFIO3c2a+TILFVXl9OB3kmchRVGgABAfDgLCwCQcgQIACAQAgQAEAgB\nAgAIhAABAARCgAAAAiFAAACBECAAgEAIEABAIAQIACAQAgQAEAgBAgAIhAABAARCgAAAAiFAAACB\nECAAgEAIEABAIAQIACAQAgQAEAgBAgAIhAABAARCgAAAAiFAAACB9I6lkZmdLmlsy/buvjhJNQEA\nuoAOA8TMfiupSNIGSU3hyS6JAAGAHiyWPZBTJY13d092MQCAriOWPpD/kZSX7EIAAF1LLHsgQyX9\n2czWSfr0yER3/2bSqgIAZLxYAmRBsosAAHQ9FkvXhpmNkPS/wy/Xufv7Sa0qIDOjqwYA4mBmcncL\n8t4O+0DM7GJJ6yT9q6SLJb1qZhcFWRkAoPvocA/EzDZKmn5kr8PMhkn6b3c/OQX1xYU9EACIT1L3\nQCRltTpk9bcY3wcA6MZi6UR/xsyelfRo+PW3Jf1X8koCAHQFsXaiXyhpSvjlC+6+PKlVBcQhLACI\nT2cOYcUUIF0FAQIA8elMgEQ9hGVma919qpntV+jeV5FZktzdBwZZIQCge2APBAB6sGRfB/LbWKYB\nAHqWWE7H/XLLF2bWW9LE5JQDAOgqogaImd0Y7v84ycz2hR/7Je2W9J8pqxAAkJGiBoi73+ru2ZJ+\n7u4Dw49sdx/i7jcmqgAzm2lmW8xsq5nNb2N+iZntNbM3wo+bErVuAEBwsV4HkiPpOEn9jkxz9zWd\nXrlZlqStks6StFPSa5IucfctLdqUSJoXy+3j6UQHgPgk5TTeFgu/QtL1kkYpNKztJEkvS/p6kBW2\nUizpHXevD69rmaTzJW1p1S7QhwMAJE8snejXK3Qr93p3nybpa5L2Jmj9+ZL+0uL1X8PTWptsZhvM\nbKWZjU/QugEAnRDLvbA+cfdPzExm9iV332JmJyS9ss+8LmmMu39sZudIekrS8dEaL1iwIPK8tLRU\npaWlya4PALqMmpoa1dTUJGRZsdzOfbmkyyTdoNBhqw8l9XH3f+r0ys0mSVrg7jPDr3+o0FXut7fz\nnjpJE939723Mow8EAOKQsnthhTu0B0l6xt3/EWSFrZbXS9LbCnWi71Jo4KpZ7r65RZsR7r47/LxY\n0mPuPjbK8ggQAIhDsjvR75G0zN1fcvfVQVYSjbs3mdm1kp5TqD9mobtvNrMrQ7P9AUkXmdnVkhol\nHVLodvIAgDSL5RDWdxX60T5B0nKFwmR9CmqLG3sgABCflBzCMrNcSRdKukShTu3jgqwwmQgQAIhP\nsoe0PWKcpBMlFeiL12kAAHqYWA5h/UzStyTVSvoPScvdPVHXgSQUeyAAEJ+kdqIrFByT3X1PkBUA\nALqnqHsgZnZi+KLBU9qa7+5vJLWyANgDAYD4JKUT3cwecPf/a2bPtzHb3T0R98JKKAIEAOKT1LOw\nzKyfu3/S0bRMQIAAQHySfRbWSzFOAwD0IFE70c0sT6E74x5lZl/TZ7dUHyipfwpqAwBksPbOwjpb\nUrlC44Dcqc8CZJ+k/5fcsgAAmS6WPpAL3f2JFNXTKfSBAEB8kt0HMtHMBrdYWY6Z3RJkZQCA7iOW\nADmn5ZXn7v6hpE6PBQIA6NpiCZBeZvalIy/M7ChJX2qnPQCgB4jlViaPSPqDmT2kUEd6uaSHk1kU\nACDzxXQ7dzObKekbklyhs7Dy3H1OkmuLG53oABCfVNzOfbdC4fGvCo2Lvrn95gCA7q69CwmPlzQr\n/Nij0K3czd2npag2AEAGa+9mis2SXpD0PXffFp623d2PTWF9ceEQFgDEJ1mHsP5F0i5Jz5vZr83s\nLH12NToAoIeL5Ur0AZLOV+hQ1tclLVZoVMLnkl9efNgDAYD4JPV27q1WlKNQR/q33f2sICtMJgIE\nAOKTsgDJdAQIAMQnFafxAgDwOQQIACAQAgQAEAgBAgAIhAABAARCgAAAAiFAAACBECAAgEAIEABA\nIAQIACAQAgQAEAgBAgAIhAABAARCgAAAAiFAAACBECAAgEAIEABAIAQIACAQAgQAEEjvdBeAzFJX\nV6+KikVqaGhWfn6WqqvLVVhYkO6yAGQgc/d015AwZubd6fOkWl1dvaZPv1e1tVWSBkg6qKKiSq1a\ndR0hAnRTZiZ3tyDv5RAWIioqFrUID0kaoNraKlVULEpjVQAyFQGCiIaGZn0WHkcM0M6dzekoB0CG\nI0AQkZ+fJelgq6kHNXIkXxMAX5T2XwYzm2lmW8xsq5nNj9LmHjN7x8w2mNmEVNfYU1RXl6uoqFKf\nhUioD6S6ujxtNQHIXGkNEDPLkvQLSWdL+rKkWWZ2Yqs250gqcvfjJF0p6f6UF9pDFBYWaNGDF2j6\n2Mn658FjNX3sZC168AI60AG0Kd17IMWS3nH3endvlLRM0vmt2pwvabEkufurkgaZ2YjUltkz1NfV\n6anLy7V8x5+0Ym+9lu/4k566vFz1dXXpLg1ABkp3gORL+kuL138NT2uvTUMbbZAAiyoqVFVb2+Ic\nLKmqtlaLKirSWRaADNXtLiRcsGBB5HlpaalKS0vTVktX09zQ0MY5WFLzzp3pKAdAEtTU1KimpiYh\ny0p3gDRIGtPi9ajwtNZtRnfQJqJlgCA+Wfn5OqjPn8h7UFLWyJFpqghAorX+j3VVVVXgZaX7ENZr\nksaZWYGZ9ZV0iaQVrdqskHSpJJnZJEl73X13asvsGcqrq1VZVNTiHCypsqhI5dXV6SwLQIZK+61M\nzGympLsVCrOF7n6bmV0pyd39gXCbX0iaqdBv2mXu/kaUZXErk06qr6vToooKNe/cqayRI1VeXa2C\nwsJ0lwUgSTpzK5O0B0giESAAEB/uhQUASDkCBAAQCAECAAiEAAEABEKAAAACIUAAAIEQIACAQAgQ\nAEAgBAgAIBACBAAQCAECAAiEAAEABEKAAAACIUAAAIEQIACAQAgQAEAgBAgAIBACBAAQCAECAAiE\nAAEABEKAAAACIUAAAIEQIACAQAgQAEAgBAgAIBACBAAQCAECAAiEAAEABEKAAAACIUAAAIEQIACA\nQAgQAEAgBAgAIBACBAAQCAECAAiEAAEABEKAAAACIUAAAIEQIACAQAgQAEAgBAgAIBACBAAQCAEC\nAAiEAAEABEKAAAACIUAAAIH0TteKzSxH0n9IKpC0Q9LF7v5RG+12SPpIUrOkRncvTmGZAIAo0rkH\n8kNJ/+3uJ0j6o6Qbo7RrllTq7l8jPFKnpqYm3SV0K2zPxGJ7ZoZ0Bsj5kh4OP39Y0gVR2pk41JZy\n/IEmFtszsdiemSGdP8zD3X23JLn7e5KGR2nnklaZ2Wtm9n9SVh0AoF1J7QMxs1WSRrScpFAg3NRG\nc4+ymCnuvsvMhikUJJvdfW2CSwUAxMnco/1uJ3nFZpsV6tvYbWZ5kp539//VwXsqJe1393+PMj89\nHwYAujB3tyDvS9tZWJJWSCqXdLuk70r6z9YNzKy/pCx3P2BmAyTNkFQVbYFBNwIAIH7p3APJlfSY\npNGS6hU6jXevmR0j6dfufp6ZFUpartDhrd6SHnH329JSMADgc9IWIACArq3Lnh5rZjlm9pyZvW1m\nz5rZoCjtdpjZRjN708zWpbrOTGdmM81si5ltNbP5UdrcY2bvmNkGM5uQ6hq7ko62p5mVmNleM3sj\n/GjrhBJIMrOFZrbbzN5qpw3fzRh1tD2DfDe7bICICxE7zcyyJP1C0tmSvixplpmd2KrNOZKK3P04\nSVdKuj/lhXYRsWzPsDXufkr4cUtKi+xaHlJoW7aJ72bc2t2eYXF9N7tygHAhYucVS3rH3evdvVHS\nMoW2a0vnS1osSe7+qqRBZjZCaEss21MKfSfRgfDp+h+204TvZhxi2J5SnN/NrvzDyoWInZcv6S8t\nXv81PK29Ng1ttEFILNtTkiaHD7msNLPxqSmtW+K7mXhxfTfTeRpvh7gQEd3Q65LGuPvH4UMwT0k6\nPs01AVKA72ZGB4i7T482L9wZNKLFhYjvR1nGrvC/H5jZcoUOMxAgIQ2SxrR4PSo8rXWb0R20QUiH\n29PdD7R4/rSZ/dLMct397ymqsTvhu5lAQb6bXfkQ1pELEaV2LkQ0s6PDz49ciPg/qSqwC3hN0jgz\nKzCzvpIuUWi7trRC0qWSZGaTJO09cugQX9Dh9mx5jN7MihU6lZ7wiM4U/bg83834Rd2eQb6bGb0H\n0oHbJT1mZpcrfCGiJLW8EFGhw1/Lw7c4OXIh4nPpKjjTuHuTmV0r6TmF/jOx0N03m9mVodn+gLv/\nl5n9k5ltk3RQ0mXprDmTxbI9JV1kZldLapR0SNK301dxZjOzpZJKJQ0xs3clVUrqK76bgXS0PRXg\nu8mFhACAQLryISwAQBoRIACAQAgQAEAgBAgAIBACBAAQCAECAAiEAAHaYWb7k7DMAjObFUO7peG2\n15sZ14sg4xAgQPuScaFUoaR/i6HdWHevl1QiaU0S6gA6hQABYhAebOd5M/udmW02s9+2mFdnZreb\n2Vtm9oqZHRue/pCZ/UuLdkf2Zm6VNDU8aM/1baxriZltknSCmb2p0C14VobvugBkjK58KxMg1SZI\nGi/pPUkvmtnp7v5SeN6H7n6SmX1H0t2S/rmN9x/Zm/mhpHnu/s22VuLus83sQoVuzPi4pDvcnUNY\nyDjsgQCxW+fuuzx0/58Nksa2mLcs/O+jkiYlYF0TJb0l6eTwv0DGYQ8EiN2nLZ436fN/P97G88MK\n/yfNzEyhG9e1KzwOw08V6ic5V6GB0g6Y2dfd/azgpQOJxx4I0L5Yh/g8cojpEkkvh5/vkHRq+Pn5\nkvqEn++XlN3WQtz9aYX2Pv7k7idL2iRpAuGBTESAAO2LdhZW6+k5ZrZR0nWS5oan/VpSSbgjfJJC\ntxyXQoekms3szbY60SV9TdJGM+sjqbe7H2yjDZB23M4d6CQzq5M0kYGh0NOwBwJ0Hv8LQ4/EHggA\nIBD2QAAAgRAgAIBACBAAQCAECAAgEAIEABAIAQIACOT/A0zIkJJpPiWxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fab9db2d150>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Inputs = [[1,0],[0,1]]\n",
    "Answers = [0,1,]\n",
    "\n",
    "Guesses = [logistic(np.dot(x,Weights.T)) for x in Inputs] #loop through inputs and find logistic(sum(input*weights))\n",
    "plt.plot(Guesses,'bo')\n",
    "plt.plot(Answers,'ro')\n",
    "plt.axis([-0.5,1.5,-0.5,1.5])\n",
    "plt.ylabel('Activation')\n",
    "plt.xlabel('Input #')\n",
    "plt.legend(['Guesses','Answers']);\n",
    "print Guesses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad! Our guesses are much closer to the correct answers than before we started running the backpropogation procedure! Now, you might say, \"HEY! But you haven't reached the *correct* answers.\" That's true, but note that acheiving the values of 0 and 1 with a logistic function are only possible at -$\\infty$ and $\\infty$, respectively. Because of this, we treat 0.05 as 0 and 0.95 as 1.\n",
    "\n",
    "Okay, all this is great, but that was a really simple problem, and I said that neural networks could solve interesting problems! \n",
    "\n",
    "Well... this post is already longer than I anticipated. I will follow-up this post with another post explaining how we can expand neural networks to solve more interesting problems. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}