{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Exercise 2 - Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook covers a Python-based solution for the second programming exercise of the machine learning class on Coursera.  Please refer to the [exercise text](https://github.com/jdwittenauer/ipython-notebooks/blob/master/exercises/ML/ex2.pdf) for detailed descriptions and equations.\n",
    "\n",
    "In this exercise we'll implement logistic regression and apply it to a classification task.  We'll also improve the robustness of our implementation by adding regularization to the training algorithm. and testing it on a more difficult problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first part of this exercise, we'll build a logistic regression model to predict whether a student gets admitted to a university.  Suppose that you are the administrator of a university department and you want to determine each applicant's chance of admission based on their results on two exams. You have historical data from previous applicants that you can use as a training set for logistic regression.  For each training example, you have the applicant's scores on two exams and the admissions decision.  To accomplish this, we're going to build a classification model that estimates the probability of admission based on the exam scores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by examining the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Exam 1</th>\n",
       "      <th>Exam 2</th>\n",
       "      <th>Admitted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 34.623660</td>\n",
       "      <td> 78.024693</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 30.286711</td>\n",
       "      <td> 43.894998</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 35.847409</td>\n",
       "      <td> 72.902198</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 60.182599</td>\n",
       "      <td> 86.308552</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 79.032736</td>\n",
       "      <td> 75.344376</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Exam 1     Exam 2  Admitted\n",
       "0  34.623660  78.024693         0\n",
       "1  30.286711  43.894998         0\n",
       "2  35.847409  72.902198         0\n",
       "3  60.182599  86.308552         1\n",
       "4  79.032736  75.344376         1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "path = os.getcwd() + '\\data\\ex2data1.txt'\n",
    "data = pd.read_csv(path, header=None, names=['Exam 1', 'Exam 2', 'Admitted'])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a scatter plot of the two scores and use color coding to visualize if the example is positive (admitted) or negative (not admitted)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0xd17d7b8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAtgAAAHuCAYAAAClGHsoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt8nGWd///3NVNC2lB6wh6gUEAt2LpoovJzV5LMECgt\n",
       "pciKBZb9URCLFn8qrM3+LIuxaauASOpp1ULRbYMcKyu/ioW2pLmbuuKCnVjlZEV/ICeLnIJt2qad\n",
       "XN8/ZiYk00mTTO6Z+zCv5+MxD2bue2buaw6h7/uaz3VdxlorAAAAAO6IeN0AAAAAIEwI2AAAAICL\n",
       "CNgAAACAiwjYAAAAgIsI2AAAAICLCNgAAACAiwoWsI0xPzbG7DLG/L7XtvnGmCeNMUljTFXW/a8z\n",
       "xvzRGPOMMWZWodoFAAAAFFIhe7D/S9LsrG2/l/TPktp6bzTGzJB0saQZ6cf8wBhD7zoAAAACp2Ah\n",
       "1lq7TdKbWduesdbuzHH3j0u621p7wFr7nKRnJZ1eqLYBAAAAheKXXuJjJb3Y6/aLko7zqC0AAABA\n",
       "3kZ43YDDOGQNd2MM67oDAACgKKy1Jp/H+aUH+yVJx/e6PTW97RDWWi6DuCxdutTzNgThwvvEe8X7\n",
       "xPvk9wvvFe8T75U3l+HwMmD3PiNYL+kSY0yZMeYkSe+V9Jg3zQIAAADyV7ASEWPM3ZJqJR1jjHlB\n",
       "0lJJb0j6nqRjJP3CGNNurZ1jrX3KGHOfpKckHZT0OTvcUwcAAADAAwUL2Nbaf+ln1wP93P8GSTcU\n",
       "qj2lJhaLed2EQOB9Gjzeq8HhfRoc3qfB470aHN6nweO9KjwTpI5iYwwd2wAAACg4Y4xsnoMc/TyL\n",
       "CAAAgGeMyStbIYDc7sAlYAMAAPSDX87DrxAnUn6Zpg8AAAAIBQI2AAAA4CICNgAAAOAiAjYAAADg\n",
       "IgI2AABAARw4cMAXgySvuOIKNTQ05PXYG2+8UVdddZXLLeorEonoz3/+c0GPUWwEbAAAABfdf//9\n",
       "OuWUD+vII0dq5MijtWDBZ7Vr166CHCsWi2n8+PHq6urq9z7GmLxnyrjuuuu0evVqSdJzzz2nSCSi\n",
       "7u7unv1r1qxRdXV1Xs8dZgRsAACAQdq7d68aGpbpmGOmacSII3XqqR/R3Xff3bP/1ltv14IF/66d\n",
       "O1fI2i7t3/9H3X33Ufrwh2v01ltv9dzv1Vdf1W233aZvfetbam9vz6stzz33nB577DFNnDhR69ev\n",
       "P+x93exJ90OvvN8RsAEAAAbh4MGDOvPMebrllh16/fWfK5l8S3/4w3ItXLhcX/vaN7R//379+79f\n",
       "r87O9ZLmKBWzJuvgwSa9/vqHdOutqZ7gW275tqZNO0X/9m9tuu66P+mMMz6us876uDo7O4fUnubm\n",
       "Zp111lm67LLLtHbt2p7t7e3tqqqq0tFHH61LLrlE+/bt69nnOI6mTp2qb37zm5o4caKOPfZYPfDA\n",
       "A9qwYYOmT5+uCRMm6Kabbuq5f2Njoy677DJJUk1NjSRp7NixOvroo/XrX/9aixYt0qOPPqrRo0dr\n",
       "/PjxkqT9+/ervr5e06ZN0+TJk3X11Vf3acM3v/lNHXvssZo6dap+/OMfD+k1BwUBGwAAYBDWr1+v\n",
       "J57YrX371kk6TdJISXPU2fmIvv71m7R582YZc4Kk9x/y2L17r9Bdd/1cGzdu1NKl39G+fb9TZ+dP\n",
       "tH//f6qz80/6n/+p0NVXf2lI7WlubtbFF1+siy66SBs3btTf/vY3dXV16YILLtDll1+uN998U/Pn\n",
       "z9f999/fp0Rk165d2r9/v1555RUtX75cCxcu1J133qn29nZt27ZNy5cv1/PPPy+p7yIs27ZtkyR1\n",
       "dHTo7bff1kc/+lHdeuut+sd//Ef9/e9/1xtvvCFJWrJkiZ599lnt2LFDzz77rF566SUtX75ckvTw\n",
       "ww+rqalJjzzyiHbu3KlHHnlkSK85KAjYAAAAg3DnnQ9o9+4rJUWz9hynI46I6bHHHhvwOb7+9e+q\n",
       "s7NR0vG9th6hffu+o/vuu7dPGcnh/PKXv9RLL72k888/X+9973s1Y8YM3Xnnnfr1r3+tgwcP6ppr\n",
       "rlE0GtWFF16oj3zkI30ee8QRR+j6669XNBrVxRdfrDfeeEPXXnutKioqNGPGDM2YMUM7duyQ1Lcc\n",
       "JFdpSPY2a61Wr16tlStXauzYsTrqqKN03XXX6Z577pEk3Xfffbryyis1Y8YMjRo1SsuWLRvU6w0a\n",
       "AjYAAMAgHDyYlHREzn3WlunEE0+UtX+R9MQh+0eOXKNLL52nP/zhGUn/lOMZ3qWysqn6y1/+Mqi2\n",
       "rF27VrNmzdLo0aMlSfPnz9fatWv1yiuv6Ljjjutz32nTpvW5PWHChJ6e6ZEjR0qSJk2a1KutI7V7\n",
       "9+5BtSPb3/72N3V2dupDH/qQxo0bp3HjxmnOnDl67bXXJEmvvPKKjj/+nZOLE044Ia/j+N0IrxsA\n",
       "AAAQBPPnz9GWLWu0e/cVknrPyvGGDh7crDlzvq0DB7r1pS/NU2fn9yXNlvSqRoy4WRMmbNdnP/sD\n",
       "3XPPg3r11SclvTfr2Xdr//4XNXny5AHbsXfvXt13333q7u7WlClTJKXqnjs6OjRlyhS99NJLfe7/\n",
       "/PPP6z3veU/+Lzwt10wk2duOOeYYjRw5Uk899VRP23qbMmVKn5OIwZ5QBA092AAAAIMwf/58HXfc\n",
       "myor+5ykv6a3/lajRs3VwoVXasqUKfrsZxequfkWTZ/+VRlTpiOPfK/+5V/26De/adPYsWNVX/8Z\n",
       "VVQsl/T3Xs9sdcQRKxSP12nixIkDtuOBBx7QiBEj9PTTT2vHjh3asWOHnn76aZ1xxhn62c9+phEj\n",
       "Rui73/2uDhw4oP/+7//W448/7srrf9e73qVIJKI//elPPdsmT56sF198UQcOHJCUmtP6qquu0rXX\n",
       "Xqu//e1vkqSXXnpJmzZtkiRddNFFWrNmjZ5++ml1dnZSIgIAAFDKjjzySD366CO65BKr8vJTdOSR\n",
       "4zRhwvlaunS+vvOdm3vud+GFF+oPf/iN9u/fq71731Zz8609JRiXXnqpLrrooxo16v2KRJZJ+q6O\n",
       "Oiqu44/foLVrfzCodjQ3N+vKK6/U1KlTNXHiRE2cOFGTJk3S5z//ed1777362c9+pjVr1mjChAm6\n",
       "7777dOGFF/Z5fHav8+HmyO49h/aoUaN0/fXX62Mf+5jGjRunxx57TGeeeaZmzpypyZMn95wcfOMb\n",
       "39B73vMeffSjH9WYMWN09tlna+fOnZKk2bNn69prr9WZZ56p6dOnq66uLu85uv3MBGkuQ2OMDVJ7\n",
       "AQBAcBlj+p3zuaurS7t379bYsWMViQy9v/Lxxx/XHXfcq927O3XOObX653/+Z5WVlQ23ychDf59z\n",
       "ente6Z+ADQAAkMPhAjbCoxABmxIRAAAAwEUEbAAAAMBFBGwAAADARQRsAAAAwEUEbAAAAMBFBGwA\n",
       "AADARQRsAAAAwEUEbAAAAAzbiSeeqJaWlrwee+655+qOO+5wuUXvcBxHxx9/fMGePxsBGwAAwA0H\n",
       "DkiXXSa9+OKh+5qapAcecO1QJ554oiZNmqTOzs6ebbfffrvi8figHh+LxfSjH/1owPvt3r1bRx11\n",
       "lM4999wB79t7WfWh2rBhgy677DJJ0po1a1RdXd1n/xVXXKGGhoa8ntsLBGwAAIDB+M1vpBUrpOxV\n",
       "/3bvlq6+WurslD7wASkW6xuyb7lF+uEPpQ9/WOrokPbsyf38r7wypOZ0d3frO9/5ztBeQ9pgg/D9\n",
       "99+vE044QY7jaNeuXXkdqxQRsAEAAAZj2jRp3TqpoeGdkL17t3Tuuane69Gjpfp6adGid0L2LbdI\n",
       "q1ZJjiNNnSr94AfS3LmHhuwbbpAuuODQ8N4PY4zq6+t1yy23qKOjI+d9fvWrX+kjH/mIxo4dq9NP\n",
       "P12PPvqoJOn666/Xtm3b9PnPf16jR4/WF7/4xX6Ps3btWi1cuFAf+9jH9JOf/KTPvjvuuEPTpk3T\n",
       "McccoxtuuKHPvsbGRs2fP1+XXXaZjj76aJ122mn64x//qBtvvFGTJk3StGnTtHnz5p77Z3rUn3nm\n",
       "GS1atEiPPvqoRo8erXHjxmn16tW66667dPPNN2v06NH6+Mc/Lkl6+eWXdeGFF2rixIk6+eST9b3v\n",
       "fa/n+fbu3asrrrhC48eP18yZM/X4448P6n11jbU2MJdUcwEAAAovZ+549VVr/+EfrL3+emv//ndr\n",
       "q6ut/fSnrU0m+97vm9+0VrK2vNzaF154Z3syae2nPmVtba21u3entn3969aecoq1L7886LadeOKJ\n",
       "9pFHHrGf+MQn7Fe+8hVrrbWrV6+2sVjMWmvt66+/bseOHWt/8pOf2GQyae+++247btw4+8Ybb1hr\n",
       "rY3FYvZHP/rRYY/x3HPP2Wg0al944QV722232dNOO61n35NPPmmPOuoou23bNrt//377pS99yY4Y\n",
       "McK2tLRYa61dunSpLS8vt5s2bbIHDx60CxYssNOmTbM33HCDPXjwoF29erU96aSTep6vd3vWrFlj\n",
       "zzjjjD5tueKKK2xDQ0OvtzFpq6qq7IoVK+yBAwfsn//8Z3vyySfbjRs3Wmut/fKXv2xramrsm2++\n",
       "aV944QU7c+ZMe/zxx+d8nf3ly/T2vDIrPdgAAACD9a53SS0t0l13pXqsp0+XbrtNigwyUkUi0u23\n",
       "SyefnOrJ/spXpOZmqbVVmjJlSE0xxmj58uX63ve+p9dee63Pvl/84hc65ZRT9K//+q+KRCK65JJL\n",
       "dOqpp2r9+vU997ED9JbfcccdOv300zV16lR94hOf0FNPPaUdO3ZIkn76059q3rx5OuOMM1RWVqYV\n",
       "K1YokvUe1NTU6Oyzz1Y0GtUnP/lJvf7661qyZImi0aguvvhiPffcc3r77bcPOW5/7eq9/fHHH9dr\n",
       "r72mr3zlKxoxYoROOukkLVy4UPfcc48kad26dbr++us1duxYTZ06Vddcc82Ar9dNI4p2JAAAgDAY\n",
       "OVIqL09dnzhRyq5nzpSFvPCCdM89qXKRTImI9E7IjkalrVulP/5xyOE6Y+bMmTrvvPN000036X3v\n",
       "e1/P9pdfflknnHBCn/tOmzZNL7/8cs/tgeqwm5ubdfXVV0uSJkyYoFgspjVr1uhb3/qWXn75ZU3N\n",
       "vB5Jo0aN0oQJE/o8fuLEiT3XR44cqWOOOabnmCNHjpSUGkR59NFHD+UlS5Kef/55vfzyyxo3blzP\n",
       "tmQyqZqaGkmp19971pDs96LQ6MEGAAAYrEzN9T/9k7Rrl/Tgg31rsrNrrrNrsjNuukl673ul2bOl\n",
       "hQv7H/g4CMuWLdPq1av10ksv9Ww77rjj9Pzzz/e53/PPP6/jjjtO0sDh+le/+pWeffZZfe1rX9OU\n",
       "KVM0ZcoUPfroo7rrrruUTCZ17LHH6oUXXui5f2dnp15//fW8X0NvudqWve2EE07QSSedpDfffLPn\n",
       "8vbbb+vBBx+UJE2ZMkV/+ctfeu7f+3oxELABAAAGIxOuM2UhEyemykXWr0+F7K4u6Xe/69tbLaVC\n",
       "9tVXp2YhkVIDGpubU73Xv/jFO+UieYbsd7/73br44ov7zCgyZ84c7dy5U3fffbcOHjyoe++9V888\n",
       "84zOO+88SdKkSZP0pz/9qd/nXLt2rWbNmqWnn35aO3bs0I4dO/TEE09o7969euihh/TJT35SDz74\n",
       "oP7nf/5HXV1d+upXv6ru7u682p9t0qRJevHFF3XgwIE+2/785z/33D799NM1evRo3Xzzzdq7d6+S\n",
       "yaSeeOIJ/Sb9Hl900UW68cYb9dZbb+nFF1/sMwCyGAjYAAAAg9HeLr3//X1rrjM12U88kZqmr7m5\n",
       "b7jOWLw4NUvI97/ft+a6d032EGYRyfbVr35VnZ2dPT29EyZM0IMPPqimpiYdc8wxuuWWW/Tggw9q\n",
       "/PjxkqRrrrlGP/3pTzV+/Hhde+21fZ5r3759Wrdunb7whS9o4sSJPZcTTzxRl112mZqbmzVjxgx9\n",
       "//vf16WXXqpjjz1W48eP71OSkWtO7IFuZ9TV1WnmzJmaPHlyT5nJpz/9aT311FMaN26cPvGJTygS\n",
       "iejBBx/Ub3/7W5188sl617vepc985jM9Nd1Lly7VtGnTdNJJJ2n27NlasGBB3nN058MUs+B7uIwx\n",
       "NkjtBQAAwWWMcX9g3PPPS2Vlh9Zcd3enerhPP93d42FA/X3O6e15pXICNgAAQA4FCdjwnUIEbEpE\n",
       "AAAAABcRsAEAAAAXEbABAAAAFxUsYBtjfmyM2WWM+X2vbeONMZuNMTuNMZuMMWN77bvOGPNHY8wz\n",
       "xphZhWoXAAAAUEiF7MH+L0mzs7YtkbTZWjtdUkv6towxMyRdLGlG+jE/MMbQuw4AAIDAKdhS6dba\n",
       "bcaYE7M2ny+pNn19rSRHqZD9cUl3W2sPSHrOGPOspNMl/bpQ7QMAABhIMedORngULGD3Y5K1dlf6\n",
       "+i5Jk9LXj1XfMP2ipOOK2TAAAIDemKIP+Sp2wO5hrbXGmMN9c3Pua2xs7Lkei8UUi8XcbRgAAABK\n",
       "juM4chzHlecq6EIz6RKRn1tr/yF9+xlJMWvtX40xUyS1WmtPNcYskSRr7U3p+z0saam19n+zno+F\n",
       "ZgAAAFBwQVpoZr2ky9PXL5f0QK/tlxhjyowxJ0l6r6THitw2DFVDg5TrTM9xUvsAAABKUCGn6btb\n",
       "0q8knWKMecEY8ylJN0k62xizU9KZ6duy1j4l6T5JT0l6SNLn6KoOgLo6af78viHbcVLb6uq8ahUA\n",
       "AICnCloi4jZKRHwoE6jXrUvdzlynNh4AAATYcEpECNgYPseR4vHU9dZWwjUAAAi8INVgAwBKTDKZ\n",
       "VDKZ9LoZAFA0BGwMT6ZEpLU1dcmuyQ4QQgDgrkQioXh8nsrKylVWVq54fJ7a29u9bhYAFBwBG/nr\n",
       "XX8di6Uu69YFLmQTAgD3JRIJ1dScI8eZq+7uDnV3d8hx5qq6epYSiYTXzQOAgiJgI38tLYcOaMyE\n",
       "7JYWr1o1JIQAoDAWL16qPXtWSFokaVT6skh79qxQfX2jp20DgEJjkCNKWjw+T44zV6kQ0NsqxeMb\n",
       "tGXLei+aBQRaMplUWVm5urs7lArWvXUqEhmjrq59ikajXjQPAAaFWUSAPBAC4CeZ+v8wfN/42wIQ\n",
       "BswiAgABkGsgbRjHAESjUdXUzJbUnGNvs2pr55R8uGZQNRBuBGyULEIAiqW/EB3mMQArVy5XRUWD\n",
       "pFWSOtOXVaqoaFBT0zJvG+ehMJ5QATgUJSIoae3t7aqunpUejLUgvbVZFRUN2rZtkyorK71sHkIg\n",
       "E6JzfcdOPXWmtm+/RGEdA5BIJFRf36itWx+SJNXWzlFT0zLX/q6CVlZzuO9CW9tGVVVVedk8AFmo\n",
       "wQaGodAhAKWt/4G0P5D0RUlvK+x1ym4H4UQiocWLl6qt7WFJUk3NbK1cudz3f7MMqgaChYANuCBo\n",
       "vWHwv8MP9vu7pLHp/4Y7YLspqL3ADPwEgodBjoALotEo/7ihiKKSKsQYgKFhfm0AQUDAxqEaGnKv\n",
       "xOg4qX0ABmWggbQf/nAlAwGHIJlMpstCFuTYu0Bbtz7k25k5GFQNlBYCNg5VV3focueZZdHr6rxq\n",
       "FRBIh5tN47bbvq22to2KxzcoEhmjSGSM4vENDLANKWZWAUoHARuHyix3ngnZmXCdvSw6gAFVVlYe\n",
       "NkRXVVVpy5b16urap66ufdqyZT3huh9B7wUe6LsAIDwY5Ij+OY4Uj6eut7YSroFhYiDt8IVlak2+\n",
       "C4D/McgRAAKAgbTDF5ZeYL4LQLjRg43cepeFSJSIAPAdeoEBFBLzYMNduWquqcMGAAAlhBIRuKul\n",
       "5dAgnRn42NLiVasAAAACgR5sAAAAIAs92AAAAIBPELABAAAAFxGwAQAAABcRsAEAnksmkz3T7gFA\n",
       "0BGwAQCeSSQSisfnqaysXGVl5YrH56m9vd3rZgHAsBCwAQCeSCQSqqk5R44zV93dHeru7pDjzFV1\n",
       "9SwlEgmvmwcAeWOaPgCAJ+LxeXKcuZIWZe1ZpXh8g7ZsWe9FswBAEis5AgACJplMqqysXN3dHZJG\n",
       "Ze3tVCQyRl1d+1gGPY1l4YHiYx5sAABCiBp1IJgI2ACAootGo6qpmS2pOcfeZtXWzin53lpq1IHg\n",
       "okQEAOCJ9vZ2VVfP0p49KyQtSG9tVkVFg7Zt26TKykovm+c5atQBb1GDDQAIpEQiofr6Rm3d+pAk\n",
       "qbZ2jpqalpV8uKZGHfAeARsAEGgM4uuLgA14j0GOAIBAi0ajhMVecteoJ9MXatQBvxvhdQMAAMCh\n",
       "Vq5cnq5R/4uk30raLMkqEjlaV131fY9bB+Bw6MEGACBLMpnsKVvxSmVlpW699duKRP5T0vmSOiS9\n",
       "re7uG3TVVV9kJhHAxwjYAACk+W3e6dtvv0fd3TcrNZPIqPRlkfbsWaH6+kbP2gXg8BjkCACA3pl3\n",
       "Ote0gW1tG1VVVVXU9jDQEfBW4AY5GmOuMcb83hjzhDHmmvS28caYzcaYncaYTcaYsV60DQBQmhYv\n",
       "XpoO1/QWAxieogdsY8z7JS2U9BFJH5B0njHm3ZKWSNpsrZ0uqSV9GwCAgksmk2pre1jv9Fz3tkBb\n",
       "tz5U9JpsVrsEgsuLHuxTJf2vtXaftTYpaaukC5UawbE2fZ+1ki7woG0AgBLjhwGN/Vm5crkqKhok\n",
       "rZLUmb6sUkVFg5qalnnbOAD98iJgPyGpOl0SMkrSuZKmSppkrd2Vvs8uSZM8aBsAoET0HtA4cmSF\n",
       "Ro8+TtLXc9zTu97iyspKtbVtVDy+QZHIGEUiYxSPb2ApecDnPBnkaIy5UtLnJO2R9KSk/ZKusNaO\n",
       "63WfN6y147MexyBHoESwsh8Kqb8BjdJiSddI+krPtoqKBl8EWv4mgOIaziBHTxaasdb+WNKPJckY\n",
       "83VJL0raZYyZbK39qzFmiqRXcz22sbGx53osFlMsFit4ewEUTyKR0OLFS9P1sFJNzWytXLnc83CD\n",
       "cOk7oDEjdX3s2Jv09tvflCTV1s5RU5P34VoiWAOF5jiOHMdx5bm86sGeaK191RhzgqSNkj4q6XpJ\n",
       "r1trv2GMWSJprLV2Sdbj6MEGQsxv06QhnAYz/d3evXtYvj3E+DUAgxG4afok/dQY86Sk9ZI+Z63t\n",
       "kHSTpLONMTslnZm+DWCI/DxgayBMkwa/IFyHk98WEkJ4eRKwrbU11tqZ1toPWmtb09vesNaeZa2d\n",
       "bq2dZa19y4u2AUEV9H84/DhNGsKJ6e9KU+YXMseZq+7uDnV3d8hx5qq6ehbLzsN1rOQIhEAYSitY\n",
       "tQ7F1N7erurqWTn/ZvwwoBHui8fnyXHmqm/dvSStUjy+QVu2rPeiWfCxIJaIAHBRGEor6FVEMTH9\n",
       "XWnhFzIUGz3YQMCFqeeXXkV4gQFv4Rem/0+ieOjBBhAK9CrCC4Md0BjkAcSlJvuz4hcyFBsBGwi4\n",
       "sP3DUVVVpS1b1qura5+6uvZpy5b1hGt4KugDiEvJ4T4rlp1HMRGwgWHwS49WGP/hYJo0+AEzTwTH\n",
       "QJ8Vv5ChmKjBBvLgx9UGE4mE6usbtXXrQ5IyK9At4x8OYBiYeSI4hvJZUXePwRhODTYBGxgiv0+J\n",
       "xz8cgDsYGBccpfJZ8f/34mKQI1BEfp8Sj9IKAAgXxgEEDwEbGALmUsVA/FKXj+EL2wDiMAvzZ8U4\n",
       "gGAiYAOAC+hhCqcwDiAOq7B+Vn7/1RS5EbCBIQhzLwnyRw9TeDHzRHCE8bPiV9PgYpAjMESsNohs\n",
       "zDQRvsFXuV5P2F5jmIXlsyqVwZt+xSBHoIjC2EuC/JV6D1PYSmMO93oYQBwcYfms+NU0uOjBBoYh\n",
       "LL0kyF8p9zD5fcrKoQrb60E48Kupd+jBBjwSll4S5K+Ue5jCNvgqbK8H4cCvpsFEDzYADFMp9jCF\n",
       "oee+9y9QYXg9CD9+NS0uerABwEP0MAVL2OrGUTr41TQ46MEGABeVUg9TEGdPOVyd9amnvk/bt1+q\n",
       "IL0eAIUznB7sEW43BgBKWSkE64yVK5enS2Ok7LDa1LTJw5b1r2+ddcYi7dkjGXOvKioaAvV6APgT\n",
       "JSIAgLwErTRmoCkVE4lfqrV1Q2BeDwD/okQEADBsQSiNGcpAxiC8HgCFxSBHAICngjD4aihTKgbh\n",
       "9QDwL3qwAQAloxSnVASQH3qwAQAYhKDVjQMIJnqwAQAliTprAIfDNH0AAAwRwRpAoVAiAgAAALiI\n",
       "gA0AAAC4iIANAAAAuIiADQAAALiIgA0AAAC4iIANAAAAuIiADQAAALiIgA0AAAC4iIANAAAAuIiA\n",
       "DQAhkEwme5b+BgB4i4ANAAGWSCQUj89TWVm5ysrKFY/PU3t7u9fNAoCSRsAGgIBKJBKqqTlHjjNX\n",
       "3d0d6u7ukOPMVXX1LCUSCa+bBwAly1hrvW7DoBljbJDaCwCFFI/Pk+PMlbQoa88qxeMbtGXLei+a\n",
       "BQChYIyRtdbk9dggBVYCNgCkJJNJlZWVq7u7Q9KorL2dikTGqKtrn6LRqBfNA4DAG07A9qRExBhz\n",
       "nTHmSWPM740xdxljjjTGjDfGbDbG7DTGbDLGjPWibQAAAMBwFD1gG2NOlHSVpCpr7T9Iikq6RNIS\n",
       "SZuttdMltaRvAwByiEajqqmZLak5x95m1dbOofcaADziRQ/225IOSBpljBmh1G+bL0s6X9La9H3W\n",
       "SrrAg7YBQGCsXLlcFRUNklZJ6kxfVqmiokFNTcu8bRwAlLCiB2xr7RuSmiT9Ralg/Za1drOkSdba\n",
       "Xem77ZI0qdhtA4AgqaysVFvbRsXjGxSJjFEkMkbx+AZt27ZJlZWVXjcPAErWiGIf0BjzbknXSjpR\n",
       "UoekdcaY/7v3fay11hiTczRjY2Njz/VYLKZYLFaopgKA71VVVWnLlvU9i8xQFgIA+XEcR47juPJc\n",
       "RZ9FxBhzsaSzrbUL07cvk/RRSWdKiltr/2qMmSKp1Vp7atZjmUUEAAAABRe0WUSekfRRY8xIY4yR\n",
       "dJakpyT9XNLl6ftcLukBD9oGHF5Dg5Tr7NZxUvsAAEDJ86IGe4dSw95/I+l36c23SbpJ0tnGmJ1K\n",
       "9WbfVOy2AQOqq5Pmz+8bsh0nta2uzqtWASUhmUz2lMIAgJ+x0AwwVJlAvW5d6nbmOuMBgIJIJBJa\n",
       "vHip2toy5B0iAAAgAElEQVQeliTV1MzWypXLGcgJoKBYyREoNseR4vHU9dZWwjU8FeYBjolEQjU1\n",
       "52jPnhWSFqS3NquiokFtbRtVVVXlZfMAhFjQarARZtQoA0WTSCQUj89TWVm5ysrKFY/PU3t7u9fN\n",
       "ctXixUvT4XqRUssmjJK0SHv2rFB9faOnbQOA/tCDDXf1Lp/I9Orm2hZklIjAB0qhZzeZTKqsrFzd\n",
       "3R1KBeveOhWJjFFX175Q9twD8B4lIvCXMAfQUjiBQCDE4/PkOHOV6tntbZXi8Q3asmW9F81yFQEb\n",
       "gJcI2PCfsNYoNzSkZgvJfj2OI7W0SCtWeNEqlJhSCp6lcCKBYAvzGIhSRw02UCwrVuQ+WYjFCNdA\n",
       "AaxcuVwVFQ2SVknqTF9WqaKiQU1Ny7xtHEpaKYyBQP4I2HBfpmSitTV1yZ43GsCwRKNR1dTMVmpJ\n",
       "gWzNqq2dE5retMrKSrW1bVQ8vkGRyBhFImMUj2/Qtm2bmKYPnsmMgXCcueru7lB3d4ccZ66qq2cp\n",
       "kUh43Tz4ACUicBc1ykBRtLe3q7p6Vs5BjmENn/wUD7+gdKk0UIMN/6BGGSiaRCKh+vpGbd36kCSp\n",
       "tnaOmpqWhTJcI3+cmLirlMZAlDoCNgCUMAIUcmEFzMIgYJcOBjkCQAmLRqP8Y44+qBEunFIaA4H8\n",
       "EbARfqwu6apkMtnTY1pKSvV1I5hYAbOwmN0GAyFgI/zq6g6dySQz8LKuzqtWBU6pTklVqq8bwZVM\n",
       "JtNlIQty7F2grVsf4mRxmJjdBgOhBhulIcyrSxZBKSzLnUupvm4Ul9s19NQIFxdjIMKLGmxgILFY\n",
       "KlDH46kL4XpISvXn5lJ93SiOQv06Qo1wcTEGArnQg43hC8rUfGFdvr3ASrU3rFRfN4qj0L+OlOI8\n",
       "6YDb6MGGt4JQ48zqkgB8pNC/jlAjDHiLHmy4w881zqwuOWylumpZqb5uFFaxfx2hRhjIDwvNwB/8\n",
       "WoIRlBIWHyvVn5tL9XWjsCg/AoKBEhHgcFasyB32YzHC9SCV6s/Npfq6UViDHYTI3OtAcNGDDXf4\n",
       "uUQErirVn5tL9XWjMA7368htt31Hq1ffzRLngMfowYa3suuZM1PiMZAwlEp1SqpSfd0ojP5+Hbnt\n",
       "tu/oM5+5hiXOgYCjBxvDR40zAOSt968jDKwF/INBjgAABByDHwF/oUQEAHyGAWoAULoI2ADgokIt\n",
       "f43wY4lzIDwI2Ci+hobcgx8dJ7UPCKjM8tcDDVCjdxv9WblyuSoqGiStktSZvqxSRUWDmpqWeds4\n",
       "AINGwEbxBWFpdSAPAy1/Te82BsLc60A4MMgR3mDebITMYAaolZePUWfn15Q973Fb20ZVVVUVt8Hw\n",
       "PeZeh5v4Pg0dgxwRPJm5suPx1IVwjZDr7rbq7Fyu/nq3gWzMvQ438MuZNwjYAOCCgQaopQL1FTn2\n",
       "LdDWrQ9Rkw3AdYMdFwL3USICb1AighDqb/nrUaMatHfvm7L2bTG/MYBiYeGi4aFEBMHC0uoIqf4G\n",
       "qP3yl5tUWztHTL8GoFiSyaTa2h7WOyf7vfHLWaHRg43iY2l1lIDsAUX99W5XVDQwQwQA17Ey6PCx\n",
       "VDoABEAikVB9faO2bn1IklRbO0dNTcsI1wAKghKR4SFgA0NFLzo8xHRZAIqBX86GhxpsYKhY7AYe\n",
       "Yvo1AMXAwkXeGbAH2xhTIelLkk6w1l5ljHmvpFOstQ8Wo4FZbaEHG+5hJhMAQIngl7OhK2iJiDHm\n",
       "PknbJS2w1s5MB+5fWWs/kM8Bh4OADdc5TmqhG0lqbSVcA8gbAQYIl0KXiLzbWvsNSV2SZK3dk8+B\n",
       "AAAII1bKA5BtMAF7vzFmZOaGMebdkvYXrklAkWRKRFpbUxfm4YYPJJNJ5qYNEFbKA5DLYAJ2o6SH\n",
       "JU01xtwlaYukL+d7QGPMKcaY9l6XDmPMF40x440xm40xO40xm4wxY/M9BjAgFruBz9ALWjxunsQs\n",
       "Xrw0PUPDIqXmGh4laZH27Fmh+vpGV44BTjwRPIcN2MaYiKRxki6U9ClJd0n6sLW2Nd8DWmv/YK2t\n",
       "tNZWSvqQpE5JP5O0RNJma+10SS3p20BhtLQcOqAxE7JbWrxqFUoUvaDF4fZJDCvlFR4nngiqwQxy\n",
       "3G6t/VBBDm7MLEkN1tpqY8wzkmqttbuMMZMlOdbaU7PuzyBHIGAY+DUwFoMovMxJTK75gNvaNqqq\n",
       "qmrIz8lKeYVViM8MGIpCD3LcbIypN8Ycny7jGG+MGZ/PwXK4RNLd6euTrLW70td3SZrk0jEAeICe\n",
       "p8GhF7Q4ClHKEY1GVVMzW1Jzjr3Nqq2dQ7geBspvEGSD6cF+TlL2nay19uRhHdiYMkkvSZphrf2b\n",
       "MeZNa+24XvvfsNaOz3qMXbp0ac/tWCymGNOqAb5Dz9Pg0QtaeIV8jwdaKe+0006TxC84Q8XfxaH4\n",
       "NbDwHMeR02sc1rJly/LuwZa11pOLpI9LerjX7WckTU5fnyLpmRyPsQD8LxY7z0o/tJLNuvzQxuPz\n",
       "vG6e7/B+FdbBgwdtJDLCSntyvMd7bCQywh48eDDv59++fbuNx+fZSGSEjURG2Hh8nr3zzjttLHZe\n",
       "z7ZY7DybSCRcfFXhVujPLEi2b9+e93fp4MGDJfM+FUI6d+aXcwe8g1Qm6RpJ90v6qaQvSDoi3wP2\n",
       "et57JF3e6/bNkr6cvr5E0k05HlOAtw+Am/iHcegSiYStqDgmHbL3pC8/tBUVxxDKXFKMk5hMmNm+\n",
       "fXu/n+f27dtdOVYp4MTT5v1dGk4oxzsKHbB/JGmtpDMl1UlaI+n2fA+Yfs4KSa9JGt1r23hJj0ja\n",
       "KWmTpLE5HleYdxCAawjY+cnVC8o/iO4p5kkMwdAdnHjm913iBM89wwnYg6nB/p219rSBthUDs4gA\n",
       "wcCsGPmjzrJwEomE6usbtXXrQ5Kk2to5ampapsrKSteOQe2wu4rxmflVvt8l/v/rnuHMIjKYgJ2Q\n",
       "dJG19tn07XdLWmetLfooJQI2EAwDDfwqhX8c4V+FPIkhYBdGKZ545vNd4vvnrkJP0/fvkrYYY7Ya\n",
       "Y7YqtZJjfT4HA1AaKisr1da2UfH4BkUiYxSJjFE8voFwDV+IRqMFCxhM3VcYhfzM/IrvUrAN2IMt\n",
       "ScaYckmnKDVd305r7b5CN6yfdtCDDQRMKfY8obTxCw7cks93iRIR9xS0B9sY83lJI621O6y1v5M0\n",
       "0hjzuXwOBqD0lGLPE0obv+DALfl8l1auXK6KigZJqyR1pi+rVFHRoKamZUVsfWkbTA32DmvtB7K2\n",
       "/dZa+8GCtix3W+jBBgAEBr/gwC1D+S6V8uBQNxV6kOPvJX3AWtudvh2V9Dtr7cx8DjgcBGwAAIDB\n",
       "4QRveIYTsEcM4j4bJd1jjLlVkpH0WUkP53MwAAAAFAfB2juD6cGOSvqMUovMSNJmpRaaSRa4bbna\n",
       "Qg82AHiAnjAApaaggxyttUlr7Q8lXSrpBkk/8yJcA6HT0CA5zqHbHSe1D/CBRCKheHyeysrKVVZW\n",
       "rnh8ntrb271uFgD4Wr8B2xhzqzHm/enrYyTtUGrJ9N8aYy4tUvuA8Kqrk+bP7xuyHSe1ra6uv0cB\n",
       "RZNIJFRTc44cZ666uzvU3d0hx5mr6upZSiQSXjcPAHyr3xIRY8xT1toZ6evXSopZay8wxkyW9DCz\n",
       "iAAuyATqdetStzPXYzEvWwVIYj5dAKWtILOIGGParbWV6esblFoe/b/St5mmD3CL40jxeOp6ayvh\n",
       "Gr7AkssIAsYGoJAKVYPdYYyZZ4ypkvRPSs8cYow5QlJ5PgcDAAAYLj+ODUgmkz2BHzhcwP6spM9L\n",
       "+i9J11prX0lvP1PSLwrdMKAkZEpEWltTl+yabMAj0WhUNTWzJTXn2Nus2to59BrCE34bG+DHsA/v\n",
       "DThNn59QIoJQ6V1/nSkLybUN8Eh7e7uqq2dpz54VkhaktzaroqKBZb/hGT+NDciE/Vx/I21tG1VV\n",
       "VVW0tsB9BV3J0U8I2AiVhobUbCHZQdpxpJYWacUKL1oVONRgFhZLLsNP/DY2wE9hH+4jYAMSgbXE\n",
       "JBIJLV68VG1tqYVla2pma+XK5QS/AuFEBn7gp4Dtp7agMAq60AwQGMwrXTL8VoNZCqLRKEEBnmNs\n",
       "AILisD3Yxpj3STpW0v9aa3f32j7bWvtwEdqX3R56sHF4zCtdEvhZFihdfhobwP+Lwq1Q82B/UdL/\n",
       "I+lpSZWSrrHWPpDe1zNHdjERsDEozCsdavwsC8AvYwP8FPbhvuEE7BGH2fcZSR+y1u42xpwo6afG\n",
       "mBOttd/O50AAAABuqKqq0pYt6z0fG1BZWam2to3psP8FSZmwT7gudYfrwX7SWjuz1+2jJN0v6SlJ\n",
       "cVZyhC9RIlIS+FkWgN94HfbhvkINcnzVGNMTotM12OdJmiDptHwOBhRU9hzSsVjqOou3hM7KlctV\n",
       "UdEgaZWkzvRllSoqGtTUtMzbxgEoSQwERm+HC9gLJP219wZr7QFJl0uqKWSjgLy0tBzaW50J2S0t\n",
       "XrUKBZD5WTYe36BIZIwikTGKxzdQ8wgA8AXmwQYQaPwsCwAoBObBBuAvDQ25y3IcJ7XPRfwsCwDw\n",
       "GwI2APex6A8AoIQNukTEGHO0ek3rZ619o1CNOkwbKBEBgoIZXQAAAVaQhWZ6PflnJS2TtF9Sd3qz\n",
       "tdaenM8Bh4OADQQMi/4AAAKqUAvNZPy7pPdba1/L5wAAAABAKRlMDfafJe0tdEMAhEymRKS1NXVh\n",
       "PnIAQIkYTIlIlaQ1kh6V1JXebK21Xyxs03K2hRIRIAiyF/3pbxsAAD5V6BKR2yQ9Iun3StVgG0mk\n",
       "XAD9G2jRHwI2ACDEBtOD3W6t9cXSaPRgAwAAoBgKvdDMQ8aYzxpjphhjxmcu+RwMAOB/yWSyZ4VM\n",
       "AMHE37G3BhOwL5W0RNKvJG3vdQEAhEgikVA8Pk9lZeUqKytXPD5P7e3tXjcLwBDwd+wPg15oxg8o\n",
       "EQGAwkgkEqqpOUd79qyQtCC9tVkVFQ1qa9uoqqoqL5sHYBD4O3ZXQReaSR/g/ZJmSCrPbLPWNudz\n",
       "wOEgYAMB09CQWho9e1Cj46QGO65Y4UWrkEM8Pk+OM1fSoqw9qxSPb9CWLeu9aBaAIeDv2F2FXsmx\n",
       "UVKtpJmSfiFpjqRfWms/mc8Bh4OAHQAEKvTGdH2BkEwmVVZWru7uDkmjsvZ2KhIZo66ufYpGo140\n",
       "D8Ag8HfsvkIPcvykpLMkvWKt/ZSkD0gam8/BUALq6g5dUCQTqOrqvGoVvJKZmi/znSBcAwBKwGAC\n",
       "9l5rbVLSQWPMGEmvSjp+OAc1xow1xvzUGPO0MeYpY8z/lZ6dZLMxZqcxZpMxhhAfRAQqZMt8J+Lx\n",
       "1IXvgu9Eo1HV1MyWlKvyr1m1tXPo9QJ8jr9jfxlMwP6NMWacpNWSfiOpXakZRYbjO5I2WGvfJ+k0\n",
       "Sc8oNVPJZmvtdEkt6dsIIgJVcDU05F7O3HFS+xBaK1cuV0VFg6RVkjrTl1WqqGhQU9MybxsHYFD4\n",
       "O/aPAQO2tfZqa+2b1tpVkmZJujxdKpKXdC94tbX2x+nnP2it7ZB0vqS16butlXRBvscAkKdClPhk\n",
       "Ht/amrpkP7+HmCf2HZWVlWpr26h4fIMikTGKRMYoHt+gbds2qbLSF2uNARgAf8f+MZhBjp+21v6o\n",
       "1+0Rkq631uZ1KmSM+aCkWyU9pVQ993ZJ10p60Vo7Ln0fI+mNzO1ej2WQYxD0LguRKBEJGjc/P58O\n",
       "ckwkElq8eKna2h6WJNXUzNbKlcv5Bygtc9LBz8lAcPF3PHzDGeQ4YhD3OcsYc6GkhZLGS/ovSW35\n",
       "HKzXMaskfd5a+7gx5tvKKgex1lpjTM4k3djY2HM9FospRmjzl1zhKVOTTcgOht4lPlKq1znfz62l\n",
       "5dDPPfP8LS2efB/6zhN7ryTJcZpVXT2LeWLT+AcZCD7+jofOcRw5Lv3COth5sC+R9J+S9kj6V2vt\n",
       "L/M+oDGTJT1qrT0pffsMSddJOllS3Fr7V2PMFEmt1tpTsx5LD7bfMU1fODiOOwHbh5gnFgAwGIWe\n",
       "B3u6pDWSnpD0PklPSlpsrd2TzwHTz9kmaaG1dmd6nu3MhI2vW2u/YYxZImmstXZJ1uMI2EChhbjE\n",
       "h3liAQCDVeiA/YxS5RyPGGMikv5N0qettTPyOWD6OT8g6XZJZZL+JOlTkqKS7pN0gqTnJF1krX0r\n",
       "63EEbKCQfFoz7RYCNgBgsAodsMekZ/novW26tXZnPgccDgI2UGAlUOJDiQgAYDAKErCNMf+vtfbm\n",
       "9PX51tp1vfbdYK39j7xaOwwEbADD1d7erurqWelBjgvSW5tVUdHAVFYAgB6FWir9X3pdzw7Tc/I5\n",
       "GAB4jXliAQCFdrge7HZrbWX29Vy3i4UebABuYp5YAEB/Cj0PNgCEEsEaAFAIh+vBTiq1iL0kjZS0\n",
       "t9fukdbaoodzerABAABQDAXpwbbW0rUDAAD6RZkVkNvhBjkCAAAcIpFIKB6fp7KycpWVlSsen6f2\n",
       "9navmwX4BgEbAAAMWiKRUE3NOXKcueru7lB3d4ccZ66qq2cpkUh43TzAFwZcaMZPqMEGAMBbLNaE\n",
       "UlHQlRz9hIANAEBhHa6uOplMqqysXN3dHZJGZe3tVCQyRl1d+6jJRigUaqEZAMXS0JBajjyb46T2\n",
       "AUCBUVcNuIeADfhBXZ00f37fkO04qW11dV61CkCJGGxddTQaVU3NbEnNOZ6lWbW1c+i9BkSJCOAf\n",
       "mUC9bl3qduZ6LOZlqwCUgKHUVbe3t6u6epb27FkhaUF6a7MqKhq0bdsmVVYWfaFnoCCowQbCwnGk\n",
       "eDx1vbWVcA2g4PKpq04kEqqvb9TWrQ9Jkmpr56ipaRnhGqHCUukAAKBoqqqqtGXLehaaAfpBDTbg\n",
       "F5kSkdbW1CW7JhsACmA4ddXRaJRwDeRAiQjgB73rrzNlIbm2AUABUFcNHIpp+oCga2k5NEjHYqlt\n",
       "LS1etQpAiaisrFRb20bF4xsUiYxRJDJG8fgGwjWQJ3qwAQBAD+qqgRQGOQIAAFcQrIHho0QEAAAA\n",
       "cBEBGwAAAHARARsAAABwEQEbAAAAcBEBGwAAAHARARsAAABwEQEbAFAQyWSyZ05lACglBGwAgKsS\n",
       "iYTi8XkqKytXWVm54vF5am9v97pZAFA0BGwAgGsSiYRqas6R48xVd3eHurs75DhzVV09S4lEwuvm\n",
       "AUBRsFQ6AMA18fg8Oc5cSYuy9qxSPL5BW7as96JZADBkw1kqnYANwN8aGqS6OikW67vdcaSWFmnF\n",
       "Ci9ahRySyaTKysrV3d0haVTW3k5FImPU1bWPpbgBBMJwAjYlIgD8ra5Omj8/FagzHCe1ra7Oq1a5\n",
       "q6Gh7+vLcJzUPgBAoBCwgVIWhGAXi0nr1r0TsjPhet26Q3u1gyokJxHRaFQ1NbMlNefY26za2jn0\n",
       "XgMoCZSIAKUsV1j1a4B1HCkeT11vbfVX29zQ+32X/PkZDEJ7e7uqq2dpz54VkhaktzaroqJB27Zt\n",
       "UmVlpZfNA4BBo0QEQH5KoXc4KDKfRTyeugT0M6isrFRb20bF4xsUiYxRJDJG8fgGwjWAkjLC6wYA\n",
       "8FjvYCf5s3c4E/xbW1O3OQnwtaqqKm3Zsr5nkRnKQgCUGnqwAfhbdq96dq97WPQ+iWhtDcXri0aj\n",
       "hGsAJYmADZQ6vwe7lpZDe6szIbulxatWuatUTiIAoEQwyBEoZUEa5BhmzPUNAL7DQjMA8kOwAwAg\n",
       "JwI2AAQNJzcA4GuBm6bPGPOcMeZ3xph2Y8xj6W3jjTGbjTE7jTGbjDFjvWgbABRFSBaXAQAcyqtB\n",
       "jlZSzFpbaa09Pb1tiaTN1trpklrStwF4LQirPQYRc5ADQGh5OYtIdpf7+ZLWpq+vlXRBcZsTcIQg\n",
       "FAo9rYUTksVlAAB9edmD/Ygx5jfGmKvS2yZZa3elr++SNMmbpgUUIQiFQk8rAABD4tVKjh+z1r5i\n",
       "jHmXpM3GmGd677TWWmNMztGMjY2NPddjsZhi/AOf0jsErVuX2kYIgluCsNpjELFCJQD4huM4clxa\n",
       "e8DzWUSMMUsl7ZZ0lVJ12X81xkyR1GqtPTXrvswiMhDHIQShMPhuuYs5yAHA1wI1i4gxZpQxZnT6\n",
       "eoWkWZJ+L2m9pMvTd7tc0gPFbhuAfjiONHeu9K1vHbraI3X++SmFFSoBoEQVvQfbGHOSpJ+lb46Q\n",
       "dKe19kZjzHhJ90k6QdJzki6y1r6V9Vh6sA+nd++XRE8Y3JH5Xl1/vfT1r/f9fvXexvcMCKxkMilJ\n",
       "ikajHrcE8I9A9WBba/9/a+0H05f3W2tvTG9/w1p7lrV2urV2Vna4xgCyf1rOHpgG5CvT03rtte98\n",
       "p6RUuP6P/yBcAwGWSCQUj89TWVm5ysrKFY/PU3t7u9fNAgLP8xrsoaAH+zBYFQ7FQi02EAqJREI1\n",
       "Nedoz54VkhaktzaroqJBbW0bVVVV5WXzAM+xVDqA4iFgA6EQj8+T48yVtChrzyrF4xu0Zct6L5oF\n",
       "+AYBG0BxUOcPHCKI9cvJZFJlZeXq7u6QNCprb6cikTHq6toXqNcEuC1QNdgAAoo6f6AP6pcB9IeA\n",
       "DWBwmFYO6JGpX3acueru7lB3d4ccZ66qq2cpkUh43bwBRaNR1dTMltScY2+zamvn0HsNDAMlIgAA\n",
       "DFEY6pfb29tVXT0r5yDHbds2qbKy0svmAZ6jRAT+1tCQu4SABUoABFAymVRb28N6J5T2tkBbtz7U\n",
       "U5ftZ5WVlWpr26h4fIMikTGKRMYoHt9AuAZcQMBG4dXVHVqnm6nnravzqlVwEydRpSHkn3MymQxE\n",
       "MHZTVVWVtmxZr66uferq2qctW9YTrgEXELBReNmD4bIHyyH4OIkqDSH9nIc6WDGM9cvRaDRwbQb8\n",
       "jBpsFA/zJ4cbU/iVhpB9zvkutkL9MhB+zIONYCBghx+fcWkI0ec8nMGKiURC9fWN2rr1IUlSbe0c\n",
       "NTUtI1wDITGcgD3C7cYAOWV6vVpbU7cD3usFIPjeGax4b469C7R16xeUTCb7LZ3I1C8HcaEZFBbf\n",
       "CVCDjcJjgZLwyTXYzXGkj39cuuyy1IkUn2849T5Z5nOWRP0y3sHiQ8ggYKPwWKAkfLIHu2XCtTHS\n",
       "lVdyEhVWITtZDuNgxVLkl9lfgr74ENxFDTaA/PQOWz/6kfTzn0sPPND3RMpxUidRK1Z41Ei4qqEh\n",
       "dXKVXdoV4M+ZwYrBlUgktHjx0nSZj1RTM1srVy737DMLw+JD6ItBjgC8EaLBbihdDFYMnnxnfymU\n",
       "ZDKpsrJydXd3SBqVtbdTkcgYdXXt4xeRgGGQIwAAefL7YEW/tstLixcvTYfr3r3Fi7Rnj1Rf30hv\n",
       "MTxHDTaA/DDYDSHjt8GKDJjLzY9L1VPPj2wEbABDF7LBboDfMGAueFauXK6KigZJqyR1pi+rVFHR\n",
       "oKamZd42DkVHwAYwdMwMAxRU3xKIUenLIu3Zs0L19Y2ets1rfu0trqysVFvbRsXjGxSJjFEkMkbx\n",
       "+AYGy5YoBjli+EI4swAAeIUBcwPz++wv1M2Hw3AGOdKDHTa5FgCRUtsaGgpzzOw5kTPHmz8/tQ8A\n",
       "ABf5vbfYb/X8KD56sMMmuza2v22FPK7EUugAMAx+mlPZ772xfm8fgot5sNGXV2GXOZEBwBV+KIHw\n",
       "20IuQ0HohhsoEUFfmcFm8Xjqkk+49qLUBAAgyfsSiKDOYsLUhvALAjZyG2pdNXMiA4CrMgvgdHXt\n",
       "U1fXPm3Zsr5ovcdBnMUkqCcFCCdKRMLIrRKRwT6PV3XfAADXBXUWEz/VrSMcKBHBO9xcAGSwpSbM\n",
       "iQyEE6ViwVSCn5sfV3dEaSNgh40XYXfFitzBOxZjDmyUlrAFG6bgDKZhfm5+XcgFCBJKRNA/pt4D\n",
       "hiaM5VL8fyCYhvm5+WEWk6GiRARuY5o+uC+MQQEohjAGUqbgDKZhfm6JREL19Y3auvUhSVJt7Rw1\n",
       "NS3zZbiWgnlSAH+jBhvuo64ayI8b02QCPuDlLCb58HpqQ6A3erABwG1h6vENY498KSjxz42FZuAG\n",
       "erABwC/CNCe8m7MS+U3YBqT2FubPbZCi0SjhGp4iYAOAW8IWbMJcKhbmGVLC/LkBAUGJCAC4paEh\n",
       "Fc5yLcbU0uLfaSuD2u7hKvEyCgCHxywiAID8lfKsQWGqlwfgquEE7BFuNwYAEDC9S1nozQWAYaMG\n",
       "GwBQmtMLhmlAarYwD+IEAoCADQAoPWEbkJotzIM4gQDwLGAbY6LGmHZjzM/Tt8cbYzYbY3YaYzYZ\n",
       "Y8Z61TYAKDlh7s3NJewzbWSfMJRKTT3gE54NcjTGfEnShySNttaeb4y5WdJr1tqbjTFfljTOWrsk\n",
       "6zEMcgQAt5XyIMewYxAnkLfALTRjjJkq6VxJt0vKNPx8SWvT19dKusCDpgFA6Ql7by4AFJknPdjG\n",
       "mHWSbpB0tKR6a+08Y8yb1tpx6f1G0huZ270eRw82AACDwTzfwLAEqgfbGHOepFette16p/e6j3SK\n",
       "JkkDGDpmTwDCP4gT8Dkv5sH+J0nnG2POlVQu6WhjzB2SdhljJltr/2qMmSLp1VwPbmxs7Lkei8UU\n",
       "40wcQG+Z2RP6qycGSsFAZT/82wkcwnEcOS6dgHq6kqMxplbvlIjcLOl1a+03jDFLJI1lkCOAvPDT\n",
       "OABgmAJVIpJDJjHfJOlsY8xOSWembwPA0JXioikA3EOpGYbJ04Btrd1qrT0/ff0Na+1Z1trp1tpZ\n",
       "1tq3vGwbAAAoUSzUg2HyQw82ALir1BZNAeCuQizUQ694SSFgAwgXZk8A4Aa3S83oFS8pBGwA4cKi\n",
       "KQD8iOXrS4qns4gMFbOIAACAoijUbEQsXx8YQZ9FBAAAwD8oNcMwEbABAAB6K1SpGQOwSwYlIgAA\n",
       "AIWWq+aaOmxfo0QEAADAzxiAXVLowQYAAACy0IMNAAAA+AQBGwAAAHARARsAAABwEQEbAAAAcBEB\n",
       "GwAAAHARARtAMDU05F6gwXFS+wAA8AgBG0Aw1dUdugpaZtGGujqvWgUAAPNgAwiw3qugSayIBgBw\n",
       "zY6HetkAAAzqSURBVHDmwSZgAwg2x5Hi8dT11lbCNQDAFSw0AwAAAPgEARtAcGVKRFpbU5fsmmwA\n",
       "ADxAwAYQTL3rr2Ox1GXdOkI2AMBzBGwAwdTScuiAxkzIbmnxqlUAADDIEQAAAMjGIEcAAADAJwjY\n",
       "AAAAgIsI2AAAAICLCNgAAACAiwjYAAAAgIsI2AAAAICLCNgAAACAiwjYAAAAgIsI2AAAAICLCNgA\n",
       "AACAiwjYAAAAgIsI2AAAAICLCNgAAACAiwjYAAAAgIsI2AAAAICLCNgAgNLV0CA5zqHbHSe1DwDy\n",
       "QMAGAJSuujpp/vy+IdtxUtvq6rxqFYCAM9Zar9swaMYYG6T2AgACIBOo161L3c5cj8W8bBUAjxlj\n",
       "ZK01eT222IHVGFMuaaukIyWVSfr/rLXXGWPGS7pX0jRJz0m6yFr7VtZjCdgAAPc5jhSPp663thKu\n",
       "AQwrYBe9RMRau09S3Fr7QUmnSYobY86QtETSZmvtdEkt6dsAAABAoHhSg22t7UxfLZMUlfSmpPMl\n",
       "rU1vXyvpAg+aBgAoNZkSkdbW1CW7JhsAhsiTgG2MiRhjfitpl6RWa+2TkiZZa3el77JL0iQv2gYA\n",
       "KCG9669jsdRl3TpCNoBhGeHFQa213ZI+aIwZI2mjMSaetd8aY3IWWzc2NvZcj8ViilEnBwDIV0vL\n",
       "oQMaMyG7pYVabKCEOI4jx6UTa89nETHGNEjaK2mhpJi19q/GmClK9WyfmnVfBjkCAACg4AI1yNEY\n",
       "c4wxZmz6+khJZ0tql7Re0uXpu10u6YFitw0AAAAYLi9KRKZIWmuMiSgV8O+w1rYYY9ol3WeM+bTS\n",
       "0/R50DYAAABgWDwvERkKSkQAAABQDIEqEQEAAADCjIANAAAAuIiADQAAALiIgA0AAAC4iIANAAAA\n",
       "uIiADQAAALiIgA0AAAC4iIANAAAAuIiADQAAALiIgA0AAAC4iIANAAAAuIiADQAAALiIgA0AAAC4\n",
       "iIANAAAAuIiADQAAAPyf9u491q6yTuP494FauYgoGMERDI0RBYFKBWR0sCKIYhRvQfEWr2iMRryN\n",
       "UUicMZPoTNQoGYPjBQoSbbwjRI1UBdGQgEKRFqyIEaUqBQWL1kGB/uaP9Z5hn332ht129ZzT9vtJ\n",
       "TvZe71rr7Pc8Wfuc31n7XevtkQW2JEmS1CMLbEmSJKlHFtiSJElSjyywJUmSpB5ZYEuSJEk9ssCW\n",
       "JEmSemSBLUmSJPXIAluSJEnqkQW2JEmS1CMLbEmSJKlHFtiSJElSjyywJUmSpB5ZYEuSJEk9ssCW\n",
       "JEmSemSBLUmSJPXIAluSJEnqkQW2JEmS1CMLbEmSJKlHFtiSJElSjyywJUmSpB5ZYEuSJEk9ssCW\n",
       "JEmSemSBLUmSJPXIAluSJEnqkQW2JEmS1CMLbEmSJKlHs15gJ9k/ySVJrkuyOsnbW/teSVYkuSHJ\n",
       "xUkeNtt9255ceumlc92FbYI5Tc6sJmNOkzGnyZnVZMxpcma19c3FGey7gXdW1ROBo4G3JjkIeB+w\n",
       "oqoOBL7flrWZfPNMxpwmZ1aTMafJmNPkzGoy5jQ5s9r6Zr3Arqpbquqa9vyvwM+BRwMnAee1zc4D\n",
       "XjjbfZMkSZK21JyOwU5yAHA4cAWwT1Wta6vWAfvMUbckSZKkzZaqmpsXTh4C/BD4j6q6IMkdVfXw\n",
       "gfW3V9VeQ/vMTWclSZK0w6mqbM5+C/ruyCSSPAj4GnB+VV3Qmtcl2beqbknyKODW4f0294eUJEmS\n",
       "Zstc3EUkwNnA9VX1iYFVFwKvac9fA1wwvK8kSZI03836EJEk/wJcBlwLTL34+4ErgS8DjwFuAl5a\n",
       "VX+e1c5JkiRJW2jOxmBLkiRJ26N5O5OjE9JMJskuSa5Ick2S65N8uLWb0whJdk6yMslFbdmcRkhy\n",
       "U5JrW1ZXtjazGpLkYUm+muTn7f33FHOaKcnj27E09bU+ydvNaqYk729/91Yl+WKSB5vTaElOazmt\n",
       "TnJaa9vhs0pyTpJ1SVYNtI3NpR1zv0yyJskJc9Pr2Tcmp5Pb++/eJEuGtt+knOZtgY0T0kykqu4C\n",
       "jq2qJwGHAce2YTjmNNppwPXcNzzJnEYr4BlVdXhVHdXazGqmM4FvV9VBdO+/NZjTDFX1i3YsHQ48\n",
       "Gfgb8A3Mapp269pTgSVVdSiwM3AK5jRDkkOANwJHAouB5yV5LGYFsAx4zlDbyFySHAy8DDi47XNW\n",
       "kvlcG/ZpVE6rgBfRDWX+f5uT07wN0QlpJldVf2tPF9L9Qr4Dc5ohyX7Ac4HPAVN3pDGn8Ybv2mNW\n",
       "A5LsCRxTVecAVNU9VbUec3ogxwM3VtXNmNWwO+lOLu2WZAGwG/B7zGmUJwBXVNVdVXUv3W1/X4JZ\n",
       "UVU/oqsDBo3L5QXA8qq6u6puAm4EjmIHMCqnqlpTVTeM2HyTc5q3BfYgJ6S5f0l2SnINXR6XVNV1\n",
       "mNMoHwf+Fdg40GZOoxXwvSQ/TXJqazOr6RYBtyVZluTqJJ9Nsjvm9EBOAZa352Y1oKpuBz4G/Jau\n",
       "sP5zVa3AnEZZDRzThj7sRnfyZD/MapxxufwTsHZgu7V0JzM13SbnNO8L7HQT0nwNOK2q/jK4rror\n",
       "NHf4qzSramMbIrIf8PQkxw6t3+FzSvI84NaqWsnMM7OAOQ15Wvs4/0S64VnHDK40K6CbR2AJcFZV\n",
       "LQE2MPRxtDlNl2Qh8HzgK8PrzAraEId3AAfQ/UF/SJJXDW5jTp2qWgP8F3Ax8B3gGuDeoW3MaoQJ\n",
       "cjGzydxvTvO6wM79TEjT1o+ckGZH1T6e/hbdGEdzmu6pwElJfk139uyZSc7HnEaqqj+0x9voxsoe\n",
       "hVkNWwusraqftOWv0hXct5jTWCcCV7XjCjymhh0BXF5Vf6qqe4CvA/+Mx9RIVXVOVR1RVUvpPuq/\n",
       "AY+pccbl8jtg/4Ht9mttmm6Tc5q3BXbihDSTSPKIqauBk+wKPAtYiTlNU1WnV9X+VbWI7iPqH1TV\n",
       "qzGnGZLslmSP9nx34AS6Cz/MakBV3QLcnOTA1nQ8cB1wEeY0zsu5b3gIeEwNWwMcnWTX9jfweLqL\n",
       "sj2mRkjyyPb4GODFwBfxmBpnXC4XAqckWZhkEfA4unlJNP3T7k3Oad7eBztOSDORJIfSXbCwU/s6\n",
       "v6o+kmQvzGmkJEuBd1fVSeY0U/vl8Y22uAD4QlV92KxmSrKY7qLZhcCvgNfRXWhsTkPaP2u/ARZN\n",
       "DffzmJopyXvpCqCNwNV0d8rYA3OaIcllwN7cd9exSzymIMlyYCnwCLrx1h8AvsmYXJKcDrweuIdu\n",
       "OO5356Dbs25ETv8G3A78d2tbD6ysqhPb9puU07wtsCVJkqRt0bwdIiJJkiRtiyywJUmSpB5ZYEuS\n",
       "JEk9ssCWJEmSemSBLUmSJPXIAluSJEnqkQW2JG1FSe5NsnLg672z+NrnJFmXZNX9bPP4JJe2vl2f\n",
       "5NOz1T9J2l55H2xJ2oqS/KWq9pij1z4G+Cvw+ao6dMw23wU+WVUXteVDqmr1Fr7uTlW1cUu+hyRt\n",
       "yzyDLUmzLMmeSdZMTbOeZHmSN7Tnn0rykySrk/z7wD43JflQO9P80yRLklyc5MYkbx71OlX1I+CO\n",
       "B+jOvsDvBvZZ3V5v5yQfTbIqyc+SvK21H5fk6iTXJjk7ycKB/v1nkquAk5OckOTyJFcl+XKbyVGS\n",
       "dggW2JK0de06NETk5KpaD7wNODfJKcCeVXV22/70qjoSWAwsTXJIay/gN1V1OHAZcC7wIuBo4INb\n",
       "0L+PAz9I8u0k70iyZ2t/E920yourajHwhSS7AMvoplk+DFgAvGWgf3+sqicD3wfOAI5ry1cB79qC\n",
       "PkrSNmXBXHdAkrZz/9uK4mmq6ntJXgp8EjhsYNXLkpxK9/v5UcDBwNSQjQvb4ypg96raAGxI8vck\n",
       "D62qOze1c1V1bhsm8hzgBcCbkywGjgM+NTXUo6ruaO2/rqob2+7nAW8FzmzLX2qPR7d+X54EYCFw\n",
       "+ab2TZK2VRbYkjQHkuwEHARsAPYCfp9kEfBu4IiqWp9kGbDLwG5/b48bgX8MtG9kC36fV9Uf6M5M\n",
       "L2sXRE6dNc/wpsM/xlDbhoHnK6rqFZvbJ0naljlERJLmxjuB64BX0hW2C4CH0hWpdybZBzhxzL7D\n",
       "he9mS/LsJA9qz/cF9gbWAivozmbv3NY9HLgBOCDJY9vurwZ+OOLbXgE8bWq7JLsneVxffZak+c4z\n",
       "2JK0de2aZOXA8nfoxk+/ATiyqjYkuQw4o6o+2LZdA9wM/HjM9yymnzkeeTuoJMuBpcDeSW4GPlBV\n",
       "y4Y2OwE4M8ldbfk9VXVrks8BBwLXJrkb+ExVnZXkdcBX2j8EVwL/M9yHqrotyWuB5Uke3JrPAH45\n",
       "5ueRpO2Kt+mTJEmSeuQQEUmSJKlHFtiSJElSjyywJUmSpB5ZYEuSJEk9ssCWJEmSemSBLUmSJPXI\n",
       "AluSJEnq0f8ButxExp+t9tQAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0xd120da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "positive = data[data['Admitted'].isin([1])]\n",
    "negative = data[data['Admitted'].isin([0])]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12,8))\n",
    "ax.scatter(positive['Exam 1'], positive['Exam 2'], s=50, c='b', marker='o', label='Admitted')\n",
    "ax.scatter(negative['Exam 1'], negative['Exam 2'], s=50, c='r', marker='x', label='Not Admitted')\n",
    "ax.legend()\n",
    "ax.set_xlabel('Exam 1 Score')\n",
    "ax.set_ylabel('Exam 2 Score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like there is a clear decision boundary between the two classes.  Now we need to implement logistic regression so we can train a model to predict the outcome.  The equations implemented in the following code samples are detailed in \"ex2.pdf\" in the \"exercises\" folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to create a sigmoid function.  The code for this is pretty simple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do a quick sanity check to make sure the function is working."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0xd120048>]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAsQAAAHfCAYAAABXrM5GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm0XWWZJ+DfR5gjk4IMAURkEJxpxaEovQpCABlFu3C2\n",
       "xHaVrb1Wre7SoqzVBqtbS7uquibLRQNiV7WlCDdAGIRCIKCFMgiIQIgERQmDIggoypDcr//YCVxC\n",
       "kntvcs7ZZ3ietc46077nvLAuOz/evPv7Sq01AAAwqjZouwAAAGiTQAwAwEgTiAEAGGkCMQAAI00g\n",
       "BgBgpAnEAACMtCkDcSnly6WUn5dSfriWY/6+lHJ7KeUHpZRXdbZEAADonul0iE9PMndNb5ZSDkuy\n",
       "R611zyT/KcmXOlQbAAB03ZSBuNb67SS/WsshRyb5vyuOvTrJ1qWU7TtTHgAAdFcnZojnJLlr0vOl\n",
       "SXbuwOcCAEDXbdihzymrPH/WftClFHtEAwDQE7XWVfPpGnUiEN+dZJdJz3de8dqz1CoT03/mzZuX\n",
       "efPmtV0GrJbfT/rVev1u1po89ljy8MPJI488fT/58XRe+/Wvk1KSjTZqbhtvPLPH6/Izq3s8a1Zz\n",
       "22CD7t9KWfvrZUUGLOXZj6f7fAiUGf6zdCIQL0jysSRfL6W8LslDtdafd+BzAYB+9NhjyYMPJtdf\n",
       "P70gu7r3Z81Kttoq2XLL5rby8eTXttsuedGLVv/eVlslW2yRbNipv+xmlE35W1RK+VqSNyXZtpRy\n",
       "V5JPJ9koSWqtJ9daLyylHFZKWZLk0SQf7GbBAEAPTEwkd92V/OhHyeLFz7y/995k002TK69cc5jd\n",
       "ZZc1v7fllskmm7T9TwhPmTIQ11qPn8YxH+tMOdB7Y2NjbZcAa+T3k6771a+eHXoXL06WLEm22SbZ\n",
       "e+9kr72a+0MOae532y1j3/lO4veTIVF6NddbSqlmiAGgBY8/ntxxx+q7vb/73TND78r7PfdsRhJg\n",
       "AJVSZnRRnUAMAMOg1uTuu58deBcvbl7fddfVB98ddhiqi6kgEYgBYLj95jfJokXPDr2335485zlN\n",
       "yF01+O6+e7MaAowIgRgAhs3ERLJwYXLKKcmFFzYBd9XQu9dezYVrgEAMAEPj3nuTr3wlOe20ZPbs\n",
       "5MMfTt797uZiN2CNZhqILd4HAP1k2bLkoouSU09Nrrgiecc7kq99LXn1q836QpcIxADQD+68s+kE\n",
       "n356s4bvCSck/+//NXPBQFcJxADQlscfTxYsaGaDr78+ec97km9+M3nZy9quDEaKQAwAvbZoUTMS\n",
       "8S//krz0pc1s8IIFze5vQM8JxADQC7/9bXLmmU03+I47kg98ILnqqmSPPdquDEaeVSYAoJtuuKEJ\n",
       "wWeckbz+9c1s8OGHWxcYusgqEwDQtocfTv71X5uxiAceSD70oeQHP0h23rntyoDV0CEGgE6otRmB\n",
       "OOWU5Jxzkre+tZkNPuigZIMN2q4ORoqNOQCgl+6/v7k47tRTmx3lTjghed/7kuc/v+3KYGQZmQCA\n",
       "bpuYSC69tAnBF1+cHHVUcvLJyQEH2DwDBpAOMQBM1913NxtnnHZasvXWzUjEu97VPAb6xkw7xIaa\n",
       "AGA6vvCFZsOMpUuTs85qNtL46EeFYRgCRiYAYG1qTT796WYN4R/+MJkzp+2KgA4TiAFgTWpN/tt/\n",
       "a+aFr7jChXIwpARiAFidiYnkP//nZjTissuS5z637YqALhGIAWBVy5Ylf/iHyZ13Jpdckmy5ZdsV\n",
       "AV0kEAPAZE88kbz73ckjjyQXXZRsvnnbFQFdJhADwEqPPZYcd1wya1ayYEGyySZtVwT0gGXXACBJ\n",
       "Hn00edvbkuc8p1lWTRiGkSEQA8DDDyeHHJLsumvy1a8mG23UdkVADwnEAIy2Bx5IDjwweeUrm62Y\n",
       "Z81quyKgxwRiAEbXffclY2NNIP6Hf0g28McijCL/5QMwmpYuTd70puQd70j+8i+TUtquCGiJVSYA\n",
       "GD0//nFy0EHNxhv/9b+2XQ3QMh1iAEbLbbc1neE/+RNhGEiiQwzAKLnppmTu3ORzn0ve//62qwH6\n",
       "hEAMwGi45prkiCOai+fe+c62qwH6iEAMwPD79reTt789Oe20JhQDTCIQAzDcLrkkede7kq99rbmQ\n",
       "DmAVLqoDYHidd17y7ncnZ58tDANrJBADMJzOOCM54YTkgguSAw5ouxqgjwnEAAyfr3wl+eM/bsYl\n",
       "XvOatqsB+pwZYgCGyz/9U7Os2mWXJS9+cdvVAANAIAZgePzVXzWB+Morkxe+sO1qgAEhEAMw+GpN\n",
       "PvOZZiWJK69Mdt657YqAASIQAzDYak0++cnkoouSK65Itt++7YqAASMQAzC4JiaSj3+82YXu8suT\n",
       "5z2v7YqAASQQAzCYli9vllW7/fbkW99Kttqq7YqAASUQAzB4nnwyec97kgcfTC6+OJk9u+2KgAEm\n",
       "EAMwWB57LHnnO5vZ4fPOSzbdtO2KgAFnYw4ABsejjyZHHtmE4PFxYRjoCIEYgMHwyCPJ3LnJjjsm\n",
       "//qvycYbt10RMCQEYgD634MPJgcdlLz0pcnppycbmvgDOqfUWnvzRaXUXn0XAEOk1uSAA5LXva7Z\n",
       "ia6UtisC+lwpJbXWaZ8sdIgB6G8XXJD85jfJ//pfwjDQFQIxAP2r1mTevOTTn0428EcW0B3OLgD0\n",
       "r/PPb9YcPvrotisBhphADEB/0h0GesQZBoD+dP75ybJlusNA1wnEAPSfld3hefN0h4Guc5YBoP+c\n",
       "d16yfHly1FFtVwKMAIEYgP5idhjoMWcaAPrLeeclExO6w0DPCMQA9A/dYaAFzjYA9I8FC5pQbGUJ\n",
       "oIcEYgD6w+TusC2agR4SiAHoDwsWNPdmh4EeE4gBaJ/uMNAigRiA9p17bnOvOwy0QCAGoF21Jied\n",
       "1HSIdYeBFgjEALRrZXf4yCPbrQMYWQIxAO3RHQb6gEAMQHvOPbcJwrrDQIs2bLsAAEbUxETTGf7M\n",
       "Z3SHgVbpEAPQjnPPbbZnPuKItisBRpwOMQC9NzHRzA7rDgN9QIcYgN4799xk1izdYaAv6BAD0Fsr\n",
       "Z4f/x//QHQb6gg4xAL11zjnJhhsmb3tb25UAJNEhBqCXVs4O6w4DfUSHGIDe0R0G+pAOMQC9oTsM\n",
       "9CkdYgB64+yzk4020h0G+o4OMQDdt7I7/NnP6g4DfUeHGIDuO/vsZOONk8MPb7sSgGeZMhCXUuaW\n",
       "Um4rpdxeSvnkat7ftpRyUSnlxlLKzaWUD3SlUgAG08ru8Lx5usNAX1prIC6lzEryj0nmJtk3yfGl\n",
       "lH1WOexjSW6otb4yyViSvy6lGMUAoDF/frLJJrrDQN+aqkO8f5IltdY7a61PJvl6kqNWOebeJFuu\n",
       "eLxlkgdqrcs6WyYAA0l3GBgAU3Vy5yS5a9LzpUleu8oxpyS5rJRyT5Itkryzc+UBMNDmz0823TQ5\n",
       "7LC2KwFYo6k6xHUan/FnSW6ste6U5JVJvlhK2WK9KwNgsOkOAwNiqg7x3Ul2mfR8lzRd4snekOR/\n",
       "Jkmt9Y5Syk+S7J3kulU/bN68eU89Hhsby9jY2IwLBmBAzJ+fbLaZ7jDQdQsXLszChQvX+edLrWtu\n",
       "Aq+4OG5xkgOT3JPkmiTH11oXTTrmb5I8XGs9qZSyfZLvJ3l5rfXBVT6rru27ABgiExPJK16RfP7z\n",
       "AjHQc6WU1Fqn/VdTa+0Q11qXlVI+luTiJLOSnFZrXVRK+ciK909O8tkkp5dSfpBmBOMTq4ZhAEbM\n",
       "+HjTHT700LYrAZjSWjvEHf0iHWKA0TAxkbz85ckXvqA7DLRiph1iO9UB0Fnj48nmm+sOAwPDBhoA\n",
       "dM7KlSW+8AUrSwADQ4cYgM4566xk9mzdYWCgmCEGoDMmJpKXvSz5q78SiIFWmSEGoB1nnZU85znJ\n",
       "3LltVwIwIzrEAKw/3WGgj+gQA9B7Z56ZbLGF7jAwkHSIAVg/y5c36w7/9V8LxEBf0CEGoLfOOqvp\n",
       "Dh9ySNuVAKwTHWIA1p3uMNCHdIgB6J2Vs8O6w8AA0yEGYN0sX96sLPG//7dADPQVHWIAeuPMM5Ot\n",
       "tkoOPrjtSgDWiw4xADOnOwz0MR1iALpPdxgYIjrEAMzM8uXJS1+a/O3f6g4DfUmHGIDu+sY3km22\n",
       "0R0GhoYOMQDTt7I7/Hd/JxADfUuHGIDuWdkdfutb264EoGN0iAGYHt1hYEDoEAPQHWeckTz3ubrD\n",
       "wNDRIQZgasuXJy95SfIP/yAQA31PhxiAzjvjjOR5z0sOOqjtSgA6TocYgLXTHQYGjA4xAJ319a/r\n",
       "DgNDTYcYgDVbvjzZd9/ki18UiIGBoUMMQOecf36z7vCBB7ZdCUDXCMQArNmZZybvfW9Spt1oARg4\n",
       "RiYAWL3HH0922CG55ZZkp53argZg2oxMANAZl17azA8Lw8CQE4gBWL3x8eTtb2+7CoCuMzIBwLMt\n",
       "W5bsuGNy3XXJC17QdjUAM2JkAoD1d8UVyW67CcPASBCIAXg24xLACDEyAcAzTUwkc+YkV16Z7Lln\n",
       "29UAzJiRCQDWz1VXJdttJwwDI0MgBuCZjEsAI2bDtgsAoI/U2gTiCy9suxKAntEhBuBp116bbLZZ\n",
       "8pKXtF0JQM8IxAA8beW4RJn2tSgAA08gBqCxclzC/DAwYgRiABo33ZQsX57st1/blQD0lEAMQMO4\n",
       "BDCiBGIAGsYlgBElEAOQ3HZb8tBDyWtf23YlAD0nEAPQdIePPTbZwB8LwOhx5gPAuAQw0gRigFH3\n",
       "4x8nS5cmv//7bVcC0AqBGGDUjY8nRx+dzJrVdiUArRCIAUadcQlgxJVaa2++qJTaq+8CYJqWLk1e\n",
       "/vLkvvuSjTduuxqAjiilpNY67UXVdYgBRtn8+ckRRwjDwEgTiAFG2fh4ctxxbVcB0CojEwCj6uc/\n",
       "T/beuxmX2HTTtqsB6BgjEwBMzznnJIceKgwDI08gBhhVVpcASGJkAmA0PfBAsvvuyT33JLNnt10N\n",
       "QEcZmQBgagsWJAceKAwDRCAGGE3GJQCeYmQCYNQ88kiy887JXXclW23VdjUAHWdkAoC1O//85Pd/\n",
       "XxgGWEEgBhg1xiUAnsHIBMAoefTRZKedkh//OHne89quBqArjEwAsGYXXZTsv78wDDCJQAwwSoxL\n",
       "ADyLkQmAUfH448kOOySLFjX3AEPKyAQAq3fJJcnLXiYMA6xCIAYYFcYlAFbLyATAKHjyyaYzfOON\n",
       "yS67tF0NQFcZmQDg2S6/PNljD2EYYDUEYoBRYFwCYI2MTAAMu+XLm804rroqedGL2q4GoOuMTADw\n",
       "TN/5TrLjjsIwwBoIxADDbnw8Oe64tqsA6FtGJgCG2cREsuuuzRrE++zTdjUAPWFkAoCnXXNNsuWW\n",
       "wjDAWgjEAMPM6hIAU9qw7QIA6JJam0A8f37blQD0NR1igGF1ww1JKckrXtF2JQB9TSAGGFYrxyXK\n",
       "tK8rARhJAjHAMFo5LmF+GGBKUwbiUsrcUsptpZTbSymfXMMxY6WUG0opN5dSFna8SgBm5tZbk0cf\n",
       "TV7zmrYrAeh7a72orpQyK8k/Jjkoyd1Jri2lLKi1Lpp0zNZJvpjkkFrr0lLKtt0sGIBpGB9Pjj02\n",
       "2cBfBAJMZaoz5f5JltRa76y1Ppnk60mOWuWYdyUZr7UuTZJa6y87XyYAM2J3OoBpmyoQz0ly16Tn\n",
       "S1e8NtmeSZ5bSrm8lHJdKeW9nSwQgBlasiT5+c+TN7yh7UoABsJU6xBPZ6/ljZLsl+TAJJsn+W4p\n",
       "5Xu11ttXPXDevHlPPR4bG8vY2Ni0CwVgmsbHk2OOSWbNarsSgJ5YuHBhFi5cuM4/X2pdc+Ytpbwu\n",
       "ybxa69wVz09MMlFr/fykYz6ZZLNa67wVz09NclGt9axVPquu7bsA6JD9908++9nkoIPargSgFaWU\n",
       "1FqnvebkVCMT1yXZs5SyWyll4yT/McmCVY45N8kBpZRZpZTNk7w2ya0zKRqADvnpT5Mf/zh505va\n",
       "rgRgYKx1ZKLWuqyU8rEkFyeZleS0WuuiUspHVrx/cq31tlLKRUluSjKR5JRaq0AM0Ib585Mjj0w2\n",
       "2qjtSgAGxlpHJjr6RUYmALrvgAOSE09MDj+87UoAWjPTkQmBGGBY3Htvsu++yX33JZts0nY1AK3p\n",
       "9AwxAIPi7LOTww4ThgFmSCAGGBbj48nb3952FQADx8gEwDD45S+TF72oGZvYfPO2qwFolZEJgFF0\n",
       "7rnJwQcLwwDrQCAGGAbGJQDWmZEJgEH30EPJrrsmd9+dbLFF29UAtM7IBMCoOf/8ZGxMGAZYRwIx\n",
       "wKAzLgGwXoxMAAyy3/wm2Wmn5Kc/TbbZpu1qAPqCkQmAUXLhhcnrXy8MA6wHgRhgkBmXAFhvRiYA\n",
       "BtXvfpfssENy++3J85/fdjUAfcPIBMCo+Ld/S171KmEYYD0JxACDanw8Oe64tqsAGHhGJgAG0RNP\n",
       "NOMSP/xhMmdO29UA9BUjEwCj4LLLkhe/WBgG6ACBGGAQWV0CoGOMTAAMmmXLkh13TK65JnnhC9uu\n",
       "BqDvGJkAGHZXXpnsuqswDNAhAjHAoDEuAdBRRiYABsnERLLzzsnllyd77912NQB9ycgEwDD77neT\n",
       "5z5XGAboIIEYYJAYlwDouA3bLgCAaao1mT8/Oe+8tisBGCo6xACD4vvfTzbeOHnpS9uuBGCoCMQA\n",
       "g2LluESZ9nUiAEyDQAwwCGo1PwzQJQIxwCC4+ebkiSeS//Af2q4EYOgIxACD4KyzkmOPNS4B0AUC\n",
       "McAgMC4B0DUCMUC/W7w4efDB5PWvb7sSgKEkEAP0u/Hx5Jhjkg2csgG6wdkVoN8ZlwDoqlJr7c0X\n",
       "lVJ79V0AQ+MnP0n23z+5995kQ5uLAkxHKSW11mlfhaxDDNDP5s9Pjj5aGAboIoEYoJ8ZlwDoOiMT\n",
       "AP3q7ruTl70sue++ZOON264GYGAYmQAYFmefnbztbcIwQJcJxAD9yrgEQE8YmQDoR7/4RbLnns24\n",
       "xGabtV0NwEAxMgEwDM45J5k7VxgG6AGBGKAfGZcA6BkjEwD95le/Sl7wguSee5LnPKftagAGjpEJ\n",
       "gEG3YEHylrcIwwA9IhAD9Jvx8eS449quAmBkGJkA6Ce//nUyZ07ys58lW2/ddjUAA8nIBMAgu+CC\n",
       "5IADhGGAHhKIAfqJ1SUAes7IBEC/+O1vkx13TO64I9l227arARhYRiYABtVFFyWvfrUwDNBjAjFA\n",
       "vzAuAdAKIxMA/eDxx5MddkhuvbUZmwBgnRmZABhE3/pW8pKXCMMALRCIAfqBcQmA1hiZAGjbk082\n",
       "neHrr0923bXtagAGnpEJgEFzxRXJ7rsLwwAtEYgB2mZcAqBVRiYA2rR8eTJnTvKd7yR77NF2NQBD\n",
       "wcgEwCC56qpk++2FYYAWCcQAbTrrLOMSAC3bsO0CAEbWxEQyf36zZTMArdEhBmjLtdcms2cn++7b\n",
       "diUAI00gBmjLytUlyrSv+wCgCwRigDbUark1gD4hEAO04Qc/aGaIX/WqtisBGHkCMUAbxseT444z\n",
       "LgHQBwRigDYYlwDoGwIxQK8tWpQ88kiy//5tVwJABGKA3hsfT449NtnAKRigHzgbA/SacQmAviIQ\n",
       "A/TSHXck99yTHHBA25UAsIJADNBL4+PJ0Ucns2a1XQkAKwjEAL1kXAKg75Raa2++qJTaq+8C6Et3\n",
       "3ZW88pXJffclG23UdjUAQ6uUklrrtBd61yEG6JX585MjjhCGAfqMQAzQKyt3pwOgrxiZAOiF++5L\n",
       "9tmnud9kk7arARhqRiYA+tE55ySHHioMA/QhgRigF6wuAdC3pgzEpZS5pZTbSim3l1I+uZbjXlNK\n",
       "WVZKObazJQIMuAceSK65Jpk7t+1KAFiNtQbiUsqsJP+YZG6SfZMcX0rZZw3HfT7JRUmmPa8BMBLO\n",
       "PTc56KBk9uy2KwFgNabqEO+fZEmt9c5a65NJvp7kqNUc9/EkZyW5v8P1AQw+4xIAfW2qQDwnyV2T\n",
       "ni9d8dpTSilz0oTkL614yVISACs9/HDy7W8nb3tb25UAsAYbTvH+dMLt3yb501prLaWUrGVkYt68\n",
       "eU89Hhsby9jY2DQ+HmCAnX9+8sY3Jltu2XYlAENr4cKFWbhw4Tr//FrXIS6lvC7JvFrr3BXPT0wy\n",
       "UWv9/KRjfpynQ/C2SX6b5MO11gWrfJZ1iIHRc+yxze50H/xg25UAjIyZrkM8VSDeMMniJAcmuSfJ\n",
       "NUmOr7UuWsPxpyc5r9Y6fzXvCcTAaHn00WSnnZKf/CR57nPbrgZgZMw0EK91ZKLWuqyU8rEkFyeZ\n",
       "leS0WuuiUspHVrx/8npVCzDMvvnN5LWvFYYB+pytmwG65fjjk7Gx5CMfabsSgJHS0ZGJThKIgZHy\n",
       "2GPJDjskixcn22/fdjUAI2WmgdjWzQDdcMklySteIQwDDACBGKAbbMYBMDCMTAB02hNPNOMSN92U\n",
       "7Lxz29UAjBwjEwBtu/zyZK+9hGGAASEQA3SacQmAgWJkAqCTli9Pdtwx+d73kt13b7sagJFkZAKg\n",
       "Td/+djJnjjAMMEAEYoBOGh9Pjjuu7SoAmAEjEwCdMjGR7LJLcumlyYtf3HY1ACPLyARAW66+Otl6\n",
       "a2EYYMAIxACdYnUJgIG0YdsFAAyFWptAfM45bVcCwAzpEAN0wvXXJ7NmJS9/eduVADBDAjFAJ6wc\n",
       "lyjTvoYDgD4hEAOsr5XjEuaHAQaSQAywvm65Jfnd75LXvKbtSgBYBwIxwPoaH0+OPda4BMCAEogB\n",
       "1pfd6QAGmkAMsD5uvz25//7kDW9ouxIA1pFADLA+xseTY45JNnA6BRhUzuAA68PqEgADr9Rae/NF\n",
       "pdRefRdAT/z0p8mrX53ce2+yoY0/AfpFKSW11mlf6axDDLCuxseTI48UhgEGnEAMsK6MSwAMBSMT\n",
       "AOvinnuSl7wkue++ZJNN2q4GgEmMTAD0wtlnJ4cfLgwDDAGBGGBdGJcAGBpGJgBm6v77kz32aMYl\n",
       "Ntus7WoAWIWRCYBuO/fc5JBDhGGAISEQA8yUcQmAoWJkAmAmHnoo2XXX5O67ky22aLsaAFbDyARA\n",
       "N513XvLmNwvDAENEIAaYCeMSAEPHyATAdP3618mcOclPf5pss03b1QCwBkYmALrlwguTN7xBGAYY\n",
       "MgIxwHQZlwAYSkYmAKbjd79LdtghWbIk2W67tqsBYC2MTAB0w8UXJ/vtJwwDDCGBGGA6xseT445r\n",
       "uwoAusDIBMBUnniiGZe4+eZkp53argaAKRiZAOi0Sy9N9tlHGAYYUgIxwFSsLgEw1IxMAKzNsmXJ\n",
       "jjsm116b7LZb29UAMA1GJgA66Yorkhe8QBgGGGICMcDaGJcAGHpGJgDWZGIimTOn6RLvtVfb1QAw\n",
       "TUYmADrlqquSbbcVhgGGnEAMsCbGJQBGwoZtFwDQl2pN5s9PLrig7UoA6DIdYoDVue66ZNNNk5e8\n",
       "pO1KAOgygRhgdVaOS5RpX5MBwIASiAFWVav5YYARIhADrOr665Mnn0z226/tSgDoAYEYYFWf+1zy\n",
       "8Y8blwAYETbmAJjsppuSgw9O7rgjmT277WoAWAc25gBYH5/5TPInfyIMA4wQHWKAlXSHAYaCDjHA\n",
       "ujrpJN1hgBGkQwyQJD/4QTJ3btMd3nzztqsBYD3oEAOsi5Wzw8IwwMjRIQbQHQYYKjrEADO1cnZY\n",
       "GAYYSTrEwGi78cbk0EN1hwGGiA4xwEx85jPJJz4hDAOMMB1iYHTpDgMMJR1igOnSHQYgOsTAqLrx\n",
       "xuSww5IlSwRigCGjQwwwHSedpDsMQBIdYmAU3XBDcvjhzezwZpu1XQ0AHaZDDDCVlbPDwjAA0SEG\n",
       "Ro3uMMDQ0yEGWJuTTko++UlhGICn6BADo0N3GGAk6BADrInuMACroUMMjAbdYYCRoUMMsDrz5ukO\n",
       "A7BaOsTA8Lv++uSII5pd6QRigKGnQwywKrPDAKyFDjEw3HSHAUZOVzrEpZS5pZTbSim3l1I+uZr3\n",
       "311K+UEp5aZSyr+XUl4+k6IBukZ3GIApTNkhLqXMSrI4yUFJ7k5ybZLja62LJh3z+iS31lofLqXM\n",
       "TTKv1vq6VT5Hhxjore9/PznyyGZliU03bbsaAHqkGx3i/ZMsqbXeWWt9MsnXkxw1+YBa63drrQ+v\n",
       "eHp1kp2nWwBA15x0UvKnfyoMA7BWG07jmDlJ7pr0fGmS167l+A8luXB9igJYb9//fnP7xjfargSA\n",
       "PjedQDztOYdSypuT/GGS31vnigA6QXcYgGmaTiC+O8kuk57vkqZL/AwrLqQ7JcncWuuvVvdB8+bN\n",
       "e+rx2NhYxsbGZlAqwDRdd12zuoTuMMBIWLhwYRYuXLjOPz+di+o2THNR3YFJ7klyTZ59Ud2uSS5L\n",
       "8p5a6/fW8DkuqgN644gjkkMOST72sbYrAaAFM72obsoOca11WSnlY0kuTjIryWm11kWllI+seP/k\n",
       "JP89yTZJvlRKSZIna637r8s/AMB6ue665IYbkjPPbLsSAAaEjTmA4aI7DDDyZtohFoiB4XHddcnR\n",
       "Rze70rmYDmBkdWWnOoCBMG9ecuKJwjAAMzKdVSYA+t+11yY33picdVbblQAwYHSIgeFw0km6wwCs\n",
       "Ex1iYPDpDgOwHnSIgcFndhiA9aBDDAy2a65JbropmT+/7UoAGFA6xMBgWzk7vMkmbVcCwIDSIQYG\n",
       "l+4wAB2gQwwMLt1hADpAhxgYTFdfnfzwh7rDAKw3HWJgMOkOA9AhOsTA4FnZHT777LYrAWAI6BAD\n",
       "g+ekk5I/+zPdYQA6QocYGCxXX53cfLPuMAAdo0MMDJZ583SHAegoHWJgcHzve8kttyTnnNN2JQAM\n",
       "ER1iYHCYHQagC3SIgcHwve8lt96qOwxAx+kQA4PB7DAAXaJDDPS/7343WbQoWbCg7UoAGEI6xED/\n",
       "Wzk7vPHGbVcCwBDSIQb6m+4wAF2mQwz0t5NOSj71Kd1hALpGIAb618ru8Ac+0HYlAAwxgRjoX/Pm\n",
       "6Q4D0HUCMdCfrroque023WEAuk4gBvqT2WEAekQgBvrPVVclixfrDgPQEwIx0H/MDgPQQwIx0F/+\n",
       "/d+TH/0oef/7264EgBEhEAP941e/Sj7+8eTP/1x3GICeEYiB/nD//clb3pK86U3Jhz7UdjUAjBCB\n",
       "GGjfPfcWGizxAAAJvUlEQVQ0Qfhtb0v+5m+SUtquCIARIhAD7frpT5M3vjF573uTv/gLYRiAnhOI\n",
       "gfbcfnsThv/Lf0lOPLHtagAYURu2XQAwom65JTn44GYDjhNOaLsaAEaYQAz03vXXJ4cd1swLv+td\n",
       "bVcDwIgTiIHe+u53k6OOSk4+OTnmmLarAQCBGOihyy9P3vnO5J//OTn00LarAYAkAjHQK9/8ZvK+\n",
       "9yVnnpmMjbVdDQA8xSoTQPfNn99sxbxggTAMQN8RiIHu+upXk49+NLnoouT1r2+7GgB4FoEY6J5T\n",
       "Tkk+8Ynk0kuT/fZruxoAWC0zxEB3/N3fNcuqLVyY7Lln29UAwBoJxEDnffazyZe/nFx5ZfKCF7Rd\n",
       "DQCslUAMdE6tyZ//eXL22U0Y3mmntisCgCkJxEBn1Jr88R8nV1zR3Lbbru2KAGBaBGJg/S1fnvzR\n",
       "HyU33ZRcdlmyzTZtVwQA0yYQA+tn2bLkAx9Ili5NLrkk2WKLtisCgBkRiIF198QTyfHHJ48+mlx4\n",
       "YbL55m1XBAAzZh1iYN387nfJ0UcnExPJuecKwwAMLIEYmLnf/CY5/PBk662Tb3wj2WSTtisCgHUm\n",
       "EAMz89BDycEHJ7vvnvzLvyQbbdR2RQCwXgRiYPp++cvkLW9JXv3q5P/8n2TWrLYrAoD1JhAD03Pv\n",
       "vcnYWHLIIc22zBs4fQAwHPyJBkztZz9L3vjGZkWJz30uKaXtigCgYwRiYO2WLGnC8Ec/mnzqU21X\n",
       "AwAdJxADa3brrc2YxIknNtsyA8AQsjEHsHo33pgcemjyhS8k731v29UAQNcIxMCzXX11cuSRyRe/\n",
       "mBx3XNvVAEBXCcTAM11xRfKOdySnn95svgEAQ04gBp528cXJe96TfP3ryYEHtl0NAPSEQAyjrtbk\n",
       "+99PTj01mT8/Oeec5Pd+r+2qAKBnrDIBo+qhh5oZ4f32a0YkdtmluZBOGAZgxJRaa2++qJTaq+8C\n",
       "1qDW5DvfSU45JVmwoNl17sMfbrZjtvMcAEOilJJa67R3kRKIYRT84hfJP/9zMxaxwQbJCSc0S6lt\n",
       "t13blQFAx800EJshhmE1MZFcckkTgi+5JDnmmOS005I3vMHWywAwiQ4xDJulS5Mvf7m5Pe95zUjE\n",
       "8ccnW23VdmUA0BM6xDCKnnwyueCCZjb4u99N/uAPmhUj9tuv7coAoO8JxDDIlixpxiC+8pVkjz2a\n",
       "2eAzz0w237ztygBgYAjEMGgee6zp/p56anLzzcn73pdcdlmyzz5tVwYAA0kghkFx883NSMRXv9qM\n",
       "QvzRHyVHHplssknblQHAQBOIoZ/95jfJGWc0QXjp0uSDH0yuvTZ54QvbrgwAhoZVJqDf1NqE3lNP\n",
       "beaB3/SmZjZ47txkQ/8PCwBTscoEDKoHH2zGIU49tekMn3BCcsstyU47tV0ZAAw1HWLotVqT++9P\n",
       "Fi9OfvSj5n7RouTb304OPbQJwm9+s62UAWAd2boZ+sVvf5vcfvvToXfyfSnJ3ns3t732am5jY8m2\n",
       "27ZdNQAMPIEYemn58uRnP3t24F28OPnFL5IXvejp0Dv5/nnPs30yAHSJQAzd8MADzwy7Kx/fcUfT\n",
       "1V1d6H3BC5JZs9quHABGjkAM6+qxx5qd31bX7V22bPWhd889k9mz264cAJik44G4lDI3yd8mmZXk\n",
       "1Frr51dzzN8nOTTJb5N8oNZ6w2qOEYjpjccfTx55JHn44eZ+8uPVvLZw8eKMPfBAcu+9yW67PTv0\n",
       "7r138vznG3GgFQsXLszY2FjbZcCz+N2kn3V02bVSyqwk/5jkoCR3J7m2lLKg1rpo0jGHJdmj1rpn\n",
       "KeW1Sb6U5HXrVD2jbdmyZwfYKcLsal9bvjzZaqtkyy2fvl/d4112SbbaKguTjM2b14ThjTZq998B\n",
       "rELooF/53WSYTLUO8f5JltRa70ySUsrXkxyVZNGkY45M8n+TpNZ6dSll61LK9rXWn3ehXlZVazIx\n",
       "8fT9dG/LlydPPvn07YknnnnfqcdTvf/oo0+H2cceezqwTg6wqwbZ7bdffdBdeb/ppjPr5i5Z0ow+\n",
       "AAAjaapAPCfJXZOeL03y2mkcs3OSZwfiQw5p7lcdnVjb85kcO52fXfnaysdtPF81vM40zE6+JU34\n",
       "22CDmd822qi5bbzx2h9P9f5GGyWbbTb9z5v8ePbsp8Ps7NnGEgCAnlvrDHEp5e1J5tZaP7zi+XuS\n",
       "vLbW+vFJx5yX5C9rrf++4vm3knyi1nr9Kp9lgBgAgJ7o5NbNdyfZZdLzXdJ0gNd2zM4rXlvnogAA\n",
       "oFem2hv2uiR7llJ2K6VsnOQ/JlmwyjELkrwvSUopr0vykPlhAAAGxVo7xLXWZaWUjyW5OM2ya6fV\n",
       "WheVUj6y4v2Ta60XllIOK6UsSfJokg92vWoAAOiQnm3MAQAA/WiqkYn1Ukp5RynlllLK8lLKfqu8\n",
       "d2Ip5fZSym2llIO7WQdMpZQyr5SytJRyw4rb3LZrYrSVUuauOD/eXkr5ZNv1wGSllDtLKTetOF9e\n",
       "03Y9jK5SypdLKT8vpfxw0mvPLaVcUkr5USnl30opW0/1OV0NxEl+mOSYJFdOfrGUsm+aeeR9k8xN\n",
       "8k+llG7XAmtTk/xNrfVVK24XtV0Qo2vSpkhz05wnjy+l7NNuVfAMNcnYivPl/m0Xw0g7Pc25crI/\n",
       "TXJJrXWvJJeueL5WXQ2htdbbaq0/Ws1bRyX5Wq31yRWbfixJswkItMlKKPSLpzZFqrU+mWTlpkjQ\n",
       "T5wzaV2t9dtJfrXKy09tGrfi/uipPqetruxOeebybUvTbPABbfp4KeUHpZTTpvPXK9BFq9vwyDmS\n",
       "flKTfKuUcl0p5cNtFwOrmLxj8s+TbD/VD0y1DvGUSimXJNlhNW/9Wa31vBl8lKv76Kq1/K5+KsmX\n",
       "knxmxfO/SPLXST7Uo9JgVc6H9Lvfq7XeW0rZLsklpZTbVnTqoK/UWut0Nodb70Bca33rOvzYtDbz\n",
       "gE6a7u9qKeXUJDP5nznotOlsigStqbXeu+L+/lLK2WnGfARi+sXPSyk71FrvK6XsmOQXU/1AL0cm\n",
       "Js8aLUjyB6WUjUspL0yyZxJXqdKaFf/BrHRMmgtCoS3T2RQJWlFK2byUssWKx7OTHBznTPrLgiTv\n",
       "X/H4/UnOmeoH1rtDvDallGOS/H2SbZNcUEq5odZ6aK311lLKN5LcmmRZko9WCyLTrs+XUl6Z5q+q\n",
       "f5LkIy3Xwwhb06ZILZcFK22f5OxSStLkiK/WWv+t3ZIYVaWUryV5U5JtSyl3JfnvSf4yyTdKKR9K\n",
       "cmeSd075OXIoAACjzNq/AACMNIEYAICRJhADADDSBGIAAEaaQAwAwEgTiAEAGGkCMQAAI+3/AzmO\n",
       "nhJtkQmYAAAAAElFTkSuQmCC\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0xd120128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nums = np.arange(-10, 10, step=1)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12,8))\n",
    "ax.plot(nums, sigmoid(nums), 'r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent!  Now we need to write the cost function to evaluate a solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def cost(theta, X, y):\n",
    "    theta = np.matrix(theta)\n",
    "    X = np.matrix(X)\n",
    "    y = np.matrix(y)\n",
    "    first = np.multiply(-y, np.log(sigmoid(X * theta.T)))\n",
    "    second = np.multiply((1 - y), np.log(1 - sigmoid(X * theta.T)))\n",
    "    return np.sum(first - second) / (len(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to do some setup, similar to what we did in exercise 1 for linear regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# add a ones column - this makes the matrix multiplication work out easier\n",
    "data.insert(0, 'Ones', 1)\n",
    "\n",
    "# set X (training data) and y (target variable)\n",
    "cols = data.shape[1]\n",
    "X = data.iloc[:,0:cols-1]\n",
    "y = data.iloc[:,cols-1:cols]\n",
    "\n",
    "# convert to numpy arrays and initalize the parameter array theta\n",
    "X = np.array(X.values)\n",
    "y = np.array(y.values)\n",
    "theta = np.zeros(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's quickly check the shape of our arrays to make sure everything looks good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((100L, 3L), (3L,), (100L, 1L))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, theta.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's compute the cost for our initial solution (0 values for theta)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.69314718055994529"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cost(theta, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good.  Next we need a function to compute the gradient (parameter updates) given our training data, labels, and some parameters theta."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def gradient(theta, X, y):\n",
    "    theta = np.matrix(theta)\n",
    "    X = np.matrix(X)\n",
    "    y = np.matrix(y)\n",
    "    \n",
    "    parameters = int(theta.ravel().shape[1])\n",
    "    grad = np.zeros(parameters)\n",
    "    \n",
    "    error = sigmoid(X * theta.T) - y\n",
    "    \n",
    "    for i in range(parameters):\n",
    "        term = np.multiply(error, X[:,i])\n",
    "        grad[i] = np.sum(term) / len(X)\n",
    "    \n",
    "    return grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we don't actually perform gradient descent in this function - we just compute a single gradient step.  In the exercise, an Octave function called \"fminunc\" is used to optimize the parameters given functions to compute the cost and the gradients.  Since we're using Python, we can use SciPy's \"optimize\" namespace to do the same thing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a single call to the gradient method using our data and initial paramter values of 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ -0.1       , -12.00921659, -11.26284221])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradient(theta, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use SciPy's truncated newton (TNC) implementation to find the optimal parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([-25.87355624,   0.21193682,   0.20722586]), 51, 1)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import scipy.optimize as opt\n",
    "result = opt.fmin_tnc(func=cost, x0=theta, fprime=gradient, args=(X, y))\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the our cost looks like with this solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.20357134412164668"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cost(result[0], X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to write a function that will output predictions for a dataset X using our learned parameters theta.  We can then use this function to score the training accuracy of our classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def predict(theta, X):\n",
    "    probability = sigmoid(X * theta.T)\n",
    "    return [1 if x >= 0.5 else 0 for x in probability]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy = 89%\n"
     ]
    }
   ],
   "source": [
    "theta_min = np.matrix(result[0])\n",
    "predictions = predict(theta_min, X)\n",
    "correct = [1 if ((a == 1 and b == 1) or (a == 0 and b == 0)) else 0 for (a, b) in zip(predictions, y)]\n",
    "accuracy = (sum(map(int, correct)) % len(correct))\n",
    "print 'accuracy = {0}%'.format(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our logistic regression classifer correctly predicted if a student was admitted or not 89% of the time.  Not bad!  Keep in mind that this is training set accuracy though.  We didn't keep a hold-out set or use cross-validation to get a true approximation of the accuracy so this number is likely higher than its true perfomance (this topic is covered in a later exercise)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularized logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the second part of this exercise, we'll improve our logistic regression algorithm from part one by adding a regularization term.  If you're not familiar with regularization, or would like some background on the equations used in this section, refer to \"ex2.pdf\" in the \"exercises\" folder.  In a nutshell, regularization is a term in the cost function that causes the algorithm to prefer \"simpler\" models (in this case, models will smaller coefficients).  The theory is that this helps to minimize overfitting and improve the model's ability to generalize.  With that, let's get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose you are the product manager of the factory and you have the test results for some microchips on two different tests.  From these two tests, you would like to determine whether the microchips should be accepted or rejected.  To help you make the decision, you have a dataset of test results on past microchips, from which you can build a logistic regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to part 1, let's start by visualizing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Test 1</th>\n",
       "      <th>Test 2</th>\n",
       "      <th>Accepted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 0.051267</td>\n",
       "      <td> 0.69956</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.092742</td>\n",
       "      <td> 0.68494</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.213710</td>\n",
       "      <td> 0.69225</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.375000</td>\n",
       "      <td> 0.50219</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.513250</td>\n",
       "      <td> 0.46564</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Test 1   Test 2  Accepted\n",
       "0  0.051267  0.69956         1\n",
       "1 -0.092742  0.68494         1\n",
       "2 -0.213710  0.69225         1\n",
       "3 -0.375000  0.50219         1\n",
       "4 -0.513250  0.46564         1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = os.getcwd() + '\\data\\ex2data2.txt'\n",
    "data2 = pd.read_csv(path, header=None, names=['Test 1', 'Test 2', 'Accepted'])\n",
    "data2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x17856dd8>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAtwAAAHuCAYAAACs89tSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt4W9Wd7//PsoIJcUIICZcACZdMaYdOae1yuNa2NE5D\n",
       "LoShhbRwfiTMnJaSmVN6IW7pzRMH95RecNppZ4ZAL3PI0KdApoWGkpBQ29umnR+kRIYyLZSmlISE\n",
       "W2lTk3FwnMjr/CHJkWU5lmRt7Yver+fRE1l7W1pb2pG+XvqstYy1VgAAAADcUeV1AwAAAIAwo+AG\n",
       "AAAAXETBDQAAALiIghsAAABwEQU3AAAA4CIKbgAAAMBFnhbcxpjvGWNeNcY8Pcb2qDGmzxjTm7p8\n",
       "odxtBAAAACZikseP/2+SviVp/RH26bbWXl6m9gAAAAAl5WkPt7X2UUl7x9nNlKMtAAAAgBv8nuG2\n",
       "ki42xjxljNlkjDnH6wYBAAAAhfA6UjKeuKQ51tr9xphFkh6QdHb2TsYY1qcHAACA66y1BacvfF1w\n",
       "W2v3ZVzfbIz5V2PM8dbaP+XYt7yNg++1traqtbXV62bAZzgvkI1zArlwXiAXY4pLOvs6UmKMOcmk\n",
       "jswYc74kk6vYBgAAAPzK0x5uY8wPJDVKmmWMeVHSaklHSZK19g5JV0n6e2PMIUn7JV3tVVsBAACA\n",
       "YnhacFtrrxln+79I+pcyNQchE41GvW4CfIjzAtk4J5AL5wVKyYQh+2yMsWE4DgAAAPiXMSZ8gyYB\n",
       "AAAqTbED81BapezMpeAGAADwGb6591ap/+jx9SwlAAAAQNBRcAMAAAAuouAGAAAAXETBDQAAALiI\n",
       "ghsAACAkrLUaHBz0uhmB0NraquXLl5flsSi4AQAAAu6///u/9bGPfUrTpp2gyZOn6LTT3qbbb7/D\n",
       "tdlOotGojj/+eE+L+zPOOEOdnZ1F/345p1+k4AYAAPC5p59+WkuWfEBHHz1VkycfqyuvXK7nnntO\n",
       "knTw4EE1NCzSnXe+rP7+bbL2oPbs+a6am+/Qpz/9heH7sNbqscce09q1a/W9731Pe/fuLaotL7zw\n",
       "grZt26YTTzxRGzduLMnxFSO1CI1nj18ICm4AAAAf6+3t1UUXNWnz5os1OLhTBw7s0AMPnKP/8T8a\n",
       "9Jvf/EYPPPCAfvvbIR04sF7SWZKMpEu0f/8W/fM//6teeeUV/fnPf9aFFzZp/vzl+uxnd+pjH9ui\n",
       "U045S9/97r8V3J7169dr/vz5Wr58ue66667h21988UW9//3v14knnqhZs2bpxhtvHN727W9/W+ec\n",
       "c46OPfZYvf3tb1dvb68k6aWXXtKVV16pE088UWeddZa+9a1vDf9Oa2urrrrqKl199dU69thj9e53\n",
       "v1u//OUvJUnLly/Xrl27tHTpUk2bNk233XabJOmxxx7TxRdfrBkzZuhd73qXuru7h+/v97//vRob\n",
       "G3XsscdqwYIFev311ws+9qJZawN/SR4GAABA8GXXNfX1i6y0zkp2xMWYL9srrvif9sorV+TcLlk7\n",
       "deoye9ddd9lFi66y1dV/b6VExvbf2ClTZtvHHnusoPbNmzfP3n333fa5556zRx11lH3ttdfsoUOH\n",
       "7Lnnnmtvuukmu3//fjswMGB/9rOfWWutve++++ypp55qn3jiCWuttTt27LA7d+60iUTC1tXV2ba2\n",
       "Nnvw4EH7/PPP27POOstu2bLFWmvt6tWr7VFHHWV/+MMf2kOHDtnbbrvNnnnmmfbQoUPWWmvPOOMM\n",
       "29HRMdyu3bt325kzZ9rNmzdba6195JFH7MyZM+3rr79urbX2wgsvtKtWrbKDg4O2p6fHTps2zS5f\n",
       "vjyv1yDr9oJrVXq4AQAAfGpgYED/+Z+dklaM2mbth/XQQ/ePex9/+tOf1NXVqcHBr2lkuOFsvfnm\n",
       "p/S1r/1L3u352c9+pj179ujyyy/XW97yFp1zzjn6/ve/r23btunll1/W1772NR1zzDE6+uijdckl\n",
       "l0iSvvOd7+jmm2/Wu9/9bknSvHnzNHfuXP3iF7/Q66+/ri984QuaNGmSzjzzTH34wx/WPffcM/x4\n",
       "5513nt7//vcrEonopptu0sDAgB577LGcbbv77ru1ePFiLVy4UJI0f/58nXfeeXrooYe0a9cuPfHE\n",
       "E2pra9NRRx2l+vp6LV26tGyRFApuAAAAnxoaGkpdm5Rja7WGhhL6wAeWaOrU9ZKGsrb/QYcOPaI5\n",
       "c+bo6KP/SlLNqHuw9mI9/fSzebfnrrvu0oIFCzRt2jRJ0rJly3TXXXdp9+7dOv3001VVNbq03L17\n",
       "t+bNmzfq9p07d+qll17SjBkzhi+33nqrXnvtteF9TjvttOHrxhiddtppeumll3K2befOndqwYcOI\n",
       "+/v5z3+uV155ZfhxjjnmmOH9Tz/99LyPe6JyvXoAAADwgSlTpuid77xQ8fh/SLoma+vdisUW633v\n",
       "e5++/OVv6de/XqEDB26RdKakn2vKlI/pH/7hH3TuuedqcPA5SQclHZV1H7/S6aefpny8+eabuu++\n",
       "+zQ0NKTZs2dLkg4cOKC+vj6ddNJJ2rVrlxKJhCKRyIjfmzNnjnbs2DHq/ubOnaszzzxzePBnLi++\n",
       "+OLw9aGhIe3evVunnHKKpNGzjMydO1fLly/XnXfeOep+du7cqb1792r//v2aMmXK8G3ZbXULPdwA\n",
       "AAA+9vWv36Jjjvm4pHslHZJ0QNL/1ZQp/6ivfKVFRx11lHp6NuuGG2arpuZ8GXOUTj31w7rtthv0\n",
       "1a9+UW95y1v09re/TZHI17Pu+Y+qqfmybrrpI3m144EHHtCkSZP0zDPP6KmnntJTTz2lZ555Ru95\n",
       "z3t0//33a/bs2frMZz6j/fv3p6Iw/ylJ+vCHP6zbbrtN8Xhc1lrt2LFDu3bt0vnnn69p06bpq1/9\n",
       "qt58800lEgn913/9l5544onhx9y+fbvuv/9+HTp0SN/4xjc0efJkXXjhhZKkk046Sb/73e+G9732\n",
       "2mv14IMPauvWrUokEhoYGJDjONqzZ49OP/10nXfeeVq9erUOHjyon/3sZ/rJT35S/ItSqGKC3367\n",
       "iEGTAAAgJHLVNY7j2He9q94eddQUO2nSMfb885vs448/Pmq/oaEhOzg4OOr23//+93b27Hl26tRL\n",
       "rfQtG4l83h5zzGzb3Py5vNu1cOFC29zcPOr2++67z86ePdu++OKL9oorrrAzZ860s2bNsh//+MeH\n",
       "91m3bp1961vfaqdOnWrf8Y532CeffNJaa+1LL71kr7nmGnvyySfbGTNm2Isuumh4IGRra6u96qqr\n",
       "7Ac/+EE7bdo0W1dXZ3t7e4fv88c//rGdO3euPe6442x7e7u11trHH3/cNjY22uOPP96ecMIJ9rLL\n",
       "LrO7du2y1lr7/PPP2/r6ejt16lT73ve+1954441lGzRpbJnC4m4yxtgwHAcAAMCR5pd+4403ZIwZ\n",
       "zlAXYmBgQPfee6+6uh7TzJnTdd11/1PnnnvuRJvrmjVr1mjHjh3693//97I/9livQer2glfMIcMN\n",
       "AAAQEMcee2zRvzt58mRdd911uu6660rYIveEqTOVDDcAAAB8xxhT1uXX3USkBAAAwEeCtGR5WJU6\n",
       "UkIPNwAAAOAiCm4AAADARRTcAAAAgIsouAEAAAAXUXADAAAALqLgBgAAgGsWL17syeI1mV544QVV\n",
       "VVVpaGjIk8en4AYAAAiyxx+XPvUpKXsau74+6dprpb17S/IwZ5xxhqZMmaJp06bp5JNP1vLly/XG\n",
       "G2+M+3ubNm3S8uXLJ/TYra2tE74PL1FwAwAA+NmaNVI8Pvr2nh7pttukt70tef1jHztcdPf1SQsW\n",
       "SDNmSMcdJ738cu773rtXevPNvJphjNFPfvIT7du3T0899ZSefvppffGLXyzyoCoLBTcAAICfvfOd\n",
       "0qJFI4vunh7pqquk2lpp+nRp61Zp27Zk0f3nPyeL7fPPl775TSmRkGIxad26kfe7d680f75URNzj\n",
       "pJNO0oIFC/SrX/1KkvTYY4/p4osv1owZM/Sud71L3d3dw/tGo1F997vfHf75e9/7ns455xwdf/zx\n",
       "WrhwoXbt2jW87Ve/+pXe+973aubMmTr55JN16623asuWLbr11lt17733atq0aaqtrZUk9fX16UMf\n",
       "+pBOOeUUnXbaaWppaRmOjAwNDam5uVknnHCC5s2bp4ceeqjgYywlCm4AAAA/u+IK6Y47Dhfd6WL7\n",
       "Bz+QmpqS+6SL7p/+NNmrnS62jZEmTZIeeki69dbDRXe62I5Gpeuvz7sp6dUXd+/erYcfflgXXHCB\n",
       "9uzZo8suu0z/+I//qL179+q2227TlVdeqT/+8Y+SRi7R/uMf/1i33nqr7r//fr3++uuqr6/XNddc\n",
       "I0nat2+f5s+fr8WLF+vll1/Wjh071NTUpEsvvVSf+9zndPXVV2vfvn3q7e2VJP3t3/6tqqur9bvf\n",
       "/U69vb3aunWrvvOd70iS7rzzTj300EN68skn9cQTT+g//uM/PF0mnoIbAADA79JF97vfLTU2jiy2\n",
       "8zFvntTZmSy6b731cLF9223JojwP1lpdccUVOvbYYzV37lzNmzdPn//853X33Xdr8eLFWrhwoSRp\n",
       "/vz5Ou+883L2Kq9bt06f/exn9da3vlVVVVX67Gc/qyeffFK7du3ST37yE51yyin65Cc/qerqak2d\n",
       "OlXnn3/+8GNnLrX+6quvavPmzfr617+uY445RieccII+8YlP6J577pEk3XffffrkJz+pU089VTNm\n",
       "zNDnPve5nEu1lwsFNwAAQBAcf/zh6zNmjNyWzmzPn5/svU7HSzKLzHnzpB/9SPrc56Tf/KagYltK\n",
       "9lT/+Mc/1htvvCHHcdTZ2ant27dr586d2rBhg2bMmDF8+fnPf65XXnll1H3s3LlTH//4x4f3mzlz\n",
       "piRpz5492r17t84666y82rJz504dPHhQs2fPHr6vlStX6g9/+IMk6eWXX9acOXOG9587d27ex+kG\n",
       "Cm4AAAC/S8dIfvpT6f77R2a608V2OkZy3HEjM93ponvvXukjH0n2ls+cmewxL1JDQ4NuvPFG3Xzz\n",
       "zZo7d66WL1+uvXv3Dl/27dunT3/606N+b+7cubrzzjtH7Nvf36+LLrpIc+bM0fPPP5/z8aqqRpas\n",
       "c+bM0dFHH60//vGPw/fT19enp59+WpI0e/bsEdnwzOteoOAGAADws+zMdnam+9lnpYaGw5lt6XCm\n",
       "e+/e5CDKzMz2j350OF6SPZCyAJ/4xCe0bds2vec979GDDz6orVu3KpFIaGBgQI7jaM+ePaN+Z+XK\n",
       "lfrSl76kX//615KSAx83bNggSbrsssv08ssv65/+6Z904MAB7du3T9u2bZOUHKT5wgsvDMdCZs+e\n",
       "rQULFuimm27Svn37NDQ0pN/97nfq6emRJH3gAx/QN7/5Te3Zs0d79+7Vl7/85aKPsxQouAEAAPzs\n",
       "hz8cndlOF90//KF0wQXS1742Oh4yfbp0993StGnJHvDMzHZmpvsHPyiqWbNmzdJ1112ntWvXauPG\n",
       "jfrSl76kE088UXPnzlV7e3vOzPQVV1yhm2++WVdffbWmT5+ud7zjHdqyZYskaerUqXrkkUf04IMP\n",
       "avbs2Tr77LPlOI4kadmyZZKkmTNn6rzzzpMkrV+/XoODg8Mznixbtmw4xnL99dfr0ksv1Tvf+U6d\n",
       "d955uvLKKz0dNGm8DJCXijHGhuE4AAAAjDGlH+D3+OPJyEl20fn889LUqdKJJ5b28TI0Njbq+uuv\n",
       "17XXXuvaY5TaWK9B6vaCK/dJJWkVAAAA/OuCC3LfnucgxWLt379fzz//vM4880xXH8fviJQAAACg\n",
       "5F577TXNnj1b0WhUl1xyidfN8RSREgAAAB9xJVKCgpQ6UkIPNwAAAOAiCm4AAADARRTcAAAAgIuY\n",
       "pQQAAMBnvJwzGqVHwQ0AAOAjDJgMHyIlAAAAgIsouAEAAAAXUXADAAAALqLgBgAAAFxEwQ0AAAC4\n",
       "iIIbAAAAcBEFNwAAAOAiCm4AAADARRTcAAAAgIsouAEAAAAXUXADAAAALqLgBgAAAFxEwQ0AAAC4\n",
       "iIIbAAAAcBEFNwAAAOAiCm4AAADARRTcAPyhpUVynNG3O05yGwAAAUXBDcAfmpqkZctGFt2Ok7yt\n",
       "qcmrVgEAMGHGWut1GybMGGPDcBxAxUsX2Bs2JH9OX49GvWwVAACSJGOMrLWm4N8LQ6FKwQ2EiONI\n",
       "sVjyelcXxTYAwDeKLbiJlAAARiJPDwAlRcENwD/SkZKuruQlO9ON8iBPDwAlRaQEgD9k5rfTMZJc\n",
       "t6E8yNMDwChkuENwHEBFa2lJ9p5mF3SOI3V0SG1tXrSqspGnB4ARii24J7nRGAAo2FgFdTRKoQcA\n",
       "CDQy3ACA0cjTA0DJUHADAEbKzs5Ho8nrFN0AUBQKbgDASB0dowdIpovujg6vWgUAgcWgSQAAACAP\n",
       "LHwDAAAA+BAFNwAAAOAiCm4AAADARRTcAAAAgIsouAEAAAAXUXADAAAALvK04DbGfM8Y86ox5ukj\n",
       "7PNNY8xvjTFPGWNqy9k+AAAAYKK87uH+N0kLx9pojFks6S+stW+R9BFJt5erYQiBlpbcq+I5TnIb\n",
       "wo3XHwDgE54W3NbaRyXtPcIul0u6K7Xv45KOM8acVI62IQSamkYvRZ1esrqpyatWoVyFMK8/AMAn\n",
       "vO7hHs+pkl7M+Hm3pNM8aguCJr0UdbroShdb2UtWo7zKVQjz+gMAfGKS1w3IQ/bymazhjvyli65Y\n",
       "LPlzVxfFltcyC+ENG5K3uVUI8/oDAHzA7wX3HklzMn4+LXXbKK2trcPXo9GoonyoAv5FIQwACADH\n",
       "ceTkikEWyFjrbYexMeYMSQ9aa9+RY9tiSR+11i42xlwo6RvW2gtz7Ge9Pg74VGaMQAp9pCCRSEiS\n",
       "IpGIxy3Jg+O4X3BX2OsPAHCXMUbW2uz0xbi8nhbwB5L+U9JbjTEvGmP+lzHmBmPMDZJkrd0k6Xlj\n",
       "zA5Jd0j6Bw+bi6DJzuxmZ3pDJB6PKxZbqurqyaqunqxYbKl6e3u9btbY0q9NV1fy4sZrUkGvPwDA\n",
       "3zzv4S4FeriRU0tLchBedm+m40gdHVJbmxetKrl4PK6GhkvV398maUXq1vWqqWlRT88W1dXVedm8\n",
       "0XINXnRjQGOFvP4AgPIptoebghsIuFhsqRxniaSVWVvWKRbbpM7OjV40a2wUwgCAgKLgDsFxAIVK\n",
       "JBKqrp6soaE+SVOytu5XVdV0DQ4OBCPTDQCAzwUyww0AAACEHQU3EGCRSEQNDQslrc+xdb0aGxfR\n",
       "uw0AgMcouIGAW7v2FtXUtEhaJ2l/6rJONTUtam9f423jUBotLblnVnGc5DYAgK9RcAMBV1tbq56e\n",
       "LYrFNqmqarqqqqYrFtukRx/dqtraWq+bh1Joaho9nWF6ZpemJq9aBQDIE4MmgVLxwewbgVr4Jgh8\n",
       "8JqOeEwW8QEATzFoEvCaD3ohI5EIxXYp+eA1HZZeuCcWS14otgEgMCZ53QAgNDJXMqQXMhx4TQEA\n",
       "JUCkBMHnp6/9048biyWvd3VRmIWBH15TIiUA4DkiJahcfvraH3BDZrEdjY7sec81ewkAwFcouBF8\n",
       "2cVHdnFSTunH7upKXiiIgs8Pr2lHx+jzOX3ed3SUty1wD9M/AqFFpATh4fXX/rkKfS+Lf0wcrynK\n",
       "ifMN8D0iJYDX6IUMH15TlJOfvq0DUFL0cCMcGFAGICy8/rYOwJjo4UblYkAZAADwMQpuBB9f+wMI\n",
       "Cz8M0gVQckRKAADwAwZNAr5HpAQAgCDj2zogtOjhBgAAAPJADzcAAADgQxTcAAAAgIsouAEAAAAX\n",
       "UXADCKREIqFEIuF1M1AqLS25p79znOQ2AAgwCm4AgRKPxxWLLVV19WRVV09WLLZUvb29XjcLE9XU\n",
       "NHrO6fSUeE1NXrUKAEqCWUoABEY8HldDw6Xq72+TtCJ163rV1LSop2eL6urqvGweJipzzmmJ+acB\n",
       "+E6xs5RQcAMoWjrSEYlEyvJ4sdhSOc4SSSuztqxTLLZJnZ0by9IOuMhxpFgseb2ri2IbgK8wLSCA\n",
       "svEi1pFIJNTT87AO92xnWqHu7s1kugEAvkTBDaAg6ViH4yzR0FCfhob65DhLVF+/QPF43OvmIcjS\n",
       "kZKuruQlO9MNAAFFwQ2gIKtWrU5lqFdKmpK6rFR/f5uam1tde9xIJKKGhoWS1ufYul6NjYvKFm2B\n",
       "CzLz29Ho4SXNKboBhAAZbiBgyp2bzn7s6urJGhrqU7LQzrRfVVXTNTg44Frbent7VV+/IOegyUcf\n",
       "3ara2lpXHhdl0NKSnI0kO7PtOFJHh9TW5kWrAGAEMtxAyDEdnlRbW6ueni2KxTapqmq6qqqmKxbb\n",
       "5J9i2w9zSfuhDcVoa8s9QDIapdgGEHgU3EAA+CU37YdYR11dnTo7N2pwcECDgwPq7Nzoj2Jb8sdc\n",
       "0n5oAwBgBCIlQAD4aTo8Yh3j8MNc0n5oAwCEEPNwh+A4gFy8zk3nEo/H1dzcqu7uzZKkxsZFam9f\n",
       "Q7Gd5oe5pP3QBgAImWIL7kluNAZAuKVjHV4O4AQmjIGaAMqEDDfgc37ITY8lEolQbGfzw1zSfmhD\n",
       "EJB3B1AmREqAACA3HRDZc0mPdVvY2xAk5N0BFIBpAYEQ8/10eEjq6BhdrKUXcOnoqJw2BEn6uYnF\n",
       "kheKbQAuoIcbCBhy00CJMcAUQJ7o4QYqBLlpoITKnXcP6sJEACaEghsAUJmys+3peImbRTcDNYGK\n",
       "RKQEAFCZvJoWkIGaQGCx8E0IjgMAUCHIjQOBRIYb4eLXnKNf21VGiURieOBmJai04wUAlB4FN/zJ\n",
       "rzlHv7arDOLxuGKxpaqunqzq6smKxZaqt7fX62a5ptKOF2XEwkRAxSFSAv/ya87Rr+2agPGmGozH\n",
       "42pouDTnwjs9PVtUV1dXnoaWSaUd70QxVWUBWJgICDQiJQgfvy5I4dd2FSHfXtxVq1anis+Vkqak\n",
       "LivV39+m5ubWkrTFT9GNchxvGPAtQBFYmAioSBTclYoscsVL9+I6zhINDfVpaKhPjrNE9fULFI/H\n",
       "h/dLJBLq6XlYh3t6M61Qd/fmCRXKfiva3D7esMj3/EGWtrbcf6BHo+7NigLAcxTclSoIWWS/5hz9\n",
       "2q4C+aEXl6ItuPxw/gBAYFhrA39JHgYK1tVl7axZyX8zr/tBrvb4oY1+bVeBDh06ZKuqJlmp30o2\n",
       "69Jvq6om2UOHDg3vH41eZqXbc+x7u43FlhbdDrfud6L82i6/KPT8KUd7yvl4ACpXquYsuFalh7uS\n",
       "+TmL7Neco1/b5bK1a29RTU2LpHWS9qcu61RT06L29jVF3aefoxtuHC9Kz29xJAAYCwU3/MmvOUe/\n",
       "tqtAkUhEDQ0LJa3PsXW9GhsXjZhxora2Vj09WxSLbVJV1XRVVU1XLLZJjz66VbW1tWVrd7lU2vEW\n",
       "qtDzxw3EkXyK8UFAbsV0i/vtIiIlxfFzpASui8fjtqZmVio60Z+63G5rambZeDw+5u8dOHDAHjhw\n",
       "oCRtCEJ0g7hCbsWeP6UShHOnIoUkdgeMRUVGSjwvlktxoeAuAm+KsNZu377dxmJLbVXVJFtVNcnG\n",
       "YkvHLJa2b99uo9HLhveNRi+bcGHlddGGiSnk/Cklv2XIkYXOHIRYsQU3C99UqpaW5Gwk2fEIx0lm\n",
       "kQMUj8DEebnwTTweV3Nzq7q7N0uSGhsXqb19DdGNACn3wjeJRELV1ZM1NNSn5Owomfarqmq6BgcH\n",
       "WIjHS46THBskJWdz8sv4IGCCil34hoIbFYPV8IoXiy2V4yxRcgq4TOsUi21SZ+fGCT8Grw8KUY5z\n",
       "0peC0llCwY2QYqVJYAzMZDAx5ZpNJBKJUGwjbxU7kwxrKACBRMGNUGMmAyCcKnYmmfQ0pOkiNl3c\n",
       "+mVa1+z2ZLcXqFBEShBqFfu1c4nxPMLPKjKO5NfIRlAiL0CRyHCH4DhQWgysKp3e3l7V1y/IOWgy\n",
       "9D2KlY4Cyp/8WnADIUeGG4BrKvbrewQjM1xpyEgDgUMPN0KNKETpVeTX95UuM5cr+SszXGlyZbb9\n",
       "luMGQoxISQiOI6y8LNCIQgAlQoTBH4j4AJ4iUgLf8cN0fEQhAHiupSV35MNxktsK0daW+4+daJRi\n",
       "G/AxerjhCjdXJiwWUQigSONESvi/NQ5iIEBoECkJwXGECdlpICSOUCw+93/+j274wYOphZGkhoaF\n",
       "Wrv2Fr49yoUcPBAKFNwhOI6wYDo+ID+B6BkeIzP83J136v6PfkKfObhWfvkWy/fIwQOBR4YbAALC\n",
       "D+Mb8jZGZviGHzyYKrZXKvmH9RRJK9Xf36bm5tayNhEA/I6CGyUXiUTU0LBQ0vocW9ersXGRv3v0\n",
       "ABelxzc4zhINDfVpaKhPjrNE9fULFI/HvW5eXhKJRCpGsiLH1hXq7t483HuPFObOBioakRK4gun4\n",
       "UKhAxCtKIAzjG4iNFYhBk0BoECmBrzAdH/IVqHjFBIWlZ5hvsQrU0TG6sI5Gk7d1dHjVKgBlRA83\n",
       "XFcpPZc4LN/X3I/TR7opTD3DfIsFoBLRww3fikQigSggMHGF9lavWrU6VbBVxsC7QnuGE4mEb3u8\n",
       "+RYLAPJHD3fYsQwwyqTQ3uow9fYWIp+e4Xg8rlWrVgdmfmu+xQJQKejhRm5NTaNHw6cH6zQ1edUq\n",
       "hFCl9VYXa7ye4SDOYsK3WABwZPRwVwJWOIPLiu2tDsOMHRORq2e4lM8JPc8AUFqsNBmC43AVK5zB\n",
       "RcUW3Ay8G6lUMZugRVIAICiIlADwTLHTxDHwrvSCGEkBgLCjh7sSEClBGUy0t5r4Q9JEIyWVHtMB\n",
       "ADcRKQnBcbiCFc48U4kFZDweV3Nzq7q7N0uSGhsXqb19Db3VBZjIHy6VOvMLAJRLICMlxpiFxphn\n",
       "jTG/NcbcnGN71BjTZ4zpTV2+4EU7A40VzsquklZOzFZXV6fOzo0aHBzQ4OCAOjs3UmwXiJgNPNfS\n",
       "MnJmqzTHSW4DUDDPeriNMRFJv5E0X9IeSb+QdI219pmMfaKSbrLWXj7OfdHD7WcVNBd4pa2cCHcV\n",
       "8y0JkRL3hf7bK74ZBcYUxB7u8yXtsNa+YK09KOkeSX+TY7+CDwo+U0FzgTMXNUqpmPmt1669RTU1\n",
       "LZLWSdroVNgwAAAgAElEQVSfuqxTTU2L2tvXuNDKylEx316lvwVNv29TbAMT5mUP91WSLrXWXp/6\n",
       "+VpJF1hrb8zYp1HSjyTtVrIXvNla++sc90UPt99VwMBN8rPwC7L0pVeR314xnSwwSrE93JPcaEye\n",
       "8qmQ45LmWGv3G2MWSXpA0tm5dmxtbR2+Ho1GFeWNwV/SPSa8eZdU6L/aRlHSWXrOj9IZ+e1V2kr1\n",
       "90vNza1EdYCQchxHTq4xDQXysof7Qkmt1tqFqZ8/K2nIWvuVI/zO7yW921r7p6zb6eEOAhd7S/xS\n",
       "WJQrP8vCJkD5VOS3VxXwrSRQjCBmuJ+Q9BZjzBnGmGpJH5Q0ohoxxpxkjDGp6+cr+QfCn0bfFXwv\n",
       "/ebd1ZW8ZGe6i+S3TGU58rMsbALAVdmZ7exMN4CCeVZwW2sPSfqopC2Sfi3pXmvtM8aYG4wxN6R2\n",
       "u0rS08aYJyV9Q9LV3rQWE+LSm7cfC89yTOnGwEygvIpdSTWwmE4WKDkWvoH7XJoW0O/Tn7kRc6nI\n",
       "r7YBH5joSqoAwoGVJkNwHMhfpRaelXrcgB+EZfYXv4x5AYIoiBluAAWquK+2AR8J+kqqfhvzAlQS\n",
       "Cm4EUiUXnixsAnirmAWJvObHMS9AJaHgRmBVauFZjoGZAMKFwdaAt8hwI9DCkqksFllMAONh7AdQ\n",
       "OgyaDMFxoHgUngCQGwU3UDoMmkRFC2KmEgDKoZLHvAB+QQ83AAAhxzziQGnQww0AY0gkEsOxI6AS\n",
       "Mdga8BY93ABCKx6Pa9Wq1erpeViS1NCwUGvX3kKBgYrGmBegePRwA0AG5h0GcmPMC1B+efVwG2PO\n",
       "kPQX1tqfGmOmSJpkrX3D5bbljR5uANlisaVynCVKzjucaZ1isU3q7NzoRbMAAAHm2rSAxpiPSLpe\n",
       "0vHW2nnGmLMl3W6tbSquqaVHwQ0gE9OgAQDc4Gak5H9Leo+kNyTJWvucpBMLfSAAAACgEuVTcB+w\n",
       "1h5I/2CMmSSJ7mQAvsW8wwAAP8mn4O42xnxe0hRjzHslbZD0oLvNAoCJWbv2FtXUtEhaJ2l/6rJO\n",
       "NTUtam9f423jAJRGS4vkOKNvd5zkNsAn8im4b5b0B0lPS7pB0iZJX3CzUQC8Eab5qpl3GKgATU3S\n",
       "smUji27HSd7W5JuhZsCRB02m4iP/Za19W/maVDgGTQITE/b5qpl3GAixdIG9YUPy5/T1aNTLViGk\n",
       "3Jyl5MeSPmat3Vls49xGwQ0ULz1fda4ln3t6tqiurs7L5gHA+BxHisWS17u6KLbhmmIL7kl57HO8\n",
       "pF8ZY7ZJ6k/dZq21lxf6YAD8Z9Wq1aliO3O+6pXq75eam1uZrxoAgAnKp4c7mrqa3tEoWXB3u9iu\n",
       "gtDD7Q98bR88zFcNIPCIlKCMXJuH21rrSHpW0rGSpkn6tZ+KbXgvHo8rFluq6urJqq6erFhsqXp7\n",
       "e71uFgAg7DKL7Wg0edmwYfRASsBj4xbcxpgPSHpc0jJJH5C0zRizzO2GIRjS+V/HWaKhoT4NDfXJ\n",
       "cZaovn6B4vG4183DOJivGkCgdXSM7s1OF90dHV61Chgln0jJLyXNt9a+lvr5BEkd1tpzy9C+vBAp\n",
       "8U4stlSOs0Qj87+StE6x2CbyvwHQ29ur+voFOQdNMoUeAACHuTlLydOSzk1XtMaYKklPWWvfUVRL\n",
       "XUDB7Q3yv/nze749Ho+rublV3d2bJUmNjYvU3r6GYhsAgAyuZbglPSxpizHmb40xf6fkwjebC32g\n",
       "isdqWBUpKPn2uro6dXZu1ODggAYHB9TZuZFiGwCAEsln0OSnJN0h6VxJ75B0h7X20243LHRCuBoW\n",
       "+d8jC2K+PRKJVPRrBgCAG/KJlJwp6RVr7Zupn4+RdJK19gX3m5efwERKQjh1EfnfsZFvBwAgXNzM\n",
       "cG+XdJG1djD189GSfm6tPa+olrogMAW3VLrVsFpakj3j2b/vOMmR2W1txbexQOR/RyPfDgBA+Li5\n",
       "0mQkXWxLkrX2gDHmqEIfCCWWjqhk9pBn96CXSTr/6/eBgQAAAF7IZ9Dk68aYv0n/kLr+untNCrF0\n",
       "QdzVlbxMZGL+7Mn9syf/9wD538PItwMAgLR8IiV/Ien7kk5J3bRb0nJr7Q6X25a3QERKchXEpSiS\n",
       "SxVRQcmRbwcqU0V/2+ejuCPgBjeXdt9hrb1A0jmS/tJae5Gfiu3AYDWsilNbW6ueni2KxTapqmq6\n",
       "qqqmKxbbRLENhFRQpgF1VQhn5AJKYcwebmPM5ZJ+mZ6NxBizWtKVkl6Q9HFr7e/L1MZxBaKH2w0h\n",
       "nPUkrCq6xwuoAOlpQHN9o9XTs0V1dXVeNq+8+GxCiJV8lpLUCpMXWGv3G2Muk/R1SVdLqpW0zFp7\n",
       "6UQaXEoVWXC7FVEBABSMaUCzEHdESLkRKRmy1u5PXX+/pO9aa7dba78j6cRiGokSIqICAL6QSCTU\n",
       "0/OwDvdsZ1qh7u7Nw99yAahMR+rh/qWkSyT1S/q9pKustb9IbXvGWvuXZWvlOCqyhxsA4AvMu5+F\n",
       "SAlCzI0e7m9I6pW0XdIzGcV2naSXimolAAAhwzSgGbKjjdlT2AIV6ojTAhpjTlMyPvKktXYoddts\n",
       "SUdZa3eVp4njo4cbAOAlpgFNYVpAhJxrS7sHAQU3AMBr8Xhczc2t6u7eLElqbFyk9vY1lVNsAxWA\n",
       "gjsExwH/YBo/AMXi/QMIL9cWvgEqCQtXAJioSCRCsQ1gBApuBFYikSjpVFvphSscZ4mGhvo0NNQn\n",
       "x1mi+voFisfjJXscAABQWcYsuI0x5xpjHjPG7DbG3GmMmZGxbVt5mgeM5lYv9KpVq1MDnlYqObXX\n",
       "FEkr1d/fpubm1gnfPwAAqExHmof755LaJD0u6UOS/peky621O4wxvdZa34wCIcNdOdxaPpl5dAEA\n",
       "wHjcWNr9l9baczN+jkn6tqRrJd1OwQ0vuLV8MgU3AAAYjxsF91OSGqy1fRm3nSvpR5JmWGtnFtvY\n",
       "UqPgrgxuF8VuFfMAACAc3Jil5KuSzsm8wVr7S0l/rWTRDYTK2rW3qKamRdI6SftTl3WqqWlRe/sa\n",
       "bxsHAAACa8yC21r7fWvt/5/j9l3W2uvdbRYwmtvLJ9fW1qqnZ4tisU2qqpquqqrpisU2VdQqcaWe\n",
       "+QUAALDwDQKmXMsnV9rCFfF4XKtWrVZPz8OSpIaGhVq79paK+UMDAIB8sPANKkK5eqEraeEK5h8H\n",
       "AMBd4/ZwG2PeY639WdZtl1hrf+5qywpAD3dlqrReaLcwWBTwB97TAP9zs4f7Wzlu++dCHwgotUrq\n",
       "hXZLIpFIxUhW5Ni6Qt3dm8l0Ay5zazEvAP4xaawNxpiLJF0s6QRjzE2S0tX8NBFFAQBgwkYu5nWv\n",
       "JMlx1qu+fsGEFvMC4C9HKpyrlSyuI6l/p6Yub0i6yv2mAXCb2zO/ADiyVatWp4rtlUquLzBF0kr1\n",
       "97epubnV07YBKJ18MtynW2t3pq5HJE3NXAzHD8hwA8Ur18wvAEZihVsgeNzMcN9qjDnWGFMj6WlJ\n",
       "vzbGfLrgFgLwJeYfBwDAXfn0cD9lrX2nMeb/k1Qn6TOS4tbad5SjgfmghxsoDWZJAMqLWYKAYCm2\n",
       "h3vMQZOZ+xhjjpJ0haR/sdYeNMZQ3QIhRKENlNfatbekIl1SdqSrvX2rhy0DUEr5RErukPSCkgMm\n",
       "e4wxZ0jyVYYbAIAgItIFVIaCl3Y3xhhJEWvtIXeaVDgiJQCAoCPSBfifa4MmjTEnG2O+a4x5OHXT\n",
       "X0q6rtAHAgAAY2MxLyC88omU/F9JWyWdkvr5t5I+6VaDAAAAAq2lRXKc0bc7TnIbKs6YBbcxJj2g\n",
       "cpa19l5JCUmy1h6U5Js4CQAAgK80NUnLlo0suh0neVtTk1etgoeONEvJNiWnAfxvY8ys9I3GmAvF\n",
       "oEkAAIDcolFpw4Zkgb1hQ/K29PVo1MuWwSNjDpo0xvRaa2uNMe+W9E1JfyXpV5JOkHSVtfap8jXz\n",
       "yBg0CQAAcmppSfYqZxe6jiN1dEhtbe49tuNIsVjyelcXxXYIuDEP9wnGmJskGUn3S9qUun5AUpMk\n",
       "3xTcAAAAOaXjHZm9y+l4R7r3GXDZkQruiKRpOW6f4lJbAAAASsureEe6qO/qKt9jwrfGjZSUuT1F\n",
       "IVICAACOqJzxjswe9Fy96hTdgeXaPNwAAAAoQEfH6MI63dPe0eFVq+ChI/Vwz7TW/rHM7SkKPdwA\n",
       "AGBM2ZlteppRpGJ7uAte2t2PKLgBAEBOxDtQQkRKAAAAshHvgA/Qw12pvJyXNGQSiYQkKRKJeNwS\n",
       "AADgJnq4URiWnZ2weDyuWGypqqsnq7p6smKxpert7XXt8RKJxHBxDwAAgoOCu1JlzkvqOOTZChSP\n",
       "x9XQcKkcZ4mGhvo0NNQnx1mi+voFisfjJX+schb2AACgtIiUVDqWnS1KLLZUjrNE0sqsLesUi21S\n",
       "Z+fGkjxOurDv72+TtCJ163rV1LSop2eL6urqSvI4AABgfMxSEoLj8AQFd8ESiYSqqydraKhPoxde\n",
       "3a+qqukaHBwoSaa7XIU9EDaMrQDgBjLcKFzmsrNdXaMz3S4gh5y/RCKhnp6HdbhnO9MKdXdv5rkE\n",
       "shDBqmAtLbk/wxwnuQ3wEAV3pcrObGdnukssTB+CkUhEDQ0LJa3PsXW9GhsX0asGeKCcYyvgQ0wG\n",
       "AB/ztOA2xiw0xjxrjPmtMebmMfb5Zmr7U8aY2nK3MbTKOC9pGD8E1669RTU1LZLWSdqfuqxTTU2L\n",
       "2tvXlOQxKOyBwqxatTo13mGlknGvKZJWqr+/Tc3NrZ62DWXAZADwMc8y3MaYiKTfSJovaY+kX0i6\n",
       "xlr7TMY+iyV91Fq72BhzgaR/stZemOO+yHD7WFhzyPF4XM3Nreru3ixJamxcpPb2NaqtLd3fhb29\n",
       "vaqvX5Bz0OSjj24t6WMBQVbOsRXwOcYmwUVBzHCfL2mHtfYFa+1BSfdI+pusfS6XdJckWWsfl3Sc\n",
       "Meak8jYTExHmHHJdXZ06OzdqcHBAg4MD6uzcWPICuLa2Vj09WxSLbVJV1XRVVU1XLLappMU2uXoA\n",
       "ANzlZcF9qqQXM37enbptvH1Oc7ldQEEikYirvWZuFfZhytUDRLAgyZPJAIB8TPLwsfPNgGR32+f8\n",
       "vdbW1uHr0WhUUb5C8oX0h6DjrNfoSAkfgoUo5fM0cn7veyVJjrNe9fULmN8bgbV27S2pCJaUHcFq\n",
       "b9/qYctQFrky2+lMNzluFMlxHDkl+KPNywz3hZJarbULUz9/VtKQtfYrGfusk+RYa+9J/fyspEZr\n",
       "7atZ90WG28fIIftPWHP1QDnGVsCnWlqSs5FkF9aOk5wMoK3Ni1YhZAK38I0xZpKSgyabJL0kaZuO\n",
       "PGjyQknfYNBkMPEh6B8MLkMlCNTCNxSKQGAUW3B7Fimx1h4yxnxU0hZJEUnftdY+Y4y5IbX9Dmvt\n",
       "JmPMYmPMDkn9kv7Oq/ZiYtI55EB9CAIIrEC9x6Tnj86MPWTGIwAEHku7AxWISAngM9kFNrljwJcC\n",
       "FykpJQpuoDDk6gEfYv5owPeCOA83AI+UY35vAACQRA83EBQuDawiVw/4AJESIBDo4YZ/tLTkXmjA\n",
       "cZLbUJz0wKrM5zb9Id3UVPTdur1wD4BxZM8fHY0enj+aRVuAUKDgRum5VBhWvOwP4VyLPAAIno6O\n",
       "0f+P0//fOzq8ahWAEiJSAnfw9ah7GFgFAIAniJRgJK9jHenemVgseaHYBgBk8vpzCigjCu6wItYR\n",
       "TunXsKsreSHjCSCo+JxCBSFSEmZexjqIlJRersw2OW4AQcZnBQKGhW9CcByu8CLvS2HoDpemBQQA\n",
       "TzEuBQFSbME9yY3GoMKNN+KeN9PijFVQp6cRAwAAvkQPd5jxVR0AwM/4nELAMEsJRmIhBQCAn/E5\n",
       "hQpCwR1WLKQAAPAzPqdQQYiUAAAAAHkgUgKMh0UWAARIIpFQIpHwuhkASoCCG5WDRRYABEA8Hlcs\n",
       "tlTV1ZNVXT1ZsdhS9fb2et0sABNApASVhRHxAHwsHo+roeFS9fe3SVqRunW9ampa1NOzRXV1dV42\n",
       "D6h4LHwTguNAmbDIAgCfisWWynGWSFqZtWWdYrFN6uzc6EWzAKSQ4QYAl5GphZsSiYR6eh7W4Z7t\n",
       "TCvU3b2Z8w8IKApuVJZ0pKSrK3lhvlfkgUwtAGAiKLhROVhkAUVIZ2odZ4mGhvo0NNQnx1mi+voF\n",
       "isfjXjcPIRKJRNTQsFDS+hxb16uxcZEikUi5mwWgBMhwo3K0tCRnI8nObDtOcpGFtjYvWgWfI1OL\n",
       "curt7VV9/YKcgyYffXSramtrvWweUPEYNBmC4wCQWzq3Wu7evUQioerqyRoa6pM0JWvrflVVTdfg\n",
       "4AC9jgHi1blUiHg8rubmVnV3b5YkNTYuUnv7GleK7SA8H4CfMGgSQOiQnUapBOlcqqurU2fnRg0O\n",
       "DmhwcECdnRtLXmwH6fkAwoCCGwiASpwdww/ZaTK14eCHc6kYkUjElfMrqM8HEGRESgAfi8fjWrVq\n",
       "dWqqMKmhYaHWrr2lInKcfslOk6kNPr+cS37B8wEUjwx3CI4DyFTJK875LTtdzkwtSstv55LXeD6A\n",
       "iaHgDsFxAJnc6IUKygApvxYFQXn+cJhfzyWvBO75YHYp+AyDJlF6LS2556d2nOQ2uKbUK84FbYCU\n",
       "X7PTbmVq4R6/nkteCdzz0dQ0eq2E9JoKTU1etQoonLU28JfkYaDkurqsnTUr+e+RbkPJHTp0yFZV\n",
       "TbJSv5Vs1qXfVlVNsocOHcrrvrZv325ramZZ6fbU/fVb6XZbUzPLbt++3eUjKV48Hh+z3fF43Ovm\n",
       "IUA4l0YK3POR+bnDZxA8lqo5C69Vi/klv10ouF3EG51notHLUh+I2QX37TYWW1r2+/HC9u3bbSy2\n",
       "1FZVTbJVVZNsLLbUnwVByB06dCjvP/D8inNppMA9H11dh9+8+AyCh4otuMlwY3yOI8ViyetdXaOz\n",
       "dHBFKWbHCFxecwxkp70RxllyOJdGCszzwecQfIIMNxAytbW16unZolhsk6qqpquqarpisU0VORUd\n",
       "2enyC+tczZxLIwXi+Uhntru6kpfsTDcQAPRw48jSb3QbNiR/Tl+nd6GsJtILxZy7KAbnjbcC0/Ps\n",
       "tszPoPTnTq7bgDKhhxull/2mFo0mr9O7UHYT6YVau/YW1dS0SFonaX/qsk41NS1qb19TwlYiLEo9\n",
       "Sw7yF7QZhVzX0TG6sE5/FnV0eNUqoGD0cGNszH8aGizcgkKEJfsfNJW82BUQFCx8E4LjKBqFsTcC\n",
       "+LzzNTXyRaSk/HjOAf8jUlLJWBjAGwF83gMxQAq+QBSpvIjxAOFGD3dYMLjRdTl7h3nec6InPRyI\n",
       "IpUPMR4gGOjhrnTpQSSxWPJC0VcyRxzExPM+AgO+wqWurk6dnRs1ODigwcEBdXZupNh2SeCWXAdQ\n",
       "EApu4AjCOhexG3iuwosoUnkQ4wHCi4I7LFgYwBWrVq1OzRiwUsmveadIWqn+/jY1N7fyvGcY97kC\n",
       "cEQsdgWEFxnuMGBhAFeMl6mMmWnqmHm8DM87+VOgxBgHAfgTGe5KxsIAnvhrazV0zz087wBKjhgP\n",
       "EC70cCN8Sjg/NvPi5o/nCgAQdvRwA2klnB+bQUz547kCACA3Cm6ETzrWkS66J5CrZhBT/niuAADI\n",
       "jUgJwstxknNjS8kZRCY4iJFBTPnjuQIAhFGxkZJJbjQGCCOKx/zxXAEAcBiREoQT82MDAACfoOBG\n",
       "+GRntrMz3QAAAGVEwY3wYV5yAADgIwyaBAAAAPLAPNwAAACAD1FwAwAAAC6i4AYAAABcRMENoHgt\n",
       "LblnfnGc5DYAyMR7BioUBTeA4jU1jZ5uMT0tY1OTV60C4Fe8Z6BCMUsJgInJnPdcGjkHOgBk4z0D\n",
       "AVbsLCUU3AAmznGkWCx5vauLD04AR8Z7BgKKaQEBBEIikVAikfC6GQAAlA0FN4CJSX893NWVvGTn\n",
       "M1Pi8bhisaWqrp6s6urJisWWqre3t+zNzUTxD3ggz/cMIEwouAEULzOLGY0mLxs2jPoAjcfjami4\n",
       "VI6zRENDfRoa6pPjLFF9/QLF4/GyN9uPxT9QEfJ8zwDChoIbQPE6OkYPdkp/gHZ0DN+0atVq9fe3\n",
       "SVopaUrqslL9/W1qbm4tZ4t9V/y7jV58+Eqe7xlA2DBoEshXS0ty2qrswT2Ok/ygaGvzolW+l0gk\n",
       "VF09WUNDfUoW2pn2q6pqugYHBxSJRMrSnlhsqRxniZLFf6Z1isU2qbNzY1na4bZ4PK5Vq1arp+dh\n",
       "SVJDw0KtXXuLamtrPW4ZAAQXgybhT2Fa5ID5YwMvkUikCtAVObauUHf35lD0BldaL/4IYXrPARAa\n",
       "FNxwV5iK1OysYXYWETlFIhE1NCyUtD7H1vVqbFxUtt7tSuGnCE/Zhek9B0BoECmB+8K2yAHzxxas\n",
       "t7dX9fULUkVgund5vWpqWvToo1vLGnMIe6TEbxEeT4TtPQeAbxApgX+le4ZjseSFD76KU1tbq56e\n",
       "LYrFNqmqarqqqqYrFttU9mJbktauvUU1NS2S1knan7qsU01Ni9rb15S1LXAJ7zkAfIaCGygE88cW\n",
       "ra6uTp2dGzU4OKDBwQF1dm70ZACfn4p/NxDhAQD/IVIC94Xl691cmW1y3KMFaDaX9ADJsBWgforw\n",
       "eCIs7zkAfIdICfwpTIscMH9sfgI0aC0SiYSu2JbC34t/RGF6zwEQGvRww10B6u1ECdHD6Bth7cUf\n",
       "E+85AFxUbA83BTcAdzCbCwAgZIiUAAAAAD5EwQ2g9JjNBQCAYRTcAEqLQWsAAIxAwQ2gtJjNBQCA\n",
       "ERg0CQAAAOSh2EGTk9xozHiMMcdLulfS6ZJekPQBa+2fc+z3gqQ3JCUkHbTWnl/GZgIAAAAT5lWk\n",
       "5DOSHrHWni2pI/VzLlZS1FpbS7ENAACAIPKq4L5c0l2p63dJuuII+xbcbQ8AfpdIJIYXpQEAhJtX\n",
       "BfdJ1tpXU9dflXTSGPtZST81xjxhjLm+PE0DAPfE43HFYktVXT1Z1dWTFYstVW9vr9fNwhj4wwhA\n",
       "KbhWcBtjHjHGPJ3jcnnmfqnRjmONeLzEWlsraZGk/22MqXervQDgtng8roaGS+U4SzQ01KehoT45\n",
       "zhLV1y9QPB73unnIwB9GE9TSknsaUMdJbgMqjGuDJq217x1rmzHmVWPMydbaV4wxsyW9NsZ9vJz6\n",
       "9w/GmPslnS/p0Vz7tra2Dl+PRqOKsow0AJ9ZtWq1+vvbJK3MuHWl+vul5uZWdXZu9KppyJD+wyj5\n",
       "Wt0rSXKc9aqvX6Ceni2qq6vztoFB0NQ0cj5+aeQc/UBAOI4jpwRrSHgyLaAx5quS/mit/Yox5jOS\n",
       "jrPWfiZrnymSItbafcaYGklbJa2x1m7NcX9MCwjA1xKJhKqrJ2toqE/SlKyt+1VVNV2DgwOKRCJe\n",
       "NC+U0lGQQp/TWGypHGeJRv5hJEnrFItt8s0fRsUeX9lkF9jZBTgQQMVOC+hVhvvLkt5rjHlO0l+n\n",
       "fpYx5hRjzEOpfU6W9Kgx5klJj0v6Sa5iGwCATBOJgyQSCfX0PCxpRY6tK9TdvdnzTHdg4i7pBa9i\n",
       "seSFYhsVzJOC21r7J2vtfGvt2dbaBek5uK21L1lrl6SuP2+tfVfq8lfW2lu9aCsAlEIkElFDw0JJ\n",
       "63NsXa/GxkX+7akMkLDn5MN+fEBYsbQ7AJTJ2rW3qKamRdI6SftTl3WqqWlRe/sabxsXEiNz8lNS\n",
       "l5Xq729Tc3PruL/v9z+MJnp8ZZWOlHR1JS/LluUeSAlUAJZ2B4Ayisfjam5uVXf3ZklSY+Mitbev\n",
       "UW1trcctC75S5eR7e3tVX78gVdimoyXrVVPTokcf3erZaxWocQCZ+e1cgyaJliCggpbhBoCKVFdX\n",
       "p87OjRocHNDg4IA6OzdSbMtf813X1taqp2eLYrFNqqqarqqq6YrFNnlabPtGvtP9dXSMLqzTme6O\n",
       "DnfbCPgQBTcAeCASifijJ9JjpRwAWMo4iB//MPJF3CU93V9m0Z3uuW5qOnxbW1vuXuxoNLkNqDBE\n",
       "SgAAnhg53/XI6Eax8137NQ5SKr44Pqb7QwUjUgIACBQ3BgCGPQ7ii+Njuj+gYPRwAwiOlpbk19bZ\n",
       "H+6Ok8yFBvGr6jAekzTucSVaW10fAOj7hWEmyNPjc5xksS0lZyCh4EaFoIcbQPjlmx8NkjAek+SL\n",
       "4wp7Tt6z42O6P6Bg9HADCJYw5kfDeEzSuMflqyXUw/pNQ6kx3R8qXLE93BTcAIInjF9nF3lMvo9N\n",
       "HOG4fDEAMLOdFJLj4w8TVDgiJQBQQUo5nZ5XfDEAMC09EDAdj6DYzo3p/oCi0MMNIFjCGL8o8Jjc\n",
       "mE7PFQUcl2966sP47QmAkqGHG0D4Zfc6ZvdKBlERx+TGdHolV+BxhX2AI4DKRsENIDjCuFx0gceU\n",
       "SCTU0/OwDvdsZ1qh7u7N/lgiPYivFbNvAHAJkRIACJBEIuH6/NUViUGTAPJApAQAKkAkElFDw0JJ\n",
       "63NsXa/GxkUU28UIYo88gMCghxsIO6bxCh1fTacHABWEHm4AuflgxT+Ulq+m0wMAjIsebqAShHEq\n",
       "PUjy0XR6AFABWGkyBMcBuIr5hQEAmBAiJQAAAIAPUXADlYD5hQEA8AwFNxB2YVydEQCAAKHgBsKO\n",
       "+YUBAPAUgyYBAACAPDBoEgAAAPAhCm4AAADARRTcAAAAgIsouAEAAAAXUXADAAAALqLgBgAAAFxE\n",
       "wQ0AAAC4iIIbAAAAcBEFNwAAAOAiCm4AAEqtpUVynNG3O05yG4CKQsENAECpNTVJy5aNLLodJ3lb\n",
       "U5NXrQLgEWOt9boNE2aMsWE4DgBAiKQL7A0bkj+nr0ejXrYKwAQYY2StNQX/XhgKVQpuAJ5oaUn2\n",
       "VmYXUI4jdXRIbW1etAp+4jhSLJa83tVFsQ0EXLEFN5ESACgWsQEAQB7o4QaAiSA2gLFwbgChQ6Qk\n",
       "BMcBIKDCFhsgKjNxmcV2+nnMdRuAQCFSAgAoDaIyE9fRMbqwjkaTt3V0eNUqAB6hhxsAJiKssYGw\n",
       "Hi+K4dUAAAjmSURBVBcATACRkhAcB4CACXtsIGxRGQCYICIlAFBuxAYAAHmghxsAMBqREgAYhR5u\n",
       "AEBpZMdi0r322QMpAQB5oeAGAIxEVAYASopICQAAAJAHIiUAAACAD1FwAwAAAC6i4AYAAABcRMEN\n",
       "AAAAuIiCGwAAAHARBTcAAADgIgpuAAAAwEUU3AAAAICLKLgBAAAAF1FwAwAAAC6i4AYAAABcRMEN\n",
       "AAAAuIiCGwAAAHARBTcAAADgIgpuAAAAwEUU3AAAAICLKLgBAAAAF1FwAwAAAC6i4AYAAABcRMEN\n",
       "AAAAuIiCGwAAAHARBTcAAADgIgpuAAAAwEUU3AAAAICLKLgBAAAAF1FwAwAAAC6i4AYAAABcRMEN\n",
       "AAAAuIiCGwAAAHARBTcAAADgIgpuAAAAwEUU3AAAAICLPCm4jTHLjDG/MsYkjDF1R9hvoTHmWWPM\n",
       "b40xN5ezjQg+x3G8bgJ8iPMC2TgnkAvnBUrJqx7upyW9T1LPWDsYYyKS/lnSQknnSLrGGPOX5Wke\n",
       "woA3S+TCeYFsnBPIhfMCpTTJiwe11j4rScaYI+12vqQd1toXUvveI+lvJD3jdvsAAACAUvFzhvtU\n",
       "SS9m/Lw7dRsAAAAQGMZa684dG/OIpJNzbPqctfbB1D5dklZZa+M5fv9KSQuttdenfr5W0gXW2htz\n",
       "7OvOQQAAAAAZrLVHjGjk4lqkxFr73gnexR5JczJ+nqNkL3euxyr4wAEAAIBy8EOkZKxi+QlJbzHG\n",
       "nGGMqZb0QUkby9csAAAAYOK8mhbwfcaYFyVdKOkhY8zm1O2nGGMekiRr7SFJH5W0RdKvJd1rrWXA\n",
       "JAAAAALFtQw3AAAAAH9ESgrCojnIxRhzvDHmEWPMc8aYrcaY48bY7wVjzC+NMb3GmG3lbifKI5//\n",
       "/8aYb6a2P2WMqS13G1F+450XxpioMaYv9f7Qa4z5ghftRPkYY75njHnVGPP0EfbhvaLCjHdeFPNe\n",
       "EbiCWyyag9w+I+kRa+3ZkjpSP+diJUWttbXW2vPL1jqUTT7//40xiyX9hbX2LZI+Iun2sjcUZVXA\n",
       "50J36v2h1lr7xbI2El74NyXPiZx4r6hYRzwvUgp6rwhcwW2tfdZa+9w4uw0vmmOtPSgpvWgOwuty\n",
       "SXelrt8l6Yoj7MusNuGWz///4fPFWvu4pOOMMSeVt5kos3w/F3h/qCDW2kcl7T3CLrxXVKA8zgup\n",
       "wPeKwBXceWLRnMpzkrX21dT1VyWN9YZoJf3UGPOEMeb68jQNZZbP//9c+5zmcrvgrXzOCyvp4lR0\n",
       "YJMx5pyytQ5+xXsFcin4vcKTpd3Hk8+iOeNgJGgIHeG8+HzmD9Zae4TFkC6x1r5sjDlB0iPGmGdT\n",
       "f8kiPPL9/5/dO8H7Rrjl8/rGJc2x1u43xiyS9ICks91tFgKA9wpkK/i9wpcFdzkXzUFwHOm8SA1u\n",
       "ONla+4oxZrak18a4j5dT//7BGHO/kl8zU3CHSz7//7P3OS11G8Jr3PPCWrsv4/pmY8y/GmOOt9b+\n",
       "qUxthP/wXoFRinmvCHqkhEVzkLZR0nWp69cp+dfmCMaYKcaYaanrNZIWKDkIF+GSz///jZJWSJIx\n",
       "5kJJf86IJCGcxj0vjDEnGWNM6vr5Sk6dS7Fd2XivwCjFvFf4sof7SIwx75P0TUmzlFw0p9dau8gY\n",
       "c4qkb1trl1hrDxlj0ovmRCR9l0VzQu/Lku4zxnxI0guSPiAlF1NS6rxQMo7yo9T/kUmSvm+t3epN\n",
       "c+GWsf7/G2NuSG2/w1q7yRiz2BizQ1K/pL/zsMkog3zOC0lXSfp7Y8whSfslXe1Zg1EWxpgfSGqU\n",
       "NCu1IN/q/9fe/YRYVcZhHP8+mUlRugilJMpAoiyLSYz+0R+CCKIsEly1SagIKyKiqEUQ1EZSclUQ\n",
       "tIhcRSm1qKbAshDUTC03bRpBiqaE0gzL8tfivoNTzDioc+iOfj9wufee95z3PfdyOTznx3vuAaaD\n",
       "x4pT2US/C47jWOGNbyRJkqQOTfUpJZIkSVJfM3BLkiRJHTJwS5IkSR0ycEuSJEkdMnBLkiRJHTJw\n",
       "S5IkSR0ycEvS/yzJuUm+ao8fkuxpr7clmfB+CUluTnLdOG2XJtmU5GCSJ4/SxwNJdibZkeTrJHef\n",
       "yGeSJB0x5W58I0knm6raCwwAJHke2F9Vq46hi1uB/cCmMdr2Ao8C94y3cZILgGeBgaran+QsYM4x\n",
       "jD9Wn6dX1V8n0ocknSyscEtS/0mSRUk2JNma5IMk57WGx5LsapXotUkuAh4CnmhV8RtHd1RVP1XV\n",
       "VuDQUcabQy+wH2jb/F5VQ228+Uk+TrI9yZdJLm7LV7ZK+M4kI3d2vSXJxiTrgW+SnNbW29z298HJ\n",
       "/ZokaWqwwi1J/SfAGmBJVf2cZBnwIrAceBqYV1WHksysqn1JXuXYq+KjbQd+BL5L8gnwTlW939re\n",
       "Al6qqvVJzgCmJbkPuAq4EpgNbEnyWVt/ALi8qna3gP1LVV2TZAbweZKPRsK8JJ0qDNyS1H9mAFcA\n",
       "g0kApgHft7adwNok64B1o7bJ8Q5WVYeBO5IsBm4DVidZBKwC5lbV+rbenwBJbgDWVlUBw0k+BRYD\n",
       "+4DNVbW7dX07sDDJ0vZ+JjAfGDrefZWkqcjALUn9J8Cuqrp+jLY7gZuAu4DnkiycrEGragu9avUg\n",
       "8Abw8gT7+K/N2/OB/yxfUVWDk7SLkjQlOYdbkvrPH8DsJNcCJJmeZEF65e4Lq2oD8AwwCzib3vzr\n",
       "cyboc9wKeJLzk1w9atEAMFRVvwF7kixp681IciawEVjW5mjPpncCsHmMMT4EHhn5p5Ukl7QLMiXp\n",
       "lGKFW5L6z9/AUmBNkln0jtWrgW+BN9uyAK9U1a9J3gPebsF4RVV9MdJRu9hyC73pHIeTPA4saGF6\n",
       "xHRgZZK5wEFgGHi4td0PvJbkBXoXXi6tqnfb3xDuoFfZfqqqhpNcxpFKN8DrwDxgWztZGAbunaTv\n",
       "SJKmjPSm4EmSJEnqglNKJEmSpA4ZuCVJkqQOGbglSZKkDhm4JUmSpA4ZuCVJkqQOGbglSZKkDhm4\n",
       "JUmSpA79A109nxJVGE9DAAAAAElFTkSuQmCC\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0xd120dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "positive = data2[data2['Accepted'].isin([1])]\n",
    "negative = data2[data2['Accepted'].isin([0])]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12,8))\n",
    "ax.scatter(positive['Test 1'], positive['Test 2'], s=50, c='b', marker='o', label='Accepted')\n",
    "ax.scatter(negative['Test 1'], negative['Test 2'], s=50, c='r', marker='x', label='Rejected')\n",
    "ax.legend()\n",
    "ax.set_xlabel('Test 1 Score')\n",
    "ax.set_ylabel('Test 2 Score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data looks a bit more complicated than the previous example.  In particular, you'll notice that there is no linear decision boundary that will perform well on this data.  One way to deal with this using a linear technique like logistic regression is to construct features that are derived from polynomials of the original features.  Let's start by creating a bunch of polynomial features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accepted</th>\n",
       "      <th>Ones</th>\n",
       "      <th>F10</th>\n",
       "      <th>F20</th>\n",
       "      <th>F21</th>\n",
       "      <th>F30</th>\n",
       "      <th>F31</th>\n",
       "      <th>F32</th>\n",
       "      <th>F40</th>\n",
       "      <th>F41</th>\n",
       "      <th>F42</th>\n",
       "      <th>F43</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.051267</td>\n",
       "      <td> 0.002628</td>\n",
       "      <td> 0.035864</td>\n",
       "      <td> 0.000135</td>\n",
       "      <td> 0.001839</td>\n",
       "      <td> 0.025089</td>\n",
       "      <td> 0.000007</td>\n",
       "      <td> 0.000094</td>\n",
       "      <td> 0.001286</td>\n",
       "      <td> 0.017551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td>-0.092742</td>\n",
       "      <td> 0.008601</td>\n",
       "      <td>-0.063523</td>\n",
       "      <td>-0.000798</td>\n",
       "      <td> 0.005891</td>\n",
       "      <td>-0.043509</td>\n",
       "      <td> 0.000074</td>\n",
       "      <td>-0.000546</td>\n",
       "      <td> 0.004035</td>\n",
       "      <td>-0.029801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td>-0.213710</td>\n",
       "      <td> 0.045672</td>\n",
       "      <td>-0.147941</td>\n",
       "      <td>-0.009761</td>\n",
       "      <td> 0.031616</td>\n",
       "      <td>-0.102412</td>\n",
       "      <td> 0.002086</td>\n",
       "      <td>-0.006757</td>\n",
       "      <td> 0.021886</td>\n",
       "      <td>-0.070895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td>-0.375000</td>\n",
       "      <td> 0.140625</td>\n",
       "      <td>-0.188321</td>\n",
       "      <td>-0.052734</td>\n",
       "      <td> 0.070620</td>\n",
       "      <td>-0.094573</td>\n",
       "      <td> 0.019775</td>\n",
       "      <td>-0.026483</td>\n",
       "      <td> 0.035465</td>\n",
       "      <td>-0.047494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td>-0.513250</td>\n",
       "      <td> 0.263426</td>\n",
       "      <td>-0.238990</td>\n",
       "      <td>-0.135203</td>\n",
       "      <td> 0.122661</td>\n",
       "      <td>-0.111283</td>\n",
       "      <td> 0.069393</td>\n",
       "      <td>-0.062956</td>\n",
       "      <td> 0.057116</td>\n",
       "      <td>-0.051818</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Accepted  Ones       F10       F20       F21       F30       F31       F32  \\\n",
       "0         1     1  0.051267  0.002628  0.035864  0.000135  0.001839  0.025089   \n",
       "1         1     1 -0.092742  0.008601 -0.063523 -0.000798  0.005891 -0.043509   \n",
       "2         1     1 -0.213710  0.045672 -0.147941 -0.009761  0.031616 -0.102412   \n",
       "3         1     1 -0.375000  0.140625 -0.188321 -0.052734  0.070620 -0.094573   \n",
       "4         1     1 -0.513250  0.263426 -0.238990 -0.135203  0.122661 -0.111283   \n",
       "\n",
       "        F40       F41       F42       F43  \n",
       "0  0.000007  0.000094  0.001286  0.017551  \n",
       "1  0.000074 -0.000546  0.004035 -0.029801  \n",
       "2  0.002086 -0.006757  0.021886 -0.070895  \n",
       "3  0.019775 -0.026483  0.035465 -0.047494  \n",
       "4  0.069393 -0.062956  0.057116 -0.051818  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "degree = 5\n",
    "x1 = data2['Test 1']\n",
    "x2 = data2['Test 2']\n",
    "\n",
    "data2.insert(3, 'Ones', 1)\n",
    "\n",
    "for i in range(1, degree):\n",
    "    for j in range(0, i):\n",
    "        data2['F' + str(i) + str(j)] = np.power(x1, i-j) * np.power(x2, j)\n",
    "\n",
    "data2.drop('Test 1', axis=1, inplace=True)\n",
    "data2.drop('Test 2', axis=1, inplace=True)\n",
    "\n",
    "data2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to modify the cost and gradient functions from part 1 to include the regularization term.  First the cost function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def costReg(theta, X, y, learningRate):\n",
    "    theta = np.matrix(theta)\n",
    "    X = np.matrix(X)\n",
    "    y = np.matrix(y)\n",
    "    first = np.multiply(-y, np.log(sigmoid(X * theta.T)))\n",
    "    second = np.multiply((1 - y), np.log(1 - sigmoid(X * theta.T)))\n",
    "    reg = (learningRate / 2 * len(X)) * np.sum(np.power(theta[:,1:theta.shape[1]], 2))\n",
    "    return np.sum(first - second) / (len(X)) + reg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the \"reg\" term in the equation.  Also note the addition of a \"learning rate\" parameter.  This is a hyperparameter that controls the effectiveness of the regularization term.  Now we need to add regularization to the gradient function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def gradientReg(theta, X, y, learningRate):\n",
    "    theta = np.matrix(theta)\n",
    "    X = np.matrix(X)\n",
    "    y = np.matrix(y)\n",
    "    \n",
    "    parameters = int(theta.ravel().shape[1])\n",
    "    grad = np.zeros(parameters)\n",
    "    \n",
    "    error = sigmoid(X * theta.T) - y\n",
    "    \n",
    "    for i in range(parameters):\n",
    "        term = np.multiply(error, X[:,i])\n",
    "        \n",
    "        if (i == 0):\n",
    "            grad[i] = np.sum(term) / len(X)\n",
    "        else:\n",
    "            grad[i] = (np.sum(term) / len(X)) + ((learningRate / len(X)) * theta[:,i])\n",
    "    \n",
    "    return grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize variables like we did in part 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set X and y (remember from above that we moved the label to column 0)\n",
    "cols = data2.shape[1]\n",
    "X2 = data2.iloc[:,1:cols]\n",
    "y2 = data2.iloc[:,0:1]\n",
    "\n",
    "# convert to numpy arrays and initalize the parameter array theta\n",
    "X2 = np.array(X2.values)\n",
    "y2 = np.array(y2.values)\n",
    "theta2 = np.zeros(11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialize our learning rate to a sensible value.  We can play with this later if necessary (i.e. if the penalization is too strong or not strong enough)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "learningRate = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try calling our new regularized functions with the default (0) values for theta to make sure the calculations are working."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6931471805599454"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "costReg(theta2, X2, y2, learningRate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.00847458,  0.01878809,  0.05034464,  0.01150133,  0.01835599,\n",
       "        0.00732393,  0.00819244,  0.03934862,  0.00223924,  0.01286005,\n",
       "        0.00309594])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradientReg(theta2, X2, y2, learningRate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use the same optimization function from part 1 to compute the optimal solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([  0.35872309,  -3.22200653,  18.97106363,  -4.25297831,\n",
       "         18.23053189,  20.36386672,   8.94114455, -43.77439015,\n",
       "        -17.93440473, -50.75071857,  -2.84162964]), 110, 1)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result2 = opt.fmin_tnc(func=costReg, x0=theta2, fprime=gradientReg, args=(X2, y2, learningRate))\n",
    "result2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can use the prediction function from part 1 to see how accurate our solution is on the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy = 91%\n"
     ]
    }
   ],
   "source": [
    "theta_min = np.matrix(result2[0])\n",
    "predictions = predict(theta_min, X2)\n",
    "correct = [1 if ((a == 1 and b == 1) or (a == 0 and b == 0)) else 0 for (a, b) in zip(predictions, y2)]\n",
    "accuracy = (sum(map(int, correct)) % len(correct))\n",
    "print 'accuracy = {0}%'.format(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we implemented these algorithms from scratch, it's worth noting that we could also use a high-level python library like scikit-learn to solve this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "model = linear_model.LogisticRegression(penalty='l2', C=1.0)\n",
    "model.fit(X2, y2.ravel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.66101694915254239"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(X2, y2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy is much lower than what we just computed, but keep in mind this result is using the default parameters provided by scikit-learn.  We'd likely need to do some parameter tuning to get the same accuracy that we obtained with our earlier result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all for Exercise 2!  Stay tuned for the next exercise where we'll tackle multi-class image classification."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}