{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 05 - Data Preparation and Advanced Model Evaluation\n",
    "\n",
    "by [Alejandro Correa Bahnsen](http://www.albahnsen.com/) & [Iván Torroledo](http://www.ivantorroledo.com/)\n",
    "\n",
    "version 1.3, June 2018\n",
    "\n",
    "## Part of the class [Applied Deep Learning](https://github.com/albahnsen/AppliedDeepLearningClass)\n",
    "\n",
    "\n",
    "This notebook is licensed under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US). Special thanks goes to [Kevin Markham](https://github.com/justmarkham)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling missing values\n",
    "\n",
    "scikit-learn models expect that all values are **numeric** and **hold meaning**. Thus, missing values are not allowed by scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass  \\\n",
       "PassengerId                     \n",
       "1                   0       3   \n",
       "2                   1       1   \n",
       "3                   1       3   \n",
       "4                   1       1   \n",
       "5                   0       3   \n",
       "\n",
       "                                                          Name     Sex   Age  \\\n",
       "PassengerId                                                                    \n",
       "1                                      Braund, Mr. Owen Harris    male  22.0   \n",
       "2            Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0   \n",
       "3                                       Heikkinen, Miss. Laina  female  26.0   \n",
       "4                 Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0   \n",
       "5                                     Allen, Mr. William Henry    male  35.0   \n",
       "\n",
       "             SibSp  Parch            Ticket     Fare Cabin Embarked  \n",
       "PassengerId                                                          \n",
       "1                1      0         A/5 21171   7.2500   NaN        S  \n",
       "2                1      0          PC 17599  71.2833   C85        C  \n",
       "3                0      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "4                1      0            113803  53.1000  C123        S  \n",
       "5                0      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import zipfile\n",
    "with zipfile.ZipFile('../datasets/titanic.csv.zip', 'r') as z:\n",
    "    f = z.open('titanic.csv')\n",
    "    titanic = pd.read_csv(f, sep=',', index_col=0)\n",
    "titanic.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Survived      0\n",
       "Pclass        0\n",
       "Name          0\n",
       "Sex           0\n",
       "Age         177\n",
       "SibSp         0\n",
       "Parch         0\n",
       "Ticket        0\n",
       "Fare          0\n",
       "Cabin       687\n",
       "Embarked      2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check for missing values\n",
    "titanic.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One possible strategy is to **drop missing values**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(183, 11)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop rows with any missing values\n",
    "titanic.dropna().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(714, 11)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop rows where Age is missing\n",
    "titanic[titanic.Age.notnull()].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes a better strategy is to **impute missing values**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.69911764705882"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mean Age\n",
    "titanic.Age.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# median Age\n",
    "titanic.Age.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Williams, Mr. Charles Eugene</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>244373</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Masselmani, Mrs. Fatima</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2649</td>\n",
       "      <td>7.2250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Emir, Mr. Farred Chehab</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2631</td>\n",
       "      <td>7.2250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>O'Dwyer, Miss. Ellen \"Nellie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330959</td>\n",
       "      <td>7.8792</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Todoroff, Mr. Lalio</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349216</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Spencer, Mrs. William Augustus (Marie Eugenie)</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17569</td>\n",
       "      <td>146.5208</td>\n",
       "      <td>B78</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Glynn, Miss. Mary Agatha</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>335677</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Mamee, Mr. Hanna</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2677</td>\n",
       "      <td>7.2292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Kraeff, Mr. Theodor</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349253</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Rogers, Mr. William John</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>S.C./A.4. 23567</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Lennon, Mr. Denis</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>370371</td>\n",
       "      <td>15.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>O'Driscoll, Miss. Bridget</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14311</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Samaan, Mr. Youssef</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2662</td>\n",
       "      <td>21.6792</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Woolner, Mr. Hugh</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>19947</td>\n",
       "      <td>35.5000</td>\n",
       "      <td>C52</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Stewart, Mr. Albert A</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17605</td>\n",
       "      <td>27.7208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Moubarek, Master. Gerios</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2661</td>\n",
       "      <td>15.2458</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Staneff, Mr. Ivan</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349208</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moutal, Mr. Rahamin Haim</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>374746</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>McDermott, Miss. Brigdet Delia</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330932</td>\n",
       "      <td>7.7875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Slocovski, Mr. Selman Francis</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>SOTON/OQ 392086</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Shorney, Mr. Charles Joseph</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>374910</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Petroff, Mr. Pastcho (\"Pentcho\")</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349215</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Moss, Mr. Albert Johan</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>312991</td>\n",
       "      <td>7.7750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Miss. Bertha</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>371110</td>\n",
       "      <td>24.1500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moore, Mr. Leonard Charles</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>A4. 54510</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>McMahon, Mr. Martin</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370372</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Peter, Miss. Anna</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2668</td>\n",
       "      <td>22.3583</td>\n",
       "      <td>F E69</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Boulos, Mrs. Joseph (Sultana)</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2678</td>\n",
       "      <td>15.2458</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Olsen, Mr. Ole Martin</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Fa 265302</td>\n",
       "      <td>7.3125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>719</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>McEvoy, Mr. Michael</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36568</td>\n",
       "      <td>15.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Mannion, Miss. Margareth</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36866</td>\n",
       "      <td>7.7375</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>733</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Knight, Mr. Robert J</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>239855</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>739</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Ivanoff, Mr. Kanio</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349201</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>740</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Nankoff, Mr. Minko</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349218</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>741</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Hawksford, Mr. Walter James</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16988</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>D45</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>761</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Garfirth, Mr. John</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>358585</td>\n",
       "      <td>14.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Brewe, Dr. Arthur Jackson</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112379</td>\n",
       "      <td>39.6000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>769</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Mr. Daniel J</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>371110</td>\n",
       "      <td>24.1500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>774</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Elias, Mr. Dibo</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2674</td>\n",
       "      <td>7.2250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>777</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Tobin, Mr. Roger</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>383121</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>F38</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>779</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Kilgannon, Mr. Thomas J</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36865</td>\n",
       "      <td>7.7375</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>784</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Mr. Andrew G</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.4500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>791</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Keane, Mr. Andrew \"Andy\"</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12460</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>793</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Sage, Miss. Stella Anna</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>69.5500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>794</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Hoyt, Mr. William Fisher</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17600</td>\n",
       "      <td>30.6958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>816</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Fry, Mr. Richard</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112058</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>B102</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>826</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Flynn, Mr. John</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>368323</td>\n",
       "      <td>6.9500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>827</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Lam, Mr. Len</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1601</td>\n",
       "      <td>56.4958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>829</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>McCormack, Mr. Thomas Joseph</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>367228</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>833</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Saad, Mr. Amin</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2671</td>\n",
       "      <td>7.2292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>838</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Sirota, Mr. Maurice</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>392092</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>840</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Marechal, Mr. Pierre</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11774</td>\n",
       "      <td>29.7000</td>\n",
       "      <td>C47</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>847</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Sage, Mr. Douglas Bullen</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>69.5500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>850</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Goldenberg, Mrs. Samuel L (Edwiga Grabowska)</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>17453</td>\n",
       "      <td>89.1042</td>\n",
       "      <td>C92</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>860</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Razi, Mr. Raihed</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2629</td>\n",
       "      <td>7.2292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Sage, Miss. Dorothy Edith \"Dolly\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>69.5500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>869</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>van Melkebeke, Mr. Philemon</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>345777</td>\n",
       "      <td>9.5000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>879</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Laleff, Mr. Kristo</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>349217</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen \"Carrie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.4500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>177 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass                                            Name  \\\n",
       "PassengerId                                                                     \n",
       "6                   0       3                                Moran, Mr. James   \n",
       "18                  1       2                    Williams, Mr. Charles Eugene   \n",
       "20                  1       3                         Masselmani, Mrs. Fatima   \n",
       "27                  0       3                         Emir, Mr. Farred Chehab   \n",
       "29                  1       3                   O'Dwyer, Miss. Ellen \"Nellie\"   \n",
       "30                  0       3                             Todoroff, Mr. Lalio   \n",
       "32                  1       1  Spencer, Mrs. William Augustus (Marie Eugenie)   \n",
       "33                  1       3                        Glynn, Miss. Mary Agatha   \n",
       "37                  1       3                                Mamee, Mr. Hanna   \n",
       "43                  0       3                             Kraeff, Mr. Theodor   \n",
       "46                  0       3                        Rogers, Mr. William John   \n",
       "47                  0       3                               Lennon, Mr. Denis   \n",
       "48                  1       3                       O'Driscoll, Miss. Bridget   \n",
       "49                  0       3                             Samaan, Mr. Youssef   \n",
       "56                  1       1                               Woolner, Mr. Hugh   \n",
       "65                  0       1                           Stewart, Mr. Albert A   \n",
       "66                  1       3                        Moubarek, Master. Gerios   \n",
       "77                  0       3                               Staneff, Mr. Ivan   \n",
       "78                  0       3                        Moutal, Mr. Rahamin Haim   \n",
       "83                  1       3                  McDermott, Miss. Brigdet Delia   \n",
       "88                  0       3                   Slocovski, Mr. Selman Francis   \n",
       "96                  0       3                     Shorney, Mr. Charles Joseph   \n",
       "102                 0       3                Petroff, Mr. Pastcho (\"Pentcho\")   \n",
       "108                 1       3                          Moss, Mr. Albert Johan   \n",
       "110                 1       3                             Moran, Miss. Bertha   \n",
       "122                 0       3                      Moore, Mr. Leonard Charles   \n",
       "127                 0       3                             McMahon, Mr. Martin   \n",
       "129                 1       3                               Peter, Miss. Anna   \n",
       "141                 0       3                   Boulos, Mrs. Joseph (Sultana)   \n",
       "155                 0       3                           Olsen, Mr. Ole Martin   \n",
       "...               ...     ...                                             ...   \n",
       "719                 0       3                             McEvoy, Mr. Michael   \n",
       "728                 1       3                        Mannion, Miss. Margareth   \n",
       "733                 0       2                            Knight, Mr. Robert J   \n",
       "739                 0       3                              Ivanoff, Mr. Kanio   \n",
       "740                 0       3                              Nankoff, Mr. Minko   \n",
       "741                 1       1                     Hawksford, Mr. Walter James   \n",
       "761                 0       3                              Garfirth, Mr. John   \n",
       "767                 0       1                       Brewe, Dr. Arthur Jackson   \n",
       "769                 0       3                             Moran, Mr. Daniel J   \n",
       "774                 0       3                                 Elias, Mr. Dibo   \n",
       "777                 0       3                                Tobin, Mr. Roger   \n",
       "779                 0       3                         Kilgannon, Mr. Thomas J   \n",
       "784                 0       3                          Johnston, Mr. Andrew G   \n",
       "791                 0       3                        Keane, Mr. Andrew \"Andy\"   \n",
       "793                 0       3                         Sage, Miss. Stella Anna   \n",
       "794                 0       1                        Hoyt, Mr. William Fisher   \n",
       "816                 0       1                                Fry, Mr. Richard   \n",
       "826                 0       3                                 Flynn, Mr. John   \n",
       "827                 0       3                                    Lam, Mr. Len   \n",
       "829                 1       3                    McCormack, Mr. Thomas Joseph   \n",
       "833                 0       3                                  Saad, Mr. Amin   \n",
       "838                 0       3                             Sirota, Mr. Maurice   \n",
       "840                 1       1                            Marechal, Mr. Pierre   \n",
       "847                 0       3                        Sage, Mr. Douglas Bullen   \n",
       "850                 1       1    Goldenberg, Mrs. Samuel L (Edwiga Grabowska)   \n",
       "860                 0       3                                Razi, Mr. Raihed   \n",
       "864                 0       3               Sage, Miss. Dorothy Edith \"Dolly\"   \n",
       "869                 0       3                     van Melkebeke, Mr. Philemon   \n",
       "879                 0       3                              Laleff, Mr. Kristo   \n",
       "889                 0       3        Johnston, Miss. Catherine Helen \"Carrie\"   \n",
       "\n",
       "                Sex  Age  SibSp  Parch           Ticket      Fare  Cabin  \\\n",
       "PassengerId                                                                \n",
       "6              male  NaN      0      0           330877    8.4583    NaN   \n",
       "18             male  NaN      0      0           244373   13.0000    NaN   \n",
       "20           female  NaN      0      0             2649    7.2250    NaN   \n",
       "27             male  NaN      0      0             2631    7.2250    NaN   \n",
       "29           female  NaN      0      0           330959    7.8792    NaN   \n",
       "30             male  NaN      0      0           349216    7.8958    NaN   \n",
       "32           female  NaN      1      0         PC 17569  146.5208    B78   \n",
       "33           female  NaN      0      0           335677    7.7500    NaN   \n",
       "37             male  NaN      0      0             2677    7.2292    NaN   \n",
       "43             male  NaN      0      0           349253    7.8958    NaN   \n",
       "46             male  NaN      0      0  S.C./A.4. 23567    8.0500    NaN   \n",
       "47             male  NaN      1      0           370371   15.5000    NaN   \n",
       "48           female  NaN      0      0            14311    7.7500    NaN   \n",
       "49             male  NaN      2      0             2662   21.6792    NaN   \n",
       "56             male  NaN      0      0            19947   35.5000    C52   \n",
       "65             male  NaN      0      0         PC 17605   27.7208    NaN   \n",
       "66             male  NaN      1      1             2661   15.2458    NaN   \n",
       "77             male  NaN      0      0           349208    7.8958    NaN   \n",
       "78             male  NaN      0      0           374746    8.0500    NaN   \n",
       "83           female  NaN      0      0           330932    7.7875    NaN   \n",
       "88             male  NaN      0      0  SOTON/OQ 392086    8.0500    NaN   \n",
       "96             male  NaN      0      0           374910    8.0500    NaN   \n",
       "102            male  NaN      0      0           349215    7.8958    NaN   \n",
       "108            male  NaN      0      0           312991    7.7750    NaN   \n",
       "110          female  NaN      1      0           371110   24.1500    NaN   \n",
       "122            male  NaN      0      0        A4. 54510    8.0500    NaN   \n",
       "127            male  NaN      0      0           370372    7.7500    NaN   \n",
       "129          female  NaN      1      1             2668   22.3583  F E69   \n",
       "141          female  NaN      0      2             2678   15.2458    NaN   \n",
       "155            male  NaN      0      0        Fa 265302    7.3125    NaN   \n",
       "...             ...  ...    ...    ...              ...       ...    ...   \n",
       "719            male  NaN      0      0            36568   15.5000    NaN   \n",
       "728          female  NaN      0      0            36866    7.7375    NaN   \n",
       "733            male  NaN      0      0           239855    0.0000    NaN   \n",
       "739            male  NaN      0      0           349201    7.8958    NaN   \n",
       "740            male  NaN      0      0           349218    7.8958    NaN   \n",
       "741            male  NaN      0      0            16988   30.0000    D45   \n",
       "761            male  NaN      0      0           358585   14.5000    NaN   \n",
       "767            male  NaN      0      0           112379   39.6000    NaN   \n",
       "769            male  NaN      1      0           371110   24.1500    NaN   \n",
       "774            male  NaN      0      0             2674    7.2250    NaN   \n",
       "777            male  NaN      0      0           383121    7.7500    F38   \n",
       "779            male  NaN      0      0            36865    7.7375    NaN   \n",
       "784            male  NaN      1      2       W./C. 6607   23.4500    NaN   \n",
       "791            male  NaN      0      0            12460    7.7500    NaN   \n",
       "793          female  NaN      8      2         CA. 2343   69.5500    NaN   \n",
       "794            male  NaN      0      0         PC 17600   30.6958    NaN   \n",
       "816            male  NaN      0      0           112058    0.0000   B102   \n",
       "826            male  NaN      0      0           368323    6.9500    NaN   \n",
       "827            male  NaN      0      0             1601   56.4958    NaN   \n",
       "829            male  NaN      0      0           367228    7.7500    NaN   \n",
       "833            male  NaN      0      0             2671    7.2292    NaN   \n",
       "838            male  NaN      0      0           392092    8.0500    NaN   \n",
       "840            male  NaN      0      0            11774   29.7000    C47   \n",
       "847            male  NaN      8      2         CA. 2343   69.5500    NaN   \n",
       "850          female  NaN      1      0            17453   89.1042    C92   \n",
       "860            male  NaN      0      0             2629    7.2292    NaN   \n",
       "864          female  NaN      8      2         CA. 2343   69.5500    NaN   \n",
       "869            male  NaN      0      0           345777    9.5000    NaN   \n",
       "879            male  NaN      0      0           349217    7.8958    NaN   \n",
       "889          female  NaN      1      2       W./C. 6607   23.4500    NaN   \n",
       "\n",
       "            Embarked  \n",
       "PassengerId           \n",
       "6                  Q  \n",
       "18                 S  \n",
       "20                 C  \n",
       "27                 C  \n",
       "29                 Q  \n",
       "30                 S  \n",
       "32                 C  \n",
       "33                 Q  \n",
       "37                 C  \n",
       "43                 C  \n",
       "46                 S  \n",
       "47                 Q  \n",
       "48                 Q  \n",
       "49                 C  \n",
       "56                 S  \n",
       "65                 C  \n",
       "66                 C  \n",
       "77                 S  \n",
       "78                 S  \n",
       "83                 Q  \n",
       "88                 S  \n",
       "96                 S  \n",
       "102                S  \n",
       "108                S  \n",
       "110                Q  \n",
       "122                S  \n",
       "127                Q  \n",
       "129                C  \n",
       "141                C  \n",
       "155                S  \n",
       "...              ...  \n",
       "719                Q  \n",
       "728                Q  \n",
       "733                S  \n",
       "739                S  \n",
       "740                S  \n",
       "741                S  \n",
       "761                S  \n",
       "767                C  \n",
       "769                Q  \n",
       "774                C  \n",
       "777                Q  \n",
       "779                Q  \n",
       "784                S  \n",
       "791                Q  \n",
       "793                S  \n",
       "794                C  \n",
       "816                S  \n",
       "826                Q  \n",
       "827                S  \n",
       "829                Q  \n",
       "833                C  \n",
       "838                S  \n",
       "840                C  \n",
       "847                S  \n",
       "850                C  \n",
       "860                C  \n",
       "864                S  \n",
       "869                S  \n",
       "879                S  \n",
       "889                S  \n",
       "\n",
       "[177 rows x 11 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.loc[titanic.Age.isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# most frequent Age\n",
    "titanic.Age.mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# fill missing values for Age with the median age\n",
    "titanic.Age.fillna(titanic.Age.median(), inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another strategy would be to build a **KNN model** just to impute missing values. How would we do that?\n",
    "\n",
    "If values are missing from a categorical feature, we could treat the missing values as **another category**. Why might that make sense?\n",
    "\n",
    "How do we **choose** between all of these strategies?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling categorical features\n",
    "\n",
    "How do we include a categorical feature in our model?\n",
    "\n",
    "- **Ordered categories:** transform them to sensible numeric values (example: small=1, medium=2, large=3)\n",
    "- **Unordered categories:** use dummy encoding (0/1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>McCarthy, Mr. Timothy J</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17463</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>E46</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Palsson, Master. Gosta Leonard</td>\n",
       "      <td>male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>349909</td>\n",
       "      <td>21.0750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)</td>\n",
       "      <td>female</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>347742</td>\n",
       "      <td>11.1333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Nasser, Mrs. Nicholas (Adele Achem)</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>237736</td>\n",
       "      <td>30.0708</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass  \\\n",
       "PassengerId                     \n",
       "1                   0       3   \n",
       "2                   1       1   \n",
       "3                   1       3   \n",
       "4                   1       1   \n",
       "5                   0       3   \n",
       "6                   0       3   \n",
       "7                   0       1   \n",
       "8                   0       3   \n",
       "9                   1       3   \n",
       "10                  1       2   \n",
       "\n",
       "                                                          Name     Sex   Age  \\\n",
       "PassengerId                                                                    \n",
       "1                                      Braund, Mr. Owen Harris    male  22.0   \n",
       "2            Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0   \n",
       "3                                       Heikkinen, Miss. Laina  female  26.0   \n",
       "4                 Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0   \n",
       "5                                     Allen, Mr. William Henry    male  35.0   \n",
       "6                                             Moran, Mr. James    male  28.0   \n",
       "7                                      McCarthy, Mr. Timothy J    male  54.0   \n",
       "8                               Palsson, Master. Gosta Leonard    male   2.0   \n",
       "9            Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)  female  27.0   \n",
       "10                         Nasser, Mrs. Nicholas (Adele Achem)  female  14.0   \n",
       "\n",
       "             SibSp  Parch            Ticket     Fare Cabin Embarked  \n",
       "PassengerId                                                          \n",
       "1                1      0         A/5 21171   7.2500   NaN        S  \n",
       "2                1      0          PC 17599  71.2833   C85        C  \n",
       "3                0      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "4                1      0            113803  53.1000  C123        S  \n",
       "5                0      0            373450   8.0500   NaN        S  \n",
       "6                0      0            330877   8.4583   NaN        Q  \n",
       "7                0      0             17463  51.8625   E46        S  \n",
       "8                3      1            349909  21.0750   NaN        S  \n",
       "9                0      2            347742  11.1333   NaN        S  \n",
       "10               1      0            237736  30.0708   NaN        C  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# encode Sex_Female feature\n",
    "titanic['Sex_Female'] = titanic.Sex.map({'male':0, 'female':1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a DataFrame of dummy variables for Embarked\n",
    "embarked_dummies = pd.get_dummies(titanic.Embarked, prefix='Embarked')\n",
    "embarked_dummies.drop(embarked_dummies.columns[0], axis=1, inplace=True)\n",
    "\n",
    "# concatenate the original DataFrame and the dummy DataFrame\n",
    "titanic = pd.concat([titanic, embarked_dummies], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Sex_Female</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass                     Name   Sex   Age  SibSp  \\\n",
       "PassengerId                                                                 \n",
       "1                   0       3  Braund, Mr. Owen Harris  male  22.0      1   \n",
       "\n",
       "             Parch     Ticket  Fare Cabin Embarked  Sex_Female  Embarked_Q  \\\n",
       "PassengerId                                                                  \n",
       "1                0  A/5 21171  7.25   NaN        S           0           0   \n",
       "\n",
       "             Embarked_S  \n",
       "PassengerId              \n",
       "1                     1  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How do we **interpret** the encoding for Embarked?\n",
    "- Why didn't we just encode Embarked using a **single feature** (C=0, Q=1, S=2)?\n",
    "- Does it matter which category we choose to define as the **baseline**?\n",
    "- Why do we only need **two dummy variables** for Embarked?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7937219730941704\n"
     ]
    }
   ],
   "source": [
    "# define X and y\n",
    "feature_cols = ['Pclass', 'Parch', 'Age', 'Sex_Female', 'Embarked_Q', 'Embarked_S']\n",
    "X = titanic[feature_cols]\n",
    "y = titanic.Survived\n",
    "\n",
    "# train/test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "\n",
    "# train a logistic regression model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "# calculate testing accuracy\n",
    "from sklearn import metrics\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ROC curves and AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict probability of survival\n",
    "y_pred_prob = logreg.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = (8, 6)\n",
    "plt.rcParams['font.size'] = 14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'True Positive Rate (Sensitivity)')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xm8VXW9//HX+xxAJpkEC5kRFHFWwKnBtEGtq13rmqaV0yUt7ZeWvyz7WdfbaHmbtIG8hpk5VFqUmJU5JwEOKCIIIgqizAIiIIfz+f2xFrE9nLPOOkfW3vuc834+HvvBXtN3ffb3HPbnfL9rre9XEYGZmVlTaiodgJmZVTcnCjMzy+REYWZmmZwozMwskxOFmZllcqIwM7NMhSUKSddJWi5pdhPbJemHkhZIekLSIUXFYmZmrVdki2IycFzG9uOB0elrIvCTAmMxM7NWKixRRMT9wOqMXU4CfhmJaUAfSQOLisfMzFqnUwXPPQhYXLK8JF33UsMdJU0kaXXQo0ePQ8eMGVOWAM3M2oJFqzawflNd5j6vv7xgZUQMaE35lUwUamRdo+OJRMQkYBLAuHHjYubMmUXGZWb2pqxYv5n6Mg6PdOFNj7Fhcx3XfmJck/vs0af7860tv5KJYgkwpGR5MLC0QrGYme0UN01/gS/e9mTZzzthRD8G9u5WSNmVTBRTgAsk3QwcBqyNiB26nczM2pLl6zYD8LUP7keNGus4Kcahw/oWVnZhiULSTcDRQH9JS4CvAJ0BIuKnwFTgBGAB8BpwVlGxmJm9WZu2bOXhZ1fx97nLeWzxGrbWN77fivVJovjohKHU1JQvURSpsEQREac1sz2ATxd1fjOzN2vx6te4d95y/j53Of94dhWb6+rp1rmWccP70rVzbaPHDO7bjT0H9Gw3SQIq2/VkZlZV1r62hZnPr+afz63mnrnLmb/8VQCG7dad0yYM5ZgxuzNhRL8mk0R75URhZh3WsnWbmP7caqY/t5oZi1Yzb9l6IqBzrZgwoh8fGT+EY8bszoj+PVAZrzdUGycKM+sQIoLnVm5gxqLVTH9uDTMWreaF1a8B0L1LLYcM7csJ+w9k/PB+HDSkD926dKxWQxYnCjNrl7bWB0+/tO5frYUZi9aw8tXkQnO/Hl0YN6wvHz9iGOOH92PfPXrRqdZjpDbFicLM2oVNW7byxJK1aYthNY8+v4b1m5OnlQf16cbbR/dn/PB+TBjRlz0H9OzQXUkt5URhZm3Suk1beOT5NcxIWwyzFq/l9fSe1b3e0pMTD9qDCSP6MX54P/boU8yDaB2FE4WZtQkr1m/+V2thxqLVPP3SOuoDOtWIfQf15hNHDmPCiN0YN6wvfXt0qXS47YoThZlVte//7Rn+8PhSnlu5AYBunWs5eGgfLjxmNBNG9OPgoX3o3sVfZUVy7ZpZVfvpfc8ytF93vnTCGMYP78d+g3rT2Reey8qJwsyq3rv23p2J79iz0mF0WE4UZlaVIoIf/X0Bm7bUs1tPX3OoJCcKM6s6W7bWc9ntT3LrzCWcfMggzjxyRKVD6tCcKMysqqzftIVP3fgoD8xfyWeOHc1F7x7tZx4qzInCzKrGy2s3cdbkGTyzbD1XfugAThk/pPmDrHBOFGYGwKKVG5j8j0U89sKaisWweM1GNm/ZynVnjuede7VqemcrgBOFWQcWETy8cBXXPbiIu+cuo1NNMmpqpW4/Hdi7GxceO4p99+hdkfNb45wozDqgTVu2MmXWUq578Dnmvryefj26cOG7RnHG4cPYvVfXSodnVcaJwqwDWb5+E7+a9gI3TnueVRteZ8xbd+XKDx3AiQft0eEm47H8nCjMqlREMPfl9Wyua2Jy5hZ4bXMdv310CX+ctZS6+uDYMbtz9lEjOGLP3XxHkTXLicKsSt0zbzlnT56508rr3qWW0w8bxieOHM6I/j12WrnW/jlRmFWp9ZuSuRS+8e/7M7D3m7tuIMHBQ/vSu1vnnRGadTBOFGattGzdJi686TGeWba+kPJfT7ucjthzN7cArKKcKMxaYd7L6znrF9NZu3EL/37IIGoL6ufv26MLw/p1L6Rss7ycKMxSP7vvWea+3HzrICK4++nldOtSy63nHeF7/q3dc6IwS13112fYpVMNfbs3P1LpfoN6891TDmSQp9i0DsCJwjqMZ1e8yuwX1za5vb4+OP2wYVx6/JgyRmVW/ZworMO4+NZZzFr8SuY+fbv7riCzhpworE2q21rPa1u2tuiYja/XcdSo3bjipP0a3V4j+cKxWSOcKKzNefqldZx7/UxefGVji48dvfuu7DmgZwFRmbVfmYlC0njgDODtwEBgIzAbuAP4dUQUcwO5WRMemL+C83/1KD136cSXThhDTQtvSz16bw9dbdZSTSYKSX8CVgF/AK4ClgNdgb2AdwF3SLoyIv5UjkCtev3h8Rf55cPPF36eiOCJJWsZtXtPfnHWeAb29h1HZuWQ1aI4JyKWNVi3CZievr4taffCIrM24y9PLWPO0nUcOqxv4ef68KGD+dL796FXV190NiuXJhPFtiQh6TzgpojY4b7CiFheYGz2Jt33zAqeXJJ9l8/OMH/5egb17cavzj2s8HOZWfnluZg9HHhU0j+B6yLib8WGZG9WRHDNPQv47l+eKds53zP2LWU7l5mVV7OJIiIulfQl4HjgPEk/AW4iSRqLCo7PGrFwxau89nrTt4b+atrz3DxjMR88aA++efIBdKotfr6BTjWe08Csvcp1e2xE1EtaBCwC9ie5A+oPkqZGxBebOk7SccAPgFrg2oj4VoPtQ4HrgT7pPpdGxNRWfI4O45Hn1/Chn/yj2f0ueNcoPvfevTwpjZm9ac0mCkmfAs4E1gH/C1wWEZsl1QALgEYThaRa4BrgPcASYIakKRExp2S3LwO3RsRPJI0FppJ0dVkT1m3aAsClx49hZBNDTw/YdRcOHlr8hWUz6xjytCgGA6dGxMLSlWkr48SM4yYAC7YdJ+lm4CSgNFEE0Ct93xtYmjfwjuDPs1/mK1NmUx/b121On0Y+fORuHDSkT4UiM7OOJE+i2KNhkpA0OSLOjIjZGccNAhaXLC8BGt4W81XgL5IuBHoA726sIEkTgYkAQ4cOzRFy+zD7xbUsW7eZ0ya88TP36tqJfQbuWqGozKyjyZMoDihdSLucxuc4rrHO8WiwfBowOSKuknQEcIOk/SLiDbPJR8QkYBLAuHHjGpbRrtXWiG+evH+lwzCzDqymqQ2SviBpDXCApNXpaw2wkuRaQnOWAENKlgezY9fSOcCtABHxMMmT3/1bEL+ZmRUsq0VxJcnQHd8ELt22MiLyDtk5AxgtaQTwInAq8NEG+7wAHAtMlrQPSaJYkbP8Nu/1unrue2YFdVvrG92+YPmrZY7IzGxHWYliVETMl3QDsO+2ldtut4yIJ7IKjog6SRcAd5Hc+npdRDwl6QpgZkRMAT4H/FzSRSTdUmdGRIfpWrr2wYVc+ed5mfv069H8bGtmZkXKShSXknQNXdPItgDe0Vzh6TMRUxusu7zk/RzgqFyRtjOb67byi4cWceSeu3H5v41tcr8BPXcpY1RmZjvKGuvpnPTft5cvnI7j94+9yIr1m/neKQcx5q29mj/AzKxCmryYvY2kRyVdImlYOQLqCOrrg0n3L2TswF4cNWq3SodjZpap2UQB/AfQGZgi6WFJn5U0qOC42rW75y7n2RUb+OQ7R3qIDTOres0mioh4NiK+EREHAmcDhwLFz1LTjk26/1kG9enGCfsPrHQoZmbNyjUooKTBwCnAR9JjLisyqPbs0RfWMGPRGv7fB8bSuTZPg87MrLLyDAr4ELAr8BvgYxFRvkkO2qFJ9y2kd7fOnDp+SPM7m5lVgTwtik82M6aT5fTcyg3cNedlPnX0nvTYJVdjzsys4pr8tpJ0WkTcBBwj6ZiG2yPih4VG1g5d+8BCOtfU8Ikjh1c6FDOz3LL+rN02ocGARrZ1mKend5aVr27mt48s4eRDBrH7rl0rHY6ZWW5ZD9z9OH17R0RMK90m6fBCo2qHbpmxmM119Zz79pGVDsXMrEXy3Hbz40bWNTash2V48ZWN9O/ZhVG796x0KGZmLZJ1jWICcAQwQNJnSjb1InkAz1rMD9eZWduTdY2iB8ncEJ1443WK9SRPa5uZWQeQdY3iHuAeSb9oOBWq5RMRTH9uNes31bFkzcZKh2Nm1ipZXU9XRcTngKsk7XCXU0ScXGhk7cAzy17lI5O23wcwckCPCkZjZtY6WV1Pt6T/Xl2OQNqjjVuSyQAv/8BYxg/vxx59fFusmbU9WV1P09N/7962TlJvYFA64ZDlNKJ/D/Yf3LvSYZiZtUqe+SjultRLUl/gSeDXkr5TfGhmZlYN8gw41C8i1kk6B7g+Iv6fpCeASwqOrU2Z+/I6vn3nXOrqt1/OWbeproIRmZntHHkeuOskaQDJLbF/LDieNuvB+Su5Z94K1m3cwobNdWzYXEet4G2j+rPPQE91amZtV54WxdeB+4AHI2K6pJHAc8WG1XbdcO5h9Orq5xHNrP1oNlFExM3AzSXLC4GTigzKzMyqR56Ji/qTTIE6vHT/iJhYXFhmZlYt8nQ9/QGYBjwIbC02HDMzqzZ5EkWP9AltMzPrgPLc9XSnpPcWHomZmVWlPIniPODPkl6VtFrSGkmriw7MzMyqQ56up/6FR9GGXfvAQpas2chTS9dWOhQzs0LkuT12q6RTgZER8Q1Jg4G3AI8UHl2Ve3VzHV+742l26VRD1861jB3Yi+6daysdlpnZTpXn9tirSWa0ewfwDeA14KfA+GJDq34RyXAdl7xvb8+FbWbtVp6upyMj4hBJjwFExGpJXQqOy8zMqkSei9lbJNUAASBpN6C+0KjMzKxq5EkU1wC/AwZI+i+SB+++XWhUbcTsF9cBUCNVOBIzs+LkuZj9S0mPAO9OV/1HRMwuNqzqd8cTL3HRrY8zon8P3n/AwEqHY2ZWmKw5s7sCWyJia0Q8JWkzcDwwEsiVKCQdB/wAqAWujYhvNbLPKcBXSbq2ZkXER1v8Kcpg5aub+cqUp9j4+la2bK3ngfkrOXRYX37+8XH06+FLNmbWfmV1Pd0F7AkgaU9gOjAWuFjS15srWFItSbfV8elxp0ka22Cf0cAXgaMiYl/gs635EOXw5JK13PHESzy/agOvvLaFjx42lBvPPcxJwszavayup34R8Uz6/hPAzRHxKUm7ADOBy5opewKwIB2WHEk3kwxPXjrf9n8C10TEGoCIWN6Kz7DT3PHES6zesLnRbfOWrQfgqlMO4qAhfcoZlplZRWUliih5fwxwFUBEbJaU566nQcDikuUlwGEN9tkLQNJDJN1TX42IPzcsSNJEYCLA0KFDc5y65Za+spFP//rRzH061Yjd3IIwsw4mK1E8JelbwIskX+h/AZDUG8hzm09j+0SD5U7AaOBoYDDwgKT9IuKVNxwUMQmYBDBu3LiGZewUdVuTYv/rxH2bvDjdtXMtPXfJ8+iJmVn7kfWtdy5wETAGOC4iNqTr9wP+J0fZS4AhJcuDgaWN7DMtIrYAz0maR5I4ZuQovxA9d+lE/567VOr0ZmZVp8lEkSaGrzWy/iHgoRxlzwBGSxpB0io5FWh4R9PvgdOAyelMensBC/OFbmZm5dDkXU+Sfi/peEk7JBNJwyRdLunspo6PiDrgApK7p54Gbk1vs71C0onpbncBqyTNAe4BLomIVW/mA7VWfRTSo2Vm1uZldT19GvgccI2kZcAKoCvJcxQvkNyt9LuswiNiKjC1wbrLS94HcHH6qqjZ6TDhw/t3r3AkZmbVJavr6UXSL3FJo4CBwEZgXkSsL1N8ZXPvvBX07taZg4b0rXQoZmZVJdctPBGxAFhQcCwVU18f3PfMCt4+uj+1NR63ycysVJ5BAdu9OS+tY8X6zRy99+6VDsXMrOo4UQD3PbMCgHfuNaDCkZiZVZ9ciUJSl/Q6Rbt077zl7DeoFwN29fMTZmYNNZsoJL0feBL4a7p8kKTbiw6sXNa+toVHnl/D0Xu528nMrDF5WhRXkIzR9ApARDwOtJvWxQMLVlAfcPTe7nYyM2tMnruetkTEK3rjLG5t+um0F1/ZyLfunMvrdVuZv+zV9LZYjwhrZtaYPIni6XRyoZp0OI7/A0wrNqxiTXt2FX+ctZSR/XvQpVMNE98xkk61vq5vZtaYPIniAuByoB64jWTYjS8WGVS5TD5rAkN385PYZmZZ8iSK90XEF4AvbFsh6WSSpGFmZu1cnv6WLzeyrrnZ7czMrJ1oskUh6X3AccAgSaXzT/Qi6YYyM7MOIKvraTkwG9gEPFWyfj1waZFBmZlZ9cgaPfYx4DFJN0bEpjLGZGZmVSTPxexBkr4OjCWZjwKAiNirsKjMzKxq5EkUk0mmRP0ucDxwFm30GsVtjy5h/vJXmfdyu5tOw8ysMHkSRfeIuEvSdyPiWeDLkh4oOrAiXPq7J6mrr6dTTQ0De3elb4/OlQ7JzKzq5UkUm5WM3/GspPOAF4E2OYJefQTnH70nl7xvTKVDMTNrM/IkiouAnsBngK8DvYGziwzKzMyqR7OJIiL+mb5dD3wMQNLgIoMyM7PqkZkoJI0HBgEPRsRKSfuSDOVxDNAmksUN057n23fOJSKoqw9q5DmxzcxaIuvJ7G8CHwJmkVzAvp1k5NhvA+eVJ7w3b87SdWytD04/bCg1NeLkQ9pEfjMzqxpZLYqTgAMjYqOkfsDSdHleeUJrvZfWbuQHf5vP+s11PLlkLT27duLLHxhb6bDMzNqkrESxKSI2AkTEaklz20KSmLN0HWdPnsHajVvYo09XOteKo0a9pdJhmZm1WVmJYqSkbUOJCxheskxEnFxoZK3wjwUrmXjDI/TcpRO3fepI9hnYq9IhmZm1eVmJ4kMNlq8uMpCd4Zt3zqVfjy7c8snDGdi7W6XDMTNrF7IGBby7nIHsDJvrtrLvHr2cJMzMdiJPFG1mZpmcKMzMLFPuRCFplyIDMTOz6tRsopA0QdKTwPx0+UBJPyo8MjMzqwp5WhQ/BD4ArAKIiFnAu4oMyszMqkeeRFETEc83WLe1iGDMzKz65BlmfLGkCUBIqgUuBJ4pNiwzM6sWeVoU5wMXA0OBZcDh6bpmSTpO0jxJCyRdmrHfhyWFpHF5yjUzs/LJ06Koi4hTW1pw2vq4BngPsASYIWlKRMxpsN+uJJMi/XPHUszMrNLyJIoZkuYBtwC3RcT6nGVPABZExEIASTeTjEg7p8F+/w1cCXw+Z7lv8MD8FVx0y+PU1QfrNm5h1O49W1OMmZk1odmup4jYE/gacCjwpKTfS8rTwhgELC5ZXpKu+xdJBwNDIuJPWQVJmihppqSZK1aseMO2eS+vZ+Wrr3P8fgP52OHDOPuoETlCMzOzvPK0KIiIfwD/kPRV4PvAjcDNzRzW2FRy8a+NUg3wPeDMHOefBEwCGDduXDS2z5dOGMOuXTs3V5SZmbVQngfueko6XdIfgenACuDIHGUvAYaULA8mmfxom12B/YB7JS0iuUg+xRe0zcyqS54WxWzgj8CVEfFAC8qeAYyWNAJ4ETgV+Oi2jRGxFui/bVnSvcDnI2JmC85hZmYFy5MoRkZEfUsLjog6SRcAdwG1wHUR8ZSkK4CZETGlpWWamVn5NZkoJF0VEZ8Dfidph+sCeWa4i4ipwNQG6y5vYt+jm422xOt19Wyq28qmLX5I3MysSFktilvSf6tuZrvX6+o54pt3s2rD6/9aV1vT2LVzMzN7s7JmuJuevt0nIt6QLNIupYrNgLepbiurNrzOu/d5C4eP7MegPt3o3iXXDVxmZtZCeYbwOLuRdefs7EBa4/CR/Tj37SM5fv+BlQ7FzKzdyrpG8RGSO5VGSLqtZNOuwCtFB2ZmZtUhq79mOskcFINJxmzaZj3wWJFBmZlZ9ci6RvEc8Bzwt/KFY2Zm1Sar6+m+iHinpDWUDL1BMjRHRES/wqMzM7OKy+p62jbdaf+MfczMrJ1r8q6nkqexhwC1EbEVOAL4JNCjDLGZmVkVyHN77O9JpkHdE/glsA/w60KjMjOzqpEnUdRHxBbgZOD7EXEhDeaVMDOz9itPoqiT9B/Ax4BtEwx54gczsw4i75PZ7yIZZnxhOmz4TcWGZWZm1aLZAZIiYrakzwCjJI0hmQf768WHZmZm1aDZRCHp7cANJJMPCXirpI9FxENFB2dmZpWXZ8jV7wEnRMQcAEn7kCQOT1lqZtYB5LlG0WVbkgCIiKeBLsWFZGZm1SRPi+JRST8jaUUAnI4HBTQz6zDyJIrzgM8A/5fkGsX9wI+KDMrMzKpHZqKQtD+wJ3B7RFxZnpDMzKyaNHmNQtKXSIbvOB34q6TGZrozM7N2LqtFcTpwQERskDQAmApcV56wzMysWmTd9bQ5IjYARMSKZvY1M7N2KqtFMbJkrmwBe5bOnR0RJxcamZmZVYWsRPGhBstXFxmImZlVp6w5s+8uZyBmZladfN3BzMwyOVGYmVmm3IlC0i5FBmJmZtWp2UQhaYKkJ4H56fKBkjyEh5lZB5GnRfFD4APAKoCImEUy452ZmXUAeRJFTUQ832Dd1iKCMTOz6pNn9NjFkiYAIakWuBB4ptiwzMysWuRpUZwPXAwMBZYBh6frzMysA2g2UUTE8og4NSL6p69TI2JlnsIlHSdpnqQFki5tZPvFkuZIekLS3ZKGteZDmJlZcZrtepL0cyAaro+Iic0cVwtcA7wHWALMkDSldFpVkpnyxkXEa5LOB64EPtKC+M3MrGB5rlH8reR9V+DfgcU5jpsALIiIhQCSbgZOAkrn376nZP9pwBk5yjUzszJqNlFExC2ly5JuAP6ao+xBvDGhLAEOy9j/HODOxjZImghMBBg6dGiOU5uZ2c7SmiE8RgB5riWokXU7dGEBSDoDGAd8p7HtETEpIsZFxLgBAwbkDtTMzN68PNco1rD9C74GWA3scGG6EUuAISXLg4GljZT/buAy4J0RsTlHuWZmVkaZiUKSgAOBF9NV9RHRaKugETOA0ZJGpMefCny0QfkHAz8DjouI5S0J3MzMyiOz6ylNCrdHxNb0lTdJEBF1wAXAXcDTwK0R8ZSkKySdmO72HaAn8BtJj0ua0rqPYWZmRclz19N0SYdExKMtLTwipgJTG6y7vOT9u1tappmZlVeTiUJSp7RV8DbgPyU9C2wguUgdEXFImWJ8g6WvbOSbU+dW4tRmZh1SVotiOnAI8MEyxZLLqg2vM/XJl+jfswtj3tqr0uGYmbV7WYlCABHxbJliyaVrp1pmfeW9lQ7DzKzDyEoUAyRd3NTGiPifAuIxM7Mqk5UoaknuSGrswTkzM+sgshLFSxFxRdkiMTOzqpT1HIVbEmZmlpkoji1bFGZmVrWaTBQRsbqcgZiZWXVqzeixZmbWgThRmJlZJicKMzPL5ERhZmaZnCjMzCyTE4WZmWVyojAzs0xOFGZmlsmJwszMMjlRmJlZJicKMzPL5ERhZmaZnCjMzCyTE4WZmWVyojAzs0xOFGZmlsmJwszMMjlRmJlZJicKMzPL5ERhZmaZnCjMzCyTE4WZmWVyojAzs0xOFGZmlsmJwszMMjlRmJlZpkIThaTjJM2TtEDSpY1s30XSLen2f0oaXmQ8ZmbWcoUlCkm1wDXA8cBY4DRJYxvsdg6wJiJGAd8Dvt1cuYP6dtvZoZqZWYYiWxQTgAURsTAiXgduBk5qsM9JwPXp+98Cx0pSVqHdu9Tu9EDNzKxpnQosexCwuGR5CXBYU/tERJ2ktcBuwMrSnSRNBCami5slzS4k4ranPw3qqgNzXWznutjOdbHd3q09sMhE0VjLIFqxDxExCZgEIGlmRIx78+G1fa6L7VwX27kutnNdbCdpZmuPLbLraQkwpGR5MLC0qX0kdQJ6A6sLjMnMzFqoyEQxAxgtaYSkLsCpwJQG+0wBPpG+/zDw94jYoUVhZmaVU1jXU3rN4QLgLqAWuC4inpJ0BTAzIqYA/wvcIGkBSUvi1BxFTyoq5jbIdbGd62I718V2rovtWl0X8h/wZmaWxU9mm5lZJicKMzPLVLWJwsN/bJejLi6WNEfSE5LuljSsEnGWQ3N1UbLfhyWFpHZ7a2SeupB0Svq78ZSkX5c7xnLJ8X9kqKR7JD2W/j85oRJxFk3SdZKWN/WsmRI/TOvpCUmH5Co4IqruRXLx+1lgJNAFmAWMbbDPp4Cfpu9PBW6pdNwVrIt3Ad3T9+d35LpI99sVuB+YBoyrdNwV/L0YDTwG9E2Xd6903BWsi0nA+en7scCiSsddUF28AzgEmN3E9hOAO0meYTsc+Geecqu1RVHI8B9tVLN1ERH3RMRr6eI0kmdW2qM8vxcA/w1cCWwqZ3Bllqcu/hO4JiLWAETE8jLHWC556iKAXun73uz4TFe7EBH3k/0s2knALyMxDegjaWBz5VZromhs+I9BTe0TEXXAtuE/2ps8dVHqHJK/GNqjZutC0sHAkIj4UzkDq4A8vxd7AXtJekjSNEnHlS268spTF18FzpC0BJgKXFie0KpOS79PgGKH8HgzdtrwH+1A7s8p6QxgHPDOQiOqnMy6kFRDMgrxmeUKqILy/F50Iul+OpqklfmApP0i4pWCYyu3PHVxGjA5Iq6SdATJ81v7RUR98eFVlVZ9b1Zri8LDf2yXpy6Q9G7gMuDEiNhcptjKrbm62BXYD7hX0iKSPtgp7fSCdt7/I3+IiC0R8RwwjyRxtDd56uIc4FaAiHgY6EoyYGBHk+v7pKFqTRQe/mO7Zusi7W75GUmSaK/90NBMXUTE2ojoHxHDI2I4yfWaEyOi1YOhVbE8/0d+T3KjA5L6k3RFLSxrlOWRpy5eAI4FkLQPSaJYUdYoq8MU4OPp3U+HA2sj4qXmDqrKrqcobviPNidnXXwH6An8Jr2e/0JEnFixoAuSsy46hJx1cRfwXklzgK3AJRGxqnJRFyNnXXwO+Lmki0i6Ws5sj39YSrqJpKuxf3o95itAZ4CI+CnJ9ZkTgAXAa8BZucpth3VlZmY7UbV2PZmZWZVwojAzs0xOFGZmlsmJwszMMjlRmJlZJieKDkrSVkmPl7yGZ+w7vKnRKFt4znvTET5npcNK7N2KMs72a0i3AAAGAElEQVST9PH0/ZmS9ijZdq2ksTs5zhmSDspxzGcldW/Fub4v6R3p+wvSUT0jfe6hpWXtncb+uKSnJe3U2d0knbhtZFZJA5SM2vyYpLdLmiqpT8axTf7cMo75m6S+O+8TWKtVerRDvyrzAl5twb7DaWI0yhae817S0VyBicCUnVXeTq6b0jjPAv6a45hFQP8WnqcfMK1k+eC0rltcVnr8XcBJJcv7F/j7cypwfZE/N5IHai8r6jP4lf/lFoX9S9pyeEDSo+nryEb22VfS9PSv1ickjU7Xn1Gy/meSaps53f3AqPTYY9O/TJ9UMp7+Lun6b2n7PBvfTdd9VdLnJX2YZFyrG9Nzdkv/mh4n6XxJV5bEfKakH7UyzocpGTRN0k8kzVQyv8N/pes+A+wB3CPpnnTdeyU9nNbjbyT1bKTsDwN/3rYQEY9FxKJm4skykGSIhm3lPZnGcqakP0j6c9pS+krJ52m0PpTM7/Bo2qq6u6Scq9MW1pXACSV1v2hbK0jSx9Of2SxJN6Trmvq5vV/S7SXxvEfSbeniFJIxmqzSKp2p/KrMi+RJ3cfT1+3puu5A1/T9aJKnWqGkRQH8CDg9fd8F6AbsA/wR6Jyu/zHw8UbOeS/b/1K/BLiFZCiFxcBe6fpfAp8l+Wt7HtsfCu2T/vtV4PMNyytdBgaQDDu9bf2dwNtaGedngW+UbOuX/lub7ndAuryItBVAMobQ/UCPdPkLwOWNnOd64N8aWf+vslr4Mz2LZBTlO4GLSursTOAlktGVuwGz03pqtD7S+lsMjGjwmc8Erm74vjRmYN/059a/wbGN/txIBqmbCwxIl39dWifAfGC3Sv9/6eivqhzCw8piY0Q07HvvDGz7i3ErydhADT0MXCZpMHBbRMyXdCxwKDBDyRAi3YCmxpy6UdJGki+WC4G9geci4pl0+/XAp4GrSeaTuFbSHUDuYcMjYoWkhUrGspmfnuOhtNyWxNmDJCGUzgJ2iqSJJMPfDCSZBOeJBscenq5/KD1PF5J6a2ggO3G8oYj4haS7gONI5h34pKQD081/jXT4jvQv9rcBdTReH4cD90cykCAR0ZLBNo8BfhsRK/McGxGRtjrOkPQL4AiSZLXNcpLWWrsbeqQtcaKwUhcBy4ADSW502GHin4j4taR/Au8H7pJ0LslfhddHxBdznOP0KBmkT1Kjc4hEMn7PBJKB3E4FLiD5EsrrFuAUkr9Wb0+/kFoUJ8lMad8CrgFOljQC+DwwPiLWSJpM0iJqSCRfzM11m2xs4vgmpV+mBwNLI2KH6TwjYilwHXCdkhsQ9tu2qeGuNPFzk3RiI/vnDrEVx/6CpGWzCfhNJPPLbNOVpJ6sgnyNwkr1Bl6KZIz+j5H8Nf0GkkYCCyPihyR9yAcAdwMflrR7uk8/5Z+3ey4wXNKodPljwH1pn37viJhK0v3T2J1H60mGFm/MbcAHSfq4b0nXtSjOiNgCfBk4XMmIo72ADcBaSW8Bjm8ilmnAUds+k6TukhprnT1Nep0mr4g4KyIOaixJpNcVOqfv30rS1fRiuvk96eftRlIvD9F0fTwMvDNNjEjq14IQ7yZpde2Wcewbfm5pcltKUteTSz6PgLeStD6tgpworNSPgU9ImkbS7bShkX0+AsyW9DgwhmRaxTkk/8n/IukJ4K8k3SrNiohNJH3rv5H0JFAP/JTki+RPaXn3kbR2GpoM/HTbBdUG5a4B5gDDImJ6uq7FcUbERuAqkv71WSRzUD9F8lf7QyW7TgLulHRPRKwg6cO/KT3PNJK6augOkpE+geSiuJIRPwcDT0i6Niu2RryX5Gczi+QOqEsi4uV024PADSTXpH4XETObqo80/onAbWlZtzQ8UVMi4ing6yTJfhbwP43sNpkdf243AovTmLY5lOSusLqGBVh5efRYswqS9CDwgShw1jlJZ5JcPL6gqHO8WZKuBh6LiP8tWfcDkluo765cZAZuUZhV2ueAoZUOopIkPULShfmrBptmO0lUB7cozMwsk1sUZmaWyYnCzMwyOVGYmVkmJwozM8vkRGFmZpn+PxvEdstpBSQuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c7536d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot ROC curve\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.0])\n",
    "plt.xlabel('False Positive Rate (1 - Specificity)')\n",
    "plt.ylabel('True Positive Rate (Sensitivity)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8386924342105262\n"
     ]
    }
   ],
   "source": [
    "# calculate AUC\n",
    "print(metrics.roc_auc_score(y_test, y_pred_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides allowing you to calculate AUC, seeing the ROC curve can help you to choose a threshold that **balances sensitivity and specificity** in a way that makes sense for the particular context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([<matplotlib.axes._subplots.AxesSubplot object at 0x7f96c7564160>,\n",
       "       <matplotlib.axes._subplots.AxesSubplot object at 0x7f96c742eac8>],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAETCAYAAADecgZGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEO5JREFUeJzt3X+w5XVdx/Hny10Iwx+ArLiBt2VqLbUZ0AxpmGlUSlFM+ENKzVgnZrYpbTQbdTWn1KzAZjKbsR8MkJulwJgIifmjFSzH5DeYuMkq4RVBFxP8NakB7/4435XLcvfec+8953vOuZ/nY+bO/Z7v+R7O637vfl587/d8zvekqpAkteVhkw4gSeqf5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlPmSRHJLkkyXeTfCnJSyadSZqEJK9Icm2S7yd516TzrDcbJx1AD/FO4AfAUcDxwOVJbqqqmycbS+rdHcBbgecAD59wlnUnvsN3eiQ5FLgb+JmquqVb927gK1W1Y6LhpAlJ8lbgmKp62aSzrCee9pkuTwDu21f8nZuAJ08oj6R1yvKfLo8Avrnfum8Cj5xAFknrmOU/Xb4DPGq/dY8Cvj2BLJLWMct/utwCbEyydcG64wBf7JU0Upb/FKmq7wLvB96S5NAkJwGnAe+ebDKpf0k2JjkE2ABsSHJIEmcojojlP31+m8G0tr3Ae4HfcpqnGvVG4H+BHcBLu+U3TjTROuJUT0lqkEf+ktQgy1+SGmT5S1KDLH9JapDlL0kN6nXO7JFHHllbtmzp8ynVgOuuu+7rVbVp0jlWwrGgcVjJWOi1/Lds2cK1117b51OqAUm+NOkMK+VY0DisZCx42keSGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUoKn5VJwtOy5f8WNuO/vUMSSRpPVvqPJPchuDDxG/D7i3qp6W5AjgImALcBvwK1V193hiSpJGaSWnfZ5ZVcdX1dO62zuAXVW1FdjV3ZYkzYC1nPM/DdjZLe8ETl97HElSH4Yt/wI+muS6JNu7dUdV1Z0A3ffHLvbAJNuT7Ely1/z8/NoTSzPKsaBpMmz5n1RVTwWeC7w8yS8M+wRVdW5Vba2qTXNzc6sKKa0HjgVNk6HKv6ru6L7vBS4BTgC+lmQzQPd977hCSpJGa9nyT3JokkfuWwaeDXwWuAzY1m22Dbh0XCElSaM1zFTPo4BLkuzb/j1V9eEk1wAXJzkLmAfOGF9MSdIoLVv+VXUrcNwi6/8HOHkcoSRJ4+XlHSSpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JatDQ5Z9kQ5Ibknywu31skquS7ElyUZKDxxdTkjRKKznyfyWwe8Htc4C3V9VW4G7grFEGkySNz1Dln+QY4FTgvO52gGcB7+s22QmcPo6AkqTRG/bI/y+A1wL3d7cfA9xTVfd2t28Hjl7sgUm2d6eG7pqfn19TWGmWORY0TZYt/yTPB/ZW1XULVy+yaS32+Ko6t6q2VtWmubm5VcaUZp9jQdNk4xDbnAS8IMnzgEOARzH4S+CwJBu7o/9jgDvGF1OSNErLHvlX1eur6piq2gK8CPh4Vf0acAXwwm6zbcClY0spSRqptczzfx3w6iRfYPAawPmjiSRJGrdhTvv8UFVdCVzZLd8KnDD6SJKkcfMdvpLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWqQ5S9JDbL8JalBlr8kNWjZ8k9ySJKrk9yU5OYkb+7WH5vkqiR7klyU5ODxx5UkjcIwR/7fB55VVccBxwOnJDkROAd4e1VtBe4GzhpfTEnSKC1b/jXwne7mQd1XAc8C3tet3wmcPpaEkqSRG+qcf5INSW4E9gIfA74I3FNV93ab3A4cPZ6IkqRRG6r8q+q+qjoeOAY4AXjiYpst9tgk27vXBe6an59ffVJpxjkWNE1WNNunqu4BrgROBA5LsrG76xjgjgM85tyq2lpVm+bm5taSVZppjgVNk2Fm+2xKcli3/HDgF4HdwBXAC7vNtgGXjiukJGm0Ni6/CZuBnUk2MPifxcVV9cEknwMuTPJW4Abg/DHmlCSN0LLlX1WfAZ6yyPpbGZz/lyTNGN/hK0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSg4Z5k9fU2rLj8hVtf9vZp44piSTNFo/8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAZZ/pLUIMtfkhpk+UtSgyx/SWrQxkkHkKRWbdlx+aoed9vZp675uZc98k/y+CRXJNmd5OYkr+zWH5HkY0n2dN8PX3MaSVIvhjntcy/we1X1ROBE4OVJngTsAHZV1VZgV3dbkjQDli3/qrqzqq7vlr8N7AaOBk4Ddnab7QROH1dISdJoregF3yRbgKcAVwFHVdWdMPgfBPDYAzxme3dq6K75+fm1pZVmmGNB02To8k/yCOCfgFdV1beGfVxVnVtVW6tq09zc3GoySuuCY0HTZKjyT3IQg+L/x6p6f7f6a0k2d/dvBvaOJ6IkadSGme0T4Hxgd1X9+YK7LgO2dcvbgEtHH0+SNA7DzPM/Cfh14D+T3NitewNwNnBxkrOAeeCM8UQcndXMqR3FfFpJmjbLln9VfRLIAe4+ebRxJEl98PIOktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUFez1+SRmC11+afFI/8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoMsf0lqkOUvSQ2y/CWpQZa/JDXI8pekBln+ktQgy1+SGmT5S1KDLH9JapDlL0kNsvwlqUGWvyQ1yPKXpAb5Gb7SFFvN58LedvapY0ii9cYjf0lqkOUvSQ2y/CWpQZa/JDVo2fJPckGSvUk+u2DdEUk+lmRP9/3w8caUJI3SMEf+7wJO2W/dDmBXVW0FdnW3JUkzYtnyr6p/A76x3+rTgJ3d8k7g9BHnkiSN0Wrn+R9VVXcCVNWdSR57oA2TbAdeAxy2adOmVT6dNPscC/3zfRIHNvYXfKvq3KraWlWb5ubmxv100tRyLGiarLb8v5ZkM0D3fe/oIkmSxm215X8ZsK1b3gZcOpo4kqQ+LHvOP8l7gWcARya5HfhD4Gzg4iRnAfPAGeMMKWn9WM15eOjvXPxq882aZcu/ql58gLtOHnEWSVJPfIevJDXI8pekBnk9/2U4T1iaDq2ci++LR/6S1CDLX5IaZPlLUoM85z/DfD1C0mp55C9JDbL8JalBlr8kNcjyl6QGWf6S1CDLX5IaZPlLUoOc5z8Gzr+XNO088pekBln+ktQgy1+SGuQ5/ynhtco1Kr7mpGF45C9JDbL8JalBlr8kNchz/o3p67UFzyG3wdeqZpdH/pLUIMtfkhpk+UtSgzznL8lz9w3yyF+SGmT5S1KDLH9JapDlL0kNWtMLvklOAd4BbADOq6qzR5JKM8+Li0nTbdVH/kk2AO8Engs8CXhxkieNKpgkaXzWctrnBOALVXVrVf0AuBA4bTSxJEnjtJbyPxr48oLbt3frJElTbi3n/LPIunrIRsl24DXAYcD3ktzc3XUk8PU1PP+omWdpY8+Tc1a0+cI8Pz7yMGOwxFiABn/fK2SeBRYZK/vyDD0WUvWQvh7ugcnPA2+qqud0t18PUFV/OuTjr62qp63qycfAPEszz3hN289jnqWthzxrOe1zDbA1ybFJDgZeBFy2hv+eJKknqz7tU1X3JnkF8BEGUz0vqKqbl3mYJGkKrGmef1V9CPjQKh9+7lqeewzMszTzjNe0/TzmWdrM51n1OX9J0uzy8g6S1CDLX5IaZPlLUoN6+ySvJGFwSYijGbwZ7A7g6pqCFx2SHAs8BfhcVf3XhDI8GjiFB++fj1TVPZPIs9A07J/1xLGwbAbHQg96OfJP8mxgD/Am4HnAqcCbgT3dfb1K8oEFy6cBHwd+Gbg0ycsmkOdM4HrgGcCPAocCzwSu6+7rO89U7Z8ux6OT/GqSVyf53W75sElkWQvHwrJ5HAvD5Vr7eKiqsX8Bu4Eti6w/FtjdR4b9nveGBcufAo7tlo8EbppAns8Dhy2y/nDgFvcPZwJfBP4aeGP39TfdujP7zrPGn8WxsHQex8LymUYyHvo67bORwYXf9vcV4KCeMiy08M/rjVX13wBV9fUk908gT1jkukjA/Sx+DaVxm7b98/vAz9Z+f/YnORy4Cvj7CWRaLcfC0hwLyxvJeOir/C8ArklyIQ9cCfTxDC4JcX5PGRY6Lsm3GPxj+pEkj6uqr3aXqdgwgTx/DFyf5KM8sH/mgF8C/mgCeaZt/0xbIayFY2FpjoXljWQ89PYmr+6DXl7A4EWcMDj6uayqPtdLgCF058yeWFX/MYHnPhx4Dg/ePx+pqrv7znIgk9o/SbYBfwAsWghV9a4+86yVY2HZ53YsLP3cIxkPvb/DN8kRQE3TL1LTbxYKYaUcC1qtUYyHvmb7zCW5MMleBuekrk6yt1u3pY8M++V5fPfc/57kDUkOWnDfB5Z67Jjy/MaC5aOT7Epyd5JPJXnCBPJ8I8l5SU7upiVOXPeP+oruaxdwxSyWpmNh2TyOhSGMYjz09Savi4BLgM1VtbWqtgKbgQ8w+PjHvl0AXAn8TpfjE0ke0903iQ8GecWC5bcDFwOPAf6MwSv6fbsLuBF4C3B7knckOXECOQBIcnySTzP4nZ3DYL98Ismnkzx1UrlWybGwNMfCMkY2HnqamrRnNfeNMc+N+91+KXAz8BPA9RPIc/2C5f2z3dBnlkXyzAGvZTD3+lbgTybx+wKevsj6E5nQdLs1/CyOhaXzOBaG+J2NYjz0NdvnuiR/BezkwTMctgE39JRhoYOSHFJV3wOoqn9I8lUGn01w6ATyHJPkLxmcu9uU5KCq+r99WSeQ54d/3lbVPPA24G1JforBrJS+HVpVV+2/sqo+nWQSv6+1cCwszbGwvJGMh77K/0zgLAbvZNz3AsWXgX9mMtPbzgOeDnxi34qq+tckZzD45fbtNQuWrwUeAdyd5HFM5tPRrlhsZVV9nsHvsG//kuRyBvOXFxbmmcCHJ5BnLRwLS3MsLG8k48Hr+WsmJHkucBoPnR652g8TkmbWKMbDxMs/yfOr6oMTDbGAeZY2bXnWk2nbt+ZZ2rTlWalpuKTzz006wH7Ms7SpypNk+6QzjNBU7VvMs5xpy7Oi8dDnO3x/mgf+TNl3mdbLqmp3LwHMM9N5DiTJb1bV3046x0pM2741z2zlWcpKxkNfb/J6HYM5zAGuBq7plt+bZEcfGcwzu3mW8YNJB1iJadu35pmtPEMYejz0cuSf5BbgyQumbO1bfzBwcw3e6NIb88xWnqUkma+quUnnGNa07VvzzFae5axkPPQ11fN+4MeAL+23fnN3X9/Ms7SpypPkMwe6CziqzywjMFX7FvMsZ9ryjGw89FX+rwJ2JdnDg69C95M8+O3cfTHPbOU5isFFrPa/dkkYfMDGLJm2fWue2coDIxoPfb7g+zAe+NzSffNSr6mq+3oJYJ6ZzZPkfODvquqTi9z3nqp6Sd+Z1mKa9q15ZjLPSMbDxOf5S5L6Nw3z/CVJPbP8JalBlr8kNcjyl6QGWf6S1KD/B3xbEU9laU1VAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c74f9eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# histogram of predicted probabilities grouped by actual response value\n",
    "df = pd.DataFrame({'probability':y_pred_prob, 'actual':y_test})\n",
    "df.hist(column='probability', by='actual', sharex=True, sharey=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f96c73a7f60>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGrhJREFUeJzt3WtsXPd55/Hvo/udGoqUowsvsi1ZUhRZJseys3Fs2YoDO23tLpC2Uje7m643RtK6C2yKBVxkkS3cN7spukELGE3VNkhboHXdvmiFQoWLirS9dazUpCQ7ljSSaIU3SdZQ4pC6kBRvz76YITWkSPGIGs7lnN8HEMCZOSKfI1I/Hf3nOf/H3B0REQmXeYUuQEREck/hLiISQgp3EZEQUriLiISQwl1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREJoQaG+cEVFhdfW1hbqy4uIlKTm5ubL7l4503EFC/fa2lqampoK9eVFREqSmbUFOU7LMiIiIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIzhruZ/dDMkmb28TSvm5n9oZm1mNlHZlaX+zJFRORuBLly/xHw3B1efx7YnPn1MvBH916WiIjcixnD3d3fBbrvcMiLwF942hFgtZmty1WBIiJhcP3mMP969jJ/8C9n+fh875x/vVzcxLQB6Mh63Jl57uLkA83sZdJX91RXV+fgS4uIFB9353xPP81tKZrbUjS1pkh8epVRBzMoX7GIHRvK5rSGXIS7TfHclFO33f0AcAAgHo9rMreIhMLQyCgnL1ylqS3F0bYUTW3dXLp6E4Bli+bzSPVqXnlmM/U1MR6pXs2qJQvnvKZchHsnUJX1eCNwIQefV0SkKPX0DXK0PX1F3tyW4sPOHgaGRgHYsHopj21aQ31NjPqaGFs/s5IF8/PfmJiLcD8IvGJmbwCPAb3uftuSjIhIKXJ3fnb5RtZVeYqW5HUA5s8zPrt+Fft3V4+H+bqypQWuOG3GcDezvwb2ABVm1gn8L2AhgLv/ADgEfAVoAfqAX5urYkVE5trA0Ag/Pd87flV+tD1F941BAFYtWUB9TYx//8gG6qpjPFxVxrJFBdt/8Y5mrMrd98/wugO/kbOKRETyKHltIH1F3pqiuT3Fx+d7GRpJvyV4f8Vyntm6lnjmqvyByhXMmzfV24zFpzj/yRERmQMjo86ZS9fGu1ia21K0d/cBsGjBPB7eWMZ/eWIT8Zpy6qpXs2bF4gJXPHsKdxEJres3h/mwo4em1nQHy/H2Hq7dHAagYsUi6mti/MfHa6iribFjwyoWL5hf4IpzR+EuIqEwube8uS3FqYu3essfum8lv7Br/fgSS3X5MsxKY4llNhTuIlKSxnrLs8P806sDQFZv+dMPUl9bzq6q1ZQtnfve8mKicBeRkjDWWz52x+fk3vLdm8oL3lteTBTuIlJ0gvSW73u0mnhtcfWWFxOFu4gU3Fhv+dhV+VS95b+4az31NeVF3VteTPQnJCJ513XtJs1t3ekwb5vYW74p01teXxMjXmK95cVE4S4ic2pk1DmbvJa+Is+EeXZv+c4N6d7y+ur0Eksp95YXE4W7iORUdm95c3uKY22pyPSWFxOFu4jMmrtzoXeAptbu8avy7N7yLWvTveX11THiteHvLS8mCncRCWxoZJRTF6+OX5U3t07sLd9Vle4tr6uJ8Uh1LHK95cVE4S4i0+rtG0rvW5558/PDjl76h0aAdG/5o5vKx+/4VG95cVG4iwhwq7c8+47Ps5N6y3/l0Sr1lpcIhbtIRA0MjfDx+V6aMkF+tC3Flaze8rqaGC/uWk9dTYxdVavVW15i9N0SiYh0b3lqvL/84/NXGRxJ376/qWI5ex5aO35V/qB6y0uewl0khEZHnTPJzL7lmTc/265kesvnz2PnxjJ+7Qu11NfEqKuJUaHe8tBRuIuEwI2bwxzv6Bm/4/NYe4prAxN7y//DY9XU15SrtzwiFO4iJWh83/LW7ql7yx9Wb3nUKdxFitxYb/nYVfnRthQXeyf2lv/G0w9Sr95yyaJwFykyY73l6TDvntBbvr5sCfHacuqrVxOvLVdvuUxL4S5SQO5O65W+8S6WptaJveXb16V7y8eGUKxfrd5yCUbhLpJHY73l2Uss6i2XuaCfHJE5NNZbfrQ9RVNr94Te8to1y9jzUGbf8lr1lktuKdxFcmR01DmbvD6+D0tz28Te8s9lesvrMkss6i2XuaRwF5mlG2P7lo/dvp/VW75meXZveYwdG8rUWy55pXAXCehCT3/WwOZuTl28xsioj/eW//zO9eM7JNasUW+5FJbCXWQKwyOjnLp4bcISy+Te8l/f84B6y6VoKdxFyPSWd2T2YWlLcbyjZ0Jv+diw5vqacratU2+5FD+Fu0SOu9N2pW98rby5rZszl9RbLuESKNzN7DngD4D5wJ+6+/+e9Ho18OfA6swxr7r7oRzXKjIrA0MjnLjQmx4Nl3nj8/L1dG/5yiULqK+J8Qs711NfG+PhjatZvljXPFL6ZvwpNrP5wOvAs0An8IGZHXT3k1mH/U/gTXf/IzPbDhwCauegXpEZXb5+c8I0oZ929k7oLX9qi3rLJfyCXKLsBlrc/RyAmb0BvAhkh7sDqzIflwEXclmkyHTGesvH9mE52paidVJv+dfH9i2vjlG5Ur3lEg1Bwn0D0JH1uBN4bNIxvwP8s5n9JrAc+FJOqhOZpG8ws295ZgDF0bYUVyf1lu/fXU28NsZn15exZKF6yyWagoT7VP9n9UmP9wM/cvffN7PPA39pZjvcfXTCJzJ7GXgZoLq6ejb1SsRcGNu3PPPr5MWrjIymf/y23LeCn9u5fryTRb3lIrcECfdOoCrr8UZuX3Z5CXgOwN3fN7MlQAWQzD7I3Q8ABwDi8fjkfyAk4sZ6y5vbusdvFrqQ6S1fuvBWb3ldTYy6qhhly9RbLjKdIOH+AbDZzDYB54F9wK9OOqYd2Av8yMy2AUuArlwWKuHT25/et/xoW4qm1om95esyveUvq7dcZFZmDHd3HzazV4C3SLc5/tDdT5jZa0CTux8Efgv4EzP776SXbL7u7royl3FjveXZW92eSV7DPd1bvm3dSn7l0SrqMkss6i0XuTdWqAyOx+Pe1NRUkK8tc+/mcNa+5a2395bXVcfG92F5uEq95SJBmVmzu8dnOk5/oyQnxnrLj2auzCf3lj+5pZJ4TTn1NTE2r1VvuchcU7jLXRsddVq6ro/f8dnc1q3ecpEio3CXGY31lh/NWi/P7i2vy/SWj+1brt5ykcJTuMttLvb2Z12VT9Vbvo76zBJLrXrLRYqSwj3ihkdGSXx6jabWbprbe2hu7b6tt/xbTz1Afa16y0VKicI9Ynr7hzjWfuuq/HhHD32DE3vLv1ETI15TztZ1K1mo3nKRkqRwDzF3p727j6bW6XvLf6l+I/W15eotFwkZhXuIpHvLr9I8Phquh8vXbwK3est/buc64uotFwk9/e0uYVcm7Vv+0fleBofTveU1a5bx5JaKzKZa5eotF4kYhXuJGOstz77j82eXbwDp3vIdG1bx9X9XS111+q5P9ZaLRJvCvUj1DQ7zYUfvhB0SJ/eW78vM+VRvuYhMpnAvEhd7+ydclZ+4cKu3fPPadG95XXWMeG25estFZEYK9wIY6y3P3iHxfE8/AEsWzrvVW565fV+95SJytxTueXB1YIijWZtqZfeWf2bVEuprY/zXL26ivibGtnWr1FsuIvdM4T4HLl0d4L2WyzS1pWhuvdVbPs9g+/pV473l9TUxNqi3XETmgMI9x3r7h/jy99+lt3+IlYsX8EhNLLMXS4xd6i0XkTxR0uTYu2e66O0f4gdfq+PL2z+j3nIRKQgt7uZYYyLJ6mULeVbBLiIFpHDPoZFR5+0zXezZUsl8BbuIFJDCPYeOd/TQfWOQp7euLXQpIhJxCvccakwkmT/PeGpLZaFLEZGIU7jnUEMiSX11jNXLFhW6FBGJOIV7jlzs7efkxatakhGRoqBwz5HGRBcAe7cp3EWk8BTuOdKQSLJh9VI2r11R6FJERBTuuTAwNMJ7LZfZu22tdmsUkaKgcM+BI+eu0D80ovV2ESkaCvccaEwkWbJwHp+/f02hSxERARTu98zdOZxI8sSDFZqGJCJFQ+F+j1qS1+lM9WtJRkSKisL9HjUkkgA8/ZDCXUSKR6BwN7PnzOy0mbWY2avTHPPLZnbSzE6Y2V/ltszidTiRZNu6VazX0A0RKSIz7uduZvOB14FngU7gAzM76O4ns47ZDPw28AV3T5lZJC5je/uGaG5L8c2n7i90KSIiEwS5ct8NtLj7OXcfBN4AXpx0zDeA1909BeDuydyWWZzeOdvFyKjzjNbbRaTIBAn3DUBH1uPOzHPZtgBbzOw9MztiZs9N9YnM7GUzazKzpq6urtlVXEQaE0liyxayqypW6FJERCYIEu5T3XLpkx4vADYDe4D9wJ+a2erbfpP7AXePu3u8srK0t8UdGXXePp1kz0NrNZhDRIpOkHDvBKqyHm8ELkxxzD+4+5C7/ww4TTrsQ+t4R4pU35BaIEWkKAUJ9w+AzWa2ycwWAfuAg5OO+XvgaQAzqyC9THMul4UWm4axwRybS/t/ICISTjOGu7sPA68AbwGngDfd/YSZvWZmL2QOewu4YmYngUbgf7j7lbkquhg0JLqor4lRtmxhoUsREbnNjK2QAO5+CDg06bnvZn3swLczv0LvQk8/py5e5bef31roUkREpqQ7VGeh8XS601MtkCJSrBTus9CYSLIxtpQHNZhDRIqUwv0uDQyN8K8tl9m7VYM5RKR4Kdzv0vvnrjAwNKoWSBEpagr3u9SYSLJ04Xwe12AOESliCve74O4cPpXkCxrMISJFTuF+F84mr3O+p19dMiJS9BTud2F8MMdW3ZUqIsVN4X4XGk4l2b5uFevKNJhDRIqbwj2g3r4hmttTWpIRkZKgcA9obDCHWiBFpBQo3ANqOHWJ8uWL2FV12zb1IiJFR+EewMio8/aZLvZsqdRgDhEpCQr3AI61p+jRYA4RKSEK9wDGBnM8uUUtkCJSGhTuATQkksRrYpQt1WAOESkNCvcZnO/pJ/HpNfZu05KMiJQOhfsMGhMazCEipUfhPoPGRJKq8qU8UKnBHCJSOhTudzAwNMJ7n1xm79b7NJhDREqKwv0O3v9EgzlEpDQp3O+gITOY47FN5YUuRUTkrijcp+HuNCSSPLFZgzlEpPQo3Kdx5pIGc4hI6VK4T2N8MMdDCncRKT0K92k0JC7x2fWr+EzZkkKXIiJy1xTuU+jpG6S5TYM5RKR0Kdyn8M6ZLkYdtUCKSMlSuE+hIZFkzfJFPLxRgzlEpDQp3CcZGXXeOdPFUw9pMIeIlK5A4W5mz5nZaTNrMbNX73DcV83MzSyeuxLza2wwx96t9xW6FBGRWZsx3M1sPvA68DywHdhvZtunOG4l8N+An+S6yHw6nEiyYJ7xxS0VhS5FRGTWgly57wZa3P2cuw8CbwAvTnHc7wLfAwZyWF/eNSaSxGtjrFqiwRwiUrqChPsGoCPrcWfmuXFm9ghQ5e7/mMPa8m58MIeWZESkxAUJ96neVfTxF83mAd8HfmvGT2T2spk1mVlTV1dX8CrzZPyuVLVAikiJCxLunUBV1uONwIWsxyuBHcDbZtYKPA4cnOpNVXc/4O5xd49XVhbfsOnGRJLq8mU8ULm80KWIiNyTIOH+AbDZzDaZ2SJgH3Bw7EV373X3Cnevdfda4Ajwgrs3zUnFc6R/cIT3Wi7zzNa1GswhIiVvxnB392HgFeAt4BTwprufMLPXzOyFuS4wX94/d5mbw6PackBEQmFBkIPc/RBwaNJz353m2D33Xlb+NSSSLFs0n8fu12AOESl9ukOVzGCOU0meeLCCxQs0mENESp/CHTh96RoXege0JCMioaFwRy2QIhI+Cneg4VSSHRtWcd8qDeYQkXCIfLinbgxytD3FMxqnJyIhEvlwf/esBnOISPhEPtwPn9JgDhEJn0iH+/DIKO+c6WLPQ2uZp8EcIhIikQ73Yx099PYPsXeblmREJFwiHe6HT6UHczyxWYM5RCRcIh3ujYkkj9aWazCHiIROZMO9M9XH6UvXtCQjIqEU2XBv1F2pIhJikQ33hkSSmjXLuL9CgzlEJHwiGe79gyP8+JMrGswhIqEVyXD/8ScazCEi4RbJcB8bzLF7kwZziEg4RS7c3Z2GRJIvbtZgDhEJr8iFe+LTa1zUYA4RCbnIhfv4YA5t8SsiIRbJcP/chjLWajCHiIRYpMI9dWOQY+0p3bgkIqEXqXB/50x6MMdehbuIhFykwv1wIknFisV8bkNZoUsREZlTkQn34ZFR3jmdZM9DlRrMISKhF5lwP9rew9WBYS3JiEgkRCbcDycusXC+BnOISDREJtzHBnOs1GAOEYmASIR7R3cfZy5d112pIhIZkQj3xtPpu1IV7iISFZEI94ZEkto1y7i/ckWhSxERyYtA4W5mz5nZaTNrMbNXp3j922Z20sw+MrPDZlaT+1Jnp29wODOY475ClyIikjczhruZzQdeB54HtgP7zWz7pMOOAXF33wn8HfC9XBc6Wz9uucKgBnOISMQEuXLfDbS4+zl3HwTeAF7MPsDdG929L/PwCLAxt2XOXsPpJMs1mENEIiZIuG8AOrIed2aem85LwD9N9YKZvWxmTWbW1NXVFbzKWXJ3GhNJvri5kkULIvH2gogIECzcp7pX36c80OxrQBz4valed/cD7h5393hlZWXwKmfp1EUN5hCRaFoQ4JhOoCrr8UbgwuSDzOxLwHeAp9z9Zm7KuzdjLZB7ts79PyQiIsUkyJX7B8BmM9tkZouAfcDB7APM7BHgj4EX3D2Z+zJn5/CpS+zcWMbalRrMISLRMmO4u/sw8ArwFnAKeNPdT5jZa2b2Quaw3wNWAH9rZsfN7OA0ny5vum8McqyjR+P0RCSSgizL4O6HgEOTnvtu1sdfynFd9+ydM0ncYe82hbuIRE9oW0gOn0oP5tixXoM5RCR6QhnuwyOjvHumi6c1mENEIiqU4d7clkoP5tCSjIhEVCjDvSGRzAzmUAukiERTaMN996ZyViwO9H6xiEjohC7cO7r7OJu8rl0gRSTSQhfuDQkN5hARCWW4b6pYzqaK5YUuRUSkYEIV7n2Dw7x/7oqu2kUk8kIV7u9pMIeICBCycG9IJFmxeAGP1mowh4hEW2jC/dZgjgoN5hCRyAtNCp68eJVPrw7wtJZkRETCE+6NmRZIbfErIhKicG9IJHl4YxmVKxcXuhQRkYILRbhfuX4zPZhDSzIiIkBIwv2dM13pwRzackBEBAhJuDckklSuXMxn168qdCkiIkWh5MN9aGSUdzSYQ0RkgpIP9+a2FNcGhrULpIhIlpIP91uDOSoKXYqISNEIRbg/tmmNBnOIiGQp6XBvv9JHS/K6NgoTEZmkpMO9IXEJ0GAOEZHJSjvcT3dxf8VyajWYQ0RkgpIN9xs3hznyiQZziIhMpWTD/b2WywyOaDCHiMhUSjbcG0+nB3PENZhDROQ2JRnu7k5DIsmTWzSYQ0RkKiWZjCcuXOXS1Zvau11EZBqBwt3MnjOz02bWYmavTvH6YjP7m8zrPzGz2lwXmq0xkcQM9ijcRUSmNGO4m9l84HXgeWA7sN/Mtk867CUg5e4PAt8H/k+uC83WcDrJzo2rNZhDRGQaQa7cdwMt7n7O3QeBN4AXJx3zIvDnmY//DthrZnOyReOV6zc53tHDM7pqFxGZVpBw3wB0ZD3uzDw35THuPgz0AmtyUeBkb5/ODObYpnAXEZlOkHCf6grcZ3EMZvaymTWZWVNXV1eQ+m6zaulCnt1+nwZziIjcQZCtFDuBqqzHG4EL0xzTaWYLgDKge/IncvcDwAGAeDx+W/gH8ez2+3h2u/ZuFxG5kyBX7h8Am81sk5ktAvYBBycdcxD4z5mPvwo0uPuswltERO7djFfu7j5sZq8AbwHzgR+6+wkzew1ocveDwJ8Bf2lmLaSv2PfNZdEiInJngSZcuPsh4NCk576b9fEA8Eu5LU1ERGarJO9QFRGRO1O4i4iEkMJdRCSEFO4iIiGkcBcRCSErVDu6mXUBbbP87RXA5RyWUwp0ztGgc46GeznnGnevnOmggoX7vTCzJnePF7qOfNI5R4POORrycc5alhERCSGFu4hICJVquB8odAEFoHOOBp1zNMz5OZfkmruIiNxZqV65i4jIHRR1uBfbYO58CHDO3zazk2b2kZkdNrOaQtSZSzOdc9ZxXzUzN7OS76wIcs5m9suZ7/UJM/urfNeYawF+tqvNrNHMjmV+vr9SiDpzxcx+aGZJM/t4mtfNzP4w8+fxkZnV5bQAdy/KX6S3F/4EuB9YBHwIbJ90zK8DP8h8vA/4m0LXnYdzfhpYlvn4W1E458xxK4F3gSNAvNB15+H7vBk4BsQyj9cWuu48nPMB4FuZj7cDrYWu+x7P+UmgDvh4mte/AvwT6Ul2jwM/yeXXL+Yr96IazJ0nM56zuze6e1/m4RHSk7FKWZDvM8DvAt8DBvJZ3BwJcs7fAF539xSAuyfzXGOuBTlnB8bmZ5Zx+8S3kuLu7zLFRLosLwJ/4WlHgNVmti5XX7+Yw72oBnPnSZBzzvYS6X/5S9mM52xmjwBV7v6P+SxsDgX5Pm8BtpjZe2Z2xMyey1t1cyPIOf8O8DUz6yQ9P+I381Nawdzt3/e7EmhYR4HkbDB3CQl8Pmb2NSAOPDWnFc29O56zmc0Dvg98PV8F5UGQ7/MC0ksze0j/7+z/mdkOd++Z49rmSpBz3g/8yN1/38w+T3q62w53H5378gpiTvOrmK/c72YwN3cazF1CgpwzZvYl4DvAC+5+M0+1zZWZznklsAN428xaSa9NHizxN1WD/mz/g7sPufvPgNOkw75UBTnnl4A3Adz9fWAJ6T1YwirQ3/fZKuZwj+Jg7hnPObNE8cekg73U12FhhnN29153r3D3WnevJf0+wwvu3lSYcnMiyM/235N+8xwzqyC9THMur1XmVpBzbgf2ApjZNtLh3pXXKvPrIPCfMl0zjwO97n4xZ5+90O8oz/Bu81eAM6TfZf9O5rnXSP/lhvQ3/2+BFuDfgPsLXXMezvlfgEvA8cyvg4Wuea7PedKxb1Pi3TIBv88G/F/gJPBTYF+ha87DOW8H3iPdSXMc+HKha77H8/1r4CIwRPoq/SXgm8A3s77Hr2f+PH6a659r3aEqIhJCxbwsIyIis6RwFxEJIYW7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSE/j/dKLk2OSddlwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c7495c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC curve using y_pred_class - WRONG!\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_class)\n",
    "plt.plot(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7809621710526315\n"
     ]
    }
   ],
   "source": [
    "# AUC using y_pred_class - WRONG!\n",
    "print(metrics.roc_auc_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use **y_pred_class**, it will interpret the zeros and ones as predicted probabilities of 0% and 100%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross-validation\n",
    "\n",
    "## Review of model evaluation procedures\n",
    "\n",
    "**Motivation:** Need a way to choose between machine learning models\n",
    "\n",
    "- Goal is to estimate likely performance of a model on **out-of-sample data**\n",
    "\n",
    "**Initial idea:** Train and test on the same data\n",
    "\n",
    "- But, maximizing **training accuracy** rewards overly complex models which **overfit** the training data\n",
    "\n",
    "**Alternative idea:** Train/test split\n",
    "\n",
    "- Split the dataset into two pieces, so that the model can be trained and tested on **different data**\n",
    "- **Testing accuracy** is a better estimate than training accuracy of out-of-sample performance\n",
    "- But, it provides a **high variance** estimate since changing which observations happen to be in the testing set can significantly change testing accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "\n",
    "# define X and y\n",
    "feature_cols = ['Pclass', 'Parch', 'Age', 'Sex_Female', 'Embarked_Q', 'Embarked_S']\n",
    "X = titanic[feature_cols]\n",
    "y = titanic.Survived"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7937219730941704\n"
     ]
    }
   ],
   "source": [
    "# train/test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "\n",
    "# train a logistic regression model\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "# calculate testing accuracy\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7802690582959642\n"
     ]
    }
   ],
   "source": [
    "# train/test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=2)\n",
    "\n",
    "# train a logistic regression model\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "# calculate testing accuracy\n",
    "print(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "res=[]\n",
    "for i in range(100):\n",
    "    # train/test split\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=3*i)\n",
    "\n",
    "    # train a logistic regression model\n",
    "    logreg = LogisticRegression(C=1e9)\n",
    "    logreg.fit(X_train, y_train)\n",
    "\n",
    "    # make predictions for testing set\n",
    "    y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "    # calculate testing accuracy\n",
    "    res.append(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f96c70eefd0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsvXmcJGd5Jvi8mZF3VtbdXaW+pW6puxESyG1xCHFpsIVmAHvtmUE2a8Ow4ANpd1g8XvCyWMZmsfHY+ED2bzEzxpZ3pdEPG1tgCXFIHBICJKELqbrVh1R9VXXXnXdGRsa3f0R8EV9cmZFXVVb19/x+/euqrDwiM7944/me932flxhjkJCQkJC4NBDZ6AOQkJCQkFg/yKAvISEhcQlBBn0JCQmJSwgy6EtISEhcQpBBX0JCQuISggz6EhISEpcQZNCXkJCQuIQgg76EhITEJQQZ9CUkJCQuIShh7kRENwP4cwBRAF9gjP2h6++7AfwdgBHzPh9ljN1PRDEAXwBwnflaf88Y+3Sz15qYmGB79+5t931ISEhIXNJ48sknFxljk63u1zLoE1EUwJ0A3gbgLIDHieg+xtgLwt0+DuBexthfE9FhAPcD2Avg3wNIMMZeSURpAC8Q0d2MsZeDXm/v3r144oknWh2WhISEhIQAIpoNc78w8s71AE4wxk4xxlQA9wB4l+s+DEDO/HkYwHnh9gwRKQBSAFQA+TAHJiEhISHRe4QJ+jsAnBF+P2veJuIOAO8horMwWP7t5u1fAlACMAfgNID/yhhbdr8AEX2QiJ4goicWFhbaewcSEhISEqERJuiTz21ua85bAXyRMbYTwC0A7iKiCIxdQgPAZQD2AfgIEV3ueTLGPs8YO8IYOzI52VKSkpCQkJDoEGGC/lkAu4Tfd8KWbzjeD+BeAGCMPQYgCWACwC8B+BpjrM4YuwjgUQBHuj1oCQkJCYnOECboPw7gABHtI6I4gHcDuM91n9MAbgIAIjoEI+gvmLe/lQxkALwWwNFeHbyEhISERHtoGfQZYxqA2wA8CGAGRpXO80T0SSJ6p3m3jwD4ABE9A+BuAO9lxnSWOwFkAfwExsXjbxljz/bhfUhISEhIhAAN2uSsI0eOMFmyKSEhIdEeiOhJxlhL+Vx25EpItIFSTcOXnzq70YchIdExZNCXkGgDDz4/jw//j2dweqm80YciIdERZNCXkGgDpZoGAKjUGxt8JBISnUEGfQmJNsCDvarpG3wkEhKdQQZ9CYk2UFGNYK82JNOX2JyQQV9Cog1wpl+TTF9ik0IGfQmJNlCV8o7EJocM+hISbaCiyqAvsbkhg76ERBuwErkNGfQlNidk0JeQaAOyekdis0MGfQmJNiA1fYnNDhn0JSTagKXpS3lHYpNCBn0JiTYg5R2JzQ4Z9CUk2oCs05fY7JBBX0KiDVRlyabEJocM+hISbYAz/brU9CU2KUIFfSK6mYiOEdEJIvqoz993E9HDRPQUET1LRLcIf7uGiB4joueJ6DkiSvbyDUhIrCekpi+x2dEy6BNRFMbYw7cDOAzgViI67Lrbx2GMUXw1jBm6f2U+VgHwDwB+nTH2CgBvBlDv2dGHxAPPzeHkQnG9X3ZTgTGGv/v+yyhU1/3r2TRgjKFa54ZrMuhLbE6EYfrXAzjBGDvFGFMB3APgXa77MAA58+dhAOfNn38GwLOMsWcAgDG2xBhbd3vC3/7HZ3HXY7Pr/bKbCqcWS/jd+57HQ0cvbvShDCzE5K1k+hKbFWGC/g4AZ4Tfz5q3ibgDwHuI6CyA+wHcbt5+JQBGRA8S0Y+J6Lf9XoCIPkhETxDREwsLC229gTCoqA2rqUbCHzWTwcrPKRi8Rh+QQV9i8yJM0Cef29zT1G8F8EXG2E4AtwC4i4giABQAbwDwy+b/P09EN3mejLHPM8aOMMaOTE5OtvUGWkFr6NB0JkvsWoAnJmUwC4Y4Lasm5R2JTYowQf8sgF3C7zthyzcc7wdwLwAwxh4DkAQwYT72O4yxRcZYGcYu4LpuD7odqDKYhQL/nOTFMRhi0JfrSWKzIkzQfxzAASLaR0RxGIna+1z3OQ3gJgAgokMwgv4CgAcBXENEaTOp+yYAL/Tq4MOAyxY1TcoWzVDXZIKyFaS8I7EVoLS6A2NMI6LbYATwKID/zhh7nog+CeAJxth9AD4C4G+I6MMwpJ/3MsYYgBUi+lMYFw4G4H7G2L/26834gTNXyWCbQ+6IWqMqmb7EFkDLoA8AjLH7YUgz4m2fEH5+AcANAY/9BxhlmxsCfqJyxi/hDx7EZDALBpd3EkpE7ogkNi22fEeuxfTlSdoU9YaRm5dBPxhc3hlOxeTnJLFpcQkEfc70pabfDFb1jrw4BoIzfRn0JTYzLoGgL2WLMJDyTmtUxaAvL44SmxRbP+jXZSI3DGQitzWkvCOxFbDlg35V+p+HApd3ZO4jGBWTQAynYnI9SWxabPmgb5dsSk2/GaS80xpc0x9KKlDlepLYpLgEgr5k+mEgbRhao1pvIBWLIhGLSk1fYtPiEgj6djAz+sUk/KDKks2WqKgNpOJRxKMR+TlJbFps/aAvdlFKdhYIVdowtESl3kBSiSCuRKAzw8xPQmKzYesHfYGRSYknGFLeaY1KvYFkPIq4Ypw28gIpsRmx5YN+VbBfkFYMwZBBvzWqqqHpx6Nm0JeflcQmxJYP+mLVjqzgCYaUd1qjYiZyY4oM+hKbF5dA0Jcj7sJANme1RrVuJHITJtOXcqHEZsQlEPRFpi9P0iBwwzX5GQWjUteRjElNX2JzY+sH/bpM5IYBbzaSTUfB4HX6cSnvSGxibPmgXxWrd6TTZiAsa2XJXgNRkYlciS2AUEGfiG4momNEdIKIPurz991E9DARPUVEzxLRLT5/LxLRb/XqwMNC1umHg6zeaY2KqelLeUdiM6Nl0CeiKIA7AbwdwGEAtxLRYdfdPg7gXsbYq2HM0P0r198/C+CB7g+3fdQ0HREyf5Ylm4Hg0pdsOgpGpd5wavryAimxCRGG6V8P4ARj7BRjTAVwD4B3ue7DAOTMn4cBnOd/IKKfA3AKwPPdH277qGkN5FIx82d5kgahLgR6LvVI2GjoDKqmS01fYtMjTNDfAeCM8PtZ8zYRdwB4DxGdhTFL93YAIKIMgP8DwO91faQdoqbpyCV50JeafhDEoC+DmRfcojsVj1iaviQREpsRYYI++dzmpoK3AvgiY2wngFsA3EVEERjB/rOMsWLTFyD6IBE9QURPLCwshDnu0KjWdeRSxvx3GcyCIX42tUb7F8e5tQou5qu9PKSBArdVTsWiSEhNfyBw/EIBpZq20YfRM/zopWU8dnKp768TJuifBbBL+H0nBPnGxPsB3AsAjLHHACQBTAB4DYDPENHLAP4zgN8hotvcL8AY+zxj7Ahj7Mjk5GTbb6IZalpDYPryJA2CKOl0cnH8yL3P4He+/FwvD2mgwKdmSU1/MNDQGd5156P462+f3OhD6Rn+8qHj+MyDR/v+OkqI+zwO4AAR7QNwDkai9pdc9zkN4CYAXySiQzCC/gJj7EZ+ByK6A0CRMfa5Xhx4WNTqUt4JA1XToUQImqldt4uFQg2x6NatALblHTvo1yXT3zAsFGooqw08c3Z1ow+lZyjVNGQTYUJyd2h5ljLGNAC3AXgQwAyMKp3nieiTRPRO824fAfABInoGwN0A3ssGxLy+ptnyjqzeCYba0JFNKtbP7aJU07BSVnt9WAMDUd6Rdfobj7m1CgDg6Hxhg4+kdyjVGkjHo31/nVCXFcbY/TAStOJtnxB+fgHADS2e444Ojq9r1LQG0nEFEZIabDPUGzoycQWr5XpHwaykNlCtN8AYA5FfGmhzg8s7snpnMDC/ZuSPFgo1LBZrmMgmNviIukexpiEzCEx/s6Nm+qUklKjU9JugrunW1rLdYMYYQ6mmoabpFiPeauDvS/rpDwbmhaKBo3Nbg+2X1QGRdzYzdJ1BbehIKBEkYpHQNgyPnljEPT863eejGyw45J02g35N06Hphpq3XNqaEk/VR96RJGLjML9WRdTsupyZy4d6zMPHLuKfnzrXz8PqCoa8I4N+V+BMLBEzaqvDnqR//e2T+LNvHu/noQ0UGGOoN5i1tay1yWDLqn0xXSnVe3psgwKL6ceiICI5J3eDMbdWxc7RFLbnEpiZDxf0//bRl/G5h0/0+cg6g6rpBvFKDIimv1nBE7cJJYpELPxJenQ+f0klfXm5Jl9w7QYzsVZ6qyZzK6rxmaRixmcUV2TQ30jMr1UxlUsiGYtiJqS8s1ZWUR7Quv6yahyX1PS7BC/RTMYioTV9IzGkoqhq0PWBKEDqO/iOqFNNv3gpBH1B3gHMoN9BE5tEbzCXr2B6OIlD0zmcuFgIVT67Vqk71uoggR9XRso73aEqMn0lEqpOn+uDjAHlLZqUdKNuBvlMh0GfsxRg62v6ybhxykh5Z+Og6wwX1mqYGk7h0PQQ6g2GUwullo9bq9RRUo0Ks0FDqWasL8n0uwQP8gklgrgSTtM/KuiDxepgsoJeo+5m+m1q+sWaqOlv3aAfIVhJXCnvbByWyyrUhm4xfaB1MlfXGdYqdTR0NpAJ+JIl7/Rf09/iQZ8z/YjJ9Ft/2aI+WKw5k5JfeeY8fuaz39lysg//XDqVd0RNf3mryjvmABXeg2DIO4MXPC4F8Br9qeEk9k1kEI9GWiZzi6oGftoOol8PPybJ9LuExfTbqNOfmctbbK7gYvo/ObeGFy8Ut1wtOmf6nco7fMEmlMiWrt5JCd2SUt7ZOFhBP5dELBrB/m3ZlsnctbK9Lku1wTt/S1LT7w14BU6SyzstgrWq6Ti5UMS1u4YBwJP0yZsXgZI6eEyhG9jVO53JO3zB7hxNbVlNnw9Q4QgrF0r0HnNmY9b0cBIAcGg6h6Mt5J21ihD0B/D85RKpbM7qElUH02/NzE4uFFFvMPz03jEA3m1gvmosnIo6eEyhG/DPhft+tBvMSubnsXM0vWWrd/hQdA6p6W8c5tcqUCKEcdN64dD0EC4Walgq1gIf4wj6Ayjv8GKItNT0u4Ndpx+uZJMncY/sHQXglXf474O4PewGdhNbtCPZolTToEQI23OJrcv0Vae8k5Ca/oZhbq2K7bmk1ZHLk7nNzNdWBXlnEMs2+TFJpt8lHIncWOvt+MxcAXElglfuGAHgI++YbKHc4+3hT86tYXUDGTLX9GNR6ojBlkyjqLFMAitldWBK4n5ybq1ng1088o7U9DcM82tVTJnSDgAcnBoC0LyCR2T65R7v1B87udT1XOlSTUM0QtaAnn5iiwd9W94xbBiaf9kzc3lcuT2LYXOmrrtks1DlQb93i4Yxhv/4/zyG//bISz17znbBg1c8Gumo6ahYayCbUDCWiaHeYAPDpH7trifxgbue7MlFqFLXpbwzIHAH/fFsAqPpGF5aDK7VX63YpKqX6/PExSJu/Zsf4FtHL3b1PNxWeT0card40LcTuWGZ/sGpHOJmiWdQIreXTL9a11FSG1hsokf2G5zpx5UIYlHqqDkrk4hiNB0H4NxKbxQYY7hYqOKZM6u4/7n5rp+vqvpo+lLeWXcwxjC3VsV0Lum4PR1XmlbV9UvTP79q+Pp3K2uu1wAVYIsHfd5FyUs2VU0PZH3cl5vrg0NJBYVa/5k+f878BjaC2fJOpCMGW6xpSMcVjGWMoD8Iun6xpllVSZ958GjXrFyWbA4G8hUNlXrDwfQBw2qlmV/WWrmOkbSxg+/l+cvJWrcXkpK6Pl76QMigT0Q3E9ExIjpBRB/1+ftuInqYiJ4iomeJ6Bbz9rcR0ZNE9Jz5/1t7/QaawZnIbe6BfsxMAh0y9cFsQnHIO6qmW7YOpR4uGl4RlK9sHDvmO6BY1HAj7aRkM5tQMMqD/gBU8PB+gX93zTRml8q4u0urbL+STRn01x/cR98b9KNN5du1Sh2T2QRiUeqpvGMH/e5iQqnWQGYdpmYBIYI+EUUB3Ang7QAOA7iViA677vZxGGMUXw1jhu5fmbcvAngHY+yVAH4VwF29OvAwqGk6IgQoQoIkSOLhSaCDJtPPJhXH4uCMHEBPnfo4w3dXCq0nOCM27CqiHcg7DWQSUYyZ8s4gWDHwC8/PvWoHXnf5OP78W8cd32G7cMs7Mcn0NwR8TOK0K+gnlIhFyvywVqljOBVDJqH0VN5ZLBrrrNvaf14MsR4Iw/SvB3CCMXaKMaYCuAfAu1z3YQBy5s/DAM4DAGPsKcbYefP25wEkiWjd5prVtAYSipEcsYJ+wMKYmc9jey5hSRRupi/KL73cHnKGn+8iILXCmeUyfufLzwUyIbe8026dfrGmIRMXmH5A0H/85WX8xbfWZ04Bv/CMZeP42C0HsVxS8YXvdZ4sN+Qd+3RJKJG25w4AwB33PY9f/sIP8Mtf+AHe84Uf4tvHuksAXmqwLRhSjtuTsagl5/ph1ZR3MnGlpyXXC4XeyDvrNSoRCBf0dwA4I/x+1rxNxB0A3kNEZ2HM0r3d53l+AcBTjLF1y1jWNB3JmPEWE4rpFR9wop5cKOHAtiHr92wi5tD0HUy/h4ncwjow/e8eX8D/98PT+Mm5Nd+/q5pdspnosE4/k1CQSyqIRiiwQeurz5xftyEW/MIzlo7jmp0juGbnMH58eqWj56o3jMlgftU77VQGqZqOL37/Zby8WEatruOJ2WX8y9PnWz9QwsLcWhVEwLYhJ3dMxqJWM6Yf1ip15FIxZBLRHjP93gT9sjpA8g4Avxoi90q/FcAXGWM7AdwC4C4isp6biF4B4I8A/JrvCxB9kIieIKInFhYWwh15CFTrDSvYJ2Kc6fsvjNNLJeweT1u/DyUVh+FavmJ/qZtN0+c7lhcC/EnE6p1OqlJKagOZhAIiwmg6juUA/51qXYeq6aG8z7sFv/Dw3cdoOu6o4GgH4tQsDu7PxKWxMODE4dfedDm+9Buvxyt3DFtyhUQ4zK9VTW3eGbpaJnJFeaeHpI0z/WLXmv5gMf2zAHYJv++EKd8IeD+AewGAMfYYgCSACQAgop0AvgzgVxhjJ/1egDH2ecbYEcbYkcnJyfbeQRPUNN0K9s3mmuardayU69gzZgd9t7wjMv1e2jBwhl/T9FB+/52A5yaC/EnULqp36g0jkHOWMpaJBWr6nImtRxv8SllFNELImXN/R9KxjoN+VfUJ+h0MR+cS4ZB5TFPDKUuukAiHuXzVo+cDxk4+iOlrDR3FmoaRVNyUd3qv6Xe7+y8OWMnm4wAOENE+IorDSNTe57rPaQA3AQARHYIR9BeIaATAvwL4GGPs0d4ddjjU6rql5VtM3yegnV4qAwB2i0HfTOTy7Ttn5EM9TgSJDL9fEg9/3qCORUdzVpvyjtsSdiQdD6ze4RfLXu6UgrBcqmM0HbeaXYZTnQd999QsQAj6bXxWnDjkkkbp4PRwEnNr1YHpYN4MmF+reCp3AIPpByVy+cV2OKWY8k5v1l9DZ1gudS/vaA0dNU1fl6HoQIigzxjTANwG4EEAMzCqdJ4nok8S0TvNu30EwAeI6BkAdwN4LzNW8m0A9gP4v4joafPftr68Ex/wRC4gaPp+QX/ZDPrjTqZfb9gDF3jgnBpOtrRWbsdvv+DYTfQn6HOmf2y+4Hts9YYOJUKIRKhtecftGTKWjjdh+mbJ63ow/ZKKsUzM+n3EDPqdzEKwgn68u6DPJcIhM+hP5ZKoafpANLNtFsyZs3HdSCjRQOmWW5yMpOPIJJSOSzbdF+flkmp59HdTBspJ0HoMUAFC1ukzxu5njF3JGLuCMfYp87ZPMMbuM39+gTF2A2PsWsbYqxhjXzdv/wPGWMa8jf9bt3IFMZEbt0o2vQtj1mT6e8Yz1m18C84DVL5SBxEwOZRoGrSOXyjg8O9+zTGBqxnEqp1+6fr8eEtqA2dWyp6/1xvM0kjjSsQanxgGvJKJuwOOZuJYCQhivLpiPWwalsuq1SEMALlUDIzB03AXBnyHkvLR9Dti+iku7xjBa+4SlXh+9NIyrv7dB0OX+JZqGgpVzVO5Axg7+WrAd8F3eMMpo3qnEynm6TOrOPSJr+Hcqp2D4UncbELpqqKv5CJO/caW78i1mX5wyebp5RLGM3HHh85/5gEqXzU0t1Zf8MmFEqp1HQ/+5EKoY1wvps/fv9+wCVXTEYt2NhGq6JJ3xjKxQNM1zsTK6+BSajB9O+hzP6W1Dlg1lw38Nf3w78WSCDnTN4P+fP7STOaeXCiiWNNwPmQym+/Id415g37S7C/x28mt8qCf5nX67a+/586toVrX8bxQAceD/p7xdFdExrZVlkG/a9Q0QdNXgr3iZ5fKDmkHsIM+D8T5ah25pLFomgV9fmI/ciJcFVK+Use4GZz6VatfqGq4ZucwiOC7A1EbOuLm52MY07Wv6fPPazQdR0NnvrYSlXVk+itl1arcAYytPYCOdP2qj7zTqtnPDwVXIpcnJOfXNs53aSPBz6OwQdjakY9lPH/jF2TfQg2B6WcTUahm8UE7mDcvTPzCA9hBf+94BiUh/9cu7AEqAyTvbFaI1TuJJsxsdqnsqNwBjEQuYAeoQlXDUFJBKh5tuj3kJ/aPT6+G6gAtVDXsGE2ZP/cn6BdrGiaHEtg3nvFN5tY1HXGT6YcZNiOCn7Bpq3onuCvXsrHoc9DXdYaVch2jaVvT50xfdFsMi14lcrlEmDUTdpPZBCJkB5RLDRW1vUl0p5cNF003QQNgybh+8q0o7/BkabsSD5fgxKDPyzX3jKehs/aHD3Gs56hEYMsHfVveiQfIO6qmY26t4qjcAYChhNNeOV8xmX482pzpmwusoTM8dnKp5THmq3XsGEmZj+2TvGNKU4emc76DJtSGjphia/pqI3zTkYfpN/Hf4Yy51/MI3ChUNTR05tD0udlWJ0zfX9MPLgwIApcII+bwDyUawbah5CWr6fOLaVgSMLtUxnAqZl3ARfDz3K+ChyfKDabvJHNhwUtr+W4DMMo140rEkuk63cGu51B0YKsH/brYkeu/HT+3WoHOgN3jzi2jH9PPpRSk4oa8E1QFUqhqSMWiSMejeOTEYstjLFQ1bM8lEaH+Mv1sIoaDU0OYXSp7Fme9oVuJyXg0AsYALWSVC2dpGaF6B/Bn+ra8019Nnzdm+Wr6nQR93pwl2DBwEuHXnMUYw9efn/fYAnCJUMTUcNIyEWuFsytlPNVhV/EgwpZ3wjL9Mvb4sHzAZvp+VgxrlToy8Shi0Yi1TtvV9ed9mP5ioYbJbMJi6J3mqtznUL+xtYO+pgsduVzzc34xs0vGltG9mCxNv2Zr+kMm0wcQWLaZrxoeH6/ZN4bvHW8e9Bu6MXCEM5B+2Cvr5mtkk4plG33MxfZVzVm9Y9wWjsHyk4cv/Gb2yrV1kneWXd24gCDvdJDI5e+F7/6A5oncp8+s4oN3PYn7n5tz3M4lQhG8Vj8M/vjBY/iNf/hxW8c+yKi0qemfXi57duQczTR9w3fHWAu8yqydrlzGmHVhPrtSRsMkRAvFGiaGElapZadMn5OggSrZ3Kwwqndcmr5rUfArt1vT5ycnl3cKVQ25pGJl2IMknoLJ5m48MImXFks4s+wtkeTgz51LxZBLxfqSyOWLeyih4OC0/1g5t7wDtBP0NUTIZlo80Lr9dxo6s6qCetkG74cVwXeHIxmLIqFEOiqLPTqfx57xtMdPH/D/nL77onGxdw/GyZv+LyKmhpOhu3JfWizhQqG6Zdw925F3tIaOcyuVQKbPz+8gps8/92zCWYodBvmqhrLawBWTGdQbzBqcslCoYTIbt3cPHa7rstT0ewPGmKN6R4kQiLxMYHapjFQsikmXgVNCiSAaIRRrdTDGUDCZftpkFEG6dL5isLk3XjkBAE0lHruET8FQMtYXTd9qnkoq2DGSwlBS8VTw1DUdCTfTD1m2yd0BeedrJm6MpnT774g7rL4z/ZJX3gEMtt8J0z86V8ChqZzjtniT6p3vHV8wj8P5Wpw4iJjKJVGsaaGkvdmlMhgDLha2Rg6AE6diiGB5frUKTWe+lTuAzfT9g76KYbM3ggfWdtYgvyi/5vJxADZRXCyqmMgmBMmoc02fyC6G6De2bNDnQYvLOtxe2S/o7x5Le2ZTEpHlv1NSG9CZ0VTDt2BBW9JCzWAVV0xmMZVLWgHAD3mhLX8oqfRF0+e7iawZmA9N5Ty1+vWGjphi1um32XRUVjUHQyEijPr474h+Rb20tvWD22yNoxP/nbKq4aWlkrVL4gjaORaqdTx1ZtU4DtdnEKTpA2jJ9tfKdevYt4pfD18TYbTw2SaVO4Cg6fus27VKHSMpYy20On/9wE3xXrNvDIAR9LkFw+RQQriQdLaui7UGMnFlXebjAls46PPgLk6X92vVPrNcxq4AnTCbMEYmcklgKGmXfFXqzZk+EeHGAxN49MSSpQH63RcAckkFuWSsL5p+QWD6AHBweshjx6A2dI+mH7b8rFRreLTI0XTcI++IJ2O/5Z3lUh3xaMRjVTucirVdsvnihSIYg5UP4QjaET120vi+oxHyVDD5a/pG5VarZK6YQNwq1T7tyDt213yQvGNq+gHyDs/pdCLF8IvsdbtHEYsSZpfKWCkbFgwG0+cXkg7lHVVbN5YPbOWgX3cyfcCsQRdOUsZY04qAoaTB9HntfS4Zs76cQKYvsLk3HJjAWqWO5wJ87O22/BhyPWD6Pzy1hDvue95xG2f6Q+ZiPzSdQ7Gm4eyKXRuuanb1ThCDDYKfO6Bhr+wK+vX+yDuMMXz0H591VLWslFSMZmIe5jScimOtTQmNO5N65J2AHdH3ji8iHY/iut0jDqbPJUK3pj8d0oqBM11gcJn+fc+cx51tzEuw5J0Q6+HMchlxJYLtQ17fHaA5018V5uN2UrLJPfynhpPYNZrG6eWSla+ZEKp3OiUz6+mwCWzhoG8NRReYflxxem4vFGqo1BuBQT9rmjOJ2rvd3OEN+owZnaiczV27cwQAcOJi0ff5RavdXCrWtffOt45exBe//7IjEBVdTH+n2Qh2QdCF636J3JCavsFSnAs2l/KaWvGtPFFvSzbXKnXc8/gZfPmpc9Ztbt8djuFUDGttzu+dmcsjE49anxtHUML7kROLeO3l49iWSzqYPpcI3Ux/W87IJbUK5Jx7yBmqAAAgAElEQVTpxqOR0CWe640vPXkW//CD2dD3581ZYXxrZpfK2DWasnoc3LDr9J3PVa03UNN062LLc3XtavoTpof/7vE0ZpfKVmPWhJjI7ULTX69yTWALB31/ecep6c8uey2VRWSTimnyZDNyzvT9ErmVegMNnVkLzNZr/TsuCy5Nv1jTOnKBtF7fPHlWhWAjavri/46h7446/faajoq1hmfBZuLeQRU8kTuajvdl8thRIU+xWlY9SVygM3vlmfkCDk7nPMHGL+ifWS7jpcUSbjww4XEb5Rd0t6afUKKYyMZbMv3TS2VMZOPYOTa4HvzzaxUsl/x9l/zQDtOfXS47DBHdCCrZ5N83Z/pEhHS8PXvlecHDf/dYGqeXyhbTnxxKGMOHopGOyUxJbUh5pxfgQYYzAP6zWEXi564pwtb0bUZu1/l6v2DxfoCxEEfTscATmt8/a2r6OutO7+YaqcgwuabPa8yHkt7tbV1jdtDvoGTT7RmSikc9yTneKTmeifdU3uEn9cx83go2yyXVk8QFjBO/pDZCT+5ijGFmLo+DU0Oev/FqMHFHxPsybjwwgdF0DKuVupXPsX13vN2kRtlmcyuG2eUSdo+lzbr+wbRtmFuroqbpLa3HOSohO7QZY8ZkuwByBgRPxhMtGDiybc7EmBfsnHePpVGoaTh+wdi9T5hVf+lEc3uWZihJeac34Fd8rvUB8Az9Pr1UQoRg2SC4YWv6Nkvj+l3F5wt2D8kAmk9HKlTrSJudgjwYd+O0aQV9gWHaLd5GYM5yewkx6IvVO226R5ZVzeMO6GdKx3chE9lETw3XxBnD3PbW7bvD0W5X7vm1KgpVzZPEBQzG6B44873jC5geTuKKySxGM3Ew5h18z22VRUzlUi2Z/pnlCvaMZzCVG0ymz22PAf/GPD+Ebc5aKqkoqcEyLGC4bAJeeccv6Lc7MnFurWIxfU4Qn5hdQVyJWLmyTLxzn/5SzXsO9RNbN+jzRK7iTOS65Z3p4ZQV6NywNX2bwXP/Fb+FKmr/HNNN2uzFag4uCXUT9PlYvxWhPrxY05CMRaCYTN7qHhTlHU2o3mlRslk1JSzx+d0sJRWLWlKX9ThzhzWWjaNa1wMrmtqF2NA2M1dAQ2eGvOOj6bfrvzNz3kziTnuZPuAkEQ2d4dETi7jxwASIyO5MNnddhapdAeZGszUCGLvW86Y/1NRwAhcKtZ59fkBvXE/F418JmJEsQtV0y+qj1eu3qtwBgFiUECGv9w7vy+Alm4DRSxJ0odF15mDsJfP85x7+/BiePbuKyWzCKhZod/cgoqQ21s1hEwgZ9InoZiI6RkQniOijPn/fTUQPE9FTRPQsEd0i/O1j5uOOEdHP9vLgm4EHmYTA9BOxqJPpN2nrBgxWXFYbWC0bxkrJWBSRCFlBzY280GHL0azjUqzb5sG/m65cX3mnqlnsHrCbU8RhIoa1criSzXf85SP4i28dB2B0SVbruqeTkF9YxM+In4wTZjDsVdmmeJE8OpdHvlKHzrw1+oD9vYRt0OJNbFdNeZk+4KwGm5nLI1/VcMN+oylv1OVBJJbnujE1nMRquR44e/nsSgWMGQFnajiFhs483b6dYq1cx5E/+Ab+7vsvd/U84hoPGpcpoiIUWrSSRc5YubdgTZ+IkIxFPTYrgUw/IEB/6cmzeN2nH7Iu0vxiJmr6gLGeJ4SGTkPe6VDTr2nr1o0LhAj6RBQFcCeAtwM4DOBWIjrsutvHYYxRfDWMGbp/ZT72sPn7KwDcDOCvzOfrO2ym70rkCoFofq1q2Rr7gVe8nF+rOk5WY86md9HYyTqB6eeSWCqpvp2CDqaf5Ey/+6AvJhCLNWdteCRiN51xiIZrrUo2z65U8MOXDPfQct3fM8TPvpa///GscaL0StfPC4m6mfm8FXCCErniY1phZq6A3WPpQL1VlHdOLRollQfNC4Tbg6gZ0+d6cRDbPy0w3elcb6dtzS4bQ38++80XO54h7D6eMJOw+AVucihhjiUNDpizS2UQwVNB5UZC8c7J5UUNw2ln0A/aXbwwl8dapW455PKL2Xbzc0/GothuVlxNZu01lm3ynM1g7CwaAyfvXA/gBGPsFGNMBXAPgHe57sMAcDo0DOC8+fO7ANzDGKsxxl4CcMJ8vr7DL5ErToXSGjouFmrWFdwPXK+bW604TtZUgL2yWM/PwSt4Lua9zCwv1G1bTL8LKwZ+IomaarFa9wQtY4EaJ7jW0KEzeA3XApKdakPHzFwBjLFAS1irwqkmMn0e9E2m36OyTf6ZH9kziqNzBSvg+JVsjrTpqT8znw+UdgBzPZlB/7Rp3MeZoNuDSJQI3Zhu0ZXLTQF3jaVbVoS1Cx6sV8t1/PW3T3b8POLxhNH0OSGYsEhAk6C/XMJULumYXOaHZCzqdTY1ZxgMCWu0mT06/w64fQr/fMQ4wa0g+LEbz9mZvMN3vIMm7+wAcEb4/ax5m4g7ALyHiM4CuB/A7W08ti/wS+QmhDr9xaKKhs6sk8gPFtNfdTH9gDmb7nF4gN1x6VdxYTB94762pt852+ILXuyG9dPcs0mblXBrYEveaaLpaw1Di1+r1DGfrzYJ+t5eBivoZ3rM9KuGbe4rLhvGS0slK5nbjOmHGZlYURt4ebFkMXc/iEF/dqmMbUMJy5SN5xS4/06+WrckQjdajU08vVxBOh7FZDYRupkrLC6Yu4s3XjmJ//7oS5aZWLuYW6tiOBVDhLxme34oC4l9oPl6OL3UXIblSLrkW8AYlZhLxhwlt83kHb7b4pVY/PMR4wS3ghCDfjrRXhkoB/8cBq1O368bwp1FuhXAFxljOwHcAuAuIoqEfCyI6INE9AQRPbGwEG7MYCvYdfrukk3jdh6E+dbaDzxYXixUQzP9WJQcFxr7hPaepMZgFuM1bE2/t9U7hapmXbw4sgnFYsic0XOmH2si74jsf2YuL9gqOwMZl3uc8o6OaISsZGqvgj43wjs0nQNjwA9fWgbgr+nb1TutX/vFCwXoLDiJCzh3jrOuzu5UPIpkLGIz/YrmqdHnaDUg/bRZrskTxL1s0JpbqyIWJXzq564GAPzpN17s6Hnm16q4bCTl243tB04CuNFhsxyP+7MNgiHveDX9EVclVzMpZt78PLhD7txaBaPpmONizS9Aokljts2KIA5rxvQgafow2Pku4fedsOUbjvcDuBcAGGOPAUgCmAj5WDDGPs8YO8IYOzI5ORn+6AWslev43EPH8dxZw/KAa/cJN9M3ZR+/K7gbPFhyszWOoK0cD0Bi+3/QCW205dtMP6GY1r/daPqqP9MfcrGIIYHp8+DOxyU2Y/ribTNzhZbyjtjLUKk3kFQitrVth0kvN/IVY7gND87fN7flftU7StR4fVHeOTqfx6M+Tqg8ietXrskhavqnl7weTmNCADTsOfxP7HRcwXAq1kTesZkuEbVlx9wK82tVbM8lsWssjfe+fi/+8cdnfecot3wes4FpNOP1XfIDJ02TltznHzDLqoaFQq1pYxZHIhb12DCIvjsc6biCmqZDc0mYhuRbxdsObwdgsP35tapVucOxx4fpN9s9NMN6T80CwgX9xwEcIKJ9RBSHkZi9z3Wf0wBuAgAiOgQj6C+Y93s3ESWIaB+AAwB+1KuDF0ER4L9+/UV813S1DOrIVS2mz7W64OSQGCzFARrpAKZvsDkvqx5KKJ6TtKbpUBu642LSrb0yT2K5Szb9mD5P5PJGJS7vxMzg76fpO4N+3rZ4CJB3Kq5EbtKcKAb0kOnXjAvtrtE0MvEoXl4qIxmLOLzvRbi7cj99/1G874uPO2QNxhj+6cfnMJFNYNdoMMPk8k613sB8vuqx/R3NxK1EYr6qYchnzB/H9HAS51a80oque/2hptoYvNIKYg36h968H0MJBX/4wNG2n8cIjkmMpmMhNX1T3hlqrumfXzXeZ6skLgAkA5i+e4eVCWiwXCjWoDPghv0TmB5O4pETC5hbq3ryftftHsX0cBKvuMwmBJl4FPUGa3vWQWmdB6gAIYI+Y0wDcBuABwHMwKjSeZ6IPklE7zTv9hEAHyCiZwDcDeC9zMDzMHYALwD4GoAPMcb64qubS8awczRlDQip1RsgspkrYNfpM8Ywv1ZFXIn4NvFwiMFSDM5BQZ8zfTemfLoo/fT/XKpz0zWtoVuBmjMtxpg1H1eEuL2tu+QdInJo1SJEvfTofMH6DNwt5H6mdNW6jmQs2vGM0iDwC20kQrjK7Jz1Y/kchv+O/Rm/MJeHqukOWePhYxfxw5eW8b/dtD/Q6wUwPrNaQ7dKCt0SxGhIpg8YksGsz8Cdi4UaapruGOc53WOmz5nscDqG2966H98+tmDtmMKgWm9gqaRiOpfEaDoeqiTWkndaaPrc42Yym/D9uwg/Tb9U8zqbBg1SEZO2Nx6YwCPHF3F+teJRA3aNpfHYx27C3gn7O+nUf2e9h6IDIev0GWP3M8auZIxdwRj7lHnbJxhj95k/v8AYu4Exdi1j7FWMsa8Lj/2U+birGGMP9OdtGBAHf/MBKqLUwh031YZuXcGbeViLwVIMzmmfjlPAYHO+HZfDSczn3VOUvHXbQ13YK/Nt7Wja6C3gRlOazrxMP2kzfR7cY+LFMeqdOwDYQX/veBqnFopYMgNaENMv152J3GTMnlHaK/8d8ULLpRg/PZ9DZPqLxRoWCjVMZBOWrNHQGf7wgaPYN5HBu6/f3fS1+c6R2x67vd4NqcNM5PowThF7xtM4s1z2eC/NuqqCALv3I6zHTRAYYx4m+yuv24sdIyl8+oGjoX2geGXa1HASY5lwmr6b6QeRAMvNcqh10HeXZAPwJT3pgDXIL6RTuRTecGAS+aqGlXK9ad6PI9MhmVnv+bjAFuvIPTQ1hFMLRSvgiUlcwDkcXfTTCIJ49RWDczrm77NRqNYdMhDHtI+3ip9lQzf2ylzP53LVSlm1/V7cmn5CQVE1zN1Ul7zDf24m71y7awQ6g2Vn7DFc44ncmlfe4R3NvXLaFF1ND/Kg34Tpj5ieOIBt0vZ773wFhhIK/uiBo/jHJ8/ixQtF/JefvcpxIfSDsSNq2B2jHk0/JjB9L+MUsXs8g5pmlBGL8BvnOZ1LQm3ooe0OgrBarqOm6VYNOmCw5Y/8zJV47twavuqa8RsEvoudHk5Zmn6rCxI/fziDDyqhFC2MW8GvZLPgK2/6r0GR6b9h/wQ4H2yW9+PINHHfbQa+G5beOx3i4HQOOgOOXyg65uNyiGZic/lK0xp9wG5kAvyZvpsJ8aSiG1PDKVws1BxGX3b3rnhh6dxemS/2y0wfoeWS6rFV5sgmFTBmMHGrZDPqCvpNqneuMS2jn5w1gn7KVYbIfVBEzbSqNayOZqMNvnum7/aoP2wmc8MyfZ6wfO3lY/jQW/bj4WML+P2vvoBX7RrB26+eavn68ahxcTy9XEY2oXjKREczcaxV6tAauqMnww88qHNmz3F6uYxohBxNhK2qfcLC3W3K8XOv2oFD0zn88YNHmzZNuZ9najiBsXQc9QZryXj5eg3D9JUIWT0WzZCMOXeojDHfQoagkYnzaxUklAhG0jGMZeK4+rJhAN7Pxw+dDkfnx5AeJE1/M4Fv72fm8gbTjznfnjg8+cJazZOV9wP/MsUTlpcoVl0nRJCmPz2cBGO2PsnvCzgvJsbIxA7lHfMk2jFiLNCVUl2wVXaXrNllk37yTqCmb77GgW1ZpGJRzK1VkYlHPbp3JGLY11ZcJZv84mAYsnUf9Kt1HfUGsxg0t0sYa5KnGU4bmr7hoFnA5FAC49kEfvX1e3HZcBKFmobfueVQqNF1/HOaXSr5jtzkF4GFYg3Vuu4JPiK4fOPW9WeXyrhsJOn4fvi67VbXt+QMV1CLRAgfe/tBnFmu4N4nzrZ8njnreVJ2U5pQTPDI8UX853uecrD/stqAYhIApYm//UKhhvFsvGluhSOhOJl+WW2AMS/pCdLf5/M1h+T7hgOGpUYYecedJ9B1htvvfsrq7A3CoJZsbhrsHksjFYtiZj6PmtbwkXeM3+fXqlAbOqZyrbeMNtN3JnIBZ6JSa+goqQ1f3daPmdmavpjIjXVcsllxM/2yioLZdeuuDOAnQaGqeap3AHjcIzk400/Fo1bSNEiLTMejzpJNtWH1Lxht8N3LO26JLJtQ8Il/dxj//siuwMcMp2JQTc+gmbm8RRSSsSj+8peuw++98xW43pyF2gpW0A/wcOIyE5d/msk7O0ZTiEbIslzgePFCAVdMZh23WQ1aXdbq+3WbcrzxyknsGU+HSujOr1UxlFDM3Y7xXYj+O/f/ZA7//PR5h0VCWW0gFY9a/vbB8o4aStoBDKYvvoZdXeau3vGfdDW/5kza/s+v3YPb3rLf8/n7wW09crFQw1eeOd90Rja/fyoWRTTERa1X2FJBP2pWcBydK6BW1x1NUoAd2F42T6wwTD/LO2ZFeccqSbQXKl9gYdvsCz6OnEMJBdW63nbZl3gsPOivlFTrouTOM2SFrajdnGUvulaafjwaserig4O+4tT0tYaVSA/yLmoXfq6m/+kN+3D1juHAx/Ca7cViDScuFnFI8Mr/qT2j+NXX7w39+vFoFNW6jrPLFd/mIc70eSBvJu/EohFcNpJ0MH1V03FyoejpFZjIJhCNEC50zfQriFBwZcxhoTCi+fNUrWDpNpoD7Pcv9kdUhMEhzZqlFou1NoJ+FFWtYe0o+K7Zy/SDNX2xhPuykRR+62evCrXLsKvSjOfkMl2rnXvRZ8Z0v7Glgj5gdFDOzOeNIBOQyOU+KWG0uiEfpm/X+dpfqMXcfU7s6ZzXiiFfrSNqyiAc3VgxcKY/NZwEEdf0jefxJrJMT/1qsLzjN2iE66VxJWIFoqAF62ZvNVHe6dCnxA0/V9NW4Ba7T51ZhdrQmzZftUJciaBSb0Bt6J7KHUBg+uZ8Wz/pT8SesYy1NgHg5EIR9QbzHGM0Qtg+lOha059bq2LbUNKy3Xbj4FQOLy+VWkpxc3k76LuN5gD7/Yv9EZV6wyH3Ba2HxUJ7QZ8x21rEImIBmr5ISnSd4UK+6khqtwP3cHR+8W61c1/vUYnAlgz6OayW65hdKnsSufwiwJl+mKDPr+BiYEn5uEj6sU6OXEpBMhZxMX2jmkPUgbsZpMK1zGzC6O5cKaueUYnu91Ss1a3gnnDJO34lm/wCkVAilidNkBbpDvqVulPe6YW1sp+raStwpv+DU4bWerCJzUIriJKYuzELsAMgl3daHefu8bRVrQMIXcE+k7uMMuDuTNfm81Vsb3IOHJoeAmPAsRZsf15o8HIbzdUbutVgJfZHGPKO8XmkE4pvhzZjDItF1WF30AxWzs7MtRUDmH46HgWRU9NfKqmoN1iomOAHt2TEdzetzuWyur62ysAWDPo8GJ1dqXiDvhl0ZpdKUCJk2fw2A18wYuDkiVwxqOV9SjA5iAjTwymHButXt81/70TX50w/FYta7f+FAMlpyEfTD5PIVQWm30rTdydrq/WGVdVjMLteaPrevEgrcB+WH5xaQixKofTaIIjry0/e4a9la/qtmH4aK+W69f3PzBUQVyLYN+G9oEwPp3B6uYznzq7hubNrHRmlza1VLatmP/AdRjOJp2661XKpdCihQImQxfTPrVSsgS+rDqavCfJOkFW5IT9OZIOrsURw+ZATIGun61qjRGROunLarAPhyjN9X9s1cN1i+i2q8Yo1Tco73eIqgRW55R1elvjyUhnbc8lQyZPtuQS2DSUc9035JHLtGaj+QXAql/Rl+iK6YfoV1U6y8lrpYlWDEiHPxU/sirXknQC7ChG8fC+uRDCciuHyiUygHiwyfcaYVadvvH6vNf3wQZ8z/VMLJezfNtSyFr8Z+HpSIuTLELntBNd3/cp5RfBkMGeJM3N5XLk96yu/7BpL48xyBe/43CN4x+cewU1/8p22K6IuCFq8H3aMpJBNKFaXux8WCjUwZle4EJHDf0fcuTjkHUHTD5L7FoTh42GQ5H04ZjK3ELDTBfj6tF/T7jXoLOgbFxLbafN0SE2/VGusu7yzvq+2DhhOxbBjJIVzqxVPIpcz/bVKHfu3hWN4v/Hm/fiPR5ydmRk/ecdnQo+I6eGk5QAJOKdmcXAJqZNafc70jWHscZxfrVi+O+5SQqt7sKpZAcVTp99E0+cX07v+l9d4HDY50nFbwlFNz35+sUz3SNMv+PQ6tIIo0zVz0AwDLu/sHE0F6uKj6bhl99zq4sTzArNLZVy9YxhH5wt405X+BoQfessVuH7fKHQdePbcGv7iW8dxbL6AV+8eDXXshWodhZrWNMhFIoSDZmFEEPwqgESjOTEx7ZZ3+E47yKGyncYswGb6nJwEeUPx28Tk8XwIA8ZWEHMTp8Nq+qqG3YnWDqK9xJZj+oB9MgeVbALhv9xsQvEk6dIJr7zTkukPJ3EhX7UaunrN9KtW0I9gLGNr+n4LPq5EkFAiKNY01IWKHOvvLUo2+c5hx0gKIwHdr0adfsM8NufjMqaGG7bNPwj5ipEMdzeHNcNQQgHftB1q4pUfBjzo727iAMl1ffcgDz9wJ8nZ5ZJlERGUaB5KxvDWg9vxbw5vxy9etxNAcxnGjTAus4CR85iZzwd22Po9z2gmZtlPnF4qIaFEECFvIpcz/SAv+naDPmf6fL1xTd+PSbuTx3NrVSgRwkQm3Gv5geeq8tU6VsrG2mzN9LVA4tQvbNGgb5woQc1ZQLiGiyD4jQPkV/Sgdurp4SQ0nWGxZCzkfMXboWkx/U40fbWBiGkwN2r6n+QDgj5gNoKJJZuKq2TTtznLe4EIQlrY6taEXQhgl4z6zRluB4WqYbYWppGKIxIh63PupnIHsD+H3WPBpb88sZmNKy1L/7IJBeOZOE4vlS127ZfEdWPnaAqZeLSpDONGGJdZwPiMClXN2q0EP48Q9NNxq2ST20LnXO6mZVHeCSjZtMzWwso7Hk1fQ0KJOBLuHO4+kgumxXSY8swg8P4TLs8d2JZFsaY1HWK/EfLOlgz6PJkbZMMAdK7dAbbtgJvpZ+LRwG2+u4vSj+ln4wqIOhukwkvgiAij6Thqmo6FYi1w58HtlX2ZflCdfkNHLEqhTox0XEGlbrB5zrz455YOaINvF/mADuhW4C393VTuAPZ68qvc4eDdwWHLSnePpzG7VLYC+MEQF6ZIhHBwOtdUhnGjWWOWCH4uBT33/Joho4qyplvT3zOexkgq5kjkVlUhxxNXoGq6p0x4sVgzBu+E/OySlrxjavo+DpscWR+m3420A5hjGGualbjn/SLFgPOZMYaSGkzM+oUtGfSD5R3vRKtOwCUFR/WOD3MXwU+uOx8+gT/46gso1LyTlLjXz0NHL+APvvoC/uCrL+Dxl5c9z/XMmVV850Vnp1+l3vCM6jtjesL4gY9MrDd0EMGRqI5Ho4HVO2FYPmDXLVfqDUe+AXAmkrtBIcDVtBWGUzFMDiVCywZBsOWdYE2WM/1m3bgi9owZZZsz83lszyV8xz764eBUcxnGDU4+trXoSueFEUG7CN7QJO62xtKGuyifBbB7LOPwPGKMoeyQd3jdvHPnt1hQMZ4JZ8EAOG1WAINUBK1/t7wzn+9B0Dd3LLwvgfvtB+3cK3XDJiItSza7x57xDK7fN4Zrd404bhcvAt0wfYDLF/aiaeWiuHcigz3jaTxyfBF3/+g0ckkF1+z0do6+Zt84Xloo4e4fncbffv9l/P5XX/Dc546vPI/f+8rzjttE5jQqNMhkA5gwZ/q1ho5Y1GlBHeyn3/DdKvuB12CXVM2RbwBsjbVdR0I38hV/V9NWeMOBCbzz2su6em3ACLRXbs/i1a51JoJfgMOWle4eS2NurYJnz661JT+1kmHcmM9XMZ6Je4iRG9mEgj3j6cB8wcxcHnt9LKUbOsOpxSLKagN7xtMYTsexZrJ/1Zy1zIOd1SHuSua2040LiPKOrem7a/Q59po9ES9eKJgW05Wm5athkDWNGE8vlTGRjVsxJijo24nm9dX0t1z1DmCw1nt/7XWe22NRAhGMErMQFgzNkE5EHTYMftU4IrIJBd/5L29p+bxf+NUj1s9//s3j+LNvvYiVkmoF8rVyHc+cWfUwQLHDkfuf8Nf1P54Yzq9WUNcYElGvDKY2jGEz4sVA9bGrDgJPTlVMb38AQkduZ46EbhSqGvZOtF/58F9+9mBXr8uxZzyDr3/4TU3v0y7T3z2egc6AExeL+DeHtoc+Fr67nZkrYGeTaV8c823IGQenhnyZ/vnVCk4ulHCra+4AX39PnV4FYFzIhlMxq4yxqjrlPosEuNbDQrEWWs8HbFLB11uhCdN/3w378Lfffxl/9MBR/Ml/uBbVut410+dEcNYcncmlx6BkbtmamiWZft9ARIhHIyACtrWxmPyQjjnLzFox/U7whgMTYAx49KRtevX9k4vQmVf3F+Ud0U8+6JiGBHknpvgnvN26vqrpoZm+aErH5Z2E6yTfKE1/PcEvzmE1fbHJq52S0qss7T1cMtdvDGAQDk3n8NJSyUFyAMM9EwBuPOAsK+Xr75mzZtAfT2M4pViafrlufO8poU4f8JKAdiwYAHsnzzV9o3rN/3MfzcTxm2/ej28dvYh/fuocgNZJ7VbgZaCnl8vYM5a2my0DSrD5+x1IeYeIbiaiY0R0gog+6vP3zxLR0+a/F4loVfjbZ4joeSKaIaK/oHZKLfqAhBLBZDbRVVMOYDB9d0duOx4wYXDtzmEMJRXr5AKA75muh3w2K0dFkHfEXUAw01es5izRbA0IHo5eayvom6Z0dc3abrvlnW6HoxvVO4Md9HkAbEfT52hH3skmFOweS2Mm5FBzt6NkMxycyhl2DBecEs93jy9g21ACV2539rzw9ff0mVUQGdVFI6k48hVD53eP2bRJgL0euAXDxFC4nAbgZfrFJolcAHjfDXsxPZzEZx48BqC7PB9gvI+apuP8WgW7xzMtS7BLteA+gn6i5RlMRHBDs+oAAB16SURBVFEAdwJ4O4DDAG4losPifRhjHzbHJL4KwF8C+Cfzsa8HcAOAawBcDeCnATTfD/cZiVi0az0fMNiJu3qn10xfiUZwwxUT+N7xRTDGwBjDd4UErriYqoK8k0vGrFr0ZjYJRdOGwR3IxWEzIlRzBGUY2AZUDatZJim4bBp/65zpN3TW8qQeBIxmvC6tzTA5lEAyFkE86m+/0AyHpps3UnFU6w2slOuhme3hae8uQtcZHj2xiDccmPCUzPIL3dG5Ai4bTiGhRDGcikFnhm7Pdwwp93pwlEAb5cRhZuNyWJq+0JzVLKAmY1F8+G1XWudxL/J8gCEf7zHLVIFgTZ+/7iDaMFwP4ARj7BRjTAVwD4B3Nbn/rTCGowMAA5AEEAeQABADcKHzw+0eyVikYyc9ESkhkcsYazkDtVPceOUEzq1WcGqxhNmlMs6uVHBkj9F1KS4mUdOPRMhmmE3q9NWGjmJN8+x64kHyjs8FIgipmN3L4D3Ju5d3ilY37mAzfVveCXdxIiLsGctg/7Zs27vRg1OGDONnx3B2pYw3//HD+OlPfRM3/OFDABD6POB9AE+fsTbweP58HivlOt54wNsxzN+zpjPLWmLYLF1dK9dtnyiXvCOuB16j356847RhaJbI5fiF63biqu1DIArfDxAE8QKzZzzdkulbA1QG0IZhB4Azwu9nAbzG745EtAfAPgAPAQBj7DEiehjAHAAC8DnG2IzP4z4I4IMAsHt382HU3eL/vOVQ10lcwEhG8sVbrRsDyPuhL9+43zipHjm+aJWu3fLKaTwxu+JYTKKmDxia5VJJDVz0fIEul1RPGWagvFNvv2SzLCRyLaYf927n20UzV9NBwmQ2gTvecRg3Xz0d+jEfu+UglEj78uOhaUOGefFCEa9yVRQ9ObuCl5fKeMe1lyGbUJBQInjrwW2hnjcSIdx89TT+6alzuP2mA9gxksJ3zeEgN+yf8Nw/HY9aFWA8R8Hr+Ncq9Sbyjr2eF9v03QHMnJ0SQVUzdpdqQ28pnUQjhD9796vw49MrPZB87dfaPZ5GLBpBKhYN1PRLAxz0/TT4oGLgdwP4EmOsAQBEtB/AIQA7zb9/g4jeyBj7ruPJGPs8gM8DwJEjR7rrzW+Bdk6+ZkgLTpGWw2YHNeOtsHs8jT3jaXzv+AIiRNg5mrJKPfMOAyvdCqqAXSrYTNMHjAlH7vsEyTu1hh7oLeSG1YClNlDVnJp+NEJIxiJd2Ss3czUdJBAR3nvDvrYe8+arwgVjN3ji9+hc3hP0eZfoZ37hGgc5CIv//WeuxFeePY8/+fox/Ol/eBUeOb6IQ9M536BMRBhLxzGfr2KXyfR5g9VquW6N0eS7waxPjqddCwaOpBJBra5bO8EwpODQdK7r7mzALr1MxaKWLNVsBCp/v9kBTOSeBSDOn9sJ4HzAfd8NW9oBgJ8H8APGWJExVgTwAIDXdnKgg4Z0zHbp85t320vceGACj51cwmMnl3DjgUlh2Iq/pg/YWnIg0zdvXympgfKO21O/neYszuLKNaFOXyj3bDYtKQxsW+XBZvrriV2j6UA7htnlMrYNJToK+IDhs/S+1+/Fl586hydnl/HE7DLeeMDL8jl4qarF9NM20+c7ZL5GkjHDm8fB9C15J3wi13guY05uM7O1foHvYMV5yc1GoG7EUHQgXNB/HMABItpHRHEYgf0+952I6CoAowAeE24+DeBNRKQQUQxGEtcj72xGpBO2zUC+zwHoxgOTKKkNFGoa3nhgwmIvfDExxkx5x/46ua4aqOmbt69W6t7qnQBN35g7HFbTt+WdSt1o6hI7KzMJxVOX3Q6sASoDrumvJyLmuNAZn0aq00tlX8//dvCbb96PXDKGX/+HH6PeYJ5STRG8Vp9bVPCJZasV1ZJ3+AXI9rcX5R0VUSE3FRaJmDEAqJmtcr/AZRqxQ7sp069piCuRrmWldtHy1RhjGoDbADwII2Dfyxh7nog+SUTvFO56K4B7mLMP/EsATgJ4DsAzAJ5hjH2lZ0e/gUjHjdFs3zm+gMdNy+R+Mf3XXTGOaIQQIeD1V0xYkgbfYdQbDA2dOZg+d79sxfQZA+Juu4oATb+d6p2IZVWhGfOKXY8Th1jwjsh20MrV9FLFoekcZua8dgzcDqEbDKdj+NBbrsBCoYaEEsGRvcE2znz97fbR9CuuoA9wEmDLOwuFGsbasGDgSCoupr+O64MHfbHsNpdswvQ3wHcHCNmRyxi7H8D9rts+4fr9Dp/HNQD8WhfHN7DgWuP7/vZx67ZuG76CkEvGcP3eMTR0huF0DIwxRCNkzeWtukoiAWPhpWJRi2G5IS62uIvpx5qUbIat3gGMZG5ZbaChM8ex8b/x7e1nv3kcn3voOL78mzd4rDOCsFk0/fXGwekc/t8fnsa51YrVmVutNzCfr3bN9AHgV163F3//2Cyu3D7k+U5F7B5LY8dIygr2vAx1rVK3NGyRpKQTUYcNw2Kx1la5Jocl72wA0x/LxBGPRhz5gaGk4hgkI6JUs/2H1hOSJnWIn3vVZdg3kbaGMA+nYlbSqh/46/dcZ/1MZBizcaZf9WFOv/hTO/GmqyYDNVyRAXk0/R40ZwFGMresNqAz5jmOTELBSknFhXwVn//uSegM+PQDM7j7A68NZZVsbd8l03fgsJXMte0YzphBpxdBPxmL4l8+dEOgmyzH//rWA3jfDXut34kIw+kY1sp1RIgQi5Jj3bldLxeLNUx0QKKSpryzEZr+cCqGh37rTY7+h1wq1rQjdyOY/iVlw9BLKNEIfmrPGF57+Thee/l4T7L/zTCSjjsGluRSipVLqLi8bfjxNWu+EY3KApuzfGwYwso7gO1FIs7H5eCOhH/2zRfR0Bk+cOM+/ODUMr59bCHg2ZzIV+pIx6PrrocOOiw7BqEzl1v99oqUjGcTLau4UvEotg05+wC402ZFbXgG32TiTnlnsai2ncQFDCuGar1hzYdeb1KwczTtcKzlmr6f+2lZ1da9XBOQQX/TYigRs5i+X9BvhWQsYi3OwOYst7zTRnMWYE7PqjdQreue0ZWZeBTnV6v4H4+fwXteuwe/ffNB7B1P4w8fONp06ARHPzqgtwIsOwahM5fLC3v6uBMNg5FUDKvlOsqq5vGbEQepMMawUOhU3omgWtetXUMnLqy9RC4Zg9rQPZVwAFDcIHlHBv1NilxKsTR9nhhLtrGAuEQEhJN3NNMONx4N/xp8Fq44FJ0jY1Y/ZeIKbn/rAcSiEfz2zQdx7EIB//jjsy2fu5Wr6aUM7q3PcdqcqxDWm79fsJh+XfcEu0wiavVtcAuGTuYdJGJRVDVD0+f9IBuJnKvSTkR5g+QdSZU2KYaSMUur7YTpAwYrXKvUPZKN1c4uyDvWfNw2TqJ0PIrFYg0JhXlm6fLF/utvvsIKRm+/egqv2jWCz3ztGH48u+J5vkxCwYfesh9jmbhk+k1waDqHb85cMGSUeBSzSyVH7fhGYTgdw9H5Aiqq5ksCeLNjJ924HAnenGUG1I1+z5b/TkXDNpdpaqm2MfKOPGs2KXLJmJXMdNschAUPmu46fW6BXBNcPDnrD9ucBZhleKoxHcjNuI7sHcMb9k/gPwndqkSE33vnK/Dhe5/Gw8cuep5vsaiiUK3jM794LfLV+oYz10HFoekh6Ax48UIB1+4awexyGVdt7240ZC/Amb44H5dDTOT+y9NG72cn4yyTsShqWgOFJvOh1xO2/46X6Rc3YCg6IIP+psVQUrG2jBXXUIqwyATIO1Y3rdAWzzXJdjT9VNyo0yfyHtubrpzEm670Nvdcu2sED33kzb7P96l/fQH/7ZGX8P43XI5CVcOe8e7qzrcqDgrJ3Kt3DOPscgVvOxx+IEu/MJKKo1jTUKhqGEk7pTme/5lfq+JvvnsK//aaaet9tAOjTl9HsVYfiJ2g5anvatBizLCYlolcidDIpWIo1jToOutK3gG8gTwWNWqqxaDPmX471TuZuFGnL3r9d4MPvWU/sgkFf/S1o6ar6caf1IOI3WNppONRzMwVcCFfhdrQLbfLjcSw6U11IV/1rFW+Fv/v+2eg6Tp++2ev6ug1jJLNxoaVQ7ox5Gqk5KhphkmjDPoSoZFLKmDcn5zLO/H2vs5s0p/pA3xIjM1OOmH6vE6/4pPI7QQj6Th+8y378dDRi1gqqQM/NWujYNkxzOWtcs09XXbj9gLcf2ehWPNJ5Bpr8b5nzuOXX7On411cQomi3mBGE9gAkAJuwui2YrAcNmX1jkRY5IT5m1W1M6bP/Xf8dPpMXHFYH/NBKGFn5AK2TFSoam0lgJvhva/faw276Ier6VYBt2M4vWzMpe1FY1a34N3hjAEpV8kmXyvZhILb37q/49fguaPFgtc9diMwFDAy0R6gIpm+REhYpmuCa2G7bDpI3gEMPb5St9lJJ/KO6C/e7gUpCMlYFB/5GWPrL0s2g3Foagj5qoYfnFqGEqGeTIvrFqI5npvp8+/y1990OcY7KNXk4OfAUqk2EJp+Jh5FhLxMfyM6hjk2/lOR6AiivXKl3vC0tYdBM3knE486mL7aibwjBPpeyDscP//qHajWG7jllb2ZjbAVwTvEvzVzATtHUy1tE9YDYvLWTQJed8U4fvcdh3Hr9d0NUeKkpN5gA8H0iQhDPqZrtq2yDPoSIeFg+h0mSu3mLG8tcyoetZq+AFvTbyuRK/iEu102u0E0QnjPa/f07Pm2Iq6aMsod81UttIldvyFaN7i9mJKxKN7X5rAZP4jnQXaDu3E5cimvvbI1QGWdvfQBKe9sWliafq3uGaASFvzC4cfeM3HFMdmqI6Yv6LadDu+Q6AxDyRh2jRneS4Og5wPOoN8v+wGxH2QQErmAYQXh1vQ3alQiIIP+poXN9DXPfNyw4EzIL5GbNhurOHhHbrveOxy9lHckwoHXuQ9C5Q5gyIi8WqVfQT/hYPqDseb8mL41FH2dRyUCMuhvWoj1v36uhWHQtGQz5i7Z7KR6x17Q7TxOojc4ZEo8uweE6QMQ/PX7FPQFUjIo8o6fpl8edKZPRDcT0TEiOkFEH/X5+2eJ6Gnz34tEtCr8bTcRfZ2IZojoBSLa27vDv3QRVyJIxiLIm4ncTk6inaMpEAFTPpUd6UTUYXXbmbwjMn3JL9YbR/aOIRohHOqgs7VfGDY9mNwum72CQ9MfEHlHtEzhKFklmwNow0BEUQB3AngbjCHpjxPRfYyxF/h9GGMfFu5/O4BXC0/x9wA+xRj7BhFlAXg9RiU6wlDSsFfuVNO/YjKLH3/8bdYQaxHpuOF6yBgDEXXkvSMOfO5VyaZEeLzxykn86Hdu6qoEstfgXbl90/QVUd4ZjKA/lFQ8mn6xpkGJUFvnU68Q5hWvB3CCMXaKMaYCuAfAu5rc/1YAdwMAER0GoDDGvgEAjLEiY8x/dphE28glla40fQC+AR8wmJjO7Kodq3qnDcYu6pVS098YDFLAB+wGrb7JO8L6HIQ6fcC0TFENyxSOsumwuREuoGHO4B0Azgi/nzVv84CI9gDYB+Ah86YrAawS0T8R0VNE9MfmzsH9uA8S0RNE9MTCQrjJSRK2Vtippt8MGZfpWq0Dpp/qU52+xOYF1/T7V70zeEyfW6YUhHGQxVpjw44vzBnsdykKGm30bgBfMgeiA4Z8dCOA3wLw0wAuB/Bez5Mx9nnG2BHG2JHJSa/zooQ/cqkY8lXNnEzV25OIa648mduJvBOJkBX4pbwjAdgNWv2TdwawZNPHXrlU0zZkahYQLuifBbBL+H0ngPMB9303TGlHeOxTpjSkAfhnANf5PlKibRjzN01Nv02ztVbgerzI9OPRCCKR9rajfGHLRK4EYHeS96tvQyQ/G1EO6QfLXrliM/3SBs3HBcIF/ccBHCCifUQUhxHY73PfiYiuAjAK4DHXY0eJiNP3twJ4wf1Yic6QS8ZsTb/n8o6xIHkTiaq1Nx+Xg188EpLpSwC48cAE/u0rpzGW7s8AHF6ymYlHHQPKNxJ+9sqlDbR+bnkWmwz9NgAPApgBcC9j7Hki+iQRvVO4660A7mHC2HdT5vktAN8ioudgSEV/08s3cCkjZw5S6UfQ50yMWzGojUZHQZ9fPKS8IwEA1+wcwZ2/fF3fvICUaARKhAZG2gFsN1hxkEppg4aiAyG9dxhj9wO433XbJ1y/3xHw2G8AuKbD45NoglwqZmnt7QxFDwOL6XN5p6635bvDkTJdBv38fSQk+oFkLDowSVwggOmrA8z0JQYXYklav5i+lchtdCbvZOIKkrHohg+olrh0kFAiyA6Q7XZOMEfk2Kih6IAM+psaop98r6t3Mq5ErmomcttFOh6V5ZoS64pkLGoNCBoEDAkDjzhKtYajeXE9IYP+JkY/mX7alcitaXpH06+GkrENaTWXuHSRSUQdjp4bDW6ZsmoyfVXToTZ0ZDeoumhwLocSbUOcRNT7On1XIrdDpn/bW/djobCr9R0lJHqET/9Pr8Rwqj/VQZ1i73gGpxaKAGzJdKPkHRn0NzEcTL/HidxYNIJ4NGIlcjst2dw3kcG+icGw9pW4NPBTe8Y2+hA8ODSdw2MnlwAItspS3pFoF6Km34+SyFTctleuaQ1pjywh0SEOTQ9hPl/FSknd0KHogGT6mxr91PQBo8HF0ZHbw5GHEhKXEvhAm5n5vCXFyuodibaRiSvgTYe9tmEA+PSs7ko2JSQkgIPTxkCbo3MFe1TiBiVy5Vm8iRGJkNXg0Y+yyLTI9DtszpKQkAC2DSUxkY1jZi6PUm3jBqgAMuhvevAa4H7IO+m4PT1LbcigLyHRDQ5O5XB03mb6siNXoiP007UwHVdQEqyVN2LKj4TEVsGh6SEcu1Cw5uX2a2RkK8izeJODJ3OTfaisScejVp1+TWtIp0wJiS5wcCoHVdPxk3N5AJLpS3SIXDKGhNK+z30YZCTTl5DoGQ5NGxU8T84uI0IbN2NCnsWbHLmk0reBFCkzkas1dOgMsnpHQqILXLEtAyVCeHmpjEx8Y+bjArJOf9PjHddehl1j6b48dyZhBH1rKLoM+hISHSOhRLF/WxZH5wsbVqMPyKC/6fGWg9vwloPb+vLc6biChs6stnHJ9CUkusPBqSEz6G9cfizUWUxENxPRMSI6QUQf9fn7Z4noafPfi0S06vp7jojOEdHnenXgEv0HN11bKasAZNCXkOgWB01df6CZPhFFAdwJ4G0wBp0/TkT3McasWbeMsQ8L978dwKtdT/P7AL7TkyOWWDfwjsHlkhH0pfeOhER34MncjRzaHoa6XQ/gBGPsFGNMBXAPgHc1uf+tAO7mvxDRTwHYDuDr3RyoxPqDJ4jXykZdsWT6EhLd4dDU/9/e3cVYedRxHP/+2O0iS1uhiqawQLcJoTaaQrMS6ltM1YSighdGwSalpgkXbbU2RlNfYrR3JmrVhDQhbaU2BmyxKqlEYqCJNxRZtEEoLdD6whaUNXbr2wWl/L145tDDsufsgT1nH3ae3yc52TOzs3tmMjv/MzvznGeK2zGUOdNvZRTPA47VpYdS3nkkLQT6gV0pPQ34LvCliVXTylBbd3ylFvR9yabZhMy5YjpzrpjOrN7yDnlp5e1mrOuKokHZNcDWiHg9pe8EtkfEsWaXJ0laD6wHWLBgQQtVsslQ+8RgbU3/Yk7OMrM3SOKRde9m9sxLO+gPAfVHH/UBxxuUXQPcVZe+CXi/pDuBy4EeSf+JiHM2gyNiI7ARYGBgoNEbik2y2kbuSC3oe6ZvNmHv6ntzqa/fStDfCyyS1A+8TBHYPzO6kKTFwGxgdy0vIm6t+/7twMDogG+Xrjdm+l7TN8vFuKM4Ik4DdwM7gEPA4xFxUNL9klbVFV0LbIkIz9Qzcd5M31fvmE15LW0hR8R2YPuovG+MSn9znN+xCdh0QbWzUs30TN8sOx7F1tAMfzjLLDsexdZQT/c0LusSI2mm73vvmE19HsXWVG9P99k1fc/0zaY+j2JrqrenizNpa95B32zq8yi2pnrr7tXv5R2zqc+j2Jqqv0eIb8NgNvV5FFtTM9K5uD3d00o76cfM2sdB35qqzfR9CwazPHgkW1O1NX1v4prlwSPZmqoFfW/imuXBI9maqt10zTN9szx4JFtTXt4xy4tHsjV1diPXd9g0y4KDvjXlmb5ZXjySramzQd+XbJplwSPZmqpt5Pp8XLM8tDSSJa2Q9IKko5LOO+5Q0gOSnk2Pw5JGUv4SSbslHZS0X9Kn290A6yzP9M3yMu7JWZK6gA3ARygOSd8raVtEPFcrExH31pX/HLA0Jf8H3BYRRyTNBfZJ2hERI+1shHWOL9k0y0srI3kZcDQiXoqIU8AWYHWT8muBzQARcTgijqTnx4GTwJyJVdkm08zptQ9n+eodsxy0EvTnAcfq0kMp7zySFgL9wK4xvrcM6AFevPBqWll89Y5ZXloZyWPdWjEalF0DbI2I18/5BdLVwGPAZyPizHkvIK2XNChpcHh4uIUq2WQ5u5HroG+WhVZG8hAwvy7dBxxvUHYNaWmnRtKVwK+Ar0fEM2P9UERsjIiBiBiYM8erP5cS33vHLC+tjOS9wCJJ/ZJ6KAL7ttGFJC0GZgO76/J6gJ8DP46IJ9pTZZtM3sg1y8u4IzkiTgN3AzuAQ8DjEXFQ0v2SVtUVXQtsiYj6pZ9PAR8Abq+7pHNJG+tvHdbTPY2vrryOj98wt+yqmFkb6NwYXb6BgYEYHBwsuxpmZlOKpH0RMTBeOf/PbmZWIQ76ZmYV4qBvZlYhDvpmZhXioG9mViEO+mZmFeKgb2ZWIQ76ZmYVcsl9OEvSMPCXCfyKtwL/aFN1pooqthmq2e4qthmq2e4LbfPCiBj35mWXXNCfKEmDrXwqLSdVbDNUs91VbDNUs92darOXd8zMKsRB38ysQnIM+hvLrkAJqthmqGa7q9hmqGa7O9Lm7Nb0zcyssRxn+mZm1kA2QV/SCkkvSDoq6b6y69MpkuZLelrSIUkHJd2T8q+S9BtJR9LX2WXXtd0kdUn6g6SnUrpf0p7U5p+mk9qyImmWpK2Snk99flPufS3p3vS3fUDSZklvyrGvJT0i6aSkA3V5Y/atCj9M8W2/pBsv9nWzCPqSuoANwC3A9cBaSdeXW6uOOQ18MSLeASwH7kptvQ/YGRGLgJ0pnZt7KE5vq/k28EBq8yvAHaXUqrN+APw6Iq4DbqBof7Z9LWke8HlgICLeCXRRHNGaY19vAlaMymvUt7cAi9JjPfDgxb5oFkEfWAYcjYiXIuIUsAVYXXKdOiIiTkTE79Pzf1MEgXkU7X00FXsU+EQ5NewMSX3AR4GHUlrAzcDWVCTHNl9JcdzowwARcSoiRsi8r4FuYIakbqAXOEGGfR0RvwX+OSq7Ud+upjhrPCLiGWCWpKsv5nVzCfrzgGN16aGUlzVJ1wBLgT3A2yPiBBRvDMDbyqtZR3wf+DJwJqXfAoykM5whzz6/FhgGfpSWtR6SNJOM+zoiXga+A/yVIti/Cuwj/76uadS3bYtxuQR9jZGX9WVJki4HfgZ8ISL+VXZ9OknSx4CTEbGvPnuMorn1eTdwI/BgRCwF/ktGSzljSWvYq4F+YC4wk2JpY7Tc+no8bft7zyXoDwHz69J9wPGS6tJxki6jCPg/iYgnU/bfa//upa8ny6pfB7wXWCXpzxRLdzdTzPxnpSUAyLPPh4ChiNiT0lsp3gRy7usPA3+KiOGIeA14EngP+fd1TaO+bVuMyyXo7wUWpR3+HoqNn20l16kj0lr2w8ChiPhe3be2AevS83XALye7bp0SEV+JiL6IuIaib3dFxK3A08AnU7Gs2gwQEX8DjklanLI+BDxHxn1NsayzXFJv+luvtTnrvq7TqG+3Abelq3iWA6/WloEuWERk8QBWAoeBF4GvlV2fDrbzfRT/1u0Hnk2PlRRr3DuBI+nrVWXXtUPt/yDwVHp+LfA74CjwBDC97Pp1oL1LgMHU378AZufe18C3gOeBA8BjwPQc+xrYTLFv8RrFTP6ORn1LsbyzIcW3P1Jc3XRRr+tP5JqZVUguyztmZtYCB30zswpx0DczqxAHfTOzCnHQNzOrEAd9M7MKcdA3M6sQB30zswr5P+rUBC2kXPuJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c70eea20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.Series(res).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "train test spliting create bias due to the intrinsic randomness in the sets selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# K-fold cross-validation\n",
    "\n",
    "1. Split the dataset into K **equal** partitions (or \"folds\").\n",
    "2. Use fold 1 as the **testing set** and the union of the other folds as the **training set**.\n",
    "3. Calculate **testing accuracy**.\n",
    "4. Repeat steps 2 and 3 K times, using a **different fold** as the testing set each time.\n",
    "5. Use the **average testing accuracy** as the estimate of out-of-sample accuracy.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diagram of **5-fold cross-validation:**\n",
    "\n",
    "![5-fold cross-validation](https://raw.githubusercontent.com/justmarkham/DAT8/master/notebooks/images/cross_validation_diagram.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration                   Training set observations                   Testing set observations\n",
      "    1     [ 5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24]        [0 1 2 3 4]       \n",
      "    2     [ 0  1  2  3  4 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24]        [5 6 7 8 9]       \n",
      "    3     [ 0  1  2  3  4  5  6  7  8  9 15 16 17 18 19 20 21 22 23 24]     [10 11 12 13 14]     \n",
      "    4     [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 20 21 22 23 24]     [15 16 17 18 19]     \n",
      "    5     [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]     [20 21 22 23 24]     \n"
     ]
    }
   ],
   "source": [
    "# simulate splitting a dataset of 25 observations into 5 folds\n",
    "from sklearn.cross_validation import KFold\n",
    "kf = KFold(25, n_folds=5, shuffle=False)\n",
    "\n",
    "# print the contents of each training and testing set\n",
    "print('{} {:^61} {}'.format('Iteration', 'Training set observations', 'Testing set observations'))\n",
    "for iteration, data in enumerate(kf, start=1):\n",
    "    print('{:^9} {} {:^25}'.format(str(iteration), str(data[0]), str(data[1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Dataset contains **25 observations** (numbered 0 through 24)\n",
    "- 5-fold cross-validation, thus it runs for **5 iterations**\n",
    "- For each iteration, every observation is either in the training set or the testing set, **but not both**\n",
    "- Every observation is in the testing set **exactly once**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create k-folds\n",
    "kf = KFold(X.shape[0], n_folds=10, random_state=0)\n",
    "\n",
    "results = []\n",
    "\n",
    "for train_index, test_index in kf:\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n",
    "\n",
    "    # train a logistic regression model\n",
    "    logreg = LogisticRegression(C=1e9)\n",
    "    logreg.fit(X_train, y_train)\n",
    "\n",
    "    # make predictions for testing set\n",
    "    y_pred_class = logreg.predict(X_test)\n",
    "\n",
    "    # calculate testing accuracy\n",
    "    results.append(metrics.accuracy_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10.000000\n",
       "mean      0.794644\n",
       "std       0.030631\n",
       "min       0.764045\n",
       "25%       0.768820\n",
       "50%       0.780899\n",
       "75%       0.820225\n",
       "max       0.842697\n",
       "dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(results).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "\n",
    "results = cross_val_score(logreg, X, y, cv=10, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10.000000\n",
       "mean      0.794665\n",
       "std       0.019263\n",
       "min       0.775281\n",
       "25%       0.779963\n",
       "50%       0.786517\n",
       "75%       0.806742\n",
       "max       0.829545\n",
       "dtype: float64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(results).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing cross-validation to train/test split\n",
    "\n",
    "Advantages of **cross-validation:**\n",
    "\n",
    "- More accurate estimate of out-of-sample accuracy\n",
    "- More \"efficient\" use of data (every observation is used for both training and testing)\n",
    "\n",
    "Advantages of **train/test split:**\n",
    "\n",
    "- Runs K times faster than K-fold cross-validation\n",
    "- Simpler to examine the detailed results of the testing process\n",
    "\n",
    "## Cross-validation recommendations\n",
    "\n",
    "1. K can be any number, but **K=10** is generally recommended\n",
    "2. For classification problems, **stratified sampling** is recommended for creating the folds\n",
    "    - Each response class should be represented with equal proportions in each of the K folds\n",
    "    - scikit-learn's `cross_val_score` function does this by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improvements to cross-validation\n",
    "\n",
    "**Repeated cross-validation**\n",
    "\n",
    "- Repeat cross-validation multiple times (with **different random splits** of the data) and average the results\n",
    "- More reliable estimate of out-of-sample performance by **reducing the variance** associated with a single trial of cross-validation\n",
    "\n",
    "**Creating a hold-out set**\n",
    "\n",
    "- \"Hold out\" a portion of the data **before** beginning the model building process\n",
    "- Locate the best model using cross-validation on the remaining data, and test it **using the hold-out set**\n",
    "- More reliable estimate of out-of-sample performance since hold-out set is **truly out-of-sample**\n",
    "\n",
    "**Feature engineering and selection within cross-validation iterations**\n",
    "\n",
    "- Normally, feature engineering and selection occurs **before** cross-validation\n",
    "- Instead, perform all feature engineering and selection **within each cross-validation iteration**\n",
    "- More reliable estimate of out-of-sample performance since it **better mimics** the application of the model to out-of-sample data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overfitting, Underfitting and Model Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've gone over the basics of validation, and cross-validation, it's time to go into even more depth regarding model selection.\n",
    "\n",
    "The issues associated with validation and \n",
    "cross-validation are some of the most important\n",
    "aspects of the practice of machine learning.  Selecting the optimal model\n",
    "for your data is vital, and is a piece of the problem that is not often\n",
    "appreciated by machine learning practitioners.\n",
    "\n",
    "Of core importance is the following question:\n",
    "\n",
    "**If our estimator is underperforming, how should we move forward?**\n",
    "\n",
    "- Use simpler or more complicated model?\n",
    "- Add more features to each observed data point?\n",
    "- Add more training samples?\n",
    "\n",
    "The answer is often counter-intuitive.  In particular, **Sometimes using a\n",
    "more complicated model will give _worse_ results.**  Also, **Sometimes adding\n",
    "training data will not improve your results.**  The ability to determine\n",
    "what steps will improve your model is what separates the successful machine\n",
    "learning practitioners from the unsuccessful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Illustration of the Bias-Variance Tradeoff\n",
    "\n",
    "For this section, we'll work with a simple 1D regression problem.  This will help us to\n",
    "easily visualize the data and the model, and the results generalize easily to  higher-dimensional\n",
    "datasets.  We'll explore a simple **linear regression** problem.\n",
    "This can be accomplished within scikit-learn with the `sklearn.linear_model` module.\n",
    "\n",
    "We'll create a simple nonlinear function that we'd like to fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "def test_func(x, err=0.5):\n",
    "    y = 10 - 1. / (x + 0.1)\n",
    "    if err > 0:\n",
    "        y = np.random.normal(y, err)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a realization of this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_data(N=40, error=1.0, random_seed=1):\n",
    "    # randomly sample the data\n",
    "    np.random.seed(1)\n",
    "    X = np.random.random(N)[:, np.newaxis]\n",
    "    y = test_func(X.ravel(), error)\n",
    "    \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEw1JREFUeJzt3X+MHOddx/HPt2cHLiXlUnyF+hLnHJQchETC0apNsVTapsUhlNiKIkikQFoFrBZRoCAjm/6RCoRiYWgpUgRYpTSF0qSk1tVqAqbEiQJRE3rOhTo/ejSk+eG1aa4KFxBcydX58sfu2ef1zu7szuzMM8+8X1KUvb3JzfPk9j777Pd55hlzdwEAqu91ZTcAAJAPAh0AIkGgA0AkCHQAiASBDgCRINABIBIEOgBEgkAHgEgQ6AAQiXVFnmzDhg0+PT1d5CkBoPKOHDnybXef7HdcoYE+PT2tubm5Ik8JAJVnZs+nOY6SCwBEgkAHgEgQ6AAQCQIdACJBoANAJApd5QIgLLPzTe07tKDjS8vaODGuXdtmtGPLVNnNwpAIdKCmZueb2nPgqJZXTkqSmkvL2nPgqCQR6hVFyQWoqX2HFk6F+arllZPad2ihpBYhKwIdqKnjS8sDPY/wEehATW2cGB/oeYSPQAdqate2GY2vHzvjufH1Y9q1baakFiErJkWBmlqd+GSVSzwIdKDGdmyZIsAjQskFACJBoANAJAh0AIgEgQ4AkWBSFEBlsPdMbwQ6gEpg75n+KLkAqAT2numvb6Cb2afM7CUze2LNc280sy+b2Tfa/z5/tM0EUHfsPdNfmhH6pyVd0/Hcbkn3u/slku5vfw0AI8PeM/31DXR3f0jSyx1Pb5d0Z/vxnZJ25NwuADhDyHvPzM43tXXvYW3efa+27j2s2flmKe0YdlL0B939hCS5+wkze1OObQKAs4S690xIk7UjX+ViZjsl7ZSkTZs2jfp0ACIW4t4zvSZri27rsKtcvmVmb5ak9r9fSjrQ3fe7e8PdG5OTk0OeDgDCFNJk7bCBflDSLe3Ht0j6Yj7NAYBqCWmyNs2yxc9J+oqkGTM7Zma3Stor6T1m9g1J72l/DQC1E9Jkbd8aurvflPCtq3NuCwAUJq9tBEKarOXSfwC1k/fKlFAma7n0H0DtxLqNACN0oAbYpfBMIa1MyROBDgypKiEZ0oUvodg4Ma5ml/Cu+jYClFyAIayGZHNpWa7TIVnWJd+9xFpeyCKklSl5ItCBIVQpJGMtL2SxY8uUbr/+Ck1NjMskTU2M6/brr6j8JxZKLsAQqhSSsZYXBpFUHqt6gHdihA4MIaSrA/uJtbyQVpXKY1kR6MAQqhSSsZYX0vrowScrUx7LipILMISQrg5MI6m8UJWVOsOanW9qaXml6/dCLI9lRaCjMkILn6rXYOuwnLHXKDzE8lhWBDoqoQ7hU7SQ9vEe1Zt1r1F4iOWxrKihoxKqtEywKkJZqTPKScukUfj5566PciBAoKMSQgmfmISyUmeUb9ZJk9e3/eyPZf7ZISLQUQmhhE9MQlmpM8o367qt8KGGjkrYtW3mjBq6FO4ywSpYrVkvr5zUmJlOumuqpInmUV/4VPXJ60EwQkclrB1pSdKY2amP5TFeIDJKa2vWknTS/dSbYxnBV8Qnhdn5prbuPazNu+/V1r2Ho33NEOiojB1bpk798Z90lxT3VX+jEtoE86jLInW6UpSSCyolpKV2VTWKmnXWZYejLIvU6TVDoGMgZV/cw2qX7PKuWYd+jUCdXjOUXJBaCB9dWe2SXd4169BKOJ3KeM2UVbMn0JFaCH+4oSy1q7K8a9ahj4CLfs2UOfCh5BK4sksca4Xwh1u1TbFClWfNOvT91ot+zZRZsyfQAxZabTKUP9w6rSuugipcI1Dka6bMgQ8ll4CFUOJYi3IHuqnb1Zj9lDnPwwg9YCGUONai3IEkfGo6rcxPLAR6wEIpcazFH279hDSPUwVlDnwyBbqZfVjSL0lySUclvd/dv5NHw1CN2iTiFto8TlWUNfAZuoZuZlOSfk1Sw90vlzQm6ca8GgZqkxitNGulQ5vHQW9ZSy7rJI2b2YqkcyUdz94krEWJA6OQduQd2jwOeht6hO7uTUl/KOkFSSckveLu/5BXw4Bu6rJr3qilHXlzZW61ZCm5nC9pu6TNkjZKer2Z3dzluJ1mNmdmc4uLi8O3FLUXwtYDsUg78maparVkWYf+bknfdPdFd1+RdEDST3Qe5O773b3h7o3JyckMp0MIyhwhU8/NT9qRN/M41ZKlhv6CpKvM7FxJy5KuljSXS6sQpLJXPFDPzc8gK6iYx6mOLDX0RyXdI+kxtZYsvk7S/pzahQCVPUKmnpsfRt5xyrTKxd1vk3RbTm1B4MoeIYe6Lj/pwpvQL8hh5B0frhRFamVfuRri1gNJZai551/WF440uSAHhTJv35uxCI1Gw+fmKLNXVWd4Sa0Rcp0/qm/de7jrm9yY2an7nq41NTGuh3e/q4imISJmdsTdG/2OY4SO1EIYIYdWxkgqN3UL817HA3kg0DGQNHXXUYVu2atsukkqQyWN0JnAxSixHzpylcfFP0lr3cteZdNN0oU3N731Qi7IQeEYoSNXWW+/1WsUXvYqm256laEaF70xqPIQ4kegI1dZQ7fXG0LZq2ySJJWhWBaIolFyQa6yXvzT6w2BfUWA3gh05Cpr6PZ6Q+DqRqA3Si7IVdaljf2uBqWMASQj0JG7LKEbwlp3oKoIdASHUTgwHGroABAJAh0AIkGgA0AkCHQAiASBDgCRYJULegptu1oAyQh0JApxu1oAySi5IFGI29UCSEagI1GI29UCSEagI1HWnRMBFItARyK2qwWqhUlRJGKjLKBaCHT0xEZZQHVQcgGASBDoABCJTIFuZhNmdo+Zfd3Mnjazt+XVMADAYLLW0D8h6e/d/QYzO0fSuTm0CQAwhKED3czeIOntkt4nSe7+qqRX82kWAGBQWUouF0talPSXZjZvZp80s9fn1C4AwICyBPo6SVdK+lN33yLpfyTt7jzIzHaa2ZyZzS0uLmY4XT3Mzje1de9hbd59r7buPazZ+WbZTQJQEVkC/ZikY+7+aPvre9QK+DO4+353b7h7Y3JyMsPp4re6u2FzaVmu07sbEuoA0hg60N39PyS9aGar14FfLempXFoVkUFG3OxuCCCLrKtcPiTps+0VLs9Ken/2JsVj0P3E2d0QQBaZAt3dH5fUyKkt0ek14u4W6BsnxtXsEt4bJ8a5cxCAvrhSdIQGHXEn7W74zh+ZpLYOoC8CfYQG3U98x5Yp3X79FZqaGJdJmpoY1+3XX6EHvr5IbR1AX+y2OEK7ts2cUUOX+u8n3m13ww/f/XjXY6mtA1iLEfoIJY24B619c+cgAGkwQu8hj4nIPPYTH2akD6B+CPQEgy45HCXuHAQgDQI9waBLDkeNOwcB6IdAT1DERT6sLQeQJwI9Qa+LfNLoF9YhlXQAxIFVLgmSLvJJMxGZZpMt9m0BkDcCPUGWJYdpwpp9WwDkjZJLF53lko///I8PVAZJE9ZZSzoA0IkReoc89iRPcyFQlpIOAHRDoHfIo7adJqzzuooUAFZRcumQVC5pLi1r8+57Uy0vTHshEGvLAeSJQO+QVNuWdEYJRuq9vJCwBlA0Si4dupVLOrG8EECIGKF36CyXeMJxLC8EEBoCvYu15ZKtew+zvBBAJVBy6YPlhQCqghF6H2xdC6AqCPQUWLECoAoouQBAJAh0AIgEgQ4AkSDQASASBDoARCJzoJvZmJnNm9mX8mgQAGA4eYzQf13S0zn8HABABpkC3cwukPQzkj6ZT3MAAMPKOkL/Y0m/Lem1HNoCAMhg6CtFzey9kl5y9yNm9o4ex+2UtFOSNm3aNOzpUum8FyiX6AOokywj9K2SrjOz5yTdJeldZvbXnQe5+353b7h7Y3JyMsPpesvjXqAAUGVDB7q773H3C9x9WtKNkg67+825tWxAedwLFACqLJp16Ek3nOBGFADqIpdAd/cH3f29efysYSXdcIIbUQCoi2hG6NyIAkDdRbMfOjeiAFB30QS6xI0oANRbNCUXAKg7Ah0AIkGgA0AkCHQAiASBDgCRINABIBIEOgBEgkAHgEgQ6AAQCQIdACJBoANAJAh0AIgEgQ4AkSDQASASBDoARIJAB4BIRHWDi06z803uYASgNqIN9Nn5pvYcOKrllZOSpObSsvYcOCpJhDqAKEVbctl3aOFUmK9aXjmpfYcWSmoRAIxWtIF+fGl5oOcBoOqiDfSNE+MDPQ8AVRdtoO/aNqPx9WNnPDe+fky7ts2U1CIAGK1oJ0VXJz5Z5QKgLqINdKkV6gQ4gLoYOtDN7EJJn5H0Q5Jek7Tf3T+RV8PSYq05ALRkGaF/V9JvuftjZnaepCNm9mV3fyqntvXFWnMAOG3oSVF3P+Huj7Uf/7ekpyUVmqKsNQeA03JZ5WJm05K2SHo0j5+XFmvNAeC0zIFuZt8n6QuSfsPd/6vL93ea2ZyZzS0uLmY93RlYaw4Ap2UKdDNbr1aYf9bdD3Q7xt33u3vD3RuTk5NZTncW1poDwGlZVrmYpL+Q9LS7fyy/JqXHWnMAOC3LKpetkn5B0lEze7z93O+4+33Zm5Uea80BoGXoQHf3f5ZkObYFAJBBtHu5AEDdEOgAEAkCHQAiQaADQCQIdACIBIEOAJEg0AEgEgQ6AESCQAeASBDoABAJAh0AIkGgA0AkCHQAiASBDgCRINABIBIEOgBEgkAHgEgQ6AAQCQIdACJBoANAJAh0AIgEgQ4AkSDQASASBDoARIJAB4BIEOgAEAkCHQAikSnQzewaM1sws2fMbHdejQIADG7oQDezMUl3SPppSZdJusnMLsurYQCAwWQZob9F0jPu/qy7vyrpLknb82kWAGBQ6zL8t1OSXlzz9TFJb83WnN5m55vad2hBx5eWtXFiXLu2zWjHlqlRnhIAKiNLoFuX5/ysg8x2StopSZs2bRr6ZLPzTe05cFTLKyclSc2lZe05cFSSCHUAULaSyzFJF675+gJJxzsPcvf97t5w98bk5OTQJ9t3aOFUmK9aXjmpfYcWhv6ZABCTLIH+VUmXmNlmMztH0o2SDubTrLMdX1oe6HkAqJuhA93dvyvpVyUdkvS0pM+7+5N5NazTxonxgZ4HgLrJtA7d3e9z90vd/Yfd/ffzalQ3u7bNaHz92BnPja8f065tM6M8LQBURpZJ0UKtTnyyygUAuqtMoEutUCfAAaA79nIBgEgQ6AAQCQIdACJBoANAJAh0AIiEuZ+1/croTma2KOn5HH7UBknfzuHnVE0d+13HPkv0u07S9Pkid++7d0qhgZ4XM5tz90bZ7ShaHftdxz5L9LvsdhQpzz5TcgGASBDoABCJqgb6/rIbUJI69ruOfZbod53k1udK1tABAGer6ggdANAh6EA3s2vMbMHMnjGz3V2+/z1mdnf7+4+a2XTxrcxXij7/ppk9ZWZfM7P7zeyiMtqZt379XnPcDWbmZhbFSog0/Tazn2v/zp80s78puo15S/Ea32RmD5jZfPt1fm0Z7cybmX3KzF4ysycSvm9m9ift/y9fM7MrBz6Juwf5j6QxSf8u6WJJ50j6V0mXdRzzK5L+rP34Rkl3l93uAvr8Tknnth9/sOp9Ttvv9nHnSXpI0iOSGmW3u6Df9yWS5iWd3/76TWW3u4A+75f0wfbjyyQ9V3a7c+r72yVdKemJhO9fK+nv1Lpf81WSHh30HCGP0N8i6Rl3f9bdX5V0l6TtHcdsl3Rn+/E9kq42s243r66Kvn129wfc/X/bXz6i1r1cqy7N71qSfk/SH0j6TpGNG6E0/f5lSXe4+39Kkru/VHAb85amzy7pDe3H368u9yquInd/SNLLPQ7ZLukz3vKIpAkze/Mg5wg50Kckvbjm62Pt57oe461b4r0i6QcKad1opOnzWreq9Y5edX37bWZbJF3o7l8qsmEjlub3famkS83sYTN7xMyuKax1o5Gmzx+VdLOZHZN0n6QPFdO00g3693+WkG9w0W2k3bkkJ80xVZK6P2Z2s6SGpJ8caYuK0bPfZvY6SR+X9L6iGlSQNL/vdWqVXd6h1qexfzKzy919acRtG5U0fb5J0qfd/Y/M7G2S/qrd59dG37xSZc6zkEfoxyRduObrC3T2R69Tx5jZOrU+nvX6SBO6NH2Wmb1b0kckXefu/1dQ20apX7/Pk3S5pAfN7Dm16osHI5gYTfsa/6K7r7j7NyUtqBXwVZWmz7dK+rwkuftXJH2vWvudxC7V338vIQf6VyVdYmabzewctSY9D3Ycc1DSLe3HN0g67O3ZhYrq2+d26eHP1QrzqtdTV/Xst7u/4u4b3H3a3afVmju4zt3nymlubtK8xmfVmgiXmW1QqwTzbKGtzFeaPr8g6WpJMrMfVSvQFwttZTkOSvrF9mqXqyS94u4nBvoJZc/89pkVvlbSv6k1K/6R9nO/q9Yfs9T6Rf+tpGck/Yuki8tucwF9/kdJ35L0ePufg2W3uYh+dxz7oCJY5ZLy922SPibpKUlHJd1YdpsL6PNlkh5WawXM45J+quw259Tvz0k6IWlFrdH4rZI+IOkDa37Xd7T/vxwd5jXOlaIAEImQSy4AgAEQ6AAQCQIdACJBoANAJAh0AIgEgQ4AkSDQASASBDoAROL/AZsGnJzp4mhRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c6f89fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = make_data(40, error=1)\n",
    "plt.scatter(X.ravel(), y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now say we want to perform a regression on this data.  Let's use the built-in linear regression function to compute a fit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VNX9//HXxwASEAUFlVUEhZ9iVWz6xaoVVJSiFahb3cWq1A1pa7Fu/Wq/aqWiVVRccEPApS4UEVFKpa1KixYEEaEogiwBEYUgSIAQzu+Pc6NDyCSz35mb9/PxyIOZO5O5n5uE95w595xzzTmHiIgUvl3CLkBERDJDgS4iEhEKdBGRiFCgi4hEhAJdRCQiFOgiIhGhQBdJgJk5Mzsg7DpEaqNAF4kgM2tkZi+Z2WfBm1GvOp6/sdpXpZk9EPP4pWa2KHjsDTNrk/WDkKQp0KXeMbOiEPfdIJFtyb5GHO8A5wOf1/VE59xuVV/APkA58GKwv57AH4D+wJ7AEuC5ZGqW3FCgR1TQMhtqZnPN7Bsze8LM9jGz181sg5n9zcxaxDz/SDP7l5mVmdkHsS06M7vYzBYE37fYzH4R81gvM1thZtea2RdmtsrMLq6lroHBa2wwsyVmdl6wvcjM7jazL4PHrwpalg1ijqd3zOvcambjYu6/aGafm9l6M3vLzLrFPDbazB42s8lm9g1wnJntGuxvmZmtNrNHzKw45nuGBsey0sx+XsfPeo/g57vKzErN7PaqN43geKeb2b1mtha4Nc62XczsZjNbGvwcx5jZHsFrdAx+FpeY2TJgWm31ADjntjrn7nPOvQNU1vX8as4AvgDeDu6fCrzonPvIObcVuA041sw6J/m6kmUK9Gg7HTgR6IL/T/k6cCPQEv+7vwbAzNoCrwG341tgvwFeNrNWwet8AfwE2B24GLjXzI6I2c++wB5AW+ASYGTsm0UVM2sK3A/0dc41A44C5gQPXxbsoztQgg+VZLwOHAjsDbwPPFPt8XOBO4Bm+JbrH/E/l8OBA4La/zeo88fBz+DE4DV7U7ungW3B63QHTgIujXm8B7A4qO2OONsGBl/HAZ2A3YAHq+2nJ3AQ0Ceoc66ZnVtHbam4CBjjvlsXxIIvYu4DHJKFfUs6nHP6iuAX8BlwXsz9l4GHY+4PBiYEt38LjK32/VOAi+K89gRgSHC7F/7jeYOYx78Ajqzh+5oCZfg3muJqj00DLo+5fxLgql43OJ7eMY/fCoyLU1/z4Hv3CO6PxgdU1eMGfAN0jtn2Q2BJcPtJYFjMY12C1zughn3tA2yJPR7gHODvwe2BwLJq31PTtjeBK2PudwUqgAZAx2D/nVL8W1gB9ErwuR3wLfr9Y7adAHwJHAoUA48C24Fzwv4719eOX2qhR9vqmNvlNdzfLbi9H3Bm0N1SZmZlwDFAawAz62tmM8xsbfDYyfhWfpWvnHPbYu5vinntbznnvgF+BlwOrDKz18zs/wUPtwGWxzx9aaIHGXTXDDOzT83sa3z4U63G2NduBTQBZsUc7xvB9mRr2Q9oGBxP1Ws9im9517TveNvaVNvPUnyY71PH62TahcA7zrklVRucc28Ct+AbBUvxP98N+DcKySMKdAEfFGOdc81jvpo654aZ2a74/8h3A/s455oDk9nxI3jCnHNTnHMn4t8s/gs8Fjy0Cmgf89QO1b71G3wIV9k35va5+BN2vfFdPx2D7bE1xi4r+iX+Da1bzPHu4fwJwURqibUc30JvGfNauzvnusU8p6YlTatvW4l/c4jd5zZ2fBPOxdKoF+K7kHbgnBvpnDvQObc3/u+hATAvB/VIEhToAjAOONXM+gSt3cbByc52QCNgV2ANsM3M+uK7Q5IWnJTtF/SlbwE28t0JuxeAa8ysXdD/fn21b58DnG1mDc2seh97s+D1vsKH/h9qq8M5tx3/RnKvme0d1NbWzPrE1DLQzA42syb41mm811oF/BW4x8x2D05udjY/MiQZzwG/MrP9zWy34Bj+XO2TT1KCE7+Ng7uNgt9r3DdiMzsKfy7hxWrbG5vZIeZ1AEYBI5xz61KtTbJDgS4455bjW7g34oN7OTAU2MU5twF/8vQFYB2+NTwxxV3tAlyLb42uxZ/kuzJ47DF8v/0H+JOa46t97++AzkENvweejXlsDL4roBSYD8xIoJbfAouAGUE3zd/w/dY4514H7sP36y+i7lElF+Lf+OYH9b1E0F2VhCeBscBb+GGBm/HnOeIys4+qRgnFsRD/SaQt/mdbTvApwMxuNLPXqz3/ImB88DuP1Rj/894IvAf8G//7kDxjzukCF5J/zKwjPtgaptNKFalP1EIXEYkIBbqISESoy0VEJCLUQhcRiYikFgVKV8uWLV3Hjh1zuUsRkYI3a9asL51zrep6Xk4DvWPHjsycOTOXuxQRKXhmltDMaXW5iIhEhAJdRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiQoEuIhIRCnQRkWz64AP41a+goiLru1Kgi4hkWmUlTJgAvXrB4YfDqFHw4YdZ360CXUQkU8rK4E9/ggMPhJ/+FJYsgeHDYcUKOOKIrO8+p1P/RUQi6eOP4f77YfRo+OYb+NGPfJD37w8NchezCnQRkVQ4B1OnwogRMHkyNGoE55wDQ4ZA9+6hlKRAFxFJxjffwNixvkW+YAHssw/ceitcfrm/HSIFuohIIpYtg5Ej4bHHYN06+P73YcwYOOss2HXXsKsDFOgiBW/C7FKGT1nIyrJy2jQvZmifrgzo3jbssqLBOZg+3Xer/OUv/v5pp8EvfwlHHQVmYVe4AwW6SAGbMLuUG8Z/SHlFJQClZeXcMN4Pj1Oop2HLFnjhBbjvPnj/fWjRAq69Fq66Cjp0CLu6uDRsUaSADZ+y8Nswr1JeUcnwKQtDqqjArV4Nv/897LcfXHghlJfDI4/A8uXwxz/mdZiDWugiBW1lWXlS2yWO99/33SrPPw9bt8LJJ/vRKieemHfdKrVRoIsUsDbNiymtIbzbNC8OoZoCs20bvPKKD/K334amTWHQIBg8GLp0Cbu6lKjLRaSADe3TleKGRTtsK25YxNA+XUOqqACsWwd33w2dO8MZZ/julHvu8bM5H3igYMMc1EIXKWhVJz41yiUB//2vHzv+9NOwaRP07Olb56eeCkVFdX9/AVCgixS4Ad3bKsDj2b4dpkzxwT1lih8vfu65cM01ftGsiFGgi0j0bNzoJ/3cfz8sXAitW8Ntt8EvfgGtWoVdXdYo0EUkOj77DB58EB5/HNavhx/8AMaNgzPP9GutRJwCXUQKm3N+lMqIEX4NcjN/snPIEDjyyIIadpguBbqIFKbNm/248REjYM4c2HNPuO46uPJKaN8+7OpCoUAXkYzK+toyq1bBww/7GZxr1kC3bv6KQOedB02aZG4/BUiBLiIZk9W1ZWbO9K3xP//ZTwo65RS/SNbxx9erbpXaaGKRiGRMxteW2bbNL5J19NH+BOeECXDFFf4KQa++CiecoDCPUWegm9mTZvaFmc2L2banmU01s0+Cf1tkt0wRKQQZW1tm7Vq/GFanTvCzn8Hnn/uVD0tLfSv9gAMyUG30JNJCHw38uNq264E3nXMHAm8G90Wknou3hkzCa8t89JEfK96uHVx/vb/Y8iuv+Bb5kCGw++4ZrDZ66gx059xbwNpqm/sDTwe3nwYGZLguESlAKa0ts307vPYanHQSHHKInxB03nkwdy68+Sb065fw1PwJs0s5etg09r/+NY4eNo0Js0vTOZyCk+pJ0X2cc6sAnHOrzGzveE80s0HAIIAOeb6WsIikJ6m1ZTZsgNGj/YJYn3wCbdrAHXf4FQ9btkx637rYB5hzru4nmXUEJjnnDgnulznnmsc8vs45V2c/eklJiZs5c2bq1YpI4Vu82M/mfOIJ+PprP/lnyBA4/XRo2DDllz162LQalxJu27yY6dcfn07FoTOzWc65krqel2oLfbWZtQ5a562BL1J8HRGpD5yDf/zDn9CcONF3oZx5pg/yHj0ysgtd7CP1QJ8IXAQMC/59JWMViUh0lJfDs8/6RbLmzvVdKTfe6Icett25GySdSUm62EdiwxafA/4NdDWzFWZ2CT7ITzSzT4ATg/siIt7KlXDzzf4anJde6lvoTzwBy5bB7bfHDfMbxn9IaVk5ju/6wBM9samLfSTQQnfOnRPnoRMyXIuIFLr33vPjxV98ESor/QiVIUOgV686JwDVNikpkVa6Lvahqf8iBSPra6SkqqICXn7Z94/PmOHHig8eDFdf7ScGJSgTfeD1/WIfCnSRApCXQ/K+/NIvivXQQ34G5wEH+L7ygQOhWbOkX0594OnTWi4idciHySoZXyMlHfPmwWWX+SVqb7oJDj4YJk3yVwYaPDilMAf1gWeCWugitciXlnHoQ/IqK/1szhEjYNo0KC6GCy/01+bs1i0ju1AfePoU6CK1SPdEXaaE1h3x9dfw1FO+K2XxYr/GyrBhfuTKXnul/fI1nRco9ElAYVKXi0gtQm8ZB3LeHbFokR+d0q6dX3N83339MrZLlsBvf5uxME9nmKLsTIEuUou0Vw/MkAHd23Lnad+jbfNiDD+d/c7TvpfZTwnO+cWwTj0VunTxVwXq3x/+8x+YPt3P7GyQuQ/1eXVeICLU5SJSi6F9uu7Qhw7hnajL2pC88nIYN853q8ybB61awe9+B5dfDq1bZ35/gZq6kGrbLnVToEvBCGMcdiGcqEv557JiBYwc6Ycerl0Lhx/u+8vPPhsaN8563UVmVNawOGCRrkCUMgW6FIQwR5vk82SVpH8uzvnJPyNGwEsv+fv9+/t+8h/9KKHLuWXqjbWmMK9tu9RNfehSENTfWrOEfy5bt/pFsnr0gKOOgjfe8CH+6acwfjwce2zCYZ6pE5lt45yHiLdd6qZAl4KQL6NN8k2dP5c1a/xiWB07+qsArV/vu1lWrIC77/bbk5DJN1ZNJMo8dblIQdC08JrF+7kcW74Sfv5z3yrfsgX69PGrHfbpA7uk3o7L5BtrIZyfKDQKdCkI+TTaJF9MmF3KN1u2fXt/l+2V9F70Hpe8/yo9ls6FJk18qA8eDAcdlJF9ZvqNNZ/PTxQiBboUhKr/9LdO/Iiy8goAGjesvz2GsSdDm235hrM++CsXvT+JDutXs2nftnDXXX42Z4s6rwyZlEy/sebtCpIFSoEuBWXLtu3f3l63qSL8FQdDMnzKQvZdvYyLZr3KmR/+jaYVm3m3XTce/cnl3DH65oxOAIqVyW6SfFknJ0oU6FIw8mVdlVA5B1OncttjN3H84plsKWrApIOO5cnv9+OjfQ/AgDuSCPNUWsiZ6ibR7zPzFOiStlx9bK7XI102bYKxY/348QULOGy3Ftx79Lk8070vXzb9rlslmb7ssFvI9fr3mSUKdElLLkOhXo50WbbMDzN87DFYtw6OOALGjGF6l6MZNenjtPqyw24hZ/v3WR/75+vvWaWIyvXFGHI54afejFt2Dt55xy+G1amTHy9+wgnw9tswcyZccAH9enRKe7GusFvI2fx91teVHNVCj5AwPkLnMhQiP255yxa/RO2IETBrFjRvDr/+NVx1Fey3305PT7cvO+xPPNn8fYb96SMsCvQICeOPONehEMlxy6tXwyOP+OVqV6/2Y8YffhguuACaNs3abvNhbH+2fp9hf/oIi7pcIiSMP+J60w2SDbNn+wsqd+gAt94K3/8+TJkCH33kl67NYphDjtZYD0m+rGOfa2qhR0gYH6Ej3w2Sadu2wSuv+G6Vt9/2oX3ZZX42Z9fMvwnWdWIwkp94yI9PH2FQoEdIWH/EUQ2FjFq3zq+l8uCDsHSpXxTrnnv81PzmzbOyy7CHJYapvjY00gp0M/sVcCnggA+Bi51zmzNRmCSvvv4R57X//tdfCejpp/1Y8p494d57oV8/KCqq+/vTUF9PDFapjw2NlAPdzNoC1wAHO+fKzewF4GxgdIZqkxTUxz/ivLN9u+8LHzHC/7vrrnDuuXDNNf6qQBlUW5dKfT0xWJ+l2+XSACg2swqgCbAy/ZJEapb3E0U2boQxY3yLfOFC2HdfuO02GDQI9t4747urq0sl7GGJknspj3JxzpUCdwPLgFXAeufcX6s/z8wGmdlMM5u5Zs2a1CuVei2vJ4p89hn85jfQrp0fM7777v6iy0uXws03ZyXMoe5JXRqBVP+kHOhm1gLoD+wPtAGamtn51Z/nnBvlnCtxzpW0atUq9Uolb+VidmreXYLOOXjrLTj9dOjcGe67z1884l//gnff9VcHatQoqyXU1aUS5WGJUrN0ulx6A0ucc2sAzGw8cBQwLhOFSWHI1UiKvOkP3rwZnn/e94/PmQN77gnXXQdXXgnt2+e0lES6VHROpX5JZ2LRMuBIM2tiZgacACzITFlSKHLVcg59osjnn8Mtt/gp+BdfDBUVMGoULF8Od96Z8zAHdanIzlJuoTvn3jWzl4D3gW3AbGBUpgqTwpCrlnNoE0VmzoQRI9j+/J+hcht/71TCxFN/w3FXngNmDL9/RmgnaTVMVapLa5SLc+4W4JYM1SIFKFcjKXIaXtu2wV/+4rtVpk+noklTnut+Mk90P4WlLdoAMPmluWBQUemA8CbtqEtFYmmmqKQlly3nrIfX2rV+3fGRI31XSqdOcO+9nFLWiY8379i1UbHd7fTt9WnSjuQnBbqkJdcf+7MyFn3+fN8aHzsWysvh+OP9FP1TToGiIj65/rWEX0qTdiRMCnRJW10t50yFcEZH1GzfDq+/7oN86lRo3BjOP9/P5vze93Z4arxupZpo0o6ESYEuWZVqCNf0JpCRtUk2bIDRo+GBB+CTT6BNG7jjDj+bs2XLGr+lpm6lhrvYDn3ooBEmEj4FumRVKiEc702g+utUSaibY/Fi343yxBPw9dfQowc895yfGNSwYa3fGq9bqaZt6j+XMCnQJatSGdYY702gyIxKt/PJyLjdHM7BP//pZ3FOnOhXNzzzTBgyxAd6EuJ1KynAJZ8o0CWrUhnWGC/sK52juGFR3SNqNm+GZ5/1/eNz58Jee8GNN8IVV0BbBbBEly5BJ1mVymzGeGFftRZJ3LVJVq70i2G1bw+XXOJb6I8/7ocg3n67wlwiTy10yapUhjXWNra9xq6P997zrfEXXoDKSn/xiCFDoFcvMMvGYYnkJQW6ZF2yE4ISehOoqICXX/ZBPmOGX7L26qv9V+fOmT4EkYKgQJe8FPdN4Kuv/KJYI0dCaSkccIC/oMTAgdCsWc7rFMknCnQpDPPm+db4uHH+pGfv3vDoo9C3L+yiU0EioECXfLZ9O7z2mh92OG0aFBfDhRf62ZzduoVdnUjeUaBL/vn6a3jqKT+b89NP/aXdhg2DSy/1QxBFpEYKdMkfixb5EH/qKT9F/6ij4A9/gJ/+tM7ZnCKiQJewOee7U0aMgEmToEEDOOssP+zwBz8IuzqRgqJAl5SkvYJiebk/wXn//f6EZ6tWflLQFVdA69bZK1wkwhTokrS0lrFdsQIeesiPUFm7Fg4/3HexnH22X8JWRFKm8V6StJQuDD1jhg/tjh3hj3+Enj39wlnvv+/HkCvMRdKmFrokLeEVFLduhZde8v3j770He+wBv/yln83ZsWP2CxWpZxTokrQ6V1Bcs8Z3qTz0EKxaBV26+LXIL7oIdtstx9WK1B/qcpGkxVtB8bb9K+HnP/erHf7ud3DooTB5MixYAFddpTAXyTK10CVpsYtnfb52I2eumsO1/32DVrf/G5o0gYsv9rM5Dzoo5EpF6hcFuqRkQKfdGLDrB/DSA7BkCXToAHfd5WdztmgRdnki9ZICXZLz8cd+Nufo0bBxIxxzDAwfDv37+0lBIhKatP4Hmllz4HHgEMABP3fO/TsThUkecQ6mTvWjVSZPhkaN/BDEIUPgiCPCrk5EAuk2qUYAbzjnzjCzRkCTDNQk+WLTJhg71s/mnD8f9tkHbr0VLr/c3xaRvJJyoJvZ7sCxwEAA59xWYGtmypJQLVvmLyDx2GOwbp1vhT/9NPzsZ7DrrmFXJyJxpNNC7wSsAZ4ys8OAWcAQ59w3GalMcss5+Ne/fLfK+PH+/mmn+W6Vo4/WtTlFCkA6gd4AOAIY7Jx718xGANcDv4t9kpkNAgYBdOjQIY3dSTxpLZS1ZYu/uPKIETBrFjRvDr/+tR83vt9+2S1cRDIqnUBfAaxwzr0b3H8JH+g7cM6NAkYBlJSUuDT2JzVIeaGs1au/m825erUfM/7ww3DBBdC0aS5KF5EMSznQnXOfm9lyM+vqnFsInADMz1xpkkjLu7aFsmoM9NmzfWv8uef8Wit9+/r1VU48Ud0qIgUu3VEug4FnghEui4GL0y9JIPGWd7yFskrLytn/+tdo07yY607oTP9lM32Qv/22b4FfdhkMHgxdu2b/YEQkJ9IKdOfcHKAkQ7VIjERb3vEWygJotnkjJ08ZT8mwSbD+C7/C4d13wyWX+L5yEYkUTe3LU4kuUTu0T9cdWvIAnb9azsBZr3L6vDdpUrGFGe0P4cFTr+LO0TdBUVH1lxSRiFCg56k6l6gNVLXW7359AZ3n/IuLZ06k15JZbClqwMSDevFUST/m79MJA+5UmItEmgI9T9XU8i5uWMTQPtX6vDduZMC/X2HAmPth4UK+bLYn9xxzHs8e3pevmn7XrVL9jUBEokeBngOpjBOPXaK2xu9butRfNOLxx6GsDEpKYNw4ph94FI+/urDuNwIRiRwFepalc0HlAd3b7vgc5+Ctt/xolQkT/DDD00/3szl/+EMwoz/gGjZKfaKRiBQsBXqWJT1OvCZbtsDzz/sgnz0b9twTrrsOrrzSXx2omp3eCESkXtAl6LIs4Qsq1+Tzz+GWW/zFIwYOhC1bmH3TMI4fMpb93TEc/cwnTJhdmtmCRaRgKdCzLN7JyNpOUv79mdd5o/uJbG3bnu233cbnXQ+FqVOZMHYK59phLN7kcHzXfaNQFxFQoGddvAsq73SScts2ePFFvur+Pxx3/skcM386z3Tvy3GXPcpxPa9lwl4HMfyvH8ftvhERUR96ltU5WmXtWr/u+MiRsHw5m/dszf8dfxkvHtqbDbsGi2QFoZ1W942IRJ4CPQdqPEk5f76/EtCYMVBeDscdBw8+yLHvQOUuO08AqnozSGSykYjUTwr0LNpp/PmJBzLg87l+tMrUqf7qP+efD9dcA4ceCsC+86fFDe2EJxuJSL2kQM+S2PHnTbds4oQ3X+Xw4ZNgbSm0aQN33AGDBkHLljt8X22hXWf3jYjUawr0LBk+ZSF7rSll4KxXOWvuVHbfuonZrbvyp1OHMqdHb37dtxsDqoU51N3nrjHmIhKPAj3TnIN//pP/ffImei96j+1mTO56DE+V9GNOm6BrZENFrbNFFdoikgoFeqZs3gzPPuv7x+fOpUeT3Xn4yDMY2/1kVjfbuSWe9GxREZE6KNDTtXKlvy7no4/Cl1/C974Hjz/OWwf3ZOTkRTuNG9/hWzXcUEQySIGeqvfe863xF16Ayko49VR/bc5evcCMfsD2xsUMn7Iw7hWFNNxQRDJJgZ6MigoYPx7uuw9mzIBmzeDqq/1X5847Pb2qL7z6ioug4YYiknkK9ER89RWMGuVnc5aWwgEH+ElBAwf6UK+DhhuKSC4o0Gszb57vVhk3zp/07N0bHnkETj4ZdkluGRyNXBGRbFOgV7d9O7z2mg/yN9+Exo3hwgv9bM5u3cKuTkQkLgV6la+/hqeeggcegE8/hXbt4M474bLLYK+9wq5ORKROCvRPP/Uh/uSTsGEDHHUU/OEP8NOfQsOGYVcnIpKw+hnozsG0ab5bZdIkaNAAzjrLX5vzBz8IuzoRkZTUr0AvL4dnnvFBPm8etGoFN98MV1wBrVuHXZ2ISFrSDnQzKwJmAqXOuZ+kX1IWrFjhZ3OOGuWHIB52mO8vP/tsf9JTRCQCMtFCHwIsAHbPwGtl1owZfhLQyy/70Sv9+/tulWOPBbOwqxMRyai0rilqZu2AU4DHM1NOBmzd6hfJ6tEDfvhDeOMNP+Rw0SI/y7NnT4W5iERSui30+4DrgLjTJc1sEDAIoEOHDmnurhZr1nw3m3PVKujSBR58EC66CHbbLeWX3emqQ5rhKSJ5KuVAN7OfAF8452aZWa94z3POjQJGAZSUlLhU9xfX3OCSbs88A1u2wEknwRNPQJ8+Sc/mrK76GiylZeW1rmMuIhKmdBLvaKCfmX0GPA8cb2bjMlJVXSorYcIEf2Hlww6D55+Hiy/2F16eMgX69k07zMGvvVJ9+duqdcxFRPJNyi1059wNwA0AQQv9N8658zNUV83Wr/cTgB54AJYsgQ4d4K674NJLoUWLjO8u3nrlWsdcRPJRYYxD/+QTv7rh6NGwcSMcc4wP8gED/KSgLGnTvLjGtcy1jrmI5KP0+yUA59w/sjoG/dJL/QnP006DWbPg7bfhjDOyGuYAQ/t0pbhh0Q7btI65iOSrwmihP/SQXyBr331zulutYy4ihaQwAj3EZWu1jrmIFIqMdLmIiEj4FOgiIhGhQBcRiQgFuohIRCjQRUQiQoEuIhIRCnQRkYhQoIuIRIQCXUQkIhToIiIRoUAXEYkIBbqISEQo0EVEIkKBLiISEQp0EZGIKIz10PPEhNmlutiFiOQtBXqCJswu5YbxH1JeUQlAaVk5N4z/EEChLiJ5QV0uCRo+ZeG3YV6lvKKS4VMWhlSRiMiOFOgJWllWntR2EZFcU6AnqE3z4qS2i4jkmgI9QUP7dKW4YdEO24obFjG0T9eQKhIR2ZFOiiao6sSnRrmISL5SoMcRb4iiAlxE8lXKXS5m1t7M/m5mC8zsIzMbksnCwlQ1RLG0rBzHd0MUJ8wuDbs0EZG40ulD3wZc65w7CDgSuMrMDs5MWeHSEEURKUQpB7pzbpVz7v3g9gZgARCJ/ggNURSRQpSRUS5m1hHoDrxbw2ODzGymmc1cs2ZNJnaXdRqiKCKFKO1AN7PdgJeBXzrnvq7+uHNulHOuxDlX0qpVq3R3lxMaoigihSitUS5m1hAf5s8458ZnpqTwaYiiiBSilAPdzAx4AljgnPtT5krKDxqiKCKFJp0ul6OBC4DjzWxO8HVyhuoSEZEkpdw/RaNNAAAF9klEQVRCd869A1gGaxERkTRoLRcRkYhQoIuIRIQCXUQkIhToIiIRoUAXEYkIBbqISEQo0EVEIkKBLiISEQp0EZGIUKCLiESEAl1EJCIU6CIiEaFAFxGJCAW6iEhEKNBFRCJCgS4iEhEKdBGRiFCgi4hEhAJdRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiQoEuIhIRCnQRkYhQoIuIRERagW5mPzazhWa2yMyuz1RRIiKSvJQD3cyKgJFAX+Bg4BwzOzhThYmISHIapPG9/wMscs4tBjCz54H+wPxMFJZLE2aXMnzKQlaWldOmeTFD+3RlQPe2YZclIpKUdLpc2gLLY+6vCLbtwMwGmdlMM5u5Zs2aNHaXHRNml3LD+A8pLSvHAaVl5dww/kMmzC4NuzQRkaSkE+hWwza30wbnRjnnSpxzJa1atUpjd9kxfMpCyisqd9hWXlHJ8CkLQ6pIRCQ16QT6CqB9zP12wMr0ysm9lWXlSW0XEclX6QT6f4ADzWx/M2sEnA1MzExZudOmeXFS20VE8lXKge6c2wZcDUwBFgAvOOc+ylRhuTK0T1eKGxbtsK24YRFD+3QNqSIRkdSkM8oF59xkYHKGaglF1WgWjXIRkUKXVqBHxYDubRXgIlLwNPVfRCQiFOgiIhGhQBcRiQgFuohIRCjQRUQiwpzbabZ+9nZmtgZYmrMdJq8l8GXYRWSIjiU/ReVYonIcUBjHsp9zrs61U3Ia6PnOzGY650rCriMTdCz5KSrHEpXjgGgdi7pcREQiQoEuIhIRCvQdjQq7gAzSseSnqBxLVI4DInQs6kMXEYkItdBFRCJCgS4iEhH1MtDN7MdmttDMFpnZ9TU8vquZ/Tl4/F0z65j7KhOTwLH82szmm9lcM3vTzPYLo85E1HUsMc87w8ycmeXlULNEjsPMzgp+Lx+Z2bO5rjFRCfx9dTCzv5vZ7OBv7OQw6qyLmT1pZl+Y2bw4j5uZ3R8c51wzOyLXNWaEc65efQFFwKdAJ6AR8AFwcLXnXAk8Etw+G/hz2HWncSzHAU2C21cU8rEEz2sGvAXMAErCrjvF38mBwGygRXB/77DrTuNYRgFXBLcPBj4Lu+44x3IscAQwL87jJwOv46+VfCTwbtg1p/JVH1vo/wMscs4tds5tBZ4H+ld7Tn/g6eD2S8AJZlbTRbHDVuexOOf+7pzbFNydgb/2az5K5PcCcBtwF7A5l8UlIZHjuAwY6ZxbB+Cc+yLHNSYqkWNxwO7B7T3I0+sKO+feAtbW8pT+wBjnzQCam1nr3FSXOfUx0NsCy2Purwi21fgc5y+1tx7YKyfVJSeRY4l1Cb4Vko/qPBYz6w60d85NymVhSUrkd9IF6GJm081shpn9OGfVJSeRY7kVON/MVuCvXjY4N6VlXLL/l/JSfbxiUU0t7epjNxN5Tj5IuE4zOx8oAXpmtaLU1XosZrYLcC8wMFcFpSiR30kDfLdLL/wnprfN7BDnXFmWa0tWIsdyDjDaOXePmf0QGBscy/bsl5dRhfJ/vlb1sYW+Amgfc78dO39M/PY5ZtYA/1Gyto9rYUnkWDCz3sBNQD/n3JYc1Zasuo6lGXAI8A8z+wzfzzkxD0+MJvr39YpzrsI5twRYiA/4fJPIsVwCvADgnPs30Bi/2FWhSej/Ur6rj4H+H+BAM9vfzBrhT3pOrPacicBFwe0zgGkuOHOSZ+o8lqCb4lF8mOdrXy3UcSzOufXOuZbOuY7OuY748wH9nHMzwyk3rkT+vibgT1ZjZi3xXTCLc1plYhI5lmXACQBmdhA+0NfktMrMmAhcGIx2ORJY75xbFXZRSQv7rGwYX/gz2h/jz+DfFGz7P3xAgP+jfBFYBLwHdAq75jSO5W/AamBO8DUx7JpTPZZqz/0HeTjKJcHfiQF/AuYDHwJnh11zGsdyMDAdPwJmDnBS2DXHOY7ngFVABb41fglwOXB5zO9kZHCcH+br31ZdX5r6LyISEfWxy0VEJJIU6CIiEaFAFxGJCAW6iEhEKNBFRCJCgS4iEhEKdBGRiPj/DSDn6lW+q9IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c6f49320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_test = np.linspace(-0.1, 1.1, 500)[:, None]\n",
    "\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error\n",
    "model = LinearRegression()\n",
    "model.fit(X, y)\n",
    "y_test = model.predict(X_test)\n",
    "\n",
    "plt.scatter(X.ravel(), y)\n",
    "plt.plot(X_test.ravel(), y_test,c='r')\n",
    "plt.title(\"mean squared error: {0:.3g}\".format(mean_squared_error(model.predict(X), y)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have fit a straight line to the data, but clearly this model is not a good choice.  We say that this model is **biased**, or that it **under-fits** the data.\n",
    "\n",
    "Let's try to improve this by creating a more complicated model.  We can do this by adding degrees of freedom, and computing a polynomial regression over the inputs. Scikit-learn makes this easy with the ``PolynomialFeatures`` preprocessor, which can be pipelined with a linear regression.\n",
    "\n",
    "Let's make a convenience routine to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll use this to fit a quadratic curve to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_poly = PolynomialFeatures(degree=2).fit_transform(X)\n",
    "X_test_poly = PolynomialFeatures(degree=2).fit_transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XeYFFXWx/HvAQYdEEEECYMCJhQT6BgQA2sC1wAqrogYENPqa1pkFzOiLqyoKOuuijlgQEUwoBjAFXENKCoCYgKBgRVUUNBRYbjvH7cGmqF7pns6VIff53n6menq6q5TE07fPvfWveacQ0REcl+dsAMQEZHUUEIXEckTSugiInlCCV1EJE8ooYuI5AkldBGRPKGELhIHM3Nmtn3YcYhURwldJE+Z2aFm9pmZ/WJmU8ysbTX77m9m75nZSjP7xMwOiHjsKDN7y8xWmNn/zOweM2uUmbOQRCihS8Exs7ohHrtePNsSfY0o+zQDxgFXA02B6cCTMfZtCjwHjACaADcBz5vZFsEujYEbgNbAzkCbYF/JMkroecrM5pvZoKC19bOZ3WdmLczspaAV9lrEPyxmtp+ZvR20wj42s24Rj/U3sznB8742s3MjHutmZovMbKCZLTWzJWbWv5q4zgheY6WZzTOzU4Ltdc3sZjP7Lnj8gqDMUS/ifA6LeJ0hZvZoxP2ngtbjj2b2ppntEvHYg2Z2p5lNNLOfgT+Y2SbB8RaY2bdmdpeZFUc8Z1BwLovN7MwaftaNg5/vEjMrM7MbKt80gvOdZmYjzewHYEiMbXXM7Coz+yb4OT5sZo2D12gX/CwGmNkCYHJ18QSOB2Y5555yzv0KDAH2MLOdouy7P/BtsG+Fc+5RYFnwGjjnHnPOveyc+8U5txy4B+gaRwySYUro+e0E4HBgR+AY4CXgCqAZ/nd/EYCZlQAv4lthTYHLgGfMrHnwOkuBo4HNgf7ASDPbM+I4LfGtuBJgAPCvyDeLSmbWEBgFHOmca4RPJB8FD58dHKMzUAr0TvBcXwJ2ALYCPgTGVHm8L3Aj0Ah4C/gH/ufSCdg+iP2aIM4ewc/g8OA1D6N6DwFrgtfpDBwBnBXx+L7A10FsN8bYdkZw+wOwLbAZcEeV4xyMbyF3D+L8xMz6xohpF+DjyjvOuZ+Br4LtVVlwq7pt1xivfRAwK8ZjEibnnG55eAPmA6dE3H8GuDPi/oXA+OD7vwGPVHn+JOD0GK89Hrg4+L4bUA7Ui3h8KbBflOc1BFbg32iKqzw2GTgv4v4RgKt83eB8Dot4fAjwaIz4mgTPbRzcfxB4OOJxA34GtovY1gWYF3x/PzA84rEdg9fbPsqxWgC/RZ4PcDIwJfj+DGBBledE2/Y6cH7E/Q7AaqAe0C44/rYJ/P7vizyHYNs04Iwo+24Z/F5OBoqA04G1wN1R9j0cWA7sGPbfuG4b39RCz2/fRnxfHuX+ZsH3bYETg3LLCjNbARwAtAIwsyPN7B0z+yF47I/4Vn6l751zayLu/xLx2us430o8CTgPWGJmL0aUAFoDCyN2/ybekwzKNcPN7Csz+wmf/KkSY+RrNwcaAB9EnO/LwfZEY2mLT4JLIl7rbnzLO9qxY21rXeU43+CTeYsaXieWVfhPVJE2B1ZW3dE59z3QE/gL/m+kB/AasChyPzPbD3gM6O2c+zyBWCRDEuqMkby1EN9CP7vqA2a2Cb51fxowwTm32szGs/FH9Lg45yYBk4J69Q34euyBwBJg64hdt6ny1J/xSbhSy4jv++IT0mH4ZN4Y34qMjDFyWtHv8G9ouzjnyqKEWVMskRbiW+jNqrypRYo2pWnVbYvxbw6Rx1yDT7BtqnmdWGbhW9rAunLXdsQolTjn/gPsHexbD1+euSXi+Z3xHadnOudeTyAOySC10AXgUeAYM+setHY3DTo72wD1gU3wnWRrzOxIfDkkYUGn7LFBcvkN34qsCB4eC1xkZm2C+vvgKk//COhjZkVmVrXG3ih4ve/xSf/v1cXhnFuLfyMZaWZbBbGVmFn3iFjOMLOOZtYAuLaa11oCvALcYmabB52b25nZwTX8OKp6HLjUzNqb2WbBOTxZzZtETZ4FdjWzE8xsU3z/wCfOuc+i7WxmnYOf7ebAzcCi4M0XM9sV/wnmQufc87WMRzJACV1wzi3Et3CvwCfuhcAgoI5zbiW+83QsvtXbF99Sq406wEB8a/QHfCff+cFj9+Dr9h/jOzXHVXnu1fgW5nLgOvxH/0oP40sUZcBs4J04Yvkb8CXwTlCmeQ1ft8Y59xJwG76u/yU1jyo5Df/GNzuI72mCclUC7gceAd4E5gG/4vs5YjKzWZWjhKpyzi3D91XcGMS0L9An4rl3mdldEU/5K/6Ty8Ig9uMiHhuIL0fdZ2argps6RbOQBR0dIlnFzNrhE1tREq1UkYKiFrqISJ5QQhcRyRMquYiI5Am10EVE8kRGx6E3a9bMtWvXLpOHFBHJeR988MF3zrnmNe2X0YTerl07pk+fnslDiojkPDOL68pplVxERPKEErqISJ5QQhcRyRNK6CIieUIJXUQkT2j6XJEcN35GGSMmzWXxinJaNylmUPcO9OpcEnZYEgIldJEcNn5GGZePm0n5aj8LcdmKci4fNxNASb0AqeQiksNGTJq7LplXKl9dwYhJc0OKSMKkhC6SwxavKE9ou+Q3JXSRHNa6SXFC2yW/KaGL5LBB3TtQXFR3g23FRXUZ1L1DSBFJmNQpKpLDKjs+NcpFQAldJOf16lyiBC6ASi4iInlDCV1EJE8ooYuI5AkldBGRPKFOURFJKc0tEx4ldBFJGc0tEy6VXEQkZTS3TLhqTOhmdr+ZLTWzTyO2NTWzV83si+DrFukNU0RygeaWCVc8LfQHgR5Vtg0GXnfO7QC8HtwXkQKnuWXCVWNCd869CfxQZXNP4KHg+4eAXimOS0RyUNhzy4yfUUbX4ZNpP/hFug6fzPgZZRk5braobadoC+fcEgDn3BIz2yqFMYlIjgpzbhl1yGZglIuZnQOcA7DNNtuk+3AiErKw5paprkO2UBJ6bUe5fGtmrQCCr0tj7eicG+2cK3XOlTZv3ryWhxMRqZ46ZGuf0J8DTg++Px2YkJpwRERqRx2y8Q1bfBz4L9DBzBaZ2QBgOHC4mX0BHB7cFxEJTdgdstmgxhq6c+7kGA8dmuJYRKTAJTNtgBb70KX/IpIlUjFKpdAX+9Cl/yKSFTRtQPLUQhfJEfk+i6FGqSRPCV2kBtmQSAvhopnWTYopi5K8C2mUSrJUchGpRmUiLVtRjmN9Is30JeWFUI7QKJXkKaGLVCNbEmnS5YiKClizxn9duxacS2F0qdGrcwnDjt+NkibFGFDSpJhhx++WN59AMkElF5FqZEtdt3WTYsqW/0LT8p9otfI7Wq78jlYrv6fd2p/hklfh++/97Ycf4Mcfobx8w9uaNdFfuF49aNgQGjTwXytvTZtC8+b+1qyZ/9qyJWyzjb9ttllKzitaOWva4ENS8tqFSAldpBoZr+uuXQsLFsDnn8Pcuf7r558zafZcipYsZpOK1Rs/591GsOWW/ta0KWy9tU/QxcXrb5tuCma+ZR55W70afv7Z3375xX9dtQoWLoQZM2DZMvj9942P2bQptG3rk/uOO8LOO8NOO/mvTZrEdaqF0C+QaUroItUY1L3DBkkHUljX/flnmDkTPvpo/e2TT3yLulKjRtChA5t13Y8v6jfhheX1mFuvMWtal9C757706LYb1K+ffCyxOOcT/LJlsGSJf7P55ht/W7AAvvgCXnppw6TfogXssgvsuae/7bUXbL891NmwwqvJtFJPCV2kGim7+tA5+PJLePvt9bdZs9bXsrfYAjp1gnPP9clwxx39rUUL37IGdgAujfLSaR2FY+bfVBo1gm23ha5dN95nzRqYPx/mzIHPPvNfZ86EUaPWJ/rNNoPOnaFLFzjgANh//6iffICY26Vm5jLYOVJaWuqmT5+eseNJfsmG4YNxc84nttdf97dp0+C77/xjjRv7xLbvvr4F26mTL5MEiTsRVcsW4D9BZEVn4urVMHu2L918+CG8/z588IHfDnyx5da836Yj7229K2+168R3Df1KlnXN+GrYH8OMPOuY2QfOudIa91NCl1yQ1Ymr0v/+B5Mm+QT+2mu+RAG+ZXvQQbD//v62884blR9qq+vwyVFbtCVNitPWuZjUG2t5OUyfDm+9xeR7x1FaNofNf/sZgNlbtefNdp2Z2n5Pxjw40Nf9BVBClzwTRuKqkXPw6afw3HP+9t57fnuzZnDooXDYYf5r+/ZpC6H94BeJ9h9swLzhR6X8eKl8Y+06fDJLflhFx6XzOGjehxw4fwZ7LZpD/bVrfEfu4YdDz55wzDF+lE0Bizehq4YuOSFbhg+ydi288w6MHQsTJvjaMcDee8P118NRR8Eee6SsBV6TTI/CSWVHZmWH86ctt+fTltvz7y5/Ykv3O3e0WUmXz99f/0ZZp47/ZNOrFxx3nP/EI1HpwiLJCaEuXuCcrwH/9a++td21K9x1F+y6K4weDYsX+9b5VVf5jr8MJXPI/NWVqXxjjXYh0dV99qHLhafBP//p3yxnzICrr/YjbS67DLbbzvc//POf8O23yZ1MHlLJRXJCKDX0r7+Ghx6Cxx/3w/Pq1YMjjoA+fXwpYPPN03PcOFXWsstWlFPXjArnKElzZ3Gopa/58+Gpp+Cxx/wQzzp1fEmrb1/fcm/cOL3HD1G8JRe10CUnVLbmmhQXrdu2aVEa/nzLy2HMGDjkEN8avP56PwJl9Gjf6fnii3DqqVmRzCvnmAGocG5dyzydncSp/kQwfkYZXYdPpv3gF+k6fHL1c+S0aweDBvlW+6xZcPnlfiho//7QqhWccQa89VZWTmuQKUroklN+W7N23ffLf1mduomyZsyACy6A1q2hXz/fGrz+en/xzOuvw9ln+ysxs0RYc8ykcr6VpCY+69gRbrgBvvrKj+k/9VR45hk48EA/jv/WW9cPEy0gKrlIzkj5x/3ff/dJYNQo39G5ySZwwgkwYAB065bRWniiUjW6Jcyx/Sn/fa5a5Tur77nH/z6LiuD44+Hii2G//Wo1zj9bqOQiGZPQx+YkpKxD7ttvYehQPxdJ375+Uqvbb/fjxivLLVmczCE1ncRhTw2c8pFLm20GZ54J//2vn0Lhz3+Gl1/2I2T228/3hayOMhdOHsnuv1rJeplMCkknsY8/htNO8xNKXXutv0Jz4kR/ufpFF/nL73NEKmrZYU8NnNaRS7vtxvgz/sphl47hmsPPY8FXZf7Nu317GDbMv4nnISV0SUomk0Ktk9jUqX58eKdO8Oyzfr6Uzz7zk0odeWTWt8ajSUUtO+yx/ekcclnZ0PiyHB7e82gOHnAn5/a5jqVbbwdXXOE7ui++GBYtSvpY2UQXFuWZTNdEM5kUEpooyznf+h42zM+j0qyZ70Q7//ycaolXJ9kV7sNe8i1lE59FUbWh4awOk9ruxad7HMC0e7aCW26Bf/8b7rwTTj8d/vY3PyNkjlNCzyNhzC+d6aRQYxJbuxbGjfMjVD75xJdXRo3yHZ0NGqQlplyV1qmB45Tsm1Is1TY0dt0VHngAhgyBESPg3nvh/vvhpJP8UMjddkt5PJmSe581JaYwaqJZsw6kc/D8837u7RNPhN9+gwcf9OOUL7xQyTyKfF7yLa76fNu2cMcdfojqZZf5v5/dd/eJfW5urtWqhJ5HwqiJhp4UnINXXvGjGI49FlauhIcf9heenH66H7omMfXqXMK0wYcwb/hRTBt8SF4kc0iwodGyJfzjH37Rjquu8hePdezoR8x8802GIk4NjUPPI1k5I2E6vf02DB7sOz233hquuUZJvIqcmkM+xWp97kuXwvDhvsa+di2cd57vSG3ZMv1Bx5CR6XPN7FLgLMABM4H+zrlfY+2vhJ5eOTFneCp89ZXvxHrmGf9PdtVVcNZZ/sIgWadg/h7SZdEi3xdz333+b+uyy/wEbQ0bZjyUtF9YZGYlwEVAqXNuV6Au0Ke2ryfJC738kW4//ACXXuoXiHj5ZbjuOl8jv+CCgk3m1V3UFfY485zXpg3cfbcf4nrMMf5itB128H0za9fW+PQwJDvKpR5QbGargQbA4uRDkmSka9RAqH77Df71L99a+uknX9scOtRPyFTAahrVFPY487yx/fbwxBN+3Pqll/rJwEaNgpEj4eCDw45uA7VuoTvnyoCbgQXAEuBH59wrqQpMBPBLuu22GwwcyLc778GpF95N+y170fWhORm7RD1b1dQCD3UO+XzUpYufVuCxx/yVpt26+blivv467MjWSabksgXQE2gPtAYamlm/KPudY2bTzWz6smXLah+pFJZvvvH/LD16APD2Px+m26GDmbppq1DmHclGNbXAs2ZIaT4xg5NP9mWYG2+EV1/1sztefz38GrP7MGOSGbZ4GDDPObfMObcaGAfsX3Un59xo51ypc660eYGvC5ivUjo512+/+X+Uyjr53/8OM2cyaFWJ6sFV1NQCz/s+lTAVF/uRL5995hc7ueYa/0ly0qRQw0qmhr4A2M/MGgDlwKGAhrAUmJRenfrKK76D88sv/TS2t97qr/Qk/HlHslE8V3rmZZ9KNikp8fX1s87yf7s9ekDv3r6+3qZNxsNJpob+LvA08CF+yGIdYHSK4pIckZKRFN9952dB7N7df6R9+WV4+ul1yRxUD45GLfAscthhfqqJG26AF16AnXbySb2ioubnppAuLJKkJLXQgnPrRw8sX+4vErrySth00412DXtMdbSLVCA9E0tJjps3z0838eKLsPfefq6Y3XdP6iXjHYeuybkkKbWenGvBAr8AwcSJsM8+fpm3aiZFSufMfDWJVlYa9NTHYLC6wq3blu6J0CRHtG/v54V58kk/z/5ee/kLkq6+OmpjJZXUQpekJNxyXrsW7rrLX+m5dq3vAL3wQqhbd+N9s0SsKRWiydtpFqR2vv8eBg6ERx+F6dP9nPy1oBa6ZERCLeeFC/1FQa+9Bkcc4a/Ca9cuoeOFMTdJIh2vhdxJK1FsuaW/svTqq2G77dJ+OCV0SVpNIynGf7iImcPv4OLn7qCeq2DulcPpfP1fE160N4z53iF2WSnWviIbyUAyB02fK2k28fWP2azvn7j6qX8wp3k7uve/g762B+M/qn6WiGhj28OamyTaBTpFdYyiuhu+IemiHQmbWuiSPhMm0OWUM2jw6ypu7HYm9+3dk7V16kKQhGO1qmO1xKsm80rpLnPEKitF26YOUQmTErqkXnk5/OUvcNddlLXYjr+cdAOfN2+3wS7VJeFYLfG6ZlRE6cTPRJkjVllJCVyyiRK6pNann0KfPn7FoEGDuGDzQ/lm1ZqNdqsuCcdK9hXOUVxUN9Q1MEWymWrokhrO+RXU997bX/k5aRLcdBOXHrVrwhNExUr2lVdC6spIkejUQpfk/fCDn8vi2Wf95fsPPQQtWgC1uyCoujlKNDeJSGxK6JKcd9+FE0+E//0Pbr7ZLwBQZ8MPfokm4TCvChXJZUroUjvOwR13+Kvg2rTxCzaX1nghW9zUEhdJnGrokrhVq6BvXz9PRY8e8MEHKU3mIlI7SuiSmNmzfcfn2LF+8Ynx42GLLcKOSkRQyUUSUTmRf8OGfj6WP/wh7IhEJIJa6FKzigoYNMivpdipE3z4oZK5SBZSC12qt3y5v1Cocnm4kSOhqCjsqEQkCiV0iW3OHL8A7vz5MHo0nH32uofCmMZWRKqnhC7RvfCCH8lSXAyTJ8MBB6x7KKxpbEWkeqqhy4acg2HD4NhjYYcd/CorEckcUrQwtIiknFrost6vv8KAAfDYY74D9N57oUGDjXaLNXmWVusRCZda6OJ99x0cdphP5sOGwZgxUZM5xJ48S6v1iIRLCV3g889hv/18eWXsWBg8uNrl4aKt4KNpbEXCp5JLoZs6FXr1grp1YcoU6NKlxqdo8iyR7KSEXsjGjIEzz4T27WHiRNh227ifqsmzRLKPSi6FyDkYOhT69fMt8rffTiiZi0h2Ugu90KxZA+eeC/ffD6ee6key1K8fdlQikgJJtdDNrImZPW1mn5nZHDOruQAr4fnlFzj+eJ/Mr7rKryykZC6SN5Jtod8OvOyc621m9YHo49wkfMuXwzHH+PLKHXf4eVlEJK/UOqGb2ebAQcAZAM6534HfUxOWpFRZmV/r84sv4Mkn/ZJxIpJ3kim5bAssAx4wsxlmdq+ZNUxRXJIqn30G++8PCxbASy8pmYvksWQSej1gT+BO51xn4GdgcNWdzOwcM5tuZtOXLVuWxOEklvEzyug6fDLtB79I1+GTGT+jzD/w3nt+HpZff4U33oBDDgk1ThFJr2QS+iJgkXPu3eD+0/gEvwHn3GjnXKlzrrR58+ZJHE6iqZz5sGxFOY71Mx9Ou+sJn8AbN4Zp02DPjX41IpJnal1Dd879z8wWmlkH59xc4FBgdupCk3jmHI828+H+c96mdPhw2HknvzBFy5aZDFtEQpLsKJcLgTHBCJevgf7JhyQQ/5zjVWc4PGrOVG574WZmt9iWPd54A5o2zVjMIhKupMahO+c+CsopuzvnejnnlqcqsEIX75zjkTMc9p75GqOeH8GM1h04rc/faX/TfzesqYtIXtOl/1kq3jnHK2c+7Pfhi9w88Tamtd2D008cyo/1izeoqSupi+Q/JfQsFe+c4706l/DUyre44dU7eXX7fTm39zWU1990g320mpBIYVBCz1JxzTnuHAwZwq633wh9+nD47Kn8Wrco6utpNSGR/KeEnqV6dS5h2PG7UdKkGANKmhQz7Pjd1neIOgfXXgvXXeenwH30USgq0mpCIgVMsy1mQDzDD6OJOed4ZTK//no46yy4+26o49+bB3XvsMHoGNBqQiKFQgk9zeIdfhi3oMwSLZlHvqZWExIpPEroaVbd8MOEk2xlMh86FAYM2CiZV9JqQiKFSTX0NIt3+GFcgmT+fOmRbLtlT7re9IaGI4rIOmqhp1nrJsWURUne1XVSRq25T7gHhg7lmT2O4LJD/oyzOsmXb0Qkr6iFnmZxDT+MEG2yrYWXDIbrruOFvXpwWff/w9n6X5vGmItIJbXQ0yzRTsqqNfez3x3HhW8+ysQ9j+CiQ8/fIJlX0hhzEQEl9LSqWjoZeVKnGksjkcm534yJXPnG/Ty/04FccugFtNqiYcLlGxEpHCq5pEmsecpr6sSsTM7Hf/o6N7zyb17dfh8uPXogLZtulnD5RkQKixJ6msQ7W2JVg7p34Ngv/8uIibfzVts9+L+egynadJN1ZZpqrx4VkYJmzrmMHay0tNRNnz49Y8cLU/vBLxLrJ1tSXR194kTW9uzFzJIOnHz8ELbYagtdGCRS4MzsA+dcaU37qYaeJrGGK0I1V4tOmQInnECd3Xdjj8mTmd24cSZCFZE8oZJLmkSrd0faqPzy7rtwzDGw3XYwaZJfC1REJAFqoadJ5HDFWC31dSNa5syBP/4RWrSAV1+FZs0yFaaI5BG10NOoV+cSpg0+hJLqprRduBCOOAKKinwyb9Uqw1GKSL5QQs+AWMMNr9h3K+jeHX76CV5+GbbdNqQIRSQfqOSSAdGuFh180NYc9ZdT4euvfc28U6eQoxSRXKeEniEbTGm7ejUceyy8/z48/TQcfHC4wYlIXlBCz7S1a6F/f19iueceOO64sCMSkTyhGnomOQeXXQZjxsCNN/oVh0REUkQJPZNuuw1GjoSLLoLLLw87GhHJM0romfL00zBwIJxwgk/qZmFHJCJ5Rgk9E95+G/r1gy5d4JFHoq4DKiKSrKQzi5nVNbMZZvZCKgLKO59/7ke0bLMNTJgAxZq7XETSIxVNxYuBOSl4nfyzdCkceaRvkb/0ki7pF5G0Siqhm1kb4Cjg3tSEk0d++cW3zJcsgeef95NuiYikUbLj0G8D/go0SkEs+aOiAk45Bd57D8aNg333DTsiESkAtU7oZnY0sNQ594GZdatmv3OAcwC22Wab2h4uNFXXBY1rsYmBA2H8eBg1Cnr1ykygIlLwkim5dAWONbP5wBPAIWb2aNWdnHOjnXOlzrnS5s2bJ3G4zKvVuqB33gm33w6XXAIXXpixWEVEap3QnXOXO+faOOfaAX2Ayc65fimLLAskvC7oq6/6JH700XDzzRmIUERkPQ2IrsbimhamiPTZZ3DiidCxIzz2GNSNvVqRiEg6pCShO+fecM4dnYrXyiatq1uYItL33/tW+Sab+BEtjdRHLCKZpxZ6NWItTDGoe4f1G37/3V/Ov3AhPPsstG2b4ShFRDxNn1uNaAtTbDDKxTk4/3z4z3/g0Udh//1DjFZECp0Seg02WJiiqpEj4b774Mor/bhzEZEQqeRSWy+84Oc2790bhg4NOxoRESX0Wpk1C04+GfbcEx56SLMnikhWUCZK1PLl0LMnNGzoZ09s0CDsiEREANXQE1NR4VvmCxbAlClQUsMUACIiGaSEnoirroJJk+Duu6Fr17CjERHZgEou8Ro7FoYPh3PPhXPOCTsaEZGNKKHH4+OPoX9/3yofNSrsaEREolJCr8n33/spcJs08Qs9168fdkQiIlGphl6dNWvgpJNg8WKYOhVatgw7IhGRmJTQq/O3v8Hrr8P998M++4QdjYhItVRyieXxx+HWW/385v37hx2NiEiNlNCjmTULzjoLDjwQbrkl7GhEROKihF7VTz/B8cf7Oc2ffBKKisKOSEQkLqqhR3IOBgyAr76CyZOhVauwIxIRiZsSeqTbbvNDE0eMgIMOCjsaEZGEqORS6a23YNAgOO44GDgw7GhERBKmFjrAt9/Cn/4E7dvDAw+AWdTdxs8oi716kYhIyJTQ16yBPn1gxQp4+WVo3DjqbuNnlHH5uJmUr64AoGxFOZePmwmgpC4iWUEll6uugjfe8DMo7r57zN1GTJq7LplXKl9dwYhJc9McoIhIfAo7oU+YAP/4h59B8dRTq9118YryhLaLiGRa4Sb0efPg9NOhtNSPbqlB6ybFCW0XEcm0wkzov//u6+bg5znfdNManzKoeweKi+pusK24qC6DundIR4QiIgkrzE7RK66A996Dp57yI1viUNnxqVEuIpKtCi+hv/CCn5/l/POhd++lGAc1AAAIiElEQVSEntqrc4kSuIhkrVondDPbGngYaAmsBUY7525PVWBpsXChr5t36lTjpFsacy4iuSaZFvoaYKBz7kMzawR8YGavOudmpyi21FqzBk4+2dfPn3yy2rq5xpyLSC6qdaeoc26Jc+7D4PuVwBwge7PdtdfCtGl+vPmOO1a7q8aci0guSskoFzNrB3QG3k3F66XcK6/AsGF+JsW+fWvcXWPORSQXJZ3QzWwz4BngEufcT1EeP8fMppvZ9GXLliV7uMQtWQL9+sHOO8OoUXE9RWPORSQXJZXQzawIn8zHOOfGRdvHOTfaOVfqnCtt3rx5ModLXEUFnHIKrFrlx5s3aBDX0zTmXERyUTKjXAy4D5jjnLs1dSGl0E03wZQpcN99sMsucT9NY85FJBeZc652TzQ7AJgKzMQPWwS4wjk3MdZzSktL3fTp02t1vIS99x507eqXk3viiZhT4oqIZDsz+8A5V1rTfrVuoTvn3gKyM0uuXOk7P1u1grvuUjIXkYKQn1eKXnSRn3xryhTYYouwoxERyYj8m5xr7Fh48EE/X4vWBRWRApJfCf2bb+Ccc2DffeGaa8KORkQko/InoVdU+EUqKipgzBgoKgo7IhGRjMqfGvqwYTB1Kjz8MGy3XdjRiIhkXH600N95B4YM8ZNv9esXdjQiIqHI/YT+00/+atA2beDOOzVEUUQKVu6XXC65BObPhzffhMaNw45GRCQ0ud1CnzABHngALr/cXxUqIlLAcjehL10KZ58NnTtriKKICLma0J3zyfynn+CRR6B+/bAjEhEJXW7W0B94AJ57zq8LmsAsiiIi+Sz3Wujz5sHFF0O3br5DVEREgFxL6BUVcMYZUKeOn6+lTm6FLyKSTrlVchk50g9PfPBBaNs27GhERLJK7jRxZ86EK6+EXr3gtNPCjkZEJOvkRkL/7Tc/8VaTJjB6tK4GFRGJIjdKLkOGwMcf+5EtmV5oWkQkR+RGC93Mz3N+zDFhRyIikrVyo4X+97/7i4lERCSm3Gihg+rmIiI1yJ2ELiIi1VJCFxHJE0roIiJ5QgldRCRPKKGLiOQJJXQRkTyhhC4ikieSSuhm1sPM5prZl2Y2OFVBiYhI4mqd0M2sLvAv4EigI3CymXVMVWAiIpKYZC793wf40jn3NYCZPQH0BGanIrBMGj+jjBGT5rJ4RTmtmxQzqHsHenUuCTssEZGEJFNyKQEWRtxfFGzbgJmdY2bTzWz6smXLkjhceoyfUcbl42ZStqIcB5StKOfycTMZP6Ms7NBERBKSTEKPNrnKRjNoOedGO+dKnXOlzbNw6tsRk+ZSvrpig23lqysYMWluSBGJiNROMgl9EbB1xP02wOLkwsm8xSvKE9ouIpKtkkno7wM7mFl7M6sP9AGeS01YmdO6SXFC20VEslWtE7pzbg3wf8AkYA4w1jk3K1WBZcqg7h0oLqq7wbbioroM6t4hpIhERGonqQUunHMTgYkpiiUUlaNZNMpFRHJdbqxYlGa9OpcogYtIztOl/yIieUIJXUQkTyihi4jkCSV0EZE8oYQuIpInzLmNrtZP38HMlgHfZOyAiWsGfBd2ECmic8lO+XIu+XIekBvn0tY5V+PcKRlN6NnOzKY750rDjiMVdC7ZKV/OJV/OA/LrXFRyERHJE0roIiJ5Qgl9Q6PDDiCFdC7ZKV/OJV/OA/LoXFRDFxHJE2qhi4jkCSV0EZE8UZAJ3cx6mNlcM/vSzAZHeXwTM3syePxdM2uX+SjjE8e5/MXMZpvZJ2b2upm1DSPOeNR0LhH79TYzZ2ZZOdQsnvMwsz8Fv5dZZvZYpmOMVxx/X9uY2RQzmxH8jf0xjDhrYmb3m9lSM/s0xuNmZqOC8/zEzPbMdIwp4ZwrqBtQF/gK2BaoD3wMdKyyz/nAXcH3fYAnw447iXP5A9Ag+P7PuXwuwX6NgDeBd4DSsOOu5e9kB2AGsEVwf6uw407iXEYDfw6+7wjMDzvuGOdyELAn8GmMx/8IvIRfK3k/4N2wY67NrRBb6PsAXzrnvnbO/Q48AfSssk9P4KHg+6eBQ80s2qLYYavxXJxzU5xzvwR338Gv/ZqN4vm9AFwP3AT8msngEhDPeZwN/Ms5txzAObc0wzHGK55zccDmwfeNydJ1hZ1zbwI/VLNLT+Bh570DNDGzVpmJLnUKMaGXAAsj7i8KtkXdx/ml9n4EtsxIdImJ51wiDcC3QrJRjediZp2BrZ1zL2QysATF8zvZEdjRzKaZ2Ttm1iNj0SUmnnMZAvQzs0X41csuzExoKZfo/1JWKsQVi6K1tKuO3Yxnn2wQd5xm1g8oBQ5Oa0S1V+25mFkdYCRwRqYCqqV4fif18GWXbvhPTFPNbFfn3Io0x5aoeM7lZOBB59wtZtYFeCQ4l7XpDy+lcuV/vlqF2EJfBGwdcb8NG39MXLePmdXDf5Ss7uNaWOI5F8zsMOBK4Fjn3G8Zii1RNZ1LI2BX4A0zm4+vcz6XhR2j8f59TXDOrXbOzQPm4hN8tonnXAYAYwGcc/8FNsVPdpVr4vpfynaFmNDfB3Yws/ZmVh/f6flclX2eA04Pvu8NTHZBz0mWqfFcgjLF3fhknq21WqjhXJxzPzrnmjnn2jnn2uH7A451zk0PJ9yY4vn7Go/vrMbMmuFLMF9nNMr4xHMuC4BDAcxsZ3xCX5bRKFPjOeC0YLTLfsCPzrklYQeVsLB7ZcO44Xu0P8f34F8ZbBuKTxDg/yifAr4E3gO2DTvmJM7lNeBb4KPg9lzYMdf2XKrs+wZZOMolzt+JAbcCs4GZQJ+wY07iXDoC0/AjYD4Cjgg75hjn8TiwBFiNb40PAM4Dzov4nfwrOM+Z2fq3VdNNl/6LiOSJQiy5iIjkJSV0EZE8oYQuIpInlNBFRPKEErqISJ5QQhcRyRNK6CIieeL/AVGIYjNsBySHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c6f56cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = LinearRegression()\n",
    "model.fit(X_poly, y)\n",
    "y_test = model.predict(X_test_poly)\n",
    "\n",
    "plt.scatter(X.ravel(), y)\n",
    "plt.plot(X_test.ravel(), y_test,c='r')\n",
    "plt.title(\"mean squared error: {0:.3g}\".format(mean_squared_error(model.predict(X_poly), y)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This reduces the mean squared error, and makes a much better fit.  What happens if we use an even higher-degree polynomial?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXeYFFX2v98zgZyzDEkkCQZQVMyoCBhhjZgDK+u6uru/3UXBsLrrV2XFtGsWZdE1i4oCCiiICgiKAgOIoEiQAclDHCYw9/fH7R5qmu6ZDtWxzvs883R31a1bpzp86sy5554rxhgURVGUzCEr2QYoiqIo7qLCriiKkmGosCuKomQYKuyKoigZhgq7oihKhqHCriiKkmGosCtKFYiIEZFOybZDUSJBhV1RMgwROUtEfhCRvSLymYi0r6LtZyKyWUR2isgiERnk2HeniOx2/BWJSLmINHO06Sci34nIHhH5RUQui/f1KdUjOkFJ8Qoikm2M2R/hMQbobIz5KcZz5xhjyqrbFmkfQdo0A1YCvwUmAvcDpxpj+oRofxTwvTGmTEROAD4FuhhjNgRpex9wmjHmTN/r7sBM4DrgE6Ah0MgYszLca1Lig3rsGYaIrBaR4SKS7/OiXhKRliLysYjsEpFPRaSxo30fEZkjIoU+j62vY98NIrLMd9zPIvI7x76+IrJORP4qIptEZIOI3FCFXdf7+tglIqtE5Crf9mwReUREtvj2/8EX/shxXE8/Rz/3icirjtfviMivIrJDRL4QkR6OfeNE5FkR+UhE9gBniEhN3/nWishGEXlORGo7jhnuu5b1InJjNe91Q9/7u0FECkTk/0Qk23G9s0XkcRHZBtwXYluWiNwtImt87+MrItLQ10cH33sxVETWAjOqssfHRcBSY8w7xph9wH3A0SLSLVhjY0y+42ZhgFygbZBrFeAa4GXH5ruB540xHxtjyowxW1XUUwMV9szkYuBsoAtwAfAxcCfQDPuZ/xFARPKAycD/AU2AvwHvikhzXz+bgPOBBsANwOMicozjPK2wXloeMBR42nnT8CMidYH/AOcYY+oDJwELfbtv8p2jF9AbuCTCa/0Y6Ay0AL4DXgvYfyXwAFAfmAX8C/u+9AQ6+Wz/u8/Ogb734Gxfn/2ompeBMl8/vYD+WE/ZzwnAzz7bHgix7Xrf3xlAR6Ae8FTAeU4HDgcG+OzMF5ErQ9jUA1jkf2GM2YP14HuEaI+ITBKRfcA8rAc+P0izU4GWwLuObX18xy/23dxeFZEmoc6jJBBjjP5l0B+wGrjK8fpd4FnH69uACb7ndwD/Czh+KnBdiL4nAH/yPe8LFAE5jv2bgD5BjqsLFGJvOLUD9s0Abna87o/1HHMc19PPsf8+4NUQ9jXyHdvQ93oc8IpjvwB7gMMc204EVvmejwVGOfZ18fXXKci5WgLFzusBrgA+8z2/HlgbcEywbdOBWxyvuwKlQA7QwXf+jhF8/i85r8G3bTZwfTXH5QLnAP+vin7HBWwr8X0+XbA3pHeB15L9G9A/ox57hrLR8bwoyOt6vuftgUt9YZhCESkETgEOARCRc0Rkrohs8+07F+v1+9lqKsd89zr6rsBYr/Fy4GZgg4hMdoQGWgO/OJqvCfcifWGcUSKyUkR2YkWGABudfTcH6gDfOq53im97pLa0x4rhBkdfz2M98WDnDrWtdcB51mBFvWU1/YRiN/Y/LCcNgF1VHWSMKTXGfAwMEJELnft8oapLqRyGAftd+q8xZoUxZjfwIPY7oiQZFXZv8wvWY2/k+KtrjBklIjWxHtgjQEtjTCPgI6zXGzHGmKnGmLOxN40fgDG+XRuoHNNtF3DoHqwY+2nleH4lMAgbMmmI9XAJsNGZHbAFK0Y9HNfb0BjjvxlVZ4uTX7AeezNHXw2MMc6QR7DMhMBt67E3Cec5y6h8M44kw2EpcLT/hS8Mdphvezjk+No7uQjYhg3TOMmP0DYlQaiwe5tXgQtEZIDP+63lGxRtA9QAagKbgTIROQcbJokY3+DthT6RKcZ6lf7slLeBP4pIG198fkTA4QuBISKSKyKBMfj6vv62YsX/warsMMaUY28oj4tIC59teSIywGHL9SLSXUTqAPdW0dcGYBrwqIg08A2CHiYip1fzdgTyBvD/RORQEannu4a3TATZMgG8DxwhIheLSC3s+EG+MeaHwIYi0s33X1lt3/t7NXAa8HlA0+uwIa1AEf8vcIOIdPS9X3cAk6K0W3ERFXYPY4z5Bevx3okV8F+A4UCWMWYXdpD1bWA71jv+MMpTZQF/xXqn27CDgbf49o3BxvUXYQc/3ws49h6sB7kd+AfwumPfK9jQRQHwPTA3DFvuAH4C5vrCN59i49r4QhFPYOP+P1F9Fsq12Bvg9z77xuMLY0XAWOB/wBfAKmAfdhwkJCKyVHxZRYEYYzZjxzIe8Nl0AjDEcexzIvKc/yV2zGIT9vP/E3C5MeY7R/s84Ezsex14rrG+7fOwn0MxvoF5JbloHruSUohIB6zA5cbgtSqKp1GPXVEUJcMIW9hFZKxvAsWSIPv+5ptI0SzYsYqiKEriiMRjHwcMDNwoIm2xEzrWumST4mGMMauNMaJhGEWJnrCF3RjzBXbgK5DHgdvRtCdFUZSUICeWg30TGQqMMYtsKYkq2w4DhgHUrVv32G7dgpauSC6LFkGjRtA+ZDE8RVESSVmZ/V22bQstWlTf3sn27fDzz9CjB2zYALt3w5FHxsfOBPHtt99uMcY0r65dRFkxvoyFScaYI3x5q58B/Y0xO0RkNdDbGLOlun569+5t5s8PVo4iybRuDeedB2PGVN9WUZT4s2GD/V0+9xz87nfVt3fy9ttw+eWwdCmMGgWzZlmhT2NE5FtjTO/q2sWSFXMYcCiwyCfqbYDvRKRVlUelMiKg6Z+KkjqUltrHnBiCCx78TUf9bhljFuOoixGJx56yZGV58kugKClLmW8MPTc38mOrCQ9nMpGkO74BfAV0FVuHe2j8zEoSIlBenmwrFEXxox57VIT9bhljrqhmf4eYrUk2GopRlNRCPfao0JmnTlTYFSW1cMNjB8/9rlXYnWiMXVFSi1g8dj/+37SHPHgVdicaY1eU1MIv7NF47B4S8kBU2J1oKEZRUgsdPI0KFXYnKuyKklro4GlUqLA70Ri7oqQWbg2eegwVdicaY1eU1MLNwVMPocLuREMxipJaxOKxO0MxHvtdq7A70VCMoqQWmu4YFSrsTjQUoyiphaY7RoUKuxMNxShKaqHpjlGhwu5EhV1RUgtNd4wKFXYnGmNXlNRC0x2jQoXdicbYFSW10HTHqFBhd6KhGEVJLTTdMSoiWWhjrIhsEpEljm2jReQHEckXkfdFpFF8zEwQKuyKklpoumNUROKxjwMGBmz7BDjCGHMUsAIY6ZJdySErS0MxipJKuOWxe4ywhd0Y8wWwLWDbNGOM75bKXOyC1umLDp4qSmpRXGwfa9ZMrh1phpsx9huBj13sL/Gox64oqUVxMWRn279o8aCz5oqwi8hdQBnwWhVthonIfBGZv3nzZjdO6z4q7IqSWpSUQI0a0R2roZjoEZHrgPOBq4wJfWs0xrxgjOltjOndvHnzWE8bH1TYFSX+FBfDiy/CrFnhtY01DONBjz2mrH8RGQjcAZxujNnrjklJRIVdUeLPbbfBmDE202XBAujRI3Rbtzx2j4l7JOmObwBfAV1FZJ2IDAWeAuoDn4jIQhF5Lk52JgYV9sxn7174979h6FB45hnYty/ZFnmLlSutqF9+uY2bP/NM1e3d8Nj9eCg0E7bHboy5Isjml1y0JflkZR3Im1Uyj+3b4fTTYfFiaN4cxo6Fl16CadOgadNkW+cNxo2zv7NHH7VC+9Zb8OSTdlswSko0FBMFOvPUiXrsmUtZGVx2GfzwA3z0EWzaBBMmwNKlcOml+rknivffh759IS8P+veHrVth+fLQ7YuLdfA0ClTYnaiwZy7PPQeffgrPPgvnnGO3DRoETz0Fn31m9yvxZc0aeyM97zz7+pRT7OOXX4Y+Rj32qFBhd6LCnpns2AH/+If1FG+8sfK+oUPh7LNhxAjYti3o4YpLzJhhH/v3t4+dOkHjxvDdd6GPUY89KlTYnWh1x8zkP/+BLVvgkUcO/rGLwGOPwa5d8MQTybHPK8yaBU2aQPfu9rWIzYhZujT0MZruGBUq7E60pEDmUVpqwy8DB8KxxwZvc8QRcPHFNlumsDCx9nmJWbPg5JMrD5T26AFLloT+3Wm6Y1SosDvRUEzm8d57sGED3Hpr1e1GjoSdO+GVVxJjl9fYuBFWrDgQV/fTo4e9mf76a/DjNN0xKlTYnaiwZx4vvAAdOx4YMA3FscfCccfB88+nrXc3YUEBJ4+awaEjJnPyqBlMWFCQbJMOMHu2fTz11Mrbu3Sxjz/9FPy4WDx2P2n6ecaCCrsTFfbMYv16m/FyzTWh86Sd3HwzfP99eFPdU4wJCwoY+d5iCgqLMEBBYREj31ucOuI+Z471vI85pvL2jh3t488/Bz8uFo/dQx56ICrsTlTYM4u33rLe2hXB5tYF4fLLoW5dePXV+NoVB0ZPXU5R6f5K24pK9zN6ahU54olk7lz7X1GgSLdvb393K1cGP07THaNChd2JCntm8cYbVky6dg2vfd26Nrd9/PgDCzykCesLiyLanlBKSuDbb6FPn4P31agBbduGFnZNd4wKFXYnKuyZQ0EBfPONnVUaCUOG2Hz2Tz+Nj11xonWj2hFtTyj5+bYmTzBhBzj0UFi9Ovg+twZPPea1q7A7UWHPHCZNso8XXBDZcf37Q6NG8Oab7tsUR4YP6Ert3MqLUdTOzWb4gDD/W4knc+fax1DCnpdnb8TBcHPw1EMevAq7ExX2zGHiRDswd/jhkR1XsyZcdJGtaZJGlR8H98rjoYuOJK9RbQTIa1Sbhy46ksG98pJtmhX21q2hTYiVM/Py7EB3sN+eDp5GRUz12DMOFfbMYO9emD4dhg2L7sd9+eW28uO0aXDhhe7bFycG98pLDSH3MWFBAaOnLue1yTNYndeJwoXrg9vXpo0d09iyBVq0OLC9vNwWb9N0x4hRj92JCntmMH269bYjDcP46dsXGjSADz901Swv4U+/LFr/Kx0KNzCneafQ6Zd5PrFft67y9pIS+6gee8SosDtRYc8MJk6E+vXhtNOiO75GDTuhaeJE/T5EiT/9sud6m265oHXX0OmX/hBNYJzdL+yxeuweJJIVlMaKyCYRWeLY1kREPhGRH32PjeNjZoLQImDpjzEwdSr06xebIFx4oa3Z/vXX7tnmIfxplr3WL6dMsljcqlOl7ZXwe+yBwl7ka1s7xsweDcVUyThgYMC2EcB0Y0xnYLrvdfqiRcDSn5UrYe1aK+yxcM45duk2DcdEhT/Nstf6H1jW4lD25daqtL0SrVrZ9zowFOMX9jp1Ij7/hAUF/PHNhQDc8uq3rNuW/ksyR0LYwm6M+QIILFg9CHjZ9/xlYLBLdiUHDcWkP9On28ezzoqtn8aNbShHhT0qhg/oSv0sQ88NK1jQuhtQRfpldrYV90CPfa9PjCMUdn98f8vuYgC27C5mccEOdhXvr+bIzCHWGHtLY8wGAN9ji1ANRWSYiMwXkfmbN2+O8bRxQoU9/Zk+3cZs/cWlYuHCC22t8FCzIpWQDO6Vx1NdyqlXUsTcdkdWn34ZLJfdL+wRhmL88X3jGDzdX27Yuqckon7SmYQNnhpjXjDG9DbG9G7evHmiThsZKuzpTXm5XaXnrLPcyYjwZ9VMnBh7Xx7k9HX5ADzz4l+ZPeLMqlMx27SpFIqZsKCAW8bYYmx//GB5RMXMAuP44guvlu33zm87VmHfKCKHAPgeN8VuUhJRYU9vFi2yiyPHGobxc9hhdrUfDcdEx/Tp0LMnNGtWfVuHx+4PpezevhOAdcVEVKnyQBy/8s09J9s7SYCxXumHwHW+59cBH8TYX3JRYU9v3IqvO7ngArvY8o4d7vXpAildex3swOecOeF/Fq1b24VO9uypCKXULrMx8qLcmhFVqgxWXiE7S2ha1ztpk5GkO74BfAV0FZF1IjIUGAWcLSI/Amf7XqcvKuzpzfTptoRA69bu9XnBBXb245Qp7vUZIylfex3srN2SkgMLV1eH/zPbsKEilFKr1Ar7vhw7QSncSpX+8grN6lkhb16vBkfmNaR+Le9MtI8kK+YKY8whxphcY0wbY8xLxpitxpizjDGdfY/pvcy7Cnv6UlICX3zhrrcOtnBV06YwaVLKeMkpX3sd4O237ft2xhnhtfcL+/r1FaGU2j5hL/IJeySVKgf3yuPJq+wat09deQxtGtUK+9hMwDtBp3BQYU9f5s61WRRuC3t2Npx7LiUTJ3H3+IUp4SWndO11sGGYDz+0xdRyc8M6ZHqhlaLbHpnEnuIycrOFWmU2i6Uot2ZslSq1uqPHUWFPX6ZPt59f377u933++dTYUUi3NUsrbU6Wl5zStdcBJk+G3bvhssvCaj5hQQF3zt0KQIs92yksKgUDTaQMgCbNGkVXqdJDQh6Id4JO4aDCnr5Mnw69e9ta6m4zYAClWdn0++lr5rfpUWlXMrzk4f27MOXhsZy19Es6bF9PaXYOS9t0ZdfV13HyqBmsLyyidaPaDB/QNTnVHv/zH7vkXZg32dFTl7MxuzbF2bm02G2juaXlhvrlJSDCZ3cP8LRIR4MKuxMV9rTAXw7WL2AjT8nj/HnzYPjw+JywYUMWHXoUZ/30NaP63lBpV8K95BUrGPyH6xg8dy47a9dnSfNDaVhewg1fv4989S4Nj72A0addQ0GhTREE4ibugZ/D8AFdGbxpic0i+ve/ISc8eSkoLAIRNtZrQsvdWyu2l+zeaycnxSrqHiwToqEYJ1oELOUJlhEy6am3bOaK2/F1BzUHD6Lz1l9ot31DxbaEr1A0ZYr9r+THH+Gll2iwYysnrVlEj1+WcdHw13m950CGzv+Ad167g+a7t7kfKtq/3xZFmziRz1/+kH+++U2lz+GRV75g7/VDoVMn+N3vwu422yfcG+s1pcXu7RXb65QVR1UnpgIPe/nqsTtRjz3lCZYRctzK7yjOqUHNk06K23mP/P3V8Oh9XLR+Af9ufEjiQx2TJ9vByB494IMP7ALQDhaX1yG//y181rE3T374MONfu51Lr/wX62ka+7n374dnn4X777cVL4HTgblZOXyX1415bY9kf1YWl+VPI6toJ3z4ZUQ11Pf7POpN9RrTbfOaiu21SmIUdj/qsXucLN/b4cEvQroQLKZ9yuoFfN2mR+zlXavisMOgWzf+vG8Fq0adV/0UeTf59lu7KPeRR9qxhABRn7CggCyfdzqj0/FcNeQBmu7dwWtv3U33nOLYzl1YaCtd3nYbHHEEvPEGfP01N110D2OPG0S9kiL+NOcN/jLrNbbVacgVQx6E446L6BR5vnDWpnpNaOEIxTSRstg+U6fH7rHftHrsTpzC7uF/41KZ1o1q25isjxa7ttJ1y1o+7nk2p8b75BdcAE88YWdINmgQ77NZ1q+3xciaN7cLdDeuvOSBPzS13yFcC1t3ZejFf+fld+7l1ff+AXecbRceiZTt220e+tKlMGYMDB1a8bv4/rhdfNL5BABq+vLNi3NrVoh0JAwf0JXh7yxiY72m1C8pok5JEaW16tCjgUCJC++zpjt6HL+wazgmZRk+oCu5WQd+oKessTW3P2t3dPxzys8/367N+fHH8T2Pn/JyuOYaW85g4kRb2jaAYKEpgPntj2LBo2NovGIpXHLJgdWIQhA4+Wri7BVw7rmwbJm9ofz2t5WE0Tltvzi3JsWx5pqLDcUANjNGIHfXztiynDwk5IGosDtRYY+ZeM/OHNwrj3qOqeEnr17IljoNyW/WIf455SefbBdbHj8+vufx8/zztlrlY4/BUUcFbRIq3bLcGE7847XW0542DW68MeT3OnBAesuWHTS7+jLKv/kG3nwTBgw46Bj/tP28RrURqL4sbxWMnrqc0v2GjfXseEDL3dso3W/Y+evm+KSvegANxThRYY+JigWMfR6kf3YmuJtyV7i31D4xhlPWLGJO+6MxkhX/nPLsbLj4Yhg3Dvbsgbp143euVats+ubZZ8NNN4VsFhiacm4H4IYbYMMGuOsuOOQQGD36oLZOr79GWSlPT3iIE1cv4p+XjuDvv/lNyHMP7pXnyufq/9w21msCUJHLXnfPLleEffaPmyn+YRNtNu7mhlEzkpffn0DUY3eSQcKejLomiaph4hetLlvW0HL3Nr7s0LPS9rhy6aV2yvxHH8XvHOXl1sPOzoaXXqoypBCskuFBIZGRI+EPf4BHHoE77zxoIPFA0a19vPjuP+m38hvu6n8L/+14invXVAWtHYOncEDYGxbvcSUUM3bWqoOcjZQqmBYHVNidZIiwJ6v6X6JqmPjF7NTVNr4+u0PPxOWUn3aaDce88078zvHMMzBzpg3BBGTABBJWSETEThgaNgweegguv9wOjPpo3ag27bev563XR3LK6oUMP+dPvNbr3IRNvvJ/njtr1mVfTg1a7t5GQ9lPzbISVzz2krIUL5gWBzQU4yRDhL0qzzme/4JWGxZwCf81tHz7Xn5ukoe0a89Difr3Ojvb5pO/8kp8wjErV8Idd8DAgdZrD4OwQiLZ2fDcc9CxI9x9t427DxkCHTrw5hfzaDl1Entr1OJ3F93FJ537JHTyld/20VOXs6luYzqU7GTUWe1sEXCXBk8lxH8pmYoKu5MMEfZkVf8bPqBrpRg7xG925uCujWFtPtx0E7NHnOl6/1UyZIgVyffes1krblFebmPiubl20NPFrI4D0/+P4NRbnuHBHybR5tVXYc8e2jZtysrLruEvHQaQb+qRl4Q6MxU3p0mdaJdTBm19zoCLg6fG8XamTMG0OOGKsIvI/wN+CxhgMXCDMWafG30nlAwR9kR5zoE4Pa+4F6L69FPYtw8GDXK/7+o47TQ7YWnsWHeF/cknbZ2VcePsGqARELRui+99DxzU/qJ2a84+/vc89NDTDO7UABo04DBSZPmz9u1h9mw7MQpcEfaaAUviJbwURBKIOcYuInnAH4HexpgjgGxgSKz9JoUMEfawBtTixOBeecwecWZcZ2dOWFDAxPufZWfNupw2d3/iB8JErGc9c6YNnbjBihV2kPP88+HaayM6tLoxlZChuWkrEjfRKlw6dYJffoFff7WvXQjF3HhyB2rXsL+HWNIy0wm3Bk9zgNoikgPUAda71G9iyRBhdzPHOCbKy+26l6++CvPnuzKte8KCAu4av5ATv/+KmR2PZe2u0uRkOVx/vf2+/Pe/sfe1f7+9UdSqZXPXIwzBVJeNlPILczg57DD7vZk5075u1y7mLk86rClndG1B15b1E1sKIonEHIoxxhSIyCPAWqAImGaMmRbYTkSGAcMA2rnwYcUF/w8qzYUd3MsxjgRnOKAXO3lh6hM0W/j1gQb9+tnY9GGHRX2O0VOX03XtMprt3cEnneyU9kQMDB9EXp4d4Bw3Du69N+yVgoLyxBMHboBRrNdanXAnKzQXFf7vxrRptpBYLOvX6szT6BGRxsAg4FCgNVBXRK4ObGeMecEY09sY07t58+axnjY+aBGwqHGGA1ru3MIzT99Kre8Xs+CuUXZa+uOPwzffwEknwaJFUZ9nfWERZ/84j9KsbD7veGyl7Qnn97+HgoLYUh8XL7aThwYPhiuvjKqL6lZUSmZoLmK6dbOP339vM3iyXAoqeOw37cbgaT9glTFmM4CIvAecBLzqQt+JJUNCMcnAHw6oUVbKmPfup25JEZdc9TC76nVndrdu9gc7cKCdSdm/v63r3b59xOdp3bAW5//wJXPaH83OWvUObE+Q9+n8rySvQS0+OrQz5h8PcO6alqzfsS+yAeN9++Cqq2wc+YUXovYwq8tGSuigdqw0bQqHH26dgY4d3enTg0XA3BD2tUAfEamDDcWcBcx3od/Eo8IeNX6P+XfzxnPkxpXcdNHd/NDiUMTpSXfrBp98An362IqFc+dGXJb1oZY7abtjI4+dclXFtkR5n4HZJet2FvNg9/MYNfkJun37OQWdToisjMKdd1qPffJkW70xSsIR7mSE5qLm5JOtsJ99dmz9eEjIA3Ejxj5PRMYD3wFlwALghVj7TQoq7FHTulFtslav4tav3mZSt1P5pHOfiu2V6NYNXn8dzjsPRoywMyIj4LSvp1JWqzbzjj4NfA5qzZzETKAONkg5/vC+3PTVO9wx82VmduzN/qzs8GL+kybZ8NQf/mCrKMZIWgl3dfzrX3YS2MCB7vTnsTAMuJQVY4y51xjTzRhzhDHmGmNMjNX9k4QKe9QMH9CVv3z1JkaE+88cChzsSVfUr/nC8PZJv7GLHk+ZUmW/zpo3/f4xmdLX3mDDGQPYnnVghZ7CosRkxgSL45dl5/Dw6dfRZetarlrwUZVtK1i6FK64Ao45Bh5+OB6mpjdNmtjFPWL1uD3ssWutGCcq7FEzuH4Rg5Z+xoQTLmBT/WYHpVgG5lrfc+I1rGjegX1XXwubNwftM/CYPrMnk7t7J/e0Pi0hxcYCCRXHn9r5RD4/9BhGfD6ODtsKqmzLhg02DFWvnl3izo2l35SqUY/d46iwR89jj5GVm8uQ8U8FnZwUGMYozqnBbRf8DdlRaHO4g/z4nMeIKeeGbyeyqFVnZjbtHNSERJRMCMwuyc0ScnOyuP2cP1KSncvz7z9Aq7K9wWP+a9dC376wcSNMmBDx7FIlQtRjVwAV9mjZs8fmYF96adBVfiC46C5v3oGH+t5oBw+feqrKYy5Y9gWHbVvHS8cNDvmDTUTJhMCJX6MvPZrRlxxNTtu23DJ4JIdu38CUSf9gcL29Bw40xgp5797WY58yBU44Ia62KgF4zGvXImBOVNij4+23YdeuqBaE+OTMS7mvVgH87W82W8axELL/mJqlxdz++cssbnkYEw8/lUa1cykuK09IsbFAQg1S2m1nwrW9qHHVVXbh54EDoWVL+OorWLLEbhs/HrqmYP54JuPBdEf12J2osEfHmDE22+WU0AszhJwkM7CbnZafl2cXi1616qBj7v7sJdrs3MyDZwylVo1c7ruwR2qUTAjGOedAfr48mDcNAAAXkklEQVSdvLRkifXUmzWzpQIWLFBRTyQeEvJA1GN3osIeOUuXWo/00Uer/CFVm2s9ebLNXz7pJOvVnnwyg3u2ptP/nueIBR/x3AkXs/boEyrVXU8JIQ9Gmza2TMATTyTbEgU8F4YBFfbKqLBHzpgxUKNGWBUJq8y1PvxwmDXL5nSfcgqceCLs2sURS5bARRdx89tvcXN2dvBjFSUYHvbYNRTjJIOKgCWEffvgf/+D3/zGhhtipXt3OxPzwQft6xYt7I3j7bftCkCKooSFeuxOtAhYZLz3Hmzbxh/r9WbiiMnu1CCpX9/WJR850j07FW/jwd+zCrsTDcVExJbHnmRvo1ZMbNq10gIPkMLxb8U7OEMxHhN3DcU4yRRhN8Z609ddB0OHwkcfuf/FXrGCZt/O5c2j+mPkwNfICyvAK2mGpjt6nEwQ9qIiuw7nxRfD1Kl22vp551mBLy117zwvvkiZZPHOkf0O2pWSK/Mo3sNDQh6ICruTdBd2Y+DGG231xPvvt4tAbNgA99xjc8Wvusodz72kBMaNY3b3k9hcr8lBu1NyZR5F8RAq7E7SXdj/8x94802bVXL33TaTJDcX/vlPGDXKrvTzr3/Ffp4PP4TNm8m9eVj6rMyjeBePxddBhb0y6Szsa9bYhRvOOw/uuOPg/bffDpddZr33xYtjO9eYMdCuHSf9/srUnQGqKB4OxbiSFSMijYAXgSMAA9xojPnKjb4TSjoL+5//bB+feSb4F1oEnn4aZsywNV1mz44uN3zlSrsK0r33QnZ2Zi3woGQm6rFHzb+BKcaYbsDRwDKX+k0s6Srs8+bZmiR33gnt2oVu16yZneY+b54V+Wh49ll7Q6ii4JeipASa7hg9ItIAOA14CcAYU2KMKYy136SQrsL+979D8+ZM7HtpxWpDJ4+aEXxFoSuvtItJ33OPHViNhL17YexYO9O0dWt3bFeUeKPpjlHREdgM/FdEFojIiyJSN7CRiAwTkfkiMn9ziBVzkk46CvuXX8K0aSy5+mZun/JzxWpD/slCB4m7CDz5pE2LvP32yM715puwfbtdp1NRUh0PCXkgbgh7DnAM8KwxphewBxgR2MgY84IxprcxpnfzGFZkjyspKuzOdT8reeLGWM+7VSv+2KhP+MvFdekCw4fbxTG++CI8I4yxWTc9esBpp8V4RYqixBM3hH0dsM4YM8/3ejxW6NOPFCwCFrjuZyVPfMYM+PxzuPNOVu0NHkMsKCwKHprxx+NvvRXKyqo3ZNIkWLTILojhYU9ISUM8Fl8HF4TdGPMr8IuI+JOXzwK+j7XfpJDoImBjx0LHjtCpkx3MrGbdTz9FpfsZPeUH6623aQM33VTlpKCgoZm6de1A6uLF1Q+kGmNz4Tt2tJOcFCUd8LAD4lZWzG3AayKSD/QEHnSp38SSyFDM66/baf6tWtnVg269FUaMOEjcQ03P7z7/c7vAxT33QK1aQVcoCuSg0MzgwTBggO1j/frQB776Ksyfbyc95eaGfYmKkhJ40GN3JY/dGLMQ6O1GX0klzsI+YUEBo6cuZ+fGrcx6/mbKjjqWpp9/Djk5dkDy4YdtDfK//rXimGBrhWaV72fk7P/ZWPmNNwIHr1AU6qtc6UbhH0g9+mjriX/66cG57Vu3HliP9LrrYn4PFCVhaLqjAhwQ9v37q24XBc5Y+RULP6Zh0S5u7n0NE5Zssl/Ap56CSy6xmSpTplQcF8wTv/yHz+m4cTU88IC9KfgY3CuP2SPOZNWo88gLEZo5KGTTubPNTZ85E267rfJNrbTUzlbdvt22ydKvi5LGeCg0o79UJ35vNQ7CXhErN4Yhi6Yyr00Pvmne6UBoJCsLxo2zK9kPGQIrVgBWrJ3T9g/PLeHeOf+D449nQsc+IfPWQy4eHayOy3XX2SyZZ5+Fq6+2C0r/+CNcdJEdoH3xRejZ0/X3RFESgse8ddCFNirj937jIOz+EMhRv/5Ix+3rebbPpZW2A3ZA84MP4LjjYNAgJj3/Hg/N2VCx+PPjlx3N4EfvgB3bmfGX1xj5/pKKgdXARS6qXTw6kH/9Cxo1spOd3njDbqtZ0w6shrGeqaKkHB7y0ANRYXfi99jDSf+LEH+s/IyV8ylH+LTT8RXbK9GhA4wfT3m/fjS48Vq2DhqBya1FQWER6/48Ar60JXnvWZVNUWlJpUP9g6N+8Y6ojouITYEcMgQ+/tj+B3H++dC2bayXrijJRT12jxOnUMyEBQXsLbE3i9NXfUv+IZ3ZXqdh6NDI6afzyPm3cvuEJ5g07s+8c1Q/eq5fwTkr5jDp2IGcf9ddrB/5UdBzxbzIRceOOrNUyQw87LFrjN1JHITdP2i6fW8pjYp20nP9CmZ2PJZGtXOrLHH7bNd+XHn5/yEYRs4cx8lrFvHoKVfx5zN/DyIh89Z1kQtFCUA9do8TB2F3TjA6ddUCsjB8fuixFBaVVgycBhP31o1qM6dDT8767XM0KN7D7hq1Kc/Krsh2GT6gKyPfW1xp8pIucqEoDjTdUQHiIuzO0MixBcvYk1uL/EM6A1UU6sKR1SLCzlr1KM/KriTcgdkyusiFolSDh0Iz6rE78WfFuDh46pxg1Gv9cvIP6cz+rANpiIEDnn7CyWrRRS4UJQw85q2DCntl4uCx+0Mm5Xv30n3Tz4w5/jcHtQk14KnCrSgx4CEPPRAVdidxEHa/ME95/l1yy/ezoHW3g9rogKeixBH12D1OnNIdB/fKY3AXG975od3hlfbpgKeixAkPe+w6eOokjiUF+OYbaNeOv157ug54Kkqi8ZjXrh67kziWFCA/H3r21Li5oiQaXfPU48SrpMC+fbB8ORx1lLv9KooSGg8JeSCuCbuIZPsWs57kVp8JJ15le5cts30eeaS7/SqKUj0eC8OAux77n4BlLvaXHLKz3Rf2/Hz7qB67oiQOD3vsrsTYRaQNcB7wAPAXN/pMGiGE3b/6UVglcAPJz4datezapoqiKHHGrcHTJ4DbgfqhGojIMGAYQLt27Vw6bRwIIuz+Ql6hap9Xy+LF0KNHpdWOFEVJEBqKiRwROR/YZIz5tqp2xpgXjDG9jTG9mzdvHutp40dOzkGDp85CXn4OWhi6KvLzNQyjKIlGi4DFxMnAhSKyGngTOFNEXnWh3+QQxGMPNeU/rNrnGzfaPx04VZTkoOmOkWOMGWmMaWOM6QAMAWYYY66O2bJkEUTYY6p9vtiGbNRjV5QE4yEhD0Tz2AMJIuwRLQwdiAq7oiQXj4VhwOWZp8aYmcBMN/tMOEGE3VlCt3RdAbUOacVfzu0e3sBpfj60agWpPK6gKJmIhz12TdMIJCcnaLrj4F55DN7yPYy8Fk48EUbOCq+//HyNryuKklA0FBNIdnbokgKPPWYfv/oKPv64+r7KymDpUg3DKEoy8WAoRoU9kFAzT8vKYOZMuPVWaNIExo+vvq8ff4TiYvXYFSUZeDjdUUMxgYQS9h9+sMW8+vSBTZtg2jT7ZakqjucvJXD00fGxVVGU6tF0RyWksC9YYB979YJ+/WD9euuRV0V+vu3v8MOrbqcoivt4SMgDUY89kFDCvmyZHVjt0uXA/m++sa9DkZ8P3bpBzZrxsVVRFCUI6rEHEiIrhjVroG1bu797d6hbF77+uuq+tJSAoiQfj8XXQYX9YEJlxaxZA/7iZdnZcOyxVQt7YSGsXavxdUVJFh4OxaiwBxIqFLN2LbRvf+D18cfbuHtJSfB+dMapoqQG6rErQYW9tBQKCg4W9uLiAwIeiC6uoSjJxcPpjirsgQQT9l9/hfJyaNMGsPXZL55fCsDo/3uVCQsKDu4nP9/mu7duHW+LFUUJBw+FZjQrJpBgwr5pk31s2fLAohs0YEudhrRfuST4ohuLFllv3UNfJkVJSTzmrYN67AcTZKGNCmFv0eLAohsiLDqkC0dvWHHwohulpVbYe/ZMnN2KolTGw06VCnsgVXnsLVpUWlwjv1VnOm/5hbrFeysvurFwoZ2letJJCTBYUZQqUY9dqU7YnYtrLDqkC1kYjti4svKiG3Pm2McTT4yzsYqihEQ99ugRkbYi8pmILBORpSLyJzcMSxoBwj5hQQGvT5zPvpwanPzU15zRrXnFohuLDukMwHEbf6y86MacOTbn3TfYqihKElGPPSrKgL8aYw4H+gB/EJHuLvSbHBzC7h8orbl9C1vqNKRgxz7e/baAi4/NI69RbQrrNGRtszZcXbKm8sDpnDkahlGUZOPhdMeYs2KMMRuADb7nu0RkGZAHfB9r30nBUVLAP1DaZO8OttduAEBR6X4++2Ezs0ecadsXngevv24HXHNyYNUqWLdOhV1RUg0PhWZcjbGLSAegFzAvyL5hIjJfROZv3rzZzdO6i6OkgH9AtMG+PRTWql/RpNJAab9+sGsXzJ5tX0+caB8HDkyIuYqiVIPHvHVwUdhFpB7wLvBnY8zOwP3GmBeMMb2NMb2bp/L6n45QjH9AtOG+3eysVbeiSaWB0oEDoVatAwtvvPOOrejYuXPCTFYUJQge8tADcUXYRSQXK+qvGWPec6PPpOEQ9uEDulI7N5uGxbvZUaseALVzsysPlNarB4MHw8svw5QpMGsW/Pa3ybBcUZRgeNBjjznGLiICvAQsM8Y8FrtJScYh7P4B0Qaj9rCzVj3yGtVm+ICulQdKAe6803rq55xjSwgMHZpoqxVFCcTDHrsbJQVOBq4BFovIQt+2O40xH7nQd+IJSHcc3K0JlJVy86Bjudk/YBrIkUfC5Mnw/vtwyy3QqFGCjFUUJSw85rW7kRUzC8icW2NgSYHCQvvYuHHVxw0YYP8URUktdM1T5aCZp9u320f1whUlvfCQkAeiwh5ItB67oiipicfCMKDCfjA1atjqjH7UY1eU9EQ9dqWCGjUqL3fnF3b12BVFSRNU2APxe+z+f9/8oRj12BUlPdFQjEKNGvbRH47RUIyipCceLgKmwh6IX9j94ZjCQqhT58B2RVHSC013VA4S9u3bNb6uKOmIh4Q8EBX2QIJ57BqGUZT0xWNhGFBhPxj12BUlM1CPXakgmLCrx64oShqhwh5IsFCMeuyKkr5oKEYhN9c+aoxdUdIbTXdUKnB67OXlsHOnCruipDOa7qhUEvadO+2XQoVdUdIPDwl5IG4tjTdQRJaLyE8iMsKNPpOGU9i1nICiKGlIzMIuItnA08A5QHfgChHpHmu/SUOFXVEyC4/F18GdpfGOB34yxvwMICJvAoOA713oO+F89nMhZwA3jZlDjaaNeRpU2BUlHdFQTEzkAb84Xq/zbauEiAwTkfkiMn/z5s0unNZ9Jiwo4PEvVgOQs7+Ukq3bAPjs15IqjlIUJaXxoMfuhrAHuy0e9E4aY14wxvQ2xvRu3ry5C6d1n9FTl7O73L4lueX7abBvDwDPLNySTLMURYkGTXeMiXVAW8frNsB6F/pNOOsLiyjJttGp3P1lNCjeDcBPxdnJNEtRlFjQdMeo+AboLCKHikgNYAjwoQv9JpzWjWpTWiHspRUee/0WzZJplqIo0eAhIQ8kZmE3xpQBtwJTgWXA28aYpbH2mwyGD+hKds2aAOSWl1G/eA97atTmL+emb5KPoijew42sGIwxHwEfudFXMhncK4+cXUfD41BjfxmtTDHSuBGDex00FqwoSrrgsfg6uCTsmcT5x3UA4J7+nWDeVihumlyDFEWJDg3FKBUETlDSHHZFSW886LGrsAeSnQ1ZWQeEvWHDZFukKEo0aLqjUomaNWHfPvXYFSWT8FBoRoU9GA0a2MqOO3aosCtKuuMxbx1U2IPTsKH11tVjV5T0xUMeeiAq7MFo2BAKCuxCGyrsipLeqMeuAFbY16w58FxRlPRDPXalEg0bwrp19nmKFixTFCUCPOa1q7AHw+mlt26dPDsURYkdLQKmACrsipIJeEjIA1FhD4ZT2Fu2TJ4diqLEjsfCMKDCHhy/sOfm2j9FUdIP9diVSviFXVMdFSX98aDHrtUdg9G/P1x2GfTtm2xLFEWJFg977DEJu4iMBi4ASoCVwA3GmEI3DEsqeXnw1lvJtkJRFLfwmNceayjmE+AIY8xRwApgZOwmKYqiuIimO0aGMWaab2k8gLnYhawVRVGSj4eEPBA3B09vBD52sT9FUZTY8VgYBsKIsYvIp0CrILvuMsZ84GtzF1AGvFZFP8OAYQDt2rWLylhFUZSw8bDHXq2wG2P6VbVfRK4DzgfOMib0rdEY8wLwAkDv3r29dwtVFEVJELFmxQwE7gBON8bsdcckRVEUF/FgKCbWGPtTQH3gExFZKCLPuWCToihK7Hh4zdOYPHZjTCe3DFEURYkLmu6oKIqSIXhIyANRYVcURckwVNgVRclsPBZfBxV2RVEyFQ3FKIqiZCjqsSuKomQIHk53VGFXFCWz0XRHRVGUDMFDQh6ICruiKEqGocKuKEpm47H4OqiwK4qSqWgoRlEUJUNRj11RFCUD8Zi4q7AriuINPBSaUWFXFCWz8Zi3DirsiqJkMh7y0p24Iuwi8jcRMSLSzI3+FEVRXEM99sgRkbbA2cDa2M1RFEVxEfXYo+Zx4HbAe7dFRVFSHw967DGteSoiFwIFxphFUs2dUUSGAcN8L3eLyPJYzp0AmgFbkm2EC2TKdYBeSyqS+tfxwAMHnletU6l/LdA+nEZiqrmbicinQKsgu+4C7gT6G2N2iMhqoLcxJtXfmLAQkfnGmN7JtiNWMuU6QK8lFcmU64DMupZqPXZjTL9g20XkSOBQwO+ttwG+E5HjjTG/umqloiiKEjZRh2KMMYuBFv7XmeaxK4qipCuaxx6aF5JtgEtkynWAXksqkinXARl0LdXG2BVFUZT0Qj12RVGUDEOFXVEUJcPwtLCLyEARWS4iP4nIiCD7a4rIW77980SkQ+KtDI8wruUvIvK9iOSLyHQRCSsfNhlUdy2Odpf4SlmkZIpaONchIpf5PpelIvJ6om0MlzC+X+1E5DMRWeD7jp2bDDurQ0TGisgmEVkSYr+IyH9815kvIsck2kZXMMZ48g/IBlYCHYEawCKge0CbW4DnfM+HAG8l2+4YruUMoI7v+e/T+Vp87eoDXwBzsdlYSbc9is+kM7AAaOx73SLZdsdwLS8Av/c97w6sTrbdIa7lNOAYYEmI/ecCHwMC9AHmJdvmaP687LEfD/xkjPnZGFMCvAkMCmgzCHjZ93w8cJZUN8U2OVR7LcaYz4wxe30v52LnHaQi4XwuAPcDDwP7EmlcBIRzHTcBTxtjtgMYYzYl2MZwCedaDNDA97whsD6B9oWNMeYLYFsVTQYBrxjLXKCRiBySGOvcw8vCngf84ni9zrctaBtjTBmwA2iaEOsiI5xrcTIU65WkItVei4j0AtoaYyYl0rAICecz6QJ0EZHZIjJXRAYmzLrICOda7gOuFpF1wEfAbYkxzXUi/S2lJDHViklzgnnegbmf4bRJBcK2U0SuBnoDp8fVouip8lpEJAtbeO76RBkUJeF8JjnYcExf7H9QX4rIEcaYwjjbFinhXMsVwDhjzKMiciLwP9+1lMffPFdJl998lXjZY18HtHW8bsPB/z5WtBGRHOy/mFX9G5cswrkWRKQftsbPhcaY4gTZFinVXUt94Ahgpm+2cx/gwxQcQA33+/WBMabUGLMKWI4V+lQjnGsZCrwNYIz5CqiFLaqVboT1W0p1vCzs3wCdReRQEamBHRz9MKDNh8B1vueXADOMb4Qlxaj2Wnzhi+exop6qsVyo5lqMMTuMMc2MMR2MMR2w4wUXGmPmJ8fckITz/ZqAHdTGt0hNF+DnhFoZHuFcy1rgLAARORwr7JsTaqU7fAhc68uO6QPsMMZsSLZREZPs0dtk/mFHwFdgR/zv8m37J1YowH453wF+Ar4GOibb5hiu5VNgI7DQ9/dhsm2O9loC2s4kBbNiwvxMBHgM+B5YDAxJts0xXEt3YDY2Y2Yhtupr0u0Och1vABuAUqx3PhS4GbjZ8Zk87bvOxan63aruT0sKKIqiZBheDsUoiqJkJCrsiqIoGYYKu6IoSoahwq4oipJhqLAriqJkGCrsiqIoGYYKu6IoSobx/wEd6teF4edN/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c6e635c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_poly = PolynomialFeatures(degree=30).fit_transform(X)\n",
    "X_test_poly = PolynomialFeatures(degree=30).fit_transform(X_test)\n",
    "\n",
    "model = LinearRegression()\n",
    "model.fit(X_poly, y)\n",
    "y_test = model.predict(X_test_poly)\n",
    "\n",
    "plt.scatter(X.ravel(), y)\n",
    "plt.plot(X_test.ravel(), y_test,c='r')\n",
    "plt.title(\"mean squared error: {0:.3g}\".format(mean_squared_error(model.predict(X_poly), y)))\n",
    "plt.ylim(-4, 14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we increase the degree to this extent, it's clear that the resulting fit is no longer reflecting the true underlying distribution, but is more sensitive to the noise in the training data. For this reason, we call it a **high-variance model**, and we say that it **over-fits** the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detecting Over-fitting with Validation Curves\n",
    "\n",
    "Clearly, computing the error on the training data is not enough (we saw this previously). As above, we can use **cross-validation** to get a better handle on how the model fit is working.\n",
    "\n",
    "Let's do this here, again using the ``validation_curve`` utility. To make things more clear, we'll use a slightly larger dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGspJREFUeJzt3X+MZWV9x/HPd2cHnVVk1rK2MrosJgpVCKxMGppNrEhTiD83KKLxd2k3amLVWpK1NsH+SNh0q9YmTduNtWq1iqJZqdiidTG2RNBZF0WFba26wEBlDSytMMCwfPvHnVnu3jnn3uf8fs4571dCmL1zZu5zzrnzPc/5Pt/nOebuAgC037qmGwAAKAcBHQA6goAOAB1BQAeAjiCgA0BHENABoCMI6ADQEQR0AOgIAjoAdMT6Ot/s5JNP9i1bttT5lgDQevv37/+5u2+atF2tAX3Lli1aWFio8y0BoPXM7FDIdqRcAKAjCOgA0BEEdADoCAI6AHQEAR0AOoKADgAdUWvZIgBUae+BRe2+7qDuOrKkU2ZndPmFp2v71rmmm1UbAjqATth7YFHv/cItWlo+KklaPLKk937hFknqTVAn5QKgE3Zfd/BYMF+1tHxUu6872FCL6kdAB9AJdx1ZyvR6F5FyAdAJp8zOaDEheJ8yOyOpH/l1eugAOuHyC0/XzPTUca/NTE/p8gtPP5ZfXzyyJNfj+fW9BxabaWxFCOgAOmH71jldefFZmpudkUmam53RlRefpe1b53qTXyflAiCTmFMX27fOJbalL/l1eugAgrU1dbGaRw99va0I6ACCtTV1MS6/3iWkXIBAMaca6tLW1MXqeer6+SOgAwGYhTgwqTQwZmn59S4h5QIEaGuqoWx9SV20FT10IEBbUw1l60vqoq0I6ECANqcaytaH1EVbkXIBApBqQBvQQwcCkGpAGxDQgUCkGhA7AjqA1mOOwIC5e21vNj8/7wsLC7W9H4D2yRqcR+cIrNq4YVpXvOx5nQjsZrbf3ecnbUcPHUA08kzgSpojIEn3Pbjcu8lfE6tczOyjZnaPmX1/6LWnmtlXzey/Vv6/sdpmAhhn74FFbdu1T6ftvFbbdu2LfrGsNHkmcI2bC9C3yV8hZYsfk3TRyGs7JX3N3Z8t6Wsr/wbQgJAVENsS8PNM4Jo0F6BPk78mBnR3/4ake0defoWkj698/XFJ20tuF4BAk3q1bVryNs8yt0lzBEJ/tmvyTiz6ZXe/W5JW/v+08poEIItJvdrQNEYMvfg8E7hWn1Q0OzO95nt9m/xV+aCome2QtEOSNm/eXPXbYQLKu7pn0rIEIWmMWFaTzDOBa/Uzff/SsmZnpmUmHXlwuZef77wB/Wdm9nR3v9vMni7pnrQN3X2PpD3SoGwx5/uhBLH80aJcl194+pqyveGeacg6NON68cOfjTo6BFkmcI1+po8sLWtmekofuvScNb+jD52ZvCmXayS9aeXrN0n6YjnNQZVYArY9sqQ/xj0cWQpLY2TpxceUi8+SToqt7VWY2EM3s09LeqGkk83sTklXSNol6bNmdpmk2yVdUmUjUQ6WgG2HPHdS43q1IWmMkF78H//zD4J68XUK/UyH3oG03cSA7u6vTfnWBSW3BRVjCdh2qCL4TEpjTErb7D2wqPseXE782SY7BKGf6b50Zlg+t0fyLgEbQ/VDn1QZfNLO5aS0zbi0XJMdgtDPdJ5yyDZi6n+P5K0gYCC1XlXdSU06l+N68eMuJk2WBYZ+psfdgVQxWNrUACwBvWeyLgHbl9xjTCalP/Iqci7TLjKzM9ONfw5CPtNpgV9S6R2WJjtBBHSM1ZfcY0zKfJjGcE8xrWY45FymXWTe//LnZW5TU5IC/7Zd+0rvsDTZCSKgYywGUptRxsM00paVHTW7Ye0My6T2SN17YlMVHZYmO0EEdIxV1e0/qjHcI19npqMBzzv4xUOPau+BxaC0RdsD+KgqOixNdoKocsFYk6ofEI/RyTMhwVySlh/z3k4uq+Lh300+UJweOibqYs+si9Ie9BCir2MiVaSSmkxPEdDRS11b12PvgcXE2/xQoemArh03qZoOS1OdIAI6eqdrtfWr+5NXaDqga8ctr5gvagR09E5dZWV1/eGPS7XMTE/plefO6apv36Hlo2tz6nMZ2tXnOQmr53LxyJJMOlYCGttFjYCO3qmjrKzO3uy4dl958VmSpKu+dcdxr0+vM+2+5OxMS8y2eU5CkYvr6LkcvSzGdFGjygW9U8e6HnUuVZzW7rnZGW3fOqfd1x3U8mPHh6GkypZJS8y2dT2Uokvnhgw2x3JRI6Cjd+ooK6uzNztpf8pYYjbkfWKV5+I6vIhZyGBzLBc1Ui7onTrKytIml5yU8NzLYXlSA5P2p6wlZts6WzQtIKe9HjrDdlVMFzUCOnqp6rKyyy88XZd/7rtrUh0PPJI+K7NI3n3c/oTO9g0J/G2ckzCVMmN2yixx+yz1/CbplefGc0xIuSBaoeuwx7he+/atc3ryE9f2l5aPps/KLDPvPnxMdl93UK88d27ibN+2plTSrB6DtBmzaa9nSYu5pOtvO5yneZWgh47SlFmmF9pbjbk2+kjGJ/yUlXdPOiaf3784ccmGtqZUkoSkTebGDPJmmaQVy4CoREBHScoOrKE1z7HURiddzNICg2uwbGue53qGKHJMqkyp1DkhZ1LaZPTOY7htIatPDotlQFQi5YKSlF2mF9pbjaE2Oq0s7vwzNq1JYaxKKp0rK+URwzEZVbR0MKtx+zqachptW9qzU5PElpIioKMUZQaRvQcWtS5lwGq0NxRDbXTaxez62w4fW6kyyegFr6yVLWM4JqPqrMuXxtfm37DzRRPv8tJs3DAd9cqjpFxQirLSBau9pbQBqwcePr5KpO712pPSBuMuZqspjNN2Xpv4xKDRny0j5RHjGvZ13zVkOQahbZiZntIVL3teVAF8FD10lKKsdMGk3tKRpeXjbtXrXK89LW2QlnMdvpjV2WtOOyaSGqsGqvuuIcvnIq0NsffGk9BDRynKqpAI6S2NDvDVVRudljZ4wvp1mpmeGtsbPP+MTfrkjbev+Z3nn7GpkraOHpOmq4GauGsI/VyktS323ngSAnpHxLCkZxmBNbRkrO5Bz9WV9pLcv7SsD116ztjjn1arnPR6Feey6WqgmEsiY25bVgT0Dmi691WGtOVJ01Q5wDccUE+amdYDjzyauPTscFsmXcxCc8hVncsmKl+SLkw37HxRZe9XRBtnwCYhh94BdVcQlG04Ny0NgvlqjcuTTlhb9lf1oOdwnvzI0vLYYB7altAcclXnsu4cdt1lihggoHdArHXHoQNwSUHMNRiUeiwhlq4GuCqCQ5YStiwDZaGDxlWdy7IGrUPPawydjBiXhKgaKZcOKKtksCxZ0wZpwWrcBI+q0kqhgXO1njlUaJ62qnNZRp44y3ltupPRhTRkHoUCupm9W9LvaNChukXSW9z9oTIahnCx1R1nHYDLunZGyO/MK6QteY9tSJ62ynNZNE+c5bw23cloehC4KblTLmY2J+n3JM27+5mSpiS9pqyGIVydtdghsvbO0tIBsxPWDh/3O/NKasv0OtPGDdO1HNvYzuWwLOe16ZUbm75DaErRlMt6STNmtixpg6S7ijcJeRTpfWUtk5u0fdbeWVo6QNLEFfPK7vHFUMIWa8VFlvPa9HFs+g6hKbkDursvmtlfSLpd0pKkr7j7V0prGWqRNdcYsn2etMG4IJZWzlhVj6+sgBrD3IAyZT2vacexjuPSRBoyhvNtnrJmxsQfNNso6fOSLpV0RNLnJF3t7p8c2W6HpB2StHnz5nMPHTpUqMEo17Zd+xJ7MmmDfmnbb9wwrQ0nrD/2YT7/jE26/rbDpX64Y/iDCZW0HvfM9FQ06ZO8ip6DOo9LnZ+XqvfLzPa7+/zE7QoE9EskXeTul638+42SznP3t6f9zPz8vC8sLOR6P1QjbdEok/STXS8J3n5UF4JXEVkvlH3R1eNS9X6FBvQidei3SzrPzDaYmUm6QNKtBX4fGpB1wkloDrJNE5uq0NdBuUm6elxi2a/cAd3db5J0taTvaFCyuE7SnpLahZokVnVMmR54+NHECRlJ26dp+x9pETGuSR6Drh6XWPar0ExRd7/C3c9w9zPd/Q3u/nBZDUM9RsvkNm6Ylnww5T1pynZSWV1aeWGWD3MbZvVlaWPTZXux6upxiWW/cufQ8yCHHr88ucCkASGT9LrzNuvPtp818T3bMICYp41tGsStU1ePS5X7VfmgaB4E9PhlHSRd9Ud7b9Gnbrx9TVlhSFBuw0BZG9qI7goN6KzlguPknZBx/W2H11wIQqdajxtQiqU3F8ugVwxiOSdYi9UWcZy8ucAiAS/tYnHSzHQ0S7DGMujVtC4ui9uG8ZtQBHQcJ+9aIkUCXtpFxEyNL8G6KpZBr6bFsCxukrxBuWsXKFIuWCPP1PciU63T1v1491U3J27fRJqj6bVJYhFj6qnIUrldW5WRgI5SFA14SReRtOd4NpXmiHXRrDrFuOhV3qC898Bi6lLJbR0bIaCjNGUHvNjWeUec5yTPXcNqrz5NW8dGCOgNqbtSIOn9pLhTCKQ54hPjOclz1zDuUYNNX6CKoA69AXVPpEl6v+kpk1xaHnpoZ2yTeVC9LpQg5vl7GrfI3F9eek50x6COxbmQU92VAknvt3zUjwvmVbcB8elKhUeeyqy03vvc7Ex0wTwLUi4NqLtSIMvvbetgELLrUoVH1vGbGMcCykBAb0Bazm92w7S27dpX+u1vlocwxzYY1IWUQKxiLEGsS4xjAWUgoDcgqXcwPWX6xUOP6r4HlyVlq6XN+35JOfSYeihZ6osJ/NnFWIJYpy6WoZJDL1nIjLWknN+TTlhfWU476f12v+ps7b7k7ONee+W5c9p93cFopkCHjjV0JRdclbTPJLNfu4ceeomy9ChHewen7bw28XeWdfub1htZfa3IbLuqhKYEupQLLlvIeeXOpjsI6CUqEliavv2NMSiGHpMu5oLLSiFNOq9dTDv0GSmXEhUJLE3f/sYYFEOPSddWQiwzhRTjeUV1COglKhJY8q5yWJYYg2LoMWn6Yli2MucpxHheUR1SLiVKqiYxSeefsSno58fd/lZdxRFrXW5ISqBrueAye9WxnldUg4Beou1b57Rw6N7jHsXmkj6/f1Hzpz41d4CpY8Cy7UGxS7ngMsdT2n5ekQ1ruZSsimdP8jzLfmnDQ7NRL54p2pAqBqEY2OoXetXIi4BesirKD5suaUT9upRCQn2ocilZFRUXXaviAFANeuglC71dDq1aWd1uafmopsx01F1z3IIDSEBAr8Ck2+XQqpXR7Y66H+uZ1xnMWfgKaAdSLg0InThS94MwkrDwFdAeBPQGhFatxFDdEsNFBUAYUi4TFEk3pP1saNVKDNUtMVxUAIQpFNDNbFbSRySdqcGkyN9292+W0bAYpOW6Fw7dq+tvOzxx0DMtTx46HTuGadsxXFQAhCnaQ/+wpH9191eZ2QmSNpTQpmikpRuGp/anDWim/ez7r/mBnvSE9UFVKzFMMInhogIgTO6AbmZPkfQCSW+WJHd/RNIj5TQrDmlphdHFEpLWDU/72SNLyzqyNHjMXEjVStMTTGK4qAAIU6SH/ixJhyX9g5mdLWm/pHe6+wPDG5nZDkk7JGnz5s0F3q5+WR6uPBrAQ3+26YdIhGj6ogIgTJEql/WSni/pb9x9q6QHJO0c3cjd97j7vLvPb9oUtoxsLJJmaFrKtqM55aSfTcMAI4AyFAnod0q6091vWvn31RoE+M5IesDC687bHDQNP+lnN26YTnwfBhgBlCF3ysXd/8fM7jCz0939oKQLJP2wvKbFISndMH/qU4NyyqM/m7YsatIAI7MzAWRVtMrlHZI+tVLh8mNJbynepPjlzSlnWeel6gdatAEXNSCbQgHd3W+WNHHR9b5KC0iTgtKkJ7X3ARc1IDum/lekyBoozM5kyQEgDwJ6RYoEJJ7UzkUNyIOAXpEiAYkHWnBRA/IgoFekSEBKKnns2wOCuagB2bHaYkWKroHS99mZLDkAZEdAD5ClfG5429kN03rC+nW6f2mZgJRD3y9qQFYE9AmylM+Nbnvfg8uamZ7Shy49h8AEoHLk0CfIUq1CqR2AJhHQJ8hSrUKpHYAmEdAnyFKtQqkdgCYR0CfIUj5HqR2AJjEoOiKpouXKi88KXl1RotQOQDPMffSBatWZn5/3hYWF2t4vq7Tlbfs2qQdAXMxsv7tPXAiRlMsQqlQAtBkBfQhVKgDajIA+hCoVAG1GQB9ClQqANqPKZUieKpXRqpjzz9ik6287TJULgNpR5VJAUlXMKKpkABRFlUsNkqpiRlElA6AuBPQCQqtfqJIBUAcCegGh1S9UyQCoQ28C+t4Di9q2a59O23mttu3ap70HFgv/zqSqmFFUyQCoSy+qXLI8pCKL1Z9911U3p27DgCiAuvSih17llP7tW+c0l5JSmZudIZgDqE0vAnraoORiSYOVTEgCEINeBPS0QUmTUnPpWXLu27fO6cqLz9Lc7IxMg545qRYAdevFxKK9Bxb17qtuVtKezs3O6IadL1qzPcvoAogFE4uGbN86lxjMpeR0DMvoAmijwgHdzKbM7ICZfamMBlUlbeAyKR3DMroA2qiMHvo7Jd1awu+pVJaBS5bRBdBGhQK6mT1D0kskfaSc5lQny8AlVSsA2qjQoKiZXS3pSkknSvoDd39pwjY7JO2QpM2bN5976NCh3O9XJ5bFBRCL0EHR3DNFzeylku5x9/1m9sK07dx9j6Q90qDKJe/71W371rljAbuqmaYAUKYiKZdtkl5uZj+V9BlJLzKzT5bSqshQ9QKgDXIHdHd/r7s/w923SHqNpH3u/vrSWhYRql4AtEEv6tCLouoFQBuUEtDd/etJA6JdQdULgDboxfK5ReV5eDQA1I2AHmi46gUAYkQOHQA6orc99NGJQ6RQALRdLwM6E4UAdFEvUy5MFALQRb0M6EwUAtBFrU+55MmFnzI7k/g8USYKAWizVvfQV3Phi0eW5Ho8Fz7u+Z8SE4UAdFOrA3reXDgPdQbQRa1OuRTJhTNRCEDXtLqHzqJZAPC4Vgd0cuEA8LhWp1xYNAsAHtfqgC6RCweAVa1OuQAAHkdAB4COIKADQEcQ0AGgI1o/KDqMNc4B9FlnAjprnAPou06kXPYeWNR7Pvtd1jgH0GutD+irPfOj7onfZ41zAH3R+oCetOLiMNZ1AdAXrQ/o43rgrOsCoE9aH9DTeuBTZqxxDqBXWh/Q01Zc/MCrzyaYA+iV1pctsuIiAAy0PqBLrLgIAFKBgG5mz5T0CUm/IukxSXvc/cNlNSwEM0MB4HFFeuiPSnqPu3/HzE6UtN/MvuruPyypbWMxMxQAjpd7UNTd73b376x8/X+SbpVUWyRNqj9nZiiAPiulysXMtkjaKummMn5fiLT6c2aGAuirwgHdzJ4s6fOS3uXu/5vw/R1mtmBmC4cPHy76dsek1Z8zMxRAXxUK6GY2rUEw/5S7fyFpG3ff4+7z7j6/adOmIm93nLT6c2aGAuirIlUuJunvJd3q7h8sr0lhqD8HgOMVqXLZJukNkm4xs5tXXvtDd/9y8WaFof4cAB6XO6C7+39IshLbAgAooPVruQAABjox9T8Js0gB9E0nAzqzSAH0USdTLswiBdBHnQzozCIF0EedDOjMIgXQR50M6MwiBdBHrRoUDa1cYRYpgD5qTUDPWrnCLFIAfdOalAuVKwAwXmsCOpUrADBeawI6lSsAMF5rAjqVKwAwXmsGRalcAYDxWhPQpbVBfXVAlKAOAC0L6Cy6BQDpWpNDlyhdBIBxWhXQKV0EgHStCuiULgJAulYFdEoXASBdqwZFKV0EgHStCugSi24BQJpWpVwAAOkI6ADQEQR0AOgIAjoAdAQBHQA6wty9vjczOyzpUAm/6mRJPy/h97QF+9tt7G+3lbG/p7r7pkkb1RrQy2JmC+4+33Q76sL+dhv722117i8pFwDoCAI6AHREWwP6nqYbUDP2t9vY326rbX9bmUMHAKzV1h46AGBEtAHdzC4ys4Nm9iMz25nw/SeY2VUr37/JzLbU38ryBOzv75vZD83se2b2NTM7tYl2lmXS/g5t9yozczNrdVVEyP6a2atXzvEPzOyf6m5jmQI+z5vN7HozO7DymX5xE+0si5l91MzuMbPvp3zfzOyvVo7H98zs+ZU0xN2j+0/SlKT/lvQsSSdI+q6k545s83ZJf7vy9WskXdV0uyve3/MlbVj5+m1d39+V7U6U9A1JN0qab7rdFZ/fZ0s6IGnjyr+f1nS7K97fPZLetvL1cyX9tOl2F9znF0h6vqTvp3z/xZL+RZJJOk/STVW0I9Ye+q9J+pG7/9jdH5H0GUmvGNnmFZI+vvL11ZIuMDOrsY1lmri/7n69uz+48s8bJT2j5jaWKeT8StKfSvpzSQ/V2bgKhOzv70r6a3e/T5Lc/Z6a21imkP11SU9Z+fokSXfV2L7Sufs3JN07ZpNXSPqED9woadbMnl52O2IN6HOS7hj6950rryVu4+6PSrpf0i/V0rryhezvsMs0uNq31cT9NbOtkp7p7l+qs2EVCTm/z5H0HDO7wcxuNLOLamtd+UL29/2SXm9md0r6sqR31NO0xmT9G88l1gdcJPW0R8txQrZpi+B9MbPXS5qX9BuVtqhaY/fXzNZJ+pCkN9fVoIqFnN/1GqRdXqjB3de/m9mZ7n6k4rZVIWR/XyvpY+7+ATP7dUn/uLK/j1XfvEbUEq9i7aHfKemZQ/9+htbekh3bxszWa3DbNu6WJ2Yh+ysz+01J75P0cnd/uKa2VWHS/p4o6UxJXzezn2qQc7ymxQOjoZ/nL7r7srv/RNJBDQJ8G4Xs72WSPitJ7v5NSU/UYM2Trgr6Gy8q1oD+bUnPNrPTzOwEDQY9rxnZ5hpJb1r5+lWS9vnK6EMLTdzflRTE32kQzNucX5Um7K+73+/uJ7v7FnffosGYwcvdfaGZ5hYW8nneq8HAt8zsZA1SMD+utZXlCdnf2yVdIElm9qsaBPTDtbayXtdIeuNKtct5ku5397tLf5emR4fHjBq/WNJ/ajBa/r6V1/5Egz9safAB+JykH0n6lqRnNd3mivf33yT9TNLNK/9d03Sbq9zfkW2/rhZXuQSeX5P0QUk/lHSLpNc03eaK9/e5km7QoALmZkm/1XSbC+7vpyXdLWlZg974ZZLeKumtQ+f3r1eOxy1VfZ6ZKQoAHRFrygUAkBEBHQA6goAOAB1BQAeAjiCgA0BHENABoCMI6ADQEQR0AOiI/wfh7kJG+oBv3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c6f89f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = make_data(120, error=1.0)\n",
    "plt.scatter(X, y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import validation_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def rms_error(model, X, y):\n",
    "    y_pred = model.predict(X)\n",
    "    return np.sqrt(np.mean((y - y_pred) ** 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "def PolynomialRegression(degree=2, **kwargs):\n",
    "    return make_pipeline(PolynomialFeatures(degree),\n",
    "                         LinearRegression(**kwargs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "degree = np.arange(0, 18)\n",
    "val_train, val_test = validation_curve(PolynomialRegression(), X, y,\n",
    "                                       'polynomialfeatures__degree', degree, cv=7,\n",
    "                                       scoring=rms_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's plot the validation curves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEKCAYAAADuEgmxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztvXmcHVWZ//9+7tZrlu50ErJvECB7QhPQYCCCmQTHBQQNow7gKMqos311dJYfOOj8Rh230ZFh0GF0HEQEWfwqO6K4ECAJCQQCZE86ayed7vTed3m+f5y63XVv3+7cdPou3f28X6/qqjrnVNXTde+tT53znPMcUVUMwzAMIxsChTbAMAzDGDqYaBiGYRhZY6JhGIZhZI2JhmEYhpE1JhqGYRhG1phoGIZhGFljomEYhmFkjYmGYRiGkTUmGoZhGEbWhAptwGBSU1OjM2fOLLQZhmEYQ4aNGzceU9Xx2ZYfVqIxc+ZMNmzYUGgzDMMwhgwisvd0ylvzlGEYhpE1JhqGYRhG1phoGIZhGFkzrHwahmGcGdFolLq6Ojo6OgptijHIlJaWMnXqVMLh8Bmdx0TDMIxu6urqGDVqFDNnzkRECm2OMUioKsePH6euro5Zs2ad0bmsecowjG46OjoYN26cCcYwQ0QYN27coNQgTTQMw0jBBGN4Mlifq4mGYRiGkTUmGgCJRKEtMAwDaGxs5Pbbbx/QsVdeeSWNjY39lrnlllt46qmnBnR+w2GiARDrANVCW2EYI57+RCMej/d77COPPMLYsWP7LXPbbbdxxRVXDNi+XHGq/62YMNEAQCHaXmgjDGPE8/nPf56dO3eyZMkSPvvZz/LrX/+aVatW8Sd/8icsXLgQgPe+971ccMEFzJ8/nzvvvLP72JkzZ3Ls2DH27NnD+eefz8c+9jHmz5/P6tWraW93v+8bbriB+++/v7v8rbfeyrJly1i4cCGvv/46APX19bzjHe9g2bJlfPzjH2fGjBkcO3Ysxc54PM4NN9zAggULWLhwId/85jcB2LFjB1dccQWLFy9m2bJl7Ny5E1Xls5/9bHfZe++9FyDj//a///u/LF++nCVLlvDxj3+ceDze57UKhXW5TRLrgEh5oa0wjKLhn/7vq7x28OSgnnPe5NHc+q75feZ/+ctfZuvWrWzevBlwD9YXXniBrVu3dncVveuuu6iurqa9vZ0LL7yQ973vfYwbNy7lPNu3b+eee+7he9/7Hu9///v52c9+xoc+9KFe16upqWHTpk3cfvvtfO1rX+P73/8+//RP/8Tb3/52/u7v/o7HHnssRZiSbN68mQMHDrB161aA7maxD37wg3z+85/nqquuoqOjg0QiwQMPPMDmzZvZsmULx44d48ILL2TlypUAKf/btm3buPfee/n9739POBzmz//8z7n77ruZP39+xmsVCqtpJLGahmEUJcuXL08ZW/Dtb3+bxYsXc/HFF7N//362b9/e65hZs2axZMkSAC644AL27NmT8dxXX311rzK/+93vWLduHQBr1qyhqqqq13GzZ89m165dfPrTn+axxx5j9OjRNDc3c+DAAa666irADaYrLy/nd7/7Hddddx3BYJCJEydy6aWX8uKLL/b6355++mk2btzIhRdeyJIlS3j66afZtWtXxmsVEqtpJInZCFjD8NNfjSCfVFRUdG//+te/5qmnnuK5556jvLycyy67LOPYg5KSku7tYDDY3TzVV7lgMEgsFgPcQLhTUVVVxZYtW3j88cf57ne/y09/+lO+9a1vZSzb3/n8/5uqcv311/Mv//IvvcqlX+uuu+46pY25wmoaSaymYRgFZ9SoUTQ3N/eZ39TURFVVFeXl5bz++uusX79+0G245JJL+OlPfwrAE088wYkTJ3qVOXbsGIlEgve973188YtfZNOmTYwePZqpU6fy0EMPAdDZ2UlbWxsrV67k3nvvJR6PU19fz7PPPsvy5ct7nfPyyy/n/vvv5+jRowA0NDSwd+/ejNcqJFbTSJKIQjwGQbslhlEoxo0bx4oVK1iwYAFr167lne98Z0r+mjVruOOOO1i0aBHnnnsuF1988aDbcOutt3Lddddx7733cumllzJp0iRGjRqVUubAgQPceOONJLzu+snawY9+9CM+/vGPc8sttxAOh7nvvvu46qqreO6551i8eDEiwle/+lXOOuusbsd7knnz5vGlL32J1atXk0gkCIfDfPe736WsrCzjtQqFZFMVGyrU1tbqgCZh6mqFY2/CuLOhZNSpyxvGMGXbtm2cf/75hTajoHR2dhIMBgmFQjz33HPcfPPN3Y75oU6mz1dENqpqbbbnsNdqP9EOEw3DGOHs27eP97///SQSCSKRCN/73vcKbVJRYaLhJ2Z+DcMY6Zxzzjm89NJLhTajaDFHuB9zhhuGYfSLiYYf63ZrGIbRLzkTDRGZJiLPiMg2EXlVRP4yQ5kPisjL3vIHEVnsy9sjIq+IyGYRGYB3ewBowvk1DMMwjIzk0qcRA/6Pqm4SkVHARhF5UlVf85XZDVyqqidEZC1wJ3CRL3+VqqYGfck1sXYIl+b1koZhGEOFnNU0VPWQqm7ytpuBbcCUtDJ/UNXkyJn1wNRc2ZM1VtMwjCFFZWUlAAcPHuSaa67JWOayyy7jVN3xv/Wtb9HW1ta9n02o9ZFIXnwaIjITWAo830+xPwMe9e0r8ISIbBSRm3JnXRrWg8owhiSTJ0/ujmA7ENJFI5tQ64Wg0GHUcy4aIlIJ/Az4K1XNGDJTRFbhRONzvuQVqroMWAt8UkRW9nHsTSKyQUQ21NfXn7nBVtMwjILxuc99LmU+jS984Qt8/etfp6Wlhcsvv7w7jPnDDz/c69g9e/awYMECANrb21m3bh2LFi3iAx/4QErsqZtvvpna2lrmz5/PrbfeCrggiAcPHmTVqlWsWrUK6Am1DvCNb3yDBQsWsGDBgu4YU/2FYPdz3333sWDBAhYvXtwd3TYej/OZz3yGhQsXsmjRIr7zne8ALmjh0qVLWbhwIR/5yEfo7OzstuW2227jkksu4b777mPnzp2sWbOGCy64gLe97W3do8szXWvQUdWcLUAYeBz4m37KLAJ2AnP7KfMF4DOnut4FF1ygA6KzRfXApp4lHh/YeQxjiPPaa6/17DzyOdW7rhzc5ZHP9Xv9TZs26cqVK7v3zz//fN27d69Go1FtampSVdX6+nqdM2eOJhIJVVWtqKhQVdXdu3fr/PnzVVX161//ut54442qqrplyxYNBoP64osvqqrq8ePHVVU1FovppZdeqlu2bFFV1RkzZmh9fX33tZP7GzZs0AULFmhLS4s2NzfrvHnzdNOmTbp7924NBoP60ksvqarqtddeqz/60Y96/U8LFizQuro6VVU9ceKEqqrefvvtevXVV2s0Gu22qb29XadOnapvvPGGqqp++MMf1m9+85vdtnzlK1/pPufb3/52ffPNN1VVdf369bpq1ao+r+Un5fP1ADboaTzXc9l7SoD/Arap6jf6KDMdeAD4sKq+6Uuv8JzniEgFsBrYmitbe2FNVIZREJYuXcrRo0c5ePAgW7ZsoaqqiunTp6Oq/P3f/z2LFi3iiiuu4MCBAxw5cqTP8zz77LPd82csWrSIRYsWdef99Kc/ZdmyZSxdupRXX32V1157ra/TAC5U+lVXXUVFRQWVlZVcffXV/Pa3vwWyC8G+YsUKbrjhBr73ve91Ny099dRTfOITnyAUcn2RqqureeONN5g1axZz584F4Prrr+fZZ5/tPs8HPvABAFpaWvjDH/7Atdde2z1Z06FDh/q81mCTy95TK4APA6+ISDJwy98D0wFU9Q7gFmAccLvTGGLqYqBMBB700kLAj1X1sRzamkq0AyIVpy5nGMOZtV8uyGWvueYa7r//fg4fPtw9r8Xdd99NfX09GzduJBwOM3PmzIwh0f14z48Udu/ezde+9jVefPFFqqqquOGGG055Hu0nPl82IdjvuOMOnn/+eX75y1+yZMkSNm/ejKr2sq+/60BPGPVEIsHYsWMzxsPKdK30CarOlFz2nvqdqoqqLlLVJd7yiKre4QkGqvpRVa3y5dd66btUdbG3zFfVf86VnRmxmoZhFIx169bxk5/8hPvvv7+7N1RTUxMTJkwgHA7zzDPPsHfv3n7PsXLlSu6++24Atm7dyssvvwzAyZMnqaioYMyYMRw5coRHH+3pe9NXWPaVK1fy0EMP0dbWRmtrKw8++CBve9vbsv5/du7cyUUXXcRtt91GTU0N+/fvZ/Xq1dxxxx3dc3g0NDRw3nnnsWfPHnbs2AG4iLmXXnppr/ONHj2aWbNmcd999wFObLZs2dLntQYbiz2VCXOGG0bBmD9/Ps3NzUyZMoVJkyYBbhrVd73rXdTW1rJkyRLOO++8fs9x8803c+ONN7Jo0SKWLFnSPX/F4sWLWbp0KfPnz2f27NmsWLGi+5ibbrqJtWvXMmnSJJ555pnu9GXLlnHDDTd0n+OjH/0oS5cu7XM2wHQ++9nPsn37dlSVyy+/nMWLF7NgwQLefPNNFi1aRDgc5mMf+xif+tSn+O///m+uvfZaYrEYF154IZ/4xCcynvPuu+/m5ptv5ktf+hLRaJR169axePHijNcabCw0OvSERk8SCMFZCwfPMMMYIlho9OHNYIRGt9hTQDSeSE1IxCAeLYwxhmEYRYyJBhCLK7FEWo3LIt4ahmH0wkQDUJTOWFr3NBMNY4QynJqsjR4G63M10fBo7UoTDetBZYxASktLOX78uAnHMENVOX78OKWlZx6MdcT3nurqivKZb/4Xi6dV8fHVS3oyrAeVMQKZOnUqdXV1DEpIHqOoKC0tZerUM48JO+JFIxIK8PX4V/jN4VWATzRiHaAKGQYIGcZwJRwOM2vWrEKbYRQx1jwVCHKg9GzO6thFaicqtZn8DMMw0jDRAFqr5zFX93CoKU0kzBluGIaRgokGEJmyhHLpZMfuXakZVtMwDMNIwUQDGH/OhQC0HNyWmmE1DcMwjBRMNIDqqefSSiklJ7anZlhNwzAMIwUTDSAQClEXnsXE9p2kdE+Pd0GisFMrGoZhFBMmGh4nR5/NObqXo82dqRnWRGUYhtGNiYZHZMK5lEkXO3ftSM2wJirDMIxuTDQ8zpo5D4DmA6+nZlhNwzAMoxsTDY/xk2fSQhnhE2+mZlhNwzAMoxsTDY9AMMB+zxmegtU0DMMwujHR8NE06hzmJPZyvMVXu9A4xLoKZ5RhGEYRYaLhIzxhLqUSZVe6MzzaVhiDDMMwioyciYaITBORZ0Rkm4i8KiJ/maGMiMi3RWSHiLwsIst8edeLyHZvuT5XdvpJOsOb6tJGhptfwzAMA8htaPQY8H9UdZOIjAI2isiTqvqar8xa4BxvuQj4D+AiEakGbgVqAfWO/bmqnsihvZw1ZQbNlBFuSK9pmF/DMAwDcljTUNVDqrrJ224GtgFT0oq9B/gfdawHxorIJOCPgCdVtcETiieBNbmyNUkwGGBfaDYT2m2shmEYRiby4tMQkZnAUuD5tKwpwH7ffp2X1ld6pnPfJCIbRGTDYMw21jTqbGYn9nGy1TcyPNYJiUTfBxmGYYwQci4aIlIJ/Az4K1U9mZ6d4RDtJ713ouqdqlqrqrXjx48/M2OB4IRzKZEou3b7x2vYhEyGYRiQY9EQkTBOMO5W1QcyFKkDpvn2pwIH+0nPOWfNPB+Axv1pI8NNNAzDMHLae0qA/wK2qeo3+ij2c+BPvV5UFwNNqnoIeBxYLSJVIlIFrPbScs6UKTM4qeWEGtLCpJsz3DAMI6e9p1YAHwZeEZHNXtrfA9MBVPUO4BHgSmAH0Abc6OU1iMgXgRe9425T1YYc2tpNKBRgb3g249vMGW4YhpFOzkRDVX9HZt+Ev4wCn+wj7y7grhyYdkoaK8/mohO/oLWjk4rSEpdoNQ3DMAwbEZ6J4IS5RCTGnl0+Z3giCvFY4YwyDMMoAkw0MjDRGxl+Yn/6yHCrbRiGMbIx0cjAjKnTadIKgsfTwqRHza9hGMbIxkQjA6FQgN3h2dS0pYdJt8CFhmGMbEw0+qCx8hxmxPfR1ekfGW41DcMwRjYmGn0QGD+XiMTZs+uNnkQTDcMwRjgmGkCmnsETvJHhx/0jwzVhfg3DMEY0JhoAoVLShWP21Gk0aiWBY+lzhlsPKsMwRi4mGgCBAITLU5Ii4QC7QhlGhltNwzCMEYyJRpJIRa+kE5VnMy2+n3iX3xluNQ3DMEYuJhpJMohGYPy5hCXO3t0+Z7jVNAzDGMGYaCSJVPZKGj/jPAAa9vmc4XGbkMkwjJGLiUaSYAiCJSlJZ0+bRoOOQo69kVrWmqgMwxihmGj4SWuiKo04Z/i4VnOGG4ZhgIlGKhmaqBoqzmZqfD/qFwqraRiGMUIx0fCTwRku488lJAnqzBluGIZhopFCuBQCqfNSjZ/uRoYf2+cLk241DcMwRigmGumk1TbmzpjCMR0N/pHhiRjEo3k2zDAMo/CYaKQTThWN8kiAncHZVPdyhluYdMMwRh4mGulk8Gs0VJ7N1Nh+1D9PuPk1DMMYgeRMNETkLhE5KiJb+8j/rIhs9patIhIXkWovb4+IvOLlbciVjRmJVJAevFBq5hIU5ci+7T2J5tcwDGMEksuaxg+ANX1lquq/quoSVV0C/B3wG1Vt8BVZ5eXX5tDG3oj0qm3UeM7w+r2v9iRaTcMwjBFIzkRDVZ8FGk5Z0HEdcE+ubDlt0kTj3OmTqdcxaH3ahEyqeTbMMAyjsBTcpyEi5bgayc98yQo8ISIbReSmvBuVJhqjSoPsCM6mqsU/Z7jaTH6GYYw4Ci4awLuA36c1Ta1Q1WXAWuCTIrKyr4NF5CYR2SAiG+rr6wfHonBvZ/jx8rOZHK+DFGe4+TUMwxhZ9CsaIhIUkX/NsQ3rSGuaUtWD3voo8CCwvK+DVfVOVa1V1drx48cPjkXBkDebn+86NXMJohzbb3OGG4YxculXNFQ1DlwgIr0n0R4ERGQMcCnwsC+tQkRGJbeB1UDGHlg5Ja2Jany3M/y1nkSraRiGMcIInboILwEPi8h9QGsyUVUf6O8gEbkHuAyoEZE64FYg7B17h1fsKuAJVW31HToReNDTqRDwY1V9LKv/ZjCJVELb8e7d82ZO5shvxpI46hsZbjUNwzBGGNmIRjVwHHi7L02BfkVDVa871YlV9Qe4rrn+tF3A4izsyi1pNY2xpQF+F5jDrBbfWI14FyTiEAjm2TjDMIzCcErRUNUb82FI0REqgUAYEj0xpo5XzOGtLZtcCJFwuUuMtkNJ75DqhmEYw5FT9p4Skaki8qA3uvuIiPxMRKbmw7iCEylP2dVx5xJAaaqzJirDMEYm2XS5/W/g58BkYArwf7204U/apEzV088F4Ig5ww3DGKFkIxrjVfW/VTXmLT8ABqlva5GT5teYN30yh7Sa+FH/hEwmGoZhjByyEY1jIvIhb8xGUEQ+hHOMD3/C5SA9t6imIsB2mcXYZn/gQmueMgxj5JCNaHwEeD9wGDgEXOOlDX9Eeo0OP1ZxNhPjh6DL6yWscYh1FsA4wzCM/NNv7ykRCQLvU9V358me4iNSAV3N3buJcecQaFVaD71JxYylLjHa7npbGYZhDHOyGRH+njzZUpyk9aCq9kaGH9njc4ZbE5VhGCOEbAb3/V5E/h24l9QR4ZtyZlUxkdaDav70szio1cSOmDPcMIyRRzai8VZvfZsvTUkdIT58CQQhVNY9U9/EyiDPyGzmtfjmDLeahmEYI4RT+TQCwH+o6k/zZE9xEqlImd61vvxsJrZtgK4WVxOJdUIiAYFiiDRvGIaRO07l00gAn8qTLcVL2niNxLi5AHQcTo4MtwmZDMMYGWTzavykiHxGRKaJSHVyybllxUSaX6Nq6nmAOcMNwxh5ZOPTSI7J+KQvTYHZg29OkRKKpAQvnD9jInV/qCFqznDDMEYY2US5nZUPQ4qeSAV0NAIwZVSAp5nNIhsZbhjGCCObKLflIvKPInKnt3+OiPxx7k0rMnxNVCJCffkcJsQOQ6c38M9qGoZhjACyjXLbRU/X2zrgSzmzqFhJc4bHPWd41xHPGZ6IQjyWb6sMwzDySjaiMUdVvwpEAVS1HcjJnOFFTbgsJXhh0hmeMmd4zGobhmEMb7IRjS4RKcM5vxGROcDIi9CXFrxw3rTx7E+Mp8uc4YZhjCCyEY1bgceAaSJyN/A08Lc5tapY8TVRzRwbZJvMYtRJnzPcRMMwjGFONr2nnhSRTcDFuGapv1TVYzm3rBjxiYaIcLTsbGo6XnDO8JJR0HkSVF2txDAMYxiSVdwLVT2uqr9U1V9kKxgicpc3r/jWPvIvE5EmEdnsLbf48taIyBsiskNEPp/dv5IHIhX43Tnx6nMAeoIXJmLQ1lAAwwzDMPJDLoMl/QBYc4oyv1XVJd5yG3TP4fFdYC0wD7hORObl0M7sCQSdQ9xjjOcMP75vW0+Z1vp8W2UYhpE3ciYaqvosMJDX7uXADlXdpapdwE8opjk9fE1U86eNZ29iAh2HX+/Jj7X3jN0wDMMYZmQzuG+OiJR425eJyF+IyNhBuv5bRGSLiDwqIvO9tCnAfl+ZOi+tOPCJxpzqIK+R5gwHq20YhjFsyaam8TMgLiJnA/8FzAJ+PAjX3gTMUNXFwHeAh7z0TF5k7eskInKTiGwQkQ319Xl4WPu63QZEOFJ2NtWxo9DR1FOmo8nmDTcMY1iSjWgkVDUGXAV8S1X/Gph0phdW1ZOq2uJtPwKERaQGV7OY5is6FTjYz3nuVNVaVa0dP378mZp1akIRCEa6d2PVbmR4/OgbqeWstmEYxjAkG9GIish1wPXAL7y08JleWETOEnF9U0VkuWfLceBF4BwRmSUiEWAd8PMzvd6g4muiGjP1XAAa9r+eWqatARLxfFplGIaRc7IJjX4j8Angn1V1t4jMAv73VAeJyD3AZUCNiNThBgmGAVT1DuAa4GYRiQHtwDpVVSAmIp8CHgeCwF2q+upp/2e5JFIJ7ScAmD9lHHuen0jocJpoaNwJR2Ueaj+GYRh5IpvBfa8Bf+Hb3w18OYvjrjtF/r8D/95H3iPAI6e6RsHw1TTm1oR4nFm8tWl773Kt9SYahmEMK7LpPfXHIvKSiDSIyEkRaRaRk/kwrmgJl4EEAQgFhMOlc6iK1XfPt9FNvBPaGzOcwDAMY2iSjU/jWzh/xjhVHa2qo1R1dI7tKn58tY2o5wxPpDvDAVpHZsQVwzCGJ9mIxn5gq+dvMJL4neFTnDO8qe713uW6mi2QoWEYw4ZsHOF/CzwiIr/BFxJdVb+RM6uGAj7RmDelml0vnkXpodepylS25ShUzcibaYZhGLkim5rGPwNtQCkwyreMbMI9wQvPqwnzqs6iIpMzHFxPK5vVzzCMdIZgA042NY1qVV2dc0uGGoEAhMsh2kpJSDhYOocx0eecQJSl1zcU2o7BqLMKYqphGAUmHnXN1LGOnnWsA4IlUD3bDRoeImQjGk+JyGpVfSLn1gw1Ik40ALqq5sJR0Po3kOkX9y7begwqJ9pcG4YxnEkkXNDSaEfqOtFHS0OsHY696YQjUp5fWwdINqLxSeBvRaQTN0+4AGo9qHB+DS9cyJgp59J1JEjb9vWMzSQaiairhZRX59lIwzAGHVUXXy5dIOIDiDmXiMLxHVA1E0qL/7Har0/DC/MxX1UDqlpmXW7TiFR2b86fUsWjiYso3/Nk372lLB6VYQxdEgn34tewGw6/DPXb4MQeaDnsgpQORDCSaBwadkHr8UEzN1f0KxpeN9sH82TL0CMYdm2SwPwJYX6i7yASbyX6xpOZy0fboLMljwYahnFGqDpBOLEHjmx1645G0EQuLgZN++DkoRyce/DIpvfUehG5MOeWDFW8rrelIWHKnIVsS8wgvvWBvntFtB7No3GGYZw2qtBxEhr3weFXXA2g/YSrDeSDlsNOnIq0Z1U2orEKeE5EdorIyyLyioi8nGvDhgy+JqoPLargh/F3UHpyD7FDfdyijpMQ68qTcYZhZE1nCzTudzWKhp3Qdjx/QpFO+wk4vrMoI2Vn4whfm3MrhjK+Hg/n1ZRwaMJKmht+TOjlBwhNXpzhAHW+jTHFMxmhYYxYulpdfLiORogX2ctcV7PXs2pOUXXJzSbK7d58GDJkSQYv1Dil4QBXzR/LT399KTfsewJtPY5UjOt9TNtxGDXJjfUwDCO/RNvdm3x745k5r/NBrKPouuTaU2sw8IUUefusMp4pW02QOJ1bH85cXuNOOAzDyA+qrmfS0deh/nVoOVL8gpEkEYXj21OnlC4gJhqDgc+vMao0zFvmz+Q38UXotl/0PajHut8aRu5JxKH5CBx51fVMig3R4KGacF19i6BLronGYOCraYjA1eeV87PAasq6Gujc8WzmY+KdRfPmYBjDjngUmg44sWg+6N7WhzzJLrkHC2qFicZgEC4nGbwQYHxlhLFzLqROa+h8uZ9hLi1W2zCMQSXaASf2OrFoPVq43k+5pOVIQbvkmmgMBsnghR6hgLBuwSh+HL+C0Q2vED22M/NxNteGYQwOnS2ui2r9NmhvAIpzjMOgUcAuuSYag4WviQpgTk0phya/g04N0/bSA30fZ74Nwxg47Y1wbLtzFHeOsFmok11y8zzuy0RjsEgTjZJQgHctnMgvEhdTuvdp4h2tmY+zuTYM4/To7gm1DU7shq4RHJon2SU3j8+QnImGiNwlIkdFZGsf+R/0Rpi/LCJ/EJHFvrw93sjzzSKyIVc2Diq+HlRJlk8r5fej1lCS6KDt1UcyH6cJN9eGYRj906snVEehLSoOEtG+e2nmgFzWNH4ArOknfzdwqaouAr4I3JmWv0pVl6hqbY7sG1yCIQiVpiRVRkIsXTCfzYnZxF99GE30FY/qWNHGmTGMvKHaM1lRZ7Orhbceg+bDXniP4dQTauiSTRiRAaGqz4rIzH7y/+DbXQ9MzZUteSNS0evtZ+3Z5dz+4mpu6biDtr0bKJ+VIfajzbVhDAcSCS/6q7q1Jtdx9yacSK5jmfdzEjnWGGxyJhqnyZ8Bj/r2FXhCRBT4T1VNr4V0IyI3ATcBTJ8+PadGnpJIZa+R3tXlESrmXkrD6/9LdPMDmUVjN0ZyAAAfVUlEQVQD3BuViYaRbxJx93afiLmXl3hyHXUPe1WvFpwuBGniMNx7KxndFFw0RGQVTjQu8SWvUNWDIjIBeFJEXlfVjKPkPEG5E6C2traw39zSMSCBlDemQACuXjiWB7at4sb6X9LeeJiysRnmCo+2um6DJb19I4Zx2vgf/omYt86wb2/3xmlSUNEQkUXA94G1qtr9iq6qB731URF5EFgO9DG0uogIBKF0rNdPvIcpY0o4PHUNcuAXnHzpIcpWfSLz8a31JhpGKom0ph2NZ5dmb/5GjiiYaIjIdOAB4MOq+qYvvQIIqGqzt70auK1AZp4+5eN6iUYkGGDNkpk8s38JF+16lOiKGwlHSnof29Hk+lwXURhkI0fEo26O6XiXb4m6h77fB2APf6PIyJloiMg9wGVAjYjUAbcCYQBVvQO4BRgH3O6mIifm9ZSaCDzopYWAH6vqY7myc9ApqXS9qNIc4vMmlPKdsWu5vPn/5/i2XzFucaZpStR1vx09OT+2GrkjHnPxxeJd7kXALw6xTkwMjKFKLntPXXeK/I8CH82QvgvINHvR0KG8Bk7WpSZFgpy/eDm7n51IySsPkVi4NvN0Gq3HoPIsm2tjKJBIOF9UtL13rcF8BcYwxZ5MuaC82jnE07h0ZgWPlaxmctsbNB/YlvlYjfdq3jKKBFU301vzYTi2A468Asd3wMkDrobYedLVME0wjGGMiUYuSDrE0xhdFqbk3NW0a4TjG/uJR9VU58I6F+H8wCOOrjZoOeqCwx1+xYVsaD7k4v6YOBgjEBONXFFR0ytJBNYumMCjsoIpR39D88kTfRysLqzz0deKYtKVEUW0wzURNuz2ROINV5PoPDk8w2wbxmliopErIhUQKuuVXFMZ4ci0d1JCF8de+mX/50jEXIydo6+7sArG4BPrgrYGNz/B4a0utHbTfuhozGs8H8MYKhR8cN+wpqLGPYB8hAPCpcvmsXHvXKbt/AUdb1lHaeQUH0Os3bWdl46B0VMglKG77nAmkfANTIu6fbRndDL+UcvZrPHiHHUNnXmiDaNIMNHIJWVVrmkjre17dnUpd1ev4YLGb7P/zeeYtuBt2Z2vowk6TkLlhOHRw6rfUcvx1HAWhmEUBSYauSQQdMKRFo+qNBxgzrLLqH/6h7RveZjovEsIB6SPk6SjbrrHtgY3nmMoxKvqanPNa9FWJwJJkbCxCoYx5DDRyDXl43qJBsAFU0fzaMnlXNP6IMeP7GP8pBmnd95EFBr3utAjo6cUV/iRWJcTic6TboIc8w0YxrBhiLdvDAEiFSnzhyepLAkRmnclCRUOvfCQa6YfCNE2N9Vlw+68T/vYTSLupt1s3A9HXoOj3iQ55kw2jGGHiUY+KB/XK0kEVs6fxm8Dtcw48hQnW/uYDjZbOhpdz5+Thxi4AmWJqovIe/IQ1L/puqae2O0GuJlj2TCGNSYa+aCsKuMI8aqyEo5OfydjaGbvpifO/DqagJbDbnxH2yCPKo+2Q0u9N8jtZVe7aTns/BTmmzCMEYP5NPJBHw7xYAAuql3Ozj1TGLXzF7S85V1Unqr7bTYk/R3NhyEQIuWh3mta2bT9TPmasGYmwzAAq2nkj/LeI8QBJo0p4eXqNcyO7aRu+yuDe814pxdQr61nibWnLR2pS7wzbekywTAMoxsTjXwRKc/oEC8JBZhRu4YWLaVly8N0xiyekWEYxYs1T+WT8hrXqyiNuVOqWF+2krc1P8ORo0eZPjnDdLCGYQwNYp3QfsK3NEBb2n5yG4EZb4FZl8LUWggW/wRsJhr5pHuEeOoI58pIiND8d1Oy8Qn2vPhLJr/rI4SyHuxnGEbeiHW6kD4Nu1xnk3QRaDvhdQ7JQKQCyqrdc6BqFkxe5pqM9/we3nzctUTMeCvMWgnTlrvJ3IoQE418Egh4DvFjvbIWnzeXzS/N45wjj3K8eR0Tx1QUwEDDMLpRdS95R1+Do9vc+vjOVB9fyWj3my6rgnHnwLTqnv2yKicS5VVuqoS+YsbFo3BwE+x+Fnb/FnY85QRj2kUw+1KYdrFr3i4STDTyTfm4jKIxpizC0envZMmef+XXm35P1+K3Ma4yQnkkWAAjDWME0nHSjXU68ppbH33dRTUA9xAffx4svBYmzIOac9xvORg+8+sGw04gpl0El/w1HHoZdv/GE5HfuPypy52AzHgrRAob/cFEI98kHeLRtpTkQAAWXHgpR/d8nzG7f0nH/LdyoLGd8kiI6oowZWETD8MYNOJRV2tI1iLqt7nJzwAQqJoJMy9xAjHhfLcfyMNvMBCCKcvc8ta/gCOv9gjI3t97+bWuCWvmChf5Os+YaBSCPhzi40dX8Pvq1VzWcC+Pv/IHZs2/mLauGG1dMSpLQlRXRCgJWYc3wzgtEjFo3Oem6D32phOIY2864QDXhDThfJi7FibOg5pzi6M5KBCESYvc8pZPOnFL1j6eXQ+/DcDkpc6JfsE4qJqeF7NEew3mGsSTi9wF/DFwVFUXZMgX4N+AK4E24AZV3eTlXQ/8o1f0S6r6w1Ndr7a2Vjds2DBY5ueORAKObM0Y8nv73gOMfvwvmMhxdsl0Ds18DxPmr4JQBAEqS0OMqyghHDRHuWH0ItrmahDHd8Cx7W59YnePQAQjUDPX1SAmerWIigkurs9QQdVFZNj1GycgTXXOt/K3uwbUXCYiG1W1NuvyORaNlUAL8D99iMaVwKdxonER8G+qepGIVAMbgFrckOWNwAWq2tf8qMAQEg1wwf0y+Dbau+Lsq29k/5ZfMbPu55zNfhoYw/4pV1K58J0kSsciwJiyMFUVEetlZYxc2o672sPxHe4henwHNB2gO8pByWioOds5qMed7Zax07woCcMEVSeKiTgsev+ATnG6opHTu6eqz4rIzH6KvAcnKAqsF5GxIjIJuAx4UlUbAETkSWANcE8u7c0rFTUZRaMsEqS8rIwZtWuJLv0jHty6kYm7H+atB+6h88D97JuwitCC99LIdE52RBlbHmFsWYSgtVoZw43k7IrRNjcBWcMuTyC8WkS77x1y1CQnCues7hGIivFDqwYxEESgejaMPz9vlyy05E4B/POh1nlpfaUPH8JlEK7I2Kd7XGUJR5tdtNgFiy8kuqCW/3l9N5XbH+bKI89QevQJDo5ZRmLeVTRMWEJTW5SqighjSsNDfjI/Y4ijCfegj3W4UP2xDvfQ72pz3/WuVrfd5YW36bVuSc1PD2EjQaie6cYxdNcg5kDJqIL8uyORQotGptcA7Se99wlEbgJuApg+PT+OoEGjogYae4tGeSTIzHHlNHfEONHWBSR4y/zZRM/7K36w/UPwxiO8r/EJxj/3/9FYPoOOue/l+LTLaGyPUF1ewujS0LB/wTLOgOQbfPqDOv2h3h2TrKsnLlnMW/zb3fsdPb6DbAiG3YtTpLxnXXmW610YSUuPVLoeTFUzhsSo6eFMoUWjDpjm258KHPTSL0tL/3WmE6jqncCd4HwauTAyZ5SOBanrcw7sUaUhRpWGaOmKcaIlCsS5/LzxdJ3zp/x4xzU0bnuG97f8kvM3/xvtr/6Q1jnv5PisK2msqKK6PMKo0kJ/vMagoeoFoGz3ljbftm8/1t7z0M/49u6lZxOEUoJufEIo4gamhUrdAztU6h7i5eMgWOLllaRuJ48LlvRMRBap8IlBuT38hyg5dYQDeD6NX/ThCH8n8Cl6HOHfVtXlniN8I7DMK7oJ5wjvd5KIIeUIT9JU56ZszYK2rjgNrV20R53IdMWVR3d2sfO1jVwbf4S3BzcTlwgt01fROOc9UD2T8kiQskiQslDI/B6DiSZcm3pLPXQ1e/Oex93DOOPi5fVXLtbpHvrRDEus3V0zGyToe0BXZHhoV3hv8P63+Yq0h3rF4AxcM/LD+PMhPLCwI0XlCBeRe3A1hhoRqQNuBcIAqnoH8AhOMHbgutze6OU1iMgXgRe9U912KsEYspSPy1o0yiNByiNltEfjnGiNQleM98wtoWvOW3hkZy3ffW0X74s+wvv2/ooZex+ndcIFNE9bxZGzlpMIl1MaClBqInJqVN0c561HocVbktut9d76mJu3ZCBIwPXgSV9CJZ6vqwzKqyHkbYfLe9JTtn37oTKIeOtgZPg7gI2CkfOaRj4ZkjUNcFOm9hXkrB86YwkaWrto6Yx5+8ovd3bx+LZjXBl9ihsiT1OjDSQkRNvEZbRMXkHrWReRiFQiuLDsw0JEVF0TXyLes07Zjvn2fdudLZ4Y1PcWhlhH6jUkCJXjXZ/+ygmuZ07lBLdfOhoCYTcYyy8CQb8o+PIyzOJoGGdEHmsaJhrFQFuDm2lvgHTFE5xo7aK5I4bixOMXO7t44PUOpnZs58rg87w7/AIT9LgTkAlLaJ28gpZJF5OIuF4nBRGRWKdrb+9sdkv3dkv/6dG23mJwRoh7s08KQlIMKn3CUFaVnzAShjEQTDQGxpAVjX5GiJ8O0bjS2NZFU3sUBVSVnY0J1h+Isf5AF6WN21kbfIF3h55nMvUkJEhbzSJap1ziBKSkJ46NE5EgpZEApaEgEujp0ibelnT/cWn+FhEhAc2HCTTuRU7sRRp3IycP9ohAVzNyip42GipFSka57pSRSrcuqXRNMoGQe/vvfoMP+vaDfeSlbYfLnDCU15yy/T7TzySZZt2cjYJjojEwhqxowGk5xE9FNKE0tUVpau8i4ft4G9oTPH8wxvoDUTqPbOdyeYF3Bp9nhhwhQYDmcQvpnLqClklvIV5ald3FNEGorZ5I8z5KmvcSObmPSPM+Is37CcQ7e2wqHUe0cjLxyGgS4Qri4UoSkUoS4Uri4QpvnZqWPnI3H630A/01BEUIBoWACEERAgEIBsQtEiAQgJAIAS/NRvIbg4qJxsAY0qIR7XCB1AaRRAJau2I0d7igh/5PujOmbD7qBOTEgZ1cHHuedwaeZ3bgEAkCNI6dR3TaJbROeSvx0mpQJdTuxCFych8lzfuInNxLpKWOQKy9+7yx0mq6Rk2nc9R0ukZPp2vUDLpGTSNR4HDOxUgwKSLihEQks//anyQivdL8tb1MnI5PvM+ikr576pOml5A+DOnrTL2Ki7tnIhAQJ9AiEECstmeiMTCGtGiAC43Q1ZKTU8cSSktHjObOGB3R1Gaw7masuiiHDuxmfstzrA08z9zAARIITaVTqYweIxzvEYfO8FhaK6fTXjmdjsrpdHoCISWVhAIQCrgftmHkA4EeMQkIAVxCUmSkW3B6vpMiPcemnKh7s3+RE5Hu6/oz/eeV5FkyNOOKuP1kzfSMMNEYGENeNM7QIZ4t0bjS0hmjpSNKR6x33/9kM9b+fXuY2vAHFrKDfTqB7TqV7YkpvKlTaeTUYRsCAkGBsNdUE/L8IgHx/8iT2+K2BQK+PHzlXb73Q02/2Cne0DPuS+q5k+dPbvvt6Cnns9MrUxISxpYIY0uFMSVuGVsijCkNUBnu+w3bMJII7jcSCLgaVMireQa9mmjAq40GAhCUgNsW6emsMlzGaRinSelYCBzIbrTuGRAOClXlYarKw3TGEk5AOmN0eQJSXRZg7ZwIzJlLZ+wcdjbGqYzDQoXzE0o8AdEExBMQUyWWwFuUuJKyH0tAXJ2fJZ5wzuME3tp7X0koKOrWvvyU7WQ51RQ/TV+kF8noyO4+p1u7ayXctu+ayTJx7/pKctvltceU9j4+slAARkfSBSWQQWCEspD0Ek0hs5j2iGiP6BpDF8X9XrL6cvtIvshMqY5TkqexmCYaxUQg4CaEaT2at0uWhAKUhCKMq4jQEU3Q3BmlpTNGLK5evjCvxr4mp6IrrjR2KE2dSmOn0tiRoKmzZ7+pU2nqUN5sjdPYGaNtgOMC+yK95tadnhQVr5D4yuPL69GcniaX1BqYqzUKdDf/+GthqTU2SSkv/uumpYn0vib4/g/fectCQnlYqAjjrd2STPPvl4VGhpC6FxjN+FKUK+xpUGyUj8uraPgpDQcoDZcwvrKEtq6414QVIz6MmjBzRSQoTKgQJlRkV74rrpz0CUpjh9IZ1141MUV926k1o5TaF6m1N6WndpXc9n+KyX1NFsC7lu84f80vpUaWobbVY49mqJ31vl7CM0Lp+R/TbfXbHFfoiCqtUaUtm7BZQLknJOU+MSkPQTDQU4NLX7od636h9DVF+pswQ4GeptdwwJUNec2wPYtvX6RXeiTobIwMoUnVTDSKjXApREa5eEYFxIUsCTK+soSOaJx48ofd/aDo+cHje2Dgz0t5+GjKAyq5kfIgS659DzF/Rl/apfSRcQo09fR5JRIUasqFmiKYVXSokVClPQqt3SKitHY5MXFrt9+att/UmeBwixOghPaIsV8Uk02QvYWyRxhzQThA71pTpKc2VRGGypT9nvSKiDAhFmdgHo3Tx0SjGBk9GVqOuIFwZzza+cwQcRNDDXf8b8OJbiH0vQH79lPeyL2HSVwTxONKXJVYXEl4a6ulDT4BESoi7mGZb5I+tYRCTH2+vG4/n7q1pvr1eue59M44tHni1xp14tbiCWJjc8KJXxa1q7GPP8fmW/8oL/fARKMYiZRD9SwvcN5JN2tZx8mBB8gzTol/jEQwteX/jFB1nQUSCSWWSJDofpAkiHsPm0QiQSzhhCZ5TK/znLElxmAgIgQFgniRV11qzq8bT7jOFn5xcdsurWritFOfZJAw0ShmRKB0jFvAxV7qaHKLb7S1UbyIQNhrFC9hcEag+Ws/kN7Ep77t9APTd1MT+guV4m+GTO6nNCdq6jE9eb2bIXub1cuwjLuJhKu5Jby39gRK3KvVDXdRDQaEyohrosrEtHPzN7GpicZQoqTSLWOmuDkWkgISbSu0ZUYe8Q8Oy5B7OmcaBGuKg0TC9SJKqLoanCcuyf2edZqQ+XxsPYLorTXVr+YX0EL6wwqNicZQJTmfwqiz3HScSQHpamFkfpWNkUwg4Ho9FUIIE4lkzzBN9X1l8oN5Y2lTOpN0+0k01UGv2pOXKJ4alYnGcCAU8cJ4j4d4zPODNHqO9CxnezMMY0D0hAAZPF9Yf8QTrrOGerUpVSWSx8lwTDSGG8GQmxuivNoJSNN+JyCGYQwLgoFkZw3PIw+nF5XyDBnpsSGHN8GQ64VVNbNXmHHDMIyBYE+SkUBZlRsweLIO2k8U2hrDMIYwVtMYKQRDrsZRNcvNZ20YhjEArKYx0igb66ZOtVqHYRgDIKc1DRFZIyJviMgOEfl8hvxvishmb3lTRBp9eXFf3s9zaeeIw2odhjFMEAiXQyB/oX5yVtMQkSDwXeAdQB3wooj8XFVfS5ZR1b/2lf80sNR3inZVXZIr+wxcraNklJufvL2h0NYYhnEqAmEXZihcAZEKTzDy62XIZfPUcmCHqu4CEJGfAO8BXuuj/HXArTm0x8hEIAhVM5yANO63+FaGUTSIG8CbFIdIBYRKCm1UTkVjCrDft18HXJSpoIjMAGYBv/Ill4rIBiAGfFlVH8qVoQYuvtWESqt1GEahSNYiIpVOJApQi8iGXIpGptEmfY2CXwfcr5oSB3y6qh4UkdnAr0TkFVXd2esiIjcBNwFMnz79TG0e2XTXOqrcoMB4V6EtOg18YWqz3QZvxLy6tSZ6R98z8oyABLyww4GeBXHTICdihZsuQIJuvFMwnMGHkOFxl3HAXVpaIOTVJipdZIchQC5Fow7wx+udChzso+w64JP+BFU96K13icivcf6OXqKhqncCdwLU1tbar30wKB0NkfPg5AFoOz7Ak4j7YSV/aIFg6kMg/aGQfDD0Su+nHORmJGx34KAMgpKS5stLxCAe7XmwJfcLPB/KoNL9GWT4LHt9vsEMn2EGIej1WWfxeapmuN9x17Tq30/m9/cZBELuDT8Qch1Eurc9YUjuB0JF+dZfCHIpGi8C54jILOAAThj+JL2QiJwLVAHP+dKqgDZV7RSRGmAF8NUc2mqkEwjC2OlQOhaaD3lpIffj7haBYM+PKRBKFYg89uYYdLofXoPwkMj4gItlTlPtebD6BTRFJL2Hq/+NvNd2prXvf/Pn9Zfmf/gX0wNTxD3Ug1n2/PN/BhrvEYqgjTgYCDm7a6oaE5FPAY/jIqTcpaqvishtwAZVTXajvQ74iSbjFTvOB/5TRBK4X+6X/b2ujDxSOtotxsA43QecMfjYZzCoSOqzemhTW1urGzZsKLQZhmEYQwYR2aiqtdmWL6I6p2EYhlHsmGgYhmEYWWOiYRiGYWSNiYZhGIaRNSYahmEYRtaYaBiGYRhZY6JhGIZhZI2JhmEYhpE1w2pwn4jUA3sHeHgNcGwQzckHQ83moWYvmM35YqjZPNTshb5tnqGq47M9ybASjTNBRDaczqjIYmCo2TzU7AWzOV8MNZuHmr0weDZb85RhGIaRNSYahmEYRtaYaPRwZ6ENGABDzeahZi+YzfliqNk81OyFQbLZfBqGYRhG1lhNwzAMw8iaEScaIrJGRN4QkR0i8vkM+SUicq+X/7yIzMy/ld22TBORZ0Rkm4i8KiJ/maHMZSLSJCKbveWWQtiaZtMeEXnFs6fXBCfi+LZ3j18WkWWFsNNnz7m++7dZRE6KyF+llSn4fRaRu0TkqIhs9aVVi8iTIrLdW1f1cez1XpntInJ9gW3+VxF53fvsHxSRsX0c2+/3KI/2fkFEDvg++yv7OLbfZ0uebb7XZ+8eEdncx7Gnf49VdcQsuBkEdwKzgQiwBZiXVubPgTu87XXAvQW0dxKwzNseBbyZwd7LgF8U+t6m2bQHqOkn/0rgUdzcohcDzxfa5rTvyGFc3/Wius/ASmAZsNWX9lXg897254GvZDiuGtjlrau87aoC2rwaCHnbX8lkczbfozza+wXgM1l8b/p9tuTT5rT8rwO3DNY9Hmk1jeXADlXdpapdwE+A96SVeQ/wQ2/7fuBykWxmux98VPWQqm7ytpuBbcCUQtgyyLwH+B91rAfGisikQhvlcTmwU1UHOkg0Z6jqs0BDWrL/+/pD4L0ZDv0j4ElVbVDVE8CTwJqcGeojk82q+oSqxrzd9cDUfNiSDX3c42zI5tmSE/qz2Xt2vR+4Z7CuN9JEYwqw37dfR++HcHcZ74vdBIzLi3X94DWTLQWez5D9FhHZIiKPisj8vBqWGQWeEJGNInJThvxsPodCsY6+f2DFdp8BJqrqIXAvGcCEDGWK+X5/BFfrzMSpvkf55FNec9pdfTQBFus9fhtwRFW395F/2vd4pIlGphpDevexbMrkFRGpBH4G/JWqnkzL3oRrSlkMfAd4KN/2ZWCFqi4D1gKfFJGVaflFd48BRCQCvBu4L0N2Md7nbCnW+/0PQAy4u48ip/oe5Yv/AOYAS4BDuOaedIryHgPX0X8t47Tv8UgTjTpgmm9/KnCwrzIiEgLGMLDq6qAgImGcYNytqg+k56vqSVVt8bYfAcIiUpNnM9NtOuitjwIP4qrufrL5HArBWmCTqh5JzyjG++xxJNm0562PZihTdPfbc8b/MfBB9RrX08nie5QXVPWIqsZVNQF8rw87ivEeh4CrgXv7KjOQezzSRONF4BwRmeW9Va4Dfp5W5udAsnfJNcCv+vpS5xqvPfK/gG2q+o0+ypyV9LmIyHLcZ3o8f1b2sqdCREYlt3FOz61pxX4O/KnXi+pioCnZxFJg+nwrK7b77MP/fb0eeDhDmceB1SJS5TWtrPbSCoKIrAE+B7xbVdv6KJPN9ygvpPnbrurDjmyeLfnmCuB1Va3LlDnge5wP734xLbieO2/iejr8g5d2G+4LDFCKa57YAbwAzC6grZfgqrgvA5u95UrgE8AnvDKfAl7F9dZYD7y1wPd3tmfLFs+u5D322yzAd73P4BWgtgi+F+U4ERjjSyuq+4wTtENAFPdm+2c4f9vTwHZvXe2VrQW+7zv2I953egdwY4Ft3oFr/09+p5O9FScDj/T3PSqQvT/yvqcv44RgUrq93n6vZ0uhbPbSf5D8/vrKnvE9thHhhmEYRtaMtOYpwzAM4www0TAMwzCyxkTDMAzDyBoTDcMwDCNrTDQMwzCMrDHRMIzTxIt6+plC22EYhcBEwzAKgIgEC22DYQwEEw3DyAIR+QdvroSngHO9tDki8pgX7O23InKeL329iLwoIreJSIuXfpm4+VF+jBsshoh8SERe8OYz+M+kmIjIahF5TkQ2ich9Xvwxwyg4JhqGcQpE5AJcWIiluFg+F3pZdwKfVtULgM8At3vp/wb8m6peSO/4Q8txI2/nicj5wAdwQeOWAHHgg15Mq38ErlAXTG4D8Dc5+wcN4zQIFdoAwxgCvA14UL04SSLyc1y4mbcC9/mmWynx1m+hZ16LHwNf853rBVXd7W1fDlwAvOidowwXcPBiYB7wey89Ajw36P+VYQwAEw3DyI70eDsBoNGrIZwOrb5tAX6oqn/nLyAi78JNmnTd6ZtpGLnFmqcM49Q8C1wlImVeVNB3AW3AbhG5FrrnPV/slV8PvM/bXtfPeZ8GrhGRCd45qkVkhnf8ChE520svF5G5g/5fGcYAMNEwjFOgbsrde3ERWX8G/NbL+iDwZyKSjBKanN7zr4C/EZEXcPO8N/Vx3tdwvosnRORl3DSsk1S1HrgBuMdLXw+cl4N/zTBOG4tyaxiDjIiUA+2qqiKyDrhOVfMyX7Rh5BrzaRjG4HMB8O/epE2NuLksDGNYYDUNwzAMI2vMp2EYhmFkjYmGYRiGkTUmGoZhGEbWmGgYhmEYWWOiYRiGYWSNiYZhGIaRNf8PWTbcoweEpUMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c6079b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_with_err(x, data, **kwargs):\n",
    "    mu, std = data.mean(1), data.std(1)\n",
    "    lines = plt.plot(x, mu, '-', **kwargs)\n",
    "    plt.fill_between(x, mu - std, mu + std, edgecolor='none',\n",
    "                     facecolor=lines[0].get_color(), alpha=0.2)\n",
    "\n",
    "plot_with_err(degree, val_train, label='training scores')\n",
    "plot_with_err(degree, val_test, label='validation scores')\n",
    "plt.xlabel('degree'); plt.ylabel('rms error')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the trend here, which is common for this type of plot.\n",
    "\n",
    "1. For a small model complexity, the training error and validation error are very similar. This indicates that the model is **under-fitting** the data: it doesn't have enough complexity to represent the data. Another way of putting it is that this is a **high-bias** model.\n",
    "\n",
    "2. As the model complexity grows, the training and validation scores diverge. This indicates that the model is **over-fitting** the data: it has so much flexibility, that it fits the noise rather than the underlying trend. Another way of putting it is that this is a **high-variance** model.\n",
    "\n",
    "3. Note that the training score (nearly) always improves with model complexity. This is because a more complicated model can fit the noise better, so the model improves. The validation data generally has a sweet spot, which here is around 5 terms.\n",
    "\n",
    "Here's our best-fit model according to the cross-validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD9CAYAAACoXlzKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VNX9//HXhxAwIBAEZAkEUBFxK9iUahEXUHFF3Fq3al1KF9v6bb+iuHyr31a/Yqlbf1pbqlZpbQUVEVxABLdaQUEiiohV9gAhIgGFCAk5vz9uJiRhljuZO2vez8djHklmbu49NzP5zJnP+ZxzzTmHiIjkjlbpboCIiARLgV1EJMcosIuI5BgFdhGRHKPALiKSYxTYRURyjO/AbmaPmtkmM/uwwX0TzexjM1tiZs+aWWFymikiIn7F02N/DDi1yX1zgMOdc0cCnwA3BtQuERFpJt+B3Tn3BvBFk/teds7V1P04H+gdYNtERKQZgsyxXwm8FOD+RESkGVoHsRMzuxmoAZ6Iss1YYCxA+/btv3nIIYcEcWgRkRZj0aJFnzvnusXaLuHAbmaXA2cCI12UhWecc5OASQAlJSVu4cKFiR5aRKRFMbPVfrZLKLCb2anADcDxzrkdiexLRESCEU+54z+Bt4GBZrbOzK4CHgA6AHPMrNTM/pSkdoqIiE++e+zOuYvC3P1IgG0REZEAaOapiEiOUWAXEckxgZQ7iogEYfriMibOXs76yip6FRYwbtRAxgwpSnezso4Cu4hkhOmLy7hx2gdUVe8GoKyyihunfQCg4B4npWJEJCNMnL28PqiHVFXvZuLs5WlqUfZSYBeRjLC+siqu+yUypWJEJCP0KiygLEwQ71VYoNx7nNRjFxHAy3EPmzCP/uNfYNiEeUxfXJbS448bNZCC/LxG9xXk53HiId24cdoHlFVW4diTe091+7KJAruI1A9cpjN4jhlSxJ3nHkFRYQEGFBUWcOe5R/DqxxXKvcdJqRgRn3I5HRBt4DKV5zhmSNFex/vllNKw2yr3Hpl67CI+ZEKPNpkyeeCyV2FBXPeLAruIL7leipfJwTNS7n3cqIFpalHmU2AX8SGTe7RByOTgGSn3nitpsGRQjl3Eh2ileLkgFCQzdQwhXO5dIlNgF/Fh3KiBjaa7Q+b0aIOi4Jk7FNhFfMj0Hm0miqeKKLRtWWUVeWbsdo4i/Y2bzXdgN7NH8a5tusk5d3jdffsBU4B+wCrgu865LcE3UyT91KP1L54FvZpuu7vu0slaBKz54hk8fQw4tcl944G5zrkBwNy6n0UkSaLNDk33zNGG4qkiCrdtrN+R6OK5NN4bZtavyd1nAyfUff848Brexa1FJGDResFA1B5yqidXxVNFFKuyKFcqj1Ip0Rx7d+fcBgDn3AYz2z+ANkkGyeXZltkmVi842mOpXuc8niqiSNtG+x2JLmV17GY21swWmtnCioqKVB1WEpDrsy2zTbRecLTH/KRFgk7jxFMXf+Ih3SLuJ/Q7mZRmygaJ9tjLzaxnXW+9J7Ap0obOuUnAJICSkhKX4HElBTJl/ZCWwM8no1i94EiPxUqL3DL9A56Yv4bQP2UQPfp4qohe/Th8Ry/PjDvPPQJI/SeObJdoj30GcHnd95cDzyW4P8kguT7bMlP4/WQUrRcc7bFoywVMX1zWKKiHBDFoOWZIEW+NH8HKCWfw1vgREYNwpNdTrXOMGVKU88s5JEM85Y7/xBso7Wpm64BbgQnAVDO7ClgDXJCMRkp6NGe2pXLy8fP7ychPL/i2GUuprKoGYJ98r98WbXLVxNnL9wrqIal6A4/1OovUjrLKKoZNmJfQay1XX6/xVMVcFOGhkQG1RTJMvLMtdTHi5onnk1G4WvqGk3uswf1bdlRz47QPuPPcI7jz3CPCBrBIS+JC6gYtY73OIgV+Y0/6qTmvtVx+vWoRMIko3sWX9JG5eRJZWbFhGgcIm1L535lLI6ZFIh3DIGXLJcR6nYVLMxnhzzWe11ouv161pIBEFc9sS+Xkm6c569A07KXHsmVHNdMXl4V9HsMd24BLji5O+wU2Gj4GjVNQkc47ntdaLr9eFdglMLm+AmKyxLsOTdMUgh+RKpmyZQ2cpoF/2IR5Cb/Wcvn1qsAugcnGFRAzZfDM7yej6YvL+O+p79evp+JXtF5o0+AeSkUE+ndwDqqrYfduaNMG8vJi/04UQbzWsvH16pcCuwQmW3p/IckaPEvWm0WovfEGdYhdyeTr77BzJ2zcCOvXe7cNG7yvmzdDZeXetx07vGBeXQ21tY0PmpcH++wDbdtCu3bQpQt06wZdu3q3vn3hgAP23Dp2bPTrsV5rDZ+Dwnb5OAdbq6obbZdtr9d4mGvGiyRRJSUlbuHChSk/rkhDkT7OFxUW8Nb4Ec3aZ7g0SUF+XiBX/InU3lgaHj/cm04oV996dw29tlVQXLmR4q0bObTqcy7dfzesWAGrV3sBvKnWrb1AXFjo3Tp12vN9u3aQn7/n1rq1F9Crq+Hrr73bzp2wfbu374oK+Pxz2LQJtm1rdJjPO3Thg279WNfnYA45dTjfuuRM6NMn7PnGSlUF9Xykg5ktcs6VxNpOPXZpsZIxeJbM2brNaZdBo6B+4zNL6LClgmM2r+WgzWvZ/sx67qpYQ9/KjfTaVkGe29Oz3pnXGg460OsxDx0KRUXQqxf07Ol97dXL62m3SkJx3ZYtsHIl78xbyGsvzufAitUM2rSSY9+cSv7r/4Qbfwr9+8Pw4XDiiXDaaUxfX+NrQLklzJ5WYJcWKxmDZ9Em04QTT9omUntDF6ZodF/tbvpUbmTA5nWMeflD+MPHDJi7gAXlq+m4a0f9dtvatGNFl94sLBrEmsNOZG1hD9YU9mBNpx607l3Ev246ye+pB6tzZ+jcmbEzP6dy6J6/R5uaag7+fDXHlX/M9e02wYsvwuTJODP6Fh3C6AOH8tLB32HVftGDdi5UvkSjwC5ZJ1Yw9BsskzF4Fm0yTdOSw3hz/OHau2/tLg7btpHu61dy4Oa1HLh5HQdtXkv/LWW03V3jbTQN6NmTyjbdWHj4iXy2X28+7dKHT7v0YdO++4EZBfl5e6ePThvU7L9DEKYvLqufRRuyq3U+H/Y4iA97HMT1E87wBmVLS3nk+vsZ+uFb3PD649zw+uMs6nUIzxwxkucPGc62ffbda9+5UPkSjXLsklKJDizGymHHm+MOYqCz4T46FeTvFYxCmubu48rxf/klLFvGoln/Ztnc+fQsW8GALevo/cVGWtVN1dltrVhT2J1Pu/Thsy59WNOtmFHnn8DxZx8HhYVRjxfKtWfKIKKfks5VE86o/77/+BdwQI9tn3P2stc474N5HLx5DTvz8pl+6Ak88q2z+aRbP8B7k733e4OzMhXjN8euwC4pE8TAYqxgmIwB0WjirSlveB3PUDBqqFPVlxy8eQ1PHd8ZPvrIuy1bBmvX7tkoPx8GDmSudWVJx15e77trH1Z17sXO1m32Ok60tiYykJjMUtFYA8Wd2+Wz+NenRJ6o5RyHl3/Ghe/P5rwP51FQs5M3+w7mL0PP4Y3+R7HqrjMDaWeqafBUMk6iA4vTF5dF/GcP3Z/s2YRNg9mOXTVxTRQqq6zilqdLabd2FeeUf0DHtSs54IsyBmxew0Gb19Jte6W34RNAQQEMGgTHH+99PfRQ73bAAdC6NVeHeWMAr0ca7k0syPK+ZK+zEu35ys8zbj3rsOhvqmZ82OMgbulxEL8/7vtcXDqLy957nslP3cr7fQ+H0zt4f9ccpcAuKZNI0A39E0cSymEnczZhuGAWkXN0/2ozB3yxnn5b1tP/izL6bymj/xfrKa7cSJvaGk6p23Rbm3Z82rUPrx5Qwqr9+zL87OM45qzjoLg4asVJpHNtZUb/8S+EDdxBXZA72Wv1Rxsonnj+NxgzpIhhE+b5elOtLOjIH4/5Ln8Zeg6XLJ3LDQufghNOgJNPhgkT4KijEm5vplFgb+FSOfMykaAb7YLH4C0INXH28qQMiEb6uN+mppqibZvoU7mRPlvLvVvlRvpvWU//LespqN5Zv+3OvHxWdu7Ff7oWM2fA0azcrxe/u/F8XtrZgZveKmdLlTfQWViQz8HHHgb9Yj8H4c4VqK+QSeZqhcn+ZBTpeWyYNvJzrPw8o32b1mytqmb/Lh0ZfMd4Cg75HTz0ENx5J5SUwI9/DLffDvvtF0jbM4ECewuW6mVLk73Y1frKqsDSDdMXl3H3Sx+xe906iivLOWZrOX0qy+m9tS6IV5bT/asv6gcuwav7Luu4P2u7FNHmpJE8u709pft0Y+V+vdjQoSvO9vS+iwoLYPhwdi4u4+uajfX3V1ZV+34Omp5rqzBlj0HXbIeej0gjc0FVm/h5HqP16mudi/7c/+pXcNVVcOut8MADMHUq/P73cPnlYLb39llGg6ctWKoHGqFxoA7VX4cb6AttG+/AZFzt3rXLG5RcvRrWrPG+rl7NpqX/oeo/n9FzawVtamvqN6/F2NChK+sKu7O2U92tsDtb9i9iS/feLKltR8/O7aMOjobcV1eVEeRzEOl4BqxsUEESjp9Pbpk2ozOwweAlS+Caa+Bf/4IzzoBJk7zJVxlIg6cSUzqWLQ39w/n5pBAr/dLUjl01jWvFt22rD9YNA3f9zxs2eHXQDfXsyabWhazsfhAvDBzG2k496gP4+o7dqM7Lb7R5tEASqUfZuV1+zHRCc56D5qa6/H5yi/Z8RHpzbi4/bzSBDQYfeSS8/rrXcx8/Hg47DP74R7go0rWFMl8ggd3MfglcjZfq/AC4wjn3dRD7luRJ17KlfgfeYgW3gl1fU7x1I323bKC4cgO9t26i498r2Mo2OlVs8BaiaqhNG299kb59YdQo72txsfe1b1/o3RvatuWsKD3thmIFs0ipp1vPOqz+5yCfg+aOLyT6fESqwmmueFKEQQ0G06oV/OIXcOqp8IMfwMUXw6uvwv33e9VJWSbhwG5mRcAvgEOdc1VmNhW4EHgs0X1LcjU3EPj92B5pG7+91F6FBWwr/5yDPl9Lvy1eNUlx5Qb6Vm6kb+WGPaWBdba1aUdZp/35oEsPjr1kxN6Bu3t3X+uaRLuQA/j/uO+nRxnkYG9ze7DxPB+p6Agku+ImqoMPhjfegP/5H69iZsECeOop7/4sElQqpjVQYGbVQDtgfUD7lSRqTiDw05uKtU24ANF1+xaO+boc/rjam5CzbBmvvP8hBZ+X129Ti7GhY1fWFPZg7oFDvTVNCnuwurAnqzv3rJ867ienHE2kqwo5/KUc4qk0Cnrp2Ob0YP0G7HB/l/xWxo5dNRHLK5sjFSnCqM9R69Zexczw4XDZZd4CaFOnwimnRN9pBglk8NTMrgXuAKqAl51zl4TZZiwwFqC4uPibq1evTvi4knp+BvsibdO5XT7t8vOw1as4vHwFh5V/xmHln3F4+Wfsv33Lng07dKifkPNhx15MrixgUZtu7O7bl2MPL+KZRWVRc+9BDP42tww0mcv2Jks8bW66fML2XTVU73Yxfy8eyR7Uj+s5Wr0azjoLli6F++6Dn/0srVUzKVtSwMw6A88A3wMqgaeAp51zf4/0O6qKyV7RKj1CvdlfTimt36Zg19cM3vAJ3yz7iJJ1yxi8YTmFX38FQI214j9di1nVewC9RwzjiFHf8WZW9uoV9Z+nYWVN04sapzuIpqPSKAjNeSNL1rkm+80x7nZ/+SVceinMmAE//Sn84Q8JXwGquVJZFXMSsNI5V1F34GnAd4CIgV2yV7T8c1llFf875V1OL1vK4OULGbp2KYduWkF+rfcPurxrMS8OHOatztf9QJZ37UvXboVRg0CkgBPuSjnJnmDl51jZeoHk5qRwknWuyb6yUdzt7tABpk2DG26Au+/2Lgbyt795g/EZKojAvgY42sza4aViRgLqjueovfKszjGoYiUnrFjEsasWU7LuI9rurmFnXj6Lew3kz98+j4VFg3ivaFDY5VOjDVL6yecHVhURg99KjVy+QHJTyTzXZD6vzWp3Xp43gal7d7j+eq/iato0aN8+KW1MVMKB3Tm3wMyeBt4DaoDFwKRE9yuZacyQIti9m9kPP8tR773GqE/epnirN8C5rFs/Hj/qLP7Vfwjn/9dF3PX6mvoeV6udNRBmOdtw65SHRKuOCD2eqmVm/VZqZPMFkuP99JOt55pQu8eN85YeGDvWG0ydNcvr0WcYzTwVf5yDd9+FyZO98q9Nm9iVl89bfY9k9oBjmHvQUCr29dbaCJernL64rFHuvaFIuc1o+fywF4ZIYm49nlmdqUwPBaW5ee10nmsix0643U8/DRdeCMccAy+9BPvu/Wk0GbQeuwRjzRovnzh5MnzyiXdl+bPOgnPP5fmiwYx7eZXvYNBv/AthDxGpPDHSIFe4S8FBcgcos3VQ1K90n1+8gTYjqo+eesqbnXrssfDCCylJy/gN7Em4Cq1kPedgzhwYM8a7YPAtt0CPHvDww7Bxo1fTe+GFnDn8EO489wiKCgswvCAQ7R+rKEIOM1Juc9yogRTkN64+KMjPCxvUIbkDlJHakulpB7/SOegbCtJllVU49oxfTF9cFvF3bpuxNGqaLiUuuMDr9Lz5pve/snNn7N9JEa0Vk8UC/xi8fTs8+ig8+CAsX87Ozl148rgLeXjgCGr79efELt149aFFEStUYok3txmpOiLSao/JHKBMdqVGuqVz0DfemabhroUakvLqo4su8haT+8EPvNsTT/ia3ZxsCuxZKtAldysrvQWQ7rsPNm+Gb3+bhb+9nyu392Obq+ulVlbx9/lr6n+lOcdrTnCM9MaRjkG7VFXgBCmdF/b2K95PC9F65WmpPrr8cigv98ohe/XySiLTTIE9SwWynsbmzd6L8MEHvZUQzzgDbrwRhg3j2gnz2Oai936as35HUMFxn/xW9edfWJDPbaMPy7qgm2zxLqYF6flEEu+nhWi98rSlxsaNg7IyuOceKCry1ntPIwX2LBXpxV1WWRV73Y4dO7xV6yZM8GbVXXCBF9AHD465f7/tSJZwg2Y7a2rDbperaRO/4n3zT9cnkng/LfhZDjnlzODee72loK+7DgYM8IoM0iT9ySAJa/riMoZNmEf/8S8wbMK8vQaSon3kjDgAtXs3PPKI96K76Sbvuo8ffghTpjQK6rH23/RY4dqXLLFq26F5g3HZLtzrJVtmwY4ZUhTXIHykgeyGyyGnRatW8Pjj3jVUL77YW18mXU1J25ElIj+BKdyLu6lGAe/dd+Hoo+Hqq70lbN98E557zlubJQw/+w9JZeD0E6z8BP9cEun1UtguP+z2mTgLdsyQIt4aP4KVE87grfEjYo67xPNGkFIFBTB9ulfXPnq0l+5MA6ViMpCfj9BNc6KRZiNs37DJu1jvpEleyeI//uFNrIixQl24nOuJh3Tj1Y8rwn4MTtV62X7ysdnSU20okdRRpNdL29atwk7kyoUSzYweyO7dG559Fo4/Hr73PW92auvUhloF9gwULX/eUMMXd7gJJqd88jYT5jxIbdWXTP3Oudzxre/RcXUXxpWu9/VPEemfJ9IszFQETj/52GxbryXRCqdIf/etVdXc+73BLX6sIS2OPhr+9Ce48kr4zW+8WwopsGegSIEp2roqDQNep6ovue2VP3POR6+xvv8hnDfydhZ36QfAl4mURcZoXyoCZ6qvSpQKiVY4RXs+Mrpnm+uuuMJLed5+uzc7NYUX6lCOPQONGzWQcIkSR+Qa3lDe8ZxNS5nzyE856+M3WfajX3HhVffXB/WQRPPN6Z6FGSsfm9E52DASTR2l+/mQKB54wBvHuvRSWJ+6C8upx56Bxgwp4r+mlIZ9LOI/e00NY556kDF/vdO7yvrf5jJoyBDWRlifJZG0STbMwsymnmqin4Cy4flosdq189aUKSnxZqnOm5eSi3QosGeoIh//7KEBN7dmDQ+9eDffWP2hV/Vy//3Qrh3TF5fRKsKCWYmmTbIpcGa6IFJHej4y2KBB8NBD3gzVCRPg5puTfkilYjJUrI/XoQG3PksW8Pxj13Lghs+4bsz1TP/pbfVB/cZpH4QN6sn6mB6r9l7Cy7bUkTTD97/v9dhvvRXeeSfph1OPPUPF+ng9cfZyzn1nJre98mdWF/Zk7Lm3sKJLb96uG3ALNyAH3pK3yQgaga5d0wKpx53jzOCPf4Svv4bOnZN/uCDWYzezQuBh4HC8Mb4rnXNvR9q+Ja/HHq5eGaLnR5v+zvUjDmDbj3/G9xe/wLwDSrh29Di+bOutBR1a2zyeC0MEId3reYu0BKm8mDXA/cAs59z5ZtYGaBfQfnNKuF7tuKfeB4Pq3a7+voY93aa/U1m+mS4XjePsFe/x56Hnctfxl1Pbak/KJpQ7T3VJYjZOChLJVQnn2M2sI3Ac8AiAc26Xc64y0f3monDpkepaVx/UQ5pe1zP0O/vt2Mo/nryZo1eWcsNpv+DOE69sFNQb5s5TXQIX6Q0jUycFieSyIAZPDwAqgL+a2WIze9jMMvPS3WkWT+81tG3oa++t5Tz1xPUM/Hw1Pzr3ZqYc2XiyQ2FBfqPceaoH5FRLLZI5gkjFtAaOAn7unFtgZvcD44H/abiRmY0FxgIUFxcHcNjsEyk9Emnb0NeCT5fz9ym3UFC9k0u+dzuLeu+9cFf7tq3DTtRJ1YCcaqlFMkcQgX0dsM45t6Du56fxAnsjzrlJwCTwBk8DOG7WCVevnN/KGuXYoXFP9zcDjG/cfhO1GBdcchefdOsXdt9NPw2kYz3yVL2RaK11kegSDuzOuY1mttbMBjrnlgMjgY8Sb1ruidSrDXffmCFF8NFHjLzmIr5s05pzzr+DT/eLHLyaTlzK1dLDXD43kaAEVe44GK/csQ2wArjCObcl0vYtudzRt2XL4MQT+Xq349zzb+ejTr0iblqQn9cof57LpYe5fG4isaS03NE5VwrEPJj49NlnMGIEmHHF9yfwUZtue22SZ0atc2FTEblcepjL5yYSFM08zTTl5TBqFOzaxdxJT/P2uzvCblbrXMSJRtm2Hnk8cvncRIKitWLSIOKaKtu2wWmnwYYNvH7vY/ysdGfEfUQLZLlcepjL5yYSFPXYUyzS4F+rXTsZfdMPYckSmDmTm95vS1V1+PRCrECWy6WHuXxuIkFRYE+xsFfL2VXDPj/6Ibw/DyZPhtNOY/3r4ddRB3xNNMrlRaVy+dxEgqBUTIqFG+S75u2pnPL+PPi///OW9yRyqqWo7nJnIiKRKLCnQMOceitrfNG7UZ/8m3Fv/o1Zg0+C8XvmdSmXLCLNpVRMkjXNqTe88MWh5Su49/m7eb9oILse+pO3ZnMd5ZJFpLkU2JMs0gUv9t+xlb9M+y1fFXRg/aP/YPTRB+61jXLJItIcCuxJFi6n3qp2N/fMmEjRri/hX//itG8elYaWiUiuUmBPsnATan7+7ykcu7oUHn4YvvnN+vsbLm5V2C4f52BrVbXSMCISFw2eJlnTQdBhq0q59q1/subM8+HKK+vvD+XiyyqrcMCWHdVUVlXj2FPrrotDi4gfCuxJ1vCCF92/3MwDz/+erw4YQPGTjzUaLI2Uiw9peFUlEZFolIpJkJ+1wccMKWLMkT28hb3cLpj5LLRvfJEpP4tYaaErEfFDgT0Bca0NPnEivPEGPPYYHLr3FZD8XF1JC12JiB9KxSQg7PIA1bv535lLG29YWgq//jVccAFcdlnYfYWbkBRuGxGRWBTYExApNbJlR/Wegc6vv4ZLL4WuXXnhJ79m2F2v7r2qI3ty8XlNZqaGFBbkqypGRHwJLLCbWZ6ZLTaz54PaZ6aLlhqpH+i85RZYupR/3/Q7rpu3rr7qJVyly5ghRdz93W+EXUrgttGHJeMURCQHBdljvxZYFuD+Ml601Mj6yip47TW45x74yU8Y91WvsGmbppUuDatoDG/RLz+rOYqIhAQyeGpmvYEzgDuAXwWxz2wwZkgRt81YSmVV9V6P9W9nXp36gQfCxIms/+1rYfcRLp2jpQREJBFB9djvA64HagPaX9a4bfRhYVMnf/50Bqxc6c0ubd8+YtrGwV75dhGRRCQc2M3sTGCTc25RjO3GmtlCM1tYUVGR6GEzRrjUyR8HwYAn/gJXXw3HHw9Er3rRzFIRCZK5BsvINmsHZncC3wdqgH2AjsA059ylkX6npKTELVy4MKHjZqyaGhg6FNavh2XLoHPn+odCk5ki1asXFRbw1vgRqWqpiGQZM1vknCuJtV3CPXbn3I3Oud7OuX7AhcC8aEE95913HyxeDP/v/zUK6uD17t8aP4LwBY2aWSoiwVAde5BWrvQmIo0eDeefH3GzSPl2zSwVkSAEGtidc685584Mcp9Z5Ze/pAbjnEEX0f/GFyMOiuqydyKSTForJiizZ8Nzz3H/iVewmA5A5LVjdNk7EUmmhAdPmyPnBk937YIjj2RdxZeMuOJBdrXOb/SwBkVFJAh+B0/VY4/Bz7K8PPAALF/Oref9eq+gDhoUFZHUUmCPwteyvBs3wm23wemn83HJ8RAmiGtQVERSSVUxUURalrfR+i433eSt4HjvvRoUFZGMoB57FJFSKPX3l5Z6F8647jo4+GDG1D2uQVERSScF9igiXdWoPrUyfrw3Cemmm+of0wJeIpJuSsVEETW1MneuV+J4881QWJimFoqI7E099igi1pt/oycMPRv69oVrrklzK0VEGlNgjyFsamXKFFi0CCZPhrZt09MwEZEIFNh9CtWzV2zexquP/pJ9Bwyi08UXp7tZIiJ7UY7dh1A9e1llFReWzqLoiw1cV3Ix05dsTHfTRET2osAew/TFZfz31Pepqt5N2+qdXDP/Keb3OZw5fQbvdb1SEZFMoMAeRainvrtuPZ2L3p9N96++4N5jLwEzLRUgIhlJgT2KhjNP21bv5CcLnubfxUeyoPgIQEsFiEhmUmCPomGPPNRbv3/YRYCWChCRzBXExaz7mNmrZrbMzJaa2bVBNCwThHrkod7628VHsKD4CPLMuPPcIzTDVEQyUhA99hrgv51zg4CjgWvM7NAA9pt240YNJL+VNeqt57cy7v7uNxTURSRjBXEx6w3Ouffqvv8SWAbkTNRrW7Ontz6/+EgiXolaRCRDBJpjN7MEWhNeAAAMfklEQVR+wBBgQZD7TZeJs5dzTunLjXLr1budyhxFJKMFFtjNbF/gGeC/nHPbwjw+1swWmtnCioqKoA6bVOVffMUP33mWRb0OYX6fI+rvV5mjiGSyQAK7meXjBfUnnHPTwm3jnJvknCtxzpV069YtiMMm3cVr36F4azl//vZ5YHtyMCpzFJFMFkRVjAGPAMucc/ck3qQM4Ry/Kn2OFV16M2fAt+vvVpmjiGS6IHrsw4DvAyPMrLTudnoA+02vuXMp/PhDtl5zLb06t8eAosIClTmKSMZLeHVH59y/yMVakbvugh49GHLTz3mrydK8oZUedfk7EclEWrY3nPfeg1degQkT9lpvPbR+TGipgbLKKm6c9gGAgruIZAQtKRDOxInQoQP8+Md7P9Rg/ZiQqurdKoEUkYyhwN7E7BffoWbqVCYNOplhDy1i+uKyRo9HKnVUCaSIZAoF9gamLy5jzR2/xxw8ftSZ9WmWhsE9UqmjSiBFJFMosDdw//NLOP+9Wbw84GjKOu0P7J1mGTdqIAX5eY1+TyWQIpJJNHjawNB/z6Lz11/y2DfPanR/wzRLaIBUVTEikqkU2EOc4+rS51nWrR8L+hze6KGmaZYxQ4oUyEUkYykVE/LGGwzYuILJ3xrdaPmA/FamNIuIZBUF9pA//IFdnQqZedgJje/PvalXIpLjFNgBVq+G6dOZMvhUvmrVptFDWqZXRLKNAjvAQw8B8KdDTwn7sGrURSSbKLDv2gWPPgqjR0Nx37CbqEZdRLKJAvtzz0FFBfzoR6pRF5GcoHLHSZOgb184+WTG5HlBXTXqIpLNWnZg/+wzbxXH3/4W6oK6atRFJNu17FTMX/7iBfQrrkh3S0REAtNyA/uuXfDXv8KZZ0KReugikjuCupj1qWa23Mw+NbPxQewz6WbMgE2bYOzYdLdERCRQQVzMOg94EDgNOBS4yMwOTXS/STdpEvTpA6NGpbslIiKBCqLHPhT41Dm3wjm3C3gSODuA/SbPihUwZw5cfXX9oKmISK4IIrAXAWsb/Lyu7r5GzGysmS00s4UVFRUBHDYBjz/uLfSlQVMRyUFBBPZwy2S5ve5wbpJzrsQ5V9KtW7cADttMtbUweTKMHOmlYkREckwQgX0d0DBC9gbWB7Df5HjzTVi1Ci6/PN0tERFJiiAC+7vAADPrb2ZtgAuBGQHsNzkefxz23RfOOSfdLRERSYqEZ54652rM7GfAbCAPeNQ5tzThliXD9u3w1FNwwQXQvn26WyMikhSBLCngnHsReDGIfSXVs8/CV18pDSMiOa1lzTx9/HHo1w+GD093S0REkqblBPa1a2HuXLjsMmjVck5bRFqelhPhnngCnPMCu4hIDmsZgd05Lw0zfDgceGC6WyMiklQtI7CXlsLHH8Oll6a7JSIiSdcyAvuTT0Lr1nDeeeluiYhI0uV+YK+t9QL7qFHQpUu6WyMiknS5H9jnz4c1a+DCC9PdEhGRlMj9wP7Pf8I++8DZmb2SsIhIUHI7sNfUwNSp3uXvOnRId2tERFIitwP7a695l79TGkZEWpDcDuxPPun11E8/Pd0tERFJmdwN7Dt3wjPPwJgxUFCQ7taIiKRM7gb2l1+Gykq46KJ0t0REJKVyN7BPmQL77QcnnZTuloiIpFRuBvadO2HmTC8Nk5+f7taIiKRUQoHdzCaa2cdmtsTMnjWzwqAalpC5c2HbNi0hICItUqI99jnA4c65I4FPgBsTb1IApk2Djh1h5Mh0t0REJOUSCuzOuZedczV1P84HeifepATV1MD06d6kpLZt090aEZGUCzLHfiXwUoD7a5433oDNm5WGEZEWK+bFrM3sFaBHmIduds49V7fNzUAN8ESU/YwFxgIUFxc3q7G+TJvm1a2PGpW8Y4iIZLCYgd05F7Ve0MwuB84ERjrnXJT9TAImAZSUlETcLiG1tV5gP+00aN8+KYcQEcl0MQN7NGZ2KnADcLxzbkcwTUrA/PmwYYPSMCLSoiWaY38A6ADMMbNSM/tTAG1qvmnTvLr1M85IazNERNIpoR67c+6goBqSMOe8tWFOPhk6dUp3a0RE0iZ3Zp6WlsKqVUrDiEiLlzuBfcYMMIOzzkp3S0RE0iq3Avsxx0C3buluiYhIWuVGYC8rg/feU29dRIRcCezPP+99HT06ve0QEckAuRHYZ86EAw6AQYPS3RIRkbTL/sC+fTu88oqXhjFLd2tERNIu+wP7K694F9ZQGkZEBMiFwD5zpjchafjwdLdERCQjZHdgr631Bk5PPVWXwBMRqZPdgf3dd6G8XGWOIiINZHdgnzkT8vK8ZXpFRATIhcB+7LGw337pbomISMbI3sC+bh0sWaIlekVEmsjewD57tvdVaRgRkUayN7DPmgVFRXDYYeluiYhIRgkksJvZdWbmzKxrEPuLqaYG5szxeuuabSoi0kjCgd3M+gAnA2sSb45P8+fD1q1e/bqIiDQSRI/9XuB6wAWwL39eeskrcxw5MmWHFBHJFgkFdjMbDZQ5594PqD3+zJoF3/kOFBam9LAiItkg5sWszewVoEeYh24GbgJO8XMgMxsLjAUoLi6Oo4lNlJd7F9W4447m70NEJIfFDOzOuZPC3W9mRwD9gffNG8DsDbxnZkOdcxvD7GcSMAmgpKSk+WmbUJmj8usiImHFDOyROOc+APYP/Wxmq4AS59znAbQrslmzYP/9YfDgpB5GRCRbZVcd++7d8PLLXm+9VXY1XUQkVZrdY2/KOdcvqH1FtHAhbN6sNIyISBTZ1e2dNcubkHTyyeluiYhIxsquwN67N1xxBXRNzQRXEZFslF2B/aqr4JFH0t0KEZGMll2BXUREYlJgFxHJMQrsIiI5RoFdRCTHKLCLiOQYBXYRkRyjwC4ikmMU2EVEcow5l7oLH9Uf1KwCWJ3yA8enK5DclSpTI1fOA3QumShXzgOy41z6Oue6xdooLYE9G5jZQudcSbrbkahcOQ/QuWSiXDkPyK1zUSpGRCTHKLCLiOQYBfbIJqW7AQHJlfMAnUsmypXzgBw6F+XYRURyjHrsIiI5pkUHdjM71cyWm9mnZjY+zONtzWxK3eMLzKxf6lvpj49z+ZWZfWRmS8xsrpn1TUc7/Yh1Lg22O9/MnJllZCWDn/Mws+/WPS9LzewfqW6jXz5eX8Vm9qqZLa57jZ2ejnbGYmaPmtkmM/swwuNmZn+oO88lZnZUqtsYCOdci7wBecBnwAFAG+B94NAm2/wU+FPd9xcCU9Ld7gTO5USgXd33P8nmc6nbrgPwBjAfKEl3u5v5nAwAFgOd637eP93tTuBcJgE/qfv+UGBVutsd4VyOA44CPozw+OnAS4ABRwML0t3m5txaco99KPCpc26Fc24X8CRwdpNtzgYer/v+aWCkmVkK2+hXzHNxzr3qnNtR9+N8oHeK2+iXn+cF4LfA74CvU9m4OPg5jx8CDzrntgA45zaluI1++TkXB3Ss+74TsD6F7fPNOfcG8EWUTc4GJjvPfKDQzHqmpnXBacmBvQhY2+DndXX3hd3GOVcDbAW6pKR18fFzLg1dhdcryUQxz8XMhgB9nHPPp7JhcfLznBwMHGxmb5nZfDM7NWWti4+fc7kNuNTM1gEvAj9PTdMCF+//UkZqne4GpFG4nnfTEiE/22QC3+00s0uBEuD4pLao+aKei5m1Au4FfpCqBjWTn+ekNV465gS8T1BvmtnhzrnKJLctXn7O5SLgMefc3WZ2DPC3unOpTX7zApUt//NRteQe+zqgT4Ofe7P3x8f6bcysNd5HzGgf49LFz7lgZicBNwOjnXM7U9S2eMU6lw7A4cBrZrYKLw86IwMHUP2+vp5zzlU751YCy/ECfabxcy5XAVMBnHNvA/vgrb2SbXz9L2W6lhzY3wUGmFl/M2uDNzg6o8k2M4DL674/H5jn6kZYMkzMc6lLX/wZL6hnai4XYpyLc26rc66rc66fc64f3njBaOfcwvQ0NyI/r6/peIPamFlXvNTMipS20h8/57IGGAlgZoPwAntFSlsZjBnAZXXVMUcDW51zG9LdqLile/Q2nTe8EfBP8Eb8b6677zd4gQK8F+dTwKfAO8AB6W5zAufyClAOlNbdZqS7zc09lybbvkYGVsX4fE4MuAf4CPgAuDDdbU7gXA4F3sKrmCkFTkl3myOcxz+BDUA1Xu/8KuDHwI8bPCcP1p3nB5n62op108xTEZEc05JTMSIiOUmBXUQkxyiwi4jkGAV2EZEco8AuIpJjFNhFRHKMAruISI5RYBcRyTH/HxY877DR7oizAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c60f0fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = PolynomialRegression(4).fit(X, y)\n",
    "plt.scatter(X, y)\n",
    "plt.plot(X_test, model.predict(X_test),c='r');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detecting Data Sufficiency with Learning Curves\n",
    "\n",
    "As you might guess, the exact turning-point of the tradeoff between bias and variance is highly dependent on the number of training points used.  Here we'll illustrate the use of *learning curves*, which display this property.\n",
    "\n",
    "The idea is to plot the mean-squared-error for the training and test set as a function of *Number of Training Points*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/al/anaconda3/lib/python3.6/site-packages/sklearn/learning_curve.py:22: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the functions are moved. This module will be removed in 0.20\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.learning_curve import learning_curve\n",
    "\n",
    "def plot_learning_curve(degree=3):\n",
    "    train_sizes = np.linspace(0.05, 1, 20)\n",
    "    N_train, val_train, val_test = learning_curve(PolynomialRegression(degree),\n",
    "                                                  X, y, train_sizes, cv=5,\n",
    "                                                  scoring=rms_error)\n",
    "    plot_with_err(N_train, val_train, label='training scores')\n",
    "    plot_with_err(N_train, val_test, label='validation scores')\n",
    "    plt.xlabel('Training Set Size'); plt.ylabel('rms error')\n",
    "    plt.ylim(0, 3)\n",
    "    plt.xlim(5, 80)\n",
    "    plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the learning curves look like for a linear model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcXFWZ+P/Pc6uqu3pPZw8JkCAJS/YQFmVfZMBREQQJ6kgYlX2U73dkhpn5DggyP535OupXRZig4MawL6KyCcIgikgSQgybSUgCSSB7b+mtqu7z++Pcqq6uvr2mqqu687xfr8pd6ta9pzvV97nnnHueK6qKMcYYk8srdgGMMcaUJgsQxhhjQlmAMMYYE8oChDHGmFAWIIwxxoSyAGGMMSZUwQKEiMRF5E8i8qqIvCYiN4ZsUy4i94rIOhF5SUSmF6o8xhhjBqeQNYgO4DRVnQ8sAM4SkeNytvk8sEdVDwW+Dfx7ActjjDFmEAoWINRpCRZjwSt3VN45wE+C+QeA00VEClUmY4wxAxct5M5FJAKsAA4FblHVl3I2mQq8C6CqSRFpBMYBO3P2cylwKUBVVdVRhx9+eCGLbYwxo86KFSt2quqEwXymoAFCVVPAAhEZAzwsInNUdU3WJmG1hR65P1R1GbAMYPH8I3X5L++AaDlEyoJpOUTLuqbGGGO6EZFNg/1MQQNEmqo2iMhzwFlAdoDYDBwIbBaRKFAH7O5/hylItLpXD9JH8CgHa8EyxpgBKViAEJEJQCIIDhXAGfTshH4UuBh4ETgf+K3uc/ZAhWS7e3WEvB0JgsWYg6y2YYwxfShkDWIK8JOgH8ID7lPVX4nITcByVX0U+BHwMxFZh6s5LClgeZxUp3u17YGaSQU/nDHGjFQFCxCquhpYGLL++qz5duCCQpWhTxYgzH4qkUiwefNm2tvbi10UUwDxeJxp06YRi8X2eV/D0gdRkpJtkGiDWEWxS2LMsNq8eTM1NTVMnz4du6t8dFFVdu3axebNm5kxY8Y+72//TrXRtqfYJTBm2LW3tzNu3DgLDqOQiDBu3Li81Q4tQBizH7LgMHrl8/92/w4QqU7oaOl/O2OM2Q/t3wECoK3/YRfGmPxpaGjgBz/4wZA++5GPfISGhoY+t7n++ut5+umnh7R/050FiLYG2NehF8aYAesrQKRSqT4/+9hjjzFmzJg+t7nppps444wzhly+QunvZytFFiA0Be2NxS6FMfuN6667jvXr17NgwQKuvfZannvuOU499VQ+/elPM3fuXAA+8YlPcNRRRzF79myWLVuW+ez06dPZuXMnGzdu5IgjjuCLX/wis2fP5swzz6StrQ2ApUuX8sADD2S2v+GGG1i0aBFz587lzTffBGDHjh18+MMfZtGiRVx22WUcfPDB7NzZLQUcqVSKpUuXMmfOHObOncu3v/1tANatW8cZZ5zB/PnzWbRoEevXr0dVufbaazPb3nvvvQChP9vPf/5zjjnmGBYsWMBll11GKpXq9VjFtv/e5pqtbQ9U9H1VYsxodOMvX+P1rU153eeRB9Ryw8dm9/r+N77xDdasWcOqVasAdxL905/+xJo1azK3Zt5xxx2MHTuWtrY2jj76aD75yU8ybty4bvtZu3Ytd999N7fffjuf+tSnePDBB/nsZz/b43jjx49n5cqV/OAHP+Cb3/wmP/zhD7nxxhs57bTT+Kd/+ieeeOKJbkEobdWqVWzZsoU1a1x2oHTT1mc+8xmuu+46zj33XNrb2/F9n4ceeohVq1bx6quvsnPnTo4++mhOOukkgG4/2xtvvMG9997L73//e2KxGFdeeSV33XUXs2fPDj1WsVkNAqCjCfyRV/0zZrQ45phjut23/93vfpf58+dz3HHH8e6777J27doen5kxYwYLFiwA4KijjmLjxo2h+z7vvPN6bPPCCy+wZIlL3HDWWWdRX1/f43OHHHIIb7/9Nn/3d3/HE088QW1tLc3NzWzZsoVzzz0XcIPSKisreeGFF7jooouIRCJMmjSJk08+mZdffrnHz/bMM8+wYsUKjj76aBYsWMAzzzzD22+/HXqsUmA1CAD1XTNT5dhil8SYYdXXlf5wqqqqysw/99xzPP3007z44otUVlZyyimnhN7XX15enpmPRCKZJqbetotEIiSTScANKOtPfX09r776Kk8++SS33HIL9913H9/5zndCt+1rf9k/m6py8cUX8/Wvf73HdrnHuuOOO/otY6FZDSLNxkQYMyxqampobm7u9f3Gxkbq6+uprKzkzTff5I9//GPey3DCCSdw3333AfDUU0+xZ0/Pv/+dO3fi+z6f/OQn+drXvsbKlSupra1l2rRpPPLIIwB0dHTQ2trKSSedxL333ksqlWLHjh08//zzHHPMMT32efrpp/PAAw+wfft2AHbv3s2mTZtCj1UKrAaR1tEEqQRE9j1/iTGmd+PGjeP4449nzpw5nH322fz1X/91t/fPOussbrvtNubNm8dhhx3GccflPql4391www1cdNFF3HvvvZx88slMmTKFmpqabtts2bKFSy65BN/3ATJX/T/72c+47LLLuP7664nFYtx///2ce+65vPjii8yfPx8R4T/+4z+YPHlyplM87cgjj+Tmm2/mzDPPxPd9YrEYt9xyCxUVFaHHKjbZ5+zaw2zx/CN1+eN3FWbntVOhemJh9m1MiXjjjTc44ogjil2Mouro6CASiRCNRnnxxRe54oorMp3mo0HY/7GIrFDVxYPZj9UgsrXtsQBhzH7gnXfe4VOf+hS+71NWVsbtt99e7CKVJAsQ2RKtkGiHWLzYJTHGFNDMmTN55ZVXil2Mkmed1Lmss9oYYwALED1ZgDDGGMACRE+pDujcW+xSGGNM0VmACGO1CGOMsQARqm2PZXg1poRUV1cDsHXrVs4///zQbU455RSWL1/e536+853v0NramlkeSPrw/ZkFiDB+0g2cM8aUlAMOOCCTqXUocgPEQNKHF0OppAa3ANEba2YypiD+8R//sdvzIL761a/yn//5n7S0tHD66adnUnP/4he/6PHZjRs3MmfOHADa2tpYsmQJ8+bN48ILL+yWi+mKK65g8eLFzJ49mxtuuAFwCQC3bt3Kqaeeyqmnngp0pQ8H+Na3vsWcOXOYM2dOJudSX2nFs91///3MmTOH+fPnZ7K4plIpvvKVrzB37lzmzZvH9773PcAl7Fu4cCFz587lb//2b+no6MiU5aabbuKEE07g/vvvZ/369Zx11lkcddRRnHjiiZlR2WHHKhQbB9Gb9kbwffAshppR7PHr4P0/53efk+fC2d/o9e0lS5ZwzTXXcOWVVwJw33338cQTTxCPx3n44Yepra1l586dHHfccXz84x/v9RnLt956K5WVlaxevZrVq1ezaNGizHv/9m//xtixY0mlUpx++umsXr2aL33pS3zrW9/i2WefZfz48d32tWLFCu68805eeuklVJVjjz2Wk08+mfr6+gGlFb/pppt48sknmTp1aqbJatmyZWzYsIFXXnmFaDTK7t27aW9vZ+nSpTzzzDPMmjWLz33uc9x6661cc801gMsO+8ILLwAub9Ntt93GzJkzeemll7jyyiv57W9/G3qsQrGzX2/Uh3ZrmzQm3xYuXMj27dvZunUrr776KvX19Rx00EGoKv/8z//MvHnzOOOMM9iyZQvbtm3rdT/PP/985kQ9b9485s2bl3nvvvvuY9GiRSxcuJDXXnuN119/vc8yvfDCC5x77rlUVVVRXV3Neeedx+9+9ztgYGnFjz/+eJYuXcrtt9+eaR56+umnufzyy4lG3XX42LFjeeutt5gxYwazZs0C4OKLL+b555/P7OfCCy8EoKWlhT/84Q9ccMEFmQcLvffee70eq1CsBtGXtj2WAtyMbn1c6RfS+eefzwMPPMD777+feS7DXXfdxY4dO1ixYgWxWIzp06eHpvnOFla72LBhA9/85jd5+eWXqa+vZ+nSpf3up6+cdANJK37bbbfx0ksv8etf/5oFCxawatUqVLVH+frLfZdODe77PmPGjAnNDxV2rNyHKeWL1SD60tEMqWSxS2HMqLNkyRLuueceHnjggcxdSY2NjUycOJFYLMazzz7Lpk2b+tzHSSedxF13ucSda9asYfXq1QA0NTVRVVVFXV0d27Zt4/HHH898prdU4yeddBKPPPIIra2t7N27l4cffpgTTzxxwD/P+vXrOfbYY7npppsYP3487777LmeeeSa33XZb5hkUu3fv5vDDD2fjxo2sW7cOcJlhTz755B77q62tZcaMGdx///2ACyyvvvpqr8cqFKtB9EldM1PV+P43NcYM2OzZs2lubmbq1KlMmTIFcI/y/NjHPsbixYtZsGABhx9+eJ/7uOKKK7jkkkuYN28eCxYsyDx/Yf78+SxcuJDZs2dzyCGHcPzxx2c+c+mll3L22WczZcoUnn322cz6RYsWsXTp0sw+vvCFL7Bw4cJen1KX69prr2Xt2rWoKqeffjrz589nzpw5/OUvf2HevHnEYjG++MUvcvXVV3PnnXdywQUXkEwmOfroo7n88stD93nXXXdxxRVXcPPNN5NIJFiyZAnz588PPVahFCzdt4gcCPwUmAz4wDJV/X8525wC/ALYEKx6SFVv6mu/BU33HSZWBRNmDd/xjCkwS/c9+o2EdN9J4O9VdaWI1AArROQ3qprbW/Q7Vf1oAcuxbxJ7IdkB0fL+tzXGmFGkYH0Qqvqeqq4M5puBN4CphTpeQdmYCGPMfmhYOqlFZDqwEHgp5O0PisirIvK4iJTGE9RzWYAwo8xIe5KkGbh8/t8WPECISDXwIHCNqubmr1gJHKyq84HvAY/0so9LRWS5iCzfsasIJ+tkO3S29r+dMSNAPB5n165dFiRGIVVl165dxOP5eehZQZ9JLSIx4FfAk6r6rQFsvxFYrKo7e9tm2Dup06omQt3IbCEzJlsikWDz5s39jg0wI1M8HmfatGnEYrFu60uqk1rcCJEfAW/0FhxEZDKwTVVVRI7B1Wh29bnjlu3Qsg2qJ+W7yH1r22MBwowKsViMGTNmFLsYZgQo5F1MxwN/A/xZRNLDAf8ZOAhAVW8DzgeuEJEk0AYs0f6qNO0NcM9nYNZfwYLPQO0BBfsBuvETbuBcec3wHM8YY4qsoE1MhbB4zixdfv0H4a1fg5+CQz8MCz8DYw4q/MErxkL9wYU/jjHG5FlJNTEVTCQKJ1wDi/4GXr0X3ngU1j4Fh5zi1o09pHDHtgyvxpj9yMgLEGmV4+CDV8KCT8Of74PXHoa3n4XpJ8Kiz8H4mfk/pqagoxEq6vO/b2OMKTEjN0CkVYyBYy6FeUtgzYOw5gHY+Ds46DgXKCYemd/jte2xAGGM2S+MnraSeC0svgQ+fS8s/jxsex0euRJ+/RV4b3X+jtPe5Po+jDFmlBs9ASKtrNr1RXz6Hjj2cti9Hn75Jfjll2HLCtjnTnmFNnuQkDFm9Bt9ASItVgnzl8BFd8MHr4bGLfDrv4dHr4Z3/rhvgcJSbxhj9gOjN0CkReMw93xYchec8L9g70544jp45sah77OzGZKd+SujMcaUoNEfINKi5XDkOXDhz2HehfD2c/v2sHarRRhjRrn9J0CkRWJw1FKIj4EVPxn6fixAGGNGuf0vQADEKlz/xJblsO21oe0j2QaJng8vN8aY0WL/DBDgmpvidbDix0Pfh9UijDGj2P4bIGIVri9i88uwPfcpqANkAcIYM4rtvwECYPYnoLx26H0RqU7oaMlvmYwxpkTs3wEiVulqEe++BNvfGNo+rBZhjBml9u8AATD7XFeLWDnEWkR7Qx5GZxtjTOmxAFFWCXMvcKOrd7w5+M/7SZcG3BhjRhkLEABzznNPilvx06F93pqZjDGjkAUIgLKqoBbxB9j5l8F/vsMyvBpjRh8LEGlzznOZYIdyR5P61sxkjBl1LECklVW7pH6bfg871w7+89bMZIwZZSxAZJvzSdfctHIIfREdzZBK5L9MxhhTJBYgspXXwJzz3SNLd60f5IftQULGmNHFAkSuuedDrGpo4yLadue/PMYYUyQWIHKV17gO6w3Pw+63B/fZRCsk2gtTLmOMGWbRYhegJM29ANY86Poizvjq4D67ay1IVtztMco6ZNR16EjsnHXiwfhZ7sFHxhgzDKwGESZeC7PPg7f/B3ZvGNxn/aRL4pd++YmcV7LnS1MhL7/7y0+6svh+YX5mY4zJYQGiN/MugFh8aHc0FUqyDRo2FrsUxpj9hAWI3sTrXCK/t5+DPRuLXZou7Y3Q/H6xS2GM2Q8ULECIyIEi8qyIvCEir4nIl0O2ERH5roisE5HVIrKoUOUZknmfcm3+K39W7JJ01/yejdw2xhRcIWsQSeDvVfUI4DjgKhE5Mmebs4GZwetS4NYClmfw4mNcLWL9b6FhU7FL092eTXbHlDGmoAoWIFT1PVVdGcw3A28AU3M2Owf4qTp/BMaIyJRClWlI5l1YmrUITcGeDZYk0BhTMMPSByEi04GFwEs5b00F3s1a3kzPIIKIXCoiy0Vk+Y5dw5zzqGIMHHlOUIt4t//th1OyvbT6R4wxo0rBA4SIVAMPAteoalPu2yEf6TEoQFWXqepiVV08YVx9IYrZt3kXgheDV0qsFgEu1XjTe8UuhTFmFCpogBCRGC443KWqD4Vsshk4MGt5GrC1kGUaksqxrhax7mlo3Fzs0vTU8r5lkzXG5F0h72IS4EfAG6r6rV42exT4XHA303FAo6qW5uXw/AvBi5ZmLQKg4R1ItBW7FMaYUaSQNYjjgb8BThORVcHrIyJyuYhcHmzzGPA2sA64HbiygOXZN5Xj4IiPw9rfQNOWYpemJ/WDkdbWaW2MyY+C5WJS1RcI72PI3kaBqwpVhrxbcBG88Sis/Dmc8o/FLk1PqQ7XaT32EJA+f/XGGNOvkTeS2itifsHKcXDEx2Dtk9BUel0lQNBpXaJlM8aMKCMvQETKoHYa/VROCmf+ReBF4JWfF+f4A7F3O7TasymMMftm5AUIgOoJQTNKZPiPXTUeDv8o/OVJl/KiVDW+C52txS6FMWYEG5kBAlxK7vGzIFKE5yPMv8g9n+GVu4b/2AOlvhtpnUoWuyTGmBFq5AYIcOm4x8+CsurhPW71RDj8r+Gtx0s7s2qq03Vahz6QyBhj+jayAwRAJArjDoWKscN73AWfdncKrSrhWgRAZ3Np3pZrjCl5Iz9AgDtR1x8MtT3SOBVOdi2iZdvwHXco9u6wTmtjzKCNjgCRVj0R6md0fyZ0IS34tJuu+u/hOd6+aHgHOvcWuxTGmBGkzzOpiERE5P8OV2HyomJM0HldVvhjVU+Cw86GNx+Dlu2FP94+UTfSOpUodkGMMSNEnwFCVVPAUUFepZEjVuGCRKyq8Mda8BlAR0Ytwk9Yp7UxZsAG0hbzCvALEfkbETkv/Sp0wfZZJBZ0Xhc4PXjNZJh1Frz5a9fWX+o6W9wYCWOM6cdAAsRYYBdwGvCx4PXRQhYqbzwP6qdDTYEfUrfwM27cwaq7C3ucfGndBXt3FrsUxpgS129iI1W9ZDgKUlA1k91jQxvecSfyvO9/Csz6K3jzl67jump8/o+Rb42bIRqH8mEeQ2KMGTH6rUGIyDQReVhEtovINhF5UESmDUfh8qqi3jU5ebHC7H/hZ12q7Vd7qUWoD37SDV5LtkOi1TX3tDdBe4O7DXXvTtfZ3fy+S7jXuNkFtT0boWFTnoObuv1ap7UxphcDSY16J/DfwAXB8meDdR8uVKEKpqwKJhwGu992J+h8qj0AZv4VrHnI9Ueo3/2VD1OPgtOvh3hdfvbnJ9ydTbUHuASE4rn8Vl7E0oUbYxDt544WEVmlqgv6WzdcFi9erMuXL9+3nfg+7F7vruDzqXUXrL4f8N2JVqTrpBs6H/LyPCBnKh40b4Pld7jHn555M4yfmd+y9yAhQSNdxt7WR93NAV7UvSzIGFMyRGSFqi4ezGcGUoPYKSKfBdJtJxfhOq1HLs9zA+p2vuWafPKlchwcd3n/2w3VAQvhN/8Kv7gKTvwKzDqzcMdCXZPYvvCirkkvEnOBJDMftWBizAgwkADxt8D3gW8DCvwhWDeyRaIuSOxaW5iO60KYeDictwyevhGe+/9cgDvuiuI+RKkvftK9kgN4VnZfwUQ83FeP7mM4VN36zLqwee2+be58RhCgMoFqX5YFomVuHI43upIVmP1Ln2cWEYkAn1TVjw9TeYZXWSWMOch11o4UFfXw19+El/4L/nw/7FoHp9/gmp5GssEEkxFDIFbp+r7KqlzW4UiJBnNjQgxkJPU5w1SW4qioh+rJxS7F4HhR+OBVcNr/ge1vwkOXwrbXil0q04NCYq97wt+eDbDtz7DtddizCfbugkR7sQtoTJ8GcjnzexH5PnAvkMn2pqorC1aq4VY7xV25tjcWuySDc+gZbiDgU/8Kv7wGjv8yHDEyxjDut1Id0NYBbUF2XS/aVbtI1zasP8aUiIHcxfRsyGpV1dMKU6S+5eUupjC+Dzv/MjKbONqb4Ldfg80vu8ehHv+l4UlWaPJPvK5AEavs6sAXD5CuPg7xuuatn8MMQN7vYhIRD7hVVe/bp5KNBJ7nnnO98619v3tnuMVr4axvwPI7YdXP3S28Z9zo0p+bkUV9d/v1UG7Bzg4ifQWU3BpKdsf+Pi9n3cKdfQt07u3RmWXJWc5+3wJfsfUZIFTVF5GrgdEfIMDdeVI/w3X89rjLpcR5ETjmCzBhFjz3dXj4MjjjqzBlfrFLZoZL+m68EfbV7Z10DxxhQltAQtb1uV3YuKSsQJv96rGt9DIfbJu+m079rvnQdfS+HcH6Huuy99XHe93u7hucgfRB/EZEvkLPPojR+Yiy8mqoOxAa3yl2SYZmxknuzqyn/hV+9b/hg1fC7POsXduMQAqaglSq2AXZbw10HATAVVnrFDgk/8UpEVXjXCqO1hGa8bR+Opx7Kzz7dfjD92DHW3Di/3bJ+YwxZoAGks11xnAUpOTUTYNkB3Q2F7skQ1NWDWd+DV75ueub2P22Wy506nNjzKgxkGyulSLyf0RkWbA8U0T6vZdSRO4IMsCu6eX9U0SkUURWBa/rB1/8AhJxV+KR8mKXZOjEg0Wfg7O+Ds3vwUOXweYC3AFmjBmVBnKbwJ1AJ/ChYHkzcPMAPvdj4Kx+tvmdqi4IXjcNYJ/DKxKFsTPcXRUj2UHHwbn/5XJFPf4P7sFG9thRY0w/BhIgPqCq/wEkAFS1jUzimd6p6vPAyO/IjlVA/cHFLsW+q5sGn7gFZpwMf/oveObG/Kc8N8aMKgMJEJ0iUkFwT5iIfADoyNPxPygir4rI4yIyu7eNRORSEVkuIst37CjCc5/jdVBzwPAfN99ile55EsdeDhueh0eudA8lMsaYEAO5i+kG4AngQBG5CzgeWJqHY68EDlbVFhH5CPAIEPqQA1VdBiwDN5I6D8cevJpJbpR1256iHD5vRGD+Evc8iadvdP0S8y5wI6/T95qnB1QhQV1Rer6XncVUPLddWTVUjnePXK0c57KxGmNGrIHcxfQbEVkJHIc7DXxZVff5/k9Vbcqaf0xEfiAi4/Ox74KpO8jd2TQammamHhWkDv8qrPhxYY4RHxMEi6ygUTXB3UZcOd7Nx2t7HwRljCmqAeUeVtVdwK/zeWARmQxsU1UVkWNwzV2l/SCidDqOHW+5x3WOdDWT4RO3ugRymRGXuGnuCE/8rvfCnq2gCh1NLktp6w433bvTjSXZu9OlMGlr6PpcmhcNAkd2IBnvgkjVBDdfMcbVcCIxCybGDKOCJacXkbuBU4DxIrIZ11QVA1DV24DzgStEJAm0AUu0v8yBpSASc3c27VzLqMhpIJK/AXTVE2Hcob2/7yehdTfs3eEez7p3Z/f5PRtcwsG+amiZJ9HFXGoUryyYxnpOI2VdgSVn6ntl+F4UJSutAlnpEiCTPkHJei9oXtPc7chqehMPr7ySaEU9UlHnRudbYDMjUMEChKpe1M/738c9qW7kKaty6SwaNhW7JCOLF3VBpL8kgp3BKPbWXS6AtDdCKhG8OrOmnTnLbqrtzWiqA5JZn/Hd+5LqRFxYGNAdGvmg4qFltWi8DuJ1SEUdEh+DBMuhr1iFpUcxRddvgAjuWtqsqh0icgowD/ipqjYUunAlrXIsJNrcw2BMfpVVQtlBLgjnUIWE75NMKSlfM/Nu2SeRUlJ9VUTV5fcRP+FeOQnSJCcxWmYZQH23faa5Lb2s3ZrhRH28ZCuRziYiHU1u2tmIl17etYlI55+JdDYhvT3u1osF/TMRupr8oM8mPnLe77W5UIMkeNnPBY+5cT/Z08yzw7OWI7G+PyeRnMewStdseib3Ma2Zt7K3z35PMr9/90qBn+pazsznrM+8l8rZLme95vz+Mr9buub7euRtt/ezP0NW8r6sDLU9stdmbeOFbZv1exlmA6lBPAgsFpFDgR8BjwL/DXykkAUbEeqmQrLdtb3jvi9JX4l6Yhd/fVCFlCq+Kr4PqkoKRX1I+YpC8J472atCMjWAk/9AiIBEUS+KUpGXn2fIVPESe4MA4l5eRxPRziZiiWaiiSYEH0EQT9xUsudds5cnbn3m5+txl1nOHWkIEJwkUwnX9Jee+glIBVM/6Zr7OpK9b5OelsJz3fs8EfdzgnY7CCZZf7zdTs7ZwU5C3s/9THBx4ftZwSgscGUFvtxApqmiDmodSIDwVTUpIucC31HV74nIK4Uu2EiQTPm0Vkyls7GZ9rZW2hOpdEZ8ohGPWMQjFhFiEY9oRCiLeMQ8b9SluU+klPZEikTKdyf+9EneJzjBK76C77ugMAp6bvJDBL+sGr+smgT7Ns5GgIgnRDwXLCIieB5EpGtd1BPEgwiCF2wb9fJ0JeOnguCRzryaW8shZ76PK+8e2wTrup3gc662M8+7GD2SvtKRTJFKuR8vElwMRLz0dJA7vGHRoMswkACREJGLgIuBjwXr9ssb3NsTKVo7U+ztSNKWSNGRcFdN4k2hPLk+s50CiZRPIhV+VRX1hGjEoywixKIuaMSCgDLo//Qi6Ez5tHX6tCXc7yGZslN+sSnuhJL0B/d/IeBONhF30ol6gicQ8TwXcETwIl2BJiK9fEfA8oRtAAAbu0lEQVS9iHuZIUkEwaAz4dOR9GlP9v93lf6/8zwhIuB5XvB/J8F6iIhbN9SL0oEEiEuAy4F/U9UNIjID+PnQDjdypHyltTNJa2cqeCXxe6lFazROZ81BlDVtZCB3Nrk/5BTtIXfKRsTVOGJRNy0Lah+xiJe/q71B6kj6tHWmaEukaE+kBn0SMqVLcbW8VHLg/6fZtZX0FW00q/bi9qvdKgeqYeu61ya7PTcns65rGwFEXAAT6VoWIWhmwzW90bWM4E6QQRNbZjsJtkOGvUafSCntyRSdSZ+OhE9Hcmh/U5n/u5S6PEjkv5lvIAPlXge+lLW8AfhG3ktSZO2JFG2dKfZ2JmnrTNGeGNwv2y+rIVk5mWjre/tUDvfHmqI95Kmn6T/MqCd4nruSc8ueu8KLuCuGSESIytD7QdoTfiYYtHWm9r3d34wqQ62tlDJPuq7GPdzfTsTrCj5eJjB1zXteOsBIJvBEcv7usoNBe8KnIzGwv6ekr2xp9tnY6NPUqdSUCdUxcdMyoaYMqmIuKBfSQO5i+ijwNeDgYHsBVFVrC1qyAkr5SlsiRWtHVw0hlYcve7JyAioeoikguFce6NZRiKA97ujIur8+rPMr2I+X2Euko4FkooWBXC144vpC0k0HmeASNClEg2nKVxcUOlO0JZKMor/7QUmklLYktCWVtqTSmlDak9CaVNoS7r3WpNKeUFqTBOvcfHuwfVswH/Ggpkyoirk/6OrMFDfNXhfMV5UJlVG6OpzNsPHV3RiRjy9/ugajaL+781V5v0XZ2JhiY6Ofmb7b7JMcwDVqVYxM0EgHkK4g0jWtKRvad2ogTUzfAc4D/jwiBrLl8INg0BZcDWf3HRRCqmJc4fYdKScVHwt+gkhHI9GOBiTZ+6AyX6FzIN+yUcBXpS0BLQmlpTN4JZTmTmVvZl3P99Mn/rakMtCvhQAVMaiICpVRycxPqPSoiLr5hE/muNv2+qwPjtfazwB8T9yVYTqQVMWEypi4Y8WgMuaOWRnrezkedSepkSrlK9tb3e/u/b0+O1rdDQ5RT4h5EPWCvjyPYFmCdQxom9zlfP6u0k0/3dapsrNNM0FgQ4PPpqYU7zT6tGc9UXVSpTB9TIRjDogyvc5jel2EMXFhb6f7LrekpwmluUNpTpD5Pjd1Kpua/Mw2+TjNDSRAvAusGQnBQVW7BYK2zhQdSX/0PfrAi5GqGE+qYjyS6iDS0UikowFJtRe7ZPsskVIaO5WmDqWhXWnscF/8luBE39ypmT+WzIk/oexN9H/xVxWj21X7AdVecPINTvY58/EoQQDoOkFXRIXyyNCv8lO+K2t22bMD1t5OpTmh7O2E5k5XQ3l/r5+pnexN6ICuLNNBrCt4uNpJer6mTBhTLoyJC3XBdEy5x5i4+1kLLeUru9rcz+aCgJt/v8Ut72jr/+o7n6IelHm4uw49KItAzBNiESgLpm69ez8WkWD73rfrTCnvNPlsaPTZ1Jhib9bFwdi4ML3O4yMfKHOBYIzHwbURKmPhv/vxg7wjW1XpSNEVUDqVT/77EH4vA9jmH4DHROR/yErzrarfGvzh8kdVM23lLhgkaU+MwmDQD42Uk6ycSLJyIpJsJ9LRQKSjEfHzlZF9H8qmrvmlsV1p7PTdtKO3l09DR99X2OURujXHjK3wOKhWqCqDmmBddgDIbuKpjFLw9tqBiHhCbTnUlg+9LJ2pdBOYCxhtCaU16QJP17yrUbUmlL1BE1lrAna3+5ng1BbSzwUQj9AVNOJe90BS7tbVx918XbkQi/T8WXxVdrdrcMIPTv7pYNDis71Vyb5JR4BxFcLkKo85E6JMrnbzk6o8Jld5TKh0d+qkFBI+JP10P4ibT2TNJ33tc5uED6lgOZHq2r4z1bWuM5i69e79lk7NWq8524dfoNSUwfS6CKcdHGN6XSSoFXjUlhe2Z1zEXeDEo8L4yqHvZyAB4t+AFiAOlA39UPmRSPms297ixhzsZ8GgPxqNk4xOJlk1GUm2ZQWLzoIe970Wn6c3JninKdXtpN/U0Xs1N+bBmHKhNjjJTK6KUBecjGqDE1H2tComlIWciPZHZRH3u6jbx6fhtifd/1NDu9LQ4QdTzZr67G7zeXuPW+7t/7IqRiaQlEfINA3lbj82Lkyq8jh8XISTD/KYUu0CwKQqYWKlN6D/36i4q32ntL4Pqayg0elDRNx3fCC1zeyxU2WR4Db44E5GTwRfXeaA9ODS9HwqWJ8epKvBcnrA6b4aSIAYq6pn5uFYeZFMKW2dqf433M9ptIJktIJk1ZRM53akoxG0l8vGQepMKX/YkuTx9Z2s3JZCgCnV7iQxqcpjVr1QF1x1dr28zHyFdcYWXTwqxKPCpCqAvscwqLoaSENHVjBJB5IgmDS0uxrjIWM8PjQ1yuQqj8nVLgBMqvQoH4amq2KKeEKFBxV9BC43YDaSGUAbiwoxz93O3vefg/T3X9SD70MKN3DVH2J73UACxNMicqaqPjWkI+SZVRoGz49V4ceqSFQdgJdoccGis8kN4x+kDQ0pnng7wW82JmjuVCZVChfPKefMGTEmVo2AUX5mSERcU15VmTC1Znj+nwWIxyJUlEUoj3lEXGYs0tfGXWMl0jNdn82Mqcgs97yiVrpvFDYmo9t2ucfotr32WO9J9ywK/QeB/Erfhss+NK1Kf33PItIMVOH6HxIU+TbX+JSZeua/3MmcA+o48oBaDptUQzxmIzgHTdUlj+tsxEt2oFnpCjTIaaNBGuvWJPzPhhaefKuRt3a0EfWED06v48wjJjB/Wi2eF8nc3iupIAme39ljvhADeczo4QlUxKJUlEWIxzzi0choy55RVDJ10QpVXTyYz/RZgxDXBjBbVd/Zp5LlUW1FjJaOJHf/6R0Ud6vaoROrmX1ALbMPqOOIKbVUlxcsi/noIYJfXodfXhf6tqry1rZmnnp9G79bu4P2hM+BYyv5/AkzOPWwidRVdGVb6bqAi6KRPhrG/VSvwSOdXXU01BHTg65Ij+yVrtZySf+bNRI4vb5bvr0s6dvz/ay8VhokO8wedTzSRCNCRSxCPBrUEqLFrIFKLxlWs54REvqK9PG+9Mw2q5qznJuRNiRDbRETIQ6kBrFCVY8apvL0a/a8hXrPY8/R0pHkzfeaeG1rE69tbWTt9haSvhtmNmN8FUceUJupZdRXFr1vfcRobEvw3Fvbeer1bbyzu5V4zOPEmRM484hJHDa5prD9BqpdabhTQfBIBw71ET+J+KmgH2V4T4sCweBC12TQY9BhMKI9/5l8JfxElVnvBlf6WS+l+zpVzwUYBB8vK71GkkQyiZ9Kkkwm8VPuhCTBSUmCk5RkTlL7dqIqi3rEox6VZVHisUhw91NfJ2av+3vdMtNKzjr6eC9s6uVsX+JVlUxK8r6mfp/bSNW4/NYgAn8UkaNV9eXB/kwFEfw/VpdHWTx9LIunjwVcqoy125pZEwSM37y+jV+tdmkvpo6pyNQwZh9Qy8SacusgzeKrsnpzI0+9/j4vrt9F0lcOm1TD1aceyokzx1NZNkw1MhE0UoZGyvpPBxkEjK6pCx7uZJZywSR4PoBoEvH90MASEQlO+C51STRIUpceeZ5JZZKPi9v08xW8KJnnMXR7RXouD/AOmAiD7sPsQdWlz0imlGTwnI2E75MK1iWSKVJBMEmlUu5ZFpoKfrddgSV9Ui8vi1JZHqOivIyq8hjRSJSewcD+DgekSEFsIDWI14FZwCZgL119EPMKX7yejjpqsd7z2LP95kpKpnzW79jLa1sbWbO1kdffa2Jvh+uUHV9dzvRxldTGY1THo9TEo1SXu1dNPNZtuao8WhL3z+eDr0pDa4Ltze3saO5gR3MH25o7WL5xN9ubO6gpj3Lq4RP58BGTmD6+atjLJ0ImMWHMc3d4ZF+VS7ppJrvJJki6lknUJt33l93EIyJBwAiCRuaKK1vI30Po30g/26WfQZD90J1RpLdggkJleZTKWARvlPzdjBZBa9CgahADCRAHh61X1U2DOVC+LF68WF9++WW2NbkT3ED5qmza1RoEjCbea2ijpSNJc7tLWd2XqvIINeUumLggkhVMgkCSHWgqy1xHW0UsMqzBJZHyMyf+Hc0dbG9uZ3tzBztautblJlirKo8wc2INZx45iWNnjKOsQO3AnkcmpXnUE8qibtqV7tzd+22MKYyCBIhSs3jxYl2+fDkAezuSbN7Tts/5hpIpn5aOpHu1J2kOAkdLRyKznD11gSVBS0f/ie3iMY/K4M6MyswrezmaWV8RC6ZZ69LvRz1hb2eKHemTfnNHZpoOBntyhiELUF9VxsSacibWlDOhppwJNXE3X13OxNryvDUfRTyhqjwSDPaRoAYQBIOIZ1eTxhTZUALEiK73VpVHmTmxmvea2tndMvTRwtGIx5jKMsYMsjPbVzdoryUTUFzgaAseLNQWPEcinTE2vb6xsa37cyYGEKM96TmUPxaR4EQfZ/H0sW4+EwzijKsuI1bgq3LPgwnV5YyvLrcgYMwoM6IDBIDnCVPHVFAbj7J5T9uwPt3ME6Eq6KeYNMRRIaqaeSBPOmC0ZgJMMhNo2hIpauJRJtbEg5pAOWMqYkXrbBeBsUHtxJqGjBmdRnyASKuJx5g1KcrWhjYa+supXEJEhHgsQjwWoX74+4WHZExljEm18YL1VxhjSsOoCRDg2sEPHFtJbTzBloa2vDwEyHSpiUeZXBe3kevG7CdGVYBIq6uMUVkeYcueNprDnt1pBqWiLMKUujhVNkLdmP3KqP2Lj0U8po+vYvfeTrY2tFlq8CEoj3lMqo13S6thjNl/jNoAkTa2qoyq8gib97TR2mFpwgciFhUm1sSpryxeJ7gxpvgK1ssoIneIyHYRWdPL+yIi3xWRdSKyWkQWFaos5dEIH5hQzeS6uI3s74PnweS6OLMm1jC2qsyCgzH7uULehvJj4Kw+3j8bmBm8LgVuLWBZAJhQU86hE6upKLO7b7KJuN/N4ZNrmVBj4xmMMU7BmphU9XkRmd7HJucAP1U3lPuPIjJGRKao6nuFKhO4B5B8YEI1Da0JGtrcSOn9VTzmUVsRY2xV4QfUGWNGnmL2QUwF3s1a3hys6xEgRORSXC2Dgw46aJ8PLCLUV5VRX1VGIuW7YNHa2W8CwNGgosyjNh6jtiJmt6saY/pUzAAR1o4Req+Rqi4DloHLxZTPQsQiXmZkcnsixZ7WThpaE8M6IrvQKsoi1FZEqauIUR61oGCMGZhiBojNwIFZy9OArUUqC+Can6bUVTClroLm9gQNrQka2xIj8hbZyvIItfEYdRUxG/FsjBmSYgaIR4GrReQe4FigsdD9D4PhngsRY6qvNLUn2NNa+v0VleUR6ipcULA+BWPMvipYgBCRu4FTgPEishm4geA5Yap6G/AY8BFgHdAKXFKosuwLz5NMptdS668QgcoyFxRqLSgYY/KskHcxXdTP+wpcVajjF0Ip9FeIuDTndRUxauNRy6RqjCmYUT+SulDy0V8h0vVYTM8LpplnqAte8DhNT9xjM6vLo9RWxEbNI1CNMaXNAkQeZPdX7O10/RTuBO9O7tB1ks+c9O0kb4wpcRYg8sjzhJq4JbYzxowO1oBtjDEmlAUIY4wxoSxAGGOMCWUBwhhjTCgLEMYYY0JZgDDGGBPKAoQxxphQFiCMMcaEsgBhjDEmlAUIY4wxoSxAGGOMCWUBwhhjTCgLEMYYY0JZgDDGGBPKAoQxxphQFiCMMcaEsgBhjDEmlAUIY4wxoSxAGGOMCWUBwhhjTCgLEMYYY0JZgDDGGBPKAoQxxphQFiCMMcaEKmiAEJGzROQtEVknIteFvL9URHaIyKrg9YVClscYY8zARQu1YxGJALcAHwY2Ay+LyKOq+nrOpveq6tWFKocxxpihKWQN4hhgnaq+raqdwD3AOQU8njHGmDwqZICYCrybtbw5WJfrkyKyWkQeEJEDC1geY4wxg1DIACEh6zRn+ZfAdFWdBzwN/CR0RyKXishyEVm+Y8eOPBfTGGNMmEIGiM1Ado1gGrA1ewNV3aWqHcHi7cBRYTtS1WWqulhVF0+YMKEghTXGGNNdIQPEy8BMEZkhImXAEuDR7A1EZErW4seBNwpYHmOMMYNQsLuYVDUpIlcDTwIR4A5VfU1EbgKWq+qjwJdE5ONAEtgNLC1UeYwxxgyOqOZ2C5S2xYsX6/Lly4tdDGOMGVFEZIWqLh7MZ2wktTHGmFAWIIwxxoSyAGGMMSaUBQhjjDGhLEAYY4wJZQHCGGNMKAsQxhhjQlmAMMYYE8oChDHGmFAWIIwxxoSyAGGMMSaUBQhjjDGhLEAYY4wJZQHCGGNMKAsQxhhjQlmAMMYYE8oChDHGmFAWIIwxxoSyAGGMMSaUBQhjjDGhLEAYY4wJZQHCGGNMKAsQxhhjQlmAMMYYE8oChDHGmFAWIIwxxoSyAGGMMSaUBQhjjDGhChogROQsEXlLRNaJyHUh75eLyL3B+y+JyPRClscYY8zAFSxAiEgEuAU4GzgSuEhEjszZ7PPAHlU9FPg28O+FKo8xxpjBKWQN4hhgnaq+raqdwD3AOTnbnAP8JJh/ADhdRKSAZTLGGDNA0QLueyrwbtbyZuDY3rZR1aSINALjgJ3ZG4nIpcClwWKLiLxVkBLD+Nxjl7iRVl4YeWUeaeUFK/NwGGnlBThssB8oZIAIqwnoELZBVZcBy/JRqL6IyHJVXVzo4+TLSCsvjLwyj7TygpV5OIy08oIr82A/U8gmps3AgVnL04CtvW0jIlGgDthdwDIZY4wZoEIGiJeBmSIyQ0TKgCXAoznbPApcHMyfD/xWVXvUIIwxxgy/gjUxBX0KVwNPAhHgDlV9TURuApar6qPAj4Cficg6XM1hSaHKM0AFb8bKs5FWXhh5ZR5p5QUr83AYaeWFIZRZ7ILdGGNMGBtJbYwxJpQFCGOMMaH2ywAhIneIyHYRWZO1bqyI/EZE1gbT+mKWMZeIHCgiz4rIGyLymoh8OVhfkuUWkbiI/ElEXg3Ke2OwfkaQVmVtkGalrNhlzSUiERF5RUR+FSyXdJlFZKOI/FlEVqVvZSzV7wWAiIwRkQdE5M3g+/zBEi/vYcHvNv1qEpFrSrzM/yv4u1sjIncHf4+D/h7vlwEC+DFwVs6664BnVHUm8EywXEqSwN+r6hHAccBVQeqSUi13B3Caqs4HFgBnichxuHQq3w7KuweXbqXUfBl4I2t5JJT5VFVdkHVvfql+LwD+H/CEqh4OzMf9rku2vKr6VvC7XQAcBbQCD1OiZRaRqcCXgMWqOgd3k9AShvI9VtX98gVMB9ZkLb8FTAnmpwBvFbuM/ZT/F8CHR0K5gUpgJW4k/U4gGqz/IPBkscuXU9ZpuD/204Bf4QZzlnqZNwLjc9aV5PcCqAU2ENwgU+rlDSn/mcDvS7nMdGWoGIu7U/VXwF8N5Xu8v9YgwkxS1fcAgunEIpenV0HW24XAS5RwuYOmmlXAduA3wHqgQVWTwSabcV/mUvId4B8AP1geR+mXWYGnRGRFkJYGSvd7cQiwA7gzaMb7oYhUUbrlzbUEuDuYL8kyq+oW4JvAO8B7QCOwgiF8jy1AjDAiUg08CFyjqk3FLk9fVDWlrlo+DZe88YiwzYa3VL0TkY8C21V1RfbqkE1LpsyB41V1ES5z8lUiclKxC9SHKLAIuFVVFwJ7KZGmmf4EbfYfB+4vdln6EvSFnAPMAA4AqnDfjVz9fo8tQHTZJiJTAILp9iKXpwcRieGCw12q+lCwuuTLraoNwHO4vpMxQVoVCE+/UkzHAx8XkY247MOn4WoUpVxmVHVrMN2Oaxs/htL9XmwGNqvqS8HyA7iAUarlzXY2sFJVtwXLpVrmM4ANqrpDVRPAQ8CHGML32AJEl+y0Hxfj2vhLhogIbuT5G6r6ray3SrLcIjJBRMYE8xW4L+0bwLO4tCpQQuUFUNV/UtVpqjod15TwW1X9DCVcZhGpEpGa9DyujXwNJfq9UNX3gXdFJJ1Z9HTgdUq0vDkuoqt5CUq3zO8Ax4lIZXDeSP+OB/89LnaHSpE6ce7Gtc0lcFc0n8e1NT8DrA2mY4tdzpwyn4CrEq4GVgWvj5RquYF5wCtBedcA1wfrDwH+BKzDVdXLi13WXsp/CvCrUi9zULZXg9drwL8E60vyexGUbQGwPPhuPALUl3J5gzJXAruAuqx1JVtm4EbgzeBv72dA+VC+x5ZqwxhjTChrYjLGGBPKAoQxxphQFiCMMcaEsgBhjDEmlAUIY4wxoSxAmBFBRMZlZdN8X0S2ZC0PKLuqiNyZdf99b9tcJSKfyVOZzwnK96qIvC4iX+hn+9OChIZh700Rkcey9vVosP5AEbk3H+U1Jpfd5mpGHBH5KtCiqt/MWS+477Qf+sFhJCLluKR0i1V1a7B8sKr+pY/P3AzsVNXvhLz3I9wo3luC5XmqurpAxTcGsBqEGeFE5NAg5/1tuIyxU0RkmYgsD/LhX5+17QsiskBEoiLSICLfCK7IXxSRicE2N4vINVnbf0Pccy3eEpEPBeurROTB4LN3B8dakFO0Olwep90AqtqRDg4iMklEHgo+9ycROU5EPgB8Abg2qHV8KGd/U3CDOgn2tzrr518VzN+ZVavaKSL/Eqy/LjjO6uzfhzH9sQBhRoMjgR+p6kJ1mSyvU/dchPnAh4PnZuSqA/5H3fMqXgT+tpd9i6oeA1wLpE+ufwe8H3z2G7jMut2oy4v0JLBJRP5bRC4SkfTf23eB/wjK+Cngh6q6Hvgh8H/VPXvgDzm7/D7wExH5rYj8czoHUM4xL1GXHPFcXGrnn4rIR4CDcKnWFwAfCgk+xoSyAGFGg/Wq+nLW8kUishJXozgCF0Bytanq48H8CtzzQcI8FLLNCbhkfqhqOsVFD6q6FPfMjuW4jKXLgrfOAG4LrvwfAeqDfFW9UtXHgA/g8nEdCbwiIuNytwv2cz9whaq+i8vNdDYu7clK4FBgVl/HMiYt2v8mxpS8vekZEZmJeyLcMaraICI/B+Ihn+nMmk/R+99CR8g2YSnAQwVNQatF5L9xyQq/EHz+GFXNLgOuC6XPfe0C7gLuEpEncIEqNzjdDtyjqs9mlfVmVf3RQMtsTJrVIMxoUws0A01BM8xfFeAYL+CahhCRuYTUUESkVro/l2EBsCmYfxq4KmvbdP9FM1ATdkAROT1dyxCRWlyu/3dytvkyEMvpvH8S+HyQ6RURmSYi4wf4c5r9nNUgzGizEpfaeA3wNvD7Ahzje7j2/dXB8dbgntqVTYB/EpHbgTagha5+jquAW0XkEtzf4LPBul8A94vIecBVOf0QRwPfF5EE7sLuVlV9RUQOzdrmK0BrutMa+L6q/lBEDgf+GNRQmoFP4/oojOmT3eZqzCCJe+hKVFXbgyatp4CZ2vU4R2NGBatBGDN41cAzQaAQ4DILDmY0shqEMcaYUNZJbYwxJpQFCGOMMaEsQBhjjAllAcIYY0woCxDGGGNC/f/x1kWgzUS9WAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c5fe90b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows a typical learning curve: for very few training points, there is a large separation between the training and test error, which indicates **over-fitting**.  Given the same model, for a large number of training points, the training and testing errors converge, which indicates potential **under-fitting**.\n",
    "\n",
    "As you add more data points, the training error will never increase, and the testing error will never decrease (why do you think this is?)\n",
    "\n",
    "It is easy to see that, in this plot, if you'd like to reduce the MSE down to the nominal value of 1.0 (which is the magnitude of the scatter we put in when constructing the data), then adding more samples will *never* get you there.  For $d=1$, the two curves have converged and cannot move lower. What about for a larger value of $d$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztvXmcXFWZ//9+aumq3qvS2fcAYUvIRlgUZZcBZ4QBUYP6HYKjLOo4OCMz6Hx/IBG/wziOMoLCKyjMODLIIiAqi4IgosgSCDEQIAkkZN87vS9V9fz+OLeqq6uruqu7q7qrK8/79arce889996nK1XnU885z3mOqCqGYRiGkYlvtA0wDMMwShMTCMMwDCMrJhCGYRhGVkwgDMMwjKyYQBiGYRhZMYEwDMMwslI0gRCRsIi8KCKvicjrInJDljohEblXRDaIyAsiMrtY9hiGYRiDo5geRCdwpqouBBYB54rIyRl1/hY4oKpHAN8F/q2I9hiGYRiDoGgCoY4W7zDovTJn5V0A/Le3/wBwlohIsWwyDMMw8idQzJuLiB9YBRwBfF9VX8ioMg3YAqCqMRE5CDQAezPuczlwOUB1VeXxRx8xu3BGKrDvbaiMQvWE3PUCIRB/4Z5rGIYxgqxatWqvqvbTyPWlqAKhqnFgkYhEgIdEZL6qrk2rks1b6JP7Q1VXAisBli48Vl9+7O7CGvrAZ6BmIpx7U+469TOgenxhn2sYhjFCiMjmwV4zIlFMqtoIPAOcm3FqKzADQEQCQD2wfyRs6kVkFhx4r/868e6RscUwDKNEKGYU0wTPc0BEKoGzgTczqj0CXOrtXwz8Vkcje2B0FjTvgFhn7joJEwjDMA4tiulBTAGeFpE1wEvAb1T1lyKyQkTO9+r8CGgQkQ3APwDXFtGe3ERmAQqN/XgR5kEYhnGIUbQxCFVdAyzOUn5d2n4H8LFi2ZA30Vlu2/gejJ+bvY4JhFEmdHd3s3XrVjo6OkbbFKMIhMNhpk+fTjAYHPa9ijpIPWaonw7igwObcteJd42YOYZRTLZu3UptbS2zZ8/GosrLC1Vl3759bN26lTlz5gz7fpZqA8BfAXVTobGfQX6NQyIxcjYZRpHo6OigoaHBxKEMEREaGhoK5h2aQCSJzOp/DAJsoNooG0wcypdC/t+aQCSJzoLGLZCI5a5j3UyGYRxCmEAkicxy3UhN23LXsYFqwxg2jY2N/OAHPxjStR/+8IdpbGzst851113Hk08+OaT7G70xgUiSjGTqb8Jcf96FYRh50Z9AxOPxfq999NFHiUQi/dZZsWIFZ5999pDtKxYD/W2liAlEkshMt7VIJsMoKtdeey0bN25k0aJFXHPNNTzzzDOcccYZfPKTn+S4444D4K//+q85/vjjmTdvHitXrkxdO3v2bPbu3cumTZs45phj+NznPse8efM455xzaG9vB2D58uU88MADqfrXX389S5Ys4bjjjuPNN91c3T179vChD32IJUuWcMUVVzBr1iz27u2VAo54PM7y5cuZP38+xx13HN/97ncB2LBhA2effTYLFy5kyZIlbNy4EVXlmmuuSdW99957AbL+bT/5yU848cQTWbRoEVdccQXxeDzns0YbC3NNEqyC6on9RzJZF5NRZtzwi9d5Y3tTQe957NQ6rv/IvJznb7rpJtauXcvq1asB14i++OKLrF27NhWaeeeddzJu3Dja29s54YQT+OhHP0pDQ0Ov+6xfv5577rmHO+64g49//OP87Gc/49Of/nSf540fP55XXnmFH/zgB3z729/mhz/8ITfccANnnnkmX/3qV3n88cd7iVCS1atXs23bNtaudenjkl1bn/rUp7j22mu58MIL6ejoIJFI8OCDD7J69Wpee+019u7dywknnMCpp54K0OtvW7duHffeey9/+MMfCAaDfP7zn+fuu+9m3rx5WZ812pgHkU50gEgmEwjDKAonnnhir7j9733veyxcuJCTTz6ZLVu2sH79+j7XzJkzh0WLFgFw/PHHs2nTpqz3vuiii/rUee6551i2bBkA5557LtFotM91hx12GO+88w5/93d/x+OPP05dXR3Nzc1s27aNCy+8EHCT0qqqqnjuuee45JJL8Pv9TJo0idNOO42XXnqpz9/21FNPsWrVKk444QQWLVrEU089xTvvvJP1WaWAeRDpRGbBul+AJtzEuUwszNUoM/r7pT+SVFdXp/afeeYZnnzySZ5//nmqqqo4/fTTs8b1h0Kh1L7f7091MeWq5/f7icXcOGI+Kd+i0SivvfYaTzzxBN///ve57777uPnmm7PW7e9+6X+bqnLppZfyr//6r33qZT7rzjvvHNDGYmMeRDrRWRDvhJZd2c+bB2EYw6a2tpbm5uac5w8ePEg0GqWqqoo333yTP/3pTwW34QMf+AD33XcfAL/+9a85cOBAnzp79+4lkUjw0Y9+lG984xu88sor1NXVMX36dB5++GEAOjs7aWtr49RTT+Xee+8lHo+zZ88enn32WU488cQ+9zzrrLN44IEH2L17NwD79+9n8+bNWZ9VCpgHkU56JFPtlCwV1ImEf/g5TgzjUKWhoYFTTjmF+fPnc9555/GXf/mXvc6fe+653H777SxYsICjjjqKk0/OXKl4+Fx//fVccskl3HvvvZx22mlMmTKF2traXnW2bdvGZZddRsLLoJD81f8///M/XHHFFVx33XUEg0Huv/9+LrzwQp5//nkWLlyIiPCtb32LyZMnpwbFkxx77LHceOONnHPOOSQSCYLBIN///veprKzM+qzRRkYju/ZwKMqCQUk6DsKPL4CTr4IFn8heZ/xRUFFVnOcbxgiwbt06jjnmmNE2Y1Tp7OzE7/cTCAR4/vnnueqqq1KD5uVAtv9jEVmlqksHcx/zINIJ10M4Agf6iWSycQjDGPO89957fPzjHyeRSFBRUcEdd9wx2iaVJCYQmVgkk2GUPXPnzuXVV18dbTNKHhukziQyy02Wy9X1ZgJhGMYhgglEJtFZ0NUC7TmWxrbZ1IZhHCKYQGQSSVtdLhuWj8kwjEMEE4hMUqGum7Kfty4mwzAOEUwgMqka7/Iy5Ypksi4mwxhxampqANi+fTsXX3xx1jqnn346L7/8cr/3ufnmm2lra0sd55M+/FDGBCITkf4jmWzpUcMYNaZOnZrK1DoUMgUin/Tho0GppAY3gchGMpIpFzYXwjCGzD//8z/3Wg/i61//Ov/xH/9BS0sLZ511Vio1989//vM+127atIn58+cD0N7ezrJly1iwYAGf+MQneuViuuqqq1i6dCnz5s3j+uuvB1wCwO3bt3PGGWdwxhlnAD3pwwG+853vMH/+fObPn5/KudRfWvF07r//fubPn8/ChQtTWVzj8Thf+cpXOO6441iwYAG33HIL4BL2LV68mOOOO47PfOYzdHZ2pmxZsWIFH/jAB7j//vvZuHEj5557Lscffzwf/OAHU7Oysz2rWNg8iGxEZsHbj0NnM4Rq+56Pd0Eg1LfcMMYaj10LO/9c2HtOPg7Ouynn6WXLlnH11Vfz+c9/HoD77ruPxx9/nHA4zEMPPURdXR179+7l5JNP5vzzz8+5xvJtt91GVVUVa9asYc2aNSxZsiR17pvf/Cbjxo0jHo9z1llnsWbNGr70pS/xne98h6effprx48f3uteqVau46667eOGFF1BVTjrpJE477TSi0WheacVXrFjBE088wbRp01JdVitXruTdd9/l1VdfJRAIsH//fjo6Oli+fDlPPfUURx55JH/zN3/DbbfdxtVXXw247LDPPfcc4PI23X777cydO5cXXniBz3/+8/z2t7/N+qxiYR5ENqIDRDLZQLVhDJnFixeze/dutm/fzmuvvUY0GmXmzJmoKl/72tdYsGABZ599Ntu2bWPXrhyJM4Fnn3021VAvWLCABQsWpM7dd999LFmyhMWLF/P666/zxhtv9GvTc889x4UXXkh1dTU1NTVcdNFF/P73vwfySyt+yimnsHz5cu64445U99CTTz7JlVdeSSDgfoePGzeOt956izlz5nDkkUcCcOmll/Lss8+m7vOJT7gUPy0tLfzxj3/kYx/7WGphoR07duR8VrEwDyIb6ZFMk7KkQzaBMMqFfn7pF5OLL76YBx54gJ07d6bWZbj77rvZs2cPq1atIhgMMnv27KxpvtPJ5l28++67fPvb3+all14iGo2yfPnyAe/TX066fNKK33777bzwwgv86le/YtGiRaxevRpV7WPfQLnvkqnBE4kEkUgka36obM/KXEypUJgHkY2ayS5ja67V5WwMwjCGxbJly/jpT3/KAw88kIpKOnjwIBMnTiQYDPL000+zeXM/OdGAU089lbvvdok7165dy5o1awBoamqiurqa+vp6du3axWOPPZa6Jleq8VNPPZWHH36YtrY2Wltbeeihh/jgBz+Y99+zceNGTjrpJFasWMH48ePZsmUL55xzDrfffntqDYr9+/dz9NFHs2nTJjZs2AC4zLCnnXZan/vV1dUxZ84c7r//fsAJy2uvvZbzWcXCPIhs+PxujeoD1sVkGMVg3rx5NDc3M23aNKZMcan1P/WpT/GRj3yEpUuXsmjRIo4++uh+73HVVVdx2WWXsWDBAhYtWpRaf2HhwoUsXryYefPmcdhhh3HKKaekrrn88ss577zzmDJlCk8//XSqfMmSJSxfvjx1j89+9rMsXrw45yp1mVxzzTWsX78eVeWss85i4cKFzJ8/n7fffpsFCxYQDAb53Oc+xxe/+EXuuusuPvaxjxGLxTjhhBO48sors97z7rvv5qqrruLGG2+ku7ubZcuWsXDhwqzPKhZFS/ctIjOAHwOTgQSwUlX/M6PO6cDPgXe9ogdVdUV/9y1quu90nloBu9+AS37a91ywGiYcWXwbDKMIWLrv8mcspPuOAf+oqq+ISC2wSkR+o6qZo0W/V9W/KqIdQyM6GzY+DbEOCIR7n7MuJsMwDgGKNgahqjtU9RVvvxlYB0wr1vMKTmQmoNCYpX/PupgMwzgEGJFBahGZDSwGXshy+n0i8pqIPCYipbGCOvQk7cs6YU5NJIwxzVhbSdLIn0L+3xZdIESkBvgZcLWqNmWcfgWYpaoLgVuAh3Pc43IReVlEXt6zr+/i4kWhfjqIL3ckkwmEMUYJh8Ps27fPRKIMUVX27dtHOBweuHIeFDWKSUSCOHG4W1UfzDyfLhiq+qiI/EBExqvq3ox6K4GV4Aapi2lzCn8Q6qb1E8nUBdja1MbYY/r06WzdupU9e/aMtilGEQiHw0yfPr0g9yqaQIibIfIjYJ2qfidHncnALlVVETkR59HsK5ZNgyY6Cxo3ZT9n60IYY5RgMMicOXNG2wxjDFBMD+IU4P8AfxaR5HTArwEzAVT1duBi4CoRiQHtwDItJb83Mhs2P+/EwJfxVlkXk2EYZU7RBEJVnwOyZ9nqqXMrcGuxbBg20ZkuvffBbT3pN5LYuhCGYZQ5lmqjP/qLZLIuJsMwyhwTiP6IzHTbbJFM5kEYhlHmmED0R7ASaiZlT/ttYxCGYZQ5JhADEc2xupwtPWoYRpljAjEQkdku3YZmEQPrZjIMo4wxgRiI6EyId0JzlpWtLGmfYRhljAnEQCQjmbJNmLNxCMMwyhgTiIFIhbpmi2QygTAMo3wxgRiIcB1URrNHMlkXk2EYZYwJRD5EckQy2SC1YRhljAlEPkRnu8lymWmi4jab2jCM8sUEIh8iM6GrFdr39y63LibDMMoYE4h8iObIyWSD1IZhlDEmEPkQne22fSKZbOlRwzDKFxOIfKgcBxXVlpPJMIxDChOIfBCxSCbDMA45TCDyJRnJlImtC2EYRpliApEvkZnQfgA6mnqXWxeTYRhliglEviQjmTK9COtiMgyjTDGByJfIbLfNFAjrYjIMo0wxgciX2kngD8GBjEgm8yAMwyhTTCDyRXwQmdE37beNQRiGUaaYQAyG6Oy+k+Vs6VHDMMoUE4jBEJkJLbugu713uXUzGYZRhphADIZkyo3MGdWWtM8wjDLEBGIwRHKFuppAGIZRfphADIb6aSD+LJFMJhCGYZQfJhCDwReA+ul9I5msi8kwjDKkaAIhIjNE5GkRWScir4vI32epIyLyPRHZICJrRGRJsezphS/gPIGhEJnVN5LJBqkNwyhDiulBxIB/VNVjgJOBL4jIsRl1zgPmeq/LgduKaI9D/DDucKiMDO366Exo2ta7W8mWHjUMowwpmkCo6g5VfcXbbwbWAdMyql0A/FgdfwIiIjKlWDYhPmg4HCqqIDxUgZgNmoCDW3vKrIvJMIwyZETGIERkNrAYeCHj1DRgS9rxVvqKCCJyuYi8LCIv79l3YIhG+GDcYW7hH4BQLfiCg79PtkgmG6Q2DKMMKbpAiEgN8DPgalVtyjyd5RLtU6C6UlWXqurSCQ3RoVjhfvmHatMNg3D94G8VmeHu12suhC09ahhG+VFUgRCRIE4c7lbVB7NU2QrMSDueDmwvsBVOHLKJQeUQxCYQhtrJfVeXM4EwDKPMKGYUkwA/Atap6ndyVHsE+Bsvmulk4KCq7iioIZGZuQekQzXgrxjCPWfZuhCGYZQ9gSLe+xTg/wB/FpHVXtnXgJkAqno78CjwYWAD0AZcVlAL6mdC1bj+64Qj0Lp7cPeNzoTtr0AiDj4vXNbWhTAMo8womkCo6nNkH2NIr6PAF4piQN00qG4YuF5ldPACEZntPIaWne45YB6EYRhlR3nOpK6dAjUT86tbUeXGFQZDcvnRAxbJZBhG+VJ+AlEzyQ0iD4bBzomIzHTb9Egm62IyDKPMKC+BqJ4AdVMHf91go5lCtVDV0DuSybqYDMMoM8pHICrHuUR6QyEYhkDl4K7JjGSyLibDMMqM8hCIcKRnXGCoDNaLiMx0ab/Vm9dnS48ahlFmjH2BCNf3rPQ2HAabvC86G7pboW1vT5l1MxmGUUaMbYEI1UF0jkubMVwCIQhW518/WySTJe0zDKOMGLsCUVFTOHFIMphuplQk0wiNQ1hKccMwRph+BUJE/CLy7yNlTN4Eq1xmVl+B9W0w3UyV41w0UzHnQqhC+wHYuwF2v24D4YZhjCj9zqRW1biIHC8i4s16Hn3E5xb88Q1xRbj+8Aehoha6mvOwQ/pGMhWqiyneDW37oHVv73u27Bp6pJZhGMYgySfVxqvAz0XkfqA1WZgjO2vxCYTAX8QUUpWR/AQCXDfT5j/2HA93kLqzBVr3QMdBsmQ9d6JRM8kJmWEYRpHJp6UdB+wDzkwrU2B0BKL/9E7DJxzxVovLw2GKzoa3HnUNerh+aOMEiQS073feQqy9/7qaMC/CMIwRY0CBUNXCZlgtdfwBN7bQmbm2URbSV5ebvGBwHkR3hwuRbdvv5lDki3kRhmGMEAOO8orIdBF5SER2i8guEfmZiJT3T9h8o5miXiRTcqA6EeuZOJcNVWhvdIPOe9a57qTBiAP0eBGGYRhFJp8woLtwC/tMxa0X/QuvrHwJ17vB8IGomeQywaYimTR70r54NzTvhN1vwIF38x/jyEXbPotoMgyj6OQjEBNU9S5VjXmv/wImFNmu0cXn771+dS7E5waqe82FSOtm6myB/e/CrteheUfhZlqbF2EYxgiQj0DsFZFPe3Mi/CLyadygdXmTbzdTZGbvuRCxTjfgvPtN2LceOhrJa8B7sJgXYRhGkclHID4DfBzYCewALvbKyptQPUgecy2is92KdN1t7rhxMxzcMnBE0nAxL8IwjCLTbxSTiPiBj6rq+SNkT+ng80G4zs1k7o9UJNN7MOHo4tuVjkU0GYZRRPr1IFQ1DlwwQraUHvl0M2VL2jdSmBdhGEYRyWei3B9E5FbgXnrPpH6laFaVCqE68AX6X060bqqr0zgKAgHmRRiGUTTyEYj3e9sVaWVK75nV5YmIC3lt62dM3hdwM5tHw4MAm11tGEbRGGgMwgfcpqr3jZA9pUdltH+BADcOsX/jyNiTDfMiDMMoAgONQSSAL46QLaVJqBZ8AzS80VnQtH30VpSzsQjDMIpAPmGuvxGRr4jIDBEZl3wV3bJSYqB1IiKzXCN9cOvI2JMNmxdhGEaByWcMIjnn4QtpZQocVnhzSpRwxOVNykV6JNO4UXpbbCzCMIwCk0821zkjYUhJE6oBf0XuLqT6GYCMXiRTEhuLMAyjgOSTzbVKRP6viKz0jueKyF/lcd2dXgbYtTnOny4iB0Vktfe6bvDmjyD9zYkIhKB2yuhFMiWxsQjDMApIvtlcu+gJd90K3JjHdf8FnDtAnd+r6iLvtWKAuqNLeIBxiOis0fcgwMYiDMMoGPkIxOGq+i2gG0BV28ljWTdVfRbYPzzzSoiKKpfaOxeRWS4HU2KQ6zsUGvMiDMMoEPkIRJeIVOKlJBWRw4HOAj3/fSLymog8JiLzclUSkctF5GUReXnPnn4Gi4tNf15EdJa37sOOkbMnF+ZFGIZRAPIRiOuBx4EZInI38BTwTwV49ivALFVdCNwCPJyroqquVNWlqrp0woRRXIqiv3GI0czJlIl5EYZhFIABBUJVfwNcBCwH7gGWquozw32wqjapaou3/ygQFJHxw71vUQmGIViV/VzEW360FMYhwLwIwzCGTT7zIFDVfcCvCvlgEZkM7FJVFZETcWJV+gsRhSM9az+kU1EDVeOLJxCqLjdU3vVtXoRhGMMjL4EYCiJyD3A6MF5EtuK6qoIAqno7buGhq0QkBrQDy1S1CEuvFZjKKDRvz34uOiv/LiZV6Gpx6020N7ptR2Pu4+52OPkqmH9R/rbavAjDMIZB0QRCVS8Z4PytwK3Fen7RCFRAsBq6W/uei8yCtx9zy412pDX0uQRAc0Q8hepceo/KKETnwNQoHHgXnr8VJhwFk3KO5/dGE9CyG+qnDf3vNQzjkGVAgfCilraqaqeInA4sAH6sqo3FNq5kqYxmF4hxc9wv/Yev7F0eCLtrKiNQPRHGH+n2w9EeIUhuw/UuhXgmXS3ws8/BUyvgojvcanf50LYXaiaaF2EYxqDJx4P4GbBURI4AfgQ8Avwv8OFiGlbSVEagKUtivrkfcoPYwcqeRj8cccfDpaIGzroeHvki/O7f4Jwb8xuTMC/CMIwhkk+Ya0JVY8CFwM2q+mVgSnHNKnH8Qaio7VseCMMRZ8Gs98PEY1z6jUKIQ5KJR8NJV8DmP8DrD+Z/Xdtei2gyDGPQ5CMQ3SJyCXAp8EuvzPor8lmvuhjMvxhmnQJ/ug32vJnfNUkvwjAMYxDkIxCXAe8Dvqmq74rIHOAnxTVrDBCuJ4+MI4VHBE77Z6gcB0+ucGMT+WBehGEYgySfiXJvqOqXVPUe7/hdVb2p+KaVOP6AW21uNAjXwVnXQctOePY/XMjsQJgXYRjGIMkn3fdficirIrJfRJpEpFlEmkbCuJJntLqZACbPhxM+C+88Det+kd815kUYhjEI8uliuhk3/tCgqnWqWquqecZYljnhepB83sIisXAZTD8Bnr8F9m0YuL55EYZhDIJ8WrctwNoxMct5pPH53aS20UJ8cMbXnA1P3pA9BUgm5kUYhpEn+QjEPwGPishXReQfkq9iGzZmqBxgIaGiPz8KZ/5/0LQNfv/dgccjzIswDCNP8hGIbwJtQBioTXsZAKF6EP/o2jB1ESy5FDb8Bt5+fOD65kUYhpEH+cykHqeq5xTdkrGKz+fGItpHefG8xZ+GHavhuZvdJL3o7Nx1bXa1YRh5kI8H8aSImED0x2h3M4EbDznz/7qlUZ/8OsQ6+q9vXoRhGAOQj0B8AXhcRNotzDUHobrsCfZGmqoGOONfXMrxP97Sf10bizAMYwD6FQgREWCeqvpUtdLCXHMg4s2sLgGmL4VFn4I3fwUbnuy/rnkRhmH0Q78C4YW2PjRCtoxtRnPSXCZLl8Ok+fD7/4DGLbnrmRdhGEY/5NPF9CcROaHolox1QrXgK5Echr6AS8XhC8JTN0CsM3fdtr3QtN0JRdt+6GiCrjbnWdjUF8M4pMmn4/wM4AoR2Qy04jLUqaouKKplY5HKCLTuGW0rHDUT4fRr4YmvucyvH7g6e73k2tW58AW8V9ANhPuD/RyP4qxywzAKTj4CcV7RrSgXKqOlIxDg1qVY8HFYc5+bK3HY6YO/RyLmXgwQFQVuPog/CP4KJxr+Cu846ATEX+GSHBqGMSYY8NuqqptHwpCyoKLaNYLxrtG2pIcTPgc7/gy/+3e31Gnd1OI9S+MQi/cfYiu+3mLhr/COgz3iYsujGkZJYD/nCk10DnQ2QWeLW7daE6Nrjz/oxiMe/Kxbz/r8W0a3AdYExDvdKyeS5nV4L8SJS6+XZGyzvJDBdX0l4u6lCSd4ibi39Y41kVaWfpxRX3zOo/IFXFecz9vvtyxQWt10qt77Ect4L9L3Y2nvQ8xdk3zvfd57ID73t6beE7/3f5Z+Ln1/FNZZKVdUez6jQ8AEotBUVLlXLe4/p6vVe7W412gIRt0Ut8jQb66DF1fC+74w8jYMCnVeWLwLChWFmykayX3NEIRCoQnviznYP0DSBCTg2eo1mKmGM1mWXj7IMtXeDX8i1tOQJMtH7ceNZBGUPISz36CKfAMuMt/rzNPp5dnqSkZdyb3f5/+zn/3UZzT9lSzTtM9bRr28/+7smEAUExEI1bgXk9x/ZHebE4zOZrctZKPUH3NOhXkXwp/vhymLYPYpI/PcUiH1hSl11InKoIWlnPDEyxh1TCBGEhE3TlFR7aKMwIWUdrVClycYxfxinHQl7FwLv7sJxv8QaiYV71mGYYx5SqjD8xClogpqJsC4w2DycTDhaKifAeFI4edVBEJw9vVOhJ76hv1KMwyjX0wgSo1gJVSPh3Fz3LKiE46B+plOMApB/XT44Fdg11p4+c7C3NMwjLLEBKLUCYahusEJRsMRLgx0uBxxFhz9V7D6f2HLi8O/n2EYZUnRBEJE7hSR3SKyNsd5EZHvicgGEVkjIkuKZUvZEKp1XVCV44Z/r/d/0YXkPv3/oHXv8O9nGEbZUcxB6v8CbgV+nOP8ecBc73UScJu3NfrD54foLJc99uCWoY8jBMJuPOKhK+GeZRCscmXBsLethECld1yZvTxY6cqS5aEa1xUWqs0vLNEwjJKmaAJlD1lAAAAfjUlEQVShqs+KyOx+qlwA/NjLGPsnEYmIyBRV3VEsm8qKyoiLhmp8z03MGwrR2fDhb8Hm56G73c2AjnX07Lfvh6b08o78wi/FW2UvXO8EozLituF6t18Z7X0uVOeEzzCMkmI0w1ynAem5qLd6ZX0EQkQuBy4HmDlz5ogYNybwB6HhcGjdB01bhxbnP3mBe+VLItYjIN0dEGt32+52F6rb3ggdjdB+0G07GmHfRrftbM5xU3FeR0pIIlCZJiAV1YDPm2ckPd6JpO2nJr95+8lJRpkTxrJuk9eQVp5xn5Spklae5brkvr+ix7MKVpbGglKGMUhG81Obbapi1ml/qroSWAmwdOlSy0GdSXWD695pfM/N1i4mvoBrzEO1g782EYOOg+6VEpJGr6yxp+zge7Cz0XlGY2JyWx74Aj1dcr22Gd14qW4779gX6D0LPDXDOPnyeykt0o5TaS7SjjNTX0DajFv1ZuVqltm6uc73cw1K7xm+2v85st1P+9bP1jwMJiV91rqZZdr/6fSCPvfLYUu67dmuSZVpr03Pccb72yu1S+as6hwzqoc4IXc0BWIrMCPteDqwfZRsGfsEQjB+LjTvguYdDHeKfVHwBdyyqFUN+dVPxHsmEGr6F0wHvdWE0hmL094Vo707RjyeQHp9OXH1UVDcORRRRUTwCfhQRMDnOQ8+z7nwkb5VBPBrN75YJ754e5rH1d7b64q1u2685o7edWx+ShruTdZ0z7HnTMZO1rMDlw+U+6nP+RypNbLePouH2ee+/aTsSHm0GaI/0A8Ff0Xfa4bAaArEI8AXReSnuMHpgzb+UABqJ0G4zq1LHWsfbWuGh8/f0+U0BBIJaO2O0doZo60zTtynUIAo4cHgEwj4ffhFCPiFgM+H3ycEvJffLwTE1ztHX7Ibr7u955dhtl+KiTx/OSYyjpPimTXBYTKJXq4EiAJ4DU9q6+VO6pVQMSORYp860vt+3rEidCWErrjSEUvQFYfO7jjxHJ5CqokVEMTrVfS2yfMibj+17WmLk+Xp90q1zd5Oet3MB0tGo99jT88903szxbsm2Tvpk55niNdt6d4R6WVnEs1wRHp8D+113MvRSR08xmApmkCIyD3A6cB4EdkKXA8EAVT1duBR4MPABqANuKxYthxyBCthwlHOk+hvMaAyJJZQ2jrjtHY5YRhtPyqh0BXzusn6Gd/PFBK/r4KAL+yExBOWgE/KKtFpPAFdsbgTglicrliczlhiUP9nqQZSvSNNLzWGSzGjmC4Z4LwCpZ5WdOwi4tZ+CNW5sYl+02uPbbrjSkunE4SO7viYbB7yERIB53F4YpESDr/nkfh9BH2lqSDdcaUjFqc7lqDTe3XHR298KZZQmruUpk4lloCADyr8QsAHQR8EfULQ78p95aTKg8RCK8qdUI2bXNe0Fdr2jbY1BaMzlqA1KQqx0WtoMlFVDnQofh/UVkhBGxcFYnElFs894Ch4nohPCKaJid8vBNPEpFiouv+brliCTs8j6IzFSRRJtVWVjjg0dbrGvqmr77a5U2nqSnAwrU7bIJLl+sUTDT8EfELQ57YVnoAEfWnC4k+e7ykP+Hqu83vlfnH3c/cWrxyv67HnmoAXmBdXJ2rxBMQUYgmIJ9Qr945VvfKe+jFvf6habAJxKODzQWSmm3vQuGXMppJu74rT0hWjtTM+qr8+4wlld5uyvSXB9uYE21sS7GhJsK0lwc6WBB1e++0XiIbFe/kYFxailcK4sI9oWHodVwZ6+q2HgwLd8QTdcejoxxPJJVw6gP+VT9BQIbQgocreNmVHq3tvd7YkaMwhAt39fBSqg1BXIdSFhPqQMKPWT21IqKtwx3Uh1wh3x12D2p3AveKucXXH6p339nOc74hBc1eizz3i6urEvfJiiWUmPnGfwaTgDAUTiDInnlAS6l7qryEROQJt3Ip2HOiJHqRnIM1F6wg+X8+Anw9JRewMB1X35VDU23dbVSWBui+O0nMOd647nqC1M/dAZTHojCk7W12jv6MlkSYGyq7WBPE0Uyr8MLXGx5QaH8dPDjClxocq7O9IcKBd2d+hHOhIsLHReRfZGoiw3xOTyjTx8IQkXWQiYaEyMLz/CIURfS9z0RFTdrQkUiLg3mdlZ0uCna2JXg2/T3oa+roKYUqNj6OSx15Zr21IqK0orrc0VBJpv/S7vV/+3fEeIUl6BEkPQNV5Hn4fBMQJWvq+zwcBcR6H39v3Z+kaO/Ibg7fVBKKESCSUuGqqUY8l1JV55cmGM57o2U+kGtqe/USip/HNzgT8sQDB1u2Dio9Oftx8Ii6CTsQl85KerpSEa/FJQE/jnxSCob81RaG1W1MewPY0EdjRkmBPe29rq4NOBOZGfZw6I8C0WicI02p8jKvMvyspoe5X74EO99rfkXDb9p7jbc0J/rzb/ULORjgA0VCPgETSBCQpJpFwYT2ToZDsbkt6WE4M3PHOlgT7O3r/fVXeezy73sf7pjmhTQrvxCrBX+DGXoBQwI/PJ70+t4lEcT+zPnHdU/ihMmVJaWICUWBi8YTXV+ga9UQiraFPNv7er4Z4IuH6C9M+kCNFPBwlEawm2LIVX3d+k+uS5sVVIZ5eUvq0dyvrD8R5a3+ct73X9pbe9kfD7pfpokmBVOM/pVaYVuOjtkIK0tD6xDXekTDMGaBud1w5mBKTREpU0o+3tSRYu9eJTrb/jZDnmUSS4hFyAhIOSk/QD/RpDPse99w/PVAoOQUs+dntjMPOtG6hjrTfHwKMrxKm1vg4cYp7j3tEQAr2HmciQEXA1/vld6+BSHq9CdyPsHTPN+69SakfaWmecSKhvd6X5PuZeh+9f9LfP/X+Q5L76f8HyXBZnxcr60uG7JJxnB5Omx7eO8S31QSigOxp7mRXU8eINvTDQf0VdNUfNugxCcn5BypoAvEWtBcv7l68GHxRrzyRvbxQgtMVV95pTPDW/jhv7Yuz/kCC95p6BkonVPk4qqGCvzgiyIxaH1OrhSnVUBUsrf+4oF8YXyWMrwLof6JTPKE0pnkmjVkEZWdLgnV7nWcy3H7wVHx/xnHAB5OrXaN//OSAt++Ed1K1jwp/cX8tB/0+Quki4G2H2kCK14/v75lVcUhhAlEAYvEEWw+009wxRmfADnLluoHaliG3Pb0Eo0do3B2TgyBuUlVyZm08obzX2MHbu9tYv7uV9Xta2byvjZjXAtZXBpk7sY73HVnD3Em1HDGxhmhV9tlyHV66h57nx1JC1lv0Ymkil1aXBEG/j5pQAJ8P4nHnNcbiXpehd1wM/D6hoVJoqBy4btwbaM3VyPeaz5tlQlkpEPAJFQE/FQEh5PcT9La+IQ7GFofkZzbXtr869BwDPVkBGCBrAAOcGxwmEMOktTPGlgNtdMdK69fnmMSbhavexzLzHU2osqOxg/W7m1m/u4X1u5rZuLc1NX+gusLPERNruGDRNOZOrGHupBom1ITyb9hEQAI5n5+LioCPusog9eEAlQF6z1bOTPmhCWLxhGuk43EScTcfIJ5IEIvFXddkIk4skSAed90azphs1vSUSXrjkDqdpczbD6qmeXBpW9JyJhUVAfGjPr/bSgD1+UAC4PMhvgCBYBC/34/fX4E/6KfC7yMc9BMO+Ajk0T00JJugd+Pd6ziPOiUkooXABGIY7G7uYHdT55jpUhoLdMcTHGjrorGtm8a2Lg60dbPjoBOFjbtbaO1yndoVAR+HT6jh3HmTnRhMrGVKJDxik5qCAaG+Mkh9ZZCqisyvUe6vleDSCQSBcB7PSR+7iqeNZ8UyAhmS416JNI9lWJ/LDPHo2Y97YpTu7WnqHKjX4HsNv881+L5AkIDfj88fIBCoIBDwe1E3PZP9UilIitL4G0PBBGIIxOIJthxop2WsdimNML0b/W63394jAI1pgpAUgHQCPmF2QzWnHjmBuRNrOGJiLTPHVRU8qmUg+heF4uDzuTDj4BByrcUTvSPi4mkvTVePHG9jZp6hVHnO+q4bKtnQW4M/9jGBGCStnTHe299GLG5uQ3c8wd6WTnY3d7KnqZP9bV19fv03tnfR2pk9lLa6wk+kqoJIVZDZ46uJVkaIVAWJVFUQ9baRyiDR6gqCo9TIjIYoFAq/r/Chocahxdj6xI8yu5s62N186HQptXXF2N3UyZ6kCDS7v393Uyd7mjs50NbVp6e6qsJPNNnoN1QRrepp9CNVQXeu0h1XDHV6Z5EJ+HtEoTpkXxHj0MU+/XlQrl1KqsrWA+1s3t+Wavz3NDsx2N3c0eeXf8AnTKgNMaE2xJJZESbWhlPHE2tDjKuuIBQYm0uHmigYRl/smzAALZ0xtpRJl1JClc37Wlm7rYm12w/y+vYmDrb3zIGoDPqZ6DX4x0ypY6LX8DsBCBOpCpZVZstgQKgNB4mYKBhGVuxb0Q+7mzrY1TR202THE8q7e1tZu+1gShBaOp0XNKE2xJKZEeZPq+eICTVMrAtTUwaNpAi9M5n6haDflxosrfD3ZDQtpbh+wyhFxn6LUAS64wm27G/LObhaqsTiCTbsaUl5COt2NNHmRQVNqQ/zvsMamD+tjvlT65lYl0+QZekgQmr9g6C/Z+2DgNfgB9PEwDCMwmACkUFzRzdbD7SPiS6l7niCt3c1ex5CE2/ubKLDS4E5PVrJqXMnMG9qHcdNq6ehJjTK1g6MzwehgI9QwE9FoHfKBAuVNIyRxwTCQ1VTETqlSmcszls7ewThrZ3NdHnrIswaV8VZR09i/rR65k2ty5lOYrQRySECAZ/9+jeMEsMEgtLtUkqo8s6eVla9d4BX3zvAWzubiSUUAeZMqObc+ZOdIEypo65ycPmUiokIfTyAkJcqoVRDWw3D6MshLxDNHd1s2d9OfKSWeRqAlo4Yr245wKrNB1j13gEavbURD59QzfkLpzJvaj3HTq0rmQFlEais8FMbDlAZ9Kdlz7QBYMMY65RGKzMKqCq7vAlfo0m6l7Bq8wHe2tlEQqEmFGDxzAjHz4yyZGaUaHXpdBklw0NrQgFqQgGbrWsYZcohKRCqyjt7W2kbpS6lpJfw8uYDvJLhJVx8/AyWzopy5KTakml4RZxg1YSdIISHkhjIMIwxxyEpEPtbu0ZUHFJewub9zkvY1VzyXkIo6KPWE4TqigC+EhErwzBGjkNOIFSVPS3F71Zq7uhm9ZZGXt50gFe29HgJR0yo4WPHz+D4EvMSfD6oDQVTXoINJhuGccgJxIG27qIt7nOwvZvHX9/Jqk37e3kJS2ZGOH5WlMUzoyUVfipCKoFeVYXfBpYNw+jFISUQbq5DR1Hu/eK7+7nl6fU0tnVzxMQaPrZ0BktnRplbQl5CEr9PGF9TwbjqCpuAZhhGTg4pgSiG99DWFeNHz73Lr9/YxeyGKm74yDwOm1BT0GcUilDQx/iaEJHKoI0pGIYxIEUVCBE5F/hPwA/8UFVvyji/HPh3YJtXdKuq/rAYtqhqwUNa1247yM1Pvc3upk4+umQ6nzppZknOBq4O+RlfG6IuXDqT6QzDKH2KJhAi4ge+D3wI2Aq8JCKPqOobGVXvVdUvFsuOJI1t3anF7YdLVyzBT17YzMOvbmNSXZh/veg45k2tL8i9C4UI1FcGGV8TorLCwlINwxg8xfQgTgQ2qOo7ACLyU+ACIFMgik4yz1Ih2Linhe/85m3e29/GufMm85lT5pRUA+zzwbjqChqqQxaJZBjGsCimQEwDtqQdbwVOylLvoyJyKvA28GVV3ZKlzrA42D587yGeUB54ZSv3vPge9eEg13/kWJbOGlcgC4dPMCA0VLtV3UptUNwwjLFJMQUiWyuVOUL8C+AeVe0UkSuB/wbO7HMjkcuBywFmzpw5aEOG6z1sO9DOd598m7d2NfPBueO58tTDSyY5XmWFG3iurwxamKphGAWlmAKxFZiRdjwd2J5eQVX3pR3eAfxbthup6kpgJcDSpUsHFYbU2NZFZ/fQvAdV5dE/7+DOP26iwu/jmnOO4tQjJwzpXoWmNhxgfG2oZJL2GYZRfhSzdXkJmCsic3BRSsuAT6ZXEJEpqrrDOzwfWFdoI4bqPext6eQ/n1rP6i2NLJkZ4Utnzh31RXdEIFpdQUN1heVDMgyj6BRNIFQ1JiJfBJ7Ahbneqaqvi8gK4GVVfQT4koicD8SA/cDyQtpwsK170N6DqvK7t/dw++82Eksonz/9cM6dN3lUu298PhhfE6LBJrYZhjGCiGpprIOQL0uXLtWXX345r7rrdzWnluDMh4Pt3dz2zAb+sHEfR0+u5ctnH8nUSOVQTR02Ik4YxteYMBiGMTxEZJWqLh3MNWXbgX2wvXtQ4vDSpv3c8tv1NHfE+Jv3zeKixdNHLRpIBBpqKphQEzJhMAxj1ChbgdiTZ86lPqkyzp/HnPGjkypDxM1hmFAbKskZ2YZhHFqUpUAcbO+mvWtg7+H17Qf57pNvs6d5dFNlJAefJ5owGIZRQpSlQAzkPfRNlbGAY6fUjZB1PSTTbU+sDdusZ8MwSo6yE4imjv69h+54gusfWcva7U2cN38yl71/5FNlJPMkTawLEQpYuKphGKVJ2QnE7qbc8x5Ule/9dj1rtzfx5bOP5MyjJ46gZSYMhmGMLcpKIJo7umnvyr3W9E9f2sIzb+3h0yfNHHFxiFQFmVAbsgluhmGMGcpKIHb14z0889Zu/vfF9zjz6Il8fOmMnPUKTdJjMGEwDGOsUTYC0Z/38MaOJv7zqfXMn1rHF884YkRmRddVBphUFzZhMAxjzFI2ApEr59KOg+1881dvMLE2xFfPO6boYaQ14QCT68IltUaEYRjGUCgLgWjpjNHW2dd7aOmIccMv3kAVrv/IvKKm6A74hamRSupLJA24YRjGcCkLgdjd1HfeQ3c8wb8+to5dTR3c+Nfzi5pTqaGmgkl1YVuoxzCMsmLMC0RrZ4zWDO9BVfnBMxtYs+0g//ChI4u2XnQ46GNatJKqijH/NhqGYfRhzLds2cYeHli1lSfX7WbZCTM446jCh7OKwMS6EBNqQraKm2EYZcuYFojWzhgtHbFeZb9fv4cf/2kzpx05gU+eOPjlSQeiJhxgaiRsE90Mwyh7xrRAZHoPb+5s4rtPvs0xU+r40plzC/rrPuAXptSHiVRVFOyehmEYpcyYFYi2rt7ew66mDr75q3U0VIf4lw8fU9Dkd9HqIFPqK20Q2jCMQ4oxKxDpOZdaOmPc8Ms36E4k+H8fOa5goaahoI9pkUqqQ2P2bTIMwxgyY7Lla++K0+x5D7F4gn97/E22N7az4vx5zIhWDfv+IjCxNsSEWhuENgzj0GVMCsRub70HVeX2Z99h9ZZG/v7MuSyYHhn2vatDfqZGKi1FhmEYhzxjTiASCk3tznt46NVtPPH6Tj52/HTOPnbSsO7r97lB6Gi1DUIbhmHAGBSIWNwtBvT8xr381x83ccoR4/n0ybOGdc9IVZAp9WECttynYRhGijEnEPGEsn5XM9/+zdscOamWL589F98QxwkqAm4mdI0NQhuGYfRhzLWMsUSCb/zqDSKVQf7lL48Z0oQ1nw/G17iZ0D4LXTUMw8jKmBOI7Y0dzIgluPGvjyM6iElrAb9QGw5QVxmkNhSw6CTDMIwBGHMC0RVPcO25RzNz3MDhrBUBH3WVAerCQZvLYBiGMUjGXKs5oTbE4pnRnOcrK3zUhYPUVQYtVNUwDGMYjDmBqA/3niUtAlUVfuorg9SGgwVNsWEYhnEoU9TWVETOFZG3RGSDiFyb5XxIRO71zr8gIrPzu69b83nGuEqOmVLHYRNqaKgJmTgYhmEUkKJ5ECLiB74PfAjYCrwkIo+o6htp1f4WOKCqR4jIMuDfgE/0d9+KoI9jp9RZ9JFhGEaRKeZP7hOBDar6jqp2AT8FLsiocwHw397+A8BZMkB4kV/ExMEwDGMEKOYYxDRgS9rxVuCkXHVUNSYiB4EGYG96JRG5HLjcO2wRkbeKYjGMz3x2iTPW7IWxZ/NYsxfM5pFgrNkLcNRgLyimQGT7ma9DqIOqrgRWFsKo/hCRl1V1abGfUyjGmr0w9mwea/aC2TwSjDV7wdk82GuK2cW0FZiRdjwd2J6rjogEgHpgfxFtMgzDMPKkmALxEjBXROaISAWwDHgko84jwKXe/sXAb1W1jwdhGIZhjDxF62LyxhS+CDwB+IE7VfV1EVkBvKyqjwA/Av5HRDbgPIdlxbInT4rejVVgxpq9MPZsHmv2gtk8Eow1e2EINov9YDcMwzCyYTPLDMMwjKyYQBiGYRhZOSQFQkTuFJHdIrI2rWyciPxGRNZ729wZAUcBEZkhIk+LyDoReV1E/t4rL0m7RSQsIi+KyGuevTd45XO8tCrrvTQrJbfGq4j4ReRVEfmld1zSNovIJhH5s4isToYylurnAkBEIiLygIi86X2e31fi9h7lvbfJV5OIXF3iNn/Z+96tFZF7vO/joD/Hh6RAAP8FnJtRdi3wlKrOBZ7yjkuJGPCPqnoMcDLwBRE5ltK1uxM4U1UXAouAc0XkZFw6le969h7ApVspNf4eWJd2PBZsPkNVF6XF5pfq5wLgP4HHVfVoYCHuvS5Ze1X1Le+9XQQcD7QBD1GiNovINOBLwFJVnY8LEkqmMhrc51hVD8kXMBtYm3b8FjDF258CvDXaNg5g/89xea5K3m6gCngFN5N+LxDwyt8HPDHa9mXYOh33ZT8T+CVuMmep27wJGJ9RVpKfC6AOeBcvQKbU7c1i/znAH0rZZnoyVIzDRar+EviLoXyOD1UPIhuTVHUHgLedOMr25MTLersYeIESttvrqlkN7AZ+A2wEGlU15lXZivswlxI3A/8EJLzjBkrfZgV+LSKrvLQ0ULqfi8OAPcBdXjfeD0WkmtK1N5NlwD3efknarKrbgG8D7wE7gIPAKobwOTaBGGOISA3wM+BqVW0abXv6Q1Xj6tzy6bjkjcdkqzayVuVGRP4K2K2qq9KLs1QtGZs9TlHVJcB5uK7HU0fboH4IAEuA21R1MdBKiXTNDITXZ38+cP9o29If3ljIBcAcYCpQjftsZDLg59gEooddIjIFwNvuHmV7+iAiQZw43K2qD3rFJW+3qjYCz+DGTiJeWhXInn5lNDkFOF9ENuGyD5+J8yhK2WZUdbu33Y3rGz+R0v1cbAW2quoL3vEDOMEoVXvTOQ94RVV3ecelavPZwLuqukdVu4EHgfczhM+xCUQP6Wk/LsX18ZcMIiK4mefrVPU7aadK0m4RmSAiEW+/EvehXQc8jUurAiVkL4CqflVVp6vqbFxXwm9V9VOUsM0iUi0itcl9XB/5Wkr0c6GqO4EtIpLMLHoW8AYlam8Gl9DTvQSla/N7wMkiUuW1G8n3ePCf49EeUBmlQZx7cH1z3bhfNH+L62t+CljvbceNtp0ZNn8A5xKuAVZ7rw+Xqt3AAuBVz961wHVe+WHAi8AGnKseGm1bc9h/OvDLUrfZs+017/U68C9eeUl+LjzbFgEve5+Nh4FoKdvr2VwF7APq08pK1mbgBuBN77v3P0BoKJ9jS7VhGIZhZMW6mAzDMIysmEAYhmEYWTGBMAzDMLJiAmEYhmFkxQTCMAzDyIoJhDEmEJGGtGyaO0VkW9pxXtlVReSutPj7XHW+ICKfKpDNF3j2vSYib4jIZweof6aX0DDbuSki8mjavR7xymeIyL2FsNcwMrEwV2PMISJfB1pU9dsZ5YL7TCeyXjiCiEgIl5Ruqapu945nqerb/VxzI7BXVW/Ocu5HuFm83/eOF6jqmiKZbxiAeRDGGEdEjvBy3t+Oyxg7RURWisjLXj7869LqPicii0QkICKNInKT94v8eRGZ6NW5UUSuTqt/k7h1Ld4Skfd75dUi8jPv2nu8Zy3KMK0el8dpP4CqdibFQUQmiciD3nUvisjJInI48FngGs/reH/G/abgJnXi3W9N2t+/2tu/K82r2isi/+KVX+s9Z036+2EYA2ECYZQDxwI/UtXF6jJZXqtuXYSFwIe8dTMyqQd+p269iueBz+S4t6jqicA1QLJx/Ttgp3ftTbjMur1QlxfpCWCziPyviFwiIsnv2/eAb3k2fhz4oapuBH4I/Lu6tQf+mHHLW4H/FpHfisjXkjmAMp55mbrkiBfiUjv/WEQ+DMzEpVpfBLw/i/gYRlZMIIxyYKOqvpR2fImIvILzKI7BCUgm7ar6mLe/Crc+SDYezFLnA7hkfqhqMsVFH1R1OW7NjpdxGUtXeqfOBm73fvk/DES9fFU5UdVHgcNx+biOBV4VkYbMet597geuUtUtuNxM5+HSnrwCHAEc2d+zDCNJYOAqhlHytCZ3RGQubkW4E1W1UUR+AoSzXNOVth8n93ehM0udbCnAs+J1Ba0Rkf/FJSv8rHf9iaqabgNuCKXfe+0D7gbuFpHHcUKVKU53AD9V1afTbL1RVX+Ur82GkcQ8CKPcqAOagSavG+YvivCM53BdQ4jIcWTxUESkTnqvy7AI2OztPwl8Ia1ucvyiGajN9kAROSvpZYhIHS7X/3sZdf4eCGYM3j8B/K2X6RURmS4i4/P8O41DHPMgjHLjFVxq47XAO8AfivCMW3D9+2u8563FrdqVjgBfFZE7gHaghZ5xji8At4nIZbjv4NNe2c+B+0XkIuALGeMQJwC3ikg37ofdbar6qogckVbnK0BbctAauFVVfygiRwN/8jyUZuCTuDEKw+gXC3M1jEEibtGVgKp2eF1avwbmas9yjoZRFpgHYRiDpwZ4yhMKAa4wcTDKEfMgDMMwjKzYILVhGIaRFRMIwzAMIysmEIZhGEZWTCAMwzCMrJhAGIZhGFn5/wFvzp52Q8+IIgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c60621d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that by adding more model complexity, we've managed to lower the level of convergence to an rms error of 1.0!\n",
    "\n",
    "What if we get even more complex?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8nGW5+P/PNUv2pEmbFtKNtlC27iv7jhxwQxahqEeKC4h4FH9HzlHP9wfK0d9Rjwc5isKvKBwXZFURPQrKoogg0Ja2FsrSFrovSdvsyazX94/7mWSSTpJJmyczyVzv12teM/PMMzNXkslzzf3c933doqoYY4wxvQVyHYAxxpj8ZAnCGGNMRpYgjDHGZGQJwhhjTEaWIIwxxmRkCcIYY0xGviUIESkRkZdEZK2IvCoiX82wT7GIPCgiG0XkRRGZ5lc8xhhjBsfPFkQEOFdV5wHzgQtF5ORe+3wcOKCqxwDfAb7pYzzGGGMGwbcEoU6rdzfsXXrPyrsY+LF3+xHgPBERv2IyxhiTvZCfLy4iQWAVcAzwfVV9sdcuk4BtAKoaF5EmYBzQ0Ot1rgWuBSgvL190/PHH+xl2Ydm7AcIlUDM915EYY3y0atWqBlUdP5jn+JogVDUBzBeRauBXIjJbVden7ZKptXBQ7Q9VXQGsAFi8eLGuXLnSl3gL0vcWw4Tj4cqf5ToSY4yPRGTLYJ8zLKOYVLUR+BNwYa+HtgNTAEQkBIwB9g9HTMYYY/rn5yim8V7LAREpBc4HXu+122PA1d7ty4Gn1aoHDj/7lRtjMvDzFFMd8GOvHyIAPKSqvxWRW4GVqvoY8CPgpyKyEddyWOZjPCYTGxNgjOmDbwlCVdcBCzJsvzntdifwQb9iMMYcLBaLsX37djo7O3MdivFBSUkJkydPJhwOH/Zr+dpJbYzJP9u3b6eyspJp06Zho8pHF1Vl3759bN++nenTD39kopXaKHh2gCg0nZ2djBs3zpLDKCQijBs3bshah5YgjClAlhxGr6H821qCMMYYk5ElCGPDXM2wamxs5Ac/+MEhPffd7343jY2N/e5z88038+STTx7S65ueLEEUOjvVYIZZfwkikUj0+9zf/e53VFdX97vPrbfeyvnnn3/I8flloJ8tH1mCMMYMqy9+8Yts2rSJ+fPnc9NNN/GnP/2Jc845hw996EPMmTMHgA984AMsWrSIWbNmsWLFiq7nTps2jYaGBt555x1OOOEEPvnJTzJr1iwuuOACOjo6AFi+fDmPPPJI1/633HILCxcuZM6cObz+upurW19fz7ve9S4WLlzIddddx1FHHUVDQ48ScCQSCZYvX87s2bOZM2cO3/nOdwDYuHEj559/PvPmzWPhwoVs2rQJVeWmm27q2vfBBx8EyPiz/exnP2Pp0qXMnz+f6667jkQi0ed75ZoNczVkKH9lCsRXf/Mqr+1sHtLXPHFiFbe8b1afj3/jG99g/fr1rFmzBnAH0Zdeeon169d3Dc285557GDt2LB0dHSxZsoTLLruMcePG9Xidt956i/vvv5+7776bK664gl/84hd85CMfOej9amtrWb16NT/4wQ/49re/zQ9/+EO++tWvcu655/KlL32Jxx9/vEcSSlmzZg07duxg/XpXPi51auvDH/4wX/ziF7nkkkvo7OwkmUzyy1/+kjVr1rB27VoaGhpYsmQJZ555JkCPn23Dhg08+OCD/PWvfyUcDvPpT3+a++67j1mzZmV8r1yzFkTBs1NMJveWLl3aY9z+d7/7XebNm8fJJ5/Mtm3beOuttw56zvTp05k/fz4AixYt4p133sn42pdeeulB+zz33HMsW+YKN1x44YXU1NQc9LwZM2awefNm/umf/onHH3+cqqoqWlpa2LFjB5dccgngJqWVlZXx3HPPcdVVVxEMBjniiCM466yzePnllw/62Z566ilWrVrFkiVLmD9/Pk899RSbN2/O+F75wFoQxhSw/r7pD6fy8vKu23/605948skneeGFFygrK+Pss8/OOK6/uLi463YwGOw6xdTXfsFgkHg8DrgJZQOpqalh7dq1PPHEE3z/+9/noYce4vbbb8+4b3+vl/6zqSpXX301//Ef/3HQfr3f65577hkwRr9ZC8IYM6wqKytpaWnp8/GmpiZqamooKyvj9ddf529/+9uQx3D66afz0EMPAfCHP/yBAwcOHLRPQ0MDyWSSyy67jH//939n9erVVFVVMXnyZB599FEAIpEI7e3tnHnmmTz44IMkEgnq6+t59tlnWbp06UGved555/HII4+wd+9eAPbv38+WLVsyvlc+sBaEsWGuZliNGzeO0047jdmzZ3PRRRfxnve8p8fjF154IXfddRdz587luOOO4+STe69UfPhuueUWrrrqKh588EHOOuss6urqqKys7LHPjh07uOaaa0gmkwBd3/p/+tOfct1113HzzTcTDod5+OGHueSSS3jhhReYN28eIsK3vvUtjjzyyK5O8ZQTTzyRr33ta1xwwQUkk0nC4TDf//73KS0tzfheuSYjrbq2LRg0xH5wClQfBR96INeRmGGyYcMGTjjhhFyHkVORSIRgMEgoFOKFF17g+uuv7+o0Hw0y/Y1FZJWqLh7M61gLwhhTcLZu3coVV1xBMpmkqKiIu+++O9ch5SVLEAZI5joAY4bVzJkzeeWVV3IdRt6zTupCZzOpjTF9sARhjDEmI0sQxhhjMrIEYazShjEmI0sQBc/6IEz+q6ioAGDnzp1cfvnlGfc5++yzGWgI/O233057e3vX/WzKhxcySxAGdOSVITaFaeLEiV2VWg9F7wSRTfnwXMiX0uCWIAwk4rmOwBSQf/3Xf+2xHsRXvvIV/uu//ovW1lbOO++8rtLcv/71rw967jvvvMPs2bMB6OjoYNmyZcydO5crr7yyRy2m66+/nsWLFzNr1ixuueUWwBUA3LlzJ+eccw7nnHMO0F0+HOC2225j9uzZzJ49u6vmUn9lxdM9/PDDzJ49m3nz5nVVcU0kEnzhC19gzpw5zJ07l+9973uAK9i3YMEC5syZw8c+9jEikUhXLLfeeiunn346Dz/8MJs2beLCCy9k0aJFnHHGGV2zsjO9l19sHkTBE1BLEAXr91+E3X8f2tc8cg5c9I0+H162bBk33ngjn/70pwF46KGHePzxxykpKeFXv/oVVVVVNDQ0cPLJJ/P+97+/zzWW77zzTsrKyli3bh3r1q1j4cKFXY99/etfZ+zYsSQSCc477zzWrVvHZz/7WW677TaeeeYZamtre7zWqlWruPfee3nxxRdRVU466STOOussampqsiorfuutt/LEE08wadKkrlNWK1as4O233+aVV14hFAqxf/9+Ojs7Wb58OU899RTHHnssH/3oR7nzzju58cYbAVcd9rnnngNc3aa77rqLmTNn8uKLL/LpT3+ap59+OuN7+cVaEAaS+dGcNYVhwYIF7N27l507d7J27VpqamqYOnUqqsqXv/xl5s6dy/nnn8+OHTvYs2dPn6/z7LPPdh2o586dy9y5c7see+ihh1i4cCELFizg1Vdf5bXXXus3pueee45LLrmE8vJyKioquPTSS/nLX/4CZFdW/LTTTmP58uXcfffdXaeHnnzyST71qU8RCrnv4WPHjuWNN95g+vTpHHvssQBcffXVPPvss12vc+WVVwLQ2trK888/zwc/+MGuhYV27drV53v5xVoQhU5wxfpinRAuyXU0Zrj1803fT5dffjmPPPIIu3fv7lqX4b777qO+vp5Vq1YRDoeZNm1axjLf6TK1Lt5++22+/e1v8/LLL1NTU8Py5csHfJ3+atJlU1b8rrvu4sUXX+R///d/mT9/PmvWrEFVD4pvoNp3qdLgyWSS6urqjPWhMr1X78WUhoq1IAygEG3NdRCmgCxbtowHHniARx55pGtUUlNTExMmTCAcDvPMM8+wZcuWfl/jzDPP5L777gNg/fr1rFu3DoDm5mbKy8sZM2YMe/bs4fe//33Xc/oqNX7mmWfy6KOP0t7eTltbG7/61a8444wzsv55Nm3axEknncStt95KbW0t27Zt44ILLuCuu+7qWoNi//79HH/88bzzzjts3LgRcJVhzzrrrINer6qqiunTp/Pwww8DLrGsXbu2z/fyi7UgCp73DSfSAuW1/e9qzBCZNWsWLS0tTJo0ibq6OsAt5fm+972PxYsXM3/+fI4//vh+X+P666/nmmuuYe7cucyfP79r/YV58+axYMECZs2axYwZMzjttNO6nnPttddy0UUXUVdXxzPPPNO1feHChSxfvrzrNT7xiU+wYMGCPlep6+2mm27irbfeQlU577zzmDdvHrNnz+bNN99k7ty5hMNhPvnJT/KZz3yGe++9lw9+8IPE43GWLFnCpz71qYyved9993H99dfzta99jVgsxrJly5g3b17G9/KLb+W+RWQK8BPgSFw1uBWq+t+99jkb+DXwtrfpl6p6a3+va+W+h9hdZ0C4FN5zGxw5O9fRmGFg5b5Hv5FQ7jsO/LOqrhaRSmCViPxRVXv3Fv1FVd/rYxxmQArJmPVDGGN68K0PQlV3qepq73YLsAGY5Nf7mUOU3okWbctdHMaYvDMsndQiMg1YALyY4eFTRGStiPxeRPJjBfVCFe17nWAzuoy0lSRN9obyb+t7ghCRCuAXwI2q2tzr4dXAUao6D/ge8Ggfr3GtiKwUkZX19fX+BlyIUh8oa0EUhJKSEvbt22dJYhRSVfbt20dJydCcKvZ1FJOIhHHJ4T5V/WXvx9MThqr+TkR+ICK1qtrQa78VwApwndR+xlzQElGIRyBUPPC+ZsSaPHky27dvx75sjU4lJSVMnjx5SF7LtwQhbobIj4ANqnpbH/scCexRVRWRpbgWzT6/YjKZ9JpoFG2zBDHKhcNhpk+fnuswzAjgZwviNOAfgb+LSGo64JeBqQCqehdwOXC9iMSBDmCZWrs3tyItUDY211EYY/KAbwlCVZ9jgMUGVPUO4A6/YjCHwGZUG2M8Vmqj0PWuZZOIQjyam1iMMXnFEoQ5mLUijDFYgjDQPcw1xRKEMQZLECZTN1HEEoQxxhKEySQRgUQs11EYY3LMEoTJLGJlN4wpdJYgDJBh6omV3TCm4FmCKHR9LAhvHdXGGEsQJrN4p/VDGFPgLEEYMp5iAmtFGFPgLEEUvH6qodhwV2MKmiUI0zfrqDamoFmCMH2Ld0AinusojDE5YgnC9NkFAVg/hDEFzBJEoetrmGuKJQhjCpYlCNM/66g2pmBZgjD0e44p3gHJxPCFYozJG5YgCt4Ap5jA6jIZU6AsQZiB2XBXYwqSJQgzMOuoNqYgWYIwB68o11vM+iGMKUSWIArdQMNcAVBrRRhTgCxBmOxYP4QxBccShKH/qdQemw9hTMGxBFHwsjnFBMTaIZn0NxRjTF6xBFHosswP1g9hTOGxBGGyZwnCmIJiCcJkzzqqjSkoviUIEZkiIs+IyAYReVVEPpdhHxGR74rIRhFZJyIL/YrH9CXrc0wuQVg/hDEFw88WRBz4Z1U9ATgZuEFETuy1z0XATO9yLXCnj/GYw6YQs1aEMYXCtwShqrtUdbV3uwXYAEzqtdvFwE/U+RtQLSJ1fsVk+jDQTOp0NtzVmIIxLH0QIjINWAC82OuhScC2tPvbOTiJICLXishKEVlZX1/vV5iFKauZ1GmsH8KYguF7ghCRCuAXwI2q2tz74QxPOejrrKquUNXFqrp4/PjxfoRpshVrG1yLwxgzYvmaIEQkjEsO96nqLzPssh2YknZ/MrDTz5hMJoM44GvSWhHGFAg/RzEJ8CNgg6re1sdujwEf9UYznQw0qeouv2IymQzyFBPYfAhjCkTIx9c+DfhH4O8issbb9mVgKoCq3gX8Dng3sBFoB67xMR6T0SEkiEgrVA59JMaY/OJbglDV5xjg6KOqCtzgVwzGJ6l+iMF2cBtjRhSbSV3ogiHobBpcx7P1QxhTECxBFLrj3g2NW2HXmoH3TWcJwphRzxJEoZvzQSitgbX3D+551lFtzKhnCaLQhUpg9mWw7SXYtyn750VbbT6EMaOcJQgDJ17sEsXaB7J/jibdIkLGmFHLEoSB4ko44X2w6Slo3ZP986wfwphRzRKEceZcDgj8/ZHsnxNp8S0cY0zuWYIodKm5DBVHwDHnwYbfZH/gj1pdJmNGs34ThIgEReQ/hysYk2PzlkG8E177dXb7awJiHf7GZIzJmX4ThKomgEVeXSUz2o2dAVNOgvW/gHgku+fYcFdjRq1sTjG9AvxaRP5RRC5NXfwOzOTIvGXQcQDe+kN2+1uCMGbUyqYW01hgH3Bu2jYFMpXvNiNOr8Zh3XwYfxyse9DNsg4E+3+6rTBnzKg1YIJQVauwWkhEYN5V8ORXYMtfYfqZ/e+f6ocIlw5LeMaY4TPgKSYRmSwivxKRvSKyR0R+ISKThyM4kyPTzoDKibDm/uxGKVkrwphRKZs+iHtxC/tMxK0X/RtvmxmtAkGYdyXUb4Dd6wbePzpC50PYCCxj+pVNghivqveqaty7/A9gC0OPdsdeCCXV2RXxG4kzqpt3Qv0bEOvMdSTG5K1sEkSDiHzEmxMRFJGP4DqtzWjQ1wjmUDHMvhS2/g32b+7/NZLxkfVtvHGbV1JEoWlbrqMxJm9lkyA+BlwB7AZ2AZd728xolyrit+7BgfcdCa0IVdj/NrQ3dG+LtkJbQ9/PMaaADTiTGrhMVd+vquNVdYKqfkBVtwxTfCaXSsbA8e+Bt56E1r3975vvdZmSSdcS6mw8+LHmnRCPDn9MxuS5bGZSXzxMsZh8NOdyQN3s6v5EWvK3FZFMwL6NEGnO/Lgm7FSTMRlkc4rpryJyh4icISILUxffIzP5obIOjj534CJ+moCGt6B5V34V8EvEXFyxAZJXpBna9w9PTMaMENnMpD7Vu741bZvSc2a1GbGyKLM190rY+KRLEvM/1M+OCq273Xn96qmuozuX4hG3Sl4iy7pSzTuguAqC2fxbGDP69fufICIB4E5VfWiY4jH5qHYmTF7i1oqYczkEi/rfP9rqhpCOmQxlY4cnxt5iHS45JGPZPycZh+btUDPNt7CMGUkG6oNIAp8ZplhMPpt7JXTsh7f+mN3+moDGLW7UUDLhb2y9RVrdaaXBJIeUjgPQ2TT0MRkzAmXTB/FHEfmCiEwRkbGpi++RmfwyaRGMm+nWrdZk9s/rbIT614dvlFNnE+zf5BLUoWra7kY9GVPgsp0HcQPwLLDKu6z0MygzjLJd6kPElQJv2gZbnh/ceySibhRR805/O7Db97sWy2ASWCaJKLTsHJqYjBnBBkwQqjo9w2XGcARn8syMs6DySNeKOBSte6DhTX/KW7Q1uFNaDFECaqu3IoSm4GVTzbVMRP6PiKzw7s8Ukfdm8bx7vAqw6/t4/GwRaRKRNd7l5sGHb4ZVIARzroA962H33w/tNWLt0PAGtA1htZaW3f7MY2jall9Ddo0ZZtlWc43SPdx1O/C1LJ73P8CFA+zzF1Wd711uHWBfkw+Ou8gNBT3UVgS4U0BNW93M5kT88OJp2g4tuw7vNfoS73TJx5gClU2COFpVvwXEAFS1gywGz6vqs4DNPBptwqUw6xK3mNCBw6y40tnkOrA7+5jh3B9V9/5t9YcXw0Ba94ysQoTGDKFsEkRURErxTu6KyNFAljOPBnSKiKwVkd+LyKy+dhKRa0VkpYisrK/3+YBQcLLspE436xIIFmdXxG8gyZgbddS0I/vTOam6Sh3D8f1DXfVXO9VkClA2CeIW4HFgiojcBzwF/MsQvPdq4ChVnQd8D3i0rx1VdYWqLlbVxePH21IUOVda7U41vfWHoauE2rY3u/UZkgmXUPqqq+SHWJv/LRVj8lA2o5j+CFwKLAfuBxar6p8O941VtVlVW73bvwPCIlJ7uK9rhsncK1xfwkBF/AYj3uE6sFv7OBgn4m64bDQHo4tadrnSHcYUkGxaEKjqPlX9X1X9raoOyVdGETlSxA3CF5GlXiy2ENFIUTURpp8Frz02tFVcNenKXezb5ArtpcSj3hDZ9qF7r8HG1WgVX01hySpBHAoRuR94AThORLaLyMdF5FMi8ilvl8uB9SKyFvgusEzVTvSOKPOWudMvG34z9K8dafY6sJvcaaeGN7MvuueXaMvQDs81Js/5VrZSVa8a4PE7gDv8en+TpWxnUmcy/jiYtNAV8Zt9GQTDQxcXuOJ5+zeDBA5/dvRQad4BJVVD/7Mak4eymSh3tIgUe7fPFpHPiki1/6GZEWHuMreE58Yn/XuPfEkO4C0utD3XURgzLLI5xfQLICEixwA/AqYDP/c1KjNyTF4CY492Q17z6UDup85G6MiwdKkxo0w2CSKpqnHgEuB2Vf08UOdvWGbESBXxO/COmzxXKJq2D38Zc2OGWTYJIiYiVwFXA7/1ttkJ2FHjMPogUo4+B6omwZNfhZdWFMbM42TM9UcYM4plkyCuAU4Bvq6qb4vIdOBn/oZlRpRACC6+A445D9b8HB5eDu/8ZfTPPm7fN3zrXBiTAzLSRpYuXrxYV6605SiGTDIJu9cO3evtWgd/vd2NPppyMpz2WTdnIt+ougN8+WHOzQwWw/jjIeDbiHFjhoSIrFLVxYN5TjajmN4rIq+IyH4RaRaRFhEZxjoHZkSpmwuXroCTb3CJ5+GrYdWP82cWcjIBm/8Mj14P910OK+89vNdLRPyrJmtMjmUzD+J2XKmNv9tENpOVQAjmfhCOPhte+AGsuhfeegJO+xxMOSk3McUjrnbUugddB3PVJNfCWf1j9/ii5Yc+J6StHkproKhsyMI1Jh9kkyC2AestOYxShzNRbiDl4+H8W2D7e+Cv/w2//1eYdiac+hmomODf+6aLtMBrv3Y1ozoOuMl9538Fpp3hfvZnv+0lCYVF1xzi70Ohcat7bT9/n8YMs2wSxL8AvxORP5NW5ltVb/MtKjO6TF4Ml/8I1j0Mq38CD70ECz8Kcz7o34zk1r1uhvfrv3GjqqYshXlXQd38ngfxM7/grlf/xPVLLP7YoR3k4x3uPSuPGJr4jckD2SSIrwOtQAlQ5G84ZtQKFsGCD8Mx58Lzd7jhsG8+AaffCBMXDN377H8b1j0Abz0JKBx9rpunMe6YzPtLwCUJEXjlp+45iz9+aEmiZReUjIFwyeH8BMbkjWwSxFhVvcD3SExhqKyDf/g6bH0B/vpd+O3n4Zjz4eTroWzcob2mqlsje+397nVDJTDrA66FUnnkwM+XAJzxz4DAKz9zS2MtOZQkoW4d69qZh/BDGJN/skkQT4rIBar6B9+jMcMvV+fMp54CExe6eRNrfg5bXoAlH4MTL3ad3NnQpJu9vfYB2POq+/a+6BqXHErGDC4eCcAZ/w8gsOZngMKSTwz+9xNtdYsoHe7wWWPywIDzIESkBSjH9T/EcFNvVVWr/A/vYDYPwgc7X8nt+zdtd53Y2192p4JO/zwc0ecKtJCIwlt/dImhaZtrlcy9wq1yFzrM0zuahL98x/VdzP8QLPnk4JOEBKCowruUu4t1XpscO5R5EP1+VfMW9JmlqlsPKzJj+jNmMlz0LXj7WXjhDvj1DXDcu+Gka6EkrXBwtBVe+w2sf8RNchs3E867GaafmX2rYyASgDM+774Grfm5O3219NrBHeA16dazSC2LKgEIl3Uni6IKCASHJl5jfNTvf5Wqqoj8Clg0TPGYQiUCM86CKUtg9U9h3UPwznOw9JNu7sSrv3Kr18XaYNIiOPtL7tqPb+YScK0YxPVrwOCTRDpNuuSWvlRqqLRnwgjZ+A+Tf7L52vU3EVmiqi/7Ho0x4TI46To49h/gue/AX/7LbZcAzDjbjUiqPdb/OCTgRljJECWJ3uId7tLureAbLOpOFkXlEC4dmvcx5jBkkyDOAa4TkS1AG919EHN9jcwUtppp8N7bYdPTcOBtd8ppuGs6SQBOu9HdXnu/O9100nX+tFoSUeiIusl8ABJ0iSJY5OaKBELuEgxDIHU/x/WfVN2qf+kXVfd7S4/ZTqeNWNkkiIt8j8LkVsWR7nx5rD3XkfQk4irE5jqG024ExM2vQOGkT/nf6ayJ7j6MPmMLZkgeIZdAgqHuRBIMZxdvMukd6GPedeLgBJBITwbZrochByeMQK/7llDy0oAJQlW3DEcgJoeq6oA698+f6lyNtLiDgPGSxOfc7XUP4pLE9bkfmaQJiGd5kO46+HrJAw5OAL6tCKiuhZSIZrl/Hwml6/ftXYv0ut37sUz3B3iMTPsdxvb09zoo3rTbuf4s9WGIhn6YUSEYgrKx7gIQbXeJItIM0TbcDLIClUoSIq4DXXGT+/L0H/sgqSRAZ64jycJgE8po0iuBZPp8DVgWb+j+Ty1BmL4VlblL5RHu22YqWURaCvOfVwRO/Swg8PeHAIWTPz1ykoQZAbyDeyoJ5Pg7mSUIk51AEEqr3QUg1tmdLKKtPp6eyDMicOo/ueu/P+z+kU+5wZKEGZUsQZhDEy5xl4oJrnMz2urWRRioY3U0EIFTPuNur38EUHffkoQZZSxBmMMXCEBJlbtE211V09GeKLqShHhJAksSZtSxBGGGVlEZjDvadWq37B7diULEnV4CSxJmVLIEYfxRVF4YiSKVJNL7JFJ9FMaMcL5NxRSRe0Rkr4is7+NxEZHvishGEVknIgv9isXkUCpR1B4LxTkpAOw/ETeaac4V8Oov4fnvZTEU0Zj852cL4n+AO4Cf9PH4RcBM73IScKd3bUaj0d6iEPHmReDmSWx9wVWprZroypGnXxeV5zpaY7LiW4JQ1WdFZFo/u1wM/ETdghR/E5FqEalT1V1+xWTyQCpRRFqhdc/oShQiboZ11STYuQZadsKmNw7+GYur3D5VdVA50V2nEkj5+MIqNaFJN1S6swk6m73rtEukuftaAl5tKu8SKup5v/e2UHH/j6f2Ee/3Lb0mqSHevLX0CWt9TWTLNFO6v5+7dwtT+35MxP3sOZDLPohJwLa0+9u9bQclCBG5FrgWYOrUqcMSnPFZcYW7jLZEIeJWxTvx4u5tkRbXamreAc27XOJo3gX1r8PmP/esaRQIuWVSK+vcpbgCgsXdB7tQUdr9Yne/63aGffw+sGjSXZK7fIT1AAAeAUlEQVQJd52IdR/UB7qk5tH0NYcmGHbrgZRUuaSq6lqgif0Qj3bPtu66xPz9WXNJAl7JkVCvEiRZ3JfQIa+XkssEkSnFZjxxq6orgBXgVpTzMygzzNITRctuiLbkOqKhV1zpLpnWqk7GobW+O2mkX9e/AbEOVzzvUAXCPRNGKolI0CUmTbh5LKkDvSZ6HvB73+66nxjc5MhA2C0Dm7qMO7rn/ZIxLgmk3w+VDH6hpkSsO2FkSiLxKCQiPW9r0vvWrt4RKHW716xmeu2Tek7XPpn27yf+vmo69X5Mk30UTxzgfjwKyfa0MiuDl8sEsR2YknZ/MrAzR7GYXCuugOJjRneiyCQQ8k4x1bn2cybJhHeAi7h/+njEu5060KXdHnAf73GS7lupBL3rQPe31NS2QODgfQKp+8G0x9P2CYZ6HfS963Cp/yO7JNCdBM3Bbhz8OKBcJojHgM+IyAO4zukm638w3YmiBVr2FE6i6E8gCIFSW0TIDDvfEoSI3A+cDdSKyHbgFiAMoKp3Ab8D3g1sBNqBa/yKxYxAqdMykRboaMSXqmWadOfCC6WOlDGD5OcopqsGeFyBG/x6fzNKpBKFXxJxt+xnW8Phnes3ZhSymdSmsAW9UUMVR7jlPtvq829lvQIXSyixRBJFEaS7O1dw93ut2SPe8NTuAajuOXm8Lk/esgRhDLgjR2qxpEiLSxSdTQM+LZmEhCqJpJLQJMkkxJPudFhAhIA3ND4gEEAQcQe0QOoayfnS0vkgkYRYIkkskSSaSBKNJ4knkkTiSV+WROhKHqkkE/D+Xt7G1N9OvETj/l6pbZK2zeu7p+ffVUQIIARH+N/WEoQxaRJJJR4sI1ExlXhRJ8mWvSTb9pFIJkgk1SWEpJJQJZ4YuoNX1wEn7UAlIgQD3ZeACKFAz22hwMj5SqwKsaRLALG4Eo0nuloHqaQ6bLGkxeRWsEvfOnRSf9dgsPvvFxAIpP6GEiAQgJAIgYC7hETypqVjCcIUHFWlJRKnuSNGLOEO9PGkawUcXEJpHBRVE+rcRzC6H0n6s5Ke4loigz1QpQ5AIe8AlEoaqYQSDErXwScUCAzLN9pYUonHk0QTSjSRIJ5QLyn40xoYrHhSaYkqzRF3SQIlQaE4BMVBocS7Lg5C8DATcOrvmogP7icPCAQDAYJeawS6r6FnC6i3jNsG9e7dLEGYgtEejdPYHqOxPUZiMN9YA0HiZROIl44nGGki1NmAxPOjn2KwByDBHfSkj6+oOtAhvJ+HVd3zh7YxECAZKiUZdvWrxJugJ5pAkwki8QQtnXFaOhI0RxM0RxI0R5SmiNIcVVoiSnM02ZUMmqNK2yDGIoQDUByE4pD0SB69r4tDXpLptW95WNylSKgIC+Vhty0c7P+QnVRIJpLketiEJQgzqnXGEjR1uKQQjR/mcFYREiXVJEqqCcTaCHXUE4iOrBIhSqqPJB++y2cgQZKhcjqkhJ0dIXa0wu7mCLua22hqj9LSGae5M+aSQmecaKLvv2lZUZDK4iBVJUVUlgWpqwlSVRzwLkJVUYCqYgiK0hlXInElkvCu40k6E0okpu66x+NJOuNKcyxtX++xaCK732tRUCgvClAWDlBRFKCsKEh5UcBdwkHKioOUF7lLcUgQVcD9rKJJpGtWt5sFLiiSGq6tSUC79zkMliDMqBNLJL2kEKUj6s8ch2S4nGi4HElECHXsIxg50LOmkhlQe0zZ2RZge0eIHW0BdrYqO5uj7G7eS0Nrz1N55cVBasqKqCwJc0RVCcdMCFFZEqaqJExlSYiqEnff3Q5TURIinIMe4qQq0bjrXO+IJWiPxGmLJmiPxmmLJGiLxN3t6MGP1TfFaY9EaYvGiRzul5khYgnCjAqJpNLcEaOxI0ZbJD5syzFosJhYxUTai8ezc/du3t69j00HonTE1HVGCgS9judgajRT2iXojYw5+DE3AqYkCKVhoSwklIahNJS6LZSFDv8cud+aI8rO1iQ7W5LsbE2yow12tsLOljiNnT0TanVZmLoxpcydVE1ddQl1Y0qpG1NC3ZgSKkvCvsSXGnmU1Ez9T4MXEKEkHKQkHGRM6aHHHE8kaY8maIvGD2r5Zorz4E09t6jCB745+DgsQZgRK9XZ3NgWo7kzNmxJoTOW4J2GNjY1tLGpvpXN9a1s2dfeNRKnJOROGyTVnY9PXSeS3fcTypCcqw8HoCwslIYDlIWEkrTbpeHAQZeycICScJBwAJJekb6kJtGkuySTiqa24WJUTfV1uNtJr68hdTup7uSHKnTGXULY1ZpkZ6vrDE5XW1FE3ZhSls7omQCOHFNCWZF/hyMRKAkHKA4FKfauU/fTJZPa4++mXX8vdTXztPtx7fH37d63+7lKItnz8Ww/o6FggKrSAFWHkWSGgiUIM+K0ReI0dsRoGmxn8yFo7YyzqcElgU31bWyub2VHY0fXwb2yJMTR4yu4eP5Ejh5fwYzaCuqqSwhkOU4x6c2hSKobQtt1AEoqiWSSzliCzrjSEUvQEU246/TbGbY1RxPs7UjQEYt3bRuuUaQBgQkVIeqqyji9rpyJ1WVdrYEjqooPOiAPtd6JoCQcpDh0cCLoM/6Am7/gF+36e3cnoEQqgSS7k1HvhNM7oh6TA3s92vOxw2MJwowIQ9rZnIGqsr8tyuaGtq5ksKm+lb0tka59aiuKOXp8OacfU8vRE1wyqK0o6nNEUDYCIgQGGNFyuFTdMNP0pJJIao+JX10TwLzTXt2Tvci4X49rb5JY19DaYTjtFQjgkkDISwLhACWhIEWh/J6ZJt6Q5JHCEoTJG8mkN14+kSTuTaCKJty36KHubG6PxlmzrZGNe72WQUMrje3dgwonjinhuCMruWh2HUePL2fG+IrDOqecSyLiHUyDVOc6mEEQgXAwQDgoFIV6nhbK90QwWliCMMMmVUohFu9OBF3lFeLq++mipo4YL729j+c37WPNtkbiSSUYEKbUlLJoag0zxldw9PhypteW+3o+3HQLeQf/omCg6zqcug72PV/DDA/7LzBDIpHs/sYfiye7SiiktwiGqxM53b7WCH/bvI/nN+9j/Y4mkgoTKot579yJnDxjLDMnVNq3UR8FAz0TQKo1kNpmCSC/WYIwfUod9BNJdad8ksmubfGEEk+6EhW5Ovj3ZXdzJy9sauD5Tft4fbdbcGhyTSmXL5rCqUePY0Ztua8HppJwgFAw0F1BNFPFUZEej3dthx5VSlOzk4Gu1S67VsL0RhK5bW5j71Uy0/fB2y91272/pN1Obc8cb/p+B8XsPRiQtIQQDBDI82G4pn+WIApYIqm0RuJ0xhLegd59848nc/eN/1Bt29/O85saeH7zPjbXtwEwY3w5HzlpKqceXcuUsWW+vbeIm7VbVeomavk9UseY4WIJosB0xhI0d8Zo7YzTHk2MqCSQTlXZVN/GC5v38fymBrYf6ADg+CMr+dhp0zhlRi1Hjinx7f0DAbpm8VaWhPN+wpoxh8ISxCiXSCqtXv2a1kiceJa1YvJRUpU3dre4lsKmfextiRAQmD1pDO+dU8fJM8YxrsK/BeuLQgFXyqE0THlR0M6fm1HPEsQo1BFN0NIZoyXiJkqN1FYCQCSe4PVdLTy/eR9/27SP/e1RQgFh/pRqli2ZwtLp43wdflpaFKTKSwolYTt1ZAqLJYhRIJ5I0hqJd1W49Hu4qB9aOmNsO9DB9gPtbNvvrrcf6GBPcycKFIcCLDqqhlNmjGPJtLGUF/vz0RWBiuJQV39CLgq+GZMvLEGMUO3R7oTQER0ZVUSTqjS0RNh+oINtXgJIXTd1dE9SCweFSdWlzDyignOOG8+M8RXMn1Lt2zf4UFC6+hIqi0M28sYYjyWIESKWSNLqJYTWSH63EmKJJDsbO9ieahEc6G4RpJcxriwOMXlsGUunj2VKTSmTa8qYUlPG+Mpi3zt9S8IBVy66NGST4ozpg/1n5KlIPJFWPz7hS/2hoRBPJHljTwtrtjXydkMb2/a3s7u5s0dxuPGVxUypKWXWxDFMrillSk0Zk2tKGVMaHraO3kDAnTpKrRlgp46MGZgliDzR4dV+b4+463wdbaSqbDvQwZptB1izrZH1O5rpiCUICEyqKWNabTlnzBzPZK9FMLmmNGeduyXhABXeqSMbdWTM4FmCyAFV7VoMpC3iVpRK5mcDAYAD7VHWbmvklW2NrN3WyL42t9pX3ZgSzj5uPAumVDNncjUVPnUcZ0vEld9OtRSshIYxh8cSxDBIJLV7ycFo/g897YwleHVnc1cr4Z197YA7+M6bXM38Ke5yRJV/E9GyVRwOdCWFiuKQtRKMGUKWIHwQSyS7ThW1R+N0xpJ5nRASSWVTfStrtzWyZlsjr+1qJp5UwkHhxLoqrj5lGvOnVDNjfHnWC+H4JTUMtbIkRIWVtTDGV74mCBG5EPhvIAj8UFW/0evx5cB/Aju8TXeo6g/9jMlvm+tbaYvk/7DTRFJZt72Rp9/Yy6p3DtASiQMwvbac982byPwp1ZxYV5UXk8NSM5grS0KUF9kwVGOGi28JQkSCwPeBdwHbgZdF5DFVfa3Xrg+q6mf8imM4HWiL5n1y2LKvjadf38uf3qxnf1uU8uIgJ08fx4KpNcybPIbqsqKcxhcKCmVFQUrDQUqKgpSFg4RsxJExOeFnC2IpsFFVNwOIyAPAxUDvBDEqJJPK7ubOXIeR0YH2KM++Wc/Tb+xlc30bwYCw+KgazjluAkunj83ZkM9QUCgNBykrcsmgNBy04afG5BE/E8QkYFva/e3ASRn2u0xEzgTeBD6vqtsy7JP3GlojeTU0NRJP8NLb+3n69b2s3nqApMLMCRVcd+YMzpg5ftiXz0wlg9Ii72LJwJi852eCyHSiuPcR9DfA/aoaEZFPAT8Gzj3ohUSuBa4FmDp16lDHedhiiWSPxe1zJanKhl3NPP36Xp7b2EB7NEFtRRGXLZzMOcdN8HVNhHTBgHeaqChIiddCsGRgzMjjZ4LYDkxJuz8Z2Jm+g6ruS7t7N/DNTC+kqiuAFQCLFy/On6/pnt1NnTkdpbSzsYNn3tjLM2/sZU9zhJJwgFOPruXc4ycwZ9IY30ceiUB1WZjKkjClYVtQ3pjRws8E8TIwU0Sm40YpLQM+lL6DiNSp6i7v7vuBDT7G44uOaILG9tjAOw6xls4Yz21s4OnX9/L67hYEmDelmg+fdBSnzBg3LKOPggFhXEUR48qLrCPZmFHItwShqnER+QzwBG6Y6z2q+qqI3AqsVNXHgM+KyPuBOLAfWO5XPH7Z1dQxbO8VSyRZvfUAT7++l5fe3k88qUwdW8Y1p07jrGPH+7pYTrricIDaimKqS8M25NSYUUw0n2dwZbB48WJduXJlrsMAoKkjxlZvlrFfIvEEf9/RxMp3DvCXt+pp7oxTXRrmzGPHc+7xE5hRWz5ss4crSkLUVhRRWTK8HdzGmMMnIqtUdfFgnmMzqQ+RqrK7aeiHtaoqOxo7WLXlAKu3HmD9jmaiiSRFoQBLp43l3OMnsGBK9bCd0hGBMaVhxlcW58WkOWPM8LEEcYgaWqNDVoK7I5pg3Y5GVm05wKotB7pGRE2uKeWi2Uey8KgaZk8cM6ydv6n+hbHlRTYCyZgCZQniEMQTSfa2HHrrQVXZur/dJYStB3htp6t9VBoOMnfyGC5fNJmFU2tyUgzP+heMMSmWIA7B3pbIoMtzt0XirNnWyOqt7tRRQ6srmT1tXBnvnzeRRUfVcEJdVc6+rZcXB6mtLKbK+heMMR5LEIPUGUuw31sPoT+qyuaGNlZ7rYQNu5pJKpQVBZk/pZqrltawcGoNtcM08igT618wxvTHEsQg9TcpLpFUnt/UwMotB3hl6wEOePMjZowv57KFk1l0VA3HHVGZ8zkDgQCMKy9mXIX1Lxhj+mYJYhBaI3FaOuMZH0sklW//4Q2e29hAZXGIBVOrWTjVtRJqynNbITWlKBRwHc9lRda/YIwZkCWIQdjdx6Q4VeX/f3YTz21s4KOnHMWlCyYTzKMDcGlRkPEVxVSV2oprxpjsWYLI0v62KB3RzD3TP39pK79fv5tLF0zig4umZNwnF6pKQ9RWFFOe47WijTEjkx05spBMKnv6WOvht+t28sDL2zj/hAksP3Xa8AaWgQjUlBdRW1Fky3EaYw6LJYgs1Pex1sOf36xnxbObOWn6WD5zzsycnr4JBoRab2JbrjvBjTGjgyWIAUTjSeozrPWwessBvvPkm5w4sYqb/uG4nPU5pCa21ZSFrX/BGDOkLEEMYE/zwcNaX9/dzP/3+w0cNbaM//c9J+bkVI5NbDPG+M0SRD/ao/GD1nrYur+dW3/zGmPLi/jK+2cNawdwamJbbUUxpUXWv2CM8ZcliH7s6lWtdW9zJzf/ej2hoHDr+2dTUzY88xuKwwHGlIapKSuy1dqMMcPGEkQfmtpjtEcS3fc7Ytz82Kt0xhL8x6VzOXKMv4X0UklhTGnYymAYY3LCEkQGqsrutGGt7dE4X/nNq9S3RLj14llMry335X1LvKRQZUnBGJMHLEFkkL7WQyyR5Ou/28Dm+lb+z3tOZNbEMUP6XqVFAaq8loLNWzDG5BNLEL2kr/WQqq+0bnsTnz9/JkumjR2S9ygtCnadPrI+BWNMvrIE0cseb60HVeXOP2/i+U37+Php0zn3+CMO63XLil1SqCqxpGCMGRksQaTpjCU44K31cN+LW3ni1d1cvnAyH1gwadCvJeLWfkj1KVhZbWPMSGMJIk1qrYfH1u7kwZXbuODEI/joKUcN6jUqSkJeSyFkJS+MMSOaJQhPS2eMls44f3pjL3f/ZTOnzBjHp88+JqvyFaGgUFNWRE25dTQbY0YPSxB4w1qbOlm5ZT+3P/UWcyeN4QsXDFxfqbw4yLhyW2fBGDM6WYIADrTHeGVrI//x+9eZNq6Mf3vPCX12JAcDQk25m9VscxWMMaNZwSeIRFJ5cfM+vvrbV6ktL+Ir75tFWdHBv5ay4iDjyosYU2pVU40xhaHgE8TabY3826PrKQ4FufXi2VSn1VcKBKCmzK2xYK0FY0yhKegEsauxgxt+vppoPMk3Lp3DEVWuvlJpUXdrIZBHa0sbY8xw8nUcpohcKCJviMhGEflihseLReRB7/EXRWSan/Gka+mM8dF7XmJfW5Sb33si02rLGVtRxDETKjhmQgU15UWWHIwxBc23FoSIBIHvA+8CtgMvi8hjqvpa2m4fBw6o6jEisgz4JnClXzGldMYSfPx/XmZzQxv/fvEszjthAtVlRTlbFc4YY/KRn6eYlgIbVXUzgIg8AFwMpCeIi4GveLcfAe4QEVHtvYZbt1d3NjPr5scPK7B4UonEk3zzsjlcuWTqYb2WMcaMVn4miEnAtrT724GT+tpHVeMi0gSMAxrSdxKRa4Frvbutr/37RW8MRYDLvgnLem6q7f3eeW6kxQsjL+aRFi9YzMNhpMULcNxgn+Bngsh0vqZ3yyCbfVDVFcCKoQiqPyKyUlUX+/0+Q2WkxQsjL+aRFi9YzMNhpMULLubBPsfPTurtwJS0+5OBnX3tIyIhYAyw38eYjDHGZMnPBPEyMFNEpotIEe5szmO99nkMuNq7fTnwdH/9D8YYY4aPb6eYvD6FzwBPAEHgHlV9VURuBVaq6mPAj4CfishGXMthWd+vOCx8P401xEZavDDyYh5p8YLFPBxGWrxwCDGLfWE3xhiTiS1YYIwxJiNLEMYYYzIqyAQhIveIyF4RWZ+2bayI/FFE3vKua3IZY28iMkVEnhGRDSLyqoh8ztuel3GLSImIvCQia714v+ptn+6VVXnLK7NSNNBrDTcRCYrIKyLyW+9+XscsIu+IyN9FZE1qKGO+fi4ARKRaRB4Rkde9z/MpeR7vcd7vNnVpFpEb8zzmz3v/d+tF5H7v/3HQn+OCTBDA/wAX9tr2ReApVZ0JPOXdzydx4J9V9QTgZOAGETmR/I07ApyrqvOA+cCFInIyrpzKd7x4D+DKreSbzwEb0u6PhJjPUdX5aWPz8/VzAfDfwOOqejwwD/e7ztt4VfUN73c7H1gEtAO/Ik9jFpFJwGeBxao6GzdIKFXKaHCfY1UtyAswDVifdv8NoM67XQe8kesYB4j/17g6V3kfN1AGrMbNpG8AQt72U4Anch1fr1gn4/7ZzwV+i5vMme8xvwPU9tqWl58LoAp4G2+ATL7HmyH+C4C/5nPMdFeoGIsbqfpb4B8O5XNcqC2ITI5Q1V0A3vWEHMfTJ6/q7QLgRfI4bu9UzRpgL/BHYBPQqKpxb5ftuA9zPrkd+Bcg6d0fR/7HrMAfRGSVV5YG8vdzMQOoB+71TuP9UETKyd94e1sG3O/dzsuYVXUH8G1gK7ALaAJWcQifY0sQI4yIVAC/AG5U1eZcx9MfVU2oa5ZPxhVvPCHTbsMbVd9E5L3AXlVdlb45w655E7PnNFVdCFyEO/V4Zq4D6kcIWAjcqaoLgDby5NTMQLxz9u8HHs51LP3x+kIuBqYDE4Fy3GejtwE/x5Yguu0RkToA73pvjuM5iIiEccnhPlX9pbc57+NW1UbgT7i+k2qvrApkLr+SS6cB7xeRd4AHcKeZbie/Y0ZVd3rXe3HnxpeSv5+L7cB2VX3Ru/8ILmHka7zpLgJWq+oe736+xnw+8Laq1qtqDPglcCqH8Dm2BNEtvezH1bhz/HlDRAQ383yDqt6W9lBexi0i40Wk2rtdivvQbgCewZVVgTyKF0BVv6Sqk1V1Gu5UwtOq+mHyOGYRKReRytRt3Dny9eTp50JVdwPbRCRVWfQ83BIAeRlvL1fRfXoJ8jfmrcDJIlLmHTdSv+PBf45z3aGSo06c+3Hn5mK4bzQfx51rfgp4y7sem+s4e8V8Oq5JuA5Y413ena9xA3OBV7x41wM3e9tnAC8BG3FN9eJcx9pH/GcDv833mL3Y1nqXV4F/87bn5efCi20+sNL7bDwK1ORzvF7MZcA+YEzatryNGfgq8Lr3v/dToPhQPsdWasMYY0xGdorJGGNMRpYgjDHGZGQJwhhjTEaWIIwxxmRkCcIYY0xGliDMiCAi49Kqae4WkR1p97Oqrioi96aNv+9rnxtE5MNDFPPFXnxrReQ1EfnEAPuf6xU0zPRYnYj8Lu21HvO2TxGRB4ciXmN6s2GuZsQRka8Arar67V7bBfeZTmZ84jASkWJcUbrFqrrTu3+Uqr7Zz3O+BjSo6u0ZHvsRbhbv9737c1V1nU/hGwNYC8KMcCJyjFfz/i5cxdg6EVkhIiu9evg3p+37nIjMF5GQiDSKyDe8b+QviMgEb5+viciNaft/Q9y6Fm+IyKne9nIR+YX33Pu995rfK7QxuDpO+wFUNZJKDiJyhIj80nveSyJysogcDXwCuMlrdZza6/XqcJM68V5vXdrPv8a7fW9aq6pBRP7N2/5F733Wpf8+jBmIJQgzGpwI/EhVF6irZPlFdesizAPe5a2b0dsY4M/q1qt4AfhYH68tqroUuAlIHVz/CdjtPfcbuMq6Pairi/QEsEVEfi4iV4lI6v/tu8C3vBivAH6oqpuAHwL/qW7tged7veQdwI9F5GkR+XKqBlCv97xGXXHES3ClnX8iIu8GpuJKrc8HTs2QfIzJyBKEGQ02qerLafevEpHVuBbFCbgE0luHqv7eu70Ktz5IJr/MsM/puGJ+qGqqxMVBVHU5bs2OlbiKpSu8h84H7vK++T8K1Hj1qvqkqr8DjsbV4zoReEVExvXez3udh4HrVXUbrjbTRbiyJ6uBY4Bj+3svY1JCA+9iTN5rS90QkZm4FeGWqmqjiPwMKMnwnGja7QR9/y9EMuyTqQR4Rt6poHUi8nNcscJPeM9fqqrpMeC6UPp9rX3AfcB9IvI4LlH1Tk53Aw+o6jNpsX5NVX+UbczGpFgLwow2VUAL0OydhvkHH97jOdypIURkDhlaKCJSJT3XZZgPbPFuPwnckLZvqv+iBajM9IYicl6qlSEiVbha/1t77fM5INyr8/4J4ONepVdEZLKI1Gb5c5oCZy0IM9qsxpU2Xg9sBv7qw3t8D3d+f533futxq3alE+BLInI30AG00t3PcQNwp4hcg/sffMbb9mvgYRG5FLihVz/EEuAOEYnhvtjdqaqviMgxaft8AWhPdVoDd6jqD0XkeOBvXgulBfgQro/CmH7ZMFdjBkncoishVe30Tmn9AZip3cs5GjMqWAvCmMGrAJ7yEoUA11lyMKORtSCMMcZkZJ3UxhhjMrIEYYwxJiNLEMYYYzKyBGGMMSYjSxDGGGMy+r8+1HVLdpLeQQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f96c6053358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an even more complex model, we still converge, but the convergence only happens for *large* amounts of training data.\n",
    "\n",
    "So we see the following:\n",
    "\n",
    "- you can **cause the lines to converge** by adding more points or by simplifying the model.\n",
    "- you can **bring the convergence error down** only by increasing the complexity of the model.\n",
    "\n",
    "Thus these curves can give you hints about how you might improve a sub-optimal model. If the curves are already close together, you need more model complexity. If the curves are far apart, you might also improve the model by adding more data.\n",
    "\n",
    "To make this more concrete, imagine some telescope data in which the results are not robust enough.  You must think about whether to spend your valuable telescope time observing *more objects* to get a larger training set, or *more attributes of each object* in order to improve the model.  The answer to this question has real consequences, and can be addressed using these metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recall, Precision and F1-Score\n",
    "\n",
    "Intuitively, [precision](http://en.wikipedia.org/wiki/Precision_and_recall#Precision) is the ability\n",
    "of the classifier not to label as positive a sample that is negative, and\n",
    "[recall](http://en.wikipedia.org/wiki/Precision_and_recall#Recall) is the\n",
    "ability of the classifier to find all the positive samples.\n",
    "\n",
    "The  [F-measure](http://en.wikipedia.org/wiki/F1_score>)\n",
    "($F_\\beta$ and $F_1$ measures) can be interpreted as a weighted\n",
    "harmonic mean of the precision and recall. A\n",
    "$F_\\beta$ measure reaches its best value at 1 and its worst score at 0.\n",
    "With $\\beta = 1$,  $F_\\beta$ and\n",
    "$F_1$  are equivalent, and the recall and the precision are equally important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import zipfile\n",
    "with zipfile.ZipFile('../datasets/titanic.csv.zip', 'r') as z:\n",
    "    f = z.open('titanic.csv')\n",
    "    titanic = pd.read_csv(f, sep=',', index_col=0)\n",
    "titanic.head()\n",
    "\n",
    "# fill missing values for Age with the median age\n",
    "titanic.Age.fillna(titanic.Age.median(), inplace=True)\n",
    "\n",
    "# define X and y\n",
    "feature_cols = ['Pclass', 'Parch', 'Age']\n",
    "X = titanic[feature_cols]\n",
    "y = titanic.Survived\n",
    "\n",
    "# train/test split\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "\n",
    "# train a logistic regression model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[107,  21],\n",
       "       [ 52,  43]])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision_score  0.671875\n",
      "recall_score     0.45263157894736844\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, f1_score\n",
    "print('precision_score ', precision_score(y_test, y_pred_class))\n",
    "print('recall_score    ', recall_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### F1Score\n",
    "\n",
    "The traditional F-measure or balanced F-score (F1 score) is the harmonic mean of precision and recall:\n",
    "\n",
    "$$F_1 = 2 \\cdot \\frac{\\mathrm{precision} \\cdot \\mathrm{recall}}{\\mathrm{precision} + \\mathrm{recall}}.$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f1_score     0.5408805031446541\n"
     ]
    }
   ],
   "source": [
    "print('f1_score    ', f1_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "We've gone over several useful tools for model validation\n",
    "\n",
    "- The **Training Score** shows how well a model fits the data it was trained on. This is not a good indication of model effectiveness\n",
    "- The **Validation Score** shows how well a model fits hold-out data. The most effective method is some form of cross-validation, where multiple hold-out sets are used.\n",
    "- **Validation Curves** are a plot of validation score and training score as a function of **model complexity**:\n",
    "  + when the two curves are close, it indicates *underfitting*\n",
    "  + when the two curves are separated, it indicates *overfitting*\n",
    "  + the \"sweet spot\" is in the middle\n",
    "- **Learning Curves** are a plot of the validation score and training score as a function of **Number of training samples**\n",
    "  + when the curves are close, it indicates *underfitting*, and adding more data will not generally improve the estimator.\n",
    "  + when the curves are far apart, it indicates *overfitting*, and adding more data may increase the effectiveness of the model.\n",
    "  \n",
    "These tools are powerful means of evaluating your model on your data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "name": "_merged"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}