{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 13 - Ensemble Methods - Random Forest\n",
    "\n",
    "\n",
    "by [Alejandro Correa Bahnsen](albahnsen.com/) and [Jesus Solano](https://github.com/jesugome)\n",
    "\n",
    "version 1.5, February 2019\n",
    "\n",
    "## Part of the class [Practical Machine Learning](https://github.com/albahnsen/PracticalMachineLearningClass)\n",
    "\n",
    "\n",
    "\n",
    "This notebook is licensed under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US). Special thanks goes to [Kevin Markham](https://github.com/justmarkham))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why are we learning about ensembling?\n",
    "\n",
    "- Very popular method for improving the predictive performance of machine learning models\n",
    "- Provides a foundation for understanding more sophisticated models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lesson objectives\n",
    "\n",
    "Students will be able to:\n",
    "\n",
    "- Explain the difference between bagged trees and Random Forests\n",
    "- Build and tune a Random Forest model in scikit-learn\n",
    "- Decide whether a decision tree or a Random Forest is a better model for a given problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Building and tuning decision trees\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AtBat</th>\n",
       "      <th>Hits</th>\n",
       "      <th>HmRun</th>\n",
       "      <th>Runs</th>\n",
       "      <th>RBI</th>\n",
       "      <th>Walks</th>\n",
       "      <th>Years</th>\n",
       "      <th>CAtBat</th>\n",
       "      <th>CHits</th>\n",
       "      <th>CHmRun</th>\n",
       "      <th>CRuns</th>\n",
       "      <th>CRBI</th>\n",
       "      <th>CWalks</th>\n",
       "      <th>League</th>\n",
       "      <th>Division</th>\n",
       "      <th>PutOuts</th>\n",
       "      <th>Assists</th>\n",
       "      <th>Errors</th>\n",
       "      <th>Salary</th>\n",
       "      <th>NewLeague</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>315</td>\n",
       "      <td>81</td>\n",
       "      <td>7</td>\n",
       "      <td>24</td>\n",
       "      <td>38</td>\n",
       "      <td>39</td>\n",
       "      <td>14</td>\n",
       "      <td>3449</td>\n",
       "      <td>835</td>\n",
       "      <td>69</td>\n",
       "      <td>321</td>\n",
       "      <td>414</td>\n",
       "      <td>375</td>\n",
       "      <td>N</td>\n",
       "      <td>W</td>\n",
       "      <td>632</td>\n",
       "      <td>43</td>\n",
       "      <td>10</td>\n",
       "      <td>475.0</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>479</td>\n",
       "      <td>130</td>\n",
       "      <td>18</td>\n",
       "      <td>66</td>\n",
       "      <td>72</td>\n",
       "      <td>76</td>\n",
       "      <td>3</td>\n",
       "      <td>1624</td>\n",
       "      <td>457</td>\n",
       "      <td>63</td>\n",
       "      <td>224</td>\n",
       "      <td>266</td>\n",
       "      <td>263</td>\n",
       "      <td>A</td>\n",
       "      <td>W</td>\n",
       "      <td>880</td>\n",
       "      <td>82</td>\n",
       "      <td>14</td>\n",
       "      <td>480.0</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>496</td>\n",
       "      <td>141</td>\n",
       "      <td>20</td>\n",
       "      <td>65</td>\n",
       "      <td>78</td>\n",
       "      <td>37</td>\n",
       "      <td>11</td>\n",
       "      <td>5628</td>\n",
       "      <td>1575</td>\n",
       "      <td>225</td>\n",
       "      <td>828</td>\n",
       "      <td>838</td>\n",
       "      <td>354</td>\n",
       "      <td>N</td>\n",
       "      <td>E</td>\n",
       "      <td>200</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>500.0</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>321</td>\n",
       "      <td>87</td>\n",
       "      <td>10</td>\n",
       "      <td>39</td>\n",
       "      <td>42</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>396</td>\n",
       "      <td>101</td>\n",
       "      <td>12</td>\n",
       "      <td>48</td>\n",
       "      <td>46</td>\n",
       "      <td>33</td>\n",
       "      <td>N</td>\n",
       "      <td>E</td>\n",
       "      <td>805</td>\n",
       "      <td>40</td>\n",
       "      <td>4</td>\n",
       "      <td>91.5</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>594</td>\n",
       "      <td>169</td>\n",
       "      <td>4</td>\n",
       "      <td>74</td>\n",
       "      <td>51</td>\n",
       "      <td>35</td>\n",
       "      <td>11</td>\n",
       "      <td>4408</td>\n",
       "      <td>1133</td>\n",
       "      <td>19</td>\n",
       "      <td>501</td>\n",
       "      <td>336</td>\n",
       "      <td>194</td>\n",
       "      <td>A</td>\n",
       "      <td>W</td>\n",
       "      <td>282</td>\n",
       "      <td>421</td>\n",
       "      <td>25</td>\n",
       "      <td>750.0</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   AtBat  Hits  HmRun  Runs  RBI  Walks  Years  CAtBat  CHits  CHmRun  CRuns  \\\n",
       "1    315    81      7    24   38     39     14    3449    835      69    321   \n",
       "2    479   130     18    66   72     76      3    1624    457      63    224   \n",
       "3    496   141     20    65   78     37     11    5628   1575     225    828   \n",
       "4    321    87     10    39   42     30      2     396    101      12     48   \n",
       "5    594   169      4    74   51     35     11    4408   1133      19    501   \n",
       "\n",
       "   CRBI  CWalks League Division  PutOuts  Assists  Errors  Salary NewLeague  \n",
       "1   414     375      N        W      632       43      10   475.0         N  \n",
       "2   266     263      A        W      880       82      14   480.0         A  \n",
       "3   838     354      N        E      200       11       3   500.0         N  \n",
       "4    46      33      N        E      805       40       4    91.5         N  \n",
       "5   336     194      A        W      282      421      25   750.0         A  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# read in the data\n",
    "url = 'https://raw.githubusercontent.com/albahnsen/PracticalMachineLearningClass/master/datasets/hitters.csv'\n",
    "hitters = pd.read_csv(url)\n",
    "\n",
    "# remove rows with missing values\n",
    "hitters.dropna(inplace=True)\n",
    "hitters.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AtBat</th>\n",
       "      <th>Hits</th>\n",
       "      <th>HmRun</th>\n",
       "      <th>Runs</th>\n",
       "      <th>RBI</th>\n",
       "      <th>Walks</th>\n",
       "      <th>Years</th>\n",
       "      <th>CAtBat</th>\n",
       "      <th>CHits</th>\n",
       "      <th>CHmRun</th>\n",
       "      <th>CRuns</th>\n",
       "      <th>CRBI</th>\n",
       "      <th>CWalks</th>\n",
       "      <th>League</th>\n",
       "      <th>Division</th>\n",
       "      <th>PutOuts</th>\n",
       "      <th>Assists</th>\n",
       "      <th>Errors</th>\n",
       "      <th>Salary</th>\n",
       "      <th>NewLeague</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>315</td>\n",
       "      <td>81</td>\n",
       "      <td>7</td>\n",
       "      <td>24</td>\n",
       "      <td>38</td>\n",
       "      <td>39</td>\n",
       "      <td>14</td>\n",
       "      <td>3449</td>\n",
       "      <td>835</td>\n",
       "      <td>69</td>\n",
       "      <td>321</td>\n",
       "      <td>414</td>\n",
       "      <td>375</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>632</td>\n",
       "      <td>43</td>\n",
       "      <td>10</td>\n",
       "      <td>475.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>479</td>\n",
       "      <td>130</td>\n",
       "      <td>18</td>\n",
       "      <td>66</td>\n",
       "      <td>72</td>\n",
       "      <td>76</td>\n",
       "      <td>3</td>\n",
       "      <td>1624</td>\n",
       "      <td>457</td>\n",
       "      <td>63</td>\n",
       "      <td>224</td>\n",
       "      <td>266</td>\n",
       "      <td>263</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>880</td>\n",
       "      <td>82</td>\n",
       "      <td>14</td>\n",
       "      <td>480.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>496</td>\n",
       "      <td>141</td>\n",
       "      <td>20</td>\n",
       "      <td>65</td>\n",
       "      <td>78</td>\n",
       "      <td>37</td>\n",
       "      <td>11</td>\n",
       "      <td>5628</td>\n",
       "      <td>1575</td>\n",
       "      <td>225</td>\n",
       "      <td>828</td>\n",
       "      <td>838</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>500.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>321</td>\n",
       "      <td>87</td>\n",
       "      <td>10</td>\n",
       "      <td>39</td>\n",
       "      <td>42</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>396</td>\n",
       "      <td>101</td>\n",
       "      <td>12</td>\n",
       "      <td>48</td>\n",
       "      <td>46</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>805</td>\n",
       "      <td>40</td>\n",
       "      <td>4</td>\n",
       "      <td>91.5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>594</td>\n",
       "      <td>169</td>\n",
       "      <td>4</td>\n",
       "      <td>74</td>\n",
       "      <td>51</td>\n",
       "      <td>35</td>\n",
       "      <td>11</td>\n",
       "      <td>4408</td>\n",
       "      <td>1133</td>\n",
       "      <td>19</td>\n",
       "      <td>501</td>\n",
       "      <td>336</td>\n",
       "      <td>194</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>282</td>\n",
       "      <td>421</td>\n",
       "      <td>25</td>\n",
       "      <td>750.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   AtBat  Hits  HmRun  Runs  RBI  Walks  Years  CAtBat  CHits  CHmRun  CRuns  \\\n",
       "1    315    81      7    24   38     39     14    3449    835      69    321   \n",
       "2    479   130     18    66   72     76      3    1624    457      63    224   \n",
       "3    496   141     20    65   78     37     11    5628   1575     225    828   \n",
       "4    321    87     10    39   42     30      2     396    101      12     48   \n",
       "5    594   169      4    74   51     35     11    4408   1133      19    501   \n",
       "\n",
       "   CRBI  CWalks  League  Division  PutOuts  Assists  Errors  Salary  NewLeague  \n",
       "1   414     375       0         0      632       43      10   475.0          0  \n",
       "2   266     263       1         0      880       82      14   480.0          1  \n",
       "3   838     354       0         1      200       11       3   500.0          0  \n",
       "4    46      33       0         1      805       40       4    91.5          0  \n",
       "5   336     194       1         0      282      421      25   750.0          1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# encode categorical variables as integers\n",
    "hitters['League'] = pd.factorize(hitters.League)[0]\n",
    "hitters['Division'] = pd.factorize(hitters.Division)[0]\n",
    "hitters['NewLeague'] = pd.factorize(hitters.NewLeague)[0]\n",
    "hitters.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# allow plots to appear in the notebook\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x19a30087b38>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAEKCAYAAAChTwphAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsnXl8FdX5/99nZu6SlSQQQgg7hE1UQNm0Llg3cN/XirbV1rVfv35r3Vrcta11abX+umit+4qCguJStAgiiyICAcKSQAIhCVnvTe42c35/3EsIeOcmhJvcLOfNa15k5p6Zec4s5zPnnOc8R0gpUSgUCoWiK6Al2gCFQqFQKFqLEi2FQqFQdBmUaCkUCoWiy6BES6FQKBRdBiVaCoVCoegyKNFSKBQKRZehQ0RLCDFQCLFICFEghFgnhPhVZPu9QohSIcTqyDKz2T53CiE2CyE2CiFO6wg7FQqFQtG5ER0xTksIkQvkSim/EUKkAauAc4GLAY+U8rED0o8FXgMmA/2BT4GRUkqz3Y1VKBQKRaelQ2paUspdUspvIn/XAwVAXoxdzgFel1L6pZTbgM2EBUyhUCgUPRijo08ohBgCTAC+Bo4FbhJCXAWsBG6TUlYTFrRlzXYr4QCRq62tVaE8FApFl6NXr17iUPZva9l3qOftLHSoI4YQIhV4B/gfKWUd8CwwHBgP7AL+tDdplN2VSCkUCkUPp8NESwjhICxYr0gp5wBIKXdLKU0ppQX8g31NgCXAwGa7DwB2dpStCoVCoeicdEjzoBBCAM8BBVLKx5ttz5VS7oqsngesjfw9D3hVCPE4YUeMfGB5R9jamSksLCQ/Pz/RZnQYKr/dm56W3/bgbxkZMX//RU1NB1nScXRUn9axwE+A74UQqyPb7gIuE0KMJ9z0VwT8AkBKuU4I8SawHggBNyrPQYVCodifDndK6AR0SJ6llF8SvZ9qQYx9HgIeajejFAqFoovjSLQBCaAnCrVCoVB0C5ISbUACUKKlUCgUXZSeWID3xDx3PSwTraoIR0MlYZ8UhUKhUM2Dis6Ir46U5y9EqyhkDBpy+8X4znok0VYpFIpOQE8swFWU906Oe/5vMbYvR2usxtm4B8eqV9B2fp9osxQKRSfA0cLSHVGi1cnRvBX7r/vq0Gp2JMgahULRmVCipeh0BEefhuVKa1o3ew/FHKRiBysUirD3YKylO9ITm0S7FMHJsxD+ehwFH+H1BdEu+AMytU+izVIoFJ2AnliA98Q8dzkCx91E4LibwmFv8pT3oEKhCNNdmwBjoURLoVAouig9sQDviXlWKBSKboGqaSkUCoWiy9BdnS1ioURLoVAouig9sQDviXlWKBSKboFqHlQoFApFl0GJlqJNWBaUlQnS0iRpaS2nVygUinjQEwvwnpjnuFJTAxdfnMK2bRouF1xzjZ/bbgsk2iyFQtEDcPTAErwHZjm+3HFHEsuX77uMf/ubiwsuCDJkiEygVQqFoieQ5Ir9e3f8fFaxBw+Rykqx33pVlWDXLnVZFQpF+2MYsZfuiCpdD5GpU03c7n21qsGDLcaMMRNokUKh6Ck4jNhLd6SbZqvjuO02P/X1sGyZgdMpefBBHxkZibZKoVD0CPREG9DxKNE6RISA++7zA/5Em6JQKHoaPbAE74FZVigUim5CDyzBe2CWFQqFopvQgvdgd0SJVgIoLha8/rqTnByLK68MdlsvH4VC0c70wLKjB2Y5saxdq3H55cls366j65K5cx28/XYDeg/sUFUoFIdIDyzBlct7B/PII262bw8rlGkKli41WLkytmK9XuzgoiXJ/LpoOEUeETOtQqHoQegtLN2QHqjTicWyfrhuxhjW9c4OB3eucVMd0IBMLlpi8sl0DxlO+30ee+wbPv54O7ouuOOOoznhhLy42K5QKDoZPbAEVzWtDubmm/30779XpSQTJ5ocfbS9as0pcUQEK0yhR2NZpf2T+q9/refJJ79l+fLdfPVVGTfe+DmlpZ54ma9QKDoTrhaWbogSrQ7mmGNMXnmlgauv9vN//+fnvfe8OGPUmtIMiUMLMDbzewalFpGsS7Jd9nENFy0qweMJNa2XlHhYsmRnPLOgUCg6C0YLSzekm2arczNhgsWECb5WpX34yN3cNPFyBqZuJGC6WL/nbCamPmybPi8vdb/19HQn+fmZh2SvQqHopPTAErwHZrlrkZE+mwzXqsiah+P6v4X0XoKwjoyafvbsyRQUVFFQUIVhaFxwwQgmTMjuOIMVCkXH0QNL8B6Y5S6GVnPAuge0MrARLbfb4N13z6Cy0ofLpZOeHqPtUaFQdG26qYdgLFSfVmcnOAOsXvvWzWEIc1LMXYQQZGcnKcFSKLo7bejTEkIMFEIsEkIUCCHWCSF+FdmeJYT4RAhRGPk/M7JdCCH+LITYLIRYI4SY2OxYsyLpC4UQs9o1rxFUTauTowUvxqIRHAvweoOk6I8iZFaizVIoFJ2BtnkIhoDbpJTfCCHSgFVCiE+Aq4HPpJSPCiHuAO4AfgPMAPIjyxTgWWCKECILmA0cDcjIceZJKasPLVOx6ZCaVjyVvTsgpaSszKS+3mo5MaAFZ6E1vEHZ5t+jWSPb2TpFNPzV1fgqK5FSzUit6ES0oaYlpdwlpfwm8nc9UADkAecA/44k+zdwbuTvc4AXZZhlQIYQIhc4DfhESlkVEapPgNPjncUD6ajmwb3KPgaYCtwohBhLWMk/k1LmA59F1mF/Zb+OsLJ3Czwei5kzKzn++AqmTavg0UfrEm2SIgZSSpbddBMLpk1jwbHHsvjKK7FijQZXKDqSQ3R5F0IMASYAXwM5UspdEBY2oG8kWR6wo9luJZFtdtvblQ4RrTgqe5fnrrtq+eqrIOXlFiUlJv/4h5fNm0Mx96kTPv7rKGRjZjUWraudKeLDjnnzKHrnHRrLyvDt3k3pwoVsePrpRJulUIQ5hDBOQohU4B3gf6SUsb6eo8WOkzG2tysd3qcVS9mFEC0p+65oxywsLGwvc+PO1q0uml/2qiqLr74qRsroYlTj9DFv5FZq3H7EUNhQt4ezNw9HRH1euh+JvrelS5ZgNTY2rctQiB0rVuBoJ7sSnd+OpqfkNz8/v30O3MYSXAjhICxYr0gp50Q27xZC5EbK4lygPLK9BBjYbPcBwM7I9hMP2P552yxqPR0qWgcquxC2Be9BKXi7PRDtwCmn1LNqVT17y8EhQwxmzhxC797RP4ueT1pKjTM8K7LUoKSXFzG6F/lmTkeZnDAKCwsTfm/7/OQnVM2di2/3bgCcmZmM+8lPyGsHuzpDfjuSnpbfdqENJbgIF7zPAQVSyseb/TQPmAU8Gvl/brPtNwkhXifsiFEbEbaFwMN7fRGAU4E725KNg6HDRCtOyt7lueWWVOrqJIsX+3E4BPfem24rWACm2F+rTSwCqD6VjiLz8MM5+g9/YMOzz4KUDL34YvJOOy3RZikUYdrmPXgs8BPgeyHE6si2uwiL1ZtCiJ8B24GLIr8tAGYCm4EG4BoAKWWVEOIBYEUk3f1Syqo2WXQQdIhoxUvZO8LW9kYIwW9/m97q9Cf48ynW91CrhcM+DTQzyDf7trCXIp4MOuccBp1zTqLNUCh+SBtKcCnll0RvzQL4cZT0ErjR5ljPA88fvBVtp6NqWnFR9p7ISLMvP/VOY7FrC4HqBq5w/winGl6nUCigR4607ZAsx1PZeyLDrGyGNWZTuKOQ5HwV5UKhUETogWGceqBOKxQKRTehB5bgPTDLCoVC0U1wJ9qAjkeJlkKhUHRVVPOgQqFQKLoMPbAE74FZ7t5ILL5wz6fM2I6GxuH+KYwJdqt4wwqFYi89sATvgVnuegRMKKzUqPQ4aSl+wLeuL9nkXI0pwgOQv3b/h1xzMBlW7/Y3VKFQdCyqeVDR2ajxwTkvprCxUscQY7mo1OSJM3226Sv0XU2CBdCoeditlSjRUii6Iz2wBFczF3dybp2fxHe7DHxBgSdg8OJqJ6t32t+2LLMvQu773W0l09fq3xGmKhSKjsbVwtIN6YE63bX4umr/W2Sagv/sMhjfPxA1/VH+E6jVqtitl6KhMc4/iUwruyNMVSgUHU0PLMF7YJa7Ftn9LHbu1miKkeuWDM6xn1NLQ+PkxguQyB4zfYlC0WPpgSW4ah7s5Nw91YdDWFAN1EL/PIvT+sWeNBJQgqVQ9AQOcebirkg3zVb3YcO3OqJMQESnxHpBvUeQmhF9ejFTwvUrklhRJXBo8PNhQa4bEexAixUKRYfRA70HVU2rk7Nsi0EgtK/WVFqtsarI/kl9dL3BOyUa27wONtU7eGSjYFOdus0KRbekB9a0VGnWyRmYZdF80uasFIv8GH1ayzwVmHLf01rtS+LLmnafl02hUCQC5T2o6Gzcd76Pwt0aG3ZpSCvET0+QjMq1F62hfUpYWpqHKR0ApLlrGJ5dAmR0kMWKOmHysrMSv7C4xJ9Ff9lNSw9F4umBJXgPzHLXwu2AObc0UNcIJcWbGTt6RMz0p2RYvOELYQoHSHAFQ4xzDOggaxUeTG5NKWaL7gdgheHlj96B5CnhUrQHPbAE7/bNg1JK7rtvM2ecsYqLLlpNcXFDok1qE+lJ4NCjO180Z85XU/HvSoKdwC6oLO3N0m192t9ABQCfOmqbBAtgpx7kZdeeBFqk6NboLSzdkG6v0w8/vJW//nU7fn+4wL/ssjV89tkkkpK65x098CtEAM5u/2kCZqTfT28HV39HyT9x7nwJgUUw+yz8Q2+3TWsgwl2QzcxoD5sUCqAHlOA/pNtn+euva5sEC2D7dh9btzZy2GGpCbSq/cib5Cdltx9vTRpgkZ5Xx7BBsVXLQlIqJC4p6NuKAlZK2FEpcBrQL7Pl2l97IpE84q7ji0jt5sSQizv9veJ2fK3ma9zbHkELhmtLorEIK3kEwZzzo6Y/OdiLD5w1bDDC8SEHmU6u8auarqKdUJNAdj/S0vQfrPfp40iQNW3D44flxTq1FSmMGAEihq7U9f+KS3+6gYLVh+NO8jFi/AaWNJxLPllR0/uQXOMMUCAsDOBUS+fhoNP2+IEgXPJoMt9t1TE0OGVikKdv8MW0KYTJNn03OhpDzL5ocWyV/sDw8abeiDTC4vkmjYw3ncwIJcXl+I6qRU2CBaCZ9eh7PrUVLTcaT3oHM89ZjV9IzgpkkCG71msmJayu16kKwOQMk7SuZX7Pons2GMWk2z+Of/rTaHbsWE1xsY+UFJ1rrx1ATk5iO8W3b/fz5pu15OQYXHZZJoZhX+KX1wvOfS6Fgt0aLmMk7283ee7SRluRyHJUIVLrmHbikqZtOcEKMKOL1h+NIEs1q6k5611hcqZpcowV/W149E0Xn68xkDK8w5wlTs6ZFuTUiWbU9H6C/CNlISX6HgSCYaEcftpwCnqchOuNhhAyZV9tzzIkb1SbzLDX3YMilD4RS09FMz0ASM2NmT4h5j5uNC4OdM2o+lLCdeuSmF9h0GgJRqdYvDvBSz9XYmvUChu6fQn+Q7p9lvv1c/Hpp5PYvt1HVpaDrKzE1rLWrm3kiiu2U1wcRNfh3XdreeutIeh6dBW6e76b9bvDAuIL6Xy4XmPljgCTBkUXiZOsHD6ztqFp4d+FmcJUq5+tPWVC7tf/0iCgSEiOsUm/o0JrEiyAxoCgcKduK1qfur5ju1EZWZNsMcr4xrGFScGWZgZrHY6KFMz0BnRXeBiA6dcwypMgTg6TZp9TCfS/GkfF+wgpCWb9iGDez+Nz8E7I6nqdBRUGDVb4o6LAq3N3oZvnxjUm2DJFVLp9Cf5DekSWnU6NESOSE20GAI88Uk5xcTiskmnCkiUNrFrVwOTJKVHTNwb3F7PGEFQ12NfMjg1MwKPVU6TvREcwLTCRTGnfx3OaqbNYt6iNHDLPghMs+1rQuccE+HS1QbUnnCavt8mMo+zDRHnF/nN/WULiEfbzgR0s1ycZfL0xk9CAcE1IlKTxy3QHYD+W7WDx5z+If8R9IC3QulbT8sFSHYBG64BnzlSOJJ0W1TyoaG8s68B1STBo3/Ry6UQ/XxXp7GkIi8TovhbThtgHzBUITved0Gp7zrYMqoOS+bqJDtwadJAn7UXrjMkmFbU+3lrsRBOS2873MyzX3v5pgVFsMnZSp4eHGmSZqYwPDm21fS0xOcviIV8yzy3PBODaIX6Ozmw5oPBBI/Tw0s2ZlGEyOsVivTec1yzD4uJ+0afBUXQCemAJ3gOzHF+CQckNNxSzZk0DTqfGnXf2Y+ZM++gTv7yhD5+PzqWxXwqELAavK2fSJPta4JmHmViykde/cRLy1fHUpQ7SY3gMSQm3rnOzrNrAEJLrhwS4YkDsgLmzTAezzNbXIK4+JcjVp7QuCO9AK5vLG49nsWs9QsKp/olkyvh6bl5a8iA/KZ4LQNBxHv5ce5f0tvCC08MHzkYsJMcH3fyPPz2ux+9MpBnw7gQvdxW6aTQFF/cLcG5OO3wEKOKD8h5UHCz33lvKnDnVmJEunTvuKOHoo1Po2ze6CHyenUXgaBd7O5IaRyVRaXnpj31t5exxIc4eF6KwcBu56bH7gp7Y6uK1Uif+SL/T/Zs0pmaaDE+JX3PZwTLM7MewBvt+tUNB3zYf55q/ogXqABCrdxLqcwTmkNPjcvyVup/nXR5qtfD92SW8DLcMzgp2jubm9iDHJVUfVhdBdv/K/w/oAcNO25cNG3xNggVQWhpk82a/bfrVNTpmM8+HUr9GQX38bsPqOr1JsAB2BzRW1sT5NksPRnAuRvBjkIn9Cnfs+KJJsAC0QC1GyRdxO/4KI9AkWAANGizXVXOZonNgGrGX7ogSrUNk6FDXfu7nubkOhg61d6kvLdOaB21HBECL40ft6FQTrUbCf4Cl0BuLI9PjV8sSsppU7wxSGmeR3HgZyQ0XtChce0SIf7rK+berAi/RvQybU1EjePAVF3//OJcab+y0Zu40LGOfE4vlSMXMndqqvLSG8SEnac0cE9wSxh9EU6pC0Z70RNHqptnqOB56KI+dOwMUFPhwOjVuvbUvubn2hdpgj8UmS4cUwAK5Czg89jm+KNZ5YY0T0zeYJwcIspLsmxLP1YM8/bqLhj0AkpQdkqEnxU+0XL4H0a3vARCYGOaXGME5hJwXR01fIQL8b8oOdkRqJ4uNev7sHUyyjdtTeY3gzHtS2FSqA3ksLTSZ/5CHjOjOlQTzz0MrX4mj6KPw+tAZhIafc2iZbMY008XlgWQ+cviwgGkhF+fHuWmwzC+4b7OLitqh/CZbY1JG4ppyFV2LkN5CvaMbzv+qROsQcbk0Xn11OFJKRKywEBFmjAzy3w8N/JGJHYdkWkzIs699fFakc/2HyZQ3aEA2294y+egyDyk2uvjYiy4a9uy1Q7Bji8ZnXxvMPC4+zXgC7wHrJkLW2qZ/ybWnSbAANht+PnTWcIHN4Nsn3nGyqUCHSBCKdR6dfy10cuv59k1y/mMfwn/MgxGD4u+efYM/nev9aeHDxzmOYHVQcM43KWxs0AEX69eavDiugaOVcClagWn0vCJcNQ/GidYIFoBuAHpkQK8AacQuZ//1nSsiWGHWVmgs2RHjQY1SCYtn8ed3XIcp+jetm9oIQo7zDsacGC4nsKtUQCngiSwlsHtXK66tEO0iWE2Hj/yLN++XGxHBCrPTr/O3EjWNiaJ1BHRnzKU7okSrg/mw0IEfDZyAE3bUaXy7094FyHXAdCROHVKd9sX+r2f5GdRvX83tqDEmJ0+On7OEZUykIenfBIxzCRgX4U16F6nZB4T9ib83eea+l2e46WJGwH5IgKteQHNzg+DwxMPyjqUuCFUBgWwh+lEvQ6IfIONJrZiCRqEACKHHXLojPa9umWDSD4jhluaW9E21L6TuPd7Hmgqdwiodh7A4aXCIqTGaE4cOtMgdb1HxncDhhMk/DuGO84e7ZUyi0XihVWn7SidPeQfxtrMKA8El/t6kxHiZ8oeaaJrEijg/GLokf2jXaiq77Xs3H+42MBEc1SvEi0c1Yth8Hp6RHeK4zBCLq8Ppx6aY/G64vfepQtEcswcW4T0vx62gtNTPggV1DBjg5PTT01ts+isrC/L++3X07+9g5sy0mOkfOdXHxkqNTZU6yQ7JBYcFGdvXvlAemC557Wwvjy914QxU8sezU9FimHP/XBdfV2sY5/kJNAheXu7g4ilBxg9OXMHfRzr4pT+nVWlv+XmAxcsMVn2ng7T40VTJFRd0nd7kT8p1Xit10GCGVerjcgePb7G4PT+6EBkavD2+gYWVBlt37mbWuN6kq7dS0UrMblqbikWHvB5CiOeBM4FyKeW4yLZ7gWuBikiyu6SUCyK/3Qn8DDCBW6SUCzvCToDVq73MmlVEcXEAl0swc2Yvnn9+iK0QFRT4uOyyYoqKgjidcNpp6bz44kDb9L2TJR9f7WVDpUYvl2RIC/NRFe0RXPJSClv26Ogil1KvyRtXNqDZfLkX+iHpjlr0XAsZgsBGBxt264wffFCXIWE4nfDiQ4W899DbeBo8XP3QL9D1rjMfVUG93iRYACEEmz2xW+ENDc7oG6Kwtop0o2tGh1ckhraKVrzKZCHE6cBThKMg/lNK+WibM9NKOuqb7gXgaeDFA7Y/IaV8rPkGIcRY4FLgMKA/8KkQYqSUsuUBPnHgoYfKKC4Oe6r5/ZJFi+opLPQzcmT0eCn337+boqJwTSAQgEWL6lm71sfhh9vP5+Qy4Mh+rav53Pepmy17wg+mKTW+2CpYVqxzzNDol8N7eiN65NjCAGN0EL3Ooqt0X9aXlvLuuWdTs2ULAO9sep+LFiwgKSv61CqdjVP6hvhruY/QmD0gJNbmDGbmqICzivbBT5udLV7gEMvkyM/PAKcAJcAKIcQ8KeX6thrVGjqkJJNS/heoamXyc4DXpZR+KeU2YDMw+RDOzQMPrOOMMxZz0UVLKS6OPVrVsvav+QQCEp/PXmBMc//0waDcb6bkQyV4QITtoAneGK1lgw5satQhOS22Pf/a5eDM71M4Z20yy2oTK26r/vKXJsECqNqwge//9a8EWnRwZKcHyT1pBymD60kZ5CHnR7sYPqAh0WYpuikmRszFjjiVyZOBzVLKrVLKAPB6JG27kujP75uEEGuEEM8LITIj2/KAHc3SlES2tYmHHy7g6ac3s2RJJZ98spvLLluGz2dfabvyyt706bPvZh9+eBJjx9rXmmbNyiI7e18V/YgjkjjiiPhFsbxuqp9+afuE6Ij+JscOsbf/Z6ZG80rcWAtOjBG1/b0Kg/uL3XxZp/FFrcG1hSmU+BNXM9D08LW00LAiLuZ7tyWK5wscnPheCse/m8Ifv43t1bLQ0UiVc9/9aXCavOFUoqVoH0z0mEsbOJgyOa5ldWtJZJfvs8ADhIftPAD8CfgpRB0ME7OqUFhYaPvbokUl+P37SvGiIg+LFq1nxIjowjJuHPz2twYLFkiysgTXXw/btm22Pf7IkTB7tuD99wUZGYIbbjApLt5im/5gyQUeOCaVdzZlk+IwuXFiKaXF9qKVjsXlw9bzqXs4LoLMqtnFzrJRtulfqR3KiHHf0juzEktqFJcM47WNqZybVGm7T3uSefY5fPTiWDbWHwlIjsxYzqknTIx5j9uT7+uSeWBtPtWh8KuytRZSGko4pW9N1PRmajl6/iBMfd+r5di1msLK1r3Licpnougp+c3Pj8+kpwcSZ0eMgy2To30Nt/t4jYSJlpRy996/hRD/AD6IrJYAA5slHQDsjHWsWA9ETs4eaBbFoVcvJxMn5pOTY18bys4OMmBADX37ujj88JanocjPhyuvbDFZm8nPh4t+BIWF21p8+Jc6PyHZtZpztW8A2OVI5rhek+htRffey/VuJSVnO3pkbNDIoQUMT5tGfkr7vGQt8dJLAVYHR+OLvCcrQ4MpLktixgz70FjBECxfH355pxxmYsTxPX7nGxfVoX3n9pgGa82B3JCfHTX9qIIPWb5zJ1/1n4wpdMZVrufuzfMwJj3d4rkKCwvbrXDrjPS0/LYH8RyL1cYy+aDK6niQMNESQuRKKXdFVs8D1kb+nge8KoR4nHCnXz6wvK3neeyxI9m+vYHt2xtISdH5+c+HxRSsrVu9nH/+SoqKGnG7NS67LI8nnjisrafvcJYFqklN2lez1B0NrKzfzWl6dNEa1buab5oNZnU6A2RlVEEwzfYcVULyjhHEDVwUdOBuIVKEaKzGseZl0F0EjrwSHPax+5YvN/H59h3P4xGsWGHaipYvAOffncKKgvAQ3SljTeY86MUVp2AAE7JDpDosPMHwR6VLl4zvbV/TFVnjef7Tq1jRZyR+3cXUstUw9v9QceEV7UE8x2m1oUwWQL4QYijhODaXApfHzSAbOsrl/TXgRKCPEKIEmA2cKIQYT7g6WQT8AkBKuU4I8SawnnBshBsPxXMwNzeJzz47kaIiL717O+ndO3afxA03fE9RUTjsus9n8fLLJdx88xCGDbOJ2NrJKK8fjDOlAKcRDitR60+nyjMYekVPP9rMYq0sJyDClzjdcpFv2nvqlQuLK5Ma2aJLkPCeI8SrDUm4bIRLeCtIefUs9MoNADjWvo73ig9shevkkw3mzg1RF5ltJDMTpk+3f0yffc/JV+t0ZGQ6lqVrdf42z8ktF8ZHJk4baHL1qADvFzmwpOBHuUGuHWvvCWPmHI85/BqmbH8XpImZezqNo2+Oiy0KxYEE2ug9GK8yWQhxE7CQsMv781LKdYeSn9bQIaIlpbwsyubnYqR/CHgoXud3OjVGjrSvOTRn06b9O82DQcmaNfVxFa3XXtvCe+8VkZRk8NBDR5OXF79jj/Mfy+dlNQzotRVLaqzfeSJX9s7Arqn5pOAQKrQGChyVaFJwcmAIeZb9tXrCGQgLFoCA1ZrFPCPERaHoNSHXl79vEiwAY9c3ONa8QvCoa6OmP+ccBxs2WLz/fohAwM+sWakcd5z9Y1peLZoEC0BKQWVtfB1JHpzi575JfizA0QrXJf+Rs/Effg9ggdb1pjHxiwq2u/+JJQL0Dh5Pn+D0RJuksKGtfVrxKpMj47gWtMmINqLG3h9AdraLqqp9X+maJhg40N578GB5440t3HXXCqqrw+fYuLGGjz+eSVpafAq3K/uEqCk7iwU7HRhyBUtoAAAgAElEQVRCck9/H7lOe5d9geBS/2HQyshBP4hiKCAoYvS9Wj+slQgzdi3oN79x8ZvfuCgs3El+fuzxWVfPCPL6Z06qIxNpZqVZXHlq/CNo6BoHVzxoOq3d47Nvdf70jhuvdxQXTze48ezENSaaNLAxZTY+fTsAHn0DIOgTPDFhNins6YrxBYUQtwCvSinb5O2VaJf3Tsfs2fmkpe2rch99dBbjx7fsjNFa5swporo6m3Cz8AgKCiSrVlW0tBveBgi1spH0pn4BFozyMm9kA5NT4xu+aaTfAc3Gjmmmxji/veD6p96KmTGkad3sM5rA4VfEzZ49tQKrWRYtCVV19uk7G0VlglueSWLpOoPvitL4w5su3v5v/GtnQSSNrXDs8ugb8Gn7vJhNrZ4qx+K426OID20dp5VgTgaKhBAfCCEuEUIcVHRUJVoHcNxxWQwZ4iI5WScjw8GFF/ZF12M3N3k8IebN28WqVdHdoJuzbVsKkAG4gSSEyMWy7B8uvx8umpXM5OmpXHjVOJ54JrHTDXxj6ni8SQT8BgG/g1pPMktjOGLIzCF4L30P//ir8R91Xbg/K8k+yvvBMvdLB7XefY9xjUdj3pddp0nus28NSvfs+1qu9WosWB7fwuZpI8h0l4/pLh9XO/0EYoiXIdPR5P6OSgeuKzoP7TBOq92RUp4NDAY+BP4HKBNC/FMIcXxr9leidQC//vV6vv++noYGk5qaIE89tZXdu+3bzjZurCc//zOuumo1J5+8jJNOWhLz+P37Z9H8skvpQMYoFO592M2niwxKd+qU7nLz9N9dbNqcuNs2WICwNPyNSfgb3SRLGNnCPjJzCL4ZT+I79Q/I5PjGERw9yMRpNPN+dEhGJzA4MEBIwichjfdDGt4WKjf5AyxS3PvsFUIyMEYA5YNlk7D4px5ihwa7NPhcs/i9Yd98mmKNIDN4DJqVDFIjyRzCYN/P42aPIr74ccZcOitSyj1SymeklNOAE4BJwCIhRJEQ4m4hRKrdvp22/pgotm7d3xFj1y4/paU+cnKi12AvuGAljY0SCM+d9O239SxaVMH06dHH8YwZk8x//+ttatLKzXXYxjUEKC7R9nM02FOlsWmzxsgRiSmYbxOwWsL3SHTgVAEnxVlDK32Cvxc6qa3O5c7BkBHj3Zs1I8jH6w3+mxF+lKfXhVrs06qx4G8NLkzgF8kBemtxDLsl4UKfgy8tDRM4UpPMdQfIsKmMHnt4iN7TAzR87UBagrQhJtdf5iP6eM6Dp1BYVDe7P1LA9hbyO9x3Kw2B8wgJDynmCHRUTauz0ombAFtECPFj4ErCoZ9WAn8AtgO/IlwLOy7afl03xwfB3LkVvPtuBX37Opk9eygpKfbV5vLyA10NRMypRhob9+9oklKydWsD020cru67rz87dgRYu7YRl0tw4405DBxoXypPOSrEf74w8EVCKw3obzHxyA6JHRwVh4DXNCgHHEBWK8rWtbUaT251oQu4Z6SPgcn2hWaFT3DGf1LYVK8DeSz5j8n8kzz0srlENVJQdImONzKOaqtDp1YKMm2awGosmFmdyvpQ+BmY63MwP8tLnzgJ1zshLSJY4QvznSV4OGDwB1f0iTg/RFJ1XSOuy30QhECm5DHgsTiJ1gSpkWsJdkXy55YwyWz5KyPZGhKX8yval87aBBgLIcRjhMd01RIO2HuPlLK02e/LgGq7/bu9aL344i5mz95GdXW40PjuOw/z5x+JYUQvFIYOTY+M07IAgcPhRLObBwSYOjWT+fPL2ftl7HRqzJxpP3eU06nx8svDkFK2OE8XwC3XByivECz52iAY8jH7Duifm9iZbYWA1s2OBevrNC5dmUKJL3wNV9XoLJjmpa8reh6eKHBGBCvM2lqd57c4uXVMdI+6p2ucFAT3pV8f1PlrjZO7e0dv0v1Hg6tJsAA2mjpPeZ08kBafiRerJU2CtZf6GOlrCPu1iGZBjRvieHv7S42Hgg7+YgQJCZhialxrdvvXvsfQFUWLcIf+eVLKFdF+lFIGhRBH2+3c7fu03nmnokmwAAoKvGzebB/A9KKL+pKZmQIkAW7GjevFmDH2ERxeemkCp5zSh4wMnT59HLz99lHk5rbcnNIawQqng4dm+/l8gZcXntnAqSclrpbVFp4tcjYJFsBmr87rJfaOElaUAjvatr1E+ylWmR/tt1jHP1gudFiMEPuabvOExXWG/T2biWBEs/P3k3BN/MwBQFQ4aVjam8Yve2NtSAu3ESq6BSH0mEtnQwihAzOANbHSSSk32P3W7T+5DgwQ7nAI3G57rb788n4kJem8+24FmZkG998/DKfTPn0oBG53DllZvXC7BRUVib2kFpLZrmq+aQzhdEgu05K5OEZIJgApoTokcAhJWpzNT9H3VwSBpJdhrxL/MybAp2UONkdqW2PSTX42wn7c0o0ZAeZ7HWyK1LZGOUyuz7BPf21ygDk+BxvMcPp83eSWlPiNi8oWMMcV4Hd+B37gf10hJuj2+c1C8DqShyVUeTz8KjWVSXH8ltzpE9xSkEyJP3zMAo9OX6fF9YO6zmzQCnu6Wp+WlNIUQpiEa1ttat5odY6FEJcBq6WUBUKIUcA/CI81vSGWKiaae+4ZypYt6yku9pOUpHHaaVkMGRJ7sPB552Vz3nnRHSl+ePwyPvigDhkpl2bP3s2xx6bQr19i3K7/n1bHcw/3wV/sRuiSguPrmHxBkCFWdHsCFlyxNpk1Xh1DSM7NDvHQcF/c7LljpJ8lexx8X6+jIZmSaXLZQPsCs1+SZP50L09vdFJXU8W901JjOmL00SXz+3t5uiac6ObMAL1jiESmJlmQ5eXPXicWcFNygL4x0reFP1e5We41MBE84QrxYr9GbFqjAchD4xmgcNdu8vPjNyYQ4OtavUmwABoswZc1DiVa3YS2hnFKME8CbwohHiYcjLfpBZRSbm1p54OR6QeBYyJ/P0Y4YKIH+Ctw0kEcp0OZODGNjz4az2efVTNokIvjjovfGCGA9RsCTYIFULozyNatgYSJ1quvpdHwXWpTE9DuhRl8cFQVNw2Knv7hIhefVRtNc1e9uEtwVu8gUzPi0wyZ4YAPj/Hw4W4Dh4CZ/ULEqLgCkOKQjOltURHykWS0HOIq25Dc16f1H21ZmuTeOPVhHcinXp1X6x00ROYw+7jBwePVFrdntc/5WmJUikWGYVETCtsjkAx0da0mZoU9XbRPa++UB6ccsF3SijAyByNa2VLK3UIIN/Aj4EIgCCRm4qWDIDfXxZVX9muXY++sSyas3WGEw4lwJ85FWFS49uuzkF4DrdQNg6LXJop8+yZbBKg3NTY0aHETLYBUAy7Ki+49dyC1QThrWQpr6gxgKO8vM3l/qpekBL6bVV7B4x87CZqCm0/yMyDLvma2LqCja/XckvMPXJqPl8uvZnOwbwdauz9jUy1+OTDAK7uchCSMTjG5d0RiBFQRfzpjv1VLSBljVtpWcDCiVSGEGAEcDqyQUvqFEMnEa0BJJ+LNN6uYN6+GjAydBx/MIyPD/jINnTSIrUUmNHpB0yB3AJrTCcSv0P+wWOfVQhfBhqE8lSfIieEyfvUYyR2rTYL+8MOc1TvE2TFG/56SFeKTKh1vpI+nnzPEiRmtE5j24PHNrohgAQhW1ug8X+zkxmGJicdX0wBnPJVCwa7w9flkvcHcm7wMtBGuU1NqGNv3BoYnbwTg+PTF1NX8GWifj6bWcMcwP/8z2I/PCtd8Fd2HrtanFQ8OJscPAKsIl8aXRLb9GPgu3kYlkpde2sNvf1tKTU1YdNata2ThwpG2zhgzf2SyomAkdZFQQqOGmBw23BM1bVtYUKxzy5fJVPo0oDfnfWSy8EwPaTZN2T/7cYiFc02+WiEQGtx5d4ABve1FbkpeKUdQy+byAQghOXZYAdkp40AeVDiwuNFgHvgNJPD+YNv+zC/V+duWsL2/HOFnZv/4fTC89JWzSbAAtlboPPmJiz9dEr3fr3/KR7giggWQ597BqMznwHd33GxqC249vCi6F12xeVAIYQA3EI6E0YdmFR8pZYuhnFotWlLKFyJzqiCl3Osz/jX7BKxb8O671U2CBbBhg5/1632MHx/d7f1n5wSxJHz0lYNkl+TRm3ykxi8oPK8WuiKCFWZ9tcbSMoPTBkWvDb3xpoOv5xvU14b3eeYRF2cfFyInJ7pwLXUWMmrUdvJHrEJoEiFgbUMmU4LD4peJg+CGoX4+Ljcobgy/jMOTTWYNtK9lrdyjceu3yZRHnA021uv0c3uZmBWfiCF6lG8VLYaGiiief9G2KRTxoCuKFvAEYT+IvxOe7uRu4Hrg9dbs3Oq3SQjxrZSyoZlgIaUsB+YflLmdHIdj/xIpKUmQlhb7Ml17bpB3ft/AS/c3kte3ZU80C4tqUU+DaNlLz32AZ5tbh142A3MB5n/ooLZ2n71FxTqLv7R/sB2mgy8eO4k5v7yMOb+8lO/fnECyjK9HkkRSL+rxCG+LaYemSN6e3MAl/QOckV7Je1O85Ljt8/v2DkeTYAHs9mm8E2Mc2MFy9dBlTHSvblof49zAHfnv2qZPDZ6OKzSmad1hDiLD/7O42aNQNMePK+bSSTkfmCGlfAoIRf4/F2jVxG0H0zw44sANIjxCNjGf5O3EAw/ksWWLn82b/aSkCM44oxfDh8fPscJHgL9bH7N2ZTJpmUFOOyyTUwOTbdM/OMXHuiqdghoNl2Zx6kCTKX3tm7+ys/evYaSkWAyyccIA2PrWNIqXJmFFokQUvDee4MgADI1PTcXE5N3kuZTpuxHAkOAQZvpOR8ToCs1PtfjbhEYKC4sYmJwf8/hDUyWGkIQizicOIRmWEr+4jJk7P+bznKf5f3W/xCddXJ/+/0jbfRo+ZkRNr+Em1/sMdc45WMJHeuA8DBl7TrDOhpTwXZFGjUdwdL5Jqgo92GnpojWtZGDv/DeNQohkKeUGIcSE1uzcomgJIV6M/Ols9vdehgDtPr1yRzJypJuPPx7J0qUecnMdHHVU/GYVBnil6huePPtUqtb3xpEcZNl5W5jwTA3ZVnRX/H7JkoVnefhyl4G3YjsXHtWfWME07p/tY906nQ0bNJxOOPOMIJMn2Ytc4XZnk2ABNHhcrN5qMjFOovW1awXbjCJkZKLIjc5NjAgNZ1SopdjwrePnwwMsKtdZvif8KE/tHeLqYfEbgxTqN4EUF/w64zEApO6iMWd8zH00ksgIxG/OsI5ESvjFX5OYv8KgMSAYPcDi3bu85GQkNnSYIjpdVLQKCEd1X044UO69Qog6oDTmXhFaU9PaYvO3BJYAb7XOzq5DVpbBmWe2fjzXkiU1vPRSGdnZTu68czDJyfYP0iu3j6FqrQF8T9DjZOO7g1j+y92ccZj98dOdMHNwiMKAN6ZgAaSkwPx5XrZt00hKluT1j13YTBkZYuFqB42B8IH7plscMyq2I8OGMo2/LHLiMuCuGX76pNqfo1qrbhIsgJAIUaVXRZkCuW3oAl6b1kixN2z/4BTZ4jU6GMxhpxE4YhaOwg9AWoQG/ojg+Gvjd4JOxnfbNOavMPBGmlzX79C560U3z93SmGDLFNHooqL1K/a5V/8v8CyQBlzXmp1bFC0p5X0QjrwrpVzYRiO7LQsX7uHmmzdRXh7+ul++vI4PPjgChyN6P1hDcQj4Bgg7F4Q8VVA2CmKI1sGi6zCilVOX/PLUIP8tM1i83oGmSX55ip/RA+z3LdilcfE/UtgRme/iq60GH97iIcPG+WRMYBQb9G2YerjQc5mp5Ad/0NJ8SAgBQ2II54F8s13jsU/cSAm3nORn2rDYIu0/4SH8x90H0gK9S0YgaDVV9YIG//6qv/eDRtH56KLjtFY0+7uQ8EzGrSamaAkhjpdS/jeyGhRCRI18IaX8z8GctDvx3HO7mgQLYPXqer77zsPRR0cPx+MKVLBXsMLU4bQagF7taqcdiwI6q2YYeE4XgOBl3clPrCDZNlN1/HmRk7zxS5gxeTVSCr5Y8GPmfDOYnx4bvUluj2cEn1edwuDM75DAlopjOT89GxyJaW7aXK4x64V9ovtdicYb1zZweF4LIq/1jPEwk0aajB5gUVASLgwzUy0uOEaFfOqsBDqvs8V+2GnHgbRGS1p6E/8KjIv8/ZzdeejEzhh+v8l11y1lw4Za3G6d+++fwAknxG+g54GzlhiGsK1lAQSDP/Sg0/X4jSuSUnLbbUtZsrQMh6Fx443juOwye2eGlxudlDcboL7V1PnIb/CTpOgFVdaI7xkx7UOSU8M1p959K3GsvwaI3pz6Qr2TxfVHs7hy30wDb2uN3JqZmMHCr65wNAkWwM5anX9/5eSxC+MXb7Erk5YEc+70cvfLbhoDgguPCSrR6sR0oeZBO/1oTqu0JKZoSSnHNft7aCtO2um4/fZVzJ27o2n91luX8/nnM0hPj49b9F13DWb9ei/bt/sxDDj++AyOOMLeeSMjwyLcnKsTvkchLCt+ovWnx9fwwr8LsczwMX9950qmTMlh2LDoNb8MsX+Nx4mkr2Zf65h2zBp2pOzr3+iVVcvYCQUQmhY1fZ8DXPYNJP3iHKD2YMhJk2hCYkW8DQWSnHTlZNCc3CzJ86oPq0vQVZoH46kfLY7TEkIsFkL8N9YSL2Pag61b95+Cb/duH8XF8YtY0b9/KsnJ43A6h5CcPJIRI8bEnCtr2DA30AA0Ao1kZoYYPDh+o5Ff/7CqSbAAPDUNzFlYYZt+dpqPCUYIA0kSklNdQU51xnCp1zP2i21oWA76yD626W/P9DPZFcKBxI3kxKQQF6cl7sv9p8cGOC4/hMuQOHXJtGEmN09XsfgUXRMTI+bSHWlNrv7Z7G8BPEM4BEeXIC9vf0Ho08fFwIHxc2O//XaLDRvcQB6BALz6KlxzjWTYsOjCdcOvxvDWBz789R7QNLJHDGLEiNSY5yjaIXjxbSdWqC93/gpcMXwBtKzeILbRFHrelYTR136alV4afJjl5ZuATrImOdKwYnrfHe0/kQp9F5X6LgQag4MjGWzaNz+maPBBnpdvfDoOARNcZsyIEu2NQ4d/XdPA/d+6MS24Z4Ift4rHp+iidKHmwSaEEOnAvUQP42QzH8U+WuM9+O8DTvj4gds6M2eckcucOdsIBCQgycqCXr1il1ILFuzkrbdK6NPHxezZh5Gaan+Z6ur2X6+thfJyGGbTMnvHs+n4B/wYgo2gG2wOOlhV4OHosdGb5NZt1Lj8xmSKS3WEGMiKNSbvPe/FYZOFWb+cwO+21GKW7wahkT7+cM6b3otY8/murdF4bpsTly65d7SfvjEibujonNFwBQ2iHg2dJNnyB4BTwNSkzjEdRk0Azlqcyrq68Mu+4kuDD07w0idGnhWKzkpXFC3CvhIDgPuBl4ErgV8D77Rm5+5Zf2zGc8+tIxDY1xxYVGSyZUstI0ZEdxx4660d/OY3a6iqCjsKrF5dzYIFx9s6V0yfLvjqK4k34l8xbBiMHWtvz4YdethH2xmOZWiZklWFuq1o/f6vLopLww+mlIJl3+p8uUJn+jHRReCG6Sa77z2ZLzbq6BrcdoqfwVn2grGyWuOqb5LZ6Quf45sag4+O8cSMBi4QpMj4TlbYUfxji6tJsAA21Os8tdHJA0eoJkJF18PfNSeBPBUYI6XcI4QwpZRzhRArgfcJxyWMSbcXrQORUu43aeOBvPba9ibBAli3ro716+s48sjoInfDDYJgEP7zH4nbDY8+KkhPt2//yhtsUV6uwV6NcsLIGNEnDrRVyvBwITuEgPvP8sNZ9mma8+w2V5NgAWzw6Cwoc3B5jNmFD5ZnFjp5d7kDIeAXJwe4cGri+rSi3fv4BX1SKDqWLtpvpQG1kb89QogMYBdRQgVGozVhnA70rzeEENPZvx2y047TmjVrDEuXlhEIhGsbgwf3YsQI+zFRur6/4DidGkkxZiAUQnDLLZIZM3xkZzvIyor95fPoLxq5rCKFqjINhGTSZJMfjbWvCf3vL/x8u9agZJcGSCYdafKjyfFravu2/MAapGR7NTAwPsd/f6XBH+e5qGkIn2d7pcbIXJMjBidGKn4+PMCcEgcb6sP3ND/V5JaRiXG/VygOlS7aPPgd4f6sz4AvCftJeIBNrdm5NTJ9oH/9HuD5ZuudepzWp59WEAg4CccNEtTW6tTWBsnIiC4uv/vdWAoL6ykqasDt1jj55Bzy8+0dJUpLG5k6dT0ej4amSX78YydvvjnONv2UfIv3H/Dy78+d9O1lcdOMAI4Yd2HCYRZv/93Lv95wglXJvb9OxRnHFoEGnwA3+4KqOASby3Q4Ij61ofnfOpoEC2B3rcbHaxwcMTgxzXFZLsn8E7z8eZMTU8LNIwMxo8grFJ2ZLipa17Kv0nMz8Ajh6ApXtWbn1jhidMnxWXvZvt1LOJvhrJaX+9mxw2srWocfnsFHH53Axx+XMWBAEtOn943pwn7mmeuprw+PSjdN+PTTIMuW1TB1qn3swsMGWfzhqtYPZh09wuL3d/soLNxFkjt21PODZaBuUZaqgSnCj5ElOSYzfjW50Xkmhi4JRSZyTHFZjBuYuJmRAXq7JPcd3n6iGZDwvsfAJwVnpQZJV9NpKdqJrjJOC0AIcRTgl1KujaxnA08ChwNfAdtbc5xu/zoNHry/d1tOjptBg2J7vPXr5+aqq4Zw0kk5MQULfug9aFk6338fv3Fg7c3c0xvos0eiBSWaXzKhzuSnR8avz+nm0wPMHB8kN8MiL9PkkmOCnD6+c3gStgcBCefuTOHa8mRurEjm9NJUqlqYeVmhaCtdbJzWk0DzcET/BEYCfyMcffUPrTlIp8tVvPnjH49mzx4/hYV1uN06v/3tkfTqFb/2tbFjHSxevDfCBRhGiJNP7jrzJyU74PljG3hqi4skTfLosfENZ6Rr8OJNjdQ1hv9O6Rqh0trMm/UOvvLpyEjrx/qAzsNVLh7LVmGiFPEn0LW8B8cAiwEizhczgHFSyk1CiHnAUloxBrjbi1ZSksFrr53Q6vSWZXHOOR+zZk0VDofG888fz/HH97dNP3fuOKZP/47Nm0NoGjz6aC5DhybHw/Q2889/Gsybp6Pr8LvfBZgwwb7PZmmVznWrkynbO129V2fhMV6ynPb7vPqqzuuvG2ga3H57kGOOadmpIj1+QT86NR5LNAnWXrwtXJ6vv9V49Gk39Z6RXHWRwVUXqlh/itbRlZoHCevNXq+nqUCZlHITgJRyR0TIWnWQHoFpWuh6y62hF1zwKYsX72ZvP+GFF37GunUXkp0dvdTVNI0vvpiAaUo0jRabEw8WKSV33LGYr77aSSgU4J57DGbOtO9mfOMNjbvudhCITC+xdp2bLxc3kpMTPf1zRc4mwQIo9Oos2G1wpY3L+4cfatxzj4OqqvA+mzcLPvjAz5AhXcuZQcqwB1G8o3NcnBbk+Tonm4LhwmSAbnFThr134o6dgp//Xwo7dmqAgy3FFpnpkrNOjW+/n5Rh1369k7RUSgmmGmtwyHTCJsBYrAMuAt4ELgU+3fuDECKPfW7wMemQPi0hxPNCiHIhxNpm27KEEJ8IIQoj/2dGtgshxJ+FEJuFEGuEEBMP5dweT5DzzvuICRPeYsqUt3nvvW0x069evYdm3vwEAhaffWY/oabPZ3HJJZuYOHENkyev5eWX7eP8tYUnnljFCy+sY82aStavr+P227+gtNS+z+zPzzibBAugolww5z37r7EUY3+xcQhJVoxpQ+bMMZoEC6CkROPDD7vU1x73LHFx1MupHPVyKr/+wh1z3N7BkqVL5vX3ckWan4tSA7ye6+Uwl33p/PEXRkSwwlTXary3ML5xpZ5tcDBpTyoTK1O5qiaJUIK/L17/2MGUq1M5/55xXHRHMo1qXHebMdFjLp2M3wB/E0JUAWcAv2/22yWEJxVukY5yxHgBOP2AbXcAn0kp8wn7698R2T4DyI8s1xGe1bLN3HbbEhYtKmX7dg8bN9Yye/Zyamrs3xK3e/8bLYSwjZAOcNdd21m4sJbi4gCFhT4eeaSUnTtjj/vxeIJ88UUJa9ZUtmj/ypW78fv3OS6UlHj49tvdtum9jT+8pVV19rf53tF+Dk83AYlDSE7sE+L0HPuv/P79LZqHhHK7JUOGdJ1P5g+2GLywzsnWOp1tdTqvbHDy+sb4ikQ/Q/JMXx//yGlkXAzBAhgy0CJpP5d7Sb++8buehSGNxz1uNps6xZbOfL+DBz2J61gsrxI89JyLTdt1dla6+eRrgzv+4k6YPV2driRaUsovgUHAKcAwKeXGZj/PB25tzXE6RLQiE0lWHbD5HGBvDMN/A+c22/6iDLMMyBBC5Lb13Dt3Nuy3XlnpY8cO+5rKW2+djNsdviyaBtOmZTN5cl/b9MXF+wtgWVmQoiJ7USwr83Lqqe9x3nnzmTlzHjfcsCim/UOGpO8XwLZPHzejRtk7esw81wQH4cqiAGe64Jyz7L31spySD6d5ePWoBt6e5OWNSQ0xm8zuvDPEcceZZGRY9OljcfbZJqef3nVEa/luHU9w32PfEBIsL0vcy33SsSbnzQjQJ8siLTXEsZNM7vlV/Koea0MaFc3mSzMRFIYSl9/iMo2yqubFjmD77m7vxNxu+HHGXOyIV+uXEGJWJH2hEGJWS/ZKKeullKuklPUHbN8opdzZmjwnskE0R0q5C0BKuUsIsVcZ8oAdzdKVRLbtsjtQYWGh7UmysyVC7Avfk5VlEAqVU1h4oIaGcblg4cKjWLmyln793IwcmRLz+Lm5PjQNrEi53bevQNd3UlgY/UW88861rF8fPrfHE+S997Ywc2YvRo1Ki5r+iiv6smZNHzZv9mAYgvPOGwBUUlgYvZb2k/MFawuGsX59MrohOf/MSlx6GTGyAISrtQBbamKnA3jsMdixw4nTKcnNDbJ5c8v7tJVY174tjBJppBnDqQ+FH/1UI8RoYweFha3IeDtx6zVw2ZlO/AHBgH5+Sl+stxwAACAASURBVHa0vE9ryRJOslNGU6GFCzCHNBlUt5vCyrL4neRg8Ov0zRhDaWW4dqUJi5xeeygsLEmMPR1Efn58x1fu5RD6tF4AngZebLZtb+vXo0KIOyLrv2H/1q8phFu/pgghsoDZwNGEm19WCSHmSSmr22pUa+iMvXjRvvNjtsLHeiCefXYYUn7Bhg3VuN069903mQkT7L0B93LkkS0mAeDJJyWh0Da++86L06lx1115TJli7wRTW7sRSIoskoaGWtzubPLz7eMm/fSn+bz2moVpernuujQGDoz9ZTrvZfD7fRgG6HoaEF0QD4VRo+J+yB9QWFgY95c9Px8qXUHe3BR+zM4aFuIXk7MB++lb2pvCeo2/V7qpDni4uZeD03LjN44tH3jUF+Qpr0ZICiY7QjyUk4Ym4v9MtJanfh3iwedC1HkCHDXW4C+/ScJhtE+h3t1paxOglPK/QoghB2w+Bzgx8ve/gc8Ji1ZT69f/b+/Mw6Oosv7/ub1nT9gCYV/CLigiKKAgAiOgLCoIM4M4g+MG81PGDZ1R3GVcXp0RdV73EUFfEBHcEBAXFBREUQKELWASlgSyd5Le7++P6qymOgE6dLpzP8/TT1KdSvWtrqr7vefcc88BvhNCVHi/RgHrpZT5AEKI9WjTQO+cVqMaSChFK0cI0c5vZbUDcv3vZ1Mz810HoEFmY11YrUbeeKN2+sTgYTIJ/vOfhmexOngwHi2Oq0J4rOTl6fv0V6/2ceedkvx8AcRyzTWSdeskCQmBw8CsYbweaunSfSxduo/y8nLmzzczaVKXoB7/tkEubhvUNPIN5joE134bTUapEUhi33YfrwwpY2Sb4AnX1TYPV9tCm4WkOmOGeBkzpLRRBiXNjSDPW52q90vv/UYllM7kNUCFD3Q2sLra+9f5/agXAkUVX+TZpKzMi88X/DAru91Iza/dyA8/6C88Xb5ckl/Nk7l3L2zZErx2SQn3FNu46GQsI/JieKss+BURH3rHyrC7Yxl+TywvfRr4+F9+eYQHHtjK5s05/PRTMXfdtZkdO+oPWGlMXvjSzPAnYxn2ZCyPfBzc0cAnR01+wdLIdRp461BYLRhVhBAPxoCvIKHn/Tplr1gwOFsh7++g5ZbqJYTIFkLMARYBY4UQ+9GiSRb5d/8EyAAOAK8QhCrJTqeXPXsKyM0tr3ffkhIPV165hQsu+ILBg7/gzTd/rfd/XC4f6eklHDtWf9aDpKTaxq2RoUP100pFRdW8L8xmaNky8Gd4PJJ9+yRZWfXfPy+VWXir3MIer5E0j4lH7Tb2uIN3W7zzlZlX1lnYnWVkV6aRp1bZ+H6v/sO0atUh8vKqAhFycsr56KPDQWvPqbL5oJGn19vYdczI7mNG/neThRXbgyfsbWxa1GZ1kgIs7FYoqhPkNE45FUFvDfR+BdUr1lDOVvTgTCllOymlWUrZQUr5mpQyT0p5mZQy1f8z37+vlFLOlVJ2l1KeI6X84Uw++8gRO5ddtoZx4z5k5MgPeOqpHQH3v/vuNDZtyuPIEQcZGWU8/fR+cnL0xSgvz8W4cd8yduw3jBq1iQcf3BPw+CtWnIPFUpExw0jXrm245hr9aMCYGBvVL5MQZhIS9Dt9u10ycaKPMWN8XHqpj9tvDxzZt9VtpLzagClXGtjiDp7L4cs0E3ZHVfvzSwx8vUv/YerZMwGTqao9NpuB1NQGLZRvFL7ab6Sg3KA5PTqC3WPg8wCie6pc3s7D2LZuoowSIz4GJXm4v79K+aRoGC4sAV+nyKl6vz4DxgkhkvyRhuP87zUqTTEQI6jcddcW0tI0/1pJiZvXXtvDrFk9adu27lRLBw/WDIc/etRBdnY5ycl1zzvdfXcaO3YU+Y/vZcmSTK67rhPdutVtPQ0YEM3OnRfz3nv5pKRYmDo1KWD7T5wwodUOcQMGXC4j+/aV0bNn3XMUCxdKvv++anvFCsnMmT6GDq17fNLf5ONjp8TtF64k4eN8U/DmUwanelj9vRmHWzt+fLSPITptB7j55n588VU+m9OSkdLL7y4sYPr07kFrz6li7poHV6RAK4PmDMnzIaNPAvrlak4Fg4ClF5Wzo9DFvsPZTB7QDlvTWl6jaMJ4fad3s/i9X6OAVkKIbLQowEXAcr8nLBMtewVo3q8JaN6vMuBPAFLKfCHEI8A2/34PVxgfjUnEi5bd7q617SIvz6ErWrm5ZdR010qMAe6LoiJ3rW0PublOXdECSE42M3euTl6lWgwe7GHjRhMOhzZq6tDBy3nn6YvKiRM1XUulpZCVBUOH1r3/32Kc7PMY2Oo2YhTwR5uLgZbgrbu6YaybtF+NfJ1mQhjgqovcjOyv3/78IiNZ3omUWbUv/WC5l2K7nYQQBbsdaifALapuh9YGjrQJroNCCDgvyUvsSbsSLMUp4TnNNXdSypk6f7qsjn0lMFfnOK9Ts75ioxPxonXRRcls3ZqLw6F1lJ06xdG9u36Giy5drBw+XIbmkpOYzYE78FGjWrF5cz5lZdrxu3WLoW/f4PWw8+e7yM01sGWLEY/Hwf33Q/v2+nMeV1wh+PprSaF/2VGXLnDxxfqRhgYBLyeW45P+9chBzk0nBPzrLw58Pu33+o7/7BsW9mZUPYg79xp5/T0L8/8Ummi/lq643zQ6yROLFgGqUIQWryfiu/DfEPFnvGCBtnh78+YcYmJMPPXURdhs+qd99dWd2Lz5BC6XJgxdusTRt2+C7v5z53bD7faxceNJbDYDTzzRj/j44E3UCwGLFmlzHA0JEZ4+3UBJiY/VqyUmEzzwgCA5uX4lCnbi2N8cv4HGia8OI8wXQn24OR5WFrg54tKuaWuzmwWBPboKxVnDG8LsJqEi4kXL4fDy889FHD3qJCrKy+7dRXTsqG8JZWTk4vHY0XIh+XC5vDgcHiw6Ne6FEMyfn8r8+Q1bb+LxSG68sYBffnFjsQhuuy2Wa68NbimTOXMMzJlzav9TWCgxmyEmpmHqVVgIJhPEBmdqp5Lb/+Riw2YzB37VHsY+3b38OYSlOlIsktWdHDyeK/ECt7V00s+mrCxF06Be0TJGXiRqxIvWXXdtZe3aqhQxCxZsZejQNiQm1i1CP/10Ap/PCWhh18eOGTlwoJBBg/TzD54KDz1UzOrVDrzequ0RI6y0b69/8zmdsGevgZMnrQR7LabbLfnDH3z88osmQlOnCh55RN8s8njguuui+OknE0ajZOJED//8Z/Ci3dq2lqx+uZSHXrdSVlrMv/5mJSkhtA9eD6uP1zvWv1xCoTjbuBz1rBuMibxI1IgXrV9/rRkNmJOjJcxNTKw7zLx23aykJBvt2ukHVZwq+/Z5KgUL4OhRH+npbl3RKigQTJoRQ/o+AyZzH66e5GXx08G7EZ94QrJhQ5UL7s03tZD5Cy+sW7ieecbKZ5+Z8fpLyC9dKrj8cjeXXqofXFHigI93mjAb4coBHiwB7jqXD27Ji+KHy4zgS+KWHMk7iWVNpg5UQ3C74aMvTDicMPFSD/FBtkYVikqaoXsw4tMrd+5cs8dITrbRsaN+L/LUUyPo168lUVFRJCZGM2/egKCKVs+ephrRiCkpBnr10p8Du22BjZ17jLgNgnKPiWXvWdi+I3iXLSND1pgzKimB9HT9/ffvN1QKFoDdbmD3bv0Hp7AMJjwfy83LYrhhSTSTXozBGSCj0L+PWdhUbKLMZ6AMExuLTLx9IvhZOhoLlxum3BzDnHujueWBGMb/OZb8wjBSXEV44TEGfkUgES9aTz45hPHjO9C9exz9+yfxxBNDdF2DANu2nSA72015uTbP88knR4OazmnhwngmT7bRo4eRvn1NLFwYT4cO+jfXtz+YtNy6Fu3lswi+2BQ8A3nsWEFUOyP0i4E+MbTpJBg5Un//yy/3EB9fpXJt2/q47DJ9FVq01srOo9r5SQTfHzKybKu+CGU7DTXK1XsQZDnD5zZd8YmZzT8Z8fm0c9i138jjL4U2EWRODtx0k5nrrrOwfn34fJeKBuARgV8RSMS7B6OjTbzzTsMT5j7wwA811l5t2ZLLzz/ncd55rYLSHpNJ8Prr+hkwauMxUDW0EIBBUhbEuISu55nxDYgGl/YhvngTCS1L0Ushds01bo4cEXz4oRmDAebNc9K7t35gQqmr5oMjEZQ49B+mP7ZxsbbQzHF/KqkOFh/TWoUuEONUOXZCIGXN88s8HrrOo7gYpk61VlrDW7YYePFFF2PHqmCSiKDp5EE+azSbYVdDraWcnNJa/+fj0KGixmhSg+jbq9ZckYAhARYXnyoPvWfD6aq6DU4WG1n2TeCxzG23udiwoZR160qZNCnwU3PrSBcdk6o6yB6tvcy8QF+EBsf6eL5rGWMS3AwzFfJKjzJ6RQW/g5WyqsZaUDHxmzSihhDmv920ycDu3VXX98QJA2+9FfFj1eaDo55XBBLxomW3u7jqqvc599w3GDr0LT74YF/A/bt3twIVoiAxmXycc07oFuYsus+BNaqid5V07uzjshGBRcsnIbtEkF9e/wj/eC1LCAGHyoN3W/Rp5+PdG0qZNsjFzAucfHBLKa3jAqvF2CQv7/Uu41/xB7goLngCXcH9i6wMHB3LwNGx3LnQFlTx6tDOhyFGauJlBKKgU4DF4KdLvkOQbRf1tj0h4bdlaqKi6t73bFJY6OHYMR9eb+SFZJ9V3PW8IpCIF6077/yCjRszycwsZu/efBYu/IbCQv0hyPPPX0qHDgIox2x2MG1ae1JTA7vzSko8fP11Hrt2lQTcrwK73cemTeXs3Fl/lof3t5hxtQQSgCRBWYIg64T+ZSv3wBXvRjHsEcGQf5pY8GXg+ZRRo91QkSBEgKG1ZMKQwNaT0wmbNxvZvt3YoA6/X4qPV2aV89LvHXRICm0n9dE6Iy++ayUz30hmvpHXP7CwbGXwAj2uuczDJYM9mGIkREkG9vFw35zgDnkXbLExfFUso1bHMvnTaBwBLtfw4T7GjPFi8WeO79PHy8MPh7aW2DPPHGfEiHSuu66MceP2UVDQDH1cwcJbzysCiXg/wZEjNYXk5MlysrJKSEysOwFu794t+eqra9m0KZuUlFiGDGkX8PhZWWVMnvw9GRl2oqJMTJvWgX//+xzd/Y8f93DVVbmkp7uJjhZMmhTNiy/qz5elZRiRJgF+Y+9EsYFdhwx0S6nbZXbvp2Y2P5QNuU4wCl79PIZpKxI4v23dYvHEJU7Si0z8ss+A2QxXnO9mdIcA4eslMGVKDD//bMRshksu8fDOO2UNzngRat5834IXUTlc80nBW6vM/CFIC5hNJlj5dBlfbTficApGXeAhWr/G5ymzNcfI2/vN2P1zfl8fEzz0g5UnLnTWub8Q8PbbLr791kBBAVxyiY8E/QQvjU5WlouXXz5BTo4mVNu3l3HnnVm89lrX0DUqnGmGeh/xotW9exLffJNdaREkJ8fQubN+7kGAli2jmDKlYat45879hYwMLdFfebmHZcsOceutXejdu+6sG/fdV8Du3VoHabdL1qwp4+abXQwYUPfEx/GDokahY+GSiADrXD9/NR+O+zswn8Szp5SPN0Rx/h/rtiaiTfDJ5FIOFhuwGSWd6nHdPfaYle3btdvG44GNG02sXm1i6tTgPT2/lgr+J92KvbgTj3UQtI0KnnUWF0fNOScBMUFOxms0wughjTPMPVgsKgVLQ3C0NPCIQQgYMaJpBF7k5Lh/Y1kVFESoSXA2aIaiFSbj49Pnn/8cyeTJqfTu3YLzzmvDc89dRnx88EKQd+8uqLHt8XjZvr1AZ28oK5O/2Q700HY2SziJVhCgFGQmBKoc0tJQ6y72SFJqv1eLDUeNPLDdxj9+sJFRHHgeLD+/5i3jdgtycoJ3G2WVCqZsiuG/h62szG/DlZtiOOkMXvTdX2c7ia0W2BFt9XH7nLqtlKbIJSleOsZWtT/e7GNcx/DpuXr2tNG5c9XzZ7MJLrwweOsgmx2eel4RSMRbWlariTffnNhox68rJWFysr4opqTY0UwnzfIxm8vp3Vu/0792mosvbkmjvPQIIOjUdRAXXaSfq/CeG23c8KOT0iKtY+vQxcTV4/Xb88VRI3/9Nppcf6HGXQVG1o4vpbWOdXP99S6+/NJEbq62f9euXqZMCd6M7/P7rRyqVn5+f4mRNzIs3NUnOMIyqL+PxQ+V8/K7VqSE2Ve5uOSC8Bnpt4+R/OeSMp740YpHCiZ2cvOHnuEz4x4fb+Stt7qyYEE2hYVljB7dmrvuahvqZoUvERohGIiIF63GpnVrH0ePVvjvJOAlKkCI9tGjh9DCelqjJeQ9SHp6PMnJKXXuf/JEOj7PTiqGTdLzJU7HaIive6Jk/PgoXnoeli0rw2IRPPJIHElJ+qK4ZL+lUrAADpYYWZttYlZq3R3hsGFeXnyxjFdftWIywT/+4aCtznzZ6WAx/PZY1iAn/Zwy1sPEUW6kBIulaSzAdDjAXo+br4Lh7bysnliKF4mlkZwlXq+WjsoWxPm4Cvr0iWL16lR/1YK673tFA4lQayoQSrTOELvd77fDiiZa5QFrRlksZjTBigd8mM3lJCXp9wyff56Ds1reo6ysUrZty2PixPa6/zNpUhSTJjUsrjnBUlMQLAZJa1tgkRgzxsuYMWUNOv6pckdvFxtzzOwu1qytcxM9/LlbcKPd7rrrOJ99ZkdKGDkyhuefb4sIdiGxU+DyKdFs22JCynNp216y43t7nRZ8BUusOaw1F+IDUr02FpZ3wlh7cdgZ8MILJl55xYTHA336+FiyxNUo4qUIAuFjZAeNiJ/TAjh5sozVq/fz44/Hg35srQKyF23SqUKw9DsQs7kv0BEtHLAlXm8f4uP1A0PatKnZWyQmmuncOfAcwLFjXh56qIAXXiiud1H1wvMdnBPrQRyQWLIkY9q5GdchdMO3JIvk01F2Fg0s566UX/l4ZCmxQRxaffBBMUuXFpGZ6SEry8PKlcUsWRK6xeNvLjXz3dcmvE6Bz2Xg6CEDV/1B3/2731DOSks+x41uco1utphLeNOaE7T2ZGQInn3WxOEsI9nHjKzfYGThwvDJ/djsUCHvkUda2glmzfqIQ4eKiI01M21ab5599jcVpU+bXr0S+fbbKjFMSYmmSxf9cLTNmytWnWp4vRa++spNly51X4pHHx3I3r3F7NtXjMHgY+bMrvTvn6h7/PR0NyNH7sXpLAcE//pXArt398BkqltIfaUC3xKB3As+E7gzBYwmkO42OglmuLmHi/3yBDEm/XM9HX780VEjGKa8XPLTTw6uuy6oH9NgvvzGCL6a4YyHMvTHkgeN5ZQYqnojn4AsQ/As0f37BSftZojyl5n2StL3RmjvFwk0Q/dgxFtaDz74TWUaJrvdzYcfHiArqzhox3/88aGcf35LoqMFiYkm5s8fQEqKviUUHV37KxfExelfhthYEx9/PIovvxzLsmW9efjhgQHbM2NGll+wACS5uUUsXqx/vg8+a2XXXiMg8HgEX24xsfaLyB3LjBsXQ4sWVYOGxEQD48YFN3rt+HEvc+cW8Je/5JOWFlhQZlztxmCqZg0bJMOG6fdEAzyxtPJWXR+bFAzwBK+IqDAJMPsFC8AoyC1SllaTRaVxijzc7ppBEU6nF7s9eI7gDRuyyMwspqzMTWGhg3ff3YfLpT8yvekmK6ZqnVRKimD06MDJ6QwGQceO0SQm1i8mLlftIBBZuZCzLkpLa5pUbo8IeimNtQeMXLM8mmkrotl+NLS33IgRMSxY0JJzz7Vy7rlW/va3lowfH7yFWvn5PiZNOsnSpeWsWOFgxox8du3SF67Lx3j5801OohMk1lgvQy/28Mrz+r1NirRwq6MdvTxR9PDYmORswVR3y6C132TmN3OyXbuqVEtNFhXyHnlMnNiNzZuPVIpXp07x9OgRPJfTW2/t4cSJqtW+O3ee5JdfTjJ4cHKd+994o5XyctiwwYPFInj0USuJicETiRtuaM0jj5QgpXbHWiw2br5ZPwXCnJkuvt1m4rg/NVRqVy8TRgfvbt+SZeS2T6PJKdOOn37SyIcz7XRJDF1HeOONLbjxxoZn2j8Vli8vY9++qkFLdraPxYtLeekl/YHJ0485efoxpz+arv5F7SM9CYz0NE5ai6GDvfTt7WPXHs0abdnCx++nhc86tmZHhApTICJetH79tRivt8r6cDg8uFw+zObgFEgzGmtaDmazEau1vizpVm67rXFqLP3tbwlI2Y1XXz2B2SxYsqQjnTvrt2fYYC8vP1XGq0utWCySB+9w0CKI+QGX7DRXChZAVrGBVelm5l+ob33cf79WHdnt7suMGQbuuSd8Os3YWANC1MwgHxPTNMLqG0JMNHywrJT7H7VSWmZg5jVOJoxTc1pNFiVakccvv5yoUZn3+PFSDh0qpH//1kE5/gMPDGHPnnwOHy7BajUwenQH+vdvnFF8Q7njjgTuuKPhI/FLhnq5ZGjjhLC3jZFoSwG0jttskKTE6a9jW77czBtvWLDbDUA0L73k44ILPIweHR4d5/TpUSxbVsZ337nw+aBvXxP33Rc4bdjZ4MABF3a7pE8fC1ZrYBFt3Uryn+cidEIk0miGIe8RL1otWtRcr5SUZKVt28AT78ePl7JuXSbt28cyenSHgGt4+vRpwdKl43n++TQ6d47jnnsGhHTND0CBU/BppokEi+Tyjh6MIZxGunu4k++yTfx03IjRILmkk4dpffWHh5s2Gf2CpVFYaOCbb0xhI1oWi+CDD1qyZk05DgdMmmQjPj6083jz5uXy0UelOJ2S3r0trFrVjsTEyCzF3uwIj8ciqES8aD377GiOHi0hM7OYqCgT8+adT6tW+tFWaWkn+eMf13H4cAk2m5Err+zKyy+P1hWi9PRiZs3aRkZGKVZrPnv3OnnjjSEhE66jpYKpa2PYW2jEJCQjUzwsH1cWMuGymWDNzFLScg2YDNCvtS/g4usRI7y8/76PUn92iIQEX8BouqaIxSK45prgRfSdCdu2lbNqlZ3SUs1f+dNPTv7xjzwWL24T4pYpgkIzNIgjPnqwRYso5swZxMCB7RkxohvTp/cJuP/DD2/l8GGtnInD4WX9+kz27y8MsP8uMjK0asdOp48NG3JISwscUv/JJ0auuy6aW2+1kZcXXHF7aJuNvYXaKNojBV8fM/HFkdCOqk0GOLetj/5tAgsWwLXXurn+ehc9enjp0qWcW25xMWZMMxxOBonjx72VglVBYaH6PiMGFT0Yebz++i4efngrhYXaxP/evQV8+ulkTKa69drjqfmAu1w+ysv1H/K6QuodDv39P/zQxO23R5GXp33+zp0mPvvMTnSAgfnLL9v5+GMHLpeVp5920a+ffiRa7Y92+wR2T/gEAgA89piTx04hmk6hz/DhUfToYebAAW3yIylJMGVKbIhbpQgazXBOK+ItrdWrD1UKFkB6ekFAy2n69B5YLFVfS6dOsfTtm6S7f+vWNdMsGQwGunXTnzN7+20zeXlHgD1AOmlpdrZv17eEliyx849/ZPHVV1ls2ZLP1VcfJS8vwDqwfi7aVkvY2zfJy+j2ETrkOk1ee62UUaNyGTUql8WLG1ZtOlxp0cLIO++05fLLoxk9OoqHH27FNdcEuYCYInSoNE6Rh5a+yEdFBJvVaiQmRn+Ff1paPh5P9RB5H6WlHt2J69zcmuHYXq9k/347LVvWHdJ++HAucKJyW8pM3O4O6I0fFi8+jstlr9w+fjyPTz5pwaxZdUekDWvr5Y3RZby6x0K0SfLgBU7iA69dblZ8842Dhx7PozhPu/UP/FpAj1QTl/+uYQmGw5HUVAvvvhu4ArciTGmG49GIt7Rmz+4FOAEX4CQqykunTvojzbS0vN+EyGdk6CdUjY+vqfsJCWZat9Zfg9WmjR1NQCuE1E2g2VS7vfYaJR9FRYHXLV3U1strl5bz/MUOWtaTsf1s4HLDd7uNbN9nrPHdhoLlG45VChaAvcDE/607GsIWKRRnQDOc04p40br77i9qbB85UkJ6ep7u/rG1UoobjYL27fXnAG6/vRtWqxfwIYSXdu2ge3f9/du3N6DdTV7Ag83mpVs3fcsvNra2q9EY0P0IkJvr44UXHCxb5sTtDq1olTrgivtiuPLvMUy8N4bpD0fjDaHbIuX8HEzRHjAngjkRo81Hu0HBy5KuiHykhLU5Rv510EJ6SYi7UJV78OwjhDgshNgphNghhPjB/14LIcR6IcR+/0/9SaV6KC+vOVPp88mAouV0OtAExQf4kNJNWZl+9oannvoOpzMbOI6Ux8jOzmTv3vxTak9xsX4v3qVLe7TaW2bAAiQjpX5xo19/9TJ+fAl//3s58+aVcfXV9t8El5xNnnzXytZ0E26PwOESfLnDxPIvQ5eAdcqYNlhbdwBbO7C1w9KqA9MnBy93nyLymZ9m488/xrAwPYop38fwyfEQRucqSytkXCqlPFdKOdi/vQD4XEqZCnzu3z4tWrSo2cELAcOH6xdQ1FI+aa5EcCKlJ6A7zuGoECE34KW0VEucq4fTWVNAPB5JQYG+aF12mZGYmB5AP6AfXbq05YIL9KMBH320nIMHNR+czwfffuth48bQhRidrJV81+MVHM8PXTTjp2/0pjS/agxUXpTA6sUDQtYeRXhR6IbPcsyU+cvJHHca+HdGCCtkKtFqMkwG/uv//b/AlNM9UIcONeevbDYjBQX6IjR+fHfiq0UupKYm0bu3/kh80qRUEhKq5rB69mzBOefop4iaOjWepKSqr71XLwuDBunf9DfeaOD22wUXXmhg0CAHL78saNNGv9N319Inr1cr5R6I//1fHxMmeLnySi+bNtU/6fT662YmTIzhiiuj2bgx8Chz9u9ctE2qOmbnNl6uGRk6ES0v/+13Vx6hbhRF8HH7BN5ajgsZSg+8u55XBNIUogclsE4IIYH/lVK+DCRLKY8BSCmPCSFOe/m+yVSzU7XZTLprtAD+8peBmEwG1q49A7N2ewAAEqdJREFUSFyclUWLRmKz6X9Ns2b1B+Cjjw4SE2PmiSdGEh2t7/669toEpIRVq4qJjjbw+ONtiIsL3PHfdZeBu+6C/fuP1btuad48K9u2eThyRBOKgQNNXHaZfnvee8/H449LivyxJocOST79VNKxY93C+NFHRh55xEZBofYdHsow8OGHpXTrVveTO6SPj//8rYz/rLFiMMB9f3DQsU3wn/IKsTbX43m84ToX768xcyBD+867dvYy9y/BK6KoiGxaWSSDEr2szxV4ESSafVyVEsL7J0LD2gMhZEiHCSCESJFSHvUL03rgr8AaKWVitX0KpJQ15rWKiooqG75//37d42/fnsfChTvIyXFgNhsYObINjz8+KKT5AT0eSXa2l9hYQatW9fvDPR7JkSNeoqMFrVsH3t/jgXnzTOzebcBggGuv9XDLLfrW0333tWH9+pqBI/fem8tVV9nr3P8f93fls89qWp7z52fy+5m59Z5HYyAlPPFKJ777WUsQPOzcIu65ITNg5o2cXDOvLklBSvjTH47Rvp0SLUXD8Uh4M78dmS4bl8YWcGmc/rrPCqoPNhMSEs6o86ne9yXODJwYu/Cdqsjn6p8rhDgMlOCPCJNSDhZCtAD+D+gCHAamSykLhNZZ/guYAJQB10spfzyTczgTQm5pSSmP+n/mCiFWAUOAHCFEO7+V1Q4I2CMGsj5SU1MZPLgnq1cfoGvXBKZO7RlSwSos9HD11bs5cKAcm83Atde25uGHu+juX1zs4ZprdrJ3bxlGo2T69LYsWtRDd/8HHyxgx47iygi9VavMzJmTTL9+dYfh9+zpY/36qoFLTAwMG5ZMamrd63r697Oybp1ESu07jIqSXHRhC1JTg1/fqSEZMd79yMxn30RR5tDa8+mm1lx+aSzTxuv7RlJTYcTwiq3OQWrtmdPcMoCE8/n+s/K31v5XiDgzF+ClUsqT1bYrYgkWCSEW+LfvAcYDqf7XUOAl/8+QENI5LSFEjBAiruJ3YByQBqwBZvt3mw2sPpPP6dGjBXfcMYSrruoV8gzsCxYcYvt2O0VFXnJy3Lz1Vg579+qXBfn73w+ydWsJRUVe8vN9LFuWwy+/1G0FARw86K4RUl5QINm5U//Ovv9+wciR0LIlJCfDjBmCiy/Wvy3uucfJpZd6aNnSR3IbH9OmuRg7NnQ+ih93GSsFC6DMIdie1lSnaoODwyF56SUnTzzh5MiRZugfUlQR3IwYerEEk4G3pMZ3QKLfmAgJoba0koFVfiExAcuklGuFENuA5UKIOUAmMC2EbayXjRuPsXRpBi1bWrn//oHExelPrBQU1AzpKSrycvSoi1696k4+mJdXc//iYi/Z2U4GDKh7LdjQoTY2bCin3F9MuV07IxdeqL/Y2WYTLFwoeeYZH9HRcO+9gSeFLBZY+V4ZR44ILBZo0wjzU6fC2BFuVnxipqBYE6qkBB/jRgQ3bCo7283jjxfh9cL8+fH07h26FCMul2Ty5DK+/15z+S5f7mblyii6dVOlRpolp3+rn0osQXsgq9r/ZvvfO3ban34GhFS0pJQZwMA63s8DLjv7LTp1Pv44i9tv38aJE1oI2o8/5vHJJ2OwWOruRMaMSeTbb4uw27VOp3t3G+edp78Y+Xe/a8GmTYWUlGjDpm7dbAwZop/RY+7cOI4d87BpkwOjUXD77fF06aIvRD/+6GXWLA9HjmjbaWku1q61EMjtLgR06NB4YrXRI3jJbaKsRWfu9wouNOp/1tjhXu64wcmKT80g4dorXIy+KHjWx4kTXqZMyeXAAa132LLFwcqVbUhNDY1wrVvnYevWqjnKQ4ckjz/u4tVXIzcNlSIApy9aw6vHEggh0gPsW1dnELLRaqgtrSaHx+Pj0ktXkJFRjNFo4JlnLmbatJ66+//3vwcrBQtg584CfvmlgMGDW9W5/1/+0o6yMh8bNhRgsRh49NEuJCbqX4bZs9tht3tZuzYPl6ucp5/uQ6tW+h2mEILHHmt45eQXX/RWChbAnj2STz/1MmNGaG6N7V7BrS4zx6UBbPHMcUg+sLlIDSBc82a5mDercYIp3n67pFKwADIzvbzwQgnPPReaBckeD0jppWoyw4jPpx7jZstpzmmdYixBNtCx2r93AEKW+yyynf+nwYQJq9i5s4DSUh/FxR5uvfULjhzRzwRee4pMCLBaA7tqbrutPR9+2J+VK/vSp0/9xQLnzu3Ahx8OZPHilpxzTnAzdFtreQ6NRgKWSTldios9lJbWbwG97TFqguXnCIIV3tDdprGxv/3s6Oj650XLyiTFxcEfjPbrJzCbXWgDXYkQHnr0CHFCR0XocNbzqoPTiCVYA1wnNC4EiirciKFAiVYt9u8voro17HZLvvoqW3f/6OjqS88lXq+L1q3DJ636Aw+Y6dVLO1+jEYYPF0yYELz5Ea9XMnv2bi688AeGDv2BO+88EHD/FCER1TwPRiQpInTzZrNmxTFkiLVycHLOOWbuvjtwpOSCBQ6GDCll6NBS/vjHMry1V6OeAdu2uWssIJcS9u6N0FWkivo5vYwYycA3Qoifga3Ax1LKtcAiYKwQYj8w1r8N8AmQARwAXgFubYxTaSjKr1CL2FgzBQVVnYDBIOjXT98V9N13vwKlaHkBfbjdLrZsOcbUqfph6U2J5GTBZ59ZWL3aS1wcTJpk9JdzCQ6LF2fz8ccn8fgfoHfeOc6YMUlcfnnd3+n/M3v5ymtgu88APh/DzTDLFDpLwmYTrFmTzJo1pf4giBji4vTHep9/7uHtt93Y/QGea9d6ee45F3fcoR8Mcyr06mUkPl5UWnFCQMeOKgij2XIa45VTjSWQ2mLeuafRukZBiVYtnn56BDNmrK1MzZKcbGPgQP2EHEZjRb2uqnktiyW8KgUnJgpmz26cW2HPntJKwQIoLfWRllaqK1pWAattbnb6BNmZmUzo0hFDiL9Om00wfXrDqv2mpXkrBQu0Oai9e4Mnuuefb+H66228/76Wwb9fPxP3368qETdbmuGKB+UerMWrr/6MlFVOYYfDwf79+lnbr722V415rTZtohk2TD8hb3NjwoSWJCRUWQJt2pgZOzZw0n6jgHONkl4eR8gF61QZO9ZEcnJVo+Pj4Xe/C64l9PDDcXz3XQs2b27JypWJWK1h9iUpgkczTJirLK1a1J5/cDq9lJXpX/2//30YpaVutm49hsVi5KGHRpCUFMKsz02MSZNak5npYNWqkwgBN92UwsCBkVvuvW9fI4sWWXnxRRc+H0yaZOLqq4M/xxkbayBWGViKCBWmQCjRqsWMGb3ZsSOH/HzN3devX2v69tWf0zIYBIsWjTpLrWs6VOSsbEiGkXnzOjJvXsd694sUpk41M3Vq6GqGhQopfxtNq2hkmmGFAiVatZg+vQ+xsRaWL0+nVatoFi4cjtmsJrorkFLy17/+yLff5mE0CmbM6Midd/YOdbMUIWTZejPPrbDickPfLl7euLcca/gE0IY3ytJSAEyY0J0JE7qHuhlNkhdeOMCKFdk4nb7K7Usuac2QIar6b3PkyAnBo29ZOXpSG9j9mmPg/tckT97SDE2AUNAMRUsFYihOiZ9+KqgULICCAjdbt+oHqigim0PHDBzPq+pGpBQcOqq6lbNGMywCqe4uxSkxbFgroqOrbptWrSyMGFF3yipF5JPawUf71tUWgxskvTs3wzjsUBHcLO9hgXIPhoCVK/N47718f+XiTiQnh8+k/Z//3JWMDDsbN+ZiMAiuv74L554bOIRdEbkkt5A8cVM5Ty6z4nILBnT3svBPOvmDFMEntEUWQoISrbPMypV53HnnrxQUaMOgPXvKWbeuD7Gx4RHsoSXkHRDqZiiaEFcM83DFsGY4uaIICco9eJZZsSKvUrAA0tPL2b5dv6ijQqFQKKpQltZZJiqq5jghOtoQsDSJQqFQ6BOh0RYBUJbWWeaxxzrSu7cNgwFiYgxccUUSAwY0Qi0QhULRDGh+eZzUEP8sk5JiZf36vmzbZqdFCxMDB0Y3KKuEQqFQ/JbmZ2kp0QoBcXFGRo8OXJNJoVAo6qe8nr9HXh5UJVoKhUIRttRnaSnRUigUCkWTITLnrQKhREuhUCjCFjWnpVAoFIqwQVlaCoVCoQgblKWlUCgUirChvujByEOJlkKhUIQtyj2oUCgUirBBuQcVCoVCETYoS0uhUCgUYYOytBQKhUIRNqhADIVCoVCEDcrSUigUCkXYoOa0IpIlPlgnIVHAI0L7qVAoFOGPsrQijpd98E8JJQiQsE9KPjKAWQmXQqEIe5qfpRXxlYs/qxAsPweAgw35R5cTpGysZikUCkUQcNfzijyatGgJIS4XQuwVQhwQQiw4nWOYa21bgdhA/1BmJ/r2KcT+fjCxf7gA88dvn87HKhQKxVmgvJ5X5NFkRUsIYQReAMYDfYGZQoi+p3qcBwV0RwKSeCRTBHQI4Bq0PXcP5u1fYszJwph1AOvrixD5uad5FgqFQtGYeOp5RR5CNlEXmBDiIuBBKeXv/Nv3AkgpnwAoKipqmg1XKBSKACQkJJzRjPrp9n1n+rlNhSZraQHtgaxq29n+9xQKhULRTGnKolXXqEBZVwqFQtGMacoh79lAx2rbHYCjFRuRYuoqFArFqdDc+76mbGltA1KFEF2FEBZgBrAmxG1SKBQKRQhpsqIlpfQA84DPgD3AcinlLghOKHw4IYQ4LITYKYTYIYT4IdTtaQyEEK8LIXKFEGnV3mshhFgvhNjv/5kUyjYGE53zfVAIccR/nXcIISaEso3BRAjRUQjxhRBijxBilxDiNv/7EXmNA5xvxF7js0WTjR7Uwx8Kvw8Yi+ZC3AbMlFLuDmnDGhEhxGFgsJTyZKjb0lgIIS4B7MBbUsr+/veeBPKllIv8g5MkKeU9oWxnsNA53wcBu5Ty6VC2rTEQQrQD2kkpfxRCxAHbgSnA9UTgNQ5wvtOJ0Gt8tmiyllYAhgAHpJQZUkoX8C4wOcRtUpwhUsqvgfxab08G/uv//b9oD31EoHO+EYuU8piU8kf/7yVo3pP2ROg1DnC+ijMkHEWrOYbCS2CdEGK7EOLGUDfmLJIspTwGWicAtAlxe84G84QQv/jdhxHhKquNEKILcB7wPc3gGtc6X2gG17gxCUfRao6h8MOllIPQsoPM9buWFJHHS0B34FzgGPBMaJsTfIQQscBK4HYpZXGo29PY1HG+EX+NG5twFK2AofCRiJTyqP9nLrAKzUXaHMjxzw1UzBFEdD4tKWWOlNIrpfQBrxBh11kIYUbrwJdKKd/3vx2x17iu8430a3w2CEfRalah8EKIGP9ELkKIGGAckBb4vyKGNcBs/++zgdUhbEujU9F5+5lKBF1nIYQAXgP2SCn/p9qfIvIa651vJF/js0XYRQ8C+MNEnwOMwOtSysdC3KRGQwjRDc26Am0x+LJIPF8hxDvAKKAVkAMsBD4AlgOdgExgmpQyIoIXdM53FJrbSAKHgZsq5nvCHSHECGATsBPw+d++D22eJ+KucYDznUmEXuOzRViKlkKhUCiaJ+HoHlQoFApFM0WJlkKhUCjCBiVaCoVCoQgblGgpFAqFImxQoqVQKBSKsEGJlkKhUCjCBiVaiohCCLFUCPF6rfdGCiHyai3sVCgUYYgSLUWk8f+ACUKIsQBCCBtaupw7grmI018iR6FQnGWUaCkiCillHvBX4GV/2quFwEEp5ZtCCIMQ4j4hxEEhxEkhxLsVWbb9f3tPCHFcCFEohPhSCNGn4rhCiLeFEC8IIdYKIUqBi4UQV/iL/JUIIbKFEPNDctIKRTNCiZYi4pBSrkAruvcOcCNwk/9PfwMmApegJVouBf5d7V8/AlKBtmg54ZbUOvTvgYeAOGAL8AYwR0oZBwwAvmqE01EoFNVQaZwUEYkQIhk4CPxdSvkv/3v7gRuklF/5tzsCB4Aof9bt6v/fCjgBxEopS4UQbwMuKeWfq+1zFHgA+D9/oT+FQtHIKEtLEZFIKXOAk8Cuam93Aj70u/8K0ZKZSqCNEMIohHhSCJEhhChGEzPQEtpWUL34KGhZuicBmX534tBGORmFQlGJEi1FcyIbGCulTKz2skkpjwPXAROA0UAC0MP/P9WLjtZwS0gpv5dSTkKrtvsR8G6jn4FC0cxRoqVoTvwHeFwI0QlACNFGCDHJ/7c4wAnkAdFAwPIvQogoIcTvhRDxUko3UAJ4G6/pCoUClGgpmhf/A6wFPhdClACbgQv8f3sDrQL2UTSX4uYGHG828KvfnTgHmBX0FisUihqoQAyFQqFQhA3K0lIoFApF2KBES6FQKBRhgxIthUKhUIQNSrQUCoVCETYo0VIoFApF2KBES6FQKBRhgxIthUKhUIQNSrQUCoVCETYo0VIoFApF2PD/AaGN4WWobj7FAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# scatter plot of Years versus Hits colored by Salary\n",
    "hitters.plot(kind='scatter', x='Years', y='Hits', c='Salary', colormap='jet', xlim=(0, 25), ylim=(0, 250))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['AtBat', 'Hits', 'HmRun', 'Runs', 'RBI', 'Walks', 'Years', 'League',\n",
       "       'Division', 'PutOuts', 'Assists', 'Errors', 'NewLeague'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define features: exclude career statistics (which start with \"C\") and the response (Salary)\n",
    "feature_cols = hitters.columns[hitters.columns.str.startswith('C') == False].drop('Salary')\n",
    "feature_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     263.000000\n",
       "mean      535.925882\n",
       "std       451.118681\n",
       "min        67.500000\n",
       "25%       190.000000\n",
       "50%       425.000000\n",
       "75%       750.000000\n",
       "max      2460.000000\n",
       "Name: Salary, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hitters.Salary.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define X and y\n",
    "X = hitters[feature_cols]\n",
    "y = (hitters.Salary > 425).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['AtBat', 'Hits', 'HmRun', 'Runs', 'RBI', 'Walks', 'Years', 'League',\n",
       "       'Division', 'PutOuts', 'Assists', 'Errors', 'NewLeague'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting if salary is high with a decision tree\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review -  Building a Decision Tree by hand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth = None\n",
    "num_pct = 10\n",
    "max_features = None\n",
    "min_gain=0.001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For feature 1 calculate possible splitting points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hits\n"
     ]
    }
   ],
   "source": [
    "j = 1\n",
    "print(X.columns[j])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the variable in num_ctp points\n",
    "splits = np.percentile(X.iloc[:, j], np.arange(0, 100, 100.0 / num_pct).tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only unique values for filter binary and few unique values features\n",
    "splits = np.unique(splits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  1. ,  52. ,  66.8,  77. ,  92. , 103. , 120. , 136. , 148.6,\n",
       "       168. ])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "split the data using split 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "k = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_l = X.iloc[:, j] < splits[k]\n",
    "\n",
    "y_l = y.loc[filter_l]\n",
    "y_r = y.loc[~filter_l]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gini \n",
    "\n",
    "The Gini Impurity of a node is the probability that a randomly chosen sample in a node would be incorrectly labeled if it was labeled by the distribution of samples in the node."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gini(y):\n",
    "    if y.shape[0] == 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return 1 - (y.mean()**2 + (1 - y.mean())**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.39928079856159704"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gini_l = gini(y_l)\n",
    "gini_l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42690311418685123"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gini_r = gini(y_r)\n",
    "gini_r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The gini impurity of the split is the Gini Impurity of each node is weighted by the fraction of points from the parent node in that node."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### putting all in a function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gini_impurity(X_col, y, split):\n",
    "    \"Calculate the gain of an split k on feature j\"\n",
    "    \n",
    "    filter_l = X_col < split\n",
    "    y_l = y.loc[filter_l]\n",
    "    y_r = y.loc[~filter_l]\n",
    "    \n",
    "    n_l = y_l.shape[0]\n",
    "    n_r = y_r.shape[0]\n",
    "    \n",
    "    gini_y = gini(y)\n",
    "    gini_l = gini(y_l)\n",
    "    gini_r = gini(y_r)\n",
    "    \n",
    "    gini_impurity_ = gini_y - (n_l / (n_l + n_r) * gini_l + n_r / (n_l + n_r) * gini_r)\n",
    "    \n",
    "    return gini_impurity_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0862547016583845"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gini_impurity(X.iloc[:, j], y, splits[k])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### test all splits on all features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def best_split(X, y, num_pct=10):\n",
    "    \n",
    "    features = range(X.shape[1])\n",
    "    \n",
    "    best_split = [0, 0, 0]  # j, split, gain\n",
    "    \n",
    "    # For all features\n",
    "    for j in features:\n",
    "        \n",
    "        splits = np.percentile(X.iloc[:, j], np.arange(0, 100, 100.0 / (num_pct+1)).tolist())\n",
    "        splits = np.unique(splits)[1:]\n",
    "        \n",
    "        # For all splits\n",
    "        for split in splits:\n",
    "            gain = gini_impurity(X.iloc[:, j], y, split)\n",
    "                        \n",
    "            if gain > best_split[2]:\n",
    "                best_split = [j, split, gain]\n",
    "    \n",
    "    return best_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 6.0, 0.1428365268140297)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "j, split, gain = best_split(X, y, 5)\n",
    "j, split, gain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_l = X.iloc[:, j] < split\n",
    "\n",
    "y_l = y.loc[filter_l]\n",
    "y_r = y.loc[~filter_l]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(263, 116, 147)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape[0], y_l.shape[0], y_r.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.49049429657794674, 0.1896551724137931, 0.7278911564625851)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.mean(), y_l.mean(), y_r.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recursively grow the tree "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tree_grow(X, y, level=0, min_gain=0.001, max_depth=None, num_pct=10):\n",
    "    \n",
    "    # If only one observation\n",
    "    if X.shape[0] == 1:\n",
    "        tree = dict(y_pred=y.iloc[:1].values[0], y_prob=0.5, level=level, split=-1, n_samples=1, gain=0)\n",
    "        return tree\n",
    "    \n",
    "    # Calculate the best split\n",
    "    j, split, gain = best_split(X, y, num_pct)\n",
    "    \n",
    "    # save tree and estimate prediction\n",
    "    y_pred = int(y.mean() >= 0.5) \n",
    "    y_prob = (y.sum() + 1.0) / (y.shape[0] + 2.0)  # Laplace correction\n",
    "    \n",
    "    tree = dict(y_pred=y_pred, y_prob=y_prob, level=level, split=-1, n_samples=X.shape[0], gain=gain)\n",
    "    \n",
    "    # Check stooping criteria\n",
    "    if gain < min_gain:\n",
    "        return tree\n",
    "    if max_depth is not None:\n",
    "        if level >= max_depth:\n",
    "            return tree   \n",
    "    \n",
    "    # No stooping criteria was meet, then continue to create the partition\n",
    "    filter_l = X.iloc[:, j] < split\n",
    "    X_l, y_l = X.loc[filter_l], y.loc[filter_l]\n",
    "    X_r, y_r = X.loc[~filter_l], y.loc[~filter_l]\n",
    "    tree['split'] = [j, split]\n",
    "\n",
    "    # Next iteration to each split\n",
    "    \n",
    "    tree['sl'] = tree_grow(X_l, y_l, level + 1, min_gain=min_gain, max_depth=max_depth, num_pct=num_pct)\n",
    "    tree['sr'] = tree_grow(X_r, y_r, level + 1, min_gain=min_gain, max_depth=max_depth, num_pct=num_pct)\n",
    "    \n",
    "    return tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'y_pred': 0,\n",
       " 'y_prob': 0.49056603773584906,\n",
       " 'level': 0,\n",
       " 'split': [6, 5.0],\n",
       " 'n_samples': 263,\n",
       " 'gain': 0.15865574114903452,\n",
       " 'sl': {'y_pred': 0,\n",
       "  'y_prob': 0.10869565217391304,\n",
       "  'level': 1,\n",
       "  'split': -1,\n",
       "  'n_samples': 90,\n",
       "  'gain': 0.01935558112773289},\n",
       " 'sr': {'y_pred': 1,\n",
       "  'y_prob': 0.6914285714285714,\n",
       "  'level': 1,\n",
       "  'split': -1,\n",
       "  'n_samples': 173,\n",
       "  'gain': 0.1127122881295256}}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_grow(X, y, level=0, min_gain=0.001, max_depth=1, num_pct=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "tree = tree_grow(X, y, level=0, min_gain=0.001, max_depth=3, num_pct=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'y_pred': 0,\n",
       " 'y_prob': 0.49056603773584906,\n",
       " 'level': 0,\n",
       " 'split': [6, 5.0],\n",
       " 'n_samples': 263,\n",
       " 'gain': 0.15865574114903452,\n",
       " 'sl': {'y_pred': 0,\n",
       "  'y_prob': 0.10869565217391304,\n",
       "  'level': 1,\n",
       "  'split': [5, 65.0],\n",
       "  'n_samples': 90,\n",
       "  'gain': 0.01935558112773289,\n",
       "  'sl': {'y_pred': 0,\n",
       "   'y_prob': 0.07407407407407407,\n",
       "   'level': 2,\n",
       "   'split': [0, 185.0],\n",
       "   'n_samples': 79,\n",
       "   'gain': 0.009619566461418955,\n",
       "   'sl': {'y_pred': 0,\n",
       "    'y_prob': 0.3333333333333333,\n",
       "    'level': 3,\n",
       "    'split': -1,\n",
       "    'n_samples': 7,\n",
       "    'gain': 0.40816326530612246},\n",
       "   'sr': {'y_pred': 0,\n",
       "    'y_prob': 0.05405405405405406,\n",
       "    'level': 3,\n",
       "    'split': -1,\n",
       "    'n_samples': 72,\n",
       "    'gain': 0.009027777777777565}},\n",
       "  'sr': {'y_pred': 0,\n",
       "   'y_prob': 0.38461538461538464,\n",
       "   'level': 2,\n",
       "   'split': [0, 470.90909090909093],\n",
       "   'n_samples': 11,\n",
       "   'gain': 0.2203856749311295,\n",
       "   'sl': {'y_pred': 0,\n",
       "    'y_prob': 0.14285714285714285,\n",
       "    'level': 3,\n",
       "    'split': -1,\n",
       "    'n_samples': 5,\n",
       "    'gain': 0},\n",
       "   'sr': {'y_pred': 1,\n",
       "    'y_prob': 0.625,\n",
       "    'level': 3,\n",
       "    'split': -1,\n",
       "    'n_samples': 6,\n",
       "    'gain': 0.4444444444444444}}},\n",
       " 'sr': {'y_pred': 1,\n",
       "  'y_prob': 0.6914285714285714,\n",
       "  'level': 1,\n",
       "  'split': [1, 103.0],\n",
       "  'n_samples': 173,\n",
       "  'gain': 0.1127122881295256,\n",
       "  'sl': {'y_pred': 0,\n",
       "   'y_prob': 0.43037974683544306,\n",
       "   'level': 2,\n",
       "   'split': [5, 22.0],\n",
       "   'n_samples': 77,\n",
       "   'gain': 0.07695385846646363,\n",
       "   'sl': {'y_pred': 0,\n",
       "    'y_prob': 0.17857142857142858,\n",
       "    'level': 3,\n",
       "    'split': -1,\n",
       "    'n_samples': 26,\n",
       "    'gain': 0.06860475087899842},\n",
       "   'sr': {'y_pred': 1,\n",
       "    'y_prob': 0.5660377358490566,\n",
       "    'level': 3,\n",
       "    'split': -1,\n",
       "    'n_samples': 51,\n",
       "    'gain': 0.09501691508611931}},\n",
       "  'sr': {'y_pred': 1,\n",
       "   'y_prob': 0.8979591836734694,\n",
       "   'level': 2,\n",
       "   'split': [2, 6.0],\n",
       "   'n_samples': 96,\n",
       "   'gain': 0.01107413837448551,\n",
       "   'sl': {'y_pred': 1,\n",
       "    'y_prob': 0.7058823529411765,\n",
       "    'level': 3,\n",
       "    'split': -1,\n",
       "    'n_samples': 15,\n",
       "    'gain': 0.16547008547008554},\n",
       "   'sr': {'y_pred': 1,\n",
       "    'y_prob': 0.927710843373494,\n",
       "    'level': 3,\n",
       "    'split': -1,\n",
       "    'n_samples': 81,\n",
       "    'gain': 0.006994315787586275}}}}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tree_predict(X, tree, proba=False):\n",
    "    \n",
    "    predicted = np.ones(X.shape[0])\n",
    "\n",
    "    # Check if final node\n",
    "    if tree['split'] == -1:\n",
    "        if not proba:\n",
    "            predicted = predicted * tree['y_pred']\n",
    "        else:\n",
    "            predicted = predicted * tree['y_prob']\n",
    "            \n",
    "    else:\n",
    "        \n",
    "        j, split = tree['split']\n",
    "        filter_l = (X.iloc[:, j] < split)\n",
    "        X_l = X.loc[filter_l]\n",
    "        X_r = X.loc[~filter_l]\n",
    "\n",
    "        if X_l.shape[0] == 0:  # If left node is empty only continue with right\n",
    "            predicted[~filter_l] = tree_predict(X_r, tree['sr'], proba)\n",
    "        elif X_r.shape[0] == 0:  # If right node is empty only continue with left\n",
    "            predicted[filter_l] = tree_predict(X_l, tree['sl'], proba)\n",
    "        else:\n",
    "            predicted[filter_l] = tree_predict(X_l, tree['sl'], proba)\n",
    "            predicted[~filter_l] = tree_predict(X_r, tree['sr'], proba)\n",
    "\n",
    "    return predicted    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 0., 1., 0., 0., 0., 1., 1., 1., 0., 1., 1., 1., 0., 0.,\n",
       "       0., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 0., 0., 0., 1.,\n",
       "       0., 0., 1., 1., 0., 1., 1., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0.,\n",
       "       0., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 0., 1., 0.,\n",
       "       0., 1., 0., 0., 1., 1., 0., 1., 1., 0., 1., 1., 0., 1., 1., 1., 0.,\n",
       "       1., 0., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0.,\n",
       "       0., 0., 0., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 0.,\n",
       "       0., 1., 1., 1., 0., 1., 0., 1., 1., 0., 0., 0., 0., 1., 0., 0., 1.,\n",
       "       0., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0., 0., 1., 0., 1., 1.,\n",
       "       0., 1., 1., 1., 0., 1., 0., 1., 1., 0., 0., 1., 1., 1., 0., 0., 1.,\n",
       "       0., 0., 0., 1., 0., 0., 0., 1., 0., 1., 0., 1., 1., 1., 1., 1., 0.,\n",
       "       1., 1., 0., 1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 0.,\n",
       "       0., 1., 1., 0., 1., 0., 1., 1., 0., 0., 0., 1., 0., 0., 1., 1., 0.,\n",
       "       0., 0., 0., 1., 1., 0., 0., 0., 1., 1., 0., 1., 0., 1., 0., 1., 1.,\n",
       "       1., 0., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 1., 0., 1., 1., 1.,\n",
       "       1., 0., 0., 1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_predict(X, tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of values to try for max_depth\n",
    "max_depth_range = range(1, 21)\n",
    "\n",
    "# list to store the average RMSE for each value of max_depth\n",
    "accuracy_scores = []\n",
    "\n",
    "# use 10-fold cross-validation with each value of max_depth\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "for depth in max_depth_range:\n",
    "    clf = DecisionTreeClassifier(max_depth=depth, random_state=1)\n",
    "    accuracy_scores.append(cross_val_score(clf, X, y, cv=10, scoring='accuracy').mean())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Accuracy')"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbAAAAEGCAYAAAAE3cBCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt8XNV57//Po7vkiyQb321sLsJgsIFcCARCQoBASICkPW3gNBdO0jQ9LyCXNueENjmEkKYn4SQhSUv6S3M5SWgbSqBpTI4p0OJAoDgYAjbYBmTAxka+25IsWfd5fn/MyNqzNSONpZk9s0ff9+ull2f27Bk92hrPo7XWs9Yyd0dERCRuKoodgIiIyEQogYmISCwpgYmISCwpgYmISCwpgYmISCxVFTuAfOjo6FAppYhImWtsbLTgfbXAREQklpTAREQklpTAiqS1tbXYIRwzxRwNxRwNxRyNQsasBCYiIrGkBCYiIrGkBCYiIrGkBCYiIrGkBFbmth0e5LWuwWKHISKSd5ElMDO73MxeNLOtZnZThsePN7O1ZvaMmW00sytSxy81s6fN7LnUv++MKua4u33jYc66Zw9n/nwPf7epq9jhiIjkVSQJzMwqgTuAdwMrgGvNbEXotC8Ad7v72cA1wHdTx/cDV7r7SuAjwJ1RxBx3vYPObc8eBsCB2zZ0or3fRKScRNUCOwfY6u6vuHs/cBdwdegcB2ambjcCbQDu/oy7t6WObwLqzKw2gphjbUv7AD1DIwnrUJ9zqC9RxIhERPIrqgS2CNgRuL8zdSzoFuCDZrYTWAPcmOF1fh94xt37ChFkOdlwYGDUsV1HlMBEpHxYFN1KZvYHwGXu/sep+x8CznH3GwPn/Fkqnm+Y2XnAD4Ez3D2Revx0YDXwLnd/Ofj6wcV84zhTvRC+urWae3dXpx379um9vLVZSUxE4qOlpeXo7fBivlGtRr8TWBK4v5hUF2HAx4DLAdz9CTOrA44D9prZYuAXwIfDySss+MOWstbW1oLGuu2FvUB6K8ya5tPSMm3Cr1nomAtBMUdDMUdDMaeLqgtxPdBiZieYWQ3JIo3VoXNeAy4GMLPTgDpgn5k1Af8P+At3fzyieGNtMOFsOjS6C3H3kaEiRCMiUhiRJDB3HwRuAB4AtpCsNtxkZrea2VWp0/4c+LiZbQB+Blznyf7NG4CTgf9lZs+mvuZGEXdcvdQxSG+GXLVLCUxEykhkG1q6+xqSxRnBYzcHbm8Gzs/wvL8C/qrgAZaRTAUcAG0q4hCRMqKVOMrQxgP9GY+rC1FEyokSWBnK1gJTF6KIlBMlsDKTcOf5g5kT2L6eBAMJrcYhIuVBCazMbDs8ROdA5iTlwB61wtK82jmorlWRmFICKzMbsox/DdNqHCO+9FQHZ9+7hxV37+YnL3YXOxwROUZKYGVmY5bxr2Ftam0AsKNrkNufS67Qn3D47Lp2NmXpehWR0qQEVmbCBRyNNWkrr6i7LOW+7b1p9wcScP1jhxjUGKFIbCiBlRF3Z2OoFXHJorq0+6pETLpve8+oY88eGOA7z2vfNJG4UAIrI21HEuzvHRnjaqgyLphfGzpHCWz3kSHW7ck8VvjVZzrZkmEZLhEpPUpgZSRcwHFGczWLp1emHdutIg5+tb2HbB2F/Qm4QV2JIrGgBFZGwgUcZ86uZn5DegJTFyKsDo1/nTu3Ju3+0/sHuGOTuhJFSp0SWBkJF3Csml3Nwob0X/Gu7qmdwPb3DvH47vT9UO+4oJn3Hp8+VvjXz3TyUru6EkVKmRJYGXnu4OgENqu2gprAb7lr0Dk8MHW7Ede81stQoHfw9OYqTmqs4hvnNdFcO1Kx2TeUrEocUleiSMlSAisTB3qH2BloXVVXwGlN1ZjZ6G7EKdwKW70tvfrwqmX1AMxrqORrb2lKe2z9vgG+u1ldiSKlSgmsTITHv05rqqamMtmiWKhxMADa+xL8ui29+/CqpfVHb//BifW8e0l6V+JXftfJ1g51JYqUIiWwMhEe/zpzdvXR2wtGJbCp2YV4/45eBgM9gqc0VnFq08iWeGbG7W9tSpv83TsENzzWrq5EkRKkBFYmxkpg88OFHFO0BTaq+3BpPWbpK5XMb6jkq6GuxHV7+/neFq2VKFJqlMDKxMaD6XPAVgUSWLgLcSpOZj48kODhtvTy+SuX1WU895qT6rlscfoE8C8/3ckrnYMFi09Ejp0SWBno7E/wcudIUqowOL050IU4LTyZeeolsAd39NIX+LGXzahk1azqjOcmuxKbmRnoSuwZcq5/7BAJV1eiSKlQAisD4Q0sW2ZWMa165Ferycywevv43YdBC6dV8tfnNKYde2JPP99XV6JIyVACKwNjjX9BhirE7qlVxHFkMMFDO0PVh8vqs5w94o9ObuCSReldiV96upNth9WVKFIKlMDKQHgNxFWhBBYu4tjdMzSlusL+fWcfRwLlh4unVfLG4zJ3HwaZGd96axMzqkdaakcGnRvUlShSEpTAykB4C5VVs9PX9muoqkgrDR9y2NczdVph4a1T3ru0bszuw6DF06v4Sqgr8bHd/fxf7eAsUnRKYDHXM+i82J7epZWpOGGqTmbuT8C/7UivPgxOXs7Fh1oauGhhelfizes72a6uRJGiUgKLuS2HBtLW9ls6vZKm2tG/1tGTmadGAvttewWHB0Yu0Lz6Ct4SWn1+PGbGd85P70rsHnQ++Xg7rq5EkaJRAou58Qo4ho2uRJwaXYgP769Ku//epfVUVuTWfRi0ZHoVt74pvSvxkV19/OSlI5OKT0QmLrIEZmaXm9mLZrbVzG7K8PjxZrbWzJ4xs41mdkXq+OzU8S4z+9uo4o2LcAHHmbMzty6mYhfiQMJ59GD6z33V0syTl3Nx3fIGLlyQ3pX4hSc7eK1LXYkixRBJAjOzSuAO4N3ACuBaM1sROu0LwN3ufjZwDfDd1PFe4H8Bn40i1rgZXcCRuQW2YNrUW07qN7v66BwcaW3Nqq3g/Pm1YzxjbGbG35zfxLSqkdfsGnQ+pa5EkaKIqgV2DrDV3V9x937gLuDq0DkOzEzdbgTaANy9290fI5nIJGAg4Ww6lGMXYv3Ua4GF1z58z/F1VE2g+zBo6YwqvvSmmWnH1rb1cWeruhJFohZVAlsE7Ajc35k6FnQL8EEz2wmsAW6MJrT4eql9MG15pPn1FcwNJaphC6dNrQQ2lHB+9Vqo+jCHycu5+Oip07hgfnpX7Ree7OD1KbzPmkgxVI1/Sl5k+rM33OdyLfBjd/+GmZ0H3GlmZ7j7MVUbtLa2TjTGyE021gf3VAIjXWIn1w1kfc3efoCGo/dfP5z93LHE5fo+3V7B/t6R8a7plc6i7h3kK/w/X2w8tbeO3kTyrd054Hz8odf51oo+cpxiNqa4XOcgxRyNqRZzS0tL1seiSmA7gSWB+4tJdREGfAy4HMDdnzCzOuA4YO+xfKOxfthS0traOulY9xxsB0Ym1J53fBMtLTMznntiwqlc33a05L5j0Fh8wsnUV+X+aZuPmKPy9+vSr817ljWwYvnivL1+C/Clyi4+99uOo8f+81Alv+qdz2dWTs95onQmcbrOwxRzNBRzuqi6ENcDLWZ2gpnVkCzSWB065zXgYgAzOw2oA/ZFFF8shUvosxVwAFRWGPPq03/de3rKs8sr4c59Gfb+yrePnzaN8+aldyXe+nQn1/7HQQ70lue1FSklkSQwdx8EbgAeALaQrDbcZGa3mtlVqdP+HPi4mW0AfgZc56nSLjPbBnwTuM7MdmaoYJxyEu48l+McsGHhycxtZTpms35vP7sDS2VNrzLeuWji5fPZVJhxxwXN1Femt7b+bUcvF/xyL4/u6svyTBHJh6i6EHH3NSSLM4LHbg7c3gycn+W5ywoaXAy92jlEV2CB2qYaY8m0zAUcw5KTmUeSXrkWcqzenl688a4ldcfUVXosTpxZxY8vmsUfP3IwbcWPXUcSXP1v+/mzVdO56eyZVE+y+lFERtNKHDGVaQLzeOMuU2Eys7tn3PurkC5bUsdvrp7Lm+akt4Ad+MbGLq5Ys09bsIgUgBJYTOU6gTkovDNzOS4n9eyBAXZ0jSTm2grnksUTn7ycq2Uzqrj/ijn82arpo0pu1+8b4MJf7uXeVzRXTCSflMBiKtc1EIPm15f/ahzhycvnNQ8xvTqat3l1hXHzGxv518uOG3WtOwecjz1yiOsfO0TXQPn94SBSDEpgMeTuE0pg5T6Z2d35ZSiBvXN29D/j2xfW8vj75nLZktGFI//YeoR3rN7Hs/v7MzxTRI6FElgMvd49xMG+kb/ip1UZJ80cvx6n3LdU2XRokFcOj/xMNRXwtlnF+Rln11Vy18Wz+NpbGqkJ/S/b2jnIpf9vH3ds6tLOziKToAQWQ+HW18pZ1VTkMHF29JYqQ2W1CG24eOOihbVMj6zOdjQz4xMrpvMfV87llMb0QAYS8PknO/jAQwfYV6bz8UQKTQkshsIFHCtz6D4EmFltaSup9w1Be38ZJbBQ9+GVeVr7cLJWzqpm7ZVz+MgpDaMee+j1Ps7/5V7Wvq61qkWOlRJYDE1k/AuSLYJyncz8UvsAL7SPlKpXGbzn+NJIYADTqiv49vnN/OSiWTTWpLeW9/YkeP+DB7h5fQf9Q+XzB4VIoRWxg0UmamOOm1hmMr+hgq2dI/d3HRni9Fm5JcBSFp68/LYFtTTXVrC/SPFkc/Wyes4+rpo/eeQQ6/am/x6/83wXv9ndx+eWGHP64lWp2DkI7Yq54OIWc2WBm0hKYDGzr2eItsD8rZoKOLUp919juU5mDlcfFnry8mQcP72KX737OG7bcJivbzhMItDoemb/ANfsr4dndhUvwAlpgHWKufDiFfO7FtfylWWFe311IcZMePxrRXP1MS1TVI6ViK92DvJc4LpUGLxnaf7XPsynqgrjL8+eyerLj2NRw9hLgIlIZkpgMbPxGFagzyRTJWLchasPz5tXk3Vjz1JzwfxaHnvfXN57fGknXJFSpC7EmJloAcew0ZOZ49Ofnk24+rCUuw8zaa6t4M53zuLO1iN8f0s32zr6qSj04EGeJYYSijkCcYt5WlVhY1UCi5lMi/geiwUN5bWc1I6uQZ7en57Ur4xZAoNkheiHT5nGh0+Zpk0LI6KYo9HaeqBgrx2fVC509Cd4NbDSRIXBiuZj+xuk3LoQ7wtVH54zp2ZUK1NEypMSWIw8FyrgWN5YRcMxNtHnh8aG9vUkGEjEd+7RfdvDk5c1liQyVSiBxUi4gCPXFTiCaiqNOXUjv3YH9sS0Fbb7yBDr9qR3qcZt/EtEJk4JLEYmO/41bHQ3YjwLOX61vYdg2/Gs2dUsnaFhXZGpQgksRp6bZAXisIVlUsgRXn3jqhJZ+1BEoqEEFhNHBhO80JG+Lf3KCS4BVQ6Tmff3DvH47r60Y1eV+ORlEckvJbCY2HxoMG3JoRNmVNIY3mgqR+VQibjmtV6C696uaK7i5Mb4r+koIrlTAouJya7AEVQOOzPHffKyiEyeElhM5KuAAzJ1IcariKO9L8Eju9K7D6/W+JfIlKMEFhPhRXwnWsAB8e9CvH9HLwOBnNvSWHVMK/KLSHlQAouBgYSzKZTAVk1iD69wFeLumCWw0d2HdZjlviK/iJQHJbAYeKF9kP5Ai2NhQwVzJrHa+qzaCoL1H4cHnMMD8ehG3NczxMNtKp8XESWwWAjvwLxyEuNfkFw4dlQ3Ync8WmHf29xNXyDUE2ZUTqo1KiLxFVkCM7PLzexFM9tqZjdlePx4M1trZs+Y2UYzuyLw2F+knveimV0WVcylYrJbqGQyemfm0m+BHR5I8P0XutKO/clp09V9KDJFRTLybWaVwB3ApcBOYL2ZrXb3zYHTvgDc7e5/Z2YrgDXAstTta4DTgYXAv5vZKe4ejyZDHoQX8T0zDy2OOE5m/vGL3XT0j0z+aq41PnxKQxEjEpFiiqoFdg6w1d1fcfd+4C7g6tA5DsxM3W4E2lK3rwbucvc+d38V2Jp6vSkh4Z7XOWDDFkyL13JSfUPOdzeNbn1Nq1YvuMhUFVXt8SJgR+D+TuAtoXNuAR40sxuBacAlgeeuCz13UbZv1NraOtlYI5NLrNuOGN2DI0UKjVVOT9urtE6y16yquwoYGUt7oe0ArfW7x31esa7v6t2V7DpSe/R+bYVzSe0eWlv3jPvcOL0nhinmaCjmaEwm5rE28MwpgZnZJ4F/cvf9E4wh08dteBOqa4Efu/s3zOw84E4zOyPH5x4Vl91Kc91ZdeMrR4BDR++fPbeOU05ZPOnvv6ryCGwbed0jNTNoaZk95nOKtRtswp27ntsLjKwF+ZHl03nzivGvQzx3sFXMUVDM0ShkzLn2v1wCbDOzX5nZB8ysdtxnpNsJLAncX8xIF+GwjwF3A7j7E0AdcFyOzy1bowo48lRxF6cxsDWv9dIaWMi40uCGM6YXMSIRKQU5JTB3vwpYCtwPfBrYbWY/MLMLc/w+64EWMzvBzGpIFmWsDp3zGnAxgJmdRjKB7Uudd42Z1ZrZCUAL8GSO3zf28rkCR1A4ge0u0SpEd+dbzx1OO/b7J9Zz/HStvCEy1eU8Au7uB9z9Dnc/D3g78GZgrZltM7PPm1nWP4ndfRC4AXgA2EKy2nCTmd1qZlelTvtz4ONmtgH4GXCdJ20i2TLbDPwbcP1UqUB091FrIOajgAMyJbAhEp61Z7ZoHt/Tz1P70pP4p86YUaRoRKSUHNOfsWZ2MfBBkpWBTwG3kWw5fYpk6+xt2Z7r7mtIlsYHj90cuL0ZOD/Lc78CfOVYYi0HO7uHONQ3klSmVxknzsxPy6O+ymiqMdpTZemDDvt6EsxrmPgKH4Xw7Y3pra93La7ldE1cFhFyL+L4Osluvw7gp8AX3P31wOPrCFYaSF6Ex79Wzq6mIo+Tdhc0VNLePzK2tOvIUEklsOcPDvDQ6+mrzn96pVpfIpKU65/zdcD73X19pgfdfcDM3pS/sARGJ7B8L5m0oKGSLe3pCeysvH6HyflOaOzrnDk1nDdvcstoiUj5yDWB/W/gSPCAmTUD9e7eBuDuL+Q5tikvXMCRr/GvYQtGbWxZOoUc2w8Pcu+r6avOf2qllo0SkRG5FnH8K8ny9aDFwC/yG44EhRfxncwmlpksqC/dUvq/3dTFUKCmZHljFe8+vq54AYlIyck1gS139+eCB1L3T81/SAKwt2corUVUWwnL87xpY6kuJ7W/d4h/eCmtwc8nV07P6/ifiMRfrglsr5mdHDyQun8g/yEJMGr9wxXN1VRX5PcDvFQnM39vczc9gebXooZK/uBELdorIulyTWA/Au41s/ea2QozuxK4B/hB4UKb2gq1AkdQKSawroEE39+Svmjvfz99GjWVan2JSLpc+6S+CgwAXye5rNMOksnrmwWKa8rbeDA8gTn/1XelmMB++tKRo3PTABprjI8sn1bEiESkVOWUwNw9Afyf1JdEoBCbWIbNqaug0jhaLHGoz+kZdOqritPa6c+wZcrHT5vODG2ZIiIZ5FwVkFrDcDnJBXaPfsK5+8MFiGtKa+9LsO3wSGuo0pJjYPlWWWHMq6+gLVAssqdniGUzirPO4D2vHGFn98jPXVcJnzhNrS8RySzXlTguAH4O1JLcdLITmEGyK/HEgkU3RYV3YF7eWFWwVtGChsq0BNbWXZwElnDnO8+nt74+2DKNOfWlszKIiJSWXPtmbgduc/dZwOHUv18GvluwyKawQi3gm0mpjIM9sKOXF9q1ZYqI5C7XBHYK8O3Qsa8Cn8lvOAKZVuAo3PJJpZLAvv1ceuvr/SfUF60rU0TiIdcE1kGy6xBgl5mtAJoB/YmcZy+0D/BoW/oCtoUo4BhWCstJPbGnj3V701udn1TrS0TGkWsC+xfgitTtHwJrgadJjotJHiTc+btNXbx99V5296QnkZUF3D6kFFpg3wq1vi5eVFvQVqeIlIdcy+g/Hbj9DTP7LckijgcKFdhUsrNrkOsfa+eRXX2jHrtkUS0zawpXRr6gobjLSW0+NMADO3rTjmnLFBHJxbgJzMwqgZeAFe7eB+DujxU6sKnA3fn5Kz18dl07nf2jd0N+37J6vvXWpoLGUOwW2LdDW6a88bhqLpiv1peIjG/cBObuQ2Y2RHJPsNFNBJmQjgH46K8P8YttPaMem1ljfP3cJv7gxPqCbx+SKYG5eyTbluzoGuTeV8JbpszQlikikpNcy7y+BdxtZn8N7ASONhfc/ZVCBFbO/uP1Xj7xTB37+0cnrwsX1PLdC5pYPD2aCrwZ1ca0KqN7MPkr7RuC9n6nubbwSeSOTV0MBhqeJ8+s4j3aMkVEcpTrp+Tfpv69NHTcAc00zVH3QIIvPtXJD17oJlw/U1cJX3xjI59YMS3SbUPMjAUNlWztHJmD1dY9RHNtYZdvOtg7xE8zbJlSmecV90WkfOVaxKHF6CbpqX39fOLRg7zcOXqM6czZ1XzvwmZObSpcteFYFjRUsLVz5P6uI0OcXsDKR4C/39LNkUDza0FDBR84SVumiEjuNFO0wAYSztc3HObrGw6n7TAMUGHwZytn8D/PmlHU7UKiLuToHkjw91u604799xXTqdWWKSJyDHJdC/E3BMa9gtz9wrxGVEZeah/gE785xDP7B0Y9tqQuwQ8vnss5c2uLEFm6qBPYP7Qe4WDfyFy3mTXGddoyRUSOUa4tsPDGlfOBjwH/kN9wykPCnR9s6ebmpzrozZAL/tvyBq6btZ8zSyB5QabVOAqXwAYSzt+Gtkz541OnFXSum4iUp1zHwH4SPmZm9wL/F7g130HFWVv3ENc/doi1baNnHMytr+Bvzm/msiV1tLbuL0J0mY1ugRVuOal/ebWHHV0jCbK2Ev50hZaNEpFjN5kxsNeBVfkKpBw8uquPDz98IG1H4WFXLa3j9rc2Mbuu9Io2o1qNw91HTVz+ryc3MFdbpojIBOQ6BvbR0KEG4PeAdbl+IzO7nOSK9pXAD9z9q6HHbwcuCrz+XHdvSj32NeA9qce+7O7/nOv3jYq7c8Njh0Ylr5nVxm3nNvGBkwo/KXmiohoDe2hnH5sPjZTrVxjceIaWjRKRicm1Bfah0P1u4D9J7hM2rtRyVHeQnEe2E1hvZqvdffPwOe7+mcD5NwJnp26/B3gDcBbJDTUfMbP73T1Q+F18bUcSvNaV/sF/wfwavvu2Zo6PaFLyRM0LtYD29SQYSDjVeZ6T9a1Q6+vqpfWcOLO0r42IlK5cx8AuGv+sMZ0DbB1etcPM7gKuBjZnOf9a4Iup2yuAR9x9EBg0sw3A5cDdk4wpr8KbUJ41u5rVlx8X6aTkiaqpNObUVbCvNzn25cCeI0N5XQ2krXuI/9yTfo0+tVJjXyIycbl2IX4YeNbdNwaOnQmscvc7c3iJRcCOwP2dwFuyfK+lwAnAw6lDG4Avmtk3SXYtXkT2xEdra2sO4eTf2u3VwMjk31NqjvDy1q1jPqdYsWYyq7KOfYHVQZ58cTs9M0cXc0w05vv2VJJsQCedMWOIaYe203poQi93TErpOudKMUdDMUdjMjG3tLRkfSzXP7G/TLILL2gHsBrIJYFlaoZknFcGXAPc4+5DAO7+oJm9mWSX5T7gCWAwy3PH/GELacf2A8DItiAXnjSHlpbsc5taW1uLFmsmy7bt58XukcrJqtkLaVlWn3bOZGLe3HYQGFn78b0nNdHSMjP7E/Kk1K5zLhRzNBRzNAoZc66Tb2YC4TGnDiDXvT52AksC9xcDbVnOvQb4WfCAu3/F3c9y90tJJsOS+xNkY6gL8cyYbchYyEKOhDtrX0+fVvDORaUxB05E4ivXBLYZ+P3QsfcDW3J8/nqgxcxOMLMakklqdfgkM1sONJNsZQ0fqzSz2anbq0iW7j+Y4/eNxL6eIdoCc6dqKuDUpngVJxRyMvPGAwMcCK288Ybj4pXgRaT05Pop+zlgjZl9AHgZOBm4GLgilye7+6CZ3UByB+dK4EfuvsnMbgWecvfhZHYtcJe7B7sXq4HfpErQO4EPpgo6SsbGg+lLRa1ors57BV+hFbIFFp7U/fYFtVTF7PqISOnJtQrxMTM7Hfgjkl2BTwKfcvcdYz8z7TXWAGtCx24O3b8lw/N6SVYilqwNB9IT2Jmzi7Oq/GQUcjWOh1/vTbv/zoXa80tEJi/XKsRaYHdw8rGZVZtZrbtP+V2awyX0cRv/gsK1wLoHEqzbm359LtL4l4jkQa5jYA8BbwwdeyPJLsEprzxaYOlvhd15SmCP7+5nINCYO3FGJctmxGt8UERKU64JbCXw29CxJ4Ez8xtO/LT3Jdh2eOTDvtKSY2BxM6u2gtpAI+zwgHN4YPLdiA+3hboPF6n7UETyI9cE1gHMCx2bR3JJqSktXMCxvKmK+qr4FSiYGfNDS0rt6p58KyxcPn/RQnUfikh+5JrA7gX+yczOMLMGM1tJcgLzzwsXWjyEx79WzYpf62tYvgs5dnYN8mLHSMFopcHbFiiBiUh+5JrAPk9yzteTQBfJVei3AF8oUFyx8dyo8a/4FXAMy3chR7h8/py5Ndq4UkTyJqdPE3fvdffrgWkkuw7PA/oowRUxolYOBRzDFkzL775g4QSm7kMRyaec/xw2sznAJ0lWHj4DvAn4VIHiioXugQQvdaTPqV4Z5wQWHgObRAIbSjhrVcAhIgU0Zj2zmVUDVwHXAZcBW0muU7gM+EN331vg+Era8wcH0lYkPnlmFTOq49tFls/lpDYeHOBQ38jVaawxzo5xcheR0jPep+0e4HvAi8C57r7C3b9Msvtwyiun7kPI7xjYw6Hqw3csrKVSy0eJSB6Nl8A2klxx/i3Am82sufAhxceGg+WdwHZPogpx1PwvLR8lInk2ZgJz93cAJ5Fc/f2zwG4zu49kMUe8P63zYGOoBbaq7BLYEAnPtm1bdocHEjwZWj7qHSrgEJE8G3fAxt23u/uX3b2F5Ar0u4AEsMHMbit0gKWqb8jZciiUwGI8Bwygvspoqhnp5ht02Ndz7K2wx3f3pS0fdfLMKpZq+SgRybNjqjhw98fc/U+A+cCNJJeYmpK2HBpgMNA4WTK9kll1ldmfEBP5GAcLj39p8V4RKYQJlcyl5oX9zN3fne+A4mJUAUfMW1/D8pFDPtA0AAARzklEQVTAwvO/3qnuQxEpgPjWfBdZuVUgDhtdSn9sXYivdQ3SGpgbV2VwgZaPEpECUAKboHLYAyyTyU5m/nWo9fXmuTWxnhsnIqVLnywTMJhwNoULOMqmBTa55aTC41/qPhSRQlECm4CXOgbpDXyuz6uvYH5D/As4YHJjYEMJ59daPkpEIqIENgHlOv4Fk0tgzx4YoL1/pDSzqcY4q4yujYiUFiWwCRi1B1iZjH/B5BLYw6+nt77esbBOy0eJSMEogU1AObfA5tRVUBnIOYf6nJ7B3FbjeDhcPq/5XyJSQEpgxyjhnmETy/JJYJUVxrz69LfFnp7xW2Gd/QnWa/koEYmQEtgxerVziK7B9HGeJdPKo4BjWLgbsa17/AT22O6+tJVJWhqrOH66lo8SkcJRAjtGmeZ/mZXXOM9ExsHWhpePUutLRApMCewYlfP417CJJLBR26do/EtECiyyBGZml5vZi2a21cxuyvD47Wb2bOrrJTNrDzx2m5ltMrMtZvYdK2KTp9z2AMvkWJeT2nZ4kJc7R5JcdQVcMF8JTEQKK5JBCjOrBO4ALgV2AuvNbLW7bx4+x90/Ezj/RuDs1O23AucDq1IPPwa8Hfh1FLEHuXuGLsQyTGDH2AILLx91ztwapmv5KBEpsKg+Zc4Btrr7K+7eD9wFXD3G+dcCP0vddqAOqAFqSW6kuaeAsWa1o3uIQ30jlQrTq4wTZ5ZfocKChmNbTio8/0u7L4tIFKJKYIuAHYH7O1PHRjGzpcAJwMMA7v4EsJbkRpq7gAfcfUtBo80iPP61cnY1FWVWwAHH1gIbTDiP7NL8LxGJXlTNh0yf8tlmx14D3OPuQwBmdjJwGrA49fhDZnahuz+a6cmtra2TjTWrX2+vJtkATDq+opvW1vbsTxhHIWOdjCODAA1H77d1DfLSS62YjY75uc4KOvpHWlyNVU79we20Hooo2ByU6nUei2KOhmKOxmRibmlpyfpYVAlsJ7AkcH8x0Jbl3GuA6wP33w+sc/cuADO7HzgXyJjAxvphJ2vntv3ASGvj7SfNoaVl2oReq7W1taCxToa7M+2pXXSnJnb1uzFn6Unsf+3lUTH/y7OdwOGj9y9e0sCppyymVJTydc5GMUdDMUejkDFH1YW4HmgxsxPMrIZkklodPsnMlgPNwBOBw68BbzezKjOrJlnAURJdiOWyB1iYmeU8mVnzv0SkWCJJYO4+CNwAPEAy+dzt7pvM7FYzuypw6rXAXe4e7F68B3gZeA7YAGxw9/uiiDto95EhdveMlJPXVcLypvIr4BiWSyFHR3+C9fvSqzKVwEQkKpF9Arv7GmBN6NjNofu3ZHjeEPCJggaXg42h1tfpzdVUlfFK65kKOZaGftzf7OpjKPCnxvLGKhZr+SgRiYgm6+Ro9BYq5Tf/KyiXSsS1oflfF6n6UEQipASWo42jVuAoz/GvYaNX4xidwDT/S0SKSQksR1NhDcSg0S2w9OWkXu0c5NXD6ctHnT+/vJO6iJQWJbAcHOpL8FrXyId1lcFpTeWewMYu4gh3H547t4ZpWj5KRCKkT5wcbAyNf53aXE1dVfkWcMD4Y2Cjug8XqftQRKKlBJaDqdZ9CDCvPj2B7etJMJjqRRxMOI/u0vwvESkuJbAcjEpgs8o/gdVUGnPqRt4eDuwfSLY6n97XT+fASP387NqKsq/KFJHSowSWg3AF4lT5sA53I+7rSyawhzOUz5fjosYiUtqUwMZxeCDB1o7Bo/cNOGMKtMBgdCHHvv5kktLyUSJSCpTAxvH8wYG0ZfNbGqumzGaNo1pg/UZ7X4Kn9oeXj1IBh4hEb2p8Ek/CVCzgGBaezLy333h0Vx+JQEY/ramKhaHzRESioAQ2jnACmyrjXzC6Bba/z1jbll4+r+WjRKRYtPLqOMJrIJb7ElJB4QS2t994PjT+peWjRKRYlMDG0DvovNg+mHZs1RQp4IDRCWxLVwXdQyMTmmsq4K1aPkpEikRdiGPYfGggbbuQpdMraaqdOpcsXIXYPZReKn/evFoaqqbO9RCR0qJPnzFM5QIOgFm1FdSOUZ/xTo1/iUgRKYGNYSqPfwGYGfPrs2cwzf8SkWJSAhvDhlF7gE2tFhiMHgcbNqeuYspM6BaR0qQElsVAwtk0RZeQCsqWwC5aqOWjRKS4lMCyeKF9kP7AHo4LGiqYO0Z3WrlaMC3zW+QibZ8iIkWmBJZFeA+wVVNs/GvYWC0wEZFiUgLLYqpXIA7LlMBWNFcxP0tiExGJihJYFhun4B5gmWRKYFp9Q0RKgRJYBkMJ5zlVIAKwMFMC0/wvESkBSmAZvNw5SPdg+o7Di6boiuvzGyoJ7h5TV5lcgUNEpNiUwDLItAK9TdGS8foq47rl047ev/GMGdRXTc1rISKlJbIEZmaXm9mLZrbVzG7K8PjtZvZs6uslM2tPHb8ocPxZM+s1s/cVMtaN6j5Mc9tbGrn/iuP40apePv+GmcUOR0QEiGg1ejOrBO4ALgV2AuvNbLW7bx4+x90/Ezj/RuDs1PG1wFmp47OArcCDhYxXFYjpzIzz5tXS2pkY/2QRkYhE1QI7B9jq7q+4ez9wF3D1GOdfC/wsw/H/Atzv7kcKECMA7j7l10AUEYmDqBLYImBH4P7O1LFRzGwpcALwcIaHryFzYsub7V1DdPSPFHDMrDaWzZiaBRwiIqUsqg0tM436e4ZjkExS97j7UPCgmS0AVgIPjPWNWltbJxTgsIf3VwIjVXYn1w/y8tatk3rNbCYbazEo5mgo5mgo5mhMJuaWlpasj0WVwHYCSwL3FwNtWc69Brg+w/E/BH7h7gMZHjtqrB82F3d1dgBdR++fu2QmLS1Nk3rNTFpbWycda9QUczQUczQUczQKGXNUXYjrgRYzO8HMakgmqdXhk8xsOdAMPJHhNbKNi+VVeAWOVbM0/iUiUooiSWDuPgjcQLL7bwtwt7tvMrNbzeyqwKnXAne5e1r3opktI9mCe6TAcfKsKhBFRGIhqi5E3H0NsCZ07ObQ/VuyPHcbWYo+8ml3T4J9vSOl4vWVRktjZJdIRESOgVbiCAiXz58xq4qqCq06ISJSipTAAkZPYNb4l4hIqVICC8i0BqKIiJQmJbCA0RWISmAiIqVKCSzlQO8QO7tH5k5XV8BpzUpgIiKlSgksJdz6Oq2pmtpKFXCIiJQqJbAUrUAvIhIvSmApSmAiIvGiBJaiLVREROJFCQzo6E/wyuGRAo4Kg9NnaQUOEZFSpgQGPH8wvfvwlMYqGqp0aURESpk+pdEEZhGROFICQ+NfIiJxpATG6DlgqkAUESl9qlQA7r50NhsODLDx4AAb9vezUktIiYiUPCUwYMn0KpZMr+K9S+uLHYqIiORIXYgiIhJLSmAiIhJLSmAiIhJLSmAiIhJLSmAiIhJLSmAiIhJLSmAiIhJL5u7FjmHSOjo64v9DiIjImBobGy14Xy0wERGJJSUwERGJpbLoQhQRkalHLTAREYklJbACMrMlZrbWzLaY2SYz+1SGc95hZh1m9mzq6+ZixBqKaZuZPZeK56kMj5uZfcfMtprZRjN7QzHiDMSzPHD9njWzTjP7dOicol9nM/uRme01s+cDx2aZ2UNm1pr6tznLcz+SOqfVzD5S5Jj/j5m9kPrd/8LMmrI8d8z3UcQx32Jmrwd+/1dkee7lZvZi6r19U5Fj/udAvNvM7Nksz438Omf7bIv8/ezu+irQF7AAeEPq9gzgJWBF6Jx3AL8qdqyhmLYBx43x+BXA/YAB5wK/LXbMgdgqgd3A0lK7zsCFwBuA5wPHbgNuSt2+CfhahufNAl5J/ducut1cxJjfBVSlbn8tU8y5vI8ijvkW4LM5vHdeBk4EaoAN4f+vUcYcevwbwM2lcp2zfbZF/X5WC6yA3H2Xu/8udfswsAVYVNyo8uJq4KeetA5oMrMFxQ4q5WLgZXffXuxAwtz9UeBg6PDVwE9St38CvC/DUy8DHnL3g+5+CHgIuLxggQZkitndH3T3wdTddcDiKGLJVZbrnItzgK3u/oq79wN3kfz9FNxYMZuZAX8I/CyKWHIxxmdbpO9nJbCImNky4GzgtxkePs/MNpjZ/WZ2eqSBZebAg2b2tJn9SYbHFwE7Avd3UjqJ+Rqy/0cvtesMMM/dd0HyQwGYm+GcUr7eHyXZGs9kvPdR1G5IdXv+KEvXVqle57cBe9y9NcvjRb3Ooc+2SN/PSmARMLPpwL3Ap929M/Tw70h2d50J/A3wr1HHl8H57v4G4N3A9WZ2Yehxy/CcopezmlkNcBXw8wwPl+J1zlWpXu/PA4PAP2Y5Zbz3UZT+DjgJOAvYRbJLLqwkrzNwLWO3vop2ncf5bMv6tAzHJnSdlcAKzMyqSf6C/9Hd/yX8uLt3untX6vYaoNrMjos4zHBMbal/9wK/INm1ErQTWBK4vxhoiya6Mb0b+J277wk/UIrXOWXPcPdr6t+9Gc4pueudGnh/L/BHnhrYCMvhfRQZd9/j7kPungC+nyWWUrzOVcDvAf+c7ZxiXecsn22Rvp+VwAoo1Xf9Q2CLu38zyznzU+dhZueQ/J0ciC7KUfFMM7MZw7dJDtg/HzptNfDhVDXiuUDHcLdBkWX9S7XUrnPAamC4CusjwC8znPMA8C4za051fb0rdawozOxy4HPAVe5+JMs5ubyPIhMao31/lljWAy1mdkKqNX8Nyd9PMV0CvODuOzM9WKzrPMZnW7Tv5ygrV6baF3AByabxRuDZ1NcVwJ8Cf5o65wZgE8mKp3XAW4sc84mpWDak4vp86ngwZgPuIFmx9RzwphK41g0kE1Jj4FhJXWeSyXUXMEDyr9CPAbOB/wBaU//OSp37JuAHged+FNia+vpvRY55K8kxjOH39P+XOnchsGas91ERY74z9V7dSPJDdkE45tT9K0hW1L1c7JhTx388/B4OnFv06zzGZ1uk72etxCEiIrGkLkQREYklJTAREYklJTAREYklJTAREYklJTAREYklJTCRGLPkKvsZ5wjl4bWXmZmnJtOKlBwlMBEBjm7LcUmx4xDJlRKYiIjEkhKYyDFKtVT+R2pl824z+6GZzUutcn/YzP59eLVzM/u5me225Gaajw6vgm9mNakNCG9M3a80s8dtnI02zazezH5sZofMbDPw5tDjC83sXjPbZ2avmtknA4/dYmb3WHKjxMNm9jszOzP12J3A8cB9ZtZlZv8z8LJ/ZGavmdn+1AK+IiVBCUxkYn4fuBQ4BbiS5JYifwkcR/L/1XDiuB9oIbmtxO9Irdzuyf2mPgjcamankdz8rxL4yjjf94skV1U/ieS+Skd3szWzCuA+kssKLSK5N9qnzeyywPOvJrla/yzgn4B/NbNqd/8Q8BpwpbtPd/fbAs+5AFieer2bU/GKFJ0SmMjE/I0nVzh/HfgNyV2pn3H3PpIrgp8N4O4/cvfDqeO3AGeaWWPqseeBv0qd/1ngQ+4+NM73/UPgK57cDHAH8J3AY28G5rj7re7e7+6vkFx5/ZrAOU+7+z3uPgB8E6gjuav2WL7k7j3uPrzm3pnjnC8SCSUwkYkJbtnSk+H+9FS34FfN7GUz6yS59TskW2nDfgIsI7k4a7YNC4MWkr4ZYHDn6aXAQjNrH/4i2SqcFzjn6HM9ubXIztRrjmV34PYRYHoOcYoUnBKYSOH8V5JddpcAjSQTFaRv6Pdd4FfAZWZ2QQ6vuYv0vZSOD9zeAbzq7k2BrxnufkXgnKPPTXU5Bvdi0sreEitKYCKFMwPoI7nNSwPw18EHzexDwBuB60iOmf0ktcPtWO4G/iK1l9Ji4MbAY08CnWb2uVSxR6WZnWFmwUKPN5rZ76Xmdn06Fd+61GN7SG7PIRILSmAihfNTkl18rwObGUkUmNnxwLeAD7t7l7v/E/AUcPs4r/ml1Gu+CjxIcp8rAFLjZ1cCZ6Ue3w/8gGTrb9gvgQ8Ah4APAb+XGg8D+N/AF1Ldj5+dyA8sEiXtByYyRZjZLcDJ7v7BYscikg9qgYmISCwpgYmUmNSE6K4MX39Z7NhESom6EEVEJJbUAhMRkVhSAhMRkVhSAhMRkVhSAhMRkVhSAhMRkVhSAhMRkVj6/wHFdhENcqCnDAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot max_depth (x-axis) versus RMSE (y-axis)\n",
    "plt.plot(max_depth_range, accuracy_scores)\n",
    "plt.xlabel('max_depth')\n",
    "plt.ylabel('Accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.8205754985754986, 4)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the best accuracy and the corresponding max_depth\n",
    "sorted(zip(accuracy_scores, max_depth_range))[::-1][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=4,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=1,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# max_depth=2 was best, so fit a tree using that parameter\n",
    "clf = DecisionTreeClassifier(max_depth=4, random_state=1)\n",
    "clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AtBat</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>League</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Division</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Assists</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Errors</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>NewLeague</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>PutOuts</td>\n",
       "      <td>0.006048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HmRun</td>\n",
       "      <td>0.010841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RBI</td>\n",
       "      <td>0.012073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Runs</td>\n",
       "      <td>0.021020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Walks</td>\n",
       "      <td>0.103473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Hits</td>\n",
       "      <td>0.298269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Years</td>\n",
       "      <td>0.548277</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      feature  importance\n",
       "0       AtBat    0.000000\n",
       "7      League    0.000000\n",
       "8    Division    0.000000\n",
       "10    Assists    0.000000\n",
       "11     Errors    0.000000\n",
       "12  NewLeague    0.000000\n",
       "9     PutOuts    0.006048\n",
       "2       HmRun    0.010841\n",
       "4         RBI    0.012073\n",
       "3        Runs    0.021020\n",
       "5       Walks    0.103473\n",
       "1        Hits    0.298269\n",
       "6       Years    0.548277"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute feature importances\n",
    "pd.DataFrame({'feature':feature_cols, 'importance':clf.feature_importances_}).sort_values('importance')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10.000000\n",
       "mean      0.820575\n",
       "std       0.083007\n",
       "min       0.692308\n",
       "25%       0.751781\n",
       "50%       0.830484\n",
       "75%       0.879630\n",
       "max       0.923077\n",
       "dtype: float64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(cross_val_score(clf, X, y, cv=10)).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Random Forests\n",
    "\n",
    "Random Forests is a **slight variation of bagged trees** that has even better performance:\n",
    "\n",
    "- Exactly like bagging, we create an ensemble of decision trees using bootstrapped samples of the training set.\n",
    "- However, when building each tree, each time a split is considered, a **random sample of m features** is chosen as split candidates from the **full set of p features**. The split is only allowed to use **one of those m features**.\n",
    "    - A new random sample of features is chosen for **every single tree at every single split**.\n",
    "    - For **classification**, m is typically chosen to be the square root of p.\n",
    "    - For **regression**, m is typically chosen to be somewhere between p/3 and p.\n",
    "\n",
    "What's the point?\n",
    "\n",
    "- Suppose there is **one very strong feature** in the data set. When using bagged trees, most of the trees will use that feature as the top split, resulting in an ensemble of similar trees that are **highly correlated**.\n",
    "- Averaging highly correlated quantities does not significantly reduce variance (which is the entire goal of bagging).\n",
    "- By randomly leaving out candidate features from each split, **Random Forests \"decorrelates\" the trees**, such that the averaging process can reduce the variance of the resulting model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting salary with a Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\albah\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.\n",
      "  from numpy.core.umath_tests import inner1d\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "clf = RandomForestClassifier()\n",
    "clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10.000000\n",
       "mean      0.817749\n",
       "std       0.062496\n",
       "min       0.703704\n",
       "25%       0.783333\n",
       "50%       0.811254\n",
       "75%       0.846154\n",
       "max       0.923077\n",
       "dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(cross_val_score(clf, X, y, cv=10)).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning n_estimators\n",
    "\n",
    "One important tuning parameter is **n_estimators**, which is the number of trees that should be grown. It should be a large enough value that the error seems to have \"stabilized\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of values to try for n_estimators\n",
    "estimator_range = range(10, 310, 10)\n",
    "\n",
    "# list to store the average Accuracy for each value of n_estimators\n",
    "accuracy_scores = []\n",
    "\n",
    "# use 5-fold cross-validation with each value of n_estimators (WARNING: SLOW!)\n",
    "for estimator in estimator_range:\n",
    "    clf = RandomForestClassifier(n_estimators=estimator, random_state=1, n_jobs=-1)\n",
    "    accuracy_scores.append(cross_val_score(clf, X, y, cv=5, scoring='accuracy').mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Accuracy')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbAAAAEGCAYAAAAE3cBCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl81dWd//HXJzshIQuL4IKgxAX3DVTc0HbcOrXTVTuiTrXLtNjW2k7t1J9D7WY7tXbRdjpWqlKrte201RmttgVxR7AgIlQCiIKIICGBkD35/P74fkPu/SYhN5C75v18PHhw7/d+773ncMP95HvO53yOuTsiIiLZJi/dDRAREdkbCmAiIpKVFMBERCQrKYCJiEhWUgATEZGsVJDuBgyFhoYGpVKKiOS4iooKi72vKzAREclKCmAiIpKVcjqA1dbWprsJKaO+5ib1NTepr0MjpwOYiIjkLgUwERHJSgpgIiKSlRTAREQkK+XEOjARyV7uzqr6Dhrb092SwXN3/l7fwZbmrkE97836PDZtak3o3JJ8OHZ0ESMKbOCThxkFMBFJmy53/vmvdTy6oYWSvBHMLWvmookj0t2shLg7s5+p577apr14dgmseCfhsw8pz+fhC8dywMj8vXiv3KUhRBFJm79sbOXRDS0AtHQZH1+4nVfrs+NS7McrGvcyeA3eup2dXLlgG62dKjoUSwFMRNLm3tW74u7v6nAun1/HjrbBDcml2sJNrcx5cUdK33PJ1na++kJDSt8z02kIUUTS4u2mTv4UXn3Fqm3oYPbT27lnZjVmmTfvs7Gxg6sX1tEVczE0qtA4bnRhwq/R3NzMiBEDD5Vuaupk7Y7O3fd//vddnDS2iMumlA6qzbkqZQHMzC4AfgjkAz9391sij08E7gEqw3NucPdHzGwSsAp4NTz1eXf/VKraLSLJ8cDaJjr6GRF76PUWfryikc8eU57aRg2gtdO56ok63mnpuUI04K5zqnn3gSUJv05tbS01NRMHPK++tYtzHt7C+p09Qey6Z7dzVFUBx44uGlTbc1FKhhDNLB+4A7gQmApcZmZTI6fdCDzo7icAlwI/iXlsrbsfH/5R8BLJcu7ea/iwOC8+ms15cQcLE8zUS5V/f6GBJVvj5+huOKF8UMFrMCqL87h3ZjUlMbkbLZ0wa34d9a2ZPcyaCqmaA5sGrHH3de7eBjwAXBI5x4FR4e0KYFOK2iYiKfbs221xQ2NFefCzY1qpLOoZMuxyuHphHW/u6uzrJVLuV7W7uOvv8UH3/AOL+dJxyb1KPHZ0EbedXhV37PXGTj7xZB1dPryTOlIVwA4ANsTc3xgeizUHuNzMNgKPANfGPDbZzJaa2UIzOzOpLRWRpItefb3n4BEcVd7FnWdXEzvr9U5LV0Zk3720rY0vPFcfd2xSeT4/O6uavBTM0102pZSrjxgZd+zxja3850s7k/7emcw8BRHczD4EnO/u14T3ZwHT3P3amHO+ELbnVjM7DbgLOBooBMrcfZuZnQT8ATjK3XenAMVuaDmcqjyLZKMdHXDRCyNo7er54r/96BamVwZDYne+UcB/vxE/v/PBCe18+dD0pNc3tMMVy0rY1Nrz+35xnjP32BYOK0tdYG3vgk++XMzLO3vGEw3ntqmtzKjO3eHEmpqa3bejG1qmKoljI3BQzP0D6T1EeDVwAYC7P2dmJcAYd98CtIbHXzSztcBhwJK+3ii2s8FEaU1fp+Uc9TU35WJf71zVSGtXTzr4xLJ8PnryIaxds4aamhpumeKs/8s2Ht/YM//127cKOW/KuJRn33W585E/b2NTa/xc3A9mVHPxPrRlbz/XBw7s5OyHtuxOInGMOWtG8MR7xzGpPDOTypP5M5yqIcTFQI2ZTTazIoIkjYci57wBnAdgZkcCJcBWMxsbJoFgZocANcC6FLVbRIaQu3PP6vjFv7NqSuOG4fLM+O+zqplUHl914rpnt7N8W1tK2tntu8t28uc344PXNUeMTFsa+wEj85l7TjV5Mdch9W3OrPl1NPeX0pnDUhLA3L0DmA08RpAS/6C7v2JmN5vZe8PTrgc+bmYvAfcDV3kwvnkWsDw8/lvgU+5el4p2i8jQemlbOyvqeoYC8ww+WjOy13mZkH33+IYWvrMsfo7plLGFfGtaRUrevz9nTShmzkmj4o69XNfOF56rJxVTQpkkZdec7v4IQXJG7LGbYm6vBGb08bzfAb9LegNFJOnujVx9vfuA4n7r+3Vn3/3rU9t3H+vOvnvgXaOTmjyxfmcHH3+yjthwMLYkj7tnjqYoP/2Lq689uozFW9t4+PWeheD3r2li2tgi/uWI3r8Q5CqVkhKRlNjV3sVv10WGDw/b85dtOrLvmjq6uHx+HQ1tPeEr32DuOdUZU0zXzLjjjCpqKuKvQf5tUT1LtqZ2mDWdMnPWT0Ryzh/XN7OjvScojBuRx/kHDbwA+FvTKnhpW1vcAuJblu7kxDFFQ76A2N25/rmGuGFOgDknjeLMCcVD+l77alRRHvPOrea8h7eyK5z/au+CK+fXcffMaoozINaOLkluIxTARCQl5kUqt390SimFeQMPxxXnG/fMHB3JvoOPL6wb8uy7X7zaxP1r4tv53oNLmH102ZC9x1A6orKQO86o4qonetIC3mzq5N3/tzWNrepxzREj+eSY5L2+hhBFJOlW17fz3NvxQ1uz+kje6E9/2XdXDGH23ZKtbXx5Ufxi5cMqCrjjzKqMLCrc7X2TRzD7qMwMsMmmACYiSRe9+poxvohDKwZ35dRX9t3yunauH4Lsu63NnVw5v472mATHsgJj3rnVlBdm/tfknJNHMWP88CvuqyFEEUmqtk7vNSx35QDJG/3pK/vuV2uaOGUfsu86upyPPVHHm03xNRdvP6OKwysT3yIlnQryjPvOHc1NSxpY+k47mZJMP6FUc2AiksUe3dASt/1IRZHxjwcPvBdWX7qz7/5ev5Xaho7dx/9tUT3HjC7k5LGDvwr5xt928NTm+OHN2UeV8b7Je9fGdKkszuNHM6oGPjHFams3J+21M//aWESy2rxI4d4PH1rKiIK9n1Pqzr4bGfMa3dl377QMrnL9Q+ub+cHLjXHHzhhfxJyTR/XzDMkkCmAikjRvNHbw10gppiv2cvgwVnf2Xaw3mzr52BPb6ehKbABtdX07n3l6e9yxCaV5zD2nmoIEsiMl/RTARCRp7qttipuPOWFMIcdUD828Ul/Zd0++1co3/7ajn2f0aGzvYtb8OnbGrEsrzIN7ZlYzbkQGLKCShCiAiUhSdHY590WyD68YROp8IvrKvrvt5UYefr253+e4O9c+Xc+rMXNoECyYnjYusxYry54pgIlIUizY1MrGmN2USwuMDxwytIkRBXnGL86pZkJp/FfZp5/aTm1D3/uH/WTlLn6/Pj7AffjQEVwzjGoI5goFMBFJiuiuy++bNIJRRUP/lTNuRD53n1NNbF7IzvZgi5HG9vjK9c9sbuWmxQ1xx46qKuAHp1dm9GJl6ZsCmIgMua3NnTzyRkvcsSsOS94eWtP3K+61zcnf6zv47DM9i5zfaurkX56oozNmUm5UkfHLc0dTWqCvwmykT01EhtwDa5qIrfB0WEUB08clt1LEx48cyYcjQ5T/81ozP125i7ZO56oFdWxpjr8i+++zqpg8Ssths5U+OREZUu7OvbXRbVNKkz5EZ2bcdnolK7a3s3J7T4LG/1vcwMJNLSzaEr9Y+UvHlXPBQdm1WFni6QpMRIbU81va4qpkFObBpYcmb/gw1sjCPH557mhGFfUEy06HxzbGr0U774Bibji+PCVtkuRRAJOctbmpk9tX7OTxDS0DnyxD5p5X45M3LppYwtgUrq06ZFQBPzuz/5JKB5Xlc+dZVeRrsXLWUwCTfnW58/UXGzjuN5v5xMI6WoZo24pUaGzv4tyHt3Dj4h18+C/b+O6ygRe3yr6b/2YLv14bn6I+FJU3BuvCiSP44nG9r7CK82HezGqqk7zRoqSGApj06zvLdnLr8kZeb+zkwXXN3Lo8edu4D7U/bWhhU1PPhP23lu7kMV2JJdUbjR1cs3B7XOWNg8vyOSdNOxl/5fhyzt0//r1vPa2S48cMv21HcpUCmPTpsQ0tfGdZfMC6d/WuhOvMpdvCTa29jn3iyTpe29HRx9myr1o6gs0l61p7fmnIM/jhjMq0DdXl5xm/mFnNrJpSThhTyI9mVHL5EFcCkfRSFqL0sn5nB594sq7X8bebu3h8YwsXTcz8zK2Fb/UOYA1tzqwFdTx+8Rit+xli/7aonmXb4itf3HjiKM7ZvyRNLQpUFOXx4zMyb4sRGRr6Xyxxmjq6uHx+HQ1tfV9p3bu6qc/jmWT9zg7eaOx7W40Vde1c9+y+7+ArPe5dvavXz8VFE0v4/DHDc5t7SR0FMNnN3fnCs/WsqOu7hhzA4xtb2LRrcHsupdqTfVx9xfr12mbmRjLlZO8sfaeNLz1fH3fs0FH5/PTMKvJUmkmSTAFMdpv76i4eiGSQXTKphKOqekaau5xe28Nnmuj813XHlHF4Rfxo+Q2LGlgcWdgqg7OtpZNZ8+tojfl9prTAmHfuaCqSUPNQJEo/ZQLA4i1t3LAovsjpYRUF3H5GFVdG0qDn1e6iK0OH4Ny91xXYRRNHMO/casqiO/gu2MaW5sy+msxUnV3ONQu3x1WbB/jRjEqmVg3Nfl8iA1EAE7Y2d3Llgm3EFu4uKzDmnVtNeWEeHz60lOKYZTPrd3by1ADDdOmyqr6DrS09HSkvNE4YU8hhlYXcEVncuqmpi489UZc1mZWZ5NtLd7IgcqX7qakj+eAhqam4IQIKYMNeR5fzsSfq4tZMAdxxZhWHVwa/SVcW53HJwfGZh5mazBEdPjx9fPHu7eEvmTSCzx4dn1jw9OY2bn5Ri5wH45E3mvleZE3gqeOK+PopFf08QyQ5FMCGua+/uIOnNsfPBV17dBmXTIoPWLMiw4gPv95MXUvmDb9F0+fPjiyivemkUZwR2cH3Rysa+eP6/nfwlR5rGzr41JPb447tNyKPu2dWU6jSTJJiCmDD2B/XN/PDFY1xx84YX8R/nDSq17lnjC/ikPKeccS2LnqVDEq3ji7n2c3xAeysSADr3sF3/8gOvp95ajuv1veffSmwq72LWQu2saO9Z8i1wODumdWML1VpJkk9BbBhanV9O7Ofjv9Nev/SPOaeU717yC2WmfW6Cpu3eldGradatq097st1TEkeU6t6r9UfOyKfe2aOpjDmp7+xI9jBd2dkB18JuDuff7Y+bpsSgK+fUsFp+6WnVJRIyipxmNkFwA+BfODn7n5L5PGJwD1AZXjODe7+SOTxlcAcd/9eqtqdixrbu8Iv654v+8K84DfpcXuoGn7ZlFK+8bcdu3e0XVnfwYvvtHPy2MyoLRed/zprQnG/a5FOGVfELdMruP65nszL1Q0dzH56O3efU53Q3lUdXU5LZ3IDeFNn8Hml272rm/jNuvgr7g9MHsGnpqo0k6RPSgKYmeUDdwDvBjYCi83sIXdfGXPajcCD7v5TM5sKPAJMinn8NuDRVLQ3l7k71z5dz6sN8b9Jf3taBdPG7fk36fGl+Zx/UEncVvH3rt6VOQFsgPmvqI8dPpLFW9ri1r79cX0Ln3xyO9UleTS2e/inK/i7I+Z2exepmQIshefeSsUbDcqRlQX8cEZl0jepFNmTVF2BTQPWuPs6ADN7ALiE4IqqmwPdky8VwKbuB8zsfcA6QOUT9tEdrzTy+0jCwkcOHcHVRyT2m/QVh5XGBbDfrWvmm9MqKC9M72h0S4ezaMue57+izIzvn17Jiu0dcdVHHlyXWXN7mWZUYbDEoizNn7lIqn4CDwA2xNzfGB6LNQe43Mw2Elx9XQtgZiOBLwNfS34zc9vTm1v5jyXxKeNHVxdy2+mJ/yb9rgNKmBCTALGrw/n9a+n/wl+0pS2uIsRBZflMKh84saC0II95M6upKNKVRKJ+emYVUyq0WFnSL1VXYH19O0QnDy4D7nb3W83sNGCemR1NELhuc/fGRL5ka2tr93g/l+2pr1tajSuWldDpPf+G5fnO1yfv4M3XGvp9Xl8uHF3I3KaeL7A7l2/jNNu0h2cMvWhf/7C+EOhp0wkjW1mzZk3Cr/e1KXlcv7KYzj5/VPtmOMV5ff9w56KKQudfDmrnsLaNJOu/lf6/5qZ96WtNTU2/j6UqgG0EDoq5fyAxQ4Shq4ELANz9OTMrAcYA04EPmtl3CRI8usysxd1v7+uNYjtbW1u7x87nkj31ta3Tmf2nd6hrj1/v9fOZY5h50OC3u/js+A7mbnh79/2Xd+bTPmZSykoI9dXXl1/dAvQMA77n8LHUHJp4VYga4OhD2vjzxhYMKCs0ygvzKCs0ygrzGFlolBXEHjNKCyzpc0D6Gc5N6uvQSFUAWwzUmNlk4E3gUuCjkXPeAM4D7jazI4ESYKu7n9l9gpnNARr7C17StxsXN7AoUrj2344v5/y9CF4Ak8oLOGf/Yp6Iyfqbt3oX355euU/t3FsNbV387Z34NVwDzX/15eSxRRmTkCIiA0vJHJi7dwCzgceAVQTZhq+Y2c1m9t7wtOuBj5vZS8D9wFWeSYuMstSDa5v471XxuS/vOqCYLx9Xvk+ve0VN/NXNA2ubaE1ySnl/nt3cSmw5wyMqC7SwVmQYSNk6sHBN1yORYzfF3F4JzBjgNeYkpXE5akVdO597Jn6vpoll+dx5dvU+b/N+8cEjqCquZ3trEDm2tzr/+3ozH0hDMddo+vzeXH2JSPZRHmyOqm/tYtb8bTTHXBWV5MO8c6upKt73j70437g0MseUrgK/0e1TFMBEhgcFsBzU5c6nntrOazvjV9reelolx40eujmeaGmphW+1sn5nRz9nJ8fW5s648kZ5BmeMVwATGQ4UwHLQ95c38qcNLXHH/uXwUv65ZmjL/kytKuSUsfGZh79M8VVY9Orr+NGFVA7BFaaIZD79T88x899s4Zt/i1+sfOKYQm5JUoZg9CrsvjW7UrpBpOa/RIYvBbAc8vrODq5eWBe3Qnx0cR73zqymOD8565XeP3kEZQU9r/1WUxd/ebNlD88YWtErsIHqH4pI7lAAyxGtXXDlgrrdWYEQzAfNPaeKA8uSl2xaVpjH+w9Jz27Nr+/sYH3MPF9RHkzfT+u4RIYLBbAc8Z9ri1i2LX4x700njuLs/fdusfJgXBEZRnxsQwubm5Jfqj06fDhtXBGlBfqRFhkuUrYOTJLn3tW7+OPb8R/leyaW8LljylLy/ieNKWRqZQEr64NswE6H+9c0cd2xwWLpzi5nZ7uzK7Idyc5wu5JRRcZ5B5QMepjzKc1/iQxrCmBZbuk7bXzxufjFylNGFXDHmVUp26upe7fmr7zQUxT4O8t28JNXGmls97i1aP2ZPq6IP5w/hhEFibXZ3TX/JTLMabwli21p7uTyv9bRFrNhb2lBsFdTRVFqP9qPHDqC2Lds6YStLV0JBS8ItkP54vP1JFo97O/1Hbzd3NPxsgLjRNUxFBlWFMCyVFunc8X8Ot6MzDXdPqOSI1NUFT5WdUk+/3jwiIFP3IP7apsSTgCJzn+dPr6Iwn0sjyUi2UVDiFnI3fnS8/U8H6kw/+mjRvL+NNQi7Pa1k0fx/NttvYJq9/YkIwuCrUm6tyMpK8xjRV07G3f1nP+l5+s5prpwwKsplY8SEQWwLHTX33dxT+RKZXplJzefXJGmFgUOLCtg+Yf2o3ZHBwXG7mBVWmDk9TMft2p7O+/6363s6giGDtu64IoFdTzx3rGMKem7onyHB7tLx0pFtqWIZBYNIWaZp95q5YZF8TsoTy7P55uHt1KQAUNo+XnGEZWFTKkoZHxpPmWFef0GL4Ajqwr58Yz4KiEbd3Vy9RPb6eynoserjXnsaOt5bHRxHkdV6XcxkeFGASyLvL6zg6sW1NER871eVmD86rzRVKR+2mvIvP+QUj59VO/CwN9cuqPP8xfXx//YnjmheI9BUkRykwJYltjV3sVH/7qNba1dccd/dlZVWpI2htrXTq7gtEgVje8vb+R/X2/ude7ihvihRaXPiwxPCmBZwN359NPbeWV7/FYlXz2hnIv3MfMvUxTmGXefU834EfE/kp9+ajtrGnoqjLR0OC/tiD/n7P0VwESGIwWwLPC9l3byx/XxBXIvmVTCF48rT1OLkmO/0nx+MbOa2LXMO9qdWfPraGwPrjxf2NpGa1fPCQeOzGdyed/JHiKS2xTAMtz/vd7MN5fujDt2VFUBd5yRukobqXTafsV8Y1p8NuWq+g4++0x9n9U3zppQnJP/DiIyMAWwDLZqezuffHJ73LHq4jzuO280ZYW5+9F98siRfChS4f5/Xmvmv1bu4slN0fR5DR+KDFfKPc5Q21uDpI3GmJTDfIN7ZlYzqTy3PzYz4wenV/JKXfvuAsEANy5u6HWuFjCLDF+5+2t8Fuvocj72RB2v7YyvaHHL9ArOHCZf2CML85h37mhGFfYMD3Z68KfbYRUFTCjV/JfIcJXQr/Jm9lngV+7+TpLbk5OaOrp4ZnMbzR2JFar988YWFkSGyq44rJRrjhjZzzNy06EVBfz0zCr+eX5dn48rfV5keEt0LOpdwLfM7AlgHvAHd2/d81ME4O2mTk7/w5Ze67cG49RxRXzv1Mphmaxw8cEjuP7YMm5d3tjrsbM0/yUyrCU0hOju7wUOBh4FPg9sNrOfm9lZyWxcLvjxisZ9Cl4HlOZz77nVFA1ys8dc8u8njGJmJFgZcOZ4BTCR4SzhOTB33+bud7j7acDZwCnAAjNbb2ZfNbPUbP+bZZ54a+8vVEvy4ZfnVTNuxPCe58nPM35+dhUHlfX8O1w4sYTKYk3higxng0pnM7PzgMuBS4AlwHeBN4DPEVydnTnUDcxm77R0sqKup4qEARdNLCGRa6mK4jw+dvhIThijTRoBRpfk8+eLx/LDl3fStKOe/5gxPt1NEpE0SzSJ43vApUADcC9wo7u/GfP488D2fp4+bD0Vufo6prqQ+84bnabWZL/xpfl8e3oltbVbqe5nqxURGT4SvQIrAf7J3Rf39aC7t5vZyUPXrNwQrRqhRbciIkMn0QD2bSBuB0UzqwJGuPsmAHf/+xC3Lest3KRdg0VEkiXRWfA/AAdGjh0I/H5om5M7NjR2sC5mIXKB0Wu7EBER2XuJBrDD3f3l2APh/SMSfSMzu8DMXjWzNWZ2Qx+PTzSzBWa21MyWm9lF4fFpZrYs/POSmf1Tou+ZTtHhw1PGFeV0/UIRkVRL9Bt1i5lNiT0Q3t+WyJPNLB+4A7gQmApcZmZTI6fdCDzo7icQJIz8JDy+AjjZ3Y8HLgB+ZmYZXwxwYSSADZcSUCIiqZJoAJsL/M7M3mNmU83sH4HfAj9P8PnTgDXuvs7d24AHCFLxYzkwKrxdAXTPrTW5e3dF15LwvIzm7r2rpiuAiYgMqUSvZG4B2oHvAQcBGwiC1/cTfP4B4XO6bQSmR86ZAzxuZtcCIwnKVwFgZtMJgujBwKyYgJaRVjd0sLm5p/pGaYFxyljNf4mIDCVzT/4FjZl9CDjf3a8J788Cprn7tTHnfCFsz61mdhpwF3C0u3fFnHMkcA9wlrvv3qK4oaFhdydqa2uT3p+BPLipgP9c1xOwTq3s5MdHq3SkiMhg1dTU7L5dUVERVwci4bkkMysCDgfGQE8xCXefn8DTNxJcuXU7kHCIMMbVBHNcuPtzZlYSvteWmPdaZWa7gKMJKoH0EtvZ2trauPupsuqNbcDu+MqFU6qoqSlP6numq6/poL7mJvU1NyWzr4lW4jgD+A1QTDBPtQMoJxgWPCSBl1gM1JjZZOBNgiSNj0bOeQM4D7g7vNIqAbaGz9ng7h1mdjBBEF2fSLvTobPLeWqz5r9ERJIt0Suw24DvuvttZrbd3avN7CYii5v7Ewaf2cBjQD4w191fMbObgSXu/hBwPXCnmV1HkKhxlbt7GDxvMLN2oAv4dCbvS7a8rp2Gtp5h2coi45jqwjS2SEQkNyUawA4Dfhg5dgvwGkFix4Dc/RHgkcixm2JurwRm9PG8eQR7kGWF6PqvMycUk583fLdCERFJlkTT6BvoSXF/K1zDVQVoC5UIlY8SEUmNRAPY/wAXhbfvAhYALxLMi0motdN57u22uGOa/xIRSY6EhhDd/fMxt281s0UESRyPJath2WjJ1jaaO3vmvyaU5lFTkfFFQ0REstKA365hGajVwFR3bwVw96eT3bBs1Ff5KDPNf4mIJMOAQ4ju3gl0EqS1yx6ofJSISOokOr71A+BBM/sWwaLk3eNk7r4uGQ3LNo3tXSzZqvkvEZFUSTSA3R7+/e7IcSdY1zXsPfd2Gx0xVbkOHZXPgWWa/xIRSZZEkzi0kdUAlD4vIpJaCkxDJJrAcfYETRmKiCRTorUQn6Kffbjc/awhbVEWqmvp5OW69rhjZ07Q9ikiIsmU6CRNdOPK8QTV4385tM3JTk9tjk/eOLq6kNElmhoUEUmmROfA7okeM7PfAb8Abh7qRmWb6PyXsg9FRJJvX+bA3gSOHaqGZLOFb7XE3VcAExFJvkTnwD4WOVQKvB94fshblGU2Nnawdkfn7vsFBqeN1/yXiEiyJToHNityfxfwLME+YcNadPuUk8YWUV6o5E4RkWRLdA5sZrIbkq2i6fNa/yUikhoJXSqY2RVmdmzk2HFmFr0yG1bcnaei67/2VwATEUmFRMe6vg5siBzbAHxjaJuTXdbs6GBTU9fu+yPyjVPGav5LRCQVEg1go4AdkWMNQOXQNie7RNPnT92viOJ8bZ8iIpIKiQawlcAHIsf+CVg1tM3JLr3LR2n4UEQkVRLNQvwy8IiZfQRYC0wBzgMuSlbDMl2X5r9ERNIqoSuwcAfmo4DFwEjgBeBod38miW3LaMu3tVPf1lMesqLIOLa6MI0tEhEZXhJdyFwMbHb3W2KOFZpZsbu37uGpOSu6/uuM8cXk52n+S0QkVRKdA/szcFLk2EnAY0PbnOwRDWCa/xIRSa1EA9gxwKLIsReA44a2OdmhrdN59u34CvSa/xIRSa1EA1gDsF/k2H4EJaWGnSVb22jq6Jn/Gj8ij8MqEs2HERGRoZBoAPsd8CszO9rMSs1lXF62AAAQ30lEQVTsGGAe8JvkNS1z9VU+ykzzXyIiqZRoAPsqwZqvF4BGgir0q4Abk9SujBad/zpLw4ciIimXaBp9i7t/hiCFfj/gNKAVqE1i2zLSrvYulmyNn/9SAV8RkdRLeN8PMxsLfJYg83ApcDLwuSS1K2Mt2dpGe0/5QyaX5zOxTPNfIiKptsdvXjMrBN4LXAWcD6wB7gcmAR929y1Jbl/GWVXfEXf/1P109SUikg4DXYG9DfwMeBU41d2nuvvXCYYPB8XMLjCzV81sjZnd0MfjE81sgZktNbPlZnZRePzdZvaimb0c/n3uYN97KK1piA9ghyv7UEQkLQYKYMsJKs5PB04xs6q9eRMzywfuAC4EpgKXmdnUyGk3Ag+6+wnApcBPwuPvAP/o7scAVxJkP6bN6kgAm6IAJiKSFnsMYO5+DnAo8DjwRWCzmT1MkMwxmMJ/04A17r7O3duAB4BLom9HsG0LQAWwKWzDUnffFB5/BSgJS1ulxZqG9rj7Wv8lIpIeAyZxuPvr7v51d68hqED/FtAFvGRm303wfQ4gfkPMjeGxWHOAy81sI/AIcG0fr/MBYGm66i/ubO+K28Ay32BSuQKYiEg6mLsPfFb0SWYlBPuBXeHuFyZw/oeA8939mvD+LGCau18bc84XwvbcamanAXcRVLzvCh8/CngI+Ad3Xxv7+g0NDbs7UVubvMz+VY3GFctG7L4/saSL353ckrT3ExEZ7mpqanbfrqioiKsYsVeXD+7eQpCNeH+CT9kIHBRz/0DCIcIYVwMXhK//XBgkxwBbzOxA4PcEAXMtexDb2dra2rj7+2rZ2iZg++77U8eWUlNzUP9PSKGh7msmU19zk/qam5LZ14TXge2jxUCNmU02syKCJI2HIue8QTBEiZkdCZQAW82sEvg/4Cvp3n8smsBRo/kvEZG0SUkAc/cOYDbBIuhVBNmGr5jZzWb23vC064GPm9lLBFd2V3kwvjmbYAfo/2dmy8I/41LR7qhoCr0CmIhI+qTsG9jdHyFIzog9dlPM7ZXAjD6e9w3gG0lvYAJWRzIQFcBERNInVUOIWa/LnXU7OuOOKYCJiKSPAliCNu7qpLmzJ2OzqtgYXax/PhGRdNE3cIJqo/Nfowq1B5iISBopgCWoVwCr1PChiEg6KYAlqPcVmAKYiEg6KYAlqFcAUwKHiEhaKYAlKFrEVwFMRCS9FMASoCK+IiKZRwEsAWsjw4eTywsoylcGoohIOimAJUCbWIqIZB4FsAREEzi0iaWISPopgCUgWsRXV2AiIumnAJYAFfEVEck8CmAD6HJn7Q4NIYqIZBoFsAFsaOykJaYIfVWxMbokP30NEhERQAFsQGt6XX0VpqklIiISSwFsANEMRCVwiIhkBgWwAaiIr4hIZlIAG4CK+IqIZCYFsAHUKoVeRCQjKYDtwc72Lt6KKeJbYDBZQ4giIhlBAWwPohU4JpUXUJinIr4iIplAAWwPNP8lIpK5FMD2QAFMRCRzKYDtgdaAiYhkLgWwPYhmIKoGoohI5lAA60dfRXw1hCgikjkUwPoRLeJbXZynIr4iIhlEAawf0SK+uvoSEcksCmD9WF2vBA4RkUymANaP3tuoKICJiGSSlAUwM7vAzF41szVmdkMfj080swVmttTMlpvZReHx0eHxRjO7PVXtXV0fn4E4RSWkREQySkoCmJnlA3cAFwJTgcvMbGrktBuBB939BOBS4Cfh8Rbg/wFfTEVbu/W6AqtUABMRySSpugKbBqxx93Xu3gY8AFwSOceBUeHtCmATgLvvcvenCQJZSvRVxHdSuQKYiEgmSdW38gHAhpj7G4HpkXPmAI+b2bXASOBde/NGtbW1e7yfiJU784CS3ff3L+li/do1e9OclNqbvmYr9TU3qa+5aV/6WlNT0+9jqQpgfZVw98j9y4C73f1WMzsNmGdmR7t7Vx/P7VdsZ2tra/fY+f4sXdsEbN99/6gxpdTUHDTo10mlve1rNlJfc5P6mpuS2ddUDSFuBGIjwIGEQ4QxrgYeBHD35wgugcakpHURq1XEV0Qk46UqgC0GasxsspkVESRpPBQ55w3gPAAzO5IggG1NUfviRPcBUwATEck8KflmdvcOM5sNPAbkA3Pd/RUzuxlY4u4PAdcDd5rZdQTDi1e5uwOY2XqCBI8iM3sf8A/uvjJZ7V0dKeKrACYiknlS9s3s7o8Aj0SO3RRzeyUwo5/nTkpq42J0ubNOZaRERDKeKnFEqIiviEh2UACL0C7MIiLZQQEsQgFMRCQ7KIBFKICJiGQHBbCIWmUgiohkBQWwCG1kKSKSHRTAYuxoUxFfEZFsoQAWY23k6mvyqAIK8/oq4ygiIummABYjWgNRm1iKiGQuBbAY0QzEwzT/JSKSsRTAYkSL+E5RABMRyVgKYDFUxFdEJHsogIU6u3oX8dUQoohI5lIAC23Y1buIb7WK+IqIZCwFsFB0/ktXXyIimU0BLNQrhV4BTEQkoymAhaJXYErgEBHJbApgIRXxFRHJLgpgIW2jIiKSXRTACIr4bm5WEV8RkWyiAEbv+S8V8RURyXwKYECt9gATEck6CmD0Mf+lKvQiIhlPAYzeGYhaAyYikvkUwNA2KiIi2WjYB7DOLu+1E7PmwEREMt+w/6Y2g8cuGkttQwerGzrY1NSpIr4iIllg2AewPDOOH1PE8WOK0t0UEREZhGE/hCgiItlJAUxERLKSApiIiGSllAUwM7vAzF41szVmdkMfj080swVmttTMlpvZRTGPfSV83qtmdn6q2iwiIpkrJUkcZpYP3AG8G9gILDazh9x9ZcxpNwIPuvtPzWwq8AgwKbx9KXAUsD/wFzM7zN07U9F2ERHJTKm6ApsGrHH3de7eBjwAXBI5x4FR4e0KYFN4+xLgAXdvdffXgDXh64mIyDBm7p78NzH7IHCBu18T3p8FTHf32THnTAAeB6qAkcC73P1FM7sdeN7dfxmedxfwqLv/tvu5DQ0NuztRW1ub9P6IiEhq1NTU7L5dUVERt01IqtaB9bU3STRyXgbc7e63mtlpwDwzOzrB5+4W21kREcldqQpgG4GDYu4fSM8QYbergQsA3P05MysBxiT4XBERGWZSNQe2GKgxs8lmVkSQlPFQ5Jw3gPMAzOxIoATYGp53qZkVm9lkoAZ4IUXtFhGRDJWSKzB37zCz2cBjQD4w191fMbObgSXu/hBwPXCnmV1HMER4lQcTdK+Y2YPASqAD+Ew0AzE6LioiIrkvJUkcIiIiQy1nK3EMtHA625nZejN72cyWmdmS8Fi1mf3ZzGrDv6vS3c69YWZzzWyLma2IOdZn3yzwo/BzXm5mJ6av5YPTTz/nmNmb4ee6LFcW9JvZQWGhglVm9oqZfS48noufa399zbnP1sxKzOwFM3sp7OvXwuOTzWxR+Ln+Opw6IpwK+nXY10VmNmmfGuDuOfeHYJhyLXAIUAS8BExNd7uGuI/rgTGRY98Fbghv3wB8J93t3Mu+nQWcCKwYqG/ARcCjBNmqpwKL0t3+feznHOCLfZw7Nfw5LgYmhz/f+enuwyD6OgE4MbxdDqwO+5SLn2t/fc25zzb8fMrC24XAovDzehC4NDz+X8C/hrc/DfxXePtS4Nf78v65egWWyMLpXHQJcE94+x7gfWlsy15z9yeBusjh/vp2CXCvB54HKsM1hRmvn372J6sX9Lv7W+7+t/D2TmAVcAC5+bn219f+ZO1nG34+jeHdwvCPA+cC3Wt1o59r9+f9W+A8M9vrHIZcDWAHABti7m9kzz9A2ciBx83sRTP7RHhsP3d/C4L/RMC4tLVu6PXXt1z8rGeHw2ZzY4aBc6af4bDRCQS/ref05xrpK+TgZ2tm+Wa2DNgC/JngCrLe3bu3uo/tz+6+ho83AKP39r1zNYANavFzlprh7icCFwKfMbOz0t2gNMm1z/qnwKHA8cBbwK3h8Zzop5mVAb8DPu/uO/Z0ah/Hsqq/ffQ1Jz9bd+909+MJ1uhOA47s67Tw7yHta64GsJxf/Ozum8K/twC/J/jBebt7mCX8e0v6Wjjk+utbTn3W7v52+IXQBdxJz1BS1vfTzAoJvtDvc/f/CQ/n5OfaV19z+bMFcPd64AmCObBKM+tephXbn919DR+vIPFh9F5yNYAlsnA6a5nZSDMr774N/AOwgqCPV4anXQn8MT0tTIr++vYQcEWYtXYq0NA9JJWNIvM8/0TwuUKWL+gP5znuAla5+/djHsq5z7W/vubiZ2tmY82sMrw9AngXwZzfAuCD4WnRz7X78/4gMN/DjI69ku4slmT9IchiWk0wHvvVdLdniPt2CEHW0kvAK939IxhL/itQG/5dne627mX/7icYYmkn+I3t6v76RjAkcUf4Ob8MnJzu9u9jP+eF/Vge/mefEHP+V8N+vgpcmO72D7KvZxAMFS0HloV/LsrRz7W/vubcZwscCywN+7QCuCk8fghBEF4D/AYoDo+XhPfXhI8fsi/vr4XMIiKSlXJ1CFFERHKcApiIiGQlBTAREclKCmAiIpKVFMBERCQrKYCJiEhWUgATyVBm9qiZXTnwmSLDk9aBiWQAM5sDTHH3y1PwXpOA14BC7ym4KpJ1dAUmIoMSU+NOJK0UwET2wIKdr78YboHREO4mWzLAc94T7rhbb2bPmtmxMY99OdyVd2e4++55ZnYB8O/AR8ys0cxeCs99wsyuCW9fZWbPmNlt4euuM7PTw+MbLNjZ+cqY97nYzJaa2Y7w8TkxTXwy/Ls+fL/TzCzPzG40s9fD17rXzCrC15pkZm5mV5vZG8D8cCfeX5rZtrA9i81sv6H4NxdJlAKYyMA+DFxAsFvuscBV/Z1oZicCc4FPEtT5+xnwUFio9XBgNnCKu5cD5wPr3f1PwLcIdqctc/fj+nn56QQ150YDvyLYqPUUYApwOXB7uIUHwC7gCqASuBj4VzPr3lSwe+udyvD9ngv7dBUwk6COXRlwe+T9zybYKuN8goKsFQSVxUcDnwKa+/t3EUkGBTCRgf3I3Te5ex3wMMF+Tv35OPAzd1/kwdYZ9wCtBFtMdBJsGz/VzArdfb27rx1EO15z91+4eyfwa4LgcbMHO/k+DrQRBDPc/Ql3f9ndu9x9OUHh4LP38Nr/DHzfg13MG4GvEFRIjx0unOPuu9y9maAA8WiCebtOd3/R97y/l8iQUwATGdjmmNtNBFcn/TkYuD4cVqs3s3qCQLO/u68BPg/MAbaY2QNmtv8g2vF2zO1mCPaYihwrAzCz6Wa2wMy2mlkDwRXSmD289v7A6zH3XwcKgNhhwdhdg+cBjwEPmNkmM/tuuAeWSMoogIkMrQ3AN929MuZPqbvfD+Duv3L3MwgCnQPfCZ831OnAvyLYsuMgd68A/oue3XD7eq9NYZu6TQQ6iA+au5/n7u3u/jV3nwqcDryHYMhSJGUUwESG1p3Ap8IrIAs3H73YzMrN7HAzO9fMioEWgiumzvB5bwOTzGyo/k+WA3Xu3mJm04CPxjy2FegimOvqdj9wXbgJbBk9c3J9ptmb2UwzO8bM8oEdBEOKnX2dK5IsCmAiQ8jdlxDMg90ObCfYuO+q8OFi4BbgHYJhyXEE2YcQbPIHsM3M/jYETfk0cLOZ7QRuAh6MaWMT8E3gmXCY81SCxJN5BBmKrxEE2Gv38Prjgd8SBK9VwELgl0PQbpGEaSGziIhkJV2BiYhIVlIAExkkM/v3cAFw9M+j6W6byHCiIUQREclKugITEZGspAAmIiJZSQFMRESykgKYiIhkJQUwERHJSv8fnO18aYblJD8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(estimator_range, accuracy_scores)\n",
    "plt.xlabel('n_estimators')\n",
    "plt.ylabel('Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning max_features\n",
    "\n",
    "The other important tuning parameter is **max_features**, which is the number of features that should be considered at each split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of values to try for max_features\n",
    "feature_range = range(1, len(feature_cols)+1)\n",
    "\n",
    "# list to store the average Accuracy for each value of max_features\n",
    "accuracy_scores = []\n",
    "\n",
    "# use 10-fold cross-validation with each value of max_features (WARNING: SLOW!)\n",
    "for feature in feature_range:\n",
    "    clf = RandomForestClassifier(n_estimators=200, max_features=feature, random_state=1, n_jobs=-1)\n",
    "    accuracy_scores.append(cross_val_score(clf, X, y, cv=5, scoring='accuracy').mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Accuracy')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbcAAAEGCAYAAADmAds7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xmc3VV9//HXe/ZkkplMSAJIAokyYBZwAUEEWWSLCEGr/RVUkBa1VKHWH7bFihRxKVVRW0HrD0UQF4pSNGjYZZMGSDCQDXACRDIQkkkymeyZ7fP7436T3LkzSW4yc9e8n4/HPOZ+z3e550wm9zPnfM/3fBQRmJmZlZOKQlfAzMxsqDm4mZlZ2XFwMzOzsuPgZmZmZcfBzczMyk5VoSuQSx0dHZ4KamZW5hobG5VZ5p6bmZmVnbwFN0nTJb0gaYmkKwbYf7CkhyTNkzRf0llp+46UNFvSIkkLJNUl5Ucl20sk/aekftHbzMz2PXkJbpIqgRuA9wJTgPMlTck47Erg9oh4G3Ae8L3k3Crgp8AlETEVOBnoSs75PvBJoDn5mp7bluRPS0tLoauQc25jeXAbS185ti9fPbdjgCUR8VJEdAK3AedmHBNAQ/K6EXgteX0GMD8ingWIiNUR0SPpQKAhImZHapmVnwDvz3VDzMys+OVrQslBwLK07Vbg2Ixjrgbuk3QZUA+clpQfBoSke4GxwG0R8fXkmq0Z1zxoZxUoxb9MSrHOe8ptLA9uY+krtfY1Nzfvcn++gttA98IyZzKeD9wcEddJOg64VdI0UnU8AXgHsAl4UNLTwLosrrnd7n4QxaalpaXk6ryn3Mby4DaWvnJsX76GJVuBCWnb49kx7LjNxcDtABExG6gDxiTnPhIRqyJiEzALeHtSPn431zQzs31Qvnpuc4BmSZOAV0lNGPlwxjGvAKcCN0uaTCq4tQH3Av8kaTjQCZwEfDsilktaL+mdwJPAhcB389IaKzkdnb08u7qLfCbBqK6Aht78vZ+Z7ZCX4BYR3ZIuJRWoKoGbImKRpGuAuRExE7gcuFHSZ0kNL16UTBRpl/QtUgEygFkR8bvk0n8H3AwMA+5Ovsy26+kNvvHser41fz2dBQg0IyqH8c3KTZx36PD8v7nZPixvK5RExCxSQ4rpZVelvV4MHL+Tc39K6nGAzPK5wLShramVi7bNPXzi0XYefm1rweqwoUdc8lg7T6zYyrXHjqKuyo9imuWDVyixsjR7xVZOnLmyoIEt3c1/2sTpv2vj5XXdha6K2T6hrNeWtH1PRHD9wg1c/fQ6ejLur+1XW8GUpvz8ygcwt62TLT07yhas6eKkmSu5/oQmZkwclpd6mO2rHNysbKzd2sun/tDOrFe29Nv3rv1r+NHJozlweGXe6rNgTRcfvvd1lm3ZMUCyriu48KE1fGpqPV86upHqCg9TmuWChyWtLDyzqpOTZq4cMLD9wxEjmDl9TF4DG8ARo6v5yVu3cO7Eun77vrdoI++btYrWDR6mNMsFBzcraRHBj57fwBm/a+PPG3r67BtVI247bTRXH91IVYF6SCOq4OaTR3PtsY1UZ/xve6qtkxNntvHgq/0DspkNjoOblawNXb188tF2Lp/d0W+a/9vHVPPIjHFMn1D4e1uSuGTKCO4+ayzj6/v2Htds7eVD963mq39cR0+v0w+aDRUHNytJz6/t4tS72vjlS5v77fvE5HruPmssh4wsrlvKR4+t4dEZYzn9oNo+5QF849n1/MV9q1m5uWfgk81sjzi4Wcm5bckm3nNXGy909L1fNaJK3HRSE9945yhqK4tzosboukr++/T9+OLbG8gcKX1k+VZO/M1K/vf14nh8wayUObhZydjSHXzm8XYueaydTd19h/CmjKrioRlj+Ys3Fv9KIBUSl79lJL8+cwzjhvX9L/j65l7OuWcV/7FgPb35XCvMrMw4uFlJeHldN6f/ro1b/rSp374PHzqcB84ZS3NjdQFqtvdOPLCWx2aM44QDavqU9wT869x1fPjBNazd6sUpzfaGg5sVvZlLN3PSzJUsWNPVp7yuEr57/Ci+9+4mhleV5q/y/sMr+fWZY7j8yBH99t2zbAsnzlzJvFWdBaiZWWkrzU8E2yd09gT/8tRaLnxoDeu6+g7RvamhkvvPHscFh9UXqHZDp6pCfPGoRm4/bT+aavveiHtlQw9n/q6NHz63gfAwpVnWHNysKLVu6Obsu1fxvUUb++07d2IdD50zjiNGl9Yw5O6cMaGOR2aM46gxfdvV2Qufe6KDjz/SzvouD1OaZcPBzYrOA61bOHFmG0+19R2Oq66Aa49t5OaTR9NQU56/ugePqOLus8byt5P790jveHkz77mrjcXtXQOcaWbpyvMTwkpST2/w1T+u4y/vX82ajIkU4+srufussVwyZQRScU7zHyo1leLf3zmKm08ezcjqvm1t6ejm1Lva+HlL/x6tme3g4GZFYeXmHj5w32q+8ex6Mu8snX5QLY/OGMvRY2sGPLdcvX/SMB46ZyxTMzIZbO4JPvWHtVz2h3Y2d/s+nNlAHNys4B5/PfXw8qPL+z68XCH44tsb+O/T92N0XX4XPS4WhzZW88DZ4/hoc//n925tSeWIe7HDiy+bZXJws4LpDfjO/PXMuGcVr2/uOww5blhFaor8W0ZSUebDkLszrEpcf0ITN5wwimEZK68sXNPFyXet5DdL+y9DZrYvc3Czgmjf2svli2sHTCp6wgE1PDZjHCceWDvwyfuojzTX88DZYzm0oe8w5fqu4GMPreHzT66lM/OHabaPytvKspKmA/8BVAI/jIhrM/YfDNwCjEqOuSIiZkmaCDwHvJAc+kREXJKc8zBwILDtz9YzImJlbltig/XHtk4+9vAalm3oP9R4+ZEj+PzbGgqWoqbYTR1dze/PGctnHl/LnRm9te8v3sgTKzvzfm+yY201javX5u39Dm2o4i/fOGyfHaq27OQluEmqBG4ATgdagTmSZkbE4rTDrgRuj4jvS5oCzAImJvtejIi37uTyH4mIuTmqug2hiOCHz2/kC0/1T1HTVCt+8O7RnDGhf2JP66uhpoKbTm7iXc/X8C9PdZD+6Nu8VV3MW5XvRwWqYXl+Z29+e/56fvKe0Rwzzr17G1i+hiWPAZZExEsR0QncBpybcUwADcnrRuC1PNXN8mB9Vy8XP9LOPz7RP7AdleRec2DLniQ+MXkE95w1lgkj9r0ezOube3nf3av40fNeucUGlq/gdhCwLG27NSlLdzXwUUmtpHptl6XtmyRpnqRHJL0747wfS3pG0hdV7g9AlahFa7o4ZWYb//Ny/0kPn0xyrx08orhyr5WKo8bW8OiMcZw5ft/rwXT1wuWzO7j08bVs8SMRlkH5+KtH0l8CZ0bEx5PtC4BjIuKytGP+b1Kf6yQdB/wImAZUAyMiYrWko4BfA1MjYp2kgyLiVUkjgTuAn0bET7Zds6OjY3vjWlpact5O6++3Kyq59sUatvb2/bujvjK4srmT08Y4OedQiIBFGypYtL6854gt3Sx+tbz/smuTR/Tw9Td3ckCdg9y+orm5efvrxsbGfh2bfP253ApMSNseT/9hx4uB6QARMVtSHTAmmSCyNSl/WtKLwGHA3Ih4NSlfL+nnpIY/f8IA0n8QpaClpaXk6pxuc3fwT0+s5daW/ilqpjZVccspo4mVS0u6jdnI57/jYcAH8vJOfeX7d/WcpZv59GPtbEjrrT23oZKLFtRz08mjOekNQ9+LLfX/j7tTju3L1595c4BmSZMk1QDnATMzjnkFOBVA0mSgDmiTNDaZkIKkNwLNwEuSqiSNScqrgbOBhXlpje3Six2p3GsDBbaPNg/ngbPHcWiJ5V6z4nHuxGE8cE7/RyJWb+3lA/et4rsL1vs+nOUnuEVEN3ApcC+paf23R8QiSddImpEcdjnwCUnPAr8ALorUb+iJwPyk/FfAJRGxBqgF7pU0H3gGeBW4MR/tsZ37zdLNnHzXShZm5F4bViluOGEU15/QxLAq3xq1wXnzqGoePGcs782YhNQb8MW56/ibh9vZ4AwK+7S83cWPiFmkJoqkl12V9noxcPwA591B6n5aZvlG4Kihr6ntjc6e4ItzOvjBc/2nhB/akBqGnFpmKWqssBprKvjZqaO57tn1fG1e3zVJ71y6mRfWdnHre/bjTY2erLQvKu+7z5YXyzZ0c9bdbQMGtg9MHMbvzxnrwGY5USHxj29t4L9P24/Gmr4jAovXdnPKb1dyzzIvTbYvcnCzQbm/dQsnzlzJ3La+w5DVFfCNdzZy08lNZZt7zYrHGRPqePiccUzJyKCwrjM474E1XDtvHb2+D7dP8aeO7ZXu3uDLT3fwl/evpn1r3w+NCSMqueessXxicvnnXrPiMamhivvfN5YPThrWb9+1z6zn/AfXsHar78PtKxzcbI+t2NTD++9dxXXzN/Tbd+b4Wh6dMY6j9rHca1Yc6qsr+OFJTXz1mEYyEihw77ItnPrblc5kvo9wcLM98tjyrZw4cyV/eL2zT3mF4F+PauAXp+1HU61/raxwJPHpqSO488wxjKnr+7v44roeTv9tG3e+3P8xFSsv/hSyrPRG8K356zn33lWsyMi9dsCwCmZOH8Nnj3TuNSseJx5Yy8PnjOXtY/pOZtrYHfz1w+1cNaeD7l7fhytXDm62W2u29HDeA6u55ul1ZH4WnHhgLY+eO44TDtj31ja04jd+RBWz3juWCwbIZP6fCzfwwftWs3qLl4ArRw5utktPt3Vy4sw27mvd2m/f594ykjvP2I9xw/a9VemtdNRVie+e0MR33jWK6oxPvEeWb+WkmW08s6pz4JOtZDm42YAigh8s3sD0WW20buz7l+3o2gp+dfp+XPn2BiqdVNRKxEWH1zPrvWM5cHjfj73WjT2cOauNn7XkNyed5ZaDm/WzrrOXv364nX9+sm8iTIB3jK3m0RljOW28c69Z6XnHuBoemTGOd+3fdzbv1h749B/W8rnZa+ns8X24cuDgZn0sXNPFKXet5NdL+6/q8HdT6vnde8cy3rnXrISNG1bJb6aP4ZIp9f32/fD5jZxzzyqWb/J9uFLn4Gbb3fqnjZz225W8uK7vf+yGanHLKaP5t2NHUZP58JBZCaquENceO4ofnNjEsIzf6SdXdnLyzJXMXtH/PrOVDgc3Y1N3L596rJ3LHl9L5sSxaaOreXjGOM6d2H/VB7NS91dvGs697xvDISP6TopasbmXc+5exY3PbXD6nBLl8aUy0tkTrOvqZV1n0NHZy7rOXjq2ve4K1qWVrUvKOjp7eW1jD21b+i9LdOFhw/n3Y0c5RY2VtSP3q+HhGeP4+CNrePDVHb217oB/fKKDP67q4sLRMHxjD509wZaeYGvytaUHOnv7lm3tgS09sf3YHftJO2/b/rTzu4OtvanjevIcULu766j64/K8vV9dpZj3oQNy+h4ObkXolQ3dPN1RQcsrm3cEorRgtC4pSw9aHZ29/Xpde2t4lbjuuFGcf2j/Z4PMylFTbQW3n7YfX5u3rt+ycr9YsolfMByeer1AtcuHCujM37qbw/PwB7ODW5H5zOPt3PKnTaQSka/J+/sf1pjKvTa5ySlqbN9SWSG+eFQjbx1Tw6cea2d9l4cjS5nvuRWRZ1d3JoEt/2oq4KPNw/n9OWMd2Gyfds4hw3jw7LE0O8lpSfO/XhGZ2za4VRIqBI01oqG6goaaChpqRGNNBQ3VoqGmIvV6e1lF6tikbP9hlc67ZpY4bFQ1D549liue7ODuZZvp7elleE0lNRWirlLUVoraSqitTG3XbPteQdp+pe3vW77zY6GmUlTleY3Wl19+mUmTJuXt/fLRPAe3IrJgdd9UHFOaqpjaVL09KG0LWtuDUrVorK1IykV9lZw/zWyINNRU8L13NwFNtLS00NzcXOgq5czG2uAN9eW1jJ6DWxFZsKZvcLv6qEbOmOCVQMzM9lTexqEkTZf0gqQlkq4YYP/Bkh6SNE/SfElnJeUTJW2W9Ezy9V9p5xwlaUFyzf9UCXdbenqDxe3dfcqO2M/3vszM9kZegpukSuAG4L3AFOB8SVMyDrsSuD0i3gacB3wvbd+LEfHW5OuStPLvA58EmpOv6blqQ669uK6bzWlr2jVVBwcM8z0wM7O9ka9Pz2OAJRHxUkR0ArcB52YcE0BD8roReG1XF5R0INAQEbMjtYTAT4D3D2218ydzSPKw+l7fPzMz20v5Cm4HAcvStluTsnRXAx+V1ArMAi5L2zcpGa58RNK7067ZuptrlozM4NZcn78HKs3Myk2+JpQM1AXJfELyfODmiLhO0nHArZKmAcuBgyNitaSjgF9LmprlNbdraWnZy6rnxxPLaoEds5UOq+8t+joPBbexPLiNpa/U2re72av5Cm6twIS07fH0H3a8mOSeWUTMllQHjImIlcDWpPxpSS8ChyXXHL+ba25X7NN4X356ObCjt3ZYfW/R13mwyn16NbiN5aLc21iO7cvXsOQcoFnSJEk1pCaMzMw45hXgVABJk0mtP9UmaWwyIQVJbyQ1ceSliFgOrJf0zmSW5IXAb/LTnKG1YlMPKzbvCGy1lXDIcC/9Y2a2t/LSc4uIbkmXAveSGnu7KSIWSboGmBsRM4HLgRslfZbU8OJFERGSTgSukdQN9ACXRMS2RRf/DrgZGAbcnXyVnIXtfe+3TR5VjRfiNzPbe3l7iDsiZpGaKJJedlXa68XA8QOcdwdwx06uOReYNrQ1zb/MlUmOGO3n28zMBsMPUhWBzJmSDm5mZoPj4FYEFmYGN69MYmY2KA5uBbapu5eWdX2X3ZrqlDNmZoPi4FZgz7V305s2MXLiSKeeMTMbLH+KFpjvt5mZDT0HtwLrd7/Nwc3MbNAc3Aoss+c2zcHNzGzQHNwKqDfCPTczsxxwcCugl9f1sLF7x2ySUTVifJmlejczKwQHtwLKXHbriNHVzuFmZjYEHNwKqN+yW35428xsSDi4FdCCNZ19tqf54W0zsyHh4FZA/Z5x26+mQDUxMysvDm4FsnpLD69t2pHDrboCDm/MW5IGM7Oy5uBWIJmPALx5VDU1lZ5MYmY2FBzcCmS+H942M8uZrIKbpL+XNCbXldmXeE1JM7PcybbndhqwVNJvJf2VpNpcVmpf4JVJzMxyJ6vgFhEzgEOAu4F/AF6X9ENJJ+aycuVqS3fwp7V9c7g5uJmZDZ2s77lFxOqIuCEijgNOAt4BPCRpqaQvSBqRs1qWmefXdpG26hbj6ysZVevbn2ZmQ2WPPlElnSrpx8DDwArgQuAC4G2kenW7One6pBckLZF0xQD7D5b0kKR5kuZLOmuA/RskfS6tbKmkBZKekTR3T9pSSL7fZmaWW1k9WCXpm8B5QAfwE+DKiHg1bf8TQPsuzq8EbgBOB1qBOZJmRsTitMOuBG6PiO9LmgLMAiam7f82AwfQUyJiVTbtKBb97rd52S0zsyGV7VPDdcAHImLOQDsjokvS0bs4/xhgSUS8BCDpNuBcID24BdCQvG4EXtu2Q9L7gZeAjVnWt6i552ZmlluKiN0fJB0EbIqI9rSyJmBYRLy28zO3H/shYHpEfDzZvgA4NiIuTTvmQOA+oAmoB06LiKcl1QMPkOr1fQ7YEBHfTM55mVSPMYAfRMT/S3/fjo6O7Y1raWnZbTvzIQJOeWIYG3t2PLB959GbGV+3+38HMzNLaW5u3v66sbGx3woY2fbcfg38DX2HHscDPwSOzeL8gZbeyPw0Px+4OSKuk3QccKukacCXgG9HxIYB0sEcHxGvSRoH3C/p+Yh4dKAKpP8gCmnp+m429qzYvt1QLU6a9iYqMtrW0tJSNHXOFbexPLiNpa8c25dtcDs8IhakF0TEAklvzvL8VmBC2vZ40oYdExcD05Nrz5ZUB4whFTw/JOnrwCigV9KWiLh+W68xIlZKupPU8OeAwa1YZA5JTh1d3S+wmZnZ4GQ7W3KlpEPTC5Lt1VmePwdoljRJUg2pySkzM455BTg1ufZkUvf52iLi3RExMSImAt8BvhYR10uqlzQyOb4eOANYmGV9CsYPb5uZ5V62PbebgDskfYHUxI43AV8mNSy5WxHRLelS4F6gErgpIhZJugaYGxEzgcuBGyV9ltSQ5UWx6xuC+wN3JkOVVcDPI+KeLNtTMJ5MYmaWe9kGt2uBLuCbpIYXl5EKbN/K9o0iYhap6f3pZVelvV4MHL+ba1yd9vol4C3Zvn+xcHAzM8u9rIJbRPQC30i+bC+t3drLsg0927crlUp1Y2ZmQyvr7JjJvbLDSU3y2D4DIiJ+n4N6laWF7X17bYc3VlFX5ckkZmZDLdsVSk4AfgnUknrQeh0wktTw5BtzVrsys2B1Rg43r0xiZpYT2c6W/Dbw9YgYDaxPvn8Z+F7OalaG+t1va3JwMzPLhWyD22HAf2SUXQt8dmirU976BTf33MzMciLb4NbBjnUflycLGzcBTnOTpc6e4IW1GcOSnilpZpYT2Qa3/wG2paD5EfAQ8DSp+3CWhT91dNPZu2P7DcMrGFNXWbgKmZmVsWwfBfiHtNfXSXqS1ISSe3NVsXKTOSTpXpuZWe7sNrgludj+BEyJiK0AEfGHXFes3CxY09ln2w9vm5nlzm6HJSOiB+ghtdaj7aWFa7r7bB8xuqZANTEzK3/ZPsT9HeB2SV8jtcL/9jUftyUgtZ2LCPfczMzyKNvgdn3y/fSM8iC1ELLtwqsbe2jfumMN6PoqManBPzYzs1zJdkJJtrMqbQD9crg1OYebmVkuOWjlQb8cbn5428wsp7JdW/Ix0u6zpYuIE4e0RmXIaW7MzPIr23tumUlJDwAuBn46tNUpTw5uZmb5le09t1syyyTdAfwYuGaoK1VO1nX28vL6HTncKgSTm7LONGRmZnthMPfcXgWOHKqKlKtFGTncDm2oYniVb3WameVStvfc/iajaDjwF8ATQ16jMtNvMomHJM3Mci7b8bELMrY3Av9LKs+b7YLvt5mZ5V9W42MRcUrG19kRcWVErM72jSRNl/SCpCWSrhhg/8GSHpI0T9J8SWcNsH+DpM9le81i4AWTzczyL6vgJulCSUdmlL1FUmaPbmfnVwI3AO8FpgDnJznh0l0J3B4RbwPOo3+W728Dd+/hNQuquzdY3O6em5lZvmU7s+HLwLKMsmXAV7I8/xhgSUS8FBGdwG3AuRnHBDsSojYCr23bIen9wEvAoj28ZkEtWdfN1h0TJRk3rIL9h3vZLTOzXMv2nlsDsC6jrAMYleX5B9E3OLYCx2YcczVwn6TLgHrgNABJ9cA/k1rX8nNpx2dzze1aWlqyrOrQuX9lJVC7ffuNtV17VI9C1Dnf3Mby4DaWvlJrX3Nz8y73ZxvcFgMfBG5PK/sA8FyW5w+0kGLmiifnAzcnyVCPA26VNA34EvDtiNigvusxZnPN7Xb3g8iFW9d2ABu2bx87vpHm5saszm1paSlInfPJbSwPbmPpK8f2ZRvc/hmYJemvgBeBQ4FTgbN2edYOrcCEtO3xpA07Ji4GpgNExGxJdcAYUr2xD0n6OqmeYq+kLcDTWVyzoPrNlPSakmZmeZHtbMk/AFOBOaSGDJ8CpkXE41m+zxygWdIkSTWkJozMzDjmFVIBE0mTSSVHbYuId0fExIiYSCqv3Nci4vosr1kwqRxunkxiZlYI2T7EXQu8HhHXppVVS6qNiK27Oz8iuiVdCtxLKv/bTRGxSNI1wNyImAlcDtwo6bOkhhcvioidDjPu7JrZtCcfVmzuZdWW3u3bwyrFoQ1edsvMLB+y/bS9H/gn+q5IchRwLXByNheIiFnArIyyq9JeLwaO3801rt7dNYtFZq9tclMVlRXO4WZmlg/ZPgpwBPBkRtlTwFuGtjrlw0OSZmaFk21w6wD2zyjbn9QyXDYArylpZlY42Qa3O4CfS5omabikI4BbgV/mrmqlzT03M7PCyTa4fYHUM21PkXpw64lk+8oc1aukbezqZUlH9/ZtAVMc3MzM8ibbRwG2RMSnST0GsD9wHLAVKK1H2vNkcXt3n6fJJ42sZGS1c7iZmeVL1p+4ksYCf09q6v084GjgMzmqV0nrd7/ND2+bmeXVLh8FkFQNzAAuAs4ElgC/ACYC/yciVua4fiWp//22mgLVxMxs37S7ntsK4AfAC8A7I2JKRHyZ1JCk7cSCNZ19tj2ZxMwsv3YX3OaTWs/xWOAdkppyX6XS1tMbLGrv7lPmBKVmZvm1y+AWEScDbwLuI5Vu5nVJd5GaWOJP7AG8tL6bTd07ppOMrq3gDcM9mcTMLJ92+6kbEX+OiC9HRDOphY2XA73As8lK/ZZmoIe3M1L1mJlZju1RlyIi/hARnwQOAC4jtSyXpfHD22ZmhbdX42XJc2+/iIj3DnWFSt2C1X2Dm++3mZnln28GDTH33MzMCs/BbQi1be7h9c07crjVVMBho5zDzcws3xzchlDmZJLJTdVUO4ebmVneObgNocwhSd9vMzMrDAe3IeT7bWZmxcHBbQg5QamZWXHIW3CTNF3SC5KWSLpigP0HS3pI0jxJ8yWdlZQfI+mZ5OtZSR9IO2eppAXJvrn5astANncHf+rwsltmZsUgL1P5JFUCNwCnA63AHEkzI2Jx2mFXArdHxPclTQFmkco+sBA4OiK6JR1IamWUuyJiWyQ5JSJW5aMdu/L82i560pK4HTKiksYad4zNzAohX5++xwBLIuKliOgEbgPOzTgmgIbkdSPwGkBEbEoLZHXJcUXHk0nMzIpHvoLbQcCytO3WpCzd1cBHJbWS6rVdtm2HpGMlLQIWAJekBbsA7pP0tKRP5qry2fBkEjOz4pGvJ4wHetgrswd2PnBzRFwn6TjgVknTIqI3Ip4EpkqaDNwi6e6I2AIcHxGvSRoH3C/p+Yh4dKAKtLS0DGV7+pnzai1QuX17zNZVtLSsGNQ1c13nYuA2lge3sfSVWvuam5t3uT9fwa0VmJC2PZ5k2DHNxcB0gIiYLakOGANsz/YdEc9J2ghMA+ZGxLahy5WS7iQ1/DlgcNvdD2IweiN48cnlpMfrM6YdzMEj9v7H29LSktM6FwO3sTy4jaWvHNuXr2HJOUCzpEmSaoDzgJkZx7xCKqUOSQ+tDmhLzqlKyg+s9BCqAAAObklEQVQBDgeWSqqXNDIprwfOIDX5JO/+vL6H9V07AltjjZhQX7mLM8zMLJfy0nNLZjpeCtxLauzupohYJOkaUj2wmcDlwI2SPkuqC3RRRISkE4ArJHWRyiP3qYhYJemNwJ1JrrQq4OcRcU8+2pNpoMkkzuFmZlY4eVvVNyJmkZookl52VdrrxcDxA5x3K3DrAOUvAW8Z+pruOU8mMTMrLn4Qawg4uJmZFRcHtyGQueyWn3EzMyssB7dBat/aS+vGnu3bVYI3j3JwMzMrJAe3Qcockjx8VBW1lZ5MYmZWSA5ug+T7bWZmxcfBbZAWrO7ss+37bWZmhefgNkj9e241BaqJmZlt4+A2CJ09/XO4HTE6b48OmpnZTji4DcLza7vo6t2xPb6+ktF1XnbLzKzQHNwGwTnczMyKk4PbIDi4mZkVJwe3QchcmcSPAZiZFQcHt70UEf16bkc6uJmZFQUHt720bGMPHZ07criNrBaHjPRkEjOzYuDgtpcWrO7ba5vaVE2Fc7iZmRUFB7e9tLDd99vMzIqVg9teyuy5HbGfg5uZWbFwcNtLXjDZzKx4ObjthY7OXv68YUcOtwrncDMzKyoObnthUUav7bDGKoZVeTKJmVmxyFtwkzRd0guSlki6YoD9B0t6SNI8SfMlnZWUHyPpmeTrWUkfyPaaueIhSTOz4paXJewlVQI3AKcDrcAcSTMjYnHaYVcCt0fE9yVNAWYBE4GFwNER0S3pQOBZSXcBkcU1c8LBzcysuOWr53YMsCQiXoqITuA24NyMYwJoSF43Aq8BRMSmiNiWV6YuOS7ba+aE15Q0Mytu+QpuBwHL0rZbk7J0VwMfldRKqtd22bYdko6VtAhYAFySBLtsrjnkunqD59c6uJmZFbN8ZdYcaLZFZGyfD9wcEddJOg64VdK0iOiNiCeBqZImA7dIujvLa27X0tKyt3XvY8lGsbVn2PbtMTW9dLS+RMeQXL2voapzMXMby4PbWPpKrX3Nzc273J+v4NYKTEjbHk8y7JjmYmA6QETMllQHjAFWbjsgIp6TtBGYluU1t9vdDyJbf3xxE9C+ffttY4fR3Dxh5yfspZaWliGrc7FyG8uD21j6yrF9+RqWnAM0S5okqQY4D5iZccwrwKkASQ+tDmhLzqlKyg8BDgeWZnnNIeeVSczMil9eem7JTMdLgXuBSuCmiFgk6RpgbkTMBC4HbpT0WVLDixdFREg6AbhCUhfQC3wqIlYBDHTNXLel32SSJgc3M7Nik69hSSJiFqmJIullV6W9XgwcP8B5twK3ZnvNXIqI/glK3XMzMys6XqFkDyzf1Mvqrb3bt4dXiTeOzNvfB2ZmliUHtz2QOSQ5tamKygovu2VmVmwc3PaAH942MysNDm57oN/9ttE1BaqJmZntioPbHliwprPPtteUNDMrTg5uWVrf1ctL63bkcBMwpcmTSczMipGDW5YWr+nqs7bXmxqqqK/2j8/MrBj50zlLC9ud5sbMrFQ4uGXJy26ZmZUOB7csOUGpmVnpcHDLQk9vsLi9u0+Zn3EzMyteDm5ZeHFdN5t7dkwnGVNXwQHD/KMzMytW/oTOwkBDkpKX3TIzK1YOblnw/TYzs9Li4JYFBzczs9Li4JaFzDUlPZnEzKy4ObjtxopNPazYvCOHW20lNDd62S0zs2Lm4LYbmSuTTGmqpso53MzMipqD2270W5nEQ5JmZkXPwW03+iUobXJwMzMrdnkLbpKmS3pB0hJJVwyw/2BJD0maJ2m+pLOS8tMlPS1pQfL9PWnnPJxc85nka9xQ17tfglKvKWlmVvTyMjNCUiVwA3A60ArMkTQzIhanHXYlcHtEfF/SFGAWMBFYBZwTEa9JmgbcCxyUdt5HImJuLuq9qbuXlnV9l92a6p6bmVnRy1fP7RhgSUS8FBGdwG3AuRnHBNCQvG4EXgOIiHkR8VpSvgiok1SbhzrzXHs3vWlJ3CaNrKShxiO5ZmbFLl9z2g8ClqVttwLHZhxzNXCfpMuAeuC0Aa7zQWBeRGxNK/uxpB7gDuArEREDnEdLS8seV/qB1yuBHXF0Ys3WvbrO3srnexWK21ge3MbSV2rta25u3uX+fAW3gebOZwah84GbI+I6SccBt0qaFhG9AJKmAv8OnJF2zkci4lVJI0kFtwuAnwxUgd39IAbStmotsHH79nEHN9Hc3LDzE4ZQS0vLXtW5lLiN5cFtLH3l2L58BbdWYELa9niSYcc0FwPTASJitqQ6YAywUtJ44E7gwoh4cdsJEfFq8n29pJ+TGv4cMLjtjc+/bSTvO7iOBWu6WLimi3cdkJfRUDMzG6R8Bbc5QLOkScCrwHnAhzOOeQU4FbhZ0mSgDmiTNAr4HfD5iHh828GSqoBREbFKUjVwNvDAUFZ6v7pKTjmoklMOqhvKy5qZWY7lZXZERHQDl5Ka6fgcqVmRiyRdI2lGctjlwCckPQv8ArgouX92KXAo8MWMKf+1wL2S5gPPkAqaN+ajPWZmVtzytkhiRMwiNb0/veyqtNeLgeMHOO8rwFd2ctmjhrKOZmZWHjyv3czMyo6Dm5mZlR0HNzMzKzsObmZmVnYc3MzMrOxoJ6tVlYWOjo7ybZyZmQHQ2NjYbxUs99zMzKzsOLiZmVnZKethSTMz2ze552ZmZmXHwa2ISJog6SFJz0laJOkzha5TrkiqlDRP0m8LXZdckDRK0q8kPZ/8ex5X6DoNNUmfTX5PF0r6RZLJo6RJuknSSkkL08pGS7pfUkvyvamQdRysnbTxG8nv6nxJdyYL1pc0B7fi0g1cHhGTgXcCn5Y0pcB1ypXPkFpEu1z9B3BPRLwZeAtl1lZJBwF/DxwdEdOASlLZPkrdzSSpt9JcATwYEc3Ag8l2KbuZ/m28H5gWEUcCfwI+n+9KDTUHtyISEcsj4o/J6/WkPhAPKmythl6Sn+99wA8LXZdckNQAnAj8CCAiOiNibWFrlRNVwLAk/dRw+udoLDkR8SiwJqP4XOCW5PUtwPvzWqkhNlAbI+K+JHsLwBOkcm6WNAe3IiVpIvA24MnC1iQnvgP8E9Bb6IrkyBuBNuDHydDrDyXVF7pSQylJFPxNUnkYlwMdEXFfYWuVM/tHxHJI/QEKjCtwfXLtb4C7C12JwXJwK0KSRgB3AP8QEesKXZ+hJOlsYGVEPF3ouuRQFfB24PsR8TZgI6U/lNVHct/pXGAS8AagXtJHC1srGyxJXyB1e+Rnha7LYDm4FZkkq/gdwM8i4n8KXZ8cOB6YIWkpcBvwHkk/LWyVhlwr0BoR23rdvyIV7MrJacDLEdEWEV3A/wDvKnCdcmWFpAMBku8rC1yfnJD0MeBs4CNRBs+IObgVEUkidZ/muYj4VqHrkwsR8fmIGB8RE0lNQPh9RJTVX/wR8TqwTNLhSdGpwOICVikXXgHeKWl48nt7KmU2aSbNTOBjyeuPAb8pYF1yQtJ04J+BGRGxqdD1GQoObsXleOACUr2ZZ5KvswpdKdsrlwE/kzQfeCvwtQLXZ0glvdJfAX8EFpD6LPl/Ba3UEJD0C2A2cLikVkkXA9cCp0tqAU5PtkvWTtp4PTASuD/53PmvglZyCHiFEjMzKzvuuZmZWdlxcDMzs7Lj4GZmZmXHwc3MzMqOg5uZmZUdBzczMys7Dm5mJUIpP5bULumpQtfHrJg5uJmVjhNIPUQ8PiKOGcyFJF0k6Q9DUy2z4uPgZlY6DgGWRsTGQlckSXNjVrQc3Mz2gKSlkv4xyVi8UdKPJO0v6W5J6yU9sC1Ts6RfSnpdUoekRyVNTcprkiWOLku2KyU9LumqXbzvxaTy3x0naYOkLyXlZyfXWivpfyUdmXbOFZJeTOq1WNIHkvLJwH+lXWttUv6wpI+nnd+ndycpJH06WYaqJSl7c5Kdeo2kFyT9n7Tjz0red72kVyV9btD/AGZZcnAz23MfJDU8eBhwDqncV/8CjCH1f+rvk+PuBppJ5f/6I0kakYjoBD4KXJMEmitIZbL+6s7eMCJ+BFwCzI6IERHxr5LeDtwE/C2wH/ADYKak2uS0F4F3A43Al4CfSjowIp7LuNaoPWj7+4FjgSlJjrr7gZ8nbTwf+N62IE5qEfC/jYiRwDTg93vwPmaD4uBmtue+GxErkoSdjwFPRsS8iNgK3EkqySwRcVNErE/KrwbeIqkx2bcQ+Epy/OeACyKiZw/r8QngBxHxZET0RMQtwFbgncl7/DIiXouI3oj4b1K9rUHdqwP+LSLWRMRmUulRlkbEjyOiO8kifwfwoeTYLlJBsCEi2rdlmTfLBwc3sz23Iu315gG2RyRDjdcmw4LrgKXJ/jFpx94CTARmRUTLXtTjEODyZEhybTK8OIFU8lAkXZg2ZLmWVO9pzC6ul41lGe9/bMb7fwQ4INn/QeAs4M+SHpF03CDf2yxrvilslhsfJpWp+jRSga0RaAeUdsz3gN8CZ0o6ISL2dPbiMuCrEdFvOFPSIcCNpPKszY6IHknPpL3/QOlANgLD07YPGOCY9POWAY9ExOkDVS4i5gDnJgl4LwVuJxV8zXLOPTez3BhJaohwNamA0Sefm6QLgKOAi0jdo7tF0og9fI8bgUskHZs8A1cv6X2SRgL1pAJRW/J+f02q57bNCmC8pJq0smeAv0gSkB4KXLyb9/8tcJikCyRVJ1/vkDQ5mTTzEUmNSabudcCeDrua7TUHN7Pc+AnwZ+BVUlm4n9i2Q9LBwHeACyNiQ0T8HJgLfHtP3iAi5pK673Y9qV7hElLBkohYDFxHKinlCuAI4PG0038PLAJel7QqKfs20JkcfwvJBJhdvP964AxSGdVfA14H/h3YNqHlAmBpMix7CalJNGZ54WSlZmZWdtxzMzOzsuPgZlZEkofBNwzw9S+FrptZKfGwpJmZlR333MzMrOw4uJmZWdlxcDMzs7Lj4GZmZmXHwc3MzMrO/weI2KjVcnjODgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(feature_range, accuracy_scores)\n",
    "plt.xlabel('max_features')\n",
    "plt.ylabel('Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting a Random Forest with the best parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features=6, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=200, n_jobs=-1,\n",
       "            oob_score=False, random_state=1, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# max_features=6 is best and n_estimators=200 is sufficiently large\n",
    "clf = RandomForestClassifier(n_estimators=200, max_features=6, random_state=1, n_jobs=-1)\n",
    "clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Division</td>\n",
       "      <td>0.006081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>League</td>\n",
       "      <td>0.008834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>NewLeague</td>\n",
       "      <td>0.009709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Errors</td>\n",
       "      <td>0.032638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Assists</td>\n",
       "      <td>0.040503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HmRun</td>\n",
       "      <td>0.047118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>PutOuts</td>\n",
       "      <td>0.051506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AtBat</td>\n",
       "      <td>0.078822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Runs</td>\n",
       "      <td>0.080185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Walks</td>\n",
       "      <td>0.082160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RBI</td>\n",
       "      <td>0.091048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Hits</td>\n",
       "      <td>0.132156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Years</td>\n",
       "      <td>0.339239</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      feature  importance\n",
       "8    Division    0.006081\n",
       "7      League    0.008834\n",
       "12  NewLeague    0.009709\n",
       "11     Errors    0.032638\n",
       "10    Assists    0.040503\n",
       "2       HmRun    0.047118\n",
       "9     PutOuts    0.051506\n",
       "0       AtBat    0.078822\n",
       "3        Runs    0.080185\n",
       "5       Walks    0.082160\n",
       "4         RBI    0.091048\n",
       "1        Hits    0.132156\n",
       "6       Years    0.339239"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute feature importances\n",
    "pd.DataFrame({'feature':feature_cols, 'importance':clf.feature_importances_}).sort_values('importance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing Random Forests with decision trees\n",
    "\n",
    "**Advantages of Random Forests:**\n",
    "\n",
    "- Performance is competitive with the best supervised learning methods\n",
    "- Provides a more reliable estimate of feature importance\n",
    "- Allows you to estimate out-of-sample error without using train/test split or cross-validation\n",
    "\n",
    "**Disadvantages of Random Forests:**\n",
    "\n",
    "- Less interpretable\n",
    "- Slower to train\n",
    "- Slower to predict"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}