{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Copyright (c) 2015, 2016 [Sebastian Raschka](sebastianraschka.com)\n",
    "<br>\n",
    "[Li-Yi Wei](http://liyiwei.org/), 2016\n",
    "\n",
    "https://github.com/1iyiwei/pyml\n",
    "\n",
    "[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Python Machine Learning - Code Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Chapter 4 - Building Good Training Sets – Data Pre-Processing\n",
    "\n",
    "Machine learns from data (training set)\n",
    "* garbage in, garbage out\n",
    "\n",
    "Data is very important\n",
    "* quality, form, etc.\n",
    "\n",
    "This part is about how to pre-process data for better machine learning\n",
    "* the first stage of the pipeline\n",
    "\n",
    "<img src=\"./images/01_09.png\" width=90%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last updated: 2016-08-26 \n",
      "\n",
      "CPython 3.5.2\n",
      "IPython 4.2.0\n",
      "\n",
      "numpy 1.11.1\n",
      "pandas 0.18.1\n",
      "matplotlib 1.5.1\n",
      "sklearn 0.17.1\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a '' -u -d -v -p numpy,pandas,matplotlib,sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "*The use of `watermark` is optional. You can install this IPython extension via \"`pip install watermark`\". For more information, please see: https://github.com/rasbt/watermark.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Overview\n",
    "\n",
    "- [Dealing with missing data](#Dealing-with-missing-data)\n",
    "  - [Eliminating samples or features with missing values](#Eliminating-samples-or-features-with-missing-values)\n",
    "  - [Imputing missing values](#Imputing-missing-values)\n",
    "  - [Understanding the scikit-learn estimator API](#Understanding-the-scikit-learn-estimator-API)\n",
    "- [Handling categorical data](#Handling-categorical-data)\n",
    "  - [Mapping ordinal features](#Mapping-ordinal-features)\n",
    "  - [Encoding class labels](#Encoding-class-labels)\n",
    "  - [Performing one-hot encoding on nominal features](#Performing-one-hot-encoding-on-nominal-features)\n",
    "- [Partitioning a dataset in training and test sets](#Partitioning-a-dataset-in-training-and-test-sets)\n",
    "- [Bringing features onto the same scale](#Bringing-features-onto-the-same-scale)\n",
    "- [Selecting meaningful features](#Selecting-meaningful-features)\n",
    "  - [Sparse solutions with L1 regularization](#Sparse-solutions-with-L1-regularization)\n",
    "  - [Sequential feature selection algorithms](#Sequential-feature-selection-algorithms)\n",
    "- [Assessing feature importance with random forests](#Assessing-feature-importance-with-random-forests)\n",
    "- [Summary](#Summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Dealing with missing data\n",
    "\n",
    "The training data might be incomplete due to various reasons\n",
    "* data collection error\n",
    "* measurements not applicable\n",
    "* etc.\n",
    "\n",
    "Most machine learning algorithms/implementations cannot robustly deal with missing data\n",
    "\n",
    "Thus we need to deal with missing data before training models\n",
    "\n",
    "We use pandas (Python data analysis) library for dealing with missing data in the examples below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A     B     C    D\n",
       "0   1.0   2.0   3.0  4.0\n",
       "1   5.0   6.0   NaN  8.0\n",
       "2  10.0  11.0  12.0  NaN"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from io import StringIO\n",
    "\n",
    "csv_data = '''A,B,C,D\n",
    "1.0,2.0,3.0,4.0\n",
    "5.0,6.0,,8.0\n",
    "10.0,11.0,12.0,'''\n",
    "\n",
    "# If you are using Python 2.7, you need\n",
    "# to convert the string to unicode:\n",
    "# csv_data = unicode(csv_data)\n",
    "\n",
    "df = pd.read_csv(StringIO(csv_data))\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       A      B      C      D\n",
       "0  False  False  False  False\n",
       "1  False  False   True  False\n",
       "2  False  False  False   True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    0\n",
       "B    0\n",
       "C    1\n",
       "D    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Eliminating samples or features with missing values\n",
    "\n",
    "One simple strategy is to simply eliminate samples (table rows) or features (table columns) with missing values, based on various criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B    C    D\n",
       "0  1.0  2.0  3.0  4.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the default is to drop samples/rows\n",
    "df.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A     B\n",
       "0   1.0   2.0\n",
       "1   5.0   6.0\n",
       "2  10.0  11.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# but we can also elect to drop features/columns\n",
    "df.dropna(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A     B     C    D\n",
       "0   1.0   2.0   3.0  4.0\n",
       "1   5.0   6.0   NaN  8.0\n",
       "2  10.0  11.0  12.0  NaN"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# only drop rows where all columns are NaN\n",
    "df.dropna(how='all')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B    C    D\n",
       "0  1.0  2.0  3.0  4.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop rows that have not at least 4 non-NaN values\n",
    "df.dropna(thresh=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A     B     C    D\n",
       "0   1.0   2.0   3.0  4.0\n",
       "2  10.0  11.0  12.0  NaN"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# only drop rows where NaN appear in specific columns (here: 'C')\n",
    "df.dropna(subset=['C'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Dropping data might not be desirable, as the resulting data set might become too small."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Imputing missing values\n",
    "\n",
    "Interpolating missing values from existing ones can preserve the original data better.\n",
    "\n",
    "Impute: the process of replacing missing data with substituted values\n",
    "<a href=\"https://en.wikipedia.org/wiki/Imputation_(statistics)\">in statistics</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1. ,   2. ,   3. ,   4. ],\n",
       "       [  5. ,   6. ,   7.5,   8. ],\n",
       "       [ 10. ,  11. ,  12. ,   6. ]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import Imputer\n",
    "\n",
    "# options from the imputer library includes mean, median, most_frequent\n",
    "\n",
    "imr = Imputer(missing_values='NaN', strategy='mean', axis=0)\n",
    "imr = imr.fit(df)\n",
    "imputed_data = imr.transform(df.values)\n",
    "imputed_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.,   2.,   3.,   4.],\n",
       "       [  5.,   6.,  nan,   8.],\n",
       "       [ 10.,  11.,  12.,  nan]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can do better than this, by selecting only the most similar rows for interpolation, instead of all rows. This is how recommendation system could work, e.g. predict your potential rating of a movie or book you have not seen based on item ratings from you and other users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Understanding the scikit-learn estimator API\n",
    "\n",
    "Transformer class for data transformation\n",
    "* imputer\n",
    "\n",
    "Key methods\n",
    "* fit() for fitting from (training) ata\n",
    "* transform() for transforming future data based on the fitted data\n",
    "\n",
    "Good API designs are consistent. For example, the fit() method has similar meanings for different classes, such as transformer and estimator. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Transformer\n",
    "<img src='./images/04_04.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Estimator\n",
    "<img src='./images/04_05.png' width=80%> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Handling different types of data\n",
    "\n",
    "There are different types of feature data: numerical and categorical.\n",
    "Numerical features are numbers and often \"continuous\" like real numbers.\n",
    "Categorical features are \"discrete\", and can be either nominal or ordinal.\n",
    "* Ordinal values are discrete but carry some numerical meanings such as ordering and thus can be sorted. \n",
    "* Nominal values have no numerical meanings.\n",
    "\n",
    "In the example below, color is nominal (no numerical meaning), size is ordinal (can be sorted in some way), and price is numerical.\n",
    "\n",
    "A given dataset can contain features of different types. It is important to handle them carefully. For example, do not treat nominal values as numbers without proper mapping.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>size</th>\n",
       "      <th>price</th>\n",
       "      <th>classlabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>green</td>\n",
       "      <td>M</td>\n",
       "      <td>10.1</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>red</td>\n",
       "      <td>L</td>\n",
       "      <td>13.5</td>\n",
       "      <td>class2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>blue</td>\n",
       "      <td>XL</td>\n",
       "      <td>15.3</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color size  price classlabel\n",
       "0  green    M   10.1     class1\n",
       "1    red    L   13.5     class2\n",
       "2   blue   XL   15.3     class1"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame([['green', 'M', 10.1, 'class1'],\n",
    "                   ['red', 'L', 13.5, 'class2'],\n",
    "                   ['blue', 'XL', 15.3, 'class1']])\n",
    "\n",
    "df.columns = ['color', 'size', 'price', 'classlabel']\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Data conversion\n",
    "\n",
    "For some estimators such as decision trees that handle one feature at a time, it is OK to keep the features as they are.\n",
    "However for other estimators that need to handle multiple features together, we need to convert them into compatible forms before proceeding: (1) convert categorical values into numerical values, and (2) scale/normalize numerical values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Mapping ordinal features\n",
    "\n",
    "Ordinal features can be converted into numbers, but the conversion often depends on semantics and thus needs to be specified manually (by a human) instead of automatically (by a machine).\n",
    "In the example below, we can map sizes into numbers. Intuitively, larger sizes should map to larger values. Exactly which values to map to is often a judgment call.\n",
    "\n",
    "Below, we use Python dictionary to define a mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>size</th>\n",
       "      <th>price</th>\n",
       "      <th>classlabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>green</td>\n",
       "      <td>1</td>\n",
       "      <td>10.1</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>red</td>\n",
       "      <td>2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>class2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>blue</td>\n",
       "      <td>3</td>\n",
       "      <td>15.3</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color  size  price classlabel\n",
       "0  green     1   10.1     class1\n",
       "1    red     2   13.5     class2\n",
       "2   blue     3   15.3     class1"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "size_mapping = {'XL': 3,\n",
    "                'L': 2,\n",
    "                'M': 1}\n",
    "\n",
    "df['size'] = df['size'].map(size_mapping)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     M\n",
       "1     L\n",
       "2    XL\n",
       "Name: size, dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inv_size_mapping = {v: k for k, v in size_mapping.items()}\n",
    "df['size'].map(inv_size_mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Encoding class labels\n",
    "\n",
    "Class labels often need to be represented as integers for machine learning libraries\n",
    "* not ordinal, so any integer mapping will do\n",
    "* but a good idea and convention is to use consecutive small values like 0, 1, ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'class1': 0, 'class2': 1}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "class_mapping = {label: idx for idx, label in enumerate(np.unique(df['classlabel']))}\n",
    "class_mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>size</th>\n",
       "      <th>price</th>\n",
       "      <th>classlabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>green</td>\n",
       "      <td>1</td>\n",
       "      <td>10.1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>red</td>\n",
       "      <td>2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>blue</td>\n",
       "      <td>3</td>\n",
       "      <td>15.3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color  size  price  classlabel\n",
       "0  green     1   10.1           0\n",
       "1    red     2   13.5           1\n",
       "2   blue     3   15.3           0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# forward map\n",
    "df['classlabel'] = df['classlabel'].map(class_mapping)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>size</th>\n",
       "      <th>price</th>\n",
       "      <th>classlabel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>green</td>\n",
       "      <td>1</td>\n",
       "      <td>10.1</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>red</td>\n",
       "      <td>2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>class2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>blue</td>\n",
       "      <td>3</td>\n",
       "      <td>15.3</td>\n",
       "      <td>class1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color  size  price classlabel\n",
       "0  green     1   10.1     class1\n",
       "1    red     2   13.5     class2\n",
       "2   blue     3   15.3     class1"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inverse map\n",
    "inv_class_mapping = {v: k for k, v in class_mapping.items()}\n",
    "df['classlabel'] = df['classlabel'].map(inv_class_mapping)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0], dtype=int64)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "class_le = LabelEncoder()\n",
    "y = class_le.fit_transform(df['classlabel'].values)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['class1', 'class2', 'class1'], dtype=object)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class_le.inverse_transform(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Performing one-hot encoding on nominal features\n",
    "\n",
    "A common mistake is to map nominal features into numerical values, e.g. for colors\n",
    "* blue $\\rightarrow$ 0\n",
    "* green $\\rightarrow$ 1\n",
    "* red $\\rightarrow$ 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 10.1],\n",
       "       [2, 2, 13.5],\n",
       "       [0, 3, 15.3]], dtype=object)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df[['color', 'size', 'price']].values\n",
    "\n",
    "color_le = LabelEncoder()\n",
    "X[:, 0] = color_le.fit_transform(X[:, 0])\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For categorical features, it is important to keep the mapped values \"equal distance\"\n",
    "* unless you have good reasons otherwise\n",
    "\n",
    "For example, for colors red, green, blue, we want to convert them to values so that each color has equal distance from one another.\n",
    "\n",
    "This cannot be done in 1D but doable in 2D (how? think about it)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "One hot encoding is a straightforward way to make this just work, by mapping n-value nominal feature into n-dimensional binary vector. \n",
    "* blue  $\\rightarrow$ (1, 0, 0)\n",
    "* green $\\rightarrow$ (0, 1, 0)\n",
    "* red   $\\rightarrow$ (0, 0, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0. ,   1. ,   0. ,   1. ,  10.1],\n",
       "       [  0. ,   0. ,   1. ,   2. ,  13.5],\n",
       "       [  1. ,   0. ,   0. ,   3. ,  15.3]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "ohe = OneHotEncoder(categorical_features=[0])\n",
    "ohe.fit_transform(X).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>size</th>\n",
       "      <th>color_blue</th>\n",
       "      <th>color_green</th>\n",
       "      <th>color_red</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10.1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13.5</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15.3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   price  size  color_blue  color_green  color_red\n",
       "0   10.1     1         0.0          1.0        0.0\n",
       "1   13.5     2         0.0          0.0        1.0\n",
       "2   15.3     3         1.0          0.0        0.0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# automatic conversion via the get_dummies method in pd\n",
    "pd.get_dummies(df[['price', 'color', 'size']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Partitioning a dataset in training and test sets\n",
    "\n",
    "Training set to train the models\n",
    "\n",
    "Test set to evaluate the trained models\n",
    "\n",
    "Separate the two to avoid over-fitting\n",
    "* well trained models should generalize to unseen, test data\n",
    "\n",
    "Validation set for tuning hyper-parameters\n",
    "* parameters are trained by algorithms\n",
    "* hyper-parameters are selected by humans\n",
    "* will talk about this later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class labels [1 2 3]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Class label</th>\n",
       "      <th>Alcohol</th>\n",
       "      <th>Malic acid</th>\n",
       "      <th>Ash</th>\n",
       "      <th>Alcalinity of ash</th>\n",
       "      <th>Magnesium</th>\n",
       "      <th>Total phenols</th>\n",
       "      <th>Flavanoids</th>\n",
       "      <th>Nonflavanoid phenols</th>\n",
       "      <th>Proanthocyanins</th>\n",
       "      <th>Color intensity</th>\n",
       "      <th>Hue</th>\n",
       "      <th>OD280/OD315 of diluted wines</th>\n",
       "      <th>Proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>13.20</td>\n",
       "      <td>1.78</td>\n",
       "      <td>2.14</td>\n",
       "      <td>11.2</td>\n",
       "      <td>100</td>\n",
       "      <td>2.65</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.28</td>\n",
       "      <td>4.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>13.16</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.67</td>\n",
       "      <td>18.6</td>\n",
       "      <td>101</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.81</td>\n",
       "      <td>5.68</td>\n",
       "      <td>1.03</td>\n",
       "      <td>3.17</td>\n",
       "      <td>1185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>14.37</td>\n",
       "      <td>1.95</td>\n",
       "      <td>2.50</td>\n",
       "      <td>16.8</td>\n",
       "      <td>113</td>\n",
       "      <td>3.85</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.24</td>\n",
       "      <td>2.18</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.45</td>\n",
       "      <td>1480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>13.24</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.87</td>\n",
       "      <td>21.0</td>\n",
       "      <td>118</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.82</td>\n",
       "      <td>4.32</td>\n",
       "      <td>1.04</td>\n",
       "      <td>2.93</td>\n",
       "      <td>735</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Class label  Alcohol  Malic acid   Ash  Alcalinity of ash  Magnesium  \\\n",
       "0            1    14.23        1.71  2.43               15.6        127   \n",
       "1            1    13.20        1.78  2.14               11.2        100   \n",
       "2            1    13.16        2.36  2.67               18.6        101   \n",
       "3            1    14.37        1.95  2.50               16.8        113   \n",
       "4            1    13.24        2.59  2.87               21.0        118   \n",
       "\n",
       "   Total phenols  Flavanoids  Nonflavanoid phenols  Proanthocyanins  \\\n",
       "0           2.80        3.06                  0.28             2.29   \n",
       "1           2.65        2.76                  0.26             1.28   \n",
       "2           2.80        3.24                  0.30             2.81   \n",
       "3           3.85        3.49                  0.24             2.18   \n",
       "4           2.80        2.69                  0.39             1.82   \n",
       "\n",
       "   Color intensity   Hue  OD280/OD315 of diluted wines  Proline  \n",
       "0             5.64  1.04                          3.92     1065  \n",
       "1             4.38  1.05                          3.40     1050  \n",
       "2             5.68  1.03                          3.17     1185  \n",
       "3             7.80  0.86                          3.45     1480  \n",
       "4             4.32  1.04                          2.93      735  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine_data_remote = 'https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data'\n",
    "wine_data_local = '../datasets/wine/wine.data'\n",
    "\n",
    "df_wine = pd.read_csv(wine_data_local,\n",
    "                      header=None)\n",
    "\n",
    "df_wine.columns = ['Class label', 'Alcohol', 'Malic acid', 'Ash',\n",
    "                   'Alcalinity of ash', 'Magnesium', 'Total phenols',\n",
    "                   'Flavanoids', 'Nonflavanoid phenols', 'Proanthocyanins',\n",
    "                   'Color intensity', 'Hue', 'OD280/OD315 of diluted wines',\n",
    "                   'Proline']\n",
    "\n",
    "print('Class labels', np.unique(df_wine['Class label']))\n",
    "df_wine.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<hr>\n",
    "\n",
    "### Note:\n",
    "\n",
    "\n",
    "If the link to the Wine dataset provided above does not work for you, you can find a local copy in this repository at [./../datasets/wine/wine.data](./../datasets/wine.data).\n",
    "\n",
    "Or you could fetch it via\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Class label</th>\n",
       "      <th>Alcohol</th>\n",
       "      <th>Malic acid</th>\n",
       "      <th>Ash</th>\n",
       "      <th>Alcalinity of ash</th>\n",
       "      <th>Magnesium</th>\n",
       "      <th>Total phenols</th>\n",
       "      <th>Flavanoids</th>\n",
       "      <th>Nonflavanoid phenols</th>\n",
       "      <th>Proanthocyanins</th>\n",
       "      <th>Color intensity</th>\n",
       "      <th>Hue</th>\n",
       "      <th>OD280/OD315 of diluted wines</th>\n",
       "      <th>Proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>13.20</td>\n",
       "      <td>1.78</td>\n",
       "      <td>2.14</td>\n",
       "      <td>11.2</td>\n",
       "      <td>100</td>\n",
       "      <td>2.65</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.28</td>\n",
       "      <td>4.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>13.16</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.67</td>\n",
       "      <td>18.6</td>\n",
       "      <td>101</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.81</td>\n",
       "      <td>5.68</td>\n",
       "      <td>1.03</td>\n",
       "      <td>3.17</td>\n",
       "      <td>1185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>14.37</td>\n",
       "      <td>1.95</td>\n",
       "      <td>2.50</td>\n",
       "      <td>16.8</td>\n",
       "      <td>113</td>\n",
       "      <td>3.85</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.24</td>\n",
       "      <td>2.18</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.45</td>\n",
       "      <td>1480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>13.24</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.87</td>\n",
       "      <td>21.0</td>\n",
       "      <td>118</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.82</td>\n",
       "      <td>4.32</td>\n",
       "      <td>1.04</td>\n",
       "      <td>2.93</td>\n",
       "      <td>735</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Class label  Alcohol  Malic acid   Ash  Alcalinity of ash  Magnesium  \\\n",
       "0            1    14.23        1.71  2.43               15.6        127   \n",
       "1            1    13.20        1.78  2.14               11.2        100   \n",
       "2            1    13.16        2.36  2.67               18.6        101   \n",
       "3            1    14.37        1.95  2.50               16.8        113   \n",
       "4            1    13.24        2.59  2.87               21.0        118   \n",
       "\n",
       "   Total phenols  Flavanoids  Nonflavanoid phenols  Proanthocyanins  \\\n",
       "0           2.80        3.06                  0.28             2.29   \n",
       "1           2.65        2.76                  0.26             1.28   \n",
       "2           2.80        3.24                  0.30             2.81   \n",
       "3           3.85        3.49                  0.24             2.18   \n",
       "4           2.80        2.69                  0.39             1.82   \n",
       "\n",
       "   Color intensity   Hue  OD280/OD315 of diluted wines  Proline  \n",
       "0             5.64  1.04                          3.92     1065  \n",
       "1             4.38  1.05                          3.40     1050  \n",
       "2             5.68  1.03                          3.17     1185  \n",
       "3             7.80  0.86                          3.45     1480  \n",
       "4             4.32  1.04                          2.93      735  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_wine = pd.read_csv('https://raw.githubusercontent.com/1iyiwei/pyml/master/code/datasets/wine/wine.data', header=None)\n",
    "\n",
    "df_wine.columns = ['Class label', 'Alcohol', 'Malic acid', 'Ash', \n",
    "'Alcalinity of ash', 'Magnesium', 'Total phenols', \n",
    "'Flavanoids', 'Nonflavanoid phenols', 'Proanthocyanins', \n",
    "'Color intensity', 'Hue', 'OD280/OD315 of diluted wines', 'Proline']\n",
    "df_wine.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## How to allocate training and test proportions?\n",
    "\n",
    "As much training data as possible for accurate model\n",
    "\n",
    "As much test data as possible for evaluation\n",
    "\n",
    "Usual rules is 60:40, 70:30, 80:20\n",
    "\n",
    "Larger datasets can have more portions for training\n",
    "* e.g. 90:10\n",
    "\n",
    "Other partitions possible\n",
    "* talk about later in model evaluation and parameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(178, 13)\n",
      "[1 2 3]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "X, y = df_wine.iloc[:, 1:].values, df_wine.iloc[:, 0].values\n",
    "\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    train_test_split(X, y, test_size=0.3, random_state=0)\n",
    "    \n",
    "print(X.shape)\n",
    "\n",
    "import numpy as np\n",
    "print(np.unique(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bringing features onto the same scale\n",
    "\n",
    "Most machine learning algorithms behave better when features are on similar scales.\n",
    "\n",
    "Exceptions\n",
    "* decision trees\n",
    "* random forests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Example\n",
    "\n",
    "Two features, in scale [0 1] and [0 100000]\n",
    "\n",
    "Think about what happens when we apply \n",
    "* perceptron\n",
    "* KNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Two common approaches \n",
    "\n",
    "### Normalization\n",
    "min-max scaler:\n",
    "$$\\frac{x-x_{min}}{x_{max}-x_{min}}$$\n",
    "\n",
    "### Standardization\n",
    "standard scaler:\n",
    "$$\\frac{x-x_\\mu}{x_\\sigma}$$\n",
    "* $x_\\mu$: mean of x values\n",
    "* $x_\\sigma$: standard deviation of x values\n",
    "\n",
    "Standardization more common as normalization sensitive to outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "mms = MinMaxScaler()\n",
    "X_train_norm = mms.fit_transform(X_train)\n",
    "X_test_norm = mms.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "stdsc = StandardScaler()\n",
    "X_train_std = stdsc.fit_transform(X_train)\n",
    "X_test_std = stdsc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Toy case: standardization versus normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>standardized</th>\n",
       "      <th>normalized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>-1.46385</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>-0.87831</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>-0.29277</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0.29277</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0.87831</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>1.46385</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   input  standardized  normalized\n",
       "0      0      -1.46385         0.0\n",
       "1      1      -0.87831         0.2\n",
       "2      2      -0.29277         0.4\n",
       "3      3       0.29277         0.6\n",
       "4      4       0.87831         0.8\n",
       "5      5       1.46385         1.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex = pd.DataFrame([0, 1, 2, 3, 4, 5])\n",
    "\n",
    "# standardize\n",
    "ex[1] = (ex[0] - ex[0].mean()) / ex[0].std(ddof=0)\n",
    "\n",
    "# Please note that pandas uses ddof=1 (sample standard deviation) \n",
    "# by default, whereas NumPy's std method and the StandardScaler\n",
    "# uses ddof=0 (population standard deviation)\n",
    "\n",
    "# normalize\n",
    "ex[2] = (ex[0] - ex[0].min()) / (ex[0].max() - ex[0].min())\n",
    "ex.columns = ['input', 'standardized', 'normalized']\n",
    "ex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Selecting meaningful features\n",
    "\n",
    "Overfitting is a common problem for machine learning. \n",
    "* model fits training data too closely and fails to generalize to real data\n",
    "* model too complex for the given training data\n",
    "\n",
    "<img src=\"./images/03_06.png\" width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## ways to address overfitting\n",
    "\n",
    "* Collect more training data (to make overfitting less likely)\n",
    "* Reduce the model complexity explicitly, such as the number of parameters\n",
    "* Reduce the model complexity implicitly via regularization\n",
    "* Reduce data dimensionality, which forced model reduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Regularization\n",
    "\n",
    "For weight vector $\\mathbf{w}$ of some model (e.g. perceptron or SVM)\n",
    "\n",
    "$L_2$:\n",
    "$\n",
    "\\|\\mathbf{w}\\|_2^2 = \\sum_{k} w_k^2\n",
    "$\n",
    "\n",
    "$L_1$:\n",
    "$\n",
    "\\|\\mathbf{w}\\|_1 = \\sum_{k} \\left|w_k \\right|\n",
    "$\n",
    "\n",
    "$L_1$ tends to produce sparser solutions than $L_2$\n",
    "* more zero weights\n",
    "* more like feature selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src='./images/04_12.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src='./images/04_13.png' width=80%> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Regularization in scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 0.983870967742\n",
      "Test accuracy: 0.981481481481\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# l1 regularization\n",
    "lr = LogisticRegression(penalty='l1', C=0.1)\n",
    "lr.fit(X_train_std, y_train)\n",
    "\n",
    "# compare training and test accuracy to see if there is overfitting\n",
    "print('Training accuracy:', lr.score(X_train_std, y_train))\n",
    "print('Test accuracy:', lr.score(X_test_std, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.38382172, -0.15807173, -0.70041684])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3 sets of parameters due to one-versus-rest with 3 classes\n",
    "lr.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.28019913,  0.        ,  0.        , -0.02814783,  0.        ,\n",
       "         0.        ,  0.7100226 ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  1.23627197],\n",
       "       [-0.64401442, -0.06882245, -0.05718991,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        , -0.92682635,\n",
       "         0.06017202,  0.        , -0.37102201],\n",
       "       [ 0.        ,  0.06153118,  0.        ,  0.        ,  0.        ,\n",
       "         0.        , -0.6360026 ,  0.        ,  0.        ,  0.49806763,\n",
       "        -0.35826867, -0.57130862,  0.        ]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 13 coefficients for 13 wine features; notice many of them are 0\n",
    "lr.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 0.983870967742\n",
      "Test accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# l2 regularization\n",
    "lr = LogisticRegression(penalty='l2', C=0.1)\n",
    "lr.fit(X_train_std, y_train)\n",
    "\n",
    "# compare training and test accuracy to see if there is overfitting\n",
    "print('Training accuracy:', lr.score(X_train_std, y_train))\n",
    "print('Test accuracy:', lr.score(X_test_std, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.58228361,  0.04305595,  0.27096654, -0.53333363,  0.00321707,\n",
       "         0.29820868,  0.48418851, -0.14789735, -0.00451997,  0.15005795,\n",
       "         0.08295104,  0.38799131,  0.80127898],\n",
       "       [-0.71490217, -0.35035394, -0.44630613,  0.32199115, -0.10948893,\n",
       "        -0.03572165,  0.07174958,  0.04406273,  0.20581481, -0.71624265,\n",
       "         0.39941835,  0.17538899, -0.72445229],\n",
       "       [ 0.18373457,  0.32514838,  0.16359432,  0.15802432,  0.09025052,\n",
       "        -0.20530058, -0.53304855,  0.1117135 , -0.21005439,  0.62841547,\n",
       "        -0.4911972 , -0.55819761, -0.04081495]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# notice the disappearance of 0 coefficients due to L2\n",
    "lr.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Plot regularization\n",
    "\n",
    "$C$ is inverse to the regularization strength"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAncAAAETCAYAAABURyCzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvXl8lNW9+P9+JpNJMlkmC9kTskMgCfuOYAAFtS61gAKK\nuNDFq1ZRr6312wJie9VWW2trf/e2oETFpVWKWFDc2BRZBCEJELaQkH3fM/v5/fEkkwxJSAiBTMJ5\n+zqvZzvzPJ+JyfCez9kUIQQSiUQikUgkksGBpr8DkEgkEolEIpH0HVLuJBKJRCKRSAYRUu4kEolE\nIpFIBhFS7iQSiUQikUgGEVLuJBKJRCKRSAYRUu4kEolEIpFIBhH9LneKoqxVFKVUUZQj7c4FKIqy\nTVGUHEVRPlUUxdCfMUokEolEIpEMFPpd7oDXgXnnnfsl8LkQYjjwJfD0FY9KIpFIJBKJZACiuMIk\nxoqixACbhRCjWo6PA9cKIUoVRQkDtgshkvs1SIlEIpFIJJIBgCtk7jojRAhRCiCEKAFC+jkeiUQi\nkUgkkgGBtr8D6CGdphcVRen/tKNEIpEMQIQQSk/rKoriodfr1wghftrc3OwL9Pi1Eomkb1EURfj4\n+Byvr6+fK4Qo6KyOq2buShVFCQVoaZYt66qiEKLbsnLlykuq19n59ucu9f69jcMVYnCVOFwtBleJ\nwxVicJU4XCEGV4njYjEYDJ/PnDnzoaysLD+LxaL0JC5ZZJHl8hSj0ag89dRTSb6+vtu6+pt1FblT\ncP4m+BFwb8v+MmDTpdw8PT39kup1dr6n97yccbhCDK4Sx0CNwVXicIUYXCUOV4jBleIAqK+vn/bh\nhx/q4+Pj0WoHSoOPRDI40el0PPXUU9qGhoauxyL0t4ECG4AiwATkA/cBAcDnQA6wDfDv4rXCFVi5\ncmV/h+ASMQjhGnG4QgxCuEYcrhCDEK4RhyvEIIRrxNHy2Xkxn9P9E6hEIumSC/0d9/tXMCHEki4u\nXXdFA7kELuUb8WCKAVwjDleIAVwjDleIAVwjDleIAVwnDolEMnhxialQeouiKGIgxy+RSCT9gaIo\niIsbUDFgP2vvu+8+oqOjefbZZ3t9j9WrV3Pq1CnefPPNfnm+RNIZF/o7dpU+dxKJRCKRXBLp6ekE\nBgZisVj6/N6KIgcISwYOUu4kEolEMuDJy8tj9+7daDQaPvroo/4ORyLpV6TcSSQSiWTAk5GRwdSp\nU7n33nt54403uqy3adMmxo4di8FgICkpiW3b1NkkiouLue222wgKCmLYsGH84x//cHqdyWRi2bJl\n+Pn5kZaWxsGDBx3Xjh8/zqxZswgICCAtLY3NmzdflvcokfQUKXcSiUQiGfBkZGRw9913s2TJEj79\n9FPKy8s71Nm3bx/Lli3jpZdeora2lp07dxIbGwvAnXfeydChQykpKeGf//wnv/rVr9i+fbvjtZs3\nb2bJkiXU1tZyyy238NBDDwFgtVq55ZZbuOGGGygvL+fPf/4zd911FydPnrwSb1si6RQpdxKJRCK5\nZBSlb0pv2L17N/n5+dxxxx2MGzeOxMRENmzY0KHeunXreOCBB5g9ezYA4eHhDBs2jIKCAvbs2cML\nL7yAu7s7o0ePZvny5WRkZDhee8011zBv3jwURWHp0qUcOXIEgD179tDY2MgvfvELtFots2bN4uab\nb+add97p3ZuRSPoAKXcSiUQiuWSE6JvSGzIyMpg7dy4BAQEALF68mPXr13eod+7cORISEjqcLyoq\nIjAwEL1e7zgXExNDYWGh4zgsLMyxr9frMRqN2O12iouLiY6Odrrf+a+VSK40/T7PnUQikUgkvcVo\nNPL+++9jt9sJDw8H1P5xtbW1juxaK9HR0Zw+fbrDPSIiIqiqqqKxsRFvb28A8vPziYyM7Pb5ERER\nnDt3zulcfn4+w4cP7+1bkkguGZm5k0gkEsmAZePGjWi1Wo4dO8bhw4c5fPgwx48fZ8aMGU7NqgAP\nPPAAr7/+Ol999RVCCIqKisjJySEqKopp06bx9NNPYzKZOHLkCGvXrmXp0qVdPrd13r/Jkyej1+t5\n8cUXsVqtbN++nY8//pjFixdf1vctkVwIKXcSiUQiGbBkZGRw//33ExkZSUhIiKM89NBDbNiwAZvN\n5qg7ceJEXn/9dR577DEMBgPp6enk5+cDsGHDBnJzc4mIiGD+/PmsWbOGWbNmdfnc1nnv3N3d2bx5\nM1u2bGHIkCE8/PDDvPnmmyQlJTnVk0iuJHKFColEIrnKuJpWqJBIBityhQqJRCKRSCSSqwQ5oEIi\nkUiuAGVlZZw4cQKtVovBYMDPzw+DwYC3t7dsupNIJH2KbJaVSCSSy0RlZSVZWVlkZ2djMplITk4G\noK6ujtraWmprazGZTPj6+mIwGBzS1yp+rVtPT88+FUDZLCuRDHwu9Hcs5U4ikUj6kOrqarKzs8nO\nzqahoYGRI0eSmppKVFRUp4JmtVqdZK91v66uzrFvt9u7FL/WfZ1O1+MYpdxJJAMfKXcSiURyGamt\nrXUIXW1tLSNGjCAlJYWhQ4ei0Vx61+bWedvOl8D2Iti+ufd88TMYDPj6+qLVqj1xpNxJJAOfC/0d\nyz53EolE0gvq6+s5evQo2dnZVFRUkJyczJw5c4iNje0ToWuPh4eHY4qPzhBC0Nzc3CH7V1ZW5jiu\nr6/Hy8sLg8HQp7FJJBLXQ2buJBKJpIc0NjY6hK60tJThw4eTkpJCfHw8bm5u/R3eBbHb7TQ0NFBX\nV0d0dLTM3EkkAxzZLCuRSCS9pLm5mWPHjpGdnU1hYSFJSUmkpKSQmJjoaOYcaMhmWYlk4CPlTiKR\nSC4Co9FITk4OWVlZjsXmU1JSSEpKwt3dvb/Du2Sk3HUkLy+PuLg4rFYrGo2Gm266icWLF19wCbK+\nxNfXl8zMTGJjYztcW79+Pf/4xz/YtWvXFYlFMjCQfe4kEomkG8xmMzk5OWRnZ3P27FliY2MZNWoU\nCxcuvKiRqJIrT2xsLCUlJRQVFREYGOg4P3bsWA4fPszZs2cZOnRot/dpP5p5y5YtlyXWrqivr7/g\ndTkXouRikHInkUiuWiwWCydPniQ7O5vTp08zdOhQUlJS+OEPf4inp2d/hyfpIYqiEBcXxzvvvMND\nDz0EQFZWFs3NzVKKJFclcvkxiURyVWG1Wjl+/DgffPABL730Et999x0JCQk8+uijLFmyhNGjR0ux\nG4AsXbqU9evXO47Xr1/PsmXLnOps2bKFcePGYTAYiImJYfXq1V3eb9asWaxbt85x/Pe//52RI0fi\n5+dHamoq33//faeve+yxxxg6dCgGg4GJEyeye/duxzW73c7vfvc7EhMTHdcLCwsB0Gg0nDlzBoCq\nqipuvfVWDAYDU6ZM4fTp0xf/A5Fc1bh05k5RlLNALWAHLEKISf0bkUQiGYhYrVZyc3PJzs4mJyeH\nsLAwUlJSuOGGG/D29u7v8CR9wJQpU3jzzTfJyckhKSmJ9957j6+//ppnnnnGUcfHx4c333yTlJQU\nsrKyuP766xk7diy33nrrBe/9z3/+k2effZZNmzYxbtw4zpw502Xfy0mTJrFq1Sr8/Px45ZVXWLhw\nIXl5eeh0Ol566SXee+89PvnkExITE8nMzESv1wPOza7/9V//hV6vp7S0lNOnTzNv3jzi4+P74Kck\nuVpwablDlbp0IUR1fwcikUgGDkIIKioqOH36NGfOnCEvL4/Q0FBSUlKYM2cOvr6+/R3ioENZ3TfN\nn2Jl7wdutGbvrr32WkaMGEFERITT9ZkzZzr2U1NTWbRoETt27OhW7tauXctTTz3FuHHjAC4oWkuW\nLHHsr1ixgjVr1pCTk0NaWhpr167lD3/4A4mJiQCkpaU56rYOWLHb7Xz44YdkZ2fj6elJSkoKy5Yt\nk4MpJBeFq8udgmw6lkgkPaCpqYnc3FxOnz7N6dOnURSFhIQExowZw+23346Xl1d/hziouRQp6yvu\nvvtuZs6cSW5uLvfcc0+H63v37uXpp58mKysLs9mM2Wxm4cKF3d63dcR0T/jDH/7AunXrKC4uBtSB\nEhUVFY77dJeBKy8vx2azERUV5TgXExMj5U5yUbi63AngM0VRbMD/CSH+3t8BSSQS18Bms1FQUOCQ\nucrKSmJiYoiPj2f69OkEBgbKzvRXGUOHDiUuLo6tW7c69Zdr5a677uLnP/85n376Ke7u7qxYsYLK\nyspu7xsdHd2jfm+7d+/m97//PV999RUjR44EIDAw0JGVa71P67XOCA4ORqvVcu7cOYYNGwZAfn5+\nt8+WSNrj6nI3XQhRrChKMKrkHRNC7G5fYdWqVY799PR00tPTr2yEEonkiiCEoKqqytHUevbsWQID\nA0lISOD6668nOjra5VeJ6C+2b9/O9u3b+zuMK8K6deuorq7Gy8sLm83mdK2hoYGAgADc3d3Zt28f\nGzZsYN68eY7rXc3lt3z5cp544gmmT5/OuHHjOH36NO7u7h2mV6mvr8fd3Z2goCDMZjPPP/+80xQn\ny5cv59e//jUjRoxw9LmLiooiICDAUUej0fCjH/2IVatWsXbtWnJzc1m/fj1xcXF98eORXCW4tNwJ\nIYpbtuWKomwEJgFdyp1EIhlcGI1Gp6ZWm81GQkICqamp3HLLLXIwRA85/4vvhUaJDkTaZ2jj4uKc\nRKj9tddee43HH3+chx9+mGuvvZY777yTmpqaTuu231+wYAFVVVUsWbKEoqIiYmNjefPNNzvI3bx5\n85g3bx7Dhg3Dx8eHFStWEB0d7bj++OOPYzabmTt3LpWVlSQnJ7Nx40YCAgKcnvfqq69y3333ER4e\nTnJyMvfffz9fffXVJf6UJFcTLrtChaIoekAjhGhQFMUb2AasFkJsa1dn0M+aLpFcTdjtdgoLCx0y\nV1ZWxtChQ4mPjycxMZEhQ4bIptY+QK5QIZEMfAbqChWhwEZFUQRqnG+3FzuJRDI4qK6udjS15ubm\nYjAYSEhIYNasWQwdOnTArt96pbEBlpZi7mZfIpEMblw2c9cT5LdJiWTgYTKZOHv2rCM7ZzKZSEhI\nICEhgfj4eHx8fK5YLAKw0iY/raWn57qtKwQeZgveRjO+JhPeJgtCCCyKgrldsSgKJo2CWQGTojgV\nY7ttc8t+s0bBqIBRUWhqOW9WFBSnAhpFwV1RcAd0gHtL2SczdxLJgOdCmTspdxKJ5LJit9spLi52\nyFxJSQlRUVHEx8eTkJBAaGhonze1CuAMsK9dOU1HCbMBbrRJT2vR9vCcl91OqMlCuNFEmMlMiNFM\nsMlEsNFMkNGMv8lMs9aNGk8P6jx11HvoUBQFdyHQti92gZsQuAnQCoFGtB6r+xoh0NgFGtR9xS5Q\nBGiEACFQztsiBNjbfTZqFFDaijJ9rJQ7iWSAI+VOIpFcUYQQFBQUkJmZydGjR9Hr9Y7sXExMTJez\n+/eWcpxFbh+gRx2B1VqG45y9ahW2CxqOzQZGM5jM6tZoatk3qccWK+jcwVMHnh7goVP3PVqOPXWg\n6cepOh2y135foHjopNxJJAMcKXcSieSKUFFRwZEjR8jKysLNzY20tDTS0tKcpnq4VBqBgziLXDUw\nkTaRmwhEdHWDVoQAq81Z1s6XOJutTdRaxe18iRuAAzzkgAqJZOAj5U4ikVw26uvryczMJCsri/r6\nelJTUxk1ahRhYWGX3NxqBbJxFrmTQBrOWbkkLrCUjdUKdY3Q2HyexJnU612Jm6cHuGsHpLx1h5Q7\niWTgI+VOIpH0KUajkWPHjpGZmUlxcTHJycmkpaURGxuLppfNkALIw1nkDgJRqAI3uWU7CvDo8iZC\nlbbaBlXo6hqg2QS+evDxbpG2dhKndRuU8tYdUu4kkoGPlDuJRHLJWK1WTp48SWZmJmfOnCEuLo60\ntDSGDRvWq+lKKoH9OMucG20SNwmYAPhf6CZ2O9Q3qiJX26DKnKKAwQf8fNStt1f/9ntzQaTcSSQD\nHyl3EomkVwghOHv2LJmZmRw/fpzQ0FDS0tIYMWIEXl5ePb5PM3AIZ5ErQ5W39s2rkXQzwMFkbsvI\n1TaoTa3enqrItRZPnaO61Wrl8OHD7Nq1i/z8fIKCghgyZIhj21qCgoL6fJCHKyPlDnbs2MHdd9/N\nuXPn+jsUiaRXDNRJjCUSST8ghKC0tNQxMEKv15OWlsbPfvYz/Pz8enSPBtR1Ar8AvgKOAiNQBW4u\n8P9QR69ecCVYIVR5a83I1TWoAyBaJS4+Eny9od16skajkX07d7Jr1y527tzJt99+S1RUFDNmzCAh\nIYHq6mry8vKoqKigsrKSiooKx763t7eT7HW23/74ahPCgUB6ejpHjhyhtLS0R/9v5GonksGKlDuJ\nRAJATU0NmZmZZGZmYrFYSE1N5e677yYkJKTb15qAb4EvUYXue2A8MBt4GXX0ard5vtaBD61ZufpG\ntV+cnw/4+0FMOHh5OvWRq62t5ZtvvmHXrl3s2rWLgwcPkpKSwowZM3jwwQd5++23GTJkSLfx2+12\n6urqHLJ3vvy1CmH7a1VVVXh7e/dIBFv3g4KCADCbzVgsFiwWi9P++cc92e/NawYjeXl57N69G39/\nfz766CPmz5/f3yFJJP2GbJaVSK5impqayM7OJjMzk8rKSkaOHElaWhrR0dEXzGrYUAc7fIEqdHuA\nZGAOqtBNB7wv9GCngQ8tgx+MJnXQg8G7LTvn7vz9s7S0lN27d7OzJTt34sQJJk6cyIwZM5gxYwZT\np069Yitc2O12amtrO80CdiWJlZWVaDQa3N3dHUWn03W6391xb6+5u7uzYMGCQdcsu2bNGrZt28bk\nyZPJyclh8+bNAGzZsoX//u//5ty5cxgMBlasWMHjjz/uaJZdsWIFL7zwAlqtlt/+9rfce++9/ftG\nJJIeIptlJRKJA4vFwvHjx8nMzCQ/P5+kpCSuueYaEhIScHPrvKFUoE5J0pqZ24naP2428F/Ae8AF\nZ7Kz2aGh0XkUq0YDfi0iFx7cYeCDEIKzubmOJtZdu3ZRVlbGtGnTmDlzJn/5y18YP348Hh5djp29\nrGg0GgICAggICCApKalfYpC0kZGRwZNPPsnEiROZMmUK5eXlBAcHs3z5cv71r38xbdo0amtryc3N\ndbympKSE+vp6ioqK2LZtGwsWLOD222/HYDD04zuRSC4dmbmTSK4C7HY7Z86cITMzk5ycHKKiokhL\nSyM5OblTORJALm2ZuS9RM3GtmblZQNiFHmixqgJXU9+jgQ+tMR47dswhcjt37sRmszmycjNnziQ1\nNbVLAZX0nMsyoKKv+q/14jN99+7dzJkzh5KSEgICAhg5ciQ//elPefTRR4mNjeWZZ55h0aJF+Pr6\nOl6zY8cObrrpJurr6x3T94SGhrJ582YmTZrUN+9FIrmMyMydRHIVIoSgsLCQzMxMsrOz8ff3Jy0t\njeuvv77Tpsti2kTuC9R+dLOB64DfAXEXepjZArX1UNOgbo0mdbCDwRfiItUM3XlSZrFYOHTokEPm\ndu/eTUBAADNmzOC6667j2WefJSEhYdB0ercKK82iGYFA0+4/RVE6HA9I+vGLdkZGBnPnznWshLJ4\n8WLWr1/Po48+ygcffMCaNWv4xS9+wejRo/mf//kfpkyZAkBQUJDTvIx6vZ6GhoZ+eQ8SSV8i5U4i\nGSQ0NDRQWFjoKEVFRfj4+JCamsr9999PYGCgU/0qYAdt2bkSIB1V6J5AHd3aqWYIoU5J0pqVq61X\nM3V+PuDvC6Ex4KPvMLdcU1MTe/fudWTl9u7dS1xcHDNnzmTJkiX87W9/IyKi20XDXAa7sGMURppE\nE8325m63Nmx4KeqwEoHA3vKfEG37duwAXcqfgoJGubAYdlm33fFgwmg08v7772O32wkPDwfUASut\nA4TGjx/Pv//9b2w2G6+++ip33HEH+fn5/Ry1RHJ5kXInkQxALBYLxcXFFBQUUFRUREFBASaTicjI\nSCIiIpg8eTIRERFOGbpGYBdtmbkTqAMf5gBvAmPoYmoSIaDJ2CZytQ3qOYOPmpmLDFH7y7XLOLX2\nlztw4AD79+9n9+7dHD58mFGjRjFjxgwee+wxpk+f3qdrzl4qQgjMmHskak2iCZMw4aF44KV4odfo\nHVu9oidEG4Je0eOl8XJsdeh6lJU7X/YueCzsFxTFro4HExs3bkSr1XL48GGn6U/uuOMOXn/9dSZM\nmMDNN9+Mn58fvr6+sllfclUg5U4icXHsdjsVFRUUFhY6ZK6iooKQkBAiIyNJSkoiPT2doKAgJ3kw\nog58aM3MHaJtepI/oa4Eoev4OFXcGpqcZc5No4qcvy/ERICXh5PMFRYWOkTuwIEDHDhwAA8PDyZO\nnMiECRP47W9/y+TJk9Hr9Zft5+T8FlRRMwszFmHBLMwYhbGDoDXZm2gWzY6tBo2TqLVuDRoD4dpw\np/Neihcape+zYIqi4Nbyn3qizx8xqMjIyOD+++8nMjLS6fxDDz3Egw8+SFZWFg8//DB2u53hw4ez\nYcOGLu81YJvEJZLzkAMqJBIXo76+noKCgg7Nq5GRkY4SFhbWYcmvcuAb4OuW8j0wElXm5nCB6Ula\nl/CqbRkAUdcIHu6qzLVm59oNfigvL3cIXKvMWSwWJkyY4JC5CRMmXHQTqxACK1bMwtx1OU/YuioW\nLLjjjk7ROYqH4uGUSdMrztk2L40X7srVMSmxXKFCIhn4yOXHJBIXxWQydWhetVqtREVFERER4die\nn/ESQA5tIvc1ap+5KagSNx01M9fpjG82mypwNfVqZq6+CfSeLZm5lvVYW5q3ampq+O6775xkrqam\nhvHjxzuJXExMTOsHDSZholE00mRvotHe6GjC7FTChEW9hrrvhhs6RYe74o6H4uHYd0gaOidha1/3\n/HoyC9M1Uu4kkoGPlDuJxAWw2+2UlZU5DXqorq4mNDTUkZGLiorC39+/g5gYgQO0idw3qOI2vV1J\npYs+cxarcxNrY7M64MG/JSvn5wNaNxobGzl06JAjG7d//36KiooYO3YsEyZNYNz0cQwfM5ygyCCa\naHKSt0Z7o0Po3BQ3vBVvvDXe6BU9eo0eT8XzgiLWun85mjklbRibLFQX1xORGCTlTiIZ4Ei5k0iu\nMEII6urqnJpXS0pK8PPzc2peDQ0N7bSDdznOWbnDqKNX28tcZIdX0bYea32jWlpXfvDzaWti9fPG\naDZz+MhhDhw+QObJTM4Un6HeWk/S6CTiRsQRGheKb7Av6KFZNGMSJrwUL1XYNHoneWt/Tq/RXzVN\nm66EsdFMVXEDVUX1XW6rixswG60Ehvuy7uzPpdxJJAMcKXcSyWXEZrNRUVFBSUkJpaWljgJ0aF71\n9PTs8PrOmlhLcW5inUQnTaxCgNHcJnL1jYiGRiwebjQbtDT7amjWK9RpTZwty6OgsoCq5iqMihGt\njxa/UD8URUFr1uLr7kuwdzC+Wt9O5c1T8ZRZtX6gucHcrbBVFTdgNdsIjPAlMNyHwAgfAsJ9CYrw\nISDch6AIX8fW298TRVFks6xEMgiQcieR9BGNjY0dJK6yshJ/f39CQ0MJDQ0lLCyM0NBQfH19O+33\n1VkTqy/OWbkU1CZWm7CpozxFM83mBpqbamg21WE019Nsa6JZa6VZZ6fZ3UqzxkwzRoRdYG+yY6w1\nUl1STdGpItzMbgzxHsLQ4KEkxyQzKmkUgfpAdEqn42UlvaUBKAIKW7ZF5x0XAxZospuostZTZWug\nyt6ytTVQbaunsmVbZW/Ajp1AjS8BGh+C3NRtoMaXQLeWbcuxt+KBcv6wWtHFPqBUSLmTSAY6Uu4k\nkoukNRvXXuJKSkqw2WwdJC44ONhpfq3zcTSxCsE+YeaMaCZVNDPG3kyyaCZaNKOztwicaKbZ3ra1\nCgsednf0Vnc8rVq8FE88FE/MTXYqyms5l1tE3ok8TmWd4tj3x/DSeJEYm8iIESNITk5m5MiRjBs3\nzmnZJUkvMKGK2fmydv6xFUSEoCHESIlvNaW6GkpEDaWmGkpqqymrqKGytB4hIDDUl8AQHwJDfQgM\n8VW3oT5t58N80ft6tH1BaP8R3tV+D+spIVLuJJKBzoCVO0VRbkCdkksDrBVCvHDedfmBI7lkGhsb\nO0hcZWUlBoOBsLAwQkJCHCLn5+fnlI2zCRuNoolyeyMl9ibKRRPV9kbqRRNV9mYaRTPu9mYMohkP\nYcQNLd4aL7wVLzw1nup8aXjiZdXiadSgbxJ4NtjxbBK4a7yoaTRyoqiQvUez2HVgP8eOH6O0tJSk\npCSSk5OdyrBhwzpdVkxyAWxAGRcWtiKgFnUx3UggAozBFso8ayjV1FBirqG0oYaSimpKC2ooza0B\nIDTOn9A4f8LiA9Rty/GQKD+8fPt3NK9slpVIBj4DUu4URdGgTqI/B/XjdT+wSAhxvF0d+YEj6TE2\nm43KykqHwLXKnMVicZK4kJAQfIL9qNCaKbI3UWFvolo00mBvpFk0YbM3gr0Jd9GEVphpUPQ0aPSY\nNXqE4o1Go8dD0ROl0ZOmeJGseOKtUSe9dUMDzSZHHzl7XQM0NNFgtZBXWc6h0yf56sBe/rPjKwR0\nELjk5GRiY2PlLPudIVCbRWuBmpZt+1KJs7AVoqZVA4GIltIib7YwO5W6OkqsNZQ01lBaUU3p2RpK\nzqjy1lDdTPBQQ6fyFhYfgE+Ap0tPxSLlTiIZ+AxUuZsCrBRC3Nhy/EtAtM/eyQ8cSVc0NTV1kLiK\nigq8/HzwCAlACfHFGuyFeYg7Rh87NppQ7E1o7U14iiaMio5GpUXYNN64KXp0Gm+8FW/8NXqCNHrC\nFG8iFU9CFaXzyYEBzBaoa8RYUYWpvBJPi51ms4ljBfl8nfk9n337DaWN9UTFxnSQuCFDhlzJH1n/\nIoBmOkpZZ5LW1bl6wBMwtBT/dvsGIAiHwIlwQZ1nEyWNNZScq6Y0V5W2VnmrKKjDEKzvUt4CI3zR\naFxX3rrjapO7++67j+joaJ599tle32PHjh3cfffdnDt3DoDU1FRee+01Zs6c2e1rL6bupXLfffex\nadMmhg0bxrffftsn91y9ejWnTp3izTff7JP7SfqGC/0du/LyY5HAuXbHBaiDBp3o7NvxypUrWbVq\nVYfzq1Zl347CAAAgAElEQVStYvXq1X1a/53V9+NjrQRgw47jvLvrRIf6i2YMY8m1yR3OD9b6t01O\nZd648TSa9TSZvWgye2Gy6tiT/RV7j2/vUH9ycjqTk2cDLf2+W/4N2Xv8S/bldKw/MSmdycNntfUR\nb9nZd+Ir9p/sWH/SuNlMTb8OxVuLpsSEW0EJGmsJX+/+jN37Pu9Qf97I6/nByHn4KY3Q2k1dUdiS\n9Qlbjn7aof4PUm7glrSbWn4XlZZVuRS2Ht3Gxu83dag/IXwMN6XOIDTAh/tH/QB3rToK9b3tn/PS\nSy91qH/H+NncOfG6Duff2/8573/3pcvXXzh+NneOnocwuUFrMbrx/vGt/PPcxx3qLwhayMKwxQh3\nN9C6qVudG//Mf4sPczM61P/R5GX86I77wANQ1F8GIeDDHW+w8ZP1HepPCP4BMU3XoNW5Ocnbt4Uf\nseGLv7dVLFTLyjkrefwKfZ5cyfqDkfT0dI4cOUJpaekF+8H2lvb/3mRlZfX4de3rrl69mtOnT5OR\n0fF3+VLZvXs3X3zxBUVFRZ2OzL8UXDkTLemIK8vdgEBna0YvagFwx9RpHXdMjjrnnx+M9UN9ypmd\n+C16j0a8dQ1465rQuln5f2Y39h7vWD8+MI/psftoNHvRZNa3bL3QKtZO769oFDTa1mk51A8cRQGN\nW+cfPu7VNvTfGcEuWmurF0rMndZvqDNSWFSFQCBabVNAdW1Dp/Urq2vJOZOPI7MhBAIoLivutH54\ngC9DA/xantXsON/cbOm0fnOzheqqxk7PD4T6xmYztcYa0NrAYFO3bnZMtaXOX99asIw4RvOMdwEb\nimJr2Vqx7zwBuR3rC4894FkHihvghqJoUHBD0WR2Gs+0HzXxuxcMeBtCgWBgCBBM/qpdsKXTlwx4\ntm/fzvbt2/s7jMtKXl4eu3fvxt/fn48++oj58+f3d0hXnLNnzxIbG9vnYicZeLiy3BUCQ9sdR7Wc\ncynmP/uOY3/XqlWwq+M35/jZ9zKnk2/Og7V++DU/I+Gx31DZYOV0mZWyWgvN1kbOhq0B/r8O9ZtF\nJAHB0Qz3ryLIpxBfjyq83KoQ1iK+6UQGb5xewaoHy8EjEHQtxSOQVX/KZe/xrzrUH6M9xUPFJxiS\nV06dt5aTYToyg200enUUFADfed6MXzGGcN9wInwjCPcJx8vdi1WrVrFn9dcd6s/78dwuMynfrt7X\n4fy4hbN5oJP651atYvPqXYOw/hzu76R+/qpVfLRvf4fzY+bMZ1kn9U+vWsW/O/l9GzXrB9y16hHA\n7FROiP/lw+2HO9QPCLPgbTiE2uGuHKho2VZ3qKvyJRCAKoJtMgidf/lwRdLT00lPT3ccd5bhG+hk\nZGQwdepUJk+ezBtvvHFBudu0aROrVq3izJkzhISE8Ne//pW5c+fyxhtv8OKLL1JQUEBISAhPPfUU\nP/nJTzq9R1xcHGvXrmX27NmsXr2ao0eP4unpycaNG4mJiWH9+vWMGzfOqa7FYuF3v/sdAP/+979J\nSEjgmWee4fnnn+fAgQOOe7/88svs2rWLjRs3dnhucXExP/vZz9i9ezdBQUE89dRTLF++nHXr1vHQ\nQw9htVrx8/PjiSeeYOXKlU6vPXPmDD/+8Y85fPgwGo2GuXPn8tprr+Hnp37ZfOGFF3j11Vepq6sj\nMjKS1157jVmzZgHqUonLli3r9P1JXA9X7nPnhjq36xzUSQj2AYuFEMfa1RnQ/UCuNoQQGG126kxW\n6kwWaoxWKhssNNmt2E1uNFZqKc1358xRLVkH3Dm0zw2Dj5WUYTWMiK8iKaaK2PAqokKqCPGvIsi3\nCk8qwVylFlPL1lgGYXMgZhFE3gzalh5xdjucPQtHj0J2NvasLGxZR9DknMDs70tlXCgF0QZOhXtw\nZIiNA/5NnLGUUdJQgqfWk3DfcMJ9wtu27fdbRNBX1/ncdpKBgBWooqP0dbVfAehwFr7u9g10nLvk\nyjMY+9wlJSXx5JNPMnHiRKZMmUJhYSHBwcGAc5+7ffv2MXfuXD788ENmz55NcXEx9fX1DBs2jK1b\ntzJixAhiY2PZtWsXN9xwA19//TVjxoxhx44dLF26lPz8fKCj3L3wwgts3LiRuXPn8swzz/DVV1+x\nZ8+eTuu2b5Y1m81ERETw9ddfM3z4cADGjRvHb37zG374wx92eJ8zZ85k9OjRvPzyyxw9epTrr7+e\n999/n/T0dNavX8/atWvZuXNnpz+j06dPc/bsWa699lpqa2uZP38+48aN4+WXX+bEiRNcd9117N+/\nn9DQUPLz87HZbMTFxXX7/iT9w4DscyeEsCmK8jCwjbapUI518zKJC6MoCl5aN7y0boR6ezjO24Wg\n0Wyj1myhbpSV2uubqTPVY7Ta8FS02I3uNJRHUpIXyxfZWk5u1pCXp3D2rLq+fUwMxMaq25gYiIus\nJbH630RVvoHfnp9iDLoRJWYRnnE3oImPh/h4uPlmNKi/WNjteJ09S1R2NlHZ2UzJzoYd2ZCTA0OG\nIFJmYxyeQJU+jEKDP6dDdZyzV1NQV8D+ov0U1RdR3FBMcX0xAkG4jyp6SYFJpIakkhKSQmpIKuE+\n4VL8XBotENJSeoJAHcXRlQAe7+R8c8tzWn8PlC5KV9f68nzf0ld37Y1C7t69m/z8fO644w4CAgJI\nTExkw4YNPProox3qrlu3jgceeIDZs9V+vuHh4YSHhwNw4403OurNmDGDuXPnsmvXLsaMGdNtDNdc\ncw3z5s0DYOnSpbzyyis9il2n03HnnXfy1ltvsWbNGrKzs8nLy+MHP/hBh7oFBQXs2bOHTz75BHd3\nd0aPHs3y5cvJyMhwysx2RUJCAgkJCQAEBQWxYsUKxyATNzc3zGYzWVlZBAUFMXToUKfX9vb9SfqH\nbuVOUZQ4IURud+cuB0KIT4Dhl/s5kv5Foyj4emjx9dCqSzW0YLWrWb5ak5W6EAuhiSbiZli4DvDT\nuWPw0OJmdaeuTEtJnpa8XA1nz8KePQbq6pZRV7cMjaWca2I+5Oa0P5IScR9bjtzG1qOLyCqfjd7H\nHT8/8PPTYDDE4+cXj5/fLfiNBUM6+PnYCG06S0h5NgHF2fhvzybs9FEmnspBCQ6GlJSWciNMTYER\nI6jX2iluKKaovoicihyyyrLYfGIzmWWZ2Ow2VfSCU52kb4j+KhoVO6hQAL+WktDD15hpa84VXZSu\nrvXVeQEM6/G77Cn9mdfLyMhg7ty5BAQEALB48WLWr1/fqdydO3euU3EC2Lp1K88++ywnTpzAbrfT\n3NzMqFGjehRDWFiYY1+v12M0GrHb7Wg03S/bd88997BkyRLWrFnDW2+9xR133NHpgJCioiICAwPR\n6/WOczExMXz33Xc9irGsrIxHH32UXbt20dDQgM1mIzAwEFDF709/+hOrVq3i6NGjzJs3j5dfftnx\nvi7l/UmuPD3J3H0AnN+w/i9gfN+HI5G0odVoCPTSEejVtkTW+U27tXYzzQGNuHlbGTnKjSkeWgI8\n3Qn19sTfQ4uiBAM/BX6KvaGQ+ZP+ycJzK3FrXkqF13zyWExe0zXU1Wuoq4PaWjh3DrKzoa7Ojdra\nBOrqEqiru9Vx3dhsI7Uyl3F7s0nbn02y+IxE05+IbsrB6BWKJTgFbUQqMVOuI+UH95M4z53QUChv\nKiO7LJussiwyyzJ5J+sdssqy8NB6kBqSSmpwm/ClBKdg8DT0289ecrnQtRRJX2E0Gnn//fex2+2O\nDJzZbKampobMzEzS0tKc6kdHR3P69OkO9zGbzSxYsIC33nqL2267DY1Gw+23305fN0d3lr2fPHky\nOp2OXbt2sWHDBt55551OXgkRERFUVVXR2NiIt7fa3SQ/P5/IyMgePftXv/oVGo2G7OxsDAYDmzZt\n4pFHHnFcX7RoEYsWLaKhoYGf/OQn/OIXv2D9+o4jziWuT5dypyhKMuoSlwZFUX7U7pIf6mxSEskV\npydNu1XNFvYXV2OxCcK8PQjz8SBE74G7TyReYx6DMY9BQy5hee8RlvdzJuvKYeydah+9oInQTdOp\nzeZGQ0MidXWJ1NbeRl0dnKqDQzU27KfO4Hk6G58zR0h64xmCXjnNVrdb+FCZT96w64kZNoukpFlM\nSoS7kiDhBoHdu4js8iyyy7P5tuBb/nHwHxwtP0qAV4BD9Fq3I4NH4q3rclY9ieSqY+PGjWi1Wg4f\nPuyU7Vq4cCEZGRn8/ve/d6r/wAMPMG/ePG6++WbS09MpLi6moaGBiIgIzGYzQ4YMQaPRsHXrVrZt\n29ZBDntKV1IYGhrK559/jhDCSfSWLl3Kww8/jE6nY9q0aZ2+NioqimnTpvH000/z+9//npycHNau\nXdulDJ5PfX09/v7++Pr6UlhY6PSzOXHiBIWFhUyfPh2dToeXlxd2u/2i35/ENbhQ5m44cDPqVKC3\ntDtfD/z4cgYlkVws7Zt2o3y9GIUfDWYrpY0mztY2811xLf6e7oT5eBDm7YGvdyxKyi8h5ZdQewzy\n3oM9S8FuUSUvZhH4p3Uqem5uYDCoJTra6QqQ1FJ+CPwGzp1j8YcfsvC9FyFrKYU+N7LPbT7bzt7A\nP/7hzalTCk1NkSQmRpKYOI+kJJieCAnT7XiG5lGuZJNdnsUXuV/wyt5XyKnIIdw33En4UkNSGT5k\nOJ5a+Z1LcvWRkZHB/fff3yF79fDDD/Poo4/ywgtOq1YyceJEXn/9dR577DFyc3MJCwvjr3/9K8OG\nDePPf/4zCxcuxGw2c8stt3Dbbbd1+dzu+s+2v95+f+HChbz11lsEBQURHx/vGCW7dOlSfv3rX3cY\n4Xo+77zzDj/96U+JiIggMDCQNWvWOEa0dsfKlSu555578Pf3JzExkaVLl/LHP/4RUEfD/vKXv+T4\n8eO4u7szbdo0/u///q9H70/ienQ7WlZRlKlCCJccEjMQRnBJXAOrXVDeZKKk0URJgwkFCPPxINTb\ng2C9B1qNos58W3MY8t5Vi5u+TfT8+qCPUkkJbNoEH3wAe/fCnDkwfz61M27mVLmBU6fg5Emcto2N\nkJAASUmQmAjxiVb0kWcw+mVRZFXFL6ssizPVZ4gxxDgJX2pIKomBibi79f1krpKBzWAcLTvQMRqN\nhIaGcvDgQcegB4nkQlzS8mOK2mnpx0As7TJ9Qoj7+zDGXiE/cCS9QQhBvdnqEL0ak4UgL53ahOvt\ngbdOq4pe5V5V8vLfB8+wFtG7E7xjLj2Iqir46CNV9HbsgBkzYP58uO02CApyVKuthdOnO0rfyZPQ\n0KAKX2IixCeZ8Yk5gX1INrW6LPKNat++ovoiRoeNZlLEJCZFqiU+IF5+677KkXLnerz88sts2bKF\nzz/vuGqORNIZlyp33wC7gO8AW+t5IcQHfRlkb5AfOJK+wGyzU9akil5powmdm0KYtyeh3h4M0evQ\nCDuU71RF79wH4DtMFb2hC8Er/NIDqKuDLVtU0du2DSZMUEXv9tshvOv719V1LX719WrGb3haPbHT\nvkMbs4+TTfvYV7iPRkujKnotwjcxciIh3j2d/kMyGJBy51rExcUB6sTGo0eP7udoJAOFS5W774UQ\n3U/y0w/IDxxJXyOEoMZkoaRBbcJtMFsJ1qsZvVAfD7w0dij5XBW9go8gcKwqetHzwSOo+wd0R1MT\nfPqpKnr/+Y861cqPfqTKXkzPM4b19aroHToEX3wBn38Ofn5w3XUw/tpivBL3c6xOlb39Rfvx9/R3\nEr5x4ePkwI1BjJQ7iWTgc6ly9xzwjRDC5VZdlB84ksuN0WqjtNHkKHp3N8J8PAnz9iDQ3YZS/Ikq\nesWfwJDpELsYom4Dd79Lf7jJBF9+qYrepk2q3M2fr5ZhF9cHUAjIylIl7/PPYdcuGD5clb3Zc+yE\njTzF4QpV9vYV7iOzLJPEwEQmRUxictRkJkVOYmTwSLQal533XHIRSLmTSAY+lyp39YA3bYs2KoAQ\nQvTBv16XhvzAkVxJ7EJQ1WyhpNFIaaOJZqudUL2OMB9PQnQWPEr+o4pe2XYIuw7SVoN/at883GqF\nnTtV0du4Ue2XN3++mtVL63xU74Uwm9UxHZ99pspeZiZMmaLK3nXXwYhUE9kVR1TZK1KFr6CugLFh\nYx199yZFTiLGECP77w1ApNxJJAOfS5I7V0Z+4Ej6kyaLmtUraTRS3mTGT6clzMeTcPcm/IreQcle\nA6krYdjDFy1fF8Ruhz174MMPVdlzd2/L6E2Y0Ktn1daq4zpaM3ulpTB7dpvsxcdDnamWA0UHHMK3\nt2AvVrvVSfYmRkwkSN8HzdOSy4qUO4lk4HOpmTsFuAuIE0KsURQlGggXQuzr+1AvDvmBI3EVbHZB\nRbOZkkYjJQ0mbEIwwbuckO/vB48hMOV18Art+wcLAQcPqpL3wQfQ3NzWR2/aNHVSvl5QWNjWV+/z\nz8HDo030Zs+GlvXYKawrdDTl7ivax4GiAwTrg52Eb2zYWLzcvfrwTUsuFSl3EsnA51Ll7m+AHZgt\nhBihKEoAsE0IMbHvQ7045AeOxFWpaDKzt6iatCFeDM17AU6vg8n/gMjO17TsE4RQ101rzeiVlcHC\nhfBf/wXJyZd022PH2kRvxw41k9cqezNmQOtSl3Zh53jF8TbhK9zH0fKjxAXEkRaSppbQNFJDUon1\nj0WjyHUp+wMpdxLJwOdS5e6gEGKcoiiHhBBjW84dFkL0+3ht+YEjcWXqTBa+LqhiWKAPCZbv4Jul\nEHkLjP09aK9AJuvkScjIgL//HUaNgkcegZtu6nU2rxWLBfbvb5O9Q4fU1uBW2Rs/HrTtxl0YrUaO\nVxxX19QtzSSzTC01xhpSglNIC1FlLy1Ulb9g7+BLfOOS7pBy1/+cO3eOlJQUamtrZb9VSa+4VLnb\nC0wD9rdIXjBq5m5s34d6ccgPHImr02SxsvtcFVF+XozwtaLsfxBqjsD0DRBwhWYYMpng/ffh1Veh\nslLN5N1/PwQE9MntGxrUsR6tsnfuHKSnt8nesGGddwOsMdZ0EL6ssiw83DzU7F5wm/DJNXX7lsEm\nd7GxsZSUlFBUVERgYKDj/NixYzl8+DBnz55l6NCh/RihRNL3XKrc3QXcCYwD1gMLgP8nhPhnXwd6\nsbj6B45EAup0Kl8XVBHkpWN0sC9K3ttw8HEY+TQkPwZXsmly715V8v7zH7jjDjWbl9pHI3pbKClR\nZ3BplT0h2kRvzhwIC+v6tUIICusLHcKXVZZFZlkmORU5RPhGOGSvNduXFJQkp2fpBYNN7uLi4vD0\n9OThhx/moYceAiArK4sFCxZw8uRJcnNzpdxJBh0X+jvu9l8VIcTbwFPA/wDFwA9dQewkkoGCp9aN\nmdFB1Jms7C+pxR57N8zbC+f+BV/Ng6aiKxfM5Mnw1ltqJ7qICJg7F2bNUqdXsVr75BFhYbBkCaxb\nB3l56sCMSZPUroAjR6ozt6xYAR9/rE623B5FUYjyi+LGpBt5avpTZNyewaGfHqLu6To2L97MktQl\nAGzI2sAt79yC3//4MfZ/x3LPxnt48esX2XpyKwV1BbiyiEguD0uXLmX9+vWO4/Xr17Ns2TLH8ZYt\nWxg3bhwGg4GYmBhWr17t9PqMjAxiY2MJDg7mueeeIy4uji+//BKA1atXc+edd7Js2TL8/PxIS0vj\n4MGDjtcWFxezYMECQkJCSEhI4NVXX3Vc279/PxMnTsRgMBAeHs6TTz4JQF5eHhqNBrvdDuD0vNZn\nLl261KnuG2+8wdChQwkKCuJ///d/OXDgAKNHjyYwMJBHHnmkr36UksGAEKLTAvi1bAM7K1297koW\nNXyJZGBgtdnFNwWVYve5SmGx2YWwWYQ4skqID0KEyP+wf4IymYTYsEGIqVOFGDpUiOefF6Ki4rI9\nzmoVYt8+IX73OyFmzxbCx0eI6dOFWLlSiF27hDCbL+5+DaYGsa9gn1h7cK14bOtjYs76OSL096HC\n/3l/MWPdDPHgxw+K1/a9Jnbl7RLF9cWiydwk7Hb7ZXlvA4mWz85B81kbGxsrvvjiC5GcnCyOHz8u\nbDabiI6OFvn5+UJRFJGXlyd27NghsrKyhBBCZGZmirCwMLFp0yYhhBDZ2dnCx8dHfPPNN8JisYgn\nn3xS6HQ68cUXXwghhFi1apXw8vISn3zyibDb7eLpp58WU6ZMEUIIYbfbxfjx48Vzzz0nrFaryM3N\nFQkJCWLbtm1CCCGmTp0q3nrrLSGEEI2NjWLv3r1CCCHOnj0rNBqNsNlsTu+hlVWrVomlS5c66iqK\nIh588EFhMpnEZ599Jjw9PcXtt98uKioqRGFhoQgJCRE7d+683D9qiQtxob/jC7VnbABuRl1Ttv3X\nYKXlOL4vJVMiGey4aRQmRwRwsKSW3ecqmRYViC5tJYRdD9/cDUVbYfwfQXsF+5bpdLB4sVq++05t\nsk1MVKdTeeQRGNO3/QLd3GDiRLU8/bS62trXX6vNt489po4Bueaatmbc1NQLT9vnrfNmYuREJkY6\nD94vbyxX+/GVZnKw+CDrD68ntyaXWmMtdmHHz8MPg6cBg4cBg6dBPfbo5Lilzvn1fXQ+cqTv+fTV\nmIBLSLq2Zu+uvfZaRowYQUREhOPazJkzHfupqaksWrSIHTt2cOutt/LBBx9w6623MnXqVACeffZZ\n/vznPzvd+5prrmHevHmO57zyyisA7Nu3j4qKCp555hlA7f+3fPly3n33Xa6//nrc3d05deoUlZWV\nBAUFMWnSpF69N0VR+M1vfoNOp+O6667D29ubxYsXExSkzis5Y8YMDh06xIwZM3p1f8ngoku5E0Lc\n3LKNu3LhSCSDG42iMD7MQGZ5PTvzK5keHYhX8DS46Xs48AhsHQfT3oagCVc+uPHj4Y03oLxcHWF7\nyy0QG6tK3u23q5Ml9zF6PVx/vVpAHe/x1Veq7P31r+pgjTlz1HLdddDTblPB3sHMjpvN7LjZHa6Z\nrCZqTbXUmeqoNdZSa6ql1thy3LJf1ljGqapTjuPz6zdZmvDR+XQqfn66LsSxnSi2Fg+tRx/+NPsZ\nF2gJv/vuu5k5cya5ubncc889Ttf27t3L008/TVZWFmazGbPZzMKFCwEoKioiOjraUdfLy8shTa2E\ntessqtfrMRqN2O128vPzKSwsdAzkEEJgt9sdMrlu3Tp+/etfk5ycTHx8PL/5zW/4wQ96NyVSSEiI\nU4yhoaFOxw0NDb26r2Tw0W1PZEVRbge+FELUthz7A+lCiH9f7uAkksGIoiikBfuS46ZhZ34l10QF\n4q3zg6nr4ey7sP0mSH4cRvw3aC5t2pJeERwMv/oVPPUU/Pvfajbv8cfhZz+Dn/wE2v0D09cEBcGC\nBWoByM1V++x99hn88pfg79+W1Zs1q3cDfj20HoRoQwjx7v37sNlt1Jvru5TDVhksqi9Sz7WTwzpT\nnaOuRtE4yV5rOV8CuyteWi85nQYwdOhQ4uLi2Lp1K+vWrQNw/Fzuuusufv7zn/Ppp5/i7u7OihUr\nqKysBCA8PJwTJ0447tPc3Oy41h3R0dHEx8eTk5PT6fWEhAQ2bNgAwAcffMCCBQuoqqrqUM/b25um\npibHcUlJSY+eL5F0Rk+Gma0UQmxsPRBC1CiKshKQcieR9BJFUUgO8kHnprDjXCXTIwMxeLpD7CII\nngZ7lkLxJzD1TfCO7v6GlwOtts20Dh+Gv/wFhg9XM3qPPKK2rV5m4uJg+XK12O3qGriff64mFu+9\nVw2nVfamTwdPz8seEgBuGjf8Pf3x9/Tv9T2EEJhsJofsXagU1Rdd8LrZZr4oGRzMrFu3jurqary8\nvLDZbI7BNQ0NDQQEBODu7s6+ffvYsGGDo5l1wYIFTJ06lW+//Zbx48ezatWqbp/Tet9Jkybh6+vL\niy++yM9//nPc3d05fvw4zc3NTJgwgbfffpt58+YxZMgQDAYDiqKg0Wic7gEwZswY3n33XW644Qa+\n//57/vWvf3HjjTd2eJ5E0hN6IneddSyRcw9IJH1AvL83Oo2G3QVVTIkIIEivA++hMPtLOPYifDIe\nJvwFYu7o30BHj1aN6vnnYe1adeWLsDBV8hYuVPvuXWY0GjWM0aPhiSfU6fu+/VbN7P3616r4TZnS\nJntjxlzyfM2XFUVR8NR64qn1vKQsIoDFZqHeXN+tJJ6tOUutqbaP3oHr0D5rGRcXR1xcXIdrr732\nGo8//jgPP/ww1157LXfeeSc1NTUAjBw5kldffZU777yTpqYmHnvsMUJCQvDw6LrZvPW+Go2Gjz/+\nmMcff5y4uDjMZjPDhw/nueeeA+CTTz7h8ccfp7m5mZiYGN577z3HfdvHvWbNGhYvXkxgYCDXXnst\nd911l1OG7/zMbHfHkqubnsxztw6oAf7acuoh1NGy917e0LrH1edekkh6SmmjiQPFNYwPMxDm0y79\nVLkfvrkLhkyDCa+Cu2//Bdkemw02b1abbI8eVZtrf/YzCA/vt5Bqa9Wl0Vrn1ystVZtuW2UvIeHC\ngzOuJgbbPHd9TWNjI/7+/pw6dYqYmJj+Dkci6ZRLncTYG/g1cF3Lqc+A54QQjX0apfMzVwI/Bspa\nTv1KCPFJJ/Wuqg8cyeCmqtnMnsJqRoX4Ee3XbnkySwMcXAGlX6qDLYZM6b8gOyM7W22yffdduOEG\nNZs3dWq/m1RhoZrV+/xzdWuxqIIXF6eujRsf37YfFeXaWb6+RspdRz7++GPmzJmD3W7niSeeYP/+\n/Xz33Xf9HZZE0iWXJHf9QYvc1QshXu6m3qD/wJFcXdS2rEc7PNCHhIDzpkQ59yHsfxCSHoKUX4Gr\nrcxQUwOvv64OczUYVMlbtOjKdYS7AEJAUZE6QCM3F86ccd6WlUF0dOfiFxcHgYH97qp9ipS7jvz4\nxz/mX//6FwATJkzgtddeIykpqZ+jkki6pldypyjKn4QQjymKsplOBrkLIW7t2zCdnr0SaBBCvNRN\nvUH/gSO5+mg0W9ldUMVQPy+Sg3yc+9I0FcKeZWBrhmlvgY8LzlRkt8PWrWqT7aFD6miIBx5QTclF\nMc000tgAACAASURBVBrV1TQ6E78zZ1Q57Er8YmNdwl8vCil3EsnAp7dyN04IcVBRlGs7uy6E2NGH\nMZ7/7JXAvUAtcAB4onUqlvPqyQ8cyaCkdT3aIV46RoX4OQuesMPxP8LR52HcHyHu7v4LtDtycuBv\nf4O331bXHlu2TB2A4esifQd7SHW1s+y13z93Tp3CpTPxi49XuyFqXGy+Yyl3EsnAp7dy94UQYo6i\nKC8IIX5xGYL6DAhtfwo1Q/gM8C1QIYQQiqI8B4QLIR7o5B5i5cqVjuP09HTS09P7OlSJpF+w2Ox8\nU1iNXqthfLg/mvPbBau/h6+XQMBYmPhX0PV+Wo7LjtkM//kPrF8P27er06ksW6aOeBjgnd1sNrV/\nX1dZv5oaiIlRRS82VvVanQ48PNRt+/3ebnW6Cwvk9u3b2b59u+N49erVUu4kkgFOb+XuKLAcWAss\n4bzFZYQQBzt7XV+jKEoMsFkIMaqTa/IDRzKosdkFe4uqAZgUEYBWc97fsbUJDv03FP1HnRMvZAAs\nPVReDu+807YaxtKlqugNH97fkV0Wmprg/2fvvsOjqtIHjn/vpE/IJDNJSCMECB0pS5AOEl1RmjQX\nA4iCLiKIBVxUliIIVtBV1pWfBQERIuq6IgQERZoGaYKAFCMlgYRimCSTRkJm3t8fE0ZiKqSH83me\neTL3nnPufWdgJm/Oveec06ftiV58PGRm2qdxyc2t2J/OzmVPBr/5RvXcKUptd6PJ3b3Aw0BPYA8F\nkzsRkcLr+lQQTdMCReR8/vMpwK0iMqqIeuoLR6nzbCLsO59G1pU8uoWYcHUqoosmcR3sGg/hf4e2\ns0FX8UuFVYqDB+29eStX2q9jPvgg3HffjS09cRMTgby8siWBOTlw990quVOU2u5Gk7seIvKDpmmz\nReSFSo2w8Lk/AjoANuA0MEFELhRRT33hKDcFEeHg7xaSs3Lp0cCEu3MRlzKzz8OP4yDXbJ8yxatp\n1Qd6o/LyYONGe6K3caN9SpUHH4S+fe1dUkqFUvfcKUrtV9LnuKTbfBfl/xxS8SGVTEQeEJF2ItJB\nRIYUldgpys1E0zTa+RsIrufOtoRLZObmFa7kEQh91kOj+2FTNzix1N6lUxs4O8OAAfDpp/ab1fr0\ngRdesM9PMm0aHD5c3REqCjk5Oeh0OpKSkq677fHjx3Fxqfoe9Y0bN6opXW5CJSV3VzRNew8I0TRt\n0Z8fVRWgoih2mqbRys+LpkZPtp25RFrOlaIqQYvH4Y4tcPxf9mlTbNaqD7Y8TCaYONG+ttiWLfbE\n7+67ISICFi2C5OTqjlCpQby8vDAYDBgMBpycnNDr9Y590dHRJba9kcSnPMt8VdcSYWppsptPScnd\nQOA74DKwr4iHoijVINzoSVt/A9+fMXMpO7foSj63QN9dkH0W9j5We3rw/qxlS3j5ZftIhFdegV27\noGlTGDoU1qyx30im3NTS09OxWCxYLBbCwsKIiYlx7Bs5cmSJbUXkuhMfdXlaqQ2KTe5EJFlEPgHu\nEZHlf35UYYyKovxJqMGDiCBvfkxM4UJmTtGVnD2g95f29WkPzq7aACuakxPcead94EV8vP0S7sKF\n9nXDnnwSfvqp9iawSoURkULJ1+XLl3nssccIDg6mYcOGPPPMM1itVsxmM8OGDePkyZOOnr6UlBRi\nY2Pp2rUrRqORBg0aMHXqVGw2W5nO361bN2bPnk2nTp0wGo387W9/Iz09vUB8y5YtIzQ0lICAABYu\nXOgos9lszJs3j/DwcOrXr8+YMWOwWCzAH5d0i2tb3Gssyrx58wgODsbb25s2bdrwww8/lPn9VWqP\nskytma1p2mZN0w4DaJrWTtO0mZUcl6IopQj0dKdriJG951I5a8kuupKLASI3QMKncOzNqg2wsnh7\n21e92LEDdu4EHx8YNgzat4fXX4fz56s7QqUGmT17NocPH+aXX35h3759bN26lddeew2TycT//vc/\nmjRp4ujpMxqNuLq68p///IeUlBR27NjBunXr+OCDD8p8vhUrVhAdHU1iYiI5OTlMnTrVUWa1Wtm3\nbx8nTpwgJiaGGTNmcPr0aQAWLFjAt99+S2xsLGfPnsXFxYWnnnqqTG2Le41/dvDgQZYtW8bBgwdJ\nS0sjJiaGBg0a3Ngbq9RsV//SKe4BbAM6A/uv2Xe4tHZV8bCHryg3t9TLuRLz23k5Yc4ovlLGaZH/\nNRA5+VHVBVaVrFaRLVtExo4V8fYW6d9f5NNPRbKzqzuyGin/u7OCv2sr6qv9xjVq1Eg2b95cYF9I\nSIhs3brVsb1mzRpp1aqViIh8/fXX0qxZsxKP+corr8ioUaNEROTy5cuiaZokJiYWWbdr164yd+5c\nx/ZPP/0k9erVExGRY8eOiU6nE7PZ7Chv166drFmzRkREGjduLLGxsY6ykydPil6vL1Pbsr7GX375\nRYKDg2XLli2Sl5dX4utWar6SPsdl6bnTi8juP+0rYqieoijVwdvNhdtCfYlLyeRocnrR9wR5hkHk\nRtj/D/uceHWNTmcfYbt0KZw9a58r7//+D0JC7IMzdu1Sl20rnVTQo2KdP3+ehg0bOrbDwsJITEws\ntv7Ro0fp378/gYGBeHt7M2/ePJKvYxBPaGhogXNlZWU5Ls06OTlhvGYOR71eT0ZGBgBnzpyhf//+\nmEwmTCYTHTt2BMBsNpfatqyvsXXr1rzyyivMmDGDgIAAxowZw8WLF8v82pTaoyzJXbKmaeHkf+ry\nJzc+V6lRKYpyXTxdnbmtoS9JGZc5+Lul6ATPuzX0/so+F97FHVUfZFWpVw8eeAA2b7bfixcSAvff\nb18D7G9/g9des4/Czb+fSanbgoKCiI+Pd2zHx8cTEhICFD2KdPz48URERHDq1CnS0tKYNWvWdQ2i\nOHPmTIFzXR29W5oGDRrw3XffYTabMZvNpKSkkJmZiclkKrVtYGBgsa/xz8aMGcMPP/zAyZMnyc7O\nZtasWWV4VUptU5bk7jHgXaClpmmJwFPAo5UalaIo183d2Yleob6kXr7CvvNp2Ir6heTXxT7B8Y7h\nkPJz1QdZ1cLCYOZM+PVX+PZbGDIEkpJgxgwICoJWreyJ4L//bZ965fLl6o5YqWBRUVHMnTsXs9nM\nxYsXeemllxgzZgwAAQEBXLx4kczMTEf9jIwMvL298fDw4JdffuH999+/rvMtW7aMuLg4MjIymDt3\nLlFRUY6ykpLECRMm8Oyzz3L27FkALl68yLp1f/Syl9R25MiRxb7Gax09epTt27eTm5uLm5sbHh4e\n6EpalFiptUr9VxWRkyLyV8AfaCkiPUUkvrR2iqJUPVcnHT0a+JJjtfFjYgpWWxG/EIL6Qqe3YWt/\nSD9R9UFWB02D5s1h9Gh4802IjYXUVPsat716waFD9su3JhN07AiPPgpLltiXR8tTd6HUFkX1xL3w\nwgu0bt2aNm3a0LFjR3r16sW0adMAaN++Pffccw9hYWGYTCZSU1N54403eP/99zEYDDz++OMFkrPi\nznGtMWPGMHLkSEJDQ3F2di4wqvXPba/dfuaZZ7jzzju5/fbb8fb2pmfPnuzfv79MbUt6jdfKzs7m\n6aefxt/fn5CQEDIzM5k3b16Jr0epnYpdfsxRQdO8geeB3vm7tgEviEhaJcdWKrUkjqIUzSbCvnOp\n5Fht9GhgKvoXUtz/wZHXoO8P4BFU9UHWRNnZcOAA7N4Ne/bYH4mJ0KED3HrrH4+mTe0JYy2llh+r\nHN26dePxxx9n1KhCS6ErSoUr6XNclkUbPwQOAyPyt8cAS4FhFROeoigVTadpdAry4fuzZo5eyqC1\nXxH3/DR7FHKSYctd8Ndt4GosXOdm4+EB3brZH1elpsK+ffZE77//heeeg/R06NTpj2Svc2f7vX2K\noig1QFmSu3ARGX7N9lxN0w5UVkCKolQMTdO4NciH7+KT8fVwJcDTrXClNjPsCd62QRC5CZz1VR9o\nTefjA3fcYX9cdf487N1r7+F77z0YPx5cXQv27nXqBL6+1Re3UuXUMl9KTVGWy7I7gWki8n3+dg9g\noYh0K7FhFVCXChSldL9n5bA7KZXbw/zwcHEqXEFs9jVoc832FS10Vb+4ea0nAqdP/3Epd88e+0hd\nf/+CCV/HjvbRvNVMXZZVlNqvpM9xWZK7DsBywDt/VwowVkSqfaid+sJRlLI5dimdC5k59Ar1RVdU\n74LtCmwfCq4+0O0j0NQIunKzWuH48T+Svd277QM0bDZwcbH39Lm4FHxU0T7t3ntVcqcotVy5krtr\nDmIAEJEaMzmU+sJRlLIREWITU/B2c+YWf0PRlfKy7PffGf8CEW/V6gEDNZbNBleuQG6u/eefH0Xt\nL+u+66irffGFSu4UpZYrb8/dS8BrIpKav20EnhaRal9fVn3hKErZ5eTZ+C7+d9oHeBNcz73oSrmp\n8O1tEHovtFWTm9ZV6rKsotR+JX2Oy3Ltpd/VxA5ARFKA/hUVnKIoVcPNWUfnYCP7z6eReaWYudtc\nfezLlJ1aDnGLqzZARVEUpUKUJblz0jTNMcxO0zQPoIhhd4qi1HS+Hq40N3myOym16AmOATwC4fZN\ncHg+xK+u2gAVRVGUcitLcrcS2Kxp2sOapj0MfIN9gIWiKLVQU6Mn7s46Dv9ewu2z9ZpAnw2w93FI\n2lh1wSlKBYmPj0en02Gz2ao7lGL179+fFStWFFlWG+JXaq4yDajQNO1u4K/5m9+ISI34tlf3gSjK\njcm12tgSn0wbfy8aeHkUX/Hi97BjKNy2Fvy6Vl2ASqWqa/fcNWrUiIsXL+Ls7IyIoGkaGzdupGfP\nnly5cqVWrp8aHx9PkyZNam38SuUr7z13iMjXIvKP/EeNSOwURblxrk72++8OXLCQnlvC2qn1e0LX\n5bB9MKT+UnUBKsp10DSNmJgYLBYL6enpWCwWgoODqzssRak26s8BRblJGd1daO1bj91JKcXffwcQ\n0h/+8gZsvRsyTldZfIpyPUrrWVy2bBmtW7fGYDDQtGlT3nvvPUdZ69atWb9+vWPbarVSv359Dhyw\nL8Y0YsQIgoKCMBqN9OnThyNHjjjqjhs3jsmTJzNw4EAMBgPdunXj1KlTjvLY2Fg6d+6M0WikS5cu\n7Ny501EWGRnJhx9+CIDNZuMf//gH/v7+NG3alJiYmELxh4eHYzAYCA8PJzo6+gbeJeVmoZI7RbmJ\nNfbR4+XqzM8X00qpOBpaTYMtfeHyxaoJTlEqUEBAAOvXr8disbB06VKmTJniSN5GjhzJqlWrHHW/\n/vpr/P396dChA2C/N+7EiRNcvHiRjh07Mnr06ALHXr16NXPnziU1NZXw8HBmzJgBQEpKCgMHDuSp\np57i0qVLTJkyhQEDBpCSklIovvfee4/169fz888/s3fvXj7//HNHWVZWFk8++SQbN27EYrEQGxvr\niE1RilJqcqdp2pNl2Xe9NE27V9O0w5qmWTVN6/insumapsVpmnZU07S+5T2XoihF0zSNvwR6k5yd\nS3xaVsmVWzwBYSNhy91wpcbMZa7UEJqmVcjjRg0ZMgSTyYTJZGLYsGGFyvv160ejRo0A6NWrF337\n9mXHjh0AjBo1iq+++orLly8DEB0dzciRIx1tx44di16vx8XFhdmzZ/Pzzz+Tnp7uKB86dCgRERHo\ndDpGjx7tSBpjYmJo3rw5o0aNQqfTERUVRcuWLVm7dm2h+D777DOeeuopgoOD8fHxYfr06QXKnZyc\nOHToEJcvXyYgIIBWrVrd8Hul1H1l6bl7sIh9Yyvg3IeAocC2a3dqmtYKGAG0AvoB72hqNWZFqTQu\nOh1dgo0c+j0dS86Vkiu3nQN+3WDbPWC9XCXxKbWDiFTI40atWbMGs9mM2Wzmiy++KFS+YcMGunXr\nhq+vL0ajkQ0bNpCcnAxAeHg4rVu3Zu3atWRnZ/PVV18xatQowH659LnnnqNp06b4+PjQuHFjNE1z\ntAUIDAx0PNfr9WRkZACQlJREWFhYgTjCwsJITEwsFF9SUhKhoaEF6l17zNWrV7N48WKCgoIYNGgQ\nx48fv5G3SblJFJvcaZo2UtO0tUBjTdO+uuaxBTCX98QiclxE4oA/J26DgU9EJE9ETgNxQOfynk9R\nlOJ5u7nQ1t+LXUkp5JU09YKmQcQicA+EH6LAVsJgDEWpQiUlhrm5udx7770888wz/P7776SkpNCv\nX78CbaKioli1ahVr1qyhTZs2NGnSBIBVq1axdu1avvvuO1JTUzl9+nSZE9Hg4GBOnz5dYF9CQgIh\nISGF6gYFBXHmzBnHdnx8fIHyO++8k02bNnH+/HlatGjB+PHjSz2/cvMqqecuFngdOJb/8+rjaeCu\nSowpBDhzzXZi/j5FUSpRmLcek7sr+8+nlfyLS+cE3T6CvGzYPR5q8BQZys3t6v/j3NxccnNz8fPz\nQ6fTsWHDBjZt2lSgblRUFJs2bWLx4sWOXjuA9PR03NzcMBqNZGZmMn369DJfPu7fvz9xcXF88skn\nWK1WVq9ezdGjRxk0aFChuiNGjGDRokUkJiaSkpLCq6++6ii7ePEiX331FVlZWbi4uFCvXj2cnJxu\n5C1RbhLOxRWISDwQD3S70YNrmvYNEHDtLkCAGSJS+KaDGzBnzhzH8z59+tCnT5+KOKyi3JTaB3iz\nNT6Z02nZNPbRF1/RyRV6fwGb/woHnoG/LKi6IJXrtnXrVrZu3VrdYVSa4pKtq/vr1avHokWL+Nvf\n/kZubi6DBg1i8ODBBeoGBgbSrVs3duzYwWeffebY/8ADD7Bx40ZCQkLw9fVl3rx5vPvuu2WKy2Qy\nsW7dOp544gkmTpzoGAVrNBoLxT1+/Hji4uJo37493t7e/OMf/2DLli2A/dLwG2+8wYMPPoimaXTo\n0IHFi9XygErxSp3EWNO0YcCrQH3syZkGiIgYKiQA+2Xep0Xkp/zt5/KP/2r+9tfA8yKyq4i2NXpi\nTUWpjdJz89iWcIkeDUwY3V1Krpxjhm97Q+MHoPUzVROgUm51bRJjRbkZlXcS49eAe0TEW0QMIuJV\nUYndNa4N7isgStM0V03TGgNNgd0VfD5FUYrh5epMh/oGdielcMVaytJHbiaI3Ahxi+G3D6omQEVR\nFKVEZUnuLojI0Yo+saZpQzRNOwN0BdZpmrYBQESOAJ8CR4D1wCT1J6OiVK0GBg8CPN3YV9r9dwD6\nEIjcBIdmw5nCoxQVRVGUqlXsZdn8y7EAtwGBwJdAztVyEan2b3F1qUBRKo/VJmxLuERDbw+aGj1L\nb2DeD1vugh6fQODtlR+gcsPUZVlFqf1K+hyXlNwtLeGYIiIPVURw5aG+cBSlcmXm5rE14RLdQoyY\nPFxLb3BhG3x/L/TZAL6dKj9A5Yao5E5Rar8bSu5qA/WFoyiVLyn9Mj9ftHB7Iz/cnMpwJ8fZNbD7\nUbhjC3i3rPwAleumkjtFqf3KldxpmraoiN1pwF4RWVMB8d0w9YWjKFXj0EULltw8uocYyzbH14ml\ncHgu/HUHeIaWXl+pUiq5U5Tar7yjZd2BDthXiogD2gENgIc1TXuzwqJUFKXGauPvxRWrjV/NmWVr\nED4Omj8OW/rC5eTS6yuKoigVpiw9dz8CPUTEmr/tDOwAegKHRKR1pUdZfGzqr0lFqSJZV6xsiU+m\nc7AP/nq3sjU6MB3Ob4bbvwFX78oNUCkz1XOnKLVfeXvujEC9a7Y9AVN+spdTdBNFUeoavYsTEUHe\n7DmXyuU8a9katX8J/LraJzrOKrxYuqLUBv/73/9o2LAhBoOBAwcO0LhxY7777rvqDqtYL7/8Mo88\n8kix5eWJX6fTcfLkyRsN7YbEx8ej0+mwlbTutVJAWScxPqBp2lJN05YB+4EFmqZ5At9WZnCKotQs\ngZ7uhHnr2XMutUwLp6NpEPEWNBoFm7pD6i+VH6Ry02nUqBEBAQFkZ2c79i1ZsoTIyMgKOf60adN4\n5513sFgsdOjQoUKOWZmmT5/Oe++9VynHLuu6unXlvLVVqcmdiCwBumOf5+5/QE8R+UBEMkVkWmUH\nqChKzdLa196Rf/RSRtkaaBq0fhbavwjf3Q4Xt1didMrNSNM0bDYbb775ZqH9FSE+Pp7WravtDqQa\nRV2erx2KTe40TWuZ/7MjEAScyX8E5u9TFOUmpGkatwb5cDotiwuZ13FnRuP7oftK2HEvJHxWen1F\nuQ7Tpk3j9ddfx2KxFFkeGxtL586dMRqNdOnShZ07dzrKIiMjmT17Nj179sRgMHD33XdjNpvJzc3F\ny8sLm81Gu3btaNasWaHj7tmzh+7du2M0GgkJCeHxxx8nLy8PgEmTJjFtWsE+kCFDhjiS0FdffZWm\nTZtiMBi45ZZb+PLLLx31li9fTq9evZg2bRomk4nw8HC+/vprR/m5c+cYPHgwvr6+NG/enA8++GP5\nv7lz5zJmzBjH9ooVK2jUqBH+/v689NJLJb6P48aNY+LEifTt2xeDwUBkZCQJCQkF6nzzzTc0b94c\nk8nE5MmTC5R9+OGHtG7dGl9fX/r161egrU6n49133y2yrYgwf/58GjVqRGBgIGPHji3233LZsmWE\nh4djMBgIDw8nOjq6xNd0UxKRIh/Ae/k/txTx+K64dlX5sIevKEp1uJh5WdbFnZfM3Lzra2jeL/JF\niMjRf1VOYEqp8r8768x3baNGjWTz5s0yfPhwmTlzpoiIfPDBBxIZGSkiImazWYxGo6xcuVKsVqtE\nR0eL0WgUs9ksIiJ9+vSRpk2bym+//SaXL1+WPn36yPTp0x3H1zRNTp48Weh8IiL79u2TXbt2ic1m\nk/j4eGndurW89dZbIiKyfft2adiwoaNdSkqKeHh4yPnz50VE5PPPP3c8//TTT8XT09OxvWzZMnF1\ndZUlS5aIzWaTxYsXS3BwsONYvXr1ksmTJ0tubq4cOHBA/P39ZcuWLSIiMmfOHBkzZoyIiPzyyy9S\nr149+f777yU3N1emTp0qLi4ujvj/bOzYsWIwGBz1n3zySenZs2eB92LQoEFisVgkISFB/P39ZePG\njSIi8uWXX0qzZs3k+PHjYrVa5cUXX5Tu3buXqe2SJUukWbNmcvr0acnMzJRhw4Y5XsPp06dFp9OJ\n1WqVzMxMMRgMEhcXJyIi58+flyNHjhT/n6MOK+lzXO0JWnkeNf0LR1HquqPJ6bI1/nex2mzX1zAj\nXmRtK5G9U0Rs1soJTilWpSR3W/dUzOMGXE22Dh8+LD4+PpKcnFwguVuxYoV06dKlQJtu3brJ8uXL\nRcSe3L344ouOsnfeeUf69evn2NY0TU6cOFHofEV58803ZdiwYY7tsLAw2bFjh4iIvP/++3LHHXcU\n+zo6dOggX331lYjYk7tmzZo5yrKyskTTNLlw4YKcOXNGnJ2dJTMz01E+ffp0GTdunIgUTO5eeOEF\nGTlypKNeZmamuLq6lpjcXVs/IyNDnJyc5OzZs473IjY21lE+YsQIefXVV0VEpF+/fvLhhx86yqxW\nq+j1eklISCi17R133CGLFy92lB0/flxcXFzEarUWSu6MRqN88cUXkp2dXex7eTMo6XPsXFrPnqZp\nemAq0FBEHtE0rRnQQkTWVWQPoqIotU8LkyeXsnP55fd02tY3lL2hZ0Po+wNsGww/jIRuy8HJvfIC\nVSrfbdW/3FybNm0YOHAgL7/8Mq1atXLsT0pKIiwsrEDdsLAwEhP/GMEdGBjoeK7X68nIKNs9pXFx\ncUydOpW9e/eSnZ1NXl4eERERjvL77ruP6OhoevbsyapVqwpcLv3oo4/417/+xenTpwHIzMwkOfmP\neSGvjcnDwwOAjIwMkpOTMZlM6PX6Aq9n3759heJLSkoiNPSPicT1ej2+vr4lvqZr63t6emIymUhK\nSiIkJASAgICAAse7+l7Fx8fz5JNP8vTTTwP2ziNN00hMTHQcs7i2f/43CgsLIy8vjwsXLhSITa/X\ns3r1ahYsWMBDDz1Ez549WbhwIS1atCjxNd1syjJadimQi31QBUAiML/SIlIUpdbQNI1OQT6cTb9M\nUsbl62vsaoTbNwECW+6C3JRKiVG5ucyZM4f333+/QOIWHBzsSKCuSkhIcCQr5TFx4kRatWrFiRMn\nSE1N5cUXXyww6GDkyJF8/vnnJCQksGvXLoYPH+44/yOPPMI777xDSkoKKSkptGnTpkwDFoKDgzGb\nzWRm/jGpeHGvJygoiDNnzji2s7KyuHTpUonHv7Z+RkYGZrO5TO9VaGgo7777LmazGbPZTEpKChkZ\nGXTt2rVMryk+Pt6xHR8fj4uLS4Fk8Ko777yTTZs2cf78eVq0aMH48eNLPf7NpizJXbiIvAZcARCR\nLECNSVYUBQA3Jx2dg33Yfz6NzNy862vs5A49PgFjR/imJ2QmlN5GUUoQHh7Offfdx6JFf6yc2b9/\nf+Li4vjkk0+wWq2sXr2ao0ePMmjQoHKfLz09HYPBgF6v59ixYyxevLhAeYcOHfD19eXvf/87d999\nNwaDvYc7MzMTnU6Hn58fNpuNpUuXcvjw4TKds0GDBnTv3p3p06eTk5PDwYMHWbJkSYFewavuvfde\n1q1bR2xsLFeuXGH27NmlJpDr168nNjaW3NxcZs2aRbdu3QgODi41rkcffZSXXnqJI0eOAJCWlsbn\nn39eptc0cuRIRy9mRkYGM2bMICoqCp3OnqZcjfnixYt89dVXZGVl4eLiQr169XBycirTOW4mZUnu\ncjVN8wAEQNO0cNTkxYqiXMPXw5XmJk92nUvFarvOqRI0HUT8C8L/Dt/0gJSfKydIpc7685Qns2fP\nJisry7HfZDKxbt06Fi5ciJ+fHwsXLiQmJgaj0Vhk+9KOf+32woULWblyJQaDgQkTJhAVFVWo/ahR\no9i8eTOjR4927GvVqhVPP/00Xbt2JTAwkF9++YWePXuWOY7o6GhOnTpFcHAww4cPZ968eUXO69e6\ndWv+85//MHLkSIKDg/H19aVBgwYlnmfUqFHMmTMHX19f9u/fz8cff1ym92LIkCE899xzREVFM9uR\nyQAAIABJREFU4ePjQ7t27QqM8C2p7UMPPcSYMWPo3bs34eHh6PX6Agn61bo2m4033niDkJAQ/Pz8\n2L59e6GEWinb8mN9gRlAa2AT0AMYKyJbKz26UqglcRSl5hARdiWl4O7sRIeAG1xqLP5T2DsZekRD\n4B0VG6DioJYfU4ozbtw4QkNDeeGFF6o7FKUU5Vp+TEQ2AcOAsUA00KkmJHaKotQsmqbRMdCHC5k5\nnLVkl96gKGEjoOdnEDsKTq2s2AAVRVFuEqUmd5qmfYw9uTshIutEJLm0Noqi3JxcnXR0DjZy4KKF\n9Ou9/+6qgNvg9u/g53/CkVdB9RgpSpVRy3zVDWW5LBsJ9Mp/hGNfW3a7iLxV+eGVTF0qUJSa6WRq\nJidTsogM88NJd4O/LLISYWs/8O9tX59Wp26arijqsqyi1H4lfY5LTe7yD+AE3ApEAo8C2SLSskKj\nvAHqC0dRaiYRYc+5VJx0GhGBPjd+oNw02DEMXLztS5c5e1RckDcxldwpSu1XrnvuNE3bDPwA3Acc\nB26tCYmdoig1l/3+O2/M2bnEp2Xd+IFcvaHPBnDygO/ugJyS5+dSFEVRyjYVykHskxjfArQDbsmf\nGkVRFKVYzjr7/XeHfk/nUnbujR/IyRW6r4D6vWFTd8g4VXFBKoqi1EFluiwLoGmaF/YRs/8AAkXE\nrRLjKhN1qUBRar7zGZfZez6NzkE+1Pcs59fGr/+BX16C274CU0Tp9ZUiqcuyilL7lfey7GRN01Zj\nH0gxGPgQ6FcBQd2radphTdOsmqZ1vGZ/mKZpWZqm/ZT/eKe851IUpfoE1nOnS7APu8+lcu56lyj7\ns+aPQae3YUs/SPq69PqKoig3Iecy1HEH3gD2icgNzm1QpEPAUODdIsp+E5GORexXFKUW8te70T3E\nyM7EFNoHCA28ynFnR+hQcA+wD7Ro/zKEj6u4QBWlAtW2CYG9vLw4dOgQjRo1qu5QlHIqyyTGC0Vk\nVwUndojIcRGJo+h1atVEO4pSx5g8XOnRwMTPFyzlG2QB4N8d/roNDs+DQy+oufBuco0aNUKv12Mw\nGAgKCmLcuHFkZZXz/9h1Wr58Ob169arSc1a09PR0ldjVEWUZUFEdGuVfkt2iaVrJi+0pilJr+Li7\n0DvUlyPJ6ZxIySzfwQwtoG8snP0Sdk8AW4X+/anUIpqmERMTg8Vi4aeffmLv3r3Mnz+/UL3KvG9Q\nRNQEwEqNUanJnaZp32iadvCax6H8n4NKaJYENMy/LPs0sErTtHrFVZ4zZ47jsXXr1gp+BYqiVDQv\nN2d6h/oSl5LJr5cyyncwj0B7D15WAmwfAnnlTBjrqK1btxb4rqyLriZuQUFB9OvXj0OHDhEZGcnM\nmTPp2bMnnp6enDp1inPnzjF48GB8fX1p3rw5H3zwgeMYe/bsoXv37hiNRkJCQnj88cfJy/vjjwad\nTse7775L8+bNMZlMTJ48GYBjx44xceJEdu7ciZeXFyaTydHGbDYzcOBADAYD3bp149SpP0Z7x8bG\n0rlzZ4xGI126dGHnzp2OspSUFB566CFCQkLw9fVl2LBhALRt25aYmBhHvby8PPz9/fn5558BGDFi\nBEFBQRiNRvr06cORI0ccdceNG8fkyZOLjUen03Hy5Mky1Z0yZQoBAQF4e3vTvn37AudRagARqdYH\nsAXoeCPl9vAVRamNMnPzZOPJC/LL7xax2WzlO5g1V2TnWJENt4pkX6iYAOuw/O/O6/merp5Ay6hR\no0ayefNmERFJSEiQNm3ayOzZs6VPnz4SFhYmR48eFavVKleuXJHevXvL5MmTJTc3Vw4cOCD+/v6y\nZcsWERHZt2+f7Nq1S2w2m8THx0vr1q3lrbfecpxH0zQZNGiQWCwWSUhIEH9/f9m4caOIiCxbtkx6\n9epVIK6xY8eKn5+f7N27V6xWq4wePVpGjhwpIiJms1mMRqOsXLlSrFarREdHi9FoFLPZLCIi/fv3\nl6ioKElLS5O8vDzZvn27iIi89tprct999znO8eWXX0q7du0c20uXLpXMzEzJzc2VKVOmSIcOHcoU\nj4iITqeTEydOlFp348aN0qlTJ7FYLCIicuzYMTl//vyN/vMpN6ikz3FZBlRUBUdftqZpfoBZRGya\npjUBmgInqy0yRVEqhd7Fid6hvvxw1kyeTWjr73Xjl7V0LtDlQzj0vH0uvMivwatpxQaslOitlIpZ\nkfJJ45M31G7IkCE4Ozvj7e3NwIED+ec//8n27dsZO3YsLVva591PSkoiNjaWDRs24OLiQvv27fn7\n3//ORx99RJ8+fejY8Y9xfA0bNuSRRx5h27ZtPPHEE47906dPx8vLCy8vLyIjIzlw4AB9+/YtNq6h\nQ4cSEWGftmf06NE8/fTTAMTExNC8eXNGjRoFQFRUFIsWLWLt2rX07duXr7/+mpSUFAwGA4Djfr77\n77+f+fPnk5GRQb169fj4448ZM2aM43xjx451PJ89ezZvvvkm6enpeHl5lRgPFL5sXVxdFxcX0tPT\nOXLkCJ07d6ZFixYl/+MoVa7akjtN04YA/wb8gHWaph0QkX5Ab+AFTdNyARswQURSqytORVEqj7uz\nE73yE7z9F9L4S4D3jSd4mgbtXgB9A/imF/T+Evy6VGzASrFuNCmrKGvWrCEyMrLQ/tDQUMfzpKQk\nTCYTer3esS8sLIx9+/YBEBcXx9SpU9m7dy/Z2dnk5eU5kpurAgICHM/1ej0ZGSXfWhAYGFhk/aSk\nJMLCwgrUDQsLIzExkTNnzuDr6+tI7K4VFBREjx49+O9//8uQIUPYsGEDixYtAsBms/HPf/6Tzz//\nnOTkZDRNQ9M0kpOTHcldcfFcT+yRkZFMnjyZxx57jISEBIYNG8bChQupV6/YO6iUKlZtAypE5EsR\nCRURDxEJyk/sEJEvROQWEekoIp1EZH11xagoSuVzddLRM9RERq6VvedSsZX3pvemj0CX92HbQDi7\ntmKCVGq8P/c6XXXtHwvBwcGYzWYyM/+4NzMhIYGQkBAAJk6cSKtWrThx4gSpqam8+OKLZR6Ecb1/\nlAQHB3P69OkC+67GEhoaitlsxmKxFNn2gQceYMWKFXz22Wd0796doKAgAFatWsXatWv57rvvSE1N\n5fTp09deWq9QkydPZu/evRw5coTjx4+zYMGCCj+HcuNq6mhZRVFuIi46Hd0bmMi1CbuSUrDayvnL\nKGQg3BYDux+BuKKm0lRuRg0aNKB79+5Mnz6dnJwcDh48yJIlSxyXNdPT0zEYDOj1eo4dO8bixYvL\nfOyAgADOnj3LlStXylS/f//+xMXF8cknn2C1Wlm9ejVHjx5l4MCBBAYG0q9fPyZNmkRqaip5eXns\n2LHD0XbIkCH89NNPLFq0iAceeMCxPz09HTc3N4xGI5mZmUyfPr1SRvDu3buX3bt3k5eXh4eHB+7u\n7uh0Kp2oSdS/hqIoNYKzTqNrsBENjZ2J9vvwysWvM9y5A44uhJ9nqrnw6rDiEpii9kdHR3Pq1CmC\ng4MZPnw48+bNc1zOXbhwIStXrsRgMDBhwgSioqJKPN6127fffjtt2rQhMDCQ+vXrlxqzyWRi3bp1\nLFy4ED8/PxYuXEhMTIxjpO2KFStwdnamZcuWBAQE8NZbf9zT6O7uzvDhwzl16pRjFC3Ye/QaNmxI\nSEgIt9xyC927dy81jpJeX3EsFgvjx4/HZDLRuHFj/Pz8mDZt2nWdS6lcZV5btiZS6x0qSt1jE+Gn\n82lkXrHSPcSIi1M5/wa9/Lv9Eq1HMLR/CbxbVUygtZhaW7b2mzdvHnFxcXz00UfVHYpSTcq1tqyi\nKEpV0mkaEYHeGFyd+f6smVyrrXwHdPeHO7aAb2f49jb4YTRYjldMsIpSDcxmM0uWLGHChAnVHYpS\nQ6nkTlGUGkfTNDoEGPDzcGV7wiUu51nLd0BnPbSZDvf8Bt6t4ZueEDsGLL9WTMCKUkU++OADGjZs\nyIABA+jRo0d1h6PUUOqyrKIoNZaIcOxSBmcs2fQM9UXv4lQxB75igeOL4PhbENQPbpkFhmYVc+xa\nQF2WVZTar6TPsUruFEWp8eLMGZxIzaJnAxP1XCtwes7cNHuS9+siCB4At8y8KSY/VsmdotR+KrlT\nFKXWO5maybFLGfRsYMLg5lKxB89Ntffi/fpvCBkEbWaCV3jFnqMGUcmdotR+KrlTFKVOSEjL4tDv\n6fRoYMLHvYITPLAnecfehLi3IeQee09evSYVf55qppI7Ran9VHKnKEqdkZiezYELFrqGGPH1cK2c\nk+SmwLF/wa//gdCh0GYG1GtcOeeqBiq5U5TaTyV3iqLUKeczL7PvXBq3BvlQ39Ot8k6UY7YneXHv\nQOiw/CSvUeWdr4qo5E5Raj81z52iKHVKoKc7nYN92HMulXMZlyvvRG4maD8PBsWBewB8HQG7J0Bm\nfOWdU6lS27ZtIzQ0tMKO179/f1asWFFhx6sIEydO5MUXX6zuMJQqpJI7RVFqJX+9G91CjPx0Po2z\n6dmVezI3E7SfD4N+BVdf2NARdj8KmQmVe16lzFatWsWtt96Kl5cXISEhDBgwgB9++KFMbSty/dX1\n69c71qotTWRkJB9++GGFnbs4ixcvZsaMGUDFJ7NKzaSSO0VRai2Thys9Gpg4eMFCfFpW5Z/QzRc6\nvAQDj4OrETb8BXZPhMwzlX9upVhvvPEGU6dOZebMmVy8eJGEhAQee+wx1q5dW6nntdnKuXpKNRCR\nCk1mlZpJJXeKotRqPu4u9Ar15UhyOidSMqvmpO5+0OFlGHgMXL1hQwfY8xhkna2a8ysOFouF559/\nnnfeeYfBgwfj4eGBk5MT/fv355VXXgEgNzeXp556ipCQEBo0aMCUKVO4cuVKkcc7duwYkZGRGI1G\n2rZtWyBBHDduHJMmTWLAgAF4eXmxdevWQu2v7Y1bvnw5vXr1Ytq0aZhMJsLDw9m4cSMAM2fOZMeO\nHUyePBmDwcATTzzhOH/fvn3x9fWlVatWfPbZZwXOP3nyZAYOHIjBYKBbt26cOnXKUT5lyhQCAgLw\n9vamffv2HDlyxNFu9uzZZGVl0b9/f5KSkvDy8sJgMHDu3Dk8PT1JSUlxHOenn36ifv36WK3lXBlG\nqTYquVMUpdbzcnOmd6gvv6Vk8uuljKo7sbs/dHjFnuQ5e8L6drBnMmQlVl0MN7mdO3eSk5PDkCFD\niq0zf/58du/ezcGDB/n555/ZvXs38+fPL1QvLy+PQYMGcffdd/P777+zaNEiRo8eTVxcnKNOdHQ0\ns2bNIj09nZ49e5Ya3+7du2nVqhWXLl1i2rRpPPTQQ46YevXqxdtvv43FYmHRokVkZWXRt29f7r//\nfpKTk/nkk0+YNGkSx44dcxxv9erVzJ07l9TUVMLDwx2XWzdt2sT333/Pb7/9RlpaGp9++im+vr4F\nYtHr9WzYsIHg4GDS09OxWCwEBQURGRnJp59+6qj38ccfM3LkSJycKmhFGKXKVeBU74qiKNXH09We\n4H1/9hJ5IrTyrVd1l5/c/eEvr0Grf8DRBfYkr9FoaP0c6IOrJoZq9sXxcxVynGEtgq6r/qVLl/Dz\n80OnK76vYtWqVfznP/9xJDvPP/88jz76KHPnzi1Qb+fOnWRmZvLss88C9l64gQMHEh0dzezZswEY\nPHgwXbt2BcDVtfSpeMLCwhwJ3YMPPsikSZO4ePEi9evXL1R33bp1NG7cmAceeACA9u3bM3z4cD77\n7DNmzZoFwNChQ4mIiABg9OjRPP300wC4uLiQnp7OkSNH6Ny5My1atCg1tqseeOABFi1axIQJE7DZ\nbERHR1f6JW2lcqnkTlGUOsPDxYleob78cNZMnk1o6+9VtfcXudeHvyyAlv+Ao6/B+lug0Rho8xx4\nXF/SUttcb1JWUXx9fUlOTsZmsxWb4CUlJdGwYUPHdlhYGElJSYXqnTt3rtBgg7CwMBIT/+iJvd7B\nCIGBgY7nHh4eAGRkZBSZ3MXHx/Pjjz9iMpkA+/1xVqvVkez9+Xh6vZ6MDHtPdWRkJJMnT+axxx4j\nISGBYcOGsXDhQurVq1dqjIMHD2bixInEx8dz9OhRfHx86NSp03W9TqVmUZdlFUWpU9yd7Qnepexc\n9l9Io1rmZ/MIgI6vw4AjoDlBTBvY9xRkV0zvlvKHbt264ebmxpdffllsnZCQEOLj/5i+Jj4+nuDg\nwj2qwcHBnDlTcHBMQkICISEhju2K/GPhz8cKDQ2lT58+mM1mzGYzKSkpWCwW3n777TIdb/Lkyezd\nu5cjR45w/PhxFixYUOo5Adzc3BgxYgQrVqzg448/LvNoX6XmUj13iqLUOa5OOnqGmth5NoW951KJ\nCPJBVx0jBD0CIeINaD0NjrwG61qCsxc46+336Dnp//S8qH1/KnfW/+n51Z/uoN18f68bDAbmzp3L\nY489hpOTE3379sXFxYVvvvmGbdu28corrxAVFcX8+fMdvVHz5s0rMoHp0qULer2e1157jalTp/L9\n99+zbt065syZUymxBwQEcPLkScf2wIEDmT59Oh9//DFRUVGICD///DNeXl6lXmbdu3cvNpuNjh07\n4uHhgbu7e5E9mQEBAVy6dAmLxYLBYHDsHzNmDA888AC///47L7/8csW9SKVaqOROUZQ6yUWno0cD\nEz8mpbArKYXOQUacdNU0BYRHEET8yz5XXo4ZrFmQlwV5mfnPM+3b1mv3ZUHOpaLLi6prvQxOHoWT\nv2uTxKuJYB0zdepUgoKCmD9/Pvfffz9eXl5EREQ4BhvMnDmT9PR02rVrh6ZpjBgxwlF2LRcXF9au\nXcvEiRN56aWXaNCgAStWrKBZs2ZA2XrtSqtzbfmTTz7Jgw8+yOLFixkzZgxvvvkmmzZtYsqUKUyd\nOhURoX379rzxxhulntdisTBlyhROnTqFu7s7d911F9OmTStUr0WLFowcOZImTZpgs9k4cuQIgYGB\ndO/eHZ1OR8eOHdU8eHWAWn5MUZQ6zSbC7qRUMq/k0cTHkwZe7rg41cEeLrGBNbvkRDAvC6yZaM0n\nqeXHlELuuOMORo8e7RgAotRsam1ZRVFuaiLChcwcTqdl83tWDkH13Anz9sDPw/WmnNBVrS2r/Nme\nPXu46667OHPmDJ6entUdjlIGJX2Oq+2yrKZprwGDgBzgBDBORCz5ZdOBh4A84EkR2VRdcSqKUvtp\nmkZgPXcC67mTk2fljOUyP1+wYBUhzNuDhgY9ehc1p5dycxo7dixr1qxh0aJFKrGrI6qt507TtL8C\n34mITdO0VwARkemaprUGVgK3Ag2Ab4FmRf3ZqP6aVBTlRokIqTlXOJ2azdn0bIzurjTy9iConnv1\n3ZtXRVTPnaLUfjWy505Evr1m80dgeP7ze4BPRCQPOK1pWhzQGdhVxSEqilKHaZqG0d0VY6Arbesb\nSMq4zKm0LA5cTCPUy4Mwbz0+7i7VHaaiKMp1qymjZR8CovOfhwA7rylLzN+nKIpSKZx1Gg0NHjQ0\neJCZm0e8JZudiSm4Omk08tYTavDAtS4OwlAUpU6q1ORO07RvgIBrdwECzBCRtfl1ZgBXRCS6iEOU\n6tr5h/r06UOfPn1uNFxFURQ8XZ1p7edFK996XMzKJT4tiyPJ6QR4uhHmrae+vvYNwti6dWuRi9wr\nilI3VetoWU3TxgLjgdtFJCd/33PY7797NX/7a+B5ESl0WVbdB6IoSlXItdo4Y8kmPi2LHKuNMG89\nYQYPPF1rysWP66PuuVOU2q9GToWiadrdwOtAbxG5dM3+qwMqumC/HPsNakCFoig1ROrlK8SnZXEm\n/TLebs6EGTwI9vLAuRYNwlDJnaLUfjU1uYsDXIGrid2PIjIpv2w68DBwhRKmQlFfOIqiVBerTTiX\neZn4tGzM2bk08PIgzNsDo7tLjb9sq5I7Ran9amRyVxHUF46iKDVB1hUrCZYs4tOycdK0/LnzPHBz\nrt6580QEqwi5ViHXauOKzUau1UYDg75OJXeNGzdmyZIl3H777Y59y5cv54MPPmDHjh3VGJmiVJ4a\nORWKoihKXaF3caKlrxctTPVIzs4lPi2bTad+x1/vRpi3BwGebujK0ZtnE+GKVci12bhitSdouTa5\n5rnNXp6/fcX2x3OdBi5OOlx1OlyddLg41exexYpU03tQFaWyqOROURSlgmiahr/eDX+9G1esNs6m\nX+b4pQz2n0+jYX5vnrNOV6AXLdean6TZ/vz8jyTNahNcdJo9SXPS4aLT4eqk5SdrOvTOTri66a5J\n4jTH87o+IXNZ6HQ6fvvtN5o0aQLAuHHjCA0N5YUXXgBg3bp1zJo1i9OnT9OmTRsWL15M27ZtqzNk\nRSkXldwpiqJUAhcnHY199DT20WPJuUJ8WjY/nDXby/J70a5NwlycdOhdNMdzVycdrvkJnYtOU71Q\n16msl5H379/Pww8/TExMDBEREXz88cfcc889/Prrr7i4qEmsldpJJXeKoiiVzODmQtv6LrStb6ju\nUCrN3LlzK+Q4zz///A21GzJkCM7Of/xKy8nJISIiotR277//Po8++iidOnUCYMyYMbz44ov8+OOP\n9OrV64ZiUZTqppI7RVEUpdxuNCmrKGvWrCEyMtKxvXz5cpYsWVJqu/j4eD766CP+/e9/A/YevytX\nrpCUlFRpsSpKZVPJnaIoilLrlXQZVq/Xk5WV5dg+f/48oaGhAISGhjJjxgymT59e6TEqSlVRiyUq\niqIodVqHDh1YtWoVNpuNr7/+mm3btjnKxo8fz//93/+xe/duADIzM1m/fj2ZmZnVFa6ilJtK7hRF\nUZRarbTBJm+99RZfffUVRqOR6Ohohg4d6iiLiIjg/fffZ/LkyZhMJpo3b87y5csrO2RFqVRqEmNF\nUZSbjFqhQlFqv5I+x6rnTlEURVEUpQ5RyZ2iKIqiKEodopI7RVEURVGUOkQld4qiKIqiKHWISu4U\nRVEURVHqEJXcKYqiKIqi1CEquVMURVEURalDVHKnKIqiKIpSh6jkTlEURVEUpQ5RyZ2iKIqiKOV2\n8eJFevfujbe3N9OmTSu1/vLly+nVq5dj28vLi9OnTwMwbtw4Zs+eXVmhFrBt2zZCQ0PLdYwzZ85g\nMBioKSu5qOROURRFqfWWLVtGu3bt8PT0JDg4mEmTJpGWluYonzt3Lq6urnh7e+Pt7U3Lli15/PHH\nOX/+vKPOrl276Nu3L76+vgQEBHDfffcVKM/NzeXRRx8lMDAQPz8/Bg8ezLlz5xzl8fHx3H777Xh6\netK6dWs2b95cKM5HH32UDz74AIDExETuv/9+/Pz88PLyomvXrsTExBSor9Pp8PLywmAw4O/vz513\n3smnn35aoM6zzz5Lw4YN8fb2pnHjxrzyyisFyidMmEDLli1xcnLio48+KlC2fPlynJ2dMRgMjvNs\n3769rG97Ae+99x7169cnLS2NBQsWlKnNtesCp6en06hRo+s+b0UkgqWtT1ya0NBQLBZLuY9TUVRy\npyiKotRqr7/+OtOnT+f111/HYrHw448/Eh8fz5133kleXp6jXlRUFGlpaZjNZv73v/9x/vx5IiIi\nuHDhAgApKSlMmDCB+Ph44uPjqVevHuPGjXO0f/PNN9m1axeHDx8mKSkJHx8fJk+e7CgfOXIkERER\nmM1m5s+fz7333sulS5cKxLphwwYGDBhASkoKPXv2xN3dnaNHj5KcnMxTTz3FqFGj+OKLLxz1NU3j\n4MGDWCwWjh8/zoMPPsjkyZOZN2+eo87DDz/MkSNHSEtLIzY2lo8//pgvv/zSUd6hQwcWL15MRERE\nke9f9+7dsVgspKenY7FY6N279w39O8THx9O6desbaqtUMBGptQ97+IqiKMr1yP/urBPftRaLRerV\nqyeff/55gf0ZGRni7+8vS5cuFRGROXPmyJgxYwrUsVqt0r59e5k2bVqRx/7pp5/EYDA4tidOnCjP\nPvusYzsmJkZatmwpIiLHjx8Xd3d3ycjIcJT37t1b3n33Xcf2wYMHpX379iIiMnPmTGnbtm2hc776\n6qsSFhbm2NY0TU6cOFGgzueffy7u7u5iNpsLtT979qy0bdtWFixYUKisZ8+esnz58gL7li1bJr16\n9Sry9Rflhx9+kFtvvVV8fHykc+fOEhsbKyIiY8eOFRcXF3F1dRUvLy/ZvHlzobaXLl2SQYMGicFg\nkC5dusisWbMKnPva1zp27FiZNWuWI8aePXsWONbVuu+99564uLiIm5ubeHl5yT333CMiIklJSTJ8\n+HDx9/eXJk2ayKJFixxts7Oz5cEHHxSj0Sht2rSRBQsWSGhoaJGv9/nnn5fHH39cRESuXLkinp6e\n8swzzziO4+7uLikpKXL69GnRNE2sVquIiPTp00dmzZolPXr0EC8vL7nrrrvk0qVLjuPu3LlTunfv\nLj4+PtKhQwfZunWro2zp0qXSpEkT8fLykiZNmsiqVauKjK2kz7FzdSWVmqa9BgwCcoATwDgRsWia\nFgYcBY7lV/1RRCZVU5hlsnXrVvr06XPTx1BT4qgJMdSUOGpCDDUljpoQQ02Ko6IN1OaVXqkM1sms\n66ofGxtLTk4OQ4cOLbDf09OT/v3788033zB27Ngi2+p0OgYPHsymTZuKLN+2bRtt2rRxbD/88MM8\n+eSTnDt3Dm9vb1auXEn//v0BOHLkCE2aNMHT09NRv3379vzyyy+O7fXr1zNgwAAAvv32W4YPH17o\nnCNGjOC5554jLi6OZs2aFRnX4MGDycvLY/fu3dx1110AvPrqq8yfP5/MzEyaNGnCqFGjimxblP37\n91O/fn1MJhP3338///znP9HpCl/YS0lJYeDAgbz99ttERUXx6aefMmDAAE6cOMHSpUsB++XJF154\nocjzTJo0Cb1ez4ULFzhx4gR33XUXTZo0cZSXdEnzz2VXt8ePH09sbGyB84oIgwYNYujQoaxevZoz\nZ87w17/+lZYtW3LnnXcyZ84cTp06xalTp8jIyODuu+8u9ry33XYbTz31FAB79uwhMDAylf4JAAAQ\n2ElEQVTQcdk6NjaWli1b4uPjQ1paWqEYo6Oj+frrr2nQoAF33303Cxcu5KWXXiIxMZGBAweycuVK\n7rrrLjZv3szw4cM5fvw4Hh4ePPnkk+zbt4+mTZty4cIFzGZzsfEVpzovy24C2ohIByAOmH5N2W8i\n0jH/UaMTO7B/WVe3mhAD1Iw4akIMUDPiqAkxQM2IoybEADUnjoq2TmZVyON6JScn4+fnV2QyEhQU\nRHJycontg4ODi/zlefDgQebNm8fChQsd+5o1a0ZoaCghISH4+Phw7NgxZs2yx5yRkYG3t3eBYxgM\nBtLT0x3bMTExjuQuOTmZoKCgImO+Wl4cZ2dn/Pz8CsT97LPPkp6ezv79+xkzZkyhWIpz2223cfjw\nYS5evMh///tfoqOji71fLiYmhubNmzNq1Ch0Oh1RUVG0bNmStWvXlnoem83GF198wbx583B3d6dN\nmzY8+OCDBerIdQxGKKnunj17SE5OZsaMGTg5OdGoUSP+/ve/88knnwDw2WefMXPmTLy9vQkJCeGJ\nJ54o9ljdunUjLi6OlJQUtm/fzsMPP0xiYiJZWVls376d2267rdi248aNIzw8HDc3N0aMGMGBAwcA\nWLlyJQMGDHAk5nfccQedOnVi/fr1ADg5OXHo0CEuX75MQEAArVq1KvP7clW1JXci8q2I2PI3fwQa\nXFNcoXcklvXLtLh6Re2/kS/oio6jJsRQU+KorTHUlDhqQgw1JY6aEENNiqOm8/PzIzk5GZvNVqjs\n3Llz+Pn5ldg+MTERk8lUYN9vv/1G//79+fe//0337t0d+ydNmkROTg4pKSlkZmYy9P/bu//gKOo0\nDeDPO0UAh0nCJEImIT84YYEqWEWIeAGEmRgVMRxyWBw5EA2HrhRF1d2einJUgENRqXhXW+sWxwne\nqkD4oaWpKDHZKpZQYiEprwRFRE5ikk0CxkyEQBLiJM/9MUNvApMQMpN0k7yfqi5mur/T/UzT0/Om\ne77dCxYYR30cDgcuXrzYYT4XLlxAZGSk8fj06dNIS0szcrfvjNE+MwCMGDGi08w+nw+1tbXX5Qb8\nRwuHDh3a7Q4Go0ePRkpKCgBg4sSJyMnJwXvvvRe0bXV1tdH2qpSUFFRVVd1wObW1tWhtbUVi4l+/\n6q+dV7iUl5cb/68xMTFwOp145ZVX8OOPPwLwv4/u5hg6dChSU1Nx6NAhHD58GG63G9OnT8enn36K\nkpKSLos7l8tlPLbb7bh06ZKRb9++fR3yHTlyBDU1NbDb7di7dy+2bt2K+Ph4zJs3D6dPn77pdWCV\nDhXLARS2ez5aRP5XRP4sIjNDnbkWNL2XwSo5btUMVslhhQxWyWGFDFbKYXVpaWkYMmRIh04IgP9I\nWmFhITIyMjp9LUkUFBR06EBwtSPG+vXrrzu1efz4cWRnZyM6OhoRERFYvXo1jh07Bq/Xi4kTJ+Ls\n2bO4fPlyh/ZXT+sWFRUhPT3dOHWXkZFxXWYA2Lt3L5KTkzF27NhOc3/44YeIiIjAtGnTgk73+Xw4\ne/Zsp6+/kc6OiiUkJBiXKrmqoqICo0aNuuE8R4wYgUGDBqGysrLDa7tj2LBhaGxsNJ6378EMXH/K\nNikpCXfccQe8Xi+8Xi/q6+tx4cIF4whjQkJChxzl5eVdLn/WrFk4ePAgvvzyS9xzzz2YNWsWioqK\nUFpa2qPOJ0lJSVi2bFmHfA0NDXj++ecBAA888ACKi4tx7tw5jB8/Hk899dRNL6O3Ozz8CcCJdsNX\ngX/ntWvzbwDeb/c8AoAz8HgKgAoAjk7mTx100EEHHW5+uMl9edAfdFvFli1b6HK5+Mknn/CXX35h\nWVkZ586dy9TUVLa0tJD0d6hYunQpSdLn8/Gbb77hokWLGB8fz5qaGpL+zghjxozh66+/HnQ52dnZ\nfOyxx3jhwgW2tLTw5ZdfZmJiojE9LS2Nzz33HJubm/n+++/T6XTyp59+Ikk+8cQTfPfdd422dXV1\nTElJ4fLly3nu3Dk2Nzdz9+7djI6O5v79+4127TsZeL1e7ty5k3FxcdywYQNJsq2tjdu2bWN9fT1J\n8vPPP2d8fDzfeOMNYx4tLS1samrijBkz+Oabb7K5uZltbW0kycLCQp4/f54keerUKU6aNImbNm0K\n+v7r6urodDqZl5dHn8/HPXv20Ol0Gh0F2neCCGbx4sXMyspiY2MjT548ycTExG51qPjuu+84dOhQ\nHj9+nM3NzXzmmWdos9mMti+88AKXLFlizKe1tZVTp07la6+9xqamJvp8Pn799dcsLS0lSa5Zs4Zu\nt5v19fWsrKzknXfe2WmHCpIsLi5mVFQUMzIySJInT55kVFQUJ02aZLQJ1qFix44dxvT2HVcqKysZ\nHx/PoqIitra2sqmpiYcOHWJVVRXPnz/P/Px8Xr58ma2trVy/fj3dbnfQXF19js3u7fokgCMAhnTR\n5s8AppiZUwcddNBhIA9WL+5I8q233uKkSZNot9vpcrm4cuVK/vzzz8b0DRs2GD05HQ4Hx40bx1Wr\nVrG6utpos3HjRtpsNkZGRhrtIiMjjel1dXVcsmQJR44cSafTyfvuu88oGEiyvLycbrebt912GydM\nmMCDBw8a01wuF2traztkrqysZFZWFmNiYuhwODht2jQWFBR0aGOz2YwcsbGxTE9P5549e4zpbW1t\nnDNnDmNjYxkZGcnx48fz1Vdf7TAPt9tNEaHNZjOGkpISkuSzzz7LuLg4OhwOjhkzhhs2bKDP5+t0\nPR85coRTp07l8OHDmZqaavSWJf3Fb1fFXW1tLTMzMxkdHc17772XOTk5HYq79gXbtYXi5s2befvt\ntzM5OZm7du3q0PbMmTOcPHkynU4nFyxYQJKsqalhVlYWXS4XY2JimJaWZvTgbWxs5LJlyzh8+HBO\nnDiRubm5XRZ3ly5d4uDBgzsUvXFxcVy1apXx/IcffqDNZjOKO4/H02lxR5LHjh3j7NmzGRMTw5Ej\nRzIzM5OVlZWsqanh7NmzOXz4cDqdTno8Hp46dSporq6KO/FP73siMgfA6wBmkaxrN/52AF6SbSJy\nB4ASAL8m+bMpQZVSaoATEZr1XdEflJaWYvXq1Th69KjZUVQ/IiIgGbSPgmmXQgHwewCDAfwpcL78\n6iVPZgH4dxFpAdAG4Dda2CmllLqVbdy40ewIagAx7cidUkqpW4MeuVPKero6cmeV3rJKKaWUUioM\ntLhTSimllOpH+mVxJyKzReSwiGwVkZ7dATk8OewiUioic01a/oTAOtgnIs+YkSGQY76I/LeI5InI\nAyZl+BsR2S4i+8xYfiCDXUT+KCLbRKT79wYKfw4rrAvTt4lADqt8RkzdVwQyWGK/qZQKXb8s7uC/\njlMDgCEA/mJijjUA9pq1cJLfklwJ4B8ATL9R+17MkU/yaQArASwyKUMZyRVmLLudvwewn+RvAPyd\nWSGssC6ssE0EcljiMwKT9xUBVtlvKqVCZOniTkR2iMh5ETlxzfg5IvKtiHwnImuufR3JwyQfAfAC\ngOB3MO7lDCKSAeAbALUI8XZqPc0QaDMPwEcADoSSIdQcAesA/MHkDGHTgyyJAK5eFr3VxBxhF0KG\nkLeJUHOE8zPSkwzh3FeEkiOc+02llMk6uwCeFQYAMwFMBnCi3TgbgP8DkAL/3Sy+BDAhMO1xAP8B\nID7wfDCAfSZk+E8AOwJZigB8YOZ6CIz7yMT/jwQArwJIt8A2sd/E7XMJgLmBx7vNytGujWnrIjA9\nLNtEONZFoF3In5EebhcvhWtfEabt4rr9Jm6BixiHQ/u7WFRUVDAyMtK4k4NSVoMuLmJs6SN3JD8F\nUH/N6GkAzpAsJ/kLgD0A5gfav0vytwD+VkT+C8DbAN4wIcO/kPynQJZdAN40IcNvAYwTkd8F1sXH\noWQIMcdCAPcDeExEnjYpwxUR2QpgcriOYt1sFgAfwL8O/gCgIBwZepJDRGLMXhcishph2iZCzDE7\nnJ+RnmQguS5c+4pQcojIgnDtN/va6NGjYbfbERUVhfj4eGRnZ3e4F+nNkMB9SpOSknDx4sXr7luq\n1K3AzIsY99Qo/PXUFuD/bUiHuyeT/AD+L1LTMrTL8o5ZGUiWwH+Hj97UnRy/h/+i1WZm8ML/+67e\n1mkWko0AlvdBhhvlsMK66O1tors5+uIz0mWGq3pxX9GtHH2w3+w1IoKPP/4YHo8HNTU1ePDBB/HS\nSy9h8+bNHdqR1GJNDQiWPnKnlFJKdYf/LBUQHx+Phx9+GF999RU8Hg/WrVuHmTNnYtiwYSgrK0NN\nTQ3mz5+P2NhYjBs3Dtu3bw86v/LycthsNrS1tQEAPB4PcnJyMHPmTERFRWHOnDnwer1G+6NHj2LG\njBlwOp24++67UVLSF38zKBXcrVjcVQFIbvc8MTBOM/R9BqvksEIGq2WxQg4rZLBKDitksFKOXlNZ\nWYkDBw5gypQpAICdO3di+/btaGhoQHJyMhYvXozk5GScO3cO+/fvx9q1a3Ho0KGg87r2KF9eXh7e\nfvtt1NbW4sqVK8jNzQUAVFVVITMzEzk5Oaivr0dubi4WLlyIurq6YLNVqtfdCqdlBR17kJUCGCsi\nKQBqACwGkKUZ+iSDVXJYIYPVslghhxUyWCWHFTL0bY7dYTrd+Y89u83Zo48+ikGDBiE6OhqZmZlY\nu3YtDh8+jCeffBITJkwAAFRXV+Ozzz5DYWEhIiIicNddd2HFihV455134Ha7b7iM7OxsjBkzBgCw\naNEiFBT4fzq7a9cuPPLII3jooYcAAPfffz9SU1Nx4MABPP744z16P0qFwtLFnYjsBuAGECsiFQDW\nk/yfwI+xi+E/8riD5CnN0LsZrJLDChmslsUKOayQwSo5rJDBlBw9LMrCJT8/Hx6P57rxSUlJxuPq\n6mrExMTAbrcb41JSUvDFF190axkul8t4bLfbcenSJQD+U7j79u0zij2S8Pl8SE9P79F7USpUli7u\nSAa9ij/JQgCFmqHvMlglhxUyWC2LFXJYIYNVclghg5Vy9JWrv7m7VvtTqwkJCfB6vbh8+TKGDRsG\nAKioqMCoUaNCWnZSUhKWLVuGbdu2hTQfpcLlVvzNnVJKKXXTEhMTMX36dLz44ou4cuUKTpw4gR07\ndnR66rSzgvFaS5cuRUFBAYqLi9HW1obm5maUlJSguro6nPGV6jYt7pRSSt3SOru8SbDxeXl5KCsr\nQ0JCAhYuXIhNmzYFPZ177eu7uoRKYmIi8vPzsXnzZowYMQIpKSnIzc01etoq1deku3+ZKKWUGphE\nhPpdoZS1iAhIBv2rQ4/cKaWUUkr1I1rcKaWUUkr1I1rcKaWUUkr1I1rcKaWUUkr1I1rcKRVmIhIn\nInkickZESkXkIxEZa3YupXrKZrO1NTU1mR1DKRXQ0tICEem0l5MWd0qF3wcADpL8Fcl7ALwIIM7k\nTEr1WGRk5GcLFixo/P777+Hz+cyOo9SA1tLSgi1btvgcDse3nbXRS6EoFUYi4oH/Nk9us7MoFS4i\nMsRut28C8HRTU1NUZ5dfUEr1PhGhw+H4tqGh4UGSfwnaRos7pcIncN/O0ST/1ewsSimlBiY9LauU\nUkop1Y9ocadUeJ0EkGp2CKWUUgOXFndKhRHJgwAGi8iKq+NE5NciMsPEWEoppQYQ/c2dUmEmIi4A\nvwMwFUATgB8A/DPJ783MpZRSamDQ4k4ppZRSqh/R07JKKaWUUv2IFndKKaWUUv2IFndKKaWUUv2I\nFndKKaWUUv2IFndKKaWUUv2IFndKKaWUUv2IFndKKaWUUv3I/wPuDR3YdQVINAAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1fe52360f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = plt.subplot(111)\n",
    "    \n",
    "colors = ['blue', 'green', 'red', 'cyan', \n",
    "          'magenta', 'yellow', 'black', \n",
    "          'pink', 'lightgreen', 'lightblue', \n",
    "          'gray', 'indigo', 'orange']\n",
    "\n",
    "weights, params = [], []\n",
    "for c in np.arange(-4, 6):\n",
    "    lr = LogisticRegression(penalty='l1', C=10**c, random_state=0)\n",
    "    lr.fit(X_train_std, y_train)\n",
    "    weights.append(lr.coef_[1])\n",
    "    params.append(10**c)\n",
    "\n",
    "weights = np.array(weights)\n",
    "\n",
    "for column, color in zip(range(weights.shape[1]), colors):\n",
    "    plt.plot(params, weights[:, column],\n",
    "             label=df_wine.columns[column + 1],\n",
    "             color=color)\n",
    "plt.axhline(0, color='black', linestyle='--', linewidth=3)\n",
    "plt.xlim([10**(-5), 10**5])\n",
    "plt.ylabel('weight coefficient')\n",
    "plt.xlabel('C')\n",
    "plt.xscale('log')\n",
    "plt.legend(loc='upper left')\n",
    "ax.legend(loc='upper center', \n",
    "          bbox_to_anchor=(1.38, 1.03),\n",
    "          ncol=1, fancybox=True)\n",
    "# plt.savefig('./figures/l1_path.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Dimensionality reduction\n",
    "\n",
    "$L_1$ regularization implicitly selects features via zero out\n",
    "\n",
    "Feature selection\n",
    "* explicit - you specify how many features to select, the algorithm picks the most relevant (not important) ones\n",
    "* forward, backward\n",
    "* next topic\n",
    "\n",
    "Note: 2 important features might be highly correlated, and thus it is relevant to select only 1\n",
    "\n",
    "Feature extraction\n",
    "* explicit\n",
    "* can build new, not just select original, features\n",
    "* e.g. PCA\n",
    "* next chapter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sequential feature selection algorithms\n",
    "\n",
    "Feature selection is a way to reduce input data dimensionality. You can think of it as reducing the number of columns of the input data table/frame. However, how do we decide which features/columns to keep? Intuitively, we want to keep important ones and remove the rest.\n",
    "\n",
    "We can select these features sequentially, either forward or backward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Backward selection\n",
    "\n",
    "Sequential backward selection (SBS) is a simple heuristic. The basic idea is to start with n features, and consider all possible (n-1) subfeatures, and remove the one that matters the least for model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.base import clone\n",
    "from itertools import combinations\n",
    "import numpy as np\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "class SBS():\n",
    "    def __init__(self, estimator, k_features, scoring=accuracy_score,\n",
    "                 test_size=0.25, random_state=1):\n",
    "        self.scoring = scoring\n",
    "        self.estimator = clone(estimator)\n",
    "        self.k_features = k_features\n",
    "        self.test_size = test_size\n",
    "        self.random_state = random_state\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        \n",
    "        X_train, X_test, y_train, y_test = \\\n",
    "            train_test_split(X, y, test_size=self.test_size,\n",
    "                             random_state=self.random_state)\n",
    "\n",
    "        dim = X_train.shape[1]\n",
    "        self.indices_ = tuple(range(dim))\n",
    "        self.subsets_ = [self.indices_]\n",
    "        score = self._calc_score(X_train, y_train, \n",
    "                                 X_test, y_test, self.indices_)\n",
    "        self.scores_ = [score]\n",
    "\n",
    "        while dim > self.k_features:\n",
    "            scores = []\n",
    "            subsets = []\n",
    "\n",
    "            for p in combinations(self.indices_, r=dim - 1):\n",
    "                score = self._calc_score(X_train, y_train, \n",
    "                                         X_test, y_test, p)\n",
    "                scores.append(score)\n",
    "                subsets.append(p)\n",
    "\n",
    "            best = np.argmax(scores)\n",
    "            self.indices_ = subsets[best]\n",
    "            self.subsets_.append(self.indices_)\n",
    "            dim -= 1\n",
    "\n",
    "            self.scores_.append(scores[best])\n",
    "        self.k_score_ = self.scores_[-1]\n",
    "\n",
    "        return self\n",
    "\n",
    "    def transform(self, X):\n",
    "        return X[:, self.indices_]\n",
    "\n",
    "    def _calc_score(self, X_train, y_train, X_test, y_test, indices):\n",
    "        self.estimator.fit(X_train[:, indices], y_train)\n",
    "        y_pred = self.estimator.predict(X_test[:, indices])\n",
    "        score = self.scoring(y_test, y_pred)\n",
    "        return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Below we try to apply the SBS class above.\n",
    "We use the KNN classifer, which can suffer from curse of dimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2clXWd//HXB4lUTKey7Kcmg1iSVI6WRGrLGCnoav4e\ntv4SQR23kvIGzXsrRJZuNNFdBTcXNUcThLZ0g7ZVcOPUsi2CCqgE3nAzCqjlbYK6KvP5/fG9Bg7T\nzHDOzLnOdZ3vvJ+PxzyY6+5cn4/jOZ9zfb/X9f2auyMiIpJHfbIOQEREpDMqUiIiklsqUiIiklsq\nUiIiklsqUiIiklsqUiIiklupFikzu83MXjCzRzvZfqCZ/cHM3jKzC9ttG2Vmq8zsSTO7LM04RUQk\nn9K+krodGNnF9peA84Bri1eaWR9gWnLsEGC0mQ1OK0gREcmnVIuUuy8EXuli+4vu/jDwbrtNQ4Gn\n3L3F3d8BZgEnphepiIjkUV77pPYBni1aXp+sExGRXiSvRUpERIS+WQfQiQ3AfkXL+ybrOmRmGoBQ\nRKQGuLuVs381rqQs+SllvzZLgAPMbICZ9QNOAeZ0dbC7R/MzceLEzGNQLsqnFn+UT75/uiPVKykz\nmwk0Ah80s2eAiUA/wN19upntBTwEvA9oNbPzgYPcfZOZnQvMIxTS29x9ZZqx5sm6deuyDqFiYsoF\nlE/eKZ/4pFqk3P3UHWx/AfhoJ9vuAw5MIy4REakNunEih5qamrIOoWJiygWUT94pn/hYd9sJ88TM\nPIY8RERiZmZ4Dm+ckDIVCoWsQ6iYmHIB5ZN3yic+KlIiIpJbau4TEZGqUHOfiIhERUUqh2Jqh44p\nF1A+ead84qMiJSIiuaU+KRERqQr1SYmISFRUpHIopnbomHIB5ZN3yic+KlIiIpJb6pMSEZGqUJ+U\niIhERUUqh2Jqh44pF1A+ead84qMiJSIiuaU+KRERqQr1SYmISFRUpHIopnbomHIB5ZN3yic+KlIi\nIpJb6pMSEZGqyF2flJndZmYvmNmjXexzo5k9ZWbLzOyQovXrzGy5mS01s8VpxikiIvmUdnPf7cDI\nzjaa2bHAIHf/GDAO+EnR5lag0d0Pcfeh6YaZLzG1Q8eUCyifvFM+8Um1SLn7QuCVLnY5Ebgz2fdB\nYA8z2yvZZmnHJyIi+ZZ6n5SZDQDmuvunO9g2F/iRu/8hWX4AuNTdHzGzNcCrwBZgurvf0sU51Ccl\nIpJz3emT6ptWMBVwhLs/Z2YfAuab2crkyqxDTU1N1NfXA1BXV0dDQwONjY3AtktmLWtZy1rWcvWW\nC4UCzc3NAFs/n8uV9ZXUzcACd5+dLK8Chrv7C+32mwi87u7Xd3KOqK6kCoXC1j94rYspF1A+ead8\n8i13d/clLPnpyBzgdAAzGwa86u4vmNmuZrZbsr4/cAzweBViFRGRHEn1SsrMZgKNwAeBF4CJQD/A\n3X16ss80YBSwGTgz6Y8aCNwLOKFJcoa7X93FeaK6khIRiVF3rqT0MK+IiFRFXpv7pExtHY8xiCkX\nUD55p3zioyIlIiK5peY+ERGpCjX3iYhIVFSkciimduiYcgHlk3fKJz4qUiIiklvqkxIRkapQn5SI\niERFRSqHYmqHjikXUD55p3zioyIlIiK5pT4pERGpCvVJiYhIVFSkciimduiYcgHlk3fKJz4qUiIi\nklvqkxIRkapQn5SIiERFRSqHYmqHjikXUD55p3zioyIlIiK5pT4pERGpCvVJiYhIVFItUmZ2m5m9\nYGaPdrHPjWb2lJktM7OGovWjzGyVmT1pZpelGWfexNQOHVMuoHzyTvnEJ+0rqduBkZ1tNLNjgUHu\n/jFgHHBzsr4PMC05dggw2swGpxyriIjkTOp9UmY2AJjr7p/uYNvNwAJ3n50srwQagYHARHc/Nll/\nOeDufk0n51CfVC+1dm0LEyY0s2FDK/vs04fJk5sYOHCAzqPz1PR5YtWdPincPdUfYADwaCfb5gKH\nFy3PBw4FvgJML1o/Frixi3O49D5r1qzzQYMuctjk4A6bfNCgi3zNmnU6j85Ts+eJWfJZXVYNyfpK\nai7wI3f/Q7L8AHAp4UpqpLuflawfCwx19/GdnMPTzqOaCoUCjY2NWYdREWnmMnbsJGbMuBjoX7R2\nM/37T2HPPSdW7DwvvjiJzZvbzlMgXOynfZ42yqe88xSodj5jxkzhrrsqd55iMX0WQPeupPqmFUyJ\nNgAfLVreN1nXD9ivg/Wdampqor6+HoC6ujoaGhq2/nHbOh9rZXnZsmW5iievyxs2tLLtgwnCh1N/\n9ttvDVdeWWDYsLD/okVhe3eXjztuDStXLklen63n+9SnWrn77p6/ftvyDTe0smiR8qm1fFasWLNd\nMcnL+yMPy4VCgebmZoCtn89lK/fSq9wfoB54rJNtxwH/nvw+DFiU/L4T8DShqbAfsAz4RBfnqNTV\nqNSQMWOuKmp68a1NMGPGXKXz6DxVO8+BB17lb75Z0VNFi24096VdoGYCG4H/BZ4BziTcxXdW0T7T\nkoK0HDi0aP0o4AngKeDyHZwnhf+cknf33rvO+/SJpy9C56m98wwceJGPHLnOGxrcV62q6Omi1J0i\npREnciimdui0cnnzTTj0UPjmN1tYsqSZjRtb2Xvv9O/qWrFiDUOG7J/6eZRP986TRT719QOYPh2+\n9z348Y+hqQmsvPvXOhXTZwF0r09KRSqHYvofM61cLrwQNm6EWbMq/tJdiulvA8qnkh5/HE45BT79\nafjJT2CPPXr+mrH9fVSkpFcoFGDMGHj0UfjgB7OORmSbN94IX6Dmz4e774ahQ7OOKF80dp9E7y9/\ngTPPhFtuUYGS/Nl1V7j55tDsd/zx4d/W1qyjqm0qUjkU03hdlc7l29+Go4+G446r6MuWLKa/DSif\ntHzlK7BkCcyZA8ceC88/373XyUs+WVKRkpoxdy4sWADXXZd1JCI7NmBAaJoeOjTc5HP//VlHVJvU\nJyU14cUXQ4f07NnwhS9kHY1IeRYsgNNOg9Gj4Qc/gH79so4oG+qTkii5w7e+FW6WUIGSWnTUUbBs\nGaxaBUceCatXZx1R7VCRyqGY2qErkcvdd8PKlTB5cs/j6amY/jagfKppzz1DH9XYsTBsGMyYseNj\n8pxPtahISa5t2AAXXAB33gk775x1NCI9Ywbjx8O8efAP/xAe/N20Keuo8k19UpJb7jBqVGgemTAh\n62hEKmvTplCw/vu/w0PphxySdUTpU5+UROXmm+GVV+CKK7KORKTydtsNfvpTuOoqOOYYuOGG8MVM\ntqcilUMxtUN3N5enn4YrrwzNfH2znlCmSEx/G1A+eTB6NCxaFPqoTjgB/vznbdtqMZ9KU5HqZdau\nbWHs2EkcddRExo6dxNq1LVmH9Fe2bIEzzggDdg4enHU0IukbNAgWLoSDDgrNfjNnhvfpBRfcntv3\nabWoT6oXWbu2haOPnsrq1ZMIkwVuZtCgicyff14qI0Z31zXXhAcfH3gA+uhrlPQyd9zRwte+NpUt\nW/L9Pu0O9UlJlyZMaC4qUAD9Wb16EhMmNGcY1fYefRSmTIHbb1eBkt5p/vzmogIFeXyfVpM+BnIo\nrXbobdOtF+vP+vXpjYBZTi5vvw2nnw7XXhuGlMmj2PoIlE/+bP8+LST/9mfjxt45Uq2KVC+yzz59\ngM3t1m5m+fI+PPFEFhFtb9KkUJzOOCPrSESy09n79Nln+/DGG1lElC31SfUia9e2cNhhU3nppe3b\nupuazuOGGwZUfFbRcvzP/8BJJ4WhY/baq/rnF8mLjvqOBw6cyKc+dR5PPz2A2bPhk5/MOsru0aSH\n0qV334X6+hY++clm3n57++m8V6yAr361srOKlmrz5nBH09VXh0Il0tt1Nk39nXfCxReHIcLGjcvm\nC2VPdKdI4e41/xPSiMeCBQtSed1Zs9yPPLLz7W+84f7Nb7rvv7/7gw9W5pyl5HLOOe5jx1bmfGlL\n62+TFeWTbx3ls2qVe0OD+0knub/0UvVj6onks7qsz/fU+6TMbJSZrTKzJ83ssg6215nZPWa23MwW\nmdlBRdvWJeuXmtnitGONmXu4IeGSSzrfZ5ddwlVUNWcVnT8/DLo5dWq65xGJxYEHhod/P/rR0AKx\ncGHWEaUr1eY+M+sDPAmMADYCS4BT3H1V0T4/Bl5398lmdiBwk7t/Kdm2BviMu7+yg/N4mnnEYMEC\nOPtsWLGitFu7W1rC1Bi77hpGffjIRyof06uvhubF224Ls+2KSHl+/Wv4+tfDe/u734Wddso6oq7l\n8TmpocBT7t7i7u8As4AT2+1zEPBbAHd/Aqg3sw8l26wKMfYK114LF11U+rNHbbOKDhuW3qyi48fD\nl7+sAiXSXccfDw8/HL6EjhgB69dnHVHlpV0A9gGeLVpen6wrthw4CcDMhgL7Afsm2xyYb2ZLzOwb\nKceaG5V+1uPxx2Hp0jCPTTn69g3TCcycGb6tXXJJeJapHJ3lcs89ocnimmvKe72sxfAcTjHlk2+l\n5LPPPmF0lqOPhs9+NjSfxyQPVylXA+83s0eAc4ClwJZk2xHufihwHHCOmR2ZUYw1bcoUOO+87s/H\n1NgYitwTT1RmVtEXXoBzzoE77oD+7Z8tFpGy7bRTaO67557QQjF+PLz1VtZRVUbafVLDgKvcfVSy\nfDnh7o5Ovz+b2VrgU+6+qd36iYS+q+s7OMbPOOMM6uvrAairq6OhoYHGxkZg27eR3ri8YQMMHlxg\n5kw44YSevd7w4Y1Mmwbf+16Bc8+FH/yg/Ndzhy98ocCAATBjRvb/fbSs5diW584tMGUKvPZaI7Nm\nwfPPZxdPoVCgubkZgPr6eiZNmpSv56TMbCfgCcKNE88Bi4HR7r6yaJ89gDfc/Z2kSe8Id28ys12B\nPu6+ycz6A/OASe4+r4Pz6MaJTlx6aWii+6d/qtxrLlsGp5wS+qumTQvz4pSquTnEsngx9OtXuZhE\nZBt3uOWWcHV1zTVw5pn5eKYqdzdOuPsW4FxCgVkBzHL3lWY2zszOSnb7BPC4ma0ERgLnJ+v3Ahaa\n2VJgETC3owIVo7ZvIj31l7+EO+e+/e2KvNxWDQ2hs7ZPH/jMZ0JTYGeKc2lpCUXzzjtrt0BV6m+T\nF8on37qbjxmcdVa4+en66+HUU+G11yoaWtWkPp2cu98HHNhu3b8U/b6o/fZk/VqgIe34YjZ9Oowc\nmc5grf37h1lF7747zCr63e/C+ed3/m2ttTV8m7voonDbuYikb8gQWLIELrwwPFN1993wuc9lHVV5\nNCxSpN5+O0ykNmdO+J8zTatXh9lFP/zhMMXGhz701/vceCPMng2//33+n+UQidEvfwnf+lb4onjJ\nJdlMhZO75j7JzqxZ4cn0tAsUbJtVdMiQcL4FC7bfvmpVGGvsjjtUoESy8pWvwEMPwdy5oYXl+eez\njqg0KlI51NN2dfdw23lXQyBVWr9+oYP2pz8NI1Wce24Lp546iYaGMzjyyEmMH9/CAQdUL560qM8j\n35RP1/bbL/RTff7z4Qtlc3OYpv6ooybmdpr61PukpPruvz/0DR1zTPXPfcwx8KtftTB8+FTefHMS\nYSSsw7jjjomMHVv701+L1Lq2h/Q/8YkWTjtt+2nqFy3K3zT16pOK0IgRYV6o007L5vxjx05ixoyL\n2X4W4M2MGTOFu+6amE1QIrKdLN6nqfRJmdl5Zvb+7ocl1fTII/Dkk+E5pqx0Nk19b53+WiSPauV9\nWkqf1F7AEjP7eTLtRg4eCYtbT9qhp0yBCy6A97yncvGUa/vprwvJv5vZe+/a7wJVn0e+KZ/SdTZN\nfd7epzuMxt2/B3wMuA1oAp4ysx+a2aCUY5MytbTAvHnwjYyH4p08uYlBgyay7Q0QpqmfPLkps5hE\nZHu18j4tuU/KzA4GzgRGAQuAYcB8d780vfBKoz6p4IIL4L3vzcfI4h1Nf52nzlgRqf77tDt9Ujss\nUmZ2PnA68CJwK/BvyTh7fQhzRWV+RaUiBa+8Ep5XeuyxMHS/iEjepPUw7weAk9x9pLv/azJ5Ie7e\nChzfjThlB7rTDv2Tn4QJBPNWoNRHkG/KJ99iy6c7SnlO6j+Al9sWzGx34BPu/mDxaOaSnbfegqlT\nYf78rCMREamsUpr7lgKHtrWnJc18DyWTEeZCb2/uu/XWMNnZb36TdSQiIp3rTnNfKVdS21UAd281\nM41UkROtreG285/8JOtIREQqr5Q+qTVmNt7M3pP8nA+sSTuw3qycduhf/zpMOphMipk7sbWpK598\nUz7xKaVIfRM4HNgArAc+B5zV5RFSNddeGwaS1SPWIhIjjd1XwxYtCvM4PfVUGDRSRCTPUumTMrOd\nga8BQ4Cd29a7+9+XHaFU1LXXhhk3VaBEJFalNPf9DPgIMBL4HbAv8HqaQfV2pbRDP/VUmOX273P+\nVSG2NnXlk2/KJz6lFKkD3H0CsNnd7wD+ltAvJRm6/nr45jehf/tBjEVEIlLKc1KL3X2omf0eOBt4\nHljs7vtXI8BS9LY+qT/9CQYPhpUrYa+9so5GRKQ0aQ2LND2ZT+p7wBzgj0DJQ5gm03usMrMnzeyy\nDrbXmdk9ZrbczBaZ2UGlHttb3XQTnHyyCpSIxK/LIpWMLvEXd3/F3X/v7vu7+4fd/V9KefHk+GmE\n/qwhwGgzG9xut+8AS939YOAM4MYyjo1SV+3Qb7wRHty96KLqxdMTsbWpK598Uz7x6bJIJYPI9mQq\njqGEkdJbkoFpZwEnttvnIOC3yfmeAOrN7EMlHtvr3H47HHEEfPzjWUciIpK+UvqkriZM0zGbomkc\n3f3lTg/aduxXgJHuflayPBYY6u7ji/b5AbCzu19kZkOBhYQbM/bf0bFFr9Er+qS2bAnF6Wc/g8MP\nzzoaEZHypDV231eTf88pWueEIlIJVwM3mNkjwGPAUmBLuS/S1NREfX09AHV1dTQ0NNCYjBXUdslc\n68t//nMjH/kIvP12gUIh+3i0rGUta7mr5UKhQHNzM8DWz+eyuXtqP4TZe+8rWr4cuGwHx6wFdivn\n2JBGPBYsWPBX61pb3Q87zP3ee6sfT090lEstUz75pnzyLfmsLquOlDLixOmdFLc7S6iBS4ADzGwA\n8BxwCjC63evvAbzhYbbfbwC/c/dNZrbDY3uT//ovePVVOOGErCMREameUvqkphYt7gyMAB5x978r\n6QRmo4AbCDdp3ObuV5vZOEJFnW5mw4A7gFZgBfA1d3+ts2M7OYfvKI9ad8IJcPzxMG5c1pGIiHRP\nd/qkyh5g1szqgFnuPqqsA1MUe5H64x/hi1+EtWthl12yjkZEpHvSepi3vc3AwG4cJyVq63hsc911\ncM45tVmg2udS65RPvimf+JTSJzWXcDcfhKJ2EPDzNIOSbZ57Du69NwwoKyLS25TSJzW8aPFdoMXd\n16caVZlibu674grYtAmmTt3xviIieZZKn5SZDQSec/e3kuVdgL3cfV13A620WIvU66/DwIGwZEn4\nV0SklqXVJ/WvhDvv2mxJ1klK2tqhb70VRoyo7QIVW5u68sk35ROfUkac6Ovub7ctuPvbZtYvxZgE\neOcd+Md/hHvuyToSEZHslNLcNx+Y6u5zkuUTgfHuPqIK8ZUkxua+GTPCldSCBVlHIiJSGWn1SQ0C\nZgB7J6vWA6e7+9PdijIFsRUpdzjkEPjhD+G447KORkSkMlLpk3L31e4+jHDr+UHufnieClSMrruu\nwLvvwrHHZh1Jz8XWpq588k35xGeHRcrMfmhmde6+KRlT7/1m9v1qBNfbrF3bwtixk5g06Xbe975J\nrFvXknVIIiKZKqW5b6m7H9Ju3SPufmiqkZUhhua+tWtbOProqaxePQnoD2xm0KCJzJ9/HgMHDsg6\nPBGRHkvrFvSdzOy9RSfZBXhvF/tLN0yY0FxUoAD6s3r1JCZMaM4wKhGRbJVSpGYA/2lmXzOzrwPz\nCaOWSwWtX9/KtgJVSP7tz8aNrR0fUCNia1NXPvmmfOKzw+ek3P0aM1sOfIkwht/9gNqfKmj9eli5\nsg9h7N7+RVs2s/fe3RkDWEQkDiVN1WFmhwCnAicTZs79pbtPSzm2ktVyn9ScOXDWWTB2bAv33juV\nNWvUJyUicaroc1Jm9nHCTLijgReB2cDF7p67T8xaLFJvvQWXXAJz58LMmXD44eHmiQkTmtm4sZW9\n9+7D5MlNKlAiEo1K3zixCvgicLy7H+nuUwnj9kkPrVoFw4bB88/DsmWhQAEMHDiAu+6ayJVXHsVd\nd02MokDF1qaufPJN+cSnqyJ1EvAcsMDMbjGzEUBZFVC25w633QZf+AKcfTb8/OdQV5d1VCIi+VXK\nc1L9gRMJzX5fBO4E7nX3eemHV5paaO577TUYNw5WrIBZs2DIkKwjEhGprrSGRdrs7jPd/QRgX2Ap\ncFk3Y+yVHnwwjMX3gQ/A4sUqUCIipSrr/mZ3f8Xdp5czArqZjTKzVWb2pJn9VXEzs93NbI6ZLTOz\nx8ysqWjbOjNbbmZLzWxxObHmQWsrXHMNnHACTJkC//zPsMsuOz4upnbomHIB5ZN3yic+pcwn1W1m\n1geYBowANgJLzOxX7r6qaLdzgBXu/mUz2xN4wszucvd3CZMtNrr7K2nGmYbnn4fTTgt38T30EOy3\nX9YRiYjUnpKek+r2i5sNAya6+7HJ8uWAu/s1RftcDuzr7ucmU9Xf7+4fT7atBT7r7i/t4Dy56pO6\n7z4488zw/NOECdA31a8CIiK1oTt9Uml/fO4DPFu0vB4Y2m6facAcM9sI7AZ8tWibA/PNbAsw3d1v\nSTPYnnr7bfjOd2D27HBzxPDhWUckIlLb8jDmzkhgqbvvDRwC3GRmuyXbjkhGWz8OOMfMjswqyB15\n+unwvNOTT4Znn3pSoGJqh44pF1A+ead84pP2ldQGoLg3Zt9kXbEzgR9BmGAxaeIbDDzk7s8l6/9s\nZvcSrsIWdnSipqYm6uvrAairq6OhoYHGxkZg2x86reXvfrfAtGnw/e83cu658Lvf9ez1li1blmq8\nWtaylrVcjeVCoUBzczPA1s/ncqXdJ7UT8AThxonngMXAaHdfWbTPTcCf3H2Sme0FPAQcDLwF9Ekm\nWuwPzAMmdfR8VlZ9Uq+/DueeG24xnz0bDj646iGIiNSMtOaT6jZ33wKcSygwK4BZ7r7SzMaZ2VnJ\nbt8HDjezRwnTgFzq7i8DewELzWwpsAiYm6cHiB9+GD7zmXBTxMMPq0CJiKQh1Supakn7Sqpt4NcN\nG8LAr/X1TdxyywBuvBFOOaXy5ysUClsvnWtdTLmA8sk75ZNveby7r+Z1NK37e987kfvvP4/hw2t/\nAFgRkTzTldQOjB07iRkzLqb9ZIRjxkzhrrsmpnJOEZEY5a5PKgYbNhRP696m9qd1FxGpBSpSO7DP\nPm3TuhdLd1r3tls4YxBTLqB88k75xEdFagcmT25i0KCJbCtUYVr3yZObMotJRKS3UJ9UCTStu4hI\nz3WnT0pFSkREqkI3TkQipnbomHIB5ZN3yic+KlIiIpJbau4TEZGqUHOfiIhERUUqh2Jqh44pF1A+\nead84qMiJSIiuaU+KRERqQr1SYmISFRUpHIopnbomHIB5ZN3yic+KlIiIpJb6pMSEZGqUJ+UiIhE\nRUUqh2Jqh44pF1A+ead84pN6kTKzUWa2ysyeNLPLOti+u5nNMbNlZvaYmTWVeqyIiMQt1T4pM+sD\nPAmMADYCS4BT3H1V0T5XALu7+xVmtifwBLAX0LqjY4teQ31SIiI5l8c+qaHAU+7e4u7vALOAE9vt\n48D7kt/fB7zk7u+WeKyIiEQs7SK1D/Bs0fL6ZF2xacBBZrYRWA6cX8axUYqpHTqmXED55J3yiU8e\nbpwYCSx1972BQ4CbzGy3jGMSEZEc6Jvy628A9ita3jdZV+xM4EcA7r7azNYCg0s8dqumpibq6+sB\nqKuro6GhgcbGRmDbt5FaWW5bl5d4erLc2NiYq3iUj/KppeVaz6dQKNDc3Ayw9fO5XGnfOLET4UaI\nEcBzwGJgtLuvLNrnJuBP7j7JzPYCHgIOBl7b0bFFr6EbJ0REci53N064+xbgXGAesAKY5e4rzWyc\nmZ2V7PZ94HAzexSYD1zq7i93dmya8eZF2zeRGMSUCyifvFM+8Um7uQ93vw84sN26fyn6/TlCv1RJ\nx4qISO+hsftERKQqctfcJyIi0hMqUjkUUzt0TLmA8sk75RMfFSkREckt9UmJiEhVqE9KRESioiKV\nQzG1Q8eUCyifvFM+8VGREhGR3FKflIiIVIX6pEREJCoqUjkUUzt0TLmA8sk75RMfFSkREckt9UmJ\niEhVqE9KRESioiKVQzG1Q8eUCyifvFM+8VGREhGR3FKflIiIVIX6pEREJCoqUjkUUzt0TLmA8sk7\n5RMfFSkREckt9UmJiEhV5LJPysxGmdkqM3vSzC7rYPvFZrbUzB4xs8fM7F0zq0u2rTOz5cn2xWnH\nKiIi+ZJqkTKzPsA0YCQwBBhtZoOL93H3Ke5+iLsfClwBFNz91WRzK9CYbB+aZqx5ElM7dEy5gPLJ\nO+UTn7SvpIYCT7l7i7u/A8wCTuxi/9HA3UXLhvrNRER6rVT7pMzsK8BIdz8rWR4LDHX38R3suwuw\nHhjUdiVlZmuAV4EtwHR3v6WT86hPSkQk57rTJ9U3rWC64QRgYVFTH8AR7v6cmX0ImG9mK919YUcH\nNzU1UV9fD0BdXR0NDQ00NjYC2y6ZtaxlLWtZy9VbLhQKNDc3A2z9fC5X2ldSw4Cr3H1Usnw54O5+\nTQf73gP83N1ndfJaE4HX3f36DrZFdSVVKBS2/sFrXUy5gPLJO+WTb3m8u28JcICZDTCzfsApwJz2\nO5nZHsBw4FdF63Y1s92S3/sDxwCPpxyviIjkSOrPSZnZKOAGQkG8zd2vNrNxhCuq6ck+ZxD6rk4t\nOm4gcC/ghGbJGe5+dSfniOpKSkQkRt25ktLDvCIiUhV5bO6TbmjreIxBTLmA8sk75RMfFSkREckt\nNfeJiEhVqLlPRESioiKVQzG1Q8eUCyifvFM+8VGREhGR3FKflIiIVIX6pEREJCoqUjkUUzt0TLmA\n8sk75ROIAXecAAAJJUlEQVQfFSkREckt9UmJiEhVqE9KRESioiKVQzG1Q8eUCyifvFM+8VGREhGR\n3FKflIiIVIX6pEREJCoqUjkUUzt0TLmA8sk75RMfFSkREckt9UmJiEhVqE9KRESiknqRMrNRZrbK\nzJ40s8s62H6xmS01s0fM7DEze9fM6ko5NlYxtUPHlAson7xTPvFJtUiZWR9gGjASGAKMNrPBxfu4\n+xR3P8TdDwWuAAru/mopx8Zq2bJlWYdQMTHlAson75RPfNK+khoKPOXuLe7+DjALOLGL/UcDd3fz\n2Gi8+uqrWYdQMTHlAson75RPfNIuUvsAzxYtr0/W/RUz2wUYBfyy3GNFRCROebpx4gRgobv3+q8O\n69atyzqEiokpF1A+ead84pPqLehmNgy4yt1HJcuXA+7u13Sw7z3Az919VjeO1f3nIiI1oNxb0NMu\nUjsBTwAjgOeAxcBod1/Zbr89gDXAvu7+ZjnHiohIvPqm+eLuvsXMzgXmEZoWb3P3lWY2Lmz26cmu\n/xe4v61AdXVsmvGKiEi+RDHihIiIxClPN06ULaaHfc1sXzP7rZmtSB5qHp91TJVgZn2SB7XnZB1L\nT5nZHmb2r2a2Mvk7fS7rmHrCzL5tZo+b2aNmNsPM+mUdUznM7DYze8HMHi1a934zm2dmT5jZ/UlX\nQk3oJJ8fJ/+/LTOzX5rZ7lnGWI6O8inadpGZtZrZB3b0OjVbpCJ82Pdd4EJ3HwJ8HjinxvNpcz7w\nx6yDqJAbgN+4+yeAg4GabX42s72B84BD3f3ThKb/U7KNqmy3E97/xS4HHnD3A4HfEgYIqBUd5TMP\nGOLuDcBT1H4+mNm+wNFASykvUrNFisge9nX35919WfL7JsIHYE0/F5b8z3gccGvWsfRU8g32C+5+\nO4C7v+vuf8k4rJ7aCehvZn2BXYGNGcdTFndfCLzSbvWJwB3J73cQ+rtrQkf5uPsD7t6aLC4C9q16\nYN3Uyd8H4B+BS0p9nVouUtE+7Gtm9UAD8GC2kfRY2/+MMXR8DgReNLPbk+bL6ckD6DXJ3TcC1wHP\nABuAV939gWyjqogPu/sLEL74AR/OOJ5K+nvgP7IOoifM7MvAs+7+WKnH1HKRipKZ7Qb8Ajg/uaKq\nSWb2t8ALydWhJT+1rC9wKHBTMs7kG4SmpZqUDOJ8IjAA2BvYzcxOzTaqVMTwBQkz+y7wjrvPzDqW\n7kq+1H0HmFi8ekfH1XKR2gDsV7S8b7KuZiXNLr8Afubuv8o6nh46Aviyma0hjMd4lJndmXFMPbGe\n8A3woWT5F4SiVau+BKxx95fdfQtwD3B4xjFVwgtmtheAmX0E+FPG8fSYmTURms1r/UvEIKAeWG5m\nawmf2Q+bWZdXu7VcpJYAB5jZgOSupFOAWr+D7KfAH939hqwD6Sl3/4677+fu+xP+Nr9199Ozjqu7\nkiakZ83s48mqEdT2DSHPAMPMbGczM0I+tXgjSPur9DlAU/L7GUCtfdnbLh8zG0VoMv+yu/9vZlF1\n39Z83P1xd/+Iu+/v7gMJX/wOcfcuv0jUbJFKvv21Pey7AphVyw/7mtkRwBjgi0Xza43KOi7Zznhg\nhpktI9zd98OM4+k2d19MuBpcCiwnfJBM7/KgnDGzmcAfgI+b2TNmdiZwNXC0mbWNVnN1ljGWo5N8\npgK7AfOTz4R/zjTIMnSSTzGnhOY+PcwrIiK5VbNXUiIiEj8VKRERyS0VKRERyS0VKRERyS0VKRER\nyS0VKRERyS0VKYlWMhXAtUXLF5nZlRV67dvN7KRKvNYOzvN3ZvZHM/vPDrZdm0zrck03XvdgMzu2\nMlGKpEdFSmL2v8BJpcxZU01mtlMZu38N+Lq7j+hg2zeAT7t7d+ZSayAMtVOWZHQKkapRkZKYvUsY\nReHC9hvaXwmZ2evJv8PNrGBm/2ZmT5vZj8zsVDN70MyWm9nAopc52syWJBNv/m1yfJ9koroHk4nq\nvlH0ur83s18RRkhpH8/oZPLBR83sR8m6CcCRwG3tr5aS19mNMPbZyWa2p5n9Ijnvg2b2+WS/w8zs\nD2b2sJktNLOPmdl7gH8A/l8yisHJZjbRzC4sev3HzGy/ZNixVWZ2h5k9BuxrZkcnr/mQmc02s12T\nY662MIniMjP7cdl/LZGOuLt+9BPlD/AXwgf5WuB9wEXAlcm224GTivdN/h0OvEyY4qEfYXyxicm2\n8cD1Rcf/Jvn9AMK0Mf0IVzffSdb3I4wxOSB53deB/TqI8/8QJoD7AOGL438SxmoDWEAY36zD/Ip+\nnwEcnvz+UcIYkCT590l+HwH8Ivn9DODGouMnEibdbFt+lDCA8wBCsT8sWf9B4HfALsnypcD3kthX\nFR2/e9Z/f/3E8dO3rIomUmPcfZOZ3UGYIfjNEg9b4smgl2a2mjA+JMBjQGPRfj9PzvF0st9g4Bjg\nU2Z2crLP7sDHgHeAxe7+TAfnOwxY4O4vJ+ecAfwN2wZM7qyJrXj9l4BPFDXH7ZZc4dQBd5rZxwhj\npZX6ni9+7RZ3X5L8Pgw4CPjv5FzvIYzP9hrwppndCvw78OsSzyPSJRUp6Q1uAB4hXP20eZekuTv5\nsO1XtK14tOnWouVWtn/PFA98aWwbMPM8d59fHICZDQc2dxFjd/p62p//cx5mqS4+702EEehPMrMB\nhCuzjmz975HYuej34rgNmOfuY9q/gJkNJVytnUwY/LmjfjSRsqhPSmLWNkXAK4Srnq8VbVsHfDb5\n/UTCFUG5TrZgEGHm3ieA+4GzLcwNRtIHtOsOXmcx8Ddm9oHkporRQKGE8xcXtnmEq0WS8x6c/Lo7\n2+ZZKx6F+vVkW5t1JPNjmdmhST4dnWcRcESSM2a2a5Jjf6DO3e8j9AF+uoT4RXZIRUpiVnylcR2h\nP6Vt3S3AcDNbSmjC6uwqp6tpAp4hFJh/B8a5+9vArYR5ph5JbjS4Gejybj4P05xfTihMSwnNjW3N\nZV2dv3jb+cBnk5s7HgfGJeuvBa42s4fZ/v2+ADio7cYJ4JfAB5OYzyYU3L86j7u/SJiv6W4zW05o\n6juQ0Of362Td74Fvd5WzSKk0VYeIiOSWrqRERCS3VKRERCS3VKRERCS3VKRERCS3VKRERCS3VKRE\nRCS3VKRERCS3VKRERCS3/j+3Km9DJofhRwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1fe513157b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "knn = KNeighborsClassifier(n_neighbors=2)\n",
    "\n",
    "# selecting features\n",
    "sbs = SBS(knn, k_features=1)\n",
    "sbs.fit(X_train_std, y_train)\n",
    "\n",
    "# plotting performance of feature subsets\n",
    "k_feat = [len(k) for k in sbs.subsets_]\n",
    "\n",
    "plt.plot(k_feat, sbs.scores_, marker='o')\n",
    "plt.ylim([0.7, 1.1])\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Number of features')\n",
    "plt.grid()\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./sbs.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Alcohol', 'Malic acid', 'Alcalinity of ash', 'Hue', 'Proline'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# list the 5 most important features\n",
    "k5 = list(sbs.subsets_[8]) # 5+8 = 13\n",
    "print(df_wine.columns[1:][k5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 0.983870967742\n",
      "Test accuracy: 0.944444444444\n"
     ]
    }
   ],
   "source": [
    "knn.fit(X_train_std, y_train)\n",
    "print('Training accuracy:', knn.score(X_train_std, y_train))\n",
    "print('Test accuracy:', knn.score(X_test_std, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 0.959677419355\n",
      "Test accuracy: 0.962962962963\n"
     ]
    }
   ],
   "source": [
    "knn.fit(X_train_std[:, k5], y_train)\n",
    "print('Training accuracy:', knn.score(X_train_std[:, k5], y_train))\n",
    "print('Test accuracy:', knn.score(X_test_std[:, k5], y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note the improved test accuracy by fitting lower dimensional training/test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Forward selection\n",
    "\n",
    "This is essetially the reverse of backward selection; we will leave this as an exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Assessing Feature Importances with Random Forests\n",
    "\n",
    "Recall\n",
    "* a decision tree is built by splitting nodes\n",
    "* each node split is to maximize information gain\n",
    "* random forest is a collection of decision trees with randomly selected features\n",
    "\n",
    "Information gain (or impurity loss) at each node can measure the importantce of the feature being split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 1) Color intensity                0.182483\n",
      " 2) Proline                        0.158610\n",
      " 3) Flavanoids                     0.150948\n",
      " 4) OD280/OD315 of diluted wines   0.131987\n",
      " 5) Alcohol                        0.106589\n",
      " 6) Hue                            0.078243\n",
      " 7) Total phenols                  0.060718\n",
      " 8) Alcalinity of ash              0.032033\n",
      " 9) Malic acid                     0.025400\n",
      "10) Proanthocyanins                0.022351\n",
      "11) Magnesium                      0.022078\n",
      "12) Nonflavanoid phenols           0.014645\n",
      "13) Ash                            0.013916\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEZCAYAAADCJLEQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xec3GW5///XO/QW+qEnIlVQOoiAEPRIVaqAEazIQUTE\nH3rETrADR38iioIgCnhoAoICEURClRIInVCkF4MgIEUOJe/vH/c92clkdneSfMrszPV8PPaRnc+U\n657dzVyfz12uW7YJIYQQus2ouhsQQgghtBMJKoQQQleKBBVCCKErRYIKIYTQlSJBhRBC6EqRoEII\nIXSlSFAhhBC6UiSo0FUkPSzpFUn/kvRi/nf5uXzNbSQ9VlQbO4x5iqRvVRlzMJKOkHRq3e0IYXbN\nW3cDQmhhYGfbVxT4msqvO2dPluax/WaB7amMpHnqbkMIcyquoEI3UtuD0uaSrpX0nKQpkrZpuu/j\nku7OV1wPSPqvfHxh4GJgxeYrstYrnNarLEkPSfqSpNuAlySNkrSCpN9JelrS3yQd0tGbkcZKmp7b\n+KikZyUdKGkTSbdJ+qek45oe/zFJ10g6TtLz+X29p+n+FSRdkF/nPkmfarrvCEnnSDpN0vPAp4Gv\nAvvk9z9lqJ9X889C0mGSpkl6QtLHm+5fUNIP89Xuc5KukrRAh7+jv+WYf5M0vpOfX+hfcQUVRgRJ\nKwJ/BPa1/SdJ7wXOlbSW7WeBacBOth+W9G5goqQbbd8qaUfgNNtjml6vXZjWq6wPATsCz+b7/gCc\nD+wDrAL8WdJU25d1+DY2A1YHts6vdQnwHmABYIqks21fnR/7TuBsYGlgT+A8SW+x/TxwFnAbsDyw\nDnCZpAdsT8rP3QX4oO2P5MSxDLCa7Y82tWXQn1e+f3lgMWBFYDvgd5LOt/0C8EPgbcDm+XXeCUwf\n6ncE/Bs4FtjY9gOSlgOW6vDnFvpUXEGFbvT7fFXxT0nn5WP7ARfZ/hOA7cuBycBO+fYlth/O318N\nXAq8ey7bcaztJ23/H7ApsIzt79p+M8c6iZTEOmHgW7Zfs/1n4GXgDNvP2n4SuBrYsOnx02z/JMc6\nG7gX2FnSysC7gMNtv277ttyO5uTzV9t/AMhtn7Uxw/+8XgO+neNfArwErKWU2T8BfM72351cb/t1\nhvkdAW8C75C0oO1ptu/p8GcX+lQkqNCNdrW9VP7aIx8bC+zdlLieA7YEVgCQtKOkv+Zur+dIVz7L\nzGU7Hm/6fiywUkv8rwD/MRuv93TT9/8mXX0031606fYTLc99hHQ1syLwT9uvtNy3UtPtYSeEdPDz\netb29Kbbr+T2LUO64nuwzcsO+jvK7d0HOAh4StIf8pVVCIOKLr7Qjdr1vz0GnGr7wFkeLM0P/I50\nBn+B7emSzm96nXYTJF4GFm66vUKbxzQ/7zHgQdtVfaiu1HJ7DHAB8CSwlKRFbL/cdF9zQmt9vzPd\n7uDnNZRngFeB1YA7Wu4b9HcEkLtCL8vdjt8Ffknq7gyhrbiCCiPF6cAHJG2XJywsmAfzVwTmz1/P\n5A/bHUnjJg3TgKUljW46diuwk6QllaaxHzpM/BuBF/PEiQUlzSNpXUmbdNj+Tj78m/2HpEMkzStp\nL2BtUvfZ48B1wPclLSBpPWB/4LQhXmsa8BYNDLwN9/MalNP+PKcAP8qTNUbliRHzMcTvSNJ/SNpF\nadLK66QuwxE5MzJUJxJU6DZtp4PnD+ZdSTPS/kHq1voiMMr2S8DngHMk/ZM0LnRB03PvBc4AHsxd\nT8uTPtBvBx4GJgJnDtWO3N31fmAD4CFSd90vgdF0Zsirmja3bwDWIF2xfBvYM0+QABgPrEq6mjoX\n+MYw0/LPISXIZyVNzj+vQxnk59VB+79Iunq6iTSB5Aek38Ogv6P8dRjpSu8Z0pXTQcPEDH1OnWxY\nKGkH4MekP7KTbR/Vcv+HgcPzzReBz9i+vZPnhhBmJuljwP62o/sr9LVhr6AkjQJ+CmwPrAuMl7R2\ny8MeBLa2vT7wHeDE2XhuCCGEMItOuvg2A+63/UieSnom6TJ+hjzN9IV883oGBniHfW4IIYTQTicJ\naiVmnrb6OLPOMGr2KdICxDl5bgh9z/ZvonsvhIKnmUvalrSIb6siXzeEEEL/6SRBPUFaZ9GwMrMu\nIiRPdz0R2MH2c7Pz3Pz8OS7mGUIIYeSy3XYZRiddfDcBqysVvJyfNCX1wuYHSBpDmu76Edt/m53n\ntjSykq8jjjiiJ2PFexuZsXr5vcXPcWTGqzLWUIa9grL9pqTPkmp1NaaK3yPpwHS3TwS+QSr8eHxe\nDPi67c0Ge+5wMUMIIYSOxqBsTwTWajl2QtP3BwAHdPrcEEIIYTh9WUli3LhxPRmr6njx3kZmvF6N\nVXW8eG/l66iSRBUkuVvaEkIIoRqS8FxMkgghhBAqFwkqhBBCV+qZBDVm7FgkFf41ZuzYut9aCCH0\npZ4Zg5LEuVOfLLBFyZ5rrzjsXP0QQghzJsagQgghjDiRoEIIIXSlSFAhhBC6UiSoEEIIXSkSVAgh\nhK4UCSqEEEJXigQVQgihK0WCCiGE0JU6SlCSdpA0VdJ9kg5vc/9akq6T9Kqkw1rue1jSbZKmSLqx\nqIaHEELobcMmKEmjgJ8C2wPrAuMlrd3ysGeBQ4Bj2rzEdGCc7Q1tbzaX7e0KUVYphBDK18mGhZsB\n99t+BEDSmcCuwNTGA2w/Azwj6f1tni96rCvxsUcfLa2sUgghhKSTxLES8FjT7cfzsU4ZuEzSTZLa\n7robQgghtOpoy/e5tKXtpyQtS0pU99i+poK4IYQQRrBOEtQTwJim2yvnYx2x/VT+9x+Szid1GbZN\nUBMmTJjx/bhx47pm2+G6jRk7lscefbTw111lzBgefeSRwl83hBAGM2nSJCZNmtTRY4fdbkPSPMC9\nwHuBp4AbgfG272nz2COAl2z/MN9eGBhl+yVJiwCXAkfavrTNc0fMdhtVb+0RW4mEEHrVUNttDHsF\nZftNSZ8lJZdRwMm275F0YLrbJ0paDpgMLAZMl3QosA6wLHC+JOdYv22XnEIIIYRWHY1B2Z4IrNVy\n7ISm76cBq7R56kvABnPTwBBCCP2pp6Z/hxBC6B2RoEIIIXSlSFAhhBC6UiSoEEIIXSkSVJhJ1BkM\nIXSLKipJhBEk6gyGELpFXEGFEELoSpGgQgghdKVIUCGEELpSJKgQQghdKRJUCCGErhQJKoQQQleK\nBBVCCKErRYIKIYTQlTpKUJJ2kDRV0n2SDm9z/1qSrpP0qqTDZue5IYQQQjvDJihJo4CfAtsD6wLj\nJa3d8rBngUOAY+bguSGEEMIsOrmC2gy43/Yjtl8HzgR2bX6A7Wds3wy8MbvPDf0tav+FEAbTSS2+\nlYDHmm4/Tko8nZib54Y+ELX/QgiDiUkSIYQQulInV1BPAGOabq+cj3Vitp47YcKEGd+PGzeOcePG\ndRgmhBDCSDBp0iQmTZrU0WM7SVA3AatLGgs8BXwIGD/E4zWnz21OUCGEEHpP68XHkUceOehjh01Q\ntt+U9FngUlKX4Mm275F0YLrbJ0paDpgMLAZMl3QosI7tl9o9d87fWgghhH7R0YaFticCa7UcO6Hp\n+2nAKp0+N4QQQhhOTJIIIYTQlSJBhRBC6EqRoEIIIXSlSFAhhBC6UiSoEEIIXSkSVAghhK4UCSr0\njShMG8LI0tE6qBB6QRSmDWFkiSuoEEIIXSkSVAghhK4UCSqEEEJXigQVQgihK0WCCiGE0JUiQYUQ\nQuhKkaBCKEkZ665izVXoJx2tg5K0A/BjBjYdPKrNY34C7Ai8DHzC9pR8/GHgBWA68LrtzYppegjd\nrYx1V7HmKvSTYROUpFHAT4H3Ak8CN0m6wPbUpsfsCKxmew1J7wR+Dmye754OjLP9XOGtDyGE0LM6\n6eLbDLjf9iO2XwfOBHZtecyuwKkAtm8AFs/bwAOowzghhBDCDJ0kjpWAx5puP56PDfWYJ5oeY+Ay\nSTdJOmBOGxpCCKG/VFGLb0vbT0lalpSo7rF9TbsHTpgwYcb348aNY9y4cRU0L4QQQlUmTZrEpEmT\nOnpsJwnqCWBM0+2V87HWx6zS7jG2n8r//kPS+aQuw2ETVAghhN7TevFx5JFHDvrYTrr4bgJWlzRW\n0vzAh4ALWx5zIfBRAEmbA8/bniZpYUmL5uOLANsBd3b+VkIIIfSrYa+gbL8p6bPApQxMM79H0oHp\nbp9o+2JJO0l6gDzNPD99OeB8Sc6xfmv70nLeSgghhF7S0RiU7YnAWi3HTmi5/dk2z3sI2GBuGhhC\nCKE/xfTvEEIIXSkSVAghhK4UCSqEEEJXigQVQgihK0WCCqEHlFE5Paqnh7pVUUkihFCyMiqnQ1RP\nD/WKK6gQQghdKRJUCGG2RHdiqEp08YUQZkvV3Yljxo7lsUcfLTzeKmPG8Ogjj9QWKwwvElQIoatV\nmRBjLK+7RBdfCCGErhQJKoQQQleKBBVCCDWpcsLJSJzcEmNQIYRQkxhfG1pHV1CSdpA0VdJ9kg4f\n5DE/kXS/pFslbTA7zw0hhBBaDZugJI0CfgpsD6wLjJe0dstjdgRWs70GcCDwi06fW4c7b7iuJ2NV\nHS/e28iM16uxqo4X7618nVxBbQbcb/sR268DZwK7tjxmV+BUANs3AItLWq7D51burhur++FXGavq\nePHeRma8Xo1Vdbx4b+XrJEGtBDzWdPvxfKyTx3Ty3BBCCGEWZc3iU0mvG0IIoU/I9tAPkDYHJtje\nId/+MmDbRzU95hfAFbbPyrenAtsAqw733KbXGLohIYQQepLtthc1nUwzvwlYXdJY4CngQ8D4lsdc\nCBwMnJUT2vO2p0l6poPnDtnAEEII/WnYBGX7TUmfBS4ldQmebPseSQemu32i7Ysl7STpAeBl4BND\nPbe0dxNCCKFnDNvFF0IIIdShL0odSVq67jaEEEKYPX2RoIDrJZ2TuyF7eqxL0ihJo0t67b0kLZa/\n/7qk8yRtVEasXiZpj6af45clnd1cfWUkkrTUUF91t28kknS0pNGS5pN0uaR/SNqv7nZVqS+6+HJS\n+k/gk8CmwNnAr23fV1K85YDvASva3lHSOsC7bJ9cUrz/BT4NvEma1DIaONb2MQXHud32epK2Ar4D\nHAN80/Y7C45zHDDoH6btzxUZryX2VsAatk+RtCywqO2HCo7R+DluAfwA+CHwFdubFxmnKd6WwARg\nLGncWaTx47cWGOMh0u9MwBjgufz9EsCjtlctKlab2HsBE22/KOnrwEbAd2zfUlbMKki61fYGknYH\n3g8cBlxle/0SYi0LHAC8haa5CbY/WXSs2dEXV1BOLrM9nvRL+Bhwo6QrJb2rhJC/Bv4ENKoo3gd8\nvoQ4DevY/hewG3AJaXr/R0qI82b+d2fgRNsXAfOXEGcycPMQX6WQdARwOPCVfGg+4PQSQjV+ju8H\nTrB9AbBACXEaTgZ+BGxFOkHbJP9bGNur5oT3Z+ADtpexvTTpPV5aZKw2vpGT01akE9GTgZ+XEUjS\n+yVNkfRPSf+S9KKkf5URi4FEsTNwju0XSooDcAGwOOn3d1HTV636opp5HoPaj/ShPQ04hDQ1fgPg\nHNIHepGWsX22pK8A2H5D0pvDPWkuzCdpPlKC+qnt10taV/aEpBOA9wFHSVqAEk5ybP+m+bakRfPx\nl4qO1WJ3YEPglhzvyUZXXMGekvQzYAdgE0nzU+7J4gu2Lynx9ZttbvuAxg3bl0g6uuSYs5w4SfpO\nSbF+DOwB3OHyu5/+mNeU/hs4KF/lvFpSrIVtd10x7764ggL+Sur22s32zrbPs/2G7cnkwrYFezkn\nRcOMxc5lnv2cADwMLAJcldedlXFWtzfpynB7288DSwH/XUIcACS9XdIU4C7gbkk3S1q3rHjAa/lD\np/F7W6SkOHsDVwI7234OWAb4ctFBJG2UxwivkHSMpHc1jpU4dvhkHp98S/76GlD8Hg8za5w47QNc\nXNaJU/YYcGcFyQnbXwa2ADbJtUxfprxapn+UtFNJrz3H+mUMam/bZ7cc28v2OSXF2wg4Dng7cCew\nLPBB27eXEW+QNsxr+40SXrf0MZqmWNcBX7N9Rb49Dvie7S1KivdFYA3SFeL3SWOW/2v7uIJef8jJ\nK7mbtjCSrhg6nN9TZLwccyngCGDrfOgq4Ejb/yw6VlPMhUlXo3fYvl/SCsA7bBfetShpU+DbpBOM\n/2sct/2jAmPsMdT9ts8rMNaLDIwdLkJ6T68zME5ZyoSrTvVLgrrF9kbDHSs45rzAWqRf9L35DKjo\nGIcNdX+R/2lyvCNI4xdr2V5T0oqkvvEti4zTFO+21gHhdscKjvk+YDvS7+1Pti8r8LUfY+DDoJVt\njykqVr+RNA+wHDMP8D9aQpxLgZeAO4DpTbGOLDDGKUPc7bonLlSpp8eglPap2glYSdJPmu4aDRR+\nddFiMwZmxGwkCdunFhyjMT6yFmnQ+8J8+wPAjQXHgurGaBoelPQN4LR8ez/gwRLjkRNSYUmp5bVX\nKeN1hyPpUOAU4EXgl6RZbl8u8gpD0o9tf17SH2gzA9P2LkXFahP7ENJV2zQGkoaB9UoIt6Ltt5fw\nujPY/kSZr99Onul5q+2X81T2jYAfl5HkZ0dPJyhS3/dkYBdmnv31IvD/lRVU0mnAasCtDAzgmrxn\nVlEaZ22SrgI2sv1ivj2BcmbgvGbbjQkYJY7RNHwSOBJodGlcnY+Voqm7A9LsxPmAl8vo5sj9/Y1u\nsEm2JxYdo8knbR8raXtgadJkodModnZd4yTifwp8zU4dSrqqf7aCWBdL2q6M7sNWkhZn5u7SK4Fv\nlTSb7+fA+pLWB74AnET6nW5TQqyO9XSCsn0bcJuk35YxHjOETUhTv6vqP10OeK3p9mv5WNHOzoPR\nS0g6gJQsfllCHADyBILS1jy1iTfjajCvndsVKHxtkqTvAlsC/5sPfUnSVra/XnSsRsj8707Aqbbv\nyu+vMLYbJ4CTgX/bng4zut7KnEIPaeJCmZOQmh0EfFFSFWM1vyKNYe+db3+EdCU85BjVHHojn3zu\nSpoJfLKk/UuIM1t6egxK0tm295Z0BzN3OzT+qMroAkDSOcDnbD9Vxuu3ifc10h/x+fnQbsBZtr9f\nQqzSxmjaxFoT+CKzLh4sfHB/iDZMsb1hwa95O7Ch7Tfz7XmBW0r8ezyFtFHoqsD6wDykq7aNS4h1\nPfCfjSUBeYnApWVNbMkxTiZ1c19ESRMX6qC8UHe4YwXFuhKYSDrpfDfwNKnLr5S/yU719BUU6dIf\n0mLBKi1DmhZ9IzP/hymlH972dyVdQvrDAviE7SklxSptjKaNc0jLAE5ioKu0NC2zp0aRroTLWncy\nmlRtAQbGEsuyP2nN34O2X8lLIMoa51iweb2a7ZfyLLsyPZq/5qecheMzSNq63XHbV5UQ7t/5yvqa\nHHtL0pqoMuwDfJj02fH3/D7L7sIfVk8nqKYrmGfI3Q75rHxtUsWFskwo8bVnkDTa9r/y1N6H81fj\nvqWKntqbP8CPAv6DdAVV9lTUN2yXUhFgEB9ojk36eZax7uRo4BZJl5N+huOAb5QQB4D8d/8QsKak\nBcuKk70saSPnMkOSNqa8D1Wg2Bl0HWhe97cgaTLUzUAZV/WfBk7NY1EC/gl8vIQ45KR0BfBhSacD\nD5EWJdeqp7v4GiTdTLq6WBK4llSv7jXb+9basLkk6Y+236+Z66A12AXWWsvxHiCVsSl1Ty8NFBf9\nHKmr4XxmvhItbU1NVSStBDRqGN5g+4kSY32K1JuwMmnizubAX0taB7UpcCZpgpKA5YF9msaoioxV\n28zBpjasQprttmeJMUZD8evk8muvSdpEdjzpRP4s4Iu2xxYda070S4K6xfZGeTrqQraPLqMvV9I1\ntrdqmQ0GXbLobW5JurasNU8tcdol3IYyEm/lxWklLU8qqto8tnZd0XFyrDtIyxCudyo+ujZpwXMZ\ng+0old1aK98sZQ1gjrOx7ZsltZ1pZvvKMuK2tEHAXbbXKeG1FwD2ZNYx2G8VGGM6aXbs/rYfyMce\nLPr/2Jzq6S6+JlIqCrsvqT8e0kBxoWxvlf8te0xhFpJ2YeZpy38sIcxkSWcBv2fmK5rCVrbn1yut\n8vUgJjd9fyRpam9pJH2PtKbrHmZet1NWqZlXbb8qCUkL2J4qaa3hnzbH1gLWIXWBlbUGcMbMwSoS\nUUPLycwo0theWVXTLyDNTryZpv9vBdsD+BCpHNZE0tVv12xJ1C9XUFuTZoNda/soSW8FPl/0mbGG\n2femrK4pST8gnSH/Nh8aD9xk+6sFx2m3wr20le35TPwgmhIvqfp3KWfkOWbhs/baxLgXWN92WRMw\nWuOdT5oU8XnSWMlzwHy2C0+IStVGxpES1MXAjsA1tj9YdKymmKVvJ9IU62NNN98AHrZ9bdFxcqw7\nXfKi4KZYi5DGW8eT/kZOBc6vYr3XkO3qhwRVlaq7ppri3g5s0LL2ZErdU0TnlqSTSItlG9XNPwK8\naftTJcYstQRWjjER2NP2y2XGGST2NqRtFSbafm24x8/B699Bmso+xfb6SnujnW77fUXHaoo5lbTw\n/maaZnu6moW7pZF0InCc7TsqjrsksBdp7PC9VcZu1RddfFWtp6mha6rZEqRZPpA+gAoj6Ut53K7t\nWE0ZYzTZpp657t5fJN1WUqwqvUiaxfdnZu4qHbK2YhEq6A5rzJZ9Iw/uPw2UXeKp9O1EalpTuRXw\n8Xzi+38lx5rBaYH8ifmrVn2RoKh4PQ1UNibU8H1gSp4mqhy3yO0bGrP2Jg/5qOK9KWk1238DyF2z\nhf/+Wia1LKyBDejKmtwyMX/1osmSliBVGLmZVFj1ryXHvELSMaSSWM0Jv8ixoTrWVO5YYayu1Bdd\nfJJuLmPV/BDxKhkTaom5AgO7pN5o++8lxJiRLKog6b2k0i4PkpLFWNJCwqG2kRgRlDYpHNOYOVVS\njAVslzW43kn8twCjXfI2M2q/rYhLmka/CG3WVJY4U7Gy7W26Ub8kqAlUuJ6mjjGhvK6mMUgMFL+6\nPZdDWZm0juxq4Kqy+8fzVNvmKcu1feAWRdLOpC3Y57e9qqQNgCNs715wnMbyitNsf6TI1+5XVa6p\nVMXb23Sjfunia8y8aV4FbqDMuf6ljQm1knQUqVTJXcw8bbnQBGV7m3zmvylpptZFkha1PeTsxTkl\n6WDgt40zcElLStrf9vFlxKvQt0iLdK8AsH2rpNVLiDO/pA8DW6jNJnhFLw+oU07665KmtgPFrhdq\nDuVULmp/4PjGmsoS4kD129t0nb5IUDVMXih7TKjVbqSzrFKvLnJ3w7vz1xLAH0lXUmU5wPbPGjds\nP6dURX2kJ6jXbT+vmQuKl9GV8WnS2r8lmLmMUyNeTyQoSb8AFga2JY0zf5By9kPL4cpfU5lVvb1N\n1+mLBKVUrPIwUp//f0lag/SBXvjEhbyy/BpSOZnGmNDhZYwJNXmQNB277O6vSaSB7+8DF5cxTbnF\nPJLk3A+du0pLLQZakXsk7Q2MkrQqqaTT9UUHcSoyeo2kybZPLvr125G0OamyQmNvstHA22zfUGLY\nLWyvJ+l220dK+iHl1dr8PPAV0hqhu/LEnbLGRCvd3qYb9csY1FmkD9aP2n57TljXuYSy9TneHbbf\nUcZrDxLvXNLak8uZeYyt6IXIS5D2MdqalHynk2q6lVLoNM/MGguckA8dCDxm+wtlxKtKPhP+Jk3b\nlgBH2n6lpHjzk66mmje++0UZA/uSppA2z2ycVIwCJpe5tkzSDbbfqbTVxx7As6QkWUa3aSPmwmX9\nvlriVLa9TTfqiysoYDXb+0gaD5D7kMss53GLpE1t31RijGYXMrDde2lyt9SDpHUtKwNbkK7cynI4\nKSkdlG9fRurCGdHyAt3D81cVjif9nhpdox8h7aBaxoLnGVe8MKOSetmfM3/MJ0/HkMZrTElXGrl7\n72RgUWCM0g60B9r+TBnxXO32Nl2nX66grgPeSyp1tJGk1YAzbG9WUrypwBqk7RpepqIFdmXLyWkq\nadzpatJ09rK7+XpOnhBxGLMuHN+upHi3tSx4bnusoFjnkbqCG9ukfAbY1vZuRccaJP4CpD2pStlh\nV9INpDGuC51LYqmkkkSqfnubrtMvV1ATSAsjV5H0W1I3VVkbtgFsX+JrzyKPqX2fgQKdALj40kqr\nN6bOl6nNav1mLuODtWK/I52Fn041C8crWfCcfRr4CfB10u/wcuC/SooFzJj6/Svgf3MVhFLHYm0/\n1tIBU9bP8mgq2N6mm/VFgrJ9af4j3px0FnKo7WeKjqO0GdyngdWBO4CTbb9RdJw2TiFV4P7/STOZ\nPkGqtFyoKpJT1m61vkhdi1+pqA1lmm77uArj/Tep2sJMC57LCGT7aVJ17CrtQ3o/N0maTPr/cKnL\n6R56TNIWgJWKGR/KQKWVok3r5+QE/dPFd7lbih62O1ZAnLOA10ndXzsCj9g+dOhnFRL3ZtsbN0/O\nqLp6RlkkbUjainov0i6f59r+ab2tmjN5RhukwqZPMevC8cI3pGuKXeqCZ9VXr7G5DaNIJzc/J13V\nnAIcW+SCfEnLAMcC/0lK9peSTngLK0zbtGZtG9KGj6Vub9PNevoKKl/RLAwso1Sht3FdPhpYqYSQ\n6zQliJMpby1Gq//L/znvl/RZ4AnSIO6IpPa7fMr2trU2bO7dxczV7ptnP5q0gWEpckIqs+RQXfUa\nAZC0HukqaifgXFKZsa2Av5D2bCpE7nkpeyfu5jVrr5Bm8c1oAj2yfq0TPX0FJelQ0rqFFUkf2o0P\nhn8Bvyz6TFwtWzW03i6L0jbb95AWZH6blICPsV3I2hpJQ1bZtv2jIuI0xevqXT5Dd8nd98+TxvXO\nbb46lHSeC9w5WKke3gHMOsGllD3R+l1PJ6gGSYdU0ecv6U3SrD1IyXAh0hlQqbNvJG3kYis3t75+\nY4fZtUjrnxpT2j9Amsm3X8HxdiONY2xJmtxyJnBSDRVBSpG72w4kneGblIx/WXYlkDJJ+gNDVMOw\nvUuJsd9q+8GyXr8l1nWk31fr3lPnlhDrraTuxM1JP9u/kjZajWKxvSYPbL6Fmc96Ct+Gug65pNLy\npNlhZ9kazMA9AAAVNklEQVS+s6Q4VwE7N1UJWAy4yPbWQz9zjuN15S6fc0vSmaQxhdPzoQ8DC9ku\nZXJBnvp9MqnqdikTXZQ2QhyUS9yHStL3gKNtP59vLwl8wfbXS4h1a1kL/NvEuh74GXBGPvQh4BDb\n76wifjfoiwQl6TRgNeBWBs56XMXAbVUkLQ/sTZrRNJqUqL5TcIx7gfUaZ/r5SuB222sN/cxCYnfN\nLp9zS9LdttcZ7liB8f6TND6zOWlvtFNs31tGrDpImtJYk9R0rJTudUnfIVWhubjo124T6/bWtZNl\nrV/rVv2SoO4hTWDo+Tcr6R3Al0gf5IXWrZP0NVISPD8f2g042/b3iozT6ySdAfyoUWlE0sbAYS5h\ny4aWuIuTrka/BjxGqrZwepEljypck9cc83bS7suNE6eFSOWV1i0h1ovAIqQr4NcpsfteaZeC50hd\n3CadfC5JqphR2nZB3aRfEtQ5wOdsP1V3W8og6W2kP949SXXIziINFj9dQqyNSNXMIe0HNaXoGL1O\n0p3A20jT5gFWJU1yeZ30YVfGmf/SwH6kMkdPMjDL7R22xxUY5xoG1uR9gLwmz/Y3i4rRJubhOdYp\n+dAnSJUeji4rZhWUtnofjPth0lC/JKgrSFNNb2Tm9QSlDdxWSdJfSWdZ59h+suRYfb3DZxFyqa1B\nueBdiyWdT5rgchrw6+YTNaVK55sUGKuWNXmSdiCtTQK4zPafSorT7uThBdKaxyoW5feVnl4H1WRC\n3Q0ok+13VRFHTTt8ks5W5yMN9PfNDp9FKDoBdeCXrWMmytvBF5mcsrrW5E0h/T06f1+W44GNSJVi\nAN4B3AksLumgkT6Bp9v0xRVUr6uq319p59ANgVs8UChzloHc0F3aTRgocRJB65q8xUkz7Arf76op\n5t6kcZlJpDGhdwP/bft3JcQ6D/iG7bvy7XVIOyR/CTivqhl+/aKnr6AkXWN7qzyw2ZyJe60qcCW1\n+IgdPkeUPLNzJWChXDKquZLKwmXE9MAWMy9RbkHmZl8jTZJ4GmYspv0zadlF0dZsJCcA23dLWtv2\ngyp1B5/+1NMJyvZW+d/F6m5LyRayfbkk2X4EmJBX1xc9MN1uh88Rvz9TD9se+Dhp767mah8vAl8t\nMpCkIfcjK3m8d1TLhKBnKecEDeAuST8njflCmpx0d15yUchsyEHGuWYoc1F+t4kuvh6QV7dvRTpj\n/Aup3/8HZaxPUp/v8Dk3JD1H+2oLjSv6pUqKu2cZlQ5aYvyDNHX9DOAGBq7WgNIX6h4DrMfAgtZ9\nSOvzCt8QMk9h/wzp/xvAtaRxqVeBhW2/VECMxhbyC5LGfG8j/TzXI02fr2TMuRtEguoBVfX7Szqq\n9T99u2OhPUnzDHW/7UL3FZK0n+3TJX2B9hXGC6uhmN/b+0jrrNYDLiJtCnrXkE8sLv6eDEzWudr2\n+UM9fiTI411H2L4j3347MMH2B+ttWXUiQYWODTLYHpMk5pCkpZh5UkuhSwQkHWj7hKZaijOxfWSR\n8ZriLkBKVMcAR3qEbo/STpULkSXd1brYuN2xXtbzCSqf2f3ZI3+rhllUVaBT0kGkbo23As1TpBcD\nrnXBxWJ7naSdSRNaViaNl6wE3Gd77VobNpdyYtqZlJzeQioq/CvbT5Qct7Kt0atciJwrjrzMQM3G\nfUnrDscXHatb9XyCApB0ObCH7RfqbkuRqirQmUvkLEk6c/xy010v9kO5laLl6frvI+36umEe19vb\n9gElxSt9iwhJpwJvBy4GznRJBYsHif0AFW2NXuVCZKX97A4CGsWYrwJ+bvvVomN1q35JUBeQ1u9c\nxsB2GJXs8lkmSWNsP1plvHbHq2xDL2hUb5B0G7BBnrpfWhFQVbBFhNIeXo3/W5Uu6ZB0re1KFotX\nOSEp9Pg08ybn0Zu7UP6etKodSefa3rPkeBcxsCPsgqQacvcCfdMnXpAXJC0KXAOcKulp4N8lxlu4\n7Ikstsua1j0oDWyNPlnSWVSzNfqhpDVknyNNSHoP8LEiA0g62/beku6g/eSWvhnz7YsrKABJ8wNr\n5pv3usAKznVR0zYDarPlQAXxNwI+Y/tTVcYd6ZT20XqFtFbno6RZl6c6bSdeRrzKtoiokqRThrjb\nRXZhVknSCrafkjS23f15rWNf6IsEJWkc8BvgYdLZ/yrAx2xfVWOz5lrzrLqyStd00IYZffGhM5K+\nZ/urwx0rMF5lW0TUQdKWtq8d7thcxqhlIbKk5Ui7WEPavbrwHQq6Wb8kqJuBDztv0iZpTdIajVIr\nLJdNA1vMN28vDyV9AEk6rOnmKFL34tK2ty8yTq8bZLp+X21EV6Qqag3WsRC5yhqD3apfxqDmc9MO\norbvkzRfnQ0qgu0hF36WoLlk1BukMalSKxT0EkkHAp8G1pTUXK5mMdIEhqLjrW176mClc0Z6yRxJ\n7wK2AJZtOXkaDRT9f2N5BhYif5hqFiJXWWOwK/VLgpos6SRmXk8wucb2jEhlLezsI2cDl9N+un4Z\nXTdfIE0v/2Gb+0wa4B/J5idt5TEvM588/QsotNpCrvIxEZjYtBB5kqQyFyJXWWOwK/VLF98CwMEM\n1M+6GjjeeYvoMLSqFgT3E0nrMrAz8dVVlQTqRZLGVjFxoOqFyFXWGOxWfZGgwtypakFwv5B0MOmE\n6ff50K7Az2wfX3CcPYa6v6Rp2JXLY8pfZNaFyIVdIda1ELkXawzOjp5OUIOtI2jop/UEoXtIuh3Y\nwrnydV4TdV3Rf4+9Og27VV7w/AtmXYhc2LhenQuR+1mvj0G9v+4G9IJYOFg4Aa813W5M/S6U7ao2\nDKzbG7Z/XmaAGhciV1JjsFv19BVUs35fTzA3YuFgMSTNa/sNSV8ijWM0ZkDuTpoR9j8lxt6ZVPGj\nuQL3t8qKVyVJE4CngfOZuZLEiK4TWWWNwW7VFwkq1hOEbtCysHozmibteGCr9DLi/oJUnmdb0g7I\nHySdpO1fVswqSXqozWG7hC0wqlRljcFu1S8J6jbgfa3rCWJhZGdyJYKhxvL6psthbtRRjirHvd32\nek3/LgpcYvvdwz451EbSsaT1V1XUGOxKvT4G1dD36wnmhu3FACR9G3gKOI10JbovsEKNTRtpWheU\nzsQF7nDbolGI9hVJK5L+/nvq96a022zrJoKn1teiQowmVYfZrumY6c3C1231S4KaKOlPzLye4JIa\n2zNS7dJy1fnzfHVa+GZtPWoe0sLSwidEDOOPkpYgdXPfQvqQO6niNpRGacfgcaQEdTGwI7lSfI3N\nmmt9NMllUH3RxQczZsQ09/n31XqCIuS9cH4GnEn6kBsPHGx7i1obNkLUVdC3pQ0LAAu6hzbvzLNL\n1wem2F4/T4g63fb7am7aXMkbFu7PrJNbemJ5QCd6uptL0uqStoTUb2v7MNuHAf+QtFrNzRuJPgzs\nDUzLX3vlY6EzVV85paDSwfkKilw9ZZSkz9TRlpL82/Z04A1Jo0kz+lapuU1FOI00BrU9cCWwMvBi\nrS2qWE8nKODHpLpcrV7I94XZYPth27vaXsb2srZ3s/1w3e0aQd5bU9wDbD/fuGH7OVKNvl4xOSfg\nX5IW694C/LXeJhViddvfAF62/RtSmaV31tymSvX6GNRytu9oPWj7Dklvqb45oZ/VuC5nHkly7s+X\nNA+p0GpPsN24GvyFpInAaNu319mmgjQ2VX0+TwL5O2nRbt/o9QS1xBD3LVRZK0Ko10TgLEkn5NsH\n5mM9Q9IuwNb55pVALySoEyUtCXyDVJh20fx93+jpSRKSzgD+YvuXLcc/RVoXtU89LRtZJB1q+9ii\ndykN1ZA0ipSUGl2MlwEn5S0kRjxJPyBVifltPjQeuMkl7VBcFUnz9MrvaE71eoJajlT+5DUGNoTb\nhNS9sbvtv9fVtpFE0q22N+iGWWghtMrFdzfIEyUaXZhTRnqNSEmPkq9+SSfavfthPYie7uKzPQ3Y\nQtK2pFL5ABfZ/kuNzRqJ7pF0P7Bi/jBoaBSvHNEfBL2qz6r5LwE0xvgWr7MhBVqbVPD6YOBXeV+2\nM21fU2+zqtPTV1ChOJKWB/4EzLI5YRSL7U6DFfdt6JXfm6TxwA+AK0gnTVsDX7Z9Vq0NK1AeizoW\n2Nd20dvZd61IUGG2SJofWDPfvNf260M9PnQfSVsB420fXHdb5pYkkdYHvcHMuxX0RPd93ix0H2AH\nYDJwlu1zh35W74gEFTqW/7OcCjxMOlNdBfiY7avqbFcYnqQNSYuq9wIeAs6zfVy9rSqGpDtsv6Pu\ndhRN0sPAFOBs4ELbLw/9jN7T02NQoXA/ArazfS/M2Gr7DGDjWlsV2sq/n/H56xnSYLtsb1trw4p3\ni6RNy9yypCbr2W5XaKBvxBVU6Fhju4bhjoXukLcpvxrY3/YD+diDI32fpFaSpgJrkK7sX6ZHJu9E\nLb64ggqzZ7Kkk4DT8+19Sf3ioTvtAXwIuCJXWDiTmuoBlmz7uhtQktOAqaT39y3S/7e+2l03rqBC\nx3Il7INpqgoPHJ8LkIYuJWkRYFdSV997SOOI59u+tNaGzaV8hfFpYHXgDuBk22/U26riNDa4bNpo\ncj7STgyb1922qkSCCqGP5OnKewH72K6reG0hJJ1Fqld3NWkPqEdsH1pvq4oj6Ubbm0m6CvgMqRbf\njb3WRTuUSFAhhBGpefaepHlJH949U+kkl2Q7F1gPOIVUi++btn9Ra8MqFAkqhDAitZbeilJcvScS\nVAhhRJL0JmnWHqTJHwsBrzAwi290XW2bG5IOG+p+2z+qqi11i1l8YViS1mvsr5MHag8HNgPuBL5j\n+5U62xf6Uw+X/Fms7gZ0i7iCCsNq7jqR9ENgaVKf+G7A0rY/Wmf7Quglko6yfbikvWyfU3d76hQJ\nKgyrMd01f38rsKnt13MdtNtG+oLIELpJrkK/HnBzv4+pRRdf6MTiknYHRgELNQrE2rakOMMJoVgT\ngeeARSU1lzoa0WNrcyKuoMKwJJ3ScujLtqflLTh+O9LX04TQjSRdYHvXuttRp0hQIYQQutKouhsQ\nRjZJ76u7DSH0Ikl7SLpf0guS/iXpxZYuv54XV1Bhrkh61PaYutsRQq+R9ADwAdt9VSC2WUySCMOS\ndOFgd5GmnIcQijetn5MTRIIKnXk3sB/wUstxkRbshhCKNzkXxP09MGPHANvn1dekakWCCp24HnjF\n9pWtd0i6t4b2hNAPRpNKN23XdMxA3ySoGIMKIYTQlWIWXwghdCFJK0s6X9LT+etcSSvX3a4qRYIK\nw5K0tqRLJF0kaTVJv5b0vKQbJb2t7vaF0KNOAS4EVsxff8jH+kYkqNCJE4HjgdOBv5BKsSwJfBv4\naY3tCqGXLWv7FNtv5K9fA8vW3agqRYIKnVjM9h9snwG8bvtMJ38gJaoQQvGelbSfpHny137As3U3\nqkqRoEInmvfdad0sbf4qGxJCH/kksDfwd+Ap4IPAJ2ptUcVimnnoxM8kLWr7JdvHNw5KWh34c43t\nCqFn2X4E2KXudtQpppmHEEIXkfTNIe627W9X1piaRRdf6IikbSWdJ+mu/PU7SePqblcIPejlNl8A\n+wOH19WoOsQVVBiWpJ1Js/W+BdxCKnG0EfB14LO2L66xeSH0LEmLAYeSktPZwA9tP11vq6oTCSoM\nS9Ik4FDbt7UcXw84zvY2tTQshB4laSngMGBf4DfAsbafq7dV1YtJEqETy7cmJwDbt0taro4GhdCr\nJB0D7EFaf/gO261FmvtGXEGFYUm62fbGs3tfCGH2SZpOql7+Bqk47Iy7SJMkRtfSsBrEFVToxGqD\n7Akl4K1VNyaEXmY7Jq9lcQUVhiVpyDGmdttwhBDC3IoEFTomaUFg9XzzAduv1tmeEEJvi0vJMCxJ\n80o6GnicNKPoVOAxSUdLmq/e1oUQelUkqNCJY4ClgFVtb2x7I2A1YAngf2ptWQihZ0UXXxiWpPuB\nNd3yxyJpHmCq7TXqaVkIoZfFFVTohFuTUz74JjNPgw0hhMJEggqduFvSR1sP5v1pptbQnhBCH4gu\nvjAsSSsD5wL/Bm7OhzcBFgJ2t/1EXW0LIfSuSFChY5LeA6ybb95t+/I62xNC6G2RoMKwJN0MXANc\nAkyK9U8hhCpEggrDkjQvsBWwA7At8CzwJ+AS2/fV2bYQQu+KBBVmm6QVSclqB9J6qBtsf6beVoUQ\nek0kqDBXJI0C3mX72rrbEkLoLZGgwrByF9/+wO7AivnwE8AFwMm2X6+rbSGE3hUJKgxL0hnA86Q6\nfI/nwysDHwOWsr1PXW0LIfSuSFBhWJLus73m7N4XQghzIypJhE78U9JeebwJSGNPkvYBnquxXSGE\nHhYJKnTiQ8AHgWmS7svFY6cBe+T7QgihcNHFF2aLpKUBbD9bd1tCCL0trqBCRyRtLWmtnJjWlvRF\nSTvX3a4QQu+KK6gwLEk/BjYD5iVVkHgvqezRNsAU2/9dY/NCCD0qElQYlqS7gLeTqpc/Aaxk+5W8\n3fsU22+vtYEhhJ4UXXyhE40NC6c3bud/pxN/QyGEksxbdwPCiHCRpKuBBYGTgLMlXU/q4ruq1paF\nEHpWdPGFjkh6F+lK6npJq5HKHj0K/M729KGfHUIIsy8SVAghhK4U4wdhWJJWkXSmpKslfTVPjmjc\n9/s62xZC6F2RoEInfgVMAg4BVgCubCzYBcbW1agQQm+LSRKhE8va/kX+/hBJ+wFXSdqFgRl9IYRQ\nqEhQoRPzSVrQ9qsAtk+X9HfSot1F6m1aCKFXRRdf6MRJwDubD9j+M7AXcGctLQoh9LyYxRdCCKEr\nxRVU6IikbSWdJ+mu/PU7SePqblcIoXdFggrDylXLfwX8AfgwsC9wMfArSTvV2bYQQu+KLr4wLEmT\ngENt39ZyfD3gONvb1NKwEEJPiyuo0InlW5MTgO3bgeVqaE8IoQ9EggqdeHkO7wshhDkW66BCJ1aT\ndGGb4wLeWnVjQgj9IcagwrAkDTnGZPvKqtoSQugfkaBCxyQtCKyebz7QqCwRQghliDGoMCxJ80o6\nGngc+A1wKvCYpKObK5uHEEKRIkGFThwDLAWsantj2xsBqwFLAP9Ta8tCCD0ruvjCsCTdD6zplj8W\nSfMAU22vUU/LQgi9LK6gQifcmpzywTeJ7TZCCCWJBBU6cbekj7YezPtCTa2hPSGEPhBdfGFYklYC\nzgP+DdycD28CLATsbvuJutoWQuhdkaBCxyS9B1g337zb9uV1tieE0NsiQYUQQuhKMQYVQgihK0WC\nCiGE0JUiQYUQQuhKkaBCCCF0pUhQIYQQutL/AxFSeuFN1S1cAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1fe52c0f048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# feature_importances_ from random forest classifier records this info\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "feat_labels = df_wine.columns[1:]\n",
    "\n",
    "forest = RandomForestClassifier(n_estimators=10000,\n",
    "                                random_state=0,\n",
    "                                n_jobs=-1)\n",
    "\n",
    "forest.fit(X_train, y_train)\n",
    "importances = forest.feature_importances_\n",
    "\n",
    "indices = np.argsort(importances)[::-1]\n",
    "\n",
    "for f in range(X_train.shape[1]):\n",
    "    print(\"%2d) %-*s %f\" % (f + 1, 30, \n",
    "                            feat_labels[indices[f]], \n",
    "                            importances[indices[f]]))\n",
    "\n",
    "plt.title('Feature Importances')\n",
    "plt.bar(range(X_train.shape[1]), \n",
    "        importances[indices],\n",
    "        color='lightblue', \n",
    "        align='center')\n",
    "\n",
    "plt.xticks(range(X_train.shape[1]), \n",
    "           feat_labels[indices], rotation=90)\n",
    "plt.xlim([-1, X_train.shape[1]])\n",
    "plt.tight_layout()\n",
    "#plt.savefig('./random_forest.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(124, 3)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "threshold = 0.15\n",
    "\n",
    "if False:\n",
    "    # deprecation warning\n",
    "    X_selected = forest.transform(X_train, threshold=threshold)\n",
    "else:\n",
    "    selector = SelectFromModel(forest, threshold=threshold, prefit=True)\n",
    "    X_selected = selector.transform(X_train)\n",
    "    \n",
    "X_selected.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now, let's print the 3 features that met the threshold criterion for feature selection that we set earlier (note that this code snippet does not appear in the actual book but was added to this notebook later for illustrative purposes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 1) Color intensity                0.182483\n",
      " 2) Proline                        0.158610\n",
      " 3) Flavanoids                     0.150948\n"
     ]
    }
   ],
   "source": [
    "for f in range(X_selected.shape[1]):\n",
    "    print(\"%2d) %-*s %f\" % (f + 1, 30, \n",
    "                            feat_labels[indices[f]], \n",
    "                            importances[indices[f]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary\n",
    "\n",
    "Data is important for machine learning: garbage in, garbage out.\n",
    "So pre-process data is important.\n",
    "This chapter covers various topics for data processing, such as handling missing data, treating different types of data (numerical, categorical), and how to avoid over-fitting which can improve both accuracy and speed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reading\n",
    "* PML Chapter 4\n",
    "* IML Chapter 6.2"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}