{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Copyright (c) 2015, 2016 [Sebastian Raschka](sebastianraschka.com)\n",
    "<br>\n",
    "2016 [Li-Yi Wei](http://liyiwei.org/)\n",
    "\n",
    "https://github.com/1iyiwei/pyml\n",
    "\n",
    "[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Python Machine Learning - Code Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Chapter 7 - Combining Different Models for Ensemble Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last updated: 2016-11-01 \n",
      "\n",
      "CPython 3.5.2\n",
      "IPython 4.2.0\n",
      "\n",
      "numpy 1.11.1\n",
      "pandas 0.18.1\n",
      "matplotlib 1.5.1\n",
      "scipy 0.17.1\n",
      "sklearn 0.18\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a '' -u -d -v -p numpy,pandas,matplotlib,scipy,sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "*The use of `watermark` is optional. You can install this IPython extension via \"`pip install watermark`\". For more information, please see: https://github.com/rasbt/watermark.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Overview\n",
    "\n",
    "- [Learning with ensembles](#Learning-with-ensembles)\n",
    "- [Implementing a simple majority vote classifier](#Implementing-a-simple-majority-vote-classifier)\n",
    "  - [Combining different algorithms for classification with majority vote](#Combining-different-algorithms-for-classification-with-majority-vote)\n",
    "- [Evaluating and tuning the ensemble classifier](#Evaluating-and-tuning-the-ensemble-classifier)\n",
    "- [Bagging – building an ensemble of classifiers from bootstrap samples](#Bagging----Building-an-ensemble-of-classifiers-from-bootstrap-samples)\n",
    "- [Leveraging weak learners via adaptive boosting](#Leveraging-weak-learners-via-adaptive-boosting)\n",
    "- [Summary](#Summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Added version check for recent scikit-learn 0.18 checks\n",
    "from distutils.version import LooseVersion as Version\n",
    "from sklearn import __version__ as sklearn_version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Learning with ensembles\n",
    "\n",
    "Challenge:\n",
    "* No single classifier best for all circumstances\n",
    "* Hard to manually decide which classifier/model to use\n",
    "\n",
    "Solution:\n",
    "* Combine multipler classifiers for better performance than each individual classifier\n",
    "* Need diverse, not just accurate, individual classifiers\n",
    "\n",
    "<a href=\"https://en.wikipedia.org/wiki/The_Wisdom_of_Crowds\"><img src=\"https://upload.wikimedia.org/wikipedia/en/9/95/Wisecrowds.jpg\" align=right></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Unanimity\n",
    "* no voting necessary\n",
    "\n",
    "#### Majority voting\n",
    "* for binary classification\n",
    "\n",
    "For sample $x$, each classifier $j$ decides whether the class $C_j(x)$ is $+1$ or $-1$\n",
    "\n",
    "Majority voting is then:\n",
    "$$\n",
    "C(x) = sign\\left[ \\sum_j C_j(x) \\right] = \n",
    "\\begin{cases}\n",
    "+1 \\; if \\; \\sum_j C_j(x) \\geq 0\\\\\n",
    "-1 \\; else\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "#### Plurality voting\n",
    "* extension of majority voting for multi-class setting\n",
    "\n",
    "$$\n",
    "C(x) = mode\\{C_1(x), C_2(x), \\cdots, C_m(x) \\}\n",
    "$$\n",
    "\n",
    "<img src='./images/07_01.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Individual classifiers can be the same or different types\n",
    "* decision tree, svm, logistic regression, etc.\n",
    "\n",
    "Random forest\n",
    "* combines multiple decision trees\n",
    "\n",
    "<img src='./images/07_02.png' width=70%> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why ensemble can better than individual classifiers\n",
    "\n",
    "A simple math model:\n",
    "* binary classification\n",
    "* $n$ base classifiers\n",
    "* each with error rate $\\epsilon$\n",
    "* classifiers make independent decisions\n",
    "\n",
    "Probability that at least $K$ classifiers are wrong:\n",
    "$$\n",
    "\\begin{align}\n",
    "\\epsilon_{ensemble} (K)\n",
    "&= \n",
    "\\sum_{k=K}^n C\\left(n, k\\right) \\epsilon^k \\left(1-\\epsilon\\right)^{n-k}\n",
    "\\end{align}\n",
    "$$\n",
    ", where $C\\left(n, k\\right)$ is the combinatorial, i.e. binomial coefficient of n choosing k.\n",
    "\n",
    "For the ensemble to be wrong, $K \\geq \\frac{n}{2}$:\n",
    "$$\n",
    "\\epsilon_{ensemble} = \\epsilon_{ensemble}\\left( \\left\\lceil \\frac{n}{2} \\right\\rceil \\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For example, if\n",
    "* $n = 11$\n",
    "* $\\epsilon = 0.25$\n",
    "\n",
    "$\\epsilon_{ensemble} = 0.034$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The above assumes classifiers are making independent decisions!\n",
    "\n",
    "Humans don't always make independent decisions\n",
    "* elections\n",
    "* financial markets\n",
    "\n",
    "<a href=\"https://books.google.com.hk/books/about/Manias_Panics_and_Crashes.html\">\n",
    "<img src=\"https://books.google.com.hk/books/content?id=Er-6QkkQkeEC&printsec=frontcover&img=1&zoom=1&edge=curl&imgtk=AFLRE73PgBQLrAIbIXBTXGzeLHPtVy8RRIMF2BN_ugz1l41rEK14FFNcYrKhN2pcU0jdU0vFvIPMCmpWAqKPnJ1PxjnQLzckEiNGJ74SMKbIl3DL2Ct7tkQEgTsRyZl-t_LQeA5W-nc3\" align=right>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# combinatorial C(n, k)\n",
    "from scipy.misc import comb\n",
    "import math\n",
    "\n",
    "def ensemble_error(n_classifier, error):\n",
    "    k_start = math.ceil(n_classifier / 2.0)\n",
    "    probs = [comb(n_classifier, k) * error**k * (1-error)**(n_classifier - k)\n",
    "             for k in range(k_start, n_classifier + 1)]\n",
    "    return sum(probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "**Note**\n",
    "\n",
    "For historical reasons, Python 2.7's `math.ceil` returns a `float` instead of an integer like in Python 3.x. Although Although this book was written for Python >3.4, let's make it compatible to Python 2.7 by casting it to an it explicitely:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from scipy.misc import comb\n",
    "import math\n",
    "\n",
    "def ensemble_error(n_classifier, error):\n",
    "    k_start = int(math.ceil(n_classifier / 2.0))\n",
    "    probs = [comb(n_classifier, k) * error**k * (1-error)**(n_classifier - k)\n",
    "             for k in range(k_start, n_classifier + 1)]\n",
    "    return sum(probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.034327507019042969"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ensemble_error(n_classifier=11, error=0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "error_range = np.arange(0.0, 1.01, 0.01)\n",
    "ens_errors = [ensemble_error(n_classifier=11, error=error)\n",
    "              for error in error_range]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FGXXwOHfk5DQCaFKRxCkSLUhIsVCUVB8kSIKgqKA\niAoqYsGCr698gogo0gQBFcFOV0QJiChYCL0rvRNaCCFlz/fHLCGBlE2Zndnk3Ne1lzuzk5mTx2FP\n5qlGRFBKKaXcKMjpAJRSSqm0aJJSSinlWpqklFJKuZYmKaWUUq6lSUoppZRraZJSSinlWrYmKWPM\nFGPMYWPMunSOGWuM2W6MiTTGNLQzHqWUUoHF7iepj4E2aX1ojGkHVBeRGkBfYILN8SillAogtiYp\nEVkBnEjnkHuAGd5jVwFhxpiydsaklFIqcDjdJlUB2Jtse793n1JKKeV4klJKKaXSlM/h6+8HKiXb\nrujddxljjE4yqJRSAUpETFZ+zh9JynhfqZkLDABmG2OaACdF5HBaJ9LJcDP22muv8dprrzkdhutp\nOWVMyyhtIrB3L/z1F4wd+xqlS7/Gxo2wbRskJKT/swULQtmyUKYMlCwJ4eHWq3hxKFbMehUtCkWK\nQKFC1vEFC0KBApA//8VXSEjKV5BL6sXiE+Np/WlrInZFAHBnjTtZ+MDCLJ/P1iRljJkJtARKGmP2\nAK8CoYCIyCQRWWiMudMYswM4C/S2M568YNeuXU6HEBC0nDKmZXTRuXOwahUsXw6//w5//glHj174\ndFeKYytWhOrV4aqr4MoroVIl61WxIpQrZyWf3CwkOIS6peuy6egmxrYdS5e6XQh6IOsZ1NYkJSLd\nfTjmCTtjUEqpzPJ4rKekRYtg8WJYvRri41MeU6IEXHcd7N8PQ4ZA3bpQu7b19JPXvXXbWwxvNZwS\nBUtk+1xOt0mpHNarVy+nQwgIWk4Zy2tlFBtrJaSvv7aS08UnJTAGGjaE5s3h5pvh+uuhalVrf0RE\nL1q2dCpqZ8UnxhMSHHLZ/qL5i+bYNUygtPMYYyRQYlVKBYaEBPj+e/j8c5g3D86cufhZlSrQrh20\nbQstWlhtRuqiBdsWMGDhAD77z2fcXPnmdI81xri644Stqlatyu7du50OQ7lUlSpVUm1biYiIoGVe\n/fPXR7m5jLZuhY8/hunT4dChi/sbNYLOnaFjR6hVy3pSykhuLqfUHI4+zFPfP8XsjbMBGP/n+AyT\nVHYEfJLavXu39vpTaTK+fMuoPMHjgYULYcwY+Omni/tr1oSHHoIuXazODip1IsLHkR/z7OJnORF7\ngkIhhfhvq/8y8MaBtl434Kv7vI+RDkSkAoHeH+rcOZgyBd57D3bssPYVLgxdu8LDD0PTpr49MeV1\nJ2NPUvP9mhyNOUqb6m0Yf9d4rgy/0qefzU51nyYplavp/ZF3xcTAhAnw9ttw2Dv6skoVePJJKzlp\nG1Pmfbv5W2LiY+her3umaimyk6RcMvxLKf+KiIhwOgTXC9QyOn8e3n3XGqP0zDNWgrr2WvjqK+tJ\navDgnE1QgVpOWXFv7Xt5oP4Dfq1G1ySVxy1btoxKlSql+Xnv3r155ZVX/BiRUlkjAl98YY1VGjwY\njhyxuorPnw9//AGdOkG+gG+Ft9/ZuLOM/HUkCZ4Mps7wE01SNqtatSqFChWiWLFiFC1alGLFivHk\nk086HVYKebFzQV7qjZVVgVRGf/xhjV/q2hX+/Rfq1LGS06pVcNdd9rY5BVI5ZeSHHT9wzfhrGLJk\nCGN+H+N0OEAu6N3ndsYYFixYQKtWrZwOxdU8Hg9Bl0w+ltq+9CQmJhIcHJzToSkXO3UKXnoJPvzQ\nepIqUwaGD4dHHtGnpsw4evYog34YxGfrPwOg4RUNaVXVHd9Z+iTlB2k13E+fPp1bbrmF5557jhIl\nSlC9enW+//77pM+nTZtG9erVKVasGNWrV+fzzz9P+mzq1KnUqVOHkiVL0q5dO/bs2ZP0WVBQEOPH\nj6dmzZqEhYXxyiuv8M8//3DzzTdTvHhxunXrRkKyWTBFhLfeeovSpUtTrVo1Zs6cmebvMn/+fBo1\nakR4eDjNmjVj/fr1aR67ZcsWWrduTcmSJalduzZffvll0me9e/fm8ccf56677qJo0aJERESkuu/0\n6dP07NmTMmXKcOWVV/Lmm2+mKL9mzZoxePBgSpUqxeuvv55mLJfKS+0IWeXmMhKBL7+0qvbGjbMm\nV33uOavNqW9f/yYoN5eTL3ZE7aD2uNp8tv4zCuQrwP/d/n+s7rOaa8tf63RoFhEJiJcV6uXS2n/x\n85x5ZVXVqlXlp59+SvWzadOmSWhoqEyZMkU8Ho+MHz9eypcvLyIiZ8+elWLFisn27dtFROTQoUOy\nadMmERH57rvvpEaNGrJ161ZJTEyUN998U5o2bZp0XmOMdOzYUaKjo2XTpk2SP39+uf3222XXrl1y\n+vRpqVOnjsyYMUNERCIiIiRfvnzy7LPPSlxcnCxbtkwKFy4s27ZtExGRXr16ybBhw0RE5O+//5Yy\nZcrIH3/8IR6PR2bMmCFVq1aVuLi4y363s2fPSqVKlWT69Oni8XgkMjJSSpUqJZs3b046b/HixeW3\n334TEZHY2NhU9/Xo0UM6duwoZ8+elV27dknNmjVl6tSpSeWXL18+GTdunCQmJkpsbOxlcaR1fyxd\nujSt/2XKy61ldPSoSKdOF/9tNmkisnatc/G4tZx85fF4pMXHLeS26bfJ9uPbbbmG999h1r77s/qD\n/n4FcpIqWrSohIeHS/HixSU8PFw++ugjEbG+ZGvUqJF0bExMjBhj5PDhw3L27FkJDw+Xb775Rs6d\nO5finO3atUv6ohYRSUxMlEKFCsmePXtExEpSF77oRUSuvfZaefvtt5O2n3nmGRk0aJCIWEkqJCQk\nxTW6dOki//3vf0UkZZLq37+/vPLKKyliufrqq2X58uWX/d6zZ8+W5s2bp9jXt29fGT58eNJ5H3ro\noRSfX7ovMTFRQkNDZcuWLUn7Jk6cKK1atUoqvypVqlx27eQyuj9UYJk3T6RsWevfZJEiIuPHiyQm\nOh1V4Dt57qR4PB7bzp+dJJXrq/tyKk1lx5w5c4iKiuLEiRNERUXxyCOPJH12xRVXJL0vWLAgANHR\n0RQqVIjZs2czfvx4ypUrR4cOHdi2bRtgzbLx1FNPUaJECUqUKEHJkiUxxrB//8X1IsuUKZPivGXL\nlk2xHR0dnbQdHh5OgQIFkrarVKnCgQMHLvs9du/ezTvvvJN03fDwcPbt25fmsb///nuKY2fOnMnh\nwxeXC0utV2HyfceOHSMhIYHKlSuniC3575lez0SVe8TEwGOPQYcOVpfy5s1h3Tro18896ygFglOx\np1LdH1YgzLUdqPR/rx9IFrPcHXfcweLFizl06BBXX301jz76KGB9MU+cOJGoqKik5BcdHU2TJk2y\ndJ0TJ05w7ty5pO09e/ZQvnz5y46rVKkSL7300mXX7dq1a6rHtmzZMsWxp0+f5oMPPkg6JrV/FMn3\nlSpVipCQkBRzM+7evZsKFSqkew5fBHo7gj+4pYy2boUmTWDyZGuxv3fegaVLrXFQbuCWckpPTHwM\nQ34cQrWx1dh/OtXFz11Lk5RLHTlyhLlz5xITE0NISAhFihRJ6unWr18//ve//7Fp0yYATp06xVdf\nfZXla4kIr776KvHx8fzyyy8sWLCALl26XHbco48+yoQJE1i9ejUAZ8+eZeHChZw9e/ayY9u3b8+2\nbdv49NNPSUhIID4+nj///JOtW7f6HFdQUBBdunThpZdeIjo6mt27d/Puu+/So0ePLP+uKrB8/rm1\nZtP69dYce6tXW2Og9OnJd0v+WUK98fUYuXIkJ86dYPHOxU6HlCn6v9oPOnToQLFixZJenTp1SvPY\nC08GHo+H0aNHU6FCBUqVKsXy5csZP348AB07dmTo0KF069aN4sWLU79+/RS9Ai99usjoaaNcuXKE\nh4dTvnx5evTowcSJE6lRo8ZlP3vttdcyefJknnjiCUqUKEHNmjWZPn16qucsUqQIixcvZtasWZQv\nX57y5cszdOhQzp8/n+HvntzYsWMpVKgQ1apVo3nz5jz44IP07p39BZxz09gWuzhZRgkJ1vRF3btD\ndDR062athlu/vmMhpcmt99LxmOP0+q4Xd3xyB/+c+Id6Zerx2yO/0btRYC2ArnP3qVxN74/Ac+KE\nNSP5kiUQGmrNWt6vn04Cm1nrDq/j2knXEmyCeaXFKzzX9LlUFyj0B527T6lMCoR2BKc5UUZbtsCN\nN1oJqkwZq+2pf393Jyi33kv1y9ZncofJrOu/jhdvedGxBJVdOiZbKeUKP/8M//mPNYtEgwYwdy4k\n69ipsqBXw15Oh5BtWt2ncjW9PwLD7NnQowfEx8O998KMGVCkiNNRBYZ1h9cxd+tcXm7+stOhpClP\nLx+vlApsY8bAoEHW+6eegtGjtfeeL87Fn+ON5W8wcqU1Y/kNFW6gdfXWToeV4/RWUHmSW9sR3MTu\nMhKB55+/mKDefttaByrQEpQT91LErggaTGjAWyveItGTyIDrB9CkYtbGSbqdPkkppfzO44EnnoDx\n463JYKdOtar7VMZmb5hNt6+7AVCndB0md5hM00pNHY7KPtompXI1vT/cJzHRmql8yhRrBomvv7bW\nfFK+iY6LpvHExjxY/0Gev/l58ufL73RIGcpOm5QmKZWr6f3hLgkJ0Ls3fPopFCwIc+bAHXc4HVXg\niUuMIzQ41OkwfKbjpJTKJG2TylhOl1FiIvTsaSWowoVh0aLckaDsupc84mHf6X2pfhZICSq7NEnZ\nLPny8SVLlqRDhw4pZvFWKi/weKBPH2suvqJFYfFiaNHC6ajca9PRTdzy8S3cNuM2YhNinQ7HUZqk\nbHZh+fjTp09z8OBBypQpw8CBA50OK02JiYk+7cvsOdzGrfOtuUlOlZEIDBwI06ZBoUKwcCE0zUXt\n/Dl5L51POM+rS1+l4YSGrNy7kjPnz7AjakeOnT8QaZLygwttIqGhodx3331Js5cDLFy4kMaNGxMW\nFkaVKlVSLIF+/vx5evToQalSpQgPD+fGG2/k6NGjAJw+fZo+ffpQvnx5KlWqxLBhw9JsexERRowY\nwVVXXUXp0qXp1q0bJ0+eBKylL4KCgpg6dSpVqlThtttuS3UfwNy5c7nmmmsoUaIEt956K1u2bEm6\nxpVXXsnbb79NgwYNKFKkCB6PJ2cLUQUkERgyBD780OokMWcONGvmdFTutHLvShpObMjw5cOJ98TT\n99q+bBqwiWvKXON0aI7KE0nKvG5Sffl6fE6JiYlh9uzZ3HTTTUn7ihQpwieffMKpU6dYsGABEyZM\nYO7cuQBMnz6d06dPs3//fqKiopgwYULSwogPPfQQoaGh/PPPP6xZs4Yff/yRjz76KNXrjh07lrlz\n5/LLL79w4MABwsPDefzxx1Mcs3z5crZs2cIPP/yQ6r7t27fTvXt3xo4dy9GjR2nXrh0dOnQgISEh\n6fhZs2axaNEiTp48mbSsiFtpm1TGcqKM3noLRo2yupl/9RXcfnv243KbnLqX9p/ez5ZjW7i65NUs\n67WMCe0nULxA8Rw5d0DL6pK+/n6RxeXjRUR4jVRfvh6fHcmXjw8JCZEKFSrIhg0b0jz+6aeflsGD\nB4uIyNSpU+Xmm2+WdevWpTjm8OHDkj9/fomNjU3a9/nnnyctq36p2rVry88//5y0feDAAQkJCZHE\nxETZtWuXBAUFya5du5I+T23fG2+8IV27dk3a9ng8UqFCBVm2bFnS7zlt2jRfisSv0ro/li5d6t9A\nAlB2y2jKFGtda2NEZs/OmZjcKKfuJY/HI5+s/UTOxZ/LkfO5CdlYPj5PDOaVVzPXBTmzx2dkzpw5\ntGrVChHhu+++o3nz5mzevJkyZcqwatUqXnjhBTZs2EBcXBxxcXF07twZgB49erBv3z66devGqVOn\nePDBB3nzzTfZvXs38fHxlCtXzorX+z+zchqzce7evZt777036elGRAgJCUmxlHvFihUv+7nk+w4c\nOECVKlWSto0xVKpUKUUnkNTO4VbaJpWx7JTR/PnWcu8AH3xgLb2RW+XUvWSM4cH6D+bIuXITd9fJ\n5BLibSsyxnDvvfcSHBzMihUrAHjggQfo2LEj+/fv5+TJk/Tt2zfp+Hz58jFs2DA2btzIypUrmTdv\nHjNmzKBSpUoUKFCA48ePJy3NfvLkSdatW5fq9StXrsyiRYtSLOV+9uzZpCR3IbZLJd9Xvnz5FMu4\nA+zduzdFYsrqUu4qd/n9dyspJSbCSy/BJTXLeZpHPEz8cyLvr3rf6VAChiYpP5szZw4nT56kTp06\nAERHRxMeHk5ISAirV69m5syZScdGRESwYcMGPB4PRYoUISQkhODgYK644gpat27NoEGDOHPmDCLC\nP//8w/Lly1O9Zt++fXnxxRfZs2cPAEePHk1q94KLSTS5S/d16dKFBQsWsHTpUhISEhg1ahQFChRI\n0b4WSLRNKmNZKaMdO6B9ezh3Dh5+GN54I+fjchtfy2nLsS20mNaCfgv6MWTJEPaf1qEovtAk5QcX\nlo8PCwtj2LBhzJgxg1q1agHw4YcfMmzYMMLCwvjvf/9L165dk37u0KFD3HfffYSFhVG3bl1atWrF\ngw9a1QEzZswgLi6OOnXqUKJECTp37syhQ4dSvf5TTz3FPffcQ+vWrQkLC6Np06asXr066fOMnqIA\natasyaeffsoTTzxB6dKlWbBgAfPmzSNfvnxpnkPlLSdOWNMbHT8Od94JEye6e7FCf4lLjOONZW/Q\nYEIDVuxZQdnCZZnRcQbli5Z3OrSAoNMiqVxN7w//iI+Htm2thQvr14cVK6xBuwr6ze/HxL8mAvBI\no0cYecdIwguGOxyVf+ncfQHyOyj/0/vDfiLQrx9MmgRly8Lq1bqibnI7onbQ6YtOjGkzhlZXtnI6\nHEe4eu4+Y0xbY8wWY8w2Y8zzqXxezBgz1xgTaYxZb4zpZXdMSmmbVMZ8LaMxY6wEVaBA3lzyPaNy\nuqrEVUT2jcyzCSq7bE1Sxpgg4AOgDVAXuN8YU+uSwwYAG0WkIdAKeMcYkye6xisV6JYsgWeftd5P\nnw433OBsPE46HH2YA2cOpPqZttlmnd1PUjcA20Vkt4jEA7OAey45RoALtddFgeMikoBSNtJxUhnL\nqIz+/Re6drUmj3355dw9Fio9LVq0YOqaqdQeV5u+8/tq9XIOs/uJpQKwN9n2PqzEldwHwFxjzAGg\nCNAVpZSrxcTAvfdCVJTVky/ZlJN5yvbj2+k7vy9Ldy0FIMGTQEx8DIVDCzscWe7hhi7obYA1IlIe\naASMM8YUcTgmlctpm1TG0iojEWvZjbVroUYN+OwzcPlUjbZ4Z+U71Btfj6VLl1KqUCk++89nLOy+\nUBNUDrP7SWo/kLwZtaJ3X3K9gbcARGSnMeZfoBbw56Un69WrF1WrVgWgePHiNGzY0IaQVW504Qv3\nQhVWZGRkiu1LP9ftCCIjI1P9/L334PPPIyhQAL77riXFi7sjXn9vr12zlvOJ52l9VWser/c4YcfD\nktqe3BCfk9tjxowhMjIy6fs6O2ztgm6MCQa2ArcBB4HVwP0isjnZMeOAIyLyujGmLFZyaiAiUZec\nK9Uu6FWrVr1suh6lLqhSpQq7du1yOoxc47ffoHlzaxn4r76CTp2cjsg55xPO89u+32hZtaXTobie\nq8dJGWPaAu9hVS1OEZERxpi+WLPiTjLGlAOmARcmkntLRD5P5TypJimllH8cOwaNGsG+fTBoEIwe\n7XREKlC4epyUiHwvIleLSA0RGeHdN1FEJnnfHxSRNiJS3/u6LEEp32lbi2+0nDKWvIw8HujRw0pQ\nTZrAiBHOxeVPR88e5YFvHmDBtgVpHqP3kr10PJJSKkNvvQXffw8lS8IXX0BoqNMR2UtEmLF2BoMX\nDybqXBR/HfiLdjXaEWTyYA8RhwX8tEhKKXstXw6tWllPU4sWWXP05WY7o3bSb0E/lvyzBIDbq93O\nhLsmUL1EdYcjC1zZqe7TJymlVJqiouCBB6wE9cILuT9BiQh3z7qbTUc3UaJgCd5t8y496vfQGSMc\npM+uuYzWj/tGyyljS5dG0KfPxXaovDBg1xjD6Naj6V6vO5sHbKZng54ZJii9l+ylT1JKqVTNmwff\nfgvFisHMmRAS4nRE/tHmqja0uaqN02EoL22TUkpdZuNGuO46iI2FWbOsOfpym6X/LqVJxSYUDCno\ndCi5nqu7oCulAktsLNx/v/Xf3r1zX4I6HnOch757iFtn3Moby/PA+vYBTpNULqP1477RckrbsGGw\nfj1UqBDB2LFOR5NzRITP1n1GrXG1mLF2BvmD81OiYIlsn1fvJXtpm5RSKklEBLzzDgQHw4svQpFc\nMtXzufhz/OeL//D9ju8BaFm1JZPaT6JGyRoOR6Yyom1SSikATp2C+vVhzx549VV47TWnI8pZ986+\nl4hdEbzT+h16N+yt3cr9yNVz9+UUTVJK2atnT/jkE7j+evj119zXm+9Q9CEArihyhcOR5D3acUIl\n0fpx32g5pfTVV1aCKljQ+m9ISOCWUVp/zF5R5ApbElSgllOg0CSlVB535Aj072+9HzUKrr7a2Xiy\nY+m/S6k/oT7bjm9zOhSVQ7S6T6k8TAQ6d4avv4bbboMff4RAbKo5ce4Ezy5+lqmRUwHo06gPk++e\n7HBU6gJtk1JKZcns2dCtm9WLb8MGqFLF6YgyR0T4ctOXDFw0kCNnjxAaHMqw5sMYcvMQQoNz+VTt\nAcS2NiljTJAxpmnWwlJO0Ppx32g5waFD8Pjj1vvRoy9PUIFQRoeiD9Hru14cOXuEZpWbsbbfWl5u\n/rJfE1QglFMgS3eclIh4vMu7N/JTPEopPxCx2qGioqB1a+jTx+mIsqZc0XK80/odgkwQj177qK73\nlAtlWN1njBkF/AZ842R9m1b3KZVzZs2ypj4qVsyq5qtUyemIVG5mdxf0vsCXQJwx5rQx5owx5nRW\nLqaUct6xYzBwoPV+1KjASFCxCbFMj5yeZvdylXtlmKREpKiIBIlIiIgU824X80dwKvO0ftw3ebmc\nnn7aSlStWqVfzeeWMvpl9y80nNCQXnN6MWvDLKfDuYxbyim38mnuPmPM3UBz72aEiMy3LySllF0W\nLIDPPrMG7U6e7O7u5qdiT/H8kueZ+NdEAGqVqkXlsMoOR6X8zZc2qRHA9cBn3l33A3+KyAs2x3Zp\nHNompVQ2nD4NderA/v3WJLKDBzsdUdo2Hd3E7TNu52D0QUKCQnih2Qu8eMuL5M+X3+nQVBbYOk7K\nGLMOaCgiHu92MLBGROpn5YJZpUlKqezp3x8mTIAbboCVK62Zzt0qLjGORhMbEZY/jMkdJlO3TF2n\nQ1LZ4I+5+4onex+WlQsp/9D6cd/ktXJascJKUPnywUcf+ZagnCyj0OBQfuzxIyseXuH6BJXX7iV/\n86VN6i1gjTFmKWCw2qaG2hqVUirHnD8Pjz1mvR86FOrVczaeS51POJ9qNV75ouUdiEa5TbrVfcZa\ncKUikIDVLgWwWkQO+SG2S2PR6j6lsuCNN+CVV6BGDVi3DgoUcDoiS1xiHCNWjGD62ums6buGYvm1\n03BuZXeb1HoRcfxvL01SSmXe1q3WQoZxcfDzz1a3czdYuXclj857lE1HNwHw2X8+o3u97g5Hpexi\nd5vU38aY6zM+TLmB1o/7Ji+Ukwj062clqN69M5+g7Cij0+dP88TCJ2g2tRmbjm6iRokaLH1oaUAn\nqLxwLznJlzapG4EHjDG7gbNY7VLi7959SqnMmTYNIiKgVCkYOdLpaCx/HviTcX+MI19QPoY0HcLL\nzV+mYEhBp8NSLuZLdV+qk/eLyG5bIko7Dq3uU8pHx49bixceP26ttPvgg05HdNGIFSO4s8ad1C+r\nf+fmFba1SXnHRG0UkVpZDS6naJJSynd9+sCUKYG9kKHKPWxrkxKRRGCrMUbnIgkQWj/um9xcTr/+\naiWo0FAYNy7rCSo7ZbT9+Ham/D0lyz8fSHLzveQGvrRJhQMbjTGrsdqkABCRu22LSimVJfHxVmcJ\ngOeft6r8/Hr9xHhGrhzJ8GXDSfAk0LhcYxqV0+XoVNb50ibVIrX9IrLMlojSjkOr+5TKwMiRMGQI\nVKtmrRNV0I99ElbvX02fuX1Yf2Q9AL0a9mLUHaMoWaik/4JQrmTrOCnvBaoANURkiTGmEBAsImey\ncsGs0iSlVPr27IHatSEmBhYtgrZt/XftGWtn0Ou7XghCtfBqTGw/kdur3e6/AJSr2TpOyhjzKPAV\nMNG7qwLwXVYupuyn9eO+yY3lNGiQlaA6d86ZBJWZMrqj2h2ULFSSIU2HsL7/+jyVoHLjveQmvrRJ\nDQBuAFYBiMh2Y0wZW6NSSmXK99/DN99A4cIwerT/r1+uaDl2PrlTpzZSOc6XNqlVInKjMWaNiDQy\nxuQD/talOpRyh9hYa9LYHTusNqlnn7XvWiLCqfOnKF6geMYHK+Vl97RIy4wxLwIFjTF3AF8C8zIR\nXFtjzBZjzDZjzPNpHNPSGLPGGLPBO9u6UspHo0ZZCapOHXjqKfuuszNqJ60/bU37me3xWMvLKWU7\nX5LUUOAosB7oCywEXvbl5MaYIOADoA1QF7jfGFPrkmPCgHFAexG5Bujsc/TqMlo/7pvcUk7//gtv\nvmm9HzcOQkJy7twXyijBk8DIX0dSb3w9lvyzhC3HtrAjakfOXSjA5ZZ7ya0ybJPyrsg72fvKrBuA\n7RemUDLGzALuAbYkO6Y78LWI7Pde71gWrqNUnvT001Z1X/fu0LJlzp//74N/02duH9YcWgPAA/Ue\n4N0271K6cOmcv5hSqfCpC3qWT25MJ6CNiDzm3X4QuEFEnkx2zLtACNaTVhFgrIh8ksq5tE1KqWQW\nLoS77oKiRa0lOcqVy/lrjPl9DIN+GESVsCpMbD+RNle1yfmLqFwvO21SvvTus1s+oDFwK1AY+M0Y\n85uIaH2CUmk4f/5i+9Nrr9mToAAG3jAQj3joe21fCocWtuciSqXD5yRljCkkIjGZPP9+IPm8fxW9\n+5LbBxzpnBd8AAAgAElEQVQTkVgg1hizHGgAXJakevXqRdWqVQEoXrw4DRs2pKW3juNCvXBe376w\nzy3xuHV7zJgxAX3/DBgQwY4dULt2SwYOtOd6kZGRPP300wy+abDjv6+bty/9t+d0PG7YHjNmDJGR\nkUnf19nhSxf0psBHQBERqWyMaQD0FZHHMzy5NYv6VuA24CCwGrhfRDYnO6YW8D7QFsiPNR6rq4hs\nuuRcWt3ng4iIiKQbRaUtkMtpzx6oVQvOnYMlS6yZzrNDRJi5fiahwaF0rnux31Igl5E/aTllzO7l\n41cB9wFzRaSRd98Gb088X4JrC7yH1ZNwioiMMMb0xVo4cZL3mGeB3kAiMFlE3k/lPJqklAK6dIEv\nv7Rmlvjii+yda9fJXfSb348fdv5AyYIl2TZwGyUKlsiZQJXysj1JJR/M6923VkQaZOWCWaVJSin4\n6Se4/XYoVAg2b4bKWVxEJ9GTyNhVY3l56cvExMcQXiCcd1q/Q6+GvTC6+JTKYXYP5t3rrfITY0yI\n96lnc0Y/pJyRvH5cpS0Qyyk+Hp709ot96aWsJyiAh+c+zODFg4mJj6Fr3a5sHrCZ3o16p0hQgVhG\nTtByspcvSaof1vx9FbA6PTT0biul/OjDD2HTJqheHQYPzt65Hr/ucaqEVWHe/fOYdd8syhYpmzNB\nKpXDbB0nlZO0uk/lZUeOQM2acOoUzJ0LHTpk/5zxifGEBOfgFBVKpcGWcVLGmPeBNLNC8gG5Sil7\nvfSSlaDatoX27X3/uahzUeQLypfq7OSaoFQgSK+670/gr3ReyoW0ftw3gVROf/4JU6ZY8/KNGQO+\n9GsQEWZtmEXtcbUZ8uOQLF03kMrISVpO9krzSUpEpiffNsYUs3b7d0VepfIyj8fqLCFizdN39dUZ\n/8yeU3t4fMHjLNi+AIDNxzYTlxhHaHCozdEqlfN86YJ+HfAxUBQwwEngYRHx69OUtkmpvOiTT6Bn\nT7jiCmt+vmIZrCn4weoPeOGnF4iOiyYsfxhv3/E2fRr3Icj40kdKKXvYPXffVOBxEfnFe7FmWEnL\nr4seKpXXnDkDz3tXYBsxIuMEBRB5KJLouGg61e7E++3ep1xRmyb1U8pPfPnzKvFCggIQkRVAgn0h\nqezQ+nHfBEI5/e9/cPAg3HAD9Ojh28+MvGMkc7rN4asuX2U7QQVCGbmBlpO90uvd19j7dpkxZiLw\nOVZvv65AhP2hKZV37dgBo0db78eOhSAfa+vCC4Zz99V32xeYUn6WZptUBsu4i4jcak9IqdM2KZWX\n3HOPNR7qoYdg2rSUn52MPcnQJUPp07gP15W/zpH4lMoMW+fucwtNUiqvWLwY2rSBIkVg27aLa0WJ\nCN9s/oaBiwZyMPog15e/nlV9Vulce8r1bJ27zxhT3BjzpDFmtDFm7IVXVi6m7Kf1475xaznFx1td\nzQGGDbuYoPaf3s+9s+/lvi/v42D0QZpWasq0jtNsTVBuLSO30XKyly+9+xYCvwPrAY+94SiVt334\noTW7+VVXXVx5N8GTQLOPm7Hr5C6KhhZlxO0j6HddP+1WrvIEX8ZJ/S0ijdM9yA+0uk/ldkePQo0a\nqc/P9/Gaj5mzdQ4f3PkBFYtVdC5IpbLA7vWkBgHRwHzg/IX9IhKVlQtmlSYpldv16wcTJ1rtUYsW\npZz+6MK9r+1PKhDZvZ5UHDAS+I2L8/b9mZWLKftp/bhv3FZOkZEwaRIEVfibd0Z7Lpufzxjj9wTl\ntjJyKy0ne/mSpJ4BrhKRqiJypfdVze7AlMorRGDA4NNIuwF4+lxHRPR4p0NSyjV86TixA4ixOxCV\nM1q2bOl0CAHBTeX0/MdzWNloABTbT76gfJw+f9rpkAB3lZGbaTnZy5ckdRaI9A7uTd4mpetJKZUN\nZ86f4aFvH+bbvV9BMbgy5EbmPDKZemXrOR2aUq7hS3Xfd8CbwEp0PSnX0/px37ihnAqFFOKPrfsg\nrjDl177Hlud+dVWCckMZBQItJ3tl+CQlItONMQWByiKy1Q8xKZUnHNgfzNFJM+BsfmbOqUyoLpSr\n1GV86YLeARgFhIrIlcaYhsBwEfHrLJbaBV3lNvffD7NmQefO8MUXTkejlH3s7oL+GnAD1mKHiEgk\noL37lPLRqn2raDmtJYejDyft++UXK0EVKAAjRzoYnFIu50uSiheRU5fs0+mRXErrx33jj3I6c/4M\nTy16ipum3MSy3ct4a8VbACQmXpzyaMgQqFLF9lCyRO8l32g52cuX3n0bjTHdgWBjTA3gSaxOFEqp\nNCzcvpD+C/qz59Qegk0wzzZ9lldavALAxx/DmjVQsaKVpJRSafOlTaoQ8BLQGjDAD8AbIhJrf3gp\n4tA2KRUQdkbtpOYHNfGIh2vLXctHd39EwysaAnDyJNSsac3T9/nn0K2bw8Eq5Qd+W0/KGBMMFBYR\nv4821CSlAsmwn4cRXjCcJ298knxBFyssBg+Gd9+FW26BZcu4bPojpXIju9eTmmmMKWaMKYy1XMcm\nY8xzWbmYsp/Wj/vG7nIa3mo4g28anCJBbd4M779vJab33nN/gtJ7yTdaTvbypeNEHe+TU0dgEXAl\n0MPWqJQKAPGJ8SzavijVzy6dDFYEBg2ChATo0wcaNfJHhEoFPl/apDYCDYGZwAcisswYs1ZEGvgj\nwGRxaHWfco0/D/xJn7l9WHt4LT/3/JlWV7ZK9/j58631ocLCYPt2KF3aT4Eq5QJ2j5OaCOwCCgPL\njTFVAHfMgKmUn52NO8szPzzDjR/dyNrDa6lavGqGK+SeP289RQG89pomKKUyI8MkJSJjRaSCiNwp\nlt1A+n82Ksdo/bhvslJO6w6v45rx1zD699EAPHPTM2zov4EWVVuk+3NjxsCOHVC7NgwYkJVonaH3\nkm+0nOyV4TgpY0x+oBNQ9ZLjh9sUk1KuVDmsMucTztPwioZM7jCZ68pfl+HPHDgAb7xhvX/vPQjR\n+fmUyhRf2qS+B05hzXyeeGG/iLxjb2iXxaFtUspxW45toXp4dUKCfcs2PXrAp5/CvffCN9/YHJxS\nLmXrOCljzAYRuSZLkeUgTVLKnzziybCtKSO//grNmkH+/Fb38yuvzKHglAowdnecWGmMcc8iNypd\nWj/um7TKKcGTwOjfRnPz1JuJT4zP8vkTE2HgQOv9kCGBmaD0XvKNlpO9fJm7rxnQyxjzL9bKvAYQ\nEalva2RK+VnkoUj6zO3DXwetNT3nbZvHf2r/J0vnmjLFmp+vUiUYOjQno1Qqb/Glui/VOZq9vfwy\nvoAxbYExWE9tU0Tk/9I47nqsiWu7ishltfda3afsEhMfw+sRr/POb++QKIlUKlaJCe0ncGeNO7N0\nvqgoa36+48dh9mzo0iWHA1YqwNhS3WeMuRWSklGQiOy+8AKu9TGwIOADoA1QF7jfGFMrjeNGYE1e\nq5Rfzds6j7dXvo1HPDx5w5NsfHxjlhMUwLBhVoJq1cpa0FAplXXptUmNSvb+60s+e9nH898AbPcm\nt3hgFnBPKscNBL4Cjvh4XpUGrR/3TfJy6lK3CwNvGMhvj/zGe+3eo2j+olk+75o1MGECBAdfnKcv\nUOm95BstJ3ul1yZl0nif2nZaKgB7k23vw0pcF09kTHmgo4i0Msak+EwpfzDGMLbd2GyfRwSeeAI8\nHmuGibp1cyA4pfK49J6kJI33qW1nxxjg+WTbAfy3p/NatmzpdAiutefUHuZtnQfYU06ffgorV0LZ\nsvDqqzl+er/Te8k3Wk72Su9JqpoxZi5W0rjwHu+2rx1q9wOVk21X9O5L7jpglrGmjS4FtDPGxIvI\n3EuOo1evXlStWhWA4sWL07Bhw6Qb5MIjt27r9qXbiZ5EnprwFFP+nkJwtWA2Pr6RfyP/zdHrLVgQ\n4V0SviVvvw1r1rjn99dt3fb39pgxY4iMjEz6vs6ONHv3GWPSnZBMRJZleHJrkcStwG3AQWA1cL+I\nbE7j+I+Bedq7L+siIiKSbhQF6w+v59F5j7Jq/yoAOtfpzPvt3mfzn5tztJwuLGbYtCn88gsEZW8c\nsCvoveQbLaeMZad3X3pPUg9grR+1RETOZOXkIpJojHkCWMzFLuibjTF9rY9l0qU/kpXrKJWaqWum\n0nd+XxI8CVQoWoEP7/qQu6++G4DNpPp3UpasXw9jx1qJ6YMPckeCUsot0nuSuhFoh/UUFIeVaL4X\nkbX+Cy9FPPokpTJl09FNXDvpWh5u+DBv3f4WxfIXy/FriEDz5rBihdVp4v33c/wSSgU8W+fu816g\nJNAaK2nVA9ZgJawvsnLRrNAkpbLi4JmDlCtazrbzz5gBDz0EZcrA1q1QvLhtl1IqYNk9dx8iclxE\nPheRniLSCBgH1MjKBZW9LjRg5iUiQmxCbKqfpZWgcqKcTp6E556z3o8cmfsSVF68l7JCy8leGSYp\nY0xZY8wUY8wi73YdoKGIvGl7dEplYN/pfXSc3ZGe3/b0+7WHDYMjR6yZznv08PvllcoTfJm7bxHw\nMfCSiDQwxuQD1oiIX2dG1+o+lZxHPEz4cwJDlwzlTNwZiuUvxob+G6gUVskv1//7b7j+emtGiTVr\noJ6uE6BUmuyu7ivlbXvyAIhIAskWP1TK3zYd3cQtH9/CgIUDOBN3ho61OrLp8U1+S1CJidCvnzWz\nxJNPaoJSyk6+JKmz3o4TAmCMaYK1Uq9yobxQPz49cjor967kiiJX8HWXr/m267dUKFYhU+fITjlN\nnAh//AEVKsDrr2f5NK6XF+6lnKDlZC9f1pMaDMwFqhtjfgVKA/fZGpVS6Xi15asEBwUz5OYhFC/g\n394Khw7Biy9a78eOhaJZn4tWKeUDX7ug5wOuxpoSaat3RnO/0jYp5QYPPAAzZ8Kdd8L8+YE9y7lS\n/mJrm5QxpjNQUEQ2Ah2B2caYxlm5mFKZ8d2W71i2K8PZt/zmp5+sBFWggDWzhCYopeznS5vUMBE5\nY4xphjX7xBRgvL1hqazKDfXjB84coNMXnbh39r08PPdhYuJjcvwamS2n2Fjo3996/8orcKWvUywH\nsNxwL/mDlpO9fElSF3ry3QVMFpEFQKh9Iam8yiMeJv01iTrj6vDN5m8oElqEQU0GkT84v9Oh8eab\nsH071KkDzzzjdDRK5R2+jJOaj7W8xh1AY+AcsFpEGtgfXoo4tE0ql7v/6/uZtWEWAO1rtufDOz/0\nW7fy9GzcCI0aQXy8NUffzTc7HZFSgcXucVJdgB+ANiJyEigBPJeViymVnm51u1GmcBlmdZrF3G5z\nXZGgPB547DErQfXtqwlKKX/LMEmJSIx3fadTxpjKQAiwxfbIVJYEcv34PbXuYeeTO+l6TVeMzb0S\nfC2nyZOt1XavuAJGjLA1JNcJ5HvJn7Sc7OVL7767jTHbgX+BZd7/LrI7MJV7nTl/hrjEuFQ/KxJa\nxM/RpO3gQXj+eev92LG5bwJZpQKBL21Sa4FbsRY/bGSMaQU8KCKP+CPAZHFom1QuMG/rPB5f+DiP\nNX6MYS2GOR1Oujp1gm++gbvugnnztMu5Ullld5tUvIgcB4KMMUEishS4LisXU3nXoehDdP2qK3fP\nupt9p/fxw84fSPS4dwrIr7+2ElSRIjB+vCYopZziS5I6aYwpAiwHPjPGvAectTcslVVuqx8XEaau\nmUrtcbX5YuMXFAopxOjWo1nWaxnBQcGOxZVeOUVFwYAB1vu334ZKzvffcITb7iW30nKyly9z992D\n1e18EPAAEAYMtzMolbt8uelLTsaepO1VbRl/13iqFq/qdEjpeuYZOHwYbrnF6tGnlHKOT3P3JR1s\nTCnguBONQ9omFbh2n9zNij0r6F6vu+299rJr8WJo0wby54d166BmTacjUirwZadNKs0k5V2SYwQQ\nBbwBfAKUwqoi7Cki32ct3KzRJKXsduaMtTbU7t1Wd/MLPfuUUtljV8eJD4D/AZ8DPwN9ROQKoDnw\nVlYupuznVP14dFw0zy1+jn9P/OvI9TMrtXIaMsRKUI0b69RHoG0tvtJysld6bVL5RGQxgDFmuIj8\nDiAiW9xeZaP86/sd39Nvfj92n9rNxqMbWfjAQqdDyrQff4QJEyAkBKZNg3y+tNYqpWyXXnXf3yLS\n+NL3qW37g1b3uc/Rs0d5+oenmbl+JgCNrmjER3d/RONygbWSy+nTcM01sHevNZHshUUNlVI5w642\nqUSsruYGKAhcWC/BAAVEJCQrF8wqTVLuEpsQy1Vjr2L/mf0UzFeQ11u+zqCbBpEvKPAeQR57zJr+\n6Lrr4Lff9ClKqZxmS5uUiASLSDERKSoi+bzvL2z7NUEp3/mrfrxAvgL0v64/t1e7nfX91/Pczc8F\nVIK6UE4//GAlqNBQmD5dE1Ry2tbiGy0ne+k/SZVlQ5sN5cVbXnR9t/K0REXBI97JvYYPt9aKUkq5\nS6bGSTlJq/ucszNqJ9VLVHc6jBx3//0waxbcdBMsX65PUUrZxe65+1QeFRMfw3OLn+PqD67m283f\nOh1Ojpo500pQhQvDjBmaoJRyK01SuUxO1Y8v+WcJ9cbXY9Rvo/CIh41HN+bIed1g71547LEIAN59\nF666ytFwXEvbWnyj5WQv/ftRpXAq9hRPff8U09dOB6BemXpM7jCZGyve6HBkOcPjgV694OxZaN8e\n+vRxOiKlVHq0TUqlcC7+HPXG12Pf6X280uIVnmv6HCHBuacz5+jR1mwSpUvD+vVQtqzTESmV+9ky\nTsptNEn5zx/7/yCsQBg1S+au2VX//huaNIH4eJgzB+6+2+mIlMobtOOESpIT9ePXV7g+1yWo6Gjo\n1s1KUAMGQLFiEU6H5Hra1uIbLSd7aZLKo9YdXkfnLztzNi5vrF85cCBs327Ncj5ypNPRKKV8pdV9\necy5+HO8sfwNRq4cSYIngVdbvMprLV9zOixbff45dO8OBQrAX3/poF2l/C071X3auy8PWfrvUh6b\n/xg7onZgMAy4fgCDbxrsdFi22rkT+vWz3o8ZowlKqUCj1X25TFr14+sOr+PWGbeyI2oHdUrX4deH\nf+WDOz+gWP5i/g3Qj2JjoXNna5bzTp2siWQv0HaEjGkZ+UbLyV62P0kZY9oCY7AS4hQR+b9LPu8O\nXFgD9QzQX0TW2x1XXlO/bH16NuhJ9fDqDG02lNDgUKdDst1TT8GaNVC9OkyZAgE6xaBSeZqtbVLG\nmCBgG3AbcAD4A+gmIluSHdME2Cwip7wJ7TURaZLKubRNKptEJGAng82sTz+FHj0gf35r+Y1GjZyO\nSKm8y81d0G8AtovIbhGJB2YB9yQ/QER+F5FT3s3fgQo2x5SrJXoSWb1/daqf5ZUEtWkT9O1rvR87\nVhOUUoHM7iRVAdibbHsf6SehPsAiWyPKxTYe2Uj95+vTbGozNh7JPXPtZcaF9qeYGHjwQXj00dSP\n03aEjGkZ+UbLyV6u6ThhjGkF9OZi+5Ty0fmE87yy9BUaTWzEpqObKFWoFEfOHnE6LL/zeOChh2DL\nFqhbFyZM0HYopQKd3R0n9gOVk21X9O5LwRhTH5gEtBWRE2mdrFevXlStWhWA4sWL07BhQ1q2bAlc\n/Gsmr22H1wqn29fd2PKH1czX976+jLh9BJG/RxKxO8Lx+Py5PWMGfPddS4oXhxdeiOCPP9I+/sI+\nN8Xvxu0L3BKPG7dbtmzpqnjcsD1mzBgiIyOTvq+zw+6OE8HAVqyOEweB1cD9IrI52TGVgZ+AHiLy\nezrn0o4Tqdhzag91P6xLxWIVmdR+ErdUucXpkBwxf/7FufgWLoS2bZ2NRyl1kWs7TohIIvAEsBjY\nCMwSkc3GmL7GmAujVoYBJYAPjTFrjDGpt/qrVFUOq8ySHkuI7BvJLVVuuewv4Lxg61Z44AEQgTff\n9C1B5cVyyiwtI99oOdnL9nFSIvI9cPUl+yYme/8okEbztvJFblnrKSuioqBDB6vDxH33wdChTkek\nlMpJOndfAPCIh0l/TWLJP0v4svOXeaYreUbi4qBNG4iIgIYN4ZdfoEgRp6NSSl1K5+7LxbYc28Kj\n8x5lxZ4VAPz4z4+0rt7a4aicJwL9+1sJqlw5mDdPE5RSuZFruqCrlOIS4xi+bDgNJjRgxZ4VlC1c\nltn3zeaOanek+3N5pX581CiYOhUKFoS5c6Fixcz9fF4pp+zQMvKNlpO99EnKpab8PYVXI14F4OGG\nDzOq9SjCC4Y7HJU7fPEFPO8dTTdjBlx3nbPxKKXso21SLhWfGE+3r7vxxPVP0OrKVk6H4xpLl1q9\n9+Li4K23tKOEUoEgO21SmqRUwFi7Fpo3t3ryDRwI772nM0ooFQhcO05KZexQ9CFW7l2ZY+fLrfXj\nu3ZBu3ZWgurcGd59N3sJKreWU07SMvKNlpO9NEk5RET46O+PqD2uNp2+6MSJc2nOBpXnHT5sdTU/\neBBatrTaoYKDnY5KKeUPWt3ngO3Ht/PY/MeI2BUBQLur2jHl7imUK1rO2cBc6PhxKzFt2AANGsCy\nZRAW5nRUSqnM0HFSAWTK31MYsHAA5xPPU7pQad5r+x7drummA3RTcfIktG5tJajatWHxYk1QSuU1\nWt3nZ1eVuIrziefp2aAnmwds5v569+dogsot9ePR0XDnnfD339by70uWQJkyOXf+3FJOdtIy8o2W\nk730ScrPWlRtwabHN1G7dG2nQ3Gt06fhrrusZd8rVYKffoLy5Z2OSinlBG2TspFHPAQZfVjNjJMn\nrXFQq1ZBhQrWuKgaNZyOSimVHdoF3WWOnj3KA988wDM/PON0KAHl+HG47TYrQVWpAsuXa4JSKq/T\nJJWDRITpkdOpNa4WM9fP5KM1H3Es5phfYwjU+vFDh6BVq4ttUMuXQ7Vq9l0vUMvJn7SMfKPlZC9t\nk8ohO6N20m9BP5b8swSA26vdzsT2EylVqJTDkbnftm1WFd+//0KtWtoGpZS6SNukckiv73oxfe10\nShYsyeg2o+lRv4d2K/fB6tVWJ4ljx+D662HBAihd2umolFI5ScdJucDbd7xN/uD8/PfW/1K6sH7L\n+uL776FTJ4iJsaY8+uILXRNKKZWStknlkDKFyzCxw0THE1Qg1I+LwNix0L69laAeegjmzPFvggqE\ncnKalpFvtJzspUkqk37c+SMbjmxwOoyAFRcHffvCU09BYiK8/DJ8/DGEhDgdmVLKjbRNykfHYo7x\nzOJnmLF2BjdWuJFfH/6V4CCd5TQzjh6F++6zeu4VKGCtrHv//U5HpZSym7ZJ2UhEmLl+Jk//8DTH\nYo6RPzg/91x9D0JgJHe3+PVX6NYN9u2zeu59953VUUIppdKj1X0Z6PZ1Nx789kGOxRyjZdWWrO+/\nnhdueYF8Qe7M726rHxeBUaOgRQsrQTVtCn/84XyCcls5uZGWkW+0nOylSSoDLaq0oHiB4ky5ewo/\n9/yZGiV1CgRfHTsG99wDzz1ntT89+yxEROgYKKWU77RNKgMe8XAs5hhlCufgFNx5wPz50KePtWBh\n8eIwfTrcfbfTUSmlnJCdNilNUl6xCbHkD86vA3Cz6cwZGDwYPvrI2m7e3EpQVas6GpZSykE6wWw2\n/fzvz9QbX4+PIz92OpRsc7J+fMECqFfPSlChoVZb1NKl7kxQ2o6QMS0j32g52StPJ6moc1E8MucR\nbptxGzuidvBx5McEypOlmxw4AJ07W4Nzd++GRo3gr7/gmWcgKE/fYUqp7MqT1X0iwhcbv+DJ75/k\nyNkjhAaHMqz5MIbcPITQ4NAcuUZecP48jBsHr71mVfMVLgzDh8OTT0I+d3Z+VEo5QNukMinBk8AN\nk29gzaE13FL5FiZ1mEStUrVy5Nx5gQh89RUMHQr//GPtu+cea6qjypWdjU0p5T7aJpVJ+YLy8dHd\nHzGx/UQiekXkqgRlZ/24CPz8M9x8M3TpYiWoOnWstqjvvgusBKXtCBnTMvKNlpO98mylTONyjWlc\nrrHTYQQEEViyxKrKW7HC2lemjLX9yCNataeUsk+uru6LTYjlnZXv0P/6/pQoWMKmyHKvhAT49lsY\nPRp+/93aV6KE1cX8ySehaFFn41NKBQaduy8Vy3cv57F5j7H1+FZ2ntjJ1HumOh1SwIiKgilT4P33\nYe9ea1/JklZvvSee0OSklPKfXNcmdTL2JH3n9aXFtBZsPb6VWqVq8XCjh50Oy2+yWj+emAg//ABd\nu0K5cjBkiJWgrr4aPvzQ6lr+wgu5J0FpO0LGtIx8o+Vkr1z1JHX6/Gmu+fAa9p/ZT0hQCC/e8iIv\nNHuB/PnyOx2aK4nAqlXw5ZfWqrj79ln7jYG2ba0qvTZtdKyTUso5ua5Nqv/8/qw7so5J7SdRt0xd\nP0QWWGJjrfWcFi6Eb765WJ0HUK0aPPww9OwJlSo5F6NSKnfRcVLJxCbEEhocSpDRP//BqsZbu9ZK\nTEuWWNMUxcRc/LxCBWshws6d4aab9KlJKZXzXJ2kjDFtgTFY7V9TROT/UjlmLNAOOAv0EpHIVI5J\nkaSOnj1K6cKlbYs7UH3zTQShoS3580+rKm/lSjh9OuUxDRtCu3bWNEZNmuTNxBQREUHLli2dDsPV\ntIx8o+WUMdf27jPGBAEfALcBB4A/jDFzRGRLsmPaAdVFpIYx5kZgAtAkrXOeTzjPiBUjGPHrCH7u\n+TM3VbrJzl/BlUTg0CHYsQO2boWNG63Xhg1w8GAk0DLF8dWqWbORt2hhtTGVK+dI2K4SGRmpXywZ\n0DLyjZaTvezuOHEDsF1EdgMYY2YB9wBbkh1zDzADQERWGWPCjDFlReTwpSdbuXclfeb2YfOxzYA1\ne3luSlIicO6ctVjgkSPWWkyHD1sTuO7da3Vs2LPHmukheZVdcqGhJ2naFK67znrdfDNUrOjf3yMQ\nnDx50ukQXE/LyDdaTvayO0lVAJI1zbMPK3Gld8x+777LklSzqc0QhMqFa/BS/clcW7gFf/2VfgAZ\n1WZe+Fwk5fvk+0TA47n43wuvxERrwGtiovWKj0/5On/eesXGWq+YmIuv6GirGu7MGeu/J05Y45Pi\n4kBdCaMAAAbmSURBVNKP94KSJaF6dahRA+rWvfiaMQNef923cyillNsFVBd0SQyGX4ewZ/kw+iYU\ncDocW+TPb83qULas9SpTxlpuvVIl64moUiWr+q548dR/fvfuXX6NN1Dt2rXL6RBcT8vIN1pO9rK1\n44Qxpgnwmoi09W4PBSR55wljzARgqYjM9m5vAVpcWt1njAmMbohKKaUu48qOE8AfwFXGmCrAQaAb\ncP8lx8wFBgCzvUntZGrtUVn9BZVSSgUuW5OUiCQaY54AFnOxC/pmY0xf62OZJCILjTF3GmN2YHVB\n721nTEoppQJHwAzmVUoplfe4bhinMaatMWaLMWabMeb5NI4Za4zZboyJNMY09HeMTsuojIwx3Y0x\na72vFcaYek7E6TRf7iXvcdcbY+KNMf/xZ3xu4eO/uZbGmDXGmA3GmKX+jtFpPvybK2aMmev9Tlpv\njOnlQJiOM8ZMMcYcNsasS+eYzH1/i4hrXlhJcwdQBQgBIoFalxzTDljgfX8j8LvTcbuwjJoAYd73\nbfNaGflaTsmO+wmYD/zH6bjdWE5AGLARqODdLuV03C4soxeAty6UD3AcyOd07A6UVTOgIbAujc8z\n/f3ttieppMG/IhIPXBj8m1yKwb9AmDGmrH/DdFSGZSQiv4vIKe/m71jjzvIaX+4lgIHAV8ARfwbn\nIr6UU3fgaxHZDyAix/wco9N8KSMBLixkUxQ4LiIJfozRFURkBXAinUMy/f3ttiSV2uDfS79g0xr8\nm1f4UkbJ9QEW2RqRO2VYTsaY8kBHERkP5NXeo77cTzWBEsaYpcaYP4wxPfwWnTv4UkYfAHWMMQeA\ntcBTfoot0GT6+zugBvOqzDHGtMLqLdnM6VhcagyQvH0hryaqjOQDGgO3AoWB34wxv4nIDmfDcpU2\nwBoRudUYUx340RhTX0SinQ4s0LktSe0HKifbrujdd+kxlTI4JjfzpYwwxtQHJgFtRSS9x+/cypdy\nug6YZYwxWO0I7Ywx8SIy108xuoEv5bQPOCYisUCsMWY50ACrnSYv8KWMegNvAYjITmPMv0At4E+/\nRBg4Mv397bbqvqTBv8aYUKzBv5d+YcwFekLSjBapDv7NxTIsI2NMZeBroIeI7HQgRjfIsJxEpJr3\ndSVWu9TjeSxBgW//5uYAzYwxwcaYQlgN3pv9HKeTfCmj3cDtAN42lprAP36N0j0MaddKZPr721VP\nUqKDfzPkSxkBw4ASwIfep4R4Ebl0Yt9czcdySvEjfg/SBXz8N7fFGPMDsA5IBCaJyCYHw/YrH++l\n/wLTknW9HiIiUQ6F7BhjzEystYJKGmP2AK8CoWTj+1sH8yqllHItt1X3KaWUUkk0SSmllHItTVJK\nKaVcS5OUUkop19IkpZRSyrU0SSmllHItTVJKZcAYk2iM+du7tMCf3kGISik/0HFSSmXAGHNaRIp5\n37cGXhSRlg7GEywiiWlt+/pzSgUCfZJSKmPJp3gJA6IAjDGFjTFLvE9Xa40xd3v3FzLGzPcuErjO\nGNPZu7+xMSbCO5P4otSWKDDGlDLGfGWMWeV93eTd/6oxZoYxZgUwwxjzkDFmjjHmJ2CJ95iR3gX3\n1hpjunj3tTDGLDfGzMFaE0qpgOKqaZGUcqmCxpi/gYLAFVizgQOcw1rqI9oYUxJr7a65WAtN7heR\n9gDGmKLGmHzA+8DdInLcm0T+v727Z40iCqM4/j+JRQrBxggK9m7iKhsQi6ggWGkr2GlhpY2Vn0Aw\nhUEQ8ROIWipaaRpfQpBYBS0UOwuLgIWICivssZibuKzCJKh4hfPrdufO3JmF3cM+d5jnMnB2ZK5r\nwFXbS5J2Aw+BqbKtA8za7ks6A/SAru2PpavwPttdSTuAF5KelP16wLTtd3/4c4n46xJSEe2+2J6B\n9Ydi3gT20lQi5iQdAQbArhIQL4F5SXM0XUgXJU2XfRbK8xTHgPe/mOsY0CljALaWh7oC3LfdHxq7\nMNTc8hBwB8D2qqTHwAHgE7CcgIr/VUIqYhNsPy8lue3ACZoWHz3bg9KeYcL2W0kzwHHgUinJ3QNe\n2Z5tmULAwdIB9sebTWZ9Hhk7+nr0OBsZF1G1rElFtFv/wZe0h+Z784FmfWq1BNRRSs8hSTuBr7Zv\nA/M0DQPfAJNrdwZK2iJpip89Yqirq6T9GzzHZ8ApSWOSJoHDwPLmLjOiPvknFdFuoqxJrYXVaduW\ndAt4IGmFprnd67K9C1yRNAD6wDnb3ySdBK5L2gaM03QGHm15cQG4UY45DjwFzredoO27JQBXaEqP\nF0vZr/Mb1x3xz+UW9IiIqFbKfRERUa2EVEREVCshFRER1UpIRUREtRJSERFRrYRURERUKyEVERHV\nSkhFRES1vgOJtW1JroE8AwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x256435bac50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.plot(error_range, \n",
    "         ens_errors, \n",
    "         label='Ensemble error', \n",
    "         linewidth=2)\n",
    "\n",
    "plt.plot(error_range, \n",
    "         error_range, \n",
    "         linestyle='--',\n",
    "         label='Base error',\n",
    "         linewidth=2)\n",
    "\n",
    "plt.xlabel('Base error')\n",
    "plt.ylabel('Base/Ensemble error')\n",
    "plt.legend(loc='upper left')\n",
    "plt.grid()\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/ensemble_err.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The ensemble outperforms individual classifiers as long as their base error $\\epsilon < 0.5$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Excercise\n",
    "\n",
    "Plot ensembe error versus different number of classifiers.\n",
    "Does the ensemble error always go down with more classifiers? Why or why not?\n",
    "Can you fix the issue?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Ensemble bias and variance\n",
    "\n",
    "$d_j$ are decisions made by $m$ individual base classifiers\n",
    "\n",
    "$y$ is the ensemble decision via summation (not really voting for mathematical tractability):\n",
    "$$\n",
    "y = \\frac{1}{m} \\sum_{j=1}^m d_j \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For iid:\n",
    "$$\n",
    "\\begin{align}\n",
    "E(y) &= E\\left( \\frac{1}{m} \\sum_{j=1}^m d_j \\right) = E(d_j)\n",
    "\\\\\n",
    "Var(y) &= Var\\left( \\frac{1}{m} \\sum_{j=1}^m d_j \\right) = \\frac{1}{m} Var(d_j)\n",
    "\\end{align}\n",
    "$$\n",
    "* The expected value E, and thus bias, remains the same.\n",
    "* The variance reduces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For general case:\n",
    "\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "Var(y) &= Var\\left( \\frac{1}{m} \\sum_{j=1}^m d_j \\right) \n",
    "\\\\\n",
    "& = \\frac{1}{m^2} \\sum_j Var(d_j) \n",
    "+ \\frac{1}{m^2} \\sum_j \\sum_{i \\neq j} Cov(d_j, d_i)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "So $Var(y)$ increases/decreases for positively/negatively correlated base estimators."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Diversity versus accuracy\n",
    "\n",
    "Not possible to have all base estimators accurate and yet negatively correlated.\n",
    "\n",
    "Why?\n",
    "* accurate estimators tend to make correct predictions\n",
    "* correct predictions, by definition, are positively correlated with themselves\n",
    "\n",
    "Sometimes need intentionally non-optimal base learners for better ensemble performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How to achieve diversification and accuracy\n",
    "\n",
    "Different models/algorithms\n",
    "* base learners complement each other\n",
    "* e.g. some parametric, some non-parametric\n",
    "\n",
    "Different hyper-parameters for the same algorithm/model\n",
    "* k in KNN\n",
    "* threshold in decision tree\n",
    "* kernel function in SVM\n",
    "* initial weights for perceptron and neural networks\n",
    "\n",
    "Different input representations of the same event\n",
    "* sensor fusion, sound and mouth shape for speech recognition\n",
    "* random subset of features (columns of the data matrix)\n",
    "\n",
    "Different training sets\n",
    "* random subset of all samples (rows of the data matrix) - bagging\n",
    "* sequential training - boosting, cascading inaccurately classified samples\n",
    "\n",
    "Base learners be reasonably instead of very accurate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How to combine multiple classifiers\n",
    "\n",
    "## Parallel: multi-expert combination\n",
    "\n",
    "Base learners work in parallel\n",
    "\n",
    "Global approach\n",
    "* all base learners generate outputs for a given input: voting, bagging\n",
    "\n",
    "Local approach\n",
    "* select only a few base learners based on the input: gating for mixture of experts\n",
    "\n",
    "## Sequential: multi-stage combination\n",
    "\n",
    "Base learners work in series\n",
    "\n",
    "Later models focus on datasets not well handled by early models: cascading\n",
    "\n",
    "Start with simpler models, and increase model complexity only if necessary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Implementing a simple majority vote classifier \n",
    "\n",
    "A (multi-class) majority vote classifier can be implemented via the weighted sum of $m$ individual classifier prediction:\n",
    "$$\n",
    "\\hat{y} = argmax_i \\sum_{j=1}^m w_j \\left( C_j(x) = i \\right)\n",
    "$$\n",
    "\n",
    "Here, $C_j(x) = i$ is a boolean expression for classifier $j$ to predict the class of $x$ to be $i$.\n",
    "\n",
    "The weights can come from confidence, accuracy, or Bayesian prior of each classifier.\n",
    "$$\n",
    "p(C(x) = i) = \\sum_{j=1}^m p(C_j) p\\left( C_j(x) = i \\; | \\; C_j \\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Equal weighting\n",
    "\n",
    "If the individual classifiers are weighed equally, the above would reduce to what we see earlier:\n",
    "$$\n",
    "\\hat{y} = mode\\{C_1(x), C_2(x), \\cdots, C_m(x) \\}\n",
    "$$\n",
    "\n",
    "$C_1(x) = 0$,\n",
    "$C_2(x) = 0$,\n",
    "$C_3(x) = 1$\n",
    "$\\rightarrow$\n",
    "$\\hat{y} = mode\\{0, 0, 1\\} = 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Unequal weighting\n",
    "\n",
    "$w_1 = 0.2$,\n",
    "$w_2 = 0.2$,\n",
    "$w_3 = 0.6$\n",
    "$\\rightarrow$\n",
    "$\n",
    "\\begin{cases}\n",
    "0.4 & class \\; 0 \\\\\n",
    "0.6 & class \\; 1\n",
    "\\end{cases}\n",
    "$\n",
    "$\\rightarrow$\n",
    "$\\hat{y} = 1$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.argmax(np.bincount([0, 0, 1], \n",
    "                      weights=[0.2, 0.2, 0.6]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Weighted probability\n",
    "\n",
    "If the individual classifiers can provide prediction probability\n",
    "$$\n",
    "p\\left(C_j(x) = i\\right)\n",
    "$$\n",
    ", such as predict_prob() in scikit-learn, we can also predict ensemble class probability:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "p\\left(C_{ensemble}(x) = i \\right) \n",
    "&= \\sum_{j=1}^m w_j p\\left( C_j(x) = i \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "From which we can decide the predicted class (i.e. predict() in scikit-learn):\n",
    "$$\n",
    "\\hat{y} = argmax_i \\; p\\left(C_{ensemble}(x) = i \\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In the 2-class 3-classifier example above,\n",
    "\n",
    "$w_1 = 0.2$,\n",
    "$w_2 = 0.2$,\n",
    "$w_3 = 0.6$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "C_1(x) &= [0.9, 0.1] \n",
    "\\\\\n",
    "C_2(x) &= [0.8, 0.2] \n",
    "\\\\\n",
    "C_3(x) &= [0.4, 0.6] \n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Then\n",
    "$$\n",
    "\\begin{align}\n",
    "p\\left(C_{ensemble} = 0\\right) &= 0.2 \\times 0.9 + 0.2 \\times 0.8 + 0.6 \\times 0.4 = 0.58\n",
    "\\\\\n",
    "p\\left(C_{ensemble} = 1 \\right) &= 0.2 \\times 0.1 + 0.2 \\times 0.2 + 0.6 \\times 0.6 = 0.42\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\hat{y} = 0\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.58,  0.42])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex = np.array([[0.9, 0.1],\n",
    "               [0.8, 0.2],\n",
    "               [0.4, 0.6]])\n",
    "\n",
    "p = np.average(ex, \n",
    "               axis=0, \n",
    "               weights=[0.2, 0.2, 0.6])\n",
    "p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Majority classifier implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.base import ClassifierMixin\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.externals import six\n",
    "from sklearn.base import clone\n",
    "from sklearn.pipeline import _name_estimators\n",
    "import numpy as np\n",
    "import operator\n",
    "\n",
    "# inherits from two classes to get some methods for free\n",
    "class MajorityVoteClassifier(BaseEstimator, \n",
    "                             ClassifierMixin):\n",
    "    \"\"\" A majority vote ensemble classifier\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    classifiers : array-like, shape = [n_classifiers]\n",
    "      Different classifiers for the ensemble\n",
    "\n",
    "    vote : str, {'classlabel', 'probability'} (default='label')\n",
    "      If 'classlabel' the prediction is based on the argmax of\n",
    "        class labels. Else if 'probability', the argmax of\n",
    "        the sum of probabilities is used to predict the class label\n",
    "        (recommended for calibrated classifiers).\n",
    "\n",
    "    weights : array-like, shape = [n_classifiers], optional (default=None)\n",
    "      If a list of `int` or `float` values are provided, the classifiers\n",
    "      are weighted by importance; Uses uniform weights if `weights=None`.\n",
    "\n",
    "    \"\"\"\n",
    "    def __init__(self, classifiers, vote='classlabel', weights=None):\n",
    "\n",
    "        self.classifiers = classifiers\n",
    "        self.named_classifiers = {key: value for key, value\n",
    "                                  in _name_estimators(classifiers)}\n",
    "        self.vote = vote\n",
    "        self.weights = weights\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        \"\"\" Fit classifiers.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Matrix of training samples.\n",
    "\n",
    "        y : array-like, shape = [n_samples]\n",
    "            Vector of target class labels.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        self : object\n",
    "\n",
    "        \"\"\"\n",
    "        if self.vote not in ('probability', 'classlabel'):\n",
    "            raise ValueError(\"vote must be 'probability' or 'classlabel'\"\n",
    "                             \"; got (vote=%r)\"\n",
    "                             % self.vote)\n",
    "\n",
    "        if self.weights and len(self.weights) != len(self.classifiers):\n",
    "            raise ValueError('Number of classifiers and weights must be equal'\n",
    "                             '; got %d weights, %d classifiers'\n",
    "                             % (len(self.weights), len(self.classifiers)))\n",
    "\n",
    "        # Use LabelEncoder to ensure class labels start with 0, which\n",
    "        # is important for np.argmax call in self.predict\n",
    "        self.lablenc_ = LabelEncoder()\n",
    "        self.lablenc_.fit(y)\n",
    "        self.classes_ = self.lablenc_.classes_\n",
    "        self.classifiers_ = []\n",
    "        for clf in self.classifiers:\n",
    "            fitted_clf = clone(clf).fit(X, self.lablenc_.transform(y))\n",
    "            self.classifiers_.append(fitted_clf)\n",
    "        return self\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\" Predict class labels for X.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Matrix of training samples.\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        maj_vote : array-like, shape = [n_samples]\n",
    "            Predicted class labels.\n",
    "            \n",
    "        \"\"\"\n",
    "        if self.vote == 'probability':\n",
    "            maj_vote = np.argmax(self.predict_proba(X), axis=1)\n",
    "        else:  # 'classlabel' vote\n",
    "\n",
    "            #  Collect results from clf.predict calls\n",
    "            predictions = np.asarray([clf.predict(X)\n",
    "                                      for clf in self.classifiers_]).T\n",
    "\n",
    "            maj_vote = np.apply_along_axis(\n",
    "                                      lambda x:\n",
    "                                      np.argmax(np.bincount(x,\n",
    "                                                weights=self.weights)),\n",
    "                                      axis=1,\n",
    "                                      arr=predictions)\n",
    "        maj_vote = self.lablenc_.inverse_transform(maj_vote)\n",
    "        return maj_vote\n",
    "\n",
    "    def predict_proba(self, X):\n",
    "        \"\"\" Predict class probabilities for X.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Training vectors, where n_samples is the number of samples and\n",
    "            n_features is the number of features.\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        avg_proba : array-like, shape = [n_samples, n_classes]\n",
    "            Weighted average probability for each class per sample.\n",
    "\n",
    "        \"\"\"\n",
    "        probas = np.asarray([clf.predict_proba(X)\n",
    "                             for clf in self.classifiers_])\n",
    "        avg_proba = np.average(probas, axis=0, weights=self.weights)\n",
    "        return avg_proba\n",
    "\n",
    "    def get_params(self, deep=True):\n",
    "        \"\"\" Get classifier parameter names for GridSearch\"\"\"\n",
    "        if not deep:\n",
    "            return super(MajorityVoteClassifier, self).get_params(deep=False)\n",
    "        else:\n",
    "            out = self.named_classifiers.copy()\n",
    "            for name, step in six.iteritems(self.named_classifiers):\n",
    "                for key, value in six.iteritems(step.get_params(deep=True)):\n",
    "                    out['%s__%s' % (name, key)] = value\n",
    "            return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 4)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "foo = np.asarray([np.random.uniform(size=4) for k in range(3)])\n",
    "foo.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Combining different algorithms for classification with majority vote\n",
    "\n",
    "Let's apply the class we wrote above for classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Load the dataset\n",
    "\n",
    "Use only 2 features for more challenge\n",
    "\n",
    "Only last 100 samples in 2 classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "if Version(sklearn_version) < '0.18':\n",
    "    from sklearn.cross_validation import train_test_split\n",
    "else:\n",
    "    from sklearn.model_selection import train_test_split\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X, y = iris.data[50:, [1, 2]], iris.target[50:]\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)\n",
    "\n",
    "X_train, X_test, y_train, y_test =\\\n",
    "       train_test_split(X, y, \n",
    "                        test_size=0.5, \n",
    "                        random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Build individual classifiers\n",
    "\n",
    "Mixed types\n",
    "* logistic regression\n",
    "* decision tree\n",
    "* knn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10-fold cross validation:\n",
      "\n",
      "ROC AUC: 0.92 (+/- 0.20) [Logistic Regression]\n",
      "ROC AUC: 0.92 (+/- 0.15) [Decision Tree]\n",
      "ROC AUC: 0.93 (+/- 0.10) [KNN]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier \n",
    "from sklearn.pipeline import Pipeline\n",
    "if Version(sklearn_version) < '0.18':\n",
    "    from sklearn.cross_validation import cross_val_score\n",
    "else:\n",
    "    from sklearn.model_selection import cross_val_score\n",
    "\n",
    "clf1 = LogisticRegression(penalty='l2', \n",
    "                          C=0.001,\n",
    "                          random_state=0)\n",
    "\n",
    "clf2 = DecisionTreeClassifier(max_depth=1,\n",
    "                              criterion='entropy',\n",
    "                              random_state=0)\n",
    "\n",
    "clf3 = KNeighborsClassifier(n_neighbors=1,\n",
    "                            p=2,\n",
    "                            metric='minkowski')\n",
    "\n",
    "pipe1 = Pipeline([['sc', StandardScaler()],\n",
    "                  ['clf', clf1]])\n",
    "pipe3 = Pipeline([['sc', StandardScaler()],\n",
    "                  ['clf', clf3]])\n",
    "\n",
    "clf_labels = ['Logistic Regression', 'Decision Tree', 'KNN']\n",
    "\n",
    "print('10-fold cross validation:\\n')\n",
    "for clf, label in zip([pipe1, clf2, pipe3], clf_labels):\n",
    "    scores = cross_val_score(estimator=clf,\n",
    "                             X=X_train,\n",
    "                             y=y_train,\n",
    "                             cv=10,\n",
    "                             scoring='roc_auc')\n",
    "    print(\"ROC AUC: %0.2f (+/- %0.2f) [%s]\"\n",
    "          % (scores.mean(), scores.std(), label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Majority vote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC AUC: 0.92 (+/- 0.20) [Logistic Regression]\n",
      "ROC AUC: 0.92 (+/- 0.15) [Decision Tree]\n",
      "ROC AUC: 0.93 (+/- 0.10) [KNN]\n",
      "ROC AUC: 0.97 (+/- 0.10) [Majority Voting]\n"
     ]
    }
   ],
   "source": [
    "# Majority Rule (hard) Voting\n",
    "\n",
    "mv_clf = MajorityVoteClassifier(classifiers=[pipe1, clf2, pipe3])\n",
    "\n",
    "clf_labels += ['Majority Voting']\n",
    "all_clf = [pipe1, clf2, pipe3, mv_clf]\n",
    "\n",
    "for clf, label in zip(all_clf, clf_labels):\n",
    "    scores = cross_val_score(estimator=clf,\n",
    "                             X=X_train,\n",
    "                             y=y_train,\n",
    "                             cv=10,\n",
    "                             scoring='roc_auc')\n",
    "    print(\"ROC AUC: %0.2f (+/- %0.2f) [%s]\"\n",
    "          % (scores.mean(), scores.std(), label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Notice the better performance of ensemble learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Evaluating and tuning the ensemble classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Plot ROC curves for test data\n",
    "\n",
    "TP (true positive) versus FP (false positive) rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEKCAYAAAAb7IIBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4VMXawH+zKUAgFQgQICH0JkWKIB1UkCJeFD9AUBSQ\nK6IiKqIIIna5tqvXAhdBBC6IhaZAEAhVBJQaEEiAhBRaOpCe+f7YZN3d7CabZHvm9zznyc6ZOTPv\nu2czc868M+8rpJQoFAqFouqhcbQACoVCoXAMagBQKBSKKooaABQKhaKKogYAhUKhqKKoAUChUCiq\nKGoAUCgUiiqKp6MFsBQhhFqvqlAoFBVASilMnXepNwAppd2O1157za7t2ftQ+rn24c76ubNujtCv\nNFxqALAnFy9edLQINkXp59q4s37urBs4l35qAFAoFIoqihoAzDBx4kRHi2BTlH6ujTvr5866gXPp\nJ8qaI3IWhBDSVWRVKBQKZ0EIgXQHI7A9iYyMdLQINkXp59q4s37urBs4l35qAFAoFIoqipoCUigU\nCjdGTQEpFAqFogRqADCDM83T2QKln2vjzvq5s27gXPqpAUChUCiqKMoGoFAoFG6MsgEoFAqFogQ2\nHQCEEEuEEFeEEMdLKfNvIcQ5IcRRIUQnW8pTHpxpns4WKP1cG3fWz511A+fSz9ZvAEuBweYyhRD3\nAs2klC2AqcCXNpZHoVAoFEXY3AYghAgDNkopO5jI+xLYKaVcU5Q+DfSXUl4xUVbZABQKhaKclGYD\ncHRAmIbAJb10QtG5EgOAQqEoH38m/UlCRgIA0THRBAYEUrt2bZW2Q/r82WME+noR6FcLZC7xcRfw\nrVkDUas9rUI70LVRx4rfWCvi6AGgXEycOJEmTZoAEBAQQKdOnejfvz/w97yatdIff/yxTet3dFrp\n59ppS/T7x/J/0O7OdgTWDOT4tuPU8q1F0x5NAZw6ff3UdRITEst1/Yltx/CtVYMm3RqDLOT0rtP4\n+NQg7I4W4OFdovyZX3+nZg0NjToEgizk4uHLeHt7EtK9HXj7lyh/ccduanjmUq+VF8hCrvyVhUYj\nqNutE7JaA05sNyx/LmIf6Zn+dO19HYSGxKgsPDy9qNWuM81+68KNLv+w2e8jMjKSZcuWAej6S3M4\n2xTQX0A/Z5gCioyM1H257ojSz7WxRL+6b9Tl53E/071Zd+sLUFgAhdlQkA0FOUWfc6BGffAOLFn+\n6h7I+EtbvjBHW7YgG0IfgEDDtR+RkZH0D/4dkraWrL/zQmj8j5L1/z4FEjaAphp4VP/7b4c3IcSE\nGTJmKaQd0ytbDTTVoeEw8GulK5afD4mJEOr/F+RlaMt5VOdSog8jx9QnPtGTunUFUVGG1aenw2+/\nwZAhJZu292+ztCkgewwATdAOALeZyBsKPCWlHCaE6AF8LKXsYaYeZQNQVG2khMLcog60qEP0DgAv\n35Jlkw/T4pvh/DJwBi1qBhR1ojnQYAgElngWgzOfweUIvQ66qP4Or0OjkSXLH3gMYr/TdYi6DrfT\ne9DovpLloxdB8kFtJ+tR7e/yjf8BASW6Bkg9DjnXS9ZfvT541Sr/d2eGvDzYvRvi4yEtDZ591jD/\nyhUYMQIOHjQ8n5MDJ09Co0ZQty5oTCyniY6Oxtvbm9DQUKvJWxEcNgAIIVYB/YHaaOf1XwO8ASml\nXFRU5jNgCHATeExK+aeZutQAoHAMhQXaTrEwR9uBedYoWSYtCm5eKPnEGtwPAtqVLB/zNVz+1bCz\nLcyBNrOg0YiS5Q9OhejFoPH8uxP1qA6dP4Cwh0qWP/sfWqybzS+3300Lv7p/d6ChD0HtriXLJx+C\nrKS/yxXXXzPM9BO9i5GbC88/D59+ang+KwuGDoXGjaFJE1iwoPJtZWdnExERwZEjRwgICODJJ5/E\n29u78hVXEIcZgaWU4ywoM92WMlQUNYXgYKQEmQ8IbadnTGYM3LpU8om1dnfwb11Sv4v/gys7/y5b\nPA3R8iloOLxk/X/MhJjF2rIy/++n0C7/hqaPlCx/eZu2Qy+eSih+yg00Y+zza12ys9VUA9+Wpst3\n/Qy6fQFC+6hZ5v1r+RRpOfNJDZ8NlkwB1e5Wdhk7UZZuGRng6wtCr0uTEoYPh9hYSEjQPrnr97le\nXnDbbVBYaPi0XqMG7NxpPdmjo6PZuHEjGRkZeHh40KVLFzw9DX+/zvS/51JGYIUDuZWgfUI0eGLN\nhoAO4Nu8ZPlL6+Dq7pIdbrPHIeTekuWPvQoxS/Tqz9Z2dl3/Ay2mmqj/R0jcZPg0rKkGPo3Av3XJ\n8jUaQFAXw+kEj2qmpx8AOiyADvO19Wu8DHsbU7SeoT0spe6d2sNSNF6Wl3VxLlzQPpnHx8P48dpO\nWp/27eHYMQjUezERAmbMgHr1tNMyXkZflxDwxBO2lXv79u3s3bsXgJCQEO6//37q1q1r20YrifIF\nVFWJeltrCDPooLOh87+YHnOG67euG5ZPPwW34rWdsvAoOjRQq5m2czUm6zLkpeuV1wAaqBYEnibm\ncAtzQBbqlfUou9N1Ek6cOEH79u2LX7U5cOAAPXv2BHBoetPZTRz951GaB5kYoJ2Af/0LHnsMilZO\n6rjnHu2tb9QIFi6EoCDHyFdezpw5w9q1a+nfvz933nknGlOGAQfgzPsAFLZESthxF/RaA9XrGOY1\nfQwaP1jyiVjjzeerPVkxagUaof8Dvt+uorsShacKuTf8XqpXr05hYSH10+rTs7W2Q3Zk+qF2D9E0\nsKnN9ZdSexj3dy+/DL//rn2SX7cO2rY1zPfz015nTESE7WS1Ja1ateKZZ57Bz8/P0aJYjpTSJQ6t\nqPZj586ddm3PJqRFSflTqJSFhSWyStNPzBeyoLDAhoLZHre4f6VgT/2uXZPyjz+kXL9eysuXS+b3\n6yfl3r0lz2/eLOW2bVKePi1lTo7l7bnCvSs08T9lKfbWr6jvNNmvqjcAdyZxi3a+3UWmUhSOZd06\naNcOWrQwPP/ssxAVpV0pExamnWfXZ9u2knPuYHoNvKtTvMInICCAvn37OlqcSqNsAO7MjruhxVPQ\nuHzTN5rXNeTPyzeaAlKY44033uCZZ57B39/f0aKUyjffwJYt2imZl1/WLn80zr/9du1qGUVJ9Ff4\neHt7M2PGDGoYW6idEGUDqIrk34TrB6DPD46WxO2pXbs2Hh4edmuvoACSkrQdeUgIGO8zevFFaN0a\nJk0yPN+woXZTU6NG2pU0xjz6qO1kdmX01/XD3yt8XKHzLwv1iGeGYt8aLsv13yGoK3iZNkg5g36F\nhYUUFBTYJB0ZGWnT+vXT06ZNo1Yt6+1OLeaPP6CozzHglVegY8dInnkGDh0qmf/qqzBhQsnzd90F\n48ZB377OvbLGGX6b+mzZsoUjR47g4eHBoEGDmDRpUqWWdzqTfmoAcFfqD4T+vzhailJZsWIFkydP\ndpt0edmzRzu//sADsGRJyfy4OO1TvjHvvgs//KB1T/DAAyXz/f0NN0EpKseAAQNo2rQpU6dOpXfv\n3k6zvNMaKBuAogTKBlB+cnK0O1A1Gq1LAX1WrICYGHjtNcPz+/ZpO/FGjaBzZ2junMv1FS6OQ53B\nWQs1ANgPewwAJ0+eJCgoiJCQEJu1YW3i47WeIbsbeVb49lvtfHtIiPbv3LmG+SkpWhcEdYy2Yiic\ni+zsbPLy8vD1NeFcz4VRQeErgDPN09kCR+v3yy+/cPLkSZvVXxH9zp+Ht9+GJ5+E998vmX/hgnba\nxpgxYyA7Gy5eLNn5g3a+3dqdv6Pvny1xhG7R0dF88cUXrFu3Dls/aDrTvVOrgBQOYdasWTZv4+ZN\nrV9245eMw4dh6VL4z38Mz2dnax2NtWsHHUx4TO7TR3sYY2oNvMI1MF7hU6tWLbKysvDx8XGwZPZB\nTQG5GwXZkHoM6txR4SpcxQZw4wacPg3djBxZHjkCAwdq5+WHDIEffzTMT0/XTuW0aWM/WRXOR0xM\nDBs2bNB57nQ2Hz7WQk0BVSWu7oYjz1e6moiICBITE22W/uGHHwzSpsjIgP/+F+bPL2lABbh6FT77\nrOT5du0gOlr7BmDc+YN2lYzq/BUJCQlkZGQQEhLilit8LKFqaVsOnGmerlwkbtZGfSqDsvQ7duwY\nycnJVk1fv55MRoY2HR0dTVpaGqA1ko4zETkiLw/279c6DDN2TwDQtKl296ox3t5w4kSkW3vAcNnf\npwXYS7fevXszYsSISq/rLy9Ode/MOQlytgPlDM4yNraW8vqhMouZ0m/NmjUyOzu7ws7gCgqkPHGi\n5PkbN6Rs1UrKmjWlbNCgZH5urpRbt5a7uVJx2ftnIe6snzvrJqVzOYNTNgB34sZF2NodRl3WRY6y\nlMLCQl544QVefvll6n1ez6QNQErYtAkuXdKueX/jDUMXwAUF0LWr1siq7xlBSvjrL60rAlfylKtw\nD6KjowFoXkU3WigbQFUhaQs0GFyuzr+wsNiXu4YPP/xQ9yr8xBPaOXR9hNCueT9+XBulKS/PMN/D\nQ2uANXaLI4R2zl11/gp7kp2dzYYNG1i5ciXr1q0jKyvL0SI5HWoAMINTzdNZSo0G2kAvely+XLKj\nBujbN5KmTaFatQJ27DhbIn/IkJIBPgC++w6+/FLrb6ZaNWsJbn1c8v6VA3fWzxq6Fa/rL/bh06NH\nD6o5yQ/Wme6d2gfg4hw9qnUzEB8PY8eOJDjYMH/MGFi8uKQR9eGHYdAg2L//R0JCSrqGHDUKNG5s\nRFW4L5GRkezatQtwndi8jkLZABxEQWEBD33/EBk5GZWq59Qp7RN+9WoQ1gSqV6+8bNvPb3eJfQAK\nhSliY2NZsWIF/fr1c8t1/eVF+QJyQrLysvB/15+fx/1s8TXHjkGDEAiu5MPMsWPHSEhIYKhxRJAi\n/Kv7071hd5N5CoUrcPPmTWrWrOloMZwCFRCmAkRGRtK/f3+btuGh8eDuZndbXP7kBmgVCl2bVa7d\n5prmbErYVK62XQ173D9H4s76lUc3KSXCxIYPZ+78neneqQHAhXjuOevUEx4ezm0q7p/ChSn24VOt\nWjUGDx7saHFcFjUF5CCy8rIIej+IrDlWWJoW9z3kpkFz88FJLly4wPz581m2bJnJJyaFwlXQj83r\n6enJs88+a5OIbO6CmgJyd2LXQMPhpRZp3LgxU6ZMUZ2/wmUxF5tXdf4Vp2qbx0uhvGt1s/OzuXrz\narkOq1CYD5d/1W4AM0HxW5Onpye9e/fWnXemtci2QOnnupjTbceOHVaNzesonOne2fwNQAgxBPgY\n7WCzREr5nlG+H7ACCAU8gA+klMtsLZc1iUuPo9+yftzMvVl2YT1a1W5lUbncXHjzTXj9dUo6OLt+\nAGqFQ436Jq99/vnnmTx5Mm3bti2XbAqFs9G/f3/S0tK4++67XbLjd0ZsagMQQmiAs8AgIBE4BIyR\nUv6lV+ZlwE9K+bIQog5wBqgnpcw3qsspbQBJmUn0XdaXp7o9xYweM2zSxoED8M9/ajd9leDYq4CE\njm+ZvPbPP/8kJCSE+vVNDxAKhcK9caQvoO7AOSllrJQyD1gNjDQqI4HiIJy+QLJx5++sXL91nbu/\nvZuJHSfarPMHbRhCU5GogCL/P+bdP99+++2q81e4FNnZ2TpX4QrbYusBoCFwSS8dX3ROn8+AtkKI\nROAY8KyNZbKIsubp0rPTGbxiMCNajuCVPq/YVJZeveCxx8xk9t8MdXqWu05nmoe0BUo/1yQ6Oprn\nn3+etWvXUlhY6GhxbIIz3TtnWAU0GDgipRwohGgGbBNCdJBS3jAuOHHiRJo0aQJAQEAAnTp10m2o\nKP5SrZU+WjTfYir/Zu5Nes3rRcvaLXl70NsIIazevn76zju16cjI8l+/b98++vbtS0FBgcX6uUNa\n6eda6YiICA4dOkR+fj43b97kzJkzbN26lXvvvdcp5HOldGRkJMuWLQPQ9ZfmsLUNoAcwX0o5pCg9\nG21wgvf0ymwC3pFS7itKbwdeklIeNqrLKWwA2fnZDF81nDD/MBbft9jp/eUoG4DC2akqsXkdhSP3\nARwCmgshwoAkYAww1qhMLHAXsE8IUQ9oCZy3sVwVIrcglwe/e5C6NeuyaMQih3b+x48f5+rVq9x1\n110WpRUKZyUlJUUXm1d57rQvNu3BpJQFwHQgAogCVkspTwshpgohnigq9iZwpxDiOLANmCWlTLGl\nXJZQ/EpVTH5hPuN/HI+HxoPl9y/HQ+Nh+kIbcu7cOXbs2AFonV3px9wtK22MsX7uhtLPdejatSv/\n+Mc/dOv63Uk3UziTfja3AUgptwCtjM59pfc5Ca0dwGkplIVM3jCZtOw0NozdgJeHl13alRLuvx+W\nLYPAQO2T0vnz5xk4cCA9O4TQs/vturI9e/akZ8+eZtMKhbMihKBDhw6OFqNKonwBlYGUkum/TOf4\n1eNseXgLNb3t52VQSti9G/r2NbEBbGsP6Pg21B9oN3kUisoQHR1NTk4O7dq1c7QoVQrlC6iCSCl5\n6deXOJh4kO2PbLdr5w/aTr9fPxMZOcmQcRrq9rKrPApFRdD34VOtWjVCQ0Px9fUt+0KFzVFmdjNE\nRkbyxu432BK9ha3jt+JXzfERzXU2gKRtENwPPCoe49SZ5iFtgdLPOTCOzdu7d+8yffW7im4VxZn0\nU28AZlhzcg072MHuibsJqhHkaHEAPRtAjT0Qcq+jxVEoSmXPnj26RQtqhY9zomwAJoiIieCJjU+w\n9/G9NPJrZJc2jZHSxLw/gCyEn0Lgnt+0TuAUCifl8uXLLF26lD59+qh1/Q5E2QDKycW0i9zT7B6H\ndf4A69bBzz/Df/9rlJGXDo3uV52/wumpX78+M2bMoEaNGo4WRWEGNSSbIfFEokPb370bmhnF/j13\n7hw79h6B7l9Wun5nmoe0BUo/+2LOb09FOn9n083aOJN+Fg0AQghvIURzWwuj+JsDB0p6AC22ASgU\nzkJ2djYbNmxg48aNjhZFUQHKtAEIIYYBHwLeUspwIUQn4DUp5T/sIaCeHHazASz6YxGHEw+zaMQi\nu7Rniuxs8PTUHgqFM6Ifm9fDw4Pp06cTEBDgaLEURlTWBrAAuAPYCSClPKreBmxP9eqOlkChMI25\n2Lyq83c9LJkCypNSGkdncI2lQ5Ug8UQieXl5REdH6845On369Gm2bdtmBe2cax7SFij9bMfevXtt\nGptX3Tv7YckAcFoI8RCgEUKECyE+Ag7YWC6n4Pr160yePNk50lJSP/5Njv6x3zrKKRQVpE+fPrRp\n04apU6fSu3dvtbzThbHEBlATmAfcU3RqK/C6lDLLxrIZy2E3G8C89fP4I+kPfv7nz3ZpT5+8PLh2\nDUJCjDLSTsCukXBfjJkNAgqFQlGSysYEHiylfElK2bnomA249TbUnNwcsrOzHdJ2TAw8+qiJjMQt\n2t2/qvNX2Ins7GyuX7/uaDEUNsSSAeBVE+fmWFsQZ6JZ02bUuOWYzSutW4PJaf4ygr+XF2eah7QF\nSr/KUezDZ82aNeTn59u0LWPUvbMfZlcBCSEGA0OAhkKID/Wy/AD3jNbsrORlQvJBqDfA0ZIo3BxT\nK3xu3bqFn5/jnSEqrE9py0CvAieBbLTRvIrJBGbbUihHEx8fjwxyooVOV3ZC7TvAq5bVqiwOJu2u\nKP3Kz/nz51m/fr3DY/Oqe2c/zA4AUsojwBEhxEoppWMmxB2EI20AJqk3APzbOloKhZtz69YtFZu3\nimHJ0N5QCLFaCHFcCHG2+LC5ZA7EUTaAkychLs5Ehpcv+Fp3750zzUPaAqVf+WnXrh2jR4+2+rr+\n8qLunf2wZABYBiwFBNrVP98Ba2woU5XljTdg505HS6GoqgghaNu2rVrXX4WwZB/AH1LKLkKIE1LK\n24rOHZZSdrWLhH/L4db7AKSEhg1h715o2tRuzSqqINHR0WRmZtK5c2dHi6KwA5X1BZQjhNAAMUKI\nfwIJgFsH9HSEDeDmTRgyBMKVm3+FjdBf4ePp6UlYWBhBQc4R7U7hGCx513sOqAk8A/QCpgCP21Io\nR+MIG0CtWvD110b7vApyIf+mTdpzpnlIW6D0M8Q4Nm+/fv2c1nmbunf2o8w3ACnl70UfM4EJAEKI\nhrYUSlHEle1weiEM2uFoSRQuzG+//UZERASgXdc/cuRIgoODHSyVwhko1QYghOgGNAT2SimvCyHa\nAS8BA6WUdo2X6O42AJMcfhZq1Id2LztWDoVLk5yczH//+1969eqlYvNWQSrkC0gI8Q6wEngY2CKE\nmI82JsAxoKUN5HQanGYfQNJmrf8fhaIS1K5dmxkzZijPnYoSlPZrGAl0lFKORusJ9EWgh5TyAynl\nLbtI5yDsbQM4cEAbAN6AzBitC4iAjjZp05nmIW1BVdXPXGzeatWq2VAa61JV750jKG0AyC52+Syl\nTAHOSilVQFoboNGAh4fRyaQt0GCw8v6psIicnBw2btzId999h72mShWuj1kbgBAiDSi2PgpggF4a\nKeUoixoQYgjwMdrBZomU8j0TZfoDHwFewDUpZQmvZ1XOBnBqIfi1gkb3OU4GhUsQExPDhg0bdD58\npk6dqtw4KHRUdB/AA0bpzyrQsKboukFAInBICLFeSvmXXhl/4D/APVLKBCFEnfK2Y22cwgbQ9kXH\ntq9wenJycoiIiODPP/8E/l7hozp/haWYnQKSUm4v7bCw/u7AOSllrJQyD1iN1ragzzjgByllQlG7\nDo9A4ch4APbCmeYhbUFV0O/gwYP8+eefeHh4MHDgQCZNmuQWyzurwr1zFizZCVwZGgKX9NLxaAcF\nfVoCXkKInUAt4N9Sym9tLJdC4fL07NmT69ev06tXL7fo+BX2xxnWhHkCt6N1NDcEmCuEsK7ry3Ji\nz3gAM2fChQt2acoAZ/JJbguqgn6enp784x//cLvOvyrcO2fB4jcAIUQ1KWVOOetPAEL10o2KzukT\nD1wvijmQLYTYDXQEoo0rmzhxIk2aNAEgICCATp066b7M4tcqa6RzcnNIOplEZGSkTeovTufmwqJF\n/VmwwDb1q7R7pHNycvj5558JCgpyCnlU2rnTkZGRLFu2DEDXX5pFSlnqgXbK5gQQV5TuCHxa1nVF\nZT3QduRhgDdwFGhjVKY1sK2orE9RW21N1CXtxVeHv5LD3hpm83Z275aya1ejk0nbpLy8w+Zt79y5\n0+ZtOBJ30S86Olp++OGH8sMPP5TZ2dm68+6inyncWTcp7a9fUd9pso+25A3g38BwYF1RL3xMCGFR\ncFopZYEQYjoQwd/LQE8LIaYWCbVISvmXEGIrcBwoABZJKU9ZUr+r07YtfPWV0cmzn0Nji1bYKtwY\nUyt8srKyXGpDl8L5sSQewEEpZXchxBEpZeeic8eklLbZompeDlmWrNbCYfsACvPgh7ow4ixUd695\nXYXlXLhwgXXr1unW9ffr149evXopNw6KClHZeACXhBDdASmE8ACeBtw6JKTD9gFc268N/ag6/ypN\nfn6+Ljav8typsCWWPFI8CcxEa8y9AvQoOue2OGwfQNIWaGAf52/FRiN3xZX1a9GiBWPHji11Xb8r\n61cW7qwbOJd+lrwB5Espx9hcEoV2AOha7g3XCjekZUu3drircBIssQHEAGfQBoL/UUqZaQ/BTMjh\nVjaARx+Fhx+Ge+7RO5m4FeoPAo2t9+cpnIGYmBiuXbtGjx49HC2Kwo2plA1AStlMCHEnMAZ4XQhx\nFFgtpVxtZTmdBnvYAP79b/A0/vZDBtu0TYVzkJOTw9atWzly5AhCCMLDw6lXr56jxVJUQSxaViCl\n3C+lfAbtjt0MtIFi3BZ72AD8/aFmTZs2USrONA9pC5xVv5iYGD7//HNdbN6BAwdWyHmbs+pnDdxZ\nN3Au/cp8AxBC1ELrwG0M0AZYD9xpY7kUCrfj0KFD/PLLL4CKzatwDiyxAVwENgLfSSn32EMoM3K4\nlQ1AUfXIyMhg0aJF9OjRQ8XmVdiNyu4DaCqlNB1nzk2xtQ0gIwP8/PROFOYrw28VwM/Pj2effRYv\nLy9Hi6JQAKUHhf+g6OMPQogfjQ87yecQbGkDSEyE5s3B4GVmWx9I+cMm7ZnDmeYhbYGj9cvPzzd5\n3lqdv6P1syXurBs4l36lPXauKfqrFqZbkT174M479UL9Zl+DjNPgf5tD5VJYh+IVPqmpqTzyyCMI\nFdNZ4cSYHQCklAeLPraRUhoMAkUO3iyNCuZy2DIewIULMEDflV5SBNQbAB7eNmnPHMVuZN0VR+hn\nHJv38uXLNGjQwCZtufP9c2fdwLn0s2Ti+XFKvgVMMnHObbClDWD2bKMTSVugwRCbtKWwD/rr+kGt\n8FG4DqXZAP5PCPETEG40/78NSLOfiPbHbr6AZCEkbYUQ+w8AzjQPaQvsqd/Ro0d16/oHDRpkl9i8\n7nz/3Fk3cC79SnsDOAgko43i9R+985nAEVsKVWW4dQkCboOaYY6WRFEJunXrxtWrV7njjjvUU7/C\npShzH4CzoPYBKBQKRfkpbR9AaVNAu4r+pgohUvSOVCFEiq2EdQZsZQPYvRtyyhtVWeE05OTkkJBg\nHNJaoXBdStuKWLxWpQ5QV+8oTrsttrABSAkffABmlofbHWeah7QF1tav2IfPqlWruHnzplXrrgju\nfP/cWTdwLv1KWwZavPu3MZAopcwVQvQGOgAr0DqFU1iIELB+vaOlUJQXUyt8cnNzqelIT34KhZWw\nxBfQUaAb2ohgW4BNQAsp5XDbi2cgh7IBKOzKxYsX+emnn3Tr+vv37698+Chcjsr6AiqUUuYJIUYB\nn0op/y2EcOtVQDaPByAlRH8FzaaAxsN27SgqhYeHh4rNq3BrLHmUyRdCjAYmoH36B3Brb1Y23weQ\ndhxOf+DQzt+Z5iFtgTX0a9y4MY888ohd1vWXF3e+f+6sGziXfpYMAI+jNQi/L6U8L4QIB/5nW7Hc\ni5gY2LBB70TSFods/lKUn/DwcDXlo3BbLNoHIITwBJoXJaOllHZfy+LKNoCPPoKzZ+GLL4pO/Nof\n2syChkOtUr+icsTExBAfH0+/fv0cLYpCYXUqZQMQQvQBvgUSAAHUF0JMkFLus66YzoO1bQB79sCD\nDxYl8jLdfzxsAAAgAElEQVS0rp/rqc7G0Riv8GnWrBmNGjVysFQKhf2w5N32I2ColLKXlPJOYBjw\niW3FcizWtgHcdZeeB9DL26FOT/B07DJCZ5qHtAVl6Wccm3fQoEGEhITYRzgr4M73z511A+fSz5JV\nQN5SylPFCSnlaSGEfX0XuzjTpukl/NtC+3kOk0UBR44cYUORUUat8FFUZSzZB7AMyEa7+QvgYcBH\nSvmobUUrIYfL2gAUzkVWVhaLFi2iS5cual2/wu2pkC8gPf4JnAdmFR3nganlaHyIEOIvIcRZIcRL\npZTrJoQo3m/gUGy+D0DhUGrUqMFTTz1F7969VeevqNKU+usXQtwGDAF+klLeV3QslFJa1DsKITRo\nA8cMBtoBY4UQrc2UexfYWl4FbIHd4gE4EGeah7QFxfrl5eWZzPf0tGT203lx5/vnzrqBc+lXmjfQ\nV4B1aKd8tgkhHq9A/d2Bc1LKWCllHrAaGGmi3NPA98DVCrThtMTHw8yZjpaiapKbm8uGDRtYunQp\nBQUFjhZHoXBKSnsMehjoIKW8KYSoC/wCfF3O+hsCl/TS8WgHBR1CiBDgfinlACGEQZ6jsFZM4Jo1\nYfBgKwhkA5wpLqm1iYmJISoqSufDJyEhgdDQUEeLZVXc+f65s27gXPqVNgDkSClvAkgprxVN09iC\njwF924BJY4U9sZYNIDBQbwBI+AXi18Ediypdr8I0KjavQlE+ShsAmgohfiz6LIBmemmklJYYaxPQ\nehEtplHROX26AquFEAJtrIF7hRB5UsoNRuWYOHEiTZo0ASAgIIBOnTrpRtPieTVrpJs1bcburbuJ\njIy0Xv0bF0P1YPrfgdXlrUj6448/ttn356h0dHQ0CQkJeHh4cPXqVfr06aPr/J1BPmum3fH+Faf1\n58idQR5X0y8yMpJly5YB6PpLc5hdBiqEGFTahVLK7aXWrK3DAzgDDAKS0MYZHiulPG2m/FJgo5Ty\nRxN5dlsGuuiPRWzYuoFNr2wqu7ClbGgOfX6EwA7Wq7MSROoNbu6ClJKIiAg6d+7MqVOn3E4/fdzx\n/hXjzrqB/fWrkCsISzr4spBSFgghpgMRaA3OS4o2kk3VZkvj+RCnCFBsLRuAjoxzUJClDQDvJLjj\nP5gQgsFFc27uPu3jjvevGHfWDZxLP5uvhZNSbgFaGZ37ykzZiqw0sjrWsAG88Qa0aAFjxqD1/tlg\niDYsmKLS5OTkkJSUVObrrUKhKB21C8YE1tgHsGkTNGhQlEg94nTun/XnIV2JYh8+//vf/0hLSzNb\nzlX1sxR31s+ddQPn0s/iNwAhRDUpZY4thXEXbt6EqCjoXryo9Y4lOMnslstiaoWPWt+vUFQOS3wB\ndQeWAP5SylAhREdgspTyaXsIqCeHy/gCklK7CaxxYysLVkWJi4vjhx9+ULF5FYoKUNmYwP8GhqPd\nFYyU8pgQYkDpl7g2lbUBCOG6nX+TJk2IjY11tBgKhaKchIWFcfHixXJdY8kAoJFSxgpDA6Zbv3s3\na9qMqHNRjhbDpphbihYbG4u93rQUCoX1EBVYZGLJAHCpaBpIFq3rfxo4W+6WFAqFQuFUWDKJ+iQw\nE+2O3itAj6Jzbktl9gFkZYHONplyBG6ct55gVsSZ1iIrFArHUOYbgJTyKjDGDrI4DZWxAaxYAadP\nw4cfAidfh9CHoFZT6wqoUCgUVsCSoPCLMbGGUUr5hE0kcgIqYwOYMqXoDaAgF67shO7/ta5wVsLd\nt9srFIqysWQK6Fdge9GxDwgG1H6AUvDwAK7vA7/WUL2Oo8VRlMGqVasYMqRiG/Xat2/P7t27rSyR\n8zN06FC+/fZbm9S9detWRo1yeGBApyI3N5c2bdqQnJxs3YqllOU60A4a+8t7XWUPraj2Ye66uXLo\nF0MrV8mfs6Q8Ns86AtkRe37PFaFJkyZy+/btDml74sSJcu7cuZWu5+LFi1IIIX19faWvr68MDw+X\n7777rhUkdA+6du0qDx486GgxLGbWrFmydu3ask6dOvKll14qtezixYtl8+bNpa+vr7z33ntlYmKi\nLm/hwoWyffv20tfXVzZt2lQuXLjQ4NqFCxfK559/3mzd5v53i86b7FcrspMmHKhnrQHIGbFKPICk\nzRByr3UEUrgdQgjS09PJyMhg7dq1vPHGG2zfXmn/iyVwtd3Shw8fJiMjg27dujlaFIv46quv2LBh\nAydOnOD48eNs3LiRRYtMx/yIjIxkzpw5bNy4kZSUFJo0acLYsWMNynz77bekpaWxefNmPvvsM777\n7jtd3tixY/nmm2/MhjmtCGUOAEKIVCFEStGRBmwDXraaBE5IRX0BRUVBdjYgCyFsHAQ574/YmfyR\nWIvFixfTokUL6tSpw/33309SUpIuLyIigtatWxMYGMhTTz1F//79+fprbYC7b775hj59+ujKPvfc\nc9SrVw9/f386duzIqVOnWLx4MStXruT999/Hz8+PkSO1kU3Dw8PZsWMHAIWFhbz99ts0b94cf39/\nunXrRkKCcfiLv5FF+y26dOlCu3btOHr0qC4vKSmJBx98kODgYJo1a8ann36qy8vOzubRRx8lKCiI\ndu3asXDhQhrr7TwMDw/n/fffp2PHjtSqVYvCwsJS6zt06BDdunXD39+fBg0a8MILLwBa9xsTJkyg\nTp06BAYGcscdd3Dt2jUABgwYoPv+pJS8+eabNGnShPr16zNx4kQyMjIA7b4SjUbD8uXLCQsLIzg4\nmLffftvsd7J582b69etncG7GjBmEhobqvtO9e/fq8h577DHmzZunS+/atcvgu4iPj+eBBx4gODiY\nunXr8swzz5htuyIsX76c559/ngYNGui+u2Jf/Mb8/PPPjB49mtatW+Pp6cncuXPZvXs3Fy5cAOCF\nF16gU6dOaDQaWrZsyciRI9m3b5/u+oYNGxIUFMSBAwesJn9ZQeEF0BGoW3QESimbSim/K+26qoiU\nMGQIXLoECA20mw0aD0eLVWXYsWMHr7zyCt9//z1JSUmEhoYyZox28dr169cZPXo07733HsnJybRq\n1YrffvvN4PriTTQRERHs3buX6Oho0tPT+e6776hduzZTpkzh4YcfZtasWWRkZLB+/foSMnzwwQes\nWbOGLVu2kJ6eztdff42Pj49ZmYsHgAMHDhAVFUXz5s1150eMGEHnzp1JSkpi+/btfPLJJ2zbtg2A\n+fPnExcXx8WLF9m2bRsrVqwosQlo9erVbN68mbS0NIQQpdb37LPPMmPGDNLT04mJieGhhx4CtANj\nRkYGCQkJpKSk8OWXX1KjRskHo6VLl7J8+XJ27drF+fPnyczMZPr06QZl9u3bx7lz5/j1119ZsGAB\nZ86cMfmdnDhxglatDJwH0717d44fP05qairjxo1j9OjR5Obmmv1ei7+LwsJChg8fTnh4OHFxcSQk\nJOh+E8b873//IzAwkKCgIAIDAw0+BwUFER8fb/K6qKgoOnbsqEt37NiRqCjLFpAUFhYCcPLkSZP5\ne/bsoV27dgbnWrduzbFjxyyq3xJKHQCK5o9+kVIWFB1VYotoRfYBxMZCXh4U/Q87PZVZATR//nzm\nz59vtbQ1WLVqFZMmTaJjx454eXnxzjvvcODAAeLi4ti8eTPt27dn5MiRaDQannnmGerVMz2L6eXl\nRWZmJqdOnUJKSatWrcyWNWbJkiW89dZbuo78tttuIzAw0GRZKSV169bFx8eHXr16MW3aNN1bxaFD\nh7h+/Tpz5szBw8ODJk2aMHnyZFavXg3A2rVrmTNnDn5+foSEhJh8qn322WcJCQmhWrVqZdbn5eVF\ndHQ0ycnJ+Pj40L3Ii6GXlxfJycmcPXsWIQSdO3emVq1aJr/7mTNnEhYWho+PD++88w6rV6/WdXBC\nCObPn4+3tzcdOnSgY8eOZjuxtLQ0fH19Dc6NGzeOgIAANBoNzz33HDk5OWYHEH1+//13kpKSeP/9\n96levTre3t7ceeedJsuOHTuW1NRUUlJSSE1NNfickpJCo0aNTF5348YN/P39dWk/Pz9u3LhhsuyQ\nIUNYu3YtJ0+eJCsriwULFqDRaLh161aJsq+99hpSSh577DGD876+vqV6wS0vluwEPiqE6CylPGK1\nVp2citgArl6FRx6pGi7/jTvvyqatQWJiIl26dNGla9asSVBQEAkJCSQmJhpMCwBm/6EHDBjA9OnT\neeqpp4iLi2PUqFH861//MtnxGXPp0iWaNrVsz4cQQrei45NPPmHVqlXk5+fj6elJbGwsCQkJBAUF\nAdrBorCwkL59++p01ZffWDdj/cqq7+uvv2bu3Lm0bt2apk2bMm/ePIYNG8aECROIj49nzJgxpKen\n8/DDD/P222/j4WH4ZpuYmEhYWJguHRYWRn5+PleuXNGd0x9EfXx8zHaSgYGBZGZmGpz717/+xddf\nf62b0svMzOT69esmr9cnPj6esLAwmzoNrFWrlm66CyA9Pd3sb2XQoEHMnz+fUaNGkZmZyYwZM/D1\n9S3xW/zss89YsWIFe/fuxcvLyyAvMzOTgIAAq8lv9psRQhQPDp2BQ0KIM0KIP4UQR4QQf1pNAiek\nIjaA7t3h/fdtJJANcDcbQEhIiIETu5s3b5KcnEzDhg1p0KABly5dMihv7pUeYPr06Rw+fJhTp05x\n5swZFi5cCJTta6Vx48bExMRYLLOUEiEEM2bMoFq1anz++ee6epo2bUpKSoruKTQ9PZ2NGzfqdNWX\nPy4urkTd+rKWVV+zZs1YtWoV165dY9asWTz44INkZWXp5qmjoqLYv38/mzZtYvny5SXaMv7uY2Nj\n8fLysvjNSZ8OHTpw9uzfnmb27t3LwoUL+f7773VP5n5+frrps5o1axo8QevbfRo3bkxcXJzuTaQ0\nVq1aha+vL35+fgZH8Tlzv5d27doZvM0cPXq0xLSNPk8++SRnz54lKSmJUaNGkZ+fT/v27XX5X3/9\nNe+//z47duyggS6gyN+cPn3aYMqpspQ2NB4s+nsf2oheQ4HRwINFfxUKh5Cbm0tOTo7uKCgoYOzY\nsSxdupTjx4+Tk5PDK6+8Qo8ePQgNDWXYsGGcPHmSDRs2UFBQwGeffWbwdKrP4cOHOXjwIPn5+dSo\nUYPq1avrniDr1avH+fPmXXtMnjyZuXPnEh0dDWjns1NTU02WNZ5NnT17Nu+99x65ubl0794dX19f\n3n//fbKzsykoKCAqKorDhw8DMHr0aN555x3S0tJISEjgP//5T6nfV1n1rVy5UvdE7e/vjxACjUZD\nZGQkJ0+epLCwkFq1auHl5VXi6R+00ycfffQRFy9e5MaNG8yZM4cxY8bovrfyzBwPHTrU4OEkMzMT\nLy8vateuTW5uLgsWLDB4Q+jUqRO//PILqampXL58mU8++cRA7wYNGjB79mxu3bpFTk4O+/fvN9nu\nuHHjyMzMJCMjw+AoPmfujfGRRx7hww8/JDExkYSEBD788MMS0zbF5OTk6OwDcXFxPPHEE8yYMUM3\nhbRy5UrmzJnDtm3bDN6oiklMTCQ1NZUePXqU/iWWg9IGAAEgpYwxdVhNAiekUjGB94yGGxesK5AN\ncOVdwMOGDcPHx4caNWrg4+PD66+/zqBBg3jjjTcYNWoUDRs25MKFC7o57tq1a7N27VpefPFF6tSp\nw19//UXXrl2pVq1aibozMjKYMmUKQUFBhIeHU6dOHV588UUAJk2aRFRUFEFBQbqNSvpP2jNnzuSh\nhx7innvuwd/fn8mTJ5OVlWVSB+O3iWHDhhEUFMTixYvRaDRs2rSJo0ePEh4eTnBwMFOmTNFNNcyb\nN4+GDRsSHh7OPffcw+jRow10Ma67rPq2bNlCu3bt8PPz47nnnmPNmjVUq1aNy5cv8+CDD+Lv70+7\ndu0YMGAA48ePL9HG448/zoQJE+jbty/NmjXDx8eHf//732blKe1NqnPnzgQEBHDo0CEABg8ezODB\ng2nZsiXh4eH4+PgYTHlNmDCBDh060KRJE4YMGWJg5NVoNGzcuJFz584RGhpK48aNDZZVWoOpU6cy\nYsQIbrvtNjp27Mh9993HlClTdPnt27fnf//7H6BdvTVu3Dh8fX3p0aMHvXr1YsGCBbqyc+fOJSUl\nhW7duunePKZNm6bLX7lyJY8++miJaaHKYDYgjBAiHvjQ3IVSSrN5tsCeAWFeWvsShxMOs31GOddl\nZ12BTa3ggWugsd5NsidFwSMcLYZNkVLSqFEjVq1aVWLJoSvy5ZdfsmbNGnbu3OloUazCtm3b+OKL\nL/jxxx8dLYrTkJubS6dOndi9ezd16pj2LmDuf7e0gDClvQF4ALUAXzOH21JeG8DmzRATA1yOgPqD\nXKLzdzcbQFlERESQnp5OTk4Ob731FoBVX6XtyeXLl9m/fz9SSs6cOcMHH3zgVq4T7r77btX5G+Ht\n7c2pU6fMdv4VpbRVQElSygWl5CuKiIuDkBDgxmZooHb/OiO//fYb48aNIy8vj7Zt27J+/XqTU0Cu\nQG5uLlOnTuXixYsEBAQwduxYnnzSrT20K2xEaVNAR6SUne0sj1mcPiZwYQH8VA+GHIGaLhoPkqox\nBaRQuCPWngIaZC3BXI0K+QJKPwnVG7h0569QKKoWZgcAKWWKPQVxJirkCyiwIwy2no8OW1PVbAAK\nhaIkttsiVxXxrOloCRQKhcJi1ABgAkv3AeTkwIwZWkdwroYr7wNQKBTWQQ0AJrDUBvDHH7B7t/v4\n/8nJUYHeFIqqhBoATGCpDWDPHtBzI+9SGNsAYmJidL5oqiqWhjn09fXl4sWLthfIzsyaNavK/waM\nOXLkiM5pnjti8wFACDFECPGXEOKsEOIlE/njhBDHio69QojbbC2TtbjnHnhiSiFc+80154GKOHPm\nDCtWrDDwauisNGnSBB8fH/z9/QkKCqJ379589dVXVlm6+ssvvzBhwoQyy2VmZtKkSZNKt6ePviMy\nDw8PfHx8dOeKXQnYkitXrrB69WomT55s87asQUpKCiNHjqRWrVo0bdq0TBcPL7/8si6gyqBBg/jr\nr790eadOnWLAgAEEBATQqlUrnZM80Lqm8PHxYevWrTbTxaGYixVpjQPtABMNhAFewFGgtVGZHoB/\n0echwAEzdZmNhWltyhUTOPlPKTe0tK1ANiY/P18uXrxY7tmzxyViAu/YsUNKKWVGRobcuHGjDA8P\nl4899piDJbMe4eHhOh3NkZ+fb9U233nnHTlt2jSr1mlLHnzwQfnwww/LrKwsuWvXLunn5yfPnDlj\nsuzKlStl48aNZWxsrCwoKJCzZs2S3bt3l1JKmZubK5s1ayY//fRTWVhYKLdt2yZr1qwpz58/r7v+\nm2++kffff79d9KoM5v53sXJM4PLQHTgnpYyVUuYBq4GRRgPQASllelHyANDQxjKVSbn2AbhB7F8P\nDw8ef/xxevfu7WhRLEIWPe37+voyfPhw1qxZwzfffMOpU6cA7U7ZF154gbCwMBo0aMC0adMM7Bvr\n16+nc+fO+Pv706JFCyIiIgDDMIcxMTH079+fgIAAgoODDWK3ajQanVfQjIwMHnnkEYKDgwkPD9e5\nmYC/Q02++OKLBAUF0axZM7Zs2WKRfsU6FjN37lzGjBnDuHHj8Pf3Z+XKlUgpdSEog4ODGTduHOnp\n6bpr9u3bR8+ePQkMDOT2229nz549Zts0DsWYkpLCsGHDCA4Opnbt2tx3330kJibq8hs3bszu3bsN\n5Hv88cd16d27d9OzZ08CAgIICwtj5cqVZeptKZmZmaxfv5633nqL6tWr07dvX4YPH86KFStMlr94\n8SJ9+/YlNDQUjUbDww8/rPutREVFkZyczPTp0xFCcNddd3HHHXcY1NW/f3+2bdvmcvGVLcHWA0BD\nQN8Rezyld/CTgc02lcgCyrUPIHELNBhiW4GsSLF3SmMbgC2DZtiabt260ahRI10H99JLLxEdHc3x\n48eJjo4mISFB53Xx4MGDPProo3zwwQekp6eze/duk9M5c+fOZfDgwaSlpREfH8/TTz+ty9P3Zjl9\n+nQyMzO5ePEikZGRLF++nKVLl+ryDx48SJs2bUhOTubFF19k0qRJFdZz3bp1jB8/nvT0dP7v//6P\nDz/8kM2bN7N3717i4+OpVauWTs5Lly4xcuRI3njjDVJTU3n33XcZNWqUWffUxqEYCwsLeeKJJ4iP\njyc2NhZvb29mzJhhkZwXLlxg2LBhvPDCC6SkpHDkyBFuu830zO4///nPEuEXiz937drV5DVnzpyh\nRo0aBi6TSwvFOHbsWM6cOUNMTAy5ubksW7aMoUOHAqY9k0opDcI0hoaGIqXk3LlzFunvSlgSEcwu\nCCEGAI8BZh9DJ06cqPtnDQgIoFOnTrrljMUdmrXSyeeTiYyMLL183g36px6F4H5Wb9/a6YiICA4d\nOkSNGjV48skndQHIjctbxPH5cPL1kufbvwYd5ltW3lzZChISEkJKinbv4uLFizlx4oTOz/rs2bN5\n+OGHeeutt/j666+ZNGkSAwcOBNAF8zbGy8tLF0mrYcOGBqEEi5/OCwsLWbNmDcePH8fHx4ewsDCe\nf/55vv32W51P+LCwMN2T8aOPPspTTz3F1atXCQ4OLreOvXv31nVc1apV46uvvmLJkiXUr18f0A5a\nLVu2ZPny5Xz77beMHDmSu+66C4B77rmHjh07smXLFoO3mWLS09MNQjHWqVNHF6LS29ub2bNnM2zY\nMIvkXLlyJUOHDuWBBx4A0HXqpvjyyy/58ssvLfwGtBiHYQRtKEbjSGLFNGzYkJ49e9KiRQs8PT0J\nCwtjx44dALRt25aAgAA+/vhjpk+fzq+//srevXsZPHiwQR3WDsVoSyIjI3WB6cuyVdl6AEgAQvXS\njYrOGSCE6AAsAoZIKU0/ooBOKVMYr2uvTDo+Pp6grkEG5/Q/Swmvv96fHz7ZAJ69wLOGVdu3djom\nJoaoqCjy8/O5desWly5dKvE0V659AR3ml6/zLm/5ClAc8vDatWvcunXLIDxkYWGhrtO+dOmSRR3Z\nwoULefXVV+nevTtBQUHMnDmzRKCP69evk5+fT2jo3z/xsLAwEhL+/okXd84ANWrUQErJjRs3KjQA\nGId+jIuLY8SIEQaBVzQaDVevXiU2NpZVq1bx008/6fLy8/O5917T05UBAQEGHejNmzd59tln2bZt\nG+np6Tq5LeHSpUs0a9as3PpZinEYRig5gOkzb948jh49SlJSEnXr1mXp0qUMGDCAU6dO4e3tzfr1\n63n66ad566236N69O6NHj8bPz8+gDmuHYrQl/fv3N/h/fv11Ew9rRdj6vf8Q0FwIESaE8AbGABv0\nCwghQoEfgAnSSQLNWGID+OgjCKxTA5pPtZNU5ScnJ4cNGzboVviEhITwxBNP0LJlS0eLZlUOHTpE\nYmIiffr0oU6dOvj4+BAVFaULgZiWlqabG7c0bGNwcDCLFi0iISGBL7/8kmnTppWIBlanTh3dm0Ix\nsbGxNGxoGzOW8XRF48aN2bZtm0Gox5s3bxIcHEzjxo15/PHHDfIyMzN5/vnnTdZtHIpx4cKFxMbG\ncvjwYdLS0nRPzMUYh2K8fPmygVzFUdHKYsqUKWZDMXbubNoXZatWrcjKyjL43o8dO2Y2FOOxY8cY\nO3Ys9erVQ6PRMGnSJK5cuaJbCdShQwd27drFtWvX+Pnnn4mOjqZ79+666+Pi4hBC0KJFC4t0ciVs\nOgBIKQuA6UAEEAWsllKeFkJMFUI8UVRsLhAEfF4Ub/igmersRlk2ACGgUycQIXdD4/vtKFn5SExM\n5MiRI3h4eDBo0CAmTZqke/J0B19AmZmZbNq0ibFjxzJhwgTatm2LEIIpU6YwY8YMrl27BmjfDooN\nvZMmTWLp0qXs3LkTKSWJiYkGHV8x33//ve5JPiAgAI1GU8JOotFoeOihh5gzZw43btwgNjaWjz76\nyKKlpNZg6tSpvPzyy7p4x1evXtUtYZwwYQI//fQTv/76K4WFhWRnZxMZGWnQUetjKhRj8XLb5OTk\nEk+RnTp1YvXq1RQUFHDw4EED//3jx49n69at/PTTTxQUFJCcnMzx48dNtrt48WKzoRiPHDli8hpf\nX19GjhzJ3LlzycrKYvfu3fzyyy+6aGXGdOvWje+++45r164hpWTp0qUIIWjatCmgtX/k5ORw69Yt\n3n33XVJTU3nkkUd01+/atYu77rrLZDhMl8fc8iBnO7Dj8sSvDn8lh701zG7t2ZL9+/fLK1eulDi/\nc+dOk+Xt+T1XhCZNmkgfHx/p5+cnAwIC5J133im/+OILWVhYqCuTk5MjX3nlFdm0aVPp7+8v27Zt\nKz/99FNd/rp162SHDh2kr6+vbNGihYyIiJBSSjlgwAC5ZMkSKaWUs2bNkg0bNpS+vr6yefPm8r//\n/a/ueo1GI2NiYqSUUqampsrx48fLunXrytDQUPnmm2/qyi1btkz26dPHQH79a80RHh4ut2/fbnDu\n1VdfLbHUtbCwUP7rX/+SLVq0kH5+frJFixZy3rx5uvwDBw7Ivn37yqCgIFmvXj05YsQImZCQYLLN\nK1euyNDQUJmbmyullDI+Pl727dtX1qpVS7Zu3Vp++eWXUqPR6MpHR0fL7t27S19fX3nffffJp59+\n2kC+Xbt2ye7du0s/Pz8ZFhYmV65cWarO5eX69evyvvvukzVr1pRNmjSRa9eu1eWdP39e+vr6yqSk\nJCmllFlZWfLJJ5+UDRo0kP7+/rJr167y119/1ZWfOXOmDAwMlL6+vnL48OHywoULBm0NHjxYbt68\n2ary2wJz/7uUsgzUbDwAZ8Pp4wG4CSoeQNVl9uzZhIaGGsShreocPXqUZ555xmDJq7NSkXgAagAw\nQWkxgQsKQKNxLv8/OTk5nD9/njZt2lS6LjUAKBSuibUDwlRZSrMB/PgjmJlqdAjFPnzWrl1LfHy8\nxde5gw1AoVBUDqfZB+Aq7NkDHcKj4a9N0NqyjTG2ICcnh61bt+oMZSEhIS4b41ahUDgGNQCYoLR4\nABMTCXMAAB2ZSURBVEePwphH14EDZ0kSExNZs2YNGRkZeHh40L9/f+68885y7eZV8QAUCoUaAExQ\n2j6AyJ0SNnwODTaYzLcHfn5+5OXlERISwsiRIyu0qUihUCjUAGCCZk2bEXXOtF8Rzc2zIPLA3/Sm\nE3tQq1YtJk6cSJ06dSrsw0ffzYVCoaiaqAGgvBQ7f3PwMiD11K9QKCqLWgVkglJjAidtsZv755iY\nGL7//nsKCwutXrd6+lcoFGoAMIEpG0BODvz1F9DzW2hg2wFA34dPVFSU2W30Cvtx6tQpunXr5mgx\nnI477riD06dPO1oMRQVRA4AJTO0DiI2FV14BqtcBTwtjBVSA4nX9+j58OnToYPV2XHUfQHh4uIFj\nstWrVxMUFMSePXuIjY1Fo9EwfPhwg2smTJigiwewa9cuNBoN06dPNyjTp08fli9fbrbdefPmMWvW\nLCtqYls++ugjGjRoQEBAAJMnTyYvL89s2R07dtClSxf8/f1p3rw5ixcvNsi/cOECI0aMwM/Pj+Dg\nYGbPnq3Le/HFF5k7d67N9FDYFjUAWEjLltpNYLbk/PnzJTx39u7d26WDtdiSb775hqeffprNmzfT\np08f3fnff/+dAwcOmL2uZs2afPvtt8TFxVnUzuXLl4mMjNT5x3d2tm7dyvvvv8/OnTuJjY0lJiaG\n1157zWTZ/Px8Ro0axZNPPkl6ejqrV69m5syZnDhxAoC8vDzuvvtu7rrrLq5evUp8fLyB07URI0aw\nc+dOrl69ahfdFNZF9SwmKNUGYEPCw8Np2rRpCc+dtsDVbQBfffUVL774IhEREdxxxx0GebNmzeKV\nV14xe21AQAATJ05k/vz5FrW1bds2br/9dry9vXXn3nvvPZo3b46fnx/t27dn3bp1urzXX3/dwCNo\n8ZtJsS0nNTWVxx9/nIYNG1K7dm1GjRplkRyWsnz5ciZNmkTr1q3x9/dn3rx5BlHK9ElJSSEzM1PX\nqXft2pU2bdroQiYuW7aMhg0b8uyzz1K9enW8vb1p37697vpq1arRpUsX9w2a7uaoAcAE5YoJbEWE\nEIwfP1499ZfB559/zvz589mxY0cJn/FCCKZNm8bZs2dL+LDXLzNnzhx++OEHi8L8GYdLBGjevDn7\n9u0jIyOD1157jfHjx3PlyhWDNozbLGb8+PFkZWVx+vRprl69ynPPPWey3X379hmESzQOnbh//36T\n10VFRdGxY0ddumPHjly9etVkOMjieMdff/01hYWF/Pbbb8TFxeneqA4cOEBYWBhDhw6lbt26DBw4\n0CBcIkCbNm04duyYSVkUzo3qZUxg0hfQrUQoyDF9QQW4efOmyfOmYpTagsrYAObP166CFUL72VS+\nufOlXWcpv/76Kz169DB4EtWnRo0azJkzh1dffdVsHcHBwfzzn/9k3rx5ZbaXlpZWItrUAw88QL16\n9QAYPXo0LVq04ODBskNZJCUlsXXrVr766iv8/Pzw8PAwmL7Sp1evXqSmpuoCuuh/TklJMQhTqY9x\nyEQ/Pz+klGZDJo4ZM4YFCxZQrVo1+vXrx1tvvUVISAigfRtes2YNM2bMICkpiaFDhzJy5Ejy8/N1\n17tSuESFIWoAsIDDh+Hcj/Ph0k+Vrqt4hc9//vMfs/+Qzs78+dqwmFKWfwAo7TpL+eKLLzh79myp\nAdYnT57MlStX2LRpk9kyL730Elu3bi1zlVVgYGCJe7V8+XI6d+6sezKPiori+vXrZcoeHx9PUFBQ\niZCD1sQ4ZGJ6ejpCCJMhE8+cOcP//d//sWLFCvLy8oiKiuK9995j8+bNgHYw7d27N/fccw+enp68\n8MILJCcnG6z8caVwiQpD1ABgAmMbwLvvFPD77xIa3F2pevVX+OTm5pbLe6e1cWUbQL169di+fTt7\n9uwx67vey8uL1157rdQVKkFBQcyYMYO5c+eW+uZlHC4xLi6OJ554gs8//1z3ZN6uXTudK17jcIlJ\nSUm6z40bNyYlJaVETFtT7N2712y4RD8/P/bt22fyunbt2hlMyRw9epR69eoRGBhYouzJkydp3bq1\nLnh8ixYtGDZsmG4A6NChQ5lvpadPnzaYclK4DmoAMIG+DUBK2LM7nz7drkO12hWrz0xsXmv476+q\n1K9fn+3bt7N161ZmzpypO6/vD338+PFkZ2frOjNTPPfcc+zfv7/Utex33303f/75J7m5uYB2+k6j\n0VCnTh0KCwtZunSpwbx4p06d2L17N5cuXSI9PZ13333XQO57772XadOmkZaWRn5+Pnv27DHZbu/e\nvc2GS8zIyKBXr14mr3vkkUdYsmQJp0+fJjU1lTfffLNEQPtiOnfuTHR0NDt37gS0DymbNm3Sdejj\nx4/nwIED7Nixg8LCQj766CPq1q2r++3m5OTwxx9/cPfdlXs4UjgGNQCYQN8GcOsWjLvnAGFmAlRb\nQnJyMkePHjUZm9dRuOo+AP2n0caNG7N9+3Z++OEH5syZUyJfo9GwYMECUlNTzT7F+vr6MmvWLFJS\nUsy2GRwczMCBA3Urfdq0acPzzz9Pjx49qF+/PlFRUfTu3VtX/q677uL//u//6NChA926dWPEiBEG\n9X377bd4enrSunVr6tWrxyeffFL+L6IUBg8ezKxZsxgwYADh4eE0a9bMYMXT0KFDdYNS06ZNWbJk\nCc888wz+/v4MGDCA0aNH66bXWrZsyYoVK5g6dSpBQUFs3LiRDRs24Omp9SKzYcMGBgwYQP369a2q\ng8I+qIhgJlj0xyI2bN3ApleK5o9/bgd3LIU63Stc5x9//EHjxo0d3vEXY84ZnIoIZprTp08zceJE\nfv/9d0eL4lT07NmTJUuW0LZtW0eLUuWpSEQw5QzOBAY2gIIcCOgEQV0qVWeXLpW73tq4sg3AEfx/\ne+ceVmWZLfDfQiUlFUJhQuQ2U2ZlZhwzyp7S1FKfpjzeUBTBp5PW1Eye7JRklrce5+I4TXVMPdaE\nN2yynEC0bErgOKZ2M7XMC6EoYsdJTbwkIuv8sT/27A17wxY27A37/T3Pftzf+63ve9f6Nr7rvX1r\nXX/99abxd8Enn3ziaxUMDcBMAbnA6T2AVldA35UQ1Kru6y5cMHF7DAZDs8E4ABfUlhPYHVU7fNau\nXUthYWEjaeY9musagMFg8B5mCqgW8vPh+HEYOdK9jKvcvI25x9tgMBi8hXEALqhaAwgLs20Ddcex\nY8fIyspqUG5eX2HWAAwGg3EALqhaA6jr3Zaqtx9Nbl6DwdAcMdtAXVB+qZyPN33M4M75cP1Ttb4A\nduLECcLCwppFr98Rsw3UYGhZ1GcbaPNqtZqI4FbBtK04AwcWQ5vQWmXDw8ObXeNvMBgM0AQOQEQG\ni8i3IrJPRJ5xI/OyiOwXkR0i0quxdfKEfteWwdUDIag1hYWFrFq1yikCYnMnkNcADh8+bI+QWR/m\nzZvHpEmTvKyVd2iobXXxwQcfeD1/QXOnvLyc66+/nh9++MHXqlw2jeoARCQIeBW4D7gRGCsi3avJ\nDAF+oarXApOBRY2pk6c8PCWWPeeG22P47N+/ny+++MLXagU88fHxtG3btkbohltuuYWgoCCPsnzF\nxMRw+vTpeofezsjIYMmSJUDNZC+Xw7Zt22jfvr1T4LgqEhMTWbhwYZ33qJ4is6G21cVzzz1HRkZG\no9y7MXjmmWfo3LkzERERTqksXbF06VKuvfZaOnbsyNChQ52C+M2aNYvg4GCnYHwHDx4EIDg4mIce\neoh58+Y1pimNQmOPAPoA+1X1kKpeBFYD1fPqPQgsA1DVbUCoiPyskfWqlfNnK/jwq2I27DrilJu3\nd+/evlTLqzTX9wBEhISEBLKysuxlu3fv5vz5802SS+HSpUtOx6pa73WT2267jZiYGNasWeNUvnv3\nbvbs2UNKSkqDdPU2n332GadPn+bWW2/1tSoesXjxYrKzs9m1axc7d+4kJyfH7rirk5eXx/Tp08nJ\nyeHEiRPEx8czduxYJ5kxY8Y4BeOLj4+3nxs7diyZmZm15l72RxrbAUQDhx2Oj1hltcmUuJBpUjb+\n7R/0uyefsjNnTW5ePyQ1NZXMzEz7cWZmJmlpaU4y69evJzExkdDQUOLi4pg1a5b9XPVee2lpKQ8+\n+CCdOnWiW7duLF261C47a9YsRo0aRWpqKmFhYWRmZjJr1iwmTJgAwN133w3YdoR17NiRgoICOnXq\nxNdff22/x/Hjx7nyyitdThFMmDChRjL65cuXM3ToUPsus+zsbHr06EF4eDj33HMPe/futV9bXFxs\nT9g+f/78Grb179+f559/njvvvJOOHTsyePBgp9HTsmXLiI+PJyIigrlz59YYUTiyYcMGu71VTJky\nhdjYWEJDQ7n11lvZvHmz/dzEiROdEu7k5+cTExNjPz5y5AgjRowgMjKSiIgIfvOb37ist74sW7aM\nqVOnEhUVRVRUFE899RRvvvmmS9nc3FxGjRpF9+7dad26NTNmzKCgoICioiKP6oqOjiY8PLzWXNT+\nSLNq0aryuM6cOZOXXnrJqRebl5fnteP+gxOJCI8kPDzcHrnTm/f3h+OqstrO+ytJSUmUlZWxd+9e\nKisreeuttxg/frxTL7x9+/YsX76cH3/8kdzcXBYtWkR2drb9vONoITk5mdjYWI4dO8bbb7/Ns88+\n6/QssrOzGT16NKdOnarRKy8oKACwh2q+6667GDt2LCtWrLDLZGVlMXDgQDp1qrmbLDU1lYKCAkpK\nSgDbiGLVqlWkp6cDsG/fPlJSUnj55Zc5fvw4Q4YM4f7776eiooJly5YRGxvLunXrOH36NE899VQN\n26rqz8zM5Pjx41y4cIH58+cD8M033/DYY4+RlZVFaWkpP/74I0ePHnX73F2lxuzTpw87d+7k5MmT\npKSkMGrUKHvYbFdU6VZZWcn9999PQkICxcXFlJSUMGbMGJfXZGVl1Zoa011eDVepMR0dc21UOVDH\nMN85OTl07tyZm266iUWLas5Ud+/e3S9SY+bl5ZGenu5Z3mtVbbQPkAS873A8DXimmswiINnh+Fvg\nZy7upU1JZWVlk9bnL3jynJmJVz71IT4+Xj/66CN98cUXNSMjQ99//3299957taKiQkVEDx065PK6\nKVOm6JNPPqmqqgcPHtSgoCC9dOmSFhcXa+vWrfXs2bN22YyMDJ04caKqqs6cOVPvvvtup3vNnDlT\nU1NTa9yrim3btmlsbKz9uHfv3vr222+7tWngwIE6b948VVXduHGjRkZGakVFhaqqzpkzR5OTk+2y\nlZWVGh0drfn5+U7Po4rq+vTr109ffPFF+/mFCxfqkCFDVFV19uzZmpKSYj937tw5DQ4OdrqfI4MG\nDdLFixe7tUNV9aqrrtKdO3eqqmp6errOmDHDfi4vL09jYmJUVXXLli0aGRnp9Ny8TatWrXTv3r32\n4/3792tQUJBL2b///e8aGRmpu3bt0nPnzumkSZO0VatWunr1alVV3bNnj5aWlmplZaVu2bJFo6Ki\n7OeqGDdunM6ZM6fR7KkLd/93rXKXbXRjvwj2KXCNiMQBpcAYYGw1mWzgMeAtEUkCTqnq9/iY/Pz8\nFr1Txt17AJ6gL/j+PYHx48dz1113UVRUZJ+OcWTbtm1kZGSwe/duysvLKS8vZ9SoUTXkSktLCQ8P\nJyQkxF4WFxfH559/bj92nLbwhD59+nDllVeSn5/P1VdfTWFhIQ888IBb+bS0NObNm8e0adNYsWIF\nY8aMoVUrW/DBo0ePEhcXZ5cVEWJiYuwjBk9wjNUfEhLCmTNn7Pd2tK1du3YuRylVuEqNOX/+fN54\n4w37gmlZWZnHqTHj4uIadVrVVWrM9u3bu5QdMGAAM2fOZPjw4ZSVlTFlyhQ6dOhA165dAVvvvorb\nb7+dJ554gjVr1pCcnGwvb46pMRt1CkhVLwGPAxuBr4HVqrpHRCaLyCRLZj1QJCIHgMWA6xx/BoMD\nsbGxJCQksGHDBpfbEseNG8ewYcMoKSnh1KlTTJ482eVCbZcuXThx4gRnz561lxUXFxMd/a9lqNoW\nl92dS0tLY/ny5SxfvpyRI0cSHBzs9h7Dhw/nyJEj5OXl8e677zqtZ3Tp0oVDhw45yR8+fNjeMDVk\n4TsqKspp+uT8+fO1bmWsnhpz8+bN/OEPf2DNmjX21JiOW1DrSo1ZXFzs0e6pVatW1Zoa090UkKvU\nmDfeeKPbeh599FH27dtHaWkpw4cPp6Kigh49eriUdbXw3xxTYzb6GoCqvq+q16nqtar6W6tssaou\ncZB5XFWvUdWbVdUv9lq25N4/tAz73njjDT7++GPatasZufXMmTNcddVVtGnThu3bt7Nq1SqX9+ja\ntSt33HEHGRkZ9nDer7/+OqmpqR7pEBERQVBQUI0IsOPGjWPt2rWsXLnS5QjFkZCQEEaMGMHEiROJ\nj48nMTHRfm706NHk5uayadMmKioqmD9/Pm3btuX2228HbL377777zul+rhydK0aOHElOTg5bt27l\n4sWLdc4XDx061GltpKysjDZt2tCpUyfKy8uZPXu20wihV69erF+/npMnT3Ls2DGnzGd9+vQhKiqK\nadOmce7cOS5cuMCWLVtc1puSklJraswqZ1idCRMmsGDBAo4ePUpJSQkLFixwmxrzwoUL9vWBqpzP\nU6ZMITTU9iJodnY2p06dAmD79u38+c9/ZtiwYfbrjx49ysmTJ0lKSqr1GfobzWoR2GBw7PEmJCQ4\nNZaO5xYuXMiMGTMIDQ1l7ty5TkP16mRlZVFUVESXLl0YMWIEc+bMoX///h7p065dO6ZPn07fvn0J\nDw9n+/btgM2xJCYmIiJO6SLdkZaWRnFxcY3dTFUpGR9//HEiIiLIzc0lJyfHnpJx2rRpzJkzh/Dw\ncBYsWFDjOdQ2Qrjhhht45ZVXSE5OtkexjYyM5IorrnApf8sttxAWFsann34K2FJP3nfffXTr1o2E\nhARCQkKcppRSU1Pp2bMn8fHxDB482GmRNygoiJycHPbv309sbCwxMTH89a9/rfM5XQ6TJ0/ml7/8\nJTfddBM333wzDzzwAA8//LD9fI8ePezbiX/66SdSUlLo0KEDSUlJ9O3bl9mzZ9tlV69ezTXXXEPH\njh1JT0/n2WefZfz48fbzK1euJC0tjTZt2njVhsbGxAJyQ0PmyJsDgRwLqKioiOuuu67W3Sre4KGH\nHiI6OtqpIfFnzp49S1hYGAcOHHBad3Dkww8/5LXXXuPdd99tYu38l/Lycnr16kVBQQGdO3f2mR4m\nJaTB4AG7du1y28B5i4MHD7J27Vp7ngh/Zd26dQwYMIDKykqmTp1Kz549a302gwYNYtCgQU2oof8T\nHBzMN99842s16oWZAnJDS+79Q8u3zx1/+tOfeOSRR/jd737XaHU8//zz9OzZk6effrrRHU1Dee+9\n9+jSpQtdu3alsLCQ1atX+1olQxNipoAMTgTCFJDB0BIx4aC9SHN5K7a+tHT7DAZD3RgHYDAYDAGK\nmQIyOGGmgAyG5onZBWRoMHFxcU0SVtlgMHiX+mw4MFNAbmjpc+Tu7Dt48GCjBghsqs+mTZt8roOx\nz9jWlPZVJai5HIwDcMOOHTt8rUKjYuxr3rRk+1qybeBf9hkH4IaquB8tFWNf86Yl29eSbQP/ss84\nAIPBYAhQjANwQ33m05oTxr7mTUu2ryXbBv5lX7PaBuprHQwGg6E5om62gTYbB2AwGAwG72KmgAwG\ngyFAMQ7AYDAYApSAdwAiMlhEvhWRfSLyjBuZl0Vkv4jsEJFeTa1jQ6jLPhFJEZGvrM9mEbnJF3rW\nB09+O0vuVhG5KCI1kwf7MR7+bfYTkS9FZLeIbGpqHRuCB3+bHUUk2/p/t0tE0n2gZr0QkddF5HsR\n2VmLjO/bFV+/FefLDzYHeACIA9oAO4Du1WSGALnW99uArb7W28v2JQGh1vfBzcU+T2xzkPsIWAcM\n97XeXv7tQoGvgWjruLOv9fayfRnAvCrbgB+A1r7W3UP77gR6ATvdnPeLdiXQRwB9gP2qekhVLwKr\ngQeryTwILANQ1W1AqIj8rGnVrDd12qeqW1X1R+twKxDdxDrWF09+O4BfA2uA/2tK5byAJ/alAO+o\nagmAqv6ziXVsCJ7Yp0AH63sH4AdVrWhCHeuNqm4GTtYi4hftSqA7gGjgsMPxEWo2gNVlSlzI+Cue\n2OfIfwAbGlUj71GnbSLSBRimqq8BzS3CnSe/XTcgXEQ2icinIpLaZNo1HE/sexW4QUSOAl8BTzSR\nbk2BX7QrJhqoAQAR6Q9MxDZ0bSm8BDjOLTc3J1AXrYFE4B7gSuATEflEVQ/4Vi2vcR/wpareIyK/\nAD4UkZ6qesbXirUUAt0BlACxDsddrbLqMjF1yPgrntiHiPQElgCDVbW2Yas/4YltvYHVYotv3RkY\nIiIXVTW7iXRsCJ7YdwT4p6r+BPwkIgXAzdjm1v0dT+ybCMwDUNVCESkCugOfNYmGjYtftCuBPgX0\nKXCNiMSJSDAwBqjeOGQDEwBEJAk4parfN62a9aZO+0QkFngHSFXVQh/oWF/qtE1Vf259ErCtA/yq\nmTT+4Nnf5nvAnSLSSkRCsC0m7mliPeuLJ/YdAgYCWPPj3YDvmlTLhiG4H3X6RbsS0CMAVb0kIo8D\nG7E5w9dVdY+ITLad1iWqul5EhorIAeAstl5Js8AT+4AZQDiw0OopX1TVPr7T2jM8tM3pkiZXsgF4\n+Lf5rYh8AOwELgFLVPUbH6rtMR7+fnOBNx22Uj6tqid8pPJlISKrgH5AJxEpBl4AgvGzdsWEgjAY\nDIYAJdCngAwGgyFgMQ7AYDAYAhTjAAwGgyFAMQ7AYDAYAhTjAAwGgyFAMQ7AYDAYAhTjAAx+g4hc\nEpEvrPDGX1gvqbmTjRORXV6oc5MVkniHiPyviFxbj3tMFpHx1vc0Ebna4dwSEenuZT23WW9v13XN\nEyLStqF1G1ouxgEY/ImzqpqoqrdY/xbXIe+tl1jGqmovbNEZ51/uxaq6WFVXWIfpOAT1UtVJqvqt\nV7T8l56v4ZmeU4AQL9VtaIEYB2DwJ2q8Nm/19AtE5DPrk+RC5garV/yF1UP+hVU+zqH8NetN59rq\nLQCqrh1gXfeViCwVkTZW+W+t5Cs7ROT3VtkLIjJVREZgiz+0wrq2rdVzT7RGCb930DlNRF6up56f\nAF0c7rVQRLaLLWnKC1bZry2ZTSLykVV2r4hssZ7jW1b4CEMAYxyAwZ9o5zAF9I5V9j0wUFV7Y4sX\n84qL6x4BXlLVRGwN8BFr2iUZuMMqrwTG1VH/A8AuEbkC+AswSlVvxpaw5FERCccWXrqH1ROf63Ct\nquo72AKVpVgjmJ8czr8D/LvDcTK2QHX10XMw8DeH42et8B03A/1EpIeqvoItuFg/VR0gIp2A6cAA\n61l+Dkytox5DCyegYwEZ/I5zViPoSDDwqthS5l0CXM3RfwJMF5EY4F1VPSAiA7CFSv7U6lG3xeZM\nXLFSRM4DB7ElkLkO+M4hOF4m8Cvgv4HzIrIUyMWWZcwVNXrwqvpPESkUkT7YonVep6pbROSxy9Tz\nCmyhnx1TCI4RkYex/X++GrgB2I1zMLIkq/wfVj1tsD03QwBjHIDB3/lP4Jiq9hSRVsD56gKqmiUi\nW4H7gVwroJgAmao63YM6UlT1y6oDq7fsqhG/ZDXgA4BRwOPWd095C1tv/1tgbVV1l6unNZX0KjBC\nROKx9eT/TVVPi8hfsDmR6giwUVXrGl0YAggzBWTwJ1zNfYcCpdb3CUCrGheJJKhqkTXtkQ30xJYH\neKSIRFgyV9Wyq6h6vXuBOBH5uXWcCuRbc+Zhqvo+8KRVT3XKgI5u6lmLLRXgGGwpEKmnns8Dt4lI\nN6uuM0CZ2EImD3GQP+2gy1agr8P6SEh9djwZWhbGARj8CVe7ehYC6SLyJbZ48GddyIy2Fma/BG4E\nlqnqHuA5YKOIfIUt7PDVLq6tUaeqXsAWnneNde0lYBG2xnSdVVaAbXRSnTeBRVWLwI73V9VT2OL1\nx6rqZ1bZZetprS38EfgvVd2JLaH6HmAFsNnhmv8B3heRj6x8wROBLKueLdimugwBjAkHbTAYDAGK\nGQEYDAZDgGIcgMFgMAQoxgEYDAZDgGIcgMFgMAQoxgEYDAZDgGIcgMFgMAQoxgEYDAZDgGIcgMFg\nMAQo/w+b9Rc12w4QKgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x25648c42630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import auc\n",
    "\n",
    "colors = ['black', 'orange', 'blue', 'green']\n",
    "linestyles = [':', '--', '-.', '-']\n",
    "for clf, label, clr, ls \\\n",
    "        in zip(all_clf,\n",
    "               clf_labels, colors, linestyles):\n",
    "\n",
    "    # assuming the label of the positive class is 1\n",
    "    y_pred = clf.fit(X_train,\n",
    "                     y_train).predict_proba(X_test)[:, 1]\n",
    "    fpr, tpr, thresholds = roc_curve(y_true=y_test,\n",
    "                                     y_score=y_pred)\n",
    "    roc_auc = auc(x=fpr, y=tpr)\n",
    "    plt.plot(fpr, tpr,\n",
    "             color=clr,\n",
    "             linestyle=ls,\n",
    "             label='%s (auc = %0.2f)' % (label, roc_auc))\n",
    "\n",
    "plt.legend(loc='lower right')\n",
    "plt.plot([0, 1], [0, 1],\n",
    "         linestyle='--',\n",
    "         color='gray',\n",
    "         linewidth=2)\n",
    "\n",
    "plt.xlim([-0.1, 1.1])\n",
    "plt.ylim([-0.1, 1.1])\n",
    "plt.grid()\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "\n",
    "# plt.tight_layout()\n",
    "# plt.savefig('./figures/roc.png', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Plot decision regions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# for visualization purposes, not necessary for pipelines with standard scalar\n",
    "sc = StandardScaler()\n",
    "X_train_std = sc.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf0AAAF4CAYAAABesSDjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt8FPW5+PHPk1ACyKWABQMqKIgtXlFBRdSIRa0W/Unb\nU4sVxapV257T9rT1tNaa1GqP2lprVawUERSqVbGK9YYHgoAggveiggpEbspNriGQ7PP7Y2aTzWZ3\nM7PZ2Z3dfd6vV15sZmZnngn77HfmO9+LqCrGGGOMKXwluQ7AGGOMMdlhhb4xxhhTJKzQN8YYY4qE\nFfrGGGNMkbBC3xhjjCkSVugbY4wxRcIK/QIjImNF5Pk03/uuiJyW6ZjCTkSeFZFLch2HMW3l9bMs\nIjtEpH/wEZmwEeunnzsishL4nqrOzsGxJwOfqOpv2riffsBKYKe7aBPwV1W9tY0hGlOQRGQV0AvY\nBzQAy4CHgPs1j7+QRWQHEI1/P6AO5/wU+L6q/j1XsZkm7XIdgCkICnRTVRWR44G5IrJEVf8vkwcR\nkVJVbcjkPo3JAQXOU9U5ItIFOB24CzgRuDynkbWBqnaJvhaRj3FuaOYk297yOTesej+kRORKEVkh\nIptE5J8iUh6z7iwReV9EtorIPSJSLSKXu+suFZF5Mdv+SUQ+FZFtIvKWiAwWkSuBi4FfiMh2EXnK\n3XaliIx0X5eIyK9E5EP3va+JSN9UIQOo6lLg38CxMTGUi8jjIvKZiHwkIj+KWddBRKaIyBYR+beI\n/FxEPolZv1JEfiEibwE73bhS7W+oG+s2EVkvIn9wl5eJyEPu33OriLwqIl9y182J+fuJiPxaRFaJ\nyAYReVBEurrr+olIRETGichq9/i/8vt/awxN+bJDVZ8Bvg1cKiKDAUSkvYj8wf2crReRe0WkrPHN\nIheIyBvu53yFiJzlLo/9LA9wvxs+dz+rf495f0REDnVfdxWRqe42K0Xk+pjtLhWReSJyu5ujH4nI\nOR7PT5otELlJRB4Rkekisg242M236PfMZ+66bjHvOUVEFro5+7qInOr7L22asUI/hNyC9xbgm0A5\nUAM84q7bH3gMuA7oCXwAnBy3C3W3PQsYAQxU1W7AfwCbVXUiMA24TVW7quoFCcL4b5wvonPc914O\n7E4VtnvMk4AjgA/d3wWYCbzhnsuZwH+JyCj3fZXAwUB/YBTwXZqqCKMuAr4GfNFdl2p/fwbudGMe\nAPzDXX4p0BXoC/QArgZqE5zHeGAczt3XoUAX4O64bU4BDgO+CvxGRA5P8XcxplWq+hqwBogWarcC\nA4Gj3X/7Ar8BEJFhwBTgv93P+WnAqgS7vQl4QVW/CBwI/CX2kDGv78b5nPcHKoBxIjI+Zv0w4D2c\n75vbgUnpnSUA/w942I37UeCnOLk9wo1xZzROETkIeAq4QVW7A/8DzBCR7m04ftGzQj+cxgKTVPUt\nVd0H/BI4SUQOxkmQd1X1KVWNqOpdwKdJ9rMPJ5kHi4io6geqmmzbeN8DrlfVDwFU9R1V3ZpkWwE2\nishuYAFwr6o+5a4bCuyvqjeraoOqrgL+hlOQA3wLuFlVt6vqOpxqznh/VtV1qlrnYX/7gIEi0lNV\nd6vq4pjlPYFB6nhDVXfGHwjnb3+Hqq5W1d04f/uLRCSaKwpUqupeVX0beAs4Jtkf0Rgf1uFckAJc\nCfxEVbep6i7gf4HvuOsux/l+mA2gqutVdXmC/e0D+olIX/fz+krMuuhFegnOxf3/uPmyGvgjENsY\ncLWqPuC2N5gCHCAivdI8x/mq+qwbdx3wfeBXqrpBVffiXKh8y932EuApVX3J3f5FnHzzUtNgkrBC\nP5z6AKujv7hJvwXnar8P8Enc9msS7cR9nnY3cA/wqYjcJyKdPcZwEPCxx20Vp0DdD6eGoEJEou1F\n+gF93arBLSKyFacgjX5p9ImLP/7ciFvf2v4uBw4H3ner8M9zlz8EvAA8IiJrRORWESlNcKxmf3v3\ndTugd8yy2Aun3YDXv6kxqfQFtriPnToBS6Ofc+A5nBwDJzc/8rC/n+N8xy8WkXfi7t6j9sf5fNfE\nLFvtxhK1IfpCVWtxLhjS/czH5/fBwMyY83wbiLgXFf2AsXG5fiJOjpo0WaEfTutwPvAAiMh+OAm/\nFliPk/SxDky2I1W9W1VPAAbjFIY/j65qJYZPcKrHvRL3DvpOnFa718bs52NV7eH+dFfVbqo62l2/\nLi7+gxOdRlxcSfenqh+p6lhV/RJwG/C4iHRU1XpVvUlVjwCGA1/HqcaP1+xv777eR/LaFGPaTESG\n4hRm83B6wOwGjoj5nH/RrRIHj7mpqp+p6lWq2hfncda90ef4MTbh1gjELOuH810ThPjvnU+AUXH5\nvJ+qfuaueyBuXRdV/WNAsRUFK/Rzr73byCz6Uwr8HRgvIke7jXduARapag3wL+BIETlfREpF5Ic0\nvwttJCIniMgw9667FtgDRNzVn+I8s07mb8BNIjLQ3ddRKZ6lSdzv/wtcJyLtgcXADnEa43VwYz5C\nRE5wt30M+KWIfFGchoI/SBETre1PRC522z0AbMP5komISIWIHOlWZ+6kqbtUvL8DPxGR/m6tyM3A\nI6oa/bvFn6sxaRORLiLydZzP3UOqusytRp8I3ClNjU37um10wHmmPl5EznAbwvURkUEJ9v1NaWp8\n+zlO7kdit3E/1/8AbhaRzuJ0wf0JTs1YNvwV+L37/B4R6SUi0RuCh4ALReSr4jTg7eDm8QFZiq0g\nWaGfe//Cuaqvdf+90e3qdgMwA+eK+xDcZ9aquhnnmdftOFfpXwaW4Nxdx+uK8+WxBacv/Sb3feB8\ncRzhVpvNcJfFXoXfgfNl8KI4LW3/BnRMcg7Nrt5V9V/uMa90v1S+jtOafyXwmRtTV3fz37rnuBJ4\nEeciIPZc4vfd2v7OAf4tItuBPwHfdp8dHgA8jnMh8G9gDvBwgmM8gPNl8zJOFepu4D+TxZPgd2O8\nmOnmVQ3O46k/0Ly73nU4jWEXicjnOLkxCBob/Y0H7sT5PFfTdKce+3kcCrzq5sI/gf9028DEb/ef\nOJ/zj3E+9w+r6uQUsXv5zHvNiz/iPLr4P/fvMR84AcBtX3AhznfhRpzGij/Fyq02scF58pzbOn4N\nMFZV5+Y6nrYSkatxCuozch2LMcYUGrtiykPi9NPv5lb9R/vULsplTOkSkQNEZLhbTXk4TkPAGa29\nzxhjjH82Il9+OhmYDnwBZwjPC9wq7HzUHue5Xn+c545/BybkMiBjjClUVr1vjDHGFIlQ3+mLiF2R\nGBMCquqp14LlrDHhkShvQ/9MX1V9/dx4442+3xP0j8WUf/FYTE0/lrPh+LGY8i+eXMaUTOgLfWOM\nMcZkhhX6xhhjTJEouEK/oqIi1yG0YDG1LmzxgMWULWE8J4vJm7DFFLZ4IHwxhbr1vjMxXHjjM6YY\niAjqoyGf5awxuZcsb0Pdet+r7TNn5joEY/JS19GjW9/IGFMwCqLQN8bkH7tYN8afTFykF9wzfWOM\nMcYklpM7fXfM+JdxhmBtBzyuqlW5iMUYY4wpFjkp9FW1TkTOUNXd7vzxC0TkOVVdnIt4jDHGmGKQ\ns+p9Vd3tvizDufiwJr/GGGNMgHJW6ItIiYi8AWwAZqnqa7mKxRhjjCkGOWu9r6oRYIiIdAX+KSKD\nVXVZ/HaVlZWNrysqKkI30IExhaa6uprq6uq03285a0z2ec3bUAzOIyI3ALtU9Y645Z4G+rCuP8ak\nx0sXoKAG57G8NcYfP132kuVtTqr3RWR/Eenmvu4IjALez0UsxhhjTLHIVfV+OTBFREpwLjweVdVn\ncxSLMcYYUxRy1WXvHeC4XBzbGGOMKVY2Ip8xxhhTJKzQN8YYY4qEFfrGGGNMkbBC3xhjjCkSVugb\nY4wxRSJnI/KZ8Nqxezcz5s9n1bp19O/ThzEjRtClU6dch2WMScJy1nhld/qmmYXLljFk/HhmT5xI\njxkzmD1xIkPGj2fhshYjJBtjQsBy1vgRimF4k7FheLNrx+7dDBk/nmm1tYyKWT4LuLhjR9588EE6\nd+yYq/BMAGwY3vxmOVtc8nYYXhNOM+bP55RIpNmXBzhjJJ8SiTBj3rxchGWMScJy1vhlhb5ptGrd\nOk6oq0u47vi6OlatX5/liIwxqVjOGr+sIZ9p1L9PH2aXlUGCL5GlZWWcWV6ekeNYoyNjMsNy1vhl\nd/qm0ZgRI1hQUsKsuOWzgAUlJYw59dQ2H8MaHRmTOZazxq+cNOQTkQOBqUBvIAJMVNW7EmxnDfmy\nbOGyZVxSVcUpkQjH19WxtKyMBSUlPHTjjZw8eHCb9m2NjsLHGvLlP8vZ4pGJhny5qt6vB36qqm+K\nSGdgqYi8qKrv5yge4zp58GDefPBBZsybx6r16zmzvJx7Tj01I4ntpdHRuLPOavNxrCrSFBPLWeNH\nrqbW3QBscF/vFJH3gL6AFfoh0Lljx4wkcrxsNDqKves5oa6O2WVl3DRpUkbueowJK8tZ41XOn+mL\nSH/gWODV3EZigta/Tx+WlJUlXLe0rIz+bWx0tGP3bi6pqmJabS1P1tVxPfBkXR3Tamu5pKqKnbW1\nbdq/McXGcrbw5LT1vlu1/zjwX6q6M9E2lZWVja8rKiqoqKjISmwm88aMGMFNkyYxC1o8H1xQUsI9\nbWx0lK2qyEJXXV1NdXV12u+3nC0clrP5w2ve5qzQF5F2OAX+Q6r6VLLtYr9ATH7r0qkTD914Ixcn\naXTU1meQ1mc5M+IL6qqqKl/vt5wtHJaz+cNr3ubyTv8BYJmq/jmHMZgsC7LRUbb6LBtTTCxnC0uu\nuuydArwMvAOo+/MrVX0+bjvrsmc8s+5F/lmXPZNLlrP+5G2XPVVdAJTm4tgm/3jtzhN0VaQxxhvL\n2fCyWfZMqMV351niYeCRnbW1jVWR/cvLGZOhqshCZHf6JtMsZ4OTiTt9K/RNaFnVX/Cs0DeZZDkb\nLJta1xQ0mzbUmPxiORt+Vuib0LLuPMbkF8vZ8LOpdU1opdudx8bxNiY3LGfDz+70TWilM22oTQNq\nTO5YzoafNeQzoeZn2tBsNiLye2cS1jsZa8hnMs1yNjjWet9lXx6FzWt3nikvvsjsiRN5MkHV4oVl\nZZx55ZUZGcfbb5ekdLowZYsV+iYIlrPByNvBeYzxw+u0odloRBQ7K1jjnUldnXNnUlXV4s7E7/bG\nFALL2fCyZ/qmYAQ9DSj475JkXZiMSc5yNvsKotDfWjOBhpppuQ7D5Fg6jYj88ntnYl2YjEnOcjb7\nCqLQf2LkObyzaxtbayZAzeJchxM6O3bvZsqLL1L14INMefFFduzeneuQAhEdx/s7HTowpF07KoAh\n7drxnQ4dMjaOt987k2zcyZjCYzlrORuUnBX6IjJJRD4Vkbfbuq+tc69g6mm38MTIc/ioYQlbayak\n/CmmC4Ni7A4jwIE41XEHur9nit87k2zcyZjCYjlrORuknLXeF5ERwE5gqqoenWQbTy2BZ87cDsDi\nte1Yf+wHKbcd3G0eZ8x6jh6lQneOh4OH+Y49XxTbONjZOl8/XZLS2T6brPV+uFjOOixnE8vr1vuq\nOl9E+mVyn8P61sPGASm3Wfzm4dwz7FS+vm0C/T5YQv+aFSm3L+WwvL0w8NIgJRPdYcIiW+d78uDB\nzJ8wgcqpU3lmzRoOPfBA5o8bxwE9emRke1O8LGcdlrPBKboue9ELg+mRO9hv2ApGHLow6bYHfvgO\nPZc4FwalB1+cxSgzo9AbpMTL1vnG3gWMrqtjyerVjFi40NNdg5ftTfGynG1iORuMoiv0o8aWRGDj\nABa/eXjSbT4C1g/7gIsi99Nzyb30L+2Wcp9hqxVIdxzsfJWNcb+tz68JkuVsk9bO12veWs42F/pC\nv7KysvF1RUUFFRUVGd3/sL71qTfYOIBJkdsZOuoBevRM/ucKY63AmBEjuGnSJGZBi+dlC0pKuCfP\nG6TES+d840feml1Wxk2TJiW9ovdbHZmP1bXV1dVUV1en/f6gc7aQWc46WjtfP3lbDDkL3vM214W+\n0EpDzdgvkFwZWxJh8dwr2JpimxfKIwwd9QBHb32Pw96YQFe6Onf+yWShRiDaHebiJA1S8vlqNZHo\n+Y6trOSk+npOrK/n1XbtWNSuHQ8nON90ruiLoc9vfEFdVVXl6/1hyNl8ZTmbOmfBf94WQ86C97zN\nWaEvItOBCqCniNQAN6rq5FzF05rWagSGAXx4GZMiJQwd6RT+sCnp9o0XBgHXCpw8eDBvPvhg4zjY\nZ5aXc0+ScbALyRpgB7AtxTYz5s/npIaGhFf0JzU0JLyi91sdWWzVtabtLGdT85u3lrPN5bL1/thc\nHTtI0VqBuSm2+bA8wtCRD3DCzOfoVzPB6ToYL4M1AV7Hwc530TuA6Xv2NH0h1Nczq74+4R3A8tWr\nOXHv3oT7GrZ3L8s/+aTFcr/VkcVWXWsyw3I2cc6C/7y1nG0u19X7BclLrcDiuVcw9bRTnXED9r3X\nfIPqVaFqG5Av/D6L27RzJx8l2dcrQPcdO1os91v9WmzVtcb4kc7zc795aznbnBX6ORLtOrj4zcO5\nrTzSbF1s24CEtQCxQtRbINf8Povbv3Nn/gUJr+jnA5d16ZJwX36rX4u1utaY1qTz/DydvLWcbWKF\nfo4N61tPi2I7rm1Az87J399jXnbaBqTip8tbkPw+izu4vJx64CLgdOA44HVgLlAP9OvdO+mx/Fa/\nFkt1rckP+ZqzkH7eWs46rNAPqWjbgEnlEdiYfLsWbQOyfOfvt8tbkNJ5Fncq8ATwKPAxcB4wFfgG\ngGRyBHBjwiHfcxYsb9vCCv0QS1gLEKdZ24BZz8HKJQm361HqJEImLwzCNoiF32dxGzZt4jSgM/C9\nuH2dCqzflLz3hTH5KN9zFixv28oK/TzXrG1Av8tTbjt00APOZEM1SzNS+IdxEAs/z+IKvWuOMfHy\nPWfB8ratrNAvEF5rBWInG+pRszTl9q1dGIR1EAuvz+IKvWuOMfHyPWfB8ratrNAvIvGTDaUSOwth\nskaC2RjrPkht6ZoTlnMwxo98z1lIP2/DdA65JF7nvs4Fr3Nzz5y5PQvRFJfFa9ux/tjoZEOrGtsE\nxNq5p56K62czva6hxRV3srmw4xsRLQnBPNU7a2sbqxb7l5czppWuOWE8h3R5mZ872bzcSbb1lLMA\n22fO9LSdyZx05q8P6+fdT96G9Rz88pKvUcny1gr9IrB583qmTq1kw9oVHND3MMaNq6RnT2/PvaZH\nStivd/JagQ1vvcErP7maUxsiDN3b0OyKOz6Z0vnCCZtCOIdYVuiH0/rNm6mcOpWVa9dySN++VI4b\nR3nPnhnZd2wBGH+XbDkbbpko9K16v8A9++xEptz3MyqAi4EFy5dy7ZxHuPTqP3DuuVe2+v7oFMRJ\n9RlAhwfOpfTDn/DeB+9zcqSeu44fROeyN2ioeaPZpo+/8hGnNOwNVSMiv8LYEMoUlonPPssN991H\nBTAaWLB8OUPmzOGmq6/mynPPbfP+/TScK4TPeyGcQyYlLfRFJHVT8Cb1qjo1Q/GYDNq8eT1T7vsZ\nT9GywcsF9/2Mk08+n+7dkw9A49Wl+3WCY/7K9KNKaDfoAR5Lst1zb2yiYm9DwnVhnr0qVlgbQpnC\nsH7zZm64774kOXsf5598Mr27d2/zcbw2nCuEz3shnEMmpbrTvx+Y52EfQ3HGRTAhM3VqJRWQ8Aq3\nApgypZIf/3hCxo43tiQCH16WdP3KjsKCskVQt6vFuqXtSzmjrAZqFod6aGHrLmSCVDl1asqcrZwy\nhQk//nHW4imEz3shnEMmlaRYV6uqZ7T2A+xL58Aico6IvC8iy0XkuvTCN6lsWLuCU5KsGw6sX5O6\nBX+mjRgxhleIMCtu+SxgdmkJh4w/mo8altBQMy2rcfkxZsQIFpSUJDyHBSUljLHuQqYNVq5dmzJn\nP16zJpvhMGbECF6GhJ/3lyEvPu+Ws82lutM/zuM+hvo9qIiUAHcDZwLrgNdE5ClVfd/vvkxyB/Q9\njAXLE/fFfwUoP/Cw7AYERBS+DZwCnAi8CiwA6uvbMWXT7xg+6h+Nkw11pWvKfeVivoFCn4HL5NYh\nffuyYPnyhOteAQ498MDsBgREVBPmbJgbgceynG0uaaGvqp5uA1X1wzSOOwxYoaqrAUTkEeACwAr9\nDBo3rpJr5zyScBCLauC+SyuzGs/8+TM4SiMsB9YCLwHbgVLgWCKUL3icw/ZrmmyoR8/k16QHfvhO\n44VBtgv/aEOoGyZP5jfPP8/lI0fy5mWXFd2Xh8m8ynHjGDJnTtKcfevSS7Maz4z58zlKlfdombNf\nUc2bRnCWs01SNeT7rZcdqOpv0jhuX+CTmN/XQKsDyhmfevYs55Krbuf8+39OBc6V+gKcL49xV92e\nkUZ8fqxZ/T7/3lfHY7T8QvvWvjp6f+Jc80UnG9qaYl8vlEdaTjaUSobbCXQqK+Ol15ejXMtLr7/A\nH75fltH9m+JU3rMnv73qKs6///4WOXvTVVdlpBGfH8tXr+btffuS5Ow+jv3kkyTvDB/LWUeq6v2D\nYl53wJnA6DVgNXAwTiH9RHChOSorKxtfV1RUUFFREfQhC0r37r2ItD+a5/ceywssRxlE+/Zv0KNH\ndgt8gG07tzCcxI2UhgPbdjQV88P61jfbJhKJMHv2dEaOHEtJSQnDiJtsaN97yQ9cvSpj8w1EPb1w\nIZu3dwPuZtO245i5aBEXDB+e8j2RSITps2czduRISkpSNafJrerqaqqrq9N+v+Vs2/Tq3h1pfwTP\n7z2+MWc7tl9C7x49sh7Lpp07U+bs5h07kr43bJ/3Qs5Z8J63qar3x0dfu9Xv31HVJ2KWjQG+lWZ8\na3EuHKIOdJe1EPsFYvyJRCJMnnwre/feDpxL9Anc3r3/YvLk6zjppNFZ/SB369yDE5KsGw683iX5\nXczChU9z110/oFOnLgwffgEQN9lQeST5gfslmGwolVYuDCKRCDdMfoxde/4CCLv2/I4bJv8Xo086\nKeXf8+mFC/nBXXfRpVOnVr9scim+oK6qqvL1fsvZ9EU/W7V7/0Jsztbu/Zenz1im7d+5M0cmWTcc\n+KxLl6TvDdPnvdBzFrznrdfBeb6GM7ZLrKeByWnEBk6NwUAR6QesBy4CvpPmvkwSixY9w6ZNNcDb\nwDsxa5SNG1ezaNEzDB9+ftbiObDfl1nYvgPs3dNi3aL2HTjkoC8nfF/04gWuZfLkW1tcrPiebKg8\nxdSbHmoFmu4YvuYuOZdN236d8s4h+qUD13LD5Mey/uUdRutGD8x1CKHz4oxZrNn0GYly9pNNnzK1\nYTVnXRB/3x2cXp8OZdGsF2FPy+5ur3Yo48TRJyT8f4xEIvzqP38OXMsvH32S42/+bk4/7y88/gKb\ndncnNmc37voNU+tXcdY3ErdJiD2HXz36z5yfA9BK02ZvvBb6HwI/AO6KWXYN8FE6B1XVBhH5IfAi\nTrfBSaqaon7WpKNXr4MYPfp7wIYEa79Hr14HJVgenBEjxjBt0q8SNlJ6pbQd404dk/B9Cxc+zdat\n7YG72bJlCIsWzWy82/cqdrIhVibfbr9hK5pPNkTzHg6RiHLDxIfZted+IDrCZet3DulULRa6FTP7\n5jqE0GlYfUzKnG1YfUxW/26DulzOn+Q2ZlHXckY7+QKXdLmcFTM7t3jfggX/5LO1HYG7+WzNEB66\n/nXfOZspkUiEW34wkd07byc2Z3fv/C23/OA6+rW7NGHOLljwTzZv6ALczab1Q3N6DlFf9j4Kb1Je\nC/0rgCdF5Bc41fB9gXog8be0B6r6PHB4uu83rRs4cAgDBw7JdRiNOnXqwnU3Ps63q77J0NpdnIoy\nD+G1jvtx3Y2P07Fjyy+P6F3+vn1Owu7bd3ObHk2MLUnxGACcxwVr7+SZYe5kQ52b1wr830sfs+bz\nz0l4J7ZxA88sWsT5cYV5ulWLpvhYzmZeOjWe0XPYs8c5hz17qnJ6DpnkKXpVfQM4DKcK/g5gLHCY\nqr4eYGx5KxKJ8NJLDxOJtFLAFKHBg0/msqv/yOx2B/Mbvsbsdgcx/po7GDz45ITbN93ln43zNOkc\ntmxpx6JFqSdracv/wbC+9VywcQCTNt7ObSub/8zSX9P7uJFI6d3AJmAusIl2JXdy1vFHc1CvXi32\nl/hxQBdmLlrkOzYTDMvZ5PIhZ1Pp1esgjjuugtK4nC0t/QvHH1+RsMZz4cKn2b69E7E5u21bh1bP\nIR+kdcmiqi8D7UVkvwzHUxCijc4K4QOSaZFIhGnT/kx9/ViU56ivv5hp0/6cMNGb7hhuBp4ELgf+\n6d453JryyyET/wdjSyItfq4deBQNNR+hDffjDFXyHHAS9ZFJ/PvjDzm43YtsrZnQ+LN51b1cP/EB\ndu35HS0fBzxmhUxIWM4ml085m8ihhx5DTc3HNMTlbEPD/dTUrOTQQ49JeA579lQRm7PO3X7qc8gH\nngp9ETkKWA5MBCa5i08HHggorrwV3+jMywekmO4yFi58mm3bOgIzgWuBZ/j887KEib5o0TNs3Lga\neBP4qbv9T4C3+OyzVSxa9EzCY6Tzf+BVU1Vhy5jWbK3lyl1nctPxNzf+XLnrTNZ+XotTtXhrzM87\njY8DTG5ZzqZWyDkbrd5PvH3LnE20fb7x+kx/AvAbVX1IRKKdqefiXASYGE3VQnezbdtQT43OEnVH\nK0TRxK6ruwBnbK+7gWHU1Y1K2Cp///0PpEOH/aitnQl0d7efB8ykY8fO7L9/4iFJM9HwL5lo48h1\n6xbw+uvdaWi4m9LSeRx33AL69Pkep5cNZWDMVMQflm2ns9sw6z0Vyru/S++PV9G1HUA5X4y8zNaa\nt1ocJxcjDRYry9nkiiFn46v3w9YAOtO8FvpHAA+7rxVAVXeJSPGNYZhCOo0/WuuOVkia7gIeBu7D\nqTqrBK7hs8+2tGhQs2nTGvbs2YXTdjS6/e+Ba6it3cmmTWsYNKj5FBFBNyIaOHAIhx56DFdddQoN\nDc4xGhp+T03NdVx//bQWx4hvmLV4bTvWX/1BY//rf7o/sQZ3m9d8pMEQzzqY7yxnU7OcLTxe/6Kr\noPmIJiLfPDpiAAAgAElEQVQyDKcrn3Gl0/ij+V1GYTQUSaZXr4M44YTTKS1t3l+2tPSLnHDC6Qmv\nuP1sD7F3DE3be2lE5EdbGvlEGwmm+tk69womHP4YT4w8h48aljRrI7C1ZkKoZyHMN5azqVnOFh6v\nhf4NwL9EpAqnAd8vgceAXwcWWZ5Jp/FH/HsKpaFIMoceegyrV6+ioeH3xP6NnKvu1S0a1Pjdvnkj\noqbtvTQi8iobjXyG9a1nbEmErXOv4J5htzRrI3DT8TezdEh3K/wzwHK2dZazhcdT9b6qPiMi5wBX\n4jzL7weMUdXE87YWoXT6gia++rwxo8+zwsTv3yid7Z2qyJbbRxsRtXUEwmyOchgdUCjepMjtzSYb\nCuMUxPnAcrZ1lrOFx1OhLyIHA2+r6rVxyw9U1TWBRJZn/Db+iH+W6CisQSDi+f0b9ep1EOedN56Z\nM3+H8yV7KPAx8BznnXd5i+2bGhEtdbeN+jhlI6IgzyEI0VkIo5MNpZqC+NAHZ1rbgCQsZ1tnOVt4\nvDbkWwXMFZFvqOqWmOXLyMxwwHnPb+OPYrz6jP0bxc+al2z7uXMfB/oD/8D5olXgSEpLafH33rRp\nDXV1u3Gan0jMmu7s2fN8wkZEbTmHXIqdbCjlFMSHj2s22VDLWoEMjOuZpyxnW2c5W3i8Fvq7gVeA\nJSLy/1T1bXe5pHiPSaHYrz69dHmqr69n5sxpOC2Hm+6s4DaefvoSxo2rpF27po9w9G+6du0C3njj\nDRoaxlJaOo3jjjsuYdecQhA/BXGL9TRNNpSoVuCy4EIrOJazlrOFQFS19Y1EtqtqVxH5NvAX4BpV\nfSK6PLDgRNRLfDNnbg8qBBOASCTCVVedwmefVdC791z++tf5Ce8cJk++gSef/BtOO9LY60sFfsuF\nF17J+PE3Jdm3M50w/Iveva9Leoxisnhty2v8m67u1Or7RARV9XSB7zVnwfI2n1jOhsPo0d6L22R5\n6+svqqqP4gyo/EcRuYk07vRF5Jsi8q6INIhI2+ptQsrvaF1hHN0ryJiaRvg6PunIXgD9+x/BgAFH\nMGDADAYMeCLmZwYDBhxJ//5HJNy3dc1JbFjf+hY/xmE5m5rlbOHwWr3fEH2hqm+4ffQfB1q/TWjp\nHeBC4K9pvDcv+B2tK4yjewUVU/MRvr5HXd0vkw5wcsYZF3HGGRf53ncxNbQymWE5m5zlbGHxOste\n97jfPwPOoHlzS09U9QNVXUGBtgfwO4Z0kGNOpyvIscj9jOPtV6GPmW2CYTlrOVtMkt7pi8hp7mx6\niMjIFPtYnfGo8pjfcbzTGfc7aEGNRe53HG+/ir2hlUmP5azlbDFJVb1/L3Ck+3pSkm2UBHf7IjIL\n6B27yN32elX1dXlYWVnZ+LqiooKKigo/b88qv+N4pzPud9CCHIvc7zjeflnXnMyorq6muro67fdb\nzmaX5awB73mbtNBX1SNjXh/i5+CqOsrP9qnEfoGEnd/RutId3ctLf9lsnUPz96S+y4iOy/3662to\naGg+Lvdxxx1lV/UhEV9QV1VV+Xq/5WxiQeWt5awB73mb6xYSBfNc3+/4zm0ZDzpaLZfp1q1Bj0Xu\nd1xuY4KUzZyFYPLWctb4leqZ/idAqx1uVfVgPwcUkf+H09d/f+AZEXlTVb/WyttCL+gxqqOCnNYz\n6LHIi3FEMxNe2cpZCC5vLWeNX6me6X835vVQ4FLgLpyGe/2AHwJT/R5QVRNNIZ730hmjOp0GLEE2\nIgp6LHJrtGPCJFs5C8HlreWs8cvriHzvAmer6tqYZQcCz8c++894cDYiXzNhG7nqlVee5rbbriES\nuZ740bdKSm7mF7+YkLG7gCDbMRQzLyN82Yh8bROmvLWczW+ZGJHP6+A8fYCdcct2An09R2DaLGzT\nembzLiCMg6EY40WY8tZy1ni9038QOAT4HbAGOAj4JVCjqpcGFpzd6TdqebcQVfjjVHsd99v4Z3f6\nwSrWvLWcDUY2x96/GliI01HzdWAC8Kq73GRBtkeuCtPY4s2fh9qY3CZ/ZDNvLWeNF61W74tIKTAW\nqFTV/wk+JJNIthvUhKVqLoyDoRjjVTFWp1vOhlurhb6qNojIHar6QDYCMollc+SqILsF+hWm56HG\n+JWtvLWcNV55/VTMFJHRgUZiQiMsVXNtHQzFmGJhOWu88tp6vwPwuIgsBJoN2qOq44IIzORGmKrm\nbGAQY1pnOWv88Frov+v+mAD47c8atrH3g2IDg5iwSicHwzT2flAsZ8PPU6Gvqv5m3DC++G2AE1SD\nHb+jdQXNZuAyYZVODgaRt5azxi/PnwYRaS8iR4nIGSIyMvoTZHDFIL4BTmvPvPxu70e2uwUak4/S\nycGg8tZy1vjl6U5fREYAjwFlQFdgO9AF5/n+oYFFVwT8jskdprH3jSlG6eRgWMbeN8briHyvAdNV\n9U8islVVu4vIb4DdqvoHXwcUuQ0YDdQBHwHjVTXh0FyFPiKf3zG5wzSGtykMNiKfP+nkoOWtyZRs\njsg3CPhz3LL/BX7iOYImLwJHqOqxwAqc4XyLUuIGOMm72/jd3hiTWenkoOWtCROvhf42nGp9gPUi\nMhjoDnT2e0BVfUlVow+0FgEH+t1HIfDbn9X6vxqTW+nkoOWtCRuvXfZm4NRLTQceAOYA+4DH23j8\ny4FH2riPvOS3P6v1fzUmt9LJQctbEzZeu+z9OOb1H0TkVZy7/BcSbS8is4DesYtwBvS5XlVnuttc\nD+xT1empjl1ZWdn4uqKigoqKCi8hh57fBjjWYMdkS3V1NdXV1Wm/33K2be8xJh1e89ZTQ75ME5HL\ngCuBkapal2K7gm7IZ0yuWUM+Y/JHJhryJb3TF5F5xAy3m4yqnuY5Cme/5wA/B05LVeAbY4wxJrNS\nVe//Leb1AJzn71OA1cDBwKU4z/f9+gvQHpglIgCLVPXaNPZjjDHGGB+SFvqqOiX6WkQWAWer6r9j\nlkUb9d3o54CqelgacRpjjDGmjbx22fsKzkA6sVYCX85sOMYYY4wJitdCfy7woIgcJiIdRWQQMAmY\nF1xoxhhjjMkkr4X+Ze6//wZ24kyzK8D4AGIyxhhjTAC89tPfAlwkIiXAl4CNMaPqGWOMMSYPeB2R\nDxHpBhyOO/Su2/IeVZ0dSGTGGGOMySivU+teBtyDU7W/O2aVYlPrGmOMMXnB653+zcA3VfW5IIMx\nxhhjTHC8NuRrhzMlrjHGGGPylNdC/1bg125DPmOMMcbkIa/V+z8BDgB+ISKbY1eo6sEZj8oYY4wx\nGee10P9uoFEYY4wxJnBe++nPDToQY4wxxgRLvM59LSLHAqcC++OMxgeAqv7G1wFFfgtcAESAT4HL\nVHVDkm09z81tjAlGsnm5k2xrOWtMCCTLW08N80TkKmABMBK4DjgK+G9gYBqx3Kaqx6jqEOBf+Jyl\nzxhjjDHp8doa/xfAOap6IVDr/vtNYJ/fA6rqzphf98O548+Y6urqTO4uIyym1oUtHrCYsiWM52Qx\neRO2mMIWD4QvJq+Ffi9Vjc6oFxGREnegntHpHFREficiNcBYwNfjgdaE7Q8MFpMXYYsHLKZsCeM5\nWUzehC2msMUD4YvJa+v9NSLSX1VXAcuBC0RkE7A30cYiMgvoHbsIZ8je61V1pqr+Gqff/3XAj4DK\nZAeurGxaVVFRQUVFhceQjTHpqK6ubtMXleWsMdnnNW+9Fvq3AV8BVgG/BR4H2gP/lWhjVR3lcb/T\ngWfxWOgbY4IXX1BXVVX5er/lrDHZ5zVvPbfeb/YmkfZA+7jn817fO1BVP3Rf/wg4VVX/I8m21gzY\nmBDw03o/6FiMMd4kyltPhb6IvOG2to9fvkRVT/AThIg8DgzCacC3GrhaVdf72Ycxxhhj/PNa6O9Q\n1S5xywTYrKo9ggrOGGOMMZmT8pm+iEx1X7aPeR3VH/h3EEEZY4wxJvNaa8j3UZLXijNYz2MZj8gY\nY4wxgfBavX+2qr6QhXiMMcYYExCvg/PsFZFDAETkABGZIiKTReSAAGMzxhhjTAZ5LfTvBRrc13cA\nX8BpfX9/EEEZY4wxJvO8Vu9vV9WuItIOZ2a8fjij8a1T1f0DjtEYY4wxGeB1RL7tItIbOBJYpqo7\n3QF6vhBcaMYYY4zJJK+F/l+A13CG3v2xu+wU4P0ggjLGGGNM5nkehldEBgENqvpRzO9lqvpOgPEZ\nY4wxJkPSGnvfGGOMMfknaet9Ean2sgMR+b+MRWOMMcaYwKR6pn+iiIwHWptdy9eEO8YYY/wRkYNw\nhj3vpmlUz4rIL4FDVPWqjAfXRm09N+NP0up9907fy3/AXlU9O5NBmXATkZXA91R1tvv7RcA9wIVA\nNfCsqn49ZvuHgBWq+lsROR2YA9yrqj+M2WYeMFFV4+d4MCavicgq4ACgj6puiVn+BnAM0F9Va7IY\nTz9gJdBOVSM+33si8H9AL1XdHbfudeBvqnpvK/to9v1hsivpnb6qVmQxDpOnRORS4A/AucAGd/GJ\nInKSqi5K8rZdwCUicls2v+yMyRHFKWS/g3NxjIgcCXTE241Vm4hIqao2xC5yj9taLW4LqvqqiHwC\nfBNovEB3z+crwPQ2hmsC5nVEPmNaEJHvA7cDZ6nqqzGrbgNuSfHWz4EHgcrAgjMmXB4CLo35/VJg\nSuwGInKuiLwuIttEZLWI3Bizrp+IRESkxP29XESeEpHNIrJcRK6I2fZGEXlMRB4Skc+BS91l0UJ6\nrvvv5yKyXUROc/dzRMw+viQiu0SkZ4JzmQqMi1t2CU4N3+fu+88XkXdFZIuIzBaRw93lU4GDgZnu\nsX+W4NzmiMhvRWS+u83zItI4hbuIjBORVSKyUUR+LSIrRWRka/8BxmGFvknXtTiF9khVfSNmueIM\n2zwoRSIqcDPwDRE5LNAojQmHRUAXETncLdy+DTxM87vtncAlqtoNOA+4WkTOj1kfWyvwKFCD89jg\nW8AtIlIRs/584B+q+kVa3n2f5v7bVVW7qurLwN+B78Zs8x3gJVXdnOBcHgJOE5G+ACIiwFicC/lo\nd+7pwH8CXwKeA54RkXaqOs6N++vusf+Q4Nyix7/UfX8Z8DN334Nxaku+A5QD3YA+CWI0SVihb9L1\nVWCRqr6bYF0tTqH+u2RvVtXPgPuA3wYTnjGhE73bHwW8B6yLXamqL6vqv93X7wKPAKfH78Rt+HYy\ncJ2q7lPVt4C/0fzue6GqznT3tSdJPLEXHFNxCu6oS9x4W1DVNTi1BZe4i76KM3Dbs+7v/wE8o6qz\n3ccKf8B5lDE8ybETmayqH6lqHfAP4Fh3+TeAp1V1oarWA79pZT8mjhX6Jl3X4NzNT0qy/m9AbxH5\nepL1ALcCZ4vI0RmPzpjweRinYL2MmOfhUSJyolsV/plbLf99INHcJuXAlriGdKuBvjG/f+InMFVd\nDOwSkdPdqvgBwNMp3jKFpkL/u8AjMe0G+rjxRPetbjx98W5DzOvdQOeYfTeem6rWAolqI0wSXofh\nxf0gHEPTHx8AVX0g00GZvPApcCbwsojcq6rXxq5U1X0iUgXcBCSqDUBVt4jIne421lXHFDRVrXFb\nrn8NuDzBJtOAu4Cz3fz5E5Domfo6oIeI7Kequ9xlBwNrYw+XKpQky6MF+QbgcVXdm2IfM4B73EcK\nY2heI7EOZ56WWAcBazzE1pr1wKDoLyLSkcR/I5OEpzt9EfkV8Bbw3zgfiujPd1O9zxQ2Vd2AU/Cf\nLSJ/dBfHVts9DHTA+ZJL5k841X5fCSRIY8Llcpx2MLUJ1nUGtroF/jCaV7c3cqvXXwF+LyJlbk3Z\n90hSHZ/ARpyp0QfELZ+G0+32YhLURMTFsBt4ApgMrFLV12NW/wM4T0TOEJF2IvIzYA+w0F2/ATg0\nbpdeexI8DowWkZNE5AtYY2DfvFbv/xgYpqonquoZMT/WYrI4NV6pq+onOAX/N4Hf43yZRNdFcJ65\ndSfJ1b2q7sBp7d8j0XpjCkBsvqyMKyBj8+Ja4CYR2Qb8GqexXjLfAQ7Buat+ArhBVed4Csa54LgZ\nWOC2rh/mLl8DvO681PkedjUFp4ahWS8EVV2Oc0N4N84FxnnAaPcZPMD/Aje4x/5p9G2xu0gR+zLg\nRzh/m3XAduAzoM5DvAaPY++LyGrgsFaqe4wxxgRARA4BPlDV9gEfZxKwVlXzooGciOyH0wV4oKqu\nbm17k3rs/ZLoD3AD8Be3b2hJ3DpjjDHBOoqYxnFBEJH+ONX7yRrnhoKIfF1EOroF/h+Bt63A9y5V\nQ756mqpZos9brohZHx3VqTSAuIwxxgAi8hPg58APW9u2Dcf4Lc5j3FvyoAC9gKb2C0uAi3IYS95J\nNfZ+Py87yIMPiDHGGGPw/kz/ZzEjJ8Uu/6mq3hFIZM7+rRuXMSGgqp5aV1vOGhMeifLW6zP5ZI06\nfp1+ON6oqq+fG2+80fd7gv6xmPIvHoup6cdyNhw/FlP+xZPLmJJJOThPzNjppSJyBs37Uh4K7PD9\njWCMMcaYnGhtRL5oK84OQOzIe4ozwMKPggjKGGOMMZmXstBX1UPAmQ5RndmRQq+ioiLXIbRgMbUu\nbPGAxZQtYTwni8mbsMUUtnggfDF5asiXKyKiYY7PmGIgIqiPhnyWs8bkXrK89TThjoh8QuKhEetw\nJlGYAUzQpmEWs2r7zJm5OKwxea/r6NE5Oa7lrDHpaWvOep1l7y6csZTvwpnW8GDgB8BjwBaciXgO\nAn7RpmiMMcYYExivhf5lwChVXRddICLPAS+q6hEiMgd4CSv0jTHGmNDyWuiXAzvjlu0C+rivlwNf\n9HpQESkDXgbauzE8rqpVXt9vjDHGGP+8Ds4zE3hKRL4qIl8Wka/iTOcYfTB3MrDK60FVtQ44Q1WH\nAMcCX4tO72iMMcaYYHgt9L8PvAr8FXgDuB94DbjaXf8xzpzJnqnqbvdlGc7dvjX5NcYYYwLkqXpf\nVfcA/+P+JFq/we+B3Wl5lwIDgHtU9TW/+zDGGGOMd16f6SMihwPHAJ1jl6vqA4nfkZqqRoAhItIV\n+KeIDFbVZfHbVVZWNr6uqKgI3UAHxhSa6upqqqur036/5awx2ec1b73OsvcrnEl33gJ2x6xSVR2Z\n+F3eicgNwC6Nm7HP60Af1ufXmPR46fMbxOA8lrPGpMdrP/02Dc4D/BgYpqpv+4gtVTD7A/tUdZuI\ndARGAf+biX0bY4wxJjGvhX4t8H4Gj1sOTHGf65cAj6rqsxncvzHGGGPieC30bwD+IiKVwKexK9xn\n876o6jvAcX7fZ4wxxpj0eS30H3T/vSJmmeB0syvNZEDGGGOMCYbXQv+QQKMwxhhjTOC89tNfDY19\n63ur6vpAozLGGGNMxnkakU9Evigi04E9wIfusvNF5HdBBmeMMcaYzPE6DO99wDagH7DXXbYQ+HYQ\nQRljjDEm87w+0z8T6KOq+0REAVR1o4j0Ci40kys7du9mxvz5rFq3jv59+jBmxAi6dOqU67CMMUlY\nzhqvvN7pbwP2j10gIgcD9my/wCxctowh48cze+JEesyYweyJExkyfjwLl7UYIdkYEwKWs8YPr4X+\n34AnROQMoERETgam4FT7mwKxY/duLqmqYlptLU/W1XE98GRdHdNqa7mkqoqdtbW5DtEYE8Ny1vjl\ntdC/FXgUuAf4AvAA8BTw54DiMjkwY/58TolEGBW3fBRwSiTCjHnzchGWMSYJy1njl9cue4pTwFsh\nX8BWrVvHCXV1CdcdX1fHqvX2NMeYMLGcNX4lLfRFxNPseao6O3PhmFzq36cPs8vKIMGXyNKyMs4s\nL8/IcazRkTGZYTlr/Eo6ta6IrPTwflXVQzMbUrMYbJrOLNqxezdDxo9nWm1ts+rCWcDFHTvy5oMP\n0rljxzYdY+GyZVxSVcUpkQgn1NWxpKyMBSUlPHTjjZw8eHCb9m38s6l185vlbPFp69S6SQv9IInI\ngcBUoDcQASaq6l0JtrMvkCyLTfDj6+pYmsEEz8YXVPQ4dlfijRX6+c9ytrjka6F/AHCAqr4pIp2B\npcAFqvp+3Hb2BZIDO2trmTFvHqvWr6d/eTljTj01I4k95cUXmT1xIk8mqIq8sKyMM6+8knFnndWm\nY9hdiT9W6BcGy9ni0dZC3+vgPBmlqhuADe7rnSLyHtAXeD/lG01WdO7Ysc2JnEjQjY5iuy813pXU\n1Tl3JVVVGbsrMSZsLGeNV1677AVGRPoDxwKv5jYSE7T+ffqwpKws4bqlZWX0b2OjI+u+ZExmWc4W\nnpzc6Ue5VfuPA/+lqjsTbVNZWdn4uqKigoqKiqzEZjJvzIgR3DRpErOgxfPBBSUl3HPqqW3av3Vf\nyozq6mqqq6vTfr/lbOGwnM0fXvM2Vet9T7UAqhrxFVnT/tsBzwDPqWrC/v/2fLDwBNnoKBvPHwuN\nPdM3rbGcDZfAGvKJSARIlb2C02Wv1FMELfc/Fdikqj9NsY19gRSgoBodZaulcSGxQt94YTkbHkEW\n+v287FhVV3uKoPm+TwFeBt7BubBQ4Feq+nzcdvYFYnx15wnyrqQQWaFvgmA5G5y87LLnlX2BmHS6\n8wR1V1KIrNA3mWY5G6ysFfoicj5wOs4Uu407UtVxnnaQBvsCKW5W9Rc8K/RNJlnOBq+thb6nxnoi\nciPwV3f7bwGbgbOBzz1HaoxP1p3HmPxiORt+XvvpXw6MUtWfAHvdf0cD/YMKzBjrzmNMfrGcDT+v\n/fS/qKrvuq/3isgXVHWxiJweVGDGpDuDmI3jbUxuWM6Gn9c7/Y9E5Aj39bvANSJyCbA1mLCMcQYG\nWVBSwqy45dGBQcYkGBhk4bJlDBk/ntkTJ9JjxgxmT5zIkPHjWbhsWVZiNqaYWc6Gn6eGfCJyLrBT\nVV8WkWHAdKAz8ANVfSKw4KxRUNHz053HGhH5Zw35TKZZzgYrKxPuqOqzMa8XAwO9BmhMW5w8eDBv\nPvhgY3eeM8vLuSdJdx4vjYgyNbqX3+pIq740xcJyNtw8FfoiskVVeyRY/pmq9sp8WP401EyjlMPg\n4GG5DsUEwOsMYtlqRBTfD3l2WRk3TZqUtB+y3+2NyXeWs+Hl9Zn+F+IXiMgXgLSG4M20pUO681HD\nEhpqpuU6FJNDQc8IBs2nAn2yro7rgSfr6phWW8slVVXsrK1t0/bGFBPL2exLWeiLyDwReRnoICIv\nx/4AHwCvZCXKVkzaeDtzRn2NpUO6s7VmghX+RSqdRkR++e2HbP2Wk6hZ7PyYomY5m32tVe//DWf0\nvaHApJjlCnwKzA4oLl/GlkTgw8uYFClh6MgHOGHmc/SrmUB3jrcq/yLSpVMnHrrxRi5O0ogoEw2C\n/FZHWr/lxBYP2UTPJavoX7OC0oMvznU4JkcsZ7MvZaGvqlMARGSRqr6fnZDSN7YkwuK5VzD1tFP5\n+rYJ9PtgifOlwmHNNyyyC4FCbZCSyMmDBzN/wgQqp0zhmbVrObRvX+ZfeikH9GjRJCUtfvshp9tv\nudBN2ng7Q0c9wNFb3+OwN9wL9FQsZy1n02Q525zXLnsCXAF8B9hfVY8WkdOAA1T1H2kdWGQS8HXg\nU1U9Osk2nrr/zJy5vcWyxWvbsf7YD5zCvzym6UH1KnqUStHUAqQz+UU+C/p8/XYxCnuXpFx12Yvm\n7PRICUMHOYV/z87Jt+8xbzVd6VoUtQKWs5azqWRlwh0RuQnnkcadwH2q+kURORR4TFVbuURPus8R\nwE5gahCFftT0SMtmC0MHPcAZs54r+MI/7B/eTMvW+S5ctozvVlZyUn09J9bX82q7dixq146HKyuT\ntgT2s3025brQB+cC/cPySMrthw5yH9vtZzlrOetfseUstLGfPnAZMERVN4nIBHfZSuBQj+9vQVXn\ni0i/dN/v1diSll8mi+dewT3D4h4BFOAdRDb7wIZBts93DbAD2BbQ9sViWN96WivCo4/tBnebxxmz\nnmt6bFdghb/lrMNyNjheC/1SnLtycBrxgTMi387Em4fbsL71sHEA0yN3sN+wFVwUuZ/D3phQcNWH\nhd4gJV665+vn+Wm0O8/0PXuavqjq65lVX8/FVVUJqwr9bG8Si+bs4jcPL+gLdsvZJq2dr9e8tZxt\nzmuh/yxwh4j8BBqf8d8EBD6WZmVlZePriooKKioqMrbvsSUR2DiASZHbGToy+lzx2aYNqlfRv7Rb\n3t5RFHqDlHjpnK/fQTj83pnk451bdXU11dXVab8/yJyNvWCPNgSMz9l8fmxnOdsk1fn6ydtiyFnw\nnrdeC/2fAlNwajm+gHOH/yIwLs34PIv9AglKtNX/pPIIbGxavt+wFXl9RzFmxAhumjSJWdDiedmC\nkhLuyUAf2DDxe76xg3A0bl9X5zxPTHJFXwzdf+IL6qqqKl/vz1bORrvpWs7mr3TO12/eFkPOgve8\n9TQin6puV9ULgYOBk4ABqnqhqu5oY5zi/uTcsL71jC2JNPu5YOMAHt1zR7OBf/JpQJHGPrAdO3Jh\nWRm/Ay4sK+Pijh0z1gc2TKLnO7ZDB0a3a8fvgNHt2jG2Q4eE55vOIBx+RxDLxohjxSxRzi5deSf3\nDLvFGalz5b15NViX5WzqnAX/eWs521xrI/J1EpFbRORpEakEtqnqa6q6oa0HFpHpOCP6DRKRGhEZ\n39Z9BmFsSYTDPryMSRtv54mR5/DmriV5VfhHJ78488or+fwb3+DMK6/kzQcfzHkL1KCtAV5y/00m\nnSt6vyOIZWPEMdPcsL71XLBxQONIne/s2mY5mwe85Cz4z1vL2eZSdtkTkcnACcBzwLnAHFX9UZZi\ny0iXvUxq1vf/g0/oUdq8kqLQGgLmG7/df/76zDO8dP/9vJBgX2cDZ33/+1x53nkt1vmZOjSd7bMp\nDF32ghTtDhjbTTeW5WxupdNlL528LbachfS77J0DHKeq60XkL8DLQNYK/bBpbEG89k6eGfZBs3WD\nu82z4X9zLJ0GOAsg8fNE4KwkhZefqUPT2d5kTrQ7YLSbbqzB3eY1jghohX9upNtozm/eWs42aa3Q\n33QLwLUAAB2YSURBVE9V1wOo6ici0i0LMYVetPCPtfjNw5lw+OVNdxQ1S63wzzK/1X417u8XAacD\nxwGvA3Pd9as//TTpsbxOHZru9iazEuWs03PH5uvIpXQesaWbt5azjtYK/XYicgZNje3if0dVQzHp\nTq7F31Hkcwtiv8IyTrjf7j+bdu7kNOAfwKPAx8B5wFTgP4DNO9raTtWEXex8HYU+8E+sfM1ZsLxt\nq9YK/c+AB2J+3xz3u9KGUfkKUbEM/BPlt597kPx2/9m/c2eOxBll6ntx+xoOfNalS5DhmpAoloF/\novI5Z8Hytq1Stt5X1f6qekiKHyvwk4h2H4q2+s+3FsRexPaXfbKujuuBJ+vqmFZbyyVVVeysrc1q\nPH67Ow3q149X27dPuK/F7dsz6KCDshC1CYtoq/987qbbmnzPWbC8bStP/fRN+saWRNg69wqmnnYL\nH1z1dbaytGC+RNLp5x40P92dxowYwaLS0oRdcxaVluZ91xyTnthuunO+ZTkbNL9dFC1v28briHym\nDYb1rWf6p4fxKV/g6Ip3oTrXEWVGNsa6T4fXBjiNdxlJuuakaqkblmeiJjgD15fw8Z7TOMdyNjQ5\nC+nnreWswwr9LJq7dB9HdlL6s4LSVucVC79sjHUftHS65gR+Dlm9q/TW57dYrVu3j/XRnK0h7xv3\nFULOgv+8DeM55ErKwXlyLWwDfbTV9EgJ+/WONu5bnfeN+/wOrFEIc4UHeQ4NNdNY1bCNzSf0z0So\nrVoz8Cgu63Vtq9vl8+A8bRU/IFf/0m6Wsym2D6NCOIdYQQ/OYzIoflY/p3vQtMC/RNZv3kzl1Kms\nXLuWQ/r2pXLcOMp79mzzfv1Ws2V09qrW7oYDuiMLZAaumsVsZSmrdylLRn+N15ZfzsD1WWhu8zZw\ndfCHyWfxs/qdse89+s+znA37jHOxCuEcMskK/RwYWxJh+vLLOfKadvS//+VAjzXx2We54b77qMCp\nyF2wfDlD5szhpquv5spzz23z/v1Us2Vi9qqGmmlsZztbTu2Xcrse84LpJpnRGbjcwn5LgzJnlFvY\nzy1hbN966BvJUMQmExpz9typ9J9nOQvhnnEuViGcQyZZoZ9j29lO95rFgdyZrt+8mRvuu4+naNkH\n9oL77uP8k0+md/fubT6O10Y4bZorPMHdcCpBjbKW9jnE1Uw0sIJVDdtYffhBPNPtGsrnHm6FfR5Y\nt9VyNqrVnA2JQjiHTMrZM30ROQe4E6fb4CRVvTXBNgX1fDBWNp4Vfv9Pf2LbnDk8m2DduUD3kSOZ\n8OMfZ/SYqezYvZujL72UR+rqWnyhXVRWxjtTp7a826hZ3FhAzhn1NZZtO5XyNw93ql1TiP59o6Os\n9SiVjBT+vp8PxtzNU9G/cfHmnfBIyVXs+vQw57FPjowe3bXVbYr5mX6s+M9U/9JuGR+5ryByNmTs\nmX5zObnTF5ES4G7gTGAd8JqIPKWq7+cinlzIxsh9K9euTdo2ezjwrzWtTWKZeRHVhGNmRxIUFNGq\n/BVD+vFIyXWUzz2cC/rWQysFPgQ3yprnZ6IxFyvRu/ldKw9rtq+xJRHIYYFv/MnGyH35nrNh1Jau\nuYUoV9X7w4AVqroaQEQeAS4AiqbQj4pv3HfCzOfosvLejOz7S912siDJulfc9R95OFZ0OtK23inP\nmD+fChHuBf4HZ77mgcC7wLUiCRvURK49n5c/GUH5s4e2enefSHxDrKO3vkfPJW37+/bqCE9WncLz\nS9fz4tufcvrRvfnZ8eV0Kqvmo5XVjds5hf11lL/pXqxYAV8QgvhMRRVCzkJM25uGpguDHqXi3NTk\nYF6DaDuGJ15+mX8tWsR5J53EPaedVnQFPuSu0O8LfBLz+xoogI7rbRCd+GPC4amfVftRcs16Xnpt\nUMJxrecAZ1/7Mrd1793qfvbrvSIjdzWr1q1j/7o6jgFG4FRXLgWOAS4IuEHN2JIIfHgZkyIlkLoN\noGefbPonCx69hMjoO/lw0AUtN9gDY/dGPNVMmPwTxGcqsJxNs6D1nbMp2t5k6nskXZ07dqRb5868\nsHQpF48aVZQFPuRBQ77KysrG1xUVFVRUVOQslqBFZ+rLmP17UztqHOfPmkoFcArOfNPVwKmjxvG9\nnl8CPNx9unc146+Z2qbeBgfsvz/3AzNo+YU2BqiMdkmK/eJ4fy8rlvfnovK9vPTSdEaOHEtJSfrd\n2TL1/DwSiXDVg78HruXDKb/n58PPazWuSCTC7NltP4egVVdXU11dnfb7iylnIXOfKSCQnD1q+QZ2\nV6dXFe83Z2N7ogyYA70+mNr0eY+rHTnsjQkpj53p3jeRSIQbJj8GXMsNkx9j9EknecrZ6bNnM3bk\nyFDnLHjP21wV+muBg2N+P9Bd1kLsF4jxJxKJ8NZbS9jDQzzPS7zAByiHA2fy9tu3EolEsv5BHg4J\n+8sOB/i8pnGAmsZGe26r9gULnuauu35Ap05dGD48wV11li1c+DRbt7YH7mbLliEsWjSz1bgWLgzX\nOSQTX1BXVVX5er/lbPryLmdFGnM2vifKgpp/tvi8x9aO7Hf8ipTHzXQ7p6cXLuTTrZ2Bu9mw5Rhm\nLlrEBcOHt/qeH9x1F106dWp121zzmre5KvRfAwaKSD9gPXAR8J0cxVKwFi16hk2banCennwF5Svu\nmrVs3LiaRYueYfjw8z3v790P6+mwaxv9alJfoSez6uP3OS3JulOBZVs+YO7ArzrPwWMa7UUiESZP\nvhW4lsmTb+Wkk0bn9Ko7Gs++fbcDwr59NzN58nUp4wrbOZhwynTO/mPpyZRE7ucwVkPNUt/xtJqz\nH73I0nFn80jJdU5PFPdxVmuf92hbplRi2zml+50TFYko109cSt2+BwGhbt/vuX7ieEb0eZOSksQd\nU6LvgWu5fuKklNtmU9c2Dp2dk0JfVRtE5IfAizR12XsvF7EUsl69DuL/t3fv4XLU9R3H35+AmEAo\nt5RgEiCFCBbQQihUC8JBqiB9YrQF8VJusajQgk+tiIKVBFFuYkF5VLCChEhRoSKoBbkdLpH4KAEh\niNxELoHcIJCkkBiy3/4xs2TOZc+Z3ezuzNn9vJ7nPGd35/adOee7v5nf/H6/mTbtY8CiQaZ+jG23\nzf8Iymqbg9kHNP4Eq0cW/pg/3v1VWD3w8Z33jB7Din2O4gerTxlwH/yee65nxYpNgYt5+eV9cl1V\nt9L6q/z3pp8cxosvnj5kXGXbByunZudstZHwcFfVteTJ2e8uPWVAT5Rm/L834zun6snbbuH5Vc+T\nzdnnVr6Jf151MH9x0ME1l1n06lLgYp5/Zd6Q87bTf23g8h5739rmlVdWcsJxb+EHr64acH/wyDFj\n+fb3HmHMmLF9lqlUKnz84/uxZMn5JM2Ifsb48adyySV3F3KlPDCeqtpxlW0fstxP34binC3HPmTl\nyVmonbeuX2yBSqXCLbfMoVJxF62sTTfdnFPPuIYjx4zlEMRZwCGII8eM5dQzrhnw5QHZK4ZDgMuB\nQ3n55dHMm3fDkNtq1d9g3ryfsnTpUyQD158NHJH+fpAlS/7IvHk/HWIf1l9l5NkHax/n7OCcs52X\nsy70W6DaYKsT/kGabbfd3sGxn7yAWzfahi8Ct260Dced8DV22+0dA+at3hdcvXoW8GNgBnAdq1fP\n4vLLzx3yy6FVf4Nx4yYxevRmJB2XfgVck/6+lzFjxjJu3KQh9qF60q1c+2Dt45ytzTkLnZSzLvSb\nrH8Dljz/IN10lVGpVJgz50LWrduC4ETWrduCOXMuHHTf1zdquh/4NHAi8G/Ab19v1FRrG/X+DfJa\ntuxZ1qx5BdiLZGyyE9Pfe7F69f+xbFnfEdPW78MDwLmZnweH3AdrH+fs0JyznZWzpe+nP9I00oBl\npHTlaoZ77rmel15aA2xDMhLzvixf/uKgx2nbbbdn6tQe5s+/gEplUjr/XYwa9VX23runZqOmRrrT\n5VVtaPXcc3OZP38r1q27mI02uoupU+cyYcLAhlbNbJhlreGcHZpzNmvk56wb8jVRI40/1i/Tw/jx\nd5SioUirVCoVjj/+b1m69E/A16keI/gU2277Ri69dG6ffa93/uoyrW6AU+ZGPvXq9oZ8ztmhOWfL\nxw35SqSRxh99rzI6o6FILUm12ZPAWLLHCDZj6dI/DKg2q3d+qNWdbuOmHtdObuTTbZyzQ3POdh4X\n+k3SSOOP/st0SkORWsaNm8jo0VsDXyF7jOArjBmzNePGTdyg+dcPmvPlPvMng+c057h2eiOfbuKc\nHZ5ztvP4nn6T9G388WBmStQcSWvws88zOnbglmXLFrJmzQoGO0arV69g2bKF7LLL3g3P37drTt/5\nq11z6hnNbDCN/J2tnJyzw3POdh4X+k1Sb+OP9WefyVCuierZ59BDuo5U9R6j6vwLFlzLE088AxwN\nXMHOO+/AHnsMnL/aNefVV+8FdspM+cOgXXPasQ9WXs7Z4TlnO48L/SaZMmUvpkzZK/f83Xj2We8x\nmjJlLyZPfiuHHz4FmENyb/BAnnzyKM4//yY23rjvv+/6rjl7s/5LGWArVq++kWXLnmWXXaa2dR+s\nvJyzw3POdh4X+gXp9rPPvI+ZveKKM6hUJpCtTq1U3sTs2TOZMeOsPvNWj+nChXO57777WLfuI2y0\n0feZOnXqoF1zzOrhnHXOdgJ32bNCzJ17Heeeewyf+9zsmvdCX3vtNQ4/fAqVSvWKoepnjBp1FNdc\n8/iAK4dO6prTDt3eZc/yc86Ww4jrsifpcEkLJK2TtGH1NiVV72hdZRzdq5UxVSoVLrvsHOAgLrvs\nnJrbuPLKWVQqaxhsZKxKZTVXXjnwedHummONcM4Ov27nbGcoonr/QeADwCUFbLst6h2tq4yje7Uy\npvUjfN3O8uU71Wz5PHny7uy88+7A/wyylj2YPHn3Pp90Y0Mraw7n7PDrds52hsKq9yXdDvx7RMwf\nYp4RV1VY72hdZRzdq5Ux9R2x6xDgppojddXrl7+8nvPOO4FK5XT6NgoKRo36Mp/97Lc6rqHVhnL1\nvnM2z7qds+Ux4qr3O129o3WVcXSvRmLKW7W4/opha5Jxubdi+fLVTdnvaqOg6dMXMX3685mfRUyb\n5kZBNjjnrHO2m7Skel/SzcD47EdAAKdHRF3/KTNnznz9dU9PDz09PU2IsDX6V1UNV0VV7/zt0GhM\neaoWq/cF164VMJPk32Ima9d+issuO2eD99tdc5qjt7eX3t7ehpd3zraXc9Ygf962pNCPiHc3a13Z\nL5Cyq3e0rkZH98rbdaYd+1CNJ/tYzFpfBOvH5d6VvuNyn8bSpY92ZD/nkah/QT1r1sDGV0Nxzg6u\nVXnrnDXIn7dFV+/nuk84EtQ7vvOGjAddPUNvdrViozHlrVqsd1xus1ZqZ85Ca/LWOWv1anvrfUnv\nB74BjAN+Kun+iHjvMIuVXr2jdTU6ulfeM/R27EM2njxVi/WOy23WSu3KWWhd3jpnrV5tL/Qj4jrg\nunZvt9UaHaO63tG9+p6h79PUB300ElM9VYvdPqKZlUu7chZal7fOWauXR+QbQco2ctXAeKo8otZI\n4S57rVemvHXOjnzustdFyjZyVd+qxb6jb1WrFpuljCOgmeVRprx1zpofuDNClHHkqnZW/ZVxBDSz\n4ZQtb52z5kJ/hGj3Yz3zdC9qVx/bVjZeNGulduatc9bycKE/QrS7QU2ZztJb2XjRrJW69craOVte\nbshnA5RpbPEyNYLqRG7I1xmcs93DDfms6co0tniZGkGZlZVz1vJyoW999B/hK+9oY+2IJVFsTGZl\n45y1erjQL4F6u7a0sitMmc7S29m9yKwejeRgq/LWOWv1cEO+Eqi3AU6rGux0c/cis3o0koOtyFvn\nrNXLhX7B6u3aUrax91vJj920MmokB8s09n4rOWfLz4V+wert2lK2sffNuk0jOVimsfetu7W9y56k\n84BpwBrgCeC4iBi0/06nd/+pt2uLu8JYs7nLXn0ayUHnrTXTSOyy9wtg94jYE3gM+HwBMZRCvQ1w\nytRgx6wbNZKDzlsrk7YX+hFxS0RUm6/OAya1O4YyqLdri7vCmBWrkRx03lrZFH1PfwZwdcExFKLe\nBjhla7Bj1m0ayUHnrZVNSwp9STcD47MfAQGcHhE3pPOcDqyNiKuGWtfMmTNff93T00NPT0+zwy1E\nvQ1w3GDH2qW3t5fe3t6Gl3fObtgyZo3Im7eFjL0v6VjgeOBdEbFmiPk6ulGQWdHckM9sZNnQhnxt\nr96XdChwCnDAUAW+mZmZNVcRrfe/AYwFbpY0X9I3C4jBzMys67T9Sj8i3tzubZqZmZkfuGNmZtY1\nXOibmZl1CRf6ZmZmXcKFvpmZWZdwoW9mZtYlXOibmZl1CRf6ZmZmXcKFvpmZWZdwoW9mZtYlXOib\nmZl1CRf6ZmZmXcKFvpmZWZdwoW9mZtYlFBHt3aB0JjAdqACLgWMjYlGNeaPd8ZlZX5KICOWc1zlr\nVgK18raIQn9sRKxKX58E7BYRJ9SY118gZgVzoW828tTK27ZX71cL/NRmJFf8TdPb29vM1TWFYxpe\n2eIBx9QuZdwnx5RP2WIqWzxQvpgKuacv6SxJTwMfAb7YzHWX7QCDY8qjbPGAY2qXMu6TY8qnbDGV\nLR4oX0wbt2Klkm4Gxmc/AgI4PSJuiIgvAF+QdCpwEjCz1rpmzlw/qaenh56enhZEbGZVvb29G/RF\n5Zw1a7+8eduSQj8i3p1z1quAn5Oz0Dez1utfUM+aNauu5Z2zZu2XN2+LaMg3JSIeT1+fBLwzIj5Y\nY163CDIrgXoa8rU6FjPLpyyt968BdiFpwPcU8MmIeL6tQZiZmXWhthf6ZmZmVgyPyGdmZtYlXOib\nmZl1iY4s9CWdKem3ku6TdKOk7UoQ03mSHpZ0v6RrJf1ZwfEcLmmBpHWSphYcy6GSfi/p0bQbZ6Ek\nfVfSYkkPFB0LgKRJkm6T9JCkByWdXHRMzeaczR1TKfLWOTu8suZtR97Tr2eo3zbG9HfAbRFRkXQO\nEBHx+QLj2ZWkMeUlwGciYn5BcYwCHgUOBp4Dfg18KCJ+X0Q8aUz7A6uA2RHxtqLiyMSzHbBdRNwv\naSxwLzC9yGPUbM7Z3DEVnrfO2XzKmrcdeaXf6qF+GxERt0RENY55wKSC43kkIh4jGTipSPsCj0XE\nUxGxFria5IFMhYmIu4HlRcaQFRGLIuL+9PUq4GFgYrFRNZdzNp+S5K1zNoey5m1LBucpA0lnAUcD\nLwEHFRxOfzNIEsWSJHgm8/5Zki8VG4SkycCewK+KjaT5nLMjhnO2TmXK2xFb6DdzqN92xZTOczqw\nNiKuKkM8NnKkVYTXAJ/qd2U8IjhnmxeTjRxly9sRW+g3c6jfZhkuJknHAocB72p1LHniKYmFwA6Z\n95PSzyxD0sYkXxxXRsRPio6nEc7ZfEZA3jpncypj3nbkPX1JUzJv309yL6VQkg4FTgHeFxFrio6n\nnyLvD/4amCJpR0mbAB8Cri8wnipRfHuHrMuA30XERUUH0grO2YYU9f/pnM2vdHnbqa33SzfUr6TH\ngE2AF9KP5kXEiQXG837gG8A4knuo90fEewuK5VDgIpKT0O9GxDlFxJGJ5yqgB9gGWAycERGXFxjP\nfsCdwIMk1bwBnBYRNxYVU7M5Z/MpS946Z3PFVMq87chC38zMzAbqyOp9MzMzG8iFvpmZWZdwoW9m\nZtYlXOibmZl1CRf6ZmZmXcKFvpmZWZdwoW9mZtYlXOibmZl1CRf6ZmZmXcKFvpmZWZdwoW/WZpKe\nlLRBT22T9HNJR9WYtqOkiqSa+Z1O3ynnti6XtEbSHxqNt1kkHSjpmeHnzL2+PsdqqOO6Ads4Q9KV\n6etNJK2U9CdJZzZzO2Z5uNC3riVpf0lzJb0kaZmkuyTtXXRceUTEYRFx5VCzVF9Iul3SjFrTczo3\nIoY9SZB0jKS76lx3vZr9wJDX15fjuG7QNiLiTxGxOfD9FmzDbFgbFx2AWREkbQ7cAHwC+BHJ09Te\nCZTxEaqt0KpHkIrmF8pNIWmjiFhXdBxmRfKVvnWrXYCIiB9GYk1E3BIRC6ozSJoh6XeSXpD0v5J2\nyEyrSDpJ0hOSlkg6LzNtJ0m3prUHSyTNkfRnwwUkabKk5Zn335G0OPN+tqST09evX71LGiXpq5KW\nSnoc+PvMMmeRnMxcLGmFpK9nNvluSY9KelHSxfUcPEnHpvu+Iv39YUlvAb4FvCOtwn4xnfcwSfMl\nvSzpKUlnZNZTrV4/Op22RNJpmemjJX0vjXEBsE+/OE6V9Hgax4L00bPVacdIulvS1yQtA84Y6lgN\nclzvT9e7It2fiqQD0mlvT2uJlku6T9KBmXVMltSb7u9NJI/BNSuHiPCPf7ruB9gcWAp8DzgU2LLf\n9OnAoyQnB6OA04C5mekV4FZgC2AS8AgwI522M3AwSU3aNkAv8LXMsk8C76oR1x+BvdLXvwceB3ZN\n3z8FvC19fXtme58EfgdMALYEbgPWAaP6z9sv/uvT47A9sAR4T42YLgfOzLzfFHgZmJK+Hw/8Zfr6\nGODOfssfAOyevt4DeB54X/p+xzSWS0hqW94GrM7s8znAHelxnkjybPKnM+v+R2B8+voIYFXm/THA\nWuDE9G/4xkaOVfr58elyY9M4lgGHpNMOTt9vk77/JXA+8AaSE64VwOyhjql//NOuH1/pW1eKiJXA\n/iQFzqXAEkk/kfTn6SyfAM6OiEcjokJS+OwpafvMas6JiJcj4lngQuDD6bqfiIhbI+K1iHgB+E/g\nQPK5EzhQ0vj0/TXp+8nA5hHxwCDLHAFcGBHPRcRLwNk5t3V2RKyMiGdICrs9cy4HSUH5VkmjI2Jx\nRDxca8aIuDMiHkpfLwCupu/xCGBmJPe7HwB+C/xVZt/OSo/zQiBbU0FEXBsRi9PXPwIeA/bNzLIw\nIr4ZEZWIWEMDx0rS/sCXgGkRsQr4KPCziLgp3e6twG+Aw9L/j78GvhgRayPiLpLbSGal4ELfulZE\nPBIRMyJiB5Ir0AkkhTckV6AXpdXKLwIvkBROEzOreDbz+ql0eSRtK+m/JT0r6SVgDvmreO8ADiK5\nOr6DpJagh6SQrNVAbgKQbdH+VM5tLc68foXkKnZYEfEKcCRwAvC8pBsk7Vprfkn7Srotrbp/ieSE\nqv/xqBXLBAYe5+y6j06r15ent0Z277fu/i396zpWaSH+A+DoiHgi/XhH4IPV/410u/sBb0rXvzwi\nXs27DbN2cqFvBkTEoyRV/XukHz0DfCIitk5/toqIsRExL7NY9qp/R+C59PXZJDUIu0fElsA/kb/h\n3B0kVcIHpq/nkhQo1feDeX6QWPrsXs5t5xYRN0fEe4DtSG5tXDrEtq4CrgMmpsfjEvIfj5r7lrax\nuBQ4Mf37bAU81G/d/eMZ7li9TtJo4Mckt2Z+kZn0DEl1ffZ/Y/OIOC9d/1aSxmTm3wGzknChb11J\n0q6SPi1pYvp+e5Lq+XvSWb4NnCZpt3T6FpIO77eaUyRtmS57Mkm1NSRXqauAlen6T8kbV0Q8DrxK\ncqJwR3obYjHwD9Qu9H8InCxpoqStgFP7TV8M5OqTn0dak/E+SZuS3DNfRXKSU93WJElvyCwyluTq\nd62kfYGP9F/lEJv7IfD59DhPAv41M22zdLvL0gZ6x7H+pG2o9Q11rLIuBx6OiAv6fT4HmCbpPel2\nRysZP2BCRDxNUtU/S9Ib0lsD04aJyaxtXOhbt1oJ/A3wK0krSRpfPQB8BiAiriO5j391WiX9AEmD\nv6yfAPcC80nu216Wfj4L2Bt4Kf382n7LDXflfQewLL2HXX1Pup3B1vEd4CaSe+G/GWR7FwFHKOmF\ncOEgy+eJKWsU8GlgIUkDtgNIqvohaRj3ELBI0pL0s38BviTpZeALJNXlQ207+34W8DRJ48cbgdmv\nz5S0I7gAmAcsIqnav3uY2Ic7VtltHwl8IG25vzJtxb9f2oZjOknjzqUk1fefYf336UeBt5PcEvoP\n4IphYjJrG0WUskutWalJqpC0Xi98lLpWk3Qp8CFgcUS8ueh4RjJJm5DUhmwMnBcRXyo4JOsyLvTN\nGtBNhb6ZdQ5X75s1xmfLZjbi+ErfzMysS/hK38zMrEu40DczM+sSLvTNzMy6hAt9MzOzLuFC38zM\nrEv8PzjZkPV+jleSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x25648f8ecc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from itertools import product\n",
    "\n",
    "all_clf = [pipe1, clf2, pipe3, mv_clf]\n",
    "\n",
    "x_min = X_train_std[:, 0].min() - 1\n",
    "x_max = X_train_std[:, 0].max() + 1\n",
    "y_min = X_train_std[:, 1].min() - 1\n",
    "y_max = X_train_std[:, 1].max() + 1\n",
    "\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),\n",
    "                     np.arange(y_min, y_max, 0.1))\n",
    "\n",
    "f, axarr = plt.subplots(nrows=2, ncols=2, \n",
    "                        sharex='col', \n",
    "                        sharey='row', \n",
    "                        figsize=(7, 5))\n",
    "\n",
    "for idx, clf, tt in zip(product([0, 1], [0, 1]),\n",
    "                        all_clf, clf_labels):\n",
    "    clf.fit(X_train_std, y_train)\n",
    "    \n",
    "    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    axarr[idx[0], idx[1]].contourf(xx, yy, Z, alpha=0.3)\n",
    "    \n",
    "    axarr[idx[0], idx[1]].scatter(X_train_std[y_train==0, 0], \n",
    "                                  X_train_std[y_train==0, 1], \n",
    "                                  c='blue', \n",
    "                                  marker='^',\n",
    "                                  s=50)\n",
    "    \n",
    "    axarr[idx[0], idx[1]].scatter(X_train_std[y_train==1, 0], \n",
    "                                  X_train_std[y_train==1, 1], \n",
    "                                  c='red', \n",
    "                                  marker='o',\n",
    "                                  s=50)\n",
    "    \n",
    "    axarr[idx[0], idx[1]].set_title(tt)\n",
    "\n",
    "plt.text(-3.5, -4.5, \n",
    "         s='Sepal width [standardized]', \n",
    "         ha='center', va='center', fontsize=12)\n",
    "plt.text(-10.5, 4.5, \n",
    "         s='Petal length [standardized]', \n",
    "         ha='center', va='center', \n",
    "         fontsize=12, rotation=90)\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/voting_panel', bbox_inches='tight', dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Notice the decision boundary from majority voting is a combination of the 3 base classifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hyper-parameter tuning for ensemble learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Get the parameter names "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'decisiontreeclassifier': DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=1,\n",
       "             max_features=None, max_leaf_nodes=None,\n",
       "             min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             presort=False, random_state=0, splitter='best'),\n",
       " 'decisiontreeclassifier__class_weight': None,\n",
       " 'decisiontreeclassifier__criterion': 'entropy',\n",
       " 'decisiontreeclassifier__max_depth': 1,\n",
       " 'decisiontreeclassifier__max_features': None,\n",
       " 'decisiontreeclassifier__max_leaf_nodes': None,\n",
       " 'decisiontreeclassifier__min_impurity_split': 1e-07,\n",
       " 'decisiontreeclassifier__min_samples_leaf': 1,\n",
       " 'decisiontreeclassifier__min_samples_split': 2,\n",
       " 'decisiontreeclassifier__min_weight_fraction_leaf': 0.0,\n",
       " 'decisiontreeclassifier__presort': False,\n",
       " 'decisiontreeclassifier__random_state': 0,\n",
       " 'decisiontreeclassifier__splitter': 'best',\n",
       " 'pipeline-1': Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', LogisticRegression(C=0.001, class_weight=None, dual=False, fit_intercept=True,\n",
       "           intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "           penalty='l2', random_state=0, solver='liblinear', tol=0.0001,\n",
       "           verbose=0, warm_start=False)]]),\n",
       " 'pipeline-1__clf': LogisticRegression(C=0.001, class_weight=None, dual=False, fit_intercept=True,\n",
       "           intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "           penalty='l2', random_state=0, solver='liblinear', tol=0.0001,\n",
       "           verbose=0, warm_start=False),\n",
       " 'pipeline-1__clf__C': 0.001,\n",
       " 'pipeline-1__clf__class_weight': None,\n",
       " 'pipeline-1__clf__dual': False,\n",
       " 'pipeline-1__clf__fit_intercept': True,\n",
       " 'pipeline-1__clf__intercept_scaling': 1,\n",
       " 'pipeline-1__clf__max_iter': 100,\n",
       " 'pipeline-1__clf__multi_class': 'ovr',\n",
       " 'pipeline-1__clf__n_jobs': 1,\n",
       " 'pipeline-1__clf__penalty': 'l2',\n",
       " 'pipeline-1__clf__random_state': 0,\n",
       " 'pipeline-1__clf__solver': 'liblinear',\n",
       " 'pipeline-1__clf__tol': 0.0001,\n",
       " 'pipeline-1__clf__verbose': 0,\n",
       " 'pipeline-1__clf__warm_start': False,\n",
       " 'pipeline-1__sc': StandardScaler(copy=True, with_mean=True, with_std=True),\n",
       " 'pipeline-1__sc__copy': True,\n",
       " 'pipeline-1__sc__with_mean': True,\n",
       " 'pipeline-1__sc__with_std': True,\n",
       " 'pipeline-1__steps': [['sc',\n",
       "   StandardScaler(copy=True, with_mean=True, with_std=True)],\n",
       "  ['clf',\n",
       "   LogisticRegression(C=0.001, class_weight=None, dual=False, fit_intercept=True,\n",
       "             intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "             penalty='l2', random_state=0, solver='liblinear', tol=0.0001,\n",
       "             verbose=0, warm_start=False)]],\n",
       " 'pipeline-2': Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "            metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "            weights='uniform')]]),\n",
       " 'pipeline-2__clf': KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "            metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "            weights='uniform'),\n",
       " 'pipeline-2__clf__algorithm': 'auto',\n",
       " 'pipeline-2__clf__leaf_size': 30,\n",
       " 'pipeline-2__clf__metric': 'minkowski',\n",
       " 'pipeline-2__clf__metric_params': None,\n",
       " 'pipeline-2__clf__n_jobs': 1,\n",
       " 'pipeline-2__clf__n_neighbors': 1,\n",
       " 'pipeline-2__clf__p': 2,\n",
       " 'pipeline-2__clf__weights': 'uniform',\n",
       " 'pipeline-2__sc': StandardScaler(copy=True, with_mean=True, with_std=True),\n",
       " 'pipeline-2__sc__copy': True,\n",
       " 'pipeline-2__sc__with_mean': True,\n",
       " 'pipeline-2__sc__with_std': True,\n",
       " 'pipeline-2__steps': [['sc',\n",
       "   StandardScaler(copy=True, with_mean=True, with_std=True)],\n",
       "  ['clf',\n",
       "   KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "              metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "              weights='uniform')]]}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mv_clf.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Grid search parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.967 +/- 0.05 {'pipeline-1__clf__C': 0.001, 'decisiontreeclassifier__max_depth': 1}\n",
      "0.967 +/- 0.05 {'pipeline-1__clf__C': 0.1, 'decisiontreeclassifier__max_depth': 1}\n",
      "1.000 +/- 0.00 {'pipeline-1__clf__C': 100.0, 'decisiontreeclassifier__max_depth': 1}\n",
      "0.967 +/- 0.05 {'pipeline-1__clf__C': 0.001, 'decisiontreeclassifier__max_depth': 2}\n",
      "0.967 +/- 0.05 {'pipeline-1__clf__C': 0.1, 'decisiontreeclassifier__max_depth': 2}\n",
      "1.000 +/- 0.00 {'pipeline-1__clf__C': 100.0, 'decisiontreeclassifier__max_depth': 2}\n"
     ]
    }
   ],
   "source": [
    "if Version(sklearn_version) < '0.18':\n",
    "    from sklearn.grid_search import GridSearchCV\n",
    "else:\n",
    "    from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {'decisiontreeclassifier__max_depth': [1, 2],\n",
    "          'pipeline-1__clf__C': [0.001, 0.1, 100.0]}\n",
    "\n",
    "grid = GridSearchCV(estimator=mv_clf,\n",
    "                    param_grid=params,\n",
    "                    cv=10,\n",
    "                    scoring='roc_auc')\n",
    "grid.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "if Version(sklearn_version) < '0.18':\n",
    "    for params, mean_score, scores in grid.grid_scores_:\n",
    "        print(\"%0.3f +/- %0.2f %r\"\n",
    "              % (mean_score, scores.std() / 2.0, params))\n",
    "\n",
    "else:\n",
    "    cv_keys = ('mean_test_score', 'std_test_score','params')\n",
    "\n",
    "    for r, _ in enumerate(grid.cv_results_['mean_test_score']):\n",
    "        print(\"%0.3f +/- %0.2f %r\"\n",
    "              % (grid.cv_results_[cv_keys[0]][r], \n",
    "                 grid.cv_results_[cv_keys[1]][r] / 2.0, \n",
    "                 grid.cv_results_[cv_keys[2]][r]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters: {'pipeline-1__clf__C': 100.0, 'decisiontreeclassifier__max_depth': 1}\n",
      "Accuracy: 1.00\n"
     ]
    }
   ],
   "source": [
    "print('Best parameters: %s' % grid.best_params_)\n",
    "print('Accuracy: %.2f' % grid.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Observations:\n",
    "* decision tree depth doesn't matter\n",
    "* lower regularization (larger $C$) better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Note**  \n",
    "By default, the default setting for `refit` in `GridSearchCV` is `True` (i.e., `GridSeachCV(..., refit=True)`), which means that we can use the fitted `GridSearchCV` estimator to make predictions via the `predict` method, for example:\n",
    "\n",
    "    grid = GridSearchCV(estimator=mv_clf, \n",
    "                        param_grid=params, \n",
    "                        cv=10, \n",
    "                        scoring='roc_auc')\n",
    "    grid.fit(X_train, y_train)\n",
    "    y_pred = grid.predict(X_test)\n",
    "\n",
    "In addition, the \"best\" estimator can directly be accessed via the `best_estimator_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', LogisticRegression(C=100.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "           intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "           penalty='l2', random_state=0, solver='liblinear', tol=0.0001,\n",
       "           verbose=0, warm_start=False)]]),\n",
       " DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=1,\n",
       "             max_features=None, max_leaf_nodes=None,\n",
       "             min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "             min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "             presort=False, random_state=0, splitter='best'),\n",
       " Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "            metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "            weights='uniform')]])]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.best_estimator_.classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "mv_clf = grid.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MajorityVoteClassifier(classifiers=[Pipeline(steps=[('sc', StandardScaler(copy=True, with_mean=True, with_std=True)), ('clf', LogisticRegression(C=100.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=0, solv...ski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "           weights='uniform'))])],\n",
       "            vote='classlabel', weights=None)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mv_clf.set_params(**grid.best_estimator_.get_params())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MajorityVoteClassifier(classifiers=[Pipeline(steps=[('sc', StandardScaler(copy=True, with_mean=True, with_std=True)), ('clf', LogisticRegression(C=100.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=0, solv...ski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=1, p=2,\n",
       "           weights='uniform'))])],\n",
       "            vote='classlabel', weights=None)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mv_clf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bagging -- Building an ensemble of classifiers from bootstrap samples\n",
    "\n",
    "Use random subsets of samples for each base classifier.\n",
    "* bootstrap samples: random sample with/without replacement of the entire training data\n",
    "* rows (samples) or cols (features)\n",
    "\n",
    "Parallel method\n",
    "* base classifiers operate independently\n",
    "\n",
    "<img src='./images/07_06.png'> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "7-sample training data example:\n",
    "<img src='./images/07_07.png'> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bagging example\n",
    "\n",
    "Classify the wine data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "remote_source = 'https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data'\n",
    "local_source = '../datasets/wine/wine.data'\n",
    "\n",
    "df_wine = pd.read_csv(local_source, header=None)\n",
    "\n",
    "df_wine.columns = ['Class label', 'Alcohol', 'Malic acid', 'Ash',\n",
    "                   'Alcalinity of ash', 'Magnesium', 'Total phenols',\n",
    "                   'Flavanoids', 'Nonflavanoid phenols', 'Proanthocyanins',\n",
    "                   'Color intensity', 'Hue', 'OD280/OD315 of diluted wines',\n",
    "                   'Proline']\n",
    "\n",
    "# drop 1 class\n",
    "df_wine = df_wine[df_wine['Class label'] != 1]\n",
    "\n",
    "y = df_wine['Class label'].values\n",
    "X = df_wine[['Alcohol', 'Hue']].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Encoding and splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "if Version(sklearn_version) < '0.18':\n",
    "    from sklearn.cross_validation import train_test_split\n",
    "else:\n",
    "    from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)\n",
    "\n",
    "X_train, X_test, y_train, y_test =\\\n",
    "            train_test_split(X, y, \n",
    "                             test_size=0.40, \n",
    "                             random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Bagging classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "tree = DecisionTreeClassifier(criterion='entropy', \n",
    "                              max_depth=None, # allow over-fitting to enhance diversity\n",
    "                              random_state=1)\n",
    "\n",
    "bag = BaggingClassifier(base_estimator=tree,\n",
    "                        n_estimators=500, \n",
    "                        max_samples=1.0, # rows\n",
    "                        max_features=1.0, # cols\n",
    "                        bootstrap=True, # replace?\n",
    "                        bootstrap_features=False, \n",
    "                        n_jobs=1, \n",
    "                        random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree train/test accuracies 1.000/0.833\n",
      "Bagging train/test accuracies 1.000/0.896\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "tree = tree.fit(X_train, y_train)\n",
    "y_train_pred = tree.predict(X_train)\n",
    "y_test_pred = tree.predict(X_test)\n",
    "\n",
    "tree_train = accuracy_score(y_train, y_train_pred)\n",
    "tree_test = accuracy_score(y_test, y_test_pred)\n",
    "print('Decision tree train/test accuracies %.3f/%.3f'\n",
    "      % (tree_train, tree_test))\n",
    "\n",
    "bag = bag.fit(X_train, y_train)\n",
    "y_train_pred = bag.predict(X_train)\n",
    "y_test_pred = bag.predict(X_test)\n",
    "\n",
    "bag_train = accuracy_score(y_train, y_train_pred) \n",
    "bag_test = accuracy_score(y_test, y_test_pred) \n",
    "print('Bagging train/test accuracies %.3f/%.3f'\n",
    "      % (bag_train, bag_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Notice bagging has less over-fitting than decision tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Visualize decision regions\n",
    "\n",
    "Use meshgrid and isocontour as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAADnCAYAAAD8QEcbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPXVx/HPyUJYVAQiqGCCC9CiIrigtn0gtLVVq7Wt\n7VOrrUut2j5tqbbaBRdAW1uX1t26oaIVq60rdalViYgLKIIsgsQCYRMisotAlvP8cW/CJEySyWSS\nmbn5vl+veTG5c+fm3CRzOL/tXnN3RERERKImJ90BiIiIiLQFFTkiIiISSSpyREREJJJU5IiIiEgk\nqcgRERGRSFKRIyIiIpGkIkfSxsyeNbMfJLDfZjPr3/YRiUhUmdk8MxuR7jikfZmukyNNMbOlQG+g\nEqgG3gMeBO7yLP7jMbPNQG383YDtBOfnwAXu/nC6YhOJspicUkWQV14HfuzuK9MZl0STenKkOQ58\nzd27A8XAn4DfABPSGlUrufvu7r6Hu+8BlBOcY+22XQocM8tt/yhFIqk2p+wB7ANUALekNySJKhU5\nkggDcPfN7v4v4LvAWWY2GMDMOpnZ9WZWbmYfmtntZlZQ92azU8xslpltNLMyM/tKuH2Kmf0wfH6g\nmZWa2QYzqzCzh2PeX2NmB4TP9zCzB8J9lpjZpTH7nWVmr5rZdWa2zsz+a2bHJ3h+Vm+D2VVm9ncz\nm2RmG4EzLDDGzD4Iv/8kM+se857Pm9kbZrbezN4xs/9p8U9apGOozSk7gH8CtbnkxPCzszHMJ2Pr\nvcnsTDNbamYfmdllYQ74YvhaZzObGH7255vZJWa2POa9sfuONbNHwv03mdlcMzs8Zt/DY+J4NMwF\nV7b9j0VSTUWOtJi7vwWsAGr/E78GOAgYEv7bF7gCwMyGAxOBX4W9QSOApXEOexXwb3ffE+hH/ZZd\n7LDYrcDuQH+gBDjTzM6JeX04sADoBVxH63qcvgH8LYz7EeCXwAnAF8IYt9TGaWb7AU8Bl7t7D+C3\nwONm1qMV318k0sysK0Gj6Y1w0xbgB+Fn7mvAj83s6+G+g4HbgO8R9AB1B/aNOdw4oIggNxwHfJ/6\nuaOhk4FJ4XEmh8fGzPKBx4F7gZ7Aw8A3W3WikjYqciRZqwgSAMB5wEXuvtHdPyEY0vpe+NoPgQnu\n/jKAu3/o7oviHK8SKDazvu6+w91fj3nNAMwshyAh/tbdt7p7OfBnIHbycrm73xvOF5oI7G1mvZM8\nx2nu/mwY93bgAmCMu68OW6BXAd8J9/0B8JS7vxju/wLwLpBIT5JIR/Okma0DNgBfBq4HcPep7j4/\nfD4P+DswMnzPqcDT7v6Gu1cRNqRifAf4g7tvcvdVwM3NxDDN3f8d5ooHCRppAMcCue5+q7tXu/sT\nwIxWna2kjYocSVZfYJ2Z7QV0BWaG3cTrgOcIelIA9gP+m8DxLiH4e5wRdh2fE2efQiAPWBazrTyM\npdbq2ifu/ilBgbRbYqe0i+UNvi4CJsec5xygJiyiioHTa18zs/XA0dRvaYpI4BR37wkUAD8HpppZ\nbzM72sxeDoeDNxA0LArD9+xLzGcy/Hx/HHPMfQl6mGs1/Pw2tDrm+Vagc9iQ2gdoOAm6uWNJhlKR\nIy1mZkcRJJRXgbUECeJgd+8ZPvYMu5shSA4HNndMd69w9/PdvS/wY+D22nk4MdYS9vjEbCtm14SU\nKg27upcDx8WcZw937+buFeFr9zZ4bXd3/3MbxSaSzWrn5HjYU1JNMAz8EPAk0Dccur6TnfPlPiQY\nJg4OYNaFnY2pXV4naJQk40PqN5wgaKxJFlKRIwkzs93N7CSCMeoH3f29sKv3buDGsFcHM+tbO7mY\nYE7MOWY2Kpy4u6+ZDYxz7G+bWW1i2QDUhI867l4DPAr8wcx2M7Ni4CKCrub2cCfwx3D+DWHL8+Tw\ntQeBb5rZl80sJ5wEWWJme7dTbCJZycxOAfYkmEu3G7De3SvD+Xynx+z6T+BkMzsmnDczrsGhHgV+\nZ2Z7hrnkpy0NJfz3DaDazH5qZrlhfMNbeCzJECpyJBGTwxVGy4DfEYyf/zDm9d8AHwBvhl3MLwAD\noW6S8jnAjcBGoJSdPTGxPSVHAdPNbBNBS260uy+Ns99ogp6jxcBUgonB9zUReyLX8kn0ej9/JhiK\neyn8eUwDjgQI5wd9E7gc+IhgcvUv0WdMJJ7J4aqmjQRz28509wUEhclV4fbLCCb8A+Du7xEMbT1C\nMCdwE8Hy8+3hLlcS9OouIchB/4h5DZr/nHv4fSqBbwE/AtYTFFqTGxxLsoQuBigiIlnHzLoR9Poe\nFDYyGr7+Y+C77j4qBd/rTeCv7j6xtceS9qVWpoiIZAUzO8nMuoQFzp+BObUFjpntbWafC4fFBwG/\nIlgKnsz3GWFmfcLhqrOAQ4HnU3Ue0n4ypsgxs37hrPr54eqa0XH2GWnBxeLeCR+XpSNWERFJi1MI\nhqpWECxoOC3mtU4E8+Y2AS8CTwB/TfL7DCK4BMR6gnl/p7r7miSPJWmUMcNV4QTNvd19tpntBswk\nWGa4MGafkQQXlft6uuIUERGR7JAxPTnhBdZmh8+3EMy0b7iMDxpcfl9EREQknrx0BxCPmfUHhgLT\n47x8rJnNJphFf0k44z7eMTKji0pEUsbdk2rkKB+IRE8i+SBjenJqhUNV/wR+EfboxJoJFLn7UIJ7\nGD3Z1LHGjh1b95gyZQrunhGPsWPHpj0GnYfOJdMfU6ZMYeTIkXWf4dZSPtB56Fyy95FsPsionhwz\nyyMocB5096cavu4xRY+7P2fB3a57uvu6eMcbN25cm8UqIm2rpKSEkpKSus/x+PHjW3U85QOR7JVs\nPsi0npx7gffc/aZ4L5pZn5jnwwkmTsctcERERKRjy5ieHDP7PHAGMNfMZhFcfXIMwdVx3d3vAr5t\nZj8huH/RpwR3pM46JSUl6Q4hJaJyHqBzyVRROpfGROUco3IeoHPJVMmcS8YsIU81M/OonptIR2Rm\neCsmHisfiERHovkg04arRERERFIiY4ar0mXy5E3pDkGkQzv55D3SHUId5QOR9Ep1PlBPjoiIiESS\nihwRERGJJBU5IiIiEkkqckRERCSSVOSIiIhIJKnIERERkUhSkSMiIiKRpCJHREREIklFjoiIiESS\nihwRERGJJBU5IiIiEkkqckRERCSSVOSIiIhIJKnIERERkUhSkSMiIiKRlDFFjpn1M7OXzWy+mc01\ns9GN7HezmZWZ2WwzG9recYqIiEh2yEt3ADGqgF+6+2wz2w2YaWYvuPvC2h3M7ATgQHcfYGZHA3cA\nx6QpXhEREclgGdOT4+6r3X12+HwLsADo22C3U4AHwn2mA93NrE+7BioiIiJZIWOKnFhm1h8YCkxv\n8FJfYHnM1yvZtRASERERyajhKgDCoap/Ar8Ie3SSNm7cuLrnJSUllJSUtCo2EWk/paWllJaWpux4\nygci2SvZfGDunvpokmRmecC/gOfc/aY4r98BTHH3R8KvFwIj3X1NnH09kXObPHlTq+MWkeSdfPIe\nCe1nZri7JfM9lA9EskOq80GmDVfdC7wXr8AJPQ2cCWBmxwAb4hU4IiIiIhkzXGVmnwfOAOaa2SzA\ngTFAMeDufpe7P2tmJ5rZB8AnwDnpi1hEREQyWcYUOe7+GpCbwH4/a4dwREREJMtl2nCViIiISEqo\nyBEREZFIUpEjIiIikaQiR0RERCJJRY6IiIhEkoocERERiSQVOSIiIhJJKnJEREQkklTkiIiISCSp\nyBEREZFIUpEjIiIikaQiR0RERCJJRY6IiIhEkoocERERiSQVOSIiIhJJKnJEREQkklTkiIiISCRl\nVJFjZhPMbI2ZzWnk9ZFmtsHM3gkfl7V3jCIiIpId8tIdQAP3AbcADzSxz1R3/3o7xSMiIiJZKqN6\nctx9GrC+md2sPWIRERGR7JZRRU6CjjWz2Wb2jJkNTncwIiIikpmyrciZCRS5+1DgVuDJNMcjzVi/\nviLdIYhIhlA+kPaWaXNymuTuW2KeP2dmt5tZT3dfF2//cePG1T0vKSmhpKSkzWOUnRYvnsNFF43k\nxhunsv/+h6Y7HMkypaWllJaWpux4ygfppXwgrZFsPjB3T300rWBm/YHJ7r7Lp8DM+rj7mvD5cOBR\nd+/fyHE8kXObPHlTa8KVJowd+31mz17G0KHFjB//YL3X1q+voEeP3mmKTDLJySfvkdB+Zoa7JzUn\nT/kg/ZQPJBGpzgcZNVxlZpOA14GBZrbMzM4xswvM7Pxwl2+b2TwzmwXcCHw3bcFKkxYvnsP8+W/h\n/hLz589gyZK59V47++xB9baJSHQpH0i6ZFSR4+6nu/u+7l7g7kXufp+73+nud4Wv3+buh7j7MHf/\nnLtPT3fMHVlT4+sTJ15LZeUlQC8qKy/h/vuvrfcaHFpvm4hkN+UDyUQZVeRI+2ntBMCmWl8VFcuY\nNWsynTs/RNeun6dz54eYNetpKiqWNdmik8RpAqekkvJBdlM+aJyKnA4oFd3DTbW+Cgv7cd11LzF2\n7NVcccUVjB17Nddd9xKFhf2abNFJYtS9L6mkfJDdlA+apiKnA2pt93Bzra+cnBwGDTqSwYOPrXsM\nGnQka9euaLRFJ4lT976kkvJBdlM+aFpWLSGX1tuZkOYxf/4hLFkyt8XLOeO1vsaPf7DZFRK1Lbrq\n6sq6bbm5+RQW9kv2dFImW1Z3pOL3J1JL+SA+5YPoUE9OB9Pa7uHGxtfffvuFZrtMG2vR5eSk988w\nm7p71b0vqaR8sCvlg2hRT04HUpuQunRZidljuG9j1qx3qKhYRu/eRQkdo7HW10MP3YD7Z+pacdkk\ntrs3k2NPxe9PpJbyQXzKB9GiIqcDSUX3cG3rK9bixXOYN+8NYD1z565pUZdpuruFs6m7N5O79yX7\nKB/sSvkgehq94rGZXZnIAdz9ipRGlCK6wmnTUplMxo79PrNmrQBqgFyGDeuXUAso9jLve+7ZJ+l4\nWnMuwVVYS3D/JWZ/YejQV9Laekt3kk8HXfE4/ZQPAsoH6deeVzzeL4GHSsYslMox59ouU1gKvAQs\nSXiFRG238F//ennS8bTmXJq6fkc6ZNNcAIkO5YOA8kE0NTpc5e7ntGcg0n5SOeZcWNiPgQNHUFZ2\nCu69MLuMAQOebrbLNLZb+P33B+LeP6l4Ys/lwgv/3KJWT6Z192bLXACJFuWDnbErH0RPwnNyzGwA\n8D2gL7ASeNjdy9oqMGkbqR5zXrt2BYsWTaVLly11k98WLXqHtWtXNDn5LXZVgPvvgHHMnbuxRfHE\nnsvcuYM4++xBXHnlkxx22MiE3h9vPkG6ZNNcAIkO5YOdlA+iKaG7kJvZycBDwL+AcqAIOAn4gbs/\n3aYRJklj8PGlesy5pqaGsrJ3dmn9DBhweKNLQSsqlvGjHx1Kly6Hs317DTU124CFwMEMG3ZgwvHE\nngscDlQD/+Wmm/6ddQkh0+YCtCfNyUkf5YPMpHzQvETzQaI9OVcDp7j7lJhvUALcCmRkkSO7aosl\nh8m0fjZtWgcYp532Te6773Jgd+DfwHeZNWt+QvHEnov7A2zbtgwoAwZw++2Xct112fNnqaWgkg7K\nB5lJ+SC1Ei1y+gGvNtg2DU08ziqZMub84IPXYzaEWbNmxIzffwWzyxMav4f65zJhwh/44INLce8F\nXMaiRb9PWfdue6xuyJTfi3QsmfJ3p3xQX6b8XqIi0SJnNvAr4JqYbb8Mt0uWyIQx5/pjzZ+lsvIj\nCgo2sX37vXTqlM+iRbObHb+HnedSUbGMsrJXyclZi/vfAcd9PXfdNY4//vGxVsdau6S1Lbu7M+H3\nIh1PJvzdKR/sKhN+L1GS6PWzfwL8yMxWmdl0M1sFnB9uFwGCVk5z6l+G/NcMHPg/FBV1BbZRVNSt\n7u7EiSos7MeYMQ9RU7OATp02UVCwnYKCQcyf/2Krl37qxnciyVM+kEyQUE+Ouy80s88CxwL7AKuA\n6e5e2fQ7paNIpJUTb6x50aJ3yM/vA0yhvPyLdOpU0KJ71+Tk5DB8+Ikp797V6gaR5CkfSKZIaHVV\nvTeY1fuLc/ealAVjNoFg1dYadx/SyD43AycAnwBnu3vcITOtpmhfwWqAZQwdWtzoKoB4Ky/uvfdq\nFi06ErgRuIhhw8p2eX86rvrZkVc3tDetrooe5QNJVnte8Tj2YIeb2Rtm9glQGT6qwn9T6T7gq03E\ncQJwoLsPAC4A7kjx95ck7GzlvMT8+TMavUJnw7sOFxb2ZdGiqeTkTAIOISdn1yuMpuOqn8le+TSR\n7nmRqFM+CCgfZIZE+wEnAlOAI4EDwsf+4b8p4+7TgPVN7HIK8EC473Sgu5n1SWUMHUFrP3wN319/\nXP2SJsesY99bWNiPCy+8k5ycGuBlcnJquOiiu+p1K6djHLx2dcPYsVdzxRVXMHbs1c3ODdAl2CVb\nKR80TfkguyVa5BQDl7r7Ancvj320ZXBx9AWWx3y9MtwmCUrkw9dU0mv4/pa0chq+Nycnh1de+RfV\n1b8BelFd/RtKS/9VNwafaIsw1Rq2MAcPPpZBg45scm5AIslXLTvJNMoHzVM+yG6JFjlPAF9py0Ck\nfdxzz1U09eGrTTyvvfZk3A9hww9vS1o5Dd/bXEJsSYswnRJJvmrZSSZSPkg95YPM0ujqKjN7EKid\nqVcAPGFm04DVsfu5+5ltF94uVhLc/bxWv3BbXOPGjat7XlJSQklJSVvFlRWmTn2MefP+A0xh/vzT\n4q4QmDjxWtwP4pprzgKMm256tW6fxlYYJHJNh3jvLS4+uNFVENl01c94ybfhpETdbK/lSktLKS0t\nTdnxlA/qUz5oG8oHbSPZfNDo6iozG5vIAdx9fIu/a1MBmfUHJrv7LuvzzOxE4Kfu/jUzOwa40d2P\naeQ4Wk3RwBlnDGHz5u7AQZh9fpcVAosXz+HXv/4OO3YcSXCLsgqGDTu6bp+GKwwOPvjfXH31P+re\n39Sqh5asTli/voJFW/ZmVfnMXRLevsVHYC1YUtqc4X2rmt1nxsrGr7Sw4eNl3HrZZ+lUcASWU4DX\nbGfH9pn87PcL2LNXkHxXL3+X+6/7FlWV88jLP4Rzfv0EffrFXTzYIV31464J7afVVamVTfmge/fC\nhO6J1dRnNRHKB+mX6nzQ6G8r1cVLIsxsElAC9DKzZcBYoFMQjt/l7s+a2Ylm9gHBEvJz2jvGbPX2\n2y+wefMGYCYwgE6d3mPWrIX1WkITJ17Ljh2nA38D3gMOYe7cV1myZC7dunWv15Kqrl7HvHkLmTnz\nPxxxxHFNXhejJa2wxYvncOFFIzn/n9PZr+Rzddu3rF3DboWpn2M+qayG03MavwrCjJV5fLBPDQMH\nFMR9vUfN/vzwwGnUVO5Mvjn5+fQ4ZP+6Yuzl866hujpo2VVXX8LLpddwxl2P1u3/ybqP2LCynD37\nFtOt516pOTGRJmRTPqg9TmwPUbwCalJNDjTxWU2E8kH0JFz2hjfkPJNgou9K4MHYG3amgrufnsA+\nP0vl9+wonn56ImZX4N4Ls8soKnqK88+/rW6svDbxQG8g+ADCr6mqepD777+WsWMn1utKnjDhD5SV\nFfD00xM54ojjmux+bXgvllWr/sstt8xm8+b1uyS1P90fHOeNO8ZwyTN/AaB89kJuGPVdrpr+KEWH\nDUrpz8V3g0mz+sdNbLUJbcCwpRy0T+PHGDCoe5ytZQCsLV/F4tefovPuy7Ccf1BTtYX/vjaPvM1P\n0P/wwUz/+7P87YKxFOXnsayyijPuGs8x3z0xRWeXLQ5JdwAdTrbkg3jHiVdAzViZR7eh7zPyiPxW\n/VyUDzJBavNBQkWOmf2I4E7k9wDTgSLgYTO73N3vTmlEknIVFcuYPftfdOmyqq7lVFb2Dj169K7r\n6i0s7MdPf3ojt912IXAX8CCwA1jIrFmzWLt2RV1LavHiOZSXlwHBePqrrz7e5NVAa1cn1La+Hnnk\nNuBQHnjg+noJ8LYP5rF24Qzw+ZRNHcS+T1/JwZ8p5LQLXwY/hBcvvIyH7xyV1M9g7bpPWbFyM/36\n7k5hzy512/sMPIRVqyqZtGZAvcRWm9C+euwDHLZ6LvtsTmqUhJoaZ9Ckb1FZVcO0N1Zw94QFDOqS\nxw0lpzPm159n0rWvMXVbNUM+hTlAyXlj+EnxgnoxRt7Bt6c7gg4lW/JBY3N+GhY+M1bm8eHQ9/nf\nI95gyKJ5Cf0MmssHM2YPqjd0FZsPjv9oHmxO7mevfJCAFOeDRHtyfg0c5+7v1m4ws0eAxwAVORku\nkbva5uTk8KUv/QDIYdWqD6ipqQYgL+94jjrqhF2uVRE7se6OO8Y3O9GutvV18cUT4iauSTU5lD05\njuodwXF27Pgtf7z8AcadcjTT31qL+/u8OWMQ//1bFYf2a9mw1WNvz+Pih/5FcW4u5dXVXH/GyZx6\n5MFUU8YQ5sER8OhM6hLbyws+ZvWgvRgwbCmHrZ5L/1fXk8uAlv7YAVi7+RO6rNuTvTrlc+89/2Jq\nZTVDCBLY//zhNQ7Iy2UIwc96CFBMHh9NzqGouFdS3y8rHZzuADqWbMgH8Y57//3XctZZl9Tbf/L0\n96g5qaCuwCkoXdfsZzWhfMCu+aC2wVOgfNC2UpwPEi1yehEMysZ6H+iZ2nCkLSR6V9u8vDy++tWz\nmtyn4Xh6Tc2nbN68lIKCe8nNbXx8vbb1FS8Bfu38h/kkdymLp/6HbruvJrfqIWqo4aUFc9maU8CO\nqjFAL3ZUjWHsv5/g8fEnJ3zuazdu5OJJ1zKlsoohlVXMAUZNeoaRX/oWhd2HU1D6EAMG5zDyiHye\nXVND4eI53PTbkZz3z+l8YZ8ZHLp5L7ZSCEXDE/6etR575RUuvuU2ivPyWFJZyZ4YtdMLhwBFnTqx\ntCqIqTbRLatxioeMgu7xurxFWi+T8sGtt9fPBzfdeR2n/3wSGz4OjtupYBWW8xhes51Zs2aydvPW\nevs/++Ll3Dv2mzsLnKIzmoy3Nflg3x5w6EfKB9km0SJnGvAXM/uNu281s27AH4HX2y40SUZb39el\nYSuwpqaGDz9czD77HFDX1d2wVbiz2/k5Nm8eRkHBg/US4LDydxh6xlEcPu0hBte8w+5PzCKnz1f4\neNPXOOPqq9mtSyU59gg1vp2XZr3LsoqzKeqd2DmWV1RQnJfHkB07gDCZ5OVRXlFBYZzEESyZPZSX\n/vIHTj35uKR/Tms3buTiW25hyo4dDNmxgznAMcCTwDcIEtiq6mrGn3ceo+65h6K8PJZVVXH96NFx\n4xJpqfa4x1Nr88Gnnwwjv8uD5OQ9jtdsY/F7M1ndcw77fGHYLhN4t25Yy6Ojv0OnbuuxnMfxmq2s\nnD6LHqV7U7CostkCB5QPOqJEi5wfA48AG81sHUEPzutAsxOFpf0kcuffloiXJOO1Avv2HdBkMt3Z\n7TwEGE1R0UzOPfdyAFavXsqNfxrB+UfN4AsnD+GovE3sOWcFuUWDqamp4aXrrqOyurruWPm536df\nYWHC51DcuzflDVtHVVUUxymS1ofJF+ZR/vYh/Pfdz/LFLsktSY2XTPsBpwH75OezyYzrR4/m1BEj\nOOnYY4P9e/euS2hrN27cZZtIolKdCyC1+WBHbT6w0ex38Ot877rzAfhoyUruOu1z/H7GPyg5tf4i\ng5qannz28IeoqgzmyvRkNp/5cH8OWLiD3JyBCZ2D8kHHk9AFR9z9Q3cfQXC/qpOB/d19pLs3eiE+\naX+pvK9LolfkbG6/+lcx/R+6dJlGWdlrFBb2ZfDgY3nllWeAIbz0lz/s8t6cnByOHDSIYwcPrnsc\nOWhQk5dTb6iwe3euHz2akfn5DCooYGR+PtePHg3AzLIy1m7eVrfvuw/8qa4rvLryEiaM+U/C36eh\n2GQKQTL9GHge2AA8f8MNnDpiRF2MRwwYUJe8HnvlFY4691x+dfnlHHXuuTw2dWrScUjHlOp7PKU6\nH+R3+Rvduh9Nt92msPjttzm+/3LOHbmVBX9/HONQXrjsGr6QN63eY0Sn1znrc5s4d+RWzh25ldP3\n2sax6/LJzxmY8PCR8kHHk+jqqq8AS919EbAi3DYIKHL35H/zkjKNrURIVmySvPDCPzfaMmvuyp2x\n3dlbtmxg/fo1FBZeSmFhv7qYa1tKy959H46o//6UtGDcMTPygGp3Xpk9m4tvvpnivDzKK7czrrgX\nVUcdwOrZwdyCGh7Dc7Yz8z8zKV91LHvR8u9bl0xvuom9Kiv5GLid4CJQ/fPz2bJtW9xzi9etPerm\nmxl52GFqwUlCYnPBvHkHtzoXQOrywZLt/Tn5nomUHDibwvkLWf/aR+zbYwSfnTyH+as28NaUxbgv\nYsbLA1kx5gkO6dej3vvXbt7GsnWfUNSzGwW7dw4mALd0fozyQYeSaN/bbcCIBts2h9sT6yeUNpXI\npcQTVT9JDuasswbWu5x7vP0aK6xqu7NffeUf3HXLzynKy2dZVSXnj76VF196sl5L6Ykxf+LcG46q\nW7kQTNS7JUg+4fj0qSNGtKjwqU0Sv92xg2uAA4BHX3yRi4Erw6RRcu6DjF/0A4679mW+VLODmUs3\n0e1zezHsgGfYb4/ObFuU1I+RU0eM4JD99+fLv/gFT1VVUcLO7vE5H3zAt3/3u13OraVzBkQaqp8L\nLuavf72ca699MunjpSofzFiZx+rD3+dnR2xhyQ1zuOiyUorzC+r+/v/20ip2VF1KsMjgUsb/+wke\nH7/zsmjx88Hw1OeDH07k9rKDOO7al9lnTTWLO33MgUds55S+89nzraWQk9znUPkgPRLt9+/t7h82\n2PYhsHeK45Ek1HYBm90DHIbZhEbv/JuIhgUT9I7b7Z3oDfM2blzLXbf8nKk7PmXO1k1M3fEpt9/0\nU2bNmkxu54fI7/p5Crrez8z/vE3F48FN5mNbMO9s3cqUHTu4+Oabue+551rUdTtn8WL2NuMaYAow\nG3gTuAX4iDBp5Oeybvlqeg08is6du/HPO0+jU7fd+Mzw/cnJSe76OLUG7bcfN154Iad26sSwrl0Z\n1akTY3/0I66cMGGXc1u7cWPcbu3G5gyINBQ7PJyTMwyYwMKFU5LOBZCafBB7LZt933yLMZeVMqWy\nuu7v/1cM/zxiAAARGUlEQVQ33cTLs6bRtfMD7NH1aLp2foCXZk1jWUVwU9D2zAfF5LDntJmcftYC\n1vTtymt/Op2jixdy+My5LRoai0f5oP0l2pOz2My+6O4vx2wrAZakPiRpqcLCflx44R3ceusYamoe\nICfnTH7xizvj3vm3ObFLQt0fZdu2DcAa5s17o17LrCWXZq+oKKdvTj5D+BQIEsn+XfI45vYr6XNQ\nMT2ZzYHLl9B1Th+K9jwMioZTXla2Swtmv9xcrrj7bl6tqkqo6/axV17hlzffzKeVlXwmPEbdsYCl\nBJX6sspqCov3hYrUrqaobV2eOmIEIw87rO7rplpnRwwYwPWjRzPq5pu1wkJarHZ4ePnyhdx+++XU\n1DxAXt6Zca8mnIhU5IOllQfUu1jf/MkrKM4vYEjlViAsLPLz+f1PfsKB++5b971jFxnE+8y0WT6o\ndg5cWsXQj+Zx63P3AYfw9zMnccZPf5BUgdOwp0n5oH0lWuSMAx43swnAf4EDCe4bpXtHZYCcnBxe\neeUZqqsvBb5GdfWllJY+w6hRp7X4WLFzaCZMuJKysi8DE6msfLXeEFgiFxSrtaL6AJbVVNZb0bCy\ncjsXfL0zvfbaSpf3tlEwv4Dc/ofUJZF4qyDKq6ooys9nSFWwuqKprtvalt8rlZVMB34RHqP2WB8A\n53TuzMqqHVxyw7fZY6+eKVtN0dgwW2yMTa3waJgEldAkUbXDw5Mm3VgvHzS8mnCiWpsPlmzvz+rD\n3693sb79h5xFedXUXf7+jz/qqEb/1tstH4TLuPvs3pPZf5vB/KlLcH+fGR8MYm71XrR0ZlNTuUD5\noH0kurrqKeArQDfga+G/Xw23S5rVX8H0eTp3fijp4araJFlY2Jeystfp0uUNunb9JV26PFzvmLX7\nDR58bN1j0KAjd1n5NGNlHh8O7MX3b/4uJZ1zOXy3AkZ1zuPW8Scx9KPN7Pfeqp1XKY1pJdVO1BsV\n0607/rzzWFVdnVDXbV3rCDgPuAk4Fji0oIBRnTrxnS9/meVVVfTLzeH6i/7Jm488m5LVFI11q6/d\nuLHJc2vYOmu4wkIkUZmSDzbudgyrDy/b5WrEifz9N9Ru+SAvj/H33MNjS7cx/skP2bHjt9RdiPT+\nf+xy3KYkkgsaOzflg9RJuJnq7jOAGW0YiySpJb0q7XnM2Bva/d/mLvxy7EmsWNeL4p7dKezaja2l\nHhy3kRUS8Vowu3ftmlDXbXHv3iwOu7CHAEcDubm5XDVmDH0LCzn+oouCbu6q8JLq513Bpm3bW72a\nItGJgk21znRNDGmNTMgHdZ/9ffPr324h/Jwn0zvRPvkg6FEpuekm1ldWslvBRHJy/9HmFyJVPmg7\njRY5ZnZlIgdw9ytSF44kI9HLtKfjmAMHFAR37N0MfXY/lD4Ht2xMu2G3bkuSY7V7sESTYLzdgSEH\nHBA/+XTKp+t5N/Ctvgfw3ke55ByWl9TqqpZcbKzhuUHj3dsiicqUfBD72Y/XkIn399+c9soHzc0R\nSkRLckG8cwPlg1RoqidnvwTen5uqQEQSlUhyLK+oYEBBAf/eupWlBIntKwUFOyf7NUg+yyurGHXk\n8Qzu0YstK/PIOSKfzwx4j5xFq1m7eUswETqBllRd13Oc1mVzLTJdE0Ok5doiHzQ1RyjRnpWmckEi\nx1E+SI1Gixx3b3RSsZkNAc5Et3WQDFWbuD4EjqJ+K6pe8slxyg3OuGs863bfC6ipd5xHn5nHz6+Y\nTHF+p4RbUvFal/c99xxX3H03Rfn5rKqujnuclnRvqwtbJHEJ54Nmhr1a2rPSWE+T8kH7SXhOjpnt\nRVDUnAUcBrxKMEldpFXa4gPaXOKqTT5L5kyk+8VfYUGPE3j22frH2PDRZn4+dnL9OxbHaUnFiz+2\ndXn/88/zu7/+lYHA8qoqfgNcHOc4iXZvqwtboiyd+SAVPSsN42/Y06R80L6aLHLMLB/4OnA28FWC\nlXYPE/T2/a+7V6QyGDM7HriRYNXXBHe/psHrI4GngMXhpsfd/fepjEHaV1t+QJtLXIXdu9OjuBcb\n9todqnZ9/5rydfTPy2UIjS9RbS7+tRs3Mvbuu3mTnYlqFLBvbu4uLbJEWpTqwpYoS3c+aOozlEjP\nivJB5mmuJ2cNQf/9/cBYd38HwMz+L9WBmFkOcCvwJWAV8JaZPeXuCxvsOtXdv57q7y/trz0+oMlM\nbqzVp7gnS6uqG21JJRJ/eUUF/fPyGFIZrEqpvfvw0srKuBMQm0vEusy7RFWm54PmelaUDzJTc9fJ\nmQPsSbDi7igz69HM/q0xHChz93J3rwT+DpwSZ7/WXWdfMkbstSug/gc0E+y51+7ccuXJjMrPi3v9\nikTiL+7dm2UNruVRBlx5/vmNJqGmromhy7xLVGV6PmjuejbKB5mpyZ4cdy8xs2KCScYXAzeb2QsE\nFwPMT3EsfYHlMV+vICh8GjrWzGYDK4FL3P29FMch7aSlSyxbKhVj+/974iEcU92f8k79djlOIvHH\ndjnvl5tLeVUVfzrvPM4+/vik4mnJJEmRbJIN+aCpnhXlg8zU7MRjdy8HrgKuMrMvEBQ8NcC7Znav\nu/+6jWOMNRMocvetZnYC8CS6C3rWassPaCrH9gt370Zh0YBdtycYf6ovy67LvEsUZU0+aGTIS/kg\nM7XoxjzuPg2YZmajgW8SFDypshKIvYNcv3Bb7PffEvP8OTO73cx6uvu6eAccN25c3fOSkhJKSkpS\nGK6kQlt8QNtzMl6i8bdmLkB7HC8TlZaWUlpamrLjKR9kPuWD5CgfNK7ldx8E3H0bwSqrh5N5fyPe\nAg4Kh8c+BE4Dvhe7g5n1cfc14fPhgDVW4ED9pCaZK9Uf0PaejNcREkw6NCxExo8f36rjKR9kB+UD\niSfZfJBUkdMW3L3azH4GvMDOJeQLzOyC4GW/C/i2mf0EqAQ+Bb6bvoglU7X12L6IZA/lg44tY4oc\nAHd/HhjUYNudMc9vA25r77gku2gynojUUj7o2DKqyBFJFU3GE5Faygcdl4ociSyNjYtILeWDjqm5\niwGKiIiIZCUVOSIiIhJJKnJEREQkklTkiIiISCSpyBEREZFIUpEjIiIikaQiR0RERCJJRY6IiIhE\nkoocERERiSQVOSIiIhJJKnJEREQkklTkiIiISCSpyBEREZFIUpEjIiIikaQiR0RERCJJRY6IiIhE\nUkYVOWZ2vJktNLNFZvabRva52czKzGy2mQ1t7xhFREQkO2RMkWNmOcCtwFeBg4HvmdlnGuxzAnCg\nuw8ALgDuaPdARUREJCtkTJEDDAfK3L3c3SuBvwOnNNjnFOABAHefDnQ3sz7tG6aIiIhkg0wqcvoC\ny2O+XhFua2qflXH2ERERESEv3QG0pXHjxtU9LykpoaSkJG2xiEjLlJaWUlpamrLjKR+IZK9k80Em\nFTkrgaKYr/uF2xrus18z+9SJTWoikl0aFiLjx49v1fGUD0SyV7L5IJOGq94CDjKzYjPrBJwGPN1g\nn6eBMwHM7Bhgg7uvad8wRUREJBtkTE+Ou1eb2c+AFwiKrwnuvsDMLghe9rvc/VkzO9HMPgA+Ac5J\nZ8wiIiKSuTKmyAFw9+eBQQ223dng65+1a1AiIiKSlTJpuEpEREQkZVTkiIiISCSpyBEREZFIUpEj\nIiIikaQiR0RERCJJRY6IiIhEkoocERERiSQVOSIiIhJJKnJEREQkklTkiIiISCSpyBEREZFIUpEj\nIiIikaQiR0RERCJJRY6IiIhEkoocERERiSQVOSIiIhJJKnJEREQkkvLSHQCAmfUAHgGKgaXA/7r7\nxjj7LQU2AjVApbsPb8cwRUREJItkSk/Ob4EX3X0Q8DLwu0b2qwFK3H2YChwRERFpSqYUOacAE8Pn\nE4FvNLKfkTkxi4iISAbLlIKht7uvAXD31UDvRvZz4D9m9paZnddu0YmIiEjWabc5OWb2H6BP7CaC\nouWyOLt7I4f5vLt/aGZ7ERQ7C9x9WopDFRERkQhotyLH3Y9r7DUzW2Nmfdx9jZntDVQ0cowPw38/\nMrMngOFAo0XOuHHj6p6XlJRQUlKSXPAi0u5KS0spLS1N2fGUD9KjkEK2NtpuFUlMsvnA3NP/x2dm\n1wDr3P0aM/sN0MPdf9tgn65AjrtvMbNuwAvAeHd/oZFjeiacm4ikhpnh7pbke5UPRCIk0XyQKUVO\nT+BRYD+gnGAJ+QYz2we4291PMrP9gScIhrLygIfc/U9NHFNJTSRCVOSISK2sKnLagpKaSLSoyBGR\nWonmg0xZXdWhpHKeQTpF5TxA55KponQujYnKOUblPEDnkqmSORcVOWkQlT+6qJwH6FwyVZTOpTFR\nOceonAfoXDKVihwRERGRkIocERERiaRITzxOdwwiklqtmXic6lhEJL069OoqERER6dg0XCUiIiKR\npCJHREREIklFjoiIiESSipw2ZGYTwpuPzonZ9m0zm2dm1WZ2eDrja4lGzuVaM1tgZrPN7DEz2yOd\nMSaqkXO50szeNbNZZvZ8eKPYjBfvXGJe+5WZ1YS3TclojfxOxprZCjN7J3wcn84YW0v5IDMpH2Se\nVOYDFTlt6z7gqw22zQW+CbzS/uG0SrxzeQE42N2HAmXA79o9quTEO5dr3f0wdx8GPAOMbf+wkhLv\nXDCzfsBxBPeCywZxzwP4i7sfHj6eb++gUkz5IDMpH2SelOUDFTltyN2nAesbbHvf3cuApJbCpksj\n5/Kiu9eEX74J9Gv3wJLQyLlsifmyG1BDFoh3LqEbgEvaOZykNXEeWfU5aYryQWZSPsg8qcwHKnIk\nVX4IPJfuIFrDzH5vZsuA04Er0h1Psszs68Byd5+b7lhS4Gfh8Mc9ZtY93cFIwpQPMkRHzwcqcqTV\nzOxSoNLdJ6U7ltZw98vcvQh4CPh5uuNJhpl1AcZQv3s9q3oJYtwOHBAOf6wG/pLmeCQBygeZQ/lA\nRY60kpmdDZxI0NqJiknAqekOIkkHAv2Bd81sCcGQwUwz653WqJLg7h/5zquV3g0clc54pHnKBxmn\nw+eDvLYLSUJG45VztlXU9c4lnN1+CTDC3benLarkNDyXg9z9g/DLbwAL0hJVcurOxd3nAXUrQcLE\ndri7xxvfzjQNfyd7u/vq8MtvAfPSElVqKR9kJuWDzJOSfKDbOrQhM5sElAC9gDUEXYbrgVuAQmAD\nMNvdT0hXjIlq5FzGAJ2Aj8Pd3nT3/0tLgC3QyLl8DRgEVBOsQPixu3+YrhgTFe9c3P2+mNcXA0e6\n+7r0RJiYRn4no4ChBJM+lwIXuPuaNIXYasoHmUn5IPOkMh+oyBEREZFI0pwcERERiSQVOSIiIhJJ\nKnJEREQkklTkiIiISCSpyBEREZFIUpEjIiIikaQiR0RERCJJRY6IiIhEkoocERERiSQVOSKSVcxs\niZl9scG2s8zs1XTFJCKZSUWOiESF7lEjIvWoyBGRSDGzGjM7IObr+8zsypivTzKzWWa23symmdmh\n6YlURNqaihwRiQJLaCezYcAE4DygJ3An8LSZ5bdhbCKSJipyRCQbPWlm62ofwG0Jvu884A53f9sD\nDwLbgWPaLFIRSRsVOSKSjU5x9561D+D/EnxfMfCrmAJpPdAP2LfNIhWRtMlLdwAiIkloanhqK9A1\n5uu9geXh8+XAH9z9j20VmIhkDvXkiEjUzAZON7McMzseGBnz2t3Aj81sOICZdTOzE82sWzoCFZG2\npSJHRLJNc0vFfwF8HVgPfA94ou6N7jMJ5uXcGs7lWQSc1UZxikiambsuLSEiIiLRo54cERERiSQV\nOSIiIhJJKnJEREQkklTkiIiISCSpyBEREZFIUpEjIiIikaQiR0RERCJJRY6IiIhEkoocERERiaT/\nB0rmntZpRNZRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2564915f400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x_min = X_train[:, 0].min() - 1\n",
    "x_max = X_train[:, 0].max() + 1\n",
    "y_min = X_train[:, 1].min() - 1\n",
    "y_max = X_train[:, 1].max() + 1\n",
    "\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),\n",
    "                     np.arange(y_min, y_max, 0.1))\n",
    "\n",
    "f, axarr = plt.subplots(nrows=1, ncols=2, \n",
    "                        sharex='col', \n",
    "                        sharey='row', \n",
    "                        figsize=(8, 3))\n",
    "\n",
    "\n",
    "for idx, clf, tt in zip([0, 1],\n",
    "                        [tree, bag],\n",
    "                        ['Decision Tree', 'Bagging']):\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    axarr[idx].contourf(xx, yy, Z, alpha=0.3)\n",
    "    axarr[idx].scatter(X_train[y_train == 0, 0],\n",
    "                       X_train[y_train == 0, 1],\n",
    "                       c='blue', marker='^')\n",
    "\n",
    "    axarr[idx].scatter(X_train[y_train == 1, 0],\n",
    "                       X_train[y_train == 1, 1],\n",
    "                       c='red', marker='o')\n",
    "\n",
    "    axarr[idx].set_title(tt)\n",
    "\n",
    "axarr[0].set_ylabel('Alcohol', fontsize=12)\n",
    "plt.text(10.2, -1.2,\n",
    "         s='Hue',\n",
    "         ha='center', va='center', fontsize=12)\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/bagging_region.png',\n",
    "#            dpi=300,\n",
    "#            bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Notice the smoother decision boundaries of bagging than decision tree.\n",
    "\n",
    "Bagging can reduce variance, but not bias.\n",
    "\n",
    "Choose base classifiers with low bias, such as unpruned decision trees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Leveraging weak learners via adaptive boosting\n",
    "\n",
    "AdaBoost for abbreviation\n",
    "\n",
    "A collection of weak base learners (diversity over accuracy)\n",
    "* sequential learning\n",
    "* later classifiers focus on the weak parts (mis-classified samples) of the earlier classifiers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Original boosting procedure [Schapire 1990]\n",
    "\n",
    "Using 3 classifiers\n",
    "\n",
    "Training:\n",
    "* Train $C_1$ with a random subset of training data\n",
    "\n",
    "* Train $C_2$ with a random subset of training data $\\bigcup$ misclassified samples from $C_1$\n",
    "\n",
    "* Train $C_3$ over the training samples for which $C_1$ and $C_2$ disagree\n",
    "\n",
    "Testing/Usage:\n",
    "* Combine $C_1$, $C_2$, $C_3$ via majority voting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## AdaBoost versus bagging\n",
    "\n",
    "Bagging\n",
    "* parallel\n",
    "* random sample with replacement\n",
    "* cannot reduce bias; same as base classifiers\n",
    "\n",
    "AdaBoost\n",
    "* sequential\n",
    "* random sample without replacement\n",
    "* can reduce bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## AdaBoost example\n",
    "\n",
    "Give previously misclassified samples higher weights\n",
    "\n",
    "Example using decision tree stumps (i.e. very shallow trees, like just one level)\n",
    "\n",
    "Steps:\n",
    "1. equal weight training of all samples by $C_1$, two blue circles are misclassified\n",
    "2. larger/lower weights to wrongly/correctly classified samples, train $C_2$\n",
    "3. larger/lower weights to wrongly/correctly classified samples, train $C_3$\n",
    "4. combine $C_1$, $C_2$, $C_3$ for weighted majority voting\n",
    "\n",
    "<img src='./images/07_09.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## AdaBoost algorithm\n",
    "\n",
    "Input data: $X$ and $y$ (label)\n",
    "\n",
    "Initial equal weight vector $w$ to all samples, $\\sum_i w_i = 1$\n",
    "\n",
    "For j in m boosting rounds, do the following:\n",
    "1. Train a weak learner: $C_j.train(X, y, w)$\n",
    "2. Predict class labels: $\\hat{y} = C_j.predict(X)$\n",
    "3. Compute weighted error rate: $\\epsilon = w . \\left(\\hat{y} \\neq y \\right)$, $0 \\leq \\epsilon < 0.5$\n",
    "4. Compute re-weighting coefficients: $\\alpha_j = 0.5 \\log_e\\left(\\frac{1-\\epsilon}{\\epsilon}\\right)$ $\\geq0$ for $0 \\leq \\epsilon < 0.5$ \n",
    " * $\\epsilon \\uparrow$ $\\rightarrow$ $\\alpha_j \\downarrow$\n",
    " * $\\epsilon = 0$ $\\rightarrow$ $\\alpha_j = \\infty$ - equal weighting if no error\n",
    " * $\\epsilon = 0.5$ $\\rightarrow$ $\\alpha_j = 0$ - no weight update if serious errors\n",
    "5. Update weights: $w \\leftarrow w \\times \\exp(-\\alpha_j . \\hat{y} \\times y)$, $\\times$ means element-wise product\n",
    " * correct/incorrect prediction will decrease/increase weight\n",
    "6. Normalize weights: $w \\leftarrow \\frac{w}{\\sum_i w_i}$\n",
    "\n",
    "Compute final prediction via weighted ensemble voting:\n",
    "$\n",
    "\\hat{y} = \\sum_j \\alpha_j \\times C_j.predict(X)\n",
    "$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHXpJREFUeJzt3XmUVdWZ9/HvIwgasR3jLKKIY5wNgqKUOAAaAY1jBhOj\nvmrMq9GsXkZjIqtjd8d0pzUuTYxpY9oYHJYvooIoKpZDwCEoahuNOBFUJNpIBFEG2e8fu8qqJhRV\nQN065977/ax1FvdWHc59PKuon/ucZ+8TKSUkSSqjtYouQJKkthhSkqTSMqQkSaVlSEmSSsuQkiSV\nliElSSqtioZURGwTEZMj4sWIeCEizlvBPoMjYl5EPNO0XVrJmiRJ1aN7hY+/FLgwpTQ9InoB0yJi\nUkrp5eX2ezSlNKLCtUiSqkxFR1IppXdTStObXi8AXgK2XsGuUck6JEnVqcvuSUVEH2Bv4MkVfHtg\nREyPiAkRsVtX1SRJKrdKX+4DoOlS3x3A+U0jqtamAb1TSgsjYjgwDtipK+qSJJVbVHrtvojoDowH\nJqaUft6B/d8A9kspzV3u6y4yKElVKqW0Wrd1uuJy32+AP7UVUBGxeavX/cnBOXdF+6aU3NrZLrvs\nssJrqIbN8+Q58jx13bYmKnq5LyIOAr4KvBARzwIJuATYDkgppeuB4yPiHGAJ8DFwUiVrkiRVj4qG\nVErpD0C3dva5Fri2knVIkqqTK07UmIaGhqJLqAqep/Z5jjrG81RZFW+c6CwRkaqlVklSi4gglbhx\nQpKk1WJISZJKy5CSJJWWISVJKi1DSpJUWoaUJKm0DClJUmkZUpKk0jKkJEmlZUhJkkrLkJIklZYh\nJUkqraoPqZTgggtg8eKiK5EkdbaqD6kIeOgheO65oiuRJHW2qg8pgIEDYerUoquQJHU2Q0qSVFqG\nlCSptGoipHbaCebPh9mzi65EktSZaiKkImDAAEdTklRraiKkwEt+klSLDClJUmlFSqnoGjokItLK\nap0/H7bcEubOhR49urAwSdJKRQQppVidv1szI6n114e+feHZZ4uuRJLUWWompMBLfpJUawwpSVJp\nGVKSpNKqqZDq1w8WLoS33y66EklSZ6ipkHJSryTVlpoKKYADDzSkJKlW1FxIeV9KkmpHzUzmbbZg\nAWy+eZ7U27NnFxQmSVopJ/O20qtXXhXdSb2SVP1qLqTAS36SVCsMKUlSaRlSkqTSqsmQ6tsXPvkE\n3nqr6EokSWuioiEVEdtExOSIeDEiXoiI89rY7+qImBER0yNi7zX/XEdTklQLKj2SWgpcmFLaHRgI\nnBsRu7TeISKGA31TSv2As4DrOuODDSlJqn4VDamU0rsppelNrxcALwFbL7fbSOCmpn2eBDaIiM3X\n9LMNKUmqfl12Tyoi+gB7A08u962tgVmt3r/N3wfZKvviF+H552HRojU9kiSpKF0SUhHRC7gDOL9p\nRFVx660HO+8MzzzTFZ8mSaqE7pX+gIjoTg6o36WU7lrBLm8D27Z6v03T1/7O6NGjP3vd0NBAQ0PD\nSj+7+ZLfwIGrVrMkafU1NjbS2NjYKceq+Np9EXET8H5K6cI2vn8UcG5K6eiIGABclVIasIL9OrR2\nX2s33wzjxsEdd6xO5ZKkzrAma/dVNKQi4iDgUeAFIDVtlwDbASmldH3TftcAw4CPgNNSSn93kW51\nQuq11+CQQ/J8qVit0yNJWlOlDanOtDohlVJeEf2Pf4TevStUmCRppVwFvQ3Nk3qnTCm6EknS6qjp\nkAI46igYO7boKiRJq6OmL/cBzJsHffrA66/Dxht3fl2SpJXzct9KbLhhHk2NGVN0JZKkVVXzIQVw\n2mnw298WXYUkaVXVRUgNGQJz5sALLxRdiSRpVdRFSHXrBt/4hqMpSao2Nd840WzGDBg0KE/sXXvt\nTixMkrRSNk50QL9+sNNOMHFi0ZVIkjqqbkIK4JvfhBtvLLoKSVJH1c3lPoD582HbbeGVV2CzzTqp\nMEnSSnm5r4PWXx9GjHDOlCRVi7oKKchzpm68MS8+K0kqt7oLqcGD4cMPYfr0oiuRJLWn7kJqrbXy\nnCkbKCSp/OqqcaLZG29A//55zlTPnp1ySElSG2ycWEXbbw9f+AKMH190JZKklanLkII8Z8plkiSp\n3Orych/ARx/BNtvASy/BFlt02mElScvxct9qWG89OPZY+N3viq5EktSWug0pgNNPh+uvh6VLi65E\nkrQidR1SBx6Yl0ny3pQklVPd3pNq9sQTcOKJeT2/ddbp9MNLUt3zntQaGDAA9t0XfvnLoiuRJC2v\n7kdSAP/933DYYfDqq3kRWklS53EktYa+8AU48ki48sqiK5EkteZIqsnrr+elkl5+GTbdtGIfI0l1\nx5FUJ9hhh9xA8ZOfFF2JJKmZI6lWZs/Ol/6eey6vRiFJWnNrMpIypJZz0UUwbx786lcV/yhJqguG\nVCeaOxd23hmmTIF+/Sr+cZJU87wn1Yk23hi++1247LKiK5EkOZJagQUL8ijqvvtgr7265CMlqWY5\nkupkvXrBxRfDpZcWXYkk1TdDqg1nnZVXonjggaIrkaT6ZUi1oWfP3OF3xhnw4YdFVyNJ9cl7Uu04\n6yxYtgx+/esu/2hJqgm2oFfQ/Pmwxx5w3XUwbFiXf7wkVb3SNk5ExA0RMScinm/j+4MjYl5EPNO0\nla5VYf314YYb4Mwz8yRfSVLXqehIKiIGAQuAm1JKe67g+4OB76WURnTgWIWMpJqdey4sXAg33lhY\nCZJUlUo7kkopPQ580M5uq1V4V7viCnjkERg/vuhKJKl+lKG7b2BETI+ICRGxW9HFtKVXL/jNb+Ds\ns+GD9mJXktQpig6paUDvlNLewDXAuILrWamGBjjuODj//KIrkaT60L3ID08pLWj1emJE/CIiNk4p\nzV3R/qNHj/7sdUNDAw0NDRWvcXn/+q95qaS77oKRI7v84yWp9BobG2lsbOyUY1W8BT0i+gD3pJT2\nWMH3Nk8pzWl63R+4PaXUp43jFNo40dpjj8FJJ8ELL8AmmxRdjSSVW2nnSUXEGKAB2ASYA1wG9ABS\nSun6iDgXOAdYAnwMXJBSerKNY5UmpAAuvBBefRXGjYO1ir5oKkklVtqQ6kxlC6nFi+Gww+DQQ+Gf\n/qnoaiSpvErbgl7LevSAO+6A//qv/KckqfM5klpDzzwDQ4fCQw/Bnn83XVmS5EiqQPvuC1dfDaNG\nwfvvF12NJNUWR1Kd5Pvfh6eegvvvh7XXLroaSSoPGydK4NNP4ZhjYMcd88hKkpR5ua8EunWDMWPy\nSOqGG4quRpJqgyOpTvbyy3DIIXn+1IEHFl2NJBXPkVSJ7LJLfpzH8cfDK68UXY0kVTdDqgKOPhp+\n/GM44giYObPoaiSpehW6wGwtO/10WLAADj8cHn0Uttyy6IokqfoYUhV0/vk5qI44AhobYdNNi65I\nkqqLjRMVlhJccglMmgSTJ8MGGxRdkSR1LedJlVxKcN558OyzuUV9vfWKrkiSuo4hVQWWLcv3qd56\nC+65B9ZZp+iKJKlrGFJV4tNP4ZRTYNGivHK6yydJqgfOk6oS3brBzTfn18ceCwsXFluPJJWdIdXF\nmp9DtdFG+REf8+YVXZEklZchVYC1184PS9x3Xxg8GGbPLroiSSqnDoVURGwXESMiYlDT++MqW1bt\nW2stuOoqOOEEGDQIXnut6IokqXw6Opn3c8DxQP+IWABMAcZWrKo6EQGXXgqbbJIXpb33Xthrr6Kr\nkqTy6GhIjQK+m1KaGxHrAkMqWFPdOeecHFRHHAFjx+aRlSSp4/ek3kopzQVIKX1cwXrq1okn5s6/\nY4+Fu+8uuhpJKoeOhtR7EXFLRBwTEXsCu1ayqHp15JEwfjycfTb89Kd5pQpJqmcdnswbETsB3wB6\nAL9OKXXp05JqYTJvR82aBaNGwa67wq9/DeuuW3RFkrT6XHGiBi1cCKedBm+8kZ/yu9VWRVckSavH\nFSdq0Oc+B7feCiNHQv/+8NRTRVckSV3PkVQVuOsuOOMMuPJK+NrXiq5GklaNl/vqwAsv5FHV8cfD\nv/wLdPdxlZKqhCFVJ95/P6+ivngxjBkDW29ddEWS1D7vSdWJTTeF++7Lrer77ZdXqJCkWuZIqko9\n9hh85St5ZPXP/+yzqSSVlyOpOnTwwflx9H/6U1737803i65IkjqfIVXFNt00L6F0/PG5Tf3OO4uu\nSJI6l5f7asSTT8LJJ+cHKf77v0OvXkVXJEmZl/vEAQfA9Om582+vveDRR4uuSJLWnCOpGjR+PJx1\nFpx0Um6qcO0/SUVyJKX/5Utfguefh3fegX32cUklSdXLkVSNu+02OO88OPNM+NGPoEePoiuSVG9K\nO5KKiBsiYk5EPL+Sfa6OiBkRMT0i9q5kPfXopJPguefyyGr//eGJJ4quSJI6rtKX+24Ehrb1zYgY\nDvRNKfUDzgKuq3A9dWmLLfIitZdcAscdB+eeC3/7W9FVSVL7KhpSKaXHgQ9WsstI4KamfZ8ENoiI\nzStZU72KyC3qL74IS5bAbrvBHXf49F9J5VZ048TWwKxW799u+poqZKON4Prr872qyy6DY46BmTOL\nrkqSVqzokFJBBg3KyyoNHJgXq/3Zz/IIS5LKpOinEr0NbNvq/TZNX1uh0aNHf/a6oaGBhoaGStVV\nF3r0gB/8AE48Md+n+s1v4Oc/h8MPL7oySdWssbGRxsbGTjlWxVvQI6IPcE9KaY8VfO8o4NyU0tER\nMQC4KqU0oI3j2IJeQSnl5ooLL8wrVvzsZ7DDDkVXJakWlLkFfQwwBdgpIv4SEadFxFkR8X8AUkr3\nAm9ExKvAr4BvV7IetS0CRo3Kq6p/8Yt5wdpLL4WPPiq6Mkn1zMm8WqG33oKLLsprAP70p7kzMFbr\n/4Mk1TsfH6+KefzxvGJFz57wb/+WGy4kaVWU9nKfqt+gQfD003DOOfDVr8Kxx8LLLxddlaR6YUip\nXd26wamnwp//DAcemJ8KfPbZ8O67RVcmqdYZUuqwddaBf/zHPJJabz3YfXcYPRrmzy+6Mkm1ypDS\nKttkk9yiPm0avPoq9OsHV10Fn3xSdGWSao0hpdXWpw/cfDPcfz88/DDsuCP88pf56cCS1BkMKa2x\nvfbKE4HHjs1/7rwz3HgjLF1adGWSqp0t6Op0jz+eJwLPnp3vWZ14Ym6+kFSfnCel0kkJJk+GH/4Q\nPvggP8vqlFOge9GrRUrqcoaUSqs5rH78Y5g1Cy6+OLez+xh7qX4YUqoKjz0Gl1+eW9gvugi+9a3c\n1i6ptrnihKrCwQfnTsDbb4eJE6Fv39zK7jwrSW0xpNTlDjgA7rkHxo/PSy5tv31+rtWcOUVXJqls\nDCkVZp994NZb4cknYd482HXXvNzSjBlFVyapLAwpFa5vX7j22nyvarPN8vqAJ5yQR1mS6puNEyqd\nBQvghhvgP/4DeveGCy6AkSOdayVVK7v7VJOWLoU774Qrr8wrrp93Xu4I/Id/KLoySavC7j7VpO7d\n82W/KVNgzBiYOjU3WVx4Ibz5ZtHVSeoKhpSqwoABcNtt8Oyz+bLffvvBccflicIOsKXa5eU+VaUF\nC+B3v4Nrrsnvv/Md+PrXoVevYuuS9Pe8J6W6lVJ+TMg118Ajj+Sg+va3Yaediq5MUjPvSaluRcCQ\nIfkxIc8+C+uuC4MGwbBhMG6cjwuRqp0jKdWcjz+GO+6A666DmTPhjDPyts02RVcm1SdHUlIr666b\nL/v94Q9w773w3nuw554walReO3DZsqIrlNRRjqRUFxYsgFtuyY+3nzcPTj8dTjsNttqq6Mqk2udI\nSmpHr15w5pkwbVpuZf/LX2D33WHECLj7bu9dSWXlSEp1a8GC/NiQ//zPfO/qm9/MK1r07Vt0ZVJt\ncSQlrYZevXIoTZkCkyblhosBA3K34E03wUcfFV2hJEdSUiuLFuVnXd14Yw6vL38537s68MDc7i5p\n1TmZV6qAd96Bm2/OgfXpp/ly4Kmn2sourSpDSqqglPKDGX/723wPa999c1gdd5zLMEkdYUhJXeST\nT/LlwJtugsceg2OOyYE1ZIjPu5LaYkhJBfjrX+HWW3NgzZ4NX/1q3vbc0/tXUmuGlFSwP/0pr8o+\nZgysv34Oq698BbbbrujKpOIZUlJJLFuWl2MaMyavH7jLLjmwTjgBNtmk6OqkYhhSUgktXpzXCvz9\n72HiRDjkEDj5ZBg50oYL1RdDSiq5+fPz8ku33JIbLoYOzYE1fHheEFeqZYaUVEXmzs3Pv7rlFnjm\nmbx+4EknweGHQ48eRVcndb5Sh1REDAOuIi/BdENK6Yrlvj8YuAt4velLY1NKl6/gOIaUas7s2fne\n1S23wJ//nB8ncuKJuaV97bWLrk7qHKUNqYhYC3gFOAx4B3gaODml9HKrfQYD30spjWjnWIaUatqs\nWTmwbr8dZsyAY4/NgXXoodC9e9HVSauvzAvM9gdmpJRmppSWALcCI1ewn7NKVPe23RYuuACmTs2P\nFNllF7j00vzMqzPPzE0YS5YUXaXUtSodUlsDs1q9f6vpa8sbGBHTI2JCROxW4Zqk0ttuO/je9/Jy\nTE89lQNr9GjYYou84O2ECXkxXKnWleFRHdOA3imlvYFrgHEF1yOVSp8+ObCmToXnnoN99oErrsiB\n9bWvwZ13wsKFRVcpVUalr3S/DfRu9X6bpq99JqW0oNXriRHxi4jYOKU0d/mDjR49+rPXDQ0NNDQ0\ndHa9Uqltsw2cd17e3n03dwlee21eof3ww/Oit0cfDRtuWHSlqmeNjY00NjZ2yrEq3TjRDfgzuXFi\nNvAUcEpK6aVW+2yeUprT9Lo/cHtKqc8KjmXjhNSG//mfvPDt2LHQ2AgHHZQDa+RI2GyzoqtTvStt\ndx981oL+c1pa0H8SEWcBKaV0fUScC5wDLAE+Bi5IKT25guMYUlIHzJ+fV7gYOxbuuw/22CO3to8a\nBX37Fl2d6lGpQ6qzGFLSqlu0CCZPhnHj4K674POfz63to0ble1uu1q6uYEhJateyZblb8M4787Zo\nUb4cOGIEDB7saheqHENK0ipJCV56KY+u7rorr3YxbFgOrOHDbbxQ5zKkJK2Rd9/NjRd33w2PPAL9\n++fAOuYY2H77oqtTtTOkJHWajz6CBx7IgTVhQr6PdcwxeTvgAOjWregKVW0MKUkV8emnecWLe+7J\n25w5cNRRObCOPDI/hVhqjyElqUu8+WZLYE2dCgMGwJe+lCcQ77hj0dWprAwpSV1u/nx48MF8SXDC\nBNhgg5bAGjTIR42ohSElqVDLluUHOE6YAOPHw6uv5mWajjoqdwtusUXRFapIhpSkUnn33bzaxYQJ\nebS14445sI4+GvbfH9Yqw9LW6jKGlKTSWrIE/vAHuPfeHFrvvQdDh+YR1tChsMkmRVeoSjOkJFWN\nmTPz2oL33psXw9199xxYw4fDfvs5yqpFhpSkqrRoETz2WA6siRPzau5Dh+bVL448Ms/RUvUzpCTV\nhDfegPvvz/ezHn4Ydt45B9awYXkVjO6VfgKeKsKQklRzFi/O97Luuy9vs2bljsFhw/Joa+uti65Q\nHWVISap5b78NkyblkdYDD8CWW7YE1sEHwzrrFF2h2mJISaorn34Kf/xjDqz774cXXsgTiIcOzfey\ndtnFZ2WViSElqa598AE89FDLSCulHFZHHgmHHWabe9EMKUlqkhK88koOq0mT4NFHYdddW0JrwACX\nbOpqhpQktWHRIpgyJd/HmjQJZszITyJuDq1+/bw0WGmGlCR10HvvtVwafOCB/HysI47Im5cGK8OQ\nkqTVkBK89FIOqwceyBOL+/XLre5HHAEHHWTXYGcwpCSpEyxeDE880RJaL74IBx7YElp77umyTavD\nkJKkCpg3L6988eCDefvgg3xJsDm0evcuusLqYEhJUheYOTPfz2oOrQ03zIF1+OFw6KGw0UZFV1hO\nhpQkdbFly+D553NoPfQQPP54XmuwObS8n9XCkJKkgjXfz3rwwRxazz8PBxzQcnlw331zJ2E9MqQk\nqWQ+/BAeeaTl8uA77+T5WYcdlrd6WrrJkJKkknv3XZg8uWWktXQpDBnSElrbblt0hZVjSElSFUkJ\nXnut5X7Www/npovmwGpogE03LbrKzmNISVIVa27CmDw5h9Zjj0HfvjmwhgzJjyJZf/2iq1x9hpQk\n1ZAlS+Dpp3NgTZ6cX++1Vw6sIUNg4MDq6hw0pCSphn38cX5K8eTJeXvxxdw52Bxa++8P3bsXXWXb\nDClJqiN/+1t+BElzaM2cmS8JNofWHnuUa/kmQ0qS6th770FjY8s9rblz8woYQ4bkP3feudh2d0NK\nkvSZt97KHYPNodXc7t689enTtfUYUpKkFUoJXn+95dLgww/Duuu2BNahh8JWW1W2BkNKktQhzc/Q\nag6sxkb4/OdbQqsSc7QMKUnSalm2DJ57rmWk9fjjsN12LaOswYPzau9rotQhFRHDgKuAtYAbUkpX\nrGCfq4HhwEfAN1NK01ewjyElSRW2dClMm9Yy0po6NTdeNDdiDBq06hOL1ySkKtqkGBFrAdcAQ4Hd\ngVMiYpfl9hkO9E0p9QPOAq6rZE21rrGxsegSqoLnqX2eo46ptfPUvXueg3XxxTBpErz/Plx5JfTq\nBVdcAVtumZ9W/IMf5KaMjz+ubD2V7qTvD8xIKc1MKS0BbgVGLrfPSOAmgJTSk8AGEbF5heuqWbX2\nD6ZSPE/t8xx1TK2fp5498xysyy7L96/eew8uvzy3tP/oR/l+1uDB8PvfV+bzKz1HeWtgVqv3b5GD\na2X7vN30tTmVLU2StKpadwYCLFiQ72P17FmZzyvxQhqSpLLr1QuGDavc8SvaOBERA4DRKaVhTe+/\nD6TWzRMRcR3wcErptqb3LwODU0pzljuWXROSVKVWt3Gi0iOpp4EdI2I7YDZwMnDKcvvcDZwL3NYU\navOWDyhY/f9ASVL1qmhIpZQ+jYjvAJNoaUF/KSLOyt9O16eU7o2IoyLiVXIL+mmVrEmSVD2qZjKv\nJKn+lGgx9ywihkXEyxHxSkRc1MY+V0fEjIiYHhF7d3WNRWvvHEXEzhExJSI+iYgLi6ixDDpwnr4S\nEc81bY9HxB5F1Fm0DpynEU3n6NmIeCoiDiqiziJ15PdS035fjIglEXFcV9ZXFh34WRocEfMi4pmm\n7dJ2D5pSKs1GDs1Xge2AtYHpwC7L7TMcmND0+gDgiaLrLuE52hTYD/gxcGHRNZf4PA0ANmh6Paze\nfpZW4Tx9rtXrPYCXiq67bOeo1X4PAeOB44quu4znCRgM3L0qxy3bSMrJv+1r9xyllN5PKU0DlhZR\nYEl05Dw9kVL6W9PbJ8jz8+pNR87TwlZvewHLurC+MujI7yWA/wvcAfy1K4srkY6ep1VqgitbSK1o\n8u/yvzjamvxbLzpyjrTq5+kMYGJFKyqnDp2niBgVES8B9wDf6qLayqLdcxQRWwGjUkq/ZBV/CdeQ\njv6bG9h0q2ZCROzW3kGdzKu6FxGHkrtKBxVdS1mllMYB4yJiEHA5cETBJZXNVUDrezD1GlTtmQb0\nTiktbFq3dRyw08r+QtlGUm8DvVu936bpa8vvs207+9SyjpwjdfA8RcSewPXAiJTSB11UW5ms0s9T\nSulxYIeI2LjShZVIR87R/sCtEfEGcDxwbUSM6KL6yqLd85RSWtB8+TilNBFYu72fpbKF1GeTfyOi\nB3ny793L7XM3cCp8tqLFCif/1rCOnKPW6vX/6No9TxHRG/h/wNdTSq8VUGMZdOQ89W31el+gR0pp\nbteWWah2z1FKaYembXvyfalvp5RW9u+yFnXkZ2nzVq/7k6dBrfRnqVSX+5KTf9vVkXPU9IPwR2B9\nYFlEnA/sllJaUFzlXasj5wn4IbAx8IuICGBJSmn5BZBrWgfP05cj4lRgMfAxcGJxFXe9Dp6j//VX\nurzIEujgeTo+Is4BlpB/lk5q77hO5pUklVbZLvdJkvQZQ0qSVFqGlCSptAwpSVJpGVKSpNIypCRJ\npWVISZJKy5CSJJWWISVJKq1SLYsk1aOIGAXsCkxvWnRTUhOXRZIKFBH9gF+llIYUXYtURl7uk4o1\nDHgnIk5tWpVdUite7pOKNQ+4LaV0T9GFSGXk5T6pQBHREzgfeBmYlVJ6tuCSpFIxpCRJpeU9KUlS\naRlSkqTSMqQkSaVlSEmSSsuQkiSVliElSSotQ0qSVFqGlCSptP4/G0ZVjJ0qfZwAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x256492280b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def alpha_func(x):\n",
    "    return 0.5*np.log((1-x)/x)\n",
    "\n",
    "delta = 0.01\n",
    "x = np.arange(delta, 0.5, delta)\n",
    "y = alpha_func(x)\n",
    "\n",
    "plt.plot(x, y)\n",
    "plt.xlabel(r'$\\epsilon$')\n",
    "plt.ylabel(r'$\\alpha$')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "10 sample example\n",
    "\n",
    "$\\epsilon = 0.3$\n",
    "\n",
    "$\\alpha = 0.5\\log_e(0.7/0.3) = 0.424$\n",
    "\n",
    "$w$ before normalization:\n",
    "* correct prediction: $0.1 \\times \\exp(-0.424 \\times 1 \\times 1) = 0.066$\n",
    "* incorrect prediction: $0.1 \\times \\exp(-0.424 \\times 1 \\times -1) = 0.153$\n",
    "\n",
    "<img src='./images/07_10.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# AdaBoost code example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "\n",
    "tree = DecisionTreeClassifier(criterion='entropy', \n",
    "                              max_depth=1,\n",
    "                              random_state=0)\n",
    "\n",
    "ada = AdaBoostClassifier(base_estimator=tree,\n",
    "                         n_estimators=500, \n",
    "                         learning_rate=0.1,\n",
    "                         random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree train/test accuracies 0.845/0.854\n",
      "AdaBoost train/test accuracies 1.000/0.875\n"
     ]
    }
   ],
   "source": [
    "tree = tree.fit(X_train, y_train)\n",
    "y_train_pred = tree.predict(X_train)\n",
    "y_test_pred = tree.predict(X_test)\n",
    "\n",
    "tree_train = accuracy_score(y_train, y_train_pred)\n",
    "tree_test = accuracy_score(y_test, y_test_pred)\n",
    "print('Decision tree train/test accuracies %.3f/%.3f'\n",
    "      % (tree_train, tree_test))\n",
    "\n",
    "ada = ada.fit(X_train, y_train)\n",
    "y_train_pred = ada.predict(X_train)\n",
    "y_test_pred = ada.predict(X_test)\n",
    "\n",
    "ada_train = accuracy_score(y_train, y_train_pred) \n",
    "ada_test = accuracy_score(y_test, y_test_pred) \n",
    "print('AdaBoost train/test accuracies %.3f/%.3f'\n",
    "      % (ada_train, ada_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Better training performance by AdaBoost, but with more overfitting than bagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjkAAADnCAYAAAD8QEcbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPX1//HXCQlhUREMoIIERcQVxIVKbSH0W+tWa1tt\na7V1qdLt6w+Xat0NWGvr8lVxq1VRca3WrcWqdSMgrYJikEUQLBAQkUhZXYAkc35/3JkwCZPkZjLJ\nTC7v5+MxD2e5c+fcCXM8n+1ec3dEREREoiYv2wGIiIiItAYVOSIiIhJJKnJEREQkklTkiIiISCSp\nyBEREZFIUpEjIiIikaQiR7LGzF4ws5+G2G6jmfVv/YhEJJvM7AwzeyPbcUh0qMiRRpnZUjP7wszW\nm9kaM5tmZr8wM2vpvt39OHd/OMR2O7r70pZ+XrJ44bQhfquJH2PiuR9n8rNEBMysLJ5DCprYNNTJ\n28ys1My2JP2O55nZ9zMQamOfqSKsnVGRI01x4Hh37wYUA38ELgEmZDWqFooXTju5+05ABcExJp57\nvP72Ztah7aMUiQYzKwa+BsSA72Rw139J+h1fADxiZj0zuP/6jJBFmOQGFTkShgG4+0Z3fx74EXCG\nme0PYGYdzewmM6sws5VmdpeZFda+2exEMyuP9wYtMrNvxZ+fbGY/i98fEG/prTOzSjN7POn9MTPb\nK35/JzN7KL7NEjO7Imm7M8zsDTO7Md5i/I+ZHRPy+Or0TJnZ78zsL2b2mJmtB06zwOVm9mH88x8z\ns25J7znSzN40s7Vm9q6Zfb3Z37RINJ0OvAk8CJyZeNLMepjZ3+O54S1gQPKbzOxWM1sWf/1tM/ta\nQx/g7i8DG5P3YWaj4zlntZk9Z2a7Jb32VTObEf+9Tjez4UmvnRnPHxvi//2xme0L/AkYHu/1XdPS\nL0Van4ocaTZ3fxv4CEj8T/x6YG9gcPy/fYCrAcxsGDAR+E28N2gEsDTFbn8H/NPddwb6Arcnf2TS\n/TuAHYH+QAlwupmdlfT6MGA+sAtwIy3rcfou8Eg87ieAC4FjCVqkfYHPEnGa2R7A34Cr3L07cCnw\njJl1b8Hni0TF6cAjwGPA0Um9LXcBXwC9gbOBn9V73wyCvNI9/t6/mlnHVB9gZscDBcD78cffAK4D\nTgZ2A5YBf4m/1h14HriVIFfcAvzDzLqbWRdgPHB0vIfoq8Asd18A/BJ4M97r26NF34i0CRU5kq6P\ngcSPfDRwgbuvd/fPCYa0EvNafgZMcPfXAdx9pbsvTLG/KqDYzPq4+xZ3/3fSawZgZnkEvUiXuvsX\n7l4B/B+QPHm5wt3v9+CibBOBXc2sV5rHOM3dX4jHvRn4BXC5u3/i7lsICrMfxLf9KfA3d381vv3L\nwHtAmJ4kkciK9770A55093eBD4FT47/n7xM0DDa5+zyC32wtd3/M3de5e8zdbwEKgUFJm/wo3mv7\nGfAccJ27b4i/dipB7nnP3auAy4AjzKwfcDywML7/mLv/BVgAnBB/bw1wkJl1cvdV7j4/89+MtAUV\nOZKuPsCaeIusCzAznmzWAC8StI4A9gD+E2J/FxP8e5xhZnPq9c4kFAH5BC2yhIp4LAmfJO64+5cE\nBdIO4Q5pG8vrPe4HTEo6ztlALF5EFRMk7jXx21rgK8DuaX62SFScDrzs7mvjjx8HzgB6EvyeP0ra\ntiL5jWZ2kZm9Hx9SWgvsRJAHEp5w9x7uvgPBMNUZZjY6/truyfuLN8DWEOSLOq8lfXYfd/+CoDH1\nK2ClmU0ys0FIu5Sf7QCk/TGzwwmSxBvAaoLu5gPcfWWKzZdTb5w9FXevBH4e3/+RwKtmNsXdFydt\ntpp4jw9Bq4v4/RVpHkqTYdV7vBw4NT5cV4eZLQfud/f/baVYRNodM+sE/BDIM7NEfigEuhEMUVUR\nNIQSvbv9kt77dYLGzyh3TwxBraHe/LkEd19mZi8S9MbcS9DbXJy0v64Eja8V8ddOqreLfgQNNNz9\nFeCV+NzC3wP3ACPRpON2Rz05EpqZ7Whm3yZoiT3s7u/Hh4XuBW5NjLObWZ/E5GKCOTFnmdmo+MTd\n3c1snxT7PtnMEj0y6whWYcSSt3H3GPAk8Hsz28GCFRsXAE0uQ8+QPwN/iM+/wcx6mVmie/th4Htm\n9k0zyzOzTmZWYma7tlFsIrnoe0A1sB8wJH7bl6CBdDrwDDDOzDpbsJDhjKT37kBQBP3XgsUNVxPM\nx0tWW/CYWV+C4eG58aceJ8g9g+PFynXAW+6+DHgBGGhmp5hZBzP7UTzG5+O/6+/E5+ZUEcy9S+Si\nVUBfa3oZvOQIFTkSxiQLVhgtIxjXvom6EwQvIRhnf8vM1gEvA/tA7STlswgm+K0HytjaukpuFR0O\nTDezDQRj62OSzo2TvN0Ygp6jxcBUgonBDzQSe5iWV9jW2f8RtPRei38f04DDAOLzg74HXAV8SjC5\n+kL0G5Pt2+kEPZwr3L0ycQPuJJgzcy5BMbMSuD9+S/hn/LYQWELwu68/hPzD+AqoDcB0guLpGgB3\nf43g9/gMQe/NnsAp8dfWAN8GLiLoIb6I4DQSawh+sxfG37OaYLHEr+Kf9zowD/jEzCpb/O1Iq7Og\nIS4iIiISLWplioiISCTlTJFjZn3N7HULTs09x8zGpNhmpAUni3s3frsyG7GKiIhI7sul1VXVwIXu\nPsvMdiBYkvxy/ARMyaa6eyZPCy4iIiIRlDM9OfETrM2K3/+M4Ky1fVJs2uILQ4qIiEj05VJPTi0z\n6w8cTDBbvr7hZjaLYOb7xYnzJ6TYh2ZUi0SMu6fVyFE+EImeMPkgZ3pyEuJDVU8B58V7dJLNBPq5\n+8EE1zB6rrF9lZaW1t4mT56Mu+fErbS0NOsx6Dh0LLl+mzx5MiNHjqz9DbeU8oGOQ8fSfm/p5oOc\n6skxs3yCAudhd/9b/dc9qehx9xctuNp1Dw/ObbCNsWPHtlqsItK6SkpKKCkpqf0djxs3rkX7Uz4Q\nab/SzQe51pNzP/C+u49P9aKZ9U66P4zgPD+63L2IiIhsI2d6cuLXKzoNmGNm5QRnob2c4Oy47u73\nACeb2a8ITrX9JcFF1NqdkpKSbIeQEVE5DtCx5KooHUtDonKMUTkO0LHkqnSOJbJnPDYzj+qxiWyP\nzAxvwcRj5QOR6AibD3JtuEpEREQkI3JmuCqbJk3akO0QRLZLJ5ywU7ZDqEO5QCS7Mp0T1JMjIiIi\nkaQiR0RERCJJRY6IiIhEkoocERERiSQVOSIiIhJJKnJEREQkklTkiIiISCSpyBEREZFIUpEjIiIi\nkaQiR0RERCJJRY6IiIhEkoocERERiSQVOSIiIhJJKnJEREQkklTkiIiISCTlTJFjZn3N7HUzm2dm\nc8xsTAPb3WZmi8xslpkd3NZxioiISPuQn+0AklQDF7r7LDPbAZhpZi+7+4LEBmZ2LDDA3Qea2VeA\nu4EjshSviIiI5LCc6clx90/cfVb8/mfAfKBPvc1OBB6KbzMd6GZmvds0UBEREWkXcqbISWZm/YGD\ngen1XuoDLE96vIJtCyERERGRnBquAiA+VPUUcF68RydtY8eOrb1fUlJCSUlJi2ITkbZTVlZGWVlZ\nxvanfCDSfqWbD8zdMx9NmswsH3geeNHdx6d4/W5gsrs/EX+8ABjp7qtSbOthj23SpA0tiltE0nPC\nCTuF3tbMcHdL53PC5gPlApHsCpsTwuaDXBuuuh94P1WBE/d34HQAMzsCWJeqwBERERHJmeEqMzsS\nOA2YY2blgAOXA8WAu/s97v6CmR1nZh8CnwNnZS9iERERyWU5U+S4+7+ADiG2O7cNwhEREZF2LteG\nq0REREQyQkWOiIiIRJKKHBEREYkkFTkiIiISSSpyREREJJJU5IiIiEgkqcgRERGRSFKRIyIiIpGk\nIkdEREQiSUWOiIiIRJKKHBEREYkkFTkiIiISSSpyREREJJJU5IiIiEgkqcgRERGRSFKRIyIiIpGk\nIkdEREQiKaeKHDObYGarzGx2A6+PNLN1ZvZu/HZlW8coIiIi7UN+tgOo5wHgduChRraZ6u7faaN4\nREREpJ3KqZ4cd58GrG1iM2uLWERERKR9y6kiJ6ThZjbLzP5hZvtnOxgRERHJTe2tyJkJ9HP3g4E7\ngOeyHI80Ye3aymyHICI5QvlA2lquzclplLt/lnT/RTO7y8x6uPuaVNuPHTu29n5JSQklJSWtHqNs\ntXjxbC64YCS33jqVPfc8KNvhSDtTVlZGWVlZxvanfJBdygfSEunmA3P3zEfTAmbWH5jk7tv8Csys\nt7uvit8fBjzp7v0b2I+HPbZJkzakG640orT0J8yatYyDDy5m3LiH67y2dm0l3bv3ylJkkitOOGGn\n0NuaGe6e1py8sPlAuaD1KB9IGGFzQth8kFPDVWb2GPBvYB8zW2ZmZ5nZL8zs5/FNTjazuWZWDtwK\n/ChrwUqjFi+ezbx5b+P+GvPmzWDJkjl1XjvzzEF1nhOR6FI+kGzJqSLH3U91993dvdDd+7n7A+7+\nZ3e/J/76ne5+oLsPdfevuvv0bMe8PWtsfH3ixBuoqroY2IWqqot58MEb6rwGB9V5TkTaN+UDyUU5\nVeRI22npBMDGWl+VlcsoL59Ep06P0qXLkXTq9Cjl5X+nsnJZoy06CU8TOCWTlA/aN+WDhqnI2Q5l\nonu4sdZXUVFfbrzxNUpLr+Pqq6+mtPQ6brzxNYqK+jbaopNw1L0vmaR80L4pHzRORc52qKXdw021\nvvLy8hg06DD233947W3QoMNYvfqjBlt0Ep669yWTlA/aN+WDxrWrJeTSclsT0lzmzTuQJUvmNHs5\nZ6rW17hxDze5QiLRoqupqap9rkOHAoqK+qZ7OBnTXlZ3ZOLvJ5KgfJCa8kF0qCdnO9PS7uGGxtff\neeflJrtMG2rR5eVl959he+ruVfe+ZJLywbaUD6JFPTnbkURC6tx5BWZP476J8vJ3qaxcRq9e/ULt\no6HW16OP3oL7vrWtuPYkubs3l2PPxN9PJEH5IDXlg2hRkbMdyUT3cKL1lWzx4tnMnfsmsJY5c1Y1\nq8s0293C7am7N5e796X9UT7YlvJB9DR4xmMzuybMDtz96oxGlCE643HjMplMSkt/Qnn5R0AM6MDQ\noX1DtYCST/O+8869046nJccSnIW1BPcLMbuZgw+ektXWW7aTfFvTGY9zg/JBQPkg+9ryjMd7hLip\nZGyHMjnmnOgyhaXAa8CS0CskEt3Cf/rTVWnH05Jjaez8HdnQnuYCSHQoHwSUD6KpweEqdz+rLQOR\ntpPJMeeior7ss88IFi06EfddMLuSgQP/3mSXaXK38Acf7IN7/7TiST6W88//v2a1enKtu7e9zAWQ\naFE+2Bq78kH0hJ6TY2YDgR8DfYAVwOPuvqi1ApPWkekx59WrP2Lhwql07vxZ7eS3hQvfZfXqjxqd\n/Ja8KsD9MmAsc+asb1Y8yccyZ84gzjxzENdc8xxDhowM9f5U8wmypT3NBZDoUD7YSvkgmkJdhdzM\nTgAeBZ4HKoB+wLeBn7r731s1wjRpTk5qmR5zjsViLFr07jatn4EDD2lwKWhl5TLOOecgOnc+hM2b\nY8Rim4AFwAEMHTogdDzJxwKHADXAfxg//p/tLiHk2lyAtqI5OdmlfJCbttd8AJmfkxO2J+c64ER3\nn5z0ASXAHUBOFjmyrdZYcphO62fDhjWAccop3+OBB64CdgT+CfyI8vJ5oeJJPhb3h9i0aRmwCBjI\nXXddwY03tp9/lloKKtmgfJCblA8yK2yR0xd4o95z09DE43YlV8acH374JswGU14+I2n8/luYXRVq\n/B7qHsuECb/nww+vwH0X4EoWLrw2Y927bbG6IVf+LrJ9yZV/d8oHdeXK3yUqwhY5s4DfANcnPXdh\n/HlpJ3JhzLnuWPN+VFV9SmHhBjZvvp+OHQtYuHBWk+P3sPVYKiuXsWjRG+Tlrcb9L4DjvpZ77hnL\nH/7wdItjTSxpbc3u7lz4u8j2Jxf+3SkfbCsX/i5REvb82b8CzjGzj81supl9DPw8/rwIELRymlL3\nNOS/ZZ99vk6/fl2ATfTr17X26sRhFRX15fLLHyUWm0/HjhsoLNxMYeEg5s17tcVLP3XhO5H0KR9I\nLgjVk+PuC8xsP2A4sBvwMTDd3asaf6dsL8K0clKNNS9c+C4FBb2ByVRUfIOOHQubde2avLw8hg07\nLuPdu1rdIJI+5QPJFaFWV9V5g1mdf3HuHstYMGYTCFZtrXL3wQ1scxtwLPA5cKa7pxwy0+qqthWs\nBljGwQcXN7gKINXKi/vvv46FCw8DbgUuYOjQRdu8Pxtn/dyeVze0Ja2uiiblA0lXW57xOHlnh5jZ\nm2b2OVAVv1XH/5tJDwBHNxLHscAAdx8I/AK4O8OfL2nY2sp5jXnzZjR4hs76Vx0uKurDwoVTyct7\nDDiQvLxtzzCajbN+pnvm0zDd8yJRp3wQUD7IDWH7AScCk4HDgL3itz3j/80Yd58GrG1kkxOBh+Lb\nTge6mVnvTMawPWjpj6/+++uOq1/c6Jh18nuLivpy/vl/Ji8vBrxOXl6MCy64p063cjbGwROrG0pL\nr+Pqq6+mtPS6JucG6BTs0l4pHzRO+aB9C1vkFANXuPt8d69IvrVmcCn0AZYnPV4Rf05CCvPjayzp\n1X9/c1o59d+bl5fHlCnPU1NzCbALNTWXUFb2fO0YfNgWYabVb2Huv/9wBg06rNG5AWGSr1p2kmuU\nD5qmfNC+hS1yngW+1ZqBSNu4777f0diPL5F4/vWv51L+COv/eJvTyqn/3qYSYnNahNkUJvmqZSe5\nSPkg85QPckuDq6vM7GEgMVOvEHjWzKYBnyRv5+6nt15421hBcPXzhL7x51IaO3Zs7f2SkhJKSkpa\nK652YerUp5k79xVgMvPmnZJyhcDEiTfgvjfXX38GYIwf/0btNg2tMAhzTodU7y0uPqDBVRDt6ayf\nqZJv/UmJuthe85WVlVFWVpax/Skf1KV80DqUD1pHuvmgwdVVZlYaZgfuPq7Zn9pYQGb9gUnuvs36\nPDM7Dvhfdz/ezI4AbnX3IxrYj1ZX1XPaaYPZuLEbsDdmR26zQmDx4tn89rc/YMuWwwguUVbJ0KFf\nqd2m/gqDAw74J9dd99fa9ze26qE5qxPWrq2kW7eiZl8DJxuSr7tj1gn3TXz55bvcd9+c2uS79Xud\nS8eOB3LjjU9p+WmcVldlTxTzwYwVoa85ndKwPtVNbtPYZ6z77zLuuHI/OhYeiuUV4rHNbNk8k3Ov\nnc/OuwT54JPl7/Hgjd+numou+QUHctZvn6V335SLibdLv/tll1DbtfjaVZkuXsIws8eAEmAXM1sG\nlAIdg3D8Hnd/wcyOM7MPCZaQn5WRzz26cyZ2k9PeefFFNm5cB8wEBtKx83zKy+fz6b6V9CouBmDi\n8TexZcupwCPA+8CBzJn7Bkt3W0jXbjsHLakdP8bynqGmag1z587nXZ/Cocccw+L3ZnH+d49g/PTp\n7Dl4SJ3PrqyoqPNej22ivHxmnc9OSN7Pvsd+vfb5tatW0b137s0x7xkbyE2DplFdtTX55hcU0POw\ngVg8+U48/iaqquMtu+qLmfj8TYx7/sna7dd/+imrKiroXVxMt5492/oQZDv0zjsv180HHd+nvHxB\nnZ6RiRNv2DYfzHmDJUvm0LVrtzo9KzU1a5g7dwEzZ77CoYce1eh5chK9Mvnx91rNZsrLZ6bslUne\nT3IPUaoC6rFYHuwWY5+BhWl/L48tinFqXsNnRZmxIp8PG/mM7rE9+dmAacSS8kFeQQHdD9yzNh+8\nPvp6amqCfFBTczGvl13PafdszQefr/mUdSsq2LlPMV17KB+0VOjz5MQvyHk6wUTfFcDDyRfszDXN\n6clZwPxWjib7zv7Webz56jG1LacDD/8rV9x2HgcdfhB5eXmsqFjBN/sfBfQCLgYuAm4BHuZrR/fk\nzy/czJy351BdFbR0rr/wDua+8yVfPWpX7vvnrZxz9Hn8+5VKjvxWb+596dY6nx2Lxeq8t2JRBVee\nfTVPzXyS/YfuX2fbVPuZP2s+Jx36A5559yn2HbJv635RGZb4XrvuOJS8vE7UVG/ki8/n8tTMv3LA\nIQfwwuP/4Nqzr6K4Yz4VW6q5csLvOO7Hx2c77DazL/uF3lY9OZlz9dWn8d57o2rzwd57/42f/7y0\ntmck0UOZKh8MHboHpaUT6/SsTJjwexYt2sDQof0YN+6RRs+TM315HjM+f5vd9wrWkMwv+5wZd57L\nLTeXMWBA3QZSqv2kKqBmrMhn5cEfMPLQghZ9L4tWwqLy/ikLnUSBM3DoUgbult7+V1d8zIUDj6bT\njkOxvEJi1Z+x+fO5XPPWE/Q/ZH+m/+UFHvlFKf0K8llWVc1p94zjiB8d16Jjam9+WnBgqO3C5oNQ\nRY6ZnUNwJfL7CPot+wFnA1e5+72hImpjzSlyqvz5Vo4muyoqKhm459nsuOMQ8vIKicU2s3Hjeyxa\nMoHi4qA1FIvFuP++V/j1L+8ABgKdgS3AAoA6286atZiRR17Ll1/Op3Pn/bj3gbMYfdYDtY+n/vsq\nhgzZc5s4Vq1aS+/e3Tn+mOt59ZWNfPNbO/GPF39b+3r9/Sb209D2zfHpp+upWLqK4v696dmzW1r7\nSEcsFuOdtxdRVVXN66/M4vYbnqY4vwPLYzGuveUcrrzgPiZ/uYXBwGxgVOeOzFl6f5vGmE0F9u3Q\n26rIyYwwQ6yxWIxXXpnInXeeT6p80Nhw7Hnn/YHx4y9LOTybXCgckPc8+6yp4OT/9x8Wv1HJXvv1\n59Y/PlQbZ0PDvPULn0SB88ND32TwwrmhvoPVa77koxUb6dtnR4p6bO3Jn73PgTw5czi7zRpUZ+gq\nEffRwx/imE/DfUYqsZgza24lVdUxpr35EfdOKKe4g7Hcnct/eyTX3fAvyjbV1OaDkk4dmPryT+vE\nGHW7HnBXqO1aPFxVz2+Bo9z9vaQPeAJ4GsjJIqc5Ct5v3lmf25u9Yrvw1mM3UlVdU/tcQf5P2Gvj\nLuTVHrsx+ohvUDDW+aDiY2pqgpZMx/wDOKHk8DrbXnnuX9m8+VJgFzZvvpTzfnlzncdXnvskL/25\nbjEya8FiDv3BhTx2w2+YNuUD3OczrWw/3n9mMUP23TPlfq8890n+eMHJDW4f1hP/mMKvS2+nf34+\nS6urueuaMfzouBENbr9q9Vp6F3Vv1mc05NM1G+i4wujRdUfuvPEZpmyuYvDmKmYDXx9zDwMKCkiM\nxg8Gijvks+L1Vex+UPi5Ku3aAdkOYPsT5irXeXl5/M///BTI4+OPPyQWC3JHfv4xHH74sducuyZ5\nou3dd49rcOJtosDZbdWj/Pgr13PfmV9hxfRF4B9Q8eGBTJr+Pid8Zf+U+33wwRs444yL60xYnjT9\nfWLfLqwtcArL1tCBgY0e/9PvzOWiR5+nuEMHKmpquOm0EzjpsAOoYRGDmQuHwpPAjHih8/r8//LJ\noJ4cPfwhhnwyh8I31jb5GQ1ZvfFzOq/ZmZ4dC7j/vueZWrW1oPn67//FXvkdGEzwXQ8Gisnn00l5\n9CveJa3Pa5cynBPC9uT8F9g1+VpVZlYIfOzB9exzTnN6cjZMmtTK0UTHsspKDjrnHHboPIQ8K6Qm\ntonPN82mS+F+5HfYkZhv5rMv32POfffRr9fWMfPvld7A5Fkb6b7DatZ9diExv5A8u5lRBz/LM+N+\nu81+E/sZfsBhTH//x9tsH9bq9es5/OyzmbwlqbekY0fenjCBom7b9pbMXryYkRdcyNRbb+GgPZtX\nTNX39JQpXHT77RTn57OkqoqdgSVJY/UHde7MR9XVTKmqChVbFO10wgmht1VPTu6p3ysUi33Jpk3l\nFBYeQIcO3er0Ei2t2ou8EQXsPfADJpSczNR/F9K96yrWfXERMb8Iy7uZ3kOmcM7PH29wAu8ee3+T\nj/5zbO0w2+7DXuT+N763tcDpd1qj8TaVD2qWPcq6X5cwrfprvPBCf45YOosLLhjJ6Kemc9IJM/jG\nwk/4osyh37Bmf1fKB+GEzQmZ7smZBtxsZpe4+xdm1hX4A/DvkO+XNlK5di29umemFyKVvkVFvHbj\njVTVBK2NWCzG4pUj2Wu33WpXORR0+Al9i4pq3zN78WL+Pe8D3F9mzcahdCl8kPwOTxDzzbxW/h7l\ni77HkAED6uwX4L8bjue0665jh85V5NnW7ZdVnlmngGpMRWUlxfn5DN6yBQhaR/3y86morEyZOEon\nPoVzEKUP/rVZxVR9q9ev56Lbbw+S6ZYtzAaOAJ4DvkuQwD6uqWHc6NGMuu8++uXns6y6mpvGjNmu\nEpq0nra4xlP9XqFYLMbKlYvZbbe9avNBopdo6crgPRWzFjB95mqcV1nz+VC6dHyQ/A6PUZ23hU/K\n5/FJj9ns9rWh20zg/WLdap4c8wM6dl0bX8DwBSuml9O9bFcKF1Y1WeBA8/NBsIT+IF67+fecdMJR\naX9PygfZE7bI+SXwBLDezNYAPQgKnFNbKzBpvkz2QkDqgikvL4/DBg2q89zAPn0aLaxKJz7FlqpL\ngcEYY9iv32R+f/b3AVj6yShG/eYi3hh/6zb7jcVi2xQ+9QuophT36kVFdTWzobZ1tKy6muIURdLW\nYmw+/5q3H3OWLEn7e0yVTPsCpwC7FRSwwYybxozhpBEj+Pbw4cH2vXrVJrTV69dv85xIWGGuAt5c\nqYqmxNmAk/XpM7DR4uqJK+5hS9Vl1OaD4sn8/tsHEeMjZu7Si6tO+SrXzvgrJSfVzwc92O+QR2sX\nMPRgFvuu3JO9FmyhQ94+oY6hOflgbfxcPjCXincO5D/v7cc3Oqe3RF35IHtCnXDE3Ve6+wiC61Wd\nAOzp7iPdvcET8UnbS+6FaKnZixcz6MyzmLNkSYu2W1ZZyevl0+jS6SF26jKcrp2nMHPRHPoUFTF8\n//15cso7YINTxpwoqIbvv3/t7bBBg5p1npyibt24acwYRhYUMKiwkJEFBdw0ZgwAMxctYvX69bXb\nbi3GdmFL1aUt+h6TkykEyfS/wEvAOuClW27hpBEjamM8dODA2uT19JQpHH722fzmqqs4/OyzeXrq\n1LTjkO2ZjnL1AAATUUlEQVRTpq/xFPYMvU1tt+7jCua+8gpdOj/ATp2Hbc0HB4zgyAFf5eVH/oNx\nEC9feT1fy59W5zai478546sbOHvkF5w98gtO7bmJ4WsKKMjbJ/TwUZP5YOOm2m3fe+iPtXOCaqou\nZsLlr4T7slJQPsieUGWpmX0LWOruC4GP4s8NAvq5e/p/ecmYTPZCQN2C6e7zRzfYU9PU8E7y8Na6\nzz5j1dq19C36Ln2LikLFnJEWjDtmRj5Q486UWbO46LbbKM7PpyLeJXz4vvvyevk0dui8Me2hsWS1\nyXT8eHpWVfFf4C6Ck0D1Lyjgs02bUh5bqm7tUbfdxsghQ9SCk1CSzyY8d+4BKc9k3FzJRdP55/9f\ngz01TZ3Jt9uue1A67VGGLv4rm/7VkVX5Pbfmg6Wf8fZ/Psf9XWa8vg8fXf4sB/atm3dWb9zEsjWf\n069HVwp37BRMAG7u/JjG8kHVZsYW70L14XvxyazgPEAxnsbzNjPzlZlUfDycnjT/d6h8kD1h+97u\nBOovR9kYfz5cP6G0qtKJT7G5Kriw3eaqS1o0pyS5+Jg2d1/2OeNM3hh/6zYFSJgiJdEbkzzpLlFY\nPPLaW9v0nCTHnOo9J40Y0azCJ5EkLt2yheuBvYAnX32Vi4BrkpLG9HvvrS3G1m7cSPcdd2z20Fh9\nJ40YwYF77sk3zzuPv1VXU8LW7vHZH37IyZddts2xNXfOgEh9dVclXcSf/nQVN9zwXNr7q1s07c8Z\nZ+xT5/IOqbZLvsxDMsvLY8CwwXzyr/u5/P4pFBd0rJsPqi8HdmFL9RWM++ezPDPu3Nr3ps4HwzKe\nD0p+NpG7Fu3NUTe8zm6raljc8b8MOHQzJ/aZx85vL4W89H6HygfZEbbfv5e7r6z33Epg1wzHI2lI\nDAnl2QRgCHk2gdfKp7GsMr2r3CYP22yuugSnV8qhm7DDO8mtkXe/+ILJW7bwm/Hjk4axvkKXTg/V\niTnVey667TYeePHFZnXdzl68mF3NuB6YDMwC3gJuBz5la9JYvno1hw0aRNdOnTjtuj+wQ+fOzR4a\nS2XQHntw6/nnc1LHjgzt0oVRHTtSes45XDNhwjbHtnr9+pTd2g3NGRCpL/kil3l5Q4EJLFgwOeVV\nwMOqv5QbeqUcBgt7Ac0Nn67hstIyJldV52Q+KCaPnafN5NQz5rOqTxf+9cdT+UrxAg6ZOadZQ2Op\nKB+0vbA9OYvN7Bvu/nrScyVA4xM2pE30LSri7vPPZ8wdD1ITe4QOeT/hrvMuSKsXIlEw7dB5I+6P\n8/mmjcAqps2tqtNTk7xdU8M7qVojxQUFXPurXzFg991rt0vuOUn1nj06dODqe+/ljerqUF23T0+Z\nwoW33caXVVXsG99H7b6ApQSVenLSyOTqqkTr8qQRIxg5ZEjt48ZaZ4cOHMhNY8Yw6rbbtMJCmi2x\n2mn58gXcdddVxGIPkZ9/Ohs3rk3rQpbJF8Z0f5JNm9YBq5g79806PTXNuYDm6oqP6V//fDC5lA9q\nnAFLqzn407nc8eIDwIH85fTHOO1/f5pWgVO/p0n5oG2FLXLGAs+Y2QTgP8AAgutGZeTaUdIyeXl5\nPDnlHaprrgKOp7rmKp4se5ZTRo1q9r6S59BcPuFRyhd9E2ciW6reqPM//vpLyaHhlU8NrWg45vDD\nG/yxpnpPRXU1/QoKGFwdrK5orOs20fKbUlXFdOC8+D4S+/oQOKtTJ1bU1DDunHMo6tYtY/OaGhpm\nS46xsRUe9ZOgEpqElVjt9Nhjt1JTcwVwPDU1V/DQQzeldbXr5CXiEyZcw6JF3wQmUlX1Rp15N2FO\nMFi7z+LdWVpdk7v5YPRoeu/Yg1mPzGDe1CW4f8CMDwcxp6YnzZ3Z1FguUD5oG2FXV/0N+BbQFTg+\n/t+j489LltVdwbRtV29zJObQ9Ckq4t1Fc+jaeRo7dRlD186P1Nlnc1Y+JSbdjUrqom2qNZLqPeNG\nj+bjmppQXbe1rSNgNDAeGA4cVFjIqI4d+cE3v8ny6mr65ucz7r77eHrq1IysrmqoWz15FVeY76P+\nCguRsJKHrLp0OZJOnR6lvPzvaQ1ZJYqmoqI+LFr0bzp3fpMuXS6kc+fH6+wzsd3++w+vvQ0adFjK\nfLBTzx784ZpRjCrIz918sHQT455byZYt8XxQfXmz80GYXNDQsSkfZE7oRf/uPgOY0YqxSJqa06uS\nrX2m0xpJ9Z4du3QJ1XVb3KsXi+Nd2IOBrwAdOnTgd5dfTp+iIo654IKgmzvegioZP561VVUtXl0V\ndqJgY9+HzokhLdGcXpVs7fPE4wZyVM1BVHTsm9v5oHAieR3+2uonIlU+aD0NFjlmdk2YHbj71ZkL\nR9KR6gR9ubjP+l206bynOcVSjXuwRJNgvN2BwXvtldYcobCac7KxVN9HQ93bImGlOkFfLu6zaMeu\nFPVr3jWg2lM+aE4uSHVsoHyQCY315OwR4v0dMhWISFhhiqWKykoGFhbyzy++YClBYvtWYeHWyX7N\nmCPUnJZUbddzitZlU/vROTFEmi9X80FjuSDMfpQPMqPBIsfdG5xUbGaDgdPRZR0kRyUS10rgcOq2\noppKPsnSaUmlal0+8OKLXH3vvfQrKODjmpqU+2lO97a6sEXCy1Y+aKinSfmg7YSek2NmPQmKmjOA\nIcAbBJPURVqkNX6gTSWuMN3cYVtSqeJPbl0++NJLXPanP7EPsLy6mkuAi1LsJ2z3trqwJcqilg/q\n9zQpH7StRoscMysAvgOcCRxNsNLucYLevh+6e3pnm2v4844BbiVY9TXB3a+v9/pI4G/A4vhTz7j7\ntZmMQdpWa/5Am0pcTXVzh2lJNRX/6vXrKb33Xt5ia6IaBezeocM2LbIwLUp1YUuUKR8oH2RaUz05\nq4AY8CBQ6u7vApjZrzMdiJnlAXcA/wN8DLxtZn9z9wX1Np3q7t/J9OdL22uLH2g6k50TmmpJhYm/\norKS/vn5DK4KVqUkrj68tKoq5QTEphKxTvMuUaV8oHzQGpo6T85sYGeCFXeHm1nqqzRmxjBgkbtX\nuHsV8BfgxBTbWSvGIG0o+dwVUPcHmguaOn9FmPiLe/ViWb1zeSwCrvn5zxtMQo2dE0OneZeoUj5Q\nPmgNjfbkuHuJmRUTTDK+CLjNzF4mOBlgQYZj6QMsT3r8EUHhU99wM5sFrAAudvf3MxyHtJHmLrFs\nrkyM7TfWkgoTf3KX8x4dOlBRXc0fR4/mzGOOSSue5kySFGlPlA+aT/mgaU1OPHb3CuB3wO/M7GsE\nBU8MeM/M7nf39C/w03wzgX7u/oWZHQs8h66C3m615g80k2P7DXVxh40/06dl12neJYqUD9KjfNC4\n0KurANx9GjDNzMYA3yMoeDJlBZB8Jbe+8eeSP/+zpPsvmtldZtbD3dek2uHYsWNr75eUlFBSUpLB\ncCUTWuMH2paT8cLG35K5AG2xv1xUVlZGWVlZxvanfJD7lA/So3zQsGYVOQnuvolgldXj6by/AW8D\ne8eHx1YCpwA/Tt7AzHq7+6r4/WGANVTgQN2kJrkr0z/Qtp6Mtz0kmGyoX4iMGzeuRftTPmgflA8k\nlXTzQVpFTmtw9xozOxd4ma1LyOeb2S+Cl/0e4GQz+xVQBXwJ/Ch7EUuuau2xfRFpP5QPtm85U+QA\nuPtLwKB6z/056f6dwJ1tHZe0L5qMJyIJygfbt5wqckQyRZPxRCRB+WD7pSJHIktj4yKSoHywfWrq\nZIAiIiIi7ZKKHBEREYkkFTkiIiISSSpyREREJJJU5IiIiEgkqcgRERGRSFKRIyIiIpGkIkdEREQi\nSUWOiIiIRJKKHBEREYkkFTkiIiISSSpyREREJJJU5IiIiEgkqcgRERGRSFKRIyIiIpGkIkdEREQi\nKaeKHDM7xswWmNlCM7ukgW1uM7NFZjbLzA5u6xhFRESkfciZIsfM8oA7gKOBA4Afm9m+9bY5Fhjg\n7gOBXwB3t3mgIiIi0i7kTJEDDAMWuXuFu1cBfwFOrLfNicBDAO4+HehmZr3bNkwRERFpD3KpyOkD\nLE96/FH8uca2WZFiGxERERHysx1Aaxo7dmzt/ZKSEkpKSrIWi4g0T1lZGWVlZRnbn/KBSPuVbj4w\nd898NGkwsyOAse5+TPzxpYC7+/VJ29wNTHb3J+KPFwAj3X1Viv152GPbMGlSBo5ARJprpxNOCL2t\nmeHuls7nhM0HkyZtSGf3EsKMFfnkjShg74EfcNTCu+hStgv0G5btsCTHhM0JYfNBLg1XvQ3sbWbF\nZtYROAX4e71t/g6cDrVF0bpUBY6IiIhIzgxXuXuNmZ0LvExQfE1w9/lm9ovgZb/H3V8ws+PM7EPg\nc+CsbMYsIiIiuStnihwAd38JGFTvuT/Xe3xumwYlIiIi7VIuDVeJiIiIZIyKHBEREYkkFTkiIiIS\nSSpyREREJJJU5IiIiEgkqcgRERGRSFKRIyIiIpGkIkdEREQiSUWOiIiIRJKKHBEREYkkFTkiIiIS\nSSpyREREJJJU5IiIiEgkqcgRERGRSFKRIyIiIpGkIkdEREQiSUWOiIiIRFJ+tgMAMLPuwBNAMbAU\n+KG7r0+x3VJgPRADqtx9WBuGKSIiIu1IrvTkXAq86u6DgNeByxrYLgaUuPtQFTgiIiLSmFwpck4E\nJsbvTwS+28B2Ru7ELCIiIjksVwqGXu6+CsDdPwF6NbCdA6+Y2dtmNrrNohMREZF2p83m5JjZK0Dv\n5KcIipYrU2zuDezmSHdfaWY9CYqd+e4+LcOhioiISAS0WZHj7kc19JqZrTKz3u6+ysx2BSob2MfK\n+H8/NbNngWFAg0XO2LFja++XlJRQUlKSXvAi0ubKysooKyvL2P6UD0Tar3Tzgbk31GnSdszsemCN\nu19vZpcA3d390nrbdAHy3P0zM+sKvAyMc/eXG9in58KxiUhmmBnubmm+V/lAJELC5oNcKXJ6AE8C\newAVBEvI15nZbsC97v5tM9sTeJZgKCsfeNTd/9jIPpXURCJERY6IJLSrIqc1KKmJRIuKHBFJCJsP\ncmV11XYlk/MMsikqxwE6llwVpWNpSFSOMSrHATqWXJXOsajIyYKo/KOLynGAjiVXRelYGhKVY4zK\ncYCOJVepyBERERGJU5EjIiIikRTpicfZjkFEMqslE48zHYuIZNd2vbpKREREtm8arhIREZFIUpEj\nIiIikaQiR0RERCJJRU4rMrMJ8YuPzk567mQzm2tmNWZ2SDbja44GjuUGM5tvZrPM7Gkz2ymbMYbV\nwLFcY2bvmVm5mb0Uv1Bszkt1LEmv/cbMYvHLpuS0Bv4mpWb2kZm9G78dk80YW0r5IDcpH+SeTOYD\nFTmt6wHg6HrPzQG+B0xp+3BaJNWxvAwc4O4HA4uAy9o8qvSkOpYb3H2Iuw8F/gGUtn1YaUl1LJhZ\nX+AogmvBtQcpjwO42d0Pid9eauugMkz5IDcpH+SejOUDFTmtyN2nAWvrPfeBuy8C0loKmy0NHMur\n7h6LP3wL6NvmgaWhgWP5LOlhVyBGO5DqWOJuAS5u43DS1shxtKvfSWOUD3KT8kHuyWQ+UJEjmfIz\n4MVsB9ESZnatmS0DTgWuznY86TKz7wDL3X1OtmPJgHPjwx/3mVm3bAcjoSkf5IjtPR+oyJEWM7Mr\ngCp3fyzbsbSEu1/p7v2AR4H/l+140mFmnYHLqdu93q56CZLcBewVH/74BLg5y/FICMoHuUP5QEWO\ntJCZnQkcR9DaiYrHgJOyHUSaBgD9gffMbAnBkMFMM+uV1ajS4O6f+tazld4LHJ7NeKRpygc5Z7vP\nB/mtF5LEGQ1Xzu2toq5zLPHZ7RcDI9x9c9aiSk/9Y9nb3T+MP/wuMD8rUaWn9ljcfS5QuxIkntgO\ncfdU49u5pv7fZFd3/yT+8PvA3KxElVnKB7lJ+SD3ZCQf6LIOrcjMHgNKgF2AVQRdhmuB24EiYB0w\ny92PzVaMYTVwLJcDHYH/xjd7y91/nZUAm6GBYzkeGATUEKxA+KW7r8xWjGGlOhZ3fyDp9cXAYe6+\nJjsRhtPA32QUcDDBpM+lwC/cfVWWQmwx5YPcpHyQezKZD1TkiIiISCRpTo6IiIhEkoocERERiSQV\nOSIiIhJJKnJEREQkklTkiIiISCSpyBEREZFIUpEjIiIikaQiR0RERCJJRY6IiIhEkoocEWlXzGyJ\nmX2j3nNnmNkb2YpJRHKTihwRiQpdo0ZE6lCRIyKRYmYxM9sr6fEDZnZN0uNvm1m5ma01s2lmdlB2\nIhWR1qYiR0SiwEJtZDYUmACMBnoAfwb+bmYFrRibiGSJihwRaY+eM7M1iRtwZ8j3jQbudvd3PPAw\nsBk4otUiFZGsUZEjIu3Rie7eI3EDfh3yfcXAb5IKpLVAX2D3VotURLImP9sBiIikobHhqS+ALkmP\ndwWWx+8vB37v7n9orcBEJHeoJ0dEomYWcKqZ5ZnZMcDIpNfuBX5pZsMAzKyrmR1nZl2zEaiItC4V\nOSLS3jS1VPw84DvAWuDHwLO1b3SfSTAv5474XJ6FwBmtFKeIZJm569QSIiIiEj3qyREREZFIUpEj\nIiIikaQiR0RERCJJRY6IiIhEkoocERERiSQVOSIiIhJJKnJEREQkklTkiIiISCSpyBEREZFI+v9Q\nSHrfiYisCwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x256494afe80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_min, x_max = X_train[:, 0].min() - 1, X_train[:, 0].max() + 1\n",
    "y_min, y_max = X_train[:, 1].min() - 1, X_train[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),\n",
    "                     np.arange(y_min, y_max, 0.1))\n",
    "\n",
    "f, axarr = plt.subplots(1, 2, sharex='col', sharey='row', figsize=(8, 3))\n",
    "\n",
    "\n",
    "for idx, clf, tt in zip([0, 1],\n",
    "                        [tree, ada],\n",
    "                        ['Decision Tree', 'AdaBoost']):\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    axarr[idx].contourf(xx, yy, Z, alpha=0.3)\n",
    "    axarr[idx].scatter(X_train[y_train == 0, 0],\n",
    "                       X_train[y_train == 0, 1],\n",
    "                       c='blue', marker='^')\n",
    "    axarr[idx].scatter(X_train[y_train == 1, 0],\n",
    "                       X_train[y_train == 1, 1],\n",
    "                       c='red', marker='o')\n",
    "    axarr[idx].set_title(tt)\n",
    "\n",
    "axarr[0].set_ylabel('Alcohol', fontsize=12)\n",
    "plt.text(10.2, -1.2,\n",
    "         s='Hue',\n",
    "         ha='center', va='center', fontsize=12)\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.savefig('./figures/adaboost_region.png',\n",
    "#           dpi=300,\n",
    "#           bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "AdaBoost has more complex decision boundary than decision tree, as expected.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary\n",
    "\n",
    "Combine multiple (independent) base learners can improve performance\n",
    "* majority voting\n",
    "* bagging - parallel, can reduce variance but not bias\n",
    "* AdaBoost - sequential, can reduce bias but might overfit\n",
    "\n",
    "Choose diverse, not just, accurate base learners\n",
    "* variance and bias trade off\n",
    "\n",
    "Ensemble learning computationally more expensive\n",
    "\n",
    "Netflix prize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reading\n",
    "\n",
    "* PML Chapter 7\n",
    "* IML Chapter 17.1-17.7"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}