{ "cells": [ { "cell_type": "code", "execution_count": 57, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# %load ../../preconfig.py\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.rcParams['axes.grid'] = False\n", "import seaborn as sns\n", "sns.set(color_codes=True)\n", "\n", "import numpy as np\n", "import pandas as pd\n", "#import itertools\n", "\n", "import sklearn\n", "\n", "import logging\n", "logger = logging.getLogger()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Exercises\n", "=========" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ex 2.1\n", "##### Question\n", "Suppose each of K-classes has an associated target $t_k$, which is a vector of all zeros, except a one in the $k$th position. Show that classifying to the largest element of $\\hat{y}$ amounts to choosing the closest target, $\\min_k \\|tk − \\hat{y}\\|$, if the elements of $\\hat{y}$ sum to one.\n", "\n", "\n", "##### Solution\n", "Given: \n", "$t_k = e_k$; \n", "$\\sum_i \\hat{y}_i = 1$.\n", "\n", "need to proof: $\\text{arg max}_i \\hat{y}_i = \\text{arg min}_k \\|t_k - \\hat{y}\\|$.\n", "\n", "Proof: \n", "\n", "\\begin{align}\n", "\\text{arg min}_k \\|t_k - \\hat{y}\\| &= \\text{arg min}_k \\left ( \\displaystyle \\sum_{j \\neq k} \\hat{y}_j + | \\hat{y}_k - 1 | \\right )\\\\\n", " &= \\text{arg min}_k \\left ( \\displaystyle \\sum_{j \\neq k} \\hat{y}_j + 1 - \\hat{y}_k \\right ) \\\\\n", " &= \\text{arg min}_k \\left ( 1 - \\hat{y}_k + 1 - \\hat{y}_k \\right ) \\\\\n", " &= \\text{arg min}_k 2(1 - \\hat{y}_k) \\\\\n", " &= \\text{arg max}_k \\hat{y}_k\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ex 2.2\n", "##### Question\n", "Show how to compute the Bayes decision boundary for the simula- tion example in Figure 2.5.\n", "\n", "\n", "##### Solution\n", "[ref: Elements of Statistical Learning - Andrew Tulloch](https://github.com/ajtulloch/Elements-of-Statistical-Learning/blob/master/ESL-Solutions.pdf)\n", "\n", "As Eq.(2.22) of textbook:\n", "\\begin{align}\n", " \\hat{G}(x) &= \\text{arg min}_{g \\in \\mathcal{G}} [ 1 - P(g | X = x) ] \\\\\n", " &= \\text{arg max}_{g \\in \\mathcal{G}} P(g | X = x)\n", "\\end{align}\n", "\n", "The optimal Bayes decision boundary is where:\n", "\\begin{align}\n", " P(\\text{orange} | X = x) &= P(\\text{blue} | X = x) \\\\\n", " \\frac{P( X = x | orange ) P( orange )}{P(X = x)} &= \\frac{P( X = x | blue ) P( blue )}{P(X = x)} \\\\\n", " P( X = x | orange ) P( orange ) &= P( X = x | blue ) P( blue )\n", "\\end{align}\n", "\n", "As descriped in Sec 2.3.3, $P(orange)$ is same as $P(blue)$, and $P(X = x | orange)$ and $P(X = x | blue)$ are generated as bivariate Gassuian distribution. Hence we can work out the optimal Bayes decision boundary exactly." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ex 2.3\n", "##### Question\n", "Derive equation (2.24).\n", "\n", "Suppose there are $N$ points, uniformly distributed in the unit sphere in $\\mathbb{R}^p$. What is the median distance from the origin to the closest data point? \n", "\n", "##### Solution\n", "[ref: Example Sheet 1: Solutions](http://www2.stat.duke.edu/~banks/cam-lectures.dir/Examples1-solns.pdf)\n", "\n", "(1)\n", "Suppose $r$ is the median distance from the origin to the closest data point.\n", "\n", "Let $r_{\\text{closest}}$ are all possible closetst points. \n", "Because $r$ is the median case, $\\forall j$\n", "$$P(r_{\\text{closest}}^j \\geq r) = \\frac{1}{2}$$\n", "\n", "Because $r_{\\text{closest}}^j$ is the closest point, so all $N$ points have distance $\\geq r_{\\text{closest}}^j \\geq r$. \n", "\n", "together, we get:\n", "$$P(\\text{all N points have distance } \\geq r) = \\frac{1}{2}$$\n", "\n", "(2)\n", "First, all points are uniformly distributed in the unit sphere in $\\mathbb{R}^p$. \n", "\n", "Second, [the p-dimensional volume of a Euclidean ball of radius R in p-dimensional Euclidean space is](https://en.wikipedia.org/wiki/Volume_of_an_n-ball):\n", "$$V_p(R) = \\frac{\\pi^{p/2}}{\\Gamma(\\frac{p}{2} + 1)}R^p$$\n", "\n", "together, for any point $x$, \n", "\\begin{align}\n", "P(x \\text{ has distance } \\geq r) &= 1 - P(x \\text{ has distance } < r) \\\\\n", " &= 1 - \\frac{\\pi^{p/2}}{\\Gamma(\\frac{p}{2} + 1)}r^p \\Big{/} \\frac{\\pi^{p/2}}{\\Gamma(\\frac{p}{2} + 1)}1^p \n", " &= 1 - r^p\n", "\\end{align}\n", "\n", "Then:\n", "\\begin{align}\n", " P(\\text{all N points have distance } \\geq r) &= P^N (x \\text{ has distance } \\geq r) \\\\\n", " &= (1 - r^p)^N\n", "\\end{align}\n", "\n", "(3)\n", "In all,\n", "$$\\frac{1}{2} = P(\\text{all N points have distance } \\geq r) = (1 - r^p)^N$$\n", "\n", "we get the solution:\n", "$$r = (1 - (\\frac{1}{2})^{1/N})^{1/p}$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ex 2.4\n", "##### Question\n", "The edge effect problem discussed on page 23 is not peculiar to uniform sampling from bounded domains. \n", "\n", "Consider inputs drawn from a spherical multinormal distribution $X \\sim N(0,I_p)$. The squared distance from any sample point to the origin has a $\\mathcal{X}^2_p$ distribution with mean $p$. Consider a prediction point $x_0$ drawn from this distribution, and let $a = x_0 \\big{/} \\| x0 \\|$ be an associated unit vector. Let $z_i = a^T x_i$ be the projection of each of the training points on this direction.\n", "\n", "Show that the $z_i$ are distributed $N(0,1)$ with expected squared distance from the origin 1, while the target point has expected squared distance $p$ from the origin.\n", "\n", "##### Solution\n", "$z_i = \\alpha^T x_i$, which is a linear combination. Moreover, $x_i \\sim N(0, I_p)$ means that its elements are all independant.\n", "\n", "As [the variance of a linear combination](https://en.wikipedia.org/wiki/Variance#Sum_of_uncorrelated_variables_.28Bienaym.C3.A9_formula.29) is:\n", "$$\\operatorname{Var}\\left( \\sum_{i=1}^N a_iX_i\\right) = \\sum_{i=1}^N a_i^2\\operatorname{Var}(X_i)$$\n", "\n", "We get:\n", "\\begin{align}\n", "E(z_i) &= \\alpha^T E(x_i) \\\\\n", " & = 0\n", "\\end{align}\n", "and\n", "\\begin{align}\n", "\\operatorname{Var} (z_i) &= \\sum \\alpha_j^2 \\operatorname{Var}(x_j^i) \\\\\n", " & = \\sum \\alpha_j^2 \\\\\n", " & = \\| \\alpha \\|^2_2 \\\\\n", " & = 1\n", "\\end{align}\n", "\n", "Thus, $z_i \\sim N(0,1)$.\n", "\n", "The target point has expected suqared distace $\\| x_i \\|_2^2 = p$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ex 2.5\n", "#### (a)\n", "Derive equation (2.27). The last line makes use of (3.8) through a conditioning argument. \n", "\n", "#### Solution\n", "First, we give:\n", "\n", "1. for $y_0 = x_0^T \\beta + \\epsilon; \\ \\epsilon \\sim N(0, \\sigma^2)$:\n", " + $E_{y_0 | x_0}(y_0) = E(y_0 | x_0) = E(x_0^T \\beta + \\epsilon) = x_0^T \\beta$\n", " + $\\operatorname{Var}_{y_0 | x_0}(y_0) = \\operatorname{Var}(y_0 | x_0) = \\operatorname{Var}(x_0^T \\beta + \\epsilon) = \\operatorname{Var}(\\epsilon) = \\sigma^2$\n", "\n", "2. for $\\hat{y}_0 = x_0^T \\hat{\\beta} = x_0^T \\beta + x_0 (X^T X)^{-1} x_0 \\epsilon$: \n", " + expected value:\n", " \\begin{equation}\n", " E_{\\tau}(\\hat{y_0}) = E(y_0 | x_0) = x_0^T \\beta \\quad \\text{unbiased}\n", " \\end{equation}\n", " + variance:\n", " \\begin{align}\n", " \\operatorname{Var}_{\\tau}(\\hat{y_0}) &= \\operatorname{Var}_{\\tau}(x_0^T \\hat{\\beta}) \\\\\n", " &= x_0^T \\operatorname{Var}_{\\tau}(\\hat{\\beta}) x_0 \\\\\n", " &= x_0^T E_{\\tau}((X^T X)^{-1} \\sigma^2) x_0 \\quad \\text{see Eq 3.8} \\\\\n", " &= E_{\\tau} x_0^T (X^T X)^{-1} x_0 \\sigma^2\n", " \\end{align}\n", " \n", "3. [Proof of variance and bias relationship](https://en.wikipedia.org/wiki/Mean_squared_error): \n", "\\begin{align}\n", " E( (\\hat{\\theta} - \\theta)^2 ) &= E( \\left (\\hat{\\theta} - E(\\hat{\\theta}) \\right )^2 ) + (E(\\hat{\\theta}) - \\theta)^ 2 \\\\\n", " &= \\operatorname{Var}(\\hat{\\theta}) + \\operatorname{Bias}^2 (\\hat{\\theta}, \\theta)\n", "\\end{align}\n", "\n", "Thus, \n", "\\begin{align}\n", " \\operatorname{EPE}(x_0) &= E_{y_0 | x_0} E_{\\tau} (y_0 - \\hat{y}_0)^2 \\\\\n", " &= E_{\\tau} E_{\\color{blue}{y_0 | x_0}} (\\color{blue}{y_0} - \\hat{y}_0)^2 \\\\\n", " &= E_{\\tau} \\left ( \\operatorname{Var}(y_0 | x_0) + (E_{y_0 | x_0}(y_0) - \\hat{y}_0)^2 \\right ) \\\\\n", " &= \\operatorname{Var}(y_0 | x_0) + E_{\\color{blue}{\\tau}} \\left( E(y_0 | x_0) - \\color{blue}{\\hat{y}_0} \\right )^2 \\\\\n", " &= \\operatorname{Var}(y_0 | x_0) + \\operatorname{Var}_{\\tau}(\\hat{y}_0) + \\left ( E_{\\tau}(\\hat{y}_0) - E(y_0 | x_0) \\right)^2 \\\\\n", " &= \\operatorname{Var}(y_0 | x_0) + \\operatorname{Var}_{\\tau}(\\hat{y}_0) + \\left ( E_{\\tau}(\\hat{y}_0) - x_0^T \\beta \\right)^2 \\\\\n", " &= \\sigma^2 + E_{\\tau} x_0^T (X^T X)^{-1} x_0 \\sigma^2 + 0^2\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (b)\n", "Derive equation (2.28), making use of the cyclic property of the trace operator [trace(AB) = trace(BA)], and its linearity (which allows us to interchange the order of trace and expectation).\n", "\n", "\n", "#### Solution\n", "[ref: A Solution Manual and Notes for: The Elements of Statistical Learning by Jerome Friedman, Trevor Hastie, and Robert Tibshirani](https://www.google.com.sg/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiF6sf2hsfLAhWSco4KHfJQCCwQFggbMAA&url=http%3A%2F%2Fwaxworksmath.com%2FAuthors%2FG_M%2FHastie%2FWriteUp%2Fweatherwax_epstein_hastie_solutions_manual.pdf&usg=AFQjCNH3VN6HgCDHtXNIbJtAjEEQNZFINA&sig2=b_zFhNYsupRwqtY62dGnwA)\n", "\n", "1. $x_0$ is $p \\times 1$ vector, and $\\mathbf{X}$ is $N \\times p$ matrix, hence $x_0^T (\\mathbf{X}^T \\mathbf{X})^{-1} x_0 = C_{1 \\times 1} = \\operatorname{trance}(C_{1 \\times 1})$\n", "\n", "2. [properity of Covariance matrix](https://en.wikipedia.org/wiki/Covariance_matrix) \n", "\\begin{align}\n", " \\operatorname{Cov}(x_0) &= E(x_0 x_0^T) - E(x_0) E(x_0)^T \\\\\n", " &= E(x_0 x_0^T) \\quad \\text{as } E(\\mathbf{X}) = 0 \\text{ and $x_0$ is picked randomly}\n", "\\end{align}\n", "\n", "Thus,\n", "\\begin{align}\n", " E_{x_0} \\operatorname{EPE}(x_0) &= E_{x_0} x_0^T (\\mathbf{X}^T \\mathbf{X})^{-1} x_0 \\sigma^2 + \\sigma^2 \\\\\n", " &= E_{x_0} \\operatorname{trance} \\left ( x_0^T (\\mathbf{X}^T \\mathbf{X})^{-1} x_0 \\right ) \\sigma^2 + \\sigma^2 \\\\\n", " &= E_{x_0} \\operatorname{trance} \\left ( (\\mathbf{X}^T \\mathbf{X})^{-1} x_0 x_0^T \\right ) \\sigma^2 + \\sigma^2 \\quad \\text{cyclic property} \\\\\n", " &\\approx E_{x_0} \\operatorname{trance} \\left ( \\operatorname{Cov}^{-1}(\\mathbf{X}) x_0 x_0^T \\right ) \\frac{\\sigma^2}{N} + \\sigma^2 \\quad \\text{as } \\mathbf{X}^T \\mathbf{X} \\to N \\operatorname{Cov}(\\mathbf{X}) \\\\\n", " &= \\operatorname{trance} \\left ( \\operatorname{Cov}^{-1}(\\mathbf{X}) \\, E_{x_0}(x_0 x_0^T) \\right ) \\frac{\\sigma^2}{N} + \\sigma^2 \\quad \\text{linearity, interchange}\\\\\n", " &= \\operatorname{trance} \\left ( \\operatorname{Cov}^{-1}(\\mathbf{X}) \\, \\operatorname{Cov}(x_0) \\right ) \\frac{\\sigma^2}{N} + \\sigma^2 \\quad \\text{see 2. above}\\\\\n", " &= \\operatorname{trance} (I_p) \\frac{\\sigma^2}{N} + \\sigma^2 \\quad \\text{as } \\operatorname{Cov}(x_0) \\to \\operatorname{Cov}(\\mathbf{X}) \\\\\n", " &= p \\frac{\\sigma^2}{N} + \\sigma^2\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ex 2.6\n", "#### Question\n", "Consider a regression problem with inputs $x_i$ and outputs $y_i$, and a parameterized model $f_{\\theta}(x)$ to be fit by least squares. Show that if there are observations with tied or identical values of $x$, then the fit can be obtained from a reduced weighted least squares problem.\n", "\n", "#### Solution\n", "[ref: A Solution Manual and Notes for: The Elements of Statistical Learning by Jerome Friedman, Trevor Hastie, and Robert Tibshirani](https://www.google.com.sg/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiF6sf2hsfLAhWSco4KHfJQCCwQFggbMAA&url=http%3A%2F%2Fwaxworksmath.com%2FAuthors%2FG_M%2FHastie%2FWriteUp%2Fweatherwax_epstein_hastie_solutions_manual.pdf&usg=AFQjCNH3VN6HgCDHtXNIbJtAjEEQNZFINA&sig2=b_zFhNYsupRwqtY62dGnwA)\n", "\n", "Assume we have $N$ train samples, and let $N_u$ be the number of *unique* inputs $x$. And for $i$th unique $x_i$, the $y_i = \\{ y_{i,1}, y_{i,2}, \\dotsc, y_{i, n_i} \\} $, namely, consists of $n_i$ observation.\n", "\n", "\\begin{align}\n", " \\displaystyle \\operatorname{argmin}_{\\theta} \\sum_{k=1}^{N} (y_k - f_{\\theta}(x_k))^2 &= \\operatorname{argmin}_{\\theta} \\sum_{i=1}^{N_u} \\sum_{j=1}^{n_i} (y_{ij} - f_{\\theta}(x_i))^2 \\\\\n", " &= \\operatorname{argmin}_{\\theta} \\sum_{i=1}^{N_u} \\sum_{j=1}^{n_i} y_{ij}^2 - 2 f_{\\theta}(x_i) y_{ij} + f_{\\theta}(x_i)^2 \\\\\n", " &= \\operatorname{argmin}_{\\theta} \\sum_{i=1}^{N_u} n_i \\left ( \\color{blue}{\\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij}^2} - 2 f_{\\theta}(x_i) \\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij} + f_{\\theta}(x_i)^2 \\right ) \\\\\n", " &= \\operatorname{argmin}_{\\theta} \\sum_{i=1}^{N_u} n_i \\left ( \\color{red}{(\\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij})^2} - 2 f_{\\theta}(x_i) \\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij} + f_{\\theta}(x_i)^2 - \\color{red}{(\\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij})^2} + \\color{blue}{\\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij}^2} \\right ) \\\\\n", " &= \\operatorname{argmin}_{\\theta} \\sum_{i=1}^{N_u} n_i \\left ( \\color{red}{\\bar{y}_i^2} - 2 f_{\\theta}(x_i) \\bar{y}_i^2 + f_{\\theta}(x_i)^2 - \\color{red}{\\bar{y}_i^2} + \\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij}^2 \\right ) \\quad \\text{def: } \\bar{y}_i = \\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij}\\\\\n", " &= \\operatorname{argmin}_{\\theta} \\sum_{i=1}^{N_u} n_i \\left ( \\left ( \\bar{y}_i^2 - f_{\\theta}(x_i) \\right )^2 - \\bar{y}_i^2 + \\frac{1}{n_i} \\sum_{j=1}^{n_i} y_{ij}^2 \\right ) \\\\ \n", " &= \\operatorname{argmin}_{\\theta} \\left ( \\sum_{i=1}^{N_u} n_i \\left ( \\bar{y}_i^2 - f_{\\theta}(x_i) \\right )^2 - \\sum_{i=1}^{N_u} n_i \\bar{y}_i^2 + \\sum_{i=1}^{N_u} \\sum_{j=1}^{n_i} y_{ij}^2 \\right ) \\\\ \n", " &= \\operatorname{argmin}_{\\theta} \\left ( \\sum_{i=1}^{N_u} n_i \\left ( \\bar{y}_i^2 - f_{\\theta}(x_i) \\right )^2 + \\mathcal{C} \\right ) \\quad \\text{as $y_{ij}$ is fixed} \\\\ \n", " &= \\operatorname{argmin}_{\\theta} \\sum_{i=1}^{N_u} n_i \\left ( \\bar{y}_i^2 - f_{\\theta}(x_i) \\right )^2 \n", "\\end{align}\n", "\n", "Thus, it's a *weighted* least squares as every $\\bar{y_i}$ is weighted by $n_i$. And also it is a *reduced* problem since $N_u < N$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ex 2.7\n", "Suppose we have a sample of $N$ pairs $x_i, y_i$ draw i.i.d from the distribution characterized as follows:\n", "\\begin{align}\n", " &x_i \\sim h(x), &\\text{the ddesign density} \\\\\n", " &y_i = f(x_i) + \\epsilon_i, &\\text{$f$ is the regression function} \\\\\n", " &\\epsilon_i \\sim (0, \\sigma^2), &\\text{mean zero, variance $\\sigma^2$}\n", "\\end{align}\n", "\n", "We construct an estimator for $f$ *linear* in the $y_i$,\n", "$$\\hat{f}(x_0) = \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) y_i,$$ \n", "where the weights $\\ell_i(x_0; \\mathcal{X}$ do not depend on the $y_i$, but do depend on the entire training sequence of $x_i$, denoted here by $\\mathcal{X}$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (a) \n", "Show that linear regression and $k$-nearest-neighbor regression are mem- bers of this class of estimators. Describe explicitly the weights $\\ell_i(x_0; \\mathcal{X}$ in each of these cases.\n", "\n", "#### solution\n", "1. for linear regression \n", " $\\hat{\\beta} = (\\mathbf{X}^T \\mathbf{X})^{-1} \\mathbf{X}^T \\mathbf{y}$\n", " \n", " so:\n", " \\begin{align}\n", " \\hat{f}(x_0) &= x_0^T \\hat{\\beta} \\\\\n", " &= x_0^T (\\mathbf{X}^T \\mathbf{X})^{-1} \\mathbf{X}^T \\mathbf{y} \\\\\n", " &= \\displaystyle \\sum_{i=1}^N \\left [ x_0^T (\\mathbf{X}^T \\mathbf{X})^{-1} \\mathbf{X}^T \\right ]_i \\ y_i\n", " \\end{align}\n", " \n", "2. for neighbor model \n", " \\begin{align}\n", " \\hat{f}(x_0) &= \\frac{1}{k} \\displaystyle \\sum_{x_i \\in N_k(x_0)} y_i \\\\\n", " &= \\displaystyle \\sum_{i=1}^N \\frac{1}{k} \\ I(x_i \\in N_k(x_0)) \\ y_i\n", " \\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (b)\n", "Decompose the conditional mean-squared error \n", "$$E_{\\mathcal{Y} | \\mathcal{X}} ( f(x_0) - \\hat{f}(x_0) )^2$$\n", "into a conditional squared bias and a conditional variance component. Like $\\mathcal{X}$, $\\mathcal{Y}$ represents the entire training sequence of $y_i$.\n", "\n", "#### solution\n", "[ref:](https://en.wikipedia.org/wiki/Mean_squared_error)\n", "Here $\\mathcal{X}$ is fixed, and $\\mathcal{Y}$ varies. Also $x_0$ and $f(x_0)$ are fixed. \n", "so: \n", "\\begin{align}\n", "E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0)) &= E_{\\mathcal{Y} | \\mathcal{X}} \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) y_i \\\\\n", " &= \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\, E_{\\mathcal{Y} | \\mathcal{X}} ( y_i ) \\\\\n", " &= \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\, E_{\\mathcal{Y} | \\mathcal{X}} ( f(x_i) + \\epsilon_i ) \\\\\n", " &= \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\, \\left ( E_{\\mathcal{Y} | \\mathcal{X}} ( f(x_i) ) + E_{\\mathcal{Y} | \\mathcal{X}} ( \\epsilon_i ) \\right ) \\\\\n", " &= \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\, E_{\\mathcal{Y} | \\mathcal{X}} ( f(x_i) ) \\\\\n", " &= \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\, f(x_i) \\\\\n", " &= C \\quad \\text{constant when $\\mathcal{X}$ is fixed}\n", "\\end{align}\n", "\n", "Thus:\n", "\n", "\\begin{align}\n", " {} & E_{\\mathcal{Y} | \\mathcal{X}} ( f(x_0) - \\hat{f}(x_0) )^2 \\\\\n", " = & E_{\\mathcal{Y} | \\mathcal{X}} ( \\hat{f}(x_0) - f(x_0) )^2 \\\\\n", " = & E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) + E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) - f(x_0) \\right )^2 \\\\\n", " = & E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\left ( \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) \\right )^2 + 2 \\left ( \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) \\right ) \\left ( E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) - f(x_0) \\right ) + \\left ( E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) - f(x_0) \\right )^2 \\right ) \\\\\n", " = & E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) \\right )^2 + E_{\\mathcal{Y} | \\mathcal{X}} 2 \\left ( \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) \\right ) \\left ( E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) - f(x_0) \\right ) + E_{\\mathcal{Y} | \\mathcal{X}} \\left ( E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) - f(x_0) \\right )^2 \\\\\n", " = & \\operatorname{Var}(\\hat{f}(x_0) + E_{\\mathcal{Y} | \\mathcal{X}} 2 \\left ( \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) \\right ) \\left ( \\underbrace{E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) - f(x_0)}_\\text{constant} \\right ) + \\color{blue}{E_{\\mathcal{Y} | \\mathcal{X}}} \\left ( \\underbrace{E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) - f(x_0)}_\\text{constant} \\right )^2 \\\\\n", " = & \\operatorname{Var}(\\hat{f}(x_0) + 2 \\left ( f(x_0) - C \\right ) \\, E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) \\right ) + \\operatorname{Bias}^2 (\\hat{f}(x_0), f(x_0)) \\\\\n", " = & \\operatorname{Var}(\\hat{f}(x_0) + 2 \\left ( f(x_0) - C \\right ) \\, \\left ( E_{\\mathcal{Y} | \\mathcal{X}} \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0) \\right ) + \\operatorname{Bias}^2 (\\hat{f}(x_0), f(x_0)) \\\\\n", " = & \\operatorname{Var}(\\hat{f}(x_0) + \\operatorname{Bias}^2 (\\hat{f}(x_0), f(x_0)) \\\\\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (c)\n", "Decompose the (unconditional) mean-squared error \n", "$$E_{\\mathcal{Y}, \\mathcal{X}} ( f(x_0) - \\hat{f}(x_0) )^2$$\n", "into a squared bias and a variance component. \n", "\n", "#### solution\n", "\\begin{align}\n", " {} & E_{\\mathcal{Y}, \\mathcal{X}} (f(x_0) - \\hat{f}(x_0))^2 \\\\\n", " = & E_{\\mathcal{X}} E_{\\mathcal{Y} | \\mathcal{X}} (f(x_0) - \\hat{f}(x_0))^2 \\\\\n", "\\end{align}\n", "use similar method as above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### (d)\n", "Establish a relationship between the squared biases and variances in the above two cases.\n", "\n", "#### solution\n", "##### 1. for variance,\n", "As we known in (b), \n", "$$E_{\\mathcal{Y} | \\mathcal{X}}(\\hat{f}(x_0)) = \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\, f(x_i)$$\n", "\n", "and also:\n", "$$ \\hat{f}(x_0) = \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) y_i $$ \n", "$$ y_i = f(x_i) + \\epsilon_i $$\n", "\n", "Thus,\n", "\\begin{align}\n", " \\operatorname{Var}(\\hat{f}(x_0)) &= E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\hat{f}(x_0) - E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\hat{f}(x_0) \\right ) \\right )^2 \\\\\n", " &= E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) y_i - \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\, f(x_i) \\right )^2 \\\\\n", " &= E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\left ( y_i - f(x_i) \\right ) \\right )^2 \\\\\n", " &= E_{\\mathcal{Y} | \\mathcal{X}} \\left ( \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\epsilon_i \\right )^2 \\\\\n", "\\end{align}\n", "\n", "##### 2. for squared bias,\n", "\\begin{align}\n", " \\operatorname{Bias}^2 (\\hat{f}(x_0), f(x_0)) &= \\left ( E_{\\mathcal{Y} | \\mathcal{X}} \\hat{f}(x_0) - f(x_0) \\right )^2 \\\\\n", " &= \\left( \\displaystyle \\sum_{i=1}^N \\ell_i(x_0; \\mathcal{X}) \\, f(x_i) - f(x_0) \\right )^2 \\\\\n", "\\end{align}\n", "\n", "##### 3. guess\n", "variance is only affected by $\\epsilon$, while squared bias is only affected by $f(x)$. Because $\\epsilon$ is independent with $f(x)$, it might not possible that there is a relation between variance and squared bias. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ex 2.8\n", "Compare the classification performance of linear regression and $k$-nearest neighbor classification on the zipcode data. In particular, consider only the 2’s and 3’s, and $k$ = 1, 3, 5, 7 and 15. Show both the training and test error for each choice. The zipcode data are available from the book website www-stat.stanford.edu/ElemStatLearn." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false }, "outputs": [], "source": [ "train_dat = pd.read_csv('./res/zip.train', header=None, sep=' ')\n", "train_dat.rename(columns={0: 'digital'}, inplace=True)\n", "train_dat = train_dat.query('digital == 2 or digital == 3')\n", "train_dat.dropna(axis=1, inplace=True)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": false }, "outputs": [], "source": [ "train_dat.set_index('digital', inplace=True)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
12345678910...247248249250251252253254255256
digital
3-1-1-1-1.00-1.000-0.928-0.2040.7510.4660.234...0.4660.6391.0001.0000.7910.439-0.199-0.883-1-1
3-1-1-1-0.830.4421.0001.0000.479-0.328-0.947...1.0000.6710.345-0.507-1.000-1.000-1.000-1.000-1-1
3-1-1-1-1.00-1.000-0.1040.5490.5790.5790.857...0.3880.5790.8111.0001.0000.7150.107-0.526-1-1
\n", "

3 rows × 256 columns

\n", "
" ], "text/plain": [ " 1 2 3 4 5 6 7 8 9 10 ... \\\n", "digital ... \n", "3 -1 -1 -1 -1.00 -1.000 -0.928 -0.204 0.751 0.466 0.234 ... \n", "3 -1 -1 -1 -0.83 0.442 1.000 1.000 0.479 -0.328 -0.947 ... \n", "3 -1 -1 -1 -1.00 -1.000 -0.104 0.549 0.579 0.579 0.857 ... \n", "\n", " 247 248 249 250 251 252 253 254 255 256 \n", "digital \n", "3 0.466 0.639 1.000 1.000 0.791 0.439 -0.199 -0.883 -1 -1 \n", "3 1.000 0.671 0.345 -0.507 -1.000 -1.000 -1.000 -1.000 -1 -1 \n", "3 0.388 0.579 0.811 1.000 1.000 0.715 0.107 -0.526 -1 -1 \n", "\n", "[3 rows x 256 columns]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_dat.head(3)" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAAFRCAYAAADJg05/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXuobVd1/7/nnH32ed0bEyN9KG0M0ghSEXxhK5G0kBL7\nl6KUaB4K9o/6gKBXkpjExEebVJEmFKLeNP0h3ii3FoWYIlaComilvkN9FSpC4jUV1Fjvveex9z77\n/P44jL2/e+wxxpxr7XXO3mff+YXFmmvOtfea6/VZY4w551oLe3t7eygqKioqmkiL065AUVFR0Tyo\nwLSoqKioARWYFhUVFTWgAtOioqKiBlRgWlRUVNSACkyLioqKGlBr2hUomkzcs033cqu6vLu7i16v\nV3kuaV7+0z/9U3zhC1/A9vY2tre3sbOzUzvd7XaxtLSExcVFLC0tTZT+h3/4B9x8881YXFzEwsIC\nFhcXR9J6nlpveXkZKysraLfbWFlZGaRTy9Zvjh8/jq2tLSwsLGBhYQEAxtJWXlRedHgqlukR16ze\nNBdddNG0q2DqD//wD8fyZqWr9dLS0rSrUDSBCkyLTM0qpIuKZlUFpkWmZsVaOwyVB0dREyowLbrg\ndSE9OIoOTrUaoPb29vCe97wH//3f/412u42/+7u/wx/8wR80XbeiokNRsUyLmlAty/TRRx9Fp9PB\n6dOnceLECdxzzz1N16uoqKjoSKkWTL/97W/jyiuvBAC84AUvwPe///1GK1VUdJgqbn5RE6oF03Pn\nzuH48eOD5VarhX6/31iliooOU8XNL2pCtWKmx44dw/nz5wfL/X4fi4ulLWta0p2262p5ebmJ6gz0\n2te+ttH/a0qnT5+edhVcHTt2bNpVKKqpWjB94QtfiC996Uu45ppr8L3vfQ9XXHFF0/WaW006Skkv\nLy8vY2dnB/1+H3t7exNNPIpJp628qPx1r3sdHnzwwcFopk6ng+3tbXQ6Hezs7LiTrMfpXq83GIHE\no5l47qX1/N/+7d/wqle9yh3xJKOHolFRPF9eXh6McOKRTTqtJ2u9yy+/HD/72c+yt52aN6Viueep\nFkyvvvpqfO1rX8O1114LAKUBqqYEjNaQ0Ny85eVl9Ho97O3tDYCamntl/X4/Cc1UmaQBYGtrawBF\nhiVP3W53MASVh6b2+/3BnENI1o0t+8Tpvb09LC0tYW9vD4uLi+j3+4MRRt1uNwRoCrCctuKtXAep\nvzXp4ynHTB4IuZPsI4DBcgHg4asWTBcWFvDe97636bpcUNKA1DBIrcPzXq83uEFTNzBPel0PljnL\nOg8YhamedD5DVf6D62YdOz5eMrFlyssCVDleFiAjeHpllkeQAqc+ZjLJMYssa2vOx0MA7zWqFcge\nnMqLTqYoD6DWcpTW8Kmbjm74qmXAPhgYlpZValmmFlC94xdNlhUH7FumFixz83hZ6qZhqsGqj3f0\nAJKXs1iTWNj9fh+tVmuwbY6bR+0XGrxFzanAdMry4JmbDwwtUw00nRctSzoCZ86k3XyBKM91mqGq\n6xfB1DsuAjEPpmyZpkCaWieCqbZMrXPEMWc5Zq1WawBPSfOcQxZ7e3totVrY3d0dsUpzXP0SDmhW\nBaZTkAdLqyw1ARi7MXOtx0nBGU3AEKYCSnbnO50Oer2eaZ1qoAqYrOMnMUILonpZwCGWaR2Y6slz\n87VVWhWmMu3u7g4A2mq1Rv5fxHWR/eZrStbxrsUC1GZUYDoDkos+p0VerwNgBEI6beWl3kkaufyR\npevBVKCZM/fcfLnhLYCKRSYw0QDlOeDDtM60u7vrxkq9B5hMy8vLgwZEcdk3NzexvLyMVqs1KNfH\nQrqw6QcBXxveNeY14HllRfkqMJ2iqoLTWgYwcoNOMuVC0wMF5wP7MLW248VJowYojgnKXMNUz+V3\nGqaWmy//6U1euTQA5bTYp8AKANvb29jd3R1AlEMEGpQWSDns4IGzQPNgVGA6JfFNEYE06s5kufmW\npRdZgZzngTEHpnp9AIM+ohqUqeUUTAUI3A1I5mydMmgYiuJS5wBT5+llgakVI2UoegC1WvPlOCwv\nL5tuPdeBey/o68KzOKP8Atr6KjCdolIWqY69WWUARmKPXkwymhiqHkRTaQumm5ubY7FBHV6Iyvn/\nBJgc15SbnyEqaS8WCgzdfCD+BEiqDBgOpdbHgSHaarVMgHa73YE7z25+u90ee6B4Fmmvtz+gYXd3\ndwTs+jqT3+n8AtTmVGA6BWmrVOZRI4a2fNil0xaP12qeSleBZaoMGLqsk0xsmWqA6skr05Ypx0yB\net9W4v/2XHsBKjcmSXxU5pwG9i1TPseeRSpWsYBU/8YCaIHnwarAdAYUWageTGUCMGLpaEDqfp3R\nsgaYBU+d760H7IMhF75emYZpFNNMrSfSbr6kc+ecFvfaipFa4RAN0FarhW63O4iZbm1tmaEc2R7H\nf7nvqX7A8rWVAmpRMyownZJyAJqadGs+AzU14sjKFzffA5xXZuUBozC1rDdvv3QZNzTlxDGjZWDU\nzZeynLmVt7S0ZFrUEqqQvqHetLS0hOXlZXQ6ncExy4mTyv9a4QDLzY+AWoDbjApMZ0SpVn1vAoZu\nfgRSa4y8TlsWZ900MO6yWg+DnGV2WVOWYk6eWKa6XOfllLNlyDAVyGorlDvfS1pip3LMtJuu3Xqx\nSOV/c9z8AsyDV4HpFJULUAtEbAFalqlANXpDk3ybXtbjBqiUtehBVJYB22VNpb1yoJorbs1F2jLV\n61QpE5gKODVMdYzUG9UkI5q2trbGtm3BVP6TPQINUrnGcoBa4Dq5CkynJN3wxOmUVaotQG6AsqxT\nBmc0Zzc/5Y6n8oH9BqjogVFlAupbjxoSbJla5bl5CwsLI7FQhihbjr1ebwSaXhrYhynvDw+FZfhK\nnJWByteEB8oCzYNTgekUxPCUea5bb8UmPZBqCzSaLJhGcc3UesB+Nx/eP2/fc9fJhV5q3W63m/0/\nqXKBnABUIMovJ2FwWlMdmLJl6sVMc4BaoNucCkynLA0Tz/X1gArkufkMzq2tLTNtWaZ1J2BomXr7\nnJNvNcJMKm2ZTiLu4+lNGq48cQwUGMLUc+1lYqs0aoAqUDw8FZhOSZElqrvX6M70PAFI9iNNdZmy\nWvMjC9kr43wAg/pFMUgvLwcAdSHBr64TWS3gOct8DnUXLF7XOtcCYT5mnU4Hy8vLg0Ypdue9Yb+e\nd6CtTAusks/pAt96KjCdgqKbS3et8dIyBxC+xd4a7eSNf48af0T65hNXFBj/FtjS0lJ2d6VUmaW6\nwD127JgbYtF5qXW0K673Q2SFchYWhq/wE3mDF+pMOb0bPHgWsFZXgemUpN14tjCsceuc5mUAppWZ\ngqhl3WiI6pilyLo5BaK8rrioDBieUq+808NAdR0s5awrMPUeHqllTjNMU/VOgRTAyEPV80a886fT\n3gOKQZmyVIvyVWA6RUWWqW6h996wBMD8BIgF0tSbmSKgajFEBQr6+0P8gTuGjJfnlXk3fd3ljY2N\nsWOfWvbWkf3W1ml0zuV/dD4QW6ZWCCgCquy3N8l29fEtEK2nAtNDlnb32DJliHpvdtL5AEZa7SML\nVVu1FlCt+jFUrRtNv19U1m21WpU/DudNetuTpI8fP27GGXnZS+tl2X/P0tbnXdLyW8mXfWzazc+Z\nUipwzVOB6ZSUipkyVKO3QQF2zNSzTBmk2pJhayuySkUaVhqmuR+Fy1nH2l7dvGPHjlW2+rT7LPFq\nACZELQDpsIluJALGYVrFGtXLqZCK9a2oAs76KjCdglIt+dolt1rjNUwjkKY+C6K7NOmb3oqZ6mVu\nMRZ5XYByJr2+3q7VoBLNOb2xsTHywLLSXpkXcsi1/HTLPx87AGMWcR2gMkwFomL9MkilHgxVPn8F\nrNVUYDolWQ1QfNN6INUfpwMwBlLv65/8vx5QdR09aQjoZWAUpnroZJU0xyF1g0qd+bFjx0xrXS9L\nH9Hd3d3Bu0MtOFqNPN5x1A8c/fu68VJrkvCLPn78aReuH9fNOp9FsQpMp6BU16gIpHoCMPKykqhb\nlIDVu/ly3XovfqqX9Qs9vLcmpfKlixVvw4NqTh7D1IpN87IANRUHTYUW+LhJ2vpNVWBGk4CUY7K6\nbgzW0oo/mQpMpygNVO1iRlAVcALp1vyoa5TVOi3yYOk17Og8ti71JG+Xt5Z1mXSytyxAD5opmFox\naOkkL8tW/1E+b9yQlAKQ9aCy8uoC1It/M0g5rb/eyl29OF2UrwLTKciyTPmGsEBqvYN0Z2cHwD5M\nrdFNVd38XIuT86IYpXbv5WubddLy3ylo5sB2Y2NjcGzkXaIM7k6nY0JUnzsGjmfV6/xUw14uMHOW\ndQOTdR6tnhh6f4vyVGA6JXmNT2yZWkNB9Sv0gCFMc4aUei36AtOcWGiOFQhgDKI8tdttM98qa7Va\nISy9PA+sx48fH3nwCFA7nc5YI5gHUx1jthrrdKNcag402zVKQ5LhyhZoNBXrNF8FplNQTj9TbZ16\nL3wGMAIGXj/q7O9ZplYDiufWW/Di32s3n2EpwLTmOq1hGsEzB67Hjh0bWPfWG52iEUz6wcfnU59b\n65xb63DactnrWqdWPQSi/CCwwFpUXQWmU1LUAOVZpyk3X3fut7pGefFS3ZKvlYKoNVrJcvM1KGV5\nZWVlJE+XV4VpBNmNjQ0sLy9jZ2dnBKaeay/nSx44u7vD95fyucyJi1rWH4dYvBhoFaAy6FPnkbtq\naau6qJoKTKck64ayWvQjoDJMrZZpr8O+B1Vg2HVGxDFBD6jWuHRg1DLlOKiG5crKyghMJc15dcCZ\nsky1VRrFSfmhw+8vBTDS2GO5/vp86/W1ZVoHqN751OeR5wxRyzItQK2mAtMpSC7SOq34+h2lwBCm\n+rfesncDAgj7IIr4hvTG1gMYa8nXVqlA1Jva7TZWV1fHLNPUC1JSFvOxY8cGcGeY6ocBQ4XPFX+2\nWR4ybF3ygykFVN3413TMVEtb9RZQed+L8lULpr1eD7fddhvOnDmDbreLv/mbv8Gf//mfN123uZSO\nk3kd93OBCmAQK82ZuPFJu/oi7nNoWaQy18DSY+kty9SySFdWVrC6uuqmtWXqwTQnH9iHqYA0Gv+f\nimkvLS2NWKM8silq3bdCPPwfdcBp/U6Lj4muqxW/LaqmWjD97Gc/i0suuQQf/OAH8X//93941ate\nVWBaQZ6LnwKq9TkSYGiZajc+yku5hQAGbqyWBywLprovqUzsyjNAV1dXB9PKygrW1taSMPXGnVvL\nwD5MrdcD6vOjh/jKXINYn1tdF3blPZDK8a8CzNRydL5y3PsC1mqqBdNXvvKVuOaaawBgEEMqqiYv\nVsrwi2Kl2s3XoLTgGY1D165pqnU3AqlA2Opjarn5DNDV1VWsra2NpNvttvu+UC9m660HDGFqWaYs\n7yEnHfzFMuVzKlY9w9k65/rc58C06mSdq1RIpLj59VWLgmtrawCAc+fO4aabbsLb3/72Rit1ocjq\nKpNjweiuOVa5hqDVRSbHwsx5o5P19icAblenKGZqWagRTKumAWB1dTWrNwXHmflTIvyQsGLK8t85\nYNUNV9Z59K6B1GS59FYnfatORdW1sFfzyD355JN429vehuuvvx6vfvWrm65XUVFR0ZFSLZj+8pe/\nxI033og777wTL3vZyw6iXnOr3d1dbG1tjU2bm5tmvi7T6z388MN45StfWbuRgqeFhYXQ+qxS9oUv\nfAGvec1rxhqTtOWZKtMNUFUan6y8Sy65BP/7v/9b+bh7+d1uNzuckkr3+3087WlPw/r6+kTTxsYG\nVldXzbdweXnRcgnj5anWUTp58iR++9vf4sMf/jDuv/9+LCws4MEHH0S73W66fnMnDwTe6+r0Z317\nvdEviALAyspKst9hTr6GaQqkUTmAQewz6vrEYQD91iir72dOt6hoHX0OvLBG6iUt4v5b51TygNFQ\njn65CLvfTUrbRzn2UnHtJ1ctmN5+++24/fbbm67LBSXrZrZuYv199Ha7PQbT1dXV7FZeC6ySZ8E0\nB57WMgCsr6+PtdpbI5302HyrM30VYEZTdOz1XL+D1QIq/58WxyBlO5IWkPL10KRSQC3wbF7Ffp+C\nUi3hliXE4NMNBdoyrZsWmFqgrAJSgena2po5sonT+sUmGmDcO6AOPD2YamvXsrojL0EapTxFjYle\nvarIa1yMGh0LUA9WBaZTEt/QfAO3Wq1BX0YNUa+lfnV11YSjN/fKqsA0yhP4bWxshK35erLcfK8/\naB1LNLJMLe/Ac/X5vPD51FBkiMp/S3oSkHqy4FiAeXgqMJ2CPMtU37y7u7uDGzcFUw3dHKjqdRmm\nKVimygAM+oh6b4fyXrdnxU1zYJqTn3P8UxDlh52WZYkKRK3wRB1F1mfR9FRgOkVZbibfyALI5eXl\nkT6HWuzme6DMKbdgWgWgGqbr6+vhO0u9d52m3uRUxfqMYGpZphFUd3d3XctU5PURXlpawu7u7sh2\nqkA1Bc1cwBb4HpwKTKcgyzKyQKqtUcD+eiS7+RYkc5dzQVkVpqm36eu8Km6+l/bKZK5jpt5+cNiF\nPQU5fnJOcgZcCFAnjZeKcqFZAHo4KjCdklIxu1arZbr2POJGbkSxTCNY5uRXganVd9GCafTdp1Q+\nW8J6n3OgmsrL6RYlIGVPQeqsQy6ee8/H17O0LdWJgVZtmMptvCpKq8B0CtKWiXUTa9h5/wHYlmnu\nxL/xWrLrgBXYj5lagEwtW6EFvc/RPHedHFef3XsddtEPOcmT8ycwFZBaMdMUUHNU3PrZUIHplOQ1\ngMiNKOO+vTH18htgCNNJpyowTY2qAYaWqS638qK0AEj235pHZXqeEysVq5Rhbx2zyMWXBxX/L1uo\nua5+LhBTLykplufBqsB0CopakzVIdYzUgql283NejmGtE8G06pBEYB+mKas2FVpg8Mgx4OORyvPK\n2EqMgMquvQCSrXnrZSkePL2BBrmqAsMqsCxgbUYFplOU7meq42xVYGrBUeellvXInxyoemXAPky9\n0UW5c3bz5RhMmo4aAL3WfKsnhAdTC6raCp608UmUskajsgLRZlVgesiKGkGWlpZci1T/hiHDbj7f\n3DlpXtYwrQpRDdO1tbWx/fNc7NR6+ljkLHtlObFS3jcvHq3debZcOUwg/6tjpnxOD0KTNmAVVVOB\n6RTkNUDpuJu1nr7hgSFMLWBGebosF6bWmHVdBgwtU+1O10nzsbOOZ5W8KMySC1Or4Um/80AsWm1x\nV4mXWoqs0WKFTk8FplMS38wMspRFuri4OOJCAqNu/iRzD6YeNHNgquOTdZf1MfGOaU5ZlZZ8Cb1I\n2mpoktZ9BqlYpdrK1vuXqvekqtp4VVRfBaZTkAapgGxvb28AIr0eQ5QtH2DcMuUbvUre4uJiEpi5\nc2AfprwfGo65+frYTXLcZa7BxiDV8U4NUp3mLlM83LTX643FYw+iJV9bqlYvEGu9omZVYDolWW6+\nV6ZvdG7kAIaWqQfJVJkF0xRIU2XAfszUawDy8nKtUOt4CiSiNB9fyyIVkGqLVM8tmLI1ajU+WRa3\nZ50W4B09FZhOWdYNLvFSDgFEVoWEB/r90W+gLy4ujtz4vCz/KQ1PlmWqW7h5zmDIcclz45reMZJj\nIsdAp71uQzrNv7Nc9qh13nLl+cuv3usNrThrykq0Gqg8y906H9a5iUB+GCGHeVeB6RTkWWP6BhHA\nAbGLJjBlkEZAZXjoUINlWVkNKJGlFbnmdSxQa9+9dBXlAtVqXIo+me11n7K8hWgfvOMWTRZQo/Nk\nAVVvuyhPBaZTUnRzWCDl7kFaEUw1QD23PwVTr1tTBFHeT53OPT4HIQ2xHJB6QOXPyfC3nLxuVNZ5\n0CDVxyxniqzRCKQWRAtI66nAdIqyLAO+CUSe1SK/5dCAd7PmQFa783Vg6rmMVdN6Xy13fhJpV5uB\nmmOVapDqdbx+qRFE9T5bFn4uVHMgav1nUX0VmM6Acm6QSFYfVQ8W0bSwYL9pnxtoLDc/1TpdFZ5a\nqfjiJLKs9Byr1HL3vc79FlD1vnmqYpFa7r6Xlv8uak4FplNWHYBqcFkwBdIQ1esITK0Gp9x4qdSP\n62qlrX06THlhD29kU5W4qQVWL7RSRx4kowddTpw7cv+L0iownZLkYhWIaYhKyzxDVdbp9/uDZWAU\npoANipx8hqkF0Kpx0wikVcDapHvPioAaQVS7+rlufm7MVPZ5kik3VpqCZwFqvgpMpyCGgwaq5MkN\nIS3uwPhb9gWqKZjqPG9ZWzipsfS5N2XVZa2mQard7AioXpcoz+3PjZV6+xRBzbpGLIBO2gBVVE8F\nplMUg1SWZc4g1TFDAann5su6uWn+38hdTE3W/qXypnXzWsfBi3NaLzJJWaVVLFNPOQC1gOlBNQKp\nly7KV4HplKRBymkNUk4zSOVG9CxTSVt5VjnfdDnxt9QNqvd3kryDcPOBvO83RVaoBdWo8clqiErt\nWw5QrXPk5Vkg5W0V1VOB6RQVXcTcbQkYbyiSG1PWlXV4XZ3nzdky1TdidJNGN6i3X6l8S4cRL7Vg\nl9vH1Bv9ZDVAWR6EJctb8QDq5TURMy3KV4HpjCi6kHV/UIYqgMG4/ioAtebeTZqTtiydXMszp6xJ\nebFlr9O+5eJHQE1Zp5aLb0HVO54RVKs+9CLrtMC1mgpMp6QUVDz307IoPZhaeVFZFesnBdQq+5t7\nPJpUBNScxieGqDU2PzUKircdHYtc997yJCyQeueSt6HrUJSnAtMpSMNB3zy6LAIpMIyZiiZJV3UL\nI3cx54bMWecg46UyZ9AxVKNO+7kNUNb/51qlks6BKoNUp6u4+AWk9VRgOkXJRcsuuy7nGKmsqy3K\nCKZ6OSqTbXo3r16O8nL3fdqKLNOcBihrFFSui+9ZpiILcjkQ9SzQnPj2rJyXo6h4nGJCv/rVr3DV\nVVfhpz/9aVP1uWCUsjQi141HJQEYpL3JeuGz9c361Lfrc0ZB8b5Z+zwLN2wUQolipgJWq8N+3fH5\nWt5DNXeq2y942udkHlTbMu31erjrrruwurraZH0uKLHlGeWnrEl+sXS0Xm6ZBUWdF5Xp/FlVDkBz\nukVVgajVLSrXO9BlkcXZ1FSUr9ow/cAHPoDXve51OHnyZJP1ueDkufqpEICOmUaqE3P0LKSq68yq\nqrboR659bgNUjmWqZV0Tkq4ysKIuMI/K+ZwF1YLpZz7zGVx66aV4+ctfjo9+9KNN1+mCVOQap5SC\n6VGTWOY63bRyWvTr9DXN6WNaFagWAHPjp5H1av1PUT0t7NW4Uq+//vrBQf/xj3+Myy+/HB/5yEdw\n6aWXNl7BoqKioqOgWjBl3XDDDXjf+96Hyy+/vKk6FU1JVVr9U+u22210Op1K/VyjdbzlnH1hXXzx\nxXjyySexvb2Nra0tbG1tmelUuaS73a7r8murNbXe+fPn8fu///s4duwYNjY2sLGxMUivr6+P5HOZ\npNfX1wfptbW17JFsqTzrvQtF45q4a1RxC+ZPOcBL5QH7jZSS53XtqpJO1TW3fGdnB9vb29jZ2RlM\nnU4HnU5nJN3pdNDtdgdznnq93hhIxbWPWuplNJsOzch9tLy8bPas8HpaWL08ouG/kavP86Lqmhim\nH//4x5uoR9GMKLIWq5QBwO7ubjJWmIonRjCtC1mxKgWonGaQ5gDV++4T15vjlnt7o9/z4jIAaLVa\nY0C1Pq2d+qRMLkgjoBawVlPptF/kyoNeThkwhKnXCFMln+vk1TUnD8CIRbq9vT1ilWpLlYGqISpT\nBFKRhqbOl/UFpJaFqkFqWaNVh5JyHSRdVE8FpkVjqmJBemUARtxeq69llJfTjSgnnmrlsSUaAdSy\nRi2oRvUWaZjyMo/Xz7FIOc0DNqLuUDluvtcNqyhPBaZFSaVccA98Fmjqprkuum5WfaNldu8joFru\nvTX6KRWa0FafuPtikcoysG+ZWm5+lU9wpwCaslALSOupwLRoRFWAGbnpANwO7AzKVAd3SXP9dH1z\nynjZipVqoHpw1Q1Q+i1QfAx1HRia3tyyTHOH++Z01E9ZqEX1VWBa5CoHrNFQSYZpNDIoVWa5+bkQ\n9dx8DVCrRT9y8QWoDHqvDkB+o45nmea05Fut+ZNORfkqMC0y5cVDvdimngMYGwnEI4ly8jit6xWl\nU2XS6JRjjaYaoPr9/gh0rHSVcgEnA9WKldaxSuuAsgA1XwWmRQNFDUpVY58ARroN6SGZUZ6V1nXM\nSXvl3A0qslBzukYJTHO6GXnxSc5niOa06OcC1dqml1dUTwWmRSPSALKA6sU9dYzTG68ejWX38rlu\ndSHKMdOUJZrbaV8akHRMErBf8Bx1VwLiBijP1Y8aonjbuh66zFouyleBadGYPMvUgqo3AeMwTX1v\nPlqH66bTVSGrXXsdK001PPEEAP1+f2TYpcylxZ7BZfUD5d96EI1A6vUvlW3K3EvrPM4vyleBaVFS\nua3vOsaZGoueM6Zd0lwXa14lb2try7RCIyvVa4ACxr8kK3ksDU4LhMDQMrU67+cOK9XWbgqi1ryo\nugpMi8ZUxzLVLfIATHBaX/ZM5QlMU92KctfxoLmzszNw63M77AOjn41hKPE22a3XrrmkAdsyjRqg\notZ8XR9rnrtOUVoFpkUjqhMz9VrfLUvU++RHVCbdo7h+Veeclphpt9sdAyjHSb24Kb/kRP53aWlp\nJB7Jlqpu5GH48QSMv+jEA2sUK9VveqrTw6CougpMi0zltOZ71qlYkjoWasUcU1O32822THPn8uo8\nC54MzpwGKIbT7u7uAJZ6m9oyZRgKLAH/RSepBqjIzRdFy6l1i9IqMC0ak7bmrI75EUgFfhqKddNs\nAep5HQuVYRpBMxpSKnNtdcrxYKCKNEQZpMvLywD8F53UAarIAqMHywLR+iowLTLFFqlejixThqnl\n5msY8dzL89z8HIBGMNXQ1FPqPaYWTPWLS4DYzWdYAnkvOkm5+tya76lAs3kVmM6BtAXk5aXK+v1+\nVvelnAmAG2fUjTjeOlZs0ptXKcttXMr58qhYoGy9W1Mkr59nqtHIszgjUBaIHpwKTOdEk7ZwCxCa\n+gQHAJzk75/pAAAgAElEQVQ/fz7pvueW5exH7rwONC1YWsc3p+eDWPGLi4uDuUDOG8jgDY6IIC5i\ngFpgL4BtRgWmcyDLHed0alnSXut7na5NALC5uVm5ocmCahOt+QwR6/81UL3RXRFI9VzHmj2QSsMV\ngBCi1hRZxfKfGqAantxIVlRfBaZzpMjNzJmsbkq5eTof2O8cz5Cs8nudL/vH+1pnDozDNAVUD6TW\n8dfnIRp+K8DkGKcGaWSd6u1YD0uWBVfLai1QracC0zmQdUPpmytalrSGWd2uTAJTcfOrWLZeGe+r\nNY/KNFQswOdYpp4bzecg19UXiApIdaNdBNKculkgtSBaANqcCkznQJ6Voi2iVNnu7q7ZYl0lzTDd\n3NzMiq+m0jIIoApIvTwAY7C2QGpZp55lqreZAqkHVgBj245A6j0ouZ4eNDkMoJd1WVGeCkznRBZI\nvfial6+hWLUbE5cBQ5jmTF4jEPcOkP3kuZf2yoFRy9TbvheXtP5PnwMNUh67H1moAFyYR+cwxzIF\nRq1TnZb6F6DWV4HpHCjHEsppzGAY6q5CnU7HLPd+A+zHTL1Wcg+aVplnmdZJA0OYpqYcYFlxSA1S\n/h/u2C8g9Rqgqrj1VpkFzwLOg1OB6ZzIurH4prTcRp0n0Iz6YKY6ujNMz58/Hzao5OZpmNZJ87LA\n1IN6CmLWsdfnwAOqwFSDVMdMvYdMrsvP9WKQenDlfSlgracC0zkQ3zyWKxjFAzltgTI11NJbB9h3\n86uC3UvrfbWWc8u63W424L3YZHQOtHvPQLUAyvDygO6B1LOYc0Gq50X1VWA6J/Lc/NyYpcDUe+FH\nKk+XA0M3PxVeiNbZ3R3vY8r77C1HZRIf9bbvxSoZUh5QvUlAymP4Jc0xYX3eci3RnPoBMUgLYCdT\ngekcKIqXphp6eJnBqF9BZ72qLnptHbDv5vPNn4oDRmWyn9a+V1kGhpYpg3Nvb69SrNI7D3w+NEB5\nYoCyqrj2+rhZ8dwCysNTgemcSFsplrsc9euMYJp6C71VBuy7+fpmtwCQU27tb908hqkFcZ3nudDy\n/5ZFmGuhakVx4yr10/vtgbTAtTkVmM6BrBvWs0690Ua7u7shJHO/mSRpYN/N15DUwLTcVatc9tPb\n/yr53W7X3E40ZwvZ++8cgALx95YmaXSyYBrFSnWe7EOBaj0VmM6Rctx8DVXr7U0WUL3vy3tpYGiZ\nRjd8ZFlpmFr7Gx0LTwJTrw6p5cgC9kAqc8AebSS/j2K5Kffee/jU6SJVoFpdtWH6wAMP4Itf/CK6\n3S5e//rX4zWveU2T9SqqIA9EnlXqjWKy4Ol9V15/LllPwH7MNOX2Wm5yymW19j/nGIk0TKvUIwfg\nEVCBNEyt3hapRrEU8Kt02C+qp1ow/cY3voHvfve7OH36NDY3N/H//t//a7peRRVkQVTH3HSDkzV6\nSTckRUDd3t4O84F9N98CVJ203t86x0jU6/VceOrjmSqLygWiks6pW26f1wisMkXj8b2hpNZyUZ5q\nwfSrX/0qrrjiCrzlLW/B+fPncfPNNzddr6KKqgpU7dpbLr6GKgPUSvMcGLr5XEdrnls2ybFhiWWa\nWwdrXW87llVqyYPVJPC0LNPoTVF6HckrIK2nWjB96qmn8POf/xwnT57EE088gTe/+c34/Oc/33Td\nijLl3cS6Jd96w5MFUsvS3N7erpQG9t18q67ePlTJn0TSD7bKNnNDCd7k9UjQ8MsZkaXPb6rHATD+\n1ijJ0/tXQFpftWB68cUX4znPeQ5arRYuv/xyrKys4Ne//jWe/vSnN12/ogrS8TPLorHcfesVet7w\n0ah7lI6ZSkPUYSsFhMjlrvqfXj9SFlt7FmBTME31FbZ6ZsgoqypTtO9NPtTmFdi1YPqiF70Ip06d\nwhvf+Eb84he/wPb2Ni655JKm61aUqY2NjWlXwdRBWJVNqA5MD0vf/e53p10FU/MKwCZVC6ZXXXUV\nvvWtb+G1r30t9vb2cNddd5WDPUWdP38+6xvwqXSuW59T3u12G78mvL6aVdJnz57F8ePHR/6z6TpZ\nFl9O/uOPP44//uM/xvr6OtbW1rC+vm6mU8vr6+tYXV0d+/RzlYm/eNpqtZIPoCrHcV5ZUbtr1Dvf\n+c4m61E0gbzGmlQDRU4MLjXCxorPHYRFmgOnXFe23W5X3nZOudWPNCoDxhu3dCiGwy+dTmfsU8+y\nX/xRPvkfhmI053RuHNnbj5zjNa8qnfbnRFH3HQ+gVfO8/z9sMUQYJFEew2Z5eXnk/+rc/HWBEfUx\nBWDGQbvdLlqtFlqt1gj8ZOIHBZ8/Bq83yXkWkC4tLY3UNepO5e3ThdqYVWA6Z8rpMuNZolW63vD2\neLsHKQ+YVeZAvmVaFQaRh5Cb1g2F3W4XS0tLA5AyQPkBobs7MUwZxNbcGmnGx9iqr7XNqOHtQlCB\n6ZyrLlyruPceIJqUdu+1ZaZjfl4esG+Z5tzkOe59rkcgyzzndUSWiy8DKiJrFBjto9rr9QYQtabd\n3d0BSFutlgtT7nGgj4uGqAXVCwmoBaZzpEnAGbn6KUv1sGRZplUnwLdMU92DPEXHx7L6AB+oVsy0\n2+2OxDkja5Qt21arheXl5bG59eJrfXz5u1W8DXmAWNv3QgIXigpM50DezVoVrHUBelhQ9Vx8qwXa\nSks8UGKmdeBpubJ87PRcrDtZlt/IXB9TC6aWRZqyTHd3d7G8vIxerzcA6PLysglR69jyerINDdII\nol56nlVgOkeK3EwGpU5by1Um3v5Bia0xD6ReC7WGKVumudD08iNL33KTJV+fI1mPQbq0tDSwSnWo\nwnoY6sar5eVltNtttNvtkQ79+oHJ+ybHlRuneF9lPZ2+0MBpqcB0TuUBVbueUfeoKg1RhyHPxdct\n1FaeTIAfM62Tp2HGabYeBaJaKcvUipFav7dGuLXb7bFhqXp71jGV9bmrVAqeqbwLQQWmcyYLdhqg\nGqqpuKn+vfyn3t5BKydmquHJ3Yksy1T+N1qO8hhgDKKFheGnSbSrrN1ztkwtmGq3Xn7jQdQaWppy\n7+XYWeDlunvufgqeFwJYC0znRCnQ5YDUch09EPN2DkNRa76AgMHpdQcC0jC18rxlaTkXkEr9rG88\nWdYpH28AAyj3er2RkAavH0FU4qTyjoWqMNUuvtWaz7+1IJsL2HlTgemcKQVSL25a1b3X29Hpg5Bu\nfNIQYKhqmMoE+A1QdZYZfl4Hejm+VsOR5+bL/ug3XOlz5b3shFvtU669PIAkTsshCz6fBZyxCkzn\nQBpgFuA8y9SyUr31Itf+sKzUyDLV7r01AeMx00mgKhDy3HCZpJ7coMPractUrFwtD6bSr7Tb7WJ5\neXnQcp8LUtkP6YNqtebz/kcAvVDhWmA6J7KswhRUvQanHAtV/+dBS8dLo0YoyyKVfpaA35ofQdMr\ni6AXWaXWusDQMrXW0+dK+pJqK1yAmmqx5+PWarXcsIDUMacVP4LrvKvAdM4UueVRDDQq80B6WNLx\nUgukOm6qQdokTDmt3xsKjINUXGfeD2s9/j8pl3m/3x+MWGJLVFxzK16sQcrH0IIpA9WLmVpA5bSG\n64WkAtM5kna7LQsy6gblxU1TgNXbO2hZUPVcfAGpTIAdM62b5oamyPL3ujfp4yiwZPH/8H5KSMDq\nAiZDRK3jZR0ribHy5IVyNESl7EIGKVBgOneyboCUtRp1i/JAqrd50PKs0sjVt6AK+DHTOkCVj/OJ\n+LhxQ5IOUVjrA6MvrtaWLf+n90o9ttD1sbPAK8cl1anfcvMlXy/r/btQwFpgOgfyAGfBLwXXusD0\n4oGTqkqc1JpkFBDDtN1u1wKolSfj/YHx/rwWVCMLFRiHqUzSeBU9THhimPKIMQ+m/MlvDVWuS+oc\ne5CVPAZySqmQgv7/aavAdA6U4/ZKjE03TPDNq/8ngpee+J2bAi35nErOBR+ts7i4OOaua0hay1Ye\nAKyuro5ts256aWkpC3B6hJZ1PKVuuutXlLb6oUZdp6xve3GcWceh5b83Nzez6pWqcxVZIQZOzxJI\ngQLTuRDfzBqo+nVrluvO/6P/Sw/PtBo6pAXZgql1wUcNPVbewsKCCVNvarVaaLfbbsxUYJqqS06a\nAaQnfS5SDyWpmw4HWCECK4/PJfcV9T7vLSCNXjot2tracvctyhOrWtJNAnDWQggFpnOgHMtUu5/W\nf6S6GVkAZTdRLEH5KumxY8cG/83b8fKidfkVcjlprxxoFqY8ft4Cpj5+Xlrc8rW1NbduXh7nW41Z\nGqKtVmvwCRR+6TRbo3o7m5ubY/uXm5b65YYKjqoKTOdEHgwZpno4IV/cua69N7JIYNrpdFzLVIM0\nAqu2wBiS1ns6c9MAsLKy4m5Xp1Nl2jLl2KQ+jnw82XNgmAroLbc25fZyWrv43mdQBKqphrKtra2x\na8EKX3CPA30Mm4borEG5wHQOZFmmfLNyH8Uc995zQ3Xjjo6/CVBlCCTD1IOkV6b3y2tgsjrmRxOQ\nb5nqZauMx9Bb1pkFUMtCZTc/6nqW6prGD0hr7L7+DApboxqo8r/APkx1fS0L22q8ZKu0yXjnrFm5\nBaZzIA+kYo1Kn0PrIrfiXB5QPYBpoPZ6PQDjMK2b5geDnnR+aj1g3zKtAtFoWYMzxxq1wKphqgdR\nRHPAfieDN27fenu/FSfla0ZgyseTR2DxdaaPk/Sz9azpWQLiJCownRNFQI2sUc8i5ZE1MnGPAIEm\nA5TnwH7MNLcRJVpHxyCtOK7XOOa50ilYWnmeZeo9eHJCI9IIJMdsbW3NHDaq8wRSAEaAyoC1RkuJ\nZer1BuD/4f/a3Nwcs/y5o//y8rJ5jYnrL+6/pVltna+qAtM5kBXzZJfLcq88mMpcIMFQ5ZtS5rqP\nomWZerC0Jm897WJabnJqma0/PnbW8YyWOc9z6T3rk8Gq86Ru3AovL2nmMfPWkFNe9ixTOZ/W2/st\n155hurW1NTjf3nekuC5yXUkfWx2CsDRrbntVFZjOgaKYqeXey2/kRooAwG8j0hDVAOUJGIXppJMX\nx7WmxcXFQcNQCqb6mFTN0w1QOeESL2widdMQFCAtLIy+J1Vgp+uohwx7n0GxHmTyv/x7YAhTqU/u\nW6nEMvWuQ62jbKUWmM6BLAvTagHW6/Hr3tiSkxum2+2OfE9IAzOagKGb703WDe2tkxt/TKUBG6be\nzZvKl65ROpxQZwL23XyGlpwP7zV/AitZZhBaVqmOjWqL1goxAPtuvnwGhUMPUWOm1N96e1VKR9FK\nLTCdE/FFvLe3NwBHFZjqm1fy9DDDaFnygHHL1IJnTp7VSm6lc8oBu2uUdTxz8tllF6uP45JRPNWz\nTNnajyxH3fIuZdYIqOgrpywNVTmXW1tb7uv5WNoq529JHUSL/iypwHQOpMHDF22r1XLDAOyGcXyO\nY6Tsamp4RmXAKEz5Bq6Tzpl0/NKagGpufqqMx90zMAWsuoFKQiayjgVTyxUHxrs/sfsv0iC0YOrB\n2QKxnMutrS2zrzJbuHwMrG9JafimoHrUrNMC0zkQQxQYvUj5fZsCTm7p140drVZrBI4WMHPSwD5M\ndauxnkdl1pyBYMEytR5QD6ZeOT+AGJzcYm/1iOBeEZIG9t18Dk1oC9ICqUwWFOWcRBauFRrQD8bN\nzc0Qjvo4a6BGbv68NEoVmM6BLDfZKud3a3J3FcuSEahawLRG1ug8YBymKUjWAWvV3wB2P9Po2Ebi\nV+KJxanByj0iGK46DQxfdBJZnHLe9Dp6vdTbqjz46hejAPuWaQ5A2aux3vZfB6hHRQWmcyKGhQar\nfktUlO71euZQxKoTMISpB8AqaS+WWnUZ8C1TOXZVJJYpu+4eWK3uZgxUqZu2RGUu50WHM6wwgAYq\nu+OiHJhKvRim2rVnmLL1bXWfku1ax7lq/qypFkx7vR5uueUWnDlzBq1WC+9///tx+eWXN123okzp\ni5vdfG7x5RvHS7NFqm+uKsvAKEw91zzHfdeWd91JFMG0yvEGMOj1YIGU4cng1HOG1tramunW62Ot\nwwAir0Vew9RrqLLCEcAQpvIfMumGNun1oV187eYfFUBWUS2YfvnLX0a/38fp06fxH//xH7j33nvx\nj//4j03XrShTclEKSOVCZcvDalG1YmY6jso3mldmzYH9rlE5cc3c2Kfs6yRpYHKYssTl1sDkF4ro\nNL9lS8qsN1rJ/1tdnVKuu4aolMl/yhBQzxrlXgfAfsw0B6a8n6lGKAuoRxmytWD67Gc/G7u7+9+I\nOXv27OBCKJqeGBg6PqVdLG+uLZVJ0sC4ZTrJZO1n3TzpGlXn+GpxzJTderZO9cAHbyAEMPrWKGlo\n0h3vuYGKgaofigsL4538rYcnd4FjiPJAh62trcFx4POiu3lxTw/PzWflAvUoQLYWTDc2NvCzn/0M\n11xzDX7zm9/g5MmTTderqIIsF04rCvyLtHvowTVnHWAcpjldl7z1co9BTnmTlqm43DwajKHKk85j\n+AhM19bWzBgmu90apCwGpZXPkNZg5nrzMrAPUwEp/07G67OlrUM/Opw0r1rYq7F3f//3f4+VlRW8\n/e1vxy9+8QvceOONeOSRR0Y+oVtUVFR0IamWZfq0pz1tEJg+fvz4oAW4aDrKeR7mrGO17ltTqrzf\n7+OKK67A//zP/2RZohJ7i2KpTalpV5H7Y3rpVJnk/cVf/AX+9V//FVtbW8lpe3sbm5ubbrrT6YTx\n6Sr5//Vf/4U/+ZM/wfr6OtbW1rCxsYH19XWsr69jY2MDa2trI8uS5onz5VxHMe6qZbOgWjB9wxve\ngNtuuw3XXXcder0eTpw40ajrVNS8OJYaraPX0y3i0jPA6nLEvwHg3qxeS7vX+n4U5O2X7Lf0qpCG\nQJnz7wGMveKO3WcNX34FnoyblzTXiSHp5bGsUIHXv9h6OOjXMlrfnJqkNwYfr1lSLZiur6/jvvvu\na7ouRQesHKDyurK+Bz2BaL/fHwGqbMOCaA5UuQ5HQfrYMDwZnBJ/1L/luQanAFODVANU97jgeqUe\nWtax163vORDlngn8vSkGaafTGVwvGuzWsk7r4zZLKp32i8akISp5GhgMUd3lBRiHaXQDeTf2LN40\nniyr1OuapsHqWaYMVS9koEehcduFhrU3F+mO9RZMNVQjoOovMbRarcGDRj9cZS7blmvLu95m7doo\nML3AJBegZ6Hyxazn7L7Lha3n/N/WjZLj6vN81pXjnrKFGv0ewBhI+Y32GqCpEWh1pbvNAeMw1SC3\ngOpZpwJTL0Yr5VIHSUvZrF4bBaYXqCKXnwEqyzznNINU34SWi18FqEdNnlUKjPb15fXZJQdsy5S7\nRnGc1LJM2c33+hTn5uluc5FlGkGVBzN0Op0RmOoucboLF1ur1jRL10qB6QWsHKBKGhiCU9LSSGGB\nQ9bJjY3JdubBxdcxU2D8dXMCUvmNHFfdV9PqtxmVaZhak1em87nengXsQTNKi5Wu+xXrRjo+tpye\n1b6qBaYXuDQ0vYvYy2OQpmDqgdSC6EGA9KDgbNVdu6vebzRMozfxs7tvAVTDNPd9DCJtpaYaoHJc\nfYapdPTnh7Du3eC91FzmltU8KyowLRqJo6aAo+Gpb0i2xqo0OFlQbVoH6RZ6D4zUetw3F/DdfHb1\n2+226XZbMLX6B8t25TxZLyGRdaVeKcvUslL5fa0CU+mfLsNYU41yPGnAz5oKTIuSqgpYzs8F6GEB\n9aBUF6YSOmGY6k8oe8D0hvgyTPU6OrSgGxf1Q9KDaSo2Ko1PAtBOpzNimfKLVqxwg3W8dDhp1lRg\nWjQmDTPPovMaBkQeTOW3EUAPAqwHaZUylBiS3L1HAKbXYfCKNcouvQZqNDFMd3d3B9vgVn4GJ3sS\nIg4FCEwtS1jHbi2w8ktQeKy/DjfwtcPXCB8fKz4/SyowLRqRjqGKNFCtWGsKprKutazTB6GDcPMZ\npDK3uozpY6XXl3K2TDmtIZuyTLnFnK3RhYUF9Hq9sRZxy5X2LNOo9d6Ll/LbpeRcWNcM10Vb0Rr8\nswbUAtOigSxARhYp33z8W5mLFeKBMyfvKMg6Vrol3zte2tLSjU3a4ky9E4GtSYaeduW5XtZ7NTyY\nRiOfvPe56tf65br1OqYc/W4WVGBaNCbLekqtC4z3T2QXl+eS1st6naY1jf+2GuyifIGpWKYCVh0n\nzWnNX1xcHLzaT8Tb0+9Z4HU0mHNCDAxW/SVW7gql68LH0Isp67DALKrAtGhEKZDqcp0HjMNU1tFp\nK88rb3K/DkLeceKbP7UMwGx8EkB6YPUsVMvC1268Drvoct2YFTVCWdYqv/tUJn2dWACNQDqrQC0w\nnQMx2Jr8P77oPXdfl3Pasz6t5ahsEmnX+iCUExIB4q8ciLSbr+eWJcrLDB6Og4oYlNwPmOXFTPWA\nAf1ya8vFt4BqeSgMUpk8F39WVWBaZCoCqWVZsHTM1Pvv3PyqavrhUmWbDDDtOqfSwKibzyCNYqba\nchMAapgyIKWlX1umsp4FUwaoBmmv1xv5mODi4v4nTaRjvuXm87HzgMqfIp91oBaYzoE8y9Bzx6so\nFS+N5I3+qarIUo72MQpFHIQiSzv3uLMrb8VCPZBqmHrWqHzvSebaMpXfWTFTyzJld58/yyJg1TD1\nXgzNIJV6MUiPAlALTOdAVWKPBwmTqF5N/led+GvT9an6n7nr6e9fsXvsfYZbRkRpAAL+R/QErPo1\niXw8dXzX6oLFcNUfE2SYchzUOjaWVSr/LVC1YqdSt0nVxLVRYFpUNGOyXF0N1H6/P7BgtTWqW8gl\nLwKi/rgeNwRp6f+yoCogtUIJGvAMcOtNUmxJ665VTakJr6XAtKhohhTFDhmoXqMTMB6TZatVhw2s\nT0dblir/H2/PA6r+H34/qd5f/Q4H3meBs7WvOZbpYXpiBaZFRTOmHKAyVBmWwHinfJ6s2CfHMz1r\n0mugsqxSDVL9+2hfrU9P61ix5+Z7VjRv6yBVYFpUNEPKtUw9V1fDS7v2GqbciJSyTL3Yq4azblTi\n3+tuW97+yj7rOLF+cLC8Y6HLDwqqBaZFRTMoD6gC0n6/P0gDNigs6MlwUOnapK1d67WJLAuo0vrO\nMI1+y/vH++g1tkVdwnjfrf3XeZJ/EEAtMC0qmjFpmFldhLR1qsEhrfHaFeeXqMhkuedezBTwgaqt\nUu83wOiLw7VbL7FSD6iWm8+WOB+HKK9poBaYFhXNkDRILdc3cvF5slx7gZR+LV4qZiqy3Hy2THPC\nAwDG9oktb/0mf6ufrd5/DczDAiirwLSoaMbkufgpS1RblV4jEUNVu/gWUEW6YUv+O6ehitcHRl/P\nqN17C6gpy1S2bUHVA2rTcC0wLSqaIaUaZeR9oN7vGFJsjcrEn5DWr8bLac33GqE0VL2eBAxTgTe/\nhV/q530s0OoaJfJAedAWqajAtKhoxhRZpsB4gwuvz63yFkz5pc0eSHnSYpAysBcWhp+t5vUYutLN\nCYBpjeqvmEZA9Vx8nU7BtUnQFpgWFc2QLMvU+2KnXk864QtQ9avydnd30e12x6BqwdSLmUo9NFT1\n2/v1sFUeIACMw1TmnmXqvZdA6uPB8zAt1ALToqIZ1OLi+Muj+csFqf6oYqUKQOVLofzFUK8BSocL\nWBqiEWytkVZi7fL2l5eXRz7C533SmkdBWdapZ5laEC2t+UVFcy4LlNoi5fik7qcp4BJrUeKkMnku\nfo5lajVAWfkSYuC5tPTLf0tdNEyjhidrFJRsNwXSCKxNqcC0qGjGpEHKsUspE6AKsBikAjCOk8o8\nBVWvn6nXAMWS7fLEEOX/ZNdeLFIN01TM1IshHwY4LWXB9LHHHsOHPvQhnDp1Co8//jhuvfVWLC4u\n4o/+6I9w1113HXQdi4ouGFmWqc5jiOrYpECO3y3K7nMKpKm3Rmk3n/OkQcoaRaXT/EUBmTRULVff\na83PcekP2kpNvr33wQcfxB133IFutwsAuOeee/COd7wDDz30EPr9Ph599NHGKlNUVGS35nM8lK06\nBlK73R7MeYrc/JxGKC2roUn3ZRUgdjqdwbSzszOYOJ8BmuPqM1ClPjy38nQ3Kr0/TSgJ08suuwz3\n33//YPkHP/gBXvziFwMAXvGKV+DrX/96IxUpKiryG5esN0Zpy84CqQVUhmoqXhpZptZLTnRYQSaG\npzVpNz+3JZ/rxfXjPGu9g1DSzb/66qtx5swZszIbGxs4e/bswdSsqOgCFcdLNRy8fJ0nbn632x0A\ntdPpDEDKDUA5MVPZDjf06Filty+WuB4C23a77Vqm2tX3GqCsrlFSj4PuJlW5AYqD4efPn8dFF13U\naIWKii5kWR3l6+rSSy9t7L/Onz/f2H81rWPHjk27CgBqwPR5z3sevvnNb+IlL3kJvvKVr+BlL3vZ\nQdRrYlU156P1pVVSr+fFZXJiOEAz71VcX1/H1tbWWKsr/7dXZq3blA6zFXWeZLmpkQvrzXd3d3Hu\n3LmR6fz58zh79izOnz9fqeyJJ57AxRdfPFJHnufmyXx1dRXHjx/HxsYGjh07NpLOmTY2NnD8+HEc\nO3YMv/u7v4vf/OY3ZiMdMB42SZVNosowveWWW/Dud78b3W4Xz3nOc3DNNddMVIFpqE4wWrtSKTcr\nWtdSzom01llfX0en0xmU51w0Oi93+0XTV04fSnZtdUu61aBlxWN5RBKw75Zb13eUlmWuE6/D8Vcr\nz3upNcdpAYwMIMiF40Fc71kwfdaznoXTp08DAJ797Gfj1KlTjVfkMJQTkPZgxyfcOvlV8oH6n1C2\nynZ2dpLdaLz83O0WzY5ygOqdb2vygCoTAPP9qZ7BYMFTpy2jQ7fUW531NWABjPS59fbdut71MZ1U\nF0ynfe/k5iwDME903bl14nLzrPxOpzNys2hLhPv+Wfnyn9Z+F8BOT5HVmbtOFbBa1imPpfcAahkR\nosiAif5Ld4Oyeg/IoAG5p/b29saAygC1jl2TuiBgmnpK5qSjJ6VXZuWnTmgOVC2YWm4cTwJOueCA\n4Uc5v7oAACAASURBVPhvvhG9Y1egenjyHmxcpt35yDKtY5FyGhi1TLWBsLCwMJIn4msqskgjOFuu\nvmWZ8gAD3m82GCzjget36DHTo6bIxahSbp3M3Dy9HMExBU5r3Z2dnZEbRN8wktZWA/9P6kIqUJ2u\nIquU07muvvWpELZGuR8rMIQpj7mXbekx+rKsQSrKCRdErj1Psj0xEmRuvR+A65IyIupo7mEqikCZ\nmgNIBsOt4LiVp10wlgfVVFosU/2qM764+H2Y/AS3XLPo4mry4itKKxeinNYWWmSdRi6+ZZmyQSAW\nqUjfL1HslBXBMzUBQ8tU9pmPEzD0wGRbfN03CdQLBqZa2t3QeTwHMHYS9RPS6lzs5QP1welZpnwz\nSB0FoBL3sv7LullTVmixUqejFEQ57VmlKevUcveBfZgyRHkSeeW8To5VWsVCle3yMRKxN8Zhrqbd\ne9FcwzSCZJUySfNJ1KMzomVdFoExp0yn+bMPciNwjFZbnvzpCM/15/0v0JyuInAC47FJfZ1EFmrU\niq/dfAaZhqQGZip0VAeg2utjy1R+67XY65FjGqpNaK5hykq5+SmYelYpD3uz5jodWaapuZe3s7Mz\nuPgtgFruvRdLsi4u76IrLv/BSXsMHlCtMpnnuPheX1OGKzCEqTyIveuBY5e6PrwOpzVIGaicZoh6\nlql3HK2paV0wMAXy4GmdaAAuSC14RpO+GCedA/uWqdRP9wcU8c0l61qWqQfVAtTpy7NKvWXP1c+N\nm2o3ny1SuY55m9ris65V/k0E0pSLr1vzvWPFxotlZDQJ1rmHqT5QGhzeU9Iqt6xTDUt+UYOVrgPT\n1Do7OzsjdUvBVKxSva7lquljV8B5eLLi2ZZVCtgPwaoufsrN1xDVknvEi5fyet59lwKqbtgFbMuU\n7xGul3VvN6W5hykQd83Qy1UtU++1Y16aYVoXoDqv0+m4T1n9Ow+kss8FqLOlHKBKHuBbptyZXXeZ\ni1x8dvMtOGoQirfDULXW17+NXP3IQgXGLVPxvvg4eG5+k0C9IGAK+E+i6CBbME0BNTX1er0kNFN5\nVWHquXdy0erjkQKqdQwLYA9OETyBUSB4104dF9+CKW9vb29vZGQU9z+1rlPLK4yAasVN9f0HDHsR\n8P7ycWOr1ANpE1C9YGAqik6gt47kpdx8fhGulxaY5oAyF6Y7OzuhW883knSdsvafL/oIqAWehy8L\nqFwmiq4BDdRU4xMPJxWoshiiMmeg6rrp30XWqAdWr5+ptc8Mdguq1jGbRBccTCNFkI2sUssC1W8R\nl3RkmaYAapUBo5apiNddXBx+2Ew69esnv3UsCkBnS5bL7K1TxyJNxUx5u9ak/9sDqV7WQPUAavXb\nBvz+rcCwS5RlnXreal1dMDBNuRXR0xLw3XyBapXPM+SAMncZGFqmkSUiNwt3L4msUzlmHkALXKev\nFKysa8GDa85bo3gbfI9ogHpgs36vl1OWqBcz9barrdOD1NzDNHr65EJV1uWT6HV90lDlj4ft7OyM\nwNSDZA5ItWWqy/WN0uv1xkZIVY2ZFrjOvrxrhUEqc28ElDecFBjeBzKX60n/vwVR+b3MIwMmBVbd\nmq8bdnm/c+7vJjSTMG1iB624SK4lalmm3sgmK1aqv8bIaQum3g0Q5Vsw1S22kVVqWaZ8rHLAWADa\nvJo6nrnufupFJ9H7TOU60kCNrFLAB2rUiu8N0QZGLVNtpYplakFV12dSzRxMLUsptY73O6s1sE4a\n2P8GjrYyrblOW19g9NxxD5rRxPvuzVPHDqh3ExeQzrYiK7XKiChg9H2mHHeXWLwVL025+jL3QGqF\n07RXCGAwl//k/Ra4ClS5+9YFZ5la6VQ5YL+YpEosRrcYnjt3LhkPtSaxRnVrvr7gtFUZlYkYplZY\nwjtuqQuIL/wCzKMtfS5TYI1a8y3XntO5MVNRbqw06ooIDGFq7bsAlGHKQG1SMwfTlJVlAcMq066B\n5SrklgFDmLIbb6WtRijOYzeEL0Ce6xjU3t7wDeLA8AuWqfiTB9eiC0u5lqmOobKbr99GxqDT1zFD\nlevAikJtnlVqtU8AQLfbHbm+I0/uIGOnMwdTwIZEtGyVWd0o6k7Avpsf9R/V3aK8sl6vN+YO6Y70\nDFCZAxiUS9o6DpMqskTrlhVNTxFYPJBqCxUYfWtU5OJHlqkXP43ipB5Uu93uAPRe3202TvR0wbj5\nQLpzfcoa6/f74ducopeV6DJgCFM5kV46ZwRUFLPS8Rx264HxV4lVOWayTq4KIOdH7NVYoSMPpGyZ\nCkT1S8jF27I8rQig1j0bxU2jmGkqPsz/zXOpTxOaOZimIKkPvDd5rkHUrWl3d3cwfl6fsHPnzo2M\ns0+Nw4/G5uuLV1sIOe6HuCzeMfOOa6Qq8CygnX15gInc/agBSqxThqhlmVZt1ed7ObJKo5ip3leG\nqNTTY0ZTmjmYiiJ4el2YeFnAmHqbk4aftQ6wD9PU73PKOFjPQXwGKae1BKISUOdjFR2/qtIXf4Hn\n0ZTVAFWlJV+7+bqsLkhFkZsv93H0SksAA6jKPlr7Frn5c2uZAnG3CS9Pz70XkPR6vRG3XC9b1iSw\n7+ZHJzVl3XK5dpk4PhpBVOYcSOfj1CQ8i+ZHVWKmKTff8qaimKmugyjyOK2GYMsqFdBzD5nFxcWx\nUBo3mB2ERSqaOZh6gLCg6aXZzfcagnIaj2QZGIWpFb/JzeOnI7v0Os3ygus65mMBVh9XnZb/rLJc\ndHQUQdQCKntJ8kJoYPhyaA1VD645lqkH0txGKI6ZCkRln8S15/huZJk2AdeZgymQfmpZgWqdZ1ml\nVnelVFcngenZs2fNYHjVPM+dsSBqyWqAspb1sbTSdVXgejRkWYYWVD2rVFumuUD13H2uk+V9MkCj\n/qUyPJq7RvE+yfqSToG0Kc0kTFk5YLUmbZWmOtZH+cC+ZcpAFCvTC5R7ZdqSjqxJwO7mIWnL2izu\nfhGrauOTBVQAY3keRHO7Q8k8cvO9/qb8rgkAplsvIOX/Y++PW/Ob0szBNBecnpXHsNPWqTVm3hpD\nr/OAIUz1ifbmXpmO/UbW4sLC6FhjabCKLFP9H1UsUevCz4FrAfDRkHbxU92idAPUQbr5lnfpgXRp\naQndbnfQbVBgKi69LPN/HIZ1OnMwFeknSA5UOZ0CqTdtb28P1pM0sN+anwJ4Ttne3t6gZZRfHqFl\nxUnlP7gnAB+rHLff2k6VsgLOo6c63aK4kz4wbpl6jVFVGqBk7t3jXryUtw1gAFYGvBVaiwyZJjST\nMPWs0xRQrdY/y9VncPJc0jof2Iept/1U/JYn3j9P+oK0/ofdfMvdt5abdmuKZl9V3HzPSgX2LVMB\nWdSSX6VrVOSBaiOJIcr1ADAGWLGiLSNHA5XrMamyYPrYY4/hQx/6EE6dOoUf/ehH+Nu//VssLS2h\n3W7jgx/8IJ7+9KdPXBGR1XiiD4J+cllpz823QJqagH0336tLlTnvlyXvwo/cEy+unANb3m607OUV\nza68BiirJZ9Bym6x7rSvwVUXqPr69IwkyzJl4AOjMGXYeg2/U7NMH3zwQTz88MPY2NgAANx99924\n88478dznPhf/8i//ggceeAC33nprYxUC6lmm+sBFLfnaCt3a2grTwBCmnqtg1dXKly5NFugiiFpP\nWOvBo9NcrtNNXEgFsLMtr1udjpvqvs+StvqZastVw1n/f6TIOtX3tm5okv/WME1Zpp7RMamSML3s\nsstw//334+abbwYA3HvvvXjGM54x2ImVlZVGKiJKQSrn6cWWqRczZWDKtL29jc3NzbF8YOjmp2KT\nqXU4T8uCKV+k1gWh/68JN79YpfOnHFefrVIdM9VWaVXLVOoA5DUyR634vD0Agw78GqRRl8SmrVIg\nA6ZXX301zpw5M1gWkH7nO9/BJz/5STz00EONVkiUe7Cjg18HqDxtbm6OwJTrZs1zyrygvL4A+YKx\nIKrDBtYxq2OBFmjOn3JjpQJUud6kgRTIa4CyQMp10KoC1cXF0U752jLt9XoD4EvaaxiW7UgdmlKt\nBqjPfe5zOHnyJB544AFccskljVVGpOHAIImgqjvKpxqgLKBubm4OICppYN/Nz6lzSh6sUnGs6Mka\nuStVgOrVrQD2aMprfLLg6o0UYss0Z0iptx1PUShP6iMwZahaMGWI6kZprwGqSQt1YS/jn86cOYMT\nJ07g9OnTePjhh/GpT30KH/nIR3DRRRc1UomioqKio65Klmm/38fdd9+NZz7zmXjrW9+KhYUFvPSl\nL8Xb3va2xiq0vb095qZ749x1Gacl/ulZnNZ8a2sL58+fH3PzO51OY9bZwsLC4Hvk7XbbTFeZPvax\nj+Gmm27CysoKVldXsbKyMkjr5Wi+srIyaLnVMa7U3MpbXV0ddCtLHY/c45abH+VJ2GQWxXWLbJyc\nsr29veR7KXLfWfGmN70J991330gPF6snzM7ODra2ttxeMpK/sLCAdrs9uIYlredRmcwfeOABvPOd\n7xxc73zdW3lra2tjZZLfbrcnOn9ZMH3Ws56F06dPAwD+8z//c6IN5ihqtLFcXa9/WtSS57XsWXU5\nyP3U6aqNb4D9VQEJcei+g5ZLJlpaWhpr+ZW0lRetC4x/mycCZ5WyKBYnvSV0GcerD/KcTqqchkQv\nT19P3tBm741nHnABjLxNzTJmooaeaB+sRqqoIVbPAZghBit2a12j1nJdzWSnfZYHFw8wVuNUCqzR\nNCv7Zz005GKyRofo2JaAMophsWWac5FH+cAoTKtakF5eyipmkOoGPz7msyrLiNDlqTSAsXvBak9I\ngVSsUwCmBWsBlQ2a1L1kNcZGQNQg1TDNid1aBkBTmjmY5gAugiSfzKg1b1rw1Puas8/RvgIYu1ks\nkMqbdayLTaSBW3UChh/7A0bfgC6aZLlK+EFAagH1KMI0BVHLi7OsUg+kEVQBJCFqgdTrysdKeTte\nH1ZugPLgG4GVt9+UZg6mLA8sHmgsi9RrzfNO9GFapHpu7adVZ8/N12/UiZ7WwPiFZMHUuxhT+VIv\nUQ4ko/WsuZcXgZSP+SxKX4s8t/K8uWWZagB6ULXe55sCaWqkkVbqevCmyDLNedjrbTSpmYWp9aSN\nwOJZpzndIlJP/4PaL52XAqmkxcX3LNNUjNQDjMBUj46pkpZlYOjm54AzskAlraHpQVSD1IPqLCqC\nqfXAj2CaGtCSAqlnmVpQje45XV/2Xlg5ANVzYLxbYfSwPwiIimYSppO4v5bLkeq8Oy13P7V/Fkh5\nP9kCtEaI5F5Asi2vA3adOTAO0xxgRnkeRCPQHlU3PxegXp4HU6/xKdUAxTFT7/faOk3dVylLMrJI\n2TJNPdwPC6gzB9MUSCNrLWWZpgCqT/hBWqf6v6MHRWSNA0MrREBqjWGOLiDZJlumqVhVKg2MfzXS\nmkdl1jw1sXXq7fNRg2mU9sosAyM1StB7nwUw6uYzUKOWfL5+c1UFrl4DVC5AmwbrzMFUS0M0Za1F\nINVPTC+uc1jWaY5lGoGU3XwZIaKBqkEkaevYWi+tsC7UnHypF29vknmVKee4z6o8SOaAVMM01ZLP\nXeiihijAb4DSbn5kmXpKeRwaoJ6bH1mm3raa1EzCdBLIeK2LEVRzT/pB7q+336m4MDCEKY9dti6i\n1DaiYYJRmTUBvptvgT21TnSD5Vgx1vGeRUXXZJV8zyq1hlp7Fik3QHldo/T/pjxAlgc6TmvPRy8D\ntmXqXRcHqZmEKeA3QFWZUvGb1MV5mPvp3RCLi4vm/uuYKV9MAtXUhaStdG/cNY+PjuDKeYDv5ueA\n0yq3bhJdbsFTjqM+5rOoOtepd+2k3PvcFn0gtkx7vV7yntPKgag+3144yYNnzoO2Sc0cTHMvlrqA\nzYHrtPaT0wIArvPCwsJYGhiFaXSxWMeObzzv2z5WWuqn8/RLeyNQVoGp7Nve3t7Ysr7h5Ph4AJ0H\nmKYmhpplmeaCNBoBleoaVdfVT3kk2gPS1mjqgau325RmDqaiOgBNnVgr1mo97bkOh7Gfen8ZpGKZ\nMkDFKtVuvnVBeg8m69hZMNUvApZt8bpST/12dg+mEUS9sgii1vpcL2DUOj0qMK3iUen1LYimRj9F\nDVCdTsfs5B913Nf3mKUceGqAWjHTlHuvLeGD0MzCVFTHIk31e9MXnQbpYVmoue6agFRgynMAA7ha\nVp23rQim1pvUdZn8ji1SAZxsixugcsAZ5QlMLYjKnAGq5/oYzKpSIM3Nr9LHNKdrVO64/Fw3H4j7\nGUdQZbgC45ZpCqrRg3sSzSRMLddAQ0C7qTkg9YCa64o0uX96X6395n3WLizHTGV/2a0V8OQ8hORm\n8F4ALBYnryO/FYjKf8syEFumVfL4BpHjxPEynlsgPQoQFeVYnbnL+n7IgWoKpt5vc/qYpqCaAqi2\nUvWDVJelLFTedhOaOZhq0Mjcu7giC9V6UkYnO6rDQe6ntY9sjWpXXyZ28wF7WGV0/CzLVL8E2IKq\nNzFYgVGY5gLTK+OGuAiqAMZAqmE6y1DVYIygGaVTILX6maaGk/LvdPw1J2ZqKddbsVx87eanXHxv\nW01p5mAK+NZayiK1Lp7o5EZQPSwLNWfSENUXg4ZpdNzk+PAbyXu93gCiMpJKQ9Va9o5rCqZV4MoP\nBoYjg1TvO4CRBxAvy7GZVeVYplG4SubWvRBZo5ZVqt8a5TVkpboiWsYJny+dTk1WzDTHKtXbaloz\nCVNRHYs01fgUAWAaQPX2MxekwOgLRfR/WhaLgFAalASsDE3JZ3ByOsfK17HcXHB6MNUAZcl6Ak+R\nBrAcm1lVynhIlfHci5tWaclny9SCp3SLsu4777oQWdcEpyPrNOoalWOpHgRUZxqmQBxP0U+kaIpc\nV7bSOL28vDy4OfnjYqzcPMlPnUQLpBZUWRqmHpS1S85wXFxcHOy7BqqXx8dO5wHAzs5OJWB65zn3\nHFtuoHXz5XwBYFra2dmpDU9tme7s7Ay+ecbfQIvevG81NAEYgSYbLtZDlaXPoTwQq9ynrVbL9Izk\nOsv9PlVOS/8kmjmYWjcQf+zLso4Af7x7lNYHkm9oPvDA/mc42LLRVk5uGYCx/9b18qxTmev9kYs6\nsm6jiY8xQ1Hcfuszv15sVfIBYHNzszI0rfUsQHrATOUDwNmzZ/0LcIr6vd/7PZw9ezZpjeZYq/1+\nf+RTIfwBSf6gJIPWG+EEYOya0damdV61FyH5AkKe5DMlqU/3tNvtkU+XABhbR/+3BdmDsFBnDqaA\nbbZzH0ZxVfXFZMHUcvH0BSBpz+UAgJWVFfN/qkxiZXoQl//l/2eQ6nVFAlNvv9nV1SER3Vovx1gs\nVf0gY8hG1j4AbG1tJeGZW+bBsmoZMPrZ7lnTuXPnQojmAFZbpgJQnWaYiktvtdADGFwbvB2tCKRc\nblmYDFVr2YLq8vIyAAzSHkQtkB4EUGcOpinXjluU9YnVoErFbHLqoC3TOhc5L+e4GJZl6ZUDozDd\n29tzuyuJFcoWKUN1cXFxBJxR39MoL7JMveObA1MLjrl5kgaOBkxT15e1rK1TC5qW689A1a6+tky9\nbbM8qFqWqQXUKh+WBGBapd5DXlumXLdJNXMwBcatEYaoZWnKb/i3UmadaL0dTltWMbAPU6vhJTfN\n9YlOnGfVaqDyvouVyFYozxmeDFA5rnqeystJA+Mw1cfZ8wRSMJ1kDsyumw9g4OZHwMwp293dHYuT\nevFT7eZbMVMOJXkg5fMm9w0bEHI9elZprjUqrr7l5mvr1ouj6vu8Cc0cTPXNpoFqWaPW76XMW0/W\n9bZpwZQD7qkeA/J7hiC76pZlyqDVEBUI8rpSN6kXW6Q62M+xUQ1WsUg9uEZzrwwYuvlVoBmtZ90E\nVfOAo2OZRlMEUiln911DMyrTb4QC4G5Pi69nDi8JTPf29lyIapCmrFQNU1lXx/T19XoQIAVmHKaW\nm89Q9Z6M/CRMbcuba5iurKyYrZlenqT1CYta5UX6ASA3B4NWLlRg1GrQx0gfOwYpQ1SDMUrnrAfY\nMVMrnSqPHnQ5ZfrGmQeY5sBW9xfVaWvZGncPDHuMaAMl5flZniTDLhUvTTVMARhz8VONT/q6mVvL\nFBgdZy1pBqi1vgVAyyr1blJt0XAaGFqmVmdoiS8KmPiJLmLY6ROorVPJkzQfB0mzFeu15uv9iSY+\nztF6ueXAPkytY141LzpXVfOAowdToPqr+RimXvennO5RbJnKf4v0/cXXpb5Ged0oVppq2WdXn2Ea\nNT5FDVH62ptEMwdTyzIVkIr7mFpfLEJvPb1sAZS3DYzC1BoBoretgckWQ85T0XpoyHa4/pIv/20B\n0tqv1HIKuqkJ2I+ZSl313MqL1s2ZLJhax/ooxEwBvwE1J7/f75sd8nPm+g1TgD0wxJJnJLBy4qUa\npAJPDVRgvzVfA9rqvneQIAVmEKYivkE0SC3Xng+UXACRm5+6KbVlurKyMgJRtkB7vd7YCdJ15Qvd\ng6hVX15/YWHBhLVYy3KsJK2tbWvfrDIPmlWADAwboOR4153XAetRhalYzSmA5qzjDR3lEVDe26A8\ny9QyUqx0VOZ1wPfcew+uHDPV8NRAPWgXH5hBmGqoeQFvD4B8wHhdnc61ctgylVZzce3lgvNODsc2\n2UW3QMHim0PKOa3nAli2fDUoc/OqQDcqA0Zjpla9vblXFgEyF6TA7Lv5gB+bzIEpMPqRRf1iEzYK\novIIpnJM+brU5bwe50WWadUuUsAQpjmTB9UmNLMwZQhFMU85MHokj+UORze/du11y7TAVCYL3h4Y\nZT+8fdHry0XKeXp9Tkt9GNb6YqmyHEE3F8rAqGWqz11OnnW+PXhWyZtly1Tq5sEzN29vb88NSVXN\nA+x35gL2NZx6qNUBp9UtSlumUf9n736da5gCNlC5jNfRXXukk7r31LRgrMFsxf9WV1fR7XYHJ0b/\nlusGjL9/NQKulgap/MYLW7Cbz8cldVFHMcYUfFN5wHgD1CRp69xVTcvyUbFMJ51bLyapkweM90Lh\nexJIfyGUl1P9TOu05nvdoHK7RjUB1JmDqXVzWmW6X6eAlEcaWf+nT67numqYrqysmPFBDUiGqFzQ\n3LfT6nzPv62SLxKrIQeg0TGpO3lPeoGpbFOf5ypl1rxq2VGA6fnz57OBmVont/teKg8Y/TQOe4LW\nuYvusYWFoWVapyHK6mfabrdN79Izjg4CpMAMwhQYtRo5z3o6eiOOUhaKdcK9pxmwb5lWcetlYpBa\n4PWky6wbSGS5YCmA5qyTgm+qTLv5sn7d5QieVeazDFPpGgWkX9QTlWvPaJIJGHbal/tBvMaU4RKF\n0aqOgLJedgLsW6aRMXQYLj6QCdPHHnsMH/rQh3Dq1KlB3iOPPIJPfOITOH36dGOVAcYvfNl53b+S\nT6YenWG1Zlvg1O6AFWvhmCn/xjoZ1kXMMVwrbiq/08tVrBPuCpZrnVXJq5u2YMrbSeXp/AisVcpn\nPWYaPTirlPG94RkdOesA493yrEZhkQdQSeeMfIriptYIKM/DtO57nT40y/TBBx/Eww8/jI2NjUHe\nD3/4Q3z6059upAJaDFE+YXwCdZk1peKBXqzUslKBfZh6TzO9bXmVnW4U49/qhqaoISEqA+zPlvA8\nKjsIy0/mkZvPqlJWx5LVy0fFMmXl5FnLOcNOU/nAaGs+GzZ6u9Z9Zg03jlx97eIzXHVDFDAKU204\nRWnLkJhESZhedtlluP/++3HzzTcDAJ566incd999uP322/Hud7974gp40jdFDlQ4bVmPlpkfuSFs\nma6srAzqwhcSMDpumWNN8nu2TLUl6yn1sLBgah236JjmlNVZz4KpVpWLN8dqTeVz3qzDFKgWAvLK\nqzygqzywPcs0Mlx0v9IcVz81CordfM9wkjpI/TwjqwklYXr11VfjzJkzAPahcccdd+DWW29Fu91O\nntQ6siAqstJeuXXQGGi5/dK0ZWoF3fUTnQP4Vt82S3WtByBvdErOBZNap2q5jICaRJNc6N5vZ93N\nt1T3XqsSFoi25zVuallGinVP1elT6nXal7llbVrLVlkTWtjLOEtnzpzBiRMncPvtt+O2227DJZdc\ngp2dHfzkJz/Ba17zGrzrXe9qrEJFRUVFR1HZrfl7e3t4/vOfj0ceeQTAELBNgzQ3XpQq293dHXup\ng/VZW+szt3p+7bXX4p//+Z/dl+zqF+16yzs7O+h2u2PBfssCjco577e//S2OHz/uHo+qy3WPty7v\ndDoDi6Hqf6TWqxtTlLwmrZEc5W5PQkOp39bJmyR97tw5XHzxxeYwzWgIp1fWbrexvr6O1dVVrK2t\nYW1tbSStl6P1LrroIvOFOpPM6yobpod9AeptezdWqsxy8cX10JACxoPpouXl5UGZdhUsV0a7L51O\nB8vLyy5MvXlqHQBYX18f1NOKl1nzVFnOcmodPmap31X9f70fck6qbO+glAs7K193B9TrpNI6LwJH\nbh6wf42loJkDVoGpgHF1dRUrKyuDieOh3lugvP6t0fwwlAXTZz3rWWNdoKy8aSkXqDqWwzEdK6DO\nsqwsjqFa/2nFharCNKfbyvr6elajQyrN80nSMm+1WmPrRL/PXTeaW9eCl38Q8mCXWo7iero8d+7F\nCKN8L29tba1Ry5RBKnPuP8pA1UNCddtFDkAPA64z2WnfUupm0OXWxVMFovpi0lYW/+fCwsJYH9VW\nq4VOpzMWXJdP5lbt/+flAUOY6n3ReTn5OfPcdSyYTgpoaz+mDVBRjqWYSuu+j3XTk066HmtraxMB\nlOcCU7ZIGabWB/IsoOYe7yqewiQ6MjAFqgNV8uTikH6fDFKZW//Fc+mG4QHauoDEEhX3nmFqQbJq\nWuq9trZmArLOBKQBlruOwDT6XZWy6KEg5967Pg4aqrk3c8qK8mCaY2laZbn9LaP1gOowjdZtt9su\nSK3X72mQWsfIO87WOTooHSmY5kgfUA3Svb1hp38J9muLU/8W2HfzPYBacVJpwJLP53a7XbTbJFwP\nsgAACytJREFUbROmEThTywCwsbExAliZUstenhyTSdLAqDXfxDznQcCyHqyHpZQLasEAsLv0TWpd\npkYCpcqAfe+nSctUJgGpnqeAqs+lB9LUOW/ymjhyME1Zp7yezC1LUls/2m3QTz/uGKwbsmTq9Xoj\nbn2320Wr1Rp7Ea8FyByIWnnA0DKt0xvAsnRTVmBqWVumnmWba+XmTnI8PB2UdVoFmKk8D6apEX0R\nSPVk5Ud5gG2Z1rFUl5aWsLy8jJWVFfOVero/qQVTyxLX5yI33aSOHEyBekAVgIplKhILVYNUhn9W\nsUwFmgJQAap+e7kFyQigUT6wbzVUibN6eTmWX24ZMLRMm7B4NTQtiMo51lA9KIha8qCZk5Z5E665\nB1LtLudMQLXW/BRkpRFKv7REj26yQOpZp7mW6kHqSMIUyAOq9bS2frOwMPo5EBntwReTvObLcu0Z\noPyWcu/N5R5Iq0DUaoDK/X2UNwlQLct0EsvWA6mcI1nWEOWbZ2/vcBqi9M1reUY5yykX3JpH7jrD\np2paloE8mOYCtspoJ90IFcVMdTpnuUkdWZgCwwPjAVLmGqjWugxUuUH5TTni5uvGJvmUSavVGvsc\nhH57eR2YpiZgaJlOOnmWX457XQWmKWB6+bK/vI7M5ZzxnEF62ECVZQ+cnoVqud0WJHPSDEfd28Qr\ns9YBhl2jGIh146cC05x5Tkt+dPyrLE+qIw1TkXejpCCq1xGQyiQ3JTB8M43+vDNbpNZklVUBXGp9\nYAhTva73Wytffy7aAmRunufmT2L1ikUq50jWBTByrjSgDguk3vXk5Xtg9aBoue2pPPagLK8qNw9I\n9zOtaqV6k7ZGc4CaA8nDcPXnAqaAb6VGlqlVZrm+wL6b733WodVqjeVFy3Wg6ZUD4zDNfZO6Ltdw\n1KD0yqz1gKFl2sTEENVQBUZfxKHP7WGB1FIORC2YRnHP3ElDUcMudwKGbv6kLn4E11SZDkHkWpqH\nBde5ganIip8wROXGkptRg5RdRJkDw+/MRKDKgVlVazGVB+xf6Braep6Tp/fbAmZUpo9ZVZh6Vq7A\n1IKoJT6vnqV4UKoCT7729EMgF5Y5ILWAlyrjNDC5ZRptNxf22ir1LFM+F1XyJ9XcwZTFbh6AEWCy\ntSPr6XJtmeZYkHXXiZajMmD/QteWcMo69izmCJxV5sD+AygCpC7jhiQrBGBJg/awoMmKtpsCaGSZ\neuDUEPWWq1qA3rrA5J32eW7FZXPTshwd79S5OijNNUwBe4SJtnisG9qK//X7/YFLz5MAwFu28lJu\nd24aGLVM605RfTUoc/KAoWWq4SjL+qHF50Bbu9a5BOwXnkwTrIAN1xzrFBiHqQVRPffKvBhl1Tyg\nWj/TFHQ9qzo3T7v4Oee4xEwnkBVP0YDMXQaGlqnn9ua6whEc6+QBNky9RrEoPwJp1TJg3DKN0toq\nZaBGbh2fK/Y4pgXVHICyZaWtrMgq1ZZaVGbBMTXxi0W4IUiusUlcew1TKyYcLVtl0TmdxkN0bmEq\n8qAKpMeE83JOy3Tk0vI6HhzrxGCBIUytblg6T/c+kPTS0tKYdRpZ2ikLHMDAirceKAxLL81hGjmP\n1jljkGqgHqZ03NNLe1CVdXTc03JzvWVOW99SspZTZcBknfb13HuY6NBHtKzjzLOgmYMpw66J/7KW\n9f+nlgEMPtNSx7LVy7kNRDlzYP9C1wMEUnOGqLZMU5MHV56AGKYRXBmiMtfSELUsUjnnB3mz6Rva\ncj9zXPyUZRrB1JsEhrl9OvU3lximua/gy4mnahBG5y3Ka/K8NvFfMwdToPknTRP/p1/B51mzqbwq\nMM3JA/YvdIakNfKKrVK5sCUt85yGs9wJGI+ZShk39FnzCKba0rfyvJvtMC2YFEA1SL2YaQTVFEgZ\njNbH6fSytw5QvTU/gq7sK58TLwaaKreWpyW7J3tRUVHRDKkpb/UgVWBaVFQ085oV6zNSgWlRUVFR\nAyowLSoqmnkVN7+oqKioARU3v6ioqKgBFcu0qKio6AJRgWlRUdHMq7j5RUVFRQ3oKLj5C3tHoZZF\nRUVFM65imRYVFRU1oALToqKiogZUYFpUVFTUgApMi4qKihpQgWlRUVFRAyowLSoqKmpAMwHTvb09\n3HXXXbj22mtx44034oknnph2lQAAvV4PN998M6677jr81V/9Fb74xS9Ou0oj+tWvfoWrrroKP/3p\nT6ddlRE98MADuPbaa/Ga17wGn/70p6ddHQD75/LEiRO49tprcf3118/EMXvsscdwww03AAAef/xx\nvP71r8f111+P9773vVOu2WjdfvSjH+G6667DjTfeiL/+67/Gr3/965mol+iRRx7BtddeO6UaDTUT\nMH300UfR6XRw+vRpnDhxAvfcc8+0qwQA+OxnP4tLLrkEn/jEJ/BP//RPeP/73z/tKg3U6/Vw1113\nYXV1ddpVGdE3vvENfPe738Xp06dx6tQpPPnkk9OuEgDgy1/+Mvr9Pk6fPo23vOUtuPfee6danwcf\nfBB33HEHut0uAOCee+7BO97xDjz00EPo9/t49NFHZ6Zud999N+688058/OMfx9VXX40HHnhgJuoF\nAD/84Q9n5oE9EzD99re/jSuvvBIA8IIXvADf//73p1yjfb3yla/ETTfdBADo9/uDTzjMgj7wgQ/g\nda97HX7nd35n2lUZ0Ve/+lVcccUVeMtb3oI3v/nN+LM/+7NpVwkA8OxnPxu7u7vY29vD2bNnB982\nmpYuu+wy3H///YPlH/zgB3jxi18MAHjFK16Br3/969Oq2ljd7r33Xjz3uc8FsP8QX1lZmYl6PfXU\nU7jvvvtw++23T6U+WjNBh3PnzuH48eODZfkQG38rZhpaW1sDsF+/m266CW9/+9unWh/RZz7zGVx6\n6aV4+ctfjo9+9KPTrs6InnrqKfz85z/HyZMn8cQTT+DNb34zPv/5z0+7WtjY2MDPfvYzXHPNNfjN\nb36DkydPTrU+V199Nc6cOTNY5oGIGxsbOHv27DSqBWC8bs94xjMAAN/5znfwyU9+Eg899NDU69Xv\n93HHHXfg1ltvHXzsctqaCcv02LFjOH/+/GB5FkAqevLJJ/GGN7wBr371q/GXf/mX064OgH2Yfu1r\nX8MNN9yAH//4x7jlllvwq1/9atrVAgBcfPHFuPLKK9FqtXD55ZdjZWVlqjE20cc+9jFceeWV+Pd/\n/3d89rOfxS233IJOpzPtag3E1/v58+dx0UUXTbE24/rc5z6H9773vXjggQdwySWXTLs6+MEPfoDH\nH38c73nPe3DixAn85Cc/mXp4cCYs0xe+8IX40pe+hGuuuQbf+973cMUVV0y7SgCAX/7yl3jTm96E\nO++8Ey972cumXZ2B2DK44YYb8L73vQ+XXnrpFGs01Ite9CKcOnUKb3zjG/GLX/wC29vbM3HzPe1p\nTxuEaY4fPz74tPWs6HnPex6++c1v4iUveQm+8pWvzNT19vDDD+NTn/oUTp06NROQ39vbw/Of/3w8\n8sgjAIAzZ87gxIkTeNe73jXVes0ETK+++mp87WtfG7TITfsJIzp58iR++9vf4sMf/jDuv/9+LCws\n4MEHH0S73Z521QaatVeTXXXVVfjWt76F1772tYNeGrNQxze84Q247bbbcN111w1a9mep8e6WW27B\nu9/9bnS7XTznOc/BNddcM+0qAdj3Eu+++24885nPxFvf+lYsLCzgpS99Kd72trdNrU6zcD1ZKm+N\nKioqKmpAsxGYLCoqKjriKjAtKioqakAFpkVFRUUNqMC0qKioqAEVmBYVFRU1oALToqKiogZUYFpU\nVFTUgApMi4qKihrQ/wet12wS5VsN1wAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.imshow(np.reshape(train_dat.iloc[0].values, (16, 16)))" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": false }, "outputs": [], "source": [ "test_dat = pd.read_csv('./res/zip.test', header=None, sep=' ')\n", "test_dat.rename(columns={0: 'digital'}, inplace=True)\n", "test_dat = test_dat.query('digital == 2 or digital == 3')\n", "test_dat.dropna(axis=1, inplace=True)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": true }, "outputs": [], "source": [ "test_dat.set_index('digital', inplace=True)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
12345678910...247248249250251252253254255256
digital
3-1.000-1.000-1.000-0.5930.7001.0001.0001.0001.0000.853...1.0000.7170.3330.162-0.393-1.000-1-1-1-1
2-0.9960.5720.3960.063-0.506-0.847-1.000-1.000-1.000-1.000...-0.668-1.000-1.000-1.000-1.000-1.000-1-1-1-1
2-1.000-1.0000.4690.4131.0001.0000.462-0.116-0.937-1.000...1.0001.0001.0000.270-0.280-0.855-1-1-1-1
\n", "

3 rows × 256 columns

\n", "
" ], "text/plain": [ " 1 2 3 4 5 6 7 8 9 10 \\\n", "digital \n", "3 -1.000 -1.000 -1.000 -0.593 0.700 1.000 1.000 1.000 1.000 0.853 \n", "2 -0.996 0.572 0.396 0.063 -0.506 -0.847 -1.000 -1.000 -1.000 -1.000 \n", "2 -1.000 -1.000 0.469 0.413 1.000 1.000 0.462 -0.116 -0.937 -1.000 \n", "\n", " ... 247 248 249 250 251 252 253 254 255 256 \n", "digital ... \n", "3 ... 1.000 0.717 0.333 0.162 -0.393 -1.000 -1 -1 -1 -1 \n", "2 ... -0.668 -1.000 -1.000 -1.000 -1.000 -1.000 -1 -1 -1 -1 \n", "2 ... 1.000 1.000 1.000 0.270 -0.280 -0.855 -1 -1 -1 -1 \n", "\n", "[3 rows x 256 columns]" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_dat.head(3)" ] }, { "cell_type": "code", "execution_count": 68, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def eva(conf, train_dat, test_dat):\n", " x_train = train_dat.values\n", " y_train = train_dat.index.values\n", " \n", " conf['cls'].fit(x_train, y_train)\n", " \n", " x_test = test_dat.values\n", " y_test = test_dat.index.values\n", " \n", " y_pred = conf['cls'].predict(x_test)\n", " \n", " accuracy = sklearn.metrics.accuracy_score(y_test, y_pred)\n", " print('{}, parameter: {}, accuracy: {:.4}'.format(conf['name'], conf['parameter'], accuracy))" ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LogisticRegression, parameter: None, accuracy: 0.9643\n", "KNeighborsClassifier, parameter: N=1, accuracy: 0.9753\n", "KNeighborsClassifier, parameter: N=3, accuracy: 0.9698\n", "KNeighborsClassifier, parameter: N=5, accuracy: 0.9698\n", "KNeighborsClassifier, parameter: N=7, accuracy: 0.967\n", "KNeighborsClassifier, parameter: N=15, accuracy: 0.9615\n" ] } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "from sklearn.neighbors import KNeighborsClassifier\n", "\n", "configuration = [\n", " {'cls': LogisticRegression(), 'name': 'LogisticRegression', 'parameter': None},\n", " {'cls': KNeighborsClassifier(n_neighbors=1), 'name': 'KNeighborsClassifier', 'parameter': 'N=1'},\n", " {'cls': KNeighborsClassifier(n_neighbors=3), 'name': 'KNeighborsClassifier', 'parameter': 'N=3'},\n", " {'cls': KNeighborsClassifier(n_neighbors=5), 'name': 'KNeighborsClassifier', 'parameter': 'N=5'},\n", " {'cls': KNeighborsClassifier(n_neighbors=7), 'name': 'KNeighborsClassifier', 'parameter': 'N=7'},\n", " {'cls': KNeighborsClassifier(n_neighbors=15), 'name': 'KNeighborsClassifier', 'parameter': 'N=15'},\n", "]\n", "\n", "for conf in configuration:\n", " eva(conf, train_dat, test_dat)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.9\n", "Consider a linear regression model with $p$ parameters, fit by least squares to a set of training data $(x_1,y_1), \\dotsc,(x_N,y_N)$ drawn at random from a population. Let $\\hat{\\beta}$ be the least squares estimate. Suppose we have some test data $(\\hat{x}_1, \\hat{y}_1), \\dotsc,(\\hat{x}_N, \\hat{y}_N)$ drawn at random from the same propulation as the training data.\n", "\n", "If $R_{tr}(\\beta) = \\frac{1}{N} \\sum_1^N (y_i - \\beta^T x_i)^2$, and $R_{te}(\\beta) = \\frac{1}{M} \\sum_1^M (\\hat{y}_i - \\beta^T \\hat{x}_i)^2$, prove that\n", "$$E[R_{tr}(\\hat{\\beta})] \\leq E[R_{te}(\\hat{\\beta})],$$\n", "where the expectation are over all that is random in each expression.\n", "\n", "#### solution\n", "Ref: Hint from [Homework 2 - Hector Corrada Bravo](http://www.cbcb.umd.edu/~hcorrada/PracticalML/assignments/hw2.pdf) \n", "\n", "define: $\\beta^{\\ast}$ is the least squares estimate in **test data**.\n", "\n", "1. \n", "As both training data and test data are picked randomly, so obviously:\n", "$$E[R_{tr}(\\hat{\\beta})] = E[R_{te}(\\beta^{\\ast})]$$\n", "\n", "2. \n", "On the other hand, because $\\beta^{\\ast}$ is the least squares estimate in **test data**, namely, \n", "$$\\beta^\\ast = \\operatorname{argmin}_\\beta \\frac{1}{M} \\sum_1^M (\\hat{y}_i - \\beta^T \\hat{x}_i)^2 $$\n", "so obviously:\n", "$$R_{te}(\\beta^\\ast) = \\frac{1}{M} \\sum_1^M (\\hat{y}_i - \\left (\\color{blue}{\\beta^{\\ast}} \\right )^T \\hat{x}_i)^2 \\leq \\frac{1}{M} \\sum_1^M (\\hat{y}_i - \\color{blue}{\\hat{\\beta}}^T \\hat{x}_i)^2 = R_{te}(\\hat{\\beta}) $$\n", "Thus:\n", "$$E[R_{te}(\\beta^\\ast)] \\leq E[R_{te}(\\hat{\\beta})]$$\n", "\n", "3. \n", "Final, we get:\n", "$$E[R_{tr}(\\hat{\\beta})] = E[R_{te}(\\beta^{\\ast})] \\leq E[R_{te}(\\hat{\\beta})]$$\n", "namely,\n", "$$E[R_{tr}(\\hat{\\beta})] \\leq E[R_{te}(\\hat{\\beta})]$$" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.1" } }, "nbformat": 4, "nbformat_minor": 0 }