{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Trunk: The Curse of Dimensionality\n", "\n", "The following theory is from a section of \"Statistical Pattern Recognition: a review\" (Jain et al., 2000) covering results from \"A Problem of Dimensionality: A Simple Example\" by Trunk (1979). The performance of a classifier is related to the number of samples seen, their dimensionality, and the complexity of the classifier. The complexity of the classifier is a tradeoff between bias and variance. What may be unintuitive is that additional information from more dimensions, even if each dimension is informative and non-redundant, may lead to a decrease in classification accuracy if the number of samples remains small compared to the number of features. Trunk (1979) provides an example of this in a simple setting." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "import seaborn as sns\n", "sns.set_context('talk')\n", "import warnings\n", "warnings.simplefilter(\"ignore\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider a two-class classification problem where each class has the same prior probability and has a $d$-dimensional Gaussian distribution with identity covariance matrix (i.e. all dimensions are independent of one another and of variance 1). The two class means $\\mu_1$ and $\\mu_2$ for class 1 and class 2, respectively, are such that $\\mu_1 = -\\mu_2 = \\mu$ where\n", "\\begin{align}\n", " \\mu &= \\big(\\frac{1}{1}, \\frac{1}{\\sqrt{2}}, \\dots, \\frac{1}{\\sqrt{d}} \\big) \\\\\n", "\\end{align}\n", "Each feature adds new information, however the information decreases monotonically with each added feature and converges to 0.\n", "\n", "We consider this classification setting, in which data are drawn independently and identically distributed with equal prior probably $\\pi = \\frac{1}{2}$ for each class and the only unknown parameter is the mean." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgkAAAD2CAYAAABcORQdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhcZdnH8e+dpGmbLkl3ugBdWMteQECWllUWlVVE5RUURVB8RRBkEUEFQTYFXEGEV0BAgbIjm5TNshVaoEChK13pkiZt06Zpk/v94zmTTqeTZGY6yZkkv8915ZrMOc85c8+ZMzP3PNsxd0dEREQkVVHcAYiIiEhhUpIgIiIiaSlJEBERkbSUJIiIiEhaShJEREQkLSUJIiIikpaShHbIzE43MzezcSnLR5jZw2a2JFp/Z7S8zMxuNrNPzazezGbHEHZBM7Nx0TE7PabHHx49/hVxPL5Ic8zsiuj8HN7cso4i+fOzsyuJO4DOLPqSfyFpUQOwApgPTALuBZ72zCezuBPYFbgKWATMiJb/FPghcD3wLrByM0MXEZFOQElCYbgXeBIwoBewPXAc8E3gOTP7irtXJZW/C7gPqEssMLOuwIHA7939+pT9Hw685+4XtN5TaPdeAroD6+IORKSduBK4BlgbdyCtoDtQH3cQhUBJQmF4293vTl5gZucB1wLnEZKIoxLr3L2eTU/gQYQkozLN/rcAPs1nwFGMvdy9Q9RKuHsDUBt3HNI5mFl3YJ27r487llxFsbfb+Jvj7vosiKhPQoFy93p3Px94BTjSzA5IrEvtkxC1nc2JVl8erfNEOWAEMDZp+RVJ+9rLzMab2VIzW2tm08zsUjPbKIE0swlmNtvMRprZA2ZWSWgaSawfbGZ/ivo91JnZAjO71cwGpuwn0Y65vZn92szmRY87xcyOTncszOxEM3vBzKrMbHUU481mVppUxszsbDObFJVZGW1zcCbHO12fhORlZvYtM5saxTrHzC7MZL/ZPIcmtvu+mT1jZvOj47rQzO5O1w5sZseY2YvRa7kmei0eMrPtkspsaWZ/i57DWjNbbGb/NbPTMngOiddutJn9LoqlxsyeN7PtozInmNnb0ePPNrMzm9jXYdHzqjKzWjN718zOSlPuCDO738xmRvusirYbm6Zs4hwdYmb3mtnyKL6nk49BC89xBzP7Y/Rar4xeq0lm9t0myvc2s6vM7MPoeSwzs1fM7JSkMndGx21AdOw/A2qAYdH64WZ2l5l9Fr0mM6L3RlnKY/U1s99G6xOPNcnMLkgp900zeyM6VjXRsbvHzAZk8PyLzOxiM5sVPcZ7ZvaNJso210+hLc+R2dFrv4OZPRG9btUWPqe2yPEYpu2TYGbfSYq9OorvgDTlPHrd97Pwnqyx8L78q5n1TCmb83uyLagmofDdDhwAHENIGNL5CzAZ+C0wHngoWv4J8D/R8qWEvgoQ+iVg4Ut5PDAduIFQC7Ef8Etgd+ArKY/TE3gReBW4FBgY7WcrYCJQGsU7A9gGOBs42Mz2cvfqlH39H6Fq//pou3OBh81sO3efnShkZlcBlwAfRM9jITAKOBH4ORuaXO4CvgY8ANwBdAW+ATxrZie4+6NNHLtMnEWoqbkdqAJOBX5jZvPc/R8tbZzFc0jnJ8BrwM2E12dn4DvAIWa2i7svix5jLPAo8B5wdRTnEOAwwmvxsYXE71lgKPBH4GOgnNCP5UDCa5KJ/wNWAb8GBgDnA0+b2WWE2q8/AX8DzgD+YmYfuHvjuRt9Kfw5el5XEb4wDwf+ZGajUprFTgf6An8H5kWxfwd43swOdveXU2LrQWg6eo1wzEcAPwIeMbOdo1q45owDDgIeB2ZF+/sKcKuZ9Xf3q5OeRwXhPbkT4bz7E1AM7AF8kdAkmOxZQl+hX0X7XWVmWwNvEF6HPxFek3HAxcD+ZnZoUm3Dv6LY/gJMAcqAHaLy10UxnUp4fV4mnFtrgK0INZEDgSUtPP8bCcfrJcK5OhD4AzCzhe1SteU5AuG8mED4PLsA2A34HtAbOCKpXIvHsClm9hvgQsLrdQmhafhM4AUzO9bdn0zZZHfCeXQH8I/oMc4g9D07M9pnvt6Trcfd9RfTH+GkceAnzZQZE5V5MGnZ6dGycUnLhkfLrkizj9nAhJRl3QgfWC8BJSnrfpxm/xOiZVem2f8jwGJgWMryvQjVkVckLbsi2s/jgCUt3ztafnXSss9Fy/4DdEvZtyW2B46Pyp2ZUqYEeIvwYW+pcTfxWpyeZtkCoCJpeRnhw3ZiBq9xps8h7esH9Eizz0OjshcmLbsxWjawmVh2Td0uy/M18do9lvLa/W+0fCWwVdLyAYQmnHuTlg2Olv0jzf5vIjSjjWrh+Q8iJL1PpixPnKMXpiy/IFr+hQyeY7rHK4r2XQ10SVr+x3TnXWKbpP/vjMrdnabcPdG6o1OWXxctPyO6Xx7d/2ML8T9EqOEraa5cE9tuT/gCex4oTlo+JlruwPA050O6ZW15jsyO9n1yStk/RMt3yOYYRmUduDPNsXkFKE1aPoSQkM9OOWYeld83Zb9PEH4c9czHe7It/tTcUPgSVfq987zfwwkftncAFWbWP/FH6EQJG2fgCRt1ijSzcsKvpkeB2pT9zCbUUqTbz00evUsA3P1NwgfItkllEtWcF3tKG6FHorunRts+nPL4FYQPq+Ep+83WHZ7UcdTdVxN+4WSyz0yfQ1ruXgON1cDl0fOaQvjC2iepaKKm5kRLaSpKU+ZgS2kGytLNKXEnfs0/4u6NfV/cfQkwjY2P00mEWp7bk1+r6Hk9RvhCPjRpHzWJ/82sp5n1I3xJvM7Gzz+hgVDrkuw/0W2Lr1fK43WLHq8v8AzhPbhDtK4IOAX4ELgtzX4a0uw+9b1TBHwZeMc3/RV6dfRcjo/uryF0ENzHmh9yWE1IYo8xM2umXDrHEhLXGz2pxsXd3yb82s1Gm50jkQXu/s+UZYnXfZvoNtNjmE7i2Fzr7o01f+6+gJAEbk2oQUo20d1fSxNTCeEzCfL3nmw1ShIKXyI5WNFsqeztGN3+jfCrOPnvo2jdoJRtlvjGoywgZNhFhGq01P0sidan7gfSV19WAv2S7m9LyLKnZPBcegGfpXn8K5p4LtlIF+uylFibkulzSMvMDjGzCYTq1io2PK9yoE9S0d8D7xB+3Vaa2ZNm9r+W1A7t7nMIVbdHAAujtthrzWzvLMNKPR7Lo9tZacouZ+PjlDjvnmPT1yrxRdT4WpnZKDO7z8yWExLBpVHZo9n4+ScsSE3GCK8VZPB6RYnI9Wb2KeFLJfF4iaa6xGP2j/6f3FKil+TjlPsDCE14U1MLunsloVlqZHS/jtAktzMwy0KfiVvMLPXL8teE/kkPA0vM7MGoHb1XBvGNjG4/SrPugwy2T9Zm50gTjwcpr3sWxzCdEdHtJq8V8H50OzJleSYx5es92WrUJ6Hw7RrdTsvzfhO/Mi4g9GdIZ0HK/dXN7Odumm4/W5NmWVNtw5byfyYfwEb4APl6M2Xeb2ZdSzZnKFSmz2HTDcMHxTOE2piLCB+wa6L93UdSku/uy6LyBxJqiQ4itCn/wsyOdveJUbmfmdnfCH1cDiS0719gZte6+08zDK2p45HpawpheO/CJsrPhPCFTWgO6wH8jtDfYiXhF/bFwCFZxJAaR1P+QagZuzV67EpCk9nRhGa4xDFP7Cvj1zaqgco2nuTt/2xmjxBeu7GEX9znmNn97n5KVOYTMxtN+KV9aFTuNsJ5cJC7z2hi98nxpHtO2dZKtMk5ksF+N9p3Jscwg/gylWlM+XhPtholCYXvjOj2iTzv95Potsbdn9uM/UwnfKiUbuZ+0pkGHElIlN5optwnwHbAa+6+Ks8xbK5Mn0M6Xyd0hDvK3Rt/gZlZD9L8io6qiCdEf5jZroRJuX5G+ABKlJsJ3ALcYmbdgKeBC83sBndfnGWM2Uqcd0szOF8OJbT5ftvd70heYWZX5juwqCPiF4G73P2slHWHpRRfQvgFvPtmPORiQtKzU5pY+hDa5jdK4N19IfBX4K9mVkzUYTd67d6MyqwlNBk+Ge3raMLnx3nAD5qJJ5FA7MimX8I70nayOUeylskxTCNxbHZK+j9hdHSbbefO5JjifE82S80NBcrMis3sesLIhifd/dU8P8TThA+pi8ysb5rH755JFaWH3vVPAieY2b5p9mOWwdCrJiRGDvzawmRRm+w7+vfvhHP56tQyUbnNaWrYXJk+h3QSv0RSy1xCyns3aq9N9RGh5qFvVKbczLokF4iq5j+M7qarvs+3fxLahX9hYa6AjUQxJo5T2udvZkeQvj/C5mrq8QYTft01ivoc3AuMNrMzSJFJf4BoH48Be5jZkSmrLyK8xuOj/ZVZypDIKCl8N7qbeI3TnQdvJ5dpxqOEhP+86Msz8VzGEEbJtJVszpGMZXoMm5A4Nhckv4eic+NbhCaed3KIqRDek81STUJhGGNh6BJsPOPi1oTq5uaq0XPi7jVm9k1C2+W0qLprOqGz3w7ACYROUxMy2N3ZhF6/L5nZ3wlvliJCG92xhC/xK3KI8Q0Lw45+Ckwys/sJIzJGEKoJPwdUufsDZnYHodpwDGHkxFLCOPT9CB2XUtsL20Smz6GJzccTqrifNLNbCUMlDyfUSixNKXubmQ0jnC9zCDPGfZVwPv09KnMwYSjfg4QajlXAnoQvwNfdPd9NWptw93lmdjbhl9yHZnZXFO8AYBfCeT+a0On1FcKxuiHqaDaP8Mv9fwhND7vkObaVZvYMcKqZrQHeJLwHv0do6knt0/AzQpPHX6PE5RVCgrEH4bP1fzJ42EsIr+nDZvZHwnvwIMJr9xIbmvC2A140s/GEprPlhF/3Z0exJToGPmNm1dG2cwnv59MJX3B3tfD8PzKzPwDnAP+JzpOB0f0pbNoxr1VkeY5kI9NjmC6maWZ2HWEI5EvR+zgxBLIn8A1veXhtOrG/J1uiJKEwfC36ayCcJPMI8xHc6+7/bq0Hdfeno3bsiwgjBAYQ3jgzCEPq3m1m8+T9zDWzPQlfhMdG+6olfEg9RvhlkGuMF5nZFMIH1YWE5GMuofZidVK5b5vZC4Q37cWEuRcWEX5FXZzr4+dDps8hzXavmtmJwGWEsfVrCJ25xhK+BJLdRfgyOI3wOq4gdDY7yd0fjMpMIQyRG0cYdVFMmInz14R5MtqEu99hZh8T5oD4HuGLbCnhQ/IywuuGu1eZ2RcI4+p/SPi8mkToH3AGeU4SIqcSphr+EuFYfkKYE2QdYSRQ8vNYbmb7Eb7oE0n1SsJxvyWTB3P3OWa2D2FuklMJx2IeoVbsSt8wR8JcQifjgwlfkl0J13i5DfhNUn+HPwEnE45rX0JHuXeAH7r7CxmE9CPC8T+TMAzzE0ITxba0UZIAmZ8jWcr0GDYV00/NbDrwfcI5UkcYZfN133S+jkwVxHuyOYkx2iIiIiIbUZ8EERERSUtJgoiIiKSlJEFERETSUsfFJGa2npA45Xt2QxERkULUG2hw97T5gDouJjGzBsDKy8vjDkVERKTVVVdXQ7iUTNqWBdUkbGxFeXl5eVVVU8PWRUREOo6Kigqqq6ubrD1XnwQRERFJK7YkwcwONbM7zWyama02s3lm9pCZZTRBioWrwz1sZtVmttLCVe9Gt7yliIiIZCLOmoSzgK0IV6o7inDxka2AN9NdAyCZhetuv0y4JvdphNkK+xKm3BzWijGLiIh0GnH2SfhB6tWtonnTZxEuX3xiM9v+hHDhi73cfUG07cRo20sJc3GLiIjIZoitJiHd5S/dvYowV3hLtQHHA88mEoRo22WE6wSckM84N8cbsyr53l1vUbe+Ie5QREREslZQHRejSwrvTLhCV1NlugOjmijzLjAwao5It21Vc39A3sY+zq9aw9dve42np37Gb5/7OF+7FRERaTMFkyRE11+/lRDT9c0U7UO4HGtlmnWJZamXdG1zQyu6892DwtWJ//ziDCbOWBZzRCIiItkpmCSBcFnS44Cz3P3DDMo3NwtU2nXuXtHcH1CdQ9xN+vFh27HL0HLc4fx/TqZ69bp87l5ERKRVFUSSYGZXAecDP3L3O1sovpyQBKSrLegb3aarZWhzpSVF/O6U3enepZgF1bVc8vB7aIZLERFpL2JPEszsl8AlwIXufnNL5d19DTCT0Hch1S7AknSdIuMyakBPfv6lMH3DE+8u5KG358cckYiISGZiTRLM7HLgMuAyd78ui03HA4eb2RZJ++oLfAl4KL9Rbr5T9t6Sw0cPAuDnj7zPp8tWxxyRiIhIy+KccfF84ArgceA5M9s36W+PpHITzCy1jv56Qv+BJ83sWDM7BngCWA/8um2eQebMjN+cuCsDenWlpq6ec+9/h/X1GhYpIiKFLc6ahC9Ft18EJqb8jW9uQ3f/DDgQmAvcBdwPVAEHufunrRXw5ujbo5QbvrIbAG9/WsXvX5gec0QiIiLN06Wik5hZVWtfBfJXj3/A7a/MosjgX2ftx55b9215IxERkVYQXQWyOhrht4nYOy52Nhd8YXt22KIXDQ7n3j+ZlbUaFikiIoVJSUIb69almJtO2YPSkiLmVq7h8kenxh2SiIhIWkoSYrD9Fr245KgdAHjo7fk8NmVBC1uIiIi0PSUJMTnt88MZu90AAC4d/x7zq9bEHJGIiMjGlCTExMy47iu70q9HKStq13Pe/ZOpb1AnUhERKRxKEmI0sFc3rj1pVwBen1XJX16aEXNEIiIiGyhJiNmhOw7i1H23AuDGZz7mvXl5vcaUiIhIzpQkFIBLjx7NqAE9WN/g/Oi+d1hdtz7ukERERJQkFILupWFYZJdiY+bSGn71eCZXyhYREWldShIKxM5Dy/nJEdsDcO8bn/L01EUxRyQiIp2dkoQC8t0DR/L5Uf0AuOjBd1m8ojbmiEREpDNTklBAioqMG07ejfLuXVi+eh3n/2sKDRoWKSIiMVGSUGAGl3fnmhN2AeDlT5Zyx39nxxuQiIh0WkoSCtBRuwzmK3sOA+A3T33EhwtXxByRiIh0RkoSCtTlX96JrfuVUVffwLn3TaZ2XX3cIYmISCejJKFA9exawu++ujvFRca0z1ZyzVMfxR2SiIh0MkoSCtgeW/Xh3EO3BeDO/87mhWmLY45IREQ6EyUJBe77B2/D3sP7AHDBv95l6aq1MUckIiKdhZKEAldcZNx48u706lrC0lVruejBd3HXsEgREWl9ShLagS37lvGr43YG4LkPF3PP65/GHJGIiHQGShLaieP2GMqxuw8B4MonPmD64pUxRyQiIh2dkoR25JfH7szQiu7UrmvgR/dNpm59Q9whiYhIB6YkoR0p796F3351d4oMpi5YwQ3PTos7JBER6cA2K0kws23MbH8zK89XQNK8z43oy/fHbQPArS/N5L/Tl8YckYiIdFQ5JQlm9kUzmwFMA14C9oyWDzSz6WZ2Uh5jlBQ/OmxbdhtWjjuc988pVK2uizskERHpgLJOEsxsHDAeqAR+AVhinbsvBmYAp+QpPkmjS3ERvztlD8pKi1m0opaLH3pPwyJFRCTvcqlJ+DkwBdgH+EOa9ROBMZsTlLRsRP8eXP6l0QA89f4i/jVpXswRiYhIR5NLkrAXcI+7N9W1fh6wRe4hSaZO3mtLjtwpHOpfPDqV2UtrYo5IREQ6klyShGKgubmB+wNqJG8DZsbVJ+zCoN5dqamr59z7J7OuXsMiRUQkP3JJEj4EDmxm/RcJzRHSBvr0KOXGk3cHYPLcKm55/pOYIxIRkY4ilyThduAkMzsjaXs3szIzuxnYD7g1XwFKy/bfpj9nHjQSgN+/MJ03Z1fGHJGIiHQEWScJ7v4n4H7gNuATwIF7gWrgHOBOd78nn0FKy84/YjtGD+5Ng8OP75/Mitp1cYckIiLtXE7zJLj7qcCJwPPAR4ThkE8CX3H3M/IXnmSqa0kxN52yO11Lipi3fA2XPzI17pBERKSdy3nGRXcf7+4nuvtO7j7a3Y919wfzGZxkZ9tBvfjZMTsCMP6d+TwyeX7MEYmISHuWy2RKJWbWu5n1vc2sZPPCklyduu/WHLLDQAB+Nv595i1fHXNEIiLSXuVSk3AD8FYz698EfpNbOLK5zIxrT9qV/j1LWbl2PefdP4X6Bs3GKCIi2cslSfgC0FyzwoPAUbmFI/nQv2dXrjtpNwDemF3Jn1+cEXNEIiLSHuWSJGxJuD5DU2ZGZSRGB+8wkNP22xqA3z77MZPnVsUckYiItDe5JAl1wOBm1m8BaNq/AnDx0Tuy7cCerG9wzr3vHWrWro87JBERaUdySRLeAU42s9LUFdGyrwLvbm5gsvm6dSnmplP2oLS4iNnLVvOrxz+IOyQREWlHckkS/gDsBDxhZnuZWWn0txfwODAa+H0+g5TcjR7SmwuP3B6A+96cy7/fXxhzRCIi0l7kMuPig8DVwKHA68BqoCb6/zDgWne/P5N9mdkwM7vJzF4xs1Vm5mY2LsNtr4jKp/4tyvY5dXTf3n8EB2zTH4CLHnqPRdW1MUckIiLtQa4zLl4K7EOoMXgaeBa4GdjH3S/OYlfbAF8DVhFmb8zF4YTrRST+js5xPx1WUZFxw8m7UVHWharV6zj/X5Np0LBIERFpQc6THrn7m4Q5ETbHS+4+EMDMjgO+nMM+3nJ3dd1vwaDe3bjmhF056+5JvDp9GX97dRbfOXBk3GGJiEgBy3la5nxwd42CaENH7rwFp+wdRqde++9pTF1QHXNEIiJSyHKqSTCzrYDvAdsC/QBLKeLufuhmxpapD81sILCY0HHyUndfnK6gmbVU41Ce7+AKzWVfHM3rsyqZtbSGH903mcfOOYDupcVxhyUiIgUol2s3HAV8DFxMmH1xJDAi5a8t6rFnAJcA3yL0S/gjcArwmpn1aYPHb5d6dC3hd1/dnZIiY/riVVz91IdxhyQiIgUql5qEq4GlwHHu3tw1HFqVu9+Vsug/ZvYa8AzwA+DKNNtUNLfPqKahw9cm7LZlBT8+fDuue3oaf584h3HbD+CQHQbFHZaIiBSYXPok7AD8Ls4EoSnu/iywkDDKQZpx1thRfG5EXwAufOBdlqxcG3NEIiJSaHJJEpYQpmYuVEVoWugWFRcZN568G726lbB0VR0XPjAFdw2LFBGRDXJJEu4CTsx3IPlgZkcAg4DX4o6lPRjWp4yrjt8FgBemLeGu1+bEHJGIiBSSXPok3AkcbGaPADcBs4D61ELu/mkmOzOzk6J/945ux5pZf6DG3Z+KykwAxrq7JW33DvB3YBqwDvg88BNgOmHqaMnAl3cbwoSPFvPQO/O56okP2W9kP7Yd1CvusEREpABYtlXMZtYAOGHYY5Mbu3tG4+rMrKl9zHH34VGZCWyaJNwL7AUMAboAc4FHgV+5e2Umj50mlqry8vLyqqrONTfTytp1HH3zy8ytXMOOg3vz8A8+T9cSDYsUEenoKioqqK6urm6qY38uScIVNJMcJLj7L7LacQHorEkCwFuzKzn5LxNpcPjugSO49JjRcYckIiKtrKUkIevmBne/YrOjkoKz1/C+nHPIttz8/Cfc9vIs+vQo5bsHjqRLcayTcoqISIz0DSCN/veQbdhz6zAP1bX/nsaXbnmFSXOWxxyViIjEJackwcx6mdnPo0s8f2Jm+0XL+0fLd8hvmNIWSoqLuPNbe3P654djBh8tWslJf/4vl45/j+o16+IOT0RE2lgu0zIPAN4CLiNct2Ek0B3A3ZcCpwFn5jFGaUO9unXhii/vxPjv78/owb1xh3te/5RDb3iRx6Ys0FwKIiKdSC41CVcCWwD7AAey6cWdHgHa6uJO0kp237KCR8/Zn58dsyNlpcUsXbWWH977Dqfd8SafLlsdd3giItIGckkSvgj80d3fJv0oh5nAlpsVlRSEkuIivnPgSJ49byyH7TgQgJc+XsLhv32RP06Yzrp6TWwpItKR5ZIk9CdMWNSUBqBbbuFIIRpa0Z3bvrkXfz51T7bo3Y216xu49t/TOObml3lrdk5TUoiISDuQS5KwCBjVzPo9gIxmW5T2w8w4cucteO78sXxr/+EUGXz82SpO+vNELn7oPapXq2OjiEhHk0uS8CRwhpkNTl1hZvsA3yT0S5AOqGfXEi7/0k48/IP92XlobwDufeNTDr1xAo9Mnq+OjSIiHUguMy5uAUwCignTIJ8B3A2UAicAC4A9c50aOU6decbFXKyvb+DvE+dwwzPTqKkLl+84cNv+XHnczmzdr0fM0YmISEvyPi0zgJltBdwCHMOG2ggn1DKc7e7zcow3VkoScrOweg1XPDqVp6d+BkDXkiL+99Bt+e6BIykt0XxdIiKFqlWShMaNzXoD2xOGQU5vj7UHyZQkbJ5npi7i8kensrC6FoBtB/bk1yfswt7D+8YcmYiIpJPXJMHMegLVwBXu/qs8xVgwlCRsvlVr1/PbZz/mjldn0RCdWqfsvSUXHbUDFWWl8QYnIiIbaSlJyKou2N1XAVXA4nwEJx1Pz64lXPbF0Tx6zgHsOqwcgPvenMuhN7zI+HfmqWOjiEg7kkvHxQeAOnf/euuEFB/VJORXfYNz18TZXP/Mx6xaux6A/bfpx5XH7cKI/urYKCISt7z3STCzEcCLwB3ADe6+YvPDLAxKElrHoupafvHYVJ56fxEApSVF/PDgbThz7Ei6lhTHHJ2ISOfVGknCTKAn4eJOAEuA1Mn83d2bm3CpIClJaF3PffAZlz86lflVawAYNaAHvz5+F/YZ2a+FLUVEpDW0RpIwgfTXbNiIux+c1Y4LgJKE1lezdj03Pf8Jt78yi/qoZ+PJew3j4qN2pE8PdWwUEWlLrToEsqNRktB2pi6o5pLx7zNlbjjWfXuUcunRO3LCmKGYpV5YVEREWoOShCwoSWhb9Q3OP16fw7X/nsbKqGPj50f148rjdmbkgJ4xRyci0vG1WpJgZgcBRwCDCB0YP4rmURgDvOvu7e6bVklCPD5bUcsvH/uAJ95bCEBpcRHfP3gUZ48bpY6NIiKtqDX6JBQD/wBOIsy06MDh7v4fM+tGuHbD9e7+680Lve0pSYjXCx8t5mcPv9/YsXHkgB5cddwu7DdKHRtFRFpDXidTivwUOBE4D9iRkCgA4O61wEh/HOEAAB1DSURBVHjg6Bz2K53cwTsM5NnzDuJ7Y0dSXGTMXFLD1257jZ/8awqVNXVxhyci0unkkiR8E/i7u98ELE2z/kOg3Q1/lMJQVlrCxUftyOM/PIDdtwyJ7QOT5nHoDRP411tzNWOjiEgbyiVJGA5MbGZ9FdAnp2hEIjsO7s1DZ3+eK4/bmV7dSli+eh0XPPAup9z6GtMXr4o7PBGRTiGXJGEl0Nxl/bYhTLAkslmKioxT992a588byxd3HQzA67MqOfqml7nx2Y+pXVcfc4QiIh1bLknCK8CplmYwu5n1Ab4NvLC5gYkkDOzdjd9/fQx3fmtvtuzbnbr6Bm5+/hOOuull/js9XYuXiIjkQy5JwlXAtsB/gC9Gy3Yzs+8BbwM9gGvyE57IBuO2H8gz547l++NGUVJkzFpaw9f/+jrn/XMyy1atjTs8EZEOJ6d5EszsaOB2whwJEIZBGuES0t9092fyFmEb0hDI9mPaopVcMv49Js1ZDkBFWRdO2XsrTtpzKNsM7BVzdCIi7UNrTqbUFTicDcMgPwGedvfUiz21G0oS2peGBue+N+dyzVMfsqJ2fePy3bas4KQxQ/nSbkOoKNP1IEREmrLZSUJ01cdz3f3R6P7PgYfc/f28RxszJQnt05KVa7n7tTk89M485lauaVxeWlzEoTsO5KQ9h3HQdgPoUpxL65qISMeVjyRhPXCau98T3W8ATnX3f+Q92pgpSWjfGhqcN2dX8uDb83ji3YXU1G0Y/dC/ZynH7j6UE8cMY/SQ3jFGKSJSOPKRJMwB7nX3i6L7DcA33P3evEcbMyUJHcfquvU8PXURD06az6szlpJ8mu84uDcn7TmMY3cfQv+eXeMLUkQkZvlIEm4GzgHeBSqBcYRZFT9rZjN390NzijhGShI6pgVVaxj/znwenDSPmUtrGpeXFBnjth/AiWOGcciOA3UxKRHpdPKRJHQHLgIOA7YgzLi4DKhpZjPcfUQuAcdJSULH5u68M7eKByfN47EpCzbq7FhR1oUv7zaEE8cMY9dh5aSZBkREpMNpjatAqk+CtHu16+p57sPPeHDSPF78eAkNSW+DbQb25MQxwzh+j6FsUd4tviBFRFpZPmoS/gb8xd1fj+6fBnzg7m/mPdqYKUnonBavrOWRdxbwwKR5TPtsZePyIoMDth3AiWOG8oWdtqBbFzVHiEjHko8kYaOaAzOrB/5HNQnS0bg7Uxes4IFJ83h0yoKNLk/dq2sJx+w6mBP3HMZeW/dRc4SIdAj5SBIWA1e7+2+j+2pukA6vbn0DL0xbzIOT5vGfjxazPqk9Yni/Mk4YM4wTxgxlWJ+yGKMUEdk8+UgSHgYOBP4OLAeuAB4ijHZoirv7r3KKOEZKEiSdypo6Hp08nwfensf781dstG7fkX05ccwwjt5lMD26lsQUoYhIbvKRJAwH/g84gDD9cuI6Dc1xd293DbhKEqQl0xat5MG35zH+nfksWbnholJlpcUcufMWnDRmGPuO7EdRkZojRKTw5W10g5mVEoZAzgbOBR5prry7z8lgn8OAC4A9gd0JV5A82N0nZBjTKOAG4GDCFS1fBn7i7h9ksn2a/SlJkIysr2/g5U+W8sDb83j2g8+oW9/QuG5oRXeO32MoJ+45jBH9e8QYpYhI81pjCOQdwJ8Tox02h5mNA/5JuMT0WuDLZJgkmNlAYDLhypNXAOuBnwGjgD3cfV4O8ShJkKxVr17H4+8t4MFJ83j7043PnT237sOJY4ZxzK6DKe/eJaYIRUTSa7WrQOaDmRW5e0P0/3HAeDJPEq4FfgiMcvcF0bJ+wCzgHnc/O4d4lCTIZpmxZBUPvT2P8W/PZ0F1bePy0pIijhg9iJP2HMaB2w6gWM0RIlIA8tEn4SAAd38p+X5LEuUzlUOS8Anwobt/OWX5PcBh7j4om8ePtlWSIHnR0OBMnLmMByfN46n3F7Fm3YaLTQ3s1bWxOWK7Qb1ijFJEOrt8zZPgQHd3r0u63+Qm5NBxMZskIZoquga4xt0vSVn3U+AaYJC7L05Z19K3f3l5eTlKEiSfVq1dz5PvLeTBSfN4fVblRuuG9enOviP7sd/Ifuw3qh9DKrrHFKWIdEYtJQmZjNn6NiEpWJdyP059CMlIZZp1iWX9CP0VRGLVs2sJJ++1JSfvtSVzK1fz4NvzeOjt+XxauZp5y9fwwKR5PDApdKHZul9ZY8Kw38h+DOytaaFFJD4tJgnufmdz92PWXLKyybqmMqWEqKahfHODEmnKln3LOPew7fjRodvy/vwVTJy5lIkzlvHm7OWsWrueOctWM2fZau57cy4AIwf0aEwa9h3ZT5e2FpE2lfXsL2b2eeAYYDugN7ACmAY84e4T8xtek5YTkoB+adb1jW7T1TKIFAQzY5dh5ewyrJwzDxrF+voG3ptfzcSZy5g4YxlvzV7OmnX1zFxSw8wlNdzz+qcAbDeoZ2PSsM+IfvTpURrzMxGRjiybeRJ6A/cCR5J+MiUHngC+4e4r06xvaf/ZdlycTrjQVGrHxbuBI9x9YA4xqOOiFIS69Q28O6+KiTOWMXHmMibNWc7apLkYAMxghy16NyYNnxvRV8MsRSQr+ZxM6RngMOAV4HbCtMwrCLUJuwLfAfYH/u3uR2cbaA5JwnXAOcAId18ULetLGAJ5r7uflUMMShKkINWuq2fy3A1Jw+RPq6ir3zhpKDLYaUh5Y3+GvUf0paemihaRZuQlSTCzLwBPATe4+wXNlLse+DFwpLs/m0mAZnZS9O/ewIWEiZGmAjXu/lRUZgIw1t0tabtBwBRgAfALNkymtB1hMqVPM3n8lFiUJEi7ULuunklzljcmDVPmVm10ESqA4iJjl6Ebkoa9hvehrFRJg4hskK8k4U5gLDDSm9nAzIqAGcAEd/9WJgGaWVP7m+Puw6MyE0hJEqLl2wLXs+m0zFMzeew0sShJkHapZu163kpKGt6fX019StLQpdjYbVhFY9IwZus+dOvS7i6xIiJ5lK8k4T3gOXf/cQZlf0uYzGiXrKONmZIE6ShW1q7jzdmVTJyxjNdmVvL+gmpS3+qlxUXssdWGpGH3rSroWqKkQaQzyVeSUAlc4u5/zqDsWcDV7t4n62hjpiRBOqrqNet4Y1ZlY03DhwtXbFKmW5ci9ty6T2NHyF2HVdCluCiGaEWkreRjMiUInRMzHbGwEuiZYVkRaQPl3btw+OhBHD46zFa+vKaO12cta0waPv5sFbXrGnh1+jJenb4MCJe/3mt438akYechvSlR0iDSqWSaJBSR3SyL+iQRKWB9epRy5M6DOXLnwQAsXbWW12ZuSBpmLqlhdV09L328hJc+XgKEmSM/N6Ivuw2rYPSQ3owe0psh5d0w08WqRDqqTJsbGoB/EC7p3JI9gVOyvXZDIVBzg0jw2YrajZKGOctWpy1X3r0LoweHhCFxO2pAT0pL9DtBpD3IV5+EhhYLbSzrCzwVAiUJIuktqFrDxBnLeGNWJR8sXMG0RSs3machobS4iG0H9WT04N7sGCUOOw7urYmeRApQvpKEsdk+sLu/mO02cVOSIJKZdfUNzFxSwwcLq/lgwQo+WLiCqQtWULV6XZPbDOvTfZNah6EV3dVcIRKjvM242BkoSRDJnbuzaEVtSBqixOGDhSuabKoA6N2tJEoayhuTh20GqrlCpK0oSciCkgSR/FtZu46PFq3cKHmY9tlK6tanb67oUmxsO7DXRjUOaq4QaR1KErKgJEGkbaRrrvhgwQqWq7lCpE0pSciCkgSR+Ki5QqTtKUnIgpIEkcKzsnYd0xatbKxt+GDhCj5a1HJzxY5JtQ7bDOxJ/56lqnUQSaEkIQtKEkTah/X1DcxcWrNRU8XUBdXNNlf06lrC8P49GN6/ByP692BE/zKG9wv/V5SVtmH0IoVDSUIWlCSItF/uzmcr1m7Sz2F2M80VCX3KuoTkIUoaEonE8P496NlVl9eWjktJQhaUJIh0PDVr1zNraQ2zl9Uwe2kNM5eG29nLVlNZU9fi9gN6dWVEvx4M71/GiP49Qw1E/x4M79dDl9qWdk9JQhaUJIh0LtWr1zErSh4SicSs6P+Vtetb3H5IebfGJoyRUeIwvH8Ptupbps6T0i4oSciCkgQRgdB0UVlT15gwhFqI1Y21EGvW1Te7fZHBsD5lURNG2UZNGEMruutqmlIwlCRkQUmCiLTE3Vm8cu2GBGLphtqHOZWrmxx1kdCl2Niyb1nUhJHoRBn+H9y7G0VFGoEhbUdJQhaUJIjI5qhvcBZWr2H20tXMWrqKWUtXN/aF+LRyNesbmv+87VpSFDVZlLFlnzKG9unOkIruDK0It33KumgYp+SVkoQsKEkQkdayvr6BecvXMGtZDbOWbOj/MHtZDfOWryGTj+JuXYo2JA3lIXEYUtGtMYnYorybOlNKVpQkZEFJgojEYe36euZWrg41D9EIjAVVa1hQtYb5VWtYXdd8H4hk/Xt2ZWhFtyiBSNREdGNoRRlDKrrRt4cmlZINlCRkQUmCiBQad6d6zTrmV61hQVXtRsnDgmjZZytrM6qJgNCkMbRiQy3EhkQi3A5WbUSnoiQhC0oSRKQ9WlffwKLqKIGoDonDhiRiDfOXr6Emq9qI0pA8pGnSGFLRXVNcdyBKErKgJEFEOiJ3Z8Wa9RsSh+o1m9RMfLailhb6VTYqLSliSPnGtRBDK7ozqLwbA3t1ZUCvrvQtK9VIjXZASUIWlCSISGe1rr6Bz1bUNiYOyTURiZqJVWtbnmAqoaTI6N+zKwN7d40Sh5BAhPvdGNCra2NC0UXzRsSmpSRBk5KLiAhdiosY1qeMYX3KmiyzonYd85cnNWOk9JFYsnJt4zDP9Q3h0t+LVtS2+Nh9e5Q2JgwDe3VLSiyi+1FyUVaqr6y2ppqEJKpJEBHJXUODs3x1HYtXrg1/K2pZvHItS1auZfHKWhaviJavrKV2XfOTTqXTs2vJhuShd5Q8pEkuyrtrPolMqbkhC0oSRERan7uzau36KJEIScOStInFWqrXNH3576aUlhQxIKmpI7l5I9HcMbBXV/r17EpxJ+83oeYGEREpKGZGr25d6NWtC6MG9Gy2bO26+saEYcnK2o0Si+RkYtmqtY0dL+vWNzA/agJpTpGFpo7EX78eXTf833PD/X49w7I+ZaWdLqlQkiAiIgWrW5dituxbxpZ9m+4rAWFGy8qausbmjOSmjQ21FCGpqKsPTR0NDktX1bF0VcuXDAcwg4ruXTZOKHqW0i9NotGvZ0gq2vvVQJUkiIhIu1dSXBT6KfTuBpQ3WS4xOVUiaVhWs5Zlq+qorKljWU0dy1atpbJmw/3k5g53WL56HctXr2PGkpqM4urdrYR+PZNqKJITip6l9O3RdaNlhTaRlfokJFGfBBERSbauvoHlq0PSULkqJA6JBKKyJiQUiSSjsqaO5avrMp5vIp0epcX0TUke+jXWVmxYtu2gnnkZ7aE+CSIiIjnqUlwUdXTsllH5+oZQU1EZ1VAsSyQUq0JSkUgyEonG8pq6ja4OWlNXT03lGuZWNt+f4sGzP8+eW/fZrOeWCSUJIiIieVJcZI1NB9sMbLl8YjbMZYlaieQkIiWxSNRY1NU30K9Haes/GZQkiIiIxMbMKC/rQnlZF0YOaLl8YvhoW00spSRBRESknUgMH20r7XtshoiIiLQaJQkiIiKSlpIEERERSUvzJCQxswbAysubnohDRESko6iurgZwd09baaAkIYmZrSfUrqzI0y4T2UZ1nvbXFhRz22mPcSvmtqGY24Ziht5Ag7unHcigJKEVmVkVQFMzWRUixdx22mPcirltKOa2oZhbpj4JIiIikpaSBBEREUlLSYKIiIikpSRBRERE0lKSICIiImkpSRAREZG0lCSIiIhIWponQURERNJSTYKIiIikpSRBRERE0lKSICIiImkpSRAREZG0lCS0AjMbZmY3mdkrZrbKzNzMxsUdV1PM7FAzu9PMppnZajObZ2YPmdkuccfWFDP7vJk9bWbzzazWzJaY2X/M7Ki4Y8uGmV0RnR+T446lKWY2Loox3d8OccfXnCj2Z8ysKjq3PzCzM+OOK53oPdjUcXYz2yLuGNMxsz3M7GEzW2BmNdExvsjMusYdW1PMbP/o86LGzFaY2ZOF8nmXzfeHmR1uZq+Z2RozW2xmfzGzvF74SUlC69gG+BqwCng+5lgycRawFfBb4CjgvOj+m2a2b5yBNaMPMA04HzgSOBNYCzxpZqfEGVimzGwn4KfAZ3HHkqGfAvul/M2OM6DmmNlpwHPADOAU4EvAH4DSOONqxq/Y9PgeBKwBXnf3RTHGllaUJP4XGA6cSzjGDwFXAbfFF1nTos+0F4AS4BvA6UB/4CUz2ybG0BIy+v6IEocngbmE4/4T4MvAE2aWv+92d9dfnv+AoqT/jwMcGBd3XM3EOzDNsgpgOfBg3PFl8TxKojfMf+KOJZNzBHgNuAWYAEyOO6ZmYh0XncPHxR1LFjFvCawGLow7ls18HidEx/67ccfSRHxXRPGNSll+F7AO6BJ3jGlifgZYAHRPWlYBVAL3FEB8GX1/AG8A76SUPzwq/9V8xaOahFbg7g1xx5ANd1+cZlkV8AkwrO0jyo27rweqCR9Ohe7HhGN7adyBdFBnRLe3xBrF5vs2Idm5P+5AmpB4r1WnLE+8D+vbNpyM7Ef4IbEmsSD6vHsZOM7MimOLjMy+P8xsKLA3cFdyeXd/FpgPnJiveJQkSFpmNgDYGXg/7liaY2ZFZlZiZkPM7BfAdoRmk4JlZiOBXwLnuPuKuOPJwl/MbL2ZVZvZ42a2Z9wBNeMg4EPghKivTX3U1+YaMyvU5oaNmNlgQlPaAwV8ntxF+AX+JzMbYWa9zexY4DTghgL9wVRKaJpMtRYoA0a2bTg52Tm6Tff5/F7S+s1Wkq8dScdhZgbcSkgir485nJb8kw1Z8wrgZHf/d4zxNCs6trcBT7v7w3HHk6Fq4HeEZpFKYEfgIuBVMxvr7q/HGFtThkR/twCXAVOBQ4CLCU0R34gvtIydBhQDt8cdSFPc/dOojf9hYGbSql+7+2UxhdWSD4B9zcw8qqM3sy7A56L1/Qm1qIWsX3RbmWZdJTAmXw+kJEHSuY7QFvYtd/8w7mBacCHwG2AL4OvAP83sNHe/N96wmvRdYC9gdNyBZMrd3yG0fSa8bGaPEn7FXAUcFktgzSsCegFfc/f7omUTzKw78BMzu9zdp8cXXkZOB6a7+0txB9IUM9saeAxYBBwPVAFjgYvNrKFAE4VbCInXzWZ2DeFc+SUbmlYLsfajKU1dVyFv11tQc4NsxMyuIowY+JG73xlzOC1y95nu/qa7P+buXwOeBv6Q1969eWJm/YFrgauBGjOriIYrlQDF0f1usQaZIQ897Z8BCnX0y7Lo9umU5U9Ft3n7pdUazOwAYHvgjrhjacE1hGTsKHd/2N0nuPsvCOf4JWY2PM7g0nH3vxFqwk4H5gGfEo51otZ0QTyRZSVxfvdLs64v6WsYclJwH6QSHzP7JXAJoUf4zXHHk6M3CMMjB8QdSBrDgHLCB+jypL/9CW2Iywm9xduLIvL4iyXP3mtiuUW3hf5r8duETn//F3cgLdgD+CC5E2DkLcL5UZDzaLj7bwjNCrsAw93984Qv1znuPjfW4DIzNbpN1/dgF/LYl0xJggBgZpcT2m4vc/fr4o4nF1F7/zhCleey5kvHYjpwcJq/KYSx/AcT+oIUvGhin8MJwzgL0UPR7dEpy48mJDZvtm04mTOzHsBXCP1W5scdTwsWADubWVnK8v2i24KN393Xuvv77j4nqvH4KvD7eKPKjLvPIyRi30iuNTWzQ4GhbDj/N5v6JLQSMzsp+nfv6HZsVN1c4+5PNbFZLMzsfMIv2MeB51ImUFobtUkXFDO7B5gDTAKWAoMJHb0OAX4YDYcsKO6+itD5byNmVhWt32RdIYiO9UzgbUJtxw6EiZW6EzoCFhx3/7eZPUVoeurPho6LPwL+7O5zYg2weV8FegJ/izuQDNwMjAeeNrPfETq5jiP0FXrO3Zuq0YmNme1G6HP1FmFEw+6E8/gNQgfd2GX4/fFTQpPfvWZ2K6Gj7m+A14F/5S2WqHOn5JmZNXVg57j78LaMpSVmNoHQ2SidgosXwMzOIfRQ345QhV9NeNP/3t0fizO2bEXHv8Ldd487lnTM7CLCjIXDgR6EWpoJwJXuXrBDZKNf5L8gzF43gND2/Ffg2gIdmgeAmb1MSMSGuHvBz/lhZocR2vh3ISQ3s4H7gBvdvSbG0NKKZom8lVBV34OQAP+dEG+6oZFtLtPvDzM7knCO7wasJIwyudDdl+ctFiUJIiIiko76JIiIiEhaShJEREQkLSUJIiIikpaSBBEREUlLSYKIiIikpSRBRERE0lKSICKY2elm5mY2Lu5Y8s3MZkdzUYhIlpQkiHQwZjYu+sJP/NWb2XIze9/M/s/MjoymsBYRaZYmUxLpYKLagBeAe4EnCRc16kW40t1xwFbAc8BX3L0q2qYY6ALUFfJshLkws66Au3td3LGItDe6doNIx/W2u9+dvMDMziNcrvo8QhJxFIC71xOuOtjhFMpUuyLtkZobRDoRd6939/OBV4AjzewASN8nIWnZoWb2czObY2ZrzOz1xEXAzGysmb1iZjVmttDMLkv3uGa2l5mNN7OlZrbWzKaZ2aVmVpJSbkLUh2CImd0bNZPUmNnTZrZdStluZnZFtK/VZlZlZu+Z2XUp5dL2STCz48zsVTNbFf29ambHpik3O4prBzN7wsxWmlm1mT0QXQ1TpMNSkiDSOd0e3R6TQdlrCM0UNxEuJjOScNW/4wiXpH0Z+AnwEfBLMzs1eWMzOxp4lXAxrhuA/wUmAr8k1Gak6gG8RKjZuAT4A+HKgo9EzSIJfwAuJ1yu+jzgUuB5wtUem2Vm3ydcvbAvcCXwq+j/h83szDSbDCVc1OpT4ALgH8AJhAsDiXRYam4Q6ZzejW63a7ZUUAzsm2jTN7MPgEeAB4D93P3NaPnthMt3/wC4O1rWjXDJ49eBQ5Iu4f0XM5sC3Ghm41Iuk90fuM7dr00sMLMlhGaSw4Cno8XHA0+5+2nZPHEz6xPtawawj7uviJb/CXgHuMHM/pnorxHZBviqu/8zaT8NwPfNbAd3/yibGETaC9UkiHROK6Lb3hmU/VNKp7+Xo9vXEgkCQFTmDWDbpLKHA4OAO4AKM+uf+CN0qgQ4IuXxGoCbU5b9J7pN3nc1sJOZ7ZzBc0h2OKG24uZEghDFvwK4hXC548NStlmQnCCkxLRNlo8v0m6oJkGkc0okByuaLRXMTL7j7sujEZSz0pRdDvRLur9jdPu3ZvY/KOX+AnevTVm2LLpN3ve5wF3Ae2Y2kzCi4zHgsRZGaIyIbqemWfd+dDsyZfnM1IJNxCTSoShJEOmcdo1up2VQtqlRD5mMhkjMx3ABMLmJMguy2G/j/A7u/oiZDQeOBsYSfv2fAbxsZoc1M+QxlzkiMopJpKNRkiDSOZ0R3T7Ryo/zSXRb4+7P5Xvn7l5J6P9wdzRB1DXAhcCxwL+a2GxGdLsToaNjstHRbbqaA5FOR30SRDoRMys2s+uBA4An3f3VVn7Ip4HFwEVm1jdNPN3NrFe2O42eR0XyMg8zw70T3d3ksZI8C9QAP0x+7Oj/HwKrojIinZ5qEkQ6rjFJwxGTZ1zcGngG+HprB+DuNWb2TeBhYJqZ/Q2YDlQAOxCGER5PGF6YjV7AQjN7lJAYLCb0NTib0C/isWZiqjKzCwlDKF83szujVacTOiF+z92rs4xHpENSkiDScX0t+msg/DqeB7wI3Ovu/26rINz9aTPbG7gIOBUYQPginwHcyIbhmNlYDfwOOJTQF6EnsBB4FLja3VP7OaTG9EczW0joK3F5tHgKcLy7P5xDPCIdkq7dICIiImmpT4KIiIikpSRBRERE0lKSICIiImkpSRAREZG0lCSIiIhIWkoSREREJC0lCSIiIpKWkgQRERFJS0mCiIiIpPX/zMkiiaYZbiIAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "d = 10\n", "mu1 = 1 / np.sqrt(np.arange(1, d+1))\n", "mu2 = -mu1\n", "\n", "f, ax = plt.subplots(1,1,figsize=(8,3))\n", "ax.plot(np.arange(1,d+1), mu1 - mu2)\n", "ax.set_xticks(np.arange(1,d+1))\n", "ax.set_xlabel('Dimension')\n", "ax.set_ylabel('Difference')\n", "ax.set_title('Difference in class mean across dimensions')\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# Simulation\n", "def simulate(n, d):\n", " n1 = np.random.binomial(n, 0.5)\n", " n2 = n - n1\n", " mu = 1 / np.sqrt(np.arange(1, d+1))\n", "\n", " X = np.vstack((\n", " np.random.normal(mu, 1, (n1, d)),\n", " np.random.normal(-mu, 1, (n2, d))\n", " ))\n", " \n", " return X, np.asarray([1]*n1 + [-1]*n2)\n", "\n", "class BayesClassifier:\n", " def __init__(self, method):\n", " self.method = method\n", " if method=='optimal':\n", " self.mu = 1 / np.sqrt(np.arange(1, d+1))\n", " \n", " def fit(self, X, y):\n", " if self.method == 'naive':\n", " self.mu = np.mean(X * y[:, None], axis=0)\n", " return self\n", "\n", " def predict(self, x):\n", " # if self.method == 'optimal':\n", " proj = x @ self.mu\n", " return 2*(proj > 0).astype(int) - 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With Gaussian models for the distributions of each class, we classify an observation as the class with the greatest likelihood. Given a mean $\\mu$ and covariance $\\Sigma$, the likelihood of an observation $x$ is \n", "\n", "$$P( x ; \\mu, \\Sigma) = \\frac{1}{\\sqrt{(2\\pi)^d|\\Sigma|}}e^{-\\frac{1}{2}(x - \\mu)\\Sigma ^{-1}(x - \\mu)^T}$$\n", "\n", "Because we know our covariance matrix to be the identity, this simplifies to\n", "\n", "$$P( x ; \\mu) = \\frac{1}{\\sqrt{(2\\pi)^d}}e^{-\\frac{1}{2}(x - \\mu)^T(x - \\mu)}$$\n", "\n", "and we make the classification $y = argmax_{y \\in \\{0,1\\}} P( x ; \\mu_y)$. In the case of the Bayes optimal classifier (where we know all the parameters), the probability of error for a given dimension $d$ can analytically be written as\n", "\n", "$$P_{error}(d) = \\int^\\infty_{\\sqrt{\\sum_{i=1}^{d}\\frac{1}{i}}} \\frac{1}{\\sqrt{2\\pi}} e^{-\\frac{1}{2}z^2} dz$$.\n", "\n", "Clearly, in the limit as $d \\rightarrow \\infty$ this error $P_{error}(d) \\rightarrow 0$ as the lower limit of integration diverges. This means, that we can perfectly discriminate the two classes in the limit **if we know the parameters**.\n", "\n", "However, as in most real cases, we don't know the true parameters and need to estimate them. Here, we assume that only the means are unknown, a generous assumption. Thus, for each class we have a our max-liklihood estimate $\\hat{\\mu}_i$ of the mean and as Trunk showed, the analytical probability of error is\n", "\n", "\\begin{align}\n", " P_{error}(n,d) &= \\int^{\\infty}_{\\theta(n,d)}\\frac{1}{\\sqrt{2\\pi}} e^{-\\frac{1}{2}z^2} dz,\\\\\n", " \\theta(n,d) &= \\frac{\\sum_{i=1}^d (\\frac{1}{i})}{\\sqrt{(1 + \\frac{1}{n})\\sum_{i=1}^d (\\frac{1}{i}) + \\frac{d}{n}}}.\n", "\\end{align}\n", "\n", "In this case, however, it is apparent that for a fixed $n$, $\\theta(n,d) \\rightarrow 0$ as $d \\rightarrow \\infty$. Thus, the probability of error integrates over half of the symmetric density and $P_{error}(n,d) \\rightarrow \\frac{1}{2}$ in the limit. Thus, if we do **not** know the parameters, increasing the number of dimensions catastrophically hurts our classification performance.\n", "\n", "This distinction between the optimal and learned classifier is demonstrated empirically in the next plot and holds, as expected." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "scrolled": true }, "outputs": [], "source": [ "dim_range = np.asarray([10,50,100,250,500,1000,2500,5e3,1e4,5e4,1e5,5e5,1e6]).astype(int)\n", "optimal_accuracies = []\n", "naive_accuracies = []\n", "for d in dim_range:\n", " X_train, y_train = simulate(n=100, d=d)\n", " X_test, y_test = simulate(n=2000, d=d)\n", " \n", " bayes_optimal = BayesClassifier('optimal')\n", " bayes_naive = BayesClassifier('naive').fit(X_train, y_train)\n", "\n", " optimal_accuracies.append(np.mean(y_test != bayes_optimal.predict(X_test)))\n", " naive_accuracies.append(np.mean(y_test != bayes_naive.predict(X_test)))" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdEAAAD7CAYAAAA1g65/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xUVdrA8d+TnhBIAqEoCKKIgoioVAEBBSlWdO0o2EURUbEXEFgrrrIuYEEBXX2xouvKKhYQF7EuigqCoFGQXhICAUIyz/vHuUkmk0nIhEkmCc/3w3wm99z23Jshz5xzzz1XVBVjjDHGhC4q0gEYY4wxNZUlUWOMMaaCLIkaY4wxFWRJ1BhjjKkgS6LGGGNMBVkSNcYYYyrIkqgJKxGZLyIZtTkOERkrIioih1bG9iuDiLQUkbdFZJMX+4xIx1QRIjJDROy+vCpUkz8vVcGS6AFERNJEZLf3n2JIpOOpzkTkbBEZG+k4wmgG0At4BLgUeCbYQt5no7yvQ6ss+hCJyKFejMPCuM19nY8e4dqXqTliIh2AqVKXAHHAb8CVwD8rYR+nAlIJ261qZwNDgbFB5k0AHgb2VGVAFSUi8UBP4B+qOnEfi18aMN0TuAZ4FvgsYN6m8ERYo3wHPF7KvOVVGYipHiyJHliuBOYB7wBPisjhqroqnDtQ1dxwbq86UtU8IC/ScYSgMe6LzdZ9Laiqxb5YiUgMLokuCpwXSETqqmr2/gRaA/y5r/NgDizWnHuAEJHjgQ7ATOBlYC9weQjri4iMEpElIpItIttFZLmIPC8isX7LlbgWWVDmNbHNFpFMEdnmXd9KFpEoEblbRH7zmpv/JyLdA7YxzGsy6x0ktnJd/xSRzt4+V4hIjnccC0VkcOD2cLXQwCa8YV5Z0Gui3vG9JCIbRGSPiKwSkQdFJClguYL1j/Tmr/GW/15EBu3rOPy2ky4ik0VktYjkeu+TRaSB3zIzgN+9yTF+x9K7vPspZd8Z3nk/TkQ+EJEsYEnA8R1a2noBZer9XrqJyKcislNENovINBFJLkcsiSLyjncOLi5juXJ9hveXiJwmIj4ReT6gPNnb3wYRaeKVRYnIPSKyQETWe8fwh4hM9f89essWNFGPFZHzReQ7EdklIitF5HJvmeYi8oaIbPWO8Z8iUjdgOzO87TQUkRdFZIt3zj8WkeNCOM6+IjJX3P/n3d55va7iZ65msprogeNKYCfwpqruFJH3gKEicr+q+sqx/r3AOOBd4GkgH2gJnAnE45JyWeoAnwALgDuBTsAVQAKwBegCPAXEAqOBd0WkRZhrNoOBo4DXcImlAS5ZviUil6jqK95yf8V9wexJ8ebNz0vbsIi0AL4CUoCpwAqgN3AX0F1ETvFqsP5m4s7bRFwz+yjgbRFpraoZZR2IiKR48bQCXgD+BxwHDAdOFpHO3rl7BtcE+QQwG3jL28SysrZfTs1xv9PXgTeBfSa8MnQA/g1MB17BnbsrAR+uJhyUl2jeBdoBg1T1ozL2sb+fYYBYEUkPUq6qusX74T0ReRK4WUQ+VNVZ3jJTgCOA01R1vVcWB9yGO3/v4P6PdsIdew8ROSFI687pwHXe9rZ6y74gIrnAg7jfyd0U/R/bDVwVJOb3vfXHAk2AEcACEemmqj+WdRJE5BrcOfwC9/9lJ9APmCquheu2stavVVTVXrX8hUtUW4EZfmVnAQoMLOc2/gcsLcdy84GMIGUK3BZQ/hbuj+Q3QKxf+Zne8tf6lQ3zynqHsM/AsjpB1k3CXctaGlA+w/33CHqMY71YDvUre9krGxSw7GNe+ZVB1v83IH7lnbzyh8pxnv/qLXt9QPkNXvl4v7JDvbKxFfjsFJz3YQHlGV75VeU5PwHrzQ8oU+9z0DWg/D1cYksO9nvxjutnYB1wXLg+w2Wsr2W8dgQsG+d9rrOAw4Ah3nITA5YTIDHIvq70lj8/yO9xJ9DCr7whLlH6gFuC/B/LDXYOvXn+n78TvG28H+S4/f92HOTt75UgcU/CfTk5vKLnuaa9rDn3wHAOkIar+RR4D9iI+6ZaHllAU6l4D8R8XE3T32e4PyJPq+regHJw39rDRlV3FvwsIkleLSYJ9829jYjUq8h2RSQKl/gXq+qcgNkP4f4wDS6xIkwqyAZefF8D2ZTvuAfjOvY8G1D+DLC5lP2F21ZczTEcFqnqFwFln+Bayw4NXFhEOuBq4gKcqKqLy7GP/f0MA3yJq3EFvk73X0hd7fECL77ZuNaJb3AtE/7Lqaru8o4pWkRSvZruJ94iXYLE8Laq/u63jU24L4I+YHLAsp/hWncODbKdRwM+f98CHwJ999GM/hdczf1575JC4QtXy48CTilj/VrFmnMPDFfi/uCuEZFWfuUfAueJSLqqbt7HNu4G3gY+E5G1uJree8AbWr7OROtUdXdA2Tbv/Tf/QlXdJiLgmlvDRkQa4XrWngU0CrJIKrC9AptuiGvK/ClwhqpuFZF1uNpIoF+DlG2lfMfdEvhGA5qIVTVPRJYDx5djG/trlarmh2lbwc7FFu892PlYAOwAupfjs1tgfz/DAJu17CbjQqq6SkRuAZ4DdgEXBXxZBEBEzgduxTXHB16bTQuy6WDnahvu/1hgj/GC/2PBzmGwJv2luB72LQjyefa08d7LOg+Ny5hXq1gSreVEpCXQB/eNeEUpiw0BnixrO6q6SEQOB/p72+sDXAzcKyI9VHVfPT/L+mNb2jz/W2XKusF+n59jcVl5Lu4PwN+Br3E1k3xcB6uLqXhHu4re0lOe467Ockopr8jvqqzPR7Dz8QpwLXATcF8Z6xYFtf+f4YooqKEmAkcCK/1nisg5wKu46+k3AatxTaXRuGuWwT6TpZ2rUM9hRZcrWOYyXFN6MMESfa1kSbT2uxz3ob8ayAwyfwKuplpmEgVQ1R24DhBvAojI9bjmoytx1/4qU8EfuPpB5rVk351C2gPHAuNUdYz/DBEJ1ukilFFxNuKaYY8OnCEiabhrSN+FsL3y+BU4UkRi/Guj4m5JaU1k/4j5/64yCgpFJAF3LlYGWSdUw3G/83tFJFZV7yzPSlX5GRaRG3GtHg/jLqnMEJH2quqfeC7FJc0+qprjt+5R4YylFG1wHYMCy/Ip6tEdzC/ee7lr5bWZXROtxbxrdcOAH1R1mqq+EfgC/g9oJyKd9rGtYD0S/+e9B0ts4VZQi+7rXygiFwEHl2P9gm/pxb5pi0g7gl8/3OHN3+exqevd/C5wnIgMCJh9J+7/2exyxBiKt3HNyIFfAK72ysO9v1AE/V0BNxOmvznetcQbcb2O7xCRv+1rnar8DIvIsbikPA+4B7gQqAe85P2/LJCP+8IW5beu4HoSV7bbvX0V7Pd43O/sY+/LRmleww008oCIJAbOFJEUcQN8HBCsJlq7nQocAjxfxjJv4npTXolr4izNMhH5AtexYi2uRnENruffrDLWCwtVXS4iHwHXev/xv8PdFjEYV7PZ131+y3DXeG4Xd9/mclyN7VrgR0peQ/wC1+V/inc70F7gS1X9jeDuxnUweVtEpngxnYTrXLKA4p26wuFR4DxgsvfHbzHumtqVuGN7NMz7C8VHuF6z47zOW78BPYCuuE5PYaOqt3i3dtzh1cpHlrF4OD7DTaX0ITMXeddB63jbywaGeF+yFovIHXhJH9fhDOAN4FzgExF5Efc5PhvX4a2ytQA+EJF/4c7FCNy12zJvT1HVNSIyHJiGO6cv4WquDYFjcPG3xa8VolaLdPdge1XeC3f/ngLH7GO55bim3hJd7f2WuROXDDbivoWu9rZ/fMBy8ynH7SZe+TBKv22lWLd6r6yJt8/tuJrif3DNT+XaJ+6Pxuu4TlY5uOtQgwl+y0oU7v7NNRTVFoZ580os75W3BF7yzlEurkn1QSApYLmg63vzMgi4BaSM30lD3L2Ca3BJfg2uaTI9YLlDqZxbXEqNE/cF5X3vPGfiai9Ng60X7Hdd2ueDUm49AsZ7y07F77aNinyGyzimsm5xKbzdB3ffro+A2528ef/2fldd/MquxnXo2Y27xvgsrmYceGtJqb9HQvg/RtEtLg29z+sW7/f0CXBCef4veuXdcS0eBZ/3tbia961AQqiftZr6Eu9kGGOMOQCIG8VqqKrWlA5s1ZpdEzXGGGMqyJKoMcYYU0GWRI0xxpgKsmuifkQkD/fFoiKj1hhjjKmd6gE+VS1xR4slUT8i4gMkJSUl0qEYY4ypJrKyssD1Ci/Remv3iRa3PSUlJSUzM9jAPsYYYw5EqampZGVlBW2htGuixhhjTAVZEjXGGGMqyJKoMcYYU0GWRI0xxpgKsiRqjDGm9snPg29nwJLXK3U31jvXGGNM7aEKv8yFD++HTT9DnUZw5ACIr1spu7MkWgE7d+5k+/bt5OXl4fP5Ih2OMSGJiooiISGB9PR0/B4naUzNt3YxzL0PMj7zCgRanQJ5uVBJTzi1JBoCn8/H2rVryc7OJioqitjYWKKjoyMdljEh2bt3Lzt27GDPnj00bdrUEqmp+bb9Dp9MgB9eKyo7rDf0Gw8Hta/UXVsSDUFWVhbZ2dmkp6fToEEDoqLskrKpmbZu3cqGDRvYvHkzDRs2jHQ4xlTMrm3w2ePw5TOQn+vKGh0N/ca5GmgVfEG0JBqCHTt2EBcXZ81gpsarX78+mZmZ7N69O9KhGBO6vD3w9TRY8JhLpAB1D4KT74VjL4KoqmshtCQaAp/PR0xMjCVQUytER0fbNX1Ts6jCT2/BRw9A5u+uLC4ZeoyCrjdAXFKVh2RJ1BhjTPX3++cw917481s3LdHQ8XLodSckR+6ShCVRY4wx1dfmX+DDMbD8vaKyo06HvmMh/YhIRVXIkqgxxpjqZ8dGmP+wGzBB811Z045w6nhocWJEQ/Nn3UsNADNmzEBEir0aNWpE3759+eSTTyIdXlj89NNPXHrppTRr1oy4uDgaN27M4MGDWbhw4X5t9+GHH+btt98uUT527NiIXT/v3bs3vXv3jsi+jdkvuTnw6WPw9+Pgm+ddAk07FP4yHa76qFolUChnEhWROiJyv4j0r+yATGS9+OKLLFq0iM8//5ypU6eye/duBgwYwNdffx3p0PbLm2++yfHHH893333HuHHj+Oijj3jyySfZsmULPXv25KmnnqrwtktLoldddRWLFi3an7CNOXD48uF/L8FTx8O8CZC7AxLToP9DcMNX0O6cKrllJVTlas5V1Z0icjcwopLjMRF2zDHH0KFDh8LpPn36kJ6ezptvvkmnTp0iGFnFrVq1issuu4wOHTowb948kpKKevCdf/75nHvuuYwaNYoTTjiBE08M37fcZs2a0axZs7Btz5haSRVWfAAfj4ONP7my6Hjoci30vBUSUyMb3z6E0py7CmhSWYHUZHn5PlZvzak2r7z88N22UKdOHaKjo8nNzS0s2717N7fccgvt27enXr16pKenc9JJJ/Hxxx8XLpOTk0P9+vW57rrrSmzzhx9+QER44YUXCsvWrFnDFVdcwUEHHURcXBytW7fmySefLLZefn4+EyZMoHXr1iQmJpKWlsZxxx3H9OnTyzyGJ554gpycHJ566qliCRTcbR6TJ09GRHjooYcKywuaYhcvXsxpp51GcnIyDRo0YPjw4ezYsaNwOREhKyuLmTNnFjaDDxs2rNg2/IkIo0aNYsqUKbRq1YrExES6dOnC4sWLycvL4/7776dZs2akpKQwePBgNm3aVGz9WbNm0a9fP5o0aUJSUhJHH300EyZMYM+ePWWeA2OqpV/nw/P94P8uKEqg7S+AG79x1z6reQKF0DoWTQFuF5GpqrolHDsXkWTgQeA8IBX4CRinqv8KYRsCfAz0ASap6qhwxBaKdVm76fnovKrebak+u70Ph9Sv2P1S+fn55OXloaqsXbuWCRMmAK7GVmDPnj1kZmZy5513cvDBB5OTk8M777xDv379mDt3Ln379iUpKYkrrriCZ555hkcffZR69eoVrj958mTS0tK46KKLAFi7di2dO3cmOTmZBx98kEMOOYQPP/yQ0aNHs2XLFsaPHw/Ao48+ykMPPcS4ceM44YQTyMnJYenSpWzdurXMY/rwww9p3LgxnTt3Djq/adOmdOzYkU8++YT8/PxiQzmec845XHLJJYwaNYovv/yScePG8ccff/Dee66n4KJFi+jXrx8nnXQS9913H8A+RwCaPXs2LVu25PHHH2fv3r3ceuutnHnmmZx66qnk5uYybdo0Vq9ezS233MI111zD7NmzC9ddtWoVZ5xxBrfccguJiYksWbKECRMmsHz5cl566aUy92tMtfHHF26YvsIxboHD+kDfMXDwcZGLqwJCSaLZwFZguYjMBH4BcgIXUtUXQ9jmbOB44HbgN2AYMFtEzlDVOeXcxtXAUSHs05ShY8eOxaaTkpJ48cUX6dq1a2FZSkpKsVpkfn4+p556Kr/++iuTJ0+mb9++ANxwww088cQTvPjii4wY4a4EbN++nZdffplrr72WxMREwNXYcnJy+PbbbznooIMA6Nu3L7m5uTz22GPcfPPN1K9fn4ULF3Lqqady6623Fu570KBB+zymP/74o1gTdTCHHnooX375JVu2bKFRo0aF5RdeeGHhF4l+/foRGxvLnXfeyaJFi+jWrRtdu3YlOjqahg0bFjtHZcnPz+c///lP4fHv3LmTYcOGkZGRUaw2v3TpUiZNmkROTk5hDfqee+4pnK+q9OjRg7S0NIYNG8akSZOoX79+uWIwJiLWLoZP/gorPywqa94NTr4PDu0eubj2QyhJdIbfzzeXsowC5UqiIjII6Auco6qzvbJ5wGHA48A+k6iINAUeBa4E3ijPfivDQSkJfHZ7n0jtvoSDUhIqvO7LL79M69atAdi4cSMzZsxg6NChxMfHc8455xQu9+abb/LEE0+wbNmyYjXBo44q+j7TsmVLTj/9dKZOnVqYRGfOnMnOnTsZPnx44XJz5szhlFNOoWHDhuTl5RWWDxo0iCeffJIvvviCQYMG0aVLFyZMmMDtt9/O6aefTseOHUs0z1aUqgKUaH698MILS0zfeeedfPrpp3Tr1q1C+zr55JMLEyhAmzZtADjttNOKLdemTRtUlT/++KPwvK5cuZLx48czb9481q1bV+x8/fLLL3Tp0qVCMRlTqTYshfkPwrJ3i8oOPs4N03d41YxxW1lCSaLhzhKDgSzgnYICVVWvlvusiLRV1aX72MZUYIGqvhnJofhioqMq3Hxa3bRt27ZYrW3AgAG0b9+eESNGFCbR119/nfPPP58LL7yQ22+/ncaNGxMdHc19993HsmXLim1v5MiR9O3bl/nz59O7d2+mTp3KgAEDOPzwwwuX2bBhA2+99RaxsbFBY9q8eTMAd911F4mJifzzn/9k4sSJxMXF0b9/fx555JFiyTtQ8+bN+e2338o87t9//52kpKQSNbkmTYp3A2jcuDEAW7ZU/IpG4D7i4uLKLC8Y33b79u307NmT5ORkxo4dyxFHHEFiYiJfffUVN9xwA7t27apwTMZUii2rYP5D8MMbuDoWboD4k++BIwfV6ORZoNxJVFU/DfO+2wFLVTWwF8wS//mlrSwiF+ESe9vy7lBEMvexSEp5t3WgiIqKom3btrz++uts3LiRRo0a8corr3DYYYfxyiuvFKu5+Xe4KXDKKafQtm1bpkyZgs/nY9myZTz22GPFlklPT+f444/ngQceCBpDy5YtAYiJiWH06NGMHj2azMxMPvjgA+644w769+9PRkZGqfdk9u3blylTpvDll18GramtWbOGb775hoEDB5Z4tN369esLEye4hA/QoEGDoPuqTPPmzWP9+vV8+umnnHTSSYXl33//fZXHYkyZMlfDp4/Ad68UDZTQoBX0vguOPgdq0ROwKjxikYikA6jq5gpuogGwIkj5Vr/5Ze17EnCPqq6u4P5NOfh8Pn788Ufi4+MLOweJCLGxscWS1k8//cSiRYuC3tJx4403MnLkSDZs2EDLli0ZOHBgsfkDBw7ko48+onXr1sU6IJUlNTWVCy64gJ9//pmxY8eSmZlJWlpa0GVHjRrF9OnTGTlyJPPnzy/WlOrz+RgxYgSqyh133FFi3VmzZnHssccWmwbo1atXYVl8fHyV1AILzndBDRVcM/Tzzz9f6fs2plyy17tHk307o+jRZKnN3fi27S+A6No3SF5IRyQiBwMPAWcBdb2y7bgm2XtU9c8Q968VnPd3XEekf4S0M9Uy+0t7NdUDujb6ww8/FDYfFlwTXbZsGTfddBMJCe5a62mnncbs2bO54YYbOOecc/j1118ZO3YsLVq0ID8/v8Q2L730Uu666y4WLFjAo48+WuI5rOPHj2fu3Ll0796dkSNHcsQRR7Bjxw5WrlzJu+++y9y5c4mOjubMM8+kXbt2dOzYkfT0dFauXMkzzzxDly5dSk2gAEcccQQzZszg0ksvpXPnztxyyy20atWKP//8kylTpvDf//6Xxx9/nB49epRYd9asWcTExHDSSSfx1Vdf8cADDzBo0KBi10PbtWvHp59+ynvvvUfjxo1JT0/n0EMPrcjpL9OJJ55Iamoq1113HQ888AAiwtNPP13iNhhjqtzOLbDwSfjqOcjzvlDWPQhOGg3HXQYxcWWvX5OparleQHNgLeAD/ge85L2+9cr+BA4JYXuLgM+DlHfBJdDzS1mvH5AP9MLdFlPwUtw10lQgprxxBGw7MyUlRUuTkZGhGRkZpc6vyaZPn67eOSx8paWlaadOnXTatGmal5dXuKzP59MJEyZo8+bNNT4+Xo899lh94403dOjQodqiRYug2x8yZIgmJCTo5s2bg85ft26dDh8+XJs3b66xsbGanp6u3bp103HjxqnP51NV1YkTJ2q3bt20QYMGGhcXpy1atNDhw4frhg0bynWMS5Ys0UsuuUQPPvhgjY2N1YYNG+pZZ52lCxYsKLHsmDFjFNDFixfrwIEDtU6dOpqWlqbXXHONZmdnF1v2u+++065du2pCQoICOnTo0GLb8AfoTTfdVKxs8eLFCuj06dOLlRf8ThYvXlxY9tlnn2mXLl00MTFRmzRpojfffLPOmTNHAZ03b17hcr169dJevXqVeT5q8+fZVJGcbaofT1D968GqY+q51yMtVRc+pZqbE+nowiYlJUWBTA2SN0S1rApfEa/Dz/nAuRpw+4mIDATeAl5V1WHl3N404FyggfpdFxWRq4DngKM1SMciERkFPLGPzQ9U1ffLE0fAtjNTUlJSMjODXzr9/Xf3/LoWLVqEuukD2u7du2nRogWDBg3a58AI1cXYsWN54IEH2LZtG6mp1f+G74qwz7OpsD074KtnYOHfYbf39zIhBU4c6UYaiq8b2fjCLDU1laysrCwN0poZSnPuqcCUwAQKoKr/EZGpwMUhbG827taUM/DroQtcBiwPlkA9bwDfBSmfB7yJa+JdEmS+qWJbtmzhl19+4ZlnnmHLli2MHj060iEZY/bH3t3wzQvw37/BTu8yQlwydB0O3UbUiBGGwi2UJJqGG2ChNL/gmlLLaw4u8T0vIg1w1ziHAj1w11wBEJH5QC9VFQBVXQOsCdyY1+lijarODyEGU4neffddLr/8cg4++GCeeeYZjj766EiHZIypiLxcWPwSLJgI2WtdWUwCdLoKetwMddIjG18EhdKcuxL4RlUvLGX+/wGdVLVVuXcuUg837N9fcAl4KW7Yv7f9lpmPXxItY1vKfg77Z8255kBin2ezT758WPKqe65npvu8EBULJwyFnqOh3kGRja+KhKs593Xc2Lm/AQ+rahYUJsI7cddLHw4lMFXdjnsyTKlPh1HV3uXcVs2/a9cYY6oDnw+Wvu0GStjs3YkoUdDhYjjpdkizL14FQkmi44GewB3AaBHx6vQcDEQDC4EJ4Q3PGGNMlVGF5f+BeX+FDT96hQLtznUDJaSXu6HxgBHKiEU5ItILuAI4G2gJCPAB8DYwQ1XzytiEMcaY6kgVfp3nnqzy57dF5UedDn3uhsbWn6E05UqiIpKIe1zZclV9DncLijHGmJru90XwyXj4fWFRWau+0OceaHp85OKqIcpbE92DS5w3AV9WXjjGGGOqxJ//czXPVUWP36NFd/dklRYnRi6uGqZcSVRVfSKyGijfwKbGGGOqpw0/wbwH4ed/F5U1PcE90/Ow3rXiySpVKZSORTOBS0VkkqruqayAjDHGVILNK11v2x/fpHBo8sbHuMeStR5gybOCQnkezedAHvCdiNwoIgNE5KTAVyXFaSrZjBkzEBESExNZs6bEWBZ06NCB3r17V2jbIsLYsWP3L8AQzZ8/HxEp9mrQoAHdu3fnjTci9vx2Y6rett/h7Rtgcif40XuuZ4Mj4C/T4doFcORAS6D7IZSa6Id+P0+i5FNWxCuLxtRYu3fvZsyYMWF9vFZpj0irCo888kjhszc3bdrEc889x3nnncfs2bM5++yzIxKTMVVi+zr4bCJ8OxN8e11Zagt3q8ox59XKx5JFQihn8fJKi8JUGwMGDGDmzJmMHj2aNm3ahGWbXbt2Dct2KqJ169bF9j9w4ECaNGnCrFmzLIma2mnnZvjvE/D1NMhzjzWk7sHQ6zboMKR2P5YsAsrVnCsi8bixbT9X1ZllvSo3XFPZ7rjjDlJSUrj77rv3uezYsWPp1KkTaWlppKSk0LlzZ1577bUSy/k3586ePRsR4dNPPy2x3IQJE4iJiWHt2rWFZXPmzKFXr17UrVuXOnXqcMopp/D1119X+PhiYmJITEwkNzc3pGNRVY466igGDBhQYpubN28mISGB+++/v7Bs27ZtjBo1ihYtWhAXF0eLFi2477772Lt3b7F1p06dSvv27alTpw716tWjbdu2PPxwSAN/GePsyoSPx8OT7WHRP1wCrdMQ+j8EIxdDxyssgVaC8tZE84GPgVspexD6A1N+HmwP9Xnklahe0wo31aSmpnLXXXdx2223sWjRomIPnw70xx9/cMMNN3DIIYeQl5fHvHnzuPjii8nOzubKK68Mus7pp59Oeno6M2bMoFevXsXmvfjii/Tr14+DDz4YgBdeeIGrrrqKCy+8kNGjR5Ofn8/f/vY3evXqxZdffskxxxyzz+Px+Xzk5bkxQDZt2sTUqW7egC0AACAASURBVFNZs2YNTzxR/Gl6+zoWEWHEiBGMHDmSlStX0qpV0cgt06ZNIz8/n2uvvRaAHTt20LNnT7Zu3co999zDUUcdxVdffcW4cePIyMjgpZdeAuD//u//uPHGG7n77rvp1asX+fn5rFixotiXCGP2ac8O+HIqfP4U7M5yZQmp0P0m6HwNxCdHNr5aLpQB6FcDE1V1UuWGFDkVHoB+2+8wqX1lh1d+Ny0JeWzLGTNmcPnll7N48WKOOuooWrduTcuWLQtrjB06dCA1NZX58+cHXd/n8+Hz+bjuuuv49ttvWbx4ceE8EWHMmDGFtdGbbrqJF154gfXr11OnTh0AFi5cSI8ePXj11Vc5//zz2blzJ82aNaNfv37FaoQ5OTm0bt2abt268frrr5d6PPPnz6dPnz4lymNiYnjkkUe45ZZbSl23tGPZsWMHTZs25eqrr2bixImFyx5++OGccMIJhR2WHnzwQe6//37+97//0b590edi0qRJjBo1ih9//JGjjz6aESNGsHDhwmLnqirZAPQ1XP5e+HaGGxw+Z7Mri0uGbjdA1+sPyMeSVZayBqAPpXfu68D5IhLKOqYGSkhI4IEHHmDBggW89957pS43b948+vfvT+PGjYmJiSE2Npbnn3+en3/+ucztX3755ezYsaNYL9mZM2eSlpbGWWe5p+AtWrSIzMxMhgwZQl5eXuErLi6O3r17B20ODmbixIl8/fXXfP3117z//vtcc801jB49mqeeeirkY0lOTmbYsGFMnz6d3bvdtab33nuPjIwMbrjhhsLl5syZQ4cOHWjbtm2x2AcOHAhQGHuXLl34/vvvue6665g7dy5ZWVnlOiZzgCsY33bqiTBntEugMYnugdg3LXHD9FkCrTKhtPlNA/oAH4rIk7hm3ZzAhVT1jzDFVnPUa+o+vNVFvab7vYmhQ4fy+OOPc9dddxX+8ff3xRdf0K9fP04++WQmT55M06ZNiY2NZerUqbzwwgtlbrtDhw4ce+yxzJgxg6FDh7Jr1y5ee+01LrnkEuLj4wHYsGEDQGFSDSTl7JJ/+OGH07Fjx8Lp/v3789tvv3HHHXdw6aWXkpqaGtKxjBgxgqeeeopZs2YxbNgwJk+eTJs2bYrVejds2MDKlSuJjY0NGtPmza7WcOmll5Kbm8u0adOYNm0aIkLPnj156KGH6NKlS7mOzxxg1n0PH9wDGZ+5aYmC44ZA77sPmMeSVTehJNEfcbewCNC7jOUOvFtcomNq3aOBoqKiePDBBznrrLN4+eWXS8x/9dVXiY2N5d133y1MfECJDjulGTp0KLfeeisZGRksWrSIrKwshg0bVjg/Pd095HfKlCl06tRp/w4mQLt27fjPf/7DihUr6Ny5c0jHcsQRRzBgwACmTJlCz549mTt3bolabXp6OsnJyTz3XPAhpguu+QJceeWVXHnlleTk5DBv3jzuvvtuTj31VDIyMkhLSwvTEZsab/ta12no+/+j8O7Cw0+GfuOhSbuIhnagCyWJjqPkvaGmFjvzzDPp3r07999/P4mJicXmiQgxMTFERRW17m/cuJF33nmnXNseMmQId9xxBzNnzmTRokUcffTRxZJl9+7dqVevHj///DPDhw8PzwF5vv/+ewAaNmwIhH4sI0eOZODAgVx77bXUqVOHyy67rNj8gQMH8thjj9G4cWMOOeSQcsWUlJTEaaedxubNmxk2bJglUePs2QELJ7lOQ3m7XFnDNnDqBDiib2RjM0Boj0IbW4lxmGrqkUceoUePHgA0atSosPy0007jiSee4JJLLuGaa65h/fr1jB8/nsaNG5Odnb3P7TZs2JCBAwfyzDPPsHHjxhK3dSQnJ/Pkk09y1VVXsXXrVgYPHkyDBg3YsGED33zzDfHx8YwfP36f+1mxYgVffPEFAFu3buWdd95h7ty5nHXWWbRs2bJCx9K/f39at27Nxx9/zPXXX0/dunWLzb/55pt544036NmzJzfffDPt2rUjNzeXjIwM5syZwz/+8Q9atGjB1VdfTVJSEt27d6dJkyasWbOGhx56iObNm9O2bdt9HpupxXz5sPgl+OSvsHOjK6vT0D1Z5bhLbaCE6kRV7eW9gMyUlBQtTUZGhmZkZJQ6vyabPn26Arp48eIS884880wFtFevXsXKn332WW3VqpXGx8dr69atdcqUKTpmzBh1H6sigI4ZM6bEdmfPnq2AxsTE6Lp164LG9dFHH2n//v01NTVV4+PjtXnz5nrOOefo3LlzyzyeefPmKa7lpPBVt25dbd++vU6cOFF37dpVoWMpcO+99yqgP/74Y9D527dv1zvuuENbtWqlcXFxmpaWpscff7zeddddmp2draqqM2fO1D59+mijRo00Li5OmzZtqkOGDNFVq1aVeWzhUps/zzXaLx+qTu6qOqaee41vpPrRONXd2yMd2QErJSVFgUwNkjfKvMVFRJ4FpqnqV950LHAGsEBVNwcs2w+4T1Vr7Pi5Fb7FxRxwjj32WOrXr8+8efMiHUqF2ee5mtmwFObeW/zRZMde5J6ukrL/nQVNxZV1i8u+2gSuAuYDX3nT9XC3uvQDPglYthHQff9CNab6ys7OZtmyZcyePZslS5bw/vvvRzokUxtkb4B5f3XNt+pzZYf2dNc9D+4Q2djMPlWkYd2G+zcHpG+//ZY+ffqQnp7O+PHj6d+/f6RDMjVZbg4smuzGud2705U1aOV63NqTVWoMuzptTDn17t2bsi5/GFMuPh8smeVuWcn2hnhMrO+ertLxcogOfn+xqZ4siRpjTFX5bYEbLGG9NzhLdBx0uQ563mqjDNVQlkSNMaay7d0F790K3/kNXNLuXDhlTK0bqOVAU54kWkdE6ns/F7zX9SsrUOsfFRAVFUVubq7r1mzXK0wNl5+fX+rQhCaMMv+AV4e4IfsADukC/R+EZh3LXs/UCOVJok97L39vVUIs1V5ycjLr169n06ZNpKenFxvhxpiaZOvWrezZs6fEQBEmzH5bAK8Pg5wtbpzbvg/AiTdap6FaZF9J1B6y7SclJYWcnBy2bNnCtm3biI2NJTr6wBsq2NRs+fn5hQm0YIxiE2aq8MUUmHsfaD4kpsFfpsPhJR/PZ2q2MpOoql5eVYHUBFFRUTRt2pTU1FS2b99OXl4ePp8v0mEZE5LY2NjCBGqXJSpBbg68OxJ+8J532+QYuOBlu/ZZS1nHogqoU6dO4cOkjTGm0LYMmDUENvzgpo85H86YBHFJEQ3LVB5LosYYEw6rPoE3roBd20Ci3YhDXYfb9c9azpKoMcbsD1X4/O/w0Vg3bF9SAzhvJrTsGenITBWwJGqMMRWVuxPeuQF+mu2mD+oAF/wTUsv3HFlT81kSNcaYitj6q7v+ufEnN33sRXD6ExCbWPZ6plaxJGqMMaH65SN48wrYnQVRMdD/Ieh8tV3/PABZEjXGmPJShf/+zQ0ej0KdhnD+i9DixEhHZiKk1CQqIhV6uLaqLqh4OMYYU03tyYa3r4dl/3LTTU+A81+yB2Yf4Mqqic4HKvLcJxvCxxhTu2xZBbMuhk0/u+njhsCgxyE2IbJxmYgrK4kGjlYkwI1Aa+BlYKlX1ha4CFgB/KMSYjTGmMhZ8QG8eTXsyYKoWBj4CHS8wq5/GqCMJKqqxcbNFZGRQEPgSFVdGzBvPLAIsNGsjTG1g88Hn02EeQ8CCsmNXfNt8y6RjsxUI6E8huRG4JnABAqgqmuAZ7xljDGmZtu93T2+bN5fAYVmneGaTy2BmhJC6Z17CJBTxvyd3jLGGFNzbVrhrn9u+cVNn3C5a8KNiY9sXKZaCqUm+hswRERKXEn3yi4DMsIUlzHGVL2f34PnTnYJNDoOzvg7nPGkJVBTqlBqoo8DzwJfi8hkYDmu924b4Abv/bqwR2iMMZXN54NPH4ZPH3HTdQ9yw/c16xjZuEy1V+4kqqrTRCQZmABMoej2FwF2Abep6nPhD9EYYyrRrkyYfS2seN9NN+/mBpCv2ziycZkaIaQRi1T1SRGZDvQHWuIS6CrgQ1XNrIT4jDGm8mz82V3/3LrKTXe6Gvo/CDFxkY3L1BghD/unqlnAa5UQizHGVJ2l/4K3h0PuDoiOd4PHH3dJpKMyNUzISVREWgKnAI2Bl1U1Q0TigCbAelXNDXOMxhgTPr58d+vKZ4+76XpN3fXPpsdHNi5TI4WUREXkEeAW3NB+ihtgIQNIwI1gdC/wZHhDNMaYMNm1Dd68ClZ+5KZb9IDzZkByw4iGZWquct/iIiLXArcBk4FTcddDAVDV7cC/gDPCHaAxxoTFhp/g2d5FCbTLcLjsbUugZr+EUhO9HpitqqNEpEGQ+UuAEeEJyxhjwujHt+CdG2BvDsQkuPs/j70g0lGZWiCUJNoamFrG/E1A+v6FY4wxYeTLh48fgIWT3HRKc7jgJTi4Q2TjMrVGKEl0N1CnjPktALvNxRhTPeRshTeugF/nuemWJ8FfZkCdYA1pxlRMKMP+fQUMDjbDG/bvUmBhKDsXkWQR+buIrBORXSLyjYicWY71rhKRf4nI7956v3jbsYsbxhhYtwSe7VWUQLuNgCGzLYGasAsliT4GdBORl4D2XlkTEemPe4B3M2BiiPufDVyC69V7Gq6H72wRGbSP9R4AtgN3AQOAvwHn44YkTA0xBmNMbbLkdXj+VMj8A2IS4dznof9fITrkO/qM2SdR1X0vVbCwyDXAJCAO1zu3YOVcYLiqzghhW4OA94BzVHW2VybAZ0ADVW1TxrqNVHVjQFkvXDIfqapPlTeOgG1kpqSkpGRmWqu0MTVOfh58NAYW/cNNp7aAC1+GJsdENi5T46WmppKVlZWlqiUqaaEO+/esiPwLOA84CpdIfwFeU9U/Q4xrMJAFvOO3fRWRmcCzItJWVZeWEsfGIMVfe+/NQozDGFPT7dwMb1wOvy1w04f1gb+8AEn1IxuXqfUqMuzfeqBCNb0A7YClquoLKF/iPz+E7Z3svf+4v4EZY2qQtd+5B2hnrXbT3UfBKfdDVHRk4zIHhHInURHJBy5V1VdKmX8B8IqqlveT2wBYEaR8q9/88sZWH/g7Xq24jOX21U6bUt59GmOqge9nwbs3Qd5uiK0DZ0+Go4P2fzSmUoRSE5X9nB9MWRdky3WxVkSSgLeB+sBJqrqnAnEYY2qS/L0w91748mk3ndYSLnwFGreNbFzmgBPO7mrNgewQlt9C8NpmwUWMrUHmFSMiibjhBo8D+qvqkrKWD3ZROGB7mVht1JjqbcdGeH0Y/O7dUdeqH5z7HCSmRTQsc2AqM4mKyFnAWX5F14hI3yCL1gf6Av8NYd8/AeeKSFTAddGCrnRlXtv07k19B+gGDFLVz0PYtzGmJvrzW3j1Utju9WPsORr63G3XP03E7Ksm2gEY5v2swEneK9AO4HNCGzt3NnAlbtD6d/zKLwOWl9YzF0BE4nFNuD2BM1T10xD2a4ypiRb/E/59C+TvgbhkGPw0tLFnXpjIKjOJquoDuIENEBEfMKS0jkUVMAeYBzzvDWj/GzAU6IFf7VdE5gO9VNX/musbQH9gHLBDRLr6zdukqqvCFKMxJtLy98L7d8LX09x0g1ZwwcvQ6KjIxmUMoV0TbQkEuz+zQrx7Qs8GHvReqbhbWs5R1Xf3sfrp3vv93svfTIpqz8aYmiw/zz3/c+nbbrr1ADjnWUiwrgumeij3iEUi0hJoV1qCE5EzgB9UNSN84VUtG7HImGokPw/euhp+estNdx8Fp4yBqFBGKzVm/4VrxKK/AocApdUSbwVW4waiN8aYisvPg9nXFiXQk26DPveAVOROOmMqTyhf6XoAH5Qxfy6uo48xxlScLx/eHg4/vuGme95qCdRUW6Ek0UbA+jLmbwQa7184xpgDmi8f3r4efvAGHus+Ck6+zxKoqbZCSaKZwOFlzG9FaIMtGGNMEV8+vDMClsxy0yeOhL5jLYGaai2UJPoZcLWINAmc4ZVdRWiDLRhjjOPzwb9GwvfeHXTdRkC/cZZATbUXaseiM4DFIvI48B1uAIbjcJ2KknG3qhhjTPn5fPDuSPjun2666/Vw6gRLoKZGKHcSVdXvROQvwHTgUYoGiBdgM3Ceqn4T/hCNMbWWzwf/HgWLX3LTXa6D/g9aAjU1RqgP5f63iDQHBuCugQqwHJirqrsqIT5jTG3l88F7t8D/ZrrpztfAgIctgZoapSIP5d6FG/fWGGMqRhXmjIZvp7vpTlfBwEctgZoaJ+Qk6o1cdArudpaXVTVDROKAJsB6Vc0Nc4zGmNpEFebcBt8876Y7XgEDH7MEamqkkMbPEpFHgBXAs7jB3w/zZiXgxr29PqzRGWNqF1VvMPnn3PQJw2DQ4zaUn6mxyv3JFZFrgduAycCpuOuhAKjqdtzDse25RMaY4FTh/bvgy6fd9PGXwWlPWAI1NVoon97rgdmqOgpYHGT+EuDIsERljKldVGHuvfDlVDfdYQicPskSqKnxQvkEtwY+LGP+JiB9/8IxxtQ6qvDhfbDoH266wyVw5lOWQE2tEMqneDdQp4z5LXBDAxpjjKMKH42Bz59y0+0vtARqapVQPslfAYODzRCRBNwj0BaGIyhjTC2gCh+Pg4WT3PQx58PZUyAqOrJxGRNGoSTRx4BuIvIS0N4rayIi/YH5QDNgYnjDM8bUSKrwyQT479/c9DHnweCnLYGaWieUYf8+EpHhwCTgYq/YG6uLXOBqVV0U5viMMTXR/IfgM+87dbtz4WxLoKZ2CnXYv2dF5F/AecBRuNtcfgFeU9U/KyE+Y0xNM/9h+PQR9/PRg2HwsxAd8rguxtQIFRn2bz3wVCXEYoypyfL2wEcPwBeT3XTbs+Cc5yyBmlqtQp9uETmSotGKflXV5eELyRhT42z4Cd66Bjb86KbbnAHnPg/RsZGNy5hKFlISFZGTcbXQowLKfwZGqurHYYzNGFPd+Xyu5vnxOMjPBYmCHrdA7zstgZoDQrmTqJdA3wf2AM/hxsoVoC1wEfAfERmgqp9URqDGmGomczW8PRwyPnPTaYe665/Nu0Q0LGOqUig10QeBDUDXwE5EIjIe+AL4K9AtfOEZY6odVVjymnuU2Z7truz4oe5h2vHJkY3NmCoWShJtDzwYrBeuqq4RkWeAu8IWmTGm+snZCv++GZa+7abrNHQjEB05MLJxGRMhoSTRLCC7jPnbsWH/jKm9Vn4Eb98AO9a76SMHwRl/h+SGkY3LmAgKJYm+DlwkIpNVNc9/hojE4q6Lvh7O4Iwx1UBuDnx4f9EzQOOSYcDDcNwQe5C2OeCFkkSfBk4EFojIE8DPgOI6Ft0MRANPi0hz/5VU9Y8wxWqMqWp/futuXdmy0k0f0tUN31e/ZWTjMqaaCCWJ/ohLmgLMCpgnfssEsrG+jKlp8vPcuLefPgK+PIiKgT53Q/dRNnyfMX5CSaLjcEnUGFObbVnlap9/fuOmGx4F5zwLBx0b2biMqYZCGYB+bCXGYYyJNFX4djp8cA/szXFlXa+HU+6H2MTIxmZMNRXWQS1FJEVVs8K5TWNMFcjeAP8aAb/MddP1mrpnfx7WO5JRGVPtlft5oiLysYg0KWN+d+C7sERljKk6y96FKV2LEugx58HwhZZAjSmHUB7KfSLwvYgM8i8U5z5gXojbM8ZE0u7t7r7PV4fArq2QkOIGjT93GiSmRTo6Y2qEUJpzu+B65b4rIpOAO4BGwD+BXsDbwJVhj9AYE17Z6+HbGfDNC7Bjgys7rDecNQVSmkYwMGNqnlA6Fi0RkROAfwCjgFOAg4E6wAhVnVI5IRpj9psq/L4QvnoOfv63u20FICYB+j4Ana+BKGtIMiZUIXUsUtVdInId0Brojrvl5UZLoMZUU3uy4ftZ8PXzsGlZUXlSAzj+Muh4BaQ2L319Y0yZQn2e6OG4Jt3jgZeBHsAkEWkATFBVu4/UmOpg4zL4eppLoLk7isqbdYJOV0PbsyA2IXLxGVNLhPI80UuAKYAPuFBVXxeRFOB54AHgZBG5WFXXVU6oxpgy5e91TbVfP1/0jE+AmEQ45i/Q6So4uEPk4jOmFgqlJvoS8CVwkapmAHj3hP7Fa+J9HPge19nIGFNVtq9zHYW+nVH0hBWA+oe5xNnhYutta0wlCSWJPgbcE/gEFwBVfVpE/gv8X9giM8aUThUy/uuabP07CkkUtB4Ana6Ew062zkLGVLJQeufesY/5P4pIp/0PqYbb+hv8tgAatYWGR0JCvUhHZGqT3dthyasueW76uajcOgoZExFhG/ZPRJKAJsCv4dpmTfTndx/QdEHR9w1fvWZENW4Ljdq4xNqoDaS3trFITWhK7SjU2TXZHn02xMRHLj5jDlBlJlERyQUuU9VZ3nRdXK/ce1T1h4DFBwMvcoA/+mz1pu1EaX0Okq0ARG1fA9vXFA2pBqhEoWktifJPrI3aQoPDITo2UqGb6qago9BX0+D3/xaVxyRC+/Og45XWUciYCNtXTTSG4kP5xQGnA09WWkQ1XObRl/J0Tm/+XLeeetkrOTJqDa1lNUfKGo6M+oP6sgNRH7J1FWxd5f5IejQqFtKPQBq1KV5zTT3Urm0dSLavhW9nWkchY2qAsD7FxcCAdgcxoN1BAGTt2suKDdn8vD6bOeuzeWLddjZuWM1Bub9zpKx2yTVqDa1lDXVlF+LbCxuXupcfjUlCGh0JDQOSa72DQSRYGKam2Z0Fa76B/82EZf8GzXflhR2FroLD+tiXKWOqGUuilSglMZZOh9an06H1C8tUlQ3b9/Dz+u0sX5/NyxuyWb5uOzs2/U5L3x9+iXU1R8ifJMheJC8H1i52Lz8aXw9p1BYaHVW8WbhOelUfqimvvFzY8gtsWAobf/Lel0LW6uLLJaV7HYUut45CxlRjlkSrmIjQJCWBJikJ9D6y6JbavHwfv2/NYfl6V3P9cH02v6zPxLftN1qzhiNkDUdGraa1rOEwWUes5CN7tsPqL9zLj9Zp6JqEi9Vcj3JP6TBVQ9UlxsBkuXlF0e0owTTrDJ29EYWso5Ax1Z4l0WoiJjqKwxsmc3jDZAYdc1Bh+a7cfFZu3MHP67fz/fpsXtuQzap1W0ne6ZqEj4haw5Hiaq4tZCNRosjOTfDbJnerjR+t18y73upXc00/EuKSqvpwa5dd24qS5IafvCb5ZbBne+nrxKdA47bu99C4LTQ62v0+ElOrLm5jzH4rTxId5Pcw7iTcoPPniUhgt8ATwhqZASAxLppjmqVwTLPitchtO3NZviGb5euzmb8+m2c3ZPP7+s002fO7S6perbV11BqayhYApKCn8MoPC7ejCFK/pXdf61F+PYVbQUxclR5rxPl8kLcLcne620hyd0Jujt/PO2Gv975zk0uUG5ZC9trStxkV6+4X9k+WjdtCvaZ2PduYWkDKGjNeRHwhbk9Vtcbe4iIimSkpKSmZmZmRDqVCVJU/M3cVdmZa7r02bNpIS9/qwubg1l6SbSil15Q0KgZJa+l6gSakeK96RT/HF/ycWrI8NjH8CcKXD3t3Qd7ufbzvcYlw7+6i9705JZNgsNfenfsXY2rzoiTZqC00Ptp9GbHbloyp0VJTU8nKyspS1RJNRftKor1C3ZmqfhrqOtVFTU+ipdmb7+O3zTsLk+rP67NZsSGbnVvX0drrHXykrC78uZ7k7N8Oo2LLSLopLiEWJrrdpSTE3cWTYVnXEStbVAzE1YG4ZO+9jjuewhqm1xQbXzdyMRpjKk2Fk+iBprYm0dLs3JPHig3ZxWuu67YTm7OeI6PW0ELWU48c6koO9cihnuykLruoJzuLlcdLBBNcgZgE94pNLP5e8HN8skuCsUkBCTGpeHKMqwOxfj/HJR94zdrGmGLKSqIR7VgkIsnAg8B5QCrwEzBOVf9VjnUPxz05pg9uQIjPgNGqurTMFU2hOvExHNc8jeOaF79xf/OOPSxfn82vm3eyOXsPv+/Yw+bsPWzasYfNO/awKXsPu/cWtfTHk1tmkvVPwvWjdxMdHYPGxENMAhKbSFRcIlFxScTEJRGbkER8YhLxiXVISKxDUlIdkuokExOX5J5/GZNY9B4TX5Qs7fqiMSYCIt07dzbuAd+3A78Bw4DZInKGqs4pbSURaYRLmhuBoUAecC/wqYgcp6prKjvw2iw9OZ70VvF0bxX8flNVZWduPpuyi5Jq8fdcVvsl3tw8v0vreysSUS7J8T5SEveQkphDalIsqUmxpCTGee+xpCYWlaV4P6cmxZIYG41YgjXGVJKINeeKyCDgPeAcVZ3tlQkuOTZQ1TZlrPsocCNwuKqu9coa4BLxy6o6vIIxHVDNuVVBVcnek+cSrJdUM3P2krVrL5k5ud77XjJ37SUrZy+Zu1yZf013f8RFR5GcEENibDTxsVEkxka7V1w0CbHuleiVJ8S5eQkFy3hlCTFRJPrP89aNjRbioqOIjY4iJlqIjYoiKsoStjG1TXVtzh0MZAHvFBSoqorITOBZEWlbRtPsYODDggTqrbtFRN4FzgEqlERN+IkI9RJiqZcQy+ENk8u93u69+YUJtiDhBibaonlFy2zfXfz6bG6+j607c8N9WKWKiRJio6Ncgo3xS7DRUYUJN7ZgOiaqaPkYN7+2VJqjRIgS9y7ezyIF5eL3c/FlogqXlaL5UW55QQrPj0CJMnDrFc3zpv3Wodh88dYpvj3vX8ntBaxTcn8lt0dAzFEiREUVPz7B/5jd8ZY4B8XOWcDyIrXmc1MZokQ4pH7l3QsfySTaDliqqoFVjiX+8wNXEpFE4HDg9SDbXAJcLCKNVHVjkHX3VcW0IX2qiYJaYuN6CSGtl+9Ttu/yara79rItJ5ecPfns2uteu3O9971+74VlvqBlu/fmk5Obh68cjTZ5PiXPl8+uCjVbG2PCLS0plsX3n1pp249kEm0ArAhSvtVvfjBpuC91W4PM81+3RBI1tV90lJBWJ460OuHtUauq7M1Xdu3NZ4+XgPfmK3vzfYWvtgznmgAAD5lJREFU3Dw3necr+rlwXr6yN8/N25uv5Ob5/Oart76P2tBXXhUURRV8qvi8d1XF53PzfOrOqc9vGTftlnHLF8xT8tXbMG60F/99FFyRcuVaIgb/eepNaGGsWur2gs0v2nbBclo4XWL5gv357dP/uPyPrzxf0Ez1FOmORWV9dPb1sQp53WDt2f68mqrVRk0JIkJcjGuiJdEGTzDhpxrsi0fxRFv8i4ebn28ZuExRldzWHckkuoXgtc2CR54Eq2kCbMMlyYqsa4wx1VLB9dso7AJnTRLJhxP+BLQRkcAYjvHefwy2kqruAn7FXTMNdAywKdj1UGOMMSbcIplEZ+MGWDgjoPwyYPk+Bk2YDfTzGxgfEanvbeutcAdqjDHGBBPJJDoHmAc8LyJXiEgfEZkB9ABuK1hIROaLSGCj/0Tc7TFzROQsETkNd89pHm4EJGOMMabSReyaqHdP6Nm4pPcgrla6FDf4wrv7WHeDiPTEJdOXKBr27yRV/WM/wqqXlZVFaqo909EYY4yTlZUFUC/YPBuA3o+I5OEScsG9qzv8ZicHlPlPB84r6OGbFcbwAvcRjnXKml/avGDldm5KL7dzU3p5WecmcLqmnJuylrFzU3PPTT3Ap6olKp6WRIMQkfkAqtq7tDL/6SDzMr3psFVpg8W0v+uUNb+0eXZu7NxUxbkJnK4p56asZezc1I5zEyiS10SNMcaYGs2SqDHGGFNBlkSNMcaYCrIkaowxxlSQdSyqBFVxMbumsnNTOjs3pbNzUzo7N6WzjkXGGGNMNWZJ1BhjjKkga841xhhjKshqosYYY0wFWRI1xhhjKsiSqDH/396ZR19RlnH88w1QUkJFLbRUXNJyK1MzUxM8oqFW0GJ4SP1lRqtLpWIqBZZ7mvsu4ZalFbimYoqJJi5UoAiu4EHwsIggCKXy9Mf7XrsM9/5+c8c7d4afz+ecOXPve9953me+M3eeebcZx3GcjHgQdRzHcZyMeBB1HMdxnIx4EC0ISSMlTZW0QtLgov0pC5LWlPQ7SbMkLZL0gKTtivarLEi6UtIcSYslTZF0UNE+lQ1Ju8f/1YlF+1IWJI2XtFzSkrj8tWifyoICwyXNjv+r8Y1s70G0OJ4DjgEeK9qRktEVeBH4HNALuB0YW6hH5eI8oI+Z9QSOAG6Q5E+qiUj6AHA+/r+qRZuZ9YjLgKKdKRFHAV8AdgHWBX7ayMYeRAvCzG4ws3HA8qJ9KRNmttTMfmVms8zsHeBiYEtJ6xftWxkws2lm9p/49W1gTeCjBbpUNr4HPAxMK9oRp/xI6gL8HBhqZrPNbIWZTWrEhgfRFEj6mKQLJE2ITSEmqW+dvD0kXRib3JZJekLSl1vscstogTafB+aa2YKmO58zeWkj6VJJy4BJwN+AqfntRT7koU280ToGGJmz+7mS43/qIknzJI2TtGN+e5AfOWizCdAdOFTS3NjF9o1GfPIgmo6tgEOAJYSLVnuMAYYApwAHEi5wYyQdkKuHxZGbNrGZ8grgpKZ521py0cbMfgj0APoD99rq+dixPLQ5AzjPzBY12ddWk4c2JwCbA5sC44C7JPVoptMtotnabExowu1N0GYoMErSVqk9MjNfOliAD1R9HggY0LdGvgPib4Oq0gRMAJ6pY3s8MLjofSybNoS7w/HAuUXvY9m0SWx7B7B/0ftatDbAzsATFbvAaODEovezDNrUKWMasE/R+1q0NsBOMd8mVWm3AUem9clroikwsxUpsw4CFgG3Vm1rwLXAJyRtm4N7hZKHNrGf4g/AK8BxzfO2tbTovOkCbJnZyYLIQZu9gG2BuZLmA4OB4ZJGNc/r1tCi8yZtGaUiB22eA956Lz55EG0u2wNTaxzoyVW/AyCpm6TuhGPQTVL3OLKws5JaG+Aq4IOE0YSrY1Nlo6TSJvbxfCuuu8a+m36Eu+vOStrz5mpga+DTcbkNuIDV+CYsBWnPm3Ul9VeYPraGpJ8A6wGPt9DXVpNKGzNbQgi0J8Rr8m7A3sADaQvqzBftIlgfeK1G+mtVv1e4ClhGuIO+Ln7+Qq7eFUsqbSRtBnyboMVC/X9e216tcbMQ0p43RpjWMgtYAJwIHGJmk2ts21lIpY2ZLbEwonuWmc0C3gQWm1mtbTsLac+bboT+4gXAq8BBwAAzeyN3D4ujkWvxDwj9xQuB64EjzOyFtAV1zeqhU5f2ak7v/mZmbUBb3s6UjA61MbOZhL6L9xtptFkK7NMad0pFqv/USonh//V+IM15M48wB/L9Rtpr8XzCjUUmvCbaXBaw8h1OhV5x3ZnvijvCtamPa1Mf16Y+rk19WqaNB9Hm8jTwyRp9mzvE9VMt9qdMuDb1cW3q49rUx7WpT8u08SDaXMYQ5hx9KZF+GDDdzFa7SfFNxLWpj2tTH9emPq5NfVqmjfeJpkTS1+PHXeN6b0kbAEvNrPIw57sIo7quiU9PeQk4HNgT+Eor/W0lrk19XJv6uDb1cW3qUzptip48u7oshI7oWsuMRL6ehOe9vkp4Lu4kYGDR/rs2rk3ZFtfGtekM2igW5jiO4zhOg3ifqOM4juNkxIOo4ziO42TEg6jjOI7jZMSDqOM4juNkxIOo4ziO42TEg6jjOI7jZMSDqOM4juNkxIOo4zSIJJM0umg/siBpLUkXSnpZ0juSZmS0M1qSdZTWGZDUJx7zEUX74pQPD6JOKZDUN16oTNKRdfKYpDta7VsnYxhwFPBHwqv4ji3UG8dZzfFn5zplZKSkG81sWdGOdEL6A1PM7PgcbH8X+H4OdotmJvBB4O2iHXHKh9dEnbLxBLAxXkMCQFIXSWs10WRvcnrPpJm9ZWbL87BdJBZYbmYeRJ1V8CDqlI2bgSeBYfHtC+1Sr39SUlv8rW9V2oiYtq2k8yXNkbRU0t8kbRPzfFXSJEnLJM2QNLSdsveV9KikNyW9KukCSWvXyLeOpLMkPS/pP5LmSbpJ0hZ1fN5X0nBJLxAenH1wBxp0lTRM0lRJyyUtkDRG0g5Vedpif+XmhLdeVJrOR3Rgu7ukcyTNjpo8Jmm/Onnr9pNKWj9+ni/pDUljJfWOeYZKeib6Pk1SzbdsSPqmpAlx+zclTax6o0d1Potl7S7pwXiM50u6WlKPRN5NJI2SNDMem7mSHpF0eFWemn2iaXRPbi/pIEmPx/xzorZdE/m3k3SLpFeiT69KekDSgTUPklMo3pzrlA0j9NvdB5wM/DSHMq4FlgCnAxsCPwPukTQcOBu4DBgFfAe4QtJUM5uQsPEZ4OvAVcB1QD/gaGB7Sf3NbAWEAAo8AmwabT4NbAT8EJgoaRczm5mw/RugW7S9GJjewf7cSAi046LvvYEfAf+QtJeZ/RP4O3Ao8FtgPnBa3HZyB7ZvAgYCtwP3AFsCfyG8WqoR7gZmAb8AtiJoNUbSX4ChwDWEG4ajgT9J2trM3i1D0q8J58PdwHBgBTAIuEXSj83skkR5nwbuAH4H/B7oSzieK2J5xOA1DvgocCnwLLAOsCOwF+E8aY80uldzAOG4X044F74CHAcsJJyLKNw43h/zX05oSt4A2AXYDbizA5+cVlP0a2188cXMIFzkDDgufr+XcFHdrCqPAXcktjNgdA17bfG3vlVpI2La7RDeYBTTj47pbwCbVqVvGH24qUaZRuK1SsAFMX1wIm0Z8KlE3s0IAXJ0DZ+nA2ul1K1/3OaPiX3akdCH91Ai/wxgfErb+9XSlxBULVw+VkofXS8NuCSRfl5MfxnomfDbgDOq0j4T006v4ePYqOOHEsdnBfC5RN47gbeAHomyTuhAhz4x34gsuldtvxToU5Uu4ClgTlXal2Peg4v6L/rS2OLNuU5ZGQasAfwqB9sXWrxiRR6K61vN7OVKopnNIwS0j9ewMd3MxibSzozrQQCSBAwh1AJfkbRBZSFcUB8lBKokl5nZmyn3ZVBcn1a9T2Y2mVAT21PShiltJRkY1+dUJ8b97qh2nOT8xPeK5teZ2eIq25MJQbFa8yGEwHJttYZRx9uADwG7J+z/w8weTaTdT2h96xO/L4rrfpI+3OD+ZNF9rJnNqMprhBdH965qZq74NEBSzwZ9cgrAg6hTSiw0hd0EDJG0Y5PNv5j4vjCuazVRLgRq9c0+k0wwsznA60Clr3PDuO1+wLwaS3/gIzVsP9u++yuxOaHWtYo/hFpOJU8Wtoi2a/lTq7z2eC+af5JQa5vGqhpeE/MkdUyWB7AgrtcHsNCMfhrh+MyR9KSksyXt2uHeZNM9jU8PEroH2oD5kh6WNFLStil8cgrA+0SdMnMKod/xLGBAg9u2d26/02C6aqTVe6iAany+j7APaUlbC02W12zas91QuWb2XjQXQe8B7eR/OqXdlWyb2SmSRgEHEvpBjwSOl3S2mQ1LY6MB0vp0uKRzCH2oexL67E+WdKyZXZyhXCdHPIg6pcXMXpJ0GXCMpH51sr0G9KqRvkWNtGaySs1A0kaEgSmVGsc8Qs20p5ndl5MfLwD7E2pryUFCFR8bHQRUbXs/YGtWDVKfyGgzC88BXwReNrNGa8AdYmYvAhcBF0nqThhAdYKkc81sbp3N8tQdM3uKUKM9W9K6wETgTEmXJLoinILx5lyn7Pya0EdWryb3LLC7quZSSloP+HbOfm0jaWAirVJzGQtgYYTujcBna03FAMjQF5ek0i/789gHW7G7PWGQyoTYt5uFW+N6pQczxP3eJqPNLFwf16dL6pL8MauGClOPulWnWZjnWgnU67WzeS66S+olaaXrspm9TgjIawHdG7Xp5IvXRJ1SY2bzY9NWvQFGFwM3APdLuh5Yl/DknJmEKQd5MQW4QdJVhJpSP0LT84OEEZsVTgb2AG6WdDNhMNF/CaNzDyDMiW3L6oSZjYt2BwPrKTwWsTLVojJlJKvteyTdDhwuqRdhesmWwPcItaTts9pu0I/HJf0SGAn8S9ItwGzCVKGdCTqukcF0P+BKSX8mDJRaEu0dCUw0s7qDp3LU/TDgJ5LGAM8TRhPvTaj13mz+FK/S4UHUWR04jzC/bqPkD2Z2o6SNgR/HfC8CpxIGfeyWo0+TCHNYTyM86m4xIaCfFGugFf8WSdqD0K91MGFu4NuEOZMTgKub4MuQ6E8bcC5h5O+DwHAzm/IebX+T0BowhDAQ6inga8AhtCiIApjZqZKeJASnY4G1gbnRn2Mymv03Yc5rX8L+dSFMuTmdoGNH5KH7eGAn4CDC+f4OoRZ6HOH8ckqGvHndcRzHcbLhfaKO4ziOkxEPoo7jOI6TEQ+ijuM4jpMRD6KO4ziOkxEPoo7jOI6TEQ+ijuM4jpMRD6KO4ziOkxEPoo7jOI6TEQ+ijuM4jpOR/wFCakx2eP75kgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "f, ax = plt.subplots(1,1,figsize=(7,3))\n", "ax.plot(dim_range, optimal_accuracies, label='Bayes Optimal')\n", "ax.plot(dim_range, naive_accuracies, label='Naive Bayes')\n", "ax.set_xscale('log')\n", "ax.set_xlabel('Number of dimensions')\n", "ax.set_ylabel('Expected Error')\n", "ax.set_title('A simulation of Trunk\\'s Example')\n", "ax.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We clearly see that the Naive Bayes classifier starts to approach chance probability. In summary, when learning a classifier from a finite amount of data, we do not want to arbitrarily utilize more features even if they are informative. Rather, as a designer of a practical implementation we wish to use those features which are most relevant and least obscured by noise given our sample size." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" } }, "nbformat": 4, "nbformat_minor": 2 }