"
]
}
],
"prompt_number": 31
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above plot will support any of the stopping criteria defined above. Assuming we haven't predefined $k$, we might choose $k=7$, as that is where the error really seems to plateau. Additionally, we could be more conservative. The red line shows the min value of (mean+1 std error). Any value of $k$ that is less than this value is statistically the same as the $k$ with the lowest mean error. If we were using that rule, we'd potentially stop at $k=3$.
\n",
"
\n",
"\n",
"##SVD Based Dimensionality Reduction\n",
"\n",
"Another way to bring the learning down to $k$ features is to find a projection matrix that produces a rank-$k$ approximation of our training data $X$. The best way to get a low-rank approximation, from a signal preservation perspective, is by the use of the Singular Value Decomposition. A review of the SVD can be found here:
\n",
"http://nbviewer.ipython.org/github/briandalessandro/DataScienceCourse/blob/master/ipython/Lecture3_PhotoSVD.ipynb\n",
"
\n",
"We'll put it to work here by doing the following:\n",
"
\n",
" - Decomposing our training data
\n",
" - For each rank-k approximation of X, get cross-validated error
\n",
" - Compare this error to the Stepwise Forward Feature Selection above
\n",
"The scale of the features influences the spectrum of the singular values, so we'll normalize the feature so they all have the same scale.\n",
" "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import numpy as np\n",
"import math\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import course_utils as bd\n",
"from sklearn import linear_model\n",
"from sklearn.preprocessing import scale\n",
"reload(bd)\n",
"\n",
"\n",
"f = '/Users/briand/Desktop/ds course/hw/Cell2Cell_data.csv'\n",
"\n",
"X_train = train.drop(lab, 1)\n",
"X_test = test.drop(lab, 1)\n",
"Y_train = train[lab]\n",
"Y_test = test[lab]\n",
"\n",
"X_train_norm = pd.DataFrame(scale(X_train, axis=0, with_mean=True, with_std=True, copy=True), columns = X_train.columns.values)\n"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 32
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's decompose the training data.
"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"U, sig, Vt = np.linalg.svd(X_train_norm)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 33
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Out of curiosity, let's plot the spectrum to get a sense of how independent the features are."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"plt.plot(np.cumsum(sig**2)/np.sum(sig**2))"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 34,
"text": [
"[]"
]
},
{
"metadata": {},
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEACAYAAABMEua6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtY1GX+//EnBG2lbQctSoYWFYLBA2J42rJGyyhbyVOF\nbZspFWvrZu2ptn67YduqVFvZ0rZ0NDugHUVbm3ZRpywlUjytuoluFE7lZqmlmMj4+f1xf8MIHUBn\n+Mzh9biuuWTknvm8mwtf3dyf+xBjWZaFiIhEhFi7CxARkcBRqIuIRBCFuohIBFGoi4hEEIW6iEgE\nUaiLiESQFkN94sSJJCQk0KtXr8O2ufnmm0lNTSUzM5NVq1YFtEAREWm9FkN9woQJuN3uw35/4cKF\nbN68merqah577DEmTZoU0AJFRKT1Wgz1wYMHc8oppxz2+/Pnz2f8+PEADBgwgJ07d7Jt27bAVSgi\nIq121GPqXq+XpKSkxucOh4OtW7ce7duKiMgRCMiN0u/vNBATExOItxURkTaKO9o3SExMpLa2tvH5\n1q1bSUxMbNYuJSWFLVu2HO3lRESiSvfu3dm8eXOr2x91Tz03N5fZs2cDUFFRwcknn0xCQkKzdlu2\nbMGyLD0si7vuusv2GkLloc9Cn0Wkfxb19RZr1ljMmmVx880Wgwdb/PCHFklJFrm5FnfdZTFvnsVH\nH1kcOND89W3tDLfYUx83bhxvvfUW27dvJykpialTp7J//34ACgoKGD58OAsXLiQlJYUOHTrw9NNP\nt6kAEZFIsWcPrF0Lq1YdfGzYAD/6EWRlmceIEebPTp2CU0OLoV5aWtrimxQXFwekGBGRcPHFF03D\ne9Uq+OgjyMg4GOATJ0Lv3tChQ/vVddRj6tJ2LpfL7hJChj6Lg/RZHBRKn4VlwdatTcO7qgp27YI+\nfUx45+TA7beD0wnx8fbWG2NZVrsckhETE0M7XUpE5Ig1NMDbb8Obb5rwXrUK4uIO9r779jV/du0K\nse2w0Upbs1OhLiJRr6EBPB54+WV47TVISoLcXMjONgF+5pn21dbW7NTwi4hEpYYGWLIEXnoJ5s2D\n5GS44gqoqDC98HClUBeRqLF/PyxebHrk8+ZBt24myCsrTahHAoW6iES0/fth0SLTIy8rg5QUE+Qr\nVpiphpFGY+oiEnHq66G83PTIy8ogLc0E+ZgxcNZZdlfXNrpRKiJRqb4e/vUv0yNfsADS0w8G+Xf2\nHAw7CnURiRr79sE//2l65AsWQI8eB4P8EFtQhSWFuohEtG++MUH+0kvw+uvQq9fBIO/Sxe7qAk+h\nLiIR55tvwO02PfJ//AMyM2HsWBg9OjKD/LsU6iISEfbuNUH+0kuwcKFZBHTFFSbIzzjD7uraj0Jd\nRMJWQ4MJ8NJSeOMNsyT/2yA/xI7eUUGhLiJh5/PP4ckn4dFHzXDK+PEmyE8/3e7K7KdtAkQkbKxY\nAcXFZi75qFHw6qtwzjl2VxXe1FMXkXa1bx+8+KIJ8//9DyZNgvz84B0aEe40/CIiIam2Fv7+d3ji\nCbMP+eTJMHw4HHOM3ZWFtrZmZzvsBiwi0cqyzAZao0ebIN+zB5YuNXuVjxihQA8GjamLSMB9/TU8\n+6wZYomNNb3y2bOhY0e7K4t8LfbU3W436enppKamUlRU1Oz7O3bsYNSoUWRmZjJgwADWr18flEJF\nJPT95z/wy1+a3Q+XLIG//Q3WrYOf/1yB3l78hrrP52Py5Mm43W42bNhAaWkpGzdubNJm2rRp9O3b\nlzVr1jB79mymTJkS1IJFJLT4fGb2yrBh4HLBySfD2rVm0ZDLBTExdlcYXfyGemVlJSkpKSQnJxMf\nH09eXh5lZWVN2mzcuJEhQ4YAkJaWRk1NDZ9//nnwKhaRkLB9O8yYYQ6aKCqC666Djz6CP/0JHA67\nq4tefkPd6/WS9J09Kx0OB16vt0mbzMxMXn31VcD8T+Cjjz5i69atQShVRELBihUmwFNTYdMmM7d8\n2TL46U/hBz+wuzrxe6M0phW/N91+++1MmTKFrKwsevXqRVZWFscc5pZ2YWFh49culwuXy9WmYkXE\nHoeaW/6Xv2hueTB4PB48Hs8Rv97vPPWKigoKCwtxu90ATJ8+ndjYWG677bbDvmHXrl1Zt24dHb93\nV0Tz1EXCj+aW2y+g89Szs7Oprq6mpqaG+vp65s6dS25ubpM2u3btor6+HoDHH3+cCy64oFmgi0j4\n0Nzy8OZ3+CUuLo7i4mJycnLw+Xzk5+fjdDopKSkBoKCggA0bNnDdddcRExNDz549efLJJ9ulcBEJ\nrLo6mDVLc8vDnbYJEIly33wDJSVmBsuAATBlClxwgaYihgrt0igirbJvnxkrnz4dsrPNPuZ9+thd\nlRwthbpIlKmvN8Ms99wDvXvDvHkm1CUyKNRFosT+/WY/lj/9Cc4+20xRHDjQ7qok0BTqIhGuoQFe\neAHuvtvsyfLss3DeeXZXJcGiUBeJUD6f6Y1PnWqOhXviCbMXi0Q2hbpIhDlwAF55BQoL4aSTzBTF\nCy/UbJZooVAXiRCWZW563nUXHHecWcafk6MwjzYKdZEwZ1nw+usmzAGmTYPLLlOYRyuFukiYsiyz\ndP+PfzQLiKZOhZEjFebRTqEuEmYsCxYtMmG+a5cZOx8zxiztF1Goi4QRj8eE+bZtJsyvvFIbbElT\nCnWRMPDOOybMP/7YjJ2PGwdx+tcrh6AfC5EQVlFhQnzTJvjDH+BnP4P4eLurklCmUTiRELRihZnB\ncuWVZrz8gw9g4kQFurRMoS4SQlavhssvN7NYLrsMqqvhxhvh2GPtrkzChUJdJAT8+9+mR37ppTB0\nKGzeDDfdpIOcpe0U6iI2+s9/IC8PLroIBg2CLVvMIRXHHWd3ZRKuFOoiNtiyBa69FgYPhsxM0zP/\nzW/ghBPsrkzCnUJdpB199BFcf705Nq57dxPmv/+9zgGVwGkx1N1uN+np6aSmplJUVNTs+9u3b+eS\nSy6hT58+9OzZk1mzZgWjTpGw5vWaMfK+fSEhwUxRvOsus4uiSCD5PXja5/ORlpZGeXk5iYmJ9OvX\nj9LSUpxOZ2ObwsJC9u3bx/Tp09m+fTtpaWls27aNuO+tjNDB0xKNPvsMZsyA2bMhPx9+9zs47TS7\nq5Jw0tbs9NtTr6ysJCUlheTkZOLj48nLy6OsrKxJmzPPPJOvvvoKgK+++opOnTo1C3SRaLN9uwnw\njAyzV8uGDXDffQp0CT6/oe71eklKSmp87nA48Hq9TdrccMMNrF+/ni5dupCZmcnMmTODU6lIGNix\nA/7f/4O0NPj6a1i7FmbOhDPOsLsyiRZ+u9QxrdjDc9q0afTp0wePx8OWLVsYNmwYa9as4cQTT2zW\ntrCwsPFrl8uFS2drSYT46isT3jNnmsVDK1dCcrLdVUk48ng8eDyeI36931BPTEyktra28XltbS0O\nh6NJm2XLlnHnnXcC0L17d7p27coHH3xAdnZ2s/f7bqiLRII9e8xxcd+eMrR8OaSm2l2VhLPvd3in\nTp3aptf7HX7Jzs6murqampoa6uvrmTt3Lrm5uU3apKenU15eDsC2bdv44IMP6NatW5uKEAk3e/fC\ngw9CSgpUVcFbb8GzzyrQxX5+e+pxcXEUFxeTk5ODz+cjPz8fp9NJSUkJAAUFBdxxxx1MmDCBzMxM\nDhw4wL333supp57aLsWLtLd9++CJJ8yRcf37m5OHeve2uyqRg/xOaQzohTSlUcLY/v0waxbccw/0\n7Al33w3nnGN3VRIN2pqdmnso4kdDAzz/vAnxbt1gzhyzR4tIqFKoixzCgQMwd645Mi4hAZ56Ci64\nwO6qRFqmUBf5jgMH4LXXzBL+E0+ERx6BCy+EVszuFQkJCnURzKrP118354DGxsK995q9zRXmEm4U\n6hLVLAv+9S9z/ufevWbs/PLLFeYSvhTqErXWr4dbb4WPP4apU+GKK0wvXSSc6UdYos727fCLX8CQ\nIfCTn8C6dXDVVQp0iQz6MZaosX+/2ZslI8ME+MaNcPPNEB9vd2UigaPhF4kKCxfCr34FP/oRLFkC\nPXrYXZFIcCjUJaJt3GjC/L//hQcegOHDdRNUIpuGXyQiffklTJkC558PF19sxs0vu0yBLpFPoS4R\npaHBbIWbnm7G0DdsMDNcjj3W7spE2oeGXyRi/POfJsDPPBMWLYJeveyuSKT9KdQl7G3aBL/+tRk/\n/8tfIDdXwywSvTT8ImFr505zE/THPzabba1fr9WgIgp1CTsNDfD3v5tx8927TZj/5jfwgx/YXZmI\n/TT8ImFl8WK45RY49VRwu6FPH7srEgktCnUJC5s3w29/C2vWwP33w6hRGmYRORQNv0hI++or+N3v\nYOBAGDDATFEcPVqBLnI4LYa62+0mPT2d1NRUioqKmn3//vvvJysri6ysLHr16kVcXBw7d+4MSrES\nPXw+c8BzWhp88YVZPHT77XDccXZXJhLa/B487fP5SEtLo7y8nMTERPr160dpaSlOp/OQ7V9//XUe\neughysvLm19IB09LK731lhk379gRHnpIBzxLdGtrdvrtqVdWVpKSkkJycjLx8fHk5eVRVlZ22PYv\nvPAC48aNa321It/x4YcwdiyMHw+//z28/bYCXaSt/Ia61+slKSmp8bnD4cDr9R6ybV1dHW+++SZj\nxowJbIUS8b7+2oR4v36QlWUWEV15pcbNRY6E39kvMW34V7VgwQLOO+88Tj755MO2KSwsbPza5XLh\ncrla/f4SeQ4cgGeegTvvhGHDYO1a6NLF7qpE7OXxePB4PEf8er+hnpiYSG1tbePz2tpaHA7HIdvO\nmTOnxaGX74a6RLfNm+Haa83X8+ZB//721iMSKr7f4Z06dWqbXu93+CU7O5vq6mpqamqor69n7ty5\n5ObmNmu3a9cu3n77bS6//PI2XVyij2XBY4/BoEEwbhy8844CXSSQ/PbU4+LiKC4uJicnB5/PR35+\nPk6nk5KSEgAKCgoAmDdvHjk5ORx//PHBr1jC1rZtcP318Mkn5iboYSZRichR8DulMaAX0pTGqDZv\nHkyaBPn58Mc/an9zkdZqa3ZqmwAJqq+/NnPOPR545RWzo6KIBI+2CZCgefdds+FWbCysXq1AF2kP\n6qlLwNXXQ2EhPP00lJSYQytEpH0o1CWgNmyAa64Bh8P0zhMS7K5IJLpo+EUC4sABmDnTnEB0001Q\nVqZAF7GDeupy1LZuheuug7o6WL4cUlLsrkgkeqmnLkdlzhzo2xeGDDFzzxXoIvZST12OyI4d8Itf\nwKpV8MYb2k1RJFSopy5ttmgRZGZC586wcqUCXSSUqKcurbZ3L9xxB7z0Ejz5JOTk2F2RiHyfeurS\nKqtWQXY2eL3m8GcFukhoUqiLXz4fzJgBF19sDrKYOxc6dbK7KhE5HA2/yGF9+KHZ8zwuzoydn3WW\n3RWJSEvUU5dmLMss8e/fH0aONDdGFegi4UE9dWni88+hoAC2bIHFi6FXL7srEpG2UE9dGv3jH2aq\nYkoKVFYq0EXCkXrqwp498JvfmEVEpaVm/xYRCU/qqUe5994ze57X1Zmpigp0kfCmnnqU2r8f7rkH\n/v53eOQRGDvW7opEJBBa7Km73W7S09NJTU2lqKjokG08Hg9ZWVn07NkTl8sV6BolwP77Xzj3XDNu\nvnq1Al0kkvg9eNrn85GWlkZ5eTmJiYn069eP0tJSnN85Bn7nzp2ce+65vPnmmzgcDrZv307nzp2b\nX0gHT4eEefPgxhvhzjvh5pshJsbuikTEn4AePF1ZWUlKSgrJyckA5OXlUVZW1iTUX3jhBcaMGYPD\n4QA4ZKCL/fbvN/u2vPgiLFgAAwbYXZGIBIPf4Rev10tSUlLjc4fDgdfrbdKmurqaL7/8kiFDhpCd\nnc2zzz4bnErliHm9MHQo/PvfUFWlQBeJZH576jGt+N18//79VFVVsWjRIurq6hg0aBADBw4kNTW1\nWdvCwsLGr10ul8bf20F5uVnq/4tfmL1bYjXfSSSkeTwePB7PEb/eb6gnJiZSW1vb+Ly2trZxmOVb\nSUlJdO7cmeOPP57jjz+e888/nzVr1rQY6hJcBw7An/8Mjz4Kzz1neuoiEvq+3+GdOnVqm17vt9+W\nnZ1NdXU1NTU11NfXM3fuXHJzc5u0ufzyy3nnnXfw+XzU1dXx3nvvkZGR0aYiJLC2b4fhw+Ff/4IV\nKxToItHEb089Li6O4uJicnJy8Pl85Ofn43Q6KSkpAaCgoID09HQuueQSevfuTWxsLDfccINC3UbL\nl0Nennn8+c9mh0URiR5+pzQG9EKa0hhUlgUPP2yC/Ikn4Hu/UIlImArolEYJD199BRMnmv3P33sP\nuna1uyIRsYvmQoS5tWvNMXOnnQbvvqtAF4l2CvUw9tRTcOGFUFhoZrkcd5zdFYmI3TT8Eobq6mDy\nZKiogLfeAt2XFpFvqaceZqqrYdAg2LfPbMilQBeR71Koh5GXXza7K06aZBYUdexod0UiEmo0/BIG\n6uvhd7+D+fPN6UTnnGN3RSISqhTqIe7jj+Gqq8zslpUr4ZRT7K5IREKZhl9CmNsN/fvDqFFmH3QF\nuoi0RD31EOTzwdSpZsriiy/C+efbXZGIhAuFeoj53//g6qvNLosrV0JCgt0ViUg40fBLCHnnHXMT\ndOBAs8OiAl1E2ko99RBgWfCXv8D998PTT8Oll9pdkYiEK4W6zXbuhOuug08/NYuJzjrL7opEJJxp\n+MVGVVVmuOWss2DpUgW6iBw9hboNLAseewxycmD6dLMP+rHH2l2ViEQCDb+0sz174Oc/h9WrzY3R\ntDS7KxKRSKKeejvauxcuvhhiYsxhFgp0EQk0HWfXTg4cMOeGHnMMPP88xOp/pyLSCm3Nzhajxe12\nk56eTmpqKkVFRc2+7/F4OOmkk8jKyiIrK4t77rmnbRVHidtvh88+g1mzFOgiEjx+x9R9Ph+TJ0+m\nvLycxMRE+vXrR25uLk6ns0m7Cy64gPnz5we10HD26KNQVgbLlsEPfmB3NSISyfz2GSsrK0lJSSE5\nOZn4+Hjy8vIoKytr1i6ah1Va8o9/wN13my1zO3WyuxoRiXR+Q93r9ZKUlNT43OFw4PV6m7SJiYlh\n2bJlZGZmMnz4cDZs2BCcSsPQypVmYdFrr0G3bnZXIyLRwO/wS0xMTItv0LdvX2praznhhBN44403\nGDlyJJs2bTpk28LCwsavXS4XLperTcWGk48/hssvh5ISs5eLiEhreDwePB7PEb/e7+yXiooKCgsL\ncbvdAEyfPp3Y2Fhuu+22w75h165dWblyJaeeemrTC0XR7JedO+G88yA/H2691e5qRCScBXT2S3Z2\nNtXV1dTU1FBfX8/cuXPJzc1t0mbbtm2NF6ysrMSyrGaBHk3q62HMGBg6FG65xe5qRCTa+B1+iYuL\no7i4mJycHHw+H/n5+TidTkpKSgAoKCjg5Zdf5tFHHyUuLo4TTjiBOXPmtEvhociy4MYbzYHQDz5o\nFhmJiLQnLT4KoLvvhgULwOOBDh3srkZEIkFbs1N7vwTI7NlmL/TlyxXoImIf9dQDYPFiGDfO9NC/\nty5LROSoBHybAPFv/XoT6HPmKNBFxH4K9aPw2Wdw2WXmKLohQ+yuRkREoX7Edu+Gn/zEzEW/5hq7\nqxERMTSmfgR8Phg5Ek47DZ58UlMXRSR4NKYeZJYFU6bAN9+YLQAU6CISSjSlsY0efBDeesscRRcf\nb3c1IiJNKdTb4JVX4IEHzFz0k06yuxoRkeYU6q20fLk5MPqf/4Tv7EYsIhJSNKbeCps3w+jR8Mwz\nkJVldzUiIoenUG/BF1/A8OFQWGj+FBEJZZrS6Mc338BFF8G558IhztwWEQm6tmanQv0wDhyAq682\nUxhLSyFWv9OIiA20S2OA3HEHbN0K5eUKdBEJHwr1QygpgVdfhWXL4Ljj7K5GRKT1NPzyPQsXmv1c\nli6FlBS7qxGRaKfhl6OwahWMHw/z5yvQRSQ8abT4/9TWwogR8OijMGiQ3dWIiByZFkPd7XaTnp5O\namoqRX7m9b3//vvExcXx6quvBrTA9rBrl5mDfuutMHas3dWIiBw5v6Hu8/mYPHkybrebDRs2UFpa\nysaNGw/Z7rbbbuOSSy4Ji3Hz79q/3wT5+efDr35ldzUiIkfHb6hXVlaSkpJCcnIy8fHx5OXlUVZW\n1qzdX//6V8aOHctpp50WtEKDwbKgoMDMcJk5U9voikj48xvqXq+XpO/sXuVwOPB6vc3alJWVMWnS\nJMDcqQ0Xf/4zrFljFhfF6ZaxiEQAv1HWmoC+5ZZbmDFjRuO0G3/DL4WFhY1fu1wuXC5XqwsNtOee\ngyeeMLsvduxoWxkiIk14PB48Hs8Rv97vPPWKigoKCwtxu90ATJ8+ndjYWG677bbGNt26dWsM8u3b\nt3PCCSfw+OOPk5ub2/RCITRP3eOBK6+EJUugRw+7qxERObyA7v3S0NBAWloaixYtokuXLvTv35/S\n0lKcTuch20+YMIERI0YwevTooy4sWDZuBJfLDLkMHWp3NSIi/gV08VFcXBzFxcXk5OTg8/nIz8/H\n6XRSUlICQEFBwdFV284++8xMXbz3XgW6iESmqNkmoK7O9NAvuwzuusu2MkRE2kRb7x6CZZnl/w0N\n8PzzmrooIuFDe78cwt/+BqtXm5kuCnQRiWQR31N/910YNcoEevfu7X55EZGj0tbsjOgNvT791Exd\nnDVLgS4i0SFiQ72+Hq64wmwDoAOjRSRaROzwy803w4cfQlmZjqMTkfClG6WYLQDeeAPef1+BLiLR\nJeJ66qtXw7BhsHgx9OoV9MuJiARVVN8o/fJLGD0a/vpXBbqIRKeI6an7fGa1aEYGPPBA0C4jItKu\noranXlgI33wDfk7cExGJeBFxo3T+fDMXfcUKiI+3uxoREfuEfahv2gTXX2+CPSHB7mpEROwV1sMv\nu3ebLQD+9CcYONDuakRE7Be2N0otC666Ck480RxLp426RCQSRc3iowceMCtGly5VoIuIfCssQ33x\nYrjvPnjvPTjuOLurEREJHWE3pl5bCz/9qdkK4Ec/srsaEZHQ0mKou91u0tPTSU1NpegQk8DLysrI\nzMwkKyuLc845h8WLFwelUDDz0MeMgVtvhYsuCtplRETClt8bpT6fj7S0NMrLy0lMTKRfv36Ulpbi\ndDob2+zZs4cOHToAsG7dOkaNGsXmzZubXygAN0pvvNFsBfDSSxpHF5HoENAVpZWVlaSkpJCcnEx8\nfDx5eXmUlZU1afNtoAPs3r2bzp07t7Hk1nn8cXNT9OmnFegiIofjN9S9Xi9JSUmNzx0OB16vt1m7\nefPm4XQ6ufTSS3n44YcDXmRlJdxxB7z2mpnCKCIih+Y31GNa2SUeOXIkGzduZMGCBfzsZz8LSGHf\n+t//zAlGjz0G6ekBfWsRkYjjd0pjYmIitbW1jc9ra2txOByHbT948GAaGhr44osv6NSpU7PvFxYW\nNn7tcrlwuVx+i2togLw8M9tl1Ci/TUVEIoLH48Hj8Rzx6/3eKG1oaCAtLY1FixbRpUsX+vfv3+xG\n6ZYtW+jWrRsxMTFUVVVxxRVXsGXLluYXOoIbpb/9LaxZY04xOuaYNr1URCQiBHRFaVxcHMXFxeTk\n5ODz+cjPz8fpdFJSUgJAQUEBr7zyCrNnzyY+Pp6OHTsyZ86co/sv+D8vvggvv2x2XlSgi4i0Tkju\n/bJ+Pbhc8Oab0LdvcOsSEQllYX9Ixq5dZvz8/vsV6CIibRVSPfUDB0ygOxzwyCPtUZWISGgL610a\np0+Hzz83K0ZFRKTtQibU3W7429/g/ffh2GPtrkZEJDyFRKj/978wfryZ7dKli93ViIiEL9tvlNbV\nwejRcOedMHiw3dWIiIQ3W2+UWpbpoft8Zn90bdQlItJUWN0ofeQRs2J02TIFuohIINjWU3/nHTPs\nsnw5dO/eHhWIiISfsFh89OmncNVV8MwzCnQRkUBq91Cvrzdb6RYUwKWXtvfVRUQiW7sPv/zyl1BT\nA2VlEGv73BsRkdAW0jdKn33WLDJ6/30FuohIMLRrT71zZ4slS6Bnz/a4oohI+AvpG6XFxQp0EZFg\nCqldGkVEpKmQ7qmLiEhwKdRFRCKIQl1EJIK0KtTdbjfp6emkpqZSVFTU7PvPP/88mZmZ9O7dm3PP\nPZe1a9cGvFAREWlZi6Hu8/mYPHkybrebDRs2UFpaysaNG5u06datG2+//TZr167lD3/4AzfeeGPQ\nCo4EHo/H7hJChj6Lg/RZHKTP4si1GOqVlZWkpKSQnJxMfHw8eXl5lJWVNWkzaNAgTjrpJAAGDBjA\n1q1bg1NthNAP7EH6LA7SZ3GQPosj12Koe71ekpKSGp87HA68Xu9h2z/55JMMHz48MNWJiEibtLhN\nQEwbNjpfsmQJTz31FO++++5RFSUiIkfIasHy5cutnJycxufTpk2zZsyY0azdmjVrrO7du1vV1dWH\nfJ/u3btbgB566KGHHm14dO/evaWYbqLFFaUNDQ2kpaWxaNEiunTpQv/+/SktLcXpdDa2+fjjjxk6\ndCjPPfccAwcO9Pd2IiISRC0Ov8TFxVFcXExOTg4+n4/8/HycTiclJSUAFBQUcPfdd7Njxw4mTZoE\nQHx8PJWVlcGtXEREmmm3vV9ERCT4gr6itKWFS9GktraWIUOG0KNHD3r27MnDDz9sd0m28vl8ZGVl\nMWLECLtLsdXOnTsZO3YsTqeTjIwMKioq7C7JNtOnT6dHjx706tWLq6++mn379tldUruZOHEiCQkJ\n9OrVq/HvvvzyS4YNG8bZZ5/NxRdfzM6dO1t8n6CGemsWLkWT+Ph4HnzwQdavX09FRQWPPPJIVH8e\nM2fOJCMjo00zrCLRlClTGD58OBs3bmTt2rVN7ldFk5qaGh5//HGqqqpYt24dPp+POXPm2F1Wu5kw\nYQJut7vJ382YMYNhw4axadMmLrzwQmbMmNHi+wQ11FuzcCmanHHGGfTp0weAjh074nQ6+eSTT2yu\nyh5bt25l4cKFXH/99VG9JfOuXbtYunQpEydOBMw9rG8X8kWbH/7wh8THx1NXV0dDQwN1dXUkJiba\nXVa7GTz5boE+AAACvElEQVR4MKecckqTv5s/fz7jx48HYPz48cybN6/F9wlqqLd14VI0qampYdWq\nVQwYMMDuUmxx6623ct999xEb5ecafvjhh5x22mlMmDCBvn37csMNN1BXV2d3WbY49dRT+fWvf81Z\nZ51Fly5dOPnkk7nooovsLstW27ZtIyEhAYCEhAS2bdvW4muC+i8q2n+tPpzdu3czduxYZs6cSceO\nHe0up929/vrrnH766WRlZUV1Lx3MlOGqqipuuukmqqqq6NChQ6t+xY5EW7Zs4aGHHqKmpoZPPvmE\n3bt38/zzz9tdVsiIiYlpVaYGNdQTExOpra1tfF5bW4vD4QjmJUPe/v37GTNmDNdccw0jR460uxxb\nLFu2jPnz59O1a1fGjRvH4sWLufbaa+0uyxYOhwOHw0G/fv0AGDt2LFVVVTZXZY8VK1bw4x//mE6d\nOhEXF8fo0aNZtmyZ3WXZKiEhgc8++wyATz/9lNNPP73F1wQ11LOzs6murqampob6+nrmzp1Lbm5u\nMC8Z0izLIj8/n4yMDG655Ra7y7HNtGnTqK2t5cMPP2TOnDkMHTqU2bNn212WLc444wySkpLYtGkT\nAOXl5fTo0cPmquyRnp5ORUUFe/fuxbIsysvLycjIsLssW+Xm5vLMM88A8Mwzz7SuI9im9adHYOHC\nhdbZZ59tde/e3Zo2bVqwLxfSli5dasXExFiZmZlWnz59rD59+lhvvPGG3WXZyuPxWCNGjLC7DFut\nXr3ays7Otnr37m2NGjXK2rlzp90l2aaoqMjKyMiwevbsaV177bVWfX293SW1m7y8POvMM8+04uPj\nLYfDYT311FPWF198YV144YVWamqqNWzYMGvHjh0tvo8WH4mIRJDonnogIhJhFOoiIhFEoS4iEkEU\n6iIiEUShLiISQRTqIiIRRKEuIhJBFOoiIhHk/wPQg9KvOxKU/QAAAABJRU5ErkJggg==\n",
"text": [
""
]
}
],
"prompt_number": 34
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see above that the first 5 or 6 singular values account for 70-80% of the sum of squares of our data. Now let's see how a rank-$k$ approximation does in cross-validation.\n",
"
"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def LrRankK_LogLoss(U, sig, Vt, Y_tr, cv):\n",
" '''\n",
" Runs cross-validation on each rank k approximation\n",
" '''\n",
" results = {}\n",
" \n",
" for k in range(len(sig)):\n",
" X_k = pd.DataFrame(np.matrix(U[:, :k+1]) * np.diag(sig[:k+1]))\n",
" mu, serr = runXVal_LogLoss(cv, X_k, Y_tr)\n",
" results[k+1] = [mu, serr]\n",
" \n",
" \n",
" return results"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 35
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"r_svd = LrRankK_LogLoss(U, sig, Vt, Y_train, cv)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 36
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's plot the results. We'll also plot it with the stepwise forward selection results above.\n",
"
"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"plt.clf()\n",
"\n",
"k_svd = []\n",
"mu_svd = []\n",
"serr_svd = []\n",
"\n",
"for i in range(len(r_svd.keys())):\n",
" k_svd.append(i+1)\n",
" mu_svd.append(r_svd[i+1][0])\n",
" serr_svd.append(r_svd[i+1][1])\n",
" \n",
"plt.plot(k_svd, mu_svd, 'b.-', label = 'SVD Rank-k Proj')\n",
"plt.plot(k_svd, np.array(mu_svd)+np.array(serr_svd), 'b+')\n",
"plt.plot(k_svd, np.array(mu_svd)-np.array(serr_svd), 'b--')\n",
"\n",
"\n",
"plt.plot(ks, mus, 'r.-', label = 'Stepwise Forward')\n",
"plt.plot(ks, np.array(mus) + np.array(serrs), 'r+-')\n",
"plt.plot(ks, np.array(mus) - np.array(serrs), 'r--')\n",
"\n",
"plt.legend(loc=1, ncol=2)\n",
"\n",
"plt.title('Stepwise Forward Feature Selection vs SVD Dim Reduction')\n",
"plt.xlabel('k')\n",
"plt.ylabel('X Validated LogLoss')\n",
"\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 41,
"text": [
""
]
},
{
"metadata": {},
"output_type": "display_data",
"png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEZCAYAAACw69OmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXdYVMfXx79LURZYYAUVAREFFRDFggIWFjvGHktsiLEm\nviYajUGNCRgTjS2xK0ZiA0tMbLEQjQpijC22KCqCoBSjBhYBF2VZzvvH/LhhYamyNOfzPPeBe2fu\nzJl7Z+fcaeeIiIjA4XA4HM4bolPVAnA4HA6ndsAVCofD4XAqBK5QOBwOh1MhcIXC4XA4nAqBKxQO\nh8PhVAhcoXA4HA6nQuAKpQJ55513sGvXrqoWo8rx9vZGcHBwVYtRLQkPD0fjxo0rPN3IyEg4OjpW\neLq1nQ8//BBff/11VYtRCG3VEwCQSCSIj4/XSto1VqGcP38enTt3hpmZGczNzdG1a1dcvXoVALB9\n+3Z069at0mU6fvw4fH19KzRNb29viMViSCQS4bh06VKF5lHRiEQiiEQijWGBgYHQ19dXK8/KlSvf\nKL/AwMAKf+4lERwcDCcnJ5iYmMDS0hL9+/dHZmZmpeWvo6ODhw8fCufdunXDvXv3Ki3/snLnzh30\n6dMH5ubmkEqlcHNzw4kTJ5CUlAQ9PT21suQxdOhQfPbZZwBYeY2NjSGRSGBhYYFevXrhp59+KjZP\nOzs7GBoawsTEBFKpFF26dEFQUBDyb73btGkTFi5cWCFlnDBhAurWrQuJRIJ69eqhZ8+euHPnToWk\nXV40fdxlZGTAzs5OK/nVSIWSnp6OAQMGYObMmZDL5UhKSkJAQADq1q1b1aJVOCKRCBs2bEBGRoZw\nuLu7lymNnJwcrchGRCjrvliRSITRo0erlefTTz/VinylpazPJyIiAp9//jn27t2L9PR03L17F6NG\njdKSdEVTk/YkDxw4EH379sXTp0/x7NkzrF27FqamprC2tkavXr0K9exTU1Nx4sQJ+Pn5Cddu3bqF\njIwMREdHY8KECZgxYwa++uqrIvMUiUQ4evQo0tPT8fjxY8ybNw/Lli3DpEmTtFJGkUgEf39/ZGRk\nIDk5Gba2tnj//fe1kldZZKpUqAZy5coVMjMz0xgWFRVFBgYGpKurS8bGxiSVSomI6NWrVzRnzhyy\ntbWlhg0b0gcffEBZWVlERHT27FmytramJUuWkIWFBdnZ2VFoaCgRET18+FAtr8mTJ1ODBg2E83Hj\nxtHq1auJiEgmk9HWrVuJiOjBgwfk5eVFpqamZGFhQe+9955wz927d6lXr15Ur149atmyJf30009F\nltXb25uCg4MLXc/NzaXFixdTkyZNqEGDBjR+/Hh68eIFERHFxcWRSCSi4OBgsrW1JS8vL/Lz86NV\nq1YREVFiYiKJRCLasGEDERHFxMRQvXr1iIgoNTWV+vfvT/Xr1yepVEoDBgygxMREIV+ZTEaff/45\nde7cmcRiMcXGxtLJkyepZcuWZGpqSjNmzFB7DgUJCAigcePGaQwLDg4mJycnkkql1LdvX3r06JEQ\n9vHHH1Pjxo3JxMSEOnToQJGRkUREdOLECapTpw7p6+uTsbExtW3bloiImjRpQr///rvGfAs+H5lM\nVmL++VmxYgUNGTJEYxhRyXXNxsZGiJuUlETvvvsu1a9fn5o2bUpr164VwlQqFX3zzTdkb29PEomE\n3NzcKCEhgbp160YikYiMjIzI2NiYfvrpp0LpRkVFkUwmIzMzM2rVqhUdOXJECPPz86Pp06dT//79\nSSKRkLu7O8XGxmosi4+PD61fv17tWps2bejgwYNERDRr1ixq0KABmZiYUOvWren27duF0nj+/DmJ\nRCKhfhZk9+7dZG9vr3Ztw4YN1L59e+FcJBIVkvHnn38mAwMDSk1N1ZiunZ0dnT59Wu3a5cuXSUdH\nh+7cuUNE7FksXLiQiP5rB5YvX07169enRo0a0cGDB+nYsWPUvHlzqlevHi1dulRjXkREEyZMoC++\n+EI4P3bsGBkaGgrnxb1rhUJBfn5+JJVKydnZmZYvX672PguWP7/cRESHDh0iV1dXMjExIXt7ewoL\nC6MFCxaQrq4uGRgYkLGxMX300UeF0kpLSyNfX1+qX78+NWnShL7++mvKzc0lIqJt27ZRly5d6NNP\nPyWpVEpNmzalEydOFFl+IvaFWeNIT08nc3Nz8vPzoxMnThSqUNu3b6euXbuqXZs1axYNHjyY5HI5\nZWRk0MCBA2n+/PlExCqSnp4ezZkzh7KzsykiIoKMjIwoOjqaiIhsbW3p2rVrRETUokULsre3p7t3\n7wphN27cICL1xn/UqFG0ZMkSIiJ6/fo1/fHHH0RElJmZSTY2NrR9+3ZSqVR0/fp1srCwoKioKI1l\n9fb21tg4BwcHk4ODA8XFxVFmZia9++675OvrS0T/NZh+fn6kUCgoKyuLfvzxRxo4cCAREYWGhpK9\nvb2g5IKDg4UGMiUlhQ4cOEBZWVmUkZFBI0aMUGs8ZTIZNWnShKKiokilUtGzZ89IIpHQL7/8Qjk5\nOfT999+Tnp6eRiVIVLRCOXToEDk4ONC9e/dIpVLR119/TZ07dxbCQ0JCKDU1lVQqFa1atYosLS3p\n9evXREQUGBgolD2Pgo1JYGBgIYWS//mUlH9+IiMjSSwWU0BAAJ0/f55evXqlFl5SXctrKFQqFbVv\n354WL15MSqWSHj58SM2aNaPffvuNiIiWL19OrVu3FurhzZs3KSUlhYgKNzD5083OziZ7e3taunQp\nKZVKOnPmDEkkErp//z4RscbI3Nycrly5Qjk5OTR27FgaNWqUxrLu3LmTunTpIpzfuXOHzMzMKDs7\nm8LCwqhDhw6Corh37x49efKkUBq5ubnUvHlzGjBgAB06dIj++ecftXCFQkGmpqZ0/vx54ZqHhwet\nWbNGONekULKzs0lPT4/CwsI0yq5JoRCx3+zmzZuJSF0J5LUDixcvppycHPrhhx/I3NycxowZQ5mZ\nmXTnzh0Si8UUHx+vMb8JEyYIjXxmZiaNGzeOunfvTkQlv2t/f3/y8vIiuVxOCQkJ1KpVK2rcuHGR\n5c8v96VLl8jU1FT4gEpKSqJ79+4RkeYP0vxp+fr60pAhQygzM5Pi4+OpRYsWQvxt27aRvr4+bd26\nlXJzc2nTpk1kZWWlsex51EiFQsS+8idMmEA2Njakp6dHgwYNoqdPnxIRexD5FUpubi4ZGRmpvZAL\nFy5Q06ZNiei/iqRQKITwkSNH0uLFi4mIPfTvvvuOnjx5Qi1btiR/f3/avHlzod5L/pc3fvx4mjp1\nqtrXPRHR3r17qVu3bmrXpk6dSosWLdJYTplMRoaGhmRmZkZmZmbUoUMHIiLq0aMHbdq0SYh3//59\n0tfXJ5VKJTSYcXFxQnhMTAxJpVLKzc2lDz74gIKCgoQGaPz48fT9999rzP/69etCLy+vjAEBAcL5\njh07yNPTU+0eGxubYhVKnTp1hPJIpVJKTk4mHx8ftXtUKhUZGhrS48ePNaYjlUrp1q1bQpoFlVTB\nxkRTDyX/8ylr/idOnKCBAweSmZkZGRsb0+zZs0mlUpWqruU994sXL5Ktra1aukuWLKH333+fiNjH\nS/6eRX6KUyjnzp0jS0tLtfijR4+mwMBAImIKZcqUKULY8ePHydHRUWM+6enpZGRkJDyHBQsW0KRJ\nk4iI6PTp09SiRQu6ePEiqVQqjffnkZiYSDNmzCB7e3vS0dEhLy8vevDggRA+efJkmjp1KhERRUdH\nU506dej58+dFljcPS0tL2r17t8Y8i1IoHh4ewsdefiVw9uxZEovFwhd6eno6iUQiunz5snBvhw4d\n6NChQxrz8/PzIwMDAzIzMyMdHR1q1qyZUIaS3nV+5UJEtGXLlmJ7KPkVytSpU2n27NkaZdL0QZqX\nVk5ODtWpU0f4OCYiCgoKIm9vbyJi7aiDg4MQ9vLlSxKJREI7q4kaOYcCAI6Ojti2bRsSEhJw+/Zt\nJCcnY9asWRrjPn/+HAqFAh06dIBUKoVUKkW/fv3w77//CnGkUinEYrFw3qRJEyQnJwMAZDIZwsPD\nERkZCS8vL8hkMkRERODcuXNFTv4vX74cRIROnTrBxcUF27ZtAwA8evQIly5dEuSQSqXYvXs3nj59\nqjEdkUiEdevWQS6XQy6XCwsPnjx5giZNmgjxbG1tkZOTo5ZO/lUi9vb2MDIywo0bNxAZGYkBAwbA\nysoK0dHROHfuHGQyGQBAoVBg2rRpsLOzg6mpKWQyGV68eKE2Xp8/3eTkZNjY2KjJXNLqlPfee08o\nT2pqKho1aoRHjx5h5syZwjMxNzcHACQlJQEAVq5cCWdnZ5iZmUEqleLFixdq76885JezpPwL4uPj\ngyNHjkAul+Pw4cPYvn07tm7din///bfEupY/z+TkZLW6sHTpUjx79gwAkJiYCHt7+zKXKzk5udA7\nyF+fRSIRGjZsKISJxeIiFxRIJBL0798fe/bsAQDs3bsXY8eOBQD06NEDM2bMwP/93/+hYcOGmDZt\nGjIyMjSmY21tjXXr1iEmJgaPHj2CkZERxo8fL4T7+flh//79eP36NXbt2gUfHx9YWFgUW06lUonn\nz5+jXr16JTwRdRITE4u8x9zcXJh3yGsPCj6rly9farxXJBJh7ty5kMvliI+PR926dbFz504AJb/r\ngu/M1ta2TOUprp4UNY/y77//QqlUFmpH8td5S0tL4X9DQ0MAKHbxSY1VKPlp2bIl/Pz8cPv2bQCF\nH6CFhQXEYjGioqKEhiwtLQ3p6elCHLlcDoVCIZw/evQI1tbWAJhCiYyMRHh4OLy9vdG1a1f88ccf\niIiIgLe3t0aZGjZsiC1btiApKQlBQUGYPn06YmNjYWtrC5lMJsghl8uRkZGBDRs2lKnMVlZWakv/\nHj9+DD09PbXKX/A5yGQy7N+/H0qlElZWVpDJZNi+fTvkcjnatm0LAFi1ahWio6Nx+fJlvHjxAhER\nEYUm3/Ona2VlhYSEBOGciNTOCyISiTROJtva2mLLli1qz+Xly5fw8PBAZGQkVqxYgf379yMtLQ1y\nuRympqZCOpp+MEZGRmo//H/++UejLKXJvyR69OiBHj164M6dO6Wqa3k0btwYTZs2VcszPT0dR48e\nFcJjYmJKzL8gee8k/3POX5/LyujRo7Fnzx78+eefePXqFbp37y6EffTRR7h69SqioqIQHR2NFStW\nlJiejY0Npk+fLvxeAaBLly6oV68eDh8+jNDQULXJ+KI4fPgw9PT00KlTp1KX5cqVK0hOTkbXrl2F\naxU5cZ33zBs3boy1a9di8eLFSE9PL/FdN2rUCI8fPxbSyf8/wBrz/O3TkydPhP+LqyfFlc3CwgL6\n+vqF2pGCH4hloUYqlPv37+O7774TNGlCQgL27NkDT09PAKwxT0xMhFKpBMCWHE6ZMgWzZs3C8+fP\nAbAvz5MnT6qlGxAQAKVSicjISBw7dgwjRowAADg4OMDAwAAhISGQyWSQSCRo0KABfvnlF+HLviD7\n9+9HYmIiAMDMzAwikQi6uroYMGAAoqOjERISAqVSCaVSiStXrhS75FNTAzx69Gh8//33iI+PR2Zm\nJhYsWIBRo0ZBR6foVyqTybB+/Xp4eXkBYEsK169fj27dugkVLzMzE2KxGKampkhNTcWiRYuKlad/\n//64c+cODh48iJycHKxdu1Zj411cWQDggw8+wJIlSxAVFQUAePHiBfbv3w+ALXPU09ODhYUFsrOz\n8dVXX6k10JaWloiPj1dLu23btti7dy9ycnJw9epV/PLLL8X+uIrLvyBHjhzBvn37IJfLQUS4fPky\nIiIi4OHhAZFIVKq6BgCdOnWCRCLB8uXLkZWVBZVKhdu3bwu90MmTJ+OLL75ATEwMiAi3bt1Camoq\nAFbHY2NjNcrn7u4OQ0NDLF++HEqlEuHh4Th69KiwEq2od1AU77zzDh49eoSAgAC11WxXr17FpUuX\noFQqYWhoCAMDA+jq6ha6Py0tDQEBAYiNjUVubi7+/fdf/Pjjj8LvFWAN3/jx4/HZZ5/hxYsXGDhw\nYKF08uROTU1FaGgoZsyYgXnz5kEqlRYpe949eY336NGj4evri1atWgnhZX0eJeWVR69eveDg4IBN\nmzbB3d292Hc9cuRILF26FGlpaUhMTMS6devU0mrbti1CQ0OhUqkQFhaGc+fOCWGTJk3Ctm3bcObM\nGeTm5iIpKQn3798HUHw90dXVxciRI/H5558jMzMTjx49wvfff49x48aV+xnUSIWStxfD3d0dxsbG\n8PT0RJs2bbBq1SoAQM+ePdGqVStYWlqiQYMGAIBly5bBwcEBHh4eMDU1Re/evREdHS2kaWlpCalU\nCisrK/j6+iIoKAgtWrQQwr29vWFhYSF85eX1TNq3b69RxqtXr8LDwwMSiQSDBw/G2rVrYWdnB2Nj\nY5w8eRJ79+6FtbU1GjVqhPnz5yM7O7vI8mpqCCdOnAhfX194eXmhWbNmMDQ0VKuEmu7x8vJCZmam\noFC6dOmCrKws4RwAZs2ahaysLFhYWKBz587o169fobTyn5ubm2P//v2YN28eLCwsEBMTo/b1p6ks\nmmQbMmQI/P39MWrUKJiamqJ169b47bffALDhJR8fH7Ro0QJ2dnYQi8VqQwJ5it/c3Bxubm4AgMWL\nFyM2NhZSqRSBgYHCME1Rz6e4/AsilUrxww8/oEWLFjA1NYWvry8+++wzjB49GkDJdS0vb11dXRw9\nehQ3btxAs2bNUL9+fUydOlVQlrNnz8bIkSPRp08fmJqaYsqUKXj16hUAtvfGz88PUqkUP//8s9pz\nrVOnDn799VecOHEC9evXx4wZM7Br1y6hPmt6B8Up2zp16uDdd9/F6dOnMWbMGOF6eno6pk6dinr1\n6sHOzg4WFhaYO3euxvsfPXqEXr16Cc9WLBZj+/btavHGjx+PhIQEvPfee9DX1y+UjqurKyQSCZo3\nb44ff/wRq1evRmBgYJFyA2y5somJCWxtbbF06VLMmTNHGH7W9CzK8lwKoum5zp07F2vXroVKpSr2\nXQcEBKBJkyZo2rQpfHx8MH78eLW01qxZg19//VUYIh86dKgQ1rFjR2zbtg2ffPIJzMzM4O3tLfRw\nZs6ciZ9//hn16tXTOCWwbt06GBkZoVmzZujWrRvGjh0rLHUuaz0BoN1lwydOnKCWLVuSg4MDffvt\ntxrjnD17ltq2bUutWrUSlm8SEa1evZpcXFyoVatWwrLc/KxcuZJEIpGw6uVNKLjkksPhcDhlR6/U\n6reMqFQqzJgxA7///jusra3RsWNHDBo0CE5OTkKctLQ0/N///R9+++032NjYCBOXt2/fxtatW3Hl\nyhXo6+vDx8cHAwYMECaeEhIScOrUKbXJJA6Hw+FULVob8rp8+TIcHBxgZ2cHfX19jBo1CocPH1aL\ns3v3bgwbNkyYBMpb1XHv3j24u7sLY7IymQwHDhwQ7ps9ezaWL19eofJW+o5SDofDqWVoTaEkJSWp\nLYOzsbEptATzwYMHSE1NRffu3eHm5iaYX3BxcUFkZCRSU1OhUChw7NgxYYL78OHDsLGxQZs2bSpM\n1vxjjhwOh8MpH1ob8irNF79SqcS1a9dw+vRpKBQKeHp6wsPDA46OjvD390efPn1gZGSEdu3aQVdX\nF1lZWViyZAlOnTolpEE1yJ4Rh8Ph1Ga0plCsra3V9iMkJCRo3ACXt25fLBbDy8sLN2/eRPPmzTFx\n4kRMnDgRALBgwQLY2toiNjYW8fHxcHV1BcA29HTo0AGXL18WVnPl4eDgUORyOQ6Hw+EUxtXVFW3b\nti20Aq/UaGu2X6lUUrNmzSguLo5ev35Nrq6uhexV3b17l3r27Ek5OTn08uVLcnFxEYy25W3vf/To\nETk6Omo0LGdnZ1fkKi8tFq1akN/8SW2jNpeNiJevplObyxcQEPBGbafWeih6enpYv349+vbtC5VK\nhUmTJsHJyQlBQUEAgGnTpsHR0RE+Pj5o06aNsPnQ2dkZADB8+HCkpKRAX18fGzduhImJSaE8+EQ6\nh8PhVB+0plAAoF+/fujXr5/atWnTpqmdf/rppxr9YeTfCVoUmpzycDgcDqdqqJE75Tko0oZYbaA2\nlw3g5avp1ObyvWnZRES1c5lUUUYIORwOh1M0b9J2anXI622hXr16kMvlVS0Gh8PhlBqpVCoYG60o\neA+lhuXF4XA4FUFR7dabtGd8DoXD4XA4FQJXKBwOh8OpELhC4XA4HE6FwBUKp0ZhZ2eH06dPlypu\nYGAgfH19tSxRYd555x3B0CmnauDvgOHt7Y3g4OBKy48rlFrO+fPn0blzZ5iZmcHc3Bxdu3bF1atX\ncfHiRRgbG6v5Xc+jXbt22LhxI+Lj46GjowOJRAKJRAJLS0sMHDgQv//+e7F56ujowNjYGBKJBNbW\n1vj444+Rk5NTIeUpyuNjUXFLy4QJE1C3bl1IJBKYm5ujT58+ghvVsnL8+PEqUWRVRVF1DAC2b9+O\nbt26VbpM2ngH3t7eEIvFwu8hz3NsdaYsv5eKgCuUWkx6ejoGDBiAmTNnQi6XIykpCQEBATAwMICH\nhwdsbGzw888/q91z+/Zt3L17V3BnCzD/6hkZGbh16xZ69+6NoUOHYseOHcXmfevWLWRkZODcuXM4\ncOAAtmzZopUyFkdZVqqIRCL4+/sjIyMDiYmJaNCgASZMmKAxTb6i7z+KqmN169atatEqHJFIhA0b\nNiAjI0M43N3dy5RGRX1YFaS61EuuUGox0dHREIlEeO+99yASiWBgYIDevXvDxcUFAODn54edO3eq\n3bNz5070798fUqm0UHoNGjTAxx9/jMDAQPj7+5dKBnt7e3Tp0gVRUVHCtZkzZ8LW1hampqZwc3PD\n+fPnhbDAwECMHDkSfn5+MDExgYuLC/766y+Nad+9exfNmjXDvn37SpRDqVRi9OjRGD58OJRKZbFx\nxWIxRo8ejdu3bwNgX6YLFy5Ely5dYGRkhLi4OFy4cAEdO3aEmZkZOnXqhD///FO4v7KHGaqSoupY\n69atcffuXXz44Yf4888/IZFIUK9ePQDA69ev8emnn6JJkyawtLTEhx9+iFevXgEAwsPDYWNjg6VL\nl6J+/fpo2rQpdu/eDQCIi4tTq5dTpkxBw4YNhXNfX1+sWbMGgPo7iImJgUwmg5mZGerXr49Ro0YJ\n99y7dw+9e/eGubk5HB0dsX///jI/AyLC119/DTs7OzRs2BB+fn6Cr/i8Xv6PP/6IJk2aoGfPnpgw\nYQK+++47AMxvlI6ODjZu3AgAiI2Nhbm5OQBALpdjwIABaNCgAerVq4eBAweq+ZTSVC9PnToFR0dH\nmJmZ4aOPPqp0RcMVipaZOhXw9gbeeQdIS6vcNFq2bAldXV1MmDABYWFhhTZfjhs3DufOnROcl+Xm\n5mLPnj3w8/MrNt2hQ4fi2bNnxQ4J5VXie/fuITIyEp06dRLCOnXqhJs3b0Iul2PMmDEYMWIEsrOz\nhfBff/0Vo0ePxosXLzBo0CDMmDGjUPrXrl2Dj48P1q9fj/fee69YeV+9eoUhQ4ZALBZj//790NfX\nL1bmzMxMhIaGon379kJYSEgItm7diszMTBgZGaF///6YNWsWUlNTMXv2bPTv3194vpU9zFCVlay4\nOubk5ITNmzfD09MTGRkZwia6efPmISYmBjdv3kRMTAySkpLw1VdfCfc9ffoUKSkpSE5Oxo4dOzB1\n6lQ8ePAATZs2hYmJCa5fvw6A2fuTSCS4d++ecJ5nOiT/O/jiiy/g4+ODtLQ0JCUl4eOPPwYAvHz5\nEr1798a4cePw/Plz7N27F9OnT8fdu3eLLK+mxnnbtm3YsWMHwsPD8fDhQ2RmZhaqs+fOncO9e/fw\n22+/QSaTITw8HAAQERGBZs2aCbYLIyIi4OXlJeQ1adIkPH78GI8fP4ZYLC6Ubv56KZFIMGzYMCxZ\nsgQpKSmwt7fHH3/8Ubl1sdx2iqs5lVm04vKSyYiAijtGjCibbHfv3qUJEyaQjY0N6enp0aBBgwTX\nAEREvXr1oiVLlhAR0cmTJ6l+/fqUk5NDRERxcXEkEolIpVKppZmVlUUikYguXLigMU+RSEQmJiZk\nZGREIpGIPvroo2JllEqldOvWLSJi5rN79+4thN25c4fEYrFwbmdnR19++SXZ2NhQREREsekGBgbS\noEGDyMvLi2bOnFlsXD8/PzIwMCAzMzOytLSkwYMH08OHD4mIyNvbW81k+c6dO8nd3V3tfk9PT9q+\nfbsQPzg4uNj8KpQqrmTF1bFt27ZR165dhbi5ublkZGREsbGxwrULFy5Q06ZNiYjo7NmzpKenRwqF\nQggfOXIkLV68mIiIfH196bvvvqMnT55Qy5Ytyd/fnzZv3kwPHz4kMzMz4Z7872D8+PE0depUSkxM\nVJN779691K1bN7VrU6dOpUWLFhXxmGVkaGhIZmZmZGZmRh06dCAioh49etCmTZuEePfv3yd9fX1S\nqVTCbyguLk4Ij4mJIalUSrm5ufTBBx9QUFAQ2djYCLJ+//33GvO/fv06SaVStTLmr5c7duwgT09P\ntXtsbGyKrItFtVtv0nbyHoqWMTRkf93cALm8fL/wPIPNbm5AWaciHB0dsW3bNiQkJOD27dtITk7G\nrFmzhHA/Pz9hNcyuXbswevRo6OrqFptmXrc7bwhDE9evX0dmZib27duHnTt34tGjR0LYypUr4ezs\nDDMzM0ilUrx48QL//vuvEJ5/GMPQ0BCvXr1Cbm4uAPbVFhQUhC5dughfcgAQGhoqTJT2799fiHvx\n4kXcvn27xCE6kUiEuXPnQi6X48mTJzh06BCaNm0qhOd3Z52cnAxbW1u1+5s0aYLk5ORi89AaVVzJ\nSqpj+Xn+/DkUCgU6dOgAqVQKqVSKfv36qb1/qVQKsVgsnOd/tnlf95GRkfDy8oJMJkNERATOnTtX\n5OT/8uXLQUTo1KkTXFxcsG3bNgDAo0ePcOnSJUEOqVSK3bt34+nTpxrTEYlEWLduHeRyOeRyubDw\n4MmTJ2jSpIkQz9bWFjk5OWrp5K8/9vb2MDIywo0bNxAZGYkBAwbAysoK0dHROHfuHGQyGQBAoVBg\n2rRpsLOxlR0qAAAgAElEQVSzg6mpKWQyGV68eKHWSypYLzU5MaxMuELRMrt3AyNGAKdOAWZmVZcG\nwIYn/Pz8hLkBgA1fJSYm4uzZszh48GCJw10AcPDgQTRs2BAtW7YsMe6IESMwYMAABAYGAgAiIyOx\nYsUK7N+/H2lpaZDL5TA1NS31OK9IJEJQUBAePXqE2bNnC9fHjh0rTJQeO3ZMiNunTx/MmzcPPXv2\nxLNnz4pNuzgZ8g8bWFtbqylIgDVO1tbWpSpDhVONKlnBOlZwuCXPQ2tUVJTQMKelpQlzDgCbO1Ao\nFMJ5/mcrk8kQGRmJ8PBweHt7o2vXrvjjjz8QERFRpKXchg0bYsuWLUhKSkJQUBCmT5+O2NhY2Nra\nQiaTCXLI5XJkZGRgw4YNZSqzlZUV4uPjhfPHjx9DT09P7cOo4HOQyWTYv38/lEolrKysIJPJsH37\ndsjlcrRt2xYAsGrVKkRHR+Py5ct48eIFIiIiCs2J5E/XyspKzUsuEamdVwZcoWgZMzPgp5/eTBGU\nN4379+/ju+++E3oUCQkJ2LNnDzw9PYU4RkZGGD58ON5//33Y2dmpzRvkkVeBnz59ivXr1+Orr77C\n0qVLSy3HvHnzsGfPHiQmJiIjIwN6enqwsLBAdnY2vvrqK7XGpDRIJBKEhYXh3LlzmD9/fpHx8uSe\nO3cuxowZg549eyIlJaXYuCWlBbA9DtHR0dizZw9ycnKwb98+3Lt3DwMGDChTOSqMKqxkJdWxhg0b\nIjExUVgIkedIb9asWXj+/DkA1uM9efKkWroBAQFQKpWIjIzEsWPHMGLECADMtbeBgQFCQkIgk8kg\nkUjQoEED/PLLL8KXfUH2798vzBOamZlBJBJBV1cXAwYMQHR0NEJCQqBUKqFUKnHlyhVhTkYTmurJ\n6NGj8f333yM+Ph6ZmZlYsGABRo0aBR2doptXmUyG9evXC71sb29vrF+/Ht26dROURGZmJsRiMUxN\nTZGamopFixYVK0///v1x584dHDx4EDk5OVi7di3++eefImXQBlyh1GLy1sm7u7vD2NgYnp6eaNOm\nDVatWqUWz8/PD48fP8b48eM1pmNmZgZjY2O0adMGYWFh+PnnnzUuqc2j4NeYi4sLevToge+++w4+\nPj7w8fFBixYtYGdnB7FYrDZ8pGlCW9OkoqmpKU6dOoUTJ04gICCgSDny7l24cCGGDBmCXr16IU3D\npHNJE+n5w+rVq4ejR49i1apVsLCwwMqVK3H06NFihwBrKyXVsZ49e6JVq1awtLREgwYNAADLli2D\ng4MDPDw8YGpqit69eyM6OlpI09LSElKpFFZWVvD19UVQUBBatGghhHt7e8PCwkLoteT1TDR9DAHA\n1atX4eHhAYlEgsGDB2Pt2rWws7ODsbExTp48ib1798La2hqNGjXC/Pnz1RaIFERTHZk4cSJ8fX3h\n5eWFZs2awdDQEOvWrSv2Hi8vL2RmZgoKpUuXLsjKylIbxp01axaysrJgYWGBzp07o1+/fsX+NszN\nzbF//37MmzcPFhYWiImJQdeuXYssizbg1oZrWF6c6o9MJsOUKVMwbty4qhalxhEeHg5fX99KH6p5\nG+HWhjmcao5CocDDhw/VJvQ5nLcFrlA4nAri2bNnaNSoEby9vdGlS5eqFqfGUqn7JjgVCh/yqmF5\ncTgcTkXAh7w4HA6HU23hCoXD4XA4FYLWFUpYWBgcHR3RvHlzLFu2TGOc8PBwtGvXDi4uLmqbk9as\nWYPWrVvDxcVFMPoGsH0FTk5OcHV1xbvvvosXL15ouxgcDofDKYlyG20pBTk5OWRvb09xcXGUnZ1N\nrq6uFBUVpRZHLpeTs7MzJSQkEBHR8+fPiYjo77//JhcXF8rKyqKcnBzq1asXxcTEEBGzOZVnX8rf\n35/8/f0L5a3lohER0dmzRAEBREZGUgLAD37wgx815shvF6yi2k6t9lAuX74MBwcH2NnZQV9fH6NG\njcLhw4fV4uzevRvDhg0TbNBYWFgAYFZq3d3dYWBgAF1dXchkMhw4cAAA0Lt3b2EXqru7u7ALtrLx\n9gYCA4HMzFQEBJBgFoEf/OAHP6r7kWf9uSLRqkJJSkpSM05mY2OjZs8fAB48eIDU1FR0794dbm5u\ngqFCFxcXREZGIjU1FQqFAseOHdOoOH788Ue888472iwGh8PhcEqBnjYTL816cqVSiWvXruH06dNQ\nKBTw9PSEh4cHHB0d4e/vjz59+sDIyAjt2rUrZBvnm2++QZ06dTBmzBiNaecZJASYeYaijMdVBFpM\nmsPhcLRGeHi44J/lTdGqQrG2tlYzoZCQkKDRvHKeBVKxWAwvLy/cvHkTzZs3x8SJEzFx4kQAwIIF\nC9RsPm3fvh3Hjx/H6dOni8w/v0LRNlyhcDicmkjBj21NRihLi1aHvNzc3PDgwQPEx8cjOzsb+/bt\nw6BBg9TiDB48GOfPn4dKpYJCocClS5fg7OwMAIK58cePH+PgwYNCTyQsLAwrVqzA4cOHYWBgoM0i\nlAofH6BLlzdzmMfhcDg1Ha32UPT09LB+/Xr07dsXKpUKkyZNgpOTE4KCggAA06ZNg6OjI3x8fNCm\nTRvBtHWeQhk+fDhSUlKgr6+PjRs3wsTEBADw0UcfITs7G7179wYAeHp6Cj6ZKxsi4Pp1IM/VxuDB\nwJkzQAk+qjgcDqfWwU2vVADvvAOcOAHY2jJ3EnI5MGECO5o1006e4eF8mI3D4VQ83PRKFZPn7O7m\nTXb8+iuQng64uwPduwM7dwIvX1ZsnhU0h8bhcDgVBlcoFUBBZ3eursDq1UBiIjBjBrBvH2BjA0yd\nCvz5Jxsm43A4nNoGH/KqJJKSgF27gB9/ZPMr778P+PoCjRqVPo3w8P96JosWAXmOCr29+fAXh8Op\nGN6k7eQKpQJ49QqoUwcoxoW0ABFw4QJTLAcOAF27AhMnAv37szRKS2AgOzgcDqci4XMoVcz27UCL\nFsDKlUBKSvFxRSK2xDg4GEhIAIYNY8NjjRsDs2cDt29XisgcDodT4XCFUgFMmwaEhAB//w3Y2wPj\nxwMXL5Y8V2JszFaCRUQA588DYjHb09KxI7BpU/F7WvgQF4fDqW7wIa8KJiUF2LYN2LwZOHIE+N+W\nmlKjUgGnTrE0fvuNLUmeOBHo0aN0Q2ocDofzJvA5FA1U9aQ8ERveehNSUoA9e9h8S0rKf3tbmjat\nCAk5HA6nMFyhaKCqFUpRxMYCV68CQ4eWbRL+xg3Wa9m9G6hbF7CzA0xM2HnecmUOh8N5U7hC0UB1\nVSjXrwNz5gBRUcCkSWxvSpMmpb//9WvWQ3nyhJ2PGMH2wHA4HE5FwFd51SDatWO2vs6eBTIzgfbt\ngUGDgHv3Snd/3bpAmzbs/wYNgC1btCcrh8PhlAWuUKoIJydgzRrg8WNmUNLIqPT37t0LDBnChrp2\n79aejBwOh1MW+JBXNSZP/KIm92Nj2Z6WkBCgV6/Kk4vD4dRe+JBXLeXSJWYXbNMmICOjcLi9PbMT\nNnYsEB1d+fJxOBxOfrhCqcZ06gR89x3w++9s4n76dLZ5Mj8yGfDNN8CAAcxsPofD4VQVXKFUY3R0\n2FDWL78wRdKgAdtJv3everzJk5lCGTECUCqrRlYOh8Phcyg1DKWS7aYv6PlYpWKrxWxtgY0b33xT\nJYfDeTvhcyhvEfr6hZUJwEzi79kDnDsHbNhQ+XJxOByOVn3KcyoXExPmLbJzZ2b9uE+fqpaIw+G8\nTfAeSi2jWTO2c37cuNJvluRwOJyKgCuUGs7du2yVV368vIBvvwUGDizZPwuHw+FUFFyh1HAaNQK2\nbgUOHVK/PnEi200/fDiQnV01snE4nLcLvsqrFnDxIjPfcuUKW+WVh0rFlIqVFfPPwld+cTickqi2\nq7zCwsLg6OiI5s2bY9myZRrjhIeHo127dnBxcYF3PjeEa9asQevWreHi4oI1a9YI11NTU9G7d2+0\naNECffr0QVpxbg3fEjw8gE8+YTvmc3L+u66ry2x9XbgArFtXdfJxOJy3BNISOTk5ZG9vT3FxcZSd\nnU2urq4UFRWlFkcul5OzszMlJCQQEdHz58+JiOjvv/8mFxcXysrKopycHOrVqxfFxMQQEdHcuXNp\n2bJlRET07bffkr+/v8b8tVi0aolKRdSrF9EXXxQOi4sjsrQkOnGi0sXicDg1jDdpO7XWQ7l8+TIc\nHBxgZ2cHfX19jBo1CocPH1aLs3v3bgwbNgw2NjYAAAsLCwDAvXv34O7uDgMDA+jq6kImk+HAgQMA\ngCNHjsDPzw8A4Ofnh0MFJw/eUnR0gF27mHn8gtjZAfv3M1/3UVGVLhqHw3lLKJNCUalUSE9PL1Xc\npKQkNG7cWDi3sbFBUlKSWpwHDx4gNTUV3bt3h5ubG3bt2gUAcHFxQWRkJFJTU6FQKHDs2DEkJiYC\nAJ4+fYqGDRsCABo2bIinT5+WpQi1GktL5glSE127AitWsJVf//5buXJxOJy3gxI3No4ePRpBQUHQ\n1dVFx44d8eLFC8ycOROfffZZsfeJSjEDrFQqce3aNZw+fRoKhQKenp7w8PCAo6Mj/P390adPHxgZ\nGaFdu3bQ1dXVmEdx+QQGBgr/e3t7q83RvI34+bFlxsOGAadOlc0FMYfDqZ2Eh4cjPDy8QtIqUaFE\nRUXBxMQEoaGh6NevH7799lu0b9++RIVibW2NhIQE4TwhIUEY2sqjcePGsLCwgFgshlgshpeXF27e\nvInmzZtj4sSJmDhxIgBgwYIFsP3f8qWGDRvin3/+gaWlJZ48eYIGDRoUKUN+hcJhLFnCejHTpwM/\n/FC6lV/h4cBbros5nFpLwY/tRYsWlTutEoe8cnJyoFQqcejQIQwcOBD6+vql6n24ubnhwYMHiI+P\nR3Z2Nvbt24dBgwapxRk8eDDOnz8PlUoFhUKBS5cuwdnZGQDw7NkzAMDjx49x8OBBjBkzBgAwaNAg\n7NixAwCwY8cODBkypGwlfssoaH1YR4c55LpyBVi9unRpVNDHC4fDqeWU2EOZNm0a7Ozs0KZNG3h5\neSE+Ph6mpqYlJ6ynh/Xr16Nv375QqVSYNGkSnJycEBQUJKTr6OgIHx8ftGnTBjo6OpgyZYqgUIYP\nH46UlBTo6+tj48aNMDExAQDMmzcPI0eORHBwMOzs7PDTTz+9SflrNTExbB/Kn38CEsl/1yUS4MgR\nwNOT2fzq37/qZORwOLWHMm9sJCKoVCro6VVvu5Jv08bG4pgyBXj9Gti5s3DYn3+yDZFnzwKtWqmH\nhYf/1zNZtAgICGD/e3vz4S8OpzbzJm1niQplzZo1eP/99yGRSDB58mRcu3YN3377Lfr27VuuDCsL\nrlAYCgXg5gbMm8eWDRckJAT48kvmbrh+fc1pBAayg8Ph1H60ulM+ODgYJiYmOHnyJFJTU7Fr1y7M\nmzevXJlxKh9DQ+Z3fs4czX7nx40DRo0C3n2X9WQ4HA6nvJSoUPI01bFjx+Dr6wsXFxetC8WpWFq3\nBhYvBqZN0xz+9deAhQXw4YeApg8TPsTF4XBKQ4lDXhMmTEBycjIePnyImzdvQqVSoXv37vjrr78q\nS8ZywYe81CEC/vmHWSfWRGYm0K0bswf26aeVKxuHw6k+aHUORaVS4ebNm2jWrBnMzMyQkpKCxMRE\nuLq6livDyoIrlLKTkMAMTW7ezHbUczictw+tKhQAOHz4MM6dOweAbYIZWANaG65QyselS8CAAcCZ\nM2yojMPhvF1odVJ+3rx5WLt2LVq1agVnZ2esXbsW8+fPL1dmnOqPuzuwZg0waBDwv72lHA6HUypK\n7KG0bt0aN27cEGxpqVQqtG3bFn///XelCFheeA+leIjYbvlOnTSHf/EF66WcOQPUrVu5snE4nKpD\nqz0UkUik5sQqLS2tVKZXONWb1FS2qTEiQnP4okVsAn/qVM0rvzgcDqcgJW53nz9/Ptq3b4/u3buD\niBAREYFvv/22MmTjaBFzc2DbNsDXF7h+nZ3nR0cH2LEDaNIEsLcHHB2Z90czs6qRl8PhVH9KNSmf\nnJyMK1euQCQSoWPHjnj06BE8PDwqQ75yw4e8SsfcucD9+8Dhw5otD3t6Mp/1ADBiBMBNp3E4tRut\nr/IqSOPGjdVM01dHuEIpHdnZzPnWuHHAxx8XDn/nHeDECbbjfvhwIDgYqOZm3Dgczhug1TkUTu2m\nTh1g714gLAzIzS0cvns365ncu8c2Rg4dCrx8Wflycjic6g/voXBKjVIJTJ7MlMvRo0Ubk+RwODWX\nN2k7ixy8KG7zYkpKSrky49Rs9PWB7dvZkuLOnVmvxt6+qqXicDjVhSJ7KCX5GK7u/tl5D0W7bN4M\nfPUVm8zv2LGqpeFwOBVFpU/K1wS4QtE+hw+zIbAdO9jkPYfDqfloVaG0bt26UAampqbo2LEjFi5c\nCPOCGxiqCVyhlJ+sLOZQ66uvSt4l/+efbKJ+yRJg4sRKEY/D4WgRrSqUuXPnQk9PD2PGjAERYe/e\nvVAoFLC0tMQff/yBX3/9tVwZaxuuUMoPEXO4ZWcHfP99yfHv3wf69QP8/Jj3R25IgcOpuWhVobRr\n1w7Xr1/XeK1169bV1qYXVyhvRmoq0K4dsGEDsz5cEv/8A/TvD7RvD2zaxPeqcDg1Fa3uQ1GpVLh0\n6ZJwfvnyZeT+b8OCHm81ai316gGhoWyOJCmp5PiWlkB4OPOpMmRIzdqrUsL6Ew6HU0pK5VN+0qRJ\nsLOzg52dHSZNmoQffvgBL1++5GbsazlduwIzZjCf81lZJceXSIBff2XuhLt3L7/5+8pu4LlC4XAq\nhhK7GB07dsTt27fx4sULAGxCPo+RI0dqTzJOtWD+fLaBsbQm7PX1mdHJL78EunRhZlscHMqWZ3g4\n92PP4dRESuyhpKWl4ZNPPkGPHj3Qo0cPzJkzR1AuJREWFgZHR0c0b94cy5Yt0xgnPDwc7dq1g4uL\ni9relqVLl6JVq1Zo3bo1xowZg9evXwNgQ26dOnVCu3bt0LFjR1y5cqVUsnDKh64uMG0asz5cWkQi\nYPFi5pvey4v5XaluhIezlWyBgcxUf97/vLfC4bwBVAJDhw6lL7/8kmJjYykmJoYCAgJo6NChJd1G\nOTk5ZG9vT3FxcZSdnU2urq4UFRWlFkcul5OzszMlJCQQEdHz58+JiCguLo6aNm1Kr169IiKikSNH\n0vbt24mISCaTUVhYGBERHT9+nLy9vTXmX4qiVSxnz1ZufjWEw4eJLCyIjh4tPt7Zs0QBAewA/vu/\nMh5rQID28+Bwagpv0naWOOQVGxuLAwcOCOeBgYFwdXUtUVFdvnwZDg4OsLOzAwCMGjUKhw8fhpOT\nkxBn9+7dGDZsGGxsbAAAFhYWAAATExPo6+tDoVBAV1cXCoUC1tbWAIBGjRoJPaS0tDThepXzlo3T\nZGczw5IlMWgQm1cZMgT4+ms2ya8Jb2/1xxcYWAFCcjicSqXEgQyxWIzIyEjh/Pz58zA0NCwx4aSk\nJDRu3Fg4t7GxQVKB5UIPHjxAamoqunfvDjc3N+zatQsAUK9ePcyZMwe2trawsrKCmZkZevXqBQD4\n9ttvhbC5c+di6dKlpSuptsjNBfbt02yqtxYzYgSwfHnpvDl6eADnzgFLlzJFUd1Wc79F3wEcjlYp\nsYeyefNmjB8/XugVSKVS7Nixo8SES+MmWKlU4tq1azh9+jQUCgU8PT3h4eEBHR0drF69GvHx8TA1\nNcWIESMQGhqKsWPHYtKkSVi7di2GDh2K/fv3Y+LEiTh16pTG9APzfeZ6e3tXvP2x8HDg7FlmMfHx\n4/8mGgp+btdCNmxg+07i44G1a0ved9KiBXDhArsnMZHZAivqnsp+dLX8VXE4xRIeHl6i7cbSUmpb\nXvlXea1evRqzZs0qNv7FixcRGBiIsLAwAGySXUdHB/7+/kKcZcuWISsrS2j4J0+eDB8fH+Tm5uLk\nyZPYunUrAGDXrl24ePEiNmzYABMTE6SnpwMAiAhmZmYaFwlU6sbG27eBTp2AuDigYcPKybMakJ7O\nnG4ZGAB79gBGRiXfk5nJejc6Osz7Y2nu4XA4lUelONgyNTUVlgyvWrWqxPhubm548OAB4uPjkZ2d\njX379mHQoEFqcQYPHozz589DpVJBoVDg0qVLcHZ2RsuWLXHx4kVkZWWBiPD777/D2dkZAODg4ICI\niAgAwJkzZ9CiRYtSF1ZruLiwbeVz51a1JJWKiQlw7BjzR+/tDWRklHyPsTFw5AjTu2+yV6UmU9v3\n2fCVcm8vWvPYqKenh/Xr16Nv375wdnbGe++9BycnJwQFBSEoKAgA4OjoCB8fH7Rp0wbu7u6YMmUK\nnJ2d4erqivHjx8PNzQ1t2rQBAEydOhUAsGXLFnz22Wdo27YtFi5ciC1btmirCGVj4UL2S/qfsntb\n0NcHfvwR+PxzpixKe09wMODjw/yqxMRoV8bqRm1v4Gt7fpxiKM/SMBsbm3IvK6ssylm0N+PAASIn\nJ6LXrys/7xpKUBCRpSXRpUtVLUnlUdnLlHl+nLLwJm1nkVOpxsbGRU6sKxQKLam3Gs6QIcDWrcxE\nb765Ik7RTJ0KNGrEJuu3bSudIcqaSHj4f1/Sixb9d11b6zdqe36c6gl3sFXRPHzIJuj/+gto0qTy\n869GpKUBpqalM2d/6RLTx199BUyZUvo8cnKYIcq8IzOz+P8LXrt8GXj9GmjcmJmJqYw1FXm78iuL\n2phfQQUWEMD+5wrszdGKT3lOOWnWDJg5E5g1Czh4sKqlqVImT2ZWiNesYSZcisPdne1V6dCBTUeJ\nxcw4ZU5O8UpCqWQrxfIOY+PC/+e/1rChenhMDPD338C//zKl4ucHjBsHdOtWNnMznMqFb4StnnCF\nog0++wxo3Ro4erT2juGUguBgYNgw5qxr9+6Slwg3b84WzP35Jzu3sADmzCleSRgYvJlDr+BgplDc\n3NiQ2/HjwEcfsd7VmDFMubi4lD99TdT2fTa8h/D2woe8tMWpU2yC4M4doBSWBWor2dlsCOvuXWaC\npaQhpXfeYUNPbm7sEZqZaVe+tDT2mrZsUc/r1i3mD2b3buYbZtw4YPRo4H9WgjjViLfM6pHW0arH\nxppKlSsUgDkScXBgRqzeYojYkERICHD1KiCVFh23qAa+qsjNZUNxoaHAL78AbdsCY8eynld1kI/D\nqWi0olCKW+UlEomE3erVlWqhUJKTgTZtgD/+AFq2rFpZqgEXLzK7XjWVV6/YkFhICHD6NNCnD1Mu\n/fqV3l8Mh1Pd0WoPZeHChbCyssK4ceMAAKGhoUhOTsbixYvLlWFlUWkKJTcX6NGDfYYbGbExkvyf\nrqtXs7mUU6febLCfU62Qy4Gff2bK5c4d1mMZN445FeOT+ZyajFYVSps2bXDr1q0Sr1U3Kk2hELEx\nnDx7YiNGMCNVeeTksAkBf382CM+pdTx6xGyZhYQw8zNjx7KjVauqlozDKTtateVlZGSEkJAQqFQq\nqFQqhIaGwri0NjbeBkQiNqwFAK6ubPA/P3p6wKZNzH1hKT1dvk0kJQEqVVVL8WY0aQLMm8dWix05\nwpYy9+3LzLutXMnKyOG8DZTYQ4mLi8PMmTNx4cIFAECXLl2wZs0awXFWdaVS51DS0phRqhYtgEOH\nNMeZMoWt9lqzpnJkqiFMmMCsFoeE1K7FcCoVM+sWGgocOMD21+RN5puYVLV0HE7R8FVeGqj0Sfms\nLLZhYcMGZvWwICkpgLMzEBbGPl05ANiy4kmT2AbDI0eA+vWrWqKKJyuLWWUOCWGT+nXrso2ezZqx\n+RYiNhWX/6+ma+X5m5nJ/urosJHZOnVY3np6hf9qulbWuGfPso64WAxMnAg0aMD2DBV1GBnxOafq\nhlYVyv379zF9+nT8888/uHPnDm7duoUjR45g4cKF5cqwsqiSVV5hYcD//R/zjyIWFw4PDgZ++IF5\nmuK/IgEi4IsvmOPL48fZBsc35fVr5vMsPZ3Na2RksP8NDYGhQwvHz81lo5faXjfRpQt7/QDQsyfz\nYqmjw/It6m9xYSX9HTTov/wGDAA2bmTTeiqV5r/FhZUmztq1zPoQANjbM4sDmZlFHy9fsp9KcUpH\n0yGRsL9BQey9aloPwykfWlUoXl5eWLFiBT744ANcv34dRAQXFxfcuXOnXBlWFlW2bHjkSLZEWNMq\nuNxc9gvz82ObLThq/PAD8OWXbC7CwkI97J9/gHXrCisIKytAkwPRO3eAwYNZw2Niwv5KJICTE8uj\nIAcPMqXm58eGpqystFPGyt64Wd3zy81lPbj8SiYjo3gllP/4/Xf28QAUXg/DKR9aVShubm64evUq\n2rVrh+vXrwMA2rZtixs3bpQrw8qiyhRKUhLb/RYZCTg6Fg6/dQvo1Yu1eLVxfOcNiYlhe0EL8uwZ\nUzgFFUT9+hUzgpibC5w/D+zcyTYwursz5TJkiObOZnmp7I2btT2/zp2ZqZ7KUphvA1pVKP369cO6\ndeswYsQIXL9+HT///DOCg4Nx4sSJcmVYWVTpxsY1a4DDh9nuN01jKLNns1/ejz9WvmycElEo2NqK\nnTuBDz5gSoVTPUlLY3NRGzbwVfkVhVYVSmxsLKZOnYo///wTZmZmaNq0KUJDQ/kqr+LIyWEm7GfP\nZrvdCpKRwcZe9u5lJnU5HE65CQlhyv/kyaqWpHag1X0oOjo6OH36NJ49e4Z79+7hjz/+qHqTJtUd\nPT1g82bmY14uLxwukTAnXB9+yDYtcGoUL14AvXuzDmY1t0D0VjBiBJt3i4qqakk4JSqUd999FwCz\n7WXyvwX0w4cP165UtYFOndhyogULNIcPH85mfteurVy5OG+MoSEwfTpb5mxryybxT56s+Rs0ayp1\n6wLTpgHr11e1JJwih7zu3r2LqKgozJ07FytXrgQRCUYhV6xYwVd5lYa0NLb35OBBNstbkAcPAE9P\n4MYNbhe9hvL8OTO7snMne8UbNlS1RG8nT56wn1pcHJ+Yf1O0Mody+PBhHDx4EL/++isGDRokXJdI\nJFCuR8sAACAASURBVBg1ahQ6d+5cPmkriUpVKK9f/7e7qyChocz+xpUrmsMDAlhfff9+7cvJ0Sqv\nX3Orw1XJ2LFstdcnn1S1JDUbrU7KX7hwodorD01UqkKZNo3txvv008JhRGyZ8MCBzC1wQUraYc+p\n8XzzDXvF77wD6OtXtTS1l0uXmJfN6OiSXU5zikarCiUrKwvBwcGIiopCVlaW4CPlx2q+5LVSFUpM\nDHP08ddfzFJgQe7fZ1ukixraOnECmDGj6B32nBrNtm1sAv/+fba0VSZjJlA0eYfOzmZm8fPMr+Qd\n+vqal8W+fs3S1xR/2rTC8ZVK1hnW12cdZn19dojFgJdX4fi5ucziQF68/Ed17I25uwMLF7LvN075\n0KpCGT58OJycnBAaGoqAgACEhITAyckJa0sxmRwWFoZZs2ZBpVJh8uTJ8Pf3LxQnPDwcn3zyCZRK\nJSwsLBAeHg4AWLp0KUJCQqCjo4PWrVtj27ZtqPu/Grxu3Tps3LgRurq66N+/P5YtW1a4YJU9h/LN\nN8yD1JEjmveefPkl84Nb1NDW8OHsMzYwUKticqqO2Fg213Ljxn+mQgqiUDDbZnkmV/IOQ0NmZqQg\nWVms41swvljMRlqLSj8nhymXvKNuXeaiuSDp6UDr1upxlUom/7NnmuO3aaO++dTEhNn00jRprlQy\n0zD540okTP7ymMEJCWGWE06dKvu9HMYbtZ1UAq6urkRE1Lp1ayIiys7Opk6dOpV0G+Xk5JC9vT3F\nxcVRdnY2ubq6UlRUlFocuVxOzs7OlJCQQEREz58/JyKiuLg4atq0Kb169YqIiEaOHEnbt28nIqIz\nZ85Qr169KDs7m4iInj17pjH/UhStYnn9msjJieiXXzSHKxRE9vZEx49rDn/8mMjcnCg6Wnsycjha\nJieH6OFDops3ic6fZ9V93z6i0FDN8VNTibp1I2rblqhZM6L69YkMDIgaNdIcPy2N6P33iT7+mOjz\nz4lWrGBp5PHqFZGlJdGdOxVftreFN2k7NcwSq1OnTh0AgKmpKf7++29YWlri+fPnJSqqy5cvw8HB\nQdgAOWrUKBw+fBhOTk5CnN27d2PYsGGw+d8wkMX/DDiZmJhAX18fCoUCurq6UCgUsLa2BgBs2rQJ\n8+fPh/7/BqPrVxfzJXXqsL0nY8cyZxhGRurhYjGbJ5k+XfPQVuPGzKnGRx+xITDu3ZFTA9HVBZo2\nLX18qRQ4d67w9ZwczfH19Nhe4DybbsePs61e33zDwvMvId64sezyc96MEvehTJkyBampqfj6668x\naNAgODs747PPPisx4aSkJDRu3Fg4t7GxQVIBT0MPHjxAamoqunfvDjc3N+zatQsAUK9ePcyZMwe2\ntrawsrKCqakpevXqJdxz7tw5eHh4wNvbG1evXi1TgbWKlxdzflGUY4++fdkylLzaX5CZM5ktsJ9/\n1p6MHE4NQNOCSIB9p02cyIb5vviCKY3du9lcTx7TpjEjFGlplSMr5z9K7KFMmTIFACCTyRAXF1fq\nhEWl+MJWKpW4du0aTp8+DYVCAU9PT3h4eEBHRwerV69GfHw8TE1NMWLECISGhmLs2LHIycmBXC7H\nxYsXceXKFYwcORIP8+xlFyAw33yEt7c3vL29Sy1/uenYsfjw779nnh3HjmXmV/Kjr8+8O44axVZ8\nSSTak5PDqQU4OjJ7q/m9QTRqBPTrxxZCzJ5ddbLVFMLDw4W56zelSIWyatUq4f+8SZr8SmJ2CW/K\n2toaCQkJwnlCQoIwtJVH48aNYWFhAbFYDLFYDC8vL9y8eRO5ubno3LkzzM3NAbDd+hcuXMDYsWNh\nY2Mj7N7v2LEjdHR0kJKSIsTNT2B1nOC2smIT9NOnA2fOFB7a6tqV2fUIDATyvQMOh6MZTd9dH3/M\nlhDPnMmXEJdEwY/tRYsWlTutIoe8MjIykJmZib/++gubNm1CcnIyEhMTsXnzZly7dq3EhN3c3PDg\nwQPEx8cjOzsb+/btU9sgCQCDBw/G+fPnoVKpoFAocOnSJTg7O6Nly5a4ePEisrKyQET4/fff4ezs\nDAAYMmQIzpw5AwCIjo5Gdna2RmVSrZk+/T+/t5pYvhzYtYt9enE4nDLj7s586hw/XtWSvGWUNGvf\ntWtXSk9PF87T09Opa9eupZrxP378OLVo0YLs7e1pyZIlRES0efNm2rx5sxBnxYoV5OzsTC4uLrRm\nzRrh+rJly4Tr48ePF1Z1ZWdn07hx48jFxYXat29PZ8+e1Zh3KYpWtVy+TNSwIVFKiubwoCCizp2J\nVKrKlYvDqSWEhBD16lXVUtQ83qTtLHEfSsuWLXHz5k0YGBgAAF69egVXV1fcv3+/EtRd+akWtrzO\nnQO2bmWbDzQxYwZbzrJ5c+Gw3Fxm52vaNDYLyeFwykR2NttnfPo0s/PFKR1aNV8/fvx4dOrUCYGB\ngQj4//buPC7Kav8D+GcGGBg3FHFlUBQXFjcQ1+s1FFHTm7mWYEGG6LXING9l/W5XbVPLulrWvXTL\nNbd2zRBNi6LMJVE0tdwgcc1E3AaY7fz++DowMzzsz8www/f9ej0vZ545M895BJ7vnOec8z3z56Nf\nv35ITEys0cHqnT59aNbWV19Jv/7yyzQR8qefyr6mVFIH/XPPAdeu2beejLmBffuAHTtKn6tUnIXY\n0SptoQDAwYMHkZmZCYVCgcGDByNCjjVX7axOtFAAmrI7fTrNPbGdmwJQqtolS4Cff5YeKzlrFlBU\nRGuqMsbKlZ5Oq0UcPFg61uXSJSA8HDh7lrMQV5VdUq/cvHkTTZo0QX5+PgCUHMA80svPz69GB3SU\nOhNQABpuEhhIgcOWEMDw4ZQ5UCpN6o0b1F7/9FPKF8YYk2QyUY7WDRusV4t46CEgMpKHEFeVXQLK\n6NGj8dVXXyEoKEhyTkl15qQ4Q50KKJcvU0Kkb76hf22dPAkMHAgcOkSBx1ZlrRjGGAAaIHn8OLB6\ndem+ffsoseapUzyEuCrsmhzSVdWpgAJQP0pICBAcLP36ggW0jumnn5Z9zZwCf8wYGljPGJN09Sq1\nUs6eBSxvonAW4qqzS0CpbK5JZGRkjQ7oKHUuoFSmqIhaL8uWAaNHl339119p0uORIzQ5kjEm6aGH\naDyM5Xev9eup1cJZiCtnl4ASHR1dYfqUb7/9tkYHdBSXCygALUw+YwZw7Jh0PrD/+z/66rVxI5CR\nATgilQxjLubKFUqDb5l/lYcQVx3f8pLgkgEFoJu9HTtKJ5DUamnIynvvAT/+yGunMFYNCxZQsPnP\nf5xdk7rN7gHl6NGjOHHiBIqKikr2JSQk1OiAjuKyAeXSJVqh6LvvpL9KbdtGqVZHjwaWL3d8/Rhz\nUTyEuGrsOrFxwYIFmDVrFlJSUvDtt9/imWeewdatW2t0MHZXURHl87II0CXatAHmz6fXbX+oGRk0\n0is4GHjrLVpDdv582s8Yq1CbNjQ6v46vXu7SKm2hdOvWDdnZ2YiMjER2djauXLmCKVOmYNeuXY6q\nY43U+RbKhAm05K9UZk+jkYalzJoFlNcSfPxxYP9+Gsry/vvSw40ZY1b276fVIXgIcfns2kJRq9Xw\n8PCAp6cnbty4gZYtW1qlpWc1tHw5rQ4klRPNw4Pyez3zDHB3YmkZLVpQWpdBg2jW1tq1ZVs0jNVj\nQlDSbsvVH/v2pT+d8rIhsdqpNKBERUXh+vXrSE5ORlRUFCIiIjBw4EBH1M29aTS05Nzf/y4dCKKi\ngEmTaFlgKdHRtCDXCy/Q6LClS4GxY2kSJWMMCgV9L/vyS+v9s2YBb7/tnDq5u3JveT322GOIj4/H\noEGDSvbl5OTg5s2b6Nmzp8MqWFN1/pYXUPmtLXPalY8/ppn0FSkuBl58EfjgA/prmTTJPnVmzIV8\n+CE13nfuLN3HQ4grZpdbXl26dMHTTz+N9u3b45lnnsGhQ4fQoUMHlwgmLsPDA0hNBf74Q/p1X19a\ntfHvfwf0+oo/y9ubhhpv2UKtlsmTOUsxq/cmTgQOH6Y+EzOViv6kuJUiv0o75XNzc7Fp0yZs3rwZ\nWq0W8fHxiIuLQ5cuXRxVxxpxiRZKVQgBjBhB29y5VXtPYSFNgty0iQIW55tg9dgzz1DiyKVLS/dd\nvgyEhtIQ4mbNnFe3ushhExsPHTqEqVOn4ujRozAajTU6oKO4TUAB6OvVgAHlJ48sz/ffA488Atxz\nD6V08fW1WxUZq6vOnKFE3efOWc+e5yzE0uw6ystgMGDr1q2Ij4/HyJEjERISgs8++6xGB2M11Lkz\n8MQT1U8MOXgw5f7y8aE8YZzIiNVDwcE0H1ilst4/axYtvlXHvxu7lHJbKDt37sSmTZvw1VdfoW/f\nvoiLi8OYMWPQqFEjR9exRtyqhQLQJMgePYA336QJjdW1cycwbRq997XXABf5OTJmT/3706JcY8Y4\nuyZ1h11ueQ0dOhRxcXGYMGFCnV9MS4rLBpT0dGqRSKW537ULSE4uP3lkZQoKKG1LZialXv3rX2td\nXcZc2fr1wKpV9KfFCCeHlOCyAeX11+m3Oz29dB1TS/HxtIJQ06YUVDZsqH5ioq1baZhLXByta295\nY5mxeoSHEJfFAUWCywYUvR7o3Zva4ZMnl3398mWgXbvSYcSTJgEffVT94/z5J6VvOXIEWLOGphAz\nVg8tXEh/VpyFmHBAkeCyAQUAfvqJcn2ZWyK2YmJoOWEfHyAtDRgypObH2ryZeieTk4F//atszyVj\nbuTaNRrtFRFRuo+HEFuzyyivc+fOlfumzMzMKn14eno6QkJC0LlzZyxZskSyTEZGBiIiItCtWzdE\nWywYtWjRIoSHh6N79+6Ij49HcXGx1fveeOMNKJVK5JeX68qVDRhAc0eef1769U8/pZbJW29RK+aF\nF6jtXhMPPghkZ1NLpU8fesyYm8rOBh5+2DrbUevWtBoEZyGWgShHhw4dxOLFi4XBYCjZd+nSJTFl\nyhQRGRlZ3ttKGAwGERwcLHJycoROpxM9e/YUx48ftypz/fp1ERYWJvLy8oQQQly9elUIIUROTo7o\n0KGDKCoqEkII8cADD4jVq1eXvO/cuXNixIgRIigoSFy7dk3y+BWcmmvIzxdiwAAhbt+uuNzFi0Lc\nd58QPXoIkZVV8+OZTEKsXi2Ev78QL78shF5f889irI4ymYTo2lWI77+33r9vnxAdOghhcbmrt2pz\n7Sy3hXLw4EGcPXsWvXr1wu7du7Fs2TL069cP/fv3x4EDByoNVPv370enTp0QFBQELy8vTJ48GVu2\nbLEqs2HDBkyYMAEajQYA4O/vDwBo0qQJvLy8oNVqYTAYoNVqERAQUPK+p556Cq+99loNwqcLadaM\nVmVs2LDicm3aULqVuXNpNv3ChZWnaZGiUACJiUBWFi3uNXAgcOIEvcbrrTA3oVDQeBTb/pK+fYGW\nLTkLcW2VG1CaNWuG1NRUJCUlITY2FkuXLsUPP/yAlJQUKJWVzofEhQsXEGgxq1uj0eDChQtWZU6d\nOoX8/HwMGTIEUVFRWLduHQDAz88Pc+fORbt27dC2bVs0bdoUw4YNAwBs2bIFGo0GPXr0qNEJuxSp\nUV7llUtIoJn0+/ZRwskjR2p2zMBAYMcO4NFHaWLkm29Sfw1jbiIxkboebVPozZpFd5FZzXmW98L1\n69cxb9487N27F9u3b8f27dtx7733Yvny5YiJian0gxVVuBjq9XpkZWVh9+7d0Gq1GDBgAPr37w+l\nUolly5YhNzcXvr6+mDRpEtavX49x48bh1VdfxdcWM75FBZ1HCyzWXI+Ojrbqo3FLAQH0FWvVKuq4\nnz0bePZZwLPcH7M089e44cMpdcvJk8DVqzQ3JjiY1rwPDubJkcwlNWsGjB9PfSaWq0NMnEgN/WPH\naKnguiIjg1arsN/nZyBDprsQ5V5pevfujZkzZ+Kdd96Bp6cnRowYgcOHD2PmzJl4//33sXHjxgo/\nOCAgwGohrry8vJJbW2aBgYHw9/eHWq2GWq3G4MGDkZ2dDZPJhIEDB6J58+YAgPHjx2PPnj3o2bMn\ncnNzSzIenz9/Hr1798b+/fvRsmXLMnWwDCj1hkJBrYthw2hm/Bdf0CTG6v6FZGTQNmQITYT8/Xdq\nASmVlFY/Jwdo3Lg0uNgGm9atq97CYszBnnsOuH3bep85C/GKFXVrCHHu6gy7RhTbL9sLpVaRrary\nOlfOnTsnud9kMonU1NRKO2f0er3o2LGjyMnJEcXFxZKd8idOnBAxMTHCYDCIO3fuiG7duoljx46J\nw4cPi/DwcKHVaoXJZBIJCQlixYoVZY7h1p3ytrRaIY4erd57TCYhUlOFaN5ciMWLa97RPn9+2X1G\noxAXLlDv5urVQrzwghDx8UL07y9EixZCNGggRHi4EGPGCDFnjhArVgiRlibEb78JUVxc8fG+/bZm\n9WSsli5dEqJpUxoTU55D//7WYfURQohv75nv0OPV5tpZbgslsJystgqFAtOnT680UHl6emLFihUY\nMWIEjEYjkpKSEBoaitTUVADAjBkzEBISgpEjR6JHjx5QKpVITk5G2N3pqgkJCYiKioJSqURkZKTk\nMatyW81tHDxIs+SPHaOWQVUoFMD06XTrKikJ+Pxzaq2EhNS+Pkol0LYtbVIpXG7epIH9Z89Sutej\nR2nwwJkzwPnzNJjAsnVj+djebXxbjj4ek5eMP7/WrSnd3cqV5a8WUfBFBjC7kuMZDMCdO8CtW9QU\nstyk9tnsv3HxNoqu3oZKdwsHCiYiYwF9bHR03f5V5YmNruSRRwA/P+oory6TidZGeeEFau/Pnk0L\nfFWF3BdcvZ5ml5mDzZkz1o91OsDfn/poKtoaNqy8jLlcRee6YAFtjuLuAdPRx6vtz08IWkPo1i3g\n1i38svc2Xnz6Fja+dwse2lulF/u7F/wL675BwLDQioOETif9u9i4cdX2nTxJE5tVKhopMH8+1dUB\nEYVnyktwy4Dy55/UF5Kebj3VtzrOnqU+Fr2eOu/rykJp5j4bIWgp4zlz6I+ye3dqUUl9o7Pd7twp\nf79KVX7AOXOGxo2q1ZR9oLb/VhaoHR3A3Ol4ej1l3rbcli4FpkwpvaibL/5VeW7eZ/79aNwYaNwY\n2Wcbo03nRmgZTM+vnr4B7e9XYfRQoWPObpzoej+MHir4DIpCp8RBZX+n1GrZ+hAzohcg2txEcYDa\nXDurOfyHOZW/P7BoEXW2f/01tVaqq2NHGgb8zjs01+SFF2itlSoMBbcry29eCoW8FyTzN1DLIPP9\n9zTEWqej23Ht29Ntinbt6KJy6xaNbCsspIuW+V/Lx+X96+FRccDJy6MBDiqV9ObtXf5rNSlXVATk\n51MrVWoTomavlff6qVN0e9X2wm/5f1jRvor2C0H/jx4e9NjTE7h+Hdi+nc5Vo6HbpuZv/c2bA0FB\npc/vBgzL52euNEJgRy+rrEPHNgBzVwK7PqbnLSx+nRx9gQ8Kctihao1bKK5GCLq56+9ffmqWqjp9\nmm6jKZV007hTJ1mqWGuu/I1aCOtv0eYLY2YmsHcvvbZ2LaXMMRqphRgcTIGtvK24uPqv3bxJLTOj\nkY7v40OBWqWix0olbQpF6WPbrTqv3bxJo/8UCiA3F+jalS72Gg3QoQMd07yZA6vtVpX9UkPga/nz\ni4mhrsYHHyzdp9PRhfzrr8sOkHR0QHH0LcRaXTtrOSCgznLjUyMmkzyfYzAI8eabNBLs7bdp9Jaz\nOXqUl9QoNj5evTneRx8Jcc89ZfcvWCDEjBll9zt6lJej1eba6eT7HKzG5Brh5uFB/RU//kirDcXE\n0BwTZ3L0MJa6PGyGVa6WP7+xY0v7wC3NmEHJuK9ft97fq7IRXvUYBxR3kpVF+blromtX4IcfgFGj\nKOvwf/9rnZLVnbl7AOPjVcjLi0bV//e/1vsthxCzquE+FHeyaBHw9tvU4T5uXM0/5/hx6lvx9QXe\nf586rBlzY3l5QK9eNJrdMh/rgQPAAw9Qd2NVR9m7Orush8Jc0HPPAR9/TPm74uJomHFNhIUBe/YA\nQ4dSyyU4mEaEnT5df1otrF4JDATeeIPGOFjq0wdo1QrYts059XI13EJxR1otDQfesAH44AO6jVVT\nUVE0Sx+gewM+PjT02HIzz3Rv355XfGRuZ8MGuu21a5eza+IYPLFRQr0OKGZ79tAw1XvuqflnjBpF\nY/yjomgMpRDUaW+e3W6ZXuXCBbrxbA4wtgHHz48TRjKXU9EQYnfEAUUCBxSZFBTQIP333pNe396S\nXk83o80BxjbgCGEdYCwDTrt21AJirA568UXg4sWyHffuiAOKBA4odYwQNP7SMsBYBpyLFynRZHEx\nzbZWq6kfqE0bCmRNm9IgAfPjpk1ptjO3eJgDXL4MhIbSr2qzZs6ujX1xQJHAAaUCS5dSG37iRGfX\npJReT2uujB9PqVAA+guOiaFWktRWWAg0aWIdZGw32yBkG5CcnXKG1TlC0Oj7uyuSl3j4YRoJVl4W\n4powmajL884dm+2mEUU3iuHxxEy0KsyFQdUQXX7eAN/2ldwlkAEHFAkcUCrw00/A1KlAjx40xLhF\ni8rf4yi2fTYV3WYzGCjdR3kBp6LXCgroL9fDg4KKtzfQuzcN6Wne3Hrz87N+7uvLLSN3ZTLh0I9a\nPJZ4Bz/uuA2ltjT/2+nDt3Hx+bfR3vMijEpPaMdNAZRKGLU6GAt1MBUWw1ikgyiilDjibhochU4H\npb4YSoMOSoMOHsZieBh18DTpoBLF8FbooIIO3tBBhWJ4CR0UENArveFh0sETJgDAHs0kDMz7yO7/\nBRxQJHBAqURhIeU/WruW0mNPmuTsGpHq9NnUlsFAAxb27KHngwbRkn3Xrllv+fnWz7Vauu9RWeCx\nfa5W2/d86guTqTSPWUoKJaP08qIU7wpFxVmnK9un1UKo1cjXNYK6RSM0aGG9TMIfX/yIloIWoz+j\n7IzDHcdD4aOC0scbSh8VPNQqeDTwhmcDFTwbqODVyBteDVVQNVLBu4k3vBur4N1EBXUTFXx8veHT\nRAWlj02iT2/vkkkvB1qMQp8/t+NYgyhojn/NLRRn4YBSRfv20STGpCTgH/9wdm0crzotIjOdjvqD\nKgo6ts+vXaOLhEJRmiW3a1e6WPn40EXEcrPdV5UyUvvmzaMb/z4+wLvv0oXRaKx8MxiqVs627KpV\nwKVLdH5xcdT6s01kaZnQsqJ95b1uMJRegAsL6TlAQb5Xr8rXy6lof4MGgIcH/vMfGib86afWP3pH\nX+Bv/F6AY4OmI/yH9xwSTAAOKJI4oFRDURF9U2ve3Nk1cTxHtYiEoJZNTAwFcQAYMgT45z/pYmm5\nFRXJt+/qVbrQA3QB9vOjwFbR5ulZeZnyyqenA1eu0PG6dKF+Ottv37aPq7rP/NjTs/SWY02+EFTB\nrVs08PCXX4CAgNL9zrjAOxoHFAkcUFidZKcLYL09nh2/EMycSdOqzIsl1hccUCRwQJHB7dt0G4A7\noOXjyD6i+nA8Ozp2jJawSUpydk0ciwOKBA4oMkhKogvEu+/S6CfGmNvj5JDMPt55pzQ5ZGgoMHo0\njarR651dM8ZYHcQtFFa5O3cof1dODk0+TEkpW0anAxITablXy7QqGo30sq2MsTqJb3lJ4IDiYMXF\nwCefWKdTOXuW+l/OnStb3mCgdcibNeM+GsbqkDodUNLT0zF79mwYjUZMmzYNzz77bJkyGRkZmDNn\nDvR6Pfz9/ZGRkQEAWLRoET788EMolUp0794dq1atgre3N55++mls27YNKpUKwcHBWLVqFXx9fa1P\njANK3WAwSLdQTp+mmekKRWmrpkMHICICmDLF8fVkrAJGY+n0IXdXZwOK0WhE165dsWvXLgQEBKBP\nnz7YuHEjQkNDS8oUFBTgL3/5C3bs2AGNRoM///wT/v7+yM3NxdChQ3HixAl4e3vjwQcfxKhRo5CY\nmIivv/4aMTExUCqVmDdvHgBg8eLF1ifGAaXuMyeMzMkpbdF4eEhPsDx0iG6p+ftbb+HhwIMPOr7u\nrF5JTKQpRAkJzq6J/dXm2mnXeLt//3506tQJQUFBAIDJkydjy5YtVgFlw4YNmDBhAjQaDQDA/25G\ntiZNmsDLywtarRYeHh7QarUIuDvDKDY2tuT9/fr1w6e201mZa1AoaKKdnx+1VioSEgKsW0erUFpu\nV69Kl8/MpJaOOfA0b07/9u5NmQEYq4bx44HFi+tHQKkNuwaUCxcuIDAwsOS5RqPBPvMs4btOnToF\nvV6PIUOG4NatW3jyySfx8MMPw8/PD3PnzkW7du2gVqsxYsQIDBs2rMwxVq5cibi4OHueBqsL1Gqg\nZ8+ql+/Xj4KKbQAqb82V9HTgscdoLVjLLSICGDBAnnNgLmv0aOCJJ4DDhym7C5Nm14CiqEJnq16v\nR1ZWFnbv3g2tVosBAwagf//+UCqVWLZsGXJzc+Hr64tJkyZh/fr1mGJxf/2VV16BSqVCfHy85Gcv\nWLCg5HF0dDSio6Nre0rMVahUtCRx+/ZVKz90KLBzJy0QZt6OHKFUJlIB5cAB4MsvywagJk3kPQ/m\nPEKUpCXyvHMHz4+9gw/faI5e6yTmZJ07RyuWmt9n1q4djXS0lZuLvB/PIT8fKC4SKCoGigqBPxsF\nYfDD7SXfYi8ZGRkl/da1ZdeAEhAQgLy8vJLneXl5Jbe2zAIDA+Hv7w+1Wg21Wo3BgwcjOzsbJpMJ\nAwcORPO7+aXGjx+PPXv2lASU1atXIy0tDbt37y73+JYBhbEKqVRAp060VYWPD92y++kn4KOPSoPQ\ntGnA8uVly1++TAmiNBrOOiwXIWjQR2Fh6UIivr7SyzH8+COQkWG96IhWC4wbJ51pe+FCWqbRy6sk\naeQ074aY9ftc/PbPZHTtalM+LQ3nXl6DWzcBowkwmhQwmYDVqum4/7NHUObmys6dKF7wIcQdQK0E\nGngo4OEB5IVOw837H5bpP6hqbL9sL1y4sMafZddOeYPBgK5du2L37t1o27Yt+vbtW6ZT/tdfX+QA\n0AAAChxJREFUf0VKSgp27NiB4uJi9OvXD5s3b4Zer8eUKVNw4MAB+Pj44JFHHkHfvn3x+OOPIz09\nHXPnzsV3331X0udS5sS4U545mhA06VOlKvva2rV0kTp/nl5XqykozZolPQhh61bg88+pjLmsjw8w\neDAg1dL+/XcKaJZl1Wq6wDZsWLb8tWuUxNFgoDobDLQFBNDia7aOHqWBEbbl+/QBBg4sW37HDmDb\nttKy5n/HjpW+gK9cCaSmUjnL9zz+ODBnTtnyL70E/OtfNIhDraZzbNiQykrNk9qxA/juu9Jy5q1X\nLyAsrGx5nY4yJdsM61q7ltKUSb3l558pW1GDBrSZkxf7+bnW6tZ1dpQXAGzfvr1k2HBSUhKee+45\npKamAgBmzJgBAFi6dClWrVoFpVKJ5ORkzJo1CwDw2muvYc2aNVAqlYiMjMT7778PLy8vdO7cGTqd\nDn5+fgCAAQMG4N1337U+MQ4orC4ymailYs4M3LChdJbno0fpClVURN/Ai4poGzQIGD68bPl162jB\nc9vyKSnA88+XLf/uu8CKFXTB9PSkK56nJw1YSE4uW/7zz2mzLT9yJHDvvWXL79sH7N9vXdbLC+je\nnRZ2s5WXR8tAm8uZ3+PvL73mrslELUSewyS7Oh1QnIUDCmOMVR/n8mKMMeZ0HFAYY4zJggMKY4wx\nWXBAYYwxJgsOKIwxxmTBAYUxxpgsOKAwxhiTBQcUxhhjsuCAwhhjTBYcUBhjjMmCAwpjjDFZcEBh\njDEmCw4ojDHGZMEBhTHGmCw4oDDGGJMFBxTGGGOy4IDCGGNMFhxQGGOMyYIDCmOMMVlwQGGMMSYL\nDiiMMcZkwQGFMcaYLOwaUNLT0xESEoLOnTtjyZIlkmUyMjIQERGBbt26ITo6umT/okWLEB4eju7d\nuyM+Ph7FxcUAgPz8fMTGxqJLly4YPnw4CgoK7HkKjDHGqshuAcVoNCIlJQXp6ek4fvw4Nm7ciBMn\nTliVKSgowOOPP44vv/wSv/zyCz755BMAQG5uLv73v/8hKysLR48ehdFoxKZNmwAAixcvRmxsLE6e\nPImYmBgsXrzYXqdQp2VkZDi7CnbjzucG8Pm5Onc+v9qem90Cyv79+9GpUycEBQXBy8sLkydPxpYt\nW6zKbNiwARMmTIBGowEA+Pv7AwCaNGkCLy8vaLVaGAwGaLVaBAQEAAC2bt2KxMREAEBiYiK++OIL\ne51Cnca/1K6Lz8+1ufP51dmAcuHCBQQGBpY812g0uHDhglWZU6dOIT8/H0OGDEFUVBTWrVsHAPDz\n88PcuXPRrl07tG3bFr6+vhg2bBgA4MqVK2jVqhUAoFWrVrhy5Yq9ToExxlg12C2gKBSKSsvo9Xpk\nZWUhLS0NO3bswEsvvYRTp07hzJkzWLZsGXJzc3Hx4kXcuXMH69evlzxGVY7DGGPM/jzt9cEBAQHI\ny8sreZ6Xl1dya8ssMDAQ/v7+UKvVUKvVGDx4MLKzs2EymTBw4EA0b94cADB+/Hjs2bMHU6ZMQatW\nrXD58mW0bt0aly5dQsuWLSWPHxwc7PbBZuHChc6ugt2487kBfH6uzl3Pr2fPniVdCjVht4ASFRWF\nU6dOITc3F23btsXmzZuxceNGqzL3338/UlJSYDQaUVxcjH379uGpp56CXq/Hiy++iMLCQvj4+GDX\nrl3o27cvAGDMmDFYs2YNnn32WaxZswZjx46VPP7p06ftdWqMMcYk2C2geHp6YsWKFRgxYgSMRiOS\nkpIQGhqK1NRUAMCMGTMQEhKCkSNHokePHlAqlUhOTkZYWBgAICEhAVFRUVAqlYiMjMT06dMBAPPm\nzcMDDzyADz74AEFBQfjoo4/sdQqMMcaqQSGEEM6uBGOMMdfndjPlqzKZ0lXl5eVhyJAhCA8PR7du\n3fDWW285u0p2YTQaERERgfvuu8/ZVZFdQUEBJk6ciNDQUISFhWHv3r3OrpJsypuM7KoeffRRtGrV\nCt27dy/Z504Tq6XO7+mnn0ZoaCh69uyJ8ePH48aNG9X6TLcKKFWZTOnKvLy88O9//xvHjh3D3r17\n8c4777jV+ZktX74cYWFhbjmo4sknn8SoUaNw4sQJHDlyBKGhoc6ukiwqmozsqqZOnYr09HSrfe40\nsVrq/IYPH45jx44hOzsbXbp0waJFi6r1mW4VUKoymdKVtW7dGr169QIANGrUCKGhobh48aKTayWv\n8+fPIy0tDdOmTYO73Y29ceMGMjMz8eijjwKgfkZfX18n10oeFU1GdlV//etf0axZM6t97jSxWur8\nYmNjoVRSWOjXrx/Onz9frc90q4BSlcmU7iI3NxeHDh1Cv379nF0VWc2ZMwevv/56yS+1O8nJyUGL\nFi0wdepUREZGIjk5GVqt1tnVkoXtZOSmTZuWTEZ2J/VpYvXKlSsxatSoar3Hrf5q3fEWiZTbt29j\n4sSJWL58ORo1auTs6shm27ZtaNmyJSIiItyudQIABoMBWVlZeOyxx5CVlYWGDRu69C0TS7aTkW/f\nvi05GdmduPPE6ldeeQUqlQrx8fHVep9bBZSqTKZ0dXq9HhMmTMBDDz1U7hwcV7Vnzx5s3boVHTp0\nQFxcHL755hskJCQ4u1qy0Wg00Gg06NOnDwBg4sSJyMrKcnKt5PHzzz+XTEb29PQsmYzsbswTqwFU\nOLHala1evRppaWk1+kLgVgHFcjKlTqfD5s2bMWbMGGdXSzZCCCQlJSEsLAyzZ892dnVk9+qrryIv\nLw85OTnYtGkThg4dirVr1zq7WrJp3bo1AgMDcfLkSQDArl27EB4e7uRaySMkJAR79+5FYWEhhBDY\ntWtXyZwyd2KeWA2gwonVrio9PR2vv/46tmzZAh8fn+p/gHAzaWlpokuXLiI4OFi8+uqrzq6OrDIz\nM4VCoRA9e/YUvXr1Er169RLbt293drXsIiMjQ9x3333OrobsDh8+LKKiokSPHj3EuHHjREFBgbOr\nJJslS5aIsLAw0a1bN5GQkCB0Op2zq1QrkydPFm3atBFeXl5Co9GIlStXimvXromYmBjRuXNnERsb\nK65fv+7sataY7fl98MEHolOnTqJdu3Yl15eZM2dW6zN5YiNjjDFZuNUtL8YYY87DAYUxxpgsOKAw\nxhiTBQcUxhhjsuCAwhhjTBYcUBhjjMmCAwpjDpKbm2uVKpwxd8MBhTHGmCw4oDDmBGfPnkVkZCQO\nHjzo7KowJhu7rSnPGJP222+/IS4uDmvWrOFbYMytcEBhzIH++OMPjB07Fp9//jlCQkKcXR3GZMW3\nvBhzoKZNm6J9+/bIzMx0dlUYkx23UBhzIJVKhc8++wwjRoxAo0aNEBcX5+wqMSYbDiiMOZBCoUCD\nBg2wbds2xMbGonHjxvjb3/7m7GoxJgtOX88YY0wW3IfCGGNMFhxQGGOMyYIDCmOMMVlwQGGMMSYL\nDiiMMcZkwQGFMcaYLDigMMYYkwUHFMYYY7L4f8ASa7PYShCaAAAAAElFTkSuQmCC\n",
"text": [
""
]
}
],
"prompt_number": 41
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is a very interesting plot. We can see that SVD approach does almost universally worse than the stepwise approach. Its not until we get to the last two features do we see them equal each other.
\n",
"My interpretation of this is that the SVD approach does not reduce the feature space with the outcome or learning task in mind. It is purely untangling the covariance structure of the matrix $X$. It is quite possible (based on the evidence above), that the most predictive features corresponded to the lowest singular values. That is why we don't see a large reduction in the cross-validated error until the last few columns are added into the matrix.
\n",
"\n",
"This is not to say or prove that dimensionality reduction with SVD is a generally inferior approach. Because the matrix $X$ is dense and not very high dimensional, the SVD approach just might not be appropriate here. It is also slower, as the cost of generating the SVD isn't always cheap.\n",
"\n",
"\n",
"
"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}
}
]
}