{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "#Regularization\n", "\n", "##Introductory Theory\n", "
Regularization is a technique for reducing the complexity of a model in supervised learning. The processing of performing regularization can be viewed with multiple interpretations:
\n",
"\n",
"Complexity Penalty
\n",
"\n",
"In the ERM framework, regularization can be viewed as the penalty or cost for adding extra complexity to a model. Usually, the complexity is a function of the weights learned for each feature. For now, we'll consider the class of linear models: $f(X) = W\\cdot X+t$. Remember in ERM we seek a function $f^{opt}$ such that:
\n",
"
As a starting point, we can visualize how regularization constrains our search space for possible optimum values of $\\beta$. We'll generate a high variance data set that represents a bournoulli process where $P(Y|X)$ is governed by the inverse logit. We'll then plot the following:\n", "
The next two plots show all of the above with different sized datasets. The first is relatively small, with 50 examples, and the 2nd has 50k examples. We know the true value of the data generating distribution is $\\beta=[1, 1]$. The dark green elipsoid shows where the log-loss is at a minimum. Its center is the optimum point of the data. With more data, we can see that this center gets closer to $[1, 1]$.
\n",
"The circular constraint boundaries basically say that the coordinates of $\\hat{\\beta}_{MAP}$ have to be within that boundary. The MAP estimator will found the lowest value of the log-loss (i.e., darker color) within the circle.
\n",
"\n",
"What we can learn here is that with a very loose regularization weight (bigger circle), we are more likely to choose $\\hat{\\beta}_{MAP}$ that fits the data well, but is far from the truth (which we know because we created the data generating distribution). With the right regulariztion strength (such as the red circle in the above plot), we can see that our MAP estimate is actually closer to the truth. The problem is less severe with more data, because the dark green elipsoid shifts closer to the true optimum point.
\n",
"\n",
"Note that with $L1$ regularization, we can do the same analysis, though the constraint boundaries would look different. Instead of circles they would be straight lines.\n",
"\n",
"
The above examples were a bit theoretical, and are mainly presented for explaining the underlying optimization process (and to reinforce the idea of the bias-variance tradeoff going on).
\n",
"Below we'll look at an actual dataset. First we'll read it in and rescale it. Rescaling is very important because the norm of $\\beta$ is going to be a function of the scale of each feature. Since we apply the same regularization factor to each feature, rescaling ensures that it has the same effect per feature. \n",
"\n",
"\n",
"
Next, we'll look at how a feature's weight evolves as we change the regularization weight.\n", "\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "L2 = {}\n", "L1 = {}\n", "\n", "for i in np.arange(-5, 5, 0.5):\n", " LR2 = linear_model.LogisticRegression(C=10**i, penalty = 'l2')\n", " LR2.fit(X_train, Y_train)\n", " L2[i] = LR2.coef_[0]\n", " LR1 = linear_model.LogisticRegression(C=10**i, penalty = 'l1')\n", " LR1.fit(X_train, Y_train)\n", " L1[i] = LR1.coef_[0]\n", "\n", "\n", "feats = X.columns.values\n", "Rpath2 = pd.DataFrame(L2, index=feats).transpose()\n", "Rpath1 = pd.DataFrame(L1, index=feats).transpose()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 20 }, { "cell_type": "markdown", "metadata": {}, "source": [ "This plot shows how the weights progress with $L2$ regularization. We can see a generally smooth progression that converges to a point as the regularization strength decreases.\n", "\n", "\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "plt.clf()\n", "fig = plt.figure()\n", "ax = plt.subplot(111)\n", "\n", "for f in feats:\n", " plt.plot(Rpath2.index.values, Rpath2[[f]], label=f)\n", "\n", "plt.xlim([-3, 2])\n", "plt.ylim([-1, 1])\n", "box = ax.get_position()\n", "ax.set_position([box.x0, box.y0, box.width*1.5, box.height * 1.5])\n", "\n", " # Put a legend below current axis\n", "ax.legend(loc='upper center', bbox_to_anchor=(1.15, 1), fancybox=True, shadow=True, ncol=1, prop={'size':10})\n", "\n", "plt.title('L2 Regularization Paths for Feature Weights.')\n", "plt.xlabel('Regularization weight C (higher C is less regularization)')\n", "plt.ylabel('Feature Weight')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 21, "text": [ "Next we'll do the same for $L1$.\n", "\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "plt.clf()\n", "fig = plt.figure()\n", "ax = plt.subplot(111)\n", "\n", "for f in feats:\n", " plt.plot(Rpath1.index.values, Rpath2[[f]], label=f)\n", "\n", "plt.xlim([-4, 2])\n", "plt.ylim([-0.5, 0.5])\n", "box = ax.get_position()\n", "ax.set_position([box.x0, box.y0, box.width*1.5, box.height * 1.5])\n", "\n", " # Put a legend below current axis\n", "ax.legend(loc='upper center', bbox_to_anchor=(1.15, 1), fancybox=True, shadow=True, ncol=1, prop={'size':10})\n", "\n", "plt.title('L1 Regularization Paths for Feature Weights.')\n", "plt.xlabel('Regularization weight C (higher C is less regularization)')\n", "plt.ylabel('Feature Weight')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 22, "text": [ "It is not so obvious from the above plot that $L1$ holds the feature weight at 0 for some time, but we can see this if we directly compare the paths of individual features.\n", "\n", "\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "\n", "fs = ['isbuyer','expected_time_visit','visit_freq','multiple_buy']\n", "\n", "fig = plt.figure()\n", "\n", "for i,f in enumerate(fs):\n", " fig.add_subplot(2, 2, i)\n", " plt.title(f)\n", " plt.plot(Rpath2.index.values, Rpath2[[f]], 'b', label='L2')\n", " plt.plot(Rpath1.index.values, Rpath1[[f]], 'b--', label='L1')\n", " \n", "fig.tight_layout()\n", "\n", "\n" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEaCAYAAABEsMO+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXlc1NX3/18DYsqugqwqwriArIaSC4mGG5apuS9goqGV\nW34VrUxtEdcWNT9aoWSoVD9zByIXzFREE7PcJTZZlFAURB0Yzu+PKyMDAwyzz3Cfj8f7AfN+33vP\necO9c97ve889R0BEBA6Hw+FwdAwjbSvA4XA4HI4suIHicDgcjk7CDRSHw+FwdBJuoDgcDoejk3AD\nxeFwOBydhBsoDofD4egk3EAZGEFBQYiOjm50vVmzZuHTTz9Vg0aKYWFhgczMzHrLnDx5El27dtWM\nQhy9IyQkBD/88EOD5eTpa7K4fv06fH19YWlpiU2bNimgIachBHwflO6QmZkJV1dXVFRUwMhIsWeH\n/v37Y8qUKZg2bVqdZWJiYhAdHY2TJ08qqqpO4uLigm3btmHAgAHaVoWjp0ydOhXt2rXDJ5980mDZ\n8PBwWFtbY/369RrQrGnC36B0EP7MoBgCgYD/7TgaIysrCx4eHnVer6ys1KA2BgpxKDc3l0aNGkW2\ntrbUsWNH2rBhAxUVFZGzszMdPHiQiIhKSkrIzc2NfvjhByIiCgsLo4iICBo4cCBZWFhQv379KCsr\nS9Lm1atXKTg4mFq3bk1dunShn376SXKtrKyM3nvvPerQoQNZWVlRYGAgPX78mNq1a0cCgYDMzc3J\n3NycUlJSiIgoOjqa3N3dqVWrVjR48GApOUlJSdSlSxeysrKid999l/r160ffffddnfd65coVatGi\nBRkbG5O5uTm1atVKcj8ffvghEREdP36cnJycaM2aNWRra0sODg60d+9eOnz4MHXq1Ilat25NUVFR\nkjYrKyspKiqK3NzcqE2bNjR27Fi6d+9evX/zIUOG0KZNm6TOeXt70969e4mISCAQUHp6OhERHT58\nmDw8PMjCwoKcnJxo3bp1Ej2dnZ2JiGjy5MlkZGRELVu2JHNzc1q7dm298jmGwapVq2j06NFS5+bM\nmUNz5syhoKAgyVi4efMmvfzyy2RlZUU2NjY0btw4SXmBQEC3bt2irVu3komJCTVv3pzMzc1p+PDh\ndcrt378/GRsbU4sWLcjCwoJu3LhBYWFhNHPmTBo6dCiZmZnR0aNHZX63VFFWVkZhYWHUqlUr8vDw\noDVr1kj6M4fR5A2UWCym7t270yeffELl5eX077//kqurK/3666+UlJRE9vb2dPfuXZo+fTqNGTNG\nUi8sLIwsLCzo5MmT9PTpU5o7dy717duXiIhKS0vJ2dmZYmJiSCwWU1paGtnY2NCVK1eIiOjtt9+m\n/v37U15eHonFYjpz5gw9ffqUMjMzSSAQkFgslsjZt28fCYVCunbtGonFYvr000+pd+/eRERUWFhI\nFhYWtGfPHqqoqKAvvviCmjVrRtHR0fXec0xMjETXKqZOnUpLly4lIvbF36xZM/rkk0+ooqKCvv32\nW2rTpg1NnDiRSktL6fLly9SyZUvKzMwkIqIvv/ySevXqRbm5uSQSiSgiIoImTJhQrw47duygPn36\nSD5fvnyZrK2tSSQSEZG0gbK3t6c//viDiIiKi4vpwoULEj2rD2gXFxc6evRovXI5hkVWVhaZmppS\nSUkJERFVVFSQg4MDnT17loKCgiRjYfz48bRy5UoiInr69CmdOnVK0kb1vlZ9HDRE9faJ2HeClZUV\nnT59moiYAarru4WIKDIykl5++WW6f/8+5eTkULdu3ahdu3ZK/kUMiyZvoFJSUqh9+/ZS51auXElv\nvvkmERHNnj2bPD09ydnZWeqtICwsTOpLuLS0lIyNjSknJ4fi4uIoMDBQqs233nqLVqxYQWKxmFq2\nbEmXLl2qpUtGRkYtAzVkyBCpQSAWi8nU1JSysrLo+++/p169ekm14ezs3KCB2r59u0wDVf0NqmXL\nllRZWUlERA8fPiSBQECpqamS8i+++CLt37+fiIi6du0qZRjy8vLIxMRE6j5q8vDhQzIzM6Ps7Gwi\nInr//fcpPDxccr36l0b79u1p69at9ODBA6k2uIHiEBH17duXduzYQURsRkEoFBKRtAEJDQ2lt956\ni27fvl2rfk0DVTUOGqL6G1pV3bCwMMnnhr5bqhsrIqJvvvmGv0HVoMmvQWVlZSEvLw+tWrWSHFFR\nUbh79y4AYMaMGbh8+TKmTp2KVq1aSeoJBAI4OztLPpuZmaF169bIy8tDVlYWzp49K9Xmrl27cOfO\nHRQVFeHJkydwc3OTW7+5c+dK2mnTpg0AIDc3F/n5+VI6AEC7du2U/ZMAANq0aQOBQAAAaNmyJQDA\nzs5Ocr1ly5YoLS2V6Dhy5EiJjh4eHmjWrBnu3LlTZ/sWFhYYNmwYdu/eDQCIi4vDpEmTZJbds2cP\n4uPj4eLigqCgIKSkpKjkHjmGwcSJEyX9aNeuXZg4cWKtMmvWrAERoWfPnvD09MT27dtVIrtqjFRR\nfTw29N2Sl5cnNV7bt2+vEp0MiWbaVkDbtG/fHh07dsSNGzdqXROLxXjrrbcQGhqKr7/+GlOnTpUY\nFiJCTk6OpGxpaSnu3bsHJycntG/fHv369UNSUlKtNisrK9GiRQvcunUL3t7eUtdqdvYq/ZYuXYoJ\nEybUunbz5k0pHWrqVBey5NR3viHat2+P7du3o1evXo2qN2HCBKxYsQKBgYF48uQJ+vfvL7Ocv78/\n9u3bB7FYjI0bN2Ls2LHIzs5Wmf4c/Wb06NFYsGABcnNzsW/fPpkPMHZ2dvjmm28AAKdOnUJwcDD6\n9esHV1dXqXLK9qHq9ev7bgEABwcHZGdnw93dHQBk9ummTpN/g+rZsycsLCywZs0aPH78GGKxGP/8\n8w/OnTuHlStXwtjYGNu3b8fChQsRGhoq5ZkTHx+PU6dOQSQSYenSpejVqxecnJwwbNgw3LhxA7Gx\nsSgvL0d5eTnOnTuHa9euwcjICNOmTcN7772H/Px8iMVinDlzBiKRCLa2tjAyMkJ6erpExsyZM7Fy\n5UpcuXIFAPDgwQP8/PPPANg+j8uXL2Pv3r2oqKjAhg0bUFBQ0OA929vb4/bt2ygvL5ecIzbdq9Df\ncObMmXj//fclA6ywsBAHDhxosF5ISAiysrKwbNkyjB8/XmaZ8vJy7Ny5Ew8ePICxsTEsLCxgbGws\ns6ydnZ3U347TNLC1tUVQUBCmTp0KV1dXdOnSpVaZn3/+Gbdv3wYAWFtbQyAQyNzKYWdnh3///Vdu\n2dXHTM3xU9d3y/nz5wEAY8eORVRUFIqLi3H79m1s3LiRP2TVoMkbKCMjIxw6dAgXL16Eq6srbG1t\n8dZbb+H48eP48ssvsWPHDggEAkRGRkIgEGD16tUA2JPSxIkTsWLFCrRp0wZpaWmIjY0FwKavkpKS\nEBcXBycnJzg4OGDJkiUQiUQAgHXr1sHLyws9evRAmzZtsGTJEhARTE1N8cEHH6BPnz5o1aoVUlNT\nMWLECERGRmL8+PGwsrKCl5cXfv31VwCAjY0Nfv75ZyxevBg2Nja4desW+vbt2+A9DxgwAN26dYO9\nvT3atm0ruZ/qg6PmQKlv4MydOxfDhw/HoEGDYGlpiV69eiE1NbVBPZo3b45Ro0bh6NGjtaZlqsuL\njY1Fx44dYWVlhW+++QY7d+6UWW7JkiX49NNP0apVK3z++ecNyucYDhMnTpTZj6o4f/48XnrpJVhY\nWOD111/Hhg0b4OLiAkC6D4WHh+PKlSto1aoVRo0a1aDcmmOm+ue6vlsePnwIAFi2bBk6dOiAjh07\nYsiQIQgNDeXbJGqizAJWQkICdenShYRCIa1atarW9X379pG3tzf5+vpS9+7dpRawG6qr6zRmMZXD\nqYk8/X/27NkkFArJ29tb4rlIxLwmPT09qVu3bvTll19qSmWOmqnp9MNRwouvoqKC3NzcKCMjg0Qi\nEfn4+EjcqKsoLS2V/H7p0iVyc3OTu66uU33fEIfTGOTp/4cPH6ahQ4cSEfMGCwgIICKiv//+mzw9\nPenx48dUUVFBwcHBdOvWLY3fA0f1cANVG4Wn+FJTUyEUCuHi4gITExOMHz8e+/fvlypjZmYm+b20\ntBQ2NjZy19V1ar7O6xozZ86EhYVFrePtt9/WmA47d+6UqYOXl5fGdNBF5On/Bw4cQFhYGAAgICAA\nxcXFKCgowNWrVxEQEIAWLVrA2NgY/fr1wy+//KKN2zB4srOzZfZfS0tLyXqWqtHl7xRtoLAXX25u\nrpSLpLOzM86ePVur3L59+7BkyRLk5+dLvNrkravLqMpNVV1s2bIFW7Zs0aoOkyZNqtN1vCkjT/+X\nVSYvLw9eXl748MMPce/ePbRo0QKHDx9Gz549NaZ7U6J9+/YoKSnRmLygoCDuyVcDhQ2UvJZ+xIgR\nGDFiBE6ePIkpU6bg2rVrcssQCoXcK4uj07i5ueHWrVuNqiPv2CEZC+Zdu3ZFZGQkBg0aBDMzM/j5\n+cn0RuNjh6PryDN2FJ7ic3Jyktpzk5OTU2vTaHUCAwNRUVGBe/fuwdnZWa666enpEvdnTRzLli3j\n8ri8Rh2KGAF5xk7NMrdv34aTkxMAYNq0aTh//jxOnDgBa2trmW7VfOzov0xDlyfP2FHYQPn7++Pm\nzZvIzMyESCTCjz/+iOHDh8scJABw4cIFACxCgTx1ORxDRZ7+P3z4cOzYsQMAkJKSAmtra0kkj6pI\nBNnZ2di7d2+drtUcjr6j8BRfs2bNsGnTJgwePBhisRjh4eFwd3fH1q1bAQARERHYs2cPduzYARMT\nE5ibmyMuLq7euhxOU0CesRMSEoL4+HgIhUKYmZlJrXmOHj0aRUVFMDExwebNm2FpaamtW+Fw1Avp\nMJpW7/jx4zovr7KSSCRiPzUhTxkMXR6R5vuovPCxo/8yDV2ePH1UpzPq8gR0tYmMBNauBYgAIyPA\nxARo1uz5z48/BmbN0raWTQdd7aO6qpeq+OknYMQIoHlzbWvCURR5+miTD3WkbyQkAGfOAJWVwNOn\nQHExkJ8PpKcDu3cDX3zBjBeHY8isWwecPq1tLTjqhhsoPaKwEMjKAl58ERAI2BtTixaAhQXQujXw\nyivMOD2LRcnRYRITE9G1a1d06tRJEt+xJnPmzEGnTp3g4+ODtLQ0yfmoqCh069YNXl5emDhxIp4+\nfaoptXWGkBAgPl7bWnDUDTdQesSJE0DfvswwyUIgACZNAqrFUuXoIGKxGO+++y4SExNx5coV7N69\nG1evXpUqEx8fj1u3buHmzZv45ptvMOvZvG1mZia+/fZbXLhwAX///TfEYrHE+agpwQ1U04AbKD3i\n+HGgjpRJEiZNAuLigIoKzejEaTyKhjq6c+cOLC0tYWJigrKyMlRUVKCsrEyyP6op4e8P3L3LZhQ4\nhgs3UHpEcnLDBqpTJ6BDB+DoUY2oxFEAWWGMcnNz5SrTunVrLFiwAO3bt4ejoyOsra0RHBysMd11\ngZQU4OZNYMgQtibLMVy4gdIT7twB8vIAX9+Gy/JpPt1GmVBH6enp+PLLL5GZmYm8vDyUlpZK5cdq\nCmzdCpw6BbzzDlAjKTXHwGjyKd/1heRk4OWXgTqSyUoxbhywbBnw6BFQLaA8R0dQJtRRcnIyevfu\njTZt2gAARo0ahdOnT8sMyrt8+XLJ70FBQQgKClLtjWiJoiLmFBQQoG1NOI0hOTkZycnJjaukzEar\nhpKuxcbGkre3N3l5eVHv3r3pr7/+klzr0KEDeXl5ka+vL/Xo0UNm+0qqZ1BERBB9/rn85QcPJtq1\nS336cBiK9NHy8nJydXWljIwMevr0aYP5oM6cOSPJB5WWlkbdunWjsrIyqqyspNDQUNq0aZNK9NIX\nevcm+v13bWvBURZ5+qjCvViepGunT5+m4uJiImLGrGqQERG5uLhQUVFR/coZ8CBrLJ07E6WlyV/+\nhx+Ihg1Tnz4chqJ9ND4+njp37kxubm60cuVKIiLasmULbdmyRVLmnXfeITc3N/L29qY///xTcn71\n6tXk4eFBnp6eFBoaSiKRSGV66QNduhBdvqxtLTjKIk8fVTiSxJkzZ7BixQokJiYCAFatWgUAWLx4\nsczy9+/fh5eXlyTRV8eOHXH+/HnJVIUsDH03vLzk5QFeXmwflIzMCjIpLQWcndlisq2tevVryuhq\nH9VVvVSBrS3wzz/As9i5HD1FrZEk5PFEqk50dDRCQkKklAsODoa/vz++/fZbRdVoEhw/DvTrJ79x\nAgBzc7ZX5Kef1KcXh6MNRo4EWrXSthYcTaD2hIUAcPz4cWzbtg2nTp2SnDt16hQcHBxQWFiIgQMH\nomvXrggMDKxV11AXehuDPPufZDF5MvDpp8zbiaMaFFro5aiUb76R/vz558wZKCJCO/pw1IfCU3wp\nKSlYvny5ZIovKioKRkZGiIyMlCp36dIljBo1ComJiRAKhTLbWrFiBczNzbFgwQJp5Qx4mqIxuLkB\n+/cDnp6Nq1deDjg5sX0jrq7q0a2po6t9VFf1Ugd79gDffcf3ROkbap3ikyfpWnZ2NkaNGoXY2Fgp\n41RWVoaSkhIAwKNHj5CUlAQvLy9FVTFosrOBkhKgW7fG1zUxAcaOBXbtUr1eHOVQNBbf9evX4efn\nJzmsrKywYcMGTaqucwQHs31RZWXa1oSjcpTxwmjIEyk8PJxat25Nvr6+Uu7k6enp5OPjQz4+PtSt\nWzdJ3ZooqZ5BEBNDNGaM4vVPn2ZeT4rkj+I0jCJ9VB4P2Opu5ikpKVIesFWIxWKyt7en7Oxsleil\nzwQFER06pG0tOI1Bnj7K80HpOFOnsg2JiuZ4IgKEQuYs8eKLKlWNA8X6qDwesDNnzkT//v0xbtw4\nAEDXrl1x4sQJSdp3AEhKSsLHH3+MP/74QyV66TNr1wKZmcDXX2tbE4688HxQeg6R4g4SVfAI57qH\norH4qrZoVBEXF4eJEyeqV1kd4+JF4OzZ2udDQlieNI5hwUMd6TAZGYBIBHTpolw7kyYxI7d2rXyh\nkjjqRdFYfNXriUQiHDx4sM71K8AwPWD37WPJOmuGOfLwAFJTtaMTRz4U8YDlBkqHqYpe3giPfpl0\n6QI4OgLHjgEDB6pENY4SKBOLr4qEhAS8+OKLsK1nF3Z1A2UoFBWxKeuaVCXw5OguNR+SVqxY0WAd\nPsWnwyg7vVed0FBg2zbVtMVRDnk8YIcPH44dO3YAYFs6rK2tpdafdu/ejQkTJmhUb12gqAioJ/gM\nx8DgBkpHqVp/UtWsTFgY8OuvQD3BPjgaolmzZti0aRMGDx4MDw8PjBs3Du7u7ti6dSu2bt0KAAgJ\nCYGrqyuEQiEiIiKwefNmSf1Hjx7hyJEjGDVqlLZuQWvcu8cNVFOCe/HpKDdvsrennBzlp/iqmD0b\nsLQEPvtMNe1xdLeP6qpeytKjB/PU69lT25pwlIV78ekxVdN7qjJOADNQ334LPH6sujY5HE3y6qtA\n+/Z1Xy8uBmR43XP0FG6gdBRVrj9V0bkzewLlkSU4+sqyZYC9fd3X79wBJkxgU+Qc/UcpA9VQuJad\nO3fCx8cH3t7e6NOnDy5duiR33aaMKvY/1cXcucBXX/EBrG0UDXUEAMXFxRg9ejTc3d3h4eGBlJQU\nTamt83TuzGYdbt7UtiYclaBomAplEhbKU/fZ2pii6uk1V64QdeignrYrK4nc3YmOHVNP+00NRfqo\nsqGOQkNDKTo6mohYdt6qMaasXoZCWBjR5s3a1oLTEPL0UYXfoFJTUyEUCuHi4gITExOMHz8e+/fv\nlyrTq1cvWFlZAQACAgIkO+HlqduUUdfbE8CeLufMYW9RHO0gT/8/cOAAwsLCALCxU1xcjDt37uDB\ngwc4efIkpk2bBoB5BFaNMQ7jlVeAo0e1rQVHFWglYWFj6zY1Tp5kCQrVxZQpbCH533/VJ4NTN8qE\nOsrIyICtrS3efPNNdO/eHTNmzEAZD+MtxYAB7CGvslLbmnCURSsJCxtT1xDDtTRESgqgziAAZmbA\ntGnApk0s2RtHflSRsFCZUEcVFRW4cOECNm3ahB49emDevHlYtWoVPv7441r1DW3s3LgBpKcDQ4fW\nX87JCZg3j6XfMDfXjG6chtFoqCN5wrUALGHhjBkzkJiYiFbP8jTLWxcwzHAt9XHnDvDgAdCpk3rl\nvPMO0L07sGIFYGGhXlmGhCLhWmqiTKgjIoKzszN69OgBABg9erQkGnpNDG3snDoFnDjRsIECgKVL\n1a8Pp3FoNNSRMgkL5anbVDl7lm1CNFLzBoAOHViUiu+/V68cTm2UCXVkb2+Pdu3a4caNGwCAI0eO\noJsi2Sz1EB7mqOmh8BtU9XAtYrEY4eHhknAtABAREYGPP/4Y9+/fx6xnyYxMTEyQmppaZ10Om957\n6SXNyJo7F5g+HXj7bfUbRM5z5Bk7ISEhiI+Ph1AohJmZGbZv3y6pv3HjRkyaNAkikQhubm5S1wyZ\noiKgdWtta8HRJDzUkY4xYACwaBEwZIj6ZRGxab7PPmP5dDiNR1f7qK7qpQxvvcX664QJwJdfspmG\nwEC+zqSvyNNHuYHSIcRioFUrlhlUU0+KMTHA7t0skCyn8ehqH9VVvZThjTeA8eNZ0sLkZMDEBDh/\nHvD1ZQ92AwYAvXoBL7ygbU058sBj8ekZly+zvE2anMaoGvBXr2pOJoejCIMGAW3bAlu2sIeq5GTg\n7l3m8VpRASxeDNjYsDesykq2jWL2bG1rzVEGbqB0CE2uP1XRogUwaxawcCEb5BzNoEyoIxcXF3h7\ne8PPzw89m1BY74gI4McfgalTnweMNTUFgoOBlSvZ+Ll9m2Wi3rULsLMDtm8HHj3SqtocZVBLDAsV\noePqqZw33yT63/80L/fJE6LBg4mmTCESizUvX59RpI8qG+rIxcWFioqKVK6XrnPzJlGbNkSFhfWX\nO3GChQp78oQoMJAoMVEj6nEaiTx9lL9B6RBnz2r+DQpgc/a//MLWvubM4YFk1Y0yoY6qoCb4T/ro\nI7YB18am/nIvvwx4eQGbN/OwR/oON1A6QnExkJ0NeHpqR76pKXDwIJsm+eAD7ejQVFA01FFVGYFA\ngODgYPj7++Pbb7/VjNJa5uJFFr5o3jz5ykdFsaNnT26g9BmF90FxVMu5c8yFtpkW/yNWVkBiIosD\naGEBLFmiPV0MGUVDHVXxxx9/wNHREYWFhRg4cCC6du2KwMDAWuUMKdTR+++zByd5Xco9PVlyw+Rk\nlnrj3j2+h0rbaDTUEUe1aMNBQhY2NsBvv7FpEktLFhKJo1qUCXUEAI6OjgAAW1tbjBw5EqmpqQ0a\nKH3mxAngn3+AMWMaV2/FCuaC/uuvgLW1enTjyI9GQx0BDXsiXbt2Db169UKLFi2wfv16qWtN1ROp\nLnTFQAHM1f2334DVq3koJHWgTKijsrIylJSUAAAePXqEpKQkeHl5afweNAURcx+fNAl49ueQm3bt\nWKSUbdt4pBR9ReE3KLFYjHfffRdHjhyBk5MTevTogeHDh0uFLGrTpg02btyIffv21aovEAiQnJyM\n1vy9G0TMQOnSckLHjkBSEstLVVoKvPkmW6fiKI8yoY4KCgowatQoAEBFRQUmTZqEQYMGae1e1M2B\nAywqua+vYllyFy9mWXbfew/g0dT0D4UjSZw5cwYrVqxAYmIiAEgiKi9evLhW2RUrVsDc3BwLFiyQ\nnOvYsSPOnz+PNvVEfzTE3fCyuHmT7eXIytK2JrW5dIkN8pQUNsUybRpbeG5ExhSDRlf7qK7q1RjE\nYsDbG1izBsjJAdLSgGc2vFGsW8fyn8l4TuZoEbVGklA26WBT9ESqC12a3quJtzcQH88MVYcObKrF\n0xNYv56lBuFw1EVsLHNsCAlRLpL5u+8y4/YsHR1Hj9BIwkJZnDp1Cg4ODk3KE6kutLX/qTE4OzNP\nqiVL2NPotm1Aly6Ajw/g5ga4uLBpQRcXdjg6AsbGWlZaDagiYSGnYZ4+BZYtY0ZKIGBeeA4OirXV\nogVzmJg/n401/vavP6g9YWFdODzrbU3FE6k+UlKAiRO1rUVtKiuBggK2RvbMgQwCAYsgHRjI0nTE\nxgL//ceeTg8cYE+6Dx+ydau2bVn2XjMztn4lFrNkjM2bM3d6gYAtXtvZMSNnbMzOGxmxa3fuPF93\nqP6l4uQEdO1a+3xuLsu6WnPWwNERCAhgT9LKooqEhQBzMJo3bx7EYjGmT5+OyMjIWmXmzJmDhIQE\nmJqaIiYmBn5+fpJrYrEY/v7+cHZ2xsGDBxXSQZfZs4cl7ezbl31++WXWTxQlIIBFQ9+/HxgxQjU6\nctSPwgaquieSo6MjfvzxR+zevVtm2ZrzjGVlZRCLxbCwsJB4Ii1btkxRVfSasjIWqLXad4/WSEkB\nvv6abRjOzgby8tgUS0SE7BT0jx8DT56w/VPW1mwx2sgI6N0bGD6cGZiysufHn38Cv//OYv5VVDBD\nUlnJpm6cnJgBqzqIWFuVlbXlikRMbk1DVFbGzldHIGDnSktV9mdSGnkcjOLj43Hr1i3cvHkTZ8+e\nxaxZs5CSkiK5/tVXX8HDw0Pi0WdoxMQA4eHPP7/+unLtde3Ktk0sWAAMG8YioXP0AGViKcXHx1Pn\nzp3Jzc2NVq5cSUREW7ZsoS1bthARUX5+Pjk7O5OlpSVZW1tTu3btqKSkhNLT08nHx4d8fHyoW7du\nkrqKxGrSd06eJOrRQ9taMK5fJ4qJITp2jOjWLRbLjFM/ivTR06dP0+DBgyWfo6KiKCoqSqpMREQE\nxcXFST536dKFCgoKiIgoJyeHXnnlFTp27Bi9+uqrKtNLV8jOJmrViqisTLXtTplC5OFBtG6datvl\nKIY8fVSpjbpDhw7F0KFDpc5FRERIfre3t5eaBqzC3NwcFy9eVEa0waBJBwki4ORJFtJozZrac/Gd\nO7ODo15kORidPXu2wTK5ubmws7PD/PnzsXbtWjx8+FBjOmuSH34Axo4FWrZUbbuvvMKmo6OiWJqZ\nqmlrju7CI0lomZQU4Nm2FrWSmMgCwZqYsKmT8nK2FsTRPIqGOiIiHDp0CG3btoWfn1+Dzhr66GBE\nxKb3GrvvpomLAAAgAElEQVQpVx5eeYXth3rzTTbVFxenehmcuuGhjvSQlBT2NqNO0tOB0FD2ZDpo\nEPdi0jbKhDras2cPDhw4gPj4eDx58gQPHz5EaGioJOpEdfTRwSglha09BgSovm1nZ/aQNmAAGw9H\njzKjxdEMCjkYqX2iUQl0XD2lyckhsrUlqqxUn4yKCqLu3Yk2blSfjKaMIn20vLycXF1dKSMjg54+\nfdpgPqgzZ85I5YOqIjk52eDWoN56i6jGchyVlhKtXataOfv2EXXtSvT0qWrb5ciPPH2UR6jSIlX7\nn9T5RmNsDGzaxIO+6hLVQx15eHhg3LhxklBHVeGOQkJC4OrqCqFQiIiICGzevFlmW8ruR9QlHj8G\nfv4ZmDJF+nxBAcvtpEqGDwdcXYEvv1RtuxzVonCoI01gCOFa6mPhQuaezfMv6S+62kd1Va/62L2b\nBSd+Fj1NQmoqe8A6d0618tLT2VRiWhoLLMvRLGoNdcRRHl0OccThaJqYGOBZEmEpiorUk8vJzY0Z\nvmohQjk6BjdQWqK8nD259eihbU04HO1z+zZ7Q5IV5UGZOHz1UV7OwnOdO8ci93N0D63lg2qorqFz\n6RIbHJaWqm87PV31bXJUizz9f86cOejUqRN8fHyQlpYGAHjy5AkCAgLg6+sLDw8PLDGQtMexsSxa\nvqy9T+oyUAALejx2LDB7Nov/x9ExFPXAqKioIDc3N8rIyCCRSCTTE+nu3bt07tw5+uCDD2hdte3b\n8tSV18tDX9m0iWj6dNW3u20bkY8PkVis+rY5tVGkj8rT/6t78aWkpEh58T169IiImDdgQEAAnTx5\nUiV6aYvKSqIuXYhOn5Z9/Y8/iH77TT2yT50icnQkGjKEqI6ANhw1IU8fVfgNKjU1FUKhEC4uLjAx\nMcH48eOxf/9+qTK2trbw9/eHSY3AV/LUNXROnAB69VJtm//8AyxaBOzaxTOI6jLy9P8DBw4g7NmC\nTEBAAIqLi3HnWX4T02eZI0UiEcRisd4n/Tx7lm3QrWs9tk8fli9NHfTuDbz2GntD+/xzYOdO9cjh\nKIZW8kEpm0tK3yktBX79lbm6qpI1a1hyQQ8P1bbLUS3y9H9ZZW7fvg2ABZv19fWFnZ0d+vfvDw89\n/4fHxABTp2pvA3lUFHDsGPDFF8BHHzHvWrFYO7pwpFHYQCmz/8KQ9m4owsGD7KnQxkZ1bZaXA4cP\nsxhjHN1G0VBHVfWMjY1x8eJF3L59G7///rte56eqa++TJmnViq1FpaayIy2NJUm8d097OnEYWskH\n1Zi6+hhPrCF27wYmTFBtm7//DgiFPACmulFFwkJlQh1Vx8rKCsOGDcP58+dljgt9GDsHDgD+/iwM\nkTYZP54dAgHbh7VoEdCzJ8sf1a2bdnUzFBQaO4oucMkTrqWKZcuWSTlJyFtXCfV0lqIiIktLogcP\nVNvuvXtEf/2l2jY5DaNIH1Um1FFhYSHdv3+fiIjKysooMDCQjhw5ohK9tMGQIUS7dmlbC9l8/z2R\njQ3R3r3a1sQwkaePaiUfVF11FbkBfePbb4neeEPbWnBUhaJ9tKGxQ0T0zjvvkJubG3l7e9Off/5J\nRESXLl0iPz8/8vHxIS8vL1qzZo1K9dIkt2/Ll/fpww+JRCLN6FST5GSidu2IPviAKD9fOzoYKvL0\nUR7qSMMEBwOzZgFvvKFtTTiqQFf7qK7qVZ3/+z+WWbm+eHgiEWBmxn5qY+k6JIStUZWXA7/9xqJP\nDBvGzvfowb1llUGePsoNlAYpKADc3VkqdVUnY+NoB13to7qqVxWFhSwN+6VL9a+b3rkDeHkBd+9q\nTrfqPHwIzJ/P1qV69ABeeAEoKQGys5lOQ4YAgwezwLMODoCdHR/b8iJPH+X5oDTITz8x13LegTlN\nnS++YBEcGnLqUWcUCXmwtASio9kew6tXgYwM9jYVHw9kZrKfv/zCQjXl5LCHUGNjwNQUsLBg9a2t\nAV9foEULZuBatGBHeTlw5Qorb2T0/KeFBeDn9/yNUSBgx6NHwN9/P/9chZkZ4OPz/HPVtdJSVr46\nAgHTzdu79r2WlrL7rImpKTBjBtChg1J/SoXgBkqDxMUBS5eqts1Hj1iWXJ4dl6Mv3L8PbN0K/Pln\nw2W1baCq8PRkR3VcXIC332YHwN62Tp8G8vOB3FxmrAoLmVHy9ASePGHhlJ48YS7sd+8yl3YioLLy\n+U8zM5aaHmDnqo7SUuDixdq6mZo+D29W/YWkpoGqumZuDty4Ubud0lL2RlsTc3P2psgNlAGTmQnc\nuqX6HfGbNwNZWSznE0d/SExMxLx58yAWizF9+nRERkbWKjNnzhwkJCTA1NQUMTEx8PPzQ05ODkJD\nQ3H37l0IBAK89dZbmDNnjhbuQHE2bABef519wTeEuiKZqwNLS/ZFzlEhanLQUAk6rl6jiIoimjlT\n9e327k2UkKD6djnyoUgfVSYWX35+PqWlpRERUUlJCXXu3Fmvtmg8eMBct2/ckK/8lStE+/erVyeO\ndpCnj3IfFA2xe7fqozwUFLA57P79VdsuR70oE4vP3t4evr6+AABzc3O4u7sjLy9P4/egKJs3A4MG\nAZ06yVfe3V31IcE4+gM3UBrgyhU2VREYqNp2Dx5kUwovvKDadjnqRdlYfFVkZmYiLS0NAQEB6lVY\nRTx6xJwjeAZpjrzwNSgNEBcHjBun+j0Te/fKzkDK0W2UjcUHAKWlpRg9ejS++uormJuby6yva6GO\nvvmGPaTpeWxbjoIoEupIKQOl6EIvALi4uMDS0hLGxsYwMTFBamqqMqroLERsem/3btW3a24ODB2q\n2nY56kfZWHzl5eV44403MHnyZIyQlYL2GdUNlLZ58gRYt44FNOY0TWo+JK1YsaLhSooucCmbdM3F\nxYWKiorqlaGEejrD+fNEbm4sKRvH8FCkjyoTi6+yspKmTJlC8+bNU7le6mTTJqJXX9W2FhxdQp4+\nqtaEhfUlXXtmHBUVrzdURS5v4hlGONVo1qwZNm3ahMGDB8PDwwPjxo2Du7s7tm7diq1btwIAQkJC\n4OrqCqFQiIiICGzevBkAcOrUKcTGxuL48ePw8/ODn58fEhMTtXk7DSISAatXAx9+2Pi6K1eyaBKc\nponCU3yyFnHPnj3bYJnc3FzY2dlBIBAgODgYxsbGiIiIwIwZMxRVRWeprAR+/JElJ+RwqjN06FAM\nrTE/GxERIfV5k4zNbX379kVlZaVadVM1O3awsEaK+HJs2gSEhqpeJ45+oLCBUnSht4o//vgDjo6O\nKCwsxMCBA9G1a1cEynBz07WF3sbwxx9skyFfFDYcVJEPqilRUcEy1sbENL4uke5EkuBoB7UmLKxv\nodfR0REAYGtri5EjRyI1NbVBA6VvREfzDLeGhkILvU2Yjz9mESMU2WLx6BGLT8djVzZdFF6D8vf3\nx82bN5GZmQmRSIQff/wRw2vsqBs+fDh27NgBAEhJSYG1tTXs7OxQVlaGkpISAMCjR4+QlJQELy8v\nJW5D99i7l2W5nTlTte1WVrJ0HU+fqrZdjmZJTExE165d0alTJ6xevVpmmTlz5qBTp07w8fFBWlqa\n5Py0adNgZ2en82MmOhrYuRPYtUux+vfu8benJo8yXhiKJl1LT08nHx8f8vHxoW7duhlcwsKMDCJb\nW6KUFNW3feYMUbduqm+XoxiK9FFlPWB///13unDhAnl6eqpUL1WSkEBkZ0d07ZribVy4QOTjozqd\nOLqFPH2U54NSMSIRm84YNw547z3Vt//uuyyB2iefqL5tTuNRpI+eOXMGK1askHjfrVq1CgCwePFi\nSZmZM2eif//+GDduHACga9euSE5Ohr29PQAWReK1117D3zXzKSihl6pIS2PhjPbtA/r0Ubyd/Hy2\njjtmjOp04+gO8vRRHupIxbz/PtC2LUtypmpu3WJRKWbPVn3bHM2haKijmmV0kaws4LXXgC1blDNO\nAEsAyI1T04aHOlIhhw4BP/8MXLignn1PkZHAggXMAHL0F1WEOpIHTXvA3r/PIpv83/8Bb7yhVlEc\nPUTjoY44z8nJAcLDWXZNdSzsFhSwJGOxsapvm6NZlPWAlRdNesA+fQqMGMHSn8+bpzGxHD1CEQ9Y\nPsWnAsrLWbSI+fOVn9aoC3t74K+/uMutIaCMB6wuUlnJghbb2gLr12tbG44hwQ2UCvjoI8DCAli0\nSL1yVB0NnaMdlAl1BAATJkxA7969cePGDbRr1w7bt2/X1q3g/HkgKIg5NPzwA++jHNXCvfiU5PBh\nICKCeS7Z2mpbG46m0dU+qm69cnKYQ9DRo2wz7ptvsk21qmTDBmDgQJa0kGN4cC8+NZKVxWKEhYez\ngLDcOHGaAiUlLOGgry+LEHH9OjB9uuqNE8DG1f37qm+Xoz9wA9VIioqYJ1337myA3rih+ky5VTx+\nzOKRcTjaprycJRzs3Bm4fZuth37yCZvaVhc8Dh9HKQOlTLgWeepqmvpcIMvKWNDLLl2Y4bh8mU1t\nWFqqRx7ADOEXXyjefmPlqRpDl6cM+jB2ioufp4uxs2OR+Q8dAr7/HqjhdKiWv/29eyzYsiy08b82\n9P6sk+NH0TAVyoRrkafus7UxRdVTiGXLlkl9LitjoYXWriVyciIaM4bo+nX1yavOlStENjZEDeR0\nVJk8dWDo8og0H+pI3WMnPZ3oiy+I+vcnsrBgSQa/+YYoN7f+eqr+24vFRMbGROXlmpEnD4benzUt\nT54+qvA+qOoJCwFIEha6V1vRlJWwsKCgABkZGQ3W1TRPnwJ5ecDWrcwz6dw5Nn3n7g74+wN79iiW\nz0ZRFi4Eliyp+wmSo7/o6tiJjWWbbF99le1lCg4GTE2VblYhiovZ9GEzvlOzSaOVhIV5eXkN1lWU\n779n8+Pl5SwXTUUFIBaz/UM2NqzjFxU9PwoKgP/+YwbK3Bz4919AKGR7msaNA1q0qHnfLFJETRwd\ngRdfrH2+vvKyyM1lm32vXmVGkWN46OrYGTOGRSmpipR/9Ojza6+8IttYHTnCpryvXwcOHpS/fE1q\nln/hBZaskNO00VrCQnlwc3NrdHgXZXj4EDh6dAWOHmVvUlOnql9mfbupaxpHdctTB4Yuz83NrdF1\nDHHsAEBcnOr/9pMn131NG7m4DL0/a1KePGNH4wkLnZ2dUV5e3mBdALh165ai6nE4OgsfOxyOfGgl\nYaE8dTkcQ4WPHQ5HPhR+g6oerkUsFiM8PFwSrgUAIiIiEBISgvj4eAiFQpiZmUlCstRVl8NpCvCx\nw+HIh06HOuJwOBxO00XnI0ls3LgR7u7u8PT0RGRkpEZkrl+/HkZGRrh3757aZS1cuBDu7u7w8fHB\nqFGj8ODBA7XI0eTG6JycHPTv3x/dunWDp6cnNmzYoFZ5VYjFYvj5+eG1115Tu6zi4mKMHj0a7u7u\n8PDwQEpKitplKoIhjx8+dlSHzo4dNe/FUopjx45RcHAwiUQiIiK6e/eu2mVmZ2fT4MGDycXFhYpU\nuUu2DpKSkkgsFhMRUWRkJEVGRqpchrybO1VFfn4+paWlERFRSUkJde7cWa3yqli/fj1NnDiRXnvt\nNbXLCg0NpejoaCIiKi8vp+LiYrXLbCyGPn742FEdujp2dPoN6n//+x+WLFkCExMTAICtBiKyvvfe\ne1izZo3a5VQxcOBAGD3LURAQEIDbt2+rXEb1jaEmJiaSzZ3qwt7eHr6+vgAAc3NzuLu7Iy8vT23y\nAOblFh8fj+nTp6s9uviDBw9w8uRJTJs2DQBbF7KyslKrTEUw9PHDx45q0OWxo9MG6ubNm/j999/x\n0ksvISgoCOfPn1ervP3798PZ2Rne3t5qlVMX27ZtQ0hIiMrbrWvTpybIzMxEWloaAtQchmP+/PlY\nu3at5AtLnWRkZMDW1hZvvvkmunfvjhkzZqCsrEztchtLUxo/fOwoji6PHa0HEhk4cCAKCgpqnf/s\ns89QUVGB+/fvIyUlBefOncPYsWPx77//qk1eVFQUkpKSJOdU9TRRl8yVK1dK5nw/++wzNG/eHBMn\nTlSJzOpoesNmFaWlpRg9ejS++uormJubq03OoUOH0LZtW/j5+Wkk4GVFRQUuXLiATZs2oUePHpg3\nbx5WrVqFjz/+WO2ya2Lo44ePnSY+dtQ+4agEQ4YMoeTkZMlnNzc3+u+//9Qi6++//6a2bduSi4sL\nubi4ULNmzahDhw50584dtcirzvbt26l37970+PFjtbR/5swZGjx4sOTzypUradWqVWqRVYVIJKJB\ngwbRF198oVY5RERLliwhZ2dncnFxIXt7ezI1NaUpU6aoTV5+fj65uLhIPp88eZKGDRumNnmK0hTG\nDx87yqHrY0enDdSWLVvoo48+IiKi69evU7t27TQmW1NOEgkJCeTh4UGFhYVqk1FeXk6urq6UkZFB\nT58+VftCb2VlJU2ZMoXmzZunNhl1kZycTK+++qra5QQGBtL1Z6Htly1bRosWLVK7zMZi6OOHjx3V\nootjR6cNlEgkosmTJ5Onpyd1796djh8/rjHZHTt21IiBEgqF1L59e/L19SVfX1+aNWuWWuTEx8dT\n586dyc3NjVauXKkWGVWcPHmSBAIB+fj4SO4rISFBrTKrSE5O1ogn0sWLF8nf35+8vb1p5MiROunF\nZ+jjh48d1aKLY4dv1OVwOByOTqLTXnwcDofDabpwA8XhcDgcnYQbKA6Hw+HoJNxAcTgcDkcn4QaK\nw+FwODoJN1AcDofD0Um4geJwOByOTsINFIfD4XB0Em6gOBwOh6OTcAPF4XA4HJ2EGygdZvny5Zgy\nZUqd13fu3InBgwfL1VZMTAwCAwOV0mfq1KlYunSpUm1wOOrG09MTv//+e53XMzMzYWRkhMrKSg1q\nxVEEreeD4tRN9Vw0mZmZcHV1RUVFhSSx2KRJkzBp0iSN6qOt/Dgcjrz8888/2laBoyL4G5QOIyuO\nr7Zj+2pbPodjKIjFYm2roPNwA6UGXFxcsG7dOnh7e8PCwgLh4eG4c+cOhg4dCisrKwwcOBDFxcVI\nTk6WSiddVffYsWOSz1VvLC+//DIAwNraGpaWlkhJSak1bWdkZISNGzfCzc0Ntra2WLRoUZ0G5dq1\naxg4cCDatGmDrl274ueff5br3v777z8MGjQIlpaWCAoKQnZ2NgDZ0yZBQUGIjo6GSCRC69atpZ5s\n7969CzMzMxQVFckll8ORFxcXFxw9ehSpqanw9/eHlZUV7O3tsWDBAqly0dHRcHJygqOjI9avXy85\nX3Mqu/o4Xbt2LUaPHi3Vzpw5czBv3jwAwIMHDxAeHg5HR0c4Oztj6dKlkjERExODPn364L333oON\njQ1WrFihlvs3JLiBUgMCgQC//PILjh49iuvXr+PQoUMYOnQoVq1ahbt376KyshIbNmyQOV1W81yV\ngTl58iQANgAePnyIl156Sabsffv24c8//8SFCxewf/9+bNu2rVaZR48eYeDAgZg8eTIKCwsRFxeH\nt99+G1evXq33vogIO3fuxEcffYT//vsPvr6+9U4xVk0JNm/eHBMmTEBsbKzk2u7duxEcHIw2bdrU\nK5PDaSxVY2ju3LmYP38+Hjx4gH///Rdjx46VKpecnIxbt24hKSkJq1evxtGjRyX165rKnjx5MhIT\nE/HgwQMALIX5jz/+iLCwMADMuDVv3hzp6elIS0tDUlISvvvuO0n91NRUuLm54e7du3j//fdVfu+G\nBjdQamL27NmwtbWFo6MjAgMD0atXL/j4+OCFF17AyJEjkZaW1qj25J1ai4yMhLW1Ndq1a4d58+Zh\n9+7dtcocOnQIHTt2RFhYGIyMjODr64tRo0bJ9Rb16quvom/fvmjevDk+++wznDlzBrm5uQ3WCw0N\nldLlhx9+qNcBhMNRlubNm+PmzZv477//YGpqioCAAKnry5YtQ8uWLeHp6Yk333xTqn/WNd4cHBwQ\nGBgoGSuJiYmwsbGBn58f7ty5g4SEBHzxxRdo2bIlbG1tMW/ePMTFxUnqOzo64p133oGRkRFatGih\nhrs2LLiBUhN2dnaS31u2bCn1uUWLFigtLVWL3OpThu3bt0deXl6tMllZWTh79ixatWolOXbt2oU7\nd+7U27ZAIICzs7Pks5mZGVq3bi1TRk0CAgLQsmVLJCcn49q1a0hPT8fw4cMbcWccjvwIBAJER0fj\nxo0bcHd3R8+ePXH48GGpMvKMFVmEhYVJZgNiY2MRGhoKgI2r8vJyODg4SMbVzJkzUVhYKFMmp2G4\nF5+GkPVEZmZmhrKyMslnsVgs1ZmrI6/3XHZ2Ntzd3SW/Ozk51SrTvn179OvXD0lJSXK1WZ2cnBzJ\n76Wlpbh37x4cHR3RvHlzAEBZWRnMzc0BAAUFBVJ1qwa2nZ0dxowZI6nD4agDoVCIXbt2AQD27NmD\n0aNH4969e5Lr2dnZ6NKli+T3qrFSc1zW7Mevv/463n77bfzzzz84fPgw1q1bB4AZnxdeeAFFRUUS\nT9uacC/YxsHfoLRI586d8eTJE8THx6O8vByffvopnj59KrOsra0tjIyMkJ6eXm+b69atQ3FxMXJy\ncrBhwwaMGzeuVplhw4bhxo0biI2NRXl5OcrLy3Hu3Dlcu3at3raJCPHx8Th16hREIhGWLl2KXr16\nwcnJCba2tnBycsIPP/wAsViMbdu21dJ18uTJ+OWXX7Bz507JUyeHow6ICLGxsZIHPisrKwgEAinD\n8emnn+Lx48e4fPkyYmJiJGPF19cX8fHxuH//PgoKCvDll19Ktd2yZUu88cYbmDhxIgICAiSzCg4O\nDhg0aBDee+89lJSUoLKyEunp6fXuyeLUDzdQGqL6k1PVIqylpSU2b96M6dOnw9nZGebm5lJTANUX\na01NTfHBBx+gT58+aN26Nc6ePStzMff111/Hiy++CD8/P7z66qsIDw+v1ZaFhQWSkpIQFxcHJycn\nODg4YMmSJRCJRA3ew6RJk7BixQq0adMGaWlpUo4P3377LdauXQsbGxtcuXIFffr0karfrl07dO/e\nHUZGRujbt68Cf0UOR35+/fVXeHp6wsLCAvPnz0dcXBxeeOEFAKwv9+vXD0KhEMHBwVi4cCGCg4MB\nAFOmTIGPjw9cXFwwZMgQjB8/vtY4CwsLwz///FNrHXXHjh0QiUTw8PBA69atMWbMGMkbGN9HqADU\nAAkJCdSlSxcSCoW0atUqmWVmz55NQqGQvL296cKFC1LXKioqyNfXl1599VXJuWXLlpGTkxP5+vqS\nr68vJSQkNKQGRw4EAgGlp6drW416mTZtGi1dulTbamidhsbVvn37yNvbm3x9fal79+509OhRybUO\nHTqQl5cX+fr6Uo8ePTSpNucZ2dnZZGpqSiUlJdpWxaCp10BVVFSQm5sbZWRkkEgkIh8fH7py5YpU\nmcOHD9PQoUOJiCglJYUCAgKkrq9fv54mTpxIr732muTc8uXLaf369aq6B84zdN1AZWRkkLW1NWVm\nZmpbFa0iz7gqLS2V/H7p0iVyc3OTfHZxcaGioiKN6cuRRiwW09y5cyk8PFzbqhg89U7xpaamQigU\nwsXFBSYmJhg/fjz2798vVebAgQOSPQABAQEoLi6WeIPdvn0b8fHxmD59ei0ngZqfOcqjiumDbt26\nwcLCotYhy129MSxduhReXl5YtGgROnTooLSe+ow848rMzEzye2lpKWxsbKSu8/GjHR49egRLS0sc\nPXqUb7TVAPV68eXm5kqtiTg7O+Ps2bMNlsnNzYWdnR3mz5+PtWvX4uHDh7Xa3rhxI3bs2AF/f3+s\nX78e1tbWyt5Lk0cVoVMuX76sAk1q88knn+CTTz5RS9v6hjzjCmCbrpcsWYL8/Hwpj0uBQIDg4GAY\nGxsjIiICM2bM0IjeHPbgoK4tIpza1PsGJe8Tuay3o0OHDqFt27bw8/OrdX3WrFnIyMjAxYsX4eDg\nUCsESRVCoVCysMgPfujiIRQK5RojioyrESNG4OrVqzh48KDUYvypU6eQlpaGhIQEfP3115IoI3zs\n8EOfDnnGTr0GysnJSWrfS05OjtRGTVllbt++DScnJ5w+fRoHDhxAx44dMWHCBBw7dkziWty2bVuJ\nktOnT0dqaqpM+enp6SC2TqaRY9myZVwel9eooyG3f0XHVXUCAwNRUVEhiVvo4OAAgG09GDlypMzx\nw8eO/ss0dHnyjJ16DZS/vz9u3ryJzMxMiEQi/Pjjj7V2/w8fPhw7duwAAKSkpMDa2hr29vZYuXIl\ncnJykJGRgbi4OAwYMEBSLj8/X1J/79698PLyalBRDsdQkGdcVRkYALhw4QIAoE2bNigrK0NJSQkA\nth6SlJTExw/HYKl3DapZs2bYtGkTBg8eDLFYjPDwcLi7u2Pr1q0AgIiICISEhCA+Ph5CoRBmZmbY\nvn27zLYEgufTGpGRkbh48SIEAgE6duwoaY/DaQrIM6727NmDHTt2wMTEBObm5pJ4bgUFBRg1ahQA\nFqh00qRJGDRokNbuhcNRK6TDaFq948ePc3lcXqPQ1SHEx47+yzR0efL0UcGzgjqJQCCADqvH4ehs\nH9VVvdRNRQUQEQFcvAhYWj4/LCyAd98FPDy0rSGnCnn6KA8Wy+FwDIZmzYCoKCAnB3j48PlRUgJU\n21omxeuvA7duAR06AH36AP37A/7+AI9lrH0afINKTEzEvHnzIBaLMX36dERGRtYqM2fOHCQkJMDU\n1BQxMTHw8/OTXBOLxfD394ezszMOHjwIALh37x7GjRuHrKwsuLi44KeffpK5D6qpPgVy9Add7aO6\nqpcucv8+M2jp6cDJk0ByMjNYly4BLi7a1s5wkaeP1mugxGIxunTpgiNHjsDJyQk9evTA7t27Jekc\nACA+Ph6bNm1CfHw8zp49i7lz5yIlJUVy/fPPP8eff/6JkpISHDhwAACwaNEi2NjYYNGiRVi9ejXu\n37+PVatWKXQDHI420dU+qqt66RqlpcCJE8CRI0BKCvDoESASAY8fs+nC8nL2uXoc5cePZbdlagpU\n+TLKSsMAACAASURBVIJV8wlDXft6zc0BWVk5SkoAWf86CwvZ7T98KLu8paV0OUXLl5YCu3cDY8bI\nvg9FUXqKr3pIFgCSkCzVDVRdoY7s7OwkoY4++OADfP7551J1Tpw4AYBFBQ4KCpJpoDgcDkeVlJcD\n584xg3TkCJCWBvToAQQHA2vWAFZWbGqv5mFi8vzLu6bBqfqOrTJQ1b9ziZhBqFkWkDY41Skulq27\nlVXt9gHgWfb5WtRlcBpqvyYPHgDPtt5pHK2EOqoyYADLPNtQJtemjlgMXL/OBtPFi+wJy9pa+rCy\nYj9NTdlgatbs+c+q3wUC9lQoFj//Wf33ysrnR83PRLUPoPZgFIuBp0+BJ0+YzGe5C6UoLGRHzfbs\n7NhRk4ICdtSUZ2cHODrKLl9tqx0AQChkaw26QkNT5/v378dHH30EIyMjGBkZYe3atRgwYIBcdTmy\nuXQJeOUVwNmZGaQPPgD69q17baouTE0bV97SsnHlrawaV76xUeLUXV6V1GugBLLMqQxqvqYRSYc6\nSk5OrldGfXKWL18u+T0oKAhBQUFy6aTP5OYC8fHMIKWlAX//zZ5g/PzY4erKnoKKi4HMzOe/378v\nPTVR/WdFBTM0VQbL2Jgd1X83NmZTDlVH1ecqwwYAL7zAPlcdVfpmZT03ZlV127dnR03y8thR/d8u\nEAD29rINTn7+cwNVHXt72U92ssp7eqrGQCUnJ9fbn+VBLBbj3XfflZo6Hz58uNTMRHBwMF5/pvDf\nf/+NkSNH4tatW3LVbYrMns2moF5+Wfb1khJg7Fjg88+BGimcODpMvQZKmVBHe/bswYEDBxAfH48n\nT57g4cOHCA0NxY4dO2BnZ4eCggLY29sjPz8fbdu2rVOH6gaqKZCbC/TuzZ7sevYExo8HfHwa/1Sl\nLOXlQEICEBMDHD/O3owiI9lTZ00ePmRz9GZmQIsWsqcJDIWaD0mKRLSWZ+q8rmjm8tRtahQUALGx\nwMcfy75OBMycycYUN076hVZCHQ0fPhzff/89AOD777/HiBEj1HFvesfDh0BICDBrFrBzJzB3Lnsi\n1LRx+usvNg2ydi0wbBibXnzwQLZxAtgUho0N0LKlYRsnVVHXtHhN9u3bB3d3dwwdOhQbNmxoVN2m\nxDffsLejVq1kX4+OZtN7z/6EHD1CK6GOFi9ejLFjxyI6OlriZt7UEYmAN95g+zC0vaTQpQvwxx9A\np07a1cNQkXfqfMSIERgxYgROnjyJKVOm4Nq1a42S0xSmx0UiYMsW4NdfZV+/dAlYsoS5jzd27Yij\nWhSZHueRJHQAImDqVLaG9MsvbF2Iox8o0kdTUlKwfPlyJCYmAgCioqJgZGRUr7ODm5sbUlNTcfPm\nTbnqNpWx8+OPwP/+x/Yu1aSkhG24/fBDPrWni8jTR+ud4uNohmXLgKtX2V4DTRqnxETgyy81J4/D\nUCaauTx1mxIXLwJz5tQ+z9edDAP+rK5lvv0W2LULOH268e6uikLEvJnWrQN+/lkzMjnPUSaaeV11\nmypRUbLPR0eztdQ6Us1x9IWGoskmJCRQly5dSCgU0qpVq2SWmT17NgmFQvL29qYLFy4QEdHjx4+p\nZ8+e5OPjQ+7u7rR48WJJ+WXLlpGTkxP5+vqSr68vJSQkyGxXDvX0msOHiezsiG7c0JzMx4+JwsKI\nfH2JsrI0J9dQ0dU+qqt6aYJLl4hsbIiuXNG2Jpz6kKeP1luioqKC3NzcKCMjg0QiEfn4+NCVGv/1\nw4cP09ChQ4mIKCUlhQICAiTXHj16RERE5eXlFBAQQH/88QcRES1fvpzWr1+vkhvQV86fZ4Po9GnN\nyczPJ3rpJaLRo4lKSzUn15DR1T6qq3qpm8ePibp0Ifr+e21rwmkIefpovWtQ1fdcmJiYSPZcVKeu\nUEcAYPrMbUYkEkEsFqNVNT9QagILuPUxZw6bYuvVS3MyxWJg1Ci2sKyp6UQOR5MkJQFt2wKhodrW\nhKMK6jVQ8uy5kFXm9u3bANiOeV9fX9jZ2aF///7wqJaMZePGjfDx8UF4eDiK6woOZaBcusQiL0ya\npFm5Tk7AwoWyA1RyOIbAvn1suwbHMFBLqKOqesbGxrh48SIePHiAwYMHIzk5GUFBQZg1axY++ugj\nAMDSpUuxYMECREdHy2zbEPdybN0KTJ/O3cn1EVWEOuIox61bwNdfA198IX2+ogI4eBB49tXCMQDU\nFuqoOlZWVhg2bBjOnz+PoKAgqdBG06dPx2uvvVanDoYW6qgqdP2lS+qV8+QJi8LM35ZUiypCHXGU\nIyND9vg5fZpFQOE5nAwHtYQ6srOzw3///SeZunv8+DF+++03SSLD/Gqhpvfu3QsvLy+V3pQuExfH\nwhfVsPMqJS+PyeABOjiGyJ07sqPe79sH8KhphoXaQh3l5+cjLCwMlZWVqKysxJQpU/DKK68AACIj\nI3Hx4kUIBAJ07NhR0l5TYMsW4NNP1df+n3+yQTpzJjBunPrkcJSjoZQZO3fuxJo1a0BEsLCwwP/+\n9z94e3sDAFxcXGBpaQljY2OYmJggtYlt9pFloIiA/ftZJBaOAaFuV0Jl0HH1Gs25c0QuLkRiserb\n/u47onffZa7r/+//qb59jmwU6aPybN84ffo0FRcXExHbi1h9+4aLiwsVFRWpXC99YeFCoqgo6XOX\nLrGxVVmpHZ04jUeePspXKDTIli3AW28pti509SqLcn7kiOzrzZsDHTuy1Bjci0m3kWf7Rq9evWD1\nLIx9QECAxDO2CmrC2zRkvUFVTe/xaPqGBfcj0xAPHgB79gCNCUhdWcni5X31FQvbEhHBEu/Jgscb\n0x/kyVRdnejoaISEhEg+CwQCBAcHw9jYGBEREZgxY4Za9dU13nsPsLWVPrdvH7B+vXb04aiPBg2U\nPOml58yZg4SEBJiamiImJgZ+fn548uQJ+vXrh6dPn0IkEuH1119H1LPAWffu3cO4ceOQlZUlSbdh\nrc28whogNhYYNEj24q4srlwBRo5kG2rnzmXz6y1aqFdHjmaQd/sGABw/fhzbtm3DqVOnJOdOnToF\nBwcHFBYWYuDAgejatSsCAwNr1TXELRoAS+BZnexstq+wb1/t6MORD4W2aNQ3/6euUEcLFy6k1atX\nExHRqlWrKDIyUuE5Sn2gspLI05Po2DH56zx6RHTiBJ9T13UU6aNnzpyhwYMHSz6vXLlSZpzLv/76\ni9zc3OjmzZt1trV8+XJat26dSvTSVzZsYPElOfqFPH1UK6GOqtcJCwvDvn37GmdV9YzTp1litcY8\nwJqaMldxPqdueMizfSM7OxujRo1CbGwshEKh5HxZWRlKSkoAAI8ePUJSUlKT2qYhi/37uXu5oVLv\nFJ88c+V1hTqys7ODWCzGiy++iPT0dMyaNUsS6ujOnTuwezbXZWdnJzFohsqWLcztmxsbDiDf9o2P\nP/4Y9+/fx6xZswBA4k5eUFCAUaNGAQAqKiowadIkDBo0SGv3om3u32cpNZrwn8Cg0Uqoo5plGzMn\nr28UFbHwK199pW1NOLrE0KFDMXToUKlzERERkt+/++47fPfdd7Xqubq64uLFi2rXT184fBgYMICn\nczdUNBrq6M8//0RQUBDs7OxQUFAAe3t75OfnS4U+qom+L/R+/z0wfDjQurW2NeGoAh6LT7ucO8ci\npKxdyz7z6BEGTn0LVOXl5eTq6koZGRn09OnTBp0kzpw5I3GSKCwspPv37xMRUVlZGQUGBtKRI0eI\niDlJVC0KR0VFGayTRGUlUadORKdOyV/nu++I9u9Xn04c1aKrfVRX9VKW3buJxoxhv5eVEVlaEhUW\nalcnjmLI00e1Eupo8eLFGDt2LKKjoyVu5obI8ePMNbwxOZ8OHgQmT1afThyOPlN9k+7Ro4CvL2Bj\no12dOOpD8MyS6SQCgUCvd8yPHw8EBgLvvCN/HVdXtjm3c2f16cVRHbraR3VVL2V5/3223vThh8CM\nGYCHBzB/vra14iiCPH2UhzpSE6WlQEJC4wK2PnzInhDd3NSnF4ejz1S9QYnFwIEDwOuva1sjjjrh\nBkpNHDwI9OnTuOmHf/4BunX7/+3de3CU1fkH8O9y8QoIImyAACGbOyHJ0sRQbH4VyeaCEAsiFIQi\nBJqmODQGcScdQWSGEKzAaDIYsIBBMaCtVMcmkQLmghBQ2DRWEIIuRWQ3QC7IreTC8/vjJWsum81e\n3nf33d3nM+MMSd7znrPyHs7uyTnfA/TuLV27mDyUlJQgJCQEgYGBWL9+fZef79q1C5GRkYiIiMBj\njz2G6nYHIPVU1pO1DVCVlYCPjzDjwDwXD1AS2b1bmOKzRXU1cPdEBebBWltb8fzzz6OkpAQnT55E\nYWEhTp061eEaf39/lJeXo7q6GitXrsTvf/97q8t6spwcIdKIV+95hx4HKGverS1btgyBgYGIjIyE\nTqcDICxJnzRpEsaOHYvw8HC8+eabputXr14NX19fqNVqqNVqlJSUiPRy5KGhASgttb0DPfUUkJUl\nSZOYjDiSZm5NWU8WHg4MGgTs3csDlDewuIqv7d3a/v37MWLECMTExCAlJQWhoaGma4qKinD27FnU\n1NTg6NGjSE9PR2VlJfr27YtNmzYhKioK169fxy9+8QskJCQgJCQECoUCmZmZyMzMlPwFusLevUB8\nPDBggG3lhg2Tpj1MXhxJM7e1rCf69lshOiwqytUtYVKzOEC1f7cGwPRurf0A1V0Wn4+PD3x8fAAA\n/fr1Q2hoKH788UeEhIQA8OzzbHbvFlYYMWaOI2nmtpR1903u3dm/X4g28uAAGo9kzyZ3SbP42pw7\ndw46nQ6xsbGm7+Xm5mLnzp2Ijo7Ghg0bPOa4jUuXhGwwD8+/ZQ6wJqEFAKqrq7FkyRKUlJSYgpat\nLQt0HKA8yYEDtq2OZfLQ+U3Sq6++2mMZSbP4AOD69euYOXMm3njjDfTr1w8AkJ6ejlWrVgEAVq5c\nieXLl2Pbtm1m7+1u7wI//BB48knOBvNUYkQdtU8zHz58OPbs2YPCwsIO13SXZm5NWU/W0iL8fvdu\nVgDzcJJm8TU3N+Ppp5/GvHnz8Jt2v9Fsn723ePFiTJs2rds2uNu7wN27ATNnOjIPYc+7wM4cSTPv\nrqw3+OwzoLAQGDXK+oM/mZuzlIPkSBbfnTt3aP78+ZSRkdHlvhcvXjT9eePGjTRnzhy7s5rk5Px5\noocfJrp92/ayqalEn30mfpuYtOT6jMq1XY7IyyN69FEiM/+kMDdkzTMqWRbfF198gffeew8RERFQ\nq9UAgHXr1iEpKQlarRZVVVVQKBQYM2aM6X7u7oMPhGPa77nH9rKHDgEZGeK3iTFPUVsr/I73bqQn\n8wKcxSei6GhhI2F8vG3lbt4EBg8Woo769pWmbUwacn1G5douR6SmArt2AZcvA/37u7o1zFGcxedE\nNTXAhQu2Heve5ptvgOBgHpwYs+TbbwE/Px6cvAkPUCLZswd45hmgj8VJU/M44oixnp07Z9vRNcz9\nuSTqqL6+HhqNBkFBQUhISEBjY6NIL8d17Mnea/Of/wCRkeK2hzFP8/DDwJw5rm4FcypLKyhaWlpI\npVKRXq+npqamHlfxVVZWmlbxGQwG0ul0RER07do1CgoKolOnThGRcKLu+vXriYgoJyfH7U/Ura4m\nGjmSqLXVvvLNzcLpoMz92PuMFhcXU3BwMAUEBJhOl27v1KlTNGHCBLr33nvp9ddf7/Cz0aNH07hx\n4ygqKopiYmJEbZdc1dcT9e9v3wpZJk/WPKMWP0FZE0xpKeoo6m5YVvuoo85lFixYgH+4eezC7t3C\nzvZedk6Y9ukD3H+/uG1i8mVNIvngwYORm5uLF198sUt5hUKB0tJS6HQ6HDt2zFnNdqnPPwcmTrRv\nhSxzXxb/STUXY9Q2yFi6pi15uU3nqKPa2lpTFJJSqURtba1jr8KFiByb3mPex5o3fkOGDEF0dDT6\ndrNyhjxshV5PDhzg5eXeyCVRR52vtVSP3KOOvvpKOGBw/HhXt4Q5gxhRR44mkisUCsTHx6N3795I\nS0vDEi9IJj5wQEiRYN7FJVFHSqUSRqMRPj4+MBgMHaKPOpN71FFhofDpiZOVvYMYUUe2JJKb88UX\nX2DYsGG4fPkyNBoNQkJCEBcX59A95ezCBeDKFV5I5I0sDlDWBFOmpKQgLy8Pv/3tb1FZWYmBAwdC\nqVSCiJCamoqwsDBkdIpISElJQUFBAbRaLQoKCjoMXu7kxg3g3XeBI0fsv0djI/DQQzzAeRNbEsnN\nGXb34LAhQ4Zg+vTpOHbsmNkBSu6zD9Y6cABQqYC1a4GVK13dGmYvu2YfelpFUVRUREFBQaRSqSg7\nO5uIiPLz8yk/P990zdKlS0mlUlFERAQdP36ciIgqKipIoVBQZGQkRUVFUVRUFBUXFxMRUV1dHU2e\nPJkCAwNJo9FQQ0OD3as8XCk3l2jGDMfuMW4c0d3/ZcwN2fOMWpNx2eaVV17psIrvxo0b9NNPPxER\n0fXr12nixIn0mZkQR7n3HVvMn080dSrRyy+7uiVMTNY8oxx1ZKfWViAoSPgENXGiffdoahI+PTU0\nAPfdJ277mHPY+4wWFxcjIyPDlHGZlZXVIePSaDQiJiYGP/30E3r16oX+/fvj5MmTuHTpEmbMmAEA\naGlpwbPPPousrCzR2iU3RMCIEcCkScIm3eefd3WLmFiseUZ5gLLT3/8OvP66Y9N71dXC769OnhSv\nXcy55PqMyrVdtjp1CkhKAmJihK0czzzj6hYxsXAWn4Q2bADMbFGxCUccMWbZgQNC+HJtLWBhLRXz\nUJJFHQHAokWLoFQqMW7cuA7Xr169Gr6+vlCr1VCr1SgpKXHwZTjX4cNCh3F0bQcPUIxZtn+/sP+p\ntpYPKfRGFgcoa3a8FxUV4ezZs6ipqcHWrVtNJ4ACwMKFC80OPgqFApmZmdDpdNDpdEhKShLp5TjH\nhg3ACy8I+58ccf06759irDstLUBZGfDEE8C//iWs5GPeRZKoI6PRCACIi4vDoEGDzN7bXefHv/sO\nKC8HFi50/F6bNwvz64yxrk6cEBZI+PgAo0fzcTTeSJKoo87XmJObm4vIyEikpqa6VZr5pk3AkiXA\ngw+6uiWMebb9+20//JN5FsmjjsxJT0/HqlWrAAArV67E8uXLsW3bNrPXymmzYX098P77wgGDzDuJ\nEXXErHPgANBpjz/zNpY2SR05coQSExNNX2dnZ3c5GiAtLY0KCwtNXwcHB5PRaDR9rdfrKTw8vNs6\nLP28h+Y53dq1RM895+pWMDmx9xl15LiNnso60i65uHmT6MEHia5edXVLmFSseUYtTvG1jzpqamrC\nnj17kJKS0uGalJQU7Ny5EwA6RB1ZYjAYTH/eu3dvl1V+cnT7NpCbC2RmurolzN05ctyGNWU9wWef\nCQuIBgxwdUuYK1kcoPr06YO8vDwkJiYiLCwMs2fPRmhoKLZs2WLa9T5lyhT4+/sjICAAaWlp2Lx5\ns6n8nDlzMHHiRJw5cwYjR47Ejh07AABarRYRERGIjIxEWVkZNm3aJOFLFMf77wthlWKNpaWlwiol\n5n0cOW7DmrKeYMsWYPFi4c8bNworZ5n3sfg7KABITk5GcnJyh++lpaV1+DovL89s2c7Bsm3aPnG5\nCyKhg4g1jl65Ajz1lPA7LeZ9HDluw9GjOtzBuXPAl18CH30kfK3XAwEBLm0Sc5EeBygmTDf07i3e\niqLKSiA21vF9VMw9OXLchi1l5bTAyBZvvw3Mm/fzKdO1tcBjj7m2Tcxx9iww4gGqB62twJo1wPLl\n4h2Jcfiw/QGzzP05ctyGLWXlfpaaOc3NwPbtwMGDP3+PUyQ8gz1nqbkk6qi+vh4ajQZBQUFISEiQ\n9T6otWuFd3LPPivePQ8fFpKZmXeyZvFRG+q0hcOWsu7o44+BwEAgNPTn7/EA5cUsLfFraWkhlUpF\ner2empqazJ5b889//pOSk5OJiKiyspJiY2NNPysvL6cTJ050WUa+YsUKWr9+PRER5eTkkFartXsZ\nopTKy4mUSqIffxTvnk1NwvLZbo7AYm7G3me0p3PWDAYD+fr60oABA2jgwIE0cuRIunbtWrdlxWqX\nq8XHE733XsfvDRpEdPmya9rDpGPNM2rxisOHD3fYB7Vu3Tpat25dh2vS0tJo9+7dpq+Dg4PJYDCY\nvja3z6n9XimDwUDBwcF2vwCp1NURjRpF9Omn4t73yhWiFSvEvSdzHbkOBHJtlyU1NUSPPEJ061bH\n7xuNRHfuuKZNTDrWPKMuiTqqra017ZVSKpWora21+hOfMxAJcUbTpwNPPinuvQcPBl57Tdx7MuYJ\n3n4bWLCg6+GdSqV4v/9l7sUlUUedr7V0vStWIm3dCnz/vbD3ibH2OOpIGk1NwDvvABUVrm4JkxOL\nA5Q1K4Y6X3PhwgWMGDHCYqVKpRJGoxE+Pj4wGAwYauEkMmevRPrmG+Dll4WOcu+9Tq2auQF7ViKx\nnu3dC4wdCwQFubolTE5cEnWUkpKCgoICAEBBQQF+4+jJfyK5dUs4gn39eiAkxNWtYcx7bNkCdNr/\nzxgU1Hl+rpPi4mJkZGSgtbUVqampyMrKMsUctSVKtGWDPfjgg9ixYwfG3z2Fb86cOSgrK0NdXR2G\nDh2KNWvWYOHChaivr8esWbNw/vx5+Pn54YMPPsDAgQO7Ns6KM+vFtHTpz4nlPOfNrOHsZ9Racm2X\nOWfOAHFxwA8/APfc4+rWMGex5hntcYByJWd2sn/8QwiC1emAhx6Spo733gOCg4GYGGnuz5zP3me0\npKTE9MZv8eLF0Gq1Xa5ZtmwZiouL8cADD+Cdd96BWq0GAPj5+WHAgAHo3bs3+vbti2PHjonWLld4\n8UUhVcXcNkutFhgzBvjDH5zfLiYta57RHjfqeoOKCmHV3vvvSzc4AcIJujduSHd/5h6sSSQvKirC\n2bNnUVNTg61btyI9Pd30M4VCgdLSUuh0OrODkzv53/+AnTuF/mfODz8A/fo5t01MPrx+gCovB2bM\nEAanCROkq+f2beDf/+ZPT8y6RPJPPvkECxYsAADExsaisbGxw3YMd/l01JOPPhJOCeguDJZTJLyb\npFFH3ZVdvXo1fH19oVaroVarUVJSIsJLsV1ZGfD000BhIaDRSFvXiRPCwgs+Kp45ur9QoVAgPj4e\n0dHRePvtt53TaIls2WJ5+o4HKO9mcZl521TE/v37MWLECMTExCAlJQWh7YKy2k9FHD16FOnp6ais\nrLRYVqFQIDMzE5kuPP2vtBR45hlg925g8mTp6+OAWNbG3v2FbQ4dOoThw4fj8uXL0Gg0CAkJQVxc\nXJfr5J5mfvSosEDCUpQgD1CeQ/Q08/ZTEQBMUxHtByhzUxFGoxF6vd5iWVdOUXz+OTBrFrBnD/DE\nE86p8/BhYOZM59TF5M3R/YXDhw8HIBxqOH36dBw7dqzHAUpu9Hphav2tt4BOZzKatLQAjY3AI484\nt21MGqKnmTsyFXHx4kWLZXNzcxEZGYnU1FSnppkfPCgMTh9+6LzBCQBeeEH6aUTmHhzZX3jz5k1c\nu3YNAHDjxg3s27evy2kBcnflCpCUBPz5z4ClLZC9ewMNDXxumjezOEA5OhXRnfT0dOj1elRVVWHY\nsGFYvny5TeXtdeAAMHs28Le/Ac6e7fjVr/idIBP06dMHeXl5SExMRFhYGGbPno3Q0FBs2bLFtMdw\nypQp8Pf3R0BAANLS0rB582YAgNFoRFxcHKKiohAbG4upU6ciISHBlS/HJjdvClN606cL+w4tUSh4\nBZ+3kyTqyNfXF83Nzd2WbR9ttHjxYkybNq3bNogxj04k5OutXCkMTr/+tc23YAyAeFl8ycnJSE5O\n7vC9tE5RCnl5eV3K+fv7o6qqyuH6XaG1VThXzd8fyM52dWuYW7AUdd7c3Ez+/v6k1+vp9u3bPZ4H\ndeTIEdN5UJbKXrx40VR+48aNNGfOHLvj2Hty6xbRokVEY8cSnTnj8O0Y60CMZ1QKcmvXnTtEf/wj\n0eTJRLdvu7o1TA6seUYtfoJqPxXRFnXUNhUBCO/4pkyZgqKiIgQEBJiijiyVBQCtVouqqiooFAqM\nGTPGdD+xnT8vLCP39wcqK3m6gDFXWb8eOHRI2HfIcUbMWh4bdXTwoDCdsHy58B9n6zEpyDVSSE7t\nevdd4YSAw4eBHg466ICI+60n88qoIyLg9deFwWnXLiHny5UPeXk58NxzrqufMVepqwPWrBH6YFGR\nbYMTIPSbXbskaRpzEx41QBEBc+cKm2+PHnXuMvLuHDoknKLLmLf473+BP/0JCAwU/lxRIZz1ZCuj\nEXj4YfHbx9yHRw1QCoXwyenQIWDUKFe3RsAJEsxbVFcD8+YB48cLh31+/TWwbZv9hxByigRzSRZf\nfX09NBoNgoKCkJCQIOpG3alTgfvus6+s2Ed5EwFHjgC//KVz6usJ1ycfUvQrZ2tuFgKQ//pX4NFH\nS5GUBIwbB3z3HfDaa7ZP6XVmaYByxd+1pz/Pcuw/FgcoR44FsFQ2JycHGo0GZ86cweTJk5GTkyPR\ny7ON2H9BZ84A/fsDd5NpJK+vJ1yfPEjVr6TU3AycOiUseFi2TJgVGDhQOIG6vBwYPrwUer1wfpOZ\ns0dtdueOkDjRbstkBzxAuX991nBJFt8nn3yCsrIyAMCCBQvw+OOPy2aQEtNXX/H0HutKyoxLR5w/\nL/y+6MIF4b8ff/z5z1euACNHCsfFREcL2zfUamDAAKHs6tXCtJ5YGhqEga67nD7mHSwOUOZy9o4e\nPdrjNd1l8bWVra2thfLuZ3elUtnhnBtHlZUJq/g6+7//A1assHz96dPA8ePWX9/T/efOFQIxGWtP\nqn7lqNOnhbQVhQK4/35hqvy++4QFDjt3AsOGdS2zcKEweLXvOwCwfTswZEj313fW+frBg4VFEsy7\nWRygxMziIyKz91MoFN3Wo1KprG5DTz79FHjppZ6vq6l51abrbb1/Z9Yk+oqJ6xOXSqWyuYxUP1ib\n0QAABPNJREFUGZftidl3gO6nqdtr6ztA91Nz3bH1esD5f9euqNOT67Om7zgti6/9cQFKpRJGoxE+\nPj4wGAwdsvnaO3v2bI8vgDF3I1XGZXvcd5gnsLhIwpFjASyVTUlJQUFBAQCgoKAAv7GUuc+Yh5Gq\nXzHmcXoK6ysqKqKgoCBSqVSUnZ1NRET5+fmUn59vumbp0qWkUqkoIiKCjh8/brEsEVFdXR1NnjyZ\nAgMDSaPRUENDgzXZgox5DCn6FWOeRtZZfIwxxryX7JMkcnNzERoaivDwcGi1WqfUuWHDBvTq1Qv1\n9fWS17VixQqEhoYiMjISM2bMwNWrVyWpx5mbO3/44QdMmjQJY8eORXh4ON58801J62vT2toKtVpt\n8XwxsTQ2NmLmzJkIDQ1FWFgYKisrJa/THp7cf7jviEe2fcfVH+EsOXjwIMXHx1NTUxMREV26dEny\nOs+fP0+JiYnk5+dHdXV1kte3b98+am1tJSIirVZLWq1W9DpaWlpIpVKRXq+npqYms+d6iclgMJBO\npyMiomvXrlFQUJCk9bXZsGEDzZ07l6ZNmyZ5Xb/73e9o27ZtRCScfdbY2Ch5nbby9P7DfUc8cu07\nsv4E9dZbbyErKwt97+7WG2JuY4XIMjMz8dprr0leTxuNRoNevYS/htjYWFy4cEH0OtpvDO3bt69p\nc6dUfHx8EBUVBQDo168fQkNDcfHiRcnqA4RVbkVFRVi8eLHkx0xcvXoVFRUVWLRoEQDh7LOHHnpI\n0jrt4en9h/uOOOTcd2Q9QNXU1KC8vBwTJkzA448/jq+++krS+j7++GP4+voiIiJC0nq6s337dkyZ\nMkX0+3a36dMZzp07B51Oh9jYWEnreeGFF/CXv/zF9A+WlPR6PYYMGYKFCxdi/PjxWLJkCW7evCl5\nvbbypv7Dfcd+cu47FvdBOYNGo4HRzJbxtWvXoqWlBQ0NDaisrMSXX36JWbNm4fvvv5esvnXr1mHf\nvn2m74n1bqK7OrOzs01zvmvXrsU999yDuXPnilJne2Ju2LTF9evXMXPmTLzxxhvoJ+Fxxp9++imG\nDh0KtVrtlDyxlpYWnDhxAnl5eYiJiUFGRgZycnKwZs0ayevuzNP7D/cdL+87kk84OiApKYlKS0tN\nX6tUKrpy5YokdX399dc0dOhQ8vPzIz8/P+rTpw+NHj2aamtrJamvvR07dtDEiRPp1q1bktz/yJEj\nlJiYaPo6OzubcnJyJKmrTVNTEyUkJNCmTZskrYeIKCsri3x9fcnPz498fHzogQceoPnz50tWn8Fg\nID8/P9PXFRUV9OSTT0pWn728of9w33GM3PuOrAeo/Px8WrVqFRERnT59mkaOHOm0up21SKK4uJjC\nwsLo8uXLktXR3NxM/v7+pNfr6fbt25L/ovfOnTs0f/58ysjIkKyO7pSWltLUqVMlrycuLo5Onz5N\nRESvvPIKvfTSS5LXaStP7z/cd8Qlx74j6wGqqamJ5s2bR+Hh4TR+/Hj6/PPPnVb3mDFjnDJABQQE\n0KhRoygqKoqioqIoPT1dknqcubmzoqKCFAoFRUZGml5XcXGxpHW2KS0tdcpKpKqqKoqOjqaIiAia\nPn26LFfxeXr/4b4jLjn2Hd6oyxhjTJZkvYqPMcaY9+IBijHGmCzxAMUYY0yWeIBijDEmSzxAMcYY\nkyUeoBhjjMkSD1CMMcZk6f8BVqCTYrFHsDUAAAAASUVORK5CYII=\n", "text": [ "Next we want to choose our regularization strength. Remember that from a Bayesian point of view, the regularization strength respresents our prior belief on the variance of the feature weights. If we assume a lower prior variance, we are effectively not allowing the weights to deviate too far from zero. Whether we take this point of view or not, the optimal strength is largely an empirical question. It is generally better to test rather than to assume what the optimal strength should be.
\n",
"\n",
"The various implementations of learning algorithms differ in how they treat the regularization parameter. Many implementations formulate the loss function as such:
\n",
"\n",
"
Now lets plot the cross-validated error as a function of C, with a separate series for the different penalization types\n", "\n", "\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "plt.clf()\n", "\n", "norm = 1\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['mu'][(xvals['norm']==norm)], 'r')\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['low'][(xvals['norm']==norm)], 'r--', label='L1')\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['up'][(xvals['norm']==norm)], 'r+')\n", "\n", "norm = 2\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['mu'][(xvals['norm']==norm)], 'b')\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['low'][(xvals['norm']==norm)], 'b--', label='L2')\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['up'][(xvals['norm']==norm)], 'b+')\n", "\n", "plt.legend(loc=4)" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 38, "text": [ "This is interesting. We can see that the AUC for $L1$ is $0.5$ where the regularization is stronger, but for $L2$ the AUC is pretty good in the same region. This is because all of the feature weights in $L1$ are zero for that range. In $L2$ they'll likely be small, but they won't be zero. This means they'll still be able to rank the instances.\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "norm = 1\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['mu'][(xvals['norm']==norm)], 'r')\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['low'][(xvals['norm']==norm)], 'r--', label='L1')\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['up'][(xvals['norm']==norm)], 'r+')\n", "\n", "norm = 2\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['mu'][(xvals['norm']==norm)], 'b')\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['low'][(xvals['norm']==norm)], 'b--', label='L2')\n", "plt.plot(xvals['e'][(xvals['norm']==norm)], xvals['up'][(xvals['norm']==norm)], 'b+')\n", "\n", "plt.legend(loc=4)\n", "plt.ylim([0.82, 0.89])\n", "\n", "plt.title('X-Val AUC by Regularization Weight')\n", "plt.xlabel('Regularization weight C (higher C is less regularization)')\n", "plt.ylabel('X-Val AUC')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 44, "text": [ "