{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "#Reciever Operator Curve\n", "
The Reciever Operator Curve is a useful tool for understanding how well a classification model ranks instances.
\n",
"\n",
"Defining the ROC
\n",
"Assume some function $f(x)$ that gives real-valued output. We define a classifier based on $f(x)$ as the following:
\n",
"
\n", "So that's enough theory for now, let's do a model bakeoff and show how the ROC can be used for model selection. First we'll build a model using four different algorithms and compare their ROCs.\n", "
" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n", " n_neighbors=10, p=2, weights='uniform')" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#Build the datasets\n", "from sklearn.metrics import roc_curve, auc\n", "from sklearn.tree import DecisionTreeClassifier\n", "from sklearn import svm, linear_model\n", "from sklearn.neighbors import KNeighborsClassifier\n", "import course_utils as bd\n", "import pandas as pd\n", "import math\n", "import numpy as np\n", "reload(bd)\n", "\n", "#Load data and downsample for a 50/50 split, then split into a train/test\n", "f='/Users/briand/Desktop/ds course/datasets/ads_dataset_cut.txt'\n", "\n", "train_split = 0.5\n", "tdat = pd.read_csv(f,header=0,sep='\\t')\n", "\n", "lab = 'y_buy'\n", "\n", "moddat = bd.downSample(tdat,lab,9)\n", "#We know the dataset is sorted so we can just split by index\n", "train = moddat[:int(math.floor(moddat.shape[0]*train_split))]\n", "test = moddat[int(math.floor(moddat.shape[0]*train_split)):]\n", "\n", "#Train the models\n", "dt = DecisionTreeClassifier(criterion='entropy',min_samples_leaf = 10,max_depth = 4)\n", "dt = dt.fit(train.drop(lab,1),train[lab])\n", "\n", "lr = linear_model.LogisticRegression(C=1000)\n", "lr.fit(train.drop(lab,1), train[lab])\n", "\n", "mm = svm.SVC(kernel='linear', C=1)\n", "mm.fit(train.drop(lab,1), 2*train[lab]-1)\n", "\n", "knn = KNeighborsClassifier(n_neighbors=10, p=2)\n", "knn.fit(train.drop(lab,1), train[lab])" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEZCAYAAACTsIJzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8VFX6+PHPzKRMyiSZ9E4CgSSUBJAqHUWkiF1RF0Wx\nsrK4fhd1XV1x17Xsruuq/HRxLaCiYsEFqQqKdEIooZcUSO9tJtNn7u+PCYGQBJKQyWSS83698kpy\n5947z6TcM+eec55HJkmShCAIgiDUkzs7AEEQBKFrEQ2DIAiC0IhoGARBEIRGRMMgCIIgNCIaBkEQ\nBKER0TAIgiAIjYiGQRAEQWhENAyCcJG4uDi8vb1RqVSEh4czZ84camtrGx7ftWsXkydPxs/Pj4CA\nAGbNmsWJEycanaO2tpannnqKXr16oVKpSEhI4Pe//z0VFRWd/XIEoV1EwyAIF5HJZKxduxaNRkNG\nRgZHjhzhlVdeAWD37t1MnTqVW2+9laKiInJyckhNTWXMmDHk5OQAYDKZuO666zhx4gSbNm1Co9Gw\ne/dugoODSUtLc+ZLE4RWk4mVz4JwQXx8PB999BGTJ08G4JlnnuH48eOsXbuWcePGkZqaypIlSxod\nM336dEJCQli+fDkffvghL7zwAtnZ2Xh7ezvjJQjCVRM9BkG4xPn3Svn5+WzcuJERI0ag0+nYvXs3\nd955Z5P977rrLn766ScANm/ezLRp00SjILg00TAIwkUkSeKWW27Bz8+P2NhY+vTpwwsvvEBlZSU2\nm42IiIgmx4SHh1NeXg5ARUVFs/sIgisRDYMgXEQmk7F69Wpqa2vZunUrP//8M+np6ajVauRyOUVF\nRU2OKSoqIiQkBIDg4GAKCws7O2xB6FCiYRCEFowfP54FCxbw7LPP4uPjw+jRo/n666+b7Pf1119z\n3XXXAXD99dezadMmdDpdZ4crCB1GNAyCcBlPPfUUaWlp7N27l9dff53ly5fz7rvvotFoqKqq4oUX\nXmDv3r289NJLAMyZM4eYmBhuv/12Tp06hc1mo6KigldffZUNGzY4+dUIQuuIhkEQLiM4OJgHHniA\nN954gzFjxrBp0yZWrVpFZGQkcXFxZGRksGPHDvr06QOAh4cHmzdvJikpiSlTpuDv78/IkSOprKxk\n1KhRTn41gtA6Dp2u+tBDD7Fu3TpCQ0M5cuRIs/v87ne/Y8OGDXh7e7Ns2TKGDBniqHAEQRCEVnBo\nj+HBBx9k48aNLT6+fv16MjMzOXPmDB988AFPPPGEI8MRBEEQWsGhDcO4ceNQq9UtPr5mzRoeeOAB\nAEaOHEl1dTUlJSWODEkQBEG4AqeOMRQUFBATE9PwfXR0NPn5+U6MSBAEQXD64POlQxwymcxJkQiC\nIAgAbs588qioKPLy8hq+z8/PJyoqqsl+CQkJZGVldWZogiAILq9Pnz5kZma2+TinNgyzZs1iyZIl\nzJ49mz179hAQEEBYWFiT/bKyspr0LHqqxYsXs3jxYmeH0SWIn8UF4mdxQVf/WcSmjMHLS4lCoUCy\nSVitNmxWW/3ni7632ZBsoFDIkCvkKBTy+s8y/FVB9IsdSt+YIUSH9KOwPIvSrM/5aNVKzBYzkX1T\nCAyN4OjOTe2K0aENwz333MOvv/5KeXk5MTExvPzyy5jNZgAee+wxpk+fzvr160lISMDHx4dPPvnE\nkeEIgiC02YbNWykuLW/HkRIGvZk6jcH+oTVQV2ugqrqWByY8ir7GCkiog1UEhvgSFKIiMMT+dWD9\n1yp/L2QyGZIkUVluJi/HQF6OAW2thaheSmLjvQjyzWXE6Ps4da6Ca1Nj2JqejZub/dLe3lvzDm0Y\nvvzyyyvuc2kKY0EQhK7kgd/+gRk3TMbD3b3RdkmSMJssGA3m+g8LRr35wvdGM3K5DE+le6OPW6dO\n46HfTSU0PABvX88WL95Wq0RhnpG8bHtjIJdDTG8lw8b4ExbhgVxhP65w00fMHK7m5UfHM+vhVxoa\nhavh1FtJQttNnDjR2SF0GeJncYH4WVzQlp9FdU0tz/3l75gt5iaPSTYJk9GCTqfnvhvvxlhno7JM\nS1WZhspyLdUVWrx9lQSGXXiH3+idf7AKTy/3Zp61ZUaDjfxzBvKyDRTmGfBXuxMTr2TKzUH4q91a\nbERe+Ms/8E++pU3PdTkuUajnfFdKEASho5iMFnbvOcScJ59i7m33oK01UKfRU6cxotUYMOhMePl4\nEBjoz7hhowgO8yMwRIU6uL4hCPbF3ePq31vX1ljqewV6KkrNhEd7EttbSXScEi9vxRWPL9z0DD6x\n1zbbMLT32il6DIIgdAtV1TVMuGk2RpMJsN/qsVml+oFc++eLv5YkCQmJAJU/iWEDCBzoS2Co/Z1+\nYIgvAYE+KNyufGFuK0mSKCsxk5etJy/HgMFgIyZOyYAhvkREe+Lm3vwqguLiYq4b0YdAXznLFg1v\n2G7RV+EbP7FDYxQNgyAIna66phaNtq7dx0uShF5norpcS3VVHdUVWrJz8ikprmTu9fOorqjDarXi\nr/YlIMibgEBf/AN9Gn2t8vNCJpeh9vcjwN+vA19dUxazzT5ekGMg76wBpZec2HglY65TExzm3nCL\nSJIkzNoSkGyNjl/07Au8/f7HqH3kvPDn14m5bXajx939mk7zvxriVpIgCJ0uefQU9HoDCkXL78jt\n7/ht2GwXfT4/vdMmIQPkChlyuRy53D6ls198bz55+00Cg1X4+imdumBWV2cl/6x94Lg430hQmDsx\n8V7ExitR+Tf/nrwubw/5qx9B4RUIgFZv5sE3D5FxVs/NI9T8/fFh9JmzBrmbslUxiFtJgiB0KcdP\nnSHj2MlmH6uqrmHLN1/iKfeiskxLRZmGyvMf5VqqyrUovdwJDFERFKJCfX4aZ7CKoFD7/X0vH89O\nfkWXJ0kS1ZWW+imlemqqLETGKonv68XY69V4eMrQZP6IrchITdNCgAAYy07gHT2SmFs+BECpVGIy\nmfh8xQruvffeTnstoscgCEKHM5ut3PvIQvLyiwjyD8JoNGMyWjDVT+O0mWHaNTMJDQ+on8d/4YJ/\nfoDXU9m2GT3OYLNKlBSZyMvRk5dtwCZBTLyS2HglYVGeKBQXeixmbQnZy6agSphy2XN6R48iYOCd\nABw4cICUlJR2T0Ft77VTNAyCILSZ0WBueHdfWXrhnf7uA/s4eDoDi9lKUU0+00dPY9LIsRdN57S/\n81cH+eLm3vEDu53BZLRRcM5+iyj/nAGVvxux8Upi4r1QBzedUqovPkzNsW+xmevQFewjYd62TotV\n3EoSBKHD6OqM9gt+uYbKMi2VZZr62z32efxGg/nCtM36lboDhsay5dgG+g6KYOLYEcgVcm6dfgNB\ngS2n3ncV2lpLw6rjsmITYZEexPT24pox/vj4Xr6B0+XvwVxbgG/CFPwSZzW7T3FxMUeOHGHKlMv3\nJjqL6DEIQjdy411zKSouveJ+NpuE1WLFWp+jx2qxXcjRY7HPiJG72fPzKBRyFJd8LZe3MKWytIx/\nvPxH7r/7tg59XZ1NkiQqSu0pKHJzDOi0VqLjlIT65iA//RcUMmOrz2U1VBEwaDYho59q9vEFCxaw\nZMkSAgICqKqq6qiXAIgegyD0WJIkUVFZjYTE7n0H+OV/X2A2SdRU1lFTqaOmqu7C19X2z27ucvwD\nffBXexMQ6GP/OsAH/yD7Ni9vj3bP6OkTF9vBr7BzmA16Cs9pyc+1kp9rxd1dRnSsguEj5QSHeiCX\nS9Qc348hNJ7gkU+26dzuqsgm24qLi0lMTKS2tpbExESOHj3aUS/lqokegyC4GKvFSlVFXcMsni3b\nd/HqB2+i9FDirvDgzqH346vyrh/Mrb/V0/C1/Xull4ezX0aXYNBfmFKan12FNzkEyfehlqfjLS9s\n9pjAIQ8RNPzRq3re870EmUzG559/7rAZR2LwWRC6mKKSUvakH2zzcRazDa1Gj7bWgLZWj6ZGT935\nr2sN6OuMePl4ovJT4uvnRVF1IRWacpa88rcOTdXgqqxGLbq8XS1eMzR1HhSV+lFUqqJGoyQ0SEtE\nqAa3zFeIv+WfKIMTHR5jeHg4AQEBHD16tEOS3rVE3EoShC7mo8+/5pvV60nu16fRdqvVhsloxWQ0\nYzJZMBst9qmc9R9WqxV3Dzc8POs/PNzx8HTD3dMNT383vEMUyGRG6qilzgRyb7hvxk0kp8a0EEnP\nos35hbKd/0QZlgKAJMmoMUZRpkugTNcXq82DYO8zxHhnkhJ1DoXcClqQRSXg7tu0HowjFBcXd8rz\ntJfoMQhCB5IkiTqtkb/98/+xaeuvDE5MYfroaRcWb5VpMZutDVk41U0ycvrip/ZBLhclbq/Eoq+k\nYt9SkKyNtpsqc5A8Q5D6vURujoH8swa8fRX2KaW9lQSFuPeYEsKixyAIncBmk9DU6BqmcFaWa+rn\n8WupLNdQUapBoZDz3paPmXntDIYkDEEd7EufpPCG+/zOTtXQXZgqs9Bmb0GdOgcAvdGTkspgiutu\norI4kFBTHTHxSoaMVOHr55xL3YIFC8jNzWX16tVOef72Eg2D0KOVV1Rx6/2PYTLZ8/FLSA1TN61m\nK5b66ZsWc/3UTosVmUyGwk2Bm5u80WeFmxy381M5ZfDZ56/g4SEGedurZNtr6PL3tvi41ajD6HUN\neeZbyc0xoKmxEB2nJHmkkqhYJR6ezU+p7QwXzzgaOnSo0+JoL9EwCN2ezWZDW6cD7Kkaaiq0VFVo\nqSqv49iJTLKy8rnvut9QW6VDW2vAK9ADf7V92qaf2gd/tRf+ah/81N74BXjj0YqBXV8fb9EotJJk\ns2AzG5ps1xfuRz34fpTBSQ3brFYoK3OjoMCNgkJ35Ho5sUZbk6pmznTxjKMVnZzjqKOIhkHoVhql\naqi/p//V2lWs2b4WhVyBJIFMZr/3av+A/n0TefjxWQQGqwgI9sXdRVM1uKrin1+i9uQPyBSXXI5k\nCrzCU8E7noJz9oVmhbkG/NVuxMR7MWWEkoDAlquaOUN0dDQFBQUN6xIcOePIkcTgs+BSdHXGhgv+\n+QHd8tJa9h0+RE2VFrPJgo+fF37+Xvj6K1H5e7H7cBrJ/frwp6d/i3+gDwqF824xdGfGijOYa/Lb\nfFzF/v+iTrkPv8QZDds0NRZy67OUVpTYq5rFxNurmnn7dN2G+4knnmDcuHFdppcg1jEILk+SJLS1\nhkbplyvqE7RV1X8tSVJD+uXAUPuMHo2xhoef+z1jRl6Dwk1Bc28g5z80hykTx3b+i+pBzn1tLx4j\n9/Rv24EyGcGjf4/WGk/u+apmehsx8Upi4pVExrRc1Uy4PNEwCC7FoDdxeN9Zsk4WNyRrqyrX4uam\nqJ+943tRcXU/e0bOEBU+vp58/MU3ZGWfazhXRVU1e9IPkrFtvRNfUc+kyfwJfbF9EV/tyTVEzXgX\nr4ghrTrWYrZRlG8kN7u+qplSbm8MensREtZzppQ6kmgYhC7PYrZy/FAe6TvOcPxgHn2Sw0keHENQ\nqF/9PP7WpWpIGnU9t8+8kUB1QMO2hN69uHla18hM2ZPk/W8ebj5heKjjkcndCBh4F3IPnxb31+us\nDVlKi/ONBIXaq5rFxCvxC3CN+/HnZxzJ5fIOT3rX0cQ6BqFLstkkMk8Ukr49k0N7s4mICWTY2ATu\nmjcWXz+vyx67+O9v8/O2XU22FxaX8Njce4mNbpqYTLg6mqzNVKR/0Or9TZVZRM14F5/Ya5t9XJIk\naqos9l5BjoHqSjNRsUri6quaeSpd6xbRpTmOuivRMAgdTpIk8nPK2bcjkwM7M/H1U3LN2ASe+/sd\nBIaoWjzOarVitV5Yxbp99z7uu+NmBg/q32g/Tw8PYqIiHBZ/TyXZrBhKj+Op7kPAoLtbd5BMjjK0\n8e/HZpMoKayvapZjwGalYaHZpVXNXEVzmVBddcZRa4hbSUKHKS2qZv+OTNJ3ZGK12hg2NoFrxiQQ\nERPYquNHTrmVY6fONNxbVijk/LrmK1IHJjsybKFe5odjseirCB33LIGD72/TsSaTjcJzRnJz9BSc\nM+Dr59YweBwY7PrjBT4+Puj1eodmQnUEcStJcIqaqjoO7MoiffsZKsu1XHNtH+Y8OYleCaFtuhjs\n3X+I4tIyDm/fQO9eIhmco9nMOvRFhxptsxqq6fd4OnL3y9/iO0+ruaiqWZGJ0EgPYuK9uObaK1c1\nczUZGRnExcV1617CxXrGqxQ6lK7OyKE92ezfmUludhkpw+OZec8I+g2MatcaAZ1Oz6Sb7+WGSeMI\n7gZlIF2BJvNHSnf8E8+ghIZtPnETkSlaHvyXJImKMjN59eMFdVor0XGe9Bvgw6Rpgbh7uNZ4QVsk\nJCRceaduRNxKElrFZLRw9MA50ref4fTRQhJTohg2ti8DhsTi4enG1p17+HVnWrvObTabef+TFVRk\nHujgqHsmQ9lJNJmbLruPsfwkck8/Im9447L7Wa0SRflGe88gW4+bu4yYeC9i45WERHh0uyywxcXF\n7NixgzvuuMPZoXQIMV1V6HBWq41TRwpI33GGI+lnie0dwrBxfUkdEY+3j2ejfR9c8Aw6nZ6UAUkt\nnO3ywkKDefg3rRzwFC6rfO8S6vL24BMz6rL7eUePwjtqWJPt9qpmRvJy9BTmGVEHuRPb2z5e4K92\nd1TYTnd+xpGvry8ajcbZ4XQI0TAIHUKSJHJOl5C+I5ODu7MIDFExbGwCQ6/tg7/6wvz0n7Zu5+/v\nXpjWeOpMNn9f/Byzb7vJGWELFynfuwTJZiVk9MJWH1NbbWlYdVxZbiYi2pOY3kpi4pQovbrXeMGl\nuvOMIzH4LFyVorxK9m0/w/6dmbi5KRg2ri9P//UWQiKaT29w+NgpwkNCePh++7t8mUzG8MEpnRmy\ncBVsNomyYlPD4LHJZE9BMegaFeHRnri5da9bRC3pDplQHUH0GHqwyjIN+3fap5fWaQxcMyaBYeP6\nEh0XhEwm48a75vLrzubz4UuSxF//+DSLFlxdUXSh4xhKj3F25V1gsxA6/o8EDpnb6HGzyUZhnn28\nIP+sAS9vObG97auOg0Jdf0ppe/Tp0wd3d/du1Uu4mOgxCK2irdVzcHc26TvOUFxQzeBRvbnzoTH0\nTopALpdx6MhxctPseYjO5RWwe+N3LY4b9MQLSVdi1pZgrslr+N5QegzvyKHE3LYcsP9udForeWft\nA8clhSaCwz2IjVeSOkKFyklVzbqSrKwsZ4fQJYm/jB7AqDdzOD2H9O2ZZJ8qpv+QWK6/ZQjJKdG4\nXVR7QFtXx5jpdzJiaCoAMVERRISH2iuSCV1O6bbXMJafRKG8MMXXO2YMVRVW+6rjbAO1NRaieinp\nk+TN+KmBTq1qJrgOcSupG8s+VcyvG45y/GAuvZPCGTa2LynD4vD0ajqz5H/rfyTj6EmWfLicstP7\nnRCtcDnVR7/BoqtotK321A8Ej1qAb++pFBfYbxHl5hiQyewpKGLjvQiL7BpVzZxtwYIFZGRksG3b\nNmeH0qm65KykjRs38tRTT2G1Wnn44Yd59tlnGz1eXl7Ob37zG4qLi7FYLPzhD39g7ty5TYMUDUOb\nVJZpWL1iL1kniphy6xCuubbPFRPWDbtuFsMGp5A6IIknHvpNJ0UqtNapJSmoU+9DJrc36maLO6Wa\nCKq4lqJC8AtwIzbei5jeXa+qmTNdPOMoNTWVQ4cOXfmgbqTLNQxWq5XExEQ2b95MVFQUw4cP58sv\nvyQ5+ULem8WLF2M0GnnttdcoLy8nMTGRkpKSJoNAomFoHaPezI//O8j2H48xYdpArp81GE/l5eed\nn8nK4Q8vvcaOPelsW/sVA5L6dVK0wnmWujKKNr8AkqXFfepydxHxm/0U5NrIyzFQXmIiLMqT2Hgl\n0fFdu6qZs1yaCbUnzjjqcoPPaWlpJCQkEBcXB8Ds2bNZvXp1o4YhIiKCw4cPA1BbW0tQUFC3nBng\naDabRNqvp/jhyzT6DYzij/+4E3Wwb6uOzTqbS3lFJd988v9I6tvHwZEKzTFrizFV5RA28YVG2yUJ\nqqo9KSzxocj/RQ6tqiEmXklyqg8RMYG4i6pmLYqPj+fs2bPdbl1CZ3HYT6ugoICYmAvJ0KKjo9m7\nt/HUx0ceeYTJkycTGRmJRqPh66+/dlQ43Vbm8UK+W7YLhZuCRxZNJa5vWIv77t1/iGl3PYhNsjVs\ns1pt3DJtCpPHje6McHs8U9VZzn55K5LNetFWG14RQ/GNG4/FIlGUZ2hYX+BZX9VsTKqS4LDul4LC\nUe69914GDBjQI3sJHcFhDUNr7nG++uqrDB48mK1bt5KVlcWUKVPIyMhApWqas3/x4sUNX0+cOJGJ\nEyd2YLSup7ykltWf7+FsZimTbhmIKlROeV0p5YdKWzxm7/5DjBl5DV9/tKTRdg+P7pvmoLMYy09j\nsxiuvF9lJu7qeHrd+UXDNr3ORkGulS1rKyjONxIY6k5svJKBQ0NcpqpZV/O3v/3N2SE4xdatW9m6\ndetVn8dhf3VRUVHk5V2YY52Xl0d0dHSjfXbt2sWf/vQnwL7QJD4+nlOnTjFsWNP8LRc3DD2ZXmfi\nx1UH2LnlBJNnpjDnyck8+5fXWPvjL4QGX7nuwfQpk/DyUnZCpD2HVV9Fzhe3oAxpXd0Ir4hh1NYq\nGnoFVRX1Vc0SvBhzXUC3T0HR0SwWi7hVVO/SN80vv/xyu87jsJ/msGHDOHPmDGfPniUyMpKVK1fy\n5ZdfNtonKSmJzZs3M2bMGEpKSjh16hS9e/d2VEguzWa1sfuXk6xbmU7y4Bief/MuDBYd7y/7jAOH\nj7LoyUd49IF7nB1mt2Y11lJzbBUStkbbbaY6FMoA4u75rsVjbTaJ0iITedkGcnP02HIqiKlfaBbu\nolXNnO38jCOz2YxOp3N2ON2KwxoGNzc3lixZwtSpU7FarcybN4/k5GSWLl0KwGOPPcbzzz/Pgw8+\nSGpqKjabjb///e8EBrau2ldPcupIAauW70Lp7cHjz00jtk8IAF+t+pmPPv+aaddPYMK1I50cZfdn\nKD5C5cFlqPre2OSxoBGPN9lmNtkoyDWSm11f1Uxlr2o2cVpgt6hq5kw9pfays4gFbl1YaVE1//ts\nDwVnK7hlzigGj+rd6GLy22f+TGVVNV/+9x0nRum6JJuVop+ew2bStmp/i64SuYcPsbd+3OI+dfVV\nzXLPVzWL8GjIUuqjErc7rlZ3zoTqCF1uuqrQfro6Ixu/3c/eX09z/axUHnzqetw9mv6qamo0DB7U\nv5kzCK0h2czUnt5A1PR/t/oYj4C4xueQJCrLzOTWjxfUaaxE9+oZVc2cISkpCY1GIzKhOphoGLqY\n4vwq3v3LDwwY2os//esu/AK8m+yz4LnFfLN6Pdo6HXffOtMJUXZtuvw08tc+CVzhnZIkofDwQdXn\n+jad32qVKM6/kIJCoYDY3l6MGOdPaDesataVnDx5kuDgYNFLcDDx0+1CKkprWfLKWm66dySjJia2\nuF9eQRHvvPYSN0wah79f06m9PZ1FX4F31DVETHn9ivtersbxxQx6KwXnjOTm6CnMtVc1i4lXcsPN\nQfirRQqKzhIeHu7sEHoE0TB0ETVVdbz7l7VMuXnIZRuF81S+PgT4+3VCZK5JJndHoWy+yFBrNapq\nVmYmIsaTmHgloyYE4OUtppQ6UnFxMWvWrOHRR0W9D2cQDUMXoNUYWPLXtYyalMiEaQOb3aesopJV\nP2xEkiTO5RV0coSuQV90CEPpMQylx9p1vM0mUV5iIje7vqqZ0V7VbOBQFRExPaeqmbOdn3Hk5eUl\nGgYnEQ2Dkxn0Jt7/2zoGDIll6m1DW9xv8687effD5Vw3bgwTx4wkZUDrFlP1JBXpHyBZTbj7x+AX\nP6lVx5jNNgpzG1c1i4n3Yuz1aoLDxJTSztTcjCPBOcR0VScyGS289+o6wqLUzH5kXIsXocLiEu59\n9CniYqNZtuQfnRxl16cr3E/Voc/QF+4nbNJLVxxM1tVZya8fOC4pMBIcdmFKqcpfvFdyBpEJ1THE\ndFUXYzFb+ehfPxIQ6MPd88Ze9p1p9rk8Kquq+eBfPTP/y5Xoiw5hM+sJHf9HfGKaJgOUJInqCkv9\nlFI9tdX2qma9E70Yd4MaT1HVzOl27twp1iV0IeI34AQ2q41Pl/yMTCZjzm8nIVdc/sK06Wd71al+\nCT0nXYhkNZH9+U3YTHVX3tdch3rIA/j1m95ou9UqkXVSx9EDWmw2iZh4JUNH+xEWKVJQdDUHDhxw\ndgjCRUTD0MkkSeKr/25DW6Pnieeno3C78uwWm9XGHbOmdUJ0zmc1arHqyrFZDFg0RfR5cHOrjru4\n7rHFbOP0cXuDEBDoxrWTAwiL9BDjBYLQSqJh6ESSJPH9Z3sozK3kyRdnNruauTkymQwvZc/IiFq8\n+Xl0hQeQu3vjFZGKm09oq481mWycPFzH8QwtoeEeTJ4eSHBY69YpCJ1jwYIFbNu2jYyMDGeHIlyG\naBg60cbvDnAyI4+Fi2eh9Gr9BevE6UxGDRviwMg6n9WkRXNmE0iNM5Waqs8RPnlxm1YjG/RWjmfU\ncepIHVG9PJl6SzDqIFFjoiu5eMbRwIHNT8kWug4x6tZJfll3mLRfT/HkCzPxUbXt3X/GsZNER0U4\nKDLn0BceoHz3v9EXHWz0oQwbhGdIUqvOodNa2be9hlWflWDQWZlxVwjjbwgUjUIXs2DBAiIiIhpy\nHB05csTZIQlXIHoMnWD3Lyf5+YfD/P6vN+Onbpr76Er8/XwZlNzPAZF1vrLd/8asKcKiLcEzuB8R\nU15t8zk0NRaOHNBw9oyehGRvbr43DB9fsRK5K0pMTOT06dNixpGLEb8lBzu4O4sfvkhj4eJZBIaI\nvEZVGV8QMnoh8mhvPIP6tunY6kozR9I15J8zkjjQm9vmhIlqZ13cE088QWhoqFiX4GJEw+BA5SW1\nfPXf7Sx4cSZhUQEt7jd/0Yts/nVni48XFpfi6eGag6i1ZzZQuv3CojybSYtf8s0oPHxbfY6KUhOH\n0zWUFJpITvXltgkBYu2Bi3jqqaecHYLQDqJhcKD8s+XE9wsjOj642cdrajXU1Go4dOQ4r//5GYam\nNj8o5+5Ip0sfAAAgAElEQVTmTlREmCNDvWqSJGHRFsMlqywNpcfx7TWWwOH2nDdyhWerG4WSQiOH\n92moqjAzYKiKsVPUuLuLBqGrErWXuw/xW3SgsuIaQiNazvB5450PUFhcilLpyaDkROJiojsxuo6l\nzd5C4cb/a7Se4Lyg4Y/h4de61yZJEoV59gZBp7Uy8BoVk2cGiQVpXdj5GUd6vR6TyeTscIQOIBoG\nByorqiEmPqTZxzZs3kppeQXrV37MgCTXGlg2VmRiKGucwVRfdAjf3pOJmvZWu84pSRK52QYOp2uw\nWiQGDVMR39dLFL3p4kTt5e5JNAwOVFpUw9BrE5pslySJW+9/nDl33UpUhOsVHqlIX4q5Jg/3S3oB\nvvGT23wum00i54yeI+kaFG4yUoapiO2tFKuUuzhRe7l7E79JByovrr3sraT//vu1Tozm6tWeWktd\n/l70RYcIHvUk/kk3t/tcVqtE5gkdR/dr8PZVMHysP5GxnqJBcBGpqami9nI3JhoGBzEZzdRpDAQE\n+jg7lA5Tc2ot7qoIgoY9gk+vce06h9ls4/QxHccOaFAHuTP2ejVhUZ4dHKngaKdOncLX11f0Erop\n8Vt1kLLiWoJC/ZrNnHrPI7/Dw8O1Vucayk5Sl/ML0bd8iG87GgWT8UIeo7BIDybPDCI41DWn4AoQ\nENDy9GvB9YmGoQPV6XTU6fQAnDxxDqW/nNLyiib77d1/iF9Wf9HZ4bWJRV/ZaOqpqTITz6B++MSO\nadN5DHorxw9pOXVUR3QvT268LZiAQNdqFHuy4uJili1bxnPPPefsUIROJCq4daCBY6dSWVWNQq7A\nZLQgSRKeyqYXQU9PT9K3rCbA388JUV5ZXf5e8r6fh8Kz8UptVd8bCZ/0UuvOobVy7KCGzBM64vp6\nMWioSlRHczHnZxx5enpiMBicHY7QDqKCWyeyWCxs2PwrZou50faKymr2bV5NdGQ4K97fSmyfEMbd\nMMBJUbZeXe5urMaahu+NpcfxiRlNzC3/bfO5amssHN2v4Wymnr7JPiKPkQu6eMZRcnIyhw8fdnZI\nQicTDUM7HD+VybyFz3Ld+GsbbZ86eTxBavu917KiGoaNbTpVtSvK+/5BfPtMaTQjyLd326aeVlWY\nObJfQ8E5I4mDfEQeIxd18boEMeOo5xINQztIkkRcTDRf/vedZh+32SRKCqsJCW95qqqjSDYr5bvf\nxmbWteUooma8066pouUl9jxGpcUm+qf6MmpCAB4ij5HLOnbsWEMvQcw46rnEb94B9v56isBgX9TB\nrU8U11FsZh2VBz8hZOyiVh8TMfXvbW4UiguMHE7XUF1pYeBQX8bfoMZN5DFyeT///LOzQxC6ANEw\ntMPrb79PRVV1s4/ptEbWrNjL43+c5vDFWhX7P6T25A+NN0o25O7eBA6+v8OfT5IkCnLteYz0OiuD\nrlHRJ8lb5DEShG5GNAztkH02j3+83Pz0vbVfpZEyIp5efVpfq/g8q0kLbZhAYCg+jKrvNHzjJzTa\nLvfs2LoPkiRxLsvAkXQNVptEyjAVcQkij5Ermz9/Pps2bSIrK8vZoQhdkGgY2sFL6UlEWNMLf15O\nOQf3ZPOnt+5u8znr8veSt+pB5O5tKfspI2DQbJQhyW1+vtaw2SRyTus5nK7B3V1G6ggVMfEij5Er\nu3jG0YABXX/GnOAcomFoJUmS2LJtFyaTicrqmiaP22wSX3+4nZvuGYHvFWo660uOYK0rv2TbYXx6\njSPm5qUdGnd7WCwSWSd1HNmvwVelYOR4fyJiRB4jVzd//nzef/99MeNIuCLRMLRSbkEhdzw4n0lj\nR5GY0JteMVGNHk/79RQ2m8SoSVcuZF+w7nd4+Mcic2vcgPjGtS//UEcxm22cPlrHsYNaAoPdGX+D\nmtAIkceoOxg4cKCYcSS0mvjraKXTmTl4uLvz/adN39HrtEZW1w84X+m+u6HsBBZNIb3u/AJ3VYSj\nwm0To9HGyQwtJw7XERblwXU3BREUIvIYdSfPP/88gOglCK3i0PmFGzduJCkpib59+/LGG280u8/W\nrVsZMmQIAwcOZOLEiY4M56pkZp9tsaBOWwactTm/4hHYBzeftg9OdzS9zsr+XTWs+rSY2horN94W\nzKRpolHoju69917RKAit5rAeg9Vq5cknn2Tz5s1ERUUxfPhwZs2aRXLyhYHS6upqfvvb37Jp0yai\no6MpLy+/zBmdR5Iknv3LGzwyZ3aTx9oy4GyzmCjf/RZBI+YjkztvVXCdxsLRg1qyTuqI7+fNzLtD\nUfmJzmN3YTAYUCrbMolBEBpz2NUgLS2NhIQE4uLiAJg9ezarV69u1DB88cUX3H777URH2yuBBQcH\nOyqcdrHZbJjNFqxWKyaTmT89/dtLHm884CxJEpLV3MLZwFafjyhw6EMOjbsltdUWjuzXcC5LT9/+\nPtxyXxjePiJtRXdxfsZRXV0dFovF2eEILsxhDUNBQQExMTEN30dHR7N3795G+5w5cwaz2cykSZPQ\naDQsXLiQOXPmOCqkNnvy2ZdY/tUqFAo5URFhBKob56C/dMC5bOc/qDywDJm85Tt0HoF9mmQtdbSq\nCjOH0zUU5hpJEnmMuqWLZxyJ2svC1XJYw9CaqY1ms5kDBw6wZcsWdDodo0ePZtSoUfTt29dRYbWK\nJEns2JvO6awcli35B3fePL3JPpcOOOsKD2CsyCRs0p9RD2p6y8kZyorteYzKSkwMGOzL6EkBeHiI\ntBXdiai9LDiCw/6CoqKiyMvLa/g+Ly+v4ZbReTExMQQHB+Pl5YWXlxfjx48nIyOj2YZh8eLFDV9P\nnDjRoQPVuQWFzLxnHmNHDiMxoXez+6xdua9hwNlq1JD77W/wjh6JZ1DzA9SdRZIkigvsDUJtlT2P\n0YQbA3FzE2sQuqORI0eK2stCg61bt7J169arPo/DCvVYLBYSExPZsmULkZGRjBgxgi+//LLRGMPJ\nkyd58skn2bRpE0ajkZEjR7Jy5Ur69+/fOMhOLNTzyRffknH0OBu2/MqpvVua3Sc/p5yP3ljJw/cY\n8fCQI1mMVB/7ln6P7+uUGJsjSRIF5+yJ7Qx6ex6j3okij1F3ZzAYcHNzE70EoVldrlCPm5sbS5Ys\nYerUqVitVubNm0dycjJLl9rXATz22GMkJSVx4403kpKSglwu55FHHmnSKHS2v/7zHe6+ZSaLn32q\n2cdtNomvP9rO9Km+GM+uwTNpFjIPd0LHPdvJkV6IJzfLwOF0DZJkz2PUS+Qx6jHE7CPBEURpz4uU\nlJXTK3UsZw9tJzw0pNl99vxyku0/HmfOjScxVZwm9rZlDo+rOTarRPZpHYfTtXh4ykgdriI6TuQx\n6q6Ki4t55513ePXVV50diuBC2nvtFCORFykuKSM4UN1io3B+wPmuh8cimXV4RQzt5AjteYxOHtby\n3WclZJ3UM2qiPzPuDCEm3ks0Ct3UwoULiYiI4J///KezQxF6CNFjqFen0xEzaAzDh6aw6Zvlze6z\n/92Z+NjONHwfOe0t/Po1nbHkCGaTjVNH6zh2SEtQiAcpw1WEhosVyt2ZmHEkXK0uN8bgaoxGEx4e\n7mz8elmzj+dll+FuziXsju8IiLGnK+6Md+hmk41jh7ScPFxHeJQnU24KJjDE3eHPKzjX//3f//Gv\nf/1LZEIVnEI0DBeRyWTNXuxtNokflq1lcpARVUhkp92ykSSJbT9WIZPBtNuD8VeLBqGnyM3NFb0E\nwWnEX1y9gqISqpqpswCQtu00Slk5cg9f3LwCOy2m08d01GmszLgrREw77WG++eYbZ4cg9GBi8Lne\noaPHm13MZjZZWPtVGuPH+OChbn6xmyPUVls4sLuW8VPVolEQBKFT9fiGoVaj5fYHnuCt9z9m+NCU\nJo/v3HyC6LggpJzleIWndkpMNqvEth8rSR2hIiBQ3D7qzs7POBKErqTH30oqr6wi/dARPnz7NZL6\nJjR6zGgw8+P3B3ni+anUrakmaPijnRJTRroGD085ySk+nfJ8Que7eMbRxdkABKEr6LE9hg8/X0lA\n71QGT5hOWEgwUyaOIyaq8Tu3tO8/474Bf0e37nrknn6dUlyntNjEqSN1jL1OLdYldFPnewnncxwd\nP37c2SEJQiM9ssdgNJrIOHqC3z40hxf+70nc3Rv/GCRJojbvKPmHdxI5fCLxs/6JTOb4NNVmk43t\nP1YyamIA3r4iLXZ3NHz4cNLT08WMI6FLa3OPQZIkVq5c6YhYOs336zbx3Q8bGJIyAC8vZZN/Tou2\nmMLv76ZfeB7q+OHI3ZTIFI6/179vRw2hEZ7EJXg5/LkE53j11VdZsWIFJ0+eFI2C0GW1+Jep1WpZ\nunQpWVlZDBw4kMcff5zVq1fzpz/9iYSEBO6++8qlLLuqjKMnmDJxHHfMmtawTZIkqo98ic2sx1Bb\njtboS/wD3xIY5tcpMeVm6ynMMzLrHufXghYcZ8qUKc4OQRCuqMWG4f7778fPz4/Ro0fz448/smzZ\nMpRKJV988QWDBw/uzBg7XE5uHr17xTbaZjPWULrtNdSpv+Hs6UJ0vjMJ7qRGQa+zsvuXaibeGCgK\n6XQjWq0WX19fZ4chCG3WYsOQmZnJ4cOHAXj44YeJiIjg3LlzeHm59m2O3PxC/rf+J1Ys/TcAlYc+\nRZe7C8lmRu7ujXLgAlZ/vJI//vPOTolHkiR2bqkiIdmbsCjPTnlOwbHOzzjSaDTYbDZnhyMIbdbi\n21OFQtHo66ioKJdvFMDeW1AH+DN18jgA6s7+imdIfwJS7iV61lI2fX+AkRMSUQd1zju908d06Ots\nDB7ZOb0TwbEunnEkai8LrqrFHsPhw4dRqS4Urdfr9Q3fy2QyamtrHR+dA2z4aSvhoSH4+vhw7uvZ\n6IsPEzT8CbyjhlFZpmHf9m958d+dU7O5psrMgd21TLs9WKxudnEiE6rQnbT4l2u1Wjszjk5RXlFF\nWWUld986EwBzbT7xc9biWZ/qYuN3+xk7pT8qf8f3jGxWie0/VjFYrG7uFq6//npRe1noNlpsGPR6\nPf/5z3/Iyspi0KBBzJs3z+XfAT3y++c4fiqTW6dPbdgmd7evLi4rquFw2ln+/M49nRJLxj4Nnko5\nSWJ1c7dw6NAhAJf/HxEEuEzD8MADD+Dh4cHYsWNZv349x48f5+233+7M2DpUVXUNv+5MY+VH7zBl\n4rgmj6//Jp2JMwbh7ev4AeDSIiOnjtUxa3aoWN3cTYgGQehOWhx8PnHiBJ9//jmPP/443333Hdu2\nbevMuDrcjj3pGIxGBiT1a/JYUV4lJw/nM3H6IIfHYTbZ2P5TFaMnBODtI1Y3u5ri4mLmz5/v7DAE\nwaFafJtz8TsgV3439Pk3/2PD5q0UFJUwY8okIsPDKPn1b1jqSrEaapDJZKxbmc71Nw9G6eX4Uplp\n22sIi/Skl1jd7HIWLlzIO++8g0Kh4L333nN2OILgMC1e8TMyMrrFrKQNm7cSERbKrTOm0j/Rnj21\n+ug3hE9ejF/SLAqLIed0MfcvmOzwWM5l6SnKN3KzWN3sUpqbcSQI3VmLt5JSU1PRaDQNHxaLpeFr\nV2kUAL77YSPjRg/njlnT6BcbRObHk5BsFlR9p6HqfR3rVqYz9faheHg6tlekq7Oye2s146aocRer\nm13GokWLGmVCFTmOhJ6gW/+FS5IEwHXjr8Wiq8BYfhokib6P7kLu5kn2qWKK8it5eNHUK5zp6uPY\nuaWKfv19CIsUq5tdiclkEusShB6nxb/0srIy/vWvfzVcXC8mk8l4+umnHRpYR9iTfhAATw8Pzn52\nEzKZDK/wFBSefuh1Jr5c+isz7x6Bu7tjB4FPHa3DoLcxeITqyjsLXYorz8QThPa67AI3jUbTmbF0\nOJPZzLjRw3F3dwebidjZ3+HuG4bNamPZvzeTkBzB8PF9HRpDTZWZg3s0TLs9GLlY3SwIggtosWEI\nDw/npZde6sxYOs33n+3BYrFyx4NjHLqOwF67uYohI8Xq5q5u4cKFfPrpp1RVVTk7FEFwum570/ST\nL77l2zXrAbAaqrHUlQGwc/Nxjh04xx9evQ2Fm2NvIR3ap0HpJSdxkFjd3FVdOuNIEITLzEravHlz\nZ8bR4b77YQMp/ZN47cVnMNcWInNTkp1jZu1X+3jsuWkOX+FcWmTk9LE6xojazV3WpbWXT5486eyQ\nBKFLkEnNjS53MTKZrNlB8JbYbDa8o/uz6dvlTLh2JLnfP4SxIpOP98xj7sLrSRwU5cBo7aub13xZ\nyrCx/vTqIxaydUUTJkxg27ZtYsaR0K219dp5Xrf8bzhfHGX86BFIkg1TZTbFvg+RlBLt8EYB7Kub\nw6M9RaPQhb311lucPHlSZEIVhGZ025VWCoUCmUyG5sxGLNoidu6sYfLMFIc/77ksPcX5RkaM83f4\ncwntN3ToUNEoCEILumXDcDorp6GehLH8FGb1BLyC4onpHeLQ521Y3XyDWN3clZSXlzs7BEFwKd3y\n6vXOB8sZkGRfn2CsyCQ3z+Tw3sL51c2JA3wIjRCrm7uC4uJi/P39CQkJwWKxODscQXAZ3a5h2JN+\nkJ1705k/bw4A2vyDlGqjGDC0l0Of99SROox6G6nDxermruDSGUdicFkQWq/b/bf8vH03Cb17MfOG\n+myppgr6jb0BudxxU0arK80c3Kth+h1idbOzidrLgnD1HNpj2LhxI0lJSfTt25c33nijxf327duH\nm5sbq1atuurnXLtpC8l9EwgLCabwVAYA10y85qrP2xJrfe3mIaP88FeL1c3Oduedd4pMqIJwlRz2\nX2O1WnnyySfZvHkzUVFRDB8+nFmzZpGcnNxkv2effZYbb7yxXfNtL6XR1jH1uvEAHPplB1GKKJQ+\nvld93pZkpNXi5SMncaC3w55DaL1ffvkFcO3iUoLgbA7rMaSlpZGQkEBcXBzu7u7Mnj2b1atXN9nv\n3Xff5Y477iAk5OpnDNVqtJzJPktUeBh1GgNRdUvxC4+/6vO2pKTQyJnjOrG6uQtxc3MTjYIgXCWH\nNQwFBQXExMQ0fB8dHU1BQUGTfVavXs0TTzwBcNUX17LyCgB6R4dwYPUnyNyURE5+/qrO2RLT+drN\nkwLw8ha1mztbcXEx9913n7PDEIRuyWENQ2su8k899RSvv/56w7Ltq72VtGNvOtGREdTl7ca35CO8\nIkfg5uOYMppp22qIiPYktrdY3dzZzs84+uqrr8Q0VEFwAIf1uaOiosjLy2v4Pi8vj+jo6Eb77N+/\nn9mzZwP2RUgbNmzA3d2dWbNmNTnf4sWLG76eOHEiEydObPR4aXkF36xez4QxIyg9+DUV+lBSb3/X\nIbORzmXqKSk0Mmu2qN3cmcSMI0G4vK1bt7J169arP5HkIGazWerdu7eUk5MjGY1GKTU1VTp+/HiL\n+8+dO1f67rvvmn2sNWH+smO31P/aG6T9GUek4/9Olta++2a7Y7+cOo1F+vLDQqmkyOiQ8wvNe/75\n5yVAkslk0ooVK5wdjiC4hPZe4h32dsvNzY0lS5YwdepUrFYr8+bNIzk5maVLlwLw2GOPdejzLX7j\nbfxVvgxNGUjGFh+iho7r0PODfXXzjvOrm8M9Ovz8QssCAgJEL0EQOkm3SLttNpvpN2Iyy997k5GD\nh3B0yUj6zV2DKiS6xWPa40SGlqxTOqbfHiIWsgltFhgYKCrECQ6hVquprKxssr1Hp92e88TT6A1G\nEuJ7cepoPgo3BUqvjl1sVl1p5lCahhl3ikZBaJ+qqqoOWasjCJfq6Ony3SJXkrZOx2fvv0lkeBhH\n03NQyms79PzW+trNQ0f74RfQLdrSLmvhwoX4+PiI2UaC4ETd6iqnqzOSf2QfQxNBoey4egiH9tbi\n7aOg3wCxutlRRO1lQeg6XL7HUFBUwuZfdwLwy7rDDEy24e7fC7mbskPOX1JgJPOEjjHXBYjVzQ7S\nXO1lMcAsCM7j8v99+w8dQeXrw4B+Sbz96Trm3+eJvK53h5zbZLKxfXMV104Wq5sdZcaMGaxfv17M\nOBKELsTlewwAE8eM4uD2swwa1gsfX0+UIUkdct70nTVExngSEy9WNzvK0qVLRS+hmyorKyM5ORmj\n0ejsULqskpIS+vfvj8lkcnYojXSLhsFqsbFt0zFuvL3j0mvX1lg4l2ngmjGidrMjRUdHi9rLXUBc\nXBze3t74+fmhVqsZM2YMS5cubZhFNW3aNFQqFSqVCg8PDzw9PRu+nz9/frPnfP3113nwwQfx9Gxc\n0XDu3Lm4u7tTXFzcZPuLL77YaNvZs2eRy+XYbLaGbV988QXDhg1DpVIRGRnJ9OnT2blzZ0f8GBoY\njUYeeugh/P39iYiI4K233rrs/h988AEJCQn4+/szfPjwRvHMnTu30c/Lz8+v4ecaFhbGpEmT+OCD\nDzo0/qvWQQvsHOpyYa5e/5M0dso90uf/7xdJkiSpbM8SqXTXW1f9nNt/qpQO7Km56vMIF+Tk5Dg7\nBKfqyv9ucXFx0pYtWyRJkqTa2lppzZo1Unx8vPTggw822Xfu3LnSiy++eNnzGQwGKTg4WCooKGi0\nXavVSr6+vlJKSor0j3/844rnzcnJkWQymWS1WiVJkqQ333xTCg0Nlb7//ntJp9NJFotFWrt2rfTM\nM8+0+TVfznPPPSeNHz9eqq6ulk6cOCGFh4dLGzdubHbfgwcPSr6+vtKBAwckSZKk999/XwoJCZFs\nNluLr+tiO3fulAYOHHhV8bb0t9XevzmX7zEYDCYqSjVMvX0oAGZNEVzlXPHaagt5OQb6D3ZcHYee\n5Hzt5fj4eDEN1QWoVCpuuukmVq5cyfLlyzl27FiTfaQr/I/t3buXgIAAIiMjG23/7rvviI+P55ln\nnmH58uVtOm9NTQ0vvfQS7733HrfccgteXl4oFApmzJhx2UJg7fHpp5/y4osv4u/vT1JSEo8++ijL\nli1rdt/jx4/Tv39/hgwZAsCcOXMoLy+ntLS0Va9rxIgRZGdnN8ot52wu3zAcTT+HX4A3wWF+AOgL\n96NQBlzVOTP2aUhO8cHT0+V/PE4nai+7ruHDhxMdHc327dvbfOyRI0eanXa8fPly7r77bmbNmkVm\nZiYHDhxo9Tl3796NwWDg1ltvbfUxr7/+Omq1utmPwMDAZo+pqqqiqKiI1NTUhm0pKSnNNpAA48aN\nIycnh7S0NKxWKx9//DFDhgwhLCysYZ/33nuPoKAghg0b1qRSpZubGwkJCRw6dKjVr8vRXPq/tE5j\n4NjBXMIi7eMAks2KqSYXr4gh7T5nbbWF/LMGbrs/7Mo7Cy0SmVDb7sk7/9Mh51nyzeMdch6AyMjI\ndqXxqK6uRqVSNdqWm5vL1q1bef/991GpVEydOpVPP/2UoUOHtuqcFRUVBAcHI5e3/g3bc889x3PP\nPdem2LVaLQD+/hfGF/38/NBoNM3uHxMTwyuvvMKYMWMAe3qK9evXNzz+u9/9jn/961/4+/uzadMm\n7r77bsLDw7n22msb9lGpVNTU1LQpTkdy6f/Un9ceJr5fGOdq7CudrcYasFnwCEpo9zkz9tWSnCp6\nC1dr/vz5Db0EMbjcOh15Qe8oBQUFLb6zvpzAwMAmF9LPPvuMgQMH0q9fP8Ben/v3v/89b775JgqF\nAjc3N8xmc6NjzGYzcrkcuVxOUFAQ5eXl2Gy2NjUObeXra7+FXFtbS3BwMGC/jXVpQ3femjVrePPN\nNzlx4gQJCQls2rSJmTNncvDgQSIiIhpuMYF9EP++++5j1apVjRoGjUZDQMDV3enoSC579TMazGz/\n8Rg6RSU6vR6A6sNfAqDwaN/YgL23YKR/qhhbuFqrVq3CZrOJRsGF7du3j4KCAsaOHdvmY1NSUjh9\n+nSjbZ9++ilnzpwhIiKCiIgInnrqKcrLy1m3bh0AsbGxnD17ttExOTk5DZUgR48ejaenJ99//32r\n43j11VcbZgNd+uHn59fsMWq1moiIiEa3djIyMhg4cGCz+2/atIkZM2aQkGB/Qzp16lQiIiLYvXt3\nq2K0WCxkZmY2unXlbC7bMBTlVRIUosIqWRk93N4VNZSdIGjkk+0+Z0ZaLf1TffAQvQWhBzo/QFpb\nW8vatWu55557mDNnDgMGDGh2v8sZPnw41dXVFBYWAvbxgezsbPbt20dGRgYZGRkcPXqUe++9l08/\n/RSA2267jXXr1vHTTz9htVopLCzklVde4Z577gHst3b+8pe/8Nvf/pbVq1ej0+kwm81s2LCBZ599\nttk4nn/+eTQaTbMftbUt51S7//77eeWVV6iurubEiRN8+OGHzJ07t9l9U1NTWbduHTk5OUiSxE8/\n/cTp06cbGpJvv/0WrVaLzWbjxx9/ZMWKFY2KkaWlpREXF9eoFLLTtW9yVOdqLsxdW05Iy9/ZIj3+\nfy9IH36+UpIkSTrx736SJntru56jutIkffFBoWQ0Wq8q1p6mqKhImj59urPDcAld+d8tLi5O8vLy\nklQqleTv7y9de+210nvvvdcw5fJirZmuKkmStGjRIumNN96QJEmSHn/8cemOO+5osk9aWpqkVCql\nqqoqSZIk6YcffpCuueYayd/fX+rVq5f0zDPPSAaDodExK1askIYNGyb5+PhI4eHh0syZM6Xdu3e3\n52W3yGg0Sg899JDk5+cnhYWFSW+91XgKvK+vr7Rjxw5JkiTJarVKixYtkqKjoyWVSiX1799f+vzz\nzxv2HTdunOTv7y/5+flJgwcPllauXNnoXPPnz5fefffdq4q3pb+t9v7NuWw9hlXLd1FnrmXVtu+Z\nOfU65t13F6f+32D6ProLuXvbk91t+7ESf7U7qcObv48oNLVw4ULeeecd5HI5dXV1KJUdk5+qu2pv\nbnxXVV5ezrhx4zh06FCTRW6CXWlpKRMnTuTQoUN4eLS/+FdLf1s9rh5DcX4Vn2z5ELkCkvu2f7AZ\noKbKTGGukVETu87gT1cmZhwJrREcHMyJEyecHUaXFhoayvHjx50dRhMuezO9KK8KhZucd19/iWTP\n4xJ4/ecAACAASURBVJSnvYdkM1/5wGZk7NPQP9UXDw+X/XF0mpdffllkQhWEbs4lr4R6nYk6rQE3\nNwVIVkq2voJkMRIyeiEyt7YlvKuutPcWklJ9HBRt99K3b1+Sk5MxmUxixpEgdFMu+VavOL+KsCg1\nsipwL90GSIRc+/t2nevwPg39B4veQmvde++9okEQhG7OJa+GRXmVRMSoAVDUniJg0D3tOk91pZnC\nPCPJKaK3IAiCcJ6LNgxVBAR7se/gYTxKfkEZNqhd58mo7y24i95CEwsXLsTT0xODweDsUARB6GQu\neSvp2PFMYhNgVF9v5J4KfOPGtfkc1ZVmivKMXDtJzES62KUzjsTAsiD0PC75Vnnp6v9iOfU2r82U\n8AxORN6OFBiH0jQMGCJ6CxcTtZcFQQAXaxgkSeI/H39BnUHLmCQ/Btz8N3rd8XmbF7RVVZgpzjeS\nNEiMLZx333338c4775CYmChmHAlXbcyYMWRkZDg7jC5t5MiRXXINA7hYw2CxWPj9i39leOIIZLpC\nPNtZ2zlD9BaaWLp0Kd98843oJfRQcXFxbNmypcn2rVu3IpfLG5LO9evX74plKH/44Qf8/f2bJIVb\ntmwZcrmcr7/+usn2ceOa3g6+NKa0tDSmT5+OWq0mKCiIkSNHtlg852q89dZbRERE4O/vz7x58y5b\nj3nHjh0MHz4cf39/+vTpw3//+9+Gx5YtW4ZCoWiUuG/btm0Nj//hD3/gz3/+c4fH3xFc6sq48n/r\nkCSJ391kbxDcVZFXOKKpqgozxYVGksRMpEZ8fX254447nB2G4CQymQyZTNbsY1FRUQ1J595++23m\nz5/fYtEagP/85z/MmTOnyfbly5czaNCghqR5bYlp9+7dXHfddUyaNImsrCwqKip4//332bhxY6vO\n1VqbNm3ijTfe4Oeff+bcuXNkZ2fz0ksvNbuv1Wrl1ltv5dFHH6WmpoaVK1fy9NNPc/jw4YZ9xowZ\n0yhx3/jx4xseu+mmm/jll18oKSnp0NfQEVyqYdi2K42Jw8YR7puHKmEqcre25+bJSNMwcIgv7u4u\n9dI71OX+qQXhcqZNm0ZQUFCLqS5MJhO//PILEyZMaLT93Llz7Ny5k08++YSffvqpzRfDRYsWMXfu\nXBYtWtRQH2Lo0KF89dVX7XshLVi+fDkPP/wwycnJBAQE8Oc//7nFXklJSQkVFRUNjeCwYcNITk5u\n9LO5XJ4ipVLJNddcw6ZNmzr0NXQEl7o6yuVyevkHoKpdh3fM6DYff763kNhDxxbO114eOHBgQ5Uq\nQWgtm83GmjVrqKmpaVR85mJnzpxBLpc3qfX86aefMmHCBIYOHcqwYcNYsWJFq59Xp9OxZ8+eNvVo\nd+zY0WJJT7Vaza5du5o97vjx401KepaUlDRbxS4yMpKUlBQ+/vhjrFYru3bt4ty5cw31K2QyGQcP\nHiQkJITExEReeeUVrFZro3MkJyd3ybEYl7qZXFFZRYDeBG5eBAya3ebjD6XV9tjewvlMqDKZjBUr\nVjRUqRK6DmVk+8bMLmUoPNkh5zmvsLAQtVqNXq/HbDbz1Vdf0adPn2b3ba6kJ9gbhkWLFgH2ym3L\nli3j6aefbtXzV1VVYbPZiIiIaHXMY8eObVdJUq1W26SkJ9grrKnV6ib7f/DBB8yYMYOFCxcC9tto\nUVFRAIwfP55jx47Rq1cvjh49yt13342bm1ujUqMqlYqioqI2x+loLtUw/LBpC3+bNQA3L3WL90Nb\nUllupqTQxNjrm/5yuzORCdV1dPQFvaNERkaSl5eHyWTiueee49VXX+X2229vtrymWq1uUtJz586d\nnD17ltv+f3t3HhXFlfYP/NvNouyyRJBFWVRkkSVicEVIxp+IRkfBhWjiMkOiMzFmEpPojMaob4xL\nlvENzoSYqIhAFpIcY1QmvioRBQdHDaggoiJbAGUTkB2e3x8MpU13Q7P0hs/nnD7HqrpVdbmnvU/X\ncu8zfz4AICwsDG+++SbS09Ph7e0tM6Un0J7WU09PD+bm5hCLxSguLhbSgiqLsbGxRAKfjjzMsoJd\nUVERZs+ejbi4OEyfPh03b97E7NmzMWzYMISEhMDJyUko6+npiXfffRe7d++WCAzV1dUyA466ac1P\n59u5eQAAVzsdGDs/2+P905/Qq4UtW7agtraWxyWwPtPX18fOnTvx4MEDxMTEyCwzcuRIEJHEr+Do\n6GgQEcaOHYthw4Zh/PjxwnqgPaVnfn6+xHHq6upw7949jBgxAoaGhpg4cSISEhIUrmtycrLclJ4m\nJiY4f/68zP08PDykUnpaW1vL7LxTUlJgb2+P6dOnAwBGjx6NWbNm4cSJE3Lr1fmZQ1ZWlkal9BT0\nKr2PigGg//kokuw9JtG//x5CZWmf9Wj/8ntN9NUXv1FzE2dnY+qjyf/dHB0d6cSJE1RfXy98Wlpa\n6MyZM2Rvby9RNjIyktzd3eUea86cORQXF0dERPX19WRmZkb79++n0tJS4bN3716ytram1tZWamho\nICcnJ/rggw+ooaGBamtrac2aNTRp0iThmCkpKWRsbEy7d++msrIyIiL69ddfafHixf3aDomJiWRj\nY0OZmZlUUVFB06ZNow0bNsgsm5mZSYaGhnT69Glqa2ujW7du0ciRI2nfvn1ERHT8+HEqKSkhIqKs\nrCzy9PSkrVu3CvvX19eThYUFFRcX97ne8r5bvf3Oae439TEdgWHRC+voymdhVHs3uUf7nz5WRtcu\n1yipdowpRtMDg0gkkvhs2rSJkpKSyMHBQaJsXV0dWVlZ0ZEjR2Qe69ixYzRz5kwiIoqPjydbW1tq\naWmROoalpSUdO3aMiNo72RkzZpCVlRVZW1vTggULqLCwUGKftLQ0mjlzJpmZmZGFhQX5+/tTTExM\nfzWB4OOPPyZra2syNTWllStXUlNTk7Bt5syZ9MEHHwjL0dHR5ObmRiYmJmRvb0/r168Xtq1bt46s\nra3JyMiInJ2dafPmzRLt8M0331BoaGi/1Lm/A4PWpPb8n48ikXomC7vDyjE88M8wGjFFoX3L7zfh\n/46WI/RFa+gO4NtIJSUlWLhwocQAGqZZnqTUnlOmTMHevXs18zaJhpgwYQL2798Pd3f3Ph/riU7t\nWVdTD1HVZYh6MH4hPa0Gnk+bDOig8PgbR1VVVRgyhCcGZOp17tw5dVdB4124cEHdVZBL6b1lYmIi\nxowZg1GjRmHnzp1S22NjY+Ht7Q0vLy9MnjxZYtRgZ9TaAgAwtPNT6Nzl95twv7QJrp4Dc9xCx7iE\nx+c44qDAGOsrpQaG1tZWvPrqq0hMTERmZibi4+OlRkw6Ozvj7NmzyMjIwKZNm/Dyyy/LORrB1apn\n7yWnp9Vg7NMm0NXt2aut2mD37t08EypjTCmU2pOkpaVh5MiRcHR0BAAsXrwYR44cgZubm1Bm4sRH\nI5j9/f1RWFgo81htbQQdMcHY+TmFzl1+vwllpU0ImGHR+z9Ag/n7+8PT0xNXrlzhgMAY61dKvWIo\nKiqCg4ODsGxvb4+ioiK55b/88kuEhITI3Nba3AaxjuK//H/9dw08xw3MqwWgfVTl1atXOSgwxvqd\nUnuVnoxOPnPmDPbv3y934Mnpk8dhVHMTH8UWYq5JEgIDA+Ueq+J+M8rvNWFa8MC8WmCMMVmSkpKQ\nlJTU5+MoNTDY2dmhoKBAWC4oKIC9vb1UuYyMDERERCAxMVHu8PApAf8Po2oaMGveLAydEtjlee/e\nqofLGMMBcbWwdu1a7N27FyUlJbCyslJ3dRhjGiwwMFDiR/OWLVt6dRylBgY/Pz/k5OTg7t27sLW1\nxddff434+HiJMvn5+Zg/fz4OHz6MkSNHyj1WS2srABH0hzjJLdOhKL8B46eYdVtOk3We44jfNmKM\nqYpSnzHo6uoiMjISM2bMgLu7OxYtWgQ3NzdERUUhKioKALB161ZUVlZi9erV8PX1xTPPPCPzWK0t\nbRCJu78CqK9rRXVVC4ba6Pfr36JKnHuZPUnCw8Nx5MgRdVdDo4WFhfV7UqIu9XIEtkoBoLXr3qeD\nbwVS5dVvuix7K+shnfqpTEU163+vvvoqASBXV1dqbm5Wd3VYP9Lk/27Jyck0ceJEYbqJyZMn08WL\nFyk1NZWMjIyotrZWah8fHx/au3cv5ebmkkgkIl9fX4nt9+/fJz09PXJ0dJR73vT0dJnzLp05c4ZE\nIhHt3LlTan3nuZuIiKZNm0ZffPGFsJydnU1hYWFkZWVFZmZm5OXlRR9//DG1tvbvfGmxsbE0fPhw\nMjIyot///vdUUVEhs1xeXh4ZGxtLfEQiEX388cdCmXv37lF4eDiZmZmRubk5LVmyRNiWlpZG48aN\nk1sPed+t3n7ntGY4cGurYlcMRXkNsB/R88xumuKTTz7B0aNH+SqBqUx1dTVmz56NtWvXorKyEkVF\nRdi8eTMGDx6MCRMmwN7eXmpm02vXriErKwvh4eHCuvr6eonsgHFxcXB2du7yJZSoqCgsXbpUan10\ndDQ8PT17lQb09u3b8Pf3F/IgVFVV4dtvv8WlS5ekpgTvi+vXr2PVqlWIjY1FaWkpDA0N8ac//Ulm\n2eHDh0uk+Lx69SrEYjFCQ0OFMvPnzxemOL9//76QvwIAxo8fj+rqaly6dKnf6t8V7QkMLW0Qd/OW\nU1sboSi/EbYjBqmoVv1PV1cXs2fPVnc12BPk5s2bEIlEWLRoEUQiEQYPHozp06fD09MTALBs2TKp\nDvrQoUOYNWuWxMsiL774ojCVNgDExMTgpZde6nKunsTERKk0oA8fPsR3332Hzz77DPn5+T3uDDdv\n3owpU6bgww8/hLW1NYD2KbEPHz4skYSnr2JjYzFnzhxMmTIFRkZG2LZtG77//ns8fPiw232jo6Mx\nbdo0DB8+HADw888/o7CwELt27YKJiQl0dHSk5pkKDAzEsWPH+q3+XdGawNDS2ooRplUA5H/Jyu81\nw8BQDGMT7filnZqaqu4qMAZXV1fo6Ohg+fLlSExMlMp8tnTpUpw9e1YYfNrW1ob4+HgsW7ZMotyS\nJUvw1VdfgYiQmZmJ2tpa+Pv7yz3vw4cPkZubC1dXV4n133//PaytrTFp0iQ8//zzEsFGEadOnepR\nGtD8/Pwu04DKyyvdOQ2os7MzBg0ahJs3b3Z5PiLCoUOHJNrvwoULcHV1xbJly2BlZYVnnnlGakJM\nVaYB1Y4eFO1XDJYGDzHY2ktumaL8BthpwW2kx984Ki4uho2NjbqrxDTAjT2u3RdSwJi12T0qb2Ji\ngnPnzmHnzp2IiIhASUkJQkJCsG/fPgwdOhQODg4IDAxETEwMNmzYgFOnTqGxsRGzZs2SOI69vT1c\nXV1x8uRJnD59Gi+99FKX562qqhLO/7jo6GgsWLAAQHsa0JdffhmffPIJdHR0FPp7ysvLe5QGdPjw\n4f2SBhRoTwXa3e2qc+fO4d69exLBq7CwED///DO+/PJLHDx4EAkJCZg7dy5u3boFS0tLAO3Z5Tra\nTNm0KjA8bNGHrqH8QWtFeY3w9ZdOwadJ1qxZg8jISCH3MgcF1qGnHXq/nnvMGBw4cAAAkJ2djaVL\nl+L1119HXFwcgPbbSdu3b8eGDRsQExOD8PBwqY5aJBLhpZdewoEDB5Camopz587hxg356Uo7XsGu\nqakROr+CggIkJSVh9+7dAIDg4GA0NDTgp59+wty5c7tNAwoAlpaW+O233/rYIt0zNjYWUn92ePDg\ngcw0oI+Ljo5GWFgYDA0NhXUGBgZwcnLCihUrAACLFi3C+++/j/Pnz2POnDkA2ttJVa+ta82tpPqG\nBnT1hKGxoQ1V5c2wttPM5wsdM6FGRkbCzc0NTU1NeOGFF9RdLcakdNzSuHbtmrBu3rx5KCwsxJkz\nZ/DDDz9I3UbqMH/+fBw/fhwuLi4yB7M+zsjICC4uLsjOfhQQY2Ji0NbWhpCQEAwbNgxOTk5oaGiQ\nSANaVlYmcR+fiJCXl4cRI0YAAH73u9/hu+++U/jvzc/P7zINaOexVx08PDwkbu3cvn0bTU1NXeal\nrq+vR0JCglT7ycpb8fgDdaA9DaiPj4/Cf1ef9OpdJhUDQHZuk+nibi9qri2VWeZO9kM6+aPmvqa6\nadMmEovFFBsbq+6qMDXR1P9uN27coI8++kjImJafn0+TJk2il19+WaLcihUraMSIEeTp6SmxvuN1\n1Y5XQS9dukR37twhIqKTJ092+brqa6+9Rtu3bxeWR48eTVu2bJFIA/rjjz/SoEGDqLy8nIiIJk2a\nRH/+85+ptraWGhoaaOfOneTs7EyNjY1ERHT79m2ysLCgt956S0itmZOTQ0uXLqWqqqq+NJWE69ev\nk6mpKSUnJ1NtbS2Fh4dTeHh4l/vExsaSk5OT1PqKigoyNzen6OhoamlpoW+//ZYsLS2Fv5movW0u\nXrwo87jyvlu9/c5p5je1EwDk4D6F0v8+Tm5gSD5ZQZnpnL6TaS5NDQxFRUW0cOFCsrOzIyMjI7Kz\ns6NVq1ZRTY3k/6ekpCQSiUS0a9cuifW5ubkkFotljhE4efKkzI6ww7Vr18jDw4OIiFJTU8nAwEDI\n6fw4Dw8P2rt3LxERFRQU0IIFC8jGxoasrKwoODiYsrKyJMpnZ2fTggULyNLSkszMzMjb25v27NnT\n7+MY4uLiJMYxVFZWCttWrVpFq1atkig/Y8YMevfdd2UeKzk5mcaOHUvGxsY0fvx4OnfunLBN1eMY\ntCa1p7vfTHyzrAyuK36CrtFQie1EhG/2l2Bm2FMwNdOaxybsCfMkpfbsiSVLlmDhwoWYO3euuqui\nscLCwvDHP/4RwcHBMrc/sak9p3pNhVgse9h8RVkzdPVEGhEUSkpKMGvWLJUNRGFM28XGxqq7Chqv\n8wBDZdOah88tza1yR1AW5TVqxGjnNWvWYNiwYbhy5YrchEOMMabp1P8TW0GDRTVAUwUglq5yUX4D\nxj6tvtdUHx+X4ObmhoyMDJ7OgjGmtbTmikEfDyEysIaugeQ4hqamNpTfa4aNnXpmU/30008lZkLN\nzMzkoMAY02pa04M1N7VAx0A6iU9xQSOGDtOHrp56Ytyzzz4Lb29v/Oc//+GAwBgbELSmJxuEOoha\n66XWF+U1wG64+p4veHh44Ndff1Xb+RljrL9pza2k4eaV0Ok0HQYRoSivEXZaPJsqY4xpGq0JDCMs\nHsDIYYLEugeVLYAIMDNX/oXPmjVroKOjg7t37yr9XIwxpk5aExjMDeuhby6Z77korwF2IwZ1mQik\nrx6f48jV1bXb+V8Y00aOjo44deqUSs/Z2NgIDw8PlJaWqvS82qSxsRFubm4oKytT6Xm1JjC0kB70\nhzhKrCvMa1Tq84WOcQn8xhEb6DpP2PY4R0dHWFtbo66uTlj3xRdfICgoSFgWi8Xw8vKSGGW7ceNG\nYbZQWT7//HNMmzZNSKbT4b333oNYLEZaWprU+hdffFHqOGKxGHfu3BGW//WvfyEgIACmpqYYOnQo\nAgMDcfToUbn16K133nkHVlZWsLKywvr16+WWi42NlZiUz8jICGKxGFeuXAEA7N69G2PHjoWpqSmc\nnZ3x4YcfCvsOGjQIK1euxI4dO/q9/l3RmsAg7lTT5uY23C9pgq2Dcp4vbNu2jWdCZey/2trasGfP\nni7LFBcXSyS16e5KPioqSqqjp/8msRk7dqxU1jhF7gwkJCRg4cKFWL58OYqKinDv3j1s3bq13wND\nVFQUjhw5goyMDGRkZODo0aOIioqSWXbJkiUSaT3/8Y9/wMXFBb6+vkKZmJgYVFVVITExEZGRkfj6\n66+FbeHh4YiOjpY53biyaE1gsDEsh3jQo0FsJYVNsBqqBz195fwJmzZtwi+//MJXCeyJk5WVBWdn\nZ6FzEolEWLduHT788EOp/AOPe/vtt7F582a0trZ2e478/HzcuXNHKsNbcnIyqqursWfPHnz11VcS\nnWF3c/4QEd544w28++67WLlypZAXISAgAJ9//nm3deqJ6OhorFu3Dra2trC1tcW6detw8OBBhfY9\nePCgRBKjt956Cz4+PhCLxRg9ejTmzp2L8+fPC9vt7e1hbm6u0oyPWhMYAGCQubPw7/bnC8p9TTUg\nIECpx2dM01y+fBnBwcGIjIzEokWLhPV+fn4IDAyUuM3R2bx582Bqaip0kF115FevXoWzszPEnW4F\nREdHY968eQgMDISBgUGPfulnZ2ejsLCwR2k94+Li5Kb0tLCwkDu1Tee0nl5eXrh+/Xq358vLy0Ny\ncrLc7HZEhLNnzwr5tjuoMq0noEXjGOpbDSSWi/IbEBRi2S/HPnnyJKZPn94vx2Kstw5+WtQvx1m+\nxq5X+/3yyy/Yv38/YmNjpX4UiUQibN26FZMnT8batWtl7i8Wi7Ft2zasXr1aobSenTOd1dXVISEh\nAd9++y0AIDQ0FIcOHcL8+fMVqn95eTkA9Cit5wsvvNCr28Sd03qampqitra22/0OHTqEgIAAIalQ\nZ++99x4ASD2bMTExUVlaT0CLAkNJ86OGrK5qQUsLwdyyb9V/fI6jGzduSCUlZ0yVetuh9wciQlRU\nFAIDA+VeKXt4eGD27NnYsWMH3NzcZJaZOXMm7O3tERUV1eUzAXNzc6ncyD/88AP09PTw3HPPAWjP\n9/zss8+ivLwclpaW0NPTk7rP3rGsp6cnpActLi6W2/H2F2NjY1RXVwvLDx48gLGxcbf7HTp0CBs3\nbpS5LTIyEocPH0ZycrKQprRDTU0NzM2lZ35QFq25ldQgepTrtPS3RtjY9e011c5vHHFQYE8ykUiE\nqKgo5OXl4Y033pBbbsuWLdi3bx+KiuRf3bz//vvYvn27xFtMnXl5eSE3NxdtbW3CuujoaNTU1MDe\n3h7Dhg1DaGgompubhWm5hw8fLjWOKDc3F7q6urCzs4OrqyscHBx6NEV15zeGHv+YmprKvZXUecaD\n9PR0qds/nZ0/fx7FxcUyb3Xt378fu3btwqlTp2Brayu1PSsrS2b6T6XpVXofFQNAh3a/LSynnKmk\na1d6l62tuLiYTE1NCQC5ublRc3Nzf1WTsS5p8n83R0dHOnXqFFVVVdG4ceNo/fr1Uts6REREkIWF\nBQUGBgrrRCIR3b59W1iePn06WVpa0ooVK+Se08vLi1JSUoiIqLCwkHR0dOjkyZNCSs+SkhJav369\nkLmstLSUzMzMKCYmhpqamqi8vJxCQ0Ml0mkmJCSQmZkZHThwgB48eECtra2UnJwslaa0rz777DNy\nc3OjoqIiKiwsJHd3d4qKiupyn4iICFq2bJnU+sOHD5ONjY1UFroOhYWFZGlpSU1NTXKPLe+71dvv\nnOZ+Ux8DgOL+vkFY/vGrUir9raFXx/rnP/9JOjo6nHuZqZw2BAai9vzD3t7eQgrKzoGhoKCABg8e\nTEFBQcI6sVgsERj+/e9/k0gk6jIw7N27l1avXk1ERB988AH5+flJlfntt99IX1+frl+/TkREKSkp\nNGXKFDI3NydbW1uKiIiQyuOcmJhIU6dOJWNjY3rqqacoKCiIjh8/3tMm6dbbb79NFhYWZGFhQe+8\n847ENg8PD4qLixOW6+vraciQIXT69Gmp4zg5OZG+vj4ZGxsLn452ISLatWsXvfnmm13Wpb8Dg9ak\n9vxm70Ys+NM2tLQQ4vcVI/yPNmqbUZWx3uDUnpKamprg6+uL06dPSw1yY+0aGxvh4+OD5ORkWFlZ\nyS33xKb21PtvEKgsa4bZEF0OCoxpOX19fYVe8XySDRo0CFlZWSo/r9b0rtZD26NlWWkTrKz1uin9\n6I2jlpYWZVeNMcYGFK0JDE6O7ZPXtQeGrrO1dbxxlJOTg+zsbFVUjzHGBgytuZVkaNA+yvl+aTM8\nfGXnd+bcy4wx1ndac8UAkRhNjW2oe9iKITIGth08eJBnQmWMsX6gNT2nsdM0lBQ3wcJKD2Kx9MC2\n4OBgjB8/HikpKRwQGGOsD7SmBxXrDkZZaY3cB882NjZS87czpknMzc2VmlSKPbn6e7oMpd5KSkxM\nxJgxYzBq1Cjs3LlTZpnXXnsNo0aNgre3t5C4Qh5FHjwzpqkqKipA7YNK+cOffv1UVFT063dVaYGh\ntbUVr776KhITE5GZmYn4+Hip93GPHz+OW7duIScnB59//jlWr17d5THL7jVjT2R7dqeLFy8qq+oa\nLSkpSd1V0BjcFo9wWzzCbdF3SgsMaWlpGDlyJBwdHaGnp4fFixfjyJEjEmV+/PFHLFu2DADg7++P\nqqoquflf794pxsHYrfjfT3djzJgxEtmPniT8pX+E2+IRbotHuC36TmmBoaioCA4ODsKyvb291IyM\nssrIm83Q29sL12+c5TeOGGNMyZTWuyr6kI1Ich4PefvNn70OH326FhZWys3axhhjTzxSktTUVJox\nY4awvH37dtqxY4dEmVdeeYXi4+OFZVdXVyopKZE6louLCwHgD3/4wx/+9ODj4uLSq/5baVcMfn5+\nyMnJwd27d2Fra4uvv/4a8fHxEmXmzJmDyMhILF68GBcuXMCQIUNkzrJ469YtZVWTMcZYJ0oLDLq6\nuoiMjMSMGTPQ2tqKP/zhD3Bzc0NUVBQA4JVXXkFISAiOHz+OkSNHwsjICAcOHFBWdRhjjClIK/Ix\nMMYYUx2NmiupvwfEabPu2iI2Nhbe3t7w8vLC5MmTkZGRoYZaqoYi3wsAuHjxInR1dfH999+rsHaq\no0g7JCUlwdfXF56enggMDFRtBVWou7YoKytDcHAwfHx84OnpiYMHD6q+kiqycuVKWFtbY+zYsXLL\n9Ljf7NWTCSVoaWkhFxcXys3NpaamJvL29qbMzEyJMseOHaOZM2cSEdGFCxfI399fHVVVOkXaIiUl\nRUhpeOLEiSe6LTrKBQUF0axZsyghIUENNVUuRdqhsrKS3N3dqaCggIiI7t+/r46qKp0ibbF582Yh\nb/X9+/fJwsJiwOZ3P3v2LF2+fJk8PT1lbu9Nv6kxVwz9PSBOmynSFhMnToSZmRmA9raQN/5D6lB3\n/QAABDJJREFU2ynSFgDw6aefIiwsDE899ZQaaql8irRDXFwcQkNDYW/fnrukq1SQ2kyRthg2bBiq\nq6sBANXV1bC0tBywY5+mTp3a5VxJvek3NSYw9PeAOG2mSFs87ssvv0RISIgqqqZyin4vjhw5Ikyp\nMhAnqlOkHXJyclBRUYGgoCD4+fkhJiZG1dVUCUXaIiIiAtevX4etrS28vb2xZ88eVVdTY/Sm39SY\nENrfA+K0WU/+pjNnzmD//v04f/68EmukPoq0xeuvv44dO3YIic87f0cGAkXaobm5GZcvX8apU6dQ\nV1eHiRMnYsKECRg1apQKaqg6irTF9u3b4ePjg6SkJNy+fRvTp09Heno6TExkJ/ka6Hrab2pMYLCz\ns0NBQYGwXFBQIFwSyytTWFgIOzs7ldVRVRRpCwDIyMhAREQEEhMT+33aXU2hSFtcunQJixcvBtD+\n0PHEiRPQ09PDnDlzVFpXZVKkHRwcHGBlZQUDAwMYGBggICAA6enpAy4wKNIWKSkp+Nvf/gYAcHFx\ngZOTE7Kzs+Hn56fSumqCXvWb/fYEpI+am5vJ2dmZcnNzqbGxsduHz6mpqQP2gasibZGXl0cuLi6U\nmpqqplqqhiJt8bjly5fTd999p8IaqoYi7ZCVlUXPPfcctbS00MOHD8nT05OuX7+uphorjyJt8Ze/\n/IXee+89IiIqKSkhOzs7Ki8vV0d1VSI3N1ehh8+K9psac8XAA+IeUaQttm7disrKSuG+up6e3oBM\nVKRIWzwJFGmHMWPGIDg4GF5eXhCLxYiIiIC7u7uaa97/FGmLv/71r1ixYgW8vb3R1taGXbt2wcLC\nQs01V47w8HD88ssvKCsrg4ODA7Zs2YLm5mYAve83eYAbY4wxCRrzVhJjjDHNwIGBMcaYBA4MjDHG\nJHBgYIwxJoEDA2OMMQkcGBhjjEngwMBYF3R0dODr6yt88vLykJSUBDMzM/j6+sLd3R1bt24FAKn1\nGzduVHPtGesdjRngxpgmMjQ0lJq/Pjc3FwEBATh69Cjq6urg4+OD559/HiKRSFjf0NAAX19fzJs3\nD+PGjVNT7RnrHb5iYKwPDA0NMW7cOKm85IMHD4aPjw/u3Lmjppox1nscGBjrQn19vXAbKTQ0VGp7\neXk5Lly4AE9PT4kZLCsqKpCWljYgp6RgAx/fSmKsCwYGBjJTISYnJ+Ppp5+GWCzGhg0b4ObmhtLS\nUiQnJ8PHxwc5OTlYtWoVPDw81FBrxvqGAwNjvTB16lQcPXpU7vq7d+8iKCgIr7/+ukSSFMa0Ad9K\nYkwJHB0dsXbtWmzbtk3dVWGsx/iKgbEuyMp0JRKJFFq/atUqjB49GoWFhTITLTGmqXjabcYYYxL4\nVhJjjDEJHBgYY4xJ4MDAGGNMAgcGxhhjEjgwMMYYk8CBgTHGmAQODIwxxiRwYGCMMSbh/wOdHYqP\nwSpKVAAAAABJRU5ErkJggg==\n", "text/plain": [ "A calibration chart
\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def getMAE(pred, truth):\n", " return np.abs(truth - pred).mean()\n", "\n", "def getLL(pred, truth):\n", " ll_sum = 0\n", " for i in range(len(pred)):\n", " if (pred[i] == 0):\n", " p = 0.0001\n", " elif (pred[i] == 1):\n", " p = 0.9999\n", " else:\n", " p = pred[i]\n", " ll_sum += truth[i]*np.log(p)+(1-truth[i])*np.log(1-p)\n", " return (ll_sum)/len(pred)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "Loop through features to get AUC and LL
" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn.metrics import roc_auc_score\n", "\n", "def evalFeat(x_train, y_train, x_test, y_test):\n", " lr_f = linear_model.LogisticRegression(C=1e30)\n", " lr_f.fit(x_train, y_train)\n", " p = lr_f.predict_proba(x_test)[:,1]\n", " ll = -1*getLL(p, y_test.values)\n", " auc = roc_auc_score(y_test, p)\n", " return [ll, auc]\n", "\n", "\n", "lls = []\n", "aucs = []\n", "feats = train.drop(lab,1).columns.values\n", "for f in feats:\n", " ll_f, auc_f = evalFeat(train[[f]], train[lab], test[[f]], test[lab])\n", " lls.append(ll_f)\n", " aucs.append(auc_f)\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A quick Lift chart
" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def liftTable(pred, truth, b):\n", " df = pd.DataFrame({'p':pred+np.random.rand(len(pred))*0.000001,'y':truth})\n", " df['b'] = b - pd.qcut(df['p'], b, labels=False)\n", " df['n'] = np.ones(df.shape[0])\n", " df_grp = df.groupby(['b']).sum()\n", " base = np.sum(df_grp['y'])/float(df.shape[0])\n", " df_grp['n_cum'] = np.cumsum(df_grp['n'])/float(df.shape[0])\n", " df_grp['y_cum'] = np.cumsum(df_grp['y'])\n", " df_grp['p_y_b'] = df_grp['y']/df_grp['n']\n", " df_grp['lift_b'] = df_grp['p_y_b']/base\n", " df_grp['cum_lift_b'] = (df_grp['y_cum']/(float(df.shape[0])*df_grp['n_cum']))/base\n", " return df_grp\n", " \n", "\n", "lifts_lr = liftTable(lr.predict_proba(test.drop(lab,1))[:,1], test[lab], 25)\n", "#lifts_svm = liftTable(mm.decision_function(test.drop(lab,1)), test[lab], 25)\n", "lifts_dt = liftTable(dt.predict_proba(test.drop(lab,1))[:,1]+np.random.rand(test.shape[0])*0.00001, test[lab], 25)\n", "lifts_knn = liftTable(knn.predict_proba(test.drop(lab,1))[:,1], test[lab], 25) \n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "