{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# About\n", "\n", "This notebook demonstrates several additional tools to optimize classification model provided by __Reproducible experiment platform (REP)__ package:\n", "\n", "* __grid search for the best classifier hyperparameters__\n", "\n", "* __different optimization algorithms__ \n", "\n", "* __different scoring models__ (optimization of arbirtary figure of merit)\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Populating the interactive namespace from numpy and matplotlib\n" ] } ], "source": [ "%pylab inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Loading data for Higgs Boson Challenge" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "['EventId',\n", " 'DER_mass_MMC',\n", " 'DER_mass_transverse_met_lep',\n", " 'DER_mass_vis',\n", " 'DER_pt_h',\n", " 'DER_deltaeta_jet_jet',\n", " 'DER_mass_jet_jet',\n", " 'DER_prodeta_jet_jet',\n", " 'DER_deltar_tau_lep',\n", " 'DER_pt_tot',\n", " 'DER_sum_pt',\n", " 'DER_pt_ratio_lep_tau',\n", " 'DER_met_phi_centrality',\n", " 'DER_lep_eta_centrality',\n", " 'PRI_tau_pt',\n", " 'PRI_tau_eta',\n", " 'PRI_tau_phi',\n", " 'PRI_lep_pt',\n", " 'PRI_lep_eta',\n", " 'PRI_lep_phi',\n", " 'PRI_met',\n", " 'PRI_met_phi',\n", " 'PRI_met_sumet',\n", " 'PRI_jet_num',\n", " 'PRI_jet_leading_pt',\n", " 'PRI_jet_leading_eta',\n", " 'PRI_jet_leading_phi',\n", " 'PRI_jet_subleading_pt',\n", " 'PRI_jet_subleading_eta',\n", " 'PRI_jet_subleading_phi',\n", " 'PRI_jet_all_pt',\n", " 'Weight',\n", " 'Label']" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy, pandas\n", "from rep.utils import train_test_split\n", "from sklearn.metrics import roc_auc_score\n", "\n", "data = pandas.read_csv('toy_datasets/Higgs.csv', sep='\\t')\n", "labels = data['Label'].values\n", "labels = labels == 's'\n", "sample_weight = data['Weight'].values\n", "\n", "train_data, test_data, train_labels, test_labels, train_weight, test_weight = train_test_split(data, labels, sample_weight)\n", "\n", "list(data.columns)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Variables used in training" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [ "features = list(set(data.columns) - set(['Weight', 'Label', 'EventId']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Metric definition\n", "\n", "In Higgs challenge the aim is to maximize `AMS` metrics.
\n", "To measure the quality one should choose not only classifier, but also an optimal threshold, \n", "where the maximal value of `AMS` is achieved.\n", "\n", "Such metrics (which require a threshold) are called threshold-based.\n", "\n", "__rep.utils__ contain class __OptimalMetric__, which computes the maximal value for threshold-based metric (and may be used as metric).\n", "\n", "Use this class to generate metric and use it in grid search." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare quality metric\n", "\n", "first we define AMS metric, and `utils.OptimalMetric` generates " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from rep.report import metrics" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def AMS(s, b): \n", " br = 10.0\n", " radicand = 2 *( (s+b+br) * numpy.log (1.0 + s/(b+br)) - s)\n", " return numpy.sqrt(radicand)\n", "\n", "optimal_AMS = metrics.OptimalMetric(AMS, expected_s=692., expected_b=410999.)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Compute threshold vs metric quality\n", "random predictions for signal and background were used here" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimal cut=0.0004, quality=1.0802\n" ] }, { "data": { "text/html": [], "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": [ "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEZCAYAAAC99aPhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n", "AAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8HEW99/HPQAiELewJkSWGQCBsAaKyRCxQRFYXdhUE\n", "UVncAvrSB7xgRCCgF5RcFuEiiYAXZbmyCGoeIFFB2XxkcUFAVllCEENCJATI7/mjepg+fWZfuqrn\n", "fN+v17zOdE/PzDeVpOpUV3cViIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiHTfrsCNwD+AZcCnGhzv\n", "gBuA54DFwAPAUT3MJyIiLVou5+9bBXgQ+DLwGmANjt8J33gcAGwJXARcAhzWw4wiIlIQi4Aj2njf\n", "T4Fru5xFRETalHePpBtGAi+HDiEiIt6w0AFatC+wO7Bz6CAiIhJeq6e2dgFeAY7pTRwREWlHUXok\n", "U4CbgVOAi+sc9ywwJpdEIiL94+/A+NAh2tFsj2RXYCEwtYljG10FNpRMCx0gItNCB4jItNABIjIt\n", "dICIdFR35t0jWQXYNHm+HLAxMAn4J/AMMB14F/CB5BiH74mcD1wFjE72vwXMzyVxcY0NHSAiY0MH\n", "iMjY0AEiMjZ0AGmPw9+IuAzfGJSfX5a8PhN4PHX8zMxx5Uf6mDT1SCpmhQ4QkVmhA0RkVugAEZkV\n", "OkBEVHemqDAqXOgAEXGhA0TEhQ4QERc6QERUd6aoMEREWtdR3VnEGxKlOS50gIi40AEi4kIHyNHL\n", "+ApSj8qjJzdzF+XyXxGRVq0JlOq87oC5uSSJh87aNEGFJCJlqg8Gq1UmOrUlIiLhqCHpXy50gIi4\n", "0AEi4kIHiIgLHaBfqCERERFJ0TlRESkrQn0wF38l1fDUvln4G6/3zxz7PQauLDscOAc/K8gi4Ink\n", "mHo0RiIi0kfGAu8GXmRgo2HAIwyci3AYcDDwGJVK/yRge/y0UqvhT9X9oZeBa1FD0r9c6AARcaED\n", "RMSFDhARF/j7jwBuBa6g0ssouwk/6/kayfaH8MuOz0sdMxm4Hngh2X4KuLJXYetRQyIiQ46BGcxJ\n", "fnb06CDGEfilw68G9gTWTb22BLgBODR17OWZ998FnAgcB2xN/XtmpAVFOCcqIvmoWR90owHpsCGZ\n", "AryGPyUFcD+VpTJmAt/GL+b3O/zy4i8AKwG/pXLKazngeOAOfMPzLI2X5tAYiYhIN5Sg1K1HmxE+\n", "BczGD5IDXEPl9FYJX7Hfie+l/Af+VNeSzGcsAy7EN0ojgTPwM6lv3mYmSahHUuFCB4iICx0gIi50\n", "gBw1qg9cHiGqGIFfNnwR8HzyeBm/ZMY2+B7Jacmx30z2vzfZTvdIqpkPfLTO6z3pkWiuLRGRfH0E\n", "eBPYFlia7Cvhx0qOSG0DzAB+g29AsqYCfwTuAd4APgGsmuyTDqhHIiJlsdYHvwC+W2X/QfjeyeVU\n", "eiRZ6R7JZ4H7gAXAv/CD73s3+O6e9Ej6jRkcHjqEiERBleNgakiaUL6S4qumS+Fc6AARcaEDRMSF\n", "DpCjWMdIQtJVWy34LrDM/DXWIiIiTTODX2au8149dCgRCUKnawbTqa0mmMEIg30MXkwakgds4IRo\n", "IjI0qHIcTA1JEyz15LRMz2SGwcohw+XMhQ4QERc6QERc6AA50hjJYBojaUUJTgU+ntr1RWCxwecN\n", "DtRgvIiIVDOoVTUYaXB6jXlyZhvsZ3BStbEUg7UM1kueL2+wgcE7kufv1/iLSNRextcJelQeL9co\n", "K53aSqlZGAbvMzi/RoNSftxv8KnkNNhVBksbHL/Y4HKDLfL8Q4qIdJkakpSGhWGwnPnB+LuTxmBB\n", "g8Yi/ZhnsLDO64c2+v4cudABIuJCB4iICx0gIi50gIh01JDkPUayK3Aj8A8GLhlZz9bAr4F/J+87\n", "pZMAJVhWgpvxk6CNL/mFY1YEvp8c8jx+Vs6bgGOAnfB3y29aglHA2sB2wGj85GppV5nGXkREemov\n", "4HTgAGAxjefOXx0/D/9PgInJ+xZS+0bDjlpVg5WsxYksk/dcnTk9Nsz8+MolBg8bXGt+MjURkRgV\n", "9tTWIho3JMfhJyRbMbXvG/ieSTXBCsNgeqYxmZ857bXI+vgqOREptL5uSC7Hn2JKexf+tNjGVY4P\n", "2ZAMM/hFgzGWyTlGcjl+V+xc6AARcaEDRMSFDhCRjurO2NcjGQ08ndk3L/XaU/nGqa3k1xfYy2A8\n", "sAPwKPDHEgOW47zX4DvAe4ANgMfxvat1gO+VYE6A6CIihdVMj+RXwKWZfRvheyTvqXJ8lN0zg92a\n", "vCpsfOisIjIk9XWP5AV8zyNtVOq1amYBTybPFwD3A3OTbZf8zHW7BHMMhn0FTtgKJh/lM93xS7gK\n", "sA/BhsmbHp1T+ZBp34XVboG75/r1nAHch2HD62Ec8OgPYd85cPuVcDIwb384/yZ4NvSfV9va1nb0\n", "2w44Mtl+kgJrpkdyLH5t4/Rg+8nAMzWOj7JH0ojBIY3GVgxKBns30avZLPlYF/LPFBkXOkBEXOgA\n", "EXGhA0SkUHXnKsCk5LEYf0/IJJLfyIHpwK2p41fH39dxFbAl8DF8w3JCjc8vVGGkmZ/KZWeDjxr8\n", "2+CZFm6UzD4+gf6TpLnQASLiQgeIiAsdICKFqjsdfnxjGfBW6vllyesz8QPQaVvhb0h8DX/apt4N\n", "iYUqjEbMD95nG4nLLJnF2GBjSyagND+1y4DZjkPnF5HC6Ku6s1N9VxgGkwyuM7988G51jlvd4I1M\n", "YzIiz6wiUlh9V3d2YsgXhsHKBjbHNyR7hM4TCRc6QERc6AARcaEDRKSjulN3WveZkp+T7NvJ5myD\n", "/UPmEREpmiHfIwEwmJA5xfWWwSZJb0W/PIhIlurOFBVGwvz6K3+tcVXXRQaPGHw1dE4RiYLqzhQV\n", "RoUDMDi2icuFf2XwLoMPGnzLYEzg7N3mQgeIiAsdICIudICIqO5MUWFUuPITg+0NNjMYb3C2wcwG\n", "DcudBh83WC1g/m5yoQNExIUOEBEXOkBEVHemqDCalNyDcpH5tVIsuXT4qSqNyjmmxbpE+p3qzhQV\n", "RhuSGxpXSJ5Pq9KYXFW+CVJE+pLqzhQVRoVr940GE5MxkzMyDco08ytClgw2MBjZxby95EIHiIgL\n", "HSAiLnSAiKjuTFFhVLhufIjBaQ3GUzboxvf0mAsdICIudICIuNABIqK6M0WF0WUGyxkc1qAx2SZ0\n", "ThHpiOrOFBVGjxisajDGYAuD6w2WZBqT/xM6o4i0TXVnigqjwvX6Cww+ZPBqpkE5rtff2wYXOkBE\n", "XOgAEXGhA0Sko7pT02VI20rwS2At4IHU7gsNdgkUSUSkY+qRBGJwaKZnoskiRYpDdWeKCiMgg7Uy\n", "jcm6oTOJSFNUd6aoMCpciC81P8Pw06nG5MgQOTJc6AARcaEDRMSFDhARjZFIPJL1UD6S2vXZUFlE\n", "RNqhHkkkzC8RXO6VTEr26RcXkTip7kxRYUQkM17yqsECgxMNTkhOf81NLiEuwt3xIv1MdWeKCqPC\n", "hQ5gcECDO+Kzj+cMPtiDKK4Hn1lULnSAiLjQASKiMRKJUwmuAz6a2nUz/t6TsgWZt6yPX2TrKYMr\n", "DDbvdUYRkSz1SCKUTK8yvs7rOxrsV6OXMif5eZvGWER6RnVnigqjwAymGFxn8B2DZVUalRcNtgyd\n", "U6QPqe5MUWFUuNABOmHwDoNzDb5QpUHZr8WPc73IWFAudICIuNABIqK6M0WFUeFCB+gW8wtpzco0\n", "Jhskp8TWa+IjXK8zFogLHSAiLnSAiKjuTFFh9DGDcTXGUd5jMN7gIIPlQ+cUKaCe1p0bAVtl9u0O\n", "zAHuAU5q4zOPB54AXgPuA6Y0OH5v4C5gITAfuB7YtMaxakj6nME5DS4h3rPOezVYL1JdT+vOa4Gz\n", "UtsbAa8CfwJuBN4AvtTC5x0CLAWOBiYAM4BFwIY1jh+fHH8WMA7YFn/56KM1jldDUuFCB+gFg2FJ\n", "D2S6wR1VGpLFBtcYfNLgLoOHdvH3s0wzWGjwsdB/hsBc6AARcaEDRKSndedTwK6p7ZOAx4AVku2v\n", "Av+vhc+7G7g4s+8R4Mwaxx8IvAmUUvt2A5bh18HIUkNS4UIHyEtyeXHNnsqc2r2XSwyGhc6fMxc6\n", "QERc6AAR6Wnd+Rq+F1L2K+A/U9sTGHxTWS3D8T2YAzL7zwfm1njP+snnfw5/7ns14Ef4U13VqCEZ\n", "opIB+dkt3EVffvzZ4EyDXQ3OSva9ZvCSwQcNjjL4usZepM/1tO58HtgueV4CXsb3Eso2w5/qasYY\n", "fE8iOyZyKvBwnfftDLyIb4Tewo+r1FrnQg3JEGawnMGIpFHZKNm3ucFpyUD8agbHGnykjQbnvw3W\n", "DP1nFOmRjurORoOP9wAn4rv/hwKrArenXt8UeKaTAA2Mww+uzwQm47uii4CrGXi6K20WMC15TGVg\n", "99UNoW3X4PW+2y7BriV4TwmsBE8DrgSjS3B7Ca4pwQ4leLgE15egtBHs9TX4evn9cxnYNc5sf+Y2\n", "eObLfsLJTQw+cCZM3wc+EerP2+b21MjyhNwe6vXDLCr1ZU9tC7yEH6dYBpyeef0K4KImP6vWqa0L\n", "8FeBVXM28IfMvnckWXaucrx6JBUudICIuHovmp/y/gKDbZLezHDzC3Stnbw+xeDeGj2VBw3WNBid\n", "y5+kcy50gIi40AEi0vO6c138QkU7VnltX+CdLXzWXVQfbD+jxvHfAe7N7Fuf6qfIQA2J9IjBSgbn\n", "G7xuML/GFWOjQucUaVOh6s6Dgdfxl/9uAZyHvz+kfPnvdODW1PFT8OMip+BPo22Pv/z3SWBElc8v\n", "VGFI8RisUB54r9ZDCZ1PpE09/bd7AP66+0aPVhyHvyFxCb63ke5ZzAQezxx/IH6AfREwDz9mUmt6\n", "cf1HrnChA0TE9eJDzd9p/x2DV1KNSeynuFzoABFxoQNEpKd157ImH7FQQ1LhQgeIiOvlhxuskemZ\n", "/DG5pPgQq36/U0gudICIuNABItLTuvNZ/GmkacDGvfyiLlFDIkEYfK7GYPzrBv9RHrgXiVRP687l\n", "8QPqP8PfnDgbP83J8F5+aQfUkEgQBismp7lq3Ycy3+L9fyOSW925Hn5KlL/gLwmeQXz/MdSQVLjQ\n", "ASLi8vwyg5EGmxqsYn6Or3Jjcrkl9z8ZfN78RSR5cwG+M1YudICI5F53boa/76PWfFchqSGpcKED\n", "RMSF/HKDyeanXanWU9kl5zgu5++LmQsdICK51J2rAJ8G7gAW429EdHl8cYvUkEiUDH5Y57TXmsmN\n", "kKPM3yclkree1p1TgMvwl97eAxwLrN7LL+yQGhKJksH6Bt9IGo6fGRxRp2H5scFYqz0NkEi3dVR3\n", "NvqHugw/Z9GPgAfrfNn/dhKiiwz95ytz1J5VeahxRFgW5u/TurbBYduV4P4ufq0jwrIIxKGyKOtp\n", "3an7SIrLhQ4QERc6QC0GH0v1RKaaX4wr3Tu5MvX8QvOnmTvhupG7T7jQASKiujNFhSGFY7CV+Zm1\n", "y9urGzxc47TX+SGzSt8KXnd+IHSAlOCFIdINBlvXaEiWWTFuDpZiCVJ3boC/Bv5xdGorVi50gIi4\n", "0AHaYfDpdC/E4PZk+4gOPtZ1J11fcKEDRKSjunO5Fo4dhh8cvAU/bcpHgR8A4zsJICLVleCyEpRK\n", "8IVkV/milg8b7NSF8RKR3GwOfBc/8+4L+MWm3gS2DBmqBvVIpG8ZbJQ5zXVO6EzSN3pad96BbzRm\n", "46dzH5bsfwOY2MsvbpMaEulrBnMzjckaoTNJX+hp3bkMuBv4UGa/GpL4udABIuJCB+iWZP6ubTON\n", "yYoGqzX5Ea6X+QrGhQ4QkZ6OkUzGr5l+FX5c5FQqqxmKSM5KsLgEDwA3pXYvARYa7BEolkhTRgCH\n", "4+8CfQPfU/kasGbATNWoRyJDgsEIg9uqXB68t8E1Bu8OnVEKJfe6czxwFvA8sBS/hnos1JDIkJFq\n", "TF6tds9J6HxSKMH+vQwDPgzcECpAFfrPU+FCB4iICx2g1wzeX6Ux+XiVQ13e2SLmQgeIiOrOFBVG\n", "hQsdICIudIA8GKyQDLzX65W4vHNFzIUOEBHVnSkqDBnyDKakGpNtQueRQlDdmaLCkCHP/CJZv8mc\n", "5to+dC6JmurOFBVGhQsdICIudIC8GaxtcEGmMTnqBPhK6GwRcaEDRER1Z4oKo8KFDhARFzpAKAb/\n", "lW5M5vifW4XOFQkXOkBEcqk7HbBjavso4E7gElLrKERADYlIhsE+Bm9meic7h84lUcml7rwff6kv\n", "wATgdeBC/PK7P8gjQJPUkIhUYbCSwbdSDcmNoTNJVHKpOxcB45LnJwM/T56/B3i2xc86HngCeA24\n", "D5jSxHumAg/jp4J4Dphe4zg1JBUudICIuNABYnHdwKV8Z4TOE5gLHSAiudSdrwCbJc9vA76YPB+L\n", "r9ybdQj+bvij8T2bGfhGqt78XecCfwP2S75vWwZPIlmmhqTChQ4QERc6QCwOgiMzp7j+aHCEVWb2\n", "Hkpc6AARyaXuvA24HL8y21Iqi1m9D9+7aNbdwMWZfY8AZ9Y4fkLyfROa/Hw1JCINGOxY5S74twz2\n", "NyiFzidB5FJ3bg08hO+ZfDO1/wLgx01+xnD8hI8HZPafj58Mspqv4U9pnYBf1vcJYBawbo3j1ZCI\n", "NMFgVYOrDP6aaVAuNFg5dD7JXdC6cyVghSaPHYOfNTg7JnIqvrGo5gf4sZTfJ++bAtyVPKr95qSG\n", "pMKFDhARFzpARFx2h8HTVXood5qf8bufudABItJR3dnsedGt8GuXPJjZPwHfy/hLJyHqWA5YEf8P\n", "+rFk3+H4MZPJwL1V3jMLv3YKwAL8FWdzk22X/NT20NqmwetDaXtSldc/CWw6Fy5N7dx5Lux8C5y2\n", "F2xS8r8ExpC/m9uTIsuT57YDjky2nyQnvwMOrbL/MPxyvM2odWrrAmBOjfd8Cz9Gklaq8TmgHolI\n", "2wy2MXgmGSu5KtU7+Z9k+xDzg/UHhs4qXZf75b9p44GFLXzOXVQfbD+jxvF74H8TSn/3Jsm+yVWO\n", "V0Mi0gUG76xyuiv9aOayfSmOXOrOBVSvuCfjG5lmHYy/mfFoYAvgPHxDVL78dzpwa+r4Ev5ek7n4\n", "buh2wK/xPaRq1JBUuNABIuJCB4iIa/ZAgwdSDcfpVRqT0clxyxscZnC0wQEGmxks37M/Qfe40AEi\n", "kkvdeSPwMwaOqayQ7Lu5xc86Dn/11RL8GEf6N5uZ+Kuz0kYDV+MbnHnAFeiqrWa40AEi4kIHiIhr\n", "9kCDDZOruNZPtj9ucGKmMRlpMLdKI/Nvg3sNPt+zP0nnXOgAEcml7twceAnfAFwBXIkfoHkJmJhH\n", "gCapIRHpMfPrwtc65fVolavAqo2vSlxyqzvH4McybsH3Qk5P9sVEDYlIDpIB+HJDMc/gKIO1Uq+v\n", "aTA/dczGIfNKQ6o7U1QYFS50gIi40AEi4rr1QQaTDSbVGg8x+ECqIfmawb6WOS1t/mrOUFzA745N\n", "z+rO7an8A9m+wSMWakgqXOgAEXGhA0TE5fllBv+ZOc31N4N1DF5I7Ts5z0wpLtD3xqhndecyYL3U\n", "81qPt3oVoA1qSEQiYrCBwWMNLiU2g5+lT41J7jqqO+tN0DYWeBrfWIxt8DlPdhKiiwxNOicSFfNT\n", "Ka2DH1vdps6hvwd2KTWo1AxWLsG/uxhRcqg7VwC+QzEGy9QjqXChA0TEhQ4QERfqiw1GpHogmyX7\n", "vp8ZlB90mstgO4MvmV/p8VLzMxUf04VIrguf0S9yqTtfpXGvJAZqSCpc6AARcaEDRMSF/HKDlQ1G\n", "VNn/51Rjsmpq/7vrnA4b2WEc1+H7+0kudef/Ap/O44s6pIZEpICSy4XLDcReqf0312lIrjC4xQZP\n", "uySty6XuPB5/V/l5+Nl3P5Z5xEINiUhBGZySui9lqsGxVW52nFyjUam3yqo0lkvdWe+qrWV5BGiS\n", "GpIKFzpARFzoABFxoQPUkoyh/K5KI/EjSw0EG+xe5Zh/tfGVrnvpC6+junO5Fo6r9xAR6UjJL2L3\n", "PvxZjpuB3+CX4f5i+kquEtyOX6n1b/j5+QDWSDUqSwLf6Cg1HIFfYCprePJaLNQjERliDA6v0kO5\n", "IXSugsnt1NZ6Vfavg05tiUhgBgcZ/D3TmHw2dK4CCdqQbAe8nEeAJqkhqXChA0TEhQ4QERc6QC8l\n", "lxeXG5KHGhzu8shUEB3VnY3WbE//RfwaeDO1vTz+JsVbOgkgItItJb8OygT8+MlWBqNL8ELoXP2u\n", "0S3x05KfpwLnAItTry3Fr09yHYPXVQ9FU6SIDHHJDY3plVv/A5iBX0zvzUZTsAxRudSdR+Lny4md\n", "/oGICAbn1rjf5Aehs0Uqt7pzBHAQ8HVgzWTfeGDtvAI0QQ1JhQsdICIudICIuNAB8mIwyvyprmxj\n", "sn5yiAuZLzI9HSMpGw/ciu8yrgFcg78B6Nhk+zOdhBAR6baSv0N+f/yUK+viZ+UA+D5wSLBgQ9jP\n", "gf/GD7AvAsYl+3cFHg8Vqgr1SESkKoMzdGlwTbnUnS+TTPvMwIbknfgBrFioIRGRmgy+nDnN1exZ\n", "mX6XyxQpUH3KgQ2BVzoJID3jQgeIiAsdICIudICQSn7i2acA5vpdG4dL0z+abUhmAydm9o0ETsPP\n", "iSMiUhTvTT0fHyzFEPQO4JHk8QZwL36w/WGq3/Eeik5tiUhDBtMzp7i+bUNwAlqDnQx2o4drtmet\n", "DBwK7IAv8D8AP8bP2BkL3ZAoIg0ZTAT+nNn9HDCuBK8HiBSEwaPA+KTSVN2ZUI+kwoUOEBEXOkBE\n", "XOgAsbgcrq9yj8nPbQhVqAaPma83cxtsH4VfJ+BY/IqJ6UcrjsdPrfIacB8wpcn3bYq/YmxRowNF\n", "RBo5Dv4LOAx4ILV7H+Ka0bzXcj2ddxi+4n8dPwHa85lHsw7Bz8t1NH5itRn4hqHRMpnD8afSfg4s\n", "rHOceiQi0pbMXfDN/oJbaAZPdKNH0qyn8FdodXrN9d3AxZl9j+BXQavne8APgU9Rv0eihkRE2mKw\n", "W6ohucbgUoPJoXP1ksFTeTYkC6jchNiu4fgrvg7I7D+fty/prmoffGOzMn7ySDUkzXGhA0TEhQ4Q\n", "ERc6QERcdofB+zJjJn8KkCs3Bk/nOUbyE3yF3ol18FOszMvsfxEYXeM9Y4BLgE8A/+7w+0VE6ir5\n", "dZfuS+3a0mD3UHly0JUxkmZPVU3Fr4H8fvxiV29kXj+tG2GquAK4CH/fSrNmAU8mzxcA91Pp8bjk\n", "51DYnhtZHm3Hs02D14fKdnlf9vU9gMPmwoXJziOB2yPI281tBxx5GKw5gfx8CX8lw4v4rt5DyaP8\n", "vBm1Tm1dAMyp8Z5lyXvKjzdT+6rNOKxTWyLSFQbHlE9xhc7SKwbP5zlG8iKDp0hpx11UH2w/o8bx\n", "EzOPk/GrNE7ET1+f1bd/4W1woQNExIUOEBEXOkBEXL0XDXZPjZWMyClTrgxeyLMh+SewSRc+52D8\n", "JcRHA1vgJ1BbSOXy3+n4dU9qORINtjfLhQ4QERc6QERc6AARcfVeNCilGpL355QpV+bXbMmtITkH\n", "v257NxyHvyFxCX7sI3299kzqr29yJLqPRERyYnB60pBcGTpLLxjM70ZD0uxUABfir5z6E/AglcH2\n", "UhLgS52E6CLNtSUiXWMwFv+L7+vAmqW45hbsmMFLwNqdzrXV7KVfE4E/4huQzYGtMw+JjwsdICIu\n", "dICIuNABIuIaHVDyV4A+C6wI/E8yOD3DYITBOw1W6HHGXhtyMx43Q6e2KlzoABFxoQNExIUOEBHX\n", "zEEGF1SZ3LH8yF48VCgG/8pzjKQoVBgi0lUG69dpSMzg+1bQJXsNXlFDMpgKQ0S6zuBzBl8xWN1g\n", "zyqNyfTQGdthsFANyWAqjAoXOkBEXOgAEXGhA0TEdfJmg5EGj6Qak292KVduDF7Nc64tERFJKcEr\n", "JdgstWtaAW9c1FWuVahHIiK5Mvhtqleyceg8rTB4Tae2BlNhiEiukjvgH0oaki+HztMKgyVqSAZT\n", "YVS40AEi4kIHiIgLHSAirlsfZPDhpCH5p/l7TgrB4HWNkYiIxOFG/Kwfa9H52k15UhtQhXokIhKE\n", "wQmpsZK1Q+dphsGbOrU1mApDRIIwWC/VkCwLnacZBsvUkAymwqhwoQNExIUOEBEXOkBEXLc/MHOT\n", "4tUGo7r9Hd1UzorGSEREorFh6vlBwHWhgjRiuoekJvVIRCQog/GpXsnLofPUYrBc6jSc6s4UFYaI\n", "BGewa6ox+UzoPNUYDEvyvYXqzgFUGBUudICIuNABIuJCB4iI6+WHp8dLevk97TJYIcn3BhojERGJ\n", "UreWJ++Vcv0fZUMXkgpERKJgsGpyee1Sg9VD58kyWCnpkSxBdecAKgwRiYbB3NQpri1C50kzv1yw\n", "mV+HXqe2pCoXOkBEXOgAEXGhA0TE5fAdP0k9/4vBAxHNENy1U1tqSEREemcW8H9T29sAJ4aJMkj5\n", "PpJC3IWfJ53aEpHoGGwd21Vc5pcNNoNFRJIpFioMEYmSwahUY/J+g+GB84xMsryC6s4BVBgVLnSA\n", "iLjQASLiQgeIiMv7CzNzcT2Y9/dnsqyZ5FiABttFRApj6/Rzg1XLGwZbGFxj8IWcshR+jOR44An8\n", "ZWf3AVPqHOuAG4DngMXAA8BRNY5Vj0REomWwvMG8TM9kmsEXktUV356G3uAig70NNu9RlnWS7/on\n", "Baw7DwGWAkcDE4AZ+MGeDWscfxJwGrATMBY4Fn9L/2FVji1cYYjI0GIwxuCwTGPS6LFCD3Ksm3z2\n", "fApYd94NXJzZ9whwZguf8VPg2ir7C1cYPeRCB4iICx0gIi50gIi4kF9uMNbgzKQyf9bgSwY71GhI\n", "xvbg+8uD/y9SsDGS4cD2wOzM/tnAzi18zkginp5ZRKSREjxZgpOB1UrwjhLMKMEfgM2AczOHn1mv\n", "MTFY0WBi6xGAAo6RjMGHzo6JnAo83ORn7Is/NTa5ymvqkYhI4SUNw2WZXskPqxy3msH9qWP2b+E7\n", "1k/e8zwFqzs7bUh2wV/zfEyN1wtVGCIi9RgclGok7qvy+uxMY9NU78LgdIObkvc8R4d157BO3tyG\n", "l/CLqGSKTDyBAAAJJUlEQVTXMR6FbxXrmQLcDJzC4DGWtFnAk8nzBcD9wNxk2yU/h8J2+XkseUJu\n", "l/fFkifk9iTg+xHlCbk9lcjrhxLMN9gUeHQu7LAnHPor+InB+Cvh3LmwR+ZN6eVza37+XPjGrGRj\n", "/dQlyEVyF9UH28+o855dgYX4v/h61COpcKEDRMSFDhARFzpARFzoAM0wWCXT65hU5+quhmd2ktNm\n", "6fc8QwHrzoOB1/GX/24BnIdvJMqX/04Hbk0d7/D3j5yN77mMTh7rVvnswhWGiEgjBjvVaTzONji4\n", "fJrKBvZKqn3WOpn3P0VB687j8DckLgHuZeCYyUzg8cz2W/hzf+lH+piyQhaGiEgjNRqROw3WMyiZ\n", "Pw1mBu+t8xklgx9nPuNJVHcOoMKocKEDRMSFDhARFzpARFzoAK0wODxV+e+RXLFVSr2+tPx6nc/Y\n", "tkpj9DgFG2wXEZH2XAOsBtxc8qejst6++91gnZK/uCmr2ozD+gU8QwUiIkOSwXmpXsYRNY7ZsUqP\n", "5FFUdw6gwhCRIclg1cx9JYMG3Q12r9KQPILqzgFUGBUudICIuNABIuJCB4iICx2g2ww+nWogxmde\n", "27LGgP3fKNhcWyIi0ju/SD3frfzE/O0Vtw4+HOjBzMJFpx6JiAxpBlOTnsYVyfbade5BMfMT4KpH\n", "IiIibyvPyTUh+TmpwfFr9DBLIalHUuFCB4iICx0gIi50gIi40AF6wWB0uadhsGeNXsjvDV5N3Xei\n", "HomIiLxtXvJzTeCXNY75N/CvfOIUj3okIjLk1RkPedbgeoOJBg91q0fSb1QYIjLkGdyTNBIfMfhT\n", "qiG5NHXMb9WQVKfCqHChA0TEhQ4QERc6QERc6AC9YjDGYIfk+TiDWwzutdRaUFZZ2KrjhkRzbYmI\n", "9JmSX/XwueT548DeVQ7TGEkN6pGIiDTB4HxdtSUiIp14q1sfpIakf7nQASLiQgeIiAsdICIudIDA\n", "unYGRw2JiMjQtCx0gFhpjEREpAkG52iMREREOqFTW9KQCx0gIi50gIi40AEi4kIHCEwNiYiIdERD\n", "ATWoYEREmmBwtsZIRESkE4+HDhAr9UgqXOgAEXGhA0TEhQ4QERc6QEgGwwxOM9gRzbUlIiKtKsGb\n", "wKmhc8RIPRIRkdYVbozkeOAJ4DX82sJTGhy/NfBr/Ipe/wBO6Wk6ERGJ2iHAUuBo/ML0M4BFwIY1\n", "jl8deAH4CTAROABYCJxY43j1SCpc6AARcaEDRMSFDhARFzpARApVd94NXJzZ9whwZo3jjwMWACum\n", "9n0D3zOpplCF0WNTQweIiMqiQmVRobKoKMypreHA9sDszP7ZwM413rMT8Fvg9czxY4CNux2wz6wR\n", "OkBEVBYVKosKlUWX5NmQrAMsD8zL7H8RGF3jPaOrHD8v9ZqIiAQW+w2JOlXVvrGhA0RkbOgAERkb\n", "OkBExoYO0C/yvI/kJfyKXKMy+0cBz9d4zwsM7nmMSr2W9XfU+KR9KnSAiKgsKlQWFSoL7++hA7Ti\n", "LqoPtp9R4/hjgVcYONh+MvBM96OJiEgRHIwfOD8a2AI4D385b/ny3+nAranjV8f3Vq4CtgQ+hm9Y\n", "Tsgpr4iIROg4/A2JS4B7GXhD4kwGTyS2Ff6GxNeAZ9ENiSIiIiIiEoqmV6lopSwccAPwHLAYeAA4\n", "qsf58tLqv4myTfGzKizqUa4Q2imLqcDD+DMEz+FPL/eDVstib/wY7kJgPnA9/t9I0e0K3Iiv/5bR\n", "3MUF/Vxv9nx6lSJptSxOAk7D3+A5Fn8RwxvAYb0O2mOtlkPZcOAPwM/x/yb6QTtlcS7wN2A//L+L\n", "bYEP9TRlPloti/HJ8WcB4/Dl8Evg0Z4n7b29gNPx9d9i4IgGx/dzvQn0fnqVImm1LKr5KXBt1xKF\n", "0W45fA/4If63s37pkbRaFhPwleeEXoYKpNWyOBA/pXoptW83/G/wa3U9XTiLaNyQtFVvxn5DYpmm\n", "V6lopyyqGQm83K1QAbRbDvskjy8ysOIosnbK4sP4C1v2Tn4+AcwC1u1NxNy0UxZ3Aq8Cn8XPvrEa\n", "cCRwD8X+P9KOturNojQkml6lop2yyNoX2B24pIu58tZOOYzB/5k/gT//2y/aKYtx+IrhYPxvqYcD\n", "mwM3UewGtp2yeB7foJ6OHytagL/dYL8eZYxZW/VmURqSdugO9+p2AX6M/438vsBZ8nYFcBH+svOh\n", "bjn86YvDgTuSx+HAu4HJAXOFMA4/uD4T/2d3+NNAV1PsRrUdbdWbRWlI8phepSjaKYuyKcAt+Ksw\n", "sueQi6adctgN+Cb+QoM3gEuBVZLnn+lNzFy0UxbP48cFHkvteyz5nI26HTBH7ZTFMfjZMr6Ov6Lx\n", "t8AngffhT/UMJW3Vm0VpSJbir7L5YGb/HsDvarzn98B7GThotAf+psanuh0wR+2UBfjLAG/BV6Qz\n", "ehMtV+2Uw1b4K3LKj1Pxl4duS7EvPGinLO7Az7U3LrVvHP600FD7/1HCD6ynlbeLUkd2S7/Wm2/T\n", "9CoVrZaFw1/6dzb+t4vRyaPoA6utlkPWkfTPVVutlkUJf2pzLjAJ2A5/70C9X0aKotWymILvxZyC\n", "v3dke/zlv08CI3JJ3Dur4P9+J+HrgFOS50Ox3nybplepaKUsZuL/oyzLPLLlVUSt/ptIO5L+uY8E\n", "Wi+L0fhxgIX4AdUrKP4vF2WtlsWB+IZ1Eb4srsdffFB0jsr/93QdcFny+lCrN0VERERERERERERE\n", "REREREREREREREREREREREREhqax+DuNtw+cQ6RtQ21CMpFYDbXpykVEJOMr+DW+l+CnJD+T2r2N\n", "ZfjJ8MrP04/bc8gq0lXDQgcQ6QPTgWPxM6T+Gr9K3/Y0t0jQu/FLuu6JXwtjaY8yiohIpFbFz5L6\n", "uSqvjaVxj6TWMSKFoTESkc5MxC8CdFvoICKhqCER6Z3yKnvpgfQVQgQR6SU1JCKd+St+Nb4PVHlt\n", "fvJzTGrfpMwx5TGR5bucS0RECuQs4GX8ioub4AfQj01e+x1+ffSJwM74wfj0GMkwKkugjgJG5hVa\n", "RETiUQK+Dvwd3zt5Gvh28trm+IZkMf6qrCkMbEjAry3+FPAmuvxXRERERERERERERERERERERERE\n", "RERERERERERERERERCTj/wNrcKHzjqavLQAAAABJRU5ErkJggg==\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "probs_rand = numpy.ndarray((1000, 2))\n", "probs_rand[:, 1] = numpy.random.random(1000)\n", "probs_rand[:, 0] = 1 - probs_rand[:, 1]\n", "labels_rand = numpy.random.randint(0, high=2, size=1000)\n", "\n", "optimal_AMS.plot_vs_cut(labels_rand, probs_rand)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The best quality" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "1.0801533437621837" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "optimal_AMS(labels_rand, probs_rand)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Hyperparameters optimization algorithms\n", "\n", "__AbstractParameterGenerator__ is an abstract class to generate new points, where the scorer function will be computed. It is used in grid search to get new set of parameters to train classifier.\n", "\n", "__Properties__:\n", "\n", "* `best_params_` - return the best grid point\n", "\n", "* `best_score_` - return the best quality\n", "\n", "* `print_results(self, reorder=True)` - print all points with corresponding quality\n", "\n", "\n", "The following algorithms inherit from __AbstractParameterGenerator__:\n", "\n", "* `RandomParameterOptimizer` - generates random point in parameters space\n", " \n", "* `RegressionParameterOptimizer` - generate next point using regression algorithm, which was trained on previous results\n", " \n", "* `SubgridParameterOptimizer` - uses subgrids if grid is huge + annealing-like technique (details see in __REP__)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Grid search" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__GridOptimalSearchCV__ implemets optimal search over specified parameter values for an estimator. Parameters to use it are:\n", "\n", "* estimator - object of type that implements the \"fit\" and \"predict\" methods\n", "\n", "* params_generator - generator of grid search algorithm (AbstractParameterGenerator)\n", "\n", "* scorer - which implement method __call__ with kwargs: \"base_estimator\", \"params\", \"X\", \"y\", \"sample_weight\"\n", "\n", "* Important members are \"fit\", \"fit_best_estimator\"" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from rep.metaml import GridOptimalSearchCV\n", "from rep.metaml.gridsearch import RandomParameterOptimizer, FoldingScorer\n", "from rep.estimators import SklearnClassifier\n", "from sklearn.ensemble import AdaBoostClassifier\n", "from collections import OrderedDict" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Grid search with folding scorer\n", "\n", "__FoldingScorer__ provides folding cross-validation for train dataset:\n", "\n", "* folds - `k`, number of folds (train on `k-1` fold, test on `1` fold)\n", "* folds_check - number of times model will be tested\n", "* score_function - function to calculate quality with interface \"function(y_true, proba, sample_weight=None)\"\n", "\n", "__NOTE__: if fold_checks > 1, the quality is averaged over tests.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# define grid parameters\n", "grid_param = OrderedDict()\n", "grid_param['n_estimators'] = [10, 20, 30]\n", "grid_param['learning_rate'] = [0.1, 0.05]\n", "\n", "# use random hyperparameter optimization algorithm \n", "generator = RandomParameterOptimizer(grid_param)\n", "# define folding scorer\n", "scorer = FoldingScorer(optimal_AMS, folds=4, fold_checks=2)\n", "\n", "grid_sk = GridOptimalSearchCV(SklearnClassifier(AdaBoostClassifier(), features=features), generator, scorer)\n", "grid_sk.fit(data, labels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Print best parameters" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "OrderedDict([('n_estimators', 30), ('learning_rate', 0.1)])" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "grid_sk.generator.best_params_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Print all qualities for used parameters" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2.245: n_estimators=30, learning_rate=0.1\n", "2.106: n_estimators=20, learning_rate=0.1\n", "2.033: n_estimators=30, learning_rate=0.05\n", "1.777: n_estimators=20, learning_rate=0.05\n", "1.771: n_estimators=10, learning_rate=0.1\n", "1.767: n_estimators=10, learning_rate=0.05\n" ] } ], "source": [ "grid_sk.generator.print_results()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Grid search with user-defined scorer\n", "\n", "You can define your own scorer with specific logic by simple way. Scorer must have just the following:\n", "\n", "* scorer(base_estimator, params, X, y, sample_weight)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Prepare reweight function" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def normed_weight(y, weight):\n", " weight[y == 1] *= sum(weight[y == 0]) / sum(weight[y == 1])\n", " return weight" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Define scorer, which will be train model on all dataset and test it on the pre-defined dataset" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from sklearn import clone\n", "def generate_scorer(test, labels, test_weight=None):\n", " \"\"\" Generate scorer which calculate metric on fixed test dataset \"\"\"\n", " def custom(base_estimator, params, X, y, sample_weight=None):\n", " cl = clone(base_estimator)\n", " cl.set_params(**params)\n", " cl.fit(X, y)\n", " res = optimal_AMS(labels, cl.predict_proba(test), sample_weight=test_weight)\n", " return res\n", " return custom" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# define grid parameters\n", "grid_param = OrderedDict()\n", "grid_param['n_estimators'] = [10, 20, 30]\n", "grid_param['learning_rate'] = [0.1, 0.05]\n", "grid_param['features'] = [features[:5], features[:10]]\n", "\n", "# define random hyperparameter optimization algorithm \n", "generator = RandomParameterOptimizer(grid_param)\n", "# define specific scorer\n", "scorer = generate_scorer(test_data, test_labels, test_weight)\n", "\n", "grid = GridOptimalSearchCV(SklearnClassifier(clf=AdaBoostClassifier(), features=features), generator, scorer)\n", "grid.fit(train_data, train_labels, train_weight)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "(187500, 62500)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(train_data), len(test_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Print all tried combinations of parameters and quality" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1.587: n_estimators=30, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n", "1.583: n_estimators=20, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n", "1.583: n_estimators=20, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n", "1.575: n_estimators=30, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n", "1.575: n_estimators=30, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n", "1.550: n_estimators=20, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n", "1.384: n_estimators=10, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n", "1.384: n_estimators=10, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n", "1.380: n_estimators=10, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n", "1.380: n_estimators=10, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n" ] } ], "source": [ "grid.generator.print_results()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Results comparison " ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from rep.report import ClassificationReport\n", "from rep.data.storage import LabeledDataStorage\n", "\n", "lds = LabeledDataStorage(test_data, test_labels, test_weight)\n", "classifiers = {'grid_fold': grid_sk.fit_best_estimator(train_data[features], train_labels, train_weight),\n", " 'grid_test_dataset': grid.fit_best_estimator(train_data[features], train_labels, train_weight) }\n", "\n", "report = ClassificationReport(classifiers, lds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ROCs" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": [ "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEiCAYAAADTSFSPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n", "AAALEgAACxIB0t1+/AAAIABJREFUeJztnXeYJFX1v9/eBMsuu2R2l7QsOYpkSV78iSgCBqIisoCI\n", "iCAGvoqiIkkwExRFgUVAgpGgRGFAQZIoUaIsaYFd2DCbw8z5/XFuUTU11T3dPV11q2fO+zz1TFe6\n", "9em7vffUvefec8AwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDKM9mQJ0+20p\n", "8CpwOTA+oCbDGLAMCS3AMHJAgNuBccB6wJHAnsBvQopqgiHY/1GjDbAfqTEQqQCLgenANNSo/A7Y\n", "OXXNt4BXgEXAY8D+qXImAFcBbwHzgX8DrsZzxwIX+WcuBJ4CDvbnJgNzU9c7tNe0SuqaDwFP+O9w\n", "vP+7Sures4FHE/u7AHd7na8CPwdWTJzfA7jflz8beADYosZ3MQzDGNRMAW5M7E8CngTuTBz7EjAH\n", "OBTYEPgusAx4lz8/CngO+DuwKzARNTSuyjMrwL2oAfgA2hPai9g4TaY+Q7LUl/Mer2sM8BpwbOpZ\n", "U4Gv+v2tfNlfAjYAdgTuQ40nwDBgFvB9YH1gY/+9N63yXQzDMAY9U9AGeS6wAG2sb6TnW/1rwKmp\n", "++4CrvCfjwE66d0TqMZeQBewSZXzk6nPkHQD705d9yPgnsT+bqjRm+D3fwP8OnXPNr6s1Xz53Wiv\n", "xDBajg1tGQOVu9HexY7ABcB7gTX9uTGo4/3e1D33Apv7z+9Gh45m1vm8dwOvA880LxlQA/Gf1LEr\n", "0V7ROn7/MKADHUID2A74FGqoou0fqK9oA/Q7TAFuBW5Cey5RWYbRb8yQGAOVhcD/0KGmLwIPA+fV\n", "cV934nOlhXq6M8obnnHdYtQAJPk38DRqQIYDB6HGJaIC/Ao1nNG2NbARsR/lKGAntGezP2rwPtDc\n", "VzEMwxj4TKGnjwS0R9KNvr2DOqS/lbqmA50mDPAZdGhr1Tqf+X50aKua3+FD/vlJB/jJZDvbszgF\n", "eBzYDx2uG504dyU6LNcIf0UnEhiGYRgZTKG3IQHtlVznP3+R2Nm+MXA6Oqy0lT+/AvA8OkS0G+qw\n", "78vZfh/q1P8A6tTeC/iIP78KaiQuRJ3oBwAvUr8hWRc1VP8Grkmd2wqdrXUROsS2IbAv8At/fn3g\n", "HNSBvx46FfpV4BtVnmUYpWYP4Ab0R9wNHFHHPVuh490LyH6LNIw0l6G/szSfQJ3w66MN/6nAy+hw\n", "0qP0nv67Ftpoz0Ib6n9R22E9FrgYnXa8EB1WOzBxPhpSWgDcjA5VddHTkHTWKP9uf/2+Gee282XO\n", "Aeah05lP8+fWAP6A/v9ZBLyEGpahNZ5lGKXlQ8CZ6NvYfODTfVw/BngD/c+8ub+vE/hyjhoNwzCM\n", "NmEufRuS49DFU8sljn0TfbMyDMMwSkDZZ229B10Qtjhx7DZ0/vx6QRQZhmEYPSi7IRkHvJk69mbi\n", "nGEYhhGYshuS9Hx6wzAMo2QMCy2gD96gd89jzcS5HoxgpWVLmG0zUQzDMBrjBXTaeFOU3ZD8EzgX\n", "dbZHfpK90DhJL6Uv9kakAiC68Gsd1JeybnLrprLuIpZfeyEjhy1kJAtYgay/c1lxySxWnvU2q855\n", "i9XmTmeN+TNYfeEMVl8yk1WWzmYluhi2HLrmYGSVv8vnVju9EXR65wL4+nA45w39zMIc/i6GSrv0\n", "GE8jngo72DkNq4uI07C6iOjX/+WiDckoNGwD6LDaemhwubfRcN7fA3ZAVwkD/Bb4DrrA7Ew0IN7X\n", "qOMfv6Kzwp7yWwpBWDh0BRaOJ2Vk6Gl4VkJ7QGv2LuMd3kDXIrzk//bYDuK6mb/noOWpbmiq/W3k\n", "2ujvcv7zSN9hG9NXPfUDAZmHrot41G9PoQ+egU7TLouxmRhaQImYGFpAiZgYWsBAoZWxhOrBEYfy\n", "lsTzp6CxgC5DQ1lMStyzJfAzNPjeTHS17hlVyk+W2W9EG+J16G1gom1t+l7UtYAMA0NseF6twJIW\n", "KR7CO4Zk5Ytg1qk0Z5Dq+Zuckl2NZahBiQIJZn3u63z0eWE/jNIUdLGfYXWRZApWFxH9ajuLNiR5\n", "01JDUsfDhqJTkdMGJml4xvZdzDu9mmo9m5mVxrueDo0dlRMyFP1uW6BBArdBe5trAqujQ4tZQQmb\n", "pZvaxmYWOuR5M1SeTN3ryLUu2gqH1UWEw+oiwgxJgkINST2INrZRryarZ7MW9fdqMofPaGmvppXI\n", "cqhBGeP/rpixX+/nRnxNT6Bxsl5Gw5XMQLMczvDbq1CZ17/vZhgDCjMkCUpnSPpC1E81nuyhs8jw\n", "9OXrEDQXxjvG5acw4iT4W+LYrCZ6NSVChqMRb6sZnFWAzYBD6BkZF33pdD0KQzMMzkhsbwLPon6e\n", "J6FSLXhiu+Owt/AIh9VFRL/azrLP2hrwVNSP8IrfMvG9mlqTAtZCh9gm4POSb6O3npgoZr7UHj57\n", "rZy9mojKUnT4albt6+SL6HDbemhPcHV4bSu0nlf320Q0cOP6Ncp5FjXOb2T8jT6/DZXuqkUYxiCh\n", "rd7e66DteiStwPdqIl9NVs9mPXrmwahSDNOoPTFgdnv3aiJked4xMqyORsedgEaa3goNK18Py9Ce\n", "zBvEQ2jT0WG1P0Flemt1G0Zu2NBWgkFpSOpBdCpzrUkBE+g70sE8sg1MslezNA/9xSIro8ON49EF\n", "sem/0eeV+yhoDrF/JvLRPAY8BDwOlVoh4w2jSMyQJDBDEuNoYPxXdIZVX72a0VULeKcYplF9UsBL\n", "wJwAvRpHLmPhshyxYYl6NuOAo9Hhxr6mSEf+mUtRA/MIMDPntTcO8wtEOKwuIsyQJDBDEuNo4X8S\n", "0Xqt1quJDM0E+q7/udQePpuWQ6/GUXiDIUPQHsuqwGqooRmProfanjgTY5o5aLiK9PYEVGa0QJjD\n", "Gs8Ih9VFhBmSBGZIAuJ7NWtRfarzemh0g1p009NX06t3U9EcNW2ODEUX274H2Bo1MBtT25f1IPAc\n", "mqL3Rf/5EajMz1erMQgwQ5LADEmJSfRqak11Hk/f/4adaCP6EJqb/W+Vnjlr2hSpoL2XDVLbtqjR\n", "qcZ8dOrys2iO+duB52xGmdEAZkgSmCGJcbRht11gBD17NemezXpomJYk84C/An8C/lrpnffc0YZ1\n", "0RNZH43OujY6bXkj4FB0GDArgsBstAfzAHC/fq68xYCoi5bhsLqIsLYzwQCYmtoyXGgBeSDqiV5V\n", "YA+B7wj8R0AS2xKBmwU+K3GwTRdSc75IRWeZyb4g3wD5I8i0nlXyzvY8XHM7yIkgO/nJAoMZF1pA\n", "iehX21mvBRqKOgg3RhfHzUaHFv6FzqUvC2ZVByGib+gf9dtuxNOYBU1FcANwF/B4RUPgD3CkgvZc\n", "dkps26PBNpMsAf6D9loeAO6EyusFCjXKQ65t5w5ohMy5qBN0DroCe47fn+fP75iXgAaxHskgR2AN\n", "gaMFbhJYlHol7xJ4SuBqga8JfFB0VtUgQIaBbANyLMhlIE9l9FiWgTwCciFIX5MijIFFbm3nn9BV\n", "uxeiyaTSi69WBvZGQ7y/CfwxLyENYIYkxoUWEBqBFQUOuhJu9AakK2O8p1vgLtHZU4MBF3+UsSDv\n", "B/kmyF9SVdMJ8guQXdUIDUhcaAElIre280Tqj7g6kp5xnUJhhiTGhRZQIhyAwEiB7X2P5XyBzpRR\n", "uUdgX+l7hX8746qfkrEgnwG5L6O3civIoSBbDiDD4kILKBHWdiawyjAaQmCcwNkCsxOt5hMCR/gZ\n", "ZIMU2QLkJ1Wc9gLyIsjFIKeDHA/yUR9axmhPCmk7rwf2o/xvamZIjKbww2BfFng10Vq+JnCG1IwS\n", "PNCR0SCfAjkX5O4ahkVAFoGcBbJ6aNVGwxTSdl6FLnqahuZV36j25cEwQxLjQgsoEa7eCwVG+N7I\n", "E6lW8u8Cx0jfgRrLjut/EbIRyMdAPgdyGsgFGcNhd+h1pcaFFlAiCms7xwLHoauJu4F7gCPoPaUw\n", "JGZIYlxoASXCNXqDwBAB52d4zUu0kIsF/iDwMakvb33ZcPkVLTuD3F+lt7IPGhamTLjQAkpEkLZz\n", "S+AnwCJ0Tckvgc1DCElhhsRoOQJjfC/ldj/LK2odZwn8UmD3Ae6gbxDZGOTyKgblVpDzQI4DcWhw\n", "SyM8hbedE4BvogsS56IhsG9GFyaeXLSYFGZIjFwRWEvgq9J7Rf2LAmcKbBpaY3mQYSCfBrm5D9/K\n", "n0BOMaMSlELazhHAQcQG40HgGHrmp9if8FFZzZDEuNACSoTLo1CBrQTOEXgl1TI+LHCSaG6SsuHC\n", "PFaGet/KfiAng1ydYVBe9UalqMkNrqDntAOFtJ1vATOBC9CQ11msDEwtQkwNzJDEuNACSoTLs/CE\n", "P+XXAnMSLWOXwC0Cn5K+k4IVhQstIEaGoDG/LskwKo+ia1ryrDeXY9ntRiFt5+HUvzgxJGZIjKAI\n", "LC9woMCfRQNIRi3jfIErRcOyDJQFfS1ERoKcDfJQyqAsAPkxyB4g40OrHMAU0nZeRnbCnVGoj6Qs\n", "mCExSoNolOLjBO5NvW6/IfATge3EgoxmIKO8b+XZjJ7KcaHVDVAKaTu70XzUaVYHuooQUCdmSGJc\n", "aAElwoUWIDBJ4FsCz6Raxv8KfFOKW/ToCnpOC5BhIF8B+W3KmPwB5BD6H1jStULlACHXtnMVNDpq\n", "NzobZZXEtjpwJLpIsSyYIYlxoQWUCBdaQIRoPpUdBM4TmJ4yKn8XOFb0/1deuBzLzhFZDuSJjB7K\n", "1/thUFwrFbY5ubad3X1sXcCpeQpoEDMkRtsgMFzgQwJXCSxItI5LBP4kcIC0h2+yIGSoX3tyRsqY\n", "vA3yMshmoRW2Mbm2nc5v3cDHEvsO2AVNiVomzJAYbYlorK/DBW6VnuHuZwlcLJoR0tZZvINU/FTi\n", "tHP+HyBHgmyo1xh1UkjbOZH2+BGbIYlxoQWUCBdaQCMIjBcNIPlIahznJdFIxf2JIuFapbMcSMUb\n", "jn9mDHsJyF41bnZFqWwDcms7t0VT7Eafa21lwQxJjAstoES40AKaRWALbzxeSrWQj3hj0+iUWJeH\n", "znIgW4N8DeSZRFW9AbJnlRtckepKTm5tZ3KmVl9+krJghsQYkIguetzDD3Mlc6d0Cdwm8GnJnqI/\n", "SJE1QJYkDMqTIO8LrarE5NZ2TiQezprYx1YWzJAYAx7RRY8fF/ij9Fz0uEDgtwL7iC16BGRVkOkg\n", "XQmDcjvFhWBpJ6ztTGCVEeNCCygRLrSAvBBYxU8Z/ntq6Gu6n2K8g/Rc9OhCaQ2HrJ3ymyzTtSnn\n", "fB+kTGkwQlJI23kC8KmM458CPt9gWZ8HXgQWAg8Du/Vx/T7A/UAnMAP4M9UTa5khiXGhBZQIF1pA\n", "EQis7xc3Pp0yKs+ILoacxCCpi2xkSzS8/VKtlrsEDcHyXZDhodUFppC28wWyG/zdgecbKOcQYAlw\n", "NLAJcD4ain6dKtdv6K8/B/1P8C7gFjSEfRZmSIxBj+iix+1Ew7C8kTIq94qGbVk1tM5wyLp+iGta\n", "ompuId8AkWWnkLZzEdm+kPX9uXp5AE2CleRZ4Owq1x+Ihq1Pds33RJ38Wat/zZAYRgKBYQJ7C1wh\n", "GjgyuejxetEAk4N00aMsB3KQ96MIyMMgZQz9XwSFtJ1T0QWJaT4OvFpnGSOApcABqeMXAh1V7hmP\n", "5jj5LDoVeUXgcnSoKwszJDEutIAS4UILKAMCo8/Q5Fu3pBY9zhENge8G2aJHp39kQ5AXEr2T60E2\n", "DqqseAppO88GXgE+AAz3297+2Ll1ljEB7Umkh8i+DTxd475dgOmoEepC/SqrV7nWDEmMCy2gRLjQ\n", "AkqEAxAYJ5p86+HU0Ncrosm6tgysswhc/FHWzFjMuCflyzOfF4W0nSOAq1FDsMxv3cA1/lw9NGNI\n", "JqFG5FzUP7I7cJffssIfCDAFOM1vJ9GzEXG2b/u233NfYDOBM2+BN+5KtKI3wfPnw0USh0Iqhd78\n", "9oc4+MWlcRXcFTnkP1oOfS3dd2hbOQVtKwt9Cd8IONhvjXb9qg1t/Qw1DFmcC/wrdWwt1CDtknG9\n", "9UgMo0n8osfdBH4hMDPxat4tcIfAZIExoXXmj6wBck2qd3JTaFU501Zt5/1kO9vPqnL994GHUsfG\n", "k92zgTarjJxxoQWUCBdaQIlw9VwksJzARwV+L7A4tejxaoEPiw5xtzOu9mlZLWVMHgLZG2QtBl5A\n", "yNzazvPRDIigudrPz9ii4/VyMLAYnf67GXAeuj4kmv77PeCOxPW7oX6Rb6G9oW3R6b9TgayFRGZI\n", "YlxoASXChRZQIlyjNwisLHCMwN0pJ8IMgQsEdpL2zPTo6rtMvpXhP5ntpxCXLQJ6s+TWdnYAKyU+\n", "35WxRccb4Th0QeIitLeR7FlcBvwvdf2BqIN9LvAmuiBx0yplmyExjBwRWE/gFIGnUi3rcwLfEV37\n", "NQCRSSA/BbkHZH7KqHwVpN2nUOfWdq5H+00FNENiGAUguujx3QI/Eng9ZVT+KXC8wGqhdeaDVEAO\n", "7d1LaWty099FHP33TuLeSZlp93/MVuJCCygRLrSAEuFaXaDAUIG9BC4XmJdoWZcK3CBwsGQPRYfG\n", "9e92GQJycsKY/LWNfSe5tZ2zgS3852RI+TJjhiTGhRZQIlxoASXC5Vm4wCiBTwj8RWBZwqh0Clwq\n", "8D6J8xyFxrWmGPlTajFjO05CyK3t/D3qk+hADck/yPaT3JmXgCYwQ2IYJUFgTYETBR5MjQG9KvB9\n", "ga1Da2wdcglIt/+KN7ahz6RfbWetbtgKwFGo8+xE1BG+sIqAE/ojooUI7Tl7xDAGNKJBWg9DI4Yn\n", "84E8DlwJ/LZSf7ilkiI7oi/XKwDHQOXXgQU1QiFtZwewct4PaQHWI4lxoQWUCBdaQIlwIR/unfS7\n", "CPxc4O1EL6Vb4E6BIwXGFiTHtb5IOd1/pQUg7219+blhbWcCq4wYF1pAiXChBZQIF1pAhMAIgf0F\n", "fiewKGFUFgpcK7Cf1B+CqRlc64uUtUD+5b9KN8gP22SYK7ehrfOBU4D56MLDrAdV/PET+yOihdjQ\n", "lmG0IaKzQg9Ah75c4tTbwLXo8Nf9lbZ4WZQRwKnAN9CJBU8Ch0Pl30Fl1Sa3trODfBYk5kkb/MgM\n", "w6iFwLoCXxN4IuWkf0Hgu9J4nL9AyE4gzya+QlleuLOwtjOBVUaMCy2gRLjQAkqECy2gXrw/5V0C\n", "PxB4LWVUHhA4QaqnlKgH1yqt1ZFRIJ1e9lyQ4/N/ZlMU0naOIHtB0UjyHcNsFDMkMS60gBLhQgso\n", "ES60gGbwix7/n8AUgbkJg7JM4CaBQ0VnSzWCy0Nrb2R573yPZJ9XzHMbopC28wY0t0eak9DYV2XB\n", "DIlhDHAEVvCG46bUose53tC8v0SLHj0yGuS5hDHpBFkutKoEhbSdbxGvck+yBTCjCAF1YobEMAYR\n", "AqsLfEHg/tTQ1zSBHwpsI6WagCMnJGS+CjIstCJPIW3nArIj7m6ORvEtC2ZIYlxoASXChRZQIlxo\n", "AXkhsJHAaQLPp4zKEwJflzhdRYQLoTOx1kS8M74MaY0LaTsfAE7POH4GGuK9LJghiXGhBZQIF1pA\n", "iXChBeSNd9LvLHChwFspo3KXwNF+urELqHJ/kDcS0nYMp0UFFfGQfdA0ub9Fk1IdjeZwXwbsV4SA\n", "OjFDYhjGO/hFj/v6BY4LEwZlkV8I+ZGcFz3WUjemp42TVcLoUDFFPeiDwL3oAsX5aBDHDxX18Dox\n", "Q2IYRiYCY0VDsPxNNCRL1IK/LXCRwK7F+1Nk54QhuU+nCwfB2s4EVhkxLrSAEuFCCygRLrSAMiCw\n", "9gVqPB5NDX39T+AM0SCTRan5YMqYhIhrWFjbORI4CPga8Yr3DYFVixJQB2ZIYlxoASXChRZQIlxo\n", "ASXCAQhsLXCuaHj7pFF5UDTeVwG9FNkY5CX/6EdBxuX/zJ4CinjIhsBUdBrwMmCSP/5DoEyhks2Q\n", "GIbRFH7R456iCbg6EwblDoHNClCwDsjT/rGLC15nUkjbeRPwK3SRz1xiQ7IH8L8iBNSJGRLDMPqN\n", "wEiBkwRmemOy1K9LWTHnJ6+e6BRdmO+zej64iIfMJA6UljQk62PrSMqKCy2gRLjQAkqECy2gRLi+\n", "LhCYIHBFwjn/qmgO+hyHu3r4TPrU2KqH9ufmIQ1cmzVFbh1gTn8EGIZhlJUKTKvA4cCOwEPAWmhY\n", "+1sltyjElVvQzJEAt4EUlegrd64BLvWfox7JWDSM/KVV7gmB9UgMw8gF70M5NjHctVjgTGk8WGQ9\n", "TxsG0uV7JSe3vvzeDyzgGawFPOu3pahlngU8DaxRhIA6MUNiGEauiMb3ujThjJ8qsH8OT9o3McS1\n", "fevL7/mwnMt/hxWAo4CfARcBnyE7tHxIzJDEuNACSoQLLaBEuNACSoTrz81+AWNyHcqNon7jFiFD\n", "QG7zxf+3deVmPyzn8tsKq4wYF1pAiXChBZQIF1pAiXD9LUBgmMAXE9OFFwp8UxrzP9d6wqhEr2TX\n", "1pSZ/aAcy+7BdsAVwL/QQI1X+GNlwgyJYRiFIzBe4KpE7+TC1s3skvt9sfeD5DVbrJC28zCgC7gd\n", "jfh7hv/chc5oKAtmSAzDCIZogMjF3phcKy3xIcsWiV5JXo73QtrOqcA3Mo6f4s+VBTMkMS60gBLh\n", "QgsoES60gBLh8ihU4KMC8xMBIY/of+9EDkoYk8+1RmnPB+RQZi/mo2FS0myEJr0qC2ZIYlxoASXC\n", "hRZQIlxoASXC5VWwwETRtSZR639b/x3xcq0vbhbIeq1RGhfe4vIy+QtwTMbxzwA3FyGgTsyQGIZR\n", "CkQTbB3ueyXieyknNe+Il2Egd3tj8hhIK8O1FNJ2Ho8GbPwlMNlvvwSmA58HPp7YQmKGxDCMUiGw\n", "hsA1id7JnQJN9ihkpURgx+tBhrZOZv50N7CFxAxJjAstoES40AJKhAstoES4Ih8mcJDAm96YdIqm\n", "/G3CdyIbgcz0xuSM1slrLz4PvAgsRKcR71bHPSehq+gXAdOA71W5ru0qI0dcaAElwoUWUCJcaAEl\n", "whX9QL8q/g+J3slNAuObKOl9iRAqrUjCFaztHN7EPYcAS9Cc75sA56Oxu9apcc+PgWfQ3PATgXeh\n", "aX+zMENiGEap8b6TwwRmJWZ2HdJESdd7QzK7NbLy54vAgYn9S9E1JM/SWErKB1DfSpJngbOrXL8J\n", "anjqfYYZEsMw2gKBtQRuSfROrpGGMs7KCYkpwWv1X07+vAC813/eA+1FHIKGU76pzjJGoAEfD0gd\n", "vxCNIpzF/6FDWl9CE2i9CEwBVq9yvRmSGBdaQIlwoQWUCBdaQIlwoQX43smxAvO8RXhd4MMNlNDh\n", "Dcmp/ZeSPwuJh59+AFzmP28GvF1nGRNQZ3zaJ/Jt1Fhk8Qv/7H/6+3YD7vdblpPKDEmMCy2gRLjQ\n", "AkqECy2gRLjQAiIENhD4e6J3conAmDru3Nvf8oIGeeyPhPx5E4jCGP8H+JT/vBG6WLEemjEkF/t7\n", "koshN/LHdsi43gyJYRhtiWi+k68ILPLG5FmBVfq4ayjIy96Y7Nm/xzfPsDqvuw3N2f4I2qhHixA3\n", "R4eb6uEt1K+yZur4msDrVe55HVgGPJ849rwvZ100L0qaKcRhW2ajhq/D7zv/1/Zt3/Ztv1T7Fdgd\n", "+JdoMNzrOmDzt+DPAu+taEOfcX8FkCnAt+CaU9Dr6nmeQ9cDQoFhrsYCFwDX03PG1Olkx+Cqxv1k\n", "O9vPqnL9XmjvY1Li2Ab+WFaiF+uRxLjQAkqECy2gRLjQAkqECy2gGgLrJ/wmPxCosfBQJvkeyUJd\n", "sNjsI9uHg4HF6PTfzYDzgE5i/8v3gDsS11fQtSYdwDbAu4G7gfuqlN9WlZEzLrSAEuFCCygRLrSA\n", "EuFCC6iFD6/S7Y3J7bVndMnf+hnQMbe2sw5HT1PXH4cOhy1Ch6aSPpPL0NlZScYB16EG5000D4rN\n", "2jIMY8Aj4ASme2MyVWDLKlce5g1J1nB/nY/Kh+nAmehQUi02RIempuclpAHMkBiGMaAQWFvgAW9M\n", "nhZYPuOqkbowUQRk6+Yekw+TgCvRoagnUSf26ejajtOBy4Gn/Pkr6enHCIUZkhgXWkCJcKEFlAgX\n", "WkCJcKEF1IvACn4Wl4gui8i66ufekPy0uUfkyxpofKxrUH/Fs+iQ1NX+eHoWVkjMkMS40AJKhAst\n", "oES40AJKhAstoBEEtk1MDf5kxhXbeUPyFshyjRdvRFhlGIYxYPGr4MXP6NosdbYC8qg3Jgc1XrQR\n", "YZVhGMaAxYdUucobkycERqWuONEbkkYTDlrbmcAqI8aFFlAiXGgBJcKFFlAiXGgBzSAwWuC/3phc\n", "Lj3CRcmqIItBukFqRVXPKLZ5+hObxTAMwyiYCsxDo7EvBD4NHJU4+zbwZ72MIwLIGxBYj8QwjEGB\n", "wBG+V7JQNE9TdCYK5Pi/BgI55tZ2XgpEyeX3oLlEVkVjhsQwjEGDwK8TAR79ovCmAjnm1nYuJU4B\n", "2Y1OAy47ZkhiXGgBJcKFFlAiXGgBJcKFFtBfBEYKPOqNyXWxv0RO94bkyvqLap5a3Z6pwAnElb0L\n", "2jPJ2gzDMIyCqaif5CDUb3IQmgQQ4pxRh4GsHEJbxEfQ0O/dfWxdoQRmYD0SwzAGHQKf8L2SpXEO\n", "E5npeyVfqa+IfFkZNRibAatV2cqCGRLDMAYdfn3JM96YvCawNcjJ3pDcWl8R+ZB0tr8Xc7a3Gy60\n", "gBLhQgsoES60gBLhQgtoJaLpel/2xuS5TXlqZ29IZuiq975uz4eks70Lc7a3Gy60gBLhQgsoES60\n", "gBLhQgtoNd75/rSAdFH5sY+7JSDb9n1r89SyUs8Bv0PT7N4JfByYWeXae/ojooUItb+TYRjGgEY0\n", "e+xDwIIhdD0iDNkNuAwqR9W+LZ+205zthmEYbUgUcn4znrzW90iu6vuWfDFne3viQgsoES60gBLh\n", "QgsoES60gLwQOEdALuHIf3pD8mDft+SPw5zt7YYLLaBEuNACSoQLLaBEuNAC8kJgDYEFc1jR+95F\n", "NItirVvp9rV9AAAfXUlEQVSKYRxwMnARcS9kN2D9ogTUgRkSwzAMQOA2ARnGkgV1hEspJPrvdsAz\n", "aFauo3knpgt7ofnaDcMwjHLxGMA6vDLD72+f14PqNSQ/As4D3o3maI+4Be2VGOXDhRZQIlxoASXC\n", "hRZQIlxoATnzGMAmPBP1NjbM60H1GpJtgSkZx9+gXDnbDcMwDOUxgP24McrfvndeD6rXkCzknfgt\n", "PdgEmN46OUYL6QgtoER0hBZQIjpCCygRHaEF5Mx/ga5VmBm97C8JKQbgYuBGYHlgLjAJdbI/Bvw0\n", "oK405mw3DMPwCDyxmOHRrK1ukFHVL82fscA/UCPSBUzzf+8BRhchoE7MkMS40AJKhAstoES40AJK\n", "hAstIG8EfiMgKzFzujcmW1e/tHnqHdqaA+yOrnb/Oup43xvNRTKvPwIMwzCM3HgAYEueWOb3fx5Q\n", "S9tgPRLDMAyPwESBZefzhe54YWK1S40IqwzDMIwEAle9yHrJFe5ZI1HWdiawyohxoQWUCBdaQIlw\n", "oQWUCBdaQBEI7CuQNCRZfpJCfCSGYRhGe3IbMPv/cUe0n9vCxIGC9UgMwzBSCPzjJH4c9Ui+m31J\n", "8zTTI1kJXZyY3AzDMIzy8vLGPBt93qbVhddrSCaicbUWoVkS30psM6rfZgTEhRZQIlxoASXChRZQ\n", "IlxoAQUyZ3kWRZ/XbnXhw+q87lK0J3IU8Do2hGQYhtFOzBvP69HnBcFEAFu1qKzPAy+i8bsepv7o\n", "wRuhK+vn1rjGDJxhGEYKge88ylaRj+SJ7Euap96hranAcn1dVAeHoLG5zkTH6e4DbgbW6eO+EcA1\n", "wN2YsTAMw2iUeSsxO/q8UigR70OnkG3Uz3IeAH6ZOvYscHYf9/0EuAQ4AuuR1IsLLaBEuNACSoQL\n", "LaBEuNACikLgs4m0u1lhrfrVdtbrI7ke7ZE8gya2WpY4J8QZE2sxAs1r8v3U8duAXWrc92G/bQMc\n", "XKdewzAMI2beisylQrcIQ0aBDIfK0lYVXq8hOaEFz1oNGAq8mTo+Hc0Hn8UENIT9RwnoIGpTOkIL\n", "KBEdoQWUiI7QAkpER2gBBTKvAoxm3rK5jBmORnR/q1WF12tIprTqgQ1yBXAR8FCg5xuGYQwE5gGs\n", "yNwub0jGEcCQgCa1OgzYDB3Oegr4LT1zuNfiLTSHSTo175oQz0tLsScaqv47fr+CThBYChwH/Drj\n", "nino5ACA2cB/iN88nP87GPajz2XRE3I/OlYWPSH3tyFORlcGPSH3T2LwtA9zO4Au7h4KnwR+eCSw\n", "qj8/lYLYHHgZbZj/jia5mg28hBqWermfbGf7WTWem9y+Acz3n7NmHpizPcaFFlAiXGgBJcKFFlAi\n", "XGgBRSGwmYCszNvLvMM97a4opO28HXW4J53qY4AbUGd5vRyM9mCORg3QeUAn8fTf70EcWSyDydis\n", "LcMwjIYQWEdAvsr3O70h+VPvS5qn3qGtXYEd0UY/ohP4Jj4DV51ch3anTgXGA48D+wCv+PPj0Hzw\n", "tTBjYRiG0RjzANZg+nC/v0IIETPJXoG+mz9XFszIxLjQAkqECy2gRLjQAkqECy2gKASGC8jzTIqG\n", "tt4EqfS8pHnqXdl+IzoNdzd0Cu9QNIf7xejwlmEYhlFSKjpBafEk/je0QvdsYA1graJ1rIz6SLrR\n", "xYjL/Oc/EXC5fQbWIzEMw8hAYLGArMqMh3yv5Kiep4tjI2B/v/U3XEoemCExDMPIQGCK5tn9z4Pe\n", "kNze83TzNJrY6jl0KOsG/9koLy60gBLhQgsoES60gBLhQgsomKcBljBivN+f2qqCa83aOh84BV23\n", "cQHZFqvij5/YKkGGYRhGLjwKcASXLzmFc0Db75ZQy5BsDURTxbaitiExykdHaAEloiO0gBLREVpA\n", "iegILaBg/gnIary1rt8f2qqCaxkSV+WzYRiG0WZUYLbA1BVYsL4/tHyryq7XR/JtshewjPTnjPLh\n", "QgsoES60gBLhQgsoES60gAA8P5p30pGMbVWh9RqS04DRGcdH+XOGYRhG+Zm5Cc9En7dPLUrMnW50\n", "AUua9wMzihTSB+avMQzDqILAxd0go5g7y08B3jI+1Tx99UjmEgdJ/F9ify6aaOo24Hf9EWAYhmEU\n", "RmcF2Iz//s/vv68VhfYVtDEKNXwpGsI9GbRxCToP+b5WCDFajmPwzUqphsPqIsJhdRHhGHx10Qmw\n", "BU/OepgdANatfXl99GVIpvi/L6IGo2U5fg3DMIzCmQOwInOjpR0jWlFovc72NdFw72k+AhzYCiFG\n", "y+kILaBEdIQWUCI6QgsoER2hBQSgE2AlZkediFGtKLSRWVsLMo4vwGZtGYZhtAudAMuzaDm/32iY\n", "rEzqLWR9NCVumueBDVohxGg5LrSAEuFCCygRLrSAEuFCCwhAJ8BIFkaLEVuyur1eQzIL2CTj+EbU\n", "Tn1rGIZhlIdOgDF0RoZk5SIffhHwBD2NyabAk2hyq7Jg60gMwzCqILCpgFzO4a/5dSR3x6eap94e\n", "yddRb/9TwKt+e9IfO7k/AgzDMIzC6AQYzbyoR7K4FYXWa0jmoGl2P4iGlz8f2BvY1Z8zyocLLaBE\n", "uNACSoQLLaBEuNACAhD5SLJiJzZNX+tIkghwu98MwzCM9mM+0D2UrpZF/m2ECnA8Opy1EJjkj38d\n", "ODiEoCqYj8QwDKMGArNuZS/xPpLb4sPNU+/Q1heBU4FfpY5PA77QHwGGYRhGoXQOY1n0eXitC1vN\n", "M8C+/vNc4h7JlsDMIoX0gfVIYlxoASXChRZQIlxoASXChRYQAoHHH2CHqEfyUHy4eertkawLPJ5x\n", "fCma3MowDMNoDzpXjJf/jSnywU8BH/efkz2Sk4B/FSmkD6xHYhiGUQOBm19hrahH8np8OH+ORP0h\n", "h6Fe/08Rx986tAgBdWKGxDAMowYC13QyOjIkEh8uhmOAl9Fsid3oosSji3p4nZghiXGhBZQIF1pA\n", "iXChBZQIF1pACAQuFmipIWlkHcmv/LY66lt5sz8PNgzDMILQ2XNXhugKDyPCeiSGYRg1EPi2gAxl\n", "aZfvlSxPjj2Sx4E90Mi/WTO2kszz15wBvNIfQYZhGEauzAEYSld3F8OG0IK1JLUMyR/QvOzR51os\n", "B/w/4CrU+BjhcQzODHBZOKwuIhxWFxGOwVkXnQDDWNa9hOX0Yz+pVcBpVT5XYwM0hIphGIZRXiJD\n", "Eu0Xurq9AmwPHAKM9sdGp0SsVKSgDMxHYhiGUQNBA22twluLvY9kAgW1nWsC96PTfruIFyT+Ejiv\n", "ifI+D7yIBoB8GA1RXw0HXI+uY5kPPIqua8nCDIlhGEYNBHZS6/FqZEg2oKAQKT8BpgOroosQI36H\n", "5iVphEOAnwJnAtsA9wE3A+tUuf49qPE4ANgCzdZ4MfCJBp872HChBZQIF1pAiXChBZQIF1pAIOYA\n", "jGZeZDxGFfXgN9EAjdAzRMokehqWengA7ckkeRY4u4EyrgV+n3G8mlWd6c/ZZlt/tzIFKW0GF1pA\n", "iXChBYRAYIKAbM+DS9AeyS56uHnq9daPRAM0plkNWNTA80YA2wLfTx2/DdilgXLGoqvs62VlbMWN\n", "0Rr69R+uBHSEFlAiOkILCEQnwIrMHer3+90jqXdo6+/A5NSxYcDXgL818LzVgKH0XhU/HRhXZxn7\n", "Au9Dh7cMwzCMxpgPdI+hM2r/R9e6uB7q7ZGcDNwD7ICuGfkhOtQ1Fs3bXhS7omtVTkCd9FlMAab6\n", "z7OB/yTOOf+3YxDsR5/LoifkfnSsleW1Ul+R+9ugPsqy6Am5fxLaPpRFTyH7Fei4HRY8xo9Gw5+B\n", "mZ+lQMYDpwN/Af6KOsvHN1jGCHSI7IDU8Z8Bd/Vx726ok+jEGtdUG3Zo9+GIZnChBZQI18Ky2v23\n", "5EILKBEutIBQCLy0M/eJ95EcSwG/6xGog3yTFpV3P9nO9rNq3LMHOq53Uh9lmyEx8sZ+S0bbI/D4\n", "8VwQGZIT6Ofvuh4fyRJg/f4+KMGPUX/L0cBm6DqUccAv/PnvAXckrnfo9OCLgKv9tePQKMSDjd2B\n", "p2ucn4LGO2sHOihfGgLDGCx0jngnAlb/V7bX62z/DZqPpBVch/YsTgX+jc7W2oc42OM44unFAEcA\n", "y6N+mtfRhYnT0F7SYOPvwKY1zkdTVKF6t93RusCaHTRvDJJa+2IqOsGiWVxBz2kHXGgBJcKFFhCQ\n", "pCEZ0d/C6nW2r4BmRdwLTa073x+voI1BLb9FFhf5LYv0qvUjM44NRoZBHBynBkVOcy5qmEco5nsV\n", "9RzDCM2cVhqSenskmwOPoLOgJgFbpTajebZFe2adaG/tWnR4yqFZKP8P7YldQu/exLvRf5dO4Bq0\n", "5xbRkfGsUegw4QR0YWkn2gOsAF8Hngfe8hpW9vcsD1zpj88CHgTWQH1auwMX+rLO7+N77oUOy80G\n", "LvDPjBrtDYA7/TNm+OeN9eeuANYFbvTP+ao//ju0XmYDd6O/0Yh90ACinWgdbpc4ty86U2cWcC/x\n", "77facwYaHaEFlIiO0AIC0jk8XhrYb0My0GjY2R55m1qxNaF3BPASOp15KPAxYDE6O+696Ay376Fj\n", "mMvT05BE937R33sA6s86vY9nvpfeQ1tfREPVTPDP+gXwW3/uWOAG//wKarxW9OfuAo6q43uuhjbq\n", "H/daT/LfLbp3AzQNwXB/7d1oWJ6IF+k95DQZNYzD/bX/Tpx7nXha+livGf/3TXQaewX4tC87GiPO\n", "ek4ac7YbbY/AD8/h/6Km61wKcLYb+bEz2rBegAbD/BP6xh/RDXwHbXTTEQR2Roe7zvP3/gF4KHHe\n", "VXlm1tDNsajPapp/1neBA722JWiMtY3QH9u/0Tf2WuWl2Qd4Avij1/pT4I3E+RfQha1L0V7JT1CD\n", "V4sp6BBrpPddxAZuCRqXbQw6ZTzq3XwWnTH4kP8uv0EN9851fIeBggstoES40AIC0lIfyaA3JBWo\n", "tGpr4vETgNdSx14hLmsGcXKxeu59ieZ0TESN2Cy/PYX6Y9ZAh3xuRYfOXgPOpadvrZ43mQnoEFOS\n", "ZK9oTV/+q2jDfwVqvKoxBDgHHYqbg/YkBO3NgPbO9kGd5x3Ew17rAV8h/p6zgLW9PsMYTATxkRj5\n", "8DqwVurYusSNc61GOuve9RL3dFS5L6vMl4EPon6RaFvBP2MZOly2BTrDbl90SKgvfUmm0TO6cyW1\n", "fzbaU4miJRxOz99m+jmHAfujw2Fj0enpSZ/Lw8BH0Snif0b9P9H3PCv1PUejPqFGvk870xFaQIno\n", "CC0gINYjGUDchzagX0Df8j+Cjt9D3z2Lf6KN/InoGP/HE/fW4k30bX9M4tgv0MZ8Xb+/OtpQg3b/\n", "t0KHueaiQ0ldibI2qOOZf0EN0cfQ73kiPWOrjUaHqTpR43hyhuYNUtcvRiPxjqJn5OjhqKEZ63XO\n", "Tej9FfA5YEe0fkcBHyaONVTv9zGMdsec7TVox5Xt2xH7Ha5DfR2noj6CdIRjlzq2HT1nbV1N7Gx3\n", "NZ55CeqLmEk8a+tL6KyqTnTI6Ex/7aH++DzUr/FT4heQnYFnfDlR/KZq7O2vjWZtJR31m6O9iLn+\n", "+3w59T33R4ftZvlzo9CeRic6rHU4ccK14ejMtJnosNcDwPEpHQ/6sqahvZHIkKSfk0WZf0v14EIL\n", "KBEutIBQCOx1FZ+InO2/paDf9R5Vtt3Rt+BVihBRB+1oSNI8gC7C7C+uBWUMFFwLy2qn31IWLrSA\n", "EuFCCwiFwE6/44DIkPyegn7XUYrd7ipbNOOosExbVWhHQ7IH2isYhhqQ+ajz2SgnZf4tGUZdCGx6\n", "PftFhuQGCpr+uw/wX3TseSO/HYYu+joQHZ/fBp3RYzTGJsQL5L6E1mc6X0ujfAMdJkpvf+lnubXY\n", "vcozO3N8pmEYzRHER/Iw8P6M4+9HQ6aAzuaZWpSgKrRjjyQvXGgBJcK1sKx2/y250AJKhAstIBQC\n", "o+/gfVGP5G8U1CPZgt5rFkCdlVv4z09Qf5ZDwzAMIxzzh7O0G6BCd2E9kkfQ+EfLJY4tjy4ce8Tv\n", "747OoAmJ9UiMvLHfkjEguJf3dILIMJY8TD9/1/VG/z0OuAntgTyOThfdEnWy7+evmQT8vD9iDMMw\n", "jGIYStc8YMWhdI2sJ6x4qxiNxmT6id8+SwuSxrcY65HEuNACSoRrYVnt/ltyoQWUCBdaQEgeYrtn\n", "QWQk85+joB4J6IK0dIpcwzAMow0ZxrK5ABVkub6ubRXJB62NRlv9AboGokwM9B5Jq1LtHodOMe4k\n", "zjtSjQ6qZ0GciK4jGkyhdgbKb8kY5DzB5h0gsgLz3iTnWVuboJFgF6JrHbZAV15/GY1ZdCcaP8ko\n", "hkZS7VZjOPAjNODhGHT9Si0aSYlrGEabMJylswGESu5BG3+IOtj3R53sf0VDio/12y+Br/VXhFEX\n", "9Q5DRsEeXZXz49AZd//tr6A2woUWUCJcaAElwoUWEJIRLJkDIFSG93VtX/RlSHZGI7HeBHweDf39\n", "c+LQKBcCm/VXxCAnr1S7WWxMbEBmA3f4z7ugyZ5mowEN31Pl/qHoy8UMNBnVh/v8doZhlJLhLI0M\n", "ydC8n9WNJjeKmItO840Y568pC034SFqWabddUu2uR0+/xiro8NZh/tihaOTcyHeSjNL7OdQQreXP\n", "34VOATcfiWG0GW+z8ul+HclSAqfatf9U/SPPVLvVSOc5+TAa3v0q/7xrUIf+/vTmYHTq92uo8Tk7\n", "ozzDMNqAUcyfCdDF0EZm72ZSTwFXoG/JFfSt+GLU+S70PZTSBlRCNoR5ptp11JcBbgK98568RHb6\n", "2fH0HFpL31dWHIM7G14Sh9VFhGMQ18VyLJkzhC666f/IVl89kt+gzvaZwNvoW+ur/vNMf+7yfqsY\n", "vOSZardeXvP3pcvJiq32OnEWRVKfDcNoL5IRgI0E7baOZDj69p9MtRv5SBw93/4h20eSTLVbj49k\n", "Itk+kk94DYegLwlRsrK0j+RJYh/J37B1JIbRlgjstSJzIv9uUB+J0T+WogbgaGKH902oQaj2jxsd\n", "W+LvnYz2EA9G/ST1kCx3JpoC4Cto+t2v+v2ZGff9Cp3+/SiaWuAPVTQahlF+OkdUHTkf3LRbjyQL\n", "S7XbelwLy2qn31IWLrSAEuFCCwiJwKbjec16JAOEdKrdLYFbgioyDGMwYD2SKrRjj+QY4A10jc5/\n", "gA+1oMwQqXYHC2X+LRlG3QiM3pBnW9IjGWi0oyEx2gv7LRkDAoHK5jxhQ1tGTVxoASXChRZQIlxo\n", "ASXChRYQkgrIMJZ1taIsMySGYRiDlDF02kKSDGxoy8gb+y0ZA4ZPcuXb7Tq09XngRTTMysPAbn1c\n", "vxVwN7AAXVX/rSaeOYt4XYZttvVn6yt/i2G0DSsyd2FoDc1wCLqQ7mg0adb56GyidapcPwad0XQN\n", "sDka4bYTTayVhbRSbJvjQgsoES60gBLhQgsoES60gNAczwXP0Yazth6gd973Z9Eoslkch+bISKb6\n", "/SbaM8mirSojZ04KLaBEWF3EWF3EDPq6+Bw/f6wVhqTIoa0RaBKn21LHb0MTK2XxHjS97OLU9RPo\n", "HWjQ6MlKoQWUCKuLGKuLmEFfF10MTaenaIoiDclqaO6NN1PHp6Mru7MYl3H9m4lzhmEYRpMsZXjb\n", "GZJmsKGq5pkYWkCJmBhaQImYGFpAiZgYWkBoljBicd9X9U2RSZ1GAPPRVK7JKLU/Qx3pe2bcczmw\n", "KhqNNmIH1NeyPhpGPcnzwAYt0msYhjFYeAHYMLSIermfbGf7WVWu/xwwh57O9m/QO0+HYRiGMUg4\n", "GHWcHw1shuYb7ySe/vs94I7E9WPQrHxXA1ug+TfmAF8qSK9hGIZRQo5DFyQuAh6i54LEy4D/pa7f\n", "El2QuBBN/9rMgkTDMAzDMAzDMIz+EyK8SllppC4ccD0wDZ3w8ChwZM76iqLR30TERsR5WgYKzdTF\n", "ScDT6AjBNHR4eSDQaF3sg/pwO4EZwJ/R30i7swdwA9r+dVNf9tWB3G7mHl6lnWi0Lk4BTkcXeE5E\n", "JzEsBT6Rt9CcabQeIkYA/wJuQn8TA4Fm6uLHwDPAfujv4l3AB3NVWQyN1sWG/vpzgEloPdwCPJe7\n", "0vz5EHAm2v7NBz7dx/UDud0E8g+v0k40WhdZXAv8vmWKwtBsPfwEuAR9OxsoPZJG62ITtPHcJE9R\n", "gWi0Lg4EltFzOcSe6Bv8Ki1XF4659G1Immo3y74gMcLCq8Q0UxdZjAVmtkpUAJqthw/77QSKXUeV\n", "J83UxUfQiS37+L8vAlOA1fORWBjN1MW9wDw07fVQYEVgMvAg7f1/pBmaajfbxZBYeJWYZuoizb7A\n", "+4CLW6iraJqphwnodz4MHf8dKDRTF5PQhuFg9C31cGBT4Eba28A2Uxevowb1TNRXNBtdbrBfThrL\n", "TFPtZrsYkmaw8CrZ7Apchb6RPxxYS9FcAVyETjsf7AxBhy8OB/7ht8OBHYHtA+oKwSTUuX4Z+t0d\n", "Ogx0He1tVJuhqXazXQzJW0AXsGbq+Jro20QWb9Dbgq6ZONeuNFMXEbsBf0VnYaTHkNuNZuphT+A7\n", "6ESDpcCvgVH+82fykVkIzdTF66hf4PnEsed9Oeu2WmCBNFMXx6LRMr6Gzmj8O/Ap4L3oUM9goql2\n", "s10MyRJ0ls0HUsf3Au6rcs8/gd3p6TTaC13UmI7R1U40Uxeg0wD/ijak5+cjrVCaqYct0Rk50fZt\n", "dHrou2jviQfN1MU/gGHo23jEJHRYaLD9/6igjvUk0X67tJGtYqC2m+9g4VViGq0Lh079Oxd9uxjn\n", "t3Z3rDZaD2kmM3BmbTVaFxV0aLMD2AZ4N7p2oNbLSLvQaF3shvZivoWuHdkWnf47FRhZiOL8GIX+\n", "+26DtgHf8p8HY7v5DhZeJaaRurgM/Y/SndrS9dWONPqbSDKZgbOOBBqvi3GoH6ATdaheQfu/XEQ0\n", "WhcHooZ1LloXf0YnH7Q7jvj/e7INuNSfH2ztpmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEY\n", "hmEYhmEYhmEYA5wOwoR1mUz5V75PQSP21mIiught2wbKfZEBlMzIMIyBzerAz4lXIb+Bhmx4f+Ka\n", "ldBQD0UzmfIbkhXRsBYRHcAFqWuGAGugcbTqZVV6hgfpRsNlGEYvhoUWYAx6/gAsDxyFRp9dE426\n", "msxMNzuArnahHkPXjebjaIS3M44NtpDqhmG0ASuhjdz7+riug55v2WsCN6DJqV5EEzM9gUY2juhG\n", "M979Ds1+9wKa0CrJOcDTiXLOpWfU08n03VAfi6ZxXQjMQIP9Jd/8jwSe8uefAU6iZ4Ncj85vowEE\n", "F6EB9S5PnJtCPLQ1hd7x1Nal59DWEDRk+hdSz9jYX7ON359KPLQ1ld4x2tbzn7dLlXOMrwd7STUM\n", "oxCGoQEDz6NnA57mLnr6SG4B/g3shIaAv8OX8+3ENd1og/lJNDz62WhE2HUS15yK5ptYF/gQGib7\n", "9MT5ydQ2JNujuUw+4cvdGvgisSE5BpiGDgmth2amfB04vgGdB6DRVz8ErI023J9P3H8ZalRBh7ju\n", "RfOsrOG3IfT2kZyLhgtP8l3UGEckfSSr+fuP8mWu6o/fAvwsVc4/gR9hGIZRIB9Hh1EWoiHMf4Bm\n", "6UuSNCSboI1a8pq10SRNaUNyVmJ/KBpG+5M1tHwOeC6xP5nahuTj6LDb6CrnX6Z37+Ik4MkGdH4Z\n", "7TVVe8OfQk9ne9roQm9DsrXfT+YieQ74emI/7WzP8pEcgOY0j14CNvPXbV5FqzFAGWxJW4zy8Uc0\n", "l/p+wM3ALsD9wClVrt8UbaySaYJfRd/80zyW+NyFDrmskTh2IJrg6XXUYPyYnj2WvrgN7cW8CFyJ\n", "DrFFRmV11MBd7MuOtu/RswHvS+d1qA/pRbSncSAwogGNWTwGPE5s5Hbymq5qsJwb0ERSkYE5CngA\n", "HcozBhFmSIwysBgdnjoDzSl/CXAa/R9nX5raF+Lf/M5o8p6b0SGnbdChrkYa6XnoW/7BaO/jFLT3\n", "MD7xnGPpmZVxC7/Vq/NVtBd2LDp89yM0A+AKDejM4kpiQ3IYml72lQbLWAr8BjUgQ9Gc75f0U5fR\n", "hpghMcrIf1EjsnzGuafR3+32iWNro72aRtgVTdpzFtowv4AOATVKFzqc9A10yGgU8GE0OdI0YEPU\n", "OZ3eGmExmib5y8AOqCHapcq1S6jPAF/tte2EGsIr+7h+KdnTh38N7In6fUYD19TxbGOAYTMrjJCs\n", "is5WugQdapmLGoj/Q3so8/x1FeKZTs8AtwK/QLPgLUb9KgvRN/laJGdLPQOshfoi7gf2Bg5tUP+H\n", "0cb4HtRXsCe6ruO//vx30Nlms9Gez3C0BzMBnTFWj87JaAP+IFofh6DG4rnetwE6w2pH1Lk/n+xp\n", "vKA9nbuBX6JO+t/V0BOV+36057IYmOWPP4sOD34fNU7zsm42DMPIixFoj+BBtCGejzZMP0SnBkek\n", "HcjR9N+FaAN3BLoG5eTENVnO4bQD+Wx0fcVc4Peos70rcX4ytVPx7grcCbyFTiF+zGtJcija41no\n", "v+M9aA+gXp0fQSchzEIb6QeAfRLXJmdtgeYcvw+tyy7i6b9d9F7ZfqR//u8zvlu6rvZF/22W0LtH\n", "9Wlfzm4YhmG0Kauhb8kfCy1kkPI1dMjRMAyjbdgTfVOfhDrN/4YO1YysdZPRckah/prX0GFGwzCM\n", "tuED6DDSPHRo6np6T6k18mcKutr+WmzijmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEY\n", "Rjvw/wG/hTQCCPPNMwAAAABJRU5ErkJggg==\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "report.roc().plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metric" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": [ "iVBORw0KGgoAAAANSUhEUgAAAaAAAAEZCAYAAADR8/HkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n", "AAALEgAACxIB0t1+/AAAIABJREFUeJztnXe8HUX5/98nFUhoIZCChAtICyIdpEhGEKmiEDrSDNIU\n", "KQIKX4V8kRZFpQSliEQRpIqgKE24WCKQSBGl5AckIC0kkEZ6md8fz5zv2bt3zr17ztnd2XPu8369\n", "9nXv7s7OfHbu3Hl2Zp6ZAUVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRlM6c\n", "BkwFFgKTgd26CDsSeAJ434V/HbgU6BsLNwr4ZyTMyelKVhRFUZqdw4ElwBhgU+AaYB6wXpXwGwHH\n", "Alu6MF9EjNEPI2E2AOYDV7s4T3RpHJy+fEVRFKVZeRq4IXZtCnBZDXH8GJgYOR8HvBoLc1MsjKIo\n", "ilIweuWYVj9gW+CR2PVHgF0SxvFJYO9YHDtXiXN7oHftMhVFUZQ8yNMADUYMwvTY9Q+Aod08OxEZ\n", "35mCtKLGRu4N8cQ5Hejj0lQURVEKSJ4GqBEOA7YBjgL2An4QVo6iKIrSKH1yTGsmsBxpsUQZArzX\n", "zbNvu5+vIK2oXwDnu/jep3MLagiwzKUZ5R1geE2qFUVRlNeRIZBUydMALUFcpb8A3Bu5vhdwdw3x\n", "9EZabr0QA/QP4KBYmL2ASe5+lOFAqYa0MsLOAVarnI+lY69ifgzj3SWHc2e/VVjAasxlVeYxkI95\n", "kEkcTxt9WUo/ljCIj1iHDxjER/SJZesi+vMiW1LC0pvl9GY5fVjG62zERHZhBb2Yw+ospS8r6IWl\n", "xAp6sZS+LGIl5rA60xnCfAawhH4soR9zWJ1lnbztx1JPPpVYYfuxZMFgZrbPYO13ltD/feBF4F4o\n", "2fpyrkFR2TKW4mmCYuoai2pKQqP/J17yNEAgHmy3As8g4zqnIK2X6939y4EdgM+782OQsZ9/IwZs\n", "e8Rj7k5gqQtzPfAN4CfAjcCuwHHAEdm+SkP8DRiA/FEtPLsZ8FLlnBWR331HV/dreXb+bziyNIq/\n", "jAQeBh4H5gClc+EHd/LKxVQMts9wW2DVlVg8bAcmr4OUp/64rt2RvMwX+QOe53sDbS5s1FFkBvAu\n", "0Gcuq/Z9mL3XeY9hq3zIWistYqVed3LPjiPZ8b+rMXfRCnr1tpRKy+ndexl9BvVl6Tq9WLHG+wzt\n", "9R7DWEpfltKXjxnIPFYrLWalAe/wif2j4m/kazd8TcYVF0eOBYhbf/Qf7tmSaPPRVuV6SNpCC6hC\n", "W2gBHtpCC/DQFlpAXuRtgO4C1gK+CwxDvkL3A/7r7g8FNoyEX4p0tW2MVGBvAuMRY1NmmovjJ8Cp\n", "SDfb6cB9Gb1DCpT2j12YAKXjAwjBVL81pwTP5SakE/OAe+IXJ7zJ/sdXe8KKQdsYKUODkLljK/+V\n", "3QY9w46f6s/iDSylYTczpvQCW7Mq85JOWF5o4VHgAaRbd0X5WAsGflTTeymKUqYA3VG5YinmOxug\n", "PbCGOIYW1GRh3V35210T2XWXM7jq4as462XEcJWPVZDWaZl1gS2qxdcuoibSeS7abKTL+fZSRt0X\n", "XWAo3t8OiqnLoJqSkEndWcTKOEuKaoCUXLFXAN9GDMcxyFSARVBa1imklJdyt+467nIJ6Wbsh8xh\n", "GxB/LsKhJU9TTlGaDK07UyDvL9GkmNACPJjQAjyYdKKxZ4G1sWMJ2HNrjWl92NfCaAtfjRwnRiK+\n", "JB3NNWECpJkEE1qABxNagAcTWoCHlnBCUJQi8AvEC/HzyPqBayML3J5Ox3UGu+VNWFiCP8WvWxkj\n", "uhlYv2G1iqK0BEVtASlBscMjLaHRqcQIe7oIn0wjPkUJTCZ1Z0/r09N+TMWDHQB8HLlwPjLpeSbw\n", "IpTm1ByjeONNobLtyFwqHnTVWIy4wf+yBC/XmqaiZIjWnSlQ1BaQCS3AgwktwIPJLmq7g2dcqHxM\n", "AdtWiyYL/SzMqBZhN8djDb6MV1MBMKEFeDChBXgwoQV40DEgRcmO0iSwmwG7IxshboLMLwNpzUwF\n", "+7/InLW/QmlKl7HBEgufQsaYSsCayNykLh5hVWQ+255WJlzPQCbFdvLOi2Dd/WdKnV3BFUUpEEVt\n", "ASmFxPYF+90qDZVfg029S8LCP+tsNS238DsrW54oStroGFAKaD+mUgd2ALLKxnrIvJ/t3Y3NofRK\n", "qinBykjLaztkftEAul61vhfi0VdeD/HeEhySpiZFQevOVChqC8iEFuDBhBbgwYQWALYX2P+4hsfr\n", "8NlCVPYWTnYtoXYKkU9eTGgBHkxoAR5MaAEeMqk7m2U/IEUpCKUVyJqGABvC/94B9jmw54dUhaz+\n", "DrB6UBWKolSlqC0gpamwA9xqCk/EVlJYV8aNAiiCDZ2QqSHSV1oerTtTQDNRSRm7tscn4Mq8DZGF\n", "QS7xWXmmq/QYtO5MgaJmogktwIMJLcCDCS3AgwF7HthXYkboFbBb5iXCQh+X8Iq+8Lm80q0RE1qA\n", "BxNagAcTWoAHHQNSlGJS+gGUNgMGI1sXg8z/+RfYp8FeAvYksJmNz5RkLtDHQOlaONbCjlmlpShK\n", "fRS1BaS0DLYE9ktgF3i65sZlmjI8FUlsse24uaOiNILWnSmgmajkhC2BHQb2aLDvOLtwc6YpwuoW\n", "vmHhQ2eE7ur+KUVJhNadKVDUTDShBXgwoQV4MKEFeDDdB7EnOAP0ROZqgFFwqIUFzgjtlkeaCTGh\n", "BXgwoQV4MKEFeNAxIEVpUha5nyaL5XviPClryJX3NfprgmV8JlronbUuRenpFLUFpLQ0do1IfT8Z\n", "7BVgtwU7UFZWyCBFGGjhhRrWktsmCx1Ky6B1ZwpoJiqBsL/qov4/KXc1UHLHrU7E6XlrUJoKrTtT\n", "oKiZaEIL8GBCC/BgQgvwYJIHtZuD3RvsBLDTYkboHrDfSKlFlFiThROdgHtSSLc7TA5p1IoJLcCD\n", "CS3AQyZ1p+4HpCi5UXoZ2en0YTm3awIfuZuj3bEDcFyOov7ifu5uoVQq7keaojQ9+s+lFAw7COwe\n", "YG9yLaFc13Jz3XDvuVbQZnmmrTQVWnemgGaiUlBsb7BznBHaOdeU4U5ngHIfi1KaBnXDbmFMaAEe\n", "TGgBHkxoAR5MOtGUlgOPuJOJzhiNBzsiB03/1w1XR1q1YDKOvx5MaAEeTGgBeRHCAJ2GLBm/EJhM\n", "1xPlDHA/8C4wH3gBOMETZoXn2CRFzYqSB2cDb7rfVwO+Lud2olvwtF9G6T7pfh5t4RfaElJalcOB\n", "JcAYZLHGa4B5yFbHPs4HLgZ2BtqAU4ClwJGRMAYxOJsB60QOn3HVLjilCbCbgb0o0iVXPv4f2HXS\n", "njtkoZeFaTHf8B9YOMTC/hb2tLCrhTXTTFdpKlqi7nwauCF2bQpwWQ1x3ElHl1GDGKC1EjzbEpmo\n", "9BRsL7CjwLbHDNE8sNuBbQO7SiopwQYWjrfwry4mq76rKyb0WJq+7uyHtF5Gx66PR/axT8pDwI2R\n", "c4MYoKlIV91jVO9DLWommtACPJjQAjyY0AI8mOyTsCWwF4J91WMXloD9YlqaLKxsYZyF2y3ca+FB\n", "C3+2sMwluFoDL1K3rgwxoQV4MKEFeGj6eUCDka+n6bHrHwBDE8ZxALAHsEvk2rtI19wkoD9wDPBn\n", "YBTwtwb0KkpBKFmkK/pisF8Gvg2sCmwB9EXK/O9TSUnGZr8dv25hJtLL0D+NdBQlb4YjLZW408GF\n", "wCsJnt8VmAOcnCDsg4jzQpyitoAUpQ7sjpGW0CGZpiTdb9bCulmmoxSWpm8BzQSWA0Ni14cA73Xz\n", "7G6IUfkenceQfDyDODz4mABMc7/PBp6n0gVo3E891/MmOF95IPypfPluuO9ZmPUanHAylGannN7i\n", "duAK+CxwRzbvo+cFOjfA8e58Gi3CU/idEC7t4pndgbnAmTWkcx8yFhSnqC0gE1qABxNagAcTWoAH\n", "EzZ5+0mwr3UcFxqf5COttlTg1RRWSzBp6UkRE1qABxNagIembwEB/Bi4FWmhTETGboYC17v7lyNr\n", "YX3enRuk5TMe+A2VsaLlyJ4nIIZpKvAS4ujwFeBLwMHZvYaiFIXSa9IVx37AjsDpsOG+MmeotCTF\n", "hMpxZTUXSVFy4VTEYCxCHAeiY0K3AG/EzpfTeZJpNMy5SCtqAfAhMqlunyppF7UFpCgpYIdFWkKz\n", "pXWUUsww2UW8fVpxKk2F1p0poJmotDj2rJib9h1uodMLwdbtwWZl11RrxRlI6Xlo3ZkCRc1EE1qA\n", "BxNagAcTWoAHE1pAZ04/07WA4nOGPgC7Rj0xWmh3kXyuAWGmgWezwoQW4MGEFuBBFyNVFCUJ1z4P\n", "DEKMxTHAte7G2njm+CRksfupY0CKUidFbQEpSsbYsZGW0Kian4bfu4cPzECcUny07kwBzUSlh2JX\n", "ATsjYoR+K/sO2QGJnpZleayFTCe8KoVFu+BaGBNagAcTWoAHE1qABxNagAfT+VJpAdIlV570fRAy\n", "FeJjsP8Gu1U3cabRBefRFRwTWoAHE1pAXqgBUpQeQ+nfwAjEEN1KZQmsLYBvgO1qRfnyPCBdC05R\n", "6kS74BSlA/aEmKfcy2BHdgoFN7oASdZiVFoP7YJTFCV1focsKLfAnW8GXOnZ9E694JTUUQNUDExo\n", "AR5MaAEeTGgBHkxoAR5M8qClWVDaD0oDIs/tC9wUC5hGF1wNunLDhBbgwYQWkBdqgBRFcZSeBM5w\n", "J18Few7Yvu5c14JTlAbRMSBF6Rb7Ymxc6LeGxyeP41w7jwHfD61OCYLWnSmgmago3WLbwN7jWcrH\n", "XshY30aPSuujdWcKFDUTTWgBHkxoAR5MaAEeTGgBHkw60dhNwO4J9pBP8/wLYG0fliwDewPYVcPp\n", "ShUTWoAHE1qAB/WCUxQlT0pToPRnKN3zfb73wBrMYhl9ewMnAf9wy/v0DixSUZqGoraAFKXQWDh5\n", "Pivbo/j1c7FuudvBHhdxVlBaE607U0AzUVHqwMIJzuJMALsp2JvBLo0Yor8kXVdOaUq07kyBomai\n", "CS3AgwktwIMJLcCDCS3Ag0k7QgtHOUtze+TqTmDvirWILs1TVwqY0AI8mNACPOgYkKIowfBMRC09\n", "DaXDgB9RWeT0PFl5W1GUOEVtASlKobFwoGvi/L6LUP/PtYK2y0+ZkhPaAlIUJRhJ1oJ7zv2cDHaD\n", "jPUoLYAaoGJgQgvwYEIL8GBCC/BgQgvwYDKIM8lacH+P/P6Gm8y6I9hShroaxYQW4MGEFpAXaoAU\n", "RUlCkrXgxgOXAA+489HA00iL6Aa4bO+IMVKUHoeOASlKHVjY3o0BTU74hAF7C9j3Y15yX8xUqJIV\n", "WnemgGaiotSBhU87C/JijU8OBHsY2FcjRuhjsG+IkVKaBK07U6ComWhCC/BgQgvwYEIL8GBCC/Bg\n", "0o7QwmbOerxaZwxD4MHXwK6IGKL/gO2TrtKaMYHT92FCC/CgXnCKogSjwR1RS9Nh/xOBPsAgYCow\n", "EjgkDXGK0gwUtQWkKIXGwrqu2fJuSjGe51pB49OJT8mYlmkBnYZ8/SxEBjR36yKsAe5HCv184AXg\n", "BE+4UcA/XZyvAyenJ1dRFNLfEfV593P7lOJTlG45HCnIY4BNgWuAecB6VcKfD1wM7Ay0AacAS4Ej\n", "I2E2QIzT1S7OE10aB3viK2oLyIQW4MGEFuDBhBbgwYQW4MGkHaGF1VwLaG4D0ZhIjENiY0HDGtXY\n", "uKbCYEIL8FDUurMmngZuiF2bAlxWQxx3AvdEzsfReWD0JmCi59miZqIJLcCDCS3AgwktwIMJLcCD\n", "STtCCys5a7G4+9BVMbFY4wuZptW6akBTITChBXgoat2ZmH5I62V07Pp4oL2GeB4Cboyc/wW4Nhbm\n", "UKQVFN8sq+kzUVFCYKFXxFKkOJnUHh0xQPumF6+SMk0/BjQYMQjTY9c/AIYmjOMAYA86GqAhnjin\n", "I942g2uXqShKnBKsAJa50xRdp0u3ARPcyX1gHwQ7TuYPKa1OaB/8WtgVuA04ncSzsb1MAKa532cj\n", "g6Ht7ty4n3mfl6+FSt93HtcWWg/AmRTj7xU93xq4qkB6iFxLNf7HYVkv6GNkPbildcRX7e/3PWBz\n", "aN8J2A/MfsBX4JAL4J4309Jf5bzH/P1qPDfA8e58Gi1AtS6464Anunl2N2AO8E3PvSeRbrwozdYF\n", "Z0IL8GBCC/BgQgvwYEIL8GCyiNTCLNdXNqjOKEwXsZfAbgz2ukiX3G11ppOSpmCY0AI8FLXurImn\n", "8DshdLWL4u6I582ZVe5fQWcnhBvpuDJvmZbIREUJgYX3nWVI2mVeb0pfdwboA7ATwF4J9gSwW2ab\n", "rtIFLVF3HoZ40YwBNkdcp+dSccO+HHgsEt4gLtbjkLGeoe5YOxKmDfgY+ImL80SXxkGe9FsiExUl\n", "BBbecgZo/YxTGgx2ccxDrnw8DvbbYOO9G0q2tEzdeSoyEXURMImOE1FvAd6InS9HBkCjRzQMSCvp\n", "ny7O14GTqqRd1Ew0oQV4MKEFeDChBXgwoQV4MFlEauE1ZwU2rjMKU0NqG4M9GOwYWS3BvhIzRDvX\n", "qaEBTblhQgvwUNS6s6koaiaa0AI8mNACPJjQAjyY0AI8mCwitfAfV/t/qs4oTIMKPgl2jjNAezYW\n", "1/9hUoonTUxoAR6KWnc2FZqJilInFp5zBmjbgCoedQZor3AaeiRNPw9IUZTmJu314OphhfupY0At\n", "gBqgYmBCC/BgQgvwYEIL8GBCC/BgMoq3vAxP/zqfNyloWO5+plV3mZTiSRMTWkBeqAFSFCUpRWoB\n", "ad2lNB06BqQodWLhj24MaP+AKh5wY0BfCqehR6JjQIqiBKXRLrg0SLsLTgmI/hGLgQktwIMJLcCD\n", "CS3AgwktwIPJKN5Gu+BMChrSdkIwKcWTJia0gLxQA6QoSlJ0DEhRGkDHgBSlTiz83I0BfS2gijvc\n", "GNAfwmnokegYkKIoQSlCC+gl93N3sGsE1KGkgBqgYmBCC/BgQgvwYEIL8GBCC/BgMoq3AGNApYuR\n", "PYVWBS4BOxrs9mAHhNOUOia0gLxQA6QoSlKK4AUH8Gv38+vAPciixjPBjgonSVG6R8eAFKVOLHzf\n", "jQFdGFjJQLDfAXsD2PvBvufGhc4Nq6ulyaTubKYtuRVFCUsRxoCA0sfIRpQO+0PgHKAUSJBSJ9oF\n", "VwxMaAEeTGgBHkxoAR5MaAEeTEbxFmEtOB/vuJ/jwN4D9hM1PGsy0NMoJrSAvFADpChKUgrSAurE\n", "3cB09/to4IiAWhSlKjoGpCh1YuE0Nwb0s9BaOmN7S1ectWAvCa2mBdF5QIqiBKUoXnAeSsuBd93J\n", "wJBKlOSoASoGJrQADya0AA8mtAAPJrQADyajeAswD6hL5rmfq9bwjMlAR6OY0ALyojsDNILO+7/v\n", "ATwBPAOcn4UoRVEKSVHHgMqUDdA+YAvYSlNq5R46uDsyAvgY+DfwALAU+GYAXfWiY0CKUicWvuzG\n", "gO4PrcWP3ceNAelacekTpO58E9g9cn4+8BrQ152fAzybt6gGUAOkKHViYT9Xu/8ptBY/dmWwt0aM\n", "0H/ATgJ7Nthvgj0N7OdDq2xSgtSdC5FWT5mHgSsj55sCs3NV1BhFNUAmtAAPJrQADya0AA8mtAAP\n", "JotILXze1ex/rjMKk6ae6tirIkbId3wlf001YUIL8BBkJYTZwFrAW8gs4x2Am2KidDUFRekZFNgL\n", "rgNnAT8GDgbWRTav6wPsCOwE3CCLmHKlLp5QbO4HbkX+eEcig5CDIvf3B14OoKteitoCUpTCY2En\n", "14R4JrSW+rADwD4baQk9HVpRExGk7twKmAksQ3YijE/wupVCTkqrihogRakTC9u4mvv50Frqx/YH\n", "e0DECPXt/hmFgHXn2sCXgc947h0AbFBjfKcBU5HxpcnAbl2E7Q9MAF5AWl9PeMIYxDjGj008YYtq\n", "gExoAR5MaAEeTGgBHkxoAR5MFpFaGOlq7Xp7PUyaehrDLpFXWaOITgkmtAAPwVZCmAH8DnjKc+8P\n", "iDFJyuHAVUhLamtgIuJRs16V8L0RQ3Ut8CBdZ8JIYGjkeK0GXYqidE/R5wHVwUnbgW2h92kuuhuB\n", "G00yy/fbhOk9jTTfT45cm4LMN7qgm2fHA1sAn4tdN8DjSEvtw27isOioo6LUhRWP2DeBt0vVPxqb\n", "BDsTcbAC+ch9BXgUeEyOUlF7S0IRpO70dW35jiT0Qyaujo5dHw+0J3h+PF13wU1F1oJ6jOpNWC1U\n", "ilInFoa4LrgPQmtpHLsj2B+B/ZfHTftbodUVkCB15zvANGAssH6DcQ1HDEV8zOdC5OujO6oZoE2A\n", "k4BtkHGq64DlnnSguAbIhBbgwYQW4MGEFuDBhBbgwWQRqYU1XQ1d79w/k6aelDBg1wJ7UGQS64dg\n", "a1lPLgNNhSPIPKARwL7AGMRI/BW4GbiPSn9waKa4o8xTQBtwLvA3T/gJiFEF+Ud6nkoLzLifeZ/T\n", "zX09F7YumB7jNBVJT5RU4x8BO/1KLvZLEt5zXtC/X6kduA96zYL7toYDP4U4Sz2d4Pkszunmfh7n\n", "BjjenU+jAKyDLL3zEuKafQ21DUZW64K7Dn/LJk61FpCPixCdcYraAlKUwmOhj2sBLQ+tJTvsvq4V\n", "9GpoJQWjMHXnJoghWEHHSalJeAq4IXZtCnBpgmdrMUD3IWNBcQqTiYrSbFgZmV/hjFDv0HqywfYB\n", "O8sZoQ1DqykQQevOAcBXkS6t+cgEVFNHPIchy3mMATYHrgbmUvGouZzOhmMk0nS/A5iETI7dOnL/\n", "TOBLwMaIl9zliHH8sif9ohogE1qABxNagAcTWoAHE1qAB5NVxBYWOQO0ch2Pm7T1pIDpfMne5QzQ\n", "13NXI5hA6XZFkDGg3RDDcygy+ewXwH6I0aiHuxDXx+8Cw4AXXXz/dfeHAvGvjgepOEBY4Dn3s/wF\n", "1hf4IfAJxJ3y3y7Oh+rUqChKdRYjE8T7I/9vrcifkDpvX2SIQMmI7vy6VyALkf4S+BfVrWDSeUCh\n", "0XlAitIAViamDwaGlFrCHduHHYZM6VgIDILSosCCikAmdWcSA5SEJCsqFAE1QIrSAFamZgwHZiGb\n", "U85AnJJmIJX2dSWZrNrk2OeR7v69ofRIaDUFoLB1516hBdSAjgElx4QW4MGEFuDBhBbgwWQVsYU7\n", "u9pox8JCC7+0HcdpM9fVAMZ/2V7uXunHuaoRTIA0uyPYWnA+1gW+B7yBjrUoSk/iCGRKxjrIeO2O\n", "yJjrccjakCsBxwJPWdmwslkp7/p6OthGJ+ErKdAHmcPzR2R7hmeB8+jsNFBkitoCUpSWwMJGFh5w\n", "raHHbUG7bbrH9gU7J9K4Ow/semCbZbghbYLVnZshXmbTgfeBcYgB2iKUoAZQA6QoGWNhLQszXM19\n", "bGg99WOP9/QyzgL7CNjrwX4X7LFgVw+tNAeC1J1/Q4zNI8AhVNy2lyLzc5qNohogE1qABxNagAcT\n", "WoAHE1qABxNagIVjXY0904rXHBRAlwfT9W3bF+y5bifV6VWGvn6er6YgBKk7VyDrIe0Tu64GKF1M\n", "aAEeTGgBHkxoAR5MaAEeTGgBbtWEx10NPcnCbXfAoxZuc8dXQmt0mNqC2/XAHiiTVO11zgClvfdY\n", "jZpyIUjduS3wU8TlchqycvV6qAFSFKUbLGwSWTkhftQ7mb1A2D5g57tXWju0mowJWneuDByDrJK6\n", "FGkZnQesGVBTPagBUpQcsbCVhaMjR7lrrkUWNLVPOgN0fGglGVOYuvOTwBXAe8iWDM3khl2YTIxh\n", "QgvwYEIL8GBCC/BgQgvwYEILqIIBsLC8QAuamsYetyc6A/RMKmoEk2JcaRFkLTgfrwHfQdZz2x9Z\n", "K05RFCUpS5D5QhdZWVtuOdKrshyp6F4tyRqQzcBtwFXADmDboDQtrBylyBS1BaQoPQYL73SzmoK1\n", "su1Lk2DvdLLPCa0kQzKpO5t0kljdFHY9I0XpKVjYFVnCqzeyGkvvyHGWC3ZlSXY1bgLsIcDdyO7K\n", "u0JpQWBBWaB1ZwoUtQVkQgvwYEIL8GBCC/BgQgvwYEILqILpLoCF7SJryg0vgqbusauA/ajikm0/\n", "H15T6hRqLThFUZTUKcE/gXuRMaLvBpaTkNIC4AvIXmQbAY+CvVomsSpKhaK2gBRFcVgY6Tzlltqm\n", "WmvS9gV7AdglrjX0FNh9W2T9OK07U0AzUVGaAAsTXFfcL0NrqR37GbBvR3wqrgqtKAW07kyBomai\n", "CS3AgwktwIMJLcCDCS3AgwktoAomaUALG1hYYmGFzXbVlcSaasOuCfZ8Z4Dm1tgdl5GmhtAxIEVR\n", "egYlmArcJL9yu4WvWBhsYRULfYq/zUNpFpQuB14GVgU+E1iQUgCK2gJSFCWGhWEW3u9irtBiC2/Z\n", "Qm8NY69ycr8fWkmDaN2ZApqJitJEWBhk4RQLf7cwx8ICC8tihuiU0DqrY/fNYKmeEGjdmQJFzUQT\n", "WoAHE1qABxNagAcTWoAHE1pAFUxaEVnoZeGmFAxQapr82AFgF4NdAXathA+ZLBXViY4BKYqiAJQq\n", "a8cVnNJ8ZGPPErBnYDFKYIraAlIUpUYsXF/8LjgA+23XDXdzaCUNoC0gRVGUJuRh9/MLYAvuvZcv\n", "aoCKgQktwIMJLcCDCS3AgwktwIMJLaAKJrQADyaHNP4FfAB8AtgsQXiTqZoCEcIAnYb4+C8EJgO7\n", "dRG2PzABeAHZQ+SJKuFGIWtILQReB05OSauiKEqDlFYAj7qTvUMq6ekcjhiSMcCmwDXAPGC9KuFX\n", "AX4GnAjcBzzuCbMBMB+42sV5okvjYE9YHQNSlBahecaAAOwxbhzoj6GV1ElL1J1PAzfErk0BLkvw\n", "7Hj8LaBxwKuxazcBEz1hWyITFUVpOgM0zBmgBWD7h1ZTB03vhNAP2BZ4JHb9EWCXBuLduUqc21OM\n", "PeeTYEIL8GBCC/BgQgvwYEIL8GBCC6iCCS3Ag8knmdJ7yFjQynQ97ADFzKdMyNMADUYMwvTY9Q+A\n", "oQ3EO8QT53Sgj0tTURSlCJQ/lL8QVEWB6BNaQAAmANPc77ORbXTb3blxP/VcjiLpIXKtKHrK53Rz\n", "X88rmLRqAmC7AAAZXklEQVTi+zUM/wQd/gj1xtfo8wnPz34fDgTMF4BvZ59eQ+cGON6dT6MF6Acs\n", "BUbHrl9Hde+2KNXGgJ5096IcijgixLvgdAxIUVqE5hoDArArgV3oxoKGhFZTI00/BrQEcZWONz/3\n", "wu8wkJR/uDjicU6iKZbqAIrZ52tCC/BgQgvwYEIL8GBCC6iCCS3Ag8kvqdIi5IMZOtdZUUz2WopB\n", "3vOAfow068YAmyOu00OB6939y4HHYs+MBLZGxnMGAlu58zLXA+sCP3FxnggcB1yZxQsoiqI0QGRV\n", "BCUEpyITURchrZSoR8gtwBux8FORhQfLiw/6FiHcHWldLUImop5UJW3tglOUFqH5uuAA7BauC+59\n", "sM20Eo3WnSmgmagoLUKTGqAS2HecEdoutJoaaPoxIKU6JrQADya0AA8mtAAPJrQADya0gCqY0AI8\n", "mHyTK1ngXndyKdhNPYFMfnrCogZIURQlX34ELEbWhXsF7ItgL3Ldc7padgujXXCK0iI0ZxdcGbsD\n", "2FvAzuq4u7h9Gez3wW5VMGOkdWcKaCYqSovQ3AaojO0Hdm+wN4GdGTNG3wmtLoKOAbUwJrQADya0\n", "AA8mtAAPJrQADya0gCqY0AI8mLDJl5ZA6WEofQ2ZkvJ5uPfv7mYzOSnUhRogRVGUQlBaBqU/w4uP\n", "dh9WaUa0C05RWoTW6ILzYQ91XXB3h1YSQbvgFEVRlNZBDVAxMKEFeDChBXgwoQV4MKEFeDChBVTB\n", "hBbgwYQW0JmxI0MryAs1QIqiKIqSAzoGpCgtgo4B5YqOASmKoiitgxqgYmBCC/BgQgvwYEIL8GBC\n", "C/BgQguoggktwIMJLaAzOgakKIqiKEqK6BiQorQIOgaUKzoGpCiKorQOaoCKgQktwIMJLcCDCS3A\n", "gwktwIMJLaAKJrQADya0gM7oGJCiKIqiKCmiY0CK0iLoGFCu6BiQoiiK0jqoASoGJrQADya0AA8m\n", "tAAPJrQADya0gCqY0AI8mNACOtNzxoD6hBZQED4C1gwtQmkZZgGDQotQlKKjBkhYEyjS/utKcxN6\n", "rLE9cPrVaA8twEN7aAGdGftSaAV5oV1wiqIoShDUABUDE1qABxNagAcTWoAHE1qABxNaQBVMaAEe\n", "TGgBnek5Y0BqgBRFUZQew2nAVGAhMBnYrZvwWwJPAguAt4Hvxe4bYIXn2MQTV7W++dB99kproeUp\n", "B3QeUK60xDygw4GrgEuArYGJwJ+A9aqEXw14FHgP2B44AzgXONsTdiQwNHK8lqbwJuKzwCtd3J8A\n", "fD8fKQ3TDowJLUJRlGzI2wCdDdwC3Ay8CnwTMS6nVgl/NLAScBzwEnAvMA6/AZoBfBA5VqQpPGNM\n", "inH9Fdisi/uW7r9mDJKHadBO/UYkrtV0EXYasEed6dRCPB2TQ5q1YkILqIIJLcCDCS2gMzoGlAX9\n", "gG2BR2LXHwF2qfLMzkiFujgWfjiwfizsZOBd4DEKWahyIalbfZ4u53l1R1nyea+80lGUlidPAzQY\n", "6A1Mj13/AOky8zHUE3565B6I0TkFONgdrwJ/pvuxpSLR3s39bYHngLnAXcCdSDeaQcbFzkNakje7\n", "a/+NPLsN8Kx79g6kRdkVA5Bu0bWAee65oUil+x2ka3Om01CevLsS8Gt3fRbwDLAOcCnSJTjexXVN\n", "N2nvhXQfzgaudWmWK/uNgAtdGjNcequ7e7cCI4Dfu3TOcdfvRvJlNjKOGP2y3A/4j3u/t4FvRe4d\n", "ADzv3uXvyDhktXTau3mnELSHFlCF9tACPLSHFtCZnjMPKE+GI91iccNwIdXHLB4Gfh67NsLFs1MX\n", "aT0I3O+5XpcTghvoTOXoKp0q9APeBE5HDPhBSIvwYmAUsBS4HOiLGAJDxQCVnz3DPTsaWOKe7YpR\n", "dDRiuDgmIn/HvsD1wO3u3snAAy79EmL0VnX3ngC+muA9ByPG4GCn9Uz3buVnNwL2dGkPRgzKTyLP\n", "T6VzF9zxiEHt68I+F7n3HrCr+311pxn3czqwg3uXY13cfbtIJ446IeSAOiHkSiZlOs+VEGYCy4Eh\n", "setDkMrAx/t0bh0NidyrxjOIw4OPCUg/PsiX8fORe8b9bI+dp4mpEn97lfQ/jVTI17rzcgsDxJHD\n", "AhchlfWu7lqZU4BVgKvd+YdICzGqJZ5emf6x+ycDNyHehe8C/wu8hXwgLEFaTEciFXQ0vjWotGKq\n", "pdeOtEjeQpZFWo44q1wAbOrCvA580d1rRwzKD5APjXJ8WyEfJ+XzaYghaXd6zwD2RVp4SxBjvjrw\n", "R8Q4GeAs4AZgkjt/CzH4n0H+DtEWpEHy+6rY+xA7971vlufla6HSr3Z+JvL/lkp8v4bhn6DDS9cT\n", "n+/vl4q++s/HjoSxwN1r468v8jg3yAccVOrLpucp5J87yhSkq8bHKcAcpDIscwGdv87j3IeMBcUp\n", "qhu26eLeEcDTsWu3I11wo5Duo3hc5fw5goqxij/bnZ64E8IC5G8xK3IsAIYhHzIXIl1a7yCOIuWP\n", "m6QtoO8g3YtRJkaeHQI8jrzvHKQL7M1I2HjLpBdwBdJlWNa9HNjA3d8e+B1i8NoRAwNijObH3vNj\n", "Kh808XSM512KXJ5CYtKMLKUWkElLT3pcdFFPaQHl7QX3Y8SqjgE2R77MhyLdOSBdSVHDcTtSyU0A\n", "tkC6Z77t4ilzJvAlYGMX5nJ3Pj6bV8iE9i7uvQesG7s2gkqB6Kpg+J5dv5tnynEujl17C9gHGfcp\n", "H6u4NJYh3XpbIA4lByBdV93pi/IuHd3xS7Hzy1xan0JaLcfQsfzG0zkaOBDptlsdMTzRMaXJwJeB\n", "tRFDVDZ+byEfRNH3HIiMefnSaU/2ernSHlpAFdpDC/DQHlpAZ3rOGFDeBuguxGB8F+ny2AXpeil/\n", "sQ8FNoyEn4sMTA9HKoxrgSvp2PffF/gh8ALwl0icv8vqJXJmIvLl/g2kVfElpFsJuvfG+gdiHL6J\n", "5NPBkWe7YjrSpbZa5Nr1iBEY4c7XRip4kK/ILZEuqnlId+DySFwbJUjzQcSAHYS85zfp2P06EGmZ\n", "zEWM6rkezRvFwi9GWjgDnPYyfREDtbrTOS+i9ybki3pHJH8HAPu7+Gp5H0VRlA40YxccwHaIwZ6H\n", "GPF7ESM+Cvlij8f1VuzZqBfcb+jeCQHEIMxEKvCyF9xZiMPIXKRr6xIX9gh3/WNkbO4qKh83n0HG\n", "nT6i0tdejb1d2LIXXLT7bqS7N8+9z9mx9zwQ6ZKb5e4NQD5C5iLdZscgRmZDxAD9yWmag3RxRqcC\n", "7I10Xc5CWmZ3UjFA8XSM5z2KXp5CYdKMTLvgciV0mW4JmtUAxXkamZybJSbj+OvBhBbgwXiuNVt5\n", "yguTZmRqgHIldJluCYpqgLpjd6QV0gcxPPPp7E2oFIeil6eWQN2wc6UlnBCU+tiUysTIs4BD6DxB\n", "t1YuQLqz4seDDcbbFZ+tkubcDNNUFEUpBEVtAZnA6fswoQV4MKEFeDCea1qe/Jg0I9MuuFzRFpCi\n", "KIqiNCtFbQEprYWWpxzQMaBc0RaQoiiK0jqoASoGJrQADya0AA8mtAAPJrQADya0gCqY0AI8mNAC\n", "OqP7ASmKoiiKkiI9YQworS25T0VcvedS2fenGu1U3/W0DVmhuid97LRSeSosOgaUKzoGpCQijS25\n", "+wI/QhbyXA2Zf9QVSeJUFEXpgBqgYmBSiietLbmHIvvevNyYnNQxoQV4MKEFeDChBVTBhBbgwYQW\n", "0BkdA1KKRZ5bcm9CxfDMprI9xi7IJm2zkYU6d67yfG9kxfIZyCZy+3f7doqiKD2AOseAUtuRu1m2\n", "5F6fjuM2g5BuuKPdtSOQlaTLY0PRVatPQQzYuu7+E8gq1D3pY0e7I3NAx4ByRceAeijlraCvRSry\n", "++i4y+kKKltyL/I82wfZ+G85so3DpARpxrvo9ke2QrjNpXcH4uhwIJ05DNmv6R3EaF3miU9RFEUN\n", "UDJKpfQOL6aLxIcjlXmU/1Kp1GcgrZqkz75J7QZhOJ33HXrTXY8zjI5dgPHnGsGkGFdamNACPJjQ\n", "AqpgQgvwYEIL6IyOASnFIcSW3HHecc/F44kbt3KaIyLnIzxhFEVR1AAVhPYu7oXYkjvOHxHnhCOd\n", "hsMRV+8/eMLe5dIrjwF9p470qtGeYlxp0R5agIf20AKq0B5agIf20AI6M/al0AryQg1Q8VmKGI4x\n", "VBwB/oB0u1Wbf1O+tsQ9ezzwITI+c2/CdKPxfgQcAHwL2ab7HHf+kee5m4CHgReAyS49HZRXFKXH\n", "U9SVEEyN4XVL7uJgPNearTzlhUkzMt0PKFfUC64HE9+S+1PAQ0EVKYqiKDVR1BZQd3wNeB/Zvvp5\n", "YN8U4gyxJXdPoejlqSXQeUC5omU6BZrVACnNhZanHFADlCvaBdfCmNACPJjQAjyY0AI8mNACPJjQ\n", "AqpgQgvwYEIL6IzOA1IURVEUJUW0C07JAy1POaBdcLnSMl1wpwFTgYXIPJHdugm/JfAksABZ+fl7\n", "njCjgH+6OF8HTq5R0ywqc2r00KPRo7v9kxRFCcDhyOTIMcCmwDWI59V6VcKvhnh/3QGMRFZznguc\n", "HQmzATAfWXBzU+BEKhMw49iG3yAbTGgBHkxoAR5MaAEeTGgBHkxoAVUwaUZmdR5QnmRSd+bdAjob\n", "uAXZt+ZVZMmW95Dtn30cjWwxcBzwEjKrfhwdDdApSMvoDBfnz4FfIrP1m4WtQwvwoJqSoZqSU0Rd\n", "BdQ0uS20grzI0wD1QzZWeyR2/RFkszMfOyNbTC+OhR9OZXHMnavEuT2yjUEzsEZoAR5UUzJUU3KK\n", "qKuAmj4eEFpBXuRpgAYjBmF67PoHyCx/H0M94adH7gEMqRKmj0tTURRFKSBFd8Mu6phN2rSFFuCh\n", "LbQAD22hBXhoCy3AQ1toAVVoCy3AQ1toAZ2ZsXZoBa1IP2Rl59Gx69ch2zb7+CWdl/zfAdmVs9wF\n", "9yQwPhbmUMQRId4F9xrhPaT00EMPPZrteI0W4Cnghti1KcClVcKfAswB+keuXUDHHTevQJwPotwI\n", "/L1+mYqiKEqrcRjiUDAG2BxxnZ5LxQ37cuCxSPjVEC+53wBbIK7Vc4CzImHagI+Bn7g4T3RpHJTR\n", "OyiKoihNyqnIRNRFwCQ6TkS9BXgjFv5TSDfbQmQLaN9E1N2RiaiLkImoJ6UrWVEURVEURVEUpQCk\n", "vazPwcgcog+QrsGngC8G1hRlN2AZ8GIBNPUDLkZarIuAN4HTA2s6BtkKfD7SdXsr4qZfC7Xo6g9M\n", "cGkuobozzSgaWyoqbU15l/Ok+VQmj3KeVFOe5TypprzLuQHuB951ab4AnOAJ12g5byqyWNbnKuA8\n", "ZBLrhsCFyD9Cd5VjlprKrIn8EzwE/Cuhniw1/RapuPYERiDeiaMCahoFLEdWxFgf2An5x3qM5NSq\n", "axXgZ8i4433A454wtSwVlZemvMt5Ek1l8irnSTXlWc6TaApRzs9HjPDOyJj7KYhH85GRMI2W86bj\n", "afwedZdVCX8qMJuOHnX/g3xNd5fOlQXQ9Fvkq/8iavsyzELTF1yYQTXoyFrTOcC02HMnIP9YWemK\n", "Mh7/F+s4Ontp3gRMDKipWjpZlfNaNOVVzpNoyrucJ9EUupyXuRO4J3JeVzkv+kTUamS1rI+P1YCP\n", "Ams6DVgbuAQoJdCStaYvIw4k5yAu8VOQL58kS4hkpelRJI8OQPJoMHAEybcYr0dXEhpZKiorTT6y\n", "LOdJybOcJyHvcp6EopTz1elYXuoq581qgLJa1ifO15FK7taAmrZEuki+gkwIq4WsNG2IdNdsiTSx\n", "vwHsg/Rfh9L0ApJHv0EM1Qfu+vEJNNWrKwmNLBWVlaY4WZfzJORdzpOQdzlPQhHK+QHAHsh8yzJ1\n", "lfNmNUD1UGuhHg38ADiKjhNf06Q7Tf2Rpu45yOBnHiTJp17IahRHIV+IjyD/nKORr7MQmj6DVAxj\n", "kS+8fZB/qHhXg9KRPMp5d4Qo50nIu5wnIXQ53xW4DXHEmNxoZH0alhOGmchAXNzzYwjiFeLjfTpb\n", "+CGRe1EOQZYBOobkTdssNA0DNkPmR93irvdCmt5LgX3pevAxq3x6D/GIifY7v+J+jgBmBNB0FpIX\n", "P3Ln/0YGRf+KDKK+24WmenUloZr2ZS7NEJrK5FXOuyNEOU9C3uU8CSHL+W5IOfkenQ1eXeW8WVtA\n", "SxB3vy/Eru9F9UGvfwCfpeNA9l7I5NboV9dhwK+QPYh+G1jT28hE3K0ix/XIukxbuefz1gTwN6TL\n", "JtoXvon72d0XbFaaSsjXapTyeZJyXo+uJPzDxRGPcxJSEYTQBPmW8+4IUc6TkHc5T0Kocr478EfE\n", "OeQaz/1GynlTksWyPkcgX1ynI9a8fCT1gslCU5yx1OYdlIWmAcBbwF2IW/SuyJfYnQE1HYX8c52C\n", "9N3vihT+SQk11aML5P23RlzEJyEVZnSTszYaWyoqC015l/MkmuKMJdtynkRT3uU8iaYQ5dwgraxx\n", "SKumXF6i3ZBt9MAl0dJe1ucJxFqviB1dzVvIWlOci6htfkRWmjYBHkYK5tvAtSTzDspS06lUuiTe\n", "QQbVh9egqR5dU6mUk+WRn1EaXSoqbU0hynmSfIqSRzlPoinvcp5EU97l/Bb85SWuXZdEUxRFURRF\n", "URRFURRFURRFURRFURRFURRFURRFURRFURRFURRFURRFUQrLH6isIwbQjn/JkFqYAPy+wTiyYgVh\n", "NvuaQON50obo37aLMNu7MCMaTEspMM26GKmixLF0XDX7y8hyM0kwyCoAg+m4x8np1LYvTRaMRVZf\n", "3jKwjjLxfFaUulEDpBSJfsg6V2kwu45n4samll0mm4E+yOrEjRDaICstRLOuhq0Un3Zkf/urkVbF\n", "R8i+M9EKbBqy5tcvgFlUNkTbBVn3rbz+1k+BVSPPrYJ0Bc1DloE/312Pxt2OrNtVph+y5fA0KmtV\n", "nY7sqFpeA20G0u3zC3c+gY7dTf2Bq1yaC5EVgHeN3Dfu+T2QbY/nI+tsbRMJs7p7z+kujteBM/Bz\n", "PLJJ2xZU1t86NnJ/LeBuZBHI14GjI/faXPgj3PstoLI21wnASy79V4Ez6Zh3JyO7fy5E8uQhKnWF\n", "dWHPQP42HyH5tXLk+e7yycc+yFYHC4G/UFl1ukwt+aYoSg+nHVlh92qkMjkUaZVEV7CehqxqfQ6y\n", "su9GSFfTPBduI2BHZJn4uyPP/RSp/PZCKue7XDy/iIR5go5jQL9BNlw7CKmcd0Mq7F7u2gpkT5p1\n", "qBi7W4AHInFcjey3si+wKbIj5Dwq+6AYF89TwCgX5iGksi9zLfAcMsaxngt3CH5WAn4IvOx0rUNl\n", "S4oV7n2OQvLuMmT14fKKxm0uzFRkrGh9YF3ga+4dytcOQFYa/7p7bnuk6/JIF9enkYq+vK3yBOTv\n", "eIN7v72Qj4fv1JBPZW3lMaD1kI+CaFl5G1kAszwGVEu+KYrSw2mnsnlXmf+h466b04D7Y2F+Bfw8\n", "dm1rpMIaDAxEKqsjI/cHIJVgNQO0sXs+vgdKGePux7cjmEClBTQAqeC/ErnfC9mz5vuxeKL7ouzi\n", "rpVXK74fuLmKDh9j8W9LsAK4NHLeG2lxHeXO21yY+NYeb9GxpQTSAvqP+/1gxMAMrKJnArIXTrTF\n", "dCPwqPs9ST6VtZUN0GX4y0rUCaHWfFOaAO2CU7LCIi2BKE8hX+EDI2Hi2/puh1Re8yLH31zYjdzR\n", "j46blM2n671jtkEqsydqfYkIGwF9gb9Hrq1wOkbGwka3ESjvMrmO+/kz4HDgeaR1s3sDmqLpLEe6\n", "y9aJhYnm79rAJ6i0SMrH5UgrCmTb6TeRltOvkS6/uDF6iY6OCO9F0q0ln8psjr+sREkz35SCoAZI\n", "yZIkA9bzPc/cRMfdMT+NtGJeaDCtLPDtUBn1vitX1OX/tYeQrq8rkRbdg3RsudVC3MvP0vl/Opq/\n", "5Xsn0zF/t3AHyHjStsimZW8h42uvINtml4k7MvjSjePLp+jz3f390sw3pSCoAVKyogTsFLv2GWQD\n", "rY+7eO5ZZPO5NzxH2XlgKbBz5JkB7plqPI+U9T2q3C973vWuch+X7hI6btzV2+l4yftEdT5EWhcn\n", "IDtHHoe0Gqpp60pXLUxHxmY+iT9/yyxHWosXIMZ/ALB/5H5Xbtj15NPL+MtKnFryTWkC1A1byZLh\n", "iDfUzxDngnOojANUYxzS/fIzKl1FmyGD5acgxutmF24G0v1zIZ0/pkpUvqqnII4KP0cG1J9DuqLW\n", "Ryq0N5FK9QBkQusCOrfM5jtN44CZyPjVWUi31k+7eacoFyO7Rr6E/P8dTMWo+pjqdG6DjJ/NpTFX\n", "9YuQAf3ZwJ+QCnxb5G91BZIHGyGeaB8Bn0OcMl6OxNFVa6WefLoe+BYdy8rJsTC15puiKD2YJ5AK\n", "51rEQeAjpO8+aiimAmd7nt0OqRznIAbnX8hgfJlVgF9SccP+H8RZoCsvuH5Ipfg20pJ6DTgtcv+7\n", "SOtgeSSeuBdcP2TP+/ddHBMRJ4Myxj0fdWZoc9fKA+4XUNlO+UPE4G3qyYNomncj+Rd1w/athBDN\n", "z3i6UY5AKvOFLt6/IF1uIO7SjyPGYwGS98dFno3nCXTePru7fPJp24+KG/ZfEWeKqBdcrfmmKEoP\n", "Jm4AFEVROqBjQEpWRLvAFEVROqEGSMkKXTNMURRFURRFURRFURRFURRFURRFURRFURRFURRFURRF\n", "URRFURQlPf4/OqsASwpPT4QAAAAASUVORK5CYII=\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "report.metrics_vs_cut(AMS, metric_label='AMS').plot()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.8" } }, "nbformat": 4, "nbformat_minor": 0 }