{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# About\n",
"\n",
"This notebook demonstrates several additional tools to optimize classification model provided by __Reproducible experiment platform (REP)__ package:\n",
"\n",
"* __grid search for the best classifier hyperparameters__\n",
"\n",
"* __different optimization algorithms__ \n",
"\n",
"* __different scoring models__ (optimization of arbirtary figure of merit)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Populating the interactive namespace from numpy and matplotlib\n"
]
}
],
"source": [
"%pylab inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Loading data for Higgs Boson Challenge"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"['EventId',\n",
" 'DER_mass_MMC',\n",
" 'DER_mass_transverse_met_lep',\n",
" 'DER_mass_vis',\n",
" 'DER_pt_h',\n",
" 'DER_deltaeta_jet_jet',\n",
" 'DER_mass_jet_jet',\n",
" 'DER_prodeta_jet_jet',\n",
" 'DER_deltar_tau_lep',\n",
" 'DER_pt_tot',\n",
" 'DER_sum_pt',\n",
" 'DER_pt_ratio_lep_tau',\n",
" 'DER_met_phi_centrality',\n",
" 'DER_lep_eta_centrality',\n",
" 'PRI_tau_pt',\n",
" 'PRI_tau_eta',\n",
" 'PRI_tau_phi',\n",
" 'PRI_lep_pt',\n",
" 'PRI_lep_eta',\n",
" 'PRI_lep_phi',\n",
" 'PRI_met',\n",
" 'PRI_met_phi',\n",
" 'PRI_met_sumet',\n",
" 'PRI_jet_num',\n",
" 'PRI_jet_leading_pt',\n",
" 'PRI_jet_leading_eta',\n",
" 'PRI_jet_leading_phi',\n",
" 'PRI_jet_subleading_pt',\n",
" 'PRI_jet_subleading_eta',\n",
" 'PRI_jet_subleading_phi',\n",
" 'PRI_jet_all_pt',\n",
" 'Weight',\n",
" 'Label']"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy, pandas\n",
"from rep.utils import train_test_split\n",
"from sklearn.metrics import roc_auc_score\n",
"\n",
"data = pandas.read_csv('toy_datasets/Higgs.csv', sep='\\t')\n",
"labels = data['Label'].values\n",
"labels = labels == 's'\n",
"sample_weight = data['Weight'].values\n",
"\n",
"train_data, test_data, train_labels, test_labels, train_weight, test_weight = train_test_split(data, labels, sample_weight)\n",
"\n",
"list(data.columns)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Variables used in training"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"features = list(set(data.columns) - set(['Weight', 'Label', 'EventId']))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Metric definition\n",
"\n",
"In Higgs challenge the aim is to maximize `AMS` metrics.
\n",
"To measure the quality one should choose not only classifier, but also an optimal threshold, \n",
"where the maximal value of `AMS` is achieved.\n",
"\n",
"Such metrics (which require a threshold) are called threshold-based.\n",
"\n",
"__rep.utils__ contain class __OptimalMetric__, which computes the maximal value for threshold-based metric (and may be used as metric).\n",
"\n",
"Use this class to generate metric and use it in grid search."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prepare quality metric\n",
"\n",
"first we define AMS metric, and `utils.OptimalMetric` generates "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from rep.report import metrics"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def AMS(s, b): \n",
" br = 10.0\n",
" radicand = 2 *( (s+b+br) * numpy.log (1.0 + s/(b+br)) - s)\n",
" return numpy.sqrt(radicand)\n",
"\n",
"optimal_AMS = metrics.OptimalMetric(AMS, expected_s=692., expected_b=410999.)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Compute threshold vs metric quality\n",
"random predictions for signal and background were used here"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Optimal cut=0.0004, quality=1.0802\n"
]
},
{
"data": {
"text/html": [],
"text/plain": [
""
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": [
"iVBORw0KGgoAAAANSUhEUgAAAZIAAAEZCAYAAAC99aPhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
"AAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8HEW99/HPQAiELewJkSWGQCBsAaKyRCxQRFYXdhUE\n",
"UVncAvrSB7xgRCCgF5RcFuEiiYAXZbmyCGoeIFFB2XxkcUFAVllCEENCJATI7/mjepg+fWZfuqrn\n",
"fN+v17zOdE/PzDeVpOpUV3cViIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiHTfrsCNwD+AZcCnGhzv\n",
"gBuA54DFwAPAUT3MJyIiLVou5+9bBXgQ+DLwGmANjt8J33gcAGwJXARcAhzWw4wiIlIQi4Aj2njf\n",
"T4Fru5xFRETalHePpBtGAi+HDiEiIt6w0AFatC+wO7Bz6CAiIhJeq6e2dgFeAY7pTRwREWlHUXok\n",
"U4CbgVOAi+sc9ywwJpdEIiL94+/A+NAh2tFsj2RXYCEwtYljG10FNpRMCx0gItNCB4jItNABIjIt\n",
"dICIdFR35t0jWQXYNHm+HLAxMAn4J/AMMB14F/CB5BiH74mcD1wFjE72vwXMzyVxcY0NHSAiY0MH\n",
"iMjY0AEiMjZ0AGmPw9+IuAzfGJSfX5a8PhN4PHX8zMxx5Uf6mDT1SCpmhQ4QkVmhA0RkVugAEZkV\n",
"OkBEVHemqDAqXOgAEXGhA0TEhQ4QERc6QERUd6aoMEREWtdR3VnEGxKlOS50gIi40AEi4kIHyNHL\n",
"+ApSj8qjJzdzF+XyXxGRVq0JlOq87oC5uSSJh87aNEGFJCJlqg8Gq1UmOrUlIiLhqCHpXy50gIi4\n",
"0AEi4kIHiIgLHaBfqCERERFJ0TlRESkrQn0wF38l1fDUvln4G6/3zxz7PQauLDscOAc/K8gi4Ink\n",
"mHo0RiIi0kfGAu8GXmRgo2HAIwyci3AYcDDwGJVK/yRge/y0UqvhT9X9oZeBa1FD0r9c6AARcaED\n",
"RMSFDhARF/j7jwBuBa6g0ssouwk/6/kayfaH8MuOz0sdMxm4Hngh2X4KuLJXYetRQyIiQ46BGcxJ\n",
"fnb06CDGEfilw68G9gTWTb22BLgBODR17OWZ998FnAgcB2xN/XtmpAVFOCcqIvmoWR90owHpsCGZ\n",
"AryGPyUFcD+VpTJmAt/GL+b3O/zy4i8AKwG/pXLKazngeOAOfMPzLI2X5tAYiYhIN5Sg1K1HmxE+\n",
"BczGD5IDXEPl9FYJX7Hfie+l/Af+VNeSzGcsAy7EN0ojgTPwM6lv3mYmSahHUuFCB4iICx0gIi50\n",
"gBw1qg9cHiGqGIFfNnwR8HzyeBm/ZMY2+B7Jacmx30z2vzfZTvdIqpkPfLTO6z3pkWiuLRGRfH0E\n",
"eBPYFlia7Cvhx0qOSG0DzAB+g29AsqYCfwTuAd4APgGsmuyTDqhHIiJlsdYHvwC+W2X/QfjeyeVU\n",
"eiRZ6R7JZ4H7gAXAv/CD73s3+O6e9Ej6jRkcHjqEiERBleNgakiaUL6S4qumS+Fc6AARcaEDRMSF\n",
"DpCjWMdIQtJVWy34LrDM/DXWIiIiTTODX2au8149dCgRCUKnawbTqa0mmMEIg30MXkwakgds4IRo\n",
"IjI0qHIcTA1JEyz15LRMz2SGwcohw+XMhQ4QERc6QERc6AA50hjJYBojaUUJTgU+ntr1RWCxwecN\n",
"DtRgvIiIVDOoVTUYaXB6jXlyZhvsZ3BStbEUg7UM1kueL2+wgcE7kufv1/iLSNRextcJelQeL9co\n",
"K53aSqlZGAbvMzi/RoNSftxv8KnkNNhVBksbHL/Y4HKDLfL8Q4qIdJkakpSGhWGwnPnB+LuTxmBB\n",
"g8Yi/ZhnsLDO64c2+v4cudABIuJCB4iICx0gIi50gIh01JDkPUayK3Aj8A8GLhlZz9bAr4F/J+87\n",
"pZMAJVhWgpvxk6CNL/mFY1YEvp8c8jx+Vs6bgGOAnfB3y29aglHA2sB2wGj85GppV5nGXkREemov\n",
"4HTgAGAxjefOXx0/D/9PgInJ+xZS+0bDjlpVg5WsxYksk/dcnTk9Nsz8+MolBg8bXGt+MjURkRgV\n",
"9tTWIho3JMfhJyRbMbXvG/ieSTXBCsNgeqYxmZ857bXI+vgqOREptL5uSC7Hn2JKexf+tNjGVY4P\n",
"2ZAMM/hFgzGWyTlGcjl+V+xc6AARcaEDRMSFDhCRjurO2NcjGQ08ndk3L/XaU/nGqa3k1xfYy2A8\n",
"sAPwKPDHEgOW47zX4DvAe4ANgMfxvat1gO+VYE6A6CIihdVMj+RXwKWZfRvheyTvqXJ8lN0zg92a\n",
"vCpsfOisIjIk9XWP5AV8zyNtVOq1amYBTybPFwD3A3OTbZf8zHW7BHMMhn0FTtgKJh/lM93xS7gK\n",
"sA/BhsmbHp1T+ZBp34XVboG75/r1nAHch2HD62Ec8OgPYd85cPuVcDIwb384/yZ4NvSfV9va1nb0\n",
"2w44Mtl+kgJrpkdyLH5t4/Rg+8nAMzWOj7JH0ojBIY3GVgxKBns30avZLPlYF/LPFBkXOkBEXOgA\n",
"EXGhA0SkUHXnKsCk5LEYf0/IJJLfyIHpwK2p41fH39dxFbAl8DF8w3JCjc8vVGGkmZ/KZWeDjxr8\n",
"2+CZFm6UzD4+gf6TpLnQASLiQgeIiAsdICKFqjsdfnxjGfBW6vllyesz8QPQaVvhb0h8DX/apt4N\n",
"iYUqjEbMD95nG4nLLJnF2GBjSyagND+1y4DZjkPnF5HC6Ku6s1N9VxgGkwyuM7988G51jlvd4I1M\n",
"YzIiz6wiUlh9V3d2YsgXhsHKBjbHNyR7hM4TCRc6QERc6AARcaEDRKSjulN3WveZkp+T7NvJ5myD\n",
"/UPmEREpmiHfIwEwmJA5xfWWwSZJb0W/PIhIlurOFBVGwvz6K3+tcVXXRQaPGHw1dE4RiYLqzhQV\n",
"RoUDMDi2icuFf2XwLoMPGnzLYEzg7N3mQgeIiAsdICIudICIqO5MUWFUuPITg+0NNjMYb3C2wcwG\n",
"DcudBh83WC1g/m5yoQNExIUOEBEXOkBEVHemqDCalNyDcpH5tVIsuXT4qSqNyjmmxbpE+p3qzhQV\n",
"RhuSGxpXSJ5Pq9KYXFW+CVJE+pLqzhQVRoVr940GE5MxkzMyDco08ytClgw2MBjZxby95EIHiIgL\n",
"HSAiLnSAiKjuTFFhVLhufIjBaQ3GUzboxvf0mAsdICIudICIuNABIqK6M0WF0WUGyxkc1qAx2SZ0\n",
"ThHpiOrOFBVGjxisajDGYAuD6w2WZBqT/xM6o4i0TXVnigqjwvX6Cww+ZPBqpkE5rtff2wYXOkBE\n",
"XOgAEXGhA0Sko7pT02VI20rwS2At4IHU7gsNdgkUSUSkY+qRBGJwaKZnoskiRYpDdWeKCiMgg7Uy\n",
"jcm6oTOJSFNUd6aoMCpciC81P8Pw06nG5MgQOTJc6AARcaEDRMSFDhARjZFIPJL1UD6S2vXZUFlE\n",
"RNqhHkkkzC8RXO6VTEr26RcXkTip7kxRYUQkM17yqsECgxMNTkhOf81NLiEuwt3xIv1MdWeKCqPC\n",
"hQ5gcECDO+Kzj+cMPtiDKK4Hn1lULnSAiLjQASKiMRKJUwmuAz6a2nUz/t6TsgWZt6yPX2TrKYMr\n",
"DDbvdUYRkSz1SCKUTK8yvs7rOxrsV6OXMif5eZvGWER6RnVnigqjwAymGFxn8B2DZVUalRcNtgyd\n",
"U6QPqe5MUWFUuNABOmHwDoNzDb5QpUHZr8WPc73IWFAudICIuNABIqK6M0WFUeFCB+gW8wtpzco0\n",
"Jhskp8TWa+IjXK8zFogLHSAiLnSAiKjuTFFh9DGDcTXGUd5jMN7gIIPlQ+cUKaCe1p0bAVtl9u0O\n",
"zAHuAU5q4zOPB54AXgPuA6Y0OH5v4C5gITAfuB7YtMaxakj6nME5DS4h3rPOezVYL1JdT+vOa4Gz\n",
"UtsbAa8CfwJuBN4AvtTC5x0CLAWOBiYAM4BFwIY1jh+fHH8WMA7YFn/56KM1jldDUuFCB+gFg2FJ\n",
"D2S6wR1VGpLFBtcYfNLgLoOHdvH3s0wzWGjwsdB/hsBc6AARcaEDRKSndedTwK6p7ZOAx4AVku2v\n",
"Av+vhc+7G7g4s+8R4Mwaxx8IvAmUUvt2A5bh18HIUkNS4UIHyEtyeXHNnsqc2r2XSwyGhc6fMxc6\n",
"QERc6AAR6Wnd+Rq+F1L2K+A/U9sTGHxTWS3D8T2YAzL7zwfm1njP+snnfw5/7ns14Ef4U13VqCEZ\n",
"opIB+dkt3EVffvzZ4EyDXQ3OSva9ZvCSwQcNjjL4usZepM/1tO58HtgueV4CXsb3Eso2w5/qasYY\n",
"fE8iOyZyKvBwnfftDLyIb4Tewo+r1FrnQg3JEGawnMGIpFHZKNm3ucFpyUD8agbHGnykjQbnvw3W\n",
"DP1nFOmRjurORoOP9wAn4rv/hwKrArenXt8UeKaTAA2Mww+uzwQm47uii4CrGXi6K20WMC15TGVg\n",
"99UNoW3X4PW+2y7BriV4TwmsBE8DrgSjS3B7Ca4pwQ4leLgE15egtBHs9TX4evn9cxnYNc5sf+Y2\n",
"eObLfsLJTQw+cCZM3wc+EerP2+b21MjyhNwe6vXDLCr1ZU9tC7yEH6dYBpyeef0K4KImP6vWqa0L\n",
"8FeBVXM28IfMvnckWXaucrx6JBUudICIuHovmp/y/gKDbZLezHDzC3Stnbw+xeDeGj2VBw3WNBid\n",
"y5+kcy50gIi40AEi0vO6c138QkU7VnltX+CdLXzWXVQfbD+jxvHfAe7N7Fuf6qfIQA2J9IjBSgbn\n",
"G7xuML/GFWOjQucUaVOh6s6Dgdfxl/9uAZyHvz+kfPnvdODW1PFT8OMip+BPo22Pv/z3SWBElc8v\n",
"VGFI8RisUB54r9ZDCZ1PpE09/bd7AP66+0aPVhyHvyFxCb63ke5ZzAQezxx/IH6AfREwDz9mUmt6\n",
"cf1HrnChA0TE9eJDzd9p/x2DV1KNSeynuFzoABFxoQNEpKd157ImH7FQQ1LhQgeIiOvlhxuskemZ\n",
"/DG5pPgQq36/U0gudICIuNABItLTuvNZ/GmkacDGvfyiLlFDIkEYfK7GYPzrBv9RHrgXiVRP687l\n",
"8QPqP8PfnDgbP83J8F5+aQfUkEgQBismp7lq3Ycy3+L9fyOSW925Hn5KlL/gLwmeQXz/MdSQVLjQ\n",
"ASLi8vwyg5EGmxqsYn6Or3Jjcrkl9z8ZfN78RSR5cwG+M1YudICI5F53boa/76PWfFchqSGpcKED\n",
"RMSF/HKDyeanXanWU9kl5zgu5++LmQsdICK51J2rAJ8G7gAW429EdHl8cYvUkEiUDH5Y57TXmsmN\n",
"kKPM3yclkree1p1TgMvwl97eAxwLrN7LL+yQGhKJksH6Bt9IGo6fGRxRp2H5scFYqz0NkEi3dVR3\n",
"NvqHugw/Z9GPgAfrfNn/dhKiiwz95ytz1J5VeahxRFgW5u/TurbBYduV4P4ufq0jwrIIxKGyKOtp\n",
"3an7SIrLhQ4QERc6QC0GH0v1RKaaX4wr3Tu5MvX8QvOnmTvhupG7T7jQASKiujNFhSGFY7CV+Zm1\n",
"y9urGzxc47TX+SGzSt8KXnd+IHSAlOCFIdINBlvXaEiWWTFuDpZiCVJ3boC/Bv5xdGorVi50gIi4\n",
"0AHaYfDpdC/E4PZk+4gOPtZ1J11fcKEDRKSjunO5Fo4dhh8cvAU/bcpHgR8A4zsJICLVleCyEpRK\n",
"8IVkV/milg8b7NSF8RKR3GwOfBc/8+4L+MWm3gS2DBmqBvVIpG8ZbJQ5zXVO6EzSN3pad96BbzRm\n",
"46dzH5bsfwOY2MsvbpMaEulrBnMzjckaoTNJX+hp3bkMuBv4UGa/GpL4udABIuJCB+iWZP6ubTON\n",
"yYoGqzX5Ea6X+QrGhQ4QkZ6OkUzGr5l+FX5c5FQqqxmKSM5KsLgEDwA3pXYvARYa7BEolkhTRgCH\n",
"4+8CfQPfU/kasGbATNWoRyJDgsEIg9uqXB68t8E1Bu8OnVEKJfe6czxwFvA8sBS/hnos1JDIkJFq\n",
"TF6tds9J6HxSKMH+vQwDPgzcECpAFfrPU+FCB4iICx2g1wzeX6Ux+XiVQ13e2SLmQgeIiOrOFBVG\n",
"hQsdICIudIA8GKyQDLzX65W4vHNFzIUOEBHVnSkqDBnyDKakGpNtQueRQlDdmaLCkCHP/CJZv8mc\n",
"5to+dC6JmurOFBVGhQsdICIudIC8GaxtcEGmMTnqBPhK6GwRcaEDRER1Z4oKo8KFDhARFzpAKAb/\n",
"lW5M5vifW4XOFQkXOkBEcqk7HbBjavso4E7gElLrKERADYlIhsE+Bm9meic7h84lUcml7rwff6kv\n",
"wATgdeBC/PK7P8gjQJPUkIhUYbCSwbdSDcmNoTNJVHKpOxcB45LnJwM/T56/B3i2xc86HngCeA24\n",
"D5jSxHumAg/jp4J4Dphe4zg1JBUudICIuNABYnHdwKV8Z4TOE5gLHSAiudSdrwCbJc9vA76YPB+L\n",
"r9ybdQj+bvij8T2bGfhGqt78XecCfwP2S75vWwZPIlmmhqTChQ4QERc6QCwOgiMzp7j+aHCEVWb2\n",
"Hkpc6AARyaXuvA24HL8y21Iqi1m9D9+7aNbdwMWZfY8AZ9Y4fkLyfROa/Hw1JCINGOxY5S74twz2\n",
"NyiFzidB5FJ3bg08hO+ZfDO1/wLgx01+xnD8hI8HZPafj58Mspqv4U9pnYBf1vcJYBawbo3j1ZCI\n",
"NMFgVYOrDP6aaVAuNFg5dD7JXdC6cyVghSaPHYOfNTg7JnIqvrGo5gf4sZTfJ++bAtyVPKr95qSG\n",
"pMKFDhARFzpARFx2h8HTVXood5qf8bufudABItJR3dnsedGt8GuXPJjZPwHfy/hLJyHqWA5YEf8P\n",
"+rFk3+H4MZPJwL1V3jMLv3YKwAL8FWdzk22X/NT20NqmwetDaXtSldc/CWw6Fy5N7dx5Lux8C5y2\n",
"F2xS8r8ExpC/m9uTIsuT57YDjky2nyQnvwMOrbL/MPxyvM2odWrrAmBOjfd8Cz9Gklaq8TmgHolI\n",
"2wy2MXgmGSu5KtU7+Z9k+xDzg/UHhs4qXZf75b9p44GFLXzOXVQfbD+jxvF74H8TSn/3Jsm+yVWO\n",
"V0Mi0gUG76xyuiv9aOayfSmOXOrOBVSvuCfjG5lmHYy/mfFoYAvgPHxDVL78dzpwa+r4Ev5ek7n4\n",
"buh2wK/xPaRq1JBUuNABIuJCB4iIa/ZAgwdSDcfpVRqT0clxyxscZnC0wQEGmxks37M/Qfe40AEi\n",
"kkvdeSPwMwaOqayQ7Lu5xc86Dn/11RL8GEf6N5uZ+Kuz0kYDV+MbnHnAFeiqrWa40AEi4kIHiIhr\n",
"9kCDDZOruNZPtj9ucGKmMRlpMLdKI/Nvg3sNPt+zP0nnXOgAEcml7twceAnfAFwBXIkfoHkJmJhH\n",
"gCapIRHpMfPrwtc65fVolavAqo2vSlxyqzvH4McybsH3Qk5P9sVEDYlIDpIB+HJDMc/gKIO1Uq+v\n",
"aTA/dczGIfNKQ6o7U1QYFS50gIi40AEi4rr1QQaTDSbVGg8x+ECqIfmawb6WOS1t/mrOUFzA745N\n",
"z+rO7an8A9m+wSMWakgqXOgAEXGhA0TE5fllBv+ZOc31N4N1DF5I7Ts5z0wpLtD3xqhndecyYL3U\n",
"81qPt3oVoA1qSEQiYrCBwWMNLiU2g5+lT41J7jqqO+tN0DYWeBrfWIxt8DlPdhKiiwxNOicSFfNT\n",
"Ka2DH1vdps6hvwd2KTWo1AxWLsG/uxhRcqg7VwC+QzEGy9QjqXChA0TEhQ4QERfqiw1GpHogmyX7\n",
"vp8ZlB90mstgO4MvmV/p8VLzMxUf04VIrguf0S9yqTtfpXGvJAZqSCpc6AARcaEDRMSF/HKDlQ1G\n",
"VNn/51Rjsmpq/7vrnA4b2WEc1+H7+0kudef/Ap/O44s6pIZEpICSy4XLDcReqf0312lIrjC4xQZP\n",
"uySty6XuPB5/V/l5+Nl3P5Z5xEINiUhBGZySui9lqsGxVW52nFyjUam3yqo0lkvdWe+qrWV5BGiS\n",
"GpIKFzpARFzoABFxoQPUkoyh/K5KI/EjSw0EG+xe5Zh/tfGVrnvpC6+junO5Fo6r9xAR6UjJL2L3\n",
"PvxZjpuB3+CX4f5i+kquEtyOX6n1b/j5+QDWSDUqSwLf6Cg1HIFfYCprePJaLNQjERliDA6v0kO5\n",
"IXSugsnt1NZ6Vfavg05tiUhgBgcZ/D3TmHw2dK4CCdqQbAe8nEeAJqkhqXChA0TEhQ4QERc6QC8l\n",
"lxeXG5KHGhzu8shUEB3VnY3WbE//RfwaeDO1vTz+JsVbOgkgItItJb8OygT8+MlWBqNL8ELoXP2u\n",
"0S3x05KfpwLnAItTry3Fr09yHYPXVQ9FU6SIDHHJDY3plVv/A5iBX0zvzUZTsAxRudSdR+Lny4md\n",
"/oGICAbn1rjf5Aehs0Uqt7pzBHAQ8HVgzWTfeGDtvAI0QQ1JhQsdICIudICIuNAB8mIwyvyprmxj\n",
"sn5yiAuZLzI9HSMpGw/ciu8yrgFcg78B6Nhk+zOdhBAR6baSv0N+f/yUK+viZ+UA+D5wSLBgQ9jP\n",
"gf/GD7AvAsYl+3cFHg8Vqgr1SESkKoMzdGlwTbnUnS+TTPvMwIbknfgBrFioIRGRmgy+nDnN1exZ\n",
"mX6XyxQpUH3KgQ2BVzoJID3jQgeIiAsdICIudICQSn7i2acA5vpdG4dL0z+abUhmAydm9o0ETsPP\n",
"iSMiUhTvTT0fHyzFEPQO4JHk8QZwL36w/WGq3/Eeik5tiUhDBtMzp7i+bUNwAlqDnQx2o4drtmet\n",
"DBwK7IAv8D8AP8bP2BkL3ZAoIg0ZTAT+nNn9HDCuBK8HiBSEwaPA+KTSVN2ZUI+kwoUOEBEXOkBE\n",
"XOgAsbgcrq9yj8nPbQhVqAaPma83cxtsH4VfJ+BY/IqJ6UcrjsdPrfIacB8wpcn3bYq/YmxRowNF\n",
"RBo5Dv4LOAx4ILV7H+Ka0bzXcj2ddxi+4n8dPwHa85lHsw7Bz8t1NH5itRn4hqHRMpnD8afSfg4s\n",
"rHOceiQi0pbMXfDN/oJbaAZPdKNH0qyn8FdodXrN9d3AxZl9j+BXQavne8APgU9Rv0eihkRE2mKw\n",
"W6ohucbgUoPJoXP1ksFTeTYkC6jchNiu4fgrvg7I7D+fty/prmoffGOzMn7ySDUkzXGhA0TEhQ4Q\n",
"ERc6QERcdofB+zJjJn8KkCs3Bk/nOUbyE3yF3ol18FOszMvsfxEYXeM9Y4BLgE8A/+7w+0VE6ir5\n",
"dZfuS+3a0mD3UHly0JUxkmZPVU3Fr4H8fvxiV29kXj+tG2GquAK4CH/fSrNmAU8mzxcA91Pp8bjk\n",
"51DYnhtZHm3Hs02D14fKdnlf9vU9gMPmwoXJziOB2yPI281tBxx5GKw5gfx8CX8lw4v4rt5DyaP8\n",
"vBm1Tm1dAMyp8Z5lyXvKjzdT+6rNOKxTWyLSFQbHlE9xhc7SKwbP5zlG8iKDp0hpx11UH2w/o8bx\n",
"EzOPk/GrNE7ET1+f1bd/4W1woQNExIUOEBEXOkBEXL0XDXZPjZWMyClTrgxeyLMh+SewSRc+52D8\n",
"JcRHA1vgJ1BbSOXy3+n4dU9qORINtjfLhQ4QERc6QERc6AARcfVeNCilGpL355QpV+bXbMmtITkH\n",
"v257NxyHvyFxCX7sI3299kzqr29yJLqPRERyYnB60pBcGTpLLxjM70ZD0uxUABfir5z6E/AglcH2\n",
"UhLgS52E6CLNtSUiXWMwFv+L7+vAmqW45hbsmMFLwNqdzrXV7KVfE4E/4huQzYGtMw+JjwsdICIu\n",
"dICIuNABIuIaHVDyV4A+C6wI/E8yOD3DYITBOw1W6HHGXhtyMx43Q6e2KlzoABFxoQNExIUOEBHX\n",
"zEEGF1SZ3LH8yF48VCgG/8pzjKQoVBgi0lUG69dpSMzg+1bQJXsNXlFDMpgKQ0S6zuBzBl8xWN1g\n",
"zyqNyfTQGdthsFANyWAqjAoXOkBEXOgAEXGhA0TEdfJmg5EGj6Qak292KVduDF7Nc64tERFJKcEr\n",
"JdgstWtaAW9c1FWuVahHIiK5Mvhtqleyceg8rTB4Tae2BlNhiEiukjvgH0oaki+HztMKgyVqSAZT\n",
"YVS40AEi4kIHiIgLHSAirlsfZPDhpCH5p/l7TgrB4HWNkYiIxOFG/Kwfa9H52k15UhtQhXokIhKE\n",
"wQmpsZK1Q+dphsGbOrU1mApDRIIwWC/VkCwLnacZBsvUkAymwqhwoQNExIUOEBEXOkBEXLc/MHOT\n",
"4tUGo7r9Hd1UzorGSEREorFh6vlBwHWhgjRiuoekJvVIRCQog/GpXsnLofPUYrBc6jSc6s4UFYaI\n",
"BGewa6ox+UzoPNUYDEvyvYXqzgFUGBUudICIuNABIuJCB4iI6+WHp8dLevk97TJYIcn3BhojERGJ\n",
"UreWJ++Vcv0fZUMXkgpERKJgsGpyee1Sg9VD58kyWCnpkSxBdecAKgwRiYbB3NQpri1C50kzv1yw\n",
"mV+HXqe2pCoXOkBEXOgAEXGhA0TE5fAdP0k9/4vBAxHNENy1U1tqSEREemcW8H9T29sAJ4aJMkj5\n",
"PpJC3IWfJ53aEpHoGGwd21Vc5pcNNoNFRJIpFioMEYmSwahUY/J+g+GB84xMsryC6s4BVBgVLnSA\n",
"iLjQASLiQgeIiMv7CzNzcT2Y9/dnsqyZ5FiABttFRApj6/Rzg1XLGwZbGFxj8IWcshR+jOR44An8\n",
"ZWf3AVPqHOuAG4DngMXAA8BRNY5Vj0REomWwvMG8TM9kmsEXktUV356G3uAig70NNu9RlnWS7/on\n",
"Baw7DwGWAkcDE4AZ+MGeDWscfxJwGrATMBY4Fn9L/2FVji1cYYjI0GIwxuCwTGPS6LFCD3Ksm3z2\n",
"fApYd94NXJzZ9whwZguf8VPg2ir7C1cYPeRCB4iICx0gIi50gIi4kF9uMNbgzKQyf9bgSwY71GhI\n",
"xvbg+8uD/y9SsDGS4cD2wOzM/tnAzi18zkginp5ZRKSREjxZgpOB1UrwjhLMKMEfgM2AczOHn1mv\n",
"MTFY0WBi6xGAAo6RjMGHzo6JnAo83ORn7Is/NTa5ymvqkYhI4SUNw2WZXskPqxy3msH9qWP2b+E7\n",
"1k/e8zwFqzs7bUh2wV/zfEyN1wtVGCIi9RgclGok7qvy+uxMY9NU78LgdIObkvc8R4d157BO3tyG\n",
"l/CLqGSKTDyBAAAJJUlEQVTXMR6FbxXrmQLcDJzC4DGWtFnAk8nzBcD9wNxk2yU/h8J2+XkseUJu\n",
"l/fFkifk9iTg+xHlCbk9lcjrhxLMN9gUeHQu7LAnHPor+InB+Cvh3LmwR+ZN6eVza37+XPjGrGRj\n",
"/dQlyEVyF9UH28+o855dgYX4v/h61COpcKEDRMSFDhARFzpARFzoAM0wWCXT65hU5+quhmd2ktNm\n",
"6fc8QwHrzoOB1/GX/24BnIdvJMqX/04Hbk0d7/D3j5yN77mMTh7rVvnswhWGiEgjBjvVaTzONji4\n",
"fJrKBvZKqn3WOpn3P0VB687j8DckLgHuZeCYyUzg8cz2W/hzf+lH+piyQhaGiEgjNRqROw3WMyiZ\n",
"Pw1mBu+t8xklgx9nPuNJVHcOoMKocKEDRMSFDhARFzpARFzoAK0wODxV+e+RXLFVSr2+tPx6nc/Y\n",
"tkpj9DgFG2wXEZH2XAOsBtxc8qejst6++91gnZK/uCmr2ozD+gU8QwUiIkOSwXmpXsYRNY7ZsUqP\n",
"5FFUdw6gwhCRIclg1cx9JYMG3Q12r9KQPILqzgFUGBUudICIuNABIuJCB4iICx2g2ww+nWogxmde\n",
"27LGgP3fKNhcWyIi0ju/SD3frfzE/O0Vtw4+HOjBzMJFpx6JiAxpBlOTnsYVyfbade5BMfMT4KpH\n",
"IiIibyvPyTUh+TmpwfFr9DBLIalHUuFCB4iICx0gIi50gIi40AF6wWB0uadhsGeNXsjvDV5N3Xei\n",
"HomIiLxtXvJzTeCXNY75N/CvfOIUj3okIjLk1RkPedbgeoOJBg91q0fSb1QYIjLkGdyTNBIfMfhT\n",
"qiG5NHXMb9WQVKfCqHChA0TEhQ4QERc6QERc6AC9YjDGYIfk+TiDWwzutdRaUFZZ2KrjhkRzbYmI\n",
"9JmSX/XwueT548DeVQ7TGEkN6pGIiDTB4HxdtSUiIp14q1sfpIakf7nQASLiQgeIiAsdICIudIDA\n",
"unYGRw2JiMjQtCx0gFhpjEREpAkG52iMREREOqFTW9KQCx0gIi50gIi40AEi4kIHCEwNiYiIdERD\n",
"ATWoYEREmmBwtsZIRESkE4+HDhAr9UgqXOgAEXGhA0TEhQ4QERc6QEgGwwxOM9gRzbUlIiKtKsGb\n",
"wKmhc8RIPRIRkdYVbozkeOAJ4DX82sJTGhy/NfBr/Ipe/wBO6Wk6ERGJ2iHAUuBo/ML0M4BFwIY1\n",
"jl8deAH4CTAROABYCJxY43j1SCpc6AARcaEDRMSFDhARFzpARApVd94NXJzZ9whwZo3jjwMWACum\n",
"9n0D3zOpplCF0WNTQweIiMqiQmVRobKoKMypreHA9sDszP7ZwM413rMT8Fvg9czxY4CNux2wz6wR\n",
"OkBEVBYVKosKlUWX5NmQrAMsD8zL7H8RGF3jPaOrHD8v9ZqIiAQW+w2JOlXVvrGhA0RkbOgAERkb\n",
"OkBExoYO0C/yvI/kJfyKXKMy+0cBz9d4zwsM7nmMSr2W9XfU+KR9KnSAiKgsKlQWFSoL7++hA7Ti\n",
"LqoPtp9R4/hjgVcYONh+MvBM96OJiEgRHIwfOD8a2AI4D385b/ny3+nAranjV8f3Vq4CtgQ+hm9Y\n",
"Tsgpr4iIROg4/A2JS4B7GXhD4kwGTyS2Ff6GxNeAZ9ENiSIiIiIiEoqmV6lopSwccAPwHLAYeAA4\n",
"qsf58tLqv4myTfGzKizqUa4Q2imLqcDD+DMEz+FPL/eDVstib/wY7kJgPnA9/t9I0e0K3Iiv/5bR\n",
"3MUF/Vxv9nx6lSJptSxOAk7D3+A5Fn8RwxvAYb0O2mOtlkPZcOAPwM/x/yb6QTtlcS7wN2A//L+L\n",
"bYEP9TRlPloti/HJ8WcB4/Dl8Evg0Z4n7b29gNPx9d9i4IgGx/dzvQn0fnqVImm1LKr5KXBt1xKF\n",
"0W45fA/4If63s37pkbRaFhPwleeEXoYKpNWyOBA/pXoptW83/G/wa3U9XTiLaNyQtFVvxn5DYpmm\n",
"V6lopyyqGQm83K1QAbRbDvskjy8ysOIosnbK4sP4C1v2Tn4+AcwC1u1NxNy0UxZ3Aq8Cn8XPvrEa\n",
"cCRwD8X+P9KOturNojQkml6lop2yyNoX2B24pIu58tZOOYzB/5k/gT//2y/aKYtx+IrhYPxvqYcD\n",
"mwM3UewGtp2yeB7foJ6OHytagL/dYL8eZYxZW/VmURqSdugO9+p2AX6M/438vsBZ8nYFcBH+svOh\n",
"bjn86YvDgTuSx+HAu4HJAXOFMA4/uD4T/2d3+NNAV1PsRrUdbdWbRWlI8phepSjaKYuyKcAt+Ksw\n",
"sueQi6adctgN+Cb+QoM3gEuBVZLnn+lNzFy0UxbP48cFHkvteyz5nI26HTBH7ZTFMfjZMr6Ov6Lx\n",
"t8AngffhT/UMJW3Vm0VpSJbir7L5YGb/HsDvarzn98B7GThotAf+psanuh0wR+2UBfjLAG/BV6Qz\n",
"ehMtV+2Uw1b4K3LKj1Pxl4duS7EvPGinLO7Az7U3LrVvHP600FD7/1HCD6ynlbeLUkd2S7/Wm2/T\n",
"9CoVrZaFw1/6dzb+t4vRyaPoA6utlkPWkfTPVVutlkUJf2pzLjAJ2A5/70C9X0aKotWymILvxZyC\n",
"v3dke/zlv08CI3JJ3Dur4P9+J+HrgFOS50Ox3nybplepaKUsZuL/oyzLPLLlVUSt/ptIO5L+uY8E\n",
"Wi+L0fhxgIX4AdUrKP4vF2WtlsWB+IZ1Eb4srsdffFB0jsr/93QdcFny+lCrN0VERERERERERERE\n",
"REREREREREREREREREREREREhqax+DuNtw+cQ6RtQ21CMpFYDbXpykVEJOMr+DW+l+CnJD+T2r2N\n",
"ZfjJ8MrP04/bc8gq0lXDQgcQ6QPTgWPxM6T+Gr9K3/Y0t0jQu/FLuu6JXwtjaY8yiohIpFbFz5L6\n",
"uSqvjaVxj6TWMSKFoTESkc5MxC8CdFvoICKhqCER6Z3yKnvpgfQVQgQR6SU1JCKd+St+Nb4PVHlt\n",
"fvJzTGrfpMwx5TGR5bucS0RECuQs4GX8ioub4AfQj01e+x1+ffSJwM74wfj0GMkwKkugjgJG5hVa\n",
"RETiUQK+Dvwd3zt5Gvh28trm+IZkMf6qrCkMbEjAry3+FPAmuvxXRERERERERERERERERERERERE\n",
"RERERERERERERERERCTj/wNrcKHzjqavLQAAAABJRU5ErkJggg==\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"probs_rand = numpy.ndarray((1000, 2))\n",
"probs_rand[:, 1] = numpy.random.random(1000)\n",
"probs_rand[:, 0] = 1 - probs_rand[:, 1]\n",
"labels_rand = numpy.random.randint(0, high=2, size=1000)\n",
"\n",
"optimal_AMS.plot_vs_cut(labels_rand, probs_rand)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The best quality"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1.0801533437621837"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"optimal_AMS(labels_rand, probs_rand)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Hyperparameters optimization algorithms\n",
"\n",
"__AbstractParameterGenerator__ is an abstract class to generate new points, where the scorer function will be computed. It is used in grid search to get new set of parameters to train classifier.\n",
"\n",
"__Properties__:\n",
"\n",
"* `best_params_` - return the best grid point\n",
"\n",
"* `best_score_` - return the best quality\n",
"\n",
"* `print_results(self, reorder=True)` - print all points with corresponding quality\n",
"\n",
"\n",
"The following algorithms inherit from __AbstractParameterGenerator__:\n",
"\n",
"* `RandomParameterOptimizer` - generates random point in parameters space\n",
" \n",
"* `RegressionParameterOptimizer` - generate next point using regression algorithm, which was trained on previous results\n",
" \n",
"* `SubgridParameterOptimizer` - uses subgrids if grid is huge + annealing-like technique (details see in __REP__)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Grid search"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__GridOptimalSearchCV__ implemets optimal search over specified parameter values for an estimator. Parameters to use it are:\n",
"\n",
"* estimator - object of type that implements the \"fit\" and \"predict\" methods\n",
"\n",
"* params_generator - generator of grid search algorithm (AbstractParameterGenerator)\n",
"\n",
"* scorer - which implement method __call__ with kwargs: \"base_estimator\", \"params\", \"X\", \"y\", \"sample_weight\"\n",
"\n",
"* Important members are \"fit\", \"fit_best_estimator\""
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from rep.metaml import GridOptimalSearchCV\n",
"from rep.metaml.gridsearch import RandomParameterOptimizer, FoldingScorer\n",
"from rep.estimators import SklearnClassifier\n",
"from sklearn.ensemble import AdaBoostClassifier\n",
"from collections import OrderedDict"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Grid search with folding scorer\n",
"\n",
"__FoldingScorer__ provides folding cross-validation for train dataset:\n",
"\n",
"* folds - `k`, number of folds (train on `k-1` fold, test on `1` fold)\n",
"* folds_check - number of times model will be tested\n",
"* score_function - function to calculate quality with interface \"function(y_true, proba, sample_weight=None)\"\n",
"\n",
"__NOTE__: if fold_checks > 1, the quality is averaged over tests.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# define grid parameters\n",
"grid_param = OrderedDict()\n",
"grid_param['n_estimators'] = [10, 20, 30]\n",
"grid_param['learning_rate'] = [0.1, 0.05]\n",
"\n",
"# use random hyperparameter optimization algorithm \n",
"generator = RandomParameterOptimizer(grid_param)\n",
"# define folding scorer\n",
"scorer = FoldingScorer(optimal_AMS, folds=4, fold_checks=2)\n",
"\n",
"grid_sk = GridOptimalSearchCV(SklearnClassifier(AdaBoostClassifier(), features=features), generator, scorer)\n",
"grid_sk.fit(data, labels)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Print best parameters"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"OrderedDict([('n_estimators', 30), ('learning_rate', 0.1)])"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"grid_sk.generator.best_params_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Print all qualities for used parameters"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2.245: n_estimators=30, learning_rate=0.1\n",
"2.106: n_estimators=20, learning_rate=0.1\n",
"2.033: n_estimators=30, learning_rate=0.05\n",
"1.777: n_estimators=20, learning_rate=0.05\n",
"1.771: n_estimators=10, learning_rate=0.1\n",
"1.767: n_estimators=10, learning_rate=0.05\n"
]
}
],
"source": [
"grid_sk.generator.print_results()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Grid search with user-defined scorer\n",
"\n",
"You can define your own scorer with specific logic by simple way. Scorer must have just the following:\n",
"\n",
"* scorer(base_estimator, params, X, y, sample_weight)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare reweight function"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def normed_weight(y, weight):\n",
" weight[y == 1] *= sum(weight[y == 0]) / sum(weight[y == 1])\n",
" return weight"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Define scorer, which will be train model on all dataset and test it on the pre-defined dataset"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from sklearn import clone\n",
"def generate_scorer(test, labels, test_weight=None):\n",
" \"\"\" Generate scorer which calculate metric on fixed test dataset \"\"\"\n",
" def custom(base_estimator, params, X, y, sample_weight=None):\n",
" cl = clone(base_estimator)\n",
" cl.set_params(**params)\n",
" cl.fit(X, y)\n",
" res = optimal_AMS(labels, cl.predict_proba(test), sample_weight=test_weight)\n",
" return res\n",
" return custom"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# define grid parameters\n",
"grid_param = OrderedDict()\n",
"grid_param['n_estimators'] = [10, 20, 30]\n",
"grid_param['learning_rate'] = [0.1, 0.05]\n",
"grid_param['features'] = [features[:5], features[:10]]\n",
"\n",
"# define random hyperparameter optimization algorithm \n",
"generator = RandomParameterOptimizer(grid_param)\n",
"# define specific scorer\n",
"scorer = generate_scorer(test_data, test_labels, test_weight)\n",
"\n",
"grid = GridOptimalSearchCV(SklearnClassifier(clf=AdaBoostClassifier(), features=features), generator, scorer)\n",
"grid.fit(train_data, train_labels, train_weight)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(187500, 62500)"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(train_data), len(test_data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Print all tried combinations of parameters and quality"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.587: n_estimators=30, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n",
"1.583: n_estimators=20, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n",
"1.583: n_estimators=20, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n",
"1.575: n_estimators=30, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n",
"1.575: n_estimators=30, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n",
"1.550: n_estimators=20, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n",
"1.384: n_estimators=10, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n",
"1.384: n_estimators=10, learning_rate=0.1, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n",
"1.380: n_estimators=10, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet']\n",
"1.380: n_estimators=10, learning_rate=0.05, features=['DER_pt_tot', 'PRI_jet_leading_phi', 'DER_mass_MMC', 'DER_met_phi_centrality', 'DER_prodeta_jet_jet', 'DER_lep_eta_centrality', 'PRI_lep_eta', 'PRI_jet_subleading_phi', 'DER_sum_pt', 'PRI_tau_phi']\n"
]
}
],
"source": [
"grid.generator.print_results()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Results comparison "
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from rep.report import ClassificationReport\n",
"from rep.data.storage import LabeledDataStorage\n",
"\n",
"lds = LabeledDataStorage(test_data, test_labels, test_weight)\n",
"classifiers = {'grid_fold': grid_sk.fit_best_estimator(train_data[features], train_labels, train_weight),\n",
" 'grid_test_dataset': grid.fit_best_estimator(train_data[features], train_labels, train_weight) }\n",
"\n",
"report = ClassificationReport(classifiers, lds)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## ROCs"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": [
"iVBORw0KGgoAAAANSUhEUgAAAZIAAAEiCAYAAADTSFSPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
"AAALEgAACxIB0t1+/AAAIABJREFUeJztnXeYJFX1v9/eBMsuu2R2l7QsOYpkSV78iSgCBqIisoCI\n",
"iCAGvoqiIkkwExRFgUVAgpGgRGFAQZIoUaIsaYFd2DCbw8z5/XFuUTU11T3dPV11q2fO+zz1TFe6\n",
"9em7vffUvefec8AwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDMMwDKM9mQJ0+20p\n",
"8CpwOTA+oCbDGLAMCS3AMHJAgNuBccB6wJHAnsBvQopqgiHY/1GjDbAfqTEQqQCLgenANNSo/A7Y\n",
"OXXNt4BXgEXAY8D+qXImAFcBbwHzgX8DrsZzxwIX+WcuBJ4CDvbnJgNzU9c7tNe0SuqaDwFP+O9w\n",
"vP+7Sures4FHE/u7AHd7na8CPwdWTJzfA7jflz8beADYosZ3MQzDGNRMAW5M7E8CngTuTBz7EjAH\n",
"OBTYEPgusAx4lz8/CngO+DuwKzARNTSuyjMrwL2oAfgA2hPai9g4TaY+Q7LUl/Mer2sM8BpwbOpZ\n",
"U4Gv+v2tfNlfAjYAdgTuQ40nwDBgFvB9YH1gY/+9N63yXQzDMAY9U9AGeS6wAG2sb6TnW/1rwKmp\n",
"++4CrvCfjwE66d0TqMZeQBewSZXzk6nPkHQD705d9yPgnsT+bqjRm+D3fwP8OnXPNr6s1Xz53Wiv\n",
"xDBajg1tGQOVu9HexY7ABcB7gTX9uTGo4/3e1D33Apv7z+9Gh45m1vm8dwOvA880LxlQA/Gf1LEr\n",
"0V7ROn7/MKADHUID2A74FGqoou0fqK9oA/Q7TAFuBW5Cey5RWYbRb8yQGAOVhcD/0KGmLwIPA+fV\n",
"cV934nOlhXq6M8obnnHdYtQAJPk38DRqQIYDB6HGJaIC/Ao1nNG2NbARsR/lKGAntGezP2rwPtDc\n",
"VzEMwxj4TKGnjwS0R9KNvr2DOqS/lbqmA50mDPAZdGhr1Tqf+X50aKua3+FD/vlJB/jJZDvbszgF\n",
"eBzYDx2uG504dyU6LNcIf0UnEhiGYRgZTKG3IQHtlVznP3+R2Nm+MXA6Oqy0lT+/AvA8OkS0G+qw\n",
"78vZfh/q1P8A6tTeC/iIP78KaiQuRJ3oBwAvUr8hWRc1VP8Grkmd2wqdrXUROsS2IbAv8At/fn3g\n",
"HNSBvx46FfpV4BtVnmUYpWYP4Ab0R9wNHFHHPVuh490LyH6LNIw0l6G/szSfQJ3w66MN/6nAy+hw\n",
"0qP0nv67Ftpoz0Ib6n9R22E9FrgYnXa8EB1WOzBxPhpSWgDcjA5VddHTkHTWKP9uf/2+Gee282XO\n",
"Aeah05lP8+fWAP6A/v9ZBLyEGpahNZ5lGKXlQ8CZ6NvYfODTfVw/BngD/c+8ub+vE/hyjhoNwzCM\n",
"NmEufRuS49DFU8sljn0TfbMyDMMwSkDZZ229B10Qtjhx7DZ0/vx6QRQZhmEYPSi7IRkHvJk69mbi\n",
"nGEYhhGYshuS9Hx6wzAMo2QMCy2gD96gd89jzcS5HoxgpWVLmG0zUQzDMBrjBXTaeFOU3ZD8EzgX\n",
"dbZHfpK90DhJL6Uv9kakAiC68Gsd1JeybnLrprLuIpZfeyEjhy1kJAtYgay/c1lxySxWnvU2q855\n",
"i9XmTmeN+TNYfeEMVl8yk1WWzmYluhi2HLrmYGSVv8vnVju9EXR65wL4+nA45w39zMIc/i6GSrv0\n",
"GE8jngo72DkNq4uI07C6iOjX/+WiDckoNGwD6LDaemhwubfRcN7fA3ZAVwkD/Bb4DrrA7Ew0IN7X\n",
"qOMfv6Kzwp7yWwpBWDh0BRaOJ2Vk6Gl4VkJ7QGv2LuMd3kDXIrzk//bYDuK6mb/noOWpbmiq/W3k\n",
"2ujvcv7zSN9hG9NXPfUDAZmHrot41G9PoQ+egU7TLouxmRhaQImYGFpAiZgYWsBAoZWxhOrBEYfy\n",
"lsTzp6CxgC5DQ1lMStyzJfAzNPjeTHS17hlVyk+W2W9EG+J16G1gom1t+l7UtYAMA0NseF6twJIW\n",
"KR7CO4Zk5Ytg1qk0Z5Dq+Zuckl2NZahBiQIJZn3u63z0eWE/jNIUdLGfYXWRZApWFxH9ajuLNiR5\n",
"01JDUsfDhqJTkdMGJml4xvZdzDu9mmo9m5mVxrueDo0dlRMyFP1uW6BBArdBe5trAqujQ4tZQQmb\n",
"pZvaxmYWOuR5M1SeTN3ryLUu2gqH1UWEw+oiwgxJgkINST2INrZRryarZ7MW9fdqMofPaGmvppXI\n",
"cqhBGeP/rpixX+/nRnxNT6Bxsl5Gw5XMQLMczvDbq1CZ17/vZhgDCjMkCUpnSPpC1E81nuyhs8jw\n",
"9OXrEDQXxjvG5acw4iT4W+LYrCZ6NSVChqMRb6sZnFWAzYBD6BkZF33pdD0KQzMMzkhsbwLPon6e\n",
"J6FSLXhiu+Owt/AIh9VFRL/azrLP2hrwVNSP8IrfMvG9mlqTAtZCh9gm4POSb6O3npgoZr7UHj57\n",
"rZy9mojKUnT4albt6+SL6HDbemhPcHV4bSu0nlf320Q0cOP6Ncp5FjXOb2T8jT6/DZXuqkUYxiCh\n",
"rd7e66DteiStwPdqIl9NVs9mPXrmwahSDNOoPTFgdnv3aiJked4xMqyORsedgEaa3goNK18Py9Ce\n",
"zBvEQ2jT0WG1P0Flemt1G0Zu2NBWgkFpSOpBdCpzrUkBE+g70sE8sg1MslezNA/9xSIro8ON49EF\n",
"sem/0eeV+yhoDrF/JvLRPAY8BDwOlVoh4w2jSMyQJDBDEuNoYPxXdIZVX72a0VULeKcYplF9UsBL\n",
"wJwAvRpHLmPhshyxYYl6NuOAo9Hhxr6mSEf+mUtRA/MIMDPntTcO8wtEOKwuIsyQJDBDEuNo4X8S\n",
"0Xqt1quJDM0E+q7/udQePpuWQ6/GUXiDIUPQHsuqwGqooRmProfanjgTY5o5aLiK9PYEVGa0QJjD\n",
"Gs8Ih9VFhBmSBGZIAuJ7NWtRfarzemh0g1p009NX06t3U9EcNW2ODEUX274H2Bo1MBtT25f1IPAc\n",
"mqL3Rf/5EajMz1erMQgwQ5LADEmJSfRqak11Hk/f/4adaCP6EJqb/W+Vnjlr2hSpoL2XDVLbtqjR\n",
"qcZ8dOrys2iO+duB52xGmdEAZkgSmCGJcbRht11gBD17NemezXpomJYk84C/An8C/lrpnffc0YZ1\n",
"0RNZH43OujY6bXkj4FB0GDArgsBstAfzAHC/fq68xYCoi5bhsLqIsLYzwQCYmtoyXGgBeSDqiV5V\n",
"YA+B7wj8R0AS2xKBmwU+K3GwTRdSc75IRWeZyb4g3wD5I8i0nlXyzvY8XHM7yIkgO/nJAoMZF1pA\n",
"iehX21mvBRqKOgg3RhfHzUaHFv6FzqUvC2ZVByGib+gf9dtuxNOYBU1FcANwF/B4RUPgD3CkgvZc\n",
"dkps26PBNpMsAf6D9loeAO6EyusFCjXKQ65t5w5ohMy5qBN0DroCe47fn+fP75iXgAaxHskgR2AN\n",
"gaMFbhJYlHol7xJ4SuBqga8JfFB0VtUgQIaBbANyLMhlIE9l9FiWgTwCciFIX5MijIFFbm3nn9BV\n",
"uxeiyaTSi69WBvZGQ7y/CfwxLyENYIYkxoUWEBqBFQUOuhJu9AakK2O8p1vgLtHZU4MBF3+UsSDv\n",
"B/kmyF9SVdMJ8guQXdUIDUhcaAElIre280Tqj7g6kp5xnUJhhiTGhRZQIhyAwEiB7X2P5XyBzpRR\n",
"uUdgX+l7hX8746qfkrEgnwG5L6O3civIoSBbDiDD4kILKBHWdiawyjAaQmCcwNkCsxOt5hMCR/gZ\n",
"ZIMU2QLkJ1Wc9gLyIsjFIKeDHA/yUR9axmhPCmk7rwf2o/xvamZIjKbww2BfFng10Vq+JnCG1IwS\n",
"PNCR0SCfAjkX5O4ahkVAFoGcBbJ6aNVGwxTSdl6FLnqahuZV36j25cEwQxLjQgsoEa7eCwVG+N7I\n",
"E6lW8u8Cx0jfgRrLjut/EbIRyMdAPgdyGsgFGcNhd+h1pcaFFlAiCms7xwLHoauJu4F7gCPoPaUw\n",
"JGZIYlxoASXCNXqDwBAB52d4zUu0kIsF/iDwMakvb33ZcPkVLTuD3F+lt7IPGhamTLjQAkpEkLZz\n",
"S+AnwCJ0Tckvgc1DCElhhsRoOQJjfC/ldj/LK2odZwn8UmD3Ae6gbxDZGOTyKgblVpDzQI4DcWhw\n",
"SyM8hbedE4BvogsS56IhsG9GFyaeXLSYFGZIjFwRWEvgq9J7Rf2LAmcKbBpaY3mQYSCfBrm5D9/K\n",
"n0BOMaMSlELazhHAQcQG40HgGHrmp9if8FFZzZDEuNACSoTLo1CBrQTOEXgl1TI+LHCSaG6SsuHC\n",
"PFaGet/KfiAng1ydYVBe9UalqMkNrqDntAOFtJ1vATOBC9CQ11msDEwtQkwNzJDEuNACSoTLs/CE\n",
"P+XXAnMSLWOXwC0Cn5K+k4IVhQstIEaGoDG/LskwKo+ia1ryrDeXY9ntRiFt5+HUvzgxJGZIjKAI\n",
"LC9woMCfRQNIRi3jfIErRcOyDJQFfS1ERoKcDfJQyqAsAPkxyB4g40OrHMAU0nZeRnbCnVGoj6Qs\n",
"mCExSoNolOLjBO5NvW6/IfATge3EgoxmIKO8b+XZjJ7KcaHVDVAKaTu70XzUaVYHuooQUCdmSGJc\n",
"aAElwoUWIDBJ4FsCz6Raxv8KfFOKW/ToCnpOC5BhIF8B+W3KmPwB5BD6H1jStULlACHXtnMVNDpq\n",
"NzobZZXEtjpwJLpIsSyYIYlxoQWUCBdaQIRoPpUdBM4TmJ4yKn8XOFb0/1deuBzLzhFZDuSJjB7K\n",
"1/thUFwrFbY5ubad3X1sXcCpeQpoEDMkRtsgMFzgQwJXCSxItI5LBP4kcIC0h2+yIGSoX3tyRsqY\n",
"vA3yMshmoRW2Mbm2nc5v3cDHEvsO2AVNiVomzJAYbYlorK/DBW6VnuHuZwlcLJoR0tZZvINU/FTi\n",
"tHP+HyBHgmyo1xh1UkjbOZH2+BGbIYlxoQWUCBdaQCMIjBcNIPlIahznJdFIxf2JIuFapbMcSMUb\n",
"jn9mDHsJyF41bnZFqWwDcms7t0VT7Eafa21lwQxJjAstoES40AKaRWALbzxeSrWQj3hj0+iUWJeH\n",
"znIgW4N8DeSZRFW9AbJnlRtckepKTm5tZ3KmVl9+krJghsQYkIguetzDD3Mlc6d0Cdwm8GnJnqI/\n",
"SJE1QJYkDMqTIO8LrarE5NZ2TiQezprYx1YWzJAYAx7RRY8fF/ij9Fz0uEDgtwL7iC16BGRVkOkg\n",
"XQmDcjvFhWBpJ6ztTGCVEeNCCygRLrSAvBBYxU8Z/ntq6Gu6n2K8g/Rc9OhCaQ2HrJ3ymyzTtSnn\n",
"fB+kTGkwQlJI23kC8KmM458CPt9gWZ8HXgQWAg8Du/Vx/T7A/UAnMAP4M9UTa5khiXGhBZQIF1pA\n",
"EQis7xc3Pp0yKs+ILoacxCCpi2xkSzS8/VKtlrsEDcHyXZDhodUFppC28wWyG/zdgecbKOcQYAlw\n",
"NLAJcD4ain6dKtdv6K8/B/1P8C7gFjSEfRZmSIxBj+iix+1Ew7C8kTIq94qGbVk1tM5wyLp+iGta\n",
"ompuId8AkWWnkLZzEdm+kPX9uXp5AE2CleRZ4Owq1x+Ihq1Pds33RJ38Wat/zZAYRgKBYQJ7C1wh\n",
"GjgyuejxetEAk4N00aMsB3KQ96MIyMMgZQz9XwSFtJ1T0QWJaT4OvFpnGSOApcABqeMXAh1V7hmP\n",
"5jj5LDoVeUXgcnSoKwszJDEutIAS4UILKAMCo8/Q5Fu3pBY9zhENge8G2aJHp39kQ5AXEr2T60E2\n",
"DqqseAppO88GXgE+AAz3297+2Ll1ljEB7Umkh8i+DTxd475dgOmoEepC/SqrV7nWDEmMCy2gRLjQ\n",
"AkqEAxAYJ5p86+HU0Ncrosm6tgysswhc/FHWzFjMuCflyzOfF4W0nSOAq1FDsMxv3cA1/lw9NGNI\n",
"JqFG5FzUP7I7cJffssIfCDAFOM1vJ9GzEXG2b/u233NfYDOBM2+BN+5KtKI3wfPnw0USh0Iqhd78\n",
"9oc4+MWlcRXcFTnkP1oOfS3dd2hbOQVtKwt9Cd8IONhvjXb9qg1t/Qw1DFmcC/wrdWwt1CDtknG9\n",
"9UgMo0n8osfdBH4hMDPxat4tcIfAZIExoXXmj6wBck2qd3JTaFU501Zt5/1kO9vPqnL994GHUsfG\n",
"k92zgTarjJxxoQWUCBdaQIlw9VwksJzARwV+L7A4tejxaoEPiw5xtzOu9mlZLWVMHgLZG2QtBl5A\n",
"yNzazvPRDIigudrPz9ii4/VyMLAYnf67GXAeuj4kmv77PeCOxPW7oX6Rb6G9oW3R6b9TgayFRGZI\n",
"YlxoASXChRZQIlyjNwisLHCMwN0pJ8IMgQsEdpL2zPTo6rtMvpXhP5ntpxCXLQJ6s+TWdnYAKyU+\n",
"35WxRccb4Th0QeIitLeR7FlcBvwvdf2BqIN9LvAmuiBx0yplmyExjBwRWE/gFIGnUi3rcwLfEV37\n",
"NQCRSSA/BbkHZH7KqHwVpN2nUOfWdq5H+00FNENiGAUguujx3QI/Eng9ZVT+KXC8wGqhdeaDVEAO\n",
"7d1LaWty099FHP33TuLeSZlp93/MVuJCCygRLrSAEuFaXaDAUIG9BC4XmJdoWZcK3CBwsGQPRYfG\n",
"9e92GQJycsKY/LWNfSe5tZ2zgS3852RI+TJjhiTGhRZQIlxoASXC5Vm4wCiBTwj8RWBZwqh0Clwq\n",
"8D6J8xyFxrWmGPlTajFjO05CyK3t/D3qk+hADck/yPaT3JmXgCYwQ2IYJUFgTYETBR5MjQG9KvB9\n",
"ga1Da2wdcglIt/+KN7ahz6RfbWetbtgKwFGo8+xE1BG+sIqAE/ojooUI7Tl7xDAGNKJBWg9DI4Yn\n",
"84E8DlwJ/LZSf7ilkiI7oi/XKwDHQOXXgQU1QiFtZwewct4PaQHWI4lxoQWUCBdaQIlwIR/unfS7\n",
"CPxc4O1EL6Vb4E6BIwXGFiTHtb5IOd1/pQUg7219+blhbWcCq4wYF1pAiXChBZQIF1pAhMAIgf0F\n",
"fiewKGFUFgpcK7Cf1B+CqRlc64uUtUD+5b9KN8gP22SYK7ehrfOBU4D56MLDrAdV/PET+yOihdjQ\n",
"lmG0IaKzQg9Ah75c4tTbwLXo8Nf9lbZ4WZQRwKnAN9CJBU8Ch0Pl30Fl1Sa3trODfBYk5kkb/MgM\n",
"w6iFwLoCXxN4IuWkf0Hgu9J4nL9AyE4gzya+QlleuLOwtjOBVUaMCy2gRLjQAkqECy2gXrw/5V0C\n",
"PxB4LWVUHhA4QaqnlKgH1yqt1ZFRIJ1e9lyQ4/N/ZlMU0naOIHtB0UjyHcNsFDMkMS60gBLhQgso\n",
"ES60gGbwix7/n8AUgbkJg7JM4CaBQ0VnSzWCy0Nrb2R573yPZJ9XzHMbopC28wY0t0eak9DYV2XB\n",
"DIlhDHAEVvCG46bUose53tC8v0SLHj0yGuS5hDHpBFkutKoEhbSdbxGvck+yBTCjCAF1YobEMAYR\n",
"AqsLfEHg/tTQ1zSBHwpsI6WagCMnJGS+CjIstCJPIW3nArIj7m6ORvEtC2ZIYlxoASXChRZQIlxo\n",
"AXkhsJHAaQLPp4zKEwJflzhdRYQLoTOx1kS8M74MaY0LaTsfAE7POH4GGuK9LJghiXGhBZQIF1pA\n",
"iXChBeSNd9LvLHChwFspo3KXwNF+urELqHJ/kDcS0nYMp0UFFfGQfdA0ub9Fk1IdjeZwXwbsV4SA\n",
"OjFDYhjGO/hFj/v6BY4LEwZlkV8I+ZGcFz3WUjemp42TVcLoUDFFPeiDwL3oAsX5aBDHDxX18Dox\n",
"Q2IYRiYCY0VDsPxNNCRL1IK/LXCRwK7F+1Nk54QhuU+nCwfB2s4EVhkxLrSAEuFCCygRLrSAMiCw\n",
"9gVqPB5NDX39T+AM0SCTRan5YMqYhIhrWFjbORI4CPga8Yr3DYFVixJQB2ZIYlxoASXChRZQIlxo\n",
"ASXCAQhsLXCuaHj7pFF5UDTeVwG9FNkY5CX/6EdBxuX/zJ4CinjIhsBUdBrwMmCSP/5DoEyhks2Q\n",
"GIbRFH7R456iCbg6EwblDoHNClCwDsjT/rGLC15nUkjbeRPwK3SRz1xiQ7IH8L8iBNSJGRLDMPqN\n",
"wEiBkwRmemOy1K9LWTHnJ6+e6BRdmO+zej64iIfMJA6UljQk62PrSMqKCy2gRLjQAkqECy2gRLi+\n",
"LhCYIHBFwjn/qmgO+hyHu3r4TPrU2KqH9ufmIQ1cmzVFbh1gTn8EGIZhlJUKTKvA4cCOwEPAWmhY\n",
"+1sltyjElVvQzJEAt4EUlegrd64BLvWfox7JWDSM/KVV7gmB9UgMw8gF70M5NjHctVjgTGk8WGQ9\n",
"TxsG0uV7JSe3vvzeDyzgGawFPOu3pahlngU8DaxRhIA6MUNiGEauiMb3ujThjJ8qsH8OT9o3McS1\n",
"fevL7/mwnMt/hxWAo4CfARcBnyE7tHxIzJDEuNACSoQLLaBEuNACSoTrz81+AWNyHcqNon7jFiFD\n",
"QG7zxf+3deVmPyzn8tsKq4wYF1pAiXChBZQIF1pAiXD9LUBgmMAXE9OFFwp8UxrzP9d6wqhEr2TX\n",
"1pSZ/aAcy+7BdsAVwL/QQI1X+GNlwgyJYRiFIzBe4KpE7+TC1s3skvt9sfeD5DVbrJC28zCgC7gd\n",
"jfh7hv/chc5oKAtmSAzDCIZogMjF3phcKy3xIcsWiV5JXo73QtrOqcA3Mo6f4s+VBTMkMS60gBLh\n",
"QgsoES60gBLh8ihU4KMC8xMBIY/of+9EDkoYk8+1RmnPB+RQZi/mo2FS0myEJr0qC2ZIYlxoASXC\n",
"hRZQIlxoASXC5VWwwETRtSZR639b/x3xcq0vbhbIeq1RGhfe4vIy+QtwTMbxzwA3FyGgTsyQGIZR\n",
"CkQTbB3ueyXieyknNe+Il2Egd3tj8hhIK8O1FNJ2Ho8GbPwlMNlvvwSmA58HPp7YQmKGxDCMUiGw\n",
"hsA1id7JnQJN9ihkpURgx+tBhrZOZv50N7CFxAxJjAstoES40AJKhAstoES4Ih8mcJDAm96YdIqm\n",
"/G3CdyIbgcz0xuSM1slrLz4PvAgsRKcR71bHPSehq+gXAdOA71W5ru0qI0dcaAElwoUWUCJcaAEl\n",
"whX9QL8q/g+J3slNAuObKOl9iRAqrUjCFaztHN7EPYcAS9Cc75sA56Oxu9apcc+PgWfQ3PATgXeh\n",
"aX+zMENiGEap8b6TwwRmJWZ2HdJESdd7QzK7NbLy54vAgYn9S9E1JM/SWErKB1DfSpJngbOrXL8J\n",
"anjqfYYZEsMw2gKBtQRuSfROrpGGMs7KCYkpwWv1X07+vAC813/eA+1FHIKGU76pzjJGoAEfD0gd\n",
"vxCNIpzF/6FDWl9CE2i9CEwBVq9yvRmSGBdaQIlwoQWUCBdaQIlwoQX43smxAvO8RXhd4MMNlNDh\n",
"Dcmp/ZeSPwuJh59+AFzmP28GvF1nGRNQZ3zaJ/Jt1Fhk8Qv/7H/6+3YD7vdblpPKDEmMCy2gRLjQ\n",
"AkqECy2gRLjQAiIENhD4e6J3conAmDru3Nvf8oIGeeyPhPx5E4jCGP8H+JT/vBG6WLEemjEkF/t7\n",
"koshN/LHdsi43gyJYRhtiWi+k68ILPLG5FmBVfq4ayjIy96Y7Nm/xzfPsDqvuw3N2f4I2qhHixA3\n",
"R4eb6uEt1K+yZur4msDrVe55HVgGPJ849rwvZ100L0qaKcRhW2ajhq/D7zv/1/Zt3/Ztv1T7Fdgd\n",
"+JdoMNzrOmDzt+DPAu+taEOfcX8FkCnAt+CaU9Dr6nmeQ9cDQoFhrsYCFwDX03PG1Olkx+Cqxv1k\n",
"O9vPqnL9XmjvY1Li2Ab+WFaiF+uRxLjQAkqECy2gRLjQAkqECy2gGgLrJ/wmPxCosfBQJvkeyUJd\n",
"sNjsI9uHg4HF6PTfzYDzgE5i/8v3gDsS11fQtSYdwDbAu4G7gfuqlN9WlZEzLrSAEuFCCygRLrSA\n",
"EuFCC6iFD6/S7Y3J7bVndMnf+hnQMbe2sw5HT1PXH4cOhy1Ch6aSPpPL0NlZScYB16EG5000D4rN\n",
"2jIMY8Aj4ASme2MyVWDLKlce5g1J1nB/nY/Kh+nAmehQUi02RIempuclpAHMkBiGMaAQWFvgAW9M\n",
"nhZYPuOqkbowUQRk6+Yekw+TgCvRoagnUSf26ejajtOBy4Gn/Pkr6enHCIUZkhgXWkCJcKEFlAgX\n",
"WkCJcKEF1IvACn4Wl4gui8i66ufekPy0uUfkyxpofKxrUH/Fs+iQ1NX+eHoWVkjMkMS40AJKhAst\n",
"oES40AJKhAstoBEEtk1MDf5kxhXbeUPyFshyjRdvRFhlGIYxYPGr4MXP6NosdbYC8qg3Jgc1XrQR\n",
"YZVhGMaAxYdUucobkycERqWuONEbkkYTDlrbmcAqI8aFFlAiXGgBJcKFFlAiXGgBzSAwWuC/3phc\n",
"Lj3CRcmqIItBukFqRVXPKLZ5+hObxTAMwyiYCsxDo7EvBD4NHJU4+zbwZ72MIwLIGxBYj8QwjEGB\n",
"wBG+V7JQNE9TdCYK5Pi/BgI55tZ2XgpEyeX3oLlEVkVjhsQwjEGDwK8TAR79ovCmAjnm1nYuJU4B\n",
"2Y1OAy47ZkhiXGgBJcKFFlAiXGgBJcKFFtBfBEYKPOqNyXWxv0RO94bkyvqLap5a3Z6pwAnElb0L\n",
"2jPJ2gzDMIyCqaif5CDUb3IQmgQQ4pxRh4GsHEJbxEfQ0O/dfWxdoQRmYD0SwzAGHQKf8L2SpXEO\n",
"E5npeyVfqa+IfFkZNRibAatV2cqCGRLDMAYdfn3JM96YvCawNcjJ3pDcWl8R+ZB0tr8Xc7a3Gy60\n",
"gBLhQgsoES60gBLhQgtoJaLpel/2xuS5TXlqZ29IZuiq975uz4eks70Lc7a3Gy60gBLhQgsoES60\n",
"gBLhQgtoNd75/rSAdFH5sY+7JSDb9n1r89SyUs8Bv0PT7N4JfByYWeXae/ojooUItb+TYRjGgEY0\n",
"e+xDwIIhdD0iDNkNuAwqR9W+LZ+205zthmEYbUgUcn4znrzW90iu6vuWfDFne3viQgsoES60gBLh\n",
"QgsoES60gLwQOEdALuHIf3pD8mDft+SPw5zt7YYLLaBEuNACSoQLLaBEuNAC8kJgDYEFc1jR+95F\n",
"NItirVvp9rV9AAAfXUlEQVSKYRxwMnARcS9kN2D9ogTUgRkSwzAMQOA2ARnGkgV1hEspJPrvdsAz\n",
"aFauo3knpgt7ofnaDcMwjHLxGMA6vDLD72+f14PqNSQ/As4D3o3maI+4Be2VGOXDhRZQIlxoASXC\n",
"hRZQIlxoATnzGMAmPBP1NjbM60H1GpJtgSkZx9+gXDnbDcMwDOUxgP24McrfvndeD6rXkCzknfgt\n",
"PdgEmN46OUYL6QgtoER0hBZQIjpCCygRHaEF5Mx/ga5VmBm97C8JKQbgYuBGYHlgLjAJdbI/Bvw0\n",
"oK405mw3DMPwCDyxmOHRrK1ukFHVL82fscA/UCPSBUzzf+8BRhchoE7MkMS40AJKhAstoES40AJK\n",
"hAstIG8EfiMgKzFzujcmW1e/tHnqHdqaA+yOrnb/Oup43xvNRTKvPwIMwzCM3HgAYEueWOb3fx5Q\n",
"S9tgPRLDMAyPwESBZefzhe54YWK1S40IqwzDMIwEAle9yHrJFe5ZI1HWdiawyohxoQWUCBdaQIlw\n",
"oQWUCBdaQBEI7CuQNCRZfpJCfCSGYRhGe3IbMPv/cUe0n9vCxIGC9UgMwzBSCPzjJH4c9Ui+m31J\n",
"8zTTI1kJXZyY3AzDMIzy8vLGPBt93qbVhddrSCaicbUWoVkS30psM6rfZgTEhRZQIlxoASXChRZQ\n",
"IlxoAQUyZ3kWRZ/XbnXhw+q87lK0J3IU8Do2hGQYhtFOzBvP69HnBcFEAFu1qKzPAy+i8bsepv7o\n",
"wRuhK+vn1rjGDJxhGEYKge88ylaRj+SJ7Euap96hranAcn1dVAeHoLG5zkTH6e4DbgbW6eO+EcA1\n",
"wN2YsTAMw2iUeSsxO/q8UigR70OnkG3Uz3IeAH6ZOvYscHYf9/0EuAQ4AuuR1IsLLaBEuNACSoQL\n",
"LaBEuNACikLgs4m0u1lhrfrVdtbrI7ke7ZE8gya2WpY4J8QZE2sxAs1r8v3U8duAXWrc92G/bQMc\n",
"XKdewzAMI2beisylQrcIQ0aBDIfK0lYVXq8hOaEFz1oNGAq8mTo+Hc0Hn8UENIT9RwnoIGpTOkIL\n",
"KBEdoQWUiI7QAkpER2gBBTKvAoxm3rK5jBmORnR/q1WF12tIprTqgQ1yBXAR8FCg5xuGYQwE5gGs\n",
"yNwub0jGEcCQgCa1OgzYDB3Oegr4LT1zuNfiLTSHSTo175oQz0tLsScaqv47fr+CThBYChwH/Drj\n",
"nino5ACA2cB/iN88nP87GPajz2XRE3I/OlYWPSH3tyFORlcGPSH3T2LwtA9zO4Au7h4KnwR+eCSw\n",
"qj8/lYLYHHgZbZj/jia5mg28hBqWermfbGf7WTWem9y+Acz3n7NmHpizPcaFFlAiXGgBJcKFFlAi\n",
"XGgBRSGwmYCszNvLvMM97a4opO28HXW4J53qY4AbUGd5vRyM9mCORg3QeUAn8fTf70EcWSyDydis\n",
"LcMwjIYQWEdAvsr3O70h+VPvS5qn3qGtXYEd0UY/ohP4Jj4DV51ch3anTgXGA48D+wCv+PPj0Hzw\n",
"tTBjYRiG0RjzANZg+nC/v0IIETPJXoG+mz9XFszIxLjQAkqECy2gRLjQAkqECy2gKASGC8jzTIqG\n",
"tt4EqfS8pHnqXdl+IzoNdzd0Cu9QNIf7xejwlmEYhlFSKjpBafEk/je0QvdsYA1graJ1rIz6SLrR\n",
"xYjL/Oc/EXC5fQbWIzEMw8hAYLGArMqMh3yv5Kiep4tjI2B/v/U3XEoemCExDMPIQGCK5tn9z4Pe\n",
"kNze83TzNJrY6jl0KOsG/9koLy60gBLhQgsoES60gBLhQgsomKcBljBivN+f2qqCa83aOh84BV23\n",
"cQHZFqvij5/YKkGGYRhGLjwKcASXLzmFc0Db75ZQy5BsDURTxbaitiExykdHaAEloiO0gBLREVpA\n",
"iegILaBg/gnIary1rt8f2qqCaxkSV+WzYRiG0WZUYLbA1BVYsL4/tHyryq7XR/JtshewjPTnjPLh\n",
"QgsoES60gBLhQgsoES60gAA8P5p30pGMbVWh9RqS04DRGcdH+XOGYRhG+Zm5Cc9En7dPLUrMnW50\n",
"AUua9wMzihTSB+avMQzDqILAxd0go5g7y08B3jI+1Tx99UjmEgdJ/F9ify6aaOo24Hf9EWAYhmEU\n",
"RmcF2Iz//s/vv68VhfYVtDEKNXwpGsI9GbRxCToP+b5WCDFajmPwzUqphsPqIsJhdRHhGHx10Qmw\n",
"BU/OepgdANatfXl99GVIpvi/L6IGo2U5fg3DMIzCmQOwInOjpR0jWlFovc72NdFw72k+AhzYCiFG\n",
"y+kILaBEdIQWUCI6QgsoER2hBQSgE2AlZkediFGtKLSRWVsLMo4vwGZtGYZhtAudAMuzaDm/32iY\n",
"rEzqLWR9NCVumueBDVohxGg5LrSAEuFCCygRLrSAEuFCCwhAJ8BIFkaLEVuyur1eQzIL2CTj+EbU\n",
"Tn1rGIZhlIdOgDF0RoZk5SIffhHwBD2NyabAk2hyq7Jg60gMwzCqILCpgFzO4a/5dSR3x6eap94e\n",
"yddRb/9TwKt+e9IfO7k/AgzDMIzC6AQYzbyoR7K4FYXWa0jmoGl2P4iGlz8f2BvY1Z8zyocLLaBE\n",
"uNACSoQLLaBEuNACAhD5SLJiJzZNX+tIkghwu98MwzCM9mM+0D2UrpZF/m2ECnA8Opy1EJjkj38d\n",
"ODiEoCqYj8QwDKMGArNuZS/xPpLb4sPNU+/Q1heBU4FfpY5PA77QHwGGYRhGoXQOY1n0eXitC1vN\n",
"M8C+/vNc4h7JlsDMIoX0gfVIYlxoASXChRZQIlxoASXChRYQAoHHH2CHqEfyUHy4eertkawLPJ5x\n",
"fCma3MowDMNoDzpXjJf/jSnywU8BH/efkz2Sk4B/FSmkD6xHYhiGUQOBm19hrahH8np8OH+ORP0h\n",
"h6Fe/08Rx986tAgBdWKGxDAMowYC13QyOjIkEh8uhmOAl9Fsid3oosSji3p4nZghiXGhBZQIF1pA\n",
"iXChBZQIF1pACAQuFmipIWlkHcmv/LY66lt5sz8PNgzDMILQ2XNXhugKDyPCeiSGYRg1EPi2gAxl\n",
"aZfvlSxPjj2Sx4E90Mi/WTO2kszz15wBvNIfQYZhGEauzAEYSld3F8OG0IK1JLUMyR/QvOzR51os\n",
"B/w/4CrU+BjhcQzODHBZOKwuIhxWFxGOwVkXnQDDWNa9hOX0Yz+pVcBpVT5XYwM0hIphGIZRXiJD\n",
"Eu0Xurq9AmwPHAKM9sdGp0SsVKSgDMxHYhiGUQNBA22twluLvY9kAgW1nWsC96PTfruIFyT+Ejiv\n",
"ifI+D7yIBoB8GA1RXw0HXI+uY5kPPIqua8nCDIlhGEYNBHZS6/FqZEg2oKAQKT8BpgOroosQI36H\n",
"5iVphEOAnwJnAtsA9wE3A+tUuf49qPE4ANgCzdZ4MfCJBp872HChBZQIF1pAiXChBZQIF1pAIOYA\n",
"jGZeZDxGFfXgN9EAjdAzRMokehqWengA7ckkeRY4u4EyrgV+n3G8mlWd6c/ZZlt/tzIFKW0GF1pA\n",
"iXChBYRAYIKAbM+DS9AeyS56uHnq9daPRAM0plkNWNTA80YA2wLfTx2/DdilgXLGoqvs62VlbMWN\n",
"0Rr69R+uBHSEFlAiOkILCEQnwIrMHer3+90jqXdo6+/A5NSxYcDXgL818LzVgKH0XhU/HRhXZxn7\n",
"Au9Dh7cMwzCMxpgPdI+hM2r/R9e6uB7q7ZGcDNwD7ICuGfkhOtQ1Fs3bXhS7omtVTkCd9FlMAab6\n",
"z7OB/yTOOf+3YxDsR5/LoifkfnSsleW1Ul+R+9ugPsqy6Am5fxLaPpRFTyH7Fei4HRY8xo9Gw5+B\n",
"mZ+lQMYDpwN/Af6KOsvHN1jGCHSI7IDU8Z8Bd/Vx726ok+jEGtdUG3Zo9+GIZnChBZQI18Ky2v23\n",
"5EILKBEutIBQCLy0M/eJ95EcSwG/6xGog3yTFpV3P9nO9rNq3LMHOq53Uh9lmyEx8sZ+S0bbI/D4\n",
"8VwQGZIT6Ofvuh4fyRJg/f4+KMGPUX/L0cBm6DqUccAv/PnvAXckrnfo9OCLgKv9tePQKMSDjd2B\n",
"p2ucn4LGO2sHOihfGgLDGCx0jngnAlb/V7bX62z/DZqPpBVch/YsTgX+jc7W2oc42OM44unFAEcA\n",
"y6N+mtfRhYnT0F7SYOPvwKY1zkdTVKF6t93RusCaHTRvDJJa+2IqOsGiWVxBz2kHXGgBJcKFFhCQ\n",
"pCEZ0d/C6nW2r4BmRdwLTa073x+voI1BLb9FFhf5LYv0qvUjM44NRoZBHBynBkVOcy5qmEco5nsV\n",
"9RzDCM2cVhqSenskmwOPoLOgJgFbpTajebZFe2adaG/tWnR4yqFZKP8P7YldQu/exLvRf5dO4Bq0\n",
"5xbRkfGsUegw4QR0YWkn2gOsAF8Hngfe8hpW9vcsD1zpj88CHgTWQH1auwMX+rLO7+N77oUOy80G\n",
"LvDPjBrtDYA7/TNm+OeN9eeuANYFbvTP+ao//ju0XmYDd6O/0Yh90ACinWgdbpc4ty86U2cWcC/x\n",
"77facwYaHaEFlIiO0AIC0jk8XhrYb0My0GjY2R55m1qxNaF3BPASOp15KPAxYDE6O+696Ay376Fj\n",
"mMvT05BE937R33sA6s86vY9nvpfeQ1tfREPVTPDP+gXwW3/uWOAG//wKarxW9OfuAo6q43uuhjbq\n",
"H/daT/LfLbp3AzQNwXB/7d1oWJ6IF+k95DQZNYzD/bX/Tpx7nXha+livGf/3TXQaewX4tC87GiPO\n",
"ek4ac7YbbY/AD8/h/6Km61wKcLYb+bEz2rBegAbD/BP6xh/RDXwHbXTTEQR2Roe7zvP3/gF4KHHe\n",
"VXlm1tDNsajPapp/1neBA722JWiMtY3QH9u/0Tf2WuWl2Qd4Avij1/pT4I3E+RfQha1L0V7JT1CD\n",
"V4sp6BBrpPddxAZuCRqXbQw6ZTzq3XwWnTH4kP8uv0EN9851fIeBggstoES40AIC0lIfyaA3JBWo\n",
"tGpr4vETgNdSx14hLmsGcXKxeu59ieZ0TESN2Cy/PYX6Y9ZAh3xuRYfOXgPOpadvrZ43mQnoEFOS\n",
"ZK9oTV/+q2jDfwVqvKoxBDgHHYqbg/YkBO3NgPbO9kGd5x3Ew17rAV8h/p6zgLW9PsMYTATxkRj5\n",
"8DqwVurYusSNc61GOuve9RL3dFS5L6vMl4EPon6RaFvBP2MZOly2BTrDbl90SKgvfUmm0TO6cyW1\n",
"fzbaU4miJRxOz99m+jmHAfujw2Fj0enpSZ/Lw8BH0Snif0b9P9H3PCv1PUejPqFGvk870xFaQIno\n",
"CC0gINYjGUDchzagX0Df8j+Cjt9D3z2Lf6KN/InoGP/HE/fW4k30bX9M4tgv0MZ8Xb+/OtpQg3b/\n",
"t0KHueaiQ0ldibI2qOOZf0EN0cfQ73kiPWOrjUaHqTpR43hyhuYNUtcvRiPxjqJn5OjhqKEZ63XO\n",
"Tej9FfA5YEe0fkcBHyaONVTv9zGMdsec7TVox5Xt2xH7Ha5DfR2noj6CdIRjlzq2HT1nbV1N7Gx3\n",
"NZ55CeqLmEk8a+tL6KyqTnTI6Ex/7aH++DzUr/FT4heQnYFnfDlR/KZq7O2vjWZtJR31m6O9iLn+\n",
"+3w59T33R4ftZvlzo9CeRic6rHU4ccK14ejMtJnosNcDwPEpHQ/6sqahvZHIkKSfk0WZf0v14EIL\n",
"KBEutIBQCOx1FZ+InO2/paDf9R5Vtt3Rt+BVihBRB+1oSNI8gC7C7C+uBWUMFFwLy2qn31IWLrSA\n",
"EuFCCwiFwE6/44DIkPyegn7XUYrd7ipbNOOosExbVWhHQ7IH2isYhhqQ+ajz2SgnZf4tGUZdCGx6\n",
"PftFhuQGCpr+uw/wX3TseSO/HYYu+joQHZ/fBp3RYzTGJsQL5L6E1mc6X0ujfAMdJkpvf+lnubXY\n",
"vcozO3N8pmEYzRHER/Iw8P6M4+9HQ6aAzuaZWpSgKrRjjyQvXGgBJcK1sKx2/y250AJKhAstIBQC\n",
"o+/gfVGP5G8U1CPZgt5rFkCdlVv4z09Qf5ZDwzAMIxzzh7O0G6BCd2E9kkfQ+EfLJY4tjy4ce8Tv\n",
"747OoAmJ9UiMvLHfkjEguJf3dILIMJY8TD9/1/VG/z0OuAntgTyOThfdEnWy7+evmQT8vD9iDMMw\n",
"jGIYStc8YMWhdI2sJ6x4qxiNxmT6id8+SwuSxrcY65HEuNACSoRrYVnt/ltyoQWUCBdaQEgeYrtn\n",
"QWQk85+joB4J6IK0dIpcwzAMow0ZxrK5ABVkub6ubRXJB62NRlv9AboGokwM9B5Jq1LtHodOMe4k\n",
"zjtSjQ6qZ0GciK4jGkyhdgbKb8kY5DzB5h0gsgLz3iTnWVuboJFgF6JrHbZAV15/GY1ZdCcaP8ko\n",
"hkZS7VZjOPAjNODhGHT9Si0aSYlrGEabMJylswGESu5BG3+IOtj3R53sf0VDio/12y+Br/VXhFEX\n",
"9Q5DRsEeXZXz49AZd//tr6A2woUWUCJcaAElwoUWEJIRLJkDIFSG93VtX/RlSHZGI7HeBHweDf39\n",
"c+LQKBcCm/VXxCAnr1S7WWxMbEBmA3f4z7ugyZ5mowEN31Pl/qHoy8UMNBnVh/v8doZhlJLhLI0M\n",
"ydC8n9WNJjeKmItO840Y568pC034SFqWabddUu2uR0+/xiro8NZh/tihaOTcyHeSjNL7OdQQreXP\n",
"34VOATcfiWG0GW+z8ul+HclSAqfatf9U/SPPVLvVSOc5+TAa3v0q/7xrUIf+/vTmYHTq92uo8Tk7\n",
"ozzDMNqAUcyfCdDF0EZm72ZSTwFXoG/JFfSt+GLU+S70PZTSBlRCNoR5ptp11JcBbgK98568RHb6\n",
"2fH0HFpL31dWHIM7G14Sh9VFhGMQ18VyLJkzhC666f/IVl89kt+gzvaZwNvoW+ur/vNMf+7yfqsY\n",
"vOSZardeXvP3pcvJiq32OnEWRVKfDcNoL5IRgI0E7baOZDj69p9MtRv5SBw93/4h20eSTLVbj49k\n",
"Itk+kk94DYegLwlRsrK0j+RJYh/J37B1JIbRlgjstSJzIv9uUB+J0T+WogbgaGKH902oQaj2jxsd\n",
"W+LvnYz2EA9G/ST1kCx3JpoC4Cto+t2v+v2ZGff9Cp3+/SiaWuAPVTQahlF+OkdUHTkf3LRbjyQL\n",
"S7XbelwLy2qn31IWLrSAEuFCCwiJwKbjec16JAOEdKrdLYFbgioyDGMwYD2SKrRjj+QY4A10jc5/\n",
"gA+1oMwQqXYHC2X+LRlG3QiM3pBnW9IjGWi0oyEx2gv7LRkDAoHK5jxhQ1tGTVxoASXChRZQIlxo\n",
"ASXChRYQkgrIMJZ1taIsMySGYRiDlDF02kKSDGxoy8gb+y0ZA4ZPcuXb7Tq09XngRTTMysPAbn1c\n",
"vxVwN7AAXVX/rSaeOYt4XYZttvVn6yt/i2G0DSsyd2FoDc1wCLqQ7mg0adb56GyidapcPwad0XQN\n",
"sDka4bYTTayVhbRSbJvjQgsoES60gBLhQgsoES60gNAczwXP0Yazth6gd973Z9Eoslkch+bISKb6\n",
"/SbaM8mirSojZ04KLaBEWF3EWF3EDPq6+Bw/f6wVhqTIoa0RaBKn21LHb0MTK2XxHjS97OLU9RPo\n",
"HWjQ6MlKoQWUCKuLGKuLmEFfF10MTaenaIoiDclqaO6NN1PHp6Mru7MYl3H9m4lzhmEYRpMsZXjb\n",
"GZJmsKGq5pkYWkCJmBhaQImYGFpAiZgYWkBoljBicd9X9U2RSZ1GAPPRVK7JKLU/Qx3pe2bcczmw\n",
"KhqNNmIH1NeyPhpGPcnzwAYt0msYhjFYeAHYMLSIermfbGf7WVWu/xwwh57O9m/QO0+HYRiGMUg4\n",
"GHWcHw1shuYb7ySe/vs94I7E9WPQrHxXA1ug+TfmAF8qSK9hGIZRQo5DFyQuAh6i54LEy4D/pa7f\n",
"El2QuBBN/9rMgkTDMAzDMAzDMIz+EyK8SllppC4ccD0wDZ3w8ChwZM76iqLR30TERsR5WgYKzdTF\n",
"ScDT6AjBNHR4eSDQaF3sg/pwO4EZwJ/R30i7swdwA9r+dVNf9tWB3G7mHl6lnWi0Lk4BTkcXeE5E\n",
"JzEsBT6Rt9CcabQeIkYA/wJuQn8TA4Fm6uLHwDPAfujv4l3AB3NVWQyN1sWG/vpzgEloPdwCPJe7\n",
"0vz5EHAm2v7NBz7dx/UDud0E8g+v0k40WhdZXAv8vmWKwtBsPfwEuAR9OxsoPZJG62ITtPHcJE9R\n",
"gWi0Lg4EltFzOcSe6Bv8Ki1XF4659G1Immo3y74gMcLCq8Q0UxdZjAVmtkpUAJqthw/77QSKXUeV\n",
"J83UxUfQiS37+L8vAlOA1fORWBjN1MW9wDw07fVQYEVgMvAg7f1/pBmaajfbxZBYeJWYZuoizb7A\n",
"+4CLW6iraJqphwnodz4MHf8dKDRTF5PQhuFg9C31cGBT4Eba28A2Uxevowb1TNRXNBtdbrBfThrL\n",
"TFPtZrsYkmaw8CrZ7Apchb6RPxxYS9FcAVyETjsf7AxBhy8OB/7ht8OBHYHtA+oKwSTUuX4Z+t0d\n",
"Ogx0He1tVJuhqXazXQzJW0AXsGbq+Jro20QWb9Dbgq6ZONeuNFMXEbsBf0VnYaTHkNuNZuphT+A7\n",
"6ESDpcCvgVH+82fykVkIzdTF66hf4PnEsed9Oeu2WmCBNFMXx6LRMr6Gzmj8O/Ap4L3oUM9goql2\n",
"s10MyRJ0ls0HUsf3Au6rcs8/gd3p6TTaC13UmI7R1U40Uxeg0wD/ijak5+cjrVCaqYct0Rk50fZt\n",
"dHrou2jviQfN1MU/gGHo23jEJHRYaLD9/6igjvUk0X67tJGtYqC2m+9g4VViGq0Lh079Oxd9uxjn\n",
"t3Z3rDZaD2kmM3BmbTVaFxV0aLMD2AZ4N7p2oNbLSLvQaF3shvZivoWuHdkWnf47FRhZiOL8GIX+\n",
"+26DtgHf8p8HY7v5DhZeJaaRurgM/Y/SndrS9dWONPqbSDKZgbOOBBqvi3GoH6ATdaheQfu/XEQ0\n",
"WhcHooZ1LloXf0YnH7Q7jvj/e7INuNSfH2ztpmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEY\n",
"hmEYhmEYhmEYA5wOwoR1mUz5V75PQSP21mIiught2wbKfZEBlMzIMIyBzerAz4lXIb+Bhmx4f+Ka\n",
"ldBQD0UzmfIbkhXRsBYRHcAFqWuGAGugcbTqZVV6hgfpRsNlGEYvhoUWYAx6/gAsDxyFRp9dE426\n",
"msxMNzuArnahHkPXjebjaIS3M44NtpDqhmG0ASuhjdz7+riug55v2WsCN6DJqV5EEzM9gUY2juhG\n",
"M979Ds1+9wKa0CrJOcDTiXLOpWfU08n03VAfi6ZxXQjMQIP9Jd/8jwSe8uefAU6iZ4Ncj85vowEE\n",
"F6EB9S5PnJtCPLQ1hd7x1Nal59DWEDRk+hdSz9jYX7ON359KPLQ1ld4x2tbzn7dLlXOMrwd7STUM\n",
"oxCGoQEDz6NnA57mLnr6SG4B/g3shIaAv8OX8+3ENd1og/lJNDz62WhE2HUS15yK5ptYF/gQGib7\n",
"9MT5ydQ2JNujuUw+4cvdGvgisSE5BpiGDgmth2amfB04vgGdB6DRVz8ErI023J9P3H8ZalRBh7ju\n",
"RfOsrOG3IfT2kZyLhgtP8l3UGEckfSSr+fuP8mWu6o/fAvwsVc4/gR9hGIZRIB9Hh1EWoiHMf4Bm\n",
"6UuSNCSboI1a8pq10SRNaUNyVmJ/KBpG+5M1tHwOeC6xP5nahuTj6LDb6CrnX6Z37+Ik4MkGdH4Z\n",
"7TVVe8OfQk9ne9roQm9DsrXfT+YieQ74emI/7WzP8pEcgOY0j14CNvPXbV5FqzFAGWxJW4zy8Uc0\n",
"l/p+wM3ALsD9wClVrt8UbaySaYJfRd/80zyW+NyFDrmskTh2IJrg6XXUYPyYnj2WvrgN7cW8CFyJ\n",
"DrFFRmV11MBd7MuOtu/RswHvS+d1qA/pRbSncSAwogGNWTwGPE5s5Hbymq5qsJwb0ERSkYE5CngA\n",
"HcozBhFmSIwysBgdnjoDzSl/CXAa/R9nX5raF+Lf/M5o8p6b0SGnbdChrkYa6XnoW/7BaO/jFLT3\n",
"MD7xnGPpmZVxC7/Vq/NVtBd2LDp89yM0A+AKDejM4kpiQ3IYml72lQbLWAr8BjUgQ9Gc75f0U5fR\n",
"hpghMcrIf1EjsnzGuafR3+32iWNro72aRtgVTdpzFtowv4AOATVKFzqc9A10yGgU8GE0OdI0YEPU\n",
"OZ3eGmExmib5y8AOqCHapcq1S6jPAF/tte2EGsIr+7h+KdnTh38N7In6fUYD19TxbGOAYTMrjJCs\n",
"is5WugQdapmLGoj/Q3so8/x1FeKZTs8AtwK/QLPgLUb9KgvRN/laJGdLPQOshfoi7gf2Bg5tUP+H\n",
"0cb4HtRXsCe6ruO//vx30Nlms9Gez3C0BzMBnTFWj87JaAP+IFofh6DG4rnetwE6w2pH1Lk/n+xp\n",
"vKA9nbuBX6JO+t/V0BOV+36057IYmOWPP4sOD34fNU7zsm42DMPIixFoj+BBtCGejzZMP0SnBkek\n",
"HcjR9N+FaAN3BLoG5eTENVnO4bQD+Wx0fcVc4Peos70rcX4ytVPx7grcCbyFTiF+zGtJcija41no\n",
"v+M9aA+gXp0fQSchzEIb6QeAfRLXJmdtgeYcvw+tyy7i6b9d9F7ZfqR//u8zvlu6rvZF/22W0LtH\n",
"9Wlfzm4YhmG0Kauhb8kfCy1kkPI1dMjRMAyjbdgTfVOfhDrN/4YO1YysdZPRckah/prX0GFGwzCM\n",
"tuED6DDSPHRo6np6T6k18mcKutr+WmzijmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEYhmEY\n",
"Rjvw/wG/hTQCCPPNMwAAAABJRU5ErkJggg==\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"report.roc().plot()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Metric"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": [
"iVBORw0KGgoAAAANSUhEUgAAAaAAAAEZCAYAAADR8/HkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
"AAALEgAACxIB0t1+/AAAIABJREFUeJztnXe8HUX5/98nFUhoIZCChAtICyIdpEhGEKmiEDrSDNIU\n",
"KQIKX4V8kRZFpQSliEQRpIqgKE24WCKQSBGl5AckIC0kkEZ6md8fz5zv2bt3zr17ztnd2XPu8369\n",
"9nXv7s7OfHbu3Hl2Zp6ZAUVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRFEVRlM6c\n",
"BkwFFgKTgd26CDsSeAJ434V/HbgU6BsLNwr4ZyTMyelKVhRFUZqdw4ElwBhgU+AaYB6wXpXwGwHH\n",
"Alu6MF9EjNEPI2E2AOYDV7s4T3RpHJy+fEVRFKVZeRq4IXZtCnBZDXH8GJgYOR8HvBoLc1MsjKIo\n",
"ilIweuWYVj9gW+CR2PVHgF0SxvFJYO9YHDtXiXN7oHftMhVFUZQ8yNMADUYMwvTY9Q+Aod08OxEZ\n",
"35mCtKLGRu4N8cQ5Hejj0lQURVEKSJ4GqBEOA7YBjgL2An4QVo6iKIrSKH1yTGsmsBxpsUQZArzX\n",
"zbNvu5+vIK2oXwDnu/jep3MLagiwzKUZ5R1geE2qFUVRlNeRIZBUydMALUFcpb8A3Bu5vhdwdw3x\n",
"9EZabr0QA/QP4KBYmL2ASe5+lOFAqYa0MsLOAVarnI+lY69ifgzj3SWHc2e/VVjAasxlVeYxkI95\n",
"kEkcTxt9WUo/ljCIj1iHDxjER/SJZesi+vMiW1LC0pvl9GY5fVjG62zERHZhBb2Yw+ospS8r6IWl\n",
"xAp6sZS+LGIl5rA60xnCfAawhH4soR9zWJ1lnbztx1JPPpVYYfuxZMFgZrbPYO13ltD/feBF4F4o\n",
"2fpyrkFR2TKW4mmCYuoai2pKQqP/J17yNEAgHmy3As8g4zqnIK2X6939y4EdgM+782OQsZ9/IwZs\n",
"e8Rj7k5gqQtzPfAN4CfAjcCuwHHAEdm+SkP8DRiA/FEtPLsZ8FLlnBWR331HV/dreXb+bziyNIq/\n",
"jAQeBh4H5gClc+EHd/LKxVQMts9wW2DVlVg8bAcmr4OUp/64rt2RvMwX+QOe53sDbS5s1FFkBvAu\n",
"0Gcuq/Z9mL3XeY9hq3zIWistYqVed3LPjiPZ8b+rMXfRCnr1tpRKy+ndexl9BvVl6Tq9WLHG+wzt\n",
"9R7DWEpfltKXjxnIPFYrLWalAe/wif2j4m/kazd8TcYVF0eOBYhbf/Qf7tmSaPPRVuV6SNpCC6hC\n",
"W2gBHtpCC/DQFlpAXuRtgO4C1gK+CwxDvkL3A/7r7g8FNoyEX4p0tW2MVGBvAuMRY1NmmovjJ8Cp\n",
"SDfb6cB9Gb1DCpT2j12YAKXjAwjBVL81pwTP5SakE/OAe+IXJ7zJ/sdXe8KKQdsYKUODkLljK/+V\n",
"3QY9w46f6s/iDSylYTczpvQCW7Mq85JOWF5o4VHgAaRbd0X5WAsGflTTeymKUqYA3VG5YinmOxug\n",
"PbCGOIYW1GRh3V35210T2XWXM7jq4as462XEcJWPVZDWaZl1gS2qxdcuoibSeS7abKTL+fZSRt0X\n",
"XWAo3t8OiqnLoJqSkEndWcTKOEuKaoCUXLFXAN9GDMcxyFSARVBa1imklJdyt+467nIJ6Wbsh8xh\n",
"GxB/LsKhJU9TTlGaDK07UyDvL9GkmNACPJjQAjyYdKKxZ4G1sWMJ2HNrjWl92NfCaAtfjRwnRiK+\n",
"JB3NNWECpJkEE1qABxNagAcTWoCHlnBCUJQi8AvEC/HzyPqBayML3J5Ox3UGu+VNWFiCP8WvWxkj\n",
"uhlYv2G1iqK0BEVtASlBscMjLaHRqcQIe7oIn0wjPkUJTCZ1Z0/r09N+TMWDHQB8HLlwPjLpeSbw\n",
"IpTm1ByjeONNobLtyFwqHnTVWIy4wf+yBC/XmqaiZIjWnSlQ1BaQCS3AgwktwIPJLmq7g2dcqHxM\n",
"AdtWiyYL/SzMqBZhN8djDb6MV1MBMKEFeDChBXgwoQV40DEgRcmO0iSwmwG7IxshboLMLwNpzUwF\n",
"+7/InLW/QmlKl7HBEgufQsaYSsCayNykLh5hVWQ+255WJlzPQCbFdvLOi2Dd/WdKnV3BFUUpEEVt\n",
"ASmFxPYF+90qDZVfg029S8LCP+tsNS238DsrW54oStroGFAKaD+mUgd2ALLKxnrIvJ/t3Y3NofRK\n",
"qinBykjLaztkftEAul61vhfi0VdeD/HeEhySpiZFQevOVChqC8iEFuDBhBbgwYQWALYX2P+4hsfr\n",
"8NlCVPYWTnYtoXYKkU9eTGgBHkxoAR5MaAEeMqk7m2U/IEUpCKUVyJqGABvC/94B9jmw54dUhaz+\n",
"DrB6UBWKolSlqC0gpamwA9xqCk/EVlJYV8aNAiiCDZ2QqSHSV1oerTtTQDNRSRm7tscn4Mq8DZGF\n",
"QS7xWXmmq/QYtO5MgaJmogktwIMJLcCDCS3AgwF7HthXYkboFbBb5iXCQh+X8Iq+8Lm80q0RE1qA\n",
"BxNagAcTWoAHHQNSlGJS+gGUNgMGI1sXg8z/+RfYp8FeAvYksJmNz5RkLtDHQOlaONbCjlmlpShK\n",
"fRS1BaS0DLYE9ktgF3i65sZlmjI8FUlsse24uaOiNILWnSmgmajkhC2BHQb2aLDvOLtwc6YpwuoW\n",
"vmHhQ2eE7ur+KUVJhNadKVDUTDShBXgwoQV4MKEFeDDdB7EnOAP0ROZqgFFwqIUFzgjtlkeaCTGh\n",
"BXgwoQV4MKEFeNAxIEVpUha5nyaL5XviPClryJX3NfprgmV8JlronbUuRenpFLUFpLQ0do1IfT8Z\n",
"7BVgtwU7UFZWyCBFGGjhhRrWktsmCx1Ky6B1ZwpoJiqBsL/qov4/KXc1UHLHrU7E6XlrUJoKrTtT\n",
"oKiZaEIL8GBCC/BgQgvwYJIHtZuD3RvsBLDTYkboHrDfSKlFlFiThROdgHtSSLc7TA5p1IoJLcCD\n",
"CS3AQyZ1p+4HpCi5UXoZ2en0YTm3awIfuZuj3bEDcFyOov7ifu5uoVQq7keaojQ9+s+lFAw7COwe\n",
"YG9yLaFc13Jz3XDvuVbQZnmmrTQVWnemgGaiUlBsb7BznBHaOdeU4U5ngHIfi1KaBnXDbmFMaAEe\n",
"TGgBHkxoAR5MOtGUlgOPuJOJzhiNBzsiB03/1w1XR1q1YDKOvx5MaAEeTGgBeRHCAJ2GLBm/EJhM\n",
"1xPlDHA/8C4wH3gBOMETZoXn2CRFzYqSB2cDb7rfVwO+Lud2olvwtF9G6T7pfh5t4RfaElJalcOB\n",
"JcAYZLHGa4B5yFbHPs4HLgZ2BtqAU4ClwJGRMAYxOJsB60QOn3HVLjilCbCbgb0o0iVXPv4f2HXS\n",
"njtkoZeFaTHf8B9YOMTC/hb2tLCrhTXTTFdpKlqi7nwauCF2bQpwWQ1x3ElHl1GDGKC1EjzbEpmo\n",
"9BRsL7CjwLbHDNE8sNuBbQO7SiopwQYWjrfwry4mq76rKyb0WJq+7uyHtF5Gx66PR/axT8pDwI2R\n",
"c4MYoKlIV91jVO9DLWommtACPJjQAjyY0AI8mOyTsCWwF4J91WMXloD9YlqaLKxsYZyF2y3ca+FB\n",
"C3+2sMwluFoDL1K3rgwxoQV4MKEFeGj6eUCDka+n6bHrHwBDE8ZxALAHsEvk2rtI19wkoD9wDPBn\n",
"YBTwtwb0KkpBKFmkK/pisF8Gvg2sCmwB9EXK/O9TSUnGZr8dv25hJtLL0D+NdBQlb4YjLZW408GF\n",
"wCsJnt8VmAOcnCDsg4jzQpyitoAUpQ7sjpGW0CGZpiTdb9bCulmmoxSWpm8BzQSWA0Ni14cA73Xz\n",
"7G6IUfkenceQfDyDODz4mABMc7/PBp6n0gVo3E891/MmOF95IPypfPluuO9ZmPUanHAylGannN7i\n",
"duAK+CxwRzbvo+cFOjfA8e58Gi3CU/idEC7t4pndgbnAmTWkcx8yFhSnqC0gE1qABxNagAcTWoAH\n",
"EzZ5+0mwr3UcFxqf5COttlTg1RRWSzBp6UkRE1qABxNagIembwEB/Bi4FWmhTETGboYC17v7lyNr\n",
"YX3enRuk5TMe+A2VsaLlyJ4nIIZpKvAS4ujwFeBLwMHZvYaiFIXSa9IVx37AjsDpsOG+MmeotCTF\n",
"hMpxZTUXSVFy4VTEYCxCHAeiY0K3AG/EzpfTeZJpNMy5SCtqAfAhMqlunyppF7UFpCgpYIdFWkKz\n",
"pXWUUsww2UW8fVpxKk2F1p0poJmotDj2rJib9h1uodMLwdbtwWZl11RrxRlI6Xlo3ZkCRc1EE1qA\n",
"BxNagAcTWoAHE1pAZ04/07WA4nOGPgC7Rj0xWmh3kXyuAWGmgWezwoQW4MGEFuBBFyNVFCUJ1z4P\n",
"DEKMxTHAte7G2njm+CRksfupY0CKUidFbQEpSsbYsZGW0Kian4bfu4cPzECcUny07kwBzUSlh2JX\n",
"ATsjYoR+K/sO2QGJnpZleayFTCe8KoVFu+BaGBNagAcTWoAHE1qABxNagAfT+VJpAdIlV570fRAy\n",
"FeJjsP8Gu1U3cabRBefRFRwTWoAHE1pAXqgBUpQeQ+nfwAjEEN1KZQmsLYBvgO1qRfnyPCBdC05R\n",
"6kS74BSlA/aEmKfcy2BHdgoFN7oASdZiVFoP7YJTFCV1focsKLfAnW8GXOnZ9E694JTUUQNUDExo\n",
"AR5MaAEeTGgBHkxoAR5M8qClWVDaD0oDIs/tC9wUC5hGF1wNunLDhBbgwYQWkBdqgBRFcZSeBM5w\n",
"J18Few7Yvu5c14JTlAbRMSBF6Rb7Ymxc6LeGxyeP41w7jwHfD61OCYLWnSmgmago3WLbwN7jWcrH\n",
"XshY30aPSuujdWcKFDUTTWgBHkxoAR5MaAEeTGgBHkw60dhNwO4J9pBP8/wLYG0fliwDewPYVcPp\n",
"ShUTWoAHE1qAB/WCUxQlT0pToPRnKN3zfb73wBrMYhl9ewMnAf9wy/v0DixSUZqGoraAFKXQWDh5\n",
"Pivbo/j1c7FuudvBHhdxVlBaE607U0AzUVHqwMIJzuJMALsp2JvBLo0Yor8kXVdOaUq07kyBomai\n",
"CS3AgwktwIMJLcCDCS3Ag0k7QgtHOUtze+TqTmDvirWILs1TVwqY0AI8mNACPOgYkKIowfBMRC09\n",
"DaXDgB9RWeT0PFl5W1GUOEVtASlKobFwoGvi/L6LUP/PtYK2y0+ZkhPaAlIUJRhJ1oJ7zv2cDHaD\n",
"jPUoLYAaoGJgQgvwYEIL8GBCC/BgQgvwYDKIM8lacH+P/P6Gm8y6I9hShroaxYQW4MGEFpAXaoAU\n",
"RUlCkrXgxgOXAA+489HA00iL6Aa4bO+IMVKUHoeOASlKHVjY3o0BTU74hAF7C9j3Y15yX8xUqJIV\n",
"WnemgGaiotSBhU87C/JijU8OBHsY2FcjRuhjsG+IkVKaBK07U6ComWhCC/BgQgvwYEIL8GBCC/Bg\n",
"0o7QwmbOerxaZwxD4MHXwK6IGKL/gO2TrtKaMYHT92FCC/CgXnCKogSjwR1RS9Nh/xOBPsAgYCow\n",
"EjgkDXGK0gwUtQWkKIXGwrqu2fJuSjGe51pB49OJT8mYlmkBnYZ8/SxEBjR36yKsAe5HCv184AXg\n",
"BE+4UcA/XZyvAyenJ1dRFNLfEfV593P7lOJTlG45HCnIY4BNgWuAecB6VcKfD1wM7Ay0AacAS4Ej\n",
"I2E2QIzT1S7OE10aB3viK2oLyIQW4MGEFuDBhBbgwYQW4MGkHaGF1VwLaG4D0ZhIjENiY0HDGtXY\n",
"uKbCYEIL8FDUurMmngZuiF2bAlxWQxx3AvdEzsfReWD0JmCi59miZqIJLcCDCS3AgwktwIMJLcCD\n",
"STtCCys5a7G4+9BVMbFY4wuZptW6akBTITChBXgoat2ZmH5I62V07Pp4oL2GeB4Cboyc/wW4Nhbm\n",
"UKQVFN8sq+kzUVFCYKFXxFKkOJnUHh0xQPumF6+SMk0/BjQYMQjTY9c/AIYmjOMAYA86GqAhnjin\n",
"I942g2uXqShKnBKsAJa50xRdp0u3ARPcyX1gHwQ7TuYPKa1OaB/8WtgVuA04ncSzsb1MAKa532cj\n",
"g6Ht7ty4n3mfl6+FSt93HtcWWg/AmRTj7xU93xq4qkB6iFxLNf7HYVkv6GNkPbildcRX7e/3PWBz\n",
"aN8J2A/MfsBX4JAL4J4309Jf5bzH/P1qPDfA8e58Gi1AtS6464Anunl2N2AO8E3PvSeRbrwozdYF\n",
"Z0IL8GBCC/BgQgvwYEIL8GCyiNTCLNdXNqjOKEwXsZfAbgz2ukiX3G11ppOSpmCY0AI8FLXurImn\n",
"8DshdLWL4u6I582ZVe5fQWcnhBvpuDJvmZbIREUJgYX3nWVI2mVeb0pfdwboA7ATwF4J9gSwW2ab\n",
"rtIFLVF3HoZ40YwBNkdcp+dSccO+HHgsEt4gLtbjkLGeoe5YOxKmDfgY+ImL80SXxkGe9FsiExUl\n",
"BBbecgZo/YxTGgx2ccxDrnw8DvbbYOO9G0q2tEzdeSoyEXURMImOE1FvAd6InS9HBkCjRzQMSCvp\n",
"ny7O14GTqqRd1Ew0oQV4MKEFeDChBXgwoQV4MFlEauE1ZwU2rjMKU0NqG4M9GOwYWS3BvhIzRDvX\n",
"qaEBTblhQgvwUNS6s6koaiaa0AI8mNACPJjQAjyY0AI8mCwitfAfV/t/qs4oTIMKPgl2jjNAezYW\n",
"1/9hUoonTUxoAR6KWnc2FZqJilInFp5zBmjbgCoedQZor3AaeiRNPw9IUZTmJu314OphhfupY0At\n",
"gBqgYmBCC/BgQgvwYEIL8GBCC/BgMoq3vAxP/zqfNyloWO5+plV3mZTiSRMTWkBeqAFSFCUpRWoB\n",
"ad2lNB06BqQodWLhj24MaP+AKh5wY0BfCqehR6JjQIqiBKXRLrg0SLsLTgmI/hGLgQktwIMJLcCD\n",
"CS3AgwktwIPJKN5Gu+BMChrSdkIwKcWTJia0gLxQA6QoSlJ0DEhRGkDHgBSlTiz83I0BfS2gijvc\n",
"GNAfwmnokegYkKIoQSlCC+gl93N3sGsE1KGkgBqgYmBCC/BgQgvwYEIL8GBCC/BgMoq3AGNApYuR\n",
"PYVWBS4BOxrs9mAHhNOUOia0gLxQA6QoSlKK4AUH8Gv38+vAPciixjPBjgonSVG6R8eAFKVOLHzf\n",
"jQFdGFjJQLDfAXsD2PvBvufGhc4Nq6ulyaTubKYtuRVFCUsRxoCA0sfIRpQO+0PgHKAUSJBSJ9oF\n",
"VwxMaAEeTGgBHkxoAR5MaAEeTEbxFmEtOB/vuJ/jwN4D9hM1PGsy0NMoJrSAvFADpChKUgrSAurE\n",
"3cB09/to4IiAWhSlKjoGpCh1YuE0Nwb0s9BaOmN7S1ectWAvCa2mBdF5QIqiBKUoXnAeSsuBd93J\n",
"wJBKlOSoASoGJrQADya0AA8mtAAPJrQADyajeAswD6hL5rmfq9bwjMlAR6OY0ALyojsDNILO+7/v\n",
"ATwBPAOcn4UoRVEKSVHHgMqUDdA+YAvYSlNq5R46uDsyAvgY+DfwALAU+GYAXfWiY0CKUicWvuzG\n",
"gO4PrcWP3ceNAelacekTpO58E9g9cn4+8BrQ152fAzybt6gGUAOkKHViYT9Xu/8ptBY/dmWwt0aM\n",
"0H/ATgJ7Nthvgj0N7OdDq2xSgtSdC5FWT5mHgSsj55sCs3NV1BhFNUAmtAAPJrQADya0AA8mtAAP\n",
"JotILXze1ex/rjMKk6ae6tirIkbId3wlf001YUIL8BBkJYTZwFrAW8gs4x2Am2KidDUFRekZFNgL\n",
"rgNnAT8GDgbWRTav6wPsCOwE3CCLmHKlLp5QbO4HbkX+eEcig5CDIvf3B14OoKteitoCUpTCY2En\n",
"14R4JrSW+rADwD4baQk9HVpRExGk7twKmAksQ3YijE/wupVCTkqrihogRakTC9u4mvv50Frqx/YH\n",
"e0DECPXt/hmFgHXn2sCXgc947h0AbFBjfKcBU5HxpcnAbl2E7Q9MAF5AWl9PeMIYxDjGj008YYtq\n",
"gExoAR5MaAEeTGgBHkxoAR5MFpFaGOlq7Xp7PUyaehrDLpFXWaOITgkmtAAPwVZCmAH8DnjKc+8P\n",
"iDFJyuHAVUhLamtgIuJRs16V8L0RQ3Ut8CBdZ8JIYGjkeK0GXYqidE/R5wHVwUnbgW2h92kuuhuB\n",
"G00yy/fbhOk9jTTfT45cm4LMN7qgm2fHA1sAn4tdN8DjSEvtw27isOioo6LUhRWP2DeBt0vVPxqb\n",
"BDsTcbAC+ch9BXgUeEyOUlF7S0IRpO70dW35jiT0Qyaujo5dHw+0J3h+PF13wU1F1oJ6jOpNWC1U\n",
"ilInFoa4LrgPQmtpHLsj2B+B/ZfHTftbodUVkCB15zvANGAssH6DcQ1HDEV8zOdC5OujO6oZoE2A\n",
"k4BtkHGq64DlnnSguAbIhBbgwYQW4MGEFuDBhBbgwWQRqYU1XQ1d79w/k6aelDBg1wJ7UGQS64dg\n",
"a1lPLgNNhSPIPKARwL7AGMRI/BW4GbiPSn9waKa4o8xTQBtwLvA3T/gJiFEF+Ud6nkoLzLifeZ/T\n",
"zX09F7YumB7jNBVJT5RU4x8BO/1KLvZLEt5zXtC/X6kduA96zYL7toYDP4U4Sz2d4Pkszunmfh7n\n",
"BjjenU+jAKyDLL3zEuKafQ21DUZW64K7Dn/LJk61FpCPixCdcYraAlKUwmOhj2sBLQ+tJTvsvq4V\n",
"9GpoJQWjMHXnJoghWEHHSalJeAq4IXZtCnBpgmdrMUD3IWNBcQqTiYrSbFgZmV/hjFDv0HqywfYB\n",
"O8sZoQ1DqykQQevOAcBXkS6t+cgEVFNHPIchy3mMATYHrgbmUvGouZzOhmMk0nS/A5iETI7dOnL/\n",
"TOBLwMaIl9zliHH8sif9ohogE1qABxNagAcTWoAHE1qAB5NVxBYWOQO0ch2Pm7T1pIDpfMne5QzQ\n",
"13NXI5hA6XZFkDGg3RDDcygy+ewXwH6I0aiHuxDXx+8Cw4AXXXz/dfeHAvGvjgepOEBY4Dn3s/wF\n",
"1hf4IfAJxJ3y3y7Oh+rUqChKdRYjE8T7I/9vrcifkDpvX2SIQMmI7vy6VyALkf4S+BfVrWDSeUCh\n",
"0XlAitIAViamDwaGlFrCHduHHYZM6VgIDILSosCCikAmdWcSA5SEJCsqFAE1QIrSAFamZgwHZiGb\n",
"U85AnJJmIJX2dSWZrNrk2OeR7v69ofRIaDUFoLB1516hBdSAjgElx4QW4MGEFuDBhBbgwWQVsYU7\n",
"u9pox8JCC7+0HcdpM9fVAMZ/2V7uXunHuaoRTIA0uyPYWnA+1gW+B7yBjrUoSk/iCGRKxjrIeO2O\n",
"yJjrccjakCsBxwJPWdmwslkp7/p6OthGJ+ErKdAHmcPzR2R7hmeB8+jsNFBkitoCUpSWwMJGFh5w\n",
"raHHbUG7bbrH9gU7J9K4Ow/semCbZbghbYLVnZshXmbTgfeBcYgB2iKUoAZQA6QoGWNhLQszXM19\n",
"bGg99WOP9/QyzgL7CNjrwX4X7LFgVw+tNAeC1J1/Q4zNI8AhVNy2lyLzc5qNohogE1qABxNagAcT\n",
"WoAHE1qABxNagIVjXY0904rXHBRAlwfT9W3bF+y5bifV6VWGvn6er6YgBKk7VyDrIe0Tu64GKF1M\n",
"aAEeTGgBHkxoAR5MaAEeTGgBbtWEx10NPcnCbXfAoxZuc8dXQmt0mNqC2/XAHiiTVO11zgClvfdY\n",
"jZpyIUjduS3wU8TlchqycvV6qAFSFKUbLGwSWTkhftQ7mb1A2D5g57tXWju0mowJWneuDByDrJK6\n",
"FGkZnQesGVBTPagBUpQcsbCVhaMjR7lrrkUWNLVPOgN0fGglGVOYuvOTwBXAe8iWDM3khl2YTIxh\n",
"QgvwYEIL8GBCC/BgQgvwYEILqIIBsLC8QAuamsYetyc6A/RMKmoEk2JcaRFkLTgfrwHfQdZz2x9Z\n",
"K05RFCUpS5D5QhdZWVtuOdKrshyp6F4tyRqQzcBtwFXADmDboDQtrBylyBS1BaQoPQYL73SzmoK1\n",
"su1Lk2DvdLLPCa0kQzKpO5t0kljdFHY9I0XpKVjYFVnCqzeyGkvvyHGWC3ZlSXY1bgLsIcDdyO7K\n",
"u0JpQWBBWaB1ZwoUtQVkQgvwYEIL8GBCC/BgQgvwYEILqILpLoCF7SJryg0vgqbusauA/ajikm0/\n",
"H15T6hRqLThFUZTUKcE/gXuRMaLvBpaTkNIC4AvIXmQbAY+CvVomsSpKhaK2gBRFcVgY6Tzlltqm\n",
"WmvS9gV7AdglrjX0FNh9W2T9OK07U0AzUVGaAAsTXFfcL0NrqR37GbBvR3wqrgqtKAW07kyBomai\n",
"CS3AgwktwIMJLcCDCS3AgwktoAomaUALG1hYYmGFzXbVlcSaasOuCfZ8Z4Dm1tgdl5GmhtAxIEVR\n",
"egYlmArcJL9yu4WvWBhsYRULfYq/zUNpFpQuB14GVgU+E1iQUgCK2gJSFCWGhWEW3u9irtBiC2/Z\n",
"Qm8NY69ycr8fWkmDaN2ZApqJitJEWBhk4RQLf7cwx8ICC8tihuiU0DqrY/fNYKmeEGjdmQJFzUQT\n",
"WoAHE1qABxNagAcTWoAHE1pAFUxaEVnoZeGmFAxQapr82AFgF4NdAXathA+ZLBXViY4BKYqiAJQq\n",
"a8cVnNJ8ZGPPErBnYDFKYIraAlIUpUYsXF/8LjgA+23XDXdzaCUNoC0gRVGUJuRh9/MLYAvuvZcv\n",
"aoCKgQktwIMJLcCDCS3AgwktwIMJLaAKJrQADyaHNP4FfAB8AtgsQXiTqZoCEcIAnYb4+C8EJgO7\n",
"dRG2PzABeAHZQ+SJKuFGIWtILQReB05OSauiKEqDlFYAj7qTvUMq6ekcjhiSMcCmwDXAPGC9KuFX\n",
"AX4GnAjcBzzuCbMBMB+42sV5okvjYE9YHQNSlBahecaAAOwxbhzoj6GV1ElL1J1PAzfErk0BLkvw\n",
"7Hj8LaBxwKuxazcBEz1hWyITFUVpOgM0zBmgBWD7h1ZTB03vhNAP2BZ4JHb9EWCXBuLduUqc21OM\n",
"PeeTYEIL8GBCC/BgQgvwYEIL8GBCC6iCCS3Ag8knmdJ7yFjQynQ97ADFzKdMyNMADUYMwvTY9Q+A\n",
"oQ3EO8QT53Sgj0tTURSlCJQ/lL8QVEWB6BNaQAAmANPc77ORbXTb3blxP/VcjiLpIXKtKHrK53Rz\n",
"X88rmLRqAmC7AAAZXklEQVTi+zUM/wQd/gj1xtfo8wnPz34fDgTMF4BvZ59eQ+cGON6dT6MF6Acs\n",
"BUbHrl9Hde+2KNXGgJ5096IcijgixLvgdAxIUVqE5hoDArArgV3oxoKGhFZTI00/BrQEcZWONz/3\n",
"wu8wkJR/uDjicU6iKZbqAIrZ52tCC/BgQgvwYEIL8GBCC6iCCS3Ag8kvqdIi5IMZOtdZUUz2WopB\n",
"3vOAfow068YAmyOu00OB6939y4HHYs+MBLZGxnMGAlu58zLXA+sCP3FxnggcB1yZxQsoiqI0QGRV\n",
"BCUEpyITURchrZSoR8gtwBux8FORhQfLiw/6FiHcHWldLUImop5UJW3tglOUFqH5uuAA7BauC+59\n",
"sM20Eo3WnSmgmagoLUKTGqAS2HecEdoutJoaaPoxIKU6JrQADya0AA8mtAAPJrQADya0gCqY0AI8\n",
"mHyTK1ngXndyKdhNPYFMfnrCogZIURQlX34ELEbWhXsF7ItgL3Ldc7padgujXXCK0iI0ZxdcGbsD\n",
"2FvAzuq4u7h9Gez3wW5VMGOkdWcKaCYqSovQ3AaojO0Hdm+wN4GdGTNG3wmtLoKOAbUwJrQADya0\n",
"AA8mtAAPJrQADya0gCqY0AI8mLDJl5ZA6WEofQ2ZkvJ5uPfv7mYzOSnUhRogRVGUQlBaBqU/w4uP\n",
"dh9WaUa0C05RWoTW6ILzYQ91XXB3h1YSQbvgFEVRlNZBDVAxMKEFeDChBXgwoQV4MKEFeDChBVTB\n",
"hBbgwYQW0JmxI0MryAs1QIqiKIqSAzoGpCgtgo4B5YqOASmKoiitgxqgYmBCC/BgQgvwYEIL8GBC\n",
"C/BgQguoggktwIMJLaAzOgakKIqiKEqK6BiQorQIOgaUKzoGpCiKorQOaoCKgQktwIMJLcCDCS3A\n",
"gwktwIMJLaAKJrQADya0gM7oGJCiKIqiKCmiY0CK0iLoGFCu6BiQoiiK0jqoASoGJrQADya0AA8m\n",
"tAAPJrQADya0gCqY0AI8mNACOtNzxoD6hBZQED4C1gwtQmkZZgGDQotQlKKjBkhYEyjS/utKcxN6\n",
"rLE9cPrVaA8twEN7aAGdGftSaAV5oV1wiqIoShDUABUDE1qABxNagAcTWoAHE1qABxNaQBVMaAEe\n",
"TGgBnek5Y0BqgBRFUZQew2nAVGAhMBnYrZvwWwJPAguAt4Hvxe4bYIXn2MQTV7W++dB99kproeUp\n",
"B3QeUK60xDygw4GrgEuArYGJwJ+A9aqEXw14FHgP2B44AzgXONsTdiQwNHK8lqbwJuKzwCtd3J8A\n",
"fD8fKQ3TDowJLUJRlGzI2wCdDdwC3Ay8CnwTMS6nVgl/NLAScBzwEnAvMA6/AZoBfBA5VqQpPGNM\n",
"inH9Fdisi/uW7r9mDJKHadBO/UYkrtV0EXYasEed6dRCPB2TQ5q1YkILqIIJLcCDCS2gMzoGlAX9\n",
"gG2BR2LXHwF2qfLMzkiFujgWfjiwfizsZOBd4DEKWahyIalbfZ4u53l1R1nyea+80lGUlidPAzQY\n",
"6A1Mj13/AOky8zHUE3565B6I0TkFONgdrwJ/pvuxpSLR3s39bYHngLnAXcCdSDeaQcbFzkNakje7\n",
"a/+NPLsN8Kx79g6kRdkVA5Bu0bWAee65oUil+x2ka3Om01CevLsS8Gt3fRbwDLAOcCnSJTjexXVN\n",
"N2nvhXQfzgaudWmWK/uNgAtdGjNcequ7e7cCI4Dfu3TOcdfvRvJlNjKOGP2y3A/4j3u/t4FvRe4d\n",
"ADzv3uXvyDhktXTau3mnELSHFlCF9tACPLSHFtCZnjMPKE+GI91iccNwIdXHLB4Gfh67NsLFs1MX\n",
"aT0I3O+5XpcTghvoTOXoKp0q9APeBE5HDPhBSIvwYmAUsBS4HOiLGAJDxQCVnz3DPTsaWOKe7YpR\n",
"dDRiuDgmIn/HvsD1wO3u3snAAy79EmL0VnX3ngC+muA9ByPG4GCn9Uz3buVnNwL2dGkPRgzKTyLP\n",
"T6VzF9zxiEHt68I+F7n3HrCr+311pxn3czqwg3uXY13cfbtIJ446IeSAOiHkSiZlOs+VEGYCy4Eh\n",
"setDkMrAx/t0bh0NidyrxjOIw4OPCUg/PsiX8fORe8b9bI+dp4mpEn97lfQ/jVTI17rzcgsDxJHD\n",
"AhchlfWu7lqZU4BVgKvd+YdICzGqJZ5emf6x+ycDNyHehe8C/wu8hXwgLEFaTEciFXQ0vjWotGKq\n",
"pdeOtEjeQpZFWo44q1wAbOrCvA580d1rRwzKD5APjXJ8WyEfJ+XzaYghaXd6zwD2RVp4SxBjvjrw\n",
"R8Q4GeAs4AZgkjt/CzH4n0H+DtEWpEHy+6rY+xA7971vlufla6HSr3Z+JvL/lkp8v4bhn6DDS9cT\n",
"n+/vl4q++s/HjoSxwN1r468v8jg3yAccVOrLpucp5J87yhSkq8bHKcAcpDIscwGdv87j3IeMBcUp\n",
"qhu26eLeEcDTsWu3I11wo5Duo3hc5fw5goqxij/bnZ64E8IC5G8xK3IsAIYhHzIXIl1a7yCOIuWP\n",
"m6QtoO8g3YtRJkaeHQI8jrzvHKQL7M1I2HjLpBdwBdJlWNa9HNjA3d8e+B1i8NoRAwNijObH3vNj\n",
"Kh808XSM512KXJ5CYtKMLKUWkElLT3pcdFFPaQHl7QX3Y8SqjgE2R77MhyLdOSBdSVHDcTtSyU0A\n",
"tkC6Z77t4ilzJvAlYGMX5nJ3Pj6bV8iE9i7uvQesG7s2gkqB6Kpg+J5dv5tnynEujl17C9gHGfcp\n",
"H6u4NJYh3XpbIA4lByBdV93pi/IuHd3xS7Hzy1xan0JaLcfQsfzG0zkaOBDptlsdMTzRMaXJwJeB\n",
"tRFDVDZ+byEfRNH3HIiMefnSaU/2ernSHlpAFdpDC/DQHlpAZ3rOGFDeBuguxGB8F+ny2AXpeil/\n",
"sQ8FNoyEn4sMTA9HKoxrgSvp2PffF/gh8ALwl0icv8vqJXJmIvLl/g2kVfElpFsJuvfG+gdiHL6J\n",
"5NPBkWe7YjrSpbZa5Nr1iBEY4c7XRip4kK/ILZEuqnlId+DySFwbJUjzQcSAHYS85zfp2P06EGmZ\n",
"zEWM6rkezRvFwi9GWjgDnPYyfREDtbrTOS+i9ybki3pHJH8HAPu7+Gp5H0VRlA40YxccwHaIwZ6H\n",
"GPF7ESM+Cvlij8f1VuzZqBfcb+jeCQHEIMxEKvCyF9xZiMPIXKRr6xIX9gh3/WNkbO4qKh83n0HG\n",
"nT6i0tdejb1d2LIXXLT7bqS7N8+9z9mx9zwQ6ZKb5e4NQD5C5iLdZscgRmZDxAD9yWmag3RxRqcC\n",
"7I10Xc5CWmZ3UjFA8XSM5z2KXp5CYdKMTLvgciV0mW4JmtUAxXkamZybJSbj+OvBhBbgwXiuNVt5\n",
"yguTZmRqgHIldJluCYpqgLpjd6QV0gcxPPPp7E2oFIeil6eWQN2wc6UlnBCU+tiUysTIs4BD6DxB\n",
"t1YuQLqz4seDDcbbFZ+tkubcDNNUFEUpBEVtAZnA6fswoQV4MKEFeDCea1qe/Jg0I9MuuFzRFpCi\n",
"KIqiNCtFbQEprYWWpxzQMaBc0RaQoiiK0jqoASoGJrQADya0AA8mtAAPJrQADya0gCqY0AI8mNAC\n",
"OqP7ASmKoiiKkiI9YQworS25T0VcvedS2fenGu1U3/W0DVmhuid97LRSeSosOgaUKzoGpCQijS25\n",
"+wI/QhbyXA2Zf9QVSeJUFEXpgBqgYmBSiietLbmHIvvevNyYnNQxoQV4MKEFeDChBVTBhBbgwYQW\n",
"0BkdA1KKRZ5bcm9CxfDMprI9xi7IJm2zkYU6d67yfG9kxfIZyCZy+3f7doqiKD2AOseAUtuRu1m2\n",
"5F6fjuM2g5BuuKPdtSOQlaTLY0PRVatPQQzYuu7+E8gq1D3pY0e7I3NAx4ByRceAeijlraCvRSry\n",
"++i4y+kKKltyL/I82wfZ+G85so3DpARpxrvo9ke2QrjNpXcH4uhwIJ05DNmv6R3EaF3miU9RFEUN\n",
"UDJKpfQOL6aLxIcjlXmU/1Kp1GcgrZqkz75J7QZhOJ33HXrTXY8zjI5dgPHnGsGkGFdamNACPJjQ\n",
"AqpgQgvwYEIL6IyOASnFIcSW3HHecc/F44kbt3KaIyLnIzxhFEVR1AAVhPYu7oXYkjvOHxHnhCOd\n",
"hsMRV+8/eMLe5dIrjwF9p470qtGeYlxp0R5agIf20AKq0B5agIf20AI6M/al0AryQg1Q8VmKGI4x\n",
"VBwB/oB0u1Wbf1O+tsQ9ezzwITI+c2/CdKPxfgQcAHwL2ab7HHf+kee5m4CHgReAyS49HZRXFKXH\n",
"U9SVEEyN4XVL7uJgPNearTzlhUkzMt0PKFfUC64HE9+S+1PAQ0EVKYqiKDVR1BZQd3wNeB/Zvvp5\n",
"YN8U4gyxJXdPoejlqSXQeUC5omU6BZrVACnNhZanHFADlCvaBdfCmNACPJjQAjyY0AI8mNACPJjQ\n",
"AqpgQgvwYEIL6IzOA1IURVEUJUW0C07JAy1POaBdcLnSMl1wpwFTgYXIPJHdugm/JfAksABZ+fl7\n",
"njCjgH+6OF8HTq5R0ywqc2r00KPRo7v9kxRFCcDhyOTIMcCmwDWI59V6VcKvhnh/3QGMRFZznguc\n",
"HQmzATAfWXBzU+BEKhMw49iG3yAbTGgBHkxoAR5MaAEeTGgBHkxoAVUwaUZmdR5QnmRSd+bdAjob\n",
"uAXZt+ZVZMmW95Dtn30cjWwxcBzwEjKrfhwdDdApSMvoDBfnz4FfIrP1m4WtQwvwoJqSoZqSU0Rd\n",
"BdQ0uS20grzI0wD1QzZWeyR2/RFkszMfOyNbTC+OhR9OZXHMnavEuT2yjUEzsEZoAR5UUzJUU3KK\n",
"qKuAmj4eEFpBXuRpgAYjBmF67PoHyCx/H0M94adH7gEMqRKmj0tTURRFKSBFd8Mu6phN2rSFFuCh\n",
"LbQAD22hBXhoCy3AQ1toAVVoCy3AQ1toAZ2ZsXZoBa1IP2Rl59Gx69ch2zb7+CWdl/zfAdmVs9wF\n",
"9yQwPhbmUMQRId4F9xrhPaT00EMPPZrteI0W4Cnghti1KcClVcKfAswB+keuXUDHHTevQJwPotwI\n",
"/L1+mYqiKEqrcRjiUDAG2BxxnZ5LxQ37cuCxSPjVEC+53wBbIK7Vc4CzImHagI+Bn7g4T3RpHJTR\n",
"OyiKoihNyqnIRNRFwCQ6TkS9BXgjFv5TSDfbQmQLaN9E1N2RiaiLkImoJ6UrWVEURVEURVEUpQCk\n",
"vazPwcgcog+QrsGngC8G1hRlN2AZ8GIBNPUDLkZarIuAN4HTA2s6BtkKfD7SdXsr4qZfC7Xo6g9M\n",
"cGkuobozzSgaWyoqbU15l/Ok+VQmj3KeVFOe5TypprzLuQHuB951ab4AnOAJ12g5byqyWNbnKuA8\n",
"ZBLrhsCFyD9Cd5VjlprKrIn8EzwE/Cuhniw1/RapuPYERiDeiaMCahoFLEdWxFgf2An5x3qM5NSq\n",
"axXgZ8i4433A454wtSwVlZemvMt5Ek1l8irnSTXlWc6TaApRzs9HjPDOyJj7KYhH85GRMI2W86bj\n",
"afwedZdVCX8qMJuOHnX/g3xNd5fOlQXQ9Fvkq/8iavsyzELTF1yYQTXoyFrTOcC02HMnIP9YWemK\n",
"Mh7/F+s4Ontp3gRMDKipWjpZlfNaNOVVzpNoyrucJ9EUupyXuRO4J3JeVzkv+kTUamS1rI+P1YCP\n",
"Ams6DVgbuAQoJdCStaYvIw4k5yAu8VOQL58kS4hkpelRJI8OQPJoMHAEybcYr0dXEhpZKiorTT6y\n",
"LOdJybOcJyHvcp6EopTz1elYXuoq581qgLJa1ifO15FK7taAmrZEuki+gkwIq4WsNG2IdNdsiTSx\n",
"vwHsg/Rfh9L0ApJHv0EM1Qfu+vEJNNWrKwmNLBWVlaY4WZfzJORdzpOQdzlPQhHK+QHAHsh8yzJ1\n",
"lfNmNUD1UGuhHg38ADiKjhNf06Q7Tf2Rpu45yOBnHiTJp17IahRHIV+IjyD/nKORr7MQmj6DVAxj\n",
"kS+8fZB/qHhXg9KRPMp5d4Qo50nIu5wnIXQ53xW4DXHEmNxoZH0alhOGmchAXNzzYwjiFeLjfTpb\n",
"+CGRe1EOQZYBOobkTdssNA0DNkPmR93irvdCmt5LgX3pevAxq3x6D/GIifY7v+J+jgBmBNB0FpIX\n",
"P3Ln/0YGRf+KDKK+24WmenUloZr2ZS7NEJrK5FXOuyNEOU9C3uU8CSHL+W5IOfkenQ1eXeW8WVtA\n",
"SxB3vy/Eru9F9UGvfwCfpeNA9l7I5NboV9dhwK+QPYh+G1jT28hE3K0ix/XIukxbuefz1gTwN6TL\n",
"JtoXvon72d0XbFaaSsjXapTyeZJyXo+uJPzDxRGPcxJSEYTQBPmW8+4IUc6TkHc5T0Kocr478EfE\n",
"OeQaz/1GynlTksWyPkcgX1ynI9a8fCT1gslCU5yx1OYdlIWmAcBbwF2IW/SuyJfYnQE1HYX8c52C\n",
"9N3vihT+SQk11aML5P23RlzEJyEVZnSTszYaWyoqC015l/MkmuKMJdtynkRT3uU8iaYQ5dwgraxx\n",
"SKumXF6i3ZBt9MAl0dJe1ucJxFqviB1dzVvIWlOci6htfkRWmjYBHkYK5tvAtSTzDspS06lUuiTe\n",
"QQbVh9egqR5dU6mUk+WRn1EaXSoqbU0hynmSfIqSRzlPoinvcp5EU97l/Bb85SWuXZdEUxRFURRF\n",
"URRFURRFURRFURRFURRFURRFURRFURRFURRFURRFURRFUQrLH6isIwbQjn/JkFqYAPy+wTiyYgVh\n",
"NvuaQON50obo37aLMNu7MCMaTEspMM26GKmixLF0XDX7y8hyM0kwyCoAg+m4x8np1LYvTRaMRVZf\n",
"3jKwjjLxfFaUulEDpBSJfsg6V2kwu45n4samll0mm4E+yOrEjRDaICstRLOuhq0Un3Zkf/urkVbF\n",
"R8i+M9EKbBqy5tcvgFlUNkTbBVn3rbz+1k+BVSPPrYJ0Bc1DloE/312Pxt2OrNtVph+y5fA0KmtV\n",
"nY7sqFpeA20G0u3zC3c+gY7dTf2Bq1yaC5EVgHeN3Dfu+T2QbY/nI+tsbRMJs7p7z+kujteBM/Bz\n",
"PLJJ2xZU1t86NnJ/LeBuZBHI14GjI/faXPgj3PstoLI21wnASy79V4Ez6Zh3JyO7fy5E8uQhKnWF\n",
"dWHPQP42HyH5tXLk+e7yycc+yFYHC4G/UFl1ukwt+aYoSg+nHVlh92qkMjkUaZVEV7CehqxqfQ6y\n",
"su9GSFfTPBduI2BHZJn4uyPP/RSp/PZCKue7XDy/iIR5go5jQL9BNlw7CKmcd0Mq7F7u2gpkT5p1\n",
"qBi7W4AHInFcjey3si+wKbIj5Dwq+6AYF89TwCgX5iGksi9zLfAcMsaxngt3CH5WAn4IvOx0rUNl\n",
"S4oV7n2OQvLuMmT14fKKxm0uzFRkrGh9YF3ga+4dytcOQFYa/7p7bnuk6/JIF9enkYq+vK3yBOTv\n",
"eIN7v72Qj4fv1JBPZW3lMaD1kI+CaFl5G1kAszwGVEu+KYrSw2mnsnlXmf+h466b04D7Y2F+Bfw8\n",
"dm1rpMIaDAxEKqsjI/cHIJVgNQO0sXs+vgdKGePux7cjmEClBTQAqeC/ErnfC9mz5vuxeKL7ouzi\n",
"rpVXK74fuLmKDh9j8W9LsAK4NHLeG2lxHeXO21yY+NYeb9GxpQTSAvqP+/1gxMAMrKJnArIXTrTF\n",
"dCPwqPs9ST6VtZUN0GX4y0rUCaHWfFOaAO2CU7LCIi2BKE8hX+EDI2Hi2/puh1Re8yLH31zYjdzR\n",
"j46blM2n671jtkEqsydqfYkIGwF9gb9Hrq1wOkbGwka3ESjvMrmO+/kz4HDgeaR1s3sDmqLpLEe6\n",
"y9aJhYnm79rAJ6i0SMrH5UgrCmTb6TeRltOvkS6/uDF6iY6OCO9F0q0ln8psjr+sREkz35SCoAZI\n",
"yZIkA9bzPc/cRMfdMT+NtGJeaDCtLPDtUBn1vitX1OX/tYeQrq8rkRbdg3RsudVC3MvP0vl/Opq/\n",
"5Xsn0zF/t3AHyHjStsimZW8h42uvINtml4k7MvjSjePLp+jz3f390sw3pSCoAVKyogTsFLv2GWQD\n",
"rY+7eO5ZZPO5NzxH2XlgKbBz5JkB7plqPI+U9T2q3C973vWuch+X7hI6btzV2+l4yftEdT5EWhcn\n",
"IDtHHoe0Gqpp60pXLUxHxmY+iT9/yyxHWosXIMZ/ALB/5H5Xbtj15NPL+MtKnFryTWkC1A1byZLh\n",
"iDfUzxDngnOojANUYxzS/fIzKl1FmyGD5acgxutmF24G0v1zIZ0/pkpUvqqnII4KP0cG1J9DuqLW\n",
"Ryq0N5FK9QBkQusCOrfM5jtN44CZyPjVWUi31k+7eacoFyO7Rr6E/P8dTMWo+pjqdG6DjJ/NpTFX\n",
"9YuQAf3ZwJ+QCnxb5G91BZIHGyGeaB8Bn0OcMl6OxNFVa6WefLoe+BYdy8rJsTC15puiKD2YJ5AK\n",
"51rEQeAjpO8+aiimAmd7nt0OqRznIAbnX8hgfJlVgF9SccP+H8RZoCsvuH5Ipfg20pJ6DTgtcv+7\n",
"SOtgeSSeuBdcP2TP+/ddHBMRJ4Myxj0fdWZoc9fKA+4XUNlO+UPE4G3qyYNomncj+Rd1w/athBDN\n",
"z3i6UY5AKvOFLt6/IF1uIO7SjyPGYwGS98dFno3nCXTePru7fPJp24+KG/ZfEWeKqBdcrfmmKEoP\n",
"Jm4AFEVROqBjQEpWRLvAFEVROqEGSMkKXTNMURRFURRFURRFURRFURRFURRFURRFURRFURRFURRF\n",
"URRFURQlPf4/OqsASwpPT4QAAAAASUVORK5CYII=\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"report.metrics_vs_cut(AMS, metric_label='AMS').plot()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.8"
}
},
"nbformat": 4,
"nbformat_minor": 0
}