{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#A Primer on Empirical Risk Minimization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Notations and Definitions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p>Let's first set up some notation and ideas:</p>\n",
"<ul>\n",
"<li>Let $X \\in \\mathbb{R}^p$ be a $p$-dimensional real valued vector of predictor variables</li>\n",
"<br>\n",
"<li>Let $Y$ be a target variable, where \n",
"<ul>\n",
"<li> $Y\\in \\mathbb{R}$ is real valued</li> \n",
"<li> $Y\\in C$ is an element in some set of classes $C=\\{c_1,c_2,...c_k\\}$</li> \n",
"</ul>\n",
"</li>\n",
"<br>\n",
"<li>$X$ and $Y$ are governed by a joint distribution $P(Y,X)$ (that we likely don't know)</li>\n",
"<br>\n",
"<li>We seek a function $f(X)$ for predicting $Y$, given $X$, whose output can be</li>\n",
"<ul>\n",
"<li> Real valued, i.e. $f(X)=E[Y|X]$</li> \n",
"<li> Discrete valued, i.e. $f(x) \\in \\{c_1,c_2,...c_k\\}$</li> \n",
"</ul>\n",
"</ul>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p>Second, let's define two more things</p>\n",
"<ul>\n",
"<br>\n",
"<li>$\\mathbb{F}$ is a family of functions, such that $f(x) \\in \\mathbb{F}$, examples are:\n",
"<ul>\n",
"<li> All linear hyper-planes, such that $f(x)=\\alpha+\\beta x$ </li>\n",
"<li> All quadratic polynomials, such that $f(x)=\\alpha+\\beta_1 x +\\beta_2x^2$ </li>\n",
"<li> All decision trees with max(depth)=$k$</li>\n",
"</ul>\n",
"</li>\n",
"<br>\n",
"<li> A loss function $\\mathbb{L}(f(X),Y)$ that measures how well $f(X)$ approximates $Y$.\n",
"<ul>\n",
"<li> Squared Loss: $\\mathbb{L}(f(x),y)=(f(x)-y)^2$</li>\n",
"<li> 0-1 Loss: $\\mathbb{L}(f(x),y)=\\mathbb{I}(f(x)==y)$</li>\n",
"<li> Logistic Loss: $\\mathbb{L}(f(x),y)=-[y*Ln(f(x))+(1-y)*Ln(1-f(x))]$</li>\n",
"<li> Hinge Loss: $\\mathbb{L}(f(x),y)=max(0,1-f(x)*y)$</li>\n",
"</ul>\n",
"</ul>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Emprical Risk Minimization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p>The main goal of Supervised Learning can be stated using the Empirical Risk Minimization framework of Statistical Learning.<br><br>\n",
"We are looking for a function <span style=\"color:red\">$f\\in \\mathbb{F}$</span> that minimizes the expected loss: \n",
"<br>\n",
"<br>\n",
"<center><span style=\"color:red\">$E[\\mathbb{L}(f(x),y)]=\\int \\mathbb{L}(f(x),y)\\, P(x,y)\\:\\mathrm{d}x\\mathrm{d}y$</span></center>\n",
"\n",
"<br>\n",
"Because we don't know the distribution <span style=\"color:red\">$P(X,Y)$</span>, we can't minimize the expected loss. However, we can minimize the empirical loss, or risk, by computing the average loss over our training data.<br><br> Thus, in Supervised Learning, we choose the function <span style=\"color:red\">$f(X)$</span> that minimizes the loss over training data:\n",
"<br><br>\n",
"<center><span style=\"color:red\">$f^{opt}= \\underset{f \\in \\mathbb{F}} {\\mathrm{argmin}} \\frac{1}{n} \\sum\\limits_{i=1}^n \\mathbb{L}(f(x_i),y_i)$</span></center>\n",
"\n",
"\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#Logistic Regression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Definition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p><b>Logistic Regression:</b> a member of the class of generalized linear models (glm) using the logit as its link function.<br><br>\n",
"\n",
"The goal of Logistic Regression is to model the posterior probability of membership in class <span style=\"color:red\">$c_i$</span> as a function of <span style=\"color:red\">$X$</span>. I.e.,\n",
"<br>\n",
"<br>\n",
"<br>\n",
"<center><span style=\"color:red\">$P(c_i|x)=f(x)=\\frac{1}{1+e^{-(\\alpha+\\beta x))}}$</span></center>\n",
"<br>\n",
"<br>\n",
"To make this a linear model in <span style=\"color:red\">$X$</span>, we take the log of the odds ratio of <span style=\"color:red\">$p$</span> (called the log-odds):\n",
"<br>\n",
"<br>\n",
"<br>\n",
"<center><span style=\"color:red\">$ln \\frac{P(c_i|x)}{1-P(c_i|x)} = ln \\frac{1}{e^{-(\\alpha+\\beta x))}} = \\alpha+\\beta x$</span></center>\n",
"<br>\n",
"<br>\n",
"And effectively we do a linear regression against the log-odds of <span style=\"color:red\">$P(c_i|x)$</span> (though we don't use least squares).\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##LogReg as ERM"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p>How do we fit Logistic Regression into the ERM framework?<br><br>\n",
"\n",
"We find the parameters <span style=\"color:red\">$\\alpha$</span> and <span style=\"color:red\">$\\beta$</span> using the method of <u>Maximum Likelihood Estimation</u>.<br><br>\n",
"\n",
"If we consider each observation to be an indepedent Bernoulli draw with <span style=\"color:red\">$p_i=P(y_i|x_i)$</span>, then the likelihood of each draw can be defined as: <span style=\"color:red\">$p_i^{y_i}(1-p_i)^{1-y_i}$</span>, with <span style=\"color:red\">$p_i$</span> given by the inverse logit function. In MLE, we wish to maximize the likelihood of observing the data as a function of the independent parameters of the model (i.e., <span style=\"color:red\">$\\alpha$</span> and <span style=\"color:red\">$\\beta$</span>). The total likelihood function looks like:<br><br>\n",
"\n",
"<center><span style=\"color:red\">$L(\\alpha,\\beta|X,Y)=\\prod\\limits_{i=1}^nP(x_i,y_i|\\alpha,\\beta)=\\prod\\limits_{i=1}^np_i^{y_i}(1-p_i)^{1-y_i}$</span></center>\n",
"<br><br>\n",
"This is actually a difficult equation to maximize directly, so we do a little trick. We take the negative log and call this our loss function for ERM!\n",
"\n",
"<center><span style=\"color:red\">$\\mathbb{L}(f(X),Y)=-Ln [L(\\alpha,\\beta|X,Y)]=-\\sum\\limits_{i=1}^n y_i\\,ln\\,(p_i)+(1-y_i)\\,ln\\,(1-p_i)$</span></center>\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Example 1 - Building and Looking at a Model"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Optimization terminated successfully.\n",
" Current function value: 0.253781\n",
" Iterations 9\n"
]
}
],
"source": [
"'''\n",
"Let's train an actual model and see how well it generalizes\n",
"'''\n",
"import math\n",
"from sklearn.metrics import confusion_matrix, roc_auc_score\n",
"from sklearn import linear_model\n",
"import numpy as np\n",
"import course_utils as bd\n",
"import pandas as pd\n",
"import statsmodels.api as sm\n",
"import matplotlib.pyplot as plt\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"reload(bd)\n",
"%matplotlib inline\n",
"\n",
"\n",
"#Load data and downsample for a 1/10 pos/neg ratio, then split into a train/test\n",
"f='/Users/briand/Desktop/ds course/datasets/ads_dataset_cut.txt'\n",
"target = 'y_buy'\n",
"tdat = pd.read_csv(f, header = 0, sep = '\\t')\n",
"moddat = bd.downSample(tdat, target, 10)\n",
"\n",
"#We know the dataset is sorted so we can just split by index\n",
"train_split = 0.75\n",
"train = moddat[:int(math.floor(moddat.shape[0]*train_split))]\n",
"test = moddat[int(math.floor(moddat.shape[0]*train_split)):]\n",
"\n",
"#Using Scikit-learn the model is built with two easy steps.\n",
"logreg = linear_model.LogisticRegression(C = 1e30)\n",
"logreg.fit(train.drop(target, 1), train[target])\n",
"\n",
"#But we are going to also build using the statsmodel package\n",
"logit_sm = sm.Logit(train[target], train.drop(target, 1))\n",
"lr_fit = logit_sm.fit()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p>Logistic Regression has long been used as a tool in statistics and econometrics so there are a lot of additional data points one can get out of logistic regression model than one might get with standard machine learning tools.<br><br>\n",
"We showed how to use scikit-learn to fit a model, but we also used statsmodel. The reason is that statsmodel returns summary statistics on each coefficient fit to the variables. In machine learning, we often only focus on the generalizability of the prediction. But in many analytical applications we want to know how statistically significant are the estimates within our model.\n",
"</p>"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<table class=\"simpletable\">\n",
"<caption>Logit Regression Results</caption>\n",
"<tr>\n",
" <th>Dep. Variable:</th> <td>y_buy</td> <th> No. Observations: </th> <td> 2087</td> \n",
"</tr>\n",
"<tr>\n",
" <th>Model:</th> <td>Logit</td> <th> Df Residuals: </th> <td> 2074</td> \n",
"</tr>\n",
"<tr>\n",
" <th>Method:</th> <td>MLE</td> <th> Df Model: </th> <td> 12</td> \n",
"</tr>\n",
"<tr>\n",
" <th>Date:</th> <td>Tue, 10 Nov 2015</td> <th> Pseudo R-squ.: </th> <td>0.1523</td> \n",
"</tr>\n",
"<tr>\n",
" <th>Time:</th> <td>20:05:51</td> <th> Log-Likelihood: </th> <td> -529.64</td> \n",
"</tr>\n",
"<tr>\n",
" <th>converged:</th> <td>True</td> <th> LL-Null: </th> <td> -624.83</td> \n",
"</tr>\n",
"<tr>\n",
" <th> </th> <td> </td> <th> LLR p-value: </th> <td>3.151e-34</td>\n",
"</tr>\n",
"</table>\n",
"<table class=\"simpletable\">\n",
"<tr>\n",
" <td></td> <th>coef</th> <th>std err</th> <th>z</th> <th>P>|z|</th> <th>[95.0% Conf. Int.]</th> \n",
"</tr>\n",
"<tr>\n",
" <th>isbuyer</th> <td> 0.9714</td> <td> 0.395</td> <td> 2.462</td> <td> 0.014</td> <td> 0.198 1.745</td>\n",
"</tr>\n",
"<tr>\n",
" <th>buy_freq</th> <td> -0.1588</td> <td> 0.203</td> <td> -0.783</td> <td> 0.433</td> <td> -0.556 0.239</td>\n",
"</tr>\n",
"<tr>\n",
" <th>visit_freq</th> <td> 0.0311</td> <td> 0.021</td> <td> 1.453</td> <td> 0.146</td> <td> -0.011 0.073</td>\n",
"</tr>\n",
"<tr>\n",
" <th>buy_interval</th> <td> -0.0030</td> <td> 0.014</td> <td> -0.208</td> <td> 0.835</td> <td> -0.031 0.025</td>\n",
"</tr>\n",
"<tr>\n",
" <th>sv_interval</th> <td> 0.0085</td> <td> 0.007</td> <td> 1.210</td> <td> 0.226</td> <td> -0.005 0.022</td>\n",
"</tr>\n",
"<tr>\n",
" <th>expected_time_buy</th> <td> 0.0101</td> <td> 0.011</td> <td> 0.935</td> <td> 0.350</td> <td> -0.011 0.031</td>\n",
"</tr>\n",
"<tr>\n",
" <th>expected_time_visit</th> <td> -0.0328</td> <td> 0.007</td> <td> -4.434</td> <td> 0.000</td> <td> -0.047 -0.018</td>\n",
"</tr>\n",
"<tr>\n",
" <th>last_buy</th> <td> 0.0115</td> <td> 0.005</td> <td> 2.284</td> <td> 0.022</td> <td> 0.002 0.021</td>\n",
"</tr>\n",
"<tr>\n",
" <th>last_visit</th> <td> -0.0597</td> <td> 0.006</td> <td> -10.295</td> <td> 0.000</td> <td> -0.071 -0.048</td>\n",
"</tr>\n",
"<tr>\n",
" <th>multiple_buy</th> <td> 1.4701</td> <td> 1.100</td> <td> 1.337</td> <td> 0.181</td> <td> -0.686 3.626</td>\n",
"</tr>\n",
"<tr>\n",
" <th>multiple_visit</th> <td> -0.1800</td> <td> 0.220</td> <td> -0.819</td> <td> 0.413</td> <td> -0.611 0.251</td>\n",
"</tr>\n",
"<tr>\n",
" <th>uniq_urls</th> <td> -0.0105</td> <td> 0.002</td> <td> -6.466</td> <td> 0.000</td> <td> -0.014 -0.007</td>\n",
"</tr>\n",
"<tr>\n",
" <th>num_checkins</th> <td>-2.867e-05</td> <td> 8.81e-05</td> <td> -0.325</td> <td> 0.745</td> <td> -0.000 0.000</td>\n",
"</tr>\n",
"</table>"
],
"text/plain": [
"<class 'statsmodels.iolib.summary.Summary'>\n",
"\"\"\"\n",
" Logit Regression Results \n",
"==============================================================================\n",
"Dep. Variable: y_buy No. Observations: 2087\n",
"Model: Logit Df Residuals: 2074\n",
"Method: MLE Df Model: 12\n",
"Date: Tue, 10 Nov 2015 Pseudo R-squ.: 0.1523\n",
"Time: 20:05:51 Log-Likelihood: -529.64\n",
"converged: True LL-Null: -624.83\n",
" LLR p-value: 3.151e-34\n",
"=======================================================================================\n",
" coef std err z P>|z| [95.0% Conf. Int.]\n",
"---------------------------------------------------------------------------------------\n",
"isbuyer 0.9714 0.395 2.462 0.014 0.198 1.745\n",
"buy_freq -0.1588 0.203 -0.783 0.433 -0.556 0.239\n",
"visit_freq 0.0311 0.021 1.453 0.146 -0.011 0.073\n",
"buy_interval -0.0030 0.014 -0.208 0.835 -0.031 0.025\n",
"sv_interval 0.0085 0.007 1.210 0.226 -0.005 0.022\n",
"expected_time_buy 0.0101 0.011 0.935 0.350 -0.011 0.031\n",
"expected_time_visit -0.0328 0.007 -4.434 0.000 -0.047 -0.018\n",
"last_buy 0.0115 0.005 2.284 0.022 0.002 0.021\n",
"last_visit -0.0597 0.006 -10.295 0.000 -0.071 -0.048\n",
"multiple_buy 1.4701 1.100 1.337 0.181 -0.686 3.626\n",
"multiple_visit -0.1800 0.220 -0.819 0.413 -0.611 0.251\n",
"uniq_urls -0.0105 0.002 -6.466 0.000 -0.014 -0.007\n",
"num_checkins -2.867e-05 8.81e-05 -0.325 0.745 -0.000 0.000\n",
"=======================================================================================\n",
"\"\"\""
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Use statsmodel if you want to understand the fit statistics of the LR model\n",
"lr_fit.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p>\n",
"<b>A Practical Aside</b><br><br>\n",
"\n",
"What exactly does the estimate of $\\beta$ really mean? How can we interpret it?<br><br>\n",
"\n",
"Recall that $Ln \\frac{p}{1-p}=\\alpha+\\beta x$. This means that a unit change in the value of $x$ changes the log-odds by the value of $\\beta$. This is a mathematical statement that IMHO does not offer much intuitive value.<br><br>\n",
"\n",
"<center><span style=\"color:red\">So what can we learn by looking at betas? (IMHO, not much!)</span></center><br><Br>\n",
"<b>Some helpful tips, garnered from theory and experience:</b>\n",
"<ul>\n",
"<li>$|\\beta_1|>|\\beta_2|$ does not guarantee that feature $X_1$ is more predictive than $X_2$. The magnitude of $\\beta$ is inversely proportional to the scale of $X$, so comparing betas only makes sense when the features have the same scale (such as binary features).</li><br>\n",
"<li>Likewise, the z-score of $\\beta$ is influences by sample size and should not be used to rank features by predictiveness</li><br>\n",
"<li>sign($\\beta$) does tell you whether $Y$ is positively or negatively correlated with $X$. However, if the features have a lot of multi-collinearity, sign($\\beta$) can be misleading.</li><br>\n",
"<li>Multi-collinearity in $X$ means the betas will have covariance with each other. The betas will \"split\" the effect. Sometimes they'll split the effect as positive numbers (i.e. 1=0.5+0.5) and other times they'll split as negatives (i.e., 1=2-1). This makes interpreting $\\beta$ that much more difficult.\n",
"</ul>\n",
"\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Example 2 - Robustness"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p>In this example we test the sensitivity of out-of-sample performance to training set sample size. Our goal is to plot test-set $AUC$ as a function of $N$, the number of samples in the training set. Because we expect a lot of variance in the lower range of $N$, we use bootstrap algorithm to compute standard errors of AUC measurements.<br><br>\n",
"\n",
"<b>To Bootstrap:</b>\n",
"<ul>\n",
" <li>Generate $k$ training sets of size $N$, where we sample from original training set with replacement.</li>\n",
" <li>Compute and store the AUC on the test set for each bootstrapped sample</li>\n",
" <li>Get Mean and Standard Error of bootstrapped AUC estimates</li>\n",
"</ul>\n",
"\n",
"\n",
"</p>"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.text.Text at 0x10c833f90>"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAtQAAAIHCAYAAAC/h8txAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XeYZFWd//F3TU9imBkGGHKYAQElIyg5lIJIlAVBSSoK\nyurqGlAUw9rmtO66K66yyooJUAEFRFBBBpQoKkkQGGAkh4EhZ7i/P763fnW7pqq7qivcqtvv1/Pc\np8K9de451T3Tnzp17jkgSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkqaAWAbvmXYkB8Hng\nQeAeYC3gcaDU4Nhh4Ee9qVbbfg28ZZyv/TbwyQ7WRZIk5WwR8BQRdB4GfgWs2YFy5wMvAZOaPP4l\nYN0OnLdXbgdeO8Yxw0S7tq7zfL3gWPsevB64GHgMeABYAOzbck3HtjUREJcADwFXAEd0oNy1id+t\nFZs8/tP0V6A+AvhDl89RBu7s8jkkdUmzf+AkFV8C7APMAlYD7ge+2cHyG/VGtnrs5HYr0mMl4K3A\ndeltVtLE6w8EfgacBKwBrAz8G80F6vlE4G/GdsAFwIXAy4jw+25gjyZfP5q1iYD+UAfKkiRJ6lu1\nPa17ATdlHi8H/JDoIV0EfIJq8C0RX10vIoL4D4DZ6b47iB7Xx9NtG2A94CLgEWIYwCnpsRenxz6R\nHnsQ0XN3F3AscG9a9hyiB/0Bojf9bCJsViwAvkT0sD4K/BJYPt03Pz3HO4G7iSEIx2ReWwI+BiwE\nFgM/zbwW4uv9f6T7Pl7nfau1c3psOb2dktk3zOg91CXi/TumzjHNmE/zgfqPjP0B6p3ALUQwPpP4\n4FXxEnA0cDPRw318+vxuRO/0i8TP9P9Y+luLdYjfh8eA36b1yL4v2wKXpuVeDeyS2bcA+Gxa/8eA\n3zCyJ3zHzGvvAN6WPj8N+HfiZ3kfMfxieoN2H0HjHuoFwJGZ4y4B/iM930Jge+Dt6bnvZ+SHqpOA\nzwEzgKepvkePAasS3xhcRfwO3wd8vUEdJElSn7id6ljgGURwPSmz/4fAL4BlgXlE2H5Huu8dRNCa\nn+4/PT2e9NjaIR+nAMel96cSoaOidrhDGXieCMhTiNCzArB/en8m0YP7i8xrFhAhfKO0LadRDWjz\n03P8BFgG2IQI5pW2v58IYKun5/sOcHK6byMi8OyY1vvrad1GC9QnAt9N798JHJDZN8zogfoV6f15\no5Q/mvk0F6hnAC8wMqjWei3x4WcLou3/TYTgipeAs4gPUmsR7+nr0327MHI4w3xG/k5cRoTbKcBO\nRKCs/P6sQXwQqfSU75Y+roTmBcTv3nrE78OFxO8KxPv2GPBmYIj4vdk83fefxAetOcTv0FnAFxu0\n/QgaB+oLqf47OIL4fXgb8WHoc8Tv4TfTtr0urc+M9PjvEx8GYOn3COJ9OSy9P4P4MCpJkvrYIiIs\nLgGeI4LAJum+IeBZIuBVvIsIExBDBf45s2+DtIxJ1B9D/QPgBEb2KlfUC9TPEiGukS2InuqKCxkZ\njjZMyyhl6rNBZv9XgO+l929kZEBeLW3LEDHU4uTMvhlpuY0C9Qyid3H39PE3iBBXMczogXqH9P5o\nbR/NfJoL1Guw9HtS60Tgy5nHyxLvy9rp45cY+cHop8BH0/tlGgfqtYkQukxm/0+oBuqPZu5XnEe1\np/dC4puCincD56b3jyM+3NUqEd+CZH/PtgNuq3MstBaob87s25Ro50qZ5xYDm6X3v0+Ebqg/hvoi\n4ndkboNzS+oTjqGWVJEA+xHDG6YB7yP+oK9M/EGfQnw9XnEH1UC8Wp19k4FVGpzrWCLUXAlcT3wl\nPpoHifBWMYMI5IuIwHoRMSQlO/Y6G07uSOs/d5T9q6f35xG93UvS7Qai93YVop13ZV73FKOPC96f\nCIsXpI9/DuxJtXf1eUYOASHz+PlM2avRvEMzdb+GCKyVxw9T/0LTJUTwG+08tT/jJ9P6ZT8U3Ze5\n/xTR8zuW1dPzP5157h9Uf5bziKE/SzLbDsSQiHrnfTpz3rWoH5JXIn6H/pwp81w6E1zvr6kLxO9v\nvfqN5UjiQ86NxL+VvduunaSuMFBLqichQuWLxPCGxUTAm585Zm2q4fKeOvteIMJFvQvv7id6uNcg\nxt3+D6PP7FFbxjFE0NiaCNK7EAEsG6jXrrn/fNqORvvvTu/fQQwvWD6zzSDaeC8R0ipmMPrMFW8j\nLvK8K33t6URgrnyNfwcj3zeI8cQvpPW5iQj+B45yjlonZ+q9WXqOyuMVGPmBoOIpYnjBaOep/Rkv\nS7T97rpHh2YuuryX6ntcMS/z2juIXvzsz2MW8NUmyr6DuMCy1mIi2G6UKXMO1XH/vZTU3GYtJD4g\nrUR8i3IaI3vyJfUJA7WkrOxFhpXe6huJYP0z4AtE79o84IPAj9PjT0kfz0/3fxE4lej1fDC9zQab\ng6j2lD5ChImX0sf3Uz8EZc0kAtGjREj8dJ12HE4M9ZhBjFP9OSNDyyeJcLIx8VX9T9Pnv5PWvxK4\nVwLekN4/jZgJZQdiGMZnafz/6BrEUJC9iXG7le0rVIcr/IYYRnM4EbRXSM99GvF+JMCHgE+ldZyd\nnm9Hood+LK3MrHJseo4PU/2QsDnVC0ZPIb5J2Jz4BuOLwOVEaG107mbO/w/iwrvPEO/BjsR7XPFj\nYkaT3YlhN9OJ4RHZnvFG5zmZGHN9EPGNyYpp/V8ixrV/g+pwjDWoDs1p1J5p6fkrW7uy79H9af2y\nof7wTP0eZeS/E0mS1IdupzoP9WPAtcAhmf1ziJ7CB4gQ9UlGBvBPpc8/QIx5XS7z2s9QnZFjGyJU\n3pWeayFwVObYo4ne0CVEj+kuLB3aViPGrj4O/J3o7X6RaritjKGuzPJxJhFWoTp+9yiid/VeIkRW\nlIgPB39P34eFxKIkFW9l5Cwft1F/DPXHgD/VeX41Ytz1Runj7YjxuQ+n9flfRr53UJ2H+nHiffw9\nMXRkLPNpPC64nlcT81A/QgznuJwIdRVHE+/HQ8RFfKtn9r3IyG8ZshfclRn5M5zPyJ/XOlTb91vi\ngsfsuOmtiYsPHyLafzbVD2TZMcwQ3wpcnHm8Y9qOR9M6VBZhmUZ8QLw13XcD8F7qexvxO5PdXiQC\nfvb8tedeLz0u606qY82z7xHEOPXFxO/CasS/t/uJ9+U6qh/sJEmSuq42ZGXNp7WFZiRJGpV/UCQV\nVSvDHSRJGjcDtaSiGu2CuGYulpMkSZIkSZIkSZIkSZJasBMxy0ivvRy4mpjdpNGMF4PqJKqrEkpS\nRzmGWpKWtojqFIKVaQRXHe0Fbapdbv0PjFzmvVeOJVZ1nA0cX2f/AmL+7+xqi7sx+vLm+xEh/VFi\nTvILWHoxm15IcOy8pC4xUEvS0hJicZFZ6Tabkctbd0M/zEoyj5iPeTRPEnOON2M94AfEvN7LEfNN\nf4ul52bulX54jyUVkIFaUr/7L2JBjkeJFfV2zOzbOn3uUSLwfr1BGXOAX1FdXOZsRq6016xFwK6Z\nx8PE4htQnd+6svDLg8TCLxWT0scLiR7vPxE9vZWFQK4hesMPIhZCuTPz2g2J3uElwPXEyoEVJxEh\n9VdpuZcz+jLubwD+lpZ1IdWe8N+n5z0+LWe9Oq9NiEVXDhnjHBVbEL3XF6aPnwDOoNq2rYklz5cQ\ni/l8k1gtseIl4N3ALWmdPkusonkZsfjMqZnjy8RiQccR7/3txLLdjexD9JwvAS4BNs3s+2ha1mPE\n0Jt6C/dIkiQNjMOIJdAnEctw30ss+w0RrA5L788gVmGsZwVgf2K56JnEMuq/GOWctzMyOGefz4ar\nT7N0oD6BWIVvM+AZYlwywEeI1SfXTx9vRnX1xtohH2WqoXMKEcI/Riyf/Roi6G2Q7j+JWF3vVcTK\nfT+mulx4rQ2IULtreuxHiLA6Od0/2oI4lf1HEh9cKu0ebcjHOsQQkf9I2zSzZv+WRKieRLV3/P2Z\n/S8RP6eZxMqSzxLBfz7xrcHfqC7jXgaeB/6deM92Tttaeb+/T3UM9SuJFQhfTfRavzVtwxTi53UH\n1SE+a9PchwdJkqSB8TDV3sSLiF7iuS2WsUVaTiOLiN7iJel2Rvp8baAeZulAnV2O+wrgTen9mxjZ\ns5w1WqDeifgQkXUyEeYhAvX/ZvbtCdzY4DyfInp1K0pET+zO6eNKYG6kErjnEj3EGzH2GOptgJ8S\n3w48TQTbZRsc+wGq7zXE+7Jd5vFVxIeAin8H/jO9XyYC9TKZ/T8FPpnezy7z/W1GLvkN0RO9M9ED\nfj/xoWMKktQEh3xI6ncfJnouHyHC7XJUA/SRRK/rjcCVwN4NyphB9BwvIoaHXJSW02hMbUJcTLd8\nuh3QQn2zY62fotoruyZwawvlVKzOyOEfEENKKsE9IQJgxdMs3RNcsRrR+1qRpGWvUfPcWBYTQ0M+\n28TxVwBvBlYmPhzsDHwi3bcBMVTlXuLn8gVgxZrX17Yt+/gZRrZ1SXpMxT+INteaBxxD9QPTEuLn\nsxrxM/oA8WHpfqK3v14ZkvT/Gagl9bOdiB7Jg4hx0MsTwasShBcS42RXAr4CnMbIHsqKY4jwtjUR\npHdJy2j1IrUnGdm72srMH3dSf1zyWO4B1mJkXecBd4+zrHmZx6W07PGU9TVi+MlWLbzmKmIIx8bp\n428TH5bWI34un6C1v0u1YX554sNTxTyizbXuIML78pltJtGjDRGid0pfnxC/W5LUkIFaUj+bBbxA\n9IhOBf6NGDtbcTgRpiGCdkIME6g1k+i5fJQYt/zpOsc042rgYGLM8auAN9L8VGzfI8bwrkcE2ewY\n6vuJoQb1XEH0dB9LDEEoExfUVYZutPKh4GdEL/5r07KOIXp5L80cM1Z5lf2PEmOpPzrKsTsAR1H9\nGb2CGPZyefp4JjG05ql037ubaEOpwf2KzxBt24lo688zx1aO/y7wz8QHrBLxIWnvtD4bEO/PNGLM\n9jPkNyuJpAFhoJbUz85Lt5uJ4RpPM3LIwuuJWS8eJ8bSHkyEoFrfIHquFxPh8VzGNyfxp4jgu4QY\nEvCTmv2jlfkfRKD9LRFGv0tcJEla1g/Scg9k5JzJzxEhdE9i9orjgbcQ7wnUn1+5UT1uJj6EfDMt\na++07BeabEPt/v9KX9voNY8Qs4pcR/yMziXGSH813f9h4huGx4hx4KfWlFWv3Nr92cf3UZ0x5EfA\n0dR/n/4MvJN4Lx8mLsysXNw4DfgS8f7cSwwvOq5B+ySpJ/YgLvS4hfq9GMsTX/9dQ/TCbFznGEmS\nxlJm6bHmkjTwhojxjfOJr9+uJuZSzfoa1QUCXg6c36vKSZIKpYyBWlJOujnkY2siUC8ipjI6lbhq\nPmtDqhP+30SE75WQJKl1Li0uKRfdDNRrMLK34C6WXpnsGqrTUW1NXFG9ZhfrJEkqpgXEIiyS1HOT\nxz5k3JrpKfgycVHLX4mLVv5K/aupF9L4CnhJkiSpU25lfNOcdsW2xNX5Fccx+vRKEKtt1VuQwK/x\nOms47woUyHDeFSiY4bwrUDDDeVegQIbzrkDBDOddgYIZzrsCBdNy7uzmkI+rgPWJcdFTiZWyzqo5\nZrl0H8QURhcBT3SxTpIkSVJHdXPIxwvAe4HfEDN+nEgsD3x0uv8EYCPgJOKTwPXEMsKSJEmSOswh\nH51VzrsCBVLOuwIFU867AgVTzrsCBVLOuwIFU867AgVTzrsCBVPY3FnYhkmSJKmv9NUYakmSJKnw\nDNSSJElSGwzUkiRJUhsM1JIkSVIbDNSSJElSGwzUkiRJUhsM1JIkSVIbDNSSJElSGwzUkiRJUhsM\n1JIkSVIbDNSSJElSGwzUkiRJUhsM1JIkSVIbDNSSJElSGwzUkiRJUhsM1JIkSVIbDNSSJElSGwzU\nkiRJUhsM1JIkSVIbDNSSJElSGwzUkiRJUhsM1JIkSVIbDNSSJElSGwzUkiRJUhsM1JIkSVIbDNSS\nJElSGwzUkiRJUhsM1JIkSRNT2bIB2An4NPA/wCHjKWByR6sjSZI0fmVgQc51mEjKdO/9zrPsLYBd\ngFUy28rAD4Bv1Tl+OWBdYCXGmY0N1JIkqRVlBjOEdVOZ/n5PlgHWJILlqplt/QbH7wUcSoxkyG6/\nAk6qc/xBwD/XHDsPWAwcX+f4A4GjgAR4KXP7C+D7dY7fJ61P5bjNge2AsxuUvxawHnA/cHl6+wBw\na4P2/irdKt7Z4LiGDNSSJKkVZRoHvMnAi0TwqfU6YFki3M3I3H49fU2tWcDj7VW1Z8r0PlBPZ+mA\nfDvw2zrHHg58DLgPeIF4759My74xPWZB5jyLgPOI8Jrdbm5Qxz8BS4BNiLCbADsSgX24pmyAPwNP\nEMG7lLm9rUH5t6bbxunjTYFridBcZun35+x06xkDtSRJGo8/ASsyMiAPEV+fP1Hn+H8lQtnTwFOZ\n29cQ4QtiHCtEwPpQWs7N6XZTensW9QP7WMq0H3pLwBxgdWCNtH6X1jluc+DNRD2zvbDXAqfXOX4j\n4A11jr+xzrEAbwG+R/S83pe5fbjB8d9Nt1rD6VbrhnRr1qJ0Oz/z3B0NyoYI/re3UP6NwKcyj4dH\nKTsXBmpJkvJRpv+GN0wCXg5sk27bAvsR40vL6TGV0HslcAnRhkpAfo7GYXffUc6bDWLDmfOsntZn\ng/R2C+DMOq+fDmxPBO67G9ShzOjv9zRgJvBQnX2vBU4gQvTz6TmeIoY0XE71PSE9x4NED3Aps1V6\nYeuZCiyfOXbtdLsX2L+m7AXAKcCPGd8HC01g/sJIkopmOO8K1DgeeIT4av1k4P1EoJ5ac9xwF+sw\n3rLXBC4iAugTwF+BU4EPNCh7A+B/gXOAq4kA/Fz6mnrmEIF+ZoP9ww2e74Rull227Lpazp32UEuS\nuq1Mf1+wlVf5k4nhEY8TX/F3Upml6z0deCUROhfVec13gM8Q4TIvC8b5uruIWR0g3tP1iQC8KSN7\nvCuuB/5CXIh2d7o9QOOfwyPpVjQLLLszDNSSJBjc0NvNslstfx1gN2AuMbZ4brpdDnw+U145vf8J\n4MNED3Bl+MCjRC/pl+qUvx6wGRHsHqUa8h4lLjSrrfctxPjkyvCNTYhxyJ+kfqC+vsl2LmjyuPHo\nRNmPAlelW63hDpRfz4IuldvtstUhBmpJ6qwyg/kHsEzz9Z5E9AJWAuPz1A8vuxBfq68J/AvV8aO/\nJabAqrUnMTa0clzl9jzgTXWO34sIpR8gvq5/Dng2Lf/ddY7fGjg2c1zlNVcDJ9Y5fj3iYrmvZ9q6\nInAFMRyi1qpEcF1MXCD2t/T+wswxCxj5Pg8TF/LNJoYVNLqgD2A+MVvDnJrtZOC9dY7fjZhu7Arg\nZ8TMCk81KLsVCzpQRtEsGNCy1SEGaknqrDKD9wdwiAhmryJC43PA7+sctwNwBrACccHV4nS7kKUD\ndRl4PfA7IkxXFlO4JD2+nvOJ0AjVMYwJ9Xtfy0Swn0wMVRgiLpL7CxGW67kL+CnRI5zdFjcof2Vg\nV2IYwFPAH4mf7V0Nyr8s3Vr1IjHl2JIxjjufkRfv1Soz8sLBzxCh/irgD+OoV9EsyLsCUt68KFFF\nV867AhNMuYNlbQV8gViM4LdEb+S/E0vZ9rvDiKmtKjMz3EPU/wcNjl+G6IWtvUhtLMPjrF/eZXe7\n/HIXyx7uYtlS0XlRojSgyth70ktlGr/fc4kLtyrzzK6R3r8c+Eqd46cCqxG9tg8Qi1dskb5mqM55\n5hNDJO6h+50FyxJjbtchhgXUOpO4GOtqYtjE8BjlPZ1u6owFeVdAUmcYqKV8lYnVq14k/j3WfrWt\nzppDdW7dg6k/RdargI8Qgfdu4iKu3xNfnddT+zX/QkYPpgcRF6JNIRZ5qGzn0XgoQUWZ0UPYEPBB\n4gPBK4nwfgMxFOIUlg7wT4xRXqd08xzdLLsX5XfLgrwrIKn/OORDRVQmQk5l1a/7gP9jsIZ/lPOu\nQBPmExfG3U6MrV1EvN+nEcG33OHzDTd53CrERWMfAk6i8RCRFYmxwtmyV6Nxh8iXgCOIldqmNFkX\nGIyfpST1QmFzZ2EbJqWGiWVq7wT+J9+qtGQ47wpkNBrXuwoxA8JWVAPmcBfrUe5weWcT8xRfRkyD\ndi9xEd26HT6PJCk4hlrqU+sSwwcazT4AMfvAOcCWPalRe2YR03et3sVzlGn8tXUJmEcsNbw9sB3V\nsc61CzPcT6wA1ysLOlze14nhJisDbwe+QcyzuzZwW4fPJUkqMHuoNahWI6YLe4gYu9tIuSe1ad8a\nxMVtTxFjcxOit3eYkW34KjFN10+B/yTGJB+avr5Zww2eLxHjmu8FTgeOIQL1tBbKLo95RH8azrsC\nkjQB2EMt9YnliQUk3kWMj305S891m7VgjPJKRO/r3R2o23isBnwZ2Bf4MbHi2m1Uw3StbxK97atn\nti2J8cv12vCvRNi+J93uBTYEVmLpZZATYGdiRo3xftheMM7XSZK0FAO11HnrEiuTnUlMn3ZnB8rc\nCLgY+C/ga/R+6rKniGEHH2DsxScg2txKu28m5jjeiQjSs4nV9R4kPogsYGQIvr+FsotkQd4VkCQN\nLod8aJCU6M4FY/OImSluA/ajuixznspdLHu4i2VLktRIYXNnYRvWhnLeFVBuXgfcCJxLLELSKSsC\nnwT27GCZ7RjOuwKSpAmp5dw5aexD1KfKeVdggisB+wBH5XDu3xFzDJ9BzPbQrvWICycXEj3r/TJz\nxIK8KyBJUpHYQz3SNCJs7ENneygHXblH5e8M/JEYU7x3l8/ZTasAvyDGKX8eWDXf6kiS1BcKmzsL\n27AxDDHyW4Qy1VkVEuBWYl7jO4iZJI7sZeX60HCXy/8OMcziduCtxM+nHy1b87jc4LipxCwktcdL\nkjSROW3egJtKrOa2M7ALsWDFLsA16f4FjPwafJgIdRsT8/DO6001+9bOwO+JfwgJscBHQiyEcW6d\n4/8F2LXO8Sek5dTamlga/J8YfYGWPJWAS4nfk08Dj9B4gZTniCW5JUlSGwzU/eMrwHuIpYUvBr4H\nHEHMtTuaF4Fr062R3Ynp1i7LbDew9IpyFWUGZ/xqmWoP7GuAHxKh8hrgOqKH/28NXnsZMd/xpPQ1\npfT+PxqU/0piiM1xLP3hpl8kxIeELwB/Jz507Ub0qv8gv2pJkqS8DeqQj3LN49k0Hqf6MmBOG2WP\nZjIRBt8D/Ii4+OwRIhjWM9xC2b1UAnYgll+uZ7jL5+92+Z1UJnraF1LtpR/Gi1klSRrLoObOMQ1q\nw74K7E8svfxn4HHgg7nWqGplYH6Dfd8iFhLpl1lgXkaEwVuJnvX3NzhuuMv16Hb53TKcdwUkSRog\ng5o7xzSIDdsbeAb4NfAxYjz01FxrNLoyIy94fJhYje/K9Lk9gFk9rlMJOJ9YFe+/iPHloy1mUu5y\nfbpdfrcM510BSZIGyCDmzqYMUsPKRID5HFHvYQbvq/bh9HYV4A3AF4ELiTHKvbYVMCWH8xZJOe8K\nSJI0QAYpd7aknxu2GjC9wb7hHtajk4ZbPP7bxDzGezP2vNjlmsclYBtijLckSVLeXCmxh7YgZk24\ngeKFwQUtHn8m8cv3AWKc883EbBvL1Dm2nN7OJ5a5/jtxoeTLWq+mJEmSmtUvPdSTiNUJLwDuIsZG\nrzDK8eUe1KnfDAGbEFP+1Rvv/A3gImAxcfHjtg2OkyRJykO/5M6O65eG7U7M1nEY/X2BYb8pM/KC\nx1OJMeblvCokSZLUQL/kzo7rl4ZVFv/Q+A3nXQFJkqRROIa6Q7ag/lCOyhLVkiRJ0kDpZogtp7eT\ngH2J6eHuBLbr4jknsnLeFZAkSRpF33We7kHM4nAL8NE6++cC5wFXA9cTF7LV082GfQF4NzEzxVXA\noTjvsSRJ0kTVV4F6CFhITI82hQjNG9YcMwx8Kb0/F3gImFynrHYaNgVYo865K/4X+AWwE46PliRJ\nmuhazp31wmunbE0E6kXp41OB/YAbM8fcC2yW3p9NBOoXmih7ErAcsKTOvpWAU4BV02028CBwLbBn\nekyZ6tCDdwKfAXYlPgQsaOL8kiRJUtcdCHw38/hw4Js1x0wiAuw9wONUA2+tBDiHmLLubuB5YgGR\neqYCrwM2JcL1WBdeDo+xX5IkSRNHX/VQN1OZjxNDQcrESnm/AzYnwnWttYHngEuIYRrnNyjzubQc\nSZIkaSxl+njShG2JCw4rjmPpCxN/DeyQeXwB8Ko6ZfVilg9JkiSpry5KnEwMy5hPDMOod1HifwCf\nTu+vQizn3Wj+Z0mSJKnb+i537gncRFyceFz63NHpBjGzx9nANcB1xJR19fRdwyRJklRIhc2dhW2Y\nJEmS+opLj0uSJEm9ZKCWJEmS2mCgliRJktpgoJYkSZLaYKCWJEmS2mCgliRJktpgoJYkSZLaYKCW\nJEmS2mCgliRJktpgoJYkSZLaYKCWJEmSAEhK43mVgVqSJEkTXDIVkqOBG/OuSTcleVdAkiRJRZNM\nhuRtkNwGyW8h2YYC587CNkySJEm9lkyC5GBI/g7JxZDskt2ZW7W6rLANkyRJUq8kJUj+CZJrIbkC\nkt3rjJsubO4sbMMkSZLUbUkJkj0g+RMkV0Oy7ygXIBY2dxa2YZIkSeqmpAzJHyG5AZKDYrjH6C/o\nRa3yUNiGSZIkqRuS7SA5H5KFkBwOyVCzL+xqtXJU2IZJkiSpk5ItIfkVJHdAchQkU1otoCvV6gOF\nbZgkSZI6IdkEktMhuQeS90IybbwFdbRafaSwDZMkSVI7kvUh+Qkk90NyDCQz2i2wI9XqQ4VtmCRJ\nksYjmQ/JiZA8CMknIJnVqYI7VE7fKWzDJEmS1IpkDUi+BclDkHwWkjmdPkGrL5jc4QpIkiRJNZJJ\nwExgNjArs432uNG+IeDbwMuhtLinzWig0YTW/SZhcOoqSZI0ASXLA2VgV+DVwByqQXgZ4CngMeDx\nzPZYg/tj7Cu90M2G0GLuHJSQaqCWJEnqK8kywA5EgN4V2BC4FLgAuAR4iGoYfhJKL+VU0VYZqCVJ\nktQNyRCwFbAbEaC3Aa4lAvT5wOVQeja/+nWMgVqSJKkYkinApPxCalICXkGE592AXYC7iAB9AXAR\nlB7Lp26ugZXTAAAgAElEQVRdZaCWJEnqD8lUYDliLHGjbbT904GXiGET9zTY7k1v74PScx2o85pU\nh3DsCrxA9D5fAPweSve1f46+Z6CWJEnqjWRN4CDgVdQPxFOBR4FHRtlG2/8EkX9WBFZPt9Uy97Pb\nKsASGgfvyvbAyAv6kuWB11DthV4RuJBqiL4VShNt+mIDtSRJUvckKwMHAgcDGwO/BBYAD7N0IH6q\nd2E0GQJWYuzgPRdYTITrErA+cQFhZRz0NQN08WC3FDZ3TrRPRpIkqW8kK0ByFCS/g2QJJD+GZJ90\nSMeASSanC6O8GpLtIZmWd436UGFzZ2EbJkmS+lEyG5LDIfkVJI9C8nNI3phOFadiK2zuLGzDJElS\nv0hmQPImSE5PQ/RZkBwKyay8a6aeKmzuLGzDJElSnpJpkOwHycmQPALJbyB5e3qxniamlnPnoAy4\nLuzgcEmS+l9SImayOIyYveIxYnaKelt235P9OUNEMoWY1eLNwH7E4iSnAqdD6cE8a6a+4CwfkiSp\nU5K1gMOBtwJTgB8BdxBzJy8HzM7cr7dNZWTAHiuIPwk8CzyX3j47xuMXmg/syRCwMzE7xwHAQiJE\n/xxK97T6zqjQWs6dk7tUEUmSNJCSmUTgfBuwBfBz4EjgstZ7m5MpNA7dledXBNZN788AphFBfFqd\n+7WPJ0FSG7gbhe9NiKnifgq8GkqLWmuL1Nig9PraQy1JUtckQ0CZCNFvAP4A/BA4G0rP5FixMSRD\nNA7ftUH8dijdklNFNVgKmzv7cPyVJEmDLtkQki9Bcickf4Hk/enCJdJEVtjcWdiGSZLUW8lcSN4L\nyZWQ3APJVyHZNO9aSX2ksLmzsA2TJKn7kmmQ7A/JL9Op4X4Cye7pkAlJIxU2dxa2YZIkdUdSgmRr\nSL4FyWJILkznV56dd82kPlfY3FnYhkmS1FnJOpB8HJK/Q3IzJJ+EZH7etZIGSMu502nzJEkaWMkk\nYGNgJ2DHdJsKnAEcAVzRnwurSMUyKFOCFHb6EkmSmpdMI1YsrATo7YHFwB+Jqe7+CCw0REttcaVE\nSZKKI5lDhOYdiRD9SuDvVAP0JVC6L7/6SYVkoJYkqX3JHGA34HngYeCh9PZhKD3XxfOuSXXoxk7E\nCoJ/otr7fDmUHu/e+SVhoJYkabySZYC9gUOBXYkA+yKwArE89grp9gwjQ3btbYPnSi/UnK8EbEg1\nPO8IzGLk8I2/QOn57rRXUgMGakmSmpdMBl5LhOj9gKuAk4FfQOmROseXiNCbDdmNbrP3lweepBqy\nnwA2AR5jZID+u+OfpdwVNnf6n4skqUOSEiTbQfJNSO6H5Ip0ye3VunjOSTGMJHkZJK+GZDdI1uje\n+SS1oeXcOSjpu7CfFCSpGJIS0RO7PrBeelu5Pxe4Dvhzul0FpftzqOPGRE/0IcCzRE/0KVBa2Pu6\nSOpjDvmQJHXLqKF5fWAScEu6LczcPgRsCmxFTPm2FfAUMbziz9Xb0gNdqPM84GAiSK8AnEIE6Wsc\nWiGpAQO1JKkd4wrNlfuLmwupSQlYh5EBeyvgcUYEbP4MpQfH0YaVgIOInugNgdOIEP1HKL3UenmS\nJhgDtSQVU7IWsBmxwu1kYKjO/Ua3zRwzC3gZHQvNLbevREwRVwnYrwK2BB5l6ZC9uM7rZxEXFR5K\nzNv8ayJE/7a709xJKiADtSQVQzKFCIZ7AXsCqxPzET9LTOX2QodvnwRupauhuVXJJCLkZ3uytwSW\nUA3YdwL7EO/RxUSIPgtKT+ZRY0mFYKCWpMGVrA7sQYToXYlw++t0uwpKL+ZYuT6RTCKGn1RC9jrA\nb4DToPRQnjWTVBiFzZ190FMiSZ2WTIZkR0i+CMlfIXkYklMheSskq+RdO0maoAqbOwvbMEkTTbIK\nJG+D5KdpgP4LJF+AZId0kRFJUr6ch1rSoEmWIS5GW69mm0WM7W1le6nJ454EHsxsi9Pbhzs/C0Qy\nBGxNjPHdixgTfD5wLnAelO7p7PkkSW1yDLWkfpTMJIJkbWheD1gJWMTIeYtvJWZ3GOrgNilzf2Z6\n3rnpbWWbTVzwVhu0R7lferZOe1cCXk+E6NcD9xDjoM8FLoXS8+N5FyVJPWGglpSXZDnqB+b1gDlU\nZ5Co3e7sn4vtksnAiiwdtke7/ywjg/Zc4BXA74kAfS6U7uxpMyRJ7TBQS2pWUgKmANOBZdKt3v1G\n+5cB1qAampehfmBeCNxTzAU1khLRq50N2k8Alzj3sSQNLAO1pGQFYszuNuntXBoH5peAZ4CnM7fN\n3K/c3kt1mMb9/TF3sSRJbTFQSxNLMhnYFNg23bYheo2vAi4HriRCb4NgXHohh0pLktTP+i5Q7wF8\ng7gI6HvAV2r2fxg4LL0/GdiQ6E17pOY4A7VGkawCHEgshHEf1WWSbwFuL9ZX78kaRGiuBOgtgX8A\nVxAB+nLgBoOyJEnj1leBegi4CdgNuJtYMvcQ4MYGx+8DfCA9vpaBWjWSucABwJuJUHkOcQHYisD6\nmW1N4C5GhuzKtqi/g2eyDLEaXDZAL0OE5kqA/hOUaj+ASpKk8Ws5d3ZzEYGtiXGVi9LHpwL70ThQ\nHwqc0sX6aOAlywP7EyF6G+A84Pi4LT3d4DVTiaWJKwH7FcC+6f1VIbmD+mH7jt7OPJGUiAv7KsM2\ntiW+sbmBCM6/BI4DbnWcsiRJ/aWbgXoNIDtV1F1EUKhnBjFX63u6WB8NpGQ28UHszcBOxIIYJwIH\nQOnJsV9feo74puSmOmVPJxYUqYTtzYA3pvdXguR2qgH7NuAF4t/MZOIbmMl1tvE8Pz099xNUe55P\nBv7a+IOCJEnqF90M1K30ou0L/JGlx05nDWfuL0g3FVIyk/ideBPwWuAi4tuLQ6D0eOfOU3qG6AG+\noU4dZhALkVTC9qbE1z8vpNuLmfuV7dk6z9U7rva556MOrpgnSVIOyunWl7YlvpKvOA74aINjfwEc\nPEpZfsVdeMkMSA6E5OeQPArJuZAcAcmcvGsmSZImlL7KnZOJldHmA1OBq4kxobWWAx4iLrZqpK8a\npk5JpkGyHyQnQ/IIJL+D5ChIVsy7ZpIkacLqu9y5JzF2dSHRQw1wdLpVvI0YLzqavmuYxiuZCsle\nkPwAkochWQDJuyFZOe+aSZIkUeDcWdiGTQzJdEj2heT7kCyG5BJI/hWS1fOumSRJUo3C5s7CNqy4\nkmXTMdGnpMM5LkpD9Fp510ySJGkUhc2dhW1YsSTLQXIYJGekFxb+FpKj05UMJUmSBkFhc2dhGzb4\nkhUheQck50DyGCRnp7NzrJB3zSRJksahsLmzsA0bn2R6BNYkp+XYk9UgeQ8kF6Q90T+H5JB0ERZJ\nkqRB1nLuzCmQtazlNdUHVzKdWGVyLWDNzG32/nLAM8TUhHfU2e5Mb+9KFy/pRL3mAQcQKwluDJwD\nnA78BkpPdeYckiRJuWs5dw5KSC1IoG46LN9DhOK70u3OmtsHoPRS2iO8FrB2zVZ5bg1gCaOH7geg\n1OCTWLI+EaDfCKwDnEmE6Aug9GxH3hJJkqT+YqDuD8lU4A3Aa6gfluuF5Jqw3JF6DAGrsHTgzm4z\nqYbryjYJ2A9YiVjF8nTgIii90Jl6SZIk9S0Ddb6SjYEjgcOBvxE9uv+gGpg7GJY7JZnB0r3c04Bf\nAZf2X30lSZK6akByZ+v6+KLEZFa6XPZlkNwDyRchWS/vWkmSJGlc+jh3tqfPGpaUINkOkhMhWQLJ\nLyHZB5LJeddMkiRJbemz3Nk5fdKwZGVIjoHkBkhuguRYSFbNu1aSJEnqmD7JnZ2XQHJwjFFOpvT4\n1EOQ7AnJaekS2idBslN+c0BLkiSpi4o8D3VyBrApcQHdLcB16XZ9entH4+nfxnXKdYC3p9u9wInA\nqVB6tHPnkCRJUp+ZCLN8JDOADYlwvUnmdiYxs0ZN0C491MJppgP/BBwFbA6cDJwIpWs70gpJkiT1\nu4kQqBsesiIRrCshuxK0n6Tai10J2jeMXN0v2YwI0YcCfyF6o8/s3CqDkiRJGhATOVDXfVmJGCKS\n7c3eFNiAmBf6+nT/KsD3Yyst6kSFJUmSNJAM1E0WNwVYnwjXjwDnQ+nFzpUvSZKkAeXCLpIkSVIb\nWs6dk7pRC0mSJGmiMFBLkiRJbTBQS5IkSW0wUEuSJEltMFBLkiRJbTBQS5IkSW0wUEuSJEltMFBL\nkiRJbTBQS5IkSW0wUEuSJEltMFBLkiRJbTBQS5IkSW0wUEuSJEltMFBLkiRJbTBQS5IkSW0wUEuS\nJEltMFBLkiRJbTBQS5IkSW0wUEuSJEltMFBLkiRJbTBQS5IkSW0wUEuSJEltGC1Q7wEcVOf5A4HX\ndac6kiRJUnFcCqxc5/mVgMt7XJekx+eTJEnSxNRy7hyth3oa8ECd5x8Elm31RJIkSVIRjRaoZwFT\n6jw/BZjenepIkiRJg2W0QH0G8L/AzMxzs4AT0n2SJEmSRjEF+DKwGPhLui0GvkL9nutucgy1JEmS\neqHl3Flq4pgZwHpp4bcCT7V6kg5IaK6ukiRJUjtazp2TR9n3RqoJvQS8BMwBrgYeH0/tJEmSpKIZ\nLVDvy9Jd3isAmwNHAhd0q1KSJElSkc0DruzxOR1DLUmSpF7o6DzUjfyD3l+UKEmSJPWl8QTqVwDP\ndLoikiRJ0iAabQz12XWeWx5YHTi8O9WRJEmSBstoU4KUax4nwEPAzcBz3apQA06bJ0mSpF7oSe7c\nCfhWt09Sw4sSJUmS1Ast587RhnxkbQkcArwJuB04vdUTSZIkSUU0WqB+ORGi3ww8CPyc6P4ud79a\nkiRJ0uB7CTgLWDvz3O051cUhH5IkSeqFjs5DfQDwNHAx8B1gV7wwUJIkSWrZTOAw4FfAk8C3gd17\nXAd7qCVJktQLXc+dKwDvAn7f7RPVMFBLkiSpFwqbOwvbMEmSJPWVjo6hliRJkjQGA7UkSZLUhmYC\n9VeafE6SJElSHX+t89x1Tb52D+DvwC3ARxscU07PcT2woMExjqGWJElSL3Q0d76bCM5PpbeVbRHw\nkyZePwQsBOYDU4CrgQ1rjpkD/A1YM308t0FZBmpJkiT1Qkdz53JEGD4VmJfenw+s2OTrtwPOyzz+\nWLplvQf4bBNlGaglSZLUCx2d5eNRojf6YGL58dekjycB6zRR9hrAnZnHd6XPZa1PzG19IXAV8JYm\nypUkSZL6xuQmjhkGtgJeDnwfmEoM+dh+jNc1k+6nAFsSy5rPAC4DLifGXEuSJEl9r5lAvT/wSuDP\n6eO7ieXIx3I3sFbm8VpEL3XWncBi4Ol0uxjYnPqBejhzfwGNL2CUJEmSmlVOt666Mr2tzPaxLHBt\nE6+bDNxKjLueSv2LEl8BnE9cwDiDuOhxozplOYZakiRJvdCV3PkR4ATgduBdxJCMf23ytXsCNxGz\nfRyXPnd0ulV8mJjp47pRyjVQS5IkqRdazp2lJo/bPd0AfgP8rtUTtSmh+bpKkiRJ49XV3DkXOIC4\nQLHX7KGWJElSL3Q0d54DbJLeXw24DzgbuAH4YCdP1AQDtSRJknqho7nzb5n7Hwd+mN6fRfNLj3eK\ngVqSJEm90NGFXZ7P3N8NODe9/zjwUqsnkiRJkopotHmo7wLeR8wn/Uqqy4jPGON1kiRJkoBViOny\nzqQ6wwfEEuQf7nFdHPIhSZKkXihs7ixswyRJktRXOjqGWpIkSdIYDNSSJEmaqDqygEszgXrHOs/t\n0ImTS5IkSTnZF/hJr0721yaf6ybHUEuSJKlTysADwKvr7Gs5d442/d12wPbASsCHqHaJz8KhIpIk\nSRpMrwZ+BrwZ+FMnChwtUE8lwvNQelvxGHBgJ04uSZIk9dBGwNnAkcCFvTzxvMz9IWC5Xp485ZAP\nSZIktevbwGFjHNOV3HkyMBtYFriBWDnx2G6caBQGakmSJLWrmVk9upI7r0lvDwO+DkwBruvGiUZh\noJYkSVIvdGVhl8lEiP4nYszJ8+M5kSRJklREzQTqE4BFwEzgYmA+8Gj3qiRJkiS1bTqRX/tSidFn\nB+kGe8QlSZLUrMnAL4Hhcby2K7lzVeBE4Lz08UbEVCO9ZKCWJElSMyYBPwR+TUwD3aqu5M7ziImv\nr00fTwGu78aJRmGgliRJ0lhKwDeBPwAzxllGR3NnZVjHVeltdrnxqzt5oiYYqCVJkjSWzwJ/ob11\nUzo6y8eV6e0TwNzM89viRYmSJEnqL5OAacAe9FFWrfRIbwVcQlTsUuAWYPMe18UeakmSJPVCy7lz\ntNVi7gL+Iz2mRCT+EvAs8GK6r1cSmlvZRpIkSWpHy7lztOnvhoBZdZ4f7wBvSZIkaUL569iH9IxD\nPiRJkpS1FTCnC+V2ZelxSZIkqZ9sQcwz3evr+lq2Yt4VyLCHWpIkSQAbAPcAB3ap/MLmzsI2TJIk\nSU3bHvgH8I4unqOwubOwDZMkSVJTXgPcCxzQ5fMUNncWtmGSJElqyhC9GZJc2NxZ2IZJkiSprzjL\nhyRJkgbe8nlXoIjsoZYkSSq+mcDxxHooeXX82kMtSZKkgfRa4Fpg2fT+S/lWp3jsoZYkSSqmWcC3\ngTuBvXKuC4wjd07uRi0kSZKkJm0ETAU2BR7JuS6FZg+1JEmSesEx1JIkSQWyJrAJMQez1BZ7qCVJ\n0kS0P3Az8Bjwe+CLwH7A3DwrNU5zgEPzrkQTCps7C9swSZKkJqwI7Al8BjgPODLf6rRsL+Kiw28B\npZzrMpbC5s7CNkySJAmYBqzSgXI+Q8zj/BZgA/IPr8sDJwG3EVPhDQLHUEuSJA2YHYCrgaM7UNZZ\nRHjdB/gdsBg4F1i3A2W36lXA9cATwGbEkJVCyvtTS7MSBqeukiRJzZgFfAk4AHgfcHoXzrEqsA1w\nITEOu9b7iN7xWscDz9R5/r0Njv9WneOXJ4L0Rc1Wtk+0nDudh1qSJKn39gBOAM4HNgaWdOk89wFn\njrJ/FWB6necbBcrVWjh+CYMXpgvNMdSSJKlI3gHslnclVFdhc2dhGyZJkqS+4kWJkiRJUi8ZqCVJ\nkrpjEvAvxEWHUu4c8iFJkgbJK4A/AJcAG+ZcF7WmsLmzsA2TJEmFMgX4BDH/83txNMAgajl3Om2e\nJElS5/yYmF96S+COnOsijWAPtSRJGgQr4WJ0g66wubOwDZMkSVJfcdo8SZIkqZcM1CqKGcBniItB\nJEnqtpWBL2OW0gBxyIfGsgLxe/LOvCsiSSq8ZYErgc/mXRF1RWFzZ2Ebpo7aGrgLWCbvikiSCmsy\ncA7wf3jxYVEVNncWtmHquNOBj+RdCUlSIZWA7wHn4hDDIits7ixsw9RxGwIPAHPyrogkqXCOBq4C\nZuZdEXVVYXNnYRumrjiOWPJVkqROmknMM61iK2zuLGzDNG7rAmfi1dWSJKmznIdaE8Jk4EfARcBL\nOddFkiRpINhDraxPABfgB0JJktR5hc2dhW2YWvYq4qLDtfKuiCSp0FYG/hsYyrsi6rm+y517AH8H\nbgE+Wmd/GXgU+Gu6fbJBOX3XMOViGeL36eAWXrM8sHZ3qiNJKigXbpnY+ip3DgELgfnEXI1XE1Oa\nZZWBs5ooq68aptxMAvZq8TXvBc7uQl0kScXkwi3qq4sStyYC9SLgeeBUYL86x/nLqma9BPy6xdd8\nF9gU2LHz1ZEkFUwJ+A6Rj47GDj01qZuBeg3gzszju9LnshJge+AaIiht1MX6aGJ6Fvg08GX88CZJ\nGt3bgS2Ag4jOQKkpk7tYdjOf6v5CXFz2FLAn8EtggwbHDmfuL0g3qRk/Bo4lhouck3NdJEn962Ri\nKOoTeVdEPVVOt760LXBe5vFx1L8wMet2YIU6z/uVy8Q1pUPl7Ed8E2IvtSRJGk1f5c7JwK3ERYlT\nqX9R4ipUA87WxHjrevqqYeqZdYGbgBkdKKsEbNaBciRJUrH1Xe7ckwhEC4keaohB/ken9/8FuJ4I\n25cSvdr19F3D1HVDwB+BD+VdEUmSNKEUNncWtmFq6OO4GqIkqXtWJmaC6ub1ZBpMhc2dhW2Y6toK\nV0OUJHXPTOBPwOfyroj6UmFzZ2EbpqVMAq4DDsm7IpKkQqos3PJ9vFBd9RU2dw5iw+aw9EWYas56\nXS6/BHwdWLHL55Ek9ZcS8D3gXDo3i5SKp+Xc6bih7jkW+CARqhflW5WBs7DL5SfAssQ0jsd2+VyS\n1Gv7Al8DngGeztz+AfhSnePXT1/zdM1r7iDWi6g1DViGWPjkOeAFBqfj6zBi4ZYyLtyiDhqUrzoS\nBqeuEHVdCHwAODvnuqi+1YmhJZsBd+dcF0nqpMlESJ5OBN/K7UPA5XWO3xR4R53jr6L+GON9gR8R\nU+JOSc/3PHAacGid418HfJVqAK/cXgR8sc7xGwNvBJ4kFlip3C4iZgVrxxRgNvFeSI20nDsHJaQO\nWqB+FXAKserjoHxqn4i+QgzNOXqsAyVJDU0iQvUkome71hxgHaoBvHK7GPhzneM3Bg4mvkmcmbm9\nEvhCnePfBPwfSwfwXwOfH2ebNLENWu5s2qCF0q/hlcOtWC2n864APEjj5e4lqd+tkncF+sAkYBaw\nKnENzubADsAmeVZKA23QcmfTBq1h/0V8wq7nZcTXaQrrElPkrZ7T+Y8F3pXTuSVpvGYAxwPX4nz9\nUqcNWu5sWpEa9t/AAuIrsInO1RAlqXWvBG4ghhYun3NdpCIqUu4coUgNm0T0YF9Lfr2y/cLVENXI\nUN4VkPrQEPGt2oPEbBWSuqNIuXOEojWsBHyMuGL5FflWJTeuhqhGNiBmYJmVd0WkPrMhcD4wP+d6\nSEVXtNz5/xW1YUcA9zIxFxg5D1dD1NJWA24Djsq7IpKkCauoubO4DSOmEpqIZuRdgQaWzbsCE9hy\nxByznxjlmA8Rc4dLktQthc2dg9CwycAPiPk1NZi2Aq7H8bt5mA5cSMxa0GjuzxKx+ui9wBnEhVlS\nUa2fdwWkCWwQcue4DELDdicmndfgKhFL8x6Rcz0mot2JGQua+TAzg1iF9B7gLGIhJakophKrB94L\nrJxzXaSJahBy57gMQsNOpHPTv62LvaR52ZG4WHRazvWYiFpdlWoZ4L3AR7pQFykPLyeW+z4HF2yR\n8jQIuXNc+r1hU4GH6NyMFT8HfkaxQt18BmdBm7OB9+ddiQ6YBPwSOIaCLqEqFUQJ+GdiKe734L9X\nKW/9njvHrd8btg+xQEmnTCdC9YXEhVqDbgox7/YBeVekSZsB9zH407YdAvw13U6iWB/QxlIixsRL\ng2BZ4DRiWjxJ+ev33Dlu/d6w/yO+eu6kIeICrauJqcQG2THAbxmsXpdDgdl5V6JNyxKzyFT+WF8K\nzM21RlXdnk1lJeBWYlXS1zBYv3uSpHz1e+4ct35v2DJ0JyCUiNUEb0nPMYjWJr7G9Ir1fE0C3kl/\n9FLvAiyk+3WZDLwVuJm42PR1GKwlSWPr99w5boVtWJMGeTXFXwD/lncl1Dc2J1bIfG0PzzlEfONw\nIxGwpbzMJC6iHdQOEmmiKGzuLGzDCm574Cb6o1dU+VsHuAt4U07nH8J54pWPqcD7iKnwfkz/DL2S\nVF9hc2dhG1ZwJfzD0e+WozerQ65EDL3o9LUGUj+bBBwO3EZMhbd5vtWR1KTC5s4Exz7WMqj2zi7A\nBnlXognTgXOB5Vt4zfuJWUDW7kqNqvYChrt8jqJbi3gfNTjKwGXAzjnXQ1JrCh2o98y7EnW8hnzG\nwg0BNwD75XDuiejDwOl5V6IJHyfmnW5FiZiF5R5iiM5E80ryG4LSirWJXs4lGKoHSQk7g6RBVOhA\n/Uf66z+m2cCjwJyczv9q4uKuTXI6/0SyDLF64htyrsdo1iIWF1p3nK/fk/h9OqJTFRoQGwP3E0uf\n96u1iCkA30986HkAmJdrjVRPP/19ktSeQgfqW4iv3vvFW4Czcq7D4cQf2hVzrkdWUS9A3I4IXvNz\nrkcjpwCfa7OMDYl/Z/u3X52BsiMRUrfMuyINrAq8PfN4Uwxv/WQesXCSsxlJxVHoQH0ksThIvzgH\nOCzvSgBfA84n5tzN21yiJ7efAn4nfRC4kv6bKWJn4A5gRgfKWp5Y2bJdq3agjF7aH7ib8ffwa+KZ\nC/wn8c3Q5yjGqraSQqED9VRg37wrklqBGO7RD0tTDwFnAFvnXRHgROAbeVeii0rACfTf8sC7AXvn\nXYmMtxFzPvfDh7xW/DNO86ixlYBPEQtWHc/gfXiUNLZCB+p+ciT9dZFaP3z9uxMxx/CgL9et9uwN\n3Ef/feholtckqBkfAF6WdyUkdU2/5c6O6beGbQdsm3cl+shU4HrgwLwroq6YC7yDsT+4bQs8iP82\n2rEq8Ala+5C8Dv19wawkDZp+y50dU9iGFcRHgV/THz3l6rx5xAemE2g8fnxDomfaKd3GbxViqMwn\nW3zdpsRFlRNx2sNumoTfWEgTVWFzZ2EbVhDr4TReRTebmNVmAfUXFdqDmPlG47MK8DfGP1PEXsSy\n1ut0rEYT10rAR4CFwJ/ov4uQJXVfYXNnbcNmAvvkUZEBsRPwXewx7oX3AIf2+JwlYhq1Xv+hHwK+\nRCwwsmmPz52XbYGjunyOlYlvAD7TZjn/SoRyZ5sYn+2BnxCL53wf2Ab/D5UmqgkTqFcgpirq9nLJ\ng2oG8Geil0XdtTkxbriXF+EdSvx8h3p4zqzDiFkOJoJ1iOn03tjFc5xKTLvWbngrAd8CziO/341B\n9nFi8Zzl866IpNxNmEAN8BXgmz2uxyD9kVqLWE56j7wrMgEcSfQwLtuDc80kZlPZoQfnUnglMUZ5\npy6VP5PO9YROJsK/PauSNH4TKlCvAjxMb+cAXUAs+T0odiCCwMvzrkjBlYAfEl8Td9uXgB/14Dwa\naTdipUwvUhtcM4F3Eb34kjSaCRWoIXqov9qjOqxNTOTfiVXkeuko4Co622O1ObGIi6qWJcavdvPC\nvPWJ38HVu3gONXYoMX7chV8Gy+bAt4kOmDOA3fOtjqQBMOEC9drEWOpeLCbyYeJCv0HUyQA2CbiM\n7hf5JtgAABeOSURBVF+oNYjWp7vfmPwbcGwXy9fY5rf5+lnEvyH1xhnAncS/nTVyroukwTHhAjX0\nbpqoq4Bde3SufvYu4BIMBXko4fs+yOYQ/4+8qcfnfQUxk8hEtAExrlySWjEhA3UvrEcsWjHR/2Ne\nmRiTvVneFZEGzBzgSuAb9P6CwU8ClwLTe3zeXplG/B8tSZ2Sd+7smrwbtjfwhZzr0A9+CHwt70pI\nA2Y54HLgv8ln9o1JwE+JOZaLNPvHFOB9RGfHt3Oui6RiyTt3dk1hG5aTt9D6WN8SMa/1zM5Xp7CG\niOkLVVw7EddXNDKbuObgePINs8sAVzD+lRj7SQnYH7gZ+A1x0aEkdVJhc2dhG5aTYWIctLMVdNdu\nxPLF7axct2KH6qLuWA24nVi5sp4ViAtJ+6FneFXgH8DBeVekTd8HrgVen3dFJBVWYXNnMw0bAj5P\n75djHkSTiKvfv0d//KEvsm8BpzO+93kHohfOCxH728uJYQd75V2RJmwGHJF3Jdo0j8FaZEvS4JnQ\ngRri678ju1mRAplJ9PK8N++KFNw0YmaH97f4uiHgL8AhHa+RumFbYgn6rfOuiCSpbRM+UO8M3IKz\ncTRrHaJn7bV5V6Tg1iVmR9mmhdccDVyM3yAMkn2ARRR3No1emkZccOg1G5LyMOEDNcAf6Fyv3hG0\nFoIG0Y40Xpp8bwzbnbI/zS9NvgKxzPUW3auOumSizvfcKSVijPdtwNm4KqikfBiogT2A62l/3GmJ\n+E99yzbLGVSzgbuIwK3/196dR8tR1Qkc/4a8BLJAWAIoBAigbEFANBFZm0UNDhgRQQwgHpfBIxpx\nQYIwzBsYcRSOYVEYV8ABZRRxQUAZR8ImS4IJBIIKMRHCJoSAIYATSOaP3+3z6jW9pvd+3885dV5V\n3VtVtys35/z69q9uNUa1o80XAhc3syFSRqe8QXB/Yq7uuUCuvU2RNMQZUBNBy13A5DqvOZlIHxmq\nP7lfQDy0qNZ7A87uodYYBswBPtnmduxOpMtMx4dwJbWfAXXSiJk+zgPObsB5utFbiNxqgzqp921P\n/H8/gvbmfztDk6ROYUDdIOsAjwC7tvi6nWA48AzdP7WWpOrtCzwIvAw8Bry1vc2RpLYyoG6QvYEH\nWnzNTjGReI3vUE11aZWNiNfZ+/O2Okn+7Z6jSpTfCSwErgMuAj4LTAPGVHHudYATgFPrb6YkNVXN\ncWe3BE1raG1b+4AJRE6f1Ax9wO+AG4kXEkndYCzxpXu7zLIt8DFiZppCRxHzc48h+vlLxKvaf9+C\ntkrS2mp13NkyPTv0riFtC+Bx4mVEl9Oj/3k1pH2bmE99DvB+7OOSukPPxp31fLBZmA+oznUIsBo4\nrd0NkSRJgAF1UTOAaxrVEKkJ9sYZDiRJ6hQG1EWMJqaEmtSgtkiSJKl3GVCXMBO4sop62wCb13kt\nSZIkdS8D6hI2IOZWfkOFet8npoGSJEnS0GRAXcZM4Ogy5SOBZcR0eZIkSRqaDKjrcBhwawuuI0mS\npM5Vc9zpW9oGHANc1e5GSJIkSc3Q7BHqUcByfCBRkiRpqDPlYy2NB05u8jUkSZLU+TouoJ4K/BF4\nCDi1TL3JwCvA+0qUN+ODDW/COSVJktTdOiqgHg48DEwERgDzgZ1L1Psd8CvgyBLnavQHmwmc2eBz\nSpIkqft11EOJU4iAegmwinjgb1qRep8GrgaebmJbCv0M+BQwtoXXlCRJUg9qZkC9JfBoZntp2ldY\nZxpwSdpu1RD7n4CbgE+06HqSJEnqUX1NPHc1wfH5RPrFGmBYWkrpz6zPTks9vgz8GvgG8HKd55Ik\nSVJ3yqWlI+1FBKx5p/HaBxP/AixOywrgKeA9Rc7VrJHrJ6k/MJckSVLv6KiHEvuARcRDiSMp/VBi\n3qW0dpYPiHmnt2rSuSVJktR9ao47m5ny8Qrx4N9viJk8vgc8CJyYyr/VxGtX66l2N0CSJElqhY4a\nepckSVLP6qhp8yRJkqSeZ0AtSZIk1cGAWpIkSapDMx9KlCRJ6hbPAhu1uxFqqeXAxu1uRCv5UKIk\nSWomY42hp9S/uQ8lSpIkSa1kQC1JkiTVwYBakiRJqoMBtSRJklQHA2pJkiSpDgbUkiRJ3WE2Mb3f\nyCZf5zJgFfC6IvvPLtg3EVjN4JhyOjAXWAE8DlwP7NP4ZnYOA2pJkqTONxGYAvwNeE8TrzMGOBJY\nCBxXULaGylPKfQ6YBfw7sBmwFfBNmttmVcm5ISVJUjN1eqxxJvBL4HTg2rRvXeA5YFKm3qbAi8D4\ntP1FYpR4KfAxYjR5uzLX+RBwH3AssKCg7FLKj1CPI0alj6zuI7Vdw+ah7hY9+8EkSVJH6PRY42Ei\nyH0j8H9E4AzwPWI0OO8kIsUCYCrwBLAzMAq4AniV8gH1/xJB+/rAS8CembJKAfVUIlWkWzIgDKgl\nSZIaqEKssWZNY5a1si8R3K6ftucDJ6f1g4lgO+92BlI1vg98OVO2PeVHqLcmAu4d0vbPgfMz5ZUC\n6mOJAL5b+KZESZKk1hk2rDHLWjkBuJFIpwD4SdoH8aDiaCK/eiKwO/CzVPZ64NHMeZZWuM7xwP3A\nnzPXmQ4MT9uvACMKjhlBBNSrgWVEqsmQiy/72t0ASZIklTQKOJoIUvOjv+sCGwK7EfnOPwY+SDyw\neC2wMtV7gngoMC+7XsyHUp38dfqATYB/IvK3H2FwvjbAtgwE7XcA/wCOAH5azYdTa5nyIUmSmqlT\nY40PEiO/E4hZMzYDNgduBs5LdaYQQfAC4PDMsVOJBxJ3IkaxL6d0ysfbifznSQXXuQK4OtWZRIyS\nv4MYtd4CuAU4J3OezwFPAtPSNUcAhwJfrf2jN5051JIkSQ3UqbHGDcC5RfYfRQTL+fSKh4BneG32\nwUwi2F4KfIIIqLcscr5LiBSPQpOJ/O0N0/ZhxBzTzwFLiEB53YJjpgNzgBfSta8F9ipy7nYzoJYk\nSWqgoRBr7EzkQQ+5HOcSDKglSZIaqFdjjSOIEeSNiDzoa9rbnI5iQC1JktRAvRpr3ECkZywjHhTc\nvL3N6SgG1JIkSQ1krDH0OA+1JEmS1AkMqCVJkqQ6GFBLkiRJdTCgliRJkupgQC1JkiTVwYBakiRJ\nqoMBtSRJUmdbDWxXsK8f+K/WN6WoLYFfEHNdPwqcWFC+mngN+Yq0fDtTdjCwmHhF+Qcy+zcE7gHG\nVLj2BsD5wF/TuR8GZgGbpPIl6RrCuSElSVJzVYo1cg24xtqeo1hA/a+0J6AuNhh7E/B1YDiwGxFY\n5zLlq4FtS5zvPmCXzHHD0v5LgPdXaMtIYA7wG2CntG9T4HRgatpeDBxU4nhf7CJJktRAlWKN/gZc\nY23PUWmEOgcsBU4DniaCyOmZupcB/wncCPwdmA1snSnfCfgfIqD9I3BUwbGXANcTo8yFwenY1L7x\nmX3fAn5Q0P7tS3y2RZn1J9J5pqTrVfIx4ElgdJk6LQmoTfmQJEnqfpsTaQ5bACcQaRU7ZMqnA2cR\nAet84Mq0fwwRTF9BjO4eA1wM7Jw59oPA2UTwfHvBdYcV/IWIL3ctqHcLETD/FNgms/9vxOj07sCr\nxGvSzwdmlP+4ABxCvFr9xSrqCkeoJUlScxWLNXLESHB/Ki+29Jc4X6lj8ufL1dC2akaoVwGjMuX/\nDZyR1i8DfpgpGwO8Akwg8pZvKTj3t4AzM8deVqF9twIXAusCexIj3Q9myvcF+oBxwEXAAiI9BCKQ\nvgm4AziQCKT/jQiyfwP8Dti/xHVvBM6p0LaWjFD31XqAJEnSEDE7LXn9NRzbX6R+sX3VeBUYUbBv\nBBFE5y0HXsps/xV4fVpfQ6SE5K0EniVGs7cB3paOz+tjIGWj8NhijgW+STyQuIgY7Z6UKb8t/X0e\n+Ez6uxPwAHAvEUiT2vt14O1EkD+DGNW+hcGj2nnL0mdoO1M+JEmSOtsjvPahvm2JGSzyNmJwLvE2\nwONpfRiwVaZsLLAx8Fg6983p+PyyPnBSje07HNiMCIY3Be4qUbdYikjeLOKBwpeJlJG5xBeDEQzO\n0c77LfAuyudQK8OUD0mS1EydPMvHOcQo75bEYOghxMOFu2TOuwo4lwg+9yMeIMznUF9GjArvQ8yM\nMYtI04AInpcAx6VjRwCTGZg14zIif7qcndJ5RqbzPM3AtHW7AHsQKR5jgQuIdJDhBed4B3BNZvsB\nIliexODZP7JGAncTedQ7EvdmE+BLwKGpjrN8ZPTsB5MkSR2hk2ON9YCvEcHhc8TI7WGZ8hyRbvEl\nIphdQqRh5F1KzNRxIzFX82wGp1DsAPyKeEDwGWLkd7fMsWdVaN9n0rEvEOkZe2bKDiRmDnkBeIoI\nmgtn/FgXmMfgUfSD0ud9DDi6zLU3IL4gPMLAPNTnESPtYEA9SM9+MEmS1BG6OdbIEQF1KZdSeZR5\nKHLaPEmSJFWlWLqEGsiAWpIkqfuVG1VdU6FcQ4SdQJIkNZOxxtBjyockSZLUCQyoJUmSpDoYUEuS\nJEl1MKCWJEmS6mBALUmSJNXBgFqSJEmqgwG1JElSZ1sCHNzuRpRxOHA/8erv24GdM2UfBl5NZfll\n/0z5+cCzwO+BLTP7pwMXVHHtKcD1wHJgGXBXuiZUfoPkkOPckJIkqZk6OdZYDBzU7kYAfUX2vRF4\nHtibGKidCTwEDE/lHwZuKXG+KalsBPA14KK0fxzwB2Bshfa8nQjQTwE2Tvv2BK5K6znKB9QNm4e6\nW/TsB5MkSR2hk2ONUgH1usQI72NpmQWMTGU3A+9L6/sAq4F3p+2DgXmZ83wEWEiMFP8a2DpTthr4\nJBEkLyrShk8Bv8psDwNeBA5M2x8Gbi3xuY4GzknrU4Hr0vo3gGNKHJN1GwNBeDE5WhRQm/IhSZLU\nnU4nRnl3T8sU4IxUNpsIKAEOAP7CQKrFAakcYBpwGnAEMJ4Ifn9UcJ1pwGRglyJtWEME0XnrpO1d\nM/veDDwN/Cm1Lz96/QCwH7AeEeTfD7wV2IGBUeZSRgN7AVdXqKeMTv7WKEmSul+lWKM/1Slc+muo\nX6puJaVGqB8mRnbz3pnqQgSo96b1G4CPAnek7ZuB92bKPpI5xzrASmCrtL2agcC8mB2BF4ggfSTw\nL0TO9KmpfFtgm7S+KxFEz8wcfzIwnwjixxM52DsCM1I7ryBSQAptmdq2Q5m25TDlY5Ce/WCSJKkj\ndHKsUSqgfpHBDwDuBPwjrY8GXgI2A54g8pSXApuk4/I5xwuJPOTlmWUlMfoLEbRuX6F9RwILgGeI\nFJQFwLEl6n4AmFui7CQiBWRSOsdwYhT+K0XqjgZeIQL5UnKY8iFJkqQyHgcmZra3TvsgguZ7iBHg\nBcAqYiaNzxMj28+meo8A/wxslFnGAHdmzlspwPwp8CZihLk/tWlOmfrDiuzbHPg4cBYxkn0fMdI9\nF9itSP0XiRH391domzI6+VujJEnqfp0caywmUjvWyyx9wNlEisT4tNxGBKR5XyZm4Dg9bX8S+DuD\nH+R7LxFw5/OjxwFHZcpXA9tVaN9biNHkTYEfE2kaeYcSwTLECPoCIi2k0JVErjZEvvaficD+K8CF\nJa6bn+XjC8TIO0QueT4HPIcpH4P07AeTJEkdoZNjjcVEYJtdziJm+biAGJV+nEi3GJk57p3EKO9+\naXvXtJ0NmAGOI0aEnydGrL+bKXuVygH1rUSgvgy4BBiVKTsXeJLIs15EjGAPLzj+IODagn2zGJif\neosy155MzEP9XLr+nenzQATUj5Q51oBakiSpgYw1hh5zqCVJkqROYEAtSZIk1cGAWpIkSaqDAbUk\nSZJUBwNqSZIkqQ4G1JIkSVId+trdAEmSpA6wHKfOG2qWt7sBrWYHlyRJUit03DzUU4E/Ag8BpxYp\nnwbcC8wj3jd/UJPbo5BrdwN6SK7dDegxuXY3oMfk2t2AHpJrdwN6TK7dDegxuXY3QM0zHHgYmAiM\nAOYDOxfUGZNZf1OqX4wj1I3V3+4G9JD+djegx/S3uwE9pr/dDegh/e1uQI/pb3cDekx/uxvQYzpq\nhHoKESAvAVYBVxEj0lkrM+tjgWea2B5JkiSp4ZoZUG8JPJrZXpr2FXov8CBwAzCjie2RJEmSGm5Y\nE899JJFD/fG0fRzwNuDTJervB3wX2LFI2cPA9o1uoCRJklRgEfCGWg5o5rR5jwFbZba3IkapS7k1\ntWcTYFlBWU0fSpIkSeoFfUSEPxEYSfGHErdnYJR8z1RfkiRJUnIo8CciZeO0tO/EtAB8EbifmDbv\nVmByqxsoSZIkSZIkSRLrAXcRqSELga+UqHch8aKYe4E3t6ZpXaeae5kDnid+HZgHnNGqxnWx4cS9\nurZEuX2zNuXuZw77Zy2WAPcR9+ruEnXsn9VZQvl7mcO+WYsNgauJ2bwWAnsVqWPfrF6l+5nD/lmN\nHRm4R/OIe1Zsprmu7Zuj098+4E5g34LydwPXp/W3pToqrtK9zAG/bGWDesDngCspft/sm7Urdz9z\nJfaruMXAxmXK7Z/Vq3Qvc9g3a3E58JG03geMKyi3b9am0v3MYf+s1TrAEwyeSANq7JvNfvV4rV5M\nf0cSo1fPFpS/h+hMECOwGwKbt6ZpXafSvYTmTpvYayYQ/7m+S/H7Zt+sTaX7SZn9Kq7c/bJ/1qZS\n37NvVmccMSXu99P2K8RIYJZ9s3rV3E+wf9bqEGJSjEcL9tfUNzstoF6HSFN4CriJ+Dkjq9jLYia0\npmldp9K9XAPsTfyMcT2wS0tb131mAacAq0uU2zdrU+l+2j9rswb4LTCXgbn/s+yf1at0L+2b1dsW\neBq4FPgD8B0Gfj3Ns29Wr5r7af+s3THAD4vsr6lvdlpAvRrYg2jw/sRPF4UKv3nV/L71IaLSvfwD\n8fPG7sBFwM9b2bgucxjwNyLPqtw3f/tmdaq5n/bP2uxD5PcdCpxEjGIVsn9Wp9K9tG9Wr4+YEvfi\n9HclMLNIPftmdaq5n/bP2owEDgd+UqK86r7ZaQF13vPAdcBbC/YXvixmQtqn0krdyxUMpIXcAIyg\nfN7gULY38dPPYuBHwEHADwrq2DerV839tH/W5on092ngZ8CUgnL7Z/Uq3Uv7ZvWWpmVO2r6aCASz\n7JvVq+Z+2j9rcyhwD/H/vVDX9s3xRH4KwCjgFuDggjrZBPG98OGFUqq5l5sz8M1rCvFkuyo7gOKz\nUtg3106p+2n/rN5oYP20Pga4HXhnQR37Z3WquZf2zdrcAuyQ1vuBrxaU2zdrU+l+2j9rcxVwQomy\nru2bbyJ+qphPTFl0StqffREMwDeIF8Xcy2u/mSlUcy9PIl6qMx/4PcWnMtJrHcDAE9T2zfqVup/2\nz+ptS9yn+cQ9K/YSLbB/VqOae2nfrM3uxIjqvcA1xGCPfXPtVbqf9s/qjQGeYeBLNNg3JUmSJEmS\nJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJDXfCw04xzuAucQ89HOBAwvKf8vA3KurgfMyZV8A/jWt\nzwCOb0B7JEmSpJZZ0YBz7AG8Lq1PIl5XnHcQ8M3M9svAImCTtP15BgLq9YG7G9AeSepp67S7AZKk\nivYgXnubfTsawGRiFHoecC6wIO2fDzyZ1hcCo4ARaXs68IvMuVcB3wY+W+S6K4BlRFAuSSrBgFqS\nOt8PgFOI1w4vYGAE+VLg48CbgVeANUWOPRK4hwicAfYh0kCyLgaOBTYocvzdwP51tF2Sep4BtSR1\ntnFpuTVtX04EuOOAscBdaf8PgWEFx04C/gM4MbNvC+DZgnoriKB9RpHrPw5MXLumS9LQYEAtSd2l\nMGgutX8CkR5yPLC4ivOeD3wUGFPkvMVGviVJiQG1JHW254HlwL5p+3hgdtq/ApiS9h+TOWZD4Drg\nVOCOgvM9zsADiFnLgR8TQXXW64Ela9VySZIkqQ1eBR7NLCcTudN3MPBQ4rhUd0raN48YYb4t7T+D\nmH5vXmYZn8q+A7wrc72/Z9Y3A1YCZ2b23YAPJUqSJKlHZdMzZgKzqjgmB1xS5fk3AObU2CZJkiSp\naxxNjD4vAK6leCpHMdkXu5QzAzhu7ZomSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZLUKP8PrxWn\nRX/nUKUAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x10b0c4510>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"'''\n",
"The datasets train and test are defined in the above examples.\n",
"'''\n",
"\n",
"target='y_buy'\n",
"\n",
"def modAUC(X_train, Y_train, X_test, Y_test):\n",
" '''\n",
" trains a model on train set and returns AUC on test set\n",
" '''\n",
" logreg = linear_model.LogisticRegression(C = 10)\n",
" logreg.fit(X_train, Y_train)\n",
" return roc_auc_score(Y_test, logreg.predict_proba(X_test)[:, 1])\n",
"\n",
"def LrBootstrapper(train, test, nruns, sampsize):\n",
" '''\n",
" Samples with replacement, runs multiple train/eval attempts\n",
" returns mean and stdev of AUC\n",
" '''\n",
" auc_res = []\n",
" for i in range(nruns):\n",
" train_samp = train.iloc[np.random.randint(0, len(train), size=sampsize)]\n",
" try:\n",
" auc_res.append(modAUC(train_samp.drop(target,1), train_samp[target], test.drop(target,1), test[target]))\n",
" except:\n",
" oops = 1\n",
" return (np.mean(auc_res), np.percentile(auc_res, 2.5), np.percentile(auc_res, 97.5))\n",
" \n",
"#Run the analysis \n",
"n_seq = np.logspace(3, 7, base=2.0, num=30)\n",
"\n",
"avg = []; lowers = []; uppers = []; sz = []\n",
"for n in n_seq:\n",
" mu, low, up =LrBootstrapper(train, test, 500, int(n))\n",
" avg.append(mu)\n",
" lowers.append(low)\n",
" uppers.append(up)\n",
" sz.append(n) \n",
"\n",
" \n",
"\n",
"\n",
"#Plot the analysis\n",
"#lower = np.ones(len(avg)) * (avg[len(avg)-1]-1.96*stderr[len(avg)-1])\n",
"\n",
"fig = plt.figure(figsize = (12, 8))\n",
"ax = fig.add_subplot(111)\n",
"plt.title('Bootstrapped AUC + Confidence Limits \\n as a Function of N Samples')\n",
"plt.plot(np.log2(n_seq), np.array(avg), label='Avg AUC')\n",
"#plt.plot(np.log2(n_seq), np.array(avg) + 1.96 * np.array(stderr), 'k--+', label = 'Upper 95% CI')\n",
"#plt.plot(np.log2(n_seq), np.array(avg) - 1.96 * np.array(stderr), 'k--', label = 'Lower 95% CI')\n",
"\n",
"plt.plot(np.log2(n_seq), np.array(uppers), 'k--+', label = 'Upper 95% CI')\n",
"plt.plot(np.log2(n_seq), np.array(lowers), 'k--', label = 'Lower 95% CI')\n",
"\n",
"#plt.plot(np.log2(n_seq), lower,'r-')\n",
"\n",
"plt.legend(loc = 4)\n",
"ax.set_xlabel('Log2(N)')\n",
"ax.set_ylabel('Test Set AUC')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<p>We can see in the above plot that Logistic Regression does fairly well with small sample sizes. The lower bound of the $95%$ at the $max(N)$ overlaps with the confidence interval at most levels of $N$, suggesting that in expectation, the smaller samples could perform as well as the larger samples.<br><br>\n",
"\n",
"While this is true, always try to use as much data as you can to reduce the variance!\n",
"\n",
"</p> "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python [Root]",
"language": "python",
"name": "Python [Root]"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 0
}