{
 "metadata": {
  "name": "REGULARIZATION_sklearn_comparison"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Comparison of different Regularizers for Linear Models"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Summary**\n",
      "\n",
      "- Linear model (with regularization) is usually useful because\n",
      "    - in large scale (online) learning, linear model is cheap to build\n",
      "    - in high dimensional data, linear model is less likely to overfit\n",
      "    - coefficients learned by linear model can be used as feature selection indicators\n",
      "    - linear combination of some features can be used as new features in feature transformation\n",
      "\n",
      "- *Ridge* penalizes `l2` and thus does *shrinkage*. it supports `Ridge` and `BayesianRidge` (for regression) and `RidgeClassifier` (for classification). *parameter: alpha, more parameters for BayesianRidge*\n",
      "\n",
      "- *Lasso* and *Lars* penalizes `l1` norm and thus achieves *sparsity*.\n",
      "    - Difference between the two is *lasso* uses coordinate-decent and thus scales with large scale data, whereas *lars* tends to give better (sparse) results for highly sparse solution (few rows and many columns). \n",
      "    - Both of them are implemented as regression in sklearn, including `Lasso`, `Lars`, `LassoLars`, and `MultiTaskLasso` (penalized with combined l1 and l2 norm)\n",
      "    - Difference between `Lars` and `LassoLars` implementations is that `Lars` accpet `n_nonzero_coefs` as parameter whereas `LassoLars` accept the traditional `alpha`\n",
      "    - `MultiTaskLasso` is specially useful to  fit multiple regression problems jointly enforcing the selected features to be the same accross tasks. In other words, the tasks are NOT independent of each other any more -- this is usually useful to model time-series data with time-varying linear models, the coeffs of models could change along time, but their importances will not. *Regression ONLY*\n",
      "    - `RandomizedLasso` works by resampling the train data and computing a Lasso on each resampling. In short, the features selected more often are good features. It is also known as *stability selection*. *Regression ONLY*\n",
      "\n",
      "- *ElasticNet* penalizes both `l1` and `l2` norms at the same time, by specifing params `alpha` and `l1_ratio`, so it achieves both shrinkage and sparsity\n",
      "    - `MultiTaskElasticNet` is the mutlitask version of it. It is trained with trained with L1/L2 mixed-norm as regularizer so it jointly enforcing selected features across tasks\n",
      "    - both of them are for *regression only* currently\n",
      "    - compared to *LASSO*, advantages of *ElasticNet* and *SGD* (p1 and p2 combined) are\n",
      "        - As for Lasso vs ElasticNet, ElasticNet will tend to select more variables hence lead to larger models (also more expensive to train) but also be more accurate in general. In particular Lasso is very sensitive to correlation between features and might select randomly one out of 2 very correlated informative features while ElasticNet will be more likely to select both which should lead to a more stable model (in terms of generalization ability so new samples). source http://stackoverflow.com/questions/12283184/how-is-elastic-net-used\n",
      "        -  For example, in the \"large p, small n problem\" case, the LASSO selects at most n variables before it saturates. Also if there is a group of highly correlated variables, then the LASSO tends to select one variable from a group and ignore the others. source http://en.wikipedia.org/wiki/Elastic_net_regularization\n",
      "\n",
      "- *Logistic Regression* is like the implementation of Ridge/Lasso for *classification* problems. It penalizes `l1` or `l2`, but not at the same time.\n",
      "    - Its parameter `C` plays the oppsite role as of `alpha` in Ridge/Lasso. `C` is the weight on the data-fitting term, `alpha` is the weight on the regulariztaion term\n",
      "    - Its parameter `dual` specifies implementing as a dual or primal formulation *for l2 norm only* usually Prefer dual=False when n_samples > n_features.\n",
      "    - As many classifiers, it is able to predict_proba.\n",
      "    - `RandomizedLogisticRegression` is similiar to `RandomizedLasso`, which does stability selection by resampling the data several times. For large data, use n_jobs=-1 and memory='memory_cache_path' for computation efficiency\n",
      "\n",
      "- *ARDRegression* - Bayesian version of linear Regression with ARD prior, see also BayesianRidge\n",
      "\n",
      "- *OrthogonalMatchingPursuit* - For regression tasks, it essentially penalized `l0` norm, i.e., number of features used\n",
      "\n",
      "- *PassiveAggressive* - `PassiveAggressiveClassifier` and `PasssiveAggressiveRegressor` for both classification and regression.\n",
      "    - One of the key features is that those models support `partial_fit`, which means they can be used in *ONLINE Learning* mode, which is specially useful for *LARGE scale* learning\n",
      "    - The parameter `C` controls regularization. Performance of algorithm is SENSITIVE to C values\n",
      "    - `n_jobs` implements parallelism, but only for multi-class learning\n",
      "    - reference: http://jmlr.csail.mit.edu/papers/volume7/crammer06a/crammer06a.pdf\n",
      "    - by default PassiveAggresive cannot generate sparse solutions\n",
      "\n",
      "- *SGD*\n",
      "    - SGD supports *ONLINE* mode `partial_fit`, similiar to PassiveAggressive\n",
      "    - SGD supports `l1`, `l2` and `elasticnet` penalties, similiar to ElasticNet\n",
      "    - It supports Classification and Regression, and Feature Selection\n",
      "    - It supports different type of loss function such as \u2018hinge\u2019 or \u2018log\u2019 or \u2018modified_huber\u2019. The loss function to be used. Defaults to \u2018hinge\u2019. The hinge loss is a margin loss used by standard linear SVM models. The \u2018log\u2019 loss is the loss of logistic regression models and can be used for probability estimation in binary classifiers. \u2018modified_huber\u2019 is another smooth loss that brings tolerance to outliers.\n",
      "    - Main complexity control parameters are `alpha` and `l1_ratio`. Parameter `n_iter` and `learning_rate` may also influence accuracies\n",
      "    - results are usually sensitive to normalization\n",
      "    - `class_weights` can be used to handle imbalanced classification problem, e.g., with value `auto`\n",
      "\n",
      "- *Perceptron*\n",
      "\n",
      "- *LinearSVM*\n",
      "\n",
      "- **Tricks**\n",
      "    - Reguarlization is seldom useful is the feature is not representative enogugh (like in the case of blackbox)\n",
      "    - Reguarlizatin (sparsity) is specially useful for a lot of features but we need to find a subset\n",
      "    - When cross-validated properly, there is no significante difference between the performances of different linear models (regularized based on cv), yet their computation cost could be very different. Online MOdels are faster and more memory-efficient in general\n",
      "    - Feature engineering by decision-tree, clustering or deep-learning will come in handy when **underfitting** is a main issue\n",
      "    - normalizaion (StandScaler) is important to SGD and clustering, but will make a sparse matrix to be dense again"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import cPickle\n",
      "import numpy as np\n",
      "from sklearn.cross_validation import cross_val_score\n",
      "from sklearn.cross_validation import KFold\n",
      "import matplotlib.pylab as plt\n",
      "from sklearn import linear_model\n",
      "from sklearn import metrics\n",
      "from sklearn import preprocessing\n",
      "from sklearn import decomposition\n",
      "from sklearn import cluster\n",
      "from IPython.parallel import Client\n",
      "from scipy import sparse\n",
      "\n",
      "def make_cv(n_samples, n_folds = 3):\n",
      "    return KFold(n_samples, n_folds=n_folds, random_state=0)\n",
      "\n",
      "client = Client()\n",
      "print len(client), 'cores running...'\n",
      "dv = client[:]\n",
      "lb = client.load_balanced_view()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "4 cores running...\n"
       ]
      }
     ],
     "prompt_number": 82
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## Some Linear Regualarizers specially Ridge, LASSO, LARS and linearSVM\n",
      "## the data used is icml2013 blackbox representation learning and MNIST (large scale?)\n",
      "## BLACKBOX data first\n",
      "## data source: http://www.kaggle.com/c/challenges-in-representation-learning-the-black-box-learning-challenge\n",
      "X, y = cPickle.load(open('data/blackbox.pkl', 'rb'))\n",
      "print X.shape, y.shape\n",
      "print np.unique(y), y.dtype ## classificaiton "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "(1000, 1875) (1000,)\n",
        "[1 2 3 4 5 6 7 8 9] int64\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## Ridge Regression - penalizes on l2 norm, alpha parameter controls penalty weight\n",
      "## (as contrast to C in SVM), testing alpha usually follow a logspace grid\n",
      "## Ridge regression gives shrinkage, but not sparsity\n",
      "alphas = np.logspace(-10, 5, 15)\n",
      "ridge = linear_model.RidgeClassifier()\n",
      "scores = [cross_val_score(ridge.set_params(alpha=alpha), X, y, cv = make_cv(X.shape[0]), n_jobs=-1) \n",
      "            for alpha in alphas]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## To guarantee the optimial selection, the cv score curve should\n",
      "## go up first and go down again\n",
      "best_alpha, best_score = max(zip(alphas, cv_scores), key = lambda (a, s): s)\n",
      "print 'best alpha and best cv_score:', best_alpha, best_score\n",
      "plt.semilogx(alphas, cv_scores)\n",
      "plt.vlines(best_alpha, 0, best_score, colors='r')\n",
      "best_ridge = linear_model.RidgeClassifier(alpha=best_alpha).fit(X, y)\n",
      "#print 'optimal coefficients for ridge', best_ridge.coef_\n",
      "print 'coeff sparse rates for different classes', np.sum(abs(best_ridge.coef_) > 0, axis = 1) *1. / X.shape[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "best alpha and best cv_score: 5.17947467923 0.221985458512\n",
        "coeff sparse rates for different classes"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " [ 1.  1.  1.  1.  1.  1.  1.  1.  1.]\n"
       ]
      },
      {
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEECAYAAAA8tB+vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGaVJREFUeJzt3X981PVhx/H3kcSqKD9NGeSyxpKURCWZMyFFRptHC8Sx\nNkrwRx5jezCIGLJBHjhbWUcfW7BWxdZRSh5KmECHII2/HoZKmiGtV2Yguako7hEYcSN6iUMpwsDQ\nGXJ89se3xByEyyXk7pv78Ho+HvdI7u7zvXtfHpd3vvnc94fHGGMEALDGMLcDAAAGF8UOAJah2AHA\nMhQ7AFiGYgcAy1DsAGCZPou9vr5emZmZysjI0KpVqy64f+vWrcrJyVF2dramTZum/fv3d9+Xlpam\n7Oxs3XzzzZoyZcrgJgcA9MoTbjv2YDCoSZMmadeuXUpJSVFeXp62bdumrKys7jF79+7VDTfcoJEj\nR6q+vl6VlZVqbGyUJF1//fV68803NWbMmOi/EgCApD7W2P1+v9LT05WWlqakpCSVlJSotrY2ZMzU\nqVM1cuRISVJ+fr7a2tpC7mf/JwCIrcRwd7a3tys1NbX7utfrVVNT00XHb9iwQbNnz+6+7vF4NGPG\nDCUkJKisrEyLFi0KGe/xeAaaGwAua+FWmsMWe3+K97XXXtPGjRvV0NDQfVtDQ4PGjx+vo0ePaubM\nmcrMzNT06dMjDjcUFRQUyOfzuR0jYvGWVyJzLMRbXonMPfXVzWGnYlJSUhQIBLqvBwIBeb3eC8bt\n379fixYt0vbt2zV69Oju28ePHy9JSk5O1pw5c+T3+/sVfihKS0tzO0K/xFteicyxEG95JTL3R9hi\nz83NVUtLi1pbW9XZ2amamhoVFRWFjPnggw9UXFysLVu2KD09vfv206dP69SpU5Kkjo4O7dy5U5Mn\nT47CS4iteHtzxVteicyxEG95JTL3R9ipmMTERFVVVamwsFDBYFClpaXKyspSdXW1JKmsrEwPPfSQ\njh8/rvLycklSUlKS/H6/jhw5ouLiYklSV1eX5s2bp1mzZkX55URfQUGB2xH6Jd7ySmSOhXjLK5G5\nP8Ju7hj1J/d44m6OHQDc1ld3sucpAFiGYgcAy1DsAGAZih0ALEOxA4BlKHYAsAzFDgCWodgBwDIU\nOwBYhmIHAMtQ7ABgGYodACxDsQOAZSh2ALAMxQ4AlqHYAcAyFDsAWIZiBwDLUOwAYJmwJ7MGgN40\nv/k7PbC0U51XjYzac1x7rTRlijR1qpSXJ11zTdSeyjqczBpAv7S1SbdOOaP7u36s7G3fi9rzfPKJ\n1Ngo7d0rvfOONGmSU/LnLl/+suTxRO3ph7S+upNiBxCxEyek6dOlvyz8WA++8jXp4MGYPO9nn0lv\nveWU/LnLmTOhRZ+bK119dUziuI5iBzAo/u//pNtuk3JypJ8sPijPnDtiVuy9CQSkPXs+L/r/+A8p\nK0u69dbPy/5LX7JzrZ5iB3DJzp6VSkqc73/+c2nYoYPSHe4W+/l+9ztnrb5n2UuhRX/LLdKVV7qb\nczD01Z18eAogLGOk+++XPv5Yqq+Xhg3RbemuukqaNs25SE7u99//vOR//nPpwAHpppuckv/TP5Vm\nzbJ0jZ41dgDh/OhH0ubN0r/9mzRq1O9vPDj01tgjcfq09MYbTtE/+6yUkCB9//vOSxmqf7B6w1QM\ngAHbskVasUJqaJC83h53xGmx93T2rPTKK9IPfuAU/ooV0t13S4lxMI/RV3fG0d8oALH06qvSAw9I\ndXXnlbolhg2Tiookv1/6p3+SnnzS+fB10yZni5t4RrEDuMC+fdK8edILL0g33uh2mujyeKTCQmeq\n6emnnSmajAzpqaecLYHiEcUOIMThw9K3viWtW+dss3658Hikr3/d+U9l2zZpxw5p4kTpJz9xpmri\nCcUOoNtvf+tsq/6970nFxW6ncc/Uqc78+yuvOGvyX/6y9Nhj0smTbieLDMUOQJKzVvqtbzmFvmSJ\n22mGhptvll58UfrVr6R333XW4FeulI4fdztZeBQ7AHV1Sffc4xyP5ZFH3E4z9Nx4o7R1q7Pz0/vv\nS+np0t//vXT0qNvJekexA5c5Y6TycmdLkKeftnOHncGSkSFt3Ci9+aZz3JxJk6S//Vvpww/dThaK\nYgcucw895GwF88ILUlKS22niQ1qas3nku+86fxhvukn6m79x1uaHgj6Lvb6+XpmZmcrIyNCqVasu\nuH/r1q3KyclRdna2pk2bpv3790e8LAB3/fM/O3uV7tjB8c4HIiVFWr3a2U9rxAjpj/9YKi2V3nvP\n5WAmjK6uLjNx4kRz+PBh09nZaXJyckxzc3PImD179pgTJ04YY4z55S9/afLz8yNeto+nBxBF27cb\n8wd/YMyhQwNY+MABYyZNGvRM8e7YMWP+8R+Nue46Y2bONGb5cmO2bTOmudmYrq7Be56+ujPsGrvf\n71d6errS0tKUlJSkkpIS1dbWhoyZOnWqRo50zqKSn5+vtra2iJcF4I7GRmnhQqm21pk3xuAYM0aq\nrJT+67+kpUul4cOl55+Xvv1tZ40+P18qK3N2ftq7V/r00+jkCHtUhPb2dqWmpnZf93q9ampquuj4\nDRs2aPbs2f1atrKysvv7goICFRQURJodwAD85386h3n5l39xTj2HwTdihFPm3/7257edPCnt3y+9\n/bbz4euGDVJzs3O4hj/6o9DL+PGhH2L7fD75fL6Inz9ssXv68fH4a6+9po0bN6qhoaFfy/YsdgDR\n9T//4xyu9tFHpd+vgyFGRoyQ/uRPnMs5XV3OH9q333Yuq1c7Xz0e54Qmn5d9gb7//YLuA5StXLky\n7HOFLfaUlBQFAoHu64FAQN5ejga0f/9+LVq0SPX19Ro9enS/lgUQGydPOmW+cKG0YIHbaSA5R5K8\n8UbnMm+ec5sxzh/gc2VfW+tsudTWJt1wg1P0fT5uuDtzc3PV0tKi1tZWTZgwQTU1Ndq2bVvImA8+\n+EDFxcXasmWL0tPT+7UsgNjo7JTmzpW++lXn8LQYujweacIE59Lzv6pPP3U2r3z77b4fI2yxJyYm\nqqqqSoWFhQoGgyotLVVWVpaqq6slSWVlZXrooYd0/PhxlZeXS5KSkpLk9/svuiyA2Dp71llLHz5c\nqqpiB6R4dc01n5/i76//OvxYTrQBWO7BB50TZeza5Zw+blBYcKKNeMY5T4HL2Jo10i9+Ib3++iCW\nOoY8ih2w1HPPOecrbWiQxo51Ow1iiWPFABbatMk59O6OHdKXvuR2GsQaa+yART77TKqokHbvln7z\nG+ccnrj8sMYOWCIQcE5ld+yYc4JmSv3yRbEDFvjVr5zDA9x1l3NskmuvdTsR3MRUDBDHjJEef9w5\n4fLWrdI3vuF2IgwFFDsQp06elP7qr6T2dmfqpccx93CZYyoGiEPNzc7Uyxe/6HxQSqmjJ4odiDPP\nPy99/evS8uXSunXSF77gdiIMNUzFAHGiq0v6u7+TXnxR+td/dU7DBvSGYgfiwEcfSffcI115pfTG\nG+xJivCYigGGuMZGKTdX+trXnD1JKXX0hTV2YIgyxjk3ZmWlcxq1nqdZA8Kh2IEh6PRpqbxc2rdP\n2rNH6nEOG6BPTMUAQ8x//7d0663Oh6V791Lq6D+KHRhC6uqcM+SUlkpbtjhnPQL6i6kYYAg4e1b6\nwQ+k9eudzRl7nske6C+KHXDZ8ePSX/yFc4iAN96Qxo93OxHiHVMxgIveftvZlDEjQ/r1ryl1DA6K\nHXDJM89IM2dKDz/sHJ0xKcntRLAFUzFAjHV2SvffL+3c6aylT57sdiLYhmIHYqi93TkZxnXXSf/+\n79KoUW4ngo2YigFi5De/kfLypD/7M+nllyl1RA9r7ECUGePMoT/2mLR5s1RY6HYi2I5iB6Lo00+l\ne++VDh2SmpqktDS3E+FywFQMECWHDklf/ap09dVSQwOljtih2IEoqK119h5dutQ5MuNVV7mdCJcT\npmKAQRQMSv/wD8426r/4hZSf73YiXI4odmCQHDsm/fmfS2fOOIcG+OIX3U6EyxVTMcAgePNN6ZZb\npJwcZ8cjSh1uYo0duEQbN0rLlztnO7rzTrfTABQ7MGCffSZVVDg7Hu3eLWVluZ0IcDAVAwxAIOCc\nXPrYMcnvp9QxtFDsQD/9+tfSlCnS3LnS889LI0a4nQgI5fpUDMfLsNu110pjx0Z+GTVKGjZEVzeM\nkX70I2n1amnrVukb33A7EdC7Pou9vr5ey5YtUzAY1L333qvly5eH3H/w4EEtWLBA+/bt0w9/+EM9\n8MAD3felpaVpxIgRSkhIUFJSkvx+/wWP39p66S8CQ5Mx0qlTznTF+Zf333e2JDn/9k8/dcr9YsV/\n3XWh1ydMkMaMkTye6L6WU6ekBQucKRi/X0pNje7zAZcibLEHg0EtWbJEu3btUkpKivLy8lRUVKSs\nHhOKY8eO1dq1a/Xyyy9fsLzH45HP59OYMWMu+hyssdtt9GjpD/8w8vFdXdInn/T+x+DYMenw4c+/\n/+1vncPgnjkjeb3OJTX180vP6yNHDrz8DxyQioudOfWtW6UvfGFgjwPESthi9/v9Sk9PV9rvD3JR\nUlKi2trakGJPTk5WcnKyduzY0etjGGMGLy2sl5jobAPen+3AT52S2tqctelzX/1+6aWXnO8DAedk\n0b0Vfs/rvc2Vv/iitHixtGqVtHDh4L1OIJrCFnt7e7tSe/zP6fV61dTUFPGDezwezZgxQwkJCSor\nK9OiRYsuGFNZWdn9fUFBgQoKCiJ+fEBy5vGzssJvmfK///t56Z/7A7BnT+gfg4SE0NLv7HQ2Zayv\nd3Y+Atzi8/nk8/kiHh+22D2XOHHZ0NCg8ePH6+jRo5o5c6YyMzM1ffr0kDE9ix2IlpEjncuNN/Z+\nvzHSiROhRf/JJ9ITTzjz+oCbzl/pXblyZdjxYYs9JSVFgUCg+3ogEJDX6404zPjfn3I9OTlZc+bM\nkd/vv6DYgaHA43E+Dxg9WsrOdjsNcGnCbliWm5urlpYWtba2qrOzUzU1NSoqKup17Plz6adPn9ap\nU6ckSR0dHdq5c6cmc9ZeAIi6sGvsiYmJqqqqUmFhoYLBoEpLS5WVlaXq6mpJUllZmY4cOaK8vDyd\nPHlSw4YN05o1a9Tc3KyPP/5YxcXFkqSuri7NmzdPs2bNiv4rAoDLnMe4uNmKx+NhqxkgHh08KN1x\nh/MVMddXdw7RffwAAANFsQOAZSh2ALAMxQ4AlqHYAcAyFDsAWIZiBwDLUOwAYBmKHQAsQ7EDgGUo\ndgCwDMUOAJah2AHAMhQ7AFiGYgcAy1DsAGAZih0ALEOxA4BlKHYAsAzFDgCWodgBwDIUOwBYhmIH\nAMtQ7ABgGYodACxDsQOAZSh2ALAMxQ4AlqHYAcAyFDsAWIZiBwDLUOwAYBmKHQAsQ7EDgGUodgCw\nDMUOAJbps9jr6+uVmZmpjIwMrVq16oL7Dx48qKlTp+rKK6/UE0880a9lAQCDL2yxB4NBLVmyRPX1\n9Wpubta2bdt04MCBkDFjx47V2rVr9Z3vfKffywIABl/YYvf7/UpPT1daWpqSkpJUUlKi2trakDHJ\nycnKzc1VUlJSv5cFAAy+xHB3tre3KzU1tfu61+tVU1NTRA8c6bKVlZXd3xcUFKigoCCixweAy4XP\n55PP54t4fNhi93g8Aw4S6bI9ix0AcKHzV3pXrlwZdnzYqZiUlBQFAoHu64FAQF6vN6Igl7IsAGDg\nwhZ7bm6uWlpa1Nraqs7OTtXU1KioqKjXscaYAS8LABg8YadiEhMTVVVVpcLCQgWDQZWWliorK0vV\n1dWSpLKyMh05ckR5eXk6efKkhg0bpjVr1qi5uVnXXHNNr8sCAKLLY85f1Y7lk3s8F6zpA4gDBw9K\nd9zhfEXM9dWd7HkKAJah2AHAMhQ7AFiGYgcAy1DsAGAZih0ALEOxA4BlKHYAsAzFDgCWodgBwDIU\nOwBYhmIHAMtQ7ABgGYodACxDsQOAZSh2ALAMxQ4AlqHYAcAyFDsAWIZiBwDLUOwAYBmKHQAsQ7ED\ngGUodgCwDMUOAJah2AHAMhQ7AFiGYgcAy1DsAGAZih0ALEOxA4BlKHYAsAzFDgCWodgBwDIUOwBY\nps9ir6+vV2ZmpjIyMrRq1apex1RUVCgjI0M5OTnat29f9+1paWnKzs7WzTffrClTpgxeagDARSWG\nuzMYDGrJkiXatWuXUlJSlJeXp6KiImVlZXWPqaur03vvvaeWlhY1NTWpvLxcjY2NkiSPxyOfz6cx\nY8ZE91UAALqFXWP3+/1KT09XWlqakpKSVFJSotra2pAx27dv1/z58yVJ+fn5OnHihD766KPu+40x\nUYgNALiYsGvs7e3tSk1N7b7u9XrV1NTU55j29naNGzdOHo9HM2bMUEJCgsrKyrRo0aILnqOysrL7\n+4KCAhUUFAzwpQCAnXw+n3w+X8Tjwxa7x+OJ6EEutlb++uuva8KECTp69KhmzpypzMxMTZ8+PWRM\nz2IHAFzo/JXelStXhh0fdiomJSVFgUCg+3ogEJDX6w07pq2tTSkpKZKkCRMmSJKSk5M1Z84c+f3+\nyF4FAGDAwhZ7bm6uWlpa1Nraqs7OTtXU1KioqChkTFFRkTZv3ixJamxs1KhRozRu3DidPn1ap06d\nkiR1dHRo586dmjx5cpReBgDgnLBTMYmJiaqqqlJhYaGCwaBKS0uVlZWl6upqSVJZWZlmz56turo6\npaena/jw4dq0aZMk6ciRIyouLpYkdXV1ad68eZo1a1aUXw4AwGNc3GzF4/Gw1QwQjw4elO64w/mK\nmOurO9nzFAAsQ7EDgGUodgCwDMUOAJah2AHAMhQ7AFiGYgcAy1DsAGAZih0ALEOxA4BlKHYAsAzF\nDgCWodgBwDIUOwBYhmIHAMtQ7ABgGYodACxDsQOAZSh2ALAMxQ4AlqHYAcAyFDsAWIZiBwDLUOwA\nYBmKHQAsQ7EDgGUodgCwDMUOAJah2AHAMhQ7AFiGYgcAy1DsAGAZih0ALEOxA4BlKHYAsAzF3k8+\nn8/tCP0Sb3klMsdCvOWVyNwffRZ7fX29MjMzlZGRoVWrVvU6pqKiQhkZGcrJydG+ffv6tWy8ibc3\nV7zllcgcC/GWVyJzf4Qt9mAwqCVLlqi+vl7Nzc3atm2bDhw4EDKmrq5O7733nlpaWrR+/XqVl5dH\nvGw8am1tdTtCv8RbXonMsRBveSUy90fYYvf7/UpPT1daWpqSkpJUUlKi2trakDHbt2/X/PnzJUn5\n+fk6ceKEjhw5EtGy8Sje3lzxllcicyzEW16JzP2RGO7O9vZ2paamdl/3er1qamrqc0x7e7s+/PDD\nPpeVJI/HM+Dwbom3zPGWVyJzLAxK3hi/5nj7GUvuZA5b7JEGMsYM6MkHuhwA4OLCFntKSooCgUD3\n9UAgIK/XG3ZMW1ubvF6vzpw50+eyAIDBF3aOPTc3Vy0tLWptbVVnZ6dqampUVFQUMqaoqEibN2+W\nJDU2NmrUqFEaN25cRMsCAAZf2DX2xMREVVVVqbCwUMFgUKWlpcrKylJ1dbUkqaysTLNnz1ZdXZ3S\n09M1fPhwbdq0KeyyAIDo8hgmugHAKux5CgCWGVLFfvjwYd1777266667JEkdHR2aP3++7rvvPj37\n7LMupwuvra1NxcXFKi0tjYu9bI0xWrFihSoqKro/I4kHHR0dysvL044dO9yOEpHa2lrdd999Kikp\n0auvvup2nLDi6fdNiq+fbU+xeA8PqWK//vrr9fTTT3dff+mll3T33Xdr/fr12r59u4vJ+vbuu+9q\n7ty52rBhQ8hhFYaql19+We3t7briiiviamulxx9/XPfcc4/bMSJ2++23a/369Vq3bp1qamrcjhNW\nPP2+SfH1s+0pFu/hqBT7woULNW7cOE2ePDnk9v4eO6bnzk8JCQnRiHqBgWa/9dZbtX79en3zm9/U\nbbfdFpOs0sDzHjp0SNOmTdOPf/xjPfXUU7GKK2ngmV999VXdcMMNSk5OjlXUbpf6nn744Ye1ZMmS\naMe8QH9yu/H7dr6B/Jzd+tme05/MMXsPmyjYvXu3eeutt8xNN93UfVtXV5eZOHGiOXz4sOns7DQ5\nOTmmubnZbN682Sxbtsy0t7d3j73zzjuNMcY888wz5pVXXjHGGFNSUhKNqIOWffXq1Wb37t0h+Ydy\n3i1btpjnnnvOGGPM3XffHbO8l5J5xYoVZtmyZWbWrFnm9ttvN2fPnh3ymc+ePWsefPBBs2vXrphl\nHWhuN37fLiWv2z/bc/qTOVbv4agUuzHGHD58OOSF7tmzxxQWFnZff/TRR82jjz4assyxY8dMWVmZ\nmThxonnsscdMR0eHWbBggSkvLzfPPvtstKIOSvZ33nnHzJ071yxevNh897vfjVlWYwaW9/Tp06a0\ntNQsXbrUPPnkkzHLes5AMp/zs5/9zOzYsSPqGc83kMxr1qwxt9xyi1m8eLFZt25dzLL2FGlut37f\nzhdp3p/+9Keu/2zP6e97I9rv4bDbsQ+mSI47M2bMGK1bty7kto0bN8YkXziRZM/OztYLL7wQ62i9\niiTvVVddFfJ5htsiyXzOuYPOuS2SzBUVFaqoqIh1tLAulvvqq68eEr9v57tY3rVr12rp0qUuJru4\nvt4b0X4Px+zD03g8eM858ZY93vJKZI6leMsdb3kl9zPHrNgjOe7MUBVv2eMtr0TmWIq33PGWV3I/\nc8yKPZ6PHRNv2eMtr0TmWIq33PGWVxoCmaMxcV9SUmLGjx9vrrjiCuP1es3GjRuNMcbU1dWZr3zl\nK2bixInmkUceicZTX7J4yx5veY0hcyzFW+54y2vM0MzMsWIAwDJDas9TAMClo9gBwDIUOwBYhmIH\nAMtQ7ABgGYodACxDsQOAZSh2ALAMxQ4Alvl/YeQcyfzWDKYAAAAASUVORK5CYII=\n"
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## Use LogisticRegression on L1 and L2 norm\n",
      "penalties = ['l1', 'l2']\n",
      "Cs = np.logspace(0, 5, 6)\n",
      "lr = linear_model.LogisticRegression(dual = True) # more features than samples\n",
      "scores = [cross_val_score(lr.set_params(penalty=penalty, C=C, dual = True if penalty =='l2' else False), \n",
      "                            X, y, cv=make_cv(X.shape[0]), n_jobs=-1) \n",
      "            for penalty in penalties\n",
      "            for C in Cs]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sorted_scores = sorted(zip([(p,C) for p in penalties for C in Cs], cv_scores), key = lambda (params, s): s, reverse=True)\n",
      "print sorted_scores\n",
      "best_params, best_score = sorted_scores[0]\n",
      "print 'best alpha and best cv_score:', best_params, best_score\n",
      "plt.plot(cv_scores)\n",
      "l1_lr = linear_model.LogisticRegression(penalty='l1', C=1.0).fit(X, y)\n",
      "print 'coeff sparse rates for different classes', np.sum(abs(l1_lr.coef_) > 0, axis = 1) *1. / X.shape[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[(('l2', 1.0), 0.20499241756726785), (('l1', 1.0), 0.20099440758123391), (('l2', 10.0), 0.18500836165506826), (('l1', 10.0), 0.18199936463409516), (('l2', 100.0), 0.17201632770495046), (('l1', 100000.0), 0.16600133067198933), (('l1', 10000.0), 0.16599533665401928), (('l1', 1000.0), 0.15901230571889255), (('l2', 10000.0), 0.1590063117009225), (('l2', 1000.0), 0.15801430172687658), (('l2', 100000.0), 0.15800231369093645), (('l1', 100.0), 0.15101328873783962)]\n",
        "best alpha and best cv_score: ('l2', 1.0) 0.204992417567\n",
        "coeff sparse rates for different classes"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " [ 0.03733333  0.04053333  0.02986667  0.02986667  0.02666667  0.02293333\n",
        "  0.02933333  0.02293333  0.024     ]\n"
       ]
      },
      {
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD9CAYAAABdoNd6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtcVHXeB/DPIKPkjRBRiUFRIQFvoKChWWyp9LjKrmaJ\nWhZe8rF1n9y0bbuDlUplStlubGpXF3lqU1wvk/FKSjMlH0hSvJCJDCTe8pKXRIbz/PEL5OYwM5yZ\n35mZz/v14pXDnDnzPa/gw5nv+Z3fT6coigIiInJrXrILICIix2PYExF5AIY9EZEHYNgTEXkAhj0R\nkQdg2BMReYBmw95oNCI8PBxhYWFIS0tr9PyaNWswcOBADBgwAMOHD0dhYWHtc9OnT0fXrl3Rv39/\ndasmIiKbWAx7s9mMuXPnwmg0oqioCJmZmThw4EC9bXr16oWvvvoKhYWFeO655/DII4/UPpecnAyj\n0eiYyomIyGoWwz4vLw+hoaEICQmBXq9HUlISsrOz620TFxcHX19fAMDQoUNRVlZW+9yIESPg5+fn\ngLKJiMgW3paeLC8vR3BwcO1jg8GA3bt333D7VatWYcyYMVa/uU6ns3pbIiK6ztbJDyye2dsSxtu2\nbcPq1aub7OtboiiK23698MIL0mvg8fH4PPH43PnYFMW+GW4sntkHBQXBZDLVPjaZTDAYDI22Kyws\nxKxZs2A0Gtm2ISLSIItn9jExMSguLkZJSQkqKyuRlZWFxMTEetuUlpZiwoQJ+OijjxAaGurQYomI\nyD4Ww97b2xsrVqxAQkICIiMjMWnSJERERCAjIwMZGRkAgIULF+Ls2bOYM2cOoqOjMWTIkNrXT548\nGcOGDcPhw4cRHByMd99917FHozHx8fGyS3AoHp9rc+fjc+djs5dOsbcBpMab63R295+IiDyVPdnJ\nO2iJiDwAw56IyAMw7ImIPADDnojIAzDsiYg8AMOeiMgDMOyJiDwAw56IyAMw7ImIPADDnojIAzDs\niYg8AMOeiMgDMOyJbuDHHwGzWXYVROpg2BPdwNixQGam7CqI1MGwJ2rCpUvAwYPAe+/JroRIHQx7\noiYUFgJ9+wIFBcCxY7KrIWo5hj1RE/Lzgbg4YNIk4MMPZVdD1HIMe6ImFBQA0dHAww+LVg4XVCNX\nx7AnakJN2MfGAq1bA19/LbsiopZh2BM1UFkJHDgADBgA6HTXz+6JXBnDnqiBoiKgZ0+gbVvx+MEH\ngX//W4zQIXJVDHuiBmpaODUCA4Fhw4B16+TVRNRSDHuiBhqGPcBWDrk+hj1RA02F/bhxwHffccw9\nuS6GPVEd1dXA3r2Nw97HR4y5/+ADOXURtRTDnqiOH34A/P0BP7/Gz3HMPbkyhj1RHU21cGrExIgz\n/B07nFsTkRqkh/2nn8qugOg6S2HPMffkypoNe6PRiPDwcISFhSEtLa3R82vWrMHAgQMxYMAADB8+\nHIWFhVa/FgCmTwfOnWvBERCpKD8fGDToxs8/8IA4QeGYe3I5igVVVVVK7969laNHjyqVlZXKwIED\nlaKionrb7Ny5Uzl37pyiKIqyZcsWZejQoVa/FoBy332K8tZblqogco7qakXp3FlRysstbzdmjKJ8\n8IFzaiJqSjPR3SSLZ/Z5eXkIDQ1FSEgI9Ho9kpKSkJ2dXW+buLg4+Pr6AgCGDh2KsrIyq18LALNm\nAe+8w4teJF95OeDlJW6isoStHHJF3paeLC8vR3BwcO1jg8GA3bt333D7VatWYcyYMTa9dseOFJSU\nALNnA1OmxCM+Pt62IyBSSU2/XqezvN24ccCcOWLMfY8ezqmNPFtubi5yc3NbtA+LYa9r7qe+jm3b\ntmH16tX4+rfpAa19bWpqClq3BkpLAeY8ydRcv75G3TH3zz3n+LqI4uPrnwinpqbavA+LbZygoCCY\nTKbaxyaTCQaDodF2hYWFmDVrFjZs2AC/3wYoW/taAEhOBj7+GLh40eb6iVRjaSROQ8nJHHNPrsVi\n2MfExKC4uBglJSWorKxEVlYWEhMT621TWlqKCRMm4KOPPkJoaKhNr61xyy3AiBFAVpYKR0RkJ1vC\nfvBg4KabOOaeXIfFsPf29saKFSuQkJCAyMhITJo0CREREcjIyEBGRgYAYOHChTh79izmzJmD6Oho\nDBkyxOJrb6TmQi2RDGfOiCHAvXpZtz3H3JOr0f02jEfOm+t0qHn7qiogJATYvFksGkHkTDk5wMKF\nwFdfWf+a48eByEigrAxo185xtRE1VDc7rSX9Dtoa3t7iBiue3ZMMBQXWXZytKzAQGD5cLGxCpHWa\nCXsAmDED+Ne/gCtXZFdCnsaWfn1dbOWQq9BU2PfoAQwZAnzyiexKyNPYG/bjxgGFhUBJieolEalK\nU2EP8EItOd/Fi+IGKQvjB26oTRsgKYnz3JP2aS7sx40DiouBgwdlV0KeorAQ6NsX0Ovte31NK6e6\nWs2qiNSlubDX64GHHgJWrpRdCXkKe1s4NQYPBtq25Zh70jbNhT0AzJwpPhZfvSq7EvIELQ17jrkn\nV6DJsA8NBfr1A5qYJJNIdfn5LQt7AJg6FVi3jlN+kHZpMuwBXqgl56isFNeHWnojX82Ye668Rlql\n2bAfPx747jvgxx9lV0LurKgI6NlT9NxbqmZyNCIt0mzY+/iIJeBWrZJdCbmzlvbr6xo7Fvj+e465\nJ23SbNgDopWzejVw7ZrsSshdqdGvr8Ex96Rlmg77yEgxC+GmTbIrIXdlz5w4lnDMPWmVpsMe4IVa\ncpzqanFDVVSUevscNEjMgLl9u3r7JFKD5sP+/vuBXbuAOoteEanihx8Af3/gt8XVVMEx96RVmg/7\ntm1FH3T1atmVkLtRs19fF8fckxZpPuwB0cpZtQowm2VXQu5E7X59jW7dxDKbnOeetMQlwj4qCuja\nFdi6VXYl5E7UHHbZEFs5pDUuEfYAL9SSuhTFsWFfM+b+6FHH7J/IVi4T9pMnA9u2ARUVsishd1BW\nBnh5iWkOHKFNG/EzyzH3pBUuE/YdOgD33suPxqSOmrN6nc5x7/Hww8D773PMPWmDy4Q9IFo5K1fy\nl4dazlEXZ+saNAho355j7kkbXCrshwwRQzG3bZNdCbk6R/bra3DMPWmJS4W9TscLtaQOR42xb2jq\nVGD9eo65J/lcKuwBMROm0QicPi27EnJVZ84A58+LeZccrWtXjrknbXC5sPfzE4uSc5QD2augQNy7\n4eWkn362ckgLXC7sgeutHEWRXQm5Imf06+saOxbYt48L8ZBcLhn2I0aIoP/6a9mVkCtyVr++RuvW\nHHNP8jUb9kajEeHh4QgLC0NaWlqj5w8ePIi4uDj4+Phg6dKl9Z5LT09H//790a9fP6Snp6tWNC/U\nUks4+8we4Jh7ks9i2JvNZsydOxdGoxFFRUXIzMzEgQMH6m3j7++PN998EwsWLKj3/X379mHlypX4\n9ttvsXfvXmzcuBFHjhxRrfBp04DsbODcOdV2SR7g4kUxXXZEhHPfNzpa3Bj41VfOfV+iGhbDPi8v\nD6GhoQgJCYFer0dSUhKys7PrbRMQEICYmBjo9fp63z948CCGDh0KHx8ftGrVCnfeeSc+/fRT1QoP\nCAASEoA1a1TbJXmAwkKxAlqDH1eH45h7ks3b0pPl5eUIDg6ufWwwGLB7926rdtyvXz8888wz+Pnn\nn+Hj44NNmzZhyJAhjbZLSUmp/Xd8fDzi4+OtqxyilbNgAfDoo4697Z3ch7P79XVNnQosXCg+XbRv\nL6cGck25ubnIzc1t0T4shr2uBQkaHh6OJ598EqNHj0a7du0QHR0NrybGutUNe1vddRdw4QKwZw8Q\nG2v3bsiDFBQAMTFy3rtrV+COO4BPPhFn+UTWanginJqaavM+LLZxgoKCYKqzHqDJZILBYLB659On\nT8eePXvw5Zdf4uabb0afPn1sLtASLy9g5kzgn/9UdbfkxpwxJ44lbOWQLBbDPiYmBsXFxSgpKUFl\nZSWysrKQmJjY5LZKE4PeT548CQAoLS3FunXrMGXKFBVKri85WZwp/fKL6rsmN1NZCRw8CPTvL6+G\nsWOB/fs55p6cz2Ibx9vbGytWrEBCQgLMZjNmzJiBiIgIZGRkAABmz56NiooKxMbG4sKFC/Dy8kJ6\nejqKiorQvn17TJw4EWfOnIFer8ff//53dOzYUfUDCAwE7rwTWLtW9PCJbqSoCOjZU0ymJ0vr1sCU\nKWLMfQs6mEQ20ylNnZI76811uiY/Edhq0yYgNRXIy1OhKHJbq1cDX3wBfPSR3DoKCoAJE4AjR5w3\nZQO5F3uy0y1+1O65Bzh+HNi7V3YlpGUybqZqSlQU0LEjx9yTc7lF2LdqBUyfzjtqyTLZF2drcMw9\nyeAWbRwAOHZM/CKbTHJ7sqRN1dWAry9QWipmTpXt5Eng1lvFz2uHDrKrIVfjsW0cAOjRAxg6VIzM\nIWqouBjo3FkbQQ8AXbqIgQX8eSVncZuwBzg5Gt2YVvr1dbGVQ87kVmE/dizwww9Ag7naiDTTr6/r\n978Xw0FVnB+Q6IbcKuz1enG2tHKl7EpIa7R4Zl93zD2Ro7nNBdoaR44At90GlJUBbdqoumtyUYoi\nZkktLARuuUV2NfUVFADjx4s7ajnmnqzl0Rdoa/TuDQwYAKxbJ7sS0oqyMjE8NzBQdiWNRUWJUUJf\nfim7EnJ3bhf2AC/UUn01/XotToPNMffkLG4Z9uPHi4/svPBFgDb79XVNnSpWXeNkfuRIbhn2bdoA\nDz7IC7UkyFywxBpdugDx8RxzT47llmEPiFbOe+8B167JroRk0/qZPcBWDjme24Z9RIS4WLtxo+xK\nSKYzZ4Dz54FevWRXYtmYMWKu/cJC2ZWQu3LbsAd4oZbEWX1UlPaHNbZuDTz/PDBvnhgqSqQ2jf8K\ntMx99wG7donJr8gzab1fX9fs2WKCtPXrZVdC7sitw75tW3GH4urVsishWVyhX1/D2xtYvhxYsAD4\n9VfZ1ZC7ceuwB0QrZ9UqwGyWXQnJoMU5cSwZOVKskbt8uexKyN24fdgPHCjunDQaZVdCznbxopgv\nPjxcdiW2ee018fXTT7IrIXfi9mEP8EKtp9q7F4iMFBPkuZLQUGDmTODpp2VXQu7EI8I+KUnMPXL8\nuOxKyJlcqV/f0DPPAFu3At9+K7sSchceEfYdOgATJwLvviu7EnImVw77Dh2ARYuA//kfDsUkdXhE\n2AOilbNypViLlDyDq12cbWjaNKCqCvjXv2RXQu7AY8I+NlacLX3xhexKyBkqK8Udqf37y67Efl5e\nQHo68OST4mIzUUt4TNjrdOLsnmPuPcP+/UDPnuJeC1c2bJhYmDwtTXYl5Oo8JuwB4A9/AHJy2AP1\nBK7cr28oLQ34xz+AkhLZlZAr86iwDw4G2rUDDh2SXQk5mqv36+syGIDHHgP++lfZlZAr86iwB4A7\n7gC++kp2FeRorjQnjjUWLAB27+byhWS/ZsPeaDQiPDwcYWFhSGuicXjw4EHExcXBx8cHS5curffc\n4sWL0bdvX/Tv3x9TpkzB1atX1avcTgx792c2i6mCo6JkV6Kem24CXn1VnOFz6g+yh8WwN5vNmDt3\nLoxGI4qKipCZmYkDBw7U28bf3x9vvvkmFixYUO/7JSUleOedd5Cfn4/vv/8eZrMZa9euVf8IbHTH\nHeLsiH179/XDD0DnzoCfn+xK1HXffUDHjhxkQPaxGPZ5eXkIDQ1FSEgI9Ho9kpKSkJ2dXW+bgIAA\nxMTEQN/gnvSOHTtCr9fj8uXLqKqqwuXLlxEUFKT+EdgoNFSMXT52THYl5Cju1K+vS6cTQzGfew44\nd052NeRqvC09WV5ejuDg4NrHBoMBu3fvtmrHnTp1wvz589G9e3fcdNNNSEhIwMiRIxttl5KSUvvv\n+Ph4xMfHW1e5nXS6662ckBCHvhVJ4m79+rqio4HERODFF4EGXVNyY7m5ucjNzW3RPiyGvU6ns3vH\nR44cwfLly1FSUgJfX1/cd999WLNmDaZOnVpvu7ph7yw1YT9tmtPfmpygoECs+OSuXnpJTPD2yCNA\nnz6yqyFnaHginJqaavM+LLZxgoKCYDKZah+bTCYYDAardrxnzx4MGzYM/v7+8Pb2xoQJE7Bz506b\nC3QEXqR1X4riXmPsm9KlC/DUU8Djj8uuhFyJxbCPiYlBcXExSkpKUFlZiaysLCQmJja5rdLgimd4\neDh27dqFK1euQFEU5OTkIDIyUr3KW6BvX7EQNWfBdD9lZWLFp8BA2ZU41p//DBQXA1u2yK6EXIXF\nNo63tzdWrFiBhIQEmM1mzJgxAxEREcjIyAAAzJ49GxUVFYiNjcWFCxfg5eWF9PR0FBUVYeDAgZg2\nbRpiYmLg5eWFQYMG4ZFHHnHKQTXHywsYPhzYvh24/37Z1ZCaavr1LehAuoTWrYFly4C//EWsbuVq\nc/aT8+mUhqfkznxzna7RJwJnee01cfv5ihVS3p4cJCVFTIK2aJHsShxPUYAxY4CEBPe+RkGN2ZOd\nHncHbQ327d2Tu/fr69LpgNdfB15+GTh1SnY1pHUee2Z/7Rrg7y/O7jt1klICOUD37sC2bUDv3rIr\ncZ5584BffwXeflt2JeQsPLO3gV4P3HYbsGOH7EpILadPA+fPi6mNPckLLwDr1ok1d4luxGPDHmAr\nx90UFIj5cLw87Kfazw9ITRVn+JwGhG7Ew34t6mPYuxdP6tc3NGsW8PPPwKefyq6EtMqjw37IELGi\n0S+/yK6E1ODJYd+qFbB8uZgK+ddfZVdDWuTRYe/jAwweDHzzjexKSA35+e45AZq1fvc7cfyvvy67\nEtIijw57gK0cd3HxImAyAeHhsiuR69VXRdiXl8uuhLSGYc+wdwt794ppMDz9TtJevYDZs8XcOUR1\neXzYx8WJj//sc7o2T+7XN/TUU8AXXwC7dsmuhLTE48O+QwcxXWxenuxKqCU8vV9fV/v2YrqIxx4D\nqqtlV0Na4fFhD7CV4w54Zl/fAw+I/65ZI7cO0g6GPRj2rq6yEjh0COjfX3Yl2uHlJZYw/NvfxMVr\nIoY9gNtvF/3Na9dkV0L22L9fTJHQtq3sSrTlttuAu+8GFi+WXQlpAcMeYiK0kBDRCiDXw379jS1e\nDGRkAEePyq6EZGPY/4atHNfFfv2NBQWJBU6eeEJ2JSQbw/43DHvXxbC37PHHgf/7PzH1M3kuj53P\nvqHjx8VNOadPe96sia7MbAZuvhkoLRWzP1LT/v1vYOFCEfreFhcjJVfA+exbIDAQ6NwZ2LdPdiVk\ni+JiICCAQd+cCRPEtamVK2VXQrIw7OtgK8f1sIVjHZ1OzIr5wgvA2bOyqyEZGPZ13HEHsH277CrI\nFgx76w0cCIwfL9o55HkY9nXUnNlr5DICWYFhb5sXXwQ++gg4cEB2JeRsDPs6evQQsyb+8IPsSsga\nisIx9rYKCACeeUYMx+RJjWdh2Neh07Fv70pMJvHHOTBQdiWu5U9/Ao4dAzZvll0JORPDvgGGvetg\nC8c+ej2wbJk4u6+slF0NOQvDvgGGvetg2NvvnnvEfEIciuk5GPYN9OkDXLokbtIhbWPYt8ySJcBL\nL3FWTE/BsG+gpm/PIZjax4uzLRMdDcTHi/H35P6aDXuj0Yjw8HCEhYUhLS2t0fMHDx5EXFwcfHx8\nsHTp0trvHzp0CNHR0bVfvr6+eOONN9St3kHYytG+06eBCxdEK4Ls9+KLIuxPn5ZdCTmaxblxzGYz\n+vTpg5ycHAQFBSE2NhaZmZmIiIio3ebUqVM4duwY1q9fDz8/P8yfP7/RfqqrqxEUFIS8vDwEBwdf\nf3MNzY1T13ffAZMncyyyln3+uWhBfPml7Epc39y5QOvWwOuvy66ErKX63Dh5eXkIDQ1FSEgI9Ho9\nkpKSkJ2dXW+bgIAAxMTEQK/X33A/OTk56N27d72g17L+/YGKCuDkSdmV0I2wX6+e554D3n9fDMck\n92Vx/rvy8vJ6AW0wGLB7926b32Tt2rWYMmVKk8+lpKTU/js+Ph7x8fE2719trVoBw4aJvv2998qu\nhpqSnw+MGSO7CvfQtasYe//CC8B778muhpqSm5uL3NzcFu3DYtjrdLoW7RwAKisr8Z///KfJfj9Q\nP+y1pKZvz7DXpoICcScoqWPBAiAsDPj+e67lq0UNT4RTU1Nt3ofFNk5QUBBMJlPtY5PJBIPBYNMb\nbNmyBYMHD0ZAQIDNxcnEi7TadfGiuHs2PFx2Je6jY0fgqaeAp5+WXQk5isWwj4mJQXFxMUpKSlBZ\nWYmsrCwkJiY2ue2NLhZkZmZi8uTJLa/UyQYPFnPknDsnuxJqaO9esdCMhctEZIc5c8SZ/Y4dsish\nR7AY9t7e3lixYgUSEhIQGRmJSZMmISIiAhkZGcjIyAAAVFRUIDg4GMuWLcNLL72E7t274+Jvd2lc\nunQJOTk5mDBhguOPRGWtWwNDhgBffy27EmqI4+sdo00bMf3xk09ykjR3xGUJLUhNBS5fBm5wuYEk\nmT5d/CH+7/+WXYn7MZvFKKeXXgJu8CGeNIDLEqqMfXtt4rBLx2nVCli8WPTuzWbZ1ZCaeGZvweXL\nYv7vkyeBdu1kV0MAcPWqWGD8zBmgbVvZ1bgnRQHuvFN8gnr4YdnVUFN4Zq+ytm2BqChg1y7ZlVCN\n/fuB3r0Z9I6k04nW5fPPA7/+KrsaUgvDvhls5WgLWzjOERcnLoK/9ZbsSkgtDPtmMOy1hWHvPIsW\niTP88+dlV0JqYNg3Y9gw4NtvRa+Y5GPYO09kJDB2LPDKK7IrITXwAq0VBg8G3ngDGD5cdiWezWwG\nfH2BsjJxkZYcz2QS16327eNav1rCC7QOwlaONhQXA126MOidKTgYSE4WN1uRa2PYW4Fhrw1s4cjx\n1FPAJ5+IP7bkutjGscKpU0BoqBjb7W1xnlDPcvasmD+orqb+dzb8nr3bZGSImRmffda2OqnlFi8W\ni/pkZcmuhAD7spPRZYWAAMBgEBNwDR4suxptOHQIGD0a8PcXd13W1dTM2A2/Z802Db+n0wF//rPt\ntVLLPfaY+EO7Zw8QEyO7GrIHw95KNa0chr2YiOz3vwdeflncZUnur21bcZPV3/4G5OTIrobswZ69\nldi3F776CrjnHnGzDYPes0yfDpSWivV/yfWwZ2+lsjJxcfDkyabbDZ5g0yYxV0pmJjBypOxqSIaP\nPwaWLBH3nnjxVFEaDr10IINBrOZz4IDsSuTIzARmzAA2bmTQe7KJE0XIf/yx7ErIVgx7G3hqK+cf\n/wCeeEL0aocOlV0NyaTTiTP7Z58Frl2TXQ3ZgmFvA08Le0URF2Ffe00cd79+sisiLbj7bqBnT2Dl\nStmVkC3Ys7fBkSNinm+Tyf379ooizuY/+wzYupW3ylN9+fli3pzDh4H27WVX43nYs3ewXr1ECB49\nKrsSx6qqAmbOFOvvfvklg54aGzRInPgsXy67ErIWw94GOp37t3KuXgUmTRKfXj7/HOjUSXZFpFUv\nvijC/vRp2ZWQNRj2NnLnsL94UXw01+mA//yHH8/JstBQcWKwaJHsSsga7NnbaP9+4A9/aDwnjKv7\n+WdxV2xkpJiDhnMAkTUqKoC+fUUPv0cP2dV4DvbsnSAiAjh3Digvl12Jeo4fF/3X4cPFCAsGPVmr\nWzfg0UeBF16QXQk1h2FvIy8vYMQIYPt22ZWo48cfgdtvByZPBl591f1HGZH6nngC2LJFLHBC2sWw\nt4O79O337RPHMn8+8PTTDHqyT8eOYoK0p5+WXQlZwrC3gzuE/e7d4uaYV14RH8OJWmLOHDEF+I4d\nsiuhG+EFWjtUVYl53I8cATp3ll2N7XJygClTgHffFRdlidTwwQfi4v6OHfyU6Gi8QOsk3t5AXJxr\nnsV8+qkI+k8+YdCTuqZOBS5cEMN2SXsY9nZyxVbOu+8Cf/oTYDSK+onU1KqVWL7w6acBs1l2NdRQ\ns2FvNBoRHh6OsLAwpKWlNXr+4MGDiIuLg4+PD5YuXVrvuXPnzmHixImIiIhAZGQkdu3apV7lkrla\n2C9bBqSkALm54lZ3Ikf4/e8BPz/gww9lV0INWezZm81m9OnTBzk5OQgKCkJsbCwyMzMRERFRu82p\nU6dw7NgxrF+/Hn5+fpg/f37tcw899BDuvPNOTJ8+HVVVVbh06RJ8fX2vv7mL9uwBMa2Avz/w009i\nNIJWKYpYTu5//1dMf9C9u+yKyN3t3CmG8h46BPj4yK7GPanes8/Ly0NoaChCQkKg1+uRlJSE7Ozs\netsEBAQgJiYGer2+3vfPnz+P7du3Y/pva9d5e3vXC3pX16aNWHh5507ZldxYdbVYoHvTJnFfAIOe\nnGHYMCAqCvj732VXQnVZvFeyvLwcwcHBtY8NBgN2795t1Y6PHj2KgIAAJCcnY+/evRg8eDDS09PR\ntm3betulpKTU/js+Ph7x8fHWVy9ZTSvnnntkV9LYtWtiCcGyMmDbNsCN/s6SC1i0CPjd78TqZvzZ\na7nc3Fzk5ua2aB8Ww17XgvFTVVVVyM/Px4oVKxAbG4t58+ZhyZIlWLhwYb3t6oa9q7njDtEH15or\nV4D77hP/NhqBm26SWw95nr59Rf/+lVfEAjjUMg1PhFNTU23eh8U2TlBQEEwmU+1jk8kEg8Fg1Y4N\nBgMMBgNiY2MBABMnTkR+fr7NBWpZXBxQUCDCVSvOnxefNHx9gXXrGPQkT2oq8PbbYu4lks9i2MfE\nxKC4uBglJSWorKxEVlYWEhMTm9y24cWCbt26ITg4GIcPHwYA5OTkoG/fviqVrQ3t2gH9+4u7UbXg\n1CngrrvE8oEffgg0uIxC5FTduwPJyUCDD/MOV10tBk7s2yfG/ZPQ7B20W7Zswbx582A2mzFjxgw8\n9dRTyMjIAADMnj0bFRUViI2NxYULF+Dl5YUOHTqgqKgI7du3x969ezFz5kxUVlaid+/eePfdd91m\nNE6Nv/5VzPv+/PNy6zh7VvRI/+u/RL+UdzCSFpw5A/TpA3zzDRAWpt5+L14UK8b9+OP1r5rHJSVi\nhJyfH1BaCrRtK1aZq/nq3fv6v4OCxP0Brsae7OR0CS20caNYrScnR14Nly4Bo0YBQ4cCr7/OoCdt\nWbRIzJtsUj2/AAAJ4klEQVSTlWX9a8xmMY143TCvG+i//AKEhNQP8ZqvkJDrC+8oCnDypJjapO5+\nah6fOSPm4W/qD0GvXtpdwIdhL8HZs+Lj6pkzQOvWzn//q1eBceMAg0HMRe/Fe6JJYy5dAm69Fdiw\nARg8+Pr3z59vHOY1gV5aKu5jaSrMe/UCunZV52f9ypXGnxBq/hAcPQp06HDjTwW33CLv941hL0lU\nlLgQddttzn3fqiqxLJxOB6xdy0VHSLvefluMuw8Pvx6uV6/eOMx79JA/uKC6Gjhxov6ngrr/Pneu\n/qeL4cOBpCTn1GZPdjIeVFAz3t6ZYV9dDcyaJXqXGzYw6EnbZswQ//X1FcHYsycQEKDtlqOXFxAY\nKL5uv73x85cu1f9UcPmy82u0Bc/sVfDJJ8B774n+vTMoCjBvHrBnD7B1qxgVRESeg20cSU6cEB9P\nT592zpX9lBQgO1vcGXvzzY5/PyLSFs5nL0nXruLLGWtwLlsGZGYCn33GoCci6zHsVeKMKY9XrxbD\nPD//HOjSxbHvRUTuhWGvEkeH/SefAM8+y2mKicg+7NmrpLQUiI0FKirUH2Hw2WfAgw+Ki7FRUeru\nm4hcD3v2EnXvLsYF/zYVkGp27AAeeEBMasagJyJ7MexVpHYrp6AAmDABWLNG3LBBRGQvhr2K1Az7\nQ4eAMWOAf/wDGD1anX0Skedi2KtIrbAvLRUBv2gRcO+9Ld8fERHDXkVhYWK+j2PH7N/HiRPAyJHA\n44+LucCJiNTAsFeRTteys/uzZ4GEBHFB9rHH1K2NiDwbw15l9ob9pUtizc677gKee079uojIszHs\nVWZP2F+9CowfD0REAEuXansmQCJyTbypSmXV1UDnzkBREdCtW/PbV1UB998vJlBbu9Y1l0gjIufi\nTVUa4OUl5r7evr35baurgZkzxTzYH33EoCcix2HYO4A1rZyaOel/+AH497+BNm2cUxsReSaGvQNY\nE/YpKeLsf+NGLj5CRI7Hnr0DXLsGdOokxtt36tT4+ddfBzIyRNhzqmIishV79hqh14v1aL/+uvFz\nq1YBb7wB5OQw6InIeRj2DtJUK+fjj8UY+q1bgeBgOXURkWdi2DtIw7A3GoG5c4EtW4Bbb5VXFxF5\nJvbsHeTKFTHe/sQJ4LvvxE1T2dnAsGGyKyMiV8eevYbcdBMwaBDw1ltiTvp//YtBT0TyMOwdqEeP\nXDz9NPD228CoUbKrUV9ubq7sEhyKx+e63PnY7NVs2BuNRoSHhyMsLAxpaWmNnj948CDi4uLg4+OD\npUuX1nsuJCQEAwYMQHR0NIYMGaJe1S6ic+dcbNokzuzdkbv/QvH4XJc7H5u9vC09aTabMXfuXOTk\n5CAoKAixsbFITExERERE7Tb+/v548803sX79+kav1+l0yM3NRaemBpt7gJtvBu65R3YVRETNnNnn\n5eUhNDQUISEh0Ov1SEpKQnZ2dr1tAgICEBMTA71e3+Q+3PUCLBGRS1Es+Pjjj5WZM2fWPv7www+V\nuXPnNrltSkqK8tprr9X7Xs+ePZWoqChl8ODByj//+c9GrwHAL37xi1/8suPLVhbbOLoWTqz+9ddf\nIzAwEKdOncKoUaMQHh6OESNG1D7Ps34iIuew2MYJCgqCyWSqfWwymWAwGKzeeWBgIADR6hk/fjzy\n8vLsLJOIiFrCYtjHxMSguLgYJSUlqKysRFZWFhITE5vctuFZ+uXLl/HLL78AAC5duoStW7eif//+\nKpVNRES2sNjG8fb2xooVK5CQkACz2YwZM2YgIiICGRkZAIDZs2ejoqICsbGxuHDhAry8vJCeno6i\noiKcPHkSE34bc1hVVYWpU6di9OjRjj8iIiJqzOYuv0q2bNmi9OnTRwkNDVWWLFkiqwyHKC0tVeLj\n45XIyEilb9++Snp6uuySHKKqqkqJiopSxo4dK7sU1Z09e1a59957lfDwcCUiIkL55ptvZJekqkWL\nFimRkZFKv379lMmTJyu//vqr7JLslpycrHTp0kXp169f7ffOnDmjjBw5UgkLC1NGjRqlnD17VmKF\nLdPU8S1YsEAJDw9XBgwYoIwfP145d+5cs/uRcgdtzfh9o9GIoqIiZGZm4sCBAzJKcQi9Xo9ly5Zh\n//792LVrF9566y23Or4a6enpiIyMbPGFfC167LHHMGbMGBw4cACFhYX17i1xdSUlJXjnnXeQn5+P\n77//HmazGWvXrpVdlt2Sk5NhNBrrfW/JkiUYNWoUDh8+jLvvvhtLliyRVF3LNXV8o0ePxv79+7F3\n717ceuutWLx4cbP7kRL21ozfd2XdunVDVFQUAKB9+/aIiIjATz/9JLkqdZWVlWHz5s2YOXOm242q\nOn/+PLZv347p06cDEO1MX19fyVWpp2PHjtDr9bh8+TKqqqpw+fJlBAUFyS7LbiNGjICfn1+9723Y\nsAEPPfQQAOChhx5q8qZPV9HU8Y0aNQpeXiK+hw4dirKysmb3IyXsy8vLEVxnQneDwYDy8nIZpThc\nSUkJCgoKMHToUNmlqOovf/kLXn311dofOHdy9OhRBAQEIDk5GYMGDcKsWbNw+fJl2WWpplOnTpg/\nfz66d++OW265BTfffDNGjhwpuyxVnThxAl27dgUAdO3aFSdOnJBckeOsXr0aY8aMaXY7Kb+p7vix\nvykXL17ExIkTkZ6ejvbt28suRzUbN25Ely5dEB0d7XZn9YAYUJCfn49HH30U+fn5aNeunUu3ARo6\ncuQIli9fjpKSEvz000+4ePEi1qxZI7ssh9HpdG6bOS+//DJat26NKVOmNLutlLBv6fh9V3Dt2jXc\ne++9eOCBB/DHP/5Rdjmq2rlzJzZs2ICePXti8uTJ+OKLLzBt2jTZZanGYDDAYDAgNjYWADBx4kTk\n5+dLrko9e/bswbBhw+Dv7w9vb29MmDABO3fulF2Wqrp27YqKigoAwPHjx9HFDdcAfe+997B582ar\n/1BLCXtbxu+7IkVRMGPGDERGRmLevHmyy1HdokWLYDKZcPToUaxduxZ33XUXPvjgA9llqaZbt24I\nDg7G4cOHAQA5OTno27ev5KrUEx4ejl27duHKlStQFAU5OTmIjIyUXZaqEhMT8f777wMA3n//fbc7\n4TIajXj11VeRnZ0NHx8f617kqOFCzdm8ebNy6623Kr1791YWLVokqwyH2L59u6LT6ZSBAwcqUVFR\nSlRUlLJlyxbZZTlEbm6uMm7cONllqO67775TYmJibBra5krS0tJqh15OmzZNqayslF2S3ZKSkpTA\nwEBFr9crBoNBWb16tXLmzBnl7rvvdouhlw2Pb9WqVUpoaKjSvXv32nyZM2dOs/uRuiwhERE5h/sN\npSAiokYY9kREHoBhT0TkARj2REQegGFPROQBGPZERB7g/wHhvN8p59QU/AAAAABJRU5ErkJggg==\n"
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## stochastic Logistic Regression for stability selection\n",
      "## ITS a feature selection method without making predictions and scoring on y\n",
      "Cs = np.logspace(0, 5, 6)\n",
      "rlr = linear_model.RandomizedLogisticRegression(sample_fraction=0.75, n_resampling=200, \n",
      "                                selection_threshold=0.25, n_jobs=-1,\n",
      "                                memory= './data/tmp/')\n",
      "rlr.set_params(C=1.0).fit(X, y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "________________________________________________________________________________\n",
        "[Memory] Calling sklearn.linear_model.randomized_l1._resample_model...\n",
        "_resample_model(<function _randomized_logistic at 0x111b1cc80>, array([[ 0.010795, ...,  0.011825],\n",
        "       ..., \n",
        "       [ 0.006166, ..., -0.009114]]), \n",
        "array([1, ..., 4]), C=1.0, n_jobs=-1, verbose=False, fit_intercept=True, scaling=0.5, n_resampling=200, random_state=None, sample_fraction=0.75, tol=0.001, pre_dispatch='3*n_jobs')"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "__________________________________________________resample_model - 39.3s, 0.7min\n"
       ]
      },
      {
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "RandomizedLogisticRegression(C=1.0, fit_intercept=True, memory='./data/tmp/',\n",
        "               n_jobs=-1, n_resampling=200, normalize=True,\n",
        "               pre_dispatch='3*n_jobs', random_state=None,\n",
        "               sample_fraction=0.75, scaling=0.5, selection_threshold=0.25,\n",
        "               tol=0.001, verbose=False)"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print 'sparseness rate for randomizedlogistic regression', np.sum(abs(rlr.scores_) > 0) * 1./ X.shape[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sparseness rate for randomizedlogistic regression 0.1328\n"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## Passive Aggressive Classifier\n",
      "Cs = np.logspace(-5, 5, 11)\n",
      "pa = linear_model.PassiveAggressiveClassifier(random_state = 0, n_iter = 50, n_jobs = 1, loss = 'hinge')\n",
      "scores = [cross_val_score(pa.set_params(C=C), X, y, n_jobs=-1, cv=make_cv(X.shape[0])) for C in Cs]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.semilogx(Cs, cv_scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "[<matplotlib.lines.Line2D at 0x109287f10>]"
       ]
      },
      {
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEECAYAAADAoTRlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH71JREFUeJzt3XtQVFeeB/BvI51g1KAiaxyahAhE8AGijSxxLHEmG6bM\nFkmMj44zeYnKWkUcZuLmWVOilUpCZtwdlcmERLEm0SCbGiNJjG2FLG1SJkIyqKgNkcnSsSWL5qFx\ngolAc/eP3iY0j6Yf9/Y9t+/3U0UV3X2771eEH4ffOfe0QZIkCUREpCtRagcgIqLwY/EnItIhFn8i\nIh1i8Sci0iEWfyIiHWLxJyLSoRGLv9VqRVpaGlJTU1FWVjbo8T179iAzMxMZGRmYP38+mpqaAABO\npxOLFi3CjBkzMHPmTGzbtk3+9EREFBSDr3X+LpcL06ZNQ21tLRISEpCdnY2qqiqkp6f3HfPRRx9h\n+vTpiI2NhdVqRWlpKY4ePYqOjg50dHRg9uzZ+O677zB37lzs37/f67lERKQOnyP/hoYGpKSkICkp\nCUajERaLBTU1NV7H5ObmIjY2FgCQk5ODc+fOAQBuuOEGzJ49GwAwduxYpKen44svvlDi30BERAGK\n9vVge3s7EhMT+26bTCbU19cPe/zOnTuxePHiQfc7HA4cO3YMOTk5XvcbDIZA8xIREYBQN2fwOfIP\npDjX1dWhsrJy0LzAd999h6VLl2Lr1q0YO3bsoOdJkjTix8aNG0M6bqj7+98X6usHm0OEDKLkEC2D\nKDlEyCBKDhEyiJJDDj5H/gkJCXA6nX23nU4nTCbToOOampqwZs0aWK1WTJgwoe/+7u5u3HPPPfjV\nr36Fu+66K+iQeXl5IR031P3+vqaSOUTIIEoOrWYQJYcIGUTJIUIGkXIMS/Khu7tbmjp1qtTW1iZd\nvXpVyszMlOx2u9cxn3/+uZScnCx99NFHXvf39vZK9913n1RSUjLs649w+rDZuHGj2hGEyCBJYuQQ\nIYMkiZFDhAySJEYOETJIkhg55KidPkf+0dHRKC8vR35+PlwuFwoLC5Geno6KigoAQFFRETZv3oyL\nFy9i3bp1AACj0YiGhgYcOXIEu3fvRkZGBrKysgAAzz77LH7xi1/I+9tLBrL/RtVoBkCMHCJkAMTI\nIUIGQIwcImQAxMkRKp9LPRU/ucEgW/+KiEgv5KidvMKXiEiHWPyJiHSIxZ+ISIdY/ImIdIjFn4hI\nh1j8BeF0Avv3q52CiPSCSz0FcPIksHgx0NMDfPEFwC2PiMgXLvWMAHV1wM9/Djz/POByuYs/EZHS\nWPxVtHcvsGIFUF0N3HsvYDYDn3yidioi0gMWf5X8x38A//7vwHvvAYsWue8zm4GPP1Y3FxHpA4t/\nmPX2Ar/5DVBZCXz4ITBr1o+PZWdz5E9E4eFzYzeS1w8/APffD5w/D3zwAdBv92sAwNy57uIvSZz0\nJSJlceQfJhcvAvn57s8PHRpc+AHgJz8Brr0WcDjCGo2IdIjFPwycTuCnPwXmzHFP8sbEDH8sWz9E\nFA4s/gpragJuvRUoLAT+8z+BqBG+4pz0JaJwYPFXUF0dcNttwO9/D/z2t/49hyN/IgoHXuGrkL17\ngfXr3Wv4PUs5/fHVV0BysnuOYKS/EohIn3iFr4AkCdiyZfAafn9NmgRMnAi0tiqTj4gI4FJPWfX2\nuts7tbXuNfyJicG9jqf1M22avPmIiDw48pfJDz8AFgtw7Jh7DX+whR/gNg9EpDwWfxn4s4Y/EFzx\nQ0RKY/EPUSBr+P01dy5w/Lh7i2ciIiWw+Icg0DX8/oqNBRISgOZmeV6PiGggFv8gBbOGPxBc709E\nSmLxD0L/ffgtFmXOwUlfIlISi38AQl3DHwhO+hKRkniFr5/6r+E/eDC0pZz+6OwE4uOBS5eAa65R\n9lxEpC28wjdM5FzD768xY9zbPJw6pfy5iEh/Riz+VqsVaWlpSE1NRVlZ2aDH9+zZg8zMTGRkZGD+\n/Ploamrqe2zVqlWYPHkyZvV/uyqNkXsNfyDY+iEipfgs/i6XC8XFxbBarbDb7aiqqkLzgPWHU6dO\nxfvvv4+mpib87ne/w9q1a/see+ihh2C1WpVJHgZKrOEPBFf8EJFSfBb/hoYGpKSkICkpCUajERaL\nBTU1NV7H5ObmIjY2FgCQk5ODc+fO9T22YMECTAjnUFlGSq3hDwRH/kSkFJ8bu7W3tyOxX4PbZDKh\nvr5+2ON37tyJxYsXBxSgtLS07/O8vDzk5eUF9Hwl1NW5l3Ju26bcUk5/ZGQAZ84A338PjB6tXg4i\nUpfNZoPNZpP1NX0Wf0MA7yJeV1eHyspKHDlyJKAA/Yu/CILdh18JMTFAejpw4gTwz/+sbhYiUs/A\ngfGmTZtCfk2fzYyEhAQ4nc6+206nEyaTadBxTU1NWLNmDd58803NtnnCuYY/EGz9EJESfBZ/s9mM\n1tZWOBwOdHV1obq6GgUFBV7HnD17FkuWLMHu3buRkpKiaFil9PYCv/kNsGuXex9+kRYncdKXiJTg\ns/hHR0ejvLwc+fn5mD59OlasWIH09HRUVFSgoqICALB582ZcvHgR69atQ1ZWFubNm9f3/HvvvRe3\n3norzpw5g8TEROzatUvZf02QHnkkvGv4A8FtHohICbq/wleSgClTgI8+Am6+WdUoQ+ruBsaPBzo6\ngHHj1E5DRCLgFb4y+PRT98SqiIUfAIxG96qfY8fUTkJEkUT3xd9mAwRYXeoTJ32JSG4s/jZtFH/2\n/YlITrou/pKkjeLPFT9EJDddF39Pvz8pSe0kvk2bBpw/795kjohIDrou/loY9QPAqFFAVhbwt7+p\nnYSIIgWLf57aKfzDvj8RyUm3xV8r/X4PrvghIjnptvhrpd/vwUlfIpKTbou/lkb9gPstHS9fBi5c\nUDsJEUUCFn+NMBiAuXM5+icieeiy+Gut3+/B1g8RyUWXxb+lRVv9fg9O+hKRXHRZ/LU46gd+XO6p\n8kaoRBQBWPw15MYbAZcL+OILtZMQkdbprvhrtd8PuCd92fohIjnorvi3tADXXae9fr8HJ32JSA66\nK/5aHfV7cORPRHJg8dcYTvoSkRx0Vfw9/f6FC9VOErwpU4DRowGHQ+0kRKRluir+Wu/3e7D1Q0Sh\n0lXx13rLx4PbOxNRqFj8NYgrfogoVAZJUm/q0GAwIFynlyTghhuA+nrtt32++sq9y+fFi0CUrn59\nExEgT+3UTemIlH4/AEyaBMTFAa2taichIq3STfGPlJaPByd9iSgULP4axUlfIgqFLop/JKzvH4iT\nvkQUihGLv9VqRVpaGlJTU1FWVjbo8T179iAzMxMZGRmYP38+mpqa/H5uuDQ3R06/32POHOD4caCn\nR+0kRKRJkg89PT1ScnKy1NbWJnV1dUmZmZmS3W73OubDDz+ULl26JEmSJB08eFDKycnx+7kjnF42\nf/qTJD34YFhOFVa33CJJTU1qpyCicJOjdvoc+Tc0NCAlJQVJSUkwGo2wWCyoqanxOiY3NxexsbEA\ngJycHJw7d87v54ZLpPX7Pdj6IaJgRft6sL29HYmJiX23TSYT6uvrhz1+586dWLx4cUDPLS0t7fs8\nLy8PeTJXaU+///e/l/VlheBZ8fPQQ2onISIl2Ww22Gw2WV/TZ/E3GAx+v1BdXR0qKytx5MiRgJ7b\nv/grobkZGDsWuOkmRU+jiuxs4LXX1E5BREobODDetGlTyK/ps/gnJCTA6XT23XY6nTCZTIOOa2pq\nwpo1a2C1WjFhwoSAnqu0SG35AMDs2cCpU0BXF3DNNWqnISIt8dnzN5vNaG1thcPhQFdXF6qrq1FQ\nUOB1zNmzZ7FkyRLs3r0bKSkpAT03HCK5+I8Z497m4eRJtZMQkdb4HPlHR0ejvLwc+fn5cLlcKCws\nRHp6OioqKgAARUVF2Lx5My5evIh169YBAIxGIxoaGoZ9bjhFcr/fwzPpO3eu2kmISEsiemM3ux34\n138F/ud/FDuF6l54AWhsBHbsUDsJEYULN3YbQSS3fDy4zQMRBYPFX+MyM4EzZ4Dvv1c7CRFpScQW\n/0jcz2co114LpKe7t3ogIvJXxBb/SF7fPxBbP0QUqIgt/npo+XhwmwciChSLfwTgG7sQUaAicqmn\nJAGTJ7sLoh7aPt3dwPjxQEcHMG6c2mmISGlc6jkMPfX7AcBoBDIy3Ov9iYj8EZHFX08tHw9O+hJR\nICKy+NfV6a/4c9KXiAIRccVfkoDDh/VX/DnpS0SBiLjib7e7Jz1vvFHtJOE1bRpw/jxw8aLaSYhI\nCyKu+Oux3w8Ao0a539T9b39TOwkRaQGLfwRh64eI/BVRxd/T74/0/XyGw0lfIvJXRBV/vfb7PTjy\nJyJ/RVTx13PLB3C/peM//uGe+CUi8oXFP4IYDO7RPyd9iWgkEVP89d7v92Drh4j8ETHFX+/9fg9u\n80BE/oiY4q/3lo+HZ8WPenu1EpEWsPhHmMREwOUC2tvVTkJEIouI4s9+/488k75s/RCRLxFR/Nnv\n95adzUlfIvItIoo/Wz7eOPInopGw+EcgT/HnpC8RDUfzxb+3l8V/oClTgNGjgbY2tZMQkag0X/zt\ndiA21r3KhX7E1g8R+TJi8bdarUhLS0NqairKysoGPd7S0oLc3FzExMRgy5YtXo9t3boVs2bNwsyZ\nM7F161b5UvfDUf/QuMMnEfnis/i7XC4UFxfDarXCbrejqqoKzc3NXsfExcVh+/bt2LBhg9f9p06d\nwo4dO/Dxxx/jxIkTePvtt/HZZ5/J/g9g8R8at3kgIl98Fv+GhgakpKQgKSkJRqMRFosFNTU1XsfE\nx8fDbDbDaDR63d/S0oKcnBzExMRg1KhRWLhwIfbt2ydr+N5eru8fzty5QGOj+2tERDRQtK8H29vb\nkdivmW4ymVBfX+/XC8+cORNPPfUUvvnmG8TExODAgQOYN2/eoONKS0v7Ps/Ly0NeAMN49vuHN2kS\nEBcHnDkDpKWpnYaIQmGz2WCz2WR9TZ/F32AwBP3CaWlpeOyxx3D77bdjzJgxyMrKQlTU4D80+hf/\nQLHl45tn0pfFn0jbBg6MN23aFPJr+mz7JCQkwOl09t12Op0wmUx+v/iqVavwySef4PDhwxg/fjym\nTZsWfNIhsPj7xklfIhqOz+JvNpvR2toKh8OBrq4uVFdXo6CgYMhjpSGuKLpw4QIA4OzZs3jjjTew\ncuVKGSK7sd8/Mk76EtFwfLZ9oqOjUV5ejvz8fLhcLhQWFiI9PR0VFRUAgKKiInR0dCA7OxuXL19G\nVFQUtm7dCrvdjrFjx2Lp0qX4+uuvYTQa8cILL+D666+XLTj7/SObMwc4cQLo6QGiff5PE5HeGKSh\nhuzhOrnBMORfDP4oLweOHwd27JA5VIRJSwNefx2YNUvtJEQkl1Bqp4dmr/Blv98/bP0Q0VA0WfzZ\n7/cft3kgoqFosviz3+8/7u1PREPRZPFny8d/s2cDp08DV6+qnYSIRKLZ4r9okdoptGHMGCA5GTh1\nSu0kRCQSzRV/9vsDx9YPEQ2kueJ/+jQwfjwQwIXGusdJXyIaSHPFn/3+wHGbByIaiMVfBzIy3Lt7\nXrmidhIiEoWmij/7/cG59logPd291QMREaCx4n/6NDBhAvv9weCkLxH1p6niz5ZP8DjpS0T9sfjr\nBIs/EfWnmV09e3uBf/on906ebPsErrvbvUS2owMYN07tNEQUCl3t6sl+f2iMRveqn8ZGtZMQkQg0\nU/zZ8gkd1/sTkQeLv45wb38i8tBE8ef6fnlw0peIPDRR/Nnvl8e0acCFC8DFi2onISK1aaL4s+Uj\nj1GjgKwsjv6JSEPFn/v3y4OtHyICNFD82e+XF7d5ICJAA8X/9Glg4kQgIUHtJJGBI38iAjRQ/Nnv\nl1dyMvCPfwDnz6udhIjUJHzxr6tj8ZeTwcDRPxEJXvzZ71cGiz8RCV38T50C4uLY75cbt3kgIqGL\nP/v9yvBs86Defq5EpDYWfx1KTHS31Nrb1U5CRGoZsfhbrVakpaUhNTUVZWVlgx5vaWlBbm4uYmJi\nsGXLFq/Hnn32WcyYMQOzZs3CypUrcfXqVb+Dsd+vHIOB6/2J9M5n8Xe5XCguLobVaoXdbkdVVRWa\nm5u9jomLi8P27duxYcMGr/sdDgdefvllNDY24uTJk3C5XNi7d6/fwdjvVxYnfYn0zWfxb2hoQEpK\nCpKSkmA0GmGxWFBTU+N1THx8PMxmM4xGo9f9119/PYxGI65cuYKenh5cuXIFCQFUcrZ8lMXiT6Rv\n0b4ebG9vR2JiYt9tk8mE+vp6v1544sSJeOSRR3DjjTdi9OjRyM/Px2233TbouNLS0r7P8/LykPf/\nFd9mA5Yu9etUFARP8ZckdxuIiMRls9lgs9lkfU2fxd8QQlX47LPP8Mc//hEOhwOxsbFYtmwZ9uzZ\ng1/+8pdex/Uv/h6efv/27UGfnkYwZQowejTQ1gZMnap2GiLypf/AGAA2bdoU8mv6bPskJCTA6XT2\n3XY6nTD5uan+J598gltvvRVxcXGIjo7GkiVL8OGHH/r1XPb7w4Pr/Yn0y2fxN5vNaG1thcPhQFdX\nF6qrq1FQUDDksQPfST4tLQ1Hjx7F999/D0mSUFtbi+nTp/sViv3+8ODbOhLpl8+2T3R0NMrLy5Gf\nnw+Xy4XCwkKkp6ejoqICAFBUVISOjg5kZ2fj8uXLiIqKwtatW2G325GZmYn7778fZrMZUVFRmDNn\nDtauXetXKJsNWLYs5H8bjcBsBp57Tu0URKQGgzRwyB7OkxsMg/5i6O0F4uOBkyeBn/xEpWA68fXX\nwM03A5cuAVFCX+5HRP0NVTsDJdyP/KlTwKRJLPzhEBfn/lqfOaN2EiIKN+GKP/v94cX1/kT6xOKv\nc9zmgUifhCr+3M8n/DjyJ9InoYr/yZPs94fb3LnAiRNAT4/aSYgonIQq/mz5hN/11wMmE2C3q52E\niMKJxZ/Y+iHSIWGKf28v8P777PergZO+RPojTPFnv189HPkT6Y8wxZ8tH/XMng2cPg0E8EZrRKRx\nLP6EMWOAlBT3X19EpA9CFH/2+9XH1g+RvghR/NnvVx+LP5G+CFH82fJRH1f8EOmLMMV/0SK1U+hb\nRgbQ2gpcuaJ2EiIKB9WLP/v9Yrj2WmD6dOD4cbWTEFE4qF78T550v3nLlClqJyH2/Yn0Q/Xiz36/\nOFj8ifSDxZ/6cNKXSD9Ufw/fiRMlnDrFto8IuruB8eOB//1f926fRCSmiHgPX/b7xWE0AllZwF//\nqnYSIlKa6iP/zk4J112nVgIa6MQJ4F/+Baiu5vJbIlHJMfJXvfireHoaRl0dsGIF8O67QGam2mmI\naKCIaPuQeBYtAv70J+COOwCHQ+00RKSEaLUDkJiWLQM6OoD8fODIEffeS0QUOdj2IZ+efBJ47z3g\nv//bvfUzEamPPX9SnCQBq1YBFy4A+/e7VwQRkbrY8yfFGQzASy+5P1+zxv3LgIi0b8Tib7VakZaW\nhtTUVJSVlQ16vKWlBbm5uYiJicGWLVv67v/000+RlZXV9xEbG4tt27bJm57CwmgE/uu/gJYW4Kmn\n1E5DRHLw2fZxuVyYNm0aamtrkZCQgOzsbFRVVSE9Pb3vmC+//BKff/459u/fjwkTJuCRRx4Z9Dq9\nvb1ISEhAQ0MDEhMTfzw52z6a8tVXwPz5QHEx8PDDaqch0i/F2z4NDQ1ISUlBUlISjEYjLBYLampq\nvI6Jj4+H2WyG0UczuLa2FsnJyV6Fn7Rn0iTg0CGgrMz9lwARaZfPpZ7t7e1eBdtkMqG+vj7gk+zd\nuxcrV64c8rHS0tK+z/Py8pDHXd6ElpQEHDjgvgo4Pp5XAROFg81mg81mk/U1fRZ/g8EQ8gm6urrw\n1ltvDTlfAHgXf9KGzEz39g+8CpgoPAYOjDdt2hTya/ps+yQkJMDpdPbddjqdMJlMAZ3g4MGDmDt3\nLuLj44NLSEJatAgoL+dVwERa5bP4m81mtLa2wuFwoKurC9XV1SgoKBjy2OEmH6qqqnDvvfeGnpSE\ns3w58Nhj7quAv/pK7TREFIgRL/I6ePAgSkpK4HK5UFhYiCeeeAIVFRUAgKKiInR0dCA7OxuXL19G\nVFQUxo0bB7vdjrFjx6KzsxM33XQT2traMG7cuMEn52qfiBCJVwFbrcC//Rtw9qzaSYgGkyRe4UsC\niKSrgL/9Fvjtb92/zHbsAH72M7UTEQ02ahSLPwmiuxu46y73CqBdu9xXBmuN1QqsXeuex3j+eWCI\nP1aJhMC9fUgonZ3ukfLPfw4884zaafzXf7S/c6c7P5HIuLcPCWXMGPc1AH/9K7B9u9pp/GO1ArNm\nAddcA5w8ycJP+sH9/ElWnquAf/pTYPJk94ogEfUf7e/axaJP+sORP8nOcxVwcbH7LSFFw9E+EXv+\npCDR3guYvX2KFOz5k9BEugqYo30ib+z5k6KWLwfOn1fvvYDZ2ycaGkf+pLiHHwbuucf9F0BnZ/jO\ny9E+0fDY86ewkCTgoYeAL79U/ipg9vYp0rHnT5phMAAvv+z+fO1a5d4LmKN9Iv9w5E9hpdRVwBzt\nk55w5E+ao8RVwBztEwWOq30o7OS6CpgreYiCx5E/qSLUq4A9o32jkaN9omCw50+qCvQq4IH77d92\nm/IZiUTDnj9pXiBXAQ8c7bPwEwWPPX9S3UhXAfcf7VdWsugTyYEjfxLCcFcBc7RPpAz2/EkY/a8C\nfuUV4NFH2dsnGgp7/hRR+l8FnJjI0T6RkjjyJ+FcuQK0tABz5qidhEhMfAN3IiIdYtuHiIiCwuJP\nRKRDLP5ERDrE4k9EpEMs/gBsNpvaEYTIAIiRQ4QMgBg5RMgAiJFDhAyAODlCNWLxt1qtSEtLQ2pq\nKsrKygY93tLSgtzcXMTExGDLli1ej126dAlLly5Feno6pk+fjqNHj8qXXEYi/GeKkAEQI4cIGQAx\ncoiQARAjhwgZAHFyhMpn8Xe5XCguLobVaoXdbkdVVRWam5u9jomLi8P27duxYcOGQc//9a9/jcWL\nF6O5uRlNTU1IT08PKqS/X+zhjhvq/mD+A+XOIUIGUXJoNYMoOUTIIEoOETKIlGM4Pot/Q0MDUlJS\nkJSUBKPRCIvFgpqaGq9j4uPjYTabYRzwjtzffvstPvjgA6xatQoAEB0djdjY2KBCsuApl0GUHFrN\nIEoOETKIkkOEDCLlGJbkw+uvvy6tXr267/arr74qFRcXD3lsaWmp9Ic//KHv9rFjx6R58+ZJDz74\noJSVlSWtXr1a6uzs9HoOAH7wgx/84EcQH6HyuaWzwWDw9bBPPT09aGxsRHl5ObKzs1FSUoLnnnsO\nmzdv7jtG4tW9RESq8Nn2SUhIgNPp7LvtdDphMpn8emGTyQSTyYTs7GwAwNKlS9HY2BhCVCIikovP\n4m82m9Ha2gqHw4Guri5UV1ejoKBgyGMHjuJvuOEGJCYm4syZMwCA2tpazJgxQ6bYREQUihE3djt4\n8CBKSkrgcrlQWFiIJ554AhUVFQCAoqIidHR0IDs7G5cvX0ZUVBTGjRsHu92OsWPH4sSJE1i9ejW6\nurqQnJyMXbt2BT3pS0RE8lF1V08iIlIHr/AlItIhIYu/zWbDggULsG7dOhw+fFjVLJ2dncjOzsaB\nAwdUOX9LSwvWrVuH5cuXY+fOnapkAICamhqsXbsWFosF7777rioZ2trasHr1aixbtkyV83d2duKB\nBx7A2rVr8dprr6mSQe2vgYcI3w+i/GwA6teJYGqmkMXfM3dw9epVv1cXKeX555/HihUrVDt/Wloa\n/vznP2Pv3r04dOiQajnuvPNOvPTSS3jxxRdRXV2tSoabb74ZO3bsUOXcALBv3z4sX74cL730Et58\n801VMqj9NfAQ4ftBlJ8NQP06EUzNVLT4r1q1CpMnT8asWbO87h9pv6AFCxbgnXfewXPPPYeNGzeq\nluPdd9/F9OnTER8fr1oGAHjrrbdwxx13wGKxqJoDAJ5++mkUFxermkFOgWRpb29HYmIiAGDUqFGq\nZFBSMDnk+H4IJYOcPxvB5pCzTgSbIaiaGfJlYj68//77UmNjozRz5sy++3p6eqTk5GSpra1N6urq\nkjIzMyW73S698sorUklJidTe3t537NWrV6WlS5eqluOpp56SSkpKpNtvv1268847pd7e3rBn6K+g\noCDo84eao7e3V3r00Uel2tpa1TJ4yPE9EUyWV199VXr77bclSZIki8WiSgYPOb8GweSQ8/sh2Az9\nyfGzEWwOOetEsBk8AqmZPq/wDdWCBQvgcDi87uu/XxCAvv2CHn/8cdx3330AgDfeeAOHDh3CpUuX\n8PDDD6uW4+mnnwYA/OUvf0F8fHxIVzwHm+Hw4cPYt28ffvjhByxatCjo84eaY9u2bXjvvfdw+fJl\n/P3vf0dRUVHYM3zzzTd48skncfz4cZSVleGxxx4LOkMwWdavX4/i4mIcOHBg2OtdlM4wefJk2b8G\nweSora2V7fsh2AwXLlyQ9Wcj2Bxy1olgM7S0tARcMxUt/kPp/6cz4L4SuL6+3uuYu+++G3fffbfq\nOTweeOAB1TIsXLgQCxcuVOT8geRYv3491q9fr2qGiRMn4sUXX1Qsw0hZrrvuOlRWVip+fl8ZwvU1\nGCnH9u3bZRmYhZIhHD8b/uTwUKpO+JPh8ccfD7hmhn3CV67fiqESIYcIGQAxcoiQwUOELCJkAMTI\nIUIGQIwccmYIe/EPZb+gSMshQgZRcoiQQaQsImQQJYcIGUTJIWsGWWYmfGhra/OasOju7pamTp0q\ntbW1SVevXh1y8iZSc4iQQZQcImQQKYsIGUTJIUIGUXIomUHR4m+xWKQpU6ZI11xzjWQymaTKykpJ\nkiTpnXfekW655RYpOTlZeuaZZ5SMIEwOETKIkkOEDCJlESGDKDlEyCBKDqUzcG8fIiIdEvIKXyIi\nUhaLPxGRDrH4ExHpEIs/EZEOsfgTEekQiz8RkQ6x+BMR6RCLPxGRDrH4ExHp0P8B1QERj/EmSmcA\nAAAASUVORK5CYII=\n"
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pa = linear_model.PassiveAggressiveClassifier(random_state = 0, n_iter = 50, n_jobs = 1, loss = 'hinge', C=1e-3)\n",
      "pa.fit(X, y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "PassiveAggressiveClassifier(C=0.001, fit_intercept=True, loss='hinge',\n",
        "              n_iter=50, n_jobs=1, random_state=0, shuffle=False,\n",
        "              verbose=0, warm_start=False)"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "np.sum(abs(pa.coef_) > 1e-6, axis = 1) * 1. / X.shape[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "array([ 1.        ,  0.99946667,  1.        ,  1.        ,  1.        ,\n",
        "        1.        ,  1.        ,  1.        ,  1.        ])"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## Stochastic Gradient Decent\n",
      "alphas = np.logspace(-6, 0, 7)\n",
      "l1_ratios = np.arange(0.1, 1.1, 0.1)\n",
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', n_jobs=1, n_iter = 20)\n",
      "params = [{'alpha': alpha, 'l1_ratio': l1_ratio} for alpha in alphas for l1_ratio in l1_ratios]\n",
      "scores = [cross_val_score(sgd.set_params(**param), X, y, n_jobs=-1, cv=make_cv(X.shape[0])) for param in params]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sorted_scores = sorted(zip(params, cv_scores), key = lambda (p,s): s, reverse=True)\n",
      "print sorted_scores[:10]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[({'alpha': 0.10000000000000001, 'l1_ratio': 1.0}, 0.21998045950141756), ({'alpha': 0.01, 'l1_ratio': 1.0}, 0.21000041958125792), ({'alpha': 0.01, 'l1_ratio': 0.90000000000000002}, 0.19596542650434867), ({'alpha': 1.0, 'l1_ratio': 1.0}, 0.19498540456624289), ({'alpha': 0.001, 'l1_ratio': 0.10000000000000001}, 0.18899438360516208), ({'alpha': 0.01, 'l1_ratio': 0.80000000000000004}, 0.18498738259217298), ({'alpha': 0.001, 'l1_ratio': 0.20000000000000001}, 0.18299736862611116), ({'alpha': 1.0000000000000001e-05, 'l1_ratio': 1.0}, 0.18000935066803328), ({'alpha': 0.001, 'l1_ratio': 0.90000000000000002}, 0.17997938057818297), ({'alpha': 0.001, 'l1_ratio': 0.30000000000000004}, 0.17899336462210713)]\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', n_jobs=-1, n_iter = 20, alpha=0.1, l1_ratio=1)\n",
      "sgd.fit(X, y)\n",
      "print 'sparse rate:', np.sum(abs(sgd.coef_) > 1e-3, axis = 1) * 1. / X.shape[1]\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sparse rate: [ 0.86666667  0.86133333  0.912       0.9552      0.94986667  0.95093333\n",
        "  0.9456      0.9568      0.95786667]\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## SGD and SVM are generally senstive to normalization\n",
      "\n",
      "XX = preprocessing.Normalizer().fit_transform(X)\n",
      "alphas = np.logspace(-6, 0, 7)\n",
      "l1_ratios = np.arange(0.1, 1.1, 0.1)\n",
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', n_jobs=1, n_iter = 20)\n",
      "params = [{'alpha': alpha, 'l1_ratio': l1_ratio} for alpha in alphas for l1_ratio in l1_ratios]\n",
      "scores = [cross_val_score(sgd.set_params(**param), XX, y, n_jobs=-1, cv=make_cv(X.shape[0])) for param in params]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## different alpha values with similar accuracy meaning redunancies in the original feature space\n",
      "sorted_scores = sorted(zip(params, cv_scores), key = lambda (p,s): s, reverse=True)\n",
      "print sorted_scores[:10]\n",
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', n_jobs=-1, n_iter = 20, alpha=0.1, l1_ratio=1)\n",
      "sgd.fit(XX, y)\n",
      "print 'sparse rate:', np.sum(abs(sgd.coef_) > 1e-3, axis = 1) * 1. / X.shape[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[({'alpha': 0.001, 'l1_ratio': 1.0}, 0.21799643955332579), ({'alpha': 1.0, 'l1_ratio': 1.0}, 0.21798445151738566), ({'alpha': 0.01, 'l1_ratio': 1.0}, 0.21498744253235269), ({'alpha': 0.0001, 'l1_ratio': 0.59999999999999998}, 0.20999142855430283), ({'alpha': 0.0001, 'l1_ratio': 0.20000000000000001}, 0.2099854345363327), ({'alpha': 0.0001, 'l1_ratio': 0.40000000000000002}, 0.20997044949140756), ({'alpha': 0.0001, 'l1_ratio': 0.80000000000000004}, 0.20898443353533178), ({'alpha': 0.001, 'l1_ratio': 0.90000000000000002}, 0.20498942055828284), ({'alpha': 0.0001, 'l1_ratio': 0.70000000000000007}, 0.20496844149538762), ({'alpha': 0.0001, 'l1_ratio': 0.10000000000000001}, 0.20397942853032672)]\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sparse rate: [ 0.2176      0.192       0.13546667  0.0816      0.0448      0.02453333\n",
        "  0.07306667  0.02613333  0.06826667]\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## TO see how the variances are distributed among features, using PCA\n",
      "from sklearn import decomposition\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 118
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pca = decomposition.PCA(whiten = True)\n",
      "pca.fit(X, y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 119,
       "text": [
        "PCA(copy=True, n_components=None, whiten=True)"
       ]
      }
     ],
     "prompt_number": 119
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## only the first pc is important (60% variance)\n",
      "plt.plot(pca.explained_variance_ratio_)\n",
      "print pca.explained_variance_ratio_[:20]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[ 0.58323866  0.06736495  0.05158571  0.03981327  0.0228172   0.01756079\n",
        "  0.01616188  0.0141026   0.01152615  0.01020264  0.00950273  0.00843913\n",
        "  0.0074265   0.00593941  0.00568563  0.00500661  0.00489166  0.00441606\n",
        "  0.00407469  0.00393579]\n"
       ]
      },
      {
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD9CAYAAABdoNd6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFIJJREFUeJzt3X9s1dX9x/HXh907FVRQfsi8t66s99J7K6HUtRI1m3f+\nSJXIFcUsZQsspGONS+c0y2KWLFk7N6TZzEZosnSLbihbaeIWixu9aMNudCK3uuI0q5Ora8Ol/EhE\nugmT3HI93z/4cktp+bTg/YE9z0fS5H7u5/Scc0/g3Ren514cY4wRAGBKm1bsCQAA8o9iDwAWoNgD\ngAUo9gBgAYo9AFiAYg8AFpiw2MdiMYVCIQWDQbW0tIzbJh6Pq6qqSosWLVIkEsn1HAEAn5Djds4+\nk8movLxc3d3d8vl8qqmpUXt7u8LhcLbN0NCQbrnlFu3YsUN+v1/vv/++5syZU5DJAwAmxzXZ9/T0\nKBAIqLS0VF6vV3V1ders7BzV5g9/+INWrlwpv98vSRR6ALgIedxuDg4OqqSkJHvt9/uVSCRGtUkm\nkxoeHtZXvvIVffjhh/rud7+r1atXj2rjOE4OpwwA9sjVhxy4JvvJFOnh4WH19vZq+/bt2rFjhx57\n7DElk8kx7YwxfBmjH/3oR0Wfw8XyxVqwFqyF+1cuuSZ7n8+nVCqVvU6lUtntmtNKSko0Z84cXXbZ\nZbrsssv05S9/Wf/4xz8UDAZzOlEAwIVzTfbV1dVKJpMaGBhQOp1WR0eHotHoqDb33nuv/va3vymT\nyeh///ufEomEKioq8jppAMD5cU32Ho9Hra2tqq2tVSaTUX19vcLhsNra2iRJDQ0NCoVCuuuuu7R4\n8WJNmzZN69ato9i74GjqCNZiBGsxgrXID9ejlzkbxHFyvv8EAFNdLmtnwd5B+/zzhRoJAHC2ghX7\nvXsLNRIA4GwFK/bs4gBA8fBBaABgAZI9AFiAZA8AFqDYA4AF2MYBAAuQ7AHAAiR7ALAAyR4ALECy\nBwALkOwBwAIkewCwAMkeACxAsgcAC5DsAcACFHsAsADbOABgAZI9AFiAZA8AFiDZA4AFSPYAYAGS\nPQBYgGQPABYg2QOABSj2AGABtnEAwAITFvtYLKZQKKRgMKiWlpYx9+PxuGbOnKmqqipVVVXpJz/5\nSV4mCgC4cB63m5lMRo2Njeru7pbP51NNTY2i0ajC4fCodrfeequ2bdvmOhDJHgCKxzXZ9/T0KBAI\nqLS0VF6vV3V1ders7BzTzlDJAeCi5prsBwcHVVJSkr32+/1KJBKj2jiOo127dqmyslI+n08///nP\nVVFRMaaveLxJTU2nHkciEUUikU88eQCYSuLxuOLxeF76di32juNM2MENN9ygVCql6dOnq6urSytW\nrNDevXvHtLv11pFiDwAY6+wg3NzcnLO+XbdxfD6fUqlU9jqVSsnv949qc8UVV2j69OmSpLvvvlvD\nw8P64IMPxvTFTg8AFI9rsa+urlYymdTAwIDS6bQ6OjoUjUZHtTl8+HB2z76np0fGGF199dX5mzEA\n4Ly5buN4PB61traqtrZWmUxG9fX1CofDamtrkyQ1NDTo2Wef1a9+9St5PB5Nnz5dW7duLcjEAQCT\n55gCHKVxHEc//KHRY4/leyQAmDocx8nZaUc+LgEALMDHJQCABUj2AGABkj0AWIBkDwAWINkDgAVI\n9gBgAYo9AFiAbRwAsADJHgAsQLIHAAuQ7AHAAiR7ALAAyR4ALECyBwALkOwBwAIUewCwANs4AGAB\nkj0AWIBkDwAWINkDgAVI9gBgAZI9AFiAZA8AFiDZA4AFSPYAYAGSPQBYgGIPABaYsNjHYjGFQiEF\ng0G1tLScs91rr70mj8ejP/3pT+PeZxsHAIrHtdhnMhk1NjYqFoupr69P7e3tevvtt8dt9+ijj+qu\nu+6SoaoDwEXHtdj39PQoEAiotLRUXq9XdXV16uzsHNNu06ZNeuCBBzR37txz9sXPAAAoHo/bzcHB\nQZWUlGSv/X6/EonEmDadnZ3auXOnXnvtNTmOM25fiUSTmppOPY5EIopEIp9o4gAw1cTjccXj8bz0\n7Vrsz1W4z/Twww9rw4YNchxHxphzbuPceONIsQcAjHV2EG5ubs5Z367F3ufzKZVKZa9TqZT8fv+o\nNn//+99VV1cnSXr//ffV1dUlr9eraDSas0kCAD4Z12JfXV2tZDKpgYEBXXvttero6FB7e/uoNv/+\n97+zj9euXavly5ePW+jZsweA4nEt9h6PR62traqtrVUmk1F9fb3C4bDa2tokSQ0NDQWZJADgk3FM\nAc5KOo6jxkajTZvyPRIATB2nfxeaC3w2DgBYgI9LAAALkOwBwAIkewCwAMkeACxAsgcAC5DsAcAC\nJHsAsADFHgAswDYOAFiAZA8AFiDZA4AFSPYAYAGSPQBYgGQPABYg2QOABUj2AGABij0AWIBtHACw\nAMkeACxAsgcAC5DsAcACJHsAsADJHgAsQLIHAAuQ7AHAAhR7ALAA2zgAYIEJi30sFlMoFFIwGFRL\nS8uY+52dnaqsrFRVVZW++MUvaufOnXmZKADgwjnGnDtzZzIZlZeXq7u7Wz6fTzU1NWpvb1c4HM62\nOX78uGbMmCFJeuutt3Tffffp3XffHT2I42jNGqPNm/P0KgBgCnIcRy4l+ry4Jvuenh4FAgGVlpbK\n6/Wqrq5OnZ2do9qcLvSSdOzYMc2ZMycnEwMA5I7H7ebg4KBKSkqy136/X4lEYky75557Tj/4wQ90\n8OBBvfDCC+P29cYbTWpqOvU4EokoEolc8KQBYCqKx+OKx+N56du12DuOM6lOVqxYoRUrVujll1/W\n6tWr9c4774xpU1k5UuwBAGOdHYSbm5tz1rfrNo7P51Mqlcpep1Ip+f3+c7b/0pe+pJMnT+rIkSNj\n7nEaBwCKx7XYV1dXK5lMamBgQOl0Wh0dHYpGo6PavPfee9lfIPT29kqSZs+enafpAgAuhOs2jsfj\nUWtrq2pra5XJZFRfX69wOKy2tjZJUkNDg/74xz/q6aefltfr1eWXX66tW7eO2xfJHgCKx/XoZc4G\ncRx9/etGW7bkeyQAmDoKdvQSADA18HEJAGABkj0AWIBkDwAWINkDgAVI9gBgAZI9AFiAZA8AFiDZ\nA4AFKPYAYAG2cQDAAiR7ALAAyR4ALECyBwALkOwBwAIkewCwAMkeACxAsgcAC1DsAcACbOMAgAVI\n9gBgAZI9AFiAZA8AFiDZA4AFSPYAYAGSPQBYgGQPABag2AOABSYs9rFYTKFQSMFgUC0tLWPu//73\nv1dlZaUWL16sW265RW+++ea4/bCNAwDF43G7mclk1NjYqO7ubvl8PtXU1CgajSocDmfbfOELX9BL\nL72kmTNnKhaL6Vvf+pZ2796d94kDACbPNdn39PQoEAiotLRUXq9XdXV16uzsHNXmpptu0syZMyVJ\nS5cu1f79+8fti2QPAMXjmuwHBwdVUlKSvfb7/UokEuds/+STT2rZsmXj3nvnnSY1NZ16HIlEFIlE\nznuyADCVxeNxxePxvPTtWuwdx5l0R3/961/11FNP6ZVXXhn3/sKFI8UeADDW2UG4ubk5Z327Fnuf\nz6dUKpW9TqVS8vv9Y9q9+eabWrdunWKxmK666qqcTQ4AkBuue/bV1dVKJpMaGBhQOp1WR0eHotHo\nqDb79u3T/fffry1btigQCJyzL/bsAaB4XJO9x+NRa2uramtrlclkVF9fr3A4rLa2NklSQ0ODfvzj\nH+vo0aN68MEHJUler1c9PT35nzkAYNIcY/KfuR3H0bJlRn/5S75HAoCpw3Ec5apE8w5aALAAxR4A\nLMCnXgKABUj2AGABkj0AWIBkDwAWINkDgAVI9gBgAZI9AFiAZA8AFqDYA4AF2MYBAAuQ7AHAAiR7\nALAAyR4ALECyBwALkOwBwAIkewCwAMkeACxAsQcAC7CNAwAWINkDgAVI9gBgAZI9AFiAZA8AFiDZ\nA4AFSPYAYAGSPQBYgGIPABaYsNjHYjGFQiEFg0G1tLSMuf+vf/1LN910ky699FI98cQT5+yHbRwA\nKB6P281MJqPGxkZ1d3fL5/OppqZG0WhU4XA422b27NnatGmTnnvuubxPFgBwYVyTfU9PjwKBgEpL\nS+X1elVXV6fOzs5RbebOnavq6mp5vV7XgUj2AFA8rsl+cHBQJSUl2Wu/369EInFBA+3b16SmplOP\nI5GIIpHIBfUDAFNVPB5XPB7PS9+uxd5xnJwNVFIyUuwBAGOdHYSbm5tz1rfrNo7P51Mqlcpep1Ip\n+f3+nA0OACgM12JfXV2tZDKpgYEBpdNpdXR0KBqNjtvWTLApz549ABSP6zaOx+NRa2uramtrlclk\nVF9fr3A4rLa2NklSQ0ODDh06pJqaGv33v//VtGnTtHHjRvX19enyyy8vyAsAAEzMMRNF8lwM4ji6\n+WajV17J90gAMHU4jjPhrslk8dk4AGABPi4BACxAsgcAC5DsAcACJHsAsADJHgAsULBi//HHhRoJ\nAHC2ghX748cLNRIA4GwFK/b/+U+hRgIAnK1gxX5oqFAjAQDOVrBif+KEdPJkoUYDAJypYMX+yivZ\nygGAYilYsZ85k2IPAMVSsGI/axb79gBQLCR7ALAAyR4ALECyBwALkOwBwAIkewCwAMkeACxAsgcA\nC5DsAcACJHsAsEDBiv28edKhQ4UaDQBwJseY/P/vsI7j6KOPjK666lS6/+xn8z0iAHz6OY6jXJXo\ngiX7Sy+VqqqkZ58t1IgAgNMKluyNMXrxRemRR6S33pIcJ9+jAsCn26cy2UvSHXdIn/mM9PzzhRz1\n4hKPx4s9hYsGazGCtRjBWuTHhMU+FospFAopGAyqpaVl3DYPPfSQgsGgKisrtWfPnnP25TjSL34h\nffObUm/vhU/604w/yCNYixGsxQjWIj9ci30mk1FjY6NisZj6+vrU3t6ut99+e1Sb7du3691331Uy\nmdSvf/1rPfjgg64D3nabtHGjVFsrtbRI+/ZJ+d9IAgC7edxu9vT0KBAIqLS0VJJUV1enzs5OhcPh\nbJtt27bpG9/4hiRp6dKlGhoa0uHDh3XNNdecs99Vq6TPf176zW+kmhpp2jQpFJL8/lNHNK+7Tpo9\n+9TZ/EsvlS655NQJnksuGfka73paQTelAODTw7XYDw4OqqSkJHvt9/uVSCQmbLN///4xxd6Z4Dey\nNp3Bb25uLvYULhqsxQjWYgRrkXuuxX6iAn3a2b8tPvv7CnDgBwDgwnXjw+fzKZVKZa9TqZT8fr9r\nm/3798vn8+V4mgCAT8K12FdXVyuZTGpgYEDpdFodHR2KRqOj2kSjUT399NOSpN27d2vWrFmu+/UA\ngMJz3cbxeDxqbW1VbW2tMpmM6uvrFQ6H1dbWJklqaGjQsmXLtH37dgUCAc2YMUO//e1vCzJxAMB5\nMHnW1dVlysvLTSAQMBs2bMj3cEW3b98+E4lETEVFhbn++uvNxo0bjTHGHDlyxNxxxx0mGAyaO++8\n0xw9ejT7PevXrzeBQMCUl5ebHTt2FGvqeXHy5EmzZMkSc8899xhj7F0HY4w5evSoWblypQmFQiYc\nDpvdu3dbuR7r1683FRUVZtGiRWbVqlXmxIkT1qzD2rVrzbx588yiRYuyz13Ia3/99dfNokWLTCAQ\nMA899NCkxs5rsT958qQpKysz/f39Jp1Om8rKStPX15fPIYvu4MGDZs+ePcYYYz788EOzcOFC09fX\nZ77//e+blpYWY4wxGzZsMI8++qgxxph//vOfprKy0qTTadPf32/KyspMJpMp2vxz7YknnjBf+9rX\nzPLly40xxtp1MMaYNWvWmCeffNIYY8zw8LAZGhqybj36+/vNggULzIkTJ4wxxnz1q181v/vd76xZ\nh5deesn09vaOKvbn89o//vhjY4wxNTU1JpFIGGOMufvuu01XV9eEY+e12O/atcvU1tZmrx9//HHz\n+OOP53PIi869995rXnzxRVNeXm4OHTpkjDn1A6G8vNwYc+on95n/4qmtrTWvvvpqUeaaa6lUytx+\n++1m586d2WRv4zoYY8zQ0JBZsGDBmOdtW48jR46YhQsXmg8++MAMDw+be+65x7zwwgtWrUN/f/+o\nYn++r/3AgQMmFApln29vbzcNDQ0TjpvXtyGNdwZ/cHAwn0NeVAYGBrRnzx4tXbp01BvNrrnmGh0+\nfFiSdODAgVEnnKbSGj3yyCP62c9+pmlnvNvNxnWQpP7+fs2dO1dr167VDTfcoHXr1un48ePWrcfV\nV1+t733ve7ruuut07bXXatasWbrzzjutW4czne9rP/t5n883qTXJa7Gf7Dn9qejYsWNauXKlNm7c\nqCuuuGLUPcdxXNdmKqzbn//8Z82bN09VVVXnfJ+FDetw2smTJ9Xb26tvf/vb6u3t1YwZM7Rhw4ZR\nbWxYj/fee0+//OUvNTAwoAMHDujYsWPasmXLqDY2rMO5TPTaP4m8FvvJnNOfioaHh7Vy5UqtXr1a\nK1askHTqJ/ah/3+b8MGDBzVv3jxJU/d9Crt27dK2bdu0YMECrVq1Sjt37tTq1autW4fT/H6//H6/\nampqJEkPPPCAent7NX/+fKvW4/XXX9fNN9+s2bNny+Px6P7779err75q3Tqc6Xz+Tvj9fvl8Pu3f\nv3/U85NZk7wW+8mc059qjDGqr69XRUWFHn744ezz0WhUmzdvliRt3rw5+0MgGo1q69atSqfT6u/v\nVzKZ1I033liUuefS+vXrlUql1N/fr61bt+q2227TM888Y906nDZ//nyVlJRo7969kqTu7m5df/31\nWr58uVXrEQqFtHv3bn300Ucyxqi7u1sVFRXWrcOZzvfvxPz583XllVcqkUjIGKNnnnkm+z2ucvEL\nBzfbt283CxcuNGVlZWb9+vX5Hq7oXn75ZeM4jqmsrDRLliwxS5YsMV1dXebIkSPm9ttvH/d41U9/\n+lNTVlZmysvLTSwWK+Ls8yMej2dP49i8Dm+88Yaprq42ixcvNvfdd58ZGhqycj1aWlqyRy/XrFlj\n0um0NetQV1dnPve5zxmv12v8fr956qmnLui1nz56WVZWZr7zne9MauyC/E9VAIDi4kOBAcACFHsA\nsADFHgAsQLEHAAtQ7AHAAhR7ALDA/wE7g7Dt+qryOgAAAABJRU5ErkJggg==\n"
      }
     ],
     "prompt_number": 120
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## Use PCA results to fit SGD\n",
      "PCA_X = decomposition.PCA(n_components=10, whiten=True).fit_transform(X)\n",
      "## draw on first 2 PC\n",
      "colors = ['r', 'g', 'b', 'm', 'y', 'k', 'c']\n",
      "for (cls,c) in zip(np.unique(y), colors):\n",
      "    plt.scatter(PCA_X[y==cls,0], PCA_X[y==cls, 1], color=c, label=str(cls))\n",
      "plt.legend()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 167,
       "text": [
        "<matplotlib.legend.Legend at 0x10a186350>"
       ]
      },
      {
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD9CAYAAAClQCyNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4FFXbh+/ZXpJACCV0Qk9AIVJEpQQRRRSwgIqIlSJW\n8OVVP8srCogNRMSOIlYQsYA0RapADC2A9BYIJZAQ0rbvzvn+OEk2SwIEktCc+7r2SnZ35syZ2d3f\nnPOcpyhCCIGGhoaGxiWH7kJ3QENDQ0Pj3NAEXENDQ+MSRRNwDQ0NjUsUTcA1NDQ0LlE0AdfQ0NC4\nRNEEXENDQ+MSpcwCnpWVRd++fYmNjSUuLo7ExMTy6JeGhoaGxhkwlLWBp59+mp49e/Ljjz/i9/tx\nOBzl0S8NDQ0NjTOglCWQJzs7m/j4ePbu3VuefdLQ0NDQKAVlGoHv27ePatWq8dBDD7Fx40batGnD\ne++9h81mA0BRlHLppIaGhsa/jdKMrctkA/f7/axfv57HHnuM9evXY7fbeeONN4p14lJ9vPLKKxe8\nD1r/L3w//o39v5T7fjn0v7SUScDr1KlDnTp1aNeuHQB9+/Zl/fr1ZWlSQ0NDQ6OUlEnAo6OjqVu3\nLjt37gRg0aJFtGjRolw6pqGhoaFxesrshfL+++8zYMAAvF4vjRo1YurUqeXRr4uChISEC92FMqH1\n/8JyKff/Uu47XPr9Ly1l8kI5Y+OKclb2HA0NDQ2N0mtnmUfgGhoaGuebKlWqcOLEiQvdjTITGRlJ\nZmbmOe+vjcA1NDQuOS4XbTnVeZT2/LRcKBoaGhqXKJqAa2hoaFyiaAKuoaGhcYmiCbiGhobGJYom\n4BoaGhrlyOTJk2nbti0Wi4WHHnqoQo+luRFqaGholCO1a9fm5ZdfZuHChbhcrgo9libgGhoa/z6c\nTpg6FY4eha5d5aOcuP322wFYu3YtBw8eLLd2S0ITcA0NjcuPXbtg7lywWODuuyEyMvie2w3t28Pe\nvfL/8ePl49FHy7UL58NPXbOBa2hoXF4kJkJ8PDz/PDzzDLRoAenpwfdnzYKUFHC5QAg5Gh85MrSN\njAzo3h3sdqhXD/7886y7cT7qIWgCrqGhcXnx5JPgcIDHI0U6IwPefTf4fk4OqGroPm536Gt9+sCy\nZVLcU1Ohd2/YvfusuqGNwDU0NDTOluPHQ5/7fJCWFnzerRsUHR2bTHD99aDTBbdPTJR/C1AUWL78\nrLqhjcA1NDQ0zpZevcBqDT632eQIuoCmTWH2bIiJgUqV4OabYebM4PsGgxT1oigKVK5cqsMHAgHc\nbjd+v59AIIDH4yEQCJThhE6DqEAquHkNDY1/KafVFo9HiAceEMJqFSIiQogJE87+AB9+KITNJoRe\nL/+2ayeE11uqXV955RWhKErI49VXXz2r8yitdmrZCDU0NC45zou2LF8OK1ZAdDQMHFh8VF4OlDUb\noSbgGhoalxyXi7Zo6WQ1NDQ0/qVoAq6hoaFxiaIJuIaGhsYliibgGhoaGpcomoBraGhoXKJoAq6h\noaFxiaIJuIaGhsYliibgGhoaGpcomoBraBQhEMhjx47BJCXFsWlTL9zu/Re6SxqXGF6vl0ceeYQG\nDRoQERFBfHw8CxYsqJBjaQKuoZGPEIJNm24lLe1rnM5tnDgxn3Xrrsbvz7nQXdO4hPD7/dSrV4/l\ny5eTk5PDmDFjuOuuu9i/v/wHA1oovYZGPj5fBqtW1UYIb+Fren0EcXHfExXV8wL2TONkyqotTp+T\nqRumctRxlK4NutI1pvxKqpVEq1atGDVqVGG5tQLKGkpf5pJqgUCAtm3bUqdOHebMmVPW5jQ0LhiK\nYgRO/tEIdLryT2KkUbHsOr6LubvmYjFYuLvF3URagyXV3H437T9rz94Te3H73YxfPZ7xN47n0bbl\nW1KtgKNHj7Jz505atGhR7m2X2YTy3nvvERcXd16Sl2toVCQGQyWqVbsLnc4GgKKYMZvrUKlSpwvc\nM42zIfFgIvGfxPP8oud5ZuEztPiwBemOYEm1WVtnkZKVgsvvQiBw+pyM/D20pFqGM4PuX3XH/rqd\neu/W48+9Z19SDcDn8zFgwAAefPBBmjZtWqbzKokyCfjBgweZN28egwYN0kwlGpcFsbHTiIkZQ1TU\nbdSp8zRXXZWITme+0N3SOAuenPckDp8DT8CDy+8iw5nBu4nBkmo5nhxUEVpSze13h7zW5/s+LNu/\nDKfPSWpOKr2n92Z35tmVVFNVlYEDB2KxWJg8eXLZTuoUlMmEMmLECN5++21yck69yDNq1KjC/xMS\nEkhISCjLITU0KhRF0VO37gjq1h1xobuicY4cd4WWVPOpPtLygiXVujXsFmIxMOlNdKnfBZ0ix7O+\ngI/EQ4khgq6gsHz/chpXaVyqPggheOSRR0hPT2fevHno9frTbr906VKWLl1aqraLcs4C/ttvv1G9\nenXi4+NPe+CiAq6hoaFR0fRq2ovP1n+Gy+8CwGa00btZsKRa06imzL5nNoPnDCbTlUlCgwSm3Tat\n8H2DzoBJb8Ltdxe+pigKlS2lK6kGMGzYMLZv386iRYswm888gzt5cPvqq6+W6jjn7IXywgsv8PXX\nX2MwGHC73eTk5HDnnXfy1VdfBRvXvFA0NDQqgNNpizfgZcicIfyw5QeMeiOjuoxixDVnN6P6aO1H\njPx9JB6/B7PBTItqLVj58EqMeuMZ992/fz8xMTFYLJaQkfenn35K//79S3Ue57Uiz7Jly3jnnXeK\neaFoAq5xubFo7yJWp66mdkRtBl45sFQ/aI3y53xoy/L9y1mxfwXRYdEMbDUQk/7iK6lWZjfCogfU\n0Lio8XohPR1q1JCVx8+Sd1a9wytLX8Htc2MxWvhiwxcse3AZet3p7Zsalyad63emc/3OF7obp0UL\n5NH4dzB3Ltx9N6gqmM0wZw507Fjq3f2qH9tYGz7VV/hamCmMmf1m0qNxj4roscZpuFy05aIZgWto\nAJCdDe++C6mp0L27FM0LPTtLS4O77gKnUz53ueCWW+DIEbDZStWEyyd9houioHDCdaK8e6uhUWo0\nAdcoP5xOaNsWDhyQ5orp02HLFhg9+sL2a9s2MJ5kqxYC9u+H2NhSNRFuDueK6lew+dhm/KofAFWo\ndKxX+lG8hkZ5oyWz0ig/fvtNjna9+blEnE544w0IBC5sv+rVC/apAJ8PoqPPqpkF9y2gY72O2Iw2\nYirHsOC+BdStVLccO6qhcXZoI3CN8sPlKv6aEFLAzxDIUKE0agQvvADjxsnFS78fJk6EyMgz71uE\n6vbqLHlgSQV18vSsTl3NkDlDOOY8xvUx1/PprZ8Sbg6/IH3RuHjQFjE1yo9Dh6RJIjdXPjeboVs3\nuYB4MbB5M+zZA3FxUAF5KSqKfSf2ccVHV+DwOQAw6810jenK/AHzL3DPLhyXi7Zoi5gaFw+1a8OK\nFTB0qFwgvP56qKAcEOfEFVfIxyXGor2LQhZQPQEPf+z5g4Aa0FwY/+VoAq5RvrRqBYmJF7oXlxVh\npjAUQj15jDpjYe4OjX8v2jdAQ6MIfj888QRUrgzVq8NHH13oHkGf5n2oE1EHs17m1LAZbYy+frQW\nPHcRc99991GzZk0iIiJo2LAhY8eOrZDjaDZwDY0iPP88vP9+0GXcZoPvv4fevU+/X0WT583jozUf\ncSj3EN0bdueWprdc2A5dYC52bdmyZQuNGjXCYrGwY8cOunTpwpdffkmPHqFBX5oNXEOjHPnxx6B4\ng/x/1qyKE/CUrBQW71tMmCmM3s16YzFYStwuzBTGf6/7b8V04l+I0wlTp8LRo9C1q3yUJydX3zEY\nDFSvXr18D4Im4BoaIZzsWajXQ1RUxRxrVeoqbvz6RkCOuOpVqkfSoCTsJnvFHPBfxK5d0vnJYpHB\nwEU/V7cb2reHvXvl/+PHy8ej5VxR7bHHHmPatGl4PB4mT57MVVddVb4HQDOhaGiEsHIl3HgjeDxS\nvCMiYNMmqFmz/I8V90Ec2zK2FT63GCyM6zaO4R2Gl//BLjNOpy2JiXDDDXI9Q6eT6xkbN0K1avL9\nb7+VjlIOR3Afux3y8oLPMzKgf39YtUrewKdOlR6xZ4sQgmXLltG3b1/mzZtH+/btS3UepdVObRFT\n45LAEQjwY3o63x49ytGToyrLkeuugzVrZPT/uHEyE0BFiDdAujM95Lnb7+ZQ7qGKOdi/iCeflOLs\n8cjYsowMmZ6ngJwcmdOsKG536Gt9+sCyZdLUkpoqTWi7z66iGiCFOCEhgX79+vH999+f2wmdBs2E\nonHRc8Lno+26dRzzyUyABkVhRsOqTFn5P9Ly0rit+W0M7zC83Nzq4uLk43QEAnLEVqnSuR+nS/0u\n/LbzNzwBDyC9S7o2KGdj7L+Q46EV1fD5ZIaHArp1C82vZjJBly5ytF6wfWJiqKArCixfDo1LV1Gt\nGD6fj6gKsMVpI3CNCscb8PL4vMepOb4mzSY3Y8HuBWe1/xsHDnDQ4yEvECAvECDb5+WWNb8za9ss\nVhxYwctLXua/v5+/Bb6pU+WUu1o1GdC5b9+5tfNFny/oVL8TekWPWW9mdNfR9GzSs3w7+y+kVy+w\nWoPPbbbQReimTWH2bIiJkTfgm2+GmTOD7xsMUtSLoijSFFMa0tPTmT59Og6Hg0AgwMKFC5k5cyZ9\n+vQ595M6BZoNXKPCGTJnCN9s+iZYo9BgY8XDK7iqZukWde7esoUf0kPNDYozFbHm/sLnVoMV54vO\nk3ctdzZskGnECzxVdDpo3lyaWs4Vv+pHr+g1v+6z4LQl1bwwZAj88INMQjlqFIw4yxrVH30EI0dK\nM4zZDC1ayPWRk5NalkRGRgZ9+/Zl48aNCCFo2rQpL730Er1LcGXS3Ag1ANiesZ0hc4ZwIPsAnep1\n4sNbPjz/yY7cbuk0/eGH8ld0553w4ov8uPXHQvEGaeuds2NOqQW8uxD85nLhzB9WGXwexPH1XIgc\nh0lJoc9VVWar9fvPqcgPIIvoXqz4fPD77zK9TefOUKvWhe7RmTGZ4Msv5eNcGTZMivaKFTJp5cCB\npRNvgKpVq55Thflz4eL95mickm3p27jzhzvZnbmbBpUb8Fmvz7h9xu1kubMQCNK2ppGak8rSB5ee\nv05lZECbNjIXeAG7dkFGBtYGVk64g4UPjHojYaawUjf9iNnMlgULmNyrF0JRSFi/jjU5H5JrUlAR\n2Iw2Hm1Tzj5gp6B27aCttIBKlc5dvC9m3G4p2tu2BW3Gf/4J7dpd2H6dLzp3lo+LGc2Econh8rlo\nMLEB6c70wgRHYaYwEJDnC/pBGXQGMp/NPH+j8AcfhG++KZ77227nu9WfMnj2YJx+J0adkShrFJsf\n20xVW9VSNb05bSNvvJhArjuXB5N13LFDZU+z6jz/0tWkuTO4vfnt5bqIeTpUFe64QwqZosjTnTED\nbr21wg993pk8GZ59NjRLcFxc2cxF5cXloi2aCeVfxq7MXbj8oeW9VKGW+GGf14rp27eXXLhBp+Pe\nK+6lZlhNZu+YTRVrFR5t+2ipxXtr+lau+eI6nPWcCAR/NlRxH2jFva//xszatcv5JM6MTgc//wyL\nF8sovvbtz90z4WInNbV4ivcjRy5MXzRKRhPwS4wq1iohhXVBBgvUCq/FwZyDeAIebEYbg+IHnTIs\nu0Lo0AGSk+WqTwFGI/xXeod0jelK15izd5H7eO3HOH3OwhuW0yAY3c7NvRdAvAtQlHML6rjU6NgR\nPvggGPBiNMI111zYPmmEorkRXkCEEGS6MvEGSh+YUieiDkPbDMVutGPUGbEb7Qy4cgDJjybzQqcX\neKDVA3zQ8wMm9phYgT0vgddflyJuNsthatWqcg7+0ktlatYX8BUrJuwX/tLt7PeH3lA0zopevaQJ\nxWCQj/h4+OqrC90rjaJoNvALxIHsA9zw1Q3sz96PEIIJN03gifZPlGpfIQTzd89ny7EtNKvajF5N\ne10cLmhCwMGDUsBr1SqXavRJh5LoOq0rTp/027MZbYzrNo6nrn7qlPsEnH7Ecy9h+Hi87FOPHtKn\nrJQV6DVC8fnkgmb4RVTB7XLRlrLawDUBv0DEfxLPpqObUIUM97IZbSwauIhr6l4Cc9Q//oBJk6RA\njxxZ4Uv1y1KW8fKSl3H6nAy6ahBD2wwt8YYlhGD38N0cnnwQ1AARbOUKXsBgCcADD8DHH5drv1RV\nJkw6ckROPq68slybPzVCQFaWTNRyIWuNXkAuF23RBPwSRAiBcbSRgAgu+pn1Zt7q/tZpR5YXBQsW\nSP/ugkgWq1W+dhH4W6V9lcbOYTtRnfKmqOClGiuIY4w06Rw7Vi6zApDifWuXXFYkmVFREHoDn3+u\n0L9/uTR/arZvl9m20tKkeE+bBnfdVcEHvfi4XLRFS2Z1CaIoSjEvDKPeSJ2IOheoR2fBG2+EJsx2\nueCddy5cf4qQtTSrULwBBCayaSmfZGbCp5+W27EWjFvPir8gz2vC6TXicikMekQgBHLVb8kSmcrO\nX0p7fWkQQor3wYNBu8ZDD8HOneV3DI1LCk3ALxDf3vEtdqOdCHMEYaYwEhokcFvz2y50t85MSaOC\nktwHT97EEUCoFTtisjayorMU/UqrmDmW/68Kn39ebsc6+t50Tj4bp1vlg/lvI5o1hdtug5tukm4b\nznIK8c/Kkr6LRT8DvR7Wry+f9jXKnV27dmGxWBg4cGCFtK+5EV4gujXsxpbHtpB0KImqtqokNEi4\nOBYiz8SIEbB2bWjNseGnzl/tTnWz6eZNOLc70Rl1NPmwCTUfqpj8rHWG1+HYjGO4U9zgcqL4PTSj\nyOzgFIuYDoeDFStWoCgKnTt3xlo0E9Ip6OBdjig6/lH8UG0bzyX+H3kNVZ5fkS+y//wDb70lE3KU\nlYiI4mGgqirDQzUuSh5//HHat29fYb/tMo3AU1NT6dq1Ky1atKBly5ZMmjSpvPr1r6B+5fr0a9GP\nrjFdLw3xBjmy/PZb6NRJ2r1//BG6dz/l5v/0/gfndicEQHWr7Hp8F7nrciuka3q7njZr2hA3I47m\nb0Vxte1R7OSH9tts8NprwY29Xhg5krSmTYmNiuLuvn3p168fLVu25PjJ+UhLIPbmGL42PIxZnwOo\nUH0LDOiJUx9g4tVFRshuN2zdWk4nqJcJPmw26RJit0PfvtJh+0x4vTIb0/Llsk//cgLOAIc+OMS+\n/+3jxJITZ97hHJg+fTqRkZF069atwuz1ZVrETEtLIy0tjdatW5OXl0ebNm345ZdfiI2NlY1fJgsN\nGueGUAXLjMugSF5lnUVHo3caUfvx04wa58yBRx6RJoNrr5U3iaqli9wMYetWaff2+aStuG3b4HsD\nB8KsWdzncjEDKLBUG41GHn74YT4+k8fK4cMwcCCvqkt57TodqjFo667ugKNv5z+x2WDMmLNPh3c6\nduyQZpPateWN9Ew3/5wcWakiJUVuW7WqTHhdATUazxdn0hbnLieZczPRWXRUu7saxshgVLLqVlnb\ndi3uvW5Ut4rOqqPx+MbUerT8MnXl5OTQrl07lixZwqeffsqePXv4+uuvS30e5yWUPjo6mujoaADC\nwsKIjY3l8OHDhQKucfYIEcDp3IYQArs9DkW5dN3EFJ2CobIBf2aRhTwDmGqZTr3Tli1wzz1BE82q\nVXD77TIt3NkSFwcTSwhoEkImMPH52EVQvEEm3t+1a9fp250zR9bb0usZYDPwTlcdDhFAILAbbIzY\nXxmsJ6R54+ab4aly9ixq1kw+SsuoUTKxWEFQk9stbyjfflvmrnhUlddSUliRnU1Tq5U3GjWiamnT\n9lUQOYk5JN+QjPALFJ1CymsptN3YFlM1+b1Ln5WOO8WN6pIjC9Wpsnvk7hAB92X42NJ/CzmrcjBG\nGWk+tTmR3SJLPF5JvPzyywwaNIhatWpV6Oy63GzgKSkpbNiwgauvvjrk9VFFbH8JCQkkJCSU1yEv\nOwKBPJKTr8fhkFNum60JrVsvw2CIuMA9OwOBgBzdWa3F8o3GfhXLlru2oOjkl7hSx0pU7XOa0fTy\n5aGLdD6fFHFVLW7/LQv5bXUENgMFKT+sViudOnU69X7Z2SE3mMY5sHqKmZdeu5Es3PS/oj9Dnh8k\nPUVMpoqrx1YCfn8uOTmJ6HRmIiKuQafLF9J//gmNSPX5ZIrBcqDvli38eeIELlXl75wclmRl8U+7\ndlgvoH/6zid3ojqkOAsEPr+Pg+8epOHrDQHw5/hDZoUgR+VCFYXf0819NpO7JhfhE3icHjb33ky7\nje2wNj7z+khycjJ//vknGzZskH0oxUh66dKl55SCtlwEPC8vj759+/Lee+8RFhaaJnRUeSze/EvY\nt+9/OBybUVVpo3Q4trFnz7M0a1bydH7qhqlMSJyAXtHzQqcXuKtF+fgD+/1Z7Nr1JLm567DZ4mja\ndDImU3TJG6enQ9eusixNICBHy998UxhgEnVLFG2T25KTmIOpmonIGyMLfyQlEhVVPDjFbi9f8VYU\nmaNlwgTGOp1sBRYDwmikR48evPjiiyXvd/w4/PprMZNFyxwLvzR8IdQXvn798utvKXC797N+fQcC\nASegYrU2IT7+L/R6m/SE+euvYGYqsxlOGmidC8d9Pn7PzMSbL1BeIUj3+ViRnc2NVaqUuf1zxX88\n1HVT+ATetGC6ishukVDkI1RMCpW7VC78Xqo+lZzEnFCRVyBreVapBHzZsmWkpKRQr149QOpjIBBg\n27ZtrF27tsR9Th7cvvrqq2c8DpSDgPt8Pu68807uu+8+brvtEnCDu4jJy9tQKN4AQnjIy0sucdtv\nNn3DE/OfKAwxf+jXh7AarPRq1qtMfRBCZcOGhHwzjheXazfr16+nffut6HQlJMcaPFj6IefXq2T2\nbGl3HjascBNbExu2JqUMY7/9dhg/XppSfD6ZhOOjj8p0TiXy2mvQqBGWuXOZX6sWmcOGodSoQWTk\nKabJK1dKc4iihJYzB7lA2LBhuXfR7Xfz7up32ZqxlWvqXMPQNkPR60oe2e7c8She91HQ5Sf9ytrE\ngZQ3iGn0GrzwAvz9t6zSqyjQujW8/XaJ7ZwNZ7O6tXkzrFsHdevC9deXWzxViUT1iuLIZ0cKTSQ6\nm46qvYOzPltTG1fMvoIdg3fgy/RROaEysdOCZl/FoKAz6VDdRYtigqFy6eRyyJAh9M+P6BJC8M47\n75CSknLmdZVzoEwCLoTgkUceIS4ujuGncSXTKB1hYa3JyUksFHFFMRMW1rrEbT9c82GheAM4fU4+\nXvdxmQXc5dqDy7ULIeSIRQgfPl8GeXnJRER0KL7D+vVB8QZpWkhKChHws8JolPbuGTNk5GTnzhVT\nQUBRZA7zBx8E4IzjxTvukGVpimK3S9POxIlQp3yDsAJqgIQvE9h4dCNuv5uftv3EXwf+4rs7vytx\ne+exdWAskmLYEMD590xo9Joccc+fD4cOyVlSvXrloqBVjUauj4xkWVYWLlXFqChUMRjoeFKl56lT\n4Ykngoe87Tb4+usiXVBVOZOrXFn2tYw0ersR/mw/6T+koxgVGoxqQNXbQs12kd0i6bC3hO8zcgGx\n0YRG7Bm5B9WjojPrsMfaiepVuqLEVqs1xBU1LCwMq9V68RU1XrlyJd988w1LliwhPj6e+Ph4Fiw4\nu4K1GkFiYsZgt1+BTmdHp7NjszWnUaM3S9y2pFSx5ZE+VtpNTx5bCRTlFPf6Jk1ArycL6AGYgCoz\nZvDddyULTakwmWDgQLbefDMJw4fTxG5nmMWCKy5O+qCfJ77+Glq2hBZxgmnHeoS+abXKm9SuXbIA\n4zkihGBHxg42H92MLxC8Ef596G+2pG/B7Zc3c6fPyU/bfuJo3tES24nYY0YpktRS54JKK7OCLyiK\nvMnUrw+KQl4ejBsnT2HmzJLjs0rDzy1b8mTt2lwTEUH/6tVJatMGWxETmM8nj+F0yomLwwG//CKX\nNQDpUVO/PjRowJ6I1qx67leysko+VmnRmXTEfhlLZ2dnOmV3ou6IumfdRu1htbly/pXEvBpDk0lN\niP8rHp3x3OTylVde4auKSuMoKpAKbv6yRFX9Ii9vs8jN3ShU1X/K7ZbsWyJsY22CUQhGIWxjbWLN\noTXlcHxVbNzYQyxbZhVLliCWLbOItWvbn7ov+/YJUbOm6GkwCJNUfgEIm80mEhMTz7kfR44cEZUq\nVRJKfnsWEL1BiPBwIQ4dOvsG3W4hhg0TonZtIVq0EGLx4tNu/sMPQthsQkhpE8KGQ3zH3UVesAmx\ncuU5np3E4/eIbtO6CdtYmwh7PUw0e7+ZSHekCyGE+HPvnyJiXETh58sohHWMVaScSCmxLd/wQWLt\nR4hlCxBLf0dseQmhdu9W4rYulxCxsUKYzcFTefnlMp3KKUlPDx6n4BEeLq+vEEKImBghFEWM5E1h\nxSEqkSUi7H6xatXp271ctOVU51Ha89ME/BJm1YFV4oGfHxAP//KwWH94/Tm343A4xMSJE8Xzzz8v\nFi5cKAIBr0hJeV1s2tRH7N37kvD7nadvIDdXWM3mQvEGhMFgEOPGjTvnPk2bNk2E2e0hbepBeMLC\nhJg+vfQNZWUJ4fMJ8cADQlitoQL8zz8l75OTI7o1OxAiOiBEAovlP3q9EGPGnPawgYBb7No1QiQl\ntRKbNvUSTufeYtuMXT5WWMdYCwXa+JpR9PuhnxBCiFxProh+J1roX9UXvtfqo1YioAZKPuCRI0KN\nriHcda3CW8MiRETEKc9v1iwhwsJCz81gEMJ/6vHCOaOqQtStK4SihF76vXuFvJPodGI5HYWd3JD+\nVK9++nYvF20pq4BrofSXMNfUvabM6Wfdbjft27dnz549uN1uJk2axLhx43jqqf8rfSNhYVSKjMSV\nllb4kslkKpPNz2w2F7PTKoBeiNIlpj58WOYi2bEj2I63iI3B65W5YFu0CN0vNxfi47Huew8InXpb\nC5wNjUYCT45ke78tHP/tODqLDu/DDUmqVov69eGuGzLZtu4mMs2bUBUvDsdmcnJW0779dozG4DVZ\nf2Q9Ln+wZplP9bHx6EZA1jlNfCSRwXMGs+v4LtrVbsfHt36MTtFx5Isj7H1uL6pbpeqdVWn2STN0\n0dEoW7cRit5jAAAgAElEQVRh/vlnmUCrZ89T2uVLSs0iVMG+5ER2V8qmfqX6xFYrn1gORYFFi+T6\n7/790vL03XcQEwMIM4SFsTOnaTGjXUaG9HwsB5P4ZY2WzOpfzi+//EJKSgru/PBqp9PJc889d1YR\ntH6/n0mTJmG1WjGbzdjtdho2bMh99913zv269dZbqVatGqZ8e6odGK7Xc6RRIxbpdOzdu/f0Ddxx\nh/R19vmkWHtPqnpkMJScG2XaNDh8mJfU17DhoMCXzIaDlxgjt1FVdg7eRsZvGahuFX+WH++E3cx6\n8QSDB6nc1iCRDONa1EKjtIqqupmychixH8Ry9ZSrWbF/BfHR8VgNwcUuo87IldWDScXrV67P7wN/\nZ9/wffzQ7weqWKuQ+Ucmu57chS/DRyAvQPoP6ewevhuAvP1Gkt5oyoqnGpLceS3etXtKvDQne4GY\ncRNnW0qrn67lnq/70ObTNry8+OXTX98SCASkp+XJX52mTWHPHhkQmpMjK/0AshPTpxNn3kuIXx9Q\no4Ym3qVBE/DLjFIkBgwhJyenmFj7fD5UVT3FHqG8+uqrWK1W+vfvT1xcHP/73/+YPHkya9asKVVS\nqFNht9tZt24dw//zH+7p1In3e/Sg1b330nTXLvrecw8tW7bk3XffPXUDGzaEXgy9Xnq4gPwbGQkD\nBhTfLysLPB6uJokVdGIQn/MIn7GMLlzLarlNtWpkLnEg3MHrZkGljZqJw6njL29xH2uP38PPO35l\ne8Z2kg4l0ePbHtzU6Caurn01dqOdcFM4DSo34INbPjjtdTk+53hIylzVpZIxOwNvupcNndbi3OUl\n4DFwYl84C9rt4o2R6fySvIRJf0/ijz1/ADLWatkyaNMGaunSuI1Z7BraG6cJsnVeXH4XExInsDFt\n42n7UpRZs2SurVq1ZIT/5s3Ft7HZSnB+uflmrtnxJc/fsx+zUSU8XBAZKYNdNc6MVtDhMmH5cujX\nT3pjNWwo3bHj4s683+7du2ndujWOfN9ms9lMp06d+OOPP864788//8x9992HM39ObjabufXWW/nx\nxx/PuK/Xm0ZKyhg8noNERd1KzZqPnDbkODs7m+jo6MKZAkh3rS1bthATE1N8hzp1pNtcAXa79GVz\nOGSw0JNPyr8nk5QkA5MK7Ax6vRxSqqr8v1YtWLiQv2/LwbUzaP7wAr8TzRxqk0aAH55pjfWG/ahW\nUPwKxwIK9yepFLgWKyi81PklRiWMYlv6NrwBLy2qt8CkD6YZ8Of6cWxyoI/QY29pR1FkWPj+MfsR\nvuDvyhZro9Fbjdh62xoCgaAnkgcd9+niOV4pA+PjbTFaPQxtM5TxN42XGwQCYDRyOEzQ+ClwFYmA\njzBHMO22aaVKcbx3L1xxRahppkYNacXS6ZCupm+8Ia/9kCHQp0+J7Rw5Ij1HmzQ5c/W7y0VbLmgu\nFI2Lg2PH4JZbIC9PPt+7V1ZNP3AgOOg8FY0bN2bevHkMGjSIjIwMunbtyhdffFGq4y5btqxQvAE8\nHg8rSpGzxOc7wdq18Xi9GYCfEycW4XbvpWHD10+5z+HDhzEYjEBQwA16Pdu2bStZwL/+Ws7VdTop\nwB06wNixxaM8VRXef1+WiatfX+YN+fprKfYZGfL9gtmIyQSffQaxsTT96ASbe21GuH14VD0GoCvp\ndOMYyYTT8l0P6n440V6Hpf41DEnbjVsNugDqdXpMehM6RYc4Jli1bBXbq2znzjvvxGQy4dzuZEOn\nDag+FeEThN8Qzqr/W4W7jZu2UW3RZelQ/So6k44m7zdBMSogQm+AOgR5qg2RVw3v5t5447/kw7Uf\n8nSHp6lXqZ68FnXqUP1QKhZ/qID7Aj5aVm95xs8SIDlZWqSKkp0tBxM1jm6SvvwFwU9Ll8KUKQTu\nuYdfMzJI83q5JiKC+PBwatY8r5kHLgs0Ab8MSE4O1SUh5FpcamrpAgQ7d+7MznOo6lK3bl0sFkvI\nqLhmKX6BGRm/4PfnUJBGSlUdpKZOICZm7ClH4SZTPRyO0BFJbl4efXr35j8jRzJu3LiQfdNiu5Iz\nZwcxh/7CWKOKvKOVFI7/xBPS7u10yrvd7Nkyi+Hhw8VH8S4X/PYb3HQTkddH0uaFA2S+Op8UtR8+\nKmOPXwNPvM814XlkrLByxac66lR+kBMP92X07u94I2kmG0540Ck6wkxhPNj6QWbPnk3//v1RVRWD\nwcD48eNZtWoVW+/diu+4r9Al/+iCo8w3zmdhq4VEPhDJt6ZvaWZrRpVbqhB2RRhCFYTHeEnfY8KA\nghsdP1EHN3pQDeCVKS5MehMZzgyiPdG49riwTPkZ8z3dmbveSfIwD2EmWJWpp0PLyTSu0viMnyXI\n6MqTTXdCSCsVoz4KjVx1Ogm8/jo9YmNJzMkhgLR+f9q0KQOiT5GuQeOUaDbwy4Dq1UODIUE+P6MT\nSGamDIurUUMaRDeW3uYJMGzYMJo0aUJYWBhhYWGEh4fzeSmq3gjhK/6i6i8eol6E8W9bUfgJCAOC\ndyt/IMDkyZP5+eef89uW1pEGDaBN79o0eeluUpp0L1m8/X45oi6YRfh80gY+b558flJEoTAYmbDz\nVq6+WnpV7N56mLq+7xBY0DXcA6+/AA1ToFoGWX3S2bnuAXaMVPlnaz+ivbMY3wpeb9uaYW2HkTw0\nmToRdRg8eDBOpxO3201eXh7bt29n+vTpuPa6QuKpzF4zYalh+N/3k/5WOje9fiPL5/QnrJE8L0Wn\nELepJ5NoyjfUYwxxTKHg7q1A44UAGHQGqi6uyuoGq0m+KZm/b3Ow/8Ex+Eb6iY2EunboX99EB1tS\n6LVyu+WMpIRpfatWcs24IE25zSaLH5lMlFhSbn7z5iTm5JCnqrhUFaeqMmTnzsvCJHK+0QT8MqB1\na5kgz26Xblo2G4weXUx/itOzpwyxPnZM2im7dJEluwpQVVizRq54Fdhn8vFP+4g9n9Vg/LjNvPBf\nH2++eT/btm2jTZs2Z+xvVNQt+ZGdcsSsc0P1xQpKfHzxcHWAnBwOfbcMVXQHjiJFPIjD4WBVfmjf\nzz/L0G2PR3Y5NVVemxJR1dOXiHv/fXkxDQawWHjZ/CYv/9WdpCRZx7njrBHstrTEzj64ZiUYgzcm\noXdz7PhMjh37AVV1oKouFOHh2rCtTLxxHPUry2RXWSeFHXq9Xo4cPYKjvgNVF1ysdBvd/LbjN8gA\nBIgADF21io133FG4jcGi53dTbb6gIauQoeMKASolTEZXdQ8xlWP4o9cf7Hx4J8IlELkC1aWy7+BG\nBMEhtKq6OHr0m2Cnxo6FiAjU2rXIbt0cUWRWMmuWjICfOVNOYJ5/Xn6VCteHBw8ONWjbbOy/pxOq\nKJIdEXCramFSrMuBhIQErFYr4eHhhIeHV1iKbU3ALxOmTIGffpJ5oBYtgmefPcMOWVnyl1bUvU4I\nmbUOpAJ27Sp9znr3lr5gKSnyvRkz2H7oCTKb52G0wzWdPbRsPoWwsAOl6qvZXJurrlpN5DYr9r1Q\n+2doNs4v1faDD2Tl9aJVcV58kV7OGdjJA2xAaKY/q9VKw3xb0aZNoQN5VZV5sUrEZJLJswq8ZXQ6\n6bt2443y+fXXy8IHo0fDm2/ykelpnM7gT8blNzCjwbO0sLyJweOHQKh9XVH0KIrxpNeM+P2Zhc87\nduyIschChcGg4yCj+fvx51l8VzqrrxYc1h1lsWExjmxHyKhcAKuLpCDdl7WHjnf/hVEnL4AJDw1I\n4eDad8k4/D7fj3ydrCvSQhOgAz61uP28sN+//w6vv85vMT4iRvqo1mcntSfHsOnoJvbvh/vvl5Yl\nt1vavSdMOMls1769HCRcfz106MDxL4YSUfUDAiJ4w9ADV9jtmMsz4+QFRlEUPvjgA3Jzc8nNzWVb\nOaXvPZnL54r9y1EUqTvDhsnsoWfEYik++lRVKEgHPHmyHH3n5Unn3WPHZJUcgM8+40RrFVHET1fF\ny4kTi4odJi0vjUGzB3Hj1zfy7up3UYUcVdrtcbT6n412j0CjT0EXQN40XntNJq+qXVt6LgD88w+P\nBD5hOBOx4MLAZxixEqEohNnttG7dmkGDBgHQuLGciRS9LiWtcRbyzTfw2GPSDtCzJyQlcUTksOqB\n6zkWZZEVgVauBK+Xk7PgKoqC4YH7sKybR/sR92G0RKIghU+ns9GgwSgoMrJFBUO2D/O+4Czjhx9+\n4Nprr0Wv1xMebmP4CD9N28XwRpNxvDmkCi+MgwELqvBel+JFJvRArfzRbXJaMq0/ac3yRl0J9BpM\ns0bT+cq0hCnKfhJdb7Ppk4a4/NHoAjb0auiNRl2agN8Lan5XlYCR+vXzU+omJZFqcnF3X3CYwKeH\nI4qZawYu5IEHVE72NnW55PJBCJ07w59/wurV7In5k7piJy8zmjByUVBpZjzB3Cuv5HwSCDg5dOgD\n9u37HydOLKmQY5wPk5C2iPlvxWKBZ56RI16HQz5v1kyOlEAu5LmCbnIEAjJtLIDdjt4BgSIzY11A\nFxJlCJDlzuKqT64i3ZmOX/WzMnUlOzN38tEt+elhb7hB2jwKig0oSugxR4+Ws4C2bVESExnjfpnR\nvAwmM+m9e7G6f3/CKlWiS5cuGPLdIO69V85Efv9dLuwaDDLy75SYzfBOsPDx5xs+58lpQzHVCuAd\nBlN/8XD3b7/BwoWMNLsZbXsBh1OHTictA/3vVaBeHCbiaOfdTOK25ziStR1huZaraw7DYGnOrjW9\nUC1+rEeg5YtelJzO8lpWrUp4eDjXXXcdXq+XSpW20raNkyd5HrdSxIfeaMT3zOOQcQjWrsWKFO9O\nwK3vvosjEOC2pPnkNfsf5O7A5v2JdxbcS5hXuhQq3uiQSEcFBYHAaXKiV/VMaT+FpA2CPg0hyghb\ncu180XGo3LhePTbWNWFU8z8jvwmmrMaZ2ZhlAR1yHhC8swUCgmrVTu0OeuSIi0qV4DpWMYfeqChU\n9t1O7U9SpQH9rrtC78DniNO5i8zMueh0FqpVuxujMZgmWFXdrFvXHrd7L6rqJjV1PI0bj6dWrUfL\nfNyi/N///R/PP/88zZo1Y+zYsXTp0qVc2wfND/zfjRCy3uSKFXLe++ijUsgBPv4Y/vOf4AKf0ShX\n7n79Fdas4fh/O7HleQ9CB0oAzLZ6tO24Fb0++OP7fvP3DPltCHneoP3coDPgftEt81rn5Ulj6bx5\nUkidztBZgc0G770nt7npJplQGmSKwD//DM4W8jngdjMqJYU0r5d4V1W6u2oSH68UWwuYM0dmgDUY\npKmpWzf5+qGcQzR5v0lIeLvVB4fHQ2W3XMT8steP/ODqTVQUvPKK9FkuYMq6z3lqwVN4Ax4sBgvR\nYdEcOpGGWHsHOGoxakcSzx9eIgVq2jS4805uv/12Fi5ciMvlQq+HKlUgb8ZcXEoJjtBz5mCfMIH7\ngD7ATUYj4rbb6PDcc6zLzkTojBBwY0s/yPQHown3BK+PQKAUEVqv3sszj4zgaOQxMiwZIYGQdqOd\nT279hAFXDgC/nw19r6NjyyScRmDXzTBzOniLVokSYMoG1URE3/+S8sUYIq3F86qrKgwc+Cb33fca\nVqv8XrndVjJf7sa9/yySd9yaNaVp7wzpEk6nLTk5iSQn34AQfhRFh8FQmbZtN2IyVQPg6NFv2bFj\nKKoatLXpdHY6dw5+T32+DLZs6U9OziqMxiiaN59KZGS30/apKElJSbRo0QKTycT333/PE088QXJy\ncqGp70znUVrt1Ewo/2YURUb/TJoEw4cHxRsQgweTfd1gMgwJuC31pFJNmSLfbNeOqImJXLWsHw23\nXkOTyJdp23FbiHgDIXbOwnaFQBSMB8PC5A3B65VifrLbjKJIm4jVKhdSN26UPpOrVxcT72NeL1et\nXctXaWnMz8zkPc9uFtXbV0y8f/0V7rlbsHixHKX3ulVQYEbee2JvSCANgCEAB/LbUPw+HmqwlPnz\npeWlQLz9PsG7HQ6Q1zaeZ15KpOovE3F4new5vg/31Dl4Fn6IZ+kY/i99DmP0I+SMZ/9+cnJymDt3\nLq78WUcgAHkOqJW9lWIpfQMByM1FAE8BNwM6n49N27ez3eGQ4g2gt+CqWpej0cUDABTcCARudEwR\n9dmW0o0Ma6h4F+AJ5I+4DQbif1rNQ/X6YNeZsYjKxbdX/HDr4zCsFeqVXzF/d/4FatBABj69+GKh\nL/2CBYPIyKiJEOD3G/hl5pP410ZKI7rDIddBPvuseIfOgp07n0RVHQjhQVVd+HwZHDwYjNqVLqyh\nth9VdSNE8LXNm/uQnb0MVXXi8aSyeXNvXK7dpe5D+/btsdvtGI1G7r//fq677jrmFXg3lSOaCeU8\nkvFLBjuf2EkgN0CVnlVo/nlz9LbzUDvQ55MBKgsXSt/mYitNoQgh2DZwBxmr7kCx3oHwC1pOaEGV\nakWS4rduTVjrH07yBwmlR+MemPVmnIoTVahYDVbuiL0Dg+6kr12B//ZPP0k7tF4vRf2hh6CgzFSB\nmJ+CWenpOFW10OLsUFXGHTjAlCNHaBseztTmzalmMvHOaBdOV9A84XIrvDsmj4SEMBpXaYw3EJoz\nJaCD+gWOIjabXNA9iRkddxGblIYFefx2G1rzQOQzZNfeBEfagi//KvkMvMobPMf7GKdOhXy7fVF8\nKtyU/Aazr32fg8Z8v2hVhdxcTLNmMQkoGmAbCAsr5jsvFMGquL+pdaQd+oAeVaeSdMU8qmyIZ4u4\njrVUY51aBZaMg0qHoMVPcseserB0FG5nbTLC2iFa5380Oh2Th/xC/wMrWbc7hWfn6PF4VRA6MLig\ndiJcmW+nEmGIjckw9IPg7G3iRDCZ2H//K0yY0Jfo6AMoChgMfm7vN4krFxsgJb/zHo9cbykDfv/x\nkOdC+PB6g4nW5Eg6eM0UxUTlyl1QFF3+5faRk5PIyTXVsrKWY7WWzjf+fKGNwM8TuWtzSb5/C95D\nXgI5ATJ+yWD7w9vPz8EffFD+iNatk/aDdu1kmNwpyFyQScbsDFSHSiBXRXUJtt579n2taqvKmsFr\n6NW0F21qtmFEhxFM7TP11Dt06iRra/78s+zr+++f/gDbt0N8PISHE3jrLcRJK2qqEBz1+ViYeZxu\nGzeiCkHajhMh2zRosIUW1d/k+PF5RIfV4LNen2E1WIkwhGHDyHdbY6lkrSwXVadMCd5Q8hFCUGPN\nESz5P3Y9YELl2k09wBVJ4Uja5oc2mYiWOeTowuDYMSIiIujVq1cwZ4xOmpi/VI/TdvGzTH5nLsNW\nC0bpq7Hu0Uc5kZnJI0UPrii0GjGC2mYzpnwRNysK8ZWq0vBZI2PvGsun3T/lxXtf5PU+H/Ni2LV8\nQjPWFdQf8tlh253y/7wa8Ml62DiQwM4bee2FSE4uZ9uudjum7HgT8UgHqL8MJXIfEVctxPqA9NPU\nK3qsRis9FuwJjat3Olny2W6uvFKlbt3lGIu4W1r0PtQ2wUAwbDbo3v0UH3j+DE49vWkhKqoXOl3w\nJq3T2ahaNXjjtdmacsUVs7FYYtDrK1Glys20aDGzyGU1oNOFzsRAwWCofNrjFpCdnc3ChQtxu934\n/X6+/fZbVqxYQY8ePc6881mi2cDPA4k5OfRKTOa4QSXyBIx5CWK3g86io8GrDYjoEEHlzqX7cpwV\n+/fLvBMnB+jY7fDhh9IHrAQOf3yY3c/sLqwpCIACXbxdUAxlLMUlRPkURHQ4pHtJfnDJwerVaTF1\nKrk2m5TMk45jQjA561qeuAO8+e4zCQkzeO65h9DrvFisVsLoROC/r+JJ80IXaD6xOVGRp4+GEkKw\nWL9cprltuwY6/oUvuxJfrLkB43928PXQ/4MoPXywHowqOl2Advu3sfSvv9BP/YaU5KN8Pn0iy5Yt\nZF/eJqx3q0zqBDZVh9FsolqNvjRvMhWlRg0ZeFWUhg1hzx4yfT6e2b2bzQ4HV4WH806jRvzfwqf4\naG1oLVHj53/jS21f5BUf1EyGyBS5kLG9DwSKlgIT5A7+j1yjqFGDBS/dw11rnyPXG/SiMegMPHvt\nsyxOWUzdiLq80/YF6l3ZqVjcQD3jYVJ9NZk7NxybLfieXmen6W/NqPHeP9KE99ZbMHRoidd573N7\nOfjeQVAhwZ9wSm1RVS87dgwhPf0HFMVIgwajqFt3xGk/x5M5dOgj9uwZiap60OnM2O0tiI9fmV+x\n6vRkZGTQs2dPtm/fjl6vJzY2ltGjR9OtW3Ebellt4JqAVzA5fj/1Vq8mu0iscVgu/HAXWD2ygKpi\nVIgZG0Pd4Wdf+umUCCG9SvbsoZivV1iYXKQsKRsfkLMmh+SE5GDWOwWsTaxcvSOYZU8IQfrMdPKS\n87A1tVFjYA0U/UnCnJ0twyKTkuQI9sAB2LsXf80abJkyDl38VcRViztlod5TndZ338HCb9Op++c0\nRnrHEom0cWyLjeU/n33GX+7j5Co2KGKqUYSf3l904ddvCn6AgnnzwrBanZw4Ie8HNStb0I8eBUlX\no7PoiOodRYsZofnCN6Sm8tqhQ2SZTPSsWpX/1K3LugE7yfJMxjjkEzD7ED49elNVrrl2Cz//4eG+\nlCP4muUWBpBafD6GUpO5N1cnLCA4hI1nr1tEvy96ciStOW41pfB4Op2dWPU5qvUcV+ihs554/sM7\nHK91JXcOqcpLLxVP8fLWyrd4ZckruAPB0W1z9wOkvv8lHo/AXyUPnt4L9Z2wMwIm1YcsC0WtqkbF\nz2eGYQzwTcVAgJ/iLTzU10COLyjARqHj2MijVA7LN689/bQcHBSNwNTpsBm9uDx6brzxK0aMGIZe\n78UvFCpXasbV7daxeLGJjz+WrvkjR8JVV4Wez+FPD7N7xO7C72RXula4tmRlLSc7ewUmUzQ1agws\nYVRedjQBv8Coqpe8vPWAnvDw+GK1I5Nycui+cSM5RQTc5oAJz0CzIulHFJNCZ0fnso9wC8jMlCv6\nJ+fB1ulk7P22bTKE7hQc/OAge57Zg6JTMFY30mpRK/SNLOQGAkQZDOwctpOj3xxFdajo7Doir4+k\n5a8tg/ZYIWQQx+bNQTdBpEZ0eghSIkHYbLSo0ZLF9y/GbgougKakyNDszZshOhq+/16eyp490gL0\nxRfgcar0IZVmZDCU/1GPeSgWC9s3beLK1L34MICik6PwgIvK2Un0eyGBKTs6ItBjNLqZN8/Gxx8L\nZs+WAhgRDu/FDSV6mTQJKAaFzt7Ohef02HPP8VGHDtJDwmDACjxQsyYvWBewZ8/wkMU9RTXQaFkz\n6hzrRL0BA0g9KaS8wQIbH7/pxoeCAcE4mpFkjGL23DCMxqJRijqq+zoR23c9Sk4uu2lEPBvIww7o\nsJn8DB4MEyeHfu+cPicdpnRgX9Y+FBR0io7JR3qx4qtVrLfGsXbykxBhAgPo1AC1Akd4dO0MJk74\nkIyMuoBAQWDDSTvWsIgbOFHFRNPHVbIUL0IHJj+0TdOxMvoF6fIJcsY3e3bol6lRI7rW3c3S5V5Q\nTcTGJtIqfhE5Lb6juy6GRn/exN3LHsPlk+dgs8l4srg4ubb+yy/wXO5mWjuCtu3zIeDnA80LpRyZ\nMWMGV111FfHx8Wcuyut243v2UdbMDGfj6o5s3JDA2rVt8ftDQ8FrmEzFQoT9ZohynSTUKgScZ5nM\n+3SU5Ial10OPHtJN6zTiDVDn8Tp0yu5Eh30dOL62CXWPrseyfDnVV66k8cpENs49guqQoyHVoXJi\n8QnykvPYn7WfLl92ofpb1ehy5XpSLB5evf9+av74I3VnzKDnf3qxMwryTODwO9l0dBOjlo0KXgZV\nuvVt3Ah+vyD3oJebu6m0bCldhCdPBpdT8DabGMx+biSXPQxnl2EkDBjAYw4Hfp0ZdHop3iKALmsj\nY5f/zvMpwwgjFxQfPp+J2XNqMneuXON1uyHjOLy6+vfCvijG4Gc0b948vtiyRbo75vucu4ApR46w\ne9+LxTwzhOonKQemHDpEzMaNFJ14W9HRZ7MTMyphBLCgMoy/CfjG8s474SQnF21JJd2YyLZn3AgF\nfuZ2PJgp+Ok6vQa++NAFv/6K3+XjxRchNhZu7Grjw9Zr+ea275l0/ZdM3NWToZO+4dOsvWyISgGz\nr3Cwrer0ZBijqN1uE5MnX4fJ5AQUBDochLGWtvxJN6pm+1j5aYBrD0LdbLhjG8z7WpXx9AXcdFNo\n6LzVCr16MWMG6GtuAcXPtp3xTD+Rxh+ebfgW/M7oRVfj8ukx5q8jOJ3w7rvw+OPS2zItDfY5zPhL\ncpf5l6N5oeTzyy+/8PDDDxemRx08eDBGo5F+/fqVvMNdd7H7ynm4IwMIEyCcOJ3bSUl5hcaNJxRu\nVt9iYUSdOkw6eLDwtaer1qR65hHUggUuPdia2zBElOPHYTTKX8F//ytV0WCQC3CzZ5faBu0+4Obv\nPZn0te7Bk99XAez1e3j5FfhkcHBbRa/gOuGi49KOHMk9QkAEyKwNrcf0w9foHpz5C3WHE4ah7siG\njOXyGH43KzIO8vmRI9QymWjtrcLhwwrVVDfjSaYaHhQvfEwjfnLLEmFx5NCM3MKFQx1GjtCTmDev\nI3X3hlAHPEWPCDj5xv03vc0BNnuupFGnBwkIE1+s9RSdHKCqkOI9Itu06aj/Yv3C0femTZvwlZCY\nSSBAMRbz+vtSPMDMHndLRwZFIdztJ0enQyhw05ZjdJsfTYGXQzbZPMbD+Mjl9999LF0Kzz0XjKkS\neMi4Vo8zRodxrx/dSS5wBuEj0LcvA8JG86urGx5PK8DEDV1NCHGrtKD5hxdG0Ae8Xjk7KYKKDqvB\nTViYi8aNk9m69drgJUQli8oQCBB7FP46OdtwUffPRx+Vs7uPP5azsJtvhjfeoLoZnvr8az5OnIZL\nnABFYPbCHZv8bEfwKiswo5KCnee4Ep/PzOzZwbiur2lAJzKw48diBjxooAl4IR988EFIbmun08mH\nH35YsoA7nTB/Po5b8sU7HyE85OVtKrb56w0bcktUFNudTmJtNq6tVIms36qxbeA2vOlewq8Kp8WP\nLSlkMPUAACAASURBVIrtV2Yee0waE9eske6DffqUWrxTJ6ay74V9TL9H4L1P5eQ48j0xSJtufj5Q\nxaiQWieV7L+zC/2/A3rIrXcDapHKPKrBSpz3afa71+GwONBH92Bd9CA275Kh4l0iKhNQr2AMm4mW\nyVABGMxedhCOo0Eq//3vA1iq74d/WsI7I8ERBkY9AYdKR7tKqlvFI/IFKuBCHNrH384udLp9NzOn\nhXP16j/J9edCRC0264+H5AapW7UuNW6pQdQtUVTvV73w9SZNmmCZNAmn1ysNtQYDisfDPbXC0J/w\nhkhqulqV6fTHU6QmmNOjwPAr0R0ysyHHiUmXDPklaeczn1xyAR9ERuJt25aJG31cc30i1vz854pH\ncNjWiR4c4W1yOIqBAAZs5PEMr9He6Gdj7ihUdQxQG1iJxxMF7MvvQZEI1927Yc9u9E2bEzAbMeOm\nLWupzjHcOjs+n4WiEZa5hMuZy8no9WCx4Bg7kcO75DKHzabD9db7vBk5keRkaBuv51kFTMDb3d+m\nirUKs7bNIsql8PbkndQ4UYMB5KHLl6J6OHhD2UTDIe1Y9v/snXeYFGW2xn9V1XlyYmCGMKQZguQo\nIEFgMZFcJagkCSqKIsqCgAhKUERFAUVRAUWRKCDJRJaco+QhzDA5T6dK94/qCT2Mq3vR3b1X33l4\nHu3uqv6quurU+c73nvfdYSyjAGRjYTAtaUsGwdL//dLJ74W/ArgP1nIM+Mp7DSg2CQg6B4VxFGuC\niLqF4OBWhqJeYKBfsGwbEkLbUl0loR1DufP67RkS/ya0bm38+404n3meTTs30XBcQ0SviD0LJFVA\nKVNsi7KYCW5tp/BUIfYadup+UZekiCQUrUyWqrn8/ldQITYthIGrXmb8Yy+h1hoNgoTiW2jdnJNF\n9wU3qPF4oV99TwAahSTS772OBATkIoo6auufOP54ItlzB1EvrBlZ0uf0zxzHGW0Ch2hiWNkfuwrj\nF2AX3bT3nuVu7qNQLkRDw5prJdwUTpY5i0B7IBaThdVbV1O3gaEcd+zYMY4fP06NGjV48MEHWbPm\na5a//S7qpJdA0xBNVqpdqEtMw3EkJ89EEKzoukLU/nux3qHgMZe6fmQRkNDyHFzFzlt6PC8IF5El\nmXzFhYJiNL7MnQuiSD7wuJ7LR8IIArUCVC2A5MvPImBnOdtZRh7XqUA+Dra2/YZTB0BVPYAHQbiM\nrj8D3ACO+AYQAtgBF2galhcm0LXPQjJ751Ap5GcGmBbjddvJvViX3hetHCaVH4n2nXmRASwlmRhM\nNi837wM5BMIC2rC97jL6d8tBlnui6xmMGNGXEydGcfSohMsF3/0A3x+QeWJhFiZB4Lk245nUfhK5\nN/J5e+IC7FSjZanpiwmoQSEd79J5802BIUNKpJILMfEdFRH8L6k/Nf4K4D5MmDCBbdu2FWfhdrud\niRMnlv9hmw0GDqTmZ19RUNtFYXVAFAiW6hDXej5kzzJ6ojduhObN/30HcZvYe30vXT/vSq0rtZgm\nTCOQQLp+D1/1g7QonaKk1iLLLGvZmKa7QyksPMXp0/04nH4FR2EC/eq0Z/m5XThlJw6zg5baefar\njXELOoIONjcMWCpR8WZTEK1+TJEibK15hdERZsgs4QtrCIQ12o9FVBBFHU2DCdMFjufZ0OJWIN6c\nzcwVMk2ayLwhjMWDBa+i8vziHVzvms0D4VdwntyJ94QXzZcve/CgKzpvr3mbJmFNaNy4MaG+tYF5\n8+Yxbtw4RFFE13UGDx6Mh8WoY38CixFwVGCG/TRfD3iJb9Y/wnF3MtEBcTRpEYVYpOpYGteKasMC\n3+tVOWAKJrrJ+yQlrIFlgrFi53AUa5enaxaWqo8x9MYWLK++jO4MNx5KSPyDtTzOUJRmTrzuRLyl\nJNZ1XQHLTnhhKLR9ySjwz/8Itt0kwHQFh8nMuMaDORav8m1IAgi1Wa91Z9z2n+jw9p3cq6fTkUxq\nUshH1ASggECuWKuQ++ElPBVAs8A18SDzpy/F7Z4OFAA6CxYcw2TKRFGmAuAKc7Fz2GEO/awjihBm\nMrGzbjO6dAzkuncMzcmmEWewl5rDmAIlBJPAo48abMSnny5R+BVFY3Hz1Knffl3/f8ZfLJRS2L9/\nP/Pnz0fXdZ5++mla/7PMVVXh3XfRf/wBd4MIGDgYW6ueCAWltEzDwgxHl9sw9/13otXCVhxIPkBI\nYQjL5izDLhvjLgiAzfd5ya25i4SUGwwODSXqzZmYkNm3r7pPHlUHRCyWipy1TuZk+lkaVGjA4MaD\n2fnldT79KRHRBfdugYqpkGtx0evuLTCqM9iD/GYrZuC0Xp/k+84ABmXR3agAW8AEePEiOFx8sz+K\nt2vMBUegcVcfP07Y7ImsWVUSCHK8Fh5L34A73IJu0RFlBX3+fNQNa0t9lxmv7s/Uyc/PJyoqCk+p\nArnd7kCutBtlbgE4Si02F0iYPqmJ6ZlLWCyg6jqhV8IoWFCNgnGn0EK92DwirrHxcLIcx5kKJ2Bk\nI7gA3P85VKrs//4PFVgz+yZhgz6GOmfhck34ZCi4FPZSjcBuGzkY8TJfrdSKM1UB0P/xD6OIXjSL\ndLth3FKs55by4INzuPPxJYyRZqGUkrsVNFjXA4J8l7ACvPW3RJo0+5G01KqMy3ybm0+40UpdzkuW\n2Fi8WAVKm3SEAT7O+vQT0CqrmEJpFgQ65ldib68qFKh2BHRe5wR3kEtgAKBBnc/rUOHvJeWrTz4x\nqoG6blD/v/sO4uL+b8WWX8Jfnpi/I1q1akWrVrc6ipcLSYIxYxDGjMEOhquwVOZ0KorRWfhb3IX/\nC5DpMmhauQG5TH14Kq+sfAWTaCJEkZi2ZSZh2kEO9e9Ay9jNpM14jzZRwbxWT6ZkBU9DVfPoV+cu\nhjYracbo8FBVrDOTybuQh6RKuAUTc9R68H1HyM+E1/ylUq2ShLS7EEEQkBUZRVf4ke0cjwzipaRK\nCFVvsCJuHIRFlJgxNm5M9l3dQVtXzK06JLWiMNwCdmN8mmSCZ56EDesAHStW+nUyKINJSZCdbXTr\nZ2RkYDKZ/AK4GBaO3jLDZzNTCiYdpc9VFEnF7YvrhRWyIaAC9G6DPVhj1tJdvBDxGV5pFPhMhx0P\nXMTUMg1XjhNZckBtJ+hHQI0CyRd0XSIcCoU3B6DXPoVg80L9M+gNTyA8uYCWWjZki1xK0Pxo1zoY\nMrilS4AWC44WdQi4uoLHHhvPLlMHlDK3vy7ofPS3RFp8fZ0GNCBkxAqe7bUeu92JLFtIdXopo0SL\nyeSltEMSgCgKSJKv9BHt8Xtb1nVSC9PR1RjfWAWmUJ/7SOKtlyGyRyQBdf01dYYONZqJCwp+g0nJ\nnwx/OhphZqbBK92y5VaK9G2hPM6112twrv+PoFedXthNRnq1P34/j0x8BOc3TtrmdCQsbxuFWal0\nrbWHlMJUNF3jRkEObsXfBk3T5FtajkWbyKyxs5h37zw+bfYDo8V6bFerGV1/P1VG/KIaVgSCRRGH\nKPJl5dpcn3IVzaUhyRJWxUrno13wYMIzZjYsGkx+YG1/J12bjeDajSArAhQJzWVl1Y1AMPnTFQST\nhdZSW2qYahDVLIpqE+MYPRpq1jRiXlwcfLmsEoWFpdLMyEgK589FG2wx0kAdI7i6RFgTC2FlLOIs\nGlQyFh89BSLP9WyPd8cYqLobrLnUnrkSdcxF8u6S4QEda4sFRjnp0nzIPmTY7agCrIsl5lQB9tpn\njeANYJER4q5A/y+R0JGjr/P+B+UYC5W1p1MUvPnnyM8fyOjRMjHexHKvgS3Ct7zBGwxgAFearyhW\nDTSbvWihdgRLScbu9VgJC+uEUVs3ZlAWi4OxY5+nVy+DzlinIAx7KcaLQxTp7c32MWk04slnOXsZ\nzmUSX0kkbVn5OiiS9FfwLg9/qgB+7pxhLDNokCHC17z5P7Vh/NdQu7ZhkBsQYCxgOhwwbRpERv76\ntn808vPhsccMmkCzZsWyrGU1JWZ2nsljDR8j0BJImC2MKd2m0LNrT0SbcZlcyLpQbMgAkOiEA9km\nEOyAiCgGUKnSUKzWmFuGcCb/DBubbuSrWmc5Z/JfHLZ8UYOl1koEvfkmnr//nT51q7PJ+6PfZxRd\nxGP2cNOeDcv7En86CEEpGYvg8vDwz83g4ZXQay2BvT7g5UUXsGol5Q5J1qn5s4dURxKXbZe5kXmD\nzavdfPjhQTyee8jPb0Nq6odMmmgGfgRiAAlh0FCEwGB0q27Ud3Tgpg1eaAwLa0J2GRqhLsAFQ8BK\n03Q0XTAYJ9fbUanNQi63Csfj0/uWBQuiJQIpohVoXjg9CeveXnyUconBxzzcFZ8GUpmFYbMKA5bC\n2Fnk5ovlE4vee88om6iq0UiVnY2y8WvcmofERI3jK85Tg4v4DdzjQTnwE07f35uz/emKuiZQMWAo\nthSBwoJg9u59gHnzvgYOAA9RrdrdvPfe28ycOZEVKwxJ+WOjanBfRDgSRiI+MDqayc1qsMPWjZYc\n5HWOEYSKFQHdo3P9revk7s4t54D+7+Grr76ibt26BAYGUqtWLXaXty5ym7jtGviWLVsYPXo0qqoy\nbNgwxo0bV7Lz/7IaeIcOhvR10ZCsVpg0yfj3u2HfPrhwwdCsbtLkFz+Wn3+U7OzvkKRgoqMfw2Qq\n03jj8RjiSVeuGKlh797/ew2Rbt0MOVZfSUAOqMSp+mvIPeRGckjUeq8WlYb8upt8cn4yNd6tUSI1\nCthNVg488iaBQg6BgQ2IiOh5izqeU1UZvH4Ea059hpobBe+dL1HoE1SqVRWx2epx4cJ5NB8bxYKV\n+bxPLZ8xbyEil2NPkJBSG4tqJSNC5dl5EjnBGqoo0PIgTJkiIGlGV2v8A+eotOV5fkxIYMi4cWSE\nhhBUcIW0F8bD9TwfPxvM1lBktxcoopA6gFeBFwDozc/cmJHCwTKEoQq5gaT3bo4e6oXle/Dr1NGB\naQmI26LRdJFAZB7lGpVwk9ZxKwsm10MTSuoKNt2J9dIccpO+L37tvosiG1dZ0E0imxfKmCI0LOYy\njV4eM56hC3nU+ThZOdotWbiUUBu9+Z1oznyjcFwqW+nSBV6cYOZt7UUOFnZCScsmf86rfv5zYZFB\nPLKsGzlCIA29h6inXmPusD2Mz7iC/uyHCJ23oniteD8ZRtrGh8gcksCLj580brLoaCNp8JWcPJqG\nCJiLbNNWrkQbPJydzjWUziNFh0itd2oRM+LWJKA0/ttiS1l8//33DB8+nBUrVtCyZUtu3ryJruvE\nxPgf13+0lV5VVRISEvjhhx+IjY2lRYsWLFu2rNjA87/tJMfFGfpOpTFoECxe/O8dR2bmBk6f7oOm\nyYiiGYulEs2bH8Vk8onkK4qhzHf8uNHJEBBgaIrMnPmL+1QLVa7NvEbhmUKC2wRTZXQVoy3f6zUW\nUUvpoRwT3yFXaISuGoFWM4tkTmjE/f8IKW6iS0mB7783HnL3319ikjJ1+1Rm7ZlVbA4wquUoZnYp\nf1y6rvPcxYt8kJyMAFiciRQefgYuNYVVy6AwGqHCWV6bf4LJfQYUB28Ddu6jJx3buGk4dgtSgIq4\n507EmS+hyjZSo8GhymQHgMVjJia5aBKvE/FgJA2yn4Nt2/iuJvTuAx4TkAnaxwK6t+SalCQLqlq2\nllYF2h2G589jcchUvg5JlcHjq6rYZBicJfPZkE44KxfCgiO38OTDTqdTcVYB1689wIdcIhIPFnTc\nMUlM+jyZE2JDZKyAhkN34d0/GMWTUby9SYH02YaRxJB+AjUe1GkXLiJIpc6R00bu1r+x35LEJ58c\nJSNDw2o1Zpb79xs/eTm9R4AhWdKrF0QFD8CxfirT107ni+Nf4PbxziVbIOaFi/BWCEIzmcHrxf7e\nalZu7k3AU4ugxzdg8z3I3VaY+wyu7+8hUphDC3WLceHUr2/Y0ZnNeDw3uXJlEh7PNUIcXRC+7oM3\nWSF1SQpKTsmDSQwQabip4a+Ku91ubHE6nSxatIjU1FQ6depEp06d/tf7Kg9t2rRh+PDhDBky5J9+\n7j/aSn/gwAFq1apFXFwcZrOZfv36sW7dutvZ5R+Kdu3813UcDiMr/3fj/Pmn0TQXoKBpLrzeZFJS\nSrW3bd9u8KSK2tAKCw23Ylf5BFhN1jja7ijXZ18n4+sMEl9J5HQ/XyZlMt2idJSr1SsO3gCqrLNh\nRi5FpvBnzkCdOsbK/9Ch0KCBscAH8ErHV/hhwA+8e8+7bHlsyy8Gb5KTWTp6NJ9euoSi68i6juyI\nI6ThVIjbiXl0HE+3eI6F2n4aDA3EqpXl3LvZIn7NK0fXMmyUG5dbRmi9j6Txn9H/Sxj6CfRZamJX\np0Jik0uarEU8xI6Ihbg4MoJN9OoLTqvRVKRafWWj4GAjytWta1AvyrZox1eBiWcgXMZrg2vVIDID\nLB4we3Xu//4qB3fch7fNeO5vugRRKCMWBuTmXuDajWFoNEEiE4uvVGFLjmXKR0l0ULcTqadRh/PM\nCj6DVEqCwSxAjBWcPn+NVTV0ZvwMhR4JXTPGWpALb871MHbdNxw7dpgPPhDZsMFgrrpcRuD+peAt\nitC9uyGzGhzVjN29d7O181a60x3J91ehcz+UiFA0i9XYwGbD9UR35vAOtN9VEryBDFsgh5/fQfqa\nUeyO60Wa2o4kZyydj8wiKBgaNsxl795mpKR8Rnb2DyRensKVjGdJejcJ1aUh2ASkEAnRJhI7MvZ3\nUea8cOECc+bMYcGCBWRn+0sIu91uWrZsydixY5k2bRoPPPAACxYsuO3vLIKqqhw+fJi0tDRq165N\nlSpVGDVqFG63+9c3/hdxWyyUpKQkqlQpUdCrXLky+/fv9/vMlFKiwh07dqRjGT3lfyfef98QSjpw\nwCijDBlirG7/u6Eo/jU+TfMgy6UusoKCYi5wMYr8IsuhJObty8N10YXm8WmTODUyN2TiTfViibYY\nNaI33jA6SK1WzIoTr1rCplAQcDTaQ6dmH7BmTSWWLh1Gfq4dGypOJLxegTfegJkzdVJSPiMwZy13\nBVakWvQv6BurKnrHDnwxoBeFpVx+vLqO4KgNOrz3yXsk3Ewo9mccz3hmMAMREbfPOUbTPLhcxmzg\n00/huee8vNKuKxkC6BKAwPI+DpqcyKLhoVBApGofhfBu4dBoBj8fW48ilhL3DwHurgmj3gEEkCSC\nrl3DO3YsroICdF3H7nAgDWpAgVmhiD6hWCAjCr69R0PCzeq7X2ZDGwUlMJNeXb8jRqjPQkZQ/CDw\netE+/5xCTcNMMmtYwlOMLB6GfXkvTm6tQlTtfWRe6cHy+CCGDHiTaqHpuFVoHQGCDhe/gGtZsC4W\nXCosvCzzNzWI7GSZDz5SScuWkWW4kghnzyosWGC0xt68easAZenLaNhQC6JoJiSkDbGxIzl46gXa\nH27PAAYwjGF48bLJEciCsrV3u40TnICCICCVkydhh3wn3zSejNmkoASa6DNnHQn3TUUAktUGFKhm\noqI24XTm43D49md1Q7dNJM1+mDRPJnG2ODpv6Yw52oy9+u1Tbvft20eXLl1QFAVRFHn11Vc5fvw4\nUVGGpdrq1atJTEwsdkJyOp28+OKLPPlkiSdmRkYG/fv3Z8+ePURERLBo0aJy5WDLQ2pqKrIss3r1\nanbv3o3JZKJnz55MmzaNadOmlbvN9u3b2V5kDfUv4LYCeNlaZ3koHcD/0wgONlTOcnKM0lxpzZ3f\nExnrjSxY82rEPh1LzFMxfucqIuJeMjLWomnGE1kU7YSH/61kB23a+O/QbDamo2G3+gwC6LJ+y1xK\nEAQ0r4Yn2UN+w5GYxjciJPlbhKpVSIhvzumBl3E6jQaZvAfX0XP4B1gsTnTdxpMDPmDMjvcwyxYy\nsPAPbyMW/LAduclA7o/OR8QLmEhPX03Llmcwm8ss1F69yuMNrrA1OglUTzEtTtB16gSF4c5LID4l\nvrgMIyDQPrYGcf3i+TknhSWrBZJzSmQNFMVw2gK4KlVBL3XZea0W9j0aRe34LDqNqIq1QxfjjYoV\nSXptLPLB8UZ5OwsjgD87ERwBxVog3rp1mfDjjxx/803y8vIYNGgQ2Xe14blzF/ALX658YqUfCG68\nmrtfTKKHDZJaLsWdWodH+Ip4zrNaf5DDu13IS1cWG0DL6KRxExXjceBB5DChpKZXJzW9OiAwfuzd\n1K9SgGQRydAj0IV8bLjRTOC2G9WZABPcmQ/PTMhH99oozbtWFEhNhatXVeLjjeWXtLSSDkbjNg8H\n4hHF5zmwO5ReW6fQMPFHhJ79qPNCe45zBgEBi++v2WEQH6fY4QiPBw4coII9EOY9w9y4F9j4o4hn\n1WQQbRQVoT639cE6yEX/JTZe5BLDCEcQ9FuWbz5bqvGFPhwzZlS3ysqMlTzQ+gF+D4waNYrCUvV+\nRVF45513mDFjBgB5eXllynVGVq5pGqIvcerZsycHDx5ElmWcTic9evTg+PHj1PonrlBFKDLoGDVq\nFNHR0QCMGTPmnwbwssnt1KlTf9Ox3lYAj42N5XrRnQVcv36dypUr/5Mt/jvwK0J8t4WkzUmc73se\nwW1csZfGXgIBYp+KLf5MQsLH6LpKZuZGJCmQWrXmEBLStmQnFSoYi46DBsGNG9CqFXz22S8uYga3\nCkYKklALVVBBsAoENAjAnejm5H0nQQRNDSW4+XDqT6lPRLSFZvXCeGtgLruOSYwdMR+LtWh65yY8\n8ipC292wvRMV8DCbo/SNXUmXyMxSzwkFVS0gPX0VYVGDuJR9iUhHJBUDK3LMdYUVdVTk1DVQqRPY\nYwCdEFMAy+5ogqiuI3lOcskBhOTA+yOpGlBAVUnnXIbI+k2g+AKQ1Qq1GgczUR1ncLlLQXGJrFlb\nnfWHK9PoqsC2bcbzjtWr2bRsoiHCsQrjAacCf4v2E3JyahreyEhWrFhR/Fqex8MzW3dAVJSxM0VB\nefdtzneIJfCFVIJ9SWLlABVvxUvobgvNbUfYN/8IB7/Gz4nLATxgSuS4EkIEMkcJ5QNqYmTrArGx\nF6hb9wA3LBV5kdk4BQcaIs/yHg+w0a+6M+NVMHqObp2K6zosWgRna95P4TPD0J4zI/z4I/qc96lW\nLYF772nP0cP3sP9gV26cT6WteJhVCRpP1F5D9ndrqHZHNXrt6oPdN/aal+GVKTJTX8xEtQUgHT2I\n7b1ZvPi2zpWxChtPmPDc1RosZUpfgsDiR6w03wsVzxtlloMHu+H1WrHZXAiCyuWzZr5cquL1/QH0\n7duX7OxsLGV59v8LZGb6W6rJskxKSomlWufOnf0SKovFQocOHYqDtyzL7Nu3zy/IC4LAzp07f1MA\nDwsL+7fFwdsK4M2bN+fChQskJiYSExPD8uXLWbZs2e81tv8aeL1epk6dys6dO6lduzZvvPFG8XSs\nNK7mXGXphKW0dZcEY82pkbwgmV1hu1iwYAF2u51JkybRtu2KW7b3Q+PGtzrp/AKkAIlm+5px/unz\nOM85CW4dTO13a3Ow4UHUgpIFotwdueytvJcq46pQY1oNxmwL4EA/BclUhscsaOAwMmARCBU8BCes\nuWXBRNc1rudepeHSKnhVL17Vy7i242hTpQ1mkwU0DxwdCaGNsWLnh74LqRsQgN4ynrTQNNRs39ia\nH0I3KQg+kaIRT2lcuQanTlkgIhzbzNdYGVcTrXQ003UEj4C+LwL3jsqAwNGjxnNu6BANBg7E1EGF\n9fg3CZ6/hHRHg+LMMkAUabpunTE1a9MGnn0Wb34+pmeeQe7QwZDlPXIE9dw5JgfVZLZmRfDl5oKo\nY5F0hGmT8DTZz9o1m1DLLDx1BSo7ujHI1QiP7Bu/oEL7JBiTSFKAwou8wU0qkU0oRVOpt3iBhQxj\nNHPoxA5UFfIKBOjzsKFtk55usJTS07GYBNDggNAS7bFnis2pLff9jXvuNDM68B0E6wl69viYb7Z1\npn+DLRyppBKgQcJZ2JcFV6OuMtYew7uFqcV9N3ddyWOTZSBHTnnxaNBoIUSYYH/MWkznzXhGjfqF\npELgWIcsGkdfIe5aKGlX4/jg1a+p+f5uzniDcDjykKRZUKqUqOs66enpxMbGlrO/fw3du3dn4cKF\nxSUSh8NBj1JepvHx8axfv57hw4eTlZVFx44dWbJkSfH7JpMJi8XiV7MWBKFYZuG3YMiQIcydO5d7\n7rkHk8nEO++8Q/fu3W/72MritgK4yWRi3rx5dOvWDVVVGTp0aDED5f8THn74Yb7//ntcLhf79+9n\n69atnD59moAA/46xJzY8QSMaoaEhlgp3m/I3MWvorGKdlZ07d7J9+3ZatGjxu43RWtlKg3UN/F7z\nptzaqaQrOpffusyorFFcveMqs96ZRVBuG3LydmEpHrLAMU9LzvaD6FRovVum0FbId6nQNRpsEqCD\n6JEZufFDMl0l9fu39r5Fy+SWtD/TjTORZ7hU8RJC9iHCAirSMNKgBAomgZanWnL4zsN4rnnQVQEV\noZiJZ7XCrFki973Vn6hx/UnDStmSrs3jQf2yOvLn8RSlqS6XzuqdZ6jd+iqJCW6CMrgFljfeIHLJ\nZ2RJEgowfOtWerz+ulEi2LwZ9u8n4quvCDKbydqwwW/b4/lXyHeaCS7l5CyYFDjaFPmnJgh8S2lp\nwyBRZGD7Dgw88BEeudTjLz4TXroIvuT1rF4HTZfKBEOBPEJ5Qx9PqJZGgv4z1hefx9Oxi7EOoqrQ\nsiXBg2bRLq8C37ERrU274uAN4BVt7AlvjlkYwSVqoFkFqne7wmU9hqridRwivFIPhhyCFJdAqrsi\nMmlIRdzwwgAsds1PC011iVzUBqMy6hc7a0yCTPhDHxHv/ZGPTePJWNSa90dMZZGnLS5Nw1YNvLNe\nNbRffA88s8VcXG64Xbz55pvk5uayYsUKzGYzU6ZMoVevXn6f6dy5M5cvXy53e0EQePvtt3nxxRfx\neDxYrVbq1q37LwXgl19+mYyMDOLj47HZbPTt2/eXtZVuA7fdSn/vvfdy7733/h5j+Y8iLc2o2RzQ\nYAAAIABJREFUjVev7puC+5Cbm8vmzZuRfQVFWZbJyspi586dtxz3hawLnGtzjo6nO2KVrYiIyBaZ\n5dryW6RqP/jgg981gJeHwEaB5B/JL1XINKDJGgGXAjgdfZqHVjzE+DajMGXv4Y5glRwZvjryCltf\njESRjJ6RoEFJqIk6cy5AgSLRT1IJS4Xq72scfSzbb5o//Jvh2I/bGSOOweP18GGXDzl570m+6f8N\nZqnkxFpjrCx8dyHLTy1H110szq9CmEnGYpFxuRxs/vYx5AgPyZpYLlfKpGnUSrzBz1SnVp1jjB8/\niPDwFE6faUX3l2rjekDD5AaO+W/nTUlFz8hAj4pCUhS+vOMOng8JoWpamrHIu3YtQkYG06dPZ+TI\nkX5ULkmQ8GQEQIQXNAFUE3w2GLwBBAIJJHCe88jIhgtOgIP6C0ajN8sDSmVvDQpAVCi6/TTBVE4r\npQEPZj7KbEnYCTOert1L7lhJAmsQEW2WMHDLeb5jo6G9qih+HarZhPE1vdB8G56gEd+J97CAJ6nK\ndRQdEgIh5XITOmmZWEs19iQHB7Lv8GvYmu6lg7iNQK+Lc4dacPhibyaRy+SfL6DVjS/5Pl1HklVq\nmM7TSfwBwaIiAe4nkjlkknD5ShJuMKhNo5+D998FdMRHRAqUAkJ/o3HwP4PFYmHx4sUsvg1+8FNP\nPUX9+vXZtWsXFStWZMCAAZhLB4ZfgclkYv78+cyfP/9/PYbfgj9VJ2Z50HV44QWoUsVoUqxRA0o/\nmP8Vjmar2FYkV0pm5PCRbGy6kR8b/8j1OdeRgm71fBTLskz+ANRfXR97jVtX9RVJITncqEG7FBf7\nkk8y87ydnntg0CEzm9s0w2MTUM3gtkFGtRr8reUUBjd+nEE/3EXrx2Ko8kIrzJdiqVjKuzYuLY5u\nh7ohukQoBKts5bmtz3F20FnqRNYxvjtXIW15GqlfprJ+/3pcqgu3BiNWdGX9xmHs3t2DBQunM/fL\nUVBjDbdQ/DB6ZmpkZbHpxFO0jdrOW291plq1nwkKyqFZ0628ft8+5LTGuAKA+8ps3KsXN4OD8QgC\nHrOZtNBQeowdW/K+KILXS79+/QgPDy9+WUIitppCRHwWSDq6LsLujtTr+RqhdxlB53Vepy1tiSSS\nulUDePddhdS0AXzxZXWqVC2Sz9NplV2IVSlzTbgko9RT9rrSFNpty8d+zHnrubBKXBmbxFNLg2ga\n1wfL6g2Ql2fMJhQFSZfRBaE4eAMgiLixsYq/+44LstwCDfYM4XE9ufgbztQ1qJoLmrRmrv4sj7uW\nkff+JDKnTuEQEVziQZZOqUdcooio6KDKND22lU+/mMk86WVMpbIGzRSIWNbxQhDgvnvh9X7wIqhV\nVLYnbr/lt/5Pon379kycOJGhQ4f+LrX5PwJ/+gC+cSN8+KHR71JQAMnJRpt9EUJDQ+nWrVvxyrLZ\nbCY0NJQO5RDI37//fRpGNyQlJoV5vedxadIlBj0xiPHjx+MoRXlxOBw89dRTf/ix2arYaHmuJQ22\nNEAKlpBCJDwWDz8l/MRPCT8BRlZ5R4U7iHJEYZEsIJXoWhRBB5IjuvFJz0+oduZuDvIJZ5jEIT5m\nxccP4vCJhff5qQ9WxX9RSzAJyOnG7MWb4uVA3QOcG3aO80+c59P3PiU6x5g25zX8lIV513jlu7as\nd12BES1Bysd6Yzk2RNDBpAuEIsOVhZy4MpTaYxKJnTIUvRQtxWyWqV3rGPZc32JTBYqlAABjilWq\nzIDJxIkiKqzVCo0aQUwMoaGh7Nmzh6ZduhF41z3U6diRt2aJxeq3gkVB6rqfCn0r0PDbhoh2kUAC\neYVXWHnfYOZ9qFCtuhtNzSM4MIcJL/cFQaOneJkJu1Opnihgcxm8cosKpoU1DQ0Uwb/OT2Yun3y+\nhbNnLtI4bTemopVdQ/wRRMiupHFi3hAahd6DbfILRO39gnuVLWgaaNyaPOiIFGg2nAocyYagfWNp\nV1gDCZ2kGPj8UZg81WDAyFZwWyUyrXZmR1Uj3lYApkIWUZ3hWe1wPdUYffE8+PwejiyfxrhLW8nJ\n9Xekj+MqUeQi6GUKYSYL2P4GiUYXsKkcaeEi7NplmNU//7zhjfoXDPzp5WRnzIDJk0v0hsEoMZaq\neODxeJg6dSo7duygdu3azJo1iwq/IFKl6zrJ+clYJAtRASULnatWreLDDz/EZrMxceLEfy5Ve7tI\nTzcI79WrF2uxyFkyBccK2J67jkSGUjdYI0+GeVcC+OKRswRaAnl99+sk5l5lY+RwCsvc+AGiSFZC\na/ZV3YNWqrQu4GHcSy/iTTYz/Yvp2BX/jN8caebOpDsRLSI/D/uZlCUpxWViTdTYU2cPL/d5GZNo\nIsIewYzOM3hm0zMIgoBFsvCEcJC3dwUgP3kRbKoR1G58AVc/AaBVGEyuFYDDUZo2ZuKeVQ1QKx3F\nKltR3lZQXb4fuEcPw/ariE8vy7B/P9rnnyPceaehIRJsdMRuzc6mx8mTSLqO7PHQwrSXKeKU4seb\nyRRKdN0DZLoyqbSrEqeGnmKlvJLcOrtp/dglQlvFc5AWBFDInQV76P+jmQ9XTCY+OR7ZBDs6wM8h\nN9lu30VWr67o4aH+HZ0eD0yYAEeOYDZDi1YiCa/2Z6vehRtCNdRSwd6uO3lHeZqC/HT+3nE/bVau\n4mZC+V1qguqmefIEGmpHeSgWFHcwFpNM4nfDGN3lQbwWo/GpbMIvbdiC4+33yX+gKZyYami+tHkL\nXOthE8YMQgJLEHz5iY3AAFAUCx5nBGJIHk9a5pKqVyx5SGkaHD6MMG4q1aZFcWbsGexm3++SmWk0\nahw4wMbg/vS5/hZOt4goGlJDeXn//bHlt+AvOdnbRK1aRkJWWtSqalX/z1it1mIO6a9BEARig29d\nSX/ooYd46KGHbmeovw1Ll8KIEcX0NxYtgj59MIebCbs7jGpHPiI0z7g3bRK8Wk8l0uLBbq/CG13f\nAGDWtWu8dPmy38JhZasVzw0PmHXwltzZBVaZwkyod7M2Ju3Wy6nR940QfaujnqseP/syURNprjen\nR3wPYoJjmNx+MpWsEfRPeIi0nCSiC0RCWldDXrbHMFEQAASo3BcKfobMnziUA1cLRaqLdiwWFx6P\ng6/2tYaYHZgEE5WiKnFz0E3URarho7hhg8HwadPGCCBpaTT64QeEc+duGXu/M2coLKKSWSwc1Fvy\nk3InCdl7CQ01cdJTnXELGmGRLIheETFIJCsjC/2UzobNbaDZy2gmMyZd5nPzQMQlXXBU2AiAWYGq\nP55nVoMPkWdN958VFEGWDQU2338eOaQxXfiCDsIuRmgfo5bS8lYEE9HmbGqH6Hy3Ko30nHL2pymY\nC68hr/iUw0eP0uNRsFUDAvMAWHJ/RdyCjl70ENH1kmDrcqHu200BBXByBwzpWLLfWZQwfVTw5gs8\n3Pc1woIfpUKFa8x7624ki5PZjOUJfQFOxWrsV5bho4+w6zDZO7kkeOs6dO1qdCPLMi+lDsHpKxZo\nGuTlnbn12P6k+NMH8IceMky1N2wwYp4gwFdf/WfHlLMrh8RXE9HdOjEjY4ju/89X55PykkgrTCNe\nCSFgxAijY7Oo7X7IEEO5KDwcTfOSl7cPoXQ9UveQmfktlSuX8Fufr1yZTZmZHC4oQMSYrX9ety72\nHBmP242Zkixb1CSuhl0lPCcc2SRj9pYEFUsVC4GNSygbYd3CyN2Ti+b0mRHbReJ7x7Ou/zrjZh44\nEFauxK7rVAMKLWHEPbCD88XB2weTGYLrQeZPqDr847yHd1v1pra5OqGRTVk85SEWA4qmYJ9uR62o\nwj3ARkDR4NVXDY631Up9p5NNJ0/eck41XSdT9qdXKrrItE+i4WsAhd6jjuIKNNYROAJkU8wBV0Y+\nByYjiHoFCS926JTD7MuZvE4MFjQ2sBG5Q5tbg7euGxno5Ml+mUWg71RW4xpNM09yNLgRukVBFDT6\nsJxQcnF6glizJhkle7dRDirat9tNpe/e4ub8H8BrDHPaVMM8uUgGJFcPQZdKnWhB8Mnn6oY9/E8/\nGVdO6UZiHfCa8Hsyq4YRc1ZWJaKibhT/dJVJ4n3vUB5f2hkN0ZCMSEpCx07B7AL0V3SDn52ebug5\n+M6/i9Kzumzgrlt+rz8r/vQBXBSNgH38uKH30bjxLzY8/luQtz+PE/ecKA5y+UfyycnJYeb+mRw6\ndIh69eoxd+7cYsrVpK2TmL1nNlaTFUnV+b6yRLPS/ggmk1FOCQ9HEIq0UEtDp6DgiN8rZlHkx8aN\n2Z6TQ66icGdwMJWsVpjwDCs77qPXrlcRfHXnKX2mkG/PZ1fdXXQ50YUWl1tgsVgwC2bqL/c3aq7y\nfBVc51zcXGQ4v0f2jKTipIrM+mkWP3/7Ba0SzzBcUxF9Q7S782gelM75sidJ1wlQUqkeDCluEckc\nRo8284h0+HeEioJoBAQdqAtsw3D+0oCEBIT+/TknCKzJyOCZmBjUQhX3FTeWShbMEWbqOBz87HQW\nz0RkrwbHzxU7on89F6zPgCcIqHAHvNQbNN3ICMq2+Vp0zL2vY5odwSSpBneqLi4TbCy8lGGOWDPT\nmdrvCu+rhaRhQxFkJIvK888b7x8+bCXtg/M0D8un9cvHqRx4mcai0TMgigpnzjSH60Hw2iwY2Nco\ny2xYzc0kI3gXweMxhNyKAngnfRtn5fpopckWgmDI0u7ZY4zNaqVV+1bsFnYb53XHZDTtPMZTzZc0\niBJmvTOyDhcvNsaVWglH7DUEi0w1IYeH075l3fdO3LixYiWeeOq56qE5NaQA6RbxtaF8wmu8jJMA\n4CC30Kr+xPjT18D/2/DzkJ9JWVzSNaahMdI2kkQ9EY/Hg9lspkqVKpw+fZpDaYfotrQbTrmkYB+T\nD0lvldqh3W70oUdEALBzp724hb8IVatOpkaNX2ndTUmBuDhea+Vhdmszdm84WYFZyKWagAJMASR3\nTkbOlAlqHmTosJQDXdHRdR1d0mm/uD1Hbx7FpbhweOHBs/D51/B1rWhOWnsQIvThH7NNeIvWRnUI\n0twsF3sh6homQadqzQ+oXXVYud/1zKZnWHRsEU7ZiVQoIHyuo1a/E33y5OLsVJRlPrfFU+WBa6CD\n7tWpMbsG3qERdD5+nBSvF49XRVo4D2XFmpLjDQDvgyA3bQx3zCzOuPF4ID0DKlYCUympVAWqXNN4\nf6jEBBpxnOsQfh98Os/YmSgiaDINvv+amXN6IHpFtrKVAgpofKcL8/ADrF2bzoYNeWiaiAWZYQEK\njy0UcFf0+XSqJhYvnsIXX0zEIu3lpQmbkRxmpue+hucHGQ75n5/KleCLJRJYzMQnLGZiVl0+Tcuk\n9JTHpCjoQ4YgpKTQpl0bLnW7RIaSgUeRYZoLVA14FvgGCKJXRGV2MJOc/ProOsQEZPDu8MGsPHqC\n4yc1KgtNaJbeihvqdSpSkXu5F0clB22SS0lIPPec4aVWWIhus/N61Ft8YnkSTdvLzZvdcLsL/l/E\nlr9q4P/fUGbh6CY3ueq5ikf3aXnLMunp6Rw9epQzlltrgTeDQA6wYZYsxhR08eLi4A0QGfkg6emr\n0H0+kKJoJzKyLNeuHKSkgNXK+N0ejlSS2Vg7FdnX0IMADrOD51s/T3Dr4F8/RJOAgMDe63s5kXrC\nKEEATgusqA9dzlYm8soHtPHakHSJca/De8+Byw71LnmZXu9R7HiKz9XNxGepHvMwJlNJY4mu63x5\n8kt0bzbjoixYVC8FqWEcLShgU+/efmULzWxm9rErvJ1XMsbL4y7T7O4wLrdqRWKuTP24DBRnSfAG\n8HgFlGAdqj1aErzBYLPExBgnxnd+ADQTJFcWcYfpTM4+zRPmOjyf/Rrrhn3I9fsbEV+7Mc/1q8tQ\n0zLy7B2JUCK4T7sPt9lF4L4DPLB3HLk0oiiN9gKLCqH7dhHbwwKYFEwmhUcfncHN69Wx7o5gyrFj\nsG4d4YfCeSFjNJ5jSnG1I8Bs5rkx02nRri+6HEZ64jGyp0+larfBJFd2oFgkbIJAh6go1l64gKIo\nfHTyIyb8OMHQhNdMoIkYegUfGfukgOMNRlHQ9i70iw+AK5zkajt5ZN1F9Cs6igyXTDs5HXiWRa7F\nBFgCEG2GhKwf5swxOk737kWoWZOXnnycl6wCmtaaLl1asG3btl+9zv4M+CuA/0FwXXJxdeZV1FyV\nCo9UIKr3ra335SH26VjSlqcVl1DMNrMxY/RzHdcxmUzF3OrSiA6siPnycaNsUqPGLY5ACQkLAb2U\nDsu7BAf/ug+oXrMme+rW5VpAADP3XODj9de5Gm1l3ht/J0XOpledXgxvOtxvm5ycHAYPHsyuXbuo\nUKECixYt8mPfuBQXouDPZJV0SFcGUNlrRzIkB+m0XafTDpBsElWWObkhqX6sIcHpwdMyDlOFVkat\nNjqaMd+N4YtjHzG3kZOwGmCRQIxLx1HbyhZNu6Wzs4zzGoJJwHnGSUC9AOJCLARYKuGU30PXn8dk\nAlkWUO+IhKgUEMpp8CheCPR/WRPA5gYbHhaZVhGo1KZNxj9gCYgOgfh3ZzGnuodnH3+CR3c/RoXc\nSrTIPkLb9LVs5G26ohYVKgAjdCbGqtQpdSfb7U6ad/qMJw7sg60KHD3K0y2eptMHW/iqyibW3tAQ\nZOjh0Flx/jLjI0LRXA4eMwUznBE8sdbKF49onK8toyac5pu7ni02Yshx5+At0k+XFKj1LcLlTgiq\n4SsvWE3caLoN2SxD3a8BCNECKLysF8vbKopGjjeVIX2G0KlRJ94c/CaBkaXaW8Eo3fTvb/wrfVpF\nkW+//fa/lpf978ZfAfwPgPuqm0NND6Hmq6BD5qZM5DkyMcP93Th0Hy9WKBXEgpoF0XhrY67OvIrm\n1qj/RH06ftSRHTt24HK5sNls1KlThyZNmmAymXi6xdPMPTAXq0/xb12/dYYY1i/QHCXJQb16X/5L\nx3Pu3DmeOHeO/dOnY/Z4UEWRBQsWMGDMGD5t3/4Xt+vZsyf79u3D6/WSlZVFly5dOH36NNWqVQOg\nRUwL7CY7Bd4CNF3DIpqpZa9AEFHFwRsMtcLrMdd5/+X3qeYK4EnNP9rquob15xw4tRU6d8Z1eD/z\nDszjvmiFELMRvAE0G7Sb7KHr+rN8W3qBT3XTd6WZIvnYfezjbOEZWn7pYESFRzC3b8+nX6XT4/4B\nILZD8V6AZt/D/R8j6mBJXI27QQJI5TA/BIOurQE2F3T/BhwuDatwHasrGrlUK4bm1ClwhzEw3UP3\nEx5uhH5AXKMOBP28FbZsoemCBdg3fYtLKXl66UCCv6IDXg3Sqn7LojvhtSMhRnMPUG/zISanQcz9\n3dkfX53ZFTNxT5rlK+LDw0o+dl9//+OLRTySzNa+NzH/vWSM99a6l7f2voVTdiLoAqOCvqeH6gAE\nToRE0HxDde7+8YbfeBqLVn7S/b0LBaBASmGlcyUHlx3k7NNn/ykPvDT+lY7I/wQCAwP9xLJcLhcj\nR47kvffe+92/668a+B+AK1OucHXaVb+1FmsVK3deM3y5dF3jwoVR3LxpTDujox8jIWEhglD+Bez1\nepk1axb79++nYcOGTJw40a8xKDEnkdSCVOpG1SXY+usljH8Fa9au4ZGpr+GZMcNPi9wqCOTddReW\nX+goLXAWEBwU7Oe7GRgYyPz58xk4cGDxaxvOb+DxdY+T782nUWAtNsxI5JD1LkgZhc3hgefmIMdd\nZObiXLbtLwAR6j0g8f7TJkRNRPe4qDcFIotk6O12sk8eIPrLpvSOkRlWHcwiJBHDEZrikF1UTm7E\nc1k78FS6HyQBktZw5xEHr6+ZxRJlCV96vsSNGwfQVBTZPmwY6vtziXitGgUpFSEw1Qh61+7CZMlm\n7qV1jBl0F564vmjWcARzKLpkZIgORPpGR2E64Sb2o1w6bvVgIYfGjOEsE8ijDkUPDlGSqaW+Swwb\nS05kQgIcPQqnTnHzO4FNU44zQZlAOumECyGsk7w0rOHkp/kGLVvXIUeGEYeh3wH4eE+E0fkSEoLe\nuDE9+vRha5MmOO12kL2wNwRe0YGqbOQSjlIXrSzKJD2ZxOD5g/1+25WnVzL6ywG0P3gPj299ArNq\n941fIXZoOO8lt+LjhgpOi+FetGD/kyxyLWTfIRWPx1ivjYwE+UnIBAItgRwcfrDcGeUv4f9KbCks\nLKRixYps3ryZdu3a3fL+XzXw/0LoXp2yc3RdLvkxrl9/h5SUxei6MadMS1uB1RpH9eqvlLs/i8XC\npEmTjLvzq68Mg4b4eMMux2wmLjSOuNC48seiQ0aG0ZtitZb7kXKR4cyg+5fd2TdqH7S4y7/TyYcc\nRaHCL0xlR24eiS74X4A6OoGBJVPlU2mn6LeqH4WykZ2dzDjN5oo6A05s5vtoG9rsfQjR6SxZorBn\nP8XsjzNrVRY3fIi5He7B1u8ZTGmlsjtFISw8lmYxzTiaewhVVzhLA8bxBrpusI6Cq4mINxfDuVeL\nNzsYB42rT6Dz2f0oviKxEzimaWz97DO6DhvG2oFL6fJ5F7hwD6xYBegogs4M2ym+ntOe3o9uR0Wk\nGYs42aoKkibw6Fcaz//NTK0hdVE75aHE3YHFlYSARl1mcIS5aEERqLLAUTWCbupqEjjLGh6khu0m\ntGgBtWqh5zs5l7+SmtRkOctRUTFJEjWUWQSd38wPn8KldpAvwE8ZIHrgATkOdmwoFp06u2ABW3Ny\ncBbNPMwWaJkDMc9B8lXe4B+8RGdsaGhoYIGHRj+Erut4dB2b72H9cP2HefjHtzixvyVZpSh+mmoi\n69ss5mTbaXk1n91VoVYW1Dzcjkkff82XNdM4dlKnciWRB/9WgSE5xmK9oik4zH+QOP8vwKmqLEpJ\nIdXrpVNoKJ3+IOrZqlWriI6OLjd4/x7407fS/xGo0K8Cot3fqLXS8BLT4KysTWhaCXNE05xkZW3+\n9R2PHAnDh8M77xgCLvfe+8vWKxj+n7VrGzovQUHGZr8VfVf15VDSISNoXrzoR3ND1wk3m4k0mw2e\n8scfw+zZxfK3qqay7Mwy6ESJ2a8JwmPCuf/++4t3s+DQAj8GjdOkM91H8W0d/DXm8JsIZoVdu4r9\nmA3IcHzHeQLbDMBUt3kJZS8gwGhiCgtj86ObaVS5F29fjuAN7UXc2PEIdlySnRzBilbxPoqc0Oxe\nePQkqBcPI4iCcZ43bID16/E+8gg5ggBJSXSu0ZkqwVVg7SKQAwxjZm8Qad5m9Gs8AJcZ6iY1YNKs\nSL7pIbC2Fzz8FVwacYm1p9ciVQjGKmUj+J7udpJpbXmcqo/e4FmxKaPlJrixcpIG3M021Lu7GsI8\nKSlk5cdTlKmrqBznODuVnSRJRgCdthICfoDdyWCWApl2/9v02nzFMALxobBePaSyjk6aaqTJuNnJ\nLC6xF8lSSF5kHi8/9DKRW/+OdcdWAnbu5I4DB7hWJLHauzc2KatYWhcAQccaa0FQNR49CR9shBf2\nQqCchun5DxgY25m3RyTwXO3ufLfNmI06zA56JPSgakiZ7rnbxAWnkzk3brAgOZnsMnx+t6bR8vBh\nxl66xLSrV3ng5EkWJCf/wp5uD0uWLPGbcf7e+CsD/wMQ2DCQRt814tK4S6h5KtGPRlNlbIn1nNVa\nBeNmLMpqRazWX9FBzsgwfMW8vgUkp9Nwrj140DB8KAc9exqm9kUxftIkaNkS2rYt9+N+2Ht9r5GJ\nRgOpN2H6dJg4ESSJAEVhfYMGiE6noQB2/brBeJk8GVasQLjfx2ppB1QAroIl1MIrL7yAVZaLpwKq\npqKXWeVTfc89QQHd99/BZatCIjSu0dhQ5PvuO+MBcuGCcXD9+gEQagtlZZ+V8D/tnXd8FGX+x98z\nW7K76QQSiKH3ACkQ4ODACyLiiaKCBTkExXboYcUT9CdnuaBiO+QUsQACehZAsYHSQrEhJDQpAQ2Q\nBAIkkJ7Nlnl+f0zYZEkCSSjr4vN+vfJ6JbMzz3x3svuZZ77PtwDNNmzwahBZIeD2XhPJWf8RhwOc\nXJUBz64Bo2Yn4u67yb36ao+7yDFmDDmFhXqCALDg+gUkP9rEyxzNbaKgTN8WURxR48lD1VTu/vBu\nLmu2l5AS7zohBsrZrVk5aKp6MtEwcCwghuzrJ9J68i2gabgIBty4gElMIoMMFBSmu+2sAZJcsPBT\nWPiVmc/n5bJ+SSAzN+n3+5MT7h5BQYSazZRVVOifPLdbr2B48KBui9FNmaLgctgIshdxR/8wJnZ6\nEGfljWNXWRnDtm9ne+/e8OijtPltCnlvFeLGijCZUQMtdJibCNNH6U+KigJuN13+3Yy0V6Jwvz4V\n4RaE9Q+h37/WE3yiGUktkrij5x2cS34sKuLyLVtwCYGqKDyzfz9bk5JoVvm0uPjYMfbb7Z7qiGWa\nxqR9+/h7tY7xeU4nt/zyC98XFRFhMjG3SxcGN3CWfuDAAdatW8fcuXPP3Zs7BSng54nQP4fSc0PP\nWl9r1y6F48eX4Xbrj/6qGkD79i+dfsDSUn0W7KiWiaGqeiJIHezY4T1Bd7t1va+PgEfYIsguyobR\nwEL0hgfDhoHNhtPpZHynTvw8fjwBBw9WZX06nTBhAmpWFsM7DWfJ7iXQCegEIQ6NEdc9CmWT9E5D\nb73FHT3vYP62+Z5ZuM0BD/2ov6/AbIXg8miKbXncd185Dz2kDy9UldDgUJ6d+qx+TrNZnzHXgRCC\nnlsV1nQEZ6W3x+KAm1q058q8v8Ly5VXXNDCQpiNGkFvd12S1suqBB+ilZDFlzhiKHcW0jP2MQ7ta\n4a6sKGg0grPVBgD2RO/xWoDV0MgLycNuLiXrpal4pzYBJhNNrkjC9b735ooKN0F3jwaRD6pKqLYD\nBSffsJo97PF0jwe4VVHYFRCAUFSeT1rG6+MgxymwGp3Mn6Pw/UYTJhNYVJUNiYnctmsXmwvzKd62\nDaY957m5qS64hBgUFIyTXmFPx1CUajcjDdhZWopT0zCpKubZL9Bnuov8r/IRLkGTK5sy2Rz9AAAg\nAElEQVRgjjTD22/rWbUHD0JiIpZu3eg7wU3ptlJUm4q5swNr5jb6W7YTHGxCaGVgOCUK5SyYmJFR\nVQJBCFxOJ69mZzOtnV6PvsjlqhGFZNc0tErBB7h2+3Z+Li7GKQRlFRUM376drb1706GWnrR1sWDB\nAgYOHOhZtD8fSBeKDwgIuIQ+fXbRqdMbdOr0On367MZqbXv6g2JidF/ISVeGoui5/0lJdR5yatMg\nk6lmnZe6mHftPGwmG4ERgQT+I1APG9A0KCnBUVHB/v37WbVp0ym+DaCggNySXJbvXlsVQifA4HAR\nXOLUxeJ//4PZs0mKTuKbUV9x2X6Fvlnw2jKY8DNgsaB8u5L4azNoFXgXf27Wla+eGcC0yVN45cVX\n2L1rd92dWxwOuP9+PQ67c2fK3/mGh/7lIjEdVLfeT/fuOTAw16LXiUlI0C+M0QgTJ9L8lKYCKmBo\nauPK96/ku6zv2HZkG/lXD6JZ+yxUVWCzCf77X4V2zXaAgJyIHKZdP41yUzku1UVuWC6P3vooblcF\nLfNOabChKLB6NUkjWjHgsmIwleoXC4FbFfSyfUk5FtA0LEGlxJv/xVFDppd4A+RYLLinvUx6569J\nXK9xr/MNrmcqDlc6u7dW8HJKObGx0KIF/HuihWVdE0mLa0PA1sehME9vLGGE2wNvJ5rKWWj8ViKM\nx7zLLgCBBgNGReHQIb2K57xPjBiuiKL52Oa6eJ98X5deCmPGeFw4BquBkL4h2LoFsGXLQHJz51Nc\n/BOHD7/Dli2DPRFZ54L8ak9bAE4hyK028RkcHu6VbmFWFC4LD/eIt1PT+LGoCGe1RUQFWFdQ0CA7\n5s+fz7hx4xpsf0OQUSj+RG4u3HorbNmiVxp87z04TQek1FS4+mrd06BpcNll8OmnNRve18W+4/tY\nf2A9Ae4AxvUbh6vaFyM4OJg5kydzQ0pKVenGgAAODhrHgH0vkfWrFQxOuOYuGLAJpeu/UM3N6JSV\nzadTp9K5f3/9MfvYMf3GVP1GEBKiZ+G1aQPJyfprqqr7utPT9e11MWGCfl0qnwpKA7qwWZ2FVg4V\nipMdYjuaxc3ob0ZzyaWVN4HCQt3PEBDA1pISBqSnU6FpqIDNYOCGsm95+4dna5zKpobQLqI1625f\ny/G/9OHyfvvICdG/7KO3wmfdbGiGCly4+eBzE9duP6V1XUyMp1vz4yv+j+f+NgqOdeWkrxtDGc+5\nn2Yy0yEtDS04mITrr2f7jh2eIRRFITk5mbfi3yJ7Vg4pFf9mAxsQCNyoKDyNYngYh1sf02qFkSNh\nwQJY8esKxv5vLPk5+SR2TGRm9kzss+16DsLc29DaHORxprGNOIRQEG4TC+Pi6HEikr599XulouhL\nD2lp+ts5E0VFP7N162W43VVPjqpqIykpHZut05kHqPa+69KWB/bu5e3Dhz0uEpuq8n5sLNdVy4lY\ndeIEd+3Zw3Gnk+SwMN7r2pXQysmREALb+vXYqz2+Bqkq73XtyohaWinWxvfff88VV1zBkSNHanTu\nqs/7qLd2ivPIeR5eUg+ysoRYvFiItWuF0LT6H+d2O8SuXePF2rUWsW5dsOjdu40wm80CEIqiiLCw\nMHHkyBEh3n9fiKZNhbBYhBg+XHSPdQnV4BaeKkhNCgTffC1Ys0awZo1QVq0SLRYtEhVPPqmfSNOE\niI4WVQcghM0mxL59QvTv771dVYW4887TG96kidcxGgbxU9MvxDLjMtGe9sKKVQSqgSIyMlJkZmbW\nOsSvZWXixYMHxSsHD4pDdrt4fOXjQn1aFTxFjR/zs2Zx65Jbhbj1VqGZTSLfinCqCGG1iiPXXCY2\ntg0Q+dZT3kNQkBCBgUKsXu0555SVUwSh+73eLggxjjlCGI1CCCE2btwoAgMD9Sl6tZ8nnnhCpPZM\nFW/whrBgOeV1szBQWOPyVueDDz4QFotFGI1G0Tq8tfg46mOR2u8Nsforq1j1lVU8v2yA+OcnY8RX\n/5cmhBBi2DAhFKVqPINBiPHj6/e5Kir6WaxbFyTWrMHzs3alUZSmvl+/ASo5nbZUuN1i3M6dwrp2\nrQhZt068cvBgg8YWQog3srOFbe1aYVizRtjWrhW9N20SDre73sffc889YuzYsWfcr673UV/tlDNw\nSa3s2zeJQ4feQNMq09zLrLz1VgKbNh0kJiaGd955h+7du3sd43LpLmmvf3nSYXh2O1iqwg2D7HY2\n9+pFp2bNqDhUwZ5Rmyn9PodAbR+dA/5LwMLX9EXbCRNqdqi57jr9MaIuWraE7GqJJGYzzsnP8cDi\nfby7810clSUEDAYDgwcP5ubnb+bttLcJNAXyVPJTDGhVM9zrtxO/kfBmAqXOUrRaHvW7NevGjlvW\n61Wh9u3TH3cGDtQzYd9803vn8HCYOVNfiKj2JLHz2E7iBu3BvfsqcOs+eJNayn+1h7h7cgQ89xxr\n165l+PDhFBUVeQ1psVgIVUO5y34Xr2qvUkpVWKURE0Z2YqcDkdgZTyYtjBWMeLkpl0y8hJ07d9Kn\nTx9Pyz9FUejYsSO7d+9mz/+tJXfLcii3EBFxDd0WJKJaVHr10mfc1bnySr2l6JkQwsWmTUmUle1G\niArUCgjKgMTJVpTZb+lul3pwIbRlXUEB6wsLaW42c2tUVJ05D2eDjAO/GCgv12O6ly7VH+VfeAHu\nrL0404UiP/9zj3gD2GzlTJvWhtjY7+s8xmjUwxWr64upxKKLaLX9HFYrTcLC0Co00gekY89ygLsZ\nFcZmpEcPok/3SNTE7h7xFsB3reBwhJmk65M57WrB9On6tSsr033b4eGYJo4lf/d9OH6p8oO63W7S\nd6WzYdkGzyLqD9k/sPa2tSRFe68rtAtvx6a7N/HChhf4Pvt7fj3+K05Nf0cm1UT3yO66MG/erCfN\nmEy6OH/wAcyfX+ViMplgwAD4299qmB3bLJZvP3Jw/bAsinNiUDWV0WErufPpBLhP797Uq1cvrFYr\nJSUlaNUe7+12O26Tm1+D9KinkyhAM1yoGHFQwltsIQgXBhf8NrkQ+9Zcfv7zNq+sQSEE+/bto6Ki\ngi4pyXR2/wWhCVRTlXhdcw3s3l31tmw2fVt9UBQjiYnryFw6nJKc9QTv1Gg7D5SKcr3dzkkBFwJy\ncnT/X/NqTSAuIJeGhXFpAzrR+wK5iPl74B//0GeVZWVw/LheiW3FirMe1q25Gz1LMZlO9fUZMZub\ne20RQvDbid/YcXQHTrcuaPPn637WoCD95/J2YYxqHkWgqmJWFAJVlUdbtqSpyUTpL6U485xVpaRd\n4DzmpGzNr/pUHl28x4yAK8fAncOhe/bjfLHni7oNv+UWvU/eAw/oYY/btkHTpgwcONArezUgIAB7\nc7t3HLqzjDnpc2odtlNEJ9699l3S7k6jV3QvAk2BBJuDaRPWhpl/nanvZDDoCVZt2+qCM3q0Xo/d\nZNJvzN266aGgdXBZtwQKMjuQtd/CkaNm5uVfi/qPez3iFRQUxA8//MDAgQNr9FR1Op2E3RLGhy9/\nSJOQJihAB2ANgm30JIX5BOLw9FnSygU5c47T4plnOBWr1UpAZSSOYlC8xBv0cNTRo/V/UUCAHgTU\nkA6BRmMIHff9lcRHVDrMBsPJ5Y+T0UwlJfqNrmNH/Vpec4139JXEg3Sh/B5o3hyOHPHe9uCDDcu8\nqUaZs4xRi0bx9d6vMagGnhj4BFP/MrVBYxQXbyI9PZmKChdut0p5eQiHDm3hvvt0EXdrbm785EaW\n71uOQTUQFRjF+tvX08TahLTtpezZGk7z5gpXXAGKIvgiP5995eXEBwV54mlLd5ayufdmT+Eu0JOe\nkpZGYRueCOXlrGgH198MpdUi+4LNwRROLvSaOZ4JTdO47bbb+PDDD1FVlcTERPJG5LGvbJ9nHwWF\ne3vfy3+v+u9px3JrbrYf3Y5LcxEXFaf3Eq2kxFHC8fLjtAhqwRGXhgJE2+0o5eUQFXXOZpL33HMP\nCxYsoLxS9Gw2G4sWLeKvf/0rAK6778b4zjuep5gcw3XsdU+k+pxNwcVA41XcFBXJN4V6lwZN05g/\nfz4jR448ow1LlugBPyUleoTp22/XLINeJ1u36l2RTk7jLRbdPfa//+l3g7lzqxa2rVaYMgWefLLK\n9otEW6QL5WIgPNxbwM3mmjGADeAfX/+DFb+twC3cuN1uXvjuBbo27cqN3W4888GV/HSsgAlPv05c\nkzxcDhupqTfhckXQsSNccYWeRfnNr994SsHaXXYGzx/MvuP7UBWVlqEtWTH8a44d24jTmcdlYX9h\neNMEr3PYutoIHRhK4bpCtHKNkkiFHbdaye4RxlUzZhB2//1kRQGKd9hcqbMUu8te1YIL0DQ7ubkL\ncDqPEhb2F0JDvX3Zqqoyf/58ZsyYgcPhIDIykgXbFjDhqwlVcegmG39P+vsZr41BNZDQPKHG9pkb\nZzLp20kYjIG4Yp9FCe2GqqgMDA3l8x49sDRQvIUQvLf1PdYfWE/HiI480PcBz3ueMWMGpaWlLFmy\nxNPy76R4c/Qo5R9+yH+E4AAwGBigBHDqA7eKHdXl5BOTidWffcbhw4fp3bs3nTt3PqNtGzfqAVEn\n9XfJEv1e8UF966TFx+uNL+67T48CGjasar3gxx+9o5LKyz0NJU4SHh7eoBv475Xws0zhlwL+e+D1\n1/XHRKezqtLPWXStX/nbSuyuKtErc5bxza/feAS8pETP0IyO9ioV7mFr7lau/fBayr4/yD679w5r\n1ugCnnY4zcv94NJc7Mrb5fn7YMGvrPspnrZBamXNF5UuXeYTGVnVF1RRFHp80YOc13PIyChk3Ijj\nOMzlkJFBUGwsmzMySMpYj/jxTqi8USgotA1re4p4O0hL609Z2R40rQJVDaBDhxlER9dcR6j+hRkb\nP5ZAUyDvpr9LkDmIKQOm6P7sU9CcGlqZRoVxL4WFGzCbI4mIuBpFqUrYSTucxuSVk/VSq23ugsD2\nlXEgGhsKC3lm/35PIsnJ67Umcw3FjmL+3PLPRAXVbJt339f38d7W9yhzlmExWvj4l4/56c6fMBlM\nWCwWFi5c6Nk3Ly+PDRs20KpVK5pt2EDvkhL2o1dC+B9wm2snp96+NQxgMKB07crgwYNrnB/QZ8Sv\nvqp/LqdM8Ti7ly3TG/WcxG7XPVclJXqe1rJlugvttdc8ybE1ufLK2lvMd+4Mv/ziaalGQECNcNnj\nx4/XMegfCyngPkYIwcwdO/ikSxeaORxMGzmSLg895ClA1BiaBzUnqyjL87fZYNZreABr11YtODmd\net38e+7xPv6LjC+ocFVAYB7YAwE7EIbVqieDAMRFxWE1Wj0zcFVR9S47lYkflzYVNDOVe9XAysi4\nx0vAAVSTSssHW/LA1iwKTzhxa7oo2t0uHi8pYd7g0fy3SQUTvtJvaFFBUXz9t6+9xsjL+5Ty8r2e\n+jKaVsa+fQ/QosUdZ5yljYwdycjYut0FB186SObjmYj+G+DxZ1EDDCiqSnBwb+LjV3hEfEvuFpST\n6SEhsV6lZcs1je8LqxpJOtwOkucls/3odk899NRxqSS2SPTsU2QvYvam2ZysXG532cnIz2DdgXUM\nbucttsuWLeOGG27AZDJRUVHBjf37kyPEydpflAFvsYMbrAKl/OT1cBFi+E3/h86eXfub//DDqgVh\n0JV4yRIYOpTwcP1BsbqIBwXpLv+vvtIn0CfX5tu00Xsz1JsZM/QyEceP69P61q3hqacaMMAfB7mI\n6WOmTp3KlClT2JCWxme//ELfV1/lYLUve2N48+o3CTIHEWgKJMgcRExIDA/+6UGcThg+HIqL9R+7\nXV/4zzil6WSgKRCDYoBLLgeCgUhQetGqVb4nOObe3vcysPVAAk2BhJhDCLeEe82KQ01gOOXT5XZ7\nh7+dRAg3GSd24KZqRutC4fujuzhUfIjbE2+neEox2Q9ns/+B/XSK8E74cLkKEMK7WqKmVVCjJGQD\nObHqBPv/tV+vJPnI82CuQBNluN0lFBf/TF7eZ55924ZVi40pOwBa1aKb2e0mtloyxzurX2JLzmZK\nHCUUVRRRVFHEuM+8M/bmb5vPqW0nXJrLc8M8icPh4MYbb6SsrIzCwkLsdjsfbdiAdsqNSygQ/Wg0\niklBCVCwtTMRuyhO73rfsiW18tprVeIN+u//1dcHxo3TXfoWS1WO1cyZ8M033t4Pu10vV3NaduzQ\nF5+vvhoWLdIH3rlTj8r66is9uqdGQRwJyBm4z5k5c6YnBlcIgd1u5+OPP2bSpEmNHrNni57svHcn\nK35bgdVoZXjn4QSaA8nK8qrpBOgBErt368ETJxkbP5ZnZj+DY1dVPLWqbKNdu3HYbF/qxxlMLP/b\ncnYc3UG5q5zukd0ZtWgUqzNXY1ANZJQ6MBkEVLaCUxQToaH9qQ2H4whJ/Mw+WlGBPnMNEOW4jiyh\n2xt3sOmuTbRv0r5G0+KThIUlQ2XStwIoipmQkH5eLo7GUPRjEZpdA0WDQO+GBEK4cDiqepcmt0nm\nbz3+xsLtCzHsfJOSkDgsShhGDaKPHyflpIBv2sT+/zxFeR/vrMyc4hyvv5ftrRlU7dJc9G/pfQ2P\nHj3qFVIIeoSNy2DwRHUEmExcmpxM56c702FyB7RSDWOEsc6nE03TOHHiBOEGQ80ZXmUzhdBQfR1y\n3jwoKNC9IX376ouaxcXVbYEmTU4dpBp79kC/fnqtHyF0H11hoT51T04+zYESkAL+u+RcLM60DG3J\n+MTxXtsiI2sGQTid0KGD97YIWwS3hN7CLOcszzZNc/Hzzz957acoCj2ienj+XjpqKesOrCOvLI8+\nl/QhoOJHMjLuweUqIiSkP926LarVVpMpgrHKBxwSTVnF5QD8RawhM/t/FFXA1DVTeX/k+zUP3LYN\nfvyR0hYteCzqCzaWCQIpY6p1JQ90r722+pnIz4d//lO/qd0YEkBPq4pWCmR0gg57wXhSLFVCQqrE\nVFEUZl8zm7+3u5HDN/6VLq/fTlbLWBQh6JudTcBbevMO7riD/g4ngQlQWhm8YsJAn/II3fcbFATT\np9MssBmqonolDvWN6UsTq7caRkVFYTKZPNEooMe4f/jhh0ybNo1Dhw4xaNAgXn/9dUCvSWKw1n1j\nW716Nddffz12u51As5kvAgL488kptc2mX5xKQkP1aM3qvPGGPpl2OHQXS3Q03HbbaS54ZeNiT8JW\nWRlMm6YLuOTM1CtfsxYmTZokunTpIuLi4sT1118vCgoKGp0O+kfmySefFDabzZOiHhISIg4cOHDe\nzvfll3oqdWionv3+/PO17zdz5kxhtVq90rLj4uJOP3hBgRBPPCHELbcI8c47ntx9rR45/IcPzxdr\n11rFFytN4vOVBnH/QgTzxgtWfi3UVd+Kf2RkCFf1cd5/XwirVQibTQycOVOYVq70pOtb164V6cXF\n9b0kHsrLhRg+fJmYPbuXmDs3Vtxy8ytibliaWBe0TqxtvUSsebujWLNGFWvX2kRubh2p3263EM2b\n1ywN8Ouv+uuRkUKAePwyhPFJhPn/EIkP2cTRCKvX/r+mfipCnwsV5mfMwviMUQRNCxLph9NrPeXq\n1atFcHCwCA4OFhaLRcyaNavB710IIfLy8kRQUJDX/zzEZhPF114rxM03C/Hjj/UaJy1N/1zNmiXE\nGf8NDz3kfa1AiNatG2X/xUR9tbPRceArVqxg8ODBqKrK5MmTAXj++ee99rlYYjXPJ0IIXn/9dRYt\nWkTTpk1JSUmpVxjX2XD0aJXrs666UBUVFVx66aXs2rXL80SQmppKYmJi7QeUl+uV/Q4c0J2gNpu+\nADZjRr3tKivbwwebn+WN9EWkBwyBDvdCZcMCm6oypVUrJrdujUvTsISEQFkZAjCtXInbUDWrDFBV\nXmzXjon1qa5UjTVrNlBRcQUWS3nlW7KxcP4zPH3t7YSoLkL6hWC+xICi1O1+APQIiqFD9UJdRqOe\n3TRyJN999x1Lx48ndN8+7tY0goxQHmIlHAvK8ROewwUqh4e8Sm7MALZXrOBw6EfcYOhKh4ee1dPz\na6G4uJjMzEyio6Np2rR2V9OZ+O677xg2bBiF1dZgQkJCWLduHfHx8Y0a84xs2aKXFTjpaw8MhGee\ngYcfPj/n8xPqq53nJJHn008/ZfHixV5hTQ0xQvL7xOl0snr1akpKShgwYABRUTVD3Tx8/rmeBl3d\nAWo06o/HDeggrgmNx1Y+xqslzXGH9/J6LcpkIt/lQgjB4J9/ZvHUqQTZ7TRZupQT1Ra5AlWVtzt3\n5pbT2VsLqal3Ae94bcvO7sjll2fQvHntx9SJELpzOCQEDAYWL17M2LFjKSsrw6QoRAjBdoOBplOn\n6o7kzEz9MGAn/yLfNADNaUTFThhb6aE8jhISrDuez6K+tMPtYG76XA4WHqRfy35c3elqz2v79+8n\nNjbWyx1jsVg4cOAAkXU0yT4n/PCDnt5ZUqL7W/7+d5+kzv+euKCJPHPmzOGWW26p9bWnqoX/JCcn\nkywXJvwGk8nE0KFD67ez3TvZRgBf/OlP7Dp4kK5hYVwTEVEv376qqLw45EWO7drFwiNHqveFJs/p\n9Py9LiGBiQ88wNwXXuCtl15i7OOPIywWjKpKt8BAbmhEIlRUlJXDhxVUteqLYzYH0MD7gI6i6Ala\nlTzyyCOexWqnEBw3m3n3qad4bMoUaN9ebwVXVkaF0oJ80Q/NqX81NSwUEEepaE1QyUHdZ1xL+nt9\ncGkuBr03iC2Ht1DmKsNmsvFo/0d5KvkpANq0acPkyZN54YUXMBgMuFwu/v3vf59f8QZ9EXPVqvN7\njt85qamppKamNvi4087AhwwZQm5ubo3t06ZN45rKYOKUlBTS0tJYvHhxzcHlDPyPw7Fj+iJcQQEI\nwYRHHmHBFVdQYTYToKqMiYrizQa4hg7Y7fTctIlyTUMIgUsITgmgoU1+Ppk33gjBwWyZP5/1vXrR\n1GSiW34zMveqdOp02nLpCCF4f/v7fLHnC1oEt+DhpBvZv/NKXK5SVFXgdlvp2nUh0dEjGnVJXMUu\njn54FHeJm8RnE8k/ke95TVEUpkyZQkpKir5h+XL44ANKXTGkLR2Ku6zqe2OglHgeJYRd8OijesGu\nRrDi1xWM+HgEJY6qWtxG1UjJlBICjFW1CtLT08nIyKBr167ExcU16lySs+OCuFDmzZvH22+/zapV\nq7BYLDVelwL+B2P3brj3XvaXltI1JQV7tUbIFlXll969adeAllRHHA4WHzuGWwh+Ky/njUOHcFR+\nnhSgf0gIG+LivBouv/KK/jRuMukRNtOm6WVlauPf6/7Ncxueo8xZhlE10tTalM3jP6X42Bzc7lJa\ntLid8PDLG3UpXIUuNiVswnHUgXAJXnG/wgrjCsorqmqXrF69mr6n9DPVnBobu2zEftBeWeTLjZnj\n9GUMBpsR1q+HnrW36jsTi3cuZvzn4ymqqIrHNxvM5D6SS7j1/HRllzSO8y7gy5cv55FHHmHt2rV1\nLppIAb84cZ5wsufOPRT9UISltYXO73YmMLYqUSWtuJhBW7ZQVC0NM8RgYFVCAknBwY06Z5HLRe/N\nmzlUWZXOAGzo2ZPu1RJkDh3SvRHVvTkWi97Y/WQGaXWCpgVR6qyK77YZbfznyv9wV6+76jYkL09P\nXd28WQ+ef+edWvvUHXjhgJ4EVKF//p04eTPsTTaYNxAcHMwrr7zC8OHDaz1FxaEKdo/fTem2UmzW\nY3RRX8TSxKmXGT4LF2RuSS6dZnai2KGvUxhVIz0ie5B2T9oZjpRcaM67D3zixIk4HA6GDBkCQL9+\n/XjjjTcaO5zETxBCsG3oNkq2liAcAkeug/QB6fTd2xdThJ7k0cVmw6yqKG63J7nGrKp0rXepupqE\nGI1sSUpi2fHj2DWNy8LDaX7K4mh2ds30brNZLytdm4C7NG+njIam1zKpjVmz9PK0J/siCqGfsF8/\nPZX1lLZZrjyXR7wBTJh42PQwi49UuRrdmhuX5vJyXwAERAcQv7x61Me1tdvUQJoHNWf1uNWM/XQs\nh0sO0ye6DwtHLDzzgZLfLY0W8L17955LOyR+givf5RFvAAQIt6Dwu0KaDtefxGwGA2sTErjhl1/4\ntbycdlYri7p1I9BwdpmRVoPhtD0JO3bEq/YK6M1xvBKVNE2ve/rTT4wKa8HHgQcoN1bOkt1OFu9a\nTFxUHANbD6w6ZtkymDTJO60c9JOVlOjtaQYO9HqpydAm5LyR4ymVq1gUmvy1KgnnqdSnSFmfghCC\n5DbJfHrzpwQHNO7ppCEkRSex876d5/08kguDrIUiaRCqRfUuMdIxA23AShy2X7z2iw0MZGefPlT8\n5S/s6tOHbqdp7HquCA+Hzz7TkxmtVr070GefgVdTlXHj9BjjuXN567X93PeTILoIVA3cws2a/Wu4\n8v0r+T6rWvnSL76oKd4n0TTdT3OqLZeH02FGB4xhRlSLStOrm9Jpll6vYNHORbz0/Uu4NBdu4WbD\nwQ3c8+U9NcaQSM6ETKWXNAhDkIHoCdEcnnMY7br5MGYhAgP7zC/hPPAkrVtPvuA25RTlMHvzbEoc\nJdzU7SaOH/8TR4/qpQMqS3foHDkCH3/s6e5idsOLK2B9KzhUrVZSmbOMGT/OqKo70qxZ1apodaxW\nva51HYuK0XdGE31ndI3tqzNXe/neK9wVpO5Pbcxbl/zBkQIuaTAdZnTA2q+EfZELPf2wNA0OHHia\n5s3HEhBQU7TOF9lF2cTNiqOoogi3cPPmpjf55MZPGNZpWM2dy8v1tmenIGoJT/eqBHj//fpi5YkT\nVT6a5GS47DI9xKWBrqGWIS0JMARQ4a4q2xcdfOGumeTiQbpQJA1GURRChjkxmANO2W7G4Th8QW15\n/efXPeINUO4q558r/ln7zq1a6Q5xr2k5TPoebNUm11ajlfv73F+1ISJCL3n68st6XOKWLXrd1Mce\n08vtNZCJfSfSLrwdQeYggsxBBJuDeeuatxo8jkQiZ+CSRmGzdQZqhjlZrR0vqJxiJokAAApNSURB\nVB1F9irxPkmJs6T2nVUVVq/Wsx43bdI7HzVvzo0hIRh7J/LaieUYVSOPD3zcexETdAd7I7okCSFq\nZKAGmYNIuyeNrzK+osxZxmVtL+OSkEsaPLZEIpsaSxpNQcE6duy4Fre7DIMhkO7dlxIWNvDMB55D\nUvenMuyDYV59Le/vez/PDX7ugtpxKrm589m7dyKaVkZY2CC6dfsYozHszAdKJFzgYlZna4TEfxFC\n4HIVYDSG+azJ7Ce/fMKUVVMod5UzpscYpg2ehkE9u5DFs6Gw8Hu2br0cTavs46mYadJkCD16fOkz\nmyT+hRRwicRHHDiQQmbmv6BaKS5VDeTSS+tw7Ugkp1Bf7ZSLmJJzSoWrggJ7ga/N8CkmUzNU1Xtx\nU7pPJOcDKeCSc8bTqU8T9FwQkS9G0nN2T46VHvO1ST4hKupWrNb2qGogqmpBVW107vy2r82SXIRI\nF4rknPBlxpfcvOhmz2KiSTUxqM0gvrn1Gx9b5hs0zc6xY4twuQoICxtEYGA3X5sk8SMuaEMHieSH\nrB884g3g1JxsPLTRhxb5FlW1EBU1xtdmSC5ypAtFck5oFdoKm9G72mCLoFpKAEokknOGFHDJOeH2\nxNuJbx7vySwMNgfz3nXv+dosieSiRvrAJecMl+ZideZqiiqK+HPLP9MiWM7AJZLGIOPAJRKJxE+R\nceASiURykSMFXCKRSPwUKeASiUTip0gBl0gkEj9FCrhEIpH4KVLAJRKJxE+RAi6RSCR+ihRwiUQi\n8VOkgEskEomfIgVcIpFI/BQp4BKJROKnSAGXSCQSP0UKuEQikfgpZy3gL7/8Mqqqcvz48XNhj0Qi\nkUjqyVkJeFZWFitWrKB169bnyh6J5OLB7YaCApAllSXnibMS8Icffpjp06efK1skkouHjz6C4GCI\njIQ2bWDPHl9bJLkIaXRT46VLlxITE0NcXNxp93vqqac8vycnJ5OcnNzYU0ok/sGePTB+PJSX639n\nZcHQoZCZCYriW9skv0tSU1NJTU1t8HGn7cgzZMgQcnNza2xPSUlh2rRpfPvtt4SEhNC2bVs2bdpE\nRESE9+CyI4/kj8gHH8Df/w7FxVXbTCY4dgxCQ31nl8RvOK8t1Xbs2MHgwYOx2fQu5NnZ2VxyySVs\n3LiRyMjIBhshkVxUrFsHV10FpaVV26xWXdANBt/ZJfEbLmhPzLZt27J582aaNGnSKCMkkosKIWDc\nOFiyBFRVX8ycOxduusnXlkn8hPpqZ6N94KeeTCKRVKIo8N57cOedkJMDPXtC586+tkpyESK70ksk\nEsnvDNmVXiKRSC5ypIBLJBKJnyIFXCKRSPwUKeASiUTip0gBl0gkEj9FCrhEIpH4KVLAJRKJxE+R\nAi6RSCR+ihRwiUQi8VOkgEskEomfIgVcIpFI/BQp4BKJROKnSAGXSCQSP0UKuEQikfgpUsAlEonE\nT5ECLpFIJH6KFHCJRCLxU6SASyQSiZ8iBVwikUj8FCngEolE4qdIAZdIJBI/RQq4RCKR+ClSwCUS\nicRPkQIukUgkfooUcIlEIvFTpIBLJBKJnyIFXCKRSPwUKeCnITU11dcmnBXSft/iz/b7s+3g//bX\nl7MS8JkzZ9K1a1e6d+/OY489dq5s+t3g7x8Cab9v8Wf7/dl28H/764uxsQeuWbOGzz//nG3btmEy\nmTh27Ni5tEsikUgkZ6DRM/BZs2YxZcoUTCYTAM2aNTtnRkkkEonkzChCCNGYAxMTE7n22mtZvnw5\nFouFl156iaSkJO/BFeWcGCmRSCR/NOojzad1oQwZMoTc3Nwa21NSUnC5XJw4cYIff/yRn3/+mZtu\nuonffvutwQZIJBKJpHGcVsBXrFhR52uzZs1ixIgRAPTu3RtVVcnPzyciIuLcWiiRSCSSWmm0D/y6\n665j9erVAGRkZOBwOKR4SyQSyQWk0T5wp9PJ+PHj2bJlC2azmZdffpnk5ORzbJ5EIpFI6qLRM3CT\nycSCBQvYvn07mzdvPqN4v/zyy6iqyvHjxxt7Sp/w5JNPEh8fT0JCAoMHDyYrK8vXJjWIRx99lK5d\nuxIfH8+IESMoLCz0tUkN4pNPPqFbt24YDAbS0tJ8bU69Wb58OV26dKFjx4688MILvjanQYwfP56o\nqCh69Ojha1MaTFZWFoMGDaJbt250796d1157zdcmNQi73U7fvn1JSEggNjaWKVOmnP4AcQE4ePCg\nGDp0qGjTpo3Iz8+/EKc8ZxQVFXl+f+2118Qdd9zhQ2sazrfffivcbrcQQojHHntMPPbYYz62qGHs\n2rVL7NmzRyQnJ4vNmzf72px64XK5RPv27UVmZqZwOBwiPj5e7Ny509dm1Zt169aJtLQ00b17d1+b\n0mAOHz4s0tPThRBCFBcXi06dOvnVtRdCiNLSUiGEEE6nU/Tt21esX7++zn0vSCr9ww8/zPTp0y/E\nqc45wcHBnt9LSkpo2rSpD61pOEOGDEFV9X9z3759yc7O9rFFDaNLly506tTJ12Y0iI0bN9KhQwfa\ntGmDyWRi1KhRLF261Ndm1ZuBAwcSHh7uazMaRfPmzUlISAAgKCiIrl27cujQIR9b1TBsNhsADocD\nt9tNkyZN6tz3vAv40qVLiYmJIS4u7nyf6rzxxBNP0KpVK9577z0mT57sa3MazZw5c7jqqqt8bcZF\nT05ODi1btvT8HRMTQ05Ojg8t+mOyf/9+0tPT6du3r69NaRCappGQkEBUVBSDBg0iNja2zn0bnUpf\nndPFiz/33HN8++23nm3idxgbXpf906ZN45prriElJYWUlBSef/55HnroIebOnesDK+vmTPaD/r8w\nm82MHj36Qpt3Rupjvz8hE9h8T0lJCTfccAMzZswgKCjI1+Y0CFVV2bJlC4WFhQwdOpTU1NQ61xjP\niYDXFS++Y8cOMjMziY+PByA7O5tevXqxceNGIiMjz8Wpzwmni3evzujRo3+XM9gz2T9v3jy+/vpr\nVq1adYEsahj1vf7+wiWXXOK12J2VlUVMTIwPLfpj4XQ6GTlyJGPGjOG6667ztTmNJjQ0lGHDhrFp\n06Y6Bfy8ulC6d+/OkSNHyMzMJDMzk5iYGNLS0n5X4n0m9u7d6/l96dKlJCYm+tCahrN8+XJefPFF\nli5disVi8bU5Z8Xv8emtNpKSkti7dy/79+/H4XDw0UcfMXz4cF+b9YdACMEdd9xBbGwsDz74oK/N\naTB5eXkUFBQAUF5ezooVK06vORdkWbWStm3b+l0UysiRI0X37t1FfHy8GDFihDhy5IivTWoQHTp0\nEK1atRIJCQkiISFBTJgwwdcmNYglS5aImJgYYbFYRFRUlLjyyit9bVK9+Prrr0WnTp1E+/btxbRp\n03xtToMYNWqUaNGihTCbzSImJkbMmTPH1ybVm/Xr1wtFUUR8fLznM79s2TJfm1Vvtm3bJhITE0V8\nfLzo0aOHmD59+mn3b3Qij0QikUh8i+zII5FIJH6KFHCJRCLxU6SASyQSiZ8iBVwikUj8FCngEolE\n4qdIAZdIJBI/5f8BlSscC5NO1VYAAAAASUVORK5CYII=\n"
      }
     ],
     "prompt_number": 167
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "alphas = np.logspace(-6, 0, 7)\n",
      "l1_ratios = np.arange(0.1, 1.1, 0.1)\n",
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', n_jobs=1, n_iter = 20)\n",
      "params = [{'alpha': alpha, 'l1_ratio': l1_ratio} for alpha in alphas for l1_ratio in l1_ratios]\n",
      "scores = [cross_val_score(sgd.set_params(**param), PCA_X, y, n_jobs=1, cv=make_cv(X.shape[0])) for param in params]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## different alpha values with similar accuracy meaning redunancies in the original feature space\n",
      "sorted_scores = sorted(zip(params, cv_scores), key = lambda (p,s): s, reverse=True)\n",
      "print sorted_scores[:10]\n",
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', n_jobs=-1, n_iter = 20, alpha=0.1, l1_ratio=1)\n",
      "sgd.fit(PCA_X, y)\n",
      "print 'sparse rate:', np.sum(abs(sgd.coef_) > 1e-3, axis = 1) * 1. / X.shape[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[({'alpha': 0.10000000000000001, 'l1_ratio': 0.80000000000000004}, 0.16497635359910809), ({'alpha': 0.01, 'l1_ratio': 0.59999999999999998}, 0.16095736455017892), ({'alpha': 0.10000000000000001, 'l1_ratio': 0.70000000000000007}, 0.15795136453819089), ({'alpha': 0.001, 'l1_ratio': 0.30000000000000004}, 0.15696834559110007), ({'alpha': 0.01, 'l1_ratio': 0.80000000000000004}, 0.15299730868593145), ({'alpha': 9.9999999999999995e-07, 'l1_ratio': 0.10000000000000001}, 0.15198431964899031), ({'alpha': 0.001, 'l1_ratio': 0.40000000000000002}, 0.15098331864798931), ({'alpha': 0.001, 'l1_ratio': 0.10000000000000001}, 0.14999430568292846), ({'alpha': 0.01, 'l1_ratio': 0.5}, 0.14996133858409308), ({'alpha': 1.0, 'l1_ratio': 1.0}, 0.14995234755713796)]\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sparse rate: [ 0.00426667  0.0016      0.0032      0.00266667  0.0032      0.00213333\n",
        "  0.00533333  0.00106667  0.0048    ]\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## based on experiments above, the linear models consistenly give UNDERFITTING results.\n",
      "## try some feature engineering methods, such as tri-kmeans\n",
      "def tri_kmeans(n_clusters, data):\n",
      "    from sklearn import cluster\n",
      "    import numpy as np\n",
      "    kmeans = cluster.KMeans(n_clusters=n_clusters, random_state = 0).fit(data)\n",
      "    dists_to_clusters = kmeans.transform(data)\n",
      "    meandist_per_clusters = np.mean(dists_to_clusters, axis = 0)\n",
      "    return np.apply_along_axis(lambda row: np.maximum(0, meandist_per_clusters-row), \n",
      "                                    1, dists_to_clusters)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 34
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "r = tri_kmeans(9, X[:,:1])\n",
      "r.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 35,
       "text": [
        "(1000, 9)"
       ]
      }
     ],
     "prompt_number": 35
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "batch_size = 2\n",
      "stride = 1\n",
      "n_samples, n_features = X.shape\n",
      "dv['X'] = preprocessing.StandardScaler().fit_transform(X) ## standization for KMeANS\n",
      "batches = [range(start, min(start+batch_size, n_features)) for start in range(n_features)[::stride]]\n",
      "dv.scatter('batches', batches)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 124,
       "text": [
        "<AsyncResult: scatter>"
       ]
      }
     ],
     "prompt_number": 124
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px print len(batches), len(batches[0]), batches[0]\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[stdout:0] 469 2 [0, 1]\n",
        "[stdout:1] 469 2 [469, 470]\n",
        "[stdout:2] 469 2 [938, 939]\n",
        "[stdout:3] 468 2 [1407, 1408]\n"
       ]
      }
     ],
     "prompt_number": 125
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "import numpy as np\n",
      "def tri_kmeans(n_clusters, data):\n",
      "    from sklearn import cluster\n",
      "    import numpy as np\n",
      "    kmeans = cluster.KMeans(n_clusters=n_clusters, random_state = 0).fit(data)\n",
      "    dists_to_clusters = kmeans.transform(data)\n",
      "    meandist_per_clusters = np.mean(dists_to_clusters, axis = 0)\n",
      "    return np.apply_along_axis(lambda row: np.maximum(0, meandist_per_clusters-row), \n",
      "                                    1, dists_to_clusters)\n",
      "kmeans_feats = [tri_kmeans(9, X[:,batch]) for batch in batches]\n",
      "print len(kmeans_feats)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[stdout:0] 469\n",
        "[stdout:1] 469\n",
        "[stdout:2] 469\n",
        "[stdout:3] 468\n"
       ]
      }
     ],
     "prompt_number": 126
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "result = dv.gather('kmeans_feats')\n",
      "result.ready()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 127,
       "text": [
        "False"
       ]
      }
     ],
     "prompt_number": 127
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "result.ready()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 128,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 128
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "kmeans_feats_from_batches = result.get()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 129
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#kmeans_feats = sparse.coo_matrix(np.hstack(kmeans_feats_from_batches+[X]))\n",
      "kmeans_feats = sparse.coo_matrix(np.hstack(kmeans_feats_from_batches))\n",
      "print kmeans_feats.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "(1000, 16875)\n"
       ]
      }
     ],
     "prompt_number": 130
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## FIT a sgd on new features\n",
      "alphas = np.logspace(-6, 0, 7)\n",
      "l1_ratios = np.arange(0.1, 1.1, 0.2)\n",
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', n_jobs=1, n_iter = 20)\n",
      "params = [{'alpha': alpha, 'l1_ratio': l1_ratio} for alpha in alphas for l1_ratio in l1_ratios]\n",
      "scores = [cross_val_score(sgd.set_params(**param), kmeans_feats, y, n_jobs=-1, cv=make_cv(X.shape[0])) for param in params]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 131
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sorted_scores = sorted(zip(params, cv_scores), key = lambda (p,s): s, reverse=True)\n",
      "print sorted_scores[:10]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[({'alpha': 0.0001, 'l1_ratio': 0.90000000000000013}, 0.32699465932998867), ({'alpha': 9.9999999999999995e-07, 'l1_ratio': 0.50000000000000011}, 0.32698866531201859), ({'alpha': 9.9999999999999995e-07, 'l1_ratio': 0.10000000000000001}, 0.32497167826509143), ({'alpha': 0.0001, 'l1_ratio': 0.70000000000000007}, 0.32398266530003056), ({'alpha': 0.001, 'l1_ratio': 0.90000000000000013}, 0.32297567028105956), ({'alpha': 0.10000000000000001, 'l1_ratio': 0.90000000000000013}, 0.31999064933196669), ({'alpha': 0.0001, 'l1_ratio': 0.10000000000000001}, 0.31896567225908545), ({'alpha': 1.0000000000000001e-05, 'l1_ratio': 0.10000000000000001}, 0.31800662938387486), ({'alpha': 0.0001, 'l1_ratio': 0.50000000000000011}, 0.31499163834493177), ({'alpha': 1.0000000000000001e-05, 'l1_ratio': 0.30000000000000004}, 0.31498864133594678)]\n"
       ]
      }
     ],
     "prompt_number": 132
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', \n",
      "                n_jobs=-1, n_iter = 20, alpha=1e-4, l1_ratio=0.9)\n",
      "sgd.fit(kmeans_feats, y)\n",
      "print 'score:', np.mean(cross_val_score(sgd, kmeans_feats, y, n_jobs=-1, cv = make_cv(X.shape[0])))\n",
      "print 'sparse rate:', np.sum(abs(sgd.coef_) > 1e-3, axis = 1) * 1. / kmeans_feats.shape[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "score:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.32699465933\n",
        "sparse rate: [ 0.97522963  0.97528889  0.97131852  0.97078519  0.97511111  0.97108148\n",
        "  0.97125926  0.97214815  0.97137778]\n"
       ]
      }
     ],
     "prompt_number": 134
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## how about supervised features such as decision tree results?\n",
      "batch_size = 2\n",
      "stride = 1\n",
      "n_samples, n_features = X.shape\n",
      "dv['X'] = preprocessing.StandardScaler().fit_transform(X) ## standization for KMeANS\n",
      "dv['y'] = y\n",
      "batches = [range(start, min(start+batch_size, n_features)) for start in range(n_features)[::stride]]\n",
      "dv.scatter('batches', batches)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 136,
       "text": [
        "<AsyncResult: scatter>"
       ]
      }
     ],
     "prompt_number": 136
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "import numpy as np\n",
      "def tree_model(n_clusters, X, y):\n",
      "    from sklearn import tree\n",
      "    import numpy as np\n",
      "    n_samples = X.shape[0]\n",
      "    ## use 1/3 data to train model\n",
      "    index = range(n_samples)\n",
      "    np.random.shuffle(index)\n",
      "    selected = index[:n_samples*1/3]\n",
      "    treemodel = tree.DecisionTreeClassifier(max_depth=5).fit(X[selected, :], y[selected])\n",
      "    probs = treemodel.predict_proba(X)\n",
      "    return probs\n",
      "tree_feats = [tree_model(9, X[:,batch], y) for batch in batches]\n",
      "print len(tree_feats)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[stdout:0] 469\n",
        "[stdout:1] 469\n",
        "[stdout:2] 469\n",
        "[stdout:3] 468\n"
       ]
      }
     ],
     "prompt_number": 160
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "result = dv.gather('tree_feats')\n",
      "result.ready()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 161,
       "text": [
        "False"
       ]
      }
     ],
     "prompt_number": 161
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print result.ready()\n",
      "tree_feats_from_batches = result.get()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "True\n"
       ]
      }
     ],
     "prompt_number": 162
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tree_feats = sparse.coo_matrix(np.hstack(tree_feats_from_batches))\n",
      "print tree_feats.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "(1000, 16875)\n"
       ]
      }
     ],
     "prompt_number": 163
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "alphas = np.logspace(-6, 0, 7)\n",
      "l1_ratios = np.arange(0.1, 1.1, 0.2)\n",
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', n_jobs=1, n_iter = 20)\n",
      "params = [{'alpha': alpha, 'l1_ratio': l1_ratio} for alpha in alphas for l1_ratio in l1_ratios]\n",
      "scores = [cross_val_score(sgd.set_params(**param), tree_feats, y, n_jobs=-1, cv=make_cv(X.shape[0])) for param in params]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 164
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sorted_scores = sorted(zip(params, cv_scores), key = lambda (p,s): s, reverse=True)\n",
      "print sorted_scores[:10]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[({'alpha': 0.001, 'l1_ratio': 0.90000000000000013}, 0.94298789807771843), ({'alpha': 0.001, 'l1_ratio': 0.70000000000000007}, 0.93100885316454185), ({'alpha': 0.01, 'l1_ratio': 0.90000000000000013}, 0.92500884117650584), ({'alpha': 0.001, 'l1_ratio': 0.50000000000000011}, 0.91599683515851182), ({'alpha': 0.001, 'l1_ratio': 0.10000000000000001}, 0.91599084114054163), ({'alpha': 0.001, 'l1_ratio': 0.30000000000000004}, 0.90899282516048974), ({'alpha': 0.0001, 'l1_ratio': 0.10000000000000001}, 0.84600768433103768), ({'alpha': 0.0001, 'l1_ratio': 0.30000000000000004}, 0.84502766239293192), ({'alpha': 0.0001, 'l1_ratio': 0.50000000000000011}, 0.83000365635096174), ({'alpha': 9.9999999999999995e-07, 'l1_ratio': 0.70000000000000007}, 0.82294570019120916)]\n"
       ]
      }
     ],
     "prompt_number": 168
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sgd = linear_model.SGDClassifier(loss = 'hinge', penalty = 'elasticnet', \n",
      "                n_jobs=-1, n_iter = 20, alpha=1e-3, l1_ratio=0.9)\n",
      "sgd.fit(kmeans_feats, y)\n",
      "print 'score:', np.mean(cross_val_score(sgd, tree_feats, y, n_jobs=-1, cv = make_cv(X.shape[0])))\n",
      "print 'sparse rate:', np.sum(abs(sgd.coef_) > 1e-3, axis = 1) * 1. / tree_feats.shape[1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "score:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.942987898078\n",
        "sparse rate: [ 0.81167407  0.82524444  0.80379259  0.79288889  0.78222222  0.77943704\n",
        "  0.77866667  0.77386667  0.76248889]\n"
       ]
      }
     ],
     "prompt_number": 170
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "***As a comparison, try SVM-RBF and RandomForest***"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn import ensemble\n",
      "forest = ensemble.RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=0)\n",
      "scores = cross_val_score(forest, X,y, cv=make_cv(X.shape[0]))\n",
      "print 'cv score:', np.mean(scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "cv score: 0.313981646317\n"
       ]
      }
     ],
     "prompt_number": 174
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn import svm\n",
      "scaled_X = preprocessing.StandardScaler().fit_transform(X)\n",
      "svc = svm.SVC()\n",
      "Cs = np.logspace(-5, -1, 5)\n",
      "gammas = np.logspace(-5, -1, 5)\n",
      "params = [{'C': C, 'gamma': gamma} for C in Cs for gamma in gammas]\n",
      "scores = [cross_val_score(svc.set_params(**param), scaled_X, y, cv=make_cv(scaled_X.shape[0], n_jobs=-1))\n",
      "            for param in params]\n",
      "cv_scores = map(np.mean, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 177
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sorted_scores = sorted(zip(params, cv_scores), key=lambda (p,s):s, reverse=True)\n",
      "print sorted_scores[:10]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[({'C': 1.0000000000000001e-05, 'gamma': 1.0000000000000001e-05}, 0.19799140457823092), ({'C': 1.0000000000000001e-05, 'gamma': 0.0001}, 0.19799140457823092), ({'C': 1.0000000000000001e-05, 'gamma': 0.001}, 0.19799140457823092), ({'C': 1.0000000000000001e-05, 'gamma': 0.01}, 0.19799140457823092), ({'C': 1.0000000000000001e-05, 'gamma': 0.10000000000000001}, 0.19799140457823092), ({'C': 0.0001, 'gamma': 1.0000000000000001e-05}, 0.19799140457823092), ({'C': 0.0001, 'gamma': 0.0001}, 0.19799140457823092), ({'C': 0.0001, 'gamma': 0.001}, 0.19799140457823092), ({'C': 0.0001, 'gamma': 0.01}, 0.19799140457823092), ({'C': 0.0001, 'gamma': 0.10000000000000001}, 0.19799140457823092)]\n"
       ]
      }
     ],
     "prompt_number": 179
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "***It seems that good representations of features go a longer way than fancy classifiers***\n",
      "\n",
      "***Another promising things are ensemble, which will be covered in other notebooks***"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}