{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img img class=\"logo\" src=\"https://raw.githubusercontent.com/albahnsen/CostSensitiveClassification/master/logo.png\" style=\"width: 300px;\" align=\"right\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Example-Dependent Cost-Sensitive Credit Scoring using CostCla\n",
    "<span class=\"right_align\">\n",
    "Last update: May 20, 2015\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "For running this tutorial you need:\n",
    "-    Python version $\\ge$ 2.7\n",
    "-    Numpy version $\\ge$ 1.8.0\n",
    "-    Pandas version $\\ge$ 0.14.0\n",
    "-    Scikit-learn version $\\ge$ 0.16.1\n",
    "-    pyea version $\\ge$ 0.1\n",
    "-    CostCla version $\\ge$ 0.4dev2\n",
    "\n",
    "The easiest way to install CostCla  is with pip:\n",
    "```\n",
    "pip install costcla\n",
    "```\n",
    "\n",
    "see https://github.com/albahnsen/CostSensitiveClassification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  ## Credit Scoring\n",
    "  \n",
    "  In order to mitigate the impact of credit risk and make more objective and accurate decisions, \n",
    "  financial institutions use credit scores to predict and control their losses.\n",
    "  The objective in credit scoring is to classify which potential customers are likely to default a \n",
    "  contracted financial obligation based on the customer's past financial experience, and with that \n",
    "  information decide whether to approve or decline a loan [1]. This tool has \n",
    "  become a standard practice among financial institutions around the world in order to predict \n",
    "  and control their loans portfolios. When constructing credit scores, it is a common practice to \n",
    "  use standard cost-insensitive binary classification algorithms such as logistic regression, \n",
    "  neural networks, discriminant analysis, genetic programing, decision trees, among \n",
    "  others [2,3]. \n",
    "  \n",
    "  Formally, a credit score is a statistical model that allows the estimation of the probability \n",
    "  $\\hat p_i=P(y_i=1|\\mathbf{x}_i)$ of a customer $i$ defaulting a contracted debt. Additionally, \n",
    "since the objective of credit scoring is to estimate a classifier $c_i$ to decide whether or not to grant a \n",
    "loan to a customer $i$, a threshold $t$ is defined such that if $\\hat p_i < t $, then the loan is \n",
    "  granted, i.e., $c_i(t)=0$, and denied otherwise, i.e., $c_i(t)=1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Kaggle Credit Competition\n",
    "\n",
    "Improve on the state of the art in credit scoring by predicting the probability that somebody will experience financial distress in the next two years.\n",
    "\n",
    "https://www.kaggle.com/c/GiveMeSomeCredit\n",
    "\n",
    "### Load dataset and show basic statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['target_names', 'cost_mat', 'name', 'DESCR', 'feature_names', 'data', 'target']\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from costcla.datasets import load_creditscoring1\n",
    "data = load_creditscoring1()\n",
    "\n",
    "# Elements of the data file\n",
    "print data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           Frequency  Percentage\n",
      "Negative (Good Customers)     105299    0.932551\n",
      "Positive (Bad Customers)        7616    0.067449\n"
     ]
    }
   ],
   "source": [
    "# Class label\n",
    "target = pd.DataFrame(pd.Series(data.target).value_counts(), columns=('Frequency',))\n",
    "target['Percentage'] = target['Frequency'] / target['Frequency'].sum()\n",
    "target.index = ['Negative (Good Customers)', 'Positive (Bad Customers)']\n",
    "print target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Full description of the dataset\n",
    "# print data.DESCR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Features</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RevolvingUtilizationOfUnsecuredLines</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NumberOfTime30-59DaysPastDueNotWorse</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DebtRatio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MonthlyIncome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NumberOfOpenCreditLinesAndLoans</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NumberOfTimes90DaysLate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NumberRealEstateLoansOrLines</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NumberOfTime60-89DaysPastDueNotWorse</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>NumberOfDependents</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               Features\n",
       "0  RevolvingUtilizationOfUnsecuredLines\n",
       "1                                   age\n",
       "2  NumberOfTime30-59DaysPastDueNotWorse\n",
       "3                             DebtRatio\n",
       "4                         MonthlyIncome\n",
       "5       NumberOfOpenCreditLinesAndLoans\n",
       "6               NumberOfTimes90DaysLate\n",
       "7          NumberRealEstateLoansOrLines\n",
       "8  NumberOfTime60-89DaysPastDueNotWorse\n",
       "9                    NumberOfDependents"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of features\n",
    "pd.DataFrame(data.feature_names, columns=('Features',))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Credit scoring as a standard classification problem\n",
    "\n",
    "Using three classifiers, a model is learned to classify customers in good and bad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Load classifiers and split dataset in training and testing\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test, cost_mat_train, cost_mat_test = \\\n",
    "train_test_split(data.data, data.target, data.cost_mat)\n",
    "\n",
    "# Fit the classifiers using the training dataset\n",
    "classifiers = {\"RF\": {\"f\": RandomForestClassifier()},\n",
    "               \"DT\": {\"f\": DecisionTreeClassifier()},\n",
    "               \"LR\": {\"f\": LogisticRegression()}}\n",
    "\n",
    "for model in classifiers.keys():\n",
    "    # Fit\n",
    "    classifiers[model][\"f\"].fit(X_train, y_train)\n",
    "    # Predict\n",
    "    classifiers[model][\"c\"] = classifiers[model][\"f\"].predict(X_test)\n",
    "    classifiers[model][\"p\"] = classifiers[model][\"f\"].predict_proba(X_test)\n",
    "    classifiers[model][\"p_train\"] = classifiers[model][\"f\"].predict_proba(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  After the classifier $c_i$ is estimated, there is a need to evaluate its performance. In \n",
    "  practice, many statistical evaluation measures are used to assess the performance of a credit \n",
    "  scoring model. Measures such as the area under the  receiver operating characteristic curve (AUC),\n",
    "  Brier score, Kolmogorov-Smirnoff (K-S) statistic,  $F_1$-Score, and misclassification are among \n",
    "  the most common [4]. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         pre        f1       acc       rec\n",
      "RF  0.492091  0.232172  0.934394  0.151926\n",
      "LR  0.529412  0.037677  0.934854  0.019533\n",
      "DT  0.251239  0.262626  0.899146  0.275095\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the performance\n",
    "from sklearn.metrics import f1_score, precision_score, recall_score, accuracy_score\n",
    "measures = {\"f1\": f1_score, \"pre\": precision_score, \n",
    "            \"rec\": recall_score, \"acc\": accuracy_score}\n",
    "results = pd.DataFrame(columns=measures.keys())\n",
    "\n",
    "# Evaluate each model in classifiers\n",
    "for model in classifiers.keys():\n",
    "    results.loc[model] = [measures[measure](y_test, classifiers[model][\"c\"]) for measure in measures.keys()]\n",
    "\n",
    "print results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAogAAAE7CAYAAABe/3D8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl0pFl95vlvLAqFdoVCW65KKZVXlaq9CipxgVkNDd12\nNQbTrrH74O6m3cYubMbQnJ7jZWz62MY9NBRge0ybGdxmpu0al8HlAuM2m11uwJ21QgHK1JVSUi7K\nTEkhhfY1It754w2FIpRaMlOxSG88n3PyZCjeUMSNqvwpHt373t8LIiIiIiIiIiIiIiIiIiIiIiIi\nIiIiIiIiIiIiIiIiIiIiIiIFE7iZBxlj7olGo9+ORqPJycnJ5zYd+5FoNPpn0Wj030Sj0UOTk5P/\nUJihioiIiEgx+Hd7gDGmGvgY8LfbPOSTwDuAVwNvMcaczt/wRERERKTYdg2IwArwo8DY5gPGmC5g\nylo7aq11gC8Db8rvEEVERESkmHYNiNbapLV2ZZvD7cBE1tfjwKF8DExERERESuNmZhB34mz62rfH\n5xMRERGREgvu8fuv4s4irjsKjO70DWtrCScYvKm9MSIiIiIl5fP5ynLy61YC4g3/gay1F40x9caY\nDtxg+M+An9rpSeLxxVsb4QHS0lLHxMRcqYchUpZUfyKlodrzpl0DojHmVcBngFYgYYx5L/DHwJC1\n9ing54E/Sz/8CWvtYKEGKyIiIiKFV/Rp0/Hx2c3nLXqGfosSKR3Vn0hpeL32Wlvry3KJea+bVERE\nRETEYxQQRURERCSHAqKIiIiI5FBAFBEREZEcCogiIiIikkMBUURERERyKCCKiIiISA4FRBERERHJ\noYAoIiIiIjkUEEVEREQkhwKiiIiIiOQIlnoAXnDlYx9l8XwfFqi+o5ejH/xQqYckUjZUfyIi+acZ\nxD268rGPsnjuB+A44DgsnvsBQx/6ZZYvjpR6aCKep/oTESkMX7FfcHx81in2axaS/dl/7X44bRYI\nED7RufH1Vo9xD2z/5Dsd2vb5drDT99zO8+34fds/344vVcRx7HRop3E4233jjkPP9zhu89/Nzv+o\nbvn5bvt97fjv4+bHkZyZ3vKhvmCQmrvvxVcZwl9Zib8yjK+yMn17/eutj/kqw+7ffv3+LHIzWlrq\nmJiYK/UwCqa1tb7oWWk/0BJzoSSTLI8M59zl893Gv7Gdviffz7fD7ws7f1vxxrHjrzTbPKfvdp/v\ndr6vAO9r+383RXpfPnZ4X768//fdsU42HUrObP0wJ5Fg/qUXdnrxXfmCQXzhMP5QJf5wJb5QJf5w\nGH8olBMk3dtu0PRXhrJuZx/bCKW+YMXt/SwQESkiBcQ9qr6j113iyhKMRDj8vvcT7jhRmkGJlInM\nEnOWYGOE9p9/jMq2dlIrK6RWlnFWVtK3V9K3l0mtrOKsrpBaXia1ukJqeSXr69Wsx62QmJ3FWVnB\nWVvb+6D9/tzgmA6evlAoHUAr8YXT998QMLNnQddDasg9Fgpp1lNE8kZLzHkw9KFfJhGPA2447Pro\n4yUekUj5KGb9OanUppC5khU+l3FWVjO3tzyWDqBOOpBmB9PbPr0iiy8zu7kxo5kJljkzoZXpIBre\nMoi6M6FZs6BBzSXI9rTE7E0KiHmwfHGEq7//Sfx+H+2/8EuaORQpIi/Un+M4OGtr28x2rmwzE+rO\ngm7cvvF7nNUVnERi7wMMBDadu5k+TzMrbGbCZ/ZsZ3jTDOmmmVBfKKTldg9QQPQmBcQ88nqRiOxn\nqr+tOYlE7hL6DUvtWwTLLWdBN82Irq7ufXA+XzpUbnde58Yy/A3ng1ZuPjc095gvENj7+OSmeL32\nyjUgat1ARMTDfMEggWCQQHVNXp/XSaVwVtNL6qsrOJkl89wl9J1nQTcdm58ntbICqdSex+cLBnM2\nDN14/uYWx7LP/wyHc4Lo+jFtMpJyoYAoIiK3zOf3u7N64XBen9dxHJxEYovl9M1BdCV3k9EOs6DJ\nuTlSsYn8bDLy+XLD4w3hc30JPcR2LZRuOB80vVyvTUaynyggiojIvuHz+fBVVEBFBYHa2rw+99ab\njFa33FS0fg5nZiZ08wxp+vuSszPurGe+NhnlzGLmLqFnn/+5VQulG46tz4QWaJORrmLkbToHMY+8\nfh6GyH6m+pNSWd9klN1Cafv2SltvJsq0V9rUdim/m4yyguPmNkk7tVDaYmn+2h/9IUvnz+W8jFdb\nvOkcRBEREbllPp8PXygEoRCBurq8Pndmk9HK6ja71Zc3ncu5VeulrBnShUUS8TjOykpexwmQiMe5\n+vufVKs3j1BAFBER2acKuslobS1zDmdmCX2HtknZx+ZfeD6v45H9RwFRRESkzPj8/szS8e3Y8ipG\n6SVm8QZtmRIREZFbcvSDHyIYiWS+Xr+KkdfOPyxnCogiIiJyyw6/7/0EIxFC0SbNHHqQlphFRETk\nloU7TtD10cfVQcCjNIMoIiIiIjkUEEVEREQkhwKiiIiIiORQQBQRERGRHAqIIiIiIpJDAVFERERE\nciggioiIiEgOBUQRERERyaGAKCIiIiI5FBBFREREJIcCooiIiIjkUEAUERERkRwKiCIiIiKSQwFR\nRERERHIoIIqIiIhIDgVEEREREcmhgCgiIiIiORQQRURERCRHcLcHGGMeB84ADvB+a+3zWcceA34a\nSALPW2t/uVADFREREZHi2HEG0RjzOqDbWvsw8B7gU1nHGoB/D7zGWvvDQK8x5kwhBysiIiIihbfb\nEvMbgb8EsNaeByLGmNr0sZX0nzpjTBCoBiYLNVARERERKY7dAmI7EMv6egI4BGCtXQZ+E7gAjADf\ntNYO5n+IIiIiIlJMt7pJxYd7LiLGmHrg1wADdAKvNsbcnd/hiYiIiEix7bZJ5SruLOK6w8C19O3T\nwJC1dgrAGPNN4BXA93Z6wkikmmAwcHujPQBaWupKPQSRsqX6EykN1Z737BYQvwJ8GPgjY8wDwKi1\ndiF9bAQ4bYwJp5ebXwF8ebcXjMcX9zDc/a2lpY6JiblSD0OkLKn+REpDtedNOwZEa+0/GmNeMMZ8\nC7eVzWPGmJ8BZqy1TxljPgr8nTEmAXzLWvvNIoxZRERERArIV+wXHB+fdYr9msWi36JESkf1J1Ia\nXq+91tb6omel/UBXUhERERGRHAqIIiIiIpJDAVFEREREciggioiIiEgOBUQRERERyaGAKCIiIiI5\nFBBFREREJIcCooiIiIjkUEAUERERkRwKiCIiIiKSQwFRRERERHIoIIqIiIhIDgVEEREREcmhgCgi\nIiIiORQQRURERCSHAqKIiIiI5FBAFBEREZEcCogiIiIikkMBUURERERyKCCKiIiISA4FRBERERHJ\noYAoIiIiIjkUEEVEREQkhwKiiIiIiORQQBQRERGRHAqIIiIiIpJDAVFEREREciggioiIiEgOBUQR\nERERyaGAKCIiIiI5FBBFREREJIcCooiIiIjkUEAUERERkRwKiCIiIiKSQwFRRERERHIoIIqIiIhI\nDgVEEREREcmhgCgiIiIiORQQRURERCSHAqKIiIiI5FBAFBEREZEcCogiIiIikkMBUURERERyKCCK\niIiISA4FRBERERHJoYAoIiIiIjkUEEVEREQkR3C3BxhjHgfOAA7wfmvt81nHjgF/BlQAL1prf75Q\nAxURERGR4thxBtEY8zqg21r7MPAe4FObHvIx4KPW2jNAMh0YRUREROQA222J+Y3AXwJYa88DEWNM\nLYAxxg+8Bvhi+vj7rLWXCzhWERERESmC3QJiOxDL+noCOJS+3QLMAY8bY/6HMeZ3CjA+ERERESmy\nXc9B3MSHey7i+u0jwCeAi8BfG2P+qbX2yzs9QSRSTTAYuOWBHhQtLXWlHoJI2VL9iZSGas97dguI\nV3FnEdcdBq6lb8eAi9baYQBjzNeBO4EdA2I8vnh7Iz0AWlrqmJiYK/UwRMqS6k+kNFR73rTbEvNX\ngJ8AMMY8AIxaaxcArLUJYMgY051+7IPA+UINVEREROQg+tEPPPXVH/3AU6n0n6+Wejw3w7fbA4wx\nHwFeCySBx4AHgBlr7VPGmJPAf8UNmi/fTJub8fFZZ7fHHFT6LUqkdFR/IqXh9dprba3fNSvtJB0I\nf2TT3aPAI1/6+NtfvJ3nNMb8K+CtQC1wDPd0v18BvgRMAp8F/i8ghJvf/u2tbiTe05u+HQqIIlII\nqj+R0vB67e0WEH/0A099FHjXDg/p2Ob+JHBlm2NPfunjb//Qdk+YDogfAO4DIsDLQAL4WWvtV4wx\n/zfwp9barxtj/inwdmvtv9vpfWx2q5tURERERKS0HOAZa20KmDTGxIETwLPp4w8Dxhjza0AAGL/V\nF1BAFBEREblN6Zm+bWf7CrHEnJa9j2S9y8xq+usV4CestWO3++QKiCIiIiIF8qWPv/3NP/qBp67g\ntgYEGP3Sx99+dI9P6wN+KH3RkibccxEns46fBX4c+LQx5o1Am7X2z27lBXbbxSwiIiIie/MI7qzh\naPr2XjnACPAk8HXgV4FU1vHfBN5ujHkG+HXg27f6AppBFBERESmg9FLyXmcNN7tgrc1e2v5/129Y\na6/h7nK+bZpBFBERETl4CtoVRjOIIiIiIgeItfZPCv0amkEUERERkRwKiCIiIiKSQwFRRERERHIo\nIIqIiIhIDm1SERERETlAjDFB4FtAH/AnwJ8D/9pa+9f5eg0FRBE50P7zEy9xbiQOPjjdEeHfP3p/\nqYckIpLjXU+896vAm9Jffv3JRz/95j0+5WEgBPwW8FHgmT0+3w20xCwiB9Z/fuIl+kbiOIDjQN9I\nnA/+wbe4eH2u1EMTEQEy4fBHcC+P5wN+5F1PvPfKu5547wN7eNrHgZPArwDvBOb3PNBNNIMoIiXl\nOA5riRTLa0lWVt0/67eXV5OsrCW2uM+93TcSv+H54nMrfOrzL/Oxx15dgncjIuXmXU+896PAu3Z4\nSMcW9x0Bnn3XE++9ss33PPnko5/+0DbHAD4A/IW19j0AxpibGuutUEAUkZuWTKVYWU25AS0T2BKs\nrKWDW1aQ2/K+1cRG0Mt6nJPn6wEsryaYnFkm2hDO7xOLiOwPvkK/gAKiiAc5jkMimdoytG03G+fe\nl9jivo3HrSVSu7/4LipDAcIVASpDARqqQ1SGAlSmv16/f+N2kPAWx9fv+8yX+ui/NH3DayytJPnQ\nH36bU0cbONPbxivuaKW+OrTnsYuIbJae6dt2ti9riTnbKPDIk49++sU8DiWvv2orIIqUWMpxcsNb\n9uzbmjtDt7qWumGmbrdZu9Qep+UCfp8bxEIB6qoraAmFqawIEA4FM6EuO7xlf73+fRu3g4QrAlRU\n+PH78veL73/4qQf44B98i/jcCgCRuko+/G8e4vn+cZ7tG6P/0jQDV2b4068O0HsiwpneNh4wLVRV\n6kefiBTHk49++s3ppeQj6btGn3z000fz/DLr5zfmjX5KityCLWfltlo2Xb99E+fVra7tfVYuVOHP\nzK5F68OZgHbjjNzm+4JUVvipTIe+7Bm6YOBg7GH7pXfew6c+/zJ+v4/3/fjd1FZV8Pr7jvD6+44Q\nn1vhuXNjnD03xveHp/j+8BR/8t/7ufdklDO9bdxzMkqoIlDqtyAi3vcI8HTW7XxwjDE/DvxH3PD5\nemPMb1prX5mPJy/4GvZm4+OzeT7baP9oaaljYkK7J/eDlOOwtpbaetn0hhm69dm3RM7sXPZj1r8v\nmdrbP1+/z5czu7bVsmn27XAouM2MnPt9oQr3Pr+/6KW87+xWf2PxRZ7tG+PsuXGuxhYACIcC3H+q\nhTO9bfSeiByYUCyyn3j9s6+1tb4sf8AqIOaR14ukUBLJFKtbBLYbl00TW+903WLWbnU1ueeTMUJB\n/5ahrDId2nJm4254zObw535PMODDl8clVtlws/XnOA5XJhY42zfGs+fGiM0sA1BbVcEr7mjlzOlW\nTh1rzOtSuIiXef2zTwGxSBQQDy7HcVhNpLbe9LBLO5LNs3bZs3qJ5N7+Sfh8ZM2+Bbeekdt831Yz\ndZvOo9Os3MFyO/XnOA4Xrs5ytm+M586PM7uwCrjnMj50upUzvW10tNUp1IvswOuffQqIRaKAWBzZ\n7UjWNzdsHdRuoR1JHmblggH/DhsZNs/IBbedkcsOfhVBvz7AZc/1l0ylOH9pmrN9Y7zQP8HSSgKA\ntkgVZ3rbONPbxqFoTb6GK+IZ++mzrxAUEIvEiwFxL5f62q4dydZNg9O7WYvQjsQHW25uCOV8Hdxi\nKXWr8Le+JOsn4Nc5XlIY+fyQWkuk+P7QJGfPjfGdgRir6Xo63lrLmd42Hjrdph6LImkKiN6kgLhH\n65f6ylYVCvBgTwvhUHDb/nP5bEcSDPhy2ols125k+1m7G4NeSLNycsAU6kNqeTXBdwZinO1zd0Kv\nb1TqPtrAmdNtvPKOVupr1GNRypcCojcpIO7Re373G7e07Lp9Q+DdmwRvF/y081KkOB9S80trvNA/\nztl0j0UHd2f66RMRzpx2eyxWh9U9TMqLAqI36SdZgdRVV/DBn7wva1bPT6gioJ2RIgdYbVUFr7vv\nCK9b77F43g2LPxie4gfDU3zub/u5J91j8V71WBSRA0wBcY9On4jcsMQcqavkl955D8fb6ko0KhEp\ntEhdJW955THe8spjjMcXOXvOvXrLi3aCF+0ElaEAD5xqTvdYbNJMv0gZ++Yj7/gq8Kb0l19/zdNf\neHMpx3MztMScB5sv9fWxx15d4hGJlJ/9ssx1ZXyes+fGONu3qcdij9uQWz0WxWv2S+0Vyl6XmNPh\ncMtrMb/m6S/c1rWYjTH1wBNAVfrPLwKNwG8DSeAJa+0njTFv3nzfzb6GAmIeXLw+l3Opr452zRyK\nFNt++5ByHIehdI/FZzf1WHzlHW6PxRPt6rEoB99+q7182y0gfvORd3wUeNcOD+nY5v4kcGWbY0++\n5ukvfGi7JzTGdAN3WWufMsa8Hjcg3g38EBAH/io9ppc33fcT1tqVHcaaoSXmPOhor+Njj73a80Ui\nIjfP5/Nx8kgDJ4808OibTnH+Upz/me6x+JXnLvOV5y7TFqniodNuj8XDzeqxKCI3bQJ4pzHmg0Al\nUAMsWWsn08d/zBjTCixn33crL6AZxDxSQBQpnYNSf2uJFN8fnuRs3xjfGYyxuub2WDyW6bHYSnND\nVYlHKXLzDkrt3a59usT8G0CltfZXjDEPAn8M+Ky1d2c9Jgo8Y62963ZeQzOIIiJFVBH0c/+pFu4/\n1eL2WByM8WzfON8bmuQv/v4Cf/H3F+g+0sCZ3jZecUcrDeqxKHKgvebpL7z5m4+84wpwJH3X6Gue\n/sLRPT5tFHf5GOCdwBwQMcYcBq4BTwP/Eghsuu+nrbWzN/MCCogiIiUSDgV5VW87r+ptv6HH4uDo\nDH/6NUtvR4SHett40LSqx6LIwfUIbkBbv71XnwM+Z4z5SeD3gZ8Efgv4i/Tx/89aO2OM+YVN991U\nOAQtMeeV16fZRfYzL9Vfdo/F4Wvuz/NgwMfdXVFedWe7eizKvuKl2ttKuTbKVkDMI68Xich+5tX6\ny+6xOBpbAFCPRdlXvFp76xQQi0QBUUQKoRzqTz0WZT/yeu0pIBaJAqKIFEI51V92j8Xnzo8zk+6x\n2FgbyrTNUY9FKRav154CYpEoIIpIIZRr/aVSDucvxTmb7rG4uJIAoDVSxRn1WJQi8HrtKSAWiQKi\niBSC6k89FqU0vF57CohFooAoIoWg+su1sprkpcGJTI/FZMr90asei5JvXq89BcQiUUAUkUJQ/W1v\nfmmNF+0EZ/vGOH8xjgP4fGT1WGyhOlxR6mHKAeX12lNALBIFRBEpBNXfzZmeX+G5c+OcPTfG0NXc\nHotnetu4t7uZSvVYlFvg9drLR0D88Aee/irwpvSXX/+Njz/y5r0+Z6EpIOaR14tEZD9T/d268ekl\nnu0b4+y5MUYnNnos3n+qmTOn27izUz0WZXder729BsR0ONzyWsy/8fFHbvdazP8KeCtwGPga8BYg\nBTxlrf24MaYR+G9AHTADPGqtXbiV11BAzCOvF4nIfqb625srE/Oc7cvtsVgTDvKKO1o5c7oNc1w9\nFmVrXq+93QLihz/w9EeBd+3wkI5t7k8CV7Y59uRvfPyRD233hOmA+O+AnwI+a619ozHGB3wTeBT4\nOeC6tfb3jTH/KzBsrf2rnd7HZrqwp4iIcLSllqOvq+Udr+1i6Fq6x+K5cZ75zlWe+c5V9VgU2V8c\n4DngIeCUMebv0vfXAieA+4FfA7DWfuJ2XkAziHnk9d+iRPYz1V/+pVIO/ZfinD3n9lhcWN7osbge\nFo+ox2LZ83rt7dMl5p8B7gK+BbzVWvveTcefBv6jtfb523l+uImAaIx5HDiDm1bfv9WLGWM+ArzK\nWvuG3Z5PAVFECkH1V1iJZIrvD01x9twYLw1MZHosHm2p5Uyvuwzd3Kgei+XI67WXp00qV4Aj6S9H\nf+Pjjxzdy/NlBcRP4Z6DeB+wDDwO/G/Ah4AVa+3/YYz5OWDJWvu5W3mNHZeYjTGvA7qttQ8bY+4A\nPgs8vOkxvcAPA6u38sIiInJwBAN+7jvVzH2nmllZTfKdwRhn+8b43tAkn39miM8/M8TJI/WcOd3G\nK0+3qceiSK5HgKezbueDY629bIz5BPAPuOc0/qW1dtkY80ngc+ml51nccxVvyY6p2BjzYeCitfaz\n6a/PAa+01s5nPeavgf8EfFgziN7+LUpkP1P9lcbC8hov9Kd7LF6K4zhuj8XTHRHOnG7jwR71WPQ6\nr9deufZB3G2TSjvwQtbXE8AhYAAyu2i+AVwsxOBERGR/qwlX8Np7D/Paew8zM7/Cs+fHebZvjL6R\nOH0jcf6fr/Srx6LIAXSru5h9uOciYoxpAv4l8E+AYzf7BJFINcGgd39AtLTUlXoIImVL9VdaLS11\ndHc281Nv6+X65AL/4zuj/MNLo7w0EOOlgRjhUIAzdx7itQ8c4X7TSkVQPRa9QrXnPbsFxKu4s4jr\nDgPX0rffkD72TaASOGmM+Zi19oM7PWE8vnibQ93/vD7NLrKfqf72lwDw+nsO8fp7DjE6Mc/Zc26P\nxWdeusIzL12hJhzkwZ5WzvS20XOsEb+/LFfxPEG15027nYP4Q7jnFr7FGPMA8Alr7Wu3eFwH8F91\nDqKKRKRUVH/7n+M4DF+b42zfGM+eH2Nm3t3b2Fgb4pV3uG1zOg+px+JB4/XaK9dzEG+mzc1HgNfi\n7o55DHgAmLHWPpX1mBOkO3nv9nwKiCJSCKq/gyWVcui/PM3ZvjFe6B/f6LHYWMVD6bY5R1pqSzxK\nuRlerz0FxCJRQBSRQlD9HVzb91is4UxvGw+dbqNFPRb3La/XngJikSggikghqP68YWU1yXcvbPRY\nTCTdj4yTh+s506sei/uR12tPAbFIFBBFpBBUf96zsLzGi/0TnD03xrmLuT0WH0r3WKxRj8WS83rt\nKSAWiQKiiBSC6s/bsnssXrg6C0Aw4OOuTrfH4n3dzVSGvNtCbT/zeu0pIBaJAqKIFILqr3xMTC/x\nbLptzpWJBQAqKwLcf6qZh3rbuKuziWBAPRaLxeu1p4BYJAqIIlIIqr/ylN1jcWJ6GUA9FovM67Wn\ngFgkCogiUgiqv/K2XY/FhtoQD6nHYkF5vfYUEItEAVFECkH1J+u267HY0hjmTG+beizmmddrTwGx\nSBQQRaQQVH+ylUQyxfeHp3i2b4yXBmKsrCUB9VjMJ6/XngJikSggikghqP5kNzv1WHyot42H7mil\nobayxKM8eLxeewqIRaKAKCKFoPqTW7G4vMYLW/RYvON4hDO96rF4K7xeewqIRaKAKCKFoPqT2zUz\nv8Jz58c5e26MC6Nuj8WA38fdXeqxeDO8XnsKiEWigCgihaD6k3yITS+l2+aMc2ViHoBQhZ/7T7Vw\n5nQbd3Wpx+JmXq89BcQiUUAUkUJQ/Um+jcYW3LY5fWOMTy8B6z0W3bDYczyiHot4v/YUEItEAVFE\nCkH1J4XiOA4j19M9Fs+NMZ3VY/GVd7gNubsO1Zdtj0Wv154CYpEoIIpIIaj+pBhSKQd7eZqz58Z4\n/nxuj8WHTrsNuY+WWY9Fr9eeAmKRKCCKSCGo/qTYEskUPxie4uy5MV6yGz0Wj7TUcOZ0Gw/1ttFa\nBj0WvV57CohFooAoIoWg+pNSWllL8t3BG3ssdh2u58zpNl55upVGj/ZY9HrtKSAWiQKiiBSC6k/2\ni8XlNV6wEzzbN0ZfGfRY9HrtKSAWiQKiiBSC6k/2o5mFVZ4/P87ZvjEGR2eAjR6LD/W2cn93y4Hv\nsej12lNALBIFRBEpBNWf7Hex6SWeTYfFy+MbPRbv627mTG8bd3dFD2SPRa/XngJikSggikghqP7k\nIBmNLfBs3xhnz40xHnd7LFZXpnss9rZxxwHqsej12lNALBIFRBEpBNWfHETZPRafOz9OfG4FgIaa\nrB6Lh/d3j0Wv154CYpEoIIpIIaj+5KBLOQ4Dl6czYXG9x2JzQ5gzvW2cOd3G0db912PR67WngFgk\nCogiUgiqP/GSRDJF38gUZ/vGeDG7x2JzDQ/1ug2590uPRa/XngJikSggikghqP7Eq7brsdh5qJ4z\nvW08VOIei16vPQXEIlFAFJFCUP1JOcj0WDw3Tt/IlNtjEbijo3Q9Fr1eewqIRaKAKCKFoPqTcjO7\nsMpz2/VYPN3KfaeaCYeCBR+H12tPAbFIFBBFpBBUf1LOYjNLPHtumx6Lp9u4qytKRTC/PRa/+MR3\nuTISBx8c7YjwY4/em9fn3y8UEItEAVFECkH1J+K6GlvgbIF7LGbCYZaauhBve+fdtLTX7em59xsF\nxCJRQBSRQlD9ieQqZI/FP/zdv9/y/pq6EO9+7OG9DHvfKdeAWPiTE0RERKTofD4fnYfq6TxUz794\nY3emx+Lz/RN87YUrfO2FKzfdY9FxHGJj8wz1TzBsY0V8F1IqmkHMI81giJSO6k/k5uT0WByIsbK6\nqcfi6VZaI9WkUg7XR2cY7o8xZCeYn3VnIANBPxUVAZaX1nKeV0vM3qKAmEf6gBIpHdWfyK1bWUvy\n8oVJzvY+IQljAAAgAElEQVSN8fKFSZLJFHVAR1WImoRDMt2gO1QZoKM7Spdp4VhnE5/+wWdxvnGU\nijW3WfdaxRJjr3qRn7vnZzhed7SE7yj/yjUg5ndLk4iIiBwYlRUB7uuK8rbedt7R3cJDwSA9+Akv\nJVheSzCBw2JLNSdf18lDbzpJxeFVvnH1Gc7HB7hoXmCtYom1iiUumheYXpnhv7z8J6V+S5InOgdR\nRESkzKwsrzEyMMmwjXFpeIpkIgVAbX0lXaaFtuONjMwu8Y/2MufnLXbw2/z5RAxfxWrmOZZrZum/\n/+9K9RakwBQQRUREysDC/AojAzGG+mNcvTRNKuWe8RWJVtPZ00yXaSHSUsWl+VH6Jl+kj37G2q8Q\nwn2cL1nJWuwwqelmAq2XCdTntrlhLcwjne8q9tuSAlFAFBER8ajZ6SWG+mMM2wmuj85m7m9pr6PT\nNNNlmvHVJTg3aXk69m3O2wEWE27vRL/Pz8nGE/Q29dAb7eFI7SHGppY42zfG098aIXzf3+ELuRtX\nnNVKlr/zev58cIIzj5mSvFfJLwVEERERj3Ach6nYAsM2xnB/jFj6qio+Hxw61uBuMuluZILr9E3+\ngL8d6md0/lrm+yOVjdzfeg+90R56IiepClblPP+haA1v/+EuvvitEVbsA1SaFwFYsQ8U701KUSgg\nioiIHGCO4zB+bY5hO8FQf4yZ9NVT/H4fx7ua6Oxppv5ogKGlIb499TX6Xx5kNemeSxj0BzndZOht\nMvRGe2irbr2pxtmnT0ToG4Hl77whc1+krpJfeuc9BXmPUnxqc5MHv/fSZ+iPDwLQE+nmF+//2RKP\nSKT8qM2NlJNUKsW1yzPu8vFAjIX0VVKCFX6Od0U5fqqRlaZp7Pwg56b6GV/caG7dWt2cWTY+1dhF\nKBC6rTF88A++lbk6S6Suko899uq9v7F9qFzb3Cgg7tHvvfQZzscHcu5rrGzwZC8okf1MAVG8LpFI\ncmUkzrCNMTIQY3kpAUBlOEhHd5RIRwWTNaOcn7EMzgyTSLnHQ4EQPZHudCg0NFdF8zKei9fn+NTn\nX8bv9/G+H7+bDo81yF6ngFgkXguI7/vGf8DhxrdUH6rjI6/59RKMSKQ8KSCKF62uJLg0NMWwneDi\nhSnW0lc9qa4Jcay7EX/7EqOVw5yLW+Ir05nvO1J7KDNL2NXQQdBfuDPKvF575RoQdQ5igcyuzvG/\nf/sjdDd2pf900lIVva2LoouISPlYXlrLtKO5MjJFMulOQtQ3hjnWW8dS8xRDfJfn5y6Sirv9C6uD\nVTzYei+noz2cbjpFY2VDKd+CeIAC4h71RLpvWGIOB8IcrTvE1fnrnL3+AmevvwBAQ6ie7sbOTGBs\nr2nF79PFbEREyt387DLDAzGGrduj0EkvTDU2V1F1JEW8cZTvrfUxl5iHWfDho6P+WGZzSUf9MX2e\nSF5piTkPfvVbv830ygzgnn/426/+VQBSToprC2MMTA8xOD3M4PQQc6vzme+rqajOhMXuxk6O1h5W\ngYvcJq8vc4n3TE8tMmxjDNkJxq9u/NttaA1B+yKjNRe4mBzKnMZUF6rNLBvfETlFbaimVEPP4fXa\nK9clZgXEPLg0d4X/8vKf4Pf7+Nm73r3t5hTHcRhfnGBwepiBdGDMPmckHAhzsvFEZpbxeN2Rgp43\nIuIlXv+QkoPPcRwmxxcYshMM2xhTEwuA26Owps3PYvMkQ5V9zAfcCQe/z8/JBrdR9eloD0dq2/fl\nJILXa08BcRvGmMeBM4ADvN9a+3zWsTcAvwMkgX7g31prdwyAXgyI626nSCaX4gxOD6X/DDO+tNGK\noMJfQWdDB6fSgfFE/XFCgYp8D1vEE7z+ISUHk+M4jI3OZkLh7PQyAP4AVLQmmGy4wuWqAZIVa4Db\nqPrOqDtLaCLdVAXDpRz+TfF67ZVrQNxxesoY8zqg21r7sDHmDuCzwMNZD/kj4PXW2lFjzJ8DbwX+\npmCj9aBoVYRo1YOcOfQgADMrs5mwODg9jI0PYtM9FgO+AB31xziVXpbuauggfAB+eIiIlJNkMsW1\ny9OZHoWL825Tan8QODTP1dohpuuvkQokCfqDmMYueqM99Db10Fbdos2Msi/stn75RuAvAay1540x\nEWNMrbV2/US6B6216xd3nACaCjTOstFQWc+DbffxYNt9AMyvLXAhHRYHp4cYnrnI0MwIf3vRXX44\nWns4ExhPNnZSU1Fd4ncgIlJ+EmtJLg/HGbITXBycZGXZ7UHoC6VYap9krH6EhfpJHH+KtuoWXtf0\nQ5yO9nCqsfO2G1WLFNJuAbEdeCHr6wngEDAAsB4OjTGHgLcAv1aAMZa12ooa7m25i3tb7gJgKbHM\n0MxIJjBenL3CpbkrfP3yPwBwuKadUxG3tc7Jhk4aKr3ZuFREpNRWlhNcvDDJsJ3g0tAUiTW35YxT\nuUa8/SozjddYqItTGaygJ3KK3uhrON3UQ3OV5lJk/7vVHRA+yO0KbYxpBZ4Gft5aG8/XwGRrVcEw\nd0bv4M7oHQCsJlcZnrmUWZYenr3I1YXrPHPl24B7SaXuhq50aOykKRwp5fBFRA60xYVVRgZjDPfH\nuDISJ5VyPxITVUvEm68yG7nOUs0MR+oO8XDTPUVpVC1SCLv9i72KO4u47jBwbf0LY0w98GXgV6y1\nX7uZF4xEqgkGA7c6zgOjpaX4M3ZH2qO8hvsBWEuuMRS/RN/4AOcmBuiPDfHta8/y7WvPuuOrbuJ0\nyyl6W09xuuUU7bU630W8oxT1J943E1/k/Peuc+5717g8PJXpUbhUPcts5DqzkesEG1Lc236a+w79\nc+5pP01TVWNpB11kqj3v2TEZGGN+CPiwtfYtxpgHgE9Ya1+bdfwzwN9ba//bzb6gdjEXVzKVZHT+\nWqYX44XpYRYSi5nj9aE6uhs70+cxdql5txxY+7H+5OCKTy4w1B9jsH+MqTH3Z6aDw2JtnNnIGHNN\nYxxuaVajarxfe+W6i/lm2tx8BHgtbiubx4AHgBngb4E48I9ZD/9Ta+1ndno+BcTSSjkpri+MpwOj\nGxpnVzfGXBOs5mRjZ6a1zpHaQwT83p3xFe84CPUn+5fjOMTG5rnQP449f52FuNt2xvGlWKibZKZp\nDKd1np72TrdRddMpaiv2R6PqUvN67SkgFokC4v7iOA4TS7Gsq70MM7W8cSppOFBJV8MJd4Yx0snx\nuqM6l0b2pYNYf1JaqZTD9dEZzveNMmRjrC24H08pX5L5xgnmIuM0d4TpbTu1rxtVl5rXa69cA6I+\n6cucz+ejtbqF1uoWXn34DOA2774wM8xAfIjBmSH6pvrpm+oH0s2764+7y9KR9ebdatEgIgdDMpni\n4vAk3/v+CNeH50mtuJ/9ycAac9FxUm3zdHY185q2uw5Mo2qRQlBAlBu4zbsjPNT+AAAzK3MbgXF6\nCDt9ATt9AUbWm3cfTV9Tuouuhg79QBWRfWVtNcn3z1/kXN8VZq6sQcI9bSYRXGW+dZz6YwF6Tx3j\nzpY306pG1SKAlpjzyuvT7OsW1hYzzbsHpoe4PDeauZi8Dx/H6g5nAuPJxhM6T0eKolzqT27O7MIi\nz718nmEbY3UsiC/lLg2vhpZItM5y6GQt93af5FTTSV3CdI+8XnvlusSsgJhHXi+S7SwnlhmauZgJ\njBdnL5N0kpnjh2va6U5veulu7KShsr6EoxWvKtf6E5fjOAyPj/Li9wcZG1rAN1WFz3FD4UrVPOHD\nSbp6Wnigu4fm6miJR+stXq+9cg2IWmKWPQsHw+51RKM9AKwm1xiZdZt3D0wPMzzjNu/+h1F3w3tr\nVXNWYOwiWqXm3SJy6xbXlvjuxX7OnRtl9nKS0GwdPnz4qSFRt0iko4K7eo9x14luba4TuUWaQcwj\nr/8WdbsSqQSX5kYzbXUuTI+wnFzOHI9UNtLd2OW21ol00VrVrHOA5Jap/rwv5aS4PDvKd0csI3aS\nxLUwVYvuioSDg79plcMn63jFPac43NJS4tGWD6/XXrnOICog5sEXn/guV0bi4IOjHRF+7NF7Sz2k\nfS3lpLgyfzXTVmdweoiFtY3m3XWh2sxy9KnGLg7VtKm1hOzK6x9S5WpudZ6+yX5+cGGE8eEFwrEo\nlSvuec2OL0VVO5w0rTx4dzc1tdogVwperz0FxCLxWkDMhMMsNXUh3vbOu2lp16WHbsZ68+71sDg4\nPcRMVvPu6mAVJxs7M4HxaO1hNe+WG3j9Q6pcJFNJRmYv84NYP/bCFZZHA9RPt1GxWuU+IOAQORbi\nzt5jGHOYyrCWjkvN67WngFgkXguIf/i7f7/l/TW1Id79voeLOxiPcJt3T+YExsms5t2VgRBdDSfS\ny9JdHK8/SoXOLyp7Xv+Q8rL48jTnpiw/GO/n8sgU4VgTddNtBBNuj1V/yOFwZwN33nmM451NBCv0\nC+J+4vXaU0AsknIJiACth+qINNcQaa6mqbmGSLSauoawzq+7DVPL8Zwl6bHFicyxCn+QE/XHM8vS\nXQ0dat5dhrz+IeUla6kEF6aH6Zvq59zYIPNXU9TH26mbbiWQcn/Zq6jyc9K0cOp0G4eONRII6DST\n/crrtaeAWCReC4hbLTEHgn5qakPMz66QSuW+3WCFn0g0KzSm/1ZwvDWzq3M5gfHq/PVML0a/z09H\n3bH0TulOTjaeoCpYVeIRS6F5/UPqoJtYnHSvyjTZz+D4RcJTTdTH26idacbvuDOCNfUVdN/RRldP\nC22H6/Uz8YDweu0pIBaJ1wIiwOf+4NsszK0C7vmH737MXVpOpVLMxJeJxxaIxxaYmlwkPrFAfGqR\nVHJTcAz6aYxuhMZIcw1NzdXUNVTh95flv81bsrC2yNDMSOaa0pfnRkk5KcBt3n207vBGa52GTmpD\nat7tNV7/kDpoVpKrDMQvZEJhfGae+ng79fE2amaj+NIfP5Hmarp6WugyLURbaxQKDyCv154CYpF4\nMSBOXJ/jbz7/Pfx+P//kx+/cdXNKKpVidtoNjlOxxXSAXCQ+tUgykcp5bCDoJ9JUnRMaI8011Dcq\nOO5kObHM8MxGL8aLs5dIZDXvPlTTllmS7m7spLGyoYSjlXzw+ofUfuc4DtcWxtxl40nL4PQQ/qVK\n6uPtNMYPEZ7fqLHWw3V0mRY6TTONTdUlHLXkg9drTwGxSLwYENfttUhSKYe5maWc0DgVW2B6cpHE\n5uAY8NHYVE2kpYamaHX6XMcaGiJh/H6dq7PZanKNi7OXMsvSQzMjrKbWMsdbqqKZTS/djZ00hSOa\nyThgvP4htR8tri3RHx+kb7Kfvql+ppdnCC/VUT/VTnTmKIEF99QOnw8OH2+ky7RwwjRTW1dZ4pFL\nPnm99hQQi0QB8dalUg7zs8tMZYXGeGyR+OQCibXc4OhPB8em5uqccx3rI1U6yTtLIpXg8txo1k7p\nrZp3b7TWaa1uUWDc57z+IbUfpJwUV+auZpaNh2cvkUqlqJpvJDpzlMbpw7DobjIJBHwc7WyiyzTT\n0R2lqlobx7zK67WngFgkCoj54zgOczPLxCezQmNsganYFsHR76OhqSqzm7qppYZItIaGJgVHcD/4\nRuevZQXGYebXFjLH6ypqM+cwnoqoefd+5PUPqVKZW53n3JSlb9JybqrfrYuUj9q5KEcWugnHIiSX\n3Y+SilCAjpNNdJoWjnc1EapU+6ly4PXaU0AsEgXEwnMch/nZlZyZxvXba6vJnMf6/T4aIlXuOY7R\nmnRwrKaxqZpAsHwDkOM4XF8cz4TFgfgQM6uzmeNVwSq6Gzd6Map5d+ntl/o76NYbVa/PEl6eG8XB\nwZfy07pwjEPzXThj1SRX3R/l4aogJ04102VaOHKikWBQdVBuvF57CohFooBYOo7jsDC3kjnHcSq2\nQHzSvb26khscfT7SwbFmY2d1tIbGaFVZfgA4jkNsaSoTGAenh4gtT2WObzTvdmcZO+qPqXl3ke33\n+tvP1htV9032cz4+wFLCPd2iIlnJydXTNEwfZvmaj2TC/fFdUxfKbDI5dKxB5z2XOa/XngJikSgg\n7j+O47Awv5qzMSY+ucDUxCKrK4mcx/p8UN9YldXH0Z1xjESry+7qBvHl6Zwl6euL45ljQX+Qzvrj\nmcDY2dBBpZp3F9RBrb9SyGlUPWm5unA9c6zF30LnSi+VExFmrq5lerk2RKro6mmm07TQeqhO5+RK\nhtdrTwGxSBQQDw7HcVhcWM3aGLMRIFeWEzc8vr4xvBEa0wGyMVpNRZkEx7nV+ZzAODp/bVPz7qOZ\n1jpq3p1/Xqu/fMtuVG3jg5ld/BX+IKfCPbTPdZK6Fmby2iJO+qd0c2stnT3u8nGkuVqhULbk9dpT\nQCwSBcSDz3EclhbXNpapM0vWiywvrd3w+LqGcM5VY9zl6moqQt5egl1cW+LCzHCmtc6luSu5zbtr\nD2UFxk7qQrUlHvHBVi71d7NWkqvY+GBm6XhiaTJzrL26FVNxmoZ4O3NXHGLX5zeOHa3PLB/XN+qX\nGNmd12tPAbFIFBC9bWlxlamJjXMb1893XFrcOjhmNsekG4BHotWe3fm4nFhhePZiZpZxZPYyidTG\nTGx7TZvbVqehk+5Il5p336Jyr7/sRtV9k/1cmB7ONIcPByoxkVN00U3FRCPXh+aJTy4C7ka1Ix2N\ndJpmTpxqpqZWPQrl1ni99hQQi0QBsTwtLa5mdlRn93JcXFi94bG19ZUbV43J6uXoteC4llxjZPZy\nJjAOzV5kNbnx36O5KrrRWqexi6iad++oHOtvcW2J8/EBzk320zdlmV6ZyRw7VnuYO5p6OLzawcpo\nkIsDk8zNrgDuFZqOdzbR2dPMie4oleGKUr0F8QCv154CYpEoIEq25SV3qTo+uZgz87gwf2NwrKmr\n3JhpXF+ujlZ75sMtmUpyaW40cw7jhZnhzG5SgMbKhpzA2Kbm3TnKof5STorLc6P0TVr6pvoZmb2U\nOW2hpqKa002GOxoMjQvtjA0tMDIQy8zehyoDdHRH6TItHOtsoiJUHucGS+F5vfYUEItEAVFuxsry\nmjvTOLlAfGKjl+PC3BbBsTaUGxrTs48HPTi6zbuv57TWyW7eXVtRk3U96S6O1LaXdfNur9bfRqPq\nfs5N2cy/AR8+TtQfpzdqMPWnYKKakYEYFwcnM22rqqor6DTNdJpmjnRE1BRfCsKrtbdOAbFIFBBl\nL1aWE5ll6nhsgan0jON8euksW3VNKLcdT/p2uOpgBkfHcRhbnGBgeigTGrOXFKuCVZzM6sV4vO5I\nWTXv9kr9JVNJhmcvpZeN+7k0N5o51hCq43S0h96mHjqrO5m4uMhwf4zLw1OZ67XX1le6m0x6mmk/\n0oDfX5afbVJEXqm97SggFokCohTC6kriho0x8dhC5pyrbFU1FTkbY9Z3Vh+0a8U6jsPkcjwnMMay\ndqqGAiG66jsys4wn6o9RETiY4fhmHOT6iy9PpzeXWPqzGlUHfAFONpygN9pDb7SHRqeJkcEYwzbG\n6MXpTI/CSLQ6046mua1Wpx5IUR3k2rsZCohFooAoxbS2mkhfq3ojNE7FFpmbWb7hseHqCpqi1URa\namhKb46JNNdQVV1xYD5wp1dmGIwPMZBur3N9YSxzLOgPcqL+WCYwdjWc8FTz7oNUf5lG1elZwmtZ\n/5+i4SY3EDYZTOQkq/MOwzbGkJ3g+pWNyz22tNfRaZrpMs1EmmtK8TZEgINVe7dDAbFIFBBlP1hb\nTTI9tdEAfD1Azk5vERyrgu5u6pYaN0Cmz3Gsqgnt++A4tzrPhenhzDmMVzY17z5edzS9JN3JyYZO\nqisObt+7/V5/44ux9JVL+rHxC1mNqis4Femit6mHO6M9NIejTE8uMWQnGO6PERt3exT6fNB+tCHT\no7CuIVzKtyOSsd9rb68UEItEAVH2s7W1JNOTuec3xmOLzE4vZa4usa4yHMxpx9PU4v5dXbt/g+Pi\n2hJDMyOZwHhxU/PuI7WHMucwdh+w5t37rf6WEysMTF/I7DjOXv5vr2mjt8nQ29TDycZOKvxBxq/N\nMWwnGOqPMRNfAtwehUdPRNLtaJqprvHOjK94x36rvXxTQCwSBUQ5iBJrSaanljLXqV7v5TgbvzE4\nhiqDOY2/m1rcv2vqKvddcFxJrjI8czFzDuPw7KXc5t3VrTmBMRJuLOFod1bq+tu5UXWYO5q66W3q\n4XTU0BSOkEqluHZ5Jr18HGNhzj1fNljh53hXlK6eZo53RakMe6v/p3hPqWuv0BQQi0QBUbwkmUhl\nLVWn/55cZGZqcYvgGMhp/L1+FZna+v0THNdSCS7OXs7qxTiS27w73JTTWqe5qmnfjL0U9bdjo+q6\nI/Q2uZtLOuuPE/AHSCSSjI5MM2QnGBmIsbzkhvHKcDCrR2GEYJlcv1y8weuffQqIRaKAKOUgmXSD\nY/Z1quOTC8xMLWV2nq6rCAWIZJ3buL6zej8Ex2QqyZX5q1k7pUdYSixljm8073YDY3t1a8nGXIz6\nu5lG1euzhPWhOsDdYX9paIphO8HFC1OsrbqzitU1IXeTSU8zh441qkehHFhe/+xTQCwSBUQpZ8lk\nipn4Um47nslFpicXbwiOwQp/7nWq0zOPdQ3hkoWwlJPi2sIYA/GN1jpza/OZ427z7o0l6SO1h4rW\nvLtQ9bdTo+rOhuOZWcJjdUcy73V5aY2RAXfp+MrwFMmk+/+2vjFMp2mhq6eZtsP1Jf8FQCQfvP7Z\np4BYJAqIIjdKJlPMTi/lXKd6KrbA9NQiqeRWwbF603K1GxyL3RTZcRzGFycYnB5mYHqIgemhTc27\nw+nm3W5gPF53tGDNu/NVfzs3qq7P9CS8I9JNdUV15tj83Aoj6XY0Vy9NZ04xaGqpSbejaSHaWqNQ\nKJ7j9c8+BcQiUUAUuXmpVIrZ6eWc61RPxRaYnlzMzEqtCwb9NEars65T7QbI+saqogVHx3GYWo5n\nAuPg9BAT2c27/RV0Za720smJ+uN5a969l/rbsVF1Y6e74zjaw+Ga9pyANxNfZKjfDYXjVzdeu/Vw\nXaYdTWNT9Q2vJ+IlXv/sU0AsEgVEkb1LpRzmZpaYyrpOdTy2SHxykWT6kmvrAgFfOjimrxqTvt0Q\nCeP3F375d3plhgvTwwykW+tkN4UO+gJ01B/nVHpZurOhg3Cw8rZe51bqby25xuDMMOfS5xJublR9\nZ3qW8FTjyZzxOI7D5PiC247GxpiaSC83++Dw8Ua6TAsnTDO1dbf3HkQOIq9/9ikgFokCokjhuMFx\nOXNuozvz6N5OrOUGR3/AR2NT7o7qphZ3xrGQGybmVxe4MLM+wzjMlbmrOc27j9Udobuxk1ONXZxs\nOJGzjLuT3epvfHEis7nExi+wltWo2kROps8lNLRUNefMEjqOw9jVWYb6YwzbiUwz9UDAx9HOJrpM\nMx3d0QN3qUaRfPH6Z58CYpEoIIoUn+OsB8eNq8e4lyBcuDE4+n00NFVlzm1c3yTTEClMcFxKLDE0\nczG98WWYi3OXc5p3H65tz2qt05nZHbzu9176DP3xQQB6It384v0/C2Q3qnb7EsaWpzLfk2lUHe2h\nu6HzhmXuZDLFtcvTbigciLE477b6qQgF6DjZRKdp4XhXE6FK9SgU8fpnnwJikSggiuwfjuMwP7uy\nERpji0ylG4Gvt2NZtx4cN/dybIxUEwjmLziuJlcZnrnEYHrTy8jsJdaymne3ZZp3d/LN0f/JhZmR\nnO8PB8K0VjczOn+N5A6NqjdLrCW5PBxnyE5wcXCSlWX3NcNVQU6ccjeZHDnRSDCoHoUi2bz+2aeA\nWCQKiCL7n+M4LMyt5DYAT/dyXF3JDY4+HzQ0VbtXjclqx9PYlJ/guJZKcGn2Slbz7mFWspp3b2er\nRtWbrSwnuDQ0yVB/jEtDk5nZ1Jq6UGaTyaFjDUU5V1PkoPL6Z58CYpEoIIocXI7jsDC/mtlNvdEI\nfOvgWB+poilaQyR9neqm5moao9V7moVbb949OD3MFwa/tOVj6kN1fOQ1v77lscWFVUYGYwz3x7gy\nEs/0n2yIVNHV00ynaaH1UJ3a0YjcJK9/9pVrQNQJNCJy03w+H7V1ldTWVXKssylzv+M4LM6vpndU\nL+Y0Ah+eijE8kP0cUN9Y5e6mbqmhKb2rujFaTcVNXGIu4A/QUX+Mjvpj9E32cz4+kHO8sbKBn7vn\nZ3Lum5tZZti6m0yuXZnJ9Chsbq2ls8ddPo40VysUioikaQYxj7z+W5TIrXIch6WF1Y3QmO7lGI8t\nZK5DnK2+MbyxMSZaQ1OLu1RdEdo+OD7+6aeonG4AYKVxhl9+79sBiE8uMGxjDPXHmLi+UZftR+sz\ny8f1jVV5fsci5cfrn33lOoOogJhHXi8SkXxxHIelxbUbNsZMxRZYXly74fF1DeGNSw5Gq2lqcf/+\n71/4AVdG4jmPrQgFCFdVMDfjtqPx+30c6Wik0zRz4lQzNbXqUSiST17/7CvXgKglZhEpOp/PR3VN\niOqaEEc6cncULy2u3nDJwfjkAhcvTHHxwtQ2z7hhbTXJ2mqSzlPNdPY0c6I7SmU4P1drEREpFwqI\nIrKvVFWHqDoe4vDxxpz7l5fWbtgYM3pxesvnqK4N8dZ33lWM4YqIeNKuAdEY8zhwBnCA91trn886\n9iPAbwNJ4MvW2t8q1EBFpLyFqyo4fKyRw8c2guMXn/juDUvMNXUh3vbOu4s9PBERT9mxuZcx5nVA\nt7X2YeA9wKc2PeSTwDuAVwNvMcacLsgoRUS28GOP3ktN3cYl7mrqQrz7sYdpaa/b4btERGQ3u3V/\nfSPwlwDW2vNAxBhTC2CM6QKmrLWj1loH+DLwpkIOVkRks7e9825q6kLUNYQ1cygikie7LTG3Ay9k\nfT2Rvm8w/fdE1rFx4GReRycisouW9jp31tDjOylFRIrpVq8ftdNW77LcBi4iIiLiNbvNIF7FnSlc\ndxi4lr49uunY0fR9OyrXfkIiIiIiB8VuM4hfAX4CwBjzADBqrV0AsNZeBOqNMR3GmCDwz9KPFxER\nER/Y1J4AAAJBSURBVJEDbNfZPGPMR4DX4rayeQx4AJix1j5ljPlh4D+lH/oX1tqPF2ykIiIiIiIi\nIiIiIiIiIiIiIiIiIiIiIiIism+p5cxtMMacAL4HrF+XujL99c8DQ8Al3E09AI619o3FHqOIV6Xr\n70lr7Suz7vtN4KdwW235gCrgI9bap0oxRhEv2vTZ5wMSwO/gtsB7DxAG7mTjAhvvttZeLv5IJR92\n64Mo2ztvrX3D+hfGmD8GfhpwgLdaaxdLNjKR8uMAn7DW/p8AxpgI8B1jzN9Ya1dKOzQRT8l89qUv\nuftF4CettW8wxnTgdjR5w47PIAfCrV5JRbZ3Fugu9SBEylhmRcRaG8dt6n+odMMR8TZr7RDw28D7\n0ndpVdJDFBDzwBhTAfxz4MX0XSoSkRIyxvQAbcCVUo9FxONeAHpLPQjJPy0x374eY8zfpW/fA/yu\ntfavjDGfBP7GGLN+DuKEtfZflGaIImXDB7zfGPMTQD3uecH/i7U2UdphiXheHRvn3IuHKCDevv6s\n8zCeBAbS9+scRJHiy5yDaIxpB76BezK9iBTWK9hYPRMP0RJzfnwI+F1jTHWpByJSpnzpP1hrrwOf\nA36jpCMS8ThjzEngl4HHSz0WyT/NIN4+Z/2GtXbEGPN54Fez7xeRgrnTGHMu6+vLQHZLm48DLxtj\n/thaew4RyZf106sqgQDwC9ba7HN99RkoIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIi\nIiIiIiIiIiIiIiIit+//B7Vh4cMMD5GZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd7f50cc710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "%matplotlib inline\n",
    "from IPython.core.pylabtools import figsize\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "\n",
    "ind = np.arange(results.shape[0])\n",
    "width = 0.2\n",
    "l = ax.plot(ind, results, \"-o\")\n",
    "plt.legend(iter(l), results.columns.tolist(), loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.25, ind[-1]+.25])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  Nevertheless, none of these measures takes into account the \n",
    "  business and economical realities that take place in credit scoring. Costs that the financial \n",
    "  institution had incurred to acquire customers, or the expected profit due to a particular client, \n",
    "  are not considered in the evaluation of the different models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Financial Evaluation of a Credit Scorecard\n",
    "\n",
    "  Typically, a credit risk model is evaluated using standard cost-insensitive measures.\n",
    "  However, in practice, the cost associated with approving \n",
    "  what is known as a bad customer, i.e., a customer who default his credit loan, is quite \n",
    "  different from the cost associated with declining a good customer,  i.e., a customer who \n",
    "  successfully repay his credit loan. Furthermore, the costs are not constant among customers. \n",
    "  This is because loans have different credit line amounts, terms, and even interest rates. Some \n",
    "  authors have proposed methods that include the misclassification costs in the credit scoring \n",
    "  context [4,5,6,7].\n",
    "  \n",
    "  In order to take into account the varying costs that each example carries, we proposed in \n",
    "  [8], a cost matrix with example-dependent misclassification costs as \n",
    "  given in the following table.\n",
    "  \n",
    "  \n",
    "|  \t| Actual Positive ($y_i=1$)  \t|  Actual Negative \t($y_i=0$)|\n",
    "|---\t|:-:\t|:-:\t|\n",
    "|   Predicted Positive ($c_i=1$)\t|   $C_{TP_i}=0$\t|  $C_{FP_i}=r_i+C^a_{FP}$ \t|\n",
    "|  Predicted Negative  ($c_i=0$) \t|   $C_{FN_i}=Cl_i \\cdot L_{gd}$\t| $C_{TN_i}=0$\t|\n",
    "  \n",
    "  First, we assume that the costs of a correct \n",
    "  classification, $C_{TP_i}$ and $C_{TN_i}$, are zero for every customer $i$. We define $C_{FN_i}$ \n",
    "  to be the losses if the customer $i$ defaults to be proportional to his credit line $Cl_i$. We \n",
    "  define the cost of a false positive per customer $C_{FP_i}$ as the sum of two real financial \n",
    "  costs $r_i$ and $C^a_{FP}$, where $r_i$ is the loss in profit by rejecting what would have been a \n",
    "  good customer. \n",
    "  \n",
    "  The profit per customer $r_i$ is calculated as the present value of the difference between the \n",
    "  financial institution gains and expenses, given the credit line $Cl_i$, the term $l_i$ and the \n",
    "  financial institution lending rate $int_{r_i}$ for customer $i$, and the financial institution \n",
    "  of cost funds $int_{cf}$.\n",
    "\n",
    "  $$  r_i= PV(A(Cl_i,int_{r_i},l_i),int_{cf},l_i)-Cl_i,$$\n",
    "  \n",
    "  with $A$ being the customer monthly payment and $PV$ the present value of the monthly payments,\n",
    "  which are calculated using the time value of money equations [9],\n",
    " \n",
    " $$ A(Cl_i,int_{r_i},l_i) =  Cl_i \\frac{int_{r_i}(1+int_{r_i})^{l_i}}{(1+int_{r_i})^{l_i}-1},$$\n",
    " \n",
    " $$ PV(A,int_{cf},l_i) = \\frac{A}{int_{cf}} \\left(1-\\frac{1}{(1+int_{cf})^{l_i}} \\right).$$\n",
    "      \n",
    "  The second term $C^a_{FP}$, is related to the assumption that the financial institution will not \n",
    "  keep the money of the declined customer idle. It will instead give a loan to an alternative \n",
    "  customer [10]. Since no further information is known about the alternative customer, \n",
    "  it is assumed to have an average credit line $\\overline{Cl}$ and an average profit $\\overline{r}$.\n",
    "  Given that, \n",
    "  \n",
    "  $$  C^a_{FP}=- \\overline{r} \\cdot \\pi_0+\\overline{Cl}\\cdot L_{gd} \\cdot \\pi_1,$$\n",
    "\n",
    "  in other words minus the profit of an average alternative customer plus the expected loss, \n",
    "  taking into account that the alternative customer will pay his debt with a probability equal to \n",
    "  the prior negative rate, and similarly will default with probability equal to the prior positive \n",
    "  rate.\n",
    "  \n",
    "  One key parameter of our model is the credit limit. There exists several strategies to calculate \n",
    "  the $Cl_i$ depending on the type of loans, the state of the economy, the current portfolio, \n",
    "  among others [1,9]. Nevertheless, given the lack of information \n",
    "  regarding the specific business environments of the considered datasets, we simply define \n",
    "  $Cl_i$ as\n",
    "\n",
    "$$      Cl_i = \\min \\bigg\\{ q \\cdot Inc_i, Cl_{max}, Cl_{max}(debt_i) \\bigg\\},$$\n",
    "  \n",
    "  where $Inc_i$ and $debt_i$ are the monthly income and debt ratio of the customer $i$, \n",
    "  respectively, $q$ is a parameter that defines the maximum $Cl_i$ in times $Inc_i$, and \n",
    "  $Cl_{max}$ the maximum overall credit line. Lastly, the maximum credit line given the current \n",
    "  debt is calculated as the maximum credit limit such that the current debt ratio plus the new \n",
    "  monthly payment does not surpass the customer monthly income. It is calculated as\n",
    " \n",
    " $$  Cl_{max}(debt_i)=PV\\left(Inc_i \\cdot P_{m}(debt_i),int_{r_i},l_i\\right),$$\n",
    "  and\n",
    "  $$ P_{m}(debt_i)=\\min \\left\\{ \\frac{A(q \\cdot Inc_i,int_{r_i},l_i)}{Inc_i},\\left(1-debt_i \\right) \\right\\}.$$\n",
    "  \n",
    "  \n",
    "### Financial savings\n",
    "\n",
    "  Let $\\mathcal{S}$ be a set of $N$ examples $i$, $N=\\vert S \\vert$, where each example is \n",
    "  represented by  the augmented feature vector \n",
    "  $\\mathbf{x}_i^*=[\\mathbf{x}_i, C_{TP_i},C_{FP_i},C_{FN_i},C_{TN_i}]$  \n",
    "  and labeled using the class   label $y_i   \\in \\{0,1\\}$. \n",
    "  A classifier $f$ which generates the   predicted label $c_i$ for each   element $i$ is trained  \n",
    "  using the set $\\mathcal{S}$. Then the cost of   using $f$ on $\\mathcal{S}$ is calculated by\n",
    "  \n",
    "  $$   Cost(f(\\mathcal{S})) = \\sum_{i=1}^N Cost(f(\\mathbf{x}_i^*)),$$\n",
    "  \n",
    "  where\n",
    "  \n",
    " $$   Cost(f(\\mathbf{x}_i^*)) = y_i(c_i C_{TP_i} + (1-c_i)C_{FN_i}) + (1-y_i)(c_i C_{FP_i} + (1-c_i)C_{TN_i}).$$\n",
    "  \n",
    "\n",
    "  However, the total cost may not be easy to interpret. We proposed an approach in [8], where the savings of using an algorithm  are defined as the cost of the algorithm versus the cost of using no algorithm at all.  To do that, the cost of the costless class is defined as \n",
    "  \n",
    "  $$  Cost_l(\\mathcal{S}) = \\min \\{Cost(f_0(\\mathcal{S})), Cost(f_1(\\mathcal{S}))\\},$$\n",
    "  \n",
    "  where \n",
    "  \n",
    "  $$  f_a(\\mathcal{S}) = \\mathbf{a}, \\text{ with } a\\in \\{0,1\\}.$$\n",
    "  \n",
    "\n",
    "  The cost improvement can be expressed as the cost savings as compared with $Cost_l(\\mathcal{S})$. \n",
    "  \n",
    "  $$    Savings(f(\\mathcal{S})) = \\frac{ Cost_l(\\mathcal{S}) - Cost(f(\\mathcal{S}))}   {Cost_l(\\mathcal{S})}.$$\n",
    "  \n",
    "\n",
    "\n",
    "  ### Parameters for the Kaggle Credit Database\n",
    "\n",
    " As this database contain information regarding the features, and more importantly about the income of each example, from which an estimated credit limit $Cl_i$ can be calculated.\n",
    "Since no specific information regarding the datasets is provided, we assume that they belong to  an\n",
    "average Portuguese financial institution. This enabled us to find the different \n",
    "parameters needed to calculate the cost measure. \n",
    "\n",
    "| Parameter \t| Value |\n",
    "|---\t|:-:\t|\n",
    "|Interest rate ($int_r$) | 4.79% |\n",
    "|  Cost of funds ($int_{cf}$) | 2.94% |\n",
    "|  Term ($l$) in months | 24 |\n",
    "|  Loss given default ($L_{gd}$) | 75% |\n",
    "|  Times income ($q$) | 3 |\n",
    "|  Maximum credit line ($Cl_{max}$) | 25,000|\n",
    "\n",
    "In particular, we obtain the average interest rates in Europe during   2013 from the European Central Bank [11]. Additionally, we use a fixed loan term $l$ to two years,\n",
    "considering that in the Kaggle Credit dataset the class was constructed to predict two years of credit behavior.\n",
    "Moreover, we set the loss given default $L_{gd}$ using information from \n",
    "the Basel II standard, $q$ to 3 since it is the average personal loan requests related to monthly income, and the maximum credit limit $Cl_{max}$ to 25,000 Euros.\n",
    "\n",
    "### Calculation of the savings of the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  1023.73054104  18750.              0.              0.        ]\n",
      " [   717.25781516   6749.25            0.              0.        ]\n",
      " [   866.65393177  12599.25            0.              0.        ]]\n"
     ]
    }
   ],
   "source": [
    "# The cost matrix is already calculated for the dataset\n",
    "# cost_mat[C_FP,C_FN,C_TP,C_TN]\n",
    "print data.cost_mat[[10, 17, 50]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         pre        f1       acc       rec       sav\n",
      "RF  0.492091  0.232172  0.934394  0.151926  0.122358\n",
      "LR  0.529412  0.037677  0.934854  0.019533  0.010948\n",
      "DT  0.251239  0.262626  0.899146  0.275095  0.191770\n"
     ]
    }
   ],
   "source": [
    "# Calculation of the cost and savings\n",
    "from costcla.metrics import savings_score, cost_loss \n",
    "\n",
    "# Evaluate the savings for each model\n",
    "results[\"sav\"] = np.zeros(results.shape[0])\n",
    "for model in classifiers.keys():\n",
    "    results[\"sav\"].loc[model] = savings_score(y_test, classifiers[model][\"c\"], cost_mat_test)\n",
    "\n",
    "# TODO: plot results\n",
    "print results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqMAAAE7CAYAAADpQojbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8lfXd//H3OTlJSEiABBJChlLQrzjuDrQO1FKttVZb\n21pH1Vq1TmQKhnH3/nXeLUjY7r2tdYvbWlGBanuLAxz0qyKYARkkZJKQcX5/nBNMGNnJ94zX8/Hw\nYc65zpXzRq8cPrnG+5IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADQRzydvcAYs1TSMZL8kqZb\na99ps+wKSb+W1CzpA2vt5M7WAQAAAFp5O1pojJko6SBr7QRJl0la0WZZoqTzJJ1grT1B0jhjzHEd\nrQMAAAC01eEwKulkSU9JkrV2o6QUY0xS8HGdtfYUa21zcDAdKmlbR+sAAAAAbXU2jGZIKmvzuFTS\nqLYvMMbMlfSZpL9Za7/oyjoAAACA1PkwuiePAueB7matXSBpjKQfGmMmdGUdAAAAQJJ8nSwvUmBP\nZ6tMSVslyRiTKunr1trXrbX1xpgXJR3f0Tr709jY5Pf5YrqbHQAAYMB5PJ5OLwBH13U2jL4i6Q+S\nbjfGjJdUaK2tDS6LlXSXMebrweeOlnS/Aofo97fOPlVU1PXmz4AeSEtLVmlptesYQL9iO0c0YDtH\nuOtwGLXWvmWMWWeMWatAfdNkY8zFkiqttU8bY/4oaZUxpknS+9baZyVpz3X6+c8AAACAMBUSu5lL\nSqo4p3SA8Zs0ogHbOaIB2/nAS08fEhLzU6To7gVMAAAAQJ9hGAUAAIAzDKMAAABwhmEUAAAAzjCM\nAgAAwBmGUQAAADjDMAoAAABnGEYBAADgDMMoAAAAnGEYBQAAgDMMowAAAHCGYRQAAADOMIwCAADA\nGYZRAAAAOMMwCgAAAGcYRgEAAOAMwygAAACcYRgFAACAMwyjAAAAcIZhFAAAAM4wjAIAAMAZhlEA\nAAA4wzAKAAAAZxhGAQAA4AzDKAAAAJxhGAUAAIAzDKMAAABwhmEUAAAAzvhcBwAAAN1XsDhPdRs/\nlpWUOO4wZc/KdR0J6BH2jAIAEGYKFuep7pOPJL9f8vtV98lH2pR7req3bHYdDeg2hlEAAMJM3caP\n93quqaJCRTcud5AG6B2GUQAAADjDMAoAQBhprqmRJzZ2r+d9KSnKnDLdQSKgd7iACQCAMNHSuEtF\nN62Qf9cueeLj5W9okBQYRMfkLXWcDugZ9owCABAG/C0t2nbn7dr5qVXSUUcr+7q58qWkKG54KntE\nEdbYMwoAQIjz+/0qfeRh1ax7RwmHjFPGZVfIGxurMXlLlZaWrNLSatcRgR5jzygAACGu4uUXteO1\nVxWXla3MyVPl3cc5o0C4YhgFACCEVb39T5U9/qh8KanKmj5TMYmDXUcC+lSnh+mNMUslHSPJL2m6\ntfadNstOkvQXSc2S/iPpckkTJT0m6cPgyzZYa6f1cW4AACJe3Scfa9s9d8mbkKCsGTMVm5rqOhLQ\n5zocRo0xEyUdZK2dYIwZJ+luSRPavOR2Sd+11hYaYx6VdJqkOkmvW2vP6a/QAABEuob8L1V00wp5\nPB5lTp6m+Kxs15GAftHZYfqTJT0lSdbajZJSjDFJbZYfaa0tDH5dKolf2QAA6KXG7WUqWLZELfX1\nyrjsSiWOO9R1JKDfdDaMZkgqa/O4VNKo1gfW2ipJMsaMknSqpBckeSQdZox5xhiz2hhzSt9GBgAg\ncjXX1Khw6WI1V+5Q2rnnK/nbR7uOBPSr7lY7eRQ4d3Q3Y0y6pJWSJllrK4wxVtLvrbWPGWPGSFpl\njBlrrW3a3zdNSUmUzxfT3ezopbS0ZNcRgH7Hdo5w0tzQoI8W36hd27Yq8yc/1tcuPLtL67GdI5x1\nNowWKbB3tFWmpK2tD4wxQxTYG/rf1tpXJclaW6TABUyy1m4yxmyTlCVpy/7epKKirkfh0XP00iEa\nsJ0jnPhbWrT11ptU88lGJX/7aA0+42dd2n7ZzhHuOjtM/4qksyXJGDNeUqG1trbN8sWSllprX2l9\nwhhzgTHmd8Gv0yWlSyoUAADYp0Cp/UOqeXedEg4Zp5G/vkIeL+2LiA6ezl5gjJkv6TsK1DdNljRe\nUqWklyVVSHqrzcsfkvSIpIcVuJgpRtIfrLUvdfQeJSVV/o6Wo+/xmzSiAds5wkX5iy+o7IlHFZeV\nrZw587rVJcp2PvDS04d0Oj+h6zo9Z9RaO2+Ppza0+XrQflY7s8eJAACIIlVv/1NlT1Bqj+jFMQAA\nAByp/fgjSu0R9RhGAQBwoP7LLdp68w2BUvsp0ym1R9RiGAUAYIA1bi9T4fKlX5XaHzLOdSTAGYZR\nAAAGULtS+/MotQcYRgEAGCAtu3ap8Mbl2rVtq1K+/wOlfP8HriMBzjGMAgAwAPwtLdp2522q/+xT\nJX/7aI045zzXkYCQwDAKAEA/o9Qe2D9+EgAA6GcVL72oHa/9Q3FZ2cqcPFXe2FjXkYCQwTAKAEA/\nqnqLUnugI53egQmRpWBxnuo2fiwrKXHcYcqeles6EgBErNqPP9K2e1tL7WdRag/sA3tGo0jB4jzV\nffKR5PdLfr/qPvlIm3KvVf2Wza6jAUDE2bvUPst1JCAkMYxGkbqNH+/1XFNFhYpuXO4gDQBErsay\nUhUuX0KpPdAFDKMAAPSh5poaFS5boubKSkrtgS5gGI0iieMO2/tJr1cjL7ls4MMAQARqV2p/6mmU\n2gNdwDAaRbJn5cqXkrL7sSc+XmppUclfH1RjRYXDZAAQ/vwtLdp2R7DU/uhjNOLsc11HAsICw2iU\nyZwyXb6UFMUNT1V27lylnHa6GrdtU8HC+Wrcvt11PAAIS36/XyV/fUg17wVL7S+9nFJ7oIs8rgNI\nUklJld91hmiTlpas0tJq+f1+bX/mKZU/t1K+ESOUM2uOYtPSXMcD+kTrdg70t/IXn1fZE48pLitb\nOXPmDWiXKNv5wEtPHxIS81Ok4Ne2KOfxeDTip2dp+E/PUlNZmfIXzteu4m2uYwFA2Kh6a63KnniM\nUnughxhGIUka/qMzNeLsc9VUUa78hQvUUFTkOhIAhLzajz7UtnvvljcxkVJ7oIcYRrFb6mmnK+0X\nF6q5cocK8uaroSDfdSQACFn1X27R1ltuDJTaT55GqT3QQwyjaCfllO8r/aKL1VxdrfxF16v+yy2u\nIwFAyNldat/QQKk90EsMo9jLsIknaeQll6mltlYFi67Xzk2bXEcCgJDRXFOjgmWLKbUH+gjDKPZp\n6AknKuOyK9Syc6cKlyzUzk8/dR0JAJxr2bVLhTcsU+O2bYFS+1NOdR0JCHsMo9ivIcdO0KirJqml\nsVEFyxapbuMnriMBgDO7S+0//4xSe6APMYyiQ8lHHa3MqyfL39SkwhVLVfvxR64jAcCAa1dqP+5Q\nSu2BPsRPEjqV9K3xypw8TWppUdGKpapZ/4HrSAAwoCpeekGVq/6huKxsZV4zRd7YWNeRgIjBMIou\nSfr6N5Q5dYbk9arophWqee9d15EAYEDsLrVPTVXWjFmU2gN9jGEUXTb48COUNX2mPD6fim69SdXv\n/Nt1JADoV+1K7afPUmxKiutIQMRhGEW3JB4yTtkzrpM3NlZbb7tFVW//03UkAOgX9V9uUdHNwVL7\nKdMptQf6CcMoui3h4IOVNXO2vAkJ2nbXHapcs9p1JADoU62l9v5dDcq4/EolmkNcRwIiFsMoeiRh\nzBhlz5ot7+DBKr73Lu14Y5XrSADQJ/YqtT+KUnugPzGMoscGHThaOdfNUUzyEJU8cJ8qXv2760gA\n0CvtSu1/QKk9MBAYRtEr8dk5ys6dq5ihw1T6yEMqf+kF15EAoEf8LS3aesetX5Xa/5xSe2AgMIyi\n1+IzM5Uze658Kakqe/xRbX9upetIANAtgVL7B1X73ruU2gMDjJ809Im4kRnKmT1PvhEjtP3pJ1X2\n9BPy+/2uYwFAl1S8+LwqV70WLLWfSqk9MIAYRtFnYtPSlJM7T7Fp6Sp/7lmVPf4oAymAkFf11lqV\nPfl4m1L7RNeRgKjCMIo+FTt8uHLmzFNsRoYqXn5RpX97mIEUQMii1B5wz9fZC4wxSyUdI8kvabq1\n9p02y06S9BdJzZL+I+lya62/o3UQ+XzDUpSTO08Fixdqx6t/l7+xSekXXsT5VwBCCqX2QGjocDow\nxkyUdJC1doKkyySt2OMlt0s621p7gqRkSad1YR1EAd/QocrJnav4nANU+cYqFd9/j/wtLa5jAYAk\nSu2BUNLZrqqTJT0lSdbajZJSjDFJbZYfaa0tDH5dKml4F9ZBlIhJTlb2rNmKH/01Va1ZrW133SF/\nc7PrWACiHKX2QGjpbBjNkFTW5nGppFGtD6y1VZJkjBkl6VRJL3S2DqJLTFKSsmfmatDYg1T9r7e0\n9Y5b5W9qch0LQJSi1B4IPd09ic+jwHmguxlj0iWtlDTJWlvelXUQXWISE5V97SwlmENU887/qei2\nm9XS2Og6FoAo077U/lhK7YEQ0dkFTEUK7OlslSlpa+sDY8wQBfaG/re19tWurLMvKSmJ8vliupoZ\nfSQtLXkA3y1ZI/70W33y5wWqfO9dbb/zFo2bmytvXNwAZkA0GtjtHKHK7/dr0213qva9dzX0v47Q\nYbNnRFSXKNs5wllnw+grkv4g6XZjzHhJhdba2jbLF0taaq19pRvr7KWioq77ydEraWnJKi2tHvj3\nvXqqGm++QRXr3tUHv/1fZU6ZJm98/IDnQHRwtZ0j9JS/8JzKXnxJcVnZGnHFNdq+o15SvetYfYLt\nHOHO09kLjDHzJX1HgfqmyZLGS6qU9LKkCklvtXn5Q9baO/dcx1q7oaP3KCmp4jD+AHP54dXS2Kit\nt92s2vffU8Ih45Q1dYa8gwY5yYLIxl/SkKSqf67VtrvvkC81VTnz/l/EdYmynQ+89PQhnc5P6LqQ\n+I/JMDrwXH94+ZuatPWOW1Wz7h0NOuhgZU2fqZiEBGd5EJlcb+dwr/ajD1W4Yqm88fHKmfObiOwS\nZTsfeAyjfYsWcjjh8fk06spJSj76WNV/9qkKl+SpubbDszkAoFsotQfCA8MonPHExCjj8is1ZMIJ\nqv9ikwoWL1RzNb/dA+i99qX2V1FqD4SwTm8HCvQnj9erkZf8Wh6fT5Vvvq78Rdcre9Zs+YYMcR0N\nQJhqW2o//LzzVZIyTCWff+o6Vr+pqEhSeXmN6xj9ZvToMYqJoXEnkjGMwjmP16v0iy6WxxejHa/9\nQwV5CwID6bBhrqMBCDPtS+1/qB1jxmh63kolDk13HQ09UFdZouW5Z2rs2INdR0E/YhhFSPB4PEo7\n/5fy+GJV8cpLys+br+xZcxSbmuo6GoAwsXep/Tmq+uJzJQ5NV1IK54sCoYpzRhEyPB6PRpxznlJP\n/5Eai4tVsHC+GstKXccCEAb8fr9K/vqgat97VwnjDtXISy+Tx8tfcUA44CcVIcXj8WjEWWdr+E9+\npsayUuUvnK9dxcWuYwEIcRUvPq/KVa8pLjtHmddMjai7KwGRjmEUIWn4j3+iET8/R03l5crPm69d\nW4tcRwIQoqr+uVZlTz4uX2pqoLM4MdF1JADdwDCKkJX6wzOUdu75at6xQ/l5C9RQWOA6EoAQU/vR\nh9p2393yJiYqa8asiLu7EhANGEYR0lJO/YHSL7xIzVVVKsi7XvVfbnEdCUCIqN+yuX2pfSYXKQHh\niGEUIW/YSd/TyIsvVXNtjQoWLVT95i9cRwLgWGNpqQpXLKXUHogADKMIC0NPnKiMX1+ulp11Kli8\nUDs//8x1JACONFdX7y61TzvvAiUf9W3XkQD0AsMowsaQ447XqCuuVktDgwqWLFKd/Y/rSAAGWEtD\ngwpvXK7G4kCpfcop33cdCUAvMYwirCQffYxGXT1Z/qZGFS5brLpPPnYdCcAAaVdqf0yg1B5A+GMY\nRdhJHn+kMq+ZKrW0qHDFUtV+uN51JAD9zO/3q+ThB1X7/ntKGHeoMi69nFJ7IELwk4ywlPSNbypz\n6gxJUtGNK1Tz/nuOEwHoTxUvPq/K178qtff4uJs1ECkYRhG2Bh9+hLKmXSt5vSq65UZVr/s/15EA\n9IPKtWt2l9pnz6DUHog0DKMIa4mHHqbsa6+TNzZWW2+7RVX/ett1JAB9qPajD1V8/z27S+19wyi1\nByINwyjCXsLBRlkzc+WNj9e2O29T5do1riMB6AOU2gPRgWEUESFhzFhlz5ojb2Kiiu+9SzvefN11\nJAC90FhaqsLlSwKl9ldQag9EMoZRRIxBo0cr57o5ihmcpJL779WO1151HQlAD+wuta+qUtovLlDy\nkZTaA5GMYRQRJT7nAGXPnquYoUNV8vCDqnjlJdeRAHTDXqX236PUHoh0DKOIOPGZWcrJnSdfSopK\nH31E259/1nUkAF1AqT0QnRhGEZHiMjKUnTtPvtTh2v7UEyp75in5/X7XsQDsR9tS+8RDD6PUHogi\n/KQjYsWlpytnzjzFpqWp/NlnVPbk4wykQIgqf+G53aX2oyZNodQeiCIMo4hoscNHKDt3nmJHZqji\nxedV+ugjDKRAiKlcu0bbn3pCvtThlNoDUYhhFBEvNjVVObPnKi4zUzv+/rJKHn5A/pYW17EASKr9\ncEObUvuZlNoDUYhhFFHBN3SYsnPnKi47R5WrXlPxA/cykAKO1W/ZrKJbKLUHoh3DKKKGL3mIcq6b\no/gDDlTV6jdVfM9dDKSAI1+V2u+i1B6IcgyjiCoxSUnKvm62Bo0Zo6q31mrbnbfJ39TkOhYQVSi1\nB9AWlysi6sQkDlb2zFwVLl+q6n//S/7GJo26ahJX7wIDoKWhQYU3LAuU2p92OqX2QB8xxoyWtEHS\nO22efk/SMklPSXrNWpsbfG2spBslHSGpKfjPJdba/IHM3Iq/fRGVvIMSlDVjlgpvWKaa99ap6OYb\nNGrSZHlj41xHAyKWv7k5UGq/6XMlH3OcRpx1tutIgDNrzjzr75K+F3z4jxNWPtkXv5lttNae1PYJ\nY8zLkl6WFNvm6QskNVlrjw++5mJJ10ia1903NMZ4rLW9qqlhGEXU8sbHK2vqDBXdfINq13+gohtX\nKHPyNHnjGEiBvrZ3qf1llNojagUH0VPaPHXKmjPPKpB05gkrn3y3j9/uLElnK7AXtNVQScmtD6y1\n97V+bYy5SNJUSS2SllhrHzXGnCvpWgX2oK6z1s4wxvxe0tckfc0Y811Jf5J0gqQYSTdaax/pakCG\nUUQ1b3y8MqdM09ZbblLt+g9UuGKpsqbOkDc+3nU0IKKUv/CcKt9YpficHI26ZiqnxSCirTnzrDxJ\nHd3P9sB9PJcl6d/BoXRfHjth5ZO53c1ira01xuz59IOSLjHGbJT0gqQnrLVrjTHJkv6fpP+SNEjS\nfcaY5yX9WdI3rLV1xpiVweHTLynWWvsdY8yJkg6w1k40xsRLWmeMecpa29CVjHwaIOp5Y+OUec1U\nbb39FtW8u06FyxYrc9q1iklIcB0NiAiVa1fvLrXPmj6Tny2g/xxijFnV5vEr1tr5kjxtX2StLZc0\n3hhzgqRTJf3VGHO3pOcVONTfIKlB0k+NMeMlfWqtrQuu/rqkbwW//nfw3xMkHdvmvT2SRkna3JXQ\nDKOAJI/Pp1FXTtK2u+9Q9b//pcKlecqaMUsxiYNdRwPCWqDU/l55Ewcra8YsSu0RFYJ7MPe7F3Mf\nh+klqVC9P0z/nz3PGQ1qd06nMSZOgXNG10haY4y5U4Ehc6X2blpqUfthNk7SzuDXjcF/N0i6y1q7\noCehOWEHCPL4fMq4/ColHzdB9Zs2qWBxnppralzHAsJW/eavSu2zpk5XfGam60hASAherFTY5qnC\nE1Y+md0P54u28uzx+C5JV7R5nCPpc0kbFdi7OtgYM8gY84okK+lgY0xS8LUT1f6KfUn6l6QfG2M8\nwfVWdCccwyjQhsfrVcall2vIid9Rw5bNyl90vZqqq1zHAsJOY2mpCle0ltpfrYSD9zpnDYh2Zyow\nkBYGv+4Le+4BzTHGbFDg4qJLjDEbjDFHKHAx0qnGmDXGmFcl/UbSpOCh+N9KelXSKkl3BJ/LlfSS\nMeZNSe9aa9e2fT9r7VvB178l6Q3tPax2aM9JeS/GmKWSjgm+4XRr7Tttlg2SdLukQ6213w4+911J\nj0n6MPiyDdbaaR29R0lJVa8qAdB9aWnJKi2tdh0jZPlbWlTy1wdVueo1xWVmKnvmbPmGDXMdC93E\ndu5Gc3W1vlzwZzUWb1PaBb9Uysl7Ho0cOJ9//qnm3f62klK41Wg4qqko1Pwrj9XYsQe7jtJOevqQ\nTucndF2H54waYyZKOshaO8EYM07S3QqcpNpqoQInrx66x6qrrLXn9mlSYAB5vF6lX3CRPL5Y7fj7\ny8rPW6Ds6+YoNoXz3YCO7FVq73AQBRAeOjtMf7ICrf2y1m6UlNLmnAEpUI767D7W4zcGhD2Px6O0\nc3+hlB+eocbibSpY+Bc1bi9zHQsIWZTaA+iJzobRDElt//YtVeBSfUmB7irtPXj6JR1mjHnGGLPa\nGMOvxQhbHo9HI846W6k//okaS0uVf/187SopcR0LCDmBUvsHKLUH0G3drXbyaI+TY/fhU0m/t9Y+\nZowZI2mVMWastbZpfyukpCTK54vpZhT0VlpacucvgiQp/fJfqWBYkrY88JCKFi/Q4X/8vRKzOQct\nHLCdD4z8Rx9X5Ruva/DXRuuI386TLzHRdSRJUkVFUucvQkhLTU3i5zjCdTaMFimwd7RVpqSte7ym\n3XBqrS1S4AImWWs3GWO2KXBXgS37e5OKirr9LUI/4cKO7ouf+H2lNTSr9NFHtH7e/yj7ujmKz2Qg\nDWVs5wOjcu1qFT/0V/lSh2vk5OmqqG2WakPjv3t5OfVs4a68vIaf4wjX2TGUVxS4n6mCDfyFwUPz\nbbU7TG+MucAY87vg1+mS0tW+SwsIWymnnqa0C36p5qoqFeQtUEN+vutIgFO1H65X8X33UGoPoMc6\n3DNqrX3LGLPOGLNWUrOkycaYiyVVWmufDnZTZUs6INhjtUSBvaIPG2PWSIpRoLdqv4fogXCTcvIp\n8sT4VPLgfcpftEDZ1+Zq0OjRrmMBAy5Qan+TPF4vpfZAezGSxvbx9/xcgVlsv4wxkyX9UoE7IiVI\n+m9r7T+68s2NMd+Q9DNr7e97mbPbOj1n1Fo7b4+nNrRZtr+Lk/qqvBUIScMmflcen0/F996lgsXX\nK2vGLCWMPch1LGDA7CotUeHyQKn9qKsnU2oPtDf26LN+95/Eoel98s3qKkv07yf/cIgCd0PaJ2PM\naEmXSzrKWttsjDEKdMF3aRi11n4g6YM+iNtt3Jse6KGhx58gj8+nbXfdroIli5Q9YyZ/ISMqNFdX\nq3DZYjVXVyntgl8q+cijXEcCQk7i0PSBvtnCUEmDJMVLqrPWWknfDbYa/UlSvaQdks6V9KikJdba\n1caYBEkfS7pMgaPZ5xhjPlOg2vP44DpnKHD9z2MK7HV9U9KJ1tqTgrf+PFKBvcG3WGvv625wejeA\nXhhyzLEaddUk+ZsaVbB0keo2fuI6EtCvviq1L6bUHgghwT2b/5b0hTHmHmPMOcYYnwJD6oXW2pMk\nVUr6gaQnJP04uOr3Jb2s9qcAfE3SfdbaCZJSJH1dgVuIPmKt/a4CA6/fGJMi6XRr7fGSTpAU25Ps\nDKNALyUf+W1lTpoitbSocPkS1X64ofOVgDBEqT0Q2qy1F0uaKOl9SbMVuBB9u6TbjTGvK3Azo1QF\nblh0WnC1n0h6fI9vVWWtbb2te4ECA+04Sa33pF8pyWOtrZBkjTFPSzpP0v09yc0wCvSBpG9+S5lT\npkmSim5crpoP3necCOhblNoDoc8YE2+t3WitXS7pGEk5CtzKfXJwj+YzkmStrZRUGDyv9DhJr6l9\nO9KeF557tJ+ueWvt6ZL+IOmb2vddOTvFJwnQRwYf8XVlTbtW8npVdPMNqn53netIQJ8pf/5ZVb7x\nuuJzcjTqmqny+LjkAAglxpjLJd1tjGkdKocpMEAmS8o3xgxTYM9ofHD5U5L+R9I/rbUtXXiLzyW1\nniD+w+B7HmiMmWatfc9amytpeE+y82kC9KHEQw9T1oxZKly+VFtvvUm6/ColH32M61hAr1SuXa3t\nTz8pX+pwZU2fqZiEBNeRgJBXV9l3t47u4ve6W9Ihkt42xtQocP7mVEnHKnB4/XNJCyX9zhizUoG9\npDcocJheCuz19Lf5ui2/pOWSHjXG/FyBc1ObFbg50nHGmPMUuLDprp78+fa8r7wTJSVVnd1iFH2M\nO9P0r52ff6bCZYvVUl+vjF9friHHHe86UlRiO++92g/Xq3DFMnkHJShn7m/Crkv0888/1bzb3x7o\nq5rRR2oqCjX/ymM1duzBrqO0k54+pLP5yUnPaH8yxhwmaZi19p/GmPMlTbTWXt0X35s9o0A/SBh7\nkLJnzVbBkkXadved8jc1aeiJE13HArqlfan9jLAbRAGHmtVBJ2iYqpZ0mzHGr8Cf79K++sYMo0A/\nGTT6a8q+brYKlyxS8X33yN/UpGEnfc91LKBL9i61D609UwAGlrU2X9KJ/fG9uYAJ6EeDDjhQ2blz\nFDNkiEoeekAVf3/ZdSSgU+1K7c+/kFJ7AP2KYRToZ/FZ2crJnauYYcNU+re/qvzF511HAvYrUGq/\nlFJ7AAOGYRQYAHGjMpWTO0++1FSVPfGYtj/7jPx+rttDaPE3N2vr7beoftMmJR9LqT2AgcEwCgyQ\nuJEjlTN7nmJHpGn7M09p+1NPMJAiZPj9fpU89IBqP3hfiYceroxLKLUHMDD4pAEGUOyINGXPnqvY\nkSNV/sJzKnvsbwykCAnlzz+ryjdbS+2nUGoPYMAwjAIDLDZ1uHJy5yluVKYqXnlJpX99UP6Wrtz8\nAugfu0vth1NqD2DgMYwCDviGDVN27lzFZWVrx2v/UMmD9zGQwonaDetVfN898iYOVtb0WfINS3Ed\nCUCUYRgFHPENGaKc3LmKP+BAVb75horvvYuBFAOqfvMXKrr1JnliYii1B+AMwyjgUExSkrJnzdag\nr41R1T8fDmhaAAAUuklEQVTXattdt8vf7Oxub4gigVL7pfLv2qWMK66m1B6AMwyjgGMxgwcra2au\nBh10sKr/9ba23n6L/E1NrmMhgjVVV+0utU8//0Iljz/SdSQAUYxhFAgBMQkJyp4xSwmHjFPNundU\ndOtNamlsdB0LEailoUFFNywLlNr/8AwNo9QegGN0d+xDc3OzNm/e5DpGv6qoSFJ5eY3rGP1m9Ogx\niomJcR2jW7yDBilr2rUqummFat9/T0U3rVDmNVPljYtzHQ0RglJ7AKGIYXQfNm/epOl5K5U4NN11\nFPRAXWWJlueeqbFjw+8cOG98vDKnTtfWm29U7Yb1KrphmTKnTJc3Pt51NIS5fZbaezyuYwEAw+j+\nJA5NV1JKlusYiELe2DiNumaqtt5+i2rfe1eFy5coa9oMeQfR/Yie+6rU/gBK7QGEFM4ZBUKQNzZW\nmVddo6SjjtZO+x8VLF2s5ro617EQpirXUGoPIHQxjAIhyuPzadQVVyn52ONU//lnKliSp+aayD3P\nF/2jdsN6Fd9/j7yDByt7xiz5hg1zHQkA2mEYBUKYJyZGGb++QkOOP1ENm79QweKFaq6udh0LYaJd\nqf2UGYobRak9gNDDMAqEOI/Xq5EXX6qhE09SQ/6Xys9boKbKStexEOIotQcQLhhGgTDg8XqV/stf\nadgp39euokLl581XY0WF61gIUZTaAwgnDKNAmPB4PEo77wKlnHa6GrdtU8HC+Wrcvt11LISYloYG\nFa2g1B5A+GAYBcKIx+PRiJ+fo9QfnanG0pLAHtLSUtexECL8zc3aetvNqv9ik5KPm0CpPYCwwDAK\nhBmPx6MRPz1Lw396lprKypS/cL52FW9zHQuOBUrt71ft+g8CpfYX/5pSewBhgWEUCFPDf3SmRpxz\nnpoqypW/cIEaiopcR4JD5c+tVOWbb1BqDyDsMIwCYSz1Bz9U2i8uVHPlDhXkzVdDQb7rSHCgcs1q\nbX/mKUrtAYQlhlEgzKWc8n2lX3Sxmqurlb/oetV/ucV1JAwgSu0BhDuGUSACDJt4kkZecplaamtV\nsOh67dy0yXUkDID6zV+o6JYbKbUHENYYRoEIMfSEE5Vx2RVq2blThUsWauenn7qOhH60qyRYat/Y\nSKk9gLDGMApEkCHHTtCoqyappbFRBcsWqW7jJ64joR9Qag8gknR6uaUxZqmkYyT5JU231r7TZtkg\nSbdLOtRa++2urAOgfyUfdbQ8MT4V3XqTClcsVeaU6Rp82OGuY6GP7C61LylW6uk/otQeQNjrcM+o\nMWaipIOstRMkXSZpxR4vWSjp391cB0A/S/rWeGVOnia1tKhoxVLVrP/AdST0gT1L7Yf/7OeuIwFA\nr3V2mP5kSU9JkrV2o6QUY0xSm+XzJD3bzXUADICkr39DmdOulbxeFd20QjXvves6EnqhXan9YZTa\nA4gcnQ2jGZLK2jwulTSq9YG1tlbSnp+GHa4DYOAMPuxwZU2fKY8vcNi++p1/d74SQlLbUvtMSu0B\nRJDuXsDkUeA80P5eB0AfSTxknLJnXCdvbKy23naLqt7+p+tI6KbKNW+2K7X3DqLUHkDk6OxX6yIF\n9nS2ypS0dY/X7DlodmWddlJSEuXzxXQSZeBUVHBWQbhLTU1SWlqy6xihI228Ukb8Xh/9/k/adtcd\nSkqI1chTTnadakCE+3ZQse5dFd9/r3zJSfqvP/5WidnZriOFFT7Pwx+f55Gvs2H0FUl/kHS7MWa8\npMLgofm29jxM35V12qmoqOtG5P5XXl7jOgJ6qby8RqWl1a5jhJaUDGXNmq2CJXn67IabVLWjRsMm\nnuQ6Vb9KS0sO6+2g/otNys/LkycmRqMmT1dt/FDVhvGfxwU+z8Mfn+eRr8PD9NbatyStM8aslbRM\n0mRjzMXGmJ9KkjHmVUkvSTrcGLPBGHPpvtbp3z8CgK4adMCByrlujmKSh6jkgftU8erfXUfCfuwq\nKVHhikCp/agrr1bCQZTaA4hMnZ4Bb62dt8dTG9os22fB3T7WARAi4rNzlJ07VwWLF6r0kYfkb2pU\n6mmnu46FNr4qta9W+oW/UtK3KLUHELm4AxMQheIzM5Uze658Kakqe/xRbX9upetICNqr1P6k6Di3\nF0D0YhgFolTcyAzlzJ4n34gR2v70kyp7+gn5/RRfuNS21H7IccdTag8gKjCMAlEsNi1NObnzFJuW\nrvLnnlXZ448ykDqyZ6n9yIsvpdQeQFRgGAWiXOzw4cqZM0+xGRmqePlFlf7tYQZSB3aX2h9wIKX2\nAKIKwygA+YalKCd3nuKysrXj1b+r5MH75W9pcR0rauwutR8xQlnTr6XUHkBUYRgFIEnyDR2qnOvm\nKD7nAFW+sUrF99/DQDoAatZ/oOL775V38GBlz5gl39BhriMBwIBiGAWwW0xysrJnzVb86K+pas1q\nbbvrDvmbm13Hilj1X2zS1ltvkicmRllTZyguY5TrSAAw4BhGAbQTk5Sk7Jm5GjT2IFX/6y1tveNW\n+ZuaXMeKOJTaA0AAwyiAvcQkJir72llKMIeo5p3/U9FtN6ulsdF1rIjRrtT+gosotQcQ1RhGAeyT\nd1CCsqbPVOKhh6n2vXe19eYb1NK4y3WssEepPQC0xzAKYL+88fHKnDpDiUf8l2o3rFfRiuVqaWhw\nHStsUWoPAHtjGAXQIW9cnDInT9Pgb35LdZ98pMIVS9VSX+86Vtjx+/0qfvC+QKn94UdQag8AQQyj\nADrljY1V5tWTlXTkUdr5n40qWLZYzTt3uo4VVsqfW6mq1W8GSu0nTabUHgCCGEYBdInH59OoKycp\n+ZhjVf/Zpypckqfm2lrXscJC5eo3KLUHgP1gGAXQZZ6YGGVcdqWGTDhB9V9sUsHihWqurnYdK6TV\nrP9AxQ/cR6k9AOwHwyiAbvF4vRp5ya819DvfVcOXW5S/6Ho1VVW5jhWS2pXaT7uWUnsA2AeGUQDd\n5vF6lX7RxRp28inaVViggrwFatqxw3WskLKruLh9qf3Yg1xHAoCQxDAKoEc8Ho/Szr9QKT84Tbu2\nFil/4Xw1lm93HSskNFVRag8AXcUwCqDHPB6PRpx9nlLP+LEaS4pVsHCBGstKXcdyqqWhQYUrlqqx\ntIRSewDoAoZRAL3i8Xg04mc/1/Cf/EyNZaXKXzhfu4qLXcdyorXUvmHzF5TaA0AXMYwC6BPDf/wT\njfj5OWoqL1d+3nzt2lrkOtKAotQeAHqGYRRAn0n94RlKO+98Ne/Yofy8BWooLHAdacCUP/sMpfYA\n0AMMowD6VMr3f6D0C3+l5qoq5ectUP2XW1xH6neVq9/Q9pVPU2oPAD3AMAqgzw076WSNvPhStdTW\nqmDRQtVv/sJ1pH5Ts/59Su0BoBcYRgH0i6EnTlTGry9Xy846FSxeqJ2ff+Y6Up8LlNrfTKk9APQC\nwyiAfjPkuOM16oqr1dLQoIIli1Rn/+M6Up9pX2o/iVJ7AOghhlEA/Sr56GM06urJ8jc1qnDZYtV9\n8rHrSL3WrtT+wouU9K3xriMBQNhiGAXQ75LHH6nMa6ZKLS0qXLFUtR+udx2px/Yqtf8upfYA0BsM\nowAGRNI3vqnMqTMkSUU3rlDN++85TtR9/uZmbb31JkrtAaAPMYwCGDCDDz9CWdNnSl6vim65UdXr\n/s91pC7z+/0qfuA+1W5YT6k9APQhhlEAAypx3KHKvvY6eWNjtfW2W1T1r7ddR+qS8mefUdUaSu0B\noK8xjAIYcAkHG2XNzJU3Pl7b7rxNlWvXuI7Uoco3KbUHgP7CMArAiYQxY5U9a468iYkqvvcu7Xjz\nddeR9qlm/fsqfvA+eZOSKLUHgH7AMArAmUGjRyvnujmKSUpSyf33asdrr7qO1M7OTW1K7afOoNQe\nAPoBwygAp+JzDlB27lzFDB2qkocfVMUrL7mOJClQal9EqT0A9DuGUQDOxWdmKSd3nnwpKSp99BFt\nf/5Zp3l2l9rXUGoPAP2NYRRASIjLyFB27jz5Uodr+1NPqOyZp+T3+wc8R7tS+zN+TKk9APQzhlEA\nISMuPV05c+YpNi1N5c8+o7InHx/QgbRdqf2E4zX8p2cN2HsDQLTqtCjPGLNU0jGS/JKmW2vfabPs\nFEl/ltQs6QVr7f8aY74r6TFJHwZftsFaO62vgwOITLHDRyh79n+rYNH1qnjxefmbmpR27i/6vWB+\nr1L7X1FqDwADocNh1BgzUdJB1toJxphxku6WNKHNS5ZLOlVSkaQ3jDFPKDC0vm6tPaefMgOIcLEp\nKcqZPVcFixdqx99flr+pUenn/1Ieb/8dzNm+8mlK7QHAgc4+2U+W9JQkWWs3SkoxxiRJkjFmjKRy\na22htdYv6QVJ3+vPsACih2/oMGXnzlVcdo4qV72m4gfulb+lpV/eq/LNN1T+7DOU2gOAA50NoxmS\nyto8Lg0+17qstM2yEkmtJXyHGWOeMcasDh7KB4Bu8yUPUc51cxR/4GhVrX5Txffc1ecDKaX2AOBW\nd49DdXQCVeuyTyX93lr7WHDv6SpjzFhrbdP+VkxJSZTPF9PNKP2noiLJdQT0UmpqktLSkl3HQF9I\nS9aIv/xRH//xf1X11lrF+aSDZ0yTt4uH0TvaDqrtp/rstlvk9fl0xG9/o+RDTF+lRojg8zz88Xke\n+Tr7NC/SV3tCJSlT0tbg14V7LMuWVGitLVLgAiZZazcZY7ZJypK0ZX9vUlFR183Y/au8vMZ1BPRS\neXmNSkurXcdAHxo59Vo1LV+qstVrVV9Tr1FXTer0vM60tOT9bge7iouVP//Patm1S5mTp6k+dZTq\n2WYiDp/n4Y/P88jX2WH6VySdLUnGmPEKDJu1kmSt3SJpiDHmQGOMT9IZkl4xxlxgjPldcJ10SekK\nDK4A0GPeQQnKmjFLCeMOVc1761R08w1qadzVo+/VrtT+l79S0je/1cdpAQBd1eEwaq19S9I6Y8xa\nScskTTbGXGyM+WnwJZMk/VXSm5IesdZ+JmmlpCONMWskPSNpUkeH6AGgq7zx8cqadq0SDz9Ctes/\nUNGNK9Syq3sDaUt9fftS+4kn9VNaAEBXdHrSlbV23h5PbWizbLXaVz3JWlsj6cw+SQcAe/DGxSlz\nyjRtveUm1a7/QIUrlipr6gx54+M7Xdff1KSiW2+m1B4AQgh3YAIQdryxccq8ZqqSxh+pnRs/CRxy\n37mzw3X8fr+KH7xPdR9Sag8AoYRhFEBY8vh8GnXlJCUffYx2fmpVuDRPzXW1+319oNR+dbDUfgql\n9gAQIhhGAYQtj8+njMuv0pDjjlf9pk0qWJyn5pq9r57e8ebrKn/2GcWOSAuW2g9ykBYAsC8MowDC\nmsfr1chLL9OQE7+jhi2blb/oejVVV+1eXvPB+yp58H55k5KURak9AIQcjlMBCHser1cjL7pEHp9P\nlate0xdzrpO/sVG2dXlsrLKmzlBcRkaH3wcAMPDYMwogIni8XqVfcJF8qcPl37VL8vt3/+OJjeMc\nUQAIUQyjACKGx+NRU0X5Xs+31Nao6MblDhIBADrDMAoAAABnGEYBRJTEcYft9ZwvJUWZU6Y7SAMA\n6AzDKICIkj0rV76UlN2PfSkpGpO3VIMOHO0uFABgvxhGAUSczCnT5UtJUdzwVPaIAkCI4/JSABFn\n0IGjNSZvqdLSklVaWu06DgCgA+wZBQAAgDMMowAAAHCGYRQAAADOMIwCAADAGYZRAAAAOMMwCgAA\nAGcYRgEAAOAMwygAAACcYRgFAACAMwyjAAAAcIZhFAAAAM4wjAIAAMAZhlEAAAA4wzAKAAAAZxhG\nAQAA4AzDKAAAAJxhGAUAAIAzDKMAAABwhmEUAAAAzvhcBwDgRnNzszZv3uQ6Rr+qqEhSeXmN6xj9\nZvToMYqJiXEdAwB6hWEUiFKbN2/S9LyVShya7joKeqCuskTLc8/U2LEHu44CAL3CMApEscSh6UpK\nyXIdAwAQxThnFAAAAM4wjAIAAMAZhlEAAAA40+k5o8aYpZKOkeSXNN1a+06bZadI+rOkZkkvWGv/\nt7N1AAAAgFYd7hk1xkyUdJC1doKkyySt2OMlyyWdJel4SacaYw7twjoAAACApM4P058s6SlJstZu\nlJRijEmSJGPMGEnl1tpCa61f0guSvtfROgAAAEBbnQ2jGZLK2jwuDT7Xuqy0zbISSaP2s86o3sUE\nAABAJOpuz6inB8s8Cpw7GlbqKktcR0AP8f+u6/hvFb74f9d1/LcKX/y/iw6dDaNF+mpPqCRlStoa\n/Lpwj2XZwdfv6mCdfUpPH9LRkAt023FP/sF1BKDfsZ0jGrCdR77ODtO/IulsSTLGjJdUaK2tlSRr\n7RZJQ4wxBxpjfJLOkPRyR+sAAAAAbXW6R9IYM1/SdxSob5osabykSmvt08aYEyVdH3zp49baJfta\nx1q7oT/CAwAAAAAAAAAAAAAAAAAAAAAAAAhtVCpFOGPMaEkbJL0TfCo++HiSpE2SvlTgQjNJ8ltr\nTx7ojEBvBbfzx6y1327z3O8lXaBADZ1HUoKk+dbap11kBHprj89zj6QmSX9RoELxMkmDJB0uaV1w\nlV9Za/MHPinQPd0tvUd42mitPan1gTHmHkkXKnAzgtOstXXOkgH9xy9pmbX2ZkkyxqRIet8Y86K1\ntsFtNKDHdn+eB2/L/ayk86y1JxljDlSg2eakDr8DEGI66xlFZPqXpINchwAGwO6jP9baCgVuwMHt\niRERrLWbJP1Z0pTgUxztRFhiGI0yxphYST+R9G7wKT68EBWMMYdIGimpwHUWoA+tk3SY6xBAb3CY\nPjocYoxZFfz665IWWGufMcYsl/SiMab1nNFSa+25biICfc4jabox5mxJQxQ4X/p8a22T21hAn0rW\nV+f9A2GJYTQ6/KfNOUaPSfo0+DznjCKS7T5n1BiTIek1BS7+ACLJUfrqSBcQljhMH31yJS0wxiS6\nDgL0M0/wH1lrt0m6X9LvnCYC+pAxZqykayUtdZ0F6A32jEYHf+sX1trNxpgnJP2m7fNABDjcGPNJ\nm8f5ktrWOC2RtN4Yc4+19hMB4an1tKt4STGSrrHWtj0Pms91AAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAQPf4/P8ehukL3xNUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd7f1cedc10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "colors = sns.color_palette()\n",
    "\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are significant differences in the \n",
    "results when evaluating a model using a  traditional cost-insensitive measure such as the \n",
    "accuracy or F1Score,  than when using the savings, leading to the conclusion of the \n",
    "importance of using the real practical financial costs of each context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bayes minimum risk\n",
    "\n",
    "As these methods (RF, LR and DT) are not performing well we then move to use cost-sensitive methods. The first model we used is the Bayes minimum risk model (BMR) [8].\n",
    "As defined in [12], the BMR classifier is a decision model based on quantifying \n",
    "tradeoffs between various decisions using probabilities and the costs that accompany such decisions. \n",
    "This is done in a way that for each example the expected losses are minimized. In  what follows, we \n",
    "consider the probability estimates $\\hat p_i$ as known, regardless of the algorithm used to \n",
    "calculate them.  The risk that accompanies each decision is calculated using the cost matrix described above.\n",
    "In the specific framework of binary classification, the risk of predicting the example $i$ as negative is \n",
    "\n",
    "$$ R(c_i=0|\\mathbf{x}_i)=C_{TN_i}(1-\\hat p_i)+C_{FN_i} \\cdot \\hat p_i, $$\n",
    "and\n",
    "$$ R(c_i=1|\\mathbf{x}_i)=C_{TP_i} \\cdot \\hat p_i + C_{FP_i}(1- \\hat p_i), $$\n",
    "\n",
    "is the risk when predicting the example as positive, where $\\hat p_i$ is the estimated positive \n",
    "probability for example $i$. Subsequently, if \n",
    "\n",
    "$$  R(c_i=0|\\mathbf{x}_i) \\le R(c_i=1|\\mathbf{x}_i), $$\n",
    "\n",
    "then  the example $i$ is classified as negative. This means that the risk associated with the \n",
    "decision $c_i$ is lower than the risk associated with classifying it as positive. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             pre        f1       acc       rec       sav\n",
      "RF      0.492091  0.232172  0.934394  0.151926  0.122358\n",
      "LR      0.529412  0.037677  0.934854  0.019533  0.010948\n",
      "DT      0.251239  0.262626  0.899146  0.275095  0.191770\n",
      "RF-BMR  0.179759  0.282039  0.782493  0.654368  0.432752\n",
      "LR-BMR  0.115987  0.188429  0.717737  0.501899  0.228092\n",
      "DT-BMR  0.251639  0.260847  0.899819  0.270754  0.192115\n"
     ]
    }
   ],
   "source": [
    "from costcla.models import BayesMinimumRiskClassifier\n",
    "ci_models = classifiers.keys()\n",
    "\n",
    "for model in ci_models:\n",
    "    classifiers[model+\"-BMR\"] = {\"f\": BayesMinimumRiskClassifier()}\n",
    "    # Fit\n",
    "    classifiers[model+\"-BMR\"][\"f\"].fit(y_test, classifiers[model][\"p\"])  \n",
    "    # Calibration must be made in a validation set\n",
    "    # Predict\n",
    "    classifiers[model+\"-BMR\"][\"c\"] = classifiers[model+\"-BMR\"][\"f\"].predict(classifiers[model][\"p\"], cost_mat_test)\n",
    "    # Evaluate\n",
    "    results.loc[model+\"-BMR\"] = 0\n",
    "    results.loc[model+\"-BMR\", measures.keys()] = \\\n",
    "    [measures[measure](y_test, classifiers[model+\"-BMR\"][\"c\"]) for measure in measures.keys()]\n",
    "    results[\"sav\"].loc[model+\"-BMR\"] = savings_score(y_test, classifiers[model+\"-BMR\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print results "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqMAAAE7CAYAAADpQojbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8lGW6//HPTBJSSAgTSCBB6VwiKqtYQCyIXVFc+9p7\n9yxni+cc97dFds+e3XM8u657WAu7Krq2FSv2ig1RlLVgwVtFEENLyJBKQpKZ3x8ziUOATEIm80yS\n7/v14sXMU6/nzjBcua/nvh8QERERERERERERERERERERERERERERERERERERERERERERERGRBPHF\n28DMbgImA2FglnPuve1s8ztginNuupkdBswHPo6uXuac+2HiQhYRERGR3iK9vZVmNg0Y65ybambj\ngTuBqW22mQAcAmyJWbzQOXdGooMVERERkd7FH2f94cBjAM655UDAzHLbbHMj8DO27mWN2+MqIiIi\nIhIvGR0KlMe8LwOKW96Y2YXAK8CqNvtNMLMnzOwNMzsyEYGKiIiISO8TLxlty0fk3lHMrAA4F/gT\nW/eEOuAG59xJwAXAHWbW7u0AIiIiItI3xUsS1xDpHW1RAqyNvp4eXfcmkAmMMbM/OOd+QmQAE865\nFWa2DhjGtr2nrRobm8Lp6Wk7dwUiIiIiSeTz+XQ7YgLFS0ZfAGYDc81sElDqnKsFcM49AjwCYGYj\ngHnOuZ+Y2TlEBj3NNrMioAgobe8kwWBdFy/DG4WFeZSVVXsdRp+iNk8+tXnyqc2TT22efGpzadFu\nMuqcW2xmS81sEdAMXGNmFwCVzrnHYzZtLd8DTwD3m9mbQBpwlXOuqRtiFxEREZEeLu69nM6569ss\nWradbVYSGXmPc64GmJmI4ERERESkd+vsACYRERERkYRRMioiIiIinlEyKiIiIiKeUTIqIiIiIp5R\nMioiIiIintGTkURE4mhubmblyhVJO18wmEtFRU3Szjdy5GjS0vTgERHxhpJREZE4Vq5cwawbF5CT\nX+R1KAlXV7mBm6+byZgx47wORUT6KCWjIiIdkJNfRG5gmNdhiIj0OrpnVEREREQ8o2RURERERDyj\nZFREREREPKNkVEREREQ8o2RURERERDyjZFREREREPKNkVEREREQ8o2RURERERDyjZFREREREPKNk\nVEREREQ8o2RURERERDyjZFREREREPKNkVEREREQ8o2RURERERDyTHm8DM7sJmAyEgVnOufe2s83v\ngCnOuekd3UdEREREpN2eUTObBox1zk0FLgH+vJ1tJgCHEEk8O7SPiIiIiAjEL9MfDjwG4JxbDgTM\nLLfNNjcCPwN80fdHdGAfEREREZG4yehQoDzmfRlQ3PLGzC4EXgFWxWwzpL19RERERERaxL1ntA0f\n35XjC4BzgWOAXTuyj4iIiIhIrHjJ6BoivaMtSoC10dfTo+veBDKBMWb2xzj7bFcgkEN6elonwk4d\nhYV5XofQ56jNk6+vt3kw2LvvNCooyO3zP2PQ59wLanOB+MnoC8BsYK6ZTQJKnXO1AM65R4BHAMxs\nBDDPOfdjMztwR/vsSDBY18XL8EZhYR5lZdVeh9GnqM2TT20OFRU1XofQrSoqavr8z1if8+RTm0uL\ndu8Zdc4tBpaa2SLgT8A1ZnaBmX2/zaatpfjt7ZP4sEVERESkN4h7z6hz7vo2i5ZtZ5uVREbe72gf\nEREREZFt6AlMIiIiIuIZJaMiIiIi4hkloyIiIiLiGSWjIiIiIuIZJaMiIiIi4hkloyIiIiLiGSWj\nIiIiIuIZJaMiIiIi4hkloyIiIiLiGSWjIiIiIuIZJaMiIiIi4hkloyIiIiLiGSWjIiIiIuIZJaMi\nIiIi4hkloyIiIiLiGSWjIiIiIuIZJaMiIiIi4hkloyIiIiLiGSWjIiIiIuIZJaMiIiIi4hkloyIi\nIiLiGSWjIiIiIuKZ9HgbmNlNwGQgDMxyzr0Xs+4y4GKgGfjQOXeNmR0GzAc+jm62zDn3w0QHLiIi\nIiI9X7vJqJlNA8Y656aa2XjgTmBqdF0OcCZwsHOu2cxeNrMDiSStrzrnTu/m2EVERESkh4tXpj8c\neAzAObccCJhZbvR9nXPuyGgimgPkA2sBX3cGLCIiIiK9R7xkdChQHvO+DCiO3cDM/gP4EviHc25l\ndPEEM3vCzN4wsyMTFayIiIiI9C6dHcDkI1KGb+Wc+z0wGjjOzKYCDrjBOXcScAFwh5nFvTdVRERE\nRPqeeEniGiK9oy1KiJTiMbMCYKJz7lXnXL2ZPQsc5Jx7i8gAJpxzK8xsHTAMWLWjkwQCOaSnp3Xh\nMrxTWJjndQh9jto8+fp6mweDuV6H0K0KCnL7/M8Y9Dn3gtpcIH4y+gIwG5hrZpOAUudcbXRdBpFe\nz4nRZQcA95jZ2cA459xsMysCioDS9k4SDNZ16SK8UliYR1lZtddh9Clq8+RTm0NFRY3XIXSrioqa\nPv8z1uc8+dTm0qLdZNQ5t9jMlprZIiLTN11jZhcAlc65x83s18BCM2sCPnDOPRkd4HS/mb0JpAFX\nOeeauvtCRERERKTniXsvp3Pu+jaLlsWsuxu4u832NcDMhEQnIiIiIr2ansAkIiIiIp5RMioiIiIi\nnlEyKiIiIiKeUTIqIiIiIp5RMioiIiIinlEyKiIiIiKeUTIqIiIiIp5RMioiIiIinlEyKiIiIiKe\nUTIqIiIiIp5RMioiIiIinlEyKiIiIiKeUTIqIiIiIp5RMioiIiIinlEyKiIiIiKeUTIqIiIiIp5R\nMioiIiIinlEyKiIiIiKeUTIqIiIiIp5RMioiIiIinlEyKiIiIiKeUTIqIiIiIp5Jj7eBmd0ETAbC\nwCzn3Hsx6y4DLgaagQ+dc9fE20dEREREpEW7PaNmNg0Y65ybClwC/DlmXQ5wJnCwc+5gYLyZHdje\nPiIiIiIiseKV6Q8HHgNwzi0HAmaWG31f55w70jnXHE1M84F17e0jIiIiIhIrXjI6FCiPeV8GFMdu\nYGb/AXwJ/MM593VH9hERERERgc4PYPIRuQ+0lXPu98Bo4Dgzm9qRfUREREREIP4ApjVEejpblABr\nAcysAJjonHvVOVdvZs8CB7W3z44EAjmkp6d1NvaUUFiY53UIfY7aPPn6epsHg737TqOCgtw+/zMG\nfc69oDYXiJ+MvgDMBuaa2SSg1DlXG12XAdxhZhOjyw4A7iFSot/RPtsVDNZ15Ro8U1iYR1lZtddh\n9Clq8+RTm0NFRY3XIXSrioqaPv8z1uc8+dTm0qLdZNQ5t9jMlprZIiLTN11jZhcAlc65x83s18BC\nM2sCPnDOPQnQdp9uvgYRERER6aHizjPqnLu+zaJlMevuBu7uwD4iIiIiItvQE5hERERExDNKRkVE\nRETEM0pGRURERMQzSkZFRERExDNxBzCJiEhynFn6IiM3R6ZlXpldzD+GHeVxRCIi3U89oyIiKeDM\n0hcZtXktPiKPrRu1eS1Xf/0wQ+o3eh2aiEi3UjIqIuKBjFAjhQ1BxtauZr9Nn7X2iMYa0FzHqWsX\nehCdiEjyqEwvItIN/OFmBjTWMrCphvzGGgY21pDfVMPAxmoGNtaQE2ro0HHSws3dHKmIiLeUjIqI\n7IxwmNzmzQxsrI4km1slndXkNW3GT3ib3ZrwU5XRn3WZBVRm5LEpI5fK9FwO2PQJJQ3bluT7hxo4\ndsNiXh00ifq0zGRcmYhIUikZFRHZjnA4TKi2lsbycsLLP+Pgzasoalwd7d2M9HKmh0Pb7gdUp+dQ\nmlXYmmhuysijMiOXTem51KRnE/Zte4fU8ryRXP31wwxorgOgKi2HJ4YeyjFlb7N31ReMq13Ny4P3\n49PcUeDzdffli4gkjZJREemzQg0NNG4sp7G8jMbycprKIn9H3pcR2ry5ddtjY/ar82dS1i8Qk2x+\nl3RWZfSn2Ze2U/E8Ujy99R7RR4qnsz5rEPN2PYH9N33KwRUfMnP9m+xV9RXPF05mU78BXbl0EZGU\noWRURHqtcHMzTRUVrcnld4lm5O/mysrt7ufr14+MwYVkjBtMxuBCKn1w75INNAwczqaMXLb4+3VL\nvOuzBnHLqNO2Whby+XknsCfLc0dwdNk7jKlbw6WrF7AoMJF3AnsQ2snEV0QkVSgZFZEeKxwO01xV\n9V2yGdOz2VReTmPFRghtW0rH7ydj0CAyd59A+uBIwhn5E3mdNmAAvphSePVXX/DZB2+Tm1mQxKvb\nWmVGHvOLj2B8zSqOLH+XaRUfsEf11zxXNIVvs4d4FpeISFcpGRWRlNZcV/ddGX07PZzhLVu2u19a\n/kCyRo2OJJmFWyec6YECfGk9sEfR52N53ki+zinh0Ir3mVT5OeeWPs+HA8aycNC+GuAkIj2SklER\n8VSosZGmjRtjeja3TjhDtbXb3c+fnU2/ocWtvZkZgweTHpNw+vt1Tyk9FTSk9ePFwsl8kjeaYze8\nzfeqvmRs7be8Mng/PtEAJxHpYZSMiki3CodCNG0KRpLLsg1bl9HLy2jatAnC206B5EtPJ2NwIemj\nxsT0bH7Xw5nWv78HV5Na1mQVMm/XGey36VMOqfiQE9e/yV5VX/J84RSCGuAkIj2EklER2a5v/3Aj\ndcs/xQE54yewy0+u2+524XCYUE3Nd72ZMQlnY3k5jRvLoXk7E7f7fKQXFJBtu22TaGYUDiZtQD4+\nvx4SF0/I52dJYE8+zx3JUWXvMLaulEtWL+Ct6ACnnR3ZLyKSLEpGRWQb3/7hRuo++6T1fd1nn/DV\nj39I4NgZ+Px+GsujCWd0wFC4oX67x0nLyyNrxIjWJHOrwUIFBfjS9RWUKJUZuTxcfDi71X7DkWVL\nOLTiAyZUf83zRVNYrQFOkkI6+ouu9B36n0BEtlG3/NNtljVXVVH+0ANbLfNlZpFR2KZXc/DgyLJB\ng/FnZSUrZAHw+fg8dwQrs4tbBzidU/o8H+aNZeHgSdSn6ech3treL7orrvsRJdfOImvESO8CE08p\nGRWRDvNnZzPk/Itak09/bu5WUyBJamgZ4PRx3hiO3bCY71V/ybja1bwyeD8+zhutAU7ime39otsU\nDLJmzs2MvvEmDyKSVKAbskRkGznjJ2yzLD0QYJef/jt5+x9A1qjRpOXlKRFNcWuzBjNv1xm8Mmhf\n0sPNnLBhEWeteZHAliqvQ5M+KFS/ebuDFUXUMyoi2yiYccJWpbT0QEC9Fj1U2OdnSWCP6BOcljC2\n7lsu+WYBiwv24u3Anl6HJ31Ac10tm15+ieBLL2x3fXogQMm1s5IclaSSuMmomd0ETAbCwCzn3Hsx\n66YD/wU0A58DlwLTgPnAx9HNljnnfpjguEWkm4Tq61k/704gMgAprV8GQ6/WP+Geriojl4eLp2O1\n33BU2RIOqfiQCdVf83j2WK9Dk16qqbqKTS++wKZXXiJUX48/pz+DTjqZTa8upLlyE6BfdCWi3WTU\nzKYBY51zU81sPHAnMDVmk7nAYc65UjN7CDgWqANedc6d3l1Bi0j3KXtkPo3lZQSOm0HhqadTWJhH\nWVm112FJIvh8uNwRrMwp5tCNH7Bv5XIuafwnoaeaaL74MtLy8ryOUHqBxmCQ4PPPUvn6q4S3bCEt\nbwCDT5jJwMOm48/Kpv/E77Fmzs34/T79oitA/J7Rw4HHAJxzy80sYGa5zrma6Pp9nXMtNx+VAQVE\nklER6YHqln9G5cKX6VdSwqCZJ3kdjnSTLf5+vFR4AB/njebotW9Qsuwjvv7F9RSefiYDph6se4Fl\npzSWlVHx3NNULXqTcFMT6YECAqcdT/7Bh271RLSsESMZfeNN+kVXWsVLRocCS2PelwHFwBcALYmo\nmRUDRwM/B74HTDCzJ4gkp7Odcy8lOG4RSbDW8rzPx5ALL8Wf0XsfpykR67IGc1v+fvxmnzTCb77B\n+rvuoGrRmww57wL6FZd4HZ70EFvWrqHimaepemcxhEJkFBZRcNwMBkw9SHMJS4d09lPiI3LvaCsz\nKwIWAFc554Jm5oAbnHPzzWw0sNDMxjjnmhITsoh0h9jyfPbo0V6HI0kS8vnxHzCZ4Ucfx4YH7qX2\ng/dZNfuXBI6bQcHxM/RLiexQw+pv2Pj0U9QsfRfCYfqVlFBw/Ank7T8ZX5qe/CUdFy8ZXUOkd7RF\nCbC25Y2ZDQCeAX7W0vvpnFtDZAATzrkVZrYOGAas2tFJAoEc0tN75ge3sFD3WCWb2jzxNn20jMqF\nL5O96y6Mv/jcrUpqoDYPBnO9DqFbFRTkUmIjKZn9cza+/Q4r5v6NiiefoG7pEsZcdQUDJ+7ldYhJ\n0dc/5x1V/blj9fxHCL4bGc/cf8xodj39VAomH9DpR/iqzQXiJ6MvALOBuWY2CSh1ztXGrP8DcJNz\nrnW+BjM7GxjnnJsd7TUtAkrbO0kw2DNvM9X9LsmnNk+8UH09K2+eAz4fg8+7mI2VDUBD63q1OVRU\n1MTfqAerqKj57mc8ZgLDZ/+W8scfZdPLL/HJL25gwIEHMfiMM0nPG+BtoN1In/P2hcNhNrvPqXjq\nydZp37LGjGXQCTPJ2XMvQj4f5Rtr4xxla2pzadFuMuqcW2xmS81sEZHpm64xswuASuB54DxgrJld\nGt3lPuBB4H4zexNII1K+V4leJEWVPTKfpvJyleellT8rm6IfnMOAKVNZf888qhYvouajDyIDnA46\nRAOc+pBwOEzdJ8vY+NST1H/5BQA5u0+gYMaJZO82Xp8FSYi494w6565vs2hZzOsdPeh45k5HJCJJ\ns/Xo+e97HY6kmKyRoxj+/37JpoUvU/7Yo6yfdydVby2i6NwLyCzRAKfeLBwKUfPB+1Q8/SQNq1YC\n0H/i9yJJ6BjNTZuKzGwkkRztvZjF7wN/IjIz0ivOueui22YAc4A9gabonwudc6uTGXMLDXMT6aNC\n9fWsm3dHzOj5DK9DkhTkS0sjcOTR5E7aj7IH7qPm/aWsmv0LCo6bQcGMEzTAqZcJNzdT/d4SKp5+\nii1rSsHnI3e//Sk4/gSyho/wOrxe5c2Zp7wIHBF9+/LBCx49KgGHXe6cmx67wMyeJ1LNjv2SPxto\ncs4dFN3mAuBqoG0HZFxm5nPOdek5r0pGRfooleelMzIKCii55l+oef+fbLj/XiqeWkD1kncoOvd8\n+k/Yw+vwpIvCTU1ULV5ExbPP0LhhPfj9DDjwIALHzVAveDeIJqJHxiw68s2Zp3wLzDx4waP/TPDp\nTgFOI9IL2iIfaB095py7u+W1mZ0H/AsQAv7onHvIzM4AfkSkB3Wpc+5fzewGYBQwyswOA34DHEzk\nFs05zrkHOxqgklGRPkjledlZuftMImf33Sl/4nE2vfQCpX+8kbwpB1J4xlmkD+i9A5x6q9CWLVS9\n+ToVzz1DU0UFpKWRf+hhBI47nn6FRV6H12O9OfOUG4H2nkS5vW7mYcCSaFK6PfMPXvDodZ2NxTlX\na2ZtF98LXGhmy4nMivSIc26RmeUBvwD2InIr5t1m9jTwW+B7zrk6M1sQTT7DQIZz7lAzOwQY7pyb\nZmaZwFIze8w519D2xNujZFSkj1F5XrrKn5VN0ZlnMWDKgaz/+91Uv72Y2o8+ovC0Mxhw8CGdnt5H\nki9UX8+m1xYSfP5Zmquq8PXrx8AjjyJw9HFkFBR4HZ7svN3MbGHM+xecc78jMk98K+dcBTDJzA4m\n8tCiB8zsTuBpIqX+lmlVvh+dTekL51zL1EevAvtEXy+J/j0VmBJzbh+RhySt7EjQSkZ3wrd/uJG6\n5Z/igJzxE9jlJ53+RUXEMyrPS6JkjRjJ8J/9gk0LX2bjY4+w/p67qFq8iKLzLiCzZJjX4cl2NNfV\nsunllwi+9AKh2lr8WVkEjptB4Khj1LOdQNEezB0mB9sp00NkGsyuluk/b3vPaFTbBxb1I3LP6JvA\nm2b2NyJJ5gKg7W+TIbZOZvsBm6OvG6N/NwB3OOd+vzNB69fXTvr2DzdG5lgLhyEcpu6zT1hx3Y+o\nj442FEllKs9Lovn8fgJHHMWIX/8XuZP2ZfMXjlWzf0n5Y48Q2rLF6/Akqqm6ivJHH+brf/8pG594\nDMIw6KSTGfXff6Dw1NOViCZZdLBS7BzspQcveHSXbrhftEXbObjuAC6Leb8r8BWwnEjvan8zyzKz\nFwAHjDOzlqd/TGPrEfsA7wAnmpkvut+fOxOcekY7qW75p9ssawoGWTPnZkbfeJMHEYl0jMrz0p0y\nCgooufpfqPng/cgAp6ef/G6A0x57xj+AdIvGYJDg889S+fqrhLdsIS1vAINPPZGB06fjz8r2Ory+\nbiaRnsiW14nQtgd0VyL3hA4EsszsWOAsIoORbo8OVqoHthCZF77OzH4JvBQ9xB+jy64DnjOzEPBG\n9P7SI1vOF52XfiGwmEji+5fOBJ0Ss9Vu2FDVpSkBkslddlGkV7SN9EBAyWgS6IkdO2/9ffdQufAV\nAsfNoPDU9u6r35raHL766guun/s2uYHeV3quCZbyu8unMGbMuIQdM1Rfz8YFjxN86QUIhcibPCUy\nwCk/P2HnSLTe9jlvLC+j4tlnqFr0BuGmJtIDBQSOPY78Q6Zt87hfr/TkNi8qGpAS+VNvoZ7RTsoZ\nP6H1UWit/H6GXn6VNwGJdECkPP+KyvOSFP6sLArP+AF5Uw5kw9/vpvqdt6ld9hGDTz2D/EMO7dAA\np+bmZlauXJGEaCOCwdykPvZ15MjRpKWlJfy4W9atpeKZp6h6ezGEQmQUFlFw3AwGTD0IX7r+y5fU\npE9mJ+3yk+tYcd2PaAoGAfBlZhJuaKDy1VfIHjtOj0aTlBNbnh96kcrzkjxZw0ew6/U/p/LVVyh/\n9GE2/H0eVW+9yZDzLyRz2C7t7rty5Qpm3biAnPzeN71QXeUGbr5uZkJ7oxtWr2bj009Ss/RdCIfp\nV1xCwYwTyNt/Mr5uSHpFEknJ6E4ouXYWa+bcjN/vY8gV11D24P1Uv/M22WPHMXD6EfEPIJJEZY88\n1Dp6PmuURs9Lcvn8fgYefiS5k/ZlwwP3UbP0PVb9+lcUHHMcBTNOxJ+ZucN9c/KLeuWtEYm0ecVX\nVDz9JLUffgBA5vARFMw4kdx9JmmKLekxlIzuhKwRIxl9402t97sUX3kN3/z6V2x48H4yR4zSdDmS\nMlSel1SRPjBAyVXXUvPRB2y47+9UPPMU1e++Q9E559N/z728Dq9HCYfDbHafU/HUk623jWWNGcug\nE2aSs+deqtBJj6NkNAEyCgoYevmVlN70v6y9bQ4jfjGbtLy8+DuKdCOV5yUV5U7cm5zddo8McHrx\neUr/9AfyDphM4ZlnkZ4/0OvwUlo4HKbuk2VUPP0Um79wAOTsPoGCGSeSvdt4JaHSYykZTZD+E/Zg\n0Ekns/HxR1n7t9sZNuvHKpGIp1Sel1Tlz8yk8PQzo09wmkf1knciA5xOO4P8Q6bpu7ONcChE7Yfv\ns/GpJ2mIzmndf+L3IknomLHeBieSAEpGE6jg+BOo/+pLapd9xMYnn2DwSSd7HZL0USrPS0+Quetw\ndv2Pn1P52quUPzqfDX+/m6q3FjHkvAu8Di0lhEMhqt9dQsXTT7JlTSn4fOTutz8Fx59A1vDtPdpc\nUkgaMCbBx/wKaG5vAzO7BjiXyBORsoGfOede7sjBzex7wMnOuRu6GGenKRlNIJ/fz9BLLmfVf95A\nxVMLyB4zhv57TvQ6LOljVJ6XnsTn9zNw+uHk7jOJDQ/eT817S1j1mxvw7X8AGeEdD27qzcJNTVS9\n/RYVzzxN44b14Pcz4MCDCBw3g8ySEq/Dk44Zc8Apv/o8UbNB1FVuYMmjs3cj8jSk7TKzkcClwH7O\nuWYzM2Au0KFk1Dn3IfBhAsLtNCWjCZaWm0vJldew+ve/Ze1fb2fEL2eTMWiw12FJH6LyvPRE6QMH\nUnLl1dQuO5j1991D09uL+aE/ixcz01jRv2+MqA9t2ULVm69T8dwzNFVUQFoa+YceRuC44+lX2Pum\nuOrtPJgNIh/IAjKBOuecAw6LPinpN0SetLQJOAN4iMjTld4ws2zgU+ASIk9hOt3MvgQeAw6K7jMD\nGAbMJ9Lr+jpwiHNuevTRn/sS6Q2+1Tl3d2cD14053SBr5CgKzzqXUG0ta279C6HGRq9Dkj6i7rNP\nVZ6XHq3/XhMZOfu3+KYcyIBQA2esfZmT1r1G/6Y6r0PrNqH6eiqef5avr7+ODfffS3NNDQOPPIpR\nv7uRIedfqERUOiTas7kE+NrM7jKz080snUiSeo5zbjpQCRwDPAKcGN31KOB5tr4FYBRwt3NuKhAA\nJhJ5hOiDzrnDiCS8YTMLAMc75w4CDgZ2qhSnZLSb5B86jQEHHkTDyq8pe+gBr8ORPiBUX8+6u++M\nPBFM5XnpwfyZmfinH84t+QfwbVYhu9es4rJvnmCfys+3+zjmnqq5rpaNTz7Bin//CeXz/0G4oYHA\ncTMY9fv/pegH55BRUOB1iNLDOOcuAKYBHwD/BrwAbATmmtmrwOFAAfAkcGx0t5OAh9scqso593H0\n9bdEEtrxwKLosgWAzzkXBJyZPQ6cCdyzM3ErGe0mPp+PonPPp9+wXahc+ApVb7/ldUjSy7WU5wuO\nPV7leekV1qfncu+wY3mucAoAx5S9w3nfPkthQ9DjyLomJ7SF0KsL+frff8rGJx6DMAw66WRG/fcf\nKDz1dNIHDPA6ROmhzCzTObfcOXczMBnYFbgTuCbao/kEgHOuEiiN3ld6IPAKEDs3WFObQ/uif7b5\nbdA5dzwwG9ibSJLbaUpGu5E/M5OSq6/Fn53N+nvm0VD6rdchSS8VW54vOPEkr8MRSRyfjw/yjb8O\n/z6f5o5kWEM5F61+isPKl5IR6lm3QOU21XF42bv8NLiI8OK38KVnMPjUMxj9P//LoBNPIq1/f69D\nlB7MzC4F7jSzlqRyIJEEMg9YbWYDifSMtowMfAz4OfCWcy7UgVN8BewXfX1c9JwjzOyHzrn3nXPX\nAYN2JnYNYOpm/YYMZchFl7L2lv9jzS1zGP7zX5GWne11WNKLqDwvfUFtejYLhh7KstoxHFP2DlM2\nfcLuNSt5vnAyK/q3/5x7r+U31jA5+DETq74knRCV/kwyj5jGqJNPw9+vn9fhSTepq9yQ7GPdCewG\nvG1mNUREWg3CAAAeg0lEQVTu3/wXYAqR8vpXwP8AvzKzBUR6Sf+PSJkeIr2e4ZjXscLAzcBDZnYq\nkXtTm4E1wIFmdiaRgU137Mz1xU1GzewmIl29YWCWc+69mHXTgf+KBvQ5cKlzLtzePn1R3qR9qT/m\nWILPP8f6eXdQfOU1elKGJExref74E1Sel17v6/7D+Fv2TKYGP2Jy8BPOWPsKn+WO4OXB+1OTnuN1\neFsp2FLJgcGP2aN6BX7CBDPyWDxwT95pzuI/99tfiWjv9lV0KqaEHrO9ldHezeu2s+pZ4Fcx72NH\nu7fOn+acew14Lfq6KGb56QBmNgG41jn3lpmdBQx2zjUCZ3XyOrbRbjJqZtOAsc65qWY2nkjWPTVm\nk7nAYc65UjN7CDjWzOri7NMnDT7ldOpXrKBm6XtsevEFAkcf43VI0guoPC99UZM/ndcHTeLT3FEc\nW/Y2u9esYlTdGl4v2If3842wz9s70AobgkwNfsT4mlX4gPKMfN4q2IvPckcS9vlpDpZ6Gp8kRTPt\nzAnaQ1UDt5tZmMj1XZSoA8frGT2cyD0FOOeWm1nAzHKdczXR9fs656qir8uI3CswJc4+fZIvLY3i\nK65m1a9/SdkjD5E1ajTZ48Z5HZb0YCrPS19Xnhng3mHH8r2qL5i+8Z8cXb6EPatX8FzRFDZkJn8k\nenF9OVODHzGuNjI+YF1mAW8F9sL1Hw6qhkkP55xbDRzSHceO9+vjUKA85n0ZUNzypiURNbNi4Gjg\nmXj79GXpAwdSfMXVEA6z5va/0FRZ6XVI0oNp9LwI4PPxYb4xd/hJfJI7ipKGci5c/TTTy99L2gCn\nXTev58zSF7ng22cYV/st32YV8lDx4czbZQYud4QSUZE4OlvL2GZYv5kVEZlv6irnXEVH9unLcnYb\nz+BTTqN50ybW/vU2ws3tPmZWZLtUnhfZWl16Nk8OPYQHS46kMiOXyZs+5dJvFjCmtptmMQmHGVVb\nyjnfPsc5pc8zavNaVmYP5f6So7l32LGRQVVKQkU6JF6Zfg2Rns4WJcDaljdmNoBIb+jPnHMvdWSf\n7QkEckhPT+tozCmlsDCv0/sMPvcMQqtXUvHOEja/+DQjzjunGyLrvXamzXuT5s2bWfX3eeD3M/7H\ns8gr6f5yZF9v82Aw1+sQulVBQW7K/Yx3ts1X5pRwx64ncmBwGVOCn3D62ldY3n84LxUekJgBTuEw\n42pXMzW4jOKGjQB8mbMLbxXsxZqswg4fJhXb3AtqA4H4yegLRCYynWtmk4BS51xtzPo/ADc5517o\nxD7bCAZ75mPeCgvzKCur3ql9A+dcSPXXq/j24UcJDd2V3L33SXB0vVNX2ry3WH/fPTRs2EDB8SdQ\nP3AI9d3cHmpzqKjo3be8V1TUpNzPuCtt3uRP541B+/BZ7iiOKXub8bXfMKpuLa8P2pt/5u+2UwOc\nfOEQ42tWMTW4jMItmwgDy/uPYHHBnqzP7PzUiqnY5smm7xZp0W4y6pxbbGZLzWwRkZFT15jZBUSe\nbfo8cB4wNjrRKsB9zrm/td2nG+PvsdJycii56hq++a/fsO6OuQz/5Ww9f1jiUnlepOPKMwdy37Bj\nmFj9JdPLl3JU+butA5w6mkD6w83sWb2CKcGPKWisJoSPj/NGsziwJxv7DezmKxDpG+LOM+qcu77N\nomUxr7M6uI9sR+auwyk693zW33UHa2+Zw67X/1zzzskOafS8yE7w+fhowDi+zNmFwzcuZc/qFVyw\n+hneyx/PG4P2ptG//X9HaaFmJlZ/wZTgJ+Q31dKMn/cHjOPtwJ5UZqi0LJJIegKTx/IPOoT6r76k\n8vXX2HD/vQy98GKvQ5IU1TJ6fuCxx1MaaoavvkjKeYPB3KSWqUeOHE1aWs+8h1xSV116Nk8NOZhl\neaM5puwdDqj8jN1qV1Hvz6RoS+RZ9yuzi3m0+DD2qXQcsOlTcps30+hL49383VkSmEB1eu9+XGdz\nczMrV65I2vn03SItlIymgMKzzqF+5Uqq3nyd7LFjyT/4UK9DkhQTW56v2msv/vXGBeTk977bOuoq\nN3DzdTMZM0Zz8Er3WJVTwh27zmRqcBlTgx+Rz3djFkZtXsuPVzyAD2jwpbN44J68O3B36tL7xiOc\nV65cwSx9t4gHlIymAH9GP0quupZVv/kVG+77O5nDR5A1fITXYUmKaFueLw01k5NfRG5gmNehifRI\nzf403hi0N1ODH22zLpKIZnDryFOoT8vcdudeTt8t4gVvn5kmrTIKCxl66eWEGxtZe+scmuvanYBA\n+pCyhzW5vUgyNfgz+mQiKuIVJaMpJHfi3hTMOJHGsjLW3fk3wqGQ1yGJx+o++5TKV1+hX8kwjZ4X\nSbCV2ds+HLAqLYdHiqd7EI1I36VkNMUMOulkcnafQO0H7xN8/lmvwxEPbVWev1ij50US7R/DjqIq\n7buJ8KvScrhl1Gmsz+r8vKEisvOUjKYYn9/P0MuuJD0QoPzRh6lb/pnXIYlHtirPjxzldTgivdIj\nxdOpSstRj6iIh5SMpqD0AQMovuJq8PtZe/utNG0Keh2SJJnK8yLJsT5rELeMOk09oiIeUjKaorLH\njqPw9DNprq5izW23EG5q8jokSRKV50VEpC9RMprCBh5xFLn7HUD9l19Q/sh8r8ORJFF5XkRE+hIl\noynM5/Mx9MKL6De0mOCLz1P93rtehyTdTOV5ERHpa5SMpjh/VjbFV1+LLzOT9fPuYMu6dV6HJN1E\n5XkREemLlIz2AJklwxhy/kWE6utZc+scQg0NXock3UDleRER6YuUjPYQAyZPIX/6EWwp/Zb1f59H\nOBz2OiRJIJXnRUSkr1Iy2oMUnvEDskaPpvrtxVS+ttDrcCRBQvWbWTfvDpXnRUSkT1Iy2oP4MzIo\nvuIa/Lm5lD14P/Vfr/A6JEmAsofn07Rxo8rzIiLSJykZ7WEyBg2i+LIrCTc3s+bWv9BcU+N1SNIF\nKs+LiEhfp2S0B+q/x54Mmvl9mio2svZvtxMOhbwOSXaCyvMiIiJKRnusghknkrPnROo+XkbF0096\nHY7sBJXnRURElIz2WD6/n+JLLye9YBAbFzxO7Scfex2SdILK8yIiIhFKRnuwtNxcSq66Bl9aGmv/\nehuNGzd6HZJ0gMrzIiIi31Ey2sNljRpN4Q/OJlRTw9rb/kKosdHrkCQOledFRES+kx5vAzO7CZgM\nhIFZzrn3YtZlAXOB3Z1z+0eXHQbMB1rqxsuccz9McNwSI3/adDZ/+QXVby+m7KEHGXLOeV6HJDug\n8ryIiMjW2k1GzWwaMNY5N9XMxgN3AlNjNvkfYAmwe5tdFzrnzkhopLJDPp+PIeddSMPq1VQufJns\nsWMZMPlAr8OSNlSeFxER2Va8Mv3hwGMAzrnlQMDMcmPWXw9sbyi3LzHhSUf5MzMpuepa/FlZrL/7\nLhpKS70OSdpQeV5ERGRb8ZLRoUB5zPsyoLjljXOulm0TzzAwwcyeMLM3zOzIhEQqcfUbOpQhF11C\neMsW1tz6f4TqN3sdkkSpPC8iIrJ9nR3A5COSbLbnC+AG59xJwAXAHWYW995USYy8ffcncNQxNK5b\nx7p5dxEOx/txSXdTeV5ERGTH4iWJa4j0jrYoAda22WarbMc5t4bIACaccyvMbB0wDFi1o5MEAjmk\np6d1NOaUUliY53UI2xh05cV8UvoNVe8toWnvPSg58QSvQ0qoVGzz9nx12wM0bdzILqedwq77T+zy\n8YLB3Pgb9WAFBbkp9zNWmyef2jz51ObilXjJ6AvAbGCumU0CSqOl+VhblenN7GxgnHNutpkVAUVA\nuzcwBoN1nYs6RRQW5lFWVu11GNs1+OLLqZ39K76+6x6aBpeQPXac1yElRCq3+fbUffYp6559nn4l\nw8g64riExF5RUZOAyFJXRUVNyv2M1ebJpzZPPrW5eKXdMr1zbjGw1MwWAX8CrjGzC8zs+wBm9hLw\nHLCHmS0zs4uABcC+ZvYm8ARwlXOuqVuvQraRPjBA8RVXQSjEmtv+QlNVldch9Tkqz4uIiMQX915O\n59z1bRYti1m3o8FJM7sSlCRGzvjdGXzKaZQ/Mp91f72NYT/6KT6/nnOQLGXzH4qMnj/+BI2eFxER\n2QFlJr1c4Njj6b/3PtR99ikbn3jM63D6jLrPPqXytYX0G7aLRs+LiIi0Q8loL+fz+Rh68aVkFBZS\n8fST1Hz4gdch9XpblecvukTleRERkXYoGe0D0nL6U3zVtfgyMlh3x1way8q8DqlXay3Pa3J7ERGR\nuJSM9hFZw0dQdM55hOrqWHPrHEKNW7wOqVdSeV5ERKRzlIz2IfkHH8qAgw+l4ZtVlD1wn9fh9Doq\nz4uIiHSektE+pujsc8ncdTiVr79G5aI3vA6nV1F5XkREpPOUjPYx/n79KL76Wvw5OWy49x4aVn/j\ndUi9gsrzIiIiO0fJaB/Ur7CIoRdfRrixkTW3zKG5ru1DtaQzVJ4XERHZeUpG+6jcvfeh4PgTaCzb\nwLq77iAcDnsdUo+l8ryIiMjOUzLahw066WSyx+9O7fv/JPj8s16H0yOpPC8iItI1Skb7MF9aGsWX\nXUnawIGUPzKfus+Xex1Sj7J1eV7PnhcREdkZSkb7uPT8fEquuAb8ftbefgtNmzZ5HVKP0VqeP24G\nWSNHeh2OiIhIj5TudQCJ1NzczMqVK5J2vmAwl4qKmqSdb+TI0aSlpSX8uNnjxlF46hmUPfQAa2+/\nhV1+8m/40nvVRyPhtirPnzDT63BERER6rF6VcaxcuYJZNy4gJ7/I61ASrq5yAzdfN5MxY8Z1y/EH\nHnU0m7/6gpql71H+2MMUnv6DbjlPb6DyvIiISOL0qmQUICe/iNzAMK/D6HF8Ph9DLryEhm+/Jfj8\nc2SNHkvevvt5HVZKai3PzzhR5XkREZEu0j2j0iotO5uSq6/F168f6+fdwZb167wOKeWoPC8iIpJY\nSkZlK5nDdmHI+RcS2ryZNbfMIdTQ4HVIKUPleRERkcRTMirbGDBlKvnTD2dL6bdsuPceTYgfpdHz\nIiIiiadkVLar8IyzyBo1mqrFi6h8/TWvw/GcyvMiIiLdQ8mobJc/I4PiK6/G378/ZQ/cS/3Kr70O\nyTMqz4uIiHQfJaOyQxmDBlN82RWEm5tZc+scmmuSN6dqKlF5XkREpPsoGZV29d9zIgUnzKRp40bW\n/m0u4VDI65CSSuV5ERGR7hV3nlEzuwmYDISBWc6592LWZQFzgd2dc/t3ZB/peQadeBL1K76i7uOP\nqHjmKQb1kaRM5XkREZHu127PqJlNA8Y656YClwB/brPJ/wBLOrmP9DA+v5/iS68gvaCAjU88Ru0n\nH3sdUlKoPC8iItL94pXpDwceA3DOLQcCZpYbs/564MlO7iM9UFpeHsVXXgt+P+v+ejuNFRu9Dqlb\nqTwvIiKSHPGS0aFAecz7MqC45Y1zrhbwdWYf6bmyR4+m6Adn01xTzdrbbiHc1OR1SN1C5XkREZHk\n6ewAJh+R+0C7ex9JUfmHHU7e5CnUr/iKsoce9DqcbqHyvIiISPLEG8C0hkhPZ4sSYG2bbdommh3Z\nZyuBQA7p6WlxQokvGOzddwMUFORSWJjndRgM+vG/8NF1pWx65SWK9tmLwkMPTur5u7MNNn34EZWv\nLSRnxHB2u+iclOwV1ec8+dTmyac2Tz61uXglXjL6AjAbmGtmk4DSaGk+VtsyfUf22UowWNeJkHes\noqJ3z4NZUVFDWVm112EAUHT51az6zWy+mHMLDfmDySwZlpTzFhbmdVsbhOo3s/LmOeD3M/i8i9m4\nqR6o75ZzdYU+58mnNk8+tXnyqc3FK+2W6Z1zi4GlZrYI+BNwjZldYGbfBzCzl4DngD3MbJmZXbS9\nfbr3EsQL/YYWM/Siiwk3NLD21r8Qqk+9pK2zyub/Q+V5ERGRJIs7z6hz7vo2i5bFrDuyg/tIL5S3\n3wFsPvJLNr30AuvvuYuhl12Jz9e2o7xnqP30Eypfe1Wj50VERJJMT2CSLik87Qyyxo6jesk7bHrl\nJa/D2Smh+s2sv/tOjZ4XERHxgJJR6RJfejrFV1xNWt4Ayh56kM1ffel1SJ2m8ryIiIh3lIxKl2UE\nAhRffiWEQqy97Raaqqu8DqnDVJ4XERHxlpJRSYic3Scw+ORTaQpWsG7u7YRDIa9DikvleREREe8p\nGZWECRx7PP0nfo+6zz5h44LHvA4nLpXnRUREvKdkVBLG5/cz9JLLyRhcSMVTT1Lz0Ydeh7RDKs+L\niIikBiWjklBp/ftTfPW1+NLTWfe3uTSWl3kd0jZC9ZtZP0/leRERkVSgZFQSLmv4CIrOOY9QXS1r\nbv0LocYtXoe0lbL5/6CpQuV5ERGRVKBkVLpF/iHTGHDQITSsWknZg/d7HU4rledFRERSi5JR6TZF\n55xH5q67Uvnaq1S9tcjrcFSeFxERSUFKRqXb+Pv1o/jKa/FnZ7P+3rtpWL3a03hay/PHqzwvIiKS\nKpSMSrfqN2QIQy++jPCWLay5dQ7NdXWexLFVeX6GyvMiIiKpQsmodLvcfSYROPZ4GjesZ/28OwiH\nw0k9v8rzIiIiqUvJqCTF4JNPJXu38dT8cynBF55L6rlVnhcREUldSkYlKXxpaRRffiVp+QMpf2Q+\nde7zpJxX5XkREZHUpmRUkiY9fyDFV1wFwNrbb6GpclO3nq95s8rzIiIiqU7JqCRVju3G4FNPp7my\nkrW330q4ubnbzlX+sMrzIiIiqU7JqCRd4OhjyZ20L5vd55Q/9ki3nCO2PD/ohJO65RwiIiLSdUpG\nJel8Ph9DLrqUjCFDCD73DDXvL03o8duW533p6Qk9voiIiCSOklHxRFp2NiVXXYuvXz/W3fk3tqxf\nn7BjqzwvIiLScygZFc9k7rIrQ867gNDmzay5dQ6hhoYuH1PleRERkZ5Fyah4asCBB5E/bTpbvl3N\nhvv+3qUJ8VWeFxER6Xni/m9tZjcBk4EwMMs5917MuiOB3wLNwDPOuf80s8OA+cDH0c2WOed+mOjA\npfco/MHZ1K9aSdVbb5I9dhz5h07bqeO0ludPOFHleRERkR6i3Z5RM5sGjHXOTQUuAf7cZpObgVOA\ng4CjzWx3Iknrq8656dE/SkSlXf6MDEquvBp/Tn823P936let7PQxVJ4XERHpmeKV6Q8HHgNwzi0H\nAmaWC2Bmo4EK51ypcy4MPAMc0Z3BSu+VMbiQ4suuINzczJpb59BcU9PhfVWeFxER6bniJaNDgfKY\n92XRZS3rymLWbQCKo68nmNkTZvZGtJQvElf/vSZSMONEmsrLWXfnXwmHQh3ar1zPnhcREemxOjuA\nydeBdV8ANzjnTgIuAO4wM3VVSYcMmvl9cibsQe1HH1Lx7NNxt6/99BMqX1d5XkREpKeKlySu4bue\nUIASYG30dWmbdbsApc65NUQGMOGcW2Fm64BhwKodnSQQyCE9Pa2ToW8rGMzt8jFSWUFBLoWFeV6H\n0e0C1/+UD/71p2x84jGG7rMnA783cav1LW3QVFfHqr/fBX4/u/9kFrnFAS/CTTp9zpNPbZ58avPk\nU5uLV+Iloy8As4G5ZjaJSLJZC+CcW2VmA8xsBJHEdAZwtpmdDYxzzs02syKgKLp+h4LBuq5eBwAV\nFR2/z7Anqqiooays2uswksDHkMuvYvX//I7lN/6R4b/8NRmBSKJZWJjX2gbr75lHQ1k5BSecyOYB\nhWzuE22jz7kX1ObJpzZPPrW5eKXdMr1zbjGw1MwWAX8CrjGzC8zs+9FNrgIeAF4HHnTOfQksAPY1\nszeBJ4CrnHNN3XYF0itljxlL4Zln0Vxdzdrb/kK4aeuPkMrzIiIivUPcezmdc9e3WbQsZt0bwNQ2\n29cAMxMSnfRpA6cfQf2XX1C95B3KHv4HRT84B2gzev5ijZ4XERHpyfS/uKQsn8/HkPMvomH1aja9\n9CKbXnoR5/ORnj+Qpk3ByOT2I0Z6HaaIiIh0gR4HKinNn5WFPzvnuwXhME2bguBPo//Evb0LTERE\nRBJCyaikvPqvv9p2YaiZtbfOSX4wIiIiklBKRkVERETEM0pGJeXljJ+wzbL0QICSa2d5EI2IiIgk\nkpJRSXm7/OQ60gPfTWifHggw+sabNHhJRESkF1AyKj1CybWzSA8E6DeoQD2iIiIivYimdpIeIWvE\nSEbfeNNWT2ASERGRnk89oyIiIiLiGSWjIiIiIuIZJaMiIiIi4hkloyIiIiLiGSWjIiIiIuIZJaMi\nIiIi4hkloyIiIiLiGSWjIiIiIuIZJaMiIiIi4hkloyIiIiLiGSWjIiIiIuIZPZteuqS5uZmVK1ck\n7XzBYC4VFTVJO9/IkaNJS0tL2vlERET6GiWj0iUrV65g1o0LyMkv8jqUhKur3MDN181kzJhxXoci\nIiLSaykZlS7LyS8iNzDM6zBERESkB4qbjJrZTcBkIAzMcs69F7PuSOC3QDPwjHPuP+PtIyIiIiLS\not0BTGY2DRjrnJsKXAL8uc0mNwOnAAcBR5vZ7h3YR0REREQEiD+a/nDgMQDn3HIgYGa5AGY2Gqhw\nzpU658LAM8AR7e0jIiIiIhIrXjI6FCiPeV8WXdayrixm3QageAf7FHctTBERERHpjTo7gMm3E+t8\nRO4dTYq6yg3JOlVSpfJ1pXJsXZHK15XKsXVFKl9XKsfWFal8XakcW1ek8nWlcmxd0Vuvq7eIl4yu\n4bueUIASYG30dWmbdbtEt9/Szj7bVVQ0oL0kV6IOfHS21yH0OWrz5FObJ5/aPPnU5smnNk9d8cr0\nLwCnAZjZJKDUOVcL4JxbBQwwsxFmlg7MAJ5vbx8RERERkVhxeyTN7HfAoUSmb7oGmARUOuceN7ND\ngP+Obvqwc+6P29vHObesO4IXERERERERERERERERERERERERERERERERkdSmKZU6wMxGAsuA96KL\nMqPvrwJWAN8QGawFEHbOHZ7sGHujaLvPd87tH7PsBuBsIlOL+YBs4HfOuce9iLE3afM59wFNwH8R\nmZ7tEiAL2ANYGt3lfOfc6uRHmrq6+l1hZhcCvwG+JPIzaADOc85tMLN5QKFzbkbM9scDTwEjicyO\nst1zO+dCCbzMlNLV7wkzW0nk59IE5AJ3OOdujx53BTDZOfduzPZLgE+ccxdFfyaTgI3R8/QDrnPO\nLUr4haaQrn5XqM2lrc5Oet+XLXfOTW95Y2Z3AecQmdD/WOdcnWeR9S1h4E/OuVsAzCwAfGBmzzrn\nGrwNrVdo/ZxHH/n7JHCmc266mY0gMmvG9HaPIF35rggDDzjn/i267y+Ai4HfR9ePNLOBzrlN0fen\nAV/FOffZwL1dvKaepjPfE60/FzPLAVaY2dzouhXAGcC70eOMAAJ89yCXMPAfzrlnoutHA88Cu3Xf\npaWMrnxXqM1lK/HmGZUdewcY63UQfVRrj75zLkjkoQp65GyCOedWAL8Fro0uUiVl53T2uyK2nYcC\n30Zfh4GXgJMBzCwDGAe01zu9JLpNX9SZ74mWbQcBZc65MJH2fhs4Ima704jMpb1d0X8zA8ysT/1b\n2cnvCrW5tFIyuhOi/wmcBPwzukj/CDxiZrsBQ/juP2xJrKXABK+D6Kl24rvCB5xpZgvNbBmwN/BI\nzPpHgdOjr48AXt3RMaPnnsl3pdI+qwPfE8+a2WtE2uo3McsbgffNbHL0/QzgGbZu89bXZnYosCaa\nWPU1nf2uUJtLK5XpO243M1sYfT0R+L1z7gkzu5nIP6qW+8DKnHNneBNin+ADZpnZacAAIvfFneWc\na/I2rF4rj+/ucZSO6cp3RRh4MKZMfy5wO3B+dP1KIr1AAeBUYA5wcJxzL0jgtfUUnf2eaCkZ5wEv\nmdmHQH103cPAGWZWCgSBmjbn+Z2Z/RQYHF13duIvp0fo7HeF2lxaKRntuM9j7o+ZD3wRXa57RpOr\n9V4wMxsKvELkRnrpHvvxXa+edEyHvyuiAzbuir79SfTv2B6gR4H/bHP8x4mULvd0zn1oZh05d1+z\nw++J6OCmacBHzrlZsTs556rN7FXgQKAlqX8J+B2wiq17qVvO8x/OuWfMbCLwN8B1xwX1ADv8rlCb\nSzwq0++c64DfR2+8luTyRf/gnFsH3AP8ytOIeikzGwP8CLjJ61h6sHa/K5xzK51z06N/tvcf+WRg\neZtlDxNJXF/t4LmzOxlzb7DD7wnn3A3R9p7VZnui9x3uT0ybO+cagfeJjBJ/km1vi2g5z0dEkrGr\nuuF6Ulq87wq1ucSjZLTjWu9Hcc6tJPLb2v+LXS7dYg8z+6zlDzCVrdv8j8BMM9vdm/B6nd2i9yu+\nBdwPXO2ci73PTp/3+Lr6XdFyz+hC4OfAD2OPHT1mFZGktCPn/nknYu+puvo98Wy0vRcDC51zb0eX\ntxxjPvCNc646uiz22LGvfw5cZ2aDu3g9PUFXvyvU5iL/vz04JAAAAAAQ9P+1I6wAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAABf7XPoHUJK8dIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd7f1cbccd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost-sensitive decision trees\n",
    "\n",
    "The next algorithm that is evaluated is the cost-sensitive decision trees algorithm [13].\n",
    "\n",
    "Decision trees are one of the most widely used machine learning algorithms [14]. \n",
    "The technique is considered to be white box, in the sense that is easy to interpret, and has a \n",
    "very low computational cost, while maintaining a good performance as compared with more complex \n",
    "techniques [15]. There are two types of decision tree depending on the objective of \n",
    "the model. They work either for classification or regression. \n",
    "\n",
    "- Construction of classification trees\n",
    "\n",
    "Classification trees is one of the most common types of decision tree, in which the objective \n",
    "is to find the $Tree$ that best discriminates between classes. In general the decision tree \n",
    "represents a set of splitting rules organized in levels in a flowchart structure.\n",
    "\n",
    "- Splitting criteria\n",
    "\n",
    "In the $Tree$, each rule is shown as a node, and it is represented as $(\\mathbf{x}^j,l^j)$, meaning \n",
    "that the set $\\mathcal{S}$ is split in two sets $\\mathcal{S}^l$ and $\\mathcal{S}^r$ according to \n",
    "$\\mathbf{x}^j$ and $l^j$:\n",
    "\n",
    "$$ \\mathcal{S}^l = \\{\\mathbf{x}_i^* \\vert \\mathbf{x}_i^* \\in \\mathcal{S} \\wedge x^j_i \\le l^j \\}, \\quad and \\quad\n",
    "  \\mathcal{S}^r = \\{\\mathbf{x}_i^* \\vert \\mathbf{x}_i^* \\in \\mathcal{S} \\wedge x^j_i > l^j \\}, $$\n",
    "\n",
    "where $\\mathbf{x}^j$ is the $j^{th}$ feature represented in the vector \n",
    "$\\mathbf{x}^j=[x_1^j,x_2^j,...,x_N^j]$, and $l^j$ is a value such that $min(\\mathbf{x}^j) \\le l^j < \n",
    "max(\\mathbf{x}^j)$. Moreover, $\\mathcal{S} = \\mathcal{S}^l \\cup \\mathcal{S}^r$.\n",
    "\n",
    "After the training set have been split, the percentage of positives in the different sets is calculated. First, the number of positives in each set is estimated by\n",
    "\n",
    "$$ \\mathcal{S}_1 = \\{\\mathbf{x}_i^* \\vert \\mathbf{x}_i^* \\in \\mathcal{S} \\wedge y_i =1 \\}, $$\n",
    "\n",
    "and the percentage of positives is calculates as $\\pi_1=\\vert \\mathcal{S}_1 \\vert / \\vert \\mathcal{S} \\vert.$\n",
    "\n",
    "Then, the impurity of each leaf is calculated using either a misclassification error, \n",
    "entropy or Gini measures:\n",
    "\n",
    "\n",
    "a) Misclassification: \n",
    "    $I_m(\\pi_1)=1-max(\\pi_1,1-\\pi_1)$ \n",
    "\n",
    "b) Entropy: $I_e(\\pi_1)=-\\pi_1\\log \\pi_1 -(1-\\pi_1) \\log (1-\\pi_1)$ \n",
    "\n",
    "c) Gini: \n",
    "    $I_g(\\pi_1)=2\\pi_1(1-\\pi_1)$\n",
    "\n",
    "Finally the gain of the splitting criteria using the rule $(\\mathbf{x}^j,l^j)$ is calculated as the \n",
    "impurity of $\\mathcal{S}$ minus the weighted impurity of each leaf:\n",
    "\n",
    "$$  Gain(\\mathbf{x}^j,l^j)=I(\\pi_1)-\\frac{\\vert \\mathcal{S}^l \\vert}{\\vert \\mathcal{S} \n",
    "\\vert}I(\\pi^l_1)  -\\frac{\\vert \\mathcal{S}^r \\vert}{\\vert \\mathcal{S} \\vert}I(\\pi^r_1), $$\n",
    "\n",
    "where $I(\\pi_1)$ can be either of the impurity measures $I_e(\\pi_1)$ or $I_g(\\pi_1)$.\n",
    "\n",
    "Subsequently, the gain of all possible splitting rules is calculated. The rule with maximal \n",
    "gain is selected\n",
    "\n",
    "$$  (best_x, best_l) = argmax_{(\\mathbf{x}^j,l^j)}Gain(\\mathbf{x}^j,l^j), $$\n",
    "\n",
    "and the set $\\mathcal{S}$ is split into $\\mathcal{S}^l$ and $\\mathcal{S}^r$ according to that rule. \n",
    "\n",
    "\n",
    "- Tree growing\n",
    "\n",
    "In order to grow a tree typical algorithms use a top-down induction using a greedy search in each\n",
    "iteration [16]. In each iteration, the algorithms evaluates all possible splitting \n",
    "rules and pick the one that maximizes the splitting criteria. After the selection of a splitting \n",
    "rule, each leaf is further selected and it is subdivides into smaller leafs, until one of the \n",
    "stopping criteria is meet. \n",
    "\n",
    "- Cost-sensitive impurity measures\n",
    "\n",
    "Standard impurity measures such as misclassification, entropy or Gini, take into account the \n",
    "distribution of classes of each leaf to evaluate the predictive power of a splitting rule,\n",
    "leading to an impurity measure that is based on minimizing the misclassification rate. However, \n",
    "as has been previously shown [8], minimizing misclassification does not \n",
    "lead to the same results than minimizing cost. Instead, we are interested in measuring how good \n",
    "is a splitting rule in terms of cost not only accuracy. For doing that, we propose a new \n",
    "example-dependent cost based impurity measure that takes into account the cost matrix of each \n",
    "example.\n",
    "\n",
    "We define a new cost-based impurity measure taking into account the costs when all the examples\n",
    "in a leaf are classified both as negative using $f_0$ and positive using $f_1$\n",
    "\n",
    "$$    I_c(\\mathcal{S}) = \\min \\bigg\\{ Cost(f_0(\\mathcal{S})), Cost(f_1(\\mathcal{S})) \\bigg\\}. $$\n",
    "\n",
    "The objective of this measure is to evaluate the lowest expected cost of a splitting rule.\n",
    "Following the same logic, the classification of each set is calculated as the prediction that \n",
    "leads to the lowest cost\n",
    "\n",
    "$$\n",
    "f(\\mathcal{S}) = \n",
    "\\begin{cases}\n",
    "  \\phantom{-}0 \\phantom{-} \\mbox{if} \\phantom{-} Cost(f_0(\\mathcal{S})) \\le \n",
    "Cost(f_1(\\mathcal{S}))\\\\\n",
    "  \\phantom{-}1 \\phantom{-}\\mbox{otherwise}\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "Finally, using the cost-based impurity, the splitting criteria cost based gain of using the \n",
    "splitting rule $(\\mathbf{x}^j,l^j)$ is calculated. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             pre        f1       acc       rec       sav\n",
      "RF      0.492091  0.232172  0.934394  0.151926  0.122358\n",
      "LR      0.529412  0.037677  0.934854  0.019533  0.010948\n",
      "DT      0.251239  0.262626  0.899146  0.275095  0.191770\n",
      "RF-BMR  0.179759  0.282039  0.782493  0.654368  0.432752\n",
      "LR-BMR  0.115987  0.188429  0.717737  0.501899  0.228092\n",
      "DT-BMR  0.251639  0.260847  0.899819  0.270754  0.192115\n",
      "CSDT    0.160237  0.267374  0.711325  0.806837  0.465266\n"
     ]
    }
   ],
   "source": [
    "from costcla.models import CostSensitiveDecisionTreeClassifier\n",
    "\n",
    "classifiers[\"CSDT\"] = {\"f\": CostSensitiveDecisionTreeClassifier()}\n",
    "# Fit\n",
    "classifiers[\"CSDT\"][\"f\"].fit(X_train, y_train, cost_mat_train)\n",
    "# Predict\n",
    "classifiers[\"CSDT\"][\"c\"] = classifiers[\"CSDT\"][\"f\"].predict(X_test)\n",
    "# Evaluate\n",
    "results.loc[\"CSDT\"] = 0\n",
    "results.loc[\"CSDT\", measures.keys()] = \\\n",
    "[measures[measure](y_test, classifiers[\"CSDT\"][\"c\"]) for measure in measures.keys()]\n",
    "results[\"sav\"].loc[\"CSDT\"] = savings_score(y_test, classifiers[\"CSDT\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAp4AAAE7CAYAAAB0T4FTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecVOXd///XlG2zvbOggqAXxa7YEAuiYkXsXWwxmuS+\nvfPL7feb3OmmmDsmt9Ff7hhJLGAvCKJiL7EgIogNwUvBBaRsYXvfnTnfP2YWlyLLlpkzM/t+Ph48\nmFPmnM+cnTP73utccy4QERERERERERERERERERERERERERERERERERERERERERERERFJaJ7eVjDG\n3A4cCTjATdbapT2WlQPrgGBk1mXW2o1RqFNEREREEpx/VwuNMccD+1hrJxljxgH3ApN6rOIAp1pr\nW6JYo4iIiIgkAW8vy08E5gFYa1cB+caYrO3W6bXVVERERESkt+A5DKjuMV0FlG23zt+NMW8ZY24d\n1MpEREREJKn0Fjy35yF8eb3bz4EfAicA+xtjzhukukREREQkyeyyjyewkXCrZ7fhwKbuCWvtg92P\njTELgQOAubvaYGdnl+P3+/peqYiIiEiMeTwedSkcRL0Fz5eAXwOzjDGHAhustc0AxphcYAHhLxe1\nAscBT/a2w9ra+P0eUnFxNlVVjW6XkbB0/AZGx29gdPz6T8duYHT8BkbHb2jZZfC01r5rjFlmjHmH\n8C2Tvm+MmQnUW2vnG2PmAouMMU3AcmvtLls7RURERGTo6q3FE2vtT7ab9UmPZXcCdw52USIiIiKS\nfPr65SIRERERkX5R8BQRERGRmFDwFBEREZGYUPAUERERkZhQ8BQRERGRmFDwFBEREZGYUPAUERER\nkZhQ8BQRERGRmFDwFBEREZGYUPAUERERkZhQ8BQRERGRmFDwFBEREZGYUPAUERERkZhQ8BQRERGR\nmPC7XYCIiIjEp2AwSHn5mqjuo7Y2i5qapqhtf9So0fh8vqhtX/pGwVNERER2qrx8DTfdtoBAbonb\npfRLS30ld9w8nTFj9nW7FIlQ8BQREZFvFcgtISt/hNtlSJJQH08RERERiQkFTxERERGJCQVPERER\nEYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERER\niQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJv9sFiEjyCwaDlJevieo+\namuzqKlpisq2R40ajc/ni8q2RUSGEgVPEYm68vI13HTbAgK5JW6X0mct9ZXccfN0xozZ1+1SREQS\nnoKniMREILeErPwRbpchIiIuUh9PEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERER\niQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJ\nCX9vKxhjbgeOBBzgJmvt0p2scytwlLV2yuCXKCIiIiLJYJctnsaY44F9rLWTgGuBO3eyzgTgWMLB\nVERERERkp3q71H4iMA/AWrsKyDfGZG23zm3AfwGewS9PRERERJJFb8FzGFDdY7oKKOueMMZcBbwG\nrB30ykREREQkqfTax3M7HiKX1I0xBcDlwDRgz0GuS0RERESSTG/BcyPhVs9uw4FNkcdTIsveBtKA\nMcaYP1trf7SrDebnB/D7ff0sN/qKi7PdLiGh6fgNTLIev9ra7XvoJJaCgqyk/dl0S/bXF23JevwS\n/dyFoXH+JpLegudLwK+BWcaYQ4EN1tpmAGvtXGAugDFmJHB/b6EToLa2ZWAVR1FxcTZVVY1ul5Gw\ndPwGJpmPX01Nk9slDEhNTVPS/mwgud97sZDMxy/Rz11I/vM30eyyj6e19l1gmTHmHeAvwPeNMTON\nMTO2W3XrJXgRERERkZ3ptY+ntfYn2836ZCfrlBP+BryIiIiIyE5p5CIRERERiQkFTxERERGJCQVP\nEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8R\nERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxER\nERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPERER\nEYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERER\niQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJ\nCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJBU8RERERiQkFTxERERGJCQVPEREREYkJ\nBU8RERERiQkFTxERERGJCX9vKxhjbgeOBBzgJmvt0h7LvgNcAwSBj6y1349WoSIiIiKS2HbZ4mmM\nOR7Yx1o7CbgWuLPHsgBwETDZWjsZGGeMOTqaxYqIiIhI4urtUvuJwDwAa+0qIN8YkxWZbrHWnmSt\nDUZCaC6wKarVioiIiEjC6i14DgOqe0xXAWU9VzDG/Bj4EnjMWls+qNWJiIiISNLotY/ndjyE+3pu\nZa39gzHmL8BCY8w71tpFu9pAfn4Av9/Xx93GTnFxttslJDQdv4FJ1uNXW5vldgkDUlCQlbQ/m27J\n/vqiLVmPX6KfuzA0zt9E0lvw3Ei41bPbcCKX040xBcCB1to3rLVtxpjngWOAXQbP2tqWAZQbXcXF\n2VRVNbpdRsLS8RuYZD5+NTVNbpcwIDU1TUn7s4Hkfu/FQjIfv0Q/dyH5z99E09ul9peA8wGMMYcC\nG6y1zZFlKcA9xpjMyPQRwKqoVCkiIiIiCW+XLZ7W2neNMcuMMe8QvmXS940xM4F6a+18Y8wtwOvG\nmC7gQ2vtMzGoWUREREQSUK99PK21P9lu1ic9ls0GZg92USIiIiKSfDRykYiIiIjEhIKniIiIiMRE\nX2+nJCISVy7a8DKjWsNjV5RnlPHYiJNdrkhERL6NWjxFJGFdtOFl9m7dhIfwTYb3bt3E9756ktK2\nLW6XJiIiO6HgKSIJKTXUubWls6ecYAvnbXrdhYpERKQ3utQuIvHJccgKtpLX2UheZxP5nY2Rx43k\ndTWRGWxzu0IREekjBU8RcY3XCZLb2bxNqMzvCgfNvM5GUpzgDs8J4qU+JZPNaQUUd9SR07XtaGid\nHh8LS46O1UsQEZE+UPAUkagKtjTjbNrEfu0VlNXWhMNlJFhmd7XgxdnhOW3eFLak5lLnz6Y2JYu6\nlOyt/xr8ARzPN72EvvfVk+QEw+GzCy8pTpAZFW/xatFEPskeAx5PzF6riIjsmoKniAyIEwrRVVdH\nZ1Vl+F9lJZ3VVXRUhqdDzeFRdi8B6DHsc4MvwNfpJdRFgmVtJFjWpmTR5k3b7cA4t2zK1j6dc8tO\noKx9C1OqP+CMykWMbyrnheKjaEjJGuRXLSIi/aHgKSK9CnV20FlVHQmXVTuETKera4fnePx+/EVF\nZIweQ0tqKs+urKc1dw9qU7Kp92fR5R2cj5+K9EL+tvf5PaaLWBMYwbSqxYxp2ci16xbwRtFhLM8x\nav0UEXGZgqeI4DgOoeZmOqsq6egOlFsDZhVddbXg7HhJ3BvIJHWPPUktLialuISU7v9LSvDn5ePx\nhi+Jr179Be+WLyYrc0RMXk9DShZPlE1l/8Y1nFT9PtOq3mN8YzkLS46mLjUnJjWIiMiOFDxFhggn\nFKKrtobOyki47A6W3ZfEW1t3fJLHgz8/nwwzlpTiElJLSrYJmL7MzNi/kN3l8fBpzhi+CpQxreo9\nTPN6rl3/DG8WHMLSvHHb9BMVkdjT4A/9Z4wZBXwCLO0xeznwF2Ae8Jq19ubIuinAX4H9ga7Iv6us\ntetjWXM3BU+RJBJqb6ezuiocKrcGzEjIrK6C4I7fEvekpISDpBn7TYtlJGT6C4vwpqS48EoGT7M/\nwFPDTmBc01pOrl7C1C1LGddUzsLSSWxJzXO7PJGhwXHwO0FSnC5SQ51M3/wWI9qrty7uHvxhbtkU\nKtILXSw0Ot6efu7LwNTI5KuTFzw1GCl7lbV2Ss8ZxpgXgReBnh/clwJd1tpjIuvMBL4H/KSvOzTG\neKy1O17+6gMFTxGXff3n22hZ9RkWCIybwB4/uvlb13Uch2BT49ZWyp6XwzsqKwnW1+30ed6sLNL3\nGhm5DP5NuEwpLsGfm7v1knjS8nhYlT2KtYFhnFS1hP2ayrl63bO8U3AQ7+XvR0itnyJA+DPG6erE\naWsn1NGOU1XFHp315LZ4t4bGlFAXqU5X+P9QZ2R+Fyk9HnfPD6/TRYrTtdM7WPTUPfhDzz7bySAS\nOk/qMeukt6ef+zUwffKCpz4Y5N2dC5xPuHWzWy6Q3T1hrZ3d/dgYcwXwb0AI+B9r7ePGmAuBHxJu\nGV1mrf0PY8yvgL2BvY0xJwC/ASYDPuCv1tpHd7dABU8RF33959toWbli63TLyhWs+c8fUnz5FXhT\nUsOtl1tDZjhghtp2cuN0jwd/YSGB8RN26GuZUlSMLxCI4auKX62+dJ4Zdhwrm/dmWuVijq9Zzrim\ncp4rPYbKtAK3y5ME05c/GqPB6eoi1NZGqL2dUHsbTnt7+HFbWzg0toXnhyLznfY2QpFAGWqLTHc/\np8fzCYW22c8NAA19qAvo8KTQ6fXT4fXT4kunwxuZ9vjp9Kawf+NqkuWrfm9PP/c24IJdrDJyJ/NG\nAEsiAXRnnpi84Kk+v6Gstc3GmO1nPwhcZYxZBSwE5lpr3zHGZAM/Bw4A0oHZxpjngN8BB1lrW4wx\nCyJB0wFSrLXHGWOOBfay1h5vjEkDlhlj5llr23enRgVPERe1rPpsh3lddbVs+uudO8z3pKZuDZWp\n3aGyO2QWFuHx63TeXV9m7sn6vUo5sXopBzV+ycz1z7E4f38WFRxI0ONzuzxJADv9o/HmHzL8BzeR\nPnLUNus6XV3bhr+eYW9rOOzYNjxGQqLT0b5duOzYGiZ31nWmrzxpaXjT0vCmpePNzsaTmoY3PR1v\nWhqetDSa2tp4c2U1BPLp9Pjp8KbQ4fVHHodDZPfjjsjjLo+v1ztIZHW1sPd2Q942+ALMLZvyLc+Q\nnRhrjOk5PvBL1tpbYdtMb62tAQ41xkwGTgEeMcbcCzxH+HJ9O9AOzDDGHAp8Ya3tHpnjDeCQyOMl\nkf8nAUf12LcHKAPKd6do/aYSiUOe1FTyp50WDpiRy+O+nFw8uh3QoGn3pfJ86SRWZo/itMp3Oab2\nE0zzOhaWTGJTerHb5Umc2+kfjbW1rPv9b0gtKd0mOO7sdmN95UlNDYfD9DT8gYJtwuHW4Ng9HVkW\nnv7msTctHU96Gt5IuPSkpPTazWb16i94cd1isvIH944Uj404eZvBHxp8gYS9xB5pmfzW1smdXGoH\n2MDAL7V/vn0fz4ht+jQYY1IJ9/F8G3jbGPNPwoFyAbD9GyDEtsE1Fej+5mln5P924B5r7R/6U7SC\np4iLMvY1tNrPt5nnz8/faauJREd5YDj37DWd47d8wGH1n3PF1y/wft543io4eNDuNSrJJdTWutPb\niwEQDBJsasSTloY/P79HAEzbGhy9qWl40rcLhD0DZHpapOUxsiw1NSn7YW87+EPytnROXvDUyZFL\n6t3pfcPkBU/tEcVdbt9CcQ/wNnB3ZHpPYDWwinCraSYQJBxEZwD7GmOyrLVNwPHAb9k2OL8H/MkY\n899AGvBHa+2/725x+lQVcZG/YNt+hf78fEbfdrtL1QxdHd4UXi4+klVZozitchFH1n3Gvs3reb5k\nEivdLk7iRqitjbrXXqHmxed3utyfl8/wf9Mfjbtr+8Efktx0wsGu+/Fg2L5lc0/CfTjzgHRjzKmE\nB437IXB35ItEbUAHcGOkD+cvgFcim/ifyLybgReMMSHgrUh/0JO692etfTdymf1dwiH3f/tSdMyv\n21VWNgzoa/jRVFycTVVVo9tlJCwdv75p/vQTNvzlz6SUDcdpbcXr8zDse/+elL+0Vq/+gp/MGvzL\nddHgD3VxbM2HHF63Ei8Oi9NGMOn7l7DPhP17f3KC0rm7a6H2dureeI3a5xcSbGrEGwiQf8qp1L3x\nGsG68J0kkvWPxkQ6d3emqXYDt15/FGPG7NvvbZSU5KiP0yBSi6eIC0Lt7VQ8OBu8Xsq+813S9xqp\nX/5xosvr5/WiiazKGsnplYs4qn0DoX/Oovna68ncL3nDp+wo1NFB/RuvU/PCcwQbGvBmZFB49jnk\nTT0ZXyBA5gEHsvGvd+D1hv9oFJHeKXiKuGDLgnl0VVeTf+rppO+1szttiNs2pRdz/55nMnHjO0xp\nWseG2/9EzuRjKb7wYnyBOB6xSQYs1NlB/Zv/ombhcwTr6/Cmp1Nw5nTyT562zWhd6SNHMfq22/VH\no0gfKHiKxFjb2nJqX3qRlOJiCs862+1yZBeCHh+vBsYw9ZKppLzyCg1vv0XzJ59QesVMsg4+pPcN\nSEIJdXbS8Pab1Cx8lq7aWjxpaRScfib5p5yKLyvL7fJEkoKCp0gMOcEgFbPvA8eh5PKZeNPS3C5J\ndoOndBh7/dfPqXnxeWqeeZqNf72D7COOoviSS/Fn57hdngyQ09VF/TtvU/PcArpqasK3Mzv1dPKn\nnaqfr8ggU/AUiaHaV16ifd1aco4+Rv0FE4zH76fwjLPIOuRQKu6/l8Yli2n5bAUll15O1uFH6B6r\nCcjp6qJh8SK2PLuArupqPCkp5J88jfxTT8efm+t2eSJJScFTJEY6q6rY8vQ8fFnZFF94sdvlSD+l\nDR/Bnj/+KXWvvEz1/LlsmnUXme+/R+llV+LPy3O7PNkNTjBIw+J3qXn2aTqrqvD4/eSddDIFp56h\nn6FIlCl4isSA4zhUPDgbp6OD4iuuwped7XZJMgAer5f8U6aRedDBVMy+l+blH1D++SqKL7qEnEmT\n1foZp5xQiMYli9nyzNN0VlTg8fvJnTKVgtPPJCU/3+3yJHH5gDGDvM3VhG/q/q2MMd8HLic8klAG\n8F/W2ld3Z+PGmIOAc6y1vxpgnX2m4CkSA41LFtOy4lMCE/Yj+6ij3S5HBklqaSl7/Of/pf7NN6h6\n4nEq7ruHxiXvUXrl1aQUFrpdnkQ4oRCNS5dQs+BpOjZvAp+P3OOnUHDGmaQU6OckAzbmiHN/+Xkg\nt2RQNtZSX8mSp349FrDfto4xZhRwHTDRWhs0xhhgFrBbwdNa+xHw0SCU22cKniJRFmxqourRh/Gk\nplJyxUy1hiUZj9dL3gknknnAQVTMuY+WFZ9S/oufUnzBheQed0JSDnWYKJxQiKYPlrFlwXw6Nm4A\nr5ecY4+j8IyzSCkqdrs8SSKB3JJY32Q/F0gnPGRli7XWAidERhj6DeERiuqAC4HHCY9K9JYxJgP4\nDLiW8OhFFxhjvgTmAcdEnnMG4eE9nyDcmvomcKy1doox5k7gMMKtvHdZa2f3tXB9IopEWdXjjxJs\nbKTw7HNILR6cv4gl/qQUFjLiP35E6dXX4fF5qXxwDl//6b/pqKhwu7Qhx3EcGj9Yxtpbfsmmv/8v\nHZs3kXPMsYz63R8YNvMahU5JeJEWyyXAV8aY+4wxFxhj/IQD6WXW2ilAPTANmAucFXnqycCLbHsZ\nf29gtrV2EpAPHEh4mM1HrbUnEA63jjEmHzjdWnsMMBlI6U/tavEEvv7zbbSs+gwLBMZNYI8f3ex2\nSZIkWlZ+RsOit0nbayT5J53idjkSZR6Ph9xjJpO53/5UPDib5g+Xs/bXP6doxnnknXSyWj+jzHEc\nmj/6kC0L5tO+bi14PGQfPYnCM6eTWjrM7fJEBpW1dqYxZhzhcPl/gBuBW4BZkRA6GngNeDqy/P8A\nZwOPbLepBmvtp5HHXxMOr+N6rLcAOMJaW2uMscaY+YRbQ+f0p+4h/yn49Z9vo2XlCnAccBxaVq5g\nzc0/pG1tudulSYILdXRQMed+8HgovfJqPD6f2yVJjPjz8hj+/X+n7Pob8aamUfX4I6z/w+9o37jR\n7dKSkuM4NH38Eet+dwsb/3oH7evXkX3kUYz6ze8pu/Z6hU5JSsaYNGvtKmvtHcCRwJ7AvcD3Iy2V\nTwNYa+uBDZF+oEcTDqM9+3x1bbdpT+Sfs/0+rbWnA78GDgae6U/dQ77Fs2XVZzvM66qtZeNf72D0\nbbe7UJEkiy3PPE1nVSX5J08jfdQot8uRGPN4PGQfcSQZ48dT9cjDNC5ZzLpbfkHBWWdTMO00PP4h\n//E7YI7j0PLZCrY8/RRta9YAkDXxCAqnn03a8Jj2txOJKWPMdcAUY8zl1loHyCMcFrOB9caYPOBE\n4OPIU+YBPwMWWWtD4Qy6S6uBicAy4LTIPkcCZ1tr7wSWG2OW9qd2ffKJREH7+nXUvvg8/sJCCmec\n63Y54iJ/dg5l199A9uFHUPHgHLbMm0vTsqWUXnUN6XuNdLu8hOQ4Dq2rVlL99DzavvwCgKzDJlJ4\n1tmk7bGny9XJUNRSXxnrbd0LjAUWG2OaCPe3/DfgKOAdwsHxj8AvjTELCLd+/v+EL7VDuDXT6fG4\nJwe4A3jcGHMe4b6kQWAjcLQx5iLCXzq6pz+vb8gHz8C4CeFL7T14AwGG/+AmlyqSROeEQmyefR+E\nQpReoWExJSzrkEPJMGOpeuJRGt5+i3W/u4WC086g4Iyz8Kb0q4/+kNTy+Sq2PD2PVvs5AJkHH0Lh\n9BkK8eKm1ZHbHw3qNne10FobAnb2hZTngV/2mO75rfOtv4ystf8C/hV5XNJj/gUAxpgJwA+stYuM\nMZcARdbaTuCSPr6OHQz54LnHj25mzc0/pKu2NjzD48Hp7FR/POm3utdeob38K7KPPIrM/Q90uxyJ\nI77MTIZddS3Zhx9Jxez7qHl2AU0fLKX0qmvJGL3z+08Hg0HKy9dEta7a2ixqapqitv1Ro0bjG+Bn\nausXluqn59G6aiUAmQceROH0c9SNReJBkF3cczNBNQJ3G2Mcwq/v6sHa8JAPngDDf3ATG/96B16v\nh5yp06h+/BE23X0Xe/3sl2qtkj7p3LKF6nlz8WZmUnzRpW6XI3Eqc7/9GXXLb6ma+wT1r7/G+lt/\nS/4p0yg8+1y8qanbrFtevoabblvAYN2cOtZa6iu54+bpjBmzb7+e37r6S7Y8PY+Wz8JXpgL7H0Dh\n9HPIGD16MMsUkR6steuBY6OxbQVPIH3kKEbfdjvFxdlUVTXSVVNN3SsvU/noQwybeY3b5UmCcByH\nyofm4LS3U3Lp5fhzctwuSeKYNz2D0suuJHviEVTMvo/aF1+g6cPllM68hoDZ9qqdCzendl3bV2uo\nfno+LZ+GvxsRGL8fhWfPIGOf/gVYEYkPCp47UXTehbRaS8NbbxIYP4GcI45yuyRJAE3vL6H544/I\nGDeenEmT3S5HEkRg7DhG/vIWtjw9j9qXX+TrP95K7pSpFJ93Pt70DLfLi7m2teVseXoezR+HR/PL\nGDeewukzdgjjIpKYFDx3wpuSQtl3b2TtLb+ics79pO89WiPOyC4Fm5upfOQhPCkplF5xlYbFlD7x\npqVRfOHFZB02kYr776X+9Vdp/vhDSq+8GtKHRnef9vXrqF4wn+blHwCQsa+h8OxzCIwb73JlIjKY\nFDy/RWrpMEovv5LN98wK9/f88U913z35VlVPPEawsYGic88ntbTU7XIkQWWM2Ye9fvFrap5bQM3C\n59hw+5/wHHgQ6aFct0uLmvYNX7NlwXyaloVvCZg+Zp9w4Bw/QX/AiSQhJaldyDl6Ei0rV9Cw6B2q\n5z1J8QUXu12SxKGWVStpePtNUvfYk/xTTnW7HElw3pQUimacR9ahE6m47x7aP/6If/ek8VK6jy8z\nk+cele0bN1LzzHwal74PjkP63qPDgXO//RU4RZKYgmcvSi69gtbVq6l98QUC4yaQeYBujyPfCHV2\nUPHA7G+GxVSruAyS9L1GstdPf8GXDz9A4M03OX/T66zI2ptXig+n1Zfudnn91rF5E1ueeZrGJe+B\n45A2chSFZ88g84CDFDhFhgD9luyFNz2dsu/eyPrf/4bN9/yDkb+6BX9evttlSZyoefYZOis2k3fS\nybq9iww6j9+P95jJ/O9HbZzftpr9mr5iVOsmXi46glVZIyGBglpHRQU1zy6gYfGicODcc08Kzz6X\nzIMOVuAUGUIUPHdD+l4jKbrwYqoefpBN/5zFHv/fzXi8XrfLEpe1b/iamhcW4i8ooEjDYkoUVfmz\neHCPU5lYt4rjapYzo+JNPm/ai5eKj6DZH3C7vF3KD7YSeu4Zyj/9FEIhUkfsQeH0GWQdcqg+R0WG\nIAXP3ZQ3ZSotn62g+cPl1Cx8lsIzp7tdkrjICYWomH0fBIOUXH7lkLztjcSW4/Hyfv4Evsjcg9Mr\n32Vs8zr2at3Mq0WH82n26Lhr/czpbGJS7Scc0PAlTp1D6vDh4cB56EQFTpEhTGf/bvJ4PAy76lr8\nBQXhUTQi4wTL0FT/xmu0rVlN1sQjyDrwYLfLkSGkLjWHh0ecwovFR+J1QpxZ+Q4XbHqVnM7oDXnZ\nF9mdzZxSuZjvrp3PwQ1fUOPNwDN9BiN/9VuyJx6h0CkyxPXa4mmMuR04EnCAm6y1S3ssmwL8nvA4\nnp8D11lrnSjV6jpfVhZl37mB9X+8lc3/uJuRv7wFX1aW22VJjHXW1FD91JN4AwFKLtGwmOICj4fl\nuWNZHRjBqVWLGdOykWvXPcPrRYfxYc6+rrR+ZnW1cHTtJxxU/wV+QtSkZPNOwUEs6Uzl9/vtp8Ap\nIkAvLZ7GmOOBfay1k4BrgTu3W2UWcL61djKQDST9vWS6b2rcVVvD5vvvwXGSNmfLTjiOQ+XDDxBq\na6P4govw5+a5XZIMYQ0pWTxeNpXnSibheDycWrWYSza+TF5nY8xqyOxqZWrV+9yw9ikOq/+cJn+A\n50om8Y+9zmZF9micOOsCICLu6q3F80RgHoC1dpUxJt8Yk2Wt7b6mc5i1tiHyuAooiFKdcaXg9DNp\nWbWS5g8H3k6zAAAc40lEQVSXU/f6q+SfeJLbJUmMNH2wlOYPl5NhxpIz+Ti3yxEBj4dPcvbhq8Bw\nTql6D9O8nmvXLeBfhYewLHccjic6LY2BrlaOrFvBofWfk+IEqfdn8k7BgXyaPYZQlPYpIomvt0+H\nYUB1j+kqoKx7ojt0GmPKgFOAhYNdYDzyeL2UXXc9vqxsqh9/lLZ1a90uSWIg2NJM5cMP4vH7w/fs\nVEuOxJEmf4Cnhp3A/NJj6fT4Oal6KZdteJHCjrpB3U9GsI3jq5dxw9p5HFn3GS2+NF4oPoq7R87g\n45x9FTpFZJf6+q12D+G+nlsZY0qABcCN1tra3jaQnx/A7/f1cbexU1ycvZsrZpP2w39j5W9+T+U/\n7+bg//kjvgx9s3m3j18C+vJvDxOsr2evyy5hxAH7RmUfyXr8amsTuy90QUGWaz+bPh07j4dV2Xuz\nNlDGyVVLmNBUztXrnuXtgoN4L3+/AbV+pgfbOaLuMw6rW0ma00WjL4PXCw7l45x9CXp2/Znu5vGL\nlWR9fYl+7sLQeP8lkt6C50bCrZ7dhgObuieMMTmEWzn/y1r7yu7ssLa2pa81xkxxcTZVVX3oGzXS\nkH/KqdS+9AKf3XkXw675TvSKSwB9Pn4JpMV+TsWLL5E6fARpx06NyutM5uNXUxMf37jur5qaJtd+\nNv05dq2+dBYMO46VTaOYVvUeJ9QsZ1zzWhaWTKIyrW89otKCHRxe9xmH160kzemkyZfBm/mH8GGO\nIejdvUYEN49fLOjcjW/J/v5LNL0Fz5eAXwOzjDGHAhustc09lv8ZuN1a+1K0Cox3ReeeT4v9nIZF\n7xAYP4Gco49xuyQZZKHOTirn3B8eFnOmhsWUxPFF1l6syyhlavVSDmxczcz1z7E4/wAWFRzQaytl\nWrCDifUrObzuM9JDnTT70nk7/yCW5xi6vDoHRKR/dvnpYa191xizzBjzDuFbJn3fGDMTqAdeBK4A\n9jHGXBd5ysPW2n9EteI44/H7Kbv+Rtbd8gsqHpxD+t5jSB02rPcnSsKoWfgsHZs3kTtlKhlj9nG7\nHJE+afelsbD0GFZmjeLUqnc5pvZjTPM6FpZMYlN60Q7rp4Y6OaxuFUfUrSAj1EGLN43XCg9lee5Y\nOr0pLrwCEUkmvf7Zaq39yXazPunxOH1wy0lMqSUllFx5FZtn/Z1Ns+5iz5/8DG+KPqCTQfvGjdQs\nfBZfXj6NhxxC0+ovorav2tqsqF7WGjVqND5f/Pavluj6KnME92RM54TqDzi0wXLF18/T4A+Q2xW+\niLUuo5Q1gREcWbuCQKidVm8qb0S+GZ/IgTMYDFJeviaq+9C5K7L7dL1kkOQccRQtn31Gw9tvUj33\ncUouvsztkmSAnFCIijnhYTGdqVP5jztfJJBb4nZZ/dJSX8kdN09nzJjofClKEkOHN5WXSo5iZfYo\nLtz4Knld3/ScGtlawcjWCto9ft4sOJileePo8Ka6WO3gKC9fw023LdC5KxInFDwHUckll9G2+kvq\nXnmZwLgJZB18iNslyQDUv/kGbV9+QdZhE2k1Ywnk1pKVP8LtskQGbH3GMPxOcKfLOrwpLCo4MMYV\nRVcgt0Tnrkic0A3XBpE3LY2y796Ix+9n833/pLOmxu2SpJ+66mqpnvsE3owMSi5R67UMHQ66P62I\nRI+C5yBL22NPii++lFBzM5v/8Xec4M5bFSS+VT78IKHWVorOvxB/Xr7b5YgMuvKMsh3mNfgCzC2b\n4kI1IjJUKHhGQe7xU8KXZ7+wbHl2gdvlSB81LV9G0wfLyNjXkHvs8W6XIxIVj404mQZfYOt0gy/A\n3/Y+n4r0QherEpFkp+AZBZ7I/R79hYXUPLuAllUr3S5JdlOwtXXrsJglV1yFx6tTRJLX3LIpNPgC\naukUkZjRb9Uo8QUyKbv+RvB42PTPuwk2atSERFD91JN01dZScPqZpA0f7nY5IlFVkV7I3/Y+Xy2d\nIhIzCp5RlDFmH4rOOY9gXR2b7/snjuP0/iRxTevqL6l/4zVSy4aTf9oZbpcjIiKSdBQ8oyx/2mkE\nJuxH88cfUffKkB1ZNO45XV1UzL4PHIfSK6/SAAAiIiJRoOAZZR6vl2HXXo8vJ4eqJx+nrbzc7ZJk\nJ2peWEjHxg3kHn8CGfsat8sRERFJSgqeMeDPzWXYtddDMMimu/9GsLXV7ZKkh47Nm6h5dgG+3DyK\nzrvA7XJERESSloJnjGTutz/5p51BZ1UllQ/OVn/POBEeFvN+nK4uSi69DF8g0+2SREREkpaCZwwV\nnX0O6aPH0PjeYhoWve12OQI0vP0WrfZzMg8+hKxDJ7pdjoiISFJT8Iwhj99P2fU34M3IoPKhB+jY\ntNHtkoa0rvo6qp58DG96OiWXXoHHo6ECRUREoknBM8ZSioopnXkNTkcHG+++i1BHh9slDVmVjzxM\nqKWFonPPJ6WgwO1yREREkp6CpwuyJx5O7vFT6Ph6PVVPPOp2OUNS00cf0rR0Celj9iH3hBPdLkdE\nRGRIUPB0SfFFl5A6Yg/qX3+NxmVL3S5nSAm1tVL50Bzw+Si9UsNiioiIxIp+47rEm5pK2XdvxJOa\nSsXse+ncUu12SUNG9byn6KqpoeC000kbsYfb5YiIiAwZCp4uShs+gpJLLiPU0sKmWX/H6epyu6Sk\n17pmDXWvvUJK6TAKzjjL7XJERESGFAVPl+VMPo7sI46kbfWXbFkw3+1yklp4WMx7ewyLmep2SSIi\nIkOKgqfLPB4PJZfPJKW4mJrnn6P5sxVul5S0al96gY4NX5Nz7HEExo5zuxwREZEhR8EzDvgCAcqu\nvxG8XjbfM4uu+nq3S0o6HRUVbHnmaXw5ORSff5Hb5YiIiAxJCp5xIn3v0RSdez7B+no23/sPnFDI\n7ZKShuM4VDxwP05nJyWXXI4vU8NiioiIuEHBM47knzyNwP4H0rLiU2pfesHtcpJGw6K3aV21kswD\nDyJr4uFulyMiIjJkKXjGEY/Xy7BrrsOXm0f1vLm0rlntdkkJr6uhgarHH8WTlkbJZVdqWEwREREX\nKXjGGX9ODmXXXQ+hEJtm3UWwpdntkhJa1WMPE2pupuic80gpLHS7HBERkSFNwTMOBcZPoOCMM+mq\nrqZizv04juN2SQmp+ZOPaXxvMel7jybvxJPcLkdERGTIU/CMU4VnzSB9n31pWvo+9W/9y+1yEk6o\nvZ2KB2dHhsW8WsNiioiIxAH9No5THp+Psu/cgDeQSdUjD9G+YYPbJSWULfOfomvLFvJPOZW0Pfd0\nuxwRERFBwTOupRQWMuzqa3A6O9l0998Itbe7XVJCaCv/itpXXiKluITCs852uxwRERGJUPCMc1mH\nHEbeiVPp2LiBqscecbucuOcEg1TMvu+bYTFTNSymiIhIvFDwTABFF1xE2p57Uv/mGzS+v8TtcuJa\n7csv0r5+HTmTJhMYP8HtckRERKQHBc8E4E1Jpey738OTlkbFnPvorKpyu6S41FFVyZYF8/FlZ1N8\n4cVulyMiIiLbUfBMEKnDyii97EpCra1smnUXTleX2yXFFcdxqHxgNk5HB8UXX4ovK8vtkkRERGQ7\nCp4JJGfSMWQfPYm2r9ZQPf8pt8uJK42LF9Hy2QoC+x9A9hFHuV2OiIiI7ISCZ4IpvewKUkpLqX1h\nIc2ffuJ2OXEh2NhI5WOP4ElNpfRyDYspIiISrxQ8E4w3PYOy62/E4/ez+Z5ZdNXVuV2S6yoff4RQ\nUxNFM84lpajY7XJERETkWyh4JqD0kaMoOv8igo2NbL5nFk4o5HZJrmle8SmN7y4ibeQo8qae7HY5\nIiIisgt+twvYXcFgkPLyNVHdR21tFjU1TVHb/qhRo/H5fIOyrbypJ9GycgXNH31I7QsLKTj9zEHZ\nbiIJtbdT+eBs8HopnXk1nkE6tiIiIhIdCRM8y8vXcNNtCwjklrhdSr+01Fdyx83TGTNm30HZnsfj\nYdjV17H21z+nev5TZJixZOwzONtOFFueeZrOqiryp51K+l4j3S5HREREepEwwRMgkFtCVv4It8uI\nG76sLIZ95wa+vu0PbJr1d0b+8hZ8mZlulxUTbevWUvvSC6QUFVM4/Ry3yxEREZHdoD6eCS5gxlI4\nfQZdNVuomH0vjuO4XVLUbR0WMxSi5IqZeNPS3C5JREREdoOCZxIoOOMsMsxYmj5YRv0br7tdTtTV\nvfoK7WvLyT7qaDL329/tckRERGQ3KXgmAY/Xy7Dv3IA3K4uqxx6mff06t0uKms7qKqrnz8WblUXx\nRZe4XY6IiIj0gYJnkkjJz2fYNdfhdHWx8e6/EWpvd7ukQec4DhUPzsHp6KDkwkvwZ+e4XZKIiIj0\ngYJnEsk68GDyTp5G5+bNVD78oNvlDLrGJe/R8uknBCbsR/bRk9wuR0RERPqo12+1G2NuB44EHOAm\na+3SHsvSgVnAeGvt4VGrUnZb0bnn02o/p+GdtwiMH0/OUckR0IJNTVQ9+hCe1FRKrpipYTFFREQS\n0C5bPI0xxwP7WGsnAdcCd263yh+BJVGqTfrBm5ISHlIzLZ2KB+bQUVHhdkmDouqJxwg2NlJ41gxS\nixPzXq4iIiJDXW+X2k8E5gFYa1cB+caYrB7LfwI8E6XapJ9SS0spvXImTnsbm+7+G6HOTrdLGpCW\nlZ/R8M5bpO25F/mnTHO7HBEREemn3oLnMKC6x3QVUNY9Ya1tBnTNMw7lHHk0OcccS/u6tVQ/9aTb\n5fRbqKODigdmg8ejYTFFREQSXF+/XOQh3NdTEkDJpZeTMmwYdS+/SNNHH7pdTr/UPLuAzsoK8k46\nhfRRe7tdjoiIiAxAb18u2ki41bPbcGDTduv0KYjm5wfw+/vealVbm9X7SnGuoCCL4uLsGO4xm6wf\n38xHN/+YyvvvYfgdfyatsDCqexzM19dcXk7ti8+TVlLMuGuvwJeRMWjb7iu9/wYm0Y+fjt3A6PgN\njI7fwLh5/GRHvQXPl4BfA7OMMYcCGyKX13vq06X22tqWvqy+VU1NU7+eF09qapqoqmqM7U6zCim+\n8GIqH3qAFX/4M3v85//F443OXbSKi7MH7fU5oRDr//K/OMEgRZdcQU1TFzTF+Nj1oPffwPedyHTs\nBkbHb2B0/AbGzeMnO9plArHWvgssM8a8A/wF+L4xZqYxZgaAMeYV4AVgP2PMJ8aYq6NesfRZ7gkn\nknXIYbTaz6l5LjG+C1b3+qu0fbWG7COOIvOAA90uR0RERAZBr/fxtNb+ZLtZn/RYdtKgVySDzuPx\nUHrVNbStLWfLgvlkjB1HwIx1u6xv1Vmzheqn5uINZGpYTBERkSSikYuGCF9mJmXfuQE8Hjb/4+8E\nm+Lz8onjOFQ+OAenvY3iCy/Gn5vrdkkiIiIySBQ8h5CMffel8Oxz6KqtZfN9/8Rx4u8GBU3L3qf5\n44/IGDeenGMmu12OiIiIDCIFzyGm4LQzCIyfQPNHH1L36itul7ONYHMzlQ8/iMfvp1TDYoqIiCQd\nBc8hxuP1Muza6/FlZ1P95GO0rS13u6Stquc+TrChgYKzzia1dFjvTxAREZGEouA5BPnz8hh27Xdw\nurrYNOsuQm2tbpdEy+erqH/zX6SO2IOCaae5XY6IiIhEgYLnEJW5/4HkTzuNzooKKh56wNVaQp0d\nVDxw/zfDYvp7vdmCiIiIJCAFzyGs6JzzSB89msZ3F9Gw6B3X6qh57lk6N28mb8pUMkaPca0OERER\niS4FzyHM4/dT9p0b8WZkUPHQHDo2bz8aavS1b9hAzfPP4c8voOjc82K+fxEREYkdBc8hLqW4mNKZ\nV+O0t7Pp7rsIdXbEbN9OKETFnPsgGKTksivwprs3FruIiIhEn4KnkD3xCHKPO4H29euofuLxmO23\n/l+v07b6S7ImHk7WwYfEbL8iIiLiDgVPAaD4oktIHT6CutdeoWn5B1HfX2dtLdVzn8CbkUHJxZdF\nfX8iIiLiPgVPAcCblkbZd7+HJzWVzffdQ2fNlqjur/LhBwi1tVF0wUX48/Kiui8RERGJDwqeslXa\niBEUX3wpoZZmNv/jbpxgMCr7afxgGc3LPyDDjCV38nFR2YeIiIjEHwVP2UbusceTNfEIWr+wbHnm\n6UHffrClhcqHHvhmWEyv3oIiIiJDhX7ryzY8Hg+lV16Fv6iImueeoWXVykHdfvXcJwjW11Fwxlmk\nlg0f1G2LiIhIfFPwlB34AgHKrr8RvF42/eNuuhobBmW7rV9Y6v/1OqnDh1Nw2hmDsk0RERFJHAqe\nslMZo8dQdM55BOvr2HzPP3FCoQFtL9TZScWc+wEovVLDYoqIiAxFCp7yrfJPOZXAfvvT8unH1L78\n4oC2VfvCQjo2bST3hBPJ2GffQapQREREEomCp3wrj9fLsGuvx5ebS/VTT9L21Zp+bad940ZqnnsG\nX14eReeeP8hVioiISKJQ8JRd8ufkMOza6yEUYtOsuwi2tPTp+U4oROUD9+N0dVF62RX4AoEoVSoi\nIiLxTsFTepU5YT8KTjuDzqqqcIh0nN1+bv1bb9L6hSXrkMPIOuSwKFYpIiIi8U7BU3ZL4fQZpI/Z\nh8b3l9Dw9pu79Zyuujqqn3wMb0YGxZdeHuUKRUREJN4peMpu8fj9lF1/A95AgMpHHqJ944Zen1P5\nyIOEWlspOvcCUvLzY1CliIiIxDMFT9ltKYVFlM68Bqejg01330Woo+Nb1236cDlNy5aSPmYfco8/\nIXZFioiISNxS8JQ+yT5sIrlTTqRjw9dUPfbITtcJtrZS+dAc8PkonXm1hsUUERERQMFT+qH4wotJ\n3WNP6v/1Oo1L399h+ZZ5T9JVW0vB6WeSNnyECxWKiIhIPFLwlD7zpqRSdv2NeFJTqZh9L53VVVuX\nta7+krrXXyNl2DAKTj/TxSpFREQk3ih4Sr+kDR9OyaVXEGptZdOsv+N0dX0zLKbjUHrl1XhTUtwu\nU0REROKIBsyWfss5ZjItK1fQ+N5ivrjhOr6IzM897gQCZqyrtYmIiEj8UYun9JvH46Grrm6H+U0f\nfUjb2vLYFyQiIiJxTcFTBqTVfr7DvGB9HRv/eocL1YiIiEg8U/AUERERkZhQ8JQBCYybsMM8f34+\nw39wkwvViIiISDxT8JQB2eNHN+PvMRymPz+f0bfdTvrIUe4VJSIiInFJwVMGbPgPbsKfn09qYYFa\nOkVERORb6XZKMmDpI0cx+rbbKS7Opqqq0e1yREREJE6pxVNEREREYkLBU0RERERiQsFTRERERGJC\nwVNEREREYkLBU0RERERiQsFTRERERGJCwVNEREREYkLBU0RERERiQsFTRERERGJCIxcNEcFgkPLy\nNVHdR21tFjU1TVHb/qhRo/H5fFHbvoiIiESXgucQUV6+hptuW0Agt8TtUvqlpb6SO26ezpgx+7pd\nioiIiPRTr8HTGHM7cCTgADdZa5f2WHYS8DsgCCy01v42WoXKwAVyS8jKH+F2GSIiIjJE7bKPpzHm\neGAfa+0k4Frgzu1WuQM4FzgGOMUYMz4qVYqIiIhIwuvty0UnAvMArLWrgHxjTBaAMWY0UGOt3WCt\ndYCFwNRoFisiIiIiiau34DkMqO4xXRWZ172sqseySqBs8EoTERERkWTS1y8Xefq5bFC01FdGexdR\nEw+1x0MN/RUPtcdDDf0VD7XHQw39EQ91x0MN/RUPtcdDDf0VD7XHQw39lci1D0nGmF8aY67vMb3a\nGJMZeTzSGLNou3W/50adIiIiIhL/ervU/hJwPoAx5lBgg7W2GcBauxbIiQRQP3BGZH0RERERkR30\nenncGHMrcBzhWyZ9HzgUqLfWzjfGHAv8d2TVJ621/xO1SkVERERERERERERERERERERERERERERE\nRERkUEX93pvxyBgzCvgE6B53Pi0yfSOwBlhH+MtUAI619sRY1xjPIsfvCWvt4T3m/Qq4FNhA+H2V\nAdxqrZ3vRo3xbLv3nwfoAn4PDCc8NG06sB+wLPKUK62162NfafQN9Fw0xlwF/Ab4kvCxbAeusNZW\nGmPuB4qttWf0WP904FlgFOG7eux039ba0CC+zKgZ6LlojCknfIy7gCzgHmvt3ZHtrgGOtNa+32P9\nJcAKa+3VkeN7KLAlsp9U4GZr7TuD/kKjYKDn4VA+drtijNkX+AtQBPiARcB/ApOAW4AQkA08YK39\nizHmBOAJ4FPC52QT8FNr7YfGmP8kfMecPGAEsCKym1OstZ0xe1EyqPp6A/lksspaO6V7whhzH3AZ\n4ACnWmtbXKssMTnAX6y1fwMwxuQDHxpjnrfWtrtbWlza+v6LDD/7DHCRtXaKMWYk4btETNnlFpLH\nQM5FB3jEWvt/Is/9OXAN8IfI8lHGmDxrbV1k+nxgdS/7vhR4cICvyU19ORe3HmNjTABYY4yZFVm2\nBrgQeD+ynZFAfuQ53c/9sbV2YWT5aOB5YGz0XtqgG8h5ONSP3Q6MMT7gSeAH1tq3IvPuBH4BXAAc\nb63dbIxJB141xjwaeeob1toLIusfAjxpjJlsrf0T8CdjzPGRbV4Q69ckg6+3+3gOJe8B+7hdRILb\n2oJura0FNqFhVHtlrV0D/A74QWTWkLwS0UNfz8Wex2sY8HXksQO8ApwDYIxJAfYFdtV6vCSyTqLr\ny7nYvW4hUGWtdQgfu8XA1B7rnc8u7tUceR/nGGMS8v3bz/NQx25bJwMru0NnxM2EWzoLCLd0Yq1t\ns9YeY63dvP0GrLXLgXuBq3rMTvTjIj0oeLL1F9LZwAeRWXqTD5AxZixQyjchQHZtGTDB7SLc1o9z\n0QNcZIx53RjzCXAwMLfH8qcIt7RAOAi88W3bjOx7Ot9cWk0Ku3EuPm+M+Rfh1/2bHvM7geXGmCMj\n02cAC9n2+G19bIw5DtgYCV+Jqq/noY7dtsYCH/WcYa1tt9Z2AD8H3jfGzDfGfM8Yk7eL7SxFn4dJ\nayhfah9rjHk98vhA4A/W2qeNMXcQ/jDp7ldWZa290J0SE4oHuMkYcz6QQ7i/3CXW2i53y0oY2XzT\nl3GoGci56ACP9rjUfjlwN3BlZHk54ZakfOA84K/A5F72vWAQX5sb+noudl8uzgZeMcZ8BLRFlj0J\nXGiM2QDUEu5/13M/t0b64RVFll06+C8npvp6HurYbcsh3K9zB9bavxtj5gHTgBnAzyIjIu5MDkP3\n8zDpDeXg+XmPvj1PAF9E5quPZ/9s7VdmjBkGvEa4477snol808o31Oz2uRj54sZ9kckfRf7v2Yr0\nFPDb7bY/n/Clzv2ttR8ZY3Zn34nsW8/FyBePjgc+ttbe1PNJ1tpGY8wbwNFAdxh/BbgVWMu2Lcnd\n+/mxtXahMeZA4J+AjcYLiqFvPQ917HbLKr7pqgCAMSYVMMBqa20FMAeYY4y5FziF8PHZ3lD+PEx6\nutQedjPwh0gHcekfT+QfkX47c4BfulpRgjDGjAF+CNzudi1xYJfnorW23Fo7JfJvZ7+YjiT8y6+n\nJwmH1Dd2c98Zfaw53nzruWit/VXk2N203fpE+hceTo/jF/nm8HLC3/J+hh27KXTv52PCQeHGKLye\nmOjtPNSx2y0vAyONMWcCGGO8wB+B/wI+iLQMd88fzrZf9COybCJwLnBPrIqW2BrKwXNrXxprbTnh\nv0h/2nO+7NJ+xpiV3f8I3yqj57H7H2C6MWa8O+XFvbGRfomLgIeB71lre/bBG0rvw4Gei919PF8H\nfgb8e89tR7bZQDiA7s6+f9aH2uPBQM/F5yPH7l3gdWvt4sj87m08Aayz1jZG5vXcds/HPwNuNsYU\nDfD1xNJAz8OhfOx2EOmjOg243hjzPvAWUGutvZRw6+/LxpjXIvNfjdw+ygMcH/k5vAfcRvjOAj2v\nOm5/7EREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREZKD+H4wZ\n9ASVre8zAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd7f2f98a10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost-Sensitive Random Patches\n",
    "\n",
    "Ensemble learning is a widely studied topic in the machine learning community. The main idea behind the ensemble methodology is to combine several individual base classifiers in   order to have a \n",
    "classifier that outperforms each of them [17]. Nowadays,   ensemble methods are  one of the most popular and well studied machine learning techniques [18], and it can be noted that since 2009 all the first-place and   second-place winners of the KDD-Cup (https://www.sigkdd.org/kddcup/)   used  ensemble methods. The core \n",
    "principle in ensemble learning, is to induce random perturbations into  the learning procedure in order to produce several different base classifiers from a single  training set, then combining the \n",
    "base classifiers in order to make the final prediction.  In order to induce the random permutations and therefore create the different base classifiers,   several methods have been proposed, in \n",
    "particular: bagging,   pasting, random forests and random patches [19]. Finally, after  the base   classifiers are trained, they are typically   combined using either   majority voting,  weighted  voting    or  stacking [18].\n",
    "\n",
    "With the objective of creating an ensemble of example-dependent cost-sensitive decision trees, we first create $T$ different random subsamples $\\mathcal{S}_j$ for $j=1,\\dots,T$, of the training  set \n",
    "$\\mathcal{S}$, and train a $CSDT$ algorithm on each one. In particular we create the different [19]. In the random patches algorithm, base classifiers are created by randomly  drawn bootstrap subsets of both examples and features. \n",
    "\n",
    "Lastly, after the base classifiers are trained, we combine them using cost-sensitive weighted voting. This method is based in based in calculating the final prediction as a weighted average of the base classifiers estimation:\n",
    "\n",
    "$$  f_{wv}(\\mathcal{S},\\mathcal{M}, \\alpha)\n",
    "  =argmax_{c \\in \\{0,1\\}} \\sum_{j=1}^T \\alpha_j \\mathbf{1}_c(M_j(\\mathcal{S})), $$\n",
    "  \n",
    "where  $\\alpha_j$ is related to the performance of each classifier $M_j$ in the  out of bag set   $\\mathcal{S}_j^{oob}=\\mathcal{S}-\\mathcal{S}_j$\n",
    "\n",
    "$$  \\alpha_j=\\frac{Savings(M_j(\\mathcal{S}_j^{oob}))}\n",
    "  {\\sum_{j_1=1}^T Savings(M_{j_1}(\\mathcal{S}_j^{oob}))}. $$\n",
    "\n",
    "This method guaranties that the base classifiers that contribute to a higher increase in savings \n",
    "have more importance in the ensemble. For more details see [20]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             pre        f1       acc       rec       sav\n",
      "RF      0.492091  0.232172  0.934394  0.151926  0.122358\n",
      "LR      0.529412  0.037677  0.934854  0.019533  0.010948\n",
      "DT      0.251239  0.262626  0.899146  0.275095  0.191770\n",
      "RF-BMR  0.179759  0.282039  0.782493  0.654368  0.432752\n",
      "LR-BMR  0.115987  0.188429  0.717737  0.501899  0.228092\n",
      "DT-BMR  0.251639  0.260847  0.899819  0.270754  0.192115\n",
      "CSDT    0.160237  0.267374  0.711325  0.806837  0.465266\n",
      "CSRP    0.164478  0.272971  0.721102  0.801953  0.469644\n"
     ]
    }
   ],
   "source": [
    "from costcla.models import CostSensitiveRandomPatchesClassifier\n",
    "\n",
    "classifiers[\"CSRP\"] = {\"f\": CostSensitiveRandomPatchesClassifier(combination='weighted_voting')}\n",
    "# Fit\n",
    "classifiers[\"CSRP\"][\"f\"].fit(X_train, y_train, cost_mat_train)\n",
    "# Predict\n",
    "classifiers[\"CSRP\"][\"c\"] = classifiers[\"CSRP\"][\"f\"].predict(X_test)\n",
    "# Evaluate\n",
    "results.loc[\"CSRP\"] = 0\n",
    "results.loc[\"CSRP\", measures.keys()] = \\\n",
    "[measures[measure](y_test, classifiers[\"CSRP\"][\"c\"]) for measure in measures.keys()]\n",
    "results[\"sav\"].loc[\"CSRP\"] = savings_score(y_test, classifiers[\"CSRP\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAp4AAAE7CAYAAAB0T4FTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVOWd9/93Lb1vdDe9oCgt6I1LYlxwCYKIC5oYcWNR\nVHCZJ/NEM+PklyG/JCYxGZNJJkxizCSZLONG1KioLBoX3BU0LrjEUfBWsAERel/pvaqeP+p0U900\ndDddVaer6vO6Li67Tp0651u3Xd2fPvc53wMiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIi\nIiIiIiIJzTPUCsaYW4FTgBBwo7X2zYjnKoFtQMBZdIW19rMY1CkiIiIiCc6/vyeNMbOAw621040x\nRwJ3ANMjVgkB51lr22JYo4iIiIgkAe8Qz58JrASw1m4CCo0xuQPWGfKoqYiIiIjIUMGzHKiNeFwD\nTBiwzu+NMS8bY34a1cpEREREJKkMFTwH8hCeXu/1feAbwBnA54wxl0apLhERERFJMvs9xxP4jPBR\nz14HATt7H1hr7+n92hjzOPB54OH9bbC7uyfk9/tGXqmIiIhInHk8Hp1SGEVDBc+1wI+APxpjTgB2\nWGt3AxhjCoA1hC8uagdOBx4aaocNDWPjOqSSkjxqalrcLmPM0bgMTuMyOI3L3jQmg9O4DE7jMjiN\nS/Lab/C01r5qjNlgjFlPuGXSDcaYJUCTtXaVMeZh4BVjTCvwtrV2v0c7RURERCR1DXXEE2vtdwYs\nei/iuV8Dv452USIiIiKSfEZ6cZGIiIiIyAFR8BQRERGRuFDwFBEREZG4UPAUERERkbhQ8BQRERGR\nuFDwFBEREZG4UPAUERERkbhQ8BQRERGRuFDwFBEREZG4UPAUERERkbhQ8BQRERGRuFDwFBEREZG4\n8LtdgIiIiCSmQCBAZeWWqG+3oSGX+vrWqG6zomIyPp8vqtuUkVPwFBERkQNSWbmFG5etIbug1O1S\n9qutqZrbls5lypQj3C4l5Sl4ioiIDEFH9vYtu6CU3MKD47Y/SWwKniIiIkPQkT2R6FDwFBERGQYd\n2RMZPV3VLiIiIiJxoeApIiIiInGh4CkiIiIicaHgKSIiIiJxoeApIiIiInGh4CkiIiIicaHgKSIi\nIiJxoeApIiIiInGh4CkiIiIicaHgKSIiIiJxoeApIiIiInGh4CkiIiIicaHgKSIiIiJxoeApIiIi\nInGh4CkiIiIiceF3uwARib1AIEBl5Zaob7ehIZf6+taobrOiYjI+ny+q2xQRkbFBwVMkBVRWbuHG\nZWvILih1u5T9amuq5ralc5ky5Qi3SxERkRhQ8BRJEdkFpeQWHux2GSIiksJ0jqeIiIiIxIWCp4iI\niIjEhYKniIiIiMSFgqeIiIiIxIWCp4iIiIjEhYKniIiIiMSFgqeIiIiIxIWCp4iIiIjEhYKniIiI\niMSFgqeIiIiIxIWCp4iIiIjEhYKniIiIiMSFf6gVjDG3AqcAIeBGa+2bg6zzU+BUa+3s6JcoIiIi\nIslgv0c8jTGzgMOttdOB64BfD7LO0cBMwsFURERERGRQQ021nwmsBLDWbgIKjTG5A9ZZBnwX8ES/\nPBERERFJFkMFz3KgNuJxDTCh94Ex5mrgOWBr1CsTERERkaQy5DmeA3hwptSNMUXAlcC5wCFRrktE\nREREksxQwfMzwkc9ex0E7HS+nu08tw7IAKYYY35hrf3m/jZYWJiN3+87wHKjq6Qkz+0SxiSNy+AS\neVwaGgaeITN2FRXlJvRYQ2J/r8RSIo+LPkOD07jISA0VPNcCPwL+aIw5Adhhrd0NYK19GHgYwBgz\nCbhrqNAJ0NDQNrqKo6SkJI+amha3yxhzNC6DS/Rxqa9vdbuEYauvb03osU7075VYSfRx0Wdo3/tK\nFIn+syVZ7PccT2vtq8AGY8x64FfADcaYJcaYiwas2jcFLyIiIiIymCHP8bTWfmfAovcGWaeS8BXw\nIiIiIiKD0p2LRERERCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC4UPEVEREQkLhQ8\nRURERCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxF\nREREJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC4UPEVE\nREQkLhQ8RURERCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC4UPEVEREQkLhQ8RURE\nRCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxFRERE\nJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC4UPEVEREQk\nLhQ8RURERCQuFDxFREREJC4UPEVEREQkLhQ8RURERCQuFDxFREREJC78Q61gjLkVOAUIATdaa9+M\neO7/ANcCAeBda+0NsSpURERERBLbfo94GmNmAYdba6cD1wG/jnguG1gIzLDWzgCONMZ8MZbFioiI\niEjiGmqq/UxgJYC1dhNQaIzJdR63WWvPttYGnBBaAOyMabUiIiIikrCGCp7lQG3E4xpgQuQKxphv\nAx8DD1hrK6NanYiIiIgkjSHP8RzAQ/hczz7W2p8ZY34FPG6MWW+tfWV/GygszMbv941wt7FRUpLn\ndgljksZlcIk8Lg0NuW6XMGxFRbkJPdaQ2N8rsZTI46LP0OA0LjJSQwXPzwgf9ex1EM50ujGmCDjW\nWvuCtbbDGPMEcBqw3+DZ0NA2inKjp6Qkj5qaFrfLGHM0LoNL9HGpr291u4Rhq69vTeixTvTvlVhJ\n9HHRZ2jf+0oUif6zJVkMNdW+FpgHYIw5Adhhrd3tPJcG3G6MyXEenwxsikmVIiIiIpLw9nvE01r7\nqjFmgzFmPeGWSTcYY5YATdbaVcaYfwOeN8b0AO9Yax+NQ80iIiIikoCGPMfTWvudAYvei3jubuDu\naBclIiIiIslHdy4SERERkbhQ8BQRERGRuBhpOyURERbueJqK9vD9IiqzJvDAwee4XJGIiCQCHfEU\nkRFZuONpDmvfiYdwY9/D2ndy/ScPUdZR53ZpIiIyxil4isiweULBviOdkfIDbVy683kXKhIRkUSi\nqXYR2UtasJuirmaKu5so7mqmyPlvYXczHreLExGRhKXgKZKqQiHyAm0UdYVDZXF3U1/YzO/Z+w5j\nXR4/tenjyOvZTW6gY6/nVk6YFa/KRUQkQSl4iiS5YHcXoepqjums4uD6Ooq7mijqbqaoq4mMUM9e\n6zf7s/kkawL16fnUpRVQl55PfXoBLb5s8ISPd17/yUPkB8LhNICH9FAPZ9e8wery02lOS5x7N4vI\n2KILF5OfgqdIEgiFQgSam+natdP5t4uunTvp3rWT7rpaCIW4PGL9bo+PhrT8cKh0wmVdWgH16fl0\ne9OG3N/DE2b3ndO5unwmxzd/xOdatnDt9sf4a+l0Pso9NEbvVESSVe+Fi716L1x8eMJsqjKLXaxM\noknBUySBhHp66KquprtqJ107d/YLmsG2vafHffn5ZB1h6MjJ5rEPW2ktOJT69Hya/TmEPAd+bWFV\nZjG/O2xe3+MdmaVsyyrjnJrXuXTXC7xecBQvjD+BoMd3wPsQkSQUCuELBUkLdZMe7CEt1ENasIf0\nYM9+L1yM/HkjiU3BU2QMCrS27gmVO3fSVeUcwayphmCw/8o+H+mlZaRPPYq08nLSyyeQPmEC6eXl\n+LJzANi8+SNe3fY3cnMOjk3BHg9/zz+CnRnjuXDXS5zctJGJHTWsLj+dJk29iyScUDBIqKuLYGcn\nwc5OQp2dBDs7wl93dRLs6CTY1Ulwx6fMbttCTmAXaaEe0oPdpDmBMr3vv/2XeQm5/fbERQqeIi4J\nBQJ019bSFXH0stuZIg+0tuy1vjcnh8zDJjuhcs+/tPHj8fjHxke5JqOQuw/5MufWvMbnWrZwjabe\nRQYVrXMZvaFgRKjrJq+nmdD2bexua3dCY4cTGjsHD5F7LXde09U17BrOAmjfe3kI6PKk0e310+X1\n0+bJpMvrp9vjp9vrLPf46XaWHd36CUXd/X/2NfuyeXjC7AMam2RmjKkA3gPejFj8NvArYCXwnLV2\nqbNuGvAb4HNAj/Pvamvt9njW3Gts/LYSSWKB9na6Bzt6WV1FqGfAxT0eD2klpWROnrwnXDpB05eX\n584bGKFubxqPlZ7Wb+r9jYKjeF5T75LKQqG+cHjxrheZ2FHT99Rh7Tu5ccv9/D3/cDq86XuOFgZ7\nSA8NOIIY7NkzTR3swU9wr10F73mDHSOpzefDm5GBNyMTb042/qIi53EGnt7lGel4MjL3Wl7dUMf/\nrP0YX355X6DsDZc9Hl/fBYnDsa74uH4XLjb7spNmin3d3EuexsnowLMz1jwSjaumNllr+6VyY8xT\nwFNA5Mn6i4Aea+1pzjpLgOuB74x0h8YYj7V2VIesFTxF9uPTXyyjbdMHWCD7yKOZ+M2lg64XCgbp\naah3jlzu6jdNHmhq3Gt9b1YW6RMP2fvoZWkp3rShL+4Z85yp988yxnPRrpc4qWkjB3dUO1PviRGg\nRcLTzZ0E2jsI1dVycE8z49q8pAe7SQ92kxHsJj0U8XWwp++5gc8PNcWcFezilMYP9ltPt8dHt8dP\nlzeNVn+287V/z3+9fnZ3dTLjhEMpLp8QERp7g+I+QuQoZkxqNn/Elufryc0sOeBtRIq8cDFZjnQ6\nofPsiEVnr5t7yafA3BlrHnkryru7BJhH+OhmrwKg7wevtfbu3q+NMVcB/wQEgV9aax80xiwAvkH4\nyOgGa+2/GGN+CBwGHGaMOQO4BZgB+IDfWGvvH26BCp4i+/DpL5bRtvH9vsdtG99n87/+CyXzF4LH\n03fVeNeuXXRV7Rp0aspfXEz2MZ8LB8yyiKOXBQV4RnAkIFHVOlPvc2pe4/PO1PvjpadhNfWeMob7\nx1u0hIJBgh0dzr/28H/b2/deNsTzoY7wVDShPWHxawBNw6sjgJdObxpd3jSa/bl0ev10edPp8vo5\nsnXroDdiaPems6ZsJl0Dp6G9aXR7fMO6ILC1YQenzzqVoilHDK/QMWbghYuJYN3cS5YB8/ezyqRB\nlh0MvO4E0MGsmLHmkRF/WKy1u40xAxffA1xtjNkEPA48bK1db4zJA74PfB7IBO42xvwV+AnwBWtt\nmzFmjRM0Q0CatfZ0Y8xM4FBr7SxjTAawwRiz0lrbOZwaFTxF9qFt095HHwKNjez60x/6LfOkp5Ne\nVt7/6OWECaSVluHNyIhXuWNWtzeNv5bNYFtWOXNqXuMSTb2njMH+eNuy9Bsc9PUbyZxU0bc8FAgM\nHgwHC4y9yzojn9+zXqhzWL/7BuVJS8ObmYk3MwtvXn7466wsvJmZtHZ18vKmekLZhXR50+j0hKeU\ne8NklzetL2h2edMI7Od7O3NA2yDYcy6j2gbJCEw1xkTeq3ittfan0P/vGmttPXCCMWYGMAf4izHm\nDuCvhKfrO4FO4CJjzAnAR9ba3jYpLwDHO1+/7vx3OnBqxL49wASgcjhFK3iKjJAnI4PxF8/rC5r+\nwkI83gNvTZQq3ss/nJ0Z47lo14uc1LSRiR3VrNLUe1Ib7I+3noYGtv37LaQVjw8HxxFeyDKQJz29\nLyD6C8b1BcW+ANn79WDLe5dlhJfvb8p58+aPeHL738gtHH1niAcOPidpz2VMRc6RyX0enRxkqh1g\nB6Ofav9w4Dmejn7ndBhj0gmf47kOWGeM+R/CgXINMPCXV5D+wTWdPZeOdTv/7QRut9b+7ECKVvAU\n2YesI4+ifWP/X5z+wsK9jtbI8NVmjHOm3l/n8y2bnan36djcwWaiJOGF9nFOYyBAsKszfCFLcXFE\nIHTCYMY+guLAAJmRiceXmEfNk/FcRhncjDWPnONMqff+1bJjxppHJsZwlwPP5LgdWAf0TtcdAmwG\nNhE+apoDBAgH0YuAI4wxudbaVmAW8GP6B+fXgP80xvwHkAH83Fr7z8MtTsFTZB/GnX5Gv+DpLyxk\n8rJbXawoOYSn3sNXvYen3l/kzYIjeX78iW6XJlHSXV9P7cMrBn3OP66Qg/5Jf7wl4rmMMipzCQe7\n3q+jYeCRzUMIn8M5Dsg0xpwHXE74QqE/OBcSdQBdwNecczh/ADzjbOKXzrKlwJPGmCDwsnM+6Nm9\n+7PWvupMs79KOOT+diRFK3iKDCLY3R3+xen14svNxZfmp/z6Yf9BJ8PQO/V+YdWLTGvaxMEdNdyX\nNdXtsmQUgl1dNKx9kvrHHyPU1UXGpAp66ur6+tLqjzdJVc6UetSOclprK4GTByzbTvhCocFcuo/t\n/AX4y4BlKwn3Ao1c9qMBj78HfG9ERTsUPEUG0fT8s3TX1jDu7HMovewKSkryqKnZu6m7jE5txjiW\nT/wy59S8zrEtm7mh6zVCmyZAgl6Rm6pCoRCtb71JzYoH6KmtxZeXz/hFV5I/fQad27fx2W9uw+v1\n6I83EVHwFBko0NpK3WNr8GZnU/yVC90uJ+l1e9N4vOy08FXv1a8SXPkw1U2NjJ9/WXL0NE1yndu3\nU/3AfbRv2gg+H4Xnfomir8zFl5UFQOakCiYvu1V/vIkIoOApspe6x9YQbGtj/PyF+HJ1n/F4+d/8\nKWzu7OHGtC00Pvcs7Zs3M+Efrye9tNTt0mQQgdZWalc/QtMLz0MoRM6xX6Bk4eWkl5W7XZqIjGEK\nniIRuqqqaHz+WdLGlzDuzIHdLyTWavy5eK++hty/vUrzupfZdsvNlC25lrxpJ7ldmjhCPT00vvg8\ndatXEWzbTXr5BEouu5yczx3rdmkikgAUPEUi1D6yAgIBxl86X9O8LvGkpVF+9XVkmyOpuududv7+\nt7TNPouSBQvxpqW7XV5K2/3B+9Tcfy9dn32GNyuLkoWXM272WaO65aKIpBb9tBBxtH/8Ea0b3iRz\n8mRydYTNdfnTTyOj4jB2/v63ND3/LB2bP2bC/71BU+8u6KqupubBv7D7nbfB46Hg9DMovvgS/Hn5\nbpcmIglGwVOE8FW5NQ/eD0DJ/MtT4j7qiSDjoIM49KYfUP2Xe2le95Iz9X4NedNOHvrFMmrBjnbq\nHnuUxmfWEurpIctMpeSyRWQeqob/IhF8wJQob3Mz4abu+2SMuQG4kvCdhLKA71prnx3Oxo0xXwAu\nttb+cJR1jpiCpwjQ+uYbdGzZTO6J08g6Qq18xhJvRgblV19L9tTeqfff0TZ7EyULLtPUe4yEgkGa\nX32F2kdWEGhqwl9UTMmCheSeeJL+KBPZ25STL7n5w+yC6MzGtDVV8/ojP5oK2H2tY4ypAP4BmGat\nDRhjDPBHYFjB01r7LvBuFModMQVPSXl9zeJ9PsZfMt/tcmQf8r84ncyKCj77/e9oev45Onqvei8r\nc7u0pNK++WNq7r+Pjk+24ElPp/jCiymccx7ejAy3SxMZs7ILSsktPHjoFaOnAMgkfMvKNmutBc5w\n7jB0C+E7FDUCC4AHCd+V6GVjTBbwAXAd4bsXzTfGfEy4YfxpzmvOJ3x7zxWEj6a+BMy01s42xvwa\nOJHwUd7/ttbePdLCB94cXiTl9DWLn32WQswYlz7hIA797vfJn3k6ndu2su2Wm2l5/TW3y0oKPY0N\n7Lz9j2z/6Y/p+GQLeSefSsWPf0rxBRcqdIqMMc4Ry9eBT4wxdxpj5htj/IQD6RXW2tlAE3Au8DBw\ngfPSc4Cn6D+Nfxhwt7V2OlAIHEv4Npv3W2vPIBxuQ8aYQuDL1trTgBnAAV2Bm3JHPD/9xTLaNn2A\nBbKPPJqJ31zqdkniov7N4qN1+1yJJW9GBuVLnKn3P9/Nzj/+N232Q0oWaur9QAS7u2hY+1T4Nped\nnWQcOonSy68g6wjjdmkish/W2iXGmCMJh8tvAV8D/g34oxNCJwPPAaud578FXMiAW2QCzdba/3W+\n/pRweD0yYr01wMnW2gZjjDXGrCJ8NHT5gdSdUkc8P/3FMto2vg+hEIRCtG18ny1Lv0HH1kq3SxOX\n9DaLLzr/AjWLTzD5p05n0vd/SPrBE2l64Tm2//uP6ara5XZZCSMUCtHy1ga2fv8m6lY+jDc9g7Il\n13Do925W6BRJAMaYDGvtJmvtbcApwCHAHcANzpHK1QDW2iZgh3Me6BcJh9HIk7V7Bmza4/wLDdyn\ntfbLwI+A44BHD6TulAqebZs+2GtZT0MDn/3mNheqEbepWXziSy+fwKE3/YCC02fRuX0bW//th5p6\nH4bOHZ+y45fL2Pm7/6K7oZ7COedR8ZOfUTBzFh5vSv1aEElIxph/AO4wxvQGyHGEw2IesN0YMw44\nk/A0OYTP4fwe8Iq1NjiMXWwGpjlff8nZ5yRjzD9ba9+21i4Fig+k9pSbahfppWbxycGbnk7Z4mvI\nmnokVcudqfcPN1Fy2eWaeh8gfJvLlTS98Fz4NpefPzZ8m8vyCW6XJpLQ2pqq472tO4CpwN+MMa2E\nz7f8J+BUYD3h4Phz4GZjzBrCRz//i/BUO4SPZoYivo4UAm4DHjTGXEr4XNIA8BnwRWPMQsIXHd1+\nIO8vpYJn9pFHh6faI3hzczno6ze6VJG4pf0jNYtPNvmnfJHMSc5V7y8+T8eWj5nwjzeQXq57h4cC\nAZpefJ7a1SsJ7t5NWlk5JQsvJ/fYL7hdmkgy2Oy0P4rqNvf3pHPUcrCLVJ4Abo54HHnVed9Vgtba\nF4EXna9LI5bPBzDGHA183Vr7ijHmcmC8tbYbuHyE72MvKRU8J35zKVuWfoOehoa+Zb7MLP21n2JC\noRA1K8LnTJcsULP4ZJJePoFDv/t9ah64j6YXX2DrLT+kbPHV5J9y6l7rBgIBKiu3RL2GhoZc6utb\no7rNiorJ+Hy+A3pt28YPqP7LvXR9tiN8m8sFlzHuzLN1m0uR6Amwn56bCaoF+IMxJkT4/V0TrQ2n\n3E+eg75+I5/95ja8Xg8Z5ihaXl1P3apHKFk46hAvCSLcLH5LuFn84WoWn2y86emUXXU1WeZIqpbf\nxa4//Z72DzdRctkivOl7pt4rK7dw47I1RKvpc6y0NVVz29K5TJkysu/Vrppqah98gNa3Nzi3uZxF\n8UWX4s/XbS5FZP+stduBmbHYdsoFz8xJFUxedislJXlU7aij4+OPaHhmLXknn0LmYZPdLk9iTM3i\nU0f+KaeSOamCnX/4LU0vvUD7ls0c9H+v7zfD4ULT55gLdnRQ//hjNKx9MnybyyNM+DaXkyrcLk1E\nJLWuah/Im55O2ZJrIBRi1913EuoZ2FFAkk3jc8+oWXwKSS8v55Dvfp+CWbPp+nQ7W2/5Ec2vvep2\nWTERvs3lej656dvUP/4Yvvx8Jnz1a0z81ncUOkVkzEi5I54DZR95FPkzT6f55Zeof+oJis+/YOgX\nSUIKtLZS/9dH1Sw+xXjT0im7aglZU6dSdfdd7PrTH2j/cBOhk09xu7Soad+ymZr776VjyxY8aWkU\nXXAhRed9WXccEpExJ+WDJ0DJvIXs/vu71D+6mrwTp+lioyTV2yy+ZMFlahafgvJPdqbef/9bml56\nETZtZHxgMh1uFzYKPY2N1D6yguZX1gOQd9LJjJ+3kLTiA2qvJyIScyk91d7Ll5ND6aKrCPX0UHX3\nnYSCw+mtKokksll8weyz3C5HXJJe5ky9n3EmVFfztcY3OLol+le2x1qwu4v6xx/jk5u+TfMr68k4\n5FAmfus7TPjH6xU6RWRM0xFPR96J02g5/kRa395A00svMO6MM90uSaJIzeKllzctnbIrF9M8roDQ\n6jXMrVrHoe27eGb8yfR4x/aPxFAoxO533qbmwb/QXVODLzeP4gWXUTDzdN1xSEQSwtj+KRtnpVdc\nSdumD6h96EFyvnA8aYWFbpckUbCnWfwUNYuXPt6jjuZ3L+xiUfuHHNf8MQd11LKqfBb16QVulzao\nzh07qHngPto+eB98Psadcy7FF8zFl53jdmkiIsOmP5Ej+McVMn7+QoIdHVTfu5xQaOBdpCTR9G8W\nf5maxUs/9b5s/nzwl9hQMJXSrkau3v5XjhljU+9ZwW6Ca59i64++T9sH75P9uc9T8cNbKF14uUKn\niCQcHfEcoGDmLFpe+xu733mb1jffIO+kk90uSUZBzeJlKAGvj6dLTmFbVhlfrnqVC5yp96ddnnr3\nhIIc1/wRMxrfIrShm7SyMkoWXk7O57+gP6BEJGHpiOcAHo+HssVX40lLo/q+ewi0RvfWdxI/ahYv\nI/FhbgV3HnI+uzKK+ELzxyz+9HGKuppcqeXQtl1cs/0xzq15DR9BPGeeRcWPfkLusccpdIpIQhsy\neBpjbjXGvGKMWW+MmTbgudnGmFeNMeuMMbcbY5LiJ2J6WTnFcy8i0NJMzYP3u12OHCA1i5eRakzP\n33vqvXlz3PZf0N3CxTtfYNFnaynpauTd/MO5ddwX8Z5yqu6tLiJJYb/B0xgzCzjcWjsduA749YBV\n/gjMs9bOAPKA82JSpQsK55xHxqGTaH5lHbs/eN/tcmSE1CxeDlTv1PvK8lkEPR4uqF7Pl6pewR+M\n3Z3N0oLdzKx7m/+zbTVTd29je2YJd008nydKp7PbqybwIpI8hjrieSawEsBauwkoNMZEdt4+0Vq7\nw/m6BiiKfonu8Ph84dtper1UL7+LYGen2yXJCPQ2iy/+ylw1i5cD8mHuJO465CvhqfeWj1ny6eMU\ndzVGdyehEMe0bOGrW1dxWsN7tHkzWV02k3sPPo+qTPXjFJHkM1TwLAdqIx7XAH239bHWNgMYYyYA\nc4DHo12gmzInVVA45zy6a2uoW73S7XJkmNQsXqKlMS2PP0/8Em8WHElJVyNLtj/O56I09V7eUcuV\nO57kgqp1ZAa7WF94LH+adCEb8w4DnccpIklqpCcNeYB+PYaMMaXAGuBr1tqGoTZQWJiN3+8b4W5j\no6Qkb8h1iq69knfeeYuGZ9ZyyJzZ5B1xeBwqc9dwxmUs23T77yEQYPI1VzH+oOgdhE/kcWloSJyj\nvkVFuXFGWgvxAAAYt0lEQVQZ6+GOScDj45mSk9meVcaXql/hK9Xrw1e9l5xMt3fkNyPI6WlnVt1b\nHNsSDrAbcyfxQvGJNKXtu554jUmsJfJ70GdocBoXGamhgudnhI969joI2Nn7wBiTT/go53ettc8M\nZ4cNDW0jrTEmSkryqKlpGda6469cwqf/+R9s+tVvmPS9m5P6JP+RjMtY1P7RR9S9+jcyJ08haD4f\ntfeS6ONSX5843Rnq61vjMtYjHZMPcyexK6OIi3a9xLEtm5nQGW44X5c+bliv94UCTGvcyPT6v5MR\n6qE6vZBnxp/EtuzyIV8brzGJJX2G4iee3y8aFxmpoaba1wLzAIwxJwA7rLW7I57/BXCrtXZtjOob\nE7KPPIr8GafT9el26p96wu1yZB/ULF5irSktj3smnudMvTcNb+o9FOLw3du5btsaZte9RcDj48mS\nU7nzkPOHFTpFRJLJfg/dWWtfNcZsMMasBwLADcaYJUAT8BRwFXC4MeYfnJfcZ639U0wrdknJ/IXs\nfu9d6h9dTd6J00gvnzD0iySuWt94Xc3iJeZ6p963ZZXx5SGm3ou7Gjmr5g0mt+8kiIc3Co5iXdGx\ndPp0pbqIpKYh54yttd8ZsOi9iK8zo1vO2OXLyaF00ZXs/O/fUnX3nUxc+m08XvXfHyuC3d3UPBJu\nFt920sls3vxRVLff0JAb9SmliorJ+Hxj43xnGTmbO4mqiKn3I1u3khYKt1zalllGdUYRJzZtwkuI\nLVkTeLbkpGFPy7spEAhQWRn924bqMyQioFtmjkjeiSfRcvyJtL69gaaXX2TcrNlulySOxueeoae2\nFs9JJ/Mv//My2QWlbpe0X21N1dy2dC5TpujIbCLrnXr/6tZVFPTsOQtpUkcVkzqqaPJn83TJKXyc\nPTFhrlSvrNzCjcvW6DMkIjGh4DlCpVdcSdumD6h96EFyjj2OtMJCt0tKeZHN4jltBtlb3iW38GC3\ny5IUEfD4yO/ZPehz3lCIj3MOiXNFo5ddUKrPkIjEhOaKR8g/rpDx8xcSbG+n+t7lhEKhoV8kMVX3\n2Oq+ZvGerCy3yxHpEyIxjnKKiMSLgucBKJhxOllmKrvfeZvWDW+4XU5K66raRePzz5FWombx4p7K\nrL0vNmz2ZfPwBJ2OIyISScHzAHi8XsqWXIPH76f63nsItCZOH7NkU/vIQxAIMP7S+XjTRt7MWyQa\nHjj4HJp92X2Pm33Z/O6webrtpYjIAAqeByi9rJziuRcRaGmmZsUDbpeTkto/+ojWDW+SOXkKuSee\n5HY5kuIenjCbZl+2jnSKiOyHLi4ahcI559Hyxus0r3+ZvFNOJefoY9wuKWWoWbyMNVWZxfzusHlu\nlyEiMqbpiOcoePx+yq6+Frxeqv98F8HOTrdLShlqFi8iIpJ4FDxHKXNSBYXnnEt3TQ11a1a6XU5K\niGwWP/7SBW6XIyIiIsOk4BkFxXMvIq2klIa1T9FR+Ynb5SS93mbx4848m/TSsd3kWkRERPZQ8IwC\nb0YGZYuvhlCIqrvvINTT43ZJSSuyWXzx+Re4XY6IiIiMgIJnlGQfdTT5M06nc/t2GtY+6XY5SSuy\nWbwvN9ftckRERGQEFDyjqGT+QnwFBdStWUXXrl1ul5N01CxeREQksSl4RpEvJ4fSRVcS6umhavmd\nhIJBt0tKKmoWLyIiktgUPKMs94Rp5Bx/Au32Q5pefsntcpJG+0dWzeJFREQSnIJnlHk8HsquuApv\nVha1Dz1Ad0OD2yUlvFAoRM2D9wNqFi8iIpLIFDxjwD+ukPHzFhJsb6f63uWEQiG3S0porW+8Tscn\nW8iddpKaxYuIiCQwBc8YKZh5OllmKrvfeZvWDW+6XU7C6tcs/pL5bpcjIiIio6DgGSMer5eyxdfg\n8fupvu/PBFpb3S4pIalZvIiISPJQ8Iyh9PJyiudeRKC5mZqHHnC7nIQTaG2l/rE1eLNz1CxeREQk\nCSh4xljhnPPIOORQmte9TNvGD9wuJ6HUPbaaYHu7msWLiIgkCQXPGPP4/ZQtuRY8HqqW30mws9Pt\nkhJC/2bxZ7pdjoiIiESBgmccZFZUUDjnXLpraqhbs9LtchJC7cMr1CxeREQkySh4xknx3ItJKyml\nYe1TdFRWul3OmNb+kaX1rQ1kTjlczeJFRESSiIJnnHgzMihbfDWEQlTdfTuhnh63SxqT1CxeREQk\neSl4xlH2UUeTP2Mmndu307D2SbfLGZP6NYufcrjb5YiIiEgUKXjGWcn8y/Dl51O3ZhVdu3a5Xc6Y\nEuzuUrN4ERGRJKbgGWe+nBxKF11FqKeHquV3EgoG3S5pzGh87ll6amspVLN4ERGRpKTg6YLcE6eR\nc/wJtNsPaXr5JbfLGRMim8UXqVm8iIhIUlLwdIHH46F00VV4s7KofegBehob3C7JdWoWLyIikvwU\nPF2SVljI+HkLCba3U3XvnwmFQm6X5Bo1ixcREUkNCp4uKph5OllmKrvffovWt950uxzX7GkWv0DN\n4kVERJKYgqeLPF4vZYuvweP3U33fPQR273a7pLjr3yx+mtvliIiISAwpeLosvbyc4rkXEWhqombF\nA26XE1dqFi8iIpJaFDzHgMI555FxyCE0r3uJto0fuF1O3LS88ZqaxYuIiKQQBc8xwOP3U7bkWvB4\nqFp+F8HOTrdLirlgd1f43E6fj/GXqlm8iIhIKlDwHCMyKw6jcM65dNdUU7dmldvlxFzjc8/SU1cX\nbhZfombxIiIiqUDBcwwpnnsxaSUlNKx9ko6tlW6XEzNqFi8iIpKa/G4XMJRAIEBl5Zaob7ehIZf6\n+taobrOiYjI+n++AX+/NyKBs8TV8+oufU3XXHRx60w/w+Mf8/6IRq3s03Cy+ZMHlahYvIiKSQsZ8\nqqms3MKNy9aQXTC2p2Pbmqq5belcpkw5YlTbyT7qaPJPm0nz+pdpePopir50fpQqHBu6qnbR+MJz\npJWUMu7Ms9wuR0REROJozAdPgOyCUnILD3a7jLgpWXAZu997l7o1q8g94UTSy8rdLilq9jSLn5+U\nR3NFRERk33SO5xjky8mhdNFVhLq7qbr7TkLBoNslRYWaxYuIiKQ2Bc8xKvfEaeQcdzzt9kOa1r3k\ndjmjFgoG1SxeREQkxSl4jlEej4fSKxbjzcqidsUD9DQ2uF3SqLS8+brTLP5kNYsXERFJUQqeY1ha\nYSHj5y0g2N5O9b33uF3OAevfLH6e2+WIiIiISxQ8x7iCmbPIMlNpfXsDLRvecLucA6Jm8SIiIgLD\nCJ7GmFuNMa8YY9YbY6YNeC7TGLPcGJOYiSgBeLxeyhZfg8fvp/q+ewjs3u12SSMSaGlRs3gREREB\nhgiexphZwOHW2unAdcCvB6zyc+D1GNUmjvTycornXkSgqYmaFQ+4Xc6I1D22hmB7O8UXzFWzeBER\nkRQ31BHPM4GVANbaTUChMSYyPXwHeDRGtUmEwjnnkXHIITSve4m2jR+4Xc6w9GsWP1vN4kVERFLd\nUMGzHKiNeFwDTOh9YK3dDagvThx4/H7KllwLHg9Vy+8i2NnpdklDUrN4ERERiTTSi4s8QCgWhcjQ\nMisOo/Ccc+muqabu0dVul7NfbfZDNYsXERGRfoY6DPUZ4aOevQ4Cdg5YZ0RBtLAwG7/fN+z1GxoS\n57zAoqJcSkryYruPf1jMO39/m4a1T3LonNnkTpkc9X2M9j2EgkH+vnIFAOYfryOvND8aZQ2Lvl8G\np3HZm8ZkcBqXwWlcBqdxkZEaKniuBX4E/NEYcwKww5lejzSiqfaGhraRrE59feuI1ndTfX0rNTUt\nMd9P8aLF7PjlMjbd+l8cetMPojqNXVKSN+r30Pz632j96GNyp51MR9EEOuIwJr30/bLvfSWKeI2L\nxmTf+0oUGpfBaVwGF89xkX3b71S7tfZVYIMxZj3wK+AGY8wSY8xFAMaYZ4AngWOMMe8ZY66JecVC\nztHHkH/aTDq3b6Ph6afcLqcfNYsXERGRfRnyUJm19jsDFr0X8dzZUa9IhqVk/kJ2v/cudWtWkXvC\niaSXlQ/9ojhofPaZcLP4c85Vs3gRERHpR3cuSlC+3FxKF11JqLubquV3EQq5f81XoKWF+r8+Gm4W\n/5W5bpcjIiIiY4yCZwLLPfEkco47nvYPN9H88ktul9O/WXxOjtvliIiIyBij4JnAPB4PpVcsxpuV\nRc2K++lpbHCtFjWLFxERkaEoeCa4tMJCxs9bQLC9ner77nGtjtqH1CxeRERE9k/BMwkUzJxFlplK\n61sbaNnwZtz332Y/pPVtNYsXERGR/VPwTAIer5eyxVfj8fupvu/PBNoGtlqNnVAwSM2D9wNQsuAy\nPB7dQVVEREQGp+CZJNLLJ1B0wYUEmpqoWfFA3Pbb8ubrdFZ+Qu60k8macnjc9isiIiKJR8EziRSd\n+yXSJx5C88sv0bZpY8z319ss3uP3q1m8iIiIDEnBM4l4/H7Kr74WPB6qlt9FsKsrpvvrbRY/7syz\n1SxeREREhqTgmWQyKw6j8Jxz6a6uom7Nqpjtp1+z+PMviNl+REREJHkoeCah4gsvJm18CQ1rn6Rj\na2VM9qFm8SIiIjJSCp5JyJuRQeniqyEYpOruOwkFAlHdftcuNYsXERGRkVPwTFI5Rx9D/vQZdG7b\nSsPap6K67dqHnWbx89QsXkRERIZPwTOJlSy4DF9ePnVrVtJVtSsq2+zXLP4ENYsXERGR4VPwTGK+\n3FxKF11JqLubquV3EQqFRrU9NYsXERGR0VDwTHK5004i57jjaf9wE80vvzSqbbW8EW4Wn3eSmsWL\niIjIyCl4JjmPx0PpoqvwZmZSs+J+ehobD2g7we4uah9xmsVfMj/KVYqIiEgqUPBMAWlFRYyft4Bg\nezvV9/35gLYR2Sw+raQkyhWKiIhIKlDwTBEFp59B1hGG1rc20LLhzRG9tq9ZfI6axYuIiMiBU/BM\nER6vl7Il1+Dx+6m+7x4CbbuH/dq6R1c7zeIvVLN4EREROWAKnikkvXwCRRdcSKCpkdqHHhzWa7p2\n7aLxxefDzeLPODPGFYqIiEgyU/BMMUXnfon0iYfQ9NKLtG3aOOT6ahYvIiIi0aLgmWI8fj/lS64B\nj4eq5XcR7Ora57p9zeIPP0LN4kVERGTUFDxTUOZhkyk8ew7d1VXUrVk16DpqFi8iIiLRpuCZooov\nuoS08SU0rH2Sjm1b93q+X7P4yVNcqFBERESSjYJnivJmZFC6+GoIBqm66w5CgUDfc2oWLyIiIrGg\n4JnCco4+hvzpM+jctpWGp5/qW65m8SIiIhILCp4prmTBZfjy8qlbvZKuqiq6m5vVLF5ERERiQsEz\nxflycylddCWh7m4qb/r/ef2qa9QsXkRERGJCwVNofOmFvZY1PPkEHVsr416LiIiIJC8FT6F9kEby\nPY0NfPab21yoRkRERJKVgqeIiIiIxIWCp5B95NF7LfMXFnLQ1290oRoRERFJVgqewsRvLsVfWNj3\n2F9YyORlt5I5qcK9okRERCTpKHgKAAd9/Ub8hYWkFxfpSKeIiIjEhN/tAmRsyJxUweRlt1JSkkdN\nTYvb5YiIiEgS0hFPEREREYkLBU8RERERiQsFTxERERGJCwVPEREREYkLBU8RERERiQsFTxERERGJ\nCwVPEREREYkLBU8RERERiQsFTxERERGJC925KEEFAgEqK7dEfbsNDbnU17dGdZsVFZPx+XxR3aaI\niIgkHgXPBFVZuYUbl60hu6DU7VL2q62pmtuWzmXKlCPcLkVERERcNmTwNMbcCpwChIAbrbVvRjx3\nNvATIAA8bq39cawKlb1lF5SSW3iw22WIiIiIDMt+z/E0xswCDrfWTgeuA349YJXbgEuA04A5xpij\nYlKliIiIiCS8oS4uOhNYCWCt3QQUGmNyAYwxk4F6a+0Oa20IeBw4K5bFioiIiEjiGip4lgO1EY9r\nnGW9z9VEPFcNTIheaSIiIiKSTEZ6cZHnAJ8blbam6lhtOmrcqFHjMnb2OVIal8HFu0aNydjZ50hp\nXAancRlcItQogDHmZmPMVyMebzbG5DhfTzLGvDJg3evdqFNERERExr6hptrXAvMAjDEnADustbsB\nrLVbgXwngPqB8531RURERET2MuT0uDHmp8DphFsm3QCcADRZa1cZY2YC/+Gs+pC19pcxq1RERERE\nREREREREREREREREREREREREREREoipmvTfHCmNMBfAe0HuP+Qzn8deALcA2whdOAYSstWfGu0Y3\nOOOywlp7UsSyHwKLgB2EvzeygJ9aa1e5UaMbBny/eIAe4N+BgwjfNjYTOAbY4LxksbV2e/wrPXCj\n/UwYY64GbgE+JjxGncBV1tpqY8xdQIm19vyI9b8MPAZUEO6kMei+rbXBKL7NERvtZ8IYU0l47HqA\nXOB2a+0fnO1uAU6x1r4Rsf7rwPvW2muccTsBqHP2kw4stdauj/obHYHRfh6ScUyGYow5AvgVMB7w\nAa8A/wpMB/4NCAJ5wJ+ttb8yxpwBrAD+l/DnoxW4yVr7jjHmXwl3jBkHHAy87+xmjrW2O25vapRG\nOSYA2cCT1tqbne11A+uc57KAO621f4jPu5HRGmkD+US1yVo7u/eBMeZO4AogBJxnrW1zrbKxJQT8\nylr7OwBjTCHwjjHmCWttp7ulxVXf94tza9hHgYXW2tnGmEmEOzjM3u8Wxr7RfCZCwF+std9yXvt9\n4FrgZ87zFcaYcdbaRufxPGDzEPteBNwzyvcUCyP5TPSNnTEmG9hijPmj89wWYAHwhrOdSUCh85re\n137bWvu48/xk4Algauze2rCN5vOQrGMyKGOMD3gI+Lq19mVn2a+BHwDzgVnW2l3GmEzgWWPM/c5L\nX7DWznfWPx54yBgzw1r7n8B/GmNmOducH+/3NFpRGhMP8LQzJuuAxojvyXTgLeczuS2+704OxFB9\nPJPVa8DhbhcxRvUdBbfWNgA7SeFboVprtwA/Ab7uLErWWYKRfiYix6Ec+NT5OgQ8A1wMYIxJA44A\n9ndU+HVnnbFqJJ+J3nWLgRprbYjwmPwNOCtivXnsp++x832X7/zCHTMO8POQ1GMywDnAxt6A5VhK\n+KheEeGjelhrO6y1p1lrdw3cgLX2beAO4OqIxWP5PQ8lGmMSInzUfa+fUdbaLsJH5Q+LQe0SAykX\nPJ1fhBcCbzmLEvkDHVPGmKlAGXtCRaraABztdhGxcgCfCQ+w0BjzvDHmPeA44OGI5x8hfCQDwsHi\nhX1t09n3XPZM1Y5pw/hMPGGMeZHw+7klYnk38LYx5hTn8fnA4/Qfl76vjTGnA585v3DHmpF+HlJh\nTHpNBd6NXGCt7XTC0feBN4wxq4wx1xtjxu1nO2+SPD9zRj0mxphc4Fz2/IyKfK6I8M+g96JeucRE\nqky1TzXGPO98fSzwM2vtamPMbYR/KPaez1ZjrV3gToljgge40RgzD8gnfP7d5dbaHnfLcl0ee855\nTBaj+UyEgPsjptqvBP4ALHaeryR8ZKoQuBT4DTBjiH2vieJ7i6aRfiZ6p5XzgGeMMe8CHc5zDwEL\njDE7gAbC5/JF7uenzjl9453nFkX/7UTFSD8PqTAmvUKEz2Hci7X298aYlYQD1EXA95w7Ag4mn+T5\nmXOgYzLL+TnhIzwj8m1r7d+d5woifoYEgX+11tbH7B1IVKVK8Pww4nyQFcBHznKd49lf3/lsxphy\n4Dn0VyTANAb5SzvBDfsz4VwIcqfz8JvOfyOPSj0C/HjA9lcRnjr9nLX2XWPMcPY9Fu3zM+FceDQL\n+Lu19sbIF1lrW4wxLwBfBHp/QT4D/BTYSv8jxL37+ba19nFjzLHA/wA2Fm8oCvb5eUjhMem1iT2n\nIQB95yAaYLO1tgpYDiw3xtwBzCH83gdKpp85BzomL0ac4/kq/X8XNSXBefYpK+Wm2gmfW/Iz50R3\n6c/j/MM5z2Y5cLOrFbnMGDMF+AZwq9u1xNB+PxPW2kpr7Wzn32C/DE8h/Msl0kOEQ+oLw9x31ghr\njpd9fiastT90xuTGAev3XgxxEhHj4lyF/Dbhq8EfZe/TD3r383fCoeNrMXg/ozLU5yEVx2SAp4FJ\nxpivABhjvMDPge8SvgAmL2L5QfS/6A7nuWnAJcDt8So6xkY9JsD/B/x2jJ/fK8OUKsGz75wga20l\n4b+sb4pcnqKOMcZs7P1HuLVF5Jj8EphrjDnKnfJcM9U5f/EV4D7gemtt5Dl9yfB9M9rPRO85ns8D\n3wP+OXLbzjabCQfQ4ez7eyOoPZZG+5l4whmTV4HnrbV/c5b3bmMFsM1a2+Isi9x25NffA5YaY8aP\n8v1Ew2g/D8k4JoNyzj89F/iqMeYN4GWgwVq7iPCR3aeNMc85y591WkN5cKaVjTGvAcsIdw2InIkb\nOC4J4wDHBPr/nHiVcBeE6wY+JyIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIi\nIiIiIiIiIiIiB+j/Ae9D2qlU0OOjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd7f1bfbb50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions\n",
    "\n",
    "CostCla is a easy to use Python library for example-dependent cost-sensitive \n",
    "classification problems. It includes many example-dependent cost-sensitive algorithms. Since \n",
    "it is part of the scientific Python ecosystem, it can be easily integrated with other \n",
    "machine learning libraries. Future work includes adding more cost-sensitive databases and \n",
    "algorithms, and support for Python $\\ge$ 3.4. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix\n",
    "\n",
    "## Tuning the parameters of random forest\n",
    "\n",
    "So far the parameters of the different hyperparameter have not being tunned, which may lead to a bias comparison. Here I quickly tuned the hyperparameter of the random forest classifier using randomized search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 20 candidates, totalling 60 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   1 jobs       | elapsed:    3.1s\n",
      "[Parallel(n_jobs=4)]: Done  50 jobs       | elapsed: 13.6min\n",
      "[Parallel(n_jobs=4)]: Done  60 out of  60 | elapsed: 30.9min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomizedSearchCV took 2088.05 seconds parameter settings.\n",
      "Model with rank: 1\n",
      "Mean validation score: 0.935 (std: 0.000)\n",
      "Parameters: {'bootstrap': True, 'min_samples_leaf': 50, 'n_estimators': 1000, 'min_samples_split': 23, 'criterion': 'gini', 'max_features': 4, 'max_depth': None}\n",
      "\n",
      "Model with rank: 2\n",
      "Mean validation score: 0.935 (std: 0.000)\n",
      "Parameters: {'bootstrap': True, 'min_samples_leaf': 86, 'n_estimators': 1000, 'min_samples_split': 95, 'criterion': 'gini', 'max_features': 6, 'max_depth': None}\n",
      "\n",
      "Model with rank: 3\n",
      "Mean validation score: 0.935 (std: 0.000)\n",
      "Parameters: {'bootstrap': True, 'min_samples_leaf': 40, 'n_estimators': 20, 'min_samples_split': 62, 'criterion': 'entropy', 'max_features': 9, 'max_depth': None}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from time import time\n",
    "from scipy.stats import randint as sp_randint\n",
    "from operator import itemgetter\n",
    "from sklearn.grid_search import RandomizedSearchCV\n",
    "\n",
    "# Utility function to report best scores\n",
    "def report(grid_scores, n_top=3):\n",
    "    top_scores = sorted(grid_scores, key=itemgetter(1), reverse=True)[:n_top]\n",
    "    for i, score in enumerate(top_scores):\n",
    "        print(\"Model with rank: {0}\".format(i + 1))\n",
    "        print(\"Mean validation score: {0:.3f} (std: {1:.3f})\".format(\n",
    "              score.mean_validation_score,\n",
    "              np.std(score.cv_validation_scores)))\n",
    "        print(\"Parameters: {0}\".format(score.parameters))\n",
    "        print(\"\")\n",
    "\n",
    "clf = RandomForestClassifier()\n",
    "\n",
    "# specify parameters and distributions to sample from\n",
    "# from http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html\n",
    "param_dist = {\"n_estimators\": [10, 20, 50, 100, 1000],\n",
    "              \"max_depth\": [3, None],\n",
    "              \"max_features\": sp_randint(1, 10),\n",
    "              \"min_samples_split\": sp_randint(1, 100),\n",
    "              \"min_samples_leaf\": sp_randint(1, 100),\n",
    "              \"bootstrap\": [True, False],\n",
    "              \"criterion\": [\"gini\", \"entropy\"]}\n",
    "\n",
    "# run randomized search\n",
    "classifiers[\"RS-RF\"] = {\"f\": RandomizedSearchCV(clf, param_distributions=param_dist,\n",
    "                                                n_iter=20, n_jobs=4, verbose=1)}\n",
    "# Fit\n",
    "start = time()\n",
    "classifiers[\"RS-RF\"][\"f\"].fit(X_train, y_train)\n",
    "print(\"RandomizedSearchCV took %.2f seconds\"\n",
    "      \" parameter settings.\" % ((time() - start),))\n",
    "\n",
    "report(classifiers[\"RS-RF\"][\"f\"].grid_scores_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                pre        f1       acc       rec       sav\n",
      "RF         0.492091  0.232172  0.934394  0.151926  0.122358\n",
      "LR         0.529412  0.037677  0.934854  0.019533  0.010948\n",
      "DT         0.251239  0.262626  0.899146  0.275095  0.191770\n",
      "RF-BMR     0.179759  0.282039  0.782493  0.654368  0.432752\n",
      "LR-BMR     0.115987  0.188429  0.717737  0.501899  0.228092\n",
      "DT-BMR     0.251639  0.260847  0.899819  0.270754  0.192115\n",
      "CSDT       0.160237  0.267374  0.711325  0.806837  0.465266\n",
      "CSRP       0.164478  0.272971  0.721102  0.801953  0.469644\n",
      "RS-RF      0.611842  0.242714  0.938326  0.151384  0.129198\n",
      "RS-RF-Sav  0.595506  0.267564  0.938326  0.172545  0.146038\n"
     ]
    }
   ],
   "source": [
    "# Predict\n",
    "classifiers[\"RS-RF\"][\"c\"] = classifiers[\"RS-RF\"][\"f\"].predict(X_test)\n",
    "# Evaluate\n",
    "results.loc[\"RS-RF\"] = 0\n",
    "results.loc[\"RS-RF\", measures.keys()] = \\\n",
    "[measures[measure](y_test, classifiers[\"RS-RF\"][\"c\"]) for measure in measures.keys()]\n",
    "results[\"sav\"].loc[\"RS-RF\"] = savings_score(y_test, classifiers[\"RS-RF\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAp4AAAE7CAYAAAB0T4FTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXLNnXSQhbggSQrwourUutgCKIqGitVVxq\nF+21vUv1lvb28vvd1rZe7XprW5fb21/be9Xa9latohYVFBVFQESlahXEr4IRErbATDYm68z8/phJ\nmISQhUzmZCbv5+ORB3OWnPP5hpPJZ77f7/kcEBERERERERERERERERERERERERERERERERERERER\nERERERERSWmu/nYwxtwBnAlEgCXW2tfjtlUBO4BQbNXnrLW7hiFOEREREUlx3r42GmPmAsdaa2cZ\nY44H7gVmxe0SAS601gaHMUYRERERSQPufrbPBx4DsNZuBXzGmPwe+/TbayoiIiIi0l/iOR7YH7dc\nC0zosc+vjTFrjTE/TmhkIiIiIpJW+ks8e3IRHV7v9F3gG8C5wInGmCsSFJeIiIiIpJk+53gCu4j2\nenaaCOzuXLDW/rHztTFmBXASsKyvA7a3d0S8Xs/gIxURERFJMpfLpSmFCdRf4rkKuBX4rTHmVKDG\nWnsQwBhTBCwnenNRM3AO8Eh/JwwEnLsPqaysgNraRsfO7xS1e3RRu0cXtXt0Ubsl1fWZeFprNxhj\nNhlj1hMtmXSjMeY6oN5a+7gxZhnwsjGmCXjDWttnb6eIiIiIjF799Xhirf1Wj1Vvx227G7g70UGJ\niIiISPoZ7M1FIiIiIiJHRYmniIiIiCSFEk8RERERSQolniIiIiKSFEo8RURERCQplHiKiIiISFIo\n8RQRERGRpFDiKSIiIiJJocRTRERERJJCiaeIiIiIJIUSTxERERFJCiWeIiIiIpIUXqcDEBERSVeh\nUIiqqu0JO14gkI/f3zTk41RWTsXj8SQgIpHBUeIpIiIyTKqqtrPk9uXkFo11OpQuwfp93LX0UqZN\nm+50KDIKKfEUEZFhN5p7/nKLxpLvKx/Wc4ikCiWeIiIy7NTzJyKgxFNERJJEPX8iorvaRURERCQp\nlHiKiIiISFIo8RQRERGRpFDiKSIiIiJJocRTRERERJJCiaeIiIiIJIUSTxERERFJCiWeIiIiIpIU\nSjxFREREJCmUeIqIiIhIUijxFBEREZGkUOIpIiIiIkmhxFNEREREkkKJp4iIiIgkhRJPEREREUkK\nr9MBiIx2oVCIqqrtCTteIJCP39805ONUVk7F4/EkICIREZEoJZ4iDquq2s6S25eTWzTW6VC6BOv3\ncdfSS5k2bbrToYiISBpR4ikyAuQWjSXfV+50GCIiIsNKczxFREREJCmUeIqIiIhIUijxFBEREZGk\nUOIpIiIiIkmhxFNEREREkkKJp4iIiIgkhRJPEREREUkKJZ4iIiIikhRKPEVEREQkKZR4ioiIiEhS\nKPEUERERkaRQ4ikiIiIiSeHtbwdjzB3AmUAEWGKtfb2XfX4MfNJaOy/xIYqIiIhIOuizx9MYMxc4\n1lo7C7gBuLuXfWYAZxNNTEVEREREetXfUPt84DEAa+1WwGeMye+xz+3AtwFX4sMTERERkXTRX+I5\nHtgft1wLTOhcMMZcD6wGPkp4ZCIiIiKSVvqd49mDi9iQujGmBPg8cAEwKcFxiYiIiEia6S/x3EW0\n17PTRGB37PW82LZ1QBYwzRjzc2vtN/s6oM+Xi9frOcpwh66srMCxcztJ7R65AoGes1dGhpKS/JT4\n+cVLtXgTJRXaPVqv89Ha7uGQavFK7/pLPFcBtwK/NcacCtRYaw8CWGuXAcsAjDGTgd/1l3QCBALB\noUU8BGVlBdTWNjp2fqeo3SOb39/kdAi98vubUuLn1ylV/r8TLVXaPVqv89Ha7kRLletc+tfnHE9r\n7QZgkzFmPXAncKMx5jpjzGU9du0aghcRERER6U2/czyttd/qsertXvapInoHvIiIiIhIr/TkIhER\nERFJCiWeIiIiIpIUSjxFREREJCmUeIqIiIhIUijxFBEREZGkUOIpIiIiIkmhxFNEREREkkKJp4iI\niIgkhRJPEREREUkKJZ4iIiIikhRKPEVEREQkKZR4ioiIiEhSKPEUERERkaRQ4ikiIiIiSaHEU0RE\nRESSQomniIiIiCSFEk8RERERSQolniIiIiKSFEo8RURERCQplHiKiIiISFIo8RQRERGRpFDiKSIi\nIiJJocRTRERERJJCiaeIiIiIJIUSTxERERFJCiWeIiIiIpIUSjxFREREJCmUeIqIiIhIUijxFBER\nEZGkUOIpIiIiIkmhxFNEREREkkKJp4iIiIgkhRJPEREREUkKJZ4iIiIikhRKPEVEREQkKZR4ioiI\niEhSKPEUERERkaRQ4ikiIiIiSaHEU0RERESSQomniIiIiCSFEk8RERERSQolniIiIiKSFEo8RURE\nRCQplHiKiIiISFIo8RQRERGRpFDiKSIiIiJJocRTRERERJLC298Oxpg7gDOBCLDEWvt63LavAH8H\nhIC3rLU3DlegIiIiIpLa+uzxNMbMBY611s4CbgDujtuWC1wNzLHWzgGON8acNZzBioiIiEjq6m+o\nfT7wGIC1divgM8bkx5aD1toF1tpQLAktAnYPa7QiIiIikrL6SzzHA/vjlmuBCfE7GGP+DfgAeMha\nW5XQ6EREREQkbfQ7x7MHF9G5nl2stT8xxtwJrDDGrLfWvtzXAXy+XLxezyBPmzhlZQWOndtJavfI\nFQjkOx1Cr0pK8lPi5xcv1eJNlFRo92i9zkdru4dDqsUrvesv8dxFtNez00Riw+nGmBLgZGvti9ba\nFmPMSmA20GfiGQgEhxDu0JSVFVBb2+jY+Z2ido9sfn+T0yH0yu9vSomfX6dU+f9OtFRp92i9zkdr\nuxMtVa5z6V9/Q+2rgMUAxphTgRpr7cHYtgzgHmNMXmz5E8DWYYlSRERERFJenz2e1toNxphNxpj1\nREsm3WiMuQ6ot9Y+boy5DXjBGNMBvGmtfSIJMYuIiIhICup3jqe19ls9Vr0dt+1+4P5EByUiIiIi\n6UdPLhIRERGRpFDiKSIiIiJJMdhySiIyQl1d8yyVzdFnOFTlTOCh8vMdjkhERKQ79XiKpIGra55l\nSvNuXESL7U5p3s1XP3yEcS0HnA5NRESkixJPkVQXiXT1dMYrDAW5YvcLDgQkIiLSOw21i6SQrFAr\nY9vqGNMWoKy1jrK2AGVtdbicDkxERGQAlHiKjECecIgx7XVdyeWYtujrwlD3J3+FcRHIKCCEh7xw\nS7dtQXcWyybMS2bYIiIifVLiKeKgSDhMxO/nhNZ9HOPf35Vo+tobcRPptm+DJ5dtuRPZn1lMbaaP\n2qxi9mcUE3J7APjqh490JaYRICvcxrhWP3uzS5PdLBFJMN08KOlCiadIEkQiEUIN9bRWV9NWU01r\nTQ2tNdW07aoh0tbG5+L2bXFnUJM9piu5rM30sT+zmBZPVp/nWDZhXtecznUlpzDvwCYW1W6gtL2e\nF0tPJeLSlG6RVNR582CnzpsHl02Ypw+WknKUeIokWLil+VBiWV0d/bemhlBTY7f9XF4vmRMm0FZY\nxMoPW2goOobaTB+N3lxwDX7W5t7sUn41ZXHX8o7c8SzetZoz67ZQ0tbA8vFn0+7OGHL7RGSYRCJk\nRDrIDHeQGW4nM9JOZrijz5sH43/nRVKBEk+RoxTp6KBt7564XszoV8f+/d13dLnIGFNG9vTpZJWX\nk1U+iczyCjLHjsXl9bJt2/us++0r5OeVJzS+uowC/lBxEZftWcP0YDWfr36aZRPm0ZCRn9DziIxW\nkUiESHs74ZaW2Fcz4ZYWIq2th5arqzk3+CH5oT3RZDLcTlaknYzO5LJHktlzik1fMiId5HY0E/Tm\nDGMrRRJLiadIPyKRCB3+A4cSzM5ezD27IRTqtq+nsJDcE2aQWV5BVkUFmRMryCovx53V9zD5cGn1\nZPLwxPNYUPsqpzZYvli9gmUT5rE7u8yReESGaqhzHT2RUFcCmNfRSKR6Jwebm+OSx9auBDLc2tK1\nPtJy6HX8esLhfs+5AKD58PXtLg9t7gzaXF7qPfnR125vbF1G1/IJTVWUtDce9v054Ta+VvUwe7JK\nqMqZwIe5E6nOHts171vSlzGmEngbeD1u9RvAncBjwGpr7dLYvhnAL4ETgY7Y1/XW2p3JjLmTEk+R\nOKGmJlqrd3YNj0f/rY7+gYnjysoie/LkaIIZ+8qsqMBbUOhQ5EcWdrlZVXYmBzKLOG//61xbs4qn\nxs5ia8EUp0MT6V8kgjcSIivcxmf2rKGipbZr05Tm3Xx9+wNszp9KqyeDrPgexM6vSPxyBx66J4rh\nP7xKzQBDcXm9uLKzcWdn4/WV4I69dmdn487K7r6cnY07O4d9dX7ueXY7nqLxsWTSS6s7g3Z3BuEB\nzrteW/rxbjcPNnhyWTZxHpXB3UwJ7qKieR/jW/18sm4z7S4PO3LGUZUzkQ9zJ7A/s/iopu5IYq27\n9PJngfNii8/PWf5oIu4O22qt7Va6xBjzDPAMED+v6lqgw1o7O7bPdcBXgW8N9oTGGJe1duDd8r1Q\n4ilpp/rntxPcugUL5B4/g4pvLj1sn3BrK227d9Nas7NrHmZrTQ2h+rruO3o8ZI4bT1Z5eawXcxKZ\n5eVklI7B5U6hm3VcLjYVn0Ago5BP73mJy/auZW17A+t9J+uPkgybSCRCpK2NcHMzkQP7qWivpzjo\nIivcRlaonexwG1nh9uhyt9ftZIUOLXv6GH7ODrdzWsN7vW4L4Yr1GmbQ5M2lzeXtWm5zeznY1sZZ\npx5DyfgJhyWM7qys7svZ2bi8g/+TWbvtfbatqSN/iKMM8TcPLpswj71ZpezNKmWj70Qywu1UNO9j\nSnAXU5p3My24i2nBXXAAGj05VOVGe0OrciZoWN4BsaRzQdyqBesuvbwauHTO8kf/muDTXQ4sJtq7\n2akIKOhcsNbe3/naGPMF4J+BMPALa+2fjTFXAd8g2jO6yVr7dWPMvwNTgCnGmHOB7wNzAA/wS2vt\ngwMNUImnpJXqn99O8N3NXcvBdzez7V++hu/8Cwm3tXbNxWzftw8i3f+YeUtKyTv5lFgvZnQuZsb4\n8bgz0ueGnO155fyh4kKu3L2as/1vUdJWz4qxszU0l4IG8gFrKOKTxnBzkFBzc/R1MEioOdi1PhyM\nrj+0rvv+8dNR/hGgof9zt7mivYJBTzaBjAJaPZm0ujM4vumjXh+WEHRn8eiEc2lzZ9DamVi6Mgi5\n3H1+sGoK1DD73E9SOm36oH8+ydbz5sF47e4MPswr58PYPPH8jmBXb2hl825OatzOSY3bo8fJ9GHd\nBUQ+HEf4mMm4MzKT1oZ0te7Sy28Hruxjl8m9rCsHXo0loL15eM7yRwf9S22tPWiM6bn6j8D1xpit\nwApgmbV2vTGmAPgucBKQDdxvjHkK+CFwirU2aIxZHks0I0CGtfYcY8zZwDHW2rnGmCxgkzHmMWtt\n60BiVOIpaSW4dcth60INDexf9ueuZXdeHjnTTdc8zKzyCjInluPJzU1mqI7Zn+Xj/opFXL7nRWY2\nVVHc3sSyCfPUE5JCevuAtX3pN5h40xKyJ1d2SxpDwWA0QWzumSAOLmkcKFdWFu6cHLwFhbjHjsOd\nk4M7J5eDHW28tNVPJLeEVncGre7MQ/96MmiJLbe5M484BJ3do6wQxIadVVaomyZvLu8UTuOdwmkQ\niTC2LRDtDY0Ny59NgPCDf2LbsofJMceRO2MmeTNPJLO8ApdGQFLJccaY+Ocir7LW/hi6fz6z1vqB\nU40xc4CFwAPGmHuBp4gO17cCrcBlxphTgfettZ1PK3kR+Hjs9auxf2cBn4w7twuYAFQNJGglnjIq\nuHNymPCPN5JVXo6nqHjUv7kGvTk8MHEhF+17mRObPuS66hU8MmE+tVk+p0OTAejtA1ZHIMCOH96G\nOzd3CEljNp7cHLyFhbjHjcedk4MnljhGE8gc3Lm5h63zxL12eXrvPd+27X2e2fkK+b6jr97wUPn5\nh811VDmhfrhc7MsqYV9WCRt9J+INdzCmdgtfOiGLjJoagpvfIbj5HfY//BCeoqJoEjrjRHJnzMBb\nVOx09Ckh1jN5xN7JXobaAWoY+lD7ez3neMZ0G84zxmQSneO5DlhnjPkfognlcqDnp7ww3RPXTA7d\nGtce+7cVuMda+5OjCVqJp6SVnGOn0/y+7bbO6/N19QTJISG3hyfHzcGfWcQ5/jf5fPVKlo8/h215\nFU6HJn2IhEKHTRM5tDEyqKTRk5vb1SPpzslJiXnLPec6yuB0uL18kFmK+7xPUjltOh11AYJbtnBw\n8zsEt2ymccPLNG54GYDMiknkzZxJ7owTyZlucGdqWP5ozFn+6PmxIfXOT101c5Y/OpxvtD17Vu4B\n1gG/iS1PArYBW4n2muYBIaKJ6GXAdGNMvrW2CZgL/IDuifNG4GfGmP8AsoCfWmu/NtDglHhKWsme\nMrVb4un1+Zh6+x0ORjTCuVy8XHIyBzILuWTveq7Y/QKrx5zGi5GRd3e+QOvOHey5755et42WD1h9\nzXWUwfMW+yicNZvCWbOJhMO01VRHk9DNm2l+/z0C1TsJPPM0rowMcqYbcmeeSN6ME8ms0LD8IF1K\nNLHrfJ0IPXs2JxGdw1kMZBtjLgQ+S/RGod/EbiRqAdqAf4rN4fwe8FzsEL+IrVsKPG2MCQNrY/NB\nF3Sez1q7ITbMvoFokvtfgwlaiaekjY6GBupeXI2noAA8HjweN+O/OuAPYaPae/mV1HvzWbz7BRbs\nf53CrHIioU84HZbEhNvb8T+1HP/KFRAKUThrDgc3v9NVhUEfsCQRXG43WZOOIWvSMZRcuIhwWxvN\n71uCm9/p6hENbtnMfh6K1iyOzQ3NPWEm3mINy/clNqSesF5Oa20V8Ike63YSvVGoN1cc4TgPAA/0\nWPcY0Vqg8etu7bH8HeA7gwo6RomnpI3AqqeJtLVRsvgqfPMXUFZWQG3t4UWXpXd7ssdwf8UiFu9e\nzSdaawj/+UFC31iKJy/P6dBGteZtH7D3d/fStnsX3pJSxn3xevJOPImWj6rY9cu7cLtd+oAlw8Kd\nmUnezBPJm3kiZUBHXR3BdzcfGpZ/ZQONr2wAILO8IpqEztSwvPRNiaekhVBjI3UvPI+nqJiis89x\nOpyU1ZiRxx8rLmTRzmc5oaqKHT/6PuVf+zqZ48Y7HdqoE25tZf9jy6h7/lmIRCiadx5lVyzGnR2t\nPpA9uZKpt9+hD1iSNN7iYgrPmk3hWbOJRCK0VVdzcEv05qTm9y2BmmoCq57G5fWSM/04cmfOJHfG\nTLIqJqXE/GFJDiWekhYCzz5DpLWVks9cobp0Q9TuzuBPBSfzg+lB2je+wo4ffZ+JX/1nco873unQ\nRo3gu1vY+/v7aK+tJWPcOMZd93fkmuOcDkuki8vlImvSJLImTaLkgosODctveYeDmzcTfHdzV8kv\nT0F0WD46ND8Tb7GqZ4xmSjwl5YWamqhb/RyewkKKzjnX6XDSQsTlwj3/PMacMIO9f7if6l/czrgv\nXEfRHPUmD6dQMMj+Rx6i/qU14Hbju3ARpZdepmFLGfG6DctfCR31ddG75bfEhuU3bqBxY9yw/IyZ\n5M6cSc7043BnZTkcvSSTEk9JeYHnVhFuaaFMf6ATrmjOOWSMKWPX//tlbJ7hbsZccaWGzYZB01tv\nsu+P99MRCJBZMYnx199AdmWl02GJHBVvUTGFZ82i8KxZ0WH5zrvlt2ym2b4XHZZ/9pnYsLyJ9ojO\nPPGwYfnhfkKXJJ8ST0lpoYMHqXv+WTwFBRTNVU2/4ZB7/Akc8+3vUfOfdxB4ZiVte/cw4cv/gDs7\n2+nQ0kJHYwO1D/yJxldfAY+H0k9/hpKLLj6q54KLjEQul4usiklkVcSG5dvbaH7//Wjx+i2bCb67\nheC7W2DZw3gKCrqG5evXraUlrjxezyd0SWrSO5uktLrnnyXc3MyYxVdpuGYYZY4bxzHf+i67f/1f\nHHzzDXb+xw+Z+M9fJ6NEjyk8WpFIhMbXNlL7p/8l1NRI9tSpjLvuBrLKj/7JPiKpwJ2RSd6MmeTN\nmAlAR319dE7o5s0c3LKZxo2v0LjxlV6/tyMQYNcv7xpp5cM8wLQEH3Mb0aLuR2SMuRH4PNEnCeUA\n37bWPj+QgxtjTgE+Y6399yHGOWhKPCVlhYJBAs+twpNfQPG5850OJ+158vIoX/Iv7Hvgj9SveZEd\nP7yN8puWkD1lqtOhpZyOugB7//h7Dr75Bq7MTMqu+izFC87XFAYZlbxFRRR+chaFn4wNy++KPsqz\n9s8POh3aQE37xOW3vJdbNDYhBwvW7+PVR289DrBH2scYUwl8GTjdWhsyxhjgt8CAEk9r7VvAWwkI\nd9CUeErKqlv9HOFgkDGXL9awb5K4vF7Gfv46MidMpPahB9j50x8z/oavUHC6is0PRCQSoWHdS9T+\n+UHCzc3kHH8C4774JTLHJuYPlkiqc7lcZJVXkFVewcG33+66M75T5xO6RprcorHk+5I6WlEEZBN9\nZGXQWmuBc2NPGPo+0ScU1QFXAX8m+lSitcaYHGALcAPRpxddaYz5gGjB+Nmx77mY6OM9Hybam/oS\ncLa1dp4x5m7gNKK9vP/PWnv/YAPXx2tJSaHmZgKrnsGdl0fx/POcDmdUcblc+BYsZOI/LwG3h92/\n/hUHnlxO5EjPDxcA2mtrqfnF7ey9/z6IRBj7heup+JelSjpFjqDim0vx+g6VXup8Qpfmd3b1WL4K\nfGiMuc8Yc6Uxxks0If2ctXYeUA9cACwDPhX71vOBZ+g+jD8FuN9aOwvwAScTfczmg9bac4kmtxFj\njA9YZK2dDcwBMo4m9lGReFb//HbsV77E+ssWU/3z250ORxKg/oXnCQcP4jv/gq6C2pJc+Sd/jGO+\ndTPe0lIOPP4oe/7nt4Tb25wOa8SJhMMEnltF1S03E3x3C3knn8Lk235E8dxzNbQu0o+JNy3B6/OR\nWVoyIns6nWStvQ6YC7wJ/B9gFXAA+K0x5kVgPlACPAFcGPu2TwOP9DhUg7X2ndjraqLJ6/HA+ti6\n5YDLWhsArDHmceBq4PdHE3fav+tV//z2aFd9JAKRSNddcS0fVTkdmhylcEsL/lVP487NpXj+AqfD\nGdWyKiZxzLe/R/bUaTRu3ED1z35KR0OD02GNGK27drHzP35E7YN/wpWZyfiv/EPspqwSp0MTSQmd\nT+g6497/Vk9nD8aYLGvtVmvtXcCZwCTgXuDGWE/lXwCstfVATWwe6FnAasAVd6iOHod2xb4OG8ay\n1i4CbgU+RjShHbS0TzyDW7cctq7zrjhJTXUvrCbc1IRvwUI8ublOhzPqeYuKqFj6fyn4xCdp2fYB\nO350G6011U6H5ahIRwcHnnqCHbd9j5ZtH1BwxieovO1HFJ55Fi6Xq/8DiIj0wRjzZeBeY0znG0ox\n0WSxANhpjCkm2uPZWe7lMeA7wMvW2vAATrENOD32+qLYOScbY75mrX3DWrsUOKqyJrq5SFJKuLWV\nwKqVuHNyKF5wvtPhSIw7I9qblzlhAgf+8hg7f/wDJvzDV8k76WSnQ0u6lh0fsfe+e2jduQNPUTHj\nPv8F8j9+mtNhicgwCtbvS/ax7gWOA14xxjQRnW/5z8AniQ6RbwN+CtxijFlOtPfzP4kOtUO0NzMS\n9zpeBLgL+LMx5gqic0lDwC7gLGPM1URvOrrnaNqX9oln7vEzUuauOOlf3YurCTU2UnLJpXhy85wO\nR+K4XC5KP/VpMsdPYM+9/03N3XdQdvW1FJ+3YFT08oXb2/A/sRz/0ysgHKZwztmUXXkNnjxdpyJp\nblus/FFCj9nXxlivZW+PcVoJ3BK3HH/XeVexa2vtGmBN7PXYuPVXAhhjZgA3WWtfNsZ8FhhjrW0H\nPjvIdhwm7RPPim8uZfvSb9ARCERXuN1M+cnPcHk8zgYmgxZubSXwzErc2dn4Fix0Ohw5goIzPoG3\ndAy7fnkntQ/+L217djP2mmsPexJPKBSiqmp7ws4bCOTj9zcN+TiVlVPxDPL9ofmD96OPFN2zG++Y\nMYz7wvXkzTxxyLGISEoI0UfNzRTVCPzGGBMh2r4vJerAaZ94QvSuuF2/vItwMEi4tZX6l9ZQPE8F\nx1NN/UsvEmpooGTRJXjy850OR/qQM3Uqx9x8C7t+eSf1L66mfd9eJvzjV7v1UldVbWfJ7ctJVNHl\nRAjW7+OupZcybdr0Ae0fbmlh/2PLqFv9HADF8xeorqyIpDxr7U7g7OE49qhIPDvviivKCLHpH25k\n/18epeDMMzVUm0LCbW34n16BKysL38IL+/8GcVxGaSmT/u/N7P7vX3PwrTfZ+aMfMPFr3+hWt9KB\nossJc3DLZvb+/j469u8nY/x4xl93AznTB5awioiMVml/V3u8zOJiSi7+FOGmJvxPLHc6HBmE+rVr\nCNXXUzzvPPV2phB3djYTb/wavoUX0rZnNzt+dBtB+57TYQ1JKHiQPb+7h5pf3E6H30/JokuYfMtt\nSjpFRAZgVCWeAMULzidjTBmB1c/RtmeP0+HIAITb2/CvfApXZia+C9TbmWpcbjdlV13DuC9+iXBz\nM9U//yn169c6HdZRaXrjr1R992Ya1q0la9IxHHPz96JD6xmZTocmIpISRl3i6c7IZMyVV0EoRO3D\nDzodjgxAw7p1hOrqKJ43H29BodPhyFEqOmcuFd/4V9xZ2ey97x7CL6zGlSKP2exoaGD3b37Frv+6\nm/DBJkovu5xjbv6eClqLiAzSqEs8AfJPPZ0ccxwH33qTg5vf6f8bxDHh9nb8K5+M9nYuvMjpcGSI\nco8/gWNu/i4Z48YTeWUDn218m4xwu9NhHVEkEqFh4waqvvdtGl97leyp0zjme7dSesmlh92lLyIi\n/RuViafL5aLsmmvB5aL2zw8SCYWcDkmOoOHl9XT4/RTNnYe3qMjpcCQBMseN55hvfxcmVzKjvZbP\n1TxDfkfQ6bAO0+73s+s/72TPf/+GSFsbZddcy6R/u5msial5M5SIyEgwKhNPgOxjJlM4+2zaaqqp\nf2mN0+E9rjjPAAAcOElEQVRILyIdHfhXPIErI4OSC9TbmU48eXm4r76G17ImMr7Vz3U7n2J8y36n\nwwKivZx1a17ko1tu5uDf3iL3hBlMvvUH+BYsxOUetW+ZIiIJMarfRcd85nLc2dns/8ujhIIHnQ5H\nemjYsJ6OAwcoOudcvMXFTocjCebyePhL3vE8P+Z08kPNfK7mGY5r+sjRmEpCQcIP/C/7/vA7AMZ9\n8UuU/8tSMstGTq1REZFUNqoTT2+RyiuNVJGODvxPPYnL68V34SKnw5Hh4nLxWvEMHpkwnzAuPrNn\nDWf534Yk33TkioQ5I7CFm+o2wkcfkXfKx5h8248oOmfuqHjcp4hIsozqxBNUXmmkati4gfb9tRSe\nPZcMn8/pcGSYbcur4I8VF1HvzWOu/w0u2bceTyQ5c6/HtNbx+eqnOe/A67S7PLg+fRkTb1qi605E\nZBj0m3gaY+4wxrxsjFlvjDm9x7Z5xpgNxph1xph7jDEp1zWg8kojTyQUwv/kE+DxUHKRejtHi9os\nH7+vWERN1hhObNzONTWryAm1DNv53JEws/x/4/qdT1Leup/N+ZXcVfxJ3DNmqpdTRGSY9Jl4GmPm\nAsdaa2cBNwB399jlt8Bia+0coABIyereKq80sjS++grttfsomnM2GSWlTocjSXTQm8MD5QvZkl/J\npJZartu5gtK2uoSfZ1zLAa7f+RTn+N+k2ZPFIxPm8cT4cwi6VQheRGQ49dfjOR94DMBauxXwGWPi\nn1d4mrW2Jva6FihJfIjDT+WVRo5IOMyBzt7ORZc4HY44oMPtZfm4s1lbcgrFHU18oXolUw7W9P+N\nA+ANdzB3/yauq17B2LYAbxYey/8c82k+yJuUkOOLiEjf+ks8xwPxNU5qgQmdC9baBgBjzARgIbAi\n0QEmS7fySmtVXskpja9tpH3vHgpnzSajdIzT4YhTXC7Wl5zCX8adjTcS4srdqzm1buuQDlnRvJcv\n7XySs+o20+DN44GJ5/P02Fm0etTLKSKSLIN99IYL6Ha7qTFmLLAc+CdrbaC/A/h8uXi9nkGeNnHK\nygqOuK3oy19k0+uv4l/+OFMuWoA3Py+JkQ2vvto9UkRCIXaufArcbqZ/4RqyExBzKrQ7EMjvfycH\nlJTkD+vPbyDtfrdgCvUZ+Vy++wUW7n+V0vZ6nhtzBhHXwO+LzAy3M/fAXzmt/j0iwGtFJ/BS6cdo\nd2f0uv9wt3s4pEK8us5HFl3n4pT+Es9dRHs9O00EdncuGGMKifZyftta+9xAThgIOPeEkrKyAmpr\nG/vYw0vJokvY/+gj2N/9L2VXfzZpsQ2n/ts9MjS+upHm6moKZ59NozuXxiHGnCrt9vubnA6hV35/\n07D+/Aba7l3ZZfy+YhGLd7/AafXv4Wtr4C/j5w6op7IyuIuL9m2gqOMg+zOKWDF2FrtyyvqNKxWu\nm066zodmpFznyabrXJzSX7fBKmAxgDHmVKDGWhtfaf3nwB3W2lXDFF/SFZ+/UOWVHBCd27kc3G5K\nLv6U0+HICNOQkc8fKy7kg9wKpjbv5gvVKylqP/IfoaxQK4v2rueaXc9R0BFkve8k7pt0Sb9Jp4iI\nDK8+E09r7QZgkzFmPXAncKMx5jpjzGXGmBzgC8CXjTEvxL6+koSYh1W38kqPPOR0OKNG01830bar\nhsIzzyJzrJ4SI4drc2ewbMK5vFo8gzHt9Vy3cwUVzXsP2880fcRXdizn5MZt7Mkq4XeTLmZt6ccJ\nuZ2b4iMiIlH9zvG01n6rx6q3415nJzackaGrvNKbb3Bwy2byZsx0OqS01tXb6XLRdNJJbNv2fkKO\nGwjkJ2SYq7JyKh6PkpaRIOJys3rM6RzIKGRh7UauqXmWQEYBY9rrAQi6s8kLt9DhcvNi6cd5tXgm\n4UHMB02GUChEVdX2hB1P17mIpJLB3lw0KrhcLsqu/iw7fnArtQ89QO73bsWlN+Rh0/TmG7RV78Q1\n80S+fu96cotGTo9nsH4fdy29lGnTpjsdisR5q8hQl1HAVbueoyyWdALRpBM3fxl3Nu/nT3YwwiOr\nqtrOktuX6zoXkVFJiecRZE+upHD2HBrWraV+7RqKz53vdEhpKRKJ4H/iL+By4Zo9m9w9H5DvK3c6\nLEkBH+VOwM3hz3T3Eub82tdGbOIJkFs0Vte5iIxKI2sMaoQZ85krcGVlc+DxxwgFD/b/DTJoB996\nk9adOyg44xO4VLdTREQkrSnx7IO3qJjSiy8h1NSI/4nlToeTdiKRCAdivZ0lF1/qdDiSgqpyJhy2\nrsGTy7IJ8xyIRkRE+qPEsx/F5y/EO2aMyisNg4Nv/43Wj6rIP+10sso17CiD91D5+TR4cruWGzy5\n/GrKYvZmlzoYlYiIHIkSz364MzIpW3y1yislWNfcTqD0EvV2ytFbNmEeDZ5c9XSKiKQA3Vw0APmn\nqbxSogU3v0PLh9vJP/U0siomOR2OpLC92aX8aspip8MQEZEBUI/nAHSWV8LlovahB4iEQk6HlNK6\n5nYCJertFBERGTWUeA5QZ3mltppq6teucTqclBZ8dwst2z4g72MfJ/uYkVvyRkRERBJLiecgqLzS\n0HWf2/lph6MRERGRZFLiOQjdyis9+YTT4aSk5ve20vy+Je/kU8iurHQ6HBEREUkiJZ6D1FVe6fln\nadur8kqDdWhup3o7RURERhslnoPUrbzSwyqvNBhB+x7N720l98STyJk61elwREREJMmUeB6F/NNO\nJ2e66SqvJAPTNbfzU+rtFBERGY2UeB4Fl8tF2TXXqrzSIDS//z7Bd7eQO2MmOdOOdTocERERcYAS\nz6PUrbzSupecDmfEO/BkZ2/nZQ5HIiIiIk5R4jkEYy6LlVd67FGVV+pD87YPCG5+h5zjTyBn+nSn\nwxERERGHKPEcAm+xyisNxIEnlgNQeql6O0VEREYzJZ5DpPJKfWv5cDvBd/5GjjmOXHOc0+GIiIiI\ng5R4DpHKK/Wts26nejtFREREiWcCxJdXCr67xelwRoyWj6o4+Le3yJluyDnueKfDEREREYcp8UyA\n+PJK+x78k8orxXQ9pehTn8blcjkcjYiIiDhNiWeCqLxSdy07PuLgm2+QPe1Yck+Y4XQ4IiIiMgIo\n8Uyg7uWVgk6H4yj/k7E72T91qXo7RUREBFDimVDdyystdzocx7RW76Tpr5vInjKV3JknOR2OiIiI\njBBKPBNM5ZXgQCzpLlFvp4iIiMRR4plgo728UmtNDU2bXidrciV5J53idDgiIiIygijxHAajubyS\n/6knIBKhVHeyi4iISA9KPIeBy+Wi7OrRV16pbfcuGl/bSNakY8g75WNOhyMiIiIjjBLPYZJdWUnh\nrNFVXulArLdTdTtFRESkN0o8h9GYz4ye8kpte/bQuPEVMismkf+xjzsdjoiIiIxASjyHUbfySk+l\nd3kl/4rY3M5LLsXl1mUlIiIih1OGMMyKz1+It7SUwHPpW16pbd8+Gl7ZQObEcvJPPc3pcERERGSE\nUuI5zNwZmZRdmd7llfwrnoBwWL2dIiIi0iev0wGMBvmnndGtvFI6Pbu8vbaWhg0vkzl+Avmnn+F0\nOCIiMgKEQiGqqrYn7HiBQD5+f9OQj1NZORWPx5OAiORoKfFMgs7ySjt+eCv7HvwTk2+5LW16Bv0r\nn4RQKPqUojRpk4iIDE1V1XaW3L6c3KKxTofSJVi/j7uWXsq0adOdDmVUU+KZJJ3llRrWr6V+7RqK\n585zOqQhaz+wn/r168gYN56CM850OhwRERlBcovGku8rdzoMGWHURZVE6VZeyb/iKQiFKL3kU+rt\nFBERkX6N6B7PdJsj0lleaf+jj+B/ajllV14z5Fic0u4/QP26l8gYO46CT3zS6XBEREQkBYzoxDMd\n54gUn7+QujUvEHjuWYrOmUfmuHEJjjA5/CtXROd2XnwJLk3UFhERkQEY0YknpN8ckc7ySrt//Stq\nH36Q8puWOB3SoLUHAjSsXUPGmDIKzzzL6XBEREQkRWhingN6lldKNYGnVxDp6Ij2dnpH/GcXERER\nGSGUeDqgs7wSLhf7HvwTkXDY6ZAGrKOujvqXXsRbWkrhWbOdDkdERERSiBJPh3SWV2qrqaZ+7UtO\nhzNg/mdWEmlvp2SRejtFRERkcJR4OihaXimLA48vS4nySh319dSveQFvSQmFs+Y4HY6IiIikGCWe\nDvIWF1Oy6BJCjY34n1rudDj9Cqx6mkhbGyUXXYw7I8PpcERERCTF9DtWaoy5AzgTiABLrLWvx23L\nBn4LnGCt1YO6j4Jv4QXUv/TiiC+v1NHYQN0Lz+P1+Sicc47T4YiIiEgK6rPH0xgzFzjWWjsLuAG4\nu8cuPwVeHabYRoXO8kqEQtQ+8pDT4RxRYNUzRNra8F24SL2dIiIiclT6G2qfDzwGYK3dCviMMflx\n278FPDFMsY0aXeWV3vjriCyvFGpqom7183iKiik6Z67T4YiIiEiK6i/xHA/sj1uuBSZ0LlhrDwKu\nYYhrVBnp5ZUCzz5DpLWFkgsvwp2R6XQ4IiIikqIGe3ORi+hcT0mwkVpeKdTURN3zz+IpLKTonHOd\nDkdERERSWH83F+0i2uvZaSKwu8c+g0pEfb5cvN6BPds7EMjvfycHlJTkU1ZWkPDjFn35OjZteo3A\n8keZctF8vHl5CTv20ca749mnCLe0UPnZqxlXMSZh8fRmtP1/d1K7Rxa1e3io3SOL2i1O6S/xXAXc\nCvzWGHMqUBMbXo83qKH2QGDg9Sr9/qbBHDpp/P4mamsbh+HIXnwXXcyBx5Zh738getNRApSVFRxV\nvKHgQWqWP4mnoADv6bOGqc2HjL7/70PHH4nU7uE7/kikdg/f8UcitVuc0udQu7V2A7DJGLMeuBO4\n0RhznTHmMgBjzHPA08BMY8zbxpgvDXvEac53/gV4S0sJPLeKtr17HY2l7vnnCDc341t4Ee6sLEdj\nERERkdTXbx1Pa+23eqx6O27bgoRHNMq5M6PllXb/+lfUPvIQ5Td+zZE4Qs3NBJ59Bnd+PsXz5jsS\ng4iIiKQXPbloBBoJ5ZXqnn+WcDBIycILcWdnOxKDiIiIpBclniNQt/JKDz2Q9PJK4ZZYb2duHkXz\nzkvquUVERCR9KfEcobIrKyk8azZt1TuTXl6pbvXzhA8exLfwAjw5OUk9t4iIiKQvJZ4j2JjLF+PK\nyuLA48sIBQdeDWAowi0tBFY9gzs3l+L5msIrIiIiiaPEcwTzFhdTsugSQo2N+J9KzpNJ615cTaip\nEd+ChXhyc5NyThERERkdlHiOcMksrxRubSXwzErcOTkUn3f+sJ5LRERERh8lniOcOzOTssVXQyhE\n7SMPDeu56te8QKixkeLzzseTwKcmiYiIiIASz5SQf/rwl1cKt7Xhf3oFrqxsfAsWDss5REREZHRT\n4pkCklFeqf6lNYQaGvCdtwBP/sh8xq6IiIikNiWeKaJbeaV1iS2vFG5vw//0U7iysvCdf0FCjy0i\nIiLSSYlnChlz+RXR8kqPJba8UsPalwjV1VE87zw8BQUJO66IiIhIPCWeKcRb7Et4eaVwezv+lStw\nZWbiW3hhQo4pIiIi0hslnikm0eWVGtavpSPgp/jc+XgLCxMQoYiIiEjvlHimmPjySvsf+fOQjhXp\n6MC/4ilcGRn4LrgoQRGKiIiI9E6JZwrqLK/U9MYmglvfPerj1L+8jg7/AYrmzsNbVJTACEVEREQO\np8QzBXUrr/Tgn46qvFK0t/NJXF4vJRcuGoYoRURERLpT4pmihlpeqeGVl+nYv5+ic87FW1w8DBGK\niIiIdKfEM4UdbXmlSCiE/6kncHm9+NTbKSIiIkmixDOFeYt9lFx08aDLKzW8soH22loKzz6HjJKS\nYYxQRERE5BAlninOt/DCQZVXioRC+Fc8AR4PJRdenIQIRURERKKUeKa4wZZXanx1I+1791I052wy\nSkuTEKGIiIhIlBLPNJB/+hlkHzu93/JKkXCYA08tj/Z2XqTeThEREUkuJZ5pwOVyMfaaawGofejI\n5ZUaX3uV9j17KJw1m4wxZckMUURERESJZ7rIrpxC4aw5tO7svbxSJBzG/+RycLspWXSJAxGKiIjI\naKfEM430VV6padPrtO3eReFZs8ksG+tQhCIiIjKaKfFMI93KK614smt9JBzmgHo7RURExGFKPNNM\nZ3mluudW0bZvHwD+ja/SVlNN4ZlnkTlunMMRioiIyGilxDPNdJZXinR0sP/hh4iEw+x86GFwuSi5\nWL2dIiIi4hyv0wFI4uWffgbZz0fLK73/938HgKewiMzxExyOTEREREYz9XimIZfLBaFQt3Whhnq2\nL/0GLR9VOROUiIiIjHpKPNNUS9WHh63rCATY9cu7HIhGRERERImniIiIiCSJEs80lXv8jMPWeX0+\nJt60xIFoRERERJR4pq2Kby7F6/N1LXt9PqbefgfZkyudC0pERERGNSWeaWziTUvw+nxklpaop1NE\nREQcp3JKaSx7ciVTb7+DsrICamsbnQ5HRERERjn1eIqIiIhIUijxFBEREZGkUOIpIiIiIkmhxFNE\nREREkkKJp4iIiIgkhRJPEREREUkKJZ4iIiIikhRKPEVEREQkKZR4ioiIiEhS6MlFI1AoFKKqanvC\njhcI5OP3Nw35OJWVU/F4PAmISEREREYjJZ4jUFXVdpbcvpzcorFOh9IlWL+Pu5ZeyrRp050ORURE\nRFJUv4mnMeYO4EwgAiyx1r4et20B8EMgBKyw1v5guAIdbXKLxpLvK3c6DBEREZGE6XOOpzFmLnCs\ntXYWcANwd49d7gIuB2YDC40xJwxLlCIiIiKS8vq7uWg+8BiAtXYr4DPG5AMYY6YCfmttjbU2AqwA\nzhvOYEVEREQkdfWXeI4H9sct18bWdW6rjdu2D5iQuNBEREREJJ0M9uYi11FuO2rB+n3Dcdijlqx4\n1O6RQe1Oj/MMlNqdHucZKLU7Pc4zUCMtHumFMeYWY8zfxy1vM8bkxV5PNsa83GPfrzoRp4iIiIiM\nfP0Nta8CFgMYY04Faqy1BwGstR8BhbEE1AtcHNtfREREROQw/Q6PG2N+DJxDtGTSjcCpQL219nFj\nzNnAf8R2fcRa+4thi1REREREREREREREREREREREREREREREREREEmpYam86yRhTCbwNdD5TPiu2\n/E/AdmAH0RulACLW2vnJjnG4xNr+sLX2jLh1/w5cC9QQ/f/OAX5srX3ciRgTrcf/twvoAH4ETCT6\nmNdsYCawKfYtX7TW7kx+pH0b6nVrjLke+D7wAdGfQyvwBWvtPmPM74Aya+3FcfsvAp4EKolWt+j1\n3NbacAKbeZihXrPGmCqiP5sOIB+4x1r7m9hxtwNnWmtfi9v/VWCztfZLsZ/LqcCB2HkygaXW2vUJ\nb+jhcVcyhOs2Vdt9JMaY6cCdwBjAA7wM/CswC7gNCAMFwB+stXcaY84FHgbeIXr9NgE3W2vfNMb8\nK9EqK8VAObA5dpqF1tr2pDVqAIbYboBc4Glr7S2x47UD62LbcoD7rLW/SU5r+tfP+9w59NLmXo5R\nxaH3QzcQBP7OWru7x7Xd6evW2reGoTlylAZbQD5VbLXWzutcMMbcB3wOiAAXWmuDjkWWfBHgTmvt\nrwCMMT7gTWPMSmttq7OhJUzX/3fsUa5PAFdba+cZYyYTrbgwr88jjAxDuW4jwAPW2v8T+97vAn8H\n/CS2vdIYU2ytrYstLwa29XPua4E/DrFNR2Mw12zXz8YYkwtsN8b8NrZtO3AV8FrsOJMBX+x7Or/3\n36y1K2LbpwIrgeOGr2ndDOW6TeV2d2OM8QCPADdZa9fG1t0NfA+4Ephrrd1jjMkGnjfGPBj71het\ntVfG9v848IgxZo619mfAz4wxc2PHvDLZbRqIBLXbBTwba/c6oC7umsoE/hr7vdmR3Nb16Ujvc9/h\n8DY/YK3d2+P7u70fGmOuI/qh+8v0uLZlZOqvjme62Agc63QQDurq2bbWBoDdpOnjTa2124EfAjfF\nVqVyr/5gr9v4to4HqmOvI8BzwGcAjDEZwHSgr57fV2P7OGUw12znvqVArbU2QrTNrwDnxe23mD5q\nDceuncLYH/OkOsrrNuXbHXM+8G5n8hWzlGjvVwnR3i+stS3W2tnW2j09D2CtfQO4F7g+bvVI/91P\nRLsjRHsPD3ufsNa2Ee1NnDIMsSdS5/ucj8Pb3DPp7I3T71UySGmfeMb+yH4a+Gts1Uh/MxpWxpjj\ngHEcSkrS0SZghtNBDMVRXLcu4GpjzAvGmLeBjwHL4rY/SrQXBaJJyYtHOmbs3JdyaJjXUQO4Zlca\nY9YQjff7cevbgTeMMWfGli8GVtC93V2vjTHnALtif8ydMNjrNl3afRzQbSjUWtsaS5y+C7xmjHnc\nGPNVY0xxH8d5ndT6vR9yu40x+cAFHHqfiN9WQvR94O2ER54gce9zm4j29Ma32dfHt8Zfy4vp/l41\nqv/Gp4J0HWo/zhjzQuz1ycBPrLV/McbcRfTNunOuXK219ipnQkwaF7DEGLMYKCQ6p+az1toOZ8Ma\nVgUcmg+ZSoZy3UaAB+OG2j8P/Ab4Ymx7FdFeLR9wBfBLYE4/516ewLYNxmCv2c4h5wLgOWPMW0BL\nbNsjwFXGmBogQHQuYPx5fhybEzgmtu3axDdnwAZ73aZLuyNE5zcexlr7a2PMY0STq8uA78Seoteb\nQlLr9/5o2z039rvqIdrT92/W2r/FthXF/R6HgX+11vqHrQVH54jvNb21ubeeXg69H04F1gL/EFsf\nf213+py1dtdwNESOTromnu/FzXN5GHg/tn5Uz/E0xowHVjOCPwEnyOn00gOQAgZ83cYm6d8XW/xm\n7N/4T/qPAj/ocfzHifYOnGitfcsYM5BzO+GI12zsxqO5wN+stUviv8la22iMeRE4C+j8w/Yc8GPg\nI7r3AHee59+stSuMMScD/wPY4WjQAB3xuk3zdm/l0BQDoGt+ogG2xYZbfw/83hhzL7CQaLt6SrXf\n+6Nt95q4OZ4b6P5+Xp8C89l7fa8xxuT01ubYHOSe137nh64bgek29ihvNMczJaT9UDvROTM/iU3A\nH41csS9inxx/D9ziaETDyBgzDfgGcIfTsQxRn9ettbbKWjsv9tXbH9szif5hi/cI0ST1xQGeO2eQ\nMSfKEa9Za+2/x9q8pMf+nTdanEFcu2N3Mb9B9E7xJzh8GK7zPH8jmrT80zC0p1/9Xbfp2u6YZ4HJ\nxphLAIwxbuCnwLeJ3hxTELd+It1viiO27XTgcuCeZAWdAENuN/AvwH85OD93qDrfa07mCG0+wrXf\n6dfAubHv75SqP4tRI10Tz665StbaKqKf+G+OX5/GZhpj3u38IlqWI77dvwAuNcac4Ex4w+K42NzG\nl4E/AV+11sbPB0yV//ehXredczxfIHqH6Nfijx07ZgPRBHQg5/7OIGIfiqFesytjbd4AvGCtfSW2\nvvMYDwM7rLWNsXXxx45//R1gqTFmzBDbM1BDvW5Ttd3dxOaWXgD8vTHmNaJDpwFr7bVEe22fNcas\njq1/3kbLPrmIDTkbYzYCtxOtCBA/mtWzzSPKUbYbuv+ubiBaxeCGnttGsN7ea66m7zYf6ftDRJPX\nX/W2XURERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERE+vD/AfCT\nOzCHyQLEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd7f2ff70d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is observed that the savings of the RF versus the RS-RF did not differ significantly, however in terms on F1Score, as expected, there is an increase in the results.\n",
    "\n",
    "To be fair, the Randomized Search algorithm is not finding the hyperparameters taking into account the savings. Therefore, those parameters could still be tunned to attempt to maximize the savings score.\n",
    "\n",
    "### Parameter search focusing on savings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 50 candidates, totalling 150 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   1 jobs       | elapsed:    4.4s\n",
      "[Parallel(n_jobs=4)]: Done  50 jobs       | elapsed: 14.5min\n",
      "[Parallel(n_jobs=4)]: Done 150 out of 150 | elapsed: 118.9min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model with rank: 1\n",
      "Mean validation score: 0.247 (std: 0.013)\n",
      "Parameters: {'bootstrap': False, 'min_samples_leaf': 4, 'n_estimators': 100, 'min_samples_split': 69, 'criterion': 'gini', 'max_features': 7, 'max_depth': None}\n",
      "\n",
      "Model with rank: 2\n",
      "Mean validation score: 0.198 (std: 0.001)\n",
      "Parameters: {'bootstrap': False, 'min_samples_leaf': 23, 'n_estimators': 20, 'min_samples_split': 73, 'criterion': 'entropy', 'max_features': 9, 'max_depth': None}\n",
      "\n",
      "Model with rank: 3\n",
      "Mean validation score: 0.197 (std: 0.011)\n",
      "Parameters: {'bootstrap': False, 'min_samples_leaf': 28, 'n_estimators': 50, 'min_samples_split': 71, 'criterion': 'entropy', 'max_features': 8, 'max_depth': None}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def _savings(clf, X, y):\n",
    "    # Temporal function that receives X=[X_train, cost_mat_train], y= Y_train\n",
    "    # reconstruct cost mat\n",
    "    x = X[:, :-4]\n",
    "    cost_mat = X[:, -4:]\n",
    "    clf.fit(x, y)\n",
    "    c = clf.predict(x)    \n",
    "    return savings_score(y, c, cost_mat)\n",
    "\n",
    "f_ = RandomizedSearchCV(clf, param_distributions=param_dist, \n",
    "                        n_iter=50, n_jobs=4, scoring=_savings, verbose=1)\n",
    "f_.fit(np.hstack((X_train, cost_mat_train)), y_train)\n",
    "\n",
    "report(f_.grid_scores_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                pre        f1       acc       rec       sav\n",
      "RF         0.492091  0.232172  0.934394  0.151926  0.122358\n",
      "LR         0.529412  0.037677  0.934854  0.019533  0.010948\n",
      "DT         0.251239  0.262626  0.899146  0.275095  0.191770\n",
      "RF-BMR     0.179759  0.282039  0.782493  0.654368  0.432752\n",
      "LR-BMR     0.115987  0.188429  0.717737  0.501899  0.228092\n",
      "DT-BMR     0.251639  0.260847  0.899819  0.270754  0.192115\n",
      "CSDT       0.160237  0.267374  0.711325  0.806837  0.465266\n",
      "CSRP       0.164478  0.272971  0.721102  0.801953  0.469644\n",
      "RS-RF      0.611842  0.242714  0.938326  0.151384  0.129198\n",
      "RS-RF-Sav  0.566102  0.274558  0.937476  0.181226  0.153802\n"
     ]
    }
   ],
   "source": [
    "# Fit a RF with the best params\n",
    "classifiers[\"RS-RF-Sav\"] = {\"f\": RandomForestClassifier(**f_.best_params_)}\n",
    "# Fit\n",
    "classifiers[\"RS-RF-Sav\"][\"f\"].fit(X_train, y_train)\n",
    "# Predict\n",
    "classifiers[\"RS-RF-Sav\"][\"c\"] = classifiers[\"RS-RF-Sav\"][\"f\"].predict(X_test)\n",
    "# Evaluate\n",
    "results.loc[\"RS-RF-Sav\"] = 0\n",
    "results.loc[\"RS-RF-Sav\", measures.keys()] = \\\n",
    "[measures[measure](y_test, classifiers[\"RS-RF-Sav\"][\"c\"]) for measure in measures.keys()]\n",
    "results[\"sav\"].loc[\"RS-RF-Sav\"] = savings_score(y_test, classifiers[\"RS-RF-Sav\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAp4AAAE7CAYAAAB0T4FTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8XNWZ//HPjEbSqDfLtlxwP9gQcKUZgzHY9A4J6Y0k\nm4QkTn5ZZ9MTks0mG29CINnshiykbLIpFBvTjDEGAsY0N8BgDtjIRW6yNVYb1Zn5/XFHQpaL2sy9\nM9L3/Xr55Zl779z7HOnO6Jlzz3kuiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiI\npDVfTxsYY24DzgJiwGJr7ctd1lUCO4FIfNGHrLV7khCniIiIiKS5wIlWGmPmA5OttXONMVOBu4G5\nXTaJAZdaa8NJjFFEREREBgF/D+svBJYBWGu3AiXGmPxu2/TYayoiIiIi0lPiORI42OV5NVDRbZv/\nNsY8Y4z5cUIjExEREZFBpafEszsfzuX1Dt8BvgJcALzHGHNDguISERERkUHmhGM8gT04vZ4dRgF7\nO55Ya//U8dgY8whwGnDfiXbY1tYeCwQy+h6piIiIiMt8Pp+GFCZQT4nnKuBW4E5jzCygylrbCGCM\nKQJW4EwuagLOB+7t6YChkHvzkMrLC6iurnfteG5T+9Kb2pe+BnPbQO1Ld2qfpLITJp7W2nXGmPXG\nmLU4JZNuMcZ8DKi11i43xtwHPGeMaQA2WmtP2NspIiIiIkNXTz2eWGu/0W3Rq13W3QHckeigRERE\nRGTw6evkIhERERGRflHiKSIiIiKuUOIpIiIiIq5Q4ikiIiIirlDiKSIiIiKuUOIpIiIiIq5Q4iki\nIiIirlDiKSIiIiKuUOIpIiIiIq5Q4ikiIiIirlDiKSIiIiKuUOIpIiIiIq4IeB2AiIhIb0QiESor\ntw94P6FQPjU1DQPax/jxE8nIyBhwLCJDjRJPERFJC5WV21m8dAW5RcM9jSNce4Dbl1zNpElTPI1D\nJB0p8RQRGUQS0SuYyj2CuUXDyS8ZnfD9iog7lHiKiAwiqdArqB5BETkeJZ4iIoOMegVFJFVpVruI\niIiIuEKJp4iIiIi4QomniIiIiLhCiaeIiIiIuEKJp4iIiIi4QomniIiIiLhCiaeIiIiIuEKJp4iI\niIi4QomniIiIiLhCiaeIiIiIuEKJp4iIiIi4QomniIiIiLhCiaeIiIiIuEKJp4iIiIi4QomniIiI\niLgi4HUAIukmEolQWbl9wPsJhfKpqWkY0D7Gj59IRkbGgGMRERFxgxJPkT6qrNzO4qUryC0a7mkc\n4doD3L7kaiZNmuJpHCIiIr2lxFOkH3KLhpNfMtrrMERERNKKxniKiIiIiCuUeIqIiIiIK5R4ioiI\niIgrlHiKiIiIiCuUeIqIiIiIK5R4ioiIiIgrlHiKiIiIiCuUeIqIiIiIK5R4ioiIiIgrlHiKiIiI\niCuUeIqIiIiIK5R4ioiIiIgrAj1tYIy5DTgLiAGLrbUvH2ObHwNnW2sXJD5EERERERkMTtjjaYyZ\nD0y21s4FbgbuOMY2pwDn4SSmIiIiIiLH1NOl9guBZQDW2q1AiTEmv9s2S4FvAr7EhyciIiIig0VP\niedI4GCX59VARccTY8zHgTXAjoRHJiIiIiKDSo9jPLvxEb+kbowpBT4MXAKMTXBcIiIiIjLI9JR4\n7sHp9ewwCtgbf7wgvu5ZIBuYZIz5mbX2qyfaYUlJLoFARj/D7bvy8gLXjuUFtc99oVD30SbeKS3N\nT8mfUYdUjm2gUrVtqXJ+JuPcTJW2gd57Xhvs7RvMeko8VwG3AncaY2YBVdbaRgBr7X3AfQDGmHHA\n73tKOgFCofDAIu6D8vICqqvrXTue29Q+b9TUNHgdQqeamoaU/BlB6v7+EiGV25Yq52cyzs1UaRvo\nveelwd6+we6EYzytteuA9caYtcAvgFuMMR8zxlzbbdPOS/AiIiIiIsfS4xhPa+03ui169RjbVOLM\ngBcREREROSbduUhEREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHEU0RERERcocRTRERERFyh\nxFNEREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHE\nU0RERERcocRTRERERFyhxFNEREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHEU0RERERcocRT\nRERERFyhxFNEREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHEU0RERERcocRTRERERFyhxFNE\nREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHEU0RE\nRERcocRTRERERFyhxFNEREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHEU0RERERcocRTRERE\nRFyhxFNEREREXKHEU0RERERcocRTRERERFyhxFNEREREXKHEU0RERERcEehpA2PMbcBZQAxYbK19\nucu6TwOfBCLAZmvtLckKVERERETS2wl7PI0x84HJ1tq5wM3AHV3W5QI3AfOstfOAqcaYc5IZrIiI\niIikr54utV8ILAOw1m4FSowx+fHnYWvtQmttJJ6EFgF7kxqtiIiIiKStnhLPkcDBLs+rgYquGxhj\nvg68DfzNWluZ0OhEREREZNDocYxnNz6csZ6drLU/Mcb8AnjEGLPWWvvciXZQUpJLIJDRx8P2X3l5\ngWvH8oLa575QKN/rEDqVluan5M+oQyrHNlCp2rZUOT+TcW6mSttA7z2vDfb2DWY9JZ57cHo9O4wi\nfjndGFMKnG6tfcpa22yMeRQ4Fzhh4hkKhQcQbt+UlxdQXV3v2vHcpvZ5o6amwesQOtXUNKTkzwhS\n9/eXCKnctlQ5P5NxbqZK20DvPS8N9vYNdj1dal8F3AhgjJkFVFlrG+PrMoG7jDF58ednAluTEqWI\niIiIpL0T9nhaa9cZY9YbY9bilEy6xRjzMaDWWrvcGPMD4EljTDuwyVr7oAsxi4iIiEga6nGMp7X2\nG90Wvdpl3R+APyQ6KBEREREZfHTnIhERERFxhRJPEREREXFFX8spicgA3VT1OOObnHstVOZU8LfR\nizyOSERExB3q8RRx0U1VjzOhaS8+nKK4E5r28vl37mVE8yGvQxMREUk6JZ4iLuro6eyqMBLmhr1P\nehCNiIiIu3SpXSRJsqKtjGgJMaKlhhEthxjRUuN1SCIiIp5S4imSADmR5niCWcPIeJJZ2nbknTVa\nfQFa/JkEo21HLA/7s7ivYoGb4YqIiHhCiadIH8RiMWL1dZzcWs34moOdvZlF7UfeCrbZn0Vlzkj2\nZ5eyP7uUfdllhDILiPn8fP6deymMONvHgGC0jWFttewPlnnQIpGhQxP70tvuny0lvPV1LJA79RTG\nfHWJ1yFJPyjxFDmOWCxGW3U1LTt30LJzB807KmnZuZNofR0f6bJdQ0aQt3NHdyaZ+7PLqA3kgc93\nzP3eV7Ggc0znP0pnsPDQy1y1/1myI61sKJ7qQstEhp6OiX0dOib23VexQF/60sDuny0l/MaWzufh\nN7awfclXGPWFxQTHjfcuMOkzJZ4iQCwapXXfXlp27KA5nmi27NxBtKnpiO0CZWVgTmb1rjZqisez\nP7uUxkBun461P1jGryfceMTzm/Y8zsUHXyQ72sq6ktOOm7SKSO9kRCNkR1vJiraRHW074cS+ru9H\nSS3RlhYaX3vliKSzQ3soxJ5f3c7Epbd5EJn0lxJPGXKibW207qlyejF37qBlxw5adu8i1tr67kY+\nH5kjRpB32nSyTzqJ4LjxZI89iYz8fLZte4un7nye/LzRCYmnOruEP4++lPfveZz5NZsIRlt5smy2\nkk8ZkmLt7USbmog0NRFtbiLa1OXfrh2c31RJQWQ/WTEnocyOtJIdbXOedzyOthEg2qvjZcbayWtv\nojGQk+SWSW9FGhpo2LyJho3rCW95jVhbW88vkrShxFMGtWhLCy27d9GyozLek7mTlqrdEIm8u1FG\nBtmjRpE9dhzZ48YRPGk82WPH4g8GXYszlFXIn8Zcyk1Vqznr8OtkR1t5rPxsYj5VPBP3DGQMpD8W\nJSvqJH757fXEdu6koaHx6OSxOZ5UNjURbW4mGg47/zc1EW0KE2tvP+FxLgYIH728xZdJiz+TcEaQ\nUGYBLf5MWv1ZtGRk0uLLZErjLkraG456XU60lS9W3kNV9jDezhvD23ljqc4q1hc/l7UfDtGwcQMN\nG9YTfnMrRJ0vDlmjRpE/czbh17fQ/M72I14TKClh1BcWexGu54wx44FXgZe7LN4I/AJYBqyx1i6J\nb5sJ/Ap4D9Ae//dxa+0uN2PuoMRTBo1IuNFJLHfuoHmHc6m8dd9eiMU6t/EFAgRPGkd2/F9w3Diy\nRo/Gn5nlYeSO+kAefx5zCTftWc2MurfJjrbx4Ih5RH0ZXocmg1xGNMJNex7npOYDncsmNO3ly9v/\nyisFE2nzZzq9i/HexOxo61GPM2ORI/YZ/fOL7OnFsX1ZWfhzcvHn5ZI5bBj+YA7+nKCzLCcYf+78\nq66t5fdr3sFfOJIWfyYt/ixa/Zm0+gM9fklbU37GERP76jJy+b/RFzMlvJvJjbsZ27Sf0S0HmV+z\nidpAHm/njeGtvLHszBmh92CStO7fT8OG9TRsXE/z9m2dy7PHT6Bg1mzyZ80ma2SFs/C6G9i+5Cu0\nh0KAk3Sm0yX2Z6++/nHgovjTJ+atuD8RM9u2WmuPKIlijHkMeAzI7LL4g0C7tfbc+DYfAz4PfKOv\nBzTG+Ky1sZ63PD4lnpJyejNzsb22lpZd7yaYLTt30FZdfcQ2/mCQnMlT3u3FHDeOrJEV+DJS949I\nU0aQv4y6mBv3rmFaww6yom0sG3kB7X69VeX4YrEYsZZmIo1hYvv3M6EtRHFDhGC0lWCk1fk/2kJ2\n5+N3l2dHW49KGjsEo62cWbv1mOvafBnxXsVM6gN5nY9b/Jk0tLUyd9Y4yipG48/N6UwmM3Jyj0ws\ng8E+vR8PbXuLt56tIz9neL9+Tl0n9t1XsYDDWYW8lHUKLxWfQnakhYnhPUxp3MXEcBWza99kdu2b\ntPgy2Z43irdzx7AtbzTNGe5dCRlsYrEYLbt2dvZstlbtdlb4fORMnUb+zFnkz5xFZumxJ3uN+sJi\n9vzqdvx+HyM//yUXIx+YeNK5sMuihc9eff1u4Op5K+7fkODDXQ/ciNO72aEIKOh4Yq39Q8djY8xH\ngC8CUeDn1tq/G2PeB3wFp2d0vbX2y8aY7wMTgAnGmAuAHwLzgAzgV9bav/Y2QP01k5RyrJmL2/7f\nYkouvZRoU3PnJfPI4cNHvM6fn0/uKac6vZgnOZfMM8uH4/On36Xqlows/jZqIdfte5pJ4Spu2rOa\neysupCXD+17ZoS6Z5Vxi0SjRcJhIOEw03EiksZFoU5hI43Geh8Px7RuJhsOdlyYBbgaoO/6xovho\n9mfRkpFFfSCXZn8W4+O3cu0u7M9mWcUF8R7GzM4E80S9gA2hKubNP5vSSVP6/fNIhu4T+7pqycjm\njYIJvFEwAX8sytim/Uxu3M3kxl1Ma9jBtIYdRPFRFSxni7+Q2KEpkGLtS0WxaJTmbdto2Liehg3r\naTvodBD4AgHyTp9O/qw55E+fQUZBQQ97guC48Uxcehvl5QVUV9f3uL1bnr36+qXAe0+wybhjLBsN\nvBhPQI/lnnkr7u/zB4y1ttEY033xn4CPG2O2Ao8A91lr1xpjCoDvAKcBQeAPxpiHgR8B0621YWPM\niniiGQMyrbXnG2POA06y1s43xmQD640xy6y1Lb2JUYmnpJTw1tePWhapq+Xg3//W+TxQUkLe9BlH\nJJmBklJ8g2hMVrs/wH0VF3Dl/rWc0lDJB/as4u8VFxHWBAjP9KacS7StLZ4khuNJYiPReGIYCYeJ\nNnZPGN993r2CQk98gQD+vDwCBYX4R4wkIzcXf24eDe1trHn9INHcYTRnZNHiz6LZn0Vzxrv/t/oy\njxrD2L3cEDiXo4diuaGoz8+O3Ap25FbwxLA5lLXVMqVxF5MbdzOm+QBjOUD0zrd554Fl5E+fSd70\nGeRMnpLSV1PcFGtvJ/zmVho2vEzDpo1EamsB8GUHKTjzLPJnzibvtNPwB/V5NkAnG2O63m95lbX2\nx3Dkd0hrbQ0wyxgzD2eY9F+MMXcDD+Ncrm8BWoBrjTGzgLestR0jqZ8CZsYfvxj/fy5wdpdj+4AK\noLI3QSvxlLTgD+ZQ8dnPkT12HIGiIq/DcUXUl8GDI+bR4s9kZt1bfKjqMf46ahH1mXlehzYkHetL\nUXsoxM4f/YCMgkJnYkzXygi94A8G8efmEigbFk8cc8nIzcOfl9fteS4ZOfH/48+PNy5527a3eKLy\nefJL+lZ14W+jFx01BlJlhgCfj0NZxRzKKub5ktPIbW9idPVr3DAmSntlJaFVKwmtWok/N4+8004n\nf8ZMck99Dxm5fSuzlu6cskev0rBxPY2bN3V+kcrIL6Bw3vnkz5pF7rRTUmI8faLFeyaP2zt5jEvt\nAFUM/FL7m93HeMYdMQbTGJOFM8bzWeBZY8z/4CSUK4DulwWjHJm4ZgEd34o7ygu0AHdZa3/Sn6CV\neEpKyZ16ylH12jpmLg7FIsExn5/Hys+mxZ/F2Ye38OGqlfx11CJCWYVehyYdYjH8WVkEiovfTRJz\n8+JJY66TRObkdkkm300qU62HrPsYSDlaOJDDxuAo3nf92Uw4aRxNb26lYdMmGjdvov6FddS/sA4y\nMsg1U8mbPoP86TPILC/3OuykiDQ20vjKJuo3xMsexb94BUpLKZw7j/xZs9UTDMxbcf+i+CX1jm+D\nVfNW3D8miYfsfvnvLuBZ4Dfx52OBbcBWnF7TPCCCk4heC0wxxuRbaxuA+cC/cmTi/ALwH8aYfwey\ngZ9aa3s96FaJp6SU0Yu/wluf+3TnTPR0m7mYFD4fTw2bTXNGFhcc2siHq1byt1ELObowjCRLe30d\n/rw8og1H/tQDxSWM+uLg+VJ0ojGQcjR/ZhZ57zmdvPecTuxDH6Fl104aN2+iYdNGwm9sIfzGFqr/\n+meyRo8hf/oM8qbPIDhhYlqOPe/QfviwMzloY7zsUbw0XdbICvLjM9Gzx40fVEOfEuRqnMSu43Ei\ndO/ZHIszhrMYCBpjLgU+gDNR6DfxiUTNQCvwufgYzu8Cq+O7+Hl82RJgpTEmCjwTHw+6sON41tp1\n8cvs63CS3P/sS9BKPCWl1L/8IsRi+LKzyczPS6uZi8n2fMlptPizuLj6BT5Y9Rh/zD/d65CGhMbX\nt7Dvrt8SbWjAFwh01pnUlyLpyufzEYyPOy+76hraQiEaX3F6QsOvb6HmkYeoeeQhMgoKyZs+nfzp\nM8g95T34s7O9Dr1HrQcOdE4Oat6+rbNjIHv8hPhM9NlkjxrlcZSpLX5JPWG9nNbaSuDMbst24UwU\nOpYbjrOfvwB/6bZsGU4t0K7Lbu32/NvAt/sUdJwST0kZsViM0KrHwOdj/K3/yqhpE1Nq5mIq2Fh0\nMi3+TK7cv5aP120ktt1oZm2SxNrbObj8fkKPPQp+P8NueB85U6ex99d3pF05F3FfZkkJxfMXUDx/\nAdGWFsKvb6Fh80YaN2+m7tlnqHv2GXyBALnTTiFvxkzyTp9BZkmJ12EDzmdx6+7dNGxcT/2G9bTu\njtcZ9/nImWKcmegzZ5FZNrQmnUliKPGUlNFk36Rl5w7yZ88hc9jgHBOVCK8XTKTVn8k1e58mes/f\nqC8ppWDOGV6HNai0HjjA3jv/i5bKd8gsH07FZz5LcMJEgJQs5yKpzZ+d3VmjMhaN0lz5Do2bNtKw\neRONr75C46uvAH8ge9x455L8jJlkjz3J1cvVsWiU5ne2OzPRN2ygrdq5mUBn2aOZs8ibMZNAgcaX\ny8Ao8ZSUEXr8MQBKLr7U40hS39t5Y/lj4Qxubn6Nvb/5NdHmT1A073yvwxoU6tatZf+f/pdYSzMF\n58xlxIc+orIvkjA+v5+ciZPImTiJYdffSNvBaicB3bSJsN1Ky45KDq1YTqCk1En4ZswkZ+rUpMwG\n7yx7tHEDDRs3EKl16iP7soPkzzmTglmzyT3tdDJydP5L4ijxlJTQum8fjZs3EZw4iZxJk70OJy28\nk1mC/7oPwb1/Z//v7yYabqLk4ku8DittRZqaOPDnP1L//Dp82UFG3vwZCs+Z63VYMshlDiun5KJF\nlFy0iEhTE+Etr9KwaSONr75C7dNPUvv0k/iys8k75T3kTZ9B3unTCRT2v9fRuez/GvUb4mWPwk75\nLH9+PoXzziN/5mxyTxmcZY8kNSjxlJQQemIVxGJKnPrIN2oUY772TXb/fCnVf/8LkaYwZVdfqxml\nfdS0fTv7fvtftFVXE5wwkZGf/ixZw/t3W0aR/srIyaFgzpkUzDmTWCRC07a3ady8kYZNm5zJPRvX\ng89HcOKk+Cz5mWSNGtX5fj/enbUi4UYaN292amy+9uq7ZY9KSik8e65T9miKGfJlj8QdSjzFc5GG\nBurWPkugrIz8mbO9DiftZI8ezdivf5Oqny+l5sEHiDY2Uv7+D6Z1yRa3xKJRQo89ysHl90M0Ssll\nVzDsmuvwBfTRKN7yZWSQa04m15xM+XvfT+u+vc4l+c2baHrL0rztbQ7efy+Z5eXkTZ9Bk7W07NzR\n+frwG1t4+0ufJ2tkBc07KjvLHmWOHElBfHJQ9vgJ+pIqrtOnq3iu9h9PEWttpeSiRfrG3U9Z5cMZ\n+y/fYvfPl3J4zWqiTU2M+Pgn9fM8gfbDIfbd9VvCb7xORlExFZ/6DLnTTvE6LJFjyhpZQenICkov\nuYxIQwONr75Cw+aNhF97lcOrHz/ma6LhMM3bt5F90rh4jc05KnuUHBnApATvcxtOUffjMsbcAnwY\n505COcA3rbVP9GbnxpjpwHXW2u8PMM4+U+Ipnoq1txNasxp/MEihJscMSKC4mLFf+wZVt/+cunVr\niTQ3UfGZz2qs1jE0bN7E/t/dRaShnrzTpzPiEzdrtq6kjYz8fArPmUvhOXOdCUL2Tap+vvTY2xYW\nMe67tx5znSTMpDOv/96buUWJGZ4Trj3Ai/ffejJgj7eNMWY88ClgjrU2YowxwJ1ArxJPa+1mYHMC\nwu0zJZ7iqfqXXiRy+DDFiy4Zcvc2ToaM/HzGfHUJVb+6g8aNG9hzxy8YdcuX8AeDXoeWEqJtrRy8\n5+8cXrMaXyBA+Qc/TPGCi3S5UdKWLxAg75RTyZ126nFvNyzJl1s0nPyS0T1vmDhFQBDnlpVha60F\nLojfYeiHOHcoOgy8D/g7zl2JnjHG5ACvAzfj3L3ovcaYt3EKxp8bf80VOLf3vAenN/UfwHnW2gXG\nmDuA2Ti9vP9lrf1DXwPXIDDxTCwWc0oo+XyUXLSw5xdIr/iDOYxe/BXyZswk/Mbr7P75UiINusFm\ny5497PzRDzm8ZjVZFaM46Vvfo+TChUo6ZVAY89UlBLoUoO+4s9ZguZ2rHCneY/ki8I4x5nfGmPca\nYwI4CemHrLULgFrgEuA+4Kr4SxcBj3HkZfwJwB+stXOBEuB0nNts/tVaewFOchszxpQAl1trzwXm\nAZn9iX1QJp67f7YU++lPsPbaG9n9s2NffhDvqWB88vgzsxj12VsoOPscmrdvY9fSn9Aer9E31MRi\nMQ7/4yl2/uv3ad29i6LzL+Ckb3+P7LFjvQ5NJKFGfWExgZISsspK1dM5BFhrPwbMBzYBXwNWAYeA\nO40xTwEXAqXAg0BHgexrgHu77arOWvta/PFunOR1KrA2vmwF4LPWhgBrjFkO3AT8sT9xD7rEc/fP\nljqXG2IxiMUIv7GF7Uu+4szqk5TSWTB+kUooJYMvEGDkJz9N0YKLaK3aza6f/BttB6u9DstVkcZG\n9v73f3Lgj7/HFwhQ8blbGPHRj6fF/bFF+io4bjwTl97GGXf/Vj2dQ4AxJttau9VaeztwFjAWuBu4\nJd5T+QCAtbYWqIqPAz0HWAN0vdTT3m3Xvvi/WPdjWmsvB24FZuAktH026BLP8NbXj1rWHgqx51e3\nexCNHI8KxrvD5/cz/IMfpvTKq2irPsCuf/83Wvbs8TosVzS9Zdlx63doWP8yOVMM4773Qwpm69ai\nIpL+jDGfAu42xnQkkMU4yWIBsMsYU4zT49nxLXsZ8G3gOWtttBeH2AbMiT++LH7MccaYL1lrN1pr\nlwBl/Yldk4vEEyoY7x6fz8ewa2/An5PLwXv+xq6f/htjvvxVguMneB1aUsQiEQ49tIKah1YAUHbN\ndZRefqVKS4lI0oRrD7i9r7uBk4HnjTENOOMtvwicjXOJfBvwU+B7xpgVOL2fv8S51A5Ob2asy+Ou\nYsDtwN+NMTfgjCWNAHuAc4wxN+FMOrqrP+0bdIln7tRTjprZl1FUpPEuKUQF471ResllZOTksv9/\nf8/u//h3Rn3xy+SePNXrsBKq7dAh9v3Pb2h6yxIoLaPi0/9EzhTjdVgiMrhti5c/Sug+T7Qy3mu5\n5BirHgW+1+V511nnnWOMrLVPA0/HHw/vsvy9AMaYU4AvWGufM8Z8ABhmrW0DPtDHdhxl0CWeY766\nhO1LvkJ7KNS5LH/mbI13SSHvFoy/WL1QLis6fz7+3Bz2/vY3VP3iZ1R87hbyT59xxDaRSITKyu0D\nPlYolE9NTf9n048fP5GMPpwf9S+/xP4//o5oOEz+7DmM+OgnyMjL6/fxRUR6KcIJam6mqXrgN8aY\nGE77PpGoHQ+6xBOcmX17fnU7fp+PaAxqn3ma0ksv08zpFHBEwfjzVDDeCwVzzsQfDLLn179iz3/+\nkpGf/DSFZ53dub6ycjuLl64gUcWQ+yNce4Dbl1zNpElTetw22tJC9d/+j9p/PI0vK4sRH/0Eheed\nrzJJIiL9ZK3dBZyXjH0PysSzY2ZfeXkB21Y8xr677uTQQysY+fGbvQ5tyDuiYHxOjtfhDFl57zmd\nMV/5Z6ruuI19//Mbos1NFM9f0Lneg2LI/dKyayd7f/NftO7bS/bYsVR85nNkVeiWgCIiqWrQzWrv\nruCss8ntbLAmAAAeyklEQVSqGEXdc2tp3b/P63CGNBWMTy05UwxjlnydjPx8DvzvH6h59GGvQ+q1\nWCxGaPXj7PzRD2jdt5fihYsY+83vKOkUEUlxgz7x9Pn9lF1zHUSjHFrxgNfhDGlNb25VwfgUEzxp\nHGP/5ZsESks5eN89VN93D7HYUaXbUkp7fR17fvkLqv/6Z/zBHEZ96csMf/+HdE96EZE0MOgTT4D8\nWbPJHjuW+hefp6WqyutwhqzQ6lWACsanmqyRFYz9l2+ROWIEoUcfJvbYSnwpmnw2vr6FHd//Lo2v\nbCZ32qmM+/4Pj5ocJSIiqWtIJJ5Or+f1EItxaMUyr8MZklQwPrVllpUx9mvfJHvsWGIbN3Bjwxb8\nsd7UGHZHrL2d6nv/TtVt/0GkoZ5hN76P0V/5KoHiYq9DExGRPhgSiSdA3vQZBCdMpGH9yzTv3OF1\nOEOOCsanvkBREWOWfB1Gj2F6636u3/sUgWj3O6m5r/XAAXb+5EeEVj5C5rByTvr6tyi99HJ8/iHz\n8SUiMmgMmU9un89H2bXXA3Bo+f0eRzO0qGB8+sjIzcP//g/wVmYpk8O7ed+eJ8iKtnoWT92659hx\n63dpqXyHwnPOZdz3biU4YaJn8YiIyMAMmcQTIPeUU8mZYmh8ZTNN2094UwBJIBWMTy++rCz+VDCd\nrXkncVLzfj5Q9Tg5kWZXY8iOthNd8QD77roTnw9GfuozjLz50/iDKsElIpLOhlTiqV5P96lgfHqK\n+Pw8MPJ8XimYREXLIT60+zHy28OuHHtk80E+X/sisS2vEZwwkZO++wMKz57ryrFFRCS5hlTiCZB7\n8lRyp51K+PUthO2bXocz6HUUjC88b74KxqeZmM/PI8Pn8mLRNIa11fLh3SspbqtP4gFjnBV6jY/s\nfpSSaBO+c+Yy9l++SdZw7+6gJCIiidVj4mmMuc0Y85wxZq0xZk63dQuMMeuMMc8aY+4yxqTFPerK\nrr0OgEPL7kv5moXpTAXjBwGfjzXD5vCP0hkUtzfw4d0rKW8JJfwwee1hbtqzmgWHNhDOCPL7wpn4\nL1iALzAob64mIjJknTDxNMbMByZba+cCNwN3dNvkTuBGa+08oAC4NClRJljOpMnknT6dprcs4de3\neB3OoKWC8YOEz8dzpafz+LAzyI808cGqx6hork7Y7ic17ubmnQ8yoWkvb+eO4e6TrmJ7ZmnC9i8i\nIqmjpx7PC4FlANbarUCJMSa/y/rZ1tqOiuzVQNr8tSi7Jt7rufx+9XomSejxxwAVjB8s1hdP46Hh\n55IdbeMDVY8zLrx3QPvLiEZYWP0i7927hqxYG6uGncm9FQtoyggmKGIREUk1PSWeI4GDXZ5XAxUd\nT6y1dQDGmArgYuCRRAeYLMFx48mfNZvmd7bT+Mpmr8MZdFr37aPxlc0EJ01WwfhB5LXCSSwfOR9/\nLMp79zzBlIad/dpPWethPrr7EebUbuVgZhF/GHMFG4qngi8tRuuIiEg/9XUAlQ84onvQGDMcWAF8\nzlrb4+CvkpJcAgH3SuqUlxccd13uxz/Mpo0bqH3oAcZfeG5aFqQ+Ufu8tO2+v0IsxrgbrmHYAGJM\nxfaFQvk9b+SS0tL8hP+MemqfzT+Je0ZdyA17n+K6fU/z8PC5bCmc1Ludx2JMr3uLhQdfIjMWYWPh\nFJ4Ydgbt/qM/ipLRtkRK1dhS5fz04tx0k85Pbw329g1mPSWee3B6PTuMAjqvrxljCnF6Ob9prV3d\nmwOGQu6UZAHnxKyuPsEs3NwSCs48i/oXnuedVU9RMPsM12JLhB7b55FIQwP7n1hDoKyM6KRT+h1j\nqravpqbB6xA61dQ0JPxn1Jv27cgdxV9HL+J9e57gqgNrCUZbWV887YSvyY60cNmB55nauINmfxYP\njpiHzR93wjhS8fcPqXtuQuqcn16dm27R+emdwd6+wa6nLr5VwI0AxphZQJW1trHL+p8Bt1lrVyUp\nvqQru+pa8Pk49MAyYtHUuTd1OlPB+KFhT7CcP4++hIaMIIsOvsTcmlfgOOOlxzTt55O7HmRq4w52\nBYdz99grT5h0iojI4HTCHk9r7TpjzHpjzFogAtxijPkYUAs8BnwEmGyM+VT8Jf9nrf1tUiNOsKyR\nIymcO4+6tc9Q/+LzKlQ9QLH2dkJPqGD8UFGdXcKfRl/KB/Y8zvk1mwhGW1lTNrtzrKYvFmVu6FXO\nrXkFgGdKp/NcyWnEfOk3rEVERAauxzGe1tpvdFv0apfHg2L6adlVV1P3/HMcWvEABWecpV66Aah/\n6UUitYfxnXkWlXt2D2hfoVD+gC+tjR8/kQz9PpPqcFYhfxpzKTdVrebMw68zrf4d8iNNALT4MwlG\n26gN5PHgiHnszhnhcbQQiUSorNw+oH3o3BQR6R9VZwYyh5VTNO98ap9+krp1aymap566/uhaMP4H\n6w/Tsu15T+MJ1x7g9iVXM2nSFE/jGArqA3n8ecwlfLZyGQXxpBMgGG2jzZfBQ8PnpkTSCVBZuZ3F\nS1eQW+TdHZF0borIUKXEM670iquoW/sMhx58gMKz5+qOKf3QUTDeN3UaLdWjyC8Z7XVI4qKmjCBZ\nsbajlmfGIly1fy2/nnCjB1EdW27RcJ2fIiIe0ECruMzSUoouWED7oUPUPvMPr8NJSx0F431nnuVx\nJCIiIpKKlHh2UXrZFfiysjj08Aqira1eh5NWWvfto3HzJoKTJuMbrZ6koaoyp+KoZXUZudxXscCD\naEREJNUo8ewiUFRM8YULiRw+TO3TT3odTloJrXYqaun2mEPb30Yvoi4jt/N5XUYuv55wI/uDZR5G\nJSIiqUKJZzell16OPxik5pGHiTY3ex1OWog0NFD33LMEysrInznL63DEY/dVLKAuI1c9nSIichQl\nnt1k5OdTvOgSIvV1HF7Tq5sxDXkqGC9d7Q+W8esJN6qnU0REjqLE8xhKFl2MPzePmpWPEgm7d4vP\ndKSC8SIiItJbSjyPISM3j5JLLiUabuTw6rS9G6gr6l96gUjtYYrOm09GTo7X4YiIiEgKU+J5HCUX\nLSIjv4DQ448RaRjYHUoGq1gsRmiVUzC+eOEir8MRERGRFKfE8zj8wSAll11OtKmJ0KqVXoeTkpre\n3ErLrp3kz55DZtkwr8MRERGRFKfE8wSKL7iQjKJiQqtX0V5X53U4KaejYLxKKImIiEhvKPE8AX92\nNmVXXEmstZXQow97HU5K6VowPmfSZK/DERERkTSgxLMHhefNJ1BayuGn1tAWCnkdTspQwXgRERHp\nKyWePfBnZlJ25TXE2tqoeeRBr8NJCZ0F44cNU8F4ERER6TUlnr1QOPdcMsuHU/uPp2k7dNDrcDzX\nWTD+wkUqGC8iIiK9psSzF3yBAGVXXwORCIceXOF1OJ5SwXgRERHpLyWevVRw1jlkjayg7rlnad2/\n3+twPKOC8SIiItJfSjx7yef3U3bNdRCNcujB5V6H4wkVjBcREZGBUOLZB/mz55A1Ziz1LzxPy54q\nr8Nx3bsF489QwXgRERHps4DXAaQTn9/PsGuvZ8+vbufQiuWM+uwtXofkqncLxl/scSQiIpJuIpEI\nlZXbB7yfUCifmpr+38p6/PiJZGhirGeUePZR3vQZZI+fQMPLL9G8cwfBk8Z5HZIrVDBeREQGorJy\nO4uXriC3aLhnMYRrD3D7kquZNGmKZzEMdUo8+8jn8zHs2uup+sXPOPTAMkZ/8cteh+QKFYwXEZGB\nyi0aTn7JaK/DEA9pjGc/5J76HnKmGBo3b6Jp+8AvG6Q6FYwXERGRRFDi2Q8+n8+Z4Q4ceuB+j6NJ\nvsNPP+kUjL9IBeNFRESk/5R49lPu1GnkTjuF8JbXCNs3vQ4naWLt7Rxe84RTMH6eCsaLiIhI/ynx\nHICya68H4NDy+4nFYh5HkxwqGC8iIiKJosRzAHImTSbvtNNpsm8SfuN1r8NJOBWMFxERkURS4jlA\nZdcM3l5PFYwXERGRRFLiOUDB8ePJnzmb5u3baHx1s9fhJFRo1UoASi5WCSUREREZOCWeCVB2zbXg\n83Fo+bJB0+vZum8fja9sdgrGT5zkdTgiIiIyCCjxTIDsMWMpOOMsWnbuoGHDeq/DSQgVjBcREZFE\nU+KZIGVXX+P0ej6wjFg06nU4A6KC8SIiIpIMSjwTJGtkBYXnnEvrnirqX3rB63AGRAXjRUREJBl0\nr/YEKrvqGupeWMehFcspmHNmWiZtnQXjc3JUMF5ExEWRSITKyoHfhjkUyqempmFA+xg/fiIZafg3\nTFKfEs8Eyiwvp2jeedQ+/RR1656jaN55XofUZx0F40sWXaKC8SIiLqqs3M7ipSvILRruaRzh2gPc\nvuRqJk2a4mkcMjgp8Uyw0iuupm7tsxx66AEKzz4HXyB9fsQqGC8i4q3couHkl4z2OgyRpNEYzwTL\nLC2laP4C2g8epPbZf3gdTp+oYLyIiIgkU8p2xyVirItX41xKL7+C2meepubhByk8dx7+zKwBxeAW\nFYwXERGRZErZxDMVxrr0d5xLoKiY4gsXElr5CLVPP0XJwouTFGHitO7bq4LxIiIiklQpm3hCeo91\nKb3kMg4/uYaahx+i6Lz5+LOzvQ7phEKPq2C8iIiIJJfGeCZJRkEBJYsuJlJfx+E1q70O54QiDQ3U\nrVurgvEiIiKSVEo8k6jk4kvw5+ZSs/IRIk1NXodzXCoYLyIiIm5Q4plEGbl5lFx8KdHGRg7H732e\naqJtbRxes1oF40VERCTplHgmWcnCRWTkFxBatZJIw8Bm2CdDw0svEqmtpei8+SoYLyIiIkmlxDPJ\n/MEcSi67nGhTU2e5olQRi8UIPb7SKRh/0UKvwxEREZFBrsfE0xhzmzHmOWPMWmPMnG7rgsaYPxpj\nXkpeiOmv+IILySgqIvTE47TX13kdTienYPwuFYwXERERV5ww8TTGzAcmW2vnAjcDd3Tb5KfAi0mK\nbdDwZ2dTesVVxFpaCD36iNfhdFLBeBEREXFTTz2eFwLLAKy1W4ESY0x+l/XfAB5MUmyDStF58wmU\nlnL4ySdoPxzyOhwVjBcRERHX9ZR4jgQOdnleDVR0PLHWNgK+JMQ16PgzMym98mpibW0cevghr8N5\nt2C8ejtFRETEJX2dXOQDYskIZCgomjuPzPJyav/xFG2HDvb8giQ5smD8bM/iEBERkaGlp1tm7sHp\n9ewwCtjbbZs+JaIlJbkEAj0XKQ+F8nvcxg2lpfmUlxckbocffD9v3f5LwqtXMvkLnxvw7voT266n\nVhFrbWXsNVcxfETRgGPoLlV+d5CE3x9qn1uS0TZQ+9wymM9NUPv6I1Xal6z3nvROT4nnKuBW4E5j\nzCygKn55vas+XWoPhcK92q6mJjVqXtbUNFBdXZ+4HZ46k6yRFex/Yg05Fywia8SIfu+qvLygz7FF\n29qoevBh/Dk5+Gecmdi2xaXK7w6S8PtD7XNLMtrWsd9UoPb1b5+pQu3r3z5TQbLee9I7J7zUbq1d\nB6w3xqwFfgHcYoz5mDHmWgBjzGpgJXCqMeZVY8wnkh5xmvP5/ZRdfS1Eoxx66AHXj6+C8SIiIuKV\nnno8sdZ+o9uiV7usU9XxfsifcwZZDz9I/fPrKL3sSrJHjXLluJ0F4/1+FYwXERER1+nORR7w+f0M\nu/Z6iMU4tGK5a8dt2vqGUzB+1hwVjBcRERHXKfH0SN6MmWSPn0DDyy/SsmunK8cMPf4YoBJKIiIi\n4g0lnh7x+XwMu/Y6AA4+sCzpx1PBeBEREfGaEk8P5Z56GsFJk2nctJHmd7Yn9VgqGC8iIiJeU+Lp\nIZ/Px7DrbgDg4PL7k3acSH09devWkjmsXAXjRURExDNKPD2WO3UaOVOnEd7yGk1v2aQc4/DTTxJr\nbaX4ooX4/PqVi4iIiDeUhaSAYddeD8DBZfcRiyX2jqTRtjYOP/kE/pwcCuedn9B9i4iIiPSFEs8U\nkDN5CrnvOZ0m+yZNW99I6L5VMF5ERERShRLPFNHZ67n8/oT1eqpgvIiIiKQSJZ4pIjh+PHkzZ9G8\n7W0aX30lIfvsKBhfMFsF40VERMR7SjxTyLBrrgOfj0MJ6vXsKBhfvEgllERERMR7SjxTSPaYsRTM\nOYOWnTto2LhhQPtSwXgRERFJNUo8U0zZ1dc6vZ4PLCMWjfZ7PyoYLyIiIqlGiWeKyaoYReE5c2mt\n2k39yy/2ax8qGC8iIiKpSIlnCiq96hrIyODQA8uJRSJ9fn1nwfiFi1QwXkRERFKGspIUlFU+nKJz\nz6Nt/z7qnn+uT6/tWjC+aN55SYpQREREpO+UeKao0iuvwhcIUPPgCmLt7b1+XdeC8f6gCsaLiIhI\n6lDimaIyS8soOv8C2g5WU7v2mV69RgXjRUREJJUp8UxhpZdfiS8ri5qHVhBta+1xexWMFxERkVSm\nxDOFBYqLKV5wIe2hELVPP93j9u8WjL802aGJiIiI9JkSzxRXcunl+LKD1DzyINGWluNu17p3T5eC\n8RNdjFBERESkd5R4prhAQSElixYRqavj8JNPHHe70GoVjBcREZHUpsQzDZQsuhR/Tg41Kx8h0tR0\n1HqnYPxzKhgvIiIiKU2JZxrIyMuj5JLLiDY0cDjes9mVCsaLiIhIOlCWkiZKFi7Cn59PaNVKIo2N\nnctVMF5ERETShRLPNOEP5lB66eVEm5oIrVrZubz+xRdUMF5ERETSghLPNFK84CIyCgsJrV5Fe31d\nvGD8YyoYLyIiImlBiWca8WdnU3r5VcRaWgitfITaV1+jdbcKxouIiEh6UOKZZormz8cXCBB6bCVb\nvvN9QAXjRUREJD0o8Uwze+64nVh7+xHL9v7XL2neUelNQCIiIiK9pMQzzYS3vn7UsvZQiD2/ut2D\naERERER6T4mniIiIiLhCiWeayZ16ylHLAiUljPrCYg+iEREREek9JZ5pZsxXlxAoKel8HigpYeLS\n2wiOG+9dUCIiIiK9oMQzDY36wmICJSVklZWqp1NERETSRsDrAKTvguPGM3HpbZSXF1BdXe91OCIi\nIiK9oh5PEREREXGFEk8RERERcYUSTxERERFxhRJPEREREXGFEk8RERERcYUSTxERERFxhRJPERER\nEXGFEk8RERERcYUSTxERERFxhe5c5JFIJEJl5fYB7SMUyqempmFA+xg/fiIZGRkD2oeIiIhIbyjx\n9Ehl5XYWL11BbtFwz2II1x7g9iVXM2nSFM9iEBERkaGjx8TTGHMbcBYQAxZba1/usm4h8CMgAjxi\nrf3XZAU6GOUWDSe/ZLTXYYiIiIi44oRjPI0x84HJ1tq5wM3AHd02uR24HjgXuNgYMy0pUYqIiIhI\n2utpctGFwDIAa+1WoMQYkw9gjJkI1Fhrq6y1MeAR4KJkBisiIiIi6aunxHMkcLDL8+r4so511V3W\nHQAqEheaiIiIiAwmfZ1c5Ovnun4J1x5I9C5T6viDuX1ety3ZMah9yaX3Xmrv38vje922ZMeg9iWX\n18eXHhhjvmeM+UyX59uMMXnxx+OMMc912/bzXsQpIiIiIqmvp0vtq4AbAYwxs4Aqa20jgLV2B1AY\nT0ADwBXx7UVEREREjtLj5XFjzI+B83FKJt0CzAJqrbXLjTHnAf8e3/Rea+3PkxapiIiIiIiIiIiI\niIiIiIiIiIiIiIiIiIiIiCRUwmtvesEYMx54Fei4j3x2/PnngO3ATpzJUQAxa+2Fbsc4EPH23WOt\nPaPLsu8DHwSqcH6POcCPrbXLvYhxILr9/nxAO/BvwCicW7UGgVOB9fGXfNRau8v9SAd+rhljPg78\nEHgbp60twEestQeMMb8Hyq21V3TZ/nLgIWA8ThWKYx7bWhtNYPv6fa4ZYypxfgbtQD5wl7X2N/H9\nbgfOsta+1GX7F4Et1tpPxNs/CzgUP04WsMRauzYRbevSvn6fa6nevuMxxkwBfgEMAzKA54B/BuYC\nPwCiQAHwv9baXxhjLgDuAV7DOe8agG9ZazcZY/4Zp4pJMTAa2BI/zMXW2rZkt+VYBtg+gFxgpbX2\ne/H9tQHPxtflAL+z1v7GndY4evisOZ9jtOsY+6jk3c8kPxAGPmmt3dvtfOzwZWvt5sEQtzHmFuDD\nOJ+xOcA3rbVPDKRtkhh9LSCfyrZaaxd0PDHG/A74EBADLrXWhj2LLDliwC+stb8GMMaUAJuMMY9a\na1u8Da1fOn9/8duxPgjcZK1dYIwZh1M1YcEJ9+CegZxrMeAv1tqvxV/7HeCTwE/i68cbY4qttYfj\nz28EtvVw7A8Cfxpgm06kL+da58/AGJMLbDfG3Blftx14H/BSfD/jgJL4azpe+3Vr7SPx9ROBR4GT\nE9yegZxr6dC+IxhjMoB7gS9Ya5+JL7sD+C7wXmC+tXafMSYIPGGM+Wv8pU9Za98b334mcK8xZp61\n9j+A/zDGzI/v873JjL8nCWqfD3g83r5ngcNdzpEsYEP8fN/pbuuO+1nzbY5u11+stfu7vf6IzyRj\nzMdwvvh+im7n42CKO578fgqYY62NGGMMcCegxDMF9FTHM529AEz2Oogk6+yxttaGgL0MgtuWWmu3\nAz8CvhBflOo9830917q2ZySwO/44BqwGrgMwxmQCU4AT9e6+GN8m2fpyrnVsWwZUW2tjOG17Hrio\ny3Y3coLav/HzoDCeFCRFP8+1tGlf3CLgjY6kLG4JTs9TKU7PE9baZmvtudbafceIdSNwN/DxLotT\n5X2ZiPbFcHrojnofW2tbcXrsJiQh9r7q+Kwp4eh2dU/ejsWtz4vu3I67COfqRXb8WNZaewGAMWah\nMWadMeZJY8wyY0xm/P/z4utzjDHvuPC+HLIGZeIZ/4N9DbAhvmjQn0DGmJOBEbybxKS79cApXgfR\nk36caz7gpviH3qvADOC+Luvvx+mlASeJeep4+4wf+2revSzsil6ca48aY57GieuHXZa3ARuNMWfF\nn18BPMKR7et8bIw5H9gTTwqSqa/nWrq172TgiMuQ1tqWeEL1HeAlY8xyY8znjTHFJ9jPy6Tme3LA\n7TPG5AOX8O77uOu6Upz36asJj7wPunzWrMfpze3arpITvLTr+XcjR35eJP1voxdxxy+7vwi8Y4z5\nnTHmvfGecXCS0g/Fe2RrcX7v9wFXxdcvAh5z4X05ZA2mS+0nG2OejD8+HfiJtfYBY8ztOH8oOsbd\nVVtr3+dNiAnlAxYbY24ECnG+2X3AWtvubVgJU8C7YyVTzUDOtRjw1y6X2j8M/Ab4aHx9JU4vWAlw\nA/ArYF4Px16RwLYdS1/PtY5L0QXAamPMZqA5vu5e4H3GmCoghDN2sOtxfhwfQzgsvu6DiW/OUfp6\nrqVb+2I44x6PYq39b2PMMpw/vtcC3zbOXeqOpZDUfE/2t33z4++lDJzetK9ba1+Jryvq8j6LAv9s\nra1JWguO77jv92O161i9ubz7mTQReAb4p/jyrudjhw9Za/cMhrittR8zxkyNH+trOGNML8QZG3qn\nce64OBFYAzwQ3+ZrOEnyXwbUejmhwZR4vtllTM49wFvx5YN+jKcxZiTOm8fTb+QJNodj9D6kiF6f\na/GxRr+LP/1q/P+u39bvB/612/6X43zDf4+1drMzPKnHYyfTcc8140w8mg+8Yq1d3PVF1tp6Y8xT\nwDlAxx+h1cCPgR0c2dPbcZyvW2sfMcacDvwPYJPRoG6Oe64NkvZt5d2hBEDnuEUDbItf6vwj8Edj\nzN3AxTjxd5eq78n+tu/pLmM813Hk52dtiowpP+b73RiTc6x2GWfccPfzteOL0i3AFBu/7TXJHePp\netzGmOU4vZn/a6292xgTtNZuBbYaY34Z//8knCEjl1lr34wvx1pba4ypio8FPQf4dBJ+JhI3KC+1\n44zv+YlxBv8PVr74P+LfFv8IfM/TiBLEGDMJ+Apwm9ex9MIJzzVrbaW1dkH837H+aJ+F84ezq3tx\nktSnennsnD7G3FfHPdestd+Pt21xt+07JmycQZf2WWfW80acGeQPcvQls47jvIKT5HwuCe3p1NO5\nlu7ti3scGGeMuRLAGOMHfgp8E2fSTEGX5aM4cjIb8XVzgOuBu1yIt68G3D7g/wH/meLj+jre76dz\nnHYd53zt8N/ABfHXd3Cjva7Eba29Nr6Pu40xnwLu6vL7LI6/5gBOz/2u+LCLC4mPAwWW4Ux+es4m\nqEqIHNtgSjw7x2NYaytxehu+1XV5mjvVGPNGxz+cMiFd2/Zz4GpjzDRvwhuwk40z7vE54P+Az1tr\nu44hTKXf40DPtY4xnk/ifNB9qeu+4/usw0lAe3Psb/ch9t4Y6Ln2aLxt64AnrbXPd4v9HmCntbY+\nvqzrvrs+/jawxBgzbIDt6W6g51qqt+8I8bFqlwCfMca8hHPZMmSt/SBO7+zjxpg18eVPWKe8k4/4\npWhjzAvAUpyZ/12vHHVvmyf62T448r20Dqcqwc3d13nsWO/3mzhxu473+ghOEvjrY61PMK/jvhvY\nAzxvjHkC5yrSl6y1zcB/Amtxrjj8FPiGMWYEzuX2mzj2566IiIiIiIiIiIiIiIiIiIiIiIiIiIiI\niIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIyODz/wEA3cD73yUeRwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd7f014c150>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "1. Anderson, R. (2007). The Credit Scoring Toolkit : Theory and Practice for Retail Credit Risk Management and Decision Automation. Oxford University Press.\n",
    "2. Hand, D. J., & Henley, W. E. (1997). Statistical Classification Methods in Consumer Credit Scoring: A Review. Journal of the Royal Statstical Society. Series A (Statistics in Society), 160(3), 523–541.\n",
    "3. Correa Bahnsen, A., & Gonzalez Montoya, A. (2011). Evolutionary Algorithms for Selecting the Architecture of a MLP Neural Network: A Credit Scoring Case. In IEEE 11th International Conference on Data Mining Workshops (pp. 725–732). Ieee. http://doi.org/10.1109/ICDMW.2011.80\n",
    "4. Beling, P., Covaliu, Z., & Oliver, R. M. (2005). Optimal scoring cutoff policies and efficient frontiers. Journal of the Operational Research Society, 56(9), 1016–1029. http://doi.org/10.1057/palgrave.jors.2602021\n",
    "5. Verbraken, T., Bravo, C., Weber, R., & Baesens, B. (2014). Development and application of consumer credit scoring models using profit-based classification measures. European Journal of Operational Research, 238(2), 505–513. http://doi.org/10.1016/j.ejor.2014.04.001\n",
    "6.  Alejo, R., & Garc, V. (2013). Making Accurate Credit Risk Predictions with Cost-Sensitive MLP Neural Networks. In J. Casillas, F. J. Martínez-López, R. Vicari, & F. De la Prieta (Eds.), Advances in Intelligent Systems and Computing (Vol. 220, pp. 1–8). Heidelberg: Springer International Publishing. http://doi.org/10.1007/978-3-319-00569-0\n",
    "7.  Oliver, R., & Thomas, L. (2009). Optimal score cutoffs and pricing in regulatory capital in retail credit portfolios (No. CRR-09-01). Southampton. Retrieved from http://eprints.soton.ac.uk/71321/\n",
    "8.  Correa Bahnsen, A., Aouada, D., & Ottersten, B. (2014). Example-Dependent Cost-Sensitive Logistic Regression for Credit Scoring. In 2014 13th International Conference on Machine Learning and Applications (pp. 263–269). Detroit, USA: IEEE. http://doi.org/10.1109/ICMLA.2014.48\n",
    "9.  Lawrence, D., & Solomon, A. (2012). Managing a Consumer Lending Business. Solomon Lawrence Partners. \n",
    "10. Nayak, G. N., & Turvey, C. G. (1997). Credit Risk Assessment and the Opportunity Costs of Loan Misclassification. Canadian Journal of Agricultural Economics, 45(3), 285–299. http://doi.org/10.1111/j.1744-7976.1997.tb00209.x\n",
    "11. ECB. (2014). European Central Bank. Retrieved from http://www.ecb.europa.eu/stats\n",
    "12. Jayanta K., G., Mohan, D., & Tapas, S. (2006). Bayesian Inference and Decision Theory. In An Introduction to Bayesian Analysis (Vol. 13, pp. 26–63). Springer New York. http://doi.org/10.1007/978-0-387-35433-0_2\n",
    "13. Correa Bahnsen, A., Aouada, D., & Ottersten, B. (2015). Example-Dependent Cost-Sensitive Decision Trees. Expert Systems with Applications, in press. http://doi.org/10.1016/j.eswa.2015.04.042\n",
    "14. Maimon, L. R. O. (2008). Data Mining with Decision Trees: Theory and Applications. World Scientific Publishing Company.\n",
    "15. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Stanford, CA: Springer.\n",
    "16. Rokach, L., & Maimon, O. (2010). Decision Trees. In L. Rokach & O. Maimon (Eds.), Data Mining and Knowledge Discovery Handbook (2nd ed., pp. 149–174). Springer US. http://doi.org/10.1007/978-0-387-09823-4_9\n",
    "17. \\citep{Rokach2009}\n",
    "18. \\citep{Zhou2012}\n",
    "19. \\citep{Louppe2012}\n",
    "20. Correa Bahnsen, A., Aouada, D., & Ottersten, B. (2015). Ensembles of Example-Dependent Cost-Sensitive Decision Trees. http://arxiv.org/abs/1505.04637"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}