{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img img class=\"logo\" src=\"https://raw.githubusercontent.com/albahnsen/CostSensitiveClassification/master/logo.png\" style=\"width: 300px;\" align=\"right\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Example-Dependent Cost-Sensitive Credit Scoring using CostCla\n",
    "<span class=\"right_align\">\n",
    "Last update: Oct 27, 2019 . Updated to work with latest version of sklearn.\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "For running this tutorial you need:\n",
    "-    Python version $\\ge$ 2.7\n",
    "-    Numpy version $\\ge$ 1.8.0\n",
    "-    Pandas version $\\ge$ 0.14.0\n",
    "-    Scikit-learn version $\\ge$ 0.16.1\n",
    "-    pyea version $\\ge$ 0.1\n",
    "-    CostCla version $\\ge$ 0.4dev2\n",
    "\n",
    "The easiest way to install CostCla \n",
    "\n",
    "1.  Clone the repo https://github.com/albahnsen/CostSensitiveClassification\n",
    "2.  cd CostSensitiveClassification\n",
    "3.  pip install .\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  ## Credit Scoring\n",
    "  \n",
    "  In order to mitigate the impact of credit risk and make more objective and accurate decisions, \n",
    "  financial institutions use credit scores to predict and control their losses.\n",
    "  The objective in credit scoring is to classify which potential customers are likely to default a \n",
    "  contracted financial obligation based on the customer's past financial experience, and with that \n",
    "  information decide whether to approve or decline a loan [1]. This tool has \n",
    "  become a standard practice among financial institutions around the world in order to predict \n",
    "  and control their loans portfolios. When constructing credit scores, it is a common practice to \n",
    "  use standard cost-insensitive binary classification algorithms such as logistic regression, \n",
    "  neural networks, discriminant analysis, genetic programing, decision trees, among \n",
    "  others [2,3]. \n",
    "  \n",
    "  Formally, a credit score is a statistical model that allows the estimation of the probability \n",
    "  $\\hat p_i=P(y_i=1|\\mathbf{x}_i)$ of a customer $i$ defaulting a contracted debt. Additionally, \n",
    "since the objective of credit scoring is to estimate a classifier $c_i$ to decide whether or not to grant a \n",
    "loan to a customer $i$, a threshold $t$ is defined such that if $\\hat p_i < t $, then the loan is \n",
    "  granted, i.e., $c_i(t)=0$, and denied otherwise, i.e., $c_i(t)=1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Kaggle Credit Competition\n",
    "\n",
    "Improve on the state of the art in credit scoring by predicting the probability that somebody will experience financial distress in the next two years.\n",
    "\n",
    "https://www.kaggle.com/c/GiveMeSomeCredit\n",
    "\n",
    "### Load dataset and show basic statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['data', 'target', 'cost_mat', 'target_names', 'DESCR', 'feature_names', 'name'])\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from costcla.datasets import load_creditscoring1\n",
    "import warnings\n",
    "warnings.simplefilter(action='ignore',category=FutureWarning)\n",
    "\n",
    "data = load_creditscoring1()\n",
    "\n",
    "#Elements of the data file\n",
    "print (data.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           Frequency  Percentage\n",
      "Negative (Good Customers)     105299    0.932551\n",
      "Positive (Bad Customers)        7616    0.067449\n"
     ]
    }
   ],
   "source": [
    "# Class label\n",
    "target = pd.DataFrame(pd.Series(data.target).value_counts(), columns=('Frequency',))\n",
    "target['Percentage'] = target['Frequency'] / target['Frequency'].sum()\n",
    "target.index = ['Negative (Good Customers)', 'Positive (Bad Customers)']\n",
    "print (target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Full description of the dataset\n",
    "# print data.DESCR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Features</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>RevolvingUtilizationOfUnsecuredLines</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>NumberOfTime30-59DaysPastDueNotWorse</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>DebtRatio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>MonthlyIncome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>NumberOfOpenCreditLinesAndLoans</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>NumberOfTimes90DaysLate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>NumberRealEstateLoansOrLines</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>NumberOfTime60-89DaysPastDueNotWorse</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>NumberOfDependents</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               Features\n",
       "0  RevolvingUtilizationOfUnsecuredLines\n",
       "1                                   age\n",
       "2  NumberOfTime30-59DaysPastDueNotWorse\n",
       "3                             DebtRatio\n",
       "4                         MonthlyIncome\n",
       "5       NumberOfOpenCreditLinesAndLoans\n",
       "6               NumberOfTimes90DaysLate\n",
       "7          NumberRealEstateLoansOrLines\n",
       "8  NumberOfTime60-89DaysPastDueNotWorse\n",
       "9                    NumberOfDependents"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of features\n",
    "pd.DataFrame(data.feature_names, columns=('Features',))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Credit scoring as a standard classification problem\n",
    "\n",
    "Using three classifiers, a model is learned to classify customers in good and bad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load classifiers and split dataset in training and testing\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test, cost_mat_train, cost_mat_test = \\\n",
    "train_test_split(data.data, data.target, data.cost_mat)\n",
    "\n",
    "# Fit the classifiers using the training dataset\n",
    "classifiers = {\"RF\": {\"f\": RandomForestClassifier()},\n",
    "               \"DT\": {\"f\": DecisionTreeClassifier()},\n",
    "               \"LR\": {\"f\": LogisticRegression()}}\n",
    "\n",
    "for model in classifiers.keys():\n",
    "    # Fit\n",
    "    classifiers[model][\"f\"].fit(X_train, y_train)\n",
    "    # Predict\n",
    "    classifiers[model][\"c\"] = classifiers[model][\"f\"].predict(X_test)\n",
    "    classifiers[model][\"p\"] = classifiers[model][\"f\"].predict_proba(X_test)\n",
    "    classifiers[model][\"p_train\"] = classifiers[model][\"f\"].predict_proba(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  After the classifier $c_i$ is estimated, there is a need to evaluate its performance. In \n",
    "  practice, many statistical evaluation measures are used to assess the performance of a credit \n",
    "  scoring model. Measures such as the area under the  receiver operating characteristic curve (AUC),\n",
    "  Brier score, Kolmogorov-Smirnoff (K-S) statistic,  $F_1$-Score, and misclassification are among \n",
    "  the most common [4]. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          f1       pre       rec       acc\n",
      "RF  0.234356  0.509532  0.152174  0.931949\n",
      "DT  0.248201  0.238436  0.258799  0.892699\n",
      "LR  0.071775  0.569231  0.038302  0.932197\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the performance\n",
    "from sklearn.metrics import f1_score, precision_score, recall_score, accuracy_score\n",
    "measures = {\"f1\": f1_score, \"pre\": precision_score, \n",
    "            \"rec\": recall_score, \"acc\": accuracy_score}\n",
    "results = pd.DataFrame(columns=measures.keys())\n",
    "\n",
    "# Evaluate each model in classifiers\n",
    "for model in classifiers.keys():\n",
    "    results.loc[model] = [measures[measure](y_test, classifiers[model][\"c\"]) for measure in measures.keys()]\n",
    "\n",
    "print (results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAo0AAAEvCAYAAAAgkfs1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXzdVb3v/9fae2eekyZN0iSd53mkLR6LFGgRAVFRcTzKOVz1eEWP+nO6crjoueLRq0fUc5VDcUQQUYYCiiJSkLbQeZ5LhyRNkzRNmnlP6/fHytykO213ujO8n4/Hfuzkm2++3xXo3nlnDZ9lrLWIiIiIiFyIJ9YNEBEREZHBT6FRRERERCJSaBQRERGRiBQaRURERCQihUYRERERiUihUUREREQi8sXqxqNGjbLjxo2L1e1FRERE+m3Lli3V1trcWLcjlmIWGseNG8fmzZtjdXsRERGRfjPGHI91G2JNw9MiIiIiEpFCo4iIiIhEpNAoIiIiIhEpNIqIiIhIRAqNIiIiIhKRQqOIiIiIRKTQKCIiIiIRKTRGWd3atRy6diX7ps/g0LUrqVu7NtZNEhERGTD6vTdyxKy493BUt3Ytp75+D7alBYBgeTmnvn4PABk33xzLpomIiESdfu+NLOppjKLK7/9nxwunnW1pofI73yFw+jSh+npsKBSj1omIiFw6GwwSqqsjUF5O6+HDNO/Ywen7v937773v/2eMWikDST2NURQ8dar345VVHF5xTcfnJjERT0oKnuRk92j/uK/n9o9Tej/HJCRgjLlCP6WIiAx21lpsSwvhxkbCTU29Pzc2EW5qe77QeW3P1u/v9/37+n0oQ5tCYxT5CgoIlpefd9ybmUnu5z53wRdkqK6OwKlT3Y7T315Jjycq4bPrOcanfxoiIleKDQbde/+FAl4/gl3XZ8Lhft3bxMV1//2Q7J59uaPO/z3R43dG+Ve/RujMmfOu6SsoiPZ/IhkElAyiKO9zn+02twNcr+Lor331oud2WGuxfv+F3zCaevyl2OM5UFHR5VgTtqmp3/c38fGRQ+iFwmfPEJqUpN5QERkWrLXY1taoBDv3fY3Y1tZ+39+0v/92ea/15mQTV1zU7T3Ym5KCSU7u9tzb+7SJj7/k/xajv/ylXn/v5X3us5d8TRm8FBqjqD0YVn7/PwmeOoWvoIC8z332kiYDG2MwCQl4EhIgKysq7bOhEOHm9uGKLgGzH29stqmJUEM9wcrThBobsY1NhJqaIBDo7w90geH4i+gljdIbnYiMHO69r/myg123Xrz+jgT5fN3f69p68eKyss4Lfr315HV/TsGTnITxDJ7lCNH8vSeDn7HWxuTGixYtsps3b47JvSV6rN/vQmSPYZX2Yz2f+/MXeX/1NaTS8y/r3p6HwpuxyEhkrcUGAj3eJy48qhJpXp5tbu73/U1SUv+m+3QLep3vP+65cxqQiYvTKMswYYzZYq1dFOt2xJJ6GuWymPh4fPHx0esNDYcjT96OFD6rqmntcs7FTN7uNuzT8UvgQj2g+oUhI5sNhwk3NbcFt4vpuest/LXN5w4G+3dzj6fX3rm4wsILBLw+Ql9KCp6kJIzXO7D/wUSGMIVGGVSMx9PRExgtNhDoe4J5P3o+QzVnCZwsvaQJ5vh8keeFJvezJ0O/1CQKus2VjjA6EDn8XeRc6YSE8/59e9PTicvPjxzwepnGosoRIleWQqMMeyYuDm9GBt6MjKhcr6OUxUWGz64fB86e7T581qPO2QV/nq4lmy5iDuh5v3hVsmnQs9Zi2+fiXeScu75KqtiLmYfcy7+huLzRF/63dYF/k6rKIDK06RUscpGMMW7eU1IS5ORE5Zo2GHQT9S8yfHb0EtXWEigvv7SSTV5v30GzP+Gzl3NjEQ7q1q6N+WR8Gwz2P9j1p0ZeUxP0c965iY/vtafOl5cXudxWL8dMYqL+mBCRbhQaRQYB4/PhTUvDm5YWlevFvGRTQkL/QmjX+aAXWikfoWTTpWxl1muPcc+AN4DFj3v7OX25uZGrCfRVNiUurt/3FhG5FFo9LSL9ctklm7rOmbvUkk19BMv6l1/uNdSapCRSli7te6V+f+emxsXhTU7GpERemd9rbbzeQrBW6osMKVo9rZ5GEekn4/XiTU3Bm5oStWtebsmmYFUV4WPH+uwFtc3NBCoq8KQk483OcsWPIyw66jX8qSaoiIhCo4jETrRKNh26dmWvW3j6CguZ8OQfLuvaIiLiaHxERIa8vM99FpOY2O2YtjITEYku9TSKyJCnrcxERAaeQqOIDAsZN9+skCgiMoA0PC0iIiIiESk0ioiIiEhECo0iIiIiEpFCo4iIiIhEpNAoIiIiIhEpNIqIiIhIRAqNIiIiIhKRQqOIiIiIRKTQKCIiIiIRKTSKiIiISEQKjSIiIiISkUKjiIiIiESk0CgiIiIiESk0ioiIiEhECo0iIiIiEpFCo4iIiIhEpNAoIiIiIhH1KzQaY1YbYw4YYw4bY77cy9dLjDF/M8ZsM8bsNMa8PfpNFREREZFYiRgajTFe4MfAjcAM4A5jzIwep/0v4HFr7Xzg/cB/RbuhIiIiIhI7/elpXAIcttYetdb6gceAW3ucY4H0to8zgPLoNVFEREREYs3Xj3PGACe7fF4KXNXjnHuBPxtj/ieQAlwXldaJiIiIyKAQrYUwdwA/t9YWAW8HfmWMOe/axpi7jDGbjTGbq6qqonRrERERERlo/QmNZUBxl8+L2o51dSfwOIC1dgOQCIzqeSFr7YPW2kXW2kW5ubmX1mIRERERueL6Exo3AZONMeONMfG4hS7P9DjnBLASwBgzHRca1ZUoIiIiMkxEDI3W2iDwaeAFYB9ulfQeY8x9xphb2k77PPDPxpgdwKPAP1pr7UA1WkRERESurP4shMFa+zzwfI9j93T5eC9wdXSbJiIiIiKDhXaEEREREZGIFBpFREREJCKFRhERERGJSKFRRERERCJSaBQRERGRiBQaRURERCQihUYRERERiUihUUREREQiUmgUERERkYgUGkVEREQkIoVGEREREYlIoVFEREREIlJoFBEREZGIFBpFREREJCKFRhERERGJSKFRRERERCJSaBQRERGRiBQaRURERCQihUYRERERiUihUUREREQiUmgUERERkYgUGkVEREQkIoVGEREREYlIoVFEREREIlJoFBEREZGIFBpFREREJCKFRhERERGJSKFRRERERCJSaBQRERGRiBQaRURERCQihUYRERERiUihUUREREQiUmgUERERkYgUGkVEREQkIoVGEREREYlIoVFEREREIlJoFBEREZGIFBpFREREJCKFRhERERGJSKFRRERERCJSaBQRERGRiBQaRURERCQihUYRERERiUihUUREREQiUmgUERERkYgUGkVEREQkon6FRmPMamPMAWPMYWPMl/s4573GmL3GmD3GmN9Et5kiIiIiEku+SCcYY7zAj4HrgVJgkzHmGWvt3i7nTAa+AlxtrT1rjMkbqAaLiIiIyJXXn57GJcBha+1Ra60feAy4tcc5/wz82Fp7FsBaWxndZoqIiIhILPUnNI4BTnb5vLTtWFdTgCnGmNeMMRuNMauj1UARERERib2Iw9MXcZ3JwDVAEfCKMWa2tba260nGmLuAuwBKSkqidGsRERGRK2/Lli15Pp/vIWAWQ39xcRjYHQwG/2nhwoW9jhj3JzSWAcVdPi9qO9ZVKfC6tTYAvGmMOYgLkZu6nmStfRB4EGDRokW2Xz+CiIiIyCDk8/keys/Pn56bm3vW4/EM6VwTDodNVVXVjIqKioeAW3o7pz+peBMw2Rgz3hgTD7wfeKbHOU/hehkxxozCDVcfvdSGi4iIiAwBs3Jzc88N9cAI4PF4bG5ubh2u17T3cyJdxFobBD4NvADsAx631u4xxtxnjGlPoi8AZ4wxe4G/AV+01p657J9AREREZPDyDIfA2K7tZ+kzG/ZrTqO19nng+R7H7unysQX+te0hIiIiIsPMUJ+0KSIiIjJiffOb38ybMGHCzFWrVk2cN2/etPj4+AX33HPP6IG4V7RWT4uIiMhItPNx+Ot9UFcKGUWw8h6Y895Yt2pQ+vXG49kP/PXQmKr61vjctAT/Z1ZOLvvQ0rE1l3PNNWvW5L744osHExMT7eHDh+OfeOKJrGi1tyf1NIqIiMil2fk4rP0M1J0ErHte+xl3XLr59cbj2d94du/YyvrWeAtU1rfGf+PZvWN/vfF49qVe8wMf+EBJaWlpwo033jj5oYceyl6xYkVTXFzcgM2xVE+jiIiIXJoX74VAc/djgWbX8zjCehu/+MSO4oMV9cl9fX3vqXMpgZA1XY+1BsOe/712z7jfbT6Z29v3TMlPa/rOe+ae7O1rAL/5zW9OrFu3LmPdunUHCwoKgpfe+v5RaBQREZGLc3ovbF4D53qWbW5TV3pl2zME9AyMkY4PRgqNIiIiElmwFfY+DZsfhhMbwJsAcckQaDr/3IyiK9++GLtQjyDAkn9/cXZlfWt8z+N5aQn+pz/9lgMD17Lo0ZxGERER6VvNm/CXe+B70+EP/wwNp+GGb8Ln98PNP4C4pO7nxyW5xTDSzWdWTi5L8HnCXY8l+Dzhz6yc3Ed37eCjnkYRERHpLhyCgy+4IejDfwXjgak3wuI7Yfw14Gnrc2qft6jV0xG1r5KO9urpdidOnPAtXrx4RmNjo9cYY3/605+O3rdv3+7s7Oxw5O/uH4VGERERceorYOsvYcsv4FwppBXAii/Bwo9CemHv3zPnvQqJ/fShpWNrohUS25WVle1q//j06dM7o3ntnhQaRURERjJr4c1XXK/i/ucgHIQJb4Mb74cpq8EbF+sWyiCh0CgiIjISNZ+F7Y+6hS1nDkFSFlz1CVj0cciZGOvWySCk0CgiIjJSWAtlW12v4u7fQ7AFihbDO38CM995/qIWkS4UGkVERIY7fyPsesKFxVM7IC4F5t7hehUL5sS6dTJEKDSKiIgMV5X73fDzjsegtQ7yZsDbvwtz3geJ6bFunQwxCo0iIiLDSdAP+9fCpofh+N/BGw8zboVFd0LJUjBDZgMSGWQUGkVERIaDs8dhy89h26+gsQoyx8J198L8D0PKqBg3ToYDhUYREZGhKhyCwy/CpjVw6M+uF3HKaterOPHaziLcMjhsWpPNum+PoaEyntQ8Pyu+VMbiO6Nat7E3wWAQn+/yI59CY7TtfFyV8UVEZGA1VLoexc0/h7oTkDoa3voFWPBRyCyOdeukN5vWZPPCV8YSbHVJvuF0PC98ZSzA5QTHAwcOxK9evXry7Nmzm3bv3p08ZcqU5t/97nfHpk2bNvOWW26pWbduXfpnP/vZiquvvrrpE5/4RElNTY0vMTEx/NBDDx2fP39+y8XcS6ExmnY+Dms/A4Fm93ndSfc5KDiKiMjlsRaOv+Z6FfethXAAxv0D3HAfTHuHinDH2lP/Ukzl3uQ+v16xK4VwoPuE0mCrhz9+aRzbfp3b6/fkzWjinT8+GenWx44dS/zpT3967IYbbmi8/fbbx33nO9/JBcjJyQnu3bt3H8CyZcumPPjgg8dnz57d+tJLL6V88pOfLNm4cePBi/gJFRqj6q/3dQbGdoFmd1yhUURELkVLnVv9vPlhqNoPiRmw5J9h4ccgd0qsWyf91TMwRjp+EfLz8/033HBDI8CHP/zhMw888EAewEc+8pGzAHV1dZ5t27al3n777R1V2/1+/0XfV6ExmupK+zh+EppqIDn7yrZHRESGrvLtrq7iricg0ASFC+DWH8PMd0F83x1aEiORegS/O2U2DafjzzueOtrPXX87cDm3Nj1WxLd/npaWFgYIhUKkpaUF9+/fv/dy7qMZstGUUdT31747GX79btj2a7d1k4iISE/+Jtj2CPz3tfDgCtj5O5j1brjrZbjrbzD/QwqMQ9WKL5XhSwh3O+ZLCLPiS2WXe+lTp07Fv/jiiykAjzzySPby5csbun49Ozs7XFRU5H/44YezAMLhMBs2bLjo7X8UGqNp5T3nb8EUlwRv+1+w7NNQfQie/hf4zmR45Hb3xqAAKSIi1YfgT1+B702Dpz8FrQ2w+tvw+f1w64+gcH6sWyiXa/GdNaz61nFSR/vBuB7GVd86Ho3V0+PGjWv54Q9/mDdhwoSZtbW1vi984QtVPc959NFHj/7sZz8bNXXq1BmTJ0+e+fvf/z7zYu+j4eloap+32Nfq6evuhfJtsOdJ2PMUHPoUrI1zZRFm3gbT3u7mqoiIyPAXCsD+59wQ9JuvgCcOpt8Mi++EsVerCPdwtPjOmoEosePz+Xj66aff7HqsrKxsV9fPp02b5n/11VcPXdZ9LuebpRdz3tv3ohdjYMwC97j+Prdp/N72APmCq9o/caULkFNv1BZPIiLDUV0pbPkFbP0lNFRARjFc+3VY8BFIzYt160T6pNAYK8ZA0UL3uP4bULalrQfySTj4R/AmwKTr2gLkakhIi3WLRUTkUoXDcOQl16t48E+ufM7k62HRD9yzxxvrFsoQNXXqVP+hQ4f2XIl7KTQOBsZA0SL3uP4bULa5cwj7wHMuQE6+3gXIKashITXWLRYRkf5orHYLILf8DM4eg5RcuPqzsPAfIWtsrFsnclEUGgcbjweKl7jHDf8OpW90Bsj9z4IvESbf0BYgV0F8SqxbLCIiXVkLJza6uop7n4KQ381RvPbrMP0W8J1fdUVkKFBoHMw8HihZ6h6rvgUnN7oAufdp2PcM+JJccJx5mwuSKsMgIhI7Ledg529h88+gcg8kpLsexUUfh7zpsW6dyGVTaBwqPB4Yu9w9Vt8PJzZ0Bsi9T0Fcshu6nnmbG8ruWfpHREQGRsUut7Xfrt+BvwHy58DNP4DZt2s0SIYVhcahyOOFcW9xjxv/w+1FuudJ2PsM7PkDxKW4xTMzb3OLaRQgRUSiK9Di/mDftMZNI/IluiLci+50FTJULkeGIYXGoc7jhfFvdY8bv9MZIPc9A7t/D/GprnzPzNtcOZ+4xFi3WERk6DpzxC1q2fYINNdAziRY9X9g7h3aKlYi+u2B32b/ZMdPxpxpPhOfk5Tj/8TcT5S9b+r7ola3MRwOY63F6x2Y1fgKjcOJ1wcTVrjH278Lx15tC5Br3bBJfJorID7zNldQ3JcQ6xaLiAx+oaArhbZpDRz9GxgvTLvJFeEev0K9itIvvz3w2+z/2PQfY/0hvwegurk6/j82/cdYgMsJjgcOHIhftWrVlPnz5zfs2rUr5e67765Ys2ZNrt/vN2PHjm197LHHjmVkZITXrVuX/NnPfrakqanJEx8fb1955ZUDWVlZ4ch36GSstZfazsuyaNEiu3nz5pjce8QJBdxuA+0BsqXWTdCedpMLkBPeptV8IiI9nSt3Bbi3/ALqyyF9jFvYMv/DkF4Q69bJFWaM2WKtXdT12I4dO47NnTu3GuDrr329+PDZw32uSN1/dn9KMBw87y8Mn8dnp2VNa+zteyZlTWr6xtXfOHmhdh04cCB++vTps//yl7/snzFjRuvNN9888aWXXjqUnp4e/trXvpbf2tpqvvnNb1ZMmjRp1iOPPHJkxYoVTTU1NZ60tLRwXFzcedfbsWPHqLlz547r7V7qaRwJvHEwaaV7vOP7cHSdC5D718KORyEhA6a/wwXI8SsUIEVk5AqH4c2XXbmc/c+DDbmpPTd9FyavciM6Ipegt8B4oeMXo6CgwL9y5crGRx99NOPIkSOJS5YsmQYQCATMwoULG3bu3JmYl5cXWLFiRRNAdnb2RfUwttO//pHGGweTr3OP4Pfh6MttPZDPwvZHIDETprUFyAkr3PkiIsNdU417D9z8M6g5AknZsOxfYNHHIHtCrFsnQ0CkHsG3Pf622dXN1ef1yoxKGuV/9B2PHriceycnJ4cBrLW85S1vObd27dpu+1C/8cYbUVkRq9A4kvniYcoN7hFshSN/61xEs/3XkJTVGSDHv1UBUkSGF2uhdLPb2m/3HyDUCsVL4ZovuyLcWjgoUfSJuZ8o6zqnESDeGx/+xNxPlEXrHtdcc03j5z//+ZLdu3cnzJo1q/XcuXOeY8eOxc2ZM6elsrIybt26dckrVqxoOnv2rCc1NbXX4ekLUWgUx5fgyvRMXd0WIF/q3Ilm26/cX93Tb3YBctw/aIhGRIau1ga3OHDzGldjMT4V5n/IFeHOnxXr1skw1b7YZSBXTxcWFgZ/+tOfHnv/+98/we/3G4B/+7d/K5szZ07rI488cuQzn/lMSUtLiycxMTH8yiuvHMzIyNBCGImiQAsc+asLkAf+6ArXJue4v8Jn3ua2xlKAFJGh4PReN1dxx2Pgr4fRs1xQnPNeSEiLdetkkIu0EGa40EIYuXRxiW6V9bSbINAMh190AXLn465WWUpulwC53NWNFBEZLIKtbuODzWvcTlreBJj5TleEu3iJyuWIXASFRum/uCQ3RD39ZvA3weG/uAC541H3hpySBzNudQGyZKkCpIjEztljblHLtl9DUzVkjYfrvwHzPggpObFunciQpNAolyY+2QXEGbeCvxEO/dkFyG2/hk3/Dan5MKOtB7J4qds7W0RkIIVDcPAFNwR9+EXXizj17W4IesLb9D4kcpkUGuXyxae4cDjzNjfB/NALLkBu/SW88SCkFXT2QBYt0Ru3iERX/em2Itw/h3Ol7o/WFf8fLPgoZIyJdetEhg2FRomuhFSY9W73aK13f/XvedINE73+E0grdPOJZt4GYxYpQIrIpbHWbZW6aQ3sfxbCQZhwDaz+Fky9USXCRAaAQqMMnIQ0mP0e92g5Bwf/5ALkpodg439BelGXALlQE9JFJLLms7D9UTcEfeaQ25Dgqk/Awo/BqEmxbp3IsKbQKFdGYrorazHnvdBSBwfaAuTrP4UNP4KMEpjZNoRduEABUkS6K9sCmx6G3b+HYLMbqXjn/3PvGXFR2exCRCJQaJQrLzED5r7PPZprXf3HPU/Cxp/A+h9CZknnHMmCeQqQIiOVvwl2P+GGoE9th7hk94fn4juhYG6sWydy0WoefSz7zH/915hgdXW8b9Qof86nPlWWfcf7o1bce6D1a0KZMWa1MeaAMeawMebLFzjv3cYYa4xZ1Nc5It0kZcK8O+CDj8MXD8Gt/wWjpsKGH8OD18AD8+DFe6F8u5vDJCLDX9UB+OOX4P9Og2f+p6u1+Pbvwuf3wy0PKDDKkFTz6GPZlfffPzZYVRWPtQSrquIr779/bM2jj2Vf7rWvu+66iTNnzpw+adKkmd/97ndHATzxxBPpM2bMmD516tQZy5YtmwJQV1fnec973jNuypQpM6ZMmTLj5z//eebF3CfijjDGGC9wELgeKAU2AXdYa/f2OC8NeA6IBz5trb3gdi/aEUYuqKkG9j/neiCPvgw25OqstfdA5s9WD6TIcBL0w/61btHcsVfBE+eqLiy+E0qW6fUuMRdpR5jyr36tuPXQoeS+vr9l//4UAoHz/yHHxdnEadMae/uehMmTmwr/z7+fjNS206dPe0ePHh1qaGgw8+fPn/HSSy8dWLx48YyXX355/7Rp0/ztX//kJz85prW11fPwww+fBKiqqvLm5uaGevxMl7UjzBLgsLX2KIAx5jHgVmBvj/O+AXwb+GI/rilyYcnZsODD7tF4xq2O3PMkvPYD+Pv3IHtiZ4AcPVO/UESGqtoTrlTO1l9BY6WbnrLy32D+hyE1N9atE4me3gLjhY5fhG9/+9ujn3vuuUyAioqKuAceeCB3yZIl9dOmTfMDjB49OgTwyiuvpD/22GNH27+vZ2CMpD+hcQzQNeWWAld1PcEYswAottY+Z4zpMzQaY+4C7gIoKSm5mHbKSJaSAws/6h6N1bBvrQuQf/8evPpdyJncGSDzpitAigx24RAc/qvbSerQn92xyatcr+LElSrFJUNSpB7BQ//w1tnBqqr4nsd9ubn+8b97/MCl3vfZZ59NW7duXdrmzZv3p6WlhZcsWTJ1/vz5TQcOHEi81Gv25bJfmcYYD/A94PORzrXWPmitXWStXZSbq78g5RKkjIJFH4OPPgOfPwg3fQ/SC1x4/H/L4MdXwd++BZX7Y91SEempoQpe/b9urvJvboeyrfCWf4W7d8IHHoPJ1yswyrCV86lPlZmEhHDXYyYhIZzzqU+VXc51a2trvRkZGaG0tLTwtm3bEnfs2JHS0tLieeONN9L2798fD274GmDFihXnvv/97+e1f29VVdVF7ffbn57GMqC4y+dFbcfapQGzgJeN6+HJB54xxtwSaV6jyGVJzXU9E4vvhIZK2PcM7HkK1n0b1t0PudM7eyBzp8S6tSIjk7VwfL3rVdz7DIQDMO4f4Lr/DdPeAb7zOl5EhqX2VdLRXj397ne/u+7BBx/MnTBhwswJEya0zJ07tzEvLy/4wAMPHLvtttsmhcNhcnJyAuvXrz/0rW9969THPvaxksmTJ8/0eDz2q1/9avlHP/rR2v7eqz8LYXy4hTArcWFxE/ABa+2ePs5/GfiCFsJIzNRXdA5hH18PWMib2RYg3wmjJse6hSLDX0sd7PitK8JdtQ8SMlylhEUfh9ypsW6dyEWLtBBmuLishTDW2qAx5tPAC4AXeNhau8cYcx+w2Vr7TFRbK3K50vJhyT+7x7lTbT2QT8Lfvukeo2d37kSTMzHWrRUZXk7tcHUVd/0OAk1QOB9u+ZHbWjS+z4WlIjIE9Ku4t7X2eeD5Hsfu6ePcay6/WSJRkl4AV/0P96gr6wyQL33DPfJnu/A4450KkCKXKtAMu//gehXLNoMvCWa/GxbdCWMWxLp1IhIl2hFGRo6MMbD0k+5RVwp7n3YB8q/3uUfB3M4AmT0+1q0VGfyqD7uguP0RaKmFUVNg9f0w9/2QlBXr1olcCeFwOGw8Hs+w2H0iHA4bINzX1xUaZWTKKIJl/+IetSc6A+SL97pH4fzOAJk1NtatFRk8QgE48Lwbgn5zHXh8MP1m16s47i0qeSUjze6qqqoZubm5dUM9OIbDYVNVVZUB7O7rnIgLYQaKFsLIoHT2eGeALN/qjo1Z2BkgM4sv/P0iw1VdWVsR7l9CQwVkFLvaqfM/AmmjY906kQHX20KYLVu25Pl8vodwVWSGer2oMLA7GKYyKhIAACAASURBVAz+08KFCyt7O0GhUaQvNW92BshT292xosVtAfJW11spMpyFw3D0Jdj0MBz8oyufM/l6twJ68g3guagSbyJDWm+hcaRRaBTpj5qjrgbkniehYqc7VnxVZ4BML4xt+0SiqfEMbP+12wf67JuQPMpt6bnwHyFrXKxbJxITCo0KjSIX78wRFx73PAWnd7ljJcvc8PWMW92KbZGhxlo4+bqbq7j3KQj5oWS5K54//WbwJcS6hSIxpdCo0ChyeaoPdfZAVu4BjAuQM2+DGbe4mpEig1lrPez8rRuCrtwD8Wlu9fOij8PoGbFuncigodCo0CgSPVUHOgNk1T7AwNirXSHxGbdCal7ES4hcMRW73dZ+Ox8Hf4OrWbroTph9OySkxrp1IoOOQqNCo8jAqNzfNoT9JFQfAONpC5C3wfRb3L7ZIldaoMUt7tq8xg1F+xJh5rvcEPSYhSqXI3IBCo0KjSIDy1qo3OfmiO3+A5w55ALkuH9oC5A3Q8qoWLdShruao64I97ZHoLkGsie64ed5H4Dk7Fi3TmRIUGhUaBS5cqyFyr2u93H3H6DmCBgvjH9rZ4DUL3CJllAQDv7J9Soeecn9W5t2kwuL41eAZ6iXlBO5shQaFRpFYsNaOL27cwi75qj7pT7hGjcHcto7FCDl0pw75Qpwb/0FnCuDtEJXKmfBR7SyX+QyKDQqNIrEnrWu9mN7gDx7zG3NNuEa1wM57Sbt4ysXFg67Lf02r4H9z4MNwcRr3cKWKavBqx1jRS6XQqNCo8jgYq3bfaY9QNaeAE8cTHybC5BT3w5JmbFupQwWTTWw/TduvmLNEUjKhvkfhIUfg5yJsW6dyLCi0KjQKDJ4Wev2v24vJF530gXISSvbAuSNkJgR61bKlWYtlG1xRbj3/AGCLW53okV3utJOcYmxbqHIsKTQCBqzEBmsjHFlUMYshOu/4YJCe4A8+CfwxsOk61yAnLIaEtNj3WIZSP5G2PU7FxYrdkJ8qlv9vOjjrsaiiMgAU0+jyFATDncJkE9CfTl4E2Dy9W0BchUkpMW6lRItlftcUNz5W2g9B3kzYfHHYc779P9Z5ApST6NCo8jQFg5D6SYXHvc+BfWnXMHmrj2Q2t1j6Am2wr61LiyeWO96lWfe5noVi69SEW6RGFBoVGgUGT7CYbfLR3uAbDgNvqTuPZDxKbFupVzI2WOw5eew9VfQVA1Z49qKcH8IUnJi3DiRkU2hUaFRZHgKh+DExrYA+TQ0VroAOWWVC5CTb4D45Fi3UsD9vzr0Z9erePhF14s45UY3BD3hWhXhFhkkFBoVGkWGv3AIjq93AXLfM9BYBXHJbuh65m2uJzIuKdatHHnqT8O2X8KWX7iV8an5rgD3wo9CRlGsWyciPSg0KjSKjCyhIBx/rS1ArnVDoHEprnzPzNvcXEiVbBk41sKxv7si3PvWQjjotvRbfKerwemNi3ULRaQPCo0KjSIjVygIx//eNoT9DDTXQHxaZ4CceK0CZLQ018KOR10R7uqDkJgJ8z4Iiz4GoybHunUi0g8KjQqNIgIQCsCxVzt7IJvPQkK66/2aeZvbkcaXEOtWDj1lW12v4q7fQ7DZ1dxcdCfMepemBIgMMQqNCo0i0lMo4PYx3vMk7HsWWmohIQOmtQXICW8DX3ysWzl4+Ztg9+9dWCzf5uaPzr7drYIunBfr1onIJVJoVGgUkQsJ+rsHyNY6t3XhtHe4ADl+hQJku6qDbvh5x2+gpQ5yp7lexbnv03aPIsOAQqNCo4j0V9APR//mAuT+59zuJImZML1LgBxpCzmCftj/rAuLx151e4PPuMWFxbHLVYRbZBhRaNTe0yLSX754V+dxyiq3Y8mRl9q2Mnwatv0akrJg+s0uQI57K3iH8dtL7cm2Ity/dDUwM0pg5T0w/8OQmhfr1omIDIhh/K4uIgPGl+BWWU+9EQItnQFy9x9ckErO6QyQY98yPAJkOOR+zk1r4NALrnzOlFWuV3HSSvB4Y91CEZEBNQzeyUUkpuIS3SKZaW+HQDMc/qsLkDt/53rjkke5IduZt8HYq4deuGqogm2/gi0/g9oTkJILb/kcLPxHyCyJdetERK4YzWkUkYERaIZDf3EB8uCfINDkAtf09gC5PKoB8rmjz/GDrT+gorGC/JR87l5wNzdNuOnSLmYtnNjgehX3Pg3hgOsxXfxxmHazFv+IjECa06jQKCJXgr/J7a+850k4+IKrWZiSBzNudQGyZOllBcjnjj7HvevvpSXU0nEs0ZvIvcvvvbjg2FIHOx93YbFqnys1NO8OVy4nd+olt09Ehj6FRoVGEbnS/I0uOO550gXJYIvbd7k9QBZfBR7PRV3yhidu4FTjqfOOF6QU8Of3/DnyBU7tcEFx1xMQaISCeW5rv1nvhviUi2qLiAxPCo2a0ygiV1p8itsRZda7oLXBDV3vedLNf3zjp5BWADPe6QJk0eKOABkMB6lorKC0oZTSevcoayijtL6018AIcKrxFDUtNWQnZp//xUCzu++mNVC2GXxJLiQu/rjbuUVERLpRT6OIDAq25Rx1e5+kbN+TnKzYQqkHSpPTKU3Po8xjONVaQ8iGOs73eXyMSR3DmNQxbK7Yhj/c3Oe1p2dPZ3nhcpYXLmeeN534bb+C7Y+47RJzJrvh53l3uLJBIiK9UE+jehpF5Aryh/yUN5R39Ba29xSerC+ltKGUxkCDO3FUJgDZeCmqL2dOIMBqk0R2zhzSCq4lZfRbyIgfRSjsIRS2vL7zIWz24xhPoONeNhxHQsN1/I8Vk1hf9hq/2PMz1uxeQ1I4zKIWP8uLp7Nszj8yYcbtmIscDhcRGYnU0ygySFlrCYUtwXDbc8gSDIcJth8LWQLhMKGwJRAKd5zbfl7n91iCoXDHddrPDYQtoVCX6513na7fYwmFwx3XO+86bV8LhMO02jpabSV+U43fVBM01QS91YQ9Z7DeOjCd7zk27MMGsgn7swl3eXbHssAmkEYT13m2cJN3I2/17CTehCi1o3g2tJTnQ1ex004ADEsyHuNM7laqfIbcoCWnagEn6q5nzew9TC1/ktamSjZnj2H9mBmst40cazgJQF5yXkcv5NKCpWQlqrdRRM6nnkaFxqh7alsZ33nhAOW1zRRmJvHFVVN55/wxsW7WsGJtL8El1B6wOoNN+8fnH3chp/N7OgNSsD0c9RW4wmFCXa7f/d5drxPuPeyFulynR8Dr7WeIBY8Bn8eDz2vwegw+j8Hn9eDztH3uDUJcDdZ3BuurJuSpIeSpJuipJmCqCRt/t+slkEWSJ48UTx4p3jzSvKNJ944mIy6fFF8m8V4v3rb7td8jzus5794+ryExWE9+xUsUlP6RrFOv4bFBWlOLeL1hNEvsThJNZ09jyHqAMAbDK+E5/N67inNFb2Pu2FHML8mkILuZPWc3s758PRtPbeSc/xwGw/ScLkPZufOIG2lbI4pIrxQaFRqj6qltZXzlD7toDnTOu0qK8/Ktd82+IsExHO4Sli6mx6mXsNL98/N7nHoGpW69UV3uFwqH23q0el6n98B1fjDretydG4pRmPK2B5oIwcbr6fzY5zE9Alhb+PK2X8dDXC/hLNJ14jxt9/a64649Xe7hPT/sxfW4Zsd1vF2u5zFgLJVNlW6xSfuiky7DydXN1d3+uyT7kilKK6Iotcg9t308Js3NN0zwJgzM/5CmGjjwPOx5Env4RXrb5TngS6HsfX9hU106207Wsu1ELQcqztH+T2jCqBTmlWQytzidjIzTnPLvYOOpDeys2knQBknyJbE4fzHLCpaxvHA54zPGY7SftMiIpNCo0BhVV9//EmW150/GT0nw8u4FRRcc4gv2CG3tgav3sNfLdcJhYvS/skuw6d5b1FvI6Rls+gxfXYKNryPo9B2A2r/e93U8HWGv+3X6Dlw92zacwkKDv6HbKuTSBvcoqy+jrKGMQLizx85jPBSkFHQEwY5w2PacmZAZ+/8292YCvb0ADNxb2+1IY2uQnaV1bDt5lm0natl24izVDa53NDHOw5wxmcwqSiA18xg1dg87qt/g2LljAIxOHt3RC3lVwVUayhYZQRQaFRqjavyXn+v11xZAVnJc/3qVugawnsGlI8z0cZ2eQ4oXDEie84NZXz1a3s7etZ5hz2OIfWCQ8/RWnqZrb2Fta/cglR6ffl5v4ZjUMRSnFpOfmk+cZ5AP0X5/FtSdPP94RjF8bvcFv9VaS+nZ5raeSBck95afwx8KA1CYkcj04iBpWW9SZ3ZzoHYr9YF6DIYZOTNYXricZYXLNJQtMswpNCo0RlVfPY1jMpN47cvXxqBFMlxZa6lrrTtv+Lj9uaKxotfyNEWpLgz2HEZOj0+P4U8TBTsfh7WfcbUX28Ulwc0PwJz3XvTlWoMh9pafcz2RbWGy9Ky7dpzXMmHMWTJz3qTRu5eTTfsJ21DHUHZ7iByfrqFskeFEoVGhMapiPadRhpee5WnaQ2F7mZqG9vI0bbITs7v3FnZ5zkvOwxvFfZ4HpZ2Pw1/vg7pSyCiClfdcUmDsS2V9C9u7hMidpXU0+UPgaSEr+wQ5ucdo8e2jLugKjeen5HcEyKX5S8lMzIxaW0TkylNoVGiMOq2elv6y1nKm5UxbncKTHWGwPSRWNlViu0x4SPAm9DmvcEzqGJLjkmP404w8wVCYg6cbus2NPFLViImrwZdyiMycNwnGHyRIU9tQ9kyWFy7TULbIEKXQqNAoMqCag82U1Zf1ugq5tL6UllBLt/PzkvPOW4Xc/pyTlIPHqAj1YFbXFGB7aW1bj+RZtp44Q4M9hi/1IAlphyHhBJgw8Z5EFo5ezIriq1leuJxx6eM0lC0yyCk0KjSKXJZQOERVcxUn6092hsEuw8lnWs50O7+v8jRFaUUUphYOXHkaiQlrLW9WN7bNjTzLlhOnONKwA5N0EF/qITzx7t9Hui+XhXlLWT3xrVw9ZhkZCRkxbrmI9KTQ2M/QaIxZDfwA8AIPWWvv7/H1fwX+CQgCVcDHrbXHL3RNhUYZKur99Z1Dxz16Cy9UnqZ92HjQlaeRmGryB9lVWse2k7VsOH6AnTWbafbuxZdyBONtAWvIjpvIvFFLuGnSCt42fvHgX70uMgIoNPYjNBpjvMBB4HqgFNgE3GGt3dvlnLcBr1trm4wxnwSusda+70LXVWiUwSIQDrjyNPWdtQq7DifXtdZ1Oz8jIaNjJXK33sLUoqFRnkYGFWst5XUtbD5ezUtHt7Cj+nWqQrswiScxJgzhBLI905mVs5jVE1Zw3aSZJMX7Yt1skRFHobF/oXEZcK+1dlXb518BsNZ+q4/z5wM/stZefaHrKjTKldJneZq2j0dceRoZ9PzBMFtOlvPswVfZUrmRU/6dhH1uJ55wIIt0O5MZmYu4dtxyrh5fwticZPVgiwwwhcb+hcb3AKuttf/U9vmHgaustZ/u4/wfARXW2m9e6LoKjRJN/pC/Ywi55yrk0oZSGgON3c4f8eVpZMjZUXGEp/b/jTcqNlDasouwacZaQ7ilCF/rNKakL+AtxQtZOHYUc4szSU9Uj7dINCk0QlTHOIwxHwIWASv6+PpdwF0AJSUl0by1DHM9y9P0XIXcV3maorQiFo5eqPI0MuTNzZ/I3PyJwD8RDAfZUbmL5w6/zPry9ZQ1v8gB/sL+Ewn8ZP8Ewo1TGJMwl0VjJrNgbDbzSzKZnJeG16PeSBG5dFEbnjbGXAf8EFhhra2MdGP1NEpPTYGmbr2EXReflDWUXVR5mlFJozRcJyPGOf85Np3axMsn/86rpa9xptUVGCeYjb9+EqHGySQEpjB3TCHzSzKZX5zFvJJMRqVqtb5If6mnsX+h0YdbCLMSKMMthPmAtXZPl3PmA0/ghrEP9efGCo0jT8/yND17C3srT1OcVnzevEKVpxG5sJPnTrK+fD3ry9ez8dTrNAUbAUNCcBwNtRMINEwm1FxMSXYa80symVecyfySLGYUpBPvUy1Qkd4oNPa/5M7bgf/Eldx52Fr778aY+4DN1tpnjDEvArOBtj9vOWGtveVC11RoHJ7q/fV9rkIuaygjGA52nNuzPE1HiRqVpxGJmkA4wO7q3R0hcnf1bsI2TJxJIs1Oo752Amerx2MDOcT7vMwqTGd+SZbrkSzJojAjUa9DERQaQcW95SL1LE/Tdfi4r/I0va1CLkorIj9F5WlErrS61jo2VWzqCJFlDWUAZMfnk+WZRUvdRN4sLaTV73ry89ISOgLk/OJMZhdlkKySPzICKTQqNEoP1lpqW2t7nVcYqTxN17I0Kk8jMvhZazlZ3zmU/UbFGzQGGvEYDxPTZzDKO4tAw2SOlmZz/EwrAF6PYVp+WsfcyPklmYwflaLeSBn2FBoVGkek1lAr5Q3lffYW9ixPk5OY0xEEVZ5GZPgKhAPsqtrF+vL1bCjfwO4zbig7NS6VebmLKIifQ6hxCkfKE9l+spaGVjfdJCMprtvcyHlFmWQkaxRBhheFRoXGYclaS3VzdZ/FrKuaqvosT9NzXqHK04iMXHWtdbxR8YbriSxbT3ljOQBFqUUsK1zG+JQFhJsmsK8syLYTtRysrKf9V8rE3JTOuZHFWUwZnYrPq0U2MnQpNCo0Dlk9y9N0XYXcW3ma0cmjz5tX2L4yWeVpRCQSay0n6k90DmWfeoOmYBNe42XWqFksL1zOvFFLCDUXs7O0nm0nzrLtRC1nGv0AJMd7mVOU0TE3cl5JJnlpiTH+qUT6T6FRoXHQCoVDVDZVnt9b2PZc01LT7fz28jQdPYUqTyMiAygQDrCzaqcr61O+sWMoOy0ujSUFS1heuJxlBcuwgRy2nXQBctuJs+wpP0cw7H7vFGUlueHs4kzml2QyszCdBJ+mu8jgpNCo0Bh1zx19jh9s/QEVjRXkp+Rz94K7uWnCTb2e27U8Tc95hT3L03iNl/yU/POKWbcHRJWnEZFYqmut4/VTr3f0RJ5qdBXYitOKXYAsXMaS/CXEmWT2lNe1hUgXJMvr3MhIvNfDjML0bqu1i7KS9N4mg4JCo0JjVD139DnuXX9vt6HhBG8CH5z+QYrSirrNKyytL+Wc/1y3728vT9Nbb6HK04jIUGGt5fi54x0Lat6o6BzKnj1qdkeInDVqFj6Pj9PnWlyAbOuR3FlaS0sgDMCo1PaSP25u5JyiDFISVPJHrjyFRoXGqLrhiRs6/rrujcrTiMhIFAgF2FG1oyNE7jmzB4slLS6NqwquYlnhMpYXLqcorajt/DAHKurZdtL1RG4/UcvRalfVwWNgan5bb2TbsPaEUal4tK+2DDCFRoXGqJrzizndViV39Zf3/IXcpFyVpxGREa+2pZbXK15nQ/kGXit/jYrGCgBK0ko6AuSS/CWkxqd2fM/ZRj/bSzuHtLefrKW+xU3hSUv0dZT7aQ+TmcnxMfnZZPhSaFRojKq+ehoLUgr483v+HIMWiYgMbtZajp071m0ouznYjNd4mZM7pyNEzsyZic/TOSwdDluOVjewtcvcyIOn62lbY8OEUSnM6zI3clp+mkr+yGVRaFRojKre5jQmehO5d/m9fS6GERGRToFQgO1V29lQvoH15evZe2avG8qOT2NpwdKOEDkmdcx539vYGmRnaV231drVDa7kT1Kcl9lFGd12shmdrpI/0n8KjQqNUXcxq6dFROTCaltq2Vix0Q1ll73G6abTAIxNH8uyAhcgF+cv7jaU3c5aS+nZ5o65kdtO1LKnvI5AyP3eG5OZ1FHux5X8ySAxTlOIpHcKjQqNIiIyRFhrefPcmx29kJsqNtEcbMZnfOcNZfc1f7wlEGLvqXMdPZHbTtRSVtsMQJzXMKMgvdtONsXZKvkjjkKjQqOIiAxR/pC/26rs9qHs9Ph0riq4iuWFy1leuJzC1MILXqeyvoXtJ2o7eiR3nKyjORACICclvlvdyDnFmaSq5M+IpNCo0CgiIsPE2Zaz3QqMtw9lj0sfx7LCZSwrWMaSgiWkxKVc8DrBUJiDpxu6zY08UuVK/hgDU0endZsbOTFXJX9GAoVGhUYRERmGrLW8WfdmR4DcfHpzt6Hs9l7IGTkz+lUKra4p0FbyxwXJ7SdrqWsOAJCW4GNul7mR84qzyE5RyZ/hRqFRoVFEREYAf8jP9srtHSFyX80+ANLj01lasLRjl5pIQ9ntwmHLm2ca2wKkC5L7K+oJtdX8GZeT3G1u5LSCNOJU8mdIU2hUaBQRkRGopqWm21B2ZVMl0DmU3b4qO9JQdldN/iC7Sus65kZuPVFLVX0rAAk+D3OKMjrmRs4vySI/QyV/hhKFRoVGEREZ4ay1HK072jmUXbGZllALPuNjbt7cjqHs6dnTL2pXL2st5XUtHUPa206cZXfZOfwht692QUZil5I/Wcweo5I/g5lCo0KjiIhIN/6Qn22V2zpWZbcPZWckZHQOZRcsoyC14KKv3RoMse9Ufbe5kSdqmgDweQzTC9I75kbOL85ibE6ySv4MEgqNCo0iIiIXdKb5TMdQ9obyDVQ2dw5lt/dCLs5fTHJc8iVdv7qhta3kjwuSO07W0uh3JX+ykuO6DWnPKc4gPTEuaj+b9J9Co0KjiIhIv1lrOVJ7xA1ln1rPlootbijb42Ne7ryOEDkte9pFDWV3FQpbDlXWdytAfqiyAXAlfybnpXaU+5lfksWkvFS8Kvkz4BQaFRpFREQuWWuotWMoe2P5xo6h7MyEzG6rsvNT8i/rPnXNAXaW1nYGyZO11Da5kj8p8d7Okj/FWcwryWRUasJl/2zSnUKjQqOIiEjUnGk+w8ZTGzuGsquaqwAYnzG+oxdy0ehFlzyU3c5ay/EzTV0KkNey99S5jpI/JdnJbSHS9UZOL0gn3qeSP5dDoVGhUUREZEBYazlce7gjQG4+vZnWUCs+j4/5efM7eiGnZ0/HYy4/0DX7Q+wur+sY0t564iynz7mSP/E+D7PHZHSEyPklmRRkJGqRzUVQaFRoFBERuSJaQ61sPb2VDeUb2HBqA/tr9gNuKHtZwTK31WEUhrK7OlXX3G1u5K6yOlqDruTP6PSEtpI/bqHN7KIMkuO1r3ZfFBoVGkVERGKiurmajac2sqF8A+vL11PdXA3AhIwJHb2Q0RjK7sofDLO/4lxHuZ9tJ85y7Iwr+eP1GKbld99Xe/yoFPVGtlFoVGgUERGJOWsth2oPdQTILae30BpqJc4Tx/y8+R271EzLnhaVoeyuzjS0sqNjkY0Lkw2tQQAyk+Ncb2RbiJxbnElGUveSP09tK+M7LxygvLaZwswkvrhqKu+cPyaqbRwMFBoVGkVERAadlmALWyu3doTIg2cPApCVkMXSws4C46NTRkf93qGw5UhVQ5edbGo5WFlPe1yYlJfK/OJM5pVkUtsU4IcvHaIlEO74/qQ4L9961+xhFxwVGhUaRUREBr3q5mo3F7ItRJ5pOQPAxIyJHb2QC0cvjOpQdlf1LQF2lnYustl2spaaRn+f54/JTOK1L187IG2JFYVGhUYREZEhxVrLwbMHuw1l+8N+4jxxLMhb0BEip2ZPjfpQdtc2nKxp5q3f+VuvXzfAm/ffNCD3jhWFRoVGERGRIa0l2MLW01tdaZ9TGzqGsrMTs1lasNStyh6goeyr73+Jstrm846rp3F40tp6ERGRISzRl8jyMctZPmY5AFVNVR0FxteXr+f5N58HYFLmpG5D2Um+pMu+9xdXTeUrf9hFcyDUcSwpzssXV0297GvL4KOeRhERkWEqbMMcOnuoI0BuPb21cyh79IKOXWqmZE255KFsrZ4eORQaRURERojmYHO3oexDZw8Bbii7fRh7WeEy8pLzYtzSwUehUaFRRERkxKpsquy2V3ZNSw3ghrLbeyEXjF4QlaHsoU6hUaFRREREcEPZB88e7DaUHQgHiPfEdxvKnpw1udtQ9nNHn+MHW39ARWMF+Sn53L3gbm6aMLxWToNCIyg0ioiISC+ag81sOb2loxfycO1hAHISczr2yW70N/K9Ld+jJdTS8X2J3kTuXX7vsAuOCo0KjSIiItIPpxtPdwxlbzy1sWMouzcFKQX8+T1/voKtG3gKjSq5IyIiIv0wOmU0t066lVsn3UrYhjlQc4D3PvveXs+taKy4wq2TK2FgSsWLiIjIsOUxHqbnTKcgpaDXr+en5F/hFsmVoNAoIiIil+TuBXeT6E3sdizRm8jdC+6OUYtkIGl4WkRERC5J+2KXkbB6WhQaRURE5DLcNOEmhcQRQsPTIiIiIhKRQqOIiIiIRNSv0GiMWW2MOWCMOWyM+XIvX08wxvy27euvG2PGRbuhIiIiIhI7EUOjMcYL/Bi4EZgB3GGMmdHjtDuBs9baScD3gW9Hu6EiIiIiEjv96WlcAhy21h611vqBx4Bbe5xzK/CLto+fAFYaY0z0mikiIiIisdSf0DgGONnl89K2Y72eY60NAnVATjQaKCIiIiKxd0UXwhhj7jLGbDbGbK6qqrqStxYRERGRy9Cf0FgGFHf5vKjtWK/nGGN8QAZwpueFrLUPWmsXWWsX5ebmXlqLRUREROSK609x703AZGPMeFw4fD/wgR7nPAN8FNgAvAd4yVprL3TRLVu2VBtjjl98k4eMUUB1rBshMgLptScSG8P9tTc21g2ItYih0VobNMZ8GngB8AIPW2v3GGPuAzZba58B1gC/MsYcBmpwwTLSdYd1V6MxZrO1dlGs2yEy0ui1JxIbeu0NfyZCh6BcIr14RGJDrz2R2NBrb/jTjjAiIiIiEpFC48B5MNYNEBmh9NoTiQ299oY5DU+LiIiISETqaRQRERGRiBQao8AYEzLGbDfG7DbGrDXGZLYdH2eMaW77WvsjPtbtFRkOurzu9hhjdhhjPm+M8RhjVnV5vTUYYw60ffzLWLdZZLgwxjT0cuxeY0xZ2+ttrzHmjli0TQaOhqejwBjTYK1Nbfv4F8BBa+2/G2PGAc9aa2fFsn0iw1GP110e439TzQAAAYxJREFU8BvgNWvtv3U552XgC9bazbFppcjw1PX11+XYvUCDtfa7xpjJwBYgx1obiEUbJfrU0xh9Gzh/b24RGUDW2krgLuDTxhgT6/aIjHTW2kNAE5AV67ZI9Cg0RpExxgusxO2Q025il6GyH8eoaSLDnrX2KG4DgrxYt0VkpDPGLAAOtf1BJ8NEf7YRlMiSjDHbcT2M+4C/dPnaEWvtvNg0S0RE5Ir6nDHmY8AU4OZYN0aiSz2N0dHcFgzHAgb4lxi3R2TEMcZMAEKAejZEYuf71tqZwLuBNcaYxFg3SKJHoTGKrLVNwGeAzxtj1IsrcoUYY3KBnwA/slrdJxJz1tpngM3AR2PdFokeBZsos9ZuM8bsBO4AXo11e0SGsfZpIXFAEPgV8L3YNklkxEg2xpR2+by31959wG+MMf9trQ1foXbJAFLJHRERERGJSMPTIiIiIhKRQqOIiIiIRKTQKCIiIiIRKTSKiIiISEQKjSIiIiISkUKjiIiIiESk0CgiIiIiESk0ioiIiEhE/z/Z7lCnt4MjyAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "%matplotlib inline\n",
    "from IPython.core.pylabtools import figsize\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "\n",
    "ind = np.arange(results.shape[0])\n",
    "width = 0.2\n",
    "l = ax.plot(ind, results, \"-o\")\n",
    "plt.legend(iter(l), results.columns.tolist(), loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.25, ind[-1]+.25])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  Nevertheless, none of these measures takes into account the \n",
    "  business and economical realities that take place in credit scoring. Costs that the financial \n",
    "  institution had incurred to acquire customers, or the expected profit due to a particular client, \n",
    "  are not considered in the evaluation of the different models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Financial Evaluation of a Credit Scorecard\n",
    "\n",
    "  Typically, a credit risk model is evaluated using standard cost-insensitive measures.\n",
    "  However, in practice, the cost associated with approving \n",
    "  what is known as a bad customer, i.e., a customer who default his credit loan, is quite \n",
    "  different from the cost associated with declining a good customer,  i.e., a customer who \n",
    "  successfully repay his credit loan. Furthermore, the costs are not constant among customers. \n",
    "  This is because loans have different credit line amounts, terms, and even interest rates. Some \n",
    "  authors have proposed methods that include the misclassification costs in the credit scoring \n",
    "  context [4,5,6,7].\n",
    "  \n",
    "  In order to take into account the varying costs that each example carries, we proposed in \n",
    "  [8], a cost matrix with example-dependent misclassification costs as \n",
    "  given in the following table.\n",
    "  \n",
    "  \n",
    "|  \t| Actual Positive ($y_i=1$)  \t|  Actual Negative \t($y_i=0$)|\n",
    "|---\t|:-:\t|:-:\t|\n",
    "|   Predicted Positive ($c_i=1$)\t|   $C_{TP_i}=0$\t|  $C_{FP_i}=r_i+C^a_{FP}$ \t|\n",
    "|  Predicted Negative  ($c_i=0$) \t|   $C_{FN_i}=Cl_i \\cdot L_{gd}$\t| $C_{TN_i}=0$\t|\n",
    "  \n",
    "  First, we assume that the costs of a correct \n",
    "  classification, $C_{TP_i}$ and $C_{TN_i}$, are zero for every customer $i$. We define $C_{FN_i}$ \n",
    "  to be the losses if the customer $i$ defaults to be proportional to his credit line $Cl_i$. We \n",
    "  define the cost of a false positive per customer $C_{FP_i}$ as the sum of two real financial \n",
    "  costs $r_i$ and $C^a_{FP}$, where $r_i$ is the loss in profit by rejecting what would have been a \n",
    "  good customer. \n",
    "  \n",
    "  The profit per customer $r_i$ is calculated as the present value of the difference between the \n",
    "  financial institution gains and expenses, given the credit line $Cl_i$, the term $l_i$ and the \n",
    "  financial institution lending rate $int_{r_i}$ for customer $i$, and the financial institution \n",
    "  of cost funds $int_{cf}$.\n",
    "\n",
    "  $$  r_i= PV(A(Cl_i,int_{r_i},l_i),int_{cf},l_i)-Cl_i,$$\n",
    "  \n",
    "  with $A$ being the customer monthly payment and $PV$ the present value of the monthly payments,\n",
    "  which are calculated using the time value of money equations [9],\n",
    " \n",
    " $$ A(Cl_i,int_{r_i},l_i) =  Cl_i \\frac{int_{r_i}(1+int_{r_i})^{l_i}}{(1+int_{r_i})^{l_i}-1},$$\n",
    " \n",
    " $$ PV(A,int_{cf},l_i) = \\frac{A}{int_{cf}} \\left(1-\\frac{1}{(1+int_{cf})^{l_i}} \\right).$$\n",
    "      \n",
    "  The second term $C^a_{FP}$, is related to the assumption that the financial institution will not \n",
    "  keep the money of the declined customer idle. It will instead give a loan to an alternative \n",
    "  customer [10]. Since no further information is known about the alternative customer, \n",
    "  it is assumed to have an average credit line $\\overline{Cl}$ and an average profit $\\overline{r}$.\n",
    "  Given that, \n",
    "  \n",
    "  $$  C^a_{FP}=- \\overline{r} \\cdot \\pi_0+\\overline{Cl}\\cdot L_{gd} \\cdot \\pi_1,$$\n",
    "\n",
    "  in other words minus the profit of an average alternative customer plus the expected loss, \n",
    "  taking into account that the alternative customer will pay his debt with a probability equal to \n",
    "  the prior negative rate, and similarly will default with probability equal to the prior positive \n",
    "  rate.\n",
    "  \n",
    "  One key parameter of our model is the credit limit. There exists several strategies to calculate \n",
    "  the $Cl_i$ depending on the type of loans, the state of the economy, the current portfolio, \n",
    "  among others [1,9]. Nevertheless, given the lack of information \n",
    "  regarding the specific business environments of the considered datasets, we simply define \n",
    "  $Cl_i$ as\n",
    "\n",
    "$$      Cl_i = \\min \\bigg\\{ q \\cdot Inc_i, Cl_{max}, Cl_{max}(debt_i) \\bigg\\},$$\n",
    "  \n",
    "  where $Inc_i$ and $debt_i$ are the monthly income and debt ratio of the customer $i$, \n",
    "  respectively, $q$ is a parameter that defines the maximum $Cl_i$ in times $Inc_i$, and \n",
    "  $Cl_{max}$ the maximum overall credit line. Lastly, the maximum credit line given the current \n",
    "  debt is calculated as the maximum credit limit such that the current debt ratio plus the new \n",
    "  monthly payment does not surpass the customer monthly income. It is calculated as\n",
    " \n",
    " $$  Cl_{max}(debt_i)=PV\\left(Inc_i \\cdot P_{m}(debt_i),int_{r_i},l_i\\right),$$\n",
    "  and\n",
    "  $$ P_{m}(debt_i)=\\min \\left\\{ \\frac{A(q \\cdot Inc_i,int_{r_i},l_i)}{Inc_i},\\left(1-debt_i \\right) \\right\\}.$$\n",
    "  \n",
    "  \n",
    "### Financial savings\n",
    "\n",
    "  Let $\\mathcal{S}$ be a set of $N$ examples $i$, $N=\\vert S \\vert$, where each example is \n",
    "  represented by  the augmented feature vector \n",
    "  $\\mathbf{x}_i^*=[\\mathbf{x}_i, C_{TP_i},C_{FP_i},C_{FN_i},C_{TN_i}]$  \n",
    "  and labeled using the class   label $y_i   \\in \\{0,1\\}$. \n",
    "  A classifier $f$ which generates the   predicted label $c_i$ for each   element $i$ is trained  \n",
    "  using the set $\\mathcal{S}$. Then the cost of   using $f$ on $\\mathcal{S}$ is calculated by\n",
    "  \n",
    "  $$   Cost(f(\\mathcal{S})) = \\sum_{i=1}^N Cost(f(\\mathbf{x}_i^*)),$$\n",
    "  \n",
    "  where\n",
    "  \n",
    " $$   Cost(f(\\mathbf{x}_i^*)) = y_i(c_i C_{TP_i} + (1-c_i)C_{FN_i}) + (1-y_i)(c_i C_{FP_i} + (1-c_i)C_{TN_i}).$$\n",
    "  \n",
    "\n",
    "  However, the total cost may not be easy to interpret. We proposed an approach in [8], where the savings of using an algorithm  are defined as the cost of the algorithm versus the cost of using no algorithm at all.  To do that, the cost of the costless class is defined as \n",
    "  \n",
    "  $$  Cost_l(\\mathcal{S}) = \\min \\{Cost(f_0(\\mathcal{S})), Cost(f_1(\\mathcal{S}))\\},$$\n",
    "  \n",
    "  where \n",
    "  \n",
    "  $$  f_a(\\mathcal{S}) = \\mathbf{a}, \\text{ with } a\\in \\{0,1\\}.$$\n",
    "  \n",
    "\n",
    "  The cost improvement can be expressed as the cost savings as compared with $Cost_l(\\mathcal{S})$. \n",
    "  \n",
    "  $$    Savings(f(\\mathcal{S})) = \\frac{ Cost_l(\\mathcal{S}) - Cost(f(\\mathcal{S}))}   {Cost_l(\\mathcal{S})}.$$\n",
    "  \n",
    "\n",
    "\n",
    "  ### Parameters for the Kaggle Credit Database\n",
    "\n",
    " As this database contain information regarding the features, and more importantly about the income of each example, from which an estimated credit limit $Cl_i$ can be calculated.\n",
    "Since no specific information regarding the datasets is provided, we assume that they belong to  an\n",
    "average Portuguese financial institution. This enabled us to find the different \n",
    "parameters needed to calculate the cost measure. \n",
    "\n",
    "| Parameter \t| Value |\n",
    "|---\t|:-:\t|\n",
    "|Interest rate ($int_r$) | 4.79% |\n",
    "|  Cost of funds ($int_{cf}$) | 2.94% |\n",
    "|  Term ($l$) in months | 24 |\n",
    "|  Loss given default ($L_{gd}$) | 75% |\n",
    "|  Times income ($q$) | 3 |\n",
    "|  Maximum credit line ($Cl_{max}$) | 25,000|\n",
    "\n",
    "In particular, we obtain the average interest rates in Europe during   2013 from the European Central Bank [11]. Additionally, we use a fixed loan term $l$ to two years,\n",
    "considering that in the Kaggle Credit dataset the class was constructed to predict two years of credit behavior.\n",
    "Moreover, we set the loss given default $L_{gd}$ using information from \n",
    "the Basel II standard, $q$ to 3 since it is the average personal loan requests related to monthly income, and the maximum credit limit $Cl_{max}$ to 25,000 Euros.\n",
    "\n",
    "### Calculation of the savings of the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1023.73054104 18750.             0.             0.        ]\n",
      " [  717.25781516  6749.25           0.             0.        ]\n",
      " [  866.65393177 12599.25           0.             0.        ]]\n"
     ]
    }
   ],
   "source": [
    "# The cost matrix is already calculated for the dataset\n",
    "# cost_mat[C_FP,C_FN,C_TP,C_TN]\n",
    "print (data.cost_mat[[10, 17, 50]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          f1       pre       rec       acc       sav\n",
      "RF  0.234356  0.509532  0.152174  0.931949  0.129533\n",
      "DT  0.248201  0.238436  0.258799  0.892699  0.184151\n",
      "LR  0.071775  0.569231  0.038302  0.932197  0.028994\n"
     ]
    }
   ],
   "source": [
    "# Calculation of the cost and savings\n",
    "from costcla.metrics import savings_score, cost_loss \n",
    "\n",
    "# Evaluate the savings for each model\n",
    "results[\"sav\"] = np.zeros(results.shape[0])\n",
    "for model in classifiers.keys():\n",
    "    results[\"sav\"].loc[model] = savings_score(y_test, classifiers[model][\"c\"], cost_mat_test)\n",
    "\n",
    "# TODO: plot results\n",
    "print (results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqoAAAEvCAYAAACNEOOcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhV5b328fu3dybCkAQIBRlFiQwySQQBRQSZXmzV9nicsSSIiAxV1OLEQThaKVoEBJkd20qrb89r1RoQB7RWJYgooEGgIiAgYwIhCUn28/5B8IQQIIGEtbLz/VzXvrLXs4bc22uDN8/eay1zzgkAAADwm4DXAQAAAIDSUFQBAADgSxRVAAAA+BJFFQAAAL5EUQUAAIAvUVQBAADgSxFeByipfv36rkWLFl7HAAAAOKWVK1fuds4lep0jXPmuqLZo0ULp6elexwAAADglM9vsdYZwxkf/AAAA8CWKKgAAAHyJogoAAABfoqgCAADAlyiqAAAA8KUyFVUzG2hmGWa2wczGl7L+HjNbZ2ZfmtkyM2tebF2hmX1R9Hi9IsMDAAAgfJ3y8lRmFpQ0S1I/SVslrTCz151z64pttkpSsnPukJndKen3kq4vWpfjnOtUwbkBAAAQ5soyo9pV0gbn3Cbn3GFJr0i6uvgGzrn3nHOHihY/kdSkYmMCAACguilLUW0saUux5a1FYyeSKukfxZZjzCzdzD4xs2tOIyMAnLY3N72p/q/2V4cXOqj/q/315qY3vY4EACijCr0zlZndIilZ0uXFhps757aZWUtJ75rZV865jSX2Gy5puCQ1a9asIiMBqMbe3PSmJn48UbmFuZKk7dnbNfHjiZKkwS0He5gMAFAWZSmq2yQ1LbbcpGjsGGZ2paSHJF3unMs7Ou6c21b0c5OZvS+ps6Rjiqpzbp6keZKUnJzsyvcSAFRnOQU52p+7X/vy9v3vz7z92p+3Xy+uffGnknpUbmGupn8+naIKAFVAWYrqCkmtzOxcHSmoN0i6qfgGZtZZ0lxJA51zPxYbT5B0yDmXZ2b1JfXUkROtAOA4eYV52pe7T5l5mccWzxI/i68vWUSPMpmcSv93747sHZX5MgAAFeSURdU5V2BmoySlSQpKWuScW2tmkySlO+delzRVUi1JfzUzSfreOfcLSW0kzTWzkI58H/aJElcLABCm8gvztT/v1EWz+M+cgpwTHq92VG0lRCcoPiZeDWIbKCkhSfHR8YqPif9pvPjPOlF1NOj/DtL27O3HHatejXqV+dIBABXEnPPXJ+3JyckuPT3d6xgAiskP5SszL/O4j9ZPNOO5P2+/svOzT3i8WpG1FB8dr4SYBMVFx5VaNIuvj4uOU2Qgsty5S35H9agawRp66f+8pAvqXlDuYwJAcWa20jmX7HWOcFWhJ1MB8L/CUKEyD2eesmgWX3/g8IETHi82IvaYmc3mcc2PK5olC2hksPyl83Qc/R7q9M+na0f2DjWs2VA3t7lZL617SalLUrWg/wK1rtv6rGQBAJQfM6pAFRZyIR04fED7co+Uy59+Hi2bxQro0fGsvKwTfnczOhithJiEnwpl8XJZ2oxnfEy8ooPRZ/lVn7ktB7YoNS1V2fnZmt9/vtrWa+t1JABVFDOqlYuiCviEc04H8g8cVy6PKaDFi2jufmUezlTIhUo9XmQg8n9LZ0z8kWJZNMsZH1166awRUeMsv2rvbD2wValpqTqQf0Dz+81Xu/rtvI4EoAqiqFYuiipQCZxzys7PPmXRLL4+My9TBa6g1ONFBCJOXjSLimjxsRoRNVR0ciNOYNvBbUpNS1VWXpbm9pur9ontvY4EoIqhqFYuiipwCs65I9fqPNkZ7CUK6L68fSoIlV46gxYs9QSikxXRmpE1KZ2VZPvB7UpJS9H+vP2a22+uOiR28DoSgCqEolq5KKqodnILcn+axTyuaJZSODPzMpVXmFfqsUx2zPc346LjTjrjGR8dr9pRtRWwsty9GGfLjuwdSklL0d7cvZpz5Rx1atDJ60gAqgiKauWiqKJKO1x4+IQFs7QZz/15+096rc46UXVOWDRLK6K1o2orGAiexVeMyrIze6dSl6Rq16FdmtNvjjo36Ox1JABVAEW1clFU4RtHr9VZcnbzRN/t3Je7T4cKDp3weLUja5/wO5ylFdE6UXUUEeCKbdXZj4d+VGpaqnYe2qlnr3xWXX7WxetIAHyOolq5KKrV2Jub3jzm+pJjLxpbYfc/LwgVHLlAfMkLw5dyYtHR9QfyT3ytzpqRNX8ql3Exccdcp7O073We7gXigV2Hdil1Sap2ZO/QrL6zdHHDi72OBMDHKKqVi6JaTZV2x56YYIwm9ph4XFktDBUeuVbnCYrmMfdmL1rOOpx1wt9dI6LG8ScOlSiaJddHBaMq7b8FUNLunN0aljZMP2T/oGf6PKOujbp6HQmAT1FUKxdFtZrq/2r/Uu+BXiOihi5pdMkxRTQzL/OEF4iPCkQduVbnKYrm0Z/x0fGKiYip7JcHnLHdObt1+5LbtfXAVs3sO1OXNLrE60gAfIiiWrn4Ql41tSN7R6njOQU52nZwm+Kj45WUkHTKGU+u1YlwVb9GfS3ov0C3L71do5aN0ow+M9TjnB5exwKAaoWiWk01rNmw1BnVRjUb6bVfvOZBIsB/6tWop4X9F2rYkmEavWy0ZvSZoZ6Ne3odCwCqDS7mWE2NvWisYoLHfgQfE4zR2IvGepQI8KeEmAQt7L9QLeNbasy7Y/Th1g+9jgQA1QZFtZoa3HKwJvaYqEY1G8lkalSzUaknUgGQ4mPitaD/Ap0Xf57GvjdWy7cu9zoSAFQLnEwFAGWUmZep4UuHa/2+9ZrWe5p6N+3tdSQAHuNkqsrFjCoAlFFcdJzm95+v1gmtdff7d+vd79/1OhIAhDWKKgCUQ52oOprXf57a1m2rce+P0zub3/E6EgCELYoqAJRT7ajamttvrtrVb6d7P7hXS75b4nUkAAhLFFUAOA21omppbr+56pDYQfcvv19vf/e215EAIOxQVAHgNNWMrKlnr3xWHRM7avzy8frHv//hdSQACCsUVQA4A0fLaqcGnTT+w/F6Y9MbXkcCgLBBUQWAMxQbGavZfWcr+WfJeuijh/T3jX/3OhIAhAWKKgBUgNjIWD3T9xld3PBiPfTRQ/p/G/6f15EAoMqjqAJABakRUUPP9HlGlzS6RI/88xH97du/eR0JAKo0iioAVKCYiBjN6DND3c/prgkfT9Br61/zOhIAVFkUVQCoYEfL6qWNL9XEf03UXzL+4nUkAKiSKKoAUAmig9GafsV09WrSS5M/mazF3yz2OhIAVDkUVQCoJFHBKE3rPU29m/TWf3/63/rzN3/2OhIAVCkUVQCoRFHBKP2h9x90RdMr9Pinj+uPX//R60gAUGVQVAGgkkUGI/XU5U+pb7O+euKzJ/Ti2he9jgQAVQJFFQDOgshgpKZePlX9mvfT1PSpen7N815HAgDfi/A6AABUF5GBSE3pNUWBDwN6auVTCimklAtTvI4FAL5FUQWAsygyEKknLntCAQU0beU0hVxIw9oP8zoWAPgSRRUAzrKIQIQev+xxmZmmfz5dhaFC3dHxDq9jAYDvUFQBwAMRgQg9funjClpQz3zxjEIupDs73el1LADwFYoqAHgkGAhqcs/JClhAs1fPVkghjew4UmbmdTQA8AWKKgB4KBgIalLPSQpYQHNWz1HIhTSq0yjKKgCIogoAngtYQBN7TFTAApr35TyFXEhjOo+hrAKo9iiqAOADAQtoQvcJClhAC75aoEJXqLsvupuyCqBao6gCgE8ELKCHL3lYAQvouTXPKRQKaVzyOMoqgGqLogoAPhKwgB7q9pACFtAL615QSCHdl3wfZRVAtURRBQCfMTM90PUBBS2ol9a9JOec7r/4fsoqgGonUJaNzGygmWWY2QYzG1/K+nvMbJ2ZfWlmy8ysebF1t5nZt0WP2yoyPACEKzPT/Rffr1vb3qqXv35Zv/vsd3LOeR0LAM6qU86omllQ0ixJ/SRtlbTCzF53zq0rttkqScnOuUNmdqek30u63szqSvovScmSnKSVRfvuq+gXAgDhxsx0X/J9ClpQz699XiEX0oPdHlTAyjTHAABVXlk++u8qaYNzbpMkmdkrkq6W9FNRdc69V2z7TyTdUvR8gKSlzrm9RfsulTRQ0p/PPDoAhD8z0z1d7lHAAlq0ZpEKXaEeueQRyiqAaqEsRbWxpC3FlrdK6naS7VMl/eMk+zYuT0AAqO7MTL+56DcKWlDzv5ov59xPl7ICgHBWoSdTmdktOvIx/+Xl3G+4pOGS1KxZs4qMBABhwcw0uvNomdlPNwU4epMAAAhXZSmq2yQ1LbbcpGjsGGZ2paSHJF3unMsrtm/vEvu+X3Jf59w8SfMkKTk5mbMFAKAUZqZRnUYpaEE9u/pZFbpCTeoxScFA0OtoAFApylJUV0hqZWbn6kjxvEHSTcU3MLPOkuZKGuic+7HYqjRJj5tZQtFyf0kPnHFqAKimzEwjO42UmWn2F7PlnNPknpMpqwDC0imLqnOuwMxG6UjpDEpa5Jxba2aTJKU7516XNFVSLUl/LbrO3/fOuV845/aa2WQdKbuSNOnoiVUAgNN3Z8c7FbSgZq6aqUJXqMcufUwRAS6NDSC8lOlvNefcW5LeKjE2odjzK0+y7yJJi043IACgdMM7DFfAApr++XQ55/T4ZY9TVgGEFf5GA4AqbFj7YQpYQNNWTlNIIf3ust8pMhDpdSwAqBAUVQCo4lIuTFHQgnoy/UmFXEhTek2hrAIICxRVAAgDt7W7TSbT1PSpCn0Q0tReUxUZpKwCqNq4AB8AhIkh7YZofNfxWvb9Mo37YJzyC/O9jgQAZ4SiCgBh5OY2N+vBbg/qvS3v6Z7379HhwsNeRwKA00ZRBYAwc2PrG/Vwt4f1/tb3dff7dyuvMO/UOwGAD1FUASAMXd/6ek3oPkHLty7X2PfGUlYBVEkUVQAIU9clXaeJ3Sfq420fa8y7Y5RbkOt1JAAoF4oqAISxXyX9So/2eFT/+uFfGv3uaOUU5HgdCQDKjKIKAGHu2lbXanLPyfp0+6cavYyyCqDqoKgCQDVw9flX67FLH9OKnSt017K7dCj/kNeRAOCUKKoAUE38/Lyf6/FLH9fKnSs1ctlIyioA3+POVEA5tBj/ptcRcJq+e2Kw1xF8YXDLwQpYQA98+IDufOdOzb5ytmpG1vQ6FgCUihlVAKhmBp07SFN6TdHqXas1YukIHTx80OtIAFAqiioAVEMDWgzQ1Munas3uNRrxzggdOHzA60gAcByKKgBUU/2a99OTlz+ptbvXasRSyioA/6GoAkA11rd5Xz3V+ymt27tOw5cMV9bhLK8jAcBPKKoAUM31adZH03pPU8a+DN2+5HZl5mV6HQkAJFFUAQCSejftraeveFrf7vuWsgrANyiqAABJUq8mvTT9iunauH+jhi0Zpv25+72OBKCao6gCAH5yWZPLNKPPDP07899KXZKqvbl7vY4EoBqjqAIAjtGzcU/N7DNTm7M2KzUtVXty9ngdCUA1RVEFAByn+zndNavvLG09sFWpaananbPb60gAqiGKKgCgVN0addPsK2frh+wfKKsAPEFRBQCc0MUNL9asvrO0PXu7UtJStOvQLq8jAahGKKoAgJO6uOHFevbKZ7Uze6dS0lK0M3un15EAVBMUVQDAKXX5WRfN6TdHu3J2KSUtRTuyd3gdCUA1QFEFAJRJ5wadNefKOdqTu4eyCuCsoKgCAMqsU4NOmtdvnvbl7tPQt4dq+8HtXkcCEMYoqgCAcumQ2EHz+s1TZl6mhqYN1baD27yOBCBMUVQBAOXWPrG95vefr6zDWRr69lBtPbDV60gAwhBFFQBwWtrVb6cF/RcoOz9bQ9OGasuBLV5HAhBmKKoAgNPWtl5bLRywUDkFORr69lB9n/W915EAhBGKKgDgjLSu21oL+y9UXmGehqYN1easzV5HAhAmKKoAgDN2Qd0LtHDAQhWECjT07aH6d+a/vY4EIAxQVAEAFSIpIUkL+y9UoStUSlqKNu3f5HUkAFUcRRUAUGHOTzhfiwYsknNOKWkp2rh/o9eRAFRhFFUAQIU6L/48LRq4SGamlLQUfbvvW68jAaiiKKoAgArXMq6lFg1YpKAFlZqWqvX71nsdCUAVRFEFAFSKc+PO1XMDn1NkMFKpaanK2JvhdSQAVQxFFQBQaZrXaa7nBjyn6GC0Upek6us9X3sdCUAVQlEFAFSqZnWa6bmBzyk2IlbDlgzTuj3rvI4EoIqgqAIAKl3T2k21aMAi1YqspWFLhmnt7rVeRwJQBVBUAQBnRZPaTbRo4CLViaqj25fcrq92feV1JAA+V6aiamYDzSzDzDaY2fhS1vcys8/NrMDM/qPEukIz+6Lo8XpFBQcAVD2NazXWcwOeU1x0nIYvHa7Vu1Z7HQmAj52yqJpZUNIsSYMktZV0o5m1LbHZ95J+LelPpRwixznXqejxizPMCwCo4hrVaqTnBj6nhJgE3bH0Dn3x4xdeRwLgU2WZUe0qaYNzbpNz7rCkVyRdXXwD59x3zrkvJYUqISMAIMw0rNlQzw14TvVr1NcdS+/Qqh9XeR0JgA+Vpag2lrSl2PLWorGyijGzdDP7xMyuKW0DMxtetE36rl27ynFoAEBV9bOaP9OiAYvUILaB7lh6h1buXOl1JAA+czZOpmrunEuWdJOkp83svJIbOOfmOeeSnXPJiYmJZyESAMAPGsQ20KIBi9SwZkPd+c6dWrFjhdeRAPhIWYrqNklNiy03KRorE+fctqKfmyS9L6lzOfIBAMJcYmyiFg1YpHNqnqO7lt2lz7Z/5nUkAD5RlqK6QlIrMzvXzKIk3SCpTGfvm1mCmUUXPa8vqackrvQMADhG/Rr1tXDAQjWu1Vh3LbtLn2z/xOtIAHwg4lQbOOcKzGyUpDRJQUmLnHNrzWySpHTn3OtmdrGkv0lKkPRzM3vUOddOUhtJc80spCOl+AnnHEUVAHCcejXqaeGAhRq2ZJhGLRulGX1mqMc5PbyOBZTbypUrG0RERCyQdKG4Zv2phCStKSgoGNalS5cfS648ZVGVJOfcW5LeKjE2odjzFTrylYCS+30sqX15EwMAqqe6MXW1sP+Rsjp62WjN6DNDPRv39DoWUC4RERELGjZs2CYxMXFfIBBwXufxs1AoZLt27Wq7Y8eOBZKOu4wpLR8A4CsJMQla2H+hWsa31Jh3x+jDrR96HQkorwsTExOzKKmnFggEXGJiYqaOzD4fv/4s5wEA4JTiY+K1oP8CnRd/nsa+N1YfbPnA60hAeQQoqWVX9N+q1E5KUQUA+FJcdJzm95+vVgmt9Jv3f6P3t7zvdSQAZxlFFQDgW0fLauuE1rr7/bv17vfveh0JqBKCwWCX1q1btz36yMjIiNqxY0ewW7duSbGxsZ2HDBnSrPj2Tz/9dL2kpKS2SUlJbVu1atXu5Zdfjvcqe3FlOpkKAACv1Imqo3n952nE0hEa9/44Tb18qq5sfqXXsYAKszhjcd05q+c03pOzJ6pejXqHR3Qcse36C67feybHjI6ODn3zzTfHXGkpKysrMGnSpB9Wr15dY82aNTWOjm/cuDHyqaeeavTFF198Xa9evcLMzMzA9u3bz6gj5ufnKzIy8kwOIYkZVQBAFVA7qrbm9purdvXb6d4P7tWS75Z4HQmoEIszFtf9/YrfN9+dszvKyWl3zu6o36/4ffPFGYvrVvTvqlOnTmjAgAEHY2JiQsXHt2/fHlmzZs1QXFxcoSTFxcWFWrdufViS1qxZE92jR4+kCy64oG3btm3brF27NjoUCumOO+5o0qpVq3ZJSUlt58+fnyBJb7zxRu0uXbpc0KdPn/NbtWp1oSTNnj27bvv27du0bt267U033dS8oKCgXJmZUQUAVAm1omppbr+5uvOdO3X/8vsVUkgDWwz0OhZwUo/885GmG/ZtiD3R+m/2fVOzIFRgxccOFx4OPPHZEy3+59v/KfW+8ucnnH9ocs/JW072e/Py8gKtW7duK0lNmzbNW7p06cYTbXvJJZccql+/fn7Tpk3b9+zZ88Avf/nLfTfddFOmJN10003n3nvvvTuGDBmy/9ChQ1ZYWGgvvvhi/FdffVXj66+/Xrt9+/aIrl27tunfv/9BSVq3bl3sqlWr1rZu3frw559/HvPqq6/WTU9P/yY6OtrdcsstzebMmVNv1KhRe06WvTiKKgCgyqgZWVPPXvmsRr4zUuOXj5dzToPOHeR1LOC0lSyppxovq9I++j+RiIgILV++/NsPPvggdsmSJXXGjx/fND09veYjjzyyY+fOnVFDhgzZL0mxsbFOkvvwww9r/+d//ufeiIgINW3atKBbt24HP/roo9i4uLhQhw4dso/Oxr799tu116xZE9uxY8c2kpSbmxto0KBBuaZUKaoAgCrlp7K6bKTGfzheha5QV7W8yutYQKlONfN5xV+uaL87Z3dUyfH6Neof/vNVf86ovGTHCgQCuuKKKw5dccUVhwYNGpQ1bNiwFo888siO8h4nNjb2p68VOOfsuuuu2zNr1qxtp53rdHcEAMArsZGxmt13tpJ/lqyHPnpIf9/4d68jAadlRMcR26KCUcd8ZzQqGBUa0XHEaZe78vruu+8iP/roo5++npCenh7buHHjwwkJCaGGDRsefumll+IlKScnxw4cOBDo1avXgVdffbVuQUGBfvjhh4jPPvus1mWXXZZd8rgDBw7MeuONNxK2bdsWIUk7d+4Mrl+//rhSfjLMqAIAqqTYyFg90/cZjX53tB766CEVukJdc/41XscCyuXo2f0Vfdb/iTRu3Lj9wYMHg/n5+ZaWlhb/1ltvra9du3bo3nvvbbJz587I6OhoV7du3fz58+d/L0kvv/zyv2+//fbmkydPPicyMtL99a9/3Xjrrbfu//jjj2u1adOmnZm5Rx99dGuzZs0Kvvzyy2N+V5cuXXIffvjhbX379k0KhUKKjIx0M2bM+D4pKelwWfOac/66cUJycrJLT0/3OgZQqhbj3/Q6Ak7Td08M9joCKkluQa7GvDtGn2z/RI/2eFTXtrrW60ioRsxspXMuufjY6tWrv+vYseNurzJVRatXr67fsWPHFiXH+egfAFClxUTEaEafGepxTg9N+HiCXlv/mteRAFQQiioAoMqLiYjR9D7TdWnjSzXxXxP1l4y/eB0JQAWgqAIAwkJ0MFrTr5iuXk16afInk/XKN694HQnAGaKoAgDCRlQwStN6T1PvJr312KeP6U9f/8nrSADOgO/O+v9qWyYnrJxlnGQCIJxEBaP0h95/0LgPxul3n/1OIRfSLW1v8ToWgNPAjCoAIOxEBiP11OVPqW+zvpqyYopeXPui15EAnAbfzagCAFARIoORmnr5VP12+W81NX2qQi6kX1/4a69joRpqMf7NLhV5vO+eGLyyLNv99re/bfjaa6/VCwQCLhAIaPbs2Zv79Olz3IX5T+b6669vfv/99+/s0qVL7umlPTMUVQBA2IoMRGpKrykKfBjQUyufUqErVGr7VK9jAZXunXfeqZmWlhb/1VdfratRo4bbvn17RF5enpX3OIsXL95cGfnKio/+AQBhLTIQqScue0KDzh2kpz9/Wgu+WuB1JKDSbdu2LbJu3boFNWrUcJLUqFGjghYtWuTfe++9jS688MI2rVq1anfjjTc2D4VCWrVqVUz79u3bHN03IyMjKikpqa0kde3a9YLly5fHSlJsbGzn0aNHN77gggvaduzYsfWWLVsiJGnt2rXRHTt2bJ2UlNR2zJgx58TGxnaWpM2bN0cmJydf0Lp167atWrVq9/bbb9cq7+ugqAIAwl5EIEKPX/q4BrccrOmfT9fc1XO9jgRUqmuuuSbrhx9+iGrRosWFt9xyS7M333yzliTdd999P65Zs+brb7/9dm1OTk7glVdeievcuXNufn6+ffPNN1GS9OKLL9a95ppr9pU8Zk5OTqB79+4HMzIy1nXv3v3gzJkzEyVp1KhRTUeOHPnj+vXr1zVp0iT/6PaLFi2q27dv38xvvvlm3ddff722W7duh8r7OiiqAIBqISIQocd6Pqaft/y5nvniGT37xbNeRwIqTVxcXGjNmjXrnnnmmc2JiYkFt91223kzZsyo949//KN2hw4dWiclJbX9+OOPa69Zs6aGJF1zzTV7X3zxxbqS9Le//S3h1ltv3VvymJGRke6GG27IlKQuXbpkb968OUqSVq1aVSslJWWvJA0bNmzP0e0vueSS7D//+c/177nnnnM+++yzGgkJCaHyvg6+owoAqDaCgaAm95ysgAU0e/VshRTSyI4jZVbur+4BvhcREaGrrrrqwFVXXXWgQ4cOOfPnz6+fkZER++mnn647//zz8++5555zcnNzA5J066237rvuuuta3nDDDfvMTO3bt88r5XguEAj8dOyCgoKT/sEZNGjQweXLl2e89tprcSkpKeeOGjVq56hRo/acbJ+SmFEFAFQrwUBQk3pO0rXnX6s5q+do5qqZcs55HQuoUKtXr47+6quvoo8ur1q1qsb555+fJ0kNGzYsyMzMDPz9739POLq+Xbt2eYFAQBMmTDjn2muvPW429WQ6dep08Pnnn0+Qjnzcf3R8/fr1UU2aNMkfN27c7iFDhuz6/PPPY8v7OphRBQBUOwELaGKPiQpYQPO/mi8npzGdxzCzikpR1stJVaSsrKzgmDFjmmVlZQWDwaBr0aJF3gsvvLA5Pj6+oE2bNu0SExMLOnbseMylqn75y1/unTx5cpMpU6ZsK8/vmjlz5pabb7753KlTpzbq06dPVq1atQolKS0trfaMGTMaRkREuNjY2MI//vGP/y7v6zC//SsyulEr1+i2p72OUa1wZ6qy465pVRfvc5Qm5EL670/+W39d/1cNvXCo7r7obsoqysXMVjrnkouPrV69+ruOHTvu9irT2XbgwIFAzZo1Q4FAQPPmzUtYvHhx3WXLlm0szzFWr15dv2PHji1KjjOjCgCotgIW0MOXPKyABfTcmucUCoU0LnkcZRUoh3/+85+xY8eObeacU506dQqff/757yrq2BRVAEC1FrCAHur2kLUQesUAAA30SURBVAIW0AvrXlChK9T9F99PWQXKaODAgQczMjLWVcaxKaoAgGrPzPRA1wcUtKBe/vplOTn99uLfUlZxukKhUMgCgYC/vl/pU6FQyCSVeukqiioAADpSVo/OpL607iUVhgr1YLcHKas4HWt27drVNjExMZOyenKhUMh27doVJ2lNaespqgAAFDEz3Zd8n4IW1PNrn5eT04PdHlTAuJojyq6goGDYjh07FuzYseNCcSnQUwlJWlNQUDCstJUUVQAAijEz3dPlHgUsoEVrFqnQFeqRSx6hrKLMunTp8qOkX3idIxxQVAEAKMHM9JuLfqOgBY9cZ9U5Teg+gbIKnGUUVQAASmFmGt15tAIW0Nwv56rQFerRHo9SVoGziKIKAMAJmJlGdR6lgAX07OpnFXIhTeoxScFA0OtoQLVAUQUA4BRGdhopM9PsL2bLOafJPSdTVoGzgKIKAEAZ3NnxTgUtqJmrZqrQFeqxSx9TRID/jQKViT9hAACU0fAOwxWwgKZ/Pl3OOT1+2eOUVaAS8acLAIByGNZ+mAIW0LSV01ToCvVErycUGYj0OhYQliiqAACUU8qFKQpaUE+mPym33GlKrymUVaASUFQBADgNt7W7TSbT1PSpCn0Q0tReUxUZpKwCFYmLwQEAcJqGtBui8V3Ha9n3yzTug3HKL8z3OhIQVspUVM1soJllmNkGMxtfyvpeZva5mRWY2X+UWHebmX1b9LitooIDAOAHN7e5WQ92e1DvbXlPd79/tw4XHvY6EhA2TllUzSwoaZakQZLaSrrRzNqW2Ox7Sb+W9KcS+9aV9F+SuknqKum/zCzhzGMDAOAfN7a+UQ93e1gfbP1Av3nvN8orzPM6EhAWyjKj2lXSBufcJufcYUmvSLq6+AbOue+cc19KCpXYd4Ckpc65vc65fZKWShpYAbkBAPCV61tfrwndJ+jDbR9q7HtjKatABShLUW0saUux5a1FY2VxJvsCAFClXJd0nSZ2n6iPt32sMe+OUW5BrteRgCrNFydTmdlwM0s3s/TCQ5lexwEA4LT9KulXerTHo/rXD//S6HdHK6cgx+tIQJVVlqK6TVLTYstNisbKokz7OufmOeeSnXPJwdi4Mh4aAAB/urbVtZrcc7I+3f6pRi+jrAKnqyxFdYWkVmZ2rplFSbpB0utlPH6apP5mllB0ElX/ojEAAMLa1edfrccufUwrdq7QXcvu0qH8Q15HAqqcUxZV51yBpFE6UjC/lvQX59xaM5tkZr+QJDO72My2SrpO0lwzW1u0715Jk3Wk7K6QNKloDACAsPfz836uxy99XCt3rtTIZSMpq0A5lenOVM65tyS9VWJsQrHnK3TkY/3S9l0kadEZZAQAoMoa3HKwAhbQAx8+oDvfuVOzr5ytmpE1vY4FVAm+OJkKAIBwNujcQZrSa4pW71qtEUtH6ODhg15HAqoEiioAAGfBgBYDNPXyqVqze43ueOcOHTh8wOtIgO9RVAEAOEv6Ne+nJy9/Uut2r9OIpSMoq8ApUFQBADiL+jbvq6d6P6V1e9dp+JLhyjqc5XUkwLcoqgAAnGV9mvXRtN7TlLEvQ7cvuV2ZedzsBigNRRUAAA/0btpbT1/xtL7d961uX3K79ufu9zoS4DsUVQAAPNKrSS/N6DNDG/dv1LAlw7Qvd5/XkQBfoagCAOChSxtfqhl9Zui7rO80bMkw7c3lvjjAURRVAAA81rNxT83sM1ObszYrNS1Ve3L2eB0J8AWKKgAAPtD9nO6a1XeWth7YqtS0VO3O2e11JMBzFFUAAHyiW6Numn3lbP2Q/QNlFRBFFQAAX7m44cWa3Xe2tmdvV0painYd2uV1JMAzFFUAAHwmuWGynr3yWe3M3qmUtBTtzN7pdSTAExRVAAB8qMvPumhOvznalbNLKWkp2pG9w+tIwFlHUQUAwKc6N+isOVfO0Z7cPZRVVEsUVQAAfKxTg06a12+e9uXu06/f/rV+OPiD15GAs4aiCgCAz3VI7KB5/eYpKy9LKWkp2nZwm9eRgLOCogoAQBXQPrG95vefr6zDWRr69lBtPbDV60hApaOoAgBQRbSr304L+i9Qdn62hqYN1ZasLV5HAioVRRUAgCqkbb22WjhgoXILcjU0bai+z/re60hApaGoAgBQxbSu21oL+i/Q4cLDGvr2UG3O2ux1JKBSUFQBAKiCLqh7gRYMWKACV6Ab37hRff7SRx1e6KD+r/bXm5ve9DoeUCEoqgAAVFFJCUka0naIDuQf0K6cXXJy2p69XRM/nkhZRVigqAIAUIUtzlh83FhuYa6mfz7dgzRAxaKoAgBQhZ3oblXcxQrhgKIKAEAV1rBmw3KNA1UJRRUAgCps7EVjFROMOWYsJhijsReN9SgRUHEivA4AAABO3+CWgyVJ0z+frh3ZO9SwZkONvWjsT+NAVUZRBQCgihvccjDFFGGJj/4BAADgSxRVAAAA+BJFFQAAAL5EUQUAAIAvUVQBAADgSxRVAAAA+BJFFQAAAL5EUQUAAIAvUVQBAADgSxRVAAAA+BJFFQAAAL5EUQUAAIAvUVQBAADgSxRVAAAA+FKZiqqZDTSzDDPbYGbjS1kfbWaLi9Z/amYtisZbmFmOmX1R9JhTsfEBAAAQriJOtYGZBSXNktRP0lZJK8zsdefcumKbpUra55w738xukDRF0vVF6zY65zpVcG4AAACEubLMqHaVtME5t8k5d1jSK5KuLrHN1ZJeKHr+qqS+ZmYVFxMAAADVTVmKamNJW4otby0aK3Ub51yBpExJ9YrWnWtmq8zsAzO77AzzAgAAoJo45Uf/Z2i7pGbOuT1m1kXS/5hZO+dcVvGNzGy4pOGSFKyTWMmRAAAAUBWUZUZ1m6SmxZabFI2Vuo2ZRUiKk7THOZfnnNsjSc65lZI2Skoq+Qucc/Occ8nOueRgbFz5XwUAAADCTlmK6gpJrczsXDOLknSDpNdLbPO6pNuKnv+HpHedc87MEotOxpKZtZTUStKmiokOAACAcHbKj/6dcwVmNkpSmqSgpEXOubVmNklSunPudUkLJb1kZhsk7dWRMitJvSRNMrN8SSFJI5xzeyvjhQAAACC8lOk7qs65tyS9VWJsQrHnuZKuK2W/1yS9doYZAQAAUA1xZyoAAAD4EkUVAAAAvkRRBQAAgC9RVAEAAOBLFFUAAAD4EkUVAAAAvkRRBQAAgC9RVAEAAOBLFFUAAAD4EkUVAAAAvkRRBQAAgC9RVAEAAOBLFFUAAAD4EkUVAAAAvkRRBQAAgC9RVAEAAOBLFFUAAAD4UoTXAQAA/tJi/JteR8Bp+u6JwV5HACoUM6oAAADwJYoqAAAAfImiCgAAAF+iqAIAAMCXKKoAAADwJYoqAAAAfImiCgAAAF+iqAIAAMCXKKoAAADwJYoqAAAAfImiCgAAAF+iqAIAAMCXKKoAAADwJYoqAAAAfImiCgAAAF+iqAIAAMCXKKoAAADwJYoqAAAAfImiCgAAAF+iqAIAAMCXKKoAAADwJYoqAAAAfImiCgAAAF+iqAIAAMCXKKoAAADwpTIVVTMbaGYZZrbBzMaXsj7azBYXrf/UzFoUW/dA0XiGmQ2ouOgAAAAIZ6csqmYWlDRL0iBJbSXdaGZtS2yWKmmfc+58SdMkTSnat62kGyS1kzRQ0uyi4wEAAAAnVZYZ1a6SNjjnNjnnDkt6RdLVJba5WtILRc9fldTXzKxo/BXnXJ5z7t+SNhQdDwAAADipshTVxpK2FFveWjRW6jbOuQJJmZLqlXFfAAAA4DgRXgeQJDMbLml40eLBzVOuyvAyT3VjU1Rf0m6vcwCVifc5qgPe555o7nWAcFaWorpNUtNiy02KxkrbZquZRUiKk7SnjPvKOTdP0ryyx0ZFMrN051yy1zmAysT7HNUB73OEm7J89L9CUiszO9fMonTk5KjXS2zzuqTbip7/h6R3nXOuaPyGoqsCnCuplaTPKiY6AAAAwtkpZ1SdcwVmNkpSmqSgpEXOubVmNklSunPudUkLJb1kZhsk7dWRMqui7f4iaZ2kAkl3OecKK+m1AAAAIIzYkYlPVGdmNrzo6xdA2OJ9juqA9znCDUUVAAAAvsQtVAEAAOBLFNVqxswKzewLM1tjZn83s/ii8RZmllO07ugjyuu8wOko9j5fa2arzWycmQXMbECx9/fBols7f2FmL3qdGSgvMztYythEM9tW9L5eZ2Y3epENqCh89F/NmNlB51ytoucvSFrvnHvMzFpIesM5d6GX+YCKUOJ93kDSnyT90zn3X8W2eV/Svc65dG9SAmem+Pu82NhESQedc0+aWStJKyXVc87le5EROFPMqFZv/xJ3CkOYc879qCM3FBlVdGtnoFpwzn0r6ZCkBK+zAKeLolpNmVlQUl8de03c84p9LDrLo2hAhXPObdKRy+s18DoLcLaY2UWSvi36xxpQJfniFqo4q2qY2Rc6MpP6taSlxdZtdM518iYWAKCC3G1mQyUlSfq512GAM8GMavWTU1RGm0sySXd5nAeodGbWUlKhJGaWUB1Mc861k/QrSQvNLMbrQMDpoqhWU865Q5LGSBpnZsysI2yZWaKkOZKecZw9imqk6M6R6frfW5wDVQ4FpRpzzq0ysy8l3SjpQ6/zABXo6FdcInXk9s0vSfqDt5GAChdrZluLLZf2Hp8k6U9mNt85FzpLuYAKw+WpAAAA4Et89A8AAABfoqgCAADAlyiqAAAA8CWKKgAAAHyJogoAAABfoqgCAADAlyiqAAAA8CWKKgAAAHzp/wNLDqaMxGRoWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "colors = sns.color_palette()\n",
    "\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are significant differences in the \n",
    "results when evaluating a model using a  traditional cost-insensitive measure such as the \n",
    "accuracy or F1Score,  than when using the savings, leading to the conclusion of the \n",
    "importance of using the real practical financial costs of each context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bayes minimum risk\n",
    "\n",
    "As these methods (RF, LR and DT) are not performing well we then move to use cost-sensitive methods. The first model we used is the Bayes minimum risk model (BMR) [8].\n",
    "As defined in [12], the BMR classifier is a decision model based on quantifying \n",
    "tradeoffs between various decisions using probabilities and the costs that accompany such decisions. \n",
    "This is done in a way that for each example the expected losses are minimized. In  what follows, we \n",
    "consider the probability estimates $\\hat p_i$ as known, regardless of the algorithm used to \n",
    "calculate them.  The risk that accompanies each decision is calculated using the cost matrix described above.\n",
    "In the specific framework of binary classification, the risk of predicting the example $i$ as negative is \n",
    "\n",
    "$$ R(c_i=0|\\mathbf{x}_i)=C_{TN_i}(1-\\hat p_i)+C_{FN_i} \\cdot \\hat p_i, $$\n",
    "and\n",
    "$$ R(c_i=1|\\mathbf{x}_i)=C_{TP_i} \\cdot \\hat p_i + C_{FP_i}(1- \\hat p_i), $$\n",
    "\n",
    "is the risk when predicting the example as positive, where $\\hat p_i$ is the estimated positive \n",
    "probability for example $i$. Subsequently, if \n",
    "\n",
    "$$  R(c_i=0|\\mathbf{x}_i) \\le R(c_i=1|\\mathbf{x}_i), $$\n",
    "\n",
    "then  the example $i$ is classified as negative. This means that the risk associated with the \n",
    "decision $c_i$ is lower than the risk associated with classifying it as positive. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['RF', 'DT', 'LR'])\n",
      "dict_keys(['RF', 'DT', 'LR', 'RF-BMR', 'DT-BMR', 'LR-BMR'])\n"
     ]
    }
   ],
   "source": [
    "from costcla.models import BayesMinimumRiskClassifier\n",
    "ci_models = classifiers.keys()\n",
    "print(ci_models)\n",
    "for model in list(ci_models):\n",
    "    classifiers[model+\"-BMR\"] = {\"f\": BayesMinimumRiskClassifier()}\n",
    "    # Fit\n",
    "    classifiers[model+\"-BMR\"][\"f\"].fit(y_test, classifiers[model][\"p\"])\n",
    "    #classifiers[model+\"-BMR\"][\"f\"].fit(y_test, classifiers[model][\"p\"])\n",
    "    # Calibration must be made in a validation set\n",
    "    # Predict\n",
    "    classifiers[model+\"-BMR\"][\"c\"] = classifiers[model+\"-BMR\"][\"f\"].predict(classifiers[model][\"p\"], cost_mat_test)\n",
    "    # Evaluate\n",
    "    results.loc[model+\"-BMR\"] = 0\n",
    "    results.loc[model+\"-BMR\", measures.keys()] = \\\n",
    "    [measures[measure](y_test, classifiers[model+\"-BMR\"][\"c\"]) for measure in measures.keys()]\n",
    "    results[\"sav\"].loc[model+\"-BMR\"] = savings_score(y_test, classifiers[model+\"-BMR\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print (ci_models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAEvCAYAAACT2dMvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deVxU5eIG8OedjWFAQBYBccF9F/ddc0+vXeuW3TaXMjU1inKvzFwqUbuZmku2uGS37PZr87ovpZVdE/cVNXcFVFBAYJgZ5v39ARgi6qDDvDDzfD+fPjHnnJnzzAiHh7O8R0gpQURERESkikZ1ACIiIiLybCykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESklE7VioODg2VkZKSq1RMRERE5bNeuXVeklCGqc7grZYU0MjIScXFxqlZPRERE5DAhxBnVGdwZD9kTERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKaXsXvZERMUVOWG16ggl6nRsH9URiIiU4B5SIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCmHCqkQopcQIl4IcUIIMeEOyz0mhJBCiBbOi0hERERE7uyuhVQIoQUwH0BvAPUBPCWEqF/EcuUAxADY4eyQREREROS+HNlD2grACSnlSSmlBcBXAB4uYrlpAGYAMDsxHxERERG5OUcKaQSAcwUen8+bdoMQohmAylLK1U7MRkREREQe4L4vahJCaAC8D2C0A8sOE0LECSHiLl++fL+rJiIiIiI34EghvQCgcoHHlfKm5SsHoCGAn4UQpwG0AfBjURc2SSkXSylbSClbhISE3HtqIiIiInIbjhTSnQBqCSGqCSEMAJ4E8GP+TCllqpQyWEoZKaWMBPA/AH2llHElkpiIiIiI3MpdC6mU0gYgGsB6AEcAfC2lPCSEmCqE6FvSAYmIiIjIvekcWUhKuQbAmkLTJt1m2c73H4uIiIiIPAXv1ERERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSDhVSIUQvIUS8EOKEEGJCEfOHCyEOCCH2CiF+FULUd35UIiIiInJHdy2kQggtgPkAegOoD+CpIgrnv6WUjaSUTQDMBPC+05MSERERkVtyZA9pKwAnpJQnpZQWAF8BeLjgAlLKtAIPfQBI50UkIiIiInemc2CZCADnCjw+D6B14YWEEC8CGAXAAKCrU9IRERERkdtz2kVNUsr5UsoaAMYDmFjUMkKIYUKIOCFE3OXLl521aiIiIiIqwxwppBcAVC7wuFLetNv5CsAjRc2QUi6WUraQUrYICQlxPCURERERuS1HCulOALWEENWEEAYATwL4seACQohaBR72AXDceRGJiIiIyJ3d9RxSKaVNCBENYD0ALYDPpJSHhBBTAcRJKX8EEC2E6A7ACuAqgEElGZqIiIiI3IcjFzVBSrkGwJpC0yYV+DrGybmIiIiIyEPwTk1EREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRkWI6vz3wqRGLxssao+c3PbH65GrVkYiIXMqhYZ+IiKhk6Pz2wBj+LYTGCgkgISMBk7dPBgD0qd5HaTYiIlfhHlIiIoW8QtZDaKw3TTPnmDFn9xxFiYiIXI+FlIhIIaG/VuT0xIxEFychIlKHhZSISBVhBWTRm+FQU6iLwxARqcNCSkSkiFfoKgiNHdKuvWWeRmiQYk5RkIqIyPVYSImIFND57YGh/B/IvtIZ5oR+sFsCICAQ7hOOQfUHIdmcjEFrByHheoLqqEREJY5X2RMRuZjGcAnG8O9gy4yE5XIPAFrY0pridOxfV9V3rdIV0ZujMWDtACzuuRjV/aurC0xEVMK4h5SIyJWEBcaILyDtepgvPAXg1sP1ANAstBmW9FoCm92GZ9c+i0PJh1ybk4jIhVhIiYhcyCvsR2i8LsF88QlIm/8dl60TWAfLey+HSW/C8+ufx87EnS5KSUTkWiykREQuovPfBUNAHCzJnZGTUduh51Txq4JlvZYhzBSG4RuHY8vZLSWckojI9XgOKRGRC2gMSTCGfQ9bRjVYLncv1nNDfUKxtNdSjNw8EqN+HoUp7abg4ZoPl1BSUi1ygnvfOrbgudJE+biHlIiopAkLjJW+gLQbYL54+/NG7yTAGIBPen6ClmEtMfG3ifj88OfOz0lEpAgLKRFRCTOGfQ+N4TLMF56EtPnd8+uY9CbM7zYfPar2wMydMzFvzzxIKZ2YlIhIDRZSIqISpPOPgz5gNyxXuiIns9Z9v55Ba8CsTrPwaK1HsXj/Yryz4x3Ypd0JSYmI1OE5pEREJUTjlQhj2A+wZVSH5Uo3p72uVqPF5LaT4e/ljyUHlyAtOw3vdHgHeq3eaesgInIlFlIiopIgsvPGG/XKG2/UuQekhBAY1XwUArwCMHvXbKRb0/F+5/fhrfN26nqI6PZ27dpVQafTfQKgIXjU+U7sAA7abLYhzZs3v1TUAiykREROJ2EM/x4awxVknX0eMqdcia1pcMPB8Df4Y+r/puKFjS/gw24fws9w7+epEpHjdDrdJ2FhYfVCQkKuajQantB9G3a7XVy+fLl+YmLiJwD6FrUM2zyVaqtPrkbPb3qi8bLG6PlNT6w+6d7DoZB70PvHQe+/B5Yr3ZCTWbPE1/dY7ccwq9MsHLhyAM+tew5Xsq6U+DqJCADQMCQkJI1l9M40Go0MCQlJRe6e5KKXcWEeomJZfXI1Jm+fjISMBEhIJGQkYPL2ySylVKppvBLgFfYDbNdrwnKlq8vW2zOyJ+Z3m49z6ecwcO1AnE8/77J1E3kwDcuoY/I+p9v2ThZSUi7Tmolzaeew99JebD67GV/Hf42F+xZi6u9TYc4x37SsOceMObvnKEpKdBeabHhHfAFp94b54hNw9Sa2XcV2+Ljnx0jNTsXAtQNx/Opxl66fiOhe8RxScjopJVKzU5FiTkGyORnJWck3/p8/LSUr7//mFGTZsor1+okZiSWUnOh+SBjDvoUwJCPr7NASPW/0TqJCorC011K8sPEFPLvuWSzovgBRIVFKshBRydNqtc1r1ap14xfpDz/8cMLf3z/n4YcfrnHgwAGffv36JS9fvvxs/vwPPvggaMGCBaEAIKUUb7311oX+/ftfU5G9IBZScojNbsNV89WbymR+0Uwxp/z1dVYKUswpsEnbLa+hFVqUN5ZHoDEQQcYgVPGrkvu1dxCCjEG3fN3nuz5IyEi45XXCfMJc8ZaJikUf8Af0/vuQfakncjKrK81Sq3wtLO+9HMM2DsPQDUPxQZcP0K5iO6WZiAhYGb8ycNG+RRHJWcmGIO8gy/Co4ReeqPNEyv28ppeXl/3o0aOHC05LS0vTTJ069eK+ffu8Dx48eGPojT///FP/r3/9K3zv3r1HgoKCclJTUzUJCQn31QWtViv0+vsfco6F1IOZbeabymTBPZg3vs6bdy276D+eDBrDjRJZwVQB9QLrIcg76EbpvPG1dxACvAKgEY4fwoxpFoPJ2yffdNheQGBE1Ij7fu9EzqTxugiv0FWwXa8FS3Jn1XEAAJXKVcLy3svxwsYX8OLmFzGj4wz0jOypOhaRx1oZvzJw5s6ZVS05Fg0AXMm6Ypi5c2ZVALjfUlqYn5+f/cEHH7weHx/vVXB6QkKC3sfHx+7v758DAP7+/nZ/f38LABw8eNBr2LBhVZOTk3VarVb+5z//OVmvXr3sESNGVNqyZYu/EEKOHTs2YejQoVf/+9//lnvrrbcq+vv755w8edJ4+vTpgwsWLAhcuHBhqNVqFc2aNctYvnz5GZ3O8ZrJQupGpJS4br1+2z2XhUtnhjWjyNfx1fveKJHV/KuheWjzv/Zcet9cNH31vhBClMj76VO9DwBgzu45SMxIhL+XP65lX8O289vQt0ZfaDXFvx84kdNpzPCu9AVkjknJeaN3EuwdjCW9liB6czTGbhuLNEsa+tXupzoWkVt687c3K5+4esJ0u/lHrx71sdltN/3CtORYNLF/xEZ+f/z7kKKeU7N8zcxp7aedu9N6s7OzNXXr1q0PAJUrV87euHHjn7dbtk2bNpnBwcHWypUrN2rfvn36o48+evXpp59OBYCnn3662pgxYxIHDhx4LTMzU+Tk5Ijly5cHHDhwwPvIkSOHEhISdK1atarXs2fP6wBw+PBh0549ew7VrVvXsnv3buM333wTGBcXd9TLy0v279+/yqJFi4Kio6OT75S9IBbSUi7HnoNr2dduu+eycOm02C23vIaAQIBXwI0S2SCowU17LgseLg80BsKoMyp4p0XrU73PjWIKAJ8f/hwzd87ErLhZGN9yfImVYSLHSBjDv4XQX0XWmaGQOb6qA93Cz+CHj3p8hFE/j8KU36cgNTsVzzd6XnUsIo9TuIzebbqjijpkfzs6nQ7btm07vnXrVtOGDRv8JkyYUDkuLs7nzTffTExKSjIMHDjwGgCYTCYJQP7yyy/l/vnPf6bodDpUrlzZ1rp16+u//vqryd/f3964ceOMunXrWgBg3bp15Q4ePGiKioqqBwBms1lToUKFW8/du1O2Yr5vj7b65Oobe+vCfMIQ0yzmprLkKGuONbdc3mbPZcGCeTX7apH3qdYJ3V8l0jsQNQJq3HyIPO/r/EPlOo17/FMPqD8ACRkJ+Pzw5wj3CcegBoNURyIPpg/YAb3ffmRf6oWcrGqq49yWt84bc7vMxRu/vYEPdn+A1OxUvNr8Vf5BR+REd9uT2eXrLo2uZF0xFJ4e7B1s+fKhL+NLLtnNNBoNunTpktmlS5fM3r17pw0ZMiTyzTffLPbVwiaT6UY5kVKKxx9/PHn+/PkX7jWXe7QUF8gfEzP/fMb8MTGB3L14mdbMoq8oL2JPZrolvch1eOu8b5TJCN8INA5u/NdezLw9mflF08/g57G/TMa0GINLmZfwXtx7qGCqgN7VequORB5IY7yQd95oHViSO6mOc1d6rR6xHWPhZ/DDkkNLkGpJxaQ2k3jqC5GLDI8afqHgOaQAYNAa7MOjht9ziSuu06dP68+fP6/v0KFDJgDExcWZIiIiLOXLl7eHhYVZPv/884ABAwZcy8rKEjabTXTq1Cn9448/DomOjk6+dOmS7o8//vCdO3fuuf379990j+JevXqlPfroozVff/31pIiICFtSUpI2NTVVW7t27VsP294GC6mD5uyeU+SYmG/8+gam/D7ltkMXlTOUu1Eia5WvhTbGNrccLs+fb9Lf9tQTKkAjNHinwzu4knUFb/z6BoK9g9EyrKXqWORJNObc8UZzfGG++E+UpvNG70QjNHij9Rvw9/LH4v2LkW5JR2zHWBi0t+y0ISIny79wydlX2d9OREREo+vXr2utVqtYv359wJo1a46VK1fOPmbMmEpJSUl6Ly8vGRgYaP3444/PAsCKFStODR06tOq0adMq6vV6+Z///OfPAQMGXNu+fbtvvXr1Gggh5JQpU85XqVLFtn///pvW1bx5c/PEiRMvdOvWrbbdboder5dz5849W5xCKqRUc4OBFi1ayLi4OCXrvheNlzWGRNGf1YD6A265ojy/aOq19z8UAhUtNTsVg9YOwqXMS1jWexlqla+lOhKVsMgJpeEuXRLGiC+gK3cYWWeGIScr0mmvfDq2+KcA3av887HbhLfBnC5z+AdxKVI6vs9Ljiu/z51JCLFLStmi4LR9+/adjoqK4r16HbRv377gqKioyKLmcQ+pg8J8woocEzPcJxzjWo5TkIj8vfyxsPtCPLPmGYzYNAIr/raCY5RSidOX/x16v4MwJ/V2ahl1tQH1B8Dfyx+TfpuEIRuGYEG3BQgwBqiORUQeqmwcZyoFYprFwKi9+epzo9aImGYxihIRAIT7hmNh94W4br2OkZtH3vb8XCJn0BjPwyt0NWzpdWFN6ag6zn3rW6MvZneejfiUeDy77lkkZSSpjkREHoqF1EF9qvfB5HaTEe4TDgGBcJ9wTG43+Z6usifnqhNYB7M7z8apa6fw6k+vwppjVR2J3JEmC94R/4a0+SIr4XG4y+azS5UuWNRjERIzEzFw7UCcSTujOhIReSD32KK6SJ/qfbCh3wbsH7QfG/ptYBktRdpWbIup7adiR+IOTPxtYpFDZRHdOwlj+DcQ+mvIuvA0kOOjOpBTtQxriU8f/BRZtiwMXDsQR1OOqo5ERB6GhZTcxt9r/B0xzWKw5tQazNk9R3UcciP68r9B73cI2Zd6wZ5VVXWcEtEgqAGW9l4Kg9aAwesGY1fSLtWRiMiDsJCSW3m+4fN4os4T+OzgZ/jy6Jeq45Ab0BjPwSt0Lazp9dzivNE7qe5fHct7LUeQdxBe2PgCtp3fpjoSEXkIXmVPbkUIgddavYakzCRM3zEdFbwroFvVbqpjUVmlycwdb9RaDuaLjwNw/5tRhPuGY1nvZRi+cThitsTg7Q5v8/QkIgdFTljd3Jmvdzq2j0OHKsaPHx/2f//3f0EajUZqNBosWLDgTNeuXTOKs64nnnii6rhx45KaN29uvvvSzsc9pOR2tBotZnaaiUbBjTD+l/HYe2mv6khUJkkYK/4HQp+OrAvPAHbPGacz0BiIzx78DE0qNMFrv7zGow1EpdimTZt81q9fH3DgwIHDx44dO/zTTz8dq169usMD0udbuXLlGVVlFGAhJTflrfPGvG7zEGoKRfSWaJxKPaU6EpUx+sBfoS93BNlJvWE3V1Ydx+V8Db5Y1GMRHqj8AN7d8S4W7VsEVTdSIaLbu3Dhgj4wMNDm7e0tASA8PNwWGRlpHTNmTHjDhg3r1apVq8FTTz1V1W63Y8+ePcZGjRrVy39ufHy8oXbt2vUBoFWrVnW2bdtmAgCTydT0pZdeiqhTp079qKiouufOndMBwKFDh7yioqLq1q5du/7LL79c0WQyNQWAM2fO6Fu0aFGnbt269WvVqtVg3bp1vsV9Hyyk5LYCjYFY1H0RtEKLEZtG4EoWb6ZBjtEYz8KrwlpY0xrAerW96jjKeGm9MLvzbPSt0Rfz987HzJ0zOYIFUSnzyCOPpF28eNEQGRnZsH///lVWr17tCwBjx469dPDgwSPHjx8/lJWVpfnqq6/8mzZtarZareLo0aMGAFi+fHngI488crXwa2ZlZWnatm17PT4+/nDbtm2vz5s3LwQAoqOjK48cOfLSsWPHDleqVOnGGIufffZZYLdu3VKPHj16+MiRI4dat26dWdz3wUJKbq2yX2XM7zYfKeYUvLj5RWRai/0zQp5GmwHvSl9AWgNgTugHTzhv9E50Gh2mtZ+G/vX6Y8WRFZj460RY7Rzrl6i08Pf3tx88ePDwhx9+eCYkJMQ2aNCgGnPnzg1au3ZtucaNG9etXbt2/e3bt5c7ePCgNwA88sgjKcuXLw8EgO+++678gAEDUgq/pl6vl08++WQqADRv3jzjzJkzBgDYs2eP7+DBg1MAYMiQIcn5y7dp0ybjyy+/DB41alTFP/74w7t8+fLF/suVhZTcXsPghnjvgfcQnxKPUVtH8Zcp3YEd3uH/gdBdzx1v1O6tOlCpoBEajGs5DtFNorHq5CqM+mkUzDZlp5oRUSE6nQ4PPfRQ+uzZsy/OmjXr7FdffRU4evToqt9+++2fx44dO9y/f/8rZrNZAwADBgy4+v3335ffv3+/lxACjRo1yi7i9aRGo7nx2jab7Y5/mffu3fv6tm3b4iMiIiyDBw+u9uGHHwYV9z2wkJJH6FSpEya2mYjfLvyGab9P47lwVCR94C/QlTuK7KS/wW6upDpOqSKEwAtRL+CN1m9g6/mtGLFpBK5brquORWWIzm8PfGrEovGyxuj5TU+sPrladSS3sG/fPq8DBw545T/es2ePd82aNbMBICwszJaamqpZtWpV+fz5DRo0yNZoNJg0aVLFf/zjH7fsHb2TJk2aXF+6dGl5IPcwff70Y8eOGSpVqmQdPXr0lYEDB17evXt3sa8C5bBP5DH61e6HxIxEfLT/I4T5hGFkk5GqI1EpovU+Da8K62FNawTr1Xaq45RaT9Z9En4GP7zx6xsYvH4wFnZfiCDvYu8MIQ+j89sDY/i3EBorJICEjARM3j4ZANxqWDFHh2lyprS0NO3LL79cJS0tTavVamVkZGT2smXLzgQEBNjq1avXICQkxBYVFXXTEFCPPvpoyrRp0yrNmDHjQnHWNW/evHPPPPNMtVmzZoV37do1zdfXNwcA1q9fX27u3LlhOp1OmkymnC+++KLYVxILVXuKWrRoIePi4pSsmzyXlBKTtk/C9ye+x5R2U/BorUdVR6JiiJxQMntUhDYDpmpzAKlHxqmXALuxRNZzN6djy84v5m3nt2H0z6MR5hOGxT0WI9w3XHUkt1FS3+cq+dSIhcZw7Zbp4T7h2NBvg4JExSeE2CWlbFFw2r59+05HRUV5zBWz6enpGh8fH7tGo8HixYvLr1y5MnDz5s1/Ovr8ffv2BUdFRUUWNY+H7MmjCCEwqe0ktK/YHlN/n8o70RAAO4wVv4bQZuSdN6qmjJY1nSp1wkc9PkJyVjIGrB2Ak6knVUeiUskGvf9OCP2tZRQAEjMSXZyH7sdvv/1mqlevXv3atWvXX7x4cYU5c+acd9ZrO1RIhRC9hBDxQogTQogJRcwfJYQ4LITYL4TYLIRwz5s9k1vQa/T4V+d/oXb52hizdQwOXTmkOhIpZAjaBp1vPLIvPQS7OUJ1nDKlWWgzLOm1BDa7Dc+ufZY/S/QXkQ194C/wqTkTxor/B0htkYuF+YS5OBjdj169el2Pj48/fOzYscNxcXHxDRs2vOWCqHt110IqhNACmA+gN4D6AJ4SQtQvtNgeAC2klI0BfANgprMCEpUEH70PFnRfgEBjIEZuHolz6edURyIFtN6nYAjZAGtqY1ivtlEdp0yqE1gHy3svh0lvwuD1g/FHwh+qI5FCQpsBQ/BG+NaKhTF0NeyWYGSeHQxzwmOQdv1Nyxq1RsQ0i1GU1Gnsdrvds8eGc1De53Tb4aAc2UPaCsAJKeVJKaUFwFcAHi64gJTyJyll/gCP/wPAy1Op1Av2DsaC7guQI3MwYtMIXDXfMjYwuTGhvQ5jxJeQlkCYEx+Fp483ej+q+FXBsl7LUNG3IkZsGoHNZzerjkQuJnTX4BX6I3xqxsIrZDNsmdWQcWokss4OQ05GbdjSmsGc8CjslgAICIT7hGNyu8nucEHTwcuXL/uzlN6Z3W4Xly9f9gdw8HbLOHKVfQSAgruPzgNofYflnwew1qGERIpV96+OeV3nYeiGoYjeEo1Pen4Cbx3HnnR/dhgrroTQZiLz3LM8b9QJQn1CsbTXUozcNBKjfh6FKe2m4JGaj6iORSVMY0iCIWgrdP57AQC21CawJD8AuyX0lmVtaU1hS2tapi7euxubzTYkMTHxk8TExIbgdTl3Ygdw0GazDbndAk4d9kkI0R9ACwAP3Gb+MADDAKBKlSrOXDXRPWtaoSliO8Zi1M+jMGHbBLzf+X1oNUWf70TuwRD0M3S+x2FO+Afs2RVVx3Eb/l7++Ljnx4j5KQZv/vYm0rLTMLDBQNWxqARojGdhCP4Z+nKHIe16WK+2gSW5I6St/N2f7EaaN29+CUBf1TncgSNt/gKAygUeV8qbdhMhRHcAbwDoK6Us8iRXKeViKWULKWWLkJCQe8lLVCK6V+2O8a3GY8u5LZj+x3QOnO/GtKY/YQjZCGtqFKzXWqmO43ZMehPmd5uPHlV7YFbcLMzdPZc/T25DQutzDN5VFsOn2gLoTKeQfbkrMk6MR3ZSX48ro+Rcjuwh3QmglhCiGnKL6JMAni64gBCiKYCPAPSSUl5yekoiF3im3jNIykjCkkNLEO4TjucbPa86EjmZ0KbDGPEVpCWI542WIIPWgFmdZmHa/6bh4wMfIzU7Fa+3fp1HHsosO3TlDsIQtBVa7wuwW/1gTvobrNdaA3avuz+dyAF3LaRSSpsQIhrAegBaAJ9JKQ8JIaYCiJNS/ghgFgBfAP8RQgDAWSkld2FTmfNK81eQmJGID3Z/gFCfUDxU/SHVkchp8s4b1WQh8+xg/iItYVqNFm+1fQv+Xv747OBnSLOk4d0O70Kv1d/9yVQ6CBv0/rthCNoKjSEZ9uxgmC8+BmtaU0DyRo/kXA59R0kp1wBYU2japAJfd3dyLiIlNEKDtzu8jWRzMt787U0EewejTTiHA3IHhuAt0PmeyL3SN5t3FXIFIQRebf4q/L38MXvXbKRb0zG782xeOFjaabKhD9gBQ+Av0OjTkZMVgazzz8CW3gC8bodKCr+ziAoxaA2Y3WU2Iv0i8epPryI+JV51JLpPWtMJGII3w5raFNZrLVXH8TiDGw7G5LaT8fvF3zFswzCkZqeqjkRFENrrMIRsgG/N6TCGroHdUgGZZ59H5ulo2NIbgZWBShK/u4iK4Gfww8LuC2HSmzBy00gkXE9QHXfpYOQAABzASURBVInuUf55o3ZLMMwJj4DnjarxWO3HMKvTLBxKPoTB6wfjcuZl1ZEoj9BdhVfoD/CpOQNewVtgy6iBjFMvIuvsUORk1AJ/ZsgVWEiJbiPMJwwLuy9Epi0TIzaN4F6dMskOY8SXEJpsmC88A0ieN6pSz8iemN9tPs6ln8PAtQN5hzTFNIYkGMNXwqfmLOjL74AtrTEy/hwF84UBsJsr3/0FiJyIhZToDmqXr405XebgTPoZvPLTK7DkWFRHomIwBG+GzuckzIkPw57Ne2aXBm0rtsUnPT9BmiUNg9YOwvGrx1VH8jga7zMwVloGnxqzofM7CGtKW2ScGAdzwuOwWyqojkceipfJuYnICatVRyhRKu/s0Sq8Fd5u/zYm/DIBb/z6BmZ0mgGN4N9ypZ3W5zgMwVtgvdYMttQWquNQAY1DGmNZr2V4YeMLeHbds1jQfQGiQqJUx3JzuWOIGoJ+hs7nFKTNhOzL3WC52g7I8VEdjoh7SIkc0ad6H7za/FWsO70O78e9rzoO3YXQpcFY8SvYLSEwJ/L2laVRzfI1saz3MgR4BWDohqHYfmG76khuKgc6v30wVZsLU5Ul0BhSYE58CNdPjIflSg+WUSo1WEiJHPRcg+fwVN2nsOzwMqw4vEJ1HLqtHBgrfgmhscB8/hlAGlQHotuoVK4SlvVehirlquDFLS9i/en1qiO5D2GFPmAHfGq8D++ILwFhQ9bFfsg4MRbWqx14PjWVOjxkT+QgIQTGtxyPS5mXMHPnTFQwVUDPyJ6qY1EhhpBN0PmcQtbFx2G3hKqOQ3cR7B2Mz3p9hujN0Ri7dSzSLenoV7uf6lhll8YMQ8AO6IN+hUaXjpysSsg63x+29PrgPigqzfjdSVQMWo0WsR1jERUShdd+eQ27knapjkQF5J8jZ7nWArbU5qrjkIP8DH74qMdHaB/RHlN+n4JPD3yqOlKZkzuG6Hr41oyFV+ha2M2hyDwzBJmnX4QtvSH4655KO36HEhWTUWfEvK7zUNG3Il7e8jJOXjupOhIBELpUGCuuhD27ArITeefissZb5425Xefib9X+hg92f4D3496HlFJ1rFJP6FPgFfo9fGrGwhD0M2wZNZFxKhpZ54YgJ7MmOIYolRU8ZE90DwKMAVjYfSH6r+mPEZtGYMXfViDEFKI6lgfLyRtv1IqsCzxvtKzSa/SY3nE6/Ax+WHJoCa5lX8OktpOg0/BXVWEar8TcK+b99gMQsKY2hSX5AUgLt0NUNvGnnOgeVSpXCfO7z8dz657DyM0jseTBJfA1+KqO5ZEMIRuhM51G1oUnOI5iGacRGrze+nX4e/njo/0fId2SjhmdZsCg5R8ZAKD1Pp1bRMsdhbQbYE1pB0tKR0ibv+poRPeFh+yJ7kODoAZ4v/P7OH71OEb9PApWu1V1JI+j9TkKr+CfYbnaCra0pqrjkBMIIRDdNBrjWo7DprObMHLzSGRYM1THUkhC63MU3lUXwRS5CBrvs8i+3B3Xj09A9qWHWEbJLbCQEt2nDhEd8Fbbt/B7wu+YvH0yz3tzIaG7Bu+KXyPHHIbspL+rjkNONqD+ALzT4R3EJcZh6IahuGa+pjqSi+VA57cXpmpzYKqyFBr9VZgT/46MExNgudIdsJtUByRyGh6yJ3KCf9T6BxIzE7Fg7wKE+YThpaYvqY7kAXL+Gl/xwjOA1KsORCWgb42+KKcvhzFbx2DQukH4qMdHCPNx89vACiv0/rtgCNoGjSEFOdkVkHXxcdhSo8Bf2+SuuIeUyEmGNx6Ox2o9hsX7F+Pr+K9Vx3F7XhXWQ2s6A3PiY7yQw811qdIFi3osQlJmEgatHYQzaWdURyoR6ZZ0fHLgE/jUnAFj+PeQOT7IOjcAmSdfyRvGjGWU3Be/u4mcRAiBiW0m4lLmJbyz4x1UMFVA58qdVcdyS1rfIzAEbYPlamvY0ngPdE/QMqwlPn3wU4zYOAID1w7ERz0+Qt3AuqpjOcWVrCtYcXgFVsavxHXrddjNtWBO7oyczOrgsE3kKbiHlMiJdBod3nvgPdQLrIexW8fiwOUDqiO5HaG7mnfeaDiykx5SHYdcqEFQAyztvRQGrQHPrXuuzN+Y4lz6Obz9v7fx4DcP4rODn6FdxXZY+dBKZJ17HjmZNcAySp6EhZTIyUx6Ez7s9iGCvYPx4uYXcTbtrOpIbsQG70r/BmBH1nmeN+qJqvtXx/JeyxHsHYwXNr6Abee3qY5UbPEp8Ri3bRwe+u4hfHv8W/y9xt+x6h+r8K/O/0L9oPqq4xEpwUJKVAKCvYOxsPtCSEgM3zQcKeYU1ZHcgleFddB6n4M54TFIa7DqOKRIuG84lvVehhoBNRCzJQb/Pflf1ZEcsitpF0ZuGol+q/ph67mtGFh/INY9tg6T201GVb+qquMRKcVCSlRCIv0jMa/rPFzKvITozdHItGaqjlSmbTm7BYagX2FJaQtbemPVcUixQGMgPu35KZqGNsVrv7yGfx/5t+pIRbJLe275XDsQz657FgevHER0k2hs6LcBo1uMRgUTb+RABLCQEpWoJhWaYGanmTiUfAjjto2DzW5THalMOp9+HhN/m4icrAhkX+qjOg6VEr4GXyzsvhCdK3fG9D+mY+G+haVmHGCb3YZVf67CYz8+hugt0UjMSMRrrV7D+n7r8ULUC/D34mD2RAWxkBKVsK5VuuL1Vq9j6/mteGfHO6XmF2ZZYc2xYuzWsZBSIuvC04Dk4CD0Fy+tF2Z3no2+Nfpiwd4FmLFzBuzSriyP2WbGl0e/xEPfPYTXf30dUkq82+FdrH50NZ6u9zS8dd7KshGVZtyyE7nAE3WfQEJGAj49+CnCfcIxrPEw1ZHKjPd3vY+DyQfxfuf3MXS/RXUcKoV0Gh2mtZ8GP4MfVhxZgdTsVExtPxV6jesuekuzpGHl0ZVYcWQFUswpaBzSGONbjscDlR+ARnDfD9HdsJASuUhMsxgkZSZh3p55CDWF4uGaD6uOVOptOrMJK46swDP1nkGPqj0ArFYdiUopjdBgXMtxKG8sj3l75uG65TpmPTALRp2xRNd7OfMyPj/yOb6O/xoZ1gy0j2iPIQ2HoHlocwjBYZuIHMVCSuQiQghMbTcVl7MuY/L2yQjxDkG7iHaqY5Va59LPYdJvk9AwqCFGNx+tOg6VAUIIDGs8DP4Gf7yz4x0M3zQc87rOQzlDOaev62zaWSw5tAQ/nPgBOTIHD1Z9EIMbDXabwfqJXI3HEYhcSK/VY3bn2ageUB2v/vwqjiQfUR2pVLLkWDBm6xhAALMemAW9luONkuOeqPsEYjvGYt+lfXh+/fNIzkp22msfST6CsVvH4u/f/x0/nPgBj9R8BKseWYWZD8xkGSW6DyykRC5WzlAOC7otgJ+XH0ZuHomL1y+qjlTqvBf3Hg4nH8a09tNQqVwl1XGoDPpb9b9hbte5OJV6Cs+ue/a+fs6klNiZuBPDNw3HP//7T/xy4RcMajAI6x9bj0ltJ6GKXxUnJifyTCykRAqE+oRiYbeFyM7JxvBNw5Ganao6Uqmx/vR6fHn0SwyoPwDdqnRTHYfKsI6VOmJxz8VIzkrGwLUDcfLayWI93y7t+OnsTxiwdgAGrx+MI8lHENMsBhv6bcCo5qMQYgopoeREnoeFlEiRmuVrYk6XOTiffh4vb3kZ2TnZqiMpdzbtLN7a/hYaBzfGq81eVR2H3EDTCk2xpNcS2Ow2DFo3CIeuHLrrc6x2640xRF/+6WVcybqCN1q/gfWPrceQRkPgZ/BzQXIiz8JCSqRQy7CWeLfDu9h9aTde++U1peMnqpadk40xW8dAK7Q8b5Scqk5gHSzvvRw+eh8MXj8YOxJ2FLlcli0LXxz5An2+7YPXf30dADC943T89x//xZN1nyzxK/aJPBmvsidSrFe1XkjKTMJ7ce9h1s5ZGNdynEcOFzNr5ywcSTmCeV3noaJvRdVxyM1U8auC5b2X44WNL2DEphF4os4T2Hx2MxIzElHBVAGNgxsjLikOV7OvoklIE7zR+g10qtTJI38WiVRgISUqBQY1GITEjESsOLICYT5hGNRgkOpILrXu1DqsjF+JZxs8i86VO6uOQ26qgqkClvZaiqdWP4UVR1bcmJ6UmYSNZzeiTvk6+KDLB2gW2kxhSiLPxEJKVEqMbTn2xp7SUFMoelXrpTqSS5xOPY23tr+FqJAovNzsZdVxyM35e/nDmmMtcl6aJY1llEgRFlKiUkIjNJjecTqSs5Lx+q+vI8g7CC3DWqqOVaLMNjPGbB0DvVaP9x54z6W3eiTPlZSZVOT0xIxEFychony8qImoFPHSemFu17moXK4yYrbE4MTVE6ojlagZO2cg/mo83u3wLsJ8wlTHIQ9xu+81fg8SqcNCSlTK+Hv5Y2H3hTDqjBi+aTiSMorem1PWrT65Gt8c+waDGw5Gp0qdVMchDxLTLAZG7c1XzBu1RsQ0i1GUiIiUHbI/cCEVkRNWq1r9fTkd20d1BHJzFX0rYkH3BRi0dhBGbh6Jpb2Wlsj9uFU5mXoSU36fgqYVmiK6abTqOORh+lTP3YbP2T0HiRmJCPMJQ0yzmBvTicj1eA4pUSlVN7AuZneZjRc3vYhXf34VC7stdIuxObNsWRj982gYtUbM7DST542SEn2q92EBJSpFeMieqBRrV7EdprSfgh0JO/Dm9jfdYuD82D9iceLaCbzbkeeNEhFRLu4hJSrl+tboi8SMRMzbMw9hpjC80vwV1ZHu2ao/V+Hb499iaKOh6BDRQXUcIiIqJVhIicqAoY2GIjEjEZ8e/BRhPmF4su6TqiMV28lrJzHtf9PQPLQ5RjYZqToOERGVIiykRGWAEAKvt34dlzMv490d7yLEFIJuVbqpjuWwTGsmRm8dDW+dN2Z2mgmdhpseIiL6C88hJSojdBodZnSagYbBDTF+23jsvbRXdSSHvbvjXfx57U9M7zgdFUwVVMchIqJShoWUqAwx6U34sNuHCDWF4qUtL+F06mnVke7q+xPf44c/f8CwxsPQrmI71XGIiKgUYiElKmMCjYFY2H0hNEKD4ZuG40rWFdWRbuvE1RN453/voGVYS4yIGqE6DhERlVIspERlUBW/Kviw64dIMafgxc0vItOaqTrSLfLPG/XR+2BGxxnQarSqIxERUSnFQkpURjUKaYRZnWbhaMpRjN46Gja7TXWkG6SUePt/b+NU6inEdopFiClEdSQiIirFWEiJyrAHKj+AiW0m4tcLv2La/6ZBSqk6EgDguxPfYdXJVRgRNQJtwtuojkNERKUcx14hKuMer/04EjMSsXj/YoSZwjCiidpzNY9dPYZ3d7yL1uGtMazxMKVZiIiobHBoD6kQopcQIl4IcUIIMaGI+Z2EELuFEDYhRD/nxySiO4luEo2+Nfpiwb4F+O74d8pyZFgzMPrn0ShnKIfYjrE8b5SIiBxy1z2kQggtgPkAegA4D2CnEOJHKeXhAoudBfAsgDElEZKI7kwIgcntJuNy5mVM+X0Kgr2D0bFSR5dmkFJi6u9TcTb9LD7p+QmCvYNdun4iIiq7HNlD2grACSnlSSmlBcBXAB4uuICU8rSUcj8AewlkJCIH6DV6zO4yG7XK18LoraNxKPmQS9f/zfFvsObUGoyMGomWYS1dum4iIirbHCmkEQDOFXh8Pm8aEZUyPnofLOi2AOW9yuPFTS/ifPp5l6w3PiUesTti0Ta8LYY0GuKSdRIRkftw6UVNQohhAIYBgNaPw8AQlYQQUwgWdl+IAWsHYMSmEVjeeznKG8uX2PquW65j9NbRCPAKwPSO03neqJuJnLBadYQSdTq2j+oIRATH9pBeAFC5wONKedOKTUq5WErZQkrZQmvyv5eXICIHVA+ojnld5+Hi9Yt4actLMNvMJbIeKSWm/D4F59LPYUanGQjyDiqR9RARkXtzpJDuBFBLCFFNCGEA8CSAH0s2FhHdr2ahzRDbKRb7L+/H+G3jkWPPcfo6vo7/GutOr8NLTV9Ci7AWTn99IiLyDHctpFJKG4BoAOsBHAHwtZTykBBiqhCiLwAIIVoKIc4DeBzAR0II115NQURF6lG1B8a1HIct57Yg9o9Ypw6cfyT5CGbsnIH2Ee0xuOFgp70uERF5HofOIZVSrgGwptC0SQW+3oncQ/lEVMr0r98fiRmJWHZ4GcJ9w51SHtMt6Ri9dTTKG8tjeofp0Aje9I2IiO4d79RE5AFGtRiFpMwkzN41G6GmUPSpfu8Xckgp8db2t3Dx+kV89uBnJXrBFBEReQYWUiIPoBEavNPhHVzJuoKJv01EsHcwWoe3vqfX+vLol9h4ZiNeafYKmoU2c3JSIiLyRDzORuQhDFoD5nSdg0i/SLzy0yuIT4kv9mscunII78W9h44RHfFcw+dKICUREXkiFlIiD+Jn8MPC7gth0pkwcvNIJGYkOvzcNEsaRm8djSDvILzb4V2eN0pERE7D3yhEHibMJwwLui9ApjUTIzaNQJol7a7PkVJi0m+TkJSRhFmdZiHAGOCCpERE5ClYSIk8UJ3AOvigywc4nXYaMVtiYMmx3HH5L458gc1nNyOmWQyaVGjiopREROQpWEiJPFTr8NaY1n4a4pLiMPHXibBLe5HLHbh8AP/a9S90rtQZgxoMcnFKIiLyBLzKnsiDPVT9ISRlJOGD3R8g1CcUo1uMvml+anYqxmwdgwreFfB2h7chhFCUlIiI3BkLKZGHG9xwMBIyErD00FKE+YThmXrPAMg9b/TN397EpcxLWNZ7Gfy9/BUnJSIid8VCSuThhBB4rdVruJx5GTP+mIFz6eew5ewWJGQkAAD6VOuDxiGNFackIiJ3xnNIiQhajRYzOs1ApXKV8MWRL26UUQDYfHYzVp9crTAdERG5OxZSIgIAGHXGIq+2N+eYMWf3HAWJiIjIU7CQEtENlzIvFTm9OAPoExERFRcLKRHdEOYTVqzpREREzsBCSkQ3xDSLgVFrvGmaUWtETLMYRYmIiMgT8Cp7IrqhT/U+AIA5u+cgMSMRYT5hiGkWc2M6ERFRSWAhJaKb9KnehwWUiIhciofsiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCmd6gBEZVXkhNWqI5So07F9VEcgIiIPwT2kRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUg4VUiFELyFEvBDihBBiQhHzvYQQK/Pm7xBCRDo7KBERERG5p7sWUiGEFsB8AL0B1AfwlBCifqHFngdwVUpZE8BsADOcHZSIiIiI3JMje0hbATghpTwppbQA+ArAw4WWeRjAsryvvwHQTQghnBeTiIiIiNyVI4U0AsC5Ao/P500rchkppQ1AKoAgZwQkIiIiIvcmpJR3XkCIfgB6SSmH5D0eAKC1lDK6wDIH85Y5n/f4z7xlrhR6rWEAhuU9rAMg3llvxMWCAVy561LkTPzMXY+fuevxM3c9fuauV1Y/86pSyhDVIdyVzoFlLgCoXOBxpbxpRS1zXgihA+APILnwC0kpFwNYfG9RSw8hRJyUsoXqHJ6En7nr8TN3PX7mrsfP3PX4mVNRHDlkvxNALSFENSGEAcCTAH4stMyPAAblfd0PwBZ5t12vRERERERwYA+plNImhIgGsB6AFsBnUspDQoipAOKklD8C+BTA50KIEwBSkFtaiYiIiIjuypFD9pBSrgGwptC0SQW+NgN43LnRSrUyf9pBGcTP3PX4mbseP3PX42fuevzM6RZ3vaiJiIiIiKgk8dahRERERKQUC+kdCCFyhBB7hRAHhRCrhBABedMjhRBZefPy/zOozusOCnzmh4QQ+4QQo4UQGiHEgwU+6+t5t7LdK4RYrjpzWSeEuF7EtMlCiAt5n/FhIcRTKrKVBfe7nRBC/Fzg+/lI3vB4+fNOCyF+KbT83ryh9iCE6CyESM2bdlQI8V5Jv9/SwBnbCSHEUiHEqQKf3VsF5v0shDhb8AYvQojv839WCv3bHhZCLBdC6F3z7tW43+1EgX+zfUKI3UKIdnnTI4UQUgjxdoFlg4UQViHEh8VdD5VdLKR3liWlbCKlbIjci7VeLDDvz7x5+f9ZFGV0N/mfeQMAPZB7y9q3pJTr8z9rAHEAnsl7PFBpWvc2O+/zfhjAR+7+C/c+OGM78UzeZ90ewIxCxbWcEKIyAAgh6hXx3F/yntsUwENCiPb3/5ZKPWdtJ8bmLdsEwCAhRLUC864h998DeX9khBd67p95z22E3OEQ/+m0d1e2OLqdyP83iwLwGoDpBeadAtCnwOPHARy6x/VQGcVC6rjfcesdqqgESSkvIfdGCtEF91SQa0kpjwPIBFBedZYy4H63E74AMgDkFJj2NYAn8r5+CsCXRT1RSpkFYO99rr/McdJ2wpj3/4wC077CXyPGPArg29usPwfAH/Cwz72wYm4n/ABcLfA4E8ARIUT+2KRPIPf7/n7XQ2UIC6kDhBBaAN1w8/irNQocGpqvKJrbk1KeRO5wYxVUZ/FUQohmAI7n/eKn27jP7cQXQoj9yL173bS8kpPv/5BbiADg7wBW3Wb95QHUArDtXt9DWXUf24lZQoi9yL0l9leFvsc3A+iU9+/6JICVRb2AEMIIoDWAdcUO7kYc2E54558eAeATANMKzf8KwJN5RwNyAFy8x/VQGeXQsE8ezDtvYxUB4AiAjQXm5R+uIXJXrwohngNQG7lFiIrmjO3EM1LKOCFECIDtQoh1UsozefOSAVwVQjyZ9/qZhZ7bUQixD7ll9AMpZeJ9vRvPMlZK+Y0QwhfAZiFEOynl9rx5OQB+RW4Z9ZZSni60A7ZG3r97NQCrpZT7XZq89HB0O5GV/7MghGgLYLkQomGB+euQW1KTUHT55/bIzXEP6Z3l/wBVBSBw87lh5AJCiOrI/cXAv4Zdb3beOXqPAfg0b08Q3apY2wkhxPq8PUWfFJ4npbwMYDdy97gVtBLAfBR9uP6XvPPyGgB4XgjhcX8oO7KdEEIsyfvc1xSeJ6W8DuBnAB0KzfoKwFwUffg4/4+NGgCaCyH63mP8su6W7YQQonKBIwPDCz9BSvk7cu9nH1JgmgXALgCjAXzjyHpK4s2QOiykDpBSZgJ4GcBoIQT3KrtI3t6iRQA+5K1o1cm7G1sc/ro9MBXB0e2ElPLBvIs7hhSeJ4QwIffipD8LzfoOwEzk3jHvdq97CkAsgPH3EL/McnQ7IaV8Lu9z/1sRr6FD7h8BhT/3X5B78U2R5+3mve4VABOQe6GOxyq4nZBSnitwId+iwssKIeoi9xSL5EKz/gVgvJQyxZH1OC89lQYspA6SUu4BsB+5FxVQyck/z+gQgE0ANgCYojiTuzMJIc4X+G9UEctMBTBKCMFtxh3cx3bii7zDv7sALJVS7ir0uulSyhkOjOaxCLnnPUYWc/1ljbO2E/nnkO4HcACFLlySud7LK5138j1yf4463kOGsuJ+txP5/2Z7kbvHf1Chc6UhpTwkpVzmQBZuj9wQ79RERERERErxrwsiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISKn/Bw8DOtKjHtotAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost-sensitive decision trees\n",
    "\n",
    "The next algorithm that is evaluated is the cost-sensitive decision trees algorithm [13].\n",
    "\n",
    "Decision trees are one of the most widely used machine learning algorithms [14]. \n",
    "The technique is considered to be white box, in the sense that is easy to interpret, and has a \n",
    "very low computational cost, while maintaining a good performance as compared with more complex \n",
    "techniques [15]. There are two types of decision tree depending on the objective of \n",
    "the model. They work either for classification or regression. \n",
    "\n",
    "- Construction of classification trees\n",
    "\n",
    "Classification trees is one of the most common types of decision tree, in which the objective \n",
    "is to find the $Tree$ that best discriminates between classes. In general the decision tree \n",
    "represents a set of splitting rules organized in levels in a flowchart structure.\n",
    "\n",
    "- Splitting criteria\n",
    "\n",
    "In the $Tree$, each rule is shown as a node, and it is represented as $(\\mathbf{x}^j,l^j)$, meaning \n",
    "that the set $\\mathcal{S}$ is split in two sets $\\mathcal{S}^l$ and $\\mathcal{S}^r$ according to \n",
    "$\\mathbf{x}^j$ and $l^j$:\n",
    "\n",
    "$$ \\mathcal{S}^l = \\{\\mathbf{x}_i^* \\vert \\mathbf{x}_i^* \\in \\mathcal{S} \\wedge x^j_i \\le l^j \\}, \\quad and \\quad\n",
    "  \\mathcal{S}^r = \\{\\mathbf{x}_i^* \\vert \\mathbf{x}_i^* \\in \\mathcal{S} \\wedge x^j_i > l^j \\}, $$\n",
    "\n",
    "where $\\mathbf{x}^j$ is the $j^{th}$ feature represented in the vector \n",
    "$\\mathbf{x}^j=[x_1^j,x_2^j,...,x_N^j]$, and $l^j$ is a value such that $min(\\mathbf{x}^j) \\le l^j < \n",
    "max(\\mathbf{x}^j)$. Moreover, $\\mathcal{S} = \\mathcal{S}^l \\cup \\mathcal{S}^r$.\n",
    "\n",
    "After the training set have been split, the percentage of positives in the different sets is calculated. First, the number of positives in each set is estimated by\n",
    "\n",
    "$$ \\mathcal{S}_1 = \\{\\mathbf{x}_i^* \\vert \\mathbf{x}_i^* \\in \\mathcal{S} \\wedge y_i =1 \\}, $$\n",
    "\n",
    "and the percentage of positives is calculates as $\\pi_1=\\vert \\mathcal{S}_1 \\vert / \\vert \\mathcal{S} \\vert.$\n",
    "\n",
    "Then, the impurity of each leaf is calculated using either a misclassification error, \n",
    "entropy or Gini measures:\n",
    "\n",
    "\n",
    "a) Misclassification: \n",
    "    $I_m(\\pi_1)=1-max(\\pi_1,1-\\pi_1)$ \n",
    "\n",
    "b) Entropy: $I_e(\\pi_1)=-\\pi_1\\log \\pi_1 -(1-\\pi_1) \\log (1-\\pi_1)$ \n",
    "\n",
    "c) Gini: \n",
    "    $I_g(\\pi_1)=2\\pi_1(1-\\pi_1)$\n",
    "\n",
    "Finally the gain of the splitting criteria using the rule $(\\mathbf{x}^j,l^j)$ is calculated as the \n",
    "impurity of $\\mathcal{S}$ minus the weighted impurity of each leaf:\n",
    "\n",
    "$$  Gain(\\mathbf{x}^j,l^j)=I(\\pi_1)-\\frac{\\vert \\mathcal{S}^l \\vert}{\\vert \\mathcal{S} \n",
    "\\vert}I(\\pi^l_1)  -\\frac{\\vert \\mathcal{S}^r \\vert}{\\vert \\mathcal{S} \\vert}I(\\pi^r_1), $$\n",
    "\n",
    "where $I(\\pi_1)$ can be either of the impurity measures $I_e(\\pi_1)$ or $I_g(\\pi_1)$.\n",
    "\n",
    "Subsequently, the gain of all possible splitting rules is calculated. The rule with maximal \n",
    "gain is selected\n",
    "\n",
    "$$  (best_x, best_l) = argmax_{(\\mathbf{x}^j,l^j)}Gain(\\mathbf{x}^j,l^j), $$\n",
    "\n",
    "and the set $\\mathcal{S}$ is split into $\\mathcal{S}^l$ and $\\mathcal{S}^r$ according to that rule. \n",
    "\n",
    "\n",
    "- Tree growing\n",
    "\n",
    "In order to grow a tree typical algorithms use a top-down induction using a greedy search in each\n",
    "iteration [16]. In each iteration, the algorithms evaluates all possible splitting \n",
    "rules and pick the one that maximizes the splitting criteria. After the selection of a splitting \n",
    "rule, each leaf is further selected and it is subdivides into smaller leafs, until one of the \n",
    "stopping criteria is meet. \n",
    "\n",
    "- Cost-sensitive impurity measures\n",
    "\n",
    "Standard impurity measures such as misclassification, entropy or Gini, take into account the \n",
    "distribution of classes of each leaf to evaluate the predictive power of a splitting rule,\n",
    "leading to an impurity measure that is based on minimizing the misclassification rate. However, \n",
    "as has been previously shown [8], minimizing misclassification does not \n",
    "lead to the same results than minimizing cost. Instead, we are interested in measuring how good \n",
    "is a splitting rule in terms of cost not only accuracy. For doing that, we propose a new \n",
    "example-dependent cost based impurity measure that takes into account the cost matrix of each \n",
    "example.\n",
    "\n",
    "We define a new cost-based impurity measure taking into account the costs when all the examples\n",
    "in a leaf are classified both as negative using $f_0$ and positive using $f_1$\n",
    "\n",
    "$$    I_c(\\mathcal{S}) = \\min \\bigg\\{ Cost(f_0(\\mathcal{S})), Cost(f_1(\\mathcal{S})) \\bigg\\}. $$\n",
    "\n",
    "The objective of this measure is to evaluate the lowest expected cost of a splitting rule.\n",
    "Following the same logic, the classification of each set is calculated as the prediction that \n",
    "leads to the lowest cost\n",
    "\n",
    "$$\n",
    "f(\\mathcal{S}) = \n",
    "\\begin{cases}\n",
    "  \\phantom{-}0 \\phantom{-} \\mbox{if} \\phantom{-} Cost(f_0(\\mathcal{S})) \\le \n",
    "Cost(f_1(\\mathcal{S}))\\\\\n",
    "  \\phantom{-}1 \\phantom{-}\\mbox{otherwise}\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "Finally, using the cost-based impurity, the splitting criteria cost based gain of using the \n",
    "splitting rule $(\\mathbf{x}^j,l^j)$ is calculated. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              f1       pre       rec       acc       sav\n",
      "RF      0.234356  0.509532  0.152174  0.931949  0.129533\n",
      "DT      0.248201  0.238436  0.258799  0.892699  0.184151\n",
      "LR      0.071775  0.569231  0.038302  0.932197  0.028994\n",
      "RF-BMR  0.279198  0.177352  0.655797  0.768253  0.434677\n",
      "DT-BMR  0.134468  0.079447  0.437371  0.614652  0.100415\n",
      "LR-BMR  0.207882  0.130423  0.511905  0.733005  0.265352\n",
      "CSDT    0.286539  0.174688  0.796584  0.728506  0.502330\n"
     ]
    }
   ],
   "source": [
    "from costcla.models import CostSensitiveDecisionTreeClassifier\n",
    "\n",
    "classifiers[\"CSDT\"] = {\"f\": CostSensitiveDecisionTreeClassifier()}\n",
    "# Fit\n",
    "classifiers[\"CSDT\"][\"f\"].fit(X_train, y_train, cost_mat_train)\n",
    "# Predict\n",
    "classifiers[\"CSDT\"][\"c\"] = classifiers[\"CSDT\"][\"f\"].predict(X_test)\n",
    "# Evaluate\n",
    "results.loc[\"CSDT\"] = 0\n",
    "results.loc[\"CSDT\", measures.keys()] = \\\n",
    "[measures[measure](y_test, classifiers[\"CSDT\"][\"c\"]) for measure in measures.keys()]\n",
    "results[\"sav\"].loc[\"CSDT\"] = savings_score(y_test, classifiers[\"CSDT\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print (results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAEvCAYAAACT2dMvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3gU5f428Pu7LZtsEkiDBAiEKj0ISBNQihSxV2yAhyICggUlFhCxAZYjIkXhKGA5cg4efVV6F0WFSJGugCCEBFII6dnd7PP+kcXfEgMkZJMn2dyf68pFdnZ25s6wm9w7+8yMKKVARERERKSLQXcAIiIiIqreWEiJiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrk64Vh4eHq5iYGF2rJyIiIiqxX375JUUpFaE7h6/SVkhjYmIQHx+va/VEREREJSYix3Vn8GX8yJ6IiIiItGIhJSIiIiKtWEiJiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrFlIiIiIi0qpEhVREBojIIRE5LCJxxdw/TESSRWSX+2uE96MSERERkS+67JWaRMQIYA6AGwCcBLBdRL5WSu0vMutSpdS4cshIRERERD6sJHtIOwE4rJQ6qpSyA/gcwK3lG4uIiIiIqouSXMu+LoATHrdPAuhczHx3ikhPAL8BeEIpdaKYeYiIiKgcxMQt1x2hTI5NH6Q7AmnkrYOavgEQo5RqC2AtgMXFzSQio0QkXkTik5OTvbRqIiIiIqrKSlJIEwBEe9yu5572F6VUqlIq331zIYAOxS1IKfWBUqqjUqpjRETEleQlIiIiIh9TkkK6HUBTEWkoIhYAgwF87TmDiER53LwFwAHvRSQiIiIiX3bZMaRKKaeIjAOwGoARwIdKqX0iMg1AvFLqawDjReQWAE4AaQCGlWNmIiIiIvIhJTmoCUqpFQBWFJk2xeP7ZwE8691oRERERFQd8EpNRERERKQVCykRERERacVCSkRERERasZASERERkVYspERERESkFQspEREREWnFQkpEREREWrGQEhEREZFWLKREREREpBULKRERERFpxUJKRERERFqxkBIRERGRViykRERERKQVCykRERERacVCSkRERERasZASERERkVYspERERESkFQspEREREWnFQkpEREREWrGQEhEREZFWLKREREREpBULKRERERFpxUJKRERERFqxkBIRERGRViykRERERKQVCykRERERacVCSkRERERasZASERERkVYspERERESklUl3ACKqXGLiluuOUCbHpg/SHYGIiEqJe0iJiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrFlIiIiIi0oqFlIiIiIi0YiElIiIiIq1YSImIiIhIKxZSIiIiItKKhZSIiIiItGIhJSIiIiKtWEiJiIiISKsSFVIRGSAih0TksIjEXWK+O0VEiUhH70UkIiIiIl922UIqIkYAcwAMBNASwH0i0rKY+YIATADws7dDEhEREZHvKske0k4ADiuljiql7AA+B3BrMfO9DGAGgDwv5iMiIiIiH1eSQloXwAmP2yfd0/4iIu0BRCullnsxGxERERFVA2U+qElEDADeBvBUCeYdJSLxIhKfnJxc1lUTERERkQ8oSSFNABDtcbuee9p5QQBaA9gkIscAdAHwdXEHNimlPlBKdVRKdYyIiLjy1ERERETkM0pSSLcDaCoiDUXEAmAwgK/P36mUOqeUCldKxSilYgD8BOAWpVR8uSQmIiIiIp9y2UKqlHICGAdgNYADAP6jlNonItNE5JbyDkhEREREvs1UkpmUUisArCgybcpF5r2+7LGIiIiIqLrglZqIiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrFlIiIiIi0oqFlIiIiIi0YiElIiIiIq1YSImIiIhIKxZSIiIiItKKhZSIiIiItGIhJSIiIiKtWEiJiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrFlIiIiIi0oqFlIiIiIi0YiElIiIiIq1YSImIiIhIKxZSIiIiItKKhZSIiIiItGIhJSIiIiKtWEiJiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrFlIiIiIi0oqFlIiIiIi0YiElIiIiIq1YSImIiIhIKxZSIiIiItKKhZSIiIiItGIhJSIiIiKtWEiJiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrFlIiIiIi0qpEhVREBojIIRE5LCJxxdw/WkT2iMguEfleRFp6PyoRERER+aLLFlIRMQKYA2AggJYA7iumcH6mlGqjlGoHYCaAt72elIiIiIh8Ukn2kHYCcFgpdVQpZQfwOYBbPWdQSmV43LQBUN6LSERERES+zFSCeeoCOOFx+ySAzkVnEpGxAJ4EYAHQ2yvpiIiIiMjnee2gJqXUHKVUYwCTALxQ3DwiMkpE4kUkPjk52VurJiIiIqIqrCSFNAFAtMfteu5pF/M5gNuKu0Mp9YFSqqNSqmNERETJUxIRERGRzyrJR/bbATQVkYYoLKKDAdzvOYOINFVK/e6+OQjA7yAiIiLyYb/88kstk8m0EEBr8FSal+ICsNfpdI7o0KHDmeJmuGwhVUo5RWQcgNUAjAA+VErtE5FpAOKVUl8DGCcifQE4AJwFMNRrPwIRERFRJWQymRZGRka2iIiIOGswGHhA90W4XC5JTk5umZSUtBDALcXNU5I9pFBKrQCwosi0KR7fTyhLUCIiIqqeTME74RexGm0XP4tIWyQmtJ+AQY0G6Y5VUq1ZRi/PYDCoiIiIc0lJSa0vNk+JCikRERGRt5mCd8Ia9T+IwQEFIDE7EVO3TgWAqlJKDSyjJePeThcd1sDxDkRERKSFX8RqiMFxwbS8gjzM2jFLUyLShYWUiIiINHBBzOnF3pOUnVTBWaouo9HYoXnz5i3Pfx06dMiSlJRk7Ny5c7OAgICrhwwZUt9z/nfeeSesWbNmLZs1a9ayadOmrT755JOaurJ74kf2ROQTqvg4NKJqRMEYeAB+EWshUvwckbbIio1UQZYeWho6f/f8uqm5qZYw/zD76NjRCfdedW9aWZbp5+fnOnjw4H7PaRkZGYZp06ad2r17t//evXv9z08/cuSI+a233oratWvXgbCwsIJz584ZEhMTy9QFHQ4HzGZzWRYBgHtIicgHnB+HZrCkQ0H9NQ5t+dHluqMR0V8UjLZDCIiZg4DoJRCDHfa0zlCuC8uM1WjFhPa+d6z00kNLQ2dun9kgJTfFoqCQkptimbl9ZoOlh5aGentdwcHBrv79+2dZrVaX5/TExESzzWZz1ahRowAAatSo4WrevLkdAPbu3evXrVu3ZldddVXLli1btti3b5+fy+XCI488Uq9p06atmjVr1nLBggUhAPDtt98GdejQ4arevXs3adq0aWsAmDt3bmibNm1aNG/evOX999/fwOl0lioz95ASUZV3qXFo3EtKpJuC0XYYfuFrYQz4Ey57CHJP3QnnufYAjCjIjYFfxGoYLeeq9Kcbk3+YHH347OGAi91/8OxBm9PlvGCfsL3Abpi+bXrMV79/VezVgpqENMl5+dqXTxR333n5+fmG5s2btwSA6Ojo/LVr1x652LxdunTJCQ8Pd0RHR7e59tprM++4446z999//zkAuP/++xtOnDgxaciQIek5OTlSUFAgS5Ysqblnzx7/AwcO7EtMTDR16tSpRb9+/bIAYP/+/QE7d+7c17x5c/uOHTusy5YtC42Pjz/o5+enHnzwwfrz588PGzduXOqlsntiISWiKo/j0IgqJ2PAEVgi1sIUcAwuRw3kJd4OR3oHeNYPZ8bVcGZcjWPTq14JLY2iZfRy00uquI/sL8ZkMuG77777ffPmzQFr1qwJjouLi46Pj7dNnjw56fTp05YhQ4akA0BAQIACoLZs2RJ0zz33pJlMJkRHRzs7d+6c9f333wfUqFHD1bZt2+zze1dXrVoVtHfv3oDY2NgWAJCXl2eoVatWqXaRspASUZWnnMEQc8bfptcKqKUhDREZ/f8oLKK2o3A5gpGXdCsc6dcAyndrx+X2ZPb6T682KbkplqLTw/3D7f++6d+Hyi/ZhQwGA3r16pXTq1evnIEDB2aMGDEiZvLkyaV+9x4QEPDXcACllNx9992pc+bMudSl5S+d60ofSERUORQALjNUMWcCVEohOSe54iMRVVMG/+Pwj/4XAmLeh8HvDPKSbkL2kafhONvVp8toSYyOHZ1gMVouGNNpMVpco2NHX3GJK61jx46Zv//++7+GFcTHxwfUrVvXHhIS4oqMjLR//PHHNQEgNzdXMjMzDT179sxctmxZqNPpxKlTp0zbtm0L7NGjR3bR5Q4YMCDj22+/DUlISDABwOnTp42//fbb38r3pVTvZwcRVXmWiDUw+KXCntYFpsCDf41DG9RwED47+BmGrByCBf0WoF5QPd1RiXyWwXoCfhHrYAo8BJfThrzTN8JxtgugStVJfNr5o+m9fZT9xdStW7dNVlaW0eFwyOrVq2uuWLHit6CgINfEiRPrnT592uzn56dCQ0MdCxYs+BMAPvnkkz9GjhzZ4OWXX65jNpvVf//73yMPPfRQ+tatWwNbtGjRSkTUSy+9dLJ+/frOX3/99YJ1dejQIe+FF15I6NOnTzOXywWz2azefffdP5s1a2YvaV5Rxe1WqAAdO3ZU8fHxWtZNRBcXE1d1jkw32g4hoP5HsJ/thPykOwDggnFoe5L34NH1j8JisOCDGz5Ak5AmuqISlTsdr12DX0JhEQ06AOUMgD2tJ+xpXQHlV+plVfYxpCLyi1Kqo+e03bt3H4uNjU3Rlamq2b17d3hsbGxMcffxI3siqpLEdA7WOv9BQV4k8k/fXOw8bSLa4KP+HwEAhq0ehr0peysyIpHPMvglwVr3Y9gazYYx4A/kn7kBWUeegT31+isqo0QspERUBRXAWvffEIMDeQn3A+riJ2VuGtIUiwcuRqA5EMNXD8f2pO0VmJPItxgsp2Gt+xlsjd6ByXYY+cl9kHV4EuypfQCXVXc8qsI4hpSIqhxLxDqYAo4hN+EeuOyXP5I+OigaSwYuwag1ozB67Wi8df1buD76+vIPSqVWlYaMFKeyf+x8pcSSDL/w9TAF7wZcZuSn9II9tQfguuhpN4lKhXtIiahKMdp+hyVsExzpHeDMaF/ix9UKqIVFAxahWUgzPL7xcXx79NtyTEnkG8ScCmvUf2Br9DZMQftgT+2J7COTYE/uzzJKXsU9pERUZYgpA9Y6n8Nlj0Be0q2lfnxNa00s7L8Q4zeMx3NbnkOWPQuDmw8uh6REVZuY02AJ3wBzjR2AMsCRdi3sqddBFQTpjkY+ioWUiKoIF6x1PocY7Mg9PuqKTydjM9swt+9cTNw8Ea/+/CqyHFkY3no4RMp0sRQinyCmdFjCN8JcMx5QgONsF9hTr4dyBuuORj6OhZSIqgRL+AaYbEeRe+ouuOy1y7QsP6Mf3r7+bUz+YTJm7ZiFjPwMPNHhCZZSqrbElAFL2EaYa24DBHCkXwN7Si8oZw3d0aqUmLjlHby5vGPTB/1SkvkmTZoU+cUXX4QZDAZlMBgwd+7c47179/7bCewv5d57723wzDPPnO7QoUPelaUtGxZSIqr0jAFHYAlfD0f61XCe887ve7PBjNe6v4YgcxA+2vcRMuwZmNxlMowGo1eWT1QViDETlrBNMIf8DIgLjvSO7iIaojsaldC6detsq1evrrlnz579/v7+KjEx0ZSfn1/qd9dLly49Xh75SooHNZFXLT+6HP2W9UPbxW3Rb1k/LD9atY+YJf3EmAlr3c/hsocjL+k2AN7bi2kQA57r/BxGthmJL37/ApO2TIKjwOG15RNVVmLMgl+tFbA1mQlz6I9wZsQi+8hTyE+6g2W0iklISDCHhoY6/f39FQBERUU5Y2JiHBMnToxq3bp1i6ZNm7a67777GrhcLuzcudPapk2bFucfe+jQIUuzZs1aAkCnTp2u+u677wIAICAg4OrHHnus7lVXXdUyNja2+YkTJ0wAsG/fPr/Y2NjmzZo1azl+/Pg6AQEBVwPA8ePHzR07dryqefPmLZs2bdpq1apVgaX9OVhIyWuWH12OqVunIjE7EQoKidmJmLp1KksplYEL1jr/gRhy3ecb9f4Jt0UE49uPx1MdnsLqY6sxfuN45Dpzvb4eokrBmA1LxCp3Ed0CZ2ZrZB95EnmJd0M5wnSnoytw2223ZZw6dcoSExPT+sEHH6y/fPnyQAB4+umnz+zdu/fA77//vi83N9fw+eef17j66qvzHA6HHDx40AIAS5YsCb3tttvOFl1mbm6uoWvXrlmHDh3a37Vr16zZs2dHAMC4ceOix4wZc+a3337bX69evb/evX/44Yehffr0OXfw4MH9Bw4c2Ne5c+ec0v4c/MieSiXXmYvU3FSk5qVe8G9aXhq+/P1L5BVcOPQkryAPs3bMwqBGvnluPipflrBNMAX+jrzE2+HKjyrXdQ1rPQxBliC89ONLGL12NN7r8x6CLDyimHyEIReW0C2whP4AGOxwZrSFPaVPic7jS5VbjRo1XHv37t2/atWqoPXr1wcNHTq08ZQpU04GBwcXvP3225F5eXmG9PR0U8uWLXMBnLvtttvSlixZEvraa68lffnllyFLly49WnSZZrNZDR48+BwAdOjQIXvdunXBALBz587ANWvWHAaAESNGpE6dOrUeAHTp0iX7kUceiXE4HIa77rrrbLdu3Ur9rp6FtJpTSiHbkf23gulZND1v5ziLf9MTZAn6Wxk9Lyk7qTx/BPJRRv+jsESsheNcLBzpnSpknXc2uxOBlkDEbYnD8NXDMa/vPIT5c68RVWGGPFhCf4AldAvEmAdHRpvCIpofqTsZeZHJZMJNN92UedNNN2W2bds2d8GCBeGHDh0K+Pnnn/c3adLE8eSTT9bJy8szAMBDDz109u677240ePDgsyKCNm3a5BezPGUwGP5attPpvORYqYEDB2Z99913h7744osa//jHPxqOGzfu9Lhx41JL9TOUZmaqGpRSyLBnFFswU/NSkZabdsHt/IK/PRchENT0q4kw/zCEWcPQOrw1wqxhf932/DfUGgqL0YJ+y/ohMTvxb8uKtPEXH5WOGLNgrfs5lCMUeUm3w5vjRi+nf0x/2Mw2PLHxCQxbNQwL+i3gc5iqHsmHJXRrYRE15cCR2RL25L5w5dfRnYy8bPfu3X4Gg+GvYrlz507/Jk2a5B86dCggMjLSee7cOcM333wTcvPNN58FgFatWuUbDAZMmTKlzu23355WmnW1a9cua9GiRSEjR448++GHH4aen/7bb79ZGjVqZH/qqadS8vPzZceOHQEAWEh9UYGrAOn56Rfdk+lZNNPy0uB0Of+2DKMYEWIN+atIxgTH/K1Ynr8dYg2ByVC6p8eE9hMwdevUC/aUCgQj2owo889P1Yl73KgxGznHxmi5Pnb3ut3x/g3vY+z6sRiycgg+uOEDxNSIqfAcRKUmdphDfoQl7DsYTNlwZjZHfkpfuPLq6U5WLZT0NE3elJGRYRw/fnz9jIwMo9FoVDExMfmLFy8+XrNmTWeLFi1aRUREOGNjYy84BdQdd9yR9vLLL9ebMWNGQmnWNXv27BMPPPBAwzfeeCOqd+/eGYGBgQUAsHr16qB333030mQyqYCAgIJPP/30j9L+HKKUKu1jvKJjx44qPj5ey7orC4fLgbN5Zy9aMD0/Mj+bfxYu5frbMkwG0yX3XHrerulXEwYp3+PYlh9djlk7ZiEpOwmh1lCk56WjTUQbLOi3AFZTxRcLKj3d1xK3hG2CX61VyEu6FY6zXUv9eG9eS/xA6gGMXjcaAPD+De+jeWhzry2biqf7+VdWuq5ln+fMQ5u3XoIlfBMMpiw4s5ohP7kvXHn1teS5Erq2XUmJyC9KqY6e03bv3n0sNjY2RVemipaZmWmw2Wwug8GADz74IGTp0qWh69evP1LSx+/evTs8NjY2prj7uIe0CM9CFWmLxIT2E0p1QI69wH7Jgum5JzM9P73YZViN1r+KZJ3AOmgT3qbYghlmDUOwJbhSncx7UKNBF2yvNcfWYOLmiXju++fw5nVvlnshpqrN6H8Mlog1cGS0geNsF91x0CKsBRYNWIRRa0fhH6v+gTl95+DqWlfrjkX0F3uBHct+W4aFexbCGpkMZ3YT5J3si4LcGN3RyAf98MMPARMmTKivlEJwcHDBokWLjnlr2SykHs6ftuj8R87nT1uU78zHNVHXXPagn7TcNGQ6Motdts1s+6tINqzREB0jOxYWzPPl0qNoBpgCKlXJLIt+Mf3wVPZTeDP+TbwZ/yaeueYZ3ZGosjJmw1r331COmshLvBMVOW70UhrWaIglA5Zg1NpReGTtI3jn+nfQrW433bGomnMUOPDl4S/xwa8f4HTOabSv1R7HD96OgpxGuqORDxswYEDWoUOH9pfHsllIPczaMavY0xa9+OOLxc4fbAn+q0g2D22OMGsxezHd31fnj6uHtByCxOxEfLz/Y9Sx1cGDLR/UHYkqHQX/qP9CTFnIOfaolnGjlxIVGIWPBnyE0WtHY+yGsZjRYwb6xfTTHYuqIYfLgW+OfIP3d7+PU9mnEBsRi1e6v4LOkZ3RcPMK3fGqI5fL5RKDwaBn/GMV4nK5BMDfxx66sZB6uNTpiaZ1m3bhnkxrGMxGcwWmq7pEBE93fBpJ2UmYuX0momxR6NOgj+5YVImYQ7+HKegg8pJurrQHX4T7h+PDAR9i3PpxePq7p5HtyMbtTW/XHYuqCafLieVHl2P+7vk4mXUSrcNaY0rXKehWp5vPfKJWRe1NTk5uGRERcY6l9OJcLpckJyfXALD3YvOwkHqItEUWe9qiKFsU//CUkdFgxOs9XseINSMwacskLPRfiHa12umORZWAwfon/GqthCOjFRxnK/dH4cGWYMzvOx9PbnoSU7ZOQaY9E0NaDdEdi3xYgasAq46twvzd83Es4xhahLbAe73fQ896PVlEKwGn0zkiKSlpYVJSUmvw6peX4gKw1+l0XvS0OyykHoo7bZHVaMWE9hM0pvId/iZ/zO49Gw+teAiPbXgMn9z4CRoEN9Adi3Qy5MC/7mdQjhqVatzopQSYAzC792xM2jIJb8S/gQx7Bsa2G8tyQF7lUi6sOb4G83bNw9FzR9EspBne6fUOekf35nOtEunQocMZALfozuEL2OY9DGo0CFO7TUWULQoCQZQtClO7TeVlL70o1BqKeX3nQSB4dN2jSMsr1Tl5yacoWOssg5gzkJtwH+AK0B2oxMxGM97o+QbuaHoH3v/1fUzfNr3Y07IRlZZLubDu+Drc+fWdeHrz0xAI3rruLfz35v+iT/0+LKPks7iHtIiipy0i76sfXB+z+8zG8NXD8dj6x7Cw/0L4m/x1x6IKZg7ZCnPQfuSdHlSlzpV4ntFgxNSuUxFoDsSS/UuQac/EtGunlfqCEkRA4RX2Np/cjDm75uBg2kHEBMdgRo8Z6B/TH0aDUXc8onLH35ykRWxELGb0mIEnNj2BuO/i8Pb1b/OXbjVisJ6EX+0VcGY2hyOtu+44V0xEMLHjRARbgvHerveQ7cjGzOtmws/opzsaVRFKKXyf8D3m7JqDfan7EB0Ujde6v4aBDQfyzQ1VK/zInrTp06APJnWahA0nNmDm9pnQddUwqmCGvMJxo85A5CbejaowbvRSRASPxD6CZzs9iw0nNmDsurHIceTojkWVnFIKW09txYMrH8SY9WOQnp+Oad2m4evbvsbNjW9mGaVqh8940uqBFg8gISuh8BylgXUwtNVQ3ZGoXClYo76AmNORc/wRoMCmO5DX3N/ifgRZgjD5h8kYuWYk5vadixp+NXTHokpoe9J2vLfzPew4swORtkhM6ToFtzW+jacSpGqNhZS0m9hxIpKyk/Bm/JuItEWif0x/3ZGonJhDfoI5eA/yzwyAK9f3zrBwc+ObYTPbMHHzRAxbNQwf3PABIgIidMeiSmLH6R2Ys2sOtiVtQy3/Wni+8/O4o+kdsBgtuqMRaceP7Ek7gxjwWvfX0C6iHZ7b8hx2nN6hOxKVA4NfAvxqfQtn1lWwp/bUHafc9K7fG/P6zkNCVgKGrByCk5kndUcizXYn78aoNaMwdNVQHEk/grhOcVhx5woMbj6YZZTIjYWUKgWryYrZvWcjKjAK4zeOxx/n/tAdibzJkAf/ep9BFdiQd+pu+Pqvns5RnbGw30Jk2DMwdGVhCaHqZ1/KPoxZNwYPrngQh84ewsSOE7HyzpV4oMUDPPCNqAjf/qtAVUpNa03M6zMPRjHi0XWPIiU3RXck8goFa+SXEHMa8hIGQxUE6g5UIdpGtMWiAYugoDBs1TDsTbnoFfPIB5iCd8LWeDraLm6LXv/phXu+uQeDlw/Grym/4vH2j2PlHSsxtNVQnuKO6CJYSKlSiQ6Oxnu930NqbioeW/8Yj1b2Aeaa22CusRv25BtQkNtId5wK1TSkKRYPWAyb2Ybhq4dje9J23ZGoHJiCd8Ia9T8YLOlQUEjJTcGBtAPo16AfVt2xCsPbDEeAuepc+IFIBxZSqnTaRLTBzJ4zsT9tPyZ9NwlOl1N3JLpCBr9E+NX+Bs6sprCnXq87jhbRwdFYPGAxomxRGL12NDad2KQ7EnmZX63lEIPjb9P3pOxBoKV6fCJAVFYlKqQiMkBEDonIYRGJK+b+J0Vkv4j8KiLrRcT3Dp+lCtWrfi/EdYrDppObMH3bdJ6jtCqSfFjrfgbl8kfeqXtQnd//1rbVxqIBi9AspBke3/g4vj36re5IVGYKxoDD8I9eCIM5q9g5krKTKjgTUdV12b8QImIEMAfAQAAtAdwnIi2LzLYTQEelVFsAywDM9HZQqn7ua34fHm71MJYeWopF+xbpjkOlomCN/AoGS4p73GiQ7kDa1bTWxML+C9G+dns8t+U5fH7wc92R6Iq4YArch4CYuQhosBAG62m4CqzFzhlpi6zgbERVV0l2WXQCcFgpdVQpZQfwOYBbPWdQSrJcZToAAB7dSURBVG1USp0f7PcTgHrejUnV1eMdHseAmAF4+5e3sfKPlbrjUAmZavwCc82dsKf0QUFOY91xKg2b2Ya5febiunrX4dWfX8XCPQu597/KKIApeCcCGs6Cf/THEGM28hJvR/bhZ5CfdCuU68KT2luNVkxoP0FTVqKqpyQnxq8L4ITH7ZMAOl9i/uEA2BzIKwxiwCvdX0FybjKe//55RPhHoGNkR92x6BIMltOwRv4/OLMbw57SW3ecSsdqsuLtXm9j8g+TMWvHLGTYM/BE+ycgUrUvoeqzxAFzjV9gCfsOBksaCvJqIzdhMJwZbQAYAQDOjKuRB8AvYjWMlnOItEViQvsJGNRokNboRFWJV6/UJCIPAugI4LqL3D8KwCgAqF+/vjdXTT7Mz+iHWb1m4aGVD2H8xvH4ZOAnaFSzeh2tXWWIHdZ6n0K5/JB36l5U53Gjl2I2mPFa99cQaA7ER3s/QqY9Ey90fgFGg1F3NDrPkAdLzZ9hDtsCgykLBTn1kXP6JhRkNUdxz2tnxtVwZlyNY9NZQomuREn+WiQAiPa4Xc897QIi0hfA8wBuUUrlF7cgpdQHSqmOSqmOERG8nB6VXA2/GpjbZy4sBgvPUVqJWSP/HwyWZOSduhfKGaw7TqVmEAOe7/w8RrYZiWW/LUPcljg4Cv5+pDZVLDFmwxKxBoFNpsOv9kq48qKQc3wkco4/ioKsluCbLKLyUZJX1nYATUWkoYhYAAwG8LXnDCJyNYD3UVhGz3g/JhFQL6ge5vSZg7P5ZzFm3Rieo7SSMQXvgLnmL7CnXo+C7Ka641QJIoLx7cfjyQ5PYtWxVRi/cTxynbm6Y1VLYkqHX+1vYGsyHZawjXBmN0b2H+OQe2K4exw0h1QQlafLFlKllBPAOACrARwA8B+l1D4RmSYit7hnewNAIID/isguEfn6IosjKpNW4a3w5nVvFl6Gb/NEnqO0kjBYzsAa9RWcOTGwJ/fVHafKebj1w3ix64v4IeEHjF47Gpn2TN2Rqg2xJMMvahlsTd6AOeRHODPaIOfoE8hLeAiuPB6fS1RRSjSGVCm1AsCKItOmeHzPv0BUYXrW64nnOz+Pl396Ga/+/CqmdJnCA0J0Eof7fKNm5CXch/MHelDp3NXsLgRaAvHslmcxfPVwzL9hPkKtobpj+SyDXwIs4ZtgCtoLKCMcZzvBntoTyhmiOxpRteTVg5qIKso9V92DxOxELNyzEHUD62JEmxG6I1VbfrW/gdGahJw/H4Zy1tAdp0obEDMANpMNT256EkNXDsWCfgt4LksvM/r/AUv4RpgCf4Mq8IM99To40rpDFfCKSkQ6cXQ2VVmPXf0Ybmx4I2btmMUr32hiCt4FS8g25Kdch4Lsq3TH8Qk96vXA/BvmIyU3BUNWDsHxjOO6I/kABWPgQfg3mIeAmPdhsCYg/0x/ZB1+FvbkASyjRJUACylVWQYx4OVrX8Y1kddg8g+TsS1xm+5I1YqYU2CN/B8KchrAntxPdxyf0qF2B/yr/7+Q58zDkJVDcCjtkO5IVZQLpuDdCGj4LgKiF8FgPoe8pFuQfXgS7Km9AFfxV1gioorHj+wrqZi45bojlElFnYvPYrTgnV7vYMiKIXh84+NYPHAxmobwCO9yJw741/sUUCbkctxouWgZ1hKLBi7CqDWj8PCqhzG371y0q9VOd6yqQZww19gBS9hmGCypKMiPQO6pu+E81w58rhJVTtxDSlVesCUY8/rOg9VkxZj1Y3Amh2ceK29+tZbDaE1EbuLdUM6auuP4rEY1GmHJwCUI9Q/FqLWjsDVhq+5IlZvkwxy6BbbGM2GN+h9UgT9yTz6InKNPwHmuA1hGiSovFlLyCVGBUZjTZw4y8jMwdv1YZDuydUfyWaagPbCE/gR7ag8UZLXQHcfn1Qmsg0UDFqF+UH2M3TAWa4+v1R2p8jHkwBK+DoFNZsBaezlc9nDk/DkcOcfGwpnZGvxTR1T58VVKPqNFWAu8df1b+P3s73hq01NwuHjVG28TcyqsUctQkBuN/DMDdMepNsL9w/HhgA/ROqw1Jm6eiC9//1J3pEpBTBnwq7W88KpKEevgzG2A7D/GIPfPUe6LM/B0cERVBQsp+ZTudbtjStcp+OHUD3jlp1eglNIdyXeIE/51PwMgHDeqQbAlGO/f8D66RHXBlK1TsGTfEt2RtBFzKvwi/wdb4xkwh34PZ1ZLZB99HHknh8KVV193PCK6AjyoiXzOHU3vwKmsU3j/1/cRZYvC6NjRuiP5BL9aK2D0T0DuiYegHDxhuw4B5gDM7j0bcVvi8Eb8G8h0ZGJM7Jhqc2EIg18iLGGbYAr+tfBk9uc6Fp7M3hGmOxoRlRELKfmkse3GIjE7EXN2zUGULQq3NrlVd6QqzRS0F5bQrbCndYMzq5XuONWaxWjBzJ4zMe3HaZi/ez4y8jMwqdMkGMR3P/Ay+B+HX9hGmIIOQhVY4EjrAXtadyhnsO5oROQlLKTkk0QEU7tOxemc05i6dSpqBdRC1zpddceqksSc5h43Whf5Z27UHYcAmAwmvNTtJQRaAvHx/o+R5cjCS91egsngS7/SFYy232EJ2wiT7Q+4nAHIT74B9rSugCtAdzgi8jJf+u1FdAGz0Yx/Xv9PDFk5BE9segKLByzGVaG8mlDpOOFf998AFHIT7gcUf2VUFiKCpzs+jWBLMObsmoMsexZmXjcTfkY/3dHKyAVT0D5YwjbB6J8AlyMYeUk3wZHeCVAW3eGIqJz47mc8RACCLEGY13cebGYbxqwfg6TsJN2RqhS/Wqtg9D+BvMS7OE6vEhIRjI4djbhOcdhwYgPGrh+LHEeO7lhXxFHgwJe/f4mARv+Ef71PIYZ85J26E9lHnoHjbHeWUSIfx0JKPi/SFom5feYi25GNMevHINOeqTtSlWAM3A9L2Pewp3WBM7ON7jh0CQ+0eACvdn8V8UnxGLlmJM7ln9MdqcRynbn49MCnuPHLGzFl65TCq3+dvB/ZR5+E49w13CtPVE2wkFK1cFXoVXj7+rfxR/ofeHLTk3AU8ByllyKmdPjX+S8K8qKQf6ZiLgNLZXNL41vw1vVv4UDaAQxbNQzJOcm6I11Shj0DC35dgAFfDMD0bdNRx1YHc/vMRc4f4+HMbAv+eSKqXviKp2qjW51ueLHbi/gp8SdM/XEqz1F6UQXucaMFyD35AKDMugNRCfWp3wdz+85FQlYChq4aipOZJ3VH+puU3BS888s76L+sP97d+S5ahbXC4gGLsXjgYvSo1wM8mT1R9cTPQqhaua3JbUjMSsTc3XNRJ7AOxrYbqztSpWOJWANjwHHkJtwH5QjXHYdKqUtUFyzotwBj1o3B0JVD8UG/D9C4ZmPdsZCQlYBFexfhy8Nfwl5gR/+Y/hjeZjiahzbXHY2IKgEWUqp2RseOxqnsU5i/ez7q2Org9qa3645UaWw5uQV+4ZthP9sJzoxY3XHoCsVGxOKjAR/hkbWPYNiqYZjfdz5ahes5f+yR9CP4155/YcUfKyAiuLXxrXi49cNoENxASx4iqpxYSKnaERFM6ToFZ3LO4KUfX0KtgFq4tu61umNpdzr7NJ7//nkU5EUi//TNuuNQGTULaYYlA5Zg5NqRGL5mOGb3no1rIq+psPXvTdmLBb8uwIYTG+Bv8sf9Le7HkJZDEGmLrLAMRFR1cAwpVUtmgxlvXfcWmtRsgic3PYmDaQd1R9LK6XLime+eQV5BHnITOG7UV0QHR2PxgMWoHVAbo9eOxuYTm8t1fUop/Jz4M0asGYH7lt+H7ae3Y3TsaKy+czWeueYZllEiuigWUqq2Ai2BmNNnDoIsQRizbgwSsxJ1R9Jm7q652HFmByZ3mQxlj9Adh7yotq02Fg1YhKYhTfH4xsex/Ohyr6/DpVzY8OcGPLDiAYxYMwJH0o/gqQ5PYe1dazG23ViEWEO8vk4i8i0spFSt1bbVxry+85DrzMWY9WOQYc/QHanCbT21FQv3LMTtTW7HzY35Ub0vCrGGYGG/hWhXqx2e3fIslh5c6pXlOl1OfHPkG9z59Z2YsHEC0vLSMLnLZKy6cxWGtR4Gm9nmlfUQke9jIaVqr2lIU7zT6x0cyziGJzY+Ua3OUZqck4xntzyLRjUaIa5TnO44VI4CLYGY13ceetbriVd+fgUL9yy84mXlF+Rj6cGluOnLm/Dc988BAKb3mI5vb/8W91x1jw9cvpSIKhoLKRGAzlGdMa3bNGxL2oYpW6dUi3OUFrgKELclDjmOHLx53ZsIMAfojkTlzGqy4p+9/okbG96IWTtm4e1f3i7Vcz3LnoUP936I/sv645WfX0GYfxhm956NL275AoMaDYLJwONkiejK8LcHkdvNjW9GYnYiZu+cjShbFMa3H687Url6/9f3sS1pG6Z1m4YmIU10x6EKYjaY8XqP1xFkCcJHez9Cpj0TL3R+AUaD8aKPSctLw6cHPsW/D/4bmfZMdI3qipFtR6Jj7Y4Q4YnsiajsWEiJPIxsMxKnsk5hwZ4FiAqMwt3N7tYdqVz8nPgz5u+ej5sb3YzbmtymOw5VMIMY8Hzn5xFkCcLCPQuRZc/Ca91fg9l44dkVkrKTsHjfYiz7bRnyC/LRp34fjGgzQts5TYnId7GQEnkQEbzQ5QUk5STh1Z9eRe2A2uhZr6fuWF6VkpuCuC1xiKkRgxe6vMA9XNWUiGBC+wkIsgThn7/8E0fTjyLDnoHTOacR7h+OBkENsCtlF6CAGxvdiOGth6NRzUa6YxORj+IYUqIiTAYT3rruLTQLaYaJmydiX+o+3ZG8xqVceHbLs8i0Z3LcKAEA/tH6H7i9ye34Lf03JOUkQUEhOTcZ8WficU3ta7D8juV4tfurLKNEVK5YSImKYTPbMKfPHNT0q4lx68chIStBdySvWLhnIX5K/AlxneLQLKSZ7jhUSfyU+FOx049lHEOdwDoVnIaIqiMWUqKLiAiIwLy+85BfkI8x68bgXP453ZHKJD4pHnN2zcHAhgNxZ9M7dcehSiQpO6lU04mIvI2FlOgSGtdsjFm9ZuFE5glM2DgB9gK77khXJC0vDZO+m4TooGi82PVFjhulC1zskp681CcRVRRtBzXtSTiHmDjvX8LOW45NH6Q7AlUS10Reg1eufQWTtkzCC9+/gOk9p8MgVee9nEu58Nz3zyE9Px1z+s7h1XPobya0n4CpW6ciryDvr2lWoxUT2k/QmIqIqhMeZU9UAjc2uhGJ2Yl4Z8c7iAqMwhMdntAdqcQ+2vsRfkj4AS90fgHNQ5vrjkOV0KBGhW/AZ+2YhaTsJETaIjGh/YS/phMRlTcWUqIS+kfrfyAxOxEf7v0QdWx1cG/ze3VHuqydZ3Zi9s7Z6NegH+656h7dcagSG9RoEAsoEWnDQkpUQiKCuE5xSMpOwmvbXkNtW21cH3297lgXlZ6Xjqc3P40oWxSmdpvKcaNERFRpVZ2BcESVgMlgwsyeM9EitAWe+e4Z7E3ZqztSsZRSeOGHF5Cal4o3r3sTQZYg3ZGIiIguioWUqJQCzAF4r897CLWGYuz6sTiReUJ3pL9Zsn8JNp/cjIkdJ/Iyj0REVOmxkBJdgXD/cMztOxdOlxNj1o1Bel667kh/2Z28G+/88g761O+D+5vfrzsOERHRZbGQEl2hRjUaYXbv2TiVdQrjN45HfkG+7kg4l38Oz2x+BrVttfFSt5c4bpSIiKoEFlKiMmhfuz1e7fEqdp7Ziee2PAeXcmnLopTClB+m4EzOGczsORM1/Gpoy0JERFQaLKREZTQgZgCe6vAU1hxfg7fj39aW47ODn2HDiQ14vMPjaBvRVlsOIiKi0uJpn4i8YGiroTiVfQqL9y9GVGAUHmjxQIWuf1/KPrwZ/yaur3c9hrQcUqHrJiIiKqsS7SEVkQEickhEDotIXDH39xSRHSLiFJG7vB+TqHITEUy6ZhJ6RffCjG0zsP7P9RW27kx7JiZunohw/3C80v0VjhslIqIq57KFVESMAOYAGAigJYD7RKRlkdn+BDAMwGfeDkhUVRgNRszoOQOtw1tj0neTsDt5d7mvUymFF7e+iMTsRLzR8w2OGyUioiqpJHtIOwE4rJQ6qpSyA/gcwK2eMyiljimlfgWg74gOokrA3+SP2b1nI8I/Ao+tfwx/ZvxZrutbemgp1h5fi/Htx6NdrXblui4iIqLyUpJCWheA55m/T7qnEVExwvzDMK/vPCgoPLruUZzNO1su6zmQegAzt89E97rdMazVsHJZBxERUUWo0IOaRGQUgFEAYAyOqMhVE1WomBoxmN17NkasGYHHNjyGhf0Wwmqyem35WfYsTNw8ESHWELzW/TUYhCfMqCxi4pbrjlAmx6YP0h2BiKqhkvwVSwAQ7XG7nntaqSmlPlBKdVRKdTQGcKwb+bZ2tdrh9R6v49fkXxG3JQ4FrgKvLFcphWk/TkNCVgJm9pyJEGuIV5ZLRESkS0kK6XYATUWkoYhYAAwG8HX5xiLyDTc0uAFPX/M01v+5Hm/Gv+mVZS77fRlWHluJse3GokPtDl5ZJhERkU6XLaRKKSeAcQBWAzgA4D9KqX0iMk1EbgEAEblGRE4CuBvA+yKyrzxDE1UlD7V8CA+2eBCfHPgEH+//uEzLOpR2CDO2zUDXqK4Y3ma4lxISERHpVaIxpEqpFQBWFJk2xeP77Sj8KJ+IijGx40QkZSfhje1vINIWiRsa3FDqZeQ4cjBx80QEWYLwWg+OGyUiIt/Bv2hEFcBoMOL1Hq+jbURbPLvlWew6s6tUj1dK4eWfXsafmX9iRo8ZCPcPL6ekREREFY+FlKiCWE1WzO49G5G2SDy24TEcO3esxI/96vBX+PbotxjddjQ6RXUqv5BEREQasJASVaAQawjm9ZkHgxjw6LpHkZqbetnHHD57GK/9/Bo6R3bGqLajKiAlERFRxWIhJapg0cHRmN17NlJyU/DYhseQ68y96Lznx40GmAMwved0GA3GCkxKRERUMVhIiTRoG9EW03tOx96UvZj03aSLnqP09W2v4+i5o5jeYzrHjRIRkc9iISXSpE/9PojrFIeNJzZi+rbpUEpdcP83R77BV4e/wsi2I9G1TldNKYmIiMpfhV46lIgudH+L+3Eq6xQW71+MuoF1Maz1MADA0XNH8fJPL6ND7Q54NPZRvSGJiIjKGQspkWZPdnwSidmJeOuXt/Cvvf/CufxzMBqMsIgFM3rMgMnAlykREfk2fmRPpJlBDOhRrwcEgvT8dCgoOF1OOJUT8afjdccjIiIqdyykRJXA3F1zoXDhGFK7y45ZO2ZpSkRERFRxWEiJKoGk7KRSTSciIvIlLKRElUCkLbJU04mIiHwJCylRJTCh/QRYjdYLplmNVkxoP0FTIiIioorDw3eJKoFBjQYBAGbtmIWk7CRE2iIxof2Ev6YTERH5MhZSokpiUKNBLKBERFQt8SN7IiIiItKKhZSIiIiItGIhJSIiIiKtWEiJiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrFlIiIiIi0oqFlIiIiIi0YiElIiIiIq1YSImIiIhIKxZSIiIiItKKhZSIiIiItGIhJSIiIiKtWEiJiIiISCsWUiIiIiLSyqQ7AFF5iIlbrjvCFTs2fZDuCERERBWKe0iJiIiISCsWUiIiIiLSioWUiIiIiLRiISUiIiIirVhIiYiIiEgrFlIiIiIi0oqFlIiIiIi0YiElIiIiIq1YSImIiIhIKxZSIiIiItKqRIVURAaIyCEROSwiccXc7yciS933/ywiMd4OSkRERES+6bKFVESMAOYAGAigJYD7RKRlkdmGAzirlGoC4J8AZng7KBERERH5ppLsIe0E4LBS6qhSyg7gcwC3FpnnVgCL3d8vA9BHRMR7MYmIiIjIV5WkkNYFcMLj9kn3tGLnUUo5AZwDEOaNgERERETk20QpdekZRO4CMEApNcJ9+yEAnZVS4zzm2eue56T79hH3PClFljUKwCj3zasAHPLWD1IOwgGkXHYuuhhuvyvHbVc23H5lw+1XNtx+V66yb7sGSqkI3SF8lakE8yQAiPa4Xc89rbh5ToqICUANAKlFF6SU+gDAB1cWtWKJSLxSqqPuHFUVt9+V47YrG26/suH2KxtuvyvHbVe9leQj++0AmopIQxGxABgM4Osi83wNYKj7+7sAbFCX2/VKRERERIQS7CFVSjlFZByA1QCMAD5USu0TkWkA4pVSXwP4F4CPReQwgDQUllYiIiIiossqyUf2UEqtALCiyLQpHt/nAbjbu9G0qxJDCyoxbr8rx21XNtx+ZcPtVzbcfleO264au+xBTURERERE5YmXDiUiIiIirVhIAYhIgYjsEpG9IvKNiNR0T48RkVz3fee/LLrzViYe226fiOwWkadExCAi/T22WZb70rO7RGSJ7syViYhkFTNtqogkuLfXfhG5T0e2ilbW16GIbPJ4nh1wn2bu/H3HRGRLkfl3uU9ZBxG5XkTOuacdFJE3y/vn9SZvvA5FZJGI/OGxDV70uG+TiPzpecETEfnq/PO3yP/RfhFZIiLmivnpy66sr0OP7b9bRHaISDf39BgRUSLyise84SLiEJH3SrueqkZEIkXkcxE5IiK/iMgKEWkmIu+6X+d7RGS7iDR0z3/MPW2Pe1u8IiJWEWnj8TxO83iertP9M5IXKaWq/ReALI/vFwN43v19DIC9uvNV5q8i264WgHUAXioyzyYAHXVnrYxfntvPY9pUABPd3zcFkAHArDtrRW6LK3kdej7PAIQCOAvA4r59DMAuANHu2y3ct/e6b18P4Fv39/4ADgK4Vvc2ucJtd0WvQwCLANzl/t4K4CiAhh6P/RVAd/ftmgB+Pr9ez/8jFB78ugHAA7q3y5VsP49pJX4dFtn+/QFs9tguRwHs9Lj/Ufdz773SrqcqfQEQAD8CGO0xLRbAZBRe0dHgnlYPQIj7+2MAwt3fBwL4DMDiiz1P+eVbX9xD+nc/4u9XoqISUEqdQeGFD8Z57kmhK6eU+h1ADoAQ3VkqWFlfh4EAsgEUeEz7D4B73d/fB+DfxT1QKZWLwsJQJX8PeOl1aHX/m+0x7XP83xlU7gDwv4usvwDANlTR7VecUr4Og1H4Zui8HAAHROT8+TXvReFzsazrqex6AXAopeafn6CU2o3C51SiUsrlnnZSKXW26IOVUlkARgO4TURCKygzacRC6kFEjAD64MLzrDb2+KhgjqZoVYZS6igK95DU0p3FF4hIewC/u0tGtVDG1+GnIvIrCq8C97K7HJ33BQqLFADcDOCbi6w/BIV7qr670p9BtzK8Dt8QkV0ovET050Wed+sB9HT//wwGsLS4BYiIFUBnAKtKHbySKsHr0P/8UAcACwG8XOT+zwEMFpFoFL5JOnWF66lKWgP4pZjp/wFws3t7vSUiV19sAUqpDAB/oPD1SD6uRKd9qgb83b+E6wI4AGCtx31HlFLt9MSiauwJEXkYQDMUlqfqwBuvwweUUvEiEgFgq4isUkodd9+XCuCsiAx2Lz+nyGN7iMhuFP7xe0cplVSmn6ZqeloptUxEAgGsF5FuSqmt7vsKAHyPwjLqr5Q6VmQHbGP3/19DAMuVUr9WaPLyUdLXYe7556eIdAWwRERae9y/CoUl9TSKL/LV5vWulDopIlcB6O3+Wi8idyul1l/kIfy0rZrgHtJC53+ZNEDhk3+s5jxVlog0QuEfLl94h6/TP5VSrQDcCeBf7r1Ovq5Ur0MRWe3ey7Kw6H1KqWQAO1C4p87TUgBzUPzH9VuUUrEAWgEYLiJV9o1oSV6HIvKRe/utKHqf++PSTQC6F7nrcwDvoviPnM+/aWgMoIOI3HKF8SuTv70ORSTaY2/96KIPUEr9iMJrskd4TLOjcG/hUygcP3nZ9ZTHD1PB9gHoUNwdSql8pdRKpdTTAF4DcFtx84lIEArH4f5WXiGp8mAh9aCUygEwHsBTIsK9x6Xk3is1H4WD9XmCWy9QhVdCi8f/XZrX55X0daiU6q+UaqeUGlH0PhEJAHA1gCNF7voSwEwUXnnuYsv9A8B0AJOuIL52JX0dKqUedm+/G4tZhgmFZb7o9tsC4HVcZPyte7kpAOIAPHsF8Sslz9ehUuqEe7u18xwfeZ6INEfhcInUIne9BWCSUiqtJOvxXnptNgDwkwvPdtFWRK4TkTru2wYAbQEcL/pg9176uQC+Km6MKfkeFtIilFI7UXg0qc+ceqOcnR87tQ+FR/auAfCS5kxVSYCInPT4erKYeaYBeNL9y7taKMPr8FP3x8a/AFiklLpgDJtSKlMpNcO9x+pS5qNwvGRMKdevi7deh+fHkP4KYA+KHLikCr3pLp2X8hUKn9s9riCDDmV9HZ7f/rtQuBd+aJHxy1BK7VNKLS5BFp94vbvfDN0OoK/7tE/7UPhmpi2Ab6TwlGu/AnACeM/joRvd920D8CeARyo2OenCKzURERERkVZV+h0YEREREVV9LKREREREpBULKRERERFpxUJKRERERFqxkBIRERGRViykRERERKQVCykRERERacVCSkRERERa/X+Y1nIvecR8FAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost-Sensitive Random Patches\n",
    "\n",
    "Ensemble learning is a widely studied topic in the machine learning community. The main idea behind the ensemble methodology is to combine several individual base classifiers in   order to have a \n",
    "classifier that outperforms each of them [17]. Nowadays,   ensemble methods are  one of the most popular and well studied machine learning techniques [18], and it can be noted that since 2009 all the first-place and   second-place winners of the KDD-Cup (https://www.sigkdd.org/kddcup/)   used  ensemble methods. The core \n",
    "principle in ensemble learning, is to induce random perturbations into  the learning procedure in order to produce several different base classifiers from a single  training set, then combining the \n",
    "base classifiers in order to make the final prediction.  In order to induce the random permutations and therefore create the different base classifiers,   several methods have been proposed, in \n",
    "particular: bagging,   pasting, random forests and random patches [19]. Finally, after  the base   classifiers are trained, they are typically   combined using either   majority voting,  weighted  voting    or  stacking [18].\n",
    "\n",
    "With the objective of creating an ensemble of example-dependent cost-sensitive decision trees, we first create $T$ different random subsamples $\\mathcal{S}_j$ for $j=1,\\dots,T$, of the training  set \n",
    "$\\mathcal{S}$, and train a $CSDT$ algorithm on each one. In particular we create the different [19]. In the random patches algorithm, base classifiers are created by randomly  drawn bootstrap subsets of both examples and features. \n",
    "\n",
    "Lastly, after the base classifiers are trained, we combine them using cost-sensitive weighted voting. This method is based in based in calculating the final prediction as a weighted average of the base classifiers estimation:\n",
    "\n",
    "$$  f_{wv}(\\mathcal{S},\\mathcal{M}, \\alpha)\n",
    "  =argmax_{c \\in \\{0,1\\}} \\sum_{j=1}^T \\alpha_j \\mathbf{1}_c(M_j(\\mathcal{S})), $$\n",
    "  \n",
    "where  $\\alpha_j$ is related to the performance of each classifier $M_j$ in the  out of bag set   $\\mathcal{S}_j^{oob}=\\mathcal{S}-\\mathcal{S}_j$\n",
    "\n",
    "$$  \\alpha_j=\\frac{Savings(M_j(\\mathcal{S}_j^{oob}))}\n",
    "  {\\sum_{j_1=1}^T Savings(M_{j_1}(\\mathcal{S}_j^{oob}))}. $$\n",
    "\n",
    "This method guaranties that the base classifiers that contribute to a higher increase in savings \n",
    "have more importance in the ensemble. For more details see [20]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              f1       pre       rec       acc       sav\n",
      "RF      0.234356  0.509532  0.152174  0.931949  0.129533\n",
      "DT      0.248201  0.238436  0.258799  0.892699  0.184151\n",
      "LR      0.071775  0.569231  0.038302  0.932197  0.028994\n",
      "RF-BMR  0.279198  0.177352  0.655797  0.768253  0.434677\n",
      "DT-BMR  0.134468  0.079447  0.437371  0.614652  0.100415\n",
      "LR-BMR  0.207882  0.130423  0.511905  0.733005  0.265352\n",
      "CSDT    0.286539  0.174688  0.796584  0.728506  0.502330\n",
      "CSRP    0.348888  0.236397  0.665631  0.829962  0.492400\n"
     ]
    }
   ],
   "source": [
    "from costcla.models import CostSensitiveRandomPatchesClassifier\n",
    "\n",
    "classifiers[\"CSRP\"] = {\"f\": CostSensitiveRandomPatchesClassifier(combination='weighted_voting')}\n",
    "# Fit\n",
    "classifiers[\"CSRP\"][\"f\"].fit(X_train, y_train, cost_mat_train)\n",
    "# Predict\n",
    "classifiers[\"CSRP\"][\"c\"] = classifiers[\"CSRP\"][\"f\"].predict(X_test)\n",
    "# Evaluate\n",
    "results.loc[\"CSRP\"] = 0\n",
    "results.loc[\"CSRP\", measures.keys()] = \\\n",
    "[measures[measure](y_test, classifiers[\"CSRP\"][\"c\"]) for measure in measures.keys()]\n",
    "results[\"sav\"].loc[\"CSRP\"] = savings_score(y_test, classifiers[\"CSRP\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print (results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAEvCAYAAACT2dMvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hUZaIG8PebnkmvJNQQWgglNGnSRFEQVGwsCtYVRESjFMVFMICuFAUBYVlQXETvgmK5ICyIujRBaaGEXgwIJKQQ0ifTvvtHAjeEBBIyyTdJ3t/z5GHmzJlz3jnMZN6cKqSUICIiIiJSRaM6ABERERHVbiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESklE7VjIOCgmR4eLiq2RMRERGV2d69e1OllMGqc9RUygppeHg49uzZo2r2RERERGUmhDirOkNNxk32RERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSZSqkQoj+QojjQohTQoiJJTz+rBAiRQixv/DnBddHJSIiIqKa6JZXahJCaAEsBNAPwHkAu4UQa6SUR4qNukpKOaYSMhIRERFRDVaWNaSdAZySUp6RUloBrATwUOXGIiIiIqLaoizXsq8H4M8i988D6FLCeI8KIXoBOAHgdSnlnyWMQ0REVKXCJ65THeGWEmYMVB2BSClXHdS0FkC4lLItgE0Alpc0khBipBBijxBiT0pKiotmTURERETVWVkK6QUADYrcr1847BopZZqUMr/w7icAOpY0ISnlEillJyllp+Dg4NvJS0REREQ1TFkK6W4AzYQQjYUQBgBDAawpOoIQIqzI3QcBHHVdRCIiIiKqyW65D6mU0i6EGANgIwAtgGVSysNCiGkA9kgp1wB4VQjxIAA7gMsAnq3EzERERERUg5TloCZIKdcDWF9s2JQit98C8JZroxEREVFlqA4HegE82Ks24ZWaiIiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKZ3qAERUuvCJ61RHKJOEGQNVRyAiomqMa0iJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISKkyFVIhRH8hxHEhxCkhxMSbjPeoEEIKITq5LiIRERER1WS3LKRCCC2AhQAGAIgC8IQQIqqE8bwBxAD43dUhiYiIiKjmKssa0s4ATkkpz0gprQBWAniohPGmA5gJwOLCfERERERUw5WlkNYD8GeR++cLh10jhOgAoIGUcp0LsxERERFRLVDhg5qEEBoAcwCMK8O4I4UQe4QQe1JSUio6ayIiIiKqAcpSSC8AaFDkfv3CYVd5A2gNYLMQIgFAVwBrSjqwSUq5RErZSUrZKTg4+PZTExEREVGNUZZCuhtAMyFEYyGEAcBQAGuuPiilzJBSBkkpw6WU4QB+A/CglHJPpSQmIiIiohrlloVUSmkHMAbARgBHAXwlpTwshJgmhHiwsgMSERERUc2mK8tIUsr1ANYXGzallHH7VDwWEREREdUWvFITERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIREZFb0fnEwbPJDLRd3hb3rr4X686sUx2JKplOdQAiIiKiq3Q+cTCFfQuhsUECSMxJROyOWADAwIiBSrNR5eEaUiIiInIbxuCNEBrbdcMsDgvm7ZunKBFVBRZSIiIicgtCfxlCf6XEx5Jykqo4DVUlbrInIiIipTTGCzAEboXO51Cp44R6hlZhIqpqLKRERESkgITW8xQMAVug8zoF6TDCdrkHnFY/GOv857rN9iatCTEdYhRmpcrGQkpERERVyAGdzyEYArdAa0qE0+6N/OT+sKZ3AZweAADp9IAxeCO0hgyEeoYipkMMD2iq4VhIiYiIqPIJK/R+u2EI2A6NIR2O/GBYLj4KW2Z7QF5fR+yZ7WHPbI+EGSyhtQULKREREVUaoc2G3n8nDP47IXS5cOQ2Qu6lB+DIjgSPraarWEiJiIjI5YQ+DYaAbdD77YHQ2GHLioItrRcceeGqo5EbYiElIiIil9GYzsMQuAU673hAamDLbA9bWi84rSGqo5EbYyElIiKiCpLQep4oKKKeZyAdRljTesGWfiek3Ud1OKoGWEiJiIjoNjmg8zlYeMR8Epw2H1gu3Q/blc6A06Q6HFUjLKRERERUPiK/4Ij5wG3Q6DPgyA9B3sXHYc+IBqsF3Q6+a4iIiKhMhDYL+oAdMPj/BqHNgz2nMSxJg+HIbgEeMU8VwUJKRERENyX0qTAEboPedy8gHLBnRcGa1htOS0PV0aiGYCElIiKiEmlM5wquMe99GJBa2DI6wHq5J6Q1WHU0qmFYSImIiKgIJ7ReJwquMe/5B6TDBGtaH9gud4d0eKsORzUUCykREREBsEPnewCGgK3Qmi7BafOF5dLAwiPmjarDUQ3HQkpERFSbaSzQ++0quMa8PhMOSyjyLgyBPTMagFZ1OqolWEiJiIhqIaHLhN7/6hHzFthzImBJfBSOnOYAhOp4VMuwkBIREdUif2T8AWPoN9D77gOEE/as1rCm9YLT0kB1NKrFWEiJiIhqgf3J+7Esfhk2/7kZel8tbBl3wJrWA9IWpDoaEQspERFRTeWUTmz5cws+O/wZ4pLj4GPwwci2I/Hh6mBIh5fqeETXlKmQCiH6A5iHgr2bP5FSzij2+CgALwNwAMgGMFJKecTFWYmIiKgMrA4r1p1Zh88Of4Y/Mv5AXc+6mNh5Ih5u+jDMejM+WLVOdUSi69yykAohtAAWAugH4DyA3UKINcUK5/9IKRcXjv8ggDkA+ldCXiIiIipFljULX5/4Gl8c+QIpeSmIDIjEzJ4zcW/4vdBpuFGU3FdZ3p2dAZySUp4BACHESgAPAbhWSKWUmUXG9wQgXRmSiIiISncp5xK+PPolvjrxFXJsOega1hXv9ngX3cK6QQgeMU/uryyFtB6AP4vcPw+gS/GRhBAvAxgLwACgr0vSERERUalOXzmNfx3+F3448wOc0on7Gt2HZ1s/i6jAKNXRiMrFZevvpZQLASwUQjwJ4G0AzxQfRwgxEsBIAGjYsKGrZk1ERFRrSCmxL3kfPov/DFvOb4FJa8LjzR/H01FPo753fdXxiG5LWQrpBQBFT05Wv3BYaVYC+EdJD0gplwBYAgCdOnXiZn0iIqIyckon/nvuv1h2eBkOphyEn9EPo6NHY2jkUPib/FXHI6qQshTS3QCaCSEao6CIDgXwZNERhBDNpJQnC+8OBHASREREVGH5jnysPb0Wyw8vR0JmAup51cOkLpPwUNOH4KHzUB2vVtu7d2+ITqf7BEBrABrVedyYE0C83W5/oWPHjskljXDLQiqltAshxgDYiILTPi2TUh4WQkwDsEdKuQbAGCHEPQBsANJRwuZ6IiIiKruM/IxrR8ynWdIQFRiF2b1n456G9/CIeTeh0+k+CQ0NbRkcHJyu0Wi45bcUTqdTpKSkRCUlJX0C4MGSxinTO1pKuR7A+mLDphS5HVORoERERLWRzicOxuCNaLv8LYR6hiKmQww61umIFUdWYPWJ1ci15+LOunfiudbPoXNoZx4x735as4zemkajkcHBwRlJSUmtSxuHf2IREREpoPOJgynsWwiNDRJAYk4iJm2fBKd0QiM06N+4P55r9RxaBLRQHZVKp2EZLZvC5VTqbg0spERERAoYgzdCaGzXDXNIBzx1nvj2oW9R16uuomREVY874BIRESkg9FdKHJ5rz2UZpTLTarUdIyMjo67+HD9+3JCUlKTt0qVLc7PZ3P7pp5++7jybH330UWDz5s2jmjdvHtWsWbNWX3zxhZ+q7EVxDSkR3baS9n8bGDFQdSwiN+eA3n9nqY+GeoZWYRaqSquOrwpYfGBxvbS8NEOgR6B1VPSoC39p8ZfLFZmm0Wh0Hjt2rOjl3JGZmamZNm3axQMHDnjEx8dfOxXD6dOn9R9++GHY/v37jwYGBjoyMjI0iYmJFeqCNpsNer2+IpMAwDWkRHSbru7/pjFcgYREYk4iYnfEYt2ZdaqjEbktredJmCPmwRT6A5z5dSCd13cBk9aEmA48TrgmWnV8VcCs3bMapealGiQkUvNSDbN2z2q06viqAFfPy8fHx3nfffdlm0wmZ9HhiYmJek9PT6evr68DAHx9fZ2RkZFWAIiPjzd27969eYsWLaKioqJaHj582Oh0OvHiiy/Wb9asWavmzZtHLV261B8AfvjhB++OHTu26Nu3b9NmzZq1BoBFixYFtGnTpmVkZGTUk08+2chut5crM9eQEtFtKWn/N4vDgnn75nEtKVExQp8GY5110HsfgdMaiNw/n4YjuyV0PvthDN4IrSGDWxmqucm/Tm5wKv2UubTHj6Uf87Q77dedJsHqsGpm7JoR/v3J74NLek5T/6a50++c/mdJj12Vn5+viYyMjAKABg0a5G/atOl0aeN27do1NygoyNagQYM2d955Z9YjjzyS/uSTT2YAwJNPPtl4/PjxSU8//fSV3Nxc4XA4xOeff+536NAhj6NHjx5OTEzUde7cueW9996bDQBHjhwxx8XFHY6MjLTu27fPtHr16oA9e/YcMxqNcvjw4Q0XL14cOGbMmLSbZS+KhZSIbktp+78l5SRVcRIiNybyYQj6LwwB2wCpRX5yf1gv9wBkwdevPbM97JntkTCDJbSmK15GbzW8rEraZF8anU6HrVu3ntyyZYv5xx9/9Jk4cWKDPXv2eE6ePDnp0qVLhqeffvoKAJjNZglAbtu2zXvIkCGXdTodGjRoYO/SpUv29u3bzb6+vs62bdvmXF27umHDBu/4+HhzdHR0SwCwWCyakJCQcq0iZSElotsibX4QhhtLKfd/IwIAZ8Haz5D/QKPPgu1KB+Sn9Ie0+6gORpXkVmsy7/rqrjapeamG4sODPIKs/x707+OVl+x6Go0Gd911V+5dd92VO2DAgMwXXnghfPLkyeVek2A2m6/tDiClFI8//njawoULb3Zp+Zvnut0nElHtZs9pVuLwMM8wOKWzxMeIagON6U+YGy2GR72vIO1+yPljNCyJQ1hGa7lR0aMuGLSG6345GrQG56joUbdd4sorISFBv3379mu7FezZs8dcr149q7+/vzM0NNS6YsUKPwDIy8sTWVlZml69emWtXr06wG634+LFi7pdu3Z59ezZM6f4dPv375/5ww8/+F+4cEEHAJcuXdKeOHHihvJ9M1xDSkTlJnSZ0PscgsMSBKGxX9v/rXVga2w6twmzds/Cm3e8yavKUK0itFkwhmyA3m8vnHZv5F18HPaM9uC6HwKAq0fTu/oo+9LUq1evTXZ2ttZms4mNGzf6rV+//oS3t7dz/Pjx9S9duqQ3Go0yICDAtnTp0nMA8MUXX/wxYsSIRtOnT6+r1+vl119/ffqpp566smPHDq+WLVu2EkLIqVOnnm/YsKH94MGD182rY8eOlrfffvvC3Xff3dzpdEKv18v58+efa968ubWseYWUai4w0KlTJ7lnzx4l8yaqLsInuuMR6xKm+iug8zyBnDOvQdqCru3/JqXE7D2zseLICrwU/RJGtxutOCtRVXyO7NAH/Apj0C+Axg5rWg9Y0/oCTmOZp1DV+5C65++WG7nTvrVCiL1Syk5Fhx04cCAhOjo6VVWm6ubAgQNB0dHR4SU9xjWkRFQuOu+D0HsfgeXSAEhb0HWPCSEwodMEZFuz8Y8D/4C3wRtPRT2lKClRZZPQeh2DKWQdNMZU2LNawnJp4A2fCyK6NRZSIiozoc2BMXQNHHn1Ybvco+RxhMA73d5Bti0bs3bPgpfeCw83e7iKkxJVLo0hGcY6P0DndQKO/GDknnsejpzmqmMRVVsspERUZsY6ayG0FuSdewyAttTxtBotZvScgRxbDmJ3xsLL4IV+jfpVXdBaipthq4DGAmPQT9AH7ACcBliSBsGW3g03+zwQ0a1xT2siKhOt11HofffDmtoHzvxbn9rJoDVgbp+5aBvUFm9sfQM7LuyogpRElcUJvd8ueDaZDX3Ar7Bd6YSc0+NhS+8BllGiimMhJaJb01hgCv0ODksdWFPvKvPTzHozFt6zEE18m+C1za9hf/L+SgxJVDm0Hgkwhy+EKexbOK3ByE0Yg/ykRyAdXqqjEdUYLKREdEvGkPUQuixYEh9Deff08TH4YHG/xQgxh2D0z6Nx/HKVnf+ZqEKELgOmuv+GOXwxhC4beReeQN7ZF+G01FMdjajG4T6kRHRTWvMpGPx3wZrWC05Lg9uaRpBHEJb0W4Kn//M0Xtz0IpYPWI5GPo1cnJTIRYQNhoCtMARtBiCRn9IX1rQ+gCzXeb6pFgqfuK6jK6eXMGPg3rKM9+abb4Z+8803gRqNRmo0GixatOhs3759bziB/c385S9/afTGG29c6tixo+X20lYMCykRlU5YCzdTBiI/5Z4KTaquV10suXcJnv3Psxjx4wh8PuBzXmaU3IyEzjsexpD10BjSYctsg/zkAZC2ANXBiEr1008/eW7cuNHv0KFDRzw8PGRiYqIuPz+/3FclWbVq1dnKyFdW3GRPN1h3Zh3uXX0v2i5vi3tX34t1Z6rHkbvkesbgjdAYLsNy8VGXrB2K8I3A4n6LkWXNwshNI3HZUikXKCEqN40xCR4Nl8Kj/peQTiNyz46A5cIwllFyexcuXNAHBATYPTw8JACEhYXZw8PDbePHjw9r3bp1y2bNmrV64oknGjmdTsTFxZnatGnT8upzjx8/bmjevHkUAHTu3LnF1q1bzQBgNpvbv/LKK/VatGgRFR0dHfnnn3/qAODw4cPG6OjoyObNm0e9+uqrdc1mc3sAOHv2rL5Tp04tIiMjo5o1a9Zqw4YN5d7BmoWUrrPuzDrE7ohFYk4iJCQScxIRuyOWpbQW0nichT5gB6zpXeHIi3DZdKMCo/Dx3R/jYvZFjNo0ClnWLJdNm6jctDkw1vlfmBvPg9aYBEviYOT+8QocuU1UJyMqk8GDB2devHjREB4e3nr48OEN161b5wUAEyZMSI6Pjz968uTJw3l5eZqVK1f6tm/f3mKz2cSxY8cMAPD5558HDB48OL34NPPy8jTdunXLPn78+JFu3bplL1iwIBgAxowZ02D06NHJJ06cOFK/fn3b1fGXLVsWcPfdd2ccO3bsyNGjRw936dIlt7yvg4W0lrI5bLiYfRH7k/dj09lN+PLol5i7dy6m7pwKi+P63UcsDgvm7ZunKCkpIewwhX0DafdBfnJ/l0++Y52OmNNnDk6mn8SYn8cgz57n8nkQ3ZwDev+d8Ir4EHr/32FL74rs0+Nhu9IVPI0TVSe+vr7O+Pj4Ix9//PHZ4OBg+zPPPNNk/vz5gf/5z3+827ZtG9m8efOoHTt2eMfHx3sAwODBgy9//vnnAQDw3Xff+T/11FM3bKrS6/Vy6NChGQDQsWPHnLNnzxoAIC4uzuv555+/DAAvvPBC2tXxu3btmvPvf/87aOzYsXV37drl4e/v7yzv6+A+pDWMw+nAZctlJOclIzknGSl5KUjO/f9/k3OTkZKbgvT8G/4ggk6jg91pL3G6STlJlR2d3Igh6GdojcnIPfcc4DRVyjx61e+F93u+jze2voFxm8dh3l3zoNfqK2VeREVpzadhrLMWWlMS7DlNkH/pgTKdW5fIXel0OgwaNChr0KBBWW3bts1bunRp0PHjx82///77kaZNm9rGjh1b12KxaADgqaeeSn/88ccjhg4dmi6EQJs2bfJLmJ7UaDTXpm2322+6T+qAAQOyt27devybb77xff755xuPGTPm0pgxY9Ju9pwb5lmekUkdKSUy8jNwKfcSUvJSkJJbctFMtaTCKa//w0QjNAg0BSLYHIy6nnURHRyNEHMIQswhCPYILvjXHAw/ox/6f9MfiTmJN8yfB5/UHhrjRRgCt8B2pQMcOS0qdV79G/dHli0L03ZOw6Ttk/B+z/eh1XDtFFUOob8MY8g66H0Ow2n1R9754bBntQJQ7uM/iNzGgQMHjBqN5lqxjIuL82jatGn+8ePHzaGhofaMjAzN2rVr/R944IF0AGjVqlW+RqPBlClT6j788MPl2pG/Xbt22f/617/8R4wYkb5s2bJrO1ifOHHCEBERYR03blxqfn6+2LdvnxkAC2l1k2PLKSiaRUpmSm7KtWFXS6fNabvhuX5GPwSbgxHiEYLm/s2v3Q42B18rnQGmAOg0ZfuvjukQg9gdsddtttcKLV5t/6rLXi+5MwdMdVdDOsywXBpUJXN8vPnjyLJmYe7eufAyeGFy18kQggWBXEhYYQjaDEPAVgAC+cn3wnq5JyC5Rp5cq6ynaXKlzMxM7auvvtowMzNTq9VqZXh4eP7y5cvP+vn52Vu2bNkqODjYHh0dfd0poB555JHL06dPrz9z5swL5ZnXggUL/hw2bFjj2bNnh/Xt2zfTy8vLAQAbN270nj9/fqhOp5Nms9nx5Zdf/lHe18FCWoksdsv/r83MS75WOIuWzuTcZOTab9z311PviWCPYNQx10H7kPYlFs0gjyAYtUaXZh4YUXCN6Xn75iEpJwleei9k2bKQYyvX6cyomjIEboXWdBF554cDTnOVzff51s8jy5qFTw59Am+DN17v+HqVzZtqLikl1v+xHp5NPoBGnwlbRruC0zjZfVVHI3KZnj175sbFxR0rPnz+/PkX58+ff7Gk50ybNu3StGnTLhUdtmvXrmtXLcnNzY27evu5555Lf+6559IBIDw83LZ///5jGo0GS5Ys8T958qQRAF555ZW0V155pVxrRIur1YV03Zl114pXqGcoYjrEXCtkN2Nz2pCWl3ataF7dXF50E3pKXgoy8jNueK5BY7hWKFsEtECPej2ubTKvY66DYI9gBJuD4an3rIyXXCYDIwZeWw5O6cTLP7+MmbtnolVQK7QOaq0sF1UujSEZhqCfYctsA3tW1f8/v9r+VWRZs7Asfhm8Dd54oc0LVZ6Bao4jaUcwY9cMxCXHQdrrIefCMDjzeDEGoor49ddfzTExMQ2llPDx8XH861//SnDVtGttIb16eqOrm6avnt4oy5qFdiHtrtsvs/jazcuWy5CQ101PK7QI8ghCiDkEDb0bomOdjteKZ9E1mz4Gn2q1OVIjNHi/x/sY8sMQjNs8DqsGrYKfyU91LHI5J0xh3wBOA/KTHlSSQAiBv3X5G7KsWZi3bx58DD4Y0mKIkixUfaXlpWFB3AJ8e/Jb+Jv8Ma37NLz+qQ48qQxRxfXv3z/7+PHjRypj2rW2kM7bN6/E0xu99/t7N4wbYAq4dgBQVGDUtTWaRYumv9G/xh6M4Wfyw4e9P8TTG57G37b/DR/f/TE0gr/caxK9/05ozWeRd2EIpMNbWQ6N0ODdHu8ix5aDd397F156L9wfcb+yPFR92Bw2/M+x/8HiA4thsVvwdNTTeDH6xYJdQMDzKFOlcTqdTqHRaOStR63dnE6nAFDq6aBqbSG92WmM5vaZe61wBnkE8VQ0ANoEt8Ebd7yBv//+d3x66FOMaDtCdSRykYIjjzfAnt0C9sz2quNAr9Hjg94f4KWfXsKk7ZPgZfBCr/q9VMciN7b9wnbM3DUTCZkJ6FGvB9644w009m2sOhbVDvEpKSlRwcHBGSylpXM6nSIlJcUXQHxp49TaQhrqGVri6Y3CPMNwT6OKXbO7phraYijikuPw8f6P0Sa4DbqGdVUdiSpMwhT2LSA1sCQ+DHc5/Y1JZ8KCvgvw1x//irGbx+If9/wDd4TeoToWuZmzmWcxa/csbD2/FY18GmHh3Qv5xwtVKbvd/kJSUtInSUlJrcH9Qm7GCSDebreXenBArS2kJZ3eyKQ1IaZDjMJU7k0IgdhusTh++Tje3Pomvhr0Fep41lEdiypA77cbOs9TsCQOhrS7177BXgYvLL5nMZ7d8Cxe+eUVfHrfp2gV2Ep1LHID2dZsLDm4BCuOroBRa8S4juMwrOUwbs2iKtexY8dkAGp2vK9ham2bHxgxELHdYxHmGQYBgTDPMMR2jy3TUfa1mVlvxtw+c5Fnz8OErRNKPDcqVQ9ClwFjyDrYcxrDdqWz6jgl8jf5Y0m/JfAz+mHUplE4c+WM6kikkFM68f2p7zHou0H47PBnGBQxCD88/AOebf0syyhRNVdrCylQUEp/fOxHHHzmIH587EeW0TKK8IvAO93eQVxyHObt5TXuqycJU+j3gHDCkvgo3PlXQR3POljSbwm0QosRm0bgQna5zuNMNcSBlAMYtm4YJv86GfW96+PfA/+N6XdOR5BHkOpoROQC7vstRG5tYMRA/KXFX7D8yHL8dPYn1XGonHQ+B6DzPor8lH6QNvf/Qm/o0xBL7l0Ci92CET+OQGpequpIVEWSc5Pxt21/w/D1w5Gcm4y/9/g7VgxYwXMiE9UwLKR029644w20DmyNyb9OxtnMs6rjUBkJbTaMddbCkdcAtss9VMcps+b+zbHonkVIzUvFyE0jS7zwBNUc+Y58fHLoEwz6bhA2JGzAiDYjsPbhtXigyQPV6lzORFQ2LKR02wxaAz7s8yG0Gi3Gbh4Li91y6yeRcsY6ayG0FlguPobq9isgOjga8+6ah4SMBIz+eTRybTdedk4JTrEAACAASURBVJeqNyklfjn3CwZ/Pxjz9s1Dt7Bu+N/B/4tXO7wKs77qLmdLRFWren0bkdup61UX7/d4HyfST5R4UQFyL1qvI9D7HoA19S44rdXzDAnd6nbD7F6zEZ8aj9f++xqsDqvqSOQip6+cxoubXkTMf2Ng0pmwpN8SzOs7Dw28G6iORkSVjIWUKqxn/Z4Y2XYkvj/1Pb47+Z3qOFQaTR5Mod/BYQmFNbWP6jQVcnejuzGt+zTsTNyJN7e+CbvTrjoSVUBGfgZm7JqBR9c8ivi0eEzsPBFfP/A1utXtpjoaEVWRWnseUnKt0dGjcSDlAN77/T20DGyJyIBI1ZGoGGOddRC6bOSdfwY14aP/UNOHkGXNwszdMxG7IxbT7pzGS9pWMw6nA9+c/AYL4hYg05qJx5o9hjHtx8Df5K86GhFVser/rURuQavRYmbPmRiydgjGbh6LlYNWwsfgozoWFdKaT8Lgtwf5qb3htNRXHcdlhkcNR5Y1C4sOLIK3wRtv3PEGD3hxYzqfOBiDN6Lt8rcQYAqAXqNHUm4SOtXphImdJ6JFQAvVEYlIEa5OIJcJ9AjEB30+QGJ2IiZvnwwpeVlftyDyYQr7Fs78IFhTa95lcUdFj8LwlsPxxdEvsPjAYtVxqBQ6nziYwr6FxnAFEhJpljQk5SbhicgnsOy+ZSyjRLUcCym5VPuQ9ni94+v45c9fsPzwctVxCIAxZCOE/gosiY8BsuZdzUYIgQl3TMBDTR7CogOL8MWRL1RHohIYQzZAaG68stvmPzdzrTYRla2QCiH6CyGOCyFOCSEmlvD4WCHEESHEQSHEz0KIRq6PStXFU1FP4Z6G9+CjfR9h76W9quPUalqPBOj9d8KW3hWOvHDVcSqNRmgQ2z0W9zS8BzN3z8T3p75XHYmuEnboA7ZB6Eo+b2xSTlIVByIid3TLQiqE0AJYCGAAgCgATwghooqNFgegk5SyLYDVAGa5OihVH0IITLtzGup51cOELRN4VR1VhA3GsG8gbb7IT+6vOk2l02l0mNlrJrqFdcM7O97Bz2d/Vh2plnNC5xMHz4gPYKqzDpAlH7IQ6hlaxbmIyB2VZQ1pZwCnpJRnpJRWACsBPFR0BCnlf6WUV89Q/RuAmnPUBN0Wb4M35vSZg0xrJk/Lo4gh6GdojSmwJD0CSKPqOFXCoDXgo7s+Quug1piwdQJ2XtypOlItJKH1PAFz4wXwqLcK0uGJ3LMvwJL4KKTz+l1GTFoTYjrEKMpJRO6kLIW0HoA/i9w/XzisNH8F8J+KhKKaoUVAC7zd9W3sStqFRfsXqY5Tq2hMF2AI3ArblY5w5DRXHadKmfVmLLp7EcJ9wxHz3xgcSDmgOlKtoTGdh0fDT2FuuAxCk4+8C08gN+FlOHKbwp7ZHpbER+C0+kFAIMwzDLHdYzEwYqDq2ETkBlx62ichxHAAnQD0LuXxkQBGAkDDhg1dOWtyU4ObDsb+5P1YemgpooOj0btBiW8NcikHTGGrIR2esFyqnV/2vkZfLOm3BM/85xm89NNL+Oy+z3gUdyUS+jQYg3+E3vcAnHZPWJIegC29C4p/xdgz28Oe2R4JM2rn+5KISleWNaQXABS9blv9wmHXEULcA2ASgAellPklTUhKuURK2UlK2Sk4OPh28lI1NLHzREQGROKt7W/hfNZ51XFqPEPgFmhNichPHAw4a++1v4M8grDk3iXw0HngxU0v4lzmOdWRahyhzYaxzhp4NpkDnfcR5KfehZzTE2BLvxM8zTURlUdZCuluAM2EEI2FEAYAQwGsKTqCEKI9gH+ioIwmuz4mVWcmnQlzes8BJDBuyzjkO0r8e4VcQGO4BEPQz7BltoU9u5XqOMrV86qHpf2WwiEdGPHjCB7R7SrCCkPgL/BsMht6/99gu9IROacnwJpyH+A0qU5HRNXQLQuplNIOYAyAjQCOAvhKSnlYCDFNCPFg4WizAXgB+FoIsV8IsaaUyVEt1cCnAd7t8S6OpB3BrF08CUPlcMIU9g2k04j8pAdvPXotEeEXgcX9FiPDmoEXN72IdEu66kjVmAN6v13wbDIbxpAfYc9tgtwzryE/6RFIO6/MRkS3r0zbVKSU6wGsLzZsSpHbNe/yL+RyfRv2xXOtnsNnhz9Du5B2eKDJA6oj1Sj6gF+hNZ9D3oW/QDq8VMdxK60CW2FB3wV46aeXMOqnUfj03k/hZeAyKjsJndcRGEI2QGtMgSO3EXIvDKvR57YloqrFKzVRlXq1w6voWKcjpv82HafST6mOU2NcPajEnhUJe2Y71XHc0h2hd2BOnzk4cfkEXvnlFVjsFtWRqgWNx1l4NFoMjwYrAAB5fz6F3LOjWEaJyKVYSKlK6TQ6zO41G2adGa9vfh05thzVkWoACVPYN4DUwJL0MABehrE0ver3wns93sPeS3sxbss42Jw3XsqSCmgMyTDV/xye4f+AxnAZlsSHkXvmtcJ9k/keIyLXYiGlKhdsDsbs3rNxLusc3tnxDqSUqiNVa3q/XdB5nkF+8v2Qdl/Vcdze/RH34+2ub2Pr+a2YtH0SHE6H6khuRegyYQz9FuaIj6Azn0Z+cj/knJoA25UuALSq4xFRDcXzcigQPnGd6ghlUpnnCrwj9A680v4VzNs3D+1D2mNYy2GVNq+aTOgyYAxZD3tOBGxXOquOU20MaTEEmdZMzNs3D956b7zd9W0IUcvX+mksMARugSFgOyCcsKV3hTW1L/dHJqIqwUJKyjzf+nnsT96PD/Z8gNZBrREdHK06UjUjYQr9DhBOWBIfBTejls8LbV5AljULy+KXwcfoU4svYWmH3v93GIJ+gUaXA1tGNPJT7oW0BaoORkS1CDfZkzIaocF7Pd5DHXMdjN8ynqfjKSedz37ovI+xPFTAax1ew+PNH8cnhz7BsvhlquNUMSd0Pgfg2WQOTKFr4cwPRc4fY2C5+ATfT0RU5VhISSlfoy8+7PMh0vLSMHHbRO7PV0YFV8hZC0duQ9gu36k6TrUlhMCkLpMwIHwA5u6di69PfK06UpXQmk/BHL4QHvX+Dek0IPfcc8g79wKclvqqoxFRLcVCSsq1CmyFt7q8hR0Xd2DJwSWq41QLxtA1EJr8wk31/BhXhFajxXs930PPej0xfed0bPhjg+pIlUZjvAiPBstgbvQJhC4HeReGIPePV+HIaQHu8kFEKnEfUnILjzV7DHGX4vCPA/9AdHA0utfrrjqS29J5HYbe5yDyk++F01pHdZwaQa/R48M+H2LUplF4a9tbMOvN6FW/l+pYLiN06TAG/wid737AaYLl0v2wpXcDpF51NCIiAFy1Qm5CCIG3u76NJn5N8Oa2N3nN8dJocmEM+x4OSxisab1Vp6lRPHQe+Pjuj9HMvxnGbh6LPUl7VEeqOE0ujCHr4NnkA+h8DsGa1gvZpybAdrkXyygRuRUWUnIbZr0Zc/rMgdVhLThpuYMnLS/OVGcdhDYHlouPgeeEdD1vgzcW91uMul518covr+BI2hHVkW6PsMEQuBleTWdBH7Ad9sx2yDk9HtaUAYDTrDodEdENWEjJrTT2bYxpd07DwZSDmLN3juo4bkXreQJ6v72wpvWCM7+e6jg1VoApAEv6LYGPwQejNo3CmYwzqiOVgxM63z3wbPIBjCEb4MgNR+4fMbAkPg5p91MdjoioVCyk5HbuC78Pw1sOxxdHv8CGhJp7gEm5iHyYQr+FIz8Y1tS7Vaep8UI9Q7Hk3iXQCA1G/jgSF7Mvqo50U1JKbD2/FebG8+BRdzWk3Qe5Z0ci7/yzcOaHqo5HRHRLLKTklsZ2HIu2wW3xzq/v4I+MP1THUc4YsgFCnwFL4mPc96+KNPJphH/2+ydy7bkY8eMIpOalqo5UooMpB/H8xufx8s8vQ2jsyDv/JHITRsORG6E6GhFRmbGQklvSa/X4sPeHMGqNGLt5LHJtuaojKaP1SIAhYCds6d3gzGukOk6t0iKgBRbdvQgpeSkYtWkUMvIzVEe65mzmWYzdPBbD1g/DmYwzmNRlEnJOj4U9qy14Ciciqm5YSMlthXqGYkbPGTh95TTe/e1dSClVR6p6wgZT2Go4rf7IT75PdZpaqV1IO3x010c4nXEaL//8svI/jlLzUvHub+9i8PeDsf3CdrwU/RLWP7IeQyOHgge6EVF1xUJKbq17ve54KfolrD2zFqtPrlYdp8oZgn6CxpgKS+IjgDSqjlNrda/bHbN6zcKh1EN4ffPrsDqsVZ4h15aLf+z/BwZ+OxCrT6zGo80fxfpH1mN0u9Hw1HtWeR4iIldiISW392L0i7iz7p14//f3cTjtsOo4VeZw2mEYArfBeqUTHLnNVMep9fo16ofYbrHYcXEHJm6bCLvTXiXztTltWHlsJQZ8OwCLDizCnfXuxPcPfY+3u76NII+gKslARFTZWEjJ7WmEBu/3fB8BpgCM2zzOrfbjqyw2hw1Tfp0CafdC/qWBquNQoYebPYw37ngDm85uwtSdU+GUzkqbl5QSPyb8iIf/92G89/t7aOzbGF/e/yXm9JmDcN/wSpsvEZEKLKRULfib/PFhnw9xKfcSJm2fVKlFwB18Gv8pTqSfgCVpMOD0UB2Hingq6imMih6F7099jw/2fFAp+zbvSdqD4euHY9yWcdBr9Pi478f47L7P0Da4rcvnRUTkDlhIqdqIDo7G+E7jseX8FiyLX6Y6TqU5mX4S/zz4TwwIHwBHdpTqOFSC0dGj8WTkk1hxZAX+efCfLpvuyfSTGPPzGDy38Tkk5SZhWvdpWP3AavRu0BtC8Mh5Iqq5dKoDEJXHk5FPIi45DgviFiA6OBp3hN6hOpJLOZwOvLPjHXjrvTGxy0R89Z+dqiNRCYQQeLPzm8i2ZWPh/oXwNnhjWMthtz29pJwkLNy/EGtOr4GnzhOvdXgNw1oOg0lncmFqIiL3xUJK1YoQAlO7T8Xxy8cxYcsEfP3A1wg2B6uO5TJfHP0Ch1IPYWbPmQgwBaiOQzehERpM7T4V2dZszNg1A94GbzzY5MFyTSPTmolPD32KL49+Cad0YnjL4RjRZgT8TLzMJxHVLtxkT9WOp94Tc/vMRa49F+O3jK+yo50r27nMc1gQtwB96vfBgMYDVMehMtBpdJjVexa6hHXBlF+n4OdzP5fpeVaHFcsPL8f9396Pz+I/Q79G/bD24bWYcMcEllEiqpVYSKlaaurfFJO7Tsa+5H2YHzdfdZwKc0on3tnxDvQaPd7u+jb3F6xGjFoj5t81H60CW2HClgn4LfG3Usd1SifWnl6LB757AB/s+QCtA1vjqwe+wvs930c9r3pVmJqIyL2wkFK19UCTBzCk+RB8Fv8Zfjn3i+o4FbL6xGrsubQH4zuNRx3POqrjUDmZ9WYsumcRGvk0wqu/vIqDKQeve1xKiV8v/Ioha4fgb9v/Bl+jL5beuxSL+y1GZECkotRERO6D+5BStfZG5zcQnxaPt7e/jVWDVqGBTwPVkcotKScJc/bOQZewLnik2SOq49Bt8jX6Ykm/JXhmwzP468a/wsvghbS8NAR6BMLX4IvTGadRz6seZvacif6N+0MjuD6AiOgq/kakas2oNWJOnzkQQmDslrGw2C2qI5WLlPLaCdZju8VyU301F2wOxtAWQ2FxWJCalwoJidS8VJzOOI1BjQdhzeA1uD/ifpZRIqJi+FuRqr16XvXwfs/3cezyMczYNUN1nHL54cwP2H5hO2I6xKC+d33VccgFvjj6RYnD9ybvhUFrqOI0RETVAwsp1Qi96vfCiDYj8M3Jb/D9qe9VxymT1LxUzNw9E+2C2+GJyCdUxyEXScpJKtdwIiJiIaUaZHS70egc2hnv/vYujl8+rjrOLf39978jz5aHqXdO5SbcGiTUM7Rcw4mISOFBTYcuZCB84jpVs79OwoyBqiOQC+g0OszsNRND1g7B2M1jsXLQSngbvFXHKtFPZ3/CprObENMhBhG+EarjkAvFdIhB7I5YWBz/vz+zSWtCTIcYhamIiNwbV8tQjRLkEYTZvWfjQvYFTPl1CqSUqiPdICM/A+/+9i5aBrTEM62eUR2HXGxgxEDEdo9FmGcYBATCPMMQ2z0WAyP4hy8RUWl42ieqcTrW6YjXOryGD/d+iBVHVuDpVk+rjnSdWbtnISM/A4v7LYZeo1cdhyrBwIiBLKBEROXANaRUIz3T6hn0bdAXc/fORVxynOo412y/sB1rTq/Bc62f4wnRiYiICrGQUo0khMD0HtMR5hWG8ZvHIy0vTXUk5NhyMG3nNET4RmBU9CjVcYiIiNwGCynVWD4GH8zpMwcZ1gy8ue1NOJwOpXnm7p2LpJwkTO0+leejJCIiKoKFlGq0yIBITOoyCb8n/o5FBxYpy7EnaQ9WHV+FYS2HoV1IO2U5iIiI3BELKdV4Dzd7GIObDsaSg0uw7fy2Kp+/xW5B7M5Y1Peqj1fav1Ll8yciInJ3LKRUK0zqMgnN/Zvjre1v4WL2xSqd96L9i3A28yxiu8fCrDdX6byJiIiqAxZSqhVMOhPm9JkDh9OBcZvHweqwVsl841PjsfzIcjza7FF0CetSJfMkIiKqblhIqdZo5NMI0++cjvi0eMzePbvS52dz2DD518kI8gjCuE7jKn1+RERE1VWZCqkQor8Q4rgQ4pQQYmIJj/cSQuwTQtiFEI+5PiaRa9zT6B48E/UMVh5fiXVnKvfStUsPLcWpK6cwpesUt72EKRERkTu4ZSEVQmgBLAQwAEAUgCeEEFHFRjsH4FkA/+PqgESuFtMxBh1COmDqzqk4feV0pczjRPoJLD24FPc3vh+9G/SulHkQERHVFGVZQ9oZwCkp5RkppRXASgAPFR1BSpkgpTwIwFkJGYlcSq/RY3bv2fDQeWDs5rHIteW6dPp2px1Tfp0CH6MPJna+YYMCERERFVOWQloPwJ9F7p8vHEZUbYWYQzCz10wkZCYgdkcspJQum/aKIytwOO0w3uryFvxN/i6bLhERUU2lq8qZCSFGAhgJAFqf4KqcNdENuoZ1xcvtXsaCuAXoUKcDhkYOrfA0EzISsHD/QvRt0Bf3NbrPBSmpJOETK3f/X1dJmDFQdQQiomqhLGtILwBoUOR+/cJh5SalXCKl7CSl7KQ1+97OJIhc6oU2L6BnvZ6YuXsmDqUcqtC0nNKJd3a8A4PWgLe7vg0hhItSEhER1WxlKaS7ATQTQjQWQhgADAWwpnJjEVUNjdDg/Z7vI8QjBOO2jMMVy5XbntZXx7/CvuR9mNBpAoLN3AJARERUVrcspFJKO4AxADYCOArgKynlYSHENCHEgwAghLhDCHEewOMA/imEOFyZoYlcydfoizl95iA1LxVvbX8LTln+Y/MuZl/E3L1z0b1udwxuOrgSUhIREdVcZToPqZRyvZSyuZSyiZTyvcJhU6SUawpv75ZS1pdSekopA6WUrSozNJGrtQpqhTfveBPbL2zH0oNLy/VcKSWm7ZwGCYkp3aZwUz0REVE58UpNRIWGtBiC+xvfj4X7F2LnxZ1lft6a02vw68Vf8VqH11DPiyegICIiKi8WUqJCQgi80+0dRPhGYOK2ibiUc+mWz0nNS8Ws3bPQIcQ1R+kTERHVRiykREWY9WbM6TMHefY8jN8yHjan7abjv/fbe7DYLYjtHguN4MeJiIjodvAblKiYCL8ITO0+FftT9mPu3rmljvdjwo/46dxPGN1uNBr7Nq7ChERERDULCylRCQY0HoAnIp/AiiMrsOnsphsev2K5gvd+fw9RgVF4ptUzChISERHVHCykRKWY0GkC2ga1xeRfJ+Ns5tnrHpu1exYy8zMxrfs06DRVesEzIiKiGoeFlKgUeq0eH/T+ADqNDq9vfh159jwAwNbzW7H2zFr8tc1f0SKgheKURERE1R9X7RDdRJhXGGb0nIGXfnoJvVf1hsVugRACIR4hGNl2pOp4RERENQLXkBLdQkZ+BnRChzx7HiQknNKJK/lXSty3lIiIiMqPhZToFubtmwe7tF83zOq0Yt6+eYoSERER1SwspES3kJSTVK7hREREVD4spES3EOoZWq7hREREVD4spES3ENMhBiat6bphJq0JMR1iFCUiIiKqWXiUPdEtDIwYCKBgX9KknCSEeoYipkPMteFERERUMSykRGUwMGIgCygREVEl4SZ7IiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSSqc6ANFV4RPXqY5wSwkzBqqOQEREVONwDSkRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqVqZAKIfoLIY4LIU4JISaW8LhRCLGq8PHfhRDhrg5KRERERDXTLQupEEILYCGAAQCiADwhhIgqNtpfAaRLKZsCmAtgpquDEhEREVHNVJY1pJ0BnJJSnpFSWgGsBPBQsXEeArC88PZqAHcLIYTrYhIRERFRTVWWQloPwJ9F7p8vHFbiOFJKO4AMAIGuCEhERERENZuQUt58BCEeA9BfSvlC4f2nAHSRUo4pMk584TjnC++fLhwntdi0RgIYWXi3BYDjrnohFRQEIPWWY9U+XC434jIpGZdLybhcSsblciMuk5K503JpJKUMVh2iptKVYZwLABoUuV+/cFhJ45wXQugA+AJIKz4hKeUSAEtuL2rlEULskVJ2Up3D3XC53IjLpGRcLiXjcikZl8uNuExKxuVSe5Rlk/1uAM2EEI2FEAYAQwGsKTbOGgDPFN5+DMAv8larXomIiIiIUIY1pFJKuxBiDICNALQAlkkpDwshpgHYI6VcA+BTACuEEKcAXEZBaSUiIiIiuqWybLKHlHI9gPXFhk0pctsC4HHXRqtSbrcbgZvgcrkRl0nJuFxKxuVSMi6XG3GZlIzLpZa45UFNRERERESViZcOJSIiIiKlal0hFUI4hBD7hRDxQoi1Qgi/wuHhQoi8wseu/hhU560KRZbJYSHEASHEOCGERghxX5FlkV14+dj9QojPVWeuCkKI7BKGxQohLhQuhyNCiCdUZHOVin4ehBCbi7wvjhae2u3qYwlCiG3Fxt9feJo4CCH6CCEyCocdE0J8UNmvtyxc8XkQQvxLCPFHkdf2TpHHNgshzhW9eIgQ4vur77diy/6IEOJzIYS+al79zVX0M1Fk2R4QQuwTQnQvHB4uhJBCiHeLjBskhLAJIT4u73zchRAiVAixUghxWgixVwixXgjRXAgxv/Azd0gIsVsI0bhw/ITCYYcKX+O7QgiTEKJNkffe5SLvrZ9Uv8bbUYHlclAIsUUI0ajItIr+DvtaCGFW98qoQqSUteoHQHaR28sBTCq8HQ4gXnU+N1gmIQB+AjC12DibAXRSnVXVcikyLBbA+MLbzQBkAtCrzuqi//tyfx6Kvi8ABABIB2AovJ8AYD+ABoX3Wxbejy+83wfAD4W3PQAcA3Cnmy2T2/o8APgXgMcKb5sAnAHQuMhzDwLoUXjfD8DvV+dbdNmj4EDSXwAMU71cii+bIsPK/JkotmzvA7ClyGs+AyCuyOMvFb5fPi7vfNzhB4AAsBPAqCLDogFMRsEVDTWFw+oD8C+8nQAgqPC2F4D/AbC8tPdWdfxxwXKZCmBpKe+pLwGMVf0a+XN7P7VuDWkxO3HjVadqNSllMgouXjCm6BocupGU8iSAXAD+qrO4SEU/D14AcgA4igz7CsBfCm8/AeDfJT1RSpmHgvLhVp9HF30eTIX/5hQZthL/fzaSRwB8W8r8HQB2wc2WS2nK+ZnwQcEfMFflAjgqhLh6zsm/oOD9U9H5qHIXAJuUcvHVAVLKAyh4HyRKKZ2Fw85LKdOLP1lKmQ1gFIDBQoiAKspcFSq0XHDz31PbADR1cV6qIrW2kAohtADuxvXnVG1SZLPIQkXRlJNSnkHBmpkQ1VncmRCiA4CThaWlWqvg5+FLIcRBFFx5bXphibrqGxQULgB4AMDaUubvj4K1Xltv9zVUlgp8HmYLIfaj4HLLK4u9T34G0KtwuQ8FsKqkCQghTAC6ANhQ7uAKlOEz4XF1NwYAnwCYXuzxlQCGCiEaoOAPm4u3OR930BrA3hKGfwXggcLl8KEQon1pE5BSZgL4AwWfjZqiosulP4Dviw8UBRflGQDgkMuSUpWqjYXUo/BLIglAHQCbijx2WkrZrvDnZTXxqBp4XQhxGAWbWd9THaaCXPF5GCalbAugIYDxRffvQsEV29KFEEMBHEXBWq2iegohDqDgam8bpZRJFX1BbmSClLIdgFAAd1/dX7KQA8B2FJRRDyllQrHnNin8f7mEgrVGB6sicAWU9TORV/h+ikRBsfi82JrnDQD6ofSSXu0/e7LgEtstALwFwAngZyHE3Td5Sq3YUlWG5fJfIcQFFJTOoltarv4O2wPgHArOi07VUG0spHmFXxKNUPBBZ/EsRggRgYIvTHde+6DSXCllKwCPAvi0cC1WdVWuz4MQYmPhGoxPij8mpUwBsA8Fa/SKWgVgIUreXL9NShkNoBWAvwoh2t3Ga6hUZfk8CCE+K1wu64s/VrjpdTOAHsUeWglgPkreLH268P+lCfB/7d2/ixNBGMbx7yuCIFpeIwiWVgG5WkQQBAvRf0ARBG0UuUPO2sJfaCeSUgtrDywECy0sBFE8DtIIIqJgozaCohw8FjPBzZqca7K5TS7PB1LkZn/MDZnlndl3d5iPiKNDVn+j/NUnImJ3YYb9bHkHSc9J65TPFf72izR7tkjKJ/znecbxz9SkA8z3K5D0U9IjSReBK8CxfttFxE5Sfu2bcVWyAcO2y0HSdWqFlEfa9aMwtAlp7gAAAbpJREFUcD6Xf0M2hWYxIAVA0nfgPLCYp/oNiIg5oE16kMAvqV2H0iplL/mzbO7UqtofJB3OF/7T5bL8dOs+4G2p6AFwg7Ta26DjvgOuAUtDVH9sqvYHSadyuxzpc4ytpCC93C7PgKsMyKvNx/0MXCLNGk28Yp+Q9KEQKLTL20bEXlIqxJdS0S1gSdLXKuepr/a1ewJsi943T7Qi4kBE7MrftwAt4H1554jYAdwBlgfkUk6rodtF0hpwATixyfJqjRkOSAEkvSY97Trxrw8Zs25eV4f0RPFjekegs2p7RHwsfBb6bHMZWMgX0Kk2Qn+4n2+ZvQLuSurJD5P0TdL1CjMXbVJe5Z7/PH/d6uoP3RzSVVJeW8+DS0pu5qBzPcuk3+L+IepQt1H7RLdtV0gz5ydLOcdI6ki6V6EuE9338gDmOHAov96oQxqAtICHkV5/tgqsAbcLuz7NZS9It6DPbGzNx2uEdunu/4k0iPPdzU3GKzWZmZmZWaMmcmRpZmZmZrPDAamZmZmZNcoBqZmZmZk1ygGpmZmZmTXKAamZmZmZNcoBqZmZmZk1ygGpmZmZmTXKAamZmZmZNeo3WCZmYN9WGL0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions\n",
    "\n",
    "CostCla is a easy to use Python library for example-dependent cost-sensitive \n",
    "classification problems. It includes many example-dependent cost-sensitive algorithms. Since \n",
    "it is part of the scientific Python ecosystem, it can be easily integrated with other \n",
    "machine learning libraries. Future work includes adding more cost-sensitive databases and \n",
    "algorithms, and support for Python $\\ge$ 3.4. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix\n",
    "\n",
    "## Tuning the parameters of random forest\n",
    "\n",
    "So far the parameters of the different hyperparameter have not being tunned, which may lead to a bias comparison. Here I quickly tuned the hyperparameter of the random forest classifier using randomized search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 15 candidates, totalling 45 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  45 out of  45 | elapsed:  3.3min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomizedSearchCV took 208.75 seconds parameter settings.\n"
     ]
    }
   ],
   "source": [
    "from time import time\n",
    "from scipy.stats import randint as sp_randint\n",
    "from operator import itemgetter\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "# Utility function to report best scores\n",
    "def report(top_grid_score):\n",
    "        print(\"Best Model :\")\n",
    "        print(\"Mean validation score: {0:.3f} (std: {1:.3f})\".format(np.mean(top_grid_score['mean_test_score']),\n",
    "                                                               np.std(top_grid_score['std_test_score'])))\n",
    "        print(\"Parameters: {0}\".format(top_grid_score['params']))\n",
    "        print(\"\")\n",
    "\n",
    "clf = RandomForestClassifier()\n",
    "\n",
    "# specify parameters and distributions to sample from\n",
    "# from http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html\n",
    "param_dist = {\"n_estimators\": [10, 20, 50, 100, 1000],\n",
    "              \"max_depth\": [3, None],\n",
    "              \"max_features\": sp_randint(1, 10),\n",
    "              \"min_samples_split\": sp_randint(2, 100),\n",
    "              \"min_samples_leaf\": sp_randint(1, 100),\n",
    "              \"bootstrap\": [True, False],\n",
    "              \"criterion\": [\"gini\", \"entropy\"]}\n",
    "\n",
    "# run randomized search\n",
    "classifiers[\"RS-RF\"] = {\"f\": RandomizedSearchCV(clf, param_distributions=param_dist,\n",
    "                                                n_iter=15, n_jobs=4, verbose=1)}\n",
    "# Fit\n",
    "start = time()\n",
    "classifiers[\"RS-RF\"][\"f\"].fit(X_train, y_train)\n",
    "print(\"RandomizedSearchCV took %.2f seconds parameter settings.\" % ((time() - start),))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Model :\n",
      "Mean validation score: 0.935 (std: 0.000)\n",
      "Parameters: [{'bootstrap': False, 'criterion': 'entropy', 'max_depth': None, 'max_features': 9, 'min_samples_leaf': 84, 'min_samples_split': 50, 'n_estimators': 100}, {'bootstrap': False, 'criterion': 'entropy', 'max_depth': 3, 'max_features': 5, 'min_samples_leaf': 26, 'min_samples_split': 32, 'n_estimators': 100}, {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 3, 'max_features': 4, 'min_samples_leaf': 24, 'min_samples_split': 68, 'n_estimators': 20}, {'bootstrap': True, 'criterion': 'gini', 'max_depth': None, 'max_features': 1, 'min_samples_leaf': 5, 'min_samples_split': 86, 'n_estimators': 20}, {'bootstrap': True, 'criterion': 'gini', 'max_depth': None, 'max_features': 9, 'min_samples_leaf': 87, 'min_samples_split': 5, 'n_estimators': 100}, {'bootstrap': False, 'criterion': 'entropy', 'max_depth': None, 'max_features': 3, 'min_samples_leaf': 62, 'min_samples_split': 35, 'n_estimators': 100}, {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 3, 'max_features': 5, 'min_samples_leaf': 28, 'min_samples_split': 55, 'n_estimators': 1000}, {'bootstrap': True, 'criterion': 'gini', 'max_depth': 3, 'max_features': 9, 'min_samples_leaf': 80, 'min_samples_split': 79, 'n_estimators': 50}, {'bootstrap': False, 'criterion': 'gini', 'max_depth': None, 'max_features': 4, 'min_samples_leaf': 89, 'min_samples_split': 49, 'n_estimators': 1000}, {'bootstrap': True, 'criterion': 'gini', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 24, 'min_samples_split': 34, 'n_estimators': 10}, {'bootstrap': False, 'criterion': 'gini', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 68, 'min_samples_split': 68, 'n_estimators': 10}, {'bootstrap': False, 'criterion': 'gini', 'max_depth': None, 'max_features': 1, 'min_samples_leaf': 81, 'min_samples_split': 25, 'n_estimators': 50}, {'bootstrap': True, 'criterion': 'gini', 'max_depth': None, 'max_features': 7, 'min_samples_leaf': 26, 'min_samples_split': 97, 'n_estimators': 50}, {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 51, 'min_samples_split': 27, 'n_estimators': 10}, {'bootstrap': True, 'criterion': 'gini', 'max_depth': 3, 'max_features': 5, 'min_samples_leaf': 94, 'min_samples_split': 43, 'n_estimators': 50}]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "report(classifiers[\"RS-RF\"][\"f\"].cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              f1       pre       rec       acc       sav\n",
      "RF      0.234356  0.509532  0.152174  0.931949  0.129533\n",
      "DT      0.248201  0.238436  0.258799  0.892699  0.184151\n",
      "LR      0.071775  0.569231  0.038302  0.932197  0.028994\n",
      "RF-BMR  0.279198  0.177352  0.655797  0.768253  0.434677\n",
      "DT-BMR  0.134468  0.079447  0.437371  0.614652  0.100415\n",
      "LR-BMR  0.207882  0.130423  0.511905  0.733005  0.265352\n",
      "CSDT    0.286539  0.174688  0.796584  0.728506  0.502330\n",
      "CSRP    0.348888  0.236397  0.665631  0.829962  0.492400\n",
      "RS-RF   0.246531  0.583012  0.156315  0.934606  0.130941\n"
     ]
    }
   ],
   "source": [
    "# Predict\n",
    "classifiers[\"RS-RF\"][\"c\"] = classifiers[\"RS-RF\"][\"f\"].predict(X_test)\n",
    "# Evaluate\n",
    "results.loc[\"RS-RF\"] = 0\n",
    "results.loc[\"RS-RF\", measures.keys()] = \\\n",
    "[measures[measure](y_test, classifiers[\"RS-RF\"][\"c\"]) for measure in measures.keys()]\n",
    "results[\"sav\"].loc[\"RS-RF\"] = savings_score(y_test, classifiers[\"RS-RF\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAEvCAYAAACT2dMvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeVhV1cIG8HedGQ6DTAIKiiioqDjhkNN1KNMss+lW5nAtM7uB5lB5nXIsbVJxuF0rU6trlt1Kc8o0MzUHnHKeUBQFBAfm4Qzr+yP0Q0VFPbDg8P6exyfOPou9350IL3vvtbeQUoKIiIiISBWN6gBEREREVLmxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVI6VRv29fWVISEhqjZPREREVGK7du1Kk1L6qc7hrJQV0pCQEMTFxanaPBEREVGJCSESVGdwZjxlT0RERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpVaJCKoToJoQ4KoQ4IYQYVcz7/xBCpAoh9hb+Gej4qERERETkjO74pCYhhBbAXAAPAUgEsFMIsVxKeeiGoUullNGlkJGIiIiInFhJjpC2BHBCShkvpSwA8DWAx0s3FhERERFVFiV5ln11AGeLvE4E0KqYcU8JIToAOAZgmJTybDFjiIioggoZtVJ1hJucntZDdQQicgBHTWpaASBEShkJYB2ARcUNEkIMEkLECSHiUlNTHbRpIiIiIqrISlJIzwEILvI6qHDZNVLKi1LK/MKXnwJoXtyKpJTzpZRRUsooPz+/e8lLRERERE6mJIV0J4AwIUQtIYQBwHMAlhcdIIQILPKyJ4DDjotIRERERM7sjteQSimtQohoAGsBaAEskFIeFEJMAhAnpVwOYIgQoicAK4BLAP5RipmJiIiIyImUZFITpJSrAKy6Ydn4Ih//C8C/HBuNiIhIPU7mIip9fFITERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREppVMdgMgZhIxaqTrCTU5P66E6AhERUYnwCCkRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKVWiQiqE6CaEOCqEOCGEGHWbcU8JIaQQIspxEYmIiIjImd2xkAohtADmAugOIALA80KIiGLGuQMYCmC7o0MSERERkfMqyRHSlgBOSCnjpZQFAL4G8Hgx4yYDmA4gz4H5iIiIiMjJlaSQVgdwtsjrxMJl1wghmgEIllKudGA2IiIiIqoE7ntSkxBCA+AjACNKMHaQECJOCBGXmpp6v5smIiIiIidQkkJ6DkBwkddBhcuucgfQEMBGIcRpAK0BLC9uYpOUcr6UMkpKGeXn53fvqYmIiIjIaZSkkO4EECaEqCWEMAB4DsDyq29KKdOllL5SyhApZQiAbQB6SinjSiUxERERETmVOxZSKaUVQDSAtQAOA/hGSnlQCDFJCNGztAMSERERkXPTlWSQlHIVgFU3LBt/i7Ed7z8WEREREVUWfFITERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIREVUQOo89MNeehshFkei6rCtWxq9UHYnIIXSqAxAREdGd6Tz2wBT4PwiNBRJAUnYSJmydAADoEdpDaTai+8UjpERERBWA0W8thMZy3bI8Wx5m7Z6lKBGR47CQEhERlXeaPAj9lWLfSs5OLuMwRI7HU/ZERETllSiAwXsrDN6bIETxQ3xdfMs2E1Ep4BFSIiKi8kZYoffaAnOd92Csuga2vGDkXegKadffNDSrIAu7UnYpCEnkOCykRERE5YYN+io7YK79PkwBK2DPr4qc04ORe3YALBc7Iy/pSdgLqkBAINAciJFRI+Fv9sfLP7+MFSdXqA5PdM94yp6IiEg5O3Qe+2D0+wUaw0XYcoORc/4Z2HJqA/j/c/XWjKawZjTF6Wn/P6u+V51eGL5xOEZvHo2EjAS81uQ1iFud3ycqp1hIiYiIlJHQuR+EwXcdtKYU2PICkXO2H2xZ9VG0iN6Op9ETHz/4MSZvm4z//PkfnMk4g8ntJsOoNZZudCIHYiElIiIqcxJa8zEY/X6G1uUcbPl+yE3sDWtmQ9zL1XR6rR4T20xETY+amLl7Js5nn8esTrPg4+Lj+OhEpYCFlIiIqAxpXeNh8PsZOtfTsBd4Iff8M7CmNwGgva/1CiHwUqOXUMOjBkb/PhovrHoBczrPQR2vOo4JTlSKOKmJiIioDGhMZ+ES/Blca86HRn8JeUm9kH1yBKzpzXG/ZbSoh2o+hM+7fY58Wz76ru6Lree2OmzdRKWFhZSIiKgUaYxJMAUtgrnWXGhM55GX0gPZJ9+A5UprlNaJyoa+DfHfR/6LQLdA/HP9P/HN0W9KZTtEjsJT9kRERKVAGFJh9F0HveefkDYT8i90RcHltoC9bCYbBboF4ovuX+CN397A5G2TcTrjNEY0HwGtxnFHY4kchYWUiIjIgYT+Eoy+66Hz3A1IPfLTOqHgYnvA7lrmWcx6M2I7x+KDuA/wxaEvcDbzLKa3nw5XfdlnIbodFlIiIiIHELoMGHw2QO+1E5AClkttUXCxI6TNTWkunUaHUS1HoYZ7DUzfOR391/TH7M6zEWAOUJqLqCgWUiIiovsgtNkw+GyE3usPQNhhudICBWmdIa2eqqNdp3f93ghyD8Ibv72BF1a+gNldZiPCJ0J1LCIAnNRERER0bzS5MPj+DHPt6dB7b4Y1IxLZJ0cgP/mJcldGr+oQ1AGLuy+GRqPBP9b8AxvObFAdiQgACykREdHdEQUw+PwKtzrvwei3AdbsusiJH4a8pL9DWsr/jejretfFkh5LUNuzNl7/9XUsOrgIUkrVsaiS4yl7IiKikhAW6Ktsh8F3IzS6LFgz6yE/9SHY86urTnbXfF18saDbAozZPAYfxH2A0xmnMbrVaOg1etXRqJJiISUiIrotG/RV4mDw3QCNPh3W7NrITewLe25N1cHui4vOBR/87QPM3jMbn+7/FImZifiw44fwMHiojkaVEE/ZExERFcNmt2HFyRUw1/4QpsDvIS2eyEkYiNwzL1f4MnqVRmgwtNlQTGozCXEpcei7qi/OZp5VHYsqIR4hJSIiKsIu7fgl4RfM3TsX8enxkPZA5Jz9B2xZdQEI1fFKxRNhTyDIPQiv//o6+qzqg1mdZqFJ1SaqY1ElwiOkREREAKSU2JS4Cc/99BxG/DYCAPDh3z5EzqkY2LLqwVnL6FUtAlrgq0e+gpveDS+tfQmr4lepjkSVSIkKqRCimxDiqBDihBBiVDHvDxZC7BdC7BVCbBZC8MZmRERUYexI2oF+q/vhtfWvIaMgA1PbTcX/ev4PXUO6ojIduwnxDMFXj3yFRn6N8Nbvb+Hf+/7NGfhUJu54yl4IoQUwF8BDABIB7BRCLJdSHioy7L9Syo8Lx/cE8BGAbqWQl4iIyGH2pe7D7D2zsT1pO6q6VsW41uPwRNgTlXq2eRVTFcx/aD4m/jER8/bOw5mMM5jYZiIMWoPqaOTESnINaUsAJ6SU8QAghPgawOMArhVSKWVGkfFmAPx1ioiIyq0jl45gzp45+C3xN3ibvPFmizfx97p/h1FrVB2tXDBoDZjSdgpqetTE7D2zcT7rPGZ2mgkvk5fqaOSkSlJIqwMoOuUuEUCrGwcJIV4DMByAAUBnh6QjIiJyoPj0eMzdMxc/J/wMd4M7hjQdghfqvwBXvavqaOWOEAKDIgehhkcNjPl9DHqv7I25D85FqGeo6mjkhBx2YYyUcq6UsjaAtwCMLW6MEGKQECJOCBGXmprqqE0TERHd1tnMsxizeQye+PEJbD63GYMiB2HNU2vwcuTLLKN30C2kGxZ0W4Acaw76rOqDbUnbVEciJ1SSQnoOQHCR10GFy27lawC9intDSjlfShklpYzy8/MreUoiIqJ7kJKdgsl/TEbP73ti7em16Fu/L1Y/tRoxTWN4A/i70NivMf7b47/wd/XHq+texXfHvlMdiZxMSU7Z7wQQJoSohb+K6HMAehcdIIQIk1IeL3zZA8BxEBERKXIx9yI+O/AZlh5ZCjvseCr8Kbzc6GX4m/1VR6uwqrtVx+LuizHyt5GY8McEJGQk4PXmr0MjKs9dCG60a9euqjqd7lMADVGZbsdw9+wADlit1oHNmze/UNyAOxZSKaVVCBENYC0ALYAFUsqDQohJAOKklMsBRAshHgRgAXAZQH+H7QIREVEJpeenY9HBRfjy8JfIt+XjsdDHMLjxYAS5B6mO5hTcDe6Y22Uupu2Yhs8Pfo4zmWfwTrt3Ku1lDzqd7tOAgID6fn5+lzUaDSd034LdbhepqakRycnJnwLoWdyYEj2pSUq5CsCqG5aNL/Lx0PsJSkREdDd0Hntg9FuLyEX/QoA5AIMjByMtLw0LDy5EZkEmuoV0w6tNXuUEnFKg0+gwptUYhHiE4L2d72HA2gGY3Xk2qrpWVR1NhYYso3em0Wikn59fenJycsNbjeGjQ4mIqELReeyBKfB/EBoLJICk7CS8/cfbAICOQR0R3TQadb3rqg3p5IQQ6BPRB0HuQXhz05t/zcDvMrcy/n/XsIyWTOH/p1te1sDrHYiIqEIx+q2F0FhuWu5r8sXsLrMrYylSpmNwRyzuvhgA0G91P2xK3KQ4EVVULKRERFShCP2VYpdfzLtYxkkIAOp518N/e/wXIZ4hiNkQg68Of8XHjZYhrVbbvF69ehFX/xw9etSQnJysbdWqVbirq2vTfv361Sg6fubMmT7h4eER4eHhEWFhYQ2+/PLLKqqyF8VT9kRO5sZr64Y2G4oeoT1UxyJyADv0Xttv+W6AOaAMs1BRVV2r4vOHP8e/fv8Xpu2YhlPppzCq5SjoNKwZRS09utT7430fV7+Ye9Hg4+JTMLjx4HPP1n320v2s02g02o8cOVL0ce7IyMjQTJo06fy+fftcDhw44HJ1+cmTJ/Uffvhh4N69ew/7+PjY0tPTNUlJSff1l2SxWKDX3/+jdnmElMiJXL22TmO4AgmJpOwkTNg6ASvjV6qORnRfhCEVLjXnwxTwI+z5/pD263+GmrQmDG3G+bUquepdMaPTDAxoMABLjy5F9IZoZBVkqY5Vbiw9utT7vZ3v1UzLTTNISKTlphne2/lezaVHl3o7elseHh72hx9+OMtkMtmLLk9KStKbzWa7p6enDQA8PT3t9erVKwCAAwcOGNu0aRNet27diIiIiPoHDx402u12vPLKK0FhYWENwsPDIz755BMvAPjpp5/cmzdvXrdz5851wsLCGgLAvHnzvBs1alS/Xr16Eb17965ptVrvKjN/dSFyIsVdW5dny8Os3bN4lJQqKBv03pth9FsHSB1yzz8Na3pz6Dz2wui3FlpDOs8ElCMaocHwqOGo4VEDU7dNRd/VfTG3y1xUc6umOlqpG7dlXPCJyyduef+rI5ePmK12qyi6rMBWoJm2Y1rID8d/KPZpQXW86uRMbjv5bHHvXZWfn6+pV69eBAAEBwfnr1u37uStxrZu3TrH19fXEhwc3Kht27aZTz755OXevXunA0Dv3r1rjRw5Mrlfv35XcnJyhM1mE4sXL66yf/9+l8OHDx9MSkrStWzZsn7Xrl2zAODQoUOue/bsOVivXr2C3bt3m5YtW+YdFxd3xGg0yj59+tT4+OOPfaKjo0t8HQ0LKZETudW1dcnZyWWchOj+aYzJMAUug9YlEZbMCOQn94K0/vV0JWtGU1gzmuL0NJbQ8ujp8KcR5B6E4b8Ox/Mrn8fszrMR6RepOpZSN5bROy0vqeJO2d+KTqfDpk2bjv/222+uP//8s8eoUaOC4+LizOPGjUtOSUkx9OvX7woAuLq6SgDy999/d//73/9+SafTITg42NqqVauszZs3u3p6etojIyOzrx5dXbNmjfuBAwdcGzduXB8A8vLyNFWrVr2rQ6QspERORFqqQBhuLqW8to4qFisMvr/C4LsR0mZCbmJvWDMbAbivn9tUxloHtsaXj3yJ19a/hhfXvogp7aagW0g31bFKzZ2OZHb6plOjtNw0w43LfV18C5Y8uuRo6SW7nkajQadOnXI6deqU071794yBAweGjBs37q6PWri6ul67HEBKKZ555pmLc+fOvd2j5W+f614/kYjKH0tGo2KXP1778TJOQnRvNKZEuNaaA6PfelgzGiEnfjismZFgGa2YQquE4qseXyHCJwJv/PYGPvnzk0o7A39w48HnDFrDddd0GrQG++DGg++5xN2t06dP6zdv3nztsoK4uDjX6tWrF3h5edkDAgIKvvjiiyoAkJubKzIzMzUdOnTIXLZsmbfVasX58+d1O3bscGvfvn32jevt1q1bxk8//eR17tw5HQCkpKRojx07dlP5vh0eISVyGlbo3Q/DbnEHpAZaQwaqulaFxWbBN8e+wZNhTyLQLVB1SKLiCQsMvr/A4LMJ0uqOnLP9YMuKUJ2KHMDb5I1Pun6C8VvGI3ZPLE5nnMaEByZAr73/mdkVydXZ9I6eZX8r1atXb5SVlaW1WCxi7dq1VVatWnXM3d3dPnLkyKCUlBS90WiU3t7elk8++eQMAHz55ZenXn755ZqTJ0+uptfr5bfffnuyb9++V7Zu3epWv379BkIIOXHixMQaNWpY//zzz+u21bx587yxY8ee69KlS7jdboder5exsbFnwsPDC0qaV6j6TSUqKkrGxcUp2TaRo4WMUj+LXe+9CSb/Vcg5MwC27LrXrq2LvxKPF1a9gOpu1bG4++JK+8xpun+l9XWudTkNU+AyaIxpKLjcAvkXHgHsLnf+RKBMriEtD/++b1QRr52VUuLjPz/GvL3zEOUfhRkdZ6CKqVzcArNEhBC7pJRRRZft27fvdOPGjdNUZapo9u3b59u4ceOQ4t7jKXsiJyC0mTD6roc1qy5s2dc/pSa0Sije/9v7OH7lOEZvHg27tN9iLURlTOTD6P8jXGr+BxA25CS8hPzkp0pcRqliEULg1cavYlr7adiXug99VvdBQkaC6lhUTrCQEjkBQ9W1gMaCvJRHi32/XfV2GBk1EuvPrMecPXPKOB3RzbTm4zCHzoDeaxsslx9AdvzrsOWEqY5FZaBHaA989vBnyMjPwAurXsDO5J2qI1E5wGtIiSo4jSkRes9dsFxqB1lQ7K3sAAB96vfBiSsn8Mn+T1C7Sm3es/E+8BTufdDkwui/EoYqcbDl+yEv4RXYckNUp6Iy1rRqU3zV4ytEr4/GoHWD8PYDb6NXnV6qY5FCPEJKVKFJGP1XQNrMyE/rctuRQgiMbTUWzf2bY/yW8fgz9c/bjidyNK3bIZhDP4Leczfy0zoi59QQltFKLNg9GF888gWi/KMwbss4xO6O5SVFlRgLKVEFpvPYB51rAgouPAzYTXccr9fqMaPjDPi5+mHor0N5w3wqE0KbBVO1JXANXgxpMyPn9D9RkNoNkJVrljXdzMPggXkPzsNTYU/hk/2fYORvI5FnzVMdixRgISWqqEQBjFVXwZZbHZb05iX+NC+TF+Z0noNcay6GbBiCHEtOKYakyk1C57EPrqEzoPM4gPzUh5BzKhr2vCDVwagc0Wv0ePuBtzEyaiR+SfgFL659EWm5nLhe2fAaUqIKyuCzERp9BnLO9cbd/m5Zx6sO3uvwHqLXR2PslrH44G8fQCP4+yk5jtBlwBjwPfTuh2HLDUbumadgz+cTw6h4Qgj0b9AfQe5B+Nfv/0Lvlb0xp8schHuFq452V0JGrSz50YESOD2tx66SjHvrrbcCvvvuOx+NRiM1Gg3mzZuX0Llz55tuYH87zz77bM0333wzpXnz5koOUfMnEFEFJPSXYPDZBEt6k3u+Bq9DUAeMiBqBdQnr8O99/3ZsQKrEJHSeO2EO/Qg683HkpTyCnNOvsoxSiXSp0QULuy2EzW5Dv9X9sPncZtWRyr1ffvnFvHbt2ir79+8/dOzYsUO//vrrsdDQ0BLfkP6qpUuXJqgqowALaaWwMn4lui7rishFkei6rCtWxpe/GcJ0d4xVVwMQyL/Q/b7W0y+iH3rV6YWP932MNafWOCYcVVpCfwkuwQvgUu072PICkR3/OiyXOoA/auhuRPhE4KseXyHYPRivrX8NXx/5WnWkcu3cuXN6b29vq4uLiwSAwMBAa0hIiGXkyJGBDRs2rB8WFtbg+eefr2m327Fnzx5To0aN6l/93KNHjxrCw8MjAKBly5Z1N23a5AoArq6uTWNiYqrXrVs3onHjxvXOnj2rA4CDBw8aGzduXC88PDxiyJAh1VxdXZsCQEJCgj4qKqpuvXr1IsLCwhqsWbPG7W73g98lnNzK+JWYsHUCkrKTICGRlJ2ECVsnsJRWYFrXk9B77EdBWkdIq+d9rUsIgXGtx6FZ1WYYu2UsDqQdcFBKqlzs0HtthTl0JrQuCchL6oXcMy9DWnxVB6MKKsAcgEXdFqFD9Q6Yun0qBq8bzAMrt9CrV6+M8+fPG0JCQhr26dOnxsqVK90A4I033rhw4MCBw8ePHz+Ym5ur+frrrz2bNm2aZ7FYxJEjRwwAsHjxYu9evXpdvnGdubm5mgceeCDr6NGjhx544IGs2bNn+wFAdHR08D//+c8Lx44dOxQUFGS5On7BggXeXbp0ST9y5Mihw4cPH2zVqtVdT05gIXUiNrsNKdkp2Je6D2tOr8Gig4sw+Y/JyLNdfwQ+z5aHWbtnKUpJ98cGo/8K2Au8UHCpg0PWaNAa8FHHj+Bj8sGQDUOQkp3ikPVS5SAMqXCpOR+mgOWw5YQgO34YLFdagz9e6H656l0xs9NMtK3WFlvOb+GBlVvw9PS0Hzhw4NCcOXMS/Pz8rP37968dGxvrs3r1avfIyMh64eHhEVu3bnU/cOCACwD06tXr0uLFi70B4Pvvv/fq27fvpRvXqdfr5XPPPZcOAM2bN89OSEgwAMCePXvcXnzxxUsAMHDgwItXx7du3Tp7yZIlvsOHD6+2Y8cOFy8vr7u+fxcnNVUQUkpkFGQgOTsZydnJSMpOuu6/ydnJuJBzAVZpLdH6eLufiklfZSe0pmTkJr7g0Fvm+Lj4YHaX2ei7qi+G/joUn3f7HC46Pr6RbscGg/dmGPzWAVKH3PPPwJreDIBQHYyciFajRXx6/E3Lrx5Y4QM+/qLT6fDoo49mPvroo5mRkZG5n3zyie/Ro0ddt2/ffqhOnTqW4cOHV8vLy9MAQN++fS8/88wzoc8999xlIQQaNWqUX8z6pEajubZuq9V623/Y3bt3z9q0adPR7777zvPFF1+sFR0dnRIdHX3xdp9z0zbvZjCVnjxr3l/FMicZSVlJSM5JRkp2ynXFM9eae93n6IQO/mZ/BJgD0NS/KQLNgQhwDUCgWyD8Xf0R6BaIp5c/jaTspJu2F2DmBIMKR5MDg9/PsGaHwprZ0OGrD/cKx/QO0zFkwxCM3zIe73V4D0KwXNDNNMZkmAKXQeuSCEtGA+SnPA5p9VAdi5zUrQ6g8MDKX/bt22fUaDTXiuWePXtc6tSpk3/06FHXgIAAa3p6umbFihVejz322GUAaNCgQb5Go8H48eOrPfHEEzcdHb2dJk2aZC1cuNDr5ZdfvrxgwQLvq8uPHTtmCA0NLRgxYkRafn6+2L17tysAFtLyxma3ITU39bZHNy/n33QJB3xMPgg0ByLUMxRtqrVBgDngr9JZ+F8fF5873qpnaLOhmLB1wnWn7TXQIKZpjMP3k0qX0e8XCG0u8lMeQ2kdheoY3BGvN38dM3bNQGiVULza+NVS2Q5VVFYYfH+FwXcjpM2E3MTesGY2Ao+KUmkKMAdUmAMrJb1NkyNlZGRohwwZUiMjI0Or1WplSEhI/qJFixKqVKlirV+/fgM/Pz9r48aNr7sF1JNPPnlp8uTJQdOnTz93N9uaPXv22RdeeKHW+++/H9i5c+cMNzc3GwCsXbvWPTY2NkCn00lXV1fbV199depu94OF9D5JKZGen359ycxJRnLWX0c7r55Kt0nbdZ/npndDgDkAAeYANPRteF3ZDHANgL/ZHwat4b7zXT2dMWv3LCRnJ8PD6IH0/HRcyb9y3+umsqMxpEDvtQ2WK61gzw8s1W0NaDAAJ6+cxLy98xDqGYqHQx4u1e1RxXAg7QBca82B1pQMS3oT5Kc8Bmkzq45FlUBxB1ZMWhOGNhuqMFX50b59+5w9e/YcuXF5bGzs+djY2PPFfc6kSZNSJk2adN2EgR07dhy9+nFOTs6eqx8PGDDg8oABAy4DQEhIiGXv3r1HNBoN5s+f73X8+HEjAMTExFyMiYm5qyOiN6pUhXRl/MprxSzAHIChzYbe8fqTXGvutaJ54yn0q0c3b5w0pNfor5XNFgEt4O/qf33hNAfA3eBemrt6nR6hPa7tp5QSMRtiMGv3LLSt3hahnqFlloPu1V/Pq4fdiPzUh0p9a0IIjH9gPBIyEjB281gEuQehgU+DUt8ulU951jzM2zsPiw4tgtC6Iedsf9iy6t/5E4kc5MYDKyX9+U2Ot2XLFtehQ4fWkFLCw8PDtnDhwtOOWnelKaRXb390tTxenaV3Oe8yInwibiqZyTl/vU7PT79uPQICfi5+CDAHIMwrDB2COtxUNpJoFVsAACAASURBVL1N3uX2qTdCCExoMwFP/PgExvw+Bl888gV0mkrzZVAh6dwOQed2AnnJjwFldETKqDViZqeZ6L2yN4ZsGIKve3wNP1e/Mtk2lR+7Unbh7a1vIyEjAU+FPYWFPzUC7CbVsagSKnpghdTp1q1b1tGjRw+VxrorTROZtXtWsbc/mr5z+nXL3A3u18plY7/G10rm1clCVV2qQq913OxmFXxdfDG29ViM/G0kPtv/GV5p/IrqSHQrwgKj/0rY8vxhudy6TDft6+KL2Z1no+/qvhiyYQg+7/Y5TDqWkcogx5KDmbtnYsmRJajuVh2fdP0ErQNbY+Fy3maH6AZ2u90uNBqNVB2kvLPb7QLALW8HVWkK6e1m43384MfXSqir3rUMU6nzcMjDWJ+wHh/v+xgdgjqgvg9PwZVHBu8t0BguISfhJQDaMt9+Xe+6eLf9u3j919cxfut4TG8/nTPvndzW81sxcetEJGUnoU/9PohpGlNpvi8S3YMDqampEX5+fukspbdmt9tFamqqJ4BbPn2l0hTSW83SCzQHom31tgoSqTem9RjsTNmJ0ZtHY+mjSx0yiYocR+gyYPDdAEtmBGw5YcpydKnRBUObDcWs3bNQp0odDIocpCwLlZ6Mggx8sPMDfH/ie4R4hGBx98VoUrWJ6lhE5ZrVah2YnJz8aXJyckPwaRC3YwdwwGq1DrzVgEpTSDlL72aeRk9MbDMRr61/DXP3zsWw5sNUR6IijH5rANiQn6L+uqmXGr6EE1dOYPae2Qj1DMWDNR9UHYkcaMOZDZiybQou5V3CwEYDMbjxYBi1RtWxiMq95s2bXwDQU3UOZ1BpCiln6RWvQ1CHvyYrHFyITsGdeESknNCYzkBfZTfy0zpCWnxUx4EQAhPbTMTZjLMYvXk0qrtV52UeTuBS3iVM2z4Nq0+vRl2vupjTZQ4ifCJUxyKiSqjSFFKAs/Ru5Y0Wb2Bb0jaM2TwG3z72La8XU84OU8AK2C3uKLjYSXWYa4xaI2Z1noXnfnoOQ34dgiU9lsDXxVd1LLoHUkqsOb0G725/F5mWTEQ3icaLjV6EXlOxJ2wSUcXF6x0IZr0Zk9tOxpnMM5i5e6bqOJWeznMPtC5nkZ/aDbCXr9OmV2fep+enY+ivQ5Fvu+kRyFTOXci5gCG/DsGbm95EkHsQvn30W7zS+BWWUSJSioWUAAAtAlqgT/0+WHJkCbYlbVMdp/LS5MNYdQ1sucGwpjdVnaZY9X3q45127+DP1D8xYesESMmJpRWBlBLfH/8evX7ohT/O/4GRUSPxRfcvUMerjupoREQspPT/hjYbihCPEIzbMg6ZBZmq41RKBp8N0Ogy/7oJfjn+5/lgzQcR3SQaP8X/hM8OfKY6Dt3BuaxzeGXdKxi/dTzqetfF/3r+D/0b9IdWU/a3EiMiKk75/YlHZc6kM+Gddu8gNScV03dMv/MnkEMJfRoM3pthudIM9rwaquPc0aDIQege0h2xu2Ox4cwG1XGoGHZpx38P/xdP/PgE9qXuw7jW4/DZw5+hhkf5//oiosqFhZSu08ivEV5q9BJ+PPkjfj3zq+o4lYrRfyUgtX9dO1oBCCEwqe0kNPBpgFG/j8LRS0dVR6IiTqefxoA1A/DujnfRzL8Zfnj8B/y97t/L7WONiahy43cmusngyMGo510PE/6YgMt5l1XHqRS05mPQux9GwcXOkFYP1XFKzKQzYVbnWXA3uCNmQwwu5l5UHanSs9qtWHBgAZ5a/hROXDmBKW2n4N9d/o1At0DV0YiIbomFlG6i1+oxtd1UZBRkYPK2yZy0UupsMPr/BHuBDwoutVMd5q5Vda2K2M6xuJx3GcM2DkOBrUB1pEpF57EH5trTELkoEp2+6YQe/+uBGbtmoH1Qe/zY60c8XudxPu6ViMo9FlIqVrhXOF5r8hrWJazD6lOrVcdxanqvbdAaLyAvpQcgK+atgRv4NMCUdlOw58IeTPxjIn+JKSM6jz0wBf4PGsMVSEik5abhfPZ5PF/veczoOIP3iSWiCoOFlG5pQIMBiPSLxNTtU3Eh54LqOE5JaLNg9FsHa1YYbFkV+8lHD4c8jFcbv4rlJ5dj4cGFquNUCka/tRAay03LN57dyKOiRFShlKiQCiG6CSGOCiFOCCFGFfP+cCHEISHEn0KI9UKImo6PSmVNq9HinXbvoMBWgPFbx/OoVykw+K0DNAXIT3kUQMUvEIMbD0bXml0xY9cMbDy7UXUcpya0mRD6K8W+l5ydXMZpiIjuzx0LqRBCC2AugO4AIgA8L4S48WHHewBESSkjASwD8J6jg5IaNT1qYljzYdhybgu+O/6d6jhORWM8D32VHbBcbg17gb/qOA6hERpMaTcF9X3q461Nb+H45eOqIzkhO/Ref8Bc+8NbjggwB5RhHiKi+1eSI6QtAZyQUsZLKQsAfA3g8aIDpJS/SilzCl9uAxDk2Jik0nP1nkOrwFZ4f+f7SMxMVB3HSUgY/VdA2lyQn/qg6jAO5aJzQWynWJj1ZsRsiMGlvEuqIzkNjSkRriHzYAr4Eba8IORfeATSfv0jP01aE4Y2G6ooIRHRvSlJIa0O4GyR14mFy27lJQCcBeNENEKDyW0mQyM0GLtlLOzSrjpShadzPwCd+RQKUrsCdlfVcRzO3+yP2M6xSMtNw7Bfh8Fiu/k6R7oLmlwY/X+Aa8hcCH06cs89j9wzL8FyqQPykp6EvaAKBAQCzYGY0GYCeoT2UJ2YiOiuOHRSkxCiD4AoAO/f4v1BQog4IURcamqqIzdNpSzQLRBvtXwLu1J24ctDX6qOU7EJC4xVV8KWFwDLlZaq05Sahr4NMbntZOy+sJu3D7tnEjqP3TDX/hB6r+2wXG6D7JMjYM1ojKvXHFszmiL75Cj82f9P/Pz0zyyjRFQhlaSQngMQXOR1UOGy6wghHgQwBkBPKWV+cSuSUs6XUkZJKaP8/PzuJS8p9Hjtx9ExqCNm7Z6F+CvxquNUWAafTdAYriA/pSec/UYX3Wt1x6DIQfj+xPdYfGix6jgVisaQApca8+FS/RvIAm/knIpBfspjgN2kOhoRkcOV5KfhTgBhQohaQggDgOcALC86QAjRFMB/8FcZ5f2BnJQQAm+3eRuueleM3jwaVrtVdaQKR+iuwOCzEZaMRrDlhKqOUyZea/IaHqr5ED7a9RE2JW5SHaf8EwUw+K2Ga+gsaE3JyEt6EjkJg2HPr6Y6GRFRqbljIZVSWgFEA1gL4DCAb6SUB4UQk4QQPQuHvQ/ADcC3Qoi9Qojlt1gdVXC+Lr4Y13ocDl48iE/3f6o6ToVjrLoagER+yiOqo5QZjdBgStspqOtVF29uehMnr5xUHamcktC5HYS59kcw+v4GS3ozZJ8cUXhZh3MfSSciKtF3OSnlKilluJSytpRyauGy8VLK5YUfPyil9JdSNin80/P2a6SKrGtIV3Sv1R3/2fcfHL54WHWcCkPrchp6z30ouPg3SKuX6jhlylXvitjOsTBpTYheH43LeZdVRypXhP4SXIIWwSX4C0ibCTmnByM/6WlIm5vqaEREZYK/dtM9GdNqDLxMXhi9eTSfXV4idhgDlsNu8UTBxb+pDqNEgDkAsZ1jcSHnAoZvHM6Z9wAgrDD4bIA59CNoXeORl9IDOadiYMsNUZ2MiKhMsZDSPfE0emJCmwk4ceUE5u6dqzpOuaevEget6TzyLzwCSIPqOMpE+kViYtuJiEuJw9TtUyv1zHut63GYa82EserPsGbVR3b8CFgutQegVR2NiKjM6VQHoIqrQ1AHPBX2FBYeXIhOwZ3QpGoT1ZHKJ00eDH5rYc0JgTUjUnUa5R4NfRQnr5zEp/s/RZ0qddAnoo/qSGVK6DJgrLoSes99sBf4IOfMANiy66qORUSkFAtpKQkZtVJ1hJucnub4+xO+0eINbEvahtGbR2PZY8vgqne+m7zfL6PvLxDanL9u2eMEz6t3hJimMYi/Eo/3495HLc9aaFu9repIZcAGvdc2GP1+BoQN+akP/nX5htTf+VOJiJwcT9nTfTHrzZjcdjISMxMxY9cM1XHKHWFIhd57KyxXomDPu90DzioXjdDg3fbvIqxKGN747Q3Epzv3fW01pjNwrTUHpoAVsOXWRHb86yhIe5BllIioEAsp3bcWAS3QJ6IPvj76Nf44/4fqOOWKyf8nwK5HQerDqqOUO656V8zuPBt6rR4x62OQnp+uOpLjabNhDPgfzLXmQWizkZv4AnLPDoC0+KpORkRUrrCQkkMMaToEtTxrYdyWccgsyFQdp1zQmo9A53YU+WkP8vY9txDoFohZnWYhKTsJIzaOgMXuLDPv7dB5xsEc+hH0VeJQcLE9suNHwJrZCLxsg4joZiyk5BAmnQlT205FWm4apu2YpjpOOWCFyf8n2PL9YLn0gOow5VqTqk0woc0EbE/ejmnbp1X4mfcaYxJcav4HLtWWwV7gi5xTQ5B/oQdgN6qORkRUbrGQksM08muElxq9hOUnl2PDmQ2q4yil994KjTEN+SmPgnMH76xn7Z4Y0HAAvjn2DZYcWaI6zr3R5MNY9Se41poNjSENueefRm7CK7DnB6hORkRU7vEnJTnU4MjB+D3xd0z8YyKaVG0Cb5O36khlTmgzYfRdD2tmPd7O5y4MbToUp66cwns730OIZwjaVGujOlIJSejcD8DovwIafQYKLrdEfurDgM2sOhgRUYXBI6TkUHqtHlPaTUFmQSambJtS4U+/3gtD1bWAxoq8C46/zZYz02q0mNZhGkKrhGLkbyNxOv206kh3JPRpcAn+HC5BX0HazMg+9U/kJz/JMkpEdJdYSMnhwr3C8VqT17AuYR1WnVqlOk6Z0pgSoffcBcultpAFfqrjVDhmvfmvmfcaPWI2lOOZ98ICg+86mENnQuuSgLzkx5BzKhr2vBqqkxERVUgspFQq/tHgH2js1xhTt09FSnaK6jhlRMLkvwLSZkZ+WmfVYSqs6m7VMaPjDCRmJWLkbyPL3cz7zec2wxw6E0a/9bBmNvjrkZ+X24KP/CQiuncspFQqtBotprabCqvdirf/eLtSnLrXeeyD1jUB+RceBuwm1XEqtGb+zfD2A29jW9I2vLfjPdVxAADJ2ckYvnE4Xv3lVUAK5CQMRN755yGtHqqjERFVeCykVGpqetTEsObDsOXcFiw7vkx1nNIlCmCsugq23OqwpjdXncYp9KrTC/0j+uPro19j6ZGlynJY7BYsOrgIPX/oiU2JmxDTNAbZp16HLaeOskxERM6GhZRK1bN1n0XrwNZ4f+f7OJt5VnWcUmPw3QiNPgN5KT3Bf1aOM6z5MHQI6oB3d7yL7Unby3z7u1N249mfnsUHcR+gRUAL/PD4DxgUOQiQvEEJEZEj8ScnlSqN0GBy28nQCi3Gbh4Lu7SrjuRw57LOweC9CZb0JrDn1lQdx6loNVpMbz8dtTxrYfjG4UjISCiT7V7Ku4RxW8ah/5r+yCrIwqxOszCn8xwEuQeVyfaJiCobFlIqdQHmAIxqOQq7L+zGF4e+UB3H4T6M+xCAQP6F7qqjOCU3gxtiO8dCIzSIXh+NjIKMUtuWXdrx7bFv8dj3j+Gnkz/hxYYv4ofHf0DnGp0hBB/5SURUWlhIqUz0rN0THYM7InZ3LOKvxKuO4zA7knZgXcI6FKR1hLR6qo7jtILdg6/NvH/jtzdgtVsdvo3DFw+j76q+mPTHJIR7hWNZz2UY1nwYXPWuDt8WERFdj4WUyoQQAm8/8DZc9a4YvXl0ubuVz72w2q2YtnMaqrtVR8GlDqrjOL2ogCiMaz0OW89vxQdxHzhsvZkFmXh3+7t4buVzSMxKxDvt3sGChxegdpXaDtsGERHdHgsplRlfF1+Maz0OBy8exKf7P1Ud5759d+w7HL98HCOiRgBSrzpOpfBk2JPoU78Pvjr8Fb499u19rUtKiVXxq9Dzh55YcmQJngl/Bst7LcdjtR/j6XkiojLGqaJUprqGdMUjZx7B/H3z8begvyHCJ0J1pHuSnp+O2Xtno2VASzxY40EAleuJVCqNiBqBUxmn8M62dxDiEYIWAS3ueh2n0k9h6vap2J60HQ18GmBO5zlo4NugFNISEVFJ8AgplbnRrUbD2+SNMZvHIN+WrzrOPZm3dx4yCzLxZos3eTStjOk0Orzf4X3U8KiBYRuH4WxGyW8nlmvNRezuWDy5/EkcSjuEsa3G4qtHvmIZJSJSjIWUypyn0RMT2kzAiSsnMHfvXNVx7trxy8ex9OhSPBP+DOp611Udp1JyN7hjTuc5AIDoDdHILMi84+f8dvY3PPHjE/hk/yfoHtIdy59YjmfrPQutho/8JCJSjYWUlGgf1B5Phz+NhQcWYs+FParjlJiUEtN3TodZb0Z0k2jVcSq1YI+/Zt6fyTiDNze9CZvdVuy481nnMXTDUERviIZRa8SChxfgnfbvwNfFt4wTExHRrbCQkjIjo0aimls1jNk8BjmWHNVxSmTD2Q3YnrQd/2zyT1QxVVEdp9JrEdACo1uPxuZzm/HRro+ue89is+Cz/Z+h14+98EfSHxjWfBiWPbbsnq45JSKi0sVJTaSMWW/GlLZT8OLaF/HRro8wtvVY1ZFuK9+Wjw92foA6Verg2brPqo5DhZ4JfwYnLp/A4kOL8ePJH5GRnwFvkzc0QoPU3FR0Du6Mt1q+hWpu1VRHJSKiW+ARUlIqKiAKfSP6YunRpdh6fqvqOLf1xaEvkJiViDdbvAmdhr/LlScNfBpAIzRIz0+HhMTFvItIy01D/4j+mNV5FssoEVE5x0JKysU0jUGoZyjGbxlfqo+FvB8Xci5g/p/z0Tm4Mx6o9oDqOHSDOXvnwC7t1y2TkPg54WdFiYiI6G6wkJJyJp0JU9tNRVpuGqbvmK46TrFm7poJq92KkVEjVUehYiRnJ9/VciIiKl943pHKhYa+DTGw0UD858//oHONzuhSo4vqSNfsS92HFfErMLDRQAR7BKuOQ8UIMAcgKTup2OVEdG9CRq1UHeEmp6f1UB2BSomyQrr/XLqyL3Z+QZdPr0S+gk2JmzDpj0loWrUpvE3eqiPBLu2Ytn0a/Fz8MLDRQNVx6BaGNhuKCVsnIM+Wd22ZSWvC0GZDFaYiIqKS4il7Kjf0Wj2mtpuKzIJMTP5jMqSUqiNhxckVOHDxAIY1Hwaz3qw6Dt1Cj9AemNBmAgLNgRAQCDQHYkKbCegRyl8+iYgqAp6yp3IlzCsM0U2jMWPXDKw8tRKPhj6qLEu2JRszd89EpG8ki00F0CO0B/+eiIgqKB4hpXKnf0R/NPFrgne2v4OU7BRlOeb/OR9puWkY1XIUNIL/VIiIiEoLf8pSuaPVaDG13VRY7Va8vfVtJafuz2ScwReHvkDP2j3RyK9RmW+fiIioMmEhpXKphkcNDG8+HFvOb8G3x74t8+2/H/c+9Bo9Xm/2eplvm4iIqLJhIaVy6+91/47Wga3xQdwHOJt5tsy2u/XcVmw8uxGDIgfBz9WvzLZLRERUWbGQUrmlERpMbjsZOqHD2M1jYbPbSn2bFrsF03dOR7B7MPpG9C317RERERELKZVzAeYAjGo1Crsv7MaXh78s9e0tPbIU8enxeCPqDRi0hlLfHhEREbGQUgXwWOhj6BTcCbG7Y3HyyslS286lvEuYt3ce2lRrg47BHUttO0RERHQ9FlIq94QQGP/AeJj1ZozePBoWu6VUtjNnzxzkWHPwZos3IYQolW0QERHRzUpUSIUQ3YQQR4UQJ4QQo4p5v4MQYrcQwiqEeNrxMamy83XxxbgHxuHQxUP4dP+nDl//kUtHsOzYMjxf73nUrlLb4esnIiKiW7tjIRVCaAHMBdAdQASA54UQETcMOwPgHwD+6+iARFc9VPMh9Ajtgfn75uPgxYMOW6+UEtN2TIOn0RODGw922HqJiIioZEpyhLQlgBNSyngpZQGArwE8XnSAlPK0lPJPAPZSyEh0zb9a/gveJm+M3TwW+bZ8h6zz54SfsStlF2KaxsDT6OmQdRIREVHJlaSQVgdQ9CaQiYXLiMqcp9ETE9tOxIkrJzB3z9z7Xl+uNRcfxn2Iul518VTYUw5ISERERHdLV5YbE0IMAjAIALQevOE43Zt21dvhmfBnsPDgQnQM7ohm/s3ueV0LDy5EUnYSprabCq1G68CUlUPIqJWqI9zk9LQeqiMQEdFdKskR0nMAgou8DipcdteklPOllFFSyiitK0+N0r0bETUC1dyqYczmMcix5NzTOpKykrBg/wJ0rdkVLQJaODghERERlVRJCulOAGFCiFpCCAOA5wAsL91YRLdn1psxtd1UnMs6h492fXRP65ixawYkJEZEjXBwOiIiIrobdyykUkorgGgAawEcBvCNlPKgEGKSEKInAAghWgghEgE8A+A/QgjHTYEmuoXm/s3RL6Iflh5diq3ntt7V5+5K2YXVp1djQMMBqOZWrZQSEhERUUmU6D6kUspVUspwKWVtKeXUwmXjpZTLCz/eKaUMklKapZQ+UsoGpRma6KqYZjEI9QzFuK3jkFGQUaLPsdltmL5jOvxd/fFiwxdLOSERERHdCZ/URBWaUWvEO+3ewcXci5i2fVqJPuf7E9/j8KXDGBE1Ai46l1JOSERERHfCQkoVXgPfBng58mWsiF+B9Qnrbzs2oyADs/fMRrOqzdAtpFsZJSQiIqLbYSElpzAochDqe9fHpG2TcDH34i3HfbzvY1zOu4xRLUfxefVERETlBAspOQW9Ro+p7aYisyATU7ZNgZTypjHx6fFYcngJngx7EvV96itISURERMVhISWnEeYVhpimMfjlzC/4Kf6n696TUuK9ne/BReeCmKYxihISERFRcVhIyan0i+iHplWb4t3t7yI5O/na8t/P/Y4t57ZgcOPB8HHxUZiQiIiIbsRCSk5Fq9FiStspsEorJmydACklLDYL3tv5HkI8QvB8vedVRyQiIqIbsJCS06nhUQMjmo/AlvNb0P7r9mj2ZTMkZCSgU3An6LV61fGIiIjoBiyk5JTMejM00CC9IP3asiVHlmBl/EqFqYiIiKg4LKTklGL3xMIO+3XL8mx5mLV7lqJEREREdCsspOSUik5oKslyIiIiUoeFlJxSgDngrpYTERGROiyk5JSGNhsKk9Z03TKT1oShzYYqSkRERES3olMdgKg09AjtAQCYtXsWkrOTEWAOwNBmQ68tJyIiovKDhZScVo/QHiygREREFQBP2RMRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVI61QHIuYSMWqk6wk1OT+uhOgIRERHdBo+QEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkVIkKqRCimxDiqBDihBBiVDHvG4UQSwvf3y6ECHF0UCIiIiJyTncspEIILYC5ALoDiADwvBAi4oZhLwG4LKWsA2AGgOmODkpEREREzqkkR0hbAjghpYyXUhYA+BrA4zeMeRzAosKPlwHoIoQQjotJRERERM6qJIW0OoCzRV4nFi4rdoyU0gogHYCPIwISERERkXMTUsrbDxDiaQDdpJQDC1/3BdBKShldZMyBwjGJha9PFo5Ju2FdgwAMKnxZF8BRR+3IXfIFkHbHUc6H+125cL8rF+535cL9Lns1pZR+irbt9HQlGHMOQHCR10GFy4obkyiE0AHwBHDxxhVJKecDmH9vUR1HCBEnpYxSnaOscb8rF+535cL9rly43+RsSnLKfieAMCFELSGEAcBzAJbfMGY5gP6FHz8NYIO806FXIiIiIiKU4AiplNIqhIgGsBaAFsACKeVBIcQkAHFSyuUAPgPwhRDiBIBL+Ku0EhERERHdUUlO2UNKuQrAqhuWjS/ycR6AZxwbrVQpv2xAEe535cL9rly435UL95ucyh0nNRERERERlSY+OpSIiIiIlHL6QiqEsAkh9gohDgghVgghqhQuDxFC5Ba+d/WPQXVeRyiyzweFEPuEECOEEBohxMNF9jWr8HGwe4UQi1VndhQhRFYxyyYIIc4V7ushIcTzKrKV1P1+zQohNhb5uz1ceLu1q++dFkL8fsP4vYW3boMQoqMQIr1w2REhxAelvb+F273vr1khxEIhxKki2d8u8t5GIcSZog/sEEL8cPXr5Yb/t4eEEIuFEPoy2vf7+pot8v9unxBitxCiTeHyECGEFEJMKTLWVwhhEULMudvtlBUhRIAQ4mshxEkhxC4hxCohRLgQIrbw38R+IcROIUStwvGnC5ftL9yHKUIIkxCiUZGvnUtFvjZ+Ub2Pt3If+/6nEOI3IUTNIusq+n3kWyGEq7o9u9ltvs9pbrW/N3z+Lb9XCSH+IYRILfL37zQ/45yalNKp/wDIKvLxIgBjCj8OAXBAdb4y2OeqAH4BMPGGMRsBRKnOWpr7XmTZBAAjCz8OA5ABQK86awn//u76a7bo3y0AbwCXARgKX58GsBdAcOHr+oWvDxS+7gjgp8KPXQAcAdC2jPf5nr5mASz8v/bONdSKKgrA36ogDYuMtAfZyyDBuL5+9EcrsbACQytQESopyOhBaWL96EdJ7yQIjQsZZmBdhVIQSgu1MpBEU29chETtYWlPoVKp1NWPtSf3Hedcz2POmXPvWR8M99y9z5691sw6a9asvWc2cGf43A/YA1wRte0Exob/zwW+SPqNjy328OZ6YEZvsNnUsZsIfBrptAfYFtU/EM73wkr7adCxEGATMCsqGwE8ha0CeFoouwQYGNn0+eHzAOAdYGkp22jWLQfdnwbe/Hlv5AAABRpJREFUKGEXy4DZRevYg93Gfm56KX1T7W+ghK8C7kls3Lfes/X5DGmKTZy8ylSfRlV/xhYjeCjODrUqqroLOAwMLFqWMqnVZgcAh4BjUdkKYGr4PB14N6uhqh7BgpeG/mZystl+4e+hqKyDE28AuR14v0T/x4DNNImvqNBmz8FuQBIOAztFJHlv41Ts/NfaT70YD/yrqu1JgaruwM7jflU9Hsr2qerBdGNV/QuYBUwWkfMaJHNe1KQ7PfuKjcBVOcubJ7HsF1Gevv9TlK9y8qVlAlIROR2YQPd3qA6NUvqLChKt7qjqHizrM7hoWYpGREYDu0LQ09TUaLPLRKQTWw1tfgiyEt7DAjKAScDqEv0PxLJmn1WrQ7XUYLMvi8h2bInjjtR5XgdcF47rNGB51g5EpB9wLbCmYsHrQBk22z8ZtgQWA/NT9R3ANBEZgt2Y/FhlP43gGmBrRvkKYFLQc4GIjCq1A1X9A9iL2W5volbdbwZWpQvFFqu5BfgqN0lzJMPPlX2uo31k+aqpka+cmbvgTu60QkDaP1ygDgAXAB9HdbtVdWTYHixGPKdBPCYiXdgw7bNFC3MK8rDZGaraBlwKPB7PLcNWUTsoItOAnVhWLGaciOzAVmBbq6oHalWogcxV1ZHAhcCEZD5l4BjwORaM9lfVb1Jth4bj/hOWoelshMA9UK7NHgn2MAwLSt5OZZbXADdROghv+t+G2rLUVwNPAseBdSIyoYcmfWY0qAzdN4jID1jQGY92JH5kC/Ad9r7wZiLTz1V4rnvyVcsjX7mkblo4udEKAemRcIG6DHNSLRd4isiV2MW46bOCdeRVVR0O3AG8GbJgzUpFNisia0MWYHG6TlV/Ab7EMn4xy4FFZA/Xb1TVEcBw4F4RGVmFDjVRjs2KyJKg9wfpujB0+wkwNlXVAbxG9rD17nDchwJjROS2KsXPi5NsVkSGRFmfWekGqroJW+t7UFT2D5Z5m4PNzTtlP/VQpky6gDFZFar6t6p+qKpzgeeAyVnfE5GzsfmzX9dLyDpRre7jMV+xHZtHmnAkCsgeDnbQTJT0c1n6isiUyPaTKSiF+yonP1ohIAVAVQ8DjwBzwhBGSyAig4B2bIJ3y790Vm1lsS2cWOq2aSnXZlV1Yrjo3JeuC0/WjgJ2p6pWAi9hK7CV2u9e4AVgXhXiV025NquqM4Pet2bs4wwsCE/rvRF4nhLzZsN+fwWewDI0hRPbrKp+HwUZ7envisgwbKrDb6mqBcA8Vf29nH7yk75i1gNnSvc3Q7SJyPUicnH4/zSgDfg23VhEBgCvA6tONe+wCalad1U9CjwK3NXb5s6m/ZyIjM7SV1VXRra/JbWPQnyVky8tE5ACqOo27Enbwl9tUmeSOWVd2NPKH9H9zrkvc5aI7Iu22RnfeQaYHZxdU1ODzS4Lw2FbgbdUtdvcNFX9U1VfLCNr0o7Nu7y8wv4rJS+bTeaQdmJz5ro9uKTGKyHo7IlVmC2Nq0KGSqnVZpNjtx3LfN+dmjOMqnap6tIyZCn0txFuQKYAN4q9+qgLu4FoA1aLvZ6sEzgKLIyabgh1m7Hh6fsbK3nt1KB70n4/dqPV60YBU35uMGXom0GjfJVTJ3ylJsdxHMdxHKdQmj5D5DiO4ziO4/RtPCB1HMdxHMdxCsUDUsdxHMdxHKdQPCB1HMdxHMdxCsUDUsdxHMdxHKdQPCB1HMdxHMdxCsUDUsdxHMdxHKdQPCB1HMdxHMdxCuU/KuTas1k173IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is observed that the savings of the RF versus the RS-RF did not differ significantly, however in terms on F1Score, as expected, there is an increase in the results.\n",
    "\n",
    "To be fair, the Randomized Search algorithm is not finding the hyperparameters taking into account the savings. Therefore, those parameters could still be tunned to attempt to maximize the savings score.\n",
    "\n",
    "### Parameter search focusing on savings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed:    6.7s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Model :\n",
      "Mean validation score: 0.075 (std: 0.011)\n",
      "Parameters: [{'bootstrap': False, 'criterion': 'gini', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 77, 'min_samples_split': 19, 'n_estimators': 20}, {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 3, 'max_features': 1, 'min_samples_leaf': 35, 'min_samples_split': 93, 'n_estimators': 50}, {'bootstrap': True, 'criterion': 'gini', 'max_depth': None, 'max_features': 3, 'min_samples_leaf': 86, 'min_samples_split': 43, 'n_estimators': 50}, {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 55, 'min_samples_split': 33, 'n_estimators': 10}, {'bootstrap': False, 'criterion': 'gini', 'max_depth': 3, 'max_features': 8, 'min_samples_leaf': 76, 'min_samples_split': 12, 'n_estimators': 10}]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "param_dist = {\"n_estimators\": [10, 20, 50, 100, 1000],\n",
    "              \"max_depth\": [3, None],\n",
    "              \"max_features\": sp_randint(1, 10),\n",
    "              \"min_samples_split\": sp_randint(2, 100),\n",
    "              \"min_samples_leaf\": sp_randint(1, 100),\n",
    "              \"bootstrap\": [True, False],\n",
    "              \"criterion\": [\"gini\", \"entropy\"]}\n",
    "\n",
    "def _savings(clf, X, y):\n",
    "    # Temporal function that receives X=[X_train, cost_mat_train], y= Y_train\n",
    "    # reconstruct cost mat\n",
    "    x = X[:, :-4]\n",
    "    cost_mat = X[:, -4:]\n",
    "    clf.fit(x, y)\n",
    "    c = clf.predict(x)    \n",
    "    return savings_score(y, c, cost_mat)\n",
    "\n",
    "f_ = RandomizedSearchCV(clf, param_distributions=param_dist, \n",
    "                        n_iter=5, n_jobs=4, scoring=_savings, verbose=1)\n",
    "f_.fit(np.hstack((X_train, cost_mat_train)), y_train)\n",
    "\n",
    "report(f_.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 f1       pre       rec       acc       sav\n",
      "RF         0.234356  0.509532  0.152174  0.931949  0.129533\n",
      "DT         0.248201  0.238436  0.258799  0.892699  0.184151\n",
      "LR         0.071775  0.569231  0.038302  0.932197  0.028994\n",
      "RF-BMR     0.279198  0.177352  0.655797  0.768253  0.434677\n",
      "DT-BMR     0.134468  0.079447  0.437371  0.614652  0.100415\n",
      "LR-BMR     0.207882  0.130423  0.511905  0.733005  0.265352\n",
      "CSDT       0.286539  0.174688  0.796584  0.728506  0.502330\n",
      "CSRP       0.348888  0.236397  0.665631  0.829962  0.492400\n",
      "RS-RF      0.246531  0.583012  0.156315  0.934606  0.130941\n",
      "RS-RF-Sav  0.132171  0.582996  0.074534  0.933012  0.066216\n"
     ]
    }
   ],
   "source": [
    "# Fit a RF with the best params\n",
    "classifiers[\"RS-RF-Sav\"] = {\"f\": RandomForestClassifier(**f_.best_params_)}\n",
    "# Fit\n",
    "classifiers[\"RS-RF-Sav\"][\"f\"].fit(X_train, y_train)\n",
    "# Predict\n",
    "classifiers[\"RS-RF-Sav\"][\"c\"] = classifiers[\"RS-RF-Sav\"][\"f\"].predict(X_test)\n",
    "# Evaluate\n",
    "results.loc[\"RS-RF-Sav\"] = 0\n",
    "results.loc[\"RS-RF-Sav\", measures.keys()] = \\\n",
    "[measures[measure](y_test, classifiers[\"RS-RF-Sav\"][\"c\"]) for measure in measures.keys()]\n",
    "results[\"sav\"].loc[\"RS-RF-Sav\"] = savings_score(y_test, classifiers[\"RS-RF-Sav\"][\"c\"], cost_mat_test)\n",
    "    \n",
    "print (results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAEvCAYAAACT2dMvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXhM1+MG8PfMmkwS2UlEiNS+JEgspfalfFUXrdIF3Shq+dq1pVWqlNLGVlu1tN+qtrrYWltbWmqJXRAl1khIiET2Wc7vj+JnCYKZOcnk/TyP5zF3ztz7TiuTd+49914hpQQRERERkSoa1QGIiIiIqGRjISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqV0qjYcEBAgw8LCVG2eiIiIqNB27tyZKqUMVJ3DVSkrpGFhYYiNjVW1eSIiIqJCE0KcVJ3BlfGQPREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkVKEKqRCivRAiXghxVAgxqoDnXxJCpAgh9lz585r9oxIRERGRK7rrnZqEEFoAswC0BXAGwA4hxHIp5cGbhi6VUvZ3QEYiIiIicmGF2UPaAMBRKWWClDIfwDcAnnBsLCIiIiIqKQpzL/sQAKeve3wGQMMCxj0thGgG4AiAwVLK0wWMISKiQggbtUp1BADAiUkdVUcgohLAXic1rQAQJqWMALAOwKKCBgkhegshYoUQsSkpKXbaNBEREREVZ4UppIkAQq97XO7KsmuklBeklHlXHi4AEFXQiqSU86SU0VLK6MDAwPvJS0REREQupjCFdAeAykKIikIIA4BuAJZfP0AIEXzdw8cBHLJfRCIiIiJyZXedQyqltAgh+gNYA0ALYKGUMk4IMQ5ArJRyOYCBQojHAVgAXATwkgMzExEREZELKcxJTZBSrgaw+qZl71z39zcBvGnfaERE5Kp40hYRXY93aiIiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqV0qgMQqRQ2apXqCDgxqaPqCEREREpxDykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESnFQkpERERESrGQEhEREZFSLKREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKVWoQiqEaC+EiBdCHBVCjLrDuKeFEFIIEW2/iERERETkyu5aSIUQWgCzAHQAUAPAc0KIGgWM8wIwCMA2e4ckIiIiItdVmD2kDQAclVImSCnzAXwD4IkCxo0H8CGAXDvmIyIiIiIXV5hCGgLg9HWPz1xZdo0Qoh6AUCnlKjtmIyIiIqIS4IFPahJCaABMAzC0EGN7CyFihRCxKSkpD7ppIiIiInIBhSmkiQBCr3tc7sqyq7wA1ALwhxDiBIBGAJYXdGKTlHKelDJaShkdGBh4/6mJiIiIyGUUppDuAFBZCFFRCGEA0A3A8qtPSinTpZQBUsowKWUYgK0AHpdSxjokMRERERG5lLsWUimlBUB/AGsAHALwrZQyTggxTgjxuKMDEhEREZFr0xVmkJRyNYDVNy175zZjWzx4LCIiIiIqKXinJiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIqISR1dqNzwemoSIRRFo9307rEpYpToSUYmmUx2AiIjImXSldsMt+AcIjRkSQFJWEsZuGQsA6BjeUWk2opKKe0iJiKhEMQaugdCYb1iWa81FzK4YRYmIiIWUiIhKDKFLg9BfKvC55KxkJ6choqt4yJ6IiFyfyIch4A8Y/DbddkiQR5ATAxHR9VhIiYjIhdmgK7UXxtK/QKPPgDk9Etbs8jCW+fWWw/atyrdSlJGIWEiJiMgladxOw63MCmhNp2DNKYesxBdgy6kAAJA2E4yBa6A1pKO0qTSMWiOWHF6C6n7V8USlJxQnJyp5WEiJiMilCF0GjIG/QO+zGzaLF3LOPgNLej1cf9qEJaMuLBl1cWLSv2fVZ5uzMej3QRi9eTSyLdl4rtpzitITlUwspERE5BqEGQa/P2EI+AOAFXmpLZB/oSVgM971pSa9CTNbz8TwjcPxwbYPkGXOwmu1X3N4ZCL6FwspEREVcxI6rwMwll4FjeESzBk1kXf+P5Bm/3tai1FrxNQWUzH6r9GI2RWDzPxMDKo3CEIIB+UmoqtYSImIqNjSGM/CWGYFdB7HYc0NQvbJXrBmP3Tf69Nr9JjYdCI89B747MBnyDRn4q2Gb0EjeJVEIkdiISUiomJHaDNhCFwLvc8OSKs7cpOegvlSfdjj8toaocGYRmPgqffE53GfI9ucjXFNxkGn4a9MIkfhTxcRERUjFuj9tsAYsAHQmGG+2Bh5qa0Bm8muWxFCYHDUYHgaPDFj9wxkW7IxudlkGLQGu26HiP7FQkpERMWAhNbzMNxKr4LGmApLZlXknesIW35ph21RCIHeEb3hoffApO2TMOC3Afik5Sdw17k7bJtEJRULKRERFWkawzkYy6yEzvMfWPMCkX3qZVizqjpt+y9UfwEmnQlj/x6LPuv6YGbrmfAyeDlt+0QlAQspEREVTZpsGAPXQ++7FbAZkJv8GMxpDwPQOj3KU5WfgrveHW9uehOvrnkVc9vOha+br9NzELkqFlIiIipirND7bocxYB2gzYH5UkPkp7SFtHooTdU+rD1MOhOG/DEEL//6Mua1m4fSJsdNGSAqSXgdCyIiKjK0pn9gqjgdbkE/w5oXjOzjA5GX/KTyMnpVs3LN8GmbT5GUlYSev/TEmctnVEcicgkspEREpJzQp8Kt3CKYKnwGoTEj53R35Jx6Dba8YNXRblE/qD4WtFuAjPwM9PylJxIuJaiORFTssZASEZE6mlwYS6+Gx0MfQ2c6hrzz7ZGVMBiWzJoAiu4dkmoH1sbn7T+HVVrx0q8v4dCFQ6ojERVrLKRERKSADXrvHfB46CMY/DfBkl4HWQnDkH+hBSD1qsMVShXfKljUYRHcdG54dc2r2H1+t+pIRMUWCykRETnVznM7Yao4E25ll0Hm+yPreH/kJnWBtJRSHe2eVShVAYvaL4Kfux9eX/c6tpzdojoSUbHEQkpERE5xNvMshm0chpd+fQlCm4WcxOeQfbIPbLnlVEd7IMGewfii/RcI9QpF/w39seHUBtWRiIodFlIiInKobHM2Zu6eicd/ehwbT29Ev8h+yDo2FJaMSBTleaL3IsA9AAsfXYjqftUx9I+hWHFshepIRMVKoQqpEKK9ECJeCHFUCDGqgOf7CCH2CyH2CCH+EkLUsH9UIiIqTqSUWJmwEp1+6oS5++aiVflWWPHUCvSt0xeQrndPeG+jN+a1m4eoMlF4+6+38W38t6ojERUbd70wvhBCC2AWgLYAzgDYIYRYLqU8eN2wr6WUc66MfxzANADtHZCXiIiKgf0p+zFpxyTsS9mHGv418FHzj1C3dF3VsRzOQ++B2W1mY+gfQzF+63hkmbPwcq2XVcciKvIKc6emBgCOSikTAEAI8Q2AJwBcK6RSyozrxnsAkPYMSURExcP57POI2RWD5ceWI8A9AOMaj8MTlZ6ARpScGWJGrREft/wYb/35FqbtnIZMcyb61+kPIVxjegKRIxSmkIYAOH3d4zMAGt48SAjxBoAhAAwAWtklHRERFQt51jwsjluM+fvnw2Kz4NVar6JXRC946IvGHZacTa/RY1LTSfDQe2DevnnIMmdhRP0RJaqYE90Lu93LXko5C8AsIcTzAEYD6HnzGCFEbwC9AaB8+fL22jQRESkipcS6k+swbec0JGYmonX51hgaPRShXqGqoymn1Wjx7sPvwqQ34cuDXyLLnIWxD4+FVqNVHY2oyClMIU0EcP0nS7kry27nGwCfFvSElHIegHkAEB0dzcP6RETF2OGLh/Hh9g8Rey4WlX0rY0G7BWgYfMsBtBJNCIHh0cPhpffC7L2zkW3OxqSmk6DXFo+L/xM5S2EK6Q4AlYUQFfFvEe0G4PnrBwghKksp/7nysCOAf0BERC7pQs4FzNwzE8uOLIO30RtjGo1B58qdodPY7aCbSxFCoG+dvjDpTfgo9iNkW7LxcYuP4aZzUx2NHtDOnTtL63S6BQBqgZfSvBMbgAMWi+W1qKio8wUNuOunh5TSIoToD2ANAC2AhVLKOCHEOACxUsrlAPoLIdoAMANIQwGH64mIqHgzW834+vDXmLN3DnItuXih+gvoE9kH3kZv1dGKhZ41e8JD74Fxf49D3/V9MaPVDHgaPFXHogeg0+kWBAUFVQ8MDEzTaDQ88nsbNptNpKSk1EhOTl4A4PGCxhTq66yUcjWA1Tcte+e6vw96kKBERFS06ErthjFwDSIWvYkgjyC0q9AOf5z5AyczTuKRkEcwvP5whHuHq45Z7DxT5RmYdCa8/dfb6LW2Fz5t8yl83HxUx6L7V4tl9O40Go0MDAxMT05OrnXbMc4MRERERZ+u1G64Bf8AjeESJCSSspKw6OAiZOVnYVbrWfi0zacsow/gP+H/wcctP8aRtCN4ec3LSMlOUR2J7p+GZbRwrvx3um3vZCElIqIbGAPXQGjMtyzXa/VoVq6ZgkSup0VoC8xuMxuJmYl46deXcDbzrOpIREqxkBIR0Q2E/lKBy5Ozkp2cxLU1DG6I+e3mIy0vDT1+6YET6SdUR6JiSKvVRlWrVq3G1T/x8fGG5ORkbcOGDauYTKa6PXr0uOE6m5988ol/lSpValSpUqVG5cqVa3711VdFYs4IT4kkUuTmOXqD6g1Cx/COqmNRiSah99l222eDPIKcmKVkiAyMxOePfo7e63qj5689Ma/tPFT1q6o6FjnI0vilfnP2zgm5kHPB4O/un98nsk9i16pdLz7IOo1Go+3w4cPX384dGRkZmnHjxp3du3ev+4EDB9yvLj927Jh+6tSpwXv27Dnk7+9vTU9P1yQlJT1QFzSbzdDrH/wyZtxDSqRAQXP0xm4Zi1UJq1RHo5JKkw23cl/CLfgn2PKCIG03/o5y07phUD2ev+oIVf2q4ov2X0Cv0ePlNS9jb8pe1ZHIAZbGL/WbvGNyhdScVIOERGpOqmHyjskVlsYv9bP3tkqVKmV79NFHM93c3GzXL09KStJ7eHjYvL29rQDg7e1tq1atWj4AHDhwwNi4ceMqVatWrVGjRo3qcXFxRpvNhtdff71c5cqVa1apUqXG/PnzfQFg5cqVXlFRUVVbtWpVqXLlyrUAYPbs2X61a9euXq1atRrPP/98BYvFck+ZuYeUSIGC5ujlWnMRsyuGe0nJ6bTuCXALWQqhy0TuuY4wX2wCXam9MAaugdaQzj34TlDRuyIWd1iM19a+hl5re2FGqxm8yUAxM2bzmNCjaUdNt3v+cNphD4vNIq5flm/N10zaPinsp39+CizoNZV8K2WPbzL+dEHPXZWXl6epVq1aDQAIDQ3NW7du3bHbjW3UqFF2QECAOTQ0tHaTJk0ud+7cOe35559PB4Dnn3++4rBhw5J79OhxKTs7W1itVrF48WKf/fv3ux86dCguKSlJ16BBg+rt2rXLBICDBw+adu/eHVetWrX8Xbt2uX3//fd+sbGxh41Go3zxxRfLz5kzx79///4X7pT9eiykRApwjh4VDVYYAn6DIeA3SLMfsk/0hS23HADAklEXloy6ODGJJdRZynqWxaL2i9B7XW/0W98PU1tMRYvQFqpjkZ3cXEbvtrywCjpkfzs6nQ6bNm36Z+PGjaa1a9eWGjVqVGhsbKzHmDFjks+dO2fo0aPHJQAwmUwSgPzzzz+9nn322Ys6nQ6hoaGWhg0bZv71118mb29vW0RERNbVvau//vqr14EDB0yRkZHVASA3N1dTunTpe9pFykJKpIA0+0AYbi2lnKNHziJ0aXAL+QY600mYL9VD7rknAJtRdawSL9AUiM8f/Rx91/fF4N8H44OmH6BDxQ6qY1Eh3G1PZstvW9ZOzUk13Lw8wD0gf8ljS+Idl+xGGo0GLVu2zG7ZsmV2hw4dMl577bWwMWPG3PPeEJPJdG06gJRSdOnS5cKsWbPudGv5O+e63xcS0f3LS2kLWcCV6x4Lf8z5YajE0Xnth0d4DLTGZOQkdkVu0rMso0WIj5sP5rebj8jSkRi5aSSWHVmmOhLZQZ/IPokGreGGOZ0GrcHWJ7LPfZe4e3XixAn9X3/9dW1aQWxsrCkkJCTf19fXFhQUlP/ll1/6AEBOTo64fPmyplmzZpe///57P4vFgrNnz+q2b9/u2bRp06yb19u+ffuMlStX+iYmJuoA4Ny5c9ojR47cUr7vhHtIiVSQOggB2CwmaHU5KGMqA4vNgh+P/ojnqz+PAPcA1QnJFYl8GMushMF3O6w55ZCT+Byk2V91KiqAp8ETn7b5FEP+GIKxf49FljkLPWr2UB2LHsDVs+ntfZb97YSEhNTOzMzUms1msWbNGp/Vq1cf8fLysg0bNqzcuXPn9EajUfr5+Znnz59/CgC++uqr47169aowfvz4snq9Xn733XfHunfvfmnLli2e1atXrymEkO+9996Z8uXLW/bt23fDtqKionJHjx6d2Lp16yo2mw16vV5Onz79VJUqVfILm1fIgnbTOEF0dLSMjY1Vsm2iq8JGqTir3QqP8I8hpRbZxwfhxKROAID4i/F4YfULqFe6Hua0nQON4AGMksze/zY1xiS4hSyB1ngeeanNkZ/SFoXZJ+GoOaRqfvZuVdTnyJqtZoz8cyTWnVyHfpH90CeyD4R4oCmHdJ+EEDullNHXL9u7d++JyMjIVFWZipu9e/cGREZGhhX0HH/jETmZznsXNMZU5Ke0w/U/glX9qmJUg1H4O+lvLNi/QF1AcjESet+/YQqbBaHNQfapV5Gf0gE8QFY86LV6TG42GU889ARm752Nj2I/gqodSUSOxE8kImcSFhgDN8CaEwpLZo1bnn668tPYkbwDs/bMQt3SdVE/qL6CkOQytFlwC/4eeq9DsGRWRe7ZLpBWT9Wp6B7pNDqMazIOngZPLD64GFnmLIxpNAZajVZ1NCK7YSElciK9zzZo9JeQffZpALcedhNC4J2H38HBCwcxatMofNvpW/i7c45fQXjI9860pmNwK7sUQpeF3OTHYE5rgoL+zVHxoBEajKw/Eh56D8zbNw9Z5ix80PQD6DUPfoccoqKAh+yJnEXkwxDwOyxZ4bBmV7rtMA+9Bz5q/hEu5V3CW3+9BZu03XYs0a2sMASuhXv5BYDNgOzj/WBOewQso8WfEAID6g7A4KjB+PXErxj8+2DkWfNUxyKyCxZSIicx+G2GRpeJvJRHcbdyUNWvKkY2GIktZ7fgs/2fOScgFXtCfxGmCnNhDPgN5vQoZB0fAFteiOpYZGev1HoFYxqNwaYzm/DG+jeQbc5WHYnogbGQEjmDJgcG/42wXK4GW06FQr2kS5Uu6BDWATP3zMTOczsdHJCKO53XPnhUnA6N8RxyErshL+kZQPLaoq7q2arPYsIjExB7Lha91vVCel666khED4RzSImcwOC/CUKbi7yUdoV+zdX5pHEX4jBi4wh89/h38HPzc2BKKpZEPoxlVsDguwPW7PLIOdsN0sx/JyVBp4c6waQ3YfjG4XhlzSuY23Yur2GsWNioVVH2XN+JSR0LtTdi5MiRQcuWLfPXaDRSo9Fg9uzZJ1u1anXLBezvpGvXrhVGjBhxLioqKvf+0j4Y7iElcjChzYTBbzPM6RGw5ZW9p9d6Gjw5n5RuS2M8C1PFGdD7xCIvtSWyT77OMlrCtC7fGjNbz8Tpy6fx8q8vIykzSXUkcrL169d7rFmzxmf//v0Hjxw5cvD3338/Eh4eXugL0l+1dOnSk6rKKMBCWuytSliFdt+3Q8SiCLT7vh1WJRSNM4/p/xn8fweEBXmpbe/r9dX9q2Nkg5HYnLgZCw8stHM6Kp4k9L6b/722qCYXOadeRX7KowB4GaCSqHHZxpjbdi5Sc1LR89eeOJVxSnUkcqLExES9n5+fxd3dXQJAcHCwJSwszDxs2LDgWrVqVa9cuXLN5557roLNZsPu3bvdateuXf3qa+Pj4w1VqlSpAQANGjSoumnTJhMAmEymugMGDAipWrVqjcjIyGqnT5/WAUBcXJwxMjKyWpUqVWoMHDiwrMlkqgsAJ0+e1EdHR1etVq1ajcqVK9f89ddf7/n6ciykxdiqhFUYu2UskrKSICGRlJWEsVvGspQWIUJ3CXrfrTCn14PMD7zv9XSp0gWPhj2KmbtnYte5XXZMSMWN0GbBvdwiuAWtgDWrMrKP//eOV22gkqFu6bpY+OhC5Fpy0XVlV7T6thV3VJQQTz75ZMbZs2cNYWFhtV588cXyq1at8gSA4cOHnz9w4MChf/75Jy4nJ0fzzTffeNetWzfXbDaLw4cPGwBg8eLFfk8++WTazevMycnRPPzww5nx8fEHH3744cwZM2YEAkD//v1D+/Xrd/7IkSMHy5UrZ746fuHChX6tW7dOP3z48MFDhw7FNWzY8J7PtGMhLUYu519G/MV4bDi1AYvjFmPc3+OQa71x73quNRcxu2IUJaSbGQI2AADyU9o80HqEEBj78FiU9SyL4ZuGIy33ls8PKgG0pqMwVfwEWo9/kJvcCTlnekJaPVTHoiKiun91vFzzZWSaM5GSk8IdFSWEt7e37cCBAwdnzpx5MjAw0NKzZ8+Hpk+f7v/LL794RUREVKtSpUqNLVu2eB04cMAdAJ588smLixcv9gOAH3/80bd79+4Xb16nXq+X3bp1SweAqKiorJMnTxoAYPfu3Z6vvPLKRQB47bXXLlwd36hRo6wlS5YEDBkypOz27dvdfX1973l+GU9qKkJyLDk4m3kWiZmJ//65nPj/f89MREZ+RqHWk5yV7OCkVBhCnwq9z06Y0xpBWnweeH2eBk9MbT4VL6x+AW//9TZmtp7J+92XGFYYAtfB4L8RtvwA5Jx++Z7nI1PJsCR+yS3Lru6o6BheNG/iQA9Op9Phscceu/zYY49djoiIyJk/f35AfHy8adu2bQcrVapkHjJkSNnc3FwNAHTv3j2tS5cu4d26dUsTQqB27dq3XMxWp9NJjUZzbd0Wi+WO1yrs0KFD5qZNm+KXLVvm/corr1Ts37//uf79+1+402tu2ea9DKYHY7aakZSVhDOZZ/6/eF5XOi/k3vj/zqg1oqxnWYR4hiAiMAIhniH//vEKQTnPcuiyoguSsm6dwB7kEeSst0R3YAxcB0gt8lNb2m2d1f2rY0T9EZiwbQK+iPsCr9R6xW7rpqJJ6C/CPWQJtO6nkZ9WH3nnOgHSoDoWFVG32yHBHRWua+/evUaNRnOtWO7evdu9UqVKefHx8aagoCBLenq6ZsWKFb6dOnVKA4CaNWvmaTQavPPOO2WfeuqpW/aO3kmdOnUyv/jiC99evXqlLVy48NoZlEeOHDGEh4fnDx06NDUvL0/s2rXLBICFVBWrzYrz2edxJvMMEjMTr5XOM5f/fXw++zwk5LXxOqFDkEcQQrxC0CK0BUI8Q64V0HJe5eDv5g8hbv+lZFC9QRi7ZewNh+0NGgMG1Rvk0PdJd6cxJkFXah/yLzSHtHrZdd1dq3bF9uTtmL5rOuqVroc6pevYdf1UdOhK7YVb0A8ABHLOPA/L5QjVkaiIC/IIKnBHBS8H5RyFvUyTPWVkZGgHDhxYPiMjQ6vVamVYWFjeokWLTvr4+FiqV69eMzAw0BIZGXnDJaA6d+58cfz48eU+/PDDxHvZ1owZM06/8MILFadMmRLcqlWrDE9PTysArFmzxmv69OlBOp1Omkwm6//+97/j9/o+hJTy7qMcIDo6WsbGxirZ9v2SUiI1J/WGw+hnM8/+W0AvJyI5KxkWabk2XkCgjEcZlPUoi3Je5W4snJ7lUNpUGlrNg50VuyphFWJ2xSA5KxkaoYGvmy/WPr0Wei3vb1wYjrofulu5RdCZjiPz6AjAZrrj2Pu5F/rl/Mt4dsWzMNvM+L7T9/Bxe/ApAcWNK9/LPtucjXqz+0Hvs1P5tUUd8f4A1/7/p9LVk11vPr+glKEUvuv0Hcp6cqrH/RJC7JRSRl+/bO/evSciIyNTVWVytsuXL2s8PDxsGo0G8+bN8126dKnfhg0bjhX29Xv37g2IjIwMK+g5l99Den1hC/IIwqB6g247j0ZKifS8dCRm3Tp/82r5vPm+wf5u/gjxDEHtgNpoX7H9tdJZzrMcgj2CHV4MO4Z3vPZ+/kr8C33X98UXcV+gV0Qvh26Xbk/jdgp6r0PIO//oXcvo/fIyeOGjFh+h++rueHvz25jRagbnk7qIQxcOYcSmEdB5n0Reaivkp7QGL+dEhXX198H1v/e6VOmCz+M+R6+1vfBF+y8QaLr/K35QybZ582bToEGDykspUapUKesXX3xxwl7rdulCevM3xatnG57NPItKPpWuFc3r53RmmW+8sUEpQymEeIbgIe+H0CykGUK8Qq7N5SzrWRbuOncVb61Aj4Q8grYV2mLuvrloX7E9Qr1CVUcqkYyl18Jm8UT+xcYO3U5N/5oYFj0ME7dPxOK4xXip1ksO3R45lpQSXx36Ch/v/Bi+Rl/knHoN1uyHVMeiYuj6HRVX1Q+qj97reqP3ut74/NHPS+RRFXpw7du3z4yPjz/oiHW7dCGN2RVT4GWRpu+efu2xu879WsGMLhN97aShq5Y1vIYAACAASURBVMu8DPad/+doI+v/ewH1idsmYlbrWXecg0r2pzUdhc7jKHKTH3PKfcSfq/YcdiTvwCe7PkGd0nU4n7SYupBzAWM2j8GfiX+iRWgLjGs8DnV3bVEdi1xIndJ1MKPVDPRb3w991/fF/Hbz4Wm452uX061sNptNaDQaNfMfixGbzSYA3PZyUC5dSO90VuGSjktQ1rMsfI2+LlXayniUwRt13sCU2CnYcGoD2lR4sOtf0r2QMJZeA5vZG+ZLDZ2yRSEE3mvyHg6tOIThm4bj+07fw9vo7ZRtk338ffZvvPXXW8jIy8BbDd9Ct6rdXOoziYqOhsENMbXFVAz+fTD6/9Yfn7b5tEgd5SumDqSkpNQIDAxMZym9PZvNJlJSUrwBHLjdGJcupLc72zDYIxi1AmopSOQcz1d/HsuPLcek7ZPQuGxjmPSOmcdIN9J6HoLW/TRykzoD0nknlZUylMLU5lPx4i8vYvRfozG91XQWmmLAbDNj5u6Z+PzA56joXRFz2sxBVb+qqmORi2sR2gIfNP0AIzeNxOA/BmNGyxk8CfYBWCyW15KTkxckJyfXAm82dCc2AAcsFstrtxvg0oW0oMsiuWndXP6ySDqNDqMbjUb3X7pj9p7ZGFZ/mOpIJYANxsC1sOUFwHwpyulbrxnw73zSSdsnYfHBxehZs6fTM1Dhnc44jZF/jsT+1P14psozGFF/BPdUkdN0qNgBOZYcvLvlXYz8cyQmN5sMncal64DDREVFnQfwuOocrsCl23zH8I4Y23gsgj2CISAQ7BGMsY3Hloi7VdQpXQdPV34aXx36CvEX41XHcXm6UvugdUtGXmobqDoj+vlqz6N1+db4ZOcn2JuyV0kGuruVCSvRZWUXnMg4ganNp+Ldh99lGSWn61y5M0bUH4F1J9fh3S3vwibv+U6PRHbl8l+JCjrbsKQYHDUYv536De9vfR+LOiziZYEcxgpj4DpYc4NgyVB34XIhBN5r/B66ruyKERtH4NtO33I+aRGSZc7CB9s+wPJjy1GvdD1MajoJwZ7BqmNRCda9RndkmjMxe89seOg98GaDNzndh5RhQ3Fh3kZvDI0eij0pe/DT0Z9Ux3FZep+d0BguIC+lHVT/SHkbvTGl2RSczzmP0ZtHQ9WNL+hGcRfi0HVlV6xMWIm+kX3x2aOfsYxSkdAnog9eqvkSlhxecsMVaIicjYXUxT3+0OOIKhOFaTunIS03TXUc1yPMMARsgDUnFNbM6qrTAABqB9bGkKgh+OP0H/jq0Feq45RoNmnDorhFeHH1i8i15OKzdp+hX51+nK9HRYYQAkOihqBLlS5YsH8BFuxfoDoSlVAspC5OCIHRDUcjKz8L03ZOUx3H5eh9tkGjT//3rkwoOoe6Xqz+IlqGtsS0ndOwP2W/6jglUmpOKvpt6IePYj9C83LNsezxZYgOir77C4mcTAiB0Y1Go2N4R8TsisHXh75WHYlKIBbSEqCSbyX0qNkDPx39CbvO7VIdx3WIPBgCfoclqxKs2ZVUp7mBEALjm4xHaffSGL5pONLz0lVHKlG2JG7BM8ufQWxyLMY0GoOPW3zM+bxUpGmEBuObjEfL0JaYuH0ifj76s+pIVMKwkJYQr0e8jrIeZTF+63iYbWbVcVyCwW8LNLos5J1vpzpKgbyN3pjSfArOZZ3DO5vf4XxSJzBbzZgaOxWvr38dvm6+WNJxCZ6t+ixPFKFiQa/RY0rzKWgU3AjvbHkH606uUx2JShAW0hLCpDdhVINROHrpKL46yHmFD0yTDYP/RpgvV4ctt7zqNLcVERiB/0b9F7+d/g1fH+ZhOEc6mXES3X/pji/ivkDXql2xpOMSVPatrDoW0T0xao2IaRmDiIAIjNg0An+e+VN1JCohWEhLkJblW6JFaAt8uvdTJGXeegcrKjyD/yZAk4f8lKK5d/R6PWr0QItyLfBR7Ec4kHrbu7bRPdKV2g2PhyYhYlEEmn7TFE/99BROXz6NT1p8gtGNRsNN56Y6ItF9MelNmNVmFir7VMbgPwZjR/IO1ZGoBGAhLWHebPAmAGDS9kmKkxRfQnsZBr/NsGREwJZX9C/dI4TA+4+8j0D3QAzbOAwZ+RmqIxV7ulK74Rb8AzSGS5CQuJR3CRZpQb86/dC6QmvV8YgeWClDKcxpOwchniEY8NsAfpklh2MhLWHKepZFn8g++O30b/jj9B+q4xRLhoDfAWFFXkpb1VEKzdvojcnNJuNc1jm8u/ldzid9QMbANRCaG+diS0gsilukKBGR/fm5+WFe23nwMfrg9XWv40jaEdWRyIUVqpAKIdoLIeKFEEeFEKMKeH6IEOKgEGKfEGKDEKKC/aOSvXSv0R2VfCph4raJyDZnq45TrAjdJeh9tsF8KRrSHKA6zj2pU7oOBtUbhPWn1nM+6YMQZgj9pQKfSs5KdnIYIscq41EGC9otgJvWDb3X9sbJjJOqI5GLumshFUJoAcwC0AFADQDPCSFq3DRsN4BoKWUEgO8BTLZ3ULIfvUaP0Y1G42zWWczbN091nGLFELgeAJCf2kpxkvvTo2YPNC/XHFNjpyIuNU51nGJH43YKporTcbuT5oM8gpwbiMgJynmVw/x282GTNvRa24vnIJBDFGYPaQMAR6WUCVLKfADfAHji+gFSyt+llFd3tW0FUM6+McneospE4YmHnsCiuEU4dumY6jjFgjCkQO+9C+a0RpAWH9Vx7otGaPB+k/fh7+6PYRuH4XL+ZdWRigdhgSHwV5jCPoXQ5CMvpTmkTX/DEDetGwbVG6QoIJFjhfuEY27bucjMz0Svdb2QmpOqOhK5mMIU0hAAp697fObKstt5FcAvDxKKnGNI9BCY9Ca8v/V9ziksBGPAOkDqkH+hpeooD8THzQdTmk1BUlYS3t3C+aR3ozGehSlsJowBf8CcHoWshMHIT+2A3KTOsOX7QEAg2CMYYxuPRcfwjqrjEjlMdf/qmN1mNs5nn0fvdb15ww2yK7ue1CSEeBFANIApt3m+txAiVggRm5KSYs9N033wc/PD4KjBiD0XixUJK1THKdI0xrPQe+9D/sUmkFZP1XEeWJ3SdTCw3kCsO7kO38R/ozpOEWWFIWADTBVnQuiykH36JeQlPQPY/r2ckyWjLrKOjcK+nvuw9pm1LKNUItQpXQcxLWNwIv0E+q7viyxzlupI5CIKU0gTAYRe97jclWU3EEK0AfA2gMellHkFrUhKOU9KGS2ljA4MDLyfvGRnnSt3RmRgJKbGTuW33TswBq6FtLoh/0Iz1VHs5qWaL6FpSFNM2TEFBy8cVB2nSNEYzsEU9imMgetgyaiNrIT/wppZTXUsoiLh4bIP46PmH+HghYMY8NsA5FpyVUciF1CYQroDQGUhREUhhAFANwDLrx8ghKgLYC7+LaPn7R+THEUjNBjTaAzS89IRsytGdZwiSeN+Ejqvw8i/0BywuauOYzcaocGERybA180XwzYOQ2Z+pupIRYANer+NMFWcAaFPQ86ZF5B79jnA6qE6GFGR0qp8K0x4ZAJik2Mx5I8hMFt5S2p6MHctpFJKC4D+ANYAOATgWyllnBBinBDi8SvDpgDwBPCdEGKPEGL5bVZHRVBVv6p4ofoL+P7I99ibsld1nCJGwhi4BjaLJ/IvNlEdxu583XwxpdkUnM08i7F/jy3R80mFPhWmCnPhVuYXWLKqIDvhv7Bcrq06FlGR1TG8I8Y8PAZ/Jv6JUX+OgtVmVR2JijFdYQZJKVcDWH3Tsneu+3sbO+ciJ+tXpx9+PfErxv89Ht889g10mkL903B5WtNR6DwSkJvcCZAG1XEcol6Zeuhftz9idsWgfpn66Fqtq+pITmaD3ncrjKV/AaQWOYldYcmoA+A213Yiomu6VOmCbHM2Por9CKa/TXiv8XvQCN5zh+4d/9UQAMBD74FRDUYhPi0eSw4vUR2niJAwll4Lm9kH5ksNVYdxqFdqvYImIU0wecdkHLpwSHUcpxG6NLiX/wxuQcthza6IrITBsGTUBcsoUeH1rNkTfSP74qejP2Hyjskl+kgL3T8WUrqmTfk2eCTkEczcPRPnss6pjqOczvMgtO6nkZfSGpCuvcdYIzT44JEP4GP0KSHzSSX03jvgEf4JtG6nkZvUGTmnX4a0eKsORlQs9Y3si+41uuN/h/6HmXtmqo5DxRALKV0jhMBbDd+CVVoxeUdJv9mWDYbAtbDlBcCSXk91GKfwc/PD5OaTcSbzDN77+z2X3cshdBlwD/0CbmWXwZobgqzj/4X5UgNwryjR/RNCYHj0cDxd+WnM2zcPCw8sVB2JihnX3u3jBGGjVqmOAAA4Mck+10AM9QpFr9q9MHPPTGxO3IwmIa53Ik9h6Ertg9btHHLOPAdAqzqO00SVicKAugP+nU8aVB/PVn1WdSQ7ktCV2gu3oJ8BYUFucieY0x4Gv5cT2YcQAmMajUG2ORsf7/wYJp0J3ap1Ux2Ligl+EtMtXq71MsJKhWHCtgkl9PpyVhgD18GaG1wiz7J+pdYraFK2CT7c/iHiL8arjmMXQpsJt5Cv4B7yDWx5gcg6PhDmtCbgRyCRfWk1WkxoOgEtyrXAhG0TsOIYb7pChcNPY7qFQWvA6EajcfryaXx24DPVcZxO7xMLjeEC8lIeRUn8Ebl6fVIfow+Gbhxa7O/EovM6AFP4x9B5HkbeuQ7IPtkHMp835iByFL1Gj49afISGQQ0xevNorD+5XnUkKgZK3m9bKpSGwQ3xn4r/wWf7P8OJ9BOq4ziPMMMQsAHW7PKwZlZVnUYZf3d/fNjsQ5y+fLr4zifVZMOt7DdwL/cVpNkH2ScGIP9ic/Bjj8jxjFojpreajtoBtTF803BsTtysOhIVcfxkptsaXn843LRueH/b+8WzkNwHve9WaPQZV/aOluyTXKKDovFGnTfwy/FfsOyfZarj3BOtRzw8wj+BrtQ+5KW0QfaJfrDlBamORVSimPQmzG4zG5V8KuG/v/8XO8/tVB2JijAWUrqtAPcADKw3ENuStuGX47+ojuN4mjwY/P+AJbMyrNkPqU5TJLxW+zU8HPwwJm2fVDzmk2pyYQxaBlP5zyGt7sg+8QbyU9ugJJ2YRlSUlDKUwpw2cxDkEYQ3NryBuNQ41ZGoiGIhpTvqUqULavrXxJTYKbicf1l1HIcy+P4FjS4LeSntVEcpMjRCg4lNJ6KUoRSGbRxWpOeTak1H4RH+CfQ+schLbYHsEwNgyw1RHYuoxPN398f8dvPhY/RBn/V9cDTtqOpIVASxkNIdaTVajHl4DC7mXsSM3TNUx3EcTTYM/ptgvlwDttxQ1WmKlKvzSU9dPoXxW8cXvekbIh/GMj/DVGEBYNMh+2Qf5Ke0d/mbGRAVJ0EeQZjfdj70Gj16reuFUxmnVEeiIoaFlO6qpn9NdK3aFUvjlyLugmsebjH4bwQ0+cjn3tEC1Q+qj76RfbEqYRV+PPqj6jjXaNxPwiM8Bga/v5F/sQmyjg+ELaeC6lhEVIDQUqGY324+LDYLeq3theSsZNWRqAhhIaVCGVB3APzc/DD+7/Gw2qyq49iV0F6GwW8LLBmRPPHlDnrV7oWGwQ3xwbYPcCTtiNIsedY8GEuvhqnCHAA2ZJ/shbxznQBpUJqLiO7sIZ+HMKftHGTkZ6DX2l64kHNBdSQqIlhIqVC8DF4YHj0ccRfi8N2R71THsStDwG+AsCIvpY3qKEWaVqPFpKaT4GXwwrCNw5BtzlaSI+5CHLqu6PrvFItL9ZF1/L88CY2oGKnpXxOzWs9CclYyXl/3OtLz0lVHoiKAhZQKrUPFDmgY3BAxu2KQmpOqOo5dCF0a9L7bYb4UDWkOUB2nyAtwD8CkppNwIv0E3t/q3MuBma1mzNozCy+segGXzZeRfepl5CV3BmxGp2UgIvuoV6YeYlrGICE9Af029CvSJ0ySc7CQUqEJITC64WjkWfMwZccU1XHswhi4HpAC+amtVUcpNhoGN0TfyL5YkbACPx39ySnbPJJ2BM+vfh5z9s7Bfyr+Bz88/gOsWSX3xgVErqBxSGNMaTYFcalxGPjbQORZ81RHIoVYSOmehHmH4dXar2L18dXYmrRVdZwHkpCeAJ33LpjTGkFavFXHKVZ6R/RGw6B/55M68hIuFpsFC/YvQNeVXXE++zw+afkJPmj6AbyN/P9F5ApaV2iN8U3GY0fyDgz9YyjMNrPqSKQICynds9dqv4ZQr1BM2DoB+dZ81XHu2+w9swGbHvkXWqiOUuxoNVpMajYJHnoPDN041CHzSY+nH0fPX3oiZlcMWoa2xI9P/IjW5bknm8jVdHqoE0Y3Go2NZzbirT/fcrkTZ6lwWEjpnhm1Rrzd8G2cyDiBzw98rjrOfTl04RDWnFiD/IuPQFo9VccplgLcAzCp2SQcTz+OCdsm2G29NmnDlwe/RJcVXXAi4wQmN5uMqc2nws/Nz27bIKKi5dmqz2JI1BD8euJXjNs6ruhd75gcjoWU7kuTkCZoV6Ed5u+fj9OXT6uOc89m7J6BUoZSyL/YTHWUYq1RcCO8Hvk6lh9bbpf5pGcun8Gra17F5B2T0Si4EX564id0qNgBQgg7pCWiouzlWi+jd0Rv/PDPD5i8YzJLaQnDQkr3bUT9EdAKLSZsm1CsPjh2n9+NPxP/xCu1XgFsbqrjFHt9IvqgflB9TNg64b7nk0op8W38t+i8vDMOXTyEcY3HYUarGQg0Bdo5LREVZf3r9MeL1V/EV4e+wuy9s1XHISdiIaX7VsajDPrX7Y/NiZux7uQ61XEKRUqJmF0xCHAPwHPVnlMdxyVoNVp82PRDmPSm+7o+aXJWMvqs74PxW8cjMjASPz7+I56q/BT3ihKVQEIIDK8/HE9Vegpz9s7BFwe+UB2JnISFlB7Ic9WeQzW/avhwx4fF4jpyf5/9GzvP7UTviN4w6U2q47iMQFMgJjadiIT0BEzcPrFQr5FSYvmx5ej8c2fsPr8bbzd8G3PbzkWwZ7CD0xJRUaYRGrz78Lt4NOxRTN05Fd/Gf6s6EjkBCyk9EJ1GhzGNxiAlOwWz9sxSHeeOpJSYvns6ynqUxTOVn1Edx+U0LtsYvSJ64aejP2H5seV3HJuak4qBvw/E23+9jcq+lfF9p+/RrVo3aAQ/kojo3yMvEx+ZiGblmuH9re9jZcJK1ZHIwfjpTw8sIjACz1R5Bl8f+hrxF+NVx7mt3079hrgLcehbpy/0Wr3qOC6pb2RfRJeJxvtb30fCpYQCx6w5sQZP/fwUtiRuwbDoYVj46EKUL1XeyUmJqKjTa/WY2nwq6gfVx+i/RmPDqQ2qI5EDsZCSXQyqNwjeRm+M3zoeNmlTHecWVpsVM3bPQFipMDwW/pjqOC5Lp9Hhw2Yfwl3njqEbhyLHknPtuUu5lzB843AM2zgM5TzL4btO36FnzZ7QarQKExNRUeamc8P0VtNR078mhm8cji1nt6iORA7CQkp24W30xtDoodibshc//POD6ji3WH18NY6lH0P/uv2h0+hUx3FppU2lMfGRiTh66SiaL22OiEURaPZNM7Rf1h7rT63HgLoD8OV/vkS4T7jqqERUDHjoPTC7zWxU9K6I//7+X+w+v1t1JHIAFlKym07hnRBdJhof7/wYF3Mvqo5zjdlmxuw9s1HdrzraVmirOk6JkJaXBp3QIceSAwmJtLw0ZFuy0S+yH3pH9OaXAiK6J95Gb8xtOxdlTGXQa00vtPy2JSIWRaDd9+2wKmGV6nhkByykZDdCCIxuNBrZ5mxMi52mOs41P/7zI85knkH/uv150oyTxOyKgUVablgmIfHdke8UJSKi4i7APQDdqnVDni0PqTmpkJBIykrC2C1jWUpdAH87k1095PMQetbsiZ+P/YzY5FjVcZBrycXcvXNRt3RdNA1pqjpOiZGclXxPy4mICmNR3KJbluVacxGzK0ZBGrInHjcju3s98nX8cvwXvL/1fXzX6TulZ7QvjV+K8znn8WGzD3mhdScK8ghCUlZSgcuJSoKwUer32J2Y1FF1BLvjl13XpayQ7k9Md+oPrCv+YBZV7jp3vNnwTQz4bQAWH1yMV2u/qiRHZn4mFuxfgMZlGyM6KFpJhpJqUL1BGLtlLHKtudeWuWndMKjeIIWpiKi445dd18VD9uQQLUJboFVoK8zdNxdnM88qyfDloS9xKe8SBtYdqGT7JVnH8I4Y23gsgj2CISAQ7BGMsY3HomM4vxgS0f0bVG8Q3LRuNyzjl13XwEP25DCjGozCEz8/gYnbJ2JGqxlO3fal3EtYFLcIbcq3Qc2Amk7dNv2rY3hHFlAisqurnykxu2KQnJWMII8gDKo3iJ81LoCFlBwm2DMYfSP7YtrOafj91O9oWb6l07a9MG4hss3ZeKPOG07bJhEROR6/7LomHrInh3qxxouo5FMJE7dPRLY52ynbPJ99HksOLcFj4Y+hkm8lp2yTiIiI7h8LKTmUXqPHmEZjkJSVhLn75jplm/P2zYPFZkHfOn2dsj0iIiJ6MCyk5HD1ytTDk5WexOK4xfgn7R+HbuvM5TNY9s8ydK7cGaFeoQ7dFhEREdkHCyk5xZCoIfAweOD9re9DSumw7Xy691NohRa9I3o7bBtERERkXyyk5BS+br4YXG8wdp3fhZ+P/eyQbSRcSsDKhJV4rtpzKONRxiHbICIiIvtjISWnearyU6gTWAfTYqfhUu4lu69/5p6ZcNe545Var9h93UREROQ4LKTkNBqhwehGo5GRn4FPdn1i13UfvHAQ606uQ48aPeDr5mvXdRMREZFjFaqQCiHaCyHihRBHhRCjCni+mRBilxDCIoR4xv4xyVVU9auKF6u/iGX/LMOe83vstt4Zu2fA2+iNHjV62G2dRERE5Bx3LaRCCC2AWQA6AKgB4DkhRI2bhp0C8BKAr+0dkFxPvzr9UMZUBuO3jofFZnng9e08txN/Jf6FV2u9Ck+Dpx0SEhERkTMVZg9pAwBHpZQJUsp8AN8AeOL6AVLKE1LKfQBsDshILsakN2FUg1E4knYE/zv0vwdal5QS03dNR6B7ILpV62anhERERORMhSmkIQBOX/f4zJVlRPetdfnWaBrSFLP3zEZyVvJ9r2fL2S3YdX4Xekf0hrvO3Y4JiYiIyFmcei97IURvAL0BQFsq0JmbpiJGCIE3G76Jp35+CpN3TMa0FtPueR1SSsTsikGIZwiervy0A1IWf2GjVqmOgBOTeM9pIiK6s8LsIU0EcP0tb8pdWXbPpJTzpJTRUsporcn7flZBLiTUKxS9I3pj3cl1+PPMn/f8+vWn1uPQxUPoV6cf9Fq9AxISERGRMxSmkO4AUFkIUVEIYQDQDcByx8aikuKlmi+hondFfLDtA+Racgv9OqvNipm7ZyLcOxwdK3IPHBERUXF210IqpbQA6A9gDYBDAL6VUsYJIcYJIR4HACFEfSHEGQBdAMwVQsQ5MjS5DoPWgNENR+NM5hnM3z+/0K9bdXwVEtIT0L9uf2g1WgcmJCIiIkcr1BxSKeVqAKtvWvbOdX/fgX8P5RPdswbBDfBY+GNYeGAhHgt/DBW9K95xvNlqxuw9s1HDvwbalG/jpJRERETkKLxTExUJQ6OHwl3njglbJ0BKecexP/zzAxIzEzGg7gAIIZyUkIiIiByFhZSKhAD3AAyqOwjbkrdh1fHbnxmeY8nB3H1zUa90PTQp28SJCYmIiMhRWEipyHimyjOo5V8LU3ZMQUZ+RoFjlh5eipScFAysN5B7R4mIiFwECykVGVqNFmMeHoNLeZcwfdf0W57PzM/EggML0CSkCaLKRClISERERI7AQkpFSg3/GuhWtRu+jf8WB1IP3PDc4oOLkZ6XjgF1ByhKR0RERI7AQkpFTv+6/eHv7o/xW8fDarMCANJy07D44GK0rdAWNf1rKk5IRERE9sRCSkWOl8ELI+uPxMELB7E0fikAYOGBhcix5KB/nf6K0xEREZG9OfVe9kSF9WjYo/jhnx8wLXYa5u+fj9ScVLjr3HHo4iGE+4SrjkdERER2xD2kVCQJIdA4pDHybHlIzUkF8O8ln8ZuGYtVCbe/LBQREREVPyykVGR9fejrW5blWnMRsytGQRoiIiJyFBZSKrKSs5LvaTkREREVTyykVGQFeQTd03IiIiIqnlhIqcgaVG8Q3LRuNyxz07phUL1BihIRERGRI/AseyqyOoZ3BADE7IpBclYygjyCMKjeoGvLiYhIjbBR6k8uPTGJvwtcCQspFWkdwzuygBIREbk4HrInIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKlWEiJiIiISCkWUiIiIiJSioWUiIiIiJRiISUiIiIipVhIiYiIiEgpFlIiIiIiUoqFlIiIiIiUYiElIiIiIqVYSImIiIhIKRZSIiIiIlKKhZSIiIiIlGIhJSIiIiKldKoDUNEWNmqV6ggAgBOTOqqOQERERA7CPaREREREpBQLKREREREpxUJKREREREqxkBIRERGRUiykRERERKQUCykRERERKcVCSkRERERKsZASERERkVIspERERESkFAspERERESlVqEIqhGgvhIgXQhwVQowq4HmjEGLplee3CSHC7B2UiIiIiFzTXQupEEILYBaADgBqAHhOCFHjpmGvAkiTUlYC8DGAD+0dlIiIiIhcU2H2kDYAcFRKmSClzAfwDYAnbhrzBIBFV/7+PYDWQghhv5hERERE5KoKU0hDAJy+7vGZK8sKHCOltABIB+Bvj4BERERE5NqElPLOA4R4BkB7KeVrVx53B9BQStn/ujEHrow5c+XxsStjUm9aV28Ava88rAog3l5vpBACAKTedVTxxfdXfLnyewP4/oo7vr/iy5XfG+D891dBShnoxO2VKLpCjEkEEHrd4/9r79yDrarqOP75oiI4VOIgqSMJkUqpiI/GqTQhaqgmG00KGJvEqZlooqZAxxpHxUdvJr4aSgAACNhJREFUnWYcdBjUBBrqXpuSgYbEFDGaYeShlwtXGIlHho8CoQwhE/z1x1onFptz7j33vPa5h99nZs/Ze6299vr99vrttX97rXXWOjOGFTtnp6TjgfcAr2cvZGZzgbmViVodktaa2aV55N0IXL++SyvrBq5fX8f167u0sm7Q+voda5TTZb8GOFvSCEn9gcnA4sw5i4Hr4/5EYLn11PTqOI7jOI7jOJTRQmpmByVNB5YBxwG/MLMuSXcCa81sMfAw8EtJfwH2EJxWx3Ecx3Ecx+mRcrrsMbOlwNJM2G3J/n+AL9ZWtJqTy1CBBuL69V1aWTdw/fo6rl/fpZV1g9bX75iixz81OY7jOI7jOE498aVDHcdxHMdxnFxpSYdU0iFJHZI2Sloi6eQYPlzSgRhX2PrnLW9vSHTrkrRe0kxJ/SRNSHTaF5d67ZC0IG+Ze4OkfUXCZkl6OerzgqQpeciWpVo7k7QiKadNcVq0QtwOSSsz53fEKdaQNFbSv2LYZkn31Em3iu1M0jxJ2xMZb8/o/lK6gIakRYXyz9zDFyQtkHRCjXWsytaSe7Re0nOSPprIbpLuTs4dIultSbN7m0+tkXSapDZJWyWtk7RU0jmS7ou2vEHSGkkj4vk7YtiGKOvdkgZIuiCxhT1JWT/ZKF1qrF+npGcknZVcK33GfyPppBz1KlXf9CulWyZ9yTpD0lRJu5LyrMl7oxlklnSLQj3WGc+7rBa6OXXAzFpuA/Yl+/OBW+L+cGBj3vLVULehwJPAHZlzVgCX5i1rtfolYbOAG+P+2cAbwAnNJGsldpaWE3AKsBfoH493AB3AsHj8wXi8MR6PBX4f9wcCm4GPNZOdAfOAiXF/ALANGJGk7QQuj8cnA88W8k3vIeHPlMuB65rJ1jL3aALwTCL7NuD5JP4bsfxm52nTgIBVwLQk7ELgVsIqe/1i2JnA4MQWh8T9QcCvgPmlyjrPrQb63QE8WKKMFwIzctStVH0zpZRumfRjKVFnAFMLttlKMgMfifZwYjweApyRt536VnxryRbSDKs4emWplsDM/kFYaGB62tLUypjZFmA/MDhvWTJUa2eDgDeBQ0nYo8CkuD8F+HWxhGZ2gODs1MXOa2RnA+Lvm0lYG4dn5PgC8LsS+R8CVtPg57iXtvZuwgdFgf3AJkmFORInEcqz2nyqZRzwtpnNSfJfTyiXV83snRi208z2ZhOb2T5gGnC1pFMaIG9vqUo/un+OVwIfqLG8lZLKeTrl6fZ/6l1nlCAPmU8HdpvZW/Eau83sFQBJt8WW2Y2S5iowStLqQuLY27GhF/k5VdDSDqmk44DxHDlv6sikif/+nESrGWa2jdCCNDRvWRqBpIuBLdFJagqqtLOFkjoJq5bdFZ2vAr8lOGoAVwFLSuQ/mNDK9qdKdeiJKuzsZ5I6CEsOt2XK7Sng4/H+TQbai11A0gDgMuDxXgteBWXY2sBCVyLwEHBXJr4NmCxpGOFD45UK86kl5wPrioQ/ClwV9blX0kWlLmBmbwDbCTbXbFSr36eBRdlAhQVfPgPk7pwUqW/KLrvkGsXqjElJnXVDi8j8BDBM0ouSHpB0ZRI328w+bGbnE1pfP2dmm4H+yfCBSZSol5za06oO6cD4EnwNeC/wxyRuq5mNids38xHPqYDvSuoidOv+IG9hIrWws+vMbDTwPuDGdPwaYbWzvZImA5sIrWgpV0haT1gpbZmZvVatQnXgJjMbA5wGjFccZxk5BPyZ4IwONLMdmbQj4/39O6E1pbMRAlO+rR2I5TuK4MgsyLQgPw58itLOdtPYtIVln88Fvg+8AzwlaXw3SfpUj0wZ+j0t6WWC05n2RBSe8bXAS4Q5t/OiaH3Ty7Lrrs5oT+qsR1pB5tiifwmhh2cX0C5paoweJ+nZ2AL6CeC8GJ72TLlD2kBa1SE9EF+CZxEqzpZ1PCW9n/Bib5oWwzrxczM7D7gWeDi2muVNr+xM0rL4Jf9QNs7MdgHPEVoCU9qB+yneXb/SzC4kVKRflTSmAh3Kohw7k/RI1G9pNi6+GFYAl2ei2oD7KN6dvTXe35HAJZI+X6H4veUoW5M0LGmJmZZNYGarCOPTTk3C/ktorZtJGC/XYz71UCZDF+EFfRRm9paZ/cHMbgJ+CFxd7DxJ7yKMk32xXkJWQaX6jSM8xx2EcaQFDiQOz7dimeZFyfqmmG6SrklstjB0pGF1Rh4yF3tOzeyQma0ws9uB6cC18Vl7gDDu+QLgQQ4PK2oHviTpnJDcttTwfjjd0KoOKQBmth/4NjAzdrm0FJJOBeYQuh6OiQllLawMtpbDS9XmTrl2ZmYT4ovta9k4hX/vXgRszUQ9BvyUsFJaqetuB34M3FyB+D1Srp2Z2Q1Rv88WucbxBGc7q99K4EeUGB8br7sb+B6hNaVhpLZmZn9LHJM52XMljSIMaXg9E3UvcLOZ7Sknn9pJX5LlwIk6ckaH0ZKulHRGPO4HjAb+mk0saRDhRb6opzF/OVGxfmZ2EPgO8BU15/hY4Oj6RtLFxXQzs8cSm12buUZd64y8ZM4+p5LOlZQOLRlDKPeC87k72vTE5BpbCR/ft+Ktow2lpR1SADN7nvBv3qaYKqgGFMatdRH++fwER37R93VOkrQz2WYUOedOYEasyJqCKuxsYezSWgfMM7Mjxr+Z2b/N7CdltMzMIYzHHN7L/EtRKzsrjCHtJIy/O+KPSxa4Jzqd3bGIYBtXVCBDKaq1tcI96iC8uK7PjAHGzLrMbH4ZsjTEpuMHxTXAJxWmReoifBCMBpYoTCvWCRwEZidJn45xqwld11+vp5yVUoV+hfSvEj6OmrpXLVPfDKUM3YpQ6zqjW3KSeRAwX2G6sk7gQ8AsM/snoVV0I+Fjf00mXTvwZUr8EdGpD75Sk+M4juM4jpMrTdPC5DiO4ziO4xybuEPqOI7jOI7j5Io7pI7jOI7jOE6uuEPqOI7jOI7j5Io7pI7jOI7jOE6uuEPqOI7jOI7j5Io7pI7jOI7jOE6uuEPqOI7jOI7j5Mr/ALH6ZwSqRhj3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results\n",
    "ind = np.arange(results.shape[0])\n",
    "figsize(10, 5)\n",
    "ax = plt.subplot(111)\n",
    "l = ax.plot(ind, results[\"f1\"], \"-o\", label='F1Score', color=colors[2])\n",
    "b = ax.bar(ind-0.3, results['sav'], 0.6, label='Savings', color=colors[0])\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\n",
    "ax.set_xlim([-0.5, ind[-1]+.5])\n",
    "ax.set_xticks(ind)\n",
    "ax.set_xticklabels(results.index)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "1. Anderson, R. (2007). The Credit Scoring Toolkit : Theory and Practice for Retail Credit Risk Management and Decision Automation. Oxford University Press.\n",
    "2. Hand, D. J., & Henley, W. E. (1997). Statistical Classification Methods in Consumer Credit Scoring: A Review. Journal of the Royal Statstical Society. Series A (Statistics in Society), 160(3), 523–541.\n",
    "3. Correa Bahnsen, A., & Gonzalez Montoya, A. (2011). Evolutionary Algorithms for Selecting the Architecture of a MLP Neural Network: A Credit Scoring Case. In IEEE 11th International Conference on Data Mining Workshops (pp. 725–732). Ieee. http://doi.org/10.1109/ICDMW.2011.80\n",
    "4. Beling, P., Covaliu, Z., & Oliver, R. M. (2005). Optimal scoring cutoff policies and efficient frontiers. Journal of the Operational Research Society, 56(9), 1016–1029. http://doi.org/10.1057/palgrave.jors.2602021\n",
    "5. Verbraken, T., Bravo, C., Weber, R., & Baesens, B. (2014). Development and application of consumer credit scoring models using profit-based classification measures. European Journal of Operational Research, 238(2), 505–513. http://doi.org/10.1016/j.ejor.2014.04.001\n",
    "6.  Alejo, R., & Garc, V. (2013). Making Accurate Credit Risk Predictions with Cost-Sensitive MLP Neural Networks. In J. Casillas, F. J. Martínez-López, R. Vicari, & F. De la Prieta (Eds.), Advances in Intelligent Systems and Computing (Vol. 220, pp. 1–8). Heidelberg: Springer International Publishing. http://doi.org/10.1007/978-3-319-00569-0\n",
    "7.  Oliver, R., & Thomas, L. (2009). Optimal score cutoffs and pricing in regulatory capital in retail credit portfolios (No. CRR-09-01). Southampton. Retrieved from http://eprints.soton.ac.uk/71321/\n",
    "8.  Correa Bahnsen, A., Aouada, D., & Ottersten, B. (2014). Example-Dependent Cost-Sensitive Logistic Regression for Credit Scoring. In 2014 13th International Conference on Machine Learning and Applications (pp. 263–269). Detroit, USA: IEEE. http://doi.org/10.1109/ICMLA.2014.48\n",
    "9.  Lawrence, D., & Solomon, A. (2012). Managing a Consumer Lending Business. Solomon Lawrence Partners. \n",
    "10. Nayak, G. N., & Turvey, C. G. (1997). Credit Risk Assessment and the Opportunity Costs of Loan Misclassification. Canadian Journal of Agricultural Economics, 45(3), 285–299. http://doi.org/10.1111/j.1744-7976.1997.tb00209.x\n",
    "11. ECB. (2014). European Central Bank. Retrieved from http://www.ecb.europa.eu/stats\n",
    "12. Jayanta K., G., Mohan, D., & Tapas, S. (2006). Bayesian Inference and Decision Theory. In An Introduction to Bayesian Analysis (Vol. 13, pp. 26–63). Springer New York. http://doi.org/10.1007/978-0-387-35433-0_2\n",
    "13. Correa Bahnsen, A., Aouada, D., & Ottersten, B. (2015). Example-Dependent Cost-Sensitive Decision Trees. Expert Systems with Applications, in press. http://doi.org/10.1016/j.eswa.2015.04.042\n",
    "14. Maimon, L. R. O. (2008). Data Mining with Decision Trees: Theory and Applications. World Scientific Publishing Company.\n",
    "15. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Stanford, CA: Springer.\n",
    "16. Rokach, L., & Maimon, O. (2010). Decision Trees. In L. Rokach & O. Maimon (Eds.), Data Mining and Knowledge Discovery Handbook (2nd ed., pp. 149–174). Springer US. http://doi.org/10.1007/978-0-387-09823-4_9\n",
    "17. \\citep{Rokach2009}\n",
    "18. \\citep{Zhou2012}\n",
    "19. \\citep{Louppe2012}\n",
    "20. Correa Bahnsen, A., Aouada, D., & Ottersten, B. (2015). Ensembles of Example-Dependent Cost-Sensitive Decision Trees. http://arxiv.org/abs/1505.04637"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}