{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains code and comments from Section 5.3 of the book [Ensemble Methods for Machine Learning](https://www.manning.com/books/ensemble-methods-for-machine-learning). Please see the book for additional details on this topic. This notebook and code are released under the [MIT license](https://github.com/gkunapuli/ensemble-methods-notebooks/blob/master/LICENSE)._\n",
    "\n",
    "## 5.3  LightGBM: A Framework for Gradient Boosting \n",
    "\n",
    "[LightGBM](https://lightgbm.readthedocs.io/en/latest/), or Light Gradient Boosted Machines, is an open source gradient boosting framework that was originally developed and released by Microsoft. \n",
    "At its core, LightGBM is essentially a histogram-based gradient boosting approach. However, it also has several modeling and algorithmic features that enable it to handle large-scale data. In particular, LightGBM offers the following advantages:\n",
    "\n",
    "* Algorithmic speedups such as gradient based one-sided sampling and exclusive feature bundling that result in faster training and lower memory usage; these are described in more detail in Section 5.3.1;\n",
    "* Support for a large number of loss functions for classification, regression and ranking as well as application-specific custom loss functions (Section 5.3.2);\n",
    "* Support for parallel and GPU learning, which enables it handle large-scale data sets (parallel/GPU-based machine learning is out-of-scope for this book).\n",
    "\n",
    "\n",
    "### 5.3.2 Gradient Boosting with LightGBM\n",
    "LightGBM is available for various platforms including Windows, Linux and MacOS, and can either be built from scratch or installed using tools such as ``pip``. See the documentation of LightGBM for [installation instructions](https://lightgbm.readthedocs.io/en/latest/). Its usage syntax is quite similar to ``scikit-learn``’s. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuing with the breast cancer data set from Section 5.2.3, we can learn a gradient boosting model using LightGBM as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "Xtrn, Xtst, ytrn, ytst = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a gradient boosting classifier using LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LGBMClassifier(max_depth=1, n_estimators=20)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LGBMClassifier</label><div class=\"sk-toggleable__content\"><pre>LGBMClassifier(max_depth=1, n_estimators=20)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LGBMClassifier(max_depth=1, n_estimators=20)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lightgbm import LGBMClassifier\n",
    "gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=20, max_depth=1)\n",
    "gbm.fit(Xtrn, ytrn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9473684210526315"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "ypred = gbm.predict(Xtst)\n",
    "accuracy_score(ytst, ypred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5.4 LigthGBM in Practice\n",
    "\n",
    "As with any ML algorithm, we need to ensure that we avoid overfitting during training. We look to set the learning rate (Section 5.4.1) or employ early stopping (Section 5.4.2) as a means to control overfitting. Specifically,  \n",
    "* by selecting an effective learning rate, we try to control the rate at which the model learns so that it doesn’t rapidly fit, and then overfit the training data. We can think of this a proactive modeling approach, where we try to identify a good training strategy so that it leads to a good model. \n",
    "* by enforcing early stopping, we try to stop training as soon as we observe that the model is starting to overfit. We can think of this as a reactive modeling approach, where we contemplate terminating training as soon as we think we have a good model.\n",
    "\n",
    "### 5.4.1 Learning Rate\n",
    "As with AdaBoost, we can aim to avoid overfitting by training with an appropriate [learning rate](https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning_rate), which adjusts the contribution of each estimator to the ensemble. ``LightGBM`` plays nicely with ``scikit-learn``, and we can combine the relevant functionalities from both packages to perform effective model learning.\n",
    "\n",
    "LightGBM provides its own functionality to perform cross validation (CV) with given parameter choices through a function called ``cv``. \n",
    "\n",
    "**Listing 5.5**: Cross Validation with LightGBM and scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Info] [cross_entropy:Init]: (objective) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 364.000000\n",
      "[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000855 seconds.\n",
      "You can set `force_col_wise=true` to remove the overhead.\n",
      "[LightGBM] [Info] Total Bins 4548\n",
      "[LightGBM] [Info] Number of data points in the train set: 364, number of used features: 30\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 91.000000\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (objective) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 364.000000\n",
      "[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000482 seconds.\n",
      "You can set `force_col_wise=true` to remove the overhead.\n",
      "[LightGBM] [Info] Total Bins 4548\n",
      "[LightGBM] [Info] Number of data points in the train set: 364, number of used features: 30\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 91.000000\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (objective) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 364.000000\n",
      "[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000889 seconds.\n",
      "You can set `force_col_wise=true` to remove the overhead.\n",
      "[LightGBM] [Info] Total Bins 4548\n",
      "[LightGBM] [Info] Number of data points in the train set: 364, number of used features: 30\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 91.000000\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (objective) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 364.000000\n",
      "[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000828 seconds.\n",
      "You can set `force_col_wise=true` to remove the overhead.\n",
      "[LightGBM] [Info] Total Bins 4548\n",
      "[LightGBM] [Info] Number of data points in the train set: 364, number of used features: 30\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 91.000000\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (objective) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 364.000000\n",
      "[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000395 seconds.\n",
      "You can set `force_col_wise=true` to remove the overhead.\n",
      "[LightGBM] [Info] Total Bins 4548\n",
      "[LightGBM] [Info] Number of data points in the train set: 364, number of used features: 30\n",
      "[LightGBM] [Info] [cross_entropy:Init]: (metric) labels passed interval [0, 1] check\n",
      "[LightGBM] [Info] [cross_entropy:Init]: sum-of-weights = 91.000000\n",
      "[LightGBM] [Info] [cross_entropy:BoostFromScore]: pavg = 0.626374 -> initscore = 0.516691\n",
      "[LightGBM] [Info] Start training from score 0.516691\n",
      "[LightGBM] [Info] [cross_entropy:BoostFromScore]: pavg = 0.629121 -> initscore = 0.528447\n",
      "[LightGBM] [Info] Start training from score 0.528447\n",
      "[LightGBM] [Info] [cross_entropy:BoostFromScore]: pavg = 0.629121 -> initscore = 0.528447\n",
      "[LightGBM] [Info] Start training from score 0.528447\n",
      "[LightGBM] [Info] [cross_entropy:BoostFromScore]: pavg = 0.629121 -> initscore = 0.528447\n",
      "[LightGBM] [Info] Start training from score 0.528447\n",
      "[LightGBM] [Info] [cross_entropy:BoostFromScore]: pavg = 0.629121 -> initscore = 0.528447\n",
      "[LightGBM] [Info] Start training from score 0.528447\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAAsTAAALEwEAmpwYAABCmUlEQVR4nO3deXhcZ3nw/+89u/bNsi3JWxw7i5OQhNeEsIelNGxJ+xbahB0KKf2VQikUKO1LWd+WbsBboIWyt+yUJdAQKGsIhBCH7JtjO7YlW7b2dfaZ+/fHc2Y0UrQcWTOakXR/rmsuzZw5c84zR6O59Wz3I6qKMcYYU2sC1S6AMcYYMx8LUMYYY2qSBShjjDE1yQKUMcaYmmQByhhjTE0KVbsAK7Fp0ybdtWtXtYthjDFmBW6//fYhVe2cu31NB6hdu3Zx4MCBahfDGGPMCojIsfm2WxOfMcaYmmQByhhjTE2yAGWMMaYmWYAyxhhTkyxAGWOMqUkWoIwxxtQkC1DGGGNqkgUoY4wxNckClDHGmJq0oQNUOpvHFmw0xpjatKZTHa3EVCpL78g044kMezY3sakxWu0iGWOMKbFhA9Tb/+tuvnt3PwAf+oNL+J1Le6pcImOMMaV8NfGJyMWVLshq62iIFO+PTKerWBJjjDHz8dsH9UMRuUtE3iIiXRUt0SppKwlQo3ELUMYYU2v8Bqgu4J3A44GHReQHIvJSEamvXNEqq70kQA1bDcoYY2qOrwClqllV/baqvgjoAb4KvBU4LSKfF5EnVbKQlVAaoEYtQBljTM1Z1jBzEWkEfge4BtgGfBl4GPiCiHy07KWroPZ664Myxpha5msUn4g8D3gZ8BzgF8AngW+patJ7/qPAceBPKlTOsmuzQRLGGFPT/A4z/zvg88CbVLV/7pOqOiIif1bOglVahw2SMMaYmuYrQKnqRT72+eTKi7N6WutLA1SGfF4JBKSKJTLGGFPK7zyoiIi8R0QeFpFp7+d7RSTm90QicqWIPCQih0Tk7fM8/0oRGRSRO73ba5bzRpYrEgrQFHXxOZdXJpKZSp7OGGPMMvlt4vs34BzgDcAxYCfwDtyIvlcv9WIRCQIfBX4L6ANuE5HrVfX+Obt+RVVf77NMK9beGGEylQVcP1RprcoYY0x1+R3FdzXwfFX9nqrer6rf87b9js/XXwYcUtUjqprGjf67etmlLbM2G8lnjDE1y2+AOgXMnZRbBzxqwMQCeoDeksd93ra5fk9E7haRr4vIdp/HPmPtNpLPGGNqlt8mvv8AbhSRf8EFl+24IeWfF5FnFHZS1R+voCzfAb6kqikR+SPgc8Az5u4kItcB1wHs2LFjBaebM1nXRvIZY0xN8Rug/sj7+Y4521/n3QAU2L3A60/gglrBNm9bkaoOlzz8JPD38x1IVT8BfAJg//79K1rMydIdGWNM7fI7zPysFZ7nNmCviJyFC0zXAC8u3UFEukrmWF0FPLDCcy7J0h0ZY0zt8r0elIiEgCfi+o76gFtUNevntaqaFZHXA98HgsCnVfU+EXkPcEBVrwfeICJXAVlgBHjlst7JGZid7siGmRtjTC3xm+roPFwfUR1usMN2ICkiL1BVXzUdVb0BuGHOtneW3P9L4C99lrssZqc7Sq3mqY0xxizB7yi+j+H6fbar6hNUdRtubtTHKlayVTBrFF/calDGGFNL/AaoS4B/VtXSQQkf8ravWdYHZYwxtctvgDoJPG3Otqd429csW3LDGGNql99BEu8ArheR7zKT6uh5wEsrVbDV0FwXIhgQcnllKpUllc0RDQWrXSxjjDH4r0F9F7gUuBdo8n7+L1X9dqUKthpEZFa6o1EbyWeMMTVjyRqUl+h1CmhV1fdVvkirq70hzNCUG8E3Mp1ma4vvBO3GGGMqaMkalKrmgINAR+WLs/os3ZExxtQmv31QXwC+KyIfxk3SLY7mW2H+vaqzdEfGGFOb/AaoP/Z+vmvO9sXy760Js/ugLEAZY0ytWK1cfDWrw5bcMMaYmuR3yfd5R+uJyDfKW5zV12YByhhjapLfYeZPX2D7FWUqR9XMTndkAcoYY2rFok18XrZxgEjJ/YLduEm7a5qlOzLGmNq0VB9UYZHBALMXHFRcVvN3VaBMq6rN0h0ZY0xNWjRAqeqrAETkl6r676tTpNXVbn1QxhhTk/yO4vt3EWkBzgUa5zy3buZBjUynUVVEpIolMsYYA/4XLHwl8FFcyqN4yVNrfh5ULBykPhIkns6RzSuTqSzNsXC1i2WMMRue34m67wdeqKrfq2RhqqWtPkI8nQDcQAkLUMYYU31+h5mHgB9UsiDV1NFo6Y6MMabW+A1QHwD+WkT87r+mWLojY4ypPX6b+N4EbAXeKiLDpU+o6o6yl2qV2Ug+Y4ypPX4D1JpeOXcpFqCMMab2+B1m/rNKF6SaLN2RMcbUHr/JYqMi8n4ROSIi4962Z4vI6ytbvNVR2gc1NJmqYkmMMcYU+B308EHgQuAlzCxWeB8z60StaaU1qAELUMYYUxP89kH9LrBHVadFJA+gqidEpKdyRVs9pQFqaMoClDHG1AK/Nag0c4KZiHQCw/PvvrZsKp0HNWV9UMYYUwv8BqivAZ8TkbMARKQL+Ajw5UoVbDV1tdQV7w9Pp8nndZG9jTHGrAa/AeodwCPAPUAr8DBwEnh3ZYq1uuoiQVrqXHqjXF6tmc8YY2qArwClqmlVfZOqNgJbgCbv8bppD+tuiRXvnxxPVrEkxhhjwH8NqkhVB1V13bWBdbfONPOdHIsvsqcxxpjVsC5z652JrSU1qOPDiSqWxBhjDFiAKiqtQR0bma5iSYwxxoAFqKKukhrUiVGrQRljTLX5TXX09NIh5iLyORH5jIhsrWzxVk/pUPNTEzZIwhhjqs1vDepjQM67/09AGMgDn6hEoaqhu3WmBjVo6Y6MMabq/AaoHlU9LiIh4LeB63B5+J7o90QicqWIPCQih0Tk7Yvs93sioiKy3++xy2FL80yAGk9kyObyq3l6Y4wxc/gNUBMisgV4GnC/qk5528N+XiwiQeCjwHOAfcC1IrJvnv2agDcCt/osV9nEwjOTdfMKgzZZ1xhjqspvgPoX4DbgC7hAA/Ak4EGfr78MOKSqR7zJvV8Grp5nv/filpevSifQluZo8b4NlDDGmOrym0niA8CzgCepaiH/3gngNT7P0wP0ljzu87YVichjge2q+t+LHUhErhORAyJyYHBw0Ofp/SkdKHFkyIaaG2NMNfkeZq6qB1X1MLhRfUCXqt5TjkKISAD4Z+DNPsrxCVXdr6r7Ozs7y3H6om1tJXOhLEAZY0xV+R1m/jMReZJ3/224Jrovisg7fJ7nBLC95PE2b1tBE25BxJ+KyFHgcuD61R4oURqg+sasic8YY6rJbw3qQuBX3v3XAk/HBZHX+Xz9bcBeETlLRCLANcD1hSdVdVxVN6nqLlXd5Z3rKlU94PP4ZVGaTaLfApQxxlSV3wAVAFREzgZEVe9X1V6gzc+LVTULvB74PvAA8FVVvU9E3iMiV51JwSuhtA/Kln43xpjq8rvk+824BQq7gG8CeMFqyO+JVPUG4IY52965wL5X+D1uOZWmOxqeXjcriRhjzJrktwb1SmAMuBt4l7ftPODDZS9RFW1tiSHe/clkllQmt+j+xhhjKsdXDUpVh3Gr6pZuW3Q4+FoUDgZorQ8zGs8A8MjwNOdtba5yqYwxZmPyO4ovLCLvFpEjIpL0fr7bG/CwrnQ2zUzWPXhqsoolMcaYjc1vE9/f4ybqvg642Pv5DFzWh3Vla0lOPpusa4wx1eN3kMSLgIu9pj6Ah0TkN8BdwJsqUrIqKR1q3jtiS78bY0y1+K1ByTK3r1k72uuL9/vHbV0oY4ypFr8B6mvAd0Tkt0XkfBG5EvgW8NWKlaxKtpcEqMHJFKpaxdIYY8zG5TdAvRX4IS6T+e247OY/Ad5WoXJVTWkT33giw3TahpobY0w1LNkH5a3l9O/AdQtNrF1PSifrTiazjEylaYz67aozxhhTLkvWoFQ1Bzwbt8T7ure5KYp4PWuJTI5TE5aTzxhjqsFvE98HgXU572muUDBAa93MQsGHB6YW2dsYY0yl+G27+lNgK/DnIjIIFEcOqOqOShSsmjqbosVsEjYXyhhjqsNvgHppRUtRY3a013PwtKs5PTI0jaoisu5G1BtjTE3zm4vvZ5UuSC3Z09nIDx8YAOD0RIpEJkd9xAZKGGPMavKbi+8bIvKUOdueIiJfr0yxqmvv1qbi/dHpNMNTtvSGMcasNr+DJJ4G/HLOtltwK+uuOxd0z2QwH4mnGbG1oYwxZtX5DVBJoGHOtkYgU97i1IY9nY0EA67PKZ7OWU4+Y4ypAr8B6vvAx0WkGcD7+RHgxkoVrJpCwQAdDTMj6h+0ZTeMMWbV+Q1QbwaagRERGQBGgBbgzypUrqrb2lK67IbNhTLGmNXmdxTfKPA8EdkKbAd6VfVURUtWZT2tddzdNw7AqfEkiXSOukiwyqUyxpiNw28NCgBVPaWqt6334ASws2Mmq/nIdJr+cUt5ZIwxq2lZAWoj2d3ZWLw/Gs/QO2oByhhjVpMFqAXs3dxYXI1xPJHhyKD1QxljzGqyALWAtvoIzSVJYw8NTJHM2NpQxhizWhYcJCEiu/0cQFWPlK84taO1Pkx7Q4TxhJvqNTyVpm80zp7NTUu80hhjTDksNorvEC5ruVCSvXyex+tyaFtzLExHQ4RHvGzmI/E0vaMJC1DGGLNKFmziU9WAqgZVNQC8BvgycB4Q835+EfjDVSllFQQCwva2meXfR6fT9FlGCWOMWTV+U3S/F9irqoWhbA+LyB8BB4HPVqJgtWD35pmRfCPTaYam0sTTWctsbowxq8DvIIkAsGvOtp2s0+a9gvNLspqPxTPk80qfDTc3xphV4bcq8EHgxyLyGaAXl03ild72daunrZ6GSJDpdI6cKuPJDH2jcc7ZYv1QxhhTaX5THf2DiNwDvAi4FOgHXq2q6zJZbEF7Q4S2hgjTaVdrGp1O0ztiNShjjFkNvjtTvGC0rgPSXIWh5oVmvZFptzbUVCpLY9T6oYwxppIWmwf1Hj8HUNV3lq84tSUaCtLdEismjR3yVtbtG41z3tbmxV5qjDFmhRarBmz38Xpdepe17dytzdx432kATnoJY48PW4AyxphKWzBAqeqrVrMgterCnmZCASGbVyaTWSaSGY4OT6OqiMjSBzDGGHNGfHekiMhe4FqgBzgBfElVH65UwWpFZ1OMrtZYcXDEybEEzbEwg5MpNjfHlni1McaYM+VrHpSIvAC4HZdBYgQ4FzggIlf5PZGIXCkiD4nIIRF5+zzPv05E7hGRO0XkZhHZ5/fYldRWH6anZSajxAlvwEQhBZIxxpjK8DtR9/8CV6vqi1X1L1X1JcDV3vYliUgQ+CjwHGAfcO08AeiLqnqRql4C/D3wzz7LVlFtDRF6SlIenRxLAnB02AKUMcZUkt8AtQ34+ZxtN3vb/bgMOKSqR1Q1jcvrd3XpDqo6UfKwgRoZgNEUDbG9vZ6A1900Enfpjvq9ZeCNMcZUht8AdSfw5jnb/tzb7kcPLgNFQZ+3bRYR+RMROYyrQb1hvgOJyHUickBEDgwODvo8/ZkTETqbomwp6W86OZZE1WpRxhhTSX4D1P8HvEZETorIrSJyErgO+ONyFkZVP6qqZwNvA/56gX0+oar7VXV/Z2dnOU+/oPb6CN2tJf1QY64f6qj1QxljTMUsNlH3YlW9C0BVHxCR84HLgW7gJHCrqmZ8nucEs+dVbfO2LeTLwL/6PHbFtTVE6Gmt4/Zjo4AbyQdwdDhOPq8EAjbc3Bhjym2xYeY/B5oBRORhVd2L63c6E7cBe0XkLFxgugZ4cekOIrK3ZNj684CaGcLe3hChu2WmiW9wMkUq6/qfTk0kZ9WujDHGlMdiAWpMRJ4P3A90ecHlUVUFP0u+q2pWRF4PfB+3RMenVfU+L53SAVW9Hni9iDwLyACjwCuW/3Yqo60+QjQcZFNjhKGpNAqcGk+ys6OBR4amLUAZY0wFLBag3gh8CLfuUwA4PM8+is81oVT1BuCGOdveWXL/jX6OUw2t9WFEoKe1rpiP78RYohignrRnU5VLaIwx689iS75/U1X3qGoYiHtLwM+9resFCwvCwQAtdWF65hkoMTiZom/UloI3xphy8zuKr6NwR0T8zn1aV7pa6mY15Z2eSJHN5QH4xaEhVGti2pYxxqwbvgKUN7m24P4KlaWmdbfGaIiGaKsPA5DLKyfHXVaJk2NJS31kjDFl5rcGteF1efn4trfXF7f1jsw07f3i8LDVoowxpozOJEBtyEk/HQ0RIqEA29tKAlRJ39PQZIoHT01Wo2jGGLMuLTtAqWpTJQpS6wIBoaslxra2umKEHphIkczM5OO75fAwubzVoowxphyWFaBEpFlEnicizxeRtkoVqlZ1tdQRCwfZ3BwF3Bj7Pm/5DYDxRIb7T04s8GpjjDHLsWiAEpEbS+4/BngI+EdcMtcHReTSyhavtnS3umwSCzXzAdx+bMT6oowxpgyWqkE9seT+3wMfVtXzVXUfM4Fqw9jaEkNk4YESAKPxDEdsRJ8xxqzYcpr4LsVllij4f8Al5SxMrYuGgmxqjNLdEiPoJYgdjWeYSmZn7fcbL6msMcaYM7dUgAqJyNNF5BlAntmLCOaB2PwvW7+6W2OEgoFZyWPnNvP1jSY4PZFc7aIZY8y6slSAGgA+DXwKSAKPLXnuccCSiWLXm/nmQx0feXSqI6tFGWPMyiyWLBZV3bXI073AtWUtzRrQXRqgDg8DrgalqojMTBE7eHqKJ+3N0BwLV6Wcxhiz1p1xJglV7VXVDZf2qLkuREM0yOamKNGQu3zTqRyj8dlrN+ZVufP4WBVKaIwx64OlOlomEaGrpY6ACNvaZpLHHh6cetS+dxwf4/iwZTo3xpgzYQHqDBSymp/d2VjcdnffOPk5WSTyqnzn7pMMTqZWtXzGGLMeLBigROTi1SzIWlJYF2rv5kbqwm5JrKlUdt75T+lsnm/feYLJZOZRzxljjFnYYjWonxfuiMjDq1CWNWNzU5RIKEAoGOCinpbi9jt7x+bdfzKZ5dt3niSdza9SCY0xZu1bLECNeTn3dgNdInKWiOyee1utgtaSQuJYgIt6WvDm7HJiLLFgc97gZIo7jtvQc2OM8WuxAPVGXOaIh4A64DBwaM5tw9asCs18jbEQe0r6ou7qG1vwNbcfH52V/dwYY8zCFgxQqvpNVd2jqmEgrqqBeW7BVSxrTSld/v3i7a3F+w+emiSxQBBKZfL8xmpRxhjji99RfB0AIhIQkS4R2fCj/7aW5OPraomxucktwZHLK/edGF/wdXccH7NalDHG+OA30ERF5PO4dEcngISIfE5EWpZ43boVDgbY2uz6oURkVi3qjt4xMrn5B0Sks3lutzRIxhizJL8B6l+ABuBCXH/URUA9LqP5htVTMlH3nM2NNEZd5qh4OsfdfQvXou7sHSOezi74vDHGGP8B6krgZap6UFVTqnoQeJW3fcPqKemHCgUDPG7XzCLDB46OkMrO35SXzua56eDQoyb2GmOMmeE3QCWBzjnbNgEbOkVCV6tbwLDggu4WmmOuFpXM5rljkVx8D/RP8K07T1h/lDHGLMBvgPok8D8i8joReY6IvA74PvCJyhWt9kVDQTq9wREAwYBw+e6O4uM7jo8tOKIP4NhwnC/9+jhDUxs6zhtjzLz8Bqj3A38HvBD4J+/n33vbN7TSZj6Ac7c20V4fASCdW3pAxFg8w1du6+XQwGTFymiMMWuRrwClzqdV9Vmqus/7+SlV3fCdKKUZzQECIly+u734+K7eMSYSi+fhS2fzfOeufn55aAi7pMYY42z4+Uwr1d1aN6sfCmDP5sbivKhsXvnRgwO+As+tj4xw/V0nrV/KGGOwALVi9ZEQO0qWfwc3L+qKc2fGlBwfiXNf/4Sv4x0ZnOb6u06SsxF+xpgNzgJUGZy3tflR27pa6rh0R2vx8c8PDvlecuPEaIIf+6x1GWPMemUBqgz2bG4kHJRHbX/C7g5a68KAGzDht6kP4N4T49yxwPIdxhizEYQWekJE3uPnAKr6zvIVZ22KhAKc3dnIg6dmj8QLBwM8a98Wvn57H+CGlT/QP8m+7kfXuOZz08FB2usj7NrUUPYyG2NMrVusBrW95LYXeDvwTGAP8Azv8d5KF3CtOL9r/qDT01rHJSV5+m56eJDplL80R6rw3btPcsfxUWvuM8ZsOIstt/Gqwg0Q4FpVfZKqvlhVnwxcs2qlXAN2tNdTH5l/9ZEnnt1RzDCRyub52cFB38fN5JSfPjTIVw/0MmwTeo0xG4jfPqjnAN+as+164Ll+TyQiV4rIQyJySETePs/zfy4i94vI3SLyIxHZ6ffYtSAQEM7d2jTvc+FggGeev6X4+OGBKQ4PTi3r+CfHknzh1uMcODpitSljzIbgN0AdAv5kzrY/xq2yuyQRCQIfxQW6fcC1IrJvzm53APtV9THA13GZKtaUhZr5wNWw9pU8/5OHBhZMJruQXF75+cND/NdvTjDhc0SgMcasVX4D1GuAPxeRPhG5VUT6gDcDr/X5+suAQ6p6RFXTwJeBq0t3UNWfqGrce/grYJvPY9eMzU1R2hsiCz7/lL2bis2A06kcNx08s8wRvSNx/vNXxzg0sLxamDHGrCV+Ux3dgRsQcS3wz8CLgb2qervP8/QAvSWP+7xtC/lD4HvzPSEi14nIARE5MDjovy9nNYgIjy9JczRXLBzkinNmJvDe3z/BbUfPbPHCVCbPDff00zcaX3pnY4xZg3wFKBH5tqpmVPXnqvoVVb1JVTMi8o1yF0hEXgrsB/5hvudV9ROqul9V93d2zl0BpPrO3dL0qMwSpfZsbuScLY3Fx7ccGeaO42cWpHJ55bt39zO+RK4/Y4xZi/w28T19ge1X+Hz9Cdxw9YJt3rZZRORZwF8BV6nqmhyyJiI847zNhAKPnrhbeP639m2ZFcRueniIe08uvALvYhLpHNffdZJ0dv4l5o0xZq1acKIuzJqsG5ln4u5u4JjP89wG7BWRs3CB6RpcM2HpuS4FPg5cqaoDPo9bk9oaIuzf1c6vjgzP+3woEOD5j+niW3ec4OR4EoAfPTBAXTjI2Z2N875mMUOTKW687xS/df4W6hYY6m6MMWvNogGKmVpPgNk1IMX1Kb3Lz0lUNSsir8ctchgEPq2q93lB74CqXo9r0msEviYuPfhxVb3K7xupNY/b1caDpyYYi8/f/BYOBrjqkm6+8ZsTDEy6yuKN957iRfu3sbkptuzzHR6Y4vDAFE2xEJubY2xtjrGro57OpigyN926McasAeJnFJmIvFZV/30VyrMs+/fv1wMHDlS7GAs6OjTNN+94VEvmLPF0lq8e6Cv2IzVGQ/zB47bTGF3qfwd/GqJBdnY08NgdbbNW/zXGmFohIrer6v652/32Qf1CRLZ4B2oUkXeLyN+IyMKjAQy7NjUsOmAC3HIdV13cTSTkfhVTqSzfueskmVx5+pSmUznuPznBF249xo33nmJ8gRqdMcbUGr8B6ktAq3f/H4GnApfj+ozMIp68d9OS+7Q3RHjuhVuLCx8OTKb4/n2nyJcxY4QqPNA/weduOcpNBwdtUIUxpub5DVC7VPUhcZ0Z/xt4EfBC4LcrVrJ1YktzjHO2zJ8CqdTOjoZZc6QOD05z08HBsqc1yuWV24+N8vlbjnJoYMrSJhljapbfjo6kiDTh0hQdV9UhEQkBy+/N34CeeHYHhwamlqwRPWZbK2PxTHEdqLv6xmmMhdi/c+HJv2dqMumaEnva6oiFg+TzSi6vlJawORbiiXs2la0/zBhjlsPvN88XgR8DTcBHvG2PBR6pRKHWm7aGCBf2NHN339JznZ6ydxNTqSwPe2mMfnFomIZIaNE8fytxYjSx6PMPD0xx+e4OLtneSnCBuV3GGFMJvgKUqr5JRJ4NZFT1J97mPPCmipVsnXn87g4e6J8gk1u8FiUiPPuCLSTSOfrGXPD4wf2n6R9P8qQ9HURDqzvPKZ3Nc9PBQe7vn+AZ522mp7VuVc9vjNm4fA0zr1W1Psx8rkMDk3z37n78XPJUJsfXbu9jeDpd3NYYDfH08zrZvWn5k3nL5cKeFp68Z5NNCDbGlM1Kh5mbMtizuYmnnuMvf2A0HOR3L+3hrJLl3t0Q9H5++MDpsg1DX657T4zzuVuOcsvhYU6OJcjlF462+bwyMJkkW6WyGmPWNqtBVcFPHxrgjuNjvvZVVQ6enuJnBwdJZGbWj2qrD3PlhVvPKOtEOUVCAba11dHdWkdXS4wtzTHi6Rz3nRjnvpMTTKWyBAPCluYo3a11bGmO0dkYpbU+bBkujDHAwjUoC1BVkM8r/31P/7LWc0qkc/z0oQEOlrwmKMITzu7g0u2tBGpkAENABEWXbMaMhAJ0NERorQ/TUhehKRZiIpFhYDLF4GSKQEDoaY3R1eICX1tDhHDQf4W/UMNczmuMMdVRlgAlIk2qOlny+GJVvatMZVy2tRqgAFLZHF/+dS8jJX1MS1FVHuif5KcHB2YNttjcFOWZ52+uem2qkkSgORamozFCW70LbG31ERqjIYJBKWaP7x1JcGhgiqPD0wDs3tTAOVub2NleT8iClTE1qVwBKgXcCXwAiANfUtW2chVyudZygAIYmkrx5V8fX3Jk31yj8TQ33nuqmGQW3Bf4Y3e0cfnudkIB+yKeKxIKsKujgd2dDZy1qYGpVJZHhqY5MjjFdCrH+V3N7OtupqUuXO2iGrPhlCtARXApjj6JW27jhar6rXIVcrnWeoACl37oxntPLft1ubxy+/FRfv3IyKyBClubYzzvoi4aYza5drlEoKe1jrb6CA3REI3RENva6mhriKx6WVTV+ujMhrFQgFpqPai3AL9Q1VsAVDUtIrtxefl+gsss8a2yl3YDOb+rmZNjCV+TeEsFA8Jlu9rZu7mRHz8wUJwzdWoiyRd/fZznXdRFT5vNWVoOVegbTdA3Z/LyjvZ6Lt7ewlmbGplOZxmZSjMaT9NcF2Z7W30x0e/McZShqTT94wlOjiVJZLLUhYPEvFs4KAQDAUIBQYRif10qm2dwMsXgVIqx6TQ9bXWct7WZszc3rPr8N2NqwaI1KBHpAy5Q1XHv8auAdwNXAlPAT1V192oUdD7roQYFkM3l+cKtx5fVH1VKVbnj+Bg3Hx4qftmJwEXdLTxmWwsdjbbMRjmUBpOCYEDcII76CFOpLBPJDBOJzLKbbRcTDgpdLXV0NEbY1BhlS3OMTY2RitWwcnm1rCGrQFXJ5JSAsOH7R8+oiU9ExoAtqprygtNfAs9W1aMiEgDGVLUyOXh8WC8BCuDEWIKv3ta7omP0jca54Z5Ts4ajA2xrreOC7mZ2bWogFrb/xNeDpliIszc3cvamRjY3R5f9e83nlWxeyebzZHLK4GSKY8PTHBuOk8jk2LO5kX1dzWxrqysGwmzO7ZtXd8vmlNF4muHpNCPTaSLBQDGIttVHiIUDxdeqKhOJLINTSUSEbW11S9YK83n1NTp1sebQ+Y6RyuYYmHAjRbtbYhUJ9Lm8ks7mEXEjW7P5PP3jSfrHkpwcTzCRyJBI58jmlUgowM6Oes7ubCwuz5PO5t1IVIFoMEgkFCAaClRstG4hDiznWqSyOcYTmbIMzjrTAHU9EAUGgGuBK1T1Zu+5q4B3qepjV1y6M7SeAhTAjx44veymvrkmkxm+d+8p+r2l5EsFvD6WszsbOXdrkwWrdSQWDnpD9sM0RkM0xkLFJL85LxHwdCrL0FSaoakUo/G0r4wm9ZEgwYCQzOSWXSsMiFAfCRILB5hIZmct8RIQV/Pc1lZHc12YJq+8E8ksx4anOT4SZyyeoaMxwpamGJ1NUeq8soQDAZLZHCfGEvSPJRmaSlEfCdJaH6G1Lkw2n2csnmE0niGZyVEfCdIUC9MQDTKeyDAyPfPem+vCnL+1ib1bmuhoiCwZALK5PNk5k9NVIa9KJpenbzTBI0Ou/OVe0qZQk97eXs+2tjraGyIL/g3n88rQVIoTYwkmk9lZ/1Qks3mSmRypTI5UNk86lyeTde+pLhKgLhykIRqiq6WOHR31bG2OIcBIPM3ARIrTE0lOjCUYmkrR3VrH7+/fPm8ZluNMA1Qb8BdABjiAW//p20AY+APgxar6nRWX7gyttwCVzOT4j1uOMZXKrug4qkrfqOvXOjw0Ne8XUSggnLu1iYu3tdpKu8bgAsCmxiibm6O0N0TpaIjQ1hAhFBAeGZrm0MAUx4any9p8u1J1kSCtdWHqIsFioMzmlMGpVNkCZKGPdb7j9bRVMUDNc5A9wKtxgyu+oaq/WnHJVmC9BShw+fq+c1d/2Y43mczwwKlJjgxOcXoiNe8+m5uinLu1iXO3NNFgS2sYY3yqdIBa1reRqh4C3rHi0pgF7dncxHlbp3jw1OTSO/vQFAtz2a52LtvVzlQyy+GhKe47McHg1EywGphMMTCZ4uaHh+hprWPXpgZ2dtTT0VC5jnhjjFmK/btcg559wVZS2TyPDE2X9biNsRAXb2vlMT0t9I8nuatvjMMD0+S8WrQCfWMJ+sYS3HwIGqJB9nj9VVubK9OZbIwxC7EAVYOCAeH5j+ni23ee5PhIvOzHFxG6W12C12Qmx6GBKR46Pfmo+T/TqRx39Y1zV984zTG3aOJFPS3WDGiMWRX2TVOjQsEAL7i4m2/deWLJVW9XIhYOcmFPCxf2tDCdynLUG2p8fCROqqRTdCKZ5dZHRrjt6Ajnbmniku2tbG5ev7n/jDHVZwGqhkVCAX730h5uOTzMb46P+hoWvBIN0RAXdLdwQXcL+bzSN5bg4OlJDg1MFYNVXuGBU5M8cGqSLc1RLuhq4ZytjZbpwBhTdr4ClIi0A28BLgFmLeeqqk8tf7FMQTgY4KnndHLu1ib+5/7TDE7OPxKv3AIBYUd7PTva67ni3E6ODE5zZ+/YrPlVpydSnJ4Y4KaHB9nZ4fbd2dFAcyxEOpcnlXFzLALiUvoERaiPBi2ZrTHGF781qC/iJux+FZfF3KyyLc0xrr1sB79+ZIRbHxmueG2qVCgQ4JwtTZyzpYlT40nu7B3j0MBUcXBFNq8cHpzm8OA0MIjgBlzMJyjC5uYoW1tibG2OFSeXRkNBVN3s+3gmh+CWuN/oKWCM2cj8BqgnAp2qujr/vpt5BQNugcJtbXXceO+pFU/oPRNbW2Jc2bKVRCbHQ6cmuffkOMNTs3MILhY7c6ou5cucTBfRUIBsTotBr6AuHKQxGqK5LkRzLExzXZhYOEA4WEi2KqQyOZKZPMlsjqR3P5XNIQiNsRBN0RBNsRDb2+ste4Yxa4jfAHU3sA04XMGyGJ+2t9fzkst38NOHBhmeTs+kLClzapXF1IWDXLK9lYu3tTAyneb4iBtY0TeaIJtXwkEhGnI5xFSVvEI2n2c6lZv3eKkFyp7I5EhkcrPmbZ2pgMDOjgbO2dJIOBhgyMscnskpnU1RurxaXX0kaEPqjakBfgPUj4EbReQzwKzFi1T102UvlVlSfSTEcy/qKj5WdcvIP3za/zLy5SAidDRG6WiMcumONvLqlntfKBt2PJ3llFeDGp5OMx7PMJ7MFNe0CgeFunCQvMJ0OlvWpsy8wiND0/POL5s7nD8cFMLBAA3REFubY3S3FoJXiHBQLIAZswr8BqinAH3Ab83ZroAFqBogIjzr/C2cGk8ymVz9pr+CgAgs8t1dHwmxu7OR3Z0zY21UlXg6RyTkmu4K8t72yWSGiYS3lEUyQzqbJ5tzyTlVIRoOuLWWQi4xaSwcJBoOkM/DVCrLZDLDqYnkgqme5pPJKZlcjng6x+BkintOzCTxFYFoMEBTLEx7Y4T2BpekVAC8JTnGExlG42nG4hkyuTyRYICw9/4CXobrgAiqrlmzkNA1r3g/lU2NUXZ11LO9vX7WdTFmo/AVoFT16ZUuiFm5WDjIcy7q4msHeld1EMVKici8k38DIi4zdzREV8vKzzMWT3Pw9BRHh6cJBlxi0M7GKOGgcHoiRf9EgsHJ1JLJQFVxGaGnUmVpelxI/3iSe06MEwwI3a0xulrq6PaaIaPWl2Y2gGXPgxLXtlH8H1lVV6/jwyypp7WOy3d3cMvh4WoXpea01ke47Kx2Ljur/VHP7d3SVLxfWDohk1NGptP0jyXoH08yNJ0ilXn0cguVlssrvSMJekdmJmw3xUK0N7jaW30kWKyRhQJCXSRIQyREXSRIMpNjKpVlKpUl5609FA0GCAUDZPOuJprO5ZlOZRlPuJpqJpenpS5Mu5fNOxwQ8gqKG2U5nXLHTGVdrbc+HCIWCdAYDdFaF6HFG8hizaBmpfzOg+oBPgI8Fbfceyn7V67GXLarndMTSY4MljeX30YREDfAIxpyQ90Li8gV5PJKMpNjLJ5heDrFyHSa6VQOLRm/2BgN0VbvvuALIxTTObcIXd4bNJLPqwssATf8PhiYueXySu9ogqND0wzPs9LyZDLLZDLLseHKzPoYnk5zZAW5IMNBoT4S8pa6D5DJKYm0G/ASEOhojLKpMUJbfYRcXkll3chLVTcHLyBuDmB7Q4SOhgjNdWHXfLyEwoCcQjMpuBGiFizXJr81qH/DzX96JvAzXKB6F3BDZYplViIQEK66uJtHhqa5+dDQo4aBm5UJBlyTZEM0RE9bXcXOs7OjgSfv2cREMsPJsURxcMngVKrmm3AzOWU8kWE8kZn3+Wlv1KdfwYAbPBMKCKGgFCd/g2tyLQS4VDb/qGsTCgjNsTBNdSGCIigukIWDAdq8ANhWHyESChD0/mEIBQLF8+Tzyngyw+h0mrFEhmBAqA8HqY+E3ChV3MAgxTUtFcqWyeXd9IdMrrjgYybnRtvG0zmm065mm825Ua9hr3bb1hChszFaXKQx7Y3QzasSCwepCwepiwRrol8yl1eODk2zo72+Iqv9+loPSkSGgR2qOi0iY6ra6mWX+KWqnlf2Uvm0HteDKrd8Xrm/f4JT40nimRzxVJYxb7lpszbl8spY3C2zPhrPkM7lyXs1hkxOiaezxNNugEc0HCj244UCQtr7gszmlFBQCAUDhAOuttNcF6KlLkw4GGA0nmZ0OsNYPE1e3cAQ8Wo1DZEQDVH3RZnK5t1UgHSOiWSmGJRqaVG/lSh8565yq64vhVpqfSRIvdes2xANEQsHiHu/j0lvJeOcKvm8zqqhBkSKtdd0Lu8m0TdF2dIcY0tzlGgoSCDg9svk8iTSOeKZHFPJrPt8xDOMxzPkVLn5bU9nW1v90oVewErXg8oBhaFhYyLSCUwAPcsowJXAh3FNgp9U1b+b8/xTgQ8BjwGuUdWv+z22WVggIMVksAWqykQiy8nxBMdH4tx/cqKKJTTLFQzMDO2vlC0rSASsqsXAFU+72kM46JYSr48ESefyDHtLz48nMoSDAaIhdwuIuC9TVZLpPMPTKYan08SX8Q9VQNw1CogUg/aZqsXAVLBULfVMHBuJc+wMVlA4PDi9ogC1EL8B6lbgucA3ge8DXwESuGXglyQiQeCjuGHqfcBtInK9qt5fsttx4JW4nH+mgkSElvowLfVhzu9q5pwtTfzgvlPL+hIwZiEi4ob9h4PM953VALTVR9izufHRTy4glcmRyuXJ5ZRMPk9+ztCsiBfgYuHgrDl4hWA5kcwwlcySVxfARIREJsfIdJphL1Dm8rOH/GdzM72Krk8xTGt9pDgtIp7OkcnnCVDS3OidszAXsC4cJBYJEgsFitMoCsG6MepqouFgoDgoJ57OMjyVZmAqxdBkiqw3sCXiTU9IZmZqrHOzrlTLluYo8QpltfEboF4GFBo8/wx4M9CEq/H4cRlwSFWPAIjIl4GrgWKAUtWj3nM2KnCVnbWpgZdcvpMb7z1FbwXWnzJmpaLh4BkNrS8Nlpublt6/VGGOGsoq5oSMsrOjYcm9inkrvb6seDrHdCrLdNoFr/pIsNjvFgsHi4NwBDdK1TX5uSAa9YJnMpPzEkC7Eau5nDeYR9X1u0WCXi3YBeu2hgj7upt5+RN2Vexq+J0HNVZyPwG8b5nn6QF6Sx73AY9f5jEAEJHrgOsAduzYcSaHMPNojIb435f2cH//BLccHq5Knj9jaomIEKrR0X8iUgzabQ2RshwzFg7SWh/h3K3+I3mlc1v6+rdARKIi8n4ROSIi4962Z4vI6ytaunmo6idUdb+q7u/s7Fzt069rhf6qVzxxF5fv7iASqv4oIWPMxuW3ie+DuFrQS4Dvedvu87Z/xMfrTwDbSx5v87aZGhQJBXjC2R08/qx2ktkc0ynXfHBqIknvSJz+8WQxd54xxlSK3wD1u8Aeb5h5HkBVT3gTeP24DdgrImfhAtM1wIuXXVqzqgKBwjDWEJ1NUXZtauDy3R1kcnkODUxx65FhRuPlG0FkjDGl/LbhpJkTzLyh5r7y6ahqFng9bgTgA8BXVfU+EXmPiFzlHe9xItIHvAj4uIjc57NsZpWFgwHO73Kdo8++YAstdeFqF8kYsw75rUF9DficiLwJQES6cCP4vuz3RKp6A3MyT6jqO0vu34Zr+jNrRCAgXNDdwr6uZvrHkxwamOLQwFRZ52UYYzYuvwHqHcAHgHuAeuBh4N+Bd1eoXGYNERG6W+vobq3jKXs3MRbPcHwkTu9onOGpNO0NETY3udQtk8ksvaNxekcSJDM278oYszC/w8zTwJuAN3lNe0PqJ0eS2XBEhDYvC/bF21vn3efi7a2oKmNxNwt+LJFhwpsomfcmShaOJbjJj/F01qVaSbtko1EvCWl9JERHQ4SOxiitdWEmk1kGp1IMTaXI5pTmukKanwAPnpqgbzQxb5mMMbVn0QAlIgtNNNpeyA6sqsfLXSiz/pUGsnJqa4iwo2P+lCsXbWthdDrN/f0TJLw8ddFQkGAA0lkllc0V0+IIEAhAPu+Wqk/nlEQ6y6nxVDFLtjGmspaqQR2FYraP+WasKbbchllD2hoiPGnPpjN+fSqbo3ckwbHhacYTGZdFO+OSaKYylgTFmHJaKkDdBdQBnwP+EzhZ8RIZU8OioSB7NjfOm0cumckxkcgwkcwWFw4s5IbLe9mkS5svwaWaiQTd0g6FpeLHE26p+NMTKU6OJSyrh9mwFg1QqnqpiFwIvAL4BW6I+OeBb3gpj4wxnmLOt+YzP0anN5ikQFWZTGXpG3GZ54+PTDOdmn9wSSwcLOZUS2ddbS4SCtDZGGVTU4RsTjk2HC9LwGuIBmmMhhmYTNb82lRm7VpykISq3gv8hYi8DZeN/JXAR0XkGar6mwqXz5gNTcQttrevO8y+7ubiUimZvFvATtWtGNsQDc1awC6Xd8lE5y69rqoMTqU4MZogIFLMsC0ysxotuEX+Cpm3c6pksm414EgowKbGKA1R99URT2c5PDDNocFJBidTxNM5C1imbPwOMwfYCzwNeAJwBzBakRIZYxZUWCplKUGviXG+129uirG56czXeypVHwlx0bYWLtrm1hvL57W4MGZhPahEJuctqOgyYxeycCcyWTJZZXNzlG1tbprCVMoFvMODUwxOpspSRrN2LTWKrx24FtfE1wT8B/BUG7lnjJlPICDFFXzPRH0kxOamGE84u4N4OstYPFOcjpDK5sjmlGw+z0Qyy+nxJNmSnJAN0SA72usJBwNMpdwSFFPJrPXhrWFLfYpOAo/gAtOvvG17RGRPYQdV/XGFymaM2cAKeSC7W+vmfT6byzMwmWJ4Ks3WlhibGiOzmjNL95tIZhlPZBiaSjE46W5TKTeYJeg1Z7bUuUU8W+vCNMVC1EVC1Hv9euBqf3mFiUSGYW+hw9Lgl80p/eOJM1rBNxoOsLU5RktdmEMDU7Z4qEcWm28rIkeZGWY+H1XV3eUulF/79+/XAwd8LeprjDEVl8nlOTYc58jgFNm8epPIIzREQ0wksozF00wks0RCAZpiIZpjYdobIrTVh4vBNZvL89DpSe7sHSOeytHWEKGjIUJrfZiGaMhbNDDonU/J5PJMpbKcnkhyeiLJ4GSKXB5EZlYPDogQEPdlnihj8Otpq+P3929fescliMjtqrp/7valRvHtWvGZjTFmgwgHAwtOQ+hq8XeMUDDABd0tXNDt8wWe87v8DR/NegFtMpkl4K2oGw0FSKRzHDw9xUOnJ5lIZAiI0NkUpas1Rn04SC6vZPKKqhILu9V12+rLO9F+rjNrKDbGGLMmhYIBWusjtM4JLk2xMJubYzxpTwcj02maYuGqL1pqAcoYY0yRiNDRGF16x1Vga3obY4ypSRagjDHG1CQLUMYYY2qSBShjjDE1yQKUMcaYmmQByhhjTE2yAGWMMaYmWYAyxhhTkyxAGWOMqUmLJoutdSIyCBxb4WE2AUNlKM56YddjNrses9n1mM2ux2xnej12qmrn3I1rOkCVg4gcmC+L7kZl12M2ux6z2fWYza7HbOW+HtbEZ4wxpiZZgDLGGFOTLEDBJ6pdgBpj12M2ux6z2fWYza7HbGW9Hhu+D8oYY0xtshqUMcaYmmQByhhjTE3asAFKRK4UkYdE5JCIvL3a5VltIrJdRH4iIveLyH0i8kZve7uI/I+IPOz9bKt2WVeTiARF5A4R+a73+CwRudX7nHxFRCJLHWM9EZFWEfm6iDwoIg+IyBM26mdERN7k/a3cKyJfEpHYRvt8iMinRWRARO4t2Tbv50Gc/+ddm7tF5LHLPd+GDFAiEgQ+CjwH2AdcKyL7qluqVZcF3qyq+4DLgT/xrsHbgR+p6l7gR97jjeSNwAMljz8AfFBV9wCjwB9WpVTV82HgRlU9D7gYd2023GdERHqANwD7VfVCIAhcw8b7fHwWuHLOtoU+D88B9nq364B/Xe7JNmSAAi4DDqnqEVVNA18Grq5ymVaVqvar6m+8+5O4L54e3HX4nLfb54DfqUoBq0BEtgHPAz7pPRbgGcDXvV022vVoAZ4KfApAVdOqOsbG/YyEgDoRCQH1QD8b7POhqjcBI3M2L/R5uBr4vDq/AlpFpGs559uoAaoH6C153Odt25BEZBdwKXArsEVV+72nTgFbqlWuKvgQ8FYg7z3uAMZUNes93mifk7OAQeAzXrPnJ0WkgQ34GVHVE8A/AsdxgWkcuJ2N/fkoWOjzsOLv2Y0aoIxHRBqB/wL+TFUnSp9TNwdhQ8xDEJHnAwOqenu1y1JDQsBjgX9V1UuBaeY0522Uz4jXr3I1Lmh3Aw08uqlrwyv352GjBqgTwPaSx9u8bRuKiIRxwekLqvoNb/PpQjXc+zlQrfKtsicBV4nIUVyT7zNw/S+tXpMObLzPSR/Qp6q3eo+/jgtYG/Ez8izgEVUdVNUM8A3cZ2Yjfz4KFvo8rPh7dqMGqNuAvd4InAius/P6KpdpVXn9K58CHlDVfy556nrgFd79VwDfXu2yVYOq/qWqblPVXbjPw49V9SXAT4AXerttmOsBoKqngF4ROdfb9EzgfjbmZ+Q4cLmI1Ht/O4VrsWE/HyUW+jxcD7zcG813OTBe0hToy4bNJCEiz8X1OQSBT6vq+6tbotUlIk8Gfg7cw0yfyztw/VBfBXbgljL5fVWd2ym6ronIFcBbVPX5IrIbV6NqB+4AXqqqqSoWb1WJyCW4QSMR4AjwKtw/thvuMyIi7wb+ADcC9g7gNbg+lQ3z+RCRLwFX4JbVOA38DfAt5vk8eIH8I7im0DjwKlU9sKzzbdQAZYwxprZt1CY+Y4wxNc4ClDHGmJpkAcoYY0xNsgBljDGmJlmAMsYYU5MsQJlZROSzIvK+Kp1bROQzIjIqIr+e5/lXisjN1ShbSRl2iMiUl3C43Mf+NxH5P+U+7jLLcJ83zH41z9npZUuvK/NxnyIiD5XzmKtBRB4jIr+sdjlqgQWoGiciR7309g0l214jIj+tYrEq5cnAbwHbVPWyahcGitf/WYXHqnpcVRtVNVfuc6nq61T1vd55rxCRvnKfo9R8/4yo6gWq+tNKnncebwc+q6qJlRxERFRE9hQeq+rPVfXcxV6zgnNV7B85Vb0bGBORF1Ti+GuJBai1IYhbBmJNOYNaxk7gqKpOV6I8G0lJ+p2aJiJRXPaB/6x2WVaTj7+NLwB/tBplqWmqarcavgFHcf9hjgCt3rbXAD/17u/CJWcMlbzmp8BrvPuvBH4BfBAYw2UDeKK3vReXN+sVJa/9LPBvwP8Ak8DPgJ0lz5/nPTcCPISbNV762n8FbsAlFn3WPO+nG5cCZQQ4BLzW2/6HQBLIAVPAu+d5beG9fASXTfpB4JlLHdt7LorLHHLSu30IiHrPbQK+612fEVyGjQDwH7gsGwmvTG+de729a/1er1yTwA+ATSXnfTludv0w8H+83+ejrkvJ9XsfLhFpwjv3lHfr9sr0duCwd7yvAu1zPgd/iEvLc5O3/Wu4DNPjwE3ABd7264AMkPaO/52Sz9uzfFyzK3C5+t6M+wz14zIFFN7Lc3GpgCZx+dfessB7fipu6ZvSbS24NFz93mvfBwS95/bgPpPjwBDwFW/7Td77n/bezx8Uyjjnb+kvgLu9/T6Fy7z9Pa+cPwTaSvZf7rU73/s8jAH3AVct9rex2DXCZahIFK73Rr1VvQB2W+IX5H1h4JJTvs/bttwAlcWlqAl6f+zHcQs2RoFne38gjd7+n/UeP9V7/sPAzd5zDbig9ipcputLvS+JfSWvHccl0QwAsXnez03Ax4AYcAluOYdnlJT15kWuReG9vAkIe19C48x8SS927PcAvwI2A53AL4H3es/9LS4oh73bU5jJsnKUkoAy93p71/owcA5Q5z3+O++5fbgvsCfjUgX9I+6LbdEA5d2/gpIvV2/bG733sM373Xwc+NKccn3e+z3VedtfDTQxE2zunO98cz9vPq7ZFd7v4j3eNXsuLp1Nm/d8P/AU734b8NgF3vOfAP89Z9s3vffW4J3718Afec99CfgrvM8X8OSS1ymwp+TxrGvovbdf4YJSDy6w/gb3OY4BPwb+pmR/39fOuwaHcOnCIrhkw5PAuQv9bSx1jYAJ4DHV/g6q5s2a+NaOdwJ/KiKdZ/DaR1T1M+r6Tb6CyzD8HlVNqeoPcP8J7inZ/79V9SZ1OcX+CniCiGwHno9rgvuMqmZV9Q5cNvQXlbz226r6C1XNq2qytBDeMZ4EvE1Vk6p6Jy7P28uX8V4GgA+pakZVv4KrxT3Px7Ff4r3nAVUdBN4NvMx7LgN04WqKGXV9F8vJAfYZVT2org/lq7jgCC6J6HdU9WZ1C2O+k5UtRfA64K9Utc/73bwLeOGc5rx3qeq0VxZU9dOqOlmy/8XeQoR+LHbNwF2393jX7AZcMD635Ll9ItKsqqPqLY45j1bcFzkAIrIFF+z+zHsfA7ja/zUlx90JdHu/5+UOmvkXVT2tbn2nnwO3quod3mf1m7hgBSz72l0ONOL+OUmr6o9xtfJrS/aZ+7ex1DWa9K7PhmUBao1Q1XtxH/gzWV77dMn9whfX3G2NJY+Li4yp6hSu2asb98XweBEZK9xwX2Jb53vtPLqBEXUr+BYcY3mLmJ2YEzyOecdd6tjd3uO5rwP4B9x/vz8QkSMistxrfKrkfpyZa9nN7GsZxzXNnamdwDdLrv0DuCbR0gUDi+cTkaCI/J2IHBaRCVwNAlyTph+LXTOAYZ1ZrA9mv/ffwwWaYyLyMxF5wgLnGMXVUgp24moj/SXv8+O4mhS4ZlYBfu2NOHy1z/dSMPdzP+/fwRlcu26gV1XzJdvmfrbn/m0sdY2acM2FG5YFqLXlb4DXMvtDXxhQUF+yrTRgnIniGi7egobtuD6IXuBnqtpacmtU1T8uee1iNYSTQLuIlH4h7WB5a8T0eFmSS19f6CNZ7NgncV9+c1+H91/ym1V1N3AV8Oci8kwf72cp/bjmOAC8YdQdPl8733l7gefMuf4xrzYw3+tejFtk71m4fp1dhaIsco5SC16zJQuvepuqXo0LLN/C1SznczeuebSgF0jh+vEK77FZVS/wjntKVV+rqt24QQQfKx25V0bLvXYnge0iUvqdOvezPes1i10jEenBNRWuuWHy5WQBag1R1UO4Jro3lGwbxP0RvNT7r+/VwNkrPNVzReTJ3lpZ7wV+paq9uBrcOSLyMhEJe7fHicj5Psvfi+vH+FsRiYnIY3Cd+ssZwbUZeIN37hfhOqZv8HHsLwF/7c252YRrbvtPcKvpisgeL/CN42olhf+ETwO7l1G+Ul8HXiAiT/Su5buY+YJbymmgY06T0r8B7xeRnV65O0Xk6kWO0YT7sh/G/QPzf+c5x2LvbcFrthgRiYjIS0SkRd3ifhPMXM+5fo1b9K8HQN16QT8A/klEmkUkICJni8jTvGO/SEQKQX8U96Vfjt/VXMu9drfiapBv9T6bVwAvwC3F8Sg+rtHTcGuSrdulO/ywALX2vAfXeVzqtbjRScPABbgv6pX4Iq62NgL8L+Cl4GoauEEV1+D+YzwFfADXiezXtbj/Rk/i2vz/RlV/uIzX3wrsxQ3OeD/wQlUtNJstduz3AQdw/7Hfg+scL8xj2YsbwTUF3AJ8TFV/4j33t7gv6TERecsyyomq3gf8Ke5Lqt87/gDui2+p1z6ICxBHvHN34wasXI9ripzEdfg/fpHDfB7XzHQCN1rsV3Oe/xSuD2RMRL41z+sXu2ZLeRlw1Gseex2uKfhRvL65z+J9xjwvx9Ue7scFoa/j+ggBHgfcKiJTuGvxRlU94j33LuBz3vv5fZ/lXMiyrp33Pl4APAf32fwY8HLv97iQxa7RS3D/kGxoth6UMavEay4dA/aq6iNVLk7N8Ab+/By4VFc4WXc98Gr/H1fVhfrtNgwLUMZUkJcN4Ee4pr1/wtV4HrvMUYLGbEjWxGdMZV3NzCCOvcA1FpyM8cdqUMYYY2qS1aCMMcbUJAtQxhhjapIFKGOMMTXJApQxxpiaZAHKGGNMTfr/Aa+2Edb49+j/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from lightgbm import cv, Dataset\n",
    "\n",
    "trn_data = Dataset(Xtrn, label=ytrn)\n",
    "params = {'boosting_type': 'gbdt', 'learning_rate': 0.25, \n",
    "          'max_depth': 1, 'objective': 'cross_entropy'}\n",
    "\n",
    "cv_results = cv(params, trn_data, \n",
    "                num_boost_round=100,\n",
    "                nfold=5, stratified=True, shuffle=True)\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(6, 4))\n",
    "m = np.array(cv_results['cross_entropy-mean'])\n",
    "s = np.array(cv_results['cross_entropy-stdv'])\n",
    "ax.fill_between(range(100), m + s, m - s, alpha=0.5)\n",
    "ax.plot(range(100), m, linewidth=3)\n",
    "ax.set_xlabel('Number of boosting iterations (estimators)', fontsize=12)\n",
    "ax.set_ylabel('Mean & std of 5-fold cross entropy', fontsize=12);\n",
    "\n",
    "fig.tight_layout()\n",
    "# plt.savefig('./figures/CH05_F19_Kunapuli.png', format='png', dpi=300, pad_inches=0)\n",
    "# plt.savefig('./figures/CH05_F19_Kunapuli.pdf', format='pdf', dpi=300, pad_inches=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternately, LightGBM plays nicely with scikit-learn, and we can combine the relevant functionalities from both packages to perform effective model learning.\n",
    "\n",
    "In the experiment below, we combine ``scikit-learn``'s [``StratifiedKFold``](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html) class to split the training data into several folds of training and validation sets. ``StratifiedKFold`` ensures that we preserve class distributions, that is, the fractions of different classes across the folds.\n",
    "\n",
    "**Listing 5.4**: Cross Validation with LightGBM and scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "import numpy as np\n",
    "\n",
    "# Initialize choices of learning rates, number of cross-validation folds\n",
    "n_learning_rate_steps, n_folds = 10, 10\n",
    "learning_rates = np.linspace(0.1, 1.0, num=n_learning_rate_steps)\n",
    "\n",
    "# Split the data into training and validation folds\n",
    "splitter = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\n",
    "\n",
    "# Initialize some structures to save training and validation errors\n",
    "trn_err = np.zeros((n_learning_rate_steps, n_folds))\n",
    "val_err = np.zeros((n_learning_rate_steps, n_folds))\n",
    "\n",
    "# Train LightGBM classifier for different learning rates and plot the results\n",
    "for i, rate in enumerate(learning_rates):\n",
    "    for j, (trn, val) in enumerate(splitter.split(X, y)):\n",
    "        gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=10, max_depth=1, learning_rate=rate)\n",
    "        gbm.fit(X[trn, :], y[trn])\n",
    "\n",
    "        trn_err[i, j] = (1 - accuracy_score(y[trn], gbm.predict(X[trn, :]))) * 100\n",
    "        val_err[i, j] = (1 - accuracy_score(y[val], gbm.predict(X[val, :]))) * 100\n",
    "\n",
    "        \n",
    "trn_err = np.mean(trn_err, axis=1)\n",
    "val_err = np.mean(val_err, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEYCAYAAABiECzgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAAsTAAALEwEAmpwYAAA+ZElEQVR4nO3dd3hUVfrA8e+bXkiBJECoASEUASkBQQlVVBBQVhcFFUSKuiLWVXfV39rWtbtrF0EBRWxrF1mVGixAUFCkhio9CRBCQvr5/XGHGCBlQjJzZybv53nmSWbunXvfG8KbM+ee8x4xxqCUUsr9/OwOQCml6ipNwEopZRNNwEopZRNNwEopZRNNwEopZZMAuwOoSGxsrElISLA7DKWUqrHVq1dnGGPiTn3dYxNwQkICqampdoehlFI1JiI7y3tduyCUUsommoCVUsommoCVUsommoCVUsomHnsTTilPVFJSwu7du8nJybE7FOVBAgMDadiwIZGRkdV6n/cn4Oc6Q9auyveJagG3/+qeeJRPy8jIQERo164dfn76AVKBMYbjx4+zZ88egGolYe9PwFm74MGsyvd5MMo9sSifd+TIERISEjT5qlIiQlhYGE2bNmXv3r3VSsD6W6RUNRQXFxMYGGh3GMoDhYaGUlhYWK33aAJWqppExO4QlAc6k98LTcBKKWUTTcBKuUFWbiHzVu7i+YVbmLdyF1m51fuo6k5Dhw5l9uzZtb6vOp3334RTyoMZY3hhURozUrbRLzGOhJhwlqdl8K/5G5iU3JpbBrWplS6NevXqlX6fm5tLcHAw/v7+ALz22mtcffXVTh/rq6++csm+6nSagJVyoRcWpfH1+v18c0d/GkWGlL5+4GgeE2evAmDa4LY1Ps+xY8dKv09ISGDGjBlccMEFp+1XVFREQIBv/bcv75qqe512/Vzc0gUhIu1EZE2Zx1ERuc0d51bKLlm5hcxI2cbM8T1PSr4AjSJDmDm+JzNStpF13HXdEUuWLKFZs2Y88cQTNG7cmAkTJnD48GGGDx9OXFwc9evXZ/jw4ezevbv0PQMGDGDGjBkAzJo1i759+3LXXXdRv359WrVqdVKrtzr7bt++nX79+hEREcEFF1zAzTffzDXXXFNh7F988QVdu3YlOjqa8847j19++aV0W0JCAk888QRdunQhPDyctLQ0RISZM2fSokULBg0aRElJCY8++igtW7akYcOGjBs3jqwsa8jqjh07TtvfDm5J+caYTUBXABHxB/YAH9fKwaNaVD3ONygCCvMgMKTy/ZSqhoc+/431e49WuP3g0Tx6tWpwWvI9oVFkCL1aNeCyF5fTsIJ9OjaJ5B8jzq5RnPv37+fQoUPs3LmTkpIScnNzmTBhAu+//z7FxcVcf/31TJ06lU8++aTc969YsYLx48eTkZHB9OnTmThxInv27Cm366SyfceOHcv555/Pt99+y8qVKxk2bBgjR44s95w///wz119/PZ9//jlJSUm8/fbbjBw5kk2bNhEcHAzAvHnz+PLLL4mNjeXAgQMALF26lA0bNuDn58esWbOYNWsWixcvLk3AU6dO5a233io9T9n97WDHWQcDW40x5dbHrLbbf7UmYlT0GDUdCrLhwwlQ7Lk3PpTvKSw2JDaKqHSftg0jKCw2Lo3Dz8+Phx56iODgYEJDQ4mJieHyyy8nLCyMiIgI7rvvPpYuXVrh+1u2bMnkyZPx9/dn/Pjx7Nu3rzThObvvrl27WLVqFQ8//DBBQUH07du3wuQLMH36dG644QbOPffc0mMFBwfz448/lu4zbdo0mjdvTmhoaOlrDz74IOHh4YSGhjJ37lzuuOMOWrduTb169fjXv/7Fu+++S1FRUbn728GOzqCrgHluO9s5V0L+UZh/F3x6M1z2KugsJlULqmqZzlu5i+VpGZXus+twLjcPasOYXi1qM7STxMXFERLyRws7NzeX22+/nQULFnD48GEAsrOzKS4uLr1xV1bjxo1Lvw8LCwNO7nN2Zt+MjAwaNGhQ+hpA8+bN+f3338s9zs6dO5k9ezYvvPBC6WsFBQXs3bv3pPefquxre/fupWXLlqXPW7ZsSVFR0Ul/PMo7hju5NROJSBAwEviggu1TRCRVRFLT09Nr78S9JsPA++GX92DBPWBc2+JQCmBYp3hSNqdz4Gheudv3Z+WRsjmdYZ3jXRrHqV0FzzzzDJs2bWLFihUcPXqUZcuWAdaIDVeJj4/n0KFD5Obmlr5WUfIFKzHed999HDlypPSRm5vLmDFjSvcprwuk7GtNmjRh584/Pmjv2rWLgIAAGjVqVOkx3MndTcGhwE/GmHI/vxhjphtjkowxSXFxpy2fVDP97oI+U2HldFj8WO0eW6lyRIUFMim5NRNnrzotCR84msekOauYlNyaqFD3Tm3Ozs4mNDSU6OhoDh06xEMPPeTyc7Zs2ZKkpCQefPBBCgoK+OGHH/j8888r3H/y5Mm8+uqrrFixAmMMOTk5fPnll2RnZzt9zjFjxvDcc8+xfft2jh07xt///neuvPJKjxoF4u5IxuDO7oeyRODCRyHvCCx7EkKjoc/NtoSi6o5bBrUBYMizS0lOjCMhJowdmbmkbE4vHQfsbrfddhtjx44lNjaWJk2acOedd1Z4A642zZ07l+uuu46YmBh69erFlVdeSXFxcbn7JiUl8frrrzN16lS2bNlCaGgoffv2pV+/fk6f7/rrr2fv3r3069ePvLw8LrroopO6NDyBuPJjx0knEgkHdgGtjTFVlC+DpKQk45JFOUuK4YPrYMNnMPJF6H5t7Z9D+awNGzbQoUOHar8vK7eQ+ev2kZ6dT1xEMMM6x7u95etprrzyStq3b++WFri7VPT7ISKrjTFJp77uthawMSYHiHHX+Srk5w+Xz4B5V8Hn0yAkEjpeandUysdFhQW69EabN1i1ahUNGjSgVatWfP3113z66afce++9dodlK8/pDHGngGC48m2Ycxn8dxIER8BZ9gzEVqqu2L9/P3/605/IzMykWbNmvPLKK3Tr1s3usGxVNxMwQFA4XP0+zBoO714N4z6F5r3sjkopnzVixAhGjBhhdxgepW4PiA2tD9d8BBGNYe4VsH+d3REppeqQup2AASIawbWfQGA4vDUKMrfaHZFSqo7QBAxQvyWM+wRKiqx+4aN7q3qHUkrVmCbgE+LawTX/heOHrSSck2l3REopH6cJuKym3WHMPDi8A+ZeDnkVV7pSSqma0gR8qlbJMHo27PsF3h0LhcftjkgplxMR0tLSALjxxht55JFHnNq3uubOncuFF154Ru/1RXV3GFpl2g2FUa/CR1Pggwlw5VvgX7dnLakz9FxnyNpV+T5RLayyqjVw8cUX06tXLx5++OGTXv/000+54YYb2L17t9M1EF599dUaxXLCjh07aNWqFYWFhaXnvvrqq6u1PJKv0wRckS6jIS9Ly1iqmsnaZdWlrkxVCwo4Yfz48dx333089NBDJ1X4euutt7j66qs9qgCN3TxpCSPNKJXpNRkGOcpYfnW3lrFUHuuyyy4jMzOTlJSU0tcOHz7MF198wbhx41i5ciV9+vQhOjqa+Ph4pk6dSkFBQbnHuu6667j//vtLnz/11FPEx8fTpEkT3njjjZP2/fLLL+nWrRuRkZE0b96cBx98sHTbicI50dHR1KtXjx9++KF02aITvv/+e3r27ElUVBQ9e/bk+++/L902YMAAHnjgAc4//3wiIiK48MILyciouL6yNy5hpH8Wq5J8Fxw/Aj+8CCFRMPgBuyNSnuKre2F/zboOSr15SfmvN+4MQx+v8u2hoaGMHj2aOXPmlCa+999/n/bt23POOeewevVqnnvuOZKSkti9ezdDhw7l5Zdf5rbbbqv0uAsWLODpp59m4cKFtGrVismTJ5+0PTw8nDlz5nD22Wezbt06hgwZQteuXbnssstYtmwZrVq14siRI6WtxU2bNpW+99ChQ1xyySU8//zzjBkzhg8++IBLLrmEtLQ0YmKssjHvvPMOX331Fc2bN2fo0KE8/fTTPP746T8Pb13CSFvAVTlRxrLbtZDyNHzvWeXslDph/PjxfPjhh+TlWbWH58yZw/jx4wHo0aMHvXv3JiAggISEBG644YZKlyE64f3332fChAl06tSJ8PDwk1q4YLVSO3fujJ+fH126dGHMmDFOHRes1nPbtm259tprCQgIYMyYMbRv3/6kOsETJkwgMTGx9A/MmjVryj2Wty5hpC1gZ4jAiP9YSxt9fb/VEu4+zu6olN2caJk63b874cuaxQL07duX2NhYPvnkE3r27MnKlSv56KOPANi8eTN33HEHqamp5ObmUlRURI8ePao85t69e0/ar+wSP2Atwnnvvfeybt06CgoKyM/P589//rNT8Z66ZNCJ4+/Zs6f0+alLHFW0FJK3LmGkLWBn+fnDn163qqZ9fiv89ondESl1mnHjxjFnzhzefvttLrrootLld2666Sbat2/Pli1bOHr0KI899phTSxDFx8eftHTQrl0nj+gYO3YsI0eO5PfffycrK4sbb7yx9LhVLfdz6pJBJ47ftGlTp661LG9dwkgTcHWcKGPZrKdVxjLtW7sjUuok48aN49tvv+X1118v7X4AaxmiyMhI6tWrx8aNG3nllVecOt7o0aOZNWsW69evJzc397Ti6dnZ2TRo0ICQkBBWrlzJO++8U7otLi4OPz8/tm3bVu6xhw0bxubNm3nnnXcoKirivffeY/369QwfPrza1+2tSxhpF0R1BYXD2Pf+KGNZVP6Ci6VqYYynUs5KSEjgvPPOY+3atSct+/70008zZcoUnnzySbp168aVV17JokWLqjze0KFDue222xg0aBB+fn48+uijzJ07t3T7yy+/zJ133snUqVPp378/o0eP5siRI4DVZXDfffdx/vnnU1hYyIIFC046dkxMDF988QW33norN910E23atOGLL74gNja22tftrUsYuW1Joupy2ZJEteXYQXi6rXNjPKvaR3mNai9J5KaJGMozeOySRD6nXkO7I1DeQBOrqoT2ASullE00AbtDfvlDZ5RSdZt2QbjD480hroNV7rJZEjTtYT331x+/UnWZZgB36Hc37EmFjV/Az45pjYFhEN/15KQc1dya9KE8mjGm1seDKu9XUlJS7fdoAnaHgX+zvhoDh7bBnp+shLxnNax83aozARDe0ErEzXpYX5t0h9Do8o+pd9dtERISQmZmJjExMZqEFWD9QS4sLOTAgQOEh4dX672agN1JBGLOsh5dHNM1iwrgwDorGe9ZDbtTYfNXf7wnpu0fLeSmPaBRJwgIcluZQ3WyZs2asXv3btLT0+0ORXmQgIAAoqKiqj2GWRNwTUS1qDrJRbWofHtAkNUN0bQ74Kg0dfwI7P3Z0Ur+CdIWwtp51jb/IGjcpaaRqzMUGBhIq1at7A5D+QhNwDXhqo/3odFw1kDrAVbXRdbuP7otdq92zXmVUm6lCdgbiEB0c+tx9ijrNe1eUMrr+UwCzsotZP66faRn5xMXEcywTvFEhek6bkopz+W2iRgiEi0iH4rIRhHZICJ9auO4xhieX7iF5CcX8V1aBgVFJSxPyyD5yUU8v3CLUyX3lFLKDu5sAf8HWGCMuUJEgoCw2jjoC4vS+Hr9fr65oz+NIkNKXz9wNI+Js1cBMG1w29o4lVJK1Sq3tIBFJAroB8wEMMYUGGOO1PS4WbmFzEjZxszxPU9KvgCNIkOYOb4nM1K2kXW8sKanUkqpWueuFnArIB14U0TOAVYDtxpjcsruJCJTgCkALVpUMXwLmL9uH8mJcacl3xMaRYaQnBjH/F/3MaZX1cfzKrUxBE4pZSt3JeAAoDtwizFmhYj8B7gXOGmJYWPMdGA6WPWAqzpoenY+rWIqn3mSEBNGenb+mcbtuSoaAvfZNFjzDkz7CaI1ASvlydx1E243sNsYs8Lx/EOshFwjcRHBbM/MqXSfHZm5xEUE1/RU3qP/3SB+sOQJuyNRSlXBLQnYGLMf+F1E2jleGgysr+lxh3WKJ2VzOgeOlr8s0P6sPFI2pzOsc3xNT+U9oppBz4mw9h3I2GJ3NEqpSrizHvAtwFwR+QXoCjxW0wNGhQUyKbk1E2evOi0JHziax3VvrmRScmuiQuvYeOC+d0BAKCz+p92RKKUq4bZhaMaYNcBpayLV1C2D2gAw5NmlJCfGkRATxo6MXBZtPEhRSQlt4qpXncgn1IuDPn+BZU9B39sh/hy7I1JKlcPrV8QQEaYNbkvK3YPo2yaW4AB/+raNZelfB9ClWTS3vb+W1B2H7A7T/fpMhZBoWKStYKU8ldcn4BOiwgIZ06sF0wa3ZUyvFjSMDGHGuCSaRYcyaU4qW9Pr2LJAodFw/q2w5X+w60e7o1FKlcNnEnB56ocHMWtCLwL8hOveXOmbw9Eqc+4NVpH3hQ9bFdWUUh7FpxMwQIuYMGaO70lGdgETZ68it6DI7pDcJygc+v0Vdn4HWxfZHY1S6hQ+n4ABzmkezYtju7FuTxZT3/mZouLqr93ktXqMt2bELXpEW8FKeRifKUdZlcEdGvHIZZ247+N1PPDpbzw2qlPdWNMrIBgG3AOf3mwtCtphhN0RqTOhawD6pDqTgAGuPrclew4f5+UlW2lWP5SbB7axOyT36HIVLP83LHoU2g0DP3+7I1LVpWsA+qQ60QVR1l8vaseobk156n+b+Pjn3XaH4x7+ATDoPkjfCL9+YHc0SimHOpeARYQnLu/CeWfFcPeHv/BdWobdIblHh0utxTyX/MtaiVkpZbs6l4ABggL8ePXaHrSOrceNb61m4/6jdofken5+MOgBOLwDfn7L7mhUdWSkOb/va/3gi9vh57lwcCOU1KEbzl6oTiZggMiQQN6c0JPw4ACue2MV+7KO2x2S67UdAs17W1OUC+vA9XqzkhLY8i28fQW82MP594VEw68fwqd/gZfPhcdbwOwR8O2DsOELOLrPVRGrM1CnbsKdqkl0KG9O6MmfX/2BCW+u4v0b+xAZ4sOFe0Rg8P/BrGGwagacd4vdEalT5WfDmnmw8jXITLMm0gz4m9V15Izxn1nJO3ML7FkNu1Otr9+/ACWOMfCRTaFpd2iaBE17QJNuEFyv4mPqCAyXqdMJGKBDfCSvXduD8W+s5Ka3V/Pmdb0ICvDhDwYJ58NZgyHlWeg+HkIi7Y5IAWRuhZWvw89vQ0G2lRj/9Dp0vAwCgpxPwGB1N8W1sx5dx1qvFR6H/b+enJQ3fG5tEz+Ia39yUm7Y0bp5CzoCw4XqfAIGOL9NLE9c3oU7P1jLvf/9hWdGn+PbY4QH3Q+vD4QfX4YB99odTd1VUgLbFsGK6bDla/ALgLNHWVPIm9Vy4cDAUGjey3qckJMJe3/6IyFv/NL6AwBWOdMmXa1krFxGE7DD5T2asffIcZ75ZjNNokO566J2Vb/JWzXtbk3I+P5F6DUFwhrYHVHdkp8Na9+FFa9ZXQXhDaH/PZA0ASIal/8eV6wBGB5j3RdoO8R6bgwc3g67V1sJeU+q1SpXLqMJuIypg9qwN+s4Ly5Oo0l0KGPP9eE11Qbeb92UWf4cXPiI3dHUDZlbrb73n9+G/KPQpDuMmg5nX2bNWKyMO/pXRaBBa+vR5c/Wa0UF8Gic689dR2kCLkNEeOTSTuzPyuP+T36lcVQwg9o3sjss12jYHs65ClZOh95/gcg6tGyTOxkD2xZbrd3N/7NmIZ49Cs69sfa7GVwhIMjuCHyaD99tOjMB/n68OLY7ZzeJ4ua5P/PL7iN2h+Q6/e+x7owve8ruSHxP/jHr4/tL58Jbo6yP9P3vhtvWweUzvCP5KpfTFnA5woMDmHldEn96+Xuun7WKj246nxYxYXaHVfsatLJGQvw02xqS1qCV3RF5NmeGY0U2gY6jHN0MWRDfFUa9ZrV6q+pmUHWOJuAKNIwIYdaEXlz+yvdc9+ZK/nvTedQP98GPY/3+CmvmwtInYNSrdkfj2ZwdjrXyNeh4qaOboafVt6pUOTQBV6JNw3rMGJ/E1TNWMHlOKm9POpeQQB+rJBYZb42E+P4Fawmjhh3sjsj73far1RL2FU6NwGjunlh8jCbgKvRMaMBzo7sydd5P3P7eGl4a2x0/Px9r0fS9HVLftJaxv/Jtu6Pxfr6UfKHyERi/fAAfTYJek90Xjw/Rm3BOuKRLPPcN68BX6/bz6Jcb7A6n9oU1gPOmWjOj9vxkdzTKm3S+AtoPt1bfTt9kdzReRxOwkyYlt2bC+Qm88d12Zi7fbnc4ta/3XyC0gVW0XZ0urw5UzDsTIjD8OWv9wU9uguI6tOZiLdAEXA33X9KRi89uzKNfruerX32sqlRIJCTfAVsXwo7ldkfjWTZ9BS/3tjsKz1WvIQx7ylH053m7o/EqmoCrwd9P+PdVXeneoj63vreG1B2H7A6pdvWcBBHxsFAX8AQg+wB8cB3MuwqCtWhRpTpdDh1GWkWDDvpgN52LaAKuppBAf14fl0Sz6FAmzUlla/oxsnILmbdyF88v3MK8lbvIyi20O8wzExhqDUv7/UdI+9buaOxjDPz0FrzU0ypQM/B+uGGZ3VF5NhG45FkIjtCuiGoQ46EtnaSkJJOammp3GBXalZnLqJeXY4CiYkO/xDgSYsLZnplDyuZ0JiW35pZBbbyvqlpRgZV4giNhylKrtGFdkrkVPr8VdqRAy/NhxH8gtq21TeviVu23j61PDYMegH532R2NxxCR1caY06Y/6jC0M9QiJoyLO8Xz47ZM3pncm0aRIaXbDhzNY+LsVQBMG9zWrhDPTEAQDPg7fDwFNnxqzeCqC4oLrf7LJU9AQAgM/7c1S7DsH6C6nFiddfYo+O0TWPI4tBsKjc62OyKP5rbmjYjsEJFfRWSNiHhu09ZJWbmFfL5272nJF6BRZAgzx/dkRso2so57YXdE5yusAt2LH6sbHyX3rIbpA2Dhw5B4EUxdaZWGrGut/9pyyTMQEuXoivDC3383qvI3TES6i8gDIvKhiHzr+PqAiJxJNZGBxpiu5TXFvc38dftITow7Lfme0CgyhOTEOOZ742gJP3+raHvGZvjlPbujcZ38Y7DgbzDjAsjNhCvnwpVvVVyTVzknPBaGPwv71lrlTlWFKuyCEJGLgH8CEcBS4Dsg2/G8AzBXRLKB+40xC9wQq0dJz86nVUx4pfskxISRnp3vpohqWfvh1lphSx63WsS+Vkhmy7fW6sFZuyBpIlzwD6vVpmpHx0utkRFLn7S6Ihp3tjuiM+Pifv/K+oAnAzcZY1ZVtIOI9ATuAZxJwAb4WkQM8JoxZno5x5sCTAFo0cKzi6HHRQSzPC2j0n12ZObSt02smyKqZScW8HxrFKyeDedOsTui2pGTAQvuhV8/gNhEmLAAWvaxOyrfNPQp2L7M6oqYvBj8vXDBWxevh1dhF4Qx5orKkq9jn1XGmCucPFdfY0x3YChws4j0K+d4040xScaYpLg4z67CP6xTPCmb0zlwNK/c7fuz8kjZnM6wzl5c6Lz1QGjZ16oXXJBjdzQ1Y4y12vCLPa2bRP3vhRuXa/J1pfAYa5bc/l8h5Rm7o/FI1b7LICIXishdIvInqcYYK2PMHsfXg8DHQK/K3+HZosICmZTcmomzV52WhA8czWPS7FVMSm5NVKgX/tU/QQQGPwA5B62VM7zVoe1WS/6TGyGmDdyYAgP/5nvdKp6owwjo/Gfrj/i+X+yOxuNUaxiaiDwM9AZSgRHAKOBaJ94XDvgZY7Id318IPFz9cD3LLYPaADDk2aUkJ8aREBPG1oM5LN50kAHt4kq3e7UWvaHtRbD839BjAoRG2x2R84qLrJWfFz9mrTg87Gmrv1dHN7jX0CdP7orQZY5KVfqbKCKXnfJSP2PMhcaYvwMXAMOdPE8jYLmIrAVWAl/6wo07EWHa4Lak3D2Ivm1iCQ7wp19iLB3iI1i98zDZ+T4yhGvQ/ZB3BH540e5InLdvLcwYBN88AGcNhJtXWCUTNfm6X1gDa1z1gXW6/NUpqmoBDxWR64FbjDE7gQ0i8ipWC3gAVjKtkjFmG3BOTQL1ZFFhgYzp9cdNw85Noxn50nJeXJTG34f5QIHz+C7WAPsfXoZeN0A9D+6fL8i16hH88JI1HOrPs6078t42I9HXtB8GXa6y+oLbXwJNutodUdU2f+3yU1SagI0xN4hIH+BdEZkP3A1cA3QH1gKvuTxCL9S5WRSjezTnze+2c1XP5rSOq2d3SDU38D5Y/6k1rvPix+yLw5lhQf7B0O1qGPIwhNZ3T1yqakMfh21LrK6IKUs8uw9+xXRYcI/LT1Pl5zFjzA/A+VhjgBcDW40xfzHGPGWM0SKpFbjronYEB/j7TgH32LbQdSysmgFZe+yL48SwoMoexfkw8gVNvp4mtL5VW+Pgemt8sCcqLoL5d8NXf4XEi11+ukpbwI5RDqOA1sBvwKXAcyIyCbjVGOOF07zcIy4imGmD2/DY/I0s3nSQge0a2h1SzfW/B355H5Y9af1HUqq62l0MXa+2Pkm1vwSadrc7oj/kZ8OH18OWr6HPVOsT1H+6OrEe3pnPWaiqD3gO0ApIAe4DlhpjRovIxcB8EXnbGKMD/Cpw3XmtmLfydx75Yj1928QS6O/lN4CiW1gjIVbNgPOmQcxZ7j1/SbF7z6dc46LHYOtiqyvihmWe0RVx5Hd450pI32jdMEyaYL3u4gJMVWWES7DqN/wNa9TDJQCOEQx9AK1SXYmgAD/uv6QD29JzmPPDTrvDqR3Jd1r/YZb8y/XnytoD6z+Db/4PZg2Hxz17dqRyUmg0jHzeSnbu+D2qyp7V8PogyNoN13z4R/J1g6pawCuBh0RkMVYCXnFigzEmD/iHC2PzCYPaN6R/Yhz//nYzl3VtQkw9D/hrXxMRjeDcG62PkH1vr71yg3lHYe/PsCfVWhh0z2rIdvRw+QVaIzG6jvXuCSHqD22HQLdr4Lv/QPsR0KyHPXGs/xQ+cozsGf85NGzv1tNXlYBHAzdg9QOvA/7P5RH5GBHhgeEduPjfKTzzzWYeG+WlRUnK+vV9wMAr51W8T2UFSooL4cBvVpI98UjfZB0ToMFZ0KofNO0BTZOgcac/PqZqAvYdpV0RN8INKRBYfmVBlzDGakQsfAia9YKr3rFleGVVw9COAjpyuobaNIxgXJ8E3vx+O1ef24Kzm3h51a2s3c4XKDEGDu9wJNqfrBbuvrVQ5Ji+HRZjJdlOl1s3ZJp0twbuK98XEmV1Rbx9OSx5zLrp5Q5FBVYlvDVvW793l77s3uRfRmXlKJ8FnjTG7K9kn8bA3caYO1wRnC+5dXBbPlmzh4c+X897U3p731JFZ2Lun63Em5tpPQ8Igfiu1nTgZj2sFm50S50kUZe1ucBaeeT7F6yuiOY9XXu+3EPw/jhryan+98CAv9n6+1dZC3gTsFJENmDVA97EH/WAE7FmwrUDHnVxjD4hKiyQOy9M5L6P1zH/1/1c0sWLq6Q568guSBxqtWybJUHDjjUvSRjVwqXDgpQNLnwUti6yRkXcmGItDusKmVvhndHW7+Wo6XDOla45TzVUmICNMa+JyBtYY3+HApcB0cBh4BfgVeBzY4yPFDxwvat6tuCtH3by2PwNDO7QkJBAf7tDcq2bV1S9T3Xpumy+JyTSmjjz1mWw6FG46J+1f46d38O7YwGBcZ95TBnSSoehGWMKjTEfGmMmGmN6GWMSjTHnGmMmG2M+1uRbPf5+wj9GnM2eI8eZvmyb3eEo5TnOGmiNMf/hJdhVy3+4174Ls0dCWCxMXugxyRfcuCinsvQ5K4ZhnRvz8pI09h45bnc4SnmOCx+BqOZWV0RBbs2PV1Jitag/vsFKupO+gQata37cWqQJ2AZ/G9oBY+CJBRvtDkUpzxEcAZe+CIe2WomzJgqPw38nWuUvu10L13zkkbVBNAHboHmDMG7o15pP1+wldcchu8NRynO07g89J1mF9Hd+f2bHOHYQZo+A3z6yhraNfMFj16MTY0zlO4j4YY14WG6MKXBHUABJSUkmNTXVXadzu9yCIgY9vZS4iGA+vfl8/Py8aCiWi1eKVXVc/jFrko+fv7VuX1Dlq4+f5OAGmDsactLhT9Oh40jXxVkNIrLaGJN06utVLklkjCkRkU+NMRGuCa1uCgsK4G/D2nPru2v4cPVuRvdsbndIztPEqlwpuB5c+hLMHg4LH4ahTzj3vrRv4YMJ1jC2CfM9q9JaBZztglgmIr1dGkkdNPKcJvRoWZ8n/7eR7LxCu8NRynO0SoZeU2DFq7BjedX7r5phtXyjW8DkRV6RfMH5BLwT+EpEZonIIyLy8ImHK4PzdSLCP0Z0JONYAS8uSrM7HKU8ywUPQv0E+PRmKMgpf5+SYljwN/jyTmtW3fULIKqZO6OsEWcTcCjwCVa1lGZAc8fDe67UQ3VpFs2fezTjje+2sz2jgl8ypeqioHCrTsPhHfDtg6dvzz9mTa748WU49yYYM88aSeFFqrwJZxdfvwlX1sHsPAY+tYQ+Z8UwY7yL58Ir5U2cueEbFgt3b3VPPGfojG/ClTlAW2AM0BTYA8wzxmypvRDrroYRIdwyuC2Pf7WRpZvT6Z/owasOK+VOJ9YArExVtUE8mFNdECIyAlgNtAcOYRXhSRURzxjj4QMmnJ9AQkwYj3yxnsLiErvDUUq5gbN9wI8Blxpjxhpj/maMuRqrSI+N65P7luAAf+6/pCNpB4/x9o8+snyRUqpSzibgZlgLc5a1HL0JV6sGd2hIcttYnvtmM5nH8u0ORynlYs4m4DXAnae8dofjdVVLRIT/G96RnIJinv1ms93hKKVczNkEfBMwSUT2isgKEdkLTHG8rmpR20YRXNu7JfNW7mL93qN2h6OUcqEqE7CjFkQToBvWIp3POL52MMZscG14ddPtFyQSFRrIw1/8hqcOE1RK1VyVCdgYUwJ8aow5ZoxZbox53/FV5866SFRYIHdc2I4ftx1iwboKl+RTSnk5Z8cBLxOR3saYH2tyMhHxB1KBPcaY4TU5lq8b07M5c3/cyT/nb2Bg+zqwfJFS5fHxNQCdTcAnakF8CvyONSUZAGPM/1XjfLcCG4DIarynTgrw9+P/RnRk7OsrmJGyjamD2todklLu5+OV99xWC0JEmgGXADOqF2Lddd5ZsQzt1JiXFm9lf1ae3eEopWpZlS1gR7fB78A/jTE1GZz6b+BurGXtKzrXFKzRFbRo4b0fK2rT34d1YOHGgzz+1Qb+fVU3u8NRStUiZ27CFWMNNzvjm24iMhw4aIxZXcW5phtjkowxSXFxWg8BrOWLpiS35pM1e1m9U5cvUsqXONsF8RZwYw3Ocz4wUkR2AO8Cg0Tk7Rocr065acBZNIoM5qHP11NSosPSlPIVzibgXsB/RGSHiKSIyLITD2fe7Kgf0cwYkwBcBSwyxlxzhjHXOeHBAdw7tD2/7M7ivz/ttjscpVQtcXYUxOuOh7LJpec0Zc4PO3liwSYu7tSYiBDPXOVVKeU8pxKwMWZ2bZ3QGLMEWFJbx6sr/PyEf4w4m8te+o6XFm/l3qHt7Q5JKVVDlXZBiMjzpzyfeMrz/7oiKFW+rs2jubx7M95Yvp0dunyRUl6vqj7g6055/tQpz4fUXijKGfdc3I5Af+Gf87UMh1LerqoELFU8V27WMDKEqYPa8s36A6RsSbc7HKVUDVSVgE8d86RjoDzA9X0TaBkTxsOfryfjWD7zVu7i+YVbmLdyF1m5WiNJKW9RVQIOEJGBIjJIRAaV81wrxNggOMCfvw9tz/aMHPo/uZjv0jIoKCpheVoGyU8u4vmFW7SMpVJeoKpREAeBN8o8zzzl+cFaj0g5ZeP+bFrFhvP2pHNpFBlS+vqBo3lMnL0KgGmDtYCPUp6s0gTsmDihPExWbiEzl2/nmzv6n5R8ARpFhjBzfE+GPLuU8eclEBWq44WV8lTOzoRTHmT+un0kJ8adlnxPaBQZQnJiHPN/3efmyJRS1aEJ2AulZ+fTKia80n0SYsJIz9aVlZXyZJqAvVBcRDDbMyufiLEjM5e4iGA3RaSUOhOagL3QsE7xpGxO58DR8ou078/KI2VzOsM6x7s5MqVUdWgC9kJRYYFMSm7NxNmrTkvCB47mcfUMa+m+dXuy7AhPKeUkZ6uhKQ9zy6A2AAx5dinJiXEkxISxIzOXlM3p/KlbU1LSMrhm5gpuG5zI1EFt8PfTSYxKeRrx1AH7SUlJJjU11e4wPF5WbiHz1+0jPTufuIhghnWOJyo0kNyCIu7/eB0f/byH5LaxPHdlV2LraZ+wUnYQkdXGmKTTXtcE7LuMMby36nf+8dlvRIcF8sKY7vRq1cDusJSqcypKwNoH7MNEhKt6teDjv5xPWFAAY17/kVeWbNVljZTyEJqA64COTSL5bOr5XHx2Y55YsJFJc1I5nFNgd1hK1XmagOuIiJBAXhzbjYcvPZuULekMf2E5P+86bHdYStVpmoDrEBFhXJ8EPrzxPERg9Gs/8Mby7Vo5TSmbaAKug85pHs2XtyTTP7EhD3+xnpve/omjeVpHWCl30wRcR0WFBfL6uB7cN6wD32w4wPDnl+vEDaXcTBNwHSYiTO7Xmvem9KagqIQ/vfI9b/+4U7sklHITTcCKpIQGfDmtL71bx3D/J+u47b015OQX2R2WUj5PE7ACIKZeMLOu68ldFyby+dq9jHxxOZv2Z9sdllI+TROwKuXnJ0wd1Ja3J51L1vEiLn1pOR+u3m13WEr5LE3A6jTnnRXL/Fv70rV5NHd9sJa7P1zL8YJiu8NSyudoAlblahgRwtxJvbllUBs+WL2bUS9/x9b0Y3aHpZRP0QSsKuTvJ9x5YTtmTejFwex8Rr6wnM/W7i3dnpVbyLyVu3h+4RbmrdxFVq6OJVaqOtxSDU1EQoBlQDBWDeIPjTH/qOw9Wg3Ns+zLOs4t7/xM6s7DXH1uc2LCg5n1/Q76JcaREBPO9swcUjanMym5NbcMaoOI1h9W6oSKqqG5qyB7PjDIGHNMRAKB5SLylTHmRzedX9VQfFQo86b05qn/beKN5dtJiA3nmzv6n7Qy84GjeUycvQqAaYPb2hWqUl7DLV0QxnKiAzHQ8dDR/l4m0N+Pmwe0ISTQn7mTzj0p+QI0igxh5viezEjZRtZx7Y5Qqipu6wMWEX8RWQMcBL4xxqwoZ58pIpIqIqnp6enuCk1Vw/x1++jfLu605HtCo8gQkhPjmP/rPjdHppT3cVsCNsYUG2O6As2AXiLSqZx9phtjkowxSXFxce4KTVVDenY+rWLCK90nISaM9Ox8N0WklPdy+ygIY8wRYDFwsbvPrWouLiKY7Zk5le6zIzOXuAhdf06pqrglAYtInIhEO74PBYYAG91xblW7hnWKJ2VzOgeO5pW7fX9WHimb0xnWOd7NkSnlfdzVAo4HFovIL8AqrD7gL9x0blWLosICmZTcmomzV52WhA8czWPsjB8Z6liZWSlVObcMQzPG/AJ0c8e5lOvdMqgNAEOeXUpyYhwJMWHsyMwlZXM6wQF+fPnLXsb3SaBjk0ibI1XKs+my9OqMZeUWMn/dPtKz84mLCGZY53iO5RdxxSvfU1Ri+O+N59EiJszuMJWyXUUTMTQBq1qXdjCbK179gciQQD68qQ8NI8ofsqZUXVFRAtZaEKrWtWkYwZvX9STjWD7j31ilkzKUqoAmYOUS3VrU59VrepB2MJvJs1PJK9RylkqdShOwcpl+iXE8O7orq3YeYuo7P1FUXGJ3SEp5FE3AyqVGnNOEh0eezbcbDnLPf3+lpMQz7zkoZQd3VUNTddi1fRLIzCng399uoUF4IH8f1kHLVSqFJmDlJrcObsvhnAJeT9lOg/Bgbhpwlt0hKWU7TcDKLUSEf4w4m0O5hTyxYCMNwgO5smcLu8NSylaagJXb+PkJz/z5HLKOF/K3j34lKjSIizs1tjsspWyjN+GUWwUF+PHqNd05p3k00979mR+2ZtodklK20QSs3C4sKIA3r+tJywZhTJ6Tyro9WXaHpJQtNAErW0SHBTFnYi+iQgMZ/8ZKtumS96oO0gSsbBMfFcpbE3thgGtnrmR/Vvk1hpXyVZqAla1ax9Vj9oReHMktYNwbKziSW2B3SEq5jSZgZbvOzaJ4fVwSOzJyuX7WKnILiuwOSSm30ASsPMJ5bWL5z1VdWfP7Ef4y9ycKtW6EqgM0ASuPMbRzPP8c1Zklm9K564O1WjdC+TydiKE8ypheLTiUU8BT/9tE/bAg/jGio9aNUD5LE7DyOH8ZcBaHcgqYuXw7MeFB3DK4rd0hKeUSmoCVxxER7hvWgcM5BTzzzWbqhwdxTe+WdoelVK3TBKw8kp+f8MQVXThyvJAHPl1H/bAgLukSb3dYStUqvQmnPFagvx8vje1OUsv63Pbez6RsSbc7JKVqlSZg5dFCg/yZMb4nZ8XV44a3VrPm9yN2h6RUrdEErDxeVGggc67vRUy9ICa8uZK0g9lk5RYyb+Uunl+4hXkrd5GVqysvK++jCVh5hYaRIbx1/bn4iXDFqz/Q98lFfJeWQUFRCcvTMkh+chHPL9yCMTp2WHkPvQmnvEZCbDhDOzfmh62ZvDO5N40iQ0q3HTiax8TZqwCYpsPWlJfQFrDyGlm5hXy2Zu9pyRegUWQIM8f3ZEbKNrKOa3eE8g6agJXXmL9uH8mJcacl3xMaRYaQnBjH/F/3uTkypc6MWxKwiDQXkcUisl5EfhORW91xXuVb0rPzaRUTXuk+LRqEsWrHIfKLit0UlVJnzl19wEXAncaYn0QkAlgtIt8YY9a76fzKB8RFBLM8LaPSfTbty2bRpoN8/dsB+ifGMaRjIwa2a0hUWKCbolTKeW5JwMaYfcA+x/fZIrIBaApoAlZOG9Ypnn/N38CBo3nldkPsz8ojdechXhzTle+2HuLbDQf48td9+PsJ57ZqwAUdGjGkYyOaNwizIXqlTifuHrYjIgnAMqCTMeboKdumAFMAWrRo0WPnzp1ujU15vucXbuHr9fuZOb5nuaMgLuzYuHQUREmJYe3uI3yz/gDfrD/AloPWunPtG0dwYcdGXNCxEZ2bRmm1NeVyIrLaGJN02uvuTMAiUg9YCvzTGPNRZfsmJSWZ1NRU9wSmvIYxhhcWpTEjZRvJiXEkxISxIzOXlM3pTEpuzS2D2lSYUHdk5PDthgN8vf4AqTsOUWKgcWQIF3RsyJCOjendugHBAf5VxpCVW8j8dftIz84nLiKYYZ3i60QXR1297tpgewIWkUDgC+B/xphnq9pfE7CqzGnJoHM8UaHOJ4NDOQUs2niQb9cfYNmWdHILiqkXHED/dnEM6VB+v3HZ5N8vMY6EmHC2Z+Y4lfy9WV297tpUUQJ2Sx+wWP86M4ENziRfpaoSFRbImF4tzvj9DcKDuKJHM67o0Yy8wmK+35rh6Ko4yJe/7CPAT+jVqgFDOjbigg5Wv/ELi9L4ev1+vrmjv22TQOxohXrCdfsqt7SARaQvkAL8CpxY7Ovvxpj5Fb1HW8DKDiUlhjW7j/DtKf3GbRvWY8+R4yy+a0C5NwAPHM1jyLNLSblnULVa4s6yqxWalVtI8pOLTku+J7j6un2FrS1gY8xyQD+jKI/n5yd0b1Gf7i3qc/fF7dmRkcM36w8wd8VOzm0VU+kkkL5tY3lv5S7G9m5JWKA/fn619yvvzlaoMYaC4hJy84uZt3IX57eJdWryS00+kdRVWgtCqUokxIYzuV9rjhcWVzm5o2WDcB77aiOPfbURgNBAf8KD/QkPDiAsKIDwIH/Cgh1fgwJKt5V9ftJXx/fFJYYZKdvKbYWemII95NmlXNChIf5+fuQUFJGTX0ROfjG5BUXkFBSTm3/y15z8ImtbOfvk5BdRVGZB1L8MOKuK6w7j4NG8M/wJ122agJVygjOTQLZn5HBZ1yac3STqjyRYNvkVFHH0eCH7s46Tk19MTkERufnFFBSXVHpcgIHtGlbaCu3RsgHDnl9e5XHK+yMQHRZEs/oBhAWd+GPxx9d1e7LYlp5T6TE37MtmxfZMtmfk0L9dHP3axhFTL7jKWLyJq/reNQEr5QRnJoF8vzXjjPpCC4pKOF7gSMgFRRzLPzlpf752L+0aR1R6jA7xEQT6C5d1a3pyIg0KICzYn3rBAYQEVL9b5EQfcGXXvWpHJgPbNSRlSwafrNmLCHRpGkX/xDj6t2tI1+bR+Ndid4w7ldf3vjwtg3/N31Arfe+agJVyQlRYIJOSWzNx9qpyJ4FMmrOKScmtz+hGVFCAH0EBfhW2qHILiqtsfe88lMvA9g0Z1rl2181z5rqn9DuLaYPbUlJiWLc3iyWb0lmy6SAvLk7j+UVpRIcFktw2zkrIiXHERXhP69jVfe9unwnnLB0FoTxNTSaB1ERVIxH2Z+Vx4XPuGYFRnes+kltAypYMlmxKZ+nmdDKO5QPQqWkk/RPjGNCuId2aRxPgX3VNMDuG39XmCBDbJ2JUlyZg5alqOgnkTFRnCrar1OS6S0oM6/cdZelmq3X8064jFJcYIkICSG4by4DEhvRvd3qpUXcMvyssLmF/Vh57jhxnr+Ox50geqTsO0aJBGDOv61nhe29+5yf6tomtcgSIrcPQlPIlNZ0EciZuGdQGgCHPLq2wFepqNbluPz+hU9MoOjWN4uaBbcg6Xsh3aRks2XSQpZvTmf/rfsCq0zGgXUMGtIujR8v6vLJka426AIwxHM0rKpNYjzsSbR57Duey90geB7LzOLUdGlsvCH8/qbLvPSEmjPTs/DP6mYC2gJXyKna0vl3NGMPG/dmOroqDpO44TFGJITzInxIDS/5a+eSX92/sQ7Yjye45cpw9h0+0ZK1W7bH8opPeF+TvR3x0CE2jQ2nieDQr/T6EJtGhhAT6M2/lLpanZfDS2O4Vxq4tYKXqEDta364mInSIj6RDfCQ3DTiL7LxCvkvLZOby7USEBFQ5/O7if6ec9Hr9sECaRIfSIiaMPmfFlEm0ITStH0pseLBTo0GcGfmSsjmdx0Z1PrMLRxOwUsrDRIQEcnGnxmw+kF3l5Jf28RH4+8G1fRJoGh1CfFQo4cG1k9ZcOfLlBE3ASimP5Mzkl12HchncoRH9E+NcEoOr+951UU6llEca1imelM3pHKhgmvOJLoDaHvtclogwbXBbUu4eRN82sQQH+NO3TSwp9wxi2uC2NR6BoS1gpZRHckcXQHVicUXfuyZgpZTH8oThd66kw9CUUh7P24ff6TA0pZTX8sXhd6A34ZRSyjaagJVSyiaagJVSyiaagJVSyiYeOwpCRNKBnXbHUU2xQOVTd3yTXnfdUlevG8782lsaY06bruexCdgbiUhqeUNNfJ1ed91SV68bav/atQtCKaVsoglYKaVsogm4dk23OwCb6HXXLXX1uqGWr137gJVSyibaAlZKKZtoAlZKKZtoAq4mEblYRDaJSJqI3FvO9jtEZL2I/CIiC0WkpR1xukJV115mv8tFxIiITwxVcua6RWS049/9NxF5x90xuoITv+stRGSxiPzs+H0fZkectU1E3hCRgyKyroLtIiLPO34uv4hIxat2VsUYow8nH4A/sBVoDQQBa4GOp+wzEAhzfH8T8J7dcbvr2h37RQDLgB+BJLvjdtO/eVvgZ6C+43lDu+N203VPB25yfN8R2GF33LV07f2A7sC6CrYPA74CBOgNrDjTc2kLuHp6AWnGmG3GmALgXeDSsjsYYxYbY3IdT38Emrk5Rlep8todHgGeAMpfR8b7OHPdk4GXjDGHAYwxB90coys4c90GiHR8HwXsdWN8LmOMWQYcqmSXS4E5xvIjEC0iZ7Qukibg6mkK/F7m+W7HaxWZiPWX0hdUee2Oj2LNjTFfujMwF3Pm3zwRSBSR70TkRxG52G3RuY4z1/0gcI2I7AbmA7e4JzTbVTcPVEgLsruIiFwDJAH97Y7FHUTED3gWuM7mUOwQgNUNMQDrE88yEelsjDliZ1BuMAaYZYx5RkT6AG+JSCdjTIndgXkLbQFXzx6geZnnzRyvnURELgDuA0YaY/LdFJurVXXtEUAnYImI7MDqG/vMB27EOfNvvhv4zBhTaIzZDmzGSsjezJnrngi8D2CM+QEIwSpW4+ucygPO0ARcPauAtiLSSkSCgKuAz8ruICLdgNewkq8v9AWeUOm1G2OyjDGxxpgEY0wCVv/3SGOMty/sV+W/OfAJVusXEYnF6pLY5sYYXcGZ694FDAYQkQ5YCTjdrVHa4zNgnGM0RG8gyxiz70wOpF0Q1WCMKRKRqcD/sO4Sv2GM+U1EHgZSjTGfAU8B9YAPRARglzFmpG1B1xInr93nOHnd/wMuFJH1QDHwV2NMpn1R15yT130n8LqI3I51Q+464xgm4M1EZB7WH9RYR//2P4BAAGPMq1j93cOANCAXmHDG5/KBn5dSSnkl7YJQSimbaAJWSimbaAJWSimbaAJWSimbaAJWSimbaAJWPk9EkkVkk91xKHUqTcDKpURkh2NmoG2MMSnGmHZ2xnCCiAxwjC1VShOw8n4i4m93DFBaJ1b/Tymn6S+LsoWI+InIvSKyVUQyReR9EWlQZvsHIrJfRLJEZJmInF1m2ywReUVE5otIDjDQ0dK+y1EgO0tE3hOREMf+J7U6K9vXsf1uEdknIntFZJKjuHybCq5jiYj8U0S+w5oV1VpEJojIBhHJFpFtInKDY99wrOp4TUTkmOPRpKqfhfJdmoCVXW4BLsOqFtcEOAy8VGb7V1gFbRoCPwFzT3n/WOCfWEWAljteGw1cDLQCulB5ZbZy93WUkrwDuABog6PGQxWuBaY4YtkJHASGY9XKnQA8JyLdjTE5wFBgrzGmnuOx14mfhfJRmoCVXW4E7jPG7HZUjHsQuEJEAgCMMW8YY7LLbDtHRKLKvP9TY8x3xpgSY8yJ4u/PG2P2GmMOAZ8DXSs5f0X7jgbeNMb85iis/6AT1zLLsX+RoyLal8aYrY6C3UuBr4HkM/1ZKN+lCVjZpSXwsYgcEZEjwAasQjaNRMRfRB53fCQ/CuxwvKdsqcPfOd3+Mt/nYhVFqkhF+zY55djlnedUJ+0jIkMdhdkPOa5tGJWXaazwZ+HEuZUX0wSs7PI7MNQYE13mEWKM2YPVvXApVjdAFJDgeI+Ueb+rqkjt4+RlpJpXtGN5sYhIMPBf4GmgkTEmGqt6lpy6bxmV/SyUD9MErNwhUERCyjwCgFeBf4pj1WgRiRORE2uORQD5QCYQBjzmxljfByaISAcRCQMeqOb7g4BgrLq4RSIyFLiwzPYDQMwp3SmV/SyUD9MErNxhPnC8zONB4D9Yha2/FpFsrALu5zr2n4N1M2sPsN6xzS2MMV8BzwOLseq9nji3UyubGGOygWlYifwwVmu+bOH6jcA8YJujy6EJlf8slA/TesBKVcKx0sM6INgYU2R3PMq3aAtYqVOIyCgRCRaR+sATwOeafJUraAJW6nQ3YI3l3Yo1GuEme8NRvkq7IJRSyibaAlZKKZtoAlZKKZtoAlZKKZtoAlZKKZtoAlZKKZv8P8zmHHM2Eb4mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5, 4))\n",
    "\n",
    "ax.plot(learning_rates, trn_err, linewidth=1.5, marker='o', markersize=9, mfc='w', label='Training error');\n",
    "ax.plot(learning_rates, val_err, linewidth=1.5, marker='s', markersize=9, mfc='w', label='Validation error');\n",
    "ax.legend(fontsize=12)\n",
    "ax.set_xlabel('Learning rate', fontsize=12)\n",
    "ax.set_ylabel('Error (%)', fontsize=12)\n",
    "\n",
    "fig.tight_layout()\n",
    "# plt.savefig('./figures/CH05_F18_Kunapuli.png', format='png', dpi=300, pad_inches=0)\n",
    "# plt.savefig('./figures/CH05_F18_Kunapuli.pdf', format='pdf', dpi=300, pad_inches=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.2\tEarly Stopping\n",
    "\n",
    "As with AdaBoost, the other important consideration for practical boosting is the number of base learners, which can be hard to specifiy ahead of time. Identifying the least number of base estimators in order to build an effective ensemble is known as early stopping. \n",
    "\n",
    "[Early stopping](https://lightgbm.readthedocs.io/en/latest/Parameters.html#early_stopping_round) with ``LightGBM`` works slightly differently than AdaBoost. In ``LightGBM``, we specify a value for the parameter ``early_stopping_rounds``. As long as the overall score (say accuracy) improves over the last ``early_stopping_rounds``, ``LightGBM`` will continue to train. However, when the score does not improve after ``early_stopping_rounds``, ``LightGBM`` stops.\n",
    "\n",
    "Small values of ``early_stopping_rounds`` make ``LightGBM`` very \"impatient\" and aggressive in that it does not wait too long to see if there is any improvement before stopping learning early. This may lead to underfitting. Large values of ``early_stopping_rounds`` make ``LightGBM`` overly patient and too passive in that it waits for longer periods to see if there is any improvement. This may lead to overfitting.\n",
    "\n",
    "``LightGBM`` needs us to **explicitly specify a validation set as well as [scoring metric](https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric)**. In the experiment below, we split the training data further into a smaller training set and a separate validation set for early stopping. The metric we will use is accuracy score.\n",
    "\n",
    "**Listing 5.6**: Early Stopping with LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] early_stopping_round is set=5, early_stopping=5 will be ignored. Current value: early_stopping_round=5\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LGBMClassifier(early_stopping=5, max_depth=1, n_estimators=50)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LGBMClassifier</label><div class=\"sk-toggleable__content\"><pre>LGBMClassifier(early_stopping=5, max_depth=1, n_estimators=50)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LGBMClassifier(early_stopping=5, max_depth=1, n_estimators=50)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a single split of training and validation sets\n",
    "Xtrn, Xval, ytrn, yval = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)\n",
    "gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=50, max_depth=1, early_stopping=5)\n",
    "gbm.fit(Xtrn, ytrn, eval_set=[(Xval, yval)], eval_metric='auc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also visualize the training and validation errors as well as the ensemble size for different choices of early_stopping_rounds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[LightGBM] [Warning] early_stopping_round is set=2, early_stopping=2 will be ignored. Current value: early_stopping_round=2\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[LightGBM] [Warning] early_stopping_round is set=3, early_stopping=3 will be ignored. Current value: early_stopping_round=3\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[LightGBM] [Warning] early_stopping_round is set=4, early_stopping=4 will be ignored. Current value: early_stopping_round=4\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[LightGBM] [Warning] early_stopping_round is set=5, early_stopping=5 will be ignored. Current value: early_stopping_round=5\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[LightGBM] [Warning] early_stopping_round is set=6, early_stopping=6 will be ignored. Current value: early_stopping_round=6\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[33]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.13718\n",
      "[LightGBM] [Warning] early_stopping_round is set=7, early_stopping=7 will be ignored. Current value: early_stopping_round=7\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[33]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.13718\n",
      "[34]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.135961\n",
      "[LightGBM] [Warning] early_stopping_round is set=8, early_stopping=8 will be ignored. Current value: early_stopping_round=8\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[33]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.13718\n",
      "[34]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.135961\n",
      "[35]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.132684\n",
      "[LightGBM] [Warning] early_stopping_round is set=9, early_stopping=9 will be ignored. Current value: early_stopping_round=9\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[33]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.13718\n",
      "[34]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.135961\n",
      "[35]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.132684\n",
      "[36]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.132215\n"
     ]
    }
   ],
   "source": [
    "number_of_steps = np.arange(1, 10, 1)\n",
    "\n",
    "\n",
    "trn_err = np.zeros((len(number_of_steps), ))\n",
    "val_err = np.zeros((len(number_of_steps), ))\n",
    "n_iters = np.zeros((len(number_of_steps), ))\n",
    "\n",
    "for i, rounds in enumerate(number_of_steps):\n",
    "    gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=50, max_depth=1, early_stopping=rounds)\n",
    "    gbm.fit(Xtrn, ytrn, eval_set=[(Xval, yval)], eval_metric='auc')\n",
    "                     \n",
    "    trn_err[i] = 1 - accuracy_score(ytrn,  gbm.predict(Xtrn))\n",
    "    val_err[i] = 1 - accuracy_score(yval,  gbm.predict(Xval))\n",
    "    n_iters[i] = len(gbm.evals_result_['valid_0']['auc']) # Get the number of estimators in the ensemble in a roundabout way  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAEYCAYAAADMEEeQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAAsTAAALEwEAmpwYAABfaklEQVR4nO3dd3xV9d3A8c83A0IgJCEECJCFLJElBJygorQMZ+vChQqullq1fVqrrQ/6WNtarc/jqBZFwa21SwUnOLBV9t7ISMIMIwNCyPo+f5yTeAnZyb0n9+b7fr3uK/ee8zv3970Yf/ne3/kNUVWMMcYYY0zrEeZ1AMYYY4wxJrAsATTGGGOMaWUsATTGGGOMaWUsATTGGGOMaWUsATTGGGOMaWUivA7AHzp37qxpaWleh2GMaeGWLl26X1UTvY6jMaydM8bUR03tXEgmgGlpaSxZssTrMIwxLZyI7PA6hsayds4YUx81tXN2C9gYY4wxppWxBNAYY4wxppWxBNAYY4wxppWxBNAYY4wxppUJyUkggZRXWMLcNbvJKThGYkxbJgxMIjY60uoLsPLycrKzszly5IjXoZgWJDIyki5dutCxY0evQzHGmCZp7r/HlgA2kqry1PwtvLBgK6P7JpKW0J6vtuznd3PXM3VUL34ypjciYvUFyP79+xER+vXrR1iYdWwb53f46NGj7Ny5E8CSQGNMUPLX32NLABvpqflb+HjdHj655xy6doyqPL43v4gpsxcDcOf5fay+AMnNzSUtLc2SP1NJRIiOjqZHjx7s2rXLEkBjTFDy199j+2vZCHmFJbywYCszJ4847j8GQNeOUcycPIIXFmwl72iJ1RcgZWVlREa2vFvTxnvt2rWjpKTl/K4aY0x9+fPvsfUANsLcNbsZ1TfxhP8YFbp2jGJEeicmPrmAhA5tm1zfgcPHGJHeqUXVN6pvInNX72bSyJQm19dcWtItadNy2O+FMSZYzV2zm1F9as83Gvv32BLARsgpOEZ6Qvtay/TtGsO3+w4T167pvVJ5hcX06xrToupLS4gmp+BYk+syxhhjzHfKy5UNewpYvP0gr32zgwsGdK21fGP/HlsC2AiJMW35asv+WstkHizktnNOapYesjcWZba4+rYfKOTs3p2bXFdLECwzncePH8/VV1/N5MmTm7WsMcYY7xSXlrN6Zx6Ltx9k0baDLNl+kPyiUgDi2kWyed/hWq9v7N9jSwAbYcLAJH43dz1784uq7Zbdk1fEgk05PHLZIKuvBQvETOcOHTpUPi8sLKRt27aEh4cD8Je//IVrr7223u/1wQcf+KWsMcaYhmts58HR4jKWZx5i4baDLN5+kGWZhygqKQfgpMT2TBycxMj0ToxI60RM20hGPTrfL3+PLQFshNjoSKaO6sWU2YtPGJi5N7+IqS8vZuqoXsQ2w+3Y1lCfVwIx0/nw4e++uaWlpfHCCy9wwQUXnFCutLSUiIjQ+t+xus/U0M8Ziv8uxpjg1tDOg7zCEpbsOMgit4dvdXYepeVKmMDJSR2ZNDKF09I7kZHWic7VjOP3199ja1kb6SdjegMw9k9fMKpvImkJ0Ww/UMiCTTmVvwChVF9qp2jW7y5gyfaD3DK6+esLtIqZVVWTP/huZtXYP33B5DPT/JLofv7551x33XX85Cc/4YknnmDs2LE8+eSTXH/99SxcuJDS0lLOOussnnvuOXr27AnAueeey3XXXcfUqVOZNWsWL7zwAqeffjozZ84kLi6OP//5z4wfP77BZbdt28bkyZNZvnw5p512Gv369SMvL49XX3212tjff/99fv3rX7N9+3YGDBjAc889x+DBgwEnyb3jjjt47bXX2LhxI2vXrqVPnz688MILPPjgg6SlpfH555/zyCOP8Pzzz3P06FHGjRvHU089RWxsLNu3byc9Pf248l9++WWz//t7RUSigC+Btjjt7zuq+t8iMgs4B8hzi96oqis8CdIYU6u6Og+OHCtlcM84Fm07wMJtB9m4twBViAwXhvSM45bRvRiZ3onhqfF0jKr774u//v5bAthIIsKd5/dh8hlplV3AZ/fuzCOXDfJLwuB1fTsOHOGzjfu4+4I+LWr9v5o8+N5a1u3Kr/H8vvwiRtYx03lkeicufforutRQZkD3jvz3Rac0OsY9e/Zw8OBBduzYQXl5OYWFhdx00028/fbblJWVcfPNNzNt2jT++c9/Vnv9woULmTx5Mvv372fGjBlMmTKFnTt3Vnvburay11xzDWeddRaffvopixYtYsKECVx88cXV1rl8+XJuvvlm3nvvPTIyMnj11Ve5+OKL2bhxI23bOt9c33jjDebMmUPnzp3Zu3cvAF988QXr168nLCyMWbNmMWvWLD777DO6dOnCDTfcwLRp03jllVcq6/EtH2KOAWNU9bCIRAJfiUjF/fr/UtV3PIzNGFOH+nQejH70M46VlhPdJpxhKfFMGOTc0h2aHEdUZHiD6/TX339LAJsoNjoyoEuheFnfB2v2sLaWpCqYlJQpfeuY6dynSwwbdhf4LYawsDAefPDBysSpXbt2/PCHP6w8f//993PeeefVeH1qaiq33HILAJMnT+ZHP/oRe/fupVu3bvUuW1xczOLFi5k3bx5t2rTh7LPPrjH5A5gxYwa33XYbp512WuV7PfLII3zzzTecc845ANx5550kJycfd9306dNp396ZOf/aa69xzz330KtXLwB+97vfMXDgQF566aVqy4cSVVWgYlxApPtQ7yIyxjTEnNW7OKt351o7D87tl8hJiR24e2xfIsOb70tsc//9twTQ1NvwlHi+3LwfVW3xa6vV1TNXr5nVhwr58Zjefku4ExMTiYr6rhEpLCzk7rvv5sMPP+TQoUMAFBQUUFZWVjlxxJdvohcdHQ0cP+awPmX3799Pp06dKo8BJCcnk5WVVe377Nixg9mzZ/PUU09VHisuLmbXrl3HXV+V77Fdu3aRmppa+To1NZXS0tLK3sKa3iNUiEg4sBToDTyjqgtF5A7gtyLyADAPuFdVT1jXQURuBW4FSElpOWtwGhOqSsvKWbsrn8XbD7Jw20EWbM7h5rPSa72md5cOtI0Ib9bkzx8CFp2IjBORjSKyRUTureb8aBFZJiKlInJ5lXMfikiuiLwfqHjNiYalxrP/8DGyDh71OpQmmzAwiQWbctibX1Tt+YqZVRMGJfkthqpJ9OOPP87GjRtZuHAh+fn5lWPfnE4j/0hKSuLgwYMUFhZWHqsp+QMnMbv//vvJzc2tfBQWFjJp0qTKMtV9OfA91r17d3bs2FH5OjMzk4iICLp27Vpt+VCjqmWqOhToCYwUkYHAr4D+wAigE/DLGq6doaoZqpqRmJgYqJCNaTWKSsr4ZusBnpq3metnLmTIgx9zyTP/5uE569m8t4BTkjrybc6RWt9j+4FCEmOavimDvwUkAXS/8T4DjAcGAJNEZECVYpnAjcDr1bzFH4Hr/Rmjqdvw1HgAlmYe9DiSpvOd6Vw1CfRqpnNBQQHt2rUjLi6OgwcP8uCDD/q9ztTUVDIyMpg+fTrFxcV8/fXXvPfeezWWv+WWW3juuedYuHAhqsqRI0eYM2cOBQX1v1U+adIknnjiCbZt28bhw4e57777uOqqq1rdbF9VzQU+A8ap6m51HANeAkZ6GpwxQSivsIQ3FmXy5LzNvLEok7zCurdHKygq4fON+3j0ww1c8dx/GDz9Y66e8Q2Pf7KJnIJj/HB4T56adCoL7zufz//rPF68cSRff7vf086D5hKoFncksEVVtwKIyJvAJcC6igKqut09V171YlWdJyLnBiJQU7O+XWPo0DaCpTsOcdmpPb0Op8kCPbO6LnfddRfXXHMNnTt3pnv37vzsZz+rcQJIc3rttde48cYbSUhIYOTIkVx11VWUlZVVWzYjI4Pnn3+eadOmsXnzZtq1a8fZZ5/N6NGj613fzTffzK5duxg9ejRFRUV8//vfP+6WcigTkUSgRFVzRaQdMBb4g4gkqepucbo+LwXWeBmnMcGkIcuyHDh8rPJ27uLtB1m3K59yhYgwYWCPWG48K42RaZ3ISIsnLrrNCXWF0jJp4s/bS5WVOLd0x6nqVPf19cBpqjqtmrKzgPerzoZzE8Cfq+qFNdThOzZmuO8tJtN8rnthIQePFDP3p6O8DuU469ev5+STT27UtScs5jkoKSj+5/WXq666iv79+wekBzJQavr9EJGlqpoRqDhEZDAwGwjHuQPztqo+JCLzgURAgBXA7apa6/L/GRkZumTJEj9HbEzL9+S8zXy8bk+1CdnNsxaTmhBNbLs2LNp2oPL2bduIME5NiWNkegKnpXfi1JQ4otvUr0/MN+GsqfOgJQ1jqamdC5l7Lqo6A5gBTsPocTgha1hqPE/P38zhY6V0aBsavz6Bnlnd0ixevJhOnTqRnp7Oxx9/zL/+9S/uvfeEYbrGJSKvUP3M3WNANvBPVV1Z3bWqugo4tZrjY5o1SGNaibqWZXnxRmdZlshwYURaJy4fnszI9HgG9YijTUTjRsEFelk2fwnUX/CdgO+0vp7uMRNkhqfGU66wMiuXs0JkL+DWbs+ePfzgBz/gwIED9OzZk2effZZTTz0hRzHfycMZk/wukIXTtl0EvAmcDPxSRG5X1Ze9C9GY0FdWrjy/YCtnnJRQ67IsY/p34ew+nbn2tNRqyzRWsHceBCoBXAz0EZF0nMTvauCaANVtmtHQ5DgAlu04ZAlgiLjooou46KKLvA4jmPQFJqjqvysOiMgZwEOqOlZExgH/C1gCaEwzOlZaxursvMrxe0u3H6LgWCk/OvekWq/rldieA4eLAxRl8AhIAqiqpSIyDfgIZ+zLi6q6VkQeApao6rsiMgL4BxAPXCQiD6rqKQAisgBniYQOIpINTFHVjwIRuzlebLtI+nbtwNLMQ16HYoxXTgMWVjm2hO9m7n6Ec5fDmFbrhLHVA5OIjW7Y7dEjx0pZlnmIxducSRsrsnI5VurME+3TpQMXDe1OWbmy/UDdy7KcbR0WJwjYIC5VnQvMrXLsAZ/ni6mh0VTVljXjoJUbnhrPnFW7KS9XwsJazkBXYwJkBc6izf+tqkXu/r7TgYpxf+lA8K+VZEwjNGRGblW5hcUs3n6IRdsOsGjbQdbsyqesXAkTOKV7LNednsqItE6MSIsnoYOzzl5eYQmjHp3P3vyiam8DVyzL8shlg/z6uYNRaIziNwF1ako8byzK4tucw/SpYzs1Y0LQZJz1SvNF5CDOws1LgGvd852AH3kUmzGeemr+Fj5et+eESRl784uYMnsxQOV+8nvyili0/SCLth1g8bZDbNzrrCfaJiKMoT3juOOckxiR3olhKXHERFXfexhKy7IEmiWApsEqFoRelnnIEkDT6rhrlp4pIslAd2C3qmb6nLe1WUyrVNeM3JmTRzDm8c/Zsu8wK7JyyTzo7EDUvk04w9M6cdGQJEamJzC4ZyxRkSduf1mTlrama7CwBNA0WK/O7YmLjmTpjkNcNSJ4Z0AZ00THgBwgQkR6AVQsdm9MazR3zW5G9U2sdUbuyLQEPl2/l1F9OnPDGamclp7AyUkxRDRh39xQWZYl0CwBNA0mIgxPiWfpDpsI4m8iwubNm+nduze33347PXr04De/+U2dZRvqtddeY/bs2Xz88cdNDTnkubN8ZwJV93pSnEluxrQ6JWXlrMrOJS0hutZyJyfFMDQ5lp9e0LfZYwj2ZVkCzRJA0yjDUuOZt2EfuYXF1W6XExSeGAR5mbWXiU2Bu1c3uopx48YxcuRIHnrooeOO/+tf/+K2224jOzu73nvgPvfcc42Ow9f27dtJT0+npKSksu5rr72Wa6+9to4rjesZ4H+A2ap61OtgjPHC0eIylmcdYpG7JMuyHbkcLSnjvH5dar1ux0GbkdtSWAJoGmVYijMOcHlmLuf1r/1/+BYrLxOm59VeZnpsk6qYPHky999/Pw8++OBxM99eeeUVrr322nonf61BaWnpCf8e1R1r6Hv4QTzwFw3EPprGNJOmLsuSd7SEpTsOsmibM0t39c48SsoUETi5W0euGpHMwO4deej9dTYjN0g0/qa7adWGJMcSHiZ2G7gOl156KQcOHGDBggWVxw4dOsT777/PDTfcwKJFizjjjDOIi4sjKSmJadOmUVxc/YKlN954I7/+9a8rX//xj38kKSmJ7t278+KLLx5Xds6cOZx66ql07NiR5ORkpk+fXnlu9OjRAMTFxdGhQwe+/vprZs2axdlnn11Z5j//+Q8jRowgNjaWESNG8J///Kfy3LnnnstvfvMbzjrrLGJiYvje977H/v37a/w3eP/99xk6dChxcXGceeaZrFq1qvJcWloaf/jDHxg8eDDt27dny5YtiAgzZ84kJSWFMWPGUF5ezsMPP0xqaipdunThhhtuIC/PSdy3b99+QvkAmAncFIiKjGkqVeXJeZsZ9eh8/r1lP8Wl5Xy1ZT+jHp3Pk/M2U9P3mJyCY8xdvZvp765lwv8tYOhDH3PzrCXM/GorIsKUs3vx0o0jWPHA95j701FMv/gULs9IrpyRuze/6Lj3sxm5LY91P5hGiW4TwYCkji03AfzgXtjT+Fu3x3lpYvXHuw2C8b+v9dJ27dpx5ZVX8vLLL1cmXm+//Tb9+/dnyJAhLF26lCeeeIKMjAyys7MZP348f/7zn7nrrrtqfd8PP/yQxx57jHnz5pGens4tt9xy3Pn27dvz8ssvc8opp7BmzRrGjh3L0KFDufTSS/nyyy9JT08nNze3srds48aNldcePHiQiRMn8uSTTzJp0iT++te/MnHiRLZs2UJCQgIAr7/+Oh988AHJycmMHz+exx57jN///sR/i+XLl3PzzTfz3nvvkZGRwauvvsrFF1/Mxo0badvWWcfrjTfeYM6cOXTu3Jm9e/cC8MUXX7B+/XrCwsKYNWsWs2bN4rPPPqtMAKdNm8Yrr7xSWY9v+QA4HbhTRO4F9vieUNXRgQjAmPqqz7IsPxnTm+xDRytv5y7adpCt+53FldtFhjMsNY67zu/LiPR4Tk2Op12bmoe62ozc4GEJoGm0YSlx/HVpNqVl5U2awRXqJk+ezIUXXsjTTz9NVFQUL7/8MpMnTwZg+PDhleXS0tK47bbb+OKLL+pMAN9++21uuukmBg4cCMD06dN54403Ks+fe+65lc8HDx7MpEmT+OKLL7j00kvrjHfOnDn06dOH66+/HoBJkybx5JNP8t5773HjjTcCcNNNN9G3rzOI+8orr+Tdd9+t9r1mzJjBbbfdxmmnnVb5b/HII4/wzTffcM455wBw5513kpycfNx106dPp3379oAzQeWee+6hV69eAPzud79j4MCBvPTSS9WWD4AX3IcxLVp9lmU594+f89rCHezNPwZAx6gIRqZ34qoRyYxM78TAHrFENqB9txm5wcMSQNNow1Ljmf31DjbsKWBgj6aNlWt2dfTMAfUf33fTnCaFcvbZZ9O5c2f++c9/MmLECBYtWsTf//53ADZt2sQ999zDkiVLKCwspLS09LiksCa7du06rlxq6vGbnC9cuJB7772XNWvWUFxczLFjx7jiiivqFe+uXbtOeL/U1FR27txZ+bpbt26Vz6Ojozl8+HC177Vjxw5mz57NU089VXmsuLiYXbt2Vb6umvxVPVY1ntTUVEpLSyt7C2t6D39R1dkBq8yYJqjPsixnnJRATsExfnxeb0akdaJf15hm2eHJZuS2fNZtYxrNd0FoU7sbbriBl19+mVdffZXvf//7dO3aFYA77riD/v37s3nzZvLz83nkkUdqHJPjKykpiaysrMrXmZnHz2a+5ppruPjii8nKyiIvL4/bb7+98n1r2oapQvfu3dmxY8dxxzIzM+nRo0e9Pquv5ORk7r//fnJzcysfhYWFTJo0qbJMdfH4HqsaT2ZmJhEREZX/hvX5TE0lItf7PL+5podfgzCmgXIKjpGeUHvPeP9uMYwd0JUbzkjj5KSOtr1nK2IJoGm0HnHt6NqxbcsdB9iC3HDDDXz66ac8//zzlbd/AQoKCujYsSMdOnRgw4YNPPvss/V6vyuvvJJZs2axbt06CgsLefDBB487X1BQQKdOnYiKimLRokW8/vrrlecSExMJCwtj69bq1yyeMGECmzZt4vXXX6e0tJS33nqLdevWceGFFzb4c99yyy0899xzLFy4EFXlyJEjzJkzh4KCgnq/x6RJk3jiiSfYtm0bhw8f5r777uOqq64K9AzqST7Pr6/hcV0gAzKmLokxbdl24EitZXYcLCQxpm2AIjItiSWAptFEhGEp8dYDWA9paWmceeaZHDlyhIsvvrjy+GOPPcbrr79OTEwMt9xyC1dddVW93m/8+PHcddddjBkzht69e58w+/XPf/4zDzzwADExMTz00ENceeWVleeio6O5//77Oeuss4iLi+Obb7457tqEhATef/99Hn/8cRISEnj00Ud5//336dy54Wt3ZWRk8PzzzzNt2jTi4+Pp3bs3s2bNatB73HzzzVx//fWMHj2a9PR0oqKijrulHAiqOsHn+Xk1PAIyBdmY+powMIkFm3JOmJFboWJZlgmDqq5pbloDCcWlrDIyMnTJEtuOMxBeWLCVh+esZ9F959OlhnEmgbB+/XpOPvnkhl0UgIWgTctQ0++HiCxV1YyGvp+IxAETcfYC3gXMUdXcJobZINbOmfp4ct5mPl63h5mTR1Q7C/h7A7px5/l9PIzQ+FtN7ZxNAjFNMsxnHOC4gUH2LdISO9MIIjIG+DuwEdgBpADPiMgPVXWep8EZU0XFsivn/vFzzuydQP9uMbYsiwHsFrBpolO6d6RNRJiNAzStydPArap6mqpeqaqnA7fgbBFnTIsiIvxkTG/aRoaRe6SYthHhnN27Mwt+OYY7z+/j9wlUpuWyHkDTJG0jwhnUI5Zlmbleh2JMoHQH/lbl2D+A5z2IxZg6ZR86Sm5hCZcO68n1p6fWfYFpFQLWAygi40Rko4hscVfQr3p+tIgsE5FSEbm8yrnJIrLZfUyueq3x1vDUeFZn53GstMzrUIwJhFeAH1c5dgfwsgexGFOnVdnO1olDeraw9VqNpwKSAIpIOM7tkfHAAGCSiAyoUiwTuBF4vcq1nYD/Bk4DRgL/LSLx/o7Z1N+wlHiKy8pZszPf0zhCcUKTabry8vLmfstTgcdFJFtEFopINvA4cKqIfFnxaO5KjWmsVdm5tAkPo3+3jl6HYlqQQN0CHglsUdWtACLyJnAJsK6igKpud89Vba2/D3yiqgfd858A44A3MC3CsNQ4AJZnHqpcHDrQoqKiOHDgAAkJCTamxQDOF4KSkhL27t3b3NvEPY/d7jVBZEVWLie747WNqRCoBLAHkOXzOhunR6+x156wJYGI3ArcCpCSYtvPBFKXmCiSO7Vj6Y5DTB3lTQw9e/YkOzubnJwcbwIwLVJERASxsbGNWsOwJrYVnAkmZeXKmp15/HB4T69DMS1MyEwCUdUZwAxw1sfyOJxWZ3hKPP/+9gCq6kkPXGRkJOnp6QGv17RO7rZvk/huHcA3gRe1HuMQRCQK+BJoi9MGv6Oq/y0i6e77JABLgetVtdhPH8G0EltzDnOkuIzBPeO8DsW0MIHqD94J+O7W3tM95u9rTYAMT40np+AY2YeOeh2KMX4lIo8Cv8RZC/C/cGYE/xz4Qz3f4hgwRlWHAEOBcSJyunv9E6raGzgETGnm0E0rtNImgJgaBCoBXAz0EZF0EWkDXA28W89rPwK+JyLx7uSP77nHTAtyasp3C0IbE+JuBM5X1WdVda6qPofTLt1Un4vVcdh9Gek+FBgDvOMenw1c2pxBm9ZpVXYuHdpG0Cuxg9ehmBYmIAmgqpYC03ASt/XA26q6VkQeEpGLAURkhDub7grgLyKy1r32IPA/OEnkYuChigkhpuXo3y2G6DbhLLMFoU3oK3AfVY/Vexq8iISLyApgH/AJ8C2Q67aVUMtYZxFZIiJLbLyrqY+VWbkM7NGR8DCbHGeOF7AxgKo6F5hb5dgDPs8X49zere7aF4EX/RpgQwV6H9kWXl9EeBhDk+NYaj2AJvT9L/B3Efk9TqKWjHMr+AkR6VVRqGLVg+qoahkw1N1T+B9A//pUbGOdTUMUl5azfncBN52V5nUopgUKmUkgAZeXCdPzai8zvRnHXARBfcNT4/nz599SWFxKdBv71TIh6//cn+dVOX4+8KT7XIHwut5IVXNF5DPgDCBORCLcXkAb62yabMOefIrLym0CiKmWLQpkms2wlHjKypWVWXUkjsYEMVUNq8ejxuRPRBLdnj9EpB0wFmdozGdAxS5Ik4F/+fmjmBBXMQFksE0AMdWwbhp/+/C+0K7Px6kpcYAzEeSMkxI8i8OYQBCRFJxxetmqmlVXeR9JwGx3h6QwnDHR74vIOuBNEXkYWA7MbPagTauyMiuXhPZt6BnfzutQTAtkCaC/LQvw9qCBrs9HXHQbenfpwFKbCGJCmIgk4azXdwZwAEgQkW+Aq1V1V13Xq+oqnO3kqh7firNrkjHNYlV2LoN7xtruSKZalgD6233ZzfM+9R3fF+j6qhieEs9H6/Z4tiC0MQHwLLASmKCqR0SkPfAI8BxwsaeRGeM6cqyULfsOM35gktehmBbKxgCaZjUsNY7cwhK27j/idSjG+MvZwM9U9QiA+/MXwJmeRmWMjzU78yhXGJJs4/9M9SwBNM1qeKqzILTdBjYh7BAwoMqxfkBu4EMxpnqrKieAxHkbiGmxLAE0zapX5w7Etou0BaFNKHsU+FREfi8id7jrAX7iHjemRViRnUuPuHZ07tDW61BMC2VjABsrNqXucXKxKa2uvrAw4dSUONsSzoQsVX1eRL4FrgEGA7uAa1R1nreRGfOdVdm5dvvX1MoSwMZqrh03gqm+pbPhvTvh9q+g26AaLx2eEs/nG3PIO1pCbLtIPwZpTOCJyBWq+ldgfpXjl6vqOzVcZkzAHDxSTNbBo1x7WqrXoZgWzG4Bm/rrNx4Q2DCn1mIV4wCXWy+gCU01rc83I6BRGFODVdm5gC0AbWpnCaCpvw5dIHlknQngkOQ4wgQbB2hCioj0cvf6DROR9IrX7uMCoMjrGI0BZwKICAzqYQmgqZndAjYN038ifPIA5GZCXPVjDtu3jaB/t44sy8wNbGzG+NcWnD1+Bfi2yrk9wPRAB2RMdVZm5XJSYgdiomwIjqmZ9QCahuk30fm58YNaiw1PjWd55iHKyjUAQRnjfz57/C6oZu/f7qpqt4CN51SVldl5dvvX1MkSQNMwnXtD536w4f1aiw1PjedIcRkb9xQEKDBjAkNVz/F97d4CTvMoHGOOszuviP2HjzHE1v8zdbAE0DRc/4mw/d9wtOYxfpULQttEEBNiROQNETnTfX4TsBZYKyJTvI3MGJsAYurPEkDTcP0ngpbBpo9rLNIz3lmAdLlNBDGh53xgifv8HuACYCRwr2cRGeNamZ1HZLhwclJHr0MxLVzAEkARGSciG0Vki4ic0FCKSFsRecs9v7DiloqItBGRl0RktYisFJFzAxWzqUH3YdChG2yseTawiDA8Nc56AE0oaqOqxSLSA+ikqv9W1bVAV68DM2ZlVi79u3UkKjLc61BMCxeQBFBEwoFngPE4e2hOEpGqe2lOAQ6pam/gCeAP7vFbAFR1EDAWeFxErOfSS2FhzpqAmz+FkppXvhieGs+OA4XsP3wsgMEZ43crRORXwG+AOQBuMpjvaVSm1SsvV1bbBBBTT4FKpEYCW1R1q6oWA28Cl1Qpcwkw233+DnC+iAhOwjgfQFX34Wy4nhGIoE0t+l8IJUdg2xc1FqkYB2jrAZoQMwUYBLQDfu0eOwN4zbOIjAG2HThCwbFSmwBi6iVQCWAPIMvndbZ7rNoyqloK5AEJwErgYhGJEJF0YDiQXLUCEblVRJaIyJKcnBw/fARznPRR0Cam1kWhT+keS2S42G1gE1JU9VtVvUZVJ7tfSlHVd1T1l17HZlq3ygkgtgewqYdguJX6Ik7CuAT4X+A/QFnVQqo6Q1UzVDUjMTExsBG2RhFtoc8FznqA5eXVFomKDGdgj1jrATTGmABYmZVHdJtw+nSJ8ToUEwQClQDu5Pheu57usWrLiEgEEAscUNVSVb1bVYeq6iVAHLDJ/yGbOvW/EI7sg51LaiwyPCWeldl5FJdWnyQaY4xpHiuzcxnYPZbwMPE6FBMEApUALgb6uPtntgGuBt6tUuZdYLL7/HJgvqqqiESLSHsAERkLlKrqugDFbWrT+wIIi6h1UejhqfEUl5azbreNjzfGGH8pKStn3a58mwBi6i0gCaA7pm8a8BGwHnhbVdeKyEMicrFbbCaQICJbcNbWqlgqpguwTETWA78Erg9EzKYe2sVB2ijYMLfGIsMqFoS228AmRIhIgtcxGFPVxj0FHCstZ3BynNehmCAREaiKVHUuMLfKsQd8nhcBV1Rz3Xagn7/jM43UfyLM/TnkbILEviec7toxih5x7Vi24xBTzk73IEBjml2miHwKvAK8665sYIynVmXnATDEegBNPQXDJBDTkvUb7/ys4zaw9QCaEJIGzMO5I7FHRGaIyNnehmRau5VZucRFR5LSKdrrUEyQsATQNE1sT0gaChtruQ2cEsee/CJ25R4NXFzG+Imq5qjqk6o6Amf9v33AKyKy1R3WkupxiKYVWpmdy+CecTjL5xpTN0sATdP1vxCyF0PBnmpPD0/tBNg4QBOSurmPjsC3OOuZLq9uu0tj/OVocRmb9x2227+mQSwBNE3Xf4Lzc+MH1Z9OiqFdZLglgCYkiMgpIvI7EdkBPAtsBoao6lhVnQIMA+6r4dpkEflMRNaJyFoR+al7fLqI7BSRFe5jQsA+kAl6a3flUVauDLYdQEwDBGwSiAlhXQZAfJqzK0jGTSecjgwPY0hyLMttRxATGr4E3gCuUNVFVU+q6nYR+d8ari0Ffqaqy0QkBlgqIp+4555Q1cf8ErEJaSttAohpBOsBNE0n4twG3vYFHCuotsiwlHjW7srnaPEJm7gYE2y6qeq06pK/Cr4rHFQ5vltVl7nPC3CWxaq6LaYxDbIyK5ek2Ci6dIzyOhQTRKwH0DSPfhPg66dhy6dwymUnnB6eGk9pubIqO5fTetkyaia4iMjNVV5XW05VX2zAe6YBpwILgbOAaSJyA862lz9T1RO6zEXkVuBWgJSUlPpWZULcquxcWwDaNJglgKZ5JJ8G0QnOotDVJICnprgLQmcesgTQBKP6LECvOHuX10lEOgB/A+5S1XwReRb4H/c9/gd4HLi56nWqOgOYAZCRkaH1C92EsrzCErYfKOSKjOS6CxvjwxJA0zzCI6DvOFj/PpSVQHjkcac7tW9Dr8T2LNuR6018xjSBqp4HIE7XXzqQ6e5w1GAiEomT/L2mqn9333+vz/nngZoX1jTGx6qduQAMsQkgpoFsDKBpPv0nwrE82P5VtaeHpcSzLPMQqtZxYYKTOr+8q4HyxlzvJpAzgfWq+ief40k+xS4D1jQlTtN6VOwAMshuAZsGsgTQNJ9e50FEuxoXhR6eGs/BI8VsP1AY4MCMaVbLgRP3Payfs3BuJ4+psuTLoyKyWkRWAecBdzdTrCbErcjKpVfn9sS2i6y7sDE+7BawaT5touGkMc44wPGPOrODfQxPdccB7jhEeuf2XkRoTHP4HPhQRGYBWTjj9oC6J4Go6ldAdTNIat5Kx5harMrO5QwbV20aoc4EUESGAROBIUAckAusBD5Q1SX+DM4Eof4TYeMc2L0Sug897lTvxA7EREWwLPMQlw/v6U18xjTdWcA24Jwqx+s9CcSY5rA3v4i9+cdsAWjTKDUmgCLyfeC3QAzwBfBvoMB9fTLwmogUAL9W1Q8DEKsJBn3HgYQ5i0JXSQDDwoRTU+JZZjuCmCBWMSHEGK+tzMoFYEiyjf8zDVdbD+AtwB2qurimAiIyAvglYAmgcbRPgJQznHGAY+4/4fTwlHj+d94m8otK6BhlY1ZMcHMndVTe0lXVRk0OMaYxVmXnER4mnNLdEkDTcDVOAlHVy2tL/twyi1X18uYPywS1fhNg7xo4tP2EU8NT41H97purMcFGRHqIyD9E5ADO1m4lPg9jAmZldi79usYQFRnudSgmCDV4FrCIfE9Efi4iP5CalsM3rVt/dx/7DSeOax+SHEuYOBNBjAlSzwHFwPnAYWAY8C5wu5dBmdZFVVmVnWe3f02jNSgBFJGHgJ8DnYCfAi834NpxIrJRRLaIyL3VnG8rIm+55xe62yQhIpEiMttdImG9iPyqITEbD3TqBV0GOOMAq4iJiqRv1xhLAE0wOxO4WVVX4CwNuBKYAvzM06hMq7LjQCF5R0tsAohptFoTQBG5tMqh0ar6PVW9D7gAuLA+lYhIOPAMMB4YAEwSkQFVik0BDqlqb+AJ4A/u8SuAtqo6CBgO3FaRHJoWrP9EyPwPHDlwwqnhqfGsyMylvNwWhDZBqQzn1i9ArogkAkeAHt6FZFqbldm5ALYHsGm0unoAx4vIuyKS6r5eLyLPichU4CVgUT3rGQlsUdWtqloMvAlcUqXMJcBs9/k7wPnuLWYF2otIBNAO59ZLfj3rNV7pNwG0HDZ/dMKp4anxFBwrZfO+wx4EZkyTLQTccQ58BLwF/B2wZbFMwKzKziMqMoy+XWO8DsUEqVoTQFW9Dfgd8KaI/Ab4Bc4agMPcn1fUs54eOAumVsjmxG/LlWXcPTbzgAScZPAIsBvIBB5T1YNVKxCRW0VkiYgsycnJqWdYxm+6nwox3au9DTws5bsFoY0JQtfjLI0FcBcwH2frtmu8Csi0PiuzcjmleyyR4bahl2mcOn9zVPVrnIVPC4DPgG9V9Ueq+kdVDURP3EicWy7dcTZh/5mI9KomzhmqmqGqGYmJiQEIy9RKxLkNvGUeFB+/9VtqQjQJ7dtYAmiCkqrmVnwJVdWjqvqwqv5SVXd7HZtpHUrLylmzK89u/5omqWsMoIjID4B7gI04t2mnisjbVTYvr8tOINnndU/3WLVl3Nu9scABnG/VH6pqiaruw1mQOqMBdRuv9J8ApUdh6+fHHRYRhqXGsyzTEkATfNwJa78Vka0ikuce+56ITPM6NtM6bN53mKKScobYBBDTBHX1AL6Mk/wlAPcDP1LVK3G2O5orIvWd9bYY6CMi6SLSBrgaZ9kEX+8Ck93nlwPzVVVxbvuOARCR9sDpwIZ61mu8lHo2tI11toarYnhqPNv2H+HgkWIPAjOmSZ4ABgLX8t0+wGuBOzyLyLQqq2wCiGkGdSWAE4HzVPVXOLN+JwK4W7+dAXSsTyXumL5pOAOm1wNvq+paEXlIRC52i80EEkRkC07SWbFUzDNABxFZi5NIvqSqq+r7AY2HItpAn7Gw8UMoLzvuVMU4QNsWzgShy4Br3OEx5QCquhObBWwCZEVWHh2jIkhLaO91KCaI1bYVHDizfB8Ukc9wEsCFFSdUtQj47/pWpKpzgblVjj1Q5f1OmFSiqoerO26CRP+JsOYdyFoEqWdUHh7cM5aIMGFp5iEuGNDVwwCNabBiqrSd7lIwJ655ZIwfrMrOZXDPOMLCbC8G03h19QBeCRzC+ca7A7jT7xGZ0NL7AgiLhA3vH3c4KjKcU3rE2kQQE4z+CswWkXQAdzz00zjLWxnjV0UlZWzcU2C3f02T1bUMTL472/dHqvpnVT0WqMBMiIjqCL3OcZaD0eMXfh6eEs+q7FxKyso9Cs6YRrkP2AasBuKAzcAu4EEPYzKtxLrd+ZSWq+0AYpqsxgRQRP4kIt1qu1hEuonIn5o/LBNS+k+EQ9sg5/i5O8NS4ygqKWf9blvX2wQPVS1W1btVtQPQFYhxX9uMJuN3q7JyAWwPYNNktY0B3AgsEpH1OIuebsRZCzAG6AucC/QDHvZzjCbY9R0P3O3cBu5ycuXh4anfLQht32ZNMFJVW3XeBNTK7Dy6xLSlW8cor0MxQa7GHkBV/QtwEvC8+/OXOONcfgH0Ap4DeqvqCwGI0wSzjknQIwM2HDcHiKTYdnSPjWJZZq43cRljTJBZ6U4AcXZKNabxap0FrKolOFuxvROYcEzI6j8B5j0E+bugY/fKw8NS420pGGOMqYf8ohK25hzhsqG24pBpOttE0ARG/wudnxuP7wUclhLPztyj7M476kFQxjRcTWOj6xozbUxTrcnOA2Bwcpy3gZiQYAmgCYzOfaHTSc5sYB8V4wCX7cj1IChjGmVTDcfXBTQK0+qsdBPAIbYEjGkGlgCawBBxZgNvWwBFeZWHB3TvSFRkmO0LbILJCYOvRKQj7q4gxvjLyqxcUhOiiYtu43UoJgTUmQCKSJiIjHH38DWm8fpPhPIS2PxJ5aHI8DAG94izBaFNiyciWSKSCbQTkUzfB7Ab+Ke3EZpQV7EDiDHNoa6t4FDVchH5l6rGBCIgE8J6joD2ic5t4EGXVx4elhrPzK+2UlRSRlRkuIcBGlOr63B6/+YC1/scV2Cvqm70JCrTKuQUHGNXXhE32+1f00zqewv4SxE53a+RmNAXFg79xjs9gKXfbSozPDWekjJl9c68Wi42xluq+oWqfg50dp9XPL6sb/InIski8pmIrBORtSLyU/d4JxH5REQ2uz/j/flZTPBZlZ0LYD2AptnU2QPo2gF8ICL/ArJwvvECoKoP+CMwE6L6TYRlL8P2Bc4+wcCwlDgAlu04xIi0Th4GZ0zdVLVQRIYCo4DO+IwJrEd7WAr8TFWXiUgMsFREPgFuBOap6u9F5F7gXpy1V40BnAkgYQIDe3T0OhQTIurbA9gOZ3yLAj2BZPfR0z9hmZDV6xyIbH/cotAJHdqSlhBt4wBNUBCRW4F/A2NwkrRBwM+A3nVdq6q7VXWZ+7wAWA/0AC4BZrvFZgOXNnvgJqitzMqlb9cYotvUt9/GmNrV6zdJVW/ydyCmlYhsB73HOOsBTngMwpzvIMNS4/lyUw6qaivcm5buF8A4VV0gIodU9TIRGQ9c3ZA3EZE04FRgIdBVVXe7p/bg7DFc3TW3ArcCpKSkNDJ8E2xUlVXZuYwdUO2vhTGNUu9lYESkj4g8ICJ/cX/28WdgJoT1vxAKdsPu5ZWHhqfGs/9wMVkHbUFo0+J1UdUF7vNyEQlT1Q+Ai+r7BiLSAfgbcJeq5vueU1XFZ5hNlXMzVDVDVTMSExMbGb4JNtmHjnKosMTG/5lmVa8EUEQuApYC/YGDQD9giYhc7MfYTKjq8z2Q8OMWha5YEHpp5kGvojKmvrLd3jtwFoW+RERGAcX1uVhEInGSv9dU9e/u4b0ikuSeTwL2NW/IJpitdCeADLEE0DSj+vYAPgJcoqrXqOqvVPVanDErj9S3IhEZJyIbRWSLO8i56vm2IvKWe35hRQMrIteKyAqfR7k7ANsEq+hOkHrmceMA+3SJoUPbCBsHaILBo8DJ7vOHgFeB+cCDdV0ozviGmcB6Vf2Tz6l3gcnu88nAv5otWhP0VmXn0SYijH7dbDU203zqmwD2BBZUOfYV9ZwEIiLhwDPAeGAAMElEBlQpNgU4pKq9gSeAPwCo6muqOlRVh+KsvbVNVVfUM27TUvW/EHLWw4FvAQgPE05NiWOpbQlnWjhVneXe8sX9GQ/Eq+qz9bj8LJx2bIzPl9oJwO+BsSKyGbjAfW0MACuychmQ1JE2EbZ5l2k+9f1tWoEzy83XPe7x+hgJbFHVrapaDLyJ04Poy3cW3DvA+XLibIBJ7rUm2PWf4Pz0uQ08LCWejXvyOXys1KOgjGk4VS1W1cP1LPuVqoqqDq74Yquqc1X1gKqer6p9VPUCVbWxEAaAsnJlzc482//XNLv6JoB3AFNFZJd7e3YXzky0O+p5fQ+c9QMrZLvHqi2jqqVAHpBQpcxVwBvVVSAit4rIEhFZkpOTU8+wjGfiUqDbIGc2sGt4ajzl6ix3YIwxBr7NOUxhcZlNADHNrl57AQPdcZYruBJ43P15sqqu9294x8VxGlCoqmuqO2+z44JQv4mQ+Q0cdhL2oSlxiGDjAI0xxlXxhXhIsvUAmuZVZwKoquXAv1T1sHv74m33Z0kD6tmJs3B0hZ7usWrLiEgEEAsc8Dl/NTX0/pkg1X8ioLDpQwA6RkXSt0uMJYDGGONamZ1Lh7YR9OrcwetQTIgJ1F7Ai4E+IpIuIm1wkrl3q5TxnQV3OTDfXQ+rohfySmz8X2jpNghiU44fB5gaz/LMQ5SXV7sMmjEtgoj0F5HfiMgzPq8Hex2XCT2rsvMY1COWsDBbIN80r/omgBV7Ac8Skf8RkYcqHvW52B3TNw34CGfro7dVda37HhVrCc4EEkRkC84EE9+lYkYDWaq6tZ7xmmAg4kwG2foZFB8BnHGA+UWlfJtTrzH1xgSciFwBfIkzbvl693AH4E81XmRMIxwrLWP97nwG2+1f4wf13VSwYi9gOH7pl3p306jqXGBulWMP+DwvAq6o4drPgab0QJqWqt8EWPgcfDsfTr6IYSlxgDMOsE9XW/PKtEgPAWNVdaWIXOUeWwkM8TAmE4I27C6gpExtAWjjF3UmgO4aflnAb1X1mP9DMq1K6pkQFecsCn3yRaR3bk9suwj+ujSLfQXHSIxpy4SBScRGR3odqTEVugCr3Ofq89PGLZhmtapiB5DkOE/jMKGpPpNAynCWe2nIpA9j6ic8EvqOg00foGUlPDV/C8WlSlx0G4pLy/lqy35GPTqfJ+dtxh0SaozXlvLdrd8KVwOLPIjFhLAVWXl07tCG7rFRXodiQlB9bwG/AtwO/NmPsZjWqv8EWPUmf//nO3y8qyef/9e5dO34XYO3N7+IKbMXA3Dn+X28itKYCncCH4vIFKC9iHwE9AW+521YJtSsys5lcM84TtwTwZimq+8kkJHA/4nIdhFZICJfVjz8GZxpJU46Hw1vS+Hqd5k5ecRxyR9A145RzJw8ghcWbCXvqHVEG2+p6gagP872lr8GXgIGqepmTwMzIeXwsVK25BxmsO0AYvykvj2Az7sPY5pf2w7s6nQa4w8to3NM22qLdO0Yxai+icxdvZtJI1MCHKAx3xGRG4BPVfXtKscnqaqtVWqaxZqdeahiE0CM39QrAVTV2XWXMqbxNsSN5vycL2HvWug2sNoyaQnR5BTYPCTjuZeA3SJyhap+7XP8L9hi9aaZVEwAsR5A4y+13gIWkServJ5S5fXf/BGUaX3yU86nHDluUeiqth8oJLGGHkJjAqgQmAr8s0qbaAO1TLNZmZVHz/h2JHSwNs/4R11jAG+s8vqPVV6Pbb5QTGs2ZvggVtKXknXvV3t+T14RCzblMGFQUoAjM+YEqqofAqOAn4nIU+5yWTZN3TSbldm5dvvX+FVdCWDVb7T2Ddf4RWx0JEfSv0/kvlXkZG857tze/CKmvryYqaN6EdvO1gM0nhMAVd0EnAGkA59Q/zHVxtTqwOFjZB86ard/jV/V1WBV/UZr33CN35w18QZ4+kleeOFpsvveQGqnaNbvLmDJjoPcMqoXPxnT2+sQjQH4rOKJquaJyEXAIziJoDFNtmpnHgCDrQfQ+FFdCWCEiJzHdz1/VV+H+y0y0+pI5z7QuR//1f5b/tq7MzsOHOHzTfuYela6rf9nWgxVvbjKawV+5T6MabJVWXmIwCDrATR+VFcCuA940ef1gSqv9zV7RKZ16z+BiH8/yaSrY6BdCqt35jFvwz7um3iyLYZqPCMi96vqb93nD9VUznd/c2Maa2V2Lr0TO9ChrY0qMP5T62+XqqYFKA5j4IlBkJfpPP9DGgCvVZx70P0ZmwJ3rw5wYMbQ0+d5smdRmJCnqqzKzuWcvl28DsWEOPt6YVqOvEyYnld7mel2S8QEnqre4fP8Ji9jMaFtV14R+w8XMyTZ2jrjX5YAGmNMA4jIAOCAqu4VkQ7AfwHlwB9VtdDb6EywW5WVC9gEEON/9d0L2BhjjOMNIM59/hgwGjgdZycQY5pkZXYekeHCyUkxXodiQpz1ABpjTMOkqepGcWYl/QAYABwFtnkblgkFK7NyOTmpI20jbJEN418B6wEUkXEislFEtojIvdWcbysib7nnF4pIms+5wSLytYisFZHVIhIVqLiNMaaKIhGJAUYCmaq6HzgGWLtkmqS8XFmzM88WgDYBEZAeQHebpGdwto7LBhaLyLuqus6n2BTgkKr2FpGrgT8AV4lIBPAqcL2qrhSRBKAkEHEbY0w1XgfmAzHA0+6xYVgPoGmirfuPUHCs1Mb/mYAIVA/gSGCLqm5V1WLgTeCSKmUuAWa7z98BzndvsXwPWKWqKwFU9YCqlgUobmOMOY6q3g3cD9yhqhUJYDlwd32uF5EXRWSfiKzxOTZdRHaKyAr3MaH5Izct3arsXACGJsd5GodpHQKVAPYAsnxeZ7vHqi2jqqVAHpAA9AVURD4SkWUi8ovqKhCRW0VkiYgsycnJafYPYIwxFVT1Y1X13RJuiarOr+fls4Bx1Rx/QlWHuo+5zRGnCS4rs3KJbhPOSYkdvA7FtALBMAkkAjgbGAEUAvNEZKmqzvMtpKozgBkAGRkZtmdxMIpNqXOdv5zwLuTtO0zvLtZAmuCkql/6jnE2psLK7DwG9oglPMx2PTL+F6gewJ0cv3p+T/dYtWXccX+xOFvPZQNfqup+d42tuTjjbUyouXu1sxC07+PcXwECdy5nz917GXHkf/lg9W6vIzXGH6aJyCr3FnF8dQXsTkfoKi4tZ93ufIbYBBATIIFKABcDfUQkXUTaAFcD71Yp8y4w2X1+OTDf3WT9I2CQiES7ieE5wDpM6zBsMkgYLHmJbrFRZKTGM8cSQBN6ngVOAoYCu4HHqyukqjNUNUNVMxITEwMYnvG3TXsLKC4ttwkgJmACkgC6Y/qm4SRz64G3VXWtiDwkIhe7xWYCCSKyBbgHuNe99hDwJ5wkcgWwTFXnBCJu0wJ0TIKTL4Tlr0DJUcYPSmLDngK25hz2OjLTiolIpIiMEpGr3NftRaR9Y99PVfeqapmqlgPP40ycM63ISpsAYgIsYOsAqupcVe2rqiep6m/dYw+o6rvu8yJVvUJVe6vqSFXd6nPtq6p6iqoOVNVqJ4GYEDZiKhw9BGv/wYRB3QD4YM0ej4MyrZWIDAI24SRqM93D5wAvNuE9k3xeXgasqamsCU0rs3KJj46kZ3w7r0MxrYRtBWdavrRR0LkvLH6BpNh2DEuJY84quw1sPPMs8ICq9ue7NUm/wJmsVicReQP4GugnItkiMgV41F3kfhVwHvVcUsaEjlXZeQzuGYez+pkx/mcJoGn5RJxewJ1LYecyJgxKYt3ufLbvP+J1ZKZ1OgVncXoABVDVI0C9um5UdZKqJqlqpKr2VNWZqnq9qg5S1cGqerGq2jecVqSwuJRNewtsAogJKEsATXAYcjVERsOSmYwf5Nwtm7vG/kYaT2wHhvseEJGRwBZPojFBb+2ufMoVmwBiAsoSQBMcomJh8JWw+h16tC1iaHIcc202sPHGb4A5IvIg0EZEfgX8Ffi1t2GZYLUyKxeAwcnWA2gCxxJAEzwypkBpEax4nQmDurFmZz6ZBwq9jsq0Mqr6Ps5OHok4Y/9SgR+o6seeBmaC1srsPLrHRtElJsrrUEwrYgmgCR5JgyH5NFg8k/GndAXsNrDxhqouV9UfqepEVb1dVZd6HZMJXquyc+32rwm4YNgKzpjvjJgKf7+F5NxFDOkZy9zVu7n9nJO8jsqEOBF5qD7lVPUBf8diQktuYTE7DhRy1Yjkugsb04ysB9AElwGXQHSC0ws4KIlV2XlkHbTbwMbvkuvx6OlZdCZorcrOA2CI9QCaALMeQBNcItrCsBvg3//HxWdM5/fAB2t2c+to6wU0/qOqN3kdgwlNq9wdQAbZEjAmwKwH0ASf4TeBKt2/fYtBPWKZs9p2BTGBJSJ9ROR+EXnG/dnH65hMcFqRlUevxPZ0jIr0OhTTylgCaIJPfCr0HQdLZzPxlARWZuWSfchuA5vAEJFrgOXAYOAIMAhY5h43pkFWZefa7V/jCUsATXAaMRWO7OOH7ZYD8KHtDWwC52Fggqpepaq/UNWrgQnAIx7HZYLMnrwi9hUcY7Dd/jUesATQBKeTxkB8GonrX+GU7h2ZY4tCm8CJwdnL19c3QHsPYjFBKq+whKc/2wzA3vxj5BWW1HGFMc3LEkATnMLCnIWhM//DdemHWZ6Zy67co15HZVqHPwGPiEgUgIi0A37rHjemVqrKk/M2M+rR+eQUHONH555E1sFCRj06nyfnbUZVvQ7RtBI2C9gEr1Ovg/kPM/HYB/yK7/PBmj1MOTvd66hMCBKRLKDiL7MA3YCfisghIN49thv4nTcRmmDx1PwtfLxuD5/ccw5dO36388fe/CKmzF4MwJ3n25wi43+WAJrgFd0JBv6Qjuv/xvBuE5m7erclgMZfrvM6ABP88gpLeGHB1hOSP4CuHaOYOXkEY//0BZPPTCO2nc0KNv5lCaAJbiOmwsrX+UnyUm5cO4Q9eUV0i7X9NE3zUtUvvI7BBL+5a3Yzqm/iCclfha4doxjVN5G5q3czaWRKgKMzrU3AxgCKyDgR2SgiW0Tk3mrOtxWRt9zzC0UkzT2eJiJHRWSF+3guUDGbINBjGCQN5cxD/wSUD2xvYONnIhIhIteLyJ9EZIbvw+vYTMuWU3CM9ITa5wqlJUSTU3AsQBGZ1iwgCaCIhAPPAOOBAcAkERlQpdgU4JCq9gaeAP7gc+5bVR3qPm4PRMwmSIjAiKm0ObiRyztnMtdmAxv/exW4FygH9lZ5GFOjxJi2bN1/uNYy2w8UkhjTNkARmdYsULeARwJbVHUrgIi8CVwCrPMpcwkw3X3+DvC0iEiA4jPBbOAP4eP7uSVqPuN2pLI3v6jGWyzGNINxQLKqFngdiAku5/XrwoPvra2xjdqTV8SCTTk8ctkgD6IzrU2gbgH3ALJ8Xme7x6oto6qlQB6Q4J5LF5HlIvKFiIyqrgIRuVVElojIkpycnOaN3rRsbaJh6HX0PfgZnTXXFoU2/rYW6OR1ECa4qCoPz1lHWbly40uL2JtfdNz5vflFTH15MVNH9bIJICYggmESyG4gRVUPiMhw4J8icoqq5vsWUtUZwAyAjIwMW0iptcm4GfnmGX4c+2/mrO7F5DPTvI7IhK7rgRdE5GOq3PZV1Ze9Ccm0dE/O28L7q3bzy3H9KClTxv7pC0b1TSQtIZrtBwpZsCmHqaN68ZMxvb0O1bQSgUoAdwLJPq97useqK5MtIhFALHBAnVUxjwGo6lIR+RboCyzxe9QmeHTuDb3O44fZn/Dw9nHsKyiiS4zdBjZ+cSMwCmf9P9/VxxWwBNCcYM6q3Tzx6SZ+MKwHt59zEiLC5DPSmLtmNzkFxzi7d2ceuWyQ9fyZgApUArgY6CMi6TiJ3tVA1Y3T3wUm42yxdDkwX1VVRBKBg6paJiK9gD7A1gDFbYLJiKnEbL2WMbKMj9YM5voz0ryOyISmnwKnqup6rwMxLd/q7Dx+9tcVDEuJ43c/GETF0PbY6Ehb6sV4KiBjAN0xfdOAj4D1wNuqulZEHhKRi91iM4EEEdkC3IMzyw5gNLBKRFbgTA65XVUPBiJuE2T6joOOPbm13We2N7Dxp71AptdBmJZvX34Rt7y8hIT2bfnL9Rm0jQj3OiRjKgVsDKCqzgXmVjn2gM/zIuCKaq77G/A3vwdogl94BGTcSMb8h8nZvpacgmG2nILxhyeAV0XkD8A+3xMVKx0YU1RSxi0vLyG/qIR3bj/T2iLT4gRsIWhjAuLUG9CwSK4J+5SP1tpsYOMXz+AsW/UfYIvPY7OXQZmWQ1X5xTurWJmdxxNXDWVA945eh2TMCSwBNKElpisMuJirIr5k3sptXkdjQpCqhtXwqNf9PRF5UUT2icgan2OdROQTEdns/oz33ycw/vbMZ1t4d+Uu/uv7/fj+Kd28DseYalkCaEKOjJhKB47QNfN9Dhy2LZWMf4hIsoic3ohLZ+EsJu3rXmCeqvYB5vHdGGgTZD5cs5vHPt7EpUO786NzT/I6HGNqZAmgCT0pZ1AU34/rwj/hI1sU2jQzEUkRkX8DG4BP3WOXi8gL9bleVb8Eqk5kuwSY7T6fDVzaPNGaQFqzM4+731rJ0OQ4fv/DwZUzfo1piSwBNKFHhLZn3MrAsO1sXPqZ19GY0PMXYA4QA5S4xz4BxjbhPbuqasXU9T1A1+oK2Y5HLde+AmfGb1x0JDNuGE5UpM34NS2bJYAmJMmQqygOi2bInnc4eKTY63BMaBkJ/F5Vy3EWf0ZV83AWr28yd/H7anczUtUZqpqhqhmJiYnNUZ1pBkUlZdz2ylJyC0t4/oYMW4TeBAVLAE1oahtDQf/LmRj2DZ8vt/V6TbPaCxy3X5eIDKBpawPuFZEk972SqLK8jGm5VJVf/X01yzNzeeKqIQzs0SzfA4zxO0sATcjqdM7ttJUSji223blMs3oMeF9EbgIiRGQS8Bbwhya8Z8VOSLg//9W0EE2gPPvFt/xj+U5+NrYv4wYmeR2OMfVmCaAJWdL1FDJjTuWs3Hc5dLjI63BMiFDVF4H/wlm4PgsnYfuNqr5Wn+tF5A2cLS/7iUi2iEwBfg+MFZHNwAXua9PCfbx2D3/8aCMXDenOtDG9677AmBYkYDuBGOOJjCmkfDaNL7/4O6MnVt1+2pjGUdV/0cheOlWdVMOp8xsfkQm0dbvyueutFQzuEcsfL7cZvyb4WA+gCWnJZ13JAeKIWT277sLG1IOITBKRk93nfUXkCxH5TET6ex2bCYycgmPc8vISOkZFMuOGDJvxa4KSJYAmpElEWzZ0v4whRxeSv/tbr8MxoeFhvlvH73FgMfAF8GfPIjIBc6y0jNtfXcqBI8d4/oYMuna0Gb8mOFkCaEJe/KhbUWDXp894HYoJDYmquldEooCzgfuBh4ChnkZl/K5ixu/SHYd4/IqhDOppM35N8LIE0IS8k/ufzFfhI+mx7R0osckgpslyRKQ3MB5YrKrHgCjABoGFuBlfbuXvy3Zy1wV9mDjYZvya4GYJoAl5IkJ272uIKc+jcOXfvA7HBL//AZYCM4E/uscuAFZ6FpHxu0/X7eX3H25g4qAk7hzTx+twjGkySwBNqzDgrIv4tjyJo/+e4XUoJsip6iwgCeipqp+4h78BrvYsKONXG/bk89M3lzOweyyPXTGEsDDr7DXBzxJA0yoMTenEe5HjSDi0AnZbR41pGlUtBDqISC8R6QV0AKI9Dsv4wYHDx5g6ewnt20bw/A0ZtGtjM35NaAhYAigi40Rko4hsEZF7qznfVkTecs8vFJG0KudTROSwiPw8UDGb0CEilAy6mqPahuJvnvc6HBPE3LZsJ7AH2OLz2OxpYKbZVcz4zSlwZvx2i7UZvyZ0BCQBFJFw4BmcQdMDgEnu3pm+pgCHVLU38AQnbqv0J+ADf8dqQteYU/vxr7IzCVvzDhzN9TocE7yewRkH2F5Vw3we1jUUQlSVX/9jDYu3H+KPVwxhSHKc1yEZ06wC1QM4EtiiqltVtRh4E7ikSplLgIrVet8Bzhd3aXURuRTYBqwNTLgmFJ2aHMcH7SYSUXYUVr7pdTgmeMUDf1HVo14HYvxn5lfb+OvSbH4ypjcXD+nudTjGNLtAbQXXA2fPzArZwGk1lVHVUhHJAxJEpAj4JTAWqPH2r4jcCtwKkJKS0nyRm5ARFib0GnwWK5b0ZtCi5wk/7Taw7ZtMw80EbgJe9DoQ0zzyCkuYu2Y3OQXHSIxpS0zbCB6Zu57xA7tx9wV9vQ7PGL8Ihkkg04EnVPVwbYVUdYaqZqhqRmJiYmAiM0Fn4qAkXi65gPCDW2Dbl16HY4LT6cCzIrJJRL70fXgdmGkYVeXJeZsZ9eh8/r1lP8Wl5Xy5KYef/XUlXTtG8dgVg23GrwlZgeoB3Akk+7zu6R6rrky2iEQAscABnJ7Cy0XkUSAOKBeRIlV92u9Rm5AzLCWeu9ufw+Gy1+iw+AXodY7XIZng84L7MEHuqflb+HjdHj6555zjtnTbm1/EzbMWM/Or7dx5vq35Z0JToBLAxUAfEUnHSfSuBq6pUuZdYDLwNXA5MF9VFRhVUUBEpgOHLfkzjRUWJpw/KJU3lpzL1A1zkPxd0NHG95j6U9XZdZcyLV1eYQkvLNh6QvIH0LVjFC/eOIKxf/qCyWemEdsu0qMojfGfgNwCVtVSYBrwEbAeeFtV14rIQyJysVtsJs6Yvy3APcAJS8UY0xwmDErilZIxoOWw1P6Wm/oRkServJ5S5bVtMxNE/rliJ2f17nxC8leha8coRvVNZO7q3QGOzJjACFQPIKo6F5hb5dgDPs+LgCvqeI/pfgnOtCoZqfEUdUhhbeQIBi6dBaN/DuH2Dd/U6UbgTp/Xf8T54lphbECjMQ2SV1jCkh0HWbT9IIu2HWRlVi63n3NSrdekJUSTU3AsQBEaE1gBSwCNaSnCwoRxA7vx1JJz+Ev4H2HD+3DKZV6HZVq+qrMBbHaAn1SdlTthYBKx0Q37kravoIjF2w6xaNsBFm47yMa9BahCZLgwuGcco/oksnX/kVrfY/uBQs7u3bkpH8WYFssSQNMqTRiUxDVfD6EwpjvRi2daAmjqQ+t4bZpIVXlq/hZeWLCV0X0TSUtoz1db9vO7ueuZOqoXPxnTG6lm6SZVJfvQURZuO8iibQdYvP0Q29zkrl1kOMNT4xk/MImR6Z04NSWOqMhw8gpLGPXofPbmF1V7G3hPXhELNuXwyGWD/P65jfGCJYCmVRqR1olOHdrxUbuJXLb9edi3Abr09zos07JFiMh5fNfzV/W17QTSRLXNyp0yezEAd57fh/JyZUvOYRZuO8jibc4t3T35RQDEtotkRFonJo1MZmR6Aqd070hk+InD3WOjI5k6qhdTZi9m5uQRJ9Q39eXFTB3VyyaAmJBlCaBplcLDhM8iphFzwB3g/eeq65K7YlPg7tXNU+kTgyAvs/YyzVmfF3WGdn37OH7x5wNVXu9rjkpaq7pm5c6cPIIxj33OssxDrMzK5VBhCQBdYtoyMr0Tp6V3YmR6An26dKj32n0/GdMbgLF/+oJRfRNJS4hm+4FCFmzKqexxNCZUWQJoWq2Yot0wPa/2QtNjm6/CvMzA1udFnSFcn6qmNcsbmWrNXbObUX0Ta52VOzI9gVXZuVxwcldGpndiZHonUjpFV3tbuD5EhDvP78PkM9Iqxxye3bszj1w2yHr+TMizBNCYupQWh3Z9XtTpxWc0LVZBUQmLth0krVN0reVOTorh1JS4Zl+cOTY6kkkjbQtR07pYAmhMXR4O8NaCga7Pizq9+IymxThw+BiLtx9k0bZDLNp+gHW78ilXOK9fl1qv23HQZuUa01wsATSmLmN+0zzvM/9/AlufF3W21PpMkzR1WZaduUdZvO2gM2lj+0G27HO2dm8bEcawlHimjenDwO4d+flfV9qsXGMCxBJAY+oy+ufN8z71TVaaqz4v6myp9ZlGacyyLKrK1v1HWOTO0F247SA7c48CEBMVQUZqPD8c1pOR6Z0Y1COWNhHfzdCduqfAZuUaEyCWABpjTAshItuBAqAMKFXVDC/jqc+yLD8+rzfrd+e7t3SdHr79h50xnp07tGFkeiduGZXOiPRO9O/WkfBaZujarFxjAscSQNMq5RWW0MzzbVtUfV7UGer1BdB5qrrf6yDqsyzLuX/8nBlffMvh4jIAesa3Y3TfREamOTN00zu3b9AMXZuVa0zgWAJoWqW5a3YzqZ5lH/94Y5PrW7Mzj5cCWJ8Xdbbk+kzD1WdZljNPSqDgWCnXnpbCiLROdI9r1yx126xcY/zPEkDTKuUUHONIu+60r2ONuJzwLjzz2ZYm16cKh+O70yFA9XlRZ0ut70i77rRvcm0Bo8DHIqLAX1R1hu9JEbkVuBUgJcW/CVJOwTHSE2r/l+ufFEPbiHAuGdrDr7EYY5qfJYCmVUqMacsvkl/jmWuG1Vjmx68v4+zendnaDD0RbyzK5JdbAlefF3W25Prq29vbApytqjtFpAvwiYhsUNUvK066CeEMgIyMDL/uRZwY05avttR+J3r7AVuWxZhgdeIGica0AhMGJrFgUw573f1Dq6pYcmLCoKSgrM+LOkO9vkBQ1Z3uz33AP4CRXsUSiv++xpjvWAJoWiXfjeCr/oHzx5ITga7PizpDvT5/E5H2IhJT8Rz4HrDGq3hC7d/XGHM8uwVsWq1ALznhxRIXof4ZQ2zZkK7AP9xZsxHA66r6oZcBhdi/rzHGh6j6dRjJdxWJjAP+DwgHXlDV31c53xZ4GRgOHACuUtXtIjISd8wLIMB0Vf1HbXVlZGTokiVLmvsjmBB1wi4Hg5L82qsR6Pq8qDNY6hORpV6vtddYgWznvPidNcY0j5rauYAkgCISDmwCxgLZwGJgkqqu8ynzI2Cwqt4uIlcDl6nqVSISDRSraqmIJAErge6qWlpTfZYAGmPqwxJAY0yoq6mdC9QYwJHAFlXdqqrFwJvAJVXKXALMdp+/A5wvIqKqhT7JXhTOMgnGGGOMMaaRApUA9gCyfF5nu8eqLeMmfHlAAoCInCYia4HVwO3V9f6JyK0iskREluTk5PjhIxhjjDHGhIagmAWsqgtV9RRgBPArETlhaXpVnaGqGaqakZiYGPggjTHGGGOCRKASwJ1Ass/rnu6xasuISAQQizMZpJKqrgcOAwP9FqkxxhhjTIgL1CSQCJxJIOfjJHqLgWtUda1PmR8Dg3wmgfxAVa8UkXQgy50Ekgp8jTNZpMYl6kWkAGieDVXrrzMQyA3cQ70+L+oM9fq8qLOl15eqqkF5y8DauZCoz4s6Q70+L+ps6fVV284FZB1AN3mbBnyEswzMi6q6VkQeApao6rvATOAVEdkCHASudi8/G7hXREqAcuBHtSV/ro2BntknIksCWWeo1+dFnaFenxd1hnp9HrN2Lsjr86LOUK/PizqDtb6ALQStqnOBuVWOPeDzvAi4oprrXgFe8XuAxhhjjDGtRFBMAjHGGGOMMc0nVBPAGXUXCfo6Q70+L+oM9fq8qDPU6/OS/f4Ef31e1Bnq9XlRZ1DWF7Ct4IwxxhhjTMsQqj2AxhhjjDGmBpYAGmOMMca0MiGXAIrIOBHZKCJbROTeANT3oojsE5E1/q7LrS9ZRD4TkXUislZEfurn+qJEZJGIrHTre9Cf9fnUGy4iy0Xk/QDVt11EVovIChFZEoD64kTkHRHZICLrReQMP9bVz/1cFY98EbnLX/X51Hu3+zuzRkTeqG4Hn2au76duXWsD8fm8ZO1cs9dn7Zx/6gvpdi7QbZxbZ/O1c6oaMg+cNQa/BXoBbYCVwAA/1zkaGAasCdBnTAKGuc9jcBbY9ttnBATo4D6PBBYCpwfgc94DvA68H6B/1+1A50DU5dY3G5jqPm8DxAWo3nBgD87CoP6spwewDWjnvn4buNGP9Q0E1gDROMtbfQr0DtR/z0A+rJ3zS33WzvmnvpBt5wLdxrl1NGs7F2o9gCOBLaq6VVWLgTeBS/xZoap+ibNwdUCo6m5VXeY+LwDW4/wi+qs+VdXD7stI9+HXmUMi0hOYCLzgz3q8IiKxOH9QZwKoarGq5gao+vOBb1V1RwDqigDauTsBRQO7/FjXycBCVS1U1VLgC+AHfqzPS9bONX991s41s1bSzgWyjYNmbudCLQHsAWT5vM7Gj42G10QkDTgV59uqP+sJF5EVwD7gE1X1a33A/wK/wNn5JVAU+FhElorIrX6uKx3IAV5yb/+8ICLt/VxnhauBN/xdiaruBB4DMoHdQJ6qfuzHKtcAo0QkQUSigQkcv/94KLF2zj/1WDvXvEK6nfOgjYNmbudCLQFsNUSkA/A34C5VzfdnXapapqpDgZ7ASBEZ6K+6RORCYJ+qLvVXHTU4W1WHAeOBH4vIaD/WFYFzO+1ZVT0VOAIEYhxXG+Bi4K8BqCsep1cqHegOtBeR6/xVn6quB/4AfAx8CKwAyvxVnwkMa+eanbVzzVdPQNs4aP52LtQSwJ0cnw33dI+FFBGJxGkUX1PVvweqXrf7/jNgnB+rOQu4WES249zaGiMir/qxPqDy2xyqug/4B85tNn/JBrJ9ehjewWko/W08sExV9wagrguAbaqao6olwN+BM/1ZoarOVNXhqjoaOIQzbiwUWTvnR9bONZtQb+cC3sZB87ZzoZYALgb6iEi6+y3gauBdj2NqViIiOGMq1qvqnwJQX6KIxLnP2wFjgQ3+qk9Vf6WqPVU1Dee/33xV9eu3KhFpLyIxFc+B7+F0tfuFqu4BskSkn3vofGCdv+rzMYkA3P51ZQKni0i0+zt7Ps44Lr8RkS7uzxSccTGv+7M+D1k71/z1WTvXzFpBOxfwNg6at52LaK6gWgJVLRWRacBHOLOAXlTVtf6sU0TeAM4FOotINvDfqjrTj1WeBVwPrHbHqwDcp6pz/VRfEjBbRMJxvjC8raoBWbIggLoC/3D+HyYCeF1VP/RznT8BXnP/gG8FbvJnZW6DPxa4zZ/1VFDVhSLyDrAMKAWW4//tkv4mIglACfDjAA44Dyhr5/zC2jn/CNl2zqM2DpqxnbOt4IwxxhhjWplQuwVsjDHGGGPqYAmgMcYYY0wrYwmgMcYYY0wrYwmgMcYYY0wrYwmgMcYYY0wrYwmgqTcRURHp7XUcTSUia0XkXK/j8BcR+VxEpnodhzHByNq54GDtXNOF1DqAJji5e31uAyLdDa79SlVP8Xcdxhjjy9o509JYD6Cpk4jYF4UmsH8/Y1o++/+0aezfL/hYAhjiRKS7iPxNRHJEZJuI3OkeHykiX4tIrojsFpGn3dXaK65TEfmxiGwGNld5zxEistddNb/i2A9EZGUdsYwUkSUiku9eX7HF05fuz1wROSwiZ4hImIj8WkR2iMg+EXlZRGLd90lz47tVRHa58f/cp57pIvKOiLwlIgUiskxEhvic3y4iF/iUfdt9/wL3tkmGT9lhIrLcPfdX9z0fruNznisi2SLySxHZA7wkIm1F5H/deHe5z9u65W8Uka+qvEflbSgRmSUiz4jIHDeOhSJykk/ZsSKyQUTyRORpQHzO9RaRL9xz+0XkrdpiNyYYWTtn7Zy1cw1nCWAIE5Ew4D1gJdADZ6/Cu0Tk+0AZcDfQGTjDPfejKm9xKXAaMMD3oKouBg7g7CVZ4Xrg5TpC+j/g/1S1I3AS8LZ7fLT7M05VO6jq18CN7uM8oBfQAXi6yvudB/Rx4/hlRWPnugT4K9AJZ6/Ef4qzuXx1LsbZkD0OZ0/VpwHcPxT/AGa57/MGcFkdn7FCN/eaVOBW4H7gdGAoMARnE/Zf1/O9wNkv9EEgHtgC/NaNsTPOJuS/xvlv+S3ONloV/gf42L2uJ/BUA+o0psWzds7aOaydaxxVtUeIPnAatcwqx34FvFRN2buAf/i8VmBMlTIK9Haf/xJ4zX3eCSgEkuqI50uc/7k7Vzme5r53hM+xecCPfF73w9n7MMKnfH+f848CM93n04FvfM6FAbuBUe7r7cAFPmU/9Sk7ADjqPh8N7MTdMtE99hXwcB2f81ygGIjyOfYtMMHn9feB7e7zG4Gvavm3ngW84HNuArDBfX5Dlc8qQDYw1X39Ms7+lD29/n20hz388bB2rvKctXPWzjXoYT2AoS0V6O7e/sgVkVzgPqCriPQVkfdFZI+I5AOP4Hyz8pVVy3u/ClwkzubbVwILVHV3HfFMAfoCG0RksYhcWEvZ7sAOn9c7cBrFrjXEt8O95oRzqlqO01j4nve1x+d5IRAlzniW7sBOdVuYauqsTY6qFvm8ru7z1BRPfWLs4PO+vp9Vq8T4C5zGcpF72+fmBtRpTDCwdg5r57B2rsEsAQxtWcA2VY3zecSo6gTgWWAD0EedWxX34TOmwqXUQFV3Al8DP8C5LfJKXcGo6mZVnQR0Af4AvOM2rNXVswunYa+QApQCe32OJVc5v6u6c+4top5VztfHbqCHiPj+uyTXVLiKqp+pus9TEc8RILrihIh0a2CMvp9VfF+r6h5VvUVVuwO3AX+WEFjiwhgf1s5h7Zy1cw1nCWBoWwQUuIN024lIuIgMFJERQAyQDxwWkf7AHY14/5dxvnkNwhmfUSsRuU5EEt1vqrnu4XIgx/3Zy6f4G8DdIpIuIh1wvrm/pccvn/AbEYkWkVOAmwDfgb/DxRmwHYFz2+cY8E0DP9/XOGOIpolIhIhcgjOmpTHeAH4tIonueJYHcHoXwBm7dIqIDBWRKJzbNfU1x7224rPeiTMuBwARuUJEerovD+E02OWN/AzGtETWzlk7Z+1cI1gCGMJUtQy4EGdA7jZgP/ACEAv8HLgGKACe5/hGpb7+gfNt7x+qWliP8uOAtSJyGGeg9NWqetS99rfAv91bOKcDL+J82/7Sjb0I+EmV9/sCZ6DwPOAxVf3Y59y/gKtwGoPrgR+oaklDPpyqFuN885+C05BfB7yP08g21MPAEmAVsBpY5h5DVTcBDwGf4sxE/KqG96guxv3AFcDvcQas9wH+7VNkBLDQ/Td/F/ipqm5tRPzGtEjWzlk7h7VzjSLH3/Y3pmFE5FvgNlX9NIB1plHLgqoiMh1nYPF1fqh7IfCcqr7U3O9tjGmZrJ0zoch6AE2jicgPcbra53sdi7+IyDki0s29NTIZGAx86HVcxpjAsHbOhCpLAE2jiMjnOAOsf+yOdak4/oE4i5xWfdznWbBN0w9n7Eou8DPgclXdLSL31fA5P/A0WmNMs7F2ztq5UGa3gI0xxhhjWhnrATTGGGOMaWUsATTGGGOMaWUsATTGGGOMaWUsATTGGGOMaWUsATTGGGOMaWX+H+6A1KqYKDv0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 648x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(9, 4))\n",
    "\n",
    "ax[0].plot(number_of_steps, trn_err, linewidth=1.5, marker='o', markersize=9, mfc='w', label='Training error');\n",
    "ax[0].plot(number_of_steps, val_err, linewidth=1.5, marker='s', markersize=9, mfc='w', label='Validation error');\n",
    "ax[0].legend(fontsize=12)\n",
    "ax[0].set_xlabel('early_stopping_rounds', fontsize=12)\n",
    "ax[0].set_ylabel('Error (%)', fontsize=12)\n",
    "ax[0].set_xticks(range(10))\n",
    "\n",
    "ax[1].plot(number_of_steps, n_iters, linewidth=1.5, marker='o', markersize=9, mfc='w');\n",
    "ax[1].set_xlabel('early_stopping_rounds', fontsize=12)\n",
    "ax[1].set_ylabel('Ensemble size at early stopping', fontsize=12)\n",
    "ax[1].set_xticks(range(10))\n",
    "\n",
    "fig.tight_layout()\n",
    "# plt.savefig('./figures/CH05_F20_Kunapuli.png', format='png', dpi=300, pad_inches=0)\n",
    "# plt.savefig('./figures/CH05_F20_Kunapuli.pdf', format='pdf', dpi=300, pad_inches=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.3. Custom Loss Functions\n",
    "The **training loss** is the function that is optimized during training. Recall that we used squared error as the loss while building our own gradient boosting algorithm; ``LightGBM`` uses cross-entropy, otherwise known as the logistic loss. \n",
    "\n",
    "The **focal loss** is a new loss function introduced by [Lin et al.](https://arxiv.org/abs/1708.02002) for object detection. It is a modification of the classical cross-entropy loss that puts more focus on harder-to-classify examples, while ignoring the easier examples. The extent of this \"focus\" is determined by a user-controllable parameter, $\\gamma \\geq 0$.\n",
    "\n",
    "Let's define the model's predicted probability of $y_{pred} = 1$ as $p$. For a binary classification problem, since the only other label is $y=0$, the probability of $y_{pred} = 0$ will be $1-p$. The classical cross-entropy loss (for binary classification) is:\n",
    "\n",
    "\\\\[\n",
    "L(y_{true}, y_{pred}) = - y_{true} \\log{p} -  (1 - y_{true}) \\log{(1 - p)}.\n",
    "\\\\]\n",
    "\n",
    "The focal loss introduces a **modulating factor** to each term:\n",
    "\n",
    "\\\\[\n",
    "L(y_{true}, y_{pred}) = - (1 - y_{true}) p^\\gamma \\log{(1 - p)} -  y_{true} (1 - p)^\\gamma \\log{p}.\n",
    "\\\\]\n",
    "\n",
    "The modulating factor essentially suppresses the contribution of well-classified examples, forcing the algorithm that uses this loss to learn a model that focuses on poorly-classified examples. This effect can be seen below, where the focal loss is plotted for different values of $\\gamma$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEYCAYAAABiECzgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAAsTAAALEwEAmpwYAABYcUlEQVR4nO3dd3yV1f3A8c+5K/dm752QhJCwCXspDlQQFbVW69aqRVtt7XS0tmqX/tqqdbVqxVlX3RNEZMjeOxBCFtl73uTmrvP74wmBkAQSuLmBy3m/XveVe59xznmS8OXkPOc5XyGlRFEURfE+3WA3QFEU5UylArCiKMogUQFYURRlkKgArCiKMkhUAFYURRkkhsFuQG8iIyNlSkrKYDdDURTlpG3ZsqVGShl19PZTNgCnpKSwefPmwW6GoijKSRNCFPW0XQ1BKIqiDBIVgBVFUQaJCsCKoiiDxCtjwEKITOC9IzalAX+QUv7TG/UriuI5DoeDkpISbDbbYDfllGM2m0lMTMRoNPbpeK8EYCllDpAFIITQA6XAx96oW1EUzyopKSEoKIiUlBSEEIPdnFOGlJLa2lpKSkpITU3t0zmDMQQxG8iTUvZ4V1BRlFObzWYjIiJCBd+jCCGIiIjo118GgzEN7VrgnZ52CCEWAAsAkpOT+1xgvdVORZONNocLm91FXKiF1MgAjzRWUZTuVPDtWX+/L14NwEIIEzAfeLCn/VLKl4CXACZNmtTndTLf3niQv3+d0/n5J+cO5b65w0+usYqiKAPM2z3gi4GtUspKTxZqNuoJoQULbVh0Vtra4z1ZvKIoyoDwdgC+jl6GH06GxajnQtffmLkjjF1DbDjaEoDxnq5GURTFo7x2E04IEQBcCHzk6bItJh1hbRMoGfJjwu0XYW9v8nQViqL4qMWLF5OZmUl6ejqPP/64V+v2WgCWUlqllBFSykZPl20x6nHrHDisX6OTAodDBWBFUY7P5XJx9913s2jRIrKzs3nnnXfIzs72Wv2n7GI8/WE26nHpnfgF34y+dRtOZ/NgN0lRfF7KA18OaPmFj1/S677c3FwuvPBCli5dSnp6Og6Hg8mTJ/P555+TlJTU5zo2btxIeno6aWlpAFx77bV8+umnjBw58qTb3xc+EYAtRj1uvROnbTsG6cbltA52kxRFGUDDhg1jwYIFfP3116Snp/Pcc88xf/78zuB79tln09zcvSP2j3/8gwsuuKDzc2lpaZeAnZiYyIYNGwb+Ajr4RgA26XHpHSAEYMDlbh3sJimKMsBGjx7N0qVLqaurY+HChV0C56pVqwaxZX3nGwHYqMeld2HwG4feVaACsKKcATIyMnj++ed55JFH+PWvf01AwOGHr/raA05ISKC4uLjzc0lJCQkJCQPb8CP4RAA2G/U49S7s1sXoiUXKtsFukqL4vGON0XrD0KFD2bp1K42Njfzzn//ssq+vPeDJkyeTm5tLQUEBCQkJvPvuu7z99tsD0Nqe+UQAtpj0OPQSo2UmuvYm3G61SpOi+Dqj0UhwcDCPP/44Ot2JTegyGAw899xzzJkzB5fLxW233caoUaM83NJj1O+1mgaQxajHqQcpXbiFDhftg90kRVG8wOFwcM4555xUGfPmzWPevHkealH/+MSC7GajHrsBXPY9uGQ9DpxI2eelJBRFOQ0VFhYyZMiQ03phIJ/oAet1go3Wm8iyRKFz2VlXdQftTjdmo36wm6YoygBJSUk5bWY79MYnesAAwhCGy34Ah7sSk1PSZncNdpMURVGOySd6wABmk54CowGHTo/dYKLN4SJssBulKIpyDD4TgC0mPR+HxXR+bnOoHrCiKKc2nwnAYQYr6S1fY3G3kRMxlDb7WYPdJEVRlGPymQAcY/iWcVU7ian3g9ht2BwLBrtJiqIox+QzAdhgCET630BJSAj+toVqCEJRlFOe7wRgfQAVAflYbFAe7lKzIBRFOeX5TADWGwL5aMp/AUhul6oHrCjKKc9n5gEbjcGENRkZWRCEXSexqQCsKEof9CUlUUpKCmPGjCErK4tJkyZ5rG6f6QGbTCFk5TkZVmpnpxvaRqgArCjKsR1KSfTNN9+QmJjI5MmTmT9/fo8ZMZYvX05kZKRH6/eZHrCfXyjDmq5DF/E3kmsn0uZwD3aTFEUZILm5uaSkpHDgwAFAW5QnKyury9q+fXFkSiKTydSZkshbfKYH7GcOxSmbkE3voA820mZ3DnaTFOXMsPwxWNnHbMITboH5z3Td9tnPYOvrhz+f8wCc9+Axi/F2SiIhBBdddBFCCO68804WLPDMNFevBWAhRCjwMjAakMBtUsp1nirf38+fVr0Rv6Crsbs/pr1dJeZUFF/mzZREq1evJiEhgaqqKi688EKGDx/OrFmzTrpcbw5BPA0sllIOB8YBez1ZuL/JgNS143YWY3QZabc1erJ4RVFOMRkZGeTk5PSakigrK6vba+nSpV3K6GtKokPboqOjufLKK9m4caNHrsErPWAhRAgwC7gVQEppB+yerMNi1OPGjtu+H5PBhM2h8sIpilec9+BxhwyOaf4z3Ycl+sBbKYmsVitut5ugoCCsVitLlizhD3/4Q7/b2xNvDUGkAtXAq0KIccAW4F4pZZf88UKIBcACgOTk5H5VYDbpaW4fQWxAOs6G3TTjvcR6iqJ430CnJJo3bx4vv/wyNpuNK6+8EgCn08n111/P3LlzPXIN3grABmAC8FMp5QYhxNPAA8DvjzxISvkS8BLApEmT+pXSwmLUY0dib/kSEynY1JNwiuLzBjIl0VdffdX5fseOHSdVR2+8NQZcApRIKQ+Nkn+AFpA9xmLUYxd69KZh6BHqSThF8XG+kJLIKwFYSlkBFAshMjs2zQayPVmHxaTDrtehMw4FFYAVxeeplET981PgLSHETiAL+KsnCzcb9QQZq2hvfAmTsGGx7fdk8YqiKB7ntXnAUsrtgOceoj6KxajHELIDMz9GtuwhQKwHfjhQ1SmKopw0n3kU2WLS49S7cdlzAXC71TQ0RVFObb4TgI166oMc2OR+9sfU4MY22E1SFEU5Jp9ZC8Js1GO11PHmuVoPeGJ12iC3SFEU5dh8pgfsZ9CBNDJpbyihTUakaMfpUiuiKYpy6vKZACyEQCeNTMhv4cblboSwY3OqAKwoyqnLZ4YgAIzuYEToPUinA538B212F4F+PnWJiqL4EJ/pAQMgDDjaVmCjGLfeqdISKYpyXLfddhvR0dGMHj3a63X7VAAWehOvzl7MwnNexKV3qafhFEU5rltvvZXFixcPSt0+FYD1OgvRdSbCmoy4dSo1vaL4Kk+lJAKYNWsW4eHhnm5in/jUAKk0RDB1n47AdklueqDqASuKF/xr+7/4945/9+nYq4ZdxSMzHumy7ZG1j/Bh7oedn3887sf8JOsnxyzHUymJBptPBWCb+VLS6hPRG/xZW9iqArCi+DBvpiQaKD4VgC1GHa2GOtyGVvydfmpNYEXxYRkZGTz//PO9piRSPWAvsxj16EQMwhiO2VmiesCK4gU/yfrJcYcMjuWRGY90G5boC0+kJBpsPnUTzmLSA9rL7HKrAKwoPswTKYkArrvuOqZPn05OTg6JiYksXLjQg608Np/qAZt1NoQ7D0d7ISNMhbTZLxnsJimKMoA8kZLonXfe8VBr+s+nesB+ejtuv0iMlrNo8GtWD2Ioig/zhZREvtUDNofikg0I+350OiNtdudgN0lRlAGiUhKdYgL8LLhxgHRicJlot3W/C6ooinKq8KkAbDEZkAYjer+RGFxG7O2Ng90kRVGUXvlWADbqcbsasLd8icFtxGFXAVhRlFOXbwVgkx5p1GOwzMTgNqkArCjKKc1rN+GEEIVAM+ACnFJKj2dIthj1uHVOcLm0HrCzxdNVKIqieIy3Z0GcJ6WsGajCzUY9DQHtmMpWUpQQgMupbsIpinLq8qlpaBaTnn1pa/litDb97OzGSwe5RYqiKL3z5hiwBJYIIbYIIRb0dIAQYoEQYrMQYnN1dXW/K/A36TG6BFn7QwBwuqwn1WBFUZSB5M0AfJaUcgJwMXC3EGLW0QdIKV+SUk6SUk6KiorqdwUWox7/hgjGlEiu3mSgwRXhgWYriuKriouLOe+88xg5ciSjRo3i6aef9mr9XhuCkFKWdnytEkJ8DEwBvvNkHWajntaSm4hxgSiv4LPY8Z4sXlEUH2MwGHjiiSeYMGECzc3NTJw4kQsvvJCRI0d6pX6v9ICFEAFCiKBD74GLgN2ersdi0tNmDKDWtYTK4ASVkkhRfJSnUhLFxcUxYcIEAIKCghgxYgSlpaUeb29vvNUDjgE+7lg0wwC8LaX0eBY8s0FHjr+Zppj51BkDaHO4kFKe1ot1KMqpbu/wEf063jxyJKkfHU5BdOj8Efv29rmMgUhJVFhYyLZt25g6dWp/LuekeCUASynzgXEDXY9BrwODDpu9hVbhh1sasLvc+Bn0A121oihe5smURC0tLVx11VX885//JDg42NNN7ZVPTUMDyPJfw8TSTbhDWtgVNQqb/SIVgBVlAPWn5+rJ8z2VksjhcHDVVVdxww038L3vfe+E2nKifC4AB5vyGFZ/NvqadnJSVtDmcBGCcbCbpSiKh3kiJZGUkttvv50RI0bwy1/+cgBaeWw+tRYEgEEEUBxmpCIyA6PbodISKYqP8kRKojVr1vDmm2+ybNkysrKyyMrK4quvvvJwS3vncz1goTcidKG4MKNzOdVMCEXxYSebkuiss85CSunBFvWPTwZgvWkYuJ3o3S7VA1YUH+ULKYl8bghCGMzYW77E6TqIzuFWeeEUxUf5Qkoin+sBGwwBmCxZSF0AepdBDUEoinLK8sEAHAiuehASvdOohiAURTll+VwANhqDcDmLQbjQO/WqB6woyinL5wKwvyUUozEItzEGnctEVbNtsJukKIrSI5+7CRcdmYzTmY/LfgDhMlBU3TDYTVIURemRzwXgzLgUdMKM0IcS2hLLwZqmwW6SoihKj3wuAKdHBeMgHqGLwNo4gtw692A3SVEUpUc+F4DjQy20ynycbcspDY6jzmqnsdUx2M1SFEXpxuduwul1gvUjZlBS00pzxyJoBbVWsvxDB7VdiqKcmlJSUggKCkKv12MwGNi8ebPX6va5AAwQH+qiuqKcOkMcAIU1VrKSQge3UYqinLKWL19OZGSk1+v1uSEIgCT9QS5r/y93Wx/gitgHKahR2ZEVxZd4KiXRYPPJHnBocAqhNXOJqc1jy5DV6GoaBrtJiuKznr9rWb+Oj0oO4prfTu52/t0vnN/nMjyZkkgIwUUXXYQQgjvvvJMFCxb063pOhk8G4EgblPvbaQrIoE6/lqCqXLQkzIqi+ApPpSRavXo1CQkJVFVVceGFFzJ8+HBmzZo1EE3uxicD8OiJ6azIfY2cuBqqA3XEl+xWyTkVZYD0p+fqyfM9lZIoISEBgOjoaK688ko2btyoAvDJGD48gbK4fVQHtgHgJ4qos9qJCPQb5JYpiuIpnkhJZLVacbvdBAUFYbVaWbJkCX/4wx8GoLU982oAFkLogc1AqZTy0oGqp6GyjIlbI2kcdxCzA0zGKgprrSoAK4oP8URKosrKSq688koAnE4n119/PXPnzvVkM4/J2z3ge4G9wIDmfY5ISMIQNY5fLxqD076cpfMbyK+2MnFI+EBWqyiKl51sSqK0tDR27NjhwRb1j9emoQkhEoFLgJcHui6X04m5IZSiIXMItMdgNbZSWKumoimKL1Epifrnn8B9QK+LMwghFgghNgshNldXV59wRTqdDqFrQEqJThdOi8vNwWq1KI+i+BJfSEnklQAshLgUqJJSbjnWcVLKl6SUk6SUk6Kiok68Pp2OhEnXIITAZg7HYdVjrco74fIURVEGgrd6wDOB+UKIQuBd4HwhxH8HskLHwW9wO6to9wsjshFcLXsGNf20oijK0bwSgKWUD0opE6WUKcC1wDIp5Y0DWeeosy5A6EOwmcNJKRzJuraxVDW3D2SViqIo/eKTa0EAxKQkgnRj8wvHXB+EXQawr6L7xGxFUZTB4vUALKVcMZBzgA8pzl6Ly56NzRxGdGsdAJsK6ga6WkVRlD7z2R7wlPlXYAyciFvvR7StFYCNKgArinIK8dkA3Fxbg0G3D4BgqWOUYRsHi4uwOVSaekVRTg19DsBCiPOEEKkd7+OEEK8LIV4VQsQOXPNOnBAC/yDtQT+nXwjW6HeZbviOHcUNg9swRVGUDv3pAf8LONR9fAIwoj1U8ZKnG+UJgeERJI3SHlG0+YWTWCsJt+xVwxCKonRx2223ER0dzejRo7tsX7x4MZmZmaSnp/P4448PSN39CcAJUsqDQggDMAdYAPwYmDEgLfOAfaueREo7u1PDqQ4R2C0VbCxUAVhRlMNuvfVWFi9e3GWby+Xi7rvvZtGiRWRnZ/POO++QnZ3t8br7E4CbhBAxwDlAtpSypWO70eOt8pBL7n2QkMvdvDd1BWURgjyzi5KiAzhdKlW9opzOPJmSaNasWYSHd12oa+PGjaSnp5OWlobJZOLaa6/l008/9Ujbj9SfAPwssAl4C3i+Y9tMYJ+nG+Up/sFGLpg0mlaTtg7EXj8Tw1272VOm1oVQlNPZkSmJgB5TEmVlZXV7LV26tE/ll5aWdpYFkJiYSGlpqcevo88BWEr5f8AFwEwp5bsdm0uBOzzeKg/Zs3Ipjqp6sspNzNniJsAK8ZZdahxYUTxo7ftvsfb9twB45ecLqCsrpTL/AG8+cC8AK954mc2ffwTAC3fdTEtdLcV7dvLeow8AsOSlZ9m5VBsCeOaWq7G3tfap3tGjR5OTk9OZkuj+++/v3Ldq1Sq2b9/e7XV0NozB1q/1gKWU+w+9F0KcB7illCs93ioPOfu6W9jwWT6zs29g0uZXabK40cccZENBHT+alTbYzVMUnzDj6hs639/2z8P35G96/GkAzr35cB/trhfeALSb5D8YNRaAixb8tHP/z15/v8/1eiolUU8SEhK6DGeUlJR0pi7ypD4HYCHESuC3Uso1Qoj7gV8CTiHE81LKv3q8ZR5Qsm8PO75Zg8M/i1Wjw6gLrsPP3Ep2QQku90T0utN3HVFFOdN5IiVRbyZPnkxubi4FBQUkJCTw7rvv8vbbb59UmT3pzxjwaGB9x/sfAecB04C7PN0oT7EEBjF8Rjqjr4ngP3NayUkU7DSbSLNns0nNhlCU05onUhIBXHfddUyfPp2cnBwSExNZuHAhBoOB5557jjlz5jBixAiuueYaRo0a5cHWa/ozBKEDpBBiKCCklNkAQogwj7fKQyISk5l1XTJu6cZQ5MQB1On1XOS3gy92ljEtLWKwm6goykk42ZREAO+8806P2+fNm8e8efNOquzj6c9/G6uB54B/AB8DdATjmgFol0c01VTx6i/uQid0THDGMyXHzagaHbtFDIt3V6jpaIpyGjvTUhLdCjQAO4FHOrYNB572aIs8KCg8kh88+gR7VpVy+baL+fVHbsI3Xsq61jnUtNjVbAhFOY35QkqiPg9BSClrgd8ete1Lj7fIg4ROR3luNiverkfIkSTpjGTWF/NVqrb/i13lzEiPHNxGKopyxurPYjxGIcSjQoh8IYSt4+ujQgjTQDbwZBVs20RIpEAisAbEk1l/sHOfGoZQFGUw9WcI4m9oD2LcBYzr+Ho+8H8D0C6PueCOnxCTGg9Ac3AySc1VBNpbEbhpsVpZn6+GIRSlv1R+xZ719/vSn1kQVwPjOoYiAHKEEFuBHcAv+lWrF+38djFCSsAPW8o4dKWruLn9H+wd1kRqyXl8sTONs4apYQhF6Suz2UxtbS0RERGn9Q0wT5NSUltbi9ls7vM5/QnAvX2nT+mfQEzaMIxmBzmbymgJTQEguLKZzVMMzLds4A87L+ehS0cS6NevhwIV5YyVmJhISUkJ1dXVg92UU47ZbCYxMbHPx/cn6rwPfC6EeBQ4CAwBHgL+168Well0ShrmwBagjCa7PxJBVr5ESEllQDXmymo+3FLCLTNSBrupinJaMBqNpKamDnYzfEJ/xoDvA5airYS2BW11tOWAfQDa5TF5mzew+p0X8A8x4XBImhPSCGmFtHJYHmBmvn4tr68txO1WY1qKonhXf1ZDs0sp/yClTJdS+ksphwF/AX51vHOFEGYhxEYhxA4hxJ6OXrRXDJ04hUt+9hsiE4MAkBMvBGBCnptsPz8uNC0jv6aFlbnqzylFUbzrZJNySvo2BtwOnC+lHAdkAXOFENNOsu4+ETodO75ZRHi8BQBH0hgAxudpPd6coBZm6Pbw2ppCbzRHURSlkyeyIh/3b3epOTKDhrEv53lKU00VYTFa4o5GwnAbDaSXQ4hV8kFQIDfql7ByfzUHqlqOU5KiKIrnHDcACyHO7+2FtiJanwgh9EKI7UAV8I2UckMPxywQQmwWQmz25B3Ws6+7hbj0aABqy1vxnzIFgOHFkkKTkSjLbmKoY+HqAo/VqSiKcjx9mQWx8Dj7Dx5nPwBSSheQJYQIBT4WQoyWUu4+6piX6MiyPGnSJI/1kHcuXYzL6cRgCsRa3074vb/hpStfYUPtIgA+CfbnWutynt8cwV3npDEkIuA4JSqKopy84wZgKaVH55tIKRuEEMuBucDu4x3vCakTJqHT6YnL0BEWF4CfxcC8+Ft440stAC8J8Ocd0zKea7uCJ7/Zz9PXjvdGsxRFOcN5Ygz4uIQQUR09X4QQFuBCvJjM0z84lLbmJmLTQvCzaP/njIwYyfDQDPxtkni74I/yCtwIPttRxt5ylbRTUZSB55UADMQBy4UQO9EyK38jpfzCS3XjaLex9OV/dX6WUmLbvYdHnq3j1e8ysLc+x3e2C5HokBKeWJLjraYpinIG80oAllLulFKOl1KOlVKOllL+0Rv1HmIOCOTaR7U1g759PZvXHliDMywGXUUNxtJ6fnVe1wSdS/dWsVmlLFIUZYB5qwc86HYtW0JlQR7NdTZaG+1U10DKe++SvvQbZo9LYkJyaJfjH/pkt1qqUlGUAXXGrEATGBaOyWxmxveiMPrpCY3xR4iozv33zR3OtS+t5lzDBnAHsKIii1fXFKr09YqiDJgzJgCnjp+ElLLH5fOstZUUF75K+rC3iLZZubPSxGr7aJ78Zj/zxsaREGoZhBYriuLrzpghiNKcvXzwl993217/7nsUnXMBea+/QaUBvg7wx2iq5mb9N7Q5XDzy2Z5BaK2iKGeCMyYAxw4dxpX3PwzAnlWlvP/YJvK3V+OXmYFwOrlotx6jQ+IWgldCgvm54QMiaOSb7Eo+3V46yK1XFMUXnTEBWG8wcHD3duxtrbQ22akqaqY0px5LVhbmUaOwWJ3M2qM9fPdpUAANRge/MbwHwEMf76a4rnUwm68oig86YwIwwMHdO7FZW0jICAWgJKceIQTht94CwPe2+oGUOIXgn+GhXKNfyViRR3O7k1+8t13NilAUxaPOqAB87k23ExwZTUxKCEY/PXVlVppq2gieMwdDdDRRlW2MKdR6wd8E+LPTbORPxlfR42JzUT3PL88b5CtQFMWXnFEBOH/rJrZ8+Sl6o44hoyMAKNhRgzCZCLv+egB+uPtwgs6/h4cxVpfPAv2XADz97X6+268WblcUxTPOqAAcmZxCyrgJAKRmaYE2f7sWUEN/cA3Cz4/E3VUk1+kB2Gn24+sAf35u+IAMUYxbwk/f2UZRrXVwLkBRFJ9yRgXg4MgojGYzbpeLIaMj0ekF5QcaaGuxYwgLI+TyywG4d9+QznOeCA/FqXPxc8OHADS2OfjRG5tpaXcOyjUoiuI7zqgADLD4+Sdpqq7Cz2IgMTMMKaFwZy0AEbffBno9SavzyGgJBsAlBMvi5/Frx12dZeyvbOHn725TN+UURTkpZ1wAvubhxwiNjQMgNUt7FLlghzYMYRoyhJArLgeXi/t3p3BNxjV8evmnXHrHW8we133Bnt9/uhspVTZlRVFOzBkXgGtLi9n57dcApI7VxoGLs+tw2F0ARP74J2A0ErRiG/dFXUdQxDCEEPztqrGMjAvuUtY7G4t5ammudy9AURSfccYFYKPJDz9/fwACQv2ISQ3G6XBTvEdbftKUmEDo968iYMaMLudZTHoW3jqJ+GA/5unWo0cL2M98m8tra1QuOUVR+u+MWYznkOCoaIIio5BuN0KnI3VcJJUFTeRtryJtvDYkEfvb3yKMxm7n7qpawhOJrzE9fxmvOOfyR+fNADzyeTZ6neCm6SnevBRFUU5zZ1wPGODzJx+jaPcOAIZOiCZ1XCQZk2M79x8ZfKWUWO1WHl77ML9a+xCP2vfRIgS3GRZzg35p53G//3QPb64r9No1KIpy+hOn6k2kSZMmyc2bNw9I2XZbG0Y/c49LUx6p/cABKv78F+Rls7m+/V80O5oBuLTFymPVtTiljlsc97PGPabznIcuGcEdZ6s1hBVFOUwIsUVKOeno7WdkD9jtcpG/deNxj2vdto3W9etxv/EBv5/2UOf2LwID+CQwAINw82/j02SKg537/vzlXv5v8T41O0JRlOM6IwOwy+GgeM/OLttKcupZ8vJuGqoOr3oW+r3vEfWLX5D82qtcnDaP+UPnd+77U2Q4e0wmgkUrb5oeJ1lUdu7794o8HvhwFw41T1hRlGM4IwNwQGgY5978oy7b9q0rJ3dzFTnrKzq3Cb2eyDsXYAgLA+B3U39Hemg6AHYhuDc2mlqdjmjRwFumx4imvvPc9zYXc9trm2hsc3jhihRFOR15JQALIZKEEMuFENlCiD1CiHu9Ue+xbPz0A/avX935efQ5CUy5LJURM+N6PN5tt9P+wac8NesJgoxBAFTqdfwqJhoHkCSqeMf8GBE0dp6zKreG7/1rjVo7QlGUHnmrB+wEfiWlHAlMA+4WQoz0Ut09GjZ1BokjD988i00NYfIlqQRH9Jz/reSuH1Px6B8J/nw1j896HIF2A2+L2cQjUZFIYCglPOv/cpfz8qqtXP78GlaqVdQURTmKVwKwlLJcSrm1430zsBdI8EbdvQmLjaepugrp7j5O29MNtLAbbwCg6smnmOZO4afjf9q577NAf14MDYGQJDJve5HxR6W4b2h1cOurG3n221zcbnVzTlEUjdfHgIUQKcB4YEMP+xYIITYLITZXVw98j3H9R+9ibajvsm3XihL++/t1NFZ3TUEUdP75BF96KdJmo/y3v+P2kT/kivQrAAg3h3PWjAfgls+ISBzGOz+axvxx8V3OlxKe+GY/t7++idqW9gG9LkVRTg9enQcshAgEVgJ/kVJ+dKxjB3Ie8LF8+1o2+9ZXMPa8RM7+QUaXfc76evIvm4+rpoaon99LyILbeXzD49w6+laSgpK6HCul5N8r8/j319tpll2HNaKD/PjnD7KYkR6Joii+b9DnAQshjMCHwFvHC77e0lhVyYo3/tNl27gLtECavbac9tauMxgMYWHEP/YYANXPPItj205+P/333YIvgBCCnyQWsTX418yx7Ouyr6q5nRsWbuCxr/Zic7g8eUmKopxGvDULQgALgb1Syie9UWdf+IeEkDwmq8u2yMQgEoeH4Wx3sWd1WbdzAs8+i4gf3QFuN6W/+jXO+vpux2TXZrN+15vw3k0Y2+t5QfyV+6LWdzlGSnjxu3zmP7ea3aWN3cpQFMX3easHPBO4CThfCLG94zXPS3X3yuhnJnnUOFqbugbAcbO1Hu2u5SW4eniYIupnP8OSlYWzooLyBx7sciNve9V27vj6Du7Z/hSrg0IAEG4nP2l+hg9SP0cvupa3v7KFK55fw9+/3qd6w4pyhvHWLIjVUkohpRwrpczqeH3ljbqPZ+fSRexe/k2XbUNGRRAW609LfTsHNlV2O0cYjSQ88Q90ISG0rFxJ9TPPAOByu3h47cM0O5ppdzv4aaiZb+OHd543qfwdtqS8wPBge5fynG7J88vzmPf0Ktbn1w7AVSqKcio6I5+EO9KEeZcz5fLvd9kmdILxFyUDsOnLwh57wcaEBBKfehJ0OmpfeJGmRYvQ6/Q8d/5zxAdoMyCc0smvzO18Pmxm53mh5av5yvx77h3R/eGM/Bor1760nl/+bzvVzWqmhKL4ujM+AANs+fJTGiorumzLnBpLSLSFxuq2Lo8nHylgxgxi7r8PgIpH/4jbaiUpOInXL36dIcFaYk+XdPFbZzELsy7j0HwTXVMxvzh4D4tm5BAdaOpW7kdbSzn/iRW8vrZQ5Z1TFB+mAjAQFBmJTq/vsk2n1zHlslQANn1ZgMvRcyAMu/lmIn50B0kvv4wuIACA2IBYXpv7Wue6EQD/bNzBY9OuxeWnjQvjamfE1kdZPfITrp+a3K3cZpuThz/bw8VPr+I79RSdovgkFYCBjKkzMVm6P4I8bGIM4fEBtNS1k72m+4wI0KabRf/qV1hGj+rcJqUk0hLJ6xe/zqSYw1P/3qlcy8+yZtMSM7pzm2nUpfz1yjH8787pZMQEdis/t6qFm1/ZyK2vbmRfRdPJXKaiKKcYFYDRAuZ7D99Pc21Nl+1CJ5h6mba4+uavCjsTdx5L4+dfcPC223C3txNsCubFC19kTsqczv3fVW3m5vho7JPvgKk/hsyLAZiSGs6XPzubhy4ZQYBJ363cFTnVXPz0Kn79/g5KG9pO5nIVRTlFqACM1ou96W/PEBTR/cm01KxIopKDMJh0NNfYjlmOu62NqiefpHXdepqXLAHApDfxt1l/47bRt3Ued0napZgueQLmPtblfKNexx3xhay6PoDvT0zsVr6U8MGWEs77+woe/nQ3VU3Hbo+iKKe2MzIlUU8c7TbWf/QeZ117c7dURU01bfiHmDAYu/dMj2bbvx/r6jVE3PbDbvs+z/ucLZVbeHj6wz2nQ2qrh+enQksVTFnA7hH38sevD7KxsK7HuvwMOm6YOoQ7z0kjJtjctwtVFMXrensUWQXgDlJKtn/9BWMvmIve0D0j8oly1tSgDw1FGHpPQF3aUorFYCF86Z9h88LDO4ITkRc/zrfuyfzf1znkVrX0eL7JoOPayUksmJVGYpi/x9quKIpnqADcR+2tVvz8A3rcZ7M62Ph5AanjIkkaEX7csuwlJRy85VbMY8aQ8Lf/Q5i6TzlzuBzcuOhGqlur+ev4XzBtw+twoOuDIWTMxXnRY3xUaOTppbm9jgEbdIL5WfHcdc5QMmKCjn+xiqJ4xaAvxnM6aKis4H+P/rbX/XvXlrNrRQlrPzrQp6SbrpoaXI2NNC9eTMlPf4bb1n3M9pltz5Bdm011WzUL1v6OJ4fPwH75v8D/iPHo/Ysx/Hsa1zS+xvKfTeZPV4wmtochB6db8tHWUi566jtufXUjaw7UqOSginIKUz3go7hdrm5zgg9xOdx8+3o2E+elEBHffcpYT9r27KH49jtwNTTgP2UKic8/hz7ocO90delqfrf6d9TZDo/zDgsbxl8n3c/wLe/Alle7FhgUD7P/QPuo7/P+ljL+vSLvmLMihscG8cOZKVyelYC5D2PYiqJ4nhqC6KPm2hp2LVvCjKuv91iZ7bm5HLztdpzV1fhlZpL00osYY2I699e01fC71b9jbdnazm0GYWDBuAXcETYe4+IHoWzr4QKD4uCnW8AUgN3p5pNtpbywMo/8mt5zz4X5G7l2SjLXT0kmKVyNEyuKN6kA3EcOezv7161m1Dmzj3mclJKCHTUMGRWB3nj8kRx7SQnFP1qAvaAAQ2wsSS++iDnz8ILvbunm3X3v8tSWp7C5Dg9VpIem8+i0hxlbthuWPgrWKrj8eRh/Y5fyXW7Jkj0VvLQqn20HG3pthxAwe3g0N0wdwqyMKPS6HmZjKIriUSoA94OUksaqSkJjYns9ZvX7uez4tpgJc4Yw/cqhfSrXWV9Pyd330LZ1K7qAAOL/8XeCzjuvyzFFTUX8bvXv2FG9o3ObQHDr6Fv55egfwdY3YeqdoDtqOOG7v0PmPIgZxZaiOv7zXQFLsis4Vgq6+BAzP5iczNWTEokP7TkZqaIoJ0/dhOuH5toaFv/rqWPewBo6PgoEbF1SRPG+nufpHs0QFkbyKwsJungubquVkp/cTc1L/+lSz5DgIbw+93Xum3wfFoMWFCWSYFMw+AXB9J90D76Fq2HZn+HfM+B/tzDRXMELN01k5W/OY8GsNILNPU+BK2u08dTS/cz8v2Xc/MpGvtxZrtYkVhQvUj3gXkgpe35Y4ggbP89n05eFWIJN/OB3kwkI8etz2bUvvED109o6wsHzLibuT3/qXMznkNKWUv607k+UWcv48LIPMep7mZ+8cA4Ud824wfBL4exfQcIEWu1OPttexhvrisguP/Z6EsFmA5eNi+eqiYmMTwo97vdAUZTjU0MQ/WRtqGfFGy8z755fIXQ9/6Hgdks+e3obpTkNJGSGMv/e8ej6Maba/O23lP3mPkSAP2kffYQhKqrbMVJK6tvrCTd3nXecW5/Lf/f+l7uz7ia6sRxWPA45Paxxn3YuzLwX0s5DAtuKG3hnw0E+31mGrZcV3g4ZEuHPFVkJXJ4VT1pU32Z9KIrSnQrA/SSl5ODuHSSPHnfMXqC1sZ33/rKJtiY7Ey8ewrTL+zYefEh7fj6uxkb8x4/X6nW5QKc7Zp1SSn70zY/YUL4Bs97MzaNu5rbRtxFQvR+++wfs+6L7SbFjYPpPYdSVYDDR2Obgs+2lvLe5mN2lx19lbXRCMPPHxXPJ2HgS1HixovSLCsAnQLrdHNi8nvRJ03rtBQOU7Kvjs6e3IyVcePtIMib3fvPueKqefAr7wYPEPfoI+pCQHo/ZXrWdmxbd1GVbuDmcO8bcwTWZ1+BXcwBWPQl7PgJ5RC9X6OBn2yFsSJdz95Q18sGWEj7dXkadtWu6pJ5MSA5l3pg4Lh4Tp4KxovSBCsAnQErJtwv/xYyrb8A/JPSYx+5YVszq/+WiN+i44pfjiU3rOXgei7O+nryL5uC2Whny3zfxnzCh12PXlq7lyS1PklOf02V7jH8Md467kyuGXoGxqRTWPa/NnHC2wcjL4Zo3jqrUDgbtEWmHy82KnGo+3lbC0r1V2J3Hz8YxLimUi0fHMmdULKmRPT/CrShnOhWAT4LL6UDodOiOnn1wBCklK9/Zz57vSrEEm/j+/RMJjuh/79B+8CCtmzYRetVVndvcdju6HtaRcLldfFnwJc9te45ya3mXfXEBcdwx5g6uSL8Ck60ZtrwCaedD4sSuhSz+LRxcC5Nug9FXgUkLoo1tDhbtKuezHWWsy6+lL78mGTGBzBkVywUjYhiTENKv8XBF8WWDGoCFEK8AlwJVUsrRxzseTq0A/PlTjzN29lyGjM065nEul5svnt1Byb56QmP8ufJXE/AP7h44+6N5xQoq//RnYh95hMCzz+rxGLvLzvv73+elnS91eaQZ4JqMa/j99N/3XLijDZ4YDrYG7bNfMIy5GibeAnHjOg+rbLLxxc5yvtxZxtZjPORxpJhgP2aPiGH28Ghmpkeqx6CVM9pgB+BZQAvwxukYgO22NkzmvvVm21sdfPzkNmpLWjjvpuGMnBl/UnUX3/VjWlasACD4kkuIefABDJHdF44HaHW08r+c//Hqnlc7A/EHl31AZnhmz4UXrYU3rgBXDxmYY8dqT9uNuRr8D8/AKG1oY9GuchbtrmBLUX2frsFs1DFjaCTnZUZxbma0ehRaOeMM+hCEECIF+OJ0DMAAhTu3UVtcxMRLrjjusa1Ndg5m1zJ8WtxJ1ysdDupef53q555H2mzogoOJ+tnPCLv2B72uMdzqaOX9/e+T35jPozMe7bKvyd7EM1uf4drMa0kPSwdrLex4Gza/CnV53QvTGSFjDmRdD8Mv6bKrotHG13sq+HpPBRsK6nAd67G7IwyNCuDczGjOyYhiSmq46h0rPu+0CMBCiAXAAoDk5OSJRUVFXmlbXzTVVGNraSY6Ja3f5zZUteIfZMJk6X1R9uOxFxdT8cc/YV21CgC/jAxiHvodAVOm9KucV3a/wlNbngJgWtw0bhhxA2cnnI1e6KBoDWx9A7I/BedRS2cmTYPbv+613HqrnWX7qli6t5KV+6tp7UP+PNCyekxNi2DWsEjOGhZJZkyQevhD8TmnRQA+0qnWAwZw2u3krFt13IV6jtRY3crH/9hKYLiZy346Dj//E8+2IaWk5dtvqXzscRylpQAEXjCb6F/9Cr/U1OOe73A7mPvBXKraqrpsTwhM4OqMq7ly2JXaAx9tDdoUtm1vQWnHz+DSp7QbdUfavwT8IyBhgrbKTwebw8X6/Fq+3VvFsn1V/UoiGhnox8z0CGYOjWRGeoTK8KH4BBWAPcDldLDq7dc469pbMPQwK6EnTTVtfPLUNoLCzVz603EYe8h43F9um43ahQupXfgKsrUVDAbCrrmGiLvuxBgd3et5Ukq2VG7hrb1vsax4GW7ZdZqZUWfkguQL+H7G95kcO1nriVbvh53vwYx7wBJ2ZGHwTBbUF0JosjbFbeQVkDCxSzCWUrK/soVl+6pYvq+KLQfr+zxUAdrTeNPTIpg+NIJpaREq951yWlIB2IOsDfUInQ7/4L7N9W2pt2GyGDD1sijOiXJUVVHz7LM0fPgRuN0Y4uNI/+YbRC8Lyh+ptKWUd/e9y8cHPqaxvbHb/qSgJBZetJC4wF7GsUu3wn/O6749OEEbKx5xGSTPAH3Xa26yOVh7oIaV+6v5bn9Nv3rHAGmRAUxNi2BaWjhTUsOJC1EPgiinvsGeBfEOcC4QCVQCD0spFx7rnFM5AK//6D1CYmIZMfOcfp/rcrn59tVsRp2dQEJm2PFP6IP23Fyqnn4a/4mTiPjhrQC4W1txt7djCDt2HTanjSVFS3hv33vsrNnZuT0+IJ5FVy1CJ3p5ArA6B9Y8A3s/hx4COKD1mIfNORyQjxrblVKSV21lVW41q3NrWJ9fi7WPY8eHJIVbmJwSzpSUcCanhpMWGaDGkJVTzqD3gPvrVA7Ah9haWjAH9m+Rmt3flbLy7Rx0OsHMq4cx5twEjwWMI1dwq3npP9S+8ALRDz5A2NVX9+n8nLocPsz9kC/yv+DGETfyk6yfdNn/Zf6XrCpdxbzUeUyPn45RZwRnO+Sv1G7c7fvi8JziI8WOgbtWH7d+h8vN9uIG1hyoYe2BWrYV1+Nw9e/3MyLAxIQhYUwaEsaklDBGJ4TgZ1CzLJTBpQKwh7W3tvLO73/NDY89hdHUt2UoQVtBbd1HB9i+tBiAzKmxnHNDpkfGho9Uet99NH32OUkLXyZw5kxAm9ImjMe/CdjuasfhchBo6vqfyw8X/5DNldrPJNQvlNnJs5mbOpdJMZMw6AzgcmgzKfZ+Afu+hOYy7cRZv4HzH+payeZXtPHl9AsgZSYYuw8ltNqdbCmqZ21eLevza9lZ0tiv8WMAk17H6IRgJiSHMWFIGBOSw4gNUePIinepADwAnA4HeoMBp70do1///lHnbqpk2Zt7cdrdRCQEcOHto/qc6LOvbDn78csY1tkrLrn357hbWwm/5RYCZkw/5gJDR6uwVnDhBxf2uC/cHM55Sedx0ZCLmBw3WesZSwnl22HfV9oNutijhv7/M/vwDAu9HwyZAUPPh6HnQczobsMVANZ2J5uL6tlYUMuG/Dp2lDT0u4cMEBdiZnxyKOMSQ8lKCmVMYgj+Js+OzyvKkVQAHiC5G9dSsH0LFy34ab/PrS1tYdGLu2isakNv1DHzqnRGn+O5IYkjuZqaOHDuebhbWwEwpaQQdv31hFx5RZcszb2RUpJTn8NX+V/xVcFXVLZW9nhckCmIWYmzuH/y/YSZexl/ttbA39OBXn73AqIgdZa2lnHqLAhL6fEwm8PF9uIGNhXUsbGwjm0HG2hpdx73Wo6mE5ARE0RWUihjE0MZmxhCZmwQRr1KGKN4hgrAA0S63TjtdoRej95g6HfwtNucrP5fLnvXaovpJI8M59wbhxMU7vk/k5319TS89z/q330XZ0UFAMJiIXjexYRdcw3msWP71H63dLOjegeLCxazpGgJNW01XfYHm4JZ+YOV2rBEhy4ZRpx2KFwFB76FA0uhpuuKbt3csxkihx23XS63ZF9FE5sL69lSpL36O8viEJNBx8i4YMYmhjAmIYQxiSGkRwViUEFZOQEqAA+wJS89S+q4iQybOuOEzj+wpYoVb+2jvdWJ0U/P9CuHMnpWAmIAVhSTTifNy5ZR/9bbtG7Y0Lndb1g6IVd+j5D5l/W63sTR3NLN9qrtfFP0Dd8e/JZyaznzh87nL2f9pctxn+V9xsJdC5mVOIuZCTOZED0Bk75jLnVDMeQvh7zlkL8C2o5YUCgwBn6V03VIojZPS0I6ZIY21S1iaI9DFqA9Lr31YD1bi+rZVtzArtLGPi2z2RM/g44RccGMTghmdHwIo+JDyIgNVDf5lONSAXiAtbe2YjKbaWtuwhIcckLDCNbGdla9u5+8bdUADJ8Wy+xbR3q6qV20FxTQ8MEHNH78Ca66jsCn1xN41llE3/cb/Ib2PcOHlJJ9dfsw6ozaOhNH+M3K37C4cHHnZ4vBwqSYSUyPn870uOkMDR2qfc/cbqjcpc2sKFipzSue/0zXija/Al/84vDngChInqY9Lp08TVtIyNDzgzJ2p5u95U1sL27ofBXUWPt8jUcz6ATDYoIYFR/MyLhgRsRpX0NO4olHxfeoAOwlH/71D5x9/a0ntGbEIXlbq1j13n4uvG2Ux+YKH4+022n57jsaPvqYlpUrwe0mfcVyjDExgJY6yZiY2OO6xMctW0pmvz+b6rbqXo+JskQxJW4KU2OnMitxFhGWiN4L/PAO2PV+7/sNZogfD4mTIHOe1lM+hoZWOztLGtlR3MDO0kZ2ljRQ2dTDCnH9kBBq6QjGQQzvCMzJ4f7o1RrJZyQVgL3E7XKh0+spzdlLXHoGuj48ldYTp8OF4YhVwpa/uRf/ED/GX5h8Uov69Knu2lpaN28heM5FQMcDExdehKuxkdT3/4cpJaXfZbY529hUsYk1pWtYU7aGoqbeF1p6fvbzzEqc1XthZdu1oYqitVo2aFsvD4KAlhl69h+6bqvN03rN5uBeT6tssrGrpJGdpY3sKmlgV2kTNS0nF5TNRh2ZMUFkxgaRGRvM8FjtfWRg36cxKqen3gKwmnvjYTq9HiklO775isCwcEKiY06onCODb2N1K9lrytEbdIw8K37AA7AhIqIz+AK46urQBQbibrdhTE7u3F79zLMYExIIPPccDBHH6LGiDTnMSpzVGViLm4pZV76O9eXr2VC+gSa7lhhUJ3RMiO6aiqnV0coVn17BmMgxTIiZQFZ0Fhkz7sZ41s+1IYvqfXBwHRRv1AJyfeHhkxN7WC3uw9u1IB45DOInQHwWxGVpD4z4aVMBY4LNxIw0c8HIwz+/yiYbu0sb2V3axO6yRrLLmvp1k8/mcLOjpJEdJV3/w4gMNJERE9T5yowNZFhMEMFmNYzh61QPeABJKdn4yfuMnHU+QRF9u6nVm/K8RurKWhh1dgKgPdCxc1kxmdNisQSeXNaNvnLW13c+2uxqbGT/jJngcoEQmMeMIfCcWQSefTbmUaP6tB7FIS63i331+9hYvpEKawUPTn2wy/51ZetY8M2CLtssBgujIkYxLmocY6PGMjZqLJGWju9xc6U2x7h4I8y8t8uC8jhs8FgiuB09tERoQTlunDaOHDcWkqb2+JDIIfVWO9nlTWSXNXV+PVDd0u8HRnoSG2xmWEwg6dGBZMQEMSxaex/q752ft+I5aghiEEgp2bPyWzKmzUQI0e+HNY4lZ0MFS1/NxmDUkTk9jnHnJxIW672kmK4WK01ffE7zt8to3bABaT+cTVkfGkrAjOkEzJhBwPTpGBMSTqquf+/4N//a/q/jHhcXEMf5yefzwJQHej+ovgjevR6q9oLsw7oTv9wLwUdkNXHaob4Awod2W2joEJvDxYGqFrLLm8ipaGZveRN7y5uob+0p6PdfZKAf6dEBpEcHkh4VSHp0EOnRgcQE+6l1ME5RKgAPsi+e/htjzr+IIWOyPFJeRX4jm78qpGh3bee2hMxQRs9KJHVcJHqD9+arultbsW7YQMvKlVhXr8FRUtJlvzE5Gf8pkwm94gr8J3X7HTwul9tFbkMu26q2sa1yG9urt3dLQnrInJQ5/OOcf3TZtrJ4JXmNeYwIH8Hw8OHaAyL2VqjYBWXbtCf2yrZr85GPXKLTPwJ+k9d1ituhVeD0fhCVCTGjIHoERI+EqOEQktjjlDgpJdXN7eytaCanool9Fc3kVDSTW9VywtPijhZg0pMWFcjQqACGRgWSFhVIWlQAqZEBKuvIIFMBeJDZ21ox+pkpzt6F02Enbfxkj5RbV25lx7Ji9m+owGnX/iGbA41kTIlh+PQ4IhMDvdorklLiKCqiZfUarOvX0bphI+7mZgBiH3mYsGuvBaBt9x5s2XsImDYN0xHjyn1Vaa1kR/UOdlbvZFfNLvbU7qHd1c6vJ/2aW0bd0uXY+1bex6LCRZ2fY/xjGB4+nIywDDLCM8gIyyA5KBmD0w6Ve6BiB5Tv0GZTzPt714q3vgmf3dN7w0xBWmCOHq5Ni5tw0zGvw+WWFNVayaloZn9lC/urmtlf0UxBjRWnB4YxQPv/ID7EQlpHYE6NDOh8xYda1MwML1AB+BRRnpuD02EnYfhI6svKiEhM8ki57W1OctZXsGdVKXVlh+e1hsX6kz4phmGTor06RHGIdLmwZWfTunETQXPmYErUhiMq//Z36l55hcif/Jion/0MAEdpKbbcXCxjxhz3pt7RHG4HB+oPEGGJINq/66L0l318GYVNhcc836QzkRaaxi8m/IIZCceYtrZpofYQSHPPPfAuhl0ENxw1XW7/Em18OmIYRKZrX/26rwFid7oprLWyv7KZA1Ut5Fa1cKCyhYIaK3aXZ3rMoC1WNCTCn5SOgDwkwp/UiABSIgOIDTajU8HZI9QsiFNE3DAtQ3FtaTGr332dy3/9EE6HA0MfVik7Fj+LgbHnJTLm3ASqDzazb10FuZsqqa9oZdMXBRTsqOYHv9NmBEgpkRKv/OMSej2WMWOwjBnTZbtl3DiC580jYPr0zm3NS5dS+djjABji47CMGo151EjMI0diHjECQ1RUr/UYdUZGRIzocd/No24muzabvbV7OdBwgPYeskDb3Xb21e1Dr+v+p/qPl/6YIFMQqSGppEamkvLDTxhiCMZSl6+NJVfu0WZiVO3tuhxnVA/ZqHO+hC2vdd0WGKs9zRcxFMLTIHwopoihZISlkhHTNau20+XmYF0rB6paOFDdwoGqFvKqreRXtdB8Autg2F1ucjsC/NH8DFpwHhIRQEqEP8kdX4eEBxAfalaPZXuA6gEPMiklb/32l1x6732ExMR6dLjA5XJTsq+eA5sqiUgMJOsC7U/92rIWPnliGxlTYzj7mgyP1XeyGj//gob33qMtO1tLtXQUfWQk5sxM/DIz8csYpr0fOhTRj4dDnG4nhY2F5NTnsL9+Pzn1OeTW51LVquXJW37N8sOzKQCrw8q0t6f1WFZsQCxDgoYwJHgIycHJJAclMTNoKKb6Am3B+rgsSJ7a9aRX52lLdvbF7Ifh7F923VawCgx+2gJFAVGd481SSqqa28mr7gjI1S3kV1vJq26htKENT/8zN+gECWEWksP9tSAdHkBSx/ukcH8C/VTf7khqCOIUZm9rxWTxZ8c3i2hramTaVdd2XbzGw3atKOG7d/eTOS2WCzoedbZZHaz/NJ+4oSHEDwsdkMWA+kq6XNgLC2nbtYv2vXuxZe/Ftncv7pbuvbSU99/HMkZb6rJ5+XLcTU34T5uOMab33Hg9aWxvJK8hj/HR47t833fX7Oa6L6/rUxkCwaYbN+GnP/xgRXVrNR/kfkBiYCIJgQnEl+4gqiYffW0e1OZCXUEvU+KAq1+HUVd03fb8VK23DWD01/LxhSZD6BAITdLehyRr7zsCtM3hoqi2lYIaLTgX1GgBurC2lTqrvVu1nhARYCIpXAvGyeEWksK090lh/sSFms+4leZUAD4NOB0O2q0t+AUE8t8H7uWGvzyJ0OnQG40eDcZSShoqtR7moXHhvK1VLH5pd+cxAaF+xKYGE50aTPSQYKKSg/Ab4AdAjkVKiaO0lPZ9+7Dl5NC+P5f23FxSP3gfnb+WOfng7XdgXbOGxH89T9D55wPQ+MWXWNevw5Q8BFNyMqbkJIxJSX1aghO0J/j21u6loLFAezUVUNRURElzCa6jprHFBsTyzfe/6bJtTeka7lp6V5dtBmEgJiCG+MB44vxjidP7E+eWpNgdTLS1aU/q1RfANW9oD4cc/ibAX2LBaevbN+32byDpiAdRpIRt/4XgOAhOhOB4Gt0WCmqtFNZogbmw1kphbSuFNVYa2zwzbe5oep0gNthMUriFxDAtKCeGWUgMs5AU7k9MsNnnbgyqMeDTgMFoxBCqPejwvQcfxWg2s/mLj7G3tTHj6uupyMslakgqesPJ/diEEN1uyEUkBDLtijTK8xqpyGvE2tBO3rbqzoWBAIKjLEQmBhKREEhEQgBxQ0PxD/bOQwFCCEyJiZgSEwm64IIejwk8Zxa64CD8Mg6PvVrXraXxw4+6HasPCcGYmKi9EhIwxsVhTIjHGB+PKS2tc80Li8HChJgJTIjp+nSew+WgtKWUoqYiipqKONh8EH+Df7d6SltKu21zSielLaXd9o2LGsd/5/23y7bvSr7j68KvifaPJtoUSlTaFCJbaohqKCOyrQG/Y/WfQo66wWtr6DaDI8QURFZwPFnBcRAUB5FxkBoHwXHUJ86mqMFOUa0WnA/WtlJU10pRrZWalhPvObvcktKGto6nCOu67TfoBHGhZhJD/UnoCMwJoR2vMAtxIRZMXpxmOZBUD/gUJ6XE1ZF544O//J7Lf/MQdSXFNFSWM3zmOQMyVCHdkoaqVirym6gsbKK6qInaUiuuo+arzvnRaNInan/qF+6soTyvkdRxkcSm9S1btDe07dqNbfcu7EUHsR88iKO4GHtxMdLWey8y7csvOleBq124kPb9+wm78abOoQ5nTQ1umw1DZCQ687GHanZU7+Dbg99S2lxKWUsZZdYy6mzdgw70PIf52W3P8tLOl3otP9gQQITBn0hh5CJ9KNe2C2gshuYK+HUupa3luNwuwsxhBNYWIF48+5jt7aQzwkNVcGTWlKq9sPhBCIzBbomkToRS6QqmxBFIfquFnBYLexuMFDXYPTaFridCQHSQHwmhFuI7gnJCqIX4kI7PoRaCLf1fm3sgqR7waUoIgaGjN3b177U1dg0mE34B2tSldx++nwtu/zGB4REU7dzmkaAsdFoPOSw2gBEztLT0Lpeb+vJWaktbqClpoa6shcjEw9OnCnfXsue7UvyDTZ0B+MCWKtZ/kkdQhJnAcDOBYX4EhvoRcOgV4ocl0Dggax4fYhkzujNwHiKlxFVTg72kBEdJKY6yss6Xs6IcY2xs57HWNWuwrl1H8CWXdG6rf/sdav6lPZmnCwzEEBGBPjISQ3g4+ohw7WtYOPrwMIaGhpIZNhfTiJTOoZJWRysV1grKrGVUWCsot5ZT3lJOVnRWt/ZXWnvOPHJIk9NKk9NKATB61ByY1PWm3dNbn2ZRgTYH2iD0hKYNJdQNoS4HoXYbIU4noW4XIW43M9psZNo7hh2CYkGno83Zhp/eT8uOXV+krdsMmIDYjte4o9okQ8KxxUxg29kvUlzXSnFdG8X1rbiqcghtyOagzUy9DKKeQBpkIC1YgL7/DkgJlU3tVDa1s/VgQ4/HBJj0xIdaiAu1EB9i1t4f8TUuxILFw3kYT4TXArAQYi7wNKAHXpZSPu6tun1NZHIKkckpAHz/t39EbzTSXFtDU402XPDtKy8QO3QYo8+9gO/efo0Z378eR7uN9tZWQmNij1Fy7/R6HZGJgUQmBpI5tfv+9AlR+AcZiR8W2rmtsbqVxuo2Gqt7X7BG6ASWICP+wSYsQSYmzBlCYscSnHVlVmrLWohICCQ8ThsycXfMgdWdxE0cIQSGqChtWtv48cc8NvKeewi+5BLMIw5PcRMmE4a4OK0n3NKCvaUFinpf3Q0g+fXXCZiqjce2vvw68osvGXPXnZx12fcBaNuzh+bFS6ld/Qq6oED0QUHoAgP5gSOLqfFxVNNChWyggkYq7bVUt1ZTZ6vrMg7d0xKe9bb6zvdO6aIGFzUC7V++oWvvPTD+QjKN0doc546ErLcuvpW9tXsJNAUSJAVB8bEEut0ESkmA202g202AWxIg3cxrsZLkdCHa6rDINmYMjSQ3PJdpejMWQxiWbSuxfPN0t6WaXUJPqy6IRgKpc/lT67Kw2j2Gha55XY4bJkqIFvU0S3+a8e/4aqEdI0cHcKvd1ev0ukNC/Y3EhWgBOrYjOMcGm4nr+BwbYh7wXIFeCcBCCD3wPHAhUAJsEkJ8JqXM9kb9vszY8SdwSHQMUy7X/jGff+sC3C4X0u0mICQUvdFIyd7dFO/ZydnX38oXT/+NsbPnEpeewYo3XubCBfdQXVSAva2NhOEjqTlYSHB0DAaTCZfD0ac1LBKHh5M4PLzLtnHnJ5E6LormOhstdTZa6ttpqbdhbbRjbWintdGOzeqgtdFOa6M2pjh61uF1I/K3V7HhswImzh3CtCu0IYGSnHo+f2YHBpMOk8WAyWzAZNZjNOsx+hkw+um1l0mPwU+H0U9PfHoocemhgLbofXVRMwGhfkQlazfiXC43TdVt6A069AYdOoNAr9e+mrPG4z+h6/hv5F13EnnXnUgpcTc24qytxVlbi6uuruNrPa76Opx19bgaGnDV12OIOjy1zV5Sgj0vD3fb4f+YbLv3UPvvF7p9Xw1AesfrEGE0ogsIIOr+h3HPnUWtrZbG5csJeXIZ9ReYCLtOm7XhqKzkgsVVZNoDaRJtWIUDuxEcenAYjvhqEDj0EJk8FueoSzrnW0unE0dTIwanm2aaaQbw633MP0v6kdRYA0jtMW7g5kU30+I4IgimJmOUErNbYpFu/KTE3PH5H9XljMUFepgwMpMp48by3/3P0mYXtLYLMpo2M8qeg0lKDEiMEoxSopfglCYmWXXYpBkrFt51ncd77pnozWVIqQN0nCN2EUs9dkzYpQm7y4S9zoSt1sh+TOx3hGDHgB0jpTKCJmEm2N9JTLCZqCA/kgONxITFMSIumLmjT6wjczSvjAELIaYDj0gp53R8fhBASvlYb+eoMeCBY2tpQW/SMhcXbNtMxrSzKNq1HVtLC5nTz2LR808y5fKr0el1fP7U49z8t2fZ+OkH6PR6Jl16JZ8/+Rjn3HwHLqeDXd9+zawbfsj+9asxBwaRPHocmz77kHEXzcPW0kxFXi4ZU2dSsm8P/sGhhMcnkLthLUMnTaWtuYnGqiqCIodQlpsH0kLiiAQaKvJJyBzB7u8OkL+tlJFnjSAy0Y0lKJjivU0senE1Ol040m0DXAhdANLdDMIM6EFaEbogpGwHKZl2xQhGzAjFHBhIwfYaFr+0mbTxQ5h9SzpCQHubjtcfWNpxvgRpR+gs2vno0OmMCNGGMASg17tBuEgbl8BZV6egM+ixt8FHf19NUEQYl94zBqe9HXNAIJ88tR5Hu0SnNyJdbRj8ApDSCU47OrfArXNjsPgjdHqcDVX4tTQzNigHZ3MT+lYbm1vTcTkNjKpZjMvahKG5hbzwabT4x6CTevzGDsecMhQBtB3Yh33PfoypyZjHjsVgMGGrLKFt9TrSD3yEwIHR5aYw8Rxa/WNILlmFqb0So8tNVeQYasIz0WPAkDaEgImT0OmNtJcfpHXFGhr8JVuGSdx6Azqnk3N2Q3TlKvTOKnTSTU3YGJpDMkl2lBPpLMMooMkvmoMBo2i0W2myuNmbCFKvR+90kpUvcAvBjjQ3br0B4XaRUSKZWrud6LZsjLhoiJpBfvBMKpq2IWzrQIBb54/bMgcp9Egh2ZcoO88fWi4Z0ebGGNqK0aRnuyuNxhZBuHMLbvs67Rdf548wXQSyHVf7V4BACInwuxQ9fqQ5XBjC2vAzGVjrzKS9tZ1wdzUVoS7qA3W4DY183Xoh52VG8eoPe1jm9Bh6GwP21q3EBKD4iM8lHdu6EEIsEEJsFkJsrq7uPXuCcnLMgYEYTX4Y/cxkTDsLgCFjssicrr2/+O5fEpGYRFhcAjf/7VkAsuZcwpjz5wAw5Yqr8Q8OwRwYxJCx2p/w/iGhmAMPT+0SCJx2B60NDQA0lJfR2qD9Obx/wxqkdNNYVUHRrq0EhZupPbgDS2ALlkADq95+VSszsJ6IuGLSJ0az+fMPqT5YQNKIIMIj1/Kjf85i5vcMjJhWy1X3TyQ6cQfTrwjjrKsTsFi+Zeb300nOrCZmyEFih4ay6Pknaagox+lsQMcSopKD2PHNV+xapk0bc7d/hn9wO37mRly2xRj89Ljsm3HZ9yIltNX/D6etBVtTCS1VX+Fod7H2f//lwIa1uBxuqg68TGNVMwd3bWf5a9pNs4M7P6V8/y7KDzRQtP1ZSnMaKN69haLdiygpbKdw5ycUbN9J4c5y8ne/TrWIo276JApHZZD49D8psu+nTB9I5L9eYMvEkWRu2kj1+OGUB0Nl7BQO5C1m37ps9q7LIbfwGyriplNQX8Le75awd205+7a9S1n0GPj+VeyePp7Qa39ASVIYJRFh6Kady4axwzBMmUxtagbFwc2UxZ9NXl0eu5etZM93pezetJDS+Ok0ByYxtKiNUZVnkX6wnfKIRHQihG3JSYRZocUSTnWAmxZbINucVsorK6ivtpHv3E5L2CzcIpKMgzpGV57N0KIG2gIysAdOYGh+MaMrzyazOBjhNtNgG8p3phTKq0MoL3OR37SD8PYRxNcEk1gZxNRcE+3OcqQpCwwjSC+oY3Tl2WQcNGBwRVAQNJtddbXstp6L3hpAsD2HGOsIEqvCGFXoz9nZJhy2PPyZyIQ8C3G1kczZJsHhwq1LIj/wfHbVlLK79WICWhyE2svQm2YRVdXKiNIUwlq1WSWxIb0vT9pf3uoBfx+YK6W8o+PzTcBUKWWvq5qoHrByKpBSIt0St0viPvTVJdHpBeYA7fFxl9NNY1UbQkeX6X2VhU24nW6k1GaWaGWBW0qto93xSDgd2w0mHcmjDo/j5m2rwuV0MzQrGr1R6ysV76ujtaEdCR1ldLyBrk+7HfE+Y2pM5wL/hbtqsDa0kzwqovNhm4qCRmqKW+heyJHfh8Pv3e12UoYHERCoQzqdVBa1UFPaSlSskchoI9LppKm2neJCG0iJMBoxRmuzZaTLTVtBHk63E4Yk4JROHC4HzsIihkVYiAoVICVNzZKDFW7KbfvxC9Rmctgdktb6BKR040JSnRqKW7pxu12EFdUxKXAY5rREdAYB/mFUFxayM/8LnMY8pNuFu92BwTEON23YdZtxS+1nYnJPxeQ2kOUwoAsTCL2T5UyhvrGZcHIoDdfRGKDDIttY0TyfX1yQwb0XHD9L95EG9UEMNQShKMrpxu2W1LS0U9Fko6LRRkWTjfJGG+dkRDEtrX+LRQ32NLRNwDAhRCpQClwLXO+luhVFUfpNpxNEB5uJDjYzNnFg6vBKAJZSOoUQ9wBfo01De0VKuccbdSuKopyqvDYPWEr5FfCVt+pTFEU51fnGA9WKoiinIRWAFUVRBokKwIqiKINEBWBFUZRBogKwoijKIDll1wMWQlQDx15iqrtIoGYAmnOqUNd3evP16wPfv8YTvb4hUspuWWVP2QB8IoQQm3t62sRXqOs7vfn69YHvX6Onr08NQSiKogwSFYAVRVEGia8F4N6TZ/kGdX2nN1+/PvD9a/To9fnUGLCiKMrpxNd6wIqiKKcNFYAVRVEGyWkZgIUQc4UQOUKIA0KIB3rY7yeEeK9j/wYhRMogNPOE9eH6fimEyBZC7BRCfCuEGDIY7TxRx7u+I467SgghhRCn1bSmvlyfEOKajp/hHiHE295u48now+9nshBiuRBiW8fv6LyeyjlVCSFeEUJUCSF297JfCCGe6bj+nUKICT0d1yeyIy3H6fJCW084D0gDTMAOYORRx/wEeKHj/bXAe4Pdbg9f33mAf8f7H/va9XUcFwR8B6wHJg12uz388xsGbAPCOj5HD3a7PXx9LwE/7ng/Eigc7Hb38xpnAROA3b3snwcsAgQwDdhwonWdjj3gKcABKWW+lNIOvAtcftQxlwOvd7z/AJgthBBebOPJOO71SSmXSylbOz6uBwZovf4B0ZefH8CfgP8DbN5snAf05fp+BDwvpawHkFJWebmNJ6Mv1yeB4I73IUCZF9t30qSU3wF1xzjkcuANqVkPhAoh4k6krtMxAPclw3LnMVJKJ9AI9C+J0+DpUwbpI9yO9r/x6eK419fxJ12SlPJLbzbMQ/ry88sAMoQQa4QQ64UQc73WupPXl+t7BLhRCFGCloThp95pmtf0999or7yWEUPxPCHEjcAk4JzBbounCCF0wJPArYPclIFkQBuGOBftr5fvhBBjpJQNg9koD7oOeE1K+URHQt43hRCjpZTuwW7YqeZ07AGXAklHfE7s2NbjMUIIA9qfQbVead3J68v1IYS4APgdMF9K2e6ltnnC8a4vCBgNrBBCFKKNsX12Gt2I68vPrwT4TErpkFIWAPvRAvLpoC/XdzvwPwAp5TrAjLaIja/o07/RvjgdA3BnhmUhhAntJttnRx3zGXBLx/vvA8tkx+j5aeC41yeEGA+8iBZ8T6fxQzjO9UkpG6WUkVLKFCllCtoY93wp5ebBaW6/9eX38xO03i9CiEi0IYl8L7bxZPTl+g4CswGEECPQAnC1V1s5sD4Dbu6YDTENaJRSlp9QSYN9x/EE71LOQ+s15AG/69j2R7R/qKD9wN8HDgAbgbTBbrOHr28pUAls73h9Ntht9uT1HXXsCk6jWRB9/PkJtGGWbGAXcO1gt9nD1zcSWIM2Q2I7cNFgt7mf1/cOUA440P5auR24C7jriJ/f8x3Xv+tkfj/Vo8iKoiiD5HQcglAURfEJKgAriqIMEhWAFUVRBokKwIqiKINEBWBFUZRBogKwoijKIFEBWFEUZZCoAKx4lRCisOMxao+f27G27rk9HXvkvoEkhMgUQmwXQjQLIX420PUppzcVgBWfIaUcJaVccbx9J/OfQB/cByyXUgZJKZ8ZoDq6EULcI4TYLIRoF0K85q16lZOjArDiMR0LH53phgB7BqHeMuDPwCuDULdyglQAVo6ro8f4YEcKnXohxKtCCPMR++4XQuwErEIIgxBihBBihRCioeNP//lHFTm5p7I6yntACJHX8Sd8thDiyn6c22vP9tA+IcSbQDLwuRCiRQhxnxDiN0KID486/hkhxNO9lNXj9QkhlqFlK3muo+yMo84LFEK4jly8WwgxWghRLoQI6vGb30dSyo+klJ9w+qz6p6ACsNJ3NwBzgKFoq3c9dMS+64BLgFC0hUo+B5YA0WiLcb8lhMjsY1l5wNloS4g+Cvz3qGwDxzr3uKSUN6Gt1nWZlDJQSvk34L/AXCFEKHT25K8F3jj6fCGEsbfrk1KeD6wC7ukoe/9RdbcA+9DS3RzyOPBXKWXzEXV80RHce3p90Z/rVU5tKgArffWclLJYSlkH/AUt6B7yTMe+NrT1ewOBx6WUdinlMuCLo47vtSwp5ftSyjIppVtK+R6Qi5YGpy/tOCFSW0rwO+Dqjk1zgRop5ZYeDu/L9R3LJjoCsBBiFtrKYS8e1Z5LpZShvbwu7fcFKqcsFYCVvjoyBUsREN/LvnigWHbNflBE15QtvZYlhLi5YxZBgxCiAW1x9si+nHuSXgdu7Hh/I/BmL8f15fqOpTMAA38Dfi+13GrKGUgFYKWvjswAkEzXRItHrmlaBiR1pBY68vgjMwb0WJYQYgjwH+AeIEJKGQrsRhvW6Es7+qqnNVg/AcYKIUYDlwJv9XJuX67vWDYBE4QQV6GtW90tJb0QYlHHGHJPr9Mp/59yHCoAK311txAiUQgRjpYK6b1ejtsAtAL3CSGMHXNvL0PLnnu8sgLQgmM1gBDih2g94BNpx7FUoqVV7ySltKFl0H4b2CilPHgS13csO4BY4AngQdnDgtxSyos7xpB7el3cU6EdNz/NaGnj9UIIs5qVcupTAVjpq7fRbjzlo90o+3NPB3X8OX0ZcDFQA/wLuFlKue94ZUkps9EC0zq0IDkGLbNCv9txHI8BD3UMc/z6iO2vd9TZ2/BDX6+vV1LL37cLKJRSerI3+xDQBjyANoTSRj9vUCrepzJiKMcltOSYd0gplw52WwaSECIZbZZCrJSyaYDqMKGlyrpGSrl+IOpQTh+qB6woQMeY7i+Bdwcq+HZ4GFijgq8CoMaIlDOeECIAbcijCG0K2kDUMQFYDuwEjn64RDlDqSEIRVGUQaKGIBRFUQaJCsCKoiiDRAVgRVGUQaICsKIoyiBRAVhRFGWQqACsKIoySFQAVhRFGST/Dy9dp+8umwV1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "p = np.linspace(0.001, 1.0, num=100)\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5, 4))\n",
    "\n",
    "for g in [0, 0.5, 1, 2, 5, 10]:\n",
    "    f = -(1 - p)**g * np.log(p)\n",
    "    \n",
    "    if g == 0.0:\n",
    "        ax.plot(p, f, linewidth=4)\n",
    "    elif 0.5 <= g <= 1:\n",
    "        ax.plot(p, f, linewidth=3, linestyle='--')\n",
    "    elif 2 <= g <= 5:\n",
    "        ax.plot(p, f, linewidth=2, linestyle='-.')\n",
    "    else:\n",
    "        ax.plot(p, f, linewidth=1, linestyle=':')\n",
    "        \n",
    "ax.set_xlabel('probability of $y = 1$', fontsize=12)\n",
    "ax.set_ylabel('Loss', fontsize=12)\n",
    "ax.legend(['$\\gamma = {0}$'.format(g) for g in [0, 0.5, 1, 2, 5, 10]])\n",
    "\n",
    "fig.tight_layout()\n",
    "# plt.savefig('./figures/CH05_F21_Kunapuli.png', format='png', dpi=300, pad_inches=0)\n",
    "# plt.savefig('./figures/CH05_F21_Kunapuli.pdf', format='pdf', dpi=300, pad_inches=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When $\\gamma=0$, the original cross-entropy loss is recovered. As $\\gamma$ increases, the part of the curve corresponding to \"well-classified\" examples becomes longer, reflecting the loss function's focus on poor classification.\n",
    "\n",
    "In order to use this focal loss to train gradient boosted decision trees, we have to provide ``LightGBM`` with two functions:\n",
    "* the actual loss function itself, which can be used for evaluation\n",
    "* the first derivative (gradient) and second derivative (Hessian), which will be used for learning\n",
    "\n",
    "**Listing 5.7**: Defining Custom Loss Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.misc import derivative\n",
    "\n",
    "def focal_loss(ytrue, ypred, gamma=2.0):   \n",
    "    p = 1 / (1 + np.exp(-ypred))\n",
    "    loss = -(1 - ytrue) * p**gamma * np.log(1 - p) - ytrue * (1 - p)**gamma * np.log(p)\n",
    "    return loss\n",
    "\n",
    "\n",
    "def focal_loss_metric(ytrue, ypred):\n",
    "    return 'focal_loss_metric', np.mean(focal_loss(ytrue, ypred)), False\n",
    "\n",
    "\n",
    "def focal_loss_objective(ytrue, ypred):\n",
    "    func = lambda z: focal_loss(ytrue, z)\n",
    "    grad = derivative(func, ypred, n=1, dx=1e-6)\n",
    "    hess = derivative(func, ypred, n=2, dx=1e-6)\n",
    "    return grad, hess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 3.38485\tvalid_0's focal_loss_metric: 0.141808\n",
      "[2]\tvalid_0's binary_logloss: 3.01701\tvalid_0's focal_loss_metric: 0.11333\n",
      "[3]\tvalid_0's binary_logloss: 1.10525\tvalid_0's focal_loss_metric: 0.0952922\n",
      "[4]\tvalid_0's binary_logloss: 0.924261\tvalid_0's focal_loss_metric: 0.0790233\n",
      "[5]\tvalid_0's binary_logloss: 0.505147\tvalid_0's focal_loss_metric: 0.0677255\n",
      "[6]\tvalid_0's binary_logloss: 0.682986\tvalid_0's focal_loss_metric: 0.0606197\n",
      "[7]\tvalid_0's binary_logloss: 0.595374\tvalid_0's focal_loss_metric: 0.0537766\n",
      "[8]\tvalid_0's binary_logloss: 0.522227\tvalid_0's focal_loss_metric: 0.0482069\n",
      "[9]\tvalid_0's binary_logloss: 0.215398\tvalid_0's focal_loss_metric: 0.0459177\n",
      "[10]\tvalid_0's binary_logloss: 0.448094\tvalid_0's focal_loss_metric: 0.0438938\n",
      "[11]\tvalid_0's binary_logloss: 0.121162\tvalid_0's focal_loss_metric: 0.0396912\n",
      "[12]\tvalid_0's binary_logloss: 0.351168\tvalid_0's focal_loss_metric: 0.0373091\n",
      "[13]\tvalid_0's binary_logloss: 0.0564804\tvalid_0's focal_loss_metric: 0.0354938\n",
      "[14]\tvalid_0's binary_logloss: 0.0137323\tvalid_0's focal_loss_metric: 0.0325854\n",
      "[15]\tvalid_0's binary_logloss: 0.00815208\tvalid_0's focal_loss_metric: 0.0325117\n",
      "[16]\tvalid_0's binary_logloss: 0.258821\tvalid_0's focal_loss_metric: 0.0321897\n",
      "[17]\tvalid_0's binary_logloss: 0.229213\tvalid_0's focal_loss_metric: 0.03054\n",
      "[18]\tvalid_0's binary_logloss: -0.0457439\tvalid_0's focal_loss_metric: 0.0303324\n",
      "[19]\tvalid_0's binary_logloss: -0.0784607\tvalid_0's focal_loss_metric: 0.0283734\n",
      "[20]\tvalid_0's binary_logloss: -0.0985534\tvalid_0's focal_loss_metric: 0.0272042\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9649122807017544"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbm_focal_loss = LGBMClassifier(objective=focal_loss_objective, learning_rate=0.25, n_estimators=20, max_depth=1)\n",
    "gbm_focal_loss.fit(Xtrn, ytrn, eval_set=[(Xval, yval)], eval_metric=focal_loss_metric)\n",
    "\n",
    "from scipy.special import expit                               # Import the sigmoid function from scipy\n",
    "probs = expit(gbm_focal_loss.predict(Xval, raw_score=True))   # Get raw scores and then compute the probability positive class\n",
    "                                                              # using the sigmoid function\n",
    "ypred = (probs > 0.5).astype(float)                           # Convert to a 0/1 label, where the prediction is class 1 if \n",
    "                                                              # probability is above 0.5 and class 0 otherwise\n",
    "accuracy_score(yval, ypred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.956140350877193"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbm_standard = LGBMClassifier(boosting_type='gbdt', learning_rate=0.25, n_estimators=20, max_depth=1)\n",
    "gbm_standard.fit(Xtrn, ytrn)\n",
    "probs = expit(gbm_standard.predict(Xval, raw_score=True))   # Get raw scores and then compute the probability positive class\n",
    "                                                              # using the sigmoid function\n",
    "ypred = (probs > 0.5).astype(float)                           # Convert to a 0/1 label, where the prediction is class 1 if \n",
    "                                                              # probability is above 0.5 and class 0 otherwise\n",
    "accuracy_score(yval, ypred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}