{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "435a256a-7d50-410e-9bfa-b303d4eaff95", "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# Install the necessary dependencies\n", "\n", "import os\n", "import sys\n", "!{sys.executable} -m pip install --quiet pandas scikit-learn numpy matplotlib jupyterlab_myst ipython" ] }, { "cell_type": "markdown", "id": "4bb18763", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "---\n", "license:\n", " code: MIT\n", " content: CC-BY-4.0\n", "github: https://github.com/ocademy-ai/machine-learning\n", "venue: By Ocademy\n", "open_access: true\n", "bibliography:\n", " - https://raw.githubusercontent.com/ocademy-ai/machine-learning/main/open-machine-learning-jupyter-book/references.bib\n", "---" ] }, { "cell_type": "markdown", "id": "e44880af", "metadata": {}, "source": [ "# Gradient boosting example" ] }, { "cell_type": "markdown", "id": "50b29d0b", "metadata": {}, "source": [ "## What is gradient boosting?\n", "\n", "If you're inside the world of machine learning, it's for sure you have heard about gradient boosting algorithms such as xgboost or lightgbm. Indeed, gradient boosting represents the state-of-start for a lot of machine learning task, but how does it work? We'll try to answer this question specifically for the case of gradient boosting for trees which is the most popular case up today\n", "\n", "For me, understanding gradient boosting is all about undestarding its link to gradient descent. Remember gradient descent is the algorithm to minimize a loss function $L(\\theta)$ by sustracting the gradient to the parameters\n", "\n", "$$\n", "\\theta = \\theta - \\frac{\\partial L(\\theta)}{\\partial \\theta}\n", "$$" ] }, { "cell_type": "markdown", "id": "f8600839", "metadata": {}, "source": [ "At first glance it doesn't make much sense; trees are based on a split-gain function (Gini, entropy), not a loss function. Moreover, what would be $\\theta$?\n", "\n", "Recall $\\theta = (\\theta_1,\\ldots,\\theta_n)$ are the learned parameters we use for making predictions, which are the paramters of the loss function. In gradient boosting, we consider the loss function as a function of the predictions instead, so we want to find $\\min_{p}L(y,p)$ and the way to achieve that is analogous to gradient descent, i.e, updating the predictions on the opposite direction of the gradients. But how can you update the predicitions? I mean, once the tree is built it has a fixed structure. \n", "\n", "Here comes the idea of additive modeling, in which you add (sequentially in this case) the predictions of several models in order to get a better performance. The first tree makes predictions on the original target, then, with the second tree, we try to minimize the loss function adding something which is not exactly minus the gradient loss, as in gradient descent, but instead we add *predictions on the gradients loss*, i.e, we fit a tree over with target the gradient loss. So, if $X,y$ represents the original data and $p = \\text{predict}(X,y)$ \n", "\n", "$$\n", "p = p-\\text{predict}\\left(X,\\dfrac{\\partial L(y,p)}{\\partial p}\\right)\n", "$$ " ] }, { "cell_type": "markdown", "id": "892cfad9", "metadata": {}, "source": [ "This equation would be the *gradient descent* version of trees. Maybe this is nothing new for you, but I hope it gives you a better or new way of seeing gradient boosting." ] }, { "cell_type": "markdown", "id": "4b3d6fd9", "metadata": {}, "source": [ "## Implementing Gradient boosting\n", "\n", "Now let's get our hands on some data and see a woking example" ] }, { "cell_type": "code", "execution_count": 2, "id": "9e0da69f", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor\n", "from sklearn.metrics import f1_score,roc_auc_score\n", "import matplotlib.pyplot as plt\n", "import warnings\n", "\n", "warnings.filterwarnings('ignore')\n", "seed = 1301" ] }, { "cell_type": "markdown", "id": "db2cfc66", "metadata": {}, "source": [ "We will work with a modified version of our familiar titanic dataset. This is a ready-to-train version: doesn't not contains missings and categorical data has been encoded." ] }, { "cell_type": "code", "execution_count": 3, "id": "993b0cca", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" }, "tags": [ "output-scoll" ] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengeridAgeFareSexsibspzerozero.1zero.2zero.3zero.4...zero.12zero.13zero.14Pclasszero.15zero.16Embarkedzero.17zero.182urvived
0122.07.25000100000...0003002.0000
1238.071.28331100000...0001000.0001
2326.07.92501000000...0003002.0001
3435.053.10001100000...0001002.0001
4535.08.05000000000...0003002.0000
\n", "

5 rows × 28 columns

\n", "
" ], "text/plain": [ " Passengerid Age Fare Sex sibsp zero zero.1 zero.2 zero.3 \\\n", "0 1 22.0 7.2500 0 1 0 0 0 0 \n", "1 2 38.0 71.2833 1 1 0 0 0 0 \n", "2 3 26.0 7.9250 1 0 0 0 0 0 \n", "3 4 35.0 53.1000 1 1 0 0 0 0 \n", "4 5 35.0 8.0500 0 0 0 0 0 0 \n", "\n", " zero.4 ... zero.12 zero.13 zero.14 Pclass zero.15 zero.16 Embarked \\\n", "0 0 ... 0 0 0 3 0 0 2.0 \n", "1 0 ... 0 0 0 1 0 0 0.0 \n", "2 0 ... 0 0 0 3 0 0 2.0 \n", "3 0 ... 0 0 0 1 0 0 2.0 \n", "4 0 ... 0 0 0 3 0 0 2.0 \n", "\n", " zero.17 zero.18 2urvived \n", "0 0 0 0 \n", "1 0 0 1 \n", "2 0 0 1 \n", "3 0 0 1 \n", "4 0 0 0 \n", "\n", "[5 rows x 28 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "path = \"https://static-1300131294.cos.ap-shanghai.myqcloud.com/data/titanic_train_and_test.csv\"\n", "data = pd.read_csv(path)\n", "\n", "data.head()" ] }, { "cell_type": "code", "execution_count": 4, "id": "59284066", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [], "source": [ "###to get train and test\n", "def split_data(data,target,drop,test_size=0.2,seed=seed):\n", " \n", " return train_test_split(data.dropna().drop(target+drop,axis=1),\n", " data.dropna()[target],\n", " test_size=test_size,\n", " random_state = seed)" ] }, { "cell_type": "code", "execution_count": 5, "id": "dd717fed", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [], "source": [ "drop,target = ['Passengerid'],['2urvived']\n", "\n", "#split the data\n", "X_train,X_test,y_train,y_test = split_data(data,target,drop=drop)" ] }, { "cell_type": "markdown", "id": "39e67e58", "metadata": {}, "source": [ "Let's fit a base learner for the data." ] }, { "cell_type": "code", "execution_count": 6, "id": "bd05fa8c", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "f1 score: 0.63\n", "AUC score: 0.86\n" ] } ], "source": [ "tcl = DecisionTreeClassifier(max_depth=3,random_state=seed)\n", "tcl.fit(X_train,y_train)\n", "y_pred = tcl.predict(X_test)\n", "y_prob = tcl.predict_proba(X_test)[:,1]\n", "print(f\"f1 score: {f1_score(y_test,y_pred):.2f}\")\n", "print(f\"AUC score: {roc_auc_score(y_test,y_prob):.2f}\")" ] }, { "cell_type": "markdown", "id": "c779dcce", "metadata": {}, "source": [ "Now we will construct a gradient booster class implementing the previous algorithm. Let's code it first and then talk a little bit about the details." ] }, { "cell_type": "code", "execution_count": 7, "id": "9cb6370d", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [], "source": [ "class gradient_booster():\n", " def __init__(self,loss,gradient_loss,max_depth,nu):\n", " self.max_depth = max_depth #max depth of all learners\n", " self.nu = nu #learning rate\n", " self.loss = loss #loss function to be optimized\n", " self.gradient_loss = gradient_loss #gradient of the loss function\n", " self.learners = [] #list with all the learners\n", " self.loss_history = [] #loss through the process\n", " \n", " def fit(self,X,y,epochs):\n", " base_learner = DecisionTreeClassifier(max_depth=self.max_depth,random_state=seed)\n", " base_learner.fit(X,y)\n", " initial_probs = base_learner.predict_proba(X)[:,1]\n", " \n", " probs = initial_probs\n", " preds = initial_probs\n", " loss_history = [] \n", " self.learners = [base_learner]\n", " target = y\n", " \n", " for i in range(epochs):\n", " target = -gradient_loss(y,probs)\n", " rt = DecisionTreeRegressor(max_depth=self.max_depth,random_state=seed) #regressor tree\n", " rt.fit(X,target)\n", " self.learners.append(rt)\n", " preds += self.nu*rt.predict(X) #these are not probabilities!!!\n", " probs = 1 / (1 + np.exp(-preds)) #these are probabilities\n", " self.loss_history.append(loss(y,probs))\n", " \n", " return self\n", " \n", " def predict_proba(self,X):\n", " try:\n", " preds = self.learners[0].predict_proba(X)[:,1]\n", " \n", " for m in self.learners[1:]:\n", " preds += self.nu*m.predict(X)\n", "\n", " return 1 / (1 + np.exp(-preds))\n", " \n", " except NotFittedError:\n", " print(\"Model not fitted yet\")" ] }, { "cell_type": "markdown", "id": "94961725", "metadata": {}, "source": [ "We intenionally picked a classification problem over a regression because has some additional technical details worth understanding.\n", "\n", "1. Note that the base learner is a classifier tree but the next learners are regressors due to the gradients being a continuous target.\n", "2. The outputs of the regressor trees aren't probabilities and therefore the cumulative predictions `preds` neither are. That's why we pass them to the sigmoid function.\n", "3. Finally, and this stand for the general case, we added a shrinkage parameter `nu` because of the same reasons we add it to gradient descent: the gradients can become crazy and overshoot the minimum.\n", "\n", "Next we define the loss which is the classic binary deviance." ] }, { "cell_type": "code", "execution_count": 8, "id": "10d66c9f", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [], "source": [ "def loss(y,p):\n", " return -2.0 * np.mean(y * np.log(p/(1-p)) - np.logaddexp(0.0, np.log(p/(1-p)))) " ] }, { "cell_type": "code", "execution_count": 9, "id": "3ccd3b59", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [], "source": [ "def gradient_loss(y,p):\n", " return (p-y)/(p*(1-p))" ] }, { "cell_type": "markdown", "id": "710a13c6", "metadata": {}, "source": [ "Let's implement the booster and see how it performs" ] }, { "cell_type": "code", "execution_count": 10, "id": "a9f18a72", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "f1 score: 0.71\n", "AUC score: 0.88\n" ] } ], "source": [ "booster = gradient_booster(loss=loss,gradient_loss=gradient_loss,max_depth=3,nu=0.01)\n", "X, y, epochs = X_train.values, y_train.values.ravel(), 50\n", "booster.fit(X,y,epochs)\n", "y_prob = booster.predict_proba(X_test)\n", "y_pred = 1*(y_prob>0.5)\n", "print(f\"f1 score: {f1_score(y_test,y_pred):.2f}\")\n", "print(f\"AUC score: {roc_auc_score(y_test,y_prob):.2f}\")" ] }, { "cell_type": "markdown", "id": "a66ed60a", "metadata": {}, "source": [ "Nice! We got decent increment in auc and a good amount of f1. To finish this section we'll plot the classic loss vs epochs plot to verify that the loss is decresing with every addition of a new tree." ] }, { "cell_type": "code", "execution_count": 11, "id": "b472389d", "metadata": { "attributes": { "classes": [ "code-cell" ], "id": "" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAH5CAYAAAAC8w0GAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABJBElEQVR4nO3dd3RUZeL/8c+dmTRCCgmQEAi9RFoILYSyiIKIygo2mtJEAcGyrLqwPxfUr7vo2rAgioIRERCQYsWC0gmQQChSpARCSUJNhdSZ3x9o3CglgSQ3M3m/zrnnbGaeufnkLmfnszPPfR7D4XA4BAAAcBUWswMAAADnQGkAAADFQmkAAADFQmkAAADFQmkAAADFQmkAAADFQmkAAADFYjM7QGmx2+06ceKEfHx8ZBiG2XEAAHAaDodDGRkZCgkJkcVy+c8TXKY0nDhxQqGhoWbHAADAaR09elR16tS57PMuUxp8fHwkXfyDfX19TU4DAIDzSE9PV2hoaOF76eW4TGn47SsJX19fSgMAANfgal/vMxESAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUC6UBAAAUS4lLw5o1a9S3b1+FhITIMAwtW7as2K9dv369bDab2rRpU+TxZ599VoZhFDnCwsJKGg0AAJShEpeGrKwshYeHa/r06SV6XWpqqoYOHaqbb775ks+3aNFCSUlJhce6detKGq3U5RXYlV9gNzsGAAAVgq2kL+jTp4/69OlT4l80ZswYDR48WFar9ZKfTthsNgUHBxf7fDk5OcrJySn8OT09vcSZrmT3iXQ9/dl23d4qRGNvbFSq5wYAwBmVy5yGDz/8UIcOHdKUKVMuO2b//v0KCQlRw4YNNWTIECUmJl7xnFOnTpWfn1/hERoaWqqZdyela9fxdL3+wy86cDKzVM8NAIAzKvPSsH//fk2cOFFz586VzXbpDzYiIyMVHR2tFStWaMaMGUpISFC3bt2UkZFx2fNOmjRJaWlphcfRo0dLNffdbWure9Mays236+nF21Vgd5Tq+QEAcDZlWhoKCgo0ePBgPffcc2ratOllx/Xp00f33nuvWrdurd69e+vrr79WamqqFi5ceNnXeHh4yNfXt8hRmgzD0NS7Wqmqh01bE1MVveFwqZ4fAABnU6alISMjQ7GxsRo/frxsNptsNpuef/55bd++XTabTT/++OMlX+fv76+mTZvqwIEDZRnvqkL8vfTP226QJL387V4dOZNlah4AAMxUpqXB19dXO3fuVHx8fOExZswYNWvWTPHx8YqMjLzk6zIzM3Xw4EHVqlWrLOMVy6COoercKFDZeXY9vXiH7HxNAQCopEpcGjIzMwsLgCQlJCQoPj6+cOLipEmTNHTo0Isnt1jUsmXLIkfNmjXl6empli1bytvbW5L05JNPavXq1Tp8+LA2bNig/v37y2q1atCgQaX0Z147wzD04l2t5eVm1aaEs/pk85UnaAIA4KpKXBpiY2MVERGhiIgISdKECRMUERGhyZMnS5KSkpKueufDHx07dkyDBg1Ss2bNdN999ykwMFAxMTGqUaNGSeOVibqBVfSPW5tJkl78eo+OnTtvciIAAMqf4XA4XOLz9vT0dPn5+SktLa3UJ0VKkt3u0H3vbVTskXPq1qS65ozsKMMwSv33AABQ3or7HsreE8VksRj67z2t5WGzaO3+01oUd8zsSAAAlCtKQwk0rFFVE3pdvHX0/77creS0bJMTAQBQfigNJTSqW0OFh/orIztf/2/pTrnItzsAAFwVpaGErBZDL9/TWm5WQyv3ntTn20+YHQkAgHJBabgGTYN89NhNTSRJUz7/Wacycq7yCgAAnB+l4RqNubGRmtfyVer5PD2zjK8pAACuj9JwjdysFr1yb7hsFkPf/pzC1xQAAJdHabgOzUN89djNF7+mmLz8Z51M524KAIDrojRcp7E3NlLL2r5Ku5Cnf3I3BQDAhVEartNvX1O4WQ39sOeklm47bnYkAADKBKWhFIQF++qJnhcXfXr2859Z9AkA4JIoDaVk9F8aqnUdP6Vn52vSkh18TQEAcDmUhlJis1r06r3hcrda9NO+U1rM3hQAABdDaShFTYJ8NOGWi19TPP/FbiWlXTA5EQAApYfSUMoe6tZQEXX9lZGTr398xt0UAADXQWkoZRf3pgiXu82iNb+c0qdbjpodCQCAUkFpKAONa1bVU7c0kyS98NUeHTt33uREAABcP0pDGRnZtYHa1aumzJx8/eOzHbLb+ZoCAODcKA1lxGox9Mq94fJ0s2j9gTOau+mI2ZEAALgulIYy1KC6tybeGiZJ+s/Xe3ToVKbJiQAAuHaUhjI2NKq+OjcKVHaeXX9ftF35BXazIwEAcE0oDWXMYjH08r3h8vGwaVtiqt5bc8jsSAAAXBNKQzmo7e+lKX9tIUma9sMv2n0i3eREAACUHKWhnNzdtrZuaR6kvAKHJiyMV05+gdmRAAAoEUpDOTEMQ/+5q5UCvd21NzlDr3+/3+xIAACUCKWhHFWv6qF/928lSZq55qBiD581OREAAMVHaShnt7YM1l1ta8vukP6+aLuycvLNjgQAQLFQGkwwpW8L1fLz1JEz5zX1mz1mxwEAoFgoDSbw83LTy/eES5LmxiRq9S+nTE4EAMDVURpM0rVJdQ2LqidJenrxdqWdzzM5EQAAV0ZpMNHEPjeoYXVvpaTn6F/Ld5kdBwCAK6I0mMjL3apX7wuX1WLo8+0ntDz+uNmRAAC4LEqDySLqVtO4Ho0lSc8s26UTqRdMTgQAwKVRGiqAR29qrPBQf2Vk5+vvC7fLbneYHQkAgD+hNFQAblaLpg1oIy83qzYeOqPZ6xPMjgQAwJ9QGiqIBtW99cwdN0iS/rtin/Yms6kVAKBioTRUIIM71tXNYTWVW2DXEwvY1AoAULFQGioQwzD04t2tCze1evW7X8yOBABAIUpDBVPDx0Mv3t1akvT+2kPaePCMyYkAALiI0lAB9WoepIEdQuVwSH9fGK+0C6wWCQAwH6WhgvrXHc1VL7CKTqRlazKrRQIAKgBKQwXl7WHT6wPayGoxtDz+hD7ffsLsSACASo7SUIG1/d/VIpfuZLVIAICpKA0V3G+rRaZn52vCwngVsFokAMAklIYKzs1q0RsD2qiKu1Uxh87qvTUHzY4EAKikKA1OoH51bz371xaSpNe++0Xbj6aaGwgAUClRGpzEve3q6PZWtZRvd+iJT+OVlZNvdiQAQCVDaXAShmHoP/1bqZafpxJOZ+m5L342OxIAoJKhNDgRvypuen1AGxmGtDD2mL7emWR2JABAJUJpcDKdGgZqbPdGkqRJS7gNEwBQfigNTuhvvZoqvI6f0i7kcRsmAKDcUBqckJvVomkDI7gNEwBQrigNTqpBdW8925fbMAEA5YfS4MTubV9Ht7UK5jZMAEC5oDQ4McMwNLV/a27DBACUC0qDk/vjbZhfsBsmAKCMlLg0rFmzRn379lVISIgMw9CyZcuK/dr169fLZrOpTZs2f3pu+vTpql+/vjw9PRUZGanNmzeXNFql1alhoMbdeHE3zH8u2amjZ8+bnAgA4IpKXBqysrIUHh6u6dOnl+h1qampGjp0qG6++eY/Pffpp59qwoQJmjJlirZu3arw8HD17t1bJ0+eLGm8SuuJnk3Utq6/MnLy9ej8bcorsJsdCQDgYgyHw3HNN/kbhqGlS5eqX79+Vx07cOBANWnSRFarVcuWLVN8fHzhc5GRkerQoYPefvttSZLdbldoaKgeffRRTZw4sVhZ0tPT5efnp7S0NPn6+l7Ln+P0jp49r9veXKuM7Hw9cmMjPX1rmNmRAABOoLjvoeUyp+HDDz/UoUOHNGXKlD89l5ubq7i4OPXs2fP3UBaLevbsqY0bN172nDk5OUpPTy9yVHahAVX04l2tJUkzVh/U+gOnTU4EAHAlZV4a9u/fr4kTJ2ru3Lmy2Wx/ev706dMqKChQUFBQkceDgoKUnJx82fNOnTpVfn5+hUdoaGipZ3dGt7eupUEdQ+VwSE98Gq8zmTlmRwIAuIgyLQ0FBQUaPHiwnnvuOTVt2rRUzz1p0iSlpaUVHkePHi3V8zuzyXe0UJOaVXUqI0dPLtouO8tMAwBKQZmWhoyMDMXGxmr8+PGy2Wyy2Wx6/vnntX37dtlsNv3444+qXr26rFarUlJSirw2JSVFwcHBlz23h4eHfH19ixy4yMvdqrcGR8jdZtFP+05p9voEsyMBAFxAmZYGX19f7dy5U/Hx8YXHmDFj1KxZM8XHxysyMlLu7u5q166dVq5cWfg6u92ulStXKioqqizjubSwYF/96/YbJEkvrdirXcfTTE4EAHB2f55kcBWZmZk6cOBA4c8JCQmKj49XQECA6tatq0mTJun48eOaM2eOLBaLWrZsWeT1NWvWlKenZ5HHJ0yYoGHDhql9+/bq2LGjpk2bpqysLI0YMeI6/jTc36me1u4/re92p+jR+dv0xaNdVdWjxP+VAwAg6Ro+aYiNjVVERIQiIiIkXXzDj4iI0OTJkyVJSUlJSkxMLNE5BwwYoFdeeUWTJ09WmzZtFB8frxUrVvxpciRKxjAM/fee35eZnrKcZaYBANfuutZpqEhYp+HyNiec1cCZG2V3SK8PCFf/iDpmRwIAVCAVap0GmKtjgwA9dnMTSdL/W7pLh05lmpwIAOCMKA2VxKM3NVGnhgE6n1ugcfO2KTuvwOxIAAAnQ2moJKwWQ28MjFCgt7v2JKXrha92mx0JAOBkKA2VSJCvp14b0EaSNDcmUV/tSDI3EADAqVAaKpnuTWvokRsbSZImfrZDR85kmZwIAOAsKA2V0IReTdW+XjVl5ORr/LxtyslnfgMA4OooDZWQzWrRm4Mi5F/FTTuPp+mlb/aZHQkA4AQoDZVUiL+XXrknXJI0e32Cvvv58juKAgAgURoqtZ7NgzSqawNJ0pOLtuvYufMmJwIAVGSUhkru6VvDFB7qr/TsfD02f5vyCuxmRwIAVFCUhkrO3WbR24Mi5ONp09bEVL3yHfMbAACXRmmAQgOq6OV7WkuS3lt9SCv3pJicCABQEVEaIEm6tWUtDe9cX5I0YeF2HT3L/AYAQFGUBhT65203qE2ov9Iu5GncvK2s3wAAKILSgELuNoumD2kr/ypu2nEsTS98ucfsSACACoTSgCJq+3tp2oA2Mgzp45gjWh5/3OxIAIAKgtKAP7mxWU092qOxJGnSkp3an5JhciIAQEVAacAlPd6zqbo0DtT53AKN/WSrsnLyzY4EADAZpQGXZLUYemNghIJ8PXTgZKYmLtkph8NhdiwAgIkoDbis6lU9NH1wW1kthr7YfkJzY46YHQkAYCJKA66off0ATeoTJkl6/svdij+aam4gAIBpKA24qge7NlDvFkHKK3Bo3CdbdS4r1+xIAAATUBpwVYZh6OV7w1UvsIqOp17QhIXxstuZ3wAAlQ2lAcXi6+mmGUPaycNm0U/7Tmn6TwfMjgQAKGeUBhRb8xBfvdCvpSTptR9+0dr9p0xOBAAoT5QGlMi97UM1qGOoHA7psfnbdDz1gtmRAADlhNKAEpvSt4Va1vbVufN5euQTNrYCgMqC0oAS83SzasaQdvLzctP2o6n691dsbAUAlQGlAdckNKCKpg1oI0mas/GIlm1jYysAcHWUBlyzHmE19dhNv29stS+Zja0AwJVRGnBdHu/ZVN2aVNeFvAKNnRunjOw8syMBAMoIpQHXxWoxNG1AG9Xy89Sh01l6evEONrYCABdFacB1C6zqoelD2srNauibXcmatS7B7EgAgDJAaUCpaFu3mv51R3NJ0tRv9mrToTMmJwIAlDZKA0rNA53q6c42ISqwOzRu3jYlp2WbHQkAUIooDSg1hmFo6l2tFBbso9OZORr7SRwLPwGAC6E0oFRVcbfpvQfaydfTpm2Jqfq/L3ebHQkAUEooDSh19QK99cbACBmGNDcmUQtjj5odCQBQCigNKBM9wmrqiZubSpKeWbZLO4+lmZwIAHC9KA0oM4/e1Fg9b6ip3Hy7xsyN09msXLMjAQCuA6UBZcZiMfTagDZqUN1bx1Mv6NH5W5VfYDc7FgDgGlEaUKZ8Pd307v3tVMXdqvUHzuiV734xOxIA4BpRGlDmmgX76L/3tJYkvbv6oL7ZmWRyIgDAtaA0oFzc0TpED/+loSTpyUXbtT+FHTEBwNlQGlBunu7dTFENA5WVW6DRH8cpnR0xAcCpUBpQbmxWi94eHKGQX3fEfGJBvArs7IgJAM6C0oByFVjVQzOHtpeHzaIf957Uq9/tMzsSAKCYKA0ody1r+xVOjHxn1UF9sf2EyYkAAMVBaYAp7mxTW6O7X5wY+dTi7dp1nBUjAaCiozTANE/3DlP3pjWUnWfX6I/jdCYzx+xIAIAroDTANFaLoTcHRRSuGDn2k63KY8VIAKiwKA0wlZ+Xm94f2k5VPWzanHBWz3/BVtoAUFFRGmC6xjV9NG1AGxmG9HHMEc3blGh2JADAJVAaUCH0bB6kJ29pJkma8vkuxR4+a3IiAMAfURpQYTxyYyPd3qqW8gocGjN3q06kXjA7EgDgf1AaUGEYhqGX722tG2r56nRmjh7+OFYXcgvMjgUA+BWlARVKFXebZj7QTgHe7tp1PF1Pf7ZDDgdLTQNARVDi0rBmzRr17dtXISEhMgxDy5Ytu+L4devWqUuXLgoMDJSXl5fCwsL0+uuvFxnz7LPPyjCMIkdYWFhJo8FFhAZU0YwhbWWzGPpi+wm9s+qg2ZEAAJJsJX1BVlaWwsPDNXLkSN11111XHe/t7a3x48erdevW8vb21rp16zR69Gh5e3vr4YcfLhzXokUL/fDDD78Hs5U4GlxIZMNAPX9nS/1z6U698t0+NQ3yUa/mQWbHAoBKrcTvzH369FGfPn2KPT4iIkIRERGFP9evX19LlizR2rVri5QGm82m4ODgksaBCxscWVd7k9M1Z+MRPbFgm5aO66KmQT5mxwKASqvc5zRs27ZNGzZsUPfu3Ys8vn//foWEhKhhw4YaMmSIEhOvfK9+Tk6O0tPTixxwPf+6o7miGgYqK7dAoz6K1bmsXLMjAUClVW6loU6dOvLw8FD79u01btw4jRo1qvC5yMhIRUdHa8WKFZoxY4YSEhLUrVs3ZWRkXPZ8U6dOlZ+fX+ERGhpaHn8Gypmb1aLpQ9oqNMBLiWfP6xGWmgYA0xiO65iabhiGli5dqn79+l11bEJCgjIzMxUTE6OJEyfq7bff1qBBgy45NjU1VfXq1dNrr72mBx988JJjcnJylJPz+wZH6enpCg0NVVpamnx9fa/p70HFtS85Q3e9s15ZuQUaFlVPz93Z0uxIAOAy0tPT5efnd9X30HKbbdigQQNJUqtWrZSSkqJnn332sqXB399fTZs21YEDBy57Pg8PD3l4eJRJVlQ8zYJ99PqANnr44zh9tPGImgX7anBkXbNjAUClYso6DXa7vcinBH+UmZmpgwcPqlatWuWYChXdLS2C9eQtTSVJk5fv0qZDZ0xOBACVS4lLQ2ZmpuLj4xUfHy/p4tcO8fHxhRMXJ02apKFDhxaOnz59ur744gvt379f+/fv16xZs/TKK6/o/vvvLxzz5JNPavXq1Tp8+LA2bNig/v37y2q1XvaTCFRe43o01h2taynf7tDYT7bq6NnzZkcCgEqjxF9PxMbGqkePHoU/T5gwQZI0bNgwRUdHKykpqcidD3a7XZMmTVJCQoJsNpsaNWqkl156SaNHjy4cc+zYMQ0aNEhnzpxRjRo11LVrV8XExKhGjRrX87fBBRmGoZfvCdfhM1nadTxdD360RZ+N7SwfTzezowGAy7uuiZAVSXEnccA1JKVd0J1vr9fJjBzd2KyGPhjaXjYrq6IDwLUo7nso/ysLp1TLz0sfDGsvTzeLVu07pRe+2mN2JABweZQGOK3Wdfz1+n1tJEnRGw7r45gj5gYCABdHaYBT69Oqlp7q3UyS9OznP2vt/lMmJwIA10VpgNN75MZGuqttbRXYHXrkk606cPLyK4kCAK4dpQFOzzAMTb2rlTrUr6aM7HyNjI7VWfaoAIBSR2mAS/CwWfXeA+0L96gY83GccvILzI4FAC6F0gCXEeDtrtnDOsjHw6bNh8/qn0t2yUXuKAaACoHSAJfSJMhHbw9pK6vF0Gdbj+mdVQfNjgQALoPSAJfTvWkNPdu3uSTp5W/36eudSSYnAgDXQGmAS3ogqr6Gd64vSfrbp/GKP5pqah4AcAWUBrisf93RXDeF1VROvl2jPorVsXNsbgUA14PSAJdltRh6c1CEwoJ9dDozRw9GxyojO8/sWADgtCgNcGlVPWyaPbyDavh4aF9KhsbP26b8ArvZsQDAKVEa4PJC/L0069fNrVb/ckrPfbGbWzEB4BpQGlAptK7jrzcGRsgwpI9jjih6w2GzIwGA06E0oNLo3SJYk/qESZL+78vdWrknxeREAOBcKA2oVB7q1lCDOobK7pAenb9Nu0+kmx0JAJwGpQGVimEYev7OlurSOFDncwv04EdblJKebXYsAHAKlAZUOm5Wi94Z0k6NangrKS1bI6O3KCsn3+xYAFDhURpQKfl5uSl6REdVr+qun0+ka/y8rdyKCQBXQWlApRUaUEUfDOsgTzeLftp3SlM+/5lbMQHgCigNqNTahPpr2oCLt2J+silR7689ZHYkAKiwKA2o9G5tGaxnbr+4K+Z/vt6rr3awKyYAXAqlAZD0YNcGv++KuTBecUfOmhsIACogSgPwq3/d0Vy9mgcp99ddMQ+fzjI7EgBUKJQG4FdWi6E3BrZReB0/nTufp+EfbtbZrFyzYwFAhUFpAP5HFXebPhjWQXWqeenwmfN6aE6ssvMKzI4FABUCpQH4gxo+Hooe0UG+njbFHTmnvy/cLrudWzEBgNIAXELjmj5674H2crMa+mpnkv7z9R6zIwGA6SgNwGVENQrUK/eGS5I+WJegD1jDAUAlR2kAruDONrX1z9subqf9wld7tDz+uMmJAMA8lAbgKh7q1lAjutSXJD25aLs2HDhtbiAAMAmlAbgKwzD0r9ub6/bWtZRX4NDDH8dp94l0s2MBQLmjNADFYLEYevXecEU2CFBmTr6Gf7hZR8+eNzsWAJQrSgNQTJ5uVs0c2l5hwT46mZGjYR9u1jkWfwJQiVAagBLw83JT9IiOCvHz1KFTWRrF4k8AKhFKA1BCwX6eih7ZsXDxp0fnb1N+gd3sWABQ5igNwDVoGuSjD4Z1kLvNou93p2jy5z/L4WDVSACujdIAXKOODQL05sA2Mgxp3qZEvbnygNmRAKBMURqA63Bry1p6/q8tJEmv//CL5m1KNDkRAJQdSgNwnR6Iqq9Hb2osSXpm2U59+3OyyYkAoGxQGoBSMKFXUw3sECq7Q3ps/jZtOXzW7EgAUOooDUApMAxDL/RrqZ43BCkn364Ho7doX3KG2bEAoFRRGoBSYrNa9NagCLWrV03p2fkaNnuzjqdeMDsWAJQaSgNQirzcrZo1rL0a16yq5PRsDZu9WannWTUSgGugNAClzL+Ku+aM7KhgX08dOJmpkdFbdCGXVSMBOD9KA1AGQvy9NOfBi6tGbk1M1fh5W1k1EoDTozQAZaRpkI9mDe8gD5tFK/ee1MQlO2W3s2okAOdFaQDKUIf6AXprUISsFkOL447pP1/vYblpAE6L0gCUsVtaBOulu1tLkj5Yl6B3Vh00OREAXBtKA1AO7mlXR8/cfoMk6eVv9+njmCMmJwKAkqM0AOVkVLeGhctNT16+S59vP2FyIgAoGUoDUI4m9GqqBzrVk8MhTfg0Xj/tO2l2JAAoNkoDUI4Mw9Bzf22hv4aHKN/u0Ni5cYplnwoAToLSAJQzi8XQq/eF68ZmNZSdZ9eI6C3afSLd7FgAcFWUBsAEblaLZgxpp/b1qikjO19DZ2/W4dNZZscCgCsqcWlYs2aN+vbtq5CQEBmGoWXLll1x/Lp169SlSxcFBgbKy8tLYWFhev311/80bvr06apfv748PT0VGRmpzZs3lzQa4FS83K2aNbyDbqjlq9OZObp/1iYlp2WbHQsALqvEpSErK0vh4eGaPn16scZ7e3tr/PjxWrNmjfbs2aNnnnlGzzzzjGbOnFk45tNPP9WECRM0ZcoUbd26VeHh4erdu7dOnmSSGFybn5eb5ozsqPqBVXTs3AU9MGuTzmaxwRWAislwXMfydIZhaOnSperXr1+JXnfXXXfJ29tbH3/8sSQpMjJSHTp00Ntvvy1JstvtCg0N1aOPPqqJEycW65zp6eny8/NTWlqafH19S5QHMNvRs+d177sblZyerdZ1/PTJqEj5eLqZHQtAJVHc99Byn9Owbds2bdiwQd27d5ck5ebmKi4uTj179vw9lMWinj17auPGjZc9T05OjtLT04scgLMKDaiiuaM6KsDbXTuOpWnUR7HKzmNnTAAVS7mVhjp16sjDw0Pt27fXuHHjNGrUKEnS6dOnVVBQoKCgoCLjg4KClJycfNnzTZ06VX5+foVHaGhomeYHylrjmj76aERHVfWwaVPCWY37ZKvy2BkTQAVSbqVh7dq1io2N1bvvvqtp06Zp/vz513W+SZMmKS0trfA4evRoKSUFzNOqjp9mDWtfuDPmk4u2szMmgArDVl6/qEGDBpKkVq1aKSUlRc8++6wGDRqk6tWry2q1KiUlpcj4lJQUBQcHX/Z8Hh4e8vDwKNPMgBkiGwbq3fvb6aE5sVoef0I+njb9350tZRiG2dEAVHKmrNNgt9uVk5MjSXJ3d1e7du20cuXKIs+vXLlSUVFRZsQDTNcjrKZeH9BGhiHNjUnUy9/uMzsSAJT8k4bMzEwdOHCg8OeEhATFx8crICBAdevW1aRJk3T8+HHNmTNH0sX1F+rWrauwsDBJF9d5eOWVV/TYY48VnmPChAkaNmyY2rdvr44dO2ratGnKysrSiBEjrvfvA5xW3/AQZWTn659Ld+qdVQfl6+WmMd0bmR0LQCVW4tIQGxurHj16FP48YcIESdKwYcMUHR2tpKQkJSYmFj5vt9s1adIkJSQkyGazqVGjRnrppZc0evTowjEDBgzQqVOnNHnyZCUnJ6tNmzZasWLFnyZHApXN4Mi6ysjO09Rv9urFb/bKx9OmIZH1zI4FoJK6rnUaKhLWaYAre/nbvZr+00EZhvTKPeG6u10dsyMBcCEVdp0GACX35C3NNLxzfTkc0lOLt+vLHSfMjgSgEqI0AE7AMAxNvqO5BnYIld0hPbEgXt/vTrn6CwGgFFEaACdhsRj6d/9W6h9RW/l2h8Z9slVrfjlldiwAlQilAXAiVouhl+9prT4tg5VbYNfDH8cq5tAZs2MBqCQoDYCTsVktemNghG4Kq6nsPLtGRm9R3JFzZscCUAlQGgAn5G6z6J0hbdW1cXWdzy3Q8A83a9fxNLNjAXBxlAbASXm6WTVzaDt1rB+gjOx83T9rk/YlZ5gdC4ALozQATqyKu02zhrdXeKi/Us/nacgHMTp4KtPsWABcFKUBcHI+nm6aM6Kjmtfy1enMXA1+P0ZHzmSZHQuAC6I0AC7Ar4qbPn6wo5oGVVVKeo4Gv79Jx86dNzsWABdDaQBcRGBVD80dFamG1b11PPWCBr+/SUlpF8yOBcCFUBoAF1LTx1PzHuqkeoFVlHj2vIa8v0kn07PNjgXARVAaABcT7HexONT299Kh01ka8sEmncnMMTsWABdAaQBcUG1/L81/qJOCfT21/2Sm7p+1Wannc82OBcDJURoAF1U3sIrmPRSp6lU9tCcpXQ/M2qz07DyzYwFwYpQGwIU1rFFV8x6KVIC3u3YeT9Ow2ZuVmZNvdiwATorSALi4pkE+mvtgpPy83LQtMVUjP9yiLIoDgGtAaQAqgeYhvpr7YKR8PG3afPisRkZv0flcigOAkqE0AJVEqzp+mjOyo3w8bNqUcFYPRsfqQm6B2bEAOBFKA1CJRNStpuiRHeXtbtXGQ2f00JxYZedRHAAUD6UBqGTa1aumj34tDusOnKY4ACg2SgNQCbWvH6DokR1Vxd2qtftP6+GP4ygOAK6K0gBUUh3qB+jD4R3k5WbVml9OaezcOOXkUxwAXB6lAajEIhsGavbwDvJ0s+infaf0yNytFAcAl0VpACq5qEaBmj2sgzxsFq3ce1LjPtmm3Hy72bEAVECUBgDq3Li6Zv1aHH7Yk6JHPuETBwB/RmkAIEnq2qS63h/avrA4jGFyJIA/oDQAKPSXpjU0a9jvcxxGUxwA/A9KA4Aiujaprtm/3lWx+pdTGvURK0cCuIjSAOBPOjeqrugRHVTl1wWg2KsCgERpAHAZkQ0DNWdkR1X1sGnjoTMazu6YQKVHaQBwWe3rB2jOgxc3udqccFbDZm9WRnae2bEAmITSAOCK2tatprmjIuXraVPskXMaOnuz0ikOQKVEaQBwVeGh/pr3UCf5eblpW2KqHvhgk9LOUxyAyobSAKBYWtb207yHIlWtipu2H0vT4A9idC4r1+xYAMoRpQFAsbUI8dOCh6NUvaq7fj6RrkHvx+h0Zo7ZsQCUE0oDgBJpFuyjBQ93Uk0fD+1NztDAmTE6mZ5tdiwA5YDSAKDEGtf00aejo1TLz1MHTmZq4MwYJadRHABXR2kAcE0aVPfWpw9Hqba/lw6dztJ9723UsXPnzY4FoAxRGgBcs7qBVfTp6E6qG1BFiWfPa8B7MUo8Q3EAXBWlAcB1qVOtihaOjlLD6t46nnpBA2ZuVMLpLLNjASgDlAYA1y3Yz1MLHu6kJjWrKiktWwPe26gDJzPMjgWglFEaAJSKmr6emv9wJ4UF++hkRo4GvBejn0+kmR0LQCmiNAAoNdWremj+Q53UsravzmTlatDMGMUdOWd2LAClhNIAoFRV83bXvIc6qX29akrPztcDszZpw4HTZscCUAooDQBKna+nm+Y82FHdmlTX+dwCDY/eopV7UsyOBeA6URoAlIkq7jZ9MKy9ercIUm6+XaM/jtMX20+YHQvAdaA0ACgzHjarpg9uq/4RtZVvd+ixBdv06ZZEs2MBuEaUBgBlyma16NV7wzUksq4cDukfn+3UrHUJZscCcA0oDQDKnMVi6IV+LfXwXxpKkv7vy916c+V+ORwOk5MBKAlKA4ByYRiGJvUJ04ReTSVJr33/i6Z+s5fiADgRSgOAcmMYhh67uYn+dUdzSdLMNYf0z6W7VGCnOADOgNIAoNw92LWB/nt3a1kMaf7mRD3xabzyCuxmxwJwFZQGAKa4r0Oo3hrUVm5WQ19sP6HRH8cpO6/A7FgAroDSAMA0t7eupZlD28vDZtGPe09q+IeblZmTb3YsAJdBaQBgqh7NamrOyI6q6mFTzKGzGvJ+jM5l5ZodC8AlUBoAmC6yYaDmPRSpalXctP1YmgbM3KiT6dlmxwLwByUuDWvWrFHfvn0VEhIiwzC0bNmyK45fsmSJevXqpRo1asjX11dRUVH69ttvi4x59tlnZRhGkSMsLKyk0QA4sdZ1/LVwdJRq+njol5RM3fPuRh09e97sWAD+R4lLQ1ZWlsLDwzV9+vRijV+zZo169eqlr7/+WnFxcerRo4f69u2rbdu2FRnXokULJSUlFR7r1q0raTQATq5JkI8Wj+msugFVlHj2vO55d4N+SckwOxaAX9lK+oI+ffqoT58+xR4/bdq0Ij//5z//0fLly/XFF18oIiLi9yA2m4KDg0saB4CLqRtYRYvGROmBWZv0S0qm7n13o2YPb6929QLMjgZUeuU+p8FutysjI0MBAUX/B2D//v0KCQlRw4YNNWTIECUmXnlTm5ycHKWnpxc5ALiGIF9PLRwdpbZ1/ZV2IU9DPtikn/aeNDsWUOmVe2l45ZVXlJmZqfvuu6/wscjISEVHR2vFihWaMWOGEhIS1K1bN2VkXP5jyalTp8rPz6/wCA0NLY/4AMqJfxV3zR0VqRub1VB2nl2j5sRqydZjZscCKjXDcR0LvxuGoaVLl6pfv37FGj9v3jw99NBDWr58uXr27HnZcampqapXr55ee+01Pfjgg5cck5OTo5ycnMKf09PTFRoaqrS0NPn6+pbo7wBQceUV2PX04h1auu24JOmZ22/QqG4NTU4FuJb09HT5+fld9T20xHMartWCBQs0atQoLVq06IqFQZL8/f3VtGlTHThw4LJjPDw85OHhUdoxAVQwbr9urR3g7a5Z6xL0wld7dCYrV0/3bibDMMyOB1Qq5fL1xPz58zVixAjNnz9ft99++1XHZ2Zm6uDBg6pVq1Y5pANQ0Vkshp65/Qb949aLt2LPWHVQ//hsh/LZrwIoVyUuDZmZmYqPj1d8fLwkKSEhQfHx8YUTFydNmqShQ4cWjp83b56GDh2qV199VZGRkUpOTlZycrLS0tIKxzz55JNavXq1Dh8+rA0bNqh///6yWq0aNGjQdf55AFyFYRgae2MjvXR3K1kMaWHsMY39ZCv7VQDlqMSlITY2VhEREYW3S06YMEERERGaPHmyJCkpKanInQ8zZ85Ufn6+xo0bp1q1ahUejz/+eOGYY8eOadCgQWrWrJnuu+8+BQYGKiYmRjVq1Ljevw+AixnQoa5m3N9O7jaLvt+doqGzNivtQp7ZsYBK4bomQlYkxZ3EAcA1xBw6o4c+ilVGTr7Cgn300ciOCvL1NDsW4JSK+x7K3hMAnFKnhoH6dHSUavh4aG9yhu6esUGHTmWaHQtwaZQGAE6reYivloztrPqBVXTs3AXd8+5GbT+aanYswGVRGgA4tdCAKlo8trNa1fbT2axcDXo/Rmv3nzI7FuCSKA0AnF71qh6a/3AndW1cXedzCzQyeouWxx83OxbgcigNAFxCVQ+bZg/voL7hIcorcOjxBfGavS7B7FiAS6E0AHAZ7jaL3hjQRsM715ckPf/lbr20Yq9c5CYxwHSUBgAuxWIxNKVvcz3Vu5mki6tHPrloh/JYPRK4bpQGAC7HMAyN69FY/727tawWQ59tPaYHP4pVVk6+2dEAp0ZpAOCy7usQqveHtpOXm1VrfjmlgTNjdCoj5+ovBHBJlAYALu2msCDNf7iTArzdtfN4mu6esUEJp7PMjgU4JUoDAJfXJtRfn43trLoBVZR49rzunrFB8SwCBZQYpQFApdCgurc++99FoGbG6Me9KWbHApwKpQFApVHDx0MLHu6kvzStoQt5BXpoTpwWbE68+gsBSKI0AKhkvD1smjWsve5uW0cFdocmLtmpaT/8wloOQDFQGgBUOm5Wi165t7XG9WgkSZr2w349tXiHcvNZywG4EkoDgErJMAw91TtM/+7fUhZDWhx3TMM/3Ky0C3lmRwMqLEoDgEptSGQ9zRreQd7uVm04eEb3zNigY+fOmx0LqJAoDQAqvR7NamrhmCgF+Xpo/8lM9Zu+QTuOpZodC6hwKA0AIKlFiJ+WjeuisGAfnc7M0YD3YvT9bm7JBP4XpQEAflXLz0uLxkQV3pL58Mexil7P9trAbygNAPA/fDzdNGtYew3qGCqHQ3r2i916/ovdKrBzSyZAaQCAP3CzWvSf/q30j1vDJEmz1ydozNw4nc9ll0xUbpQGALgEwzA09sZGemtQhNxtFn2/O0UDZ8boZEa22dEA01AaAOAK+oaHaN6oSFWr4qYdx9LUf/oG7UvOMDsWYApKAwBcRfv6AVr6SBc1rO6t46kXdM+MDVrzyymzYwHljtIAAMVQv7q3ljzSWR0bBCgjJ18jorew2RUqHUoDABSTfxV3ffxgR/WPqF242dVLK/bKzp0VqCQoDQBQAh42q167L1yP39xEkjRj1UE9On+bsvMKTE4GlD1KAwCUkGEY+luvpnrtvnC5WQ19tTNJg96P0amMHLOjAWWK0gAA1+iutnU0Z2SkfD1t2paYqn7T12tPUrrZsYAyQ2kAgOsQ1ShQS8d1UYP/ubPiB/asgIuiNADAdWpUo6qWPtJZnRsFKiu3QA99HKv3Vh+Uw8EESbgWSgMAlAL/Ku76aGRHDY6sK4dDmvrNXj21eIdy8pkgCddBaQCAUuJmtejf/Vrq2b7NZTGkxXHH9MAHm3UmkwmScA2UBgAoRYZhaHiXBvpwREf5eNi0+fBZ9XtnvX5JYelpOD9KAwCUge5Na2jpuM6qG1BFR89e0F3vbNBPe0+aHQu4LpQGACgjjWv6aPm4LopsEKDMnHw9+NEWfbD2EBMk4bQoDQBQhqp5u+vjByM1sEOo7A7pha/26B+f7VBuvt3saECJURoAoIy52yyaelcrTb7j4gTJhbHHdP8Hm5ggCadDaQCAcmAYhkZ2baDZwzsUTpC8c/p67U1mBUk4D0oDAJSjG5vV1NJxnVUvsIqOnbugu99hBUk4D0oDAJSzxjV9tOyRLopq+PsKku+ygiScAKUBAExQzdtdcx7sqCG/riD54jd79fdF29liGxUapQEATOJmteiFfi313F9byGoxtGTrcQ2cGaOU9GyzowGXRGkAABMZhqFhnesrekQH+Xm5Kf5oqvq+tU5bE8+ZHQ34E0oDAFQA3ZrU0Ofju6hpUFWdzMjRwPdi9OmWRLNjAUVQGgCggqgX6K0lj3RR7xZByi2w6x+f7dTk5buUV8BCUKgYKA0AUIFU9bBpxpB2mtCrqSRpzsYjLASFCoPSAAAVjMVi6LGbm+j9oe1V1cOmTQln9de312vX8TSzo6GSozQAQAXVq3mQlo3rrAbVvXU89YLueXeDlscfNzsWKjFKAwBUYI1r+mjZuC66sVkNZefZ9fiCeP3n6z3KZ54DTEBpAIAKzs/LTbOGddDYGxtJkmauOaQR0VuUej7X5GSobCgNAOAErBZD/7g1TNMHt5WXm1Vr959W37fXaU8SG16h/FAaAMCJ3N66lpY80lmhAV46evaC7npng77cccLsWKgkKA0A4GRuqOWrL8Z3Vbcm1XUhr0Dj523Ti9/sVYGdDa9QtigNAOCE/Ku468PhHTT6Lw0lSe+uPsg8B5S5EpeGNWvWqG/fvgoJCZFhGFq2bNkVxy9ZskS9evVSjRo15Ovrq6ioKH377bd/Gjd9+nTVr19fnp6eioyM1ObNm0saDQAqFZvVokm33aA3BraRp5tFa345pb++vV57k5nngLJR4tKQlZWl8PBwTZ8+vVjj16xZo169eunrr79WXFycevToob59+2rbtm2FYz799FNNmDBBU6ZM0datWxUeHq7evXvr5MmTJY0HAJXOnW1q67OxnVXb30uJZ8+r//QN+mI78xxQ+gyHw3HNX4IZhqGlS5eqX79+JXpdixYtNGDAAE2ePFmSFBkZqQ4dOujtt9+WJNntdoWGhurRRx/VxIkTi3XO9PR0+fn5KS0tTb6+viXKAwCu4GxWrh6dv1XrD5yRJI3q2kAT+4TJZuWbaFxZcd9Dy/1fkt1uV0ZGhgICAiRJubm5iouLU8+ePX8PZbGoZ8+e2rhx42XPk5OTo/T09CIHAFRmAd7u+mhER43pfnE9hw/WJej+WZt0mn0rUErKvTS88soryszM1H333SdJOn36tAoKChQUFFRkXFBQkJKTky97nqlTp8rPz6/wCA0NLdPcAOAMbFaLJvYJ04whbeXtblXMobPq+9Y6bUs8Z3Y0uIByLQ3z5s3Tc889p4ULF6pmzZrXda5JkyYpLS2t8Dh69GgppQQA59enVS0tG9dFDat7KyktWwPei9H8zYlmx4KTK7fSsGDBAo0aNUoLFy4s8lVE9erVZbValZKSUmR8SkqKgoODL3s+Dw8P+fr6FjkAAL9rEuSj5eO76JbmQcotsGvSkp36x+Idys4rMDsanFS5lIb58+drxIgRmj9/vm6//fYiz7m7u6tdu3ZauXJl4WN2u10rV65UVFRUecQDAJfl4+mmd+9vp6d6N5NhSJ/GHtV9723UsXPnzY4GJ1Ti0pCZman4+HjFx8dLkhISEhQfH6/ExIsfe02aNElDhw4tHD9v3jwNHTpUr776qiIjI5WcnKzk5GSlpf2+L/yECRP0/vvv66OPPtKePXs0duxYZWVlacSIEdf55wEALBZD43o01kcjOsq/ipt2HEvTHW+t00/7uK0dJVPiWy5XrVqlHj16/OnxYcOGKTo6WsOHD9fhw4e1atUqSdKNN96o1atXX3b8b95++229/PLLSk5OVps2bfTmm28qMjKy2Lm45RIAru7o2fMaN2+rdhy7+H/cHr2psZ7o2VRWi2FyMpipuO+h17VOQ0VCaQCA4snJL9D/fblbc2MufkLcpXGg3hgYoepVPUxOBrNU2HUaAADm8rBZ9UK/Vpo2oI283Kxaf+CM7nhznWIPnzU7Gio4SgMAVFL9Imrr8/Fd1KiGt5LTszVwZow+WHtILvIBNMoApQEAKrEmQT76fHxX9Q0PUb7doRe+2qOxc7cqPTvP7GiogCgNAFDJeXvY9ObANnr+zhZysxpa8XOy/vrWOv18Iu3qL0alQmkAAMgwDA2Nqq9FYy7ulnn4zHnd9c4GLdzCarv4HaUBAFCoTai/vny0q25sVkM5+XY9/dkOPblouy7ksookKA0AgD+o5u2u2cM66KnezWQxpMVxx9T/nfU6eCrT7GgwGaUBAPAnv60iOXdUpKpX9dDe5Az99a11+nLHCbOjwUSUBgDAZXVuVF1fP9ZVkQ0ClJVboPHztmnK8l3KyefrisqI0gAAuKKavp76ZFSkHrmxkSTpo41HdN97MTp6lk2vKhtKAwDgqmxWi56+NUyzh7eXn5ebth9N1e1vrtWKXUlmR0M5ojQAAIrtprAgffVYV0XU9Vd6dr7GzN2qyct3KTuPrysqA0oDAKBE6lSrooWjozS6e0NJ0pyNR3TXOxt0iLsrXB6lAQBQYm5Wiyb1uUHRIzoo0Ntdu5PSdcdb67Rk6zGzo6EMURoAANfsxmY19fXj3dSpYYDO5xZowsLtenLRdp3PzTc7GsoApQEAcF2CfD31yahO+lvPpoWLQfV9a532JKWbHQ2ljNIAALhuVouhx3s20byHOinI10MHT2XpzunrNTfmCFttuxBKAwCg1HRqGKivH+umHs1qKDffrmeW7dIjn2xV2gW22nYFlAYAQKkKrOqhWcM66Jnbb5Cb1dA3u5J12xtrtTXxnNnRcJ0oDQCAUmexGBrVraEWj+msugFVdDz1gu57d6PeXX1QdjtfVzgrSgMAoMyEh/rrq8e6qm94iPLtDr34zV4N+3CzTmXkmB0N14DSAAAoUz6ebnpzYBu9dHcrebpZtHb/ad325lqt23/a7GgoIUoDAKDMGYahAR3q6ovxXdUsyEenMnL0wOxN+u+KvcorsJsdD8VEaQAAlJsmQT5aPr6LBkfWlcMhvbPqoO6ZsUGHT2eZHQ3FQGkAAJQrTzer/tO/lWYMaXtxx8xjabr9zbVaFHuUNR0qOEoDAMAUfVrV0je/LkGdlVugpxbv0Pj525R2njUdKipKAwDANCH+XvpkVCc91buZbBZDX+1IUp831mjToTNmR8MlUBoAAKayWgyN69FYn43trPqBVXQiLVuD3o/RK9/uY5JkBUNpAABUCOGh/vrysW66p10d2R3S2z8d0L3vbtSRM0ySrCgoDQCACqOqh02v3BuutwdHyMfTpvijqbrtjbVaHHeMSZIVAKUBAFDh3NE6RCue+Is6Nrg4SfLJRduZJFkBUBoAABVSbX8vzX/oz5MkY5gkaRpKAwCgwvptkuTiP0ySZCVJc1AaAAAVXptQf331WDfd175O4UqSd8/YoARWkixXlAYAgFPw9rDpv/eE651fV5LccSxNt72xVgs2JzJJspxQGgAATuW2VrW04oluimoYqAt5BZq4ZKcemhOrkxnZZkdzeZQGAIDTqeXnpbmjIjWpT5jcrRb9sOeker++Rl/vTDI7mkujNAAAnJLVYmh090b6/NEuuqGWr86dz9Mjn2zVEwu4NbOsUBoAAE4tLNhXy8d10bgejWQxpGXxJ9R72hqt+eWU2dFcDqUBAOD03G0WPdU7TIvHdlaD6t5KTs/W0Nmb9a9lu3Q+N9/seC6D0gAAcBlt61bTV4911bCoepKkj2OO6LY31iruyFmTk7kGSgMAwKVUcbfpuTtbau6Dkarl56nDZ87r3nc36r8r9io3nwWhrgelAQDgkro2qa4VT/xFd0XUlv3XBaHunL5ee5PTzY7mtCgNAACX5eflptcGtNGMIW1VrYqb9iSl669vrdd7qw+qwM6CUCVFaQAAuLw+rWrp27/9RTeH1VRugV1Tv9mrQTNjdPTsebOjORVKAwCgUqjp46kPhrXXS3e3kre7VZsPn9Wt09awDHUJUBoAAJWGYRga0KGuvnn8L+pYP0BZuReXoR71UaxOprMM9dVQGgAAlU7dwCqa/3An/fO2i8tQr9x7Uj1fW60lW4/xqcMVUBoAAJWS1WLo4b800hePdlWr2n5Kz87XhIXbNeqjWKXwqcMlURoAAJVas2AfLXmks57q3azwU4der63W4jg+dfgjSgMAoNJzs1o0rkdjfflYV4XXufipw5OLtmtE9BYlpV0wO16FQWkAAOBXTYN89NnYzvrHrRfnOqzad0q3vLZGC7cc5VMHURoAACjCZrVo7I2N9NVjXdUm1F8ZOfl6+rMdGvbhFh1PrdyfOlAaAAC4hCa/fuowqU+Y3G0WrfnllG55bbXmxhyRvZKuJklpAADgMqwWQ6O7N9I3j3dTu3rVlJVboGeW7dKQDzbpyJkss+OVO0oDAABX0ahGVS0cHaUpfZvLy82qjYfO6NZpazV7XUKl2sOC0gAAQDFYLYZGdGmgFU90U6eGAbqQV6Dnv9yt+97bqAMnM82OVy5KXBrWrFmjvn37KiQkRIZhaNmyZVccn5SUpMGDB6tp06ayWCx64okn/jQmOjpahmEUOTw9PUsaDQCAMlcv0FvzRnXSv/u3VFUPm+KOnNNtb67VjFUHlV9gNztemSpxacjKylJ4eLimT59erPE5OTmqUaOGnnnmGYWHh192nK+vr5KSkgqPI0eOlDQaAADlwmIxNCSynr7921/0l6Y1lJtv10sr9qrfO+u163ia2fHKjK2kL+jTp4/69OlT7PH169fXG2+8IUmaPXv2ZccZhqHg4OCSxgEAwDS1/b300YgOWhx3TP/35W7tOp6uO6ev10PdGuqJnk3k6WY1O2KpqjBzGjIzM1WvXj2Fhobqzjvv1M8//3zF8Tk5OUpPTy9yAABQ3gzD0L3tQ/XD37vr9la1VGB36N3VB9V72hptOHja7HilqkKUhmbNmmn27Nlavny55s6dK7vdrs6dO+vYsWOXfc3UqVPl5+dXeISGhpZjYgAAiqrp46npQ9pq5gPtFOTroSNnzmvw+5v0j8U7lHY+z+x4paJClIaoqCgNHTpUbdq0Uffu3bVkyRLVqFFD77333mVfM2nSJKWlpRUeR48eLcfEAABc2i0tgvX9hO66v1NdSdKnsUd182ur9fXOJKdfirpClIY/cnNzU0REhA4cOHDZMR4eHvL19S1yAABQEfh6uumFfq20aEyUGtbw1unMHD3yyVY9/HGcktOcd9vtClkaCgoKtHPnTtWqVcvsKAAAXLMO9QP09WPd9NhNjWWzGPp+d4p6vbZa8zYlOuVS1CUuDZmZmYqPj1d8fLwkKSEhQfHx8UpMTJR08WuDoUOHFnnNb+MzMzN16tQpxcfHa/fu3YXPP//88/ruu+906NAhbd26Vffff7+OHDmiUaNGXcefBgCA+TzdrJpwS7OL227/ugHWP5fu1OAPYnT4tHMtRW04SvgFy6pVq9SjR48/PT5s2DBFR0dr+PDhOnz4sFatWvX7LzGMP42vV6+eDh8+LEn629/+piVLlig5OVnVqlVTu3bt9MILLygiIqLYudLT0+Xn56e0tDS+qgAAVEgFdoc+XJ+gV7/7RRfyCuRhs+hvvZpqVNcGslnN+/C/uO+hJS4NFRWlAQDgLI6ePa9JS3Zq3YGLt2S2rO2rl+5urRYhfqbkKe57aIWc0wAAgCsLDaiijx/sqJfvaS1fT5t2HU/XX99er5dW7FV2XoHZ8S6L0gAAgAn+d1Go21oFq8Du0IxVB3XrtDVat79iLgpFaQAAwEQ1fTz1zpB2eu+Bdqrp46HDZ87r/lmb9PiCbTqVkWN2vCIoDQAAVAC9WwRr5d+7a3jn+jIMaXn8Cd306irNjTlSYW7PZCIkAAAVzI5jqfrn0p3adfzivkoRdf31736t1DykbN7fmAgJAICTal3HX8vHddWUvs1V1cOmbYmp6vv2Ov37q93Kysk3LRelAQCACshqMTSiSwP9MOH3iZLvr01Qr9dW64fdKaZkojQAAFCBBftdnCj54fAOqlPNSyfSsrUp4YwpWWym/FYAAFAiPcJq6vuG3TVr3SGN6NLAlAyUBgAAnISXu1Xjb2pi2u/n6wkAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAslAYAAFAsNrMDlBaHwyFJSk9PNzkJAADO5bf3zt/eSy/HZUpDRkaGJCk0NNTkJAAAOKeMjAz5+fld9nnDcbVa4STsdrtOnDghHx8fGYZRKudMT09XaGiojh49Kl9f31I5J7iuZYXrWna4tmWD61o2ruW6OhwOZWRkKCQkRBbL5WcuuMwnDRaLRXXq1CmTc/v6+vIPugxwXcsG17XscG3LBte1bJT0ul7pE4bfMBESAAAUC6UBAAAUC6XhCjw8PDRlyhR5eHiYHcWlcF3LBte17HBtywbXtWyU5XV1mYmQAACgbPFJAwAAKBZKAwAAKBZKAwAAKBZKAwAAKBZKAwAAKBZKw2VMnz5d9evXl6enpyIjI7V582azIzmdNWvWqG/fvgoJCZFhGFq2bFmR5x0OhyZPnqxatWrJy8tLPXv21P79+80J60SmTp2qDh06yMfHRzVr1lS/fv20b9++ImOys7M1btw4BQYGqmrVqrr77ruVkpJiUmLnMGPGDLVu3bpwFb2oqCh98803hc9zTUvHiy++KMMw9MQTTxQ+xrUtuWeffVaGYRQ5wsLCCp8vq2tKabiETz/9VBMmTNCUKVO0detWhYeHq3fv3jp58qTZ0ZxKVlaWwsPDNX369Es+/9///ldvvvmm3n33XW3atEne3t7q3bu3srOzyzmpc1m9erXGjRunmJgYff/998rLy9Mtt9yirKyswjF/+9vf9MUXX2jRokVavXq1Tpw4obvuusvE1BVfnTp19OKLLyouLk6xsbG66aabdOedd+rnn3+WxDUtDVu2bNF7772n1q1bF3mca3ttWrRooaSkpMJj3bp1hc+V2TV14E86duzoGDduXOHPBQUFjpCQEMfUqVNNTOXcJDmWLl1a+LPdbncEBwc7Xn755cLHUlNTHR4eHo758+ebkNB5nTx50iHJsXr1aofDcfE6urm5ORYtWlQ4Zs+ePQ5Jjo0bN5oV0ylVq1bN8cEHH3BNS0FGRoajSZMmju+//97RvXt3x+OPP+5wOPj3eq2mTJniCA8Pv+RzZXlN+aThD3JzcxUXF6eePXsWPmaxWNSzZ09t3LjRxGSuJSEhQcnJyUWus5+fnyIjI7nOJZSWliZJCggIkCTFxcUpLy+vyLUNCwtT3bp1ubbFVFBQoAULFigrK0tRUVFc01Iwbtw43X777UWuocS/1+uxf/9+hYSEqGHDhhoyZIgSExMlle01dZldLkvL6dOnVVBQoKCgoCKPBwUFae/evSalcj3JycmSdMnr/NtzuDq73a4nnnhCXbp0UcuWLSVdvLbu7u7y9/cvMpZre3U7d+5UVFSUsrOzVbVqVS1dulTNmzdXfHw81/Q6LFiwQFu3btWWLVv+9Bz/Xq9NZGSkoqOj1axZMyUlJem5555Tt27dtGvXrjK9ppQGwImNGzdOu3btKvJdJq5ds2bNFB8fr7S0NC1evFjDhg3T6tWrzY7l1I4eParHH39c33//vTw9Pc2O4zL69OlT+J9bt26tyMhI1atXTwsXLpSXl1eZ/V6+nviD6tWry2q1/mmWaUpKioKDg01K5Xp+u5Zc52s3fvx4ffnll/rpp59Up06dwseDg4OVm5ur1NTUIuO5tlfn7u6uxo0bq127dpo6darCw8P1xhtvcE2vQ1xcnE6ePKm2bdvKZrPJZrNp9erVevPNN2Wz2RQUFMS1LQX+/v5q2rSpDhw4UKb/XikNf+Du7q527dpp5cqVhY/Z7XatXLlSUVFRJiZzLQ0aNFBwcHCR65yenq5NmzZxna/C4XBo/PjxWrp0qX788Uc1aNCgyPPt2rWTm5tbkWu7b98+JSYmcm1LyG63Kycnh2t6HW6++Wbt3LlT8fHxhUf79u01ZMiQwv/Mtb1+mZmZOnjwoGrVqlW2/16vaxqli1qwYIHDw8PDER0d7di9e7fj4Ycfdvj7+zuSk5PNjuZUMjIyHNu2bXNs27bNIcnx2muvObZt2+Y4cuSIw+FwOF588UWHv7+/Y/ny5Y4dO3Y47rzzTkeDBg0cFy5cMDl5xTZ27FiHn5+fY9WqVY6kpKTC4/z584VjxowZ46hbt67jxx9/dMTGxjqioqIcUVFRJqau+CZOnOhYvXq1IyEhwbFjxw7HxIkTHYZhOL777juHw8E1LU3/e/eEw8G1vRZ///vfHatWrXIkJCQ41q9f7+jZs6ejevXqjpMnTzocjrK7ppSGy3jrrbccdevWdbi7uzs6duzoiImJMTuS0/npp58ckv50DBs2zOFwXLzt8l//+pcjKCjI4eHh4bj55psd+/btMze0E7jUNZXk+PDDDwvHXLhwwfHII484qlWr5qhSpYqjf//+jqSkJPNCO4GRI0c66tWr53B3d3fUqFHDcfPNNxcWBoeDa1qa/lgauLYlN2DAAEetWrUc7u7ujtq1azsGDBjgOHDgQOHzZXVNDYfD4bi+zyoAAEBlwJwGAABQLJQGAABQLJQGAABQLJQGAABQLJQGAABQLJQGAABQLJQGAABQLJQGAABQLJQGAABQLJQGAABQLJQGAABQLP8f8QnGJM1dhK0AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig,ax = plt.subplots(1,1,figsize=(6,6))\n", "plt.plot(range(epochs),booster.loss_history)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "299a9855", "metadata": {}, "source": [ "## Your turn! 🚀\n", "\n", "TBD" ] }, { "cell_type": "markdown", "id": "f1432d8d", "metadata": {}, "source": [ "## Acknowledgments\n", "\n", "Thanks to Alexis Moraga for creating the open-source course [Understanding Gradient Boosting](https://www.kaggle.com/code/senoratiramisu/understanding-gradient-boosting). It inspires the majority of the content in this chapter." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" } }, "nbformat": 4, "nbformat_minor": 5 }