{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Decision tree for classification\n", "> A Summary of lecture \"Machine Learning with Tree-Based Models in Python\n", "\", via datacamp\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Chanseok Kang\n", "- categories: [Python, Datacamp, Machine_Learning]\n", "- image: images/decision-boundary.png" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Decision tree for classification\n", "- Classification-tree\n", " - Sequence of if-else questions about individual features.\n", " - **Objective**: infer class labels\n", " - Able to caputre non-linear relationships between features and labels\n", " - Don't require feature scaling(e.g. Standardization)\n", "- Decision Regions\n", " - Decision region: region in the feature space where all instances are assigned to one class label\n", " - Decision Boundary: surface separating different decision regions\n", "![decision region](image/decision_boundary.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Train your first classification tree\n", "In this exercise you'll work with the [Wisconsin Breast Cancer Dataset](https://www.kaggle.com/uciml/breast-cancer-wisconsin-data) from the UCI machine learning repository. You'll predict whether a tumor is malignant or benign based on two features: the mean radius of the tumor (```radius_mean```) and its mean number of concave points (```concave points_mean```)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Preprocess" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
iddiagnosisradius_meantexture_meanperimeter_meanarea_meansmoothness_meancompactness_meanconcavity_meanconcave points_mean...texture_worstperimeter_worstarea_worstsmoothness_worstcompactness_worstconcavity_worstconcave points_worstsymmetry_worstfractal_dimension_worstUnnamed: 32
0842302M17.9910.38122.801001.00.118400.277600.30010.14710...17.33184.602019.00.16220.66560.71190.26540.46010.11890NaN
1842517M20.5717.77132.901326.00.084740.078640.08690.07017...23.41158.801956.00.12380.18660.24160.18600.27500.08902NaN
284300903M19.6921.25130.001203.00.109600.159900.19740.12790...25.53152.501709.00.14440.42450.45040.24300.36130.08758NaN
384348301M11.4220.3877.58386.10.142500.283900.24140.10520...26.5098.87567.70.20980.86630.68690.25750.66380.17300NaN
484358402M20.2914.34135.101297.00.100300.132800.19800.10430...16.67152.201575.00.13740.20500.40000.16250.23640.07678NaN
\n", "

5 rows × 33 columns

\n", "
" ], "text/plain": [ " id diagnosis radius_mean texture_mean perimeter_mean area_mean \\\n", "0 842302 M 17.99 10.38 122.80 1001.0 \n", "1 842517 M 20.57 17.77 132.90 1326.0 \n", "2 84300903 M 19.69 21.25 130.00 1203.0 \n", "3 84348301 M 11.42 20.38 77.58 386.1 \n", "4 84358402 M 20.29 14.34 135.10 1297.0 \n", "\n", " smoothness_mean compactness_mean concavity_mean concave points_mean \\\n", "0 0.11840 0.27760 0.3001 0.14710 \n", "1 0.08474 0.07864 0.0869 0.07017 \n", "2 0.10960 0.15990 0.1974 0.12790 \n", "3 0.14250 0.28390 0.2414 0.10520 \n", "4 0.10030 0.13280 0.1980 0.10430 \n", "\n", " ... texture_worst perimeter_worst area_worst smoothness_worst \\\n", "0 ... 17.33 184.60 2019.0 0.1622 \n", "1 ... 23.41 158.80 1956.0 0.1238 \n", "2 ... 25.53 152.50 1709.0 0.1444 \n", "3 ... 26.50 98.87 567.7 0.2098 \n", "4 ... 16.67 152.20 1575.0 0.1374 \n", "\n", " compactness_worst concavity_worst concave points_worst symmetry_worst \\\n", "0 0.6656 0.7119 0.2654 0.4601 \n", "1 0.1866 0.2416 0.1860 0.2750 \n", "2 0.4245 0.4504 0.2430 0.3613 \n", "3 0.8663 0.6869 0.2575 0.6638 \n", "4 0.2050 0.4000 0.1625 0.2364 \n", "\n", " fractal_dimension_worst Unnamed: 32 \n", "0 0.11890 NaN \n", "1 0.08902 NaN \n", "2 0.08758 NaN \n", "3 0.17300 NaN \n", "4 0.07678 NaN \n", "\n", "[5 rows x 33 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wbc = pd.read_csv('./dataset/wbc.csv')\n", "wbc.head()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "X = wbc[['radius_mean', 'concave points_mean']]\n", "y = wbc['diagnosis']\n", "y = y.map({'M':1, 'B':0})" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1 0 0 1 0]\n" ] } ], "source": [ "from sklearn.tree import DecisionTreeClassifier\n", "\n", "# Instantiate a DecisionTreeClassifier 'dt' with a maximum depth of 6\n", "dt = DecisionTreeClassifier(max_depth=6, random_state=1)\n", "\n", "# Fit dt to the training set\n", "dt.fit(X_train, y_train)\n", "\n", "# Predict test set labels\n", "y_pred = dt.predict(X_test)\n", "print(y_pred[0:5])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluate the classification tree\n", "Now that you've fit your first classification tree, it's time to evaluate its performance on the test set. You'll do so using the accuracy metric which corresponds to the fraction of correct predictions made on the test set." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test set accuracy: 0.89\n" ] } ], "source": [ "from sklearn.metrics import accuracy_score\n", "\n", "# Predict test set labels\n", "y_pred = dt.predict(X_test)\n", "\n", "# Compute test set accuracy\n", "acc = accuracy_score(y_test, y_pred)\n", "print(\"Test set accuracy: {:.2f}\".format(acc))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Logistic regression vs classification tree\n", "A classification tree divides the feature space into rectangular regions. In contrast, a linear model such as logistic regression produces only a single linear decision boundary dividing the feature space into two decision regions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Helper function" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from mlxtend.plotting import plot_decision_regions\n", "\n", "def plot_labeled_decision_regions(X,y, models):\n", " '''Function producing a scatter plot of the instances contained \n", " in the 2D dataset (X,y) along with the decision \n", " regions of two trained classification models contained in the\n", " list 'models'.\n", " \n", " Parameters\n", " ----------\n", " X: pandas DataFrame corresponding to two numerical features \n", " y: pandas Series corresponding the class labels\n", " models: list containing two trained classifiers \n", " \n", " '''\n", " if len(models) != 2:\n", " raise Exception('''Models should be a list containing only two trained classifiers.''')\n", " if not isinstance(X, pd.DataFrame):\n", " raise Exception('''X has to be a pandas DataFrame with two numerical features.''')\n", " if not isinstance(y, pd.Series):\n", " raise Exception('''y has to be a pandas Series corresponding to the labels.''')\n", " fig, ax = plt.subplots(1, 2, figsize=(10.0, 5), sharey=True)\n", " for i, model in enumerate(models):\n", " plot_decision_regions(X.values, y.values, model, legend= 2, ax = ax[i])\n", " ax[i].set_title(model.__class__.__name__)\n", " ax[i].set_xlabel(X.columns[0])\n", " if i == 0:\n", " ax[i].set_ylabel(X.columns[1])\n", " ax[i].set_ylim(X.values[:,1].min(), X.values[:,1].max())\n", " ax[i].set_xlim(X.values[:,0].min(), X.values[:,0].max())\n", " plt.tight_layout()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAskAAAFgCAYAAABJzuRWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde5xVdb3/8ddnLgwDA4MyXkHzAnmtY6h0t5OdTD2oeUPJRE2zLDqVHkuzU52yMstLqaFoqRwlJYvih5paJpSUjpeC0ATFC4hAA8wwA8wMM/P5/bHWhj17bmvP7LWv7+fjsR/MXmuvtb57A+/57u/6XszdERERERGRncpyXQARERERkXyjSrKIiIiISApVkkVEREREUqiSLCIiIiKSQpVkEREREZEUqiSLiIiIiKRQJVkKjpmdY2aPDvLYZWb27xkuUt4xs1vN7H9yXQ4RyV9m9rCZnRfhdS1mdkA2ypQtZvYfZvZajOe/w8y+lvR8hpmtDz/L2vDPfeO6vmSGaZ5kiVsYRBe5+++zfN27gNXu/vVBHOvAVsCBJuB+4HJ378xoIUVEBhBm6B5AB9AJvADMBma5e1cOixaZmS0D3hY+rQa2E7wfgO+5+/diuOZ7gG8B7yX43FYAt7j7bDP7D+AOd98v09ftpRzDgUbgSHdfFvf1JHPUkizSt39z9xrgQ8BZwKcyfQEL6P+hiAzkJHcfRVDRvAb4KvCz3BYpOnc/zN1rwkz9EzAj8by3CrKZVQzlemb2AeD3wB+AA4CxwAzgxKGcd5D2BKoyUUEe6uci6dEvZ8kZM/u0mb1sZhvNbL6Z7Z207zgze8nMmszsp2a20MwuCvedb2Z/Dn82M7shvI3VZGZLzOxwM7sYOAf4Snhb6/+Fr38tbEHAzMrN7Gtm9oqZNZvZs2a2T2o53f1l4EngiKTy1ZrZz8zsLTN708yuNrPypPNeZ2YNZvZqeJvNE+FmZk+Y2XfN7EmC1uoDBjjfhPD9N4XnvL+/9x7uu8vMro74WbuZfdbMVpjZJjO7xcwsE3/HIpJZ7t7k7vMJvrifF+ZdlZn9yMzeMLN1YXer6sQxZnaKmf3NzDaHeXd8uP2JpFztNWfCfW5mE8Kfa81stpn9y8xeN7OvJ77oJ7I5LMumMP9OiPK+zOwiM1tkZj8xs43A15O2/zM838PJGW1mh5rZ78Nc+6eZnZ50yh8BP3P3H7r7Bg/Uu/vZfVz/62a2MvxdsMzMTk7a9/awbInPZk64vSwsb3IGHxruu8fMvmVmhwDLwm0tZvaomVWEn+l+4fbhZna9ma0K//5+akHr845uIRb8rloL3B7l85TMUCVZcsLMjgW+D0wF9gJeB+4L99UBDwBXEnz7fwl4Xx+nOg44Bng7MIbgF8cGd58F3AtcG7ZUnNTLsZcC0whaFkYTtBRv7aWsBwMfBF5O2nw3wa3CCcC7wnJcFO77NHACQaV6EvDxXq59LnAxMCp87/2d7zvAo8AuwHjgpv7eey/l7/OzTjIFOBr4t/B1H+ulzCKSJ9z9aWA1QTb9gCAHjiDIkHHANwDMbDJB14zLCXLiGOC1Xk7ZV86kugmoJWid/RAwHbggaf+7CTK7DrgW+FkaX7rfB7wI7Ab8wMzOCMt9SrjtKSBRQR0FPBa+t90JGkVmmdlB4b7JBL9HoloOvD98b98F5pjZHuG+7wIPsvOzuSXcfgLwHmBiuO9sYGPySd39RYJcJfxddFwv1/4RsD/wzvBc+wFXJe0fD9QA+wKfS+M9yRCpkiy5cg7wc3d/zt3bCCrE7w2/WZ8ILHP3X7t7B/ATYG0f59lOUNE8mKCP/Yvu/lbEMlwEfN3dXwpbGf7u7smVzOfMbAtBaD8B/BQgDM4TgC+5+xZ3Xw/cQBCQEFQyf+zuq919E8Gt0VR3ufuy8P3tOsD5thPcYt3b3Vvd/c9pvvf+PuuEa9y90d3fAP5IUqu5iOStNQT58Wngy+6+0d2bge+xMz8uJPj//5i7d7n7m+7+z17O1VfO7GDB3a2zgCvdvdndXwOuI/jSn/C6u98ejt+4m+CL+R6p5+rDG+4+09073X0b8BmC/sovhVl5NTDZzMYBJwPL3X22u3e4+7PAb4Azws/EgKi/C3D3ue7+VvgZzSH4InFU0mezH7BX+Nk8mbR9NEEG4+4vuHtfv6t6FbbCX0SQ/5vcfTNBo0Zyi3cH8C13bw8/F8kSVZIlV/YmaNEEwN1bCFpBx4X7ViXtc4IWkx7c/XHgZoJv9uvMbJaZjY5Yhn2AV/rZP4ng2/tZBK0jI8PtbwMqgbfMrNHMGoHbCFozEu9tVdJ5kn/ubdtA5/sKQeA/Hd4G/BSk9d77+6wTkoN9a/i+RSS/jQMqgBHAs0n58TuCllcYOOcSes2ZFHXAMJLyJPy51yxx98Sduah5kpqVbwNuSXpfDUAXQcvq24D3J/aF+88iqJRvJBh0vVfE6ya6ivw96VwHE7xfgMsIMvoZM1tq4Ywg7v4ocCswkyCDbw1bsdOxJ1AFJF97ATvzH2Cdu7eneV7JAFWSJVfWsHOkM2Y2kqBrxZsE3/7HJ+2z5Oep3P0n7n4kcBjBLcfLE7sGKMMq4MD+XhC2MM8F/kJ4+zI8rg2oc/cx4WO0ux8W7u9WfoJfUj1OnVKOPs/n7mvd/dPuvjdBy8pPE/0D+3nvyfr7rEWkAJnZ0QSV098A24DDkvKjNhwgBxFyDvrPmSQN7GxxTtiXzGVJamavAi5Mel9j3L3a3Z8K9/0hZV+Nu88IW9OfBk5PvUBvLJjebiZwCTDW3ccA/yT40kDYwnyRu+8FfJ6gW8f+4b4b3X0ScDhwKEE3vnSsA9qBg1L+/mr7+VwkS1RJlmypDAcnDA8HJMwFLjCzI8ysiuD24FPh7bsHgXeY2cctGOz2eYJv2z2Y2dFm9m4zqwS2AK0EU/1AED79ze15B/AdM5togXea2dg+XnsNcLGZ7Rl2aXgUuM7MRoeDNw40sw+Fr50LfNHMxpnZGIJR6H0a6HxmdqaZJSrdmwgCs3OA955sDn1/1iJSQMKMmEIwruAed/87wWCuG8xs9/A148wsMa7gZwT//z8SZsu4cJxF6nl7zZnk14RdKOYC3zWzUWb2NoJK4T0xvFUIWmmvCge/YWZjwn7KAPOBw8zsE2ZWGT4mm9lB4f7LgYvM7FIz2zU8/l0WDrpLUUPwfv8VvMwuIuxCER43NeziAcFUbokMnhw+KggyuJ3eM7hP4Wd6B3Cjme0W/i4ab2a99V2WLFMlWbLlIYLWjsTjg8D/AL8iaHk9kLAPlrs3AGcSDPrYQPDt/BmC1tZUowl+QWwiuO23gWAQBAS/HA4Nb2H9ppdjrycI/EeBzeHrq3t5He6+FFjIzpba6QS3HV8Ir/0AO2/t3R6ecwnwfPjeE/Ob9qW/8x0NPGVmLQS/GL7o7q8O8N6Ty/4H+visRaRg/D8zayZoQb2KIL8SA+a+SjCw+K9mtplg6rODYMcAvwsIxjk0EeTY2+ipr5xJ9QWCCuFK4M8EX8J/nok3mMrdf0nwPn8Zvq8lhIOK3b0p/PmTBLm2lqAvb1W4/0/Af4Svec2CGTNmEuRx6nWWEIx9eTo818EEgwQT3g3UWzBG5dfA58PxG2MIfm80EvRhfovgc07XZQQZ/jTB39GjBAP4JMe0mIjkvXBgw2rgHHf/Y67Lky4LpkC61d17+8UkIiIieUgtyZKXzOxj4a21KuBrBH3D/prjYkViZtVmdqIFc2GOA74JzMt1uURERCQ6VZIlX72XYER2A3AS8PECmvrGgP8l6AbxPMEUct/o9wgRERHJK+puISIiIiKSQi3JIiIiIiIpKnJdgGy5fdFKNZkXgL/+8mZ+fMre1IyoynVRRLLu+nsfhTef5dIjtjzLt5qOGviI/FNMWbts8aOcu/srHHnwvrkuioj0YUduHlPL9YuaYNyRXHpOxBn09jgMDjy2z2XT1ZIseaVyeA3NW1tzXQyRrGtobGHBwnqmTxo58ItFRKRHbk6fNJIFC+vZ0LQlI+dXJVnySkX1SFq29TYdskhxm/3gYqZMKKOupmRu8ImIDElqbtbVVDBlQhl3L3gyI+dXGkteqRg+iuatG3JdDJGse+K55axZ38acpetZva65bu23cl0iEZH8lpybyfZetzx6l4t+qJIseaW8epS6W0hJmn/djJ1P3veF/XJWEBGRAtEtN2NQ0pVkw6mt7GJ4OZj12W87Z9yd1k5o2l6Gk3/li0PViBqamjtyXQwRySBlrYgUopKuJNdWdjFm5HC6rALyMLhxZ7h3wJZWGreX57o0WVFVPYLm9aokixQTZa2IFKKSHrg3vJz8DW0AM7qsguEllNnDR4ykaVt7roshIhmkrBWRQlTSlWQzy9/QTjDLy9uTcamqHknTVlWSRYqJslZEClFJV5Il/1RUDmN7Z1euiyEiIiIlTpXkHHvmz49z4Ukf4IIT38v9d9yU6+KIiBQt5a2IpEOV5Bzq7Ozklu9+jat/ei+zfruQJx7+Da+/8lKuiyUiUnSUtyKSrpKe3SIdX5x+Kk2bN/fYXjt6ND+ePW9Q53xp6fPste9+7LXP2wD40Amn8Jc/PsLbDjxoSGUVESlUcWQtKG9FJH2qJEfUtHkzEy++ucf2FbMGP5H1hvVr2W3PcTue1+2xFy8teX7Q5xMRKXRxZC0ob0UkfepukUPu3mObRleLiGSe8lZE0qVKcg7V7bEX/1r75o7nDeveYtfd98hhiUREipPyVkTSpUpyDh10+BGsef1V1q5+g+3b21n48G95z79/LNfFEhEpOspbEUlX7JVkMzvezF4ys5fN7Ipe9h9jZs+ZWYeZnZG0/cNm9rekR6uZfTzcd5eZvZq074i430ccyisq+NzXvsdVn53GxScfwzEfO4n9JmgQiYikT1nbP+WtiKQr1oF7ZlYO3AJ8FFgN1JvZfHd/IellbwDnA/+dfKy7/xE4IjzPrsDLwKNJL7nc3R+Ir/Td1Y4e3evAkdrRo4d03snHfITJx3xkSOcQkdKmrI1GeSsi6Yh7dovJwMvuvhLAzO4DTgF2BLe7vxbu62+ZtTOAh919a3xF7d9Qph4SEYmZslZEJMPi7m4xDliV9Hx1uC1dZwO/SNn2XTNbYmY3mFlVbweZ2cVm9oyZPbNofurhIiJFQ1krIpJhcVeSe5tfp+c8PP2dwGwv4B3AI0mbrwQOBo4GdgW+2tux7j7L3Y9y96OOOXlaOpcVESkkyloRkQyLu5K8Gtgn6fl4YE2a55gKzHP37YkN7v6WB9qAOwluNYqIlCplrYhIhsVdSa4HJprZ/mY2jOBW3vw0zzGNlNt/YYsHFswE/3HgHxkoq4hIoVLWiohkWKyVZHfvAGYQ3L57EZjr7svM7NtmdjKAmR1tZquBM4HbzGxZ4ngz24+gdWRhyqnvNbOlwFKgDrg6zvchIpLPlLUiIpkX9+wWuPtDwEMp276R9HM9wa3B3o59jV4Gn7j7sZktZe5c/z9f5qlFjzFm1zpum/dErosjIgVKWds/Za2IpEsr7uXYR0+ZytUz5+S6GCIiRU1ZKyLpUiU5TU2bNvDd//okmxs3ZuR87zjqvYyq3SUj5xIRKRbKWhHJNVWS0/T4b+6la83f+cO8e3JdFBGRoqWsFZFcUyU5DU2bNvD8Yw9w42njef6xBzLWwiEiIjspa0UkH6iSnIbHf3MvJ02AiXtUc9IE1MIhIhIDZa2I5ANVkiNKtGx84shaAD5xZK1aOEREMkxZKyL5QpXkiBItG2NrKoHgz0y0cHz/K5fw5U9OYfVrr/DJj0zid7/W6GsRKV3KWhHJF7HPk1wslj79J/70Viu/WLK62/Yx//oTp17wX4M+75XXzhxq0UREioayVkTyhSrJEX1j5i9zXQQRkaKnrBWRfKHuFiIiIiIiKUq6kuzu4J7rYvTPPSiniEiBUtaKSCEq6UpyayeUeUf+hrc7Zd5Ba2euC5I9ZWVldHTm6d+HiAyKslZEClFJ90lu2l4GW1oZXg5mluvi9ODutHaG5SwRFZXD2Nau31QixURZKyKFqKQryY7RuL0ctue6JJJgZnSZflGJFBNlrYgUItVGJO90uf5ZioiISG6pNiJ5Ry3JIiIikmuqjUje6dQ/SxEREckx1UYk76i7hYiIiOSaaiOSd9TdQkRERHJNtRHJO136ZykiIiI5ptqI5J1Oz795VEVERKS0qJIsecdRJVlERERyS5VkEREREZEUqiSLiIiIiKRQJVlEREREJIUqySIiIiIiKVRJFhERERFJoUqyiIiIiEgKVZJFRERERFKokiwiIiIikiL2SrKZHW9mL5nZy2Z2RS/7jzGz58ysw8zOSNnXaWZ/Cx/zk7bvb2ZPmdkKM7vfzIbF/T5ERPKZslZEJLNirSSbWTlwC3ACcCgwzcwOTXnZG8D5wJxeTrHN3Y8IHycnbf8BcIO7TwQ2ARdmvPAiIgVCWSsiknlxtyRPBl5295Xu3g7cB5yS/AJ3f83dlwBdUU5oZgYcCzwQbrob+HjmiiwiUnCUtSIiGRZ3JXkcsCrp+epwW1TDzewZM/urmSXCeSzQ6O4dA53TzC4Oj39m0fxfpFt2EZFCoawVEcmwuCvJ1ss2T+P4fd39KOATwI1mdmA653T3We5+lLsfdczJ09K4rMjQNTS2cPoVt7KhaUuuiyLFT1krJUtZK3GJu5K8Gtgn6fl4YE3Ug919TfjnSuAJ4F1AAzDGzCoGc06RbJn94GI2rV3F3QuezHVRpPgpa6VkKWslLnFXkuuBieEI6WHA2cD8AY4BwMx2MbOq8Oc64P3AC+7uwB+BxOjs84DfZrzkIkPQ0NjCgoX1zDytjgUL69XCIXFT1kpJUtZKnGKtJId92WYAjwAvAnPdfZmZfdvMTgYws6PNbDVwJnCbmS0LDz8EeMbM/k4Q1Ne4+wvhvq8Cl5rZywT95n4W5/sQSdfsBxczZUIZB+1exZQJZWrhkFgpa6VUKWslThUDv2Ro3P0h4KGUbd9I+rme4DZe6nGLgXf0cc6VBKO5RfJOomVj7tRRAEyfNJKpc+s5b8r7GVs7Mselk2KlrJVSo6yVuGnFPZEMS7Rs1NUE30HrairUwiEikmHKWolb7C3JIqXmieeWs2Z9G3OWru+2fe91y7n0nONyVCoRkeKirJW4qZIskmHzr5uR6yKIiBQ9Za3ETd0tRERERERSqJIsIiIiIpJClWQRERERkRSqJIuIiIiIpFAlWUREREQkhSrJIpK2hsYWTr/iVi0BKyKSIcrV/KNKsoikbfaDi9m0dpUm7RcRyRDlav5RJVmKhr6FZ0diKdiZp9WxYGG9Pm+REqbczQzlan5SJVmKhr6FZ0diKdiDdq/SErAiJU65mxnK1fykSrIUBX0Lz47E5zx90kgApk8aqc9bpEQpdzNDuZq/VEmWoqBv4dmR+JzraoIV7etqKvR5i5Qo5W5mKFfzV0WuCyAyVIlv4XOnjgKCb+FT59Zz3pT3M7Z2ZI5LV1yeeG45a9a3MWfp+m7b9163nEvPOS5HpRKRbFPuZo5yNX+pkiwFr79v4QqYzJp/3YxcF0FE8oByN3OUq/lLlWQpePoWLiKSXcpdKQWqJEvB07dwEZHsUu5KKdDAPRERERGRFKoki4iIiIikUCVZ8k4XlusiiIiISIlTJVnyjlVU0da+PdfFEBERkRKmSrLknYrqGlq2tee6GCIiIlLCVEmWvDOsuobmra25LoaIiIiUMFWSJe9UVNfQvLUt18UQERGREqZKsuSdsuGjaNmmSrKIiIjkjirJkncqq0exeav6JIuIiEjuqJIseWf4iJE0bdXsFiIiIpI7qiRL3qmqHsHmbaokZ0pDYwunX3ErG5q25LooIiJFQ9la/CJVks1sNzP7mpnNMrOfJx4Rjz3ezF4ys5fN7Ipe9h9jZs+ZWYeZnZG0/Qgz+4uZLTOzJWZ2VtK+u8zsVTP7W/g4IkpZpDCUV1TS4Z7rYhSN2Q8uZtPaVdy94MlcF0VipKwVyS5la/GL2pL8W6AW+D3wYNKjX2ZWDtwCnAAcCkwzs0NTXvYGcD4wJ2X7VmC6ux8GHA/caGZjkvZf7u5HhI+/RXwfIiWlobGFBQvrmXlaHQsW1qvFo0gpa0WyS9laGqJWkke4+1fdfa67/yrxiHDcZOBld1/p7u3AfcApyS9w99fcfQnQlbJ9ubuvCH9eA6wHdotYXilyus0VzewHFzNlQhkH7V7FlAllavEoXspaiZ1ydydla2mIWkleYGYnDuL844BVSc9Xh9vSYmaTgWHAK0mbvxveGrzBzKoGUTYpYLrNNbBES8f0SSMBmD5ppFo8ipeyVmKn3A0oW0tH1EryFwkqytvMbLOZNZvZ5gjHWS/b0upsamZ7Af8HXODuiRaQK4GDgaOBXYGv9nHsxWb2jJk9s2j+L9K5rOQx3eaKJtHSUVdTAUBdTYVaPIqXslZipdzdSdlaOiqivMjdRw3y/KuBfZKejwfWRD3YzEYT9H3+urv/Nak8b4U/tpnZncB/93a8u88CZgHcvmilRoIVie63uVq5e8GTXHrOcbkuVt554rnlrFnfxpyl67tt33vdcn1exUdZK7FS7u6kbC0dkSrJAGa2CzARGJ7Y5u6LBjisHphoZvsDbwJnA5+IeL1hwDxgtrv/MmXfXu7+lpkZ8HHgH1HfhxS2RGvG3KnB97bpk0YydW495015P2NrR+a4dPll/nUzcl0EyR5lrcRGududsrV0RJ0C7iJgEfAI8L/hn98a6Dh37wBmhK9/EZjr7svM7NtmdnJ47qPNbDVwJnCbmS0LD58KHAOc38v0Q/ea2VJgKVAHXB3p3UrB020ukZ6UtRIn5a6UqqgtyV8k6JP2V3f/sJkdTFBZHpC7PwQ8lLLtG0k/1xPcGkw97h7gnj7OeWzEckuR0W0ukd4payUuyl0pVVErya3u3mpmmFmVu//TzA6KtWQivSiU21wNjS185pp7mHXluSV5O1JEikc+566yVuIUdXaL1eHk8r8BHjOz35LGoBCRUqOpkkRE4qeslThFqiS7+6nu3uju3wL+B/gZwSAOEUmhqZJEROKnrJW4RW1Jxsw+YGYXuPtC4C8MYqJ6kVKglZhEROKnrJW4RZ3d4psEk8hfGW6qpI+BHiKlTCsxiYjET1kr2RC1JflU4GRgC4C7rwEGu8CISNHSVEkiIvFT1ko2RJ3dot3d3cwcwMw0hFSkF5oqSUQkfspayYaoleS5ZnYbMMbMPg18Crg9vmKJFKZ8nipJRKRYKGslGyJVkt39R2b2UWAzcBDwDXd/LNaSiYiIiIjkSNSWZNz9MTN7KnGMme3q7htjK5mIiIiISI5EqiSb2WeAbwPbgC7AAAcOiK9oIiIiIiK5EbUl+b+Bw9y9Ic7CiIiIiIjkg6hTwL0CbI2zICIiIiIi+SJqS/KVwOKwT3JbYqO7/1cspRIRERERyaGoleTbgMeBpQR9kkVEREREilbU7hYd7n6pu9/p7ncnHrGWTCSChsYWTr/iVi1FKiKSJcpdKRVRK8l/NLOLzWwvM9s18Yi1ZCIRzH5wMZvWrtJSpCIiWaLclVIRtZL8CcJ+ycCz4eOZuAolEkVDYwsLFtYz87Q6FiysV6uGiEjMlLtSSiJVkt19/14eO+ZIDlfjE8mq2Q8uZsqEMg7avYopE8rUqiEiEjPlrpSSqC3JA/lBhs4jQnlFBe3b+x8fmmjNmD5pJADTJ41Uq4aISIyUu1JqMlVJtgydR4Th1SPYvK2939ckWjPqaoIJWupqKtSqISISI+WulJqoU8ANxDN0HhGGDR9By7bt/b7mieeWs2Z9G3OWru+2fe91y7n0nOPiLJ6ISElS7kqpyVQlWXKsuXEj9/3wcqZ95UfU1O6S6+IMSVlZGR3e/82J+dfNyFJpRETS09DYwmeuuYdZV57L2NqRuS5Oxih3pdRkqrvFaxk6jwxS/cP3U7FuKU8/dF+ui5IRXRn7pykikl2aIk2kOESqiZjZmWY2Kvz562b2azOblNjv7qfFVUAZWHPjRl5aNI/rTh3HS4vm0dK0KddFGrIuUyVZRAqPpkgTKR5RayL/4+7NZvYB4GPA3cDM+Iol6ah/+H5OmggTdq/mpIkURWtyl6uSLCKFR1OkiRSPqDWRzvDP/wRmuvtvgWHxFEnSkWhFnjapFoBpk2qLojW5y8ozfk4tpSoicdIUad0pc6XQRa0kv2lmtwFTgYfMrCqNYyVGiVbksSMrgeDPYmhN7oxhVkH1ExSROGmKtO6UuVLoos5uMRU4HviRuzea2V7A5fEVS6Ja8fyTPL++lfuXrO62vWbtkxw77ZIclWroOjPc3SK5n+AlC+o5b8r7i2rUuYjknqZI20mZK8UgaiX5Nnc/N/HE3d8ys2uBR+MplkT1mWvvyXURYmEZbkju3k+wlbsXPFlyv7REJF6aIm0nZa4Ug6jNdYclPzGzcuDIKAea2fFm9pKZvWxmV/Sy/xgze87MOszsjJR955nZivBxXtL2I81saXjOn5hlukolxUT9BKUUKGslXyhzpVj0W0k2syvNrBl4p5ltDh/NwHrgtwOdPKxM3wKcABwKTDOzQ1Ne9gZwPjAn5dhdgW8C7wYmA980s8QqGTOBi4GJ4eP4gcoipWuo/QQ1+ETynbJW8slQMld5K/mk3+4W7v594Ptm9n13v3IQ558MvOzuKwHM7D7gFOCFpGu8Fu7rSjn2Y8Bj7r4x3P8YcLyZPQGMdve/hNtnAx8HHh5E+aQEDLWfYPLgE90ulDylrJW8MZTMVd5KPonUJ9ndrzSzccDbko9x90UDHDoOWJX0fDVBa0UUvR07Lnys7mV7D2Z2MUErCJ+87GqOOXlaxEtLMRlKP0ENPpECoayVvDHYzFXeSr6JuuLeNcCTwNcJZrW4HPjvKIf2ss0jlq2vYyOf091nuftR7n6UQlsGQwsDSIFQ1krBU95Kvok6cO9U4CB3P9HdTwofJ0c4bjWwT9Lz8cCaiNfs69jV4c+DOadIZPk0+ET99GQAylopaLnMW+Wr9CVqJXklUDmI89cDE81sfzMbBpwNzI947CPAcWa2SziI5DjgEXd/C2g2s/eEI62nE2EQoUi68mlhABCY9pYAACAASURBVE3KLwNQ1kpBy2XeKl+lL1HnSd4K/M3M/gC0JTa6+3/1d5C7d5jZDIIQLgd+7u7LzOzbwDPuPt/MjgbmAbsAJ5nZ/7r7Ye6+0cy+QxD+AN9ODCwBLgHuAqoJBpFoIIlkXL4sDKB+ejIQZa0UulzlrfJV+mPuA3dbS543M5m7353xEsXk9kUro/bPkzzwzP99h9sumJTrYuSF6+99FN58lkuPqeX6RU0w7kiN+i527/tCwc5HXExZu2zxo5y7+yscefC+uS6KxET5WuL2OAwOPLbPvI3U3cLd7+7tkblSikhv8qlftIhIMVG+ykAGWkxkbvjnUjNbkvrIThFFSlc+9YsWESkmylcZyEB9kr8Y/jkl7oKISE/50i9aRKTYKF9lIAOtuPdW+OfrZrYHcHS462l3X9/3kSKSCUNZCEVERPqmfJWBRF1MZCrwNHAmMBV4yszOiLNgIiIiIiK5EnUKuKuAoxOtx2a2G/B74IG4CiYiIiIikitRFxMpS+lesSGNY0XStq21NZYVkIptZaViez8ikluZzpRCzqhCLrtkRtSK7u/M7BEzO9/MzgceBB6Kr1hS6lYsXx7LCkjFtrJSsb0fEcmtTGdKIWdUIZddMiPqPMmXA7cB7wT+DZjl7l+Ns2BSupobN7L61VeYeVpdRuesTF5ZqRjmwiy29yMiuZXpTCnkjCrkskvmpNNlYjGwEHgc+Es8xRGB+ofv5+27wkG7V2V0zsrEnJiZPm+uFNv7EZHcynSmFHJGFXLZJXOizm5xEcHsFqcCZwB/NbNPxVkwKU3NjRt5adE83rFXFZC5FZBSV1b6z4OGc/2cx1ixKnczGQ6lv5tWihKRTMp0puRD5g42Y5WvkhC1Jfly4F3ufr67nwccCai7hWRc/cP3c9JEGDEs+KeZqRWQUldWevDFFvap6eQrN/1yyGUeSpkG299NK0WJSCZlOlPyIXMHm7HKV0mIOgXcaqA56XkzsCrzxZFS0dy4kft+eDnTvvIjamp32bF9xfNP8vz6VjZtaubRl9t3bB/qCkjJKyt1dHaxubmFm06o4tJHX2ND0xbG1o4c0vtJV3J/t0sW1HPelPenVQatFCVSONydF15dS5d77NfadfQIxu02ps/9DY0tfOaae5h15bndMic5U1rbO3GCsu726lKOf987OGS/PTGzyOXIdeYOJWOVr5JgHuE/rZnNBt4B/BZw4BSC7hfLAdz9+hjLmBG3L1oZfzpJZI//Yiav/+FO3vaRCzh22iU99j/zf9/htgsmxXLt6+99FN58lkuPqeX6RU0w7sisB18+lEHy2Pu+EL02kmeKKWuXLX6Uc3d/hSMP3ndI53ntrQ184b6XOODf3puhkvVtw99/zz1f/mif+6+/91EWPLaQKR/9UK+Z07K1jdOuf4JD3rdz34pnFvHzC9/FnmNHD6pMucg7ZaxEssdhcOCxfeZt1JbkV8JHwm/DP0cNtlySX/pq2Y3rWi8tmsctp47j8wvmMfnEs2O/ZkKidWHu1OCf7vRJI5k6N/2W3EIvg4hkT8u2Ng5453s4/Jgp/b4uEzn83OvP9bkvSutqy7Y2DjjsXd3K2tm2jS3bhtY3OZt5p4yVTIk6Bdz/9veIu5ASv/qH76di3VKefui+rFzrpIkwYfdqTppIVq6ZkA99zfKhDCKSPS1b2yirGrhylokc7qTvmxBRZmzYsq2N8pSylg8fQcu29h6vjSIXeaeMlUyJ2pIsRSybLbuJa33zrFoApk2q5RP3Z681OR/6muVDGUQke1pa26morun3NZnK4S7vvZIctXV1S2s7FcPruh1bVjWSLa1taZcFcpN3yljJFFWSJaVldwtPP3Rfr/2EM3mtsSMrgeDPRGtyXNdMNv+6GbFfoxDK0NfgHRHJvObWTqp2qe73NZnK4c4+bhD317qaXHHc0tpO2fDuFfph1TVs3jq4luRc5F0uM1bZWlzSWUxEilCi9WLapJ0tuy8tmkdL06ZYrrfi+Se5f0krH7xl9Y7H/UtaWfG8boNlk5ZbFcmejk6nrLzvNqlM5rD30d3iieeWM2dpG0fdsn7HY87SNp54bnm313V2dmFl5d22lZVX0NGVdlFKkrK1uERqSTaztwMzgT3c/XAzeydwsrtfHWvpJHbZbtn9zLX3ZPyckp6hTj8nIpmVjRzOhztYxU7ZWnyidre4nWBBkdsA3H2Jmc0BVEkucIl5ie9fsrrb9pq1T2al+4NkX/fBO609breKSHYph4uDsrX4RK0kj3D3p1MmEu+IoTySZYmW3WxOASe5o6mRRPKPcrjwKVuLU9Q+yQ1mdiDBQiKY2RnAW7GVSrIum1PASe5oaiSR/KUcLlzK1uIUtZL8eYKuFgeb2ZvAl4DPxlYqyarEoJHrTh0X66C9YtXQ2MLpV9zKhqbBTbafTVEH74hIdimHB5bPWatsLU5Ru1u87u7/YWYjgTJ3b46zUJJd2ZwCrhglj2bO9/5nGrwjkp+UwwPL56xVthanqC3Jr5rZLOA9QEuM5ZEsy/YUcMUmeTTzgoX1ednCISL5TTk8MGWt5ELUSvJBwO8Jul28amY3m9kH4iuWZEt/Uw8Vmzhu1Q20zGs+3x4UkfxQrDmcyfzrL2uVsxKXSJVkd9/m7nPd/TTgXcBoYGGsJZOsyNfFPbo88+fM9CTviZaN6ZOCkcvTJ43s0cKhieVFZCD5msNDlan8GyhrlbMSl8jLUpvZh4CzgBOAemBqXIWS7MnXxT3cMrsYZByTvA+0zKsmlheRKPI1h4cik/nXX9ZO/8/3KWclNlFX3HsV+BswF7jc3XVPQ2LV0ZXZSnIck7w/8dxy1qxvY87S9d22771uOZeec5wmlheRkpXJ/OsvawHlrMQmakvyv7n75sFcwMyOB34MlAN3uPs1KfurgNnAkcAG4Cx3f83MziFY5S/hncAkd/+bmT0B7AVsC/cd5+7d//dIQesqK6ezs4vy8qFXluOa5L2/0cyaWF6yTVkr+SLT+ddX1jY0tjD1Kz9WzkpsotZA2s3s82b2UzP7eeIx0EFmVg7cQtBF41BgmpkdmvKyC4FN7j4BuAH4AYC73+vuR7j7EcC5wGvu/rek485J7FdoF59hw0ewpbUtI+fKxSTvmlhesklZK/kkW/mnnJW4RW1J/j/gn8DHgG8D5wAvRjhuMvCyu68EMLP7gFOAF5JecwrwrfDnB4CbzczcPXno1jTgFxHLKjHLxtKpFdU1tGxtZ/TI6iGfa6BuEXHIxTWlpClrS0i+L1+drfxTzkrcolaSJ7j7mWZ2irvfbWZzgEciHDcOWJX0fDXw7r5e4+4dZtYEjAUakl5zFkHAJ7vTzDqBXwFXpwQ9AGZ2MXAxwCcvu5pjTp4WocgykOSlU+Oa7L6yehTNW5uA2iGfKxeTvGtieckyZW0JyUYGD0W28k85K3GL2t1ie/hno5kdTlBz2S/CcdbLttSA7fc1ZvZuYKu7/yNp/znu/g7gg+Hj3N4u7u6z3P0odz9KoZ0Z2Vo6tXx4DS3bMtPdQqQEKGtLhJavFsmeqJXkWWa2C/B1YD7BLbxrIxy3Gtgn6fl4YE1frzGzCoIK+Mak/WeTcvvP3d8M/2wG5hDcapQs6L50anyT3VdUj6J5qyrJIhEpa0tEtjJYRKIvJnKHu29y90XufoC77+7ut0Y4tB6YaGb7m9kwghCen/Ka+cB54c9nAI8nbueZWRlwJrAjBcyswszqwp8rgSnAP5DYZXPp1KrqGppaOzJ+XpEipawtAVq+WiS7IlWSzex7ZjYm6fkuZnb1QMe5ewcwg6D/8ovAXHdfZmbfNrOTw5f9DBhrZi8DlwJXJJ3iGGB1YjBKqAp4xMyWEMzd/CZwe5T3IUOTzaVTrbyMrjiW3RMpQsra0lCsy1eL5KuoA/dOcPevJZ64+yYzO5Gg+0W/3P0h4KGUbd9I+rmVoAWjt2OfAN6Tsm0LwTyfkmUrnn+S59e3cv+S1d2216x9Mi8Hj4iUEmVt8VMGi2RX1EpyuZlVuXsbgJlVE7QySAkpxqVTRUQKhTJYJLuiVpLvAf5gZncSjIb+FHB3bKUSEREREcmhqAP3rgW+CxwCHAZ8J9wm0qfmxo3cftWFGlQiIpJlDY0tnH7FrWxo2pLroogUrKhTwOHuD7v7f7v7Ze4eZSERKXHJE96LiEj2zH5wMZvWrtISzSJDEHV2i9PMbIWZNZnZZjNrNrPNcRdOCpcmvBcRyY2GxhYWLKxn5ml1LFhYr9ZkkUGK2pJ8LXCyu9e6+2h3H+Xuo+MsmBQ2TXgvIpIbsx9czJQJZRy0exVTJpSpNVlkkKIO3Fvn7i/GWhIpGolW5G+etXPC+0/cP4/JJ55NTe0u3V77/RnTaGlp7nGOyvIyjvz0B7NSXhGRYtHa2sqCv9Yzd+ooAKZPGsnUufWcN+X9jK0dueN1ky+5hYbmnquaVpV1csLn3pW18orks6iV5GfM7H7gN8CO/1Xu/utYSiUFrbcJ76cc0MXNXzqTGTf+Enfnvh9ezrSv/IiWlmYOuOimHud46ZaLsl1sEZGCt3L5Pzl7Qhl1NcGv97qaCk44AD76hRt47KYv4+585pp7WNe4lXd+9sYexz970+d7bCsvL2d7Z1fsZRfJN1EryaOBrcBxSdscUCVZeuhtwvvWLS1sb9u6o9uFBvSJiGTeurfeZM6GNuYsXb9j2+YtrWxrbdvR7WLT2lW0b4366x+GDR9BS2tHxssqku8i/S9x9wviLogUj9QJ75sbN3LPlWdzy5SRXDL/Abq64LZTx/H5BfPo6tzZLX7tqpV0dnaGx2zis9f9mstu/R11o6p4embP1g0REenuwx/7T+761Dt3PG9obGHqV37MzCl7cPFvn6bLnTtOq+Njs96kfWszw0aM4sU31tPR6QD8q3ELP//hNxhWPZKamlFcefMvqKquplmVZClBUWe3GG9m88xsvZmtM7Nfmdn4uAsnxSF5EN9x47YwftjmHQP6urY27nhdZ2cnVXX7UlW3LxUjRrPvxz7FYZ++rtd+cyIiMrDkQXwfGtdOXWUrB+1exSkHl7Pu2UcB6Oh0qncbT/Vu46kYMYq9T/g8B1x0047xIlXVI2lu3Z7LtyGSE1Hvt9wJzAHODJ9/Mtz20TgKJcUjeRBfZ2cnp+y/ncWvtLJxy3amTarlzkWr2b61icoRtbkuqohIUUlMBTd36ii2d3Rx4v6dLHyljQ1bOjn9kGE8/MdFtB953IDnqRpezYL65WxrG1pFecZ/HkHdmJohnUMkm6JWkndz9zuTnt9lZl+Ko0BSXJIH8TVv/Bf77VLGyQdVMrd+PZ/993Gcekgl/3fbZ6kYVcemhvVU1uwKQNmw4TkuuYhIYUu0ItfVVLB2w2b226WcUw6q5O76Rk7Yv4IT9tzA3Fu/zMY2o7ImmHmoclhVj/NUVA7jtG/eHYxEGqQVS+pZ+ko9Hz7y7YM/iUiWRa0kN5jZJ4FfhM+nARviKZIUk+RBfM2bGvCuoM9xh7dz7zIHajngwH35zLX3cNX5U3bMdLHlzRXQ/HoOSy4iUtieeG45a9YHg/jWb2qmM5yhoot25iwbBVRz9IQ6/rGuncM+fS0AG197kU29NBj3VnlOx4hRo2lqUr9mKSxRK8mfAm4GbiD4Lrk43CZFpK85ixODNwYjdRCf5IeGxhY+c809zLry3G5zp4pIbmUyh+dfNyPS6w745PVpnXcwqoaPoGVtcVeSlavFJ+rsFm8AJ8dclpIXRyU1HX3NWbzyji/Efm0I3mfiWtvbtjGsYysb/nwfdaOG1oIhPc1+cDGb1q7i7gVPcuk5A/dJFClF9912Pb9sb6GqqnsGxTnjTi5yuG5UFctuvwwIFiPZ6sNoCGe3yJSqESOKfoYM5WrxiVRJNrO7gS+6e2P4fBfgOndXa3IG5bqSmmvJXwReX/4Pjtz4MKd+4NAclqg4JQbzzDytjksW9FyJS0QC27Zt452nf4kx4yd0256oUEbV5Y6ZZbJoGZVc4f/z31/h19vexSFHZXbF02FV1bQU8QwZytXiFGkKOOCdiQoygLtvArRupUiaGhpbOP2KW9nQtGVQ+zMheUqoKRPKdiwwICLdmZXRuX3oU1BuaetkWFVpD0a2sjKytWZffzkaV8YqV4tT1EpyWdh6DICZ7Ur0/swiJS05lJNvx/VmoP2ZKMuChfVMnxS0cEyfNJIFC+tjrZSLFCqzMjraW4d8npbW7VRVq1UxTlFzNo6MVa4Wr6iV5OuAxWb2HTP7NsHAvWvjK5b0prlxI7dfdSEtTZtyXRRJQyKUf/rAH3fcjustQJNv18UVsMlTQgHU1VSo1UOkD1ZWRmd7e6/70mmR3Lytnarq6kwXT5JEydm4Mla5WryiDtybbWbPAMcCBpzm7i/EWjLpof7h+6lYt5SnH7qPY6ddkvHzJw+cS91eDLI98rihsYULvnM3jZs2csdpdZw15y+cfviI8HZca4/BHd1v1/XcnwnJU0Il23vdcg00EUlhZrRta+7RmtzV1cXtv1lEw5o3mDVvIf911kf6PU9jSyu7VUWrJBdTDjc3buS+H17OtK/8CCuL2iaXnnRyNq6MVa4Wr8hdJsJKsSrGMeovHBMr191y6jg+v2Aek088m5raXXo5y+BlYwaNXMr2yOPZDy6mYc3r7Dm6kgN23YWP7LMdOoOBK9MnjWTq3J2DO5JXxuptf6ZEnRJKRGB07S68vmgebyz8dbftlZUV/HT+U5x2yEh+Ov8p/tG1L8OH910JLt/lUMrKyyNds5hyOLlh591TPhHLNaLmrLvHlrHK1eKlfsV5pL9wfPwXMzlpIkzYvZqTJm6JrTW5WGV75HFDYwu/ffxpvnFMBdf8uY0Vb23mU+8axhce3sLnPtDZ7Xbcpecc1+/tOrVEiOTG1356f6/bH//FTN7+1jxmfLCOMbUNLK8czXumKo+TpTbsHH7MiRmvcKSTs4AyVtKmSnIBSITNN8+qBWDapFo+cX88rcnFKhtdGVKv96Fx7bxrrwpOOci557kmPvnOSv5tDzj6ptXsOipodUrcjtPtOpHCoDyOpv7h+7s17Dz32K+ZPCaz10gnZwFlrKRNleQCkAibsSMrgeDPkyai1uSIhtKVYTD9mBPX+/4HYOO2Lt493jh33nYeer2KivKRHH5gXY/bc7pdJ1IYlMcD6+2LxNQ5Czj8w0f1e1w6eTuYnBVJlyrJBWDF80/y/PpW7l+yutv2mrVPxhrKuV4BMFOG0pVhMP2YE9d799v33LFtRkMTjDtSLRYiBS6beZyLDHaHYHz+4PX2ReL4A+DZZSuAY/o8Lp28Vc5KNqiSXAA+c+09OblusawAONiuDIPtx6yuEyLFK5t5PJgM9iFes62jg4rKqoFf2I/evkh0dXVR5m/1eUy6eauclWxQJbkEJE/DU4p95gZ7y22w/Zh1i09EepONLLbyYbRv72BY5eB+vW/e1smwsUOb07m3LxJdXV28OueKPo9JN2+Vs5INqiSXgKjzK6fe2tvUsJ4lN19C+fARHHbRddkoat7I1pRsIlI6omRxcg4nMhiInMOV1TVs2dY+6Epyc+t2hsewOmBZWRmd3vtcycpbyVfxzO6dxMyON7OXzOxlM+vxNdLMqszs/nD/U2a2X7h9PzPbZmZ/Cx+3Jh1zpJktDY/5iZkNrQNVEUsMoLju1HG8tGhev6v1JW7tJR57nn01e59/I52tW7NY4vygFZSk0Chr81vULE7O4UQGp5PDlcNH0LKtbfDl3NbBsH7mfB6Krj76OitvJV/F2pJsZuXALcBHgdVAvZnNT1mt70Jgk7tPMLOzgR8AZ4X7XnH3I3o59UzgYuCvwEPA8cDDMb2NgpY6DY9GYEej/m5SSJS1+S9bWTxyt/F855fzqRk5uIruijfWcdy/x7O632sbWrn0zp4V39/+7mm2tLRw7cKmbttH1jzN6na1JEt8jpzcyTkHHtvn/ri7W0wGXnb3lQBmdh9wCt1X7jsF+Fb48wPAzf21VpjZXsBod/9L+Hw28HEU3D0MdT7P8vJy2hreYHvLxm4DRQpxedR0qb+bFBhlbR4bbBYnMhjolsP9ZfDbj/4wHP3hQZf1kEEfObDjv3RD79c8N8aLivRj3736r8/EXUkeB6xKer4aeHdfr3H3DjNrAsaG+/Y3s+eBzcDX3f1P4euT595ZHW6TFEOdz3PPfQ4AoK1ud75714JYyyoiQ6KszWODzeJEBoNyWCQX4q4k99ZKkTpDTV+veQvY1903mNmRwG/M7LCI5wxObHYxwa1CPnnZ1Rxz8rTIBS8GuZpfWUSyTlmbx5TFIoUp7kryamCfpOfjgTV9vGa1mVUAtcBGd3egDcDdnzWzV4C3h68fP8A5CY+bBcwCuH3RyqFOH1lw0p3Ps6ZmVK/zb5ZC94pMG8xKfSJDoKzNY+lksXI4PaU+xanEK+5Kcj0w0cz2B94EzgY+kfKa+cB5wF+AM4DH3d3NbDeCAO80swOAicBKd99oZs1m9h7gKWA60HO2dUlbPq2iV2ZldHQV7u/awazUJzIEytoikU85XAiiTnEqMhixTgHn7h3ADOAR4EVgrrsvM7Nvm9nJ4ct+Bow1s5eBS4HE1EXHAEvM7O8Eg0w+6+4bw32XAHcALwOvoIEkRaeqegQtrR25LsagJK8ctWBhPRuatuS6SFLklLVSitKZ4lRkMGJfTMTdHyKYOih52zeSfm4FzuzluF8Bv+rjnM8Ah2e2pJJPqqpHsnlre66LMSiDXalPZCiUtVJqNMWpxE0r7smQpa7Ul1BTM2rQtw6HjxjJpm3bh1q0rNPKUSKSbXFkcL4b6hSnIlGokpyGUgyiKBIrRKXqbfBJVMOGV9PSWniV5P5WjlJrskh8Sjmf48jghE3/Wkvzpg1DPk+m7LHP/lRVjxjyFKciUaiSnIY4g0i6MzM6vfBWwNVKfSK5oXyOx5IHbuDUI8YO/MIs2Lh5C8+9PIFJJ56rafUkK1RJlrzlVp7rIqRNK/WJSDEZU1PN6R88LNfFAGDT5q08/btgTGm6U5yKDEass1uIDEWX/nmKiORUGV25LsIOI6uHsX1bS66LISVEtRDJW5365ykiklP5VEkeVlmBdxbmrEdSmNTdQoYsrhWiOntdFVdERJLFuUqf5VElGaC895XRRWKhSnIatFxo7+IbOa5KsohEU8r5HOfsHfl2P0+/FSSbVElOQ7FPIyQiUqiUzyKSafn2JVFEREREJOfUklzkSnmCfRGRfKEsFik8qiQXoeQw3tSwnj3PvhqA8vJy9tznACC/Jtjv65eHbd3ILRccnYMSiYhkRiLfkrMYduZxLrNYFXeR/qmSXISSV55acvMlVNXtC0Bbwxu5LFaf+lop64UfTs1BaUREMieRb8lZDPmRx1qlUKR/6pMsIiIiIpJCLcmSV5bdcRmdrVsBaG3ZQtXx3wKgzDvZa7ddAKgbVcXTMz+fqyKKiBSdRPa2t2zCuzr43JTJAHhrC/P+8JRyV0qSKsmSVzpbt7L3+TcCsO7hmxl/4iUArLrzyxz26WsBWHb7ZTkrn4hIMUpk77b1b1AxZg8qh1UB8MZNn+SwT1+n3JWSpEpykSsfPoI1d30JgO0tG2mr2x0ojQn2RUTyRXIWw848zvcs1vp2UspUSS4gUUciJ688VQ0wPPhrrqk7MO0RywNdMxOjo5PLu71lY14MaBER6U+6eZycxRA9j+PM4N6y17s6tKqdSEiV5AISdSRyJqfuGeiamRgdnVzeq86fwrj9JgKw+VmNKxWR/JStPI4zg3vL3jdfW0FF2NVCpNSpklyE/uvkd9PR1fMmWUWZ8ZP5T+WgRCIipUdZLFLYVEkuMt+fMY32jg7Gf/bn3bYb8OasT+emUGlIvv23Zc1rrFzxFyCY3SIxcKRulFo5RCT/be/oZNxnf9Zj+5u3XpiD0vQvkb1NGxro8q4d27vatvDUDZ9lVHUlcx5fumP7f06eQG1NdS6KKpI1qiTniUytfNTS0oxZOVYxrNt272gfchmzIfm9/u3/vqUV90QkJzKRyY73yOLE9nzT13vavKmBlqZNAKwIt61+5Z+MXvoSU957SJZKJ5IbqiTnCa181POXUuumdTz8hz9pfk4RybpSyuSBvhCM3qWu2/auzi5a1ryQreKJ5IwqyQUkuStC6vZMSg7Mpg0NPHvNWQCYdzFmtz27XTMTZUpcb1PDeiprdt2xvYsKzc8pInkp7jxO5GJyBsPOHM5UBn9/xjRWvfZKt+yFYMo6eqk4A1RVV9PSuj3qWxEpWKokF5BMzlrRn/5aUL5714KMlik5oG34KDo9uA1ZNmwEnc0NAHR2dnL6Fbcy68pzGVs7ckjXExHJhLjzOGoOD7YcyY0Tqdk77pzvs+auL9FVCbdfdSHTvvIjamp32XFs1fARtLR2DOq6IoVEleQC1dftscZ/rQWM1bec1227eyfDKqL9dTf+ay1Lbr6kx/aOsNKaSS0tzex59tVU1e3L9va2Hf333rp756T77Vtb2LS2kbsXPMml5xyX8TKIiAxFX3nsHdt7ZDEAnQNXMOPO4UQl/M3XVlA2eo9es7drayMV6zby9EP3cey0nWUZNryaRrUkSwlQJblA9dXK8PwPprHr2Loe29MabGJlO5aGTvb6zdN3/JypgYYJrQ2r6erq3PG8s2UjXW3b+NOtVzGmcwszTxvHJQvqOW/K+9WaLCJ5pa88bvrBNGr7yOOBDJTDmcrgzo7tdDau3fm8ZSOr7vwiHZvXU1fVxnWnHsTnF8xj8oln72hNLisvp6uXqe1Eio0qyTmUTt/fqGrH1vXoEjFYnVub2Py76xl9/GWUjxjdbV+6g1r6CvSmDQ3sDnhXF5Vjx2MWLCBSXrMr1RPfQ+erf+WUQ8o5aPcqTjhgKx/9wg08dtOXVVEWkYzLdCZnIo/7yuF0Mri3/N3UsJ61q1YC1iN7hTJD/QAAGlVJREFU9zrvRtb95ExOObiCCbtXM+WAZm7+0pnMuPGX3bpdiBQ7VZJzKJ2+v31ZdsdldLZu3fF8e8tGrjp/yqBbdJO1/eMR9m17mXX/+B0jJk/dsf37M6axqWF9j1uB5cNH0NesmX29124DUgBPzM/pjm9vo7ZrM2cdHvxSOvFAuO3JDfz0gT/yPxdOGdJ7ExFJNdRMjiOPe8vhdDO4t/e15OZL6OwM7t6lZm/H5gZ2rWznzMODCvHJE7q4d/Gb/OnXd3LCBZcO6n2IFCJVkgtcZ+vWbrfk2hreYNx+E4c0TVGZlbF19QtUvvQI/3virnzhoUfYuvfhlIUtDcn9iJOtuetLMDz9f1Ll5eV0drThSf30vKsTGldzxmGVvO/gPdje0UVFZysXHTmMu3+3mM+d8WG1JotIXslkHveXw5nO4NTsbX9+HmceXMkhBx1IZ2cnIzqb+dSkKu545Bd88LQLGDZci4hIaYi9kmxmxwM/BsqBO9z9mpT9VcBs4EhgA3CWu79mZh8FrgGGAe3A5e7+eHjME8BewLbwNMe5+/q430upqB1bx4iGf3LyoVVM3LuW0w/tZH7DP+nopW/dUJl3sfWRG3pM/1ZVU4ttXsvcZdt5ct16Nm9phY52Rg83qujUID6RFMra4hJ3DpcPH8Ha+74O0CN7Kxrf4FdruvjDLatp3dJCRedWRg8vo8a6ePqh+/jAaRdkpAwi+S7WSrKZlQO3AB8FVgP1Zjbf3ZNnIb8Q2OTuE8zsbOAHwFlAA3CSu68xs8OBR4BxSced4+7PxFn+gQxl4ETidtmbr63otr28vDzStRr/tZZnrzkL7+qireGNAY9PR/Xw4bT/9R5OOnUEbf9q4aTxXTww7x6q9zy0z2PevPdK2jf/i7bN8Lkpk3dsN+9i/P4T+zxuzG578t27FnDV+VN63A5c//jPYeVifnfTxUz9yo+ZO3UUdTUVNLR0MHWuBvGJJBR71qZrsNnctKGhRyYntvd3jUQed3V1snXtSsrDmYSGksf95fC21tYer19192V0NPXM4IoyY9Suu/V4/WEXXbejhbuvLiYX3zibe648mzln1TJ2ZCUbtmznE/fP44iPfHzQ70ukkMTdkjwZeNndVwKY2X3AKUBycJ8CfCv8+QHgZjMzd38+6TXLgOFmVuXubTGXuYf+plubdOUve2yPcmutpaWZyppde9wuS67wpr6+rz694/bruxI6GO/70Ed4+1ubqatppr2tldGVxkkTjZ8v/SefmzIZ7+qi4qEfs9uJX9xxTOe2zewx9dsM23U8FcOqdmxfc9eXev3sUvU2IX7rpnUcvO9uzH5wMVMmlFFXE/xzraupYMqEMrUmi+xUFFmbrkxnc5d39cjkxPZUffXzLa+oyEgmJ3J49LAGOju6GF3JjhxuL6uiYsGN7D5l53Rtvr2VPaZ9j2Fj9uyWwcmzEv3/9u49zK26zuP4+5uZaUtpt5QO0FIuLVpdwUUFwV1XEdcV0adQXVEpXiiiFaSu+uC69Kn6sCwK1QcvWNfVclPktuqCXS6Wi7o8sqJAgUIX1kIpdAq9TNtpOwxzS777xzmZJpmcTGYmJzlJPq/nmWcyJycn35w5+eSX37n8iik1GMlDd93CafNgxv5tQPD7tHnw8Oqf8/rho22LNJy4G8mzgU05f3cAb4max90HzWw3MIOgdyPrg8CjBaF9rZmlgV8Cl7r7sOvRmNliYDHAxy68lJNOXzimF1HOSWdj0TJpcnAMWY6B7p0cPudV41rueK1/9AEe3dbLj37bRcadlv0PBCYxZfZMpn3oG/Rs2UDXr6/M+yDY0doGWF445xppVKhivTuPXX8xPzjnBE6/cAUvbuvjxify9/IeuvXPaiSLBBoia0er0tlsnhmWydnp1ZbN4R/e203L1OwhFkEOT3jHeWxftTwvg7cCFpHBpfK3VM/6j778MR7d1sstazvypk9u/wOvP2HumF6XSD2Ju5FsRaYVBmzJeczsGILdgrmtoY+6+2Yzm0oQ3B8nONYufyHuPwZ+DLDy/g2Ju6jjMZ+6Yti0DVd9rqyzoLNnUWcG+vM+EFKWYtqM9nENjfqZb/4MgGWL5tPdO1jkWp3OQPfOvNAd6N7J8H/tPuO50saqK5aM+bEiTUJZWwEHHDRzVJe2zMrmcf+e7bx441K2pILDLMaTx9kc/uz8EznynGvy7uvt7CDd05VXV7qnC0ulii5rrPmbraFQf18vnbddMqZlitSTuBvJHcDhOX8fBrwYMU+HmbUC04CdAGZ2GHAr8Al3fzb7AHffHP7ea2Y3EuxqHBbcjSzd24O37UfL1Bm0trYNTR/o3lmRy7+V0tLaxvT2g/MuibRs0XxacuqoBF2rXqRsytoa2pfH7Xh6cCiT48rjSe2HkUq15GXwZ+efyKT2wyr6PCLNLu5G8kPAPDObC2wGzgTOKphnFXA28AfgDOA37u5mdgBwB7DU3R/IzhyG+wHu3mlmbcB84N6YX0ciZfp7mPWJb9OWs3ut4ycXsmnjsyxblH8d4bgbznFwG/9JiCJNQllbY5n+Hmad/V18sJ+2CRPZfMNSbNLUYXlcj1ks0qxibSSHx70tIThbugW4xt3XmdklwMPuvgq4GrjezJ4h6NU4M3z4EuDVwFfN7KvhtFOAl4HVYWi3EIT2yjhfR5SUpUoeZ1vKSMfojjT/QPdObNLUYftPMwO9zDzz0mEnjoznusm5Nt+wlPQre/DevUPBn0kP8sq25+m66St4QUHmGaa0j+0klsHChYlIUY2etaM11mweTS7nzpvNYx/sH8rkbCdGZs/WvDwebxZvvmEpmf4eMpl0Xga39e1m4/c/NjRyXlZrSjkqMlaxXyfZ3e8E7iyY9rWc273Ah4o87lLg0ojFHl/JGkcSFZyzj5wb2SMw0iWIRtuTUDh/9njhqBPlKmHKlKl0bV+fd3Z0JpPmkA//KxMmTmTm4UcB0Pn7mzhs4Ke8bsE5/N3C86MWN2oZih9fJyLDNULWjtZos/myJQuH7WXLLme0uZw7bzaP22LI49aUFc1gAw5/9euAbAZv5HULzq1oBos0O424V4ax7BorNbxpUoylIb9s0XyOCIMZYODl3fjT93DxO/fnkvtv5X/++76i1/Acyy7GVbffwdz77sUsvyekfepE/vTDC0a1LBFpPKPNlCTm8kg5fOWqP+ZNr0YGj2cMAJFGokZynZoyZSq7Op8ddl3lwgZlKZX4wNj96J0smGfMmd7CafPg+w908JovDg/RsXwI9fUPcuJnv0NLW/4FOdetvHDUyxIRiUuxPPb0YNHLiRQabw7HkcFJ/DIhUgtqJNeppStuCr7t33tl3vTMK3vGPepe1/Ytkbskc2V7MM44Yxr0dLPwuGlce38HAz27aZs8bVw1AFgqxWDfK8MaySIiSVIsj713L+k9W8ecx1E5nDv6X9wZLNLs1EiuY1HH3HWv/g4bCqaP5jqdbqmyehGyPRjT92+lrycYjWnBX7Zy15o7aX9b6cEEytmdZ5Yi3d9LcKUqEZHkKszj3EZzbh6Xm8VROZx7XfyxZrAOpxApjxrJDWakgMuGY9f2LaTd2XlZGLhmTJgynZZJk0s+Pu+M7s7nudEGufHBlzAzph3YQdeOAVKHroERGsll7c4Le5JFROpNOVnc8dz6/BwGMGO/9tmRjzPPjDuDdTiFSHnUSI7JaC/xVmlRPQVd27dw3NKfs3bF+cw6aznWGhzKMNC5if0OPqLokKy5Rgr+ZYvmM+fj3xp74TnMWkgP9I08o4hIGWqRy6WyuHVqe14OQ5DFu+78duTyDjhoZt4gIoUqmcEizU6N5JjUepdVVE/Bw18/g7Urzqd/704GurbsuyM9SF/nCwx07yRlY7/0WiU/hCbttx8bfnUlkyZNypvePjW+y96JSOOqRS4Xy+Itmzaw4/ovkynIYTPDM4PjyuFKZHCtO3lEkkKN5Aqp5TFexZ57V+c21l11Icd86oq86W5w6KLvsunazzOh/Yih6f3bnmP2nHn0tR88rloq+VrPWnIRZ0xZy1v/6qiKLVNEmks1s7ncLE6n07RMPgBrac3L4YHOTbS0tjJ9HDlciddU604ekaRQI7lCCnsL1l11IeneHnZ1xj8kaXf3Xia/54uk0+mhaQcNDtJ5+xVs2bRhaNCPcqkXQUQaRW42Z3MZyMvmSuVy9rm2bNowlMfZLN68cT0tLS1l57FyWKT2mqaRHHdDtVC6t4dDF32Xvs4XKjokaeTzpdNMzOmReGXbC1hLa17DuVxDlzMq6BHp7t7LZUsWlrXuRuq90QeAiFR7D1w2l4G8bK50LufmcTaLJ7YfMey69nEqtW6VvyLlaZpGcm4vb6Oewbv5hqVk+oNeEk8PknllD1tvW07XlGlDu/oMo6/zBSzVyovXLBl6bKZnN4MHzxwKyfGe/TzS47U7T0Qa9SoL2+/4Hp4ZBPZl8aZrP4+lWpl9wQogOP64MIfTPV20trQx+8i5sWZwqRP/RGSfpmkkN4NMfw+zzg56SXywn4GdmwHY/ouLh4I1ZTB7zryhoM5ScIqIVEZmoJdDPxlkbDaLLdXKlpuXsuGqz7GrcxuT2g8bds5Ibg4XG0hERKpLjeQGZa0TsFQrqRRMbz94KHgvW7JQu9lERKokm8X7HXwEqVQLX7/u9qFDIQqzWDkskixqJFdI4TFeA9076et8YVRDko71+LwpU6ay6eav4B6cHZ1lqRSQyZt3vIc57N7RGTlktQ6hEJGkyRsAKcxloKxsHm0mZ58r3dNVJIv3GU9WKoNFqkeN5AqpxJCkYz0GLXui3aaNz5KfxZlRNdLLkfFMQx5DKCKNKTebc3MZ9mVzVC6PNpOzz7XktL8elsV9nS+M6xr0Q0tSBotUTdM0knMDpBq7tKr9jX7piptYtmh+3pU0sjYUmX8kUWc/m2eKzL1Ptudl945OHrl831Cr5hkOOGimdieKyJBqX2WhWrk8bUZ70Szum9Fe9jKUwSK11zSN5GY4Ka2SHzhRHyYjnUyiM6pFpFyNenhAJbJYGSxSe03TSG4GpT5wajkioIhIM1EWizQGNZKLaMQQa9TrkYpIc6n3fFYWi9QPNZKLqFWI1cMoSPVQo4g0rmrmcxLzLok1iTQqNZITpFgvSLbXpPA4tFr1mtRDT42ISCUkMZOVwSLVo0ZywtXbrjn1cohII0t6JiuDRSpHjeQqqvdj6crRKK9DRJpDo+VyPdYsklRqJFdRLXsg4updaLQPGBFpLtXO5UpnsTJYJD5qJBfRiLur4grLpO96FJHGUu/5XOksVgaLxEeN5CKa8du3eiNEpB40ch4ph0WSRY3khKtWr4l6I0RERhZnJiuHRZJFjeSEU++BiEhyKJNFmocayVVU78fSiYg0GuWyiERRI7mKGrEHQh8wIlLP6j2XlcEi8Ym9kWxmpwLfA1qAq9z98oL7JwI/BY4HdgAfcfeN4X1LgXOBNPCP7r66nGU2kqSfyJGEGkREWVsNScxjZbBIfGJtJJtZC/AD4N1AB/CQma1y9//Nme1cYJe7v9rMzgSWAx8xs6OBM4FjgEOBe83sNeFjRlpmw6jWiRzqjRCpX8ra6og7j5XDIskSd0/yicAz7r4BwMxuBhYAuSG7ALg4vP0LYIWZWTj9ZnfvA54zs2fC5VHGMmWU1BshUteUtQ1AOSySLHE3kmcDm3L+7gDeEjWPuw+a2W5gRjj9wYLHzg5vj7RMAMxsMbA4/PMz7v7jMbyGijOzxeXWct7WlzZO3tPbWTh9x9aX2j990lFzallb7E46auhmouoqkNTakloXJLe2pNZVhrrM2iSt73JqqVYe18V6effNyamlBlRLcY1WS9yNZCsyzcucJ2p6qoxlBhODlZOIf1aBxZRZV/rlXXOK3nH5rgqWk6fs2qosqXVBcmtLal2Q3NqSWtdI6jVrk7S+R6ylinlcV+ulilRLcaqluHHXUiwEK6kDODzn78OAF6PmMbNWYBqws8Rjy1mmiEgzUdaKiFRY3I3kh4B5ZjbXzCYQnByyqmCeVcDZ4e0zgN+4u4fTzzSziWY2F5gH/KnMZYqINBNlrYhIhcV6uEV43NsSYDXBJYSucfd1ZnYJ8LC7rwKuBq4PTxbZSRDEhPP9B8FJIoPABe6eBii2zDhfRwySsiuimKTWltS6ILm1JbUuSG5tSa2rpDrO2iStb9VSnGopTrUU11C1WNCRICIiIiIiWXEfbiEiIiIiUnfUSBYRERERKaBGcgWZ2TVmts3MnsyZdqCZ3WNm68Pf0yMee3Y4z3ozO7vYPDHU9i0ze9rM1prZrWZ2QMRjN5rZE2b2mJk9XIW6LjazzeHzPWZm74t47Klm9n9m9oyZXVTJukrUdktOXRvN7LGIx8a5zg43s9+a2VNmts7MPh9Or+m2VqKuJGxnUbUlYltrZGb22pz1+5iZ7TGzLxTMc7KZ7c6Z52sVfP7E5HKScjhJ2ZuUrE1StiYpT5OUnyVqiWd7cXf9VOgHOAk4DngyZ9o3gYvC2xcBy4s87kBgQ/h7enh7ehVqOwVoDW8vL1ZbeN9GoL2K6+xi4EsjPK4FeBY4CpgAPA4cHXdtBfdfAXytButsFnBceHsq8Gfg6FpvayXqSsJ2FlVbIra1ZvkJ1+UW4MiC6ScDt8f0nInJ5STlcJKyNylZm6RsTVKeJik/o2qJa3tRT3IFufv9BGeN51oA/CS8/RPg/UUe+h7gHnff6e67gHuAU+Ouzd3vdvfB8M8HCa6DWlUR66wcQ8Pwuns/kB0ytyq1mZkBHwaqPo6su7/k7mvC23uBpwhGSKvpthZVV0K2s6h1Vo7Yt7Um8i7gWXd/vlpPmKRcTlIOJyl7k5K1ScrWJOVpkvJzpFoqvb2okRy/Q9z9JQj+ucDBReYpNqRsuRtgpXwSuCviPgfuNrNHLBh+thqWhLuTronYtVXrdfZ2YKu7r4+4vyrrzMzmAG8C/kiCtrWCunLVfDsrUlvSt7VGcibRH15/Y2aPm9ldZnZMzHUk5r1SoObvD5L3fqhJ1iYpW5OUp0nKz4j1UtHtRY3kZChnSNn4ntxsGcH1UW+ImOVv3f044L3ABWZ2Uswl/RB4FfBG4CWCXSeFarrOgIWU/qYa+zozsynAL4EvuPuech9WZFpF11tUXUnYzorUVg/bWkOwYECS04GfF7l7DcEhGG8Avg/cVs3aIlT1/56E9wfJfD9UPWuTlK1JytMk5WeJ/1FFtxc1kuO31cxmAYS/txWZp2bDv4YnF8wHPurhATuF3P3F8Pc24FaC3Sexcfet7p529wywMuL5arnOWoF/AG6JmifudWZmbQQBcYO7/2c4uebbWkRdidjOitWW9G2twbwXWOPuWwvvcPc97t4d3r4TaDOz9hhrqfl7JVcS3h/hshP1fqhF1iYpW5OUp0nKzxLrpeLbixrJ8csdCvZs4FdF5lkNnGJm08PdFaeE02JlZqcC/wyc7u49EfPsb2ZTs7fD2p4sNm8F65qV8+cHIp6vlkPm/j3wtLt3FLsz7nUWHnN1NfCUu387566abmtRdSVhOytRW9K3tUYS2cNjZjPD/xFmdiLBZ9OOGGtJTC4n4f2R8zxJez9UNWuTlK1JytMk5WeJ/xHEsb14Bc581M/QWZM3EexyGCD49nQuMAO4D1gf/j4wnPfNwFU5j/0k8Ez4c06VanuG4Fihx8Kffw/nPRS4M7x9FMHZqI8D64BlVajreuAJYC3Bm2lWYV3h3+8jOLP12UrXFVVbOP064LyCeau5zt5GsLtqbc7/7n213tZK1JWE7SyqtkRsa43+A0wmaPROy5l2XvZ9BCwJ/++PE5yM9NYKPndicjmilpq8PyJqqcn7oVgt4fTrqGLWlsiJqm8vJWqp+vZSopaqby9RtcS1vWhYahERERGRAjrcQkRERESkgBrJIiIiIiIF1EgWERERESmgRrKIiIiISAE1kkVERERECqiRLCIiIiJSQI1kaUpmdrKZ3R7ePt3MLqp1TSIijUZZK/WstdYFiFRSOBqPeTBMZlncfRUaQU1EpGzKWmkG6kmWumdmc8zsKTP7N2ANcLWZPWxm68zsX3LmO9XMnjaz3xOM756dvsjMVoS3rzOzM3Lu6w5/zzKz+83sMTN70szeXqKebjNbbmaPmNm9Znaimf3OzDaY2enhPC1m9i0ze8jM1prZZ8LpU8zsPjNbY2ZPmNmCgte4Mnxdd5vZfhVdkSIiJShrpdmokSyN4rXAT939TcCF7v5m4FjgHWZ2rJlNAlYCpwFvB2aOcvlnAavd/Y3AGwiGwoyyP/A7dz8e2AtcCrybYGz7S8J5zgV2u/sJwAnAp81sLtALfMDdjwPeCVwR9tgAzAN+4O7HAF3AB0f5GkRExktZK01Dh1tIo3je3R8Mb3/YzBYTbN+zgKMJvhA+5+7rAczsZ8DiUSz/IeAaM2sDbnP3UsHdD/w6vP0E0OfuA2b2BDAnnH4KcGxOT8o0gmDuAL5hZicBGWA2cEg4z3M5z/tIzrJERKpFWStNQz3J0iheBgh7CL4EvMvdjwXuACaF83gZyxkkfF+EvQoTANz9fuAkYDNwvZl9osQyBtw9+1wZoC9cRoZ9X0wN+Jy7vzH8mevudwMfBQ4Cjg97Urbm1N+X8xxp9CVXRKpPWStNQ41kaTR/QRDiu83sEOC94fSngblm9qrw74URj98IHB/eXgC0AZjZkcA2d18JXA0cN846VwPnh70lmNlrzGx/gl6ObWFvyDuBI8f5PCIicVDWSsPTtyNpKO7+uJk9CqwDNgAPhNN7w92Cd5hZJ/B74PVFFrES+JWZ/Qm4j7DXBDgZ+CczGwC6gVK9G+W4imAX3pqwF2U78H7gBuC/zOxhgmPxnh7n84iIVJyyVpqB7dtTISIiIiIioMMtRERERESG0eEWImNkZn8EJhZM/ri7P1GLekREGpGyVmpFh1uIiIiIiBTQ4RYiIiIiIgXUSBYRERERKaBGsoiIiIhIATWSRUREREQK/D/p3UrSxUYWJwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "\n", "# Instantiate logreg\n", "logreg = LogisticRegression(random_state=1)\n", "\n", "# Fit logreg to the training set\n", "logreg.fit(X_train, y_train)\n", "\n", "# Define a list called clfs containing the two classifiers logreg and dt\n", "clfs = [logreg, dt]\n", "\n", "# Review the decision regions of the two classifier\n", "plot_labeled_decision_regions(X_test, y_test, clfs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classification tree Learning\n", "- Building Blocks of a Decision-Tree\n", " - Decision-Tree: data structure consisting of a hierarchy of nodes\n", " - Node: question or prediction\n", " - Three kinds of nodes\n", " - **Root**: no parent node, question giving rise to two children nodes.\n", " - **Internal node**: one parent node, question giving rise to two children nodes.\n", " - **Leaf**: one parent node, no children nodes --> prediction.\n", "- Information Gain (IG)\n", "![information gain](./image/ig.png)\n", "$$ IG(\\underbrace{f}_{\\text{feature}}, \\underbrace{sp}_{\\text{split-point}} ) = I(\\text{parent}) - \\big( \\frac{N_{\\text{left}}}{N}I(\\text{left}) + \\frac{N_{\\text{right}}}{N}I(\\text{right}) \\big) $$\n", " - Criteria to measure the impurity of a note $I(\\text{node})$:\n", " - gini index\n", " - entropy\n", " - etc...\n", "- Classification-Tree Learning\n", " - Nodes are grown recursively.\n", " - At each node, split the data based on:\n", " - feature $f$ and split-point $sp$ to maximize $IG(\\text{node})$.\n", " - If $IG(\\text{node}) = 0$, declare the node a leaf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using entropy as a criterion\n", "In this exercise, you'll train a classification tree on the Wisconsin Breast Cancer dataset using entropy as an information criterion. You'll do so using all the 30 features in the dataset, which is split into 80% train and 20% test.\n", "\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='entropy',\n", " max_depth=8, max_features=None, max_leaf_nodes=None,\n", " min_impurity_decrease=0.0, min_impurity_split=None,\n", " min_samples_leaf=1, min_samples_split=2,\n", " min_weight_fraction_leaf=0.0, presort='deprecated',\n", " random_state=1, splitter='best')" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.tree import DecisionTreeClassifier\n", "\n", "# Instantiate dt_entropy, set 'entropy' as the information criterion\n", "dt_entropy = DecisionTreeClassifier(max_depth=8, criterion='entropy', random_state=1)\n", "\n", "# Fit dt_entropy to the training set\n", "dt_entropy.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Entropy vs Gini index\n", "In this exercise you'll compare the test set accuracy of dt_entropy to the accuracy of another tree named ```dt_gini```. The tree ```dt_gini``` was trained on the same dataset using the same parameters except for the information criterion which was set to the gini index using the keyword ```'gini'```." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',\n", " max_depth=8, max_features=None, max_leaf_nodes=None,\n", " min_impurity_decrease=0.0, min_impurity_split=None,\n", " min_samples_leaf=1, min_samples_split=2,\n", " min_weight_fraction_leaf=0.0, presort='deprecated',\n", " random_state=1, splitter='best')" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dt_gini = DecisionTreeClassifier(max_depth=8, criterion='gini', random_state=1)\n", "dt_gini.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy achieved by using entropy: 0.8947368421052632\n", "Accuracy achieved by using gini: 0.8859649122807017\n" ] } ], "source": [ "from sklearn.metrics import accuracy_score\n", "\n", "# Use dt_entropy to predict test set labels\n", "y_pred = dt_entropy.predict(X_test)\n", "y_pred_gini = dt_gini.predict(X_test)\n", "\n", "# Evaluate accuracy_entropy\n", "accuracy_entropy = accuracy_score(y_test, y_pred)\n", "accuracy_gini = accuracy_score(y_test, y_pred_gini)\n", "\n", "# Print accuracy_entropy\n", "print(\"Accuracy achieved by using entropy: \", accuracy_entropy)\n", "\n", "# Print accuracy_gini\n", "print(\"Accuracy achieved by using gini: \", accuracy_gini)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Decision tree for regression\n", "- Information Criterion for Regression Tree\n", "$$ I(\\text{node}) = \\underbrace{\\text{MSE}(\\text{node})}_{\\text{mean-squared-error}} = \\dfrac{1}{N_{\\text{node}}} \\sum_{i \\in \\text{node}} \\big(y^{(i)} - \\hat{y}_{\\text{node}} \\big)^2 $$\n", "$$ \\underbrace{\\hat{y}_{\\text{node}}}_{\\text{mean-target-value}} = \\dfrac{1}{N_{\\text{node}}} \\sum_{i \\in \\text{node}}y^{(i)}$$\n", "- Prediction\n", "$$ \\hat{y}_{\\text{pred}}(\\text{leaf}) = \\dfrac{1}{N_{\\text{leaf}}} \\sum_{i \\in \\text{leaf}} y^{(i)}$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Train your first regression tree\n", "In this exercise, you'll train a regression tree to predict the mpg (miles per gallon) consumption of cars in the [auto-mpg dataset](https://www.kaggle.com/uciml/autompg-dataset) using all the six available features.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Preprocess" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mpgdisplhpweightacceloriginsize
018.0250.088313914.5US15.0
19.0304.0193473218.5US20.0
236.191.060180016.4Asia10.0
318.5250.098352519.0US15.0
434.397.078218815.8Europe10.0
\n", "
" ], "text/plain": [ " mpg displ hp weight accel origin size\n", "0 18.0 250.0 88 3139 14.5 US 15.0\n", "1 9.0 304.0 193 4732 18.5 US 20.0\n", "2 36.1 91.0 60 1800 16.4 Asia 10.0\n", "3 18.5 250.0 98 3525 19.0 US 15.0\n", "4 34.3 97.0 78 2188 15.8 Europe 10.0" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mpg = pd.read_csv('./dataset/auto.csv')\n", "mpg.head()" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "mpg = pd.get_dummies(mpg)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mpgdisplhpweightaccelsizeorigin_Asiaorigin_Europeorigin_US
018.0250.088313914.515.0001
19.0304.0193473218.520.0001
236.191.060180016.410.0100
318.5250.098352519.015.0001
434.397.078218815.810.0010
\n", "
" ], "text/plain": [ " mpg displ hp weight accel size origin_Asia origin_Europe \\\n", "0 18.0 250.0 88 3139 14.5 15.0 0 0 \n", "1 9.0 304.0 193 4732 18.5 20.0 0 0 \n", "2 36.1 91.0 60 1800 16.4 10.0 1 0 \n", "3 18.5 250.0 98 3525 19.0 15.0 0 0 \n", "4 34.3 97.0 78 2188 15.8 10.0 0 1 \n", "\n", " origin_US \n", "0 1 \n", "1 1 \n", "2 0 \n", "3 1 \n", "4 0 " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mpg.head()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "X = mpg.drop('mpg', axis='columns')\n", "y = mpg['mpg']" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=3)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DecisionTreeRegressor(ccp_alpha=0.0, criterion='mse', max_depth=8,\n", " max_features=None, max_leaf_nodes=None,\n", " min_impurity_decrease=0.0, min_impurity_split=None,\n", " min_samples_leaf=0.13, min_samples_split=2,\n", " min_weight_fraction_leaf=0.0, presort='deprecated',\n", " random_state=3, splitter='best')" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.tree import DecisionTreeRegressor\n", "\n", "# Instantiate dt\n", "dt = DecisionTreeRegressor(max_depth=8, min_samples_leaf=0.13, random_state=3)\n", "\n", "# Fit dt to the training set\n", "dt.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluate the regression tree\n", "In this exercise, you will evaluate the test set performance of ```dt``` using the Root Mean Squared Error (RMSE) metric. The RMSE of a model measures, on average, how much the model's predictions differ from the actual labels. The RMSE of a model can be obtained by computing the square root of the model's Mean Squared Error (MSE).\n", "\n" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test set RMSE of dt: 4.37\n" ] } ], "source": [ "from sklearn.metrics import mean_squared_error\n", "\n", "# Compute y_pred\n", "y_pred = dt.predict(X_test)\n", "\n", "# Compute mse_dt\n", "mse_dt = mean_squared_error(y_test, y_pred)\n", "\n", "# Compute rmse_dt\n", "rmse_dt = mse_dt ** (1/2)\n", "\n", "# Print rmse_dt\n", "print(\"Test set RMSE of dt: {:.2f}\".format(rmse_dt))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Linear regression vs regression tree\n", "In this exercise, you'll compare the test set RMSE of ```dt``` to that achieved by a linear regression model. We have already instantiated a linear regression model ```lr``` and trained it on the same dataset as ```dt```.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Preprocess" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.linear_model import LinearRegression\n", "\n", "lr = LinearRegression()\n", "\n", "lr.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Linear Regression test set RMSE: 5.10\n", "Regression Tree test set RMSE: 4.37\n" ] } ], "source": [ "# Predict test set labels\n", "y_pred_lr = lr.predict(X_test)\n", "\n", "# Compute mse_lr\n", "mse_lr = mean_squared_error(y_test, y_pred_lr)\n", "\n", "# Compute rmse_lr\n", "rmse_lr = mse_lr ** 0.5\n", "\n", "# Print rmse_lr\n", "print(\"Linear Regression test set RMSE: {:.2f}\".format(rmse_lr))\n", "\n", "# Print rmse_dt\n", "print(\"Regression Tree test set RMSE: {:.2f}\".format(rmse_dt))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }