{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 02 - Ensembling\n", "\n", "Often for tabular problems, we deal with ensembling from other models. For today, we'll look at using XGBoost (Gradient Boosting) mixed in with `fastai`, and you'll notice we'll be using `fastai` to prepare our data!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install fastai" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from fastai.tabular.all import *" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's first build our `TabularPandas` object:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "path = untar_data(URLs.ADULT_SAMPLE)\n", "df = pd.read_csv(path/'adult.csv')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race']\n", "cont_names = ['age', 'fnlwgt', 'education-num']\n", "procs = [Categorify, FillMissing, Normalize]\n", "y_names = 'salary'\n", "y_block = CategoryBlock()\n", "splits = RandomSplitter()(range_of(df))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "to = TabularPandas(df, procs=procs, cat_names=cat_names, cont_names=cont_names,\n", " y_names=y_names, y_block=y_block, splits=splits)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# XGBoost\n", "\n", "* Gradient Boosting\n", "* [Documentation](https://xgboost.readthedocs.io/en/latest/)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import xgboost as xgb" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll need our `x`'s and our `y`'s" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X_train, y_train = to.train.xs, to.train.ys.values.ravel()\n", "X_test, y_test = to.valid.xs, to.valid.ys.values.ravel()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model = xgb.XGBClassifier(n_estimators = 100, max_depth=8, learning_rate=0.1, subsample=0.5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now we can fit our classifier:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "xgb_model = model.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And we'll grab the raw probabilities from our test data:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "xgb_preds = xgb_model.predict_proba(X_test)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.89155704, 0.10844298],\n", " [0.6882768 , 0.31172317],\n", " [0.79331285, 0.20668715],\n", " ...,\n", " [0.49610275, 0.50389725],\n", " [0.90957344, 0.09042657],\n", " [0.9879613 , 0.01203871]], dtype=float32)" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "xgb_preds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And check it's accuracy" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor(0.8340)" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "accuracy(tensor(xgb_preds), tensor(y_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can even plot the importance" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from xgboost import plot_importance" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc8AAAEWCAYAAAAASRzMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdeXxV1bn/8c+XmRIGEYKgIuIAIWQo\nIIh1SKqICipYWq8XVKYiWgtexakWRa9XqUrLoNWCFqiXqkVBrVirP/CgZZBBCCgF9ZZYRFTQggQC\nJOH5/bE36SEk5BxJSA553q9XXmfvtdde+9knmoe19rBkZjjnnHMudrWqOgDnnHMu0XjydM455+Lk\nydM555yLkydP55xzLk6ePJ1zzrk4efJ0zjnn4uTJ0zlXaST9QtLTVR2HcxVN/pync9WTpFygFVAU\nVXymmX1+hG0ON7P/d2TRJR5J44DTzWxQVcfiEp/3PJ2r3i43s6Son++cOCuCpDpVefzvKlHjdtWX\nJ0/nEoykppKekbRF0mZJD0qqHW47TdICSV9L2iZplqRm4bZngbbAnyXlSbpDUpakz0q0nyvponB5\nnKQXJf2vpG+BwYc7fimxjpP0v+FyO0kmaYikTZL+JWmkpLMkrZG0XdLjUfsOlrRI0uOSdkhaL+nC\nqO1tJL0q6RtJn0j6aYnjRsc9EvgFcHV47jlhvSGS/i5pp6R/SLohqo0sSZ9Juk3SV+H5Dona3lDS\nBEmfhvH9TVLDcNvZkhaH55QjKes7/bJdteXJ07nEMwMoBE4Hvg9cDAwPtwl4GGgDpAAnA+MAzOxa\n4J/8uzf7SIzHuxJ4EWgGzCrn+LHoAZwBXA1MBO4BLgJSgZ9IuqBE3f8DWgD3AXMkNQ+3PQ98Fp7r\nAOAhST8sI+5ngIeAF8JzzwjrfAX0BZoAQ4DfSOoS1cYJQFPgRGAY8ISk48JtjwFdgXOA5sAdwH5J\nJwLzgAfD8jHAS5JaxvEduWrOk6dz1dvLYe9lu6SXJbUCLgNuMbNdZvYV8BvgPwDM7BMze8vM9prZ\nVuDXwAVlNx+TJWb2spntJ0gyZR4/Rv9tZnvM7E1gF/CcmX1lZpuBdwkS8gFfARPNrMDMXgA2AH0k\nnQz8ALgzbGs18DRwXWlxm1l+aYGY2Twz+z8LLATeBM6LqlIAPBAe/3UgD+ggqRYwFBhtZpvNrMjM\nFpvZXmAQ8LqZvR4e+y1gRfi9uWOEXwdwrnrrF31zj6TuQF1gi6QDxbWATeH2VsAkggTQONz2ryOM\nYVPU8imHO36Mvoxazi9lPSlqfbMdfFfjpwQ9zTbAN2a2s8S2bmXEXSpJlxL0aM8kOI/vAWujqnxt\nZoVR67vD+FoADQh6xSWdAvxY0uVRZXWBt8uLxyUOT57OJZZNwF6gRYk/6gc8BBiQZmbfSOoHPB61\nveTt9bsIEgYA4bXLksOL0fuUd/yKdqIkRSXQtsCrwOdAc0mNoxJoW2Bz1L4lz/WgdUn1gZcIequv\nmFmBpJcJhr7Lsw3YA5wG5JTYtgl41sx+eshe7pjhw7bOJRAz20IwtDhBUhNJtcKbhA4MzTYmGFrc\nEV57u71EE18C7aPWPwIaSOojqS7wS6D+ERy/oiUDoyTVlfRjguu4r5vZJmAx8LCkBpLSCa5J/u9h\n2voSaBcOuQLUIzjXrUBh2Au9OJagwiHs3wO/Dm9cqi2pZ5iQ/xe4XFLvsLxBePPRSfGfvquuPHk6\nl3iuI/jDv45gSPZFoHW47X6gC7CD4KaVOSX2fRj4ZXgNdYyZ7QBuIrheuJmgJ/oZh3e441e09whu\nLtoG/A8wwMy+DrddA7Qj6IXOBe4r5/nV2eHn15LeD3uso4A/EZzHfxL0amM1hmCIdznwDfAroFaY\n2K8kuLt3K0FP9Hb87+0xxV+S4JyrliQNJnihw7lVHYtzJfm/hJxzzrk4efJ0zjnn4uTDts4551yc\nvOfpnHPOxcmf86wBmjVrZqeffnpVhxGzXbt20ahRo6oOIyaJFCskVryJFCskVryJFCtUXbwrV67c\nZmalvlbRk2cN0KpVK1asWFHVYcQsEomQlZVV1WHEJJFihcSKN5FihcSKN5FihaqLV9KnZW3zYVvn\nnHMuTp48nXPOuTh58nTOOefi5MnTOeeci5MnT+eccy5Onjydc865OHnydM455+LkydM555yLkydP\n55xzLk6ePJ1zzrk4efJ0zjnn4uTJ0znnnIuTJ0/nnHMuTp48nXPOuTh58nTOOVft7dmzh+7du5OR\nkUFqair33XffQdtHjRpFUlJS8fqnn37KhRdeSHp6OllZWXz22WfF2+644w5SU1NJSUlh1KhRmFnc\n8XjyrGCSRkn6u6RZh6mTVwHHGSypzZG245xziaB+/fosWLCAnJwcVq9ezRtvvMHSpUsBWLFiBf/6\n178Oqj9mzBiuu+461qxZw7333svdd98NwOLFi1m0aBFr1qzhgw8+YPny5SxcuDDueHwy7Ip3E3CR\nmX1Wbs0jMxj4APi8vIr5BUW0u2teJYdTcW5LK2RwgsSbSLFCYsWbSLFCYsWbSLECzLikEZKKe5YF\nBQUUFBQgiaKiIm6//Xb++Mc/Mnfu3OJ91q1bx69//WsAsrOz6devHwCS2LNnD/v27cPMKCgooFWr\nVnHH5D3PCiTpKaA98BdJOyT9XlJE0j8kjSql/hOSrgiX50r6fbg8VNL/hMtjJW2Q9DdJz0kaI2kA\n0A2YJWm1pIZH7yydc65qFBUVkZmZSXJyMr169aJHjx48/vjjXHHFFbRu3fqguhkZGcyZMweAuXPn\nsnPnTr7++mt69uxJdnY2rVu3pnXr1vTu3ZuUlJS4Y9F3Get1ZZOUS5DYbgYuBrKBxsAG4AQzK5CU\nZ2ZJkv4D6Gpmt0taBuw3s7MlTQeeB74BpgFnA3WB94HfmdljkiLAGDNbUUYcI4ARAC1atOx678Rp\nlXfSFaxVQ/gyv6qjiE0ixQqJFW8ixQqJFW8ixQpwatPaB13PzMvLY+zYsQwePJinn36aiRMnUrt2\nbS699FL+8pe/ALBt2zYmT57Mli1bSE9P55133mH69Ons2LGDKVOmFF8zHTNmDDfccAPp6emHHDc7\nO3ulmXUrLSYftq1c88xsL7BX0ldAKyB6OPdd4BZJnYB1wHGSWgM9gVHAMOAVM9sD7JH051gPbGZT\ngakAbdufbhPWJs6v+ra0QhIl3kSKFRIr3kSKFRIr3kSKFYJh26ysrIPK3n//fbZv387WrVsZNmwY\nAHv37mX48OF88sknAAwYMAAIkm3Hjh3p27cvjz76KH369OHSSy8FYPny5ezZs+eQ9suTON9eYtob\ntVxEie/bzDZLagZcArwDNAd+AuSZ2U5JFRJEw7q12TC+T4W0dTREIhFyB2ZVdRgxSaRYIbHiTaRY\nIbHiTaRYIYh369at1K1bl2bNmpGfn89bb73FnXfeyRdffFFcLykpqThxbtu2jebNm1OrVi0efvhh\nhg4dCkDbtm2ZNm0ad999N2bGwoULueWWW+KOya95Vr2lwC0EyfNdYEz4CbAIuFxSA0lJQN+o/XYS\nDAc759wxb8uWLWRnZ5Oens5ZZ51Fr1696Nu3b5n1I5EIHTp04Mwzz+TLL7/knnvuAYLe6GmnnUZa\nWhoZGRlkZGRw+eWXxx2P9zyr3rvAxWb2iaRPCXqf7wKY2XJJrwJrgC+BtcCOcL8ZwFOS8oGeZpZA\nVzCccy4+6enprFq16rB18vL+/RTggAEDiodto9WuXZvf/e53RxyPJ88KZmbtwsVxJco7Ry0nRS0/\nAzwTLhcAjUo0+ZiZjZP0PYLe6cqw7kvASxUcvnPOuRh48qz+poY3FDUAZprZ+1UdkHPO1XSePKs5\nM/vPqo7BOefcwfyGIeeccy5Onjydc865OHnydM455+LkydM555yLkydP55xzLk6ePJ1zzrk4efJ0\nzjnn4uTJ0znnnIuTJ0/n3DFn06ZNZGdn06lTJ1JTU5k0aRIAs2fPJjU1lVq1arFixcFT4a5Zs4ae\nPXuSmppKWloae/bsAeCSSy4hIyOD1NRURo4cSVFR0VE/H1f9ePJ0zh1z6tSpw4QJE1i3bh1Lly7l\niSeeYN26dXTu3Jk5c+Zw/vnnH1S/sLCQQYMG8dRTT/Hhhx8SiUSoW7cuAH/605/Iycnhgw8+YOvW\nrcyePbsqTslVM/56vhogv6CIdnfNq+owYnZbWiGDEyTeRIoVEiveI4k1d3wfWrduDUDjxo1JSUlh\n8+bN9OrVq9T6b775Junp6WRkZABw/PHHF29r0qQJECTYffv2UVHz7LrE5j3PakDSy5JWSvpQ0oiw\nbJikjyQtkzRN0uNheUtJL0laHv78oGqjd656y83NZdWqVfTo0aPMOh999BGS6N27N126dOGRRx45\naHvv3r1JTk6mcePGpU5z5WoemVlVx1DjSWpuZt9IaggsB3oTTITdhWDS6wVAjpndLOmPwG/N7G+S\n2gJ/NbOUUtocAYwAaNGiZdd7J047WqdzxFo1hC8TZHbSRIoVEiveI4k17cSmAOTn5zN69GgGDRp0\n0FDtLbfcwo033kiHDh0AeOGFF3j55Zd56qmnqF+/PrfddhtDhw6la9euxfvs27ePBx98kCuuuIJu\n3bodcsy8vDySkpIOKa+OEilWqLp4s7OzV5rZob9sfNi2uhglqX+4fDJwLbDQzL4BkDQbODPcfhHQ\nKWroqImkJDPLi27QzKYCUwHatj/dJqxNnF/1bWmFJEq8iRQrJFa8RxJr7sAsCgoK6Nu3LyNHjuTW\nW289aHuzZs3o2rVrcRL84osv2L17N1deeSUAy5cvZ//+/WRlZR203xdffMGyZcsYM2bMIceMRCKH\n1K+uEilWqJ7xJsb/RccwSVkECbGnme2WFAHWA4f0JkO1gLPNbE+sx2hYtzYbxvc50lCPmkgkQu7A\nrKoOIyaJFCskVrxHEquZMWzYMFJSUg5JnKXp3bs3jzzyCLt376ZevXosXLiQ//qv/yIvL4+dO3fS\nunVrCgsLmTdvHuedd953iskdW/yaZ9VrCvwrTJwdgbOBRsAFko6TVAf4UVT9N4GfH1iRlHlUo3Uu\nASxatIhnn32WBQsWkJmZSWZmJq+//jpz587lpJNOYsmSJfTp04fevXsDcNxxx3Hrrbdy1llnkZmZ\nSZcuXejTpw+7du3iiiuuID09nczMTJKTkxk5cmQVn52rDrznWfXeAEZK+juwAVgKbAYeApYB3xD0\nRHeE9UcBT0haQ/D7ewfw/5udi3LuuedS1v0c/fv3L7V80KBBDBo06KCyVq1asXz58gqPzyU+T55V\nzMz2ApeWLJe0wsymhj3PucDLYf1twNVHN0rnnHPRfNi2+honaTXwAbCRMHk655yret7zrKbM7NDb\n+ZxzzlUL3vN0zjnn4uTJ0znnnIuTJ0/nnHMuTp48nXPOuTh58nTOOefi5MnTOeeci5MnT+eccy5O\nnjydc865OHnydC7BDB06lOTkZDp37lxcNnbs2OKXl1988cV8/vnnAKxfv56ePXtSv359HnvsseL6\nmzZtIjs7m06dOpGamsqkSZOO+nk4l8g8eVYhSZmSLotav0LSXVUZk6v+Bg8ezBtvvHFQ2e23386a\nNWtYvXo1ffv25YEHHgCgefPmTJ48+ZD5J+vUqcOECRNYt24dS5cu5YknnmDdunVH7RycS3T+er6q\nlQl0A14HMLNXgVcr+iD5BUW0u2teRTdbaW5LK2RwgsR7NGPNDedkPf/888nNzT1oW5MmTYqXd+3a\nxYHJ0pOTk0lOTmbevINjbN26Na1btwagcePGpKSksHnzZjp16lSJZ+DcsaNG9jwl3Srpg/DnlrDs\nOklrJOVIejYsayVpbliWI+kcSe0kfRDV1hhJ48LliKRJklaHbXcPy7tLWiJplaTFkjpIqgc8AFwd\n1r9a0mBJj4f7tJO0IIxpvqS2YfkMSZPDdv4hacBR/fJctXXPPfdw8sknM2vWrOKeZyxyc3NZtWoV\nPXr0qMTonDu2qKw5745VkroCMwgmnRbwHjACmA6cY2bbJDU3s28kvQAsMbOJkmoDScBxwGtm1jls\nbwyQZGbjJEWAj83sp5LOB35rZp0lNQF2m1mhpIuAG83sR5IGA93M7OawreJ1SX8GXjSzmZKGAleY\nWT9JMwgmy74a6Ai8amanl3KeI8LzokWLll3vnTitor/KStOqIXyZX9VRxOZoxpp2YtPi5S+++IK7\n776b6dOnH1Jv1qxZ7Nu3jyFDhhSXzZgxg4YNG9KnTx+SkpKKy/Pz8xk9ejSDBg3i/PPPr9wTiFNe\nXt5BsVZ3iRRvIsUKVRdvdnb2SjPrVtq2mjhsey4w18x2AUiaQzB0OjucKxMz+yas+0PgurCsCNgh\n6bhy2n8urP+OpCaSmgGNgZmSzgAMqBtDnD2Bq8LlZ4FHora9bGb7gXWSWpW2s5lNBaYCtG1/uk1Y\nmzi/6tvSCkmUeI9mrLkDs/69nJtLo0aNyMrKOqRe+/btueyyy5g5c2ZxWSQSISkpiaSkpOJ9CgoK\n6Nu3LyNHjuTWW2+t5OjjF4lESj2/6iqR4k2kWKF6xpsYf6Gql0IOHu5uUGJ7ya68Af8NvG1m/SW1\nAyJHGMPeqGWVV7lh3dpsCK+XJYJIJHJQoqjOqkusH3/8MWeccQYAr7zyCh07djxsfTNj2LBhpKSk\nVMvE6Vx1VxOT57vADEnjCRJPf+AGYLqkX5vZ1weGbYH5wI1A9LDtl0CypOOBPKAvEH3r49XA25LO\nBXaY2Q5JTYHN4fbBUXV3EvRKS7MY+A+CXufAMG7nuOaaa4hEImzbto2TTjqJ+++/n9dff50NGzZQ\nq1YtTjnlFJ566ikgGN7t1q0b3377LbVq1aJu3br83//9H2vWrOHZZ58lLS2NzMxMAB566CEuu+yy\nwx3aOReqccnTzN4PrxsuC4ueNrNFkv4HWCipCFhFkORGA1MlDQOKCK5VLpH0QLj/ZmB9iUPskbSK\nYGh2aFj2CMGw7S+B6Nse3wbukrQaeLhEOz8nSOi3A1uBITgHPPfcc4eUDRs2rNS6J5xwAp999lnx\neiQSoUmTJpx77rnUtPsdnKtINS55ApjZr4FflyibCcwsUfYlcGUp+08GJpfR/P+a2S0l6i8Bzowq\n+mVY/g1wVon9Z4TbPiW45lry2INLrCfOVX/nnDtG1MhHVZxzzrkjUSN7npXFzLKqOgbnnHOVz3ue\nzjnnXJw8eTrnnHNx8uTpnHPOxcmTp3POORcnT57OOedcnDx5Ouecc3Hy5Omcc87FyZOnO+Zt2LCB\nzMzM4p8mTZowceJEZs+eTWpqKrVq1WLFihXF9QsKCrj++utJS0sjJSWFhx8u+eZE51xN5y9JqGCS\ncgnm5NwWQ91xQJ6ZPVbZcdVkHTp0YPXq1QAUFRVx4okn0r9/f3bv3s2cOXO44YYbDqo/e/Zs9u7d\ny9q1a9m9ezedOnXimmuuoV27dlUQvXOuOvLkWYHCmVeqnfyCItrdNa/8itXEbWmFDK6geHNLTMU2\nf/58TjvtNE455ZQy95HErl27KCwsJD8/n3r16tGkSZMKicc5d2zwYduQpNsljQqXfyNpQbj8Q0mz\nJF0jaa2kDyT9Kmq/PEkTJOUQTGB9oLyhpL9I+mm4fp2kNZJyJD1byvF/Kml5uP0lSd8Ly38cHjNH\n0jthWaqkZZJWh22eUalfzjHk+eef55prrjlsnQEDBtCoUSNat25N27ZtGTNmDM2bNz9KETrnEoH3\nPP/tXeA2gtlSugH1JdUFzgM+An4FdAX+BbwpqZ+ZvQw0At4zs9sg6LUQzPv5PPAHM/uDpFSCmVTO\nMbNtkkr7SzzHzKaFbTwIDAOmAPcCvc1ss6RmYd2RwCQzmyWpHnBIj1fSCGAEQIsWLbk3rfAIv56j\np1XDoPdZESKRSPFyQUEBL730En379j2ofPv27axcuZK8vDwA1q5dy7Zt23juuefYuXMno0ePJikp\niTZt2hzSfl5e3kFtVXeJFG8ixQqJFW8ixQrVM15Pnv+2EugqqQmwF3ifIImeB/wZiJjZVgBJs4Dz\ngZcJ5vl8qURbrwCPmNmscP2HwOwD10HDqchK6hwmzWYEyfevYfkigsm7/wTMCcuWAPdIOokg6X5c\nsjEzmwpMBWjb/nSbsDZxftW3pRVSUfHmDswqXn7llVfo0aMHV1111UF1mjVrRteuXenWrRsQXPO8\n/vrrueiiiwD485//TJ06dcjKyqKkSCRSanl1lUjxJlKskFjxJlKsUD3jTZy/qJXMzAokbSSYBHsx\nsAbIBk4Hcgl6naXZY2ZFJcoWAZdI+qPFPuPwDKCfmeVIGgxkhXGNlNQD6AOslNTVzP4o6b2w7HVJ\nN5jZgrIabli3NhtKXPurziKRyEFJr6I899xz5Q7ZArRt25YFCxZw7bXXsmvXLpYuXcott9xS7n7O\nuZrDr3ke7F1gDPBOuDwSWAUsAy6Q1CK8KegaYOFh2rmXYHj3iXB9AfBjSccDlDFs2xjYEg4VDzxQ\nKOk0M3vPzO4FtgInS2oP/COclPsVIP27nnBNsWvXLt56662Dep1z587lpJNOYsmSJfTp04fevXsD\n8LOf/Yy8vDxSU1M566yzGDJkCOnp/hU75/7Ne54Hexe4B1hiZrsk7QHeNbMtku4C3gYEzDOzV8pp\nazTwe0mPmNkdkv4HWCipiCAhDy5RfyzwHkGCfI8gmQI8Gt4QJGA+kAPcCVwrqQD4AnjoiM66BmjU\nqBFff/31QWX9+/enf//+h9RNSkpi9uzZRys051wC8uQZxczmA3Wj1s+MWn4OeK6UfZJKrLeLWh0S\nVT4TmFmi7rio5SeBJ0tp/6qSZcD48Mc551wV8GFb55xzLk6ePJ1zzrk4efJ0zjnn4uTJ0znnnIuT\nJ0/nnHMuTjElT0mnSaofLmdJGhX1qjjnnHOuRom15/kSUCTpdIJXvp0M/LHSonLOOeeqsViT534z\nKwT6A1PM7HagdeWF5ZxzzlVfsSbPAknXANcDr4VldQ9T3znnnDtmxZo8hxDMVfk/ZrZR0qnAIXNS\nOuecczVBTMnTzNYRvE/1/XB9o5n96vB7OXf0bN++nQEDBtCxY0dSUlJYsmQJY8eOJT09nczMTC6+\n+GI+//xzANavX0/Pnj2pX78+jz32WBVH7pxLRLHebXs5sBp4I1zPlPRqZQZW3UgaLOnxCm6zn6RO\nUesPSLqoIo9RU4wePZpLLrmE9evXk5OTQ0pKCrfffjtr1qxh9erV9O3blwceeACA5s2bM3nyZMaM\nGVPFUTvnElWsL4YfB3QHIgBmtjqcFssdmX4E15DXAYTTjlW4/IIi2t01rzKarhS3pRUyOMZ4c8f3\nYceOHbzzzjvMmDEDgHr16lGvXr2D6u3atQtJACQnJ5OcnMy8eYnznTjnqpeYbxgysx0lyvZXdDBV\nSdIgScskrZb0O0m1JQ2R9JGkZcAPourOkDQgaj0vavlOSWsl5UgaH5b9VNLysOwlSd+TdA5wBcGU\nY6vDZ2mL25V0oaRVYVu/j3rONlfS/ZLeD7d1PEpfUbW1ceNGWrZsyZAhQ/j+97/P8OHD2bVrFwD3\n3HMPJ598MrNmzSrueTrn3JGSmZVfSXqGYC7Ju4AfAaOAumY2snLDOzokpQCPAFeZWYGk3xLMqfnf\nQFdgB8FcnqvM7GZJM4DXzOzFcP88M0uSdCnBvJwXmdluSc3N7BtJx5vZ12HdB4EvzWxKKe3MIOiJ\nvgZ8DFxoZh9J+gPwvplNlJQLTAj3vwnoYmbDSzmnEcAIgBYtWna9d+K0SvjmKkerhvBlfmx1005s\nyoYNG7jpppuYMmUKnTp1YsqUKTRq1IihQ4cW15s1axb79u1jyJDiWeKYMWMGDRs25Oqrr/7Osebl\n5ZGUlFR+xWoikeJNpFghseJNpFih6uLNzs5eaWbdStsW67Dtzwkmid5L8HKEvwIPVkx41cKFBEly\neTi01xA4B4iY2VYASS8AZ5bZQuAiYLqZ7QYws2/C8s5h0mwGJBF8f4fTAdhoZh+F6zOBnwETw/U5\n4edKoLT5PjGzqQQvtKBt+9NtwtrEmbr1trRCYo03d2AWHTt25OGHH+amm24CoHbt2owfP56srKzi\neu3bt+eyyy5j5sx/T6kaiURISko6qF68IpHIEe1/tCVSvIkUKyRWvIkUK1TPeMv9CyWpNjDPzLIJ\nEuixSMBMM7u7uEDqRxmJCSgkHPKWVAuoV0a9A2YA/cwsR9JgIOsI490bfhYRw++wYd3abBjf5wgP\nefREIhFyB2bFXP+EE07g5JNPZsOGDXTo0IH58+fTqVMnPv74Y8444wwAXnnlFTp2rPEj3M65ClLu\nH14zK5K0X1LTUq57HivmA69I+o2ZfSWpObAKmCTpeOBb4MdATlg/l6Cn+ieC65YHXhjxFnCvpFnR\nw7ZAY2CLpLrAQGBzWH9nuK2kDUA7Saeb2SfAtcDCij3lY8uUKVMYOHAg+/bto3379kyfPp3hw4ez\nYcMGatWqxSmnnMJTTz0FwBdffEG3bt349ttvqVWrFhMnTmTdunU0adKkis/COZcoYh3LywPWSnoL\n2HWg0MxGVUpUR5mZrZP0S+DNsCdZQDBMOg5YAmwneFTngGkEyTaH4PGdXWE7b0jKBFZI2ge8DvyC\n4Droe8DW8PNAwnwemCZpFFB8A5KZ7ZE0BJgtqQ6wHHiqMs79WJGZmcmKFSsOKnvppZdKrXvCCSfw\n2WefHY2wnHPHqFiT5xz+fZ3tmGRmLwAvlCheCkwvpe6XwNlRRXdGbRsPjC9R/0ngyVLaWQR0iioa\nHLVtPvD9UvZpF7W8giMfAnbOORenmJKnmc0sv5ZzzjlXM8SUPCVtBA55psXM/EUJzjnnapxYh22j\nn3NpQHDzTPOKD8c555yr/mJ9MfzXUT+bzWwikDjPPjjnnHMVKNZh2y5Rq7UIeqKJ89S9c845V4Fi\nTYATopYLgY3ATyo+HOecc676izV5DjOzf0QXhBNiO+ecczVOrLOqvBhjmXPOOXfMO2zPM5zuKhVo\nKin6Pa9NCO66dc4552qc8oZtOwB9CWYDuTyqfCfw08oKyjnnnKvODjtsa2avmNkQoK+ZDYn6GWVm\ni49SjM6Vafv27QwYMICOHYiEyaQAACAASURBVDuSkpLCkiVLmD17NqmpqdSqVeug990uW7aMzMxM\nMjMzycjIYO7cuVUYuXMukcV6w9AqST8jGMItHq41s6Fl73J0hFN8dTOzmyuwzX7AR2a2Llx/AHjH\nzP5fRR3DVYzRo0dzySWX8OKLL7Jv3z52795Ns2bNmDNnDjfccMNBdTt37syKFSuoU6cOW7ZsISMj\ng8svv5w6dfypK+dcfGL9q/EssB7oDTxAMK3W3ysrqGqgH/AasA7AzO6t2nCOTH5BEe3umlfVYcTs\ntrRCBscQb87d5/LOO+8wY8YMAOrVq0e9evVo1qxZqfW/973vFS/v2bOHcOJz55yLW6x3255uZmOB\nXeFL4vsAPSovrH+TNEjSMkmrJf1OUm1JQyR9JGkZ8IOoujMkDYhaz4tavlPSWkk5ksaHZT+VtDws\ne0nS9ySdQzBH56PhMU+LblfShZJWhW39XlL9sDxX0v2S3g+3lTrzcln1JI2TNCaq3geS2oU/68MY\nPpI0S9JFkhZJ+lhS9wr9whPIxo0badmyJUOGDOH73/8+w4cPZ9euXYfd57333iM1NZW0tDSeeuop\n73U6576TWP9yFISf2yV1Br4AkisnpH+TlAJcDfzAzAok/RYYBNxPMBn1DuBtgomrD9fOpcCVQI8D\nk1SHm+aY2bSwzoMEz7NOkfQq8JqZvRhuO9BOA2AGcKGZfSTpD8CNwMSwvW1m1kXSTcAYYHgZIcVa\n74DTCd4nPJRgbs//BM4lSPK/IOgplzznEcAIgBYtWnJvWmE5h6g+WjUMep/lee+991i5ciWDBw9m\n8ODBTJkyhRtvvJGhQ4OrCdu3b2flypXk5eUdtN8TTzzBp59+yi9+8QsaNWpEvXr1vnOseXl5RCKR\n77z/0ZZI8SZSrJBY8SZSrFA94401eU6VdBzBpM6vAknA0RjKvJAgSS4PE1hD4BwgYmZbASS9AJxZ\nTjsXAdPNbDeAmX0TlncOk2YzgnP6azntdAA2mtlH4fpMgkmzDyTPA3OergSuomyx1jtgo5mtBZD0\nITDfzEzSWqBdaTuY2VRgKkDb9qfbhLWJ08O6La2QWOJdesuVPPzww9x0000A1K5dm/Hjx5OVlQVA\ns2bN6Nq1K926dSt1/5kzZ9K8efMyt8ciEokUHy8RJFK8iRQrJFa8iRQrVM94Y53P8+lwcSFwNKch\nEzDTzO4uLghu5ikr4RQSDkVLqgWU16WYAfQzs5zwxqOsI4x3b/hZRPjdSvor0ApYYWbDy6oXHXso\n+jnavVHL+6PW9xPD77Bh3dpsGJ847/GPRCLkDsyKqe7JJ5/Mhg0b6NChA/Pnz6dTp05l1t24cSMn\nn3wyderU4dNPP2X9+vW0a9euYoJ2ztUoMV3zlNRK0jOS/hKud5I0rHJDA2A+MEBScnjc5gRDtBdI\nOl5SXYLhzANyCXqqEAxp1g2X3wKGSPpeVDsAjYEtYTsDo9rZGW4raQPQTtLp4fq1BP+gKJOZ9Taz\nzKjEWZZcoEsYXxfAX38YgylTpjBw4EDS09NZvXo1v/jFL5g7dy4nnXQSS5YsoU+fPvTu3RuAv/3t\nb2RkZJCZmUn//v357W9/S4sWLar4DJxziSjWsbwZwHTgnnD9I+AF4JlKiKmYma2T9EvgzbAnWUAw\nTDoOWAJsB1ZH7TINeEVSDvAGsCts5w1JmcAKSfuA1wmuFY4F3gO2hp8HEubzwDRJo4DiG5DMbI+k\nIcBsSXUIrj8+VUGn+xJwXTgs+x7Bd+zKkZmZedCznAD9+/enf//+h9S99tprufbaa49WaM65Y1is\nybOFmf1J0t0AZlYoqagS4ypmZi8QJOpoSwmSecm6XwJnRxXdGbVtPDC+RP0ngSdLaWcRED3+Nzhq\n23zg+6Xs0y5qeQVlDAGXVc/M8oGLS9sH6By1T3QsudHbnHPOHR2xPqqyS9LxgAFIOpvgTlfnnHOu\nxom153krwV22p0laBLQkajjTOeecq0nKm1WlrZn908zel3QBwaMaAjaYWcHh9nXOOeeOVeUN274c\ntfyCmX1oZh944nTOOVeTlZc8o1/+eTSf73TOOeeqrfKSp5Wx7JxzztVY5d0wlCHpW4IeaMNwmXDd\nzKxJpUbnnHPOVUOHTZ5mVvtoBeKcc84lilif83TOOedcyJOnc845FydPnu6o27NnD927dycjI4PU\n1FTuu+8+ABYsWECXLl0YMmQI119/PYWFwZyeO3bs4PLLLy+uP336IW9mdM65o8qTZ0hSXjnbm4WT\nVx9YbyPpxQqOISLpkMklJXWTNLkij1WV6tevz4IFC8jJyWH16tW88cYbLF68mOuvv57nn3+e6dOn\nc8oppzBz5kwgmLy6U6dO5OTkEIlEuO2229i3b18Vn4VzriZLnBmSK4CCGbVlZvu/w+7NgJuA3wKY\n2eccpVcUhi+QX1FuxTLkFxTR7q55FRjRd5c7vg+SSEpKAqCgoICCggJq165NvXr1OPPMM/n888/p\n1asXDz/8MMOGDUMSO3fuxMzIy8ujefPm1KlTo/7Tdc5VM8d8z1NSO0kbJP0B+AAYK2m5pDWS7i+l\nfpKk+ZLel7RW0pXhpvEE7/ZdLenRsN0Pwn0aSJoe1l8lKTssHyxpjqQ3JH0s6ZGwvLakGZI+CPf5\nr6gQfixpmaSPJJ0X1s+S9Fq4PE7Ss5KWhG3+tNK+vEpUVFREZmYmycnJ9OrVi+7du1NYWFg8vdiL\nL77Ipk2bALj55pv5+9//Tps2bUhLS2PSpEnUqnXM/6frnKvGaso/388ArgeaEPQWuxM8q/qqpPPN\n7J2ounuA/mb2raQWwFJJrwJ3AZ3NLBOCpBy1z88InntNk9SRYP7RM8NtmQRTmO0FNkiaAiQDJ5pZ\n57CtZlFt1TGz7pIuA+4DLirlfNIJpl5rBKySNC/sCReTNAIYAdCiRUvuTSuM9buqVJFIpHh54sSJ\n5OXlMXbsWDp27Mgdd9zB0KFD2bNnDz169CA/P59IJMLChQtp0aIFf/zjH/n8888ZPnw4Tz/9NI0a\nNaq6Ewnl5eUddE7VXSLFm0ixQmLFm0ixQvWMt6Ykz0/NbKmkxwjmzFwVlicRJNbo5CngIUnnA/uB\nE4FW5bR/LjAFwMzWS/oUOJA855vZDgBJ64BTgA+B9mEinQe8GdXWnPBzJdCujOO9Es7/mS/pbYJ/\nDES/hxgzmwpMBWjb/nSbsLZ6/KpzB2YdUvb+++/z9ddfM2bMGH72s58RiUTYt28fe/fuJSsri0cf\nfZS77rqL8847D4BnnnmGli1b0r1796Mc/aEikQhZWVlVHUbMEineRIoVEiveRIoVqme81eMvauXb\nFX4KeNjMfneYugMJplzramYFknKBBkdw7L1Ry0UEPct/ScoAegMjgZ8AQ0vUL6Ls30/JVyUe9tWJ\nDevWZsP4PnEFXZm2bt1K3bp1adasGfn5+bz11lvceeedfPXVVyQnJ7Nv3z5+9atfcc899wDQtm1b\n5s+fz3nnnceXX37Jhg0baN/eX7XsnKs6Ne3C0V+BoZKSACSdKCm5RJ2mwFdh4swm6CkC7AQal9Hu\nuwRJl3C4ti2woawgwuHgWmb2EvBLoEuc53FleJ31eCALWB7n/lVqy5YtZGdnk56ezllnnUWvXr3o\n27cvjz76KCkpKQwfPpzLL7+cH/7whwCMHTuWxYsXk5aWxoUXXsivfvUrWrRoUcVn4ZyryWpKzxMA\nM3tTUgqwJLjxljxgEPBVVLVZwJ8lrSW4w3V9uO/XkhaFNwn9BXgiap/fAk+G+xQCg81sb3iM0pwI\nTJd04B8vd8d5KmuAt4EWwH+XvN5Z3aWnp7Nq1apDyh999FEeffTRQ4Zo2rRpw5tvvnlIfeecqyrH\nfPI0s1ygc9T6JGBSKfWSws9tQM8y2vrPEkWdw/I9wJBS6s8AZkSt943afEhv08yyopa3EV7zNLMI\nEImqusbMristRuecc5Wvpg3bOuecc0fsmO95HmvMbFxVx+CcczWd9zydc865OHnydM455+LkydM5\n55yLkydP55xzLk6ePJ1zzrk4efJ0zjnn4uTJ0znnnIuTJ09XYTZt2kR2djadOnUiNTWVSZOCFzmt\nXr2as88+m8zMTLp168ayZcsAmDVrFunp6aSlpXHOOeeQk5NTleE751zManzylNRG0ovhcmY4j2Z5\n+xRPTh3jMfpJ6lRR9aqrOnXqMGHCBNatW8fSpUt54oknWLduHXfccQf33Xcfq1ev5oEHHuCOO+4A\n4NRTT2XhwoWsXbuWsWPHMmLEiCo+A+eci02NTp6S6pjZ52Y2ICzKBMpNnt9BPyCWpBhrvWqpdevW\ndOkSvLK3cePGpKSksHnzZiTx7bffArBjxw7atGkDwDnnnMNxxx0HwNlnn81nn31WNYE751ycEvL1\nfJLaAW8AS4FzCKbkmg7cDyQTTg9G8AL4BkA+MMTMNkgaDFxFMBF2bUnXA68RvKj9AaChpHOBh4GN\npbVRTmzjgSsIZld5k2By6yuACyT9EvgR8ENgBFAP+AS4liBxl6z3DDDGzFaE05itMLN2klLD861H\n8A+gH5nZx2XFlF9QRLu75h0u7COWW2K+0NzcXFatWkWPHj2YOHEivXv3ZsyYMezfv5/Fixcfsv8z\nzzzDpZdeWqkxOudcRZHZYedRrpbC5PkJ8H3gQ4LkmQMMI0hAQ4DrgN1mVijpIuBGM/tRmDwfBNLN\n7JuwrdfMrHO4rZuZ3Rwep0kZbWQRJLXoWVII59dcDHQ0M5PUzMy2S5oRHuPA8PDxZvZ1uPwg8KWZ\nTSmlXoTSk+cUYKmZzZJUD6htZvklYhlBkKBp0aJl13snTjuyL70caSc2LV7Oz89n9OjRDBo0iPPP\nP5/JkyeTkZHBBRdcwNtvv81rr73GhAkTiuuvWrWKiRMnMnnyZJo2bUpeXh5JSUmVGm9FSaRYIbHi\nTaRYIbHiTaRYoerizc7OXmlm3UrblpA9z9BGM1sLIOlDYH6YsNYSTOXVFJgp6QzAgLpR+75lZt/E\ncIzDtVGaHcAe4JnwmmhZ10U7h0mzGUEP+K8xxBJtCXCPpJOAOaX1Os1sKjAVoG37023C2sr9VecO\nzAKgoKCAvn37MnLkSG699VYArrzySl566SUkccEFF/Cb3/ymeL7ONWvW8Pjjj/PWW29x5plnAhwy\nn2d1lkixQmLFm0ixQmLFm0ixQvWMN5GT596o5f1R6/sJzuu/gbfNrH/Yu4xE1d8V4zEO1wYAkv4K\ntCLoFQ6X1B24EBgA3EwwRFvSDKCfmeWEvd2sMo5fyL+vSzc4UGhmf5T0HtAHeF3SDWa2oKyTaFi3\nNhtKDKtWBjNj2LBhpKSkFCdOCCazXrhwIVlZWSxYsIAzzjgDgH/+859cddVVPPvss8WJ0znnEkEi\nJ8/yNAU2h8uDY9xnJ9A4njbMrPeBZUlJwPfM7HVJi4B/lNFuY2CLpLoE12c3l1EvF+gKLCNIxgeO\n0x74h5lNltQWSAfKTJ5Hy6JFi3j22WdJS0sjMzMTgIceeohp06YxevRoCgsLadCgAVOnTgXggQce\n4Ouvv+amm24Cgrt1V6xYUWXxO+dcrI7l5PkIwZDrL4FY75Z5G7hL0mqCG4bibaMx8IqkBoCAA92v\n54FpkkYRJMGxwHvA1vCzcRn1HgP+FF6/jD7+T4BrJRUAXwAPxXh+lercc8+lrGvoK1euPKTs6aef\n5umnn67ssJxzrsIlZPI0s1ygc9T64DK2RY8F/jLcPoNg2PSQ+uF10LNKHK60NiKUMoRrZluA7qWU\nL+LgR1CeDH/KqwdBr7Lk8ccD40vu75xz7uio0c95Ouecc9+FJ0/nnHMuTp48nXPOuTh58nTOOefi\n5MnTOeeci5MnT+eccy5Onjydc865OHnydM455+LkydM555yLkydP55xzLk6ePF2F2bRpE9nZ2XTq\n1InU1FQmTZoEwOrVqzn77LPJzMykW7duLFu2DID169fTs2dP6tevz2OPPVaVoTvnXFwS8t22rnqq\nU6cOEyZMoEuXLuzcuZOuXbvSq1cv7rjjDu677z4uvfRSXn/9de644w4ikQjNmzdn8uTJvPzyy1Ud\nunPOxcWTZxWTJEBmtr+yjpFfUES7u2KdWOa7yR3fh9atW9O6dWsAGjduTEpKCps3b0YS3377LQA7\nduygTZs2ACQnJ5OcnMy8eZUbm3POVTRPnlUgnFj7rwTTkXUFlklKAxoCL5rZfWG9s4BJQCOCyb4v\nBHYTzKiSBdQHnjCz3x3dMyhfbm4uq1atokePHkycOJHevXszZswY9u/fz+LFi6s6POecOyIqa/5F\nV3nC5PkP4BwzWyqpuZl9I6k2MB8YBawPf642s+WSmhAkzqFAspk9KKk+sAj4sZltLHGMEcAIgBYt\nWna9d+K0Sj2ntBObFi/n5+czevRoBg0axPnnn8/kyZPJyMjgggsu4O233+a1115jwoQJxfVnzJhB\nw4YNufrqqwHIy8sjKSmpUuOtKIkUKyRWvIkUKyRWvIkUK1RdvNnZ2SvNrFtp2zx5VoEweb5tZqeG\n6yMJEl0doDXwc+BD4Ckz+0GJfV8kmONzd1jUFLjBzN4s63ht259utX4yqYLP4mC54/sAUFBQQN++\nfenduze33hrMBd60aVO2b9+OJMyMpk2bFg/jAowbN46kpCTGjBkDQCQSISsrq1LjrSiJFCskVryJ\nFCskVryJFCtUXbySykyePmxbdXYBSDoVGAOcZWb/kjQDaHCY/QT83Mz+GuuBGtatzYYwuVUmM2PY\nsGGkpKQUJ06ANm3asHDhQrKysliwYAFnnHFGpcfinHOVyZNn1WtCkEh3SGoFXApEgA1Aa0lnhcO2\njYF8gmulN0paYGYFks4ENpvZriqKv9iiRYt49tlnSUtLIzMzE4CHHnqIadOmMXr0aAoLC2nQoAFT\np04F4IsvvqBbt258++231KpVi4kTJ7Ju3bqqPAXnnIuJJ88qZmY5klYRXN/cRHANEzPbJ+lqYIqk\nhgSJ8yLgaaAd8H54p+5WoF9VxF7SueeeS1mXAVauXHlI2QknnMBnn31W2WE551yF8+RZBcwsF+gc\ntT64jHrLgbNL2fSL8Mc551wV8DcMOeecc3Hy5Omcc87FyZOnc845FydPns4551ycPHk655xzcfLk\n6ZxzzsXJk6dzzjkXJ0+ezjnnXJw8eTrnnHNx8uTpnHPOxcmTZw20Z88eunfvTkZGBqmpqdx3330A\nDBs2jIyMDNLT0xkwYAB5eXlVHKlzzlVPnjxroPr167NgwQJycnJYvXo1b7zxBkuXLuU3v/kNOTk5\nrFmzhrZt2/L4449XdajOOVctVWnylDRYUoX+hZbUT1KnqPUHJF1UkcdINPkFRbS7ax7t7poHgKTi\nWdkLCgooKChAEk2aNAGCeTnz8/MJJm1xzjlX0rHY8+wHFCdPM7vXzP5fFcZTLRUVFZGZmUlycjK9\nevWiR48eAAwZMoQTTjiB9evX8/Of/7yKo3TOuepJZc2/WCGNS4OAUUA94D3gJuA64G5gO5AD7DWz\nmyXNAF4zsxfDffPMLClcvhMYBOwH/mJmd0n6KTAibPsT4FogE3gN2BH+/AgYe6BdSRcCjxFMxbYc\nuNHM9krKBWYClwN1gR+b2fpSzqfUepLGAXlm9lhY7wOgb7jbG8BS4JzwmNOB+4FkYKCZLSvjuxsH\ntAXah58TzWxyuO1l4GSgATDJzKaWsv+I8PuhRYuWXe+dOA2AtBObHlQvLy+PsWPHMmrUKE499VQg\nSKyTJ0+mY8eOXHrppaWFV6ny8vKKe8bVXSLFCokVbyLFCokVbyLFClUXb3Z29koz61bqRjOrlB8g\nBfgzUDdc/y1wPfBPoCVB0lsEPB5unwEMiNo/L/y8FFgMfC9cbx5+Hh9V90Hg52W0MwMYQJBoNgFn\nhuV/AG4Jl3Oj9r8JeLqMcyq1HjAOGBNV7wOCCavbAYVAGkEvfyXwe0DAlcDLh/n+xoXnXR9oAXwd\n9V0e+A4ahsc6vqx2zIyTTz3NTrnzNTvlztesNPfff789+uijB5UtXLjQ+vTpU2r9yvb2229XyXG/\ni0SK1Syx4k2kWM0SK95EitWs6uIFVlgZf1crc9j2QqArsFzS6nD9v4CImW01s33ACzG0cxEw3cx2\nA5jZN2F5Z0nvSloLDARSy2mnA7DRzD4K12cC50dtnxN+riRIemWJtd4BG81srZntBz4E5oe/lLUx\n7D/PzPaa2TbgK6BVWD5KUg5Bj/Zk4IzDNdKwbm1yx/chd3wfALZu3cr27dsByM/P56233qJDhw58\n8sknQPAPqldffZWOHTvGcHrOOVfz1KnEtgXMNLO7iwukfsBVZdQvJLwGK6kWQc/0cGYA/cwsR9Jg\nIOsI490bfhYRfi+S/kqQsFaY2fCy6kXHHmpQSrsQDDvvjVou7/uP3rcIqCMpi+AfFD3NbLekSInj\nlWvLli1cf/31FBUVsX//fn7yk5/Qp08fzjvvPL799lvMjIyMDJ588sl4mnXOuRqjMpPnfOAVSb8x\ns68kNQdWAZMkHQ98C/yY4LonBEOiXYE/AVcQXFMEeAu4V9KsMFk0D3ufjYEtkuoS9Dw3h/V3httK\n2gC0k3S6mR24RrrwcCdgZr1jPNdcwmuckroAp8a433fRFPhX+F10BM6Ot4H09HRWrVp1SPmiRYsq\nIDznnDv2VdqwrZmtA34JvClpDUESbE1wLW8JwfXOv0ftMg24IByO7AnsCtt5A3gVWBEO/44J648l\nuAlpERB9c8/zwO2SVkk6LSqePcAQYHY41LsfeKqCTvcloLmkD4GbgY/KqX8k3iDogf4dGE8wdOuc\nc+4oqsyeJ2b2Aode11xKcMdpybpfcnAv6s6obeMJEkV0/SeBQ8YVzWwRUY+qAIOjts0Hvl/KPu2i\nlldQxhBwWfXMLB+4uLR9gM5R+0THkhu9rZRjjSuxHl336N8C65xzrtix+Jync845V6kqtefpyidp\nCDC6RPEiM/tZVcTjnHOufJ48q5iZTaeUYWznnHPVlw/bOuecc3Hy5Omcc87FyZOnc845FydPns45\n51ycPHk655xzcfLk6ZxzzsXJk6dzzjkXJ0+ezjnnXJw8eTrnnHNx8uTpnHPOxUlmVtUxuEomaSfB\nfKaJogWwraqDiFEixQqJFW8ixQqJFW8ixQpVF+8pZtaytA3+btuaYYOZdavqIGIlaUWixJtIsUJi\nxZtIsUJixZtIsUL1jNeHbZ1zzrk4efJ0zjnn4uTJs2aYWtUBxCmR4k2kWCGx4k2kWCGx4k2kWKEa\nxus3DDnnnHNx8p6nc845FydPns4551ycPHke4/T/27u3WDvKMozj/4eeQAvSDYQ0gNoaEm0IqTto\naqxNg1JKvWiJvWgkoVETgke4wFhCYsoFF5oIiQmRBK2FegAFxN4YqLTJVpIeFHbLRizd0CbalBbk\noNVyEF8v5l3syWLt1cyua2ayfX7JZM36ZtL1zJuZ/XV9M2tGWilpv6RxSRuazgMg6ZCkpySNSvpD\ntg1J2ibpQL7Oy3ZJ+n7m3ydpuIZ8myQdkzRWaqucT9L6XP+ApPU1Zt0o6XDWd1TSqtKymzPrfklX\nltpr2U8kXSRph6Q/SXpa0g3Z3rr69snayvpKOl3Sbkl7M++t2b5A0q787Pslzc72Ofl+PJd/8GTb\nUUPWzZIOlmq7ONsbPc56ighP03QCZgDPAQuB2cBeYFELch0Czu1q+y6wIec3AN/J+VXAbwABS4Bd\nNeRbBgwDY1PNBwwBz+frvJyfV1PWjcBNPdZdlPvAHGBB7hsz6txPgPnAcM6fCTybuVpX3z5ZW1nf\nrNHcnJ8F7Mqa/QJYl+13AV/O+a8Ad+X8OuD+fttRU9bNwNoe6zd6nPWa/M1zevs4MB4Rz0fEm8B9\nwOqGM01mNXBPzt8DrCm13xuFncDZkuYPMkhEjAAvn2K+K4FtEfFyRLwCbANW1pR1MquB+yLijYg4\nCIxT7CO17ScRcSQinsj5fwDPABfQwvr2yTqZRuubNTqeb2flFMDlwAPZ3l3bTs0fAD4tSX22o46s\nk2n0OOvFnef0dgHwl9L7v9L/4K9LAI9K+qOk67Lt/Ig4kvMvAOfnfFu2oWq+pnN/LYe3NnWGQPtk\naiRrDhN+lOJbR6vr25UVWlpfSTMkjQLHKDqS54BXI+LfPT77nVy5/DXgnLrydmeNiE5tb8va3iFp\nTnfWrkyNHWfuPK0JSyNiGLgK+KqkZeWFUYzHtPY3VG3PB/wA+BCwGDgCfK/ZOO8maS7wIHBjRPy9\nvKxt9e2RtbX1jYi3I2IxcCHFt8UPNxxpUt1ZJV0C3EyR+WMUQ7HfajBiX+48p7fDwEWl9xdmW6Mi\n4nC+HgN+RXGQH+0Mx+brsVy9LdtQNV9juSPiaP5h+g9wNxNDbq3IKmkWRWf004h4KJtbWd9eWdte\n38z4KrAD+ATFEGfnPublz34nVy5/H/C3uvOWsq7MofKIiDeAH9PC2na485ze9gAX59V2sykuCtja\nZCBJ75V0ZmceWAGMZa7OlXLrgV/n/Fbg2rzabgnwWml4r05V8z0CrJA0L4f1VmTbwHWdE76aor6d\nrOvyKssFwMXAbmrcT/Kc2o+AZyLi9tKi1tV3sqxtra+k8ySdnfNnAFdQnKfdAazN1bpr26n5WmB7\nfuufbDsGnfXPpf9AieLcbLm2rTrOBn5FkqdmJ4qr1J6lOPdxSwvyLKS4km8v8HQnE8W5lseAA8Bv\ngaFsF3Bn5n8KuKyGjD+nGI57i+Icypemkg/4IsXFFuPAF2rMuiWz7KP4ozO/tP4tmXU/cFXd+wmw\nlGJIdh8wmtOqNta3T9ZW1he4FHgyc40B3y4dc7uzTr8E5mT76fl+PJcvPNl21JB1e9Z2DPgJE1fk\nNnqc9Zp8ez4zM7OKPGxrZmZWkTtPMzOzitx5mpmZVeTO08zMrCJ3nmZmZhXNPPkqZma9SXqb4qcD\nHWsi4lBDccxq45+qG8DulQAAAa1JREFUmNmUSToeEXNr/LyZMXGfVrPGeNjWzAZG0nxJI/lsxjFJ\nn8r2lZKeyOc5PpZtQ5IezpuC75R0abZvlLRF0uPAlrw7zYOS9uT0yQY30f5PedjWzE7FGflkDICD\nEXF11/LPA49ExG2SZgDvkXQexT1hl0XEQUlDue6twJMRsUbS5cC9FDdfh+IZk0sj4oSknwF3RMTv\nJb2f4nZsHxngNpq9iztPMzsVJ6J4MsZk9gCb8gbrD0fEqKTlwEgUz4okIjrPI10KfC7btks6R9JZ\nuWxrRJzI+c8Ai4rbnwJwlqS5MfF8SLOBc+dpZgMTESP5yLnPApsl3Q68MoV/6p+l+dOAJRHx+v8i\no9lU+JynmQ2MpA8ARyPibuCHwDCwE1iWT+ygNGz7O+CabFsOvBRdz/pMjwJfL31Gv2++ZgPhb55m\nNkjLgW9Kegs4DlwbES9Kug54SNJpFM/uvALYSDHEuw/4FxOPy+r2DeDOXG8mMAJcP9CtMOvin6qY\nmZlV5GFbMzOzitx5mpmZVeTO08zMrCJ3nmZmZhW58zQzM6vInaeZmVlF7jzNzMwq+i+ZMDVXi0NB\nwwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "plot_importance(xgb_model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bring in `fastai`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dls = to.dataloaders()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "learn = tabular_learner(dls, layers=[200,100], metrics=accuracy)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_lossvalid_lossaccuracytime
00.3785140.3725430.82954500:04
10.3635590.3718110.82724200:04
20.3560180.3627370.83031300:04
30.3595890.3593650.83660900:04
40.3431870.3628380.83845200:04
" ], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "learn.fit(5, 1e-2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, our neural network has 83.84%, slighlty higher than the GBT" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we'll grab predictions" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "nn_preds = learn.get_preds()[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[0.9685, 0.0315],\n", " [0.6587, 0.3413],\n", " [0.5426, 0.4574],\n", " ...,\n", " [0.3229, 0.6771],\n", " [0.9517, 0.0483],\n", " [0.9978, 0.0022]])" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "nn_preds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's check to see if our feature importance changed at all" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class PermutationImportance():\n", " \"Calculate and plot the permutation importance\"\n", " def __init__(self, learn:Learner, df=None, bs=None):\n", " \"Initialize with a test dataframe, a learner, and a metric\"\n", " self.learn = learn\n", " self.df = df if df is not None else None\n", " bs = bs if bs is not None else learn.dls.bs\n", " self.dl = learn.dls.test_dl(self.df, bs=bs) if self.df is not None else learn.dls[1]\n", " self.x_names = learn.dls.x_names.filter(lambda x: '_na' not in x)\n", " self.na = learn.dls.x_names.filter(lambda x: '_na' in x)\n", " self.y = dls.y_names\n", " self.results = self.calc_feat_importance()\n", " self.plot_importance(self.ord_dic_to_df(self.results))\n", "\n", " def measure_col(self, name:str):\n", " \"Measures change after column shuffle\"\n", " col = [name]\n", " if f'{name}_na' in self.na: col.append(name)\n", " orig = self.dl.items[col].values\n", " perm = np.random.permutation(len(orig))\n", " self.dl.items[col] = self.dl.items[col].values[perm]\n", " metric = learn.validate(dl=self.dl)[1]\n", " self.dl.items[col] = orig\n", " return metric\n", "\n", " def calc_feat_importance(self):\n", " \"Calculates permutation importance by shuffling a column on a percentage scale\"\n", " print('Getting base error')\n", " base_error = self.learn.validate(dl=self.dl)[1]\n", " self.importance = {}\n", " pbar = progress_bar(self.x_names)\n", " print('Calculating Permutation Importance')\n", " for col in pbar:\n", " self.importance[col] = self.measure_col(col)\n", " for key, value in self.importance.items():\n", " self.importance[key] = (base_error-value)/base_error #this can be adjusted\n", " return OrderedDict(sorted(self.importance.items(), key=lambda kv: kv[1], reverse=True))\n", "\n", " def ord_dic_to_df(self, dict:OrderedDict):\n", " return pd.DataFrame([[k, v] for k, v in dict.items()], columns=['feature', 'importance'])\n", "\n", " def plot_importance(self, df:pd.DataFrame, limit=20, asc=False, **kwargs):\n", " \"Plot importance with an optional limit to how many variables shown\"\n", " df_copy = df.copy()\n", " df_copy['feature'] = df_copy['feature'].str.slice(0,25)\n", " df_copy = df_copy.sort_values(by='importance', ascending=asc)[:limit].sort_values(by='importance', ascending=not(asc))\n", " ax = df_copy.plot.barh(x='feature', y='importance', sort_columns=True, **kwargs)\n", " for p in ax.patches:\n", " ax.annotate(f'{p.get_width():.4f}', ((p.get_width() * 1.005), p.get_y() * 1.005))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Getting base error\n" ] }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Calculating Permutation Importance\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " 100.00% [9/9 00:05<00:00]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdIAAAD4CAYAAABYIGfSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deXxV5bX/8c8XwiAiCAgWAxriwaBh\niHACWvVWQEQvNuptKqhFEK1VcYD+VLjXltbbwWjt9aqg1JaW2KuEgpVYkTigVsUhBAlTioYhlkRq\nIyooSiBh/f44m/QEkhA4GWG9X6/9yt7PXvvZ6zkZVvZw9pGZ4ZxzzrnD06qpE3DOOedaMi+kzjnn\nXAy8kDrnnHMx8ELqnHPOxcALqXPOOReDuKZOwNXuhBNOsISEhKZOwznnWpQVK1Z8YmbdG2NfXkib\nuYSEBPLy8po6Deeca1EkfdhY+/JTu84551wMvJA655xzMfBC6pxzzsXAC6lzzjkXA7/Z6AiXMH1x\nU6fgnHOHrChjTFOnUGd+ROqcc87FwAupc845F4MWXUglnSRpYTCfIunf67DN+ZKeO4R9XCbpjPqK\nc845d2RpsYVUUpyZfWRm6UFTCnDQQnoYLgPqUiDrGuecc+4Q5eTkkJSURCgUIiMj44D1ZWVljB07\nllAoxLBhwwDaAkhKkPS1pPxgmr1vG0lXSlojabWkHEknRK27VdJ6Sesk3V9bbo1eSINBrZc0V9IH\nkp6UdIGkZZIKJQ0NprclrZT0lqSkYNuJkp6V9AqwNOhrraS2wH8DY4MXamxNfRwktwxJBcGL+oCk\nbwJpwK+Cfk+V9H1JyyWtkvS0pA41xL0mKRz0e4KkomA+WVJuELdaUt+GeaWdc+7IUFFRweTJk1my\nZAkFBQXMmzePgoKCKjFz5syhS5cubNiwgalTpwL0ilq90cxSgulGiByMAQ8Bw81sILAauCVYNxy4\nFBhkZsnAA7Xl11RHpCHg10C/YLoKOBe4A/gvYD1wnpmdCcwAfhm17WAg3cy+ta/BzHYHcfODF2r+\nQfo4gKRuwOVAcvCi/tzM3gKeBe4M+t0I/NnMUs1sEPA34Loa4mpyI/CQmaUAYaC4mlxukJQnKa+0\ntLS2tJ1z7oiXm5tLKBQiMTGRtm3bMm7cOLKzs6vEZGdnM2HCBADS09MBjpOkWrpVMB0bxHUCPgrW\n3QRkmFkZgJn9s7b8mqqQbjazNWa2F1gHLDUzA9YACUBnYIGktcCDQHLUti+Z2ad12EdtfVRnO7AL\nmCPpP4CvaojrL+kNSWuAq+vQ7/7eBv5L0jTgFDP7ev8AM3vczMJmFu7evVGeueycc81WSUkJvXv3\nrlzu1asXJSUlNcbExcUBVADdgtV9grOTf5V0HoCZ7SFSMNcQKaBnAHOC+NOA8yS9G2yTWlt+TVVI\ny6Lm90Yt7yXy3tafAa+aWX/g20D7qPidddxHbX0AIOmF4BTr78ysHBgKLAQuAXJq6HcucIuZDQDu\nqa7fQDn/en0rY8zsKSKngb8Gnpc0oo7jcc45d+i2AicHZyd/CDwlqZOkNkQK6ZnASURO7f5nsE0c\n0BU4C7gT+FNtR7fN9YEMnYF9/25MrOM2XwDHHUofZjZ637ykjkAHM3te0jJgUw39HgdsDb4JV0ft\nY/+4ImAIkAvsuyEKSYnAJjN7WNLJwEDglTqO0Tnnjjrx8fFs2bKlcrm4uJj4+PhqY3r16kV5eTlA\na2BbcLZz3ynaFZI2EjniVNC2EUDSn4Dp+3ZB5DKeAbmS9gInANVea2uud+3eD9wraSV1L/avAmfs\nu9noMPo4DnhO0mrgTSL/uQBkAXcGpwVOBX4MvAssI3IdlhriHgBuCvZ/QlTcFcBaSflAf+CJOo7P\nOeeOSqmpqRQWFrJ582Z2795NVlYWaWlpVWLS0tLIzMwEYOHChQBfmJlJ6i6pNVQeyPQlcqBUQqRm\n7Lt+NorIfS8Ai4DhwTanEbkD+JOa8lOk4LrmKhwOWyyfR+qPCHTOtUT7PyLw+eefZ8qUKVRUVDBp\n0iTuvvtuZsyYQTgcJi0tjV27djF+/HhWrlxJ165dWb58+RozGyjpO0Te1bGHyOXDn5jZXwAk3Qjc\nHqz7EJhoZtuCd4L8nsjbKncDd5hZjWcOvZA2c7EWUuecOxpJWmFm4cbYV3M9teucc861CF5InXPO\nuRh4IXXOOedi4IXUOeeci4EXUueccy4GXkidc865GHghdc4552LghdQ555yLgRdS55xzLgZeSJ1z\nzrkYeCF1zjnnYuCF1DnnnItBc/08UldP/NNf3JFm/08Fca6p+RGpc845FwMvpIdBUoqkf49aTpM0\nvbZtnHPOHZm8kB6eFKCykJrZs2aW0YT5OOecayItqpBK+qGktcE0JWi7RtJqSask/TFoO1HSM0Hb\nKknflJQgaW1UX3dI+mkw/5qkhyTlB30PDdqHSnpb0kpJb0lKCj45/b+BsUH8WEkTJc0MtkmQ9EqQ\n01JJJwftcyU9HPSzSVJ6o754zh2BcnJySEpKIhQKkZFx4P+yZWVljB07llAoxLBhwygqKgIgNzeX\nlJQUUlJSGDRoEM8880zlNgkJCQwYMICUlBTC4X99LvSnn37KqFGj6Nu3L6NGjeKzzz5r8PG5lqHF\nFFJJQ4BrgWHAWcD3JZ0D/AgYYWaDgNuD8IeBvwZtg4F1ddhFBzNLAW4Gfh+0rQfOM7MzgRnAL81s\ndzA/38xSzGz+fv08AmSa2UDgySCXfXoC5wKXAH4E61wMKioqmDx5MkuWLKGgoIB58+ZRUFBQJWbO\nnDl06dKFDRs2MHXqVKZNmwZA//79ycvLIz8/n5ycHH7wgx9QXl5eud2rr75Kfn4+eXl5lW0ZGRmM\nHDmSwsJCRo4cWW3hdkenFlNIiRSgZ8xsp5l9CfwZCAMLzOwTADP7NIgdATwWtFWY2fY69D8viH8d\n6CTpeKAzsCA4kn0QSK5DP2cDTwXzfwzy3meRme01swLgxJo6kHSDpDxJeaWlpXXYpXNHn9zcXEKh\nEImJibRt25Zx48aRnZ1dJSY7O5sJEyYAkJ6eztKlSzEzOnToQFxc5E0Lu3btQtJB9xfd14QJE1i0\naFE9j8i1VC2pkMaqnKrjbb/feqtm+WfAq2bWH/h2NdscqrKo+Rp/c83scTMLm1m4e/fuMe7SuSNT\nSUkJvXv3rlzu1asXJSUlNcbExcXRuXNntm3bBsC7775LcnIyAwYMYPbs2ZWFVRIXXnghQ4YM4fHH\nH6/s6+OPP6Znz54AfOMb3+Djjz9u0PG5lqMlFdI3gMskdZB0LHA5kAd8V1I3AEldg9ilwE1BW2tJ\nnYGPgR6SuklqR+T0arSxQfy5wPbgKLYzsO83c2JU7BfAcTXk+RYwLpi/OsjbOdfMDBs2jHXr1rF8\n+XLuvfdedu3aBcCbb77Je++9x5IlS5g1axavv/76AdtKqtNRrDs6tJhCambvAXOBXOBd4Hdmtgz4\nBfBXSauA/wnCbweGS1oDrADOMLM9RG4SygVeInL9M9ouSSuB2cB1Qdv9wL1Be/TDK14Fzth3s9F+\n/dwKXCtpNTCef123dc7Vo/j4eLZs2VK5XFxcTHx8fI0x5eXlbN++nW7dulWJOf300+nYsSNr166t\n3AagR48eXH755eTm5gJw4oknsnXrVgC2bt1Kjx49GmZgrsVpUU82MrP/4V/Fcl9bJpC5X9vHwKXV\nbP8wVW/+ifZ/ZjZlv/i3gdOimn4UtH8KpO63/dxg3YdErtHuv++J+y13rCEP51wdpKamUlhYyObN\nm4mPjycrK4unnnqqSkxaWhqZmZmcffbZLFy4kBEjRiCJzZs307t3b+Li4vjwww9Zv349CQkJ7Ny5\nk71793Lcccexc+dOXnzxRWbMmFGlr+nTp5OZmcmllx7wJ8YdpVpUIXXOuX3i4uKYOXMmo0ePpqKi\ngkmTJpGcnMyMGTMIh8OkpaVx3XXXMX78eEKhEF27diUrKwuInL7NyMigTZs2tGrVikcffZQTTjiB\nTZs2cfnllwORI9irrrqKiy66CIDp06dzxRVXMGfOHE455RT+9Kc/NdnYXfMis/3vsXHNSTgctuhb\n8J1zzh2cpBVmFj54ZOxazDVS55xzrjnyQuqcc87FwAupc845FwMvpM4551wMvJA655xzMfBC6pxz\nzsXAC6lzzjkXAy+kzjnnXAy8kDrnnHMx8ELqnHPOxcALqXPOORcDf2j9ES5h+uKmTsG5mBVljGnq\nFJyrkR+ROuecczHwQuqcc87FwAupc845FwMvpM65FiUnJ4ekpCRCoRAZGRkHrC8rK2Ps2LGEQiGG\nDRtGUVERALm5uaSkpJCSksKgQYN45plnqmxXUVHBmWeeySWXXHJAn7fddhsdO3ZskPG4ls8LaYwk\nLZK0QtI6STcEbddJ+kBSrqTfSpoZtHeX9LSk5cF0TtNm71zLUlFRweTJk1myZAkFBQXMmzePgoKC\nKjFz5syhS5cubNiwgalTpzJt2jQA+vfvT15eHvn5+eTk5PCDH/yA8vLyyu0eeughTj/99AP2mZeX\nx2effdawA3MtmhfS2E0ysyFAGLhNUjzwY+As4BygX1TsQ8CDZpYKfAf4XXUdSrpBUp6kvNLS0obN\n3rkWJDc3l1AoRGJiIm3btmXcuHFkZ2dXicnOzmbChAkApKens3TpUsyMDh06EBcXeaPCrl27kFS5\nTXFxMYsXL+b666+v0ldFRQV33nkn999/fwOPzLVkXkhjd5ukVcA7QG9gPPBXM/vUzPYAC6JiLwBm\nSsoHngU6STrgfJGZPW5mYTMLd+/evRGG4FzLUFJSQu/evSuXe/XqRUlJSY0xcXFxdO7cmW3btgHw\n7rvvkpyczIABA5g9e3ZlYZ0yZQr3338/rVpV/ZM4c+ZM0tLS6NmzZ0MOy7VwXkhjIOl8IsXxbDMb\nBKwE1teySSvgLDNLCaZ4M/uyEVJ1zgHDhg1j3bp1LF++nHvvvZddu3bx3HPP0aNHD4YMGVIl9qOP\nPmLBggXceuutTZStaym8kMamM/CZmX0lqR+R07nHAt+S1EVSHJFTuPu8CFT+VkpKadRsnWvh4uPj\n2bJlS+VycXEx8fHxNcaUl5ezfft2unXrViXm9NNPp2PHjqxdu5Zly5bx7LPPkpCQwLhx43jllVf4\n3ve+x8qVK9mwYQOhUIiEhAS++uorQqFQww/StTheSGOTA8RJ+huQQeT0bgnwSyAXWAYUAduD+NuA\nsKTVkgqAGxs9Y+dasNTUVAoLC9m8eTO7d+8mKyuLtLS0KjFpaWlkZmYCsHDhQkaMGIEkNm/eXHlz\n0Ycffsj69etJSEjg3nvvpbi4mKKiIrKyshgxYgT/93//x5gxY/jHP/5BUVERRUVFdOjQgQ0bNjT6\nmF3z548IjIGZlQEX798uKc/MHg+OSJ8BFgXxnwBjGzdL544ccXFxzJw5k9GjR1NRUcGkSZNITk5m\nxowZhMNh0tLSuO666xg/fjyhUIiuXbuSlZUFwJtvvklGRgZt2rShVatWPProo5xwwglNPCJ3JJCZ\nNXUORxxJDxC5dtqeyOnc2+0wX+hwOGx5eXn1mZ5zzh3xJK0ws3Bj7MuPSBuAmd3R1Dk455xrHH6N\n1DnnnIuBF1LnnHMuBl5InXPOuRh4IXXOOedi4IXUOeeci4EXUueccy4GXkidc865GHghdc4552Lg\nhdQ555yLgRdS55xzLgb+iMAjXML0xU2dgjuCFWWMaeoUnGtyfkTqnHPOxcALqXPOOReDJi2kkiZK\nmlnPfV4m6Yyo5f+WdEF97sM555zb50g8Ir0MqCykZjbDzF5uwnycO+Ll5OSQlJREKBQiIyPjgPVl\nZWWMHTuWUCjEsGHDKCoqAuCll15iyJAhDBgwgCFDhvDKK69UbjN//nwGDhxIcnIy06ZNq2yfPXs2\nAwYMICUlhXPPPZeCgoIGH59ztWnQQirpe5JyJeVL+o2k1pKulfSBpFzgnKjYuZLSo5a/jJqfJmmN\npFWSMoK270taHrQ9LamDpG8CacCvgn2eGt2vpJGSVgZ9/V5Su6C9SNI9kt4L1vWrYTzVxkn6qaQ7\nouLWSkoIpvVBDh9IelLSBZKWSSqUNLReX3DnmkBFRQWTJ09myZIlFBQUMG/evAOK25w5c+jSpQsb\nNmxg6tSplYXxhBNO4C9/+Qtr1qwhMzOT8ePHA7Bt2zbuvPNOli5dyrp16/jHP/7B0qVLAbjqqqtY\ns2YN+fn53HXXXfzwhz9s3AE7t58GK6SSTgfGAueYWQpQAXwPuIdIAT2XqCPHWvq5GLgUGGZmg4D7\ng1V/NrPUoO1vwHVm9hbwLHCnmaWY2caoftoDc4GxZjaAyB3LN0Xt6hMzGww8BtT2wdx1jdsnBPwa\n6BdMVxEZ+x3Af9Uw5hsk5UnKKy0trcMunGs6ubm5hEIhEhMTadu2LePGjSM7O7tKTHZ2NhMmTAAg\nPT2dpUuXYmaceeaZnHTSSQAkJyfz9ddfU1ZWxqZNm+jbty/du3cH4IILLuDpp58GoFOnTpX97ty5\nE0mNMUznatSQR6QjgSHAckn5wfJU4DUzKzWz3cD8OvRzAfAHM/sKwMw+Ddr7S3pD0hrgaiD5IP0k\nAZvN7INgORP4t6j1fw6+rgASaumnrnH7bDazNWa2F1gHLDUzA9bUtL2ZPW5mYTML7/tD4lxzVVJS\nQu/evSuXe/XqRUlJSY0xcXFxdO7cmW3btlWJefrppxk8eDDt2rUjFArx/vvvU1RURHl5OYsWLWLL\nli2VsbNmzeLUU0/lrrvu4uGHH27A0Tl3cA1ZSAVkBkeGKWaWBPy0lvjyfflIagW0PUj/c4FbgqPL\ne4D2MeZbFnytIHh/raQXglPEv6stLjr3QPtq4gH2Ri3vxd/H6xwA69atY9q0afzmN78BoEuXLjz2\n2GOMHTuW8847j4SEBFq3bl0ZP3nyZDZu3Mh9993Hz3/+86ZK2zmgYQvpUiBdUg8ASV2BlcC3JHWT\n1Ab4blR8EZEjWIhc52wTzL8EXCupQ1Q/AMcBW4N+ro7q54tg3f7eBxIkhYLl8cBfaxuAmY0O/gm4\n/iBjLQIGB/kNBvocJN65I0Z8fHyVo8Xi4mLi4+NrjCkvL2f79u1069atMv7yyy/niSee4NRTT63c\n5tvf/jbvvvsub7/9NklJSZx22mkH7HvcuHEsWrSoIYblXJ01WCE1swLgR8CLklYTKYg9iRyVvg0s\nI3Jtc5/fEimyq4CzgZ1BPzlErnvmBaeI912X/DHwbtDP+qh+soA7g5uKKn8rzWwXcC2wIDgdvBeY\nXU/DfRroKmkdcAvwwUHinTtipKamUlhYyObNm9m9ezdZWVmkpaVViUlLSyMzMxOAhQsXMmLECCTx\n+eefM2bMGDIyMjjnnHOqbPPPf/4TgM8++4xHH32U66+P/D9bWFhYGbN48WL69u3bkMNz7qAUuVzn\nmqtwOGx5eXmHvb0/ItA1pH2PCHz++eeZMmUKFRUVTJo0ibvvvpsZM2YQDodJS0tj165djB8/npUr\nV9K1a1eysrJITEzk5z//Offee2+VYvjiiy/So0cPrrzySlatWgXAjBkzGDduHAC33347L7/8Mm3a\ntKFLly7MnDmT5OSD3SLhjjaSVphZuFH25YW0eYu1kDrn3NGoMQvpkfhABuecc67ReCF1zjnnYnDQ\nQqqI70maESyf7E/kcc455yLqckT6KJG7aK8Mlr8AZjVYRs4551wLUpcHAgwzs8GSVgKY2WeSDvaw\nBOecc+6oUJcj0j2SWgMGIKk7kfdgOuecc0e9uhTSh4FngB6SfgG8CfyyQbNyzjnnWohaT+0Gz7zd\nDNxF5KHzAi4zs7/Vtp1zzjl3tKi1kJrZXkmzzOxMqj6GzznnnHPU7dTuUknfkX/on3POOXeAuhTS\nHwALgDJJOyR9IWlHA+flnHPOtQgHffuLmVX3kWSuhfCH1v/LvgesO+dcfTpoIZX0b9W1m9nr9Z+O\nc84517LU5YEMd0bNtweGAiuAEQ2SkXPOOdeC1OXU7rejlyX1Bv63wTJqIpKKgLCZfVKH2J8CX5rZ\nAw2dl3POuebtcD79pRg4vb4TaUrBk5ucc865Q1aXT395RNLDwTQTeAN4r+FTqxtJd0q6LZh/UNIr\nwfwISU9KulLSGklrJd0Xtd2Xkn4taRWRh/Lvaz9G0hJJ3w+Wr5G0WtIqSX+sZv/fl7Q8WP+0pA5B\n+3eDfa6S9HrQliwpV1J+0GffBn1xXLVycnJISkoiFAqRkZFxwPqysjLGjh1LKBRi2LBhFBUVAbBt\n2zaGDx9Ox44dueWWW6psc9FFFzFo0CCSk5O58cYbqaioAODTTz9l1KhR9O3bl1GjRvHZZ581+Pic\nc42rLkekeUSuia4A3gammdn3GjSrQ/MGcF4wHwY6SmoTtH0A3Efkem4KkCrpsiD2WOBdMxtkZm8G\nbR2BvwDzzOy3kpKBHwEjzGwQcHs1+/+zmaUG6/8GXBe0zwBGB+1pQduNwENmlhLkWlzdgCTdIClP\nUl5paekhvyCuZhUVFUyePJklS5ZQUFDAvHnzKCgoqBIzZ84cunTpwoYNG5g6dSrTpk0DoH379vzs\nZz/jgQcOPKP/pz/9iVWrVrF27VpKS0tZsGABABkZGYwcOZLCwkJGjhxZbeF2zrVsdSmkx5tZZjA9\naWbLJFVXUJrKCmCIpE5AGZFiHyZSSD8HXjOzUjMrB54E9t2FXAE8vV9f2cAfzOyJYHkEsGDfdVMz\n+7Sa/feX9IakNcDVQHLQvgyYGxzZ7jt1/DbwX5KmAaeY2dfVDcjMHjezsJmFu3fvXvdXwh1Ubm4u\noVCIxMRE2rZty7hx48jOzq4Sk52dzYQJEwBIT09n6dKlmBnHHnss5557Lu3btz+g306dOgFQXl7O\n7t272ff8kui+JkyYwKJFixpyeM65JlCXQjqhmraJ9ZzHYTOzPUSeBzwReIvIEepwIAQU1bLpLjOr\n2K9tGXDRIT7FaS5wi5kNAO4hcmczZnYjkaPZ3sAKSd3M7CkiR6dfA89L8jufG1lJSQm9e/euXO7V\nqxclJSU1xsTFxdG5c2e2bdt20L5Hjx5Njx49OO6440hPTwfg448/pmfPngB84xvf4OOPP66voTjn\nmokaC2lwbfEvQB9Jz0ZNrwLVHZk1pTeAO4DXg/kbgZVALvAtSScENxRdCfy1ln5mAJ/xrw8ufwX4\nrqRuAJK6VrPNccDW4HTy1fsaJZ1qZu+a2QygFOgtKRHYZGYPEzn6HXi4A3bNzwsvvMDWrVspKyvj\nlVdeOWC9JA7tfzTnXEtQ2xHpW8CviTys/tdR0/8DRjd8aofkDaAn8LaZfQzsAt4ws63AdOBVYBWw\nwsyya+4GiFwHPUbS/Wa2DvgF8NfgpqT/qSb+x8C7RI5mox/s/6t9NzkReS1XAVcAayXlA/2BJ/bv\nzDWs+Ph4tmzZUrlcXFxMfHx8jTHl5eVs376dbt261an/9u3bc+mll1aeLj7xxBPZunUrAFu3bqVH\njx71MQznXDNS4/tIzexD4EOi7mhtrsxsKdAmavm0qPl5wLxqtum433JC1OK1Ue2ZQOZ+sT+Nmn8M\neKya/v+jmlQzgsk1kdTUVAoLC9m8eTPx8fFkZWXx1FNPVYlJS0sjMzOTs88+m4ULFzJixIhajyS/\n/PJLvvjiC3r27El5eTmLFy/mvPPOq9LX9OnTyczM5NJLL23Q8TnnGp/MrPYA6SzgESLvHW1L5MaZ\nnWbWqeHTc+Fw2PLy8po6jSPK888/z5QpU6ioqGDSpEncfffdzJgxg3A4TFpaGrt27WL8+PGsXLmS\nrl27kpWVRWJiIgAJCQns2LGD3bt3c/zxx/Piiy/SrVs3LrnkEsrKyti7dy/Dhw/nwQcfJC4ujm3b\ntnHFFVfw97//nVNOOYU//elPdO1a3RUC51x9krTCzMKNsq86FNI8YByRT4AJA9cAp5nZfzZ8es4L\nqXPOHbrGLKR1erKRmW0AWptZhZn9AbioYdNyzjnnWoa6PLT+K0ltgXxJ9wNbObxHCzrnnHNHnLoU\nxPFB3C3ATiLvi/xOQyblnHPOtRR1+fSXDyUdA/Q0s3saISfnnHOuxajLQ+u/DeQDOcFyiqRnGzox\n55xzriWoy6ndnxL5MO/PAcwsH+jTgDk555xzLUZdCukeM9u+X1vt75lxzjnnjhJ1uWt3naSrgNbB\n52feRuSRd84559xRr7aH1u/7EOuNRD4arIzIo/Z2AFMaPjXnnHOu+avtiHSIpJOAsUQ+luzXUes6\nEHkwvHPOOXdUq62QzgaWAolA9DPqROQaaWID5uXqScL0xU2dQpMqyhjT1Ck4545wNZ7aNbOHzex0\n4Pdmlhg19TEzL6LOOeccdbhr18xuaoxEnHPOuZboqHhmrqSJkmbWc5+XSTojavm/JV1Qn/twzjnX\n/B0VhbSBXAZUFlIzm2FmLzdhPs4555rAEVFIJX1PUq6kfEm/kdRa0rWSPpCUC5wTFTtXUnrU8pdR\n89MkrZG0SlJG0PZ9ScuDtqcldZD0TSAN+FWwz1Oj+5U0UtLKoK/fS2oXtBdJukfSe8G6fo30Eh3V\ncnJySEpKIhQKkZGRccD6srIyxo4dSygUYtiwYRQVFQGwbds2hg8fTseOHbnlllsq47/66ivGjBlD\nv379SE5OZvr06ZXrZs+ezYABA0hJSeHcc8+loKCgwcfnnGtaLb6QSjqdyFt0zjGzFKAC+B5wD5EC\nei5RR4619HMxcCkwzMwGAfcHq/5sZqlB29+A68zsLeBZ4E4zSzGzjVH9tAfmAmPNbACRO6OjrzN/\nYmaDgceAO2rI5QZJeZLySktL6/pSuGpUVFQwefJklixZQkFBAfPmzTuguM2ZM4cuXbqwYcMGpk6d\nyrRp0wBo3749P/vZz3jggQcO6PeOO+5g/fr1rFy5kmXLlrFkyRIArrrqKtasWUN+fj533XUXP/zh\nDxt+kM65JtXiCykwEhgCLJeUHyxPBV4zs1Iz2w3Mr0M/FwB/MLOvAMzs06C9v6Q3JK0BribycIra\nJAGbzeyDYDkT+Leo9X8Ovq4AEqrrwMweN7OwmYW7d+9eh9RdTXJzcwmFQiQmJtK2bVvGjRtHdnZ2\nlZjs7GwmTJgAQHp6OkuXLtgpwqwAABnZSURBVMXMOPbYYzn33HNp3759lfgOHTowfPhwANq2bcvg\nwYMpLi4GoFOnTpVxO3fuRFJDDs851wwcCYVUQGZwZJhiZklEHrRfk3KCcUtqBbQ9SP9zgVuCo8t7\ngPa1hx9UWfC1gro9otHFoKSkhN69e1cu9+rVi5KSkhpj4uLi6Ny5M9u2batT/59//jl/+ctfGDly\nZGXbrFmzOPXUU7nrrrt4+OGH62EUzrnm7EgopEuBdEk9ACR1BVYC35LUTVIb4LtR8UVEjmAhcp2z\nTTD/EnCtpA5R/QAcB2wN+rk6qp8vgnX7ex9IkBQKlscDfz384bnmqry8nCuvvJLbbruNxMR/vbV6\n8uTJbNy4kfvuu4+f//znTZihc64xtPhCamYFwI+AFyWtJlIQexI5Kn0bWEbk2uY+vyVSZFcBZwM7\ng35yiFz3zAtOEe+7fvlj4N2gn/VR/WQBdwY3FZ0alc8u4FpgQXA6eC+Rp0S5JhAfH8+WLVsql4uL\ni4mPj68xpry8nO3bt9OtW7eD9n3DDTfQt29fpkyp/tHT48aNY9GiRTFk75xrCY6IU4tmNp8Dr4O+\nA/yhmtiPgbOimqZFrcsAMvaLf4zIjUH797OMqjcxTYxatxQ4s5ptEqLm84DzDxyNq0+pqakUFhay\nefNm4uPjycrK4qmnnqoSk5aWRmZmJmeffTYLFy5kxIgRB722+aMf/Yjt27fzu9/9rkp7YWEhffv2\nBWDx4sWV8865I5fM/KNFm7NwOGx5eXkHD3Q1ev7555kyZQoVFRVMmjSJu+++mxkzZhAOh0lLS2PX\nrl2MHz+elStX0rVrV7KysipP1SYkJLBjxw52797N8ccfz4svvkinTp3o3bs3/fr1o127dgDccsst\nXH/99dx+++28/PLLtGnThi5dujBz5kySkw92f5pzrr5JWmFm4UbZlxfS5s0LqXPOHbrGLKQt/hqp\nc84515S8kDrnnHMx8ELqnHPOxcALqXPOORcDL6TOOedcDLyQOuecczHwQuqcc87FwAupc845FwMv\npM4551wMvJA655xzMTgiHlrvapYwfXG99FOUMaZe+nHOuSONH5E655xzMfBC6pxzzsXgiCikkr48\nyPrjJd0ctXySpIX1nMNrkg74pAFJYUkP1+e+nHPONR8tppAq4nDzPR6oLKRm9pGZpddPZrUzszwz\nu60x9uWcc67xNetCKilB0vuSngDWAj+WtFzSakn3VBPfUdJSSe9JWiPp0mBVBnCqpHxJvwr6XRts\n017SH4L4lZKGB+0TJf1ZUo6kQkn3B+2tJc2VtDbYZmpUCt+VlCvpA0nnBfHnS3oumP+ppD9Kejvo\n8/sN9uI1kJycHJKSkgiFQmRkZBywvqysjLFjxxIKhRg2bBhFRUWV6+69915CoRBJSUm88MILAOza\ntYuhQ4cyaNAgkpOT+clPflIZP3HiRPr06UNKSgopKSnk5+c3+Picc+5QtYS7dvsCE4BOQDowFBDw\nrKR/M7PXo2J3AZeb2Q5JJwDvSHoWmA70N7MUiBToqG0mA2ZmAyT1A16UdFqwLgU4EygD3pf0CNAD\niDez/kFfx0f1FWdmQyX9O/AT4IJqxjMQOAs4FlgpabGZfRQdIOkG4AaAk08+ua6vU4OrqKhg8uTJ\nvPTSS/Tq1YvU1FTS0tI444wzKmPmzJlDly5d2LBhA1lZWUybNo358+dTUFBAVlYW69at46OPPuKC\nCy7ggw8+oF27drzyyit07NiRPXv2cO6553LxxRdz1llnAfCrX/2K9PRGOXngnHOHpVkfkQY+NLN3\ngAuDaSXwHtCPSJGNJuCXklYDLwPxwIkH6f9c4P8AzGw98CGwr5AuNbPtZrYLKABOATYBiZIekXQR\nsCOqrz8HX1cACTXsL9vMvjazT4BXifxjUIWZPW5mYTMLd+/e/SDpN57c3FxCoRCJiYm0bduWcePG\nkZ2dXSUmOzubCRMmAJCens7SpUsxM7Kzsxk3bhzt2rWjT58+hEIhcnNzkUTHjh0B2LNnD3v27EFS\no4/NOecOV0sopDuDrwLuNbOUYAqZ2Zz9Yq8GugNDgqPPj4H2Mey7LGq+gsgR52fAIOA14Ebgd9XE\nV1Dz0b4dZLnZKikpoXfv3pXLvXr1oqSkpMaYuLg4OnfuzLZt22rdtqKigpSUFHr06MGoUaMYNmxY\nZdzdd9/NwIEDmTp1KmVlZTjnXHPTEgrpPi8AkyR1BJAUL6nHfjGdgX+a2Z7gWucpQfsXwHE19PsG\nkQJMcEr3ZOD9mpIIThm3MrOngR8Bgw9xHJcG12W7AecDyw9x+yNO69atyc/Pp7i4mNzcXNauXQtE\nrqmuX7+e5cuX8+mnn3Lfffc1cabOOXegFlNIzexF4CngbUlrgIUcWByfBMLB+muA9cG224BlwQ1C\nv9pvm0eBVsE284GJZlbboU888JqkfCKnhP/zEIeymsgp3XeAn+1/fbQ5i4+PZ8uWLZXLxcXFxMfH\n1xhTXl7O9u3b6datW522Pf744xk+fDg5OTkA9OzZE0m0a9eOa6+9ltzc3IYamnPOHbZmfbORmRUB\n/aOWHwIeqiauY/D1E+DsGvq6ar+m/kH7LuDaauLnAnOjli+JWn3AUaiZnR81/wnBNVIze43IaeB9\nVpvZNdXl2NylpqZSWFjI5s2biY+PJysri6eeeqpKTFpaGpmZmZx99tksXLiQESNGIIm0tDSuuuoq\nfvjDH/LRRx9RWFjI0KFDKS0tpU2bNhx//PF8/fXXvPTSS0ybNg2ArVu30rNnT8yMRYsW0b9//+rS\ncs65JtWsC6mLXX0+IzcuLo6ZM2cyevRoKioqmDRpEsnJycyYMYNwOExaWhrXXXcd48ePJxQK0bVr\nV7KysgBITk7miiuu4IwzziAuLo5Zs2bRunVrtm7dyoQJE6ioqGDv3r1cccUVXHJJ5H+Wq6++mtLS\nUsyMlJQUZs+eXW9jcc65+iKzFnOvy1EpHA5bXl5eU6fhnHMtiqQVZnbA0+YaQou5Ruqcc841R15I\nnXPOuRh4IXXOOedi4IXUOeeci4EXUueccy4GXkidc865GHghdc4552LghdQ555yLgRdS55xzLgZe\nSJ1zzrkYeCF1zjnnYuCF9AiXMH0xCdMXN3Uazjl3xPJC6pxzzsXAC2kUSbdJ+pukJ2uJ+bIe9jNR\n0kmx9uOcc67p+eeRVnUzcIGZFTfwfiYCa4GPGng/zjnnGpgfkQYkzQYSgSWStkv6vaTXJG2SdFs1\n8bMkpQXzz0j6fTA/SdIvgvkfS3pf0puS5km6Q1I6EAaelJQv6ZjGGyXk5OSQlJREKBQiIyPjgPVl\nZWWMHTuWUCjEsGHDKCoqqlx37733EgqFSEpK4oUXXqhzn7fddhsdO3ZskPE451xT80IaMLMbiRwh\nDgceBPoBo4GhwE8ktdlvkzeA84L5eOCMYP484HVJqcB3gEHAxUSKJ2a2EMgDrjazFDP7usEGtZ+K\nigomT57MkiVLKCgoYN68eRQUFFSJmTNnDl26dGHDhg1MnTqVadOmAVBQUEBWVhbr1q0jJyeHm2++\nmYqKioP2mZeXx2effdZYQ3TOuUbnhbRmi82szMw+Af4JnLjf+jeA8ySdARQAH0vqCZwNvAWcA2Sb\n2S4z+wL4S113LOkGSXmS8kpLS+tlMAC5ubmEQiESExNp27Yt48aNIzs7u0pMdnY2EyZMACA9PZ2l\nS5diZmRnZzNu3DjatWtHnz59CIVC5Obm1tpnRUUFd955J/fff3+9jcE555obL6Q1K4uar2C/68lm\nVgIcD1wEvE6ksF4BfBkUzsNmZo+bWdjMwt27d4+lqypKSkro3bt35XKvXr0oKSmpMSYuLo7OnTuz\nbdu2Gretrc+ZM2eSlpZGz549620MzjnX3Hghjc07wBT+VUjvCL4CLAO+Lam9pI7AJVHbfQEc15iJ\nNraPPvqIBQsWcOuttzZ1Ks4516C8kMbmDSDOzDYA7wFdgzbMbDnwLLAaWAKsAbYH280FZjf2zUbx\n8fFs2bKlcrm4uJj4+PgaY8rLy9m+fTvdunWrcdua2leuXMmGDRsIhUIkJCTw1VdfEQqFGniEzjnX\nBMzMpwaagI7B1w5EbjAafKh9DBkyxGJxyrTn7JRpz5mZ2Z49e6xPnz62adMmKysrs4EDB9ratWur\nxM+cOdN+8IMfmJnZvHnz7Lvf/a6Zma1du9YGDhxou3btsk2bNlmfPn2svLy8Tn2amR177LExjcM5\n5w4FkGeN9Lfe30fasB4PbkZqD2Sa2XtNmUxcXBwzZ85k9OjRVFRUMGnSJJKTk5kxYwbhcJi0tDSu\nu+46xo8fTygUomvXrmRlZQGQnJzMFVdcwRlnnEFcXByzZs2idevWANX26ZxzRwtFCrdrrsLhsOXl\n5TV1Gs4516JIWmFm4cbYl18jdc4552LghdQ555yLgRdS55xzLgZeSJ1zzrkYeCF1zjnnYuCF1Dnn\nnIuBv4/UOefq0Z49eyguLmbXrl1NncpRoX379vTq1Ys2bfb/gK7G44XUOefqUXFxMccddxwJCQlI\naup0jmhmxrZt2yguLqZPnz5Nloef2nXOuXq0a9cuunXr5kW0EUiiW7duTX7074XUOefqmRfRxtMc\nXmsvpM4551wM/BrpESxh+mKKMsY0dRrOHdUSpi+u1/7q8jv9zW9+k7feeqte91uboqIi3nrrLa66\n6qpG22dz4kekzjl3hGnMIlpeXk5RURFPPfVUo+2zufFC6pxzR5iOHTsC8Nprr/Gtb32LSy+9lMTE\nRKZPn86TTz7J0KFDGTBgABs3bgRg4sSJ3HjjjYTDYU477TSee+45IHLj1LXXXsuAAQM488wzefXV\nVwGYO3cuaWlpjBgxgpEjRzJ9+nTeeOMNUlJSePDBBykqKuK8885j8ODBDB48uLKwv/baa5x//vmk\np6fTr18/rr766n2f3czy5cv55je/yaBBgxg6dChffPEFFRUV3HnnnaSmpjJw4EB+85vfNPZLWSd+\natc5545gq1at4m9/+xtdu3YlMTGR66+/ntzcXB566CEeeeQR/vd//xeInJ7Nzc1l48aNDB8+nA0b\nNjBr1iwksWbNGtavX8+FF17IBx98AMB7773H6tWr6dq1K6+99hoPPPBAZQH+6quveOmll2jfvj2F\nhYVceeWV7Ps4yJUrV7Ju3TpOOukkzjnnHJYtW8bQoUMZO3Ys8+fPJzU1lR07dnDMMccwZ84cOnfu\nzPLlyykrK+Occ87hwgsvbNK3ulTHC2kMFLldTGa2t6lzcc656qSmptKzZ08ATj31VC688EIABgwY\nUHmECXDFFVfQqlUr+vbtS2JiIuvXr+fNN9/k1ltvBaBfv36ccsoplYV01KhRdO3atdp97tmzh1tu\nuYX8/Hxat25duQ3A0KFD6dWrFwApKSkUFRXRuXNnevbsSWpqKgCdOnUC4MUXX2T16tUsXLgQgO3b\nt1NYWNjsCqmf2j1EkhIkvS/pCWAtMEdSnqR1ku6JikuV9JakVZJyJR0nqbWkX0laLmm1pB80Zu5m\nxm233UYoFGLgwIG899571catWLGCAQMGEAqFuO222ypPvXz66aeMGjWKvn37MmrUKD777LNa+83P\nz+fss88mOTmZgQMHMn/+/MYZqHOuUrt27SrnW7VqVbncqlUrysvLK9ft/zaSg72t5Nhjj61x3YMP\nPsiJJ57IqlWryMvLY/fu3dXm07p16yo57M/MeOSRR8jPzyc/P5/NmzdX/iPQnHghPTx9gUfNLBn4\nf8GnsA8EviVpoKS2wHzgdjMbBFwAfA1cB2w3s1QgFfi+pAP+tZJ0Q1Cc80pLS+st6SVLllBYWEhh\nYSGPP/44N910U7VxN910E7/97W8rY3NycgDIyMhg5MiRFBYWMnLkSDIyMmrtt0OHDjzxxBOsW7eO\nnJwcpkyZwueff15v43HO1Z8FCxawd+9eNm7cyKZNm0hKSuK8887jySefBOCDDz7g73//O0lJSQds\ne9xxx/HFF19ULm/fvp2ePXvSqlUr/vjHP1JRUVHrvpOSkti6dSvLly8H4IsvvqC8vJzRo0fz2GOP\nsWfPnsocdu7cWV9Drjd+avfwfGhm7wTzV0i6gchr2RM4AzBgq5ktBzCzHQCSLgQGSkoPtu1MpChv\nju7czB4HHgcIh8NWX0lnZ2dzzTXXIImzzjqLzz//nK1bt1ae9gHYunUrO3bs4KyzzgLgmmuuYdGi\nRVx88cVkZ2fz2muvATBhwgTOP/987rvvvhr7Pe200yr7Pemkk+jRowelpaUcf/zx9TUk55q9lvIW\ntJNPPpmhQ4eyY8cOZs+eTfv27bn55pu56aabGDBgAHFxccydO7fKEeU+AwcOpHXr1gwaNIiJEydy\n8803853vfIcnnniCiy66qNajV4C2bdsyf/58br31Vr7++muOOeYYXn75Za6//nqKiooYPHgwZkb3\n7t1ZtGhRQ70Eh8/MfDqECUgA1gbzfYANQJdgeS4wERgALKtm26eB0YeyvyFDhtjhOmXac1WWx4wZ\nY2+88Ubl8ogRI2z58uVVYpYvX24jR46sXH799ddtzJgxZmbWuXPnyva9e/dWLtel33fffdf69etn\nFRUVhz0e51qCgoKCpk7hkE2YMMEWLFjQ1GkctupecyDPGqku+Knd2HQCdgLbJZ0IXBy0vw/0lJQK\nEFwfjQNeAG6S1CZoP01S7f+qNVOS6vxorq1btzJ+/Hj+8Ic/0KqV/8g5544sfmo3Bma2StJKYD2w\nBVgWtO+WNBZ4RNIxRK6PXgD8jsgR7XvBHb+lwGUNmeOsWbP47W9/C0Tu3tuyZUvluuLiYuLj46vE\nx8fHU1xcXG3MiSeeWHkqeOvWrfTo0aNym5r63bFjB2PGjOEXv/hF5eli51zzMnfu3KZOoUXzw4ND\nZGZFZtY/anmimZ1mZiPN7D/MbG7QvtzMzjKzQcHXL81sr5n9l5kNMLP+ZjbczLY3ZL6TJ0+uvOPt\nsssu44knnsDMeOeddypvOY/Ws2dPOnXqxDvvvIOZ8cQTT3DppZcCkJaWRmZmJgCZmZlV2qvrd/fu\n3Vx++eVcc801pKen49zRwqzebm1wB9EsXuvGOofs0+FNsVwj3d/evXvt5ptvtsTEROvfv3+V65iD\nBg2qnF++fLklJydbYmKiTZ482fbu3WtmZp988omNGDHCQqGQjRw50rZt21Zrv3/84x8tLi7OBg0a\nVDmtXLmy3sbjXHO0adMmKy0trfy9cQ1n7969Vlpaaps2bTpgHY14jVSR/bnmKhwO274ngjjnmr89\ne/ZQXFzc5J+RebRo3749vXr1ok2bNlXaJa2wyFsTG5xfI3XOuXrUpk2bZvfkHdew/Bqpc845FwMv\npM4551wMvJA655xzMfCbjZo5SaXAhzF0cQLwST2l0xz4eJqvI2ks4ONp7g42nlPMrHtjJOKF9Agn\nKa+x7lxrDD6e5utIGgv4eJq75jQeP7XrnHPOxcALqXPOORcDL6RHvsebOoF65uNpvo6ksYCPp7lr\nNuPxa6TOOedcDPyI1DnnnIuBF1LnnHMuBl5IWzBJF0l6X9IGSdOrWd9O0vxg/buSEqLW/WfQ/r6k\n0Y2Zd3UOdyySukl6VdKXkmY2dt41iWE8oyStkLQm+DqisXOvTgzjGSopP5hWSbq8sXOvTiy/O8H6\nk4OfuTsaK+eaxPC9SZD0ddT3Z3Zj516dGP+uDZT0tqR1we9Q+0ZJurE+Zsan+p2A1sBGIBFoC6wC\nztgv5mZgdjA/DpgfzJ8RxLcD+gT9tG6hYzkWOBe4EZjZ1N+XehjPmcBJwXx/oKSFj6cDEBfM9wT+\nuW+5JY4nav1CYAFwR0sdC5AArG3qn696HE8csBoYFCx3a6y/a35E2nINBTaY2SYz2w1kAZfuF3Mp\nkBnMLwRGSlLQnmVmZWa2GdgQ9NdUDnssZrbTzN4EmtNnVsUynpVm9lHQvg44RlK7Rsm6ZrGM5ysz\nKw/a2wPN4e7GWH53kHQZsJnI96epxTSWZiiW8VwIrDazVQBmts3MKhojaS+kLVc8sCVquThoqzYm\n+GO2nch/aXXZtjHFMpbmqL7G8x3gPTMra6A86yqm8UgaJmkdsAa4MaqwNpXDHo+kjsA04J5GyLMu\nYv1Z6yNppaS/SjqvoZOtg1jGcxpgkl6Q9J6kuxohX8A/j9S5ZklSMnAfkf+yWzQzexdIlnQ6kClp\niZk1pzMIh+KnwINm9mXzPairs63AyWa2TdIQYJGkZDPb0dSJHaY4Ipd5UoGvgKWKfLj30obesR+R\ntlwlQO+o5V5BW7UxkuKAzsC2Om7bmGIZS3MU03gk9QKeAa4xs40Nnu3B1cv3x8z+BnxJ5NpvU4pl\nPMOA+yUVAVOA/5J0S0MnXIvDHktwaWcbgJmtIHJt8rQGz7h2sXxvioHXzewTM/sKeB4Y3OAZ44W0\nJVsO9JXUR1JbIhfdn90v5llgQjCfDrxikavwzwLjgrvf+gB9gdxGyrs6sYylOTrs8Ug6HlgMTDez\nZY2Wce1iGU+f4I8dkk4B+gFFjZN2jQ57PGZ2npklmFkC8L/AL82sKe8Wj+V7011SawBJiUT+Dmxq\npLxrEsvfgheAAZI6BD9z3wIKGiXrprgzy6f6mYB/Bz4g8p/k3UHbfwNpwXx7IncWbiBSKBOjtr07\n2O594OIWPpYi4FMiRzvF7HeXX0saD/AjYCeQHzX1aMHjGU/kppx84D3gsqYeS6w/b1F9/JQmvms3\nxu/Nd/b73ny7qccS6/cG+F4wprXA/Y2Vsz8i0DnnnIuBn9p1zjnnYuCF1DnnnIuBF1LnnHMuBl5I\nnXPOuRh4IXXOOedi4IXUOeeci4EXUueccy4G/x/QSaxx74YCbwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "imp = PermutationImportance(learn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And it did! Is that bad? No, it's actually what we want. If they utilized the same things, we'd expect very similar results. We're bringing in other models to hope that they can provide a different outlook to how they're utilizing the features (hopefully differently)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Ensembling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And perform our ensembling! To do so we'll average our predictions to gather (take the sum and divide by 2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "avgs = (nn_preds + xgb_preds) / 2" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([[0.9300, 0.0700],\n", " [0.6735, 0.3265],\n", " [0.6679, 0.3321],\n", " ...,\n", " [0.4095, 0.5905],\n", " [0.9307, 0.0693],\n", " [0.9929, 0.0071]])" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "avgs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now we'll take the argmax to get our predictions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "argmax = avgs.argmax(dim=1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([0, 0, 0, ..., 1, 0, 0])" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "argmax" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How do we know if it worked? Let's grade our predictions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0, 0, 0, ..., 0, 1, 0], dtype=int8)" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "y_test" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor(0.8385)" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "accuracy(tensor(nn_preds), tensor(y_test))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor(0.8340)" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "accuracy(tensor(xgb_preds), tensor(y_test))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor(0.8391)" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "accuracy(tensor(avgs), tensor(y_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see we scored a bit higher!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bringing in Random Forests\n", "\n", "Let's also try with Random Forests" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestClassifier" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tree = RandomForestClassifier(n_estimators=100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's fit" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tree.fit(X_train, y_train);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we are not going to use the default importances. Why? Read up here:\n", "\n", "[Beware Default Random Forest Importances](https://explained.ai/rf-importance/) by Terence Parr, Kerem Turgutlu, Christopher Csiszar, and Jeremy Howard" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Instead, based on their recommendations we'll be utilizing their `rfpimp` package" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install rfpimp" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from rfpimp import *" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "imp = importances(tree, X_test, to.valid.ys)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "plot_importances(imp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Which as we can see, was also very different." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can get our raw probabilities:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "forest_preds = tree.predict_proba(X_test)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.99, 0.01],\n", " [0.72, 0.28],\n", " [0.75, 0.25],\n", " ...,\n", " [0.42, 0.58],\n", " [0.72, 0.28],\n", " [1. , 0. ]])" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "forest_preds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now we can add it to our ensemble:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "avgs = (nn_preds + xgb_preds + forest_preds) / 3" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor(0.8354)" ] }, "execution_count": null, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "accuracy(tensor(avgs), tensor(y_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, it didn't quite work how we wanted to. But that is okay, the goal was to experiment!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 1 }