{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Previously I've shown how to create a [linear model and neural net from scratch](https://www.kaggle.com/code/jhoward/linear-model-and-neural-net-from-scratch), and used it to create a solid submission to Kaggle's [Titanic](https://www.kaggle.com/competitions/titanic/) competition. However, for *tabular* data (i.e data that looks like spreadsheet or database tables, such as the data for the Titanic competition) it's more common to see good results by using ensembles of decision trees, such as Random Forests and Gradient Boosting Machines.\n", "\n", "In this notebook, we're going to learn all about Random Forests, by building one from scratch, and using it to submit to the Titanic competition! That might sound like a pretty big stretch, but I think you'll be surprised to discover how straightforward it actually is.\n", "\n", "We'll start by importing the basic set of libraries we normally need for data science work, and setting numpy to use our display space more efficiently:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19", "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5", "execution": { "iopub.execute_input": "2022-05-23T04:37:24.640765Z", "iopub.status.busy": "2022-05-23T04:37:24.640339Z", "iopub.status.idle": "2022-05-23T04:37:25.174055Z", "shell.execute_reply": "2022-05-23T04:37:25.172992Z", "shell.execute_reply.started": "2022-05-23T04:37:24.640663Z" } }, "outputs": [], "source": [ "from fastai.imports import *\n", "np.set_printoptions(linewidth=130)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data preprocessing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll create `DataFrame`s from the CSV files just like we did in the \"*linear model and neural net from scratch*\" notebook, and do much the same preprocessing (so go back and check that out if you're not already familiar with the dataset):" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:26.152841Z", "iopub.status.busy": "2022-05-23T04:37:26.152518Z", "iopub.status.idle": "2022-05-23T04:37:26.205303Z", "shell.execute_reply": "2022-05-23T04:37:26.204112Z", "shell.execute_reply.started": "2022-05-23T04:37:26.152806Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "titanic.zip: Skipping, found more recently modified local copy (use --force to force download)\n" ] } ], "source": [ "import os\n", "iskaggle = os.environ.get('KAGGLE_KERNEL_RUN_TYPE', '')\n", "\n", "if iskaggle: path = Path('../input/titanic')\n", "else:\n", " import zipfile,kaggle\n", " path = Path('titanic')\n", " kaggle.api.competition_download_cli(str(path))\n", " zipfile.ZipFile(f'{path}.zip').extractall(path)\n", "\n", "df = pd.read_csv(path/'train.csv')\n", "tst_df = pd.read_csv(path/'test.csv')\n", "modes = df.mode().iloc[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One difference with Random Forests however is that we don't generally have to create *dummy variables* like we did for non-numeric columns in the linear models and neural network. Instead, we can just convert those fields to *categorical variables*, which internally in Pandas makes a list of all the unique values in the column, and replaces each value with a number. The number is just an index for looking up the value in the list of all unique values." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:26.941371Z", "iopub.status.busy": "2022-05-23T04:37:26.941013Z", "iopub.status.idle": "2022-05-23T04:37:26.9692Z", "shell.execute_reply": "2022-05-23T04:37:26.968016Z", "shell.execute_reply.started": "2022-05-23T04:37:26.941337Z" } }, "outputs": [], "source": [ "def proc_data(df):\n", " df['Fare'] = df.Fare.fillna(0)\n", " df.fillna(modes, inplace=True)\n", " df['LogFare'] = np.log1p(df['Fare'])\n", " df['Embarked'] = pd.Categorical(df.Embarked)\n", " df['Sex'] = pd.Categorical(df.Sex)\n", "\n", "proc_data(df)\n", "proc_data(tst_df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll make a list of the continuous, categorical, and dependent variables. Note that we no longer consider `Pclass` a categorical variable. That's because it's *ordered* (i.e 1st, 2nd, and 3rd class have an order), and decision trees, as we'll see, only care about order, not about absolute value." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:27.728358Z", "iopub.status.busy": "2022-05-23T04:37:27.728077Z", "iopub.status.idle": "2022-05-23T04:37:27.733013Z", "shell.execute_reply": "2022-05-23T04:37:27.732245Z", "shell.execute_reply.started": "2022-05-23T04:37:27.728328Z" } }, "outputs": [], "source": [ "cats=[\"Sex\",\"Embarked\"]\n", "conts=['Age', 'SibSp', 'Parch', 'LogFare',\"Pclass\"]\n", "dep=\"Survived\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Even although we've made the `cats` columns categorical, they are still shown by Pandas as their original values:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:28.64687Z", "iopub.status.busy": "2022-05-23T04:37:28.645995Z", "iopub.status.idle": "2022-05-23T04:37:28.657163Z", "shell.execute_reply": "2022-05-23T04:37:28.656275Z", "shell.execute_reply.started": "2022-05-23T04:37:28.646831Z" } }, "outputs": [ { "data": { "text/plain": [ "0 male\n", "1 female\n", "2 female\n", "3 female\n", "4 male\n", "Name: Sex, dtype: category\n", "Categories (2, object): ['female', 'male']" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.Sex.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However behind the scenes they're now stored as integers, with indices that are looked up in the `Categories` list shown in the output above. We can view the stored values by looking in the `cat.codes` attribute:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:29.606313Z", "iopub.status.busy": "2022-05-23T04:37:29.605974Z", "iopub.status.idle": "2022-05-23T04:37:29.613586Z", "shell.execute_reply": "2022-05-23T04:37:29.612892Z", "shell.execute_reply.started": "2022-05-23T04:37:29.606276Z" } }, "outputs": [ { "data": { "text/plain": [ "0 1\n", "1 0\n", "2 0\n", "3 0\n", "4 1\n", "dtype: int8" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.Sex.cat.codes.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Binary splits" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before we create a Random Forest or Gradient Boosting Machine, we'll first need to learn how to create a *decision tree*, from which both of these models are built.\n", "\n", "And to create a decision tree, we'll first need to create a *binary split*, since that's what a decision tree is built from.\n", "\n", "A binary split is where all rows are placed into one of two groups, based on whether they're above or below some threshold of some column. For example, we could split the rows of our dataset into males and females, by using the threshold `0.5` and the column `Sex` (since the values in the column are `0` for `female` and `1` for `male`). We can use a plot to see how that would split up our data -- we'll use the [Seaborn](https://seaborn.pydata.org/) library, which is a layer on top of [matplotlib](https://matplotlib.org/) that makes some useful charts easier to create, and more aesthetically pleasing by default:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:31.229486Z", "iopub.status.busy": "2022-05-23T04:37:31.228601Z", "iopub.status.idle": "2022-05-23T04:37:32.45072Z", "shell.execute_reply": "2022-05-23T04:37:32.449794Z", "shell.execute_reply.started": "2022-05-23T04:37:31.229441Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAApkAAAFNCAYAAABL6HT2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmj0lEQVR4nO3de7xdZX3n8c+XcLHcREq4lJBCNUrB8RqpttZiUUFHjW1F46VCy0yGKWjbqSI4rbXVVMSOU6rQNOMgeCOmaiU6qUixeEHQhBaBcKkZQAjXACJgHWjCb/5YK7A52UlOkrXOPif5vF+v8zp7PevZa/+Onjx8z3rWWk+qCkmSJKlLO4y6AEmSJG17DJmSJEnqnCFTkiRJnTNkSpIkqXOGTEmSJHXOkClJkqTOGTI15SRZkORPOjjOuUk+0EVNkjTRkqxIcuSo65A2xJCpTiR5cZLvJPlxkvuSXJrkBX18VlWdWFXv7+PYWyPJzUleNuo6JG0bho0pSY5P8m2Aqjq8qi7ZxDEOTlJJduyxVGkof+m01ZLsCXwF+K/AYmBn4FeBh7fgWAFSVY92WuRWSrJjVa0ZdR2SNJk4NmpjPJOpLjwdoKrOr6q1VfXTqvpaVV0FkOR9ST69rvPYv6yTXJJkfpJLgX8D3pNk+eAHJPnDJEva149Ncye5LsmrB/rtmOSeJM9rt/8uyZ3tGdZvJjl8PD9Qe7bg0iT/M8l9wPuSPDXJ15Pc237GZ5Ls1fb/FDAT+HKSh5Kc0ra/sD3De3+S7zu1Jakrg2c6kxyRZHmSB5LcleQjbbdvtt/vb8emFyXZIckfJ/lhkruTfDLJkweO+7Z2371J/mTM57wvyeeTfDrJA8Dx7Wdf1o5zdyT5WJKdB45XSX4vyQ+SPJjk/e14ellb7+LB/tp2GDLVhX8F1iY5L8krkzxlC47x28A8YA/go8Azkswa2P9m4LND3nc+8KaB7aOBe6rqn9vtfwBmAfsC/wx8ZjNq+iXgxva984EAHwR+DvhF4CDgfQBV9dvALcBrqmr3qjojyYHA/wE+AOwNvBP4QpLpm1GDJI3HmcCZVbUn8FSaWSWAl7Tf92rHpsuA49uvlwK/AOwOfAwgyWHA2cBbgAOAJwMHjvmsOcDngb1oxtS1wB8C+wAvAo4Cfm/Me44Bng+8EDgFWNh+xkHAM3niOK5thCFTW62qHgBeDBTwv4DVSZYk2W8zDnNuVa2oqjVV9WPgAtpBpw2bhwJLhrzvs8Brk+zabj8hjFbVOVX1YFU9TBMInz34F/sm3F5VH21r+mlVrayqi6rq4apaDXwE+LWNvP+twNKqWlpVj1bVRcBy4FXj/HxJ+lJ7hvD+JPfTBMBh/h14WpJ9quqhqrp8I8d8C/CRqrqxqh4CTgPmtrNLrwe+XFXfrqpHgPfSjO2DLquqL7Xj2k+r6oqqurwdK28G/pb1x8YPVdUDVbUCuAb4Wvv5P6Y5GfDccf8voinDkKlOVNV1VXV8Vc2g+av054C/2oxD3Dpm+7M8/pftm4EvVdW/DfnclcB1wGvaoPna9r0kmZbk9CT/t53Wubl92z5bUlOSfZMsSnJbe7xPb+JYPw8cO+Y/EC+mOTsgSePxuqraa90X658hXOcEmkuXrk+ybPAyoiF+DvjhwPYPae7R2K/d99jY14679455/9ix8elJvtJemvQA8BesPzbeNfD6p0O2d99IvZqiDJnqXFVdD5xLEzYBfgLsOtBl/2FvG7P9NWCfJM+hCZvDpsrXWTdlPge4tg2e0ITTOcDLaKZ8Dm7bM44fY1hNH2zbntVOSb11zLHG9r8V+NTgfyCqareqOn2cny9J41JVP6iqN9Fc3vMh4PNJdmP9cQngdpo/gteZCayhCX53ADPW7UjyM8DPjv24Mdt/A1wPzGrHxvcw/nFW2zBDprZakkOT/FGSGe32QTShb910zZXAS5LMbKeqT9vUMdu7FT8PfJjmesaLNtJ9EfAKmrvbB8PoHjR3uN9LE3L/YjN+rGH2AB6iuYD+QOBdY/bfRXN90zqfpjnDenR7VvVJSY5c97+TJHUlyVuTTG+fzHF/27wWWA08yhPHpvOBP0xySJLdacbGzw2Mu69J8svtzTh/xqYD4x7AA8BDSQ6lGYslQ6Y68SDNTTLfTfITmnB5DfBHAO21iJ8DrgKuoHnc0Xh8luYs5N9t7BEZVXUHcBnwy+3nrPNJmmmg24BreTz0bqk/A54H/Jjmhp4vjtn/QeCP26nxd1bVrTRnUt9DM9DfShNM/XcnqWvHACuSPERzE9Dcqvp/7XT3fODSdmx6IXAO8CmaO89vAv4f8HaA9prJt9P88X4Hzfh+Nxt/JN07aWaOHqS5Lv9zG+mr7Uiqhp1JlyRJ27v2TOf9NFPhN424HE0xnlGRJEmPSfKaJLu213T+JXA1j984KY2bIVOSJA2aQ3Nz0O00zxmeW057ags4XS5JkqTOeSZTkiRJnTNkStIkkGSvdk3o65Nc164xvXeSi9o1ny8aXLI1yWlJVia5IcnRo6xdkoaZctPl++yzTx188MGjLkPSNuaKK664p6pGtq58kvOAb1XVx9vnE+5K8/ir+6rq9CSnAk+pqne360ufDxxBs0LLPwJPr6q1Gzq+Y6ekPmxs7NxxoovZWgcffDDLly8fdRmStjFJfrjpXr199p7AS4DjAdo1ox9JMgc4su12HnAJ8G6aGzMWVdXDwE1JVtIEzss29BmOnZL6sLGx0+lySRq9X6B5YP8nkvxLko+3j4/Zr11sYN2iA/u2/Q/kietHr2rbJGnSMGRK0ujtSLOa1N9U1XOBnwCnbqT/sGX+1rv2Kcm8JMuTLF+9enU3lUrSOBkyJWn0VgGrquq77fbnaULnXUkOAGi/3z3Q/6CB98+geabhE1TVwqqaXVWzp08f2eWmkrZThkxJGrGquhO4Nckz2qajgGuBJcBxbdtxwAXt6yXA3CS7JDmE5oHZ35vAkiVpk3q98SfJMcCZwDTg41V1+pj9TwY+Dcxsa/nLqvpEnzVJ0iT1duAz7Z3lNwK/Q3MiYHGSE4BbgGMBqmpFksU0QXQNcNLG7iyXpFHoLWQmmQacBbycZmpnWZIlVXXtQLeTgGur6jVJpgM3JPlMe2elJG03qupKYPaQXUdtoP98YH6fNUnS1uhzuvwIYGVV3diGxkU0j90YVMAeSQLsDtxH81e5JEmSprA+Q+Z4HrHxMeAXaS5Yvxr4/ap6tMeaJEmSNAH6DJnjecTG0cCVNCtWPAf4WPtQ4iceyMdwSJIkTSl9hszxPGLjd4AvVmMlcBNw6NgD+RgOSZKkqaXPu8uXAbPax2vcBswF3jymzy00F7V/K8l+wDNo7qrUZjjllFO488472X///TnjjDNGXY4kSVJ/IbOq1iQ5GbiQ5hFG57SP3Tix3b8AeD9wbpKraabX311V9/RV07bqzjvv5Lbbbht1GZKk7dgtf/4fRl2CtsLM917d+TF7fU5mVS0Flo5pWzDw+nbgFX3WIEmSpInnij+SJEnqnCFTkiRJnTNkSpIkqXOGTEmSJHXOkClJkqTOGTIlSZLUOUOmJEmSOmfIlCRJUucMmZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkdc6QKUmSpM4ZMiVJktS5HUddwER5/rs+OeoSerPHPQ8yDbjlnge32Z/zig+/bdQlSJKkzeCZTEmSJHXOkClJkqTOGTIlSZLUOUOmJEmSOmfIlCRJUucMmZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkda7XkJnkmCQ3JFmZ5NQh+9+V5Mr265oka5Ps3WdNkiRJ6l9vITPJNOAs4JXAYcCbkhw22KeqPlxVz6mq5wCnAd+oqvv6qkmSJEkTo88zmUcAK6vqxqp6BFgEzNlI/zcB5/dYjyRJkiZInyHzQODWge1Vbdt6kuwKHAN8YQP75yVZnmT56tWrOy9UkiRJ3eozZGZIW22g72uASzc0VV5VC6tqdlXNnj59emcFSpIkqR99hsxVwEED2zOA2zfQdy5OlUvajiW5OcnV7Y2Qy9u2vZNclOQH7fenDPQ/rb2p8oYkR4+uckkars+QuQyYleSQJDvTBMklYzsleTLwa8AFPdayTXt0591Yu8uePLrzbqMuRdLWeWl7M+TsdvtU4OKqmgVc3G7T3kQ5Fzic5lKjs9ubLSVp0tixrwNX1ZokJwMXAtOAc6pqRZIT2/0L2q6/AXytqn7SVy3bup/MesWoS5DUjznAke3r84BLgHe37Yuq6mHgpiQraW62vGwENUrSUL2FTICqWgosHdO2YMz2ucC5fdYhSVNAAV9LUsDfVtVCYL+qugOgqu5Ism/b90Dg8oH3Dr2xMsk8YB7AzJkz+6xdktbTa8iUJI3br1TV7W2QvCjJ9RvpO64bK9uguhBg9uzZG7rxUpJ64bKSkjQJVNXt7fe7gb+nmf6+K8kBAO33u9vum3NjpSSNhCFTkkYsyW5J9lj3GngFcA3NzZLHtd2O4/EbJJcAc5PskuQQYBbwvYmtWpI2zulySRq9/YC/TwLNuPzZqvpqkmXA4iQnALcAxwK0N1EuBq4F1gAnVdXa0ZQuScMZMiVpxKrqRuDZQ9rvBY7awHvmA/N7Lk2StpjT5ZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkdc6QKUmSpM4ZMiVJktQ5Q6YkSZI6Z8iUJElS5wyZkiRJ6pwhU5IkSZ0zZEqSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjpnyJQkSVLnDJmSJEnqnCFTkiRJnes1ZCY5JskNSVYmOXUDfY5McmWSFUm+0Wc9kiRJmhg79nXgJNOAs4CXA6uAZUmWVNW1A332As4GjqmqW5Ls21c9kiRJmjh9nsk8AlhZVTdW1SPAImDOmD5vBr5YVbcAVNXdPdYjSZKkCdJnyDwQuHVge1XbNujpwFOSXJLkiiRv67EeSZIkTZDepsuBDGmrIZ//fOAo4GeAy5JcXlX/+oQDJfOAeQAzZ87soVRJkiR1qc8zmauAgwa2ZwC3D+nz1ar6SVXdA3wTePbYA1XVwqqaXVWzp0+f3lvBkiRJ6kafIXMZMCvJIUl2BuYCS8b0uQD41SQ7JtkV+CXguh5rkiRJ0gTobbq8qtYkORm4EJgGnFNVK5Kc2O5fUFXXJfkqcBXwKPDxqrqmr5okSZI0Mfq8JpOqWgosHdO2YMz2h4EP91mHJEmSJpYr/kiSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjpnyJQkSVLnDJmSJEnqnCFTkiRJnTNkSpIkqXOGTEmaBJJMS/IvSb7Sbu+d5KIkP2i/P2Wg72lJVia5IcnRo6takjbMkClJk8PvA9cNbJ8KXFxVs4CL222SHAbMBQ4HjgHOTjJtgmuVpE0yZErSiCWZAfxH4OMDzXOA89rX5wGvG2hfVFUPV9VNwErgiAkqVZLGzZApSaP3V8ApwKMDbftV1R0A7fd92/YDgVsH+q1q29aTZF6S5UmWr169uvOiJWljDJmSNEJJXg3cXVVXjPctQ9pqWMeqWlhVs6tq9vTp07e4RknaEjuOugBJ2s79CvDaJK8CngTsmeTTwF1JDqiqO5IcANzd9l8FHDTw/hnA7RNasSSNg2cyJWmEquq0qppRVQfT3NDz9ap6K7AEOK7tdhxwQft6CTA3yS5JDgFmAd+b4LIlaZM8kylJk9PpwOIkJwC3AMcCVNWKJIuBa4E1wElVtXZ0ZUrScIZMSZokquoS4JL29b3AURvoNx+YP2GFSdIWcLpckiRJnTNkSpIkqXOGTEmSJHXOkClJkqTOGTIlSZLUOUOmJEmSOtdryExyTJIbkqxMcuqQ/Ucm+XGSK9uv9/ZZjyRJkiZGb8/JTDINOAt4Oc0yaMuSLKmqa8d0/VZVvbqvOiRJkjTx+jyTeQSwsqpurKpHgEXAnB4/T5IkSZNEnyHzQODWge1VbdtYL0ry/ST/kOTwHuuRJEnSBOlzWckMaasx2/8M/HxVPZTkVcCXgFnrHSiZB8wDmDlzZsdlSpIkqWt9nslcBRw0sD0DuH2wQ1U9UFUPta+XAjsl2WfsgapqYVXNrqrZ06dP77FkSZIkdaHPkLkMmJXkkCQ7A3OBJYMdkuyfJO3rI9p67u2xJkmSJE2A3qbLq2pNkpOBC4FpwDlVtSLJie3+BcDrgf+aZA3wU2BuVY2dUpckSdIU0+c1meumwJeOaVsw8PpjwMf6rEGSJEkTzxV/JEmS1DlDpiRJkjpnyJQkSVLnDJmSJEnq3EZv/EnyIOs/QP0xVbVn5xVJkiRpyttoyKyqPQCS/DlwJ/ApmpV83gLs0Xt1kiRJmpLGO11+dFWdXVUPtqv0/A3wW30WJkmSpKlrvCFzbZK3JJmWZIckbwHW9lmYJEmSpq7xhsw3A28A7mq/jm3bJEmSpPWMa8WfqroZmNNvKZIkSdpWjOtMZpKnJ7k4yTXt9rOS/HG/pUmSJGmqGu90+f8CTgP+HaCqrgLm9lWUJE1FSS4eT5skbQ/GNV0O7FpV30sy2Lamh3okacpJ8iRgV2CfJE+hedQbwJ7Az42sMEkaofGGzHuSPJX2wexJXg/c0VtVkjS1/BfgD2gC5RU8HjIfAM4aUU2SNFLjDZknAQuBQ5PcBtxE80B2SdruVdWZwJlJ3l5VHx11PZI0GYw3ZP6wql6WZDdgh6p6sM+iJGkqqqqPJvll4GAGxteq+uTIipKkERlvyLwpyVeBzwFf77EeSZqyknwKeCpwJY8vWFGAIVPSdme8IfMZwGtops3/d5KvAIuq6tu9VSZJU89s4LCqqlEXIkmjNt6Hsf8UWAwsbu+cPBP4BjCtx9okaaq5BtifbejGyOe/y5OwU9kVH37bqEvQdmy8ZzJJ8mvAG4FXAstolpmUJD1uH+DaJN8DHl7XWFWvHV1JkjQa4wqZSW6iucZoMfCuqvpJn0VJ0hT1vi15U/uczW8Cu9CMy5+vqj9NsjfNtfAHAzcDb6iqH7XvOQ04gebaz3dU1YVbW7wkdWm8ZzKfXVUP9FqJJE1xVfWNLXzrw8CvV9VDSXYCvp3kH4DfBC6uqtOTnAqcCrw7yWE0q64dTvNszn9M8vSqWruhD5CkibbRkJnklKo6A5ifZL0L2avqHb1VJklTTJIHaRetAHYGdgJ+UlV7bux97Y1CD7WbO7VfBcwBjmzbzwMuAd7dti+qqodpnv6xEjgCuKyrn0WSttamzmRe135f3nchkjTVVdUeg9tJXkcT/jYpyTSa1YKeBpxVVd9Nsl9V3dEe+44k+7bdDwQuH3j7qrZNkiaNjYbMqvpy+/KqqvqXzT14kmNo7kSfBny8qk7fQL8X0AyYb6yqz2/u50jSZFRVX2qnucfTdy3wnCR7AX+f5Jkb6Z4hbevNNiWZB8wDmDlz5njKkKTOjPeazI8kOQD4O5opmhWbekP7V/lZwMtp/spelmRJVV07pN+HAC9alzSlJfnNgc0daJ6buVnPzKyq+5NcAhwD3JXkgPYs5gHA3W23VcBBA2+bAdw+5FgLaZYEZvbs2T67U9KE2mE8narqpTTXBa0GFia5Oskfb+JtRwArq+rGqnoEWERzHdFYbwe+wOODpyRNVa8Z+DoaeJDh494TJJnensEkyc8ALwOuB5YAx7XdjgMuaF8vAeYm2SXJIcAs4Hvd/RiStPXG/ZzMqroT+Osk/wScArwX+MBG3nIgcOvA9irglwY7JDkQ+A3g14EXjLcWSZqMqup3tvCtBwDntTM7OwCLq+orSS6jWQTjBOAW4Nj2c1YkWQxcC6wBTvLOckmTzXifk/mLNA9ifz1wL81ZyT/a1NuGtI2drvkr4N1VtTYZ1v2xz/e6IkmTXpIZwEeBX6EZ774N/H5VrdrY+6rqKuC5Q9rvBY7awHvmA/O3tmZJ6st4z2R+AjgfeEVVrXfdzwaM55qh2cCiNmDuA7wqyZqq+tJgJ68rkjRFfAL4LO0ZR+CtbdvLR1aRJI3IJkNmO33zf6vqzM089jJgVnu90G00Dw5+82CHqjpk4HPOBb4yNmBK0hQyvao+MbB9bpI/GFUxkjRKm7zxp73O52eT7Lw5B66qNcDJNHeNX0dzjdGKJCcmOXGLqpWkye2eJG9NMq39eivNJUaStN0Z73T5D4FLkywBHlu3vKo+srE3VdVSYOmYtgUb6Hv8OGuRpMnqd4GPAf+T5prM7wBbejOQJE1p4w2Zt7dfOwB7bKKvJG2v3g8cV1U/AkiyN/CXNOFTkrYr4wqZVfVnfRciSduAZ60LmABVdV+S9e4al6TtwXgfYfRPDFm1oqp+vfOKJGnq2iHJU8acyRz384glaVsy3sHvnQOvnwT8Fs0DgCVJj/sfwHeSfJ7mD/M34LMsJW2nxjtdfsWYpkuTfKOHeiRpyqqqTyZZTrOKWYDfrKprR1yWJI3EeKfL9x7Y3IHmIer791KRJE1hbag0WEra7o13uvwKHr8mcw1wM3BCHwVJkiRp6ttoyEzyAuDWdSvzJDmO5nrMm/EvdUmSJG3Aplb8+VvgEYAkLwE+CJwH/Jh2LXFJkiRprE1Nl0+rqvva128EFlbVF4AvJLmy18okSZI0ZW3qTOa0JOuC6FHA1wf2+ew3SZIkDbWpoHg+8I0k9wA/Bb4FkORpNFPmkiRJ0no2GjKran6Si4EDgK9V1bo7zHcA3t53cZIkSZqaNjnlXVWXD2n7137KkSRJ0rZgU9dkSpIkSZvNkClJkqTOGTIlSZLUOUOmJEmSOmfIlCRJUucMmZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkdc6QKUmSpM71GjKTHJPkhiQrk5w6ZP+cJFcluTLJ8iQv7rMeSZIkTYxNrl2+pZJMA84CXg6sApYlWVJV1w50uxhYUlWV5FnAYuDQvmqSJEnSxOjzTOYRwMqqurGqHgEWAXMGO1TVQ1VV7eZuQCFJkqQpr8+QeSBw68D2qrbtCZL8RpLrgf8D/O6wAyWZ106nL1+9enUvxUqSJKk7fYbMDGlb70xlVf19VR0KvA54/7ADVdXCqppdVbOnT5/ebZWSJEnqXJ8hcxVw0MD2DOD2DXWuqm8CT02yT481SZIkaQL0GTKXAbOSHJJkZ2AusGSwQ5KnJUn7+nnAzsC9PdYkSZKkCdDb3eVVtSbJycCFwDTgnKpakeTEdv8C4LeAtyX5d+CnwBsHbgSSJEnSFNVbyASoqqXA0jFtCwZefwj4UJ81SNJkl+Qg4JPA/sCjwMKqOjPJ3sDngIOBm4E3VNWP2vecBpwArAXeUVUXjqB0SdogV/yRpNFbA/xRVf0i8ELgpCSHAacCF1fVLJrnCp8K0O6bCxwOHAOc3T6bWJImDUOmJI1YVd1RVf/cvn4QuI7mkW9zgPPabufRPIWDtn1RVT1cVTcBK2meTSxJk4YhU5ImkSQHA88FvgvsV1V3QBNEgX3bbuN6DrEkjZIhU5ImiSS7A18A/qCqHthY1yFt69006UIWkkbJkClJk0CSnWgC5meq6ott811JDmj3HwDc3baP6znELmQhaZQMmZI0Yu3zgv83cF1VfWRg1xLguPb1ccAFA+1zk+yS5BBgFvC9iapXksaj10cYSZLG5VeA3wauTnJl2/Ye4HRgcZITgFuAYwHaZw4vBq6luTP9pKpaO+FVS9JGGDIlacSq6tsMv84S4KgNvGc+ML+3oiRpKzldLkmSpM4ZMiVJktQ5Q6YkSZI6Z8iUJElS5wyZkiRJ6pwhU5IkSZ0zZEqSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjrn2uXSduSUU07hzjvvZP/99+eMM84YdTmSpG2YIVPajtx5553cdtttoy5DkrQdcLpckiRJnTNkSpIkqXO9hswkxyS5IcnKJKcO2f+WJFe1X99J8uw+65EkSdLE6C1kJpkGnAW8EjgMeFOSw8Z0uwn4tap6FvB+YGFf9UiSJGni9Hkm8whgZVXdWFWPAIuAOYMdquo7VfWjdvNyYEaP9UiSJGmC9BkyDwRuHdhe1bZtyAnAP/RYjyRJkiZIn48wypC2GtoxeSlNyHzxBvbPA+YBzJw5s6v6JEmS1JM+z2SuAg4a2J4B3D62U5JnAR8H5lTVvcMOVFULq2p2Vc2ePn16L8VKkiSpO32GzGXArCSHJNkZmAssGeyQZCbwReC3q+pfe6xFkiRJE6i36fKqWpPkZOBCYBpwTlWtSHJiu38B8F7gZ4GzkwCsqarZfdUkSZKkidHrspJVtRRYOqZtwcDr/wT8pz5rkLbELX/+H0ZdQi/W3Lc3sCNr7vvhNvszznzv1aMuQZKEK/5IkiSpB4ZMSZIkdc6QKUmSpM4ZMiVJktQ5Q6YkSZI6Z8iUJElS5wyZkiRJ6pwhU5IkSZ0zZEqSJKlzhkxJkiR1zpApSZKkzvW6drmkyWWfJz0KrGm/S5LUH0OmtB1557PuH3UJGiLJOcCrgbur6plt297A54CDgZuBN1TVj9p9pwEnAGuBd1TVhSMoW5I2yulySRq9c4FjxrSdClxcVbOAi9ttkhwGzAUOb99zdpJpE1eqJI2PIVOSRqyqvgncN6Z5DnBe+/o84HUD7Yuq6uGquglYCRwxEXVK0uYwZErS5LRfVd0B0H7ft20/ELh1oN+qtk2SJhVDpiRNLRnSVkM7JvOSLE+yfPXq1T2XJUlPZMiUpMnpriQHALTf727bVwEHDfSbAdw+7ABVtbCqZlfV7OnTp/darCSNZciUpMlpCXBc+/o44IKB9rlJdklyCDAL+N4I6pOkjfIRRpI0YknOB44E9kmyCvhT4HRgcZITgFuAYwGqakWSxcC1wBrgpKpaO5LCJWkjDJmSNGJV9aYN7DpqA/3nA/P7q0iStp7T5ZIkSeqcIVOSJEmdM2RKkiSpc72GzCTHJLkhycokpw7Zf2iSy5I8nOSdfdYiSZKkidPbjT/tWrpnAS+nea7bsiRLquragW73Ae/g8eXSJEmStA3o80zmEcDKqrqxqh4BFtGsufuYqrq7qpYB/95jHZIkSZpgfYZM19eVJEnaTvUZMse9vu4mD+T6u5IkSVNKnyFz3Ovrborr70qSJE0tfYbMZcCsJIck2RmYS7PmriRJkrZxvd1dXlVrkpwMXAhMA85p19w9sd2/IMn+wHJgT+DRJH8AHFZVD/RVlyRJkvrX69rlVbUUWDqmbcHA6ztpptElSZK0DXHFH0mSJHXOkClJkqTOGTIlSZLUOUOmJEmSOmfIlCRJUucMmZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkdc6QKUmSpM4ZMiVJktQ5Q6YkSZI6Z8iUJElS5wyZkiRJ6pwhU5IkSZ0zZEqSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjpnyJQkSVLnDJmSJEnqnCFTkiRJnTNkSpIkqXO9hswkxyS5IcnKJKcO2Z8kf93uvyrJ8/qsR5K2FZsaXyVp1HoLmUmmAWcBrwQOA96U5LAx3V4JzGq/5gF/01c9krStGOf4Kkkj1eeZzCOAlVV1Y1U9AiwC5ozpMwf4ZDUuB/ZKckCPNUnStmA846skjVSfIfNA4NaB7VVt2+b2kSQ9kWOnpElvxx6PnSFttQV9SDKPZjod4KEkN2xlbduifYB7Rl1EX/KXx426hG3JNv27wp8OG1bG5ee7LKNnjp3d2ab/PTh2dmqb/l3pY+zsM2SuAg4a2J4B3L4FfaiqhcDCrgvcliRZXlWzR12HJj9/V7YJjp0d8d+Dxsvflc3X53T5MmBWkkOS7AzMBZaM6bMEeFt7l/kLgR9X1R091iRJ24LxjK+SNFK9ncmsqjVJTgYuBKYB51TViiQntvsXAEuBVwErgX8DfqeveiRpW7Gh8XXEZUnSE6Rqvct4NAUlmddOjUkb5e+K9Dj/PWi8/F3ZfIZMSZIkdc5lJSVJktQ5Q+YkkeQdSa5L8pmejv++JO/s49ia2pIcmeQro65D2hKOnRoVx85N6/MRRto8vwe8sqpuGnUhkjSFOHZKk5RnMieBJAuAXwCWJPnvSc5JsizJvySZ0/Y5PsmXknw5yU1JTk7y39o+lyfZu+33n9v3fj/JF5LsOuTznprkq0muSPKtJIdO7E+sriU5OMn1ST6e5Jokn0nysiSXJvlBkiPar++0vzPfSfKMIcfZbdjvnzQZOXZqazl29suQOQlU1Yk0D1J+KbAb8PWqekG7/eEku7Vdnwm8mWbd4vnAv1XVc4HLgLe1fb5YVS+oqmcD1wEnDPnIhcDbq+r5wDuBs/v5yTTBngacCTwLOJTmd+XFNP8fvwe4HnhJ+zvzXuAvhhzjv7Ph3z9pUnHsVEccO3vidPnk8wrgtQPXAD0JmNm+/qeqehB4MMmPgS+37VfT/OMAeGaSDwB7AbvTPEfvMUl2B34Z+LvksSWkdunh59DEu6mqrgZIsgK4uKoqydXAwcCTgfOSzKJZgnCnIcfY0O/fdX0XL20lx05tKcfOnhgyJ58Av1VVT1hjOMkvAQ8PND06sP0oj/9/eS7wuqr6fpLjgSPHHH8H4P6qek6nVWsy2NTvx/tp/mP7G0kOBi4Zcoyhv3/SFODYqS3l2NkTp8snnwuBt6f9UznJczfz/XsAdyTZCXjL2J1V9QBwU5Jj2+MnybO3smZNDU8GbmtfH7+BPlv7+yeNimOn+uLYuYUMmZPP+2lOxV+V5Jp2e3P8CfBd4CKa60iGeQtwQpLvAysAL1DePpwBfDDJpTRLEQ6ztb9/0qg4dqovjp1byBV/JEmS1DnPZEqSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjpnyNSU1q5XvCLJVUmubB+8LEnaCMdOTQRX/NGUleRFwKuB51XVw0n2AXYecVmSNKk5dmqieCZTU9kBwD1V9TBAVd1TVbcneX6SbyS5IsmFSQ5I8uQkNyR5BkCS85P855FWL0mj4dipCeHD2DVlJdkd+DawK/CPwOeA7wDfAOZU1eokbwSOrqrfTfJy4M+BM4Hjq+qYEZUuSSPj2KmJ4nS5pqyqeijJ84FfBV5KM1B+AHgmcFG7hOw04I62/0XtusNnAa45LGm75NipieKZTG0zkrweOAl4UlW9aMj+HWj+Uj8EeFVVXTXBJUrSpOPYqb54TaamrCTPSDJroOk5wHXA9PbCdpLslOTwdv8ftvvfBJyTZKeJrFeSJgPHTk0Uz2Rqymqnez4K7AWsAVYC84AZwF8DT6a5JOSvaP4KvwA4oqoeTPIR4MGq+tOJr1ySRsexUxPFkClJkqTOOV0uSZKkzhkyJUmS1DlDpiRJkjpnyJQkSVLnDJmSJEnqnCFTkiRJnTNkSpIkqXOGTEmSJHXu/wNqeVkkNDm5xAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import seaborn as sns\n", "\n", "fig,axs = plt.subplots(1,2, figsize=(11,5))\n", "sns.barplot(data=df, y=dep, x=\"Sex\", ax=axs[0]).set(title=\"Survival rate\")\n", "sns.countplot(data=df, x=\"Sex\", ax=axs[1]).set(title=\"Histogram\");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we see that (on the left) if we split the data into males and females, we'd have groups that have very different survival rates: >70% for females, and <20% for males. We can also see (on the right) that the split would be reasonably even, with over 300 passengers (out of around 900) in each group.\n", "\n", "We could create a very simple \"model\" which simply says that all females survive, and no males do. To do so, we better first split our data into a training and validation set, to see how accurate this approach turns out to be:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:32.688753Z", "iopub.status.busy": "2022-05-23T04:37:32.688257Z", "iopub.status.idle": "2022-05-23T04:37:32.908453Z", "shell.execute_reply": "2022-05-23T04:37:32.907399Z", "shell.execute_reply.started": "2022-05-23T04:37:32.688718Z" } }, "outputs": [], "source": [ "from numpy import random\n", "from sklearn.model_selection import train_test_split\n", "\n", "random.seed(42)\n", "trn_df,val_df = train_test_split(df, test_size=0.25)\n", "trn_df[cats] = trn_df[cats].apply(lambda x: x.cat.codes)\n", "val_df[cats] = val_df[cats].apply(lambda x: x.cat.codes)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(In the previous step we also replaced the categorical variables with their integer codes, since some of the models we'll be building in a moment require that.)\n", "\n", "Now we can create our independent variables (the `x` variables) and dependent (the `y` variable):" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:33.154552Z", "iopub.status.busy": "2022-05-23T04:37:33.153932Z", "iopub.status.idle": "2022-05-23T04:37:33.164771Z", "shell.execute_reply": "2022-05-23T04:37:33.163736Z", "shell.execute_reply.started": "2022-05-23T04:37:33.154505Z" } }, "outputs": [], "source": [ "def xs_y(df):\n", " xs = df[cats+conts].copy()\n", " return xs,df[dep] if dep in df else None\n", "\n", "trn_xs,trn_y = xs_y(trn_df)\n", "val_xs,val_y = xs_y(val_df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here's the predictions for our extremely simple model, where `female` is coded as `0`:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:33.932259Z", "iopub.status.busy": "2022-05-23T04:37:33.931549Z", "iopub.status.idle": "2022-05-23T04:37:33.940681Z", "shell.execute_reply": "2022-05-23T04:37:33.93969Z", "shell.execute_reply.started": "2022-05-23T04:37:33.932218Z" } }, "outputs": [], "source": [ "preds = val_xs.Sex==0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll use mean absolute error to measure how good this model is:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:34.580131Z", "iopub.status.busy": "2022-05-23T04:37:34.579605Z", "iopub.status.idle": "2022-05-23T04:37:34.588669Z", "shell.execute_reply": "2022-05-23T04:37:34.58794Z", "shell.execute_reply.started": "2022-05-23T04:37:34.580088Z" } }, "outputs": [ { "data": { "text/plain": [ "0.21524663677130046" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.metrics import mean_absolute_error\n", "mean_absolute_error(val_y, preds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternatively, we could try splitting on a continuous column. We have to use a somewhat different chart to see how this might work -- here's an example of how we could look at `LogFare`:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:36.079566Z", "iopub.status.busy": "2022-05-23T04:37:36.078932Z", "iopub.status.idle": "2022-05-23T04:37:36.428338Z", "shell.execute_reply": "2022-05-23T04:37:36.427385Z", "shell.execute_reply.started": "2022-05-23T04:37:36.079515Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAo8AAAE9CAYAAABqeoiYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABE/UlEQVR4nO3deXxU9b3/8ddnJhvZgISw74soKojiimuh1bZS609t7Xrb2lpbF7pX216t7b21ve2vt1rttVzR2+W2Krb0J9VqxX0XcAFZhMi+JpBAQkL2z++PmWAIARKSmXNm8n4+HnmQOXPmnHdknHz4rubuiIiIiIh0RiToACIiIiKSOlQ8ioiIiEinqXgUERERkU5T8SgiIiIinabiUUREREQ6TcWjiIiIiHRaRtAB2howYICPHj066BgikmaWLFmy091Lgs6RCPrcFJFEONznZqiKx9GjR7N48eKgY4hImjGzDUFnSBR9bopIIhzuc1Pd1iIiIiLSaSoeRURERKTTVDyKiIiISKepeBQRERGRTlPxKCIiIiKdpuJRJI28/PLLQUcQEZE0p+JRJE3ceeed3HTTTdx5551BRxERkTSm4lEkDdTW1vLQQw8B8NBDD1FbWxtwIhERSVcqHkXSwLXXXnvYxyIiIj1FxaNIiluyZAnr1q074Ni6det44403AkokIiLpTMWjSIr79a9/3eHxO+64I8lJRIL34KJNfOF/FvH/3tyCuwcdRyQtqXgUSXGzZ8/u8PgNN9yQ5CQiwdq+p44fLljOi6U7mX3/mzyzujzoSCJpScWjSIqbOnUqY8aMOeDYmDFjmDp1akCJRILxH4+toqnFeXT2OQwqzGbu8+uO/CIR6TIVjyJp4K677jrsY5F0t6+hmb8v3cYnTxvJuJJ8PnfWGF4o3cmKrVVBRxNJOyoeRdJAbm4ul19+OQCXX345ubm5AScSSa4lGyppaG7h/IklAHzytJFkRIyH39oacDKR9JMRdAAR6RnXXXcdp5xyCmeeeWbQUUSS7qV3d5IRMU4dXQRA39xMThrRj5fX7go4mUj6UcujSBpR4Si91Uvv7mLKiH7kZb/XJnLWuGKWbd5NVV1jgMlE0o+KRxERSWnVdY0s27KHs8YVH3D8zHEDaHFYtK4ioGQi6UnFo4iIpLTlW6tobnFOHtX/gONTR/YjKyPCy++q61qkJyW0eDSzfmb2kJmtMrOVZqY+NRER6VFryvYCMHFQwQHHczKjnDSiH4s3VAYRSyRtJbrl8XbgMXc/FpgCrEzw/UREpJdZs6OaguwMhvTNOei5E4b25Z3t1TS3aLcZkZ6SsOLRzAqBc4G5AO7e4O67E3U/ERHpnVbvqGb8oHzM7KDnjh9ayL7GZtbt3BtAMpH0lMiWx7FAOXCfmb1hZveYWV4C7yciIr1QadleJgzM7/C544cVArFxkSLSMxJZPGYAJwP/5e5TgRrgxvYnmdnVZrbYzBaXl2sfUhER6byKmgZ27m3gmHbjHVuNK8knKyOi4lGkByWyeNwMbHb3V+OPHyJWTB7A3ee4+zR3n1ZSUpLAOCIikm5W76gGYMIhisfMaISJgwq0TaFID0pY8eju24FNZjYxfmgGsCJR9xMRkd6ndab1obqtITbucfnWPbhr0oxIT0j0bOvrgf81s6XAScBPEnw/ERHpRTZX1JKVEWFw4cEzrVtNHFxAZW0jO/c2JDGZSPpK6N7W7v4mMC2R9xARkd5rc+U+hvfrQyRy8EzrVuPjrZKlZXspKchOVjSRtKUdZkREksTMLjKzd8ys1MwOmkAYP+d8M3vTzJab2bPJzphqNlfWMqx/n8OeM64kXjyWa7kekZ6g4lFEJAnMLArcBXwQmAR8wswmtTunH/Ab4CPufjxwRbJzpprNlfsY3j/3sOcM6ZtDXlaUd8tUPIr0BBWPIiLJcRpQ6u5r3b0BuB+4pN05nwT+6u4bAdy9LMkZU0ptQxO7ahoYfoSWRzNj3MB83lXLo0iPUPEoIpIcw4BNbR5vjh9r6xigv5k9Y2ZLzOyzHV1I6+PGbN29D+CIxSPA+JJ8StXyKNIjVDyKiCRHRzM62q8dkwGcAnwYuBD4VzM75qAXaX1cADZVdr54HDcwn2176thb35ToWCJpT8WjiEhybAZGtHk8HNjawTmPuXuNu+8EngOmJClfytm8v3g8/JhHeG/GtcY9inSfikcRkeRYBEwwszFmlgVcCTzc7pz/B5xjZhlmlgucDqxMcs6UsbmylqxohJL8Iy+/M3ZAHgDrd9UkOpZI2kvoOo8iIhLj7k1mdh3wOBAF7nX35WZ2Tfz5u919pZk9BiwFWoB73P3t4FKH2+bKfQztl3PYNR5bjSjKxQw27KpNQjKR9KbiUUQkSdz9UeDRdsfubvf458DPk5krVW3fU8eQvkce7wiQkxllSGGOWh5FeoC6rUVEJCXtqKpjUGHnd4wZWZzLRrU8inSbikcREUk57k5ZVT2DDrOndXuji/NYr+JRpNtUPIqISMrZXdtIQ3MLA7tQPI4qzmPn3not1yPSTSoeRUQk5ZRV1wN0qdt6VHFsSZ8NGvco0i0qHkNu1apVQUeQFPLyyy8HHUEkKXZU1QF0qdu6tXjUuEeR7lHxGGLz58/nmmuuYf78+UFHkRRw5513ctNNN3HnnXcGHUUk4fYXjwVd67YGNO5RpJtUPIZUfX39/iLgzjvvpL6+PuBEEma1tbU89NBDADz00EPU1uqXo6S31m7rgV3ots7PzmBAfpa6rUW6ScVjSP3oRz+iubkZgObmZn784x8HnEjC7Nprrz3sY5F0s6Oqjr59MsnJjHbpdaOK87TWo0g3qXgModLSUl588cUDjr3wwgusXbs2oEQSZkuWLGHdunUHHFu3bh1vvPFGQIlEEq+razy2GqW1HkW6TcVjCN13330dHr/33nuTnERSwa9//esOj99xxx1JTiKSPDu6uMZjq9HFeWzdU0ddY3MCUon0DioeQ+iqq67q8PgXvvCFJCeRVDB79uwOj99www1JTiKSPGVVdQzswmSZVq0zrjdVqPVR5GipeAyhsWPHMn369AOOnX322YwdOzagRBJmU6dOZcyYMQccGzNmDFOnTg0okUhitbQ45XvruzRZplXrjOsN6roWOWoqHkPq5ptvJhqNDQSPRqP867/+a8CJJMzuuuuuwz4WSSdVdY00Njsl+V0vHkfHWx41aUbk6Kl4DKns7GwuvfRSAC699FKys7v+ISm9R25uLlOmTAHgpJNOIjc3N+BEIomzc29smZ7i/Kwuv7ZfbhaFORlqeRTpBhWPIdXU1LR/xvVLL71EU5P2YpVDq6+v5+233wZg2bJlWhdU0trOvQ0ADDiKlkeA0QO0XI9Id6h4DKn58+dTWVkJQEVFhXaZkcPSuqDSm7S2PB5t8TiyKFctjyLdoOIxhHbt2sXcuXOpq4ttv1VXV8fcuXOpqKgIOJmEkdYFld5mV7zl8Wi6rSG2XM+W3ftobG7pyVgivYaKxxB66qmnaGk58EOtpaWFJ598MqBEEmZaF1R6m11764kY9M89uuJxZHEuzS3O1t37ejiZSO+g4jGEZsyYQSRy4F9NJBJhxowZASWSMNO6oNLblO9toCgvi2jEjur1o4piE8rUdS1ydDKCDiAHKyoq4qqrrtrfdZ2Tk8NVV11FUVFR0NEkYAsWLGDhwoUHHS8sLKSqquqAx7fffvsB58ycOZNZs2YlPKNIou3aW3/U4x2hzVqPWihc5KioeAypSy+9lPnz57N161aKior2L9sjvdvChQspXbOGcWNGHnB89IihwNADjnnTezOu3123EUDFo6SFnXvrj3q8I8DAgmyyMyJs1IxrkaOi4jGkMjIyuPHGG7nhhhu48cYbycjQX5XEjBszkl/+6KYuveYbN9+WoDQiyberpoEp/fsd9esjEWNkUS7r1W0tclRUkYTY5MmTmTdvHiUlJUFHEREJjZ3V3eu2hljX9UYVjyJHRRNmQk6Fo4jIe/Y1NFPT0NytbmuAUcW5bKyoxd17KJlI76HiUUREUkbrAuFHs691W6OKc9nX2Ex5tXZjEukqFY8iIpIydtV0b4HwViNbl+vRjGuRLlPxKCIiKaOiJtZSWJTX3W7r2HI963dqxrVIV6l4FBGRlFFZ0wgc/e4yrYb160M0YmxUy6NIl6l4FBGRlFFZG+u27m7xmJURYWi/HO0yI3IUVDyKiEjK2F3bSMSgIKf7K82NKsrTmEeRo6DiUUQkSczsIjN7x8xKzezGDp4/38z2mNmb8a+bg8gZZpW1DfTLzSJylPtatzWyOJcN2mVGpMu0SLiISBKYWRS4C3g/sBlYZGYPu/uKdqc+7+4XJz1githd20i/3Mweudaoolx21zayZ18jffv0zDVFegO1PIZceXl50BFEpGecBpS6+1p3bwDuBy4JOFPKqaxt6PZ4x1atM66104xI16h4DLGlS5dyxRVXsHTp0qCjiEj3DQM2tXm8OX6svTPN7C0z+4eZHZ+caKmjsraR/j3V8ljcutajuq5FukLFY0g1NTXxgx/8AIAf/OAHNDU1BZxIRLqpo0F67ffGex0Y5e5TgF8Df+vwQmZXm9liM1vc23ondsfHPPaE/QuFq+VRpEtUPIbUvHnzqKqqAqCqqop58+YFnEhEumkzMKLN4+HA1rYnuHuVu++Nf/8okGlmA9pfyN3nuPs0d59WUlKSyMyhE+u27pmWx7zsDAbkZ2vSjEgXJbR4NLP1ZrYsPmtwcSLvlU527drFnDlzDjg2Z84cKioqAkokIj1gETDBzMaYWRZwJfBw2xPMbLCZWfz704h9Ru9KetKQqmtspq6xpcdaHiHWda2WR5GuScZs6wvcfWcS7pM2HnjgAdwP7M1ydx588EGuueaagFJJsixYsICFCxd2+FxpaSl4C9+4+bYuXfPddRvAIsyePbvD52fOnMmsWbO6nFU6z92bzOw64HEgCtzr7svN7Jr483cDlwNfMbMmYB9wpbf/MOjFemqB8LZGFefy8ruqz0W6Qkv1hND69es7PL527drkBpFALFy4kNJ3VjJmaPFBz40ZVAhAy749XbrmmMH9AGiuLjvouXVbY784VTwmXrwr+tF2x+5u8/2dwJ3JzpUq3tuasOeW1RlVlMf8N7ZQ19hMTma0x64rks4SXTw68E8zc+C37j7nSC8QuOaaa3jttdc6PC69w5ihxdz25eQs9XfTb/+elPuIdNfueMtjT3dbu8PmylrGDyzoseuKpLNET5iZ7u4nAx8ErjWzc9uf0JtnDR7K2LFjOeussw44Nn36dMaOHRtQIhGR4FXWxlse83qu5XFkfLme9Ts17lGksxJaPLr71vifZcB8Yovktj+n184aPJzWZXpaff/73w8oiYhIOCRkzGPrcj3a41qk0xJWPJpZnpkVtH4PfAB4O1H3SzePPPIIGRmxUQUZGRk88sgjAScSEQnWe93WPdfyWJSXRX52Bhu1XI9IpyWy5XEQ8IKZvQW8Bjzi7o8l8H5pY9euXcydO3f/wuBNTU3MnTtXS/WISK9WWdtIblaU7Iyem9hiZowZkMfanSoeRTorYcVjfP/WKfGv49393xN1r3Tz1FNP0dLScsCxlpYWnnzyyYASiYgEryf3tW5rXEke75bt7fHriqQr7TATQjNmzCASOfCvJhKJMGPGjIASiYgEb3dtY492WbcaPzCfrXvqqKnXNrAinaHiMWQWLFjArbfeSlFREfGNJjAzioqKuPXWW1mwYEHACUVEgpG4lsd8ANap61qkU7RIeMgsXLiQNWtKGTl6DIX9ig54bs2aUkCLOYtI77SntpFh/fr0+HXHD4wVj6VlezlhWN8ev75IulHxGEIjR4/hplt/ctDx2275XgBpRETCobK2ISHd1iOLc4lGjHfLNe5RpDPUbS0iIqHX0uLs2deYkG7r7IwoI4tyKdWkGZFOUfEoIiKhV1XXSIv37NaEbY0ryVfLo0gnqXgUEZHQ2781YQK6rQHGDcxj/c5amppbjnyySC+n4lFEREIvEVsTtjWuJJ+G5hY2Ve5LyPVF0omKRxERCb1EbE3YVuuMay0WLnJkKh5FRCT0Kmtau60T1/IIUKpxjyJHpOJRRERCL9Hd1n37ZFJSkK2WR5FOUPEoIiKht7u2kYhBQU7iliceV5KnGdcinaDiUUREQi+2QHgWkYgl7B7jSvIpLduLuyfsHiLpQMWjiIiE3u7axoRNlmk1fmA+VXVN7NzbkND7iKQ6FY8iIhJ6lbUNCRvv2GrCwAIAVu+oTuh9RFKdikcREQm9ytrGhC0Q3urYIbHicdV2FY8ih6PiUUREQm93fMxjIg3Iz2ZAfjartlUl9D4iqU7Fo4iIhF6s2zqxLY8Axw0pUMujyBGoeBQRkVCra2ymrrEl4S2PAMcOLmD1jmrtcS1yGIlbMEukl1uwYAELFy7s8utKS0vx5kZu+u3fE5DqYGu37sKiVcyePbvLr505cyazZs1KQCqR9yR6gfC2jh1cSH1TC+t31TA+PoFGRA6k4lEkQRYuXMialcsYXZzTpdeNKgTIoGXfnoTkam90/wzAaSxb06XXrd9VB6DiURLuva0JE99t3TppZuW2ahWPIoeg4lEkgUYX53DrrFFBx0iIWxZsCDqC9BK74y2Pyei2Hj8wn8yosWJbFbOmDE34/URSkcY8iohIqFXWxlse8xLf8pidEeWYQQW8vSU5Lf8iqUjFo4iIhFoyxzwCnDisL0s379E2hSKHoOJRRERC7b1u68S3PAKcMKwve/Y1srlyX1LuJ5JqNOYxYO1n5JaWltLizm23fO+gczesX0vEbP+sWM10FUktZnYRcDsQBe5x958e4rxTgVeAj7v7Q0mMGEqVtY3kZkXJzogm5X4nDusLwLItexhRlJuUe4qkEhWPAVu4cCGr3lnNwGEjASgaPByAmvrGg84dMGQEABV76yjbshHQTFeRVGFmUeAu4P3AZmCRmT3s7is6OO9nwOPJTxlOydjXuq2JgwvIiBjLtuzhQycOSdp9RVKFiscQGDhsJJ+49rtdes2f7/pZgtKISIKcBpS6+1oAM7sfuARY0e6864G/AKcmN1547altTFqXNUBOZpSJgwtYtlmTZkQ6ojGPIiLJMQzY1Obx5vix/cxsGHApcHcSc4VeslseAU4a0Y+3Nu2mpUWTZkTaU/EoIpIc1sGx9pXJr4DvunvzYS9kdrWZLTazxeXl5T2VL7R2J7nlEWDqyP5U1zexpmxvUu8rkgpUPIqIJMdmYESbx8OBre3OmQbcb2brgcuB35jZR9tfyN3nuPs0d59WUlKSoLjhEUTL48kj+wHwxsbKpN5XJBWoeBQRSY5FwAQzG2NmWcCVwMNtT3D3Me4+2t1HAw8BX3X3vyU9aYi0tDh79jUmZWvCtsYMyKN/biavq3gUOYgmzIiIJIG7N5nZdcRmUUeBe919uZldE39e4xw7UFXXSIsnZ2vCtsyMqSP78/rG3Um9r0gqUPGYBO3XcmyrtLSUpuaWLs+eLtuykYpoZP+aj+1pDUiR8HH3R4FH2x3rsGh0988lI1PYJXNrwvZOHtmPp1aVUVnTQP+85BavImHW6eLRzPoAI939nQTmSUsLFy5k+cp3KCwZetBz2f0GkQ3U1B28ruPh5BXH1h7btLP6oOeqymPDqFQ8ikiqq9y/u0zyi7fTxxYD8Nr6Ci48fnDS7y8SVp0qHs1sFvALIAsYY2YnAT9y948kMFtaKSwZyplXXJOUe708T71fIolkZn8B7gX+4e4tQedJZ7uTvK91W5OH9yU7I8Kra1U8irTV2QkzPyS2wO1uAHd/ExidiEAiIingv4BPAmvM7KdmdmzQgdJVZU282zrJE2YAsjOinDKqP6+s3ZX0e4uEWWeLxyZ311L7IiKAuy90908BJwPrgSfM7CUz+7yZJb/KSWNBdlsDnD6mmJXbq9hT27WhRSLprLPF49tm9kkgamYTzOzXwEsJzCUiEmpmVgx8Dvgi8AZwO7Fi8okAY6Wd3bWNRCNGYU4w8zvPGFuEO7yyTq2PIq06WzxeDxwP1AN/AvYAX0tQJhGRUDOzvwLPA7nALHf/iLs/4O7XA/nBpksvlbUN9O2TiVlHG/Qk3tSR/cnLivL8mvTfyUeks474TzkziwIPu/tM4PuJjyQiEnr3xJfd2c/Mst293t2nBRUqHQWxNWFbWRkRzhxXzLOry3H3wIpYkTA5YstjfI/VWjPrm4Q8IiKp4N86OPZy0lP0AkFsTdjeeceUsKliH+t31QaaQyQsOjuIpA5YZmZPADWtB939hoSkEhEJITMbDAwD+pjZVKC1GaqQWBe29LDK2kaG9csJNMO5x8T2D39udTljBuQFmkUkDDpbPD4S/xIR6c0uJDZJZjjwyzbHq4HvBREo3e2ubeD4oYWBZhhVnMeYAXk8uaqMfzlrdKBZRMKgU8Wju//uaG8QHzO5GNji7hcf7XVEuutw20QmQmlpKd5Yxy0LNiTtnsm0flcdVlV6yC0yEyHobTfjn4W/M7PL3P0vgQXpRWLd1sGvfvSBSYO498V1VNU1UpgTfB6RIHV2h5kJwG3AJGB//4G7j+3Ey2cDK4l164gEZuHChaxZ/gYj85uTcr8RmUAmtOyrT8r9km1kLkA99RsWJ+V+G/dGgWC33TSzT7v7H4HRZvaN9s+7+y87eJkcpbrGZuoaWwJb47GtDxw/iN8+t5Zn3innI1MO3mpWpDfpbLf1fcAtwH8CFwCf572xPodkZsOBDwP/Dhz0QSuSbCPzm/neyVVBx5Cj8JPXQ/Hvz9YBb1qOJwkqA9yasL2TRvRnQH42jy/fruJRer3OFo993P1JMzN33wD80MyeJ1ZQHs6vgO8ABd3IKCISCu7+2/iftwadpTcIcmvC9qIR4wPHD2L+61uoqW8iLzuYRctFwqCzi4TXmVmE2D6u15nZpcDAw73AzC4Gytx9yRHOu9rMFpvZ4vJyLcIqIuFnZv9hZoVmlmlmT5rZTjP7dNC50s3ugLcmbO+SKUPZ19jMEyt2BB1FJFCdLR6/RmwZihuAU4BPA/9yhNdMBz5iZuuB+4H3mdkf25/k7nPcfZq7TyspKelsbhGRIH3A3auAi4HNwDHAt4ONlH4q4/tJ988LvuUR4NTRRQztm8Pf3twSdBSRQB22eDSzfwK4+yLgenff7O6fd/fL3P2Vw73W3W9y9+HuPhq4EnjK3fUvcxFJB63VzIeAP7t7RZBh0lWYxjwCRCLGJVOH8fyanZRV1wUdRyQwR2p5bNsUeEUig4iIpJAFZrYKmAY8aWYlxDZTkB60Z1+s5THI7Qnbu+KU4TS3OA8u2hR0FJHAHKl49J64ibs/ozUeRSRduPuNwJnANHdvJLbz1iXBpko/lTUN5GZFyc6IBh1lv7El+Zw9fgB/enUjzS098itSJOUcabrYWDN7mNiyPK3f7+fuH0lYMhGRcDuO2HqPbT9Hfx9UmHRUWdsYmi7rtj59xiiu+eMSnlixg4tOGBx0HJGkO1Lx2PZf0r9IZBARkVRhZn8AxgFvAq2rzjsqHnvU7tqGUHVZt5p53EBGFedy19OlXHj8IMyOuOyxSFo5bPHo7s8mK4iISAqZBkxyd/VbJlBsa8LwtTxmRCN89fxxfPcvy3hmdTkXTDzsynUiaaez2xMu4+Dxj3uI7Vn9b+6+q6eDiYiE2NvAYGBb0EHS2e7aRob26xN0jA5dOnU4v36qlJ/9YxXnjB9ARrRzK9+5O397cwsPLdnM2vIaivKy+OTpI/nYtBFkdvIaIkHr7Dv1H8AjwKfiXwuA54HtwP8kJJmISHgNAFaY2eNm9nDrV9Ch0k1YWx4BsjIi/ODDx7FqezW/e3lDp15TXl3PF3+3mK8/8Bbbdtdx5rhiohHj+/Pf5ku/X0xdY/ORLyISAp3dX2m6u09v83iZmb3o7tO1q4KI9EI/DDpAumtpcfbsawzF1oSHcuHxgzl/Ygm/ePwdzhpXzHFDDr3/+uPLt3PTX5ext76Jmy+exOfOGk0kYrg7//vqRn7wt7f55oNvcecnp2oMpYReZ1se883s9NYHZnYakB9/2NTjqUREQiw+Hnw9kBn/fhHweqCh0kxVXSMtHp6tCTtiZvzHZZMpyMng6j8sZuvufQedU13XyLfnvcWX/7CEIX1zeOT6s/nC2WOIRGz/NT59xii+feFEHlm2jb8v1UgICb/OFo9fBO4xs3Xx7QbvAb5oZnnAbYkKJyISRmb2JeAh4LfxQ8OAvwUWKA2FbWvCQxlYmMNvP3MKlTWNXHLXi/x96Vbqm5rZXdvAn17dyAf+8zn+8vpmrrtgPPO/Op0Jgwo6vM6Xzx3LlOF9+eHDy6mua0zyTyHSNZ3qto5vT3iimfUFzN13t3n6wUQEExEJsWuB04BXAdx9jZlpym0Pat2aMMwtj62mjuzPX796Fl/54xKu+9MbBzw3ZUQ/7vzkyZwyqv9hr5ERjfCjS07gkrte5Pcvb+DaC8YnMrJIt3R2tnVf4Bbg3PjjZ4EfufueBGYLnQULFrBw4cIuv660tJT6xmZennd3AlIdrKp8K6W7o8yePbvLr505cyazZs1KQCqRtFLv7g2tY9PiC4Vr2Z4etDtk+1ofyTGDCvjn18/jmXfKWLG1ikjEOGNsESeP7N/pMYxTRvTjvGNKmPvCOj4/fTS5WZ2dliCSXJ19Z95LbGmKj8Uffwa4D/g/iQgVVgsXLmTpilVk9e1iA0OfIqJ9oKY+OV0R0cISmoBVWyq69LqGPWUAKh5FjuxZM/se0MfM3g98ldgqFNJDKmvi3dYhnjDTXjRizDhuEDOOG3TU17j+feO5/O6X+cuSzXzmzNE9F06kB3W2eBzn7pe1eXyrmb2ZgDyhl9V3IIPP+0TQMRJi+7N/DjqCSKq4EbgKWAZ8GXiU2Fhw6SGp1G3dk04Z1Z/jhhTy59c28ekzRmnmtYRSZyfM7DOzs1sfmNl04OBpZSIivYC7txCbIPNVd7/c3f+7M7vNmNlFZvaOmZWa2Y0dPH+JmS01szfNbHHbz93eZndtI9GIUZjTu7puzYxPnjaCFduqWLalV40MkxTS2eLxGuAuM1sfn219J7F/bYuI9BoW80Mz2wmsAt4xs3Izu7kTr40CdwEfBCYBnzCzSe1OexKY4u4nAV+gF7dmVtY20K9PZq9sebtk6jByMiM8uHhT0FFEOtSp4tHd33L3KcBkYLK7TwXel9BkIiLh8zVgOnCquxe7exFwOjDdzL5+hNeeBpS6+1p3bwDuBy5pe4K7723TgplHL56Es7u2kX4pNN6xJxXmZDLjuEH8Y9l2mppbgo4jcpAubaTp7lXuXhV/+I0E5BERCbPPAp9w93WtB9x9LfDp+HOHMwxo25S0OX7sAGZ2qZmtIrYl7Be6nThFVdQ0UJyXHXSMwMyaPIRdNQ28srZrEx9FkqE7u7D3vr4EEentMt19Z/uD7l4OHKmZrKPPzINaFt19vrsfC3wU+HGHFzK7Oj4mcnF5efmRU6egipqG0C8QnkjnTxxIXlaUvy/dGnQUkYN0p3jstd0pItJrNRzlcxBraRzR5vFw4JCVgbs/B4wzswEdPDfH3ae5+7SSkpIj3DY1VdQ2UJTXu2Zat5WTGWXGcYN4fPl2mlv061bC5bDFo5lVm1lVB1/VwNAkZRQRCYsph/lMPPEIr10ETDCzMWaWBVwJPNz2BDMbb/EZImZ2MpAF7ErAzxFq7k5lTe8uHgFmHDeQytpG3tq8O+goIgc47BoI7t7xJpwiIr2Qu0e78domM7sOeByIAve6+3Izuyb+/N3AZcBnzayR2HJoH+/MEkDppqquiaYWT5ndZRLlvGNKiBg8vaqMk0cefntDkWTqXQtoiYgEyN0fJbageNtjd7f5/mfAz5KdK2wqa2IjAHp7y2O/3CxOGdWfp1aV8c0PTAw6jsh+3RnzKCIi0uN2qXjc7/yJA1m+tYodVXVBRxHZT8WjiIiEiloe3/O+YwcCsa5rkbBQ8SgiIqFSEd/XurePeQQ4dnABQ/rm8JSKRwkRFY8iIhIqFfGWx+J8FY9mxgXHDuSF0p3UNzUHHUcEUPEoIiIhU1nTQHZGhD6ZRz25Pa28b+JAahuaeW2ddpuRcFDxKCIioVIRX+MxvuRlr3fW+GKyohGefSc9dxOS1KPiUUREQqWipkHjHdvIzcrg1DH9eX7NQTtjigRCxaOIiIRKRW2Dxju2c86EEt7ZUc32PVqyR4Kn4lFEREKlUi2PBzl3QmwP8+fWqOtagqfiUUREQmWX9rU+yHFDCigpyOa51SoeJXgqHkVEJDQam1uormtS8diOmXHOhAG8ULqT5pZet925hEzK7229YMECFi5cmJR7lZaW0lDfxPZn/5yU+yVbw+4ySvdVMHv27KTdc+bMmcyaNStp9xORcKtsXSBcxeNBzp1Qwl9f38LbW/YwZUS/oONIL5byxePChQt58+2VNOcWJf5mkQLoA431jYm/VxD69GcPsGTtjqTcLlobW7NMxaOItGpdILxIYx4PcvaEAQA8t7pcxaMEKuWLR4Dm3CL2HfuhoGNIF/VZ9WjS77mjqp5n1muXhlS0o6qekUGHkISr0L7WhzQgP5sThhXy/JqdXD9jQtBxpBfTmEcREQmNyppYz46Kx46dM6GE1zdWUl2Xpj1gkhLSouVRpLMGFWZz/uj6oGPIUXipIjvoCJIEFTWx/z/752UGnCSczp1Qwn898y4vvbuLC48fHHQc6aXU8igiIqFREW951DqPHTtlVH/ysqI8r/UeJUAqHkVEJDQqaxsozMkgM6pfTx3Jyohw5rhinlutrQolOPq/U0REQkMLhB/ZORNK2FhRy/qdNUFHkV5KxaOIiIRGpYrHIzr3GG1VKMFS8SgiIqGhlscjG12cy4iiPuq6lsCoeBQRkdCorGnQZJkjiG1VWMLL7+6koakl6DjSC6l4FBGRUHB3KmrV8tgZ504ooaahmdc3VgYdRXohFY8iIhIKNQ3NNDS1qHjshLPGFxONGM+8o3GPknwJKx7NLMfMXjOzt8xsuZndmqh7iYhI6quMb03YX8XjERXmZHL6mCIWrtwRdBTphRLZ8lgPvM/dpwAnAReZ2RkJvJ+IiKSwXfHisVjFY6d8YNIgSsv2srZ8b9BRpJdJWPHoMa3v6Mz4lyfqfiIiktp2Vse2JhyQr60oO2PmpEEAPLFCrY+SXAkd82hmUTN7EygDnnD3Vzs452ozW2xmi8vLNXZDRKS32rk3XjwWqHjsjOH9czl+aCGPL98edBTpZRJaPLp7s7ufBAwHTjOzEzo4Z467T3P3aSUlJYmMIyIiIdZaPKrbuvM+dOIQXt+4my279wUdRXqRpMy2dvfdwDPARcm4n4iIpJ6dexsoyMkgJzMadJSUcfHkIQA8snRrwEmkN0nkbOsSM+sX/74PMBNYlaj7iYhIaivfW0+Jxjt2yajiPKaM6MeCt7YFHUV6kUS2PA4BnjazpcAiYmMe/57A+4mIhJqZXWRm75hZqZnd2MHznzKzpfGvl8xsShA5g7Kzul6TZY7CrMlDWLZlD2t2VAcdRXqJRM62XuruU919sruf4O4/StS9RETCzsyiwF3AB4FJwCfMbFK709YB57n7ZODHwJzkpgzWzr31DCjQeMeu+ujUYWRGjQcWbQo6ivQS2mFGRCQ5TgNK3X2tuzcA9wOXtD3B3V9y99b95l4hNtmw19i5t0Etj0dhQH42M48bxF/f2EJ9U3PQcaQXUPEoIpIcw4C2TUOb48cO5SrgHx09kY5LnDU0tbBnX6OKx6P08VNHUFHTwGNva9keSTwVjyIiyWEdHOtw4wQzu4BY8fjdjp5PxyXOdtVogfDuOHdCCWMH5PHfz6/FXftxSGKpeBQRSY7NwIg2j4cDB62vYmaTgXuAS9x9V5KyBW5ndWxrwgH5GvN4NCIR44vnjOXtLVW8/G6vedtIQFQ8iogkxyJggpmNMbMs4Erg4bYnmNlI4K/AZ9x9dQAZA6PdZbrv/5w8jAH52fzqyTVqfZSEUvEoIpIE7t4EXAc8DqwEHnT35WZ2jZldEz/tZqAY+I2ZvWlmiwOKm3Tl8eJR6zwevZzMKLNnTuC1dRUsXFkWdBxJYxlBBxAR6S3c/VHg0XbH7m7z/ReBLyY7VxiUV2vMY0+48tQR3PfiOn7y6ErOHj+APlnarUd6nloeRUQkcOXV9RTkZKjY6abMaIQfX3IC63bW8LPHtKmbJIaKRxERCdyOqjoGarxjj5g+fgCfnz6a/3lpPX97Y0vQcSQNqdtaREQCV1Zdz6DCnKBjpI3vXnQsK7dV8a15b5ERNS6ePDToSJJG1PIoIiKB21FVp+KxB+VkRpnz2WlMHt6X6/70Bv/6t7f3z2gX6S61PIqISKDcnbLqenVb97DCnEz+fPUZ3PboKv7wygbuX7SR08YUMWFgAVkZERqaWtjX0ExlbUP8q5H6pmYKczIZMyCPs8YN4KITBlOUp7U35UAqHkVEJFB79jXS0NTCQLU89rjsjCg//MjxfPbMUTywaBPPr9nJvMWbaGpxsjMi5GRG6Z+bRf+8TCYMzCcnM8ru2gaWbKjk70u38eO/r+CzZ41i9owJ5GapZJAYvRNERCRQO6pi3alqeUycsSX53PSh47ipk+e7Oyu2VXHP8+v47bNr+efyHdz96VOYOLggoTklNWjMo4iIBKqsug5AYx5DxMw4fmhf/vPjJ/HnL51BTX0Tl9/9Eq+tqwg6moSAikcREQlUa8vjoEK1PIbRmeOKmX/tdAYWZPOF/1nE21v2BB1JAqZua+lVNu6N8pPXC4OOIUdh494oE4IOIQmxoyrW8jiwQC2PYTWsXx/++MXTuew3L/GF/1nEIzecQ4mGGfRaKh6l15g5c2ZS71daWoo37mN0cXr+Qly/qw7L7MP48eOTcr8JJP/vUJJDu8ukhiF9+zD3c6dy6W9e5IY/v8Efv3g60YgFHUsCoOJReo1Zs2Yxa9aspN1v9uzZNJat4dZZo5J2z2S6ZcEGMgeO5/bbbw86iqQ47S6TOo4bUsiPLzmBbz+0lLkvrOXqc8cFHUkCoDGPIiISqO1aIDylXH7KcC48fhC/+OdqSsuqg44jAVDxKCIigdq2u46h/foEHUM6ycz4t4+eSJ/MKLcuWIG7Bx1JkkzFo4iIBKapuYWy6jqG9lXLYyopKcjm6zMn8PyanTyxYkfQcSTJVDyKiEhgdlTX0+IwRC2PKedTZ4xiwsB8/u2RldQ1NgcdR5IoLSbM1FVX0rhlRdAxpIusuhIYFHQMEQnQ1t37ABiilseUkxmNcMus4/n03FeZ+8I6rr0gOSsvSPDU8igiIoFpLR415jE1nT1hAO+fNIjfPF1KRU1D0HEkSdKi5TGnoD8+bFLQMaSLcqrXBx1BRAK2bU9sgXC1PKau71w4kQ+s3MFvn32Xmz50XNBxJAnU8igiIoHZtnsfBTkZFORkBh1FjtKEQQV89KRh/O7l9fv3KZf0puJRREQCs3VPHUP7qss61c2eMYHGZuc3T78bdBRJAhWPIiISmG179jGkn7qsU93oAXlcfvJw/vTqxv3jWCV9qXgUEZHAbNtdxxC1PKaF62eMx3HufLo06CiSYCoeRUQkEPsamtlV08AwtTymheH9c7ny1JE8uGgTmypqg44jCaTiUUREArG5MlZgjCjKDTiJ9JRrLxhPJGLc/uSaoKNIAql4FBGRQGysUPGYbgb3zeEzZ4zir69vprSsOug4kiAqHkVEJBCtXZsjVTymla+eP44+mVF++cTqoKNIgqh4FBGRQGys2EduVpTivKygo0gPKs7P5qpzxvLosu0s27wn6DiSACoeRUQkEBsrahnRPxczCzqK9LAvnjOGfrmZ/Pyf7wQdRRJAxaOIiARic2WtxjumqcKcTL56/jieW13OK2t3BR1Helha7G0dra2gz6pHg44hXRStrQAGBR0jodbvquOWBRuCjpEQ63fVMWFg0ClSi5ldBNwORIF73P2n7Z4/FrgPOBn4vrv/Ivkpk8Pd2VhRy1njBgQdRRLks2eOZu4L6/j54+/w0DVnqoU5jaR88Thz5syk3au0tJSa2losmp7jc7y5gbzcXMaPH5+kOw5K6t9fsh3tz1ZaWsq+2lpyspOz129dfSN9juLvfcLA5P7/l+rMLArcBbwf2AwsMrOH3X1Fm9MqgBuAjyY/YXLtqmmgtqGZkUVaIDxd5WRGuWHGBL4//22eWlXGjOPSu7GgN0n54nHWrFnMmjUrKfdasGABCxcu7PLrSktLqW9sprBkaAJSHayqfCvZmdGjKgJnzpyZtP+e6e5o35uHe5+VlpaCtzBuzKguXfPddRvAIod8T+jvPSlOA0rdfS2Amd0PXALsLx7dvQwoM7MPBxMxebRMT+/wsWkjuOf5dfz7Iys5e8IAsjOiQUeSHpDyxWMyHW0xMHv2bDbtrObMK65JQKqDvTzvbkYMKOD2229Pyv2kZx3ufTZ79my8qZ5f/uimLl3zGzffhmVk6z0RrGHApjaPNwOnB5QlcOt31gAwqljFYzrLjEa4ZdYkPnffIua+sI6vnp+sni1JJBWPSVJVvpWX592dtHsxYGJS7iXJ9+66jXzj5tu6/JrxEyYkKJF0UkcDvvyoLmR2NXA1wMiRI7uTKTBry2vIiBijivOCjiIJdv7EgVx4/CB+/WQpl5w0jGH9NFQh1al4TILDjQsrLS2ltnYfWdnZXbpmQ309ubl9Ou6GHDBRY9HSVEd/r2+99dYhz58yZQoA4ydM0HsieJuBEW0eDwe2Hs2F3H0OMAdg2rRpR1WABu3d8r2MLMolM6pFP3qDf714EjN/+Sw/XrCCuz9zStBxpJtUPCbB4boh249vKy0tpcWdUaPHHnTuhvVriZjtLxg1Tq336ei99K1vfYvFixcfdO6pp57Kz3/+82RFkyNbBEwwszHAFuBK4JPBRgrOu+V7GVuSH3QMSZLh/XO5/n0T+Pnj7/DI0m18ePKQoCNJNySseDSzEcDvgcFACzDH3TXgqp32xcDs2bNZs6a0w3ONWOGocWvS1oknnthh8XjCCScEkEYOxd2bzOw64HFiS/Xc6+7Lzeya+PN3m9lgYDFQCLSY2deASe5eFVTuRGhucdbvrOWCY7XWU29y9blj+eeKHdz016VMHdmPoeq+TlmJbHlsAr7p7q+bWQGwxMyeaLcshbTT2rW4fPlympqa9h/PyMjg+OOPV9ejHGTWrFncd999HR6XcHH3R4FH2x27u83324l1Z6e1zZW1NDS3MG6AWh57k8xohNs/fhIfvuN5vv7Am/zpS2cQjWjtx1SUsMEm7r7N3V+Pf18NrCQ221AOY9asWXz0ox89oHAEaGpq4rLLLlNBIAcpKiriK1/5ygHHvvKVr1BUVBRQIpHDW1sem2k9bqAmy/Q2owfk8cOPHM+r6yq46+mOe9kk/JIyUtnMRgNTgVeTcb9Ud8cdd3R4/Fe/+lVyg0jKuOyyyyguLgaguLiYyy67LOBEIof2bvleAMaq5bFXuvyU4Xz0pKH88onVPLJ0W9Bx5CgkvHg0s3zgL8DXOhq3Y2ZXm9liM1tcXl6e6Dgp4etf/3qHx7/2ta8lN4ikjIyMDG655RYAbrnlFjIyNBdOwmv1jmqK87Lon5eeu3XJ4ZkZP71sMtNG9efrD77Jkg2VQUeSLkpo8WhmmcQKx/919792dI67z3H3ae4+raSkJJFxUsa55557UJdjUVER5557bkCJJBVMnjyZefPmMXny5KCjiBzWym3VTBpaGHQMCVBOZpQ5n53G0L45fOn3iyktqw46knRBwopHi+2APhdY6e6/TNR90tXvfve7wz4W6Yj+ASZh19Tcwjs7qjluiIrH3q4oL4v7Pn8aETM+/ttXWLU9rRYVSGuJbHmcDnwGeJ+ZvRn/+lAC75dWCgoKOO+88wA4//zzKSgoCDiRiEj3rd1ZQ0NTC5NUPAowZkAeD375DDKjEa6c8wpvb9kTdCTphIQNjHL3F+h4Oy7ppFtvvZXHHnuMiy66KOgoIiI9YsXWWOuSWh6l1diSfB748hl88r9f5eO/fZnbr5zKzEmDgo4lh6F9oUJOhaOIpJOV26rIikYYW6JleuQ9o4rz+MtXzmJsST5f+sNi/vu5tbin5M6bvYKKx5DTDHQRSScrtlVxzOB87WktBxncN4cHv3wmHzxhMP/+6Epu/MsyGppago4lHdD/vSG2dOlSrrjiCpYuXRp0FBGRbmtpcZZu3sMJQ/sGHUVCqk9WlDs/cTLXv288DyzexCf++xW27dkXdCxpR8VjSDU1NXHbbbcB8NOf/vSgHWdERFLN2p172bOvkZNH9Q86ioRYJGJ88wMT+fUnprJqWxUfvuMFnlutXrgwUfEYUvPnz6eyMrZwakVFBfPnzw84kYhI97QuBn2KikfphFlThvLw9WdTkp/Nv9z3Gr98YjXNLRoHGQYqHkNo165dzJ07l7q6OgDq6uqYO3cuFRUVAScTETl6SzZU0i83k7EDNFlGOmdcST5/u3Y6l508nDueXMNn732V8ur6oGP1eioeQ+ipp56ipeXAQcItLS08+eSTASUSEem+JRsqOWVkf2J7SIh0Tp+sKL+4Ygr/cflkFq+v5IO3P8c/l28POlavpuIxhGbMmEEkcuBfTSQSYcaMGQElEhHpnsqaBt4tr9F4RzlqH5s2ggXXn82gwhyu/sMSvjXvLarqGoOO1SupeAyhoqIirrrqKnJycgDIycnhqquuOmi/axGRVPFC6U4AzhxXHHASSWXHDCpg/lenc/37xjP/jS188FfP88w7ZUHH6nVUPIbUpZdeur9YLCoq4tJLLw04kYjI0Xt2dTl9+2QyZXi/oKNIisvKiPDND0zkoWvOJDszwufuW8SX/7CYzZW1QUfrNVQ8hlRGRgY33ngjADfeeCMZGQnbSVJEJKHcnedWl3P2hAFEIxrvKD1j6sj+/GP2OXz7wok8t3onM/7vs/z47yvYuVcTahJNxaOIiCTUqu3VlFXXc94xJUFHkTSTnRHl2gvGs/Cb53Hx5KHc9+I6zvnZ0/zssVVU1jQEHS9tqXgMKS0SLiLp4okVOzBDxaMkzLB+ffi/H5vCE984j/dPGsTdz77L2T97ip88upKyqrqg46UdFY8hpUXCRSQduDt/e3MLp40uYlBhTtBxJM2NK8nnjk9M5fGvncvMSYO45/m1nP2zp7npr8vYsKsm6HhpQ8VjCGmRcBFJF8u3VrG2vIZLThoWdBTpRY4ZVMDtV07l6W+dz+XThvOXJZu54BfPcP2f32Dltqqg46U8FY8hpEXCRSRd/O2NLWRGjQ+eMDjoKNILjSrO4yeXnsgL372AL50zlqdW7uCDtz/PF/5nEW9t2h10vJSl4jGEtEi4iKSDmvomHly8ifdPGkT/vKyg40gvNrAwh5s+dBwv3TiDb77/GN7YWMkld73I7PvfYFOFlvjpKhWPIaRFwkUkHTy0ZDNVdU1cdfbYoKOIANA3N5PrZ0zgue9cwLUXjOOxt7cz45fPctujK9mzT7vVdJaKx5DSIuEiksoamlqY+8I6po7sxynaklBCpiAnk29feCxPf+t8Zk0eypzn1/K+XzzD/766geYWDzpe6Kl4DCktEi4iqex3L61nY0UtN8yYEHQUkUMaGl/iZ8F1ZzOuJJ/vz3+bD9/xPC/Ft9OUjql4DLHJkyczb948Jk+eHHQUEZFO21FVxx1PruH8iSVcMHFg0HFEjuiEYX154Mtn8JtPncze+iY+ec+rfOn3iyktqw46WiipeAy5khItqisiqaOpuYUb/vwGTS3OzRdPCjqOSKeZGR86cQgLv3Ee37loIi+V7uT9//kc1/7pdVZs1fI+bakvVEREeoS7c+uCFby6roL/e8UUxpbkBx1JpMtyMqN89fzxXHnqSOa+sJbfvbSBR5Zu46QR/fjYtBFcdMJginr56gEqHkVEpNsamlr44YLl/OnVjXz53LFcdsrwoCOJdEtRXhbfvvBYvnTOWP7y+hYeXLSJ781fxvfmL+O4IYWcPqaIcQPzGV2cS0lBNnlZGeRmRcmIRIhEIGJGNGJEzIgYRCOGmQX9Y/UIFY8iIkliZhcBtwNR4B53/2m75y3+/IeAWuBz7v560oN20ZubdnPz/3ubpZv38JXzx/GdCycGHUmkx/TLzeKqs8fwhemjeXtLFc+tKeeld3fywKJN7Gts7tK1zCAzGmFgQTaDCnMYXJjD0H45TBhYwPhB+YwfmE9hTmaCfpKeo+JRRCQJzCwK3AW8H9gMLDKzh919RZvTPghMiH+dDvxX/M/Q2bOvkWdXlzNv8SaeX7OTAfnZ/NenTuaDJw4JOppIQpgZJw7vy4nD+3LtBeNpaXHKqutZv6uGipoG9tY3sa+hmaYWx91pbnGa3WlpcVocmlucFnfqm1oor65n+546Vm6vYuHKHdQ3vber3JC+OYwfmM+EgQVMGJTPhIGxorJfbni6ylU8iogkx2lAqbuvBTCz+4FLgLbF4yXA793dgVfMrJ+ZDXH3bYkI1PoLrin+S62pJfaLrqnFqWtsprquib31TVTXNVJWVc/WPXVs3FXD0i17WFteA8Cgwmy++f5j+Nz00RSkQIuJSE+JRIzBfXMY3DenW9dpbnE2V9ayZsdeVpdVUxr/88+vbTygZbM4L4vh/fswrH8fhvbtw6DCHPr2yaSwTyZ9418FORlkZUTIjEbIjFr8zwjRSM92l6t4FBFJjmHApjaPN3Nwq2JH5wwDeqx4/Mofl/DkyjKaWlro6lrIZjCkMIfjh/Xl0pOGcca4Yk4e2b/HfzGJ9CbRiDGqOI9RxXnMnDRo//GWFmfL7n2Ulu1lTVk175bVsHXPPlZtr+apVWXUNbYc5qoHihh8/8OTuOrsMT2SOVTF45IlS3aa2Yagc4TMAECrlUpn6f3SsVFBBwA6qrDal2+dOQczuxq4Ov5wr5m9081s0Mn3znrg5R64WTekwns8FTJCauRMhYyQAjm/+FP4YtdyHvJzM1TFo7trUcN2zGyxu08LOoekBr1fQm0zMKLN4+HA1qM4B3efA8zpyXCp8t5JhZypkBFSI2cqZITel1OLhIuIJMciYIKZjTGzLOBK4OF25zwMfNZizgD2JGq8o4jI0QpVy6OISLpy9yYzuw54nNhSPfe6+3Izuyb+/N3Ao8SW6SkltlTP54PKKyJyKCoew69Hu6Yk7en9EmLu/iixArHtsbvbfO/AtcnOFZcq751UyJkKGSE1cqZCRuhlOS32WSUiIiIicmQa8ygiIiIinabiMcTM7CIze8fMSs3sxqDzSHiZ2b1mVmZmbwedRVJLKrx3zGyEmT1tZivNbLmZzQ46U0fMLMfMXjOzt+I5bw0606GYWdTM3jCzvwed5VDMbL2ZLTOzN81scdB5DiW+mP9DZrYq/h49M+hMbZnZxPh/w9avKjP7WreuqW7rcIpvZbaaNluZAZ9ot5WZCABmdi6wl9juJCcEnUdSRyq8d8xsCDDE3V83swJgCfDRsH0exvcmz3P3vWaWCbwAzHb3VwKOdhAz+wYwDSh094uDztMRM1sPTHP3UK+faGa/A55393viKynkuvvugGN1KF5bbAFOd/ejXldbLY/htX8rM3dvAFq3MhM5iLs/B1QEnUNSTyq8d9x9m7u/Hv++GlhJbOedUPGYvfGHmfGv0LXQmNlw4MPAPUFnSXVmVgicC8wFcPeGsBaOcTOAd7tTOIKKxzA71DZlIiK9lpmNBqYCrwYcpUPx7uA3gTLgCXcPY85fAd8BOr+/XTAc+KeZLYnvqhRGY4Fy4L74MIB7zCwv6FCHcSXw5+5eRMVjeHVqmzIRkd7CzPKBvwBfc/eqoPN0xN2b3f0kYrsDnWZmoRoKYGYXA2XuviToLJ0w3d1PBj4IXBsfYhE2GcDJwH+5+1SgBgjlHIV4l/pHgHndvZaKx/Dq1DZlIiK9QXwM4V+A/3X3vwad50jiXZfPABcFm+Qg04GPxMcT3g+8z8z+GGykjrn71vifZcB8YsO5wmYzsLlNC/NDxIrJMPog8Lq77+juhVQ8hldntjITEUl78Ykoc4GV7v7LoPMcipmVmFm/+Pd9gJnAqkBDtePuN7n7cHcfTez3ylPu/umAYx3EzPLik6OIdwN/AAjdigDuvh3YZGYT44dmAKGayNXGJ+iBLmtQ8Rha7t4EtG5lthJ40N2XB5tKwsrM/gy8DEw0s81mdlXQmSQ1pMh7ZzrwGWKtZK3LjXwo6FAdGAI8bWZLiTUAPOHuoV0KJ+QGAS+Y2VvAa8Aj7v5YwJkO5Xrgf+N/7ycBPwk2zsHMLJfY6i090mqvpXpEREREpNPU8igiIiIinabiUUREREQ6TcWjiIiIiHSaikcRERER6TQVjyIiIiLSaSoeJSnM7PtmttzMlsaX2Ti9B675ETPrkZX8zWzvkc8SEUm+nvh8MrPzzWxPm6WOFvZENumdMoIOIOnPzM4ELgZOdvd6MxsAZHXytRnxNS8P4u4Po4XTRUQ663l3v7grLzjcZ7D0Xmp5lGQYAux093oAd9/p7lvNbH28kMTMppnZM/Hvf2hmc8zsn8DvzexVMzu+9WJm9oyZnWJmnzOzO82sb/xakfjzuWa2ycwyzWycmT1mZkvM7HkzOzZ+zhgze9nMFpnZj5P830NEpFvM7CQzeyXemzPfzPrHj58aP/aymf3czA65K4uZnWZmL5nZG/E/J8aPf87M5pnZAuCf8d1e7o1/Xr5hZpck6ceUkFLxKMnwT2CEma02s9+Y2XmdeM0pwCXu/kli+69+DMDMhgBD3X1J64nuvgd4C2i97izgcXdvBOYA17v7KcC3gN/Ez7md2Eb2pwLbu/0Tiogk1++B77r7ZGAZcEv8+H3ANe5+JtDc7jXntOm2/j6xrRPPdfepwM0cuDPKmcC/uPv7gO8T28bwVOAC4OfxLQOll1K3tSScu+81s1OAc4h98DzQibGKD7v7vvj3DwJPEPtw/Bgwr4PzHwA+DjxNbL/W35hZPnAWMC+2NS4A2fE/pwOXxb//A/Czrv5cIiJBMLO+QD93fzZ+6HfEPuf6AQXu/lL8+J+IDRlqdUC3tZmNAH5nZhMABzLbnPuEu1fEv/8A8BEz+1b8cQ4wktjWudILqXiUpHD3ZuAZ4BkzWwb8C9DEe63fOe1eUtPmtVvMbJeZTSZWIH65g1s8DNxmZkXEWi2fAvKA3e5+0qFiHd1PIyISSnbkUw7wY+Bpd7/UzEYT+4xuVdPmewMuc/d3uhdP0oW6rSXhzGxi/F+2rU4CNgDriRV68F4r4KHcD3wH6Ovuy9o/6e57gdeIdUf/3d2b3b0KWGdmV8RzmJlNib/kRWItlACf6vIPJSISkPhQnUozOyd+6DPAs+5eCVSb2Rnx41d2eIH39AW2xL//3GHOexy43uJdOGY29aiCS9pQ8SjJkE+sa2SFmS0FJgE/BG4Fbjez5zl4bE57DxH7IHzwMOc8AHw6/merTwFXmdlbwHKgdaD3bOBaM1tE7ANURCSscs1sc5uvbxDrvfl5/DP1JOBH8XOvAuaY2cvEWgz3HOa6/0Gsx+ZFIHqY835MrEt7aXwCjiYZ9nLmrp47ERGRdGBm+fGeGOJjy4e4++yAY0ma0ZhHERGR9PFhM7uJ2O/3DRy+O1rkqKjlUUREREQ6TWMeRURERKTTVDyKiIiISKepeBQRERGRTlPxKCIiIiKdpuJRRERERDpNxaOIiIiIdNr/B4oHLL+rB6ZBAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "df_fare = trn_df[trn_df.LogFare>0]\n", "fig,axs = plt.subplots(1,2, figsize=(11,5))\n", "sns.boxenplot(data=df_fare, x=dep, y=\"LogFare\", ax=axs[0])\n", "sns.kdeplot(data=df_fare, x=\"LogFare\", ax=axs[1]);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The [boxenplot](https://seaborn.pydata.org/generated/seaborn.boxenplot.html) above shows quantiles of `LogFare` for each group of `Survived==0` and `Survived==1`. It shows that the average `LogFare` for passengers that didn't survive is around `2.5`, and for those that did it's around `3.2`. So it seems that people that paid more for their tickets were more likely to get put on a lifeboat.\n", "\n", "Let's create a simple model based on this observation:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:38.68033Z", "iopub.status.busy": "2022-05-23T04:37:38.680007Z", "iopub.status.idle": "2022-05-23T04:37:38.68637Z", "shell.execute_reply": "2022-05-23T04:37:38.685259Z", "shell.execute_reply.started": "2022-05-23T04:37:38.680295Z" } }, "outputs": [], "source": [ "preds = val_xs.LogFare>2.7" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...and test it out:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:40.107202Z", "iopub.status.busy": "2022-05-23T04:37:40.106891Z", "iopub.status.idle": "2022-05-23T04:37:40.114544Z", "shell.execute_reply": "2022-05-23T04:37:40.113805Z", "shell.execute_reply.started": "2022-05-23T04:37:40.107167Z" } }, "outputs": [ { "data": { "text/plain": [ "0.336322869955157" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mean_absolute_error(val_y, preds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is quite a bit less accurate than our model that used `Sex` as the single binary split.\n", "\n", "Ideally, we'd like some way to try more columns and breakpoints more easily. We could create a function that returns how good our model is, in order to more quickly try out a few different splits. We'll create a `score` function to do this. Instead of returning the mean absolute error, we'll calculate a measure of *impurity* -- that is, how much the binary split creates two groups where the rows in a group are each similar to each other, or dissimilar.\n", "\n", "We can measure the similarity of rows inside a group by taking the standard deviation of the dependent variable. If it's higher, then it means the rows are more different to each other. We'll then multiply this by the number of rows, since a bigger group has more impact than a smaller group:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:40.630562Z", "iopub.status.busy": "2022-05-23T04:37:40.62976Z", "iopub.status.idle": "2022-05-23T04:37:40.636571Z", "shell.execute_reply": "2022-05-23T04:37:40.635377Z", "shell.execute_reply.started": "2022-05-23T04:37:40.630514Z" } }, "outputs": [], "source": [ "def _side_score(side, y):\n", " tot = side.sum()\n", " if tot<=1: return 0\n", " return y[side].std()*tot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we've got that written, we can calculate the score for a split by adding up the scores for the \"left hand side\" (lhs) and \"right hand side\" (rhs):" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:41.160896Z", "iopub.status.busy": "2022-05-23T04:37:41.160555Z", "iopub.status.idle": "2022-05-23T04:37:41.166645Z", "shell.execute_reply": "2022-05-23T04:37:41.165803Z", "shell.execute_reply.started": "2022-05-23T04:37:41.160861Z" } }, "outputs": [], "source": [ " \n", "def score(col, y, split):\n", " lhs = col<=split\n", " return (_side_score(lhs,y) + _side_score(~lhs,y))/len(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For instance, here's the impurity score for the split on `Sex`:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:42.530883Z", "iopub.status.busy": "2022-05-23T04:37:42.529881Z", "iopub.status.idle": "2022-05-23T04:37:42.542981Z", "shell.execute_reply": "2022-05-23T04:37:42.541924Z", "shell.execute_reply.started": "2022-05-23T04:37:42.530826Z" } }, "outputs": [ { "data": { "text/plain": [ "0.40787530982063946" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "score(trn_xs[\"Sex\"], trn_y, 0.5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...and for `LogFare`:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:43.657577Z", "iopub.status.busy": "2022-05-23T04:37:43.656642Z", "iopub.status.idle": "2022-05-23T04:37:43.665306Z", "shell.execute_reply": "2022-05-23T04:37:43.66471Z", "shell.execute_reply.started": "2022-05-23T04:37:43.657534Z" } }, "outputs": [ { "data": { "text/plain": [ "0.47180873952099694" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "score(trn_xs[\"LogFare\"], trn_y, 2.7)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we'd expect from our earlier tests, `Sex` appears to be a better split.\n", "\n", "To make it easier to find the best binary split, we can create a simple interactive tool (note that this only works in Kaggle if you click \"Copy and Edit\" in the top right to open the notebook editor):" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:44.159662Z", "iopub.status.busy": "2022-05-23T04:37:44.159112Z", "iopub.status.idle": "2022-05-23T04:37:44.20316Z", "shell.execute_reply": "2022-05-23T04:37:44.202647Z", "shell.execute_reply.started": "2022-05-23T04:37:44.159612Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "cca27efa8e1643e990264062f595df43", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(Dropdown(description='nm', options=('Age', 'SibSp', 'Parch', 'LogFare', 'Pclass'), value…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def iscore(nm, split):\n", " col = trn_xs[nm]\n", " return score(col, trn_y, split)\n", "\n", "from ipywidgets import interact\n", "interact(nm=conts, split=15.5)(iscore);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Try selecting different columns and split points using the dropdown and slider above. What splits can you find that increase the purity of the data?\n", "\n", "We can do the same thing for the categorical variables:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:45.542149Z", "iopub.status.busy": "2022-05-23T04:37:45.541373Z", "iopub.status.idle": "2022-05-23T04:37:45.587574Z", "shell.execute_reply": "2022-05-23T04:37:45.58671Z", "shell.execute_reply.started": "2022-05-23T04:37:45.542101Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d6078c0b725e464998e37b5c00c5de08", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(Dropdown(description='nm', options=('Sex', 'Embarked'), value='Sex'), IntSlider(value=2,…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "interact(nm=cats, split=2)(iscore);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That works well enough, but it's rather slow and fiddly. Perhaps we could get the computer to automatically find the best split point for a column for us? For example, to find the best split point for `age` we'd first need to make a list of all the possible split points (i.e all the unique values of that field)...:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:47.225685Z", "iopub.status.busy": "2022-05-23T04:37:47.225336Z", "iopub.status.idle": "2022-05-23T04:37:47.234728Z", "shell.execute_reply": "2022-05-23T04:37:47.233526Z", "shell.execute_reply.started": "2022-05-23T04:37:47.225646Z" } }, "outputs": [ { "data": { "text/plain": [ "array([ 0.42, 0.67, 0.75, 0.83, 0.92, 1. , 2. , 3. , 4. , 5. , 6. , 7. , 8. , 9. , 10. , 11. , 12. ,\n", " 13. , 14. , 14.5 , 15. , 16. , 17. , 18. , 19. , 20. , 21. , 22. , 23. , 24. , 24.5 , 25. , 26. , 27. ,\n", " 28. , 28.5 , 29. , 30. , 31. , 32. , 32.5 , 33. , 34. , 34.5 , 35. , 36. , 36.5 , 37. , 38. , 39. , 40. ,\n", " 40.5 , 41. , 42. , 43. , 44. , 45. , 45.5 , 46. , 47. , 48. , 49. , 50. , 51. , 52. , 53. , 54. , 55. ,\n", " 55.5 , 56. , 57. , 58. , 59. , 60. , 61. , 62. , 64. , 65. , 70. , 70.5 , 74. , 80. ])" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nm = \"Age\"\n", "col = trn_xs[nm]\n", "unq = col.unique()\n", "unq.sort()\n", "unq" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...and find which index of those values is where `score()` is the lowest:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:49.070221Z", "iopub.status.busy": "2022-05-23T04:37:49.069942Z", "iopub.status.idle": "2022-05-23T04:37:49.150766Z", "shell.execute_reply": "2022-05-23T04:37:49.14992Z", "shell.execute_reply.started": "2022-05-23T04:37:49.070191Z" } }, "outputs": [ { "data": { "text/plain": [ "6.0" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "scores = np.array([score(col, trn_y, o) for o in unq if not np.isnan(o)])\n", "unq[scores.argmin()]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Based on this, it looks like, for instance, that for the `Age` column, `6` is the optimal cutoff according to our training set.\n", "\n", "We can write a little function that implements this idea:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:50.094341Z", "iopub.status.busy": "2022-05-23T04:37:50.093867Z", "iopub.status.idle": "2022-05-23T04:37:50.175498Z", "shell.execute_reply": "2022-05-23T04:37:50.174679Z", "shell.execute_reply.started": "2022-05-23T04:37:50.094305Z" } }, "outputs": [ { "data": { "text/plain": [ "(6.0, 0.478316717508991)" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def min_col(df, nm):\n", " col,y = df[nm],df[dep]\n", " unq = col.dropna().unique()\n", " scores = np.array([score(col, y, o) for o in unq if not np.isnan(o)])\n", " idx = scores.argmin()\n", " return unq[idx],scores[idx]\n", "\n", "min_col(trn_df, \"Age\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's try all the columns:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:51.295064Z", "iopub.status.busy": "2022-05-23T04:37:51.294528Z", "iopub.status.idle": "2022-05-23T04:37:51.577522Z", "shell.execute_reply": "2022-05-23T04:37:51.57682Z", "shell.execute_reply.started": "2022-05-23T04:37:51.29503Z" } }, "outputs": [ { "data": { "text/plain": [ "{'Sex': (0, 0.40787530982063946),\n", " 'Embarked': (0, 0.47883342573147836),\n", " 'Age': (6.0, 0.478316717508991),\n", " 'SibSp': (4, 0.4783740258817434),\n", " 'Parch': (0, 0.4805296527841601),\n", " 'LogFare': (2.4390808375825834, 0.4620823937736597),\n", " 'Pclass': (2, 0.46048261885806596)}" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cols = cats+conts\n", "{o:min_col(trn_df, o) for o in cols}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "According to this, `Sex<=0` is the best split we can use.\n", "\n", "We've just re-invented the [OneR](https://link.springer.com/article/10.1023/A:1022631118932) classifier (or at least, a minor variant of it), which was found to be one of the most effective classifiers in real-world datasets, compared to the algorithms in use in 1993. Since it's so simple and surprisingly effective, it makes for a great *baseline* -- that is, a starting point that you can use to compare your more sophisticated models to.\n", "\n", "We found earlier that our OneR rule had an error of around `0.215`, so we'll keep that in mind as we try out more sophisticated approaches." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating a decision tree" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How can we improve our OneR classifier, which predicts survival based only on `Sex`?\n", "\n", "How about we take each of our two groups, `female` and `male`, and create one more binary split for each of them. That is: find the single best split for females, and the single best split for males. To do this, all we have to do is repeat the previous section's steps, once for males, and once for females.\n", "\n", "First, we'll remove `Sex` from the list of possible splits (since we've already used it, and there's only one possible split for that binary column), and create our two groups:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:52.822448Z", "iopub.status.busy": "2022-05-23T04:37:52.821912Z", "iopub.status.idle": "2022-05-23T04:37:52.829494Z", "shell.execute_reply": "2022-05-23T04:37:52.82831Z", "shell.execute_reply.started": "2022-05-23T04:37:52.822405Z" } }, "outputs": [], "source": [ "cols.remove(\"Sex\")\n", "ismale = trn_df.Sex==1\n", "males,females = trn_df[ismale],trn_df[~ismale]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's find the single best binary split for males...:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:54.372844Z", "iopub.status.busy": "2022-05-23T04:37:54.372319Z", "iopub.status.idle": "2022-05-23T04:37:54.585566Z", "shell.execute_reply": "2022-05-23T04:37:54.584693Z", "shell.execute_reply.started": "2022-05-23T04:37:54.372801Z" } }, "outputs": [ { "data": { "text/plain": [ "{'Embarked': (0, 0.3875581870410906),\n", " 'Age': (6.0, 0.3739828371010595),\n", " 'SibSp': (4, 0.3875864227586273),\n", " 'Parch': (0, 0.3874704821461959),\n", " 'LogFare': (2.803360380906535, 0.3804856231758151),\n", " 'Pclass': (1, 0.38155442004360934)}" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "{o:min_col(males, o) for o in cols}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...and for females:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:55.970401Z", "iopub.status.busy": "2022-05-23T04:37:55.969784Z", "iopub.status.idle": "2022-05-23T04:37:56.171762Z", "shell.execute_reply": "2022-05-23T04:37:56.170711Z", "shell.execute_reply.started": "2022-05-23T04:37:55.970348Z" } }, "outputs": [ { "data": { "text/plain": [ "{'Embarked': (0, 0.4295252982857327),\n", " 'Age': (50.0, 0.4225927658431649),\n", " 'SibSp': (4, 0.42319212059713535),\n", " 'Parch': (3, 0.4193314500446158),\n", " 'LogFare': (4.256321678298823, 0.41350598332911376),\n", " 'Pclass': (2, 0.3335388911567601)}" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "{o:min_col(females, o) for o in cols}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the best next binary split for males is `Age<=6`, and for females is `Pclass<=2`.\n", "\n", "By adding these rules, we have created a *decision tree*, where our model will first check whether `Sex` is female or male, and depending on the result will then check either the above `Age` or `Pclass` rules, as appropriate. We could then repeat the process, creating new additional rules for each of the four groups we've now created.\n", "\n", "Rather than writing that code manually, we can use `DecisionTreeClassifier`, from *sklearn*, which does exactly that for us:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:57.022761Z", "iopub.status.busy": "2022-05-23T04:37:57.022405Z", "iopub.status.idle": "2022-05-23T04:37:57.201325Z", "shell.execute_reply": "2022-05-23T04:37:57.200214Z", "shell.execute_reply.started": "2022-05-23T04:37:57.022724Z" } }, "outputs": [], "source": [ "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n", "\n", "m = DecisionTreeClassifier(max_leaf_nodes=4).fit(trn_xs, trn_y);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One handy feature or this class is that it provides a function for drawing a tree representing the rules:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:57.253844Z", "iopub.status.busy": "2022-05-23T04:37:57.253469Z", "iopub.status.idle": "2022-05-23T04:37:57.274531Z", "shell.execute_reply": "2022-05-23T04:37:57.27374Z", "shell.execute_reply.started": "2022-05-23T04:37:57.253805Z" } }, "outputs": [], "source": [ "import graphviz\n", "\n", "def draw_tree(t, df, size=10, ratio=0.6, precision=2, **kwargs):\n", " s=export_graphviz(t, out_file=None, feature_names=df.columns, filled=True, rounded=True,\n", " special_characters=True, rotate=False, precision=precision, **kwargs)\n", " return graphviz.Source(re.sub('Tree {', f'Tree {{ size={size}; ratio={ratio}', s))" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:58.207169Z", "iopub.status.busy": "2022-05-23T04:37:58.206306Z", "iopub.status.idle": "2022-05-23T04:37:59.286087Z", "shell.execute_reply": "2022-05-23T04:37:59.284841Z", "shell.execute_reply.started": "2022-05-23T04:37:58.207112Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "Tree\n", "\n", "\n", "\n", "0\n", "\n", "Sex ≤ 0.5\n", "gini = 0.47\n", "samples = 668\n", "value = [415, 253]\n", "\n", "\n", "\n", "1\n", "\n", "Pclass ≤ 2.5\n", "gini = 0.38\n", "samples = 229\n", "value = [59, 170]\n", "\n", "\n", "\n", "0->1\n", "\n", "\n", "True\n", "\n", "\n", "\n", "2\n", "\n", "Age ≤ 6.5\n", "gini = 0.31\n", "samples = 439\n", "value = [356, 83]\n", "\n", "\n", "\n", "0->2\n", "\n", "\n", "False\n", "\n", "\n", "\n", "3\n", "\n", "gini = 0.06\n", "samples = 120\n", "value = [4, 116]\n", "\n", "\n", "\n", "1->3\n", "\n", "\n", "\n", "\n", "\n", "4\n", "\n", "gini = 0.5\n", "samples = 109\n", "value = [55, 54]\n", "\n", "\n", "\n", "1->4\n", "\n", "\n", "\n", "\n", "\n", "5\n", "\n", "gini = 0.41\n", "samples = 21\n", "value = [6, 15]\n", "\n", "\n", "\n", "2->5\n", "\n", "\n", "\n", "\n", "\n", "6\n", "\n", "gini = 0.27\n", "samples = 418\n", "value = [350, 68]\n", "\n", "\n", "\n", "2->6\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "draw_tree(m, trn_xs, size=10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that it's found exactly the same splits as we did!\n", "\n", "In this picture, the more orange nodes have a lower survival rate, and blue have higher survival. Each node shows how many rows (\"*samples*\") match that set of rules, and shows how many perish or survive (\"*values*\"). There's also something called \"*gini*\". That's another measure of impurity, and it's very similar to the `score()` we created earlier. It's defined as follows:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:37:59.76912Z", "iopub.status.busy": "2022-05-23T04:37:59.76879Z", "iopub.status.idle": "2022-05-23T04:37:59.776055Z", "shell.execute_reply": "2022-05-23T04:37:59.774896Z", "shell.execute_reply.started": "2022-05-23T04:37:59.769086Z" } }, "outputs": [], "source": [ "def gini(cond):\n", " act = df.loc[cond, dep]\n", " return 1 - act.mean()**2 - (1-act).mean()**2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What this calculates is the probability that, if you pick two rows from a group, you'll get the same `Survived` result each time. If the group is all the same, the probability is `1.0`, and `0.0` if they're all different:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:38:01.166595Z", "iopub.status.busy": "2022-05-23T04:38:01.166139Z", "iopub.status.idle": "2022-05-23T04:38:01.177753Z", "shell.execute_reply": "2022-05-23T04:38:01.176747Z", "shell.execute_reply.started": "2022-05-23T04:38:01.166562Z" } }, "outputs": [ { "data": { "text/plain": [ "(0.3828350034484158, 0.3064437162277842)" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gini(df.Sex=='female'), gini(df.Sex=='male')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's see how this model compares to our OneR version:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:38:02.057241Z", "iopub.status.busy": "2022-05-23T04:38:02.056429Z", "iopub.status.idle": "2022-05-23T04:38:02.068392Z", "shell.execute_reply": "2022-05-23T04:38:02.067471Z", "shell.execute_reply.started": "2022-05-23T04:38:02.057192Z" } }, "outputs": [ { "data": { "text/plain": [ "0.2242152466367713" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mean_absolute_error(val_y, m.predict(val_xs))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's a tiny bit worse. Since this is such a small dataset (we've only got around 200 rows in our validation set) this small difference isn't really meaningful. Perhaps we'll see better results if we create a bigger tree:" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:38:02.926983Z", "iopub.status.busy": "2022-05-23T04:38:02.926216Z", "iopub.status.idle": "2022-05-23T04:38:02.981361Z", "shell.execute_reply": "2022-05-23T04:38:02.98003Z", "shell.execute_reply.started": "2022-05-23T04:38:02.926945Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "Tree\n", "\n", "\n", "\n", "0\n", "\n", "Sex ≤ 0.5\n", "gini = 0.47\n", "samples = 668\n", "value = [415, 253]\n", "\n", "\n", "\n", "1\n", "\n", "Pclass ≤ 2.5\n", "gini = 0.38\n", "samples = 229\n", "value = [59, 170]\n", "\n", "\n", "\n", "0->1\n", "\n", "\n", "True\n", "\n", "\n", "\n", "8\n", "\n", "LogFare ≤ 3.31\n", "gini = 0.31\n", "samples = 439\n", "value = [356, 83]\n", "\n", "\n", "\n", "0->8\n", "\n", "\n", "False\n", "\n", "\n", "\n", "2\n", "\n", "Age ≤ 28.5\n", "gini = 0.06\n", "samples = 120\n", "value = [4, 116]\n", "\n", "\n", "\n", "1->2\n", "\n", "\n", "\n", "\n", "\n", "5\n", "\n", "LogFare ≤ 2.7\n", "gini = 0.5\n", "samples = 109\n", "value = [55, 54]\n", "\n", "\n", "\n", "1->5\n", "\n", "\n", "\n", "\n", "\n", "3\n", "\n", "gini = 0.11\n", "samples = 53\n", "value = [3, 50]\n", "\n", "\n", "\n", "2->3\n", "\n", "\n", "\n", "\n", "\n", "4\n", "\n", "gini = 0.03\n", "samples = 67\n", "value = [1, 66]\n", "\n", "\n", "\n", "2->4\n", "\n", "\n", "\n", "\n", "\n", "6\n", "\n", "gini = 0.49\n", "samples = 59\n", "value = [25, 34]\n", "\n", "\n", "\n", "5->6\n", "\n", "\n", "\n", "\n", "\n", "7\n", "\n", "gini = 0.48\n", "samples = 50\n", "value = [30, 20]\n", "\n", "\n", "\n", "5->7\n", "\n", "\n", "\n", "\n", "\n", "9\n", "\n", "Age ≤ 20.5\n", "gini = 0.24\n", "samples = 320\n", "value = [275, 45]\n", "\n", "\n", "\n", "8->9\n", "\n", "\n", "\n", "\n", "\n", "18\n", "\n", "SibSp ≤ 0.5\n", "gini = 0.43\n", "samples = 119\n", "value = [81, 38]\n", "\n", "\n", "\n", "8->18\n", "\n", "\n", "\n", "\n", "\n", "10\n", "\n", "gini = 0.43\n", "samples = 55\n", "value = [38, 17]\n", "\n", "\n", "\n", "9->10\n", "\n", "\n", "\n", "\n", "\n", "11\n", "\n", "Age ≤ 32.5\n", "gini = 0.19\n", "samples = 265\n", "value = [237, 28]\n", "\n", "\n", "\n", "9->11\n", "\n", "\n", "\n", "\n", "\n", "12\n", "\n", "Age ≤ 24.75\n", "gini = 0.22\n", "samples = 181\n", "value = [158, 23]\n", "\n", "\n", "\n", "11->12\n", "\n", "\n", "\n", "\n", "\n", "17\n", "\n", "gini = 0.11\n", "samples = 84\n", "value = [79, 5]\n", "\n", "\n", "\n", "11->17\n", "\n", "\n", "\n", "\n", "\n", "13\n", "\n", "LogFare ≤ 2.18\n", "gini = 0.16\n", "samples = 114\n", "value = [104, 10]\n", "\n", "\n", "\n", "12->13\n", "\n", "\n", "\n", "\n", "\n", "16\n", "\n", "gini = 0.31\n", "samples = 67\n", "value = [54, 13]\n", "\n", "\n", "\n", "12->16\n", "\n", "\n", "\n", "\n", "\n", "14\n", "\n", "gini = 0.21\n", "samples = 50\n", "value = [44, 6]\n", "\n", "\n", "\n", "13->14\n", "\n", "\n", "\n", "\n", "\n", "15\n", "\n", "gini = 0.12\n", "samples = 64\n", "value = [60, 4]\n", "\n", "\n", "\n", "13->15\n", "\n", "\n", "\n", "\n", "\n", "19\n", "\n", "gini = 0.48\n", "samples = 60\n", "value = [36, 24]\n", "\n", "\n", "\n", "18->19\n", "\n", "\n", "\n", "\n", "\n", "20\n", "\n", "gini = 0.36\n", "samples = 59\n", "value = [45, 14]\n", "\n", "\n", "\n", "18->20\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m = DecisionTreeClassifier(min_samples_leaf=50)\n", "m.fit(trn_xs, trn_y)\n", "draw_tree(m, trn_xs, size=25)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:38:03.953946Z", "iopub.status.busy": "2022-05-23T04:38:03.953608Z", "iopub.status.idle": "2022-05-23T04:38:03.965301Z", "shell.execute_reply": "2022-05-23T04:38:03.964474Z", "shell.execute_reply.started": "2022-05-23T04:38:03.953912Z" } }, "outputs": [ { "data": { "text/plain": [ "0.18385650224215247" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mean_absolute_error(val_y, m.predict(val_xs))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It looks like this is an improvement, although again it's a bit hard to tell with small datasets like this. Let's try submitting it to Kaggle:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:38:04.158937Z", "iopub.status.busy": "2022-05-23T04:38:04.158557Z", "iopub.status.idle": "2022-05-23T04:38:04.183949Z", "shell.execute_reply": "2022-05-23T04:38:04.182841Z", "shell.execute_reply.started": "2022-05-23T04:38:04.158902Z" } }, "outputs": [], "source": [ "tst_df[cats] = tst_df[cats].apply(lambda x: x.cat.codes)\n", "tst_xs,_ = xs_y(tst_df)\n", "\n", "def subm(preds, suff):\n", " tst_df['Survived'] = preds\n", " sub_df = tst_df[['PassengerId','Survived']]\n", " sub_df.to_csv(f'sub-{suff}.csv', index=False)\n", "\n", "subm(m.predict(tst_xs), 'tree')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When I submitted this, I got a score of 0.765, which isn't as good as our linear models or most of our neural nets, but it's pretty close to those results.\n", "\n", "Hopefully you can now see why we didn't really need to create dummy variables, but instead just converted the labels into numbers using some (potentially arbitary) ordering of categories. For instance, here's how the first few items of `Embarked` are labeled:" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:40:33.30121Z", "iopub.status.busy": "2022-05-23T04:40:33.300898Z", "iopub.status.idle": "2022-05-23T04:40:33.310952Z", "shell.execute_reply": "2022-05-23T04:40:33.309927Z", "shell.execute_reply.started": "2022-05-23T04:40:33.301175Z" } }, "outputs": [ { "data": { "text/plain": [ "0 S\n", "1 C\n", "2 S\n", "3 S\n", "4 S\n", "Name: Embarked, dtype: category\n", "Categories (3, object): ['C', 'Q', 'S']" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.Embarked.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...resulting in these integer codes:" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:41:09.343513Z", "iopub.status.busy": "2022-05-23T04:41:09.34246Z", "iopub.status.idle": "2022-05-23T04:41:09.351674Z", "shell.execute_reply": "2022-05-23T04:41:09.350699Z", "shell.execute_reply.started": "2022-05-23T04:41:09.343456Z" } }, "outputs": [ { "data": { "text/plain": [ "0 2\n", "1 0\n", "2 2\n", "3 2\n", "4 2\n", "dtype: int8" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.Embarked.cat.codes.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So let's say we wanted to split into \"C\" in one group, vs \"Q\" or \"S\" in the other group. Then we just have to split on codes `<=0` (since `C` is mapped to category `0`). Note that if we wanted to split into \"Q\" in one group, we'd need to use two binary splits, first to separate \"C\" from \"Q\" and \"S\", and then a second split to separate \"Q\" from \"S\". For this reason, sometimes it can still be helpful to use dummy variables for categorical variables with few levels (like this one).\n", "\n", "In practice, I often use dummy variables for <4 levels, and numeric codes for >=4 levels." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The random forest" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can't make the decision tree much bigger than the example above, since some leaf nodes already have only 50 rows in them. That's not a lot of data to make a prediction.\n", "\n", "So how could we use bigger trees? One big insight came from Leo Breiman: what if we create lots of bigger trees, and take the average of their predictions? Taking the average prediction of a bunch of models in this way is known as [bagging](https://link.springer.com/article/10.1007/BF00058655).\n", "\n", "The idea is that we want each model's predictions in the averaged ensemble to be uncorrelated with each other model. That way, if we average the predictions, the average will be equal to the true target value -- that's because the average of lots of uncorrelated random errors is zero. That's quite an amazing insight!\n", "\n", "One way we can create a bunch of uncorrelated models is to train each of them on a different random subset of the data. Here's how we can create a tree on a random subset of the data:" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:51:20.332256Z", "iopub.status.busy": "2022-05-23T04:51:20.331981Z", "iopub.status.idle": "2022-05-23T04:51:20.338451Z", "shell.execute_reply": "2022-05-23T04:51:20.337562Z", "shell.execute_reply.started": "2022-05-23T04:51:20.332229Z" } }, "outputs": [], "source": [ "def get_tree(prop=0.75):\n", " n = len(trn_y)\n", " idxs = random.choice(n, int(n*prop))\n", " return DecisionTreeClassifier(min_samples_leaf=5).fit(trn_xs.iloc[idxs], trn_y.iloc[idxs])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can create as many trees as we want:" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:51:23.113575Z", "iopub.status.busy": "2022-05-23T04:51:23.113271Z", "iopub.status.idle": "2022-05-23T04:51:23.392216Z", "shell.execute_reply": "2022-05-23T04:51:23.391179Z", "shell.execute_reply.started": "2022-05-23T04:51:23.113547Z" } }, "outputs": [], "source": [ "trees = [get_tree() for t in range(100)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our prediction will be the average of these trees' predictions:" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:51:24.13079Z", "iopub.status.busy": "2022-05-23T04:51:24.130425Z", "iopub.status.idle": "2022-05-23T04:51:24.27515Z", "shell.execute_reply": "2022-05-23T04:51:24.274245Z", "shell.execute_reply.started": "2022-05-23T04:51:24.130754Z" } }, "outputs": [ { "data": { "text/plain": [ "0.2272645739910314" ] }, "execution_count": 50, "metadata": {}, "output_type": "execute_result" } ], "source": [ "all_probs = [t.predict(val_xs) for t in trees]\n", "avg_probs = np.stack(all_probs).mean(0)\n", "\n", "mean_absolute_error(val_y, avg_probs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is nearly identical to what `sklearn`'s `RandomForestClassifier` does. The main extra piece in a \"real\" random forest is that as well as choosing a random sample of data for each tree, it also picks a random subset of columns for each split. Here's how we repeat the above process with a random forest:" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:51:24.302789Z", "iopub.status.busy": "2022-05-23T04:51:24.302442Z", "iopub.status.idle": "2022-05-23T04:51:24.586127Z", "shell.execute_reply": "2022-05-23T04:51:24.584999Z", "shell.execute_reply.started": "2022-05-23T04:51:24.302754Z" } }, "outputs": [ { "data": { "text/plain": [ "0.18834080717488788" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.ensemble import RandomForestClassifier\n", "\n", "rf = RandomForestClassifier(100, min_samples_leaf=5)\n", "rf.fit(trn_xs, trn_y);\n", "mean_absolute_error(val_y, rf.predict(val_xs))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can submit that to Kaggle too:" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:51:26.868302Z", "iopub.status.busy": "2022-05-23T04:51:26.867535Z", "iopub.status.idle": "2022-05-23T04:51:26.902301Z", "shell.execute_reply": "2022-05-23T04:51:26.901088Z", "shell.execute_reply.started": "2022-05-23T04:51:26.868252Z" } }, "outputs": [], "source": [ "subm(rf.predict(tst_xs), 'rf')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I found that gave nearly an identical result as our single tree (which, in turn, was slightly lower than our linear and neural net models in the previous notebook)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One particularly nice feature of random forests is they can tell us which independent variables were the most important in the model, using `feature_importances_`:" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "execution": { "iopub.execute_input": "2022-05-23T04:53:08.654508Z", "iopub.status.busy": "2022-05-23T04:53:08.654163Z", "iopub.status.idle": "2022-05-23T04:53:08.906439Z", "shell.execute_reply": "2022-05-23T04:53:08.904724Z", "shell.execute_reply.started": "2022-05-23T04:53:08.654445Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaQAAAD4CAYAAACjd5INAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXvUlEQVR4nO3dfZRddX3v8fcnEBMgGJ7URkADGFKFAEJAQZGAd3GraBFFBK2CFRHX1WWv197mLlqkIq23topwqxgfCvb6QFGxFqxgxQQUBCcaCJGHSzC2wScCNhqehPC9f8wOjsNk5mQyM2efmfdrrVk5Z+/f3vuzz8rkk/07e86kqpAkqdumdTuAJElgIUmSWsJCkiS1goUkSWoFC0mS1ArbdjtAr9ptt91q7ty53Y4hST1l+fLl66rqaUOts5BGae7cufT19XU7hiT1lCQ/3tw6p+wkSa1gIUmSWsFCkiS1gu8hSVKXPfroo6xdu5aHH36421HGzMyZM9ljjz2YPn16x9tYSJLUZWvXrmXHHXdk7ty5JOl2nK1WVdx3332sXbuWvfbaq+PtnLKTpC57+OGH2XXXXSdFGQEkYdddd93iKz6vkEZp5T3rmbv4ym7HmLTWfOC4bkeQJtRkKaNNRnM+XiFJklrBKyRJapmxnn0ZacbhiCOO4Prrrx/TY46GV0iSNMW1oYzAQpKkKW/WrFkALF26lKOOOoqTTjqJfffdl8WLF/PZz36Www47jAULFrB69WoATjvtNM4880yOPPJI9t13X6644ooxydETU3ZJNgIr6c97G3BqVT24mbHnABuq6m8nLqEkTQ4333wzt912G7vssgt77703p59+OjfddBMf+chHuPDCCzn//PMBWLNmDcuWLWP16tUcffTR3HXXXcycOXOrjt0rV0gPVdVBVbU/8BvgzG4HkqTJ6NBDD2XOnDnMmDGDffbZh2OPPRaABQsWsGbNmifGnXTSSUybNo158+ax9957c/vtt2/1sXulkAa6DngOQJI3Jbklyc1J/nHwwCRvTfK9Zv2XkmzfLH9tklub5dc2y/ZLclOSFc0+503oWUlSC8yYMeOJx9OmTXvi+bRp03jssceeWDf4tu6xuG29pwopybbAy4CVSfYDzgKOqaoDgXcNscmXq+rQZv1twFua5WcD/7VZ/ofNsjOBj1TVQcBCYO0Qxz8jSV+Svo0Prh/LU5OknnLZZZfx+OOPs3r1au6++27mz5+/1fvsifeQgO2SrGgeXwd8Cngb8MWqWgdQVfcPsd3+Sd4P7ATMAq5qln8HuDjJPwFfbpbdAJyVZA/6i+z/Dd5ZVS0BlgDMmDOvxuC8JOlJeuEHw+fPn89RRx3Fz3/+cy666KKtfv8IeqeQHmquXJ6Q/uvDkUrhYuBVVXVzktOARQBVdWaSFwDHASuSHFRVn0tyY7PsqiSnV9U1Y3saktQ+GzZsAGDRokUsWrToieVLly594vHgdS960Yv48Ic/PKY5emrKbpBvAicl2RUgyS5DjNkR+GmS6cAbNi1Msk9V3VhVZwPrgD2T7A3cXVUXAF8FDhj3M5AkPaFXrpCepKpWJTkPWNbcFv4D4LRBw/4CuBH4Mf23je/YLP9gc9NC6C+2m4HFwB8leRT4GfC+cT8JSepBF1988bjstycKqapmbWb5JcAlg5adM+Dxx4CPDbHdq4fY3V83X5I04apqUn3AatWWv83ey1N2kjQpzJw5k/vuu29U/4i30abfh7SlNzr0xBWSJE1me+yxB2vXruXee+/tdpQxs+k3xm6JTJZGnmgLFy6svr6+bseQpJ6SZHlVLRxqnVN2kqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFawkCRJrWAhSZJawUKSJLWChSRJagULSZLUCv4+pFFaec965i6+ctyPs+YDx437MSSpDbxCkiS1goUkSWoFC0mS1AoWkiSpFVpTSEk2jME+FiVZn2RF8/VvY5FNkjT+JuNddtdV1Su2ZIMk21bVY+MVSJI0stZcIQ0lyUFJvpvkliSXJ9m5WX5os+yGJB9Mcusw+zgsyfVJftD8Ob9ZflqSy5L8C3B1kh2SfDrJ95qxx0/QaUqSaHkhAZ8B/qyqDgBWAu9tlv8DcGZVHQ5sHLTNkQOm7M4CbgdeUlXPB84G/mrA2MOBU6vqGOAs4JqqOhQ4Gvhgkh0G7jjJGUn6kvRtfHD9GJ+qJE1trZ2ySzIb2KmqljWLLgEuS7ITsGNVXd8s/xwwcIrud6bskuwJXJJkHlDA9AFjv1FV9zePjwX+MMl7muczgWcBt20aXFVLgCUAM+bMq60/S0nSJq0tpGFkC8efC3yrqk5IMhdYOmDdA4P2+5qqumPr4kmSRqO1U3ZVtR74ZZIjm0VvBJZV1S+BXyd5YbP85BF2NRu4p3l82jDjrgLemSQASZ4/quCSpFFp0xXS9knWDnj+IeBU4KIk2wN3A29u1r0F+ESSB+i/4hnuDZ2/oX/K7t3ANcOMOxc4H7ilKaU1/O5UoCRpHKWq994KSTKrqjY0jxcDc6rqXROZYcaceTXn1PPH/Th+uKqkySTJ8qpaONS6Nl0hbYnjkvwv+vP/mOGn4iRJPaAnC6mqLgUu7XYOSdLY6clCaoMFu8+mz+k0SRozrb3LTpI0tVhIkqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFawkCRJrWAhSZJawUKSJLWChSRJagULSZLUChaSJKkVLCRJUitYSJKkVrCQJEmt0JOFlGRjkhVJbk1yWZLtt3J/c5PcOlb5JElbricLCXioqg6qqv2B3wBndrJRkm3HN5YkabR6tZAGug54TpJXJrkxyQ+S/FuSZwAkOSfJkiRXA59J8owklye5ufk6otnPNkk+kWRVkquTbNe1M5KkKainC6m54nkZsBL4NvDCqno+8AXgfw4YeghwfFW9HrgAWFZVBwIHA6uaMfOAv6+q/YD/BF4zxPHOSNKXpO/ee+8dp7OSpKmpV6ewtkuyonl8HfApYD5waZI5wFOAHw0Y/9Wqeqh5fAzwJoCq2gisT7Iz8KOq2rTP5cDcwQetqiXAEoCFCxfWGJ6PJE15vVpID1XVQQMXJLkQ+FBVfTXJIuCcAasf6GCfjwx4vBFwyk6SJlBPT9kNMhu4p3l86jDjvgm8HSDJNkmeOt7BJEkjm0yFdA5wWZLrgHXDjHsXcHSSlfRPze03AdkkSSNIlW+FjMbChQurr6+v2zEkqackWV5VC4daN5mukCRJPcxCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFawkCRJrWAhSZJawUKSJLWChSRJagULSZLUChaSJKkVLCRJUitYSJKkVrCQJEmtYCFJklph224H6FUr71nP3MVXPmn5mg8c14U0ktT7tvgKKcm0JE8djzCSpKmro0JK8rkkT02yA/BD4I4kfzq+0SRJU0mnV0jPq6pfAa8CvgY8C3jjeIWSJE09nRbS9CTT6S+kf66qR4Eat1SSpCmn00L6OLAG2AG4NsmzgV+NV6iRJDkryaoktyRZkeQFST6Z5HnN+g2b2e6FSW5strktyTkTGlyStFkd3WVXVRcAFwxY9OMkR49PpOElORx4BXBwVT2SZDfgKVV1egebXwKcVFU3J9kGmD+eWSVJnRu2kJK8e4TtPzSGWTo1B1hXVY8AVNU6gCRLgfdUVV/z/O+Ao4FfAidX1b3A04GfNtttpP8GDZorpX2A3YE9gb+pqk9M3ClJkkaasttxhK9uuBrYM8mdST6a5KghxuwAfL+qDgaWAe9tln+Y/jsEL0/ytiQzB2xzAHAccDhwdpJnDt5pkjOS9CXp2/jg+jE9KUma6oa9Qqqqv5yoIJ2qqg1JDgGOpP8K6NIkiwcNexy4tHn8f4EvN9u+L8lngWOB1wOnAIuacf9cVQ8BDyX5FnAY8JVBx14CLAGYMWeeN3VI0hjq9OeQ9miuKn6R5OdJvpRkj/EOtzlVtbGqllbVe4F3AK8ZaZMB266uqo8BLwUOTLLr4DGbeS5JGked3mX3D8BXgWfS/z7LvzTLJlyS+UnmDVh0EPDjQcOmASc2j18PfLvZ9rgkaZbPAzYC/9k8Pz7JzKagFgHfG/PwkqTN6vSz7J5WVQML6OIkfzIOeToxC7gwyU7AY8BdwBnAFweMeQDYL8lyYD3wumb5G4EPJ3mw2fYNVbWx6aibgCvp/6Hfc6vqJxNwLpKkRqeFtC7JHwGfb56fAtw3PpGGV1XLgSOGWLVowJhZzcO/GLTtycPs+s6qOmOrA0qSRqXTKbs/Bk4Cfkb/bdMnAm8er1CSpKmn0yukc4FTq+qXAEl2Af6W/qLqeVV1TrczSNJU12khHbCpjACq6v4kzx+nTD1hwe6z6fN3H0nSmOl0ym5akp03PWmukPzlfpKkMdNpqfwdcH2SL9L/8zknAeeNWypJ0pTT6YerfiZJH3AMEODVVfXDcU0mSZpSOp52awrIEpIkjYtO30OSJGlcWUiSpFawkCRJrWAhSZJawUKSJLWChSRJagULSZLUChaSJKkVLCRJUitYSJKkVvATu0dp5T3rmbv4yjHZ1xp/jYUkeYUkSWoHC0mS1AoWkiSpFSwkSVIrTNpCSnJCkkry+93OIkka2aQtJOAU4NvAyd0OIkka2aQspCSzgBcBb6EppCTTknw0yaokVyT5WpITm3WHJFmWZHmSq5LM6WJ8SZqSJmUhAa8Cvl5VdwL3JzkYeDUwF1gAnA4cDpBkOnAhcGJVHQJ8GjhvqJ0mOSNJX5K+jQ+uH/eTkKSpZLL+YOwpwPnN4y80z6cDl1XV48DPknyrWT8f2B/4RhKAbYCfDrXTqloCLAGYMWdejVd4SZqKJl0hJdkVOAbYP0nRXzAFXL65TYBVVXX4BEWUJA1hMk7ZnQh8pqqeXVVzq2pP4EfAOuA1zXtJzwAWNePvAJ6W5IkpvCT7dSO4JE1lk7GQTuHJV0NfAp4JrAVuBT4O3Aisr6rf0F9i/zvJzcAK4IgJSytJAibhlF1VLRpi2QXQf/ddVW1opvVuAlY261cAL5nAmJKkQSZdIY3giiQ7AU8Bzq2qn3U5jySpMaUKaairJ0lSO0ypQhpLC3afTZ+/x0iSxsxkvKlBktSDLCRJUitYSJKkVrCQJEmtYCFJklrBQpIktYKFJElqBQtJktQKFpIkqRUsJElSK1hIkqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFYYt0JKsjHJigFfi7dg20VJrtjK4y9NsnCU216c5MStOb4kactsO477fqiqDhrH/W9Wkm26cVxJ0uhN+JRdkjVJ/irJDUn6khyc5Kokq5OcOWDoU5NcnuSHSS5KMq3Z/mPNdquS/OWg/Z6d5NvAawcsn5bkkiTvT7JNkg8m+V6SW5K8rRmTJP+nOdaVwNMn6OWQJDXGs5C2GzRl97oB6/6jqg4HrgMuBk4EXgi8b8CYw4D/ASwA9gFe3Sw/q6oWAgcARyU5YMA2D1fVi6vqC83zbYHPAndW1Z8DbwHWV9WhwKHAW5PsBZwAzG+O9VbgiKFOKMkZTRn23XvvvaN5TSRJm9GtKbuvNn+uBGZV1a+BXyd5OMlOzbqbqupugCSfB14MfBE4KckZ9GefAzwPuKXZ5tJBx/k48E9VdV7z/FjggAHvD80G5gEvAT5fVRuBnyS5ZqjQVbUEWAKwcOHCGuH8JUlboFt32T3S/Pn4gMebnm8qycH/4FdzNfMe4KVVdQBwJTBzwJgHBm1zPXB0kk1jAryzqg5qvvaqqqs3czxJ0gRq823fhyXZq3nv6HXAt4Gn0l8665M8A3jZCPv4FPA14LIk2wJXAW9PMh0gyb5JdgCuBU5u3mOaAxw9PqckSdqc8Zyy2y7JigHPv15VHd/6DdwAfID+93WuBS6vqseT/ABYBdwNfGeknVTVh5LMBv4ReAMwF/h+kgD3Aq8CLgeOoX8K8U5g2RbklCSNgVQ5UzUaCxcurL6+vm7HkKSekmR5c2Pak7R5yk6SNIVYSJKkVrCQJEmtYCFJklrBQpIktYKFJElqBQtJktQKFpIkqRUsJElSK1hIkqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFbYttsBetXKe9Yzd/GV3Y4hSRNqzQeOG7d9e4UkSWoFC0mS1AoWkiSpFSZ1ISU5K8mqJLckWZHkBd3OJEka2qS9qSHJ4cArgIOr6pEkuwFP6XIsSdJmTOYrpDnAuqp6BKCq1lXVT5IckmRZkuVJrkoyJ8nsJHckmQ+Q5PNJ3trV9JI0xUzmQroa2DPJnUk+muSoJNOBC4ETq+oQ4NPAeVW1HngHcHGSk4Gdq+oTg3eY5IwkfUn6Nj64fiLPRZImvUk7ZVdVG5IcAhwJHA1cCrwf2B/4RhKAbYCfNuO/keS1wN8DB25mn0uAJQAz5syr8T4HSZpKJm0hAVTVRmApsDTJSuC/Aauq6vDBY5NMA54LPATsAqydwKiSNOVN2im7JPOTzBuw6CDgNuBpzQ0PJJmeZL9m/X9v1p8CfLqZ3pMkTZDJfIU0C7gwyU7AY8BdwBn0T7ldkGQ2/ed/fpJHgdOBw6rq10muBf4ceG9XkkvSFDRpC6mqlgNHDLFqHfCSIZY/d8C27x6vXJKkoU3aKTtJUm+xkCRJrTBpp+zG24LdZ9M3jh/DLklTjVdIkqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFVLlb1EYjSS/Bu7odo5R2I3+j0/qNb2Yuxczg7knUi9mhq3L/eyqetpQK/zB2NG7o6oWdjvElkrSZ+6J0YuZwdwTqRczw/jldspOktQKFpIkqRUspNFb0u0Ao2TuidOLmcHcE6kXM8M45famBklSK3iFJElqBQtJktQKFtIIkvxBkjuS3JVk8RDrk+SCZv0tSQ7uRs7BOsj9+0luSPJIkvd0I+NgHWR+Q/Ma35Lk+iQHdiPnYB3kPr7JvCJJX5IXdyPnYCPlHjDu0CQbk5w4kfk2k2Wk13pRkvXNa70iydndyDlYJ691k31FklVJlk10xiHyjPRa/+mA1/nW5u/ILlt10KryazNfwDbAamBv4CnAzcDzBo15OfCvQIAXAjf2SO6nA4cC5wHv6ZHMRwA7N49f1kOv9Sx++37tAcDtvZB7wLhrgK8BJ7Y9M7AIuKLbr+8ocu8E/BB4VvP86W3PPGj8K4Frtva4XiEN7zDgrqq6u6p+A3wBOH7QmOOBz1S/7wI7JZkz0UEHGTF3Vf2iqr4HPNqNgEPoJPP1VfXL5ul3gT0mOONQOsm9oZrvWmAHoA13EnXydxvgncCXgF9MZLjN6DRz23SS+/XAl6vq36H/+3OCMw62pa/1KcDnt/agFtLwdgf+Y8Dztc2yLR0z0dqYaSRbmvkt9F+ZdltHuZOckOR24Ergjyco23BGzJ1kd+AE4KIJzDWcTv+OHJ7k5iT/mmS/iYk2rE5y7wvsnGRpkuVJ3jRh6YbW8fdjku2BP6D/Py5bxY8OGl6GWDb4f7edjJlobcw0ko4zJzma/kJqw3sxHeWuqsuBy5O8BDgX+C/jHWwEneQ+H/izqtqYDDV8wnWS+fv0f1bahiQvB74CzBvvYCPoJPe2wCHAS4HtgBuSfLeq7hzvcJuxJf+GvBL4TlXdv7UHtZCGtxbYc8DzPYCfjGLMRGtjppF0lDnJAcAngZdV1X0TlG04W/RaV9W1SfZJsltVdfNDNTvJvRD4QlNGuwEvT/JYVX1lQhI+2YiZq+pXAx5/LclHe+S1Xgusq6oHgAeSXAscCHSrkLbk7/XJjMF0HeBNDcN90V/YdwN78ds39vYbNOY4fvemhpt6IfeAsefQjpsaOnmtnwXcBRzR7bxbmPs5/PamhoOBezY9b3PuQeMvpvs3NXTyWv/egNf6MODfe+G1Bp4LfLMZuz1wK7B/mzM342YD9wM7jMVxvUIaRlU9luQdwFX033Xy6apaleTMZv1F9N999HL6/6F8EHhzt/Ju0knuJL8H9AFPBR5P8if030Xzq83tt9uZgbOBXYGPNv9rf6y6/EnJHeZ+DfCmJI8CDwGvq+a7uVs6zN0qHWY+EXh7ksfof61P7oXXuqpuS/J14BbgceCTVXVrmzM3Q08Arq7+K7ut5kcHSZJawbvsJEmtYCFJklrBQpIktYKFJElqBQtJktQKFpIkqRUsJElSK/x//m0ShhO2Xx0AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "pd.DataFrame(dict(cols=trn_xs.columns, imp=m.feature_importances_)).plot('cols', 'imp', 'barh');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that `Sex` is by far the most important predictor, with `Pclass` a distant second, and `LogFare` and `Age` behind that. In datasets with many columns, I generally recommend creating a feature importance plot as soon as possible, in order to find which columns are worth studying more closely. (Note also that we didn't really need to take the `log()` of `Fare`, since random forests only care about order, and `log()` doesn't change the order -- we only did it to make our graphs earlier easier to read.)\n", "\n", "For details about deriving and understanding feature importances, and the many other important diagnostic tools provided by random forests, take a look at [chapter 8](https://github.com/fastai/fastbook/blob/master/08_collab.ipynb) of [our book](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So what can we take away from all this?\n", "\n", "I think the first thing I'd note from this is that, clearly, more complex models aren't always better. Our \"OneR\" model, consisting of a single binary split, was nearly as good as our more complex models. Perhaps in practice a simple model like this might be much easier to use, and could be worth considering. Our random forest wasn't an improvement on the single decision tree at all.\n", "\n", "So we should always be careful to benchmark simple models, as see if they're good enough for our needs. In practice, you will often find that simple models will have trouble providing adequate accuracy for more complex tasks, such as recommendation systems, NLP, computer vision, or multivariate time series. But there's no need to guess -- it's so easy to try a few different models, there's no reason not to give the simpler ones a go too!\n", "\n", "Another thing I think we can take away is that random forests aren't actually that complicated at all. We were able to implement the key features of them in a notebook quite quickly. And they aren't sensitive to issues like normalization, interactions, or non-linear transformations, which make them extremely easy to work with, and hard to mess up!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you found this notebook useful, please remember to click the little up-arrow at the top to upvote it, since I like to know when people have found my work useful, and it helps others find it too. (BTW, be sure you're looking at my [original notebook here](https://www.kaggle.com/jhoward/how-random-forests-work) when you do that, and are not on your own copy of it, otherwise your upvote won't get counted!) And if you have any questions or comments, please pop them below -- I read every comment I receive!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.10" }, "toc": { "base_numbering": 1, "nav_menu": { "height": "133.002px", "width": "196.553px" }, "number_sections": false, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }