{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Overfitting II\n", "\n", "Last time, we saw a theoretical example of *overfitting*, in which we fit a machine learning model that perfectly fit the data it saw, but performed extremely poorly on fresh, unseen data. In this lecture, we'll observe overfitting in a more practical context, using the Titanic data set again. We'll then begin to study *validation* techniques for finding models with \"just the right amount\" of flexibility. " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from matplotlib import pyplot as plt\n", "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SurvivedPclassNameSexAgeSiblings/Spouses AboardParents/Children AboardFare
003Mr. Owen Harris Braundmale22.0107.2500
111Mrs. John Bradley (Florence Briggs Thayer) Cum...female38.01071.2833
213Miss. Laina Heikkinenfemale26.0007.9250
311Mrs. Jacques Heath (Lily May Peel) Futrellefemale35.01053.1000
403Mr. William Henry Allenmale35.0008.0500
...........................
88202Rev. Juozas Montvilamale27.00013.0000
88311Miss. Margaret Edith Grahamfemale19.00030.0000
88403Miss. Catherine Helen Johnstonfemale7.01223.4500
88511Mr. Karl Howell Behrmale26.00030.0000
88603Mr. Patrick Dooleymale32.0007.7500
\n", "

887 rows × 8 columns

\n", "
" ], "text/plain": [ " Survived Pclass Name \\\n", "0 0 3 Mr. Owen Harris Braund \n", "1 1 1 Mrs. John Bradley (Florence Briggs Thayer) Cum... \n", "2 1 3 Miss. Laina Heikkinen \n", "3 1 1 Mrs. Jacques Heath (Lily May Peel) Futrelle \n", "4 0 3 Mr. William Henry Allen \n", ".. ... ... ... \n", "882 0 2 Rev. Juozas Montvila \n", "883 1 1 Miss. Margaret Edith Graham \n", "884 0 3 Miss. Catherine Helen Johnston \n", "885 1 1 Mr. Karl Howell Behr \n", "886 0 3 Mr. Patrick Dooley \n", "\n", " Sex Age Siblings/Spouses Aboard Parents/Children Aboard Fare \n", "0 male 22.0 1 0 7.2500 \n", "1 female 38.0 1 0 71.2833 \n", "2 female 26.0 0 0 7.9250 \n", "3 female 35.0 1 0 53.1000 \n", "4 male 35.0 0 0 8.0500 \n", ".. ... ... ... ... ... \n", "882 male 27.0 0 0 13.0000 \n", "883 female 19.0 0 0 30.0000 \n", "884 female 7.0 1 2 23.4500 \n", "885 male 26.0 0 0 30.0000 \n", "886 male 32.0 0 0 7.7500 \n", "\n", "[887 rows x 8 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# assumes that you have run the function retrieve_data() \n", "# from \"Introduction to ML in Practice\" in ML_3.ipynb\n", "\n", "titanic = pd.read_csv(\"data.csv\")\n", "titanic" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Recall that we diagnosed overfitting by testing our model against some new data. In this case, we don't have any more data. So, what we can do instead is *hold out* some data that we won't let our model see at first. This holdout data is called the *validation* or *testing* data, depending on the use to which we put it. In contrast, the data that we allow our model to see is called the *training* data. `sklearn` provides a convenient function for partitioning our data into training and holdout sets called `train_test_split`. The default and generally most useful behavior is to randomly select rows of the data frame to be in each set. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((620, 8), (267, 8))" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "np.random.seed(1234)\n", "train, test = train_test_split(titanic, test_size = 0.3) # hold out 30% of the data\n", "train.shape, test.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we have two data frames. As you may recall from a previous lecture, we need to do some data cleaning, and split them into predictor variables `X` and target variables `y`. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from sklearn import preprocessing\n", "\n", "def prep_titanic_data(data_df):\n", " df = data_df.copy()\n", " \n", " # convert male/female to 1/0\n", " le = preprocessing.LabelEncoder()\n", " df['Sex'] = le.fit_transform(df['Sex'])\n", " \n", " # don't need name column\n", " df = df.drop(['Name'], axis = 1)\n", " \n", " # split into X and y\n", " X = df.drop(['Survived'], axis = 1)\n", " y = df['Survived']\n", " \n", " return(X, y)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "X_train, y_train = prep_titanic_data(train)\n", "X_test, y_test = prep_titanic_data(test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we're able to train our model on the `train` data, and then evaluate its performance on the `val` data. This will help us to diagnose and avoid overfitting.\n", "\n", "Let's try using the decision tree classifier again. As you may remember, the `DecisionTreeClassifier()` class takes an argument `max_depth` that governs how many layers of decisions the tree is allowed to make. Larger `max_depth` values correspond to more complicated trees. In this way, `max_depth` is a model complexity parameter, similar to the `degree` when we did polynomial regression. \n", "\n", "For example, with a small `max_depth`, the model scores on the training and validation data are relatively close. " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(0.8290322580645161, 0.8164794007490637)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn import tree\n", "\n", "T = tree.DecisionTreeClassifier(max_depth = 3)\n", "\n", "T.fit(X_train, y_train)\n", "T.score(X_train, y_train), T.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "On the other hand, if we use a much higher `max_depth`, we can achieve a substantially better score on the training data, but our performance on the validation data has not improved by much, and might even suffer. " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(0.9903225806451613, 0.7602996254681648)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "T = tree.DecisionTreeClassifier(max_depth = 20)\n", "\n", "T.fit(X_train, y_train)\n", "T.score(X_train, y_train), T.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That looks like overfitting! The model achieves a near-perfect score on the training data, but a much lower one on the test data. " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Text(0, 0.5, 'Performance (score)'), Text(0.5, 0, 'Complexity (depth)')]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAGqCAYAAABOLOPmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df5Tkd13n++e7J0GmYvihDDGTSVdhiIYsG1DagBuioIAJF3fExV2guEp0rc2BIO7ZsKC1K+xy65xczLqXveGYbSD8MAWIQjZBWH4ExYR7F5keCPlFojFOd8YJZGKUCD0QZua9f9S3h55O/6juqR/fqno+zukzXZ+q+ta7vqkuXny+nx+RmUiSJKkcpoZdgCRJkr7HcCZJklQihjNJkqQSMZxJkiSViOFMkiSpRAxnkiRJJXJSvw4cEdcALwEeyMynr3J/AG8HXgwsAq/OzC8V911U3LcNeFdmXtHNaz7pSU/KWq3WmzcgSZLUR3v37n0wM3esbO9bOAPeC1wFvH+N+y8Gzi5+ng38PvDsiNgGvAN4IbAf2BMRN2TmnRu9YK1WY25urgelS5Ik9VdEzK/W3rfLmpl5E/DQOg/ZDbw/O74APCEiTgfOB+7JzHsz8xHgQ8VjJUmSxt4wx5ydAdy37Pb+om2t9lVFRCMi5iJi7uDBg30pVJIkaVCGGc5ilbZcp31VmTmbmTOZObNjx6Mu20qSJI2Ufo4528h+4Mxlt3cBB4DHrNEuSZI09obZc3YD8MvR8RzgG5l5P7AHODsinhIRjwFeXjxWkiRp7PVzKY0PAs8DnhQR+4E3AycDZObVwCfoLKNxD52lNC4p7jscEZcBn6KzlMY1mXlHv+qUJEkqk76Fs8x8xQb3J/DaNe77BJ3wJkmSNFHcIUCSJKlEDGeSJEklYjiTJEkqEcOZJElSiRjOJEmSSsRwJkmSVCKGM0nSlrTbbWq1GlNTU9RqNdrttscpeU0eZzDHOWGZOTY/z3rWs1KS1H/XXnttViqVpLP3cQJZqVTy2muv9TglrcnjDOY4mwHM5Sp5ZuiBqpc/hjNJ2ti1116b1Wo1IyKr1eqW/senWq0e9z9iSz/VatXjlLQmjzOY42zGWuEsOveNh5mZmZybmxt2GZJUWu12m0ajweLi4rG2SqXC7Ows9Xq96+NMTU2x2v9+RARHjx6d+OOUsSaPM5jjbEZE7M3MmUfV0pdXkySVUrPZPC6YASwuLtJsNjd1nOnp6U21T9pxyliTxxnMcXrBcCZJE2RhYWFT7WtptVpUKpXj2iqVCq1Wy+OUtCaPM5jj9MRq1zpH9ccxZ5K0vl6Oq+nF2LVxPk4Za/I4gzlOt3DMmSSNtna7TbPZZGFhgenpaVqt1qbGiS0doxdjziSdOMecSdIIWwpV8/PzZCbz8/M0Go1Nr8NUr9eZnZ2lWq0SEVSrVYOZVDL2nElSn/Wix6tWqzE/P/+o9mq1yr59+3pUqaRBWqvn7KRhFCNJk2LlZcSlHi9gUwGtVwP5JZWflzUlqY/KtnSFpPIznElSH5Vt6QpJ5Wc4k6Q+6lWPlwP5pclhOJOkNbTbbWq1GlNTU9RqtU3PjITe9njV63X27dvH0aNH2bdvn8FMGlOGM0lahUtXSBoWl9KQpFW4dIWkfnMRWknaBJeukDQshjNJY6UX48TApSskDY/hTNLY6NU4MXDpCknDYziTNDZ6teArOJBf0vA4IUDS2JiammK177SI4OjRo0OoSJLW5oQASWPPcWKSxoHhTNLYcJyYpHFgOJM0NhwnJmkcOOZMkiRpCBxzJkmSNAIMZ5IkSSViOJNUCr1a2V+SRt1Jwy5AkpZW9l9aQHZpZX/AwfySJo49Z5KGrpcr+0vSqDOcSRq6hYWFTbVL0jgznEkaOlf2l6TvMZxJGjpX9pek7zGcSRo6V/aXpO9xhwBJkqQhcIcASZKkEWA4kyRJKhHDmSRJUokYziRJkkrEcCZJklQihjNJkqQSMZxJOiHtdptarcbU1BS1Wo12uz3skiRppJ007AIkja52u02j0Ti2afn8/DyNRgPABWQlaYvsOZO0Zc1m81gwW7K4uEiz2RxSRZI0+gxnkrZsYWFhU+2SpI0ZziRt2fT09KbaJUkbM5xJ2rJWq0WlUjmurVKp0Gq1hlSRJI0+w5mkLavX68zOzlKtVokIqtUqs7OzTgaQpBMQmTnsGnpmZmYm5+bmhl2GJEnShiJib2bOrGy350ySJKlEDGfShHLxWEkqJxehlSaQi8dKUnnZcyZNIBePlaTyMpxJE8jFYyWpvAxn0gRy8VhJKi/DmTSBXDxWksrLcCZNIBePlaTy6usitBFxEfB2YBvwrsy8YsX9TwSuAc4Cvg38ambeXty3D/hH4AhweLVF2lZyEVpJkjQq1lqEtm9LaUTENuAdwAuB/cCeiLghM+9c9rDfBm7JzJdGxDnF43922f3Pz8wH+1WjJElS2fTzsub5wD2ZeW9mPgJ8CNi94jHnAp8FyMy7gFpEnNbHmiRJkkqtn+HsDOC+Zbf3F23LfQX4RYCIOB+oAruK+xL4dETsjYjGWi8SEY2ImIuIuYMHD/aseEmSpGHoZziLVdpWDnC7AnhiRNwCvA74MnC4uO+CzPxx4GLgtRHxU6u9SGbOZuZMZs7s2LGjR6VLkiQNRz+3b9oPnLns9i7gwPIHZObDwCUAERHA3xQ/ZOaB4t8HIuI6OpdJb+pjvZIkSUPXz56zPcDZEfGUiHgM8HLghuUPiIgnFPcB/Gvgpsx8OCJOiYhTi8ecArwIuL2PtUqSJJVC33rOMvNwRFwGfIrOUhrXZOYdEXFpcf/VwNOA90fEEeBO4NeKp58GXNfpTOMk4AOZ+cl+1SpJklQWfV3nbNBc50ySJI2KtdY5c4cASZKkEjGcSZIklYjhTJIkqUQMZ9KIabfb1Go1pqamqNVqtNvtYZckSeqhfq5zJqnH2u02jUaDxcVFAObn52k0Ohto1Ov1YZYmSeoRe86kEdJsNo8FsyWLi4s0m80hVSRJ6jXDmTRCFhYWNtUuSRo9hjNphExPT2+qXZI0egxn0ghptVpUKpXj2iqVCq1Wa0gVSZJ6zXAmjZB6vc7s7CzVapWIoFqtMjs762QASRojbt8kSZI0BG7fJEmSNAIMZ5IkSSViOJMkSSoRw5kkSVKJGM4kSZJKxHAmSZJUIoYzSZKkEjGcSZIklYjhTJIkqUQMZ5IkSSViOJMkSSoRw5kkSVKJGM4kSZJKxHAmDUi73aZWqzE1NUWtVqPdbg+7JElSCZ007AKkSdBut2k0GiwuLgIwPz9Po9EAoF6vD7M0SVLJ2HMmDUCz2TwWzJYsLi7SbDaHVJEkqawMZ9IALCwsbKpdkjS5DGfSAExPT2+qXZI0uQxn0gC0Wi0qlcpxbZVKhVarNaSKJEllZTiTBqBerzM7O0u1WiUiqFarzM7OOhlAkvQokZnDrqFnZmZmcm5ubthlSJIkbSgi9mbmzMp2e84kSZJKxHAmSZJUIoYzSZKkEjGcSZIklYjhTJIkqUQMZ5IkSSViOJMkSSoRw5kkSVKJGM4kSZJKxHAmSZJUIoYzSZKkEjGcSZIklYjhTJIkqUQMZ9IG2u02tVqNqakparUa7XZ72CVJksbYScMuQCqzdrtNo9FgcXERgPn5eRqNBgD1en2YpUmSxpQ9Z9I6ms3msWC2ZHFxkWazOaSKJEnjznAmrWNhYWFT7ZIknSjDmbSO6enpTbVLknSiDGfSOlqtFpVK5bi2SqVCq9UaUkWSpHFnOJPWUa/XmZ2dpVqtEhFUq1VmZ2edDCBJ6pvIzGHX0DMzMzM5Nzc37DIkSZI2FBF7M3NmZbs9Z5IkSSViOJMkSSoRw5kkSVKJGM4kSZJKxHAmSZJUIoYzSZKkEjGcSZIklcimwllEnBIR2/pVjCRJ0qRbN5xFxFREvDIiPh4RDwB3AfdHxB0R8bsRcfZgypQkSZoMG/Wc/RlwFvBbwA9l5pmZ+WTgQuALwBUR8ao+1yhJkjQxTtrg/hdk5ndXNmbmQ8BHgI9ExMl9qUySJGkCrdtztjyYRcRzI+KS4vcdEfGUlY9ZKSIuioi7I+KeiHjTKvc/MSKui4hbI+KLEfH0bp8rSZI0jrqaEBARbwbeSOfyJsDJwLUbPGcb8A7gYuBc4BURce6Kh/02cEtmngf8MvD2TTxXkiRp7HQ7W/OlwD8HvgWQmQeAUzd4zvnAPZl5b2Y+AnwI2L3iMecCny2OeRdQi4jTunyuJEnS2Ok2nD2SmQkkdJbU6OI5ZwD3Lbu9v2hb7ivALxbHPB+oAru6fC7F8xoRMRcRcwcPHuyiLEmSpPLqNpx9OCL+O/CEiPh14EbgnRs8J1ZpyxW3rwCeGBG3AK8Dvgwc7vK5ncbM2cycycyZHTt2bFCSJElSuW00W5OICOAPgXOAh4EfBX4nMz+zwVP3A2cuu70LOLD8AZn5MLA0ySCAvyl+Khs9V5IkaRxtGM4yMyPif2Tms4CNAtlye4Czi1mdfwu8HHjl8gdExBOAxWJc2b8GbsrMhyNiw+dKkiSNow3DWeELEfETmbmn2wNn5uGIuAz4FLANuCYz74iIS4v7rwaeBrw/Io4AdwK/tt5zu35XkiRJIyo64/w3eFDEnXQuZ+6jM2Mz6HSqndfX6jZpZmYm5+bmhl2GJEnShiJib2bOrGzvtufs4h7XI0mSpFV0NVszM+eBJwA/X/w8oWiTJElSD3W7Q8DrgTbw5OLn2oh4XT8LkyRJmkTdrnP2a8CzM/N3MvN3gOcAv96/sqQT1263qdVqTE1NUavVaLfbwy5JkqQNdTvmLIAjy24fYfWFYqVSaLfbNBoNFhcXAZifn6fRaABQr9eHWZokSevqtufsPcBfRMRbIuItwBeAd/etKukENZvNY8FsyeLiIs1mc0gVSZLUna56zjLz9yLic8Bz6fSYXZKZX+5nYdKJWFhY2FS7JEll0VU4i4jnAHdk5peK26dGxLMz8y/6Wp20RdPT08zPP3pC8fT09BCqkSSpe91e1vx94JvLbn+raJNKqdVqUalUjmurVCq0Wq0hVSRJUne6DWeRy7YSyMyjdD+ZQBq4er3O7Ows1WqViKBarTI7O+tkAElS6XW7fdNHgc/xvd6y1wDPz8xf6F9pm+f2TZIkaVSstX1Ttz1nlwL/DPhbYD/wbKDRu/IkSZIE3c/WfAB4eZ9rkSRJmnjdbt/0toh4XEScHBGfjYgHI+JV/S5OkiRp0nR7WfNFmfkw8BI6lzV/BHhD36qSJEmaUN2Gs5OLf18MfDAzH+pTPZIkSROt2+UwPhYRdwGHgNdExA7g2/0rS5IkaTJ11XOWmW8CfhKYyczvAovA7n4WJkmSNInWDWcR8dyl3zPz7zPzSPH7tzLza8Ukgaf3u0hJkqRJsdFlzX8REW8DPgnsBQ4CjwWeCjwfqAL/rq8VSpIkTZB1w1lm/tuIeCLwMuCXgNPpjDv7KvDfM/Pz/S9RkiRpcmw4ISAz/x54Z/EjSZKkPup2KQ1pYNrtNrVajampKWq1Gu12e9glSZI0MN0upSENRLvdptFosLi4CMD8/DyNRmcb13q9PszSJEkaCHvOVCrNZvNYMFuyuLhIs9kcUkWSJA1Wt3trViLiP0bEO4vbZ0fES/pbmibRwsLCptolSRo33facvQf4Dp2FaKGzv+b/1ZeKNNGmp6c31S5J0rjpNpydlZlvA74LkJmHgOhbVZpYrVaLSqVyXFulUqHVag2pIkmSBqvbcPZIRGwHEiAizqLTkyb1VL1eZ3Z2lmq1SkRQrVaZnZ11MoAkaWJEZm78oIgXAv8BOBf4NHAB8OrM/Fxfq9ukmZmZnJubG3YZkiRJG4qIvZk5s7K9q6U0MvMzEfEl4Dl0Lme+PjMf7HGNkiRJE6/b2ZovBQ5n5scz80+AwxHxC/0tTZIkafJ0O+bszZn5jaUbmfkPwJv7U5IkSdLk6jacrfY4dxeQJEnqsW7D2VxE/F5EnBURPxwR/xXY28/CJEmSJlG34ex1wCPAHwJ/BHwbeG2/ipIkSZpU3c7W/Bbwpj7XIkmSNPG6CmcR8SPA5UBt+XMy82f6U5YkSdJk6nZQ/x8BVwPvAo70rxxJkqTJ1m04O5yZv9/XSiRJktT1hICPRcRrIuL0iPiBpZ++ViZJkjSBuu05+5Xi3zcsa0vgh3tbjiRJ0mTrdrbmU/pdiCRJkjaxyn9EPB04F3jsUltmvr8fRUmSJE2qbpfSeDPwPDrh7BPAxcDnAcOZJElSD3U7IeBlwM8CX8vMS4BnAN/Xt6okSZImVLfh7FBmHgUOR8TjgAdwMoAkSVLPdTvmbC4ingC8k86G598Evti3qiRJkiZUt7M1X1P8enVEfBJ4XGbe2r+yJEmSJtNmZmuex7K9NSPiqZn50T7VJUmSNJG6na15DXAecAdwtGhOwHAmSZLUQ932nD0nM8/tayWSJEnqerbm/4oIw5kkSVKfddtz9j46Ae1rwHeAADIzz+tbZZIkSROo23B2DfB/ArfxvTFnkiRJ6rFuw9lCZt7Q10okSZLUdTi7KyI+AHyMzmVNAFxKQ5Ikqbe6DWfb6YSyFy1rcykNSZKkHtswnEXENuDBzHzDAOqRJEmaaBsupZGZR4AfH0AtkiRJE6/by5q3RMQNwB8B31pqdMyZJElSb3Ubzn4A+DvgZ5a1OeZMkiSpx7oKZ5l5Sb8LkSRJUpfbN0XEroi4LiIeiIivR8RHImJXv4uTJEmaNN3urfke4AZgJ3AGnfXO3rPRkyLiooi4OyLuiYg3rXL/4yPiYxHxlYi4IyIuWXbfvoi4LSJuiYi5LuvUkLTbbWq1GlNTU9RqNdrt9rBLkiRpJHU75mxHZi4PY++NiN9c7wnFEhzvAF4I7Af2RMQNmXnnsoe9FrgzM38+InYAd0dEOzMfKe5/fmY+2GWNGpJ2u02j0WBxcRGA+fl5Go0GAPV6fZilSZI0crrtOXswIl4VEduKn1fRmSCwnvOBezLz3iJsfQjYveIxCZwaEQF8P/AQcHgT9asEms3msWC2ZHFxkWazOaSKJEkaXd2Gs18F/iXwNeB+4GVF23rOAO5bdnt/0bbcVcDTgAN0NlV/fWYubayewKcjYm9ENNZ6kYhoRMRcRMwdPHiwy7ejXlpYWNhUuyRJWtu64Swi/u/i12dn5j/PzB2Z+eTM/IXMnN/g2LFKW664/XPALXTGsj0TuCoiHlfcd0Fm/jhwMfDaiPip1V4kM2czcyYzZ3bs2LFBSeqH6enpTbVLkqS1bdRz9uKIOBn4rS0cez9w5rLbu+j0kC13CfDR7LgH+BvgHIDMPFD8+wBwHZ3LpCqhVqtFpVI5rq1SqdBqtYZUkSRJo2ujcPZJ4EHgvIh4OCL+cfm/Gzx3D3B2RDwlIh4DvJzOjM/lFoCfBYiI04AfBe6NiFMi4tSi/RQ6G67fvql3poGp1+vMzs5SrVaJCKrVKrOzs04GkCRpC9YNZ5n5hsx8PPDxzHxcZp66/N8NnnsYuAz4FPBV4MOZeUdEXBoRlxYPeyvwzyLiNuCzwBuL2ZmnAZ+PiK8AXyxe/5Mn9E61ql4tgVGv19m3bx9Hjx5l3759BjNJkrZow6U0iiUxTtnKwTPzE8AnVrRdvez3A3R6xVY+717gGVt5TXXPJTAkSSqfDWdrZuYRYDEiHj+AejRALoEhSVL5dLsI7beB2yLiM8C3lhoz8zf6UpUGwiUwJEkqn27D2ceLH42R6elp5ucfvSKKS2BIkjQ8XYWzzHxfRGwHpjPz7j7XpAFptVrHjTkDl8CQJGnYutohICJ+ns5isZ8sbj8zIlYui6ER4xIYkiSVT2SuXLR/lQdF7AV+BvhcZv5Y0XZbZv7TPte3KTMzMzk3NzfsMiRJkjYUEXszc2Zle7d7ax7OzG+saNs41UmSJGlTup0QcHtEvBLYFhFnA78B/P/9K0uSJGkyddtz9jrgnwDfAT4AfAP4zX4VJUmSNKnW7TmLiMcClwJPBW4DfrLYlkmSJEl9sFHP2fuAGTrB7GLgyr5XJEmSNME2GnN27tKMzIh4N51NyCVJktQnG/WcfXfpFy9nSpIk9d9GPWfPiIiHi98D2F7cDiAz83F9rU6SJGnCrBvOMnPboAqRJElS90tpSJIkaQAMZ5IkSSViOJMkSSoRw5kkSVKJGM4kSZJKxHAmSZJUIoazEdVut6nVakxNTVGr1Wi328MuSZIk9cBGi9CqhNrtNo1Gg8XFRQDm5+dpNBoA1Ov1YZYmSZJOkD1nI6jZbB4LZksWFxdpNptDqqi39l9/PTdeeCEfe+pTufHCC9l//fXDLkmSpIGx52wELSwsbKp9lOy//npubTY5cugQAIcOHODWInTu2r17mKVJkjQQ9pyNoOnp6U21j5K7rrzyWDBbcuTQIe668sohVSRJ0mAZzkZQq9WiUqkc11apVGi1WkOqqHcO3X//ptolSRo3hrMRVK/XmZ2dpVqtEhFUq1VmZ2fHYjLA9tNP31S7JEnjxnA2our1Ovv27ePo0aPs27dvLIIZwDmXX8627duPa9u2fTvnXH75kCqSJGmwDGfqiV7NsNy1ezfntVps37kTIti+cyfntVpOBpAkTQxna+qE9XqG5a7duw1jkqSJZc+ZTpgzLCVJ6h3DmU6YMywlSeodw5lOmDMsJUnqHcOZTpgzLCVJ6h0nBOiELQ3ev+vKKzl0//1sP/10zrn8cgf1S5K0BYYz9YQzLCVJ6g0va0qSJJWI4UySJKlEDGeSJEklYjgbUb3aLqlXxymjcX5vkqTx5YSAEdSr7ZJ6ve1SmYzze5MkjTd7zkZQr7ZLGudtl8b5vUmSxpvhbAT1arukcd52aZzfmyRpvBnORlCvtksa522Xxvm9SZLGm+FsBPVqu6Rx3nZpnN+bJGm8OSFgBPVqu6Rx3nZpnN+bJGm8RWYOu4aemZmZybm5uWGXIUmStKGI2JuZMyvbvawpSZJUIoYzSZKkEjGcSZIklYjhTBoxbkslSePN2ZrSCHFbKkkaf/acSSPEbakkafwZzqQR4rZUkjT+DGfSCHFbKkkaf4YzaQNlGoDvtlSSNP4MZwPWbrep1WpMTU1Rq9Vot9vDLknrWBqAf+jAAcg8NgB/WAFt1+7dnNdqsX3nTohg+86dnNdqORlAksaI2zcNULvdptFosLi4eKytUqkwOztLvV4fYmVay40XXtgJZits37mTF9x88xAqkiSNC7dvKoFms3lcMANYXFykWSyFoPJxAL4kadAMZwO0sLCwqXYNnwPwJUmDZjgboOnp6U21a/gcgC9JGjTD2QC1Wi0qlcpxbZVKhVarNaSKtJFeDsAv06xPSVJ59XX7poi4CHg7sA14V2ZeseL+xwPXAtNFLVdm5nu6ee4oWhr032w2WVhYYHp6mlar5WSAktu1e/cJz4Z02yVJUrf6NlszIrYBfwm8ENgP7AFekZl3LnvMbwOPz8w3RsQO4G7gh4AjGz13NWWfranJ5axPSdJKw5iteT5wT2bem5mPAB8CVnYRJHBqRATw/cBDwOEunyuNDGd9SpK61c9wdgZw37Lb+4u25a4CngYcAG4DXp+ZR7t8rjQynPUpSepWP8NZrNK28hrqzwG3ADuBZwJXRcTjunxu50UiGhExFxFzBw8ePJF6pb5x1qckqVv9DGf7gTOX3d5Fp4dsuUuAj2bHPcDfAOd0+VwAMnM2M2cyc2bHjh09K17qJbddkiR1q5+zNfcAZ0fEU4C/BV4OvHLFYxaAnwVujojTgB8F7gX+oYvnSiOlF7M+JUnjr2/hLDMPR8RlwKfoLIdxTWbeERGXFvdfDbwVeG9E3EbnUuYbM/NBgNWe269aJUmSysKNzyVJkobAjc8lSZJGgOFMkiSpRAxnkiRJJWI4kyRJKhHDmSRJUokYziRJkkrEcCZNqP3XX8+NF17Ix576VG688EL2X3/9sEuSJNHfHQIkldT+66/n1maTI4cOAXDowAFubTYB3MVAkobMnrMutdttarUaU1NT1Go12u32sEuStuyuK688FsyWHDl0iLuuvHJIFUmSlthz1oV2u02j0WBxcRGA+fl5Go0GAPV6fZilSVty6P77N9UuSRoce8660Gw2jwWzJYuLizSLy0DSqNl++umbapckDY7hrAsLCwubapfK7pzLL2fb9u3HtW3bvp1zLr98SBWpG07ikCaD4awL09PTm2qXym7X7t2c12qxfedOiGD7zp2c12o5GaDEliZxHDpwADKPTeIwoEnjJzJz2DX0zMzMTM7NzfX8uCvHnAFUKhVmZ2cdcyZpIG688MJOMFth+86dvODmm4dQkaQTFRF7M3NmZbs9Z12o1+vMzs5SrVaJCKrVqsFM0kA5iUOaHM7W7FK9XjeMSRqa7aefvnrPmZM4pLFjz5kkjQAncUiTw3AmSWso0+xIJ3FIk8PLmpK0ijJucbVr927DmDQB7DmTpFW4xZWkYTGcSdIqnB0paVgMZ5K0Cre4kjQshrMulWlgsKT+6+XsSL8/JG2GEwK6UMaBwZL6a+lv+64rr+TQ/fez/fTTOefyyzf9N+/3h6TNcvumLrhtiqSt8vtD0lrcvukEODBY0lb5/SFpswxnXXBgsKSt8vtD0mYZzrrgtimStsrvD0mbZTjrgtumSGtzJuL6/P7QOPLvvr+cECBpy1bORIROr5DhQxpf/t33jhMCJPWcWxxJk8e/+/4znEnaMmciSpPHv/v+M5xJ2jJnIkqTx7/7/jOcSdqyMs5EdKDyxjxHOhFl/LsfN27fJGnLerXFUa+4VdLGPEc6UWX7ux9HztaUNDbcKmljniOpPJytKWnsOVB5Y54jqfwMZ5LGhgOVN+Y5ksrPcCZpbDhQeWOeI6n8DGeSSqEXMwjdKmljnqPR5Azb9Y3b+XFCgKShczsYaW3+faxvlM+PEwIklZbbwUhr8+9jfeN4fgxnkobOGYTS2vz7WN84nh/DmaShcwahtDb/PtY3jufHcDZg4zZoUeoFZxCqTMr2Pe3fx/rG8fy4fdMAuW2KtDq3g1FZlPF72r+P9Y3j+XG25gC5bYoklZvf0xokZ2uWwDgOWpSkceL3tMrAcDZA4zhoUZLGid/TKgPD2XfGuakAAAtcSURBVACN46BFSRonfk+rDAxnA+S2KZL0aGWaHdnr7+kyvbcy8vyszgkBkqShGeWtdzYyzu+tFzw/TgiQJJXQOG69s2Sc31sveH7WZjiTJA3NOM+OHOf31guen7UZziRJQzPOsyPH+b31gudnbYYzSdKW9GIw9zjPjizjeyvTAHzPz9rcvkmStGm92uZoHLfeWVK291a2rak8P2tztqYkadPc5mj0+N9sfcM4P87WlCT1jIO5R4//zdZXpvNjOJMkbZqDuUeP/83WV6bzYziTJG1aGQdza33+N1tfmc6PEwIkSZtWtsHc0BnQXaZ6yqaM/83KpEznxwkBkqSR51ZAGkVOCJAkjS23AtI46Ws4i4iLIuLuiLgnIt60yv1viIhbip/bI+JIRPxAcd++iLituM/uMEnSmso00046UX0LZxGxDXgHcDFwLvCKiDh3+WMy83cz85mZ+Uzgt4A/z8yHlj3k+cX9j+rykyRpSZlm2kknqp89Z+cD92TmvZn5CPAhYL0L/68APtjHeiRJY6pMM+2kE9XPcHYGcN+y2/uLtkeJiApwEfCRZc0JfDoi9kZEo29VSpJG3q7duzmv1WL7zp0QwfadO50MoJHVz6U0YpW2taaG/jzw/624pHlBZh6IiCcDn4mIuzLzpke9SCe4NQCmp6dPtGZJ0ojatXu3YUxjoZ89Z/uBM5fd3gU8etOqjpez4pJmZh4o/n0AuI7OZdJHyczZzJzJzJkdO3accNGSJEnD1M9wtgc4OyKeEhGPoRPAblj5oIh4PPDTwPXL2k6JiFOXfgdeBNzex1olSZJKoW+XNTPzcERcBnwK2AZck5l3RMSlxf1XFw99KfDpzPzWsqefBlwXEUs1fiAzP9mvWiVJksrCHQIkSZKGwB0CJEmSRoDhTJIkqUQMZ5IkSSViOJMkSSoRw5kkSVKJGM4kSZJKxHAmSZJUIoYzSZKkEhmrRWgj4iAwv8bdTwIeHGA5k8rzPDie68HxXA+G53lwPNeDsdF5rmbmozYGH6twtp6ImFttFV71lud5cDzXg+O5HgzP8+B4rgdjq+fZy5qSJEklYjiTJEkqkUkKZ7PDLmBCeJ4Hx3M9OJ7rwfA8D47nejC2dJ4nZsyZJEnSKJiknjNJkqTSM5xJkiSVyNiHs4i4KCLujoh7IuJNw65nnEXEvoi4LSJuiYi5YdczTiLimoh4ICJuX9b2AxHxmYj4q+LfJw6zxnGwxnl+S0T8bfG5viUiXjzMGsdFRJwZEX8WEV+NiDsi4vVFu5/rHlrnPPu57rGIeGxEfDEivlKc6/9UtG/6Mz3WY84iYhvwl8ALgf3AHuAVmXnnUAsbUxGxD5jJTBc27LGI+Cngm8D7M/PpRdvbgIcy84ri/3g8MTPfOMw6R90a5/ktwDcz88ph1jZuIuJ04PTM/FJEnArsBX4BeDV+rntmnfP8L/Fz3VMREcApmfnNiDgZ+DzweuAX2eRnetx7zs4H7snMezPzEeBDwO4h1yRtWmbeBDy0onk38L7i9/fR+cLVCVjjPKsPMvP+zPxS8fs/Al8FzsDPdU+tc57VY9nxzeLmycVPsoXP9LiHszOA+5bd3o8fyn5K4NMRsTciGsMuZgKclpn3Q+cLGHjykOsZZ5dFxK3FZU8vs/VYRNSAHwP+Aj/XfbPiPIOf656LiG0RcQvwAPCZzNzSZ3rcw1ms0ja+13GH74LM/HHgYuC1xSUiadT9PnAW8EzgfuC/DLec8RIR3w98BPjNzHx42PWMq1XOs5/rPsjMI5n5TGAXcH5EPH0rxxn3cLYfOHPZ7V3AgSHVMvYy80Dx7wPAdXQuK6t/vl6MJ1kaV/LAkOsZS5n59eIL9yjwTvxc90wxLucjQDszP1o0+7nusdXOs5/r/srMfwA+B1zEFj7T4x7O9gBnR8RTIuIxwMuBG4Zc01iKiFOKwaZExCnAi4Db13+WTtANwK8Uv/8KcP0QaxlbS1+qhZfi57onisHT7wa+mpm/t+wuP9c9tNZ59nPdexGxIyKeUPy+HXgBcBdb+EyP9WxNgGJ68P8DbAOuyczWkEsaSxHxw3R6ywBOAj7gue6diPgg8DzgScDXgTcD/wP4MDANLAC/lJkOZj8Ba5zn59G59JPAPuDfLI0f0dZFxHOBm4HbgKNF82/TGQ/l57pH1jnPr8DPdU9FxHl0Bvxvo9P59eHM/M8R8YNs8jM99uFMkiRplIz7ZU1JkqSRYjiTJEkqEcOZJElSiRjOJEmSSsRwJkmSVCKGM0kDERE/FBEfioi/jog7I+ITEfEjfXy950XEn2zxuZdGxC8Xv786InZu4Rh/XCwxs7L91RFx1RbrqkXEK7s5VkTc6JY80mgynEnqu2IhzOuAz2XmWZl5Lp21lk4bbmWry8yrM/P9xc1XA5sKZxHxT4BtmXlvj0urAa/c6EGFPwBe0+PXlzQAhjNJg/B84LuZefVSQ2bekpk3R8fvRsTtEXFbRPwrONbz9ecR8eGI+MuIuCIi6hHxxeJxZxWPe29EXB0RNxePe8nKFy92sLgmIvZExJcjYnfR/t8i4neK338uIm6KiKmIeEtEXB4RLwNmgHZE3BIR/0dEXLfsuC+MiI+ufD2gzrJVwCPikqK2PwcuWNa+IyI+UtS1JyIuKNrfEhF/EBF/GhF/FRG/XjzlCuDCopZ/W7TtjIhPFo9727IabqCz0KikEXPSsAuQNBGeDuxd475fpLNS+TPorMy/JyJuKu57BvA04CHgXuBdmXl+RLweeB3wm8XjasBP09nI+c8i4qkrXqMJ/Glm/mqxvcoXI+JG4E3F690M/DfgxZl5tNPRB5n5xxFxGXB5Zs4VPYD/JSJ2ZOZB4BLgPau8pwuAD8KxbXL+E/As4BvAnwFfLh73duC/ZubnI2Ia+FTxfgHOA54DnAJ8OSI+XtR7eWa+pDj2q4tz92PAd4C7I+L/zcz7MvPvI+L7IuIHM/Pv1jj3kkrInjNJw/Zc4IPFJsxfB/4c+Inivj2ZeX9mfgf4a+DTRfttdALZkg9n5tHM/Cs6Ie6cFa/xIuBNEXELnc2IHwtMZ+Yi8OvAZ4CrMvOv1ys0O1uq/AHwqiLk/STwP1d56OnAweL3Z9O5nHswMx8B/nDZ414AXFXUdQPwuKU9aoHrM/NQZj5IJ9CttTH1ZzPzG5n5beBOoLrsvgfY5CVZScNnz5mkQbgDeNka98U6z/vOst+PLrt9lOO/v1buQ7fydgD/IjPvXuU1/inwd3QfYt4DfAz4NvBHmXl4lcccohMA16pnyRTwk5l56LhiOz13G72nJcvP0RGOPy+PLWqRNELsOZM0CH8KfN+ysVNExE9ExE8DNwH/KiK2RcQO4KeAL27y+L9UjBU7C/hhYGUI+xTwuuKyJBHxY8W/VeDf0bkseHFEPHuVY/8jsNSbRWYeAA4A/wF47xr1fBVYurT6F8DzIuIHI+Jk4JeWPe7TwGVLNyLimcvu2x0Rjy02TX4esGdlLesp3usP0dnUWtIIMZxJ6rvicuBLgRcWS2ncAbyFTsi5DrgV+AqdEPfvM/Nrm3yJu+lcDv2fwKXFJb7l3gqcDNwaEbcDby3Cy7vpjOE6APwa8K6IeOyK574XuLoYhL+9aGsD92XmnWvU83E6gYrMvL94r/8LuBH40rLH/QYwExG3RsSdwKXL7vticZwvAG8tarwVOBwRX1k2IWAtzwK+sEbPnqQSi853piSNpoh4L/AnmfnHA3zNq4AvZ+a717h/O51xYhdk5pEtHP8twDcz88oTqPHtwA2Z+dmtHkPScNhzJkmbEBF76cykvHatxxRjyN4MnDGoulZxu8FMGk32nEmSJJWIPWeSJEklYjiTJEkqEcOZJElSiRjOJEmSSsRwJkmSVCL/G9MY7VuJiiTxAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "fig, ax = plt.subplots(1, figsize = (10, 7))\n", "\n", "for d in range(1, 30):\n", " T = tree.DecisionTreeClassifier(max_depth = d)\n", "\n", " T.fit(X_train, y_train)\n", " \n", " ax.scatter(d, T.score(X_train, y_train), color = \"black\")\n", " ax.scatter(d, T.score(X_test, y_test), color = \"firebrick\")\n", "\n", "ax.set(xlabel = \"Complexity (depth)\", ylabel = \"Performance (score)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Observe that the training score (black) always increases, while the test score (red) tops out around 83\\% and then even begins to trail off slightly. It looks like the optimal depth might be around 5-7 or so, but there's some random noise that can prevent us from being able to determine exactly what the optimal depth is. \n", "\n", "Increasing performance on the training set combined with decreasing performance on the test set is the trademark of overfitting. \n", "\n", "This noise reflects the fact that we took a single, random subset of the data for testing. In a more systematic experiment, we would draw many different subsets of the data for each value of depth and average over them. This is what *cross-validation* does, and we'll talk about it in the next lecture." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }