{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Fine-tuning your model\n", "> A Summary of lecture \"Supervised Learning with scikit-learn\", via datacamp\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Chanseok Kang\n", "- categories: [Python, Datacamp, Machine_Learning]\n", "- image: images/roc-curve.png" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How good is your model?\n", "- Classification metrics\n", " - Measuring model performance with accuracy:\n", " - Fraction of correctly classified samples\n", " - Not always a useful metrics\n", "- Class imbalance example: Emails\n", " - Spam classification\n", " - 99% of emails are real; 1% of emails are spam\n", " - Could build a classifier that predicts ALL emails as real\n", " - 99% accurate!\n", " - But horrible at actually classifying spam\n", " - Fails at its original purpose\n", "- Diagnosing classification predictions\n", "\n", " - Confusion matrix\n", "![cm](image/confusion_matrix.png)\n", "\n", " - Accuracy:\n", " $$ \\dfrac{tp + tn}{tp + tn + fp + fn} $$\n", " \n", " - Precision (Positive Predictive Value):\n", " $$ \\dfrac{tp}{tp + fp}$$\n", " \n", " - Recall (Sensitivity, hit rate, True Positive Rate):\n", " $$ \\dfrac{tp}{tp + fn}$$\n", " \n", " - F1 score: Harmonic mean of precision and recall\n", " $$ 2 \\cdot \\dfrac{\\text{precision} \\cdot \\text{recall}}{\\text{precision} + \\text{recall}} $$\n", " \n", " - High precision : Not many real emails predicted as spam\n", " - High recall : Predicted most spam emails correctly\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Metrics for classification\n", "Accuracy is not always an informative metric. In this exercise, you will dive more deeply into evaluating the performance of binary classifiers by computing a confusion matrix and generating a classification report.\n", "\n", "You may have noticed in the video that the classification report consisted of three rows, and an additional support column. The support gives the number of samples of the true response that lie in that class - so in the video example, the support was the number of Republicans or Democrats in the test set on which the classification report was computed. The precision, recall, and f1-score columns, then, gave the respective metrics for that particular class.\n", "\n", "Here, you'll work with the [PIMA Indians](https://www.kaggle.com/uciml/pima-indians-diabetes-database) dataset obtained from the UCI Machine Learning Repository. The goal is to predict whether or not a given female patient will contract diabetes based on features such as BMI, age, and number of pregnancies. Therefore, it is a binary classification problem. A target value of 0 indicates that the patient does not have diabetes, while a value of 1 indicates that the patient does have diabetes. As in Chapters 1 and 2, the dataset has been preprocessed to deal with missing values." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Preprocess" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pregnanciesglucosediastolictricepsinsulinbmidpfagediabetes
061487235033.60.627501
11856629026.60.351310
28183640023.30.672321
318966239428.10.167210
40137403516843.12.288331
\n", "
" ], "text/plain": [ " pregnancies glucose diastolic triceps insulin bmi dpf age \\\n", "0 6 148 72 35 0 33.6 0.627 50 \n", "1 1 85 66 29 0 26.6 0.351 31 \n", "2 8 183 64 0 0 23.3 0.672 32 \n", "3 1 89 66 23 94 28.1 0.167 21 \n", "4 0 137 40 35 168 43.1 2.288 33 \n", "\n", " diabetes \n", "0 1 \n", "1 0 \n", "2 1 \n", "3 0 \n", "4 1 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_csv('./dataset/diabetes.csv')\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "X = df.iloc[:, :-1]\n", "y = df.iloc[:, -1]" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[176 30]\n", " [ 56 46]]\n", " precision recall f1-score support\n", "\n", " 0 0.76 0.85 0.80 206\n", " 1 0.61 0.45 0.52 102\n", "\n", " accuracy 0.72 308\n", " macro avg 0.68 0.65 0.66 308\n", "weighted avg 0.71 0.72 0.71 308\n", "\n" ] } ], "source": [ "from sklearn.metrics import classification_report, confusion_matrix\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.neighbors import KNeighborsClassifier\n", "\n", "# Create training and test set\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)\n", "\n", "# Instantiate a k-NN classifier: knn\n", "knn = KNeighborsClassifier(n_neighbors=6)\n", "\n", "# Fit the classifier to the training data\n", "knn.fit(X_train, y_train)\n", "\n", "# Predict the labels of the test data: y_pred\n", "y_pred = knn.predict(X_test)\n", "\n", "# Generate the confusion matrix and classification report\n", "print(confusion_matrix(y_test, y_pred))\n", "print(classification_report(y_test, y_pred))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Logistic regression and the ROC curve\n", "- Logistic regression for binary classification\n", " - Logistic regression outputs probabilities\n", " - If the probability is greater than 0.5:\n", " - The data is labeled '1'\n", " - If the probability is less than 0.5:\n", " - The data is labeled '0'\n", "- Probability thresholds\n", " - By default, logistic regression threshold = 0.5\n", " - Not specific to logistic regression\n", " - k-NN classifiers also have thresholds\n", "- ROC curves (Receiver Operating Characteristic curve)\n", "![roc](./image/roc.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Building a logistic regression model\n", "Time to build your first logistic regression model! As Hugo showed in the video, scikit-learn makes it very easy to try different models, since the Train-Test-Split/Instantiate/Fit/Predict paradigm applies to all classifiers and regressors - which are known in scikit-learn as 'estimators'. You'll see this now for yourself as you train a logistic regression model on exactly the same data as in the previous exercise. Will it outperform k-NN? There's only one way to find out!" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[168 38]\n", " [ 36 66]]\n", " precision recall f1-score support\n", "\n", " 0 0.82 0.82 0.82 206\n", " 1 0.63 0.65 0.64 102\n", "\n", " accuracy 0.76 308\n", " macro avg 0.73 0.73 0.73 308\n", "weighted avg 0.76 0.76 0.76 308\n", "\n" ] } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "\n", "# Create training and test sets\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)\n", "\n", "# Create the classifier: logreg\n", "logreg = LogisticRegression(max_iter=1000)\n", "\n", "# Fit the classifier to the training data\n", "logreg.fit(X_train, y_train)\n", "\n", "# Predict the labels of the test set: y_pred\n", "y_pred = logreg.predict(X_test)\n", "\n", "# Compute and print the confusion matrix and classification report\n", "print(confusion_matrix(y_test, y_pred))\n", "print(classification_report(y_test, y_pred))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plotting an ROC curve\n", "Great job in the previous exercise - you now have a new addition to your toolbox of classifiers!\n", "\n", "Classification reports and confusion matrices are great methods to quantitatively evaluate model performance, while ROC curves provide a way to visually evaluate models. As Hugo demonstrated in the video, most classifiers in scikit-learn have a ```.predict_proba()``` method which returns the probability of a given sample being in a particular class. Having built a logistic regression model, you'll now evaluate its performance by plotting an ROC curve. In doing so, you'll make use of the ```.predict_proba()``` method and become familiar with its functionality." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU9fX/8dcxccOqVUErsu8JixajiKiIKIvFirYoyhelBihQ3NCquCBS9QcIouybCCKLSkGx5VtqtVa/VEQERIgikR1RFlnEhSWc3x8zsdOYZUJyM5mZ9/PxmIdzZ+7MPTfEnPtZ7vmYuyMiIsnrmFgHICIisaVEICKS5JQIRESSnBKBiEiSUyIQEUlySgQiIklOiUBEJMkpEUhCMbMNZva9me03sy/NbKqZ/SzPPheb2Vtm9o2Z7TWz180sPc8+p5jZM2a2Kfxd2eHtigUc18zsDjNbZWbfmtkWM3vFzBoHeb4ipUGJQBLRNe7+M+A84JdA/9w3zKw58HfgNaAyUBP4CFhkZrXC+xwHvAk0BNoBpwAXA7uACws45rPAncAdwOlAPeBV4FfFDd7MUov7GZGSMN1ZLInEzDYA3d39H+HtoUBDd/9VePtd4GN375Pnc/8L7HD3W8ysO/AEUNvd90dxzLrAp0Bzd19SwD5vAy+6++TwdrdwnJeEtx3oC9wFpAILgf3ufm/Ed7wG/MvdnzazysAo4DJgPzDC3UdG8SMS+Qm1CCRhmVkVoD2QHd6uQOjK/pV8dn8ZuCr8/Ergb9EkgbDWwJaCkkAxdASaAenATOBGMzMAMzsNaAPMNrNjgNcJtWTOCR//LjNrW8LjS5JSIpBE9KqZfQNsBrYDj4ZfP53Q7/y2fD6zDcjt/z+jgH0KUtz9C/L/3P1rd/8eeBdw4NLwe78F3nP3L4ALgEruPsjdD7r7OmAS0LkUYpAkpEQgiaiju58MXA404D9/4HcDR4Cz8/nM2cDO8PNdBexTkOLuX5DNuU881Gc7G7gp/NLNwIzw8+pAZTPbk/sAHgTOKoUYJAkpEUjCcvd/AVOBYeHtb4H3gE757H4DoQFigH8Abc3spCgP9SZQxcwyCtnnW6BCxPYv8gs5z/Ys4LdmVp1Ql9Gfw69vBta7+88jHie7+9VRxivyX5QIJNE9A1xlZueFtx8Abg1P9TzZzE4zs8eB5sBj4X2mE/pj+2cza2Bmx5jZGWb2oJn95I+tu68FxgKzzOxyMzvOzE4ws85m9kB4txXA9WZWwczqAJlFBe7uy4EdwGRgobvvCb+1BNhnZveb2YlmlmJmjczsgqP5AYkoEUhCc/cdwAvAI+Ht/wPaAtcT6tffSGiK6SXhP+i4+wFCA8afAm8A+wj98a0IvF/Aoe4ARgNjgD3A58B1hAZ1AUYAB4GvgGn8p5unKLPCscyMOKcc4BpC02PXE+rSmgycGuV3ivwXTR8VEUlyahGIiCQ5JQIRkSSnRCAikuSUCEREklzcFbeqWLGi16hRI9ZhiIjElQ8//HCnu1fK7724SwQ1atRg6dKlsQ5DRCSumNnGgt5T15CISJJTIhARSXJKBCIiSU6JQEQkySkRiIgkucASgZlNMbPtZraqgPfNzEaGFwVfaWZNg4pFREQKFmSLYCqhhb8L0h6oG370BMYFGIuIiBQgsPsI3P0dM6tRyC7XAi+EV2JabGY/N7Oz3b00lvwTESk3Zr6/iddWbD3qzx85ksPBg4doWutMHr2mYSlGFhLLMYJziFiaD9gSfu0nzKynmS01s6U7duwok+BERErLayu2krVt31F9ds+ePXzwwVJWr15NUMsGxPLOYsvntXzP0t0nAhMBMjIytICCiAAlv9IuK1nb9pF+9im89PvmUX9mz549/PGPf+TlyZOpU6cOkydPpmXLRoHEF8tEsAWoGrFdBfgiRrGISBzKvdJOP/uUWIdSqPSzT+Ha8/Lt8MhXTk4OF198MWvWrOG+++5j4MCBnHjiiYHFF8tEMB/oa2azCS3MvVfjAyLlS3m/4j6aK+3ybNeuXZx++umkpKTwxBNPULVqVTIyMgI/bpDTR2cB7wH1zWyLmWWaWS8z6xXeZQGwDsgGJgF9gopFRI5OSfq2y0Jxr7TLK3fnxRdfpF69ekyePBmA6667rkySAAQ7a+imIt534A9BHV9Eiie/q/9Eu+IujzZv3kyvXr1YsGABF110ES1atCjzGHRnsYgA+V/9J8oVd3k1a9YsGjZsyNtvv80zzzzD//3f/5Genl7mccTdegQi8lOl0Zevq/+yd9ppp9GsWTMmTpxIzZo1YxaHEoFIAiiN2TO6+g/e4cOHGTFiBAcPHuShhx6iXbt2tG3bFrP8ZtOXHSUCkThS0JW/rubLv48++ojMzEw+/PBDbrjhBtwdM4t5EgCNEYjElYJm8ehqvvw6cOAAjzzyCBkZGWzevJlXXnmF2bNnl4sEkEstApGABDEHX1f+8Wft2rUMGTKEm2++maeffpozzjgj1iH9hFoEIgEJYg6+rvzjw/79+5kxYwYAjRo14tNPP2XatGnlMgmAWgQigdLVe/J544036NmzJxs3bqRp06akpaVRq1atWIdVKLUIRERKwe7du8nMzKRNmzYcd9xx/Otf/yItLS3WYUVFLQKRUhQ5LhAPxdCkdOTk5NCiRQs+++wz+vfvz4ABAzjhhBNiHVbUlAhESlHkfH715ye+nTt3/lgk7sknn6RatWo0bRp/q+4qEYgU4Ghm/WhWT3Jwd6ZPn85dd93F4MGD6dmzJx07dox1WEdNYwQiBTiaWT9qBSS+jRs30r59e2699VbS0tK47LLLYh1SialFIIIqb0p0XnzxRXr37o27M2rUKPr06cMxx8T/9XT8n4FIKVDlTYlGpUqVaNGiBatXr6Zv374JkQRALQJJYvnN8NHVv0Q6dOgQw4cP59ChQzzyyCO0bduWNm3alKvyEKUhMdKZyFGIbAXo6l/yWr58Oc2aNaN///5kZWURWkuLhEsCoBaBJDm1AiSvH374gUGDBjF06FAqVqzIn//8Z66//vpYhxUoJQKJayUp7KYbviQ/2dnZDBs2jFtuuYXhw4dz2mmnxTqkwKlrSOJaSQq7qTtIcu3fv5/p06cDoSJxa9asYcqUKUmRBEAtAkkA6t6Rkli4cCE9e/Zk8+bNZGRkkJaWFtNlI2NBLQIRSUq7du3i1ltvpV27dlSoUIF33303borElTYlAolbM9/fxPvrv451GBKHcovEzZgxg4ceeojly5fTokWLWIcVM+oakriVO0isfn6J1o4dOzjjjDNISUlhyJAhVK9enfPOOy/WYcWcWgQSl3JbA81qns7NzarFOhwp59yd559/nnr16jFp0iQArr32WiWBMCUCiUtqDUi0NmzYQNu2bbntttto3LgxrVq1inVI5Y66hiRu5C0JodaAFGX69On07t0bM2Ps2LH8/ve/T5j6QKVJPxGJGyoJIcV11llncdlll7F69Wp69+6tJFAAtQgkcCW5+zeSCsNJUQ4dOsTQoUPJyclhwIABtGnThjZt2sQ6rHJP6VECV5K7fyOpFSCFWbZsGRdccAEPP/wwa9as+bFInBRNLQIJTG5LQFfyEqTvv/+exx57jGHDhlGpUiXmzZsX18tGxkKgLQIza2dma8ws28weyOf9amb2TzNbbmYrzezqIOORshWZBHQlL0FZt24dTz/9NN26dSMrK0tJ4CgE1iIwsxRgDHAVsAX4wMzmu3tWxG4PAy+7+zgzSwcWADWCikmClXcsQC0BCcq+ffuYO3cu3bp1o2HDhqxdu5bq1avHOqy4FWSL4EIg293XuftBYDZwbZ59HMitA3wq8EWA8UjA8o4FqCUgQViwYAGNGjUiMzOTTz75BEBJoISCHCM4B9gcsb0FaJZnn4HA383sduAk4Mr8vsjMegI9AapV07zx8kwtAAnKzp07ufvuu3nxxRdJT09n0aJFSVskrrQF2SLIbz23vMP4NwFT3b0KcDUw3cx+EpO7T3T3DHfPqFSpUgChikh5llskbvbs2QwYMIBly5Zx0UUXxTqshBFki2ALUDViuwo/7frJBNoBuPt7ZnYCUBHYHmBcUoryWwBepLR89dVXVKpUiZSUFIYNG0b16tVp0qRJrMNKOEG2CD4A6ppZTTM7DugMzM+zzyagNYCZpQEnADsCjElKme72lSC4O8899xz169dn4sSJAFxzzTVKAgEJrEXg7ofNrC+wEEgBprj7ajMbBCx19/nAPcAkM7ubULdRN9ddIHEjsgKoxgWktKxbt44ePXrw1ltv0bJlS668Mt+hQylFgd5Q5u4LCE0JjXxtQMTzLCB5V4OIc6oAKqVt2rRp9OnTh5SUFMaPH0+PHj1UH6gM6M5iKRZVAJUgVa5cmSuuuIJx48ZRpUqVWIeTNJQIpFgi7xbWmICU1MGDBxk8eDBHjhxh4MCBXHXVVVx11VWxDivpKBFIseleASkNH3zwAbfddhurVq2ia9euuDtm+c06l6Cp801EytR3333Hvffey0UXXcTu3buZP38+L7zwgpJADKlFIAXKbx0B3SsgJbV+/XpGjRpFjx49GDJkCKeeemqsQ0p6ahFIgfJbR0DjAnI09u7dy/PPPw9Aw4YNyc7OZvz48UoC5YRaBFIojQdISf31r3/l97//Pdu2baN58+Y0aNCAqlWrFv1BKTNqEYhIIHbs2EGXLl3o0KEDp512Gu+99x4NGjSIdViSD7UIRKTU5eTkcMkll7B+/Xoee+wxHnjgAY477rhYhyUFiCoRhGsFVXP37IDjkRhTETkpiS+//JIzzzyTlJQUhg8fTo0aNWjUqFGsw5IiFNk1ZGa/Aj4G3ghvn2dm84IOTGJDReTkaBw5coQJEyZQr149JkyYAECHDh2UBOJENC2CQYQWlPkngLuvMLM6gUYlZSq/VoAGiCVa2dnZ9OjRg7fffpsrrriCtm3bxjokKaZoBosPufuePK+pQmgCUStAjtbzzz9P48aNWbZsGZMmTeIf//gHtWrVinVYUkzRtAg+MbMbgGPMrCZwJ7A42LCkrKiUtJREtWrVaNu2LWPGjOGcc3QBEa+iaRH0Bc4HjgBzgR8IJQNJAColLcVx4MABBg4cyIABoWryrVu35tVXX1USiHPRJIK27n6/u/8y/HgAaB90YFJ2VEpaovH+++9z/vnn89hjj7Fp0ya0hlTiiCYRPJzPaw+VdiAiUj59++239OvXj+bNm7N3717+8pe/MHXqVBWJSyAFjhGYWVtCC8ufY2ZPR7x1CqFuIoljuTOFdK+AFGXjxo2MHTuWXr16MXjwYE45Rb8viaawweLtwCpCYwKrI17/BnggyKAkeJFJQOMDkteePXuYM2cO3bt3Jz09nezsbK0YlsAKTATuvhxYbmYz3P2HMoxJyojuF5D8vPbaa/Tu3Zvt27dzySWX0KBBAyWBBBfNGME5ZjbbzFaa2We5j8AjE5EytX37djp37kzHjh2pVKkSixcvVpG4JBHNfQRTgceBYYRmC/0OjRHEjfwWlwHVEZL/lpOTQ4sWLdi0aROPP/449913H8cee2ysw5IyEk0iqODuC81smLt/DjxsZu8GHZiUjoIGhDU2IABffPEFv/jFL0hJSeHZZ5+lRo0apKenxzosKWPRJIIDFpon9rmZ9QK2AmcGG5YUV1FX/hoLkEi5ReLuv/9+Bg8eTJ8+fbj66qtjHZbESDRjBHcDPwPuAFoAPYDbggxKii+/ZSVBV/7yU5999hmtWrWiT58+NGvWjPbtdX9osiuyReDu74effgN0BTAzTSEoh3TlL0V57rnn6Nu3LyeccAJTpkyhW7duujFMCm8RmNkFZtbRzCqGtxua2Quo6JxIXKpRowbt27cnKyuL3/3ud0oCAhR+Z/H/A34DfERogHgeoWJzQ4BeZROe5CpoDCCXZgFJfg4cOMCf/vQnAB5//HFat25N69atYxyVlDeFdQ1dC5zr7t+b2enAF+HtNWUTmkQqqhyExgIkr3//+99kZmby6aefctttt+HuagFIvgpLBD+4+/cA7v61mX2qJFA28rv61+wfidb+/ft56KGHGDVqFFWrVuVvf/ubVg2TQhU2RlDLzOaGH/OAGhHbc6P5cjNrZ2ZrzCzbzPKtT2RmN5hZlpmtNrOZR3MSiSa/GUC64pdobdq0iQkTJvCHP/yBVatWKQlIkQprEfwmz/bo4nyxmaUAY4CrgC3AB2Y2392zIvapC/QHWrj7bjPT/QlhuvqX4ti9ezevvPIKPXv2JD09nXXr1lG5cuVYhyVxorCic2+W8LsvBLLdfR2Amc0mNO6QFbFPD2CMu+8OH3N7CY8pknTmzZtHnz592LFjBy1btqR+/fpKAlIs0dxQdrTOATZHbG8JvxapHlDPzBaZ2WIza5ffF5lZTzNbamZLd+zYEVC4IvHlyy+/pFOnTlx//fX84he/YMmSJdSvXz/WYUkciqbExNHKb3pC3rXtUoG6wOVAFeBdM2vk7nv+60PuE4GJABkZGVofT5JeTk4Ol156KZs3b+bJJ5/k3nvvVZE4OWpRJwIzO97dDxTju7cAVSO2qxCagpp3n8XufghYb2ZrCCWGD4pxnLilyqBSXFu2bKFy5cqkpKQwcuRIatasqVLRUmJFdg2Z2YVm9jGwNrx9rpmNiuK7PwDqmllNMzsO6AzMz7PPq0Cr8PdWJNRVtK4Y8cc11QeSaB05coRRo0bRoEEDxo0bB0D79u2VBKRURNMiGAl0IPRHG3f/yMxaFfUhdz9sZn2BhUAKMMXdV5vZIGCpu88Pv9fGzLKAHOCP7r7rKM8lLml2kBTl008/pXv37ixatIi2bdvSoUOHWIckCSaaRHCMu2/Mc0diTjRf7u4LgAV5XhsQ8dyBfuGHiOQxefJk+vbtS4UKFZg2bRpdu3bV3cFS6qJJBJvN7ELAw/cG3A5oqUqRMlC7dm2uueYaRo8ezVlnnRXrcCRBRZMIehPqHqoGfAX8I/yaiJSyH374gUGDBgHw5JNP0qpVK1q1KrInVqREokkEh929c+CRiCS5RYsWkZmZyZo1a+jevbuKxEmZiSYRfBCe1vkSMNfdvwk4prhXVMnoXJomKgDffPMNDz74IGPGjKF69eosXLiQNm3axDosSSJFTh9199rA48D5wMdm9qqZqYVQiIKmhealaaICoXsDJk+ezO23387HH3+sJCBlLqobytz938C/zWwg8AwwA5gdYFxxT9NCpTC7du3i5Zdfpnfv3qSlpbFu3TrOPvvsWIclSSqaG8p+ZmZdzOx1YAmwA7g48MhEEpC7M2fOHNLT07njjjtYsya0xIeSgMRSNEXnVgEXAUPdvY673xOxoL2IRGnbtm385je/oVOnTlStWpWlS5eqSJyUC9F0DdVy9yOBRyKSwHKLxG3dupWhQ4dy9913k5oaZM1HkegVtnj9cHe/B/izmf2k4qe7Xx9oZCIJYPPmzZxzzjmkpKQwZswYatasSb169WIdlsh/KeyS5KXwf4u1MpmIhFoAY8aMoX///gwdOpQ//OEPWjJSyq3CVihbEn6a5u7/lQzCxeRKuoJZwsm9f0D3ByS3Tz75hMzMTN577z3at2/PNddcE+uQRAoVzWDxbfm8llnagSSCyCSg+wOS08SJEznvvPP47LPPmD59On/961+pVq1arMMSKVRhYwQ3ElpDoKaZzY1462RgT/6fEt0/kNzq1q3Lddddx8iRIznzzDNjHY5IVAobI1gC7CK0stiYiNe/AZYHGZRIvPj+++8ZOHAgZsbgwYNVJE7iUmFjBOuB9YSqjYpIHu+88w7du3dn7dq19OrVS0XiJG4VOEZgZv8K/3e3mX0d8dhtZl+XXYgi5cu+ffvo06cPLVu2JCcnhzfffJNx48YpCUjcKqxrKLd9W7EsAhGJF1988QVTp06lX79+DBo0iJNOOinWIYmUSIEtgoi7iasCKe6eAzQHfg/oN1+Sys6dOxk7diwADRo0YP369QwfPlxJQBJCNNNHXyW0TGVt4AUgDZgZaFQi5YS789JLL5Gens5dd93FZ5+FVmnVspGSSKJJBEfc/RBwPfCMu98OaJK8JLwvvviCjh070rlzZ6pXr86HH36o8hCSkKJaqtLMOgFdgY7h144NLiSR2MvJyeGyyy5j69atDBs2jDvvvFNF4iRhRfObfRvQh1AZ6nVmVhOYFWxYIrGxceNGqlSpQkpKCmPHjqVWrVrUqVMn1mGJBCqapSpXAXcAS82sAbDZ3Z8IPLI4M/P9Tby/XrNq41VOTg5PP/00aWlpjBs3DoA2bdooCUhSKLJFYGaXAtOBrYABvzCzru6+KOjg4knuYvWqMRR/Vq1aRWZmJkuWLKFDhw507Nix6A+JJJBouoZGAFe7exaAmaURSgwZQQYWj5rVPJ2bm6nAWDwZP348d9xxB6eeeiozZ86kc+fOujFMkk40s4aOy00CAO7+CXBccCGJBM89tNZSWloanTp1Iisri5tuuklJQJJSNC2CZWY2gVArAKALKjoH/Gf9AUBrEMSJ7777jgEDBpCSksKQIUNo2bIlLVu2jHVYIjEVTYugF/A5cB9wP7CO0N3FSS93/QFAaxDEgbfffpsmTZowfPhw9u/f/2OrQCTZFdoiMLPGQG1gnrsPLZuQ4kPuLKFmNU/X+gPl3N69e7nvvvuYOHEitWvX5q233lKpaJEIhVUffZBQeYkuwBtmlt9KZUlLs4Tix7Zt23jxxRe59957WblypZKASB6FdQ11AZq4eyfgAqB3cb/czNqZ2RozyzazBwrZ77dm5mYWVzORNEuo/NqxYwejRo0CQkXiNmzYwFNPPUWFChViHJlI+VNYIjjg7t8CuPuOIvb9CTNLIbSyWXsgHbjJzNLz2e9kQjesvV+c74+Vme9v4sYJ7/04NiDli7szc+ZM0tLSuOeee34sElepUqUYRyZSfhX2x72Wmc0NP+YBtSO25xbyuVwXAtnuvs7dDwKzgWvz2e9PwFDgh2JHHwNaoL782rx5M9dccw1dunShTp06LF++XEXiRKJQ2GDxb/Jsjy7md58DbI7Y3gI0i9zBzH4JVHX3v5jZvQV9kZn1BHoCVKsW+64YLVBf/hw+fJjLL7+cL7/8khEjRnD77beTkpIS67BE4kJhaxa/WcLvzu/OnB/n65nZMYTuWu5W1Be5+0RgIkBGRobm/MmPNmzYQNWqVUlNTWXChAnUqlWLWrVqxToskbhSrH7/YtpCaHWzXFWALyK2TwYaAW+b2QbgImB+eR4wVmG58uPw4cMMGzaMtLS0H1cOu/LKK5UERI5CkAXWPwDqhstWbwU6Azfnvunue4lYD9nM3gbudfelAcZUIpoyWj6sXLmSzMxMli5dyrXXXstvfpO3F1NEiiPqFoGZHV+cL3b3w0BfYCHwCfCyu682s0Fm9uvihVl+aMpobI0dO5bzzz+fjRs38tJLLzFv3jwqV64c67BE4lo0ZagvBJ4DTgWqmdm5QPfwkpWFcvcFwII8rw0oYN/LowlYkpO7Y2Y0atSIzp07M2LECCpWrFj0B0WkSNF0DY0EOhC6yxh3/8jMdGumlIlvv/2Whx9+mNTUVJ566ikuu+wyLrvssliHJZJQoukaOsbdN+Z5LSeIYEQivfnmmzRu3JhnnnmGAwcOqEicSECiSQSbw91DbmYpZnYX8FnAcUkS27NnD927d+fKK68kNTWVd955h5EjR2qtAJGARJMIegP9gGrAV4SmeRa77pBItL766itmz57N/fffz0cffcSll14a65BEElqRYwTuvp3Q1E+RwOT+8b/zzjupX78+GzZs0GCwSBmJZtbQJCLuCM7l7j0DiUiSirszY8YM7rzzTvbv38/VV19N3bp1lQREylA0XUP/AN4MPxYBZwIHggxKksOmTZv41a9+RdeuXalfvz4rVqygbt26sQ5LJOlE0zX0UuS2mU0H3ggsIkkKuUXitm/fzsiRI+nTp4+KxInEyNGUmKgJVC/tQMqz3EXqtUB9ya1bt47q1auTmprKpEmTqF27NjVq1Ih1WCJJrciuITPbbWZfhx97CLUGHgw+tPJDaxCU3OHDhxkyZAjp6emMGTMGgNatWysJiJQDRS1eb8C5hIrGARzxJL2rR2sQHL0VK1aQmZnJsmXLuO666+jUqVOsQxKRCIW2CMJ/9Oe5e074kZRJQI7e6NGjueCCC9i6dStz5sxh7ty5nH322bEOS0QiRDNraImZNQ08EkkoudcMTZo0oUuXLmRlZalctEg5VWDXkJmlhktJXwL0MLPPgW8JrTzm7q7kID+xf/9+HnroIY499liGDRumInEicaCwMYIlQFOgYxnFInHu73//Oz179mTTpk3cfvvtP5aOFpHyrbBEYADu/nkZxSJxavfu3fTr14+pU6dSv3593nnnHS655JJYhyUiUSosEVQys34FvenuTwcQT7mTu05xs5qnxzqUcmv79u3MmTOH/v37M2DAAE444YRYhyQixVBYIkgBfka4ZZCstE5x/r788ktmzZrF3Xff/WORuDPOOCPWYYnIUSgsEWxz90FlFkk5knsnMUDWtn1apziCu/PCCy9w9913891339GhQwfq1q2rJCASxwqbPpq0LYHcO4kB3U0cYcOGDbRr145u3bqRnp6uInEiCaKwFkHrMouiHNKdxP/t8OHDtGrVip07dzJmzBh69erFMcdEcxuKiJR3BSYCd/+6LAOR8ik7O5uaNWuSmprKlClTqFWrFtWrJ1XNQZGEp0u6CDPf38SNE977sVsomR06dIgnn3yShg0b/lgkrlWrVkoCIgnoaMpQJyxVGQ1ZtmwZmZmZrFixgk6dOnHjjTfGOiQRCZASQR7JPjYwcuRI+vXrR6VKlZg7dy7XXXddrEMSkYCpa0iA/xSJ++Uvf8ktt9xCVlaWkoBIklCLIMl988039O/fn+OPP57hw4dz6aWXcumll8Y6LBEpQ2oRJLG//e1vNGrUiLFjx+LuaLkJkeSkRJCEdu3axa233kr79u056aSTWLRoEU8//bQqhYokKSWCJLRr1y7mzZvHI488wvLly2nePHkHx0Uk4ERgZu3MbI2ZZZvZA/m838/MssxspZm9aWaapHRNzYcAAA7WSURBVB6Qbdu2MWzYMNydevXqsXHjRgYNGsTxxx8f69BEJMYCSwRmlgKMAdoD6cBNZpaeZ7flQIa7NwHmAEODiidZuTtTpkwhLS2NRx55hOzsbABOO+20GEcmIuVFkC2CC4Fsd1/n7geB2cC1kTu4+z/d/bvw5mKgSoDxJJ3169fTpk0bMjMzOffcc/noo49UJE5EfiLI6aPnAJsjtrcAzQrZPxP43/zeMLOeQE+AatVUDjoahw8f5oorrmDXrl2MGzeOnj17qkiciOQryESQ3xSUfOcnmtn/ABlAy/zed/eJwESAjIwMzXEsxNq1a6lVqxapqak8//zz1K5dm6pVq8Y6LBEpx4K8RNwCRP4FqgJ8kXcnM7sSeAj4tbsfCDCeQuUuSRmvDh06xOOPP06jRo0YPXo0AJdffrmSgIgUKcgWwQdAXTOrCWwFOgM3R+5gZr8EJgDt3H17gLEUKZ6XpFy6dCmZmZmsXLmSzp07c9NNN8U6JBGJI4G1CNz9MNAXWAh8Arzs7qvNbJCZ/Tq821OE1kV+xcxWmNn8oOKJRjwuSfnss8/SrFkzdu7cyWuvvcasWbM488wzYx2WiMSRQGsNufsCYEGe1wZEPL8yyOMnMnfHzMjIyCAzM5OhQ4fy85//PNZhiUgcSvqic7kL1eeuQ1De7du3j/vvv58TTjiBESNG0KJFC1q0aBHrsEQkjiX9fMJ4WoxmwYIFNGzYkIkTJ5KamqoicSJSKpK+RQDlfzGanTt3ctdddzFjxgwaNmzInDlzaNassFsyRESil/Qtgniwe/duXn/9dR599FGWLVumJCAipUotgnJq69atzJgxgz/+8Y/UrVuXjRs3ajBYRAKhFkE54+5MmjSJ9PR0Bg4cyOeffw6gJCAigUnqRFDe7ib+/PPPad26NT179qRp06asXLmSOnXqxDosEUlwSd01VJ7uJj58+DCtW7fm66+/ZsKECXTv3l1F4kSkTCR1IoDY3028Zs0aateuTWpqKtOmTaN27dpUqaJq3CJSdnTJGSMHDx7kscceo3HjxowZMwaAli1bKgmISJlL+hZBLCxZsoTMzExWrVrFzTffTJcuXWIdkogkMbUIytgzzzxD8+bNf7w3YMaMGVSsWDHWYYlIElMiKCO55SAuvPBCevTowerVq+nQoUOMoxIRUddQ4Pbu3ct9993HiSeeyDPPPMPFF1/MxRdfHOuwRER+pBZBgF5//XXS09OZPHkyxx9/vIrEiUi5lHQtgtyy00Bgpad37NjBnXfeyaxZs2jcuDGvvvoqF1xwQakfR0SkNCRdiyC37DQQWOnpvXv3smDBAh577DGWLl2qJCAi5VrStQggmLLTmzdv5sUXX+SBBx6gTp06bNy4kVNPPbVUjyEiEoSkaxGUtiNHjjB+/HgaNmzI448//mOROCUBEYkXSgQlsHbtWq644gp69+7NhRdeyMcff6wicSISd5Kya6g0HD58mKuuuoo9e/bw3HPP8bvf/Q4zi3VYIiLFpkRQTJ988gl169YlNTWV6dOnU7t2bSpXrhzrsEREjpq6hqJ04MABHn30UZo0acLo0aMBuPTSS5UERCTuqUUQhcWLF5OZmUlWVhZdu3ala9eusQ5JRKTUqEVQhOHDh3PxxRfzzTffsGDBAl544QXOOOOMWIclIlJqlAgKcOTIEQCaN29Or169WLVqFe3bt49xVCIipU9dQ3ns2bOHe+65hwoVKjBq1CgViRORhKcWQYRXX32V9PR0pk2bxsknn6wicSKSFJQIgO3bt3PDDTdw3XXXcdZZZ7FkyRKefPJJ3RcgIklBiQDYt28fb7zxBk888QRLliyhadOmsQ5JRKTMJO0YwaZNm5g+fToPPvggderUYdOmTZx88smxDktEpMwFmgjMrB3wLJACTHb3wXnePx54ATgf2AXc6O4bgogldx2CrG37+Lnvp2HDNhw5coQbb7yROnXqKAmISNIKrGvIzFKAMUB7IB24yczS8+yWCex29zrACGBIUPG8tmIrq7bu4eBX61jx2iSaN2/O6tWrVSRORJJekGMEFwLZ7r7O3Q8Cs4Fr8+xzLTAt/HwO0NoCGqF1d77d8inbZz/IqLs6s3DhQmrUqBHEoURE4kqQXUPnAJsjtrcAzQrax90Pm9le4AxgZ+ROZtYT6AlQrVq1owqm4TmnclqzRgx8Iouzzz77qL5DRCQRBZkI8ruyzzsxP5p9cPeJwESAjIyMo5rc/+g1DYGGR/NREZGEFmTX0BagasR2FeCLgvYxs1TgVODrAGMSEZE8gkwEHwB1zaymmR0HdAbm59lnPnBr+Plvgbdct/OKiJSpwLqGwn3+fYGFhKaPTnH31WY2CFjq7vOB54DpZpZNqCXQOah4REQkf4HeR+DuC4AFeV4bEPH8B6BTkDGIiEjhVGJCRCTJKRGIiCQ5JQIRkSSnRCAikuQs3mZrmtkOYONRfrwiee5aTgI65+Sgc04OJTnn6u5eKb834i4RlISZLXX3jFjHUZZ0zslB55wcgjpndQ2JiCQ5JQIRkSSXbIlgYqwDiAGdc3LQOSeHQM45qcYIRETkp5KtRSAiInkoEYiIJLmETARm1s7M1phZtpk9kM/7x5vZS+H33zezGmUfZemK4pz7mVmWma00szfNrHos4ixNRZ1zxH6/NTM3s7ifahjNOZvZDeF/69VmNrOsYyxtUfxuVzOzf5rZ8vDv99WxiLO0mNkUM9tuZqsKeN/MbGT457HSzJqW+KDunlAPQiWvPwdqAccBHwHpefbpA4wPP+8MvBTruMvgnFsBFcLPeyfDOYf3Oxl4B1gMZMQ67jL4d64LLAdOC2+fGeu4y+CcJwK9w8/TgQ2xjruE53wZ0BRYVcD7VwP/S2iFx4uA90t6zERsEVwIZLv7Onc/CMwGrs2zz7XAtPDzOUBrM8tv2cx4UeQ5u/s/3f278OZiQivGxbNo/p0B/gQMBX4oy+ACEs059wDGuPtuAHffXsYxlrZoztmBU8LPT+WnKyHGFXd/h8JXarwWeMFDFgM/N7MSLcSeiIngHGBzxPaW8Gv57uPuh4G9wBllEl0wojnnSJmErijiWZHnbGa/BKq6+1/KMrAARfPvXA+oZ2aLzGyxmbUrs+iCEc05DwT+x8y2EFr/5PayCS1mivv/e5ECXZgmRvK7ss87RzaafeJJ1OdjZv8DZAAtA40oeIWes5kdA4wAupVVQGUgmn/nVELdQ5cTavW9a2aN3H1PwLEFJZpzvgmY6u7Dzaw5oVUPG7n7keDDi4lS//uViC2CLUDViO0q/LSp+OM+ZpZKqDlZWFOsvIvmnDGzK4GHgF+7+4Eyii0oRZ3zyUAj4G0z20CoL3V+nA8YR/u7/Zq7H3L39cAaQokhXkVzzpnAywDu/h5wAqHibIkqqv/fiyMRE8EHQF0zq2lmxxEaDJ6fZ5/5wK3h578F3vLwKEycKvKcw90kEwglgXjvN4Yiztnd97p7RXev4e41CI2L/Nrdl8Ym3FIRze/2q4QmBmBmFQl1Fa0r0yhLVzTnvAloDWBmaYQSwY4yjbJszQduCc8eugjY6+7bSvKFCdc15O6HzawvsJDQjIMp7r7azAYBS919PvAcoeZjNqGWQOfYRVxyUZ7zU8DPgFfC4+Kb3P3XMQu6hKI854QS5TkvBNqYWRaQA/zR3XfFLuqSifKc7wEmmdndhLpIusXzhZ2ZzSLUtVcxPO7xKHAsgLuPJzQOcjWQDXwH/K7Ex4zjn5eIiJSCROwaEhGRYlAiEBFJckoEIiJJTolARCTJKRGIiCQ5JQIpd8wsx8xWRDxqFLJvjYKqNBbzmG+HK1x+FC7PUP8ovqOXmd0Sft7NzCpHvDfZzNJLOc4PzOy8KD5zl5lVKOmxJXEpEUh59L27nxfx2FBGx+3i7ucSKkj4VHE/7O7j3f2F8GY3oHLEe93dPatUovxPnGOJLs67ACUCKZASgcSF8JX/u2a2LPy4OJ99GprZknArYqWZ1Q2//j8Rr08ws5QiDvcOUCf82dbhOvcfh+vEHx9+fbD9Z32HYeHXBprZvWb2W0L1nGaEj3li+Eo+w8x6m9nQiJi7mdmoo4zzPSKKjZnZODNbaqF1CB4Lv3YHoYT0TzP7Z/i1Nmb2Xvjn+IqZ/ayI40iCUyKQ8ujEiG6heeHXtgNXuXtT4EZgZD6f6wU86+7nEfpDvCVccuBGoEX49RygSxHHvwb42MxOAKYCN7p7Y0J34vc2s9OB64CG7t4EeDzyw+4+B1hK6Mr9PHf/PuLtOcD1Eds3Ai8dZZztCJWUyPWQu2cATYCWZtbE3UcSqkPTyt1bhctOPAxcGf5ZLgX6FXEcSXAJV2JCEsL34T+GkY4FRof7xHMI1dDJ6z3gITOrAsx197Vm1ho4H/ggXFrjREJJJT8zzOx7YAOhUsb1gfXu/ln4/WnAH4DRhNY3mGxmfwWiLnPt7jvMbF24Rsza8DEWhb+3OHGeRKjkQuTqVDeYWU9C/1+fTWiRlpV5PntR+PVF4eMcR+jnJklMiUDixd3AV8C5hFqyP1loxt1nmtn7wK+AhWbWnVDJ3mnu3j+KY3SJLEpnZvmuURGuf3MhoUJnnYG+wBXFOJeXgBuAT4F57u4W+qscdZyEVuoaDIwBrjezmsC9wAXuvtvMphIqvpaXAW+4+03FiFcSnLqGJF6cCmwL15jvSuhq+L+YWS1gXbg7ZD6hLpI3gd+a2ZnhfU636Ndr/hSoYWZ1wttdgX+F+9RPdfcFhAZi85u58w2hUtj5mQt0JFRH/6Xwa8WK090PEeriuSjcrXQK8C2w18zOAtoXEMtioEXuOZlZBTPLr3UlSUSJQOLFWOBWM1tMqFvo23z2uRFYZWYrgAaElvPLIvQH8+9mthJ4g1C3SZHc/QdClR1fMbOPgSPAeEJ/VP8S/r5/EWqt5DUVGJ87WJzne3cDWUB1d18Sfq3YcYbHHoYD97r7R4TWKl4NTCHU3ZRrIvC/ZvZPd99BaEbTrPBxFhP6WUkSU/VREZEkpxaBiEiSUyIQEUlySgQiIklOiUBEJMkpEYiIJDklAhGRJKdEICKS5P4/eQGFOqPDQKQAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from sklearn.metrics import roc_curve\n", "\n", "# Compute predicted probabilities: y_pred_prob\n", "y_pred_prob = logreg.predict_proba(X_test)[:, 1]\n", "\n", "# Generate ROC curve values: fpr, tpr, thresholds\n", "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n", "\n", "# Plot ROC curve\n", "plt.plot([0, 1], [0, 1], 'k--')\n", "plt.plot(fpr, tpr)\n", "plt.xlabel('False Positive Rate')\n", "plt.ylabel('True Positive Rate')\n", "plt.title('ROC Curve')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Precision-recall Curve\n", "When looking at your ROC curve, you may have noticed that the y-axis (True positive rate) is also known as recall. Indeed, in addition to the ROC curve, there are other ways to visually evaluate model performance. One such way is the precision-recall curve, which is generated by plotting the precision and recall for different thresholds. As a reminder, precision and recall are defined as:\n", "$$ \\text{Precision} = \\dfrac{TP}{TP + FP} \\\\\n", " \\text{Recall} = \\dfrac{TP}{TP + FN}$$\n", " Study the precision-recall curve. Note that here, the class is positive (1) if the individual has diabetes." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Text(0.5, 1.0, 'Precision / Recall plot')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU1f3/8dcnCWFLWJOw7/siKiDijuKCtC5trcWlLl/Ubta2Wr9av/1aq11t7W5rrVrXurb6RaWiUBdQBMK+BiJbFpaEkJCQkPXz+2MGfiGEZICZTJJ5Px+PPJx758y9n5Pg/cw5555zzd0REZHYFRftAEREJLqUCEREYpwSgYhIjFMiEBGJcUoEIiIxTolARCTGKRFIi2Rma81sSiNl+ptZiZnFN1FYUWVmD5jZ88HXA83MzSzhOI4zxcyywx+hNFdKBBJWZrbVzMqCF+BdZvZ3M0sK93ncfYy7f9BIme3unuTu1eE+P4CZnWlmn9Sz/+BFuCT4s9XM7o1EDNFmZk+b2U+iHYecGCUCiYTL3D0JGA+cBvywbgELaOn//qYDsxt4v0vw93AV8L9mdlHThCVybFr6/4jSjLl7DvBvYCyAmX1gZj81s4+BUmCwmXU2syfNbIeZ5ZjZT2p35ZjZrWa23syKzWydmY0P7t9qZhcGX08ys3Qz2xdshfwmuP+w7hEz621ms8yswMwyzezWWud5wMxeMbNng+daa2YTG6liY4ng4O8hHVgLnFLrfL3N7J9mlmdmW8zsjlrvxZvZfWb2WTCWpWbWL/je780sK1jXpWZ2TmPnr0/w9/eD4O90b7Dl1u4oZUcF/3aFwd/L5cH9twHXAf8dbPm8eTyxSPQpEUjEBC9e04HltXZ/FbgNSAa2Ac8AVcBQ4FTgYuCW4Oe/DDwA3AB0Ai4H9tRzqt8Dv3f3TsAQ4JWjhPQikA30JvAt/WdmNrXW+5cDLwFdgFnAnxqoWy+gR526Ha3sZALJMDO4HQe8CawE+gBTge+a2SXBj9wJXEPgd9cJ+C8CiRNgCYGE0g34B/Dq0S7gIbgOuITA72w49bfc2gRjfRdIA74NvGBmI9z9ceAF4OFgF9xlxxmHRJkSgUTCG2ZWCCwAPgR+Vuu9p919rbtXEbiYXQp81933u/tu4LfAjGDZWwhcZJZ4QKa7b6vnfJXAUDNLcfcSd/+0boFgUjobuMfdD7j7CuAJAonpoAXuPjs4pvAccHIDdZwOvOMNL9aVb2ZlwELgz8Abwf2nAanu/qC7V7j7ZuBvder9Q3fPCNZ7pbvvAXD35919j7tXufsjQFtgRAMxNORP7p7l7gXATwkkn7omA0nAL4Kx/gd46yhlpYU65jsKREJwpbvPPcp7WbVeDwDaADvM7OC+uFpl+gGfhXC+mcCDwAYz2wL82N3fqlOmN1Dg7sW19m0Danf/7Kz1uhRoZ2YJwaRV13QC38gbkgI48F0CF842QAWBevcOJsuD4oH5wddHrbeZ3UUgUfQOHrtT8DzHo/bfYlvwmHX1BrLcvaZO2T7HeU5phtQikKZW+xt0FlAOpLh7l+BPJ3cfU+v9IY0e0H2Tu19DoOvil8BrZtaxTrFcoJuZJdfa1x/IOdYKBLtLzgPeCyG26uA39wPAN4O7s4Attercxd2T3X16rfePqHdwPOAe4Gqgq7t3AYoAq1s2RP1qve5P4HdUVy7Qr87Afu3fm5YvbgWUCCRq3H0Hgb7nR8ysk5nFmdkQMzsvWOQJ4PtmNiF4l9FQMxtQ9zhmdr2ZpQa/tR78ln3YLaPungV8AvzczNqZ2TgCLYkXjiP0c4BV7r7vGD7zCwKDqu2AxcA+M7vHzNoHB4fHmtlpwbJPAA+Z2bBgvceZWXcC4ypVQB6QYGb3E2gRHK9vmVlfM+sG3Ae8XE+ZRcD+YOxtLDB34zICYykAu4DBJxCDNANKBBJtNwCJwDpgL/Aa0AvA3V8l0Hf9D6CYQB97t3qOMQ1Ya2YlBAaOZ7j7gXrKXQMMJPAt93XgR+7e6Lf6eoR0t1AdbxOo363BMYjLCAz6bgHyCVz8OwfL/obAgPe7wD7gSaA9MIfAXVgbCXTPHODw7p1j9Y/gOTYHf46YD+DuFQQG0S8Nxvln4AZ33xAs8iQwOnhH0Rt1Py8tg+nBNCLHxszWAVe5+7pox3K8zGwrcEsDYzkSQ9QiEDkGZpYIPNuSk4BIXbprSOQYBLtKfhHtOETCSV1DIiIxTl1DIiIxrsV1DaWkpPjAgQOjHYaISIuydOnSfHdPre+9FpcIBg4cSHp6erTDEBFpUcysvuVZAHUNiYjEPCUCEZEYp0QgIhLjlAhERGKcEoGISIyLWCIws6fMbLeZrTnK+2Zmfwg+MnDVwUcQiohI04pki+BpAqtCHs2lwLDgz23AXyIYi4iIHEXE5hG4+0dmNrCBIlcQWLzLgU/NrIuZ9QquUR92S7YWMH9j3qHt/t07ctWEvpE4lYhIixLNCWV9OHwt9ezgviMSgZndRqDVQP/+/Y/rZMu27eWP72cCcHB5pctP7k1igoZJRCS2RfMqWN/j9epdAc/dH3f3ie4+MTW13hnSjfraeUPY8vPPseXnn+PuS0YET6YF90REopkIsjn8mal9qf+ZqSIiEkHRTASzgBuCdw9NBooiNT4gIiJHF7ExAjN7EZgCpJhZNvAjoA2Auz9G4Jmv04FMoBS4OVKxiIjI0UXyrqFrGnnfgW9F6vwiIhIa3TIjIhLjlAhERGKcEoGISIxTIhARiXFKBCIiMU6JQEQkxikRiIjEOCUCEZEYp0QgIhLjlAhERGKcEoGISIxTIhARiXFKBCIiMU6JQEQkxikRiIjEOCUCEZEYp0QgIhLjlAhERGKcEoGISIxTIhARiXFKBCIiMS6iicDMpplZhpllmtm99bw/wMzmmdkqM/vAzPpGMh4RETlSxBKBmcUDjwKXAqOBa8xsdJ1ivwaedfdxwIPAzyMVj4iI1C+SLYJJQKa7b3b3CuAl4Io6ZUYD84Kv36/n/RYpY2cx/9mwK9phiIiEJJKJoA+QVWs7O7ivtpXAl4KvvwAkm1n3ugcys9vMLN3M0vPy8iISbLhk7i7m6r8u5H9eXxPtUEREQhLJRGD17PM6298HzjOz5cB5QA5QdcSH3B9394nuPjE1NTX8kYbJrn0HuPGpJRSVVVJdU7eqIiLNU0IEj50N9Ku13RfIrV3A3XOBLwKYWRLwJXcvimBMEVNUVsmNTy2msLSCU/t3IWdvWbRDEhEJSSRbBEuAYWY2yMwSgRnArNoFzCzFzA7G8APgqQjGEzHuzp0vryBzdwmPfXUCI3smRzskEZGQRSwRuHsVcDswB1gPvOLua83sQTO7PFhsCpBhZhuBHsBPIxVPJD336TbmbdjNfdNHcc6w5tt1JSJSn0h2DeHus4HZdfbdX+v1a8BrkYwh0jJ2FvPTt9czZUQqN581MNrhiIgcM80sPgEHKqu548XlJLdL4FdXnYxZfePjIiLNW0RbBK3d7+ZuImNXMX+/6TRSk9tGOxwRkeOiFsFxyioo5akFW/jS+L6cPzIt2uGIiBw3JYLj9PCcDOLi4O5LRkQ7FBGRE6JEcByWb9/Lmytzue2cwfTs3C7a4YiInBAlgmPk7vzk7fWkJLXla+cNiXY4IiInTIngGL2zZidLt+3lrouH07GtxtpFpOVTIjhGj320mcEpHbl6Yr/GC4uItABKBMdgVXYhK7MKueGMAcTHac6AiLQOSgTH4NmF2+iQGM8XJ+hBaiLSeigRhGjv/greXJnLlaf2oVO7NtEOR0QkbJQIQvTq0izKq2q44YwB0Q5FRCSslAhCUFPjPP/pdiYN7MbInp2iHY6ISFgpEYTgw415bC8o5atqDYhIK6REEII3VuTQrWMil4zpGe1QRETCTomgEVXVNXyQkcf5I9JITNCvS0RaH13ZGrF0216Kyiq5cFTTrDC68LM9zFu/q0nOJSICeh5Bo/6zYTdt4o2zh6VE/FwLP9vDjU8tZkhaElNH9Yj4+UREQC2CRs1dv4vJg7uTHOG5A+ty93Hbs+lUVNfg7hE9l4hIbUoEDdiav5/P8vZzQYQfPJNVUMqNf19MUrsETu7XJaLnOlb5JeWUVlRFOwwRiSAlggb8Z8NuAKaOjFw3zf7yKmY+s4Tyymqe+a9J9OzU+CMvt+8pZWv+/ojFBIHnMf9qzgYm/2wev3l3Y0TPJSLRFdFEYGbTzCzDzDLN7N563u9vZu+b2XIzW2Vm0yMZz7Gat2EXw9KS6N+9Q0SO7+7c/dpKMneX8OfrJjC8R3Kjn1mytYDpf5jPfa+vjkhMAAs25TPtdx/x6PufUePO3tLKiJ1LRKIvYonAzOKBR4FLgdHANWY2uk6xHwKvuPupwAzgz5GK51gVH6hk0eYCLojg3UJ//Wgzs1fv5N5LR4Y0GL1gUz43PLmYkvIqyqtqwh7PnpJyvvfyCq5/chFmxj9uOZ1enduH/Twi0rxE8q6hSUCmu28GMLOXgCuAdbXKOHBwzYbOQG4E4zkmH23Mp6rGuTBCd+/M35THw+9s4PPjenHrOYMbLf+fDbv4+vPLGJzSkTgL7xLY7s6/luXw0Nvr2F9exbcvGMq3zh9Kuzbx9ZavrnHmrN3J2N6dI9ZaEpGmE8muoT5AVq3t7OC+2h4ArjezbGA28O36DmRmt5lZupml5+XlRSLWI3zyWT7JbRM4NQKDt4Fv3isZmpbEw1eNwxq5sC/YlM/Xn1vGiB7JvHjrZLp1TAxrLF97bil3vbqSoalJzL7jHO66eMRRk8DKrEKueHQB33xhGU8s2By2OEQkeiLZIqjv6lb3vshrgKfd/REzOwN4zszGuvth/R7u/jjwOMDEiROb5N7KNTlFjOnTiYT48OZKd+cH/1rNvrJKnr9lEh0SG/4TLN22l1ufTWdQSkeemzmJLh3ClwTmrd/FPf9cxb6yKv5n+ihmnj2IuKM8cKeotJJfvbuBFxZtJzWpLW0T4qiq0W2uIq1BJFsE2UDt5zn25ciun5nAKwDuvhBoB0R+5lYjKqtrWL+zmJP6dA77sV9dms2763Zx9yUjGl3JdG1uETf/fTE9OrXluVsaTgJ791cw4/GF/Oj/1jQaQ0VVDQ+9tY6Zz6STktSWWd8+i1vPHXzUJLAyu5ALHvmAfyzazs1nDmLeXedFfF6FiDSdSLYIlgDDzGwQkENgMPjaOmW2A1OBp81sFIFE0DR9Pw3YuKuYiqoaxoY5EWQVlPLjWWuZPLgbM88e1GDZ3MIybv77EpLaJvD8LaeTltyuwbI3PLWYzN0ljY4fZBWUcvs/lrEyu4ibzhzID6aPpG1C/d1AB2XuLmF8/y48O3MSY3qHPzmKSHRFLBG4e5WZ3Q7MAeKBp9x9rZk9CKS7+yzgLuBvZvY9At1GN3kzmFa7NmcfQFhbBO7Ovf9aRZwZv/7yyUf99g1QWlHNzGfSKauo5p/fPJO+XY8+IJu5u5ivPrmYkgNVpCU3PAfhg4zd3PHichx47PrxTBvbq9G4775kBNU1zhdO7dNgzCLSckV0rSF3n01gELj2vvtrvV4HnBXJGI7H6pwiktomMLB7x7Ad861VO/g4cw8PXTGmwQs7wPaCUuLjjKduOq3BuQUbdu7j2r8tIs6Ml742mQdmra23nLvz5w8+49fvZjCyZyf+ev2EkO/2ufLUuuP7zUPB/greW7eTaWN60bmDuqlEToQWnavH6pwixvTuFLZvwCXlVfzk7XWM7dOJa08P7eE2D1w2mvOGpx71/R2FZVz7t0Ukxsfx4m2TGZRSf9Iqq6jmrldXMHv1Ti4/uTe//NI42ic23BXUnGUVlPLE/M28nJ7Fgcoaqmqc60L8nYpI/ZQI6qiqrmH9jn18dXL4Li6/n7uRXfvKeez6CcQ3klyuntiP0wd156tnDGywXG7RAXp1bseLt05m4FGSQF5xObc8m86q7ELumz6SW88Z3Oitqs3Vmpwi/vrRZt5elUt8nHHR6B7MXr2T6jp3Lm3OK2HO2l3MOK0fXcN4m61Ia6ZEUMem3SWUV9VwUt/wjA9k7CzmqY+3MuO0fpzav2uj5UNZfrpT+wR6dW7HS7dNZsBRuq827Srm5qeXkF8SSEAt9elqS7YW8Id5m5i/KZ+ktgnces5gbj5rEG3ijdmrdwKBrq9FWwp4Yv4W5m3YhTt0T0rk6on9Gjm6iIASwRFW5xQBhO2OoZ+8vY6ktgn897SRYTkewCNfPoUadzq2rf/Pt2z7Xm7++xISE+J45WtnMK5v81nRdHNeCU8u2MJXTuvXYFzpWwv43dxNLMjMJyUpkXumjeS6yf3pFLxtdU9JOQAfZ+bzano2q3OK6NqhDddO6s8Li7YftpR3UVklr6Zn8d66XfzsiycxJDUpspUUaWGUCOpYExwoHhSGgeJFm/cwf1M+900fGdbZwA318X+WV8J1f1tEWqe2PD/zdPp1i+wSEO7O/E35jOiZTI9OR7/Ftaiskj/M28Qzn2ylqsbp2iGx3kRQNwH88HOjuO70AUfU+WAX15y1uxic2pGffeEkvji+DwX7K3hh0XYg8Lt45pOtvLY0m9KKaiDw91UiEDmcEkEdq3OKGB2GgWIHHnl3I6nJbfnq5IFhiS0Uu/aVM7JnMs/OnNTg3INw2BGcvzB/Uz63nD2IH36+7pqCgTGXF5dk8Zt3Mygsq+TqCf14ZWnWEeXW5BTxy3c2MH9TwwngoG4dE/nfz49mUEoHpgxPO+Lv9Yd5meQUlpEYH8dlJ/fmvBGp3PHi8kPvu3uLHS8RCTclgloODhSH4y6U/JJy8orLefCKMU12l86Q1CTaxMfxl+sn0Ll95G+pfD8jj6S2CSTEGRXVR66GumBTPg++tZaNu0o4fVA37r9sNGN6d+a1ZdmHymQVlPLIuxm8sSKXrh3a8D/TR3H95KMngNrqm5TXMTGBNvGBeO68aDjXTOpPanJbNueVALB8eyFvr9rB+xm7eflrZzA+hHGbSKqucRZt3sPWPaVcM6mfkpNEhRJBLZ/l7edAZU1YJpK5Q58u7fnKaU03YPnzL57UZBeSiQO64jg/umwMn/vD/MPeyysu56G31jFrZS79u3UIDlb3OCy2wrIKfvLWOp5duA0z+OaUIXx9ypBDYwDHq3OHNnx49/mkJLUlMeHIFVSe/mQr7dvEU1nt5BaWhT0RVNc4H2fmszqniK+dO7jetarcnWXbC3lzZS5vr95BXnFgvOPsoSlazVWiQomglv8/UNzwGkChumPq0EaXbwinpvw2+dhXJxyxr6bGeSU9i5//ewNlFdV898JhfGPKkHp/B89/uh0zuGp8X+68eHhYn3vQu8uRx+rfrQM3nzWQ0b06MapXJz7/xwVhOx8ExiP+uTSbfy3LYee+AwCcMyzl0DiIu7M2dx9vrsrlrZU7At1WCXFMHZlGp3ZteDk9i+roT6qXGKVEUMvGXcUkJsQxKOXEBhOHpiVzav8ufGl83zBF1vxtLyhlxuOfsnhrAacP6sZPv3ASQ9Pq/z2O69uZzu3bcO+lIxtdeC9cEuLj+NFlY4DArbXhsO9AJW+t3MFrS7NYtr2QOIMpI9KYNrYnT3+yFXfYkr+fN5bn8OaqXDbn7SchzjhnWAp3XTyci0b3ILldG95YnsPL6UeOm4g0FSWCWnIKy+jbpX2jk74aM/PsQY0uKtfafJCRR+f2bXj4S+P48sS+DbZOXv9ms1tV5JDGBpFrapyPP8vntaXZvLNmJ+VVNQxLS+IHl47kC6f2Ia1TO/6zYRdPf7KV77y0nK17SjGDMwZ359ZzBjNtTE9NdJNmR4mglpy9ZfV2K0jDzhueSlyccd/0UaQkNbzwXXPk7izcvIe/f7yVDzPyeP1bZx6xymrB/gpeTc/ihUXb2V5QSqd2CVw9sR9XTejLuL6dD0seB38H7drEc9/0kVxxSp8Gb60ViTYlglpyC8s4f0TknlHcWv1uxqnRDuG4HKis4aXF23n6k61s2FlM24Q4KqpryC08wJjenYODunt5/tPtvL16BxVVNUwa2I27Lh7OJWN6HvUpbuP6dmHVAxef8MC3SFMJORGYWR9gQO3PuPtHkQgqGsqrqtldXK4WQQy555+rqK5xRvZM5uEvjWNwakeuemwhpRVVvLBoG89/up31O/aR1DaBGaf147rTBzCi59FXg61NSUBakpASgZn9EvgKgQfPVwd3O9BqEsHOosCdHn26KhG0dqnJbenduR0n9e3MzWcN4vRB3TAz1gTvGvvuyytwh1G9OvHTL4zlylP6HHU5D5HWINR/3VcCI9y9PJLBRFPO3jIAendRX25r16VDIp/8YOoR+3t3ac+glI6c2q8L100ewPj+XTTBS2JCqIlgM9AGaL2JoDCQCPp20YSeWNWtYyLvf39KtMMQaXKhJoJSYIWZzaNWMnD3OyISVRTkFJZhBj07q0UgIrEl1EQwK/jTauXsLSMtuf5lCUREWrOQEoG7P2NmicDw4K4Md6+MXFhNL7dIcwhEJDaF9PXXzKYAm4BHgT8DG83s3AjG1eRy9pbRR4lARGJQqP0gjwAXu/t57n4ucAnw28Y+ZGbTzCzDzDLN7N563v+tma0I/mw0s8JjCz88amqc3KIDSgTSIpSUV7F9T2m0w5BWJNQxgjbunnFww903mlmDM2bMLJ5AC+IiIBtYYmaz3H1dreN8r1b5bwNRmaKav7+ciqoazSGQZququob5mfm8sTyHOWt3UlMDy+6/iCTNb5AwCPVfUbqZPQk8F9y+DljayGcmAZnuvhnAzF4CriAwKa0+1wA/CjGesMotDE4mU4tAmhF3Z3VOEf9alsNbq3LJL6mgc/s2DE5JYt2OfZRXVisRSFiE+q/oG8C3gDsAIzCj+M+NfKYPUHtt3Wzg9PoKmtkAYBDwn6O8fxtwG0D//v1DDDl0/38ymRKBRF9WQSlvLM/h9RU5bM7bT2J8HFNHpXHlqX2YMiKVl5dkcf//rY12mNKKhHrXUDnwm+BPqOqbknm0J2/MAF5z9+r63nT3x4HHASZOnBj2p3fkBieTqWtIoumtlbksyMxn0ZYCACYN7Mat5wxm+thedO6gtYskchpMBGb2irtfbWarqeci7u7jGvh4NlD7OY19gdyjlJ1BoMURFTmFZSS3TdBCYRIVB1exeOS9jQzs3oG7LhrOlaf2oV83zXKXptFYi+A7wf9+/jiOvQQYZmaDgBwCF/tr6xYysxFAV2DhcZwjLHIKy9QakKg5d1gqd1wwlLOHpXLawK5a30iaXIOJwN13BF/mA2XuXmNmw4GRwL8b+WyVmd0OzAHigafcfa2ZPQiku/vBmcrXAC+5R++BrXogjURT146J3HnxiLAc60BlNXPX7+KN5TnsL6/mhVtOJ+4En7gnrV+og8UfAeeYWVdgHpBOYFnq6xr6kLvPBmbX2Xd/ne0HQg02UnKLypgwoGu0wxA5LtU1zief5fPG8lzmrN1JSXkV8XFGdY1TUV1Du7j6H6AjclCoicDcvdTMZgJ/dPeHzWx5JANrKvvLqygsrVSLQFqcNbn7+GhjHrNW5pJXXE5y2wSmn9STK0/pw7Lte/n1uxujHaK0ECEnAjM7g0ALYOYxfrZZ0x1D0lLd+NRiEuPjmDIilS+c2ofzR6YdenzmiuyoTNKXFirUi/l3gR8Arwf7+QcD70curKaTfTAR6IE00kJMHtyd6Sf15Jxhqbq1VMIi1HkEHwIf1treTGByWYt3cDJZHz2QRlqI4T2S+fN1E0IqW+POgk357K+o4pIxPSMcmbRUjc0j+J27f9fM3qT+eQSXRyyyJrK7uByzwHNsRVqbcx9+n/ySCuIMNv7kUhLi9bwNOVJjLYKDawv9OtKBRMu+skqS2iYQr1vspBXp17UD7drEMWFAV6qqnXkbdkc7JGnGGptHcHBhuXSC8wjg0MqireIrdPGBKs0ollbnspN78/lxvTAz/jhvkxKBNCjUduI8oHYnentgbvjDaXrFBypJbtcqboASOYxmKEuoQk0E7dy95OBG8HWrGF3dd6BSLQIRiWmhJoL9Zjb+4IaZTQDKIhNS0yo+UKUWgYjEtGOZR/CqmR1cPbQXgSUmWrziA1UMS1MiEJHYFeo8giVmNhIYQeA5AxvcvTKikTWRfQcq6dReXUMiErtC6hoysw7APcB33H01MNDMjmdp6mbFXV1DIiKhXgH/TuAZxWcEt7OBV4G3IhFUUzlQWU11jZOswWKJIZXVNczflMfry3Nxd/507fjGPyStWqiJYIi7f8XMrgFw9zJrBfemFR+oAlCLQGLCiqxC3lyZy5urdlCwvwJAEykFCD0RVJhZe4LLTJjZEKA8YlE1kaKywDCHbh+VWHDVYwtpmxDHRaN78IVT+7B4awFPzN8S7bCkGQg1EfwIeAfoZ2YvAGcBN0UqqKaiFoHEgrOGpbA2dx8XjEpj2tieh774LN+upaoloNErYLALaAPwRWAygbuGvuPu+RGOLeKKDwRaBBojkNZsfP+uPPbV0FYrldjUaCJwdzezN9x9AvB2E8TUZPYFWwSd1CIQkRgW6hXwUzM7zd2XRDSaJnawRaB5BBKr3J131uzg9eU5JMTH8ajuIIpJoSaC84Gvm9lWYD+B7iF393GRCqwpaIxAYl2Nw9efXwZAYoKeVRCrQr0CXno8BzezacDvgXjgCXf/RT1lrgYeIHBH0kp3v/Z4znU8ig9UEh9ntA8+51Uklkwb25PiA5VMHdWD+ZvyeGbhtmiHJFHS2BPK2gFfB4YCq4En3b0qlAMHn1nwKHARgQloS8xslruvq1VmGIFnIZ/l7nvNLO34qnF89pUFZhW3gikRIsdsbJ/OjO3TGYBPPtsT5WgkmhprETwDVALzCbQKRgPfCfHYk4DM4PONMbOXgCuAdbXK3Ao86u57Ady9SZ+eUVyuJahFjqayuoYFm/KJjzPOHZ4KQFZBKW+t2sGbK3Pp1bkdj143ng835pHcLoEzh6REOWI5Xo0lgtHufhKAmT0JLD6GY/cBsmptZwOn1ykzPHjsjwl0Hz3g7u/UPZCZ3QbcBtC/f/9jCKFhWmdI5HDuzrLthfzfihzeCs5ATm6XwJ0XDWfWytxDcw86JMazcVcxE29reMMAABErSURBVH8yl5LyKvp2bc+Cey6IcvRyvBq7Ch5aYdTdq46xC6W+wl7P+YcBU4C+wHwzG+vuh810cffHgccBJk6cWPcYx21fmZ5OJnJQZXUN5/3qA7YXlNI2IY4LR/egsLSCjzP38OM31zG6VyfumTaSz4/rxbvrdvGXDzKZOrIHm/NLyN5bRlFZJfPW76Jbx0SmjKi/l3fbnv28s2YnA7p3ZNrYnk1cQzmaxq6CJ5vZvuBrA9oHtw/eNdSpgc9mA/1qbfcFcusp82lwSestZpZBIDE0yW2qxQeq6NGpXVOcSqRZS01uiwH9u3Xg2xcMZdrYniS3a0Pm7mLmrt/NhaPSGJqWfKj8zLMHMfPsQQD892srWba9kNN+MpeK6hoGdO/Ah3f//0SQubuEd9bsYPbqnazbEbicjOyZrETQjDT28PoTuZ1mCTDMzAYBOcAMoO4dQW8A1wBPm1kKga6izSdwzmOy70CVZhWLAP911kBmnNaPjm0PvyQMTUs+LAHUZ0zvzizeUsCFo3qwOqeI7L1lZOwsZvbqHfx7zQ427go85XbCgK788HOjeG/dLgpLD3+cSXlVNQs/20PPzu0Y2bOh75cSCRHrFwl2Jd0OzCHQ//+Uu681sweBdHefFXzvYjNbB1QDd7t7k92+sE8PrhcBAg+6r5sEQnXjmQO58cyBANz5ygoWbSngkt99hBlMGtiNBy4bzbSxvejZOdD6Tt+6l8LSSkrKq3h/w27mrN3JBxl5lJRXMXFAV177xpnhqpaEKKJXQXefDcyus+/+Wq8duDP40+Qqqmq0vIRIGJ0/Io3C0kouGJnGxWN6kJZcf9frlvz9jH/wPSqqa+jeMZHLTu7Fsm2FlJRX8e7ancxdv4tT+3fli+P7sGTLXnp3acfg1KQmrk3siPmroJaXEAmfy07uzWUn926wzNg+ndiwcx9TR/XgkjE9mTCgK/Fxxs1/X8z7GXnc9txSAN5atYOfvr2ekvIqzhmWwnMzAzcdllZUsWBTPu9n5HHGkO5c3sj5pHExnwjUNSTStG6/YBi3XzDsiP3Xnj6AQSlJXDAyjXfX7eTfa3YydWQai7YUsKekgn8s2s7c9bv4ODOf8qoaALL3lioRhEHMXwU1WCzSPFw0ugcXje4BwNnDUnjwirEAXPu3T/nksz3c9/pq+nVrz7Wn9+eiUT34xTsbohluq6JEoBaBSLP2rfOHcs6wVKaOSmNYWtKhJWES4oycvWXc+coK3l61g8+N68U5w1L4MCOPC0b1UEvhGMT8VVBLTIg0b2cNTeGsoUcuX9GuTTyb8wvZs7+C8qoa/rUsh38tywGgpLxKieAYxHwiUItApGX6xRfHsbv4AKf068Lc9btZv2MfF4xM495/rY52aC1OzF8FNUYg0jL1796B/t07AIEltQ/OVI7TYsLHLOafRKEWgYjEuphOBIkJcbTTQ2lEJMbFdCLQrGIRkRhPBBofEBGJ8USgFoGISIwnArUIRERiPhGoRSAiokQgIhLjYjoRaHkJkdYpv6SCR9/P5Ct/Xchzn24L6TO5hWX8c2k22/bsj3B0zU9MfyXWGIFI6xMfZ6zIKmRFViFxBg64Ox9tzOfc4Sl065jIgk35XDAyjaS2CXywMY8PMnYfeqTmNZP68/MvnhTdSjSxGE8EMV19kVbpvumj2L6nlCkjUrnhqcUs3lLA4i0FAMxdv+tQuZeWZAHQJt44bWA3rprQl8c+3ExVdU1U4o6mmL4SKhGItD6TB3dn8uDuANx9yQi25O/ngpFprMgqZHPefs4fmcqybYVs3bOfKSPSOHNI90PPa376461RjDx6YvpKqMdUirRuU0f1OPS69jOPJwzoFo1wmq2YHixWi0BEJMKJwMymmVmGmWWa2b31vH+TmeWZ2Yrgzy2RjKcu3TUkIhLBriEziwceBS4CsoElZjbL3dfVKfqyu98eqTgaohaBiDSmsrqGtbn7GJzasdV+eYzklXASkOnumwHM7CXgCqBuIoia1vpHFZETk1VQyocb8/hwYx4LP9tDSXkVXztvMDNO68/iLXs4c0gK/bp1iHaYYRPJRNAHyKq1nQ2cXk+5L5nZucBG4HvunlW3gJndBtwG0L9//xMObEhqEiN6JGuwWESO8MaKHF5dmg1A367tueKU3vxrWQ5Pzt/CXz/cDMAtZw/ih58fHc0wwyqSiaC+B8Z5ne03gRfdvdzMvg48A1xwxIfcHwceB5g4cWLdYxyz2o+1ExE56OIxPdm6Zz/nDU/lvOGpDErpiJkRZ8bOfQc4d3gqP5+9noxdxTz45jrStxXwzSlDmDa2V7RDPyGRTATZQL9a232B3NoF3H1Prc2/Ab+MYDwiIg164PIx9e5/6Mqxh17/fu5G5m/KZ/GWAsqrakjfuleJoAFLgGFmNgjIAWYA19YuYGa93H1HcPNyYH0E4xEROWF/uX4CZRXVTBrUjQkPvRftcMIiYonA3avM7HZgDhAPPOXua83sQSDd3WcBd5jZ5UAVUADcFKl4RETC4bSBrW8yWkTvn3T32cDsOvvur/X6B8APIhmDiIg0LKZnFouIRJr7Cd/fEnGaUSUicgIydhXzwzdWk751L9+9cBgXjurByuwiPsnMZ0FmPiuyCvnpF07iqgl9ox3qUSkRiIgcp8SEOOZvymfptnhKK6p5YNY67n51FcXlVZjB6F6dKK+qYXtBabRDbZASgYjIcXrixtOornFO6deFL/3lE/YdqOT8kb05e2gKZwzpTreOiQy89+1oh9koJQIRkeM0YUDXQ6/f/PbZUYzkxGiwWEQkxikRiIjEOCUCEZEYp0QgIhLjlAhERCKstLyK9zN28/u5m9i+p/ndSqq7hkREIuyJBVt4YsEWAOIMvj11WJQjOpwSgYhIBH1n6jAqq2s4fXB3bnxqMUu37+WuV1ayOqeQH35uNOcOT412iEoEIiKR9L2LhgNQU+MkJsTxQUYendu3oaisktU5RUoEIiKxIi7OeO3rZ9AmPo5BKR0Z+b/vRDukQ5QIRESayLi+XQAor6qOciSH011DIiIxTolARCTGKRGIiETJquxC/veNNVz56Mcs2VoQtTg0RiAi0sTizEiIM+as3UX7NvGUVVbz6zkZJCbEsTZ3H49cfTLnj0hrsniUCEREmlib+Dj+cetkEhPiGN4jifEPvceiLQUMTUuiYH8Fm/P2c/6IpotHiUBEJAomDep26PXcO88jqW0CZsbJP363yWOJ6BiBmU0zswwzyzSzexsod5WZuZlNjGQ8IiLNUd+uHejSITFq549YIjCzeOBR4FJgNHCNmY2up1wycAewKFKxiIjI0UWyRTAJyHT3ze5eAbwEXFFPuYeAh4EDEYxFRESOIpKJoA+QVWs7O7jvEDM7Fejn7m81dCAzu83M0s0sPS8vL/yRiojEsEgmAqtnnx960ywO+C1wV2MHcvfH3X2iu09MTY3+Ak0iIq1JJBNBNtCv1nZfILfWdjIwFvjAzLYCk4FZGjAWEWlakUwES4BhZjbIzBKBGcCsg2+6e5G7p7j7QHcfCHwKXO7u6RGMSURE6ohYInD3KuB2YA6wHnjF3dea2YNmdnmkzisiIscmohPK3H02MLvOvvuPUnZKJGMREZH6aWaxiEgz8/6G3XySmc+GncX86svjOHNISkTPp9VHRUSaibYJcSQmxLEgM59Nu0vIKSxj066SiJ9XLQIRkWaiXZt45n7vPDq2jcfMGP/Qe01yXiUCEZFmpH/3DgAU7K9osnOqa0hEJMYpEYiIxDglAhGRGKdEICIS45QIRERinBKBiEiMUyIQEWnGSsqr+Dgznz/M28Ta3KKInEPzCEREmrFfzck49Lprx0TG9O4c9nMoEYiINENdO7ThG1OGkBgfx/gBXTmlXxc6t28TkXMpEYiINENmxj3TRjbJuTRGICIS45QIRERinBKBiEiMUyIQEYlxSgQiIjFOiUBEJMYpEYiIxDglAhGRGGfuHu0YjomZ5QHbjvPjKUB+GMNpCVTn2KA6x4YTqfMAd0+t740WlwhOhJmlu/vEaMfRlFTn2KA6x4ZI1VldQyIiMU6JQEQkxsVaIng82gFEgeocG1Tn2BCROsfUGIGIiBwp1loEIiJShxKBiEiMa5WJwMymmVmGmWWa2b31vN/WzF4Ovr/IzAY2fZThFUKd7zSzdWa2yszmmdmAaMQZTo3VuVa5q8zMzazF32oYSp3N7Org33qtmf2jqWMMtxD+bfc3s/fNbHnw3/f0aMQZLmb2lJntNrM1R3nfzOwPwd/HKjMbf8IndfdW9QPEA58Bg4FEYCUwuk6ZbwKPBV/PAF6OdtxNUOfzgQ7B19+IhToHyyUDHwGfAhOjHXcT/J2HAcuBrsHttGjH3QR1fhz4RvD1aGBrtOM+wTqfC4wH1hzl/enAvwEDJgOLTvScrbFFMAnIdPfN7l4BvARcUafMFcAzwdevAVPNzJowxnBrtM7u/r67lwY3PwX6NnGM4RbK3xngIeBh4EBTBhchodT5VuBRd98L4O67mzjGcAulzg50Cr7uDOQ2YXxh5+4fAQUNFLkCeNYDPgW6mFmvEzlna0wEfYCsWtvZwX31lnH3KqAI6N4k0UVGKHWubSaBbxQtWaN1NrNTgX7u/lZTBhZBofydhwPDzexjM/vUzKY1WXSREUqdHwCuN7NsYDbw7aYJLWqO9f/3RrXGh9fX982+7j2yoZRpSUKuj5ldD0wEzotoRJHXYJ3NLA74LXBTUwXUBEL5OycQ6B6aQqDVN9/Mxrp7YYRji5RQ6nwN8LS7P2JmZwDPBetcE/nwoiLs16/W2CLIBvrV2u7LkU3FQ2XMLIFAc7KhplhzF0qdMbMLgf8BLnf38iaKLVIaq3MyMBb4wMy2EuhLndXCB4xD/bf9f+5e6e5bgAwCiaGlCqXOM4FXANx9IdCOwOJsrVVI/78fi9aYCJYAw8xskJklEhgMnlWnzCzgxuDrq4D/eHAUpoVqtM7BbpK/EkgCLb3fGBqps7sXuXuKuw9094EExkUud/f06IQbFqH8236DwI0BmFkKga6izU0aZXiFUuftwFQAMxtFIBHkNWmUTWsWcEPw7qHJQJG77ziRA7a6riF3rzKz24E5BO44eMrd15rZg0C6u88CniTQfMwk0BKYEb2IT1yIdf4VkAS8GhwX3+7ul0ct6BMUYp1blRDrPAe42MzWAdXA3e6+J3pRn5gQ63wX8Dcz+x6BLpKbWvIXOzN7kUDXXkpw3ONHQBsAd3+MwDjIdCATKAVuPuFztuDfl4iIhEFr7BoSEZFjoEQgIhLjlAhERGKcEoGISIxTIhARiXFKBCJ1mFm1ma0wszVm9qaZdQnz8W8ysz8FXz9gZt8P5/FFjpUSgciRytz9FHcfS2CeybeiHZBIJCkRiDRsIbUW9DKzu81sSXAd+B/X2n9DcN9KM3suuO+y4PMulpvZXDPrEYX4RRrV6mYWi4SLmcUTWLrgyeD2xQTW7ZlEYOGvWWZ2LrCHwBpOZ7l7vpl1Cx5iATDZ3d3MbgH+m8AsWJFmRYlA5EjtzWwFMBBYCrwX3H9x8Gd5cDuJQGI4GXjN3fMB3P3gAoZ9gZeDa8UnAluaJHqRY6SuIZEjlbn7KcAAAhfwg2MEBvw8OH5wirsPdfcng/vrW6vlj8Cf3P0k4GsEFkMTaXaUCESOwt2LgDuA75tZGwILn/2XmSUBmFkfM0sD5gFXm1n34P6DXUOdgZzg6xsRaabUNSTSAHdfbmYrgRnu/lxwmeOFwRVcS4Drg6th/hT40MyqCXQd3UTgyVmvmlkOgWWwB0WjDiKN0eqjIiIxTl1DIiIxTolARCTGKRGIiMQ4JQIRkRinRCAiEuOUCEREYpwSgYhIjPt/UD1ImeB6xQkAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from sklearn.metrics import precision_recall_curve\n", "\n", "precision, recall, thresholds = precision_recall_curve(y_test, y_pred_prob)\n", "\n", "plt.plot(recall, precision)\n", "plt.xlabel('Recall')\n", "plt.ylabel('Precision')\n", "plt.title('Precision / Recall plot')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Area under the ROC curve (AUC)\n", "- Larger area under the ROC curve = better model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### AUC computation\n", "Say you have a binary classifier that in fact is just randomly making guesses. It would be correct approximately 50% of the time, and the resulting ROC curve would be a diagonal line in which the True Positive Rate and False Positive Rate are always equal. The Area under this ROC curve would be 0.5. This is one way in which the AUC, which Hugo discussed in the video, is an informative metric to evaluate a model. If the AUC is greater than 0.5, the model is better than random guessing. Always a good sign!\n", "\n", "In this exercise, you'll calculate AUC scores using the ```roc_auc_score()``` function from ```sklearn.metrics``` as well as by performing cross-validation on the diabetes dataset." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "AUC: 0.8243384732533791\n", "AUC scores computed using 5-fold cross-validation: [0.81240741 0.80777778 0.82574074 0.87283019 0.84471698]\n" ] } ], "source": [ "from sklearn.metrics import roc_auc_score\n", "from sklearn.model_selection import cross_val_score\n", "\n", "# Compute predicted probabilites: y_pred_prob\n", "y_pred_prob = logreg.predict_proba(X_test)[:, 1]\n", "\n", "# Compute and print AUC score\n", "print(\"AUC: {}\".format(roc_auc_score(y_test, y_pred_prob)))\n", "\n", "# Compute cross-validated AUC scores: cv_auc\n", "cv_auc = cross_val_score(logreg, X, y, cv=5, scoring='roc_auc')\n", "\n", "# Print list of AUC scores\n", "print(\"AUC scores computed using 5-fold cross-validation: {}\".format(cv_auc))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Hyperparameter tuning\n", "- Linear regression: Choosing parameters\n", "- Ridge/Lasso regression: Choosing alpha\n", "- k-Nearest Neighbors: Choosing n_neighbors\n", "- Hyperparameters: Parameters like alpha and k\n", "- Hyperparameters cannot be learned by fitting the model\n", "- Choosing the correct hyperparameter\n", " - Try a bunch of different hyperparameter values\n", " - Fit all of them separately\n", " - See how well each performs\n", " - Choose the best performing one\n", " - It is essential to use cross-validation\n", "- Grid search cross-validation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Hyperparameter tuning with GridSearchCV\n", "Like the alpha parameter of lasso and ridge regularization that you saw earlier, logistic regression also has a regularization parameter: $C$. $C$ controls the inverse of the regularization strength, and this is what you will tune in this exercise. A large $C$ can lead to an overfit model, while a small $C$ can lead to an underfit model.\n", "\n", "The hyperparameter space for $C$ has been setup for you. Your job is to use GridSearchCV and logistic regression to find the optimal $C$ in this hyperparameter space. \n", "\n", "You may be wondering why you aren't asked to split the data into training and test sets. Good observation! Here, we want you to focus on the process of setting up the hyperparameter grid and performing grid-search cross-validation. In practice, you will indeed want to hold out a portion of your data for evaluation purposes, and you will learn all about this in the next video!" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tuned Logistic Regression Parameters: {'C': 0.006105402296585327}\n", "Best score is 0.7734742381801205\n" ] } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "from sklearn.model_selection import GridSearchCV\n", "\n", "# Setup the hyperparameter grid\n", "c_space = np.logspace(-5, 8, 15)\n", "param_grid = {'C':c_space}\n", "\n", "# Instantiate a logistic regression classifier: logreg\n", "logreg = LogisticRegression(max_iter=1000)\n", "\n", "# Instantiate the GridSearchCV object: logreg_cv\n", "logreg_cv = GridSearchCV(logreg, param_grid, cv=5)\n", "\n", "# Fit it to the data\n", "logreg_cv.fit(X, y)\n", "\n", "# Print the tuned parameters and score\n", "print(\"Tuned Logistic Regression Parameters: {}\".format(logreg_cv.best_params_))\n", "print(\"Best score is {}\".format(logreg_cv.best_score_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Hyperparameter tuning with RandomizedSearchCV\n", "GridSearchCV can be computationally expensive, especially if you are searching over a large hyperparameter space and dealing with multiple hyperparameters. A solution to this is to use ```RandomizedSearchCV```, in which not all hyperparameter values are tried out. Instead, a fixed number of hyperparameter settings is sampled from specified probability distributions. You'll practice using ```RandomizedSearchCV``` in this exercise and see how this works.\n", "\n", "Here, you'll also be introduced to a new model: the Decision Tree. Don't worry about the specifics of how this model works. Just like k-NN, linear regression, and logistic regression, decision trees in scikit-learn have ```.fit()``` and ```.predict()``` methods that you can use in exactly the same way as before. Decision trees have many parameters that can be tuned, such as ```max_features```, ```max_depth```, and ```min_samples_leaf```: This makes it an ideal use case for ```RandomizedSearchCV```." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tuned Decision Tree Parameters: {'criterion': 'entropy', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 4}\n", "Best score is 0.7448603683897801\n" ] } ], "source": [ "from scipy.stats import randint\n", "from sklearn.tree import DecisionTreeClassifier\n", "from sklearn.model_selection import RandomizedSearchCV\n", "\n", "# Setup the parameters and distributions to sample from: param_dist\n", "param_dist = {\n", " \"max_depth\": [3, None],\n", " \"max_features\": randint(1, 9),\n", " \"min_samples_leaf\": randint(1, 9),\n", " \"criterion\": [\"gini\", \"entropy\"],\n", "}\n", "\n", "# Instantiate a Decision Tree classifier: tree\n", "tree = DecisionTreeClassifier()\n", "\n", "# Instantiate the RandomizedSearchCV object: tree_cv\n", "tree_cv = RandomizedSearchCV(tree, param_dist, cv=5)\n", "\n", "# Fit it to the data\n", "tree_cv.fit(X, y)\n", "\n", "# Print the tuned parameters and score\n", "print(\"Tuned Decision Tree Parameters: {}\".format(tree_cv.best_params_))\n", "print(\"Best score is {}\".format(tree_cv.best_score_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Hold-out set for final evaluation\n", "- How well can the model perform on never before seen data?\n", "- Using ALL data for cross-validation is not ideal\n", "- Split data into training and hold-out set at the beginning\n", "- Perform grid search cross-validation on training set\n", "- Choose best hyperparameters and evaluate on hold-out set" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Hold-out set in practice I: Classification\n", "You will now practice evaluating a model with tuned hyperparameters on a hold-out set. The feature array and target variable array from the diabetes dataset have been pre-loaded as ```X``` and ```y```.\n", "\n", "In addition to $C$, logistic regression has a ```'penalty'``` hyperparameter which specifies whether to use ```'l1'``` or ```'l2'``` regularization. Your job in this exercise is to create a hold-out set, tune the ```'C'``` and ```'penalty'``` hyperparameters of a logistic regression classifier using ```GridSearchCV``` on the training set." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tuned Logistic Regression Parameter: {'C': 3.727593720314938, 'penalty': 'l2'}\n", "Tuned Logistic Regression Accuracy: 0.7608695652173914\n" ] } ], "source": [ "from sklearn.model_selection import train_test_split, GridSearchCV\n", "from sklearn.linear_model import LogisticRegression\n", "\n", "# Create the hyperparameter grid\n", "c_space = np.logspace(-5, 8, 15)\n", "param_grid = {'C': c_space, 'penalty': ['l1', 'l2']}\n", "\n", "# Instantiate the logistic regression classifier: logreg\n", "logreg = LogisticRegression(max_iter=1000, solver='liblinear')\n", "\n", "# Create train and test sets\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)\n", "\n", "# Instantiate the GridSearchCV object: logreg_cv\n", "logreg_cv = GridSearchCV(logreg, param_grid, cv=5)\n", "\n", "# Fit it to the training data\n", "logreg_cv.fit(X_train, y_train)\n", "\n", "# Print the optimal parameters and best score\n", "print(\"Tuned Logistic Regression Parameter: {}\".format(logreg_cv.best_params_))\n", "print(\"Tuned Logistic Regression Accuracy: {}\".format(logreg_cv.best_score_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Hold-out set in practice II: Regression\n", "Remember lasso and ridge regression from the previous chapter? Lasso used the $L1$ penalty to regularize, while ridge used the $L2$ penalty. There is another type of regularized regression known as the elastic net. In elastic net regularization, the penalty term is a linear combination of the $L1$ and $L2$ penalties:\n", "$$ a * L1 + b * L2 $$\n", "\n", "In scikit-learn, this term is represented by the ```'l1_ratio'``` parameter: An ```'l1_ratio'``` of 1 corresponds to an $L1$ penalty, and anything lower is a combination of $L1$ and $L2$.\n", "\n", "In this exercise, you will ```GridSearchCV``` to tune the ```'l1_ratio'``` of an elastic net model trained on the Gapminder data. As in the previous exercise, use a hold-out set to evaluate your model's performance." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Preprocess" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
populationfertilityHIVCO2BMI_maleGDPBMI_femalelifechild_mortality
034811059.02.730.13.32894524.5962012314.0129.904975.329.5
119842251.06.432.01.47435322.250837103.0130.124758.3192.0
240381860.02.240.54.78517027.5017014646.0118.891575.515.4
32975029.01.400.11.80410625.355427383.0132.810872.520.0
421370348.01.960.118.01631327.5637341312.0117.375581.55.2
\n", "
" ], "text/plain": [ " population fertility HIV CO2 BMI_male GDP BMI_female life \\\n", "0 34811059.0 2.73 0.1 3.328945 24.59620 12314.0 129.9049 75.3 \n", "1 19842251.0 6.43 2.0 1.474353 22.25083 7103.0 130.1247 58.3 \n", "2 40381860.0 2.24 0.5 4.785170 27.50170 14646.0 118.8915 75.5 \n", "3 2975029.0 1.40 0.1 1.804106 25.35542 7383.0 132.8108 72.5 \n", "4 21370348.0 1.96 0.1 18.016313 27.56373 41312.0 117.3755 81.5 \n", "\n", " child_mortality \n", "0 29.5 \n", "1 192.0 \n", "2 15.4 \n", "3 20.0 \n", "4 5.2 " ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_csv('./dataset/gm_2008_region.csv')\n", "df.drop(labels=['Region'], axis='columns', inplace=True)\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "X = df.drop('life', axis='columns').values\n", "y = df['life'].values" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\kcsgo\\anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 282.48621758506584, tolerance: 5.58941590909091\n", " positive)\n", "C:\\Users\\kcsgo\\anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 309.8466391486277, tolerance: 5.893071666666667\n", " positive)\n", "C:\\Users\\kcsgo\\anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 255.50344008061325, tolerance: 5.890250303030303\n", " positive)\n", "C:\\Users\\kcsgo\\anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 287.6728412633782, tolerance: 5.814186865671641\n", " positive)\n", "C:\\Users\\kcsgo\\anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 311.1827114768199, tolerance: 5.801944179104479\n", " positive)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Tuned ElasticNet l1 ratio: {'l1_ratio': 0.20689655172413793}\n", "Tuned ElasticNet R squared: 0.8668305372460283\n", "Tuned ElasticNet MSE: 10.057914133398445\n" ] } ], "source": [ "from sklearn.linear_model import ElasticNet\n", "from sklearn.metrics import mean_squared_error\n", "from sklearn.model_selection import GridSearchCV, train_test_split\n", "\n", "# Create train and test sets\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)\n", "\n", "# Create the hyperparameter grid\n", "l1_space = np.linspace(0, 1, 30)\n", "param_grid = {'l1_ratio': l1_space}\n", "\n", "# Instantiate the ElasticNet regressor: elastic_net\n", "elastic_net = ElasticNet(max_iter=100000, tol=0.001)\n", "\n", "# Setup the GridSearchCV object: gm_cv\n", "gm_cv = GridSearchCV(elastic_net, param_grid, cv=5)\n", "\n", "# Fit it to the training data\n", "gm_cv.fit(X_train, y_train)\n", "\n", "# Predict on the test set and compute metrics\n", "y_pred = gm_cv.predict(X_test)\n", "r2 = gm_cv.score(X_test, y_test)\n", "mse = mean_squared_error(y_pred,y_test)\n", "print(\"Tuned ElasticNet l1 ratio: {}\".format(gm_cv.best_params_))\n", "print(\"Tuned ElasticNet R squared: {}\".format(r2))\n", "print(\"Tuned ElasticNet MSE: {}\".format(mse))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }