{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Training and Testing Datasets\n", "Author: Ravin Poudel" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "Our goal in statistics or machine learning is to build a model. Often we start with a set of data, fit a model of choice to the data, publish the model. However, it is equally important to test the model with new data and check/evaluate the model performance. Model validation requires a new set of data; the data that has not been used in fitting a model or the model has never seen these data. From an agricultural perspective, it means an additional experiment to generate data for model validation. Instead, we can __randomly__ divide a single dataset into two sets. Then use one set for training the model and other sets for testing/evaluating the learned model.\n", "\n", "\n", "\n", "> Train data set: A data set used to __construct/train/learn__ a model. \n", "\n", "> Test data set: A data set used to __evaluate__ the model.\n", "\n", "\n", "\n", "#### How do we spilit a single dataset into two?\n", "\n", "There is not a single or one best solution. Conventionally more data is used for model training than for model testing. Often convention such as `75%/ 25% train/ test or 90%/10% train/test` scheme are used. Regardless of how we decide to split the dataset, there are some pros and some cons. For instance, a larger training dataset allows us to learn the model better, while the larger testing dataset increases confidence in the model evaluation. _(Don't forget to evaluate sd in model accuracy among various approaches discussed in the sections below)_.\n", "\n", "Before we split the data, we also need to keep in mind the following question.\n", "\n", "> Can we apply similar data-splitting scheme when we have a small dataset? Often the case in agriculture or life sciences - \"as of now.\"\n", "\n", "> Does a single random split make our predictive model random? Do we want a stable model or a random model?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's start working on python. Here using `iris dataset`, we will explore the data splitting scheme, then build and evaluate the model. We will also explore various cross-validation methods briefly. The main goal of this module is to provide a general overview of creating train and test dataset, apply them to build a model, and evaluate the model performance. Beyond this model, you will be using this concept of train/test data throughout the other advanced modules in the workshop or the rest of your research." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `iris dataset` contains:\n", "\n", "- 50 samples of 3 different species of iris flower (150 samples in total)\n", "- Iris flower: Setosa, Versicolour, and Virginica\n", "- Measurements: sepal length, sepal width, petal length, petal width\n" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "# import modules\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "from sklearn import datasets\n", "from sklearn.model_selection import train_test_split\n" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "# import iris data from scikit and data preparation\n", "\n", "iris = datasets.load_iris() # inbuilt data \n", "iris_X = iris['data'] # features data\n", "iris_y = iris['target'] # this has information about the flower type, has been coded as 0, 1, or 2.\n", "names = iris['target_names'] # flower type\n", "feature_names = iris['feature_names'] # features name\n" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(150, 4)" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# check data shape\n", "\n", "iris_X.data.shape\n" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n", " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n", " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n", " 2 2]\n" ] } ], "source": [ "print(iris_y)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['setosa' 'versicolor' 'virginica']\n" ] } ], "source": [ "print(names)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n" ] } ], "source": [ "print(feature_names)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "# splitting into train and test data. For example, test dataset = 25% of the original dataset\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(iris_X, iris_y, test_size=0.25, random_state=0)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((112, 4), (112,))" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# shape of train dataset\n", "\n", "X_train.shape, y_train.shape" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((38, 4), (38,))" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# shape of test dataset\n", "\n", "X_test.shape, y_test.shape" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "# instantiate a K-Nearest Neighbors(KNN) model, and fit with X and y\n", "\n", "from sklearn.neighbors import KNeighborsClassifier\n", "model = KNeighborsClassifier()\n", "model_tt = model.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "NOTE: Here we are using KNeighborsClassifier model. Any other model, approprite to your study can be deployed. If you are interested to learn more on models, please follow [scikit-learn](https://scikit-learn.org/stable/)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9732142857142857" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# check the accuracy on the training set\n", "model_tt.score(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2 1 0 2 0 2 0 1 1 1 2 1 1 1 1 0 1 1 0 0 2 1 0 0 2 0 0 1 1 0 2 1 0 2 2 1 0\n", " 2]\n" ] } ], "source": [ "# predict class labels for the test set\n", "predicted = model_tt.predict(X_test)\n", "print (predicted)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2 1 0 2 0 2 0 1 1 1 2 1 1 1 1 0 1 1 0 0 2 1 0 0 2 0 0 1 1 0 2 1 0 2 2 1 0\n", " 1]\n" ] } ], "source": [ "print(y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> Did you see any differences is the predicted and test output?" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Text(0, 0.5, 'Predicted Values')" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAaqUlEQVR4nO3dfbRddX3n8feHEEZECtZc0eaBoGYxBZSHngY0VmGWaCLaVMexYayODzUNhaW2M9ToGqHVdg0V6yq6QIyYUazA1JEwUR4Cth3TBUVzw/Ojk4Y4XEInFxAImDFc/Mwfe9/J4Wafc/dN7j7nJvfzWuuse/bvYZ/vPWzuN3vv3/79ZJuIiIixDuh3ABERMTUlQURERKUkiIiIqJQEERERlZIgIiKi0oH9DmAyzZo1y/Pnz+93GBER+4yNGzc+Znugqm6/ShDz589ncHCw32FEROwzJP20U10uMUVERKUkiIiIqJQEERERlZIgIiKiUhJERERUSoKIiIhKjSUISXMl/YOk+yXdK+njFW0k6UuSNkm6S9JJbXWLJT1Y1q1sKs6IiKjW5HMQI8B/tH2bpEOBjZJusn1fW5slwILydTLwFeBkSTOAi4HTgSFgg6S1Y/pGTHnX3P4IF657kK1P7uDXDj+Yc992NL9z4ux+hxX7gaNWXkv7Yg0CHrrgjEn9jMbOIGw/avu28v124H5g7P8ZS4HLXbgVOFzSK4GFwCbbm23vBK4q20bsM665/RE+dfXdPPLkDgw88uQOPnX13Vxz+yP9Di32cWOTA4DL8snUk3sQkuYDJwI/GlM1G3i4bXuoLOtUHrHPuHDdg+x47vkXlO147nkuXPdgnyKK/UWnZd4me/m3xhOEpJcA3wU+YfvpsdUVXdylvGr/yyUNShocHh7eu2AjJtHWJ3dMqDxiqmk0QUiaSZEcvm376oomQ8Dctu05wNYu5buxvcp2y3ZrYKByvqmIvvi1ww+eUHnEVNPkKCYBXwfut/3FDs3WAh8oRzOdAjxl+1FgA7BA0lGSDgKWlW0j9hnnvu1oDp454wVlB8+cwblvO7pPEcX+ouoSS7fyPdXkKKZFwPuBuyXdUZZ9GpgHYPtS4Drg7cAm4OfAh8q6EUnnAOuAGcBq2/c2GGvEpBsdrZRRTDHZHrrgjJ6MYpI92bc1+qfVajnTfUdE1Cdpo+1WVV2epI6IiEpJEBERUSkJIiIiKiVBREREpSSIiIiolAQRERGVkiAiIqJSEkRERFRKgoiIiEpJEBERUSkJIiIiKiVBREREpSSIiIiolAQRERGVkiAiIqJSEkRERFRqbEU5SauBdwDbbB9XUX8u8L62OH4dGLD9hKQtwHbgeWCk02IWERHRnCbPIL4BLO5UaftC2yfYPgH4FPBD20+0NTmtrE9yiIjog8YShO31wBPjNiycCVzZVCwRETFxfb8HIenFFGca320rNnCjpI2Slo/Tf7mkQUmDw8PDTYYaETGt9D1BAO8Ebh5zeWmR7ZOAJcDZkt7UqbPtVbZbtlsDAwNNxxoRMW1MhQSxjDGXl2xvLX9uA9YAC/sQV0TEtNbXBCHpMODNwP9oKztE0qGj74G3Avf0J8KIiOmryWGuVwKnArMkDQHnAzMBbF9aNnsXcKPtZ9u6HgGskTQa3xW2b2gqzoiIqNZYgrB9Zo0236AYDttethk4vpmoIiKirqlwDyIiIqagJIiIiKiUBBEREZWSICIiolISREREVEqCiIiISkkQERFRKQkiIiIqJUFERESlJIiIiKiUBBEREZWSICIiolISREREVEqCiIiISkkQERFRqbEEIWm1pG2SKleDk3SqpKck3VG+zmurWyzpQUmbJK1sKsaIiOisyTOIbwCLx2nzj7ZPKF+fBZA0A7gYWAIcA5wp6ZgG44yIiAqNJQjb64En9qDrQmCT7c22dwJXAUsnNbiIiBhXv+9BvF7SnZKul3RsWTYbeLitzVBZVknSckmDkgaHh4ebjDUiYlrpZ4K4DTjS9vHAl4FrynJVtHWnndheZbtluzUwMNBAmBER01PfEoTtp20/U76/DpgpaRbFGcPctqZzgK19CDEiYlrrW4KQ9ApJKt8vLGN5HNgALJB0lKSDgGXA2n7FGRExXR3Y1I4lXQmcCsySNAScD8wEsH0p8B7gLEkjwA5gmW0DI5LOAdYBM4DVtu9tKs6IiKim4m/y/qHVanlwcLDfYURE7DMkbbTdqqrr9yimiIiYopIgIiKiUhJERERUGjdBSFok6ZDy/e9J+qKkI5sPLSIi+qnOGcRXgJ9LOh74E+CnwOWNRhUREX1XJ0GMlMNPlwIX2b4IOLTZsCIiot/qPAexXdKngPcDv1XOtjqz2bAiIqLf6pxB/C7wC+DDtv+FYuK8CxuNKiIi+m7cBFEmhe8C/6osegxY02RQERHRf3VGMX0U+O/AV8ui2eyaeTUiIvZTdS4xnQ0sAp4GsP2/gJc3GVRERPRfnQTxi3JlNwAkHUiX9RkiImL/UCdB/FDSp4GDJZ0OfAf4XrNhRUREv9VJECuBYeBu4A+A64D/3GRQERHRf+M+B2H7l8DXyldEREwT4yYISQ9Rcc/B9qsaiSgiIqaEOk9Sty8k8SLg3wG/Ol4nSauBdwDbbB9XUf8+4JPl5jPAWbbvLOu2ANuB5ymm+qhczCIiIppT50G5x9tej9j+a+Df1Nj3N4DFXeofAt5s+3XA54BVY+pPs31CkkNERH/UucR0UtvmARRnFONO1md7vaT5Xepvadu8FZgz3j4jIqJ36lxi+qu29yPAFuC9kxzHR4Dr27YN3CjJwFdtjz27+P8kLQeWA8ybN2+Sw4qImL7qjGI6rckAJJ1GkSDe2Fa8yPZWSS8HbpL0gO31HeJbRXl5qtVq5QG+iIhJ0jFBSPrjbh1tf3FvP1zS64DLgCW2H2/b99by5zZJa4CFQGWCiIiIZnQ7g2h0USBJ84Crgffb/klb+SHAAba3l+/fCny2yVgiImJ3HROE7T/bmx1LuhI4FZglaQg4n3KhIduXAucBLwMukQS7hrMeAawpyw4ErrB9w97EEhERE1dnFNOLKO4RHEvxHAQAtj/crZ/tM8ep/33g9yvKNwPHjxdXREQ0q85cTN8CXgG8DfghxXDU7U0GFRER/VcnQbzG9meAZ21/EzgDeG2zYUVERL/VSRDPlT+flHQccBgwv7GIIiJiSqjzoNwqSS+lmOJ7LfAS4DONRhUREX3X7TmII2z/H9uXlUXrgczgGhExTXS7xHSnpJskfVjSYT2LKCIipoRuCWI28AXgt4CfSLpG0u9KOrg3oUVERD91TBC2n7e9zvaHgLnAfwV+B3hI0rd7FWBERPRHnVFM2N4J3AfcDzwNHNNkUBER0X9dE4SkeZLOlXQb8H1gBrDU9ok9iS4iIvqm2yimWyjuQ3wHWG57sGdRRURE33V7DuJTwHrbWWMhImIa6jab6w97GUhEREwttW5SR0TE9JMEERERlRpbclTSauAdwDbbx1XUC7gIeDvwc+CDtm8r6xaXdTOAy2xfMM7vETElzV957W5lWy44ow+RRExctzOIQ8tXCziLYkTTbGAF9Z6D+AawuEv9EmBB+VoOfAVA0gzg4rL+GOBMSXnuIvY5VcmhW3nEVDPukqOSbgROsr293P5TiqGvXdleL2l+lyZLgcvLUVK3Sjpc0ispphLfVK4sh6Sryrb31fh9IiJiktS5BzEP2Nm2vZPJWQ9iNvBw2/YQu85SqsorSVouaVDS4PDw8CSEFRERUG89iG8BP5a0BjDwLuDySfhsVZS5S3kl26uAVQCtVivPbERETJJxE4Ttv5B0PcWsrgAfsn37JHz2EMUkgKPmAFuBgzqUR0RED9Ud5vpi4GnbFwFDko6ahM9eC3xAhVOAp2w/CmwAFkg6StJBwLKybcQ+pdNopYxiin3FuGcQks6nGMl0NMWU3zOBvwEWjdPvSuBUYJakIeD8si+2LwWuoxjiuolimOuHyroRSecA6yiGua62fe8e/G4RfZdkEPuyOvcg3gWcCNwGYHurpEPH62T7zHHqDZzdoe46igQSERF9UucS087yj7kBJB3SbEgRETEV1EkQfyvpq8Dhkj4K/AC4rNmwIiKi3+qMYvqCpNMpVpI7GjjP9k2NRxYREX1V5yb1X9r+JHBTRVlEROyn6lxiOr2ibMlkBxIREVNLt9lczwL+EHi1pLvaqg4Fbmk6sIiI6K9ul5iuAK4H/guwsq18u+0nGo0qIiL6ruMlJttP2d5CsS7DE7Z/avunwHOSTu5VgBER0R917kF8BXimbfvZsiwiIvZjdRKEygflALD9S+o9gR0REfuwOglis6SPSZpZvj4ObG46sIiI6K86CWIF8AbgEYopuk+mWCI0IiL2Y3WepN5GMeV2RERMI92eg/gT25+X9GUqVnSz/bFGI4uIiL7qdgZxf/lzsBeBRETE1NIxQdj+Xvnzm3u6c0mLKZ6jmAFcZvuCMfXnAu9ri+XXgQHbT0jaAmwHngdGbLf2NI6IiJi4bpeYvkfFpaVRtn+7244lzQAuppjLaQjYIGmt7fva9nEhcGHZ/p3AH415Svs024/V+UUiImJydbvE9IXy57uBV1AsMwpwJrClxr4XAptsbwaQdBWwFLivQ/szgStr7DciInqg2yWmHwJI+pztN7VVfU/S+hr7ng083LY9OkR2N5JeDCwGzmkPAbhRkoGv2l7Voe9yymG38+bNqxFWRETUUec5iAFJrxrdkHQUMFCjnyrKOl2yeidw85jLS4tsn0QxtfjZkt5U1dH2Ktst262BgTphRUREHXWmzPgj4H9KGn16ej7wBzX6DQFz27bnAFs7tF3GmMtLtreWP7dJWkNxyarOmUtEREyCOg/K3SBpAfCvy6IHbP+ixr43AAvKM45HKJLAvx/bSNJhwJuB32srOwQ4wPb28v1bgc/W+MyIiJgkdZYcfTHwx8CRtj8qaYGko21/v1s/2yOSzgHWUQxzXW37XkkryvpLy6bvAm60/Wxb9yOANZJGY7zC9g0T/eUiImLPqW2i1uoG0n8DNgIfsH2cpIOBf7J9Qi8CnIhWq+XBwTzXFxFRl6SNnZ4zq3OT+tW2Pw88B2B7B9U3oCMiYj9SJ0HsLM8aDCDp1UCdexAREbEPqzOK6XzgBmCupG8Di4APNhlURET0X9cEoeIu8QMUT1OfQnFp6eOZ/iIiYv/XNUHYtqRrbP8GcG2PYoqIiCmgzj2IWyX9ZuORRETElFLnHsRpwIpy+u1nKS4z2fbrmgwsIiL6q06CWNJ4FBERMeV0Ww/iRcAK4DXA3cDXbY/0KrCIiOivbvcgvgm0KJLDEuCvehJRRERMCd0uMR1j+7UAkr4O/Lg3IUVExFTQ7QziudE3ubQUETH9dDuDOF7S0+V7AQeX26OjmH6l8egiIqJvui05OqOXgURExNRS50G5iIiYhpIgIiKiUqMJQtJiSQ9K2iRpZUX9qZKeknRH+Tqvbt+IiGhWnSep94ikGcDFwOnAELBB0lrb941p+o+237GHfSMioiFNnkEsBDbZ3mx7J3AVsLQHfSMiYhI0mSBmAw+3bQ+VZWO9XtKdkq6XdOwE+yJpuaRBSYPDw8OTEXdERNBsgqhat9pjtm8DjrR9PPBl4JoJ9C0K7VW2W7ZbAwMDexxsRES8UJMJYgiY27Y9B9ja3sD207afKd9fB8yUNKtO34iIaFaTCWIDsEDSUZIOApYBa9sbSHpFuawpkhaW8Txep29ERDSrsVFMtkcknQOsA2YAq23fK2lFWX8p8B7gLEkjwA5gmW0DlX2bijUiInan4u/x/qHVanlwcLDfYURE7DMkbbTdqqrLk9QREVEpCSIiIiolQURERKUkiIiIqJQEERERlZIgIiKiUhJERERUSoKIiIhKSRAREVEpCSIiIiolQURERKUkiIiIqJQEERERlZIgIiKiUhJERERUajRBSFos6UFJmyStrKh/n6S7ytctko5vq9si6W5Jd0jKIg8RET3W2IpykmYAFwOnU6wxvUHSWtv3tTV7CHiz7Z9JWgKsAk5uqz/N9mNNxRgREZ01eQaxENhke7PtncBVwNL2BrZvsf2zcvNWYE6D8URExAQ0mSBmAw+3bQ+VZZ18BLi+bdvAjZI2SlreqZOk5ZIGJQ0ODw/vVcAREbFLY5eYAFWUVS6ALek0igTxxrbiRba3Sno5cJOkB2yv322H9iqKS1O0Wq39Z4HtiIg+a/IMYgiY27Y9B9g6tpGk1wGXAUttPz5abntr+XMbsIbiklVERPRIkwliA7BA0lGSDgKWAWvbG0iaB1wNvN/2T9rKD5F06Oh74K3APQ3GGhERYzR2icn2iKRzgHXADGC17XslrSjrLwXOA14GXCIJYMR2CzgCWFOWHQhcYfuGpmKNiIjdyd5/Ltu3Wi0PDuaRiYiIuiRtLP9hvps8SR0REZWSICIiolISREREVEqCiIiISkkQERFRKQkiIiIqJUFERESlJIiIiKiUBBEREZWSICIiolISREREVEqCiIiISkkQERFRKQkiIiIqJUFERESlRhOEpMWSHpS0SdLKinpJ+lJZf5ekk+r2jYiIZjW2opykGcDFwOkU61NvkLTW9n1tzZYAC8rXycBXgJNr9p0081deu1vZlgvOaOKjIiL2GU2eQSwENtnebHsncBWwdEybpcDlLtwKHC7plTX7Toqq5NCtPCJiumgyQcwGHm7bHirL6rSp0zciIhrUZIJQRdnYBbA7tanTt9iBtFzSoKTB4eHhCYYYERGdNJkghoC5bdtzgK0129TpC4DtVbZbtlsDAwN7HXRERBSaTBAbgAWSjpJ0ELAMWDumzVrgA+VoplOAp2w/WrNvREQ0qLEEYXsEOAdYB9wP/K3teyWtkLSibHYdsBnYBHwN+MNufZuIs9NopYxiiojpTnblpf19UqvV8uDgYL/DiIjYZ0jaaLtVVZcnqSMiolISREREVEqCiIiISkkQERFRKQkiIiIqJUFERESl/WqYq6Rh4Kd7sYtZwGOTFM5kSlz1TcWYIHFNxFSMCfbfuI60XTkNxX6VIPaWpMFO44H7KXHVNxVjgsQ1EVMxJpieceUSU0REVEqCiIiISkkQL7Sq3wF0kLjqm4oxQeKaiKkYE0zDuHIPIiIiKuUMIiIiKiVBREREpWmTICQtlvSgpE2SVlbUS9KXyvq7JJ1Ut2+DMb2vjOUuSbdIOr6tboukuyXdIWlS5zivEdepkp4qP/sOSefV7dtwXOe2xXSPpOcl/WpZ18j3JWm1pG2S7ulQ3/PjqmZcPT+2asTUr+NqvLj6cVzNlfQPku6XdK+kj1e0af7Ysr3fv4AZwD8DrwIOAu4EjhnT5u3A9RTrYZ8C/Khu3wZjegPw0vL9ktGYyu0twKw+fVenAt/fk75NxjWm/TuBv+/B9/Um4CTgng71PT2uJhBXP46t8WLq+XFVJ64+HVevBE4q3x8K/KQff7OmyxnEQmCT7c22dwJXAUvHtFkKXO7CrcDhkl5Zs28jMdm+xfbPys1bKdbmbtre/L5NfVd7su8zgSsn6bM7sr0eeKJLk14fV7Xi6sexVeO76qSv39UYvTquHrV9W/l+O8XKmrPHNGv82JouCWI28HDb9hC7f9md2tTp21RM7T5C8a+FUQZulLRR0vJJiGeicb1e0p2Srpd07AT7NhkXkl4MLAa+21bc1Pc1nl4fV3uiV8dWHb0+rmrr13ElaT5wIvCjMVWNH1sH7kmnfZAqysaO7+3Upk7fPVF7v5JOo/if+I1txYtsb5X0cuAmSQ+U/xLqRVy3Uczf8oyktwPXAAtq9m0yrlHvBG623f6vwqa+r/H0+riakB4fW+Ppx3E1ET0/riS9hCIhfcL202OrK7pM6rE1Xc4ghoC5bdtzgK0129Tp21RMSHodcBmw1Pbjo+W2t5Y/twFrKE4rJ8O4cdl+2vYz5fvrgJmSZtXp22RcbZYx5jJAg9/XeHp9XNXWh2Orqz4dVxPR0+NK0kyK5PBt21dXNGn+2JrsmytT8UVxprQZOIpdN22OHdPmDF54w+fHdfs2GNM8YBPwhjHlhwCHtr2/BVjcw+/qFex6yHIh8L/L762R72oi/x2AwyiuJx/Si++r3Od8Ot947elxNYG4en5s1Yip58dVnbj6cVyVv/flwF93adP4sTUtLjHZHpF0DrCO4g7/atv3SlpR1l8KXEcxKmAT8HPgQ9369iim84CXAZdIAhhxMWvjEcCasuxA4ArbN+xtTBOI6z3AWZJGgB3AMhdHZiPf1QTiAngXcKPtZ9u6N/Z9SbqSYvTNLElDwPnAzLaYenpcTSCunh9bNWLq+XFVMy7o8XEFLALeD9wt6Y6y7NMUib1nx1am2oiIiErT5R5ERERMUBJERERUSoKIiIhKSRAREVEpCSIiIiolQcS0IullbTNz/oukR9q2D5qkzzhU0uPlU7Dt5d+X9O4u/d4i6ZrJiCFiMkyL5yAiRrl4YvgEAEl/Cjxj+wvtbVQMbJftX+7hZ2yX9PcUE6R9u9znS4GTKcb6R+wTcgYRAUh6TTnX/6UUcwLNlfRkW/0ySZeV74+QdLWkQUk/lnRKxS6vpJiaYdS/Ba61/X8lnSLpnyTdLulmSQsq4vlzSZ9o235A0pzy/X8oP/cOSZdIOkDSgZK+pWJtgnskfWxyvpmYzpIgInY5Bvi67ROBR7q0+xLw+fLJ4/dSzGc01rXAKeWZA7xwHp/7gTeWn/M54M/rBijpOIqnet9g+wSKqwDLgN+gWJfgtbaPo5imIWKv5BJTxC7/bHtDjXZvAY4up1gAeKmkg23vGC2w/QtJ1wLvlvR94Fjg78rqw4HLJb16D2J8C/CbwGD5+QdTTO28rozpIoopGG7cg31HvEASRMQu7fPs/JIXTpv8orb3Aha6WIylmyuB/0TxR/xq2yNl+V8A62xfIuk1QNX8PSO88Ax/9PNFMbfOZ8Z2KGdnXQJ8jOKSVq/Xcoj9TC4xRVQob1D/TNICSQdQXNYZ9QPg7NENSSd02M0PKM4cVvDCaaIPY9clrA926LuF4rIRkhaya/rmHwDvLafBHh2VNU/SAMWN9e9QTDZ30u67jJiYJIiIzj5J8a/7v6OYY3/U2cAiFQvF3wd8tKqz7ecp1gj4FeDmtqq/BC6UdHNVv9J3gCMk3U6xoM/mcp93A38G/EDSXRSXko6gSCDry5k/v0Yx82fEXslsrhERUSlnEBERUSkJIiIiKiVBREREpSSIiIiolAQRERGVkiAiIqJSEkRERFT6f2U6la8+JZjwAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# scatter plot\n", "plt.scatter(y_test, predicted)\n", "plt.xlabel(\"True Values\")\n", "plt.ylabel(\"Predicted Values\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> Note: There are overlapping of predicted values and true values in the scatter plot. There are more than four points; more precisely there are 38 points." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.9736842105263158\n" ] } ], "source": [ "# generate evaluation metrics\n", "from sklearn import metrics\n", "print (metrics.accuracy_score(y_test, predicted))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Confusion Matrix\n", "Also known as an error matrix. Each row of the matrix represents the instances in a predicted class while each column represents the instances in an actual class (or vice versa)." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[13 0 0]\n", " [ 0 15 1]\n", " [ 0 0 9]]\n" ] } ], "source": [ "print (metrics.confusion_matrix(y_test, predicted))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## NOTE:\n", "\n", "> Never train model on your test dataset.\n", "\n", "> Be suspesious: If you ever happen to have 100% accuracy in your model __(overf-fitting)__ with test data, be suspecious and double check if you have not used test dataset for traning your model. \n", "\n", "> __over-fitting:__ model performs very well on the training data but poorly on the test data. Model follows exactly the same trend as the training dataset. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Model Evaluation via Cross-Validation\n", "\n", "Results of train and test split are based on a single random split. Given we are randomly splitting datasets, each time we ran the model, we might get a slightly different results. To minimize stochasticity in the model, rather we can use cross-validation approaches which are robust to these issues. Besides, these approaches are much suitable, especially when you have a smaller dataset. There are many [methods](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection) of cross-validation available in scikit-learn, but just to get started we will be learning:\n", "\n", "- K-Folds Cross-Validation\n", "\n", "- Leave One Out Cros-Validation (LOOCV)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### K-Folds Cross-Validation\n", "In K-Folds Cross-Validation, first, we divide the dataset randomly into k subset/bins. One of the subset/bin is used to validate the model, whereas the rest of the bins are used for training the model. We repeat the process for multiple rounds. Model performances at each round are averaged to define the overall performance of the model.\n", "\n", "" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "from sklearn import model_selection\n", "model = KNeighborsClassifier()\n", "kfold = model_selection.KFold(n_splits=5, random_state=12323, shuffle=True) # note shuffle is true so that samples are randomly assigned to the folds." ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0.93333333, 0.96666667, 0.96666667, 1. , 0.96666667])" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = model_selection.cross_val_score(model, iris_X, iris_y, cv=kfold)\n", "results" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy: 96.667% (2.108%)\n" ] } ], "source": [ "print(\"Accuracy: %.3f%% (%.3f%%)\" % (results.mean()*100.0, results.std()*100.0))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Leave One Out Cross Validation (LOOCV)\n", "\n", "In LOOCV, first, we randomly select one data point for testing and use the remaining data points for building a model. Given the smaller size of the test data, sd% is higher for model accuracy.\n", "\n", "" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy: 96.667% (17.951%)\n" ] } ], "source": [ "model = KNeighborsClassifier()\n", "loocv = model_selection.LeaveOneOut()\n", "results = model_selection.cross_val_score(model, iris_X, iris_y, cv=loocv)\n", "print(\"Accuracy: %.3f%% (%.3f%%)\" % (results.mean()*100.0, results.std()*100.0))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Comparision of model accuracy among various approches:\n", "\n", "|Methods|Accuracy%|Sd%|Notes|\n", "| :---: | :---: | :---: |:---:|\n", "| Train/Test (75/25)|97.36|NA| |\n", "| K-Folds (5)|96.67|2.20||\n", "| LOOCV|96.67|17.95|Higher sd|" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### !!! Now your turn :)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Exercises" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- How does train/test split ratio affects the model performance ? \n", " - Try by changing the percentage of test data to 50% and evaluate the model.\n", " \n", "- Evalate the model performance by incleasing K-folds size. \n", " - Check the model accuracy and sd.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }