{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear Regression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Linear regression is a method that allows us to study the statistical relationship between independent/predictor variables and dependent/response/outcome variables. It is simple to use and interpreting a linear regression model is straightforward. There are two variants:\n",
"\n",
" - Simple linear regression\n",
" - One predictor variable, often denoted $x$\n",
" - One response variable, often denoted $y$\n",
" - Multiple linear regression\n",
" - Two or more predictor variables\n",
" - One dependent variable"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-02T07:41:27.083386Z",
"start_time": "2017-08-02T07:41:26.168157Z"
},
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simple Linear Regression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Simple linear regression assumes that there is a linear relationship between two continuous variables; the predictor variable $x$ and the response variable $y$. This can be described as: \n",
"\n",
"\\begin{equation*}\n",
"y \\thickapprox \\beta_0 + \\beta_1 x\n",
"\\end{equation*}\n",
"\n",
"where\n",
"\n",
" - $\\thickapprox$ can be read as \"*is approximately modeled as*\". \n",
" - $\\beta_0$ is called the *intercept* e.g. if $x=0$ then $y=\\beta_0$\n",
" - $\\beta_1$ is called the *slope* (sometimes referred to as the coefficient). This basically describes the slope of the regression line e.g. if $x$ increases by 1 unit then $y$ increases by $\\beta_1$ units. \n",
" \n",
"Together, $\\beta_0$ and $\\beta_1$ are called the model *coefficients* or *parameters*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Learning the Model Coefficients"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since these coefficients are unknown to begin with, we use our data to estimate or learn them using least squares criteria. Essentially, we find the best fitting line by minimising the sum of squared errors/residuals. The errors or residuals are the differences between the observed value and the estimated value. We square the difference to avoid negative values. The *residual sum of squares (RSS)* is defined as:\n",
"\\begin{equation*}\n",
"RSS = \\sum_{i=1}^n{(y_i - \\hat{y}_i)^2}\n",
"\\end{equation*}\n",
"where\n",
"\n",
" - $y_i$ is the observed value\n",
" - $\\hat{y}_i$ is the estimated/fitted value\n",
" \n",
"We can illustrate this with a plot:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-02T07:41:27.315293Z",
"start_time": "2017-08-02T07:41:27.085190Z"
},
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAD3CAYAAABciF63AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9wVeW97/H33nsRIBAxSPw1WinH5AHEoiCHIggoIAmt\nxVjsbzmt3jM41TvkaK/32qnndno803FaKZxqT532OKenc8/QOfREbcdAAgqCv0BQihCeRMUfWKTB\nBAoESNbe6/7xhL2JJAG2e6+dvfN5zThmr/zY31kT8/FZz1rfbyQIAkRERCQ80VwXICIiMtAofEVE\nREKm8BUREQmZwldERCRkCl8REZGQeWG8SUvLYd1SDZSWFtPW1p7rMgqeznM4dJ7DofMcnkyf67Ky\nkkhvn9PKN0SeF8t1CQOCznM4dJ7DofMcnjDPtcJXREQkZApfERGRkCl8RUREQqbwFRERCZnCV0RE\nJGQKXxERyYja5lXMWjmNS/61lFkrp1HbvCrXJfVbZ/WcrzFmKvCItXa2MeYa4OdAHDgBLLbW7s9i\njSIi0s/VNq9iScOdydeNrTuTr6vLF+WqrH7rjCtfY8wDwK+BIV2HVgD/01o7G/hv4H9nrToREckL\ny7c+2uPxFduWhVxJfjible/bwG3Ab7tef81au++U7z9+ph9QWlqsB8W7lJWV5LqEAUHnORw6z+HI\nh/Pc1La71+P5UP9JYdV6xvC11v7eGDP6lNf7AIwx1wP3AjPP9DPUGs0pKyuhpeVwrssoeDrP4dB5\nDke+nOeK0rE0tu7s8Xg+1A+ZP9d9BXlaN1wZY74K/BL4grW2Jc26RESkQNRMvr/H40sn3RdyJfnh\nnAcrGGO+BSwBZltrWzNfkoiI5JuTN1U9vnIJu0o7qSibwNJJ9+lmq16cU/gaY2LAvwDvA/9tjAHY\nYK39v1moTURE8kh1+SLueuqHALRufSm3xfRzZxW+1tp3gc93vRyZtWpEREQGADXZEBERCZnCV0RE\nJGQKXxERkZApfEVEREKm8BUREQmZwldERCRkCl8REZGQKXxFRERCpvAVEREJmcJXREQkZApfERGR\nkCl8RUREQqbwFRERCZnCV0REJGQKXxERkZApfEVEREKm8BUREQmZwldERCRkCl8REZGQKXxFRERC\npvAVEREJmcJXREQkZN7ZfJExZirwiLV2tjHmSuDfgQB4E7jHWpvIXokiIiKF5YwrX2PMA8CvgSFd\nh5YBP7DW3gBEgIXZK09ERKTwnM3K923gNuC3Xa8nAxu6Pq4DbgZq+/oBpaXFeF4s3RoLSllZSa5L\nGBB0nsOh8xyOvDrP0QiQZzWfIqy6zxi+1trfG2NGn3IoYq0Nuj4+DIw4089oa2tPr7oCU1ZWQkvL\n4VyXUfB0nsOh8xyOfDvPIxMuHlrzqOaTMn2u+wrydG64OnV/twQ4mMbPEBERGbDSCd/XjTGzuz6u\nAjZmrhwREZHCd1Z3O3/C/cCvjDFFQCOwKrMliYiIFLazCl9r7bvA57s+bgJmZbEmERGRgqYmGyIi\nIiFT+IqIiIRM4SsiIhIyha+IiEjIFL4iIiIhU/iKiIiETOErIiISMoWviIgMaLW1HrNmFeN5MGtW\nMbW16fSfOjfZfwcREZF+qrbWY8mSocnXjY2xrtfHqK72s/a+WvmKiMiAFATwyCODe/zcihVFWX1v\nrXxFRGTAiMdhy5YYq1d71NV57NnT8xq0qSm7a1OFr4iIFLRjx2DDBhe49fUeBw64YC0uDigpCTh8\nOHLa91RUJE47lkkKXxERKTgffxyhoSFGXZ3Hhg0e7e0uYMvKEtxxRwdVVT4zZsSpq+u+53vS0qUd\nWa1P4SsiIgVhz54Iq1d7rF7t8eqrMRIJF7jl5XEqK32qqnwmTUoQPeWKsrup6hiP37uHXZ3lVIyP\nsnRpR1ZvtgKFr4iI5KkggO3bo8n928bGGACRSMB11yW6AreTK68M+vw51dU+dz1cRSwaoWX9jjBK\nV/iKiEj+6OiAF1+MJVe4+/a5ZezgwQE33+xTWelz880+F17Yd+DmmsJXRET6tb/+Fdatc2G7dq2X\nvEHq/PMDbr+9k6oqn9mzfYYPz3Gh50DhKyIi/c6f/5zav33xxRidnS5wP/OZBF//ugvcqVPjeHma\nYnlatoiIFJIggN27o9TVucB9441Y8nOf+1zqhqnx4xNETn8yKO8ofEVEJCficdi82T0OVFfn8d57\nbv/W8wJmznRhO3++z2WX9e/923QofEVEJDTt7bB+vdfV8CJGa6sL3GHDAhYu7KSy0mfuXJ8RI3Jc\naJYpfEVEJCNqaz0e+2i1e152VoSaGve87IEDEerr3R3K69d7HD/urhtfdFGCxYs7WLDAZ/r0OIN7\nbrNckNIKX2PMIOA3wGggDvy9tXZ3BusSEcmY2uZVLN/6KE1tu6koHUvN5PupLl+U67IKSmo60DgA\nGhthyZKh/OQncd55J5pseGFMav/2mmu6N7wYSNJd+S4APGvt9caYecA/A1/OXFkiIplR27yKJQ13\nJl83tu5MvlYAZ87y5T1PAXrrrShTp6YCd8yYwtu/TUe64dsEeMaYKHAe0Jm5kkREMmf51kd7PL5i\n2zKF76d04oRreOG6S/W8hPU8+MMfjoVcWf+XbvgewV1y3g2MAr7Y1xeXlhbjebG+vmTAKCsryXUJ\nA4LOczjy4Tw3tfW8I9bUtjsv6of+dZ4PHoRnn4Wnn4a6Ojh82B2Pxdzdy580fnykX9Xfq+jJwQvh\n1Jpu+P4DsMZa+6Ax5nLgOWPM1dba4z19cVtbe9oFFpKyshJaWg7nuoyCp/Mcjnw5zxWlY2ls3dnj\n8Xyovz+c5w8/jCT7J7/0Ugzfd0F1xRUJvvENnwULfD78MMJ3v3v6dKB77jlGS0t2hxRkwshE4Ho7\nZ/Bc9xXk6YZvG6lLza3AIEBLWxHpd2om399tz/ekpZPuy0E1+SEIYNeu1MCCP/0p9ef9mmtS+7dj\nx3ZveBGLHWPFiiKamqJUVCRCmQ6Ur9IN358BTxpjNgJFwPettUczV5aISGac3Nd9fOUSdpV2UlE2\ngaWT7tN+7yf4Prz6aiwZuO+/n2p4MXu2G1hQWelz6aW93zBVXe0rbM9SWuFrrT0CfCXDtYiIZEV1\n+SLueuqHALRufSm3xfQjR4/C88+7hhcNDR5tbW4ZO3x4wK23uv7Jc+b4nHdejgstQGqyISIygLS0\nRKivd6vbF16IJRteXHJJgm9/2wXu9Olxinp+ckgyROErIlLg3n470tU/eRCvvRYlCFzgjhsXT15O\nnjhx4Da8yAWFr4hIgUkkYNu21A1Tzc3uhqloNGDq1DhVVS5wP/tZNbzIFYWviEgBOH4cNm1yDS/W\nrPH4y1/cMnbo0IDKyk4WLPCZOzfOqFEK3P5A4SsikqcOHoS1a93q9rnnPI4edZeTL7jADZyvrPSZ\nNcunuDjHhcppFL4iIv1Qba3H8uVFNDVBRUVxckLQ3r2phhcvv5xqeDF6dILFi90NU1OmxImp80K/\npvAVEelnUhOCnMbGGEuWDOWf/inB3r2pu6KuvTa1f2tM94YX0r8pfEVE+pneJgTt3RvhxhtTDS8u\nuUT7t5lQ27yKx6r3sev8TipWTgtl5KTCV0SkHzhyxDW8ONOEoN/9ThOCMik5cnKkex3WyEmFr4hI\njuzfn2p4sXFjjBMn3HVjzwvwe+jSWFGRCLnCwperkZMKXxGREDU3R6mrcy0dt27t3vCiqsoNLHj7\n7Sh33336hKClSzvCLrfg9TVyMpsUviIiWZRIwNatqcB9661Uw4tp01zgzp/vM3p0av924sQEkcgx\nHr93D7s6y6kYH9WEoCzpa+RkNil8RUQy7Phx2Lgx1fCipcXt4RYXByxY4J6/nTcvzgUX9D0h6K6H\nq9yM2fU7wip9wMnVyEmFr4hIBrS1QUODW90+95xHe7u7nDxqVIJvfrODykqfmTPjDD39arLk0Ml9\n3RXbltHUtpuK0rGhjJxU+IqIpOn9913Di9WrXcOLeNwF7pgxCaqq3Ar3uuvU8KK/qy5fRHX5IsrK\nSmhpORzKeyp8RUTOUhDAm29GefZZF7g7d6ZSdfLkVMOL8nI1vJC+KXxFRPrQ2QkvvxxLrnBPdpgq\nKgqYM8dP3jB10UVqeCFnT+ErIvIJR47Ac8+552/XrvU4dMgtY0eMCPjyl13/5Jtu8hk+PMeFSt5S\n+IqI4BpenFzdbtwYo6PDBe5llyW4/Xa3fzttWpxBg3JcqBQEha+IFLzaWo/HPlrtnpmdFaGmpoNb\nb/Vpbk4NnN+6NbV/e9VVqYYXEyZo/1YyT+ErIgUtNSFoHACNjbBkyVAeeiiRHDgfiwXMmJHav/3M\nZ7R/K9ml8BWRgvazn/U8IailJcIXv+j2b+fO9SktDbkwGdDSDl9jzIPAl4Ai4BfW2n/LWFUiaaht\nXsXyrY8mH5QPYyyY9E+trVBf7/Zvd+/ueUJQLAZPPnk85MpEnLTC1xgzG7gemA4UA9/LYE0i5yw5\nFqxLWGPBpP94771Icv/2lVdiJBJuo7aoKKCjh3kEmhAkuZTuync+sAOoBc4D/lfGKhJJQ67Ggknu\nBAH86U9uYIGbgetumIpEAiZPTnTdMNXJjh2xrj3f7jQhSHIpEgTnfmOBMeZXwBXAF4HPAs8AY621\nPf4w348Hnqf+apI93o884kH89ONRj86HOnNQkWRDRwds2ABPP+3+2bvXHR88GObOhYUL4ZZb4OKL\nu3/fypXw4x/Drl0wfjw8+CB87Wvh13/ORo92/3733VxWIenr9T75dFe+HwO7rbUdgDXGHAfKgL/0\n9MVtbe1pvk1hCbNv6EDT11gwnfPsCOv3+fBhWLfO7d+uXevx17+6v2fnnx9w++2uneONN3ZveNHS\n0v1nzJnj/jnVJ7+mPxqZCNxUI/0OhyLTv9NlZSW9fi7d8N0ELDXGLAMuAYbhAlkkJ3I1FkyyY9++\nVMOLTZtidHa6wL388gRf+5preDF1qhpeSP5KK3yttX80xswENgNR4B5r7enX/ERCcnJf9/GVS9hV\n2klF2YRQxoJJZgQBWJtqePH666ltqquvTg0suOoqNbyQwpD2o0bW2gcyWYjIp1Vdvoi7nvqhu0y3\n5aVclyNnEI/Dli2x5A1T777rHgnyvIAbbkg1vLj8cjW8kMKjJhsiEpr2dtiwwV1Orq+P8fHHLnCH\nDQv40pfc5eS5c33OPz/HhYpkmcJXRLLqwIEIDQ1uhbthg8exY+668YUXJli8uIOqKp8ZM+IMHpzj\nQkVCpPAVkYx7553UDVObN6caXlRUpPZvr702QbTn5lMiBU/hKyLnpLbWY/nyIpqaoKKimJqaDhYu\n9Nm+3TW8cC0dUw0vpkyJU1np9nD/5m+0fysCCl8ROQepCUFOY6PrHvXAAwkOHXLL2CFDAubPd6vb\nefN8LrxQgSvySQpfETlry5b1PCHoyJEIX/2qu2Fq9myfYcNCLkwkzyh8RaRPf/5zJHk52dqeN2kj\nEfj5zzUhSORsKXxFpJsggMbGVMOL7dtTDS+GDAk43kPGakJQ5tU2r+Kx6n3sOr+TipXTNCKzwCh8\nRQTfh82bUw0v3n8/1fBi1iy3f1tZ6bN5syYEhSE5InOke60RmYVH4SsyQB09CuvXu8vJDQ0xWltd\n4A4fHrBwYSdVVT5z5viMGJH6nupqHzjG4/fuYVdnORXjoyxd2tF1XDJFIzILn8JXZABpaYlQX+8C\nd8OGGMePu+dvL744wd/9nWt4MX163w0vqqt97nq4yrXxXL8jpMoHlqa23ed0XPKPwlekwL3zTiR5\nOXnLlhhB4AJ37NjU87cTJ6rhRX/S14hMKQwKX5ECk0jA66+nbphqanI3TEWjAVOnxpP7t2PG6Pnb\n/kojMgufwlekAJw4AZs2uRum1qzx2L8/1fCistLt386bF2fUKAVuPji5r7ti2zKa2nZTUTpWIzIL\njMJXJE8dOgRr17rV7bp1HkePusvJI0e6gfNVVT6zZvkUF+e4UElLdfkiqssXUVZWQkvL4VyXIxmm\n8BXJI3v3RlizxuPZZz1efjmG77vAveKKBHfc4QJ3ypQ4nv7LFunX9J+oSD8WBLBzZ2r/dseOVMOL\na65JTQgaOzZBJJLDQkXknCh8RXIoNSEoSkVFgpqaDm65xeeVV2LJkXwnG14MGhQwe3aq4cWll2r/\nViRfKXxFcqS3CUH33Rck929LSgKqq93AgjlzfM47L1fVikgmKXxFcuSnP+15QtCJE/Cd73RQWeka\nXhT1/GUikscUviIheuutCHV1g6ir82hu7r2rxSOPnAixKhEJm8JXJIsSCdi6NZrcv21uTjW8KC6G\n9vbTv0cTgkQK36cKX2PMhcBWYJ61Vk1He1HbvIrlWx9NPiyv0WCF7fjx7g0v/vIXt8IdOjSgqirV\n8OKFFzQhSGSgSjt8jTGDgCeAY5krp/AkR4N10WiwwnTwIDQ0uNXtunUe7e3uhqkLLkjwjW+4/duZ\nM+PdG15MWAlfXgObHoSW8VC2C2b8GCbMB/S7IVLIPs3K96fAL4EHM1RLQdJosML1wQeR5OXkl16K\nEY+7wP3sZxNUVbk7lKdMiROL9fz9y7c+ClfvhKt/1+34im079bshUuDSCl9jzLeBFmvtGmPMGcO3\ntLQYz+vlL1CB62s0WFlZScjVDABRF4DZOLdBANu3w1NPwdNPwxtvpD73t38Lt94KCxfCuHFRIpEi\noO/blPP6dyOL51lOp/McnrDOdbor3zuBwBgzF7gG+A9jzJestR/19MVtbT3cVTJA9DUaTP1aM29k\nInBzZjN0bjs74ZVXUvu3H3zg9m+LigJuuik1Iejii1MNLw4cOLufnc+/G5k+z9I79XYOT6bPdV9B\nnlb4WmtnnvzYGLMeuLu34B3oNBos/xw5As8/79o5NjR4HDrkVnnnnRdw223uhqmbbvIp+ZT/g6zf\nDZGBS48aZdnJvbvHVy5hV2knFWUTNBqsH9q/3w0sWL3a44UXYnR0uMC99NIEixa5/dvrr48zaFDm\n3lNj40QGrkgQZL8/bEvL4QHfhHbk5AnuMt2WHbkupaCdy3lubo5SV+dWuNu2RQkCF7jjx7uBBVVV\nPldfrYEFPdHvc3h02Tk8Wbjs3OtfD618ZcBIJOC1105OCBrE22+7/dtYLOD6613gzp/vc8UVA/7/\nFUUkyxS+UjBqaz0e+2g1uzrLqZgVoabGPV+7cWNqQtCBAy5wi4sDvvAFt387d67PyJE5Ll5EBhSF\nrxSE1ISgcQA0NsKSJUMpKgqS+7ejRiX41rdcIN9wQ5yhpzeXEhEJhcJXCsJPftL7M7X33nuCykqf\nyZMTvTa8EBEJk8JX8lIQwI4dqRum3nqr51RNJOAf/1G9kkWkf1H4St7o7ISXXkrt3374odu/HTw4\nYPjwgCNHTr+xUBOCRKQ/UvhKv3b4MDz3nFvdrluXanhx/vkBixa5G6ZuvNGnocHThCARyRsKX+l3\n9u+PdD0O5LFpU6rhxWWXJfjKV1zDi89/vnvDi+pqHzjGihVFNDXFqKiIs3RpR9dxEZH+ReErORcE\n0NQUTQbutm2p/dsJE9zzt5WVPhMm9N3worrap7ra73pQfuD2ExeR/k/hKzkRj8OWLbFk4O7Zk2p4\nccMNfnJgweWXq+GFiBQeha+E5tgx2LDBBW59ffeGF7fc4i4nz53rU1qa40JFRLJM4StZ9fHHERoa\n3Ei+DRs82ttPzoFNcMcdHVRV+cyYEWfIkBwXKiISIoWvZNyePZHk40CvvhojkXCBW17u5t9WVflM\nmpQgGs1xoSIiOaLwlU8tCGD79tQNU42N7oapSCTguusSXYHbyZVXav9WRAQUvpKmjg548cVUw4t9\n+1INL26+2d0sdfPNPhdeqMAVEfkkha/0qLbWY/nyIpqaolRUJKip6WDOHJ9161zYrl3rcfhwquHF\n7be7hhezZ/sMH57j4kVE+jmFr5wmNSHIaWyMsWTJUGKxgHjcBe5nPpPg6193gTt1ahxPv0kiImdN\nfzKlmyCARx4Z3OPnBg2C++93E4KuuqrvhhciItI7ha8Qj8PmzbHkhKD33uv5NmTfh+99T72SRUQ+\nLYXvANXeDuvXe10NL2K0trrAHTYsoKQkSO7nnkoTgkREMkPhO4AcOBChvt7dobxhg8exYy5gL7oo\nweLFHSxY4DN9epxnn9WEIBGRbFL4Frh33klNCNqyJdXwwph4sn/ytdd2b3hxckLQ4/fuYVdnORXj\no5oQJCKSQWmFrzFmEPAkMBoYDDxsrX0mg3VJmhIJeOONVMMLa1MNL6ZMcROCqqp8xozp+/nb6mqf\nux6uAqB1/ZtZr1vyR23zKh6r3seu8zupWDmNmsn3U12+KNdlieSVdFe+3wI+ttbeYYwZCbwBKHxz\n5MQJ1/Cirs5jzRqPjz5yy9ghQwLmz3fdpebNi1NWpoYX8unUNq9iScOdMNK9bmzd6V6DAljkHKQb\nvv8FrOr6OALoemTIDh2Cdevc6nbdOo8jR9zl5JEjE3z1q+7521mzfIYNy3GhUlCWb320x+Mrti1T\n+Iqcg0gQpL8aMsaU4Fa8v7LW/mdvX+f78cDzYr19emAYPdr9+9130/4RH3wAzzwDTz0F69e7R38A\nxoyBhQvdP9Onk7mGFxmoWQqL9yOPeBA//XjUo/OhzhxUJNKv9doNIe0/08aYy4Fa4Bd9BS9AW1t7\num9TMEYmAmLRCC0th8/6e4IAdu2KJvsnb9+e+h+YiRPd/m1lpc+4camGF21tma0ZoPUcau4PyspK\nzuk8y9mrKB1LY+vOHo/rnGeHfp/Dk+lzXVZW0uvn0r3h6iKgHrjXWrsuzbqkB77fveHF+++7/VvP\nC5g1y08G7qWXav9Wwlcz+f7kHu+plk66LwfViOSvdFe+3wdKgYeMMQ91Hauy1h7LTFkDy9GjruFF\nXZ1HQ4NHW5tbxg4fHnDrrZ1UVvrMmeMzYkSOC5UB7+S+7opty2hq201F6ViWTrpP+70i5+hT7fme\nrZaWwwN6mVZb6/HYve+6Z2bHRaip6WDGjDj19V5Xw4sYx4+7wL344kTy+dvp0+MM7rnNcihGTp4A\nQOvW/HrUSJfpwqHzHA6d5/Bk4bJz5vd85eykJgSNA6Cxka7XASf34seOTe3fTpzYveGFiIgUHoVv\nFiUS8OMf97x0LS6GBx44TmXlmRteiIhIYVH4ZtiJE7BpU4xnn/Wor/fYv7/nZWxHB3z3u3o0Q0Rk\nIFL4ZsDBg7B2rbth6rnnPI4edZeTL7ggwYgRAYcOaUKQiIikKHzTtHdvamDByy/H8H0XsKNHJ1i8\n2HWYmjIlzjPPaEKQiIh0p/A9S0EAb76ZanixY0eq4cW117oJQVVVPsakGl5AakLQihVFNDXFqKiI\na0KQiMgAp/Dtg+/DK6+4hherV3t88IHbvx00KODGG/3kI0GXXHLmCUHV1X7Xbezq9iUiMtApfD/h\nyBF4/nl3OXntWo+DB90ytqQk4LbbUg0vSnrvGlYQNDZORCR7FL7A/v0R6utd4G7cGOPECRe4l16a\nSAbu9dfHKSrKcaEh0dg4EZHsGrDh29wcTfZP3rYtShC4wB03LjVw/nOf675/O1BobJyISHYNmPBN\nJOC111I3TL31lrthKhoNmDbNBe78+T6jR6vhRVPb7nM6LiIi56agw/f4cXjhhRirV3usWePR0uJu\nmCouDliwwF1OnjcvzgUXKHBP1dfYOBER+fQKLnzb2qChwV1Ofv55j/Z2d9141KgE3/xmB5WVPjNn\nxhl6+qO30kVj40REsivvwre21mP58iKamqJUVCSoqelg8uR4suHFK6/EiMdd4I4Zk6Cqyq1wr7su\nTix2hh8ugMbGiYhkW16NFExNCOrd5MmpCUHl5f3rhimNBguHznM4dJ7DofMcHo0U7MXy5T0/6zN8\neMAPf3iC+fN9LrpI+7ciItK/5VX4NjX1PCHo+HFYvFgTgkREJD/k1dj23iYBaUKQiIjkk7wK35qa\nnicBaUKQiIjkk7wK3+pqnyeeOMbnBu3Co5Px4+M88cQxTQgSEZG8kld7vuAC+K6HqwBoXf9mjqsR\nERE5d3m18hURESkECl8REZGQpXXZ2RgTBX4BTAROAP/DWvtWJgsTEREpVOmufG8FhlhrpwH/B+h5\nBp2IiIicJt0brmYAqwGsta8YY67r64tLS4vxvAw2Vo66jl1lZSWZ+5khycea85HOczh0nsOh8xye\nsM51uuF7HnDolNdxY4xnre3xmZ+2tvY036ZnIxOuhWRrnvU7VY/WcOg8h0PnORw6z+HJQm/nXj+X\n7mXnvwKn/tRob8ErIiIi3aUbvi8CCwCMMZ8HdmSsIhERkQKX7mXnWmCeMeYlIAJ8J3MliYiIFLa0\nwtdamwDuznAtIiIiA4KabIiIiIRM4SsiIhIyha+IiEjIFL4iIiIhU/iKiIiETOErIiISMoWviIhI\nyBS+IiIiIcu78K1tXsW11fsY/J33mbVyGrXNq3JdkoiIyDlJt71kTtQ2r2JJw50w0r1ubN3pXgPV\n5YtyWJmIiMjZy6uV7/Ktj/Z4fMW2ZSFXIiIikr68Ct+mtt3ndFxERKQ/yqvwrSgde07HRURE+qO8\nCt+ayff3eHzppPtCrkRERCR9eXXD1cmbqlZsW0ZT224qSseydNJ9utlKRETySl6FL7gAVtiKiEg+\ny6vLziIiIoVA4SsiIhIyha+IiEjIFL4iIiIhU/iKiIiELBIEQa5rEBERGVC08hUREQmZwldERCRk\nCl8REZGQKXxFRERCpvAVEREJmcJXREQkZApfERGRkOXdVKN8Y4wZBDwJjAYGAw9ba5/JaVEFzBhz\nIbAVmGet3Z3regqVMeZB4EtAEfALa+2/5bikgtP1t+M3uL8dceDv9TudWcaYqcAj1trZxpgrgX8H\nAuBN4B5rbSJb762Vb/Z9C/jYWnsDUAk8luN6ClbXH6sngGO5rqWQGWNmA9cD04FZwOU5LahwLQA8\na+31wI+Af85xPQXFGPMA8GtgSNehZcAPuv5WR4CF2Xx/hW/2/RfwUNfHEcDPYS2F7qfAL4E/57qQ\nAjcf2AH5bYMfAAABk0lEQVTUAn8A/pjbcgpWE+AZY6LAeUBnjuspNG8Dt53yejKwoevjOmBuNt9c\n4Ztl1toj1trDxpgSYBXwg1zXVIiMMd8GWqy1a3JdywAwCrgOuB24G/h/xphIbksqSEdwl5x3A78C\n/iWn1RQYa+3v6f4/NBFr7cl+y4eBEdl8f4VvCIwxlwPPA7+11v5nruspUHcC84wx64FrgP8wxlyc\n25IK1sfAGmtth7XWAseBshzXVIj+AXeeK4CJwG+MMUPO8D2SvlP3d0uAg9l8M91wlWXGmIuAeuBe\na+26XNdTqKy1M09+3BXAd1trP8pdRQVtE7DUGLMMuAQYhgtkyaw2UiuzVmAQEMtdOQXvdWPMbGvt\neqAKt2DKGoVv9n0fKAUeMsac3PutstbqpiDJS9baPxpjZgKbcVfP7rHWxnNcViH6GfCkMWYj7q7y\n71trj+a4pkJ2P/ArY0wR0IjbJswajRQUEREJmfZ8RUREQqbwFRERCZnCV0REJGQKXxERkZApfEVE\nREKm8BUREQmZwldERCRk/x/rGRG0SfofqAAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from sklearn.linear_model import LinearRegression\n",
"\n",
"# Create artificial data\n",
"X = np.array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])\n",
"Y = np.array([-1.1, 4, 1, 6, 4, 2, 8, 5, 12, 7])\n",
"\n",
"# Create a model\n",
"lrm = LinearRegression()\n",
"lrm.fit(X.reshape(-1, 1), Y)\n",
"model_line = lrm.intercept_ + X * lrm.coef_[0]\n",
"\n",
"fig, ax = plt.subplots(figsize=(8,4))\n",
"\n",
"# Plot the data\n",
"ax.plot(X, Y, 'go') \n",
"\n",
"# Plot the regression line\n",
"ax.plot(X, model_line, 'bo-') # \n",
"\n",
"# Plot the residuals\n",
"ax.vlines(x=X, ymin=np.minimum(Y, model_line), ymax=np.maximum(Y, model_line), color='red')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The green dots represent our artificial data i.e. the observed data. The red vertical lines indicate the errors or the residuals. The blue line represent the regression line that best fit the observed data. This line is found by minimising the residuals. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Multiple Linear Regression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Typically, we have more than one predictor variable (features) in our data. In such instances, we use Multiple Linear Regression:\n",
"\n",
"\\begin{equation*}\n",
"y \\thickapprox \\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 + \\cdots + \\beta_n x_n\n",
"\\end{equation*}\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Explore the Boston House Prices Dataset"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-02T07:41:27.364304Z",
"start_time": "2017-08-02T07:41:27.317292Z"
},
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.datasets import load_boston\n",
"boston = load_boston()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-02T07:41:27.370305Z",
"start_time": "2017-08-02T07:41:27.366305Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['data', 'target', 'feature_names', 'DESCR'])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"boston.keys()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-02T07:41:27.383310Z",
"start_time": "2017-08-02T07:41:27.371305Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Boston House Prices dataset\n",
"===========================\n",
"\n",
"Notes\n",
"------\n",
"Data Set Characteristics: \n",
"\n",
" :Number of Instances: 506 \n",
"\n",
" :Number of Attributes: 13 numeric/categorical predictive\n",
" \n",
" :Median Value (attribute 14) is usually the target\n",
"\n",
" :Attribute Information (in order):\n",
" - CRIM per capita crime rate by town\n",
" - ZN proportion of residential land zoned for lots over 25,000 sq.ft.\n",
" - INDUS proportion of non-retail business acres per town\n",
" - CHAS Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
" - NOX nitric oxides concentration (parts per 10 million)\n",
" - RM average number of rooms per dwelling\n",
" - AGE proportion of owner-occupied units built prior to 1940\n",
" - DIS weighted distances to five Boston employment centres\n",
" - RAD index of accessibility to radial highways\n",
" - TAX full-value property-tax rate per $10,000\n",
" - PTRATIO pupil-teacher ratio by town\n",
" - B 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
" - LSTAT % lower status of the population\n",
" - MEDV Median value of owner-occupied homes in $1000's\n",
"\n",
" :Missing Attribute Values: None\n",
"\n",
" :Creator: Harrison, D. and Rubinfeld, D.L.\n",
"\n",
"This is a copy of UCI ML housing dataset.\n",
"http://archive.ics.uci.edu/ml/datasets/Housing\n",
"\n",
"\n",
"This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n",
"\n",
"The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n",
"prices and the demand for clean air', J. Environ. Economics & Management,\n",
"vol.5, 81-102, 1978. Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n",
"...', Wiley, 1980. N.B. Various transformations are used in the table on\n",
"pages 244-261 of the latter.\n",
"\n",
"The Boston house-price data has been used in many machine learning papers that address regression\n",
"problems. \n",
" \n",
"**References**\n",
"\n",
" - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n",
" - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n",
" - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)\n",
"\n"
]
}
],
"source": [
"print(boston['DESCR'])"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-02T07:41:27.410315Z",
"start_time": "2017-08-02T07:41:27.384308Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" CRIM | \n",
" ZN | \n",
" INDUS | \n",
" CHAS | \n",
" NOX | \n",
" RM | \n",
" AGE | \n",
" DIS | \n",
" RAD | \n",
" TAX | \n",
" PTRATIO | \n",
" B | \n",
" LSTAT | \n",
" MEDV | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0.00632 | \n",
" 18.0 | \n",
" 2.31 | \n",
" 0.0 | \n",
" 0.538 | \n",
" 6.575 | \n",
" 65.2 | \n",
" 4.0900 | \n",
" 1.0 | \n",
" 296.0 | \n",
" 15.3 | \n",
" 396.90 | \n",
" 4.98 | \n",
" 24.0 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0.02731 | \n",
" 0.0 | \n",
" 7.07 | \n",
" 0.0 | \n",
" 0.469 | \n",
" 6.421 | \n",
" 78.9 | \n",
" 4.9671 | \n",
" 2.0 | \n",
" 242.0 | \n",
" 17.8 | \n",
" 396.90 | \n",
" 9.14 | \n",
" 21.6 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0.02729 | \n",
" 0.0 | \n",
" 7.07 | \n",
" 0.0 | \n",
" 0.469 | \n",
" 7.185 | \n",
" 61.1 | \n",
" 4.9671 | \n",
" 2.0 | \n",
" 242.0 | \n",
" 17.8 | \n",
" 392.83 | \n",
" 4.03 | \n",
" 34.7 | \n",
"
\n",
" \n",
" | 3 | \n",
" 0.03237 | \n",
" 0.0 | \n",
" 2.18 | \n",
" 0.0 | \n",
" 0.458 | \n",
" 6.998 | \n",
" 45.8 | \n",
" 6.0622 | \n",
" 3.0 | \n",
" 222.0 | \n",
" 18.7 | \n",
" 394.63 | \n",
" 2.94 | \n",
" 33.4 | \n",
"
\n",
" \n",
" | 4 | \n",
" 0.06905 | \n",
" 0.0 | \n",
" 2.18 | \n",
" 0.0 | \n",
" 0.458 | \n",
" 7.147 | \n",
" 54.2 | \n",
" 6.0622 | \n",
" 3.0 | \n",
" 222.0 | \n",
" 18.7 | \n",
" 396.90 | \n",
" 5.33 | \n",
" 36.2 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" count | \n",
" mean | \n",
" std | \n",
" min | \n",
" 25% | \n",
" 50% | \n",
" 75% | \n",
" max | \n",
"
\n",
" \n",
" \n",
" \n",
" | CRIM | \n",
" 506.0 | \n",
" 3.593761 | \n",
" 8.596783 | \n",
" 0.00632 | \n",
" 0.082045 | \n",
" 0.25651 | \n",
" 3.647423 | \n",
" 88.9762 | \n",
"
\n",
" \n",
" | ZN | \n",
" 506.0 | \n",
" 11.363636 | \n",
" 23.322453 | \n",
" 0.00000 | \n",
" 0.000000 | \n",
" 0.00000 | \n",
" 12.500000 | \n",
" 100.0000 | \n",
"
\n",
" \n",
" | INDUS | \n",
" 506.0 | \n",
" 11.136779 | \n",
" 6.860353 | \n",
" 0.46000 | \n",
" 5.190000 | \n",
" 9.69000 | \n",
" 18.100000 | \n",
" 27.7400 | \n",
"
\n",
" \n",
" | CHAS | \n",
" 506.0 | \n",
" 0.069170 | \n",
" 0.253994 | \n",
" 0.00000 | \n",
" 0.000000 | \n",
" 0.00000 | \n",
" 0.000000 | \n",
" 1.0000 | \n",
"
\n",
" \n",
" | NOX | \n",
" 506.0 | \n",
" 0.554695 | \n",
" 0.115878 | \n",
" 0.38500 | \n",
" 0.449000 | \n",
" 0.53800 | \n",
" 0.624000 | \n",
" 0.8710 | \n",
"
\n",
" \n",
" | RM | \n",
" 506.0 | \n",
" 6.284634 | \n",
" 0.702617 | \n",
" 3.56100 | \n",
" 5.885500 | \n",
" 6.20850 | \n",
" 6.623500 | \n",
" 8.7800 | \n",
"
\n",
" \n",
" | AGE | \n",
" 506.0 | \n",
" 68.574901 | \n",
" 28.148861 | \n",
" 2.90000 | \n",
" 45.025000 | \n",
" 77.50000 | \n",
" 94.075000 | \n",
" 100.0000 | \n",
"
\n",
" \n",
" | DIS | \n",
" 506.0 | \n",
" 3.795043 | \n",
" 2.105710 | \n",
" 1.12960 | \n",
" 2.100175 | \n",
" 3.20745 | \n",
" 5.188425 | \n",
" 12.1265 | \n",
"
\n",
" \n",
" | RAD | \n",
" 506.0 | \n",
" 9.549407 | \n",
" 8.707259 | \n",
" 1.00000 | \n",
" 4.000000 | \n",
" 5.00000 | \n",
" 24.000000 | \n",
" 24.0000 | \n",
"
\n",
" \n",
" | TAX | \n",
" 506.0 | \n",
" 408.237154 | \n",
" 168.537116 | \n",
" 187.00000 | \n",
" 279.000000 | \n",
" 330.00000 | \n",
" 666.000000 | \n",
" 711.0000 | \n",
"
\n",
" \n",
" | PTRATIO | \n",
" 506.0 | \n",
" 18.455534 | \n",
" 2.164946 | \n",
" 12.60000 | \n",
" 17.400000 | \n",
" 19.05000 | \n",
" 20.200000 | \n",
" 22.0000 | \n",
"
\n",
" \n",
" | B | \n",
" 506.0 | \n",
" 356.674032 | \n",
" 91.294864 | \n",
" 0.32000 | \n",
" 375.377500 | \n",
" 391.44000 | \n",
" 396.225000 | \n",
" 396.9000 | \n",
"
\n",
" \n",
" | LSTAT | \n",
" 506.0 | \n",
" 12.653063 | \n",
" 7.141062 | \n",
" 1.73000 | \n",
" 6.950000 | \n",
" 11.36000 | \n",
" 16.955000 | \n",
" 37.9700 | \n",
"
\n",
" \n",
" | MEDV | \n",
" 506.0 | \n",
" 22.532806 | \n",
" 9.197104 | \n",
" 5.00000 | \n",
" 17.025000 | \n",
" 21.20000 | \n",
" 25.000000 | \n",
" 50.0000 | \n",
"
\n",
" \n",
"
\n",
"