{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Resampling Methods – Conceptual\n",
    "\n",
    "Excercises from **Chapter 5* of [An Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/) by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Using basic statistical properties of the variance, as well as single- variable calculus, derive (5.6). In other words, prove that α given by (5.6) does indeed minimize Var(αX + (1 − α)Y ).\n",
    "\n",
    "![5_1.jpg](./images/5_1.jpg)\n",
    "\n",
    "\n",
    "## 2. We will now derive the probability that a given observation is part of a bootstrap sample. Suppose that we obtain a bootstrap sample from a set of n observations.\n",
    "\n",
    "- (a) What is the probability that the first bootstrap observation is not the jth observation from the original sample? Justify your answer.\n",
    "- (b) What is the probability that the second bootstrap observation is not the jth observation from the original sample?\n",
    "- (c) Argue that the probability that the jth observation is not in the bootstrap sample is (1 − 1/n)n.\n",
    "- (d) When n = 5, what is the probability that the jth observation is in the bootstrap sample?\n",
    "- (e) When n = 100, what is the probability that the jth observation is in the bootstrap sample?\n",
    "- (f) When n = 10, 000, what is the probability that the jth observa- tion is in the bootstrap sample?\n",
    "\n",
    "![5_2.jpg](./images/5_2.jpg)\n",
    "\n",
    "### 2(g) Create a plot that displays, for each integer value of n from 1 to 100,000, the probability that the jth observation is in the bootstrap sample. Comment on what you observe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'probability')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGMhJREFUeJzt3X+QJ3V95/Hni8UFTUQXWQ1hF3fJrR4bNYJTiGL8ecDKpcREy9vNJaLxjsopemq8HJQWmM1ZehdjLAylrLmNaEUIh7lky1qLcPwwGn/tcPyS1YVhjTAuCeOBQEUUl33fH9+e0Awz8+1dtmdmd56Pqm99uz/d/f2+e3v4vujuT3enqpAkaZhD5rsASdKBwcCQJHViYEiSOjEwJEmdGBiSpE4MDElSJwaGJKkTA0OS1ImBIUnq5ND5LmB/Oeqoo2rVqlXzXYYkHVCuv/76H1bV8i7zHjSBsWrVKkZHR+e7DEk6oCT5ftd5PSQlSerEwJAkdWJgSJI6MTAkSZ0YGJKkTnoLjCSbk9yT5NszTE+SC5OMJbk5yYmtaWclub15ndVXjZKk7vrcw/gMsG6W6a8F1jSvs4FPAiQ5ErgAeDFwEnBBkmU91ilJ6qC3wKiqvwPunWWWM4HP1sA3gKcnORo4Hbiqqu6tqvuAq5g9eJ6QHz+8m4/97Q5uuPO+vr5Ckg4K83kO4xjgrtb4eNM2U/vjJDk7yWiS0YmJiX0q4qGHH+HCa8a45Qf379PykrRYzGdgZJq2mqX98Y1Vm6pqpKpGli/vdGW7JGkfzWdgjAMrW+MrgF2ztEuS5tF8BsYW4M1Nb6mTgfur6m7gSuC0JMuak92nNW2SpHnU280Hk1wKvBI4Ksk4g55PTwKoqk8BW4EzgDHgx8Bbm2n3JvlDYFvzURuraraT55KkOdBbYFTVhiHTC3jHDNM2A5v7qEuStG+80rtR055WlyRNWvSBkUzXKUuSNNWiDwxJUjcGhiSpEwNDktSJgSFJ6sTAaJTdpCRpVos+MOwjJUndLPrAkCR1Y2BIkjoxMCRJnRgYkqRODAxJUicGRsNOtZI0u0UfGN57UJK6WfSBIUnqxsCQJHXSa2AkWZdkR5KxJOdOM/3ZSa5OcnOS65KsaE17JMmNzWtLn3VKkobr85neS4CLgFOBcWBbki1Vtb0120eBz1bVJUleDXwY+O1m2kNV9cK+6pMk7Z0+9zBOAsaqamdVPQxcBpw5ZZ61wNXN8LXTTJckLRB9BsYxwF2t8fGmre0m4A3N8K8DT03yjGb88CSjSb6R5PU91gn4TG9JGqbPwJiuw+rUn+X3Aa9IcgPwCuAHwO5m2rFVNQL8JvDxJL/0uC9Izm5CZXRiYmIfi7RfrSR10WdgjAMrW+MrgF3tGapqV1X9RlWdALy/abt/clrzvhO4Djhh6hdU1aaqGqmqkeXLl/eyEpKkgT4DYxuwJsnqJEuB9cBjejslOSrJZA3nAZub9mVJDpucBzgFaJ8slyTNsd4Co6p2A+cAVwLfAS6vqluTbEzyuma2VwI7ktwGPAv4UNN+PDCa5CYGJ8M/MqV3lSRpjvXWrRagqrYCW6e0nd8avgK4YprlvgY8v8/aJEl7xyu9JUmdGBgNe9VK0uwMDHvVSlInBoYkqRMDQ5LUiYEhSerEwJAkdWJgNMq7D0rSrAwMSVIniz4wYrdaSepk0QeGJKkbA0OS1ImBIUnqxMCQJHViYEiSOln0gWEnKUnqZtEHhiSpGwNDktRJr4GRZF2SHUnGkpw7zfRnJ7k6yc1JrkuyojXtrCS3N6+z+qxTkjRcb4GRZAlwEfBaYC2wIcnaKbN9FPhsVb0A2Ah8uFn2SOAC4MXAScAFSZb1Vaskabg+9zBOAsaqamdVPQxcBpw5ZZ61wNXN8LWt6acDV1XVvVV1H3AVsK7HWiVJQ/QZGMcAd7XGx5u2tpuANzTDvw48NckzOi67X3mzWkmaXZ+BMV2P1ak/y+8DXpHkBuAVwA+A3R2XJcnZSUaTjE5MTOxbkd59UJI66TMwxoGVrfEVwK72DFW1q6p+o6pOAN7ftN3fZdlm3k1VNVJVI8uXL9/f9UuSWvoMjG3AmiSrkywF1gNb2jMkOSrJZA3nAZub4SuB05Isa052n9a0SZLmSW+BUVW7gXMY/NB/B7i8qm5NsjHJ65rZXgnsSHIb8CzgQ82y9wJ/yCB0tgEbmzZJ0jw5tM8Pr6qtwNYpbee3hq8Arphh2c08uschSZpnXuktSerEwGjU4zthSZJaFn1g2KlWkrpZ9IEhSerGwJAkdWJgSJI6MTAkSZ0YGA1vPihJszMwJEmdLPrA8Ga1ktTNog8MSVI3BoYkqRMDQ5LUiYEhSerEwGjYq1aSZtcpMJIs6buQ+RJvPyhJnXTdwxhL8kdJ1vZajSRpweoaGC8AbgP+LMk3kpyd5Ige65IkLTCdAqOqHqyqT1fVS4HfBy4A7k5ySZJ/NdNySdYl2ZFkLMm500w/Nsm1SW5IcnOSM5r2VUkeSnJj8/rUPq6fJGk/6fRM7+Ycxr8F3gqsAv4Y+AvgVxk8s/s5MyxzEXAqMA5sS7Klqra3ZvsAcHlVfbI53LW1+XyAO6rqhfuwTpKkHnQKDOB24Frgj6rqa632K5K8fIZlTgLGqmonQJLLgDOBdmAUMHlo62nArq6FS5LmVtdzGG+uqre1wyLJKQBV9a4ZljkGuKs1Pt60tX0Q+K0k4wz2Lt7Zmra6OVT15SS/2rHOfebdaiVpdl0D48Jp2j4xZJnp+qtO/VneAHymqlYAZwCfS3IIcDdwbFWdALwX+Px0J9mbk++jSUYnJiaGrsS0RdqrVpI6mfWQVJKXAC8Flid5b2vSEcCwazPGgZWt8RU8/pDT24B1AFX19SSHA0dV1T3AT5v265PcweA8yWh74araBGwCGBkZcR9Bkno0bA9jKfDzDILlqa3XA8Abhyy7DViTZHWSpcB6YMuUee4EXgOQ5HjgcGAiyfLJiwWTHAesAXZ2XSlJ0v436x5GVX0Z+HKSz1TV9/fmg6tqd5JzgCsZ7I1srqpbk2wERqtqC/B7wKeTvIfB4aq3VFU1J9I3JtkNPAL8blXdu/erJ0naX4Ydkvp4Vb0b+NMkjzvkU1Wvm235qtrK4GR2u+381vB24JRplvsC8IXZS5ckzaVh3Wo/17x/tO9CJEkL27BDUtc371+em3LmT3m/Wkma1bBDUrcwy52/q+oF+70iSdKCNOyQ1K/NSRWSpAVv2CGpveoZJUk6eM16HUaSrzbvDyZ5YOr73JQoSVoIhu1hvKx5f+rclCNJWqi63q2WJCcCL2NwEvyrVXVDb1XNA28+KEmz6/pM7/OBS4BnAEcBn0nygT4LkyQtLF33MDYAJ1TVTwCSfAT4v8B/66uwueLdaiWpm663N/8HBjcGnHQYcMd+r0aStGANu3DvEwzOWfwUuDXJVc34qcBX+y9PkrRQDDskNfn8ieuB/91qv66XaiRJC9awbrWXzFUhkqSFrdNJ7yRrgA8Da2mdy6iq43qqS5K0wHQ96f3nwCeB3cCrgM/y6K3PD2iZ9tHjkqSpugbGk6vqaiBV9f2q+iDw6v7KkiQtNF2vw/hJkkOA25vHrv4AeGZ/ZUmSFpquexjvBp4CvAt4EfDbwFnDFkqyLsmOJGNJzp1m+rFJrk1yQ5Kbk5zRmnZes9yOJKd3rFOS1JNOexhVtQ2g2ct4V1U9OGyZJEuAixhcszEObEuypXmO96QPAJdX1SeTrGXw/O9VzfB64JeBXwT+T5LnVNUje7FukqT9qOu9pEaap+/dDNyS5KYkLxqy2EnAWFXtrKqHgcuAM6fMU8ARzfDTgF3N8JnAZVX106r6HjDWfJ4kaZ50PSS1GXh7Va2qqlXAOxj0nJrNMcBdrfHxpq3tg8BvJRlnsHfxzr1Ydr8qb1crSbPqGhgPVtVXJkeq6qvAsMNS0/VXnfqrvAH4TFWtAM4APtcc9uqyLEnOTjKaZHRiYmJIOTMUaa9aSepk2L2kTmwGv5XkYuBSBj/c/47htwcZB1a2xlfw6CGnSW8D1gFU1deTHM7g9uldlqWqNgGbAEZGRtxFkKQeDTvp/cdTxi9oDQ/7gd4GrEmymkE33PXAb06Z507gNQyer3E8g6vIJ4AtwOeTfIzBSe81wLeGfJ8kqUfD7iX1qn394Kra3VyzcSWwBNhcVbcm2QiMVtUW4PeATyd5D4MAeksNTibcmuRyYDuDq8vfYQ8pSZpfXe8l9TQGexcvb5q+DGysqvtnW66qtjI4md1uO781vB04ZYZlPwR8qEt9kqT+7U0vqQeBNzWvBxjeS0qSdBDpemuQX6qqN7TG/yDJjX0UNF/sVStJs+u6h/FQkpdNjiQ5BXion5Lmlr1qJambrnsYvwt8tjmXAXAfHe4lJUk6eAwNjOZCuudW1a8kOQKgqh7ovTJJ0oIy9JBUVe0BzmmGHzAsJGlx6noO46ok70uyMsmRk69eK5MkLShdz2H8DoML694+pf2geaa3naQkaXZdA2Mtg7B4GYPf1q8An+qrKEnSwtM1MC5hcLHehc34hqbtTX0UNZfi7WolqZOugfHcqvqV1vi1SW7qoyBJ0sLU9aT3DUlOnhxJ8mLg7/spSZK0EHXdw3gx8OYkdzbjxwLfaR7bWlX1gl6qkyQtGF0DY12vVUiSFrxOgVFV3++7kPnmzQclaXZdz2EctOwjJUndLPrAkCR1Y2BIkjrpNTCSrEuyI8lYknOnmf4nSW5sXrcl+VFr2iOtaVv6rFOSNFzXXlJ7LckS4CLgVGAc2JZkS/McbwCq6j2t+d8JnND6iIeq6oV91SdJ2jt97mGcBIxV1c6qehi4DDhzlvk3AJf2WI8k6QnoMzCOAe5qjY83bY+T5NnAauCaVvPhSUaTfCPJ6/src6C8X60kzaq3Q1JM32N1pl/l9cAVVfVIq+3YqtqV5DjgmiS3VNUdj/mC5GzgbIBjjz1234q0X60kddLnHsY4sLI1vgLYNcO865lyOKqqdjXvO4HreOz5jcl5NlXVSFWNLF++fH/ULEmaQZ+BsQ1Yk2R1kqUMQuFxvZ2SPBdYBny91bYsyWHN8FHAKcD2qctKkuZOb4ekqmp3knOAK4ElwOaqujXJRmC0qibDYwNwWdVjbs5xPHBxkj0MQu0j7d5VkqS51+c5DKpqK7B1Stv5U8Y/OM1yXwOe32dtkqS945XekqRODIyGd6uVpNkt+sDwmd6S1M2iDwxJUjcGhiSpEwNDktSJgSFJ6sTAaNhJSpJmZ2BIkjoxMCRJnRgYkqRODAxJUicGhiSpEwNDktSJgTHJuw9K0qwMDHyutyR1YWBIkjoxMCRJnfQaGEnWJdmRZCzJudNM/5MkNzav25L8qDXtrCS3N6+z+qxTkjRcb8/0TrIEuAg4FRgHtiXZUlXbJ+epqve05n8ncEIzfCRwATDC4DZP1zfL3tdXvZKk2fW5h3ESMFZVO6vqYeAy4MxZ5t8AXNoMnw5cVVX3NiFxFbCux1olSUP0GRjHAHe1xsebtsdJ8mxgNXDN3iyb5Owko0lGJyYmnlCxdqqVpNn1GRjTdVad6Xd5PXBFVT2yN8tW1aaqGqmqkeXLl+9jmdN/mSTpsfoMjHFgZWt8BbBrhnnX8+jhqL1dVpI0B/oMjG3AmiSrkyxlEApbps6U5LnAMuDrreYrgdOSLEuyDDitaZMkzZPeeklV1e4k5zD4oV8CbK6qW5NsBEarajI8NgCXVT16b46qujfJHzIIHYCNVXVvX7VKkobrLTAAqmorsHVK2/lTxj84w7Kbgc29FSdJ2ite6S1J6sTAaHizWkmanYEBxNvVStJQBoYkqRMDQ5LUiYEhSerEwJAkdWJgNMrbD0rSrAwMSVInBgberVaSujAwJEmdGBiSpE4MDElSJwaGJKkTA6PhzQclaXYGBuC9ByVpOANDktSJgSFJ6qTXwEiyLsmOJGNJzp1hnjcl2Z7k1iSfb7U/kuTG5rVlumUlSXOnt2d6J1kCXAScCowD25JsqartrXnWAOcBp1TVfUme2fqIh6rqhX3VJ0naO33uYZwEjFXVzqp6GLgMOHPKPP8RuKiq7gOoqnt6rEeS9AT0GRjHAHe1xsebtrbnAM9J8vdJvpFkXWva4UlGm/bXT/cFSc5u5hmdmJh4QsXaq1aSZtfbISmmv6ff1N/lQ4E1wCuBFcBXkjyvqn4EHFtVu5IcB1yT5JaquuMxH1a1CdgEMDIyss+/+fH2g5I0VJ97GOPAytb4CmDXNPP8TVX9rKq+B+xgECBU1a7mfSdwHXBCj7VKkoboMzC2AWuSrE6yFFgPTO3t9NfAqwCSHMXgENXOJMuSHNZqPwXYjiRp3vR2SKqqdic5B7gSWAJsrqpbk2wERqtqSzPttCTbgUeA/1JV/y/JS4GLk+xhEGofafeukiTNvT7PYVBVW4GtU9rObw0X8N7m1Z7na8Dz+6xNkrR3vNJbktSJgQEsPfQQfvKzR+a7DEla0AwM4FlHHMZt//QgD+/eM9+lSNKC1es5jAPFa45/Fpv+bifP+cCXOPxJh/CUpYdySEIChwQOSVrj6XQ79K5XdqTDh3X6rP1UU5d6JC0sxx99BJ/Y0P+VBwYG8F/X/WtOPHYZO/7xQf754d38+OHd7CmoKvbsgaLYU7Cnij17hl8f2PUKwi4PberyWdXhgzrV5OXu0gFp5bInz8n3GBjAkkPCuuf9Auue9wvzXYokLView5AkdWJgSJI6MTAkSZ0YGJKkTgwMSVInBoYkqRMDQ5LUiYEhSeokXa4SPhAkmQC+/wQ+4ijgh/upnAPFYlvnxba+4DovFk9knZ9dVcu7zHjQBMYTlWS0qkbmu465tNjWebGtL7jOi8VcrbOHpCRJnRgYkqRODIxHbZrvAubBYlvnxba+4DovFnOyzp7DkCR14h6GJKmTRR8YSdYl2ZFkLMm5813P3kqyMsm1Sb6T5NYk/7lpPzLJVUlub96XNe1JcmGzvjcnObH1WWc189+e5KxW+4uS3NIsc2EWwGP5kixJckOSLzbjq5N8s6n9L5MsbdoPa8bHmumrWp9xXtO+I8nprfYF9zeR5OlJrkjy3WZbv2QRbOP3NH/T305yaZLDD7btnGRzknuSfLvV1vt2nek7hqqqRfsClgB3AMcBS4GbgLXzXddersPRwInN8FOB24C1wP8Azm3azwX+ezN8BvAlBk9sPRn4ZtN+JLCzeV/WDC9rpn0LeEmzzJeA1y6A9X4v8Hngi8345cD6ZvhTwH9qht8OfKoZXg/8ZTO8ttnehwGrm7+DJQv1bwK4BPgPzfBS4OkH8zYGjgG+Bzy5tX3fcrBtZ+DlwInAt1ttvW/Xmb5jaL3z/R/CPP9RvgS4sjV+HnDefNf1BNfpb4BTgR3A0U3b0cCOZvhiYENr/h3N9A3Axa32i5u2o4HvttofM988reMK4Grg1cAXm/8YfggcOnW7AlcCL2mGD23my9RtPTnfQvybAI5ofjwzpf1g3sbHAHc1P4KHNtv59INxOwOreGxg9L5dZ/qOYa/Ffkhq8o9y0njTdkBqdsNPAL4JPKuq7gZo3p/ZzDbTOs/WPj5N+3z6OPD7wJ5m/BnAj6pqdzPervFf1quZfn8z/97+O8yn44AJ4M+bw3B/luTnOIi3cVX9APgocCdwN4Ptdj0H93aeNBfbdabvmNViD4zpjtMekN3Gkvw88AXg3VX1wGyzTtNW+9A+L5L8GnBPVV3fbp5m1hoy7YBY38ahDA5bfLKqTgD+mcFhhJkc8OvcHFM/k8FhpF8Efg547TSzHkzbeZh5X8fFHhjjwMrW+Apg1zzVss+SPIlBWPxFVf1V0/xPSY5uph8N3NO0z7TOs7WvmKZ9vpwCvC7JPwCXMTgs9XHg6UkObeZp1/gv69VMfxpwL3v/7zCfxoHxqvpmM34FgwA5WLcxwL8BvldVE1X1M+CvgJdycG/nSXOxXWf6jlkt9sDYBqxpel4sZXCybMs817RXml4P/xP4TlV9rDVpCzDZW+IsBuc2Jtvf3PS4OBm4v9klvRI4Lcmy5v/uTmNwjPdu4MEkJzff9ebWZ825qjqvqlZU1SoG2+uaqvr3wLXAG5vZpq7v5L/DG5v5q2lf3/SuWQ2sYXCCcMH9TVTVPwJ3JXlu0/QaYDsH6TZu3AmcnOQpTU2T63zQbueWudiuM33H7ObzxNZCeDHoeXAbgx4T75/vevah/pcx2M28GbixeZ3B4Pjt1cDtzfuRzfwBLmrW9xZgpPVZvwOMNa+3ttpHgG83y/wpU06+zuO6v5JHe0kdx+CHYAz4X8BhTfvhzfhYM/241vLvb9ZpB61eQQvxbwJ4ITDabOe/ZtAb5qDexsAfAN9t6vocg55OB9V2Bi5lcI7mZwz2CN42F9t1pu8Y9vJKb0lSJ4v9kJQkqSMDQ5LUiYEhSerEwJAkdWJgSJI6MTAkSZ0YGJKkTgwMqUdJVmXw/IpPN892+NskT57vuqR9YWBI/VsDXFRVvwz8CHjDPNcj7RMDQ+rf96rqxmb4egbPP5AOOAaG1L+ftoYfYXC7cumAY2BIkjoxMCRJnXi3WklSJ+5hSJI6MTAkSZ0YGJKkTgwMSVInBoYkqRMDQ5LUiYEhSerEwJAkdfL/Ab5Bi83pB09DAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def prob_j_in_sample(n):\n",
    "    return 1 - (1 - 1/n)**n\n",
    "\n",
    "x = np.arange(1, 100000)\n",
    "y = np.array([prob_j_in_sample(n) for n in x])\n",
    "\n",
    "ax = sns.lineplot(x=x, y=prob_j_in_sample(x))\n",
    "plt.xlabel('n')\n",
    "plt.ylabel('probability')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2(h) We will now investigate numerically the probability that a bootstrap sample of size n = 100 contains the jth observation. Here j = 4. We repeatedly create bootstrap samples, and each time we record whether or not the fourth observation is contained in the bootstrap sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6353635363536354"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store = [] \n",
    "for i in np.arange(1, 10000):\n",
    "    store += [np.sum((np.random.randint(low=1, high=101, size=100) == 4)) > 0]\n",
    "\n",
    "np.mean(store)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comment**\n",
    "\n",
    "The result observed from a numerical approach above is simimilar to to our probabilistic estimation for a sample size of 100 which was P = 0.634.\n",
    "\n",
    "It is interesting to note that there is a suprisingly high level of variability between results given that results are averaged over 10000 tests. This can be observed by running the above cell multiple time (note that no random seed is set)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. We now review k-fold cross-validation.\n",
    "\n",
    "### (a) Explain how k-fold cross-validation is implemented.\n",
    "\n",
    "In k-fold cross validation k independant samples are taken from the set of all available observations each of size, $1 - (\\frac{n}{k})$\n",
    "\n",
    "The model is then fitted to each of these training samples, and then tested on the observations that were excluded from that sample. This produces k error scores which are then averaged to produce the final cross-validation score.\n",
    "\n",
    "Note that the proportion of observations that are included in each training increases increases with k.\n",
    "\n",
    "### (b) What are the advantages and disadvantages of k-fold cross- validation relative to:\n",
    "\n",
    "### - i. The validation set approach?\n",
    "\n",
    "When $k>2$, cross-validation provides a larger training set than the validation set approach. This means that there is less bias in the training setting. This means crossvalidation can produce more accurate estimates for more flexible models that benefit from a larger number of observations in the training set.\n",
    "\n",
    "Cross-validation results will exhibit more variability than the validation set approach. The approach is also more computationally expensive because the model must be fitted and tested for each fold in k.\n",
    "\n",
    "### - ii. LOOCV?\n",
    "\n",
    "Cross-validation for k<n provides less variance in accuracy scores than LOOCV. It is also less computationally expensive in most settings (assuming that a linear regression model isn't used and a certain mathematical shortcut implemented for LOOCV).\n",
    "\n",
    "LOOCV provides the maximum proportion of the original observations for training, which could provide an improved accuracy estimate if the model is limited by the size of training set available in k-fold cross validation (for some small value of k). LOOCV can also be more computationally efficient for linear regression assuming that the afformentioned mathematical shortcut is implemented.\n",
    "\n",
    "\n",
    "## 4. Suppose that we use some statistical learning method to make a prediction for the response Y for a particular value of the predictor X. Carefully describe how we might estimate the standard deviation of our prediction.\n",
    "\n",
    "**INCORRECT: Need to revisit**\n",
    "\n",
    "\n",
    "Given n observations of X sampled from the population, we use our model to make the corresponding predictions for the response Y. We can then estimate the standard deviation of the the responses using the formula:\n",
    "\n",
    "$\\hat{σ} = \\sqrt{\\frac{1}{n - 1}\\sum_{i=1}^{n}{(\\hat{y_i} - \\bar{y})^2}}$\n",
    "\n",
    "The accuracy of this estimate is limited by its variability. \n",
    "\n",
    "We can use the bootstrap method to improve the accuarcy of this estimate. We randomly select n observations from the original dataset to produce a boostrap dataset $X^{*1}$, the sampling is done with replacement. We can use $X^{*1}$ to produce a new bootstrap estimate $\\hat{σ}^{*1}$. We repeat this procedure $B$ times for some large value of $B$ to produce $B$ different bootstrap datasets, $X^{*1}$, $X^{*2}$,..., $X^{*B}$, and $B$ resultant $σ$ estimates, $\\hat{σ}^{*1}$, $\\hat{σ}^{*2}$,...,$\\hat{σ}^{*B}$. The estimate $\\hat{σ}$ is then:\n",
    "\n",
    "$\\hat{σ} = E[\\hat{σ}^{*B}]$\n",
    "\n",
    "For large values $B$ should provide an improved estimate $\\hat{σ}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}