{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 4. 모델 훈련"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "matplotlib.rc('font', family='NanumBarunGothic')\n",
    "plt.rcParams['axes.unicode_minus'] = False\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "np.random.seed(42)\n",
    "\n",
    "m = 100\n",
    "X = 6 * np.random.rand(m, 1) - 3\n",
    "y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 4.4 학습곡선"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Figure4-14](./images/Figure4-14.png)\n",
    "**<center>그림 4-14 고차(300차) 다항 회귀</center>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(80, 1) (20, 1) (80, 1) (20, 1)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAERCAYAAACD9ivUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xe8HGXZ//HPldOSk94IEEwoSYDQSaRIiwJCEAREio/wQ6Q9oGIDQRQSFEGUIqiAFEEQsPBEEBQEEkGIBDh0CCUJJKSQRnIISU6SU67fH/dOds9mz57dzbaz+b5fr3ntzuw9M9e2uea+Z+Yec3dERESy0a3UAYiISNej5CEiIllT8hARkawpeYiISNaUPEREJGtKHiIikjUlDxERyVpRk4eZ/dTMXjWz583sJTM7p4Ny3zCzt83sDTN72MyGFDNOERFJr9g1j+XAWHffCzgM+KWZbZNYwMzGARcCB7j7zkADcHuR4xQRkTSsVFeYm9mngYeA7d3944TpNwHL3f3i2Hh/YAkwMLGciIiUTnWxV2hmI4F/AgOBE1IkhG2Bv0Qj7r7czD4GtgZeTVrWWcBZAD179hyzww47FDByEZHK8+KLLy5198HZzlf05OHuM4CRZrYH8A8zO9Td30woYkBr0mwtpGhic/dbgFsAxo4d6w0NDQWKWkSkMpnZnFzmK9nZVu7+MvAs8Nmkl+YBw6IRM6sn1FLmFS86ERFJp2jJw8x2MbMTzMxi40OBvYGXzWyymR0QK3o38FUz6xsb/wYw1d2XFCtWERFJr5jNVnOA/wUuNLNmoBa4BHgR2AYYAODu/zaz3wJPxcotAE4qYpwiItKJoiUPd19BqEWksm1S2RuAGwoelIiI5ERXmIuISNaKfraViEg+rFixgsWLF9Pc3FzqUMpWTU0Nm222GX369Mn7spU8RKTLWbFiBYsWLWLo0KH06NGD2Hk4ksDdaWpqYv78+QB5TyBqthKRLmfx4sUMHTqU+vp6JY4OmBn19fUMHTqUxYsX5335Sh4i0uU0NzfTo0ePUofRJfTo0aMgTXtKHiLSJanGkZlCfU5KHiIiRfad73wHM2s37L777lkvZ9y4cVx99dUFiLBzOmAuIlIk06ZNY+HChey///7ss88+7V7r1q0bDzzwAFVVVRx11FFcdNFFXHXVVe3K9OzZk5UrVxYz5A4peYiIFMn06dOZPn162jI1NTUcddRRXHHFFVx++eXrpz/11FMcffTRAIwYMYLZs2fT2trKkUceWdCYO6LkISIycWIYCuzrX/86bW1t3Hrrrdx///28//77tLa2MmTIEA466CDOP/98Bg8OvaPPnDmTd999d/28r7/++vrnM2fOBOCQQw4peMwdUfIQEbnssqIkD4Cf//zn/OEPf+COO+5gl112oaamhtmzZ/ODH/yAr3zlKzzxxBMA3HvvvVx22WXt5u3Zs2dRYsyEDpiLSGUwy33YmPmz1NLSQlVVFbW1tVRVVdGtWzdqamqora2ltbX9rYwOPvhgli9fvn6ILvgbMWIE1dXVTJ48eaM/tlyp5iEiUkQXX3wxdXV1nHPOObz//vu0tLSw+eabM27cOK6//vp2ZauqqujVqxetra2sWbOGZcuW8frrr3PTTTdx6KGHMm7cuNK8CZQ8RKRSuOc+r9nGzZ+F6upqLrzwQi688MJOyz722GPU1NTQvXt3evXqRb9+/dhuu+04+uijOfTQQ4sQbceUPEREimCnnXZizpz2d3x1d1avXp2ym5U//elPeJESWi6UPEREJkwo+CrefPNNABYuXEi/fv3o3r07M2fOZOTIkbzzzjtstdVWKeeLrg055phjNnjtlFNOYYcddiho3B3RAXMRkSKdaQUwduxYHn30USBcGNizZ0+6det4U/zoo49y8803p3zt9NNPZ7/99itInJ1RzUNEpMhWrVpFY2MjAwYMYN68eQA0NjYC4SLB5FNyW1tbO7yyvLq6mu7duxc24FTrLfoaRUQ2cSeffHKHrx122GHrayaRJ554gt69e6csf/DBB6+/NqSYlDxERIooqmlkauLEiUwsYrNapnTMQ0REsqbkISIiWVPyEBGRrCl5iIhI1pQ8REQka0oeIiKSNSUPERHJWlGTh5mdbWavmlmDmb1mZuemKDPQzFaZ2bSEoTR3eBcRKbL777+frbfeutRhdKpoycPMqoCRwH7uPhYYD1xjZkOTig4C/uvu+yQM5xcrThGRQjr99NOprq5uN5gZDQ0NKcvfeOON7cpGXbTX1dVRX1/PK6+8UuR3EBQtebh7q7uf7+5RBy0fAeuAqqSig4BdzexZM3vBzH5tZkOKFaeISCHdeOONNDY20tjYyIoVK3jggQcYNGgQu+22W8ry55xzDmvWrGHNmjWsXbuWpqYmrr32WgYMGMCf/vQndt999yK/g6CU3ZP8Cvizu3+QNL0B2NLdW82sJ/AT4J9mNtbLuXN7EZEM1NXVUVdXB0BbWxtXX3013/ve92hqamLVqlWsXr26XXkzo7q6mvfee49JkyZx0003sXDhQg4//HBGjhxZircAlOiAuZldDgwFvpn8mruvdffW2PNVwA+AHYARKZZzVuz4ScOSJUsKHLWISP6sWbOGs846CzPjggsuYNSoUfTv359TTz11fZlXX32VE044ge23357x48fT2NjIY489xvz58/n0pz/Nl770JUaPHs0ll1xS/Dfg7kUdgKuBvwG1GZavBpqAIenKjRkzxkVk0zB9+vQNpoX7yJZmyMa6dev8/vvv9+22285PPPFEX7VqVbvX//rXv/rw4cPXl50yZYo3NjZ2uLwZM2b4G2+8kfXnFQEaPIdtedGarcysG3Aj0B843t1bYtOrgMeAie7+tJl9EXjB3T+0cF/GnwBPuvuiYsUqIlIIS5YsYc8992Trrbfm17/+NePHj++w7KRJk/jBD36Q8bJHjx7N3//+93yEmZFiHvM4AjibcEzjmYT79V4ObAMMiI07MMnMaoA24DXgq0WMU0SkIAYPHsz06dPp3bv3+oPgyQ488EAeeughRo0axYEHHpjxsquri3sIu2hrc/eHAevg5YcTyj0EPFSUoESkYnSV02l69+7N0qVLGTx4cIdlhg8fzuzZs9cfWJ82bRrXXHMNL7zwAkuXLqW6upphw4Yxfvx4LrroIvr371+s8NermCvMW1pKHYGISHbefvttmpqa2g333ntvuzLTpk3jwAMPZPvtt2fKlCk0Njby4YcfctNNN/H8888zbtw42traih57xdxJ8LXX4MMPYYstSh2JiEhm6urqNrj/eE1NTbvxf/3rXwwbNozLL798/bTq6mr2228/rrrqKvbee2/mzp3L8OHDixJzpGJqHu5w++2ljkJEJHOrV69m5cqV7Ybk4yCHH344H3zwARMnTmTOnDm0trayZs0annvuOX74wx+y66678qlPfarosVdM8gC45RZobS11FCIimdlpp53o3bt3u+GUU05pV2bvvffm6aef5o033mDfffele/fubLHFFpxxxhmMHTuWf//733TrVvxNecU0WwHMnQv//CccdVSpIxER6digQYOi69gysvfee3P//fcXMKLsVVTNA+Dmm0sdgYhI5au45PHIIzB7dqmjEBGpbBWTPPr0CY/u4diHiIgUTsUkj8TrbW6/HdatK10sIiKVrmKSR9++MDR2W6nFi+FvfyttPCJSWNkccN6UFepzqpjkYQZnnhkf14FzkcpVU1NDU1NTqcPoEpqamja48DAfKiZ5AJxxBlTF7kv45JPw1lslDUdECmSzzTZj/vz5rF69WjWQDrg7q1evZv78+Wy22WZ5X35FXecxdCh88YvxJqubboIbbihtTCKSf31iZ8gsWLCA5ubmEkdTvmpqahgyZMj6zyufrFKy9tixY72hoYEnnoBDDw3TevWCefPC8RAREdmQmb3o7mOzna+imq0ADj4YRo8Oz1euhN//vrTxiIhUoopLHmbwne/Ex2+4Qf1diYjkW8UlD4CTT4aBA8Pz2bOhiHdmFBHZJFRk8ujRA84+Oz7+q1+VLhYRkUpUkckD4NxzIbql73/+Ay+9VNp4REQqScUmj6FD4YQT4uPXX1+6WEREKk3FJg+Ab387/vy++8JtakVEZONVdPLYay/4zGfC8+ZmuPHG0sYjIlIpKjp5QPvTdm+9NSQRERHZOBWfPI49FrbcMjxftAgefLC08YiIVIKKTx7V1XD66fHx3/2udLGIiFSKik8eEHrb7RZ7p088ATNnljYeEZGubpNIHsOGwfjx8fFbb83/Olpa4N13oa0t/8sWESk3m0TygPZXnN9xR35vU9vWBp/9LGy/PXzta/lbrohIuSpq8jCzs83sVTNrMLPXzOzcDsp9w8zeNrM3zOxhMxuysesePx622io8X7Ikv7epnTIFnnkmPL/77lADScU99LV1113hroe77AKHHKLrT0Sk6yla8jCzKmAksF+s7/jxwDVmNjSp3DjgQuAAd98ZaABu39j1V1eHYx+RfB44T+72PdWy//tfGDkSttkGTj0VbrsN3ngDJk+Gc87JXywiIsVQsptBmVl3YBGwi7t/kDD9JmC5u18cG+8PLAEGuvvHHS0vuhlUOvPmwfDh8eMS77wDo0ZlFu/cuaG796j2Elm+HLbYAtaujU/r3x/mzw8dNEI4HrLzzmF9HXn4YfjCFzKLRUQkX7rizaB+Bfw5MXHEbAvMikbcfTnwMbB18gLM7KxYE1jDkiVLOl3hVlvBkUfGx2+5JX35996Dn/8c9twzHHTfZht44IH2Ze69t33igJBQ/vKX+Phdd8UTR20tHHYYXH45HHdcvMx558GaNZ2+BRGR8uDuRR+Ay4GHgNoUrz0GfC1p2iJgj3TLHDNmjGfiH/9wD0cf3Lt3D+PJXnjBfd994+UShyFD3Bsb42X33DP+2ujR8ef77BNeb2py/9Sn4tN/+tP4vIsXu/fvH39t4sT4ax995H766e5Dh7qPGOG+997uRxzhftpp7k8+mdFbFRHpFNDguWzHc5lpYwbgauBvqRJH7PXfA5cmjNcDLcDgdMvNNHm0tLjvsEN8g11d7X7ffeG1tjb3G25wr6lJnTii4bzzQvmXX26fiN55p/28L7/sft118fHNNnP/5JP28dx8c/z1ujr3WbPcH3/cfcstO16/WUg0ra0ZvWURkQ6VffIgNJHdDPwZqE6YXgVMJhwgB/gs8A7QNzZ+AfBUZ8vPNHm4u8+Y4T58ePuN8bXXun/5y+030jU17l/4gvtdd7nfdlt8erdu7q+84v6tb8Wn/c//hGWfdFL7aYMGxcevv37DWFpa3MeOjZdJjKuz4cgj3Zcvz/hti4hsoCskjyMBB14ApiUMRwLvAUcnlD0PeCVW9kFgi86Wn03ycHefN699M1PysMceIclE2trcDzkk/vo++7RvcnriiVDuqadSL2/4cPc1a1LH8vzzIYElzzNokPv997u/9Zb7M8+4P/ig++c+177MiBHur72W1VsXEVmv7JNHoYdsk4e7+9Kl7nvtteFG+9xzw7GKZG+/nbpJa+ut401IbW3uO+20YZk77kgfy9lnty9/xBHuH364YbnmZvcLLmhftnt392uuCbWYQvvkk1CDOvNM9+OOC8ls993d99/f/Y9/VFOaSFeTa/Io2am6+ZbJqbqpfPIJfOlLoc+r3r3D9ReJdyBMdvHFcOWV7adddhlceml8/De/gW99Kz6+447w+utQVdXxcpcvD2eCzZgBEyeGaz/MOi7/17/CaafBqlXxafvvH66eHzGi4/k609YG778P/frBwIHx6S0tYdmXXgoLF3Y8/5gx8Mtfhivu3UM/YlOmwFtvhYsijz8e+vTJPb6NsWhR6ONs8OANX5s1Cy68EP797/D6jjuGYdQoqKsLXfmvWxce+/ULZ+5ttVXosbmurvjvRSRfcj1Vd5NPHhA2mM8/H7oX6d8/fdlVq2D0aPggdoKxWbhqfNiweJmPPw4bldWrw/j//V9IUJlwT580Ek2fDl/9KrzySnxafT18//thw7fVVuF2vDU14bqTefPC47p1IdHstVc8oTU1havjr7kmfoX8DjuEm2ntvHNIqtOnZxYXhPk++CCsM1GPHuEU5a99LXyOq1aFz2nVqhD7sGFh45zpZ5DOypXw1FPw+OPw2GMhgQEcdFC4wv+440JS/NnP4Nprc++yZrvtQuL/4hfhgAPC5y3SVSh5bETyyNYDD4T7hEC4sO/hhzcsc8cdoZZyzDHhDob52Bimsm5d2Pj97GfQ2prdvP37w6GHhgsn77wzdNuSiS23DDfZGj48LKNfv9Ddy3XX5edalV694kmkqSkkl9Wrw2e4556w334hOY0Zk3qvf8aMkAzuvDN9PP37h/nT1aSy1a9fuI5nzBjYbbcwDNnoznVECkfJo4jJA+Cee+Dll0NTR6pmkGJ78cXQ7cmbb278snr2jDfRJOrVCy66KCSOnj03nG/uXLjkknBRZPSz6tMHxo0LtZeHH4bXXtv4+CJ1daEpbM89YY89Qk3rjjtCIkv1s66tDTWNjno+3nvvUPPq3h3efjvUVGbNCsuqrQ01iupqWLYs1Kjmzg39knXWk/Jmm4XayTbbhGHYsNB8tnZt+JzXrQvr6NYtJMhu3WDAgNCdzYgRIfkUauej0ixaFP6Xq1eHHYc1a8IOyKpVoSb6ySfh+aBBoWa9ww6hxaFnz9B0vHQpfPRR+D623Tb0HtGtwruPVfIocvIoR2vXwh//GI6vRE1U8+aFJDB0aHxYsyY04yxY0H7+YcNCYjj99LChbGiAqVPh1VfDH+nb3w4bws689VZIZqNGhQ17dXX8tVdeCTWCBx4IcfTsGYYePcIfe86ceHNfPuy4IxxxRKhhHXBA2EDceSfcfns4tgOw+eZw1VVw8snZbyjWrg1NY3//exjmzs1f7JFevULS2WKLEGv0OHhw2AgOGhRvbm1tjQ9tbWFwD49RYooGs/ZJqWfPkIDLcWO5fHn4Xb31VnzjP2hQ+AyWLQu/58cea9+Emw/du8cTf/R5DxwYapjNzSExNTWF33Lv3uH/EQ21te13EGprQ619yy1DEy2EhDZjRhgWLAjLj8psuWVYZqF3HJQ8lDyy4h46Znz00bDB3n9/+PKX22/oSxXXsmUhppUrwwatvj4Mn3wSOpicOjU8prup1+GHww9+EGo9qf58bW1hoz93bmiC7N07P7G/8go8+2xIuK++GhJ5PpNhofXqFWqJu+wS9sjNQm2tuTk8ppPqJHWzcFytujoMa9aEY4Tvvx8eFywIOyo9esSH2tp4+aqq8B3ls2mxHPTtGxLTokXpy1VXh9p7nz5hnr594+N9+oTfbfT/qK8Pn1diLWvlylCLTT7JJ5GSR6Umj4kTwyAbWLo0bKxfeik0VcycGTZ63/1ueCwHra3hxIH3348P8+eHvfu6urChrK0NG9mohtDWFprDoj3SFStK/S66jurq0Pw4eHDYOEdDr17xob4+JK233grNkzNmhOTYv3+oVQwcGL6DWbNCE1ZXt9126Xe0lDwqJXlEyaKpKewajx8fnnfvXurIpATcQ5KMjq8sXBgeFy0K05cuDSc6NDbG9/ITh6ipKqp9RckpGhItWVK+G8u6unB8YvTosJGP3vvSpeH1Aw4IJyqMG5d9LTI60STVqfSNjSGJzJkTPpvomEhjY4gpqi3V1YUkv3hxfGhtje8c1NaGGuiHH4bEFR1PrKkJTcIjR8KnPhVq3QsWxIemppw/svWGDElfc8s1eZS4kULaWbYsXDTy/PPhgoPoVKEvfxkmTQq/wEx0ldpKOcaZKqYSxmkW9qKLcVKGe9jIvP56GGbPDhvUmpr4yQKdtb9Hx1GiwT1+DKalJSwv6qF6663DBrOtLX7soKkp3kQWPQ4cGDaw6a6T2hgplxv7zvv1C2fOjXloYt5+F21t4a++alU4BpmuqXjdunDq/4oV4TF6Ho1Hp7pHQ0tL+1pWr16dX36Qs1yuLCzHIZcrzEtiwoTU09esSd9fCrhfckn6ZTc3u997byhbjpLfeznEmRjT9OkhpmuvDZfR//a37rfeGqa1taWep1yUY0xdWfTbbG4OHcgl/wYSy0Q29jtINX/ytM7Gc4C6J+kiySPVBnPChNTJIprep094POWU0P9Hqh/QI4+0Tz6/+MWGP/ZSamsLcV1/vftRR8V7jPzvf+NlSrEBhJBwDzwwfeLebjv3733P/T//6XyjUYA/eForVoSYknvJVELJ3IQJ4Tc6ZUroDRXca2vb/wZ69gx9Dx15ZOhaG0IPqevWhWVs7O8icf5Vq9zfeCNMS+wrKXkdeUhgSh7lnjzmzHHfbz9PuQfzwQfu9fXxH2kicJ86NfxwId4J1vvvu7/3nvu0aek3ej/+cek3IlOnhg7A0sUZva9iuv329DGlG777XfeGhnhSTBT94T/+2H3JktTva2MTzIQJofOziy9279cvrKO62v3QQ91vvNF9wYK8bFg2Cc3N4bPaYovcfw/RPRTOOCN0Pvezn4XxN94Iy3ePfx+rV8cTwz33uP/mN+FGP+D++c+7Dxu24fL793ffccfw/HOfCwnshBPC+G9+E3pmnTcvp+9cyaOck8cPf5j6Bxd9sVFf8NFjoqjM5Mnhhh8d/Xj79HH/5S9D8xe49+gRHg86qPgb5UTf+EbqeM8919fvzSVOf+ih+J+tUBu7jmp6F12U+rOCcBeuVPMMHBgeDzzQfdQo9759U5fbfHP38ePDxv4vfwnT5s3bcMOSGGOq8ebmUFsD96qqjn8P0XDFFe5vvpk6yWXyOeVbsWtl6da7erX7TTe5b7tt/PMaNCjcLAfC69GOHoSa3csvu0+a5H7VVfGNemffQV1d/K5xW23VefmNHfbcM9wb4tJLw/j776f9aJQ8yjV5tLa6f+lLG/5wrrsuvP7442G8vj7UQDr6M3W0wUuVkCDsFSfuSb36apHecILFi+N/zKOO2nDjBe7nn5/6fUS1tM7k0mT05z+Hm7JE60qsCXaUPCLRRjhVd8y5DGYhsYD7r38d7igWraOtLdxS8oUXwvgxx7RPTmbuxx4bTyZLl7offXTq9UQbucmT3deu7fiz6eh9p/s8M7VoUVjmP/7h/txzoeacyzoyfb2tLdzP4L77wnqmTnV/9934HnpibT/VfyhRut/F8uXh9p4QagS5/A723z88Pvhg+A0kNoW1toYa7GuvhfHHHw/love1xx7pl33jjWk/roIkD+DYTheQQZliDGWbPA44IHzMffuGH3Lil3rnnfE91yuvzHyZnf2wJ0xIfxylGHt7F18cbnoC7mPGuK9cmTrOSNTMMmpU+3iPOCL8Wdra2m8Upk8PSQDCBjYSbXQXLQobqOTk8JWvhOYdiDcVdBRTR9OieZqb48l/ypQQ00cfpf5+Zs50P/74zDYkUYKIjnV1NqTa2EWf9267pZ4nOj62YMGG7/ODD+K3wLzuupC8Mq0hdaSxsf0tPBOHUaNCze622+L/kUw+/+TX29rCMYjodzNgQOef3R57xH9H6daZze8iefzjj92ffjqMz5rV8WeZLkFlM/71r3f8O0mhUMljQdL4B52VKdVQlskjOvOpWzf3f/0rTJswIRzMTv7zRHuDmcjkRxdZvbr9us48M335fGhtja9v+PD4jUky2dONqtrJw2abhcdjj21/e8bE16O9t+SmsEGDwh7hhRfGp11wQfuElI1MNxodjSdOW7cubFAg9Y1gMk0Wqd5HVKalJX6gP9U6dt01fpOYoUNTr7Omxv3gg8Pzhob4jVsy+S1dcklm7ytxOPzw0Hz0yCNh/P77w9lv0Xv//e9DLWrmzDD+ox9tuOOR6ZDq88zFxv4u8nG2VTbbhvVFCpM8Pkw33tG0UgxllzymTw93aYIN7z/b1hbO3Il+vFFiyVS27cYQ9uoSj5ncc09ILJnMn62ozbhv39DenqlUf4TLL4836+Rr+N//ze+ZaLk0nXX0J29pcX/xxTC+ZEn7Nvd082cSV+KGMt3Qo0f8+Ntpp3W8Bx8losQ29VTvNTozKUr6yTE991w4UJzP7zgaLr00s8+vFMd3irHOEiaP5JrHBrUM1TxSSNdklMnrhYino3VGB/KSb0OY6x55Pt9Xuo1d4kahtdV99uxwxgm4L1vWfhnRxqtYn3cmNnYPMl81ph/9qOPPJnGdCxfGD/R3dFJA9FtatCg+X+Jpr1HTTbr3FR3viZJX8hCdUr3LLqlfj2o5yXJJvpWgVGdbKXlshMSbjadT7B9xdE3Apz/d/k/Xt29oJ77yynDD9M42Vql+lK2t8aajfLyvTPaiOtsodNWNRjH2UnNJUBA28O++G04R7SiZ7Lyz++mnx8fvvDP1MrOpleU6nul6N1EFSx5ADVAL1CWNr5+Wy4rzPZRV8li3ruPrNpKVYmOWbo8+cTjllHBa4qpVmf1Bo6uxhwwpzPvKZaOQQzV+k5HLZ5Nqns6OaVx4YX7jyvZ1SatQyaMNaE0YksdbgdZcVpzvoaySx/PPh4921KiNP+2wEFL9+ebOjZ9SnDxEZyeNHh32KHfdNYzPnBlfxsKF8YvV7ruvOO25pVpGpSpGU1gh4tJ3ulFyTR5pe9U1s4My7B/rqUzKFVJZ9ap73XXwve/B178e7jpU7qIe7JKnHXwwTJ7c+fw//CE89FC4Qcjhh8M//6lb322qkn9LqX5bUlYK0qtuOSSFLumZZ8Lj/vuXNo5MTZiQevoTT4THRYvCretefz10Cbp2Ley1V7zcbbeF/rx79CjsDdul/HX0W5KKkzZ5mFkdUOvunyRM6wGcB3wKmOTuUwobYhfjDk8/HZ53leSRqlvpxI3AkCHhceed25d59lk47zx44YX4PNtsU5AQpYtI/i0pmVSszu5W/AvgzGjEzLoBjwITgH2Ah8zsqMKF1wXNmBH2wocMgREjSh1N7jrbCEyYEO5hGyUOgIsuCrWOcrtHh5SOfgsVq7ObQX0BGJMwfjKwF7Cvu79iZocDPwIeKlB8XU9ik1UlNd+kuhFO4qPatkU2KZ3VPOrd/eOE8e8Bt7n7KwDu/iiwdYFi65q62vEOEZEcdJY81sWOcWBm+wE7A79JKpPx7rWZ1ZgbaOpFAAATfElEQVTZ+WbWbGYndVBmjJktM7NpCcP3M11HyW2qyUNt2yKblM6arR4DbjazfwCXAQ+6+zvRi2a2I/BJRzOncCbgwLQ0ZQYB/+fuZ6YpU54WLQrHPHr2hN13L3U0xaW2bZFNSmc1jx8Cw4E7gPeAs6IXzMyA64D7Ml2Zu9/o7tcQLi7syCDg82b2XGz4mZn1znQdJTV1anjcZ5/0d7UXEeniOrvO4yNgXAevuZmdCKzMc0yTgHtjyx9AaCa7Czg2uaCZnUUsoQ0bNizPYeRgU22yEpFNTmc1j7Tc/WN3T1eLyGWZTbFL5nH3ZcD3gaOiYy9JZW9x97HuPnbw4MH5DCM30fUdBxxQ2jhERAqss4sE38tkIe6+bX7CSakKWAOsLeA6Nt7KlfDii1BVBXvvXepoREQKqrOG+a2B+cBfgYdJf6wiJ2Y2EPg78DV3nxE7C+tRd280s2rgSuBud2/L97rz6rnnwnUOe+wBvXqVOhoRkYLKJHmcBnwNOBH4A+E6j4xqJBmqJxyU7xsb7wFMNrM2wplZTwGX5nF9hdHVuiQREdkIaXvVXV8onFl1KCGRHA08C9xK6NtqXUEjzFDJetWdOBEuu2zD6RMm6PRVESl7ufaqm1HySFrRAOCrhEQyjNCk9N1sV5xvJe2SffHi0Ousezj20bNnaeIQEclSrskj67OtYmdA/Qd4hnBXQZ1a9Mgj8X6dlDhEZBOQcfIws75mdo6ZvUg4DlENfDaXjFUShWxC+sc/wuP48YVbh4hIGem02crMPgucQbhI7xXCsY4/u/vqwoeXuU6brQrV62tzMwwaBCtWwHvv6X4WItKlFOROgmY2C6gF7gfGuvv0HOMrrZdfDo+rVuW/WemZZ0LiGD1aiUNENhmdNVttAwwFvg28bmatSUObmeX92o+8mTgx1Dj23DOM9+qV/5sVPfxweDzyyPwtU0SkzHV2nUcmu9Jb5iOQgpg4MQyHHAKTJ4fmpTlzoL4+f+uIjnd84Qv5W6aISJnrrOaxALgAeIFwbceZ7j7H3ecAHwDHAI8UNsQ8WLUqPC5dCnfemfl8ndVQZs6Ed96Bfv3gM5/JNToRkS6ns+RxEbAv8C3gx8DxZnaWme0DvAz8BLimsCHmQZQ8AK65BlpaMpsv1cV/iaJax+GHqwt2EdmkdLbF+wpwuLt/AGBmrwFPAHXA7cCh7r6ksCHmQZQ8+vQJZ0RNmgQnnNBx+TVr4PTTO1+ujneIyCaqs5pH7yhxxLxISDi7uPs3u0TigHjy+M53wuMvftHxabsTJ0KPHnDvvWHcLPVB9k8+gSlToFu3UPMQEdmEdJY8qhJHYvfZWOHuMwsXUgFEyePcc2Hw4NB1+r//nfqYxtlnt+8Vd9SoUBNJLvv449DWBvvuCwMHFipyEZGy1GnNw8yuSByAXimmlS/3ePIYNAjOOy88v+qq1Mc0fvSj0D9VdLX4u+/Cz3/evkxbG9x9d3iuJisR2QSlvcLczJ4kdIuejrv75/IZVC46vMK8qSmcmltXF2oQy5bBsGHxhJL4/hsaYK+9wsHvN9+EK64IZ2fV1sLrr4dayCWXwOWXb7ge9aIrIl1QQa4wd/dxOUdULqIkEV1ZfsMN7c++MguPl14Kd9wRksm3vw0jR4Zxs/B4zjkhsbz0Uijfq1eooRSiyxMRkTK3Ufcw7xKSk8fEiWGj/5WvxMucdBIMHQpz54ZjIj/+cfy1X/wiHNOYMiU0Xz32GGy2GTz5ZLHegYhI2dn0kkf0/J574s//9KdwoBxCU1XfvvGygwbB1VfHx7fdFqZOhTFjQlOViMgmaNNMHhCaoyZMgNNOaz/9zDPbn5o7cWL7Mu+9F5q0oq5PREQ2QVnfSbBcdXjA/Mkn4bOfhQMPhKeeSj3z2rXw4INw4onpj2EUqlt3EZESKdqdBLucjmoeierq0l9xLiIi7Sh5JOrsGIaOcYiIAEoe7XV2DEPHOEREACUPERHJgZKHiIhkTclDRESypuQhIiJZU/IQEZGsFTV5mFmNmZ1vZs1mdlIHZczMfmpm75jZdDP7o5nlvuVX8hARybti1zzOJHTxPi1NmVOBI4Dd3X000AxclfMalTxERPKuqMnD3W9092uA1jTFTgR+5+5NsfHrCfdSz42Sh4hI3pXjMY9tgVkJ47OAAWbWN7mgmZ1lZg1m1rBkSQe3U1fyEBHJu3JMHkb7mklL7HGDWN39Fncf6+5jBw8enHppSh4iInlXjsljHjAsYXwYsBJozGlpSh4iInlX8uRhZgPNbKqZjYxNuhs4w8xqY+PfAiZ5rn3HK3mIiORd2nuYF0k9MByIjmncBYwAnjezFmA68M2cl67kISKSdyVJHu4+LuH5XGCrhPFW4EexYeMpeYiI5F3Jm60Kat06aGmB6mqore28vIiIZKSyk4dqHSIiBaHkISIiWVPyEBGRrCl5iIhI1pQ8REQka0oeIiKSNSUPERHJmpKHiIhkTclDRESypuQhIiJZU/IQEZGsKXmIiEjWlDxERCRrSh4iIpI1JQ8REcmakoeIiGRNyUNERLKm5CEiIllT8hARkawpeYiISNaUPEREJGtKHiIikrXKTR6trbB2LZhB9+6ljkZEpKJUbvKIah319SGBiIhI3lR+8lCTlYhI3il5iIhI1oqaPMzsIDN7ycxeM7MGM9snRZkxZrbMzKYlDN/PemVKHiIiBVNdrBWZWT9gEnCkuz9rZuOAB81sG3dfnVB0EPB/7n7mRq1QyUNEpGCKWfM4DHjH3Z8FcPcngQ+Bg5PKDQI+b2bPxYafmVnvrNem5CEiUjDFTB7bArOSps2KTU80Cdja3fcGxgPbAHdlvTYlDxGRgilasxVgQGvStBaSEpi7NyU8XxY73jHXzHokvgZgZmcBZwEMGzas/ZKVPERECqaYNY95QNIWnmGx6elUAWuAtckvuPst7j7W3ccOHjy4/YtKHiIiBVPM5PEgsKuZ7QJgZnsBOwBTzGyqmY2MTT8pdnAdM6sGrgTudve2rNam5CEiUjBFa7Zy94/N7Hjg92bmhCarI4B6YDjQN1a0BzDZzNoAB54CLs16hUoeIiIFU8xjHrj7v4FPp3hpq4QydwB3bPTKlDxERApGV5iLiEjWlDxERCRrSh4iIpI1JQ8REcmakoeIiGRNyUNERLKm5CEiIllT8hARkawpeYiISNaUPEREJGuVmTza2mB17OaE9fWljUVEpAJVZvJoit32o3t3qKoqbSwiIhWoMpOHmqxERApKyUNERLKm5CEiIllT8hARkawpeYiISNaUPEREJGtKHiIikjUlDxERyZqSh4iIZE3JQ0REsqbkISIiWVPyEBGRrCl5iIhI1pQ8REQka0oeIiKSNSUPERHJWlGTh5kdZGYvmdlrZtZgZvukKGNm9lMze8fMppvZH82s8yywYEH8uZKHiEhBFS15mFk/YBLwDXffFTgfeNDMku8TeypwBLC7u48GmoGrOl3Bhx/Ciy+G4aOPwjQlDxGRgqgu4roOA95x92cB3P1JM/sQOBh4KKHcicDv3D12L1muByYD3+x0DWPHth/v1WujgxYRkQ0VM3lsC8xKmjYrNj1duVnAADPr6+4fJxY0s7OAswAGAkmpA0aNAmARfDgPFiS/XCKDgKWlDiIDijN/ukKMoDjzravEuX0uMxUzeRjQmjSthQ2bzpLLtcQeN2hic/dbgFsAzKxhqfsG+aPcmFmDK8686QpxdoUYQXHmW1eKM5f5innAfB4wLGnasNj0dOWGASuBxsKFJiIi2Shm8ngQ2NXMdgEws72AHYApZjbVzEbGyt0NnGFmtbHxbwGT3N2LGKuIiKRRtGYrd//YzI4Hfm9mTmiOOgKoB4YDfWNF7wJGAM+bWQswnUwOlsear7oAxZlfXSHOrhAjKM58q+g4TTv0IiKSrcq8wlxERApKyUNERLLW5ZNHJl2elIKZ1ZjZ+WbWbGYnJUwvq3jN7GwzezUWy2tmdm5s+s6xExleM7PXzezIEsf501icz8c+v3PKMc6Ime1kZsvMbGJsvNy+99fN7AUzmxYbppRjnLGYvm9mb8TibTCz+nKJ08y+lfAZRsOKWHxlEWNCrBfEYnku9nhxbHpu/yF377ID0A/4CNg3Nj4OWATUl0Fs5wLfB54GTirHeIEq4GqgV2x8KNAUe5wJnBibviOwDNiqhJ/n94Ca2PPBhNO3tyu3OBO+5yeB3wITy+17j8UwG+iWIu5yi/My4OdAdUKMA8otzoR4twDmxP5DZRMjcAAwHxgYGx8Q+w0clut/qKvXPDbo8gSIujwpKXe/0d2vof0Fj2UVr7u3uvv57r4yNukjYB3hbLcewF9i5d4C/gMcW4o4YzFc6+7NsdGtCcljK8osTjPrBvwBuBhYEptcVt97zADgKTN72cz+Yma7U2ZxxvrDOwz4BJhqZk8B+wKHllOcSS4i7DTsT3nFGP0W+8ce+xBantaQ43+oqyePTLs8KRflHu+vgD8DWwLveWxXJKbkcZrZSDObAfwLOJnyjPNy4HF3/2/CtHL83oe4+wHAGMI1WE8QrrsqpzjHADsB89x9b+A7wL2EnZtyihMAM9sC+BJwI2X2nbv724SunF4ws3eBVwifZ87/oa6ePDLt8qRclG28ZnY5oar9Tco0Tnef4e4jCXtvdxF+4GUTp5kdBwxz998kv0QZxQngsY5H3b3N3e8BXoy9VE5xbgbMdvc/ALj7y4ROUqG84oz8EPhtrCZfVt+5mY0CbgYOdfdRhK4ArwC2Icc4S/1hb6xMuzwpF2UZr5ldTdjDO87d11GmcUZiG5FnCcdsyinO8cCO0YFT4IzY8DPKK85Uqght4OUU52JgRdK0Nsrve8fMtiTUOqIdh3L7D30ReMbdGwDcfSbwN+AQco2z1AeYNvIgUF9Cr5W7xMb3ApYD/UodW0KMTxI/YF5W8RJ2Hm4mNFVVJ0yvIlRdx8fGtyUcRBtZojh3AU4gflHr0NiPe79yijNF3BNjQ7l9758GxiSMH0E4mDq8zOKsJRwn2D82vn0svt3KKc5YDL8GLk4YL7fv/IvAXGCL2Hgf4CXgG7n+h4rZq27eeQddnrh7WXaiWIbxHgGcDTQAz5hZNP3HhANmvzWzn8SmneruM4ofIhDOXvlf4EIzayZsVC5x96lmVk5xplSG3/tK4Foz2xxYS9hYfN7d55RTnO6+zsyOAG6LnYjQRvh+Xy2nOM1sKHAcCV2bl9t37u5/N7MRwKNm1kToFuofhJ3Hp8nhP6TuSUREJGtd/ZiHiIiUgJKHiIhkTclDRESypuQhIiJZU/IQEZGsKXlIRTGzX5mZJw2v5LisJ83s/CznmWhmD8SezzazL+ey7kKxhPOxY7E+XMp4pOvq0td5iERi3V1vDjwDTEt6uc3MjgFa3f2hWPmfAxcmlVvl7r2yXO9Cwnnx/8qg7M7A62mKvOnuO6eZ/wnSd6x3pbtH3WyPAZ5Ler0b4artzTuLVaQzSh5SKUbHhnSagYdizy8mXAwZOYjQQSBmNpPQc28V0NmeeS2hZ9JsDAJWpZje1sl8R9Lxf/YPSct8mdB9eeKyzwWOyTBGkbSUPKQiuPvvY1chnwl8mdDhWxXhHgpPAVe7+5KEWUYAoxLGd0lY1ghYv6ffoVgTUA/i3V1naq27Z5twSDePmXUndF0elW0jXEmeWGZ/YGq26xVJRclDKslFwKnAaYTmoWZCDeIXwH2ETuAi/wNMSJo/VW0gnR2A7kC9mX0HuC42/cFO5mtMOPSQ6HZ3PzubAMyshnCPhi0JTVIdlRsCHE7opFFko+mAuVSSakL30utij22EBLKOUAtJNpmw4Y2GoRCarcyshc5v3LMP4c6LJxNuANQbuDKDOPu5e3WKIW3iMLOFZtaSMDgh4T1P6CxwUZrZLwUa3P2FDOIT6ZRqHlJJriB09HcTodmqGlhI6Nn42ynKtxKadqoINYgBZrYLcI67P25mT3ayvhMI93C4APilu883s3V5eB8d2Y5wnwgAB5pjnQf2JLyPD1LNZGbjCV3DH1DA2GQTo+QhFcPdW4CrYkMmPk+omawhbHwbCd1TPwg8nm5GM/sMoZvt44EhwL1mdki6eRJ80kGzFUCPjo5tuHtHzWrbEJLJ3BRxfpZwi9GL3P35DOMT6ZSSh3R5ZvYm4V4U7SYTup1eTdiwJjrJ3ScS7rWRy/oGAncDP3H3lWZ2DeFA/fWkOe7g7m8Qrznk0w7ABx5u5BXFWEOoEU0gdF9/XUczi+RCyUO6PHffCSB2f4pGd18Tu3fBDGB7d+/wrmjR9SHu/kCKl+8G3k4x/VrgLcINgHD3j8zs/xHuuZ33/5SZVXey3N2AmbEzriDUps4k3LP6KHd/LN8xiSh5SCVpINyD/QHCwfJVdH7txOGEA98bJA93v72Dec4CusVOh43KPgI8YmYTkwub2dbA+51GH/ecu++TMP5jNjwzLJWm2OM57n6jmf0+l1OCRTKh5CGVpqeZ9SPcHW8rgNg4hAPMqY4bVJlZR1eWtyRvgN19bZYxzSFcD5Kpdgkv1yY2JQ4pJCUPqTR/TPPavwg1jWSHkHCBXZLJtL8+JGsebtepDblUFN2GVkREsqaLBEVEJGtKHiIikjUlDxERyZqSh4iIZE3JQ0REsqbkISIiWVPyEBGRrCl5iIhI1v4/6P43pUTEgSgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "def plot_learning_curves(model, X, y):\n",
    "    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=10)\n",
    "    train_errors, val_errors = [], []\n",
    "    print(X_train.shape, X_val.shape, y_train.shape, y_val.shape)\n",
    "    for m in range(1, len(X_train)):\n",
    "        model.fit(X_train[:m], y_train[:m])\n",
    "        y_train_predict = model.predict(X_train[:m])\n",
    "        y_val_predict = model.predict(X_val)\n",
    "        train_errors.append(mean_squared_error(y_train[:m], y_train_predict))\n",
    "        val_errors.append(mean_squared_error(y_val, y_val_predict))\n",
    "\n",
    "    plt.plot(np.sqrt(train_errors), \"r-+\", linewidth=2, label=\"훈련\")\n",
    "    plt.plot(np.sqrt(val_errors), \"b-\", linewidth=3, label=\"검증\")\n",
    "    plt.legend(loc=\"upper right\", fontsize=14)\n",
    "    plt.xlabel(\"훈련 세트 크기\", fontsize=14)\n",
    "    plt.ylabel(\"RMSE\", fontsize=14)\n",
    "    \n",
    "lin_reg = LinearRegression()\n",
    "plot_learning_curves(lin_reg, X, y)\n",
    "plt.axis([0, 80, 0, 3])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ▣ 단순 선형 회귀 모델(직선)의 학습 곡선\n",
    "+ 훈련 세트\n",
    "    + 샘플이 하나 혹은 두 개일 때는 모델이 완벽하게 작동\n",
    "    + 샘플이 추가됨에 따라 노이즈도 있고 데이터가 비선형이기 때문에 모델이 완벽히 학습하는 것이 불가능\n",
    "    + 오차가 계속 상승하다가 어느 정도를 유지\n",
    "+ 검증 세트\n",
    "    + 샘플의 수가 적으면 제대로 일반화될 수 없어서 검증 오차가 초기에 매우 큼\n",
    "    + 샘플이 추가됨에 따라 학습이 되고 검증 오차가 천천히 감소\n",
    "    + 선형 회귀의 직선은 데이터를 잘 모델링할 수 없으므로 오차의 감소가 완만해져서 훈련 세트의 그래프와 가까워짐\n",
    "+ 과소적합 모델의 전형적인 모습  \n",
    "→ 훈련 샘플을 더 추가해도 효과가 없음, 더 복잡한 모델을 사용하거나 더 나은 특성을 선택해야 함"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### ▣ 10차 다항 회귀 모델의 학습 곡선"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(80, 1) (20, 1) (80, 1) (20, 1)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAERCAYAAACD9ivUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPXV+PHPIQkhYd8RkEVWRcAlbsUFH0SFutRata1au4l1a21dwBVoXVulWq1UfdSfdalaRFxQtGqLyyNKsOKCKKigIMgaIiSBLOf3x5lxJsMkMxMms+W8X6/7yr137sycbPfMdxdVxTnnnEtEq3QH4JxzLvt48nDOOZcwTx7OOecS5snDOedcwjx5OOecS5gnD+eccwnz5OGccy5hKU0eIvIHEVksIm+LyDsicm4D150vIktF5AMReVZEeqYyTuecc41LdcljM1CiqgcCxwB/EpGB4ReIyFhgMnCYqu4NlAL3pjhO55xzjZB0jTAXkQOAZ4Bhqrol7PxMYLOqXhE47gysB7qGX+eccy598lP9hiIyBHgO6AqcGiUh7AE8HjxQ1c0isgUYACyOeK1JwCSAtm3b7j98+PC4Yli0KLS/334gkuh3kXo1NbA48N3n5cE++6Q3Hudcbli0aNEGVe2e6PNSnjxUdRkwRET2BeaKyHhV/TDsEgFqI55WQ5QqNlW9G7gboKSkREtLS+OKIT8fagPv8NZbdpzpNm2Crl1tv0MHiPNbdc65RonIyqY8L229rVT1v8CbwJERD60C+gUPRKQYK6WsSl10mc3nsnTOpVvKkoeIjBSRU0WskkhE+gAHAf8VkZdF5LDApQ8Cp4tIx8Dx+cAbqro+VbFmomyoWnPOtRyprLBZCfwKmCwi1UBr4GpgETAQ6AKgqv8Wkb8C8wPXfQX8MIVxZjwveTjn0i1lyUNVy7FSRDR7RFz7F+AvzR5UFvGSh3Muk/gIc+eccwnLgn5GyZft1T7ZHr9zyVBeXs66deuorq5OdygZq6CggB49etChQ4ekv3aLTB7hsqU6KFvidC4VysvL+frrr+nTpw9FRUWI/4PsRFWprKxk9erVAElPIF5tlYW85OFaunXr1tGnTx+Ki4s9cTRARCguLqZPnz6sW7cu6a/vySNL+P+HcyHV1dUUFRWlO4ysUFRU1CxVe548nHNZyUsc8Wmun5Mnjyzk1VbOZbeLLroIEam37dOECevGjh3LzTff3AwRxpazDeaquVXVk0vfi3Mt1YIFC1i7di2HHnooBx98cL3HWrVqxZw5c8jLy+P4449nypQp3HTTTfWuadu2LVu3bk1lyA3KueRRUQHHHgurVsGsWTZrbq7xkodz2WnJkiUsWbKk0WsKCgo4/vjjuf7667n22mu/PT9//nxOPPFEAAYPHsyKFSuora3luOOOa9aYG5JzyeOf/4TXXrP9v/4V7o2yjFQ23ny95OFcM5o2zbZm9vOf/5y6ujruueceZs2axeeff05tbS09e/bkiCOO4JJLLqF7d5sdffny5XzyySffPvf999//dn/58uUAHHXUUc0ec0NyLnl89FFof1Uc8/Bm4005G5Ofcxlt+vSUJA+AG2+8kQceeID777+fkSNHUlBQwIoVK7jsssv40Y9+xEsvvQTAI488wvTp0+s9t23btimJMR4512AeSMgANEPX5rTJxiTnXEqJNH3blecnqKamhry8PFq3bk1eXh6tWrWioKCA1q1bU1tbfymjcePGsXnz5m+34IC/wYMHk5+fz8svv7zLP7amyrmSx7Jlof31LXoSd+dcJrriiisoLCzk3HPP5fPPP6empoZevXoxduxYbrvttnrX5uXl0a5dO2pra6mqqmLTpk28//77zJw5k/HjxzN27Nj0fBPkWPJQrV/yWL8+93pdgVdbORfVrvxjiKTsHys/P5/JkyczefLkmNe++OKLFBQU0KZNG9q1a0enTp0YNGgQJ554IuPHj09BtA3LqeSxZo31tgrasQPKy6Fjx4afky1yLQE619KMGDGClSvrr/iqqlRUVESdZuXRRx9FM/iTYk4lj/Aqq6D163MjeYTL4L8n57LT1KnN/hYffvghAGvXrqVTp060adOG5cuXM2TIED7++GP69u0b9XnBsSHf+973dnrszDPPZPjw4c0ad0NyqsE8vMoqKFcazb3k4VwzSlFPK4CSkhLmzZsH2MDAtm3b0qpVw7fiefPm8be//S3qY7/4xS8YM2ZMs8QZS4soeUTyT+7OuXTatm0bZWVldOnShVWBMQVlZWWADRKM7JJbW1vb4Mjy/Px82rRp07wBR3vflL9jM4o3eYTLxk/0nvycy25nnHFGg48dc8wx35ZMgl566SXat28f9fpx48Z9OzYklXIqeXi1lXMu062KZ/RymGnTpjEthdVq8cr5No9cHOvhJQ/nXLrlTPKorq7fTTfISx7OOZd8OZM8qqqin/eSh3POJV/OJI/t20P7w4aF9r3k4ZxzyZeTyeM73wnt52LJwznn0i1nkkd4tVVk8giv5qmsDO3n5TV/XM3Bq62cc+mWM8kjvOQxahQEx9hUV8OWLaHHvvgitL/77tlTHZQtcTrnWoaUJg8ROUdEFotIqYi8JyLnRbmmq4hsE5EFYVvMFd7DSx6DB0OPHqHj8Kqr8HnJ+vdv2veRbl7ycC53zZo1iwEDBqQ7jJhSljxEJA8YAoxR1RJgAnCLiPSJuLQb8H+qenDYdkms1w/eULt0sS2wkiNQv9F8xYrQfhb8fr7lJQ/ncsMvfvEL8vPz620iQmlpadTr77zzznrXBqdoLywspLi4mHfffTfF34FJWfJQ1VpVvURVgxO0bAR2AJEtD92AUSLypogsFJHbRaRnvO8zeLB9zeWSh3Mue915552UlZVRVlZGeXk5c+bMoVu3bowePTrq9eeeey5VVVVUVVWxfft2KisrmTFjBl26dOHRRx9ln332SfF3YNI5PcmtwGOq+kXE+VKgt6rWikhb4PfAcyJSonFMbj9kiH1tqOSRC8nDq62cy16FhYUUFhYCUFdXx80338zvfvc7Kisr2bZtGxURo51FhPz8fD777DNmz57NzJkzWbt2LcceeyxDgje8NEhLg7mIXAv0AS6IfExVt6tqbWB/G3AZMBwYHOV1JgXaT74t70VLHuElj/Bqq2xKHl5t5VxuqaqqYtKkSYgIl156KUOHDqVz586cddZZ316zePFiTj31VIYNG8aECRMoKyvjxRdfZPXq1RxwwAF8//vfZ6+99uLqq69OefwpL3kEGr8HASer6o54noIlufLIB1T1buBue90ShcSqrbKpzSOclzyc21k6P2Al8j9ZXV3N008/zeTJkykpKWHu3Lnk5+ezdu1awBrML7nEmnn32msvzj33XPbbbz86RqxqN2XKFKZMmcLy5cvZHt7dNEVSljxEpBVwJ9AZOEVVawLn84AXgWmq+pqInAAsVNU1Yusy/h74j6p+Hc/7NFZttWMHfPVVMB7rqpstvOThXPZbv349++23HwMGDOD2229nwoQJDV47e/ZsLrvssrhfe6+99uLpp59ORphxSWXJYyJwDtam8XrYer3XAgOBLoFjBWaLSAFQB7wHnB7vmwSTR7SSx6pVUFdn+7vtBq1bN+XbSD8veTiXnbp3786SJUto3779t43gkQ4//HCeeeYZhg4dyuGHHx73a+fnp7YiKWXvpqrPYlVQ0Twbdt0zwDNNeY8uXaBzZ9uPVvLI5iorL3k417hs+VDVvn17NmzYQPfwm1SE/v37s2LFim8b1hcsWMAtt9zCwoUL2bBhA/n5+fTr148JEyYwZcoUOgdvfCmUMyPMIVTqgOgN5rnQ08o5lzuWLl1KZWVlve2RRx6pd82CBQs4/PDDGTZsGK+88gplZWWsWbOGmTNn8vbbbzN27FjqglUqKZRTKwkODuuPFZk8VD15OOcyS2Fh4U7rjxcUFNQ7fuGFF+jXrx/XXnvtt+fy8/MZM2YMN910EwcddBBffvkl/VN8U8vZkkdREbRrZ/s1NVBWlr2jy8GrrZzLRRUVFWzdurXeFtkOcuyxx/LFF18wbdo0Vq5cSW1tLVVVVbz11ltcfvnljBo1it3T0Psnp0oekeNlevSArYHx7OvX51bJQ9UTinPZbsSIEVHPh5ciDjroIF577TX+9Kc/ccghh7B+/XratWtH3759mThxIpMnT6ZVq9SXA3ImeQwaBGPH1j/XvTt89pntr1uXW8nDOZe9unXrRhwTZnzroIMOYtasWc0YUeJyptqqUyfo3bv+ufB2j7Vr60/H7snDOeeaLmeSRzThYz3ee8/aPsCSSnFxemJKlmzpluicy005nTzCSx4LF4b2s7XU4W0czrlMkdPJI7zkEZ48sq2nVTRe8nDOpVNOJ4/wksfGjaF9L3k4l/0SaXBuyZrr55TTySO85BEuW5NHOP+/cS1ZQUEBlZWV6Q4jK1RWVu408DAZcjp5NDR1TLZWW3nJwznTo0cPVq9eTUVFhZdAGqCqVFRUsHr1ano09El6F+TMOI9oGkoeuVDycK4l69ChAwBfffUV1dXVaY4mcxUUFNCzZ89vf17J5MkjS/mHLdfSdejQoVluii4+OV1t1aYNtG9f/1ynThCxIFfW8Gor51ymyOnkATs3mudCqQO85OGcS6+cTx6RVVfZnDy85OGcyxQtLnlka08r55zLJDmfPLzayjnnki/nk4dXWznnXPLlfPKILHnkSrWVlzycc+mU88nDSx7OOZd8LSp5FBdD167piyWZvOThnEunnE8effuG9gcN8k/vzjmXDDmfPPbaC04/3UocV12V7mh2jSc+51ymyOm5rcBuuA89ZNU82X7zbd0aqqpsf8sWKCpKbzzOuZYr50seQdmeOABGjAjtv/NO+uJwzrkWkzxyQUlJaL+0NH1xOOdcSpOHiJwjIotFpFRE3hOR8xq47nwRWSoiH4jIsyLSM5VxZqr99w/tL1qUvjiccy5lyUNE8oAhwBhVLQEmALeISJ+I68YCk4HDVHVvoBS4N1VxZjIveTjnMkXKkoeq1qrqJaq6NXBqI7ADyIu49DTgIVVdHzi+DThWRLJ0FY7kGT7cxqoAfPUVrFmT3niccy1XOts8bgUeU9UvIs7vAXwaPFDVzcAWYEDkC4jIpEAVWOn69esjH845eXmw776hY6+6cs6lS1qSh4hcC/QBLoj2MFAbca6GKLGq6t2qWqKqJd0bWnM2x3jVlXMuE6R8nIeI3AwMAk5W1R1RLlkF9Au7vhjoGjjf4nnycM5lglQ2mLcSkb8BuwOnBBOHiOSJyMsicljg0geB08PaOM4H3ghrA2nRIntc+RxXzrl0SGXJYyJwDtZ76nUJjdq7FhgIdAFQ1X+LyF+B+SJSDXwF/DCFcWa0oUOhXTvYuhXWrrWG8z59Yj/POeeSKWXJQ1Wfxdozonk24tq/AH9p9qCyULDR/LXX7Li01JOHcy71fIR5Fgpv9/AeV865dPDkkYW80dw5l26ePLJQeKN5aak3mjvnUs+TRxYaMgTat7f99ethlXdids6lmCePLNSq1c6lD+ecSyVPHlnKk4dzLp08eWQp73HlnEsnTx5ZyhvNnXPp5MkjSw0aBB0DE7hs3Ah33ukJxDmXOp48slSrVjBuXOj4ggvgtNOgrCx9MTnnWg5PHlns1lth5MjQ8T//aVOXvPVW+mJyzrUMnjyy2O67W6L41a9C51asgMMPhwUL0haWc64F8OSR5YqKYOZMePxx6NDBzu3YAb/4BWzfnt7YnHO5y5NHjjjlFFi4ENq2teMlS+Cmm9Ibk3MudzWaPETkpFgvEM81LjWGDoXrrgsdX3cdfPRR+uJxzuWuWCWPv4YfiMgXsa5x6XXBBXDggba/YwdMmgR1demNyTmXe2Ilj8jFmwriuMalUV4e3HMP5AeW+Xr9dTt2zrlkirWSYOSws2jD0HxoWoYZNQouvRRuuMGOL70U5s2zfRHo1QsmT4b+/dMXo3Muu6VyDXOXQldfbeM+li+Hb76BOXPqP/7FF/Dss9Gf65xzscTsbSUiBSLSWkQKI46/PecyT1ER3HsvFESraAReftm78jrnmi5WyaMXUBV2LBHHLoMdfrh12f3gA5v3ShV++1srdVRVWdfeQw9Nd5TOuWwUK3kcmZIoXLMZPNi2oLlz4b77bH/+fE8ezrmmaTR5qOr8VAXiUuOII0LJ49VX4cor0xuPcy47xRokWCgi7SPOFYnIZBG5Q0T+p3nDc8l2+OGh/TfegOrq9MXinMtesRrM/wicHTwQkVbAPGAqcDDwjIgc33zhuWQbMAD69bP9bdvgnXfSGo5zLkvFSh7fBe4NOz4DOBD4jqqWACcDlzVTbK6ZHHFEaP/VV9MXh3Mue8VKHsWquiXs+HfA/6rquwCqOg8Y0EyxuWYSXnU131u1nHNNECt57BCRIgARGQPsDdwRcU3c05MExohcIiLVIvLDBq7ZX0Q2iciCsO3ieN/DxRZe8njtNaitTV8szrnsFKur7ovA30RkLjAdeEpVPw4+KCJ7At8k8H5nY9OZNLZUUTfgCVU9u5Fr3C4YPBh22w3WrIHycnjvPVuB0Dnn4hWr5HE50B+4H/gMmBR8QEQE+DPwj3jfTFXvVNVbgMY+63YDjhaRtwLbdZE9vtyuEfGqK+fcrmk0eajqRlUdq6ptVfW7qrox7DEFTgOua/gVmmQ2MEBVDwImAAOBv0e7UEQmiUipiJSuX78+yWHktvCqK08ezrlE7dJKgqq6RVWTWmOuqpWBxISqbgIuBo4Ptr1EXHu3qpaoakn37t2TGUbOi2z38DU/nHOJaLTNQ0Q+i+dFVHWP5IQTVR42n5ZP45dEe+4J3brBhg2wcaPNgbX33tGvraiATZtg61bbtm2DwkLYfXeb3j0vL7WxO+fSL1aD+QBgNfBP4Fkab6toEhHpCjwN/FRVlwV6Yc1T1TIRyQduAB5UVf9snETBdo/Zs+14/nxLHhs2WCJZtCi0ffyxTaoYTX4+9O4NI0fCn/8MQ4ak7ntwzqVPPMnjZ8BPsfaNB7BxHnGVSOJUjDXKdwwcFwEvi0gd1jNrPnBNEt/PBRxxRCh5TJsGU6daKSQRNTU2S+8XgQWKfY0Q51oG0YY+UoZfZD2rxmOJ5ETgTeAeYLaq7mjWCONUUlKipaWl6Q4jqyxeDPvsE/u6vDzo0QPat4e2bW2rrLSEEd5PoaAA1q2DTp2aL2bnXHKJyKLAjCEJiWslwUAD9ovAiyLSBTgdm5bkDhF5UFV/m+gbu/QbORJGj7YkElRcDMOG2fn997dt9Gg7H01lJYwZA//9r02y+OyzcMYZqYnfOZc+CS9Dq6qbRORVYAhwFnBY0qNyKdGqFbz4Ivz739ClCwwfDn362Pl4FRXBKadY8gB44glPHs61BHHfJkSko4icKyKLsHaIfODIphR3XObo0QNOOw3Gj7feU4kkjqCTTw7tz5tnPbKcc7ktnjXMjxSRh4E1wJnY3Fa9VfU8VfUJvR1Dh4a6+VZVwfPPpzce51zzi7UY1KfY6O51QImqfkdV71fVipRE57JGeOlj1qz0xeGcS41YJY+BQB/gN8D7IlIbsdWJiM/J6uolj7lzrSE96KmnrFvwaafBH/8IL71kgw6dc9krVoP5wDheo3cyAnHZbe+9bYDgsmU2Av3FF+HEE+GVV+AHP7DxIACPPx56Ttu20K5d6GuHDtZw37mzbUOHwi9/aV2AnXOZJVby+AqbOfdUoAa4T1Wvgm/Hfvwam6rde/a3cCJW+rjxRjt+4gnrvXXyyaHEEWnbNtsa8/HHcOutyY3VObfrYlVbTQEOAS4ErgJOCcxkezDwX+D3wC3NG6LLFuFVV08/DccdB2VldrzbbnDHHTBpko0dad06vte84w5YujT5sTrndk2sksePgGNV9QsAEXkPeAkoxNY2H6+qPhe6Aywp9O8PK1fCli22gQ0wfOYZezyors5KHcHJFrdutes3b7btrrvg7bdtlcNLL7XnO+cyR6zk0T6YOAIWBZ4zUlWXN19YLhuJwPe/bxMkhp976KH6iQNsPEn79rZFs+++9hxVG7X+0ktw1FHNF7tzLjGxqq3qTbYdmKak3BOHa0h41RVYG8hJJyX+OvvuCz/9aej4d7/ztdadyyQxSx4icn3EuXaR51T1iuSG5bLVIYdYL6unn7Yb/qWXNv21rr0WHnvM1hN5/324/37rfeWcS79YyWMR1mDe2LnY0/K6FqNVK5gzx274DU2mGK/evWHyZJsqHuCqq6wL70cfwQcfwJo1cNBBNpfWqFG7HrtzLn5xTcmeDXxK9txUUWHjPVavbvy6UaMsiYwebeNCWrcO9eiqqQltbdvC4ME2niQWVasqy094+lDnskezTsnuXLoUF8MNN8BPftL4de+9B5ddFv/rduliSaRPH0sSweRSVWWj3zdutFUVt2+3dpxHH/Uk4lw4/3dwGe/002HhQptwsX9/G82+99626NQTT1g1WVVVYq+5aZN1BY7HE09YYpoxI/HYnctVXm3lsl55OTz5pHXp3bIFduywham2b7euwnl5VmrIy7MxJMuX1597K14PPww//nHy42+JVG2D+JcBWLoU7rkH1q6FAQNgjz1sGzwY+va137VLXFOrrTx5uBanrs4a25cts2Vz8/NtKyiwrUsX6NrVvp51liUmsIWv3nzT2lVakuBYmzlzLOk2dsuoqAgN9Ny82QaC1tRY21Fwq6urnzhKSmw2guOOs2WRw5OAqi1WNmOGTbjZkE6drN1r1CgYMcJ+f+3aheZOq6622Csq7MPFPvtYAnKePDx5uGZRXm49uoJTpAwcCKWl8M03tnri++9bu8mPfmTJJZfU1tpEljfcYN9nKvTubVtdnb1/eTl8/nny36dVK5sqZ/p0WxCtJfPk4cnDNZOlS+HAAy1hgJVOqqvrXzNwoM3DNXFi6uNLtmXLbBr9u+6yKr7mItJ4KSba9ccfD9/9LqxaBZ99Bp9+ar+f4BxqiWrfHq64Ai66CNq0qf+YqiWvNWvs9QsKoLDQevEVFoZKNoWFTXvvTOHJw5OHa0ZPPmlTr8Ry0kk2C3C/fs0fU1Nt2gQzZ9on+k6dbOvcGb780gZ3fvTRzs8pLoZzzrEqJojevqBqpa/glPqdOtnNtaAg1OaUl2ef+kVs27QJXnjBqqSefz76Oi9FRfCzn8FvfmPdtqO976pVsHix9bpbtswS/Tff2Jxp27bZDb+oyL6PzZvhrbfqv0br1hZrUZFttbXWthJP21hBgT23WzcrxfTsaVtRUahtp67Ofh4HHmgl2W7dYr9usqg23h7kycOTh2tmf/gDXHON7XfsaFOoDB9uo+A3bw5dl59vjbhDh9bfhg2zm0q6GnZrauDuu+Hqq+NfjKtTJ/j1r23r2rV546uttcGfVVWhRJOXZ6W6huZAawpVS1QXX5y+GZuHDLG2s9atQ8m0dWv7Gxk92raePS3WdetsstEvvrAkVFxsW9u29pxgQs7Lsw4jH31k31fw6/Dh9qGgIZ48PHm4FFi2LHRDCyaB9ettJPz998d+fvv2doM4+GAYO9ZWWGyOT6Fbt9on72Aj8eefW/XMBx/Efm5RERx9NJxwgi3k1aFD8uPLBNXV1nvr+usbHoRaVGTLCXTtasl3xw7bqqpCP+OG1qvZVd262Xsk2g090oABjbcbefLw5OHS7PXX4be/tQb1RIwcaY3u4YLVOsFPpe3a2U0guHXrZqWdjRttW7vW6v8//dTaKTZsaPw9Bw6ECy6wG19Zmb1WQQGMHw/jxu361DLZRDWUZCsrbROBXr3s5x6rpLh9u7WNbNhgpYSvv7Zt+/b6VXQrV1pvvXfe2bnNrDnl5VkSimzTCfLk4cnDZYjycruBf/KJbR9/HNovL09vbG3bwpVXWpJr6GbimldVlfXU+/xzq4YKdl3eutV6tQXbbioq7PpOnWxwbL9+1jhfURFahbO6OtT9ubbWfqfDhllV1Z572tehQxv/MODJw5OHy3DB+uvFi2H+fBu/sHBh81R7FBbaTSfYAFxUZA21V11lXWFdZquttaq0Tp2av9owK+a2EpEC4DfADcCZqvpolGsEW972VKAWeAc4R1VjrHbtXGYTsUbQo4+2DezT5qJFoU+ZEOqhE+ylU1dnVUsrVtj2+edWzRQczNi1q1VjBUdbDx5sCSLekdsu8+TlZXaPPUj93FZnY1O4L2jkmrOAicA+qlopIvcDNwEXpCA+51KqXTtrNHcu26T0s4mq3qmqt2AlioacBtylqsEe1rdha6k755zLEJlYsN0D+DTs+FOgi4h0jLxQRCaJSKmIlK5fvz5lATrnXEuXiclDqF8yCTYn7hSrqt6tqiWqWtK9e/eUBOeccy4zk8cqILypqB+wFWji7DXOOeeSLe3JQ0S6isgbIjIkcOpB4JciElhElAuB2ZorfYqdcy4HZMJKgsVAfyDYpvF3YDDwtojUAEvwnlbOOZdR0pI8VHVs2P6XQN+w41rgysDmnHMuA6W92so551z28eThnHMuYZ48nHPOJcyTh3POTZuW2HFD51oQTx7OuezSlBt9Y95+G6ZPh7//HR54wFb1mj7d1h5+6SVYsMCOy8rqL7o+ffquxdWUuDMpYalqTmz777+/OudaAEjsWFV16tSdj197TfV//id8EuPYW1GR6qBBqocdZseTJqlecYXqjBl2PHu26iuvqL7zjh1v357cuCOvifZ9NXYcBVCqTbjn+noezrn4TZtW/9Nv5HGiz0/kms8+g0susRLB3nvblMTt2lnp4PjjQ0v2PfUUnHWWLTTfoYN9vfRSeOghu14VTjop/ph3Vdeutizhhx/aSk3ffBPaBgywdW579bLv6ze/sYXJCwpsmz7d1g+urbXt5pth5kzo0QO6d4fDD7elJDt3Dn3/4ff0yOMomrqeR9pLDMnavOThsloSPkGm5D1jfVqO9Rqxro92zVVXqR51VGIlhHi2Dh1Ur7lGdfPm+EsF5eWqS5da6QJUJ05MflxN2fLzVXv1sv199lE98EDVMWPs+Lnndv4Z1/vWmlbySPtNP1mbJw+XlerqVN980/4VH3tM9R//UH3oITv+7DPVmhq7LtGQHYT4AAAUXElEQVSbdEPnwsVzw6yqsqqYk09Wbd/ezpWUqJ54ouq559rxww9bFdDKlXZcV1f/NRYvVr3rLtWf/cyOzzhD9fLLVWfOtON771W97jrVCy9UPe00Ozdtmurdd6s++mj9m+SPfxx6zTfeUH3hBTueM0f1ySct1uBrHnNMfDfeqVPj+1k05edXU6P69deq775rx0uWqH75pWpZmR0vW2Y/u8cft+MZMxpOlOPG2df99mtagmng78GThycPlyniLSVUV6ued17j//CFhaojRtj+r3+tevPNoRvNV1+p1tbaa8VzcwvG9cEHqpMn2+M//7nqBReoXnaZHV9zTWgD1Y4dE79J5eerduqk2rdv025y0baBA1XnzYv+fSXjRt+UUlisUtWuHifynKoqS0pg7S0LFqi++qodr1698+vWewlPHo3+gJxLmVg3JlX75D1kyK7fVAsKVPfYw/aPOUb1hBNUTzlF9cwz7dxdd6n+61+qH31kx717N+19xo9XXbHC9t98U3XWLNXbbrPjvfaK7zWOPda+nnhi9MePPjpU6go2SEf79Nwc1W3JkGhcTYl7VxNWFJ48PHm4dJk61aonXnlF9Ve/sn+r889XvfVW1blz7biiInT96tWhm2GXLlZt0dA/fXm5amlp6AaejE/xHTpYLyFQveceSwI33GDH06aFNlD98MPocTV2vH276saNoWSTyPPjvSZRqWhDag6x4kxjb6u03/STtXnycGmxYIH9GxUXx75pd+2qOnq0as+edjxokOonn9jrNLWKY9s21Y8/tuPnnrN6/0cfVb3/fjs3enT0WJpaz59og3iin5zjvcYljScPTx4uVaZOVV24MFQNE9wGD7Y+/8EqmHhLAk2piknGJ/jm6OGV6GvE8x7ZUkrIUk1NHj7Ow7lEvPcejB7d8ONTp1rf/PD/KxHro79uHaxaBVu3wpFH1r8mUfGMt4g8F0eff9fyNHWcRyYsBuVc8iQ6aC0RDz9sg88Aiorgwgtt8Fn37rFvyq1a2UCwXr2SE0tT5l6aOjU57+0cPreVyybx3CBjzTcU6/kNXX/CCXDGGVaCAKishD/+Ee64I/ZrRrtpp+NGnknzIrms59VWLnuIWNVP+/ZQWGjnWrWC556DDz6ATz6B//1fePBB2GsvGD4c2ratXyoIlkxU4csvoX9/+PxzmyKisDB61c6ECTBvnu3fcANcfnn013QuCzW12sqTh8t8dXXwk59YtVFQcO6fiorYzz/ySBg8GAYNgilT4LvfhYULLRGF69YNNmyAH/zA5iAaOhSWL4c//MEev+02+PWvve3A5RRv83CZJVmfxqdN27kqCqC62rZwEybA88/DnnvCRx+Fzv/737YFzZ0b/b02bLCvs2bt/Ng998Avf2n73nbgnLd5uGYS7YbflPULDjggNFsohDq4VlXZ+grh5557zo6XLLHj7dvt+PTTo7/2NdeEnl9TA2vW2PFDD9lspeHOPtti8Coq5wCvtnLN4ZVXYNw4+NGPYPfdbevXD0480W7SeXl2XWT1T+SN+cILbYGe8nK47jq48sqdq4tivUasKarjmcLaq6lcDvNqK5d+kVVM//jHzte0a2cN2SNG2PHjj1vbwpAh9tzgjb+8PNST6eSTrZF6x46dXy+yCinR7qmRj3uVlHNx8ZKHS66LL4YZM2z/gQfg//2/+u0NsbRrZyWT2lobTDdihC0D2q5dcuJrSrWTV1W5HOa9rTx5pN/778O++1rvqGA7RDgRa6dYssRWVTv7bOvV9PHHsV976lS/gTvXDLzayqWXKpx/vpUYzjvPRl1H07EjHHKIbWefDUuX2vnaWsjPhy1bbL+uzrrO5siHG+dyjScPlxzf/z689poljWuvtTWVIzXWvhBsRO/QoflidM4lTUq76orIESLyjoi8JyKlInJwlGv2F5FNIrIgbLs4lXG6BJWVwZw5tv+nP0VPHBC7q643XjuXNVJW8hCRTsBs4DhVfVNExgJPichAVQ0fJtwNeEJVz05VbG4XBW/yhx5qI8GbqinjQJxzaZHKkscxwMeq+iaAqv4HWAOMi7iuG3C0iLwV2K4TkfYpjNPFa9o0awT/y1/s+PXXba4pv+k7l/NS2eaxB/BpxLlPA+fDzQYeUVUVkS7AHcDfgZOaP0SXkGnToLgYJk+2Y2/cdq7FSGXyEKA24lwNEaUfVa0M298UaO/4UkSKwh8DEJFJwCSAfv36NUvQrhG1tXDnnemOwjmXBqmstloFRN7h+wXONyYPqAK2Rz6gqneraomqlnRvqGuoaz7PPgsrV9pstcF5opxzLUIqk8dTwCgRGQkgIgcCw4FXROQNERkSOP/DQOM6IpIP3AA8qKp1KYzVxeP22+3r+edHnwjROZezUlZtpapbROQU4D4RUazKaiJQDPQHOgYuLQJeFpE6QIH5gH+szTRLlsDLL1ubx89+lu5onHMpltJBgqr6b+CAKA/1DbvmfuD+lAXloos1n1Nw0sIzz4ROnVIRkXMug/h6Hi66xtYC37LFlnsFuOCClIXknMscPj2Jq2/pUrj+etv/1a9stHjnzpZMxoyBPn3g6adtFb8jj4S9905vvM65tPDk4axUMXUqfO97lhiC7rqr/nVHH13/2EsdzrVYXm3VEsQa8T19uq36F0wcwWVbJ05s/HknnxxamtU516J48sh1qg23X6xYEWr4fuwxaNvWln196CE7N3du/XU5gvuRx548nGtxPHlko3gnEHznndByr/37w7HHwkUXWTLp0QMGDrR1woO2bYNPAzPI+Iy2zrlG+EqC2aS6Gp55xqqLNm6ELl3svEj9eaWmTrWlW198sfHX69DB2jFmzYKqKigsbPjayK67vjSrczmhqSsJeskjm/z+95Y4ALp2hV69YFxgUuLbb7dk8cEHdl0wcZx/vn2N1rhdXh4qmTSWOMCnS3fO1eMlj2wxeTL88Y/xX9+5M9x3n/WgiiyZRB57KcK5FstLHrmuTRv7On68fa2ttQbvuXPteL/96l+/eTOcdFKoG25jPHE45xLk4zyywZYtcNtttn/11fCvf9miS/372wawaFHo+siSRSRvDHfO7SIveWSD22+3BDJ2LBx22K6v9e0lDefcLvLkkem++QZmzLD94JoZsRqvvWThnGtmnjwy3UknWfvFmDFW8oiHlyycc83Mk0cm27bN1swAK3WIpDce55wL8OSRaYKlhjVrbFZbgIMOCvWycs65DODJI9NMnw6HHAK9e4fmmHrrLetd5dVRzrkM4V11M4Uq3HCD7S9YYF9POgmefLLxbrfOOZcGXvLIBFdfbSWLK6+sf37UqPTE45xzMXjJI90uvxyC06oUF0NFhZc0nHMZz+e2SqfVq6FvX9vv0cOmGjngAE8ezrmUaercVl7ySJft20PjNoYOheefhz328AF+zrms4G0e6TBtmk10uHy5HX/yCQwa5LPbOueyhpc80uHqq+Hhh0PJw6upnHNZxkse6TBrliWOPfZIdyTOOdckXvJINVW48Ubbv+wyG0nunHNZxpNHqr3wArz7ri0he9ZZoUWenHMui3i1VaoFR5H/9reeOJxzWSulyUNEjhCRd0TkPREpFZGDo1wjIvIHEflYRJaIyEMi0jbmi3/1Vf3jWGtexHPNrh5Hnvu//4NXX4WOHUOTHjrnXBZK2SBBEekEfAocp6pvishY4DFgoKpWhF33U+BC4FBVrRSR+4FtqnpBY69fIlJ/kGBJSWjkdrTjeK7Z1ePIc1deadVWV1wB113X2LfjnHMp0dRBgqlMHqcBv1HV74Sdexe4WlWfCTv3PPCkqt4dON4HeFlVuzb2+iUimjXjy7/+2kaUO+dcmmXDCPM9sJJHuE8D5xu77lOgi4h0VNUt4ReKyCRgEkBXIOHvPl169gTga1izCr6KcXW6dAM2pDuIOGRDnNkQI3icyZYtcQ5rypNSmTwEqI04V8PO7S6R19UEvu7UPhMonQRLKKUbmpA9U01ESpuS5VPN40yebIgRPM5ky6Y4m/K8VDaYrwL6RZzrFzjf2HX9gK1AWfOF5pxzLhGpTB5PAaNEZCSAiBwIDAdeEZE3RGRI4LoHgV+KSOvA8YXAbM2V6X+dcy4HpKzaSlW3iMgpwH0iolh11ESgGOgPdAxc+ndgMPC2iNQAS4BGe1oF3J38qJuFx5lc2RBnNsQIHmey5XScObOeh3POudTxEebOOecS5snDOedcwrI+ecQz5Uk6iEiBiFwiItUi8sOw8xkVr4icIyKLA7G8JyLnBc7vHejI8J6IvC8ix6U5zj8E4nw78PM7NxPjDBKRESKySUSmBY4z7ff+vogsFJEFge2VTIwzENPFIvJBIN5SESnOlDhF5MKwn2FwKw/ElxExhsV6aSCWtwJfrwicb9r/kKpm7QZ0AjYChwSOxwJfA8UZENt5wMXAa8APMzFeIA+4GWgXOO4DVAa+LgdOC5zfE9gE9E3jz/N3QEFgvzvWfXtQpsUZ9nv+D/BXYFqm/d4DMawAWkWJO9PinA7cCOSHxdgl0+IMi3c3YGXgfyhjYgQOA1YDXQPHXQJ/A8c09X8o20sexwAfq+qbAKr6H2ANMC6dQQGo6p2qegv1BzxmVLyqWquql6jq1sCpjcAOrLdbEfB44LqPgFeBk9IRZyCGGapaHTgcgCWPvmRYnCLSCngAuAJYHzidUb/3gC7AfBH5r4g8HpgGKKPiDMyHdwzwDfCGiMwHDgHGZ1KcEaZgHxoOJbNiDP4tdg587YDVPFXRxP+hbE8e8U55kikyPd5bsckqewOfaeCjSEDa4xSRISKyDHgBOIPMjPNa4F+q+n9h5zLx995TVQ8D9sfGYL2EjbvKpDj3B0YAq1T1IOAi4BHsw00mxQmAiOwGfB+4kwz7navqUmwqp4Ui8gnwLvbzbPL/ULYnj3inPMkUGRuviFyLFbUvIEPjVNVlqjoE+/T2d+wPPGPiFJGTgX6qekfkQ2RQnACqWhn4WqeqDwOLAg9lUpw9gBWq+gCAqv4XeDnwWCbFGXQ58NdAST6jfuciMhT4GzBeVYdiUwFeDwykiXGm+4e9q+Kd8iRTZGS8InIz9gnvZFXdQYbGGRS4ibyJtdlkUpwTgD2DDafALwPbdWRWnNHkYXXgmRTnOqA84lwdmfd7R0R6Y6WO4AeHTPsfOgF4XdUmH1fV5cCTwFE0Nc50NzDtYiNQR2zWypGB4wOBzUCndMcWFuN/CDWYZ1S82IeHv2FVVflh5/OwouuEwPEeWCPakDTFORI4ldCg1j6BP+4xmRRnlLinBbZM+70fAOwfdjwRa0ztn2FxtsbaCQ4NHA8LxDc6k+IMxHA7cEXYcab9zk8AvgR2Cxx3AN4Bzm/q/1BWr2GuDUx5oqoZOYliBsY7ETgHKAVeF5Hg+auwBrO/isjvA+fOUtVlqQ8RsN4rvwImi0g1dlO5WlXfEJFMijOqDPy9bwVmiEgvYDt2szhaVVdmUpyqukNEJgL/G+iIUIf9fhdnUpwi0gc4mbCpzTPtd66qT4vIYGCeiFRi00LNxT48vkYT/od8ehLnnHMJy/Y2D+ecc2ngycM551zCPHk455xLmCcP55xzCfPk4ZxzLmGePFxOEZFbRUQjtneb+Fr/EZFLEnzONBGZE9hfISI/aMp7NxcJ648diPXZdMbjsldWj/NwLigw3XUv4HVgQcTDdSLyPaBWVZ8JXH8jMDnium2q2i7B912L9Yt/IY5r9wbeb+SSD1V170ae/xKNT6x3g6oGp9neH3gr4vFW2KjtXrFidS4WTx4uV+wV2BpTDTwT2L8CGwwZdAQ2QSAishybuTcPiPXJvDU2M2kiugHbopyvi/G842j4f/aBiNf8LzZ9efhrnwd8L84YnWuUJw+XE1T1vsAo5LOBH2ATvuVhayjMB25W1fVhTxkMDA07Hhn2WoPh20/6DQpUARURmu46XttVNdGEQ2PPEZE22NTlwWvrsJHk4dccCryR6Ps6F40nD5dLpgBnAT/DqoeqsRLEH4F/YJPABf0YmBrx/GilgcYMB9oAxSJyEfDnwPmnYjyvLKzpIdy9qnpOIgGISAG2RkNvrEqqoet6AsdikzQ6t8u8wdzlknxseukdga91WALZgZVCIr2M3XiDWx+waisRqSH2wj0HYysvnoEtANQeuCGOODupan6UrdHEISJrRaQmbFMs4b2NTRb4dSNPvwYoVdWFccTnXExe8nC55Hpsor+ZWLVVPrAWm9n4N1Gur8WqdvKwEkQXERkJnKuq/xKR/8R4v1OxNRwuBf6kqqtFZEcSvo+GDMLWiQBQoDoweWBb7Pv4ItqTRGQCNjX8Yc0Ym2thPHm4nKGqNcBNgS0eR2Mlkyrs5luGTU/9FPCvxp4oIt/Bptk+BegJPCIiRzX2nDDfNFBtBVDUUNuGqjZUrTYQSyZfRonzSGyJ0Smq+nac8TkXkycPl/VE5ENsLYp6p7FppyuwG2u4H6rqNGytjaa8X1fgQeD3qrpVRG7BGupvo5F2B1X9gFDJIZmGA1+oLeQVjLEAKxFNxaav/3NDT3auKTx5uKynqiMAAutTlKlqVWDtgmXAMFVtcFW04PgQVZ0T5eEHgaVRzs8APsIWAEJVN4rIT7A1t5P+PyUi+TFedzSwPNDjCqw0dTa2ZvXxqvpismNyzpOHyyWl2Brsc7DG8m3EHjtxLNbwvVPyUNV7G3jOJKBVoDts8NrngedFZFrkxSIyAPg8ZvQhb6nqwWHHV7Fzz7BoKgNfz1XVO0XkvqZ0CXYuHp48XK5pKyKdsNXx+gIEjsEamKO1G+SJSEMjy2sib8Cquj3BmFZi40HiVS/hNbWKzROHa06ePFyueaiRx17AShqRjiJsgF2El6k/PiRhast1+o3c5RRfhtY551zCfJCgc865hHnycM45lzBPHs455xLmycM551zCPHk455xLmCcP55xzCfPk4ZxzLmGePJxzziXs/wOf/bshXcFG8wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "polynomial_regression = Pipeline([\n",
    "        (\"poly_features\", PolynomialFeatures(degree=10, include_bias=False)),\n",
    "        (\"lin_reg\", LinearRegression()),\n",
    "    ])\n",
    "\n",
    "plot_learning_curves(polynomial_regression, X, y)\n",
    "plt.axis([0, 80, 0, 3])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 훈련 데이터의 오차가 선형 회귀 모델보다 훨씬 낮음\n",
    "+ 두 곡선 사이에 공간이 있음. 훈련 데이터에서의 모델 성능이 검증 데이터에서보다 훨씬 낫다는 뜻으로 과대적합 모델의 특징임. 그러나 더 큰 훈련 세트를 사용하면 두 곡선이 점점 가까워짐\n",
    "+ 과대적합 모델을 개선하는 한 가지 방법은 검증 오차가 훈련 오차에 근접할 때까지 더 많은 훈련 데이터를 추가하는 것"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### ▣ 편향/분산 트레이드오프\n",
    "\n",
    "※ 모델의 일반화 오차는 세 가지 다른 종류의 오차의 합으로 표현할 수 있음\n",
    "+ **편향**  \n",
    "일반화 오차 중에서 편향은 잘못된 가정으로 인한 것임. (예를 들어 데이터가 실제로는 2차인데 선형으로 가정하는 경우) 편향이 큰 모델은 훈련 데이터에 과소적합되기 쉬움\n",
    "+ **분산**  \n",
    "분산<sup>variance</sup>은 훈련 데이터에 있는 작은 변동에 모델이 과도하게 민감하기 때문에 나타남. 자유도가 높은 모델(예를 들면 고차 다항 회귀 모델)이 높은 분산을 가지기 쉬워 훈련 데이터에 과대적합되는 경향이 있음\n",
    "+ **줄일 수 없는 오차**  \n",
    "줄일 수 없는 오차<sup>irreducible error</sup>는 데이터 자체에 있는 노이즈 때문에 발생함. 이 오차를 줄일 수 있는 유일한 방법은 데이터에서 노이즈를 제거하는 것(예를 들어 고장 난 센서 같은 데이터 소스를 고치거나 이상치를 감지해 제거함)\n",
    "\n",
    "※ 모델의 복잡도가 커지면 통상적으로 분산이 늘어나고 편향은 줄어듬. 반대로 모델의 복잡도가 줄어들면 편향이 커지고 분산이 작아짐."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 4.5 규제가 있는 선형 모델"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5.1 릿지 회귀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▣ **릿지 회귀**(또는 **티호노프**<sup>Tikhonov</sup> 규제) : 규제가 추가된 선형 회귀\n",
    "+ 규제항 : 가중치 벡터의 $l_2$ 노름의 제곱을 2로 나눈 것을 사용\n",
    "+ $\\alpha$로 모델을 얼마나 많이 규제할지 조절\n",
    "+ 규제항은 훈련하는 동안에만 비용 함수에 추가됨\n",
    "+ 모델의 훈련이 끝나면 모델의 성능을 규제가 없는 성능 지표로 평가\n",
    "\n",
    "![Equation4-8](./images/Equation4-8.png)\n",
    "**<center>식 4-8 릿지 회귀의 비용 함수</center>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "![Figure4-17](./images/Figure4-17.png)\n",
    "**<center>그림 4-17 릿지 회귀</center>**\n",
    "+ 왼쪽 그래프는 평범한 릿지 모델\n",
    "+ 오른쪽 그래프는 PolynomialFeatures(degree=10)을 사용해 데이터를 확장하고 StandardScaler를 사용해 스케일을 조정한 릿지 모델\n",
    "+ $\\alpha$가 커질수록 모든 가중치가 거의 0에 가까워지고 결국 데이터의 평균을 지나는 수평선이 됨  \n",
    " → 모델의 분산은 줄지만 편향은 커짐"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "![Equation4-9](./images/Equation4-9.png)\n",
    "**<center>식 4-9 릿지 회귀의 정규방정식</center>**\n",
    "+ A는 편향에 해당하는 맨 왼쪽 위의 원소가 0인 $(n+1)\\times(n+1)$의 단위행렬<sup>identity matrix</sup>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.linear_model import SGDRegressor\n",
    "\n",
    "np.random.seed(42)\n",
    "m = 20\n",
    "X = 3 * np.random.rand(m, 1)\n",
    "y = 1 + 0.5 * X + np.random.randn(m, 1) / 1.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.55071465]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 정규방정식을 사용한 릿지 회귀\n",
    "ridge_reg = Ridge(alpha=1, solver=\"cholesky\", random_state=42)\n",
    "ridge_reg.fit(X, y)\n",
    "ridge_reg.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ solver 매개변수의 기본값은 'auto'며 희소 행렬이나 특이 행렬(singular matrix)이 아닐 경우 'cholesky'가 됨"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.13500145])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 확률적 경사 하강법을 사용한 릿지 회귀\n",
    "sgd_reg = SGDRegressor(max_iter=5, penalty=\"l2\", random_state=42)\n",
    "sgd_reg.fit(X, y.ravel())\n",
    "sgd_reg.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.5507201]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_reg = Ridge(alpha=1, solver=\"sag\", random_state=42)\n",
    "ridge_reg.fit(X, y)\n",
    "ridge_reg.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 확률적 평균 경사 하강법(Stochastic Average Gradient Descent, SAG) - SGD의 변종  \n",
    ": 현재 그래디언트와 이전 스텝에서 구한 모든 그래디언트를 합해서 평균한 값으로 모델 파라미터를 갱신"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.5.2 라쏘 회귀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▣ **라쏘**<sup>Least Absolute and Selection Operator</sup>(Lasso) **회귀** : 선형 회귀의 또 다른 규제된 버전\n",
    "+ 규제항 : 가중치 벡터의 $l_1$ 노름을 사용\n",
    "![Equation4-10](./images/Equation4-10.png)\n",
    "**<center>식 4-10 라쏘 회귀의 비용함수</center>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "![Figure4-18](./images/Figure4-18.png)\n",
    "**<center>그림 4-18 라쏘 회귀</center>**\n",
    "+ 덜 중요한 특성의 가중치를 완전히 제거하려고 함(즉, 가중치가 0이 됨)\n",
    "+ 자동으로 특성 선택을 하고 **희소 모델**<sup>sparse model</sup>을 만듬(즉, 0이 아닌 특성의 가중치가 적음)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "![Figure4-19](./images/Figure4-19.png)\n",
    "**<center>그림 4-19 라쏘 및 릿지 규제</center>**\n",
    "+ 파란 배경의 등고선(타원형) : 규제가 없는($\\alpha$=0) MSE 비용 함수\n",
    "+ 하얀색 원 : 비용 함수에 대한 배치 경사 하강법의 경로\n",
    "+ 무지개색 등고선 : $l_1$(다이아몬드형, 왼쪽 위), $l_2$(타원형, 왼쪽 아래) 페널티에 대한($\\alpha$→**∞**) 배치 경사 하강법의 경로\n",
    "+ 오른쪽 위 그래프 : $\\alpha$=0.5의 $l_1$ 페널티가 더해진 비용 함수 → 라쏘 회귀의 비용함수\n",
    "+ 오른쪽 아래 그래프 : $\\alpha$=0.5의 $l_2$ 페널티가 더해진 비용 함수 → 릿지 회귀의 비용함수  \n",
    "※ 규제가 있는 경우의 최젓값이 규제가 없는 경우보다 $\\theta$값이 0에 더 가까움(가중치가 완전히 제거되지는 않음)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "+ 라쏘의 비용 함수는 $\\theta_i$=0($i$=1~n)에서 미분 불가능  \n",
    "→ **서브그래디언트 벡터**<sup>subgradient vector</sup> **g**를 사용하여 해결\n",
    "![Equation4-11](./images/Equation4-11.png)\n",
    "**<center>식 4-11 라쏘 회귀의 서브그래디언트 벡터</center>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.53788174])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "# 정규방정식을 사용한 라쏘 회귀\n",
    "lasso_reg = Lasso(alpha=0.1)\n",
    "lasso_reg.fit(X, y)\n",
    "lasso_reg.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.13498188])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 확률적 경사 하강법을 사용한 라쏘 회귀\n",
    "sgd_reg = SGDRegressor(max_iter=5, penalty=\"l1\", random_state=42)\n",
    "sgd_reg.fit(X, y.ravel())\n",
    "sgd_reg.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.5.3 엘라스틱넷"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▣ **엘라스틱넷**<sup>Elastic Net</sup> : 릿지 회귀와 라쏘 회귀를 절충한 모델\n",
    "+ 규제항은 릿지 회귀와 라쏘 회귀의 규제항을 단순히 더해서 사용\n",
    "+ 혼합 정도는 혼합 비율 r을 사용해 조절\n",
    "    + r=0 → 릿지 회귀\n",
    "    + r=1 → 라쏘 회귀\n",
    "![Equation4-12](./images/Equation4-12.png)\n",
    "**<center>식 4-12 엘라스틱넷 비용 함수</center>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.54333232])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import ElasticNet\n",
    "\n",
    "elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5, random_state=42)\n",
    "elastic_net.fit(X, y)\n",
    "elastic_net.predict([[1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "▣ **보통의 선형 회귀, 릿지, 라쏘, 엘라스틱넷**\n",
    "+ 적어도 규제가 약간 있는 것이 대부분의 경우에 좋으므로 일반적으로 평범한 선형 회귀는 피해야 함\n",
    "+ 릿지가 기본, 실제로 쓰이는 특성이 몇 개뿐이라고 의심되면 라쏘나 엘라스틱넷이 좋음(불필요한 특성의 가중치를 0으로 만듦)\n",
    "+ 특성 수가 훈련 샘플 수보다 많거나 특성 몇 개가 강하게 연관되어 있을 때는 보통 라쏘보다 엘라스틱넷이 선호됨(라쏘는 특성 수가 샘플 수(n)보다 많으면 최대 n개의 특성을 선택함. 또, 여러 특성이 강하게 연관되어 있으면 이들 중 임의의 특성 하나를 선택함)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.5.4 조기 종료"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▣ **조기 종료**<sup>early stopping</sup> : 검증 에러가 최솟값에 도달하면 바로 훈련을 중지시키는 방식\n",
    "![Figure4-20](./images/Figure4-20.png)\n",
    "**<center>그림 4-20 조기 종료 규제</center>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(287,\n",
       " SGDRegressor(alpha=0.0001, average=False, early_stopping=False, epsilon=0.1,\n",
       "        eta0=0.0005, fit_intercept=True, l1_ratio=0.15,\n",
       "        learning_rate='constant', loss='squared_loss', max_iter=1,\n",
       "        n_iter=None, n_iter_no_change=5, penalty=None, power_t=0.25,\n",
       "        random_state=42, shuffle=True, tol=None, validation_fraction=0.1,\n",
       "        verbose=0, warm_start=True))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.base import clone\n",
    "\n",
    "X_train, X_val, y_train, y_val = train_test_split(X[:50], y[:50].ravel(), test_size=0.5, random_state=10)\n",
    "\n",
    "poly_scaler = Pipeline([\n",
    "        (\"poly_features\", PolynomialFeatures(degree=90, include_bias=False)),\n",
    "        (\"std_scaler\", StandardScaler()),\n",
    "    ])\n",
    "\n",
    "X_train_poly_scaled = poly_scaler.fit_transform(X_train)\n",
    "X_val_poly_scaled = poly_scaler.transform(X_val)\n",
    "\n",
    "sgd_reg = SGDRegressor(max_iter=1, warm_start=True, penalty=None,\n",
    "                       learning_rate=\"constant\", eta0=0.0005, random_state=42)\n",
    "\n",
    "minimum_val_error = float(\"inf\")\n",
    "best_epoch = None\n",
    "best_model = None\n",
    "for epoch in range(1000):\n",
    "    sgd_reg.fit(X_train_poly_scaled, y_train)  # 이어서 학습합니다\n",
    "    y_val_predict = sgd_reg.predict(X_val_poly_scaled)\n",
    "    val_error = mean_squared_error(y_val, y_val_predict)\n",
    "    if val_error < minimum_val_error:\n",
    "        minimum_val_error = val_error\n",
    "        best_epoch = epoch\n",
    "        best_model = clone(sgd_reg)\n",
    "\n",
    "best_epoch, best_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**※** warm_start=True로 지정하면 fit() 메서드가 호출될 때 처음부터 다시 시작하지 않고 이전 모델 파라미터에서 훈련을 이어감  \n",
    "◈ SGDRegressor : https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDRegressor.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.6 로지스틱 회귀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▣ **로지스틱 회귀**<sup>Logistic Regression</sup>(또는 **로짓 회귀**<sup>Logit Regression</sup>) : 샘플이 특정 클래스에 속할 확률을 추정  \n",
    "e.g.) 이메일이 스팸일 확률(이진 분류기)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.6.1 확률 추정"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Equation4-13](./images/Equation4-13.png)\n",
    "**<center>식 4-13 로지스틱 회귀 모델의 확률 추정(벡터 표현식)</center>**\n",
    "**※** $\\sigma(·)$ → logistic 또는 logit이라고 부름  \n",
    ": 0과 1사이의 값을 출력하는 **시그모이드 함수**<sup>sigmoid function</sup> (S자 형태)\n",
    "![Equation4-14](./images/Equation4-14.png)\n",
    "**<center>식 4-14 로지스틱 함수</center>**\n",
    "![Figure4-21](./images/Figure4-21.png)\n",
    "**<center>그림 4-21 로지스틱 함수</center>**\n",
    "![Equation4-15](./images/Equation4-15.png)\n",
    "**<center>식 4-15 로지스틱 회귀 모델 예측</center>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.6.2 훈련과 비용 함수"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Equation4-16](./images/Equation4-16.png)\n",
    "**<center>식 4-16 하나의 훈련 샘플에 대한 비용 함수</center>**\n",
    "\n",
    "\n",
    "![Equation4-17](./images/Equation4-17.png)\n",
    "**<center>식 4-17 로지스틱 회귀의 비용 함수(로그 손실<sup>log loss</sup>)</center>**\n",
    "\n",
    "\n",
    "![Equation4-18](./images/Equation4-18.png)\n",
    "**<center>식 4-18 로지스틱 비용 함수의 편도함수</center>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.6.3 결정 경계"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['feature_names', 'target_names', 'target', 'filename', 'data', 'DESCR']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "list(iris.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((150, 4), (150,))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris.data.shape, iris.target.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _iris_dataset:\n",
      "\n",
      "Iris plants dataset\n",
      "--------------------\n",
      "\n",
      "**Data Set Characteristics:**\n",
      "\n",
      "    :Number of Instances: 150 (50 in each of three classes)\n",
      "    :Number of Attributes: 4 numeric, predictive attributes and the class\n",
      "    :Attribute Information:\n",
      "        - sepal length in cm\n",
      "        - sepal width in cm\n",
      "        - petal length in cm\n",
      "        - petal width in cm\n",
      "        - class:\n",
      "                - Iris-Setosa\n",
      "                - Iris-Versicolour\n",
      "                - Iris-Virginica\n",
      "                \n",
      "    :Summary Statistics:\n",
      "\n",
      "    ============== ==== ==== ======= ===== ====================\n",
      "                    Min  Max   Mean    SD   Class Correlation\n",
      "    ============== ==== ==== ======= ===== ====================\n",
      "    sepal length:   4.3  7.9   5.84   0.83    0.7826\n",
      "    sepal width:    2.0  4.4   3.05   0.43   -0.4194\n",
      "    petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)\n",
      "    petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)\n",
      "    ============== ==== ==== ======= ===== ====================\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "    :Class Distribution: 33.3% for each of 3 classes.\n",
      "    :Creator: R.A. Fisher\n",
      "    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n",
      "    :Date: July, 1988\n",
      "\n",
      "The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken\n",
      "from Fisher's paper. Note that it's the same as in R, but not as in the UCI\n",
      "Machine Learning Repository, which has two wrong data points.\n",
      "\n",
      "This is perhaps the best known database to be found in the\n",
      "pattern recognition literature.  Fisher's paper is a classic in the field and\n",
      "is referenced frequently to this day.  (See Duda & Hart, for example.)  The\n",
      "data set contains 3 classes of 50 instances each, where each class refers to a\n",
      "type of iris plant.  One class is linearly separable from the other 2; the\n",
      "latter are NOT linearly separable from each other.\n",
      "\n",
      ".. topic:: References\n",
      "\n",
      "   - Fisher, R.A. \"The use of multiple measurements in taxonomic problems\"\n",
      "     Annual Eugenics, 7, Part II, 179-188 (1936); also in \"Contributions to\n",
      "     Mathematical Statistics\" (John Wiley, NY, 1950).\n",
      "   - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.\n",
      "     (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.\n",
      "   - Dasarathy, B.V. (1980) \"Nosing Around the Neighborhood: A New System\n",
      "     Structure and Classification Rule for Recognition in Partially Exposed\n",
      "     Environments\".  IEEE Transactions on Pattern Analysis and Machine\n",
      "     Intelligence, Vol. PAMI-2, No. 1, 67-71.\n",
      "   - Gates, G.W. (1972) \"The Reduced Nearest Neighbor Rule\".  IEEE Transactions\n",
      "     on Information Theory, May 1972, 431-433.\n",
      "   - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al\"s AUTOCLASS II\n",
      "     conceptual clustering system finds 3 classes in the data.\n",
      "   - Many, many more ...\n"
     ]
    }
   ],
   "source": [
    "print(iris.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((150, 1), (150,))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = iris[\"data\"][:, 3:]  # 꽃잎 넓이\n",
    "y = (iris[\"target\"] == 2).astype(np.int)  # Iris-Virginica이면 1 아니면 0\n",
    "\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='warn',\n",
       "          n_jobs=None, penalty='l2', random_state=42, solver='liblinear',\n",
       "          tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "log_reg = LogisticRegression(solver='liblinear', random_state=42)\n",
    "log_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 다른 선형 모델처럼 로지스틱 회귀 모델도 $l_1, l_2$ 페널티를 사용하여 규제할 수 있음\n",
    "+ 사이킷런은 $l_2$ 페널티를 기본으로 함\n",
    "+ LogisticRegression 모델의 규제 강도를 조절하는 하이퍼파라미터는  \n",
    "alpha가 아니고 그 역수에 해당하는 C(높을수록 규제가 줄어듦)  \n",
    "\n",
    "**※** https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgAAAADZCAYAAABW1og0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XdcV9X/wPHXYSggyEYEBFy4cKPmDFeaO81RuRqO1MzSHKVpjpbmN3P0c6UtR6Y5cqZm5p64UxBFZSiIgCAyz++Py1RAEPh8GOf5eNzH5/O599x736B8zrlnCikliqIoiqKULgb6DkBRFEVRFN1TBQBFURRFKYVUAUBRFEVRSiFVAFAURVGUUkgVABRFURSlFFIFAEVRFEUphVQBQFEURVFKIVUAUBRFUZRSSBUAFEVRFKUUMtLlzYQQxsD7wBfAICnluizSCGAm0A9IAs4AI6SUMTld287OTrq7uxd4zIpS0iUkJ+D/wJ8q1lUwNjDO07mPEh5x7f41atjWwNTY9LnuG30zGoDGjRvrJGa/cD8iH0diZWpFVeuqeTo3NCaUW5G3cLNyw87MLk/nhkSHEBgViEt5FyqYVyj08wCi4qLwDffFw8YDi7IWeTr3duRt7sXcw8HcgUrlK+nsvvk5t7Q5ffp0mJTS/nnP12kBABgGSOBYDmmGAF2ABlLKWCHEKuArYExOF3Z3d+fUqVMFFqiilBajto/i4umLtGrcisVdF+fpXM8lniSFJiHsBadG5e3vL/W+XRK7MLLJSLp3766TmMVnAoAIIjg1PW8xG35mCMBtbnNz+s3nuu8d7nB7+u1CPw/A5isbeAyhJqFcnXQ1T+em3vce97g7/a7O7pufc0sbIURAfs7XaROAlHKJlPIbtCf77PQHlkopY1M+LwBeK/TgFKUUCn4YzCqfVSTLZFb5rCIkOiTX5/oE+3Ap9BIAl0Ivcf7u+ee6799l/6ZJ2yY6ibnXr70yfX51/au5Pvf7E9+TTDIAySSz/MzyXJ/75cEvM32ed2ReoZ4H8Nf1v3jw+AEADx4/YP+N/bk+d+z2sZk+f7j7Q53cNz/nKnlXFPsAVAGuZ/h8HbARQlg+mVAIMVwIcUoIcSo0NFRnASpKSTHr4CySpZapJckkZv0zK9fnDvxjYKbPr298XSf3zc+5W/y2ZPq88b+NuT53zM7MlZAjt43M9blT/p6S6fNHf31UqOcB9P+9f6bPr/6W+8LOwlMLM33+37H/6eS++TlXybuiWAAQZK4hSEx5fSpWKeUyKaWXlNLr7l17GjaEVq2gc2d49VUYMgRGj4aTJ9PPuXwZ1qyBrVth3z44fhwuXYKbN+H+/cL8sRSlaEl9ko5PigcgPik+10/UGZ/+U+W2FuCp+x6PZ/ny5bm6b35ifvLpP1VuagEyPv2nym0twJNP8ame9TT/vOdB5ifpVLl9on7y6T9VbmoB8nPf/JyrPB+hj+WAhRAHgP/LphPgfmC1lPKnlM81gFNAeZlDsEJ4SS3Z09atg/4pBcu5c2HixKyvYWYGMRm6GnbvrhUKLC21zcoq/bVVK2jdWkv38CEEBICNDdjaQtmyz/oNKIr+jdo+ipVnV6ZlpgBlDMvwTsN3ntmu7rnE86kCAEAd+zpcHHUxb/edkbL/z1HPvG9+Yk5t086KnJ7z96DhZ4ZPFQAADDAgaXpOLZrPf9/8xGvzlc1TmSmAtYk14ZPCczxXX/fNz7mljZSSRwmPMC9rflpK6fW819F1J8CnCCFsga3AUCmlL/Az8I4QYp2UMh54D9iUU+YPUKcO/PwzREenbzEx2mvGzsU1asBrr2Wdrly5zNc8fRqCg7O+3yefpBcADh+Gl19OP2ZhoRUE7Oy019WrwdFRO7Z3Lzx4oO23tQUHB7C3ByO9/0sopc3RO0czZaSgPVEfuXPkmedef3A9T/tzvG8j7SU3981PzPmRVeaf0359i3gckaf9ReG++oq5KEhMTiTsURihMaGEPgp9+jXlfdijMO7H3ic8Nvypv4PnofcaACFEJeAo0EtKeUoIYYg2DLArWvX/ZWCMlDIqp2t6eXnJgh4FcOUKhIdDZCRERGR+bdcOXnpJS7d3L4wdq6UNC4OkJx4I7t/XagcAOnWCPXsyHxdCKwS8+iosTnmIiY6GFSu0gkPFitqroyOUL6+lVxRFUYouKSUP4x8S9DCI4IfBBD0M0t5Hp7+/F3OP0EehhMfmvYajrGFZ4qbFFb8aACmld4b3twGXDJ+TgE9SNr2qVSt36Tp00PoWAEipFRDu39cKA2FhWpNBKm9vrYYg9di9e+mvsbHp6W7dgg8+ePpeZmbg4qLVdjRtqu07c0Y7v1IlbStf/rl+XEVRFCWXIh5HEBARwK3IWwREBhAQEcCdh3fSM/qHwcQk5Dh9TRqBwNbMFnsze+zL2WuvGd+nvNqZ2WFrZoutqS2mxqaIafl7GlQVzwVMCC3Dt7KCqlnMMTJlytP7EhLgyUEMZmYwZgyEhKRvwcFac8W1a5mbKxYvhh9+SP9saakVBFxdoUULrbki1ePHYGKSv59RUQpSUFAQAE5OTnqORFHSRTyOwC/cj+vh19My+IBIbbsVeYuouBwrpQEwMzbDycKJiuYVcbJwStsqmlekokVFHM0dsTezx8bUBkMDQx38VJmpAkARYGwMT373ubvDwoVPp42Kgtu3oXr19H21a2tNErdva1tkpLZdvAiJiekFgKgorXDg5ARVqmgFlNTXGjW0fhSmeZvMTVHyzdnZGdCqTIsTKSVCtccVa+Gx4fiF++F73xe/cD/8Hvilfb4fm/OwMDNjM9ws3XCzcsPN0g1XS1cqla+Ec3nntEy+fNnyRfr/iF76ABSGwugDUBxJqTUp3L6tjUywsNCaKEBrpqhfXysUZOXPP6FrV+39X3+Bry/UrKk1hTg6qr4HSuFIffJPrQkoDlb7rGbcrnF4u3vTs0ZP2ldpj6ulq77DUrIgpSTwYSCXQy9n2q6EXcmx7d3UyJRqNtWoZlMNdyt3XC1dM2X4NqY2es/chRDFrw+AUnhSOxTa20OjRpmP1a6t9TO4cwf8/eH6dW3z84P//svc5+GXX+Cnn9I/W1pqx+vWhTZtYGDmOWAU5bkVp4w/1fXw60TGRbLl6hb2+e8jUSZibWJNp6qd6ObRjbaV22JjaqPvMEudezH38Anx4fzd85ky+4fxD7NMb17GPC2Tr25TPe19NZtqVDSvqPcMvrCpAkApY2SkNS+4u2vNBtl56SUt7ZUr2hYRAceOaVtgYHoBIDpae1+/PtSrBw0aaM0KJfzvRlHSRCdoixkFRwez+txqNl7ZSFxSHG6WbnTz6MbL1V6mpWtLzIzN9BxpyZEsk7kefh2fEB/OhpzFJ8QHnxAfgqOzHrdta2pLHYc61LarTW17batlX6tUZPI5UU0AyjNJqXVSvHQJzp3TRiG8mjJ52rFj0Lx55vTW1uDlpY1SGDXq6f4NivIk3/u+2T6lFUULTyxktc/qZ6YzwADzsuY8TniMZwVPetboSaeqnWjm0qzwgywhpJT4P/DneOBxjt85zqngU5wLOZdlD3uLMhbUd6xP/Qr1qWNfJy2zty/33AvmFWn5bQJQBQAlX8LCtP4C587B+fPakMS7GRYOu3FDq20AbbRCRIRWYGjW7OmJl5TSqWGjhviE+MAIfUeiG0YGRiRMS9B3GEXWg9gHnAg8oWX4gcc5EXiCsEdhT6VztnCmgWMDGjo2pIFjAxo4NqCydWUMRFGc4b5wqAJAClUAKBqk1PoYnDypFQimT09vDqhXDy5c0N4bGmp9FFq1St8cHPQXt6I/qVWwxem7aNr+acz+d/Yz01mUsSAuKY5K5SulNQe0cm1FuTKq9JsqICKAf2/9y8GAgxwMOMjV+08vAWxvZk8zl2Y0c25GU+emNHRsWGKf6vNCdQJUihQh0ick6t0787GPP4ajR7Wpk8+e1QoJJ0/C//4HEyZo6zSANlXyw4faPAZKyVeSCu7ljMuRJJOwLGtJp2qd6FZd6xBoZ2an79CKBCklV+9f5WDAwbRM/1bkrUxpTIxMaFSxUVpm38y5Ge5W7qW6rb6wqAKAojMDBmgbaBn88eNw6JC2ZeyQ+NtvMHIkVKsG7dtrwxjbttXWTlBKnsYZF+soZkyMTDAyMEIgMg0JdLdy13doRUZgVCB/+f/FX/5/sc9/H3dj7mY6bmViRWvX1rR2bU0btzY0rNiQMoZl9BRt6aIKAIpepM5PkDpHQUZRUdp0xn5+2rZ0qVaz0LChtkLjjBk6D1dRMmnj1oY2AW3oWbMnHat0xNPBUz2hpoiJj+GfgH/Yc30Pf/n/xeXQy5mOO5o7ar8/1za0dmuNp4NnqWq3L0pUHwClSEpM1FZj3LdPW2zp8GGIj9eGJ+7enZ7mxx+hc2dImUxOKYZmpJToZqiSXbF1Newqf177kz99/+TwrcMkJKd3cixnXI62ldvSsUpHOlbpSE27mqqwVEBUJ8AUqgBQssXGaoUAY2N48UVt36FD6Usy16+vzWLYtas2wsBQ99NqK8+pOHYCLO0SkhI4fPsw265uY9u1bfiG+6YdMxAGNHFqomX4VTvygssLqkq/kKhOgEqpYGr6dHNBmTLQs2f6MMRz5+Dzz7VZEF95BebN05oalKJt+vTp+g5ByYXo+Gi2X9vO5qub2eW3i4jHEWnHrE2s6VK9C909uvNS1ZewNrXWY6RKbqkaAKXYe/wY/vkHtm/X1jO4cUNrErh1CwxSmhYPHdImJ1IrISpK7j2Me8if1/5kw+UN7PTbyePEx2nHatrVpLtHd7p7dKd5peYYGajnSV1TTQApVAFAAW0egvPntbkIUhc2Cg3VFjMyM9P2DRgAL78MZcvqN1ZFKYoexj1k27VtbLi8gV1+uzJl+i0qtaBPrT70qNGDajbV9BilAqoJQFEyEULrD1C/fvq+oCBtjYIzZ2D9em2zttamM37jDa0fgYHqhKw3p0+fBor3cMDiLj4pnl1+u/jl/C9svbqVuKS4tGMtK7Wkb+2+9KndB5fyLnqMUiloqgZAKTVu3IANG2DNGq2/AGgZf2CgVkOg6IfqBKgfUkpOBJ7g5/M/s+7iOu7H3gdAIGjpmpLp1+qDc3k1xKaoUjUAipJLlSvDxInadukS/Por3LuXnvknJ2uzF7Zvr9UM2KjVXHWi0ZPrViuFKiAigB/P/cgv53/J1Hu/jn0dBtUbxOt1X6eSZSU9RqjoiqoBUJQUBw5oMw6C1j/glVfg7be1WQpVE4FSnCUkJbDt2jaWn1nObr/dSLTvfUdzR173fJ2B9QbSwLGBGp9fzKhOgClUAUDJr7g42LwZVq7UJh9K/dNwd4c334QPPwRzc72GqCh54hfux4ozK1jtszptCt4yhmXoXas3Q+sPpX2V9qr3fjGmCgApVAFAKUgBAbB6Naxapb2vWFF7NTbWd2SKkrOEpAQ2/7eZ/zv9f+y/sT9tf2372gxrNIxB9QZha6YW1igJVB8ARSkEbm7aUsbTpmnTEUdGpmf+4eHalMRvvgmDB6vJhvLLyckJgKCgID1HUryFxoSy/MxylpxcQuDDQABMjUzp79mfYY2G0dyluariVzLRaQFACPEi8L+U+8YDY6SUx55IUwFYBFQDElLSTZZSHtJlrIoCWtt/x46Z9/3yi7ZOwenTMGWKVggYMwZq1tRPjMVdcHCwvkMo1s4En2HhiYWsvbA2bfheTbuajG4ymkH1BmFpYqnnCJWiSmdNAEIIK+A60E1KeVQI4Q2sBypLKR9lSLcSLdMfJaWUQojewEIpZY5jUVQTgKIrCQnwxx+weDEcPJi+v1s3mDAhfa0CJXdSn/xTawKUZ0tKTmLTlU0sOL6Aw7cPA9rwva4eXRnbdCwdqnRQT/ulQHFqAugEXJVSHgWQUh4QQgQD7YFtGdIFAtWBssBjwD5l31OEEMOB4QCurq6FF7miZGBsDP36aduFC7BoEfz0kzYNcXQ0/P23viMsXlTGn3uxCbGs9lnNvKPz8H/gD4BlWUveavgWo5uMpqpNVT1HqBQnuiwAVEGrAcjoesr+jKYDy4B7QogIIBTomtUFpZTLUtLi5eVVMnozKsVK3bqwdCnMng1LlkDLlunHzp3TCgNvv636CSj5c//RfZacXMLCEwsJfRQKQBXrKnz4wocMaTAE8zJqeIqSd7osAAgg6Yl9icCTI6ynAM5AJSllpBBiGPCnEKKZlPLJ8xWlSLC31zoNZvTVV7B2LXz2Gbz/PowdqyYXysrw4cMBWLZsmZ4jKXoCIgKYf3Q+K86u4FGC1lLauGJjJrWcRO9avTE0UOteK89Pl30ABgNDpZTtMuw7CsyXUm7IsO8iMF5KuTvDvmCgq5TyTHbXV30AlKLmzz+1QsChlO6rFhYwejR88AE4OOg3tqJETQX8NP8H/nz+7+f8eO5HEpMTAehcrTMTW0zE291bte8rQP77AOhyfrMtQD0hRF0AIURToCawXwhxWAhRPSXdNaC3EMIgJV0boDxwS4exKkq+desG//6rLVXcsSM8fAhffqlNLLRhwzNPLzWWLl3K0qVL9R1GkeAX7sebW97EY6EHK8+uJFkm80bdNzg38hw739hJ28ptVeavFBidNQGkVOf3BX4QQki06v8ugBngBqSOVRmFNlTwjBAidUmqPlLKMF3FqigFqU0b2LMHjh+HOXNg507wylBmT04u3VMNpzYBlGbX7l9j9sHZ/HrhV5JlMobCkKENhvJxq4+pblv92RdQlOegZgJUFB27cwdcUlZVTU7WliN+4QWYPFnrS6CUHn7hfsw4MIO1F9eSLJMxMjBiSP0hTGk1RfXoV56pOA0DVBSF9MwftFqBI0e0bdkyGDcOxo8HKyv9xadr27Zpo4C7d++u50h0J+hhEDP/mcnKsytJTE7EyMCItxu+zZRWU6hsXVnf4SmlhKoBUBQ9O3tWm3J4+3bts5WVtmTx2LFQrpx+Y9OF0tQJMDw2nC8PfcnCEwt5nPgYA2HAkPpDmP7idNys3PQdnlLMqBoARSnmGjbURgwcOQKffKItS/zxx9qUwxculPz+Ad26ddN3CIUuOj6aBccW8PWRr4mKiwKgT60+zGo7i1r2tfQcnVJaqQKAohQRLVrA/v3a9skn0KdPeuafmAiGhlASO4CnNgGURAlJCSw7vYyZB2dyL+YeAB2rdOTz9p/j5fTcD26KUiBUAUBRihAhoH17aNcOkjJMe/Xtt7B5M3zzDTRrpr/4lNyRUrLddzsf/fUR/4X9B0Az52Z80f4L2lZuq+foFEWjCgCKUgQJAUYpf51JSbB8OVy7po0W6N8fPv8cqjw5ibZSJJwLOcf4PePZd2MfANVsqvF1h6/pVbOXGsOvFCklvHVRUYo/Q0M4eVLrF2BiAuvXa0sPjx8PERH6ji7/hBAlImMMfhjM21vepuHShuy7sQ9rE2v+1+l/XBp1iVdqvVIifkalZFGjABSlGLl9G6ZOhZ9/BinBzk7rPFi9GM8VU9xHAcQmxDLvyDy+OvwVMQkxGBkYMabJGKa9OA0bU7X4g1J41CgARSlFKlWCH3/U5gsYNw7i4qBqMZ8vprhm/FJKNv+3mQ92f0BAZAAAvWr24qsOX+Fh66Hn6BTl2VQBQFGKoYYNteGCDx6kjxS4eVNbeXDOHHBy0md0Jd/VsKuM3TWWPdf3AFCvQj0WdF6At7u3fgNTlDxQfQAUpZgSIvPywpMnw+rV4OGhrUIYF5ftqcpzehj3kEl/TaLu93XZc30PViZWLHp5EaeHn1aZv1LsqAKAopQQX3wBvXpBTIxWGPD0hN27n32evnXv3r3ITwMspWTthbXUXFyTr498TWJyIu80fIdrY64xuulojAxUZapS/KhOgIpSwuzZA++/D/9pw8/p10+bR6BiRf3GlZ2i3gnw0r1LjN4xmn8C/gGgqXNTFr28iCbOTfQcmVLa5bcToKoBUJQS5qWX4Px5+PprMDODTZsgrAgvpr1161a2bt2q7zCeEpsQyyf7PqHB0gb8E/APdmZ2rOyxkqNvH1WZv1IiqBoARSnBAgLg4EEYNEj7LKU2oVCNGvqNq6jb7bebUTtG4f/AH4CRjUfyefvPsTa11nNkipJO1QAoipItN7f0zB/gjz+gVi147z2IjNRfXEVVSHQIr298nc6/dsb/gT91Hepy5K0jfN/te5X5KyWOKgAoSini66sNG1y0SJtN8LfftFoBfVq2bBnLli3TawzJMpmlp5ZSc1FN1l5ci6mRKV91+IrTw0/TvFJzvcamKIWl1DQBREVFce/ePRISEnQYlVKaGRsb4+DgQPny5fUdSibnz8PIkXD0qPa5e3dYsgRcXPQTj747AV64e4ERf47g6B3tF/JytZdZ3GUxla0r6yUeRcktNRNgLkRFRXH37l2cnZ0xNTVVc3IrhU5KSWxsLIGBgQBFqhBQrx4cOgQrVsBHH8G2bfDPP7Bzp7Yksa4NGzZM9zcF4hLjmHVwFl8d/orE5EQczR35rvN3vFr7VfUdoZQKpaIGwM/PDycnJ8zMzHQclVLaPXr0iKCgIKpVq6bvULIUGAijRsGFC9pWrpy+I9KNY3eO8daWt7gSdgWAd73e5Yv2X2BpYqnnyBQl91QNQC4kJCRgamqq7zCUUsjU1LRINzs5O8PmzdowwdTMPyoKVq3SCgbGxvqNr6DFxMcwdf9UFhxfgETiYevByh4raeXaSt+hKYrOlZpOgKpKT9GH4vD/Tgiwt0//PGWKttBQs2Zw5kzh3z8oKIigoKBCv8/fN/6m3v/V49vj3yKEYFLLSfiM8FGZv1Jq6bQAIIR4UQhxRghxXghxSgjxQjbp7IQQvwshLgohTgshvtBlnIpSmvXsCe7ucPYsNG2qTSv8+HHh3c/Z2RlnZ+dCu37k40hGbBtBu5/a4f/An3oV6nH8neN82eFLTI1VzaBSeumsACCEsAI2AaOllPWACcAWIYTZE+nKAn8CC6SUnlLKxsA8XcWpKKXdSy9p/QHefx+Sk7WFhRo2hOPHC+d+FStWpGIhzVO8w3cHnt97suzMMowNjJnpPZOTw07i5fTczaaKUmLosgagE3BVSnkUQEp5AAgG2j+RbjBwDBibUkvwM1DCWiJzx9vbm8mTJ2d5LDg4GFtbW7Zt25avewwbNoyOHTtmeez999+na9euxMXF4ebmxuLFi/N1r379+tGvX79cpS2oeyrPx9xcWz/gyBFtvoD//tNGCFy5UvD3KowmgPuP7jPoj0F0XdOVO1F3aOrclDMjzjDtxWmUMSxToPdSlOJKl50AqwDXn9h3PWV/Rm2ABkB34BYwE/iVpwsKCCGGA8MBXF1dCzjcos3ExISGDRtia2ubr+sMHDiQ9u3bc/fuXSpUqJC2Pzk5mQ0bNjBv3jyMjIyoV68eTvlcZN7DwyPXbeIFdU8lf154QWsK+PRTuHtXm0WwqNt2dRvD/xxOSHQIJkYmzG47m3EvjMPQwFDfoSlKkaLLAoAAkp7Yl8jTtRAOwGop5U0AIcRXQKQQwlxKGZ0xoZRyGbAMtGGAhRF0UWVtbc3evXvzfZ02bdrg7OzMhg0bGDNmTNr+AwcOEBUVRa9evTA0NMx3TQPA7Nmzc522oO6p5J+JibawUMYRw6dPa9MKT5sGZcvqL7aMIh9H8sHuD1jlswqA1q6t+aHnD1SzKZpDMBVF33TZBHAHePIx3TVlf0b3gKgMn5MzbAVGfCb0sj2PGTNm0LBhQyZMmICjoyMvvPACiYmJCCE4cOAAAIcPH6Zp06Y4OTnh7OzMm2++SURExLN/D0Lw+uuvs3bt2kz7165dyyuvvJI2d4KLiwurV6/ONh6A33//nbp162Jvb0+9evVYvHgxQghu3rwJaLUNQ4cOBbQChhCCXbt20bRp07RzDh06lBZDxnuCtmpcixYtcHR0xMXFhcGDBxMfH4+Pjw99+vTB1dWVihUrUq9evbTfi1JwUitvkpPhrbdgzhzw8sr/SIHGjRvTuHHjfF1jr/9e6n5fl1U+qyhrWJb5L83nwNADKvNXlBzosgCwBagnhKgLIIRoCtQE9gshDgshqqek2wQMF0JYpHweB+yXUj7SYaxFjo+PDxYWFgQGBvL3338/dXzs2LG0b9+eoKAgrl27RuPGjYnM5WovgwYN4ujRo9y6dQuA+Ph4Nm7cyMCBA3Mdz8mTJ3nttdf44osvCA0NZfPmzaxYseKZ9549eza///47oaGh9OzZkyFDhmSZ7vfff2fQoEFMnjyZwMBAbt68SYcOHUhKSuLixYv07duXq1evEhwczLBhwxgxYkSufnYl7wwMYPFiqFYNLl7URgp8+inExz/f9c6cOcOZ5yxFRMdHM3r7aDr+3JHbUbdp4tSEsyPO8kHzDzAQpWaUs6I8F501AUgpI4UQfYEfhBASrfq/C2AGuAGWKen+EEJUA04KIWKBAGBogcczvXi1GDg7O/Ppp58ihMDU1JTExMRMx5s1a8b+/fs5deoUXl5emarzmzRpwu3bt5+6ZkhICAC1a9emfv36rFu3jokTJ7J7927KlClDhw4dch3PqlWr6Ny5M926dQOgSpUqzJs3L8drAHz//fdp/Tf69u3L7NmzCQ8Px8bGJlO6qVOn8uGHH9KjR4+0fYMHDwa0moXY2FhOnz6Nv78/V69exc/PL8f7KvnTqhWcOwcffwzffQezZsGWLfDjj9CgQd6u9bzLeB+6dYihm4dy/cF1jA2MmeE9g4ktJ2JkUCrmN1OUfNNpEVlK+beUsomUsqmUsoWU8qiU8raU0kVKeSpDurlSyppSyoZSyl5SysKfJaSIc3FxybED3eLFixk1ahQjR46kcePGrF+/Pu3YyZMnCQkJeWrLaODAgaxbtw7Qqv8HDBiAoWH2naaejOfmzZtPTXdbtWrVZ/5cnp6eae9TmxuioqKeSufv70+9evWyvMb8+fOpXr06a9as4f79+9SoUYPk5AJtMVKyYGamjRQ4cACqVNEWGerQAWJi8nadvDYBPE58zIQ9E2izqg3XH1ynXoV6nBx2ko9bf6wyf0XJgzz9tQghjkop1dqYRZAQgiFDhjBkyBCOHz9Oly5dKFu2LL169crV+a+//jqTJk0RRUdtAAAgAElEQVTi7NmzbN26Nc9t6Pb29ty5k7k7R26GduV2VICzszNXr159av+NGzeYMGECPj4+aQWEY8eO5eqaSsFo00bL/CdP1voEFOZ6AicDTzJk8xCuhF3BUBgyudVkPn3xUzW0T1GeQ15rAOpntTO1XV/Rj4iICKZPn55Wze/u7k6ZMmVy3QcAtMlY2rZtyzvvvIOLiwteXnmbKGXAgAFs2bKFPXv2AHD37l0++eSTPF0jJx9//DFff/01u3fvJjk5maSkJNasWcO9e/eQUnL37l0Arl+/zvvvv19g91Vyp1w5WLgQMnbh+P57bd+zKmNmzJjBjBkzckwTnxTPtP3TaL6yOVfCrlDTriZH3j7C7HazVeavKM8pyxoAIcSmlLdSStlHCBEFVM9w/BRa2320lLIpcBpQf4V6YmpqSmxsLO3atSMqKoqyZcsydOjQHDvxZWXQoEEMGTKEmTNn5jmGl19+mSVLljBu3DjCw8NxcnLinXfeSevtn1/Dhg3D1NSUqVOnMmjQIIyNjenYsSNLlixhzpw5DBkyhMTERKpVq8aoUaM4ceJEvu+pPL/gYPjwQ20K4S1btMWFKlXKOu1nn30GkG0h4Pzd8wz+YzDn7p5DIPjwhQ+Z3W62msZXUfIpy+WAhRDJwGfAJCmlmRDiIVANuAm8CPwNjEGbrre8ECJBSqnX2fpyWg74ypUr1CoOM5iUMKdOnaJJkybcvXsXBwcHfYejN6X1/9+mTTBihLbSoKWlVhswcGD6cMJUqRn/kwWAxORE5h6ey/QD00lITqCKdRVW91xNa7fWuvkBFKWIy+9ywNk1AUgp5WdA6hIgMmUrizZNrwnacD1FydbZs2exsrLCPuNSc0qp0bu3NkywRw+IjITBg+HVVyE0NHO6rJoAroZdpdUPrfh4/8ckJCfwrte7nBt5TmX+ilKA8tIHQKBV+RsAeeznq5R0kZGR/PLLL8SnDAY/d+4cn3/+OUOHDi0WS+IqhaNCBdi8GX74ASwstFqBt97KPn2yTGbRiUU0XNqQ44HHcSnvwu6Bu1nSdQnmZcx1F7iilAJqzIxSIIQQrFu3jvHjxxMfH4+FhQUDBgzI0/S/SskkBLz5JrRtC6NGwbwn1vY8ffo0AA7VHHhr61vs9demuB5UbxDfvfwdViZWug5ZUUqFvBYAjFMW4FEFByWT8uXL8+eff+o7DKUIc3eHHTvSP0sJY8bAkiXjod4/WL5mSWRcJLamtizttpQ+tfvoLVZFKQ3y0gQg0Xr6D6OULs+rKErB2bQJliwBOABm3xAZ/ZjuHt25OOqiyvwVRQeyKwCkNtoKIcRltCF/AoiTUjZB6xwoATMhxJEcrqMoipIlgxp/Uq7jPBCJcOxDKq4J5tPqW3A0d9R3aIpSKmRXlT8o5XU46eP7My4t9yEQC6TOuPJrwYemKEpJFBUXxYe7P2Tl2ZXQEhq+4EvkuoX4+1rTvLm2xPCUKWCs6hkVpVBlWQCQUv6a8roBQAhhBPQhpWZASrk8JeliHcSoKEoJcTDgIEM2D+FmxE3KGpbl8/afM+6FccRNMWDKFFiwAKZPh/LlYdw4fUerKCVbbjvzlQPWAIMLMRZFUUqox4mPmbp/KvOPzkciaVSxET/1+ok6DnVwcnICtLUjevSAb76BkSP1HLCilALZTQX86RO7TNCe/qumHDsGvJDh+CEp5f7CCVFRlOLsTPAZBv0xiMuhlzEUhnzc+mOmtZmGsaFWxx8cHJyWtl07bUv14IE2dPCLL7RRBIqiFJzsOu81eWKrC2xPee8FuALTgaYp+5wLPdJSyNvbGyEEy5Yty7R/9erVeHt75/o6Bw4cYMWKFdkeX716NY6OOXe86tevH/369cv1PbNy+PBhDAwM0hYtyujs2bMYGRkREhLCxIkTadasWb7utW3bNmxtbTNlLjkpiHsqmSUmJzL74GyarWjG5dDLeNh6cOTtI8xsOzMt8wcIDAwkMDAwy2tMnQrr1kHdurBihTZ0UFGUgpFlAUBK2R0YB8ySUnbPuAHhwEq0GoEeKft/1l3IpUvHjh2ZNWsWcXFxz32NZxUAcsPDw4MaNWrk6xotW7akcuXKrF+//qlj69ato127djg6OuLu7k6dOnXydS9bW1saNGiAqWnuFowpiHsq6a6GXaXlDy2Z9vc0EpMTGdt0LGdHnKWpc9On0jo5OaU1Azxpxgzo0weio2HYMOjaFbIpKyiKkkc5Dd9rBQwUQpQRQnwkhEidhLspUHpXdtGxAQMGYGFhwffff6/XOGbPns2sWbPyfZ3XX3+ddevWZdonpWTdunVpqxeOGjWKH374IV/3adGiBfv27cPKKnezyBXEPZXMU/meCDyBS3kX9g7ay4KXF2BmbJbn69nbw4YNsGYNWFvDzp3g6Qm//qpqAxQlv7IsAAghNqKN8xfAp8DrwCYhhCMQhlYAKPZ/fkJkv2WsdV+2LOe0GTVunLt0uWVgYMDMmTP54osviInJegmGW7du0adPH2xtbbGxsaFDhw5cunQJ0JoR5s2bx5kzZ3B0dKRnz5453u/mzZsIIZg3bx41atTAwsKCixcvMnDgQIYOHQpAdHQ0gwYNwtnZGQcHB1q1asXBgwdz9fMMGjSI06dP4+vrm7bvyJEjhIaG8sorrwAwderUtCaO7OIJCAige/fuVKhQARcXFz755BOaNGmStqjM3r17M61B4O7uzpw5cxg8eDAVK1bE2dmZadOmpR3PeE+AgIAABg4ciLu7O46OjjRv3pyTJ08SGxvLlClTqFWrFo6Ojmn3VuB25G06/dKJ93a+R2xiLIPrD+bCuxdoX6V9jucNHz6c4cOHZ3tcCHjtNW1hoa5dISJCW1Xw7NmC/gkUpXTJrgbgpQzvOwMdgWXAG0AyYFPIcSkZ9OnTB2dnZxYsWPDUsdjYWF588UVq1apFSEgIYWFh9OvXj9atW3Pv3j0OHDjAhAkTaNSoESEhIWzZsiVX99ywYQP//PMPDx48eKrqf9WqVRw6dIjr168TEhLCl19+SVBQUK6u6+HhgZeXV6ZagLVr19KzZ08sLCxyFU/16tXp3r07NjY2BAQEcP36deLj49PmlM/OV199RZ8+fQgKCmLv3r189dVX/Pvvv0+li4iIoGXLllhZWXHu3DlCQkJYunQpjx8/JjIykqSkJPbu3UtISAjHjh1j0aJFHD16NFc/f0kkpWTlmZV4fu/JXv+92JnZsbHfRn7s9WOu5vFfvnw5y5cvf2Y6JyfYtg1WroSPPoJGjQoiekUpvXIzg5+plDIMuAXMBVoD2X9TFyNSZr9lfCAZPjzntBmdPp27dHkhhGD27NnMnTuXiIiITMe2bt1KdHQ0n332GcbGxhgYGDB8+HCqVq3Kzz8/f9eM2bNn4+joiJGREcZPzMjSqFEj7t+/z7p160hISKBVq1YMGDAAgHnz5uHo6PjUlrHdf+DAgaxduxaApKQkNmzYkFb9n5t4zp07x4ULF5g/fz4mJiaULVuWr7/+GhcXlxyvMWzYMHr27IkQglq1auHp6cnJkyefSvd///d/mJiY8N1332FpaQlAvXr1aN26NY6Ojnz11VdERkayadMmNmzYgImJCX5+fs/+pZZAd6Lu0GVNF97Z9g5RcVH0qtmLi+9epHet3rm+xtKlS1m6dGmu0gqhrSb49dfp+w4dgkGDtBEDiqLkXm4KAHFCiPKAI9rMgGvRhgUqOtSlSxdq167N3LlzM+0PCAjA2dkZQ0PDTPvd3Nzw9/d/7vtVqlQp22MtW7bk33//Zc+ePVStWpXJkycTGRkJwIQJEwgJCXlq69+/f9r5r732Gr6+vpw/f559+/aRnJxMp06dch3PzZs3sbKywtbWNm2fEIIqVarkeA1PT89Mn83MzIiKinoqnb+/P3Xq1MHA4Ok/Dx8fH2rVqsWkSZO4fPky1tbWODg4kJSUlOO9SxopJat9VuO5xJNdfruwNrHm196/sqnfJiqYV8jTtZ7VBJBzHDB6NPzyi9Y3YOfO57qMopRK2RUA/srw/gCwFRgB7ESbAtgAOEgJ6AdQnMyZM4cFCxYQGhqats/V1ZXAwMCnMqCbN29SuXLlQoulfv36rFmzhvPnz3P58mWGDBmS63MdHBzo0KEDa9euZe3atfTv3x8jo9wvMGlvb09kZCQPHz7MtP9ZzRAilx0xnJ2duXbtGjKLaptx48bRpUsXtm3bxtSpUxk6dCgJCQm5jr0kCHoYRPe13Xlzy5tExkXS3aM7l0Zd4vW6r+f6d1xQhIDff4cWLSAoCLp00UYLZFGuUxTlCdkNA+yN1gFQonUCPAoMl1IGZkjTVmb1DZkDIcSLQogzQojzQohTQogXnpF+uhAiQQjhnpf7lFTe3t40b94804iAnj17Ym5uzvTp00lISCA5OZlly5bh5+fH4MHaxI1mZmZEREQQHx+Pv79/vp9Wv/vuOw4dOkRSUhKWlpZUqFAhrQYgtwYOHMj69evZvHkzb7zxRp7Obd68Oa6urowfP574+HiSkpKYO3dugVXDDxs2jLCwMD766COio6MBuHz5Mjt37iQuLo779++TlJTE48ePmTx5cqmp/pdS8vO5n6mzpA7bfbdjZWLFj71+ZMuALVS0qPjc1922bRvbtm177vOrV4eDB2HuXChTRpsvoG5d2K+mJlOUHOXUBHAI+EVKGS2lnCKlTF3s/TraSIA8EUJYAZuA0VLKesAEYIsQIsuxQUKI7kBFQI36zWD27NncuHEj7bOpqSkHDx7k0qVLODo6Ymdnx/r16zl8+DAODtpozd69e2NsbIyFhQXdunUjMTExXzFUqlSJSZMm4eTkhIuLC3fu3MnzPAOvvPIKoaGh2NjY0Lx58zyda2Jiwo4dO7h58yYuLi5pHSBbtmxZIE+gTk5OHDx4kICAAGrWrImjoyNvvfUWNjY2fPvtt1y4cAF7e3tq166Nra0tjRs3zvc9i7rgh8H0XNeTwZsHE/E4gi7Vu3Dx3YsMrj8437/zHj160KNHj3xdw9AQJkyAM2e0kTi3bmkjBx49ytdlFaVEE3l8iH/+GwnRH3hfStkiwz4fYJqUctsTaWsAS4AuwFXAW0p5M6fre3l5yVOnTmV57MqVK9SqVSt/P4BS5Hl6ejJ48GAmTpyo71AyKc7//6SUrLmwhvd2vseDxw8oX7Y8CzovYEj9IQVW3d+9e3eAfNUCZJSQoE0dXKsW9O2r7ZPy+YfiKkpRJYQ4LaX0et7zc9/wmn9V0GoPMrqesj9NSofDlcAbUsq4nL5khBDD0Tom4urqWqDBKsVLZGQk/v7+eHh46DuUEiMwKpDRO0az5ao2dLRztc4s774cl/I5j7bIq4LK+FMZG8OnT6xmMmsW+PvD/PlgowYxKwqQu1EABUUATzY+J2aMQWi5/Y9oUxAHPOuCUsplUkovKaWXvb19gQarFG2//fZb2vzxUVFRjB07Fmtra1566aVnnKk8S7JMZtnpZdReUpstV7dgUcaCFd1XsOP1HQWe+etCZKS2wuCPP0Lt2rBpk74jUpSiQZcFgDtoiwhl5JqyP5UF0AD4TAhxTAhxDK0fwB9CiKE6iVIpFgICAmjdujX29va4ubkRGhrKX3/9hZlZ3qebVdL53vel3Y/tGPHnCKLiouhRowdXRl/h7UZv67yHf0GxtISTJ6FVK7h7V1tboG9f7b2ilGa67ANgiVbl31ZKeUEI0RTYDVRDG2Y4VErpm8V5N1F9AJRirDj8/0tMTmT+0flMPzCdx4mPsTezZ1GXRfSt3bfQM/7U6xf2d1FyMnz/PUyaBDExWlPAt99q0woX07KNUsrltw+AzmoApJSRQF/gByHECeBbtE5+ZoAbYKmrWBRFSecT4kOzFc2YtHcSjxMfM7j+YK6MvkK/Ov2K7VN/VgwMtEmDLl2Cl16C8HD46Sd9R6Uo+qPLToBIKf8GmmRxKNuGRSmle0HcOzk5OcuZ3RSlMCUnJ+s7hGzFJsQy85+ZzD0ylySZhKulK8u6LaNTtZxnZSxouqqFTOXmBrt2werV4O2d/vQfGqqtOJiHOakUpVgrFf/Vy5UrR2BgIBUqVMDY2LhEPdUoRZOUkoSEBO7evUu5cuX0Hc5T9vnv493t7+Ib7otAMLbpWOa0n4N5GXN9h6YTQsCbb6Z/llLrFxAVpa3+6fXclaqKUnyUigKAi4sLYWFhBAQE5HsSHEXJLSMjIywtLbGzs9N3KGlCokMYv2c8ay6sAaCWXS1W9lhJ80p5m4yppAkMhBs3tAmEmjWDMWO0oYPly+s7MkUpPDrrBFjYcuoEqCilXVJyEstOL2PKvilExkViYmTCp20+ZXyL8ZQxLKPX2Ap6IqDnFR0Nn30G//sfJCVpyw8vXAivvKI6CSpFU347AaoCgKKUcGeDzzJy+0hOBJ4A4OVqL7OoyyKqWOe8eqKu6GoUQG75+MCIEXBC+3XRrx+sW6cKAUrRU5xmAlQURYcexj3k078/5bsT35Esk3GycOK7zt/Ru1bvItUPZuvWrfoOIZMGDeDIEVi6FKZMgfr1VeavlEyqBkBRShgpJb9d+o3xe8YT+DAQA2HA2KZjmdl2JhZlLfQdXrESFAR2dtoqg6AtPWxhAZ10O1BCUbKkagAURUlzLuQcY3eN5WDAQQCaODVhabelNKzYUM+RFU9OTunvw8Nh5Ei4fx969dL6Cri76y00Rck3NTBeUUqA8NhwRm8fTaNljTgYcBA7MzuWdlvK0bePFvnMf9myZSxbtkzfYTyTuTlMnAjlysHmzdpqgzNnQmysviNTlOejmgAUpRhLSk5i+ZnlfLL/E8JjwzEUhoxuMpoZ3jOwNrXWd3i5UtQ6AT5LYKBWEFijjaTE3V2bUrhHD9VXQNEt1QSgKKXUvwH/8t7O9zh39xwAbd3b8t3L3+Hp4KnnyPJm2LBh+g4hT5yd4ddftZECY8bAhQswbRp06waGhvqOTlFyTxUAFKWYuR5+nSn7prDh8gYAXC1dmf/S/CLXuz+3ikP1f1batIEzZ7QFhho0SM/8Q0K0V0dH/cWmKLmhCgCKUkyEx4Yz++BsFp1YREJyAiZGJkxuOZmPWn6EmbFaBlkfjIzgvfcy75s4Ef74AyZPhg8+ALVCtVJUqU6AilLExSXG8c2Rb6j6XVX+d+x/JCYnMqT+EHzf82W69/Rin/kHBQURFBSk7zAKRGIiPHyozSo4dSrUqAG//KItRawoRY0qAChKESWlZN3FddRcXJMJf00g4nEE7Su358yIM6zutRqX8tkuolmsODs74+zsrO8wCoSRkfb0//ff0LAh3LkDgwZp6wvs3avv6BQlM9UEoChFjJSSfTf28cn+T9Km761jX4e5HefSuVrnYtnOn5OKFSvqO4QC5+0Np07Bzz/Dxx9r7zt1Aj8/qFxZ39EpikYVABSlCDly+wif7P+EAzcPAOBo7sistrMY2mAoRgYl88+1pFT/P8nAAIYMgVdf1RYVCglJz/ylBH9/qFpVvzEqpVvJ/EZRlGLmbPBZpv49lR2+OwCwMrFiYouJjG02lnJlyuk5OiU/ypXTOgRmtGuXNmxw4EBtBUI1o6CiD6oPgKLo0ZXQK/Td0JdGyxqxw3cH5mXMmdZmGjfev8GU1lNU5l9CXb6s1RD89BN4eMDo0XD7tr6jUkobVQBQFD24EnqFgZsG4vm9J79f/h0TIxPGNx+P/1h/ZradiZWJlb5D1JnGjRvTuHFjfYehU+PHw9WrWg1AYiIsWaI1B7z7LgQE6Ds6pbRQUwErig6dCznH7H9ns/HyRiQSIwMjhjUaxietP8G5fMnoCZ9XxW0q4IJ26RLMng3r12t9A2bN0oYQKsqzqKmAFaUYOBF4gtkHZ7Pt2jYAyhiW4e2GbzOx5UTcrdz1G5yelfaCe506sHYtfPopfPNN5omF9uzRagZUZ0GlMKgaAEUpJFJKDgYc5PNDn7Pn+h4ATI1MGek1kgktJuBk4fSMKyilWWysNmogNFQbSTBxIpSylhLlGfJbA6D6AChKAUtMTmT9xfU0W9EM7x+92XN9D+ZlzJnccjI3x91kfqf5KvNXnikmBrp00dYY+O038PKC9u1h926tqUBR8kunBQAhxItCiDNCiPNCiFNCiBeySFNBCLFUCHFFCHFCCPGvEKKuLuNUlOcRHR/NgmMLqPZdNQZsHMDJoJPYmdkx/cXpBIwL4IsOX+BQzkHfYRaq33//HfdcjGnLWPPo4eGBt7d3nu6Ten5YWBhCCG7evAlAfHw8ERERaVt4eDhRUVEAuLi4sHnzZlavXo2nZ9FfMdHODn74QZsvYMIEsLCA/fuhc2dt8aHgYH1HqBR3OisACCGsgE3AaCllPWACsEUI8eRE5o2A3VLKWlLKpsBm4BtdxakoeRX0MIgpe6dQ6X+VGLd7HAGRAVS3qc73Xb8nYFwAM7xnYGNqo+8wC8SQIUMwMjLKtAkhOHz4MImJiSQ/Men94cOHs0w/e/bstDS+vr78888/Wd7vu+++y/L87FYQ3LhxI56ennh6elK/fn1cXV2ZOHFiwf0C9MDFBebOhVu34MsvtVUGk5KgQoX0NDEx+otPKb50WQPQCbgqpTwKIKU8AAQD7TMmklLulFJuyrArmGw6KwohhqfUJJwKDQ0tnKgVJQtSSv4N+JcBvw/A7Vs3vjz8JRGPI2jl2orN/Tfz35j/GOk1stgv1POkFStWEB0dTXR0NI8ePWLFihU4Oztz9uxZjI2N6d+/f6b0zZs3T3saj4yMxMfHByMjI/r27ZuWpnr16rz44otZ3u/dd9/NdP6tW7cwMDDA0dERIQT29vaZ0r/22mvcuXOHO3fuEBAQgJeXF25ubgX/i9ADKyuYNAlu3tTWGzBI+fa+cUMrDLz9Nvj46DVEpZjRZQGgCnD9iX3XU/ZnSQhRAZgJfJbVcSnlMimll5TS68kvAkUpDDHxMSw7vYwGSxvQZnUb1l9aj5SSV2u/ytG3j/Lvm//Ss2ZPDETJ7F5jbGyMiYkJJiYmXL58mcmTJzN//nzGjBmDlJI//vgjU3oDAwPMzc0xNzcHYNiwYYwfP56aNWtibm6OEAJfX99smwCMjY3Tzi9XrhzLly+nTZs29OzZk4SEBMLCwnKM99KlS3z88ccIIQgMDCyQ34G+lS0L1aunf963T6sB+OEHbQGiNm3g99+1+QUUJSe6HAYogKQn9iWSTSFECGEL7ABmSCmzrh9UFB3xve/LkpNLWOWzisi4SAAcyjkwvNFwRniNKDEr8+VGVFQUCxYs4JtvvmHmzJn069cPb2/vtGr8rJ647927R79+/bhx4wZNmzYlOTmZwMBApJQMGDAgV/fduXMnCxcu5PDhw7maM+DYsWMkJSURExNDmTJlqFatWt5+0GLinXe0TH/xYli1Cv79V9tcXGDUKJgyRd8RKkWVLh9T7gCuT+xzTdmfiRCiIrAP+EZK+YsOYlOUpzxKeMRP537Ce7U3Hos8+Pb4t0TGRdKiUgt+7f0rt8bdYla7WaUm84+JiaFXr144Ojqya9cudu7cydixYwHYs2cPsbGxrFmzJtM5cXFx/PDDD9StW5fOnTtz48YN/P39adWqFdeuXcPKyoqYmJgcFwSKjo7mww8/5M0332Tbtm3UqFGDOXPmYGxsjJ2dXbbnLV68mP79+2NmZoaRUcme8sTDAxYsgMBAWLQIatTQliI+fz49TVISJCToL0alCJJS6mQDLIEwoG7K56bAA8AWOAxUT9nvBlwG+uTl+o0bN5aKkl/JycnyxJ0TcsS2EbL8F+UlM5DMQJrNMZNvb3lbngk6o+8Q9eqnn36Sly5dyvb4+fPn5Zw5c6SUUiYkJMiGDRvKOnXqyL/++istTVJSkvz2229lly5dZGJiogSk9lWUWXx8vBw+fLi0srKSgwcPlsHBwU+lCQ0NlYC8ceNGpv2nTp2S5cqVkzdu3Ei7PiD/+OMPuWrVKlmnTp3n/A0UD0lJUu7fL+W5c+n7tmyR0tFRysmTpfTz019sSsEBTsl85Ms6nQhICNEW+DrljzERGI9WA3AU6CWlPCWE+B1oC/hmODVOSpl1L6EUaiIgJT/uRt9l3cV1rDy7kgv3LqTtb+bcjLcbvk1/z/6UL1tejxEWHTdv3qRyDovau7m5pQ3LCw8Px9raOm2636w4ODhgYmLCrVu3njp26NAhqlatSsWKFbM8Nyoqit69e/Prr79SIaVbfHBwMM2bN+ett97i008/JTGlMdzd3Z1FixYRERHBvHnzuHjxYm5/5BJh1Cj4/vv0z61awaBB0LcvWFvrLy7l+eV3IiCd1QAU9qZqAJS8ehj3UP7k85Ps9HMnafiZYdrTvt3XdvKDXR/Ii3cv6jvEIishISHLbePGjdLNze2p9ElJSXLBggWyWbNm0s7OTgohpLW1tezUqZPctWvXM+8XEBAgx4wZI6tWrSpNTEykoaGhrFy5snz33XflnTt30tKdP39eVq5cWb7yyisyOTk50zWcnZ1LTQ1AVpKTpTx0SMohQ6Q0NZVSm05IyjJlpPzoI31HpzwP8lkDUDK7KitKNhKSEvjz2p+8tvE1HOY6MHjzYHZf340Qgq7Vu7Kh7wYCPwxkfqf51HGoo+9wi6wnx+anbgYGWX+lTJgwgblz5zJx4kSuXLnC48ePOXv2LO3ataNHjx7s2LEj23vFxMTQvHlz7t69y4YNGwgLC+PBgwf89ttvBAcH06JFC2JjYwGYNm0a3t7erFu3Lsdah9JICGjZElavhrt3tdf27bV+ATYZpqkICoKDB7U+A0rJptYCUEq8+KR49t/Yz8bLG/njvz+4H3s/7Vgr11a87vk6fev0xc4s+w5lSrrUJgBDQ8Msj7u7u+Pn55dpX82aNdOGAD6pU6dOeHh4sHDhwiyvd+zYMZo3b05YWBi2traZjgUHB+Pk5MSpU6do3Lgx8fHxlClTJsvruLi4lOomgOwEBod89zsAABVMSURBVIKJCaT+aj/7DGbM0CYc6tNHayJo1UqbklgpWtRqgIqShdiEWHZf382mK5vYenVr2tA9gDr2dXij7hu8Vve1Ur8SX34EBgZiaWmZ5TEpZaYn8JdffpmFCxdSrVo1WrRogbm5Offu3WPTpk3s2bOHkydPZlsAqFu3Ls7OzowePZpJkybh4eGBlJKrV68yZ84cKlWqRO3atQGyzfyV7Dk/sQq1paW2CNGNG9rQwsWLtcJA797w2mtaYUApGVQBQCkxHsQ+SMv0d/juICYhfX7Uug516VOrD31q96GOfR1VPVwAHB0dsz125coVatasmfZ53rx5uLm58cUXX+Dv7094eDhWVlY0Tlne7sGDB9leq1y5chw7dox58+YxYMAAgoKCiI+Px9nZmZdeeonDhw9jampacD9YKTduHLz/Ppw5Axs2aJu/PyxZApGR6QWAx4+15gMLC/3Gqzw/1QSgFFtSSi6FXmL7te1s993OkdtHSJLpDZdeTl5apl+rD9Vtq+dwJUWfUucAcHJSKyQWRVLC2bNaQaBDB63fAGizDb7xBnh7Q/fu2lZCZl0uNlQTgFKqxMTH8PfNv9nhu4Ptvtu5FZk+dMzIwIi2bm3p7tGd3rV642alvo2KA5XxF21CQKNG2pbRf/9p0w3v2aNt770Hnp7w8svQqVN6QUEpulQNgFKkJSQlcDLoJPv897H3xl6O3j5KQnL6dGYO5RzoUr0LXat3pWOVjliaZN0mrShKwQsLgx07YNs22L0bHj7U9jdpAidOaO+lhAsXtMJBNoNElOekagCUEiW1Wj81w//n5j88jH+YdlwgaOLUJC3Tb+zUuMQuvPM8YmPBzw/q1tV3JLk3fPhwgGyX+FWKLjs7GDxY2+LjteGDe/ZonQhTXb4M9euDg4PWhNCunbZ2QbVqWu2Coj+qBkDRq/ikeM4En+HQrUMcunWIw7cPE/Yo8wpvHrYedKjcgfZV2uPt7o2NqU02Vyt9kpLg9GntS3fzZu1JKykJTp7UVoYrDlI7ZJaU7yIlsz3/397dR8dd1Xkcf3+Spk3T0KQkaZvWPkLp00Ir9GG3sosc5GER2T0iqIsKnqOAi7LqKrqueFy3srLL6nZFkYcVhbIruy5aYBUFKbALBUFoC2JrrfSBtoEmpJOmbZ6a7/5xf5NMJpNk0iaZmfy+r3PumZnf3N9v7txzk3vn3vu79+dhq+LX0nZ9mTIlNARuvbX7FkQ3ON4D4ApKoiXBhtc2dFX4z+55lpaOlh5xastrOWfuOV2Vflw228mGGWzdCo88Eir8DRtgzBhobQ2/wCDcxpV8Xghuu+22XCfBDaPzzoNdu0K5/fnPQy/Bk0+GxYgefjiU16QvfAEmToSVK2HZMr/DYLh5D4AbNi0dLWys28hze57jub0hbK3fitGzzC2sXsiZM8/sCnMq5/htein27Al7vj/wQHhsawsNgWjxu14qKsJ47MqVI5tO57KVbMhu3w7vfGc41toaym5ra3gtweLFsGJFKMsXXAAz0/eTjbnj7QHwBoAbEi0dLbyy/xVe2PdCV4X/0hsv0dHZ0SNeSVEJy6Yt66rsV81Y5SvwZbBzJ6xeDT/9aZhoVVICzc3ZnVtSArW1hfXP8rrrwopzLr5aWsKths88EyYQbtwY7jJI+v73w1wDCHG2bIGlS2HRIojr+k8+BOBGlJmx9+BeNr++mU2vb+p63Fq/tcc9+BAm7C2uWczy6ctZPm05K6av4NTJpzJuzLgcpb5wNDTAY4+FddknTMi+8oewOMuuXSEUhgeprIRLL31XrhPicqi0NOxO+MEPhtctLWH9gWefDQ2Ct72tO+6998Itt4TnJSWhp2Dp0hCWL4dVq0Y+/YXIewBcRmZGXXMdv6n/DVvqt7Clfgu/3v9rNtVt6rGWflKRijil6hSWTFnC8mnLWT59OafXnk752PIcpH702L8f1q8P3f+PPBJus5Lg8OG+zym0IQCfBOgG6957w9/Exo2wbVsYUkhatQqeeio8b2+Hz34WFi4MPQWLFo2uCYfeA+COS2tHKzsO7GBL/Zauyj752NTalPGcytJKlkxZwmlTTmPJlCUsmbqERTWLKCspG+HUj341NXDZZSFAWJ/90Udh3bowkcos/JNLjpsWoosuuijXSXAF5vLLQ4DQO7Z5c2gMbNwIp5zSHW/bNlizpue5kyd3NwY+/Wk46aRw3Cx+tyV6D0AMNB5pZHvjdra/uZ3fN/6e7Y3dj7sTu3tNykuaVDqJhTULWVC1gAXVC1hYs5DTppzGjIkzfJJeHujsDLf9Pfoo/OhH8PzzMG5c6Dp98snC6QFwbrjs2wf33BPWIkiGQ91bhLB5c/eaGddeGxY1mjcvrFGQGmbNCkNx+cYnAUbi2gA42nmUfc372J3Yze6m3d2PTbvZeWAn2xu3c6DlQJ/nF6mImRUzmV81n4XVC7sq+gXVC6gpq/GKvoC0tYXx0s2b4SMfCY0B51y3zs6wHkGyMXDNNVAWdVy+4x3hLptMzj033MIIocfhhhvCJNtZs7pDVdXI9yB4AyAy2hoAZsbBtoPsO7iPuua6rrDn4J6uin5XYhd7D+7tNfku3YSSCZx04knMnTSXkyb1fJxVOYuxxTGdQuucc5HWVtixI6yk+bvfheGD5PPzzw/bIgO8/HLmlTbLymDGDFi7NqxhAPD007B7d7grZ9q08DiUPQk+B6CAHO08SmNLI/WH66k/XE/D4QbeOPQGdc117GvuWdHXNddxpKOPG73TTJ4wmRkTZzCjYgYzJs5gZsXMrse5k+YyecJk/yXv8pZPAnT5YNw4mD8/hP5UVcFNN4W7bHbu7H5MJMLaBqk7U995J9x1V8/zJ04MjYEzz4Q77gjHzMLz6uow7ycZJk0a3v0TvAFwDDo6O0i0JDjQcoBEa6LrecORhq6Kvf5IdyVff7iehiMNNB5p7HO8PZOykjJqy2uZWj61K9SW14YKPqrsp0+cTumY0mH8ts4555Jqa+H663sfTyTCr/2TT+4+tmIFNDWF23n37QuPTU0hpK7TceAAXH1172sWF4cGxx13wMUXh2OPPRaGKobibobYNADMjMPth2lua+ZQ+yGa25q7wqG28Ppg20ESLQkSrd2V+4GWAz2PtSQ41H5o4A/sw4njT6RqfBXVZdVUlVVRU1bTq5KfWj6V2hNq/RY6Fwv+y9+NBhUVPZc1hjDH4Jprul+bwZtvhsZAqs7OsF/C/v09QyIBb7zRc6Gj9evhxhuHJs2jZg5A9cnVdtaNZ/Wq1JPhcPvhQf367k+RiqgYV0FFaQUV4yqoLK2korSiu2KPHpOVfPLYpPGTGFMUmzaXc86549DWFlYCnTSpe2jhiSfCXT4NDbBmTQFNApR0FvANQs9DG/BxM3smLY6ArwCXAUeBF4Crzazfn92aJiNDF0qqspIyJpRMoHxsOa9uLYe2CdBWHkJ79Lylkhu/FCr3ytLKHhV88vnE0nIg85j6QNnZ31B8f+ce63kAU6eGjTfSTZkCdXXDd24hOp58zsW5hZbe4z3Xy7Jz3QrmLgBJlcB24CIz2yDp7cB9wBwzO5wS70rgE8CZZnZE0l3AITP7eH/Xn7Nojt38w5spH1vOhLGhki8fW95V4ZeVlFFcVJySnr6vlY///AotvYWq0PK50NKb+dzkEsAP5m2anctHhXQXwPnAVjPbAGBmj0vaB5wDPJgS773AbWaWnAK/BvgF0G8DoKqsiksWXTL0qXbODbOHcp0A52JpJBsAcwk9AKm2R8f7i7cdOFFShZklUiNKugq4CmBmIW195pxL8UCuE+BcLA3jHYa9iDCmn6ojQxrS4yU3hOyVVjO73cyWmdmympqaIUuoc24kvYvuYQDn3EgZyQbAa0D6z/SZ0fH+4s0EmoG+17N1zjnn3KCM5CTACkJ3/tlm9pKkFcDPgJMJfYBXmtk2SR8GPgy8w8zaJN0CnGBmVwxw/YPA1uxTdMYZfb/3q1/l37nH85lLlsCYDMM9HR2wadPwnVuI4lQuCvHcgi/L1UD9CH1WIfN8ys58MzvhWE8esTkAZpaQdCnwXUlG6Nq/ECgDZgHJJRTuJjQKfimpA3iFASYARrYez2zIuJD0vOdTdjyvsuP5lD3Pq+x4PmVH0nFtgDOiq9KY2XpgeYa33pIS5yjwt1Fwzjnn3DAYyTkAzjnnnMsTo6kBcHuuE1AgPJ+y53mVHc+n7HleZcfzKTvHlU+jZi8A55xzzmVvNPUAOOeccy5L3gBwzjnnYsgbAM4551wMFVQDQNJZkl6QtFnS85L+MEMcSfp7SVslvSJpraQJuUhvrmSZT2dIelPSMynhr3OR3lyRVCLpM5LaJb2vjzixL0+QdV55mZKulrQp+rvbLOkv+4h3raQtkl6W9JCkKSOd1lzLJq8kVUk6lFambs5FenMl+v+zSdIvo//rH+sj3qDL1IiuA3A8ou2E76fndsLrJPXYThi4grDA0NKU7YRvIrvFhAreIPKpGvhvM/toLtKZJz4KGPBMP3FiXZ5SZJNXsS5TkoqBecDbzKxZ0nTgd5LWmdmelHhvBz4HnGFm+yV9Gfg34KIcJDsnss0rQpl62szOzUlC80MjsMzM2iXVAK9KetjMXk1GONYyVUg9AL22EwaS2wmnyrSd8PtHKpF5INt8qgbOk/RsFL4q6ZiXlCxEZvZtM/tnem9SlSru5QnIOq9iXabM7KiZfcbMmqNDDUAbUJwW9b3AWjPbH71eA1wQLZceC4PIq2rgNEkbJD0n6Ztx6y0xs6+bWXv0cjZhb5w306IdU5kqpAbAcW8nPIxpyyfZ5tP9wGwzWwn8KTCHsAyz6ynu5WkwvEz19C/AfWa2K+14jzJlZo1AgvDPPa76yqvngWlm9kfA2wmNhJ9I0ginL6ckzZO0jbB/zgfMLJEW5ZjKVMEMATAM2wmPUlnlU8ovWszszWisdrek8anvudiXp6x5meomaTUwHbgk09tk978sFvrLKzNrTXl+SNL1QBNhv5htI5bIHDOzbcA8SW8F/kfSuWb265Qox1SmCqnA+XbC2ck2n9IVAy1A6wDx4ibu5el4xLJMRZPUFgOXmFlbhig9ypSkMqCKgf9GR50s8qrXKYR6q2lYE5anzOxFYANwdtpbx1SmCqkBsI4wFnQqgMJ2wguAxyQ9JWleFO8e4COSxkavPwHcb/FZ8jCrfJL0vmjCIJLGAP8A3GNmnTlKd16IZh17ecpCel7FvUxJKpL0HWAGcGmyQpNULOkXkv44inoPcHnKMNK1wFMp47ejXrZ5JeliSbXRcwFfAR43s9dzlfaRJOlUSZclhzyiyZIrgReHokwVzBDACGwnPCoMIp/GA7+Q1EmY3f0E8KUcJDnfeHnKnpepni4EriaMW/9fyjD1asJ8iBMh7Ioq6VvAE5Lagb1AxlsrR7Gs8opQju6XVAJ0ApuBy0c2qTm1E7gG+FxUVsYCNwC/YgjKlO8F4JxzzsVQIQ0BOOecc26IeAPAOeeciyFvADjnnHMx5A0A55xzLoa8AeCcc87FkDcAnBslJCXX4W9W2L3wurT3n5E0qFsYJe2QdMHQptQ5lw+8AeBcHpL0RUmWEj6R9tokfSAl/unAj4FvEe7Nvxr4vKSrBvicCxW2GW2W9FtJ1yvs1JZNGldnSFN66LUVdYbrvEvSumw+czAkvTVaqKh0qK/t3GjgDQDn8tNNwAnA6YTFUL5D2PFrdnR8C2Gzj6RLgZ+a2d1m1hDtAnkz/SyaImkl8CPgdmA+odFwLWFb0Wx8mbD4TwthHffxUfhWdN3xwLP9XSBa2ew2oNde8McrWjb1fwn54JxL4w0A5/KQmbVHW6WOBRqj7UD/HZgTHa8GUpf5LKL3ZiADrfL1fuBhM7vTzPaY2Xrgq8AVWaaxw8xaCA2RDjNriV4fBRLR64HS8DXgP9P2gB9K/wR8UNLiYbq+cwXLGwDO5bdq4MXUA5LKCb+6X005/EPgIknvkTQh6nr/NHBfP9cW3bsbJmW9K52kh6PlpqcA65Ld/sB1wJXR68/0c341YbnS76Qdv1jS05LqJL0m6W5JYyV9T9IaSXdG7+2R9H5JsyU9Jmm/pBclLUtey8waCL0RH8vmOzkXJ94AcC7PRBPvkpXpQ8A50fNiYD1wEHgLUBfF+3Mze45Qmd4ANAA/AG4Bbu3no+6nu9FQprDV6Ofpv9HQxcwuMDMlA2F98k+lHjOz/rrf/wzYY2ZbUr77ewgbm3yNsEXsbODR6LsDXAU8DtQShj1uJQwhfMrMagg7pf1r2uf8DHh3Nt/JuTgpmM2AnIsLM5t9jOc9ADwwiPhPSLqC0O1/H1APfBf4u/7Ok1REGJro9RYwJsOkO0vd1z3FfOC3acdWA1+PvkvS3dHnAvzEzNZGx5+W1Ab82Mw2RcceBD6Uds3fArWSKswsgXMO8AaAc3kt2tnxnrTDT5rZeX3ELyZUxJm8EziSesDMfgD8QFKxmaXPIejLnxB6IjI5nTDunioBVGaIWw4cSjs2l7DjW19eTnt9mDAhMukIMCEtTnP0OJGeEyedizVvADiX3x4ibEecqlWS+phg10p3d3kma4BPph9MVv7Rr/cTgWlAY6YLRHcYKGpslPTzWX398k96DXhr2rE9hJ6BPq+Z5bFUtYS5DXsHiOdcrPgcAOfylKQdhPH+HVF4FdgG1AF7M93fbmZjgG8Cd6WMzZdHby8ys09G1/68pAMpoVlSB+EX9UvA94EVAyTxbwi/uPsKDQOc/xSwVFJZyrEbgeslnS+pSFKxpL+QVDPAtfqzCtgwiB4O52LBGwDO5Skzm21mY5IBWAr8F6Eb/JzolrtM2oBxKa+TPX2pM/6/DfxBFBYDJxF++Y8xsyozW2xm/zFA+lanTfhLnRB4dhZf8UngdcLQRPKadxDuIlhNaOjsAs6j91DBYFwCrB0wlnMx40MAzuUpSVWEyvlsQiW5ECgFvgG8RVITYRa9pZwzhlDRl0bPobsxYMmxfjNrAppG6KtkZGYm6Qbgs4SGTfL4WjJX2FdmuMbstNePkzIHQtI5QAXwvSFIsnOjivcAOJeHJH0R+A1hRv5Ywgp9EwmT7BLAXxFW2Tsj5ZzZQDthJb93R8/bCb+yIQwfPDES6Y8+tz6LeGuBN6K7EYaUpBLgH4GPmlnbUF/fuUKngRfqcs654SNpIvAhM7tliK+7GJhnZj8eyus6N1p4A8A555yLIR8CcM4552LIGwDOOedcDHkDwDnnnIshbwA455xzMeQNAOeccy6GvAHgnHPOxdD/A8QWyYHPBegwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x216 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_new = np.linspace(0, 3, 1000).reshape(-1, 1)\n",
    "y_proba = log_reg.predict_proba(X_new)\n",
    "decision_boundary = X_new[y_proba[:, 1] >= 0.5][0]\n",
    "\n",
    "plt.figure(figsize=(8, 3))\n",
    "plt.plot(X[y==0], y[y==0], \"bs\")\n",
    "plt.plot(X[y==1], y[y==1], \"g^\")\n",
    "plt.plot([decision_boundary, decision_boundary], [-1, 2], \"k:\", linewidth=2)\n",
    "plt.plot(X_new, y_proba[:, 1], \"g-\", linewidth=2, label=\"Iris-Virginica\")\n",
    "plt.plot(X_new, y_proba[:, 0], \"b--\", linewidth=2, label=\"Not Iris-Virginica\")\n",
    "plt.text(decision_boundary+0.02, 0.15, \"결정 경계\", fontsize=14, color=\"k\", ha=\"center\")\n",
    "plt.arrow(decision_boundary, 0.08, -0.3, 0, head_width=0.05, head_length=0.1, fc='b', ec='b')\n",
    "plt.arrow(decision_boundary, 0.92, 0.3, 0, head_width=0.05, head_length=0.1, fc='g', ec='g')\n",
    "plt.xlabel(\"꽃잎의 폭 (cm)\", fontsize=14)\n",
    "plt.ylabel(\"확률\", fontsize=14)\n",
    "plt.legend(loc=\"center left\", fontsize=14)\n",
    "plt.axis([0, 3, -0.02, 1.02])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.61561562])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decision_boundary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg.predict([[1.7], [1.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Figure4-24](./images/Figure4-24.png)\n",
    "**<center>그림 4-24 선형 결정 경계</center>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.6.4 소프트맥스 회귀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▣ **소프트맥스 회귀**<sup>softmax Regression</sup> 또는 **다항 로지스틱 회귀**<sup>Multinomial Logistic Regression</sup>\n",
    "\n",
    "![Equation4-19](./images/Equation4-19.png)\n",
    "**<center>식 4-19 클래스 $k$에 대한 소프트맥스 점수</center>**\n",
    "+ 각 클래스는 자신만의 파라미터 벡터 $\\theta^{(k)}$가 있음\n",
    "+ 이 벡터들은 **파라미터 행렬<sup>parameter matrix</sup>** $\\Theta$에 행으로 저장됨\n",
    "\n",
    "![Equation4-20](./images/Equation4-20.png)\n",
    "**<center>식 4-20 소프트맥스 함수</center>**\n",
    "+ K는 클래스 수\n",
    "+ $s(x)$는 샘플 x에 대한 각 클래스의 점수를 담고 있는 벡터\n",
    "+ $\\sigma(s(x))_k$는 샘플 x에 대한 각 클래스의 점수가 주어졌을 때 이 샘플이 클래스 k에 속할 추정 확률\n",
    "\n",
    "![Equation4-21](./images/Equation4-21.png)\n",
    "**<center>식 4-21 소프트맥스 회귀 분류기의 예측</center>**\n",
    "+ 추정 확률이 가장 높은 클래스 → 가장 높은 점수를 가진 클래스\n",
    "\n",
    "**※** 소프트맥스 회귀 분류기는 다중 출력<sup>multioutput</sup>이 아니라 한 번에 하나의 클래스만 예측하는 **다중 클래스**<sup>multiclass</sup>이기 때문에 상호 배타적인 클래스에서만 사용해야 함"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "![Equation4-22](./images/Equation4-22.png)\n",
    "**<center>식 4-22 크로스 엔트로피 비용 함수</center>**\n",
    "+ $i$번째 샘플에 대한 타깃 클래스가 $k$일 때 $y_k^{(i)}$가 1이고, 그 외에는 0\n",
    "+ 딱 두 개의 클래스가 있을 때(K=2) 이 비용 함수는 로지스틱 회귀의 비용 함수와 같음(식 4-17의 로그 손실)\n",
    "\n",
    "![Equation4-23](./images/Equation4-23.png)\n",
    "**<center>식 4-23 클래스 $k$에 대한 크로스 엔트로피의 그래디언트 벡터</center>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='multinomial',\n",
       "          n_jobs=None, penalty='l2', random_state=42, solver='lbfgs',\n",
       "          tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = iris[\"data\"][:, (2, 3)]  # 꽃잎 길이, 꽃잎 넓이\n",
    "y = iris[\"target\"]\n",
    "\n",
    "softmax_reg = LogisticRegression(multi_class=\"multinomial\",solver=\"lbfgs\", C=10, random_state=42)\n",
    "softmax_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ LogisticRegression은 클래스가 둘 이상일 때 기본적으로 일대다(OvA) 전략을 사용\n",
    "+ multi_class 매개변수를 \"multinomial\"로 바꾸면 소프트맥스 회귀를 사용할 수 있음\n",
    "+ 소프트맥스 회귀를 사용하려면 solver 매개변수에 \"lbfgs\"와 같이 소프트맥스 회귀를 지원하는 알고리즘을 지정해야 함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "softmax_reg.predict([[5, 2]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[6.38014896e-07, 5.74929995e-02, 9.42506362e-01]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "softmax_reg.predict_proba([[5, 2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Figure4-25](./images/Figure4-25.png)\n",
    "**<center>그림 4-25 소프트맥스 회귀 결정 경계</center>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 4.7 연습문제"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. 수백만 개의 특성을 가진 훈련 세트에서는 어떤 선형 회귀 알고리즘을 사용할 수 있을까요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ 수백만 개의 특성이 있는 훈련 세트를 가지고 있다면 **<font color=\"blue\">확률적 경사 하강법(SGD)</font>**이나 **<font color=\"blue\">미니배치 경사하강법</font>**을 사용할 수 있습니다. 훈련 세트가 메모리 크기에 맞으면 배치 경사 하강법도 가능합니다. 하지만 정규방정식은 계산 복잡도가 특성 개수에 따라 매우 빠르게 증가하기 때문에 사용할 수 없습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. 훈련 세트에 있는 특성들이 각기 아주 다른 스케일을 가지고 있습니다. 이런 데이터에 잘 작동하지 않는 알고리즘은 무엇일까요? 그 이유는 무엇일까요? 이 문제를 어떻게 해결할 수 있을까요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ 훈련 세트에 있는 특성의 스케일이 매우 다르면 비용 함수는 길쭉한 타원 모양의 그릇 형태가 됩니다. 그래서 **<font color=\"blue\">경사 하강법(GD) 알고리즘이 수렴하는 데 오랜 시간이 걸릴 것입니다.</font>** 이를 해결하기 위해서는 **<font color=\"blue\">모델을 훈련하기 전에 데이터의 스케일을 조절</font>**해야 합니다. 정규방정식은 스케일 조정 없이도 잘 작동합니다. 또한 규제가 있는 모델은 특성의 스케일이 다르면 지역 최적점에 수렴할 가능성이 있습니다. 실제로 규제는 가중치가 커지지 못하게 제약을 가하므로 특성값이 작으면 큰 값을 가진 특성에 비해 무시되는 경향이 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. 경사 하강법으로 로지스틱 회귀 모델을 훈련시킬 때 지역 최솟값에 갇힐 가능성이 있을까요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ 로지스틱 회귀 모델의 비용 함수는 **<font color=\"blue\">볼록 함수</font>**이므로 경사 하강법이 훈련될 때 지역 최솟값에 갇힐 가능성이 없습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. 충분히 오랫동안 실행하면 모든 경사 하강법 알고리즘이 같은 모델을 만들어낼까요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ 최적화할 함수가 (선형 회귀나 로지스틱 회귀처럼) **<font color=\"blue\">볼록 함수이고 학습률이 너무 크지 않다고 가정하면 모든 경사 하강법 알고리즘이 전역 최적값에 도달</font>**하고 결국 비슷한 모델을 만들 것입니다. 하지만 학습률을 점진적으로 감소시키지 않으면 SGD와 미니배치 GD는 진정한 최적점에 수렴하지 못할 것입니다. 대신 전역 최적점 주변을 이리저리 맴돌게 됩니다. 이 말은 매우 오랫동안 훈련을 해도 경사 하강법 알고리즘들은 조금씩 다른 모델을 만들게 된다는 뜻입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. 배치 경사 하강법을 사용하고 에포크마다 검증 오차를 그래프로 나타내봤습니다. 검증 오차가 일정하게 상승되고 있다면 어떤 일이 일어나고 있는 걸까요? 이 문제를 어떻게 해결할 수 있나요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 훈련 에러도 같이 올라간다면 학습률이 너무 높아 알고리즘이 **<font color=\"blue\">발산</font>**하는 것이기 때문에 학습률을 낮추어야 함\n",
    "+ 훈련 에러는 올라가지 않는다면 모델이 훈련 세트에 **<font color=\"blue\">과대적합</font>**된 것이므로 훈련을 멈추어야 함"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. 검증 오차가 상승하면 미니배치 경사 하강법을 즉시 중단하는 것이 좋은 방법인가요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ 무작위성 때문에 확률적 경사 하강법이나 미니배치 경사 하강법 모두 매 훈련 반복마다 학습의 진전을 보장하지 못합니다. 검증 에러가 상승될 때 훈련을 즉시 멈춘다면 최적점에 도달하기 전에 너무 일찍 멈추게 될지 모릅니다. 더 나은 방법은 **<font color=\"blue\">정기적으로 모델을 저장하고 오랫동안 진전이 없을 때(즉, 최상의 점수를 넘어서지 못하면), 저장된 것 중 가장 좋은 모델로 복원하는 것</font>**입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. (우리가 언급한 것 중에서) 어떤 경사 하강법 알고리즘이 가장 빠르게 최적 솔루션의 주변에 도달할까요? 실제로 수렴하는 것은 어떤 것인가요? 다른 방법들도 수렴하게 만들 수 있나요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ **<font color=\"blue\">확률적 경사 하강법은 한 번에 하나의 훈련 샘플만 사용하기 때문에 훈련 반복이 가장 빠릅니다.</font>** 그래서 가장 먼저 전역 최적점 근처에 도달합니다(그다음이 작은 미니배치 크기를 가진 미니배치 GD입니다). 그러나 **<font color=\"blue\">훈련 시간이 충분하면 배치 경사 하강법만 실제로 수렴할 것입니다.</font>** 앞서 언급한 대로 **<font color=\"blue\">학습률을 점진적으로 감소</font>**시키지 않으면 SGD와 미니배치 GD는 최적점 주변을 맴돌 것입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. 다항 회귀를 사용했을 때 학습 곡선을 보니 훈련 오차와 검증 오차 사이에 간격이 큽니다. 무슨 일이 생긴 걸까요? 이 문제를 해결하는 세 가지 방법은 무엇인가요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ 검증 오차가 훈련 오차보다 훨씬 더 높으면 모델이 훈련 세트에 과대적합되었기 때문일 가능성이 높습니다.\n",
    "1. **<font color=\"blue\">다항 차수 낮추기</font>** : 자유도를 줄이면 과대적합이 훨씬 줄어들 것임\n",
    "2. **<font color=\"blue\">모델을 규제</font>** : 예를 들어 비용 함수에 $l_2$ 페널티(릿지)나 $l_1$ 페널티(라쏘)를 추가(이 방법도 모델의 자유도를 감소시킴)\n",
    "3. **<font color=\"blue\">훈련 세트의 크기 증가시키기</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. 릿지 회귀를 사용했을 때 훈련 오차와 검증 오차가 거의 비슷하고 둘 다 높았습니다. 이 모델에는 높은 편향이 문제인가요, 아니면 높은 분산이 문제인가요? 규제 하이퍼파라미터 $\\alpha$를 증가시켜야 할까요, 아니면 줄여야 할까요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ 훈련 에러와 검증 에러가 거의 비슷하고 매우 높다면 모델이 훈련 세트에 **<font color=\"blue\">과소적합</font>**되었을 가능성이 높습니다. 즉, **<font color=\"blue\">높은 편향</font>**을 가진 모델입니다. 따라서 규제 하이퍼파라미터 **<font color=\"blue\">$\\alpha$를 감소</font>**시켜야 합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. 다음과 같이 사용해야 하는 이유는?\n",
    "   1. **평범한 선형 회귀(즉, 아무런 규제가 없는 모델) 대신 릿지 회귀**\n",
    "   2. **릿지 회귀 대신 라쏘 회귀**\n",
    "   3. **라쏘 회귀 대신 엘라스틱넷**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. **<font color=\"blue\">규제가 있는 모델이 일반적으로 규제가 없는 모델보다 성능이 좋습니다.</font>** 그래서 평범한 선형 회귀보다 릿지 회귀가 선호됩니다.\n",
    "2. 라쏘 회귀는 $l_1$ 페널티를 사용하여 가중치를 완전히 0으로 만드는 경향이 있습니다. 이는 가장 중요한 가중치를 제외하고는 모두 0이 되는 희소한 모델을 만듭니다. 또한 자동으로 특성 선택의 효과를 가지므로 **<font color=\"blue\">단지 몇 개의 특성만 실제 유용할 것이라고 의심될 때 사용하면 좋습니다.</font>** 만약 확신이 없다면 릿지 회귀를 사용해야 합니다.\n",
    "3. **<font color=\"blue\">라쏘가 어떤 경우(몇 개의 특성이 강하게 연관되어 있거나 훈련 샘플보다 특성이 더 많을 때)에는 불규칙하게 행동</font>**하므로 엘라스틱넷이 라쏘보다 일반적으로 선호됩니다. 그러나 추가적인 하이퍼파라미터가 생깁니다. 불규칙한 행동이 없는 라쏘를 원하면 엘라스틱넷에 l1_ratio를 1에 가깝게 설정하면 됩니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 11. 사진을 낮과 밤, 실내와 실외로 분류하려 합니다. 두 개의 로지스틱 회귀 분류기를 만들어야 할까요, 아니면 하나의 소프트맥스 회귀 분류기를 만들어야 할까요?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "☞ 실외와 실내, 낮과 밤에 따라 사진을 구분하고 싶다면 **<font color=\"blue\">이 둘은 배타적인 클래스가 아니기 때문에(즉, 네 가지 조합이 모두 가능하므로) 두 개의 로지스틱 회귀 분류기를 훈련시켜야 합니다.</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12. 조기 종료를 사용한 배치 경사 하강법으로 소프트맥스 회귀를 구현해보세요(사이킷런은 사용하지 마세요)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**※** https://github.com/rickiepark/handson-ml/blob/master/04_training_linear_models.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}