{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Programming Exercise 2: Logistic Regression\n",
"#### Author - Rishabh Jain"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"import pandas as pd\n",
"import numpy as np\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1 Logistic Regression\n",
"\n",
"##### Problem Statement\n",
"In this part of the exercise, we will build a logistic regression model to predict whether the student gets admitted into a university. \n",
"\n",
"Suppose that we are the adminstrator of a department and want to determine each applicant's chance of admission based on their results in two exams. We have hisotrical data from previous applicants that we can use as a training set for logistic regression. For each applicant, we have their scores in two exams and their admission decision.\n",
"\n",
"Our task is to build a classification model that estimates an applicant's probability of admission based on the score from those two exams."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TRAINING DATASET SHAPE : 100 X 3\n"
]
},
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
exam1
\n",
"
exam2
\n",
"
decision
\n",
"
\n",
" \n",
" \n",
"
\n",
"
3
\n",
"
60.182599
\n",
"
86.308552
\n",
"
1
\n",
"
\n",
"
\n",
"
37
\n",
"
64.176989
\n",
"
80.908061
\n",
"
1
\n",
"
\n",
"
\n",
"
79
\n",
"
82.226662
\n",
"
42.719879
\n",
"
0
\n",
"
\n",
"
\n",
"
85
\n",
"
68.468522
\n",
"
85.594307
\n",
"
1
\n",
"
\n",
"
\n",
"
64
\n",
"
44.668262
\n",
"
66.450086
\n",
"
0
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" exam1 exam2 decision\n",
"3 60.182599 86.308552 1\n",
"37 64.176989 80.908061 1\n",
"79 82.226662 42.719879 0\n",
"85 68.468522 85.594307 1\n",
"64 44.668262 66.450086 0"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data=pd.read_csv('./ex2data1.csv')\n",
"print(f'TRAINING DATASET SHAPE : {data.shape[0]} X {data.shape[1]}')\n",
"data.sample(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 1.1 Visualizing the data\n",
"Before starting to implement any learning algorithms, it is always good to visualize the data if possible."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XuUFPW16PHvHmB4OcrD0SDjCPGixhhEMj4QrxFR1GhUjk/iER9JMEe9ijErMcm5aIxGPZqrsnKNIXrwcXBEUZBrTI5KICZziMmgqAiyjB6ECQQmIygvHaD3/aOqh2bsnunp6ar6VdX+rDWrp2uqu39d3VO7fq/9E1XFGGOMaa8i6gIYY4xxkwUIY4wxeVmAMMYYk5cFCGOMMXlZgDDGGJOXBQhjjDF5WYAwxhiTlwUIY4wxeVmAMMYYk1fPqAvQHfvuu68OGzYs6mIYY0ysLFmy5B+qWt3ZfrEOEMOGDaOxsTHqYhhjTKyIyAfF7GdNTMYYY/IKLECIyL+LyAYRWZazbZCIvCQi7/q3A/3tIiLTReSvIvKmiIwOqlzGGGOKE2QN4hHg9HbbbgIWqOoIYIF/H+AMYIT/MwX4RYDlMsYYU4TAAoSqvgJ82G7zOcCj/u+PAufmbH9MPX8CBojIkKDKZowxpnNh90Hsr6rrAPzb/fztQ4E1Ofs1+duMMcZExJVRTJJnW96VjERkCl4zFLW1tUGWqaBMRmnZ2krrzl1U9uzB4P6VVFTkewvGGBNfYQeI9SIyRFXX+U1IG/ztTcCBOfvVAGvzPYGqzgBmANTV1YW+HF4mo6xcv5lvPdZI08bt1Azsy68m13Ho/lUWJIwxiRJ2E9N84DL/98uA53K2T/ZHMx0HfJRtinJNy9bWtuAA0LRxO996rJGWra0Rl8wYY8orsBqEiNQDJwH7ikgTcDNwJ/CUiHwDWA1c4O/+AvBV4K/ANuCKoMrVXa07d7UFh6ymjdtp3bkrohIZY0wwAgsQqjqpwJ/G59lXgWuCKks5VfbsQc3AvnsEiZqBfans2SO4F81kYFsz7GyFnpXQrxoqbI5jotlnbhxg37guGty/kl9NrqNmYF+Atj6Iwf0rg3nBTAY2LIeHToH7jvBuNyz3tptkss/cOEK8i/d4qqur0yhyMYU6imnLeu8EsWn17m0DauGbL8Ne+wfzmsYT1VW8feYmYCKyRFXrOtvPlWGusVJRIVRX9Q7nxXa27nmiAO/+TusUL6v2waDvYGh+B56c5B3vAbVwcT3sd3jwQSIOn7k1gaWCfaKu61npnZxyDaj1tpvyyNek89Ga3cEBvNsnJ3knxaCV+plnMl7tY9Ma7zaoJilrAksNCxCu61ftXblmTxjZK9l+naZyN8Xa1vzZYLBlfXRX8aV85mGetPMdr7CCpwmVNTF1wIkZ0xUVXrPGN1+26nxQ8jXpbG32Tszt+wHCqLmV8pkXOmkH0W8RhyYwUxYWIApwasZ0RYV1TgYp26STe9JbWg8XzYLZl+zZBxFWza2rn3mYJ+18x8uaPRPJLkMLsBnTKZKvSWfcD3dfxU9d5t2G0UFdqjD7qqzZMzWsBlGAzZhOkY6adOJSc8uetNuPugripG3NnqlhAaKASGZMm+jEKRjkE/ZJO+7HyxTFQn4Boc+YNqa7siftAQd6t3ZFb7rJahAFVFQIh+5fxdyrx9q6D8aYVLIA0YFQZ0wbUwybwWxCZAHCmLjIToaLIv2HSSX7VhkTFzaD2YTMahDGxEVcZjBbM1hi2KdmTFzEIXGjJfJLFAsQxsRFHGYwWzNYolgTkzFxEYcZzNlmsJo6GDsV+g6E7RutBhFTFiCMiRPXZzD3rIRDz4Rjr4L51+4ebXXRLJu8F0ORfFoicr2ILBORt0Vkqr9tkIi8JCLv+rcDoyibMalTzoWG+lXDabfvDg7g3c6+xJqZYij0ACEiRwDfAo4BjgTOEpERwE3AAlUdASzw7xtjglTuTuWKCqjoGY/RVqZTUdQgvgD8SVW3qepO4PfAROAc4FF/n0eBcyMomzHpkK01fLQaNq+FvfbztpejUzkOo61MUaIIEMuAE0VksIj0A74KHAjsr6rrAPzb/SIom4laWOsqp1lureH+I+HXN8LJ07yOZej+1X4cRluZooTeSa2qK0TkLuAlYAvwBrCz2MeLyBRgCkBtbW0ne5tYsVQS4cg3FHX+tXDaT2H2P3f/aj8Oo61MUSL5xFT1YVUdraonAh8C7wLrRWQIgH+7ocBjZ6hqnarWVVfbFUmgwr6atzH04Sg0I7vvwPJd7cc99bjVZIGIhrmKyH6qukFEaoF/AsYAw4HLgDv92+eiKJvxRXE1H5dUEnFXaE3pAQd5V/1pv9q3mmybqN7tMyKyHPh/wDWquhEvMJwqIu8Cp/r3TVSiuJq3zs1wFOoj2HtoPK/2y81qsm0iqUGo6v/Ms60FGB9BcUw+UVzNh7mucpoV6iMArzklbf0G7ZMLZjJWk/XZTOokCCJ7ZqFmiCCv5q1zMzztZ2SntVkl3/u+aJY3G3zlr3fvl9KabII/+ZQIKntmVEMV4965GVdpbVbJ975nX+LNBg/ru+9wh7jVIOKu0D/2N1/uXs4eu5pPl7QOECj0vit6hvPdd7zmFn0JTPcE+Y+d5Kt5h6/aIpHWAQIdve8wvvuO19wS9B+fUmn9x+4OW9Rmt2yg3NkKk+d7be+QngECUc/6drzmZk1McWcjf7ouqGa57ohimc5CHbRn/sx77TQ0KUbdlBrFYJAusAARd1F/wePItau2qNqhC3XQRhkooxDlGhuOX+BZgEgC1xeRcY1rV21h1mhyayrgZXHNPQ4ONW+kguMXeG6UwpgwRd3u3F5YNZr2fS+PfBXG37I7iys41byRGg4PBrEahEkf167awqrR5KupPHe11+cw64LoA6VxjgUIk04uNcuF1Q5dqKay7yEwdVn0gdI4xwKEMVELq0ZTqKbSq687wdI4xS4VjHFBGO3QrvW9GOdZDcKYtHCt78U4zwKEMWniUt+LcZ5dOhhjjMnLAoQxxpi8LEAYY4zJy/ogjOmOKJLsGRMSCxDGlMrxxV6M6a5IvsUicoOIvC0iy0SkXkT6iMhwEXlVRN4VkdkiYglhjNscX+zFmO4KPUCIyFDgOqBOVY8AegAXA3cB96rqCGAj8I2wy2ZMl7iWNtyYMouqHtwT6CsiPYF+wDrgZGCO//dHgXMjKpsxxbHV/EzChR4gVPVvwD3AarzA8BGwBNikqjv93ZqAoWGXzZgusdQVJuFC76QWkYHAOcBwYBPwNHBGnl21wOOnAFMAamtr8+2SKpmM0rK1ldadu6js2YPB/SupqJCoi5UOlrrCJFwUo5hOAf5bVZsBRORZ4HhggIj09GsRNcDafA9W1RnADIC6urq8QSQtMhll5frNfOuxRpo2bqdmYF9+NbmOQ/evsiARFhdSV9hQWxOQKL5Fq4HjRKSfiAgwHlgOLATO9/e5DHgugrLFSsvW1rbgANC0cTvfeqyRlq3WSZoa7VeJe+gU734mE3XJTAJE0QfxKl5n9GvAW34ZZgDfB74jIn8FBgMPh122UmQySvPmT/nbxm00b/6UTCa8Sk3rzl1twSGraeN2WnfuCq0MqZbJwJb1sGmNdxvFSdmG2poARTJRTlVvBm5ut/l94JgIilOyqJt4Knv2oGZg3z2CRM3AvlT27BH4a6eeK5PkbKitCZA1VHZD1E08g/tX8qvJddQM7AvQFqAG97dhloFz5crdhtpGz4WaZEAs1UY3RN3EU1EhHLp/FXOvHmujmMLmypV7WOtZm/xcqUkGxAJEN7jQxFNRIVRX9Q7t9Yyv0PrOYV+5x3WobVJGXhWqSX7z5ehHt5VBDD8RdwTVxBNlx7cpkkuT5MJYz7qckjTyypWaZECsBtENQTTxRN3x3W0uXhkGUaa4Xrm7IElX3a7UJANi3+ZuyjbxDB3Yj+qq3t0+iUfd8d0tLl4ZBlmmuF25B6GUDtokXXW7VJMMgNUgShRUiouoO767xcUrQxfLlBSldtAm6ao74TXJZLyLkGWbgSY+0MDYuxYy8YEGVq7fXJa+gmzHd67YzG1w8crQxTIlRalDfZN21Z3gmmRy3kmIgmwGivXcBhfH5LtYpqQoNfjmXnVPXebdJmRYaNJYE1MJgmwGivXcBhfH5LtYpqTI11R06Jkg4vVJdNTc4kKSQ9MpCxAlCHr+Q2znNrjYHutimZKiffA99Ez4yvdg5hmJnDSWRqIa3zH2dXV12tjYGPrrxn4oqjHlkjuEWGR3cMgaUGsDAhwkIktUta6z/awGUYLOmoFsEZ8YcXHeRpzkNhVtWmMDAhLGAkSJCjUDWe0iRgoN06z6HOzYbgGjq5I0fNUANoqp7GI90S1tCg3TXPuaOxP94iRpw1ddElHGWKtBlFmsJ7qlTaFhmr367f7dJtUVzwYEBCPCjLH2yZVZrCe6pU2hORLbN+6+b23oXZPgSWORiXDtEfv0yqzcE90ss2uA8jWJnPMANNy3ex9rQzdRizAbgDUxlVk5J7pZh3fA2jeJ9KiETzfDlg3e360N3bggws5/mwfhsObNnzLxgYbPTMibe/XYeE6kiwMb9mpcE0AfhLPzIETkUGB2zqbPA9OAx/ztw4BVwIWqurH949PEOrwjYCkg9mQBM3oRdv6H/kmr6kpVHaWqo4AvA9uAucBNwAJVHQEs8O+nmnV4OyBJC9J39b24uL5Hd8T5s4yo8z/qS4HxwHuq+gFwDvCov/1R4NzISuWIWGd2zSdu/6BJOkGW8l4iHD1Tdkn6LEMUaR+EiPw78Jqq/lxENqnqgJy/bVTVgR09Pul9EJCgtB0RjuUu2Zb13okkCbmFSnkvm9Z4J9P2pi7zrmTjJEmfZRkU2wdR1H+miPTKs23fUgqW8/hK4Gzg6S4+boqINIpIY3NzDK9kuqjcS5pGJo5Xo0labKiU91JongjEowaYy9XPsrNadcS17g4DhIiME5EmYK2IvCgiw3L+/GI3X/sMvNrDev/+ehEZ4r/uEGBDvgep6gxVrVPVuupqG34YG67+g3YkSYsNlfJeCs0TmXN5/JpoXPwsO2v2cqBZrLMaxL8Bp6lqNTADeElEjvP/1t1L2UlAfc79+cBl/u+XAc918/mNS1z8B+1MknILlfJeckfPXP8GnPkzWHALNDXGowaYy8XPsrNatQO17s6GuVaq6tsAqjpHRFYAz4rITUDJnRci0g84FbgqZ/OdwFMi8g1gNXBBqc9vHBTHld2SlFuo1PeSHT2zaQ3Mavcv6XoNMJeLn2VntWoHat2dBYgdIvI5Vf07gKq+LSLjgeeBg0t9UVXdBgxut60Fb1STSSIX/0GLkaR5Ed15L0lI5e3aZ9nZMXXgmHf233kTsMcRVdUm4CS8K35jimeJ3OLLxSaauOvsmDpwzC3VhjGmODaruvw6O6YBHfOyptoQkbOAnwAH+Y8RQFV1726V0oQqMXMq4i6uJ1rXmmiSoLNjGvExLzYX033APwFvaZyrHClmmWEdEccJgya1iv1GrgGWWXCIL1sK1REODF00RD4BLS6KrUF8D3hBRH4PfJrdqKr/J5BSmbKzzLCOcGDoYuoFVYuLa9NhB4ot/e14WVf7AFU5P6ZMgl45zjLDOiKOEwaTJohanAOznoNQbA1ikKpOCLQkKRZG/0A2M2z714hNZtikXJ3FccJg0gRRiysUdGKeDLDYAPGyiExQ1e7mXzJ5FOofKOfKceVcCjV0SerYjeuEwSQJYgJaQpsOi/1WXgP8VkS2i8jHIrJZRD4OsmBpElb/QGwzwyatY9cmDEYriAloCW06LKoGoarW3xCgbP9A+7WnrX/Al9CrMxORIGpxCW06LHpNahEZCIzA66gGQFVfCaJQaRP7/oGgOZCTxiRMuSegJbTpsNiZ1N8ErgdqgKXAccBi4OTgipYese4f6KKSZnMn9OrMJEwCZ5oXW4O4Hjga+JOqjhORw4AfB1es9Mn2DyRZyaO1Enp1Zozriv0P+0RVPwEQkd6q+g5waHDFMknUrdnc1rFrTOiKrUE0icgAYB7eqnIbgbXBFcskkc3mNiZeih3FNNH/9RYRWQjsA/w2sFKZRLLRWsbES1H1dBE5Jfu7qv5eVefjrSltTNGyo7WyKT9stJYxbiu2iWmaiJwHfBfYC3gIL2nfo0EVzCRPmkZr5ZWUdCEmHA58X4p9ta8A7+ENcf0j8ISqnh9YqUxiVVQIg/tXUtmzB607d9GytbXsiQmdlNBkbiYgjnxfig0QA4Fj8YLEp8BBIlLyZZ+IDBCROSLyjoisEJExIjJIRF4SkXf924GlPn+Ygs7CmjTZoa4TH2hg7F0LmfhAAyvXb07+cUtauhATLEe+L8UGiD8Bv1HV0/HmQxwANHTjde8HfquqhwFHAiuAm4AFqjoCWODfd1pqT3btdCVIpnbhIksXYrrCke9LsQHiFGCHiExT1e3APZR4AheRvYETgYcBVLVVVTcB57C7T+NR4NxSnj9MqT3Z5ehqkAxsqKvrK4QlNJmbCYgj35diA8QP8NJrZEcubQZ+VuJrfh5oBmaKyOsi8pCI9Af2V9V1AP7tfiU+f2hsXH/Xg2QgCxc50l7boSAyiJrkcuT7UuwopmNVdbSIvA6gqhtFpNRQ1hMYDfwvVX1VRO6nC7UREZkCTAGora3tZO9g2bj+rgfJQBITxmGxFksXYrrCke9LsQFih4j0ABRARKqBUi/PmoAmVX3Vvz8HL0CsF5EhqrpORIYAG/I9WFVnADMA6urqIm3styysXQ+SgQx1daS9tlMJTOZmAuTA96XYADEdmAvsJyK3A+cD/1rKC6rq30VkjYgcqqorgfHAcv/nMuBO//a5Up4/TKkf109pQbLsiQktHbgxgRDV4i7C/Qyu4wHBG220ouQXFRmFN9muEngfuAKvP+QpoBZYDVygqh929Dx1dXXa2NhYajFMmZSUwru8BUjOkqTGhEBElqhqXaf7FRsgXGQBwrRxYNapMXFRbIAoekU5Y5zmQHutCYFdCITKAoQxJh6sKTF0dlRTyNKDGKcVmvToSPqJNLEaRMqUvOynMWHoqJYQl+HMCWI1iJSx9CDGaR3VEhxJP5EmFiBSxtKDGKd1VEtwJP1EmlgTU8pYehDjtI4mPTqSfiJN7MimjC37aZzWWS0hO5x5wIHerQWHQNlEuRSKfOazMR2xuQ6Bs4lypqCy50Iyppxs0qMzLCwbY4zJywKEMcaYvCxAGGOMycv6IIxxjXXSGkdYgDDOSuVoK0tIZxxiAcKUJOiTtws5oyIJUHFYX9ukhgUI02VhnLwL5Yyae/XYUIboRhagLCGdcYjVWU2XhZHwL+qcUZElNbSEdMYhFiBMl4Vx8s7mjMoVZs6oyAKUJaQzDrEAYbosjJN31DmjIgtQuQnppi7zbq2D2kQkklxMIrIK2AzsAnaqap2IDAJmA8OAVcCFqrqxo+exXEzRCKt9PspRTDt3Zli5YTNXPb7EFlYyiVNsLqYoA0Sdqv4jZ9u/AR+q6p0ichMwUFW/39HzWICITqGTdxKGpmYD4L0vreS8Lx/I4P6V7FfVmwP26UvPnnYlb+Ivjsn6zgFO8n9/FFgEdBggSpGEE5gL8iX8c2FoajnkdlC/uHwD4DUvhTWCyhhXRHU5pMCLIrJERKb42/ZX1XUA/u1+5X7R7Als4gMNjL1rIRMfaGDl+s1kMvFNee6SpCxnGvUIKmNcEVWAGKuqo4EzgGtE5MRiHygiU0SkUUQam5ubu/SiSTmBuSopJ9aoR1AZ44pIAoSqrvVvNwBzgWOA9SIyBMC/3VDgsTNUtU5V66qruzb0LyknMFcVOrHuymisamlRj6BKrUwGtqyHTWu820wm6hKlXugBQkT6i0hV9ndgArAMmA9c5u92GfBcuV/brgyDNbh/Jb+89Mt7nFjvOm8kt/16eaxqaRUVwqH7VzH36rE0fH8cc68eG7t+lNjJ5qB66BS47wjvdsNyCxIRC30Uk4h8Hq/WAF4n+ROqeruIDAaeAmqB1cAFqvphR8/V1VFMSelEddn6j7aztOkjBvTtxabtO3hw0Xu8vmYTDd8fx9CB/aIunnHVlvVeUMhNMzKg1nJQBcTZUUyq+j5wZJ7tLcD4IF8798rQRjEFo6Kigp88v3yPpjyrpcVUmGnHLQeVk1I3qDs7PHPowH5UV/Uu+8Su5s2f8reN22je/Gms2t3LxdrvEyLsJh/LQeWkSCbKlYtLE+Ws+Wq39nNNBvbtxcbtO6zWloez83LCbvKxdTBC5WwTU1JFnZ7aJbmT6CxwFhb5semoCSnsJp/cHFS2kp4z7OiXiQ2hzc/mnhQW6bHprAkpiiafigqvdjLgQO/WgkPk7BMoExtC+1mZjNK6cxc/u+BIfnnplznqwAGABc6sSC8qCq1ct82ffGppx0uTsLkc1sRUJtnO2fbNBWntnM3XfHLXeSO55z9X0rzl01QHzqzsRUUkI746a0JKQ5NPuUdpJbAfxTqpy8jZDscING/+lIkPNHzm5PeTc47gc/v0sT4IIu6DSPu8gyBO5jE6ptZJHYF8GU7TqlDzycH77UXNgL6pDw4Q8bycbBNS+xNkOZqQwpw/UapCTWzdOZkncC6HBQgTiELNJ3179bDgkCOyi4qgmpDi0swSxMk827HfvgYR47kcDn1i4SnHhDabFNcxmzAXA0GMGuqs89sVQYzSSmDHfupqEOVo9418/HoMWFqTlCrnlXmQTVVBNLElsGM/dZ3UhTpPC01oy9fx3LK1tUvPYUxqlKujNoymqtwA1KMSKnrAju2JOLF3pthO6uQegQK6Mva88Ap0GZsUZ7olsU2U5WpmCaOpKtvEtvdQ2NoMvxpnqcbbSV0TU1fGnhea6frUVWOiG79uyiqKocmJbqIsspml0+Me5oigIEY0JUTqahBd6TwtVNtQVeuATYCo1ihPfPqRTjq/izruYab6SODw1HJJXQ2iK52nHdU2Dt2/r3XAxlxUCRbTnrerqOMe5DyN9hI4PLVcUhcgoPix5x2lz7BJcfEX1Yk60hQbvihn/Rd13MMcERRmMIqZVAaIYtlQzWSL6kQddd6uqPtAij7u2aaqoCVweGq5pG6YqzFZYZ4oXVpEqatDvcst6gBlLBeTMZ0Kq4bo2gkx6j6QigphRPVePHXVGHbuytCzRwX77VXe5X9NeURWhxKRHiLyuog8798fLiKvisi7IjJbRKyHyAQuyDXKs1wbtRT12iWZjPJu8xYu/OViTrx7ERf+cjHvNm9JzlyQBImyke16YEXO/buAe1V1BLAR+EYkpTKmzKK+Ym8v6jxZrgVMU1gkTUwiUgOcCdwOfEdEBDgZ+Lq/y6PALcAvoiifMeXkwqilXFEPvnAtYJrCouqDuA/4HlDl3x8MbFLVnf79JmBoKU+8Y8cOmpqa+OSTT7pfSlM2ffr0oaamhl69epXl+eK0OFPUo5byiXKYtmsB0xQWeoAQkbOADaq6REROym7Os2veBkkRmQJMAaitrf3M35uamqiqqmLYsGF4FRMTNVWlpaWFpqYmhg8f3u3nc63TtzNRX7G7xsWA2RVxujjprtCHuYrIHcClwE6gD7A3MBc4Dficqu4UkTHALap6WkfPlW+Y64oVKzjssMMsODhGVXnnnXf4whe+0O3ninqYpum+uJ5k43ZxUoiz2VxV9QeqWqOqw4CLgd+p6iXAQuB8f7fLgOdKfQ0LDu4p52dibdjxF8bosSCU0sEe58y9Lk0V/D5eh/Vf8fokHo64PCUTEW688ca2+/fccw+33HJLh4+ZN28ey5cv73CfI488kkmTJhX8+6pVqzjiiCO6VNZp06bx8ssvA3Dfffexbdu2tr/99Kc/7dJzATzyyCNce+21XX5cV0Q9TNOkV1cvTqJKCFkukQYIVV2kqmf5v7+vqseo6v9Q1QtU9dMoy9YdvXv35tlnn+Uf//hH0Y/pLECsWLGCTCbDK6+8wtatW8tRTABuvfVWTjnlFKA8ASIMUQ/TNOnV1YuTuA/pTf1M6iDaQnv27MmUKVO49957uf322/f42wcffMCVV15Jc3Mz1dXVzJw5k6amJubPn8/vf/97brvtNp555hkOPvjgPR73xBNPcOmll7JixQrmz5/fVpNYsmQJV155Jf369eOEE05o2/+RRx5h3rx57Nq1i2XLlnHjjTfS2trK448/Tu/evXnhhRcYNGgQl19+OWeddRZr165l7dq1jBs3jn333Zdjjz2W7du3M2rUKL74xS8ya9Ys/uM//oPp06fT2trKscceywMPPECPHj2YOXMmd9xxB0OGDOGQQw6hd+9g+wGs09dEpasd7HFvDnWpiSl0QVb/rrnmGmbNmsVHH320x/Zrr72WyZMn8+abb3LJJZdw3XXXcfzxx3P22Wdz9913s3Tp0s8EB4DZs2dz0UUXMWnSJOrr69u2X3HFFUyfPp3Fixd/5jHLli3jiSee4M9//jM/+tGP6NevH6+//jpjxozhscce22Pf6667jgMOOICFCxeycOFC7rzzTvr27cvSpUuZNWsWK1asYPbs2TQ0NLB06VJ69OjBrFmzWLduHTfffDMNDQ289NJLnTaTlUtc27BNvOVenDR8fxxzrx7bYQd13JtDUx0ggqz+7b333kyePJnp06fvsX3x4sV8/evefMBLL72UP/7xj50+11/+8heqq6s56KCDGD9+PK+99hobN27ko48+YtOmTXzlK19pe75c48aNo6qqiurqavbZZx++9rWvAfClL32JVatWden9LFiwgCVLlnD00UczatQoFixYwPvvv8+rr77KSSedRHV1NZWVlVx00UVdel5j4qYrFydxbw5NdRNT0NW/qVOnMnr0aK644oqC+xQzuqe+vp533nmHYcOGAfDxxx/zzDPPcP7553f4+NymnoqKirb7FRUV7Ny5s9DD8lJVLrvsMu644449ts+bN89GjRlTQNybQ1Ndgwi6+jdo0CAuvPBCHn5494Cs448/nieffBKAWbNmtfUbVFVVsXnz5s88RyaT4emnn+bNN99k1apVrFq1iueee476+noGDBjAPvvs01YLmTVrVrfK274MvXr1YseOHQCMHz+eOXPmsGFMutGfAAAMPElEQVTDBgA+/PBDPvjgA4499lgWLVpES0sLO3bs4Omnn+5WGYxJmjg3h6Y6QIRR/bvxxhv3GM00ffp0Zs6cyciRI3n88ce5//77Abj44ou5++67Oeqoo3jvvffa9n/llVcYOnQoQ4fuzjxy4oknsnz5ctatW8fMmTO55pprGDNmDH377hnsumrKlCmcccYZjBs3ru3+yJEjueSSSzj88MO57bbbmDBhAiNHjuTUU09l3bp1DBkyhFtuuYUxY8ZwyimnMHr06G6VoZA4jyU3Jq4St2DQihUrujRbN64zOuOoq59NVlJmrxrjCmdnUrsmztW/tIj7WHJj4ir1AcK4L+5jyY2JKwsQxnlxH0tuTFxZgDDOi/tYcmPiKtXzIEw8xH0suTFxZQHCxEKUK6AZk1bWxBSQuXPnIiK88847ef9++eWXM2fOnKKfb+3atZx/vrdcxtKlS3nhhRfa/rZo0SL+67/+q8tlHDZsWJcyzhpj0sUCREDq6+s54YQT2mZNd9cBBxzQFlDKFSCMMaYjFiAyGdiyHjat8W4zmW4/5ZYtW2hoaODhhx9uCxCqyrXXXsvhhx/OmWee2ZayArwr+R/+8IeMGTOGuro6XnvtNU477TQOPvhgHnzwQWD3YkCtra1MmzaN2bNnM2rUKO666y4efPBB7r33XkaNGsUf/vAHmpubOe+88zj66KM5+uijaWhoAKClpYUJEyZw1FFHcdVVVxHnSZLGmOCluw8ik4ENy+HJSbBpNQyohYvrYb/DoaL02Dlv3jxOP/10DjnkEAYNGsRrr73GqlWrWLlyJW+99Rbr16/n8MMP58orr2x7zIEHHsjixYu54YYbuPzyy2loaOCTTz7hi1/8It/+9rfb9qusrOTWW2+lsbGRn//85wBs376dvfbai+9+97sAfP3rX+eGG27ghBNOYPXq1Zx22mmsWLGCH//4x5xwwglMmzaNX//618yYMaPk92iMSb50B4htzbuDA3i3T06Cb74Me+1f8tPW19czdepUwMuxVF9fz44dO5g0aRI9evTggAMO4OSTT97jMWeffTbgpeLesmULVVVVVFVV0adPHzZt2tSl13/55Zf3WJfh448/ZvPmzbzyyis8++yzAJx55pkMHDiw5PdojEm+dAeIna27g0PWptXe9hK1tLTwu9/9jmXLliEi7Nq1CxFh4sSJRaXmzk3Lnb3f1dTcmUyGxYsX503eZ6m5jTHFSncfRM9Kr1kp14Bab3uJ5syZw+TJk/nggw9YtWoVa9asYfjw4QwaNIgnn3ySXbt2sW7dOhYuXFjya7RPy93+/oQJE9qan8Dr1AYvC2w2JfhvfvMbNm7cWHIZTPxZhlzTmdADhIj0EZE/i8gbIvK2iPzY3z5cRF4VkXdFZLaIBD9Ntl+11+eQDRLZPoh+1SU/ZX19PRMnTtxj23nnncff//53RowYwZe+9CX+5V/+pW0VuFKMGzeO5cuXM2rUKGbPns3XvvY15s6d29ZJPX36dBobGxk5ciSHH354W0f3zTffzCuvvMLo0aN58cUXqa2t7eSVTFIFudyuSY7Q032L18bRX1W3iEgv4I/A9cB3gGdV9UkReRB4Q1V/0dFzlSPdN5mM1xexs9WrOfSr7lYHtSms1HTfpvyaN3/KxAca9kiCWDOwL3OvHmsTElPA2XTf6tni3+3l/yhwMpCdOfYocG4oBaqo8DqkBxzo3VpwMClgGXJNMSI5G4pIDxFZCmwAXgLeAzaparY3tgkYWujxxpjusQy5phiRBAhV3aWqo4Aa4BggX7tD3rYvEZkiIo0i0tjc3BxkMY1JLMuQa4oR6TBXVd0kIouA44ABItLTr0XUAGsLPGYGMAO8PogC+9hwTsfYrG23WIZcU4woRjFVi8gA//e+wCnACmAhcL6/22XAc6U8f58+fWhpabETkkNUlZaWFvr06RN1UUwOW27XdCaKGsQQ4FER6YEXoJ5S1edFZDnwpIjcBrwOPFzKk9fU1NDU1IQ1P7mlT58+1NTURF0MY0wXhB4gVPVN4Kg829/H64/oll69ejF8+PDuPo0xxqSejek0xhiTlwUIY4wxeVmAMMYYk1foqTbKSUSagQ9KfPi+QFzW27SyBidO5bWyBiNOZYXylPcgVe006VysA0R3iEhjMblIXGBlDU6cymtlDUacygrhlteamIwxxuRlAcIYY0xeaQ4QcVqQ2coanDiV18oajDiVFUIsb2r7IIwxxnQszTUIY4wxHUhFgHBqmdMi+OtlvC4iz/v3nSwngIisEpG3RGSpiDT62waJyEt+eV8SkYFRlxNARAaIyBwReUdEVojIGBfLKiKH+scz+/OxiEx1saxZInKD/7+1TETq/f85J7+3InK9X863RWSqv82JYysi/y4iG0RkWc62vGUTz3QR+auIvCkio8tdnlQECOBT4GRVPRIYBZwuIscBdwH3quoIYCPwjQjLmOt6vAy3Wa6WM2ucqo7KGXp3E7DAL+8C/74L7gd+q6qHAUfiHWPnyqqqK/3jOQr4MrANmIuDZQUQkaHAdUCdqh4B9AAuxsHvrYgcAXwLL+/bkcBZIjICd47tI8Dp7bYVKtsZwAj/ZwrQ4RLNJVHVVP0A/YDXgGPxJpv09LePAf7TgfLV+F+Ck4HnAXGxnDnlXQXs227bSmCI//sQYKUD5dwb+G/8fjeXy9qufBOABpfLirf64xpgEF4C0OeB01z83gIXAA/l3P/fwPdcOrbAMGBZzv28ZQN+CUzKt1+5ftJSg4jTMqf34X1hM/79wbhZziwFXhSRJSIyxd+2v6quA/Bv94usdLt9HmgGZvrNdw+JSH/cLGuui4F6/3cny6qqfwPuAVYD64CPgCW4+b1dBpwoIoNFpB/wVeBAHD22vkJlywbmrLIf49QECO3GMqdhEZGzgA2quiR3c55dXRp6NlZVR+NVd68RkROjLlABPYHRwC9U9ShgK4400RTit9mfDTwddVk64reJnwMMBw4A+uN9H9qL/Hurqivwmr5eAn4LvAHs7PBB7gr83JCaAJGlqpuAReQsc+r/qeAypyEaC5wtIquAJ/Game7DvXK2UdW1/u0GvHbyY4D1IjIEwL/dEF0J2zQBTar6qn9/Dl7AcLGsWWcAr6nqev++q2U9BfhvVW1W1R3As8DxOPq9VdWHVXW0qp4IfAi8i7vHFgqXrQmv9pNV9mOcigAhAS9zWi6q+gNVrVHVYXhNC79T1UtwrJxZItJfRKqyv+O1ly8D5uOVExwpr6r+HVgjIof6m8YDy3GwrDkmsbt5Cdwt62rgOBHpJyLC7mPr6vd2P/+2FvgnvGPs6rGFwmWbD0z2RzMdB3yUbYoqm6g7jULq9BmJt4zpm3gnsGn+9s8Dfwb+ileN7x11WXPKfBLwvMvl9Mv1hv/zNvAjf/tgvI72d/3bQVGX1S/XKKDR/x7MAwY6XNZ+QAuwT842J8vql+3HwDv+/9fjQG+Hv7d/wAtgbwDjXTq2eMFqHbADr4bwjUJlw2ti+r94/alv4Y0iK2t5bCa1McaYvFLRxGSMMabrLEAYY4zJywKEMcaYvCxAGGOMycsChDHGmLwsQBgTMT/tw0IR2SIiP4+6PMZk9ex8F2NMwD7BSxp3hP9jjBOsBmFMASLyz+KtI7JURH4pIgf5Ofn3FZEKEfmDiEzw953nJyx8OydpIX6t4C7/by+LyDEiskhE3heRswFUdauq/hEvUBjjDAsQxuQhIl8ALsJLRjgK2AV8BS/R24PAjcByVX3Rf8iVqvploA64TkQG+9v7A4v8v20GbgNOBSYCt4b1fowphTUxGZPfeLzFev7ipReiL16m3VtE5ALg23ipO7KuE5GJ/u8H4i3i0gK04mUNBS8dwqequkNE3sLL+2+MsyxAGJOfAI+q6g/22OitIVDj390L2CwiJ+ElgByjqttEZBHQx99nh+7OZ5PBW90QVc3kZDo1xknWxGRMfguA83Myfw4SkYPwmphmAdOAX/n77gNs9IPDYXip5I2JPbuCMSYPVV0uIv+Kt1peBV52ze8AR+P1S+wSkfNE5ArgCeDbIvIm3rKPf+rq6/lrgOwNVIrIucAEVV1eprdjTEksm6sxxpi8rInJGGNMXhYgjDHG5GUBwhhjTF4WIIwxxuRlAcIYY0xeFiCMMcbkZQHCGGNMXhYgjDHG5PX/AZzbS+tHPFUrAAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"sns.scatterplot(x='exam1',y='exam2',data=data[data['decision']==0],label='Not Admitted');\n",
"sns.scatterplot(x='exam1',y='exam2',data=data[data['decision']==1],label='Admitted');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 1.2 Implementation\n",
"We will add another dimension to the Design Matrix to accomodate the $\\theta_0$ intercept term. We will also initialize the fitting parameters to 0."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"X : (100, 3)\n",
"y : (100, 1)\n",
"theta : (1, 3)\n"
]
}
],
"source": [
"m=data.shape[0]\n",
"X=np.ones(shape=(m,3))\n",
"X[:,1:]=data.values[:,:2]\n",
"print(f'X : {X.shape}')\n",
"\n",
"y=data.values[:,2]\n",
"y=y[:,np.newaxis]\n",
"print(f'y : {y.shape}')\n",
"\n",
"theta=np.zeros(shape=(1,X.shape[1]))\n",
"print(f'theta : {theta.shape}')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### 1.2.1 Sigmoid/Logistic Function\n",
"\n",
"Before starting with the implementation, let's try to understand the sigmoid function. \n",
"For large positive values of $x$, sigmoid function is close to 1, while for large negative values, sigmoid function is close to 0. And sigmoid of 0 is exactly 0.5 .\n",
"\n",
"$$ \\sigma(x)=\\frac{1}{1+e^{-x}} $$"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def sigmoid(z):\n",
" sigma=1/(1+np.exp(-z))\n",
" return sigma"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VPW9//HXZyYrJGFJwh4FBBRQQYiC2goqIlIvorUKt70V24re1l791WuLtdr+tP4eWK2/3v5qq7i3Lri0thQpghVbrQs7QkAkIktIIGEnhKzz/f0xwziECQSSk5NJ3s/HI4+c5Tszn5w5mfec79nMOYeIiAhAwO8CRESk9VAoiIhIlEJBRESiFAoiIhKlUBARkSiFgoiIRCkUREQkSqEgIiJRCgUREYlK8ruAE5WTk+P69u3rdxkiIgll2bJlO51zucdrl3Ch0LdvX5YuXep3GSIiCcXMNjemnbqPREQkSqEgIiJRCgUREYlKuH0K8dTU1FBUVERlZaXfpUgzSEtLo0+fPiQnJ/tdiki70yZCoaioiMzMTPr27YuZ+V2ONIFzjl27dlFUVES/fv38Lkek3fGs+8jMnjazUjNb08B8M7Nfm1mhmX1sZiNO9rUqKyvJzs5WILQBZkZ2dra2+kR84uU+hWeBCceYfwUwMPIzHfhdU15MgdB26L0U8Y9n3UfOuX+aWd9jNLkK+L0L3w/0QzPrbGY9nXMlXtUkIonPOUdtyFFVG6K6NkRVbR01tY7qujqqax21oRA1dY7auhB1IUdNyFEXClEX4ovfzhEKOULOURdyOAch5whFfrsjhsO/w68dmRYZBgiPfTF+uMYv5h/dtn77I/6+I//YI+ZdOrg7w/I6n9yCayQ/9yn0BrbGjBdFph0VCmY2nfDWBKecckqLFHeiHnjgAV588UWCwSCBQIDHH3+cUaNGMXbsWB5++GHy8/MpLy/nzjvvZMGCBWRlZREIBLjlllu46aab2LRpE/369eMnP/kJ999/PwA7d+6kZ8+e3HzzzfzmN78BYNasWTzyyCMAZGVl8cgjj/ClL30J4IjX6tu3L5mZmQDU1dVxzTXXcM8995CamnpU7du3b+f2229nyZIlpKam0rdvX371q19RXFzMww8/zNy5c6Ntp02bxpVXXsm1117L2LFjKSkpIS0tjZSUFJ544gmGDx/OtGnTOP/887n55pujj/vzn//MrFmzmDdvHsFgkLPOOis6b8qUKcyYMaOZ3xFprWrrQuw6WM3O8ip2H6xm98Fq9hysZt+hWvYdquFAZQ3lVbUcqKylvKqWQ9V1VNSEfx+qrqOyNvxh317Ebjh3y0pr06EQr48g7jvtnJsFzALIz89vdWvDBx98wNy5c1m+fDmpqans3LmT6urqo9p95zvfoX///mzYsIFAIEBZWRlPP/10dH7//v2ZO3duNBReffVVhg4dGp0/d+5cHn/8cd577z1ycnJYvnw5kydPZvHixfTo0eOo11u0aBE5OTmUl5czffp0pk+fznPPPXdEG+ccV199NTfccAOzZ88GYOXKlezYsaNRf/sLL7xAfn4+zzzzDHfeeScLFy5k6tSpzJw584hQmD17NlOnTgUgPT2dlStXNur5JfHUhRzFew+xcedBPi8rZ+ueQ2zbc4htew+xfX8lu8qraOgzvWNKkE7pyWSkJZGRmkRmWhLds1LpmJJEWkqQ9OTwT2pSgNTkAKlJQZKDAVKSwj/JASM5GCApGPkdMJKCRjAQIGhGMHD4BwJmBCLTAgHDgGDAMAMjPN0IfyibHZ4eftzhNvV7Og9P/2L48HSLGY5t3/q6Sv0MhSIgL2a8D1DsUy1NUlJSQk5OTvRbeE5OzlFtPvvsMxYvXsyLL75IIBDelZObm8uPfvSjaJv09HQGDx7M0qVLyc/P5+WXX+a6666juDi8WB588EEeeuih6POPGDGCG264gUcffTQaJPFkZGTw2GOPkZeXx+7du+natWt03qJFi0hOTuaWW26JThs+fDgA77zzTqOXwfnnn89DDz0EwLhx45g2bRolJSX07NmTiooK3nrrLZ544olGP58khqraOtZs28+qrXtZV7KfT7Yf4NMdB6iqDUXbpCUH6N05nd5dOjCkZxbds1LJzUojNyOFrh1T6doxhS4dkslKTyY5qFOn/OZnKMwBbjWz2cAoYF9z7E/4338tYG3x/iYXF2tIryx++m9DG5w/fvx47rvvPgYNGsS4ceO4/vrrGTNmzBFtCgoKGDZsWDQQGjJlyhRmz55Njx49CAaD9OrVKxoKBQUFjBw58oj2+fn5R337jycrK4t+/fqxYcMGRo0aFZ2+Zs2ao57zZMyfP5/JkycDEAwGueaaa3jllVe47bbbmDNnDhdffHG0O+vQoUPR4AG46667uP7665tcg3ivqraOZZv38N6GnXywcRcF2/ZTXRcOgJyMVAb3zOQ/Rp/KgG4Z9MvpSL/cjuRmpLbKb8QSn2ehYGYvAWOBHDMrAn4KJAM45x4D5gETgUKgArjRq1q8lpGRwbJly3j33XdZtGgR119/PTNnzmTatGkNPuaBBx7g1VdfpbS0NPqhDzBhwgTuueceunfv3qgPSudco//hXEN7thrQ0PPGTv/617/OwYMHqaurY/ny5dHpU6dO5c477+S2225j9uzZfPOb34zOU/dRYtlfWcPb60qZt7qEf24oo7ImRDBgDM/rzI0X9uWcU7ow4pTOdMtK87tUaQZeHn009TjzHfC95n7dY32j91IwGGTs2LGMHTuWs846i+eee+6IUBgyZAirVq0iFAoRCAS4++67ufvuu8nIyDjieVJSUhg5ciS//OUvKSgo4K9//esRz7Fs2TIuueSS6LTly5czZMiQ49Z34MABNm3axKBBg46YPnToUF577bW4j8nOzmbPnj1HTNu9e/cR3WMvvPACw4YNY8aMGXzve9/jT3/6EwAXXnghJSUlrFq1ivfffz+6v0ISQyjkeLdwJy8v2cJba0uprgvRIyuN6/LzuGhgLqP6dyUzTWect0Vt4oxmv61fv55AIMDAgQOB8I7aU0899Yg2AwYMID8/P3p0UTAYpLKyMu639zvuuIMxY8aQnZ19xPQf/vCH/OhHP2L+/PlkZ2ezcuVKnn32WT766KNj1ldeXs53v/tdJk+eTJcuXY6Yd8kll/DjH/+YJ554gptuugmAJUuWUFFRwejRoykuLmbdunUMHjyYzZs3s2rVqiO6fgCSk5P5+c9/zmmnnRZta2Zcd9113HDDDUycOJG0NH2LTATlVbW88OFmfv/BZrbtPUSXDsl8Y/SpfOXsnpyT15lAQN1AbZ1CoRmUl5fz/e9/n71795KUlMSAAQOYNWvWUe2efPJJ7rzzTgYMGEDXrl1JT0/nwQcfPKrd0KFDjzjq6LBJkyaxbds2LrjgAsyMzMxMnn/+eXr27Bm3rosvvjh8vHUoxNVXX80999xzVBsz4/XXX+f2229n5syZpKWlRQ9JTU1N5fnnn+fGG2+ksrKS5ORknnzySTp16nTU86Snp3PHHXfw8MMP89RTTwHhLqSHHnqImTNnHtG2/j6FCRMmHNVGWtbeimqe/tcmnnt/E/sO1TC6f1dmXHEG44d2JzUp6Hd50oLsRPuZ/Zafn+/q32Tn8LdTaTv0nraMmroQz3+4mV+9tYF9h2oYP6Q73714AMM9PhZeWp6ZLXPO5R+vnbYURNqpdzeU8dM5BWwsO8iXBuRw91cGM7hnlt9lic8UCiLtTEV1Lf9n3jqe/3AL/XI68tQN+VxyRjcdNipAGwqFEzk0U1q3ROvSTCTLNu/hB6+sZMvuCr7zpX789+Wnk5asfQbyhTYRCmlpaezatUuXz24DDt9PQUcrNb+XFm/h3r+soXtWGi/dNJrR/bOP/yBpd9pEKPTp04eioiLKysr8LkWaweE7r0nzqKkL8fO5a3nug81cNCiX/zf1HDql6xwDia9NhEJycrLu0iUSR0V1LTf/YRnvbtjJTV/ux4wrBhPUuQZyDG0iFETkaOVVtXzrmSUs3bybX3z1bK47N+/4D5J2T6Eg0gbtO1TDtGcW83HRPn499RyuPLuX3yVJglAoiLQxB6tq+eZTH7G2ZD+//foILh969L02RBqiUBBpQ2rrQnz/pRWs3raPx/8jn8uGdPe7JEkwCgWRNsI5x0/nFPD2J6U8cPWZCgQ5KbrNkUgb8fg/N/LCR1u4ZcxpfH3Uqcd/gEgcCgWRNuDdDWU8OP8Trjy7Jz+8/HS/y5EEplAQSXCl+yv5Xy+vZEBuBg9dO0z3PJAm0T4FkQRWF3LcNnsl5VW1vHjTaNJTdB0jaRqFgkgC+83bhXywcRe/+OrZDOqe6Xc50gao+0gkQa3cupf/+funTB7ei6/l61pR0jwUCiIJqKYuxIw/fkxuZir3TT5TVweWZqPuI5EENOufG/lk+wFm/cdIstJ0xVNpPtpSEEkwG8vK+Z+/b2DiWT0Yr0tYSDNTKIgkEOccd/1pNWlJAX42aajf5UgbpFAQSSBzVhXz0ee7+fHEwXTL1N3ppPkpFEQSRGVNHb+Yv54hPbO4Ll/3RhBvKBREEsSz729i295D/OQrg3XWsnhGoSCSAHaVV/Ho24VcekY3LhiQ43c50oYpFEQSwK//voGKmjrumniG36VIG6dQEGnlPt95kBc+2sKUc/MY0E2XshBveRoKZjbBzNabWaGZzYgz/xQzW2RmK8zsYzOb6GU9Iono0UWFBAPGbeMG+l2KtAOehYKZBYFHgSuAIcBUMxtSr9lPgFecc+cAU4DfelWPSCLauruC11ds499HnaJDUKVFeLmlcB5Q6Jzb6JyrBmYDV9Vr44CsyHAnoNjDekQSzm/fKSRoxs0XneZ3KdJOeHnto97A1pjxImBUvTY/AxaY2feBjsA4D+sRSSjb9h7itWVFXH9uHj06aStBWoaXWwrxDqR29canAs865/oAE4E/mNlRNZnZdDNbamZLy8rKPChVpPV5/B+f4RzcMkZbCdJyvAyFIiD2tMs+HN099G3gFQDn3AdAGnDUQdjOuVnOuXznXH5ubq5H5Yq0HqX7K5m9ZCvXjuxDny4d/C5H2hEvQ2EJMNDM+plZCuEdyXPqtdkCXApgZoMJh4I2BaTde+6DTdTUhfjPsdpKkJblWSg452qBW4E3gXWEjzIqMLP7zGxSpNkdwE1mtgp4CZjmnKvfxSTSrlTW1PHiR1u4bHB3Ts3u6Hc50s54epMd59w8YF69affGDK8FLvSyBpFE8+cV29hTUcONF/bzuxRph3RGs0gr4pzj6X99zuCeWYzu39XvcqQdUiiItCLvf7aLT3eUc+OFfXXfZfGFQkGkFXn6vc/J7pjCpGG9/C5F2imFgkgrsWnnQd5eX8rXR51CWnLQ73KknVIoiLQSLy7eQtCMb4w+1e9SpB1TKIi0AtW1If64rIhLB3ejW5YuaSH+USiItAIL1+5g18Fqppx3it+lSDunUBBpBWYv2ULvzulcNFCXcRF/KRREfLZ1dwXvbtjJ1/L7EAzoMFTxl0JBxGevLN2KGVyXn3f8xiIeUyiI+Ki2LsQrS7cyZlAuvTqn+12OiEJBxE//+LSMHfurmHKudjBL66BQEPHRH5cXkd0xhUsHd/O7FBFAoSDim32HanhrXSn/NqwXyUH9K0rroDVRxCd/W11CdW2Iq8/p7XcpIlEKBRGfvL5iG/1zOnJ2n05+lyISpVAQ8UHRngo++nw3V5/TW5fIllZFoSDig7+sLAZgsrqOpJVRKIi0MOccr6/Yxrl9u5DXtYPf5YgcQaEg0sIKivdTWFqurQRplRQKIi3szyu2kRw0vnJWT79LETmKQkGkBYVCjjdWl3DRwFw6d0jxuxyRoygURFrQiq17KNlXyZXDtJUgrZNCQaQFzf24hJSkAOMGd/e7FJG4FAoiLSQUcsxbXcKYQblkpiX7XY5IXAoFkRaydPMeduyv4sqz1XUkrZdCQaSFvPFxMalJAS5V15G0YgoFkRZQF3LMW7OdS87oRkZqkt/liDRIoSDSAhZ/vpuyA1V8RV1H0sopFERawBuri0lLDnDJGbqZjrRuCgURj4VCjjcLdnDx6d3okKKuI2ndPA0FM5tgZuvNrNDMZjTQ5jozW2tmBWb2opf1iPhhxdY9lB2oYsKZPfwuReS4PPvaYmZB4FHgMqAIWGJmc5xza2PaDATuAi50zu0xM21bS5szf812UoLqOpLE4OWWwnlAoXNuo3OuGpgNXFWvzU3Ao865PQDOuVIP6xFpcc455hds58IB2TphTRKCl6HQG9gaM14UmRZrEDDIzP5lZh+a2YR4T2Rm081sqZktLSsr86hckea3tmQ/W3cfUteRJAwvQyHePQZdvfEkYCAwFpgKPGlmnY96kHOznHP5zrn83NzcZi9UxCvz12wnYOhaR5IwvAyFIiAvZrwPUBynzV+cczXOuc+B9YRDQqRNmL9mO6P6ZZOdkep3KSKN4mUoLAEGmlk/M0sBpgBz6rX5M3AxgJnlEO5O2uhhTSItprC0nA2l5eo6koTiWSg452qBW4E3gXXAK865AjO7z8wmRZq9Cewys7XAIuBO59wur2oSaUlvFmwHYPxQdR1J4vD0TBrn3DxgXr1p98YMO+AHkR+RNmVBwXaG5XWmZ6d0v0sRaTSd0Szige37KllVtI/LtZUgCUahIOKBhet2ADB+iEJBEotCQcQDCwq20z+nI6flZvhdisgJUSiINLP9lTV8uHEXlw3pjlm803VEWi+Fgkgze2d9GTV1TkcdSUJSKIg0swUF28nJSGV4Xhe/SxE5YQoFkWZUVVvHO+vLGDe4G8GAuo4k8SgURJrRhxt3U15Vq64jSVgKBZFmtKBgOx1SglxwWo7fpYicFIWCSDMJhRwL1+5gzKBc0pKDfpcjclIUCiLNZPW2fZQeqOIynbAmCazBUDCzeWbWt+VKEUlsC9fuIBgw3XZTEtqxthSeBRaY2d1mpvsIihzHwrU7OLdvFzp3SPG7FJGT1uBVUp1zr5jZG8C9wFIz+wMQipn/SAvUJ5IQtuyqYP2OA9xz5RC/SxFpkuNdOrsGOAikApnEhIKIfGHB2si9E7Q/QRJcg6FgZhOARwjfLW2Ec66ixaoSSTAL1+7gjB6Z5HXt4HcpIk1yrC2Fu4GvOecKWqoYkUS052A1Szbt5nsXD/C7FJEmO9Y+hS+3ZCEiiertT0oJOXQoqrQJOk9BpIkWrt1Bj6w0zurdye9SRJpMoSDSBJU1dfxzQxnjhnTTvROkTVAoiDTB+5/tpKK6jvFDevhdikizUCiINMGCgh1kpiYxun+236WINAuFgshJqgs53lq3g7FndCMlSf9K0jZoTRY5SSu27GFnebVOWJM2RaEgcpIWrt1BctAYe3qu36WINBuFgshJcM7xZsF2zj8th8w0XS9S2g6FgshJKCwtZ9OuCnUdSZujUBA5CQvW7gB0FrO0PQoFkZOwYO0OhuV1pntWmt+liDQrhYLICSree4hVW/eq60jaJE9DwcwmmNl6Mys0sxnHaHetmTkzy/eyHpHmsKAgfO+EK87UWczS9ngWCmYWBB4FrgCGAFPN7KjbUplZJvBfwEde1SLSnOYXbGdQ9wz652b4XYpIs/NyS+E8oNA5t9E5Vw3MBq6K0+5+4BdApYe1iDSLXeVVLP58NxOGaitB2iYvQ6E3sDVmvCgyLcrMzgHynHNzPaxDpNm8tW4HIQeXq+tI2igvQyHedYRddKZZAPi/wB3HfSKz6Wa21MyWlpWVNWOJIidm/prtnNK1A0N6ZvldiognvAyFIiAvZrwPUBwzngmcCbxjZpuA0cCceDubnXOznHP5zrn83FxdUkD8sb+yhvcKdzLhzB66d4K0WV6GwhJgoJn1M7MUYAow5/BM59w+51yOc66vc64v8CEwyTm31MOaRE7aok9KqalzXK79CdKGeRYKzrla4FbgTWAd8IpzrsDM7jOzSV69rohX5q/ZTrfMVM7J6+x3KSKeSfLyyZ1z84B59abd20DbsV7WItIUFdW1vLO+jGtH9iEQUNeRtF06o1mkERZ9UsahmjomntXT71JEPKVQEGmEuR8Xk5uZynn9uvpdioinFAoix3Gwqpa3Pyll4pk9CKrrSNo4hYLIcfz9k1KqakN85exefpci4jmFgshxzF1VTPesVPJP7eJ3KSKeUyiIHMOByhre+bSMiWf11FFH0i4oFESO4e/rSqmuDXHl2TrqSNoHhYLIMcz9uJhendI4J09dR9I+KBREGrCvooZ/frqTK9R1JO2IQkGkAfPWlFBdF2Ly8N7HbyzSRigURBrw+vJtnJbbkTN76zLZ0n4oFETi2Lq7gsWbdnPNiD66TLa0KwoFkTj+snIbAJOG6YQ1aV8UCiL1OOd4fcU2zuvblbyuHfwuR6RFKRRE6lm9bR+flR3k6hHawSztj0JBpJ7XV2wjJRhg4pk6YU3aH4WCSIzauhB/XVXMpYO70alDst/liLQ4hYJIjLc/KWVneTVXn6OuI2mfFAoiMV5espXczFQuPqOb36WI+EKhIBJRsu8Qi9aX8rWRfUgO6l9D2iet+SIRry4tIuTg+nPz/C5FxDcKBREgFHK8vGQrFw7I5tTsjn6XI+IbhYII8G7hTrbtPcSUc0/xuxQRXykURICXl2yhS4dkxg/t7ncpIr5SKEi7V3agioVrd3DNiD6kJgX9LkfEVwoFafde/GgLNXWOfx+lriMRhYK0a9W1IZ7/aDNjT8/ltNwMv8sR8Z1CQdq1N1YXU3agihsv7Od3KSKtgkJB2i3nHE+/t4kB3TK4aGCO3+WItAoKBWm3lm3ew+pt+5h2QV/dXU0kQqEg7dYz/9pEp/RkrtF9E0SiPA0FM5tgZuvNrNDMZsSZ/wMzW2tmH5vZ383sVC/rETls295DzC/YzpTz8uiQkuR3OSKthmehYGZB4FHgCmAIMNXMhtRrtgLId86dDbwG/MKrekRiPf6PzwgY3HB+X79LEWlVvNxSOA8odM5tdM5VA7OBq2IbOOcWOecqIqMfAn08rEcEgB37K5m9ZCvXjuxDr87pfpcj0qp4GQq9ga0x40WRaQ35NvC3eDPMbLqZLTWzpWVlZc1YorRHj/9jI3Uhx3+OGeB3KSKtjpehEO9wDhe3odk3gHzgoXjznXOznHP5zrn83NzcZixR2pud5VW8uHgzk4f35pTsDn6XI9LqeLmHrQiIvTB9H6C4fiMzGwfcDYxxzlV5WI8IT7y7keraEN+7+DS/SxFplbzcUlgCDDSzfmaWAkwB5sQ2MLNzgMeBSc65Ug9rEWHPwWr+8MFm/m1YL/rrkhYicXkWCs65WuBW4E1gHfCKc67AzO4zs0mRZg8BGcCrZrbSzOY08HQiTfbookIO1dRx68XalyDSEE8P0HbOzQPm1Zt2b8zwOC9fX+SwzbsO8twHm7huZB4Du2f6XY5Iq6UzmqVd+MX89SQFAvxg/CC/SxFp1RQK0uYt27ybN1aXcPOY/nTPSvO7HJFWTaEgbZpzjp+/sY5umalMv6i/3+WItHoKBWnT5qwqZsWWvfz3+NN1jSORRlAoSJu1t6Ka++eu5ew+nfjqSF1BRaQx9NVJ2qwH3ljHnooafv+tUQQDul+CSGNoS0HapPc27OTVZUXcfFF/hvTK8rsckYShUJA251B1HT9+fTX9cjryX5cO9LsckYSi7iNpc2b+bR1bdlcwe/po0pKDfpcjklC0pSBtyvw1JTz3wWa+dWE/RvfP9rsckYSjUJA2Y+vuCu587WOG9enEjCvO8LsckYSkUJA2obo2xK0vrQDgN/8+gpQkrdoiJ0P7FCThOee4f+5aVm3dy+++PoK8rrp5jsjJ0tcpSXhPvfc5f/hwM9Mv6s8VZ/X0uxyRhKZQkIQ2b3UJP39jHRPP6sGMCdqPINJUCgVJWEs37eb2l1cy8tQuPHLdcAI6a1mkyRQKkpCWbNrNtGeW0LtzOk98M1/nI4g0E4WCJJz3P9vJN59aTLesVF66aTRdO6b4XZJIm6FQkITyzvpSbnxmCX26pDN7+mh6dNJNc0Sakw5JlYTgnOOZf23i52+s5fQeWTz/7fPIzkj1uyyRNkehIK1eVW0dP3l9Da8uK2L8kO48cv1wMlK16op4Qf9Z0qp9VlbOD15eyaqiffzXJQO4fdwgHWUk4iGFgrRKoZDjuQ82MfNvn5CeEuSxb4xgwpk6MU3EawoFaXXWFu/nZ38tYPHnu7n49Fwe/OrZdMvSDmWRlqBQkFaj7EAVjyxcz+wlW+mUnszMa87i+nPzMFN3kUhLUSiI77bvq+TJdzfy4uItVNeGuPGCftx26UA6dUj2uzSRdkehIL5wzrF62z5e+HALr6/YRp1zTBrWi1svGcBpuRl+lyfSbikUpEWVHqjkb6u38/KSrawt2U9acoCv5ffhljGn6ZLXIq2AQkE85Zzjs7Jy/vHpTuavKWHp5j04B0N7ZXH/5DOZNKwXndLVTSTSWigUpFmFQo4NpeUs37KHpZv28K/CnWzfXwnAGT0yue3SgVxxZk9O75Hpc6UiEo+noWBmE4D/AYLAk865mfXmpwK/B0YCu4DrnXObvKxJmodzjrLyKj4vO8hnZQf5ZPt+1pXsZ13JAcqragHo0iGZC07L4cIBOXx5YI66h0QSgGehYGZB4FHgMqAIWGJmc5xza2OafRvY45wbYGZTgAeB672qSRqnLuTYU1HN7oPV7CyvonR/FTv2V1Kyr5Jtew9RtOcQRbsrOBD58AfISE3ijB6ZXH1Ob4bndWbEqV3om91Bh5OKJBgvtxTOAwqdcxsBzGw2cBUQGwpXAT+LDL8G/MbMzDnnPKwroTjnqAs56g7/jvzUhhy1dY6aulBkOERVbYiauhDVtSGqI7+rakNU1tRRWRPiUE0dh6prqaiuo6K6jvKqWsoraymvqmV/ZQ17K2rYd6iG/ZU1xHsHOqYE6dOlA727pHNu3y70y+lI/9wM+ud0pE+XdAWASBvgZSj0BrbGjBcBoxpq45yrNbN9QDaws7mLeWXJVma9uzE63lDuuAZGDg8652KG4fCYcxzxQRqvXSjaJjwccg5X73fIOUKh8HBdZHpzSwqXWtYHAAAITklEQVQY6SlBMlOTyEhLIiM1ia4dU+iX05FO6cl07pBCdscUunZMITsjhe5ZaXTPStNF6ETaAS//y+N9baz/EdeYNpjZdGA6wCmnnHJSxXTpmMLp3evt3Gzgi23s5NhvvxadFjtsX7Q3ODx2uM3hhxtGIBAZMgiaRdsEAkYg8jzBgGFmBCw8HDAjGIj5MSMpaCQFjGAgQFLQSA4aSYEAKUkBUoIBkoMBUpMDpCaFp6UnB0lLDpKWFCQ9JUhKkm6jISLxeRkKRUBezHgfoLiBNkVmlgR0AnbXfyLn3CxgFkB+fv5JfXe+bEh3LhvS/WQeKiLSbnj5lXEJMNDM+plZCjAFmFOvzRzghsjwtcDb2p8gIuIfz7YUIvsIbgXeJHxI6tPOuQIzuw9Y6pybAzwF/MHMCglvIUzxqh4RETk+T/ccOufmAfPqTbs3ZrgS+JqXNYiISONpj6OIiEQpFEREJEqhICIiUQoFERGJUiiIiEiUJdppAWZWBmw+yYfn4MElNJqB6joxquvEtdbaVNeJaUpdpzrnco/XKOFCoSnMbKlzLt/vOupTXSdGdZ241lqb6joxLVGXuo9ERCRKoSAiIlHtLRRm+V1AA1TXiVFdJ6611qa6TozndbWrfQoiInJs7W1LQUREjqHNhYKZfc3MCswsZGb59ebdZWaFZrbezC5v4PH9zOwjM9tgZi9HLvvd3DW+bGYrIz+bzGxlA+02mdnqSLulzV1HnNf7mZlti6ltYgPtJkSWYaGZzWiBuh4ys0/M7GMze93MOjfQrkWW1/H+fjNLjbzHhZF1qa9XtcS8Zp6ZLTKzdZH1/7Y4bcaa2b6Y9/feeM/lQW3HfF8s7NeR5fWxmY1ogZpOj1kOK81sv5ndXq9Niy0vM3vazErNbE3MtK5mtjDyWbTQzLo08NgbIm02mNkN8dqcEOdcm/oBBgOnA+8A+THThwCrgFSgH/AZEIzz+FeAKZHhx4D/9LjeXwL3NjBvE5DTgsvuZ8B/H6dNMLLs+gMpkWU6xOO6xgNJkeEHgQf9Wl6N+fuB7wKPRYanAC+3wHvXExgRGc4EPo1T11hgbkutT419X4CJwN8I34hwNPBRC9cXBLYTPo7fl+UFXASMANbETPsFMCMyPCPeeg90BTZGfneJDHdpSi1tbkvBObfOObc+zqyrgNnOuSrn3OdAIXBebAML33vzEuC1yKTngMle1Rp5veuAl7x6DQ+cBxQ65zY656qB2YSXrWeccwucc7WR0Q8J38XPL435+68ivO5AeF261GLv6+oB51yJc255ZPgAsI7wPdATwVXA713Yh0BnM+vZgq9/KfCZc+5kT4ptMufcPzn6rpOx61FDn0WXAwudc7udc3uAhcCEptTS5kLhGHoDW2PGizj6nyYb2BvzARSvTXP6MrDDObehgfkOWGBmyyL3qW4Jt0Y24Z9uYHO1McvRS98i/K0ynpZYXo35+6NtIuvSPsLrVouIdFedA3wUZ/b5ZrbKzP5mZkNbqKTjvS9+r1NTaPiLmR/L67DuzrkSCIc+0C1Om2Zfdp7eZMcrZvYW0CPOrLudc39p6GFxptU/9KoxbRqlkTVO5dhbCRc654rNrBuw0Mw+iXyjOGnHqgv4HXA/4b/5fsJdW9+q/xRxHtvkQ9gas7zM7G6gFnihgadp9uUVr9Q40zxbj06UmWUAfwRud87trzd7OeEukvLI/qI/AwNboKzjvS9+Lq8UYBJwV5zZfi2vE9Hsyy4hQ8E5N+4kHlYE5MWM9wGK67XZSXjTNSnyDS9em2ap0cySgGuAkcd4juLI71Ize51w10WTPuQau+zM7AlgbpxZjVmOzV5XZAfalcClLtKZGuc5mn15xdGYv/9wm6LI+9yJo7sGmp2ZJRMOhBecc3+qPz82JJxz88zst2aW45zz9Bo/jXhfPFmnGukKYLlzbkf9GX4trxg7zKync64k0p1WGqdNEeF9H4f1Ibw/9aS1p+6jOcCUyJEh/Qgn/uLYBpEPm0XAtZFJNwANbXk01TjgE+dcUbyZZtbRzDIPDxPe2bomXtvmUq8f9+oGXm8JMNDCR2mlEN70nuNxXROAHwGTnHMVDbRpqeXVmL9/DuF1B8Lr0tsNBVlzieyzeApY55x7pIE2PQ7v2zCz8wj//+/yuK7GvC9zgG9GjkIaDew73G3SAhrcWvdjedUTux419Fn0JjDezLpEunvHR6advJbYs96SP4Q/zIqAKmAH8GbMvLsJHzmyHrgiZvo8oFdkuD/hsCgEXgVSParzWeCWetN6AfNi6lgV+Skg3I3i9bL7A7Aa+DiyQvasX1dkfCLho1s+a6G6Cgn3m66M/DxWv66WXF7x/n7gPsKhBZAWWXcKI+tS/xZYRl8i3G3wccxymgjccng9A26NLJtVhHfYX9ACdcV9X+rVZcCjkeW5mpijBj2urQPhD/lOMdN8WV6Eg6kEqIl8fn2b8H6ovwMbIr+7RtrmA0/GPPZbkXWtELixqbXojGYREYlqT91HIiJyHAoFERGJUiiIiEiUQkFERKIUCiIiEqVQEGkCC1+d9HMz6xoZ7xIZP9Xv2kROhkJBpAmcc1sJXx5kZmTSTGCW8/HiaiJNofMURJoocnmJZcDTwE3AOS58BVWRhJOQ1z4SaU2cczVmdicwHxivQJBEpu4jkeZxBeHLFJzpdyEiTaFQEGkiMxsOXEb4rmH/q4VvECPSrBQKIk0QuYrm7wjfv2AL8BDwsL9ViZw8hYJI09wEbHHOLYyM/xY4w8zG+FiTyEnT0UciIhKlLQUREYlSKIiISJRCQUREohQKIiISpVAQEZEohYKIiEQpFEREJEqhICIiUf8fnS4lzmbbqYcAAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"df=pd.DataFrame({'X':np.linspace(-10,10,100)})\n",
"df['Y']=sigmoid(df['X'])\n",
"sns.lineplot(x='X',y='Y',data=df,label='SIGMOID CURVE');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"###### Model Representation\n",
"\n",
"The hypothesis in Linear Regression (i.e. $h_\\theta(x)=\\theta^Tx$) cannot be used for the classification problems because the linear hypothesis output ranges from $-\\infty$ to $\\infty$, whereas $y$ (i.e. target variable) in classification problems are either 0 or 1. Hence the range of hypothesis output in Logistic Regression should be:\n",
"\n",
"$$ 0\\le h_\\theta(x) \\le 1 $$\n",
"\n",
"In order to keep $h_\\theta(x)$ in the range mentioned above, Sigmoid Function comes to rescue:\n",
"\n",
"$$ h_\\theta(x)=g(\\theta^Tx) $$\n",
"$$ g(z)=\\frac{1}{1+e^{-z}} $$\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"def predict(X,theta):\n",
" '''Predicts by applying logistic function on linear model'''\n",
" z=np.dot(X,theta.T)\n",
" h=sigmoid(z)\n",
" return h"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### 1.2.2 Cost function\n",
"We learnt about the cost function $J(\\theta)$ in **Linear Regression**, the cost function represents the optimization objective i.e. we create a cost function and minimize it to develope an accurate model with minimum error.\n",
"$$ J(\\theta)=\\frac{1}{2m}\\sum_{i=0}^m(h_\\theta(x^{(i)})-y^{(i)})^2 $$ \n",
"On replacing $h_\\theta(x)$ with $\\frac{1}{1+e^{(-\\theta^Tx)}}$ in the above equation,it becomes '**Non Convex**' which does not guarrante to converge and find the global minimum. So we need a new cost function for Logistic Regression. \n",
"\n",
"\n",
"\n",
"The figure in left is a '**Convex**' function and has only one minimum called global minimum and will always converage whereas the figure in right is a '**Non Convex**' function and has mutiple minimums also called local minimums. And there is no guarrante that the non convex function will always converge.\n",
"\n",
"For **Logistic Regression** cost function is defined as: \n",
"\n",
"$$\\begin{equation}\n",
" Cost(h_\\theta(x),y)=\\begin{cases}\n",
" -log(h_\\theta(x)) & \\text{if }y=1 \\\\\n",
" -log(1-h_\\theta(x)) & \\text{if }y=0\n",
" \\end{cases}\n",
"\\end{equation}$$\n",
"\n",
"Let's try to understand the cost function. \n",
"\n",
"For $y=1$\n",
"- $J(\\theta)\\approx\\infty$ if $h_\\theta(x)\\approx0$ (Penalty for wrong prediction)\n",
"- $J(\\theta)\\approx0$ if $h_\\theta(x)\\approx1$ \n",
"\n",
"For $y=0$\n",
"- $J(\\theta)\\approx\\infty$ if $h_\\theta(x)\\approx1$ (Penalty for wrong prediction)\n",
"- $J(\\theta)\\approx0$ if $h_\\theta(x)\\approx0$ "
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd4VNed//H3Ue+9o46EQPSOwdjGuPdu4hI7seNssm7xJvHPm03fbOIUb+xN4rjFnRiXuOFu44LBFNFBiKYCEqijjtrM+f1xRiDbgEagmTvl+3qeeUbSjLjfC/bnnjn3FKW1RgghhO8LsLoAIYQQ7iGBL4QQfkICXwgh/IQEvhBC+AkJfCGE8BMS+EII4Sck8IUQwk9I4AshhJ+QwBdCCD8RZHUBgyUlJenc3FyryxBCCK+ybt26Rq118lDv86jAz83NpaSkxOoyhBDCqyilqpx5n3TpCCGEn5DAF0IIPyGBL4QQfsKj+vCFGI6+vj6qq6vp7u62upQRFxYWRmZmJsHBwVaXInyIBL7wWtXV1URHR5Obm4tSyupyRozWmqamJqqrq8nLy7O6HOFDpEtHeK3u7m4SExN9KuwBlFIkJib65CcXYS0JfOHVfC3sB/jqeQlr+Ubg93TAJ/fD3tVWVyKEEMP2+sYanl3l1FD6k+IbgR8QCJ/8D1Qut7oSIYZUUVHB7NmzKSws5Nprr6W3t9fqkoTFXllfw8sl+1x+HN8I/OBwCIuD9gNWVyLEkO69915+8IMfsGvXLuLj43niiSesLklYrK61m5SYMJcfxzcCHyAmA9ok8IX7/PSnP+XBBx88/P1PfvITHnrooeP+jtaaZcuWcdVVVwFw00038dprr7m0TuH5atu6SXND4PvOsMzodGnh+7FfvrmN0v1tI/pnFmfE8POLxx/z9VtuuYUrrriCu+66C7vdzgsvvMCyZcuYMmXKUd+/ePFiUlJSiIuLIyjI/K+XmZlJTU3NiNYtvEt3n43WQ32kxUrgOy86HepLra5C+JHc3FwSExPZsGEDdXV1TJ06lZycHDZu3HjM32loaPjaz2REjn+razPDb1OiQ11+LN8J/Jh06KgDu83cxBV+5XgtcVe69dZbeeqpp6itreXb3/427e3tzJ8//6jvXbx4MePGjaOlpYX+/n6CgoKorq4mIyPDzVULT1LX1gMgLfxhiU4DbYeOehP+QrjB5Zdfzs9+9jP6+vpYvHgxgYGBx23hAyxYsICXX36ZRYsW8fTTT3PppZe6qVrhiWodLfxUuWk7DNGOVlL7fmvrEH4lJCSEBQsWcM011xAY6Nwny/vvv58HHniAgoICmpqauOWWW1xcpfBkda3uC3zfaeEPtOrba62tQ/gVu93OqlWreOmll5z+nfz8fNasWePCqoQ3qWvrJjw4kJgw18exD7XwHYHfJi184R6lpaUUFBSwcOFCCgsLrS5HeKnatm5SY0LdcvPed1r4kcmgAqWFL9ymuLiY8vJyq8sQXq6+rcct3TngSy38gECISpWx+EIIr2Ja+BL4wxcjk6+EEN5Da01dW7dbhmSCrwV+dLosryCE8Bqth/ro6bdLC/+ERKfLsEwhhNc4Mgbf9bNswdcCPyYduluht8vqSoQ4pr/85S8UFBSglKKxsdHqcoSFDs+ylRb+CRgYmin9+MKDzZs3jw8//JCcnByrSxEWc+ekK/DZwJehmcL1TmR5ZICpU6eSm5vrwsqEtzi8cJqbunR8Zxw+mDXxQVr4/uid/we1W0b2z0ybCOf/7pgvn8jyyMXFxSNbo/BqtW3dJESGEBrkngUffSvwo9PMs8y2FW5wIssjCzFYXVu3W5ZFHuBbgR8aA8GR0qXjj47TEnel4S6PLC18MVhdW4/bxuCDrwW+UqaVL0MzhZucyPLIQgyobeumOD3GbcfzrZu2YPrxpYUv3ORElkd+6KGHyMzMpLq6mkmTJnHrrbe6uErhifptdho7ekj1tRa+UioQKAFqtNYXufRg0emwb7VLDyHEgBNZHvnOO+/kzjvvdGFVwhs0dPSgtfvG4IP7Wvh3AdvdcqToNNPC19othxP+S5ZHFiejttW9s2zBDS18pVQmcCHwG+AeVx+PmAyw9cChgxCR4PLDCf8lyyOLkzEwy9Zdk67APS38PwM/BuxHe1EpdZtSqkQpVdLQ0HDyR5OhmX5F++gnOV89L3FEnRv3sh3g0sBXSl0E1Gut1x3rPVrrR7XWM7TWM5KTk0/+oIf3tpUbt74uLCyMpqYmnwtHrTVNTU2EhbkvCIT71bV1ExyoSIwMcdsxXd2lMw+4RCl1ARAGxCilntNa3+CyIx7e21Za+L5uYKTLiHwy9DBhYWFkZmZaXYZwodq2blKiwwgIcP3WhgNcGvha6/uA+wCUUmcAP3Rp2ANESZeOvwgODiYvL8/qMoQ4IXVt3W5bQ2eA743DDwqB2Gxo3GV1JUIIcUx76jvJS4x06zHdNtNWa/0J8IlbDpZaDHXb3HIoIYQYrpauXmrbuilKi3brcX2vhQ+QOh6adkF/r9WVCCHE15TVtgMw1o3LKoCvBn5KMdj7oXGn1ZUIIcTX7BgIfGnhj4DU8eZZunWEEB6orLaNuIhgty6NDL4a+IkFEBgC9RL4QgjPU1bbzti0aJRy35BM8NXADwyGpCKoK7W6EiGE+BK7XbOztp2xae7tvwdfDXyQkTpCCI9UffAQnb02t4/QAV8O/JRiM9v20EGrKxFCiMPKatsAJPBHVOoE8yzdOkIIDzIwQqcoVQJ/5KQ69g6Vbh0hhAcpq20nOyGCyFD37zDru4EfnQ5hcTJSRwjhUcpq2yzpzgFfDnylTLeOdOkIITxEd5+NisZOxkngu0BqMdSXgv2oe68IIYRb7a7vwK6hyIIhmeDrgZ9SDL0d0LrX6kqEEOLwGjrSpeMKh5dYkG4dIYT1dtS2ERIUQG5ihCXH9+3ATxlnnmWkjhDCA5TVtjMmNYqgQGui17cDPzQaEkZDzTG31BVCCLew2zVbalopdvOSyIP5duAD5MyFvSvBbrO6EiGEHyurbaelq4/ZeYmW1eD7gZ87H7pbpVtHCGGp1RVNAMzOT7CsBj8I/HnmufJza+sQQvi1VeVNZCWEkxlvzQ1b8IfAj82E+DwJfCGEZex2zeqKZuZY2J0D/hD4ALmnQtUKmYAlhLDEjjrTfz8nXwLf9XJPhe4WqNtqdSVCCD+0qtz6/nvwl8DPcfTjV62wtg4hhF9aXd5MZry1/ffgL4EflwXxudKPL4RwO9N/32R5dw74S+CD6dap/Fz68YUQbrWzvp2DHtB/D/4U+DmOfnxZH18I4Uar9jj67/Os7b8Hfwp8GY8vhLDAqvJmRsWFk5Vgbf89+FPgx2VDXA6Uf2J1JUIIP9Fvs7PKQ/rvwZ8CH6DofNjzMfR0WF2JEMIPrKlspqWrj7PGpVhdCuBvgT/uErD1wO4PrK5ECOEH3ttaS1hwAKcXJVtdCuBvgZ89ByKSoPQNqysRQvg4u13z7rZaTh+TTERIkNXlAP4W+AGBMPZC2PU+9HVbXY0QwodtrG6hrq2H8yakWV3KYf4V+GC6dXo75OatEMKl3ttaS1CA4syxqVaXcpj/BX7eaRAaA9vftLoSIYSP0tp058wtSCI2PNjqcg5zaeArpcKUUmuUUpuUUtuUUr905fGcEhQCY86DHW+Brd/qaoQQPqistp2qpi7OG+853Tng+hZ+D3Cm1noyMAU4Tyk1x8XHHFrxJXDooCymJoRwiXe31qIUnDPec7pzwMWBr42BQe/Bjod25TGdMnohBIXDdhmtI4QYee9urWVmbgJJUaFWl/IlLu/DV0oFKqU2AvXAB1rr1V95/TalVIlSqqShocHV5RghEVB0Hmz9F/T3uOeYQgi/sLWmlR117Vw4Md3qUr7G5YGvtbZpracAmcAspdSEr7z+qNZ6htZ6RnKyGycnTL0RDjVD2VL3HVMI4fOWrN1HSFAAl00ZZXUpX+O2UTpa6xbgE+A8dx3zuPIXQGw2rH/G6kqEED7iUK+N1zbWcMGENGIjPGd0zgBXj9JJVkrFOb4OB84Cylx5TKcFBMC0G814/OYKq6sRQviAt7ccoL27n0Wzsq0u5ahc3cJPBz5WSm0G1mL68D2nD2XK9aACYMOzVlcihPABL6zdS15SpEesfX80rh6ls1lrPVVrPUlrPUFr/StXHm/YYkdBwdmw4XkZky+EOCm76ztYW3mQa2dmoZSyupyj8r+Ztl81/SboqDXr6wghxAlasnYvQQGKK6Z53s3aARL4hedAVCqse8rqSoQQXqqn38Yr62tYOC6FlOgwq8s5Jgn8wGCYfrNp4TfstLoaIYQXenV9Dc2dvdw4J9fqUo5LAh9g1m0QFAYrH7S6EiGEl7HZNY9+Vs74jBjmFXjGVobHIoEPEJkEU2+ATUugbb/V1QghvMgHpbWUN3byb6eP9tibtQMk8AfMvR20DVb9zepKhBBeQmvNw5+Wk50QwfketNHJsUjgD4jPhfGXQ8lTcKjF6mqEEF5gVXkzm/a18J3T8gkK9Pw49fwK3WneXdDbDiVPWF2JEMIL/P3TPSRFhXD19EyrS3GKBP5g6ZNh9Jnwxd+gp2Po9wsh/NaW6lY+3dnAzXNzCQsOtLocp0jgf9UZ/wldjdKXL4Q4rvvfLSM+Ipib5uZaXYrTTirwlVLfGqlCPEbWTBh7Eax4EDobra5GCOGBlu9q4PPdjdx+ZiHRYZ63KuaxnGwL3/o9al1h4c+grws++6PVlQghPIzdrrn/3TJGxYVzwxzPXBXzWIKGeoNjpcujvgR41oaNIyW5yIzLL3kC5nwP4nOsrkgI4SHe2nKArTVt/OnqyYQGeUff/YAhAx8T6ucCB7/ycwWsHPGKPMUZ98HmF+Hj38AVj1pdjRDCA/TZ7Pzp/R0UpUZz2VTPXSTtWJzp0lkKRGmtq77yqMTsYOWbYjJg9r/B5iWwb63V1QghPMCTKyqobOri3vOLCAzw7Fm1RzNk4Gutb9Faf36M164b+FopFT+ShXmE034I0Rnw1j2yXr4Qfu5A6yH+/OEuFo5NYUFRitXlnJCRHJb50Qj+WZ4hNBrO+x+o3SyTsYTwc79eWorNrvnFJeM9fs2cYxnJwPfOv4GhFF9mJmMt+29or7O6GiGEBT7d2cDbW2q5fUEBWQkRVpdzwkYy8PUI/lmeQym44I/Q3w3v/5fV1Qgh3Ky7z8bPX99KXlIkt52eb3U5J0Vm2jojcTTMuxu2vAg7ZStEIfzJgx/torKpi19dOt7rhmF+lXTpOOu0H0JKMbxxB3Q1W12NEMIN1u89yCOf7uHaGVnML0y2upyT5nTgK6WeHeJnC0ekIk8VFAqXPWzW2XnnXqurEUK42KFeGz98cRPpseH810XjrC5nRAynhT9+8DdKqUBg+sD3Wmvfb/ZmTIHTfmy6dkpft7oaIYQL/f69MsobO/nDVZO8ar2c4xky8JVS9yml2oFJSqk2x6MdqAf8L/Xm3wPpU2DpD6C91upqhBAusGJ3I0+uqOSmU3KYW5BkdTkjxpmJV7/VWkcDf9Baxzge0VrrRK31fW6o0bMEBsPlj0BvF7xyK9htVlckhBhB9e3d3PXCRkYnR3Lv+WOtLmdEDadLZ6lSKhJAKXWDUuoBpZR/riqWMhYu/BNULodP77e6GiHECLHZNXe/sJGOnj7+dv10IkKcWW7Mewwn8B8GupRSk4EfA1XAMy6pyhtMvR6mXA+f/h72fGx1NUKIEfB/y3axck8Tv7p0AkVp0VaXM+KGE/j9WmsNXAo8qLV+EPC9v5HhuOAPZinlf30H2vZbXY0Q4iR8vquRBz/axRXTRnnNHrXDNZzAb1dK3QfcCLzlGKXjG7euT1RIJFz9NPQdgheuM89CCK9T2djJvy9eT2FKFL++dILXrpUzlOEE/rVAD/BtrXUtMAr4g0uq8iYpY+GKx2D/RjMpS/vmChNC+Kr27j5ufaaEAAWPf3MmkaG+1W8/mNOB7wj554FYpdRFQLfW2n/78AcbewEs/ClseQk+/1+rqxFCOMlm19z1wkYqGzv52/XTyU703oXRnDGcmbbXAGuAq4FrgNVKqatcVZjXOfUemHAVfPQrKH3D6mqEEE747dvbWVZWz88vGc8poxOtLsflhvPZ5SfATK11PYBSKhn4EHjZFYV5HaXg0r9Ay14zPj/yNciZa3VVQohjeHx5OY9/XsHNc3O5cY5/jDAfTh9+wEDYOzQN8/d9X3A4XLcE4rLhn4ugfrvVFQkhjuL1jTX891vbuWBiGj+9qNjqctxmOIH9rlLqPaXUzUqpm4G3gLeP9wtKqSyl1MdKqe1KqW1KqbtOplivEJEAN7wCQWHw3JXQWm11RUKIQVbsbuSHL21iVl4CD1wzxSv3pj1RzqylU6CUmqe1/hHwCDAJmAx8ATw6xK/3A/+htR4HzAH+XSnl+5fT+By4/mXoaYdnLpWdsoTwECWVzdz6dAn5SVE8duMMwoK9e3374XKmhf9noB1Aa/0vrfU9WusfYFr3fz7eL2qtD2it1zu+bge2Y4Zz+r70SSb02w6Y0O9ssroiIfzapn0t3PzkWtJjw3ju1tnERvjfNCJnAj9Xa735qz/UWpcAuc4eSCmVC0wFVn/l57cppUqUUiUNDQ3O/nHeIXu26dM/WAHPXgqHDlpdkRB+qXR/G9/8xxriI4N5/juzSY4OtbokSzgT+GHHeS3cmYMopaKAV4C7tdZtg1/TWj+qtZ6htZ6RnOz9O8p8Td58WPQ8NOyApy+GzkarKxLCr2yubuG6x1cRGRLI4lvnkB7rVGz5JGcCf61S6jtf/aFS6hZg3VC/rJQKxoT981rrfw2/RB9QcBZ845/QuAueulDW0RfCTdZVNXP9Y6uJCg1iyXdPISvBtydWDUXpIZYCUEqlAq8CvRwJ+BlACHC5YwbusX5XAU8DzVrru4cqZsaMGbqkpMTJ0r1QxXJYfC1Ep8FNb0Csby7QJIQn+GJPE7c8vZbUmDAWf2e2T7fslVLrtNYzhnqfMxug1Gmt5wK/BCodj19qrU85Xtg7zMMstnamUmqj43HBkNX7qrz5cOOr0NkAT5wj4/SFcJF3thzgpifXMCounCW3+Xc3zmBDtvDdyedb+ANqt5gx+v095qZu9hyrKxLCZzy7qoqfvb6VKVlx/OOmmcRHhlhdksuNWAtfuEDaRLjlfYhINEM2ty+1uiIhvJ7Wmj+9v4OfvraVM4tSWHzrHL8I++GQwLdKfK4J/dTxsOQGWPGQLK0sxAnq7rNxxz838H/LdnPNjEweuXE64SH+NanKGRL4VopMgpuWQvEl8MFP4c07wdZndVVCeJWG9h6+8dgqlm4+wI/PK+L+KycRFCjRdjS+u9K/twiJgKuego9/A8v/CM0VcPVT5mIghDiurTWtfPfZdTR19vDw9dM4f2K61SV5NLkMeoKAALOByuWPQvVaeOR02L/B6qqE8GivbajhyodXYteaF797ioS9EyTwPcnka+Hb75m19f9xHmxcbHVFQnicPpudXy8t5e4lG5mSFcebd5zKpMw4q8vyChL4niZjCtz2CWTOhNe+B6/fLpujC+Gwv+UQ1z7yBU84Ni557tbZJEX557o4J0L68D1RZBLc+Bp88lvTr1+zHq55GpIKra5MCMt8vKOee5ZspM+m+ct1U7loUobVJXkdaeF7qsAg069//SvQfgAeOQ3WPytDN4Xf6em38eulpXzrybWkxYbz5h2nStifIAl8T1d4FnxvBYyaDm/cDi/dLMssC7+xu76dy/66kic+r+CmU3J49ftzyUuKtLosryVdOt4gJgO++TqsfAiW/bcZyXPpX2D0mVZXJoRL2O2aZ76o5HfvlhEREsQTN81g4bhUq8vyetLC9xYBgXDqD+CWDyAkEp69HN76D+jttLoyIUZU9cEurn98Nb94s5RT8hN59675EvYjRFr43mbUNPjuZ/DRr2HV32D3R3DJ/5mVOIXwYna75p9r9/Lbt8vQWvO7KyZy7cwszCrrYiRIC98bBYfDef8DNzsWXXv6InjzbuhutbYuIU5QZWMn1z2+ip+8upVJmbG8e/dpLJqVLWE/wqSF781yT4XvrTTLMqz6G+x8D86/H8ZdbCZvCeHh+mx2HltezkMf7SI4IEBa9S4mge/tQiLg3N/A+CvM4msv3ghjzoML/gBx2VZXJ8QxlVQ285+vbmFnXQfnjk/ll5dMIC32eFtoi5Mlge8rMqebGbqrHjYTtv46G+b/B8y9A4JkJqLwHI0dPdz/ThkvratmVFw4j39zBmcVy01Zd5Adr3xRy1549z4oWwoJ+XD+76HwbKurEn6u32bnuVVV/OmDnXT32fj2qXnceWYhkaHS7jxZzu54JX/TviguGxY9D7s/hHfuheevgsJz4JzfQPIYq6sTfmj5rgZ+vbSUnXUdzC9M4ucXj6cgJcrqsvyOBL4vKzjL3NRd/Xf47I/w8Ckw8ztw+o8hIsHq6oQf2F3fwW/f3s5HZfVkJ0Tw9xumce74NLkpaxHp0vEXHQ1mNM/6pyE02vTvz/ouBMtNMjHy6tu7efDDXbywdh/hwYHccWYBN8/LJTRIth10BWe7dCTw/U3dNvjwF7DrfYjNggX/CZOuNTN5hThJ7d19PL68gseWl9Pbb+f62dncsbBQljB2MQl8cXzln8IHP4MDGyF5LJz5XzD2Ihm/L05Id5+N51ZV8dePd3Owq4/zJ6Txo3OLyE+Wfnp3kJu24vjyTzfDOLe/YRZkW3IDpE+BBT8xI3ok+IUTevptvLh2H3/9eA+1bd3ML0ziR+cWyQ5UHkoC358pBcWXQtGFsPkF+PT3sPhqGDUDzrgPChZK8Iuj6um38cq6Gv768W5qWg4xIyeeB66dzNzRSVaXJo5DunTEEbY+2Pg8fPoHaKuGjKlw+r1m5q4Ev8B03bxYso+HP9nDgdZupmTFcc/ZY5hfmCQjbywkffjixPX3wqbFsPwBaKmC1Ilw6t0w/nK5ueunOnr6eX5VFY9/XkFDew8zcuK5c2GhBL2HkMAXJ8/WB1tegs//Fxp3mlm7c++Eyd+Q4Zx+orGjh2dWVvLUykrauvs5tSCJ7y8YzSn5iRL0HkQCX4wcu90s0/D5A7B/A0Qmw+zvwoxbZAKXj6po7OSx5eW8sq6aXpudc4pT+f4ZBUzOkpuxnkgCX4w8raFyOax4CHZ/AMERMOU6mPN9SBxtdXXiJGmtWVPRzOOfV/Dh9jqCAwO4ctoobp2fz2gZXunRZFimGHlKQd5p5lG3Db74G6x/BtY+YW7szv4u5J8hN3i9TE+/jaWbDvDkygq21rQRHxHMHQsKuOGUHFKipevOl0gLX5yc9jpY+ziU/AO6Gs0krlm3mdm7odIq9GS1rd0sXl3F4jV7aezopSAlim/Ny+XKaZmEBcvNeW8iXTrCvfq6YdursPphOLAJQmPMzd2Zt8oKnR5Ea80X5U08+0UV75fWYdeahWNTuHluHvMK5East5LAF9bQGqrXwprHoPQ1sPVC7nyYfrPZelE2Y7FES1cvL6+rZvGavZQ3dBIXEcy1M7K4fnYO2YkRVpcnTpJHBL5S6h/ARUC91nrCUO+XwPcxHQ2w8TkoedKM549INK3+6TdDUqHV1fk8rTWrK5p5Yc1e3t5aS2+/nWnZcVw3O4eLJqVLt40P8ZTAPw3oAJ6RwPdjdjvsWQbrnoSd74K9H3LmwdQbzNIOIZFWV+hT6tu6+deGGl5cu4/yxk6iQ4O4dGoG183KoTgjxuryhAt4ROA7CskFlkrgC8Dc5N202IzuaS6HkGiYcDlMuR6yZssInxPU229nWVkdL6+r5uMdDdjsmhk58Syalc2FE9MJD5HWvC/zmsBXSt0G3AaQnZ09vaqqyqX1CA+hNez9AjY8B9teg75OSBhtunwmXQPxOVZX6PG01myubuVf66t5Y9N+Dnb1kRIdypXTM7l6eqYsTexHvCbwB5MWvp/q6TDLNG9cbCZ2genymXSN6fIJj7e2Pg+zr7mL1zfW8K8NNZQ3dBISFMA5xalcNT2TUwuSCAoMsLpE4WYS+MI7HayCLS/CpiXQtAsCQ8wG7BOvhjHnQnC41RVaoqmjh7e3HOC1jftZV3UQgFm5CVw+bRQXTEwnNjzY4gqFlWSmrfBO8Tlw2o9g/g9h/3rY8jJsfcWs5RMSDWMvhAlXwugFEOjbIdfW3cf72+p4Y9N+VuxuxGbXjEmN4kfnFnHJ5AyyEmQ4pRgeV4/S+SdwBpAE1AE/11o/caz3SwtfHJXdBhWfmeDf/gZ0t5punrEXwYQrIPc0CPSNtktHTz8fba/jzU0H+GxnA702O5nx4Vw8OYOLJ2UwLj1aJkeJr/GYLp3hkMAXQ+rvhT0fmVm9ZW9DbzuEJ5iWf/FlZp2foBCrqxyWtu4+Ptpex9tbavl0ZwO9/XbSYsK4cFI6F05KZ2pWnIS8OC7p0hG+KSgEis43j75u2P0hlL5uRvpseBbCYmHM+WZW7+gzIcQzuz2aOnr4oLSOd7fVsmJ3I302TXpsGNfPzuaCielMz44nIEBCXowsCXzhvYLDYNxF5tHfYyZ3bX8Tyt4ye/QGR5jQH3uRueFr8dr9+5q7eG9bLe9vq6Okqhm7hqyEcL41L49zx6cxNStOQl64lAS+8A1BoUda/rY+qPzc3Ogte8s8q0DImQtFF5j3JOS5vCS7XbN1fysflNbxQWkdZbXtAIxNi+b2Mws5pziV8Rkx0l0j3Eb68IVvs9vNLl073jJ9/g3bzc+Tx5o1/IvOh8yZI7ZXb1dvPyt2N/HR9jqWldVT395DgIIZuQmcU5zK2cWp5CTKUhJiZMlNWyGOprkcdrwLO9+BqpVmXZ/weCg423T7jD5z2F0/+5q7+HhHPcvK6lm5p4nefjtRoUGcPiaZheNSWFCUQnykd91IFt5FAl+IoRxqMf3+u943j64mUAEwaoaZ7FWwENKnQMCXZ65299lYW9nMJzsa+GRHPXsaOgHIS4pkQVEKZ45NYVZeAiFBMuNVuIcEvhDDYbeZrp+B8N+/wfw8IgldcCYNKaeyrG8C71fZ+WJPE4f6bIQEBjA7P4EFRSmcUZQsa9cIy0jgC3HH+MW1AAAPqElEQVQS2hr2U7HmTey7PiS3ZRXxtAGwMyCfuqS5RI0/m6KZZxMRIf3xwnoS+EIMQ0+/jfVVLazY3cjnuxvZXN2CXUNUaBCn5MVzWVoDc9lE/P7lUL3G9P0HhZuRP/lnmEfqhK91/wjhDjLxSojjsNk12/a3smJ3Eyv3NLK2spnuPjuBAYopWXHcvqCA+WOSmZIVR/Dh1ScvNE897WbY556PofwT+OCn5ucRiWY7x/zTIe90SMiX9f2FR5HAF37BbteU1bazqryJlXuaWF3RRHt3PwBjUqNYNDObeQVJzM5PICZsiEXZQqOPjPkHaNsP5Z9CxafmufQ18/OYUWaph9z5kHuqrPEvLCddOsInDQ741RVNrK5opqWrD4CcxAjmjk5kTn4ip4xOJCU6bOQOrDU07TaLvVV8Ztb372oyr8Vlm/DPmScXADGipA9f+JU+m51t+9tYU9HEmopm1lYepPWQCfjshAhm5yVwiiPkM+LcuKa+3Q4NZaYLqPIzqFwBh5rNa7FZ5h5AzlxzEUgskC4gcUIk8IVP6+rtZ+PeFtZWHmRNZRMb9rbQ1WsDID8pkll5CczKS2B2fiKj3BnwQxl8AahaYSZ/ddab1yKTIXsOZM81z2mTfGbZZ+FactNW+JT69m7WVR6kpOogJZXNbNvfRr9doxSMTYvh6umZzMxLYFZuAikxI9hFM9ICAiC12Dxm33akC6hqpXnsXWkWgAMIjoTMGZB9CmTPNktAhEZbW7/watLCFx6n32ZnR1076/e2sL7qIOuqDrK3uQuA0KAAJmfFMTM3nhm5CUzLjve97f1aa2DfKqj6wjzXbQNtN7OAU8dD1mzzyJwJ8bnSDSSkS0d4j6aOHjbua2HD3hbW7z3Ipn0tdDq6Z5KjQ5mRE8+07Him58YzISPW/5Ys6G6D6rWwbw3sWw3VJWbjF4DIFMiaZT4JZM6EjKkQIpPB/I106QiP1NNvo3R/Gxv3tRx+VDWZ1ntggKI4PYarpmcyzRHymfHhsnxwWIxZ16dgofneboP6UnMBGLgQlC01r6lA0100ynEBGDUdksbIhDABSAtfuJDdrqls6mRTdQub9rWyYV8L2/e30WuzA5AaE8rUrHimZMcxLTueiaNiCQ8ZmWWK/U5nE9SUmPCvKYGa9dBjloMgNMa0/EdNP/KISbe2XjGipEtHuJXWmv2t3WypbmFTdSubq1vYXN16eHJTREggE0bFMjUrjilZcUzOinPv8Eh/Y7dD406oWWcuANUl5lOB3fx7EJ0OGdNg1FTznDHV8h3BxImTLh3hMlpr6tp62FzdwtaaVrY4Ho0dvQAEBSjGpkdz8eQMJmfGMiUrnoKUKAJl+z73CQiAlLHmMfV687O+Q1C7xXERWA/715uNYQbE55rgT58CGVMgfbLZK0D4DAl8cVxaa2paDrG1po1t+02wbx0U7gEKxqRGc0ZRCpMzY5mYGcfYtGjCgqVrxuMEh5sbvFmzjvzsUAsc2GTCv2a9uRhse/XI6/G5X74ApE+RTwJeTAJfHGazayoaO9m2v5XS/W1s29/G1v2th5ckCAxQFKZEcUZRChNHxTJhVCzF6THS7+7NwuPMYm/5px/5WVez2Q/gwEbYv9FcDAbWBwIzQzhtkuMCMMl8HZMhw0O9gAS+nzrUa2NHXfvhcC890EbZgXYO9ZnhkCGBARSlRXPe+DTGj4plQkYM49JjpOXuDyISvjwqCMxFoHazuQDUboYDm2HH24DjHmB4AqRNPHIBSJsIiYUyU9jDyL+Gjxvob99+wIT6wHNlYyd2x/+r0aFBjMuIYdGsLIrTYxifEUthatSgZYGF34tIOLLu/4CeDjMprHaz6Raq3QKrHwGb6e4jMBRSxkHaBEid6HgeL/cFLCSB70MO9drYWddOWW0bZbXtbD9gnge6ZABGxYVTnBHDRZMyKE6PZnxGrIx1FycmNMos+ZA9+8jPbH3QuMtcBGq3QN1W2PEObHjuyHtiMk3wp02AlGKzcUxigXwacAP5G/ZC/TY7Vc1d7Khtp6y2nR21beyobaequYuBUbbhwYEUpUVz/oQ0xqaZ7pix6dFDr/UuxMkIDD6yVtDkReZnWkN7rfk0ULfF8bwN9nx0ZJhoYAgkFZnfSyk2F4SUcWZPAWmMjBgJfA82MEJmZ107O+s62Fnbzo66dnbVd9DbbyYvBSjITYxkXHoMl00dxdi0GMamRZOdEEGADIMUnkApM9ErJh0Kzzry8/4eM1egrtR8EqgvhYrlsHnJkfeExprgTxlnLgQDX0cmuf88fIBMvPIAA5OWdtW1s6uug52OUN9d30FHT//h96XHhjEmNZoxqVEUpcVQlBpNQUqUjJIRvqWr2SwhXV9qLgb126F+G3S3HnlPZDIkjzWPlLGQPM58HZloXd0WkolXHshu11QfPMSu+nZ213ewy/HYXdd+eLEwgKSoUMakRnHV9EwKU6NMyKdEExsh3THCD0QkHNkYZoDW0H7AhH9D2ZHnTS8cWUgOBl0Iisxz0hjzHJUiXUNI4LtET7+NysYudjta6XsazHN5YwfdffbD70uJDqUgJYqrZ2RRkGKCvSAlioTIEAurF8IDKWXG+sdkfHm4qNbQVuO4CJRBw3Zo2AmbX4KeQZ8IwuLMRSCp0NwrGPg6LgcC/OcTsgT+SWju7KW8wQR6eUPn4WDf29x1eMgjmJExBSlRzB2dSGFqFAUpURQkS4tdiJOmFMRmmkfBoPsDA58IGnaY+wQNZeZCsPO9L48YCgozI4SSCs2ngaQx5vvEAjMKycdI4A+ht9/O3uZOyhs6KW/sZE99B+WNnZQ3dHBw0HDHkKAA8pMiKc6I4ZLJGYxOiWJ0chT5yZFEhMhfsxBuNfgTwegFX36tq9lcBAYeDTvNPILS181GMwNiRjkuBmPMBWHgwhCT6bXLTUsSYfrWa9u6qWg0oV7R0ElFYwcVjZ3sO3gI26DmenJ0KPlJkZw3IZ3RyZGHQz0zPkIWBxPCG0QkOPYOnvPln/f3QNMeaNrluBjsMo/NS44sNQ3mU0HCaEgcfeTTwMAjIsGj7xW4PPCVUucBDwKBwONa69+5+phHo7WmoaOHysYuKhs7qWgywV7ZZB6D+9bDgwPJS4pk/KhYLp6cQX5yJPlJUeQlR8o4diF8VVDokTkEg2kNHfVm7+Emx0WgaY+5cbzj7SNzCQDCYk3wD1wQDl8YRpvXLObSwFdKBQJ/Bc4GqoG1Sqk3tNalrjie1pqG9h4qm7qobOqkqqmTysYuKhrN14NHwgQHKrISIshLjOTUgiRykyIPB3tqTKjMPBVCGEpBdKp55M778mu2fmipOvLJoGmPuTDs/QK2vMThtYYAIhIHXQjyv/wIj3PLqbi6hT8L2K21LgdQSr0AXAqMaOAf7OzlG4+tYm9zF12DQj0oQJEZH05uUiSz8hLIS4okJzGC/KQoMuLCCJK1YoQQJyMw6EgLnnO+/FrfIThYaS4CzXscz+VQ8Rls+ueX3xvuGIq66HmXluvqwB8F7Bv0fTUwe/AblFK3AbcBZGdnn9BBYsODyUqIYO7oJHKTIshOiCAvKZJRceES6kIIawSHH5kZ/FV9h6C5wlwAmsvhYIVbNp93deAfrV/kS1N7tdaPAo+CmWl7IgcJCFA89s0hJ5kJIYRnCA4/+v0CF3N187cayBr0fSaw38XHFEIIcRSuDvy1QKFSKk8pFQIsAt5w8TGFEEIchUu7dLTW/Uqp24H3MMMy/6G13ubKYwohhDg6l4/D11q/Dbzt6uMIIYQ4PhnCIoQQfkICXwgh/IQEvhBC+AkJfCGE8BMetcWhUqoBqBrmryUBjS4ox9PJefsXOW//MtzzztFaJw/1Jo8K/BOhlCpxZi9HXyPn7V/kvP2Lq85bunSEEMJPSOALIYSf8IXAf9TqAiwi5+1f5Lz9i0vO2+v78IUQQjjHF1r4QgghnOAVga+UOk8ptUMptVsp9f+O8nqoUmqJ4/XVSqlc91c58pw473uUUqVKqc1KqY+UUjlW1OkKQ537oPddpZTSSimfGMnhzHkrpa5x/LtvU0otdneNruDEf+vZSqmPlVIbHP+9X2BFnSNJKfUPpVS9UmrrMV5XSqmHHH8nm5VS0076oFprj35gVtncA+QDIcAmoPgr7/k+8HfH14uAJVbX7abzXgBEOL7+ni+ct7Pn7nhfNPAZsAqYYXXdbvo3LwQ2APGO71OsrttN5/0o8D3H18VApdV1j8B5nwZMA7Ye4/ULgHcwG0nNAVaf7DG9oYV/eF9crXUvMLAv7mCXAk87vn4ZWKi8fxfyIc9ba/2x1rrL8e0qzAYzvsCZf3OAXwO/B7rdWZwLOXPe3wH+qrU+CKC1rndzja7gzHlrIMbxdSw+sJGS1vozoPk4b7kUeEYbq4A4pVT6yRzTGwL/aPvijjrWe7TW/UArkOiW6lzHmfMe7BZMa8AXDHnuSqmpQJbWeqk7C3MxZ/7NxwBjlFIrlFKrlFLnua0613HmvH8B3KCUqsYst36He0qz1HAzYEguXw9/BAy5L66T7/E2Tp+TUuoGYAZwuksrcp/jnrtSKgD4X+BmdxXkJs78mwdhunXOwHyiW66UmqC1bnFxba7kzHl/A3hKa/0npdQpwLOO87a7vjzLjHiueUML35l9cQ+/RykVhPnId7yPSt7Aqf2AlVJnAT8BLtFa97ipNlcb6tyjgQnAJ0qpSkz/5hs+cOPW2f/WX9da92mtK4AdmAuAN3PmvG8BXgTQWn8BhGHWm/FlI74nuDcEvjP74r4B3OT4+ipgmXbc9fBiQ563o1vjEUzY+0Jf7oDjnrvWulVrnaS1ztVa52LuX1yitS6xptwR48x/669hbtajlErCdPGUu7XKkefMee8FFgIopcZhAr/BrVW63xvANx2jdeYArVrrAyfzB3p8l44+xr64SqlfASVa6zeAJzAf8XZjWvaLrKt4ZDh53n8AooCXHPeo92qtL7Gs6BHi5Ln7HCfP+z3gHKVUKWADfqS1brKu6pPn5Hn/B/CYUuoHmG6Nm729UaeU+iemay7JcW/i50AwgNb675h7FRcAu4Eu4FsnfUwv/zsTQgjhJG/o0hFCCDECJPCFEMJPSOALIYSfkMAXQgg/IYEvhBB+QgJfiCEopXKPtaKhEN5EAl8IIfyEBL4QzglUSj3mWIP+faVUuNUFCTFcEvhCOKcQsyzxeKAFuNLieoQYNgl8IZxTobXe6Ph6HZBrYS1CnBAJfCGcM3glUhtesA6VEF8lgS+EEH5CAl8IIfyErJYphBB+Qlr4QgjhJyTwhRDCT0jgCyGEn5DAF0IIPyGBL4QQfkICXwgh/IQEvhBC+AkJfCGE8BP/H/DudeeNp6VjAAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"sns.scatterplot(x='exam1',y='exam2',data=data[data['decision']==0],label='Not Admitted');\n",
"sns.scatterplot(x='exam1',y='exam2',data=data[data['decision']==1],label='Admitted');\n",
"# Only need two points to define a line, so choose two endpoints\n",
"x1=np.array([min(X[:,1]),max(X[:,1])])\n",
"x2=np.multiply((-1/theta[0][2]),(theta[0][0]+np.multiply(theta[0][1],x1)))\n",
"# De-normalizing the data\n",
"x1=x1*data['exam1'].std()+data['exam1'].mean()\n",
"x2=x2*data['exam2'].std()+data['exam2'].mean()\n",
"sns.lineplot(x=x1,y=x2,label='Decision Boundary',color='green');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### 1.2.4 Evaluating Logistic Regression"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
actual
\n",
"
prediction
\n",
"
\n",
" \n",
" \n",
"
\n",
"
64
\n",
"
0
\n",
"
0.075530
\n",
"
\n",
"
\n",
"
73
\n",
"
1
\n",
"
0.878757
\n",
"
\n",
"
\n",
"
63
\n",
"
0
\n",
"
0.000157
\n",
"
\n",
"
\n",
"
10
\n",
"
0
\n",
"
0.905205
\n",
"
\n",
"
\n",
"
49
\n",
"
1
\n",
"
0.999988
\n",
"
\n",
"
\n",
"
21
\n",
"
1
\n",
"
0.998391
\n",
"
\n",
"
\n",
"
40
\n",
"
1
\n",
"
0.967775
\n",
"
\n",
"
\n",
"
38
\n",
"
0
\n",
"
0.209379
\n",
"
\n",
"
\n",
"
99
\n",
"
1
\n",
"
0.999695
\n",
"
\n",
"
\n",
"
6
\n",
"
1
\n",
"
0.998787
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" actual prediction\n",
"64 0 0.075530\n",
"73 1 0.878757\n",
"63 0 0.000157\n",
"10 0 0.905205\n",
"49 1 0.999988\n",
"21 1 0.998391\n",
"40 1 0.967775\n",
"38 0 0.209379\n",
"99 1 0.999695\n",
"6 1 0.998787"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"h=predict(X,theta).reshape(m)\n",
"result=pd.DataFrame({'actual':data['decision'],'prediction':h})\n",
"result.sample(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Calculating the admission probability of student with Exam 1 score of 45 and Exam 2 score of 85."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ADMISSION PROBABILITY : 0.770\n"
]
}
],
"source": [
"x=[1,45,85]\n",
"# Normalizing the data\n",
"x[1]=(x[1]-data['exam1'].mean())/data['exam1'].std()\n",
"x[2]=(x[2]-data['exam2'].mean())/data['exam2'].std()\n",
"print('ADMISSION PROBABILITY : %.3f'%predict(x,theta)[0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Calculating the training accuracy."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TRAINING ACCURACY : 89.0%\n"
]
}
],
"source": [
"p=np.round(h).reshape(m,1)\n",
"accuracy=np.mean(y==p)*100\n",
"print(f'TRAINING ACCURACY : {accuracy}%')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2 Regularized Logistic Regression\n",
"\n",
"##### Problem Statement\n",
"In this part of the exercise, we will implement regularized logistic regression to predict whether microchips from a fabrication plant passes quality assurance (QA). During QA,e ach microchip goes through various tests to ensure it is functioning correctly.\n",
"\n",
"Suppose we are the product manager of the factory and we have the rest results for some microchips on two different tests. From these two tests, we would like to determine whether the microchips should be accepted or rejected. To help us make a decision, we have a dataset of test results on past microchips, from which we can build a logistic regression model."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TRAINING DATASET SHAPE : 118 X 3\n"
]
},
{
"data": {
"text/html": [
"