{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Logistic Regression/Classification\n", "> In this post, it will cover the basic concept of Logistic Regression, which is widely used in classification tasks. And it will explain what the hypothesis and cost function, and how to solve it with gradient descent as we saw previously.\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Chanseok Kang\n", "- categories: [Python, Tensorflow, Machine_Learning]\n", "- image: images/cross_entropy.png" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "\n", "plt.rcParams['figure.figsize'] = (16, 10)\n", "plt.rcParams['text.usetex'] = True\n", "plt.rc('font', size=15)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classification\n", "\n", "Classification is the task to classify the data with labels. If we have two kinds of labels, its task is called **binary classification**, and labels more than 2, then that task is **multi-class classification**. In binary classification, variable (or label) is either 0 or 1, or True or False. For example,\n", "\n", "- Exam: Pass or Fail\n", "- Spam: Not Spam or Spam\n", "- Face: Real or Fake\n", "- Tumor: Malignant or Benign (or Not Malignant)\n", "\n", "To interpret it in model side, we must encode the label through one-hot encoding.\n", "\n", "## Logistic Regression\n", "\n", "Unlike linear regression, **Logistic Regression** is an regression approach to handle the classification problem. So what's the difference between Logistic Regression and Linear Regression? At first, let's look at the type of data we care.\n", "\n", "![logistic_linear](image/logistic_linear_diff.png)\n", "\n", "There are two types of data, discrete (counted) and continuous (measured). Roughly speaking, discrete data can count it by hand, so it can classify it with its value. But continous data can only measure its information, and cannot classify it with its value.\n", "\n", "## Hypothesis Representation\n", "\n", "While we use linear regression, the output $Y$ can calculated from the product of the weight vector $\\theta$ and observation $X$, adding bias $b$. Mathematically, we can derive this,\n", "\n", "$$ H_{\\theta}(X) = \\theta^TX + b$$\n", "\n", "And we can define the cost function by measuring the error between the hypothesis and actual data. But what about logistic regression? In binary classification task, its output is only 0 or 1. So we cannot bring the same hypothesis and cost functions from linear regression.\n", "\n", "So we need a new function to classify it correctly, and here is a new hypothesis.\n", "\n", "$$ H_{\\theta}(X) = g((\\theta^T X)) $$\n", "\n", "There is a new function $g(\\dot)$ If this function can outputs the probability of True or False, compare it with decision boundary, we can handle the classification problem with logistic regression.\n", "\n", "![logistic](image/logistic.png)\n", "\n", "## Sigmoid (Logistic) function\n", "\n", "And, Sigmoid function is introduced. (Also called logistic function). It has following term,\n", "\n", "$$ g(z) = \\frac{1}{1 + e^{-z}} $$\n", "\n", "and it generates the output between 0 and 1. Suppose that $z$ is changed from $-\\infty$ to $\\infty$, then\n", "\n", "$$ \\lim_{z \\to -\\infty} g(z) = \\frac{1}{1 + \\infty} = 0 \\quad \\lim_{z \\to \\infty} g(z) = \\frac{1}{1 + 0} = 1 $$\n", "\n", "When the $z = 0$, then $g(z) = 0.5$, so it is reasonable to define the decision boundary to 0.5. If $g(z)$ is less than 0.5, then the function classify its data to negative, and vice versa. \n", "\n", "## Cost Function\n", "Cost function is also changed. Because it cannot measure the difference between the hypothesis and label. In common sense, $H_{\\theta}(x) = y$ then cost function must be 0. To do this, we can introduce the new type of cost function, [Cross-Entropy](https://en.wikipedia.org/wiki/Cross_entropy). Cross Entropy measures the average number of bits needed to identify an event. Mathematically, it can expressed like this, if the probability of event happening is $p$,\n", "$$ \\text{C.E.} = -y\\log(p) - (1-y)\\log(1-p) $$\n", "\n", "so we can calculate the Cross Entropy in terms of $y$,\n", "\n", "$$ \\text{C.E.} = \\begin{cases} -\\log(p) & \\text{if } y = 1 \\\\ -\\log(1-p) & \\text{if } y=0 \\end{cases} $$\n", "\n", "We can apply it in cost function,\n", "\n", "$$ \\text{Cost}(H_{\\theta}(x), y) = -y \\log(H_{\\theta}(x)) - (1-y)\\log(1-h_{\\theta}(x)) $$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's visualize the cost function." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6oAAAJnCAYAAACefa7AAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dfYxc130f/O8hKdtaSwZNSlrasbkM7diy0yYRTQe18tZCVIKmKYoUkpKWTp0YCd0X5J88faz44V8tQiR2UqBFWgRiEBhOtGhtOUXbuG0Q0S1goG5TU1LcNKlsVwxJx8lSlGTalNeOXnifP+6Od7jcl5ndmbl35n4+wGJ5D2d3jq7IPfzec87vlKqqAgAAAG2xq+kOAAAAQD9BFQAAgFYRVAEAAGgVQRUAAIBWEVQBAABoFUEVAACAVtnTdAeA5pRS9iZ5IMm9VVXd33R/AKDrSiknkjy3cnm4qqoPNdkfaIqgCh1VSjmS5HDqwfBww90BgM7rhdSqqj6+cn24lPJQVVXva7hrMHGCKnRUVVWPJ3l8JbACAM17X1VV7+hdVFV1rpRytMkOQVPsUQUAgIatbMdZb4XTc6WUY5PuDzRNUIWGlFKOlVLuK6U8tKb9/StLfwCACWp4bO5tx1nrSmzRoYMs/YUGrDw13VtV1cdLKY+UUh6squrKym//WJJf3ORrH9ro99aypwUABtOCsXnfJl+2d9DvD7NCUIVmHF0ZCI8kudIbCFcGySNJzmz0hcInAIyFsRlaxNJfaEBVVb3B7n1JPtb3W0eTnOt7ggsATECLx2azqXSSoArNeiDJI33X92aTJ7YAwNg1NTafzfrLf/cleXwC7w+tYukvNKRvL0z/4Hcsm+yBWfm6Rzb7/T7PWYoEAINrcmyuqupKKeW5UsreNbO3a/sDnSCoQkNWBqRvXq/sidl0D8zK190/4q5sVrwBADqjBWPzB5OcSPKhvvcXUumkUlVV032AziqlvD/J/iRPJXlHkmNVVb1pQu99OMl9qZc0HUs9KD5VVdXpSbw/ALRRk2PzyvufyOoxNYerqvrQpN4b2kRQhZZYKW3/lAEJANrB2AzNEVShIaWUp5LcX1XV4yt7Yv4kybeq+AsAzTA2Q3uo+gsNWBn8zlRV1avi9+tJfsZACADNMDZDu5hRhYb07UE5nORxFf0AoFnGZmgPQRUAAIBWsfQXAACAVmn1Oaq33XZbdejQoaa7AcCMeOyxx56pqur2pvsxzYzNAIzSRmNzq4PqoUOHcvbs2aa7AcCMKKVcaLoP087YDMAobTQ2W/oLAABAqwiqAAAAtIqgCgAAQKsIqgAAALSKoAoAAECrCKoAAAC0iqAKAABAqwiqAAAAtIqgCgAAQKsIqgAAALSKoAoAAECrCKoAAAC0iqAKAABAqwiqAAAAtIqgCgAAQKsIqgAAALSKoAoAAECrCKoAAAC0iqAKAABAqwiqAAAAtMrsBtUDB5JSbvw4cKDpngEAAEydxcXk0KFk16768+Li+N5rz/i+dcMuXRquHQAAgHUtLiYnTiTLy/X1hQv1dZIcPz7695vdGVUAAABG4uTJ1ZDas7xct4+DoAoAAMCmLl4crn2nBFUAAAA2dfDgcO07JagCAACwqVOnkrm569vm5ur2cZjdoDo/P1w7AAAA6zp+PDl9OnnFK+rrhYX6ehyFlJJZDqpLS0lVJWfO1Nef+lR9vbTUbL8AAACm0PHjybd+a/LAA8n58+MLqcksB9WeW26pP1+92mw/AAAAptzVq8mtt47/fWY/qPbuoqAKAACwI1evrs4FjpOgCgAAwJaqKnn+eTOqoyGoAgAA7NjXvlaHVUF1FOxRBQAA2LFepBJUR2HPnuTmmwVVAACAHRBUR+3WWwVVAACAHRBUR01QBQAA2BFBddQEVQAAgB0RVEdNUAUAANgRQXXUBFUAAIAdEVRHTVAFAADYEUF11G65JXn++aZ7AQAAMLV6keqWW8b/Xt0IqmZUAQAAduTq1eTVr052TSBFdieoPv98UlVN9wQAAGAqXb06mWW/SZeCalUlX/ta0z0BAACYSoLqqPXupuW/AAAA23L16mT2pyaCKgAAAAMwozpqgioAAMCOCKqjJqgCAADsiKA6aoIqAADAjgiqoyaoAgAA7IigOmqCKgAAwLa9/HKyvCyojpagCgAAsG3PP19/FlRH6dWvrj8LqgAAAEPrRSlBdZR27arDau8xAAAAAAMzozout95qRhUAAGAbzKiOi6AKAACwLYLquAiqAAAA2yKojougCgAAsC2C6rgIqgAAANsiqI6LoAoAALAtvSh1yy2TeT9BFQAAgE1dvZqUUp/6OQmCKgAAAJu6erWeTS1lMu/XraC6vJy8/HLTPQEAAJgqV69Obn9q0rWgmiTPP99sPwAAAKaMoDouvbtq+S8AAMBQBNVx6ZWnElQBAACGIqiOi6W/AAAA2/L884LqeFj6CwAAsC1mVMdFUAUAANgWQXVcBFUAAIBtEVTHRVAFAAAY2ksvJd/4hqA6HoIqAADA0HoRSlAdh7m5ZNcuQRUAAGAIguo4lVKfpSqoAgAADExQHbdbbxVUAQAAhtCLULfcMrn3FFQBAADYkBnVcRNUAQAAhiKojpugCgAAMBRBddwUUwIAABiKoDpuZlQBAACGIqiO2623Js8/33QvAAAApsbzzye7dyevetXk3rN7QdWMKgAAwMCuXq2jVCmTe8/uBdVvfCN56aWmewIAADAVekF1kroXVBOzqgAAAAPqRFAtpRwrpdw36fdNIqgCwDoaHZsBaL2ZD6qllL1JHkqyb5Lv+02CKgBcp/GxGYDWm/mgmuSBJGcm/J6rBFUAWKvZsRmA1pvpoFpKOZamB0JBFQC+qRVjMwCtN7NBdWVZ0d6qqs4N8NoTpZSzpZSzly9fHm1HBFUASNKisRmA1pvZoJrkWFVVHx/khVVVna6q6mhVVUdvv/320fZCUAWAnnaMzQC03tWryS23TPY9xx5USylHkjw+7vcZSO/uCqoAdFirxmYAWu0v/iJ58cXJz6jumcB77EtytJTSuz6WZF8pJVVVnZ7A+68yowoASZvGZgBarRedZi6oVlV1XZGGUsq9SR5tZCB81auS3bsFVQA6rVVjMwCtNrNBtV8p5UTqp7Z7SynPDbo3ZoQdqO/w889P9G0BoK0aH5sBaLVedJrVYkpJvlmM4bVVVd078YHwwIE6qF65kvzqr9a/LqVuB4COanRsBqDVFheTe++tf/2P/lF9PSkTnVFt1KVLw7UDAAB01OJicuJEsrxcXz/9dH2dJMePj//9JzqjCgAAQPudPLkaUnuWl+v2SRBUAQAAuM7Fi8O1j5qgCgAAwHUOHhyufdQEVQAAAK5z6lQyN3d929xc3T4J3Qmq8/PDtQMAAHTU8ePJ6dOrYXVhob6eRCGlpEtBdWkpqarkwx+ur8+dq6+XlprtFwAAQAsdP568613J3Xcn589PLqQmXQqqPXfcUX9++ulm+wEAANByTz+9GqEmSVAFAABgXYLqpAiqAAAAW7p2Lbl8WVCdjNtvrz8LqgAAABt67rk6rAqqk3DzzcmttyaXLjXdEwAAgNbqRSZBdVLuuMOMKgAAwCZ6kUlQnRRBFQAAYFOC6qQJqgAAAJsSVCdtfl5QBQAA2MTTTye7diX790/+vbsZVO+4o66zfO1a0z0BAABopaefrg9N2dVAauxuUL12ra63DAAAwA2efrqZZb9Jl4NqYvkvAADABgTVSRNUAQAANiWoTpqgCgAAsClBddJ6d/vSpWb7AQAA0ELf+Eby1a8KqpO1b19dusqMKgAAwA2aPEM16WpQ3b07ue02QRUAAGAdgmpT7rhDUAUAAFiHoNqU+XlBFQAAYB29qDQ/38z7dzeomlEFAABYlxnVpgiqAAAA63r66WRuLnn1q5t5/24H1a9+ta67DAAAwDc1eYZq0vWgmiSXLzfbDwAAgJYRVJvSu+uW/wIAAFxHUG1K765futRsPwAAAFrm0iVBtRlmVAEAAG5QVWZUmyOoAgAA3ODKleSllwTVZrz61cnNNwuqAAAAfZo+QzXpclAtxVmqAAAAawiqTZufF1QBAAD69CLS/Hxzfeh2UDWjCgAAcB0zqk0TVAEAAK7Ti0i33dZcHwTVp5+u6y8DAACQp59O9u9P9uxprg+C6osvJl/5StM9AQAAaIWmz1BNBNX686VLzfYDAACgJS5dElSb1bv79qkCAAAkMaParAMHkh/8wfrX3//99bmqpdTtAAAAHbS4mHzhC8kjjySHDtXXTehuUN1oua9lwAAAQActLiYnTiTXrtXXFy7U102E1e4GVQAAAL7p5Mlkefn6tuXlun3SBFUAAABy8eJw7eMkqAIAAJCDB4drHydBFQAAgJw6ldx00/Vtc3N1+6R1N6jOzw/XDgAAMMOOH09+4AdWD0RZWEhOn67bJ627QXVpKamq5F/+y/r6z/6svl5aarZfAAAADXnlK5Pv+I668u/5882E1KTLQbVnYaH+fOFCs/0AAABo2IULqxGpSYKqoAoAAJCqElTbQ1AFAADIlSvJ1avJoUNN90RQTV7zmuS1rxVUAQCATutFIjOqbbGwUO8UBgAA6KheJBJU22JhwYwqAADQaWZU26YXVKuq6Z4AAAA04sKF5Oabk9tua7ongmptYSF5/vnky19uuicAAACN6FX8LaXpngiqtV5ZK8t/AQCAjrpwoR0VfxNBteaIGgAAoOPacoZqIqjWBFUAAKDDvva15JlnBNV22b8/mZtzRA0AANBJbar4mwiqtVIcUQMAAHSWoNpWgioAANBRgmpbHTokqAIAAJ104UJy003J617XdE9qgmrPwkLy7LP1LmIAAIAOuXAheeMbk927m+5JTVDtUfkXAADoqDYdTZMIqqsEVQAAoKME1bbq/V9xRA0AANAhL7yQ/NmfCart9LrX1buHzagCAAAd8sUvJlUlqLbTrl3JwYOCKgAA0Cm9CHToUKPduI6g2s9ZqgAAQMe07QzVRFC9nqAKAAB0zIULSSnJG97QdE9WCao9Bw4kH/5wvYu4lNWPAwea7hkAAMBYLC4mv/Ir9R7Vt7ylvm6DPU13oDUuXRquHQAAYIotLiYnTiTLy/X1hQv1dZIcP95cvxIzqgAAAJ108uRqSO1ZXq7bmyaoAgAAdNDFi8O1T5KgCgAA0EEHDw7XPkmCKgAAQAedOpW84hXXt83N1e1NE1R75ueHawcAAJhix48nf/Nv1r8upT6t8/Tp5gspJYLqqqWluibzL/xCfX31an29tNRsvwAAAMbkla+sA+q1a8n58+0IqYmgeqM776w/f/7zzfYDAABgzJ58cjUCtYmgutbb3lZ/fvLJZvsBAAAwRteuCarT401vSnbvFlQBAICZ9qUv1eemCqrT4JWvTA4fFlQBAICZ1os8vUWlbSKorufOO5P/83+a7gUAAMDY9CKPGdVpceeddTGll19uuicAAABj8eSTyd69yR13NN2TGwmq67nzzuSFF+r6zAAAADOoV0iplKZ7ciNBdT0q/wIAADOurRV/E0F1fW99a/1ZUAUAAGbQV76S/PmfC6rTZd++eqG2oAoAAMygz32u/iyoThuVfwEAgBnV5oq/iaC6sTvvNKMKAADMpCefTG66KTl8uOmerE9Q3ciddybPPps880zTPQEAABipJ59M3vzmOqy2kaC6EZV/AQCAGdXmir9JsmcSb1JKOZJkX5K9SQ4nSVVVH5rEe29b7//ak08m3/u9zfYFAEZsKsdmAEbixReT//t/kx/90aZ7srGJBNUkn0xyT1VVZ5KklFKVUs5UVfX4hN5/eAcPJq96lRlVAGbV9I3NAIzEuXPJSy+1e0Z1Ukt/7+kNfKWUvSttVyb03sM7cCDZvTv5xjeSf/bPklLqjwMHmu4ZAIzKdI3NAIzE4mLyfd9X//rBB+vrNprIjOqap7MPJPl4VVXnJvHe23Lp0nDtADBlpm5sBmDHFheTEyeS5eX6emmpvk6S48eb69d6JlZMqZRyuJRyIsm9VVXdv8nrTpRSzpZSzl6+fHlS3QOAzjE2A3TLyZOrIbVneblub5uJBdWqqs5VVXU6yaOllEc2ed3pqqqOVlV19Pbbb59U9wCgc4zNAN1y8eJw7U2a+PE0KwPisVLK+yf93gDAjYzNAN1w8OBw7U0ae1AtpRwppTy1pvlckjeN+70BgBsZmwG66dSp+mCTfnNzdXvbTGJG9UqSM2vaDid5dALvvT3z88O1A8B0mb6xGYAdO358tWhSKcnCQnL6dPsKKSUTqPpbVdW5UsojK8UakuQdSR6squrj437vbVtaWv31z/1c8mu/lly9muyZ1LGzADA+Uzk2AzASr3xlcuutyZUrya6JbwQd3KSOp1n71HZ63HVXfZ7q5z6XfPu3N90bABiJqR6bAdi2J55IvvM72x1SkwaKKU2dI0fqz0880Ww/AAAAduDll5PPfnY14rSZoLqVt7613nEsqAIAAFPsC1+oz029666me7I1QXUre/Yk3/EdyeOPN90TAACAbevNvQmqs+Kuu5I/+IOkqpruCQAAwLY88UTyilckb3970z3ZmqA6iLvuqstinT/fdE8AAAC25Yknkr/0l5Kbbmq6J1sTVAfRmxu3TxUAAJhCVVXvZpyGZb+JoDqYv/yXk927BVUAAGAqffGLyXPPCaqz5eabk7e9TVAFAACmUi/KTMPRNImgOri77lL5FwAAmEpPPJGUUh9oMg0E1UHddVfy53+eXLrUdE8AAACG8sQTyVvfmrz61U33ZDCC6iAOHEh+7udWf11K/XHgQLP9AgAA2MTiYnLoUPIf/kNy8WJ9PQ32NN2BqbDRLKrZVQAAoKUWF5MTJ5Ll5fp6ebm+TpLjx5vr1yDMqAIAAMygkydXQ2rP8nLd3naCKgAAwAy6eHG49jYRVAEAAGbQwYPDtbeJoAoAADCDTp1K5uaub5ubq9vbTlAdxPz8cO0AAAANO348+af/dPV6YSE5fbr9hZQSVX8Hs7S0+uvv/d7k2rXk059urj8AAAAD2L+//vxHf5S8/e3N9mUYZlSHdffdyWOPJd/4RtM9AQAA2NR/+2/Ja1+b3Hln0z0ZjqA6rLvvTl54IXn88aZ7AgAAsKlPfzp517uSXVOW/Kasuy1w9931Z0t/AQCAFvvyl5M//uPVCDNNBNVh3XFH8uY313PoAAAALfU//kf9WVDtirvvrmdUq6rpngAAAKzr059Odu9O3vnOpnsyPEF1O+6+O3n66eTcuaZ7AgAAsK5Pfzr5zu9Mbrml6Z4MT1DdDvtUAQCAFnvppeT3fz/5nu9puifbI6hux9vfnrzmNYIqAADQSv/rfyVf+9p07k9NBNXt2b07+St/RVAFAABaqRdVBNUuOXAg+b3fqx9TlLL6ceBA0z0DAAA6bnExefDB+tff93319bQRVLfj0qXh2gEAACZgcTE5cSJZXq6vL16sr6ctrAqqAAAAM+LkydWQ2rO8XLdPE0EVAABgRly8OFx7WwmqAAAAM+LgweHa20pQBQAAmBGnTtV1XvvNzdXt00RQ3Y75+eHaAQAAJuC7vzupqmTfvjqwLiwkp08nx4833bPh7Gm6A1NpaWn11z/wA8nzzyePPdZcfwAAAJJ88pP15//+35O3vKXZvuyEGdWduuee5Iknkueea7onAABAx33yk8kb3pB827c13ZOdEVR36p576rn1//pfm+4JAADQYdeu1bHknntu3Kc6bQTVnfru705uuWV1jh0AAKABn/1s8uyzdVCddoLqTt10U/L93y+oAgAAjepFEkGV2rFjyec/n/zpnzbdEwAAoKM++cnkbW9LXv/6pnuyc4LqKPQeWZhVBQAAGvDCC8mnPjUbs6mJoDoaP/iD9eef/Ml613Lv48CBRrsFAADMvsXF5ODBZHk5+ehH6+tp5xzVUbh0abh2AACAEVhcTE6cqENqkly+XF8nyfHjzfVrp8yoAgAATKmTJ1dDas/yct0+zQRVAACAKXXx4nDt00JQBQAAmFIHDw7XPi0EVQAAgCl16lSye/f1bXNzdfs0E1RHYX5+uHYAAIAR+PEfT171quTVr64PHllYSE6fnu5CSomgOhpLS0lVJS++mLz2tfUxNVVVtwMAAIzJ7/9+8rWvJb/xG8m1a8n589MfUhNBdbT27En++l9P/uN/rP+UAAAAjNEnPlHHkB/6oaZ7MlqC6qj9yI/Uhxd95jNN9wQAAJhxn/hE8n3fl+zd23RPRktQHbUf+qF6N/Pv/E7TPQEAAGbYhQvJH/5hPVc2awTVUdu3L/me76kfbQAAAIxJL3IIqgzmR34k+exnky9+semeAAAAM+oTn0i+7duSt7yl6Z6MnqA6agcOJO9/f/3rgwfrGtGl1O0AAAA7tLhYR43f/d36oJHFxaZ7NHpDBdVSyi+u03bXeu2ddenScO0AsAPGZoBuWVxMTpxYXbx59Wp9PWthddgZ1fvWNlRV9cR67QDARBibATrk5Mlkefn6tuXlun2W7BnkRX1PZQ+v84T2HUn2jbRXAMCmjM0A3XTx4nDt02rQGdXnVj7KOh+PJjk6lt4BABsxNgN00MGDw7VPq4FmVKuq+uUkKaW8s6qqnx9vlwCArRibAbrp1KnkJ38yeeml1ba5ubp9lgy1R7Wqqgf6r0sph0oph0bZoak3Pz9cOwDsgLEZoFt+7MeSm2+uw2kpycJCcvp0cvx40z0brWGr/n60lPJdK7/+f5OcS/JYKeX/GUfnptLSUlJV9ceLLya33Zb8+I/X7QAwYsZmgG751KfqSr8f+Uhy7Vpy/vzshdRk+Kq/x6qq+oOVX/9SkiNJ3pTk/xtpr2bFnj3J3/7bye/8TvL1rzfdGwBmk7EZoEMeeaSeTf3hH266J+M1bFAtSVJKuSfJV6qq+oOqqq702lnH/fcnX/ta8p//c9M9AWA2GZsBOuKll5Lf/u3kb/yNOqzOsmGD6plSykeTPJLkdFIfKp56mRHr+at/tV7++8gjTfcEgNlkbAboiE99Krl8uZ4Lm3XbKaZ0JslP91UY3JfkwVF3bGa84Q3JM88k/+bf1Ludex8HDjTdMwBmgLEZYPYtLiaHDiX33FNHiatXm+7R+A07o5okn0ny46WUz6w8wX22qqpPjrhfs+PSpeHaAWB4xmaAGbW4mJw4kVy4UF9XVfKzP1u3z7Jhq/7ek+S/pB4QfynJY0k+WUr5a2PoGwCwBWMzwGw7eTJZXr6+bXm5bp9le4Z8/S8leUdVVX/SayilnEny0STfNsqOAQADMTYDzLCLF4drnxXDLv19U/9AmCRVVT2eZP/ougQADMHYDDDDDh4crn1WDBtUz5ZSfrS/oZTy00nOjq5LAMAQjM0AM+zUqeQVr7i+bW6ubp9lwy79vT/JY6WUv5+67P07k3xrkneMumMzY35+/cJJ8/OT7wsAs8jYDDDDjh9PfuVXkj/8w+TatXom9dSpun2WDXs8zVeqqnpz6nPavpLkoaqq9ldVdX4cnZsJS0t1aa6qSr7whbrtgx+s2wFgh4zNALPt6afrkPqP/3EdVM+fn/2QmmwRVEsp31VK+a617VVV/XZVVT9fVdWvl1J+er3XsI43vzm5++7kIx+pgysADMnYDNAt//pfJy+/nPzETzTdk8naakb115Mc3eI1f5Lkg6PpTgf8vb+X/PEfJ0880XRPAJhOxmaADvnN30ze8Y7k27+96Z5M1lZB9U1JPrbZC1YOFD82sh7NugceqHdD/+ZvNt0TAKaTsRmgI/73/04ef7ye6+qaLfeoVlX11QG+z1dG0JdueNvbkhdeSP7Fv0hKWf04cKDpngEwJYzNALNvcTH5nu+pf/3Lv1xfd8lWQfVsKeWvbfaCUso9qasMMoj1KgBv1g4A1zM2A8y4xcXkxInkqyuPJf/0T+vrLoXVrYLqh5J8vJSysN5vllIOpV5+9P7RdgsA2ICxGWDGnTyZLC9f37a8XLd3xabnqFZVdaaU8sEkf1JKeSTJo6mf0B5Ocm+S+5L8fFVV/2XsPQUAjM0AHXDx4nDts2iQPaofSvLmJCXJzyc5s/K5JDlaVdUvj7WHAMB1jM0As+31r1+//eDByfajSZvOqPZUVXUuyQNj7gsAMCBjM8Dsete7ko9//Pq2ubnk1Klm+tOELWdUGbH5+eHaAQCAznj55eR//s/63NSFhfqAkIWF5PTp5Pjxpns3OQPNqDJCS0urv37hheSNb6wfmfy7f9dcnwAAgFb43d+t96I+8khy331N96Y5ZlSb9IpXJD/1U8knPpF86UtN9wYAAGjYQw/Viy3/1t9quifNElSb9hu/Uc/vv+EN9bx+7+PAgaZ7BgAATMjiYh0Jfud3kq9/PfnYx5ruUbMs/W3aM8+s337p0mT7AQAANGJxMTlxYvXs1K9+tb5OurUvtZ8ZVQAAgAadPLkaUnuWl+v2rhJUAQAAGnTx4nDtXSCoAgAANOjgweHau0BQBQAAaNB733tj29xccurU5PvSFoJq0+bnh2sHAABmyuc+l7zqVckb31gfALKwkJw+3d1CSomg2rylpaSq6o8/+qO67dSpuh0AAJhpX/pSfRTNP/gH9Z7Ua9eS8+e7HVITQbVd3v725Ad/MPlX/yp58cWmewMAAIzZr/1aHU5/9meb7km7OEe1TQ4cWD0/9RWvWG2fnzfDCgAAM2RxMfnAB5IvfjG5+ebk059OvvVbm+5VewiqbdILqYO2AwAAU2dxMTlxYvXs1K9/vb5OLPntsfQXAABggk6eXA2pPcvLdTu1icyollKOJDm2cvnOJA9VVXVmEu8NANzI2AzQnIsXh2vvokkt/T1WVdWHkqSUsjfJn5RS7qmq6vEJvT8AcD1jM0BD3vjG9UPpwYOT70tbjX3p78oT2w/0rququpLkbFaf4gIAE2RsBmjWevtQ5+bqUyqpjT2orjyZvX9N8+EkV8b93lNnfn799ttum2w/AJhpxmaAZn3mM8lrXlPPrJaSLCwkp08rpNRvIsWU+ve8lFIOJ9mX5GPrvbaUcqKUcraUcvby5cuT6F57LC0lVbX6cfvtdfszz9R/gnsfBw40208App6xGWCyFheTQ4eSXbuSM34qjKIAAB0HSURBVGeSH/7hevnvtWvJ+fNC6lpNVP19KMk9K8uMblBV1emqqo5WVXX09l5Q66qN/jHguBoARsvYDDBGveNoLlyo56OS5N//+7qd9U00qJZS3p/kgwo1AEA7GJsBxm+942i+/nXH0WxmYkG1lHJfkjO9pUYry4wAgIYYmwEmw3E0w5tIUC2lHEtypfe0dqUM/pFJvDcAcCNjM8DkbHTsjONoNjaJ42kOJ3k0yaOllKqUUiX5cpJz435vAOBGxmaAyTp1Ktm9+/o2x9Fsbs+436CqqnNJyrjfZybNz69fOGmjY2wAYADGZoDJuuuu5OWX6yNprl6tZ1JPnVLpdzNjD6rswNLS9dfveU/ysY8lTzzRTH8AAICh/ZN/ktxyS/LUU8lttzXdm+nQxPE0bNd/+k/JN76RvP71zlUFAIAW6z839WMfS+65R0gdhqA6TZ55Zv1256oCAEBrrHdu6u/9nnNThyGoAgAAjJBzU3dOUAUAABgh56bunKAKAAAwQs5N3TlBFQAAYIR+4Rfqmqf9nJs6HEF1mmx0fqpzVQEAoDV2766LKO3fXwfWhYXk9Gnnpg5DUJ0mS0v1n/jex6231u2XLjmuBgAAGra4WIfSv/t3k5tuSv75P0+uXUvOnxdSh7Wn6Q6wA1evrt/uuBoAAJio3pE0vWq/L76YvO999TySkDo8M6oAAAA7tN6RNMvLjqTZLkEVAABghxxJM1qCKgAAwA697nXrtzuSZnsEVQAAgB26/fYb2xxJs32C6jTb7FgaFYABAGCsFheTQ4eSXbuSz342ede76qq/jqTZOVV/p9nS0uqv154o3KMCMAAAjNzaKr9JHVaF09EwowoAADAkVX7HS1AFAAAYkiq/4yWoAgAADGmjar6q/I6GoAoAADCkv/N3bmxT5Xd0BNVZsVEF4Ne+drL9AACAGfcXf5H823+b3HFH8sY3qvI7DoLqrFhaSqpq9aMXXL/85dWjahxXAwAA29Y7juZVr0o+//nkp36q3pN67Vpy/ryQOkqC6qza6Fgax9UAAMDQesfRXLiw2varv1q3M3qCKgAAwBYcRzNZgioAAMAWHEczWYIqAADAFr7lW9ZvdxzNeAiqXaS4EgAAbKlXPGnXrvVLvTiOZnwE1Vm10XE1aymuBAAAN+gvnlRVyYsv1oF1/37H0UzCnqY7wJgsLV1/XUoz/QAAgCm0XvGka9eSW25JnnmmmT51iRlVAACANRRPapagCgAAsMZGRZIUT5oMQRUAAGCNBx+8sU3xpMkRVLtis+JKqgADAECSuojSwkLyD/9hfb13r+JJTVBMqSsGLa6kCjAAAB3Vq/TbX0TphReS3/otAXXSzKgCAABk/Uq/y8t1O5MlqAIAAESl3zYRVAEAAJK85jXrt6v0O3mCKjdSXAkAgI5YXEwOHUp27Uq+8pX6cz+VfpshqHbVZlWA+ymuBADAjOoVT7pwIamqum337mT/fpV+m6bqb1cNWgUYAABm1HrFk158MbnlluSZZ5rpEzUzqgAAQCcpntRegioAANBJG5VjUTypeYIqW1NYCQCAGdFfPGntbrhE8aS2EFSpDVJcSWElAACm2NriSVWleFJbKaZErf9xksJKAADMoPWKJ738suJJbWRGFQAA6ATFk6aHoAoAAHTCt3zL+u2KJ7WPoMpweoWVFFcCAGAKKJ40nQRVbjRIYaVEcSUAAFptbfGkl15SPGlaKKbEjdY+alJcCQCAKaR40vQyowoAAMwkxZOml6DKztizCgBAy/T2pVbV+r+veFL7WfrL6NizCgBAw3r7Utcu+e1RPGk6mFFla4MWVwIAgIatty+1R/Gk6WFGla0prgQAwJTYaP9pKcn58xPtCjtgRhUAAJgZt922frt9qdNFUGW0FFcCAGDCesWTdu1KLl++cQGgfanTR1BleIPuWVVcCQCAMesVT7pwYbXK7549yf79dWC1L3U62aPK8OxZBQCgJdYrnvTii8kttyTPPNNMn9g5M6oAAMDU2qh40kbtTAdBlfGyXxUAgBHr35O6EcWTppulv0yG/aoAAIxAb0/qRmelJoonzQIzquzcoMWVAABgh9bbk5oku3crnjRLzKiyc/3FlRRWAgBgjDbae3rtWv3BbDCjyuQ4YxUAgG3o35O60byIPamzxYwqzbBnFQCAAazdk9o7K7WfPamzx4wqo2W/KgAAI2RPajeZUWW0+verJvasAgCwI/akdpMZVZpjzyoAABvo7Utdb6lvYk/qrDOjSjvYswoAwIqtzkq1J3X2mVFlvOxZBQBgSBvtS03sSe0KM6qMlz2rAAAMaaN9qaUk589PtCs0xIwq7WHPKgBAZ/WflWpfKmZUaSd7VgEAOmOrPamJfaldY0aVybJnFQCANZyVylpmVJmsYfas9n5vfv7GrwMAYKotLtYB9eLFjZf6Oiu1uwRV2s8yYACAmTLIUt/EntQus/QXAACYqM2On+mxJ7XbBFWaZc8qAEDnbHT8TGJPKjVBlWYtLdWbEjbamNDj6BoAgKnWf/zMRmVKFhbqPannzwupXWePKtPHnlUAgKmydk/qenMUlvrSz4wq7WEZMADATHL8DMMyo0p7bOfomsTxNQAALeT4GXZCUGX6WQoMANAqjp9hpyz9BQAARsrxM+yUoEp72bMKADCVHD/DTgmqtFf/0TWOrwEAaL3eETQb/dPN8TMMyh5VZo89qwAAE7fVvlRLfRmGGVWmh6XAAACttdm+VEt9GZagyvTYzlJgy4ABAMamt9R3167kwoX1X1OKpb4Mz9JfZptlwAAAY+EIGsbJjCoAADA0R9AwToIq02vQPasqAgMAjMQgS30TR9Cwc5b+Mr2WllZ/XcpgX2MpMADAtgy61Hdhod6TCjthRhUAANiSpb5MkqDKbBjm6BpLgQEABmKpL02ZSFAtpRwupTxSSjk2ifejg4Y5uqafpcBARxmbga30lvpeuLD5P68WFpJr1xxBw2iNfY9q3wB4eNzvBQBszdgMDMJSX5o09hnVqqrOVFV1Jslz434v+CZLgQE2ZGwGNtNb7mupL01S9ZfZ1F8ROFEVGABgAINU9lXVl0loXTGlUsqJUsrZUsrZy5cvN90dAOg8YzN0x1bLfS31ZVJaF1SrqjpdVdXRqqqO3n777U13h1lhKTDAthmbYbYNWtnXUl8mydJfusFSYACAGwyy1Dex3JfJa92MKgAAMBkq+9JWgirdtJ2lwJYBAwAzYNClvir70qRJnKN6JMmxJEeTPFhKOVxV1elxvy9sajtLgS0DBmaEsRm6y1JfpsXYg2pVVY8neTzJh8b9XjB2/YF2fv7GwAswBYzN0C2Li/US34sX61nUl1/e/PWW+tIGlv5CMtxS4B4zrABAy/VmUC9cSKpq85BqqS9touovJNfPjA5aERgAoOUGKZaUWOpL+5hRhZ1w5ioA0EK9gkmbFUvqsdSXNhJUYa3tLANOLAUGAFqhf7nvRnbvttSXdrP0F9baTkVgAICW2Gq579yccEr7mVGFrWznzFVLgQGACRr0bFQzqEwLM6qwle3OsFoKDABMgLNRmUVmVGGczK4CAGPQP4P6nvdsHVIVTGLaCKowLGeuAgANcjYqXWDpLwxLsSUAoEHORqULzKjCpCi0BABs06DFknos9WXaCaqwU5YCAwBjtHap70acjcosEVRhp5aW6lFjs5FjPWZYAYBN9GZR3/3uwYolfeQjybVr9XJfIZVpJ6jCKG1ndjUxwwoAXKd/FnUzZlCZVYopwSjtpNBS/2vn52/8XgDATFtcrAslXbxY70XdrJpvolgSs01QhTYywwoAndKbQe0t8d0qpCqWxKyz9BfGabtLgQGAThn0yJnEUl+6QVCFceovtLTdYksKLQHATNrOkTMPP6xYEt0gqMIkOcoGAIgjZ2ArgipM0nZnWB1lAwBTr38G9T3vceQMbEZQhSaZYQWATlg7g7pZsSQzqKDqLzSr/wgaR9kAwEwZ9riZxJEz0COowrQzwwoArTPscTOJI2egn6W/0BY7OcrGHlYAaIXePtR3v3uw42YUS4L1mVGFtli7fHeYpcD9zLACQCPWzqJuZW5OOIWNmFGFttrJDCsAMHEnT24dUs2gwmAEVWir7R5lk1gGDAAT0n/kzIULm7/WcTMwOEEVpoWjbACgVdYeObMZM6gwHEEVpsV2Z1gVWgKAkemfQX3Pe7Ze6js3lzz8sBlUGJagCtPKDCsATNTaGdTNjpyxDxV2RtVfmFb9VYKHqRDc/9r5+RurDQMA37S4WBdJunixnkUd5DzUhYV6BhXYPkEVuswMKwBsaO1xM4OE1Lm55NSp8fYLusDSX5gFOznKxh5WALhObx/qu9892JmojpyB0TOjCrNg7fLdYZYC9zPDCkDHrZ1F3crcnHAK42BGFWaRGVYAGNiwlXzNoML4mVGFWWSGFQAGMuw+VDOoMBlmVKELRjHDanYVgBkx7AxqjxlUmBxBFbpgaak+8K33sR2XLlkWDMBU6g+mt92WvPe9g52F2jM3lzz8cH3kjJAKkyGoQhftZIa1x7JgAKZAb2lvL5g++2zywgtbf519qNAsQRW6aBQzrIkZVgBaa9gjZvrNzSUf+Uhy7ZpZVGiKoAqYYQVgpvTPog7KDCq0i6AKXD/DuhNmWAFoyHYLJCVmUKGNBFXgeqOYXU3MsAIwMWv3oW5VIOmmm5L9+82gQpsJqsD11u5fdbQNAC20kyNmPvzh5JlnzKBCm+1pugNAyy0tXX9dyvDfw+wqACPUm0HthdNBj5gxcwrTw4wqMJztzrDavwrADmxnBlWBJJheZlSB4ZhhBWDCzKBC95hRBXbGDCsAYzLsWahmUGF2CKrAzoziaJtLlwRXAK5b3nvbbcl73zv4WaiOmIHZIqgCo+NoGwC2ae0RM88+m7zwwuZfYwYVZpegCozO2qNtdsLsKsDM2+4RM4kZVJh1giowPqOYYTW7CjCT1s6gDlIgqccMKsw+QRUYn1HNsNq/CjATdjKDmtSzqA8/bAYVukBQBSZnVDOsgivAVNioONKgM6g33ZTs328fKnSRc1SByVl7BuuBAztf2mtpMEArrT379NlnB/u63bvrfacHDyanTgmm0FVmVIHmjOJom8QMK0CLDHv2aT8FkoAeQRVoh1EebSO0AkzMTs4+TRwxA6zP0l+gHdYuCy5l59/TsmCAsdru8t6euTnhFFifGVWgnUY1w2pZMMBI7aRyr+JIwKDMqALtpPASQOusnUEd9uxTxZGAQZlRBaaDM1kBGuHsU6AJgiownZzJCjA2vXBaSvITP+HsU2DyLP0FplP/0uBRLAtOLA0GyI3LewdZxOLsU2DUzKgC029Uy4ITM6xAJ+1kea+zT4FxEFSB2eNMVoBNbXT26aDLe519Coybpb/A7HEmK8CGnH0KTAMzqsDscyYr0HE7rdzbe95nBhWYFEEVmH1r97CqGAx0wE4q9ybXL+/9rd+qv9YeVGBSLP0FumdcS4P7v8/8/I3vAzBGi4vJyZPJxYvJvn3J1avJCy/UvzdsnTnLe4GmmVEFGNXS4H72tAIT1Nt32ps1ffbZ1ZA6CGefAm0jqAL0Lw0eZWi1LBgYo53uO+1f2vvhDyfPPOOIGaA9LP0F6GdZMNBSmy3tHXTfaY+lvUDbmVEF2IxlwUAL7HRpb6JyLzBdBFWAzYyjYnCiYjAwkN7y3ne/e/ilvWv3narcC0wTQRVgGI66Acaof9/pbbcl731vPYs6KPtOgVlhjyrATvTvNR3Fftbk+j2t9rPCTNts3+mzzw73vew7BWaJGVWAUbGfFRiCI2UANiaoAoyK/azAFnZ6pEyPpb3ArBNUAcZlbXAdBftZYapstOe0qoY/Uiapl/c+/LBgCsw+QRVgUsa1NFhwhVbphdNSkp/4iZ0dKWN5L9BVgirApIxraXC//uAqtMJEbFapd9jFFGuDqeW9QFep+gvQlP5qvgcOjL5wkkJMMHa9gki9vabDVupN6iNlrl1LDh5MTp0SRgESM6oA7TCO/ayJZcEwBqMqiJTUe04/8hEzpgBrCaoAbWQ/K7TGKAsi2XMKMBhBFaCNJr2fVXCF64yyIFIp9Wd7TgEGJ6gCTAPBFcZqnAWRfuu36u8hmAIMTjElgGk07kJMiWJMdIaCSADtY0YVYNqNqxBTYnaVmaUgEkC7CaoAs8ayYLiBgkgA00VQBZg19rNCEgWRAKaZoAow6wRXOkJBJIDZIagCdM2kg6vQyphstpx3O7Omu3evBlOzpgDNUvUXoOv6Kwj31jeOUi+09szPX/+eMKDFxeTkyeTixWTfvuTq1dUwup1Kvf3m5uw1BWgTM6oArBrH7Opajr1hG3pHyOxkxrSfgkgA7SaoArBqEsuCE/tZGVhvee+7372zI2QSBZEApomgCsDGFGJiwjYriDQsBZEAppegCsDgFGJixEZZEGltMDVrCjC9JlJMqZSyN8mJJOeSHE5ypqqqxyfx3gCMUX9RpAMHRr//VCGmsWlqbB5lQaSbbkpe85rkueeSgweTU6eEUYBZMamqv48keV9VVeeSpJTyaCnl/qqqrkzo/QEYt7UBclzBlVGZ+NjcK4jU22u6k0q9CwuCKcAsG/vS35Untod7A+GKc0mOjfu9AWjQ2mXCtEZTY/PJkzsviDQ3lzz8sKW8ALNuEntUjyZZ+3T2SpJ7J/DeALTFJI6+YVCNjM0XLw7/NY6RAeimSQTVvUmeW9P2bJJ96724lHKilHK2lHL28uXLY+8cABMyqaNvGEQjY/PBg1u/RkEkAJLJVf1dd+BbT1VVp6uqOlpV1dHbb799nH0CoEn9wVVobcLEx+ZTp+qlu/0EUwDWM4mgeiX1k9t++3Pjk1wAumrQ2VaBdlQaGZuPH6+X7i4sCKYAbG4SVX/P5santnuTPDqB9wZgGjmCZtwaG5uPHxdGAdja2GdUV8rcny2lHO5rPprkzLjfGwC4kbEZgLab1Dmq9yc5UUo5l/oJ7s84QxUAGmVsBqC1JhJUVwa+D03ivQCArRmbAWizSVX9BQAAgIEIqgAAALSKoAoAAECrCKoAAAC0iqAKAABAqwiqAAAAtIqgCgAAQKsIqgAAALSKoAoAAECrCKoAAAC0iqAKAABAqwiqAAAAtIqgCgAAQKsIqgAAALSKoAoAAECrCKoAAAC0iqAKAABAq5Sqqpruw4ZKKZeTXNjGl96W5JkRd2eWuV+Dc68G514Nx/0a3E7u1UJVVbePsjNdY2yeGPdrcO7V4Nyr4bhfgxv52NzqoLpdpZSzVVUdbbof08L9Gpx7NTj3ajju1+Dcq+nk/9tw3K/BuVeDc6+G434Nbhz3ytJfAAAAWkVQBQAAoFVmNaiebroDU8b9Gpx7NTj3ajju1+Dcq+nk/9tw3K/BuVeDc6+G434NbuT3aib3qAIAADC9ZnVGFQAAgCklqAIAANAqe5ruwHaUUvYmOZHkXJLDSc5UVfX4Tl87i4a8V0eSHFu5fGeSh6qqOjORjrbEdv+8lFKOJdlbVdXHx9zF1hj2XvX9+TqXZF9VVZ3Z97GNn1kP9DWd6+Dfw8NJPpgtfgZ1/ed72xibB2dsHo6xeXDG5sEZm4fTyNhcVdXUfSR5NMnhNdd7d/raWfwY8l69v+/Xe5N8OcmRpv8b2nq/1tyrp5KcaLr/bb1XSY4keaTv+rEu/dna7t/DlesPduxn1rGVj8eSHBvVffUxkf93xubx3Ctjs7F5LPfK2GxsHuJeNTI2T93S35WUfriqqnN9zeey+rRxW6+dRUPeqyNJPtC7rqrqSpKz6712Vu3gz8sDSbr2VG3Ye/XrSR7su76n6sjsyTbu1Y+tuX429RPJTqiq6kxVP6l9brPXdf3ne9sYmwdnbB6OsXlwxubBGZuH09TYPHVBNcnRJFfWtF1Jcu8OXzuLBv7vX/nBdP+a5sPrfP0sG/rPy8qyok4NhCsGvlf9P7RKKUdKKYdX/rHVFcP+uTpXSnmslHJ4ZZnN/q78w2FIXf/53jbG5sEZm4djbB6csXlwxubxGOnP92kMqntzY5p/Nsm+Hb52Fg3131/1rTdf+Uu4L8nHxta79hnqfq38kN+75qlRVwxzr44mea6Ucl9W9iuUUh4ac//aZNi/h/envk9Ppd4H8uB6r6PzP9/bxtg8OGPzcIzNgzM2D87YPB4j/fk+jUE1Ge4/tisD30a2+9//UOolIF16upYMd7+OVR0q0LCOQe/V3qxupr+y8o+uwyuDY1cM/Odq5b48mvrp4+GVJ7h7x9az6db1n+9tY2wenLF5OMbmwRmbB2dsHo+R/XyfxqB6JfVfrn77s/6a6WFeO4u29d9fSnl/kg92cEnDwPdrZd9Q1+5Pv2H/Hl5Z8w+rc+nOMr9h/lwdTvLOqqpOr+wHeVPqe/WBta+l8z/f28bYPDhj83CMzYMzNg/O2DweI/35Po3H05zNjUl9b+qnHDt57Swa+r9/5YnRN8tIr+xZ6MrymWHu174kR0spvetjSfaVUlJ1o7T7sH8P19OVGYFh7tWRJJ9Z0/YzqasLcr2u/3xvG2Pz4IzNwzE2D87YPDhj83iM9Of71M2o9irerTzd6DmalU3zfZuct3ztrBvmXq1cH0v9dK03EO5N/ZezE4b8s3Vm5cna6ZXB7/Ekj3ZkINzO38Mz67z2o5Pqb5OG/Ht4Jjc+zT6a5JGxd3QK+PneXsbmwRmbh2NsHpyxeXDG5tEZ58/3snK+zVRZc5DsviRn+36A9841et9Wr+2CQe/Vyh+op9b5Fu9wv9b/s9X3NSdSP1U7m3qDfSf2xmzj7+EHUm+o35/6Hw6d+EdpMvS96h2+3nuq/VxX/kwl1/33fyD136lHev/I9PO93YzNgzM2D8fYPDhj8+CMzYNramyeyqAKAADA7Jq6pb8AAADMNkEVAACAVhFUAQAAaBVBFWaMA6gBoF2MzTA8QRVaoJRyXynlsVLKUyuVCnvt7y+lVKWUR1aOKNjq+xxJXQZ80Pd9/za7DAAzzdgMzVL1F1qilPJo+sp9r7TtTfLlJK9dOZtqq+/x/qqqPjTEe+5N8oGqqh7cTp8BYJYZm6E5ZlShPY7lxgORjyY5N+hAmGSoM71Wvu+zaw5mBgBqxmZoiKAKLbCyLOhKVVXn1vzWvUkGPST5net8/SA+nuR9W74KADrE2AzN2tN0B4Ak9RPbsxu0P7TVF68MpjcMhCvLh44lOZx6UO0Vc7hSVdWZJKmq6tzK1wMAq4zN0CBBFdrhnUmu9BdrSD1wHcmaJUcrr3kufQNa6sHuqXW+7wO9fTWllC8neUeS+1Z+b+1SJgBglbEZGiSoQjscS3JPVVXfXEq03pKjUsp9qffFnCmlPJTVAe1w1nlqm5UnwSv7XM6tfK/1CjpcKaUc3ubyJACYRcZmaJA9qtCwlYFqb/9AuGK9JUcfWBkI9ybZ19e+P8kNRR36vud98ZQWAAZibIbmCarQvCNZvyjDvUke7V309qqsPLk9keuf0j6b1T0u67k3yUdXvn6jKoLPDd5lAJhpxmZomKAKzbs36z9RPbqm/WiSX6yq6uOpn9J+pu/3zuX6p7gppZwopXxw5Qnv4aw+1b0vN9o7SJl9AOgIYzM0TFCFhpRS9q6cr/ZAkiN9T2WPrOxx2Zvkx1YGsyR5U1af7h5ZGRR7zqQuxtDvXOoiDkeT3J/kvpUnvqcDANzA2AztUaqqaroPwABWKgp+LPUT2KO9ioF9v/9IVVX3b+P7Hlnv+wEAmzM2w/iYUYXpcSb1E96NBq5HSynHtvF9fyz1IAsADMfYDGNiRhVmSCnlg1VVPTjE648kObxmqRIAMCLGZtgeM6owW35xZa/LoI4aCAFgrIzNsA1mVAEAAGgVM6oAAAC0iqAKAABAqwiqAAAAtIqgCgAAQKsIqgAAALSKoAoAAECrCKoAAAC0yv8PkdKP7dE3HbIAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "x = np.arange(0.01, 1, 0.01)\n", "y_1 = -np.log(x)\n", "y_2 = -np.log(1-x)\n", "\n", "fig, ax = plt.subplots(1, 2, figsize=(16, 10))\n", "ax[0].plot(x, y_1, color='red', marker='s');\n", "ax[0].set_xlabel(r'$H_{\\theta}(x)$')\n", "ax[0].set_ylabel('Cost')\n", "ax[0].set_title('$y=1$')\n", "ax[1].plot(x, y_2, color='blue', marker='o');\n", "ax[1].set_xlabel(r'$H_{\\theta}(x)$')\n", "ax[1].set_ylabel('Cost')\n", "ax[1].set_title('$y=0$')\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optimization\n", "\n", "We defined the cost function with Cross-entropy, we need to minimize it to make the model correctly classified, and Gradient Descent can apply this. As you noticed before, it requires to calculate the gradient.\n", "\n", "Actually, when you try implement it with tensorflow, you don't need to calculate the gradient manually. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Logistic Regression in Tensorflow\n", "Let's build the Logistic Regression model with tensorflow. Before beginning, let's analyze the sample data patterns.\n", "\n", "We made a sample data like this," ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA70AAAJTCAYAAADE5HHqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de4zl533f98+XWtG6LKwhqXiDWrHkYWEjjQA3s0OgQOt0txxGcP8rsKvE1T8GCu4YAZoUiLtbwkVdFAjcXQm9qEBQrp3+YVftkkvXSYvm0h064wBGg2J3kxRKobbhyIJVJbJEaiQNlxeR+/SP85vl4dmZ2SW5cy7PvF7AYOf8fr8z8yzwcJbveX6Xaq0FAAAAevTQrAcAAAAAh0X0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANCtY7MewDR88pOfbJ/5zGdmPYx7evXVV/Pxj3981sPgiDMPmQfmIfPAPGQemIfMg0WYhzdu3Phua+1P7LXvSETvZz7zmVy/fn3Ww7inzc3NnDp1atbD4IgzD5kH5iHzwDxkHpiHzINFmIdV9Y399jm9GQAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADo1rFpfJOqWknyaJKlJMtJ0lq7tM+xS0nOJdkajt1ord281z4AAACYNJXoTfJikidbaxtJUlWtqvYL1qtJ1ltrW8Ox16rqbGtt+x77gA/ov/n3fzM/+2//1KyHATBzt1/97aR9YtbD4Ii7/drfTW6/OuthwMKb1unNT06s1ibJXaE67FvejdrBVpK1g/Yd0pjhSPnqH3wt//Nf/3v54z96edZDAZipdns7+eEXk7e/ldbenPVwOKJaezP5wa8lt7+ddvt7sx4OLLSpRO/Eiu7nk7wwEa+7VnN3DG8neeoe+4AP6PL5305a8vqrr+f/vbnXf54AR0N79TeTtCS3025dnfVwOKLard9J8sbo853Lsx0MLLip3ciqqpar6lySp1prZ/c5bCnJKxPbXs471wPvtw/4AL76B1/L1j/5RpKk3W6jAAY4gtrt7eTWb2cUG7eTnf/aai9T19qbyc5/lbRbSW4nt75itRc+gGqtTfcbHhC+VXUmyTOttZNj284neSLJc/vt2+drncvoplc5ceLEyStXrjzwv8uDtrOzk+PHj896GBxBf/R/fyuv77yeJHnkU5/I9rd+kD/1sz+ZH/vYwzMeGUeVn4fMzO1vJ7dfTnI7O6+dyPGPfid56E8mD/kdO1N0+5Xk9r/Iu+fho6O5CDOwCP8unz59+kZrbXWvfdO6kdUdrbXLVXWxqs7vcQfn7YxWdMc9ltEK70H79vw+SS4nyerqajt16tQHHfqh29zczCKMk7589Q++li/91d/MG7dGp1B9/ou/kKvn/25+7vSfyRc3fm3Go+Oo8vOQWWi3t9O+828m7bUkyT/46l/On/vsl5P6ROon/iBVfhHI4WvtzbQ//vmkjVZ278zDfCT1E7+feuiR2Q6QI2nR/10+9NObq2qlql6a2LyV5PE9Dr+eu09XXkpy7R77gPfphf/if8nbb72dj3/iY/n4Jz6Whz70UD5y/Mfyj3/vq/n2N74z6+EBTM9rfytpbyR1fPSRD43+bD9IXt+Y9eg4Kt74+0nbvnse5kdpt3531qODhTSNld7tJJP/Uiwn+fVkdK1vkrTWtlpr21V1varG79K8muTCQfum8HeAbv2Vv/50vvPNd+7Y/PVv/7N86ff+0xz78LH8xE99coYjA5iyj55JPXzyndcf+mbq0d8afX7sZ2YzJo6eHzudeux33nk9Pg8/9JmZDAkW3aFHb2ttq6quDtfYJsnJjCL2heH1ekYrtuvD67NJzlXVVkYru0+PPYf3oH3A+/DIiaU8cuKdKwe+tflH+ZmTe52IAdC3eujjyUOfHdvw3dSHP7v/G+AQVD2cfNg8hAdpKtf0ttb2PSeotXZh4vV2kslrfe+5DwAAACZN7ZFFAAAAMG2iFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALp1bBrfpKpWkqwNL59I8mxrbWOfY68mudBa29pj3/kkjyV5LsmjSc621tYPZ9QAAAAsuqlEb5K11tqlJKmqpSRfr6onW2s39zj2TJIzVTW+bau19vjw+bnhYyPJ04c4ZgAAABbcoZ/ePKzyPrP7urW2neR63ln5HT92KaPV29r9SPJUkrPDIduttUeGj7PD1wIAAIA9HXr0Dqu5Zyc2Lye5K1hba9uttRd2Xw8RvDS5IlxVK1W1fBjjBQAAoB/VWpvuNxzF6o0kP32vldqquthauzD2+lySVzI6tXktyRPj+yfeu3sadE6cOHHyypUrD+hvcHh2dnZy/PjxWQ+DI848ZB6Yh8wD85B5YB4yDxZhHp4+ffpGa211r32ziN5rGd2oaq/recePO5Mk4yu/exzzUpL1/W6KtWt1dbVdv379/Qx3qjY3N3Pq1KlZD4MjzjxkHpiHzAPzkHlgHjIPFmEeVtW+0TvVRxYNd1++eK/gHTyT0Yru+PtXJo65mdE1vwAAAHCXqUXvsHK7sbsqe9A1ucO1vCvjpz8PwfvixKFLSV46hOECAADQgalEb1WtZXTn5ZvD66UkK8Pny3sE8GombnQ1vHfy+t3lJM8fyqABAABYeIf+nN4haK8Nn4/vOjn8uZ7Riu36xFv3ugj3+nCK9HaSxzN6vJHHFgEAALCnQ4/e1tpWkjpg/113Xx5Ogb7r5lTDau/9XA8MAAAA072RFQAAAEyT6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbx6bxTapqJcna8PKJJM+21jb2OfZ8kseSPJfk0SRnW2vrw76lJOeSbCVZTrLRWrt5yMMHAABgQU0lepOstdYuJXfC9etV9eQBwXpu+NhI8vTY9qtJ1ltrW8PXulZVZ1tr24c4dgAAABbUoZ/ePKzyPrP7egjU63ln5XfSdmvtkeHjTtAOsby8G7yDrQO+DgAAAEfcoUfvsJp7dmLzcpIDV2eraqWqlsc2re7xnu0kT33gQQIAANClqdzIavz63SFkH03y/H7HV9WZjFZxV6rq4rB5KckrE4e+PHwtAAAAuEu11qb7DauuJblwvzegqqqXkqxnFL3PtNZOju07n+SJ1trkSnKqave64Jw4ceLklStXHsTwD9XOzk6OHz8+62FwxJmHzAPzkHlgHjIPzEPmwSLMw9OnT99ora3utW9aN7JKcidSLx4UvFW1MrH/ZkanMF/LKHzHPZa7V3+TJK21y0kuJ8nq6mo7derUBxj5dGxubmYRxknfzEPmgXnIPDAPmQfmIfNg0efh1J7TO5yyvLF7qvPE9bq7x6wkeXFi81KSlzK6+dXkqcxLGcUwAAAA3GUq0VtVaxndlfnm8Hopycrw+fJuAA/7L0y8fTnJ87t3fd7j5lZ7Pu8XAAAADv305iFSrw2fj+/avTZ393rd9eH19eE06O0kjycZfw7v2STnqmoro1Xfpz2jFwAAgP0cevQOz9WtA/ZfmHh9M6PrePc6djvJpQc6QAAAALo1tWt6AQAAYNpELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHTr2DS+SVWtJFkbXj6R5NnW2sZ7Pbaqzid5LMlzSR5Ncra1tn6YYwcAAGBxTSV6k6y11i4lSVUtJfl6VT3ZWrv5Po49N3xsJHl6CmMHAABgQR366c3Dyu0zu69ba9tJrued1dz3cux2a+2R4ePssB8AAAD2dOjRO6zQnp3YvJzkrmC932OraqWqlh/kOAEAAOhPtdam+w1HsXojyU/fa6V28tiqOpfklYxObV5L8kRr7cI+7909DTonTpw4eeXKlQf4tzgcOzs7OX78+KyHwRFnHjIPzEPmgXnIPDAPmQeLMA9Pnz59o7W2ute+WUTvtSQX9rme9z0dW1UvJVnf76ZYu1ZXV9v169ff13inaXNzM6dOnZr1MDjizEPmgXnIPDAPmQfmIfNgEeZhVe0bvVN9ZNFw9+WL9xm8dx07XPM77maSpx7sKAEAAOjFtO7enKo6k2RjN2Krarm1tnW/xyZZSvJikkfGDl1K8tKhDhwAAICFNZWV3qpay+jOy7sRu5RkZfh8efymVPsdO7yevH53OcnzU/grAAAAsIAOfaV3CNprw+fju04Of65ntGK7fh/HXh9Oe95O8ngSjy0CAABgX4cevcMpzHXA/gvv4dibGV3HCwAAAPc01RtZAQAAwDSJXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFv3Fb1V9eNV9SvDx7+6x/5ff/BDAwAAgA/mntFbVX82yXaSv5jkzyd5sar+vYnDzh/C2AAAAOADOXYfx1xOst5a+40kqaqlJM9XVVprf2M4pg76AlW1kmRtePlEkmdbaxv7HLuU5FySrSTLSTZaazfvtW/R3b59e9ZDAAAA6M79nN68vBu8SdJa226t/fkkn6uqf2d38z2+xlpr7VJr7VKSp5NcHUJ4L1eTvNBae2E4/uIQu/fat7Bu/fC1/Ls/9ct5/dU3Zj0UAADmwVe+knzmM8mNG6M/v/KVWY+II+r2q/9D8vZWWrtX8s2v+4ner1fVpyc3ttY+n+QXq+rfOujNQ9w+M/a+7STX887K7/ixSxlF9tbY5q0kawftu4+/w1z73S//7Xzv29/Pd7/58qyHAgDArH3lK8m5c8k3vjF6/Y1vjF4LX6astTeSnf8yaa8nb/7vsx7O+3Y/0Xsuye9U1a9M7hjC95cPevNw+vHZic3LGV0nPGl1j+3bSZ66x76FdeuHr+W5i38zt9++nddvvZH/6x/+P7MeEgAAs/Srv5rcuvXubbdujbbDFLVbzyXtR0lup/3w4sKu9tb9Dryq/mxr7R/ts+/J1tqL9/l1lpPcSPLTw6rv+L4zGV0//NTYtvMZXQf83H77WmuTUZ2qOpdRsOfEiRMnr1y5cj/Dm7pX/sV2Xvnn30u73fLIpz6R17ffyKd+9l+a9bA4wnZ2dnL8+PFZD4MjzjxkHpiHzMyNG3c+3fnUp3L8m998Z9/JkzMYEEdTS976WpK3s/PaiRz/6HeSD306qY/PemB7On369I3W2upe++7nRlZJktbaP6qqH2+t/WCv3e9hPM8meXIyeMc8esB7D9r37gG1djmjm3BldXW1nTp16r4HOC23fvha/uJPnstrO68nST7/xV/I3/q1F3Np49fyr/xrPzPj0XFUbW5uZh7/e+FoMQ+ZB+YhM/NLv3Tn1ObNL30pp35lOOHy059O/vAPZzYsjpbbr/5W8sNnk9zKP/jqX86f++yXk2N/OvXY30zVgfcxnjv39ZzeMX9YVafHN1TVf5hRyN7TsDJ78YA7Lm8nmbwx1WNJXrnHvoX0u1/+23n7rbffte3N197Mb5z/7RmNCACAmftrfy352Mfeve1jHxtthykYXcv75SQTp9m/9fWFvLb3vUbvWpLfqKq/miRV9XxG19ruuYw8bjh1eWP3UUXDac6Trufu1dylJNfusW8h/ZPNf5pU5eGPPpyHP/pw6qHKhz/ycP7oa/9f3njNnZwBAI6kL3whuXx5tLKbjP68fHm0HabhrX+W5O0kHxk+Hhr+TNqb/3B243qf7vv05uTOTan+5aq6XlWXklxorX3pXu+rqrUk2xPP211JsrUbv621rdba9vC1x+/SvDp8n333vZe/wzy5dO0/edfrzc3N/K+vuisfAMCR94UvjD42N53SzNTVh/9M6sTY7Zy+tpmH/uT/ObsBfUDvKXqTZLiL81JGd22+UFU3Wmt//4DjlzOsxk6c+717Ff768PXWh9dnk5yrqq2MVnafHrv+96B9AAAA8C7vKXqr6n/L6KZVJ1tr36+qF5M8X1X/R2vtL+31nmFVdt8rnVtrFyZebye5tM+x++4DAACASe/1mt6brbXPtda+n9w5JXn1fXwdAAAAOHTvKVZba//RPtt/+cEMBwAAAB4cK7QAAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0ayrRW1XLVXW1qtbucdzVqlreZ9/5qrpYVStVtVZVzx7OaAEAAOjFoUfvELrLw8e9nEnyUlW1sY+XxvafS/JikvUkFx78aAEAAOjJscP+Bq21jSSpqlcOOq6qlpKcba29MLZtLcnu+7Zba48c2kABAADoztxc09ta254I3qUkS621m+PHDac338+qMQAAAEdctdam842qriW5uLvyex/HX2ytXRh7fS6jVd+NJGtJnhjfv8f7z2V0OnROnDhx8sqVKx9k+FOxs7OT48ePz3oYHHHmIfPAPGQemIfMA/OQebAI8/D06dM3Wmure+2by+itqjNJMr7yu8cxLyVZv5+vt7q62q5fv/5ehjsTm5ubOXXq1KyHwRFnHjIPzEPmgXnIPDAPmQeLMA+rat/onZvTmyc8k9GK7h1VtTJxzM0kT01tRAAAACycuYve4Vreldba9ti2lYzu2jxuKclLAQAAgH3MNHqH5/dO3pRqNcn2+IbhZlaT1+8uJ3n+EIcHAADAgjv0RxYNq7RrGcXshapabq1dHnavZ7Riuz7xtr0uwL1eVeczCuLHM3q80fYexwEAAECS6Tyn92ZG199e2mPfXXdfHm5MddfNqca+DgAAANyXubumFwAAAB4U0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3phK9VbVcVVerau0ex52vqotVtVJVa1X17Ni+pWH/meHPlcMfOQAAAIvs2GF/g7HQXb7Pt5wbPjaSPD22/WqS9dba1vB1r1XV2dba9gMbLAAAAF059OhtrW0kSVW9ch+Hb7fWHpncWFVLSZZ3g3ewlWQtyQsPZKAAAAB0Zy6v6R1Obx5fGV5NMrmiu53kqemNCgAAgEVz6Cu971VVncno1Oa1qlpvrV1IspRkcqX45RxwynRV7Z4mnRMnTmRzc/NwBvwA7ezsLMQ46Zt5yDwwD5kH5iHzwDxkHiz6PJyr6G2tXR57+cJwU6trw+tH38fXupwkq6ur7dSpUw9mkIdoc3MzizBO+mYeMg/MQ+aBecg8MA+ZB4s+D+fq9OY97sh8M6NTmLczWu0d91juXv0FAACAO+YmeofgfXFi81KSl5Jcz90rvUtJrgUAAAD2MdPoHZ7fu5wkrbWbSS5MHLKc5PnhsUTX97i51cZ0RgoAAMAimsZzelcyerTQapILVbU8du3uekYrtuvD6+tVdT6j05kfTzL+HN6zSc5V1VZGq75Pe0YvAAAAB5nGc3pvZnRt7qU99l3Y59i9vs72Xl8DAAAA9jM31/QCAADAgyZ6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6NaxaXyTqlpOcjHJs621jQOOW0myNrx8Yvz4qjqf5LEkzyV5NMnZ1tr6oQ4cAACAhXbo0VtVuxG7fB+Hr7XWLg3vW0ry9ap6srV2c9h/bvjYSPL0Ax8sAAAAXTn06B1bqX3loOOGVd5nklwa3rddVdczWvm9mWS7tfbIIQ8XAACAjszNNb3Dau7Zic3LSbbHN1TVynC6NAAAABxobqI3eWdVOLlzHfCjSZ4f23YmyVaSlaq6OP0RAgAAsEiqtTadb1R1LcnFg25ktcfxF8au553c/1KS9f2+XlXtXv+bEydOnLxy5cr7G/gU7ezs5Pjx47MeBkececg8MA+ZB+Yh88A8ZB4swjw8ffr0jdba6l77pnL35vdquFPzxfHgraqViQC+meSpjG5qdZfW2uUkl5NkdXW1nTp16vAG/IBsbm5mEcZJ38xD5oF5yDwwD5kH5iHzYNHn4dxF73AK88Zu4A6nOS8leTHJ+I2slpK8NP0RAgAAsChmek1vVS2P35RqeLzR9ljwLiXZXeG9MPH25Yxd7wsAAACTpvGc3pWMHju0muRCVS0Ppx4nyXpGK7brQ/xeG94z/iVODn9eH0573k7yeJKzrbV33dkZAAAAxk3jOb03M7r+9tIe+y6Mfb6VpCaP2ePrAAAAwH2Zq0cWAQAAwIMkegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAALolegEAAOiW6AUAAKBbohcAAIBuiV4AAAC6JXoBAADolugFAACgW6IXAACAboleAAAAuiV6AQAA6JboBQAAoFuiFwAAgG6JXgAAAIKGXoQAAAeYSURBVLolegEAAOiW6AUAAKBbx6bxTapqOcnFJM+21jYOOG4pybkkW0mWk2y01m7eax8A/fjnW99Ou91mPQwAoBOHHr1VtTZ8unwfh19Nst5a2xree62qzrbWtu+xD4AOvPWjt/JX/o3/OGf+88/NeigAQCcO/fTm1trGsLr7ykHHDSu5y7tRO9hKsnbQvgc+YABm5tpv/X5uff9WfvDyD/Pdbx34zwYAwH2Zp2t6V5NMrtpuJ3nqHvsA6MBbP3or/92v/o9547U301ry3/9nV2c9JACgA1O5pvc+LeXu1eCXMzot+qB9e6qqcxldA5wTJ05kc3PzgQ30sOzs7CzEOOmbecis/OC7P8znLvzrabdbHvnJH0/VD7Jx7cUc+/CHZj00jig/D5kH5iHzYNHn4TxFb5I8+j733aW1djnJ5SRZXV1tp06d+gDDmo7Nzc0swjjpm3nILLz1o7fyi3/ql7P9x99Pknz+i7+Q/+lXr+Vzv3Qq/8F/uz7j0XFU+XnIPDAPmQeLPg/n6fTm7YxWdMc9ltEK70H7AFhw137r97PzvZ13bXvrzbfyd/7G77m2FwD4QOZppfd67l7NXUpy7R77AFhwD3/k4fzcqc/eef2xH/9oTj71c/nQsYfyxq03ZjgyAGDRzTR6h+f3prW21VrbrqrrVTV+l+bVJBcO2jeLcQPwYD35hZ/Pk1/4+TuvNzc384W/94szHBEA0ItpPKd3JaNHC60muTCE6+Vh93pGK7a7F2ydTXKuqrYyWtl9euw5vAftAwAAgLscevS21m4muZnk0h77Lky83t7ruHvtAwAAgL3M042sAAAA4IESvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB0S/QCAADQrWqtzXoMh66qvpPkG7Mex334ZJLvznoQHHnmIfPAPGQemIfMA/OQebAI8/DTrbU/sdeOIxG9i6KqrrfWVmc9Do4285B5YB4yD8xD5oF5yDxY9Hno9GYAAAC6JXoBAADoluidL5dnPQCIech8MA+ZB+Yh88A8ZB4s9Dx0TS8AAADdstILAABAt0QvAAAA3To26wEcdVW1nORikmdbaxuzHg9HU1WtJFkbXj4R85EZGObho0mWkiwnSWvt0kwHxZFWVWtJllprL8x6LBwtVXU+yWNJnsvo5+LZ1tr6bEfFUTX2/4lbSR5trS3c9b2id4aGf0yT4X/uYIbWduOiqpaSfL2qnmyt3ZzxuDhaXkzy5O4vXKqqVdWGecgsDD8Ln83oF9MwC+eGj40kT894LBxRQ/A+01o7O7y+MTyzd6H+bRa9MzT2P3avzHosHF27P8ySXEqS1tp2VV3P6Dd6C/UDjYV35xctQ3AkyfYMx8PR9vmMYgNmYbu19sisBwFJfiPJ2bHXT7bWFu7fZtf0whE3RMbZic3LERtM2cRvjT+f5IXW2tasxsPRNZyJJXiZuapaGS6Fg6kbfgG93Frb2p2Lixi8iegF8s5ZB8md68wfTfL87EbEUVVVy1V1LslTu6dSwTQN/5O35BcuzFpVncnoGsqVqnKaPbOwmuSVsbm4XFXPznhM74voBSY9mwU9dYXF11rbGm6Qca2qrs56PBxJa25cxay11i631l5orW0P8/HM2L1gYFp2byy5MczFjYzC98yMx/WeiV7gjuFukRcX7eYE9GcI37VhTsJUDPc48POPmRvm4ribSZ6axVg40rYzur58fCFkKws4F93ICkhy5zSqjbEbCS07vY9pGf4H72pr7fGxzVtJHt/nLXAYHk2yWlW7r9eSPFpVu7+IgUM3/Dx8Mcn4jayWkrw0mxFxhF3fZ/vCnQ0oeoHdm7ZsT9w5dyWj6IBp2M7dNw5aTvLrMxgLR9Tk88mr6qkk1wQv09Rau1lVFyY2L8e9Npiy4YkeGxMLIatZwEdoVWtt1mM4ssYe9PxMRr9JueofVqZtuHHVXr89Puk0Z6Zp+OXL7l1KTya54WciszLcUO1iRv8+P+s6X6Zp7P8RtzM64+U5/yYzC8NCyDNJXk7yWEa/CFy4u9uLXgAAALrlRlYAAAB0S/QCAADQLdELAABAt0QvAAAA3RK9AAAAdEv0AgAA0C3RCwAdq6rlqnp2eO4sABw5x2Y9AADgcFTVs0mWk6wmuTHj4QDATIheAOhUa209SapK8AJwZDm9GQAWVFVdrKprY69Xqup7VbU0y3EBwDwRvQCwoFprF5Kkqs4Pm64mOdta257dqABgvji9GQAW29kkN6rqLyR5trW2MesBAcA8Eb0AsMBaa9tVdTGj4D056/EAwLxxejMALLCqWk5yIcmFqro66/EAwLwRvQCw2K4mWW+tXUredX0vAJCkWmuzHgMA8D4Mz+EdfzTRUkbP411vrW0Mpz2fyehZvdtJXknyVGtta0ZDBoCpE70AAAB0y+nNAAAAdEv0AgAA0C3RCwAAQLdELwAAAN0SvQAAAHRL9AIAANAt0QsAAEC3RC8AAADdEr0AAAB06/8HUvXdVUcI0qYAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Training data\n", "x_train = [[1., 2.], [2., 3.], [3., 1.], [4., 3.], [5., 3.], [6., 2.]]\n", "y_train = [[0.], [0.], [0.], [1.], [1.], [1.]]\n", "\n", "# Test data\n", "x_test = [[5., 2.]]\n", "y_test = [[1.]]\n", "\n", "x1 = [x[0] for x in x_train]\n", "x2 = [x[1] for x in x_train]\n", "\n", "# Visualize it\n", "colors = [int(y[0] % 3) for y in y_train]\n", "plt.scatter(x1, x2, c=colors, marker='^')\n", "plt.scatter(x_test[0][0], x_test[0][1], c='red')\n", "\n", "plt.grid()\n", "plt.xlabel('x1')\n", "plt.ylabel('x2')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Actually, we defined the dataset with python list. But to handle it with tensorflow, it need to convert tensor type. Let's define the whole process." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n", "\n", "# Initialize Weight and bias\n", "W = tf.Variable(tf.zeros([2, 1]), name='weight')\n", "b = tf.Variable(tf.zeros([1]), name='bias')\n", "\n", "# Sigmoid Function\n", "def sigmoid(X):\n", " h = tf.divide(1., 1. + tf.exp(-(tf.matmul(X, W) + b)))\n", " return h\n", "\n", "# Loss function \n", "def loss_fn(h, y):\n", " cost = -tf.reduce_mean(y * tf.math.log(h) + (1 - y) * tf.math.log(1 - h))\n", " return cost\n", "\n", "# Optimizer (Stochastic Gradient Descent)\n", "optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)\n", "\n", "# Accuracy function for decision boundary\n", "def accuracy_fn(h, y):\n", " y_hat = tf.cast(h > 0.5, dtype=tf.float32)\n", " accuracy = tf.reduce_mean(tf.cast(tf.equal(y_hat, y), dtype=tf.int32))\n", " return accuracy\n", "\n", "# Gradient function\n", "def grad(X, y):\n", " with tf.GradientTape() as tape:\n", " h = sigmoid(X)\n", " loss = loss_fn(h, y)\n", " return tape.gradient(loss, [W, b])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With this function, we will build the training step," ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch: 0, Loss: 0.6931\n", "Epoch: 100, Loss: 0.5781\n", "Epoch: 200, Loss: 0.5352\n", "Epoch: 300, Loss: 0.5056\n", "Epoch: 400, Loss: 0.4840\n", "Epoch: 500, Loss: 0.4673\n", "Epoch: 600, Loss: 0.4537\n", "Epoch: 700, Loss: 0.4421\n", "Epoch: 800, Loss: 0.4320\n", "Epoch: 900, Loss: 0.4229\n" ] } ], "source": [ "for e in range(1000):\n", " for x, y in iter(dataset.batch(len(x_train))):\n", " h = sigmoid(x)\n", " grads = grad(x, y)\n", " optimizer.apply_gradients(grads_and_vars=zip(grads, [W, b]))\n", " \n", " if e % 100 == 0:\n", " print('Epoch: {}, Loss: {:.4f}'.format(e, loss_fn(h, y)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see the cost is decreasing in each epoch. Weight $W$ and bias $b$ is stored in memory, so we can test it with test dataset." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test Result = [[1]]\n", "Test Accuracy: 1.0000\n" ] } ], "source": [ "test_accuracy = accuracy_fn(sigmoid(x_test), y_test)\n", "print('Test Result = {}'.format(tf.cast(sigmoid(x_test) > 0.5, dtype=tf.int32)))\n", "print('Test Accuracy: {:.4f}'.format(test_accuracy))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the model correctly classified test set based on trained weight and bias." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Logistic Regression with diabetes classification" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's look at more complicated dataset. Diabetes dataset, released from [UCI ML repository](https://archive.ics.uci.edu/ml/datasets/diabetes), is common dataset for classification. Let's apply the same approach in this dataset." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pregnanciesglucosediastolictricepsinsulinbmidpfagediabetes
061487235033.60.627501
11856629026.60.351310
28183640023.30.672321
318966239428.10.167210
40137403516843.12.288331
\n", "
" ], "text/plain": [ " pregnancies glucose diastolic triceps insulin bmi dpf age \\\n", "0 6 148 72 35 0 33.6 0.627 50 \n", "1 1 85 66 29 0 26.6 0.351 31 \n", "2 8 183 64 0 0 23.3 0.672 32 \n", "3 1 89 66 23 94 28.1 0.167 21 \n", "4 0 137 40 35 168 43.1 2.288 33 \n", "\n", " diabetes \n", "0 1 \n", "1 0 \n", "2 1 \n", "3 0 \n", "4 1 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_csv('./dataset/diabetes.csv')\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the scale is different from each variable. So we need to standardizing it." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(460, 8) (460, 1)\n", "[[-5.4791862e-01 -1.1859907e+00 -2.1224350e-01 ... 6.1015427e-01\n", " 4.7453222e-01 -7.8628618e-01]\n", " [ 4.6014335e-02 6.5895163e-02 5.6322277e-01 ... 9.4197702e-04\n", " -8.7209895e-02 6.4591348e-02]\n", " [ 4.6014335e-02 -3.4096771e-01 -1.6054575e-01 ... -1.1749996e-02\n", " -2.6465813e-03 -3.6084738e-01]\n", " ...\n", " [ 1.2338802e+00 2.1002097e+00 4.5982724e-01 ... 2.0189581e+00\n", " -1.0113661e+00 8.3038110e-01]\n", " [ 1.2338802e+00 -1.1546935e+00 2.5303626e-01 ... 8.0053312e-01\n", " -4.4928238e-02 4.9003014e-01]\n", " [-5.4791862e-01 -6.8523633e-01 4.6245251e-02 ... -1.4713213e+00\n", " -7.1539456e-01 -5.3102291e-01]]\n" ] } ], "source": [ "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split\n", "\n", "X = df.iloc[:, :-1].to_numpy()\n", "y = df.iloc[:, [-1]].to_numpy()\n", "\n", "X = X.astype(np.float32)\n", "y = y.astype(np.float32)\n", "\n", "# Standardize the data\n", "ss = StandardScaler()\n", "X = ss.fit_transform(X)\n", "\n", "# Split with training and test dataset\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4)\n", "\n", "print(X_train.shape, y_train.shape)\n", "print(X_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's try!" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train)).batch(len(X_train))" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "W = tf.Variable(tf.random.normal((8, 1)), name='weight')\n", "b = tf.Variable(tf.random.normal((1,)), name='bias')" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch: 0, Loss: 0.9894\n", "Epoch: 100, Loss: 0.8589\n", "Epoch: 200, Loss: 0.7700\n", "Epoch: 300, Loss: 0.7060\n", "Epoch: 400, Loss: 0.6584\n", "Epoch: 500, Loss: 0.6217\n", "Epoch: 600, Loss: 0.5928\n", "Epoch: 700, Loss: 0.5695\n", "Epoch: 800, Loss: 0.5504\n", "Epoch: 900, Loss: 0.5344\n", "Epoch: 1000, Loss: 0.5210\n", "Epoch: 1100, Loss: 0.5095\n", "Epoch: 1200, Loss: 0.4996\n", "Epoch: 1300, Loss: 0.4910\n", "Epoch: 1400, Loss: 0.4836\n", "Epoch: 1500, Loss: 0.4771\n", "Epoch: 1600, Loss: 0.4714\n", "Epoch: 1700, Loss: 0.4664\n", "Epoch: 1800, Loss: 0.4620\n", "Epoch: 1900, Loss: 0.4582\n" ] } ], "source": [ "for e in range(2000):\n", " for x, y in iter(dataset.batch(len(X_train))):\n", " h = sigmoid(x)\n", " grads = grad(x, y)\n", " optimizer.apply_gradients(grads_and_vars=zip(grads, [W, b]))\n", " \n", " if e % 100 == 0:\n", " print('Epoch: {}, Loss: {:.4f}'.format(e, loss_fn(h, y)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After training, we can validate model performance with test dataset. In this time, we want to measure whether model is correctly classified or not, so it can write the test_accuracy manually." ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test Accuracy: 0.7468\n" ] } ], "source": [ "y_hat = tf.cast(sigmoid(X_test) > 0.5, dtype=tf.int32)\n", "# print('Test Result = {}'.format(tf.cast(sigmoid(X_test) > 0.5, dtype=tf.int32)))\n", "print('Test Accuracy: {:.4f}'.format(np.sum(y_test.reshape(-1) == y_hat.numpy().reshape(-1)) / y_test.shape[0]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "In this post, we cover the basic definition of logistic regression. Logistic regression is the approach to handle the classification task. So its hypothesis and cost function are different from that in linear regression. For cost function, Cross-Entropy is introduced, and we can implement whole process with tensorflow 2.x." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }