{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Sveučilište u Zagrebu
\n",
"Fakultet elektrotehnike i računarstva\n",
"\n",
"# Strojno učenje\n",
"\n",
"http://www.fer.unizg.hr/predmet/su\n",
"\n",
"Ak. god. 2015./2016.\n",
"\n",
"# Bilježnica 7: Logistička regresija\n",
"\n",
"(c) 2015 Jan Šnajder\n",
"\n",
"Verzija: 0.2 (2015-11-16)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Populating the interactive namespace from numpy and matplotlib\n"
]
}
],
"source": [
"import scipy as sp\n",
"import scipy.stats as stats\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"%pylab inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Sadržaj:\n",
"\n",
"* Model logističke regresije\n",
"\n",
"* Gubitak unakrsne entropije\n",
"\n",
"* Minimizacija pogreške\n",
"\n",
"* Poveznica s generativnim modelom\n",
"\n",
"* Usporedba linearnih modela\n",
"\n",
"* Sažetak\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Model logističke regresije\n",
"\n",
"### Podsjetnik: poopćeni linearni modeli\n",
"\n",
"$$\n",
"h(\\mathbf{x}) = \\color{red}{f\\big(}\\mathbf{w}^\\intercal\\tilde{\\mathbf{x}}\\color{red}{\\big)}\n",
"$$\n",
"\n",
"\n",
"* $f : \\mathbb{R}\\to[0,1]$ ili $f : \\mathbb{R}\\to[-1,+1]$ je **aktivacijska funkcija**\n",
"\n",
"\n",
"* Linearna granica u ulaznom prostoru (premda je $f$ nelinearna)\n",
" * Međutim, ako preslikavamo s $\\boldsymbol\\phi(\\mathbf{x})$ u prostor značajki, granica u ulaznom prostoru može biti nelinearna\n",
"\n",
"\n",
"* Model nelinearan u parametrima (jer je $f$ nelinearna)\n",
" * Komplicira optimizaciju (nema rješenja u zatvorenoj formi)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Podsjetnik: klasifikacija regresijom\n",
"\n",
"* Model:\n",
"$$\n",
" h(\\mathbf{x}) = \\mathbf{w}^\\intercal\\boldsymbol{\\phi}(\\mathbf{x}) \\qquad (f(\\alpha)=\\alpha)\n",
"$$\n",
"\n",
"\n",
"* [Skica]\n",
"\n",
"\n",
"* Funkcija gubitka: kvadratni gubitak\n",
"\n",
"\n",
"* Optimizacijski postupak: izračun pseudoinverza (rješenje u zatvorenoj formi)\n",
"\n",
"\n",
"* Prednosti:\n",
" * Uvijek dobivamo rješenje\n",
" \n",
"\n",
"* Nedostatci:\n",
" * Nerobusnost: ispravno klasificirani primjeri utječu na granicu $\\Rightarrow$ pogrešna klasifikacija čak i kod linearno odvojivih problema\n",
" * Izlaz modela nije probabilistički\n",
" \n",
"### Podsjetnik: perceptron\n",
"\n",
"* Model:\n",
"$$\n",
"h(\\mathbf{x}) = f\\big(\\mathbf{w}^\\intercal\\boldsymbol\\phi(\\mathbf{x})\\big)\n",
"\\qquad f(\\alpha) = \\begin{cases}\n",
"+1 & \\text{ako $\\alpha\\geq0$}\\\\\n",
"-1 & \\text{inače}\n",
"\\end{cases}\n",
"$$\n",
"\n",
"\n",
"* [Skica]\n",
"\n",
"\n",
"* Funkcija gubitka: količina pogrešne klasifikacije \n",
"$$\n",
"\\mathrm{max}(0,-\\tilde{\\mathbf{w}}^\\intercal\\boldsymbol{\\phi}(\\mathbf{x})y)\n",
"$$\n",
"\n",
"\n",
"* Optimizacijski postupak: gradijentni spust\n",
"\n",
"\n",
"* Prednosti:\n",
" * Ispravno klasificirani primjeri ne utječu na granicu
\n",
" $\\Rightarrow$ ispravna klasifikacija linearno odvojivih problema\n",
"\n",
"\n",
"* Nedostatci:\n",
" * Aktivacijska funkcija nije derivabilna
\n",
" $\\Rightarrow$ funkcija gubitka nije derivabilna
\n",
" $\\Rightarrow$ gradijent funkcije pogreške nije nula u točki minimuma
\n",
" $\\Rightarrow$ postupak ne konvergira ako primjeri nisu linearno odvojivi\n",
" * Decizijska granica ovisi o početnom izboru težina\n",
" * Izlaz modela nije probabilistički"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Logistička regresija\n",
"\n",
"* Ideja: upotrijebiti aktivacijsku funkciju s izlazima $[0,1]$ ali koja jest derivabilna\n",
"\n",
"\n",
"* **Logistička (sigmoidalna) funkcija**:\n",
"$$\n",
" \\sigma(\\alpha) = \\frac{1}{1 + \\exp(-\\alpha)}\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEACAYAAABMEua6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGHRJREFUeJzt3XuUVOWZ7/HvExBijAwYDPcTby1I1KBHgVnOaCkaG6Og\nxghosmKOQ5gxuGatrEmMoyvpOdGoo+eMEZSgISaZg2JEGhvFWzQVL6NE4gUUGprxdOQmGpXjZcKl\nu5/zx1tK2TRdVd3V/dbe9fustVfVrtpdPLa1frw8+93vNndHRETS4VOxCxARkfJRqIuIpIhCXUQk\nRRTqIiIpolAXEUkRhbqISIoUDHUz+4WZbTOz1Z0cc4uZNZnZy2Z2XHlLFBGRYhUzUr8TqN3Xm2Z2\nFnCEu9cA3wbmlak2EREpUcFQd/engHc7OWQK8KvcsSuAgWY2pDzliYhIKcrRUx8BbMzb3wSMLMPn\niohIicp1otTa7WvtARGRCPqW4TM2A6Py9kfmXvsEM1PQi4h0gbu3HzjvUzlCvQGYDSwys4nAdnff\nto/CyvDHCUBdXR11dXWxy0gF/S475w5vvQXr10NT0ycfX38dduyAYcP2bBs31jF1ah2DB8OgQWEb\nOHDP87/6K+hbjuSpEmZF5zlQRKib2d3AKcBgM9sI/AjYD8Dd57v7cjM7y8w2AB8C3yq5ahGpGFu3\nwnPP7dlefhk+9SkYPRpqauDII+FrXwvPDzkkBHV+7tTVwVVXxapeCoa6u88o4pjZ5SlHRHpbYyM8\n9NCeEP/gA5g4MWxXXw3HHQeDB8euUoqlfwQlVCaTiV1CalTb79I9jL6XLIH77oPt2+Gcc+Dss+Ga\na+CIIz458i5Vtf0+K431Vp/bzFw9dZE43OH552Hx4hDmbW3w1a/C+efDhAmhvSKVycx6/USpiFSo\ntjZ44AG47rpwsnP6dLj3Xhg3rnujcalcCnWRFGppgd/8JoT5fvvBlVeGUXmfPrErk56mUBdJkR07\n4Fe/gn/9Vxg5Em68Ec48U6PyaqJQF0mJRx+Fv/s7OOaYEOx/8zexK5IYFOoiCffhh/D974fe+Z13\nwqRJsSuSmHTOWyTBnn02nPT84IMwTVGBLhqpiyTQrl3wL/8CCxbAbbeFk6AioFAXSZxXX4Wvfz2c\nCH3pJRg6NHZFUknUfhFJkGeegVNPhe98BxoaFOiyN43URRLiySfhggvg3/89TFMU6YhCXSQBnngC\npk2DRYt0MlQ6p/aLSIV79NFwef/ixQp0KUyhLlLBli8PJ0Xr6+GUU2JXI0mgUBepUA0NcMkl4fGk\nk2JXI0mhnrpIBWpogJkzw0j9hBNiVyNJovXURSrMa6+Fuw498ACMHx+7Gomt1PXU1X4RqSC7doWT\noldfrUCXrtFIXaSCfO97sG4d3H+/lsuVQHc+Ekmohx6Ce+6BF19UoEvXKdRFKsCWLfCtb4W7FX3u\nc7GrkSRTT10kstbWMBf9ssvg5JNjVyNJp1AXiey668INoq+6KnYlkgZqv4hE9NRTMHcu/PGPuim0\nlIdG6iKRvPNOaLssWAAjRsSuRtJCUxpFIpk1C/r1gzlzYlcilUxTGkUSYO1aWLIE1q+PXYmkjdov\nIhH84AdhGzQodiWSNhqpi/SyJ5+EVavCnHSRctNIXaQXucM//RNcey307x+7GkkjhbpIL7r33nCx\n0fTpsSuRtNLsF5FesnMnjB0Ld9wBp50WuxpJCi29K1KhfvYzGD1agS49SyN1kV6wfXsI9Mcfh6OP\njl2NJIlG6iIV6Prr4eyzFejS8wqO1M2sFrgZ6AP83N1vaPf+YOD/AEMJUyRvcvdfdvA5GqlLVXr9\ndRg3LkxjHDkydjWSNKWO1DsNdTPrA6wDTgc2A88DM9x9bd4xdUB/d78yF/DrgCHu3tLusxTqUpUu\nuSSs7XLttbErkSQq9zIB44EN7t6c+/BFwFRgbd4xW4Fjc88HAG+3D3SRavXqq+GORk1NsSuRalEo\n1EcAG/P2NwET2h1zB/CEmW0BDgQuLF95Isn205/C7NkwYEDsSqRaFAr1Yvol/wy85O4ZMzsceMzM\nvuTu77c/sK6u7uPnmUyGTCZTQqkiyfLOO+Fio8bG2JVIkmSzWbLZbJd/vlBPfSJQ5+61uf0rgbb8\nk6Vmthy41t2fye0/Dlzh7ivbfZZ66lJVbrwRVq+GX/86diWSZOWe0rgSqDGzQ8ysHzANaGh3TCPh\nRCpmNgQYDbxWfMki6dPaCrfeCpdfHrsSqTadtl/cvcXMZgOPEKY0LnD3tWY2K/f+fOAnwJ1m9jLh\nL4nvu/s7PVy3SEVbtgyGDYMTT4xdiVQbXVEq0gMmTYJLL4WLLopdiSRdWeepl5NCXarFK6/Al78M\nzc3hdnUi3aFlAkQimzt3z/1HRXqbRuoiZfTuu3DYYeEepEOHxq5G0kAjdZGIfvEL+MpXFOgSj0bq\nImXS2gpHHAGLFsGE9tddi3SRRuoikTz4IHz+8wp0iUuhLlImc+boYiOJT+0XkTJYsybMTW9uhv79\nY1cjaaL2i0gEc+fCt7+tQJf4NFIX6ab33oMvfCGsnT58eOxqJG00UhfpZfX1cPLJCnSpDAp1kW5a\nuFBrvEjlUPtFpBveeAPGjIEtW+Azn4ldjaSR2i8iveiee2DKFAW6VA6Fukg33HUXXHxx7CpE9lCo\ni3RRU1OYlz5pUuxKRPZQqIt00d13w7Rp0LfQ7dtFepFCXaQL3DXrRSqTQl2kC154AVpatHiXVB6F\nukgXfDRKt6Inmon0Ds1TFylRayuMGgWPPw5HHRW7Gkk7zVMX6WHZbLizkQJdKpFCXaREmpsulUzt\nF5ES7NgRFu5avRpGjIhdjVQDtV9EetDy5TBunAJdKpdCXaQEmpsulU7tF5Eibd8ebobR3AyDBsWu\nRqqF2i8iPaS+Hk47TYEulU2hLlKkhQs160Uqn9ovIkX485/h8MPDTTH23z92NVJN1H4R6QHLlsEZ\nZyjQpfIp1EWKsHQpnHtu7CpEClP7RaSADz+EYcPgT3/SSVLpfWq/iJTZo4+GJXYV6JIECnWRAurr\n1XqR5CgY6mZWa2aNZtZkZlfs45iMmb1oZq+YWbbsVYpEsns3PPggTJkSuxKR4nR6d0Uz6wPMBU4H\nNgPPm1mDu6/NO2YgcCtwprtvMrPBPVmwSG966ik47LCwfrpIEhQaqY8HNrh7s7vvBhYBU9sdcxFw\nn7tvAnD3P5e/TJE46uvhvPNiVyFSvEKhPgLYmLe/KfdavhrgIDP7nZmtNLNvlLNAkVjcNZVRkqfT\n9gtQzBzE/YDjgUnAZ4Bnzew5d2/qbnEiMb3wQrjYSHc4kiQpFOqbgfxu4ijCaD3fRuDP7v4X4C9m\n9iTwJWCvUK+rq/v4eSaTIZPJlF6xSC9ZujS0XnRzaelN2WyWbDbb5Z/v9OIjM+sLrCOMwrcAfwBm\ntDtROoZwMvVMoD+wApjm7mvafZYuPpJEOfpouOMO+Ou/jl2JVLNSLz7qdKTu7i1mNht4BOgDLHD3\ntWY2K/f+fHdvNLOHgVVAG3BH+0AXSZqmprCI14QJsSsRKY2WCRDpwE03hWCfPz92JVLttEyASBlo\nKqMklUbqIu288QaMGQPbtkH//rGrkWqnkbpINy1bBrW1CnRJJoW6SDu64EiSTO0XkTzvvQcjR8Km\nTTBgQOxqRNR+EemWhx+Gk05SoEtyKdRF8qj1Ikmn9otIzu7dMGQIvPIKDB8euxqRQO0XkS56+mk4\n/HAFuiSbQl0kZ9kyOOec2FWIdI9CXYSwdnpDg0Jdkk+hLgI0NsLOnTBuXOxKRLpHoS5CaL2cfbbW\nTpfkU6iLoH66pIemNErVe/ttOPRQePNN+PSnY1cj8kma0ihSouXL4bTTFOiSDgp1qXrLlsGUKbGr\nECkPtV+kqu3aFa4ibWwMjyKVRu0XkRI8+SSMHq1Al/RQqEtV06wXSRuFulQtd4W6pI9CXarWmjXQ\n2grHHBO7EpHyUahL1fpolK6rSCVNFOpStdR6kTTSlEapSm+9BTU1sG0b9O8fuxqRfdOURpEiLF8O\np5+uQJf0UahLVVLrRdJK7RepOjt3houNmprg4INjVyPSObVfRAr4/e/hi19UoEs6KdSl6ui2dZJm\nar9IVXGHUaPgt7+FMWNiVyNSmNovIp344x/hs59VoEt6KdSlqixdCuedF7sKkZ6jUJeqUl8P554b\nuwqRnqNQl6qxfj28+y6ceGLsSkR6jkJdqsb998PUqfApfeslxQp+vc2s1swazazJzK7o5LgTzazF\nzM4vb4ki5VFfr366pF+nUxrNrA+wDjgd2Aw8D8xw97UdHPcY8F/Ane5+XwefpSmNEs3WrTB2bFjA\nq1+/2NWIFK/cUxrHAxvcvdnddwOLgKkdHHc5sBh4q+hKRXrRsmUwebICXdKvUKiPADbm7W/KvfYx\nMxtBCPp5uZc0HJeKo9aLVItCoV5MQN8M/CDXW7HcJlIx3nsPnnkGamtjVyLS8/oWeH8zMCpvfxRh\ntJ7vvwOLLNwTbDAw2cx2u3tD+w+rq6v7+HkmkyGTyZResUiJHnoI/vZv4cADY1ciUlg2myWbzXb5\n5wudKO1LOFE6CdgC/IEOTpTmHX8nsMzdl3Twnk6UShTTp8OkSTBzZuxKREpX1hOl7t4CzAYeAdYA\n97j7WjObZWazuleqSM/buRMefhimTIldiUjv0CqNkmoPPwzXXANPPx27EpGu0SqNInm01otUG43U\nJbXa2mD48DBKP+KI2NWIdI1G6iI5K1aEW9Yp0KWaKNQltdR6kWqkUJdUcleoS3VSqEsqrV0bpjMe\nf3zsSkR6l0JdUmnx4rDWi2nRCqkyCnVJHXdYuBBmzIhdiUjvU6hL6rzwArS0wIQJsSsR6X0KdUmd\nhQvhoovUepHqpIuPJFVaW2HUKHj8cTjqqNjViHSfLj6SqpbNwtChCnSpXgp1SZW77oKLL45dhUg8\nar9IauzYEdZ6Wb0aRowofLxIEqj9IlVr+XIYN06BLtVNoS6p8dGsF5FqpvaLpML27fCFL0BzMwwa\nFLsakfJR+0Wq0pIlcNppCnQRhbqkgma9iARqv0jibdkCX/xieNx//9jViJSX2i9Sde65J6ybrkAX\nUahLCmjWi8geCnVJtHXrYPPmcJJURBTqknB33QXTpkGfPrErEakMfWMXINJV7iHU77ordiUilUMj\ndUmsFSvC4wknxK1DpJIo1CWxbr0VZs3SzTBE8mmeuiTSG2+ENdP/8z/hoINiVyPSczRPXarC7bfD\nhRcq0EXa00hdEmfXLjjkEHjkETjmmNjViPQsjdQl9e67D8aMUaCLdEShLokzZw5cfnnsKkQqk0Jd\nEmXlynAF6TnnxK5EpDIp1CVR5syB73wH+uqyOZEO6USpJMabb8Lo0bBhA3zuc7GrEekdOlEqqXX7\n7XDBBQp0kc4UFepmVmtmjWbWZGZXdPD+xWb2spmtMrNnzOzY8pcq1Wz3bpg3TydIRQopGOpm1geY\nC9QCY4EZZnZUu8NeA05292OBHwO3l7tQqW5LlkBNDRyr4YJIp4oZqY8HNrh7s7vvBhYBU/MPcPdn\n3f3/5XZXACPLW6ZUO01jFClOMaE+AtiYt78p99q+XAos705RIvleeAFefx2mTi18rEi1K2ZiWNFT\nVszsVOB/ACd19H5dXd3HzzOZDJlMptiPlio2Zw5cdpmmMUp1yGazZLPZLv98wSmNZjYRqHP32tz+\nlUCbu9/Q7rhjgSVArbtv6OBzNKVRSrZ1K4wdC01NMHhw7GpEel9PTGlcCdSY2SFm1g+YBjS0+0P/\nGyHQv95RoIt01Y9+BDNnKtBFilXwH7Tu3mJms4FHgD7AAndfa2azcu/PB34IDALmWbhjwW53H99z\nZUs1WLMGli4NN5cWkeLoilKpWOecA6eeCt/9buxKROIptf2iU09SkbJZeOUVWLw4diUiyaJlAqTi\ntLXB974HP/kJ9O8fuxqRZFGoS8W5915wh2nTYlcikjzqqUtF2bkz3FB6wYLQTxepdlqlURJt3rwQ\n6gp0ka7RSF0qxvbtcOSR8MQTcPTRsasRqQwaqUtiXX89TJmiQBfpDo3UpSK8/jocdxysWgUjOlsu\nTqTKaKQuiXT11fAP/6BAF+kuXXwk0TU0wO9/D6tXx65EJPkU6hLVpk1hwa4lS2DAgNjViCSf2i8S\nTUsLXHQR/OM/wkkdrsAvIqVSqEs0P/4x9OsHV+x1K3MR6Sq1XySK3/0Obr893KquT5/Y1Yikh0bq\n0uveegu+8Q345S9h2LDY1Yiki+apS69yh7PPDhcY3XBD4eNFqp3mqUtFu/lmePttuOaa2JWIpJNG\n6tJrVq6Es86CFSvg0ENjVyOSDBqpS0VqaoLzz4fbblOgi/Qkhbr0uMbGsJTuD38IF1wQuxqRdNOU\nRulRr74KZ5wB110H3/xm7GpE0k+hLj1m1So480y46Sa4+OLY1YhUB4W69IgXX4TJk+GWW+DCC2NX\nI1I9FOpSds8/H+aiz5sXTo6KSO/RiVIpq6efhq98Be64Q4EuEoNCXcpi9+4wu+WrX4Vf/zrclk5E\nep/aL9Jta9aEtVyGDoWXXtJ6LiIxaaQuXdbWBv/2b3DKKfD3fw8PPKBAF4lNI3XpkuZmuOQSaG0N\nl/0fdljsikQENFKXEr3/Ptx4I5x4Yjghms0q0EUqiUbqUpS33w5zzm+7DU4/PdwoeuzY2FWJSHsa\nqUunNm+G734Xampgyxb4j/+Au+9WoItUKoW67KWtDZ57DmbOhGOOCa+tWhXmntfUxK1NRDqn9osA\n4YTnU0/BffdBfT0MGADTp8P69TB4cOzqRKRYCvUq9v778MwzsGQJ3H8/DB8eLh567DE46qjY1YlI\nVxQMdTOrBW4G+gA/d/e97ixpZrcAk4H/Ai5x9xfLXah0T1sbrFsX2iofbRs2wHHHwdSp8OyzmsUi\nkgad3s7OzPoA64DTgc3A88AMd1+bd8xZwGx3P8vMJgA/dfeJHXyWbmdXRtlslkwms9frra3wpz+F\ntklTU3hsbAy3kjvoIJg4cc/2pS9Bv369X3ul2dfvUrpGv8/yKvV2doVG6uOBDe7enPvwRcBUYG3e\nMVOAXwG4+wozG2hmQ9x9W0mVS0G7dsGbb8LWrTB/fpbGxgxbt4b9LVvgtdfCNmQIHHlk2GpqwhK4\nJ5wAn/987P+CyqQQKi/9PuMqFOojgI15+5uACUUcMxKoqlB3D4tatd927ICdO/d+/Mtf4MMP92wf\nfLDn+Xvvwbvv7tm2bw+Pu3bBwQeHS/Hffx8OPDA8P/74cCHQ4YeHbf/9Y/82RCSWQqFebL+k/T8N\nOvy5yZM7+AN83/sfPe/sNfd9v7avra1t34/5W2vrJ5+3tITH9s937w6PffvCfvt9cvv0p8PWv//e\nzz/7WTjggD3bwIEwYkQI60GD9mwDB4bHAw4Ay/2m6+rCJiKSr1BPfSJQ5+61uf0rgbb8k6Vm9jMg\n6+6LcvuNwCnt2y9mpoa6iEgXlLOnvhKoMbNDgC3ANGBGu2MagNnAotxfAts76qeXUpSIiHRNp6Hu\n7i1mNht4hDClcYG7rzWzWbn357v7cjM7y8w2AB8C3+rxqkVEpEOdtl9ERCRZenTtFzP7mpm9amat\nZnZ8u/euNLMmM2s0sy/3ZB1pZGZ1ZrbJzF7MbbWxa0oiM6vNfQebzOyK2PUknZk1m9mq3HfyD7Hr\nSRIz+4WZbTOz1XmvHWRmj5nZejN71MwGFvqcnl7QazVwHvBk/otmNpbQnx8L1AK3mZkWFyuNA//b\n3Y/LbQ/HLihpchfXzSV8B8cCM8xMCyR0jwOZ3HdyfOxiEuZOwncx3w+Ax9z9SODx3H6nejRI3b3R\n3dd38NZU4G533527sGkD4UInKY1OPnfPxxfXuftu4KOL66R79L3sAnd/Cni33csfX9yZezy30OfE\nGh0PJ1yk9JFNhIuYpDSXm9nLZragmH+WyV46unBO38PuceC3ZrbSzGbGLiYF8q/O3wYMKfQD3V6l\n0cweA4Z28NY/u/uyEj5KZ2zb6eR3exUwD/ifuf0fA/8LuLSXSksLfefK7yR332pmBwOPmVljbgQq\n3eTuXsz1Pt0OdXc/ows/thkYlbc/Mvea5Cn2d2tmPwdK+QtUgvbfw1F88l+QUiJ335p7fMvM6gkt\nLoV6120zs6Hu/oaZDQPeLPQDvdl+ye+zNQDTzayfmR0K1AA6U16C3P/gj5xHOCktpfn44joz60c4\ned8QuabEMrPPmNmBuecHAF9G38vuagC+mXv+TWBpoR/o0ZtkmNl5wC3AYOBBM3vR3Se7+xoz+w2w\nBmgBLtO6vCW7wczGEVoI/xeYFbmexNnXxXWRy0qyIUC9hQWK+gIL3f3RuCUlh5ndDZwCDDazjcAP\ngeuB35jZpUAzcGHBz1GWioikh+aGi4ikiEJdRCRFFOoiIimiUBcRSRGFuohIiijURURSRKEuIpIi\nCnURkRT5/8/itESqh3f1AAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def sigm(x): return 1 / (1 + sp.exp(-x))\n",
"\n",
"xs = sp.linspace(-10, 10)\n",
"plt.plot(xs, sigm(xs));"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Nagib sigmoide može se regulirati množenjem ulaza određenim faktorom:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEACAYAAABMEua6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VVXW8PHfTiiC9CIgQUEJIQgqHXsAR5qAEBEEVEZF\npL0+DzOKWONYxzKKqAjIWB5BkBIICkiRKCAt1ABpgEgLLRAILaTs94+dQAhJ7r255/b1/cyZ287Z\nZxmvKzvr7LO30lojhBDCPwR5OgAhhBDWkaQuhBB+RJK6EEL4EUnqQgjhRySpCyGEH5GkLoQQfsRm\nUldK/VcpdUQpFV/CPp8qpVKUUluVUi2tDVEIIYS97Ompfw10Le5DpVR3oLHWOhR4BphoUWxCCCEc\nZDOpa61XAidL2KUX8G3evuuAakqpOtaEJ4QQwhFW1NTrA/sLvD4AhFjQrhBCCAdZdaFUFXotcw8I\nIYQHlLGgjYNAgwKvQ/Leu4JSShK9EEKUgta6cMe5WFYk9RhgFDBDKdUBSNdaHykmMAtOJwCioqKI\niorydBh+wdt+llrD0qXwyy8QFwebN0OtWtC6NbRpA61aQaNGULcuVKrkjng0x84dIzktmZS0FPN4\nwjzuO7WPC9kXqFe5HvUq1aNe5Xrsn7+f3sN6U6tiLapXqE71a6pT7Zpql55XvaYqZYKsSD1eJCsL\njh0zW1qa2U6cuPw8LQ1OnoRTp8yWnm4eT5+G8uWhShWoXNn8C61c+Yrn6quvHArF5k9WKfUDcB9Q\nSym1H3gdKAugtZ6ktV6olOqulNoFnAX+7vAPRAjBhQvw/ffw8cdQtiz07w+vvGKSeM2a7osjNSOV\ntQfWmu3gWrYe3kqQCiKsVhihNUJpUrMJ/Zr1I7RmKA2rNaT6NdVR6nJHMmpnFC/f+7L7AnalzEw4\ndAgOHoQDB8x28CAcPgxHjlze0tPNv6TrrjOPNWtCjRrmsW5duOUWqF4dqlaFatUuP1apYv5ll8Tq\npK61ftSOfUY5dFYhxCVHj8LEiWZr3RomTICOHUHZ/Qe3cxKPJ7IoZRFrD5pEfubiGTqEdKBD/Q68\ncs8rtKzXkloVa7knGHfLyoJ9+2DPniu3P/+E/ftN77pePQgJMVv9+uaxTRuoU8ck8Tp1zJ9SwcGe\n/qcBrCm/CA+IiIjwdAh+w1M/ywMH4I03YPZseOQRWLECwsNdf16tNVuPbGVuwlzmJMwh/UI6PZv0\n5MHQB3mr41s0rtH4ip63o7zyu5meDgkJV28HDpikfdNNl7fISFPfuuEGk7S9JFnbS7mrzq2U0lJT\nF8LYtQvuvx8GDIB//ANq13bt+bTWbDi0gdk7ZzM3YS65OpfI8Ej6hvelfUh7gpSfzBiSm2t62ps2\nXd62b4eMDGja1PzWLLg1agTlynk66hIppRy6UCpJXQg3S0w0Cf3VV2HYMNeeK1fn8lPyT7y76l2O\nnT3GgOYDiAyP5Pa6tzvVG/cahw7BqlWwbh1s3GiuKlerZi5E5G8tWkCDBu6rZ1lMkroQXiw+Hrp0\ngXffhSeecN15snOz+XHHj7y76l3KBpVl3N3j6Bvel+Ag3yolXEFrSEoySXzlSrOdOgV33w133GEu\nSLRsaerbfkSSuhBeauNG6NEDxo83I1tc4UL2Bb7d8i3v//E+IVVCGHf3OLrc3MV3e+VHjpixnYsW\nwfLlULEi3HOPSeT33GNKKkF+UjoqhiR1IbzQmjXQuzdMngwPPeSacyzZvYSnY56mRZ0WjLt7HHff\ncLdrTuRKOTmmlLJokdl27YLOnaFbN3jgAXPxMsBIUhfCy/z+Ozz8MHz7rclNVjt78SwvLH2Bn1J+\n4r+9/kvnmzpbfxJXys6G2FiYMQPmzTPDBrt1M9udd9oex+3nJKkL4UW2bTMXRWfMgE6drG9/zf41\nPD7vce5scCfju46n2jXVrD+JK+TmwurV5gczezbceKOpSfXrF5C98ZJIUhfCi/TqZaoHzz1nbbsX\ncy7yRuwbTN08lS96fEHf8L7WnsBVkpJgyhSTzGvWNIm8f3+4+WZPR+a1HE3qcvOREC4SF2eGSf/4\no7Xt7ji6g8HRgwmpEsKWZ7dQt1Jda09gtawsmD/f3DK7Ywf8/e+wZAk0a+bpyPySJHUhXCQqCl58\nEa65xro2V+9bTZ+ZfXin8zs81fIp7x7Vsn+/6ZV/9RWEhsLw4dC3r9ff7OPrJKkL4QLr18PWraZc\nbJXf//qdh398mP/r8390adzFuoattnEjvPOOmfdg0CAz5eQtt3g6qoAhSV0IF4iKgnHjrOul//rn\nr/Sf3Z8ZkTO8d3TL2rXw5pvmt9nzz5vhPu6YG1hcQZK6EBZbu9ZMNxIdbU17S3YvYfDcwczuN5v7\nGt5nTaNW+v13k8yTk029ac4ca2tOwiGS1IWwWFQUvPSSWfvAWQtTFjJk3hCi+0dz1w13Od+glX77\nDV57zcwv/tJLMHiw1Mu9gCR1ISz0xx9mRteYGOfbikmK4emYp4l5NIYOIR2cb9Aqf/5pyitxcaaH\n/uijUEZSibfw70kThHCz1183qxU522GNSYph6IKhLBy00HsS+tmz5h+uTRu4/Xbz2+uxxyShexlJ\n6kJYZNUqM1XJkCHOtbPn5B6ejnmaBY8uoM31bSyJzSlaw7RpZvKsvXvNhdBXXoEKFTwdmSiC/IoV\nwiL5vXRnpiq5mHORAbMH8Mq9r9CufjvrgiutLVtgxAhzA9HMmWYuFuHVpKcuhAV+/910Yh9/3Ll2\nXl7+MnUr1WV0u9GWxFVqWVmmXv7AA/DUU2bmREnoPkF66kJYwIpe+qKURczcMZPNwzZ79k7RnTvN\nb6datcw8ByEhnotFOEx66kI4KSnJDNF+7LHSt3Eo4xB/n/93vu/7PTUr1rQuOEfk5MCHH8J998Ez\nz5j5zCWh+xzpqQvhpCVLoGvX0g8CycnNYfDcwYxoO4J7b7zX2uDslX+Ft0wZU2q56SbPxCGcJj11\nIZy0ZIkpPZfWu6veJVfn8vI9L1sXlCO++w46dDBzmf/6qyR0HyfzqQvhhIsXTel5z57SrXe88q+V\n9JvVj43PbKR+lfrWB1iSixdhzBjzW2nuXGje3L3nF3aR+dSFcKM1ayAsrHQJ/cT5EwyOHszUXlPd\nn9BTU03PvEYNM6VkNR9ZMUnYJOUXIZzgTOll3LJx9GrSix5NelgblC1//AFt20KXLmZNUEnofkV6\n6kI4YelS+OADx49LOJbA3MS5JI9Ktj6o4mhtVh964w34+mvo3t195xZuI0ldiFJKSzPDGe+4w/Fj\nX1z+Ii/e9SLVK1S3PrCiZGaalYfi4syCz40bu+e8wu0kqQtRSsuXwz33OD551+9//c62I9v48WGL\nFy8tTkYG9OkDVaqYiwDXXuue8wqPkJq6EKVUmnq61pp/Lvknb3d6m/JlLJhw3ZajR6FjR9MznzVL\nEnoAkKQuRCloXbqkPmvnLHJ0DgOaD3BNYAXt3Qt3321q5xMnQnCw688pPE7KL0KUQlKSeQwLs/+Y\nzOxMxi0fx5SeUwhSLu5PxcebZD52LIwa5dpzCa8iSV2IUsjvpTsy79aXcV8SVjOMTo06uS4wMBO7\nR0bC+PEwwA1/EQivIkldiFJYutSxCbzSL6Tzzqp3WP74ctcFBbBgATz5pFnUwpm5C4TPkmkChHDQ\nxYtQu7aZGqCmnRMqvrjsRY6dPcbU3lNdF9j8+WZ2xQULoJ0XLLAhLOHoNAE2C3tKqa5KqUSlVIpS\namwRn9dSSi1WSm1RSm1XSg1xMGYhfEr+1AD2JvR9p/YxeeNk3uj4huuC+uUXGDoUfv5ZEnqAKzGp\nK6WCgc+ArkAz4FGlVHih3UYBm7XWtwMRwEdKKSnrCL/l6KiX11a8xvA2wwmp4qK5yWNjTS1o3jyz\nKLQIaLZ66u2AXVrrvVrrLGAG0LvQPqlAlbznVYA0rXW2tWEK4T0cSeo7ju5g0a5FjL37qj9yrbFm\njZmYa8YMWW5OALYvlNYH9hd4fQBoX2ifKcCvSqlDQGXgEevCE8K7HD9uVjnq0MG+/cevG8+otqOo\nUr6K7Z0dtXEj9O5t5kPv5OIRNcJn2Erq9lzZfAnYorWOUErdDCxVSt2mtc4ovGNUVNSl5xEREURE\nRDgQqhCet3w53HuvfVMDnDh/glk7Z5E4MtH6QLZvhx49YNIk6NbN+vaFx8TGxhIbG1vq40sc/aKU\n6gBEaa275r0eB+Rqrf9dYJ+FwNta69V5r5cDY7XWcYXaktEvwuc9/TTcdhuMHm173w9Wf0D80Xi+\n6/OdtUEkJZme+UcfyTj0AGD16Jc4IFQp1VApVQ7oD8QU2icRuD/v5HWAMGCP/SEL4RscmRogJzeH\nzzd8zuh2dmR/Rxw+bOZBf/NNSeiiSCWWX7TW2UqpUcAvQDAwVWudoJQalvf5JOAd4Gul1FbML4kX\ntNYnXBy3EG6XlGTuIG3SxPa+C5IXUK9yPdrWb2tdAOfOQa9e5uaiJ5+0rl3hV+TmIyHs9OmnZkqV\nKVNs79v5u8481fIpBrYYaM3Jc3PNKJdrr4Vvv3VsfgLh0yy/+UgIYdhbetl+dDsJxxJ4uNnD1p18\n3Dgz9GbKFEnookSS1IWwg9awciXYM2Drs/WfMaz1MMoFO7h6RnGmTIHoaJg7F8q7YQ524dPkzk8h\n7PDXX1C5spnzpSQnz59k5o6ZJIxMsObEy5bBq6+a3yj2zksgApokdSHssG0btGhhe7//bv4vPUJ7\nULdSXedPunMnDBwIs2dDaKjz7YmAIEldCDvEx8Ott5a8T05uDp9t+IwZkTOcP+HRo/Dgg/Dhh+Zu\nJyHsJDV1IexgT0/955Sfue7a62gfUngmDQdlZcHDD8OgQfD44861JQKOJHUh7GBPT33C+gnW3Gz0\n0ktQqRK84cKpeoXfkvKLEDZcuAB//glNmxa/z85jO9l+dDv9mvVz7mRz58KsWWayriDpcwnHSVIX\nwoaEBGjcuORJvD5b/xnPtHqG8mWcGHKYkgLPPmsWupCRLqKUJKkLYYOtevrpzNP8sP0HdozYUfqT\nnDtnFov+17+grYVTC4iAI3/fCWGDrXp6dEI09954L9dXvr50J9Aahg830z8OG1a6NoTII0ldCBts\n9dSnxU9jYHMn5niZMsXU0L/8UqYAEE6TpC6EDSX11A+fOcz6g+vpGdazdI1v3Agvvwxz5pjJuoRw\nkiR1IUpw7JgZ/RJSzJrRM7fPpFdYLyqWreh44ydOmJkXJ06EsDDnAhUijyR1IUoQH29KL8VVRaZv\nn86gFoMcb1hrGDoUevY0NxoJYREZ/SJECbZtK770kpKWwt70vXS+qbPjDX/9tRnCOG2acwEKUYgk\ndSFKEB8P7doV/dkP23+g/y39KRPk4H9Gu3fD2LGwYgVcc43zQQpRgJRfhChBcT11rbUZ9eLoykbZ\n2TB4MLzyCjRvbk2QQhQgSV2IYuTkmNlvi8q9m1I3kZ2bTfv6Dk7e9dZbUKUKjLZ4QWoh8kj5RYhi\n7N4NdeqYxTEKyx+brhwZV75mjRmLvmmTzOsiXEa+WUIUI3/kS2E5uTnM2D7DsdJLRoYpu0ycCNeX\n8s5TIewgSV2IYhRXT4/dG0vdSnUJrx1uf2PPPQcdO0KfPtYFKEQRpPwiRDHi42HAgKvfnx7v4Nj0\nOXPMGqObN1sXnBDFkJ66EMUoqqd+IfsC0YnRDGheRLYvyqFDMGIEfP+9WfhCCBeTpC5EEc6cMfm4\nceMr31+YspDb695O/Sr1bTeitZl18dlnob2TS9wJYScpvwhRhB07IDwcyhT6L8ShsenTp8Pevab8\nIoSbSE9diCIUNfIl/UI6y/YsIzI80nYDR47AmDFmOoCSlkwSwmKS1IUoQlH19OiEaDo16kT1CtVt\nNzByJDz5JLRp45oAhSiGJHUhilBUT31a/DT7Rr3Mnm3qN6+/7prghCiB1NSFKETrq3vqx88dZ8Oh\nDSwIXVDywcePmykA5s6VybqER0hPXYhCUlMhONhMEZBvQdIC/nbT36hQtkLJBz/3HAwcCHfc4dog\nhSiG9NSFKKSoevq8pHn0a9av5ANjYmD9eti61XXBCWGD9NSFKKRwPf3sxbOs+HMFPUJ7FH/QyZPm\nJqOpU6FiKZa2E8IiktSFKKRwT33J7iW0D2lf8qiXMWPMvC733uv6AIUogZRfhCgkPt6UxvNFJ0bz\nUNhDxR+wdKlZxWj7dtcHJ4QNNnvqSqmuSqlEpVSKUmpsMftEKKU2K6W2K6ViLY9SCDfJyoLkZGjW\nLO91ThY/p/xMr7BeRR9w/jwMHw5ffCFzuwivUGJPXSkVDHwG3A8cBDYopWK01gkF9qkGfA500Vof\nUErVcmXAQrhScjI0aHC5LL5y30puqn4TDao2KPqAt9+G1q2he3f3BSlECWyVX9oBu7TWewGUUjOA\n3kBCgX0GAnO01gcAtNbHXRCnEG5RuJ4enRBNn6bFzIG+YwdMmmQOEsJL2Cq/1Af2F3h9IO+9gkKB\nGkqpFUqpOKXUY1YGKIQ7FRz5orVmXtI8HmpaRD09N9fMwPivf0G9eu4NUogS2Erq2o42ygKtgO5A\nF+BVpVSos4EJ4QkFk/qm1E1UKFOB8FpFrHA0dSpkZ5vELoQXsVV+OQgULCY2wPTWC9oPHNdanwfO\nK6V+B24DUgo3FhUVdel5REQEERERjkcshAslJFy+SDovcR59mva5enHpI0fg5Zdh2TJZQFpYLjY2\nltjY2FIfr7QuvjOulCoDJAGdgUPAeuDRQhdKm2IupnYBygPrgP5a652F2tIlnUsIT8vMhKpVzRrR\nZctC8y+aM6XnFO5oUOiW/4ED4YYb4L33PBOoCChKKbTWyvaeRok9da11tlJqFPALEAxM1VonKKWG\n5X0+SWudqJRaDGwDcoEphRO6EL5g92648UaT0FPSUjh+7jjtQwqtWPTLL7B2LXz1lWeCFMIGmzcf\naa0XAYsKvTep0OsPgQ+tDU0I90pKgiZNzPP5SfPpHdabIFWgvHLunJkK4PPPZSoA4bWkIChEnqQk\nCAszz6MTo+kTXmgo41tvQdu20K2b+4MTwk4yTYAQeZKS4M474fCZw+w4uoOODTte/nDnTpgyxQyP\nEcKLSU9diDz5PfUFSQvo2rgr5cuUNx9obcour78Odet6NkghbJCkLkSe5GST1K+64Wj6dDh92szx\nIoSXk6QuBJCWZu4luqbqaVb+tZLuoXlzuZw6Bc8/bybsCg72bJBC2EGSuhBcLr38snsxd91wF1XK\nVzEfvPYa9OgBHTp4NkAh7CQXSoXgclKflzjv8tzpmzfDjBnmIqkQPkJ66kJgknrj0BwW71pMz7Ce\nZsKuESPM1Lo1a3o6PCHsJkldCExSz62RwM01bub6ytfD11+bD5580rOBCeEgKb8IgUnqlbssomeT\nnuaq6UsvweLFMmGX8DmS1EXAy86GP//UXDjzDf/b5HsYNw7694eWLT0dmhAOk6QuAt7evVDzuiyy\ngk9z+1+Z8NNPZg5eIXyQJHUR8JKSoGLdg3Ru3AM1ciS8/76Zg1cIHyQFQxHwkpLgTOWN9NwdDJUq\nwaBBng5JiFKTnroIeFt3XOBExT/o9O8fYekKKLzSkRA+RHrqIuCt35pOqwqHuebRx6B5c0+HI4RT\npKcuAt7elLI8VW4bTPnS06EI4TTpqYuAdvz4eS6cL8+goSOgShVPhyOE0ySpi4A2850JVKy2m3pD\nZFpd4R8kqYvAdfQoMSsTuDk0Wy6OCr8hSV0ELP3iWNZXCOfODiGeDkUIy0hSF4Hpjz/Yuf5nLp4O\n555W13k6GiEsI0ldBJ7sbBg5kgXPdqLC6ZY0bSqlF+E/JKmLwPPll1C1KjHlD3DmcF2aNPF0QEJY\nR5K6CCxHjsAbb3Dso3+xfXc6NasHU7myp4MSwjpy85EILM8/D0OGsDD4T24r8whlw6T0IvyLJHUR\nOH77DVasgIQEFiwcQsPs57lWSi/Cz0j5RQSGrCwYORI+/pjMa8qybM8yyqffSliYpwMTwlqS1EVg\nGD8e6teHyEh+++s3brnuFvbtqSBJXfgdKb8I/3fgALz3HqxZA0oRkxRDzyY9mZiEJHXhd6SnLvzf\n//4vjBgBoaForZmXOI+uN/bh6FFo2NDTwQlhLempC/+2ZAls3AjffQfAxtSNVCpXiaCTYdx0EwQH\nezg+ISwmPXXhvy5cMBdHJ0yAChUAmJc4jz5N+5AkpRfhpySpC//1wQdwyy3Qo8elt6ITo3mo6UOS\n1IXfkqQu/NOff5oRL+PHX3orOS2Zk+dP0rZ+W0nqwm9JUhf+R2tzYfSf/4Qbb7z09vzE+fQO602Q\nCpKkLvyWzaSulOqqlEpUSqUopcaWsF9bpVS2UqqvtSEK4aBZs8wwxn/844q3oxOj6RPeB62RpC78\nVomjX5RSwcBnwP3AQWCDUipGa51QxH7/BhYDMpmG8Jz0dDOEcdYsKFv20tupGakkHE8gomEER45A\nuXJQo4YH4xTCRWz11NsBu7TWe7XWWcAMoHcR+40GZgPHLI5PCMeMGwe9esGdd17x9oLkBXRr3I1y\nweWkly78mq1x6vWB/QVeHwDaF9xBKVUfk+g7AW0BbWWAQthtzRqYPx927rzqo+jEaJ68/UlASi/C\nv9nqqduToD8BXtRaa0zpRcovwv2ysuCZZ+Djj6FatSs+Op15mtX7VtO1cVfAJHVZGEP4K1s99YNA\ngwKvG2B66wW1BmYosxp7LaCbUipLax1TuLGoqKhLzyMiIoiIiHA8YiGK8p//QEgIPPLIVR8tSlnE\nPTfeQ+XyZjWM+Hj4n/9xd4BC2Cc2NpbY2NhSH69MB7uYD5UqAyQBnYFDwHrg0cIXSgvs/zWwQGs9\nt4jPdEnnEqLU9uyBdu1gwwZo1OiqjwfMHkDnRp0Z2nooAPXqwfr10KDBVbsK4XWUUmit7a6AlFh+\n0VpnA6OAX4CdwEytdYJSaphSaphzoQphgfwx6S+8UGRCz8zOZPGuxfQK6wXAsWNm9oCQEHcHKoR7\n2JzQS2u9CFhU6L1Jxez7d4viEsI+M2fCoUNmGGMRVuxdQfPrmlOnUh3AlF5atAAlV36En5JZGoXv\nOnkSxoyBuXOvGJNeUHSCmesl37ZtcOut7gpQCPeTaQKE7/rHP6BvX+jQociPc3Uu85PmX5HU83vq\nQvgr6akL37R4sVlEOj6+2F3WHVhH7Wtr07hG40vvbdsGTz/tjgCF8AxJ6sL3nD4Nw4bB1KlQqVKx\nu0UnRvNQ2OVeek6OuS+peXN3BCmEZ0j5RfieF16ABx6A++8vdhet9aW50/Pt3g116kDlyu4IUgjP\nkJ668C2//go//wzbt5e4W8LxBDKzM2lVr9Wl96SeLgKB9NSF7zhzxhTEJ02CqlVL3HX2ztn0adoH\nVWDsoox8EYFAkrrwHS+/DPfcA927l7ib1ppp8dN4tMWjV7wvPXURCCSpC9+wahXMnm0m7LJhU+om\nsnOzaV//iglFpacuAoIkdeH9zp+HJ5+Ezz+3a2WLafHTGNh84BWllzNnzI2njRuXcKAQfkCSuvB+\nr70GrVvDQw/Z3DUnN4cZ22cwsMXAK97fsQPCw6GMDA0Qfk6+4sK7rV4N339vaid2iN0bS91KdQmv\nHX7F+1JPF4FCeurCe50+DY89BpMnQ+3adh0yPX46g1oMuup9qaeLQCFJXXiv0aPNTUY9e9q1+4Xs\nC0QnRjOg+YCrPpOeuggUUn4R3unHH2HtWti0ye5DFqYs5Pa6t1O/Sv0r3tdaeuoicEhSF95n/37T\nS//5Z7j2WrsPmxY/7aoLpACpqRAcbKYIEMLfSflFeJfcXHjiCXjuOWjTxu7D0i+ks2zPMiLDI6/6\nTHrpIpBIUhfe5T//gawsGDvWocPmJsylU6NOVK9Q/arPpJ4uAomUX4T32LIF3n/frAodHOzQodPj\np/Nsm2eL/GzbNujUyYoAhfB+0lMX3uH8eRg40PTUGzZ06NBDGYfYmLqRHqE9ivxceuoikEhSF97h\nhRfgtttg0NVjzG2ZuX0mDzV9iAplK1z1WVYWJCVBs2ZWBCmE95Pyi/C82bPhp5/M8MUC87XYa1r8\nNN7t/G6RnyUnww03QMWKzgYphG+QpC48KzkZhg+HRYug+tUXOW1JOp7EwYyDdGpUdNF82zYpvYjA\nIuUX4Tlnz0JkJLz1lkPDFwuaHj+d/rf0Jzio6Aur8fEynFEEFknqwjO0Nj30li3hmWdK2YRm+vai\n53rJJz11EWik/CI8Y/Jk2LwZ1q0rVR0dYN3BdQC0ub74Xr701EWgkaQu3C8uDl55xUyr68QVzM83\nfM6w1sOuWAyjoPR0SEuDRo1KfQohfI6UX4R7paXBww/DxInQpEmpmzl85jA/Jf/Eky2fLHaf7duh\neXMIkm+5CCDydRfuk5tr5kePjDSJ3QmTN07mkWaPUKNC8cvbST1dBCIpvwj3efttyMiA995zqpmL\nORf5Mu5Lfhn8S4n7ST1dBCLpqQv3mDXLXBydORPKlnWqqTk759C0VlNa1Cm5Gy49dRGIJKkL11u7\nFkaMgAUL4PrrnW5uwvoJjG43usR9tDY1dUnqItBIUheutXcv9O0LX38Nt9/udHNxh+I4mHGQnmEl\nL3H3119QqRLUrOn0KYXwKZLUheucOgU9epi50R980JImJ6yfwMi2IykTVPLlIKmni0AlSV24RlYW\n9OsHHTvC//t/ljR59OxRYpJieKrlUzb3/eMPaNvWktMK4VMkqQvraQ2jRkGZMvDJJ6W+Y7SwyRsn\n83D4w9SsaLumsmQJPPCAJacVwqfYldSVUl2VUolKqRSl1FXrjCmlBimltiqltimlViul5A/fQPbR\nR+bi6MyZJrFbICsni4lxExndvuQLpADHjsHu3dC+vSWnFsKn2PwvTikVDHwG3A8cBDYopWK01gkF\ndtsD3Ku1PqWU6gpMBjq4ImDh5ebONb3zNWugcmXrmk2YS2iNUG6tY7u/sGwZREQ4PXJSCJ9kT0+9\nHbBLa71Xa50FzAB6F9xBa71Ga30q7+U6IMTaMIVPWLwYnn0WYmKgQQNLm7ZnGGO+pUul9CIClz1J\nvT6wv8CDbNyBAAAPUElEQVTrA3nvFecpYKEzQQkfFBsLjz8O8+ZBq1aWNr0pdRP7Tu2jd9PeNvfV\nWurpIrDZU/DU9jamlOoIPAncVdTnUVFRl55HREQQERFhb9PCm61ZA488Aj/+CHfeaXnzE9ZPYETb\nETaHMQIkJJiyy803Wx6GEG4RGxtLbGxsqY9XWpecs5VSHYAorXXXvNfjgFyt9b8L7XcrMBfoqrXe\nVUQ72ta5hA/auBG6d4fvvoMuXSxvPjUjlWZfNCNldAq1Ktayuf8nn5jEPmmS5aEI4RFKKbTWdg8h\ns6f8EgeEKqUaKqXKAf2BmEInvQGT0AcXldCFn4qPNzcXTZ7skoQO8Hrs6wxtNdSuhA5SehHCZk8d\nQCnVDfgECAamaq3fVUoNA9BaT1JKfQX0AfblHZKltW5XqA3pqfuTpCRzY9HHH0P//i45xc5jO4n4\nJoKkUUlUr2B7UerMTKhd20wRUIo1rIXwSo721O1K6laQpO5Hdu82Cf3NN+GJJ1x2mp4/9KRjw46M\nuWOMXfv/+iu8/LIp8QvhL1xRfhHisq1b4d574dVXXZrQY/fGsv3odka2HWn3MTKUUQhJ6sIRv/0G\nf/sbjB8PQ4e67DS5Opfnlz7PO53eoXyZ8nYfJ/V0ISSpC3tFR5sJumbMcHopOltm7ZiF1pr+ze2v\n1edPDdCune19hfBnspydsO2rr+C112DRImjd2qWnyszOZNzycUztNZUgZX+fQ6YGEMKQpC6KpzW8\n+65J6r/9BqGhLj/lxLiJhNcOp2Ojjg4dJ6UXIQwZ/SKKlpsLY8bAihVmTpd69Vx+yvQL6TSZ0IRf\nn/iV5tc1t/s4rSEkxPzeadzYhQEK4QGOjn6Rnrq4Wno6PPYYZGSYTFmtmltO+96q9+gV1suhhA6w\ncyeULy9TAwgBcqFUFLZ9u1kyqFEjM0bQTQl936l9TNk0hTci3nD42PyhjBatxSGET5OkLi6bOdPc\nVPTaa/Dpp2696vjKr68wvM1w6lcpaQLQokk9XYjLpPwiIDsbXnzRLHCxZAm0bOnW08ckxfDbX78R\nPzze4WMzM2HVKpg+3QWBCeGDJKkHumPHzNwtZcvChg1Q0/b6n1Y6cPoAQxcMZe4jc6lSvorDx69e\nDbfc4rYqkRBeT8ovgWzFCmjTBu64AxYudHtCz87NZuCcgTzX/jnuuqHIKfhtktKLEFeSnnogOncO\nxo2DOXPMtLndu3skjDd/e5NyweUYe9dVa5nbbckSmDDBwqCE8HGS1APNmjVmIq62bWHbNqhRwyNh\nrPhzBZM3TWbTM5sIDgouVRtHj8KePTI1gBAFSVIPFJmZ8Prr8M038PnnEBnpsVCOnT3GY9GP8U3v\nb6hXufQ3NX31FfTqJVMDCFGQJPVAsHmzWRS6cWPTO7/uOo+ForVmyPwhDGoxiC6NS79a0unTZum6\nlSstDE4IPyAXSv3ZiRMwerRZam7sWDNk0YMJHeCTtZ+Qdi6Ntzq95VQ748dD164QFmZRYEL4CUnq\n/ig7Gz77DJo2NROjJCTA4MEev+Uy7lAc7656lx8if6BscOlrJunp5t6oV1+1MDgh/ISUX/zNsmXw\nP/8DderA8uXQooWnIwIgJS2FvjP78kWPL2hUvZFTbY0fb9a7dsOkkUL4HJml0V/s2gX//CfEx8NH\nH0Hv3h7vmedLPJ7I/d/dT1REFE+3etqpttLTzaWBdetkAi8RGGSN0kCTnAxDhkCHDtC+PezYAQ89\n5DUJfcfRHXT6thNvd3rb6YQO8PHHZsSLJHQhiiblF1+1Ywe8/baZonD0aNNT97J75bcd2UaX77vw\n4d8+ZNCtg5xu78QJc6lgwwYLghPCT0lP3dds2WLWCO3UCW691SzM+dprXpfQN6du5oH/e4DxXcdb\nktAB/vMf6NMHbrrJkuaE8EtSU/cFOTnw888wcSJs3Wpq58OGwbXXejqyIm04uIEHf3iQiT0m0je8\nryVtpqVBkyawcSM0bGhJk0L4BFn5yJ8cPmxum5w8Ga6/HoYPN2PNK1TwdGTFWrVvFX1n9uWrXl/R\nK6yXZe1+9JH5A0USuhAlk6TubbQ2S8h98YWpl/frB/Pnu32Oc0dl5WTx5u9vMmnjJL7r8x1dG3e1\nrO1jx2DSJHNjrBCiZJLUvYHWpqwyc6bZKlQwvfIpU6BqVU9HZ9POYzt5LPox6laqy5ZhW5yaz6Uo\nH34IjzwCN9xgabNC+CVJ6p6UkGCS+IwZcOECDBhgyiu33eY1QxJLkqtzGb92PO+seod3Or3D062e\nRlkc95YtpgK1ZYulzQrhtySpu1NODqxfD4sWmZJK/qpD33xjxpj7QCLPtzd9L0PmDSFH57Du6XXc\nVN36ISnr10PPnuaSQoMGljcvhF+S0S+udvQoLF5sEvmSJVC/PnTrZu5zv+suCC7dXOKekpGZwZdx\nX/L+H+/zwp0vMOaOMaWeD70kq1ZB377w3//Cgw9a3rwQPkNGv3jasWNm4cxVqyA21twU1LmzSeQf\nfAAhIZ6OsFTSzqXx6bpP+SLuC+6/6X5+G/IbzWo3c8m5fv3VVKKmTYO//c0lpxDCb0lP3Rlam6V3\nVq82E3uvWgWpqWbNz3vuMVuHDj69isPB0wf5aM1HfLPlGyLDI3nhrhcIrem6mbQWLTILM82aBffd\n57LTCOEzpKfuKrm5kJICmzZduVWqBHfeCXffDSNHmlkRfaykUliuzmX9wfVM3TSVOQlzGHL7ELYN\n30ZIFdf+lTFvnrmnKibG/C4UQjhOeuqF5ebCvn1mZEpCAuzcaR63bYPataFVK2jd2jy2bOnxRSes\nkpObw8p9K5mzcw7RidFUKV+FAc0HMKLtCGpVrOXSc2sNP/wAY8bAwoXmRyuEMBztqQdmUs/JgYMH\nTemk4JaUZLZq1SA83GzNmpnHW2+F6tU9HbmlMjIzWL1/NXMT5jI/aT7XV76eyPBIIsMjCa8d7vLz\nZ2fDnDlmTpcTJ8xoTi+Z/l0Ir2F5UldKdQU+AYKBr7TW/y5in0+BbsA5YIjW+qp7/9yW1HNzTYY4\ncKDo7a+/TE+8dm0zM1TBLTTUrBbkAzf8OCpX55J0PIm1B9aa7eBadp3YRcu6Lekd1pvIZpEuGZZY\nlFOnYOpUs9jFjTeaHnrPnj5ftRLCJSxN6kqpYCAJuB84CGwAHtVaJxTYpzswSmvdXSnVHhivtb6q\nIlrqpJ6dDSdPmhmdTpwwj/nbkSNXbkePmtEnlSubUSZFbQ0amAlErrnG8Vi8SGxsLBEREVe9n5Ob\nw1+n/iI5LZmUtBSS05JJTEsk7lAcNSrUoENIBzrU70CHkA7cVvc2ygWXc0u8Wpup3ydNMsPyu3Qx\nybxtW7ecvkTF/SxF6cjP01pWXyhtB+zSWu/Na3wG0BtIKLBPL+BbAK31OqVUNaVUHa31kataW7IE\nzpyBjIzLjxkZZmn49HTThTt16vLz9HSzX7VqULMm1KhhHvOf16ljSiN16lzerrsOyrknUbnbxZyL\nHD17lNSMVCbNnkRipURSM1JJPZPKoYxD7Dm5hz0n91CnUh2a1GxCkxpNCK0ZSrfQbrS5vg3XXeue\n+r/WsHcvxMWZWRXj4sw15WuvNUMVt2zxrlv+JQlZS36enmUrqdcH9hd4fQBob8c+IcDVSf2DD0wv\nulIl85j/vG5dU/KoWtUk8IKPVatCkPdP+661Jis3i6ycrCseL2RfIDM70zzmZF56fT77PGcvnuVs\n1lnOXjzLmYtnLj0/ffE0J8+f5OSFk5w8f5L0C+mcvHCSizkXqV2xNvUq1yMjNYPKqZWpV6kereq1\nokdoD26ucTM3V7+ZCmWtncUxJwfOn4dz58x29qz5fVv4D6X8LTHRTF/TujW0aWN65K1bm9+5QgjX\nspXU7a2XFP7ToMjjrkt7C9IK73nlrubVBeBw3lZ0c/qKJ7qIvXSB1zr/f1d8crkclPeONo+X/l/n\nPdcajUbnP+Y/z3udq83+CoVSQQSpvEeCCFJBBKlgglQQwSqIoKDyBKkKBKsaBAeVIVgFUyYomGAV\nTHBQmbznZSgbVJaywWWpEVSWOsFlKRNUlmAVfOlHfSo5iv1fRLFPX/4x6kLPC2+5uWbLybn8PDfX\nVLiysq58zM6GixdNEs/MhIoVr9yqVDFJum5d89i8ubnHqk4ds4ZoPWvn9BJC2MlWTb0DEKW17pr3\nehyQW/BiqVLqSyBWaz0j73UicF/h8otSykuGvgghhG+xsqYeB4QqpRoCh4D+wKOF9okBRgEz8n4J\npBdVT3ckKCGEEKVTYlLXWmcrpUYBv2CGNE7VWicopYblfT5Ja71QKdVdKbULOAv83eVRCyGEKJLb\nbj4SQgjhei4dVqKU6qeU2qGUylFKtSr02TilVIpSKlEp9YAr4/BHSqkopdQBpdTmvM269eMCiFKq\na953MEUpNdbT8fg6pdRepdS2vO/kek/H40uUUv9VSh1RSsUXeK+GUmqpUipZKbVEKVXNVjuuHisY\nD/QBfi/4plKqGaY+3wzoCnyhlPL+cYveRQP/0Vq3zNsWezogX5N3c91nmO9gM+BRpZTr50fwbxqI\nyPtOtvN0MD7ma8x3saAXgaVa6ybA8rzXJXJpItVaJ2qtk4v4qDfwg9Y6K+/Gpl2YG52EY+Tis3Mu\n3Vyntc4C8m+uE86R72UpaK1XAicLvX3p5s68x4dsteOp3vH1mJuU8h3A3MQkHDNaKbVVKTXVnj/L\nxFWKunFOvofO0cAypVScUmqop4PxAwXvzj8C2LyFz+n51JVSS4G6RXz0ktZ6gQNNyRXbQkr42b4M\nTAT+lff6TeAj4Ck3heYv5Dtnvbu01qlKqdrAUqVUYl4PVDhJa63tud/H6aSutS7NgmMHgYJLCYfk\nvScKsPdnq5T6CnDkF6gwCn8PG3DlX5DCQVrr1LzHY0qpaEyJS5J66R1RStXVWh9WStUDjto6wJ3l\nl4J1thhggFKqnFKqERAKyJVyB+T9C87XB3NRWjjm0s11SqlymIv3MR6OyWcppSoqpSrnPb8WeAD5\nXjorBngi7/kTwDxbB7h0OTulVB/gU6AW8LNSarPWupvWeqdS6kdgJ5ANjPCeFTR8xr+VUrdjSgh/\nAsM8HI/PKe7mOg+H5cvqANFKKTC5ZZrWeolnQ/IdSqkfgPuAWkqp/cBrwHvAj0qpp4C9wCM225Fc\nKoQQ/kPGhgshhB+RpC6EEH5EkroQQvgRSepCCOFHJKkLIYQfkaQuhBB+RJK6EEL4EUnqQgjhR/4/\nldtQ8RZzpngAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(xs, sigm(0.5*xs), 'r');\n",
"plt.plot(xs, sigm(xs), 'g');\n",
"plt.plot(xs, sigm(2*xs), 'b');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Derivacija sigmoidalne funkcije:\n",
" \n",
"$$\n",
"\\frac{\\partial\\sigma(\\alpha)}{\\partial\\alpha} = \n",
"\\frac{\\partial}{\\partial\\alpha}\\big(1 + \\exp(-\\alpha)\\big) =\n",
"\\sigma(\\alpha)\\big(1 - \\sigma(\\alpha)\\big)\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Model **logističke regresije**:\n",
"$$\n",
" h(\\mathbf{x}|\\mathbf{w}) = \\sigma\\big(\\mathbf{w}^\\intercal\\boldsymbol{\\phi}(\\mathbf{x})\\big) =\n",
" \\frac{1}{1+\\exp(-\\mathbf{w}^\\intercal\\boldsymbol{\\phi}(\\mathbf{x}))}\n",
"$$\n",
"\n",
"\n",
"* **NB:** Logistička regresija je klasifikacijski model (unatoč nazivu)!\n",
"\n",
"### Probabilistički izlaz\n",
"\n",
"* $h(\\mathbf{x})\\in[0,1]$, pa $h(\\mathbf{x})$ možemo tumačiti kao **vjerojatnost** da primjer pripada klasi $\\mathcal{C}_1$ (klasi za koju $y=1$):\n",
"\n",
"$$\n",
"h(\\mathbf{x}|\\mathbf{w}) = \\sigma\\big(\\mathbf{w}^\\intercal\\mathbf{\\phi}(\\mathbf{x})\\big) = \\color{red}{P(y=1|\\mathbf{x})}\n",
"$$\n",
"\n",
"* Vidjet ćemo kasnije da postoji i dublje opravdanje za takvu interpretaciju\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Funkcija logističkog gubitka\n",
"\n",
"\n",
"* Definirali smo model, trebamo još definirati **funkciju gubitka** i **optimizacijski postupak**\n",
"\n",
"\n",
"* Logistička funkcija koristi **gubitak unakrsne entropije**\n",
"\n",
"\n",
"### Definicija\n",
"\n",
"\n",
"* Funkcija pokriva dva slučajeva (kada je oznaka primjera $y=1$ i kada je $y=0$):\n",
"\n",
"$$\n",
"L(h(\\mathbf{x}),y) =\n",
"\\begin{cases}\n",
"- \\ln h(\\mathbf{x}) & \\text{ako $y=1$}\\\\\n",
"- \\ln \\big(1-h(\\mathbf{x})\\big) & \\text{ako $y=0$}\n",
"\\end{cases}\n",
"$$\n",
"\n",
"\n",
"* Ovo možemo napisati sažetije:\n",
"$$\n",
"L(h(\\mathbf{x}),y) = \n",
"- y \\ln h(\\mathbf{x}) - (1-y)\\ln \\big(1-h(\\mathbf{x})\\big)\n",
"$$\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHpJJREFUeJzt3XuUVNWZ9/HvQwODchGVERAwoCgq0gjIRUApAiqiQaPJ\nBDUSQZdORrxlvSaRMbGzJq+Os8y8Co6KRo2aCJOYxCDiFS00KvdLIxcF0QyItkTEQe5NP+8fu5pu\nm6a7uruqTl1+n7XO6qru7anHs5ofm3323sfcHRERyS/Noi5ARERST+EuIpKHFO4iInlI4S4ikocU\n7iIieUjhLiKSh5IKdzMrMrNlZvbcIX4+1czWmdkKM+uX2hJFRKShku253wSsBg6aFG9mY4Ge7n4i\ncC3wYOrKExGRxqg33M2sKzAW+DVgtTQZBzwB4O4LgPZm1jGVRYqISMMk03P/f8CtQMUhft4F2Fjt\n/SagaxPrEhGRJqgz3M3sQuAzd19G7b32A01rvNeeBiIiEWpez8+HAuMS4+qtgHZm9qS7T6jW5mOg\nW7X3XRPf+xozU+CLiDSCu9fVua5VnT13d5/i7t3cvQcwHnitRrADzAImAJjZEGCbu5cd4nw63Lnj\njjsiryFbDl0LXQtdi7qPxqqv535QPidC/LpEWE939zlmNtbM1gM7gImNrkZERFIi6XB393nAvMTr\n6TV+NjnFdYmISBNohWoEYrFY1CVkDV2LKroWVXQtms6aMqbToA8y89o+6+abYeRIuOiijJQhIpJT\nzAxP9Q3VTGjbFpYujboKEZH8Enm49+kDK1dGXYWISH7JinAvLY26ChGR/BL5mHt5ObRrB1u2QOvW\nGSlFRCRn5OyYe/Pm0KsXrFoVdSUiIvkj8nAHKC7WuLuISCplRbjrpqqISGplTbjrpqqISOpkTbiv\nXAkZurcrIpL3siLcO3eGigooq3UvSRERaaisCHcz3VQVEUmlrAh30Li7iEgqZVW4q+cuIpIaCncR\nkTwU+fYDlbZvh44dw9eiooyUJCKS9XJ2+4FKbduGWTPr10ddiYhI7suacAfdVBURSZV6w93MWpnZ\nAjNbbmarzeyuWtrEzOxLM1uWOG5vTDEadxcRSY16H5Dt7rvNbKS77zSz5sBfzWy4u/+1RtN57j6u\nKcX06QMzZjTlDCIiAkkOy7j7zsTLlkARsLWWZg0e8K9JC5lERFIjqXA3s2ZmthwoA15399U1mjgw\n1MxWmNkcMzu1McX07AmbN8NXXzXmvxYRkUrJ9twr3P10oCtwtpnFajRZCnRz977ANODZxhTTvDmc\nfLIe3CEi0lT1jrlX5+5fmtnzwBlAvNr3t1d7/YKZPWBmR7n714ZvSkpKDryOxWLEYrGDPqPypurg\nwQ2pTEQkP8TjceLxeJPPU+8iJjPrAJS7+zYzOwx4CfiFu8+t1qYj8Jm7u5kNAn7v7t1rnKfORUyV\n7rkHNm6E++5r+P+MiEi+aewipmR67p2BJ8ysGWEY5yl3n2tm1wG4+3TgO8APzawc2AmMb2ghlYqL\nYc6cxv7XIiICWbT9QKVPPglDM1u2hK2ARUQKWc5vP1CpU6fw9dNPo61DRCSXZV24m2mlqohIU2Vd\nuIPCXUSkqbIy3LVSVUSkabIy3LU7pIhI02TdbBkI2w8ccwz87/+GVasiIoUqb2bLALRpowd3iIg0\nRVaGO+imqohIU2RtuBcXa9xdRKSxsjbc1XMXEWk8hbuISB7KytkyAOXl0K4dfPZZuMEqIlKI8mq2\nDIQpkMXFsHBh1JWIiOSerA13gAsugNmzo65CRCT3ZHW4f+tbMGsWZGjkSEQkb2R1uPftC3v3wtq1\nUVciIpJbsjrczULv/bnnoq5ERCS3ZHW4Q9XQjIiIJC9rp0JW2r0bOnaEDz6ADh3SUJiISBZLy1RI\nM2tlZgvMbLmZrTazuw7RbqqZrTOzFWbWr6FF1KVVKxg9Wg/NFhFpiDrD3d13AyPd/XSgGBhpZsOr\ntzGzsUBPdz8RuBZ4MNVFamhGRKRh6h1zd/ediZctgSJga40m44AnEm0XAO3NrGMqixw7Fl59Ffbs\nSeVZRUTyV73hbmbNzGw5UAa87u6razTpAmys9n4T0DV1JYYHd/TuDfPmpfKsIiL5q97nHLl7BXC6\nmR0BvGRmMXeP12hWc7C/1junJSUlB17HYjFisVjShVYOzZx7btL/iYhIzonH48Tj8Safp0GzZczs\nZ8Aud7+n2vceAuLuPjPxfi0wwt3Lavy3jZotU2nVqjA889FHYf67iEghSNdsmQ5m1j7x+jDgHGBZ\njWazgAmJNkOAbTWDPRVOPRWKirQNsIhIMuoblukMPGFmzQh/ETzl7nPN7DoAd5/u7nPMbKyZrQd2\nABPTUagZjBsXVqsWF6fjE0RE8kfWL2Kqbu5cmDIFFixIUVEiIlmuscMyORXue/eG1apr1kCnTikq\nTEQki+Xdwzpq07IlnHcePP981JWIiGS3nAp30GpVEZFk5NSwDMDWrdC9O5SVwWGHNb0uEZFsVhDD\nMgBHHQX9+8Nrr0VdiYhI9sq5cAcNzYiI1CfnhmUA3n8fRo6EjRuhWU7+9SQikpyCGZYBOOkk6NwZ\nXngh6kpERLJTToY7wA03wLRpUVchIpKdcnJYBsLj977xDXjjDejVK2WnFRHJKgU1LAPh8XvXXAP/\n9V9RVyIikn1ytucO4YZq377wt79B27YpPbWISFYouJ47QLdu8M1vwpNPRl2JiEh2yelwh3Bj9f77\nIUP/ABERyQk5H+5nnw0tWoQHaIuISJDz4W6maZEiIjXl9A3VSjt3wnHHwcKFcPzxafkIEZFIFOQN\n1UqHHw4TJ8IDD0RdiYhIdsiLnjvAhx/CwIFhWmTr1mn7GBGRjEpbz93MupnZ62a2yszeNbMba2kT\nM7MvzWxZ4ri9oYU0VY8eMGwY/O53mf5kEZHsU2/P3cw6AZ3cfbmZtQGWABe7+5pqbWLAj9x9XB3n\nSWvPHcKMmVtugdLScKNVRCTXpa3n7u6fuvvyxOuvgDXAsbXV0NAPT7VRo2D/fpg3L+pKRESi1aAb\nqmbWHegHLKjxIweGmtkKM5tjZqempryGMYPJk2Hq1Cg+XUQkezRPtmFiSOYZ4KZED766pUA3d99p\nZucDzwIn1TxHSUnJgdexWIxYLNaIkuv2gx/AL38JS5bAgAEpP72ISFrF43Hi8XiTz5PUbBkzawHM\nBl5w93uTaP8hMMDdt1b7XtrH3Cs9/DDMnAlz52rsXURyWzpnyxjwKLD6UMFuZh0T7TCzQYS/NLbW\n1jYTJk2CTz7Rk5pEpHAlM1tmOPAGUEoYWweYAhwH4O7Tzex64IdAObCTMHNmfo3zZKznDuEB2lOm\nwIoVUFSUsY8VEUmpxvbc82YRU03uEIvBhAlw9dUZ+1gRkZRSuNdi4UK45BJ47z2tWhWR3FTQe8sc\nyqBBMHw43FvvLWARkfyS1z13gA0bQsivXg3HHJPxjxcRaRINy9Thlltg377wxCYRkVyicK/D55/D\nySfDW2/BSQctrRIRyV4ac6/D0UfDrbfCbbdFXYmISGYURM8dYNcu6NULZswIWwOLiOQC9dzrcdhh\ncOedcOONYfxdRCSfFUy4A1xxBfzjP8Ldd0ddiYhIehXMsEylTZugf3945RXo2zfqakRE6qZhmSR1\n7Qr/8R9w1VWwd2/U1YiIpEfBhTuEPd+7dAlj8CIi+ajghmUqbd4Mp58OL74YhmlERLKRhmUa6Nhj\n4T//MwzP7NkTdTUiIqlVsOEOYfbM8cfDv/1b1JWIiKRWwQ7LVPr00zBrZvZsGDgw6mpERL5OwzKN\n1KkT3HdfGJ7ZvTvqakREUqPge+4Qntr03e+GaZLa+11Esol67k1gBo88As89B08/HXU1IiJNV2+4\nm1k3M3vdzFaZ2btmduMh2k01s3VmtsLM+qW+1PQ68kj485/hpptg+fKoqxERaZpkeu77gFvcvTcw\nBLjezE6p3sDMxgI93f1E4FrgwZRXmgHFxTBtWnju6uefR12NiEjj1Rvu7v6puy9PvP4KWAMcW6PZ\nOOCJRJsFQHsz65jiWjNi/PgQ7pdfDvv3R12NiEjjNGjM3cy6A/2ABTV+1AXYWO39JqBrUwqL0r//\nO5SXw+23R12JiEjjNE+2oZm1AZ4Bbkr04A9qUuP9QVNjSkpKDryOxWLEYrFkPz6jmjeHmTPDvPcz\nzoBLL426IhEpFPF4nHg83uTzJDUV0sxaALOBF9z9oMmCZvYQEHf3mYn3a4ER7l5WrU3WToU8lCVL\nYMwYiMehd++oqxGRQpS2qZBmZsCjwOragj1hFjAh0X4IsK16sOeqAQPgnnvg29+GbduirkZEJHn1\n9tzNbDjwBlBK1VDLFOA4AHefnmh3PzAG2AFMdPelNc6Tcz33SjffHKZHvvgitGoVdTUiUkga23PX\nCtUkVFSE2TN79sAf/hDG5EVEMkErVNOoWTN48knYuRP++Z/DdgUiItlM4Z6kli3hj3+ElSthypSo\nqxERqZvCvQHatIHnn4dnnw0P+hARyVYaPW6gDh3g5Zdh+PDwesKEqCsSETmYwr0RunULM2dGjoSj\njoILL4y6IhGRr9OwTCOdcgrMmgWTJsHcuVFXIyLydQr3Jhg0KNxkveyyMBYvIpItFO5NdNZZ4SEf\nkyaFoBcRyQYac0+BwYPDGPzYseE5rFdcEXVFIlLoFO4p0q8fvPoqnHtuCPirr466IhEpZAr3FOrd\nG15/HUaPDqtZb7gh6opEpFAp3FPspJNg3jwYNSoE/E9+EnVFIlKItHFYmmzaFIZozjknrGYtKoq6\nIhHJRdoVMgt98UV4ilObNvD00+GriEhDaFfILHTkkWEWzdFHw4gRsHlz1BWJSKFQuKdZy5bw2GOh\nB3/mmVBaGnVFIlIINCyTQTNnwo03wlNPwXnnRV2NiOQCDcvkgPHj4c9/hh/8AB56KOpqRCSfqece\ngfXrYdy4MExz//1w2GFRVyQi2SptPXcze8zMysxs5SF+HjOzL81sWeK4vaFFFJqePWHhQvjqq7Av\n/EcfRV2RiOSbZIZlHgfG1NNmnrv3Sxy/TEFdea9NmzAGf+WVMGRImFUjIpIq9Ya7u78JfFFPswb/\nk0HADG6+GZ55Bq65Bn7xC6ioiLoqEckHqbih6sBQM1thZnPM7NQUnLOgDB8OixaFh35ceCFs3Rp1\nRSKS61Kxt8xSoJu77zSz84FngZNqa1hSUnLgdSwWIxaLpeDj80PnziHcf/rTsMPkU0/B2WdHXZWI\nZFo8Hicejzf5PEnNljGz7sBz7t4nibYfAgPcfWuN72u2TJLmzAnDNBMnQkkJtGgRdUUiEpXI5rmb\nWUczs8TrQYS/MDSw0ARjx8KyZbB8OQwbBuvWRV2RiOSaZKZCzgDeBnqZ2UYzm2Rm15nZdYkm3wFW\nmtly4F5gfPrKLRwdO8Ls2XDVVTB0KDz6KOgfPiKSLC1iygGrV8Pll8MJJ8DDD4eNyESkMGj7gTx2\n6qmwYAH06AF9+uhB3CJSP/Xcc8zbb8OkSXDaaWHrgk6doq5IRNJJPfcCMXRouNF60knQty88+aTG\n4kXkYOq557ClS0Mv/thjYfp06NYt6opEJNXUcy9A/fuHla1Dh4bX06ZBeXnUVYlINlDPPU+sWQP/\n8i+wbRs8+GDYjExEcp967gXulFPgtdfg1lvDI/2uuQb+/veoqxKRqCjc84hZmA+/enXYUrh37zAv\nXjtNihQeDcvksRUrwlBNeTlMnQqDB0ddkYg0lIZl5CB9+8Kbb4aAv+SS0Kv/29+irkpEMkHhnuea\nNQsP5H7//TA3vn9/mDIFtm+PujIRSSeFe4Fo3TpsH7xiBWzeHIL+kUdg//6oKxORdNCYe4FasgR+\n9KPw1Ke77oILLgg3ZEUkuzR2zF3hXsDcYdYs+Nd/hSOOCCGvpz+JZBeFuzTa/v3w9NPw85/DySfD\nnXeGR/2JSPQ0W0YaragIrrwS3nsvPKD7ggvge98LN2FFJDcp3OWAli3h+uvDY/1OPz084q8y9EUk\ntyjc5SCtW8Ntt8H69dCrFwwfDldcEfavEZHcoHCXQzriCLj9dvjgg/BwkBEjYPx4WLUq6spEpD7J\nPCD7MTMrM7OVdbSZambrzGyFmelWXJ5p1y705D/4ICyC+uY34TvfCdsNi0h2Sqbn/jgw5lA/NLOx\nQE93PxG4FngwRbVJlmnbFn78Y9iwIQzVXHopjB4Nr7yip0GJZJt6w93d3wS+qKPJOOCJRNsFQHsz\n65ia8iQbtW4NN98cxuSvvDK8PuMM+P3vteJVJFukYsy9C7Cx2vtNQNcUnFeyXMuWYd+alSvhjjvg\n3nvDPPkHH4QdO6KuTqSwNU/ReWpOsK/1H+klJSUHXsdiMWKxWIo+XqLUrBmMGwff+hb89a/wq1+F\nBVHXXAOTJ0OXLlFXKJI74vE48Xi8yedJaoWqmXUHnnP3PrX87CEg7u4zE+/XAiPcvaxGO61QLSDr\n14c95H/7Wzj/fLjlljB0IyINE+UK1VnAhEQRQ4BtNYNdCk/PniHcN2wIWxlceimcdRY88wzs2xd1\ndSL5r96eu5nNAEYAHYAy4A6gBYC7T0+0uZ8wo2YHMNHdl9ZyHvXcC1h5OfzpT3D//SHwr702HJ06\nRV2ZSHbTxmGSM0pL4YEH4L//G8aMCVseDBumLYdFaqNwl5yzbRs88UQI+sMOg+uuC48CPOKIqCsT\nyR4Kd8lZFRUwdy48/DC8+mp43uu118KgQerNiyjcJS+UlcFvfhOCvk2bEPJXXAHt20ddmUg0FO6S\nVyoq4PXXQ8i/9FKYQz9pUti8rJm2u5MConCXvLVlC/zud/DYY7B9O1x1VVgZ27171JWJpJ/CXfKe\nOyxdCo8/DjNnQt++MHEifPvbYb8bkXykcJeCsnt3eLj3b34D77wTtj/4/vfDdsRFRVFXJ5I6Cncp\nWGVlMGNG2Orgk0/CdMorr4Ti4qgrE2k6hbsIsHp1CPnf/jbMsLnssvD0qB49oq5MpHEU7iLVVFSE\nHSpnzAj72fTsGXr0//RP0FFPG5AconAXOYR9+8LiqBkz4Lnnwu6U48fDxRfD0UdHXZ1I3RTuIknY\ntQuefz7sa/PyyzBkCHz3u2HGjYJespHCXaSBduwIQf+HP3w96C++GDp0iLo6kUDhLtIENYN+wICw\nx83FF0NXPTRSIqRwF0mRnTvhlVfC/vOzZ8OJJ4agv+SScGNWJJMU7iJpsG8fxOMh6J99Fo46Ci66\nKBwDB2qfG0k/hbtImlVUwMKF8Je/hGPbtrCh2UUXhZWxrVpFXaHkI4W7SIatW1cV9KWlMGoUXHgh\njB2rxwdK6qQ13M1sDHAvUAT82t3vrvHzGPAXYEPiW39091/WaKNwl7y1ZQu8+GIYo3/55TBOf+GF\n4ejXTw8dkcZLW7ibWRHwHjAa+BhYBFzm7muqtYkBP3L3cXWcR+EuBWHfvrA6dvbssGjqq6/g/PPD\ncc45eoygNExjwz2Z20GDgPXu/pG77wNmAhfVVkNDP1wkH7VoASNHwq9+Be+/D/Pmhe2JH300TKsc\nMQLuvjsM5ai/I+nSPIk2XYCN1d5vAgbXaOPAUDNbQejd/x93X52aEkVy24knhuPGG8M0y3gcXngh\nrIrdvRvOPRfOOy/06rVKVlIlmZ57Mn2LpUA3d+8LTAOebVJVInnq8MPDDddp02D9+hD0AwaEJ031\n6BEeCv6zn4VhnX37oq5WclkyPfePgW7V3ncj9N4PcPft1V6/YGYPmNlR7r61eruSkpIDr2OxGLFY\nrBEli+QHs6pe/eTJsGcPvP12eGbsDTfAhg1hCOecc8LRq5duzBaCeDxOPB5v8nmSuaHanHBDdRSw\nGVjIwTdUOwKfubub2SDg9+7evcZ5dENVpAE++wxeey2sln3llTA+P3p0OEaN0nTLQpHuqZDnUzUV\n8lF3v8vMrgNw9+lmdj3wQ6Ac2EmYOTO/xjkU7iKN5B7m1VcG/bx5cOyxYfHUqFGhh3/kkVFXKemg\nRUwiBWT/fli2LPTs584Nwzm9eoWgHzkShg2Dtm2jrlJSQeEuUsD27AlbI8ydG27SLl4MffpALBbC\nfuhQaNMm6iqlMRTuInLArl0wf34I+ngcliwJDwwfMQLOPjv07Nu1i7pKSYbCXUQOaefOEPZvvBHG\n6xctgpNPDkE/YgQMH6459tlK4S4iSduzJwR8Zdi/8w506wZnnRWO4cPhG9+IukoBhbuINEF5OaxY\nAW++GRZQvfkm/MM/VAX9sGHQuzcUFUVdaeFRuItIyriHFbSVYf/WW1BWBoMHh6AfNiy81k3a9FO4\ni0habdkSply+9VY4li8P0y/PPDPMxjnzzLCFglbRppbCXUQyavfuMNf+7bfDmP0774ThncqwHzIk\n7JvTunXUleY2hbuIRMod/ud/qoJ+/nx4993Qux8yJAzjDBkS9tLRs2eTp3AXkayze3cYvpk/PxwL\nFsCXX4bdL6sfxxwTdaXZS+EuIjnh00/DNMwFC8Kq2kWLoH37qqAfOBD699fN2koKdxHJSRUVYVO0\nhQurwn7lynBzduDAqqO4OEzPLDQKdxHJG3v3hvH6RYuqjnXr4JRT4IwzwjFgAJx2GrRsGXW16aVw\nF5G8tnNnWGi1eHHYK2fx4vBAk9NOC0Hfv3/42rt3fvXwFe4iUnB27Ag3bBcvDtMylyyBDz4I++ZU\nBn7//mGHzMMPj7raxlG4i4gQevilpbB0aQj7Zctg7dowht+/P/TrF47TT8+NB5wo3EVEDmHvXli1\nKgT9smUh+EtLoUOHEPLVj+OOy65Vtgp3EZEG2L8/DOEsX/71Y9euEPJ9+1Ydp54KrVpFU6fCXUQk\nBcrKwo3b6sf69XDCCVVhX1wcjs6d09/LT1u4m9kYqh6O/Wt3v7uWNlOB8wkPx77K3ZfV0kbhLiI5\nac8eWL26KuxXrgxf3auCvrg43Ljt3Tu1N2/TEu5mVgS8B4wGPgYWAZe5+5pqbcYCk919rJkNBu5z\n9yG1nEvhnhCPx4nFYlGXkRV0LaroWlTJhWvhHnr5paVfP957D7p2DUFf/ejZs3H74Tc23JvX8/NB\nwHp3/yjxITOBi4A11dqMA54AcPcFZtbezDq6e1lDiykUufCLmym6FlV0LarkwrUwg06dwnHuuVXf\n37cP3n8/9O5XroQnnwxfy8rCFM3TTgtHnz7ha9eu6RnaqS/cuwAbq73fBAxOok1XQOEuIgWnRYsw\nNNO7N4wfX/X97dvD0M6774bj1VfD1x07qtpXP5o6nl9fuCc7jlKzBI2/iIhU07Zt2PZ4cI3u8eef\nh2malcezz4av+/eHkG+s+sbchwAl7j4m8f42oKL6TVUzewiIu/vMxPu1wIiawzJmpsAXEWmEdIy5\nLwZONLPuwGbge8BlNdrMAiYDMxN/GWyrbby9McWJiEjj1Bnu7l5uZpOBlwhTIR919zVmdl3i59Pd\nfY6ZjTWz9cAOYGLaqxYRkTplbBGTiIhkTsqfZGhmY8xsrZmtM7OfHKLN1MTPV5hZv1TXkC3quxZm\ndkXiGpSa2VtmVhxFnZmQzO9Fot1AMys3s0syWV8mJflnJGZmy8zsXTOLZ7jEjEniz0gHM3vRzJYn\nrsVVEZSZdmb2mJmVmdnKOto0LDfdPWUHYehmPdAdaAEsB06p0WYsMCfxejAwP5U1ZMuR5LU4Ezgi\n8XpMIV+Lau1eA2YDl0Zdd4S/F+2BVUDXxPsOUdcd4bUoAe6qvA7A50DzqGtPw7U4C+gHrDzEzxuc\nm6nuuR9Y9OTu+4DKRU/VfW3RE9DezDqmuI5sUO+1cPd33P3LxNsFhPUB+SiZ3wuAG4BngC2ZLC7D\nkrkWlwN/dPdNAO7+9wzXmCnJXItPgHaJ1+2Az929PIM1ZoS7vwl8UUeTBudmqsO9tgVNXZJok4+h\nlsy1qO5qYE5aK4pOvdfCzLoQ/mA/mPhWvt4MSub34kTgKDN73cwWm9mVGasus5K5Fo8Avc1sM7AC\nuClDtWWbBudmfVMhG0qLnqok/f9kZiOBScCw9JUTqWSuxb3AT93dzcw4+HckXyRzLVoA/YFRwOHA\nO2Y2393XpbWyzEvmWkwBlrt7zMxOAF4xs77uvj3NtWWjBuVmqsP9Y6BbtffdCH/D1NWma+J7+SaZ\na0HiJuojwBh3r+ufZbksmWsxgLBWAsLY6vlmts/dZ2WmxIxJ5lpsBP7u7ruAXWb2BtAXyLdwT+Za\nDAX+L4C7f2BmHwK9CGtwCkmDczPVwzIHFj2ZWUvCoqeafzhnARPgwArYWhc95YF6r4WZHQf8Cfi+\nu6+PoMZMqfdauPvx7t7D3XsQxt1/mIfBDsn9GfkLMNzMiszscMINtNUZrjMTkrkWawm70pIYY+4F\nbMholdmhwbmZ0p67a9HTAclcC+DnwJHAg4ke6z53HxRVzemS5LUoCEn+GVlrZi8CpUAF8Ii75124\nJ/l7cSfwuJmtIHRGf+zuWyMrOk3MbAYwAuhgZhuBOwjDc43OTS1iEhHJQylfxCQiItFTuIuI5CGF\nu4hIHlK4i4jkIYW7iEgeUriLiOQhhbuISB5SuIuI5KH/D/6Ia+lO1gAkAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"xs = linspace(0, 1)\n",
"plt.plot(xs, -sp.log(xs));"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHapJREFUeJzt3XmUVOWZx/HvI6KOEkSDArIMGgkKAiIKLUoojBgExIkm\nRhMlmkQJRx3HmUlMjJN0PEmczDiDEjei0ShRGINGwLC4loAIiDQ7bcAVEFlUCAiJNP3MH2+1tE0v\nVV3LreX3Oeceqrpeqh6v3b++PPe97zV3R0REistBURcgIiKZp3AXESlCCncRkSKkcBcRKUIKdxGR\nIqRwFxEpQkmFu5m9bWbLzazCzBY1MGa8ma01s2Vm1jezZYqISCoOTnKcAzF3/7C+F81sOHCiu3cz\nswHAvUBZhmoUEZEUpdKWsUZeGwU8DODuC4E2ZtYuncJERKT5kg13B54zs8VmdnU9r3cE1td6vgHo\nlG5xIiLSPMm2Zc5y901mdgzwrJlVuvvcOmPqHtlrXQMRkYgkFe7uvinx51Yz+xPQH6gd7huBzrWe\nd0p87VNmprAXEWkGd2+sLV6vJtsyZna4mX0u8fgI4DxgRZ1h04DRiTFlwHZ331xPgdrc+dnPfhZ5\nDfmyaV9oX2hfNL41VzJH7u2AP5lZzfhH3f0ZMxuTCOwJ7j7DzIab2TrgY+CqZlckIiJpazLc3f0t\n4NR6vj6hzvPrMliXiIikQVeoRiAWi0VdQt7QvthP+2I/7Yv0WTo9nZQ+yMxz9VkiIoXi17+GI4+E\n73+//tfNDM/GCVUREcmeBQvg85/P/Psq3EVEIrRiBfTqlfn3VVtGRCQiu3bBscfCX/8KBzcwvUVt\nGRGRArNqFZx0UsPBng6Fu4hIRLLVkgGFu4hIZFasgN69s/PeCncRkYjoyF1EpMi4w/LlCncRkaLy\n/vvhz/bts/P+CncRkQjUtGQs5UmOyVG4i4hEIJsnU0HhLiISiWz220HhLiISiWzOlAEtPyAiknNV\nVdC6NWzZAq1aNT5Wyw+IiBSIdeugQ4emgz0dCncRkRzL9slUULiLiORctk+mQpLhbmYtzKzCzKbX\n81rMzHYkXq8ws1syX6aISPHI9slUSOIG2Qk3AKuBzzXw+kvuPiozJYmIFLdchHuTR+5m1gkYDjwA\nNHTGNkvXWImIFJddu8LSAyeemN3PSaYtMw74AVDdwOsODDSzZWY2w8x6ZKw6EZEis3Jl9m7QUVuj\n4W5mI4Et7l5Bw0fnS4DO7t4H+A3wVGZLFBEpHrloyUDTPfeBwCgzGw4cBrQ2s0fcfXTNAHffWevx\nTDO7x8yOdvcP675ZeXn5p49jsRixWCzN8kVECktT4R6Px4nH42l/TtJXqJrZYODf3f2COl9vRzi6\ndzPrDzzu7l3r+fu6QlVESt6QIfDjH8N55yU3vrlXqKba9fHEh40BcPcJwNeAsWZWBewGLk21CBGR\nUuCemwuYQGvLiIjkzHvvQZ8+YU2ZZNdx19oyIiJ5Lts36KhN4S4ikiO5mikDCncRkZxRuIuIFKFc\nnUwFnVAVEcmJmht0bN0KRxyR/N/TCVURkTy2di0cd1xqwZ4OhbuISA7kst8OCncRkZxQuIuIFKHl\ny3N3MhUU7iIiOZHrI3fNlhERybKdO6Fdu/Bnixap/V3NlhERyVOrVsHJJ6ce7OlQuIuIZNnChdCv\nX24/U+EuIpJl06fDiBG5/Uz13EVEsmjHDujcGTZtat4FTOq5i4jkoVmzYNCg3F2ZWkPhLiKSRdOn\nw6hRuf9ctWVERLKkqipMgVy+HDp2bN57qC0jIpJnXn4ZunZtfrCnI6lwN7MWZlZhZtMbeH28ma01\ns2Vm1jezJYqIFKaoWjKQ/JH7DcBq4IC+ipkNB050927ANcC9mStPRKRwTZsGF1wQzWc3Ge5m1gkY\nDjwA1Nf3GQU8DODuC4E2ZtYuk0WKiBSa11+H3buhb0S9jGSO3McBPwCqG3i9I7C+1vMNQKc06xIR\nKWjTp4ejdkv5VGhmHNzYi2Y2Etji7hVmFmtsaJ3n9U6LKS8v//RxLBYjFmvsLUVECte0afCjH6X+\n9+LxOPF4PO3Pb3QqpJn9CrgCqAIOA1oDT7j76Fpj7gPi7j458bwSGOzum+u8l6ZCikhJ+OADOOEE\n2LwZDjssvffKylRId7/Z3Tu7+/HApcALtYM9YRowOlFEGbC9brCLiJSSmTNhyJD0gz0djbZl6uEA\nZjYGwN0nuPsMMxtuZuuAj4GrMlyjiEhBiXIKZA1doSoikkGffBKuSq2sDH+mS1eoiojkgTlzoHv3\nzAR7OhTuIiIZlA8tGVC4i4hkjHu0V6XWpnAXEcmQVatCwJ9yStSVKNxFRDKmpiUT1VWptSncRUQy\nJF9aMqCpkCIiGbF5c5gls2ULHHJI5t5XUyFFRCL04INw0UWZDfZ06MhdRCRNVVVhLZmpUzO/xK+O\n3EVEIjJ1KnTpEt3a7fVRuIuIpOk3v4Hrr4+6is9SW0ZEJA0rVsCwYfD229CyZebfX20ZEZEI3HUX\njBmTnWBPh47cRUSa6aOPwonUNWugffvsfIaO3EVEcuzBB2HEiOwFezp05C4i0gz79kG3bjBpEgwY\nkL3P0ZG7iEgOzZgBbdtmN9jToXAXEWmGfJz+WJvaMiIiKaqshFgM3nkHDj00u5+VtbaMmR1mZgvN\nbKmZrTaz2+oZEzOzHWZWkdhuSbUQEZFCcdddcPXV2Q/2dBzc1AB3/5uZDXH33WZ2MDDPzM5293l1\nhr7k7nlwcykRkez561/hscfCxUv5LKmeu7vvTjw8BGgBfFjPsDxYnl5EJLt+/3sYOhQ6doy6ksYl\nFe5mdpCZLQU2Ay+6++o6QxwYaGbLzGyGmfXIdKEiIlHbty+0ZPL5RGqNJtsyAO5eDZxqZkcCs80s\n5u7xWkOWAJ0TrZvzgaeAL9Z9n/Ly8k8fx2IxYrFY8ysXEcmxP/wBjjkGzjore58Rj8eJx+Npv0/K\ns2XM7D+APe5+eyNj3gL6ufuHtb6m2TIiUrD27Al3Wpo8GQYOzN3nZnO2TFsza5N4/A/AUKCizph2\nZuGWsGbWn/BLo76+vIhIQbrzTjjjjNwGezqSact0AB42s4MIvwwmuvvzZjYGwN0nAF8DxppZFbAb\nuDRbBYuI5Nq2bXD77TB/ftSVJE8XMYmINOGGG/afTM215rZlFO4iIo1Ytw7KymD1ajj22Nx/vhYO\nExHJgp/8BG68MZpgT4eO3EVEGrBwIVx8MfzlL3D44dHUoCN3EZEMcocf/ABuvTW6YE+Hwl1EpB7T\npoXb6H3721FX0jxJXaEqIlJKqqrgpptg3Dho0SLqappHR+4iInU88AB06gTDhkVdSfPphKqISC1b\nt0Lv3vDnP8Npp0Vdjea5i4hkxCWXQNeu8F//FXUlQXPDXT13EZGExx8PN+F45JGoK0mfjtxFRIDN\nm6FPnzBLpn//qKvZT20ZEZFmcg8XK510EvzqV1FX81lqy4iINNOkSeEq1EmToq4kc3TkLiIlbdMm\nOPVUmDED+vWLupoDafkBEZEUucOYMWHLx2BPh9oyIlKyJk6Ed96BKVOiriTz1JYRkZK0cSP07QvP\nPBPaMvlKbRkRkSTt2wdXXQXXXpvfwZ4OhbuIlJxbboHq6nAjjmLVaLib2WFmttDMlprZajO7rYFx\n481srZktM7O+2SlVRCR9U6aEKY+TJ8PBRXzWsdH/NHf/m5kNcffdZnYwMM/Mznb3eTVjzGw4cKK7\ndzOzAcC9QFl2yxYRSd2qVTB2LMyeDW3bRl1NdjXZlnH33YmHhwAtgA/rDBkFPJwYuxBoY2btMlmk\niEi6tm+Hr34V/ud/8mO1x2xrMtzN7CAzWwpsBl5099V1hnQE1td6vgHolLkSRUTSU10NV1wBX/kK\njB4ddTW50WTHyd2rgVPN7EhgtpnF3D1eZ1jdaTr1znksLy//9HEsFiMWi6VSq4hIs9x6K+zYAf/7\nv1FX0rR4PE48Hk/7fVKa525m/wHscffba33tPiDu7pMTzyuBwe6+uc7f1Tx3Ecm5adPClMdXX4X2\n7aOuJnVZmeduZm3NrE3i8T8AQ4GKOsOmAaMTY8qA7XWDXUQkCn/5C3zve/DHPxZmsKejqbZMB+Bh\nMzuI8Itgors/b2ZjANx9grvPMLPhZrYO+Bi4Krsli4g0bdMmOP98uO02KCvB+XtafkBEis727TB4\nMHz96+GCpUKmm3WIiAB79oRZMX37wh13gKUci/lF4S4iJa+qKtxRqVWrsOLjQUWwwIoWDhORkuYO\nV18Nn3wCDz1UHMGejiJeWUFESskPfwiVlfDcc3DIIVFXEz2Fu4gUvP/+73CbvLlz4Ygjoq4mPyjc\nRaSg3Xcf3H03zJsHRx8ddTX5Q+EuIgVr3DgYPx6efx46aUWrz1C4i0hB+uUv4fe/h5degi5doq4m\n/yjcRaSguIcLk556CubMgQ4doq4oPyncRaRguMO//Ru88ALE43DMMVFXlL8U7iJSEKqrw+qOS5bA\niy/CUUdFXVF+U7iLSN6rqgqrO77xBjz7LLRuHXVF+U/hLiJ5bedOuPTSEPCzZmkee7JK/AJdEcln\nGzbAoEFw3HHw9NMK9lQo3EUkLy1dCmeeCZddBr/9LbRsGXVFhUVtGRHJO3/+M1x5JdxzT1iTXVKn\ncBeRvHL33fCLX4R7n555ZtTVFC6Fu4jkhaqqsLLjjBnw8stwwglRV1TYFO4iErktW0Jv3QxeeUVz\n2DOhyROqZtbZzF40s1VmttLM/rmeMTEz22FmFYmtwO9aKCK5smABnH46DBgAs2cr2DMlmSP3vcCN\n7r7UzFoBr5nZs+6+ps64l9x9VOZLFJFi5A733gvl5XD//XDhhVFXVFyaDHd3fx94P/F4l5mtAY4D\n6oZ7gd+GVkRyZfduGDMGli2D+fPhxBOjrqj4pDTP3cy6An2BhXVecmCgmS0zsxlm1iMz5YlIsVm3\nDsrKwuMFCxTs2ZL0CdVES2YKcIO776rz8hKgs7vvNrPzgaeAL9Z9j/Ly8k8fx2IxYrFYM0oWkUL1\n6KPwL/8CP/85jB0bTqDKZ8XjceLxeNrvY+7e9CCzlsDTwEx3vyOJ8W8B/dz9w1pf82Q+S0SKz/bt\nYUXHigp47DE49dSoKyocZoa7p/xrMJnZMgb8DljdULCbWbvEOMysP+GXxof1jRWR0jJ3bgjzo46C\nxYsV7LmSTFvmLOByYLmZVSS+djPQBcDdJwBfA8aaWRWwG7g0C7WKSAHZuxduvRUeeCDMhhk5MuqK\nSktSbZmMfJDaMiIlY906+Na34POfh4cegnbtoq6ocGWtLSMikqx9+2DcuLAmzOWXhwXAFOzR0PID\nIpIRq1fDd78Lhx4alhDQFMdo6chdRNKyd29YxXHwYPj2t8PNqxXs0dORu4g0W0UFfOc70L49vPYa\ndOkSdUVSQ0fuIpKyXbvgpptg2DC48cawTK+CPb8o3EUkae7wxBPQowe8915YG2b0aF1pmo/UlhGR\npKxdC9dfD+vXw8SJoccu+UtH7iLSqD174Kc/DdMbzz033LhawZ7/dOQuIvVyh6lT4V//NdxMY+lS\n6NQp6qokWQp3ETnAkiUh1LdtgwkTYOjQqCuSVKktIyKf2rgRrrwSRoyAb34zHK0r2AuTwl1E+Pjj\nsMZ6797QoQO8/jpccw0crH/bFyyFu0gJq6oKqzZ27w6VleFCpNtug9ato65M0qXfyyIlqLoapkyB\nW24JJ0mnTNl/6zspDgp3kRLiDrNmwU9+AgcdBHffHaY36iKk4qNwFykR8+fDj38MW7aEhb4uukih\nXszUcxcpci+/DOedB5ddFmbCrFgBF1+sYC92CneRIjVnTmi5XH45fP3rYfmAq67SDJhSof/NIkUm\nHg/TGt99N/TWr7gCWraMuirJtSaP3M2ss5m9aGarzGylmf1zA+PGm9laM1tmZn0zX6qINMQ9LLv7\npS/B1VeHm2ZUVoa11hXspSmZI/e9wI3uvtTMWgGvmdmz7r6mZoCZDQdOdPduZjYAuBfQxCqRLKuq\ngscfh1//OgT8TTfBN76h1oskEe7u/j7wfuLxLjNbAxwHrKk1bBTwcGLMQjNrY2bt3H1zFmoWKXm7\nd8NDD8Htt4ebZPznf4YbZ+gkqdRI6fe7mXUF+gIL67zUEVhf6/kGoBOgcBfJoK1b4b77wvz0sjJ4\n7LGwFK9IXUmHe6IlMwW4wd131TekznOvO6C8vPzTx7FYjFgsluzHi5S0VavgjjvClaQXXxxuQt2j\nR9RVSTbE43Hi8Xja72PuB2TwgYPMWgJPAzPd/Y56Xr8PiLv75MTzSmBw7baMmXkynyUigTvMng3j\nxsHy5TB2LHz/+3DssVFXJrlkZrh7yg23Jo/czcyA3wGr6wv2hGnAdcBkMysDtqvfLtI8u3bBo4/C\nnXeGmS433gjTpsGhh0ZdmRSSJo/czexsYA6wnP2tlpuBLgDuPiEx7i5gGPAxcJW7L6nzPjpyF2nE\n66/DPffAH/4QpjRefz0MGaKTpKWuuUfuSbVlMkHhLnKgffvg6afDCdJly+B734MxY8IMGBHIYltG\nRDLvvffCVMbf/haOOw6uvTYsEaDWi2SKwl0kR/btg2eeCYEej8Mll8CTT0K/flFXJsVI4S6SZRs3\nwoMPhjseHXtsuH3dI4/A5z4XdWVSzBTuIlnwyScwfXpovcyfH5YE+NOf4LTToq5MSoXCXSSDli0L\ngf7oo9CzZ1hi9//+D444IurKpNQo3EXStG0bTJoUQn3btnBDjAUL4AtfiLoyKWWaCinSDH/7W2i7\nTJwYbooxYkQI9XPOgRYtoq5OionmuYtkWXU1zJ0bLjJ64onQP7/iinAvUp0clWzRPHeRLHAPffRJ\nk2DyZDjyyBDoy5dDp05RVyfSMIW7SD3WrQuB/thjoQVz6aWhDdO7d9SViSRHbRmRhHfeCUvqTp4M\n69eHi4wuuyysm671XSQq6rmLNENNoP/xj+Fo/Z/+KRylx2K6VZ3kB4W7SJLefjucEK0d6JdcElZg\n1M2kJd8o3EUasWZNWMflySfh3XfhwgvDQl3nnKNAl/ymcBepxR2WLNkf6Dt3wle/GqYtDhqklosU\nDoW7lLxPPoGXXoKpU/ffuejii0Ogn346HHRQ1BWKpE7z3KUk7dgBM2eGQJ81C7p3Dy2XWbPg5JM1\ny0VKl47cpeC88Ua4e9HTT8PChaHNcuGFcMEF0KFD1NWJZFbW2jJm9iAwAtji7r3qeT0GTAXeTHzp\nCXf/RT3jFO7SLHv3hmVzawJ9+/awlsvIkXDuudCqVdQVimRPNtsyDwG/AR5pZMxL7j4q1Q8Xacim\nTaG1MnMmPPccnHBCCPOJE8OaLuqfizSuyXB397lm1rWJYepsSlqqqkKLZeZMmDED3noLhg6F88+H\nO+9Uu0UkVZk4oerAQDNbBmwE/t3dV2fgfaXIrV8Ps2eH7fnn4R//cX+Yn3mmpiuKpCMTPz5LgM7u\nvtvMzgeeAr6YgfeVIrN7d1j7vCbQt24NR+cjR8L48To6F8mktMPd3XfWejzTzO4xs6Pd/cO6Y8vL\nyz99HIvFiMVi6X685LF9+6CiAp59NmyLFoV++Ve+Em4Qrd65yIHi8TjxeDzt90lqKmSi5z69gdky\n7QgzadzM+gOPu3vXesZptkyRc4c33wwnQJ97Dl54Adq3DzNahg6FwYN1UwuRVGVttoyZTQIGA23N\nbD3wM6AlgLtPAL4GjDWzKmA3cGmqRUjheu+9EOIvvBD65nv3hvVaRo6EO+6Ajh2jrlCkNOkiJknJ\n1q3hEv8XXwyBvmVLWB73nHPCdtJJuipUJJO0toxkxbZt4SRoPB4C/d134eyzQ6B/+cvQp49uCC2S\nTQp3yYgtW0KYz5kTjtDffhvOOiuE+ZAh0LevpiiK5JLCXZplw4b9QT5nTrgy9Oyzw8nPL30J+vVT\nmItESeEuTXKHykqYOzds8+bBrl1h4a2aMO/dW20WkXyicJcD/P3v4YYVL78cgnzePGjdOhyZDxoU\ntu7ddQJUJJ8p3IVt28LqifPnh0CvqIBu3ULPfNCgEOqamihSWBTuJaa6GlavhldeCWH+yiuhXz5g\nAAwcGAK9rEwXDYkUOoV7kfvoo3D5fk2YL1oExxwTFtiq2Xr1Ur9cpNgo3ItIVRWsWBGWwF2wIGwb\nN4b7gJaVhSPzsrIQ7iJS3BTuBco9XBi0aFEI80WLQq+8S5fQYikrC1uPHpqSKFKKFO4F4oMPYPFi\nePXVEOSLFoXZKgMGQP/+YTv9dGjTJupKRSQfKNzz0M6dYSriq6+GbfHiMKPltNPgjDP2B3qnTpqO\nKCL1U7hH7OOPYenSEOCvvRb+fOedcJLzjDP2b927aw1zEUmewj2Hdu2CZctCiC9ZEoL8zTehZ8/Q\nUjn99HDZfs+e0LJl1NWKSCFTuGfJ9u3hiHzJkrC99lo4Ij/llNBeOe20EOannAKHHBJ1tSJSbBTu\nGbBpU5ipUrMtWRJWSezdOxyJ14R5jx46IheR3FC4p2DfPli7NhyR19727g1L2vbtG0K8b99w+b4u\nDBKRqCjcG7BzJyxfHnrkNX+uWAHt2sGpp35206wVEck3JR/u1dXhxhI1AV6zvf9+aKP06fPZ7cgj\ns1aKiEjGZC3czexBYASwxd17NTBmPHA+4QbZV7p7RT1jMhbu27eHo+8VK0KYL18OK1eGC3969w7T\nD2tCvFs3XdkpIoUrm+E+CNgFPFJfuJvZcOA6dx9uZgOAO929rJ5xKYf73/8ebi5RE+Q12/btYZph\n7977t1694KijUnr7yMTjcWKxWNRl5AXti/20L/bTvtivueHe5DGtu881s66NDBkFPJwYu9DM2phZ\nO3ffnGwR+/aFeeIrV352e+MNOP74ENy9esE114Qg79q1sC8E0jfuftoX+2lf7Kd9kb5MNCw6Autr\nPd8AdAIOCPfq6jBHfNWq/dvKlfD66+EEZ8+eYb74qFFw883has7DDstAhSIiJSZT3ei6/2Sot//S\nunVonfTsGbZYDK69Njxu1SpDlYiISHKzZRJtmekN9NzvA+LuPjnxvBIYXLctY2b5McldRKTAZKXn\nnoRpwHXAZDMrA7bX129vTnEiItI8TYa7mU0CBgNtzWw98DOgJYC7T3D3GWY23MzWAR8DV2WzYBER\naVrOLmISEZHcyfiEQjMbZmaVZrbWzG5qYMz4xOvLzKxvpmvIF03tCzP7VmIfLDezl82sdxR15kIy\n3xeJcWeYWZWZXZTL+nIpyZ+RmJlVmNlKM4vnuMScSeJnpK2ZzTKzpYl9cWUEZWadmT1oZpvNbEUj\nY1LLTXfP2Aa0ANYBXQmtm6XAyXXGDAdmJB4PABZksoZ82ZLcF2cCRyYeDyvlfVFr3AvA08DFUdcd\n4fdFG2AV0CnxvG3UdUe4L8qB22r2A/ABcHDUtWdhXwwC+gIrGng95dzM9JF7f2Cdu7/t7nuBycCF\ndcZ85qInoI2ZtctwHfmgyX3h7q+4+47E04WE6wOKUTLfFwDXA1OArbksLseS2RffBJ5w9w0A7r4t\nxzXmSjL7YhPQOvG4NfCBu1flsMaccPe5wEeNDEk5NzMd7vVd0NQxiTHFGGrJ7IvavgvMyGpF0Wly\nX5hZR8IP9r2JLxXryaBkvi+6AUeb2YtmttjMrshZdbmVzL64H+hpZu8By4AbclRbvkk5NzO9pFay\nP5BJXfRU4JL+bzKzIcB3gLOyV06kktkXdwA/cnc3M+PA75Fikcy+aAmcBnwZOBx4xcwWuPvarFaW\ne8nsi5uBpe4eM7MvAM+aWR9335nl2vJRSrmZ6XDfCHSu9bwz4TdMY2M6Jb5WbJLZFyROot4PDHP3\nxv5ZVsiS2Rf9CNdKQOitnm9me919Wm5KzJlk9sV6YJu77wH2mNkcoA9QbOGezL4YCPwSwN3fMLO3\ngO7A4pxUmD9Szs1Mt2UWA93MrKuZHQJ8g3CRU23TgNEAjV30VASa3Bdm1gV4Erjc3ddFUGOuNLkv\n3P0Edz/e3Y8n9N3HFmGwQ3I/I1OBs82shZkdTjiBtjrHdeZCMvuiEjgXINFj7g68mdMq80PKuZnR\nI3d3rzKz64DZhDPhv3P3NWY2JvF6yVz0lMy+AH4KHAXcmzhi3evu/aOqOVuS3BclIcmfkUozmwUs\nB6qB+9296MI9ye+LXwEPmdkywsHoD939w8iKzpJsXCyqi5hERIpQAa+KLiIiDVG4i4gUIYW7iEgR\nUriLiBQhhbuISBFSuIuIFCGFu4hIEVK4i4gUof8H9/lpC4wyJXoAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(xs, 1 - sp.log(1 - xs));"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Ako $y=1$, funkcija kažnjava model to više što je njegov izlaz manji od jedinice. Slično, ako $y=0$, funkcija kažnjava model to više što je njegov izlaz veći od nule\n",
"\n",
"\n",
"* Intutivno se ovakva funkcija čini u redu, ali je pitanje kako smo do nje došli"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Izvod\n",
"\n",
"* Funkciju gubitka izvest ćemo iz **funkcije pogreške**\n",
" * Podsjetnik: funkcija pogreške = očekivanje funkcije gubitka\n",
" \n",
"\n",
"* Budući da logistička regresija daje vjerojatnosti oznaka za svaki primjer, možemo izračunati kolika je vjerojatnost označenog skupa primjera $\\mathcal{D}$ pod našim modelom, odnosno kolika je izglednost parametra $\\mathbf{w}$ modela\n",
"\n",
"\n",
"* Želimo da ta izglednost bude što veća, pa ćemo funkciju pogreške definirati kao **negativnu log-izglednost** parametara $\\mathbf{w}$:\n",
"$$\n",
"E(\\mathbf{w}|\\mathcal{D}) = -\\ln\\mathcal{L}(\\mathbf{w}|\\mathcal{D})\n",
"$$\n",
"\n",
"\n",
"* Želimo maksimizirati log-izglednost, tj. minimizirati ovu pogrešku\n",
"\n",
"\n",
"* Log-izglednost:\n",
"$$\n",
"\\begin{align*}\n",
"\\ln\\mathcal{L}(\\mathbf{w}|\\mathcal{D})\n",
"&= \\ln p(\\mathcal{D}|\\mathbf{w})\n",
"= \\ln\\prod_{i=1}^N p(\\mathbf{x}^{(i)}, y^{(i)}|\\mathbf{w})\\\\\n",
"&= \\ln\\prod_{i=1}^N P(y^{(i)}|\\mathbf{x}^{(i)},\\mathbf{w})p(\\mathbf{x}^{(i)})\\\\\n",
"&= \\sum_{i=1}^N \\ln P(y^{(i)}|\\mathbf{x}^{(i)},\\mathbf{w}) + \\underbrace{\\color{gray}{\\sum_{i=1}^N \\ln p(\\mathbf{x}^{(i)})}}_{\\text{ne ovisi o $\\mathbf{w}$}}\n",
"\\end{align*}\n",
"$$\n",
"\n",
"\n",
"* $y^{(i)}$ je oznaka $i$-tog primjera koja može biti 0 ili 1 $\\Rightarrow$ **Bernoullijeva varijabla**\n",
"\n",
"\n",
"* Budući da $y^{(i)}$ Bernoullijeva varijabla, njezina distribucija je:\n",
"$$\n",
"P(y^{(i)}) = \\mu^{y^{(i)}}(1-\\mu)^{y^{(i)}}\n",
"$$\n",
"gdje je $\\mu$ vjerojatnost da $y^{(i)}=1$\n",
"\n",
"\n",
"* Naš model upravo daje vjerojatnost da primjer $\\mathcal{x}^{(i)}$ ima oznaku $y^{(i)}=1$, tj.:\n",
"$$\n",
"\\mu = P(y^{(i)}=1|\\mathbf{x}^{(i)},\\mathbf{w}) = \\color{red}{h(\\mathbf{x}^{(i)} | \\mathbf{w})}\n",
"$$\n",
"\n",
"\n",
"* To znači da vjerojatnost oznake $y^{(i)}$ za dani primjer $\\mathbf{x}^{i}$ možemo napisati kao:\n",
"\n",
"$$\n",
"P(y^{(i)}|\\mathbf{x}^{(i)},\\mathbf{w}) = \n",
"\\color{red}{h(\\mathbf{x}^{(i)}|\\mathbf{w})}^{y^{(i)}}\\big(1-\\color{red}{h(\\mathbf{x}^{(i)}|\\mathbf{w})}\\big)^{1-y^{(i)}}\n",
"$$\n",
"\n",
"\n",
"* Nastavljamo s izvodom log-izglednosti:\n",
"\n",
"$$\n",
"\\begin{align*}\n",
"\\ln\\mathcal{L}(\\mathbf{w}|\\mathcal{D}) \n",
"&= \\sum_{i=1}^N \\ln P(y^{(i)}|\\mathbf{x}^{(i)},\\mathbf{w}) \\color{gray}{+ \\text{konst.}}\\\\\n",
"&\\Rightarrow \\sum_{i=1}^N\\ln \\Big(h(\\mathbf{x}^{(i)}|\\mathbf{w})^{y^{(i)}}\\big(1-h(\\mathbf{x}^{(i)}|\\mathbf{w})\\big)^{1-y^{(i)}}\\Big) \\\\\n",
"& = \\sum_{i=1}^N \\Big(y^{(i)} \\ln h(\\mathbf{x}^{(i)}|\\mathbf{w})+ (1-y^{(i)})\\ln\\big(1-h(\\mathbf{x}^{(i)}|\\mathbf{w})\\big)\\Big)\n",
"\\end{align*}\n",
"$$\n",
"\n",
"\n",
"* Empirijsku pogrešku definiramo kao negativnu log-izglednost (do na konstantu):\n",
"\n",
"$$\n",
"E(\\mathbf{w}|\\mathcal{D}) = \\sum_{i=1}^N \\Big(-y^{(i)} \\ln h(\\mathbf{x}^{(i)}|\\mathbf{w}) - (1-y^{(i)})\\ln \\big(1-h(\\mathbf{x}^{(i)}|\\mathbf{w})\\big)\\Big)\n",
"$$\n",
"\n",
"* Alternativno (kako ne bi ovisila o broju primjera):\n",
"\n",
"$$\n",
"E(\\mathbf{w}|\\mathcal{D}) = \\color{red}{\\frac{1}{N}} \\sum_{i=1}^N\\Big( - y^{(i)} \\ln h(\\mathbf{x}^{(i)}|\\mathbf{w})- (1-y^{(i)})\\ln \\big(1-h(\\mathbf{x}^{(i)}|\\mathbf{w})\\big)\\Big)\n",
"$$\n",
"\n",
"$\\Rightarrow$ **pogreška unakrsne entropije** (engl. *cross-entropy error*)\n",
"\n",
"\n",
"* Iz pogreške možemo iščitati funkciju gubitka:\n",
"\n",
"$$\n",
"L(h(\\mathbf{x}),y) = - y \\ln h(\\mathbf{x}) - (1-y)\\ln \\big(1-h(\\mathbf{x})\\big)\n",
"$$\n",
"\n",
"$\\Rightarrow$ **gubitak unakrsne entropije** (engl. *cross-entropy loss*)\n",
"\n",
"\n",
"* **NB:** Izraz kompaktno definira grananje za dva slučaja (za $y=1$ i za $y=0$)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def cross_entropy_loss(h_x, y):\n",
" return -y * sp.log(h_x) - (1 - y) * sp.log(1 - h_x)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEVCAYAAADzUNLBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VWW2wOHfoktHkBAIvQok9CJFQpMiRUEdiog6FOtY\n5s5VGR1xnNHRKVgR0LEwKFiv0pFi6B0SQm8qPYCUoZOy7h/7BEIMcFLO2aes93n2k1O+7L3YJGfl\n66KqGGOMMdeTz+0AjDHGBAdLGMYYY7xiCcMYY4xXLGEYY4zxiiUMY4wxXrGEYYwxxiuuJQwRyS8i\n60Vk2lXef0tEdohIgog08Xd8xhhjruRmDeMJYDPwq4kgItITqKWqtYERwHt+js0YY0wmriQMEYkC\negIfAJJFkT7AJwCquhIoLSIR/ovQGGNMZm7VMMYAfwDSrvJ+JWBvhuf7gChfB2WMMebq/J4wRKQX\ncFhV15N17eJS0UzPbQ0TY4xxUQEXrtkG6OPppygClBSRiap6X4Yy+4HKGZ5HeV67gohYEjHGmBxQ\n1Wv9wZ4lv9cwVHWUqlZW1erAAGBBpmQBMBW4D0BEWgMnVDXpKuezQ5UXX3zR9RgC5bB7YffC7sW1\nj5xyo4aRmQKIyEgAVR2vqjNFpKeI7ATOAA+4GaAxxhiXE4aqLgQWeh6Pz/TeY64EZYwxJks20ztE\nxMbGuh1CwLB7cZndi8vsXuSe5KY9y20iolnF/+TsJ+lYrSN96/V1ISpjjHHXa69BqVLw0ENZvy8i\naA46vQOhDyPPlShUgnUH11nCMCYMiWT7czBkPfwwuerkziwkE0Z0RDRTNk5xOwxjjEuCueUkL+V1\n8gzJPozo8tFsSNrgdhjGGBNSQjJh1C5bmwOnDnDm4hm3QzHGmJARkgmjQL4C1C1Xl01HNrkdijHG\nhIyQTBgAMRExJCYluh2GMcaEjJBNGNHlo0k8bAnDGBM6Lly4wIMPPkipUqWIjIxkzJgxfr1+SI6S\nAidhzNwx0+0wjDEmz4wePZpdu3axZ88eDh48SMeOHalfvz7dunXzy/VDt4YR4dQwbHidMSYQ/P3v\nf+euu+664rXf/e53PPnkk16fY+LEibzwwguUKlWKevXqMWLECD7++OM8jvTqQjZhRBaPJE3TSDqT\n5SK3xhjjV0OGDGH27NmcPHkSgJSUFD7//HOGDh3KI488QpkyZbI8GjduDMDx48c5ePAgjRo1unTO\nmJgYNm3y3+CekE0YImId38aYXxHJmyO7KlSoQPv27fnyyy8BmD17NjfddBNNmjRh7NixHD9+PMsj\nPj4egNOnTwNQqlSpS+csWbIkp06dyv1N8VLIJgywCXzGmF9TzZsjJ4YOHcqkSZMAmDRpEkOGDPH6\ne4sXLw7Af//730uvnTx5khIlSuQsmBwI+YRhI6WMMYGib9++bNiwgY0bNzJjxgwGDx4MwEMPPUSJ\nEiWyPKKjowEoU6YMkZGRl2ocAAkJCTRs2NBv8YfkarXpVuxbwaMzH2XtiLV+jMoY4ybPSqxuh3FV\nw4cPZ+XKlZQvX5558+Zl63ufe+45li9fzrfffsvBgwfp1KkTn3zyCbfddluW5a92L3K6Wm1I1zAa\n3NSALUe2kJqW6nYoxhgDOM1SGzduzFZzVLqXXnqJmjVrUrVqVTp27Mgzzzxz1WThCyFdwwCo+VZN\nZg6aSd1ydf0UlTHGTYFew9i7dy/16tUjKSnpUr+Er1gNI5us49sYEyjS0tL45z//ycCBA32eLHzB\nlYQhIkVEZKWIxIvIZhF5NYsysSJyUkTWe47nc3It6/g2xgSCM2fOULJkSebPn89LL73kdjg54srS\nIKp6XkQ6qupZESkALBGRdqq6JFPRharaJzfXio6IZvLGybk5hTHG5FqxYsUuzaUIVq41SanqWc/D\nQkB+4FgWxXK9XZRN3jPGmLzhWsIQkXwiEg8kAT+o6uZMRRRoIyIJIjJTROrn5Dq1bqzFgVMHOH0x\nuDO7Mca4zc0aRpqqNgaigFtFJDZTkXVAZVVtBLwNfJuT6xTIV4B65eqx6bBtpmSMMbnh+vLmqnpS\nRGYAzYG4DK+fyvB4loiMFZEbVfWKpqvRo0dfehwbG0tsbOyvrpG+cm2rqFZ5Hr8xxgS6uLg44uLi\ncn0eV+ZhiEg5IEVVT4jIDcAc4CVVnZ+hTARwWFVVRFoCX6hqtUznue48DIB/LPsHe0/u5c0eb+bp\nv8MYE3gCfR6GP+X1PAy3ahiRwCcikg+nWew/qjpfREYCqOp44C7gYRFJAc4CA3J6sZiIGNtMyRhj\ncinkZ3oDHDx1kOj3ojnyhyNITtYlNsYEjVCuYXzxxRe88cYbJCQk0LJlS3744YdrlreZ3jlQoXgF\nAA6dPuRyJMYYk3Nly5bl6aef5tlnn3Xl+mGRMETkUse3Mca4IS+2aO3cuTN33XUXkZGReR2eV8Ii\nYYBniRCbwGeMcUlut2gNBK4Pq/WXmIgYluzJvPKIMSbcyEt504+pL2avnyTjFq3Dhg371RatY8eO\nzZO4fClsEkZ0+WjGrg78/xBjjG9l94M+Lw0dOpRx48YxbNiwbG/RGgjCpkmqQfkGbD26lZS0FLdD\nMcaEqdxs0ZqRW6M9wyZhFC9UnMgSkew8ttPtUIwxYeqGG26gf//+DBo0iFatWhEVFQXAuHHjOHXq\nVJZHYuLlvte0tDTOnz9PcnIyaWlpXLhwgeTkZL/FHzYJA6zj2xjjvtxs0Tpx4kSKFi3KI488wuLF\ni7nhhhsYOXKkD6LMWlhM3Ev3px/+hKrycqeXfRiVMcZNgT5xz7ZoDRK2+54xxk3BvkVr2IySAs+q\ntfMtYRhj/O/MmTNERERQvXp1Zs+e7XY4ORJWCaPWjbU4eOogpy+epnih4MvuxpjgZVu0BpkC+QoQ\nExHDqv2r3A7FGGOCTlglDIDba9/O9O3T3Q7DGGOCTtgljN51ezN129SAHkVhjDGBKKz6MAAaRTTi\nYupFth7dys033ex2OMYYH7B9b3wj7GoYIkLvOr2Ztn2a26EYY3xAVUP+GDxYGTfOu7J5KewSBlxu\nljLGmGCTkgKzZkGvXv6/dlgmjNhqsSQeTuTo2aNuh2KMMdmydClUqwaVKvn/2n5PGCJSRERWiki8\niGwWkVevUu4tEdkhIgki0iQvYyhSoAhdanRh5o6ZeXlaY4zxuWnToE8fd67t94ShqueBjqraGIgB\nOopIu4xlRKQnUEtVawMjgPfyOo7edaxZyhgTfKZOhd693bm2K01SqnrW87AQkB84lqlIH+ATT9mV\nQGkRicjLGHrW7sm83fO4kHIhL09rjDE+s20bnD0LTfK0zcV7riQMEcknIvFAEvCDqm7OVKQSsDfD\n831AVF7GUL5YeRqUb8DCnxfm5WmNMcZnpk1zahdujRp2ZR6GqqYBjUWkFDBHRGJVNS5Tscy3JMvx\nYaNHj770ODY2ltjYWK/jSG+Wuq3mbV5/jzHGuGXqVHj22ex/X1xcHHFxcbm+vuv7YYjIC8A5Vf1H\nhtfGAXGqOsXzfCvQQVWTMn1vtvbDyGzT4U30/KwnPz3xk030McYEtF9+gRo1ICkJihTJ3bmCZj8M\nESknIqU9j28AugLrMxWbCtznKdMaOJE5WeSF+jfVJ7/ktz0yjDEBb9Ys6Ngx98kiN9zow4gEFnj6\nMFYC01R1voiMFJGRAKo6E9gtIjuB8cAjvghEROhTtw/Tttmsb2NMYHNzOG0615ukciO3TVIA83fP\nZ9SCUawctjKPojLGmLx18SJERMDWrc7X3AqaJqlA075qe7b/sp1Dpw+5HYoxxmRp0SKoWzdvkkVu\nhH3CKJS/EN1qdmPG9hluh2KMMVkKhOYosIQBeIbXbrdZ38aYwKPq7uzujCxhAD1q9+CHH3/gXPI5\nt0MxxpgrbNrkJI2GDd2OxBIGADfecCNNI5uy4McFbodijDFXSG+OCoSpYpYwPGwxQmNMIAqU5iiw\nhHFJ77q9mb5jOmma5nYoxhgDOLO6t2yBDh3cjsRhCcOjTtk6RBaPZNaOWW6HYowxAHz4IfTrB4UK\nuR2JwxJGBo+3fJy3V73tdhjGGENKCrz3Hjz+uNuRXGYJI4PfNPwN6w+tZ9vRbW6HYowJc999B1Wq\nuLf3RVYsYWRQpEARhjUZxrur33U7FGNMmHv77cCqXYCtJfUre0/updG4Rvz85M+UKFwiT89tjDHe\nSEyE7t3hp5+gYMG8P7+tJZVHKpeqTKfqnZiYMNHtUIwxYeqdd2DkSN8ki9ywGkYWFv60kIdmPMTm\nRzbbxkrGGL86ftzZKGnLFqhQwTfXsBpGHrq16q0UzFeQebvnuR2KMSbMfPgh3H6775JFbljCyIKI\n2BBbY4zfpabCu+8GXmd3OksYVzE4ZjDL9i5j9/HdbodijAkTM2dCuXLQqpXbkWTNEsZVFC1YlAca\nP8DY1WPdDsUYEyYCcShtRtbpfQ0/Hv+RFu+34Ocnf6ZYoWI+u44xxmzdCrGx8PPPULiwb68VVJ3e\nIlJZRH4QkU0islFEfpdFmVgROSki6z3H8/6Os3qZ6rSt0pZPEz/196WNMWHmnXdg+HDfJ4vccKWG\nISIVgAqqGi8ixYG1wB2quiVDmVjgaVW96saEvq5hAMzbPY+n5jzFhoc22BBbY4xP/Pe/UK2aM2Gv\nUiXfXy+oahiqekhV4z2PTwNbgIpZFHX9E7pz9c6kpqWy8OeFbodijAlRH38MXbv6J1nkhuud3iJS\nDWgCrMz0lgJtRCRBRGaKSH1/xwZOJn6s5WO8tfItNy5vjAlxqalOc1Qgd3anK+DmxT3NUV8BT3hq\nGhmtAyqr6lkR6QF8C9TJfI7Ro0dfehwbG0tsbGyexzm00VD+sugvrD2wlmYVm+X5+Y0x4WvSJLjp\nJmjb1nfXiIuLIy4uLtfncW2UlIgUBKYDs1T1DS/K/wg0U9VjGV7zeR9GuglrJzBl4xTm3zff+jKM\nMXni3DmoWxemTIE2bfx33aDqwxDnE/ffwOarJQsRifCUQ0Ra4iS3Y1mV9YcHmzzIwdMHmbXTduQz\nxuSNN9+EFi38myxyw61RUu2ARcAGnL4KgFFAFQBVHS8ijwIPAynAWZwRUysyncdvNQyAqdumMmr+\nKBIeSiB/vvx+u64xJvQcPQr16sGyZVDnV43tvpXTGoZN3MsGVSX2k1jui7mP3zb9rd+ua4wJPU88\ncbnD298sYfjJqv2r6Pd5P7Y9ts1mfxtjcmTnTmjdGjZvhvLl/X/9oOrDCGYtK7WkXZV2vLHiuv30\nxhiTpT/+EZ56yp1kkRtWw8iB3cd30/L9lmx+dDPliwXZ/7gxxlUrV0L//rB9OxQt6k4M1iTlZ0/N\nforktGTe6elCA6QxJiipQocOcP/98OCD7sVhTVJ+9vytz/P5ps/Z/st2t0MxxgSJqVOdLViHDnU7\nkpzxuoYhIsWAwUBDID9QBEgDTgMrgC9VNc1HcV4tJtdqGACvL32dlftX8vU9X7sWgzEmOKSkQMOG\nMGYM9Ojhbiw+bZISka5AfWC6qu7K9J4AjYAuwLz0RQX9we2EcS75HHXfqcvk/pNpW8WH8/qNMUFv\n3Dj46iuYOxfcXizCZwlDRIoAUaq604sgolU1MbtB5JTbCQNg0oZJjFkxhhW/XUHB/AVdjcUYE5iO\nHIGYGJgxA5o2dTsaP3Z6i8jHwBFgKbBcVZOye9G8EggJQ1Xp8WkP2lVpx/O3+n2PJ2NMELjnHme/\ni9dfdzsSh19HSYnIzUBrz9EM+AL4R7j1YaTb9999NB3flLlD5tKoQiO3wzHGBJAvvoAXX4T166FI\nEbejcfizhtHa833LPc/vBhKAW1X1g+wGkBuBkjAAPo7/mDdXvsnKYSsplL+Q2+EYYwJAUhI0auSM\njmrZ0u1oLvPnsNouwK0i8rmIfIQzaqoS4FrTVCAY2mgolUpU4pXFr7gdijEmAKjCww878y0CKVnk\nRk5qGA2Boqq6KsNrw4C9qjonj+O7XiwBU8MAOHDqAI3HNWb2vbNpGhkAPVvGGNd89hm88gqsXQuF\nC7sdzZVspneAmLRhEq8vfZ3Vw1dTuECA/ZQYY/zi4EFo3BhmzoRmAbhJpyszvUWkuojsEpFYEemW\nm3OFisHRg6lRpgYvL3rZ7VCMMS5QhZEjnSMQk0Vu5LqGISKVVHV/HsWT3WsHXA0D4NDpQzQa14jp\nA6fTolILt8MxxvjRxInwz3/C6tVQKEDHv1iTVICZsnEKLy96mbUj1lKkQICMpTPG+NT+/dCkCXz/\nvdMkFah8njBsLansUVXu/vJuokpG8UZ32zvDmFCXmuqsEdW2rTPvIpDZWlIB6Pi54zR/vzkvd3yZ\nQdGD3A7HGONDzz3nNEPNng0FCrgdzbW5tpaUiJRU1f96Hnu1lpSIVAYmAuUBBSao6ltZlHsL6AGc\nBe5X1fWZ3g/ohAGwIWkDnSd2Zu6QuTSuEMB1VGNMjn31FfzP/8CaNVCunNvRXJ/PRkmp6vmMyUJE\nXsvwOD/wYYay3i48mAw8paoNcJYXedSz3MglItITqKWqtYERwHtenjugxETE8HaPt+n3eT9+OfuL\n2+EYY/LYpk3OBL1vvgmOZJEbORlWe4eIPCMiXYD1QNfsnkBVD6U3XanqaWALUDFTsT7AJ54yK4HS\nIhKRg3hdN6DhAPrd3I9B3wwiNS3V7XCMMXnkxAm4805nVFQgrELrazlJGP8EugHfAjOADrkJQESq\nAU2AlZneqgTszfB8HxCVm2u56W9d/kZKWgrPL7AVbY0JBWlpMGQIdOsG993ndjT+kZOumTeB14He\nOKOmpgD1cnJxESkOfAU84alp/KpIpue/6rAYPXr0pcexsbHExsbmJBSfK5CvAFP6T6HF+y1oXrE5\n/ev3dzskY0wu/PnPcPIk/OtfbkdyfXFxccTFxeX6PDlZS+p3GTuoReSPqvrXbF9YpCAwHZilqr8a\ndyoi44A4VZ3ieb4V6JBx/41g6PTObO2BtXT/tDtxQ+NoUL6B2+EYY3Jg6lR49FFnVFSFCm5Hk30+\n6/QWkcIicqkrJ/NopozJQkSqeHNRz1DcfwObs0oWHlOB+zzlWwMn3NysKa80q9iMf3T9B3d+ficn\nzp9wOxxjTDZt3w7DhsGXXwZnssgNb+dh9AJKAv+nqueyeL8McDewRVUXe3G+dsAiYAOXm5lGAVUA\nVHW8p9w7QHfgDPCAqq7LdJ6gq2Gke3L2k8Qfimf2vbNtJrgxQeLgQWjXDkaNgt/+1u1ocs4fM70j\ngQdw5k4UAQoCqThzJPYB76vqyewGkBvBnDDSNI1BXw/iQuoFvrz7SwrkC/CZPsaEuRMnoEMHuPtu\neD7Ix67YWlJB6GLqRXpP7k3lkpV5v/f7OC11xphAc+6cMxqqSRN44w0I9l9Vt5Y3b+rpvDY5UCh/\nIb6+52sSDycyav4ot8MxxmQhJQUGDIDKlWHMmOBPFrmR7YQhIoNEZIyIDAIOA0PyPqzwUbxQcWYM\nmsG3277lX8uDYHyeMWFEFYYPh4sX4aOPIF+u/sQOfjn556cCLwMngGeBoJx9HUjKFS3H9/d+z5sr\n32RiwkS3wzHGePzv/8LWrc5aUYG6t4U/edXTKiJLgVXAGpwZ2PlUdSYw04exhZXKpSoze/BsOn7S\nkRtvuJFedXq5HZIxYe3vf3e2WF28GIoVczuawODtsNo+wA7gFpzFAusBx4DlwA+qusqXQV4jrqDu\n9M7Kqv2r6PVZLyb3n0znGp3dDseYsDRuHPztb7BkCUQF7YJEV+f3UVKeZT1aAPVU1ZWVZEMxYQAs\n/nkx/b/oz0d9P+L2Ore7HY4xYWXMGHjrLZg3D2rWdDsa3/DlfhiFgRKqetSLIKqo6p7sBpFToZow\nAFbuW0mfKX0Y23OsrTtljJ/89a/w8ccwfz5U8WrdiuCU04Rx3T4MVb0gIl1FxKuZ3oDfEkYoaxXV\nitmDZ9Pzs56cTznP4JjBbodkTMhSdSbjffstLFoEkZFuRxSYbKZ3gNt0eBO3TbqNP8f+md82DeK1\nCIwJUKrw+9/DggUwdy7cdJPbEfmeX/swRCQfzp4YK1T1eLZPkEfCIWEAbP9lO10mduEPbf7A460e\ndzscY0JGWpqz6uy6dc5e3GXKuB2Rf/h0pnfG1WoBVDUNmA/cKSIzsntRkz11ytZh4f0LGbNiDK8t\nee3632CMua6UFHjwQdi40alZhEuyyA1vJ+7dk/kFVb2oqh8C2/M2JJOV6mWqs+iBRXyS8AlPzHrC\ntno1JhdOnYK+fZ3VZ2fPhpIl3Y4oOHibMF4Wka9E5H9FJNYzpDbdJl8EZn4tqmQUSx9cSuLhRO78\n/E5OX8xqk0JjzLXs2wft20PFijB9uk3Kyw5vE8Yfgb/h7EtxP7BcRBJF5COc/SqMn5S5oQyz751N\n2aJl6fBxBw6cOuB2SMYEjfh4uOUWGDgQJkyAgrZ0arbkZuJeSZyJe0+oap88jcr7GMKi0zsrqsqr\nS15l/NrxTBs4jZiIGLdDMiagzZgB998PY8c6e1qEM9f2wxCRFqq6Olcnyfm1wzZhpJuycQq/m/U7\n/nPnf+hWq5vb4RgTkN59F/7yF/jmG6eGEe5sA6UwtnTPUvp/0Z/RsaN5qPlDbodjTMBISXFWnJ05\n0zlq1HA7osBgCSPM7Ty2kz6T+3BL1C280/Mdbih4g9shGeOqw4edvgoR+PJLGzabkSs77uWUiHwo\nIkkikniV92NF5KSIrPccQb6Dru/VurEWq4av4nTyadp91I6fTvzkdkjGuGbFCmjeHFq1gjlzLFnk\nFbf2j/JmdNVCVW3iOf7ij6CCXfFCxZnSfwpDYobQ+oPWzN452+2QjPErVadTu08fePtteOUVyJ/f\n7ahCh1cbKOU1VV0sItWuUyyMd87NORHhydZP0rxicwZ8NYDhTYfzQocXyCdhvrekCXlnz8LIkZCQ\nAMuWQa1abkcUegL1U0SBNiKSICIzRaS+2wEFm3ZV2rF6+Grm/zifXp/14ti5Y26HZIzP7NwJrVs7\nj1essGThK67UMLywDqisqmdFpAfwLVAnq4KjR4++9Dg2NpbY2Fh/xBcUIktEMv+++Tw771majG/C\nf+78D7dWvdXtsIzJU59+Ck8+CS+9BA8/7HRymyvFxcURFxeX6/O4NkrK0yQ1TVWjvSj7I9BMVY9l\net1GSXlp5o6ZDJs6jAcaP8Do2NEUzG9TXE1wO3HCWWl2/Xr47DNo3NjtiIJHUI2Suh4RiRBx/k4Q\nkZY4ic3aVHKhZ+2erB+5nvikeNp+2JYdv+xwOyRjcmzxYidBlCkDa9ZYsvAXt4bVTgaWAXVFZK+I\nPCgiI0VkpKfIXUCiiMQDbwAD3Igz1EQUj2D6wOnc3/h+2nzYhn+v+zdWQzPBJDkZXngB7rkH3nnH\nOYoWdTuq8GET98LU5iObGfT1IGreWJMJvSZQtmhZt0My5pp27oTBg6FsWfjoI4iIcDui4BVSTVLG\n9+rfVJ+Vw1ZSvXR1ot+L5uvNX7sdkjFZSk2FMWOcNaDuvddZRNCShTushmFYtncZD373IA3LN+Sd\nnu9QoXgFt0MyBoDNm+G3v4XCheGDD2y4bF6xGobJsTaV2xD/UDx1ytah0bhGTEyYaH0bxlXJyc7q\nsh06wNChsGCBJYtAYDUMc4V1B9fx4HcPUrFERcb3Gk/lUpXdDsmEmfXrnb22K1SA8eOhShW3Iwo9\nVsMweaJpZFNWD19Nm8ptaDqhKW+vfJuUtBS3wzJh4PRpeOYZ6N4dnnrKWY7ckkVgsRqGuaotR7bw\nyMxHOHH+BO/d/h6to1q7HZIJQarOxkZPPeU0Qf39707twviO7YdhfEJVmbxxMn+Y+wd61OrB37r8\njXJFy7kdlgkRO3bA44/D3r3OKrMdOrgdUXiwJinjEyLCoOhBbH5kM8ULFafB2AZMWDuBNE1zOzQT\nxM6dgz/9yRkq26ULxMdbsggGVsMw2ZJwKIFHZj5CSloKb3V/i1ZRrdwOyQQRVfjuO3j6aWeDo3/9\nC6Ki3I4q/FiTlPGbNE3jPwn/YdSCUXSo2oFXO79K1dJV3Q7LBLh165xEcfSoMxGva1e3Iwpf1iRl\n/Caf5GNo46Fsf2w7dcrWoemEpoyaP4pTF065HZoJQPv3w/33w+23w6BBTvOTJYvgZAnD5FixQsUY\nHTuahIcSOHDqAHXeqcP7a98nNS3V7dBMADhzxtmjIiYGIiNh2zYYMQIKBOouPOa6rEnK5Jm1B9by\n9PdPc+zcMV7t/Cq3174dsd1swk5KCnz8MYweDe3bw6uvQrVqLgdlrmB9GCYgqCpTt03ljwv+SKki\npXi186u2y1+YSEuDr76C5593OrJfeeXytqkmsFjCMAElNS2VzxI/409xf6JeuXq80ukVmkQ2cTss\n4wOqMHs2/PGPkC+fU6Po0sW2Sg1kljBMQLqYepH3177PXxf/lfZV2/Nyx5epUzbL7dlNEFq2DJ57\nDg4fdhYL7NfPEkUwsFFSJiAVyl+IR1s+yo7Hd9A4ojFtP2zLkP8bwraj29wOzeTC0qVw220wcKAz\nAioxEfr3t2QR6qyGYfzq5PmTvL3qbd5c+Sa31byN59s/z8033ex2WMZLixbBn/8Mu3bBqFHO0uOF\nCrkdlckua5IyQeW/F/7Lu6veZcyKMXSq3okXbn2BBuUbuB2WuYq4OGeI7J49Tl/FkCFQsKDbUZmc\nCqomKRH5UESSRCTxGmXeEpEdIpIgItZbGmJKFi7Jc+2fY9fvdtE0simdJnbiri/uYvX+1W6HZjxU\nnSXGb70Vhg93ahNbtzp7VViyCE+u1DBEpD1wGpioqtFZvN8TeExVe4pIK+BNVf3VAD2rYYSOMxfP\n8P669/nX8n9Rp2wdnmn7DF1qdLF5HC5ISYEvvoDXXnOSxjPPwG9+YxPuQknQNUmJSDVg2lUSxjjg\nB1X93PN8K9BBVZMylbOEEWIupl5kcuJkXl/2OkUKFOGZts/Q/+b+5M+X3+3QQt7Zs/DRR/CPfzgb\nFz37rLOZkeXs0BNUTVJeqATszfB8H2BrWoaBQvkLMbTxUBIfTuTFDi/yxoo3qPduPd5b/R5nLp5x\nO7yQdOQvFeP3AAAWnUlEQVQIvPwy1KgBc+fCZ5/BwoXQo4clC3OlQK5kZv5RzbIqMXr06EuPY2Nj\niY2N9V1Exm/yST761O1D7zq9WbJnCf9c/k/+FPcnhjUZxmMtH6NSyUpuhxj0Nm2CN95wZmf37w8L\nFkD9+m5HZXwhLi6OuLi4XJ8nkJuk4lR1iue5NUkZdh7byVsr32LShkn0qN2Dp1o/RfOKzd0OK6io\nwpw5zvLiGzbAww/DQw9B+fJuR2b8KdT6MDJ2ercG3rBOb5PuxPkTfLDuA95e9TZVSlXhiVZP0Ldu\nXwrmt6E7V3P6NHz6Kbz5pjPC6amnnEl3hQu7HZlxQ1AlDBGZDHQAygFJwItAQQBVHe8p8w7QHTgD\nPKCq67I4jyWMMJaSlsI3W77hnVXvsPv4bkY0G8GIZiOoULyC26EFjG3bnL2yJ01yhsc+/jh07Gh9\nE+EuqBJGXrGEYdJtSNrA2NVj+XzT53Sv1Z1HWzxK28ptw3JYbmoqTJ8O774LCQkwbBiMHOmMfDIG\nLGEYAzjNVZ/Ef8LYNWO5ocANjGw2kkHRgyhVpJTbofncgQPOsNgJE6BiRXj0Ubj7bmt2Mr9mCcOY\nDNI0jfm75zNh3QTm7Z5Hv3r9GNFsBC0rtQypWkdqKnz/vZMk4uLgnnucXe2aNXM7MhPILGEYcxVJ\np5P4OP5jJqybQPFCxRnRdASDYwZTukhpt0PLsf374cMP4YMPnBFOI0bAgAFQooTbkZlgYAnDmOtI\n0zR++PEHJqybwJydc+hdtzcPNn6QDtU6kE8CdQ7rZRcvwrRpTrPTsmXOch3Dh0PTpm5HZoKNJQxj\nsuHImSN8mvgpH67/kFMXT3F/o/sZ2ngo1UpXczu0X0lIcJLEp59CgwbwwANw111QrJjbkZlgZQnD\nmBxQVdYdXMdH8R8xZeMUGlVoxAONH+DOendSrJB7n8hHj8LkyU6iOHrU2aRo6FCoWdO1kEwIsYRh\nTC6dTznP1G1T+Tj+Y5bvW06fun24N/peOlXv5JfFD8+fd5qc/vMfZ6Oi2293EkWnTpDf1l40ecgS\nhjF5KOl0EpM3TmbShkkcPH2QQQ0HMaTREGIiYvL0OmlpsHixM7Hu66+d/oghQ5y9sa0D2/iKJQxj\nfGTzkc1M2jCJSRsmUbpIaQY2HMiAhgOoXqZ6js6n6vRLTJ4MU6ZAqVJOkhg4EKJsTWbjB5YwjPGx\nNE1jyZ4lTE6czFdbvqLWjbUY1HAQ9zS4h4jiEdf9/p07nSTx2WdO89OAAU6SiMnbSosx12UJwxg/\nSk5NZt7ueUzeOJlp26fRvGJzBjQYwB317qBs0bKXyv38s7N8+JQpsHevM7Fu4EBo3drWczLusYRh\njEvOJZ9jxo4ZfL7pc77f9T2Nyramwi93s2vGnfy8pSx33OHUJmJjbZtTExgsYRjjop9+cjqtp3xz\nhq2pMyjX4UsOl/ietlVac0/Du7mj3h2UK1rO7TCNASxhGON3W7bAN984x5490Levs9hfp07OnhNn\nLp5hxo4ZfLn5S77f9T3NIpvR7+Z+3FHvDqJKWu+2cY8lDGN8TBXWrbucJE6dgjvvdIbAtm9/7eam\ns8lnmbtrLt9s/Ybp26dT+8ba9Lu5H/1u7ketG2v57x9hDJYwjPGJixdh4UL47juYOtVZKrx/fydJ\nNG8O+XKwBFVyajJxP8XxzZZv+Hbbt9x4w430rduXvnX70qJSi6BY18oEN0sYxuSRkydh1iwnScye\nDXXrOs1NffvCzTfn7eimNE1j1f5VfLf1O77b9h0nzp+gd53e9K3Xl07VO1GkQJG8u5gxHpYwjMmF\nXbucXeqmT4eVK50mpr59oXdviIz0Xxw7ftnBd9uc5LEhaQOdq3emV51e9Kzd07aeNXnGEoYx2ZCc\n7CwRnp4kTpxw1m7q1Qu6dIHixd2O0FlRd/bO2UzfMZ3vd31P7Rtr06tOL3rV6UWTCk1CaiMo419B\nlzBEpDvwBpAf+EBVX8v0fizwHbDb89LXqvqXTGUsYRivHTzoNDHNmgXz5kGNGk6C6NXLWcMpJ/0R\n/pKcmsySPUuYvn0607ZP4/TF0/So1YMetXvQtUbXsNiC1uSdoEoYIpIf2AZ0AfYDq4GBqrolQ5lY\n4GlV7XON81jCMFeVkuI0L82aBTNnwo8/Qteu0KMHdO/u36amvLbjlx3M2jmLWTtnsWTPEppGNqVn\nrZ70qN2D6PLRVvsw1xRsCeMW4EVV7e55/iyAqv4tQ5lY4Peq2vsa57GEYa6wdy/MmeMc8+dD1apO\ngujRA265JTRnWp9NPkvcT3HM2jGLmTtncj7lPLfVvI1uNbvRtUbXK5YqMQaCL2HcBXRT1eGe5/cC\nrVT18QxlOgDfAPtwaiH/o6qbM53HEkaYO3vW2TsiPUkcOeLUIrp1g9tuC+5aRE6oKjuP7WTOrjnM\n2TWHhT8tpF65enSr2Y1utbrRqlIrCuYv6HaYxmU5TRhu/b3lzaf8OqCyqp4VkR7At0CdzIVGjx59\n6XFsbCyxsbF5FKIJRKmpsH49zJ3rHKtWOf0P3brBxImB3xfhayJC7bK1qV22No+1fIwLKRdYtncZ\nc3bN4fFZj7P7+G46VO1A1xpd6VqzK3XL1rXmqzAQFxdHXFxcrs/jVg2jNTA6Q5PUc0Ba5o7vTN/z\nI9BMVY9leM1qGCFOFXbvdjqp582DBQugQgVnJFPXrtChg200lB2HzxxmwY8LmLtrLnN3z0VRutTo\nQpfqXehco7MN3Q0TwdYkVQCn07szcABYxa87vSOAw6qqItIS+EJVq2U6jyWMEHTggJMYFixw+iGS\nk531mbp2dRJFpUpuRxgaVJUdx3ZcSh4Lf15IxRIV6VStE51rdKZD1Q6UuaGM22EaHwiqhAHgaWZK\nH1b7b1V9VURGAqjqeBF5FHgYSAHO4oyYWpHpHJYwQsCRI87yGz/84CSJw4edpcA7dXKOevVs7wh/\nSE1LZf2h9Sz4cQHzf5zPsr3LqFu2Lp2rd6Zj9Y60rdyWEoWtOhcKgi5h5AVLGMHp6FGnozouzkkS\ne/ZAu3ZOkujcGRo1gvz53Y7SXEi5wKr9q5j/43zifopjzYE1REdEE1s1lo7VO9KmchuKFwqAGY4m\n2yxhmIB1+LCTIBYtcmoSP/0Ebds6CaJjR2jSJDSHu4aac8nnWLFvBXE/xRH3cxxrD6wlJiKGDlU7\ncGvVW2lbpS0lC5d0O0zjBUsYJmDs23c5OSxa5MywbtfO6aC+9VZo1swSRCg4m3yWFftWsOjnRSz8\neSGr96+mXrl63Fr1VjpU7UC7Ku1sDkiAsoRhXKEKW7fC4sXOsWQJnD7tLN6XniBiYqyJKRxcSLnA\n6gOrLyWQ5XuXU7lUZdpXaU/7Ku1pV6UdVUtXdTtMgyUM4ycXLjibCC1d6iSHJUugZEmnBtG+vXPU\nrWud1AZS0lJIOJTA4j2LWbJnCYv3LKZw/sK0r9qedpXb0bZKWxrc1ID8+eyvCX+zhGF84uhRZ1XX\nZcucJLF+PdSu7fRBtG/vJAob5mq8kT4LPT2BLN27lKTTSbSKakXbym1pW7ktraJaWUe6H1jCMLmW\nlgabN8Py5U6CWL7c6X9o1QratHGSROvWNlHO5J0jZ46wbO8ylu5dytK9S4k/FE/dsnW5JeoW2lRu\nwy2Vb6F66eo2Gz2PWcIw2Xb8uLO0RnqCWLUKbrrJWaQv/YiOtv4H4z/nU86z/uB6lu1dxvJ9y1m+\nbzkpaSmXEkjrqNY0i2xGsULF3A41qFnCMNeUkgKJic5y3ytWOMf+/c6+1K1bOzWI1q2dhGFMoFBV\n9pzc4ySPvctZsX8FGw9vpG7ZurSOak2rSq1oHdWa2mVr217o2WAJw1yi6kyGW7XKSRCrVjl9D1Wq\nOM1LrVs7R/36NrzVBJ/zKeeJPxTPin0rWLFvBSv3r+Tk+ZO0rNTyiqN8sfJuhxqwLGGEsV9+gTVr\nYPVqJzmsWuWMUmrVClq2dI7mzaF0abcjNcY3Dp0+xOr9q1m5fyWr9q9i9YHVlC5S2kkeFVvSolIL\nmkY2tQ51D0sYYeLUKWdY6+rVzrFmjTOSqWlTaNHicpKIirKhrSZ8pWkaO37Zwar9qy4lkMTDiVQv\nXZ0WlVrQoqJzxETEULhAYbfD9TtLGCHozBmIj3eSwtq1zteff3Y6olu0uHzUrRvee0AY442LqRfZ\neHgjq/evZvUB59jxyw5uvulmmkc2p3nF5jSr2IyG5RtSKH8ht8P1KUsYQe70aUhIcBLDunVOcti9\nGxo0cJqTmjd3ltRo0AAK2oZpxuSJs8lnSTiUwJoDa1h7cC1rDqxh9/HdNCzfkGaRzWga2ZRmFZvR\n4KYGIVUTsYQRRE6ccGoO69Y5x9q1Ts2hYUOnaalpUydBNGwIhUL7Dx1jAs6Zi2eIPxTPmgNrWH9o\nPWsPrmXXsV3UK1fvUhJpGtmU6IhoihYs6na4OWIJI0AdPOiMUEo/1q1zVm+NiXFqDOkJon59qzkY\nE6jOJp9lQ9IG1h1cx9oDa1l/aD1bj26lepnqNI1sSpMKTWhSoQmNKzQOik2nLGG4LDUVduxwag4Z\nj+RkZ/nuJk2cxNCkibO0hk2GMya4XUy9yKbDm1h/aD3rD65n3aF1bEjaQLmi5WhcoTGNIxo7Xys0\npkqpKgE1W90Shh+dOgUbNjh9DulfExMhIgIaN77ysNFKxoSP1LRUdh3fRfyh+CuOcynnaFyhMY0i\nGjlHhUbUv6k+RQoUcSVOSxg+kJbmbPaTnhTSj0OHnCakRo2uPEqV8lkoxpgglnQ6iYSkBBIOJThf\nkxLYeWwnNcvUpFEFJ4nERMQQExFDZPFIn9dGgiphiEh3Lu/n/YGqvpZFmbeAHjj7ed+vquuzKJNn\nCePECaeWkJjoJIgNG2DjRmeyW0yMM5Q1PTHUrm0zpI0xuXMh5QKbj2y+lEgSDyeSkJSAql5KHjER\nMUSXj6ZB+QZ52sEeNAlDRPID24AuwH5gNTBQVbdkKNMTeExVe4pIK+BNVW2dxbmynTAuXHA2/ElP\nDunHiRPOkNWYmMtHdDSUCfz+KwDi4uKIjY11O4yAYPfiMrsXlwXDvVBVks4ksSFpwxXHtl+2EVUy\niujy0c4R4XytdWOtHO0nktOE4cbfyS2Bnar6E4CITAH6AlsylOkDfAKgqitFpLSIRKhqkrcXSU11\n5jFs3HjlsWsXVK/uJIPoaBgxwkkO1aoF9+S3YPhl8Be7F5fZvbgsGO6FiFCheAUqFK/AbTVvu/R6\ncmoy23/ZTuLhRBKTEpmYMJHEw4kknU6iXrl6NCzfkIblGxJdPpqG5RsSVTLKJ81abiSMSsDeDM/3\nAa28KBMFeJUwnn4axo+H8uWduQwNG0KfPjBqlDMruog7/UzGGJMjBfMXpEH5BjQo34ABDQdcev3U\nhVNsPrKZjYc3svHwRubunsvGwxs5m3yW17u8zsjmI/M0DjcShrdtSJnTo9dtT7//Pbz0km30Y4wJ\nbSUKl6BVVCtaRV35N/fRs0fxRXeDG30YrYHRqtrd8/w5IC1jx7eIjAPiVHWK5/lWoEPmJikRCd4h\nXsYY46Jg6cNYA9QWkWrAAeA3wMBMZaYCjwFTPAnmRFb9Fzn5BxtjjMkZvycMVU0RkceAOTjDav+t\nqltEZKTn/fGqOlNEeorITuAM8IC/4zTGGHOloJ64Z4wxxn+CYiCpiHQXka0iskNEnrlKmbc87yeI\nSBN/x+gv17sXIjLYcw82iMhSEYlxI05/8ObnwlOuhYikiEg/f8bnT17+jsSKyHoR2SgicX4O0W+8\n+B0pJyKzRSTecy/udyFMnxORD0UkSUQSr1Eme5+bqhrQB06z1U6gGlAQiAduzlSmJzDT87gVsMLt\nuF28F7cApTyPu4fzvchQbgEwHejvdtwu/lyUBjYBUZ7n5dyO28V7MRp4Nf0+AL8ABdyO3Qf3oj3Q\nBEi8yvvZ/twMhhrGpYl+qpoMpE/0y+iKiX5AaRGJ8G+YfnHde6Gqy1X1pOfpSpz5K6HIm58LgMeB\nr4Aj/gzOz7y5F4OAr1V1H4CqHvVzjP7izb04CJT0PC4J/KKqKX6M0S9UdTFw/BpFsv25GQwJI6tJ\nfJW8KBOKH5Te3IuMfgvM9GlE7rnuvRCRSjgfFu95XgrVDjtvfi5qAzeKyA8iskZEhvgtOv/y5l68\nDzQQkQNAAvCEn2ILNNn+3AyGJfR8PtEviHj9bxKRjsCDQFvfheMqb+7FG8CzqqrirJMQqsOwvbkX\nBYGmQGegKLBcRFao6g6fRuZ/3tyLUUC8qsaKSE1grog0UtVTPo4tEGXrczMYEsZ+oHKG55VxMuG1\nykR5Xgs13twLPB3d7wPdVfVaVdJg5s29aIYzlwectuoeIpKsqlP9E6LfeHMv9gJHVfUccE5EFgGN\ngFBLGN7cizbAXwFUdZeI/AjUxZkjFk6y/bkZDE1Slyb6iUghnIl+mX/hpwL3waWZ5FlO9AsB170X\nIlIF+Aa4V1V3uhCjv1z3XqhqDVWtrqrVcfoxHg7BZAHe/Y58B7QTkfwiUhSnk3Ozn+P0B2/uxVac\n1bLxtNnXBXb7NcrAkO3PzYCvYahN9LvEm3sB/AkoA7zn+cs6WVVbuhWzr3h5L8KCl78jW0VkNrAB\nSAPeV9WQSxhe/ly8AnwkIgk4fzT/r6oecy1oHxGRyUAHoJyI7AVexGmazPHnpk3cM8YY45VgaJIy\nxhgTACxhGGOM8YolDGOMMV6xhGGMMcYrljCMMcZ4xRKGMcYYr1jCMMYY4xVLGMYYY7xiCcOENREp\nfJXXi/j7msYEOksYJuSJSB8RWZrF672AElf5tigR6eqDWK51zfQyd4nILhF5zctzFhaRRSJiv8/G\np+wHzISDHcCqjC+ISCRQ8mobCXkWbqwvIsXyKojrXTPDtb8CFqjqVbedzVT+ArAYuCP3URpzdZYw\nTDi4hV8vXf0A8H/X+b7pwOA8jMOba+bUVGCgj85tDGAJw4SH1kAlEfmNiAzyvFbeszcEIvI7ETkn\nIiNEZFt6U5Cq7gIa5uSCItJLRKaJyCHPLncPZ7pmLRF5SkTu8jyf6Pk65zrnzTJWnL2r2+QkVmO8\nFfDLmxuTB+rh7LKWBrwAfAZc6tRW1bdEpCwwAPhQVTP2HeTPeCIRqQ9crW/jE1U94dmTpLeq9haR\nO3BWhf4/ERmXoexNOPuM5/OUP+t5fYbna5Yb2VwtVlW9ICL5RKSIqp6/9u0wJmcsYZiQJiLFgWOq\nelREegKrPW8VzFT0deD3OHslZHTFaCnPHhLX20fiPuBNz+OyXN6c59I1VXW5iDwCDMfpe1jmeWuz\n5/3R1zj/1WIVQnNrYhMgLGGYUNcCWO553BP4m4g0BVIzlRsMvA38S0SaqGr6+2kZC12nhjHRsyVu\nGWCP57VbgImex5mvqap6XkQaADM9O8Rd8OLf9KtYPUN1Uz0d4Mb4hCUME+rqAT94Hu/H+bCfxOUm\nIETkj8AQnGaeR4DPReRB4BRwOuPJvKxhfAAMEGfLwzdVNdnz+tlM5RJEpB9wAugI1MRpLruqa8Ra\nn8uJ0RifsB33TFgSkf/B2b7z+DXKNAbqqurn/rpmLs79CrBaVX01CssYGyVlwtb7wN3XKdMF+NLP\n18w2T3NUO+DbvD63MRlZDcOELRFpD/ysqnuyeC8ayK+q8f66pjGBzhKGMcYYr1iTlDHGGK9YwjDG\nGOMVSxjGGGO8YgnDGGOMVyxhGGOM8YolDGOMMV6xhGGMMcYrljCMMcZ45f8BWbm+Jyn+4cEAAAAA\nSUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"xs = linspace(0, 1)\n",
"plt.plot(xs, cross_entropy_loss(xs, 0), label='y=0')\n",
"plt.plot(xs, cross_entropy_loss(xs, 1), label='y=1')\n",
"plt.ylabel('$L(h(\\mathbf{x}),y)$')\n",
"plt.xlabel('$h(\\mathbf{x}) = \\sigma(w^\\intercal\\mathbf{x}$)')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* **Q:** Koliki je gubitak na primjeru $\\mathbf{x}$ za koji model daje $h(\\mathbf{x})=P(y=1|\\mathbf{x})=0.7$, ako je stvarna oznaka primjera $y=0$? Koliki je gubitak ako je stvarna oznaka $y=1$?\n",
"\n",
"\n",
"* Gubitaka nema jedino onda kada je primjer savršeno točno klasificiran ($h(x)=1$ za $y=1$ odnosno $h(x)=0$ za $y=0$)\n",
"\n",
"\n",
"* U svim drugim slučajevima postoji gubitak: čak i ako je primjer ispravno klasificiran (na ispravnoj strani granice) postoji malen gubitak, ovisno o pouzdanosti klasifikacije\n",
"\n",
"\n",
"* Ipak, primjeri na ispravnoj strani granice ($h(\\mathbf{x})\\geq 0.5$ za $y=1$ odnosno $h(\\mathbf{x})< 0.5$ za $y=0$) nanose puno manji gubitak od primjera na pogrešnoj strani granice"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#TODO: konkretan primjer u ravnini"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Minimizacija pogreške\n",
"\n",
"$$\n",
"\\begin{align*}\n",
"E(\\mathbf{w}) &=\n",
"\\sum_{i=1}^N L\\big(h(\\mathbf{x}^{(i)}|\\mathbf{w}),y^{(i)}\\big)\\\\\n",
"L(h(\\mathbf{x}),y) &= - y \\ln h(\\mathbf{x}) - (1-y)\\ln \\big(1-h(\\mathbf{x})\\big)\\\\\n",
"h(\\mathbf{x}) &= \\sigma(\\mathbf{w}^\\intercal\\mathbf{x}) = \\frac{1}{1 + \\exp(-\\mathbf{w}^\\intercal\\mathbf{x})}\n",
"\\end{align*}\n",
"$$\n",
"\n",
"\n",
"* Ne postoji rješenje u zatvorenoj formi (zbog nelinearnosti funkcije $\\sigma$)\n",
"\n",
"\n",
"* Minimiziramo **gradijentnim spustom**:\n",
"$$ \n",
"\\nabla E(\\mathbf{w}) =\n",
"\\sum_{i=1}^N \\nabla L\\big(h(\\mathbf{x}^{(i)}|\\mathbf{w}),y^{(i)}\\big)\n",
"$$\n",
"\n",
"* Prisjetimo se:\n",
"$$\n",
"\\frac{\\partial\\sigma(\\alpha)}{\\partial\\alpha} = \n",
"\\sigma(\\alpha)\\big(1 - \\sigma(\\alpha)\\big)\n",
"$$\n",
"\n",
"* Dobivamo: \n",
"$$\n",
"\\nabla L\\big(h(\\mathbf{x}),y\\big) = \n",
"\\Big(-\\frac{y}{h(\\mathbf{x})} + \\frac{1-y}{1-h(\\mathbf{x})}\\Big)h(\\mathbf{x})\\big(1-h(\\mathbf{x})\\big)\n",
"\\tilde{\\mathbf{x}} = \\big(h(\\mathbf{x})-y\\big)\\tilde{\\mathbf{x}}\n",
"$$\n",
"\n",
"\n",
"* Gradijent-vektor pogreške:\n",
"$$\n",
" \\nabla E(\\mathbf{w}) = \\sum_{i=1}^N \\big(h(\\mathbf{x}^{(i)})-y^{(i)}\\big)\\tilde{\\mathbf{x}}^{(i)}\n",
"$$\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Gradijentni spust (*batch*)\n",
"\n",
"> $\\mathbf{w} \\gets (0,0,\\dots,0)$
\n",
"> **ponavljaj** do konvergencije
\n",
"> $\\quad \\Delta\\mathbf{w} \\gets (0,0,\\dots,0)$
\n",
"> $\\quad$ **za** $i=1,\\dots, N$
\n",
"> $\\qquad h \\gets \\sigma(\\mathbf{w}^\\intercal\\tilde{\\mathbf{x}}^{(i)})$
\n",
"> $\\qquad \\Delta \\mathbf{w} \\gets \\Delta\\mathbf{w} + (h-y^{(i)})\\, \\tilde{\\mathbf{x}}^{(i)}$
\n",
"> $\\quad \\mathbf{w} \\gets \\mathbf{w} - \\eta \\Delta\\mathbf{w} $\n",
"\n",
"#### Stohastički gradijentni spust (*on-line*)\n",
"\n",
"> $\\mathbf{w} \\gets (0,0,\\dots,0)$
\n",
"> **ponavljaj** do konvergencije
\n",
"> $\\quad$ (slučajno permutiraj primjere u $\\mathcal{D}$)
\n",
"> $\\quad$ **za** $i=1,\\dots, N$
\n",
"> $\\qquad$ $h \\gets \\sigma(\\mathbf{w}^\\intercal\\tilde{\\mathbf{x}}^{(i)})$
\n",
"> $\\qquad$ $\\mathbf{w} \\gets \\mathbf{w} - \\eta (h-y^{(i)})\\tilde{\\mathbf{x}}^{(i)}$\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#TODO kod + primjer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Regularizacija\n",
"\n",
"* Regularizacija sprečava (smanjuje mogućnost) prenaučenosti\n",
" \n",
" \n",
"* Trostruki učinak:\n",
" * **(1)** Ako je model nelinearan, regularizacijom sprečavamo prenaučenost\n",
" * **(2)** Ako imamo puno značajki, regularizacijom efektivno smanjujemo broj značajki jer težine potiskujemo prema nuli\n",
" * **(3) Specifično za logističku regresiju:** Ako je problem linearno odvojiv, sprječavamo \"otvrdnjivanje\" sigmoide\n",
" * $\\|\\mathbf{w}\\|$ raste $\\Rightarrow$ $\\mathbf{w}\\tilde{\\mathbf{x}}$ raste $\\Rightarrow$ sigmoida je strmija\n",
"\n",
"\n",
"* [Skica za (1)]\n",
"\n",
"\n",
"* [Skica za (3)]\n",
"\n",
"\n",
"* L2-regularizacija:\n",
"\n",
"$$\n",
"\\begin{align*}\n",
"E(\\mathbf{w}|\\mathcal{D}) = \\sum_{i=1}^N \\Big( - y^{(i)} \\ln h(\\mathbf{x}^{(i)}) - (1-y^{(i)})\\ln\n",
"\\big(1-h(\\mathbf{x}^{i})\\big)\\Big)\n",
"+ \\color{red}{\\frac{\\lambda}{2}\\mathbf{w}^\\intercal\\mathbf{w}}\n",
"\\end{align*}\n",
"$$\n",
"\n",
"* Korekcija težina:\n",
"$$\n",
"\\mathbf{w} \\gets \\mathbf{w} - \\eta\\Big(\n",
"\\sum_{i=1}^N\\big(h(\\mathbf{x}^{(i)}) - y^{(i)}\\big)\\mathbf{x}^{(i)} + \\color{red}{\\lambda \\mathbf{w}}\\Big) \n",
"$$\n",
"\n",
"* Ekvivalentno:\n",
"$$\n",
" \\mathbf{w} \\gets \\mathbf{w}(1\\color{red}{-\\eta\\lambda}) - \\eta\n",
"\\sum_{i=1}^N\\big(h(\\mathbf{x}^{(i)}) - y^{(i)}\\big)\\mathbf{x}^{(i)}\n",
"$$\n",
"gdje $\\mathbf{w}(1-\\eta\\lambda)$ uzrokuje **prigušenje težina** (engl. *weight decay*)\n",
"\n",
"\n",
"* **NB:** Težinu $w_0$ ne regulariziramo!\n",
" * Ako bismo regularizirali težinu $w_0$, ona bi $w_0\\to 0$. Kako $w_0$ određuje udaljenost ravnine od ishodišta (ta udaljenost je $-w_0|\\|\\mathbf{w}\\|$, v. prethodnu bilježnicu), to znači da bi ravnina uvijek morala prolaziti kroz ishodište i da ne bismo mogli dobro razdvojiti dvije klase u slučajevima kada granica ne prolazi baš kroz ishodište (a općenito granica ne mora prolaziti kroz ishodište).\n",
"\n",
"\n",
"#### L2-regularizirana logistička regresija (gradijentni spust, *batch*)\n",
"\n",
"> $\\mathbf{w} \\gets (0,0,\\dots,0)$
\n",
"> **ponavljaj** do konvergencije
\n",
"> $\\quad \\color{red}{\\Delta w_0 \\gets 0}$
\n",
"> $\\quad \\Delta\\mathbf{w} \\gets (0,0,\\dots,0)$
\n",
"> $\\quad$ **za** $i=1,\\dots, N$
\n",
"> $\\qquad h \\gets \\sigma(\\mathbf{w}^\\intercal\\tilde{\\mathbf{x}}^{(i)})$
\n",
"> $\\qquad \\color{red}{\\Delta w_0 \\gets \\Delta w_0 + h-y^{(i)}}$
\n",
"> $\\qquad \\Delta\\mathbf{w} \\gets \\Delta\\mathbf{w} + (h-y^{(i)})\\mathbf{x}^{(i)}$
\n",
"> $\\quad \\color{red}{w_0 \\gets w_0 - \\eta \\Delta w_0}$
\n",
"> $\\quad \\mathbf{w} \\gets \\mathbf{w}(1\\color{red}{-\\eta\\lambda}) - \\eta \\Delta\\mathbf{w}$\n",
"\n",
"\n",
"#### L2-regularizirana logistička regresija (stohastički gradijentni spust, *on-line*)\n",
"\n",
"> $\\mathbf{w} \\gets (0,0,\\dots,0)$
\n",
"> **ponavljaj** do konvergencije:
\n",
"> $\\quad$ (slučajno permutiraj primjere u $\\mathcal{D}$)
\n",
"> $\\quad$ za $i=1,\\dots, N$
\n",
"> $\\qquad h \\gets \\sigma(\\mathbf{w}^\\intercal\\mathbf{x}^{(i)})$
\n",
"> $\\qquad \\color{red}{w_0 \\gets w_0 - \\eta (h-y^{(i)})}$
\n",
"> $\\qquad \\mathbf{w} \\gets \\mathbf{w}(1\\color{red}{-\\eta\\lambda}) - \\eta (h-y^{(i)})\\mathbf{x}^{(i)}$
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Poveznica s generativnim modelom"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Usporedba linearnih modela"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Sažetak\n",
"\n",
"* Logistička regresija je **diskriminativan klasifikacijski model** s probabilističkim izlazom\n",
" \n",
" \n",
"* Koristi se **logistička funkcija gubitka** odnosno **pogreška unakrsne entropije**\n",
"\n",
"\n",
"* Optimizacija se provodi **gradijentnim spustom**, a prenaučenost se može spriječiti **regularizacijom**\n",
"\n",
"\n",
"* Model **odgovara generativnom modelu** s normalno distribuiranim izglednostima i dijeljenom kovarijacijskom matricom, ali je broj parametara logističke regresije manji \n",
"\n",
"\n",
"* Logistička regresija vrlo je dobar algoritam koji **nema nedostatke** koje imaju klasifikacija regresijom i perceptron\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}