{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### 从一元函数的角度理解梯度下降法" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import math" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "生成一个拥有两个极小值点的四次函数:" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "f = lambda x: (x+2)*(x+1)*(x-2)*(x-1)\n", "X = np.linspace(-3, 3, 10000) \n", "Y = np.array([f(x) for x in X])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "离散点的最小的 Y 和对应的 X:" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(-2.2499992119664576, -1.5808580858085808)" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Y[np.argmin(Y)], X[np.argmin(Y)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "绘制出该四次函数曲线可以发现有两个极小值点，并且最小值是左边的极小值点。" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl4nXWZ//H3neUkzcm+NvvWdC+0NC0USqFQaFkGcBRxQ3BQ0J/j6IwL+pvfNaOjzujouP90QBBRcdABFWSTpa1toVtaSrekTZukTZp935dzznf+yEmnltImac55zvOc+3VdvZqTnPTcj8RPvud+7uf7iDEGpZRS9hdhdQFKKaVmhga6Uko5hAa6Uko5hAa6Uko5hAa6Uko5hAa6Uko5hAa6Uko5hAa6Uko5hAa6Uko5RFQwXyw9Pd0UFRUF8yWVUsr29uzZ026MybjQ84Ia6EVFRVRUVATzJZVSyvZE5MRknqctF6WUcggNdKWUcggNdKWUcggNdKWUcggNdKWUcohJB7qIRIrImyLynP9xsYjsFJFqEfmNiLgCV6ZSSqkLmcoK/dNA5RmPvwl81xhTBnQB981kYUoppaZmUoEuInnALcAj/scCXAc85X/K48AdgSgQ4Jl9p/jVjkmNYSqlVEipbunjO68cpbVvOOCvNdkV+veALwA+/+M0oNsY4/E/bgByz/WNInK/iFSISEVbW9u0inzhQBOPvV47re9VSikr7TnRxQ9eq2ZkzHfhJ1+kCwa6iNwKtBpj9pz56XM89Zx3mzbGPGyMKTfGlGdkXPDK1XMqSndT3zmE16c3tFZK2UttxwDRkUJ2UmzAX2syl/5fBdwmIjcDsUAi4yv2ZBGJ8q/S84DGQBVZnOZm1OujsXuI/NS4QL2MUkrNuLr2AfJT44iKDPxQ4QVfwRjzJWNMnjGmCHgfsNEY80FgE/Ae/9PuAZ4JVJFF6W4AatsHAvUSSikVEHXtgxSnuYPyWhfzK+NB4B9E5BjjPfVHZ6aktyv2B3pdhwa6Uso+fD5DXcfA6UVpoE1pt0VjzGZgs//jGmDlzJf0dpkJMcyKjqSufTAYL6eUUjOiuXeYEY8vaIFuiytFRYTCtDhdoSulbKXO3ya2Q8slqIrT3af/x1FKKTuo9S9Ci9KDM8xhm0AvSndzsnMQjzfws5xKKTUT6toHcEVFkJM0KyivZ5tAL05z4/EZTnUPWV2KUkpNSm37IIWpcUREnOvSnZlnm0DX0UWllN0Ec8IFbBXo4z0o7aMrpezA6zOc7Bg8PXYdDLYJ9Iz4GNyuSOo6dHRRKRX6mnqGGPX6KArShAvYKNBFhKJ0t7ZclFK2MHHdTLAmXMBGgQ7jfXSdRVdK2cHEyKK2XN5BcZqbhq4hxnR0USkV4uraB4iNjiArIfC7LE6wVaAXpbvx+gz1ndpHV0qFtrr2AYrS3EEbWQSbBXrxxKSLtl2UUiGutmMgqCdEwWaBXpg2MYuuK3SlVOjyeH3Udw4GdQYdbBboaW4XCTFRnNAVulIqhDV2DzPmNae7CsFiq0DX0UWllB2c3pRLWy7nV5zupqZNA10pFbpq2/qB4I4sgg0DvTQjnsaeIYZGvVaXopRS53S8bYCE2CgyEmKC+rr2C/RMN8boJl1KqdB1vK2f0ox4RII3sgh2DPSMeABq2vstrkQppc6tpm2AkozgtlvAhoFenO5GBI636gpdKRV6+kc8NPcOn158BpPtAj02OpLc5Fkcb9MVulIq9NT4s0kDfZJKM+I10JVSIWkim+ZkastlUkoz4qlpG8DnM1aXopRSf+F46wCREUJBqgb6pJRmuhka89LUO2x1KUop9ReOt/VTmBqHKyr48WrPQPf3po63attFKRVajrf1U2JB/xzsHujaR1dKhRCvz1DXPkipBSOLYNNAT493kRgbpYGulAopDV2DjHp9lky4gE0DXUQozYzXWXSlVEiZWGSWWjDhAjYNdNDRRaVU6JlYZJak6wp9Skoz4mntG6FveMzqUpRSChhfoae5XaS4XZa8vo0DffwtjW6lq5QKFRObclnFvoGeqZMuSqnQUtM2YFn/HGwc6AWpcURFiAa6UiokdA2M0jEwaln/HGwc6NGRERSkxemki1IqJExs6a0r9GkqzYjnmK7QlVIh4GjLeBaVZSZYVoOtA31uVjx17QOMenxWl6KUCnNHW/qY5d/e2yo2D/QEPD6jt6NTSlmuuqWfsqx4IiKCe9u5M10w0EUkVkR2ichbInJIRL7i/3yxiOwUkWoR+Y2IBH3wcuKtzdGWvmC/tFJK/YWjLX2Wtltgciv0EeA6Y8ylwFJgg4hcAXwT+K4xpgzoAu4LXJnnVpLhJjJCNNCVUpbqGRyjtW+EuVnWTbjAJALdjJs48xjt/2OA64Cn/J9/HLgjIBWeR2x0JIVpcRroSilLHW0dz6C5WaG/QkdEIkVkH9AKvAIcB7qNMR7/UxqA3MCUeH5zMxOobtFJF6WUdSYWlWWhvkIHMMZ4jTFLgTxgJbDgXE871/eKyP0iUiEiFW1tbdOv9B3MzYqnrmOA4THvjP/bSik1GdUt/bhd1k64wBSnXIwx3cBm4AogWUSi/F/KAxrf4XseNsaUG2PKMzIyLqbWcyrLSsBndE8XpZR1jrb0MScrARHrJlxgclMuGSKS7P94FrAOqAQ2Ae/xP+0e4JlAFXk+82brpItSylpHW/qZm2ltuwUg6sJPIRt4XEQiGf8F8FtjzHMichh4UkS+BrwJPBrAOt9RUZqbKJ10UUpZpGtglPb+EctPiMIkAt0Ysx9Ydo7P1zDeT7eUKyqC4nT36ctulVIqmELlhCjY/ErRCXOzEqhu1RW6Uir4jraOLyZDYYXuiEAvy4rnZOcgQ6M66aKUCq7qlj4SYqLIToq1uhRnBPq8rASMgWOt2nZRSgXX+IRLvOUTLuCQQC/L0kkXpZQ1qlv6mWvxHi4THBHoRWlxuCIjTl9+q5RSwdDRP0LHwGhInBAFhwR6VGQEJRlujjZroCulgqfKnznzZydaXMk4RwQ6jF9gdEQDXSkVRJVNvQAsyNaWy4xakJ1IY88w3YOjVpeilAoTlU19ZCTEkBYfY3UpgIMCfb5/C4DKJl2lK6WCo6q593T2hALHBPrC7PEeVlVzr8WVKKXCgcfro7ql/3T2hALHBHpGQgxpbtfpnpZSSgVSTfsAo14f80Okfw4OCnQRYUF2orZclFJB8b8nRHWFHhALshM40tKHx+uzuhSllMNVNvURHSmUpIfGDDo4LNDnz05k1OOjrkNvdqGUCqyq5l5KM+JxRYVOjIZOJTNg4q3PYW27KKUCrLKpN6ROiILDAn1OZjzRkaInRpVSAdU5MEpL70hInRAFhwW6KyqC0ox4DXSlVEBVheAJUXBYoMP4PHqVtlyUUgFUGWJ7uExwXKDPz06guXeYrgHdAkApFRiVTb2kx7vISAiNS/4nOC7QJ94CadtFKRUoVc29IdduAQcH+mENdKVUAIx5fRxt6Q+pPVwmOC7Q0+NjyEiI0StGlVIBUd3Sz6jHx+LcJKtLeRvHBTrAopxEDjX2WF2GUsqBDvqzRQM9SBbnJFHd2s/wmNfqUpRSDnPwVA9uVyTFaW6rS3kbZwZ6bhJen9ETo0qpGXfwVA+LcpKIiBCrS3kbRwb6krzxt0IHT2nbRSk1c7w+w+GmXhblht6ECzg00HOSYkl1uzigga6UmkE1bf0Mj/lYnBN6/XNwaKCLCItyEjl4SlsuSqmZM3FCdKILEGocGegAS3KTONrSpydGlVIz5kBDL7HREZSkh94JUXB4oHt8hiPNOo+ulJoZBxt7WJCdSFRkaEZnaFY1AyZmRA/qPLpSagb4fIbDjb0h2z8HBwd6XsoskmZF66SLUmpGnOgcpH/Ew5IQvKBogmMDXURYkpukky5KqRkxsTgM1ZFFcHCgw3jb5UhzHyMePTGqlLo4Bxt7cEVGUJYZeptyTXB4oCcy5jVUt/RbXYpSyuYONPQwb3ZCSN0U+myhW9kMmOh1adtFKXUxvD7D/oYeluYnW13KeTk60AtS40iaFc3+hm6rS1FK2VhNWz/9Ix4u1UC3johwSV4Sb57UQFdKTd+++vEMsf0KXUTyRWSTiFSKyCER+bT/86ki8oqIVPv/Tgl8uVO3LD+Zoy19DIx4rC5FKWVT++q7SYiJCtkrRCdMZoXuAT5rjFkAXAF8UkQWAl8EXjPGlAGv+R+HnKUFyfiM9tGVUtP3VkM3l+SH5pa5Z7pgoBtjmowxe/0f9wGVQC5wO/C4/2mPA3cEqsiLsTR//I3DxFsmpZSaiuExL1VNfSHfboEp9tBFpAhYBuwEsowxTTAe+kDmTBc3E1LdLgrT4tinfXSl1DQcauzB4zNcmuegQBeReOBp4DPGmEnvSysi94tIhYhUtLW1TafGi7Y0P1lX6EqpaZkYqnDMCl1EohkP8yeMMb/zf7pFRLL9X88GWs/1vcaYh40x5caY8oyMjJmoecqW5ifT3DtMU8+QJa+vlLKvtxp6yEmKJTMx1upSLmgyUy4CPApUGmO+c8aXngXu8X98D/DMzJc3MyZ+s2rbRSk1VW/Vd4f8/PmEyazQrwLuBq4TkX3+PzcD3wBuEJFq4Ab/45C0MCcRV2SEtl2UUlPS0T/Cyc5BW7RbAKIu9ARjzDbgnWZ1rp/ZcgIjJiqSBTmJvKmBrpSagv0N4+POTlqhO8Ky/GQONPTg8fqsLkUpZRN7TnQRGSEhvQf6mcIm0JfmJzM05uWo7ryolJqkihOdLMxOxB1zwWZGSAibQF9WMP6Wac/JLosrUUrZwZjXx776bpYXhuSuJucUNoFekBpHRkIMe+o6rS5FKWUDhxt7GR7zUV6kgR5yRIQVRSnsrtMVulLqwipOjGdFeWGqxZVMXtgEOoz/hznVPURjt15gpJQ6vz0nOslNnsXspNC/oGhCWAX6iqLx37QTv3mVUupcjDFU1HXZqt0CYRboC7ITiHNFUqF9dKXUeTR0DdHaN0K5jU6IQpgFelRkBJcVaB9dKXV+FSfGF33LbdQ/hzALdIDyohSqmnvpHR6zuhSlVIiqqOsiISaKebMTrC5lSsIu0FcUpWIM7NU+ulLqHew50cXSgmQiQ/wORWcLu0Bfmj/+H6lC2y5KqXPoGRrjSEufrcYVJ4RdoLtjoliUk8huPTGqlDqH3bWdGAOXl2ig20J5YSr76rsZ9ehGXUqpv7SjpgNXVIRttsw9U1gG+sriFEY8PvY36Ha6Sqm/tKO2g8sKkomNjrS6lCkLy0C/vDgNEXjjeIfVpSilQkjP0BiHGntZVZJudSnTEpaBnuJ2sWB2Its10JVSZ5jon19hw/45hGmgA1xZmsaek10Mj3mtLkUpFSK213QQExVhmzsUnS1sA31VaRqjHh97dX90pZTfjpoOLitIsWX/HMI40FcWpxIZIdp2UUoB0DM4xuGmXq4oSbO6lGkL20BPiI1mcW6SBrpSCoBddeP981WlGui2dGVpGvvquxkY8VhdilLKYjtO98/tcUPocwnrQF9VkobHZ3R/dKUUrx9rZ3lhCjFR9uyfQ5gHenlRCtGRwhvH260uRSllodbeYaqa+7i6LMPqUi5KWAd6nCuKpfnJ2kdXKsxtOza+qLu6zJ4XFE0I60AHuLosgwOneugcGLW6FKWURbZWt5PmdrEwO9HqUi5K2Af6mrkZGANbq9usLkUpZQFjDFur21ldlk6EzfY/P1vYB/qS3CSS46LZclT76EqFo6rmPtr7R1g9x97tFtBAJzJCuLosgy3VbRhjrC5HKRVkE+/O7X5CFDTQAVhTlk5b3wiVTX1Wl6KUCrKt1e3MzYpndlKs1aVcNA104Jq547+Z/3xU++hKhZPhMS87azsdsToHDXQAMhNjmT87gS0a6EqFlZ21nYx6fKy2+bjiBA10v2vmZVBxolO3AVAqjGyqaiU2OoJVNt6Q60wa6H7XlGUw5jV6kZFSYcIYw2tVLVxVmm7b7XLPpoHut7woBbcrko1HWq0uRSkVBNWt/dR3DnH9giyrS5kxGuh+MVGRXDMvg9cqW/D5dHxRKad7rXJ88Xbd/EyLK5k5GuhnWLcgi5beEQ6c6rG6FKVUgL1W2cKinERHjCtO0EA/w9p5mUQIvFrZYnUpSqkA6hwYZe/JLke1W2ASgS4iPxORVhE5eMbnUkXkFRGp9v+dEtgygyPF7aK8KJVXDmugK+Vkfz7ais/A9Q5qt8DkVug/Bzac9bkvAq8ZY8qA1/yPHeHGhVlUNfdR3zlodSlKqQB5tbKVjIQYluTa9+5E53LBQDfGbAE6z/r07cDj/o8fB+6Y4bosM/EWTNsuSjnTiMfLliNtXDcv0/a7K55tuj30LGNME4D/b8e8bylOdzMnM14DXSmHev1YO30jHjYsmW11KTMu4CdFReR+EakQkYq2NntcWr9uQRY7azrpGRyzuhSl1Ax74UAzCbFRXFXqjMv9zzTdQG8RkWwA/9/veDWOMeZhY0y5MaY8I8MeG+DctHg2Hp/h5cPNVpeilJpBY14frxxu4YYFWbiinDfkN90jeha4x//xPcAzM1NOaLgkL4m8lFk8f6DJ6lKUUjNoR00HPUNjbFjsvHYLTG5s8b+A7cA8EWkQkfuAbwA3iEg1cIP/sWOICLdcks226na6B/Veo0o5xQsHmnG7Ilkz1x7dgqmazJTL+40x2caYaGNMnjHmUWNMhzHmemNMmf/vs6dgbO+WJdnjbZdDenJUKSfw+gwvH2pm7fxMx2zGdTbnNZFmyJLcJPJTZ/Gctl2UcoRdtZ10DIxy85Jsq0sJGA30dyAi3LIkhzeOtdM1oG0Xpezu2bcaiXNFcu08Z7ZbQAP9vG69JFunXZRygBGPl+f3N7J+0WziXFFWlxMwGujnsSgnkaK0OJ7Z12h1KUqpi7D5SBu9wx5uX5pjdSkBpYF+HiLCu5blsb2mg1PdQ1aXo5Sapmf2nSI93sXqOc67mOhMGugX8NeX5WIM/OHNU1aXopSaht7hMV6tbOXWS3KIinR25Dn76GZAfmocK4tTeXpvA8bonYyUspuXDjQz6vFxx7Jcq0sJOA30SXj3ZbnUtA2wr77b6lKUUlP0+zdPUZQWx6V5ztoq91w00CfhpiXZxERF8Lu92nZRyk5OdAywvaaDd1+Wh4iztso9Fw30SUiMjWb9otk8+1YjIx6v1eUopSbptxX1RAjcWZ5vdSlBoYE+Se9ZnkfP0Bh/0q0AlLIFj9fHf1c0sHZepqNuBH0+GuiTtHpOOvmps3hixwmrS1FKTcKmI2209o3wvpUFVpcSNBrokxQRIXxgZSE7azs51tpndTlKqQt4ctdJMhNiWOvgS/3PpoE+BXeW5xEdKTyx86TVpSilzqOpZ4hNR1q5szzP8bPnZwqfI50B6fExbFiczdN7Ghga1ZOjSoWqJ3acxADvWxE+7RbQQJ+yD11eQO+wh+f26/4uSoWi4TEvv951knULsshPjbO6nKDSQJ+ilcWplGXG89jrdXrlqFIh6Nm3GukcGOUjVxZZXUrQaaBPkYhw3+piDjf1sv14h9XlKKXOYIzhsdfrmJeVwKrSNKvLCToN9Gm4Y1ku6fEufrq1xupSlFJn2FnbSWVTL/deVRQWV4aeTQN9GmKjI/nwqiI2HWmjukVHGJUKFY9srSU5Lpo7ljp/I65z0UCfpg9dUUhsdASPbK21uhSlFFDV3MurlS3ce2URs1zOvAn0hWigT1Oq28V7lufx+zdP0dI7bHU5ahqMMQyPeekZHKNveIzhMS8er09PdtvUTzYfx+2K5N4wPBk6wbk31wuC+68u5cld9fxk83G+fNsiq8tRZxkc9VDd0s+Rlj6Ot/XT1D1Mc88wTb1DdA+OMTjqxet7e3i7oiJIc7tIdbtIi4+hIHUWRWluitPdzM1KIC9lVlj2Z0NZXfsAf3yrkY9dXUJynMvqciyjgX4RCtLiePdlefx610k+cW0pWYnhsQFQKDLGUNs+QEVdF7vqOtlzoova9oHTX3dFRpCdHMvsxFiWF6SQHOfCHRNJnCuK2OhIfD6Dx2fweH30j3joHBilc2CUtv4R9p3sonfYc/rfSomLZkleMkvzkri8JI3lhSnERofnW/xQ8dCW40RFRnDf6mKrS7GUBvpF+uTaOTy9t0FX6RYYGvXyxvF2XqtqZWNlK83+1ldKXDTLC1N517Jc5mYlMG92AgWpcURGTG9VbYyha3CM2vZ+Kpv62N/Qzf6GHn60qY0fbDxGTFQEK4tTubosnfWLZlOY5p7Jw1QXUN85yFN7GrhrRT6ZYb6o0kC/SLpKD64Rj5dNVa384c1GNh1pZcTjw+2K5OqyDNbMzWBlcQol6fFETDO8z0VESHW7SHWnsrwwFSgEoH/Ew67aDrZWt/P6sXb+9YUq/vWFKhZmJ3LzktncvCSbkoz4GatDndt3XzlKhAh/u7bM6lIsJ8E8AVReXm4qKiqC9nrBUt85yNpvb+auFfl8/V1LrC7HcYwx7DnRxVN7Gnj+QBN9wx7S42O4Zcls1i3MYmVxKjFR1rc8GroGeelgMy8ebGbPiS4Alhem8N7yPG65JIf4GF0/zbSq5l5u+v5W7l9TwpduWmB1OQEjInuMMeUXfJ4G+sz48rOH+MX2Ov70mTWUZSVYXY4jDIx4+MO+U/xy+wmqmvtwuyJZv3g2dyzN5crStJDeRa+pZ4hn9zXy24p6jrcNEOeK5NZLsrnnyiIW5Tj/3pbB8tHHd7OztpOtX1jr6JOhGuhB1jkwyjXf2kR5YQqPfWSl1eXYWn3nII9uq+WpPQ30j3hYmJ3Ih1cVctvSHOJc9lrlGmPYe7Kb3+6u54/7Gxkc9XJFSSr3rS7huvmZ0+7rK9hZ08FdD+/g8+vn8cm1c6wuJ6A00C3w8Jbj/OsLVfzyvpVcXRY+m+rPlMqmXv7zz8d5bn8TEQK3XpLDh64o5LKCZEeMCfYMjvGbipM8/sYJTnUPUZgWx0evLuG95Xkh0TKyE4/Xx60/3EbfsIdX/+Eax19IpIFugRGPl3Xf+TNx0VE893eriQ7hlkAo2V3XyY83HWPTkTbcrkg+cHkB960ucex9ID1eH3861MJPt9awr76b2YmxPHBNCe9fWaDjj5P0i+11/NMzh/jxBy/j5iXZVpcTcBroFnnlcAsf+0UFD26YzyeuLbW6nJB2pLmPb75UxcaqVlLdLj5yZRF3ryp0dC/0TMYY3jjewQ9eq2ZnbSfp8TE8sKaED15RYLvWUjB1Doyy9tubWZSTyBMfvdwR794uZLKBrj81M+yGhVmsX5TF9187yi1LsilIC68N9iejqWeI77x8lKf3NuCOieLBDfPDcv8NEeGqOelcNSedHTUd/HBjNV9/oZKHttTw6evncNeKAlxR+i7vbF9/vpL+EQ9fvm1RWIT5VOgKPQCaeoZY9x9/ZnlRKo9/ZIX+0Pn1DI3xk83Heez1WoyBD68q5JNr55DiDo8V+WRU1HXy7y8dYVddJwWpcXz2xrn81SU5MzpXb2cbq1r4m59X8Ldr5/C59fOsLidotOVisZ+/XsuX/3iYb757CXeF2X0Nzzbi8fLL7Sf40aZjdA+OccfSHD5747ywuz3YZBlj2HykjW++VEVVcx8LsxP5woZ5XDM3I6wXB73DY9z4nS0kzorij59aHVYnkrXlYrEPryri5cMtfOWPh1lZnEZxevhdDu7zGZ59q5Fvv3yEhq4hri5L58EN81mcq3PY5yMirJ2fyTVzM/jj/kb+4+Wj3PvYbi4vTuVLNy9gaX6y1SVa4l/+eJjWvmEeuvuqsArzqdAVegA19Qyx4XtbKUp389THV4XV1Mu26nb+7cVKDjX2sjA7kS/dPF9HOadp1OPjyd0n+f6r1XQMjHLLkmw+v34eRWG0SPj9mw38/W/e4lPXzeGzN4ZPq2WCtlxCxPP7m/jkr/fywJoSvnSzcy9NnnCosYdvvFjF1up2cpNn8fn187jtUu0Bz4T+EQ8Pb6nhka01jHp8fODyAv7u+jLS42OsLi2gjrf181c/3MbinCR+/bHLQ/oK4UAJSstFRDYA3wcigUeMMd+4mH/PiW65JJvtNQU8tKWGRblJ3HZpjtUlBURD1yD/8fJR/rDvFEmzovl/tyzg7lWF+tZ4BsXHRPEPN8zlQ1cU8P1Xq3li50me3tPA/WtK+ejVxbgduFdM/4iHTz6xl5ioCL7//qVhGeZTMe0VuohEAkeBG4AGYDfwfmPM4Xf6nnBcocP4W+YPPrKDA6d6ePoTVzpqL4/uwVF+tPEYv9h+AhH4yFXFfOLaUpJmRVtdmuMdb+vnWy8d4aVDzaTHx/CZdWXctSLfMa09r89w/y8q2Hy0jcfuXcGaueHbsgt4y0VEVgFfNsas9z/+EoAx5t/e6XvCNdABWvuGue2HrwPw1CdWkZdi7wmP4TEvP3+jjh9vOkbfiIf3XJbH398wl5zkWVaXFnb2nOjiGy9Wsruui5J0N1/YMI/1i2bbfiLmq88d5tFttXz19kXcvarI6nIsNdlAv5hf5blA/RmPG/yfO7uQ+0WkQkQq2traLuLl7C0zIZbHPrKCgVEPH350Fx39I1aXNC0er4/f7q5n7bc3840XqygvSuWlT6/hW3deqmFukeWFKfz2gVX89MPlREQIH//VXt79kzfYXddpdWnT9v83HePRbbXce2VR2If5VFzMCv1OYL0x5qP+x3cDK40xn3qn7wnnFfqE3XWdfOiRnczJjOeX911Oqk0uqjHG8KdDzXzrT0c43jbApfnJfHHDfFaVplldmjqDx+vjqT0NfPfVo7T0jrBuQRZfvGkeczLts6XzI1tr+NrzlbxrWS7fvvNS3ZGS4KzQG4D8Mx7nAY0X8e+FhRVFqTx093KOtfZz10Pbae4ZtrqkC3rjeDt3/PgNPv6rvQD854eW84f/c6WGeQiKiozgfSsL2Py5tXx+/Tx21nRw43e38MWn99PSG9o/a8YYfrSxmq89X8ktS7L51nsu0TCfootZoUcxflL0euAU4ydFP2CMOfRO36Mr9P+1/XgHH318NyluFz+7dwVzQ/CmGDtrOvjhxmNsO9ZOdlIsf79uLn99Wa5OGthI58AoP9xYza/9BFAwAAAHmUlEQVR2nCAyQrhvdTEPXFNKYmxonbT2+gxffvYQv9xxgjuW5vDv77lU97E5Q1Dm0EXkZuB7jI8t/swY8/XzPV8D/S/tb+jmvscrGBjx8J33XsqGxdZvA2qM4fVjHfxgYzW7ajtJj3fxwJpS7l5VqFu72tjJjkG+/fIRnn2rkZS4aD62poQPXl4YEtNI7f0jfPrJN3n9WAcPXFPCg+vn63ULZ9ELi2yipXeYB365h3313dx9RSFfvGm+JfPEY14ffzrUzCNba3WPbgc70NDDt14+wpaj43vPv29lAX+zuphci05ob6tu57P/vY/uwTG+evti3rsi/8LfFIY00G1kxOPl3186ws9eryU/JY6v3L6Ia4O0EVNH/whP7q7nl9tP0Nw7TEFqHPevKeFOvYuOox081cNPt9bw3P4mBLhpSTbvX5HPFSVpQVkddw6M8rXnD/O7vacoSXfzow9cxsKcxIC/rl1poNvQrtpOHnx6P7XtA1w1J43P3TiPZQUpM/46Ix4vm6pa+d3eU2w60sqY13B1WTr3XlnEtfP0Ppfh5FT3ED/bVst/V9TTO+yhMC2O95bnc8ey3ICs2nuHx3h0ay2PbqtleMzLA9eU8KnryvRd4AVooNvUqMfHr3ac4Acbq+keHGNZQTJ3X1HIuoVZF3Uiq2dojC1H29hY1crGqlZ6hsbISIjhjqU53LUi31ZjbWrmDY95eelgM/+16yQ7a8fn1y/JS2LD4tmsW5BFWWb8tN8xGmM4eKqXJ3ef5A9vnmJg1MtNi2fz2Rvn6s/dJGmg21z/iIenKur5+Rt11HUM4oqM4Mo5aawoSmVZQTKlGfFkxMe87e2xMYbuwTHqOgY43jbAW/XdvFnfRWVTH16fISUumrXzMrltaQ6r56TrxIp6m7r2AV461MyLB5t5q74bgFS3i5VFqSwtSGZuVjxlmQlkJ8We8+dncNTDiY5BDpzq4c2T3Ww+0kpTzzAxURH81aU53HtlkW6hPEUa6A7h8xnerO/mxQNNbDrSyvG2gdNfc0VGkBwXTUx0BFEREfSPeOgbHmN4zHf6OfExUVyan8RlBSlcOy+Dpfkp2lJRk9bYPcS2Y+3srOlkZ20HDV1Df/H1pFnRpydlvD5D79AYfSOe01+Pj4niytI01i3I4sZFWWFzv9iZpoHuUN2Do+xv6OFE5yANXYP0DI4x4vHh8RniYyJJiI0mKzGWwtQ4itLjKE6P1wBXM6Z7cJRjrf1Ut/bT0jtM18AoPUNjiAgikOj/+ctJjmVxbhLFaW4dQZwBescih0qOc4X1rnPKWslxLsqLUikvSrW6FHUO2kBVSimH0EBXSimH0EBXSimH0EBXSimH0EBXSimH0EBXSimH0EBXSimH0EBXSimHCOqVoiLSBpyY5renA+0zWI6VnHIsTjkO0GMJVU45los9jkJjzAWvKAxqoF8MEamYzKWvduCUY3HKcYAeS6hyyrEE6zi05aKUUg6hga6UUg5hp0B/2OoCZpBTjsUpxwF6LKHKKccSlOOwTQ9dKaXU+dlpha6UUuo8bBXoIvJVEdkvIvtE5GURybG6pukSkW+JSJX/eH4vIslW1zQdInKniBwSEZ+I2HIaQUQ2iMgRETkmIl+0up7pEpGfiUiriBy0upaLISL5IrJJRCr9P1uftrqm6RKRWBHZJSJv+Y/lKwF9PTu1XEQk0RjT6//474CFxpiPW1zWtIjIjcBGY4xHRL4JYIx50OKypkxEFgA+4CHgc8YYW92SSkQigaPADUADsBt4vzHmsKWFTYOIrAH6gV8YYxZbXc90iUg2kG2M2SsiCcAe4A6b/jcRwG2M6ReRaGAb8GljzI5AvJ6tVugTYe7nBuzz2+gsxpiXjTETN1/cAeRZWc90GWMqjTFHrK7jIqwEjhljaowxo8CTwO0W1zQtxpgtQKfVdVwsY0yTMWav/+M+oBLItbaq6THj+v0Po/1/ApZbtgp0ABH5uojUAx8E/snqembI3wAvWl1EmMoF6s943IBNw8OJRKQIWAbstLaS6RORSBHZB7QCrxhjAnYsIRfoIvKqiBw8x5/bAYwx/2iMyQeeAP7W2mrP70LH4n/OPwIexo8nJE3mOGzsXHcwtu07PycRkXjgaeAzZ707txVjjNcYs5Txd+ErRSRg7bCQu0m0MWbdJJ/6a+B54J8DWM5FudCxiMg9wK3A9SaET2ZM4b+JHTUA+Wc8zgMaLapF+fn7zU8DTxhjfmd1PTPBGNMtIpuBDUBATlyH3Ar9fESk7IyHtwFVVtVysURkA/AgcJsxZtDqesLYbqBMRIpFxAW8D3jW4prCmv9E4qNApTHmO1bXczFEJGNigk1EZgHrCGBu2W3K5WlgHuNTFSeAjxtjTllb1fSIyDEgBujwf2qHHSd2RORdwA+BDKAb2GeMWW9tVVMjIjcD3wMigZ8ZY75ucUnTIiL/BVzL+M5+LcA/G2MetbSoaRCR1cBW4ADj/18H+L/GmBesq2p6ROQS4HHGf7YigN8aY/4lYK9np0BXSin1zmzVclFKKfXONNCVUsohNNCVUsohNNCVUsohNNCVUsohNNCVUsohNNCVUsohNNCVUsoh/gdqKCXvOPqOxwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(X, Y)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$f(x)$ 为:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$f(x)=(x+1)(x-1)(x+2)(x-2)$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "对 $f(x)$ 求导可得:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$f'(x)=4x^3-10x$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "通过对 $f'(x)$ 等于零可得 $f(x)$ 的最小值为 $x=\\sqrt{\\frac{5}{2}}$。" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.5811388300841898" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = lambda x: 4*x**3-10*x\n", "math.sqrt(5/2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "利用梯度下降法求函数的极小值，基本公式为:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$x= x-\\lambda f'(x)$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "其中 $\\lambda$ 是控制 $x$ 更新的幅度，更具体点就是深度学习中的学习率。\n", "\n", "下面就是利用梯度下降法求函数的极值点。" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "-1.581138831026283" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr = 0.001\n", "x = -3\n", "for i in range(1000):\n", " x -= lr*df(x)\n", "x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$1000$ 其实就是控制 $x$ 更新的次数，对应的是深度学习中的最大迭代次数，可以看出该值基本上和 $\\sqrt{\\frac{5}{2}}$ 相差无几了。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 一个好的权重初始化的意义" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "为了方便理解，同样从一元函数的角度出发，**为什么一个好的权重初始化非常重要？**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "首先要知道梯度下降法最大的问题是什么：迭代终止于**局部最优解**，很难找到**全局最优解**。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "当 $x$ 初始化为 3 呢？" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.581138831026283" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr = 0.001\n", "x = 3\n", "for i in range(1000):\n", " x -= lr*df(x)\n", "x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "同样的参数取得了不同的结果(ps: 为了计算方便这个四次函数关于 Y 轴对称，两个极值点都为最小值)。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "所以这就是一个好的初始化方法的重要性，caffe 权重的初始化方法一般为 Gaussian 和 XavierFiller，偏置项一般常数初始化为 0，具体如何初始化可以参考 [caffe中权值初始化方法](https://www.cnblogs.com/tianshifu/p/6165809.html)。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 动画展示梯度下降的过程" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "%%capture\n", "%matplotlib inline\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "from matplotlib import animation\n", "from IPython.display import HTML\n", "\n", "def show_anim(X, Y, f, df, max_iterations, lr, initial_x):\n", " fig, ax = plt.subplots()\n", " ax.plot(X, Y)\n", " line, = ax.plot(0, 0, 'ro')\n", " annotation = ax.annotate(\"\", xy=(0, 0), xytext=(0, 0), arrowprops=dict(arrowstyle=\"->\"))\n", " # 返回梯度下降的点，生成动画的箭头\n", " def gradient_descent(f, df, Y, max_iterations=30, lr=0.001, x=-3):\n", " points = [(x, f(x))]\n", " print(lr)\n", " for _ in range(max_iterations):\n", " x -= lr*df(x)\n", " points.append((x, f(x)))\n", " arrows = []\n", " for i in range(len(points)-1):\n", " arrows.append((points[i], points[i+1]))\n", " return arrows\n", " \n", " arrows = gradient_descent(f, df, Y, max_iterations, lr, initial_x)\n", "\n", " def init():\n", " return line,\n", "\n", " def update(index):\n", " start, end = arrows[index]\n", " line.set_data(end[0], end[1])\n", "\n", " annotation.set_position(start)\n", " annotation.xy = end\n", "\n", " return line, annotation\n", "\n", " anim = animation.FuncAnimation(fig, update, interval=500, blit=False, frames=max_iterations, init_func=init)\n", " return anim\n", "\n", "lr = 0.001\n", "initial_x = -3\n", "max_iterations = 100\n", "anim = show_anim(X, Y, f, df, max_iterations, lr, initial_x)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "HTML(anim.to_html5_video())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "改变一些参数，例如学习率，初始值时会有什么不同。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "学习率非常大:" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "%%capture\n", "%matplotlib inline\n", "lr = 0.075\n", "initial_x = -3\n", "max_iterations = 100\n", "anim = show_anim(X, Y, f, df, max_iterations, lr, initial_x)" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "HTML(anim.to_html5_video())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "设定权重初始值（即 $x$）、学习率、最大迭代次数等，会发现以下几点:\n", "* 越在前面 loss 收敛速度越快，后面则改变幅度很小。\n", "* 学习率过大，或导致 loss 震荡，一直达不到极小值点，虽然这里 lr=0.07 时仍然收敛了，这是因为如果 lr 过大，会直接导致超出计算机能表示的数值范围，在这里就能看出 loss 的变化幅度已经很大了。超出数值范围，对应到 caffe 中就是 Nan（not a number），这是新手容易遇到的问题。\n", "* 学习率过小，收敛速度慢。\n", "\n", "因此，如何选择一个合适的学习率很重要，同时也要应用相应的学习率衰减策略（略）。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "所以深度学习的基础算法之一，梯度下降其实并不难理解，只需要一些数值计算的知识即可。通过一元函数的梯度下降法求极值点，可以对应到深度学习中的很多知识。真正的深度学习当然不可能只是一个一元函数，而是非常复杂的函数，但不管在哪，梯度下降法都是适用的。" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2" } }, "nbformat": 4, "nbformat_minor": 2 }