{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.学习率调整策略\n",
    "\n",
    "Q:_LRScheduler的主要属性和方法有哪些？\n",
    "- 属性：\n",
    "- optimizer：关联的优化器\n",
    "- last_epoch：记录epoch数\n",
    "- base_lrs：记录初始学习率\n",
    "- 方法：\n",
    "- step()：更新下一个epoch的学习率\n",
    "- get_lr()：虚函数，计算下一个epoch的学习率\n",
    "\n",
    "Q:如何等间隔调整学习率？\n",
    "- `torch.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1)`\n",
    "- 调整方式：lr = lr * gamma\n",
    "- step_size：调整间隔数\n",
    "- gamma：调整系数\n",
    "\n",
    "Q:StepLR代码示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5RU5Znv8e+vmvtFMIAOigiJmAQ1ONgBGc8EieIloyEmOkJIgsZ1cCajRk1yNMslGgZPYiajTpQ1CZ4QxGhk4plEjjFqQnAmcbxwCUaQGFsGYwtRLgqicu3n/FG727Kphl0Nu6u79++zVq2ueve7q57eVV1Pv5f9bkUEZmZmzRWqHYCZmbVPThBmZlaWE4SZmZXlBGFmZmU5QZiZWVldqh3AwTJw4MAYNmxYtcMwM+tQli1btjEiBpXb1mkSxLBhw1i6dGm1wzAz61AkvdTSNncxmZlZWU4QZmZWlhOEmZmV1WnGIMwsnV27dlFfX8/27durHYq1oR49ejBkyBC6du2aeh8nCLOcqa+vp2/fvgwbNgxJ1Q7H2kBEsGnTJurr6xk+fHjq/TLtYpJ0lqTnJdVJurbM9o9JWi5pt6Tzm22bJumF5DYtyzjN8mT79u0MGDDAySFHJDFgwICKW42ZJQhJNcBs4GxgJDBF0shm1f4EXATc22zf9wE3AGOBMcANkg7NKlazvHFyyJ/WvOdZdjGNAeoiYg2ApPuAScBzjRUiYm2yraHZvmcCv4yIzcn2XwJnAT8+2EH+ect27n2qxWnA1kzv7l24+JThdOvi+Q1mnV2WCeJI4OWSx/UUWwSt3ffI5pUkTQemAwwdOrRVQb66dTu3L65r1b5503jpkJOOPpTaYe+rbjDWod10003ce++91NTUUCgU+P73v8/YsWO57bbbmD59Or169Trg15g3bx5Lly7ljjvueE/5sGHD6Nu3L5I49NBDmT9/PkcfffRe+8+dO5dbb70VSTQ0NHDTTTcxadKksq/12GOP8Z3vfIcHH3ywVbGuXbuWc845h5UrV+6zXuMJwQMHDmzV61QqywRRrj2T9upEqfaNiDnAHIDa2tpWXflo1FH9+e9v/k1rds2d/3pxI5+98yl2N/giU9Z6TzzxBA8++CDLly+ne/fubNy4kZ07dwJw22238bnPfe6gJIh9Wbx4MQMHDuSGG25g1qxZ3Hnnne/ZXl9fz0033cTy5cvp168f27ZtY8OGDZnGlIU9e/ZQU1PT6v2z7CeoB44qeTwEWNcG+1pGCkkfZoMThB2A9evXM3DgQLp37w7AwIEDOeKII/jud7/LunXrmDBhAhMmTADg0UcfZdy4cYwePZoLLriAbdu2AcX/pK+55hrGjBnDmDFjqKtrXS/AuHHjeOWVV/Yqf+211+jbty99+vQBoE+fPk2zf+rq6jj99NMZNWoUo0eP5sUXXwRg27ZtnH/++XzoQx9i6tSpNF6tc9myZYwfP56TTjqJM888k/Xr1zeVjxo1inHjxjF79uym1543bx6XXXZZ0+NzzjmHxx57bK8Yf/SjHzFmzBhOPPFELr30Uvbs2dMU64wZMxg7dixPPPFEq45LoyxbEEuAEZKGA68Ak4HPptz3EeB/lwxMnwF8/eCHaJVoShDOD53GN/7fKp5bt/WgPufIIw7hhnOPa3H7GWecwcyZMzn22GM5/fTTufDCCxk/fjxXXHEFt9xyS9N/9xs3bmTWrFn86le/onfv3tx8883ccsstzJgxA4BDDjmEp59+mvnz53PllVe2qnvn4Ycf5lOf+tRe5aNGjeLwww9n+PDhnHbaaXz605/m3HPPBWDq1Klce+21nHfeeWzfvp2GhgZefvllfve737Fq1SqOOOIITjnlFB5//HHGjh3L5ZdfzgMPPMCgQYNYsGAB1113HXPnzuXiiy/m9ttvZ/z48Xzta1+rKO7Vq1ezYMECHn/8cbp27cqXvvQl7rnnHr7whS/w1ltvcfzxxzNz5syKj0dzmSWIiNgt6TKKX/Y1wNyIWCVpJrA0IhZK+ijwU+BQ4FxJ34iI4yJis6R/pJhkAGY2Dlhb9RSSjr8GX8fcDkCfPn1YtmwZv/nNb1i8eDEXXngh3/rWt7joooveU+/JJ5/kueee45RTTgFg586djBs3rmn7lClTmn5eddVVFcUwYcIEXn31VQ477DBmzZq11/aamhoefvhhlixZwqJFi7jqqqtYtmwZX/nKV3jllVc477zzgOLJZ43GjBnDkCFDADjxxBNZu3Yt/fv3Z+XKlUycOBEodvkMHjyYLVu28MYbbzB+/HgAPv/5z/OLX/widfyLFi1i2bJlfPSjHwXgnXfe4bDDDmuK/TOf+UxFx6MlmZ4oFxEPAQ81K5tRcn8Jxe6jcvvOBeZmGZ9VRk0tCCeIzmJf/+lnqaamhlNPPZVTTz2VE044gbvuumuvBBERTJw4kR//uPzkxdJpm5VO4Vy8eDG9e/fmoosuYsaMGdxyyy1ln7+xC2vixIlcfPHFXH311S0+Z2OXWePvt3v3biKC4447bq+unjfeeKPFmLt06UJDw7sTO8uduxARTJs2jW9+85t7bevRo8cBjTuU8lxFS60maUI4P9iBeP7553nhhReaHq9YsaJpFlHfvn158803ATj55JN5/PHHm8YX3n77bf74xz827bdgwYKmn6Uti7R69uzJbbfdxvz589m8+b0dFOvWrWP58uV7xXjIIYcwZMgQfvaznwGwY8cO3n777RZf44Mf/CAbNmxoShC7du1i1apV9O/fn379+vHb3/4WgHvuuadpn2HDhrFixYqmrqunn356r+c97bTTuP/++3nttdcA2Lx5My+9dPCn63upDUutsYtpjwch7ABs27aNyy+/nDfeeIMuXbpwzDHHMGfOHACmT5/O2WefzeDBg1m8eDHz5s1jypQp7NixA4BZs2Zx7LHHAsUv57Fjx9LQ0NBiK2PevHlNX+ZQ7LYqNXjwYKZMmcLs2bO5/vrrm8p37drFV7/6VdatW0ePHj0YNGgQ3/ve9wC4++67ufTSS5kxYwZdu3blJz/5SYu/a7du3bj//vu54oor2LJlC7t37+bKK6/kuOOO44c//CFf/OIX6dWrF2eeeWbTPqeccgrDhw/nhBNO4Pjjj2f06NF7Pe/IkSOZNWsWZ5xxBg0NDXTt2pXZs2eXna57IBSd5N/B2tra8AWDsrXylS2cc/tvmfP5kzjjuL+odjjWSqtXr+bDH/5wtcM4IG19PkBnUe69l7QsImrL1XcXk6WmpkHq6sZhZm3DXUyWWuM0187S6rSOa+3atdUOIRfcgrDUGgep3YLo+Jzk86c177kThKXWNEjtL5cOrUePHmzatMlJIkcarwdRet5GGu5istTkLqZOYciQIdTX13fItYWs9RqvKFcJJwhLreAT5TqFrl27VnRVMcsvdzFZak1LbTS/eoeZdUpOEJaaWxBm+eIEYakVCk4QZnniBGGpFXyinFmuOEFYau5iMssXJwhLzUttmOWLE4SlVuPzIMxyxQnCUmvsYvJy32b54ARhqfma1Gb54gRhqSn5tLiLySwfnCAsNc9iMssXJwhLzedBmOWLE4Sl5kFqs3xxgrDUfEU5s3xxgrDU3MVkli9OEJaaB6nN8sUJwlLzUhtm+eIEYalJoiBocIYwywUnCKtIQXIXk1lOOEFYRYoJotpRmFlbcIKwikie5mqWF04QVhF3MZnlR6YJQtJZkp6XVCfp2jLbu0takGx/StKwpLyrpLskPStptaSvZxmnpVdTEHsaqh2FmbWFzBKEpBpgNnA2MBKYImlks2qXAK9HxDHArcDNSfkFQPeIOAE4Cbi0MXlYdUk+D8IsL7JsQYwB6iJiTUTsBO4DJjWrMwm4K7l/P3CaJAEB9JbUBegJ7AS2ZhirpVSQPAZhlhNZJogjgZdLHtcnZWXrRMRuYAswgGKyeAtYD/wJ+E5EbG7+ApKmS1oqaemGDRsO/m9geynIJ8qZ5UWWCUJlypp/tbRUZwywBzgCGA58RdL796oYMSciaiOidtCgQQcar6XgQWqz/MgyQdQDR5U8HgKsa6lO0p3UD9gMfBZ4OCJ2RcRrwONAbYaxWkqFghOEWV5kmSCWACMkDZfUDZgMLGxWZyEwLbl/PvDrKHZw/wn4uIp6AycDf8gwVkupuNRGtaMws7aQWYJIxhQuAx4BVgP/FhGrJM2U9Mmk2g+AAZLqgKuBxqmws4E+wEqKieaHEfH7rGK19NzFZJYfXbJ88oh4CHioWdmMkvvbKU5pbb7ftnLlVn1easMsP3wmtVXES22Y5YcThFWkpiD2OEGY5YIThFXEXUxm+eEEYRXxUhtm+eEEYRXxUhtm+eEEYRXxeRBm+eEEYRUpyIPUZnnhBGEVcReTWX44QVhFCgWv5mqWF04QVhEvtWGWH04QVhH5PAiz3HCCsIrUCBqcIcxywQnCKuIuJrP8cIKwijhBmOWHE4RVRL4mtVluOEFYRXwehFl+OEFYRWoKYo+bEGa54ARhFXEXk1l+OEFYRdzFZJYfThBWkYJbEGa54QRhFfE0V7P8cIKwihQ8SG2WG/tNEJJ6Sbpe0p3J4xGSzsk+NGuPCgI3IMzyIU0L4ofADmBc8rgemJVZRNauuYvJLD/SJIgPRMS3gV0AEfEOoEyjsnbLCcIsP9IkiJ2SegIBIOkDFFsUlkNyF5NZbnRJUedG4GHgKEn3AKcAF2cZlLVfNQVfk9osL/abICLiUUnLgJMpdi19OSI2Zh6ZtUvuYjLLjzSzmBZFxKaI+HlEPBgRGyUtaovgrP2RoKGh2lGYWVtosQUhqQfQCxgo6VDeHZg+BDiiDWKzdshLbZjlx75aEJcCy4APJT8bbw8As9M8uaSzJD0vqU7StWW2d5e0INn+lKRhJds+IukJSaskPZskLKsyL7Vhlh8ttiAi4l+Af5F0eUTcXukTS6qhmEgmUjx3YomkhRHxXEm1S4DXI+IYSZOBm4ELJXUBfgR8PiKekTSAZJqtVZcHqc3yI80g9e2SjgdGAj1KyufvZ9cxQF1ErAGQdB8wCShNEJMozpICuB+4Q5KAM4DfR8QzyWttSvXbWObkLiaz3EgzSH0DcHtymwB8G/hkiuc+Eni55HF9Ula2TkTsBrYAA4BjgZD0iKTlkv5XC7FNl7RU0tINGzakCMkOlLuYzPIjzYly5wOnAX+OiIuBUUD3FPuVO9u6+VdLS3W6AP8DmJr8PE/SaXtVjJgTEbURUTto0KAUIdmB8jRXs/xIkyDeiYgGYLekQ4DXgPen2K8eOKrk8RBgXUt1knGHfsDmpPw/ImJjRLwNPASMTvGalrGCRIObEGa5kCZBLJXUH7iT4iym5cDTKfZbAoyQNFxSN2AysLBZnYXAtOT++cCvo9jB/QjwkWQl2S7AeN47dmFVUmxBVDsKM2sL+xykTgaMvxkRbwDfk/QwcEhE/H5/TxwRuyVdRvHLvgaYGxGrJM0ElkbEQuAHwN2S6ii2HCYn+74u6RaKSSaAhyLi563/Ne1gKY5BOEOY5cE+E0REhKSfASclj9dW8uQR8RDF7qHSshkl97cDF7Sw748oTnW1dqRQ8BiEWV6k6WJ6UtJHM4/EOgR5FpNZbqRZzXUCcKmkl4C3KM48ioj4SKaRWbvkpTbM8iNNgjg78yisw6iRr0ltlhdpzqR+qS0CsY7BJ8qZ5UeaMQizJsWJbbibySwHnCCsIoUkQbgVYdb5OUFYRQrJ4iie6mrW+e13DELSm+y9htIWYCnwlcbVWi0fCoXGFoQThFlnl2YW0y0U11C6l+IU18nAXwDPA3OBU7MKztqfpi4mX3bUrNNL08V0VkR8PyLejIitETEH+ERELAAOzTg+a2fcxWSWH2kSRIOkv5VUSG5/W7LN3xI58+4gtd96s84uTYKYCnye4jLfryb3PyepJ3BZhrFZO6SmFkR14zCz7KU5UW4NcG4Lm397cMOx9q6m4PMgzPIizSymQcD/BIaV1o+IL2YXlrVXjV1MXm7DrPNLM4vpAeA3wK+APdmGY+1dwV1MZrmRJkH0iohrMo/EOgQvtWGWH2kGqR+U9InMI7EOwUttmOVHmgTxZYpJ4h1JWyW9KWlr1oFZ+1STfGI8zdWs80szi6lvWwRiHYM8SG2WGy0mCEkfiog/SBpdbntELM8uLGuvCk1jEFUOxMwyt68WxNXAdOCfy2wL4OOZRGTtmpfaMMuPFhNERExPfk5ou3CsvfNSG2b5kWaaK5L+ir1PlJufUUzWjr273HeVAzGzzKU5k/pu4APACt49US4AJ4gccheTWX6kaUHUAiPDZ0YZ7mIyy5M050GspHiBILN3WxC+YJBZp5emBTEQeE7S08COxsKI+GRmUVm7JbcgzHIjTYK4MesgrOOo8XkQZrmxzwQhqQa4PiJOb6N4rJ0rJJ2Se5whzDq9fY5BRMQe4G1J/dooHmvn3MVklh9pupi2A89K+iXwVmNhRFyRWVTWbhW83LdZbqSZxfRz4HrgP4FlJbf9knSWpOcl1Um6tsz27pIWJNufkjSs2fahkrZJ+mqa17Ps+YJBZvmRZjXXu1rzxMn4xWxgIlAPLJG0MCKeK6l2CfB6RBwjaTJwM3BhyfZbgV+05vUtG42D1A3OEGad3n5bEJJGSLpf0nOS1jTeUjz3GKAuItZExE7gPmBSszqTgMYEdD9wmpJObkmfAtYAq9L+Mpa9puW+3cVk1uml6WL6IfCvwG5gAsUlNu5Osd+RwMslj+uTsrJ1ImI3sAUYIKk3cA3wjX29gKTpkpZKWrphw4YUIdmBauxicn4w6/zSJIieEbEIUES8FBE3km6pb5Upa/610lKdbwC3RsS2fb1ARMyJiNqIqB00aFCKkOxAvbtYnzOEWWeXahaTpALwgqTLgFeAw1LsVw8cVfJ4CLCuhTr1kroA/YDNwFjgfEnfBvoDDZK2R8QdKV7XMuRBarP8SJMgrgR6AVcA/0ixm2laiv2WACMkDaeYVCYDn21WZ2HyXE8A5wO/ThYF/OvGCpJuBLY5ObQPXqzPLD/SzGJaAiApIuLitE8cEbuTFscjQA0wNyJWSZoJLI2IhcAPgLsl1VFsOUxuzS9hbafgWUxmuZHmehDjKH6R9wGGShoFXBoRX9rfvhHxEPBQs7IZJfe3Axfs5zlu3N/rWNt5twVR5UDMLHNpBqlvA84ENgFExDPAx7IMytov+YJBZrmRJkEQES83K9pTtqJ1el5qwyw/0gxSv5xckzokdaM4WL0627CsvarxNanNciNNC+LvgH+geFJbPXAisN/xB+ucGqe57nGGMOv00sxi2ghMLS2TdCXFsQnLGS/3bZYfqcYgyrj6oEZhHYaX2jDLj9YmiHJLZFgO+EQ5s/xobYLwt0NOeZDaLD9aHIOQ9CblE4GAnplFZO1a03kQzhBmnV6LCSIi+rZlINYxuIvJLD9a28VkOeWlNszywwnCKlLwUhtmueEEYRVpvGCQl9ow6/ycIKwijV1MPpParPNzgrCK+IpyZvnhBGEV8VIbZvnhBGEV8VIbZvnhBGEVefdMamcIs87OCcIq0jRI7QRh1uk5QVhF5C4ms9xwgrCKNJ1J7WlMZp2eE4RVxEttmOWHE4RVxEttmOWHE4RVRBKSE4RZHjhBWMUKkhOEWQ44QVjFCvIYhFkeOEFYxeQWhFkuOEFYxWoknwdhlgNOEFaxgrzct1keOEFYxTxIbZYPmSYISWdJel5SnaRry2zvLmlBsv0pScOS8omSlkl6Nvn58SzjtMpIXmrDLA8ySxCSaoDZwNnASGCKpJHNql0CvB4RxwC3Ajcn5RuBcyPiBGAacHdWcVrlCgW3IMzyIMsWxBigLiLWRMRO4D5gUrM6k4C7kvv3A6dJUkT8LiLWJeWrgB6SumcYq1Wgxl1MZrmQZYI4Eni55HF9Ula2TkTsBrYAA5rV+Qzwu4jYkVGcViFJ7GmodhRmlrUuGT63ypQ1/7dzn3UkHUex2+mMsi8gTQemAwwdOrR1UVrFCoJwC8Ks08uyBVEPHFXyeAiwrqU6kroA/YDNyeMhwE+BL0TEi+VeICLmRERtRNQOGjToIIdvLfEsJrN8yDJBLAFGSBouqRswGVjYrM5CioPQAOcDv46IkNQf+Dnw9Yh4PMMYrRW81IZZPmSWIJIxhcuAR4DVwL9FxCpJMyV9Mqn2A2CApDrgaqBxKuxlwDHA9ZJWJLfDsorVKuNZTGb5kOUYBBHxEPBQs7IZJfe3AxeU2W8WMCvL2Kz1CpKvKGeWAz6T2irmLiazfHCCsIp5kNosH5wgrGJeasMsH5wgrGI1HqQ2ywUnCKtYQfJy32Y54ARhFSteUa7aUZhZ1pwgrGJeasMsH5wgrGKexWSWD04QVrHimdTVjsLMsuYEYRUrnijnDGHW2TlBWMXcxWSWD04QVrGCoMEXDDLr9JwgrGJyC8IsFzJdzdU6pxqJV954h7v+a221Q+kQ/uoDAxhxeN9qh2FWMScIq9gR/XvyxJpN3LBwVbVD6RDGHzuIu744ptphmFXMCcIq9k/nf4Tr/ubD1Q6jQ7hqwQpee3NHtcMwaxUnCKtYoSDe17tbtcPoEAb368Hq9VurHYZZq3iQ2ixDA/p0Y/NbO30FPuuQnCDMMjSgd3d2NwRbt++qdihmFXOCMMvQgD7FrriN23ZWORKzyjlBmGVoYJ/uAGza5oFq63icIMwy1NiC2PSWWxDW8ThBmGWocbaXWxDWETlBmGXofb08BmEdlxOEWYa61BQ4tFdXNr3lFoR1PE4QZhkb0Kc7m9yCsA7ICcIsYwN6d3OCsA7JCcIsYwP7dGeju5isA3KCMMvYgD5uQVjH5ARhlrEBvbuz5Z1d7Nzty/BZx+IEYZaxxpPlXn/brQjrWDJd7lvSWcC/ADXA/4mIbzXb3h2YD5wEbAIujIi1ybavA5cAe4ArIuKRLGM1y8rAJEHMeGAl/Xp2rXI07V//Xt24/OPH0LeHj1W1ZZYgJNUAs4GJQD2wRNLCiHiupNolwOsRcYykycDNwIWSRgKTgeOAI4BfSTo2IvZkFa9ZVo4/sh8fGNSb39dvqXYoHcKrW7ezev1W5l70UbrWuJOjmrJsQYwB6iJiDYCk+4BJQGmCmATcmNy/H7hDkpLy+yJiB/DfkuqS53siw3jNMjHk0F4s+sqp1Q6jw7jv6T9x7b8/y6n/9Bi9utVUO5wO4dQPDuK6vxl50J83ywRxJPByyeN6YGxLdSJit6QtwICk/Mlm+x7Z/AUkTQemAwwdOvSgBW5m1TN5zFD2RPB43cZqh9JhHH5Ij0yeN8sEoTJlzS+r1VKdNPsSEXOAOQC1tbW+ZJdZJzF17NFMHXt0tcPIvSw7+OqBo0oeDwHWtVRHUhegH7A55b5mZpahLBPEEmCEpOGSulEcdF7YrM5CYFpy/3zg1xERSflkSd0lDQdGAE9nGKuZmTWTWRdTMqZwGfAIxWmucyNilaSZwNKIWAj8ALg7GYTeTDGJkNT7N4oD2ruBf/AMJjOztqXiP+wdX21tbSxdurTaYZiZdSiSlkVEbbltnmRsZmZlOUGYmVlZThBmZlaWE4SZmZXVaQapJW0AXjqApxgItMdTNx1XZRxX5dprbI6rMq2N6+iIGFRuQ6dJEAdK0tKWRvKryXFVxnFVrr3G5rgqk0Vc7mIyM7OynCDMzKwsJ4h3zal2AC1wXJVxXJVrr7E5rsoc9Lg8BmFmZmW5BWFmZmU5QZiZWVm5TxCSzpL0vKQ6SddWMY6jJC2WtFrSKklfTspvlPSKpBXJ7RNVim+tpGeTGJYmZe+T9EtJLyQ/D23jmD5YclxWSNoq6cpqHDNJcyW9JmllSVnZ46Oi7yafud9LGt3Gcf2TpD8kr/1TSf2T8mGS3ik5bt/LKq59xNbieyfp68kxe17SmW0c14KSmNZKWpGUt9kx28d3RHafs4jI7Y3iMuQvAu8HugHPACOrFMtgYHRyvy/wR2AkxWt2f7UdHKu1wMBmZd8Grk3uXwvcXOX38s/A0dU4ZsDHgNHAyv0dH+ATwC8oXjnxZOCpNo7rDKBLcv/mkriGldar0jEr+94lfwvPAN2B4cnfbU1bxdVs+z8DM9r6mO3jOyKzz1neWxBjgLqIWBMRO4H7gEnVCCQi1kfE8uT+m8BqylyHu52ZBNyV3L8L+FQVYzkNeDEiDuRs+laLiP+keE2TUi0dn0nA/Ch6EugvaXBbxRURj0bE7uThkxSv2NjmWjhmLZkE3BcROyLiv4E6in+/bRqXJAF/C/w4i9fel318R2T2Oct7gjgSeLnkcT3t4EtZ0jDgL4GnkqLLkibi3LbuxikRwKOSlkmanpQdHhHrofjhBQ6rUmxQvNhU6R9tezhmLR2f9vS5+yLF/zIbDZf0O0n/IemvqxRTufeuvRyzvwZejYgXSsra/Jg1+47I7HOW9wShMmVVnfcrqQ/wf4ErI2Ir8K/AB4ATgfUUm7fVcEpEjAbOBv5B0seqFMdeVLyk7SeBnyRF7eWYtaRdfO4kXUfxio33JEXrgaER8ZfA1cC9kg5p47Baeu/axTEDpvDef0Ta/JiV+Y5osWqZsoqOWd4TRD1wVMnjIcC6KsWCpK4U3/h7IuLfASLi1YjYExENwJ1k1Kzen4hYl/x8DfhpEserjU3W5Odr1YiNYtJaHhGvJjG2i2NGy8en6p87SdOAc4CpkXRYJ903m5L7yyj28x/blnHt471rD8esC/BpYEFjWVsfs3LfEWT4Oct7glgCjJA0PPkvdDKwsBqBJH2bPwBWR8QtJeWlfYbnASub79sGsfWW1LfxPsVBzpUUj9W0pNo04IG2ji3xnv/q2sMxS7R0fBYCX0hmmZwMbGnsImgLks4CrgE+GRFvl5QPklST3H8/MAJY01ZxJa/b0nu3EJgsqbuk4UlsT7dlbMDpwB8ior6xoC2PWUvfEWT5OWuL0ff2fKM40v9Hipn/uirG8T8oNv9+D6xIbp8A7gaeTcoXAoOrENv7Kc4geQZY1XicgAHAIuCF5Of7qhBbL2AT0K+krM2PGcUEtR7YRfE/t0taOj4Um/6zk8/cs0BtG8dVR7FvuvFz9r2k7meS9/cZYDlwbhWOWYvvHXBdcqCff88AAAHFSURBVMyeB85uy7iS8nnA3zWr22bHbB/fEZl9zrzUhpmZlZX3LiYzM2uBE4SZmZXlBGFmZmU5QZiZWVlOEGZmVpYThFkFJO3Re1eQPWgrACcrg1brnA2zvXSpdgBmHcw7EXFitYMwawtuQZgdBMk1Am6W9HRyOyYpP1rSomTxuUWShiblh6t4LYZnkttfJU9VI+nOZL3/RyX1rNovZbnnBGFWmZ7NupguLNm2NSLGAHcAtyVld1BccvkjFBfF+25S/l3gPyJiFMVrD6xKykcAsyPiOOANimfqmlWFz6Q2q4CkbRHRp0z5WuDjEbEmWVDtzxExQNJGistF7ErK10fEQEkbgCERsaPkOYYBv4yIEcnja4CuETEr+9/MbG9uQZgdPNHC/ZbqlLOj5P4ePE5oVeQEYXbwXFjy84nk/n9RXCUYYCrw2+T+IuDvASTVVOG6C2b75f9OzCrTU8kF6xMPR0TjVNfukp6i+I/XlKTsCmCupK8BG4CLk/IvA3MkXUKxpfD3FFcQNWs3PAZhdhAkYxC1EbGx2rGYHSzuYjIzs7LcgjAzs7LcgjAzs7KcIMzMrCwnCDMzK8sJwszMynKCMDOzsv4/N5mmVdVM3LIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "torch.manual_seed(1)\n",
    "\n",
    "LR = 0.1\n",
    "iteration = 10\n",
    "max_epoch = 200\n",
    "# ------------------------------ fake data and optimizer  ------------------------------\n",
    "\n",
    "weights = torch.randn((1), requires_grad=True)\n",
    "target = torch.zeros((1))\n",
    "\n",
    "optimizer = optim.SGD([weights], lr=LR, momentum=0.9)\n",
    "\n",
    "# ------------------------------ 1 Step LR ------------------------------\n",
    "\n",
    "scheduler_lr = optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.1)  # 设置学习率下降策略\n",
    "\n",
    "lr_list, epoch_list = list(), list()\n",
    "for epoch in range(max_epoch):\n",
    "\n",
    "    # 获取当前lr，新版本用 get_last_lr()函数，旧版本用get_lr()函数，具体看UserWarning\n",
    "    lr_list.append(scheduler_lr.get_lr())\n",
    "    epoch_list.append(epoch)\n",
    "\n",
    "    for i in range(iteration):\n",
    "\n",
    "        loss = torch.pow((weights - target), 2)\n",
    "        loss.backward()\n",
    "\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "    scheduler_lr.step()\n",
    "\n",
    "plt.plot(epoch_list, lr_list, label=\"Step LR Scheduler\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q:如何按给定间隔调整学习率？\n",
    "- `torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.1, last_epoch=-1)`\n",
    "- milestones：设定调整时刻数\n",
    "- gamma：调整系数\n",
    "\n",
    "Q:MultiStepLR的代码示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3xU1bn/8c8zAQIYAQ1g0YBBQStCiBhADvXSUgW8gBdUlCoXFWpra23lV3x5CpRyqlbP8bRqtVhvWBWsbZW2VLB4vKFys+EO5WKUCCoEb4gQSJ7fH7MzhGQSZiCTCdnf9+uVFzNrrz37mZ1hnqy19lrb3B0REZGqIukOQEREGiYlCBERiUsJQkRE4lKCEBGRuJQgREQkribpDqCutG3b1nNzc9MdhojIYWXJkiXb3L1dvG2NJkHk5uayePHidIchInJYMbP3atqmLiYREYlLCUJEROJSghARkbgazRiESEOzZ88eiouL2bVrV7pDEaF58+bk5OTQtGnThPdRghBJkeLiYo488khyc3Mxs3SHIyHm7pSUlFBcXEznzp0T3i+lXUxmNsjM1prZejObEGf7WWb2jpntNbNhVbaNNLN1wc/IVMYpkgq7du0iOztbyUHSzszIzs5OujWbsgRhZhnAA8BgoBtwlZl1q1LtfWAU8HSVfY8GJgF9gT7AJDM7KlWxiqSKkoM0FAfzWUxlF1MfYL27bwQwsxnAUGBVRQV3Lwq2lVfZdyDwkrtvD7a/BAwCnqnrID/8bBdPL6jxMmCp4ojMJozu35lmTXR9g0hjl8oEcRywqdLzYqItgoPd97iqlcxsLDAWoFOnTgcV5Eef7+K+/1t/UPuGTcWtQ04//igKco9ObzCSEDPjO9/5Dk8++SQAe/fupUOHDvTt25e//e1vte6blZXFjh07KCoq4s033+Tqq68GYPHixUyfPp3f/OY3+9XfuXMnN9xwA8uWLcPdadOmDS+++CJ79+7l6aef5nvf+16dvKdRo0Zx4YUXMmzYvl7poqIiTjnlFE4++WRKS0spKCjgkUceqTYgW15ezo9+9CNefvllzIzmzZvz7LPP1tgvP3nyZLKysrj11lsPKtbHH3+cxYsXc//999dYp6ioiAsvvJAVK1Yc1DFSKZUJIl57JtG7EyW0r7tPA6YBFBQUHNSdj3p2bMO7d1xwMLuGzpsbtnH1wwvYW66bTB0ujjjiCFasWMFXX31FixYteOmllzjuuGp/a9WqqKiIp59+OpYgCgoKKCgoqFbv17/+NccccwzLly8HYO3atTRt2pRt27bx29/+ts4SRE1OPPFECgsLKSsr49xzz+XZZ59lxIgR+9WZOXMmmzdvZtmyZUQiEYqLizniiCNSGlddKysrIyMjo16Olcp+gmKgY6XnOcDmethXUiQS9GGWK0EcVgYPHszf//53AJ555hmuuuqq2LbJkydzzz33xJ53796doqKi/fafMGECr7/+Ovn5+dx777288sorXHjhhdWOs2XLlv2Sz8knn0xmZiYTJkxgw4YN5OfnM378eADuvvtuevfuTV5eHpMmTQKiiejrX/86I0eOJC8vj2HDhrFz586k329GRgZ9+vThgw8+iBtjhw4diESiX305OTkcdVR0ePPFF1+kV69e9OzZkwEDBsT2WbVqFeeccw4nnHDCfq2mP/zhD/Tp04f8/HzGjRtHWVkZAI899hgnnXQSZ599NvPnz4/VHzVqFM8991zseVZWVrX4ysrKGD9+fOzc/O53vwPglVde4Zvf/CZXX301PXr0SPqcHKxUJohFQFcz62xmzYDhwKwE950DnGdmRwWD0+cFZZJGsQSh/HBYGT58ODNmzGDXrl0sW7aMvn0T7emNuvPOOznzzDMpLCzklltuqbHemDFjuOuuu+jXrx//+Z//ybp162L7V/x1f/fddzN37lzWrVvHwoULKSwsZMmSJbz22mtAtNUxduxYli1bRqtWrfjtb3+b9PvdtWsXCxYsYNCgQdW2XXHFFfz1r38lPz+fn/zkJ/zrX/8CYOvWrdxwww386U9/YunSpfzxj3+M7bNmzRrmzJnDwoUL+fnPf86ePXtYvXo1M2fOZP78+RQWFpKRkcFTTz3Fli1bmDRpEvPnz+ell15i1apV1WKozSOPPELr1q1ZtGgRixYt4uGHH+bdd98FYOHChfzXf/1X0q95KFKWINx9L3AT0S/21cCz7r7SzKaY2RAAM+ttZsXA5cDvzGxlsO924BdEk8wiYErFgLWkTyTo+CvXfcwPK3l5eRQVFfHMM89w/vnnp+w4+fn5bNy4kfHjx7N9+3Z69+7N6tWrq9WbO3cuc+fO5bTTTqNXr16sWbMmlkw6duxI//79AfjOd77DG2+8kfDxK1op2dnZdOrUiby8vGp1cnJyWLt2LXfccQeRSIQBAwYwb9483n77bc4666zYWMTRR+8bY7vgggvIzMykbdu2tG/fno8++oh58+axZMkSevfuTX5+PvPmzWPjxo0sWLCAc845h3bt2tGsWTOuvPLKpM7h3LlzmT59Ovn5+fTt25eSkpLYuenTp09ScxjqQkonyrn7bGB2lbKJlR4vItp9FG/fR4FHUxmfJMdiLQgliMPNkCFDuPXWW3nllVcoKSmJlTdp0oTy8n0XER7qrO+srCwuvfRSLr30UiKRCLNnz+ayyy7br467c9tttzFu3Lj9youKiqpdipnMpZkVrZQtW7ZwzjnnMGvWLIYMGVKtXmZmJoMHD2bw4MEcc8wxPP/885x77rk1HiszMzP2OCMjg7179+LujBw5kjvuuGO/us8//3yNr1P5XLs7paWl1eq4O/fddx8DBw7cr/yVV15Jy1iJrlWUhGUETQjlh8PPmDFjmDhxYrX+69zcXN555x0A3nnnnVh3RmVHHnkkX3zxxQGPMX/+fD755BMASktLWbVqFccff3y1/QcOHMijjz7Kjh07APjggw/4+OOPAXj//fd56623gOh4yTe+8Y2k32uHDh248847q315V7zHzZujw5nl5eUsW7aM448/nn79+vHqq6/G3v/27bV3WAwYMIDnnnsuFvf27dt577336Nu3bywJ79mzZ7+uqtzcXJYsWQLACy+8wJ49e6q97sCBA3nwwQdj2/7973/z5ZdfJn0O6ooShCSsooupTIMQh52cnBxuvvnmauWXXXYZ27dvJz8/nwcffJCTTjqpWp28vDyaNGlCz549uffee2s8xoYNGzj77LPp0aMHp512GgUFBVx22WVkZ2fTv39/unfvzvjx4znvvPO4+uqr6devHz169GDYsGGxBHLKKafwxBNPkJeXx/bt27nxxhvjHmvcuHHk5OSQk5NDv379qm2/+OKL2blzJ6+//vp+5R9//DEXXXQR3bt3j72vm266iXbt2jFt2jQuvfRSevbsecCuoW7dujF16lTOO+888vLyOPfcc2MD4JMnT6Zfv358+9vfplevXrF9brjhBl599VX69OnDggUL4rYIrr/+erp160avXr3o3r0748aNY+/evbXGkkrmjeTPwYKCAtcNg1JrxQefceF9bzDtmtM579SvpTucBm/16tWccsop6Q7jsNGQ5wM0FvE+k2a2xN2rX7eMWhCSBIsNUqc3DhGpH1rNVRJWcZlrY2l11qef/3UlqzZ/Xqev2e3YVky66NQ6fc10ys3NVeuhgVELQhJWMUitFoRIOKgFIQmLDVKrBZG0xvSXvoSHWhCSMFMXU6M0a9Ys7rzzTqD60huJKiwsZPbs2QeumAKTJ0/muOOOY+LE6BSrV155hdatW5Ofn09+fj5TpkyJ1X3xxRc5+eST6dKlS+w91+a1116jV69eNGnSZL9lMgoLC+nXrx+nnnoqeXl5zJw5M7Zt1KhRdO7cOXb8wsLCAx5n0KBBtGnTptoSJu7O7bffzkknncQpp5wSW+rD3fnhD39Ily5dyMvLi12qXDFZMN4yHgdDLQhJWEQT5RqlIUOGxJ1QlozCwkIWL16c0pnatbnlllv2W3H1zDPPrLZabVlZGd///vd56aWXyMnJoXfv3gwZMoRu3arepmafTp068fjjj1dLmi1btmT69Ol07dqVzZs3c/rppzNw4EDatGkDRNeaqrza7IGMHz+enTt3xtZeqvD444+zadMm1qxZQyQSic27+Mc//sG6detYt24dCxYs4MYbb2TBggWxyYJ1lSDUgpCExZbaqHr3DmmQKha/u/766+nevTsjRozgn//8J/3796dr164sXLgQiH4J3XTTTdX237BhA4MGDeL000/nzDPPZM2aNQD88Y9/pHv37vTs2ZOzzjqL0tJSJk6cyMyZM8nPz2fmzJls376diy++mLy8PM444wyWLVsGRP/aHzNmTMKL35WVlTFq1Ci6d+9Ojx49ap2HcSALFy6kS5cunHDCCTRr1ozhw4fzwgsv1LpPbm4ueXl5scX9Kpx00kl07doVgGOPPZb27duzdevWg45twIABHHnkkdXKH3zwQSZOnBg7fvv27YHoRLtrr70WM+OMM87g008/ZcuWLQd9/JooQUjC1II4/Kxfv56bb76ZZcuWsWbNGp5++mneeOMN7rnnHn75y1/Wuu/YsWO57777WLJkCffcc09sue4pU6YwZ84cli5dyqxZs2jWrBlTpkzhyiuvpLCwkCuvvJJJkyZx2mmnsWzZMn75y19y7bXXxl43mcXvCgsL+eCDD1ixYgXLly9n9OjRADz00EM89NBDNcb+1ltv0bNnTwYPHszKlSuB6Iztjh33LRKdk5MTd8XXZC1cuJDS0lJOPPHEWNntt99OXl4et9xyC7t37z7o196wYQMzZ86koKCAwYMHx9ZlStV7qUpdTJKwSEQJ4nDTuXPn2PIap556KgMGDMDM6NGjR7VlvSvbsWMHb775JpdffnmsrOKLrn///owaNYorrriCSy+9NO7+b7zxBn/6058A+Na3vkVJSQmfffYZsG/xu8zMzLiL3wF89dVXtG/fnosuuoiNGzfygx/8gAsuuIDzzjsPgO9+97s1xt6rVy/ee+89srKymD17NhdffDHr1q2LO3Z2qLeE3bJlC9dccw1PPPFE7K/8O+64g6997WuUlpYyduxY7rrrrtj4SLJ2795N8+bNWbx4MX/+858ZM2YMr7/+ekreSzxqQUjCIpood9ipvNBcJBKJPY9EIrUu4VBeXk6bNm0oLCyM/VSszPrQQw8xdepUNm3aRH5+/n6L/1Wo7QustsXvKo61du1aJk+ezFFHHcXSpUs555xzeOCBB7j++usP+J5btWoV64M///zz2bNnD9u2bSMnJ4dNm/bdqLK4uJhjjz32gK9Xk88//5wLLriAqVOncsYZZ8TKO3TogJmRmZnJ6NGjY115ByMnJye22OEll1wS66qr6/dSEyUISZi6mMKjVatWdO7cObbYnLuzdOlSINrt0bdvX6ZMmULbtm3ZtGlTtQX5zjrrLJ566ikgelVR27ZtadWqVY3Hq2nxu23btlFeXs5ll13GL37xi9jVOrX58MMPYwlq4cKFlJeXk52dTe/evVm3bh3vvvsupaWlzJgxIzY4f9ttt/GXv/wl4fNTWlrKJZdcwrXXXrtfKwuIjQW4O88//zzdu3ePxVK5qy0RF198MS+//DIAr776amytrCFDhjB9+nTcnbfffpvWrVvToUOHpF47EepikoRpqY1weeqpp7jxxhuZOnUqe/bsYfjw4fTs2ZPx48fHumwGDBhAz5496dSpE3feeSf5+fncdtttTJ48mdGjR5OXl0fLli154oknaj1W5cXvysvLadq0KQ888AAtWrRg9OjRsWWyK1ZorRh/iNfV9Nxzz/Hggw/SpEkTWrRowYwZMzAzmjRpwv3338/AgQMpKytjzJgxnHpqdH7K8uXL417JtWjRIi655BI++eQT/vrXvzJp0iRWrlzJs88+y2uvvUZJSQmPP/44EB3sz8/PZ8SIEWzduhV3Jz8/Pxbr+++/T4sWLeK+/4qLAHbs2EFOTg6PPPIIAwcOZMKECYwYMYJ7772XrKwsfv/73wPRltHs2bPp0qULLVu25LHHHjvQr/OgaLE+SVjJjt2cPvWfTBl6Ktf2y013OA2eFuurH5MnTyYrK2u/y1yTNXDgQObMSe1NK8ePH88111wT90ZGdS0rKyu2nHplWqxPUqaii0nLfUtDkpWVxbRp0w56IBhIeXKA6NyIVCeHiolyxxxzTJ28nrqYJGG6J3Xy3D0lV5fIPrfeeushtR4ak4qJcvEcTG+RWhCSMAs+LY2lWzLVmjdvTklJic6XpJ27U1JSQvPmzZPaTy0ISZiuYkpOTk4OxcXFhzTDVqSuNG/enJycnKT2UYKQhGkeRHKaNm1K586d0x2GyEFTF5MkTIPUIuGiBCEJ0x3lRMJFCUISpi4mkXBRgpCEaZBaJFyUICRhWmpDJFyUICRhZkbEoFwZQiQUlCAkKREzdTGJhIQShCQlmiDSHYWI1AclCEmKmS5zFQkLJQhJirqYRMIjpQnCzAaZ2VozW29mE+JszzSzmcH2BWaWG5Q3NbMnzGy5ma02s9tSGackLiNilJWnOwoRqQ8pSxBmlgE8AAwGugFXmVm3KtWuAz5x9y7AvcBdQfnlQKa79wBOB8ZVJA9JLzPNgxAJi1S2IPoA6919o7uXAjOAoVXqDAUq7kX4HDDAoovnO3CEmTUBWgClwOcpjFUSFDHTGIRISKQyQRwHbKr0vDgoi1vH3fcCnwHZRJPFl8AW4H3gHnffXvUAZjbWzBab2WItqVw/IqaJciJhkcoEEe82WlW/Wmqq0wcoA44FOgM/MbMTqlV0n+buBe5e0K5du0ONVxKgQWqR8EhlgigGOlZ6ngNsrqlO0J3UGtgOXA286O573P1jYD4Q96baUr8iESUIkbBIZYJYBHQ1s85m1gwYDsyqUmcWMDJ4PAx42aMd3O8D37KoI4AzgDUpjFUSFF1qI91RiEh9SFmCCMYUbgLmAKuBZ919pZlNMbMhQbVHgGwzWw/8GKi4FPYBIAtYQTTRPObuy1IVqyROXUwi4ZHSW466+2xgdpWyiZUe7yJ6SWvV/XbEK5f001IbIuGhmdSSFC21IRIeShCSlIyIUaYEIRIKShCSFHUxiYSHEoQkRUttiISHEoQkRUttiISHEoQkRfMgRMJDCUKSEjENUouEhRKEJEVdTCLhoQQhSYlEtJqrSFgoQUhStNSGSHgoQUhSTPMgREJDCUKSkmFQrgwhEgpKEJIUdTGJhIcShCRFCUIkPJQgJCmme1KLhIYShCRF8yBEwkMJQpKSETHK1IQQCQUlCEmKuphEwkMJQpKiLiaR8FCCkKRE1IIQCQ0lCEmKLnMVCQ8lCElKRIPUIqFxwARhZi3N7Gdm9nDwvKuZXZj60KQhihioASESDom0IB4DdgP9gufFwNSURSQNmrqYRMIjkQRxorv/CtgD4O5fAZbSqKTBUoIQCY9EEkSpmbUAHMDMTiTaopAQMnUxiYRGkwTqTAZeBDqa2VNAf2B0KoOShisjontSi4TFAROEu881syXAGUS7lm52920pj0waJHUxiYRHIlcxzXP3Enf/u7v/zd23mdm8+ghOGh4zKC9PdxQiUh9qbEGYWXOgJdDWzI5i38B0K+DYeohNGiAttSESHrW1IMYBS4CvB/9W/LwAPJDIi5vZIDNba2brzWxCnO2ZZjYz2L7AzHIrbcszs7fMbKWZLQ8SlqSZltoQCY8aWxDu/mvg12b2A3e/L9kXNrMMoonkXKJzJxaZ2Sx3X1Wp2nXAJ+7excyGA3cBV5pZE+APwDXuvtTMsgkus5X00iC1SHgkMkh9n5l1B7oBzSuVTz/Arn2A9e6+EcDMZgBDgcoJYijRq6QAngPuNzMDzgOWufvS4FglCb0bSTlTF5NIaCQySD0JuC/4+SbwK2BIAq99HLCp0vPioCxuHXffC3wGZAMnAW5mc8zsHTP7fzXENtbMFpvZ4q1btyYQkhwqdTGJhEciE+WGAQOAD919NNATyExgv3izrat+tdRUpwnwDWBE8O8lZjagWkX3ae5e4O4F7dq1SyAkOVS6zFUkPBJJEF+5ezmw18xaAR8DJySwXzHQsdLzHGBzTXWCcYfWwPag/FV33+buO4HZQK8EjikpFjGjXE0IkVBIJEEsNrM2wMNEr2J6B1iYwH6LgK5m1tnMmgHDgVlV6swCRgaPhwEve7SDew6QF6wk2wQ4m/3HLiRNoi2IdEchIvWh1kHqYMD4Dnf/FHjIzF4EWrn7sgO9sLvvNbObiH7ZZwCPuvtKM5sCLHb3WcAjwJNmtp5oy2F4sO8nZvY/RJOMA7Pd/e8H/zalrkTHIJQhRMKg1gTh7m5mzwOnB8+Lknlxd59NtHuoctnESo93AZfXsO8fiF7qKg1IJKIxCJGwSKSL6W0z653ySOSwYLqKSSQ0ElnN9ZvAODN7D/iS6JVH7u55KY1MGiQttSESHokkiMEpj0IOGxmme1KLhEUiM6nfq49A5PCgiXIi4ZHIGIRITPTCNtTNJBICShCSlEiQINSKEGn8lCAkKZFgcRRd6irS+B1wDMLMvqD6GkqfAYuBn1Ss1irhEIlUtCCUIEQau0SuYvofomsoPU30EtfhwNeAtcCjwDmpCk4anlgXk247KtLoJdLFNMjdf+fuX7j75+4+DTjf3WcCR6U4Pmlg1MUkEh6JJIhyM7vCzCLBzxWVtulbImT2DVLrVy/S2CWSIEYA1xBd5vuj4PF3zKwFcFMKY5MGyGItiPTGISKpl8hEuY3ARTVsfqNuw5GGLiOieRAiYZHIVUztgBuA3Mr13X1M6sKShqqii0nLbYg0folcxfQC8DrwT6AsteFIQxdRF5NIaCSSIFq6+09THokcFrTUhkh4JDJI/TczOz/lkchhQUttiIRHIgniZqJJ4isz+9zMvjCzz1MdmDRMGcEnRpe5ijR+iVzFdGR9BCKHB9MgtUho1JggzOzr7r7GzHrF2+7u76QuLGmoIrExiDQHIiIpV1sL4sfAWOC/42xz4FspiUgaNC21IRIeNSYIdx8b/PvN+gtHGjottSESHolc5oqZ/QfVJ8pNT1FM0oDtW+47zYGISMolMpP6SeBEoJB9E+UcUIIIIXUxiYRHIi2IAqCba2aUoC4mkTBJZB7ECqI3CBLZ14LQDYNEGr1EWhBtgVVmthDYXVHo7kNSFpU0WKYWhEhoJJIgJqc6CDl8ZGgehEho1JogzCwD+Jm7f7ue4pEGLhJ0SpYpQ4g0erWOQbh7GbDTzFrXUzzSwKmLSSQ8Euli2gUsN7OXgC8rCt39hymLShqsiJb7FgmNRK5i+jvwM+A1YEmlnwMys0FmttbM1pvZhDjbM81sZrB9gZnlVtneycx2mNmtiRxPUk83DBIJj0RWc33iYF44GL94ADgXKAYWmdksd19Vqdp1wCfu3sXMhgN3AVdW2n4v8I+DOb6kRsUgdbkyhEijd8AWhJl1NbPnzGyVmW2s+EngtfsA6919o7uXAjOAoVXqDAUqEtBzwAALOrnN7GJgI7Ay0TcjqRdb7ltdTCKNXiJdTI8BDwJ7gW8SXWLjyQT2Ow7YVOl5cVAWt4677wU+A7LN7Ajgp8DPazuAmY01s8Vmtnjr1q0JhCSHqqKLSflBpPFLJEG0cPd5gLn7e+4+mcSW+rY4ZVW/Vmqq83PgXnffUdsB3H2auxe4e0G7du0SCEkO1b7F+pQhRBq7hK5iMrMIsM7MbgI+ANonsF8x0LHS8xxgcw11is2sCdAa2A70BYaZ2a+ANkC5me1y9/sTOK6kkAapRcIjkQTxI6Al8EPgF0S7mUYmsN8ioKuZdSaaVIYDV1epMyt4rbeAYcDLwaKAZ1ZUMLPJwA4lh4ZBi/WJhEciVzEtAjAzd/fRib6wu+8NWhxzgAzgUXdfaWZTgMXuPgt4BHjSzNYTbTkMP5g3IfUnoquYREIjkftB9CP6RZ4FdDKznsA4d//egfZ199nA7CplEys93gVcfoDXmHyg40j92deCSHMgIpJyiQxS/y8wECgBcPelwFmpDEoaLtMNg0RCI5EEgbtvqlJUFreiNHpaakMkPBIZpN4U3JPazawZ0cHq1akNSxqqDN2TWiQ0EmlBfBf4PtFJbcVAPnDA8QdpnCoucy1ThhBp9BK5imkbMKJymZn9iOjYhISMlvsWCY+ExiDi+HGdRiGHDS21IRIeB5sg4i2RISGgiXIi4XGwCULfDiGlQWqR8KhxDMLMviB+IjCgRcoikgYtNg9CGUKk0asxQbj7kfUZiBwe1MUkEh4H28UkIaWlNkTCQwlCkhLRUhsioaEEIUmpuGGQltoQafyUICQpFV1Mmkkt0vgpQUhSdEc5kfBQgpCkaKkNkfBQgpCkaKkNkfBQgpCk7JtJrQwh0tgpQUhSYoPUShAijZ4ShCTF1MUkEhpKEJKU2ExqXcYk0ugpQUhStNSGSHgoQUhStNSGSHgoQUhSzAwzJQiRMFCCkKRFzJQgREJACUKSFjGNQYiEgRKEJM3UghAJBSUISVqGmeZBiISAEoQkLWJa7lskDJQgJGkapBYJh5QmCDMbZGZrzWy9mU2Isz3TzGYG2xeYWW5Qfq6ZLTGz5cG/30plnJIcMy21IRIGKUsQZpYBPAAMBroBV5lZtyrVrgM+cfcuwL3AXUH5NuAid+8BjASeTFWckrxIRC0IkTBIZQuiD7De3Te6eykwAxhapc5Q4Ing8XPAADMzd/+Xu28OylcCzc0sM4WxShIy1MUkEgqpTBDHAZsqPS8OyuLWcfe9wGdAdpU6lwH/cvfdKYpTkmRmlJWnOwoRSbUmKXxti1NW9c/OWuuY2alEu53Oi3sAs7HAWIBOnTodXJSStIiBqwUh0uilsgVRDHSs9DwH2FxTHTNrArQGtgfPc4C/ANe6+4Z4B3D3ae5e4O4F7dq1q+PwpSa6ikkkHFKZIBYBXc2ss5k1A4YDs6rUmUV0EBpgGPCyu7uZtQH+Dtzm7vNTGKMcBC21IRIOKUsQwZjCTcAcYDXwrLuvNLMpZjYkqPYIkG1m64EfAxWXwt4EdAF+ZmaFwU/7VMUqydFVTCLhkMoxCNx9NjC7StnESo93AZfH2Vx+vpMAAAo2SURBVG8qMDWVscnBi5jpjnIiIaCZ1JI0dTGJhIMShCRNg9Qi4aAEIUnTUhsi4aAEIUnL0CC1SCgoQUjSImZa7lskBJQgJGnRO8qlOwoRSTUlCEmaltoQCQclCEmarmISCQclCEladCZ1uqMQkVRTgpCkRSfKKUOINHZKEJI0dTGJhIMShCQtYlCuGwaJNHpKEJI0UwtCJBRSupqrNE4ZZnzw6Vc88WZRukORQ9ClfRb9u7RNdxjSgClBSNKObdOCtzaWMGnWynSHIoegZbMMVv58IGbx7vwrogQhB+HuYXncfsEp6Q5DDsH0t4r433+u48vSMrIy9TUg8emTIUmLRIyjj2iW7jDkEOQc1RKAkh27lSCkRhqkFgmh7Kxogt+2ozTNkUhDpgQhEkJtj8gEoi0IkZooQYiEUEULouRLtSCkZkoQIiFUMYa0XQlCaqEEIRJCzZtmcGRmE7api0lqoQQhElLZWc0o0SC11EIJQiSksrMyKflSLQipmRKESEhlH6EWhNROCUIkpLKzMjUPQmqlBCESUm2zmrH9y92U6/aAUgMlCJGQyj6iGeUOn361J92hSAOlBCESUtlZmk0ttVOCEAkprcckB6IEIRJSbStaELrUVWqQ0nV+zWwQ8GsgA/i9u99ZZXsmMB04HSgBrnT3omDbbcB1QBnwQ3efk8pYRcImO1hu47H5Rbz2761pjqZuZEQiXNvveE7p0CrdoTQKKUsQZpYBPACcCxQDi8xslruvqlTtOuATd+9iZsOBu4ArzawbMBw4FTgW+KeZneTuZamKVyRsjmrZjDNOOJr3Snay+dOv0h1Onfjsqz28tOpD/vK9/nQ8umW6wznspbIF0QdY7+4bAcxsBjAUqJwghgKTg8fPAfdb9P6HQ4EZ7r4beNfM1gev91YK4xUJlUjEmDG2X7rDqFPrP/6CS377JkPufyPWhRYG55zcjtsv6Fbnr5vKBHEcsKnS82Kgb0113H2vmX0GZAflb1fZ97iqBzCzscBYgE6dOtVZ4CJyeOrS/kgeH92Hx+a/S7mHZ37HMa2ap+R1U5kg4t0JvepvrKY6ieyLu08DpgEUFBSE59MgIjU6/fijOP34o9IdRqOQyquYioGOlZ7nAJtrqmNmTYDWwPYE9xURkRRKZYJYBHQ1s85m1ozooPOsKnVmASODx8OAl93dg/LhZpZpZp2BrsDCFMYqIiJVpKyLKRhTuAmYQ/Qy10fdfaWZTQEWu/ss4BHgyWAQejvRJEJQ71miA9p7ge/rCiYRkfpl3kgGcgoKCnzx4sXpDkNE5LBiZkvcvSDeNs2kFhGRuJQgREQkLiUIERGJSwlCRETiajSD1Ga2FXjvEF6iLbCtjsKpS4orOYoreQ01NsWVnION63h3bxdvQ6NJEIfKzBbXNJKfToorOYoreQ01NsWVnFTEpS4mERGJSwlCRETiUoLYZ1q6A6iB4kqO4kpeQ41NcSWnzuPSGISIiMSlFoSIiMSlBCEiInGFPkGY2SAzW2tm681sQhrj6Ghm/2dmq81spZndHJRPNrMPzKww+Dk/TfEVmdnyIIbFQdnRZvaSma0L/q3Xu7SY2cmVzkuhmX1uZj9Kxzkzs0fN7GMzW1GpLO75sajfBJ+5ZWbWq57jutvM1gTH/ouZtQnKc83sq0rn7aFUxVVLbDX+7szstuCcrTWzgfUc18xKMRWZWWFQXm/nrJbviNR9ztw9tD9ElyHfAJwANAOWAt3SFEsHoFfw+Ejg30A3ovfsvrUBnKsioG2Vsl8BE4LHE4C70vy7/BA4Ph3nDDgL6AWsOND5Ac4H/kH0zolnAAvqOa7zgCbB47sqxZVbuV6azlnc313wf2EpkAl0Dv7fZtRXXFW2/zcwsb7PWS3fESn7nIW9BdEHWO/uG929FJgBDE1HIO6+xd3fCR5/Aawmzn24G5ihwBPB4yeAi9MYywBgg7sfymz6g+burxG9p0llNZ2focB0j3obaGNmHeorLnef6+57g6dvE71jY72r4ZzVZCgww913u/u7wHqi/3/rNS4zM+AK4JlUHLs2tXxHpOxzFvYEcRywqdLzYhrAl7KZ5QKnAQuCopuCJuKj9d2NU4kDc81siZmNDcqOcfctEP3wAu3TFBtEbzZV+T9tQzhnNZ2fhvS5G0P0r8wKnc3sX2b2qpmdmaaY4v3uGso5OxP4yN3XVSqr93NW5TsiZZ+zsCcIi1OW1ut+zSwL+BPwI3f/HHgQOBHIB7YQbd6mQ3937wUMBr5vZmelKY5qLHpL2yHAH4OihnLOatIgPndmdjvROzY+FRRtATq5+2nAj4GnzaxVPYdV0++uQZwz4Cr2/0Ok3s9ZnO+IGqvGKUvqnIU9QRQDHSs9zwE2pykWzKwp0V/8U+7+ZwB3/8jdy9y9HHiYFDWrD8TdNwf/fgz8JYjjo4oma/Dvx+mIjWjSesfdPwpibBDnjJrPT9o/d2Y2ErgQGOFBh3XQfVMSPF5CtJ//pPqMq5bfXUM4Z02AS4GZFWX1fc7ifUeQws9Z2BPEIqCrmXUO/godDsxKRyBB3+YjwGp3/59K5ZX7DC8BVlTdtx5iO8LMjqx4THSQcwXRczUyqDYSeKG+Ywvs91ddQzhngZrOzyzg2uAqkzOAzyq6COqDmQ0CfgoMcfedlcrbmVlG8PgEoCuwsb7iCo5b0+9uFjDczDLNrHMQ28L6jA34NrDG3YsrCurznNX0HUEqP2f1MfrekH+IjvT/m2jmvz2NcXyDaPNvGVAY/JwPPAksD8pnAR3SENsJRK8gWQqsrDhPQDYwD1gX/Ht0GmJrCZQArSuV1fs5I5qgtgB7iP7ldl1N54do0/+B4DO3HCio57jWE+2brvicPRTUvSz4/S4F3gEuSsM5q/F3B9wenLO1wOD6jCsofxz4bpW69XbOavmOSNnnTEttiIhIXGHvYhIRkRooQYiISFxKECIiEpcShIiIxKUEISIicSlBiCTBzMps/xVk62wF4GBl0HTN2RCppkm6AxA5zHzl7vnpDkKkPqgFIVIHgnsE3GVmC4OfLkH58WY2L1h8bp6ZdQrKj7HovRiWBj//EbxUhpk9HKz3P9fMWqTtTUnoKUGIJKdFlS6mKytt+9zd+wD3A/8blN1PdMnlPKKL4v0mKP8N8Kq79yR674GVQXlX4AF3PxX4lOhMXZG00ExqkSSY2Q53z4pTXgR8y903Bguqfeju2Wa2jehyEXuC8i3u3tbMtgI57r670mvkAi+5e9fg+U+Bpu4+NfXvTKQ6tSBE6o7X8LimOvHsrvS4DI0TShopQYjUnSsr/ftW8PhNoqsEA4wA3ggezwNuBDCzjDTcd0HkgPTXiUhyWlhww/rAi+5ecalrppktIPqH11VB2Q+BR81sPLAVGB2U3wxMM7PriLYUbiS6gqhIg6ExCJE6EIxBFLj7tnTHIlJX1MUkIiJxqQUhIiJxqQUhIiJxKUGIiEhcShAiIhKXEoSIiMSlBCEiInH9f1hTRGcZ3kx8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "torch.manual_seed(1)\n",
    "\n",
    "LR = 0.1\n",
    "iteration = 10\n",
    "max_epoch = 200\n",
    "# ------------------------------ fake data and optimizer  ------------------------------\n",
    "\n",
    "weights = torch.randn((1), requires_grad=True)\n",
    "target = torch.zeros((1))\n",
    "\n",
    "optimizer = optim.SGD([weights], lr=LR, momentum=0.9)\n",
    "\n",
    "# ------------------------------ 2 Multi Step LR ------------------------------\n",
    "milestones = [50, 125, 160]\n",
    "scheduler_lr = optim.lr_scheduler.MultiStepLR(optimizer, milestones=milestones, gamma=0.1)\n",
    "\n",
    "lr_list, epoch_list = list(), list()\n",
    "for epoch in range(max_epoch):\n",
    "\n",
    "    lr_list.append(scheduler_lr.get_lr())\n",
    "    epoch_list.append(epoch)\n",
    "\n",
    "    for i in range(iteration):\n",
    "\n",
    "        loss = torch.pow((weights - target), 2)\n",
    "        loss.backward()\n",
    "\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "    scheduler_lr.step()\n",
    "\n",
    "plt.plot(epoch_list, lr_list, label=\"Multi Step LR Scheduler\\nmilestones:{}\".format(milestones))\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q:如何按指数衰减调整学习率？\n",
    "- `torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma, last_epoch=-1)`\n",
    "- gamma：指数的底，通常设为一个接近1的数字，如0.95\n",
    "- 调整方式：lr = lr * gamma ** epoch\n",
    "\n",
    "Q:ExponentialLR代码示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXwUdZr48c/TnQvIAYFEbhLuKyRAQBwRQVQOD0QZBXUEdZbRlR13nZnV2fEc/bmjOzuooy8dvHV0xFtmxRvwGhQCciNySzjDkRASEnI8vz+6kmlCJ3Qgneqkn/fr1a+u+ta3qp+udPrpqu+3viWqijHGGFOTx+0AjDHGhCdLEMYYYwKyBGGMMSYgSxDGGGMCsgRhjDEmoCi3A2go7dq107S0NLfDMMaYJmXZsmX7VTUl0LJmkyDS0tLIyclxOwxjjGlSRGR7bcvsFJMxxpiALEEYY4wJyBKEMcaYgJpNG4Qx4aisrIzc3FxKSkrcDsVEuLi4ODp37kx0dHTQ61iCMCaEcnNzSUhIIC0tDRFxOxwToVSVAwcOkJubS3p6etDrhfQUk4iMF5ENIrJJRO4IsHyUiCwXkXIRmVJj2XQR2eg8pocyTmNCpaSkhLZt21pyMK4SEdq2bVvvI9mQJQgR8QJPABOA/sA0Eelfo9qPwAzg1RrrJgP3AGcCw4F7RKRNqGI1JpQsOZhwcCqfw1AeQQwHNqnqFlU9BrwGTPKvoKrbVHUVUFlj3XHAJ6p6UFUPAZ8A40MR5M78o/zvxxv48UBxKDZvjDFNVigTRCdgh998rlPWYOuKyEwRyRGRnLy8vFMK8vDRMv68YBMrcvNPaX1jwp3X6yUrK6v68Yc//MHtkGr1yCOPUFz8zx9rEydOJD+/7v/NtLQ09u/fH1T5Cy+8QEpKCllZWfTt25fZs2cH3ObevXu5+OKLyczMpH///kycOLHOGEaPHn1aF+rOmDGDN998s846L7zwArNmzTrl1zgVoWykDnQ8E+zdiYJaV1XnAHMAsrOzT+nOR+ntWiECm/YdOZXVjQl7LVq0YMWKFW6HEZRHHnmEa6+9lpYtWwIwf/78Bn+Nq666iscff5wDBw7Qp08fpkyZQpcuXY6rc/fdd3PBBRdw6623ArBq1aoGjyPUysvLiYo6va/4UB5B5AL+e70zsKsR1q2XuGgvXdq0ZHOeJQgTOQoKCujTpw8bNmwAYNq0aTz99NMAxMfH86tf/YohQ4YwduxYqo7OV6xYwYgRIxg0aBCTJ0/m0KFDgO/X8+23387w4cPp3bs3X375JQAVFRX85je/YdiwYQwaNIi//OUvACxatIjRo0czZcoU+vbtyzXXXIOq8thjj7Fr1y7GjBnDmDFjgOOPAi677DKGDh3KgAEDmDNnzmnvg7Zt29KzZ0927959wrLdu3fTuXPn6vlBgwZVTz/88MNkZGSQmZnJHXf8s+/NG2+8EfQ+UFVmzZpF//79ueiii9i3b1/1dvzfc05ODqNHjz4hvry8PK644gqGDRvGsGHD+PrrrwG49957mTlzJhdeeCHXXXfdaewdh6qG5IHv6GQLkA7EACuBAbXUfQGY4jefDGwF2jiPrUByXa83dOhQPVXXP79Ex83+/JTXN6Y269atczsE9Xg8mpmZWf147bXXVFX1448/1hEjRujf/vY3HTduXHV9QP/617+qqup9992nt9xyi6qqZmRk6KJFi1RV9a677tJbb71VVVXPPfdcve2221RV9f3339exY8eqqupf/vIXvf/++1VVtaSkRIcOHapbtmzRhQsXamJiou7YsUMrKip0xIgR+uWXX6qqardu3TQvL686Fv/5AwcOqKpqcXGxDhgwQPfv3x9wnUDrVnn++eer38/27ds1MzNTjx49esK6H374oSYlJeno0aP1gQce0J07d6qq6vz58/Wss87SoqKi42Kq7z5466239Pzzz9fy8nLduXOnJiUl6RtvvHFC3EuXLtVzzz33hNinTZtWvc+2b9+uffv2VVXVe+65R4cMGaLFxcUnvCfVwJ9HIEdr+V4N2SkmVS0XkVnAR4AXeE5V14rI752A5onIMOAdJwlcIiL3qeoAVT0oIvcDS53N/V5VD4Yq1p6p8Xy1aT8VlYrXYz1OTPNS2ymmCy64gDfeeINbbrmFlStXVpd7PB6uuuoqAK699louv/xyCgoKyM/P59xzzwVg+vTp/PSnP61e5/LLLwdg6NChbNu2DYCPP/6YVatWVZ9bLygoYOPGjcTExDB8+PDqX+hZWVls27aNkSNH1vk+HnvsMd555x0AduzYwcaNG2nbtm2998fcuXNZuHAhGzZs4OmnnyYuLu6EOuPGjWPLli18+OGHfPDBBwwePJg1a9bw6aefcv3111efAktOTj6lffDFF18wbdo0vF4vHTt25LzzzqvXe/j0009Zt25d9fzhw4cpLCwE4NJLL6VFixb12l5tQnqhnKrOB+bXKLvbb3opvtNHgdZ9DngulPFV6ZHSimPlleQeKqZb21aN8ZLGuK6yspL169fTokULDh48eNwpFX/BdI+MjY0FfA3i5eXlgO/sxJ///GfGjRt3XN1FixZV16+5Tm0WLVrEp59+yuLFi2nZsiWjR48+5avTq9ogFi9ezEUXXcSECRNo3779CfWSk5O5+uqrufrqq7n44ov54osvUNVa90d99sH8+fNr3U5UVBSVlb6OnbW9x8rKShYvXhwwEbRq1XDfYTYWE74jCLCGahNZZs+eTb9+/fjb3/7GDTfcQFlZGeD78qn6xfvqq68ycuRIkpKSaNOmTfW59Zdffrn6aKI248aN48knn6ze7g8//EBRUVGd6yQkJFT/EvZXUFBAmzZtaNmyJd9//z3ffPNNvd9vTWeddRY/+9nPePTRR09YtmDBgureVIWFhWzevJmuXbty4YUX8txzz1UvO3iw7hMbte2DUaNG8dprr1FRUcHu3btZuHBh9TppaWksW7YMgLfeeivgdi+88EIef/zx6vlQdUKwoTaAHim+BLE57whj+53hcjTGNKyjR4+SlZVVPT9+/HhuuOEGnnnmGZYsWUJCQgKjRo3igQce4L777qNVq1asXbuWoUOHkpSUxNy5cwF48cUXuemmmyguLqZ79+48//zzdb7uz3/+c7Zt28aQIUNQVVJSUnj33XfrXGfmzJlMmDCBDh06HPelOX78eJ566ikGDRpEnz59GDFiRFDvfdCgQXg8vt/BV1555XGNzQC33347Q4YM4b/+679ISEioLl+2bBmzZs2q/jX/85//nGHDhgG+L+Ps7GxiYmKYOHEiDz74YL33weTJk1mwYAEZGRn07t37uGR7zz33cOONN/Lggw9y5plnBtzuY489xi233MKgQYMoLy9n1KhRPPXUU0Htk/oQXxtF05edna2n0w85+4FPOK9vKg9PyWzAqEykW79+Pf369XM7jHqJj4/nyBE7mm6OAn0eRWSZqmYHqm+nmBw9UuLtFJMxxvixU0yOnqnx/N+q3XU2QhlzOu77+1rW7TrcoNvs3zGRey4Z0KDbtKMHU8WOIBw9UuIpOFrG/iPH3A7FGGPCgh1BOPx7MqUkxJ6ktjH119C/9JuTZcuWMWPGDI4ePcrEiRN59NFHTziSP3ToEDfccAObN28mLi6O5557joEDBwK+nj8JCQl4vV6ioqJOa1wk8092BOHokfrPnkzGmMZ18803M2fOHDZu3MjGjRv58MMPT6jz4IMPkpWVxapVq3jppZeqx0mqsnDhQlasWGHJoQFZgnB0TIqjZYzXGqpNs3T//ffTt29fLrjgAqZNm8Yf//hHnn76aYYNG0ZmZiZXXHFFdd/+GTNmcPPNNzNmzBi6d+/O559/zg033EC/fv2YMWNG9Tbj4+O5/fbbGTp0KOeffz5Llixh9OjRdO/enXnz5gGwbds2zjnnHIYMGcKQIUP4xz/+cUJsu3fv5vDhw5x11lmICNddd13A7rDr1q1j7NixAPTt25dt27axd+/eEOwtU8UShENE6JESb0cQptnJycnhrbfe4rvvvuPtt9+u/oV9+eWXs3TpUlauXEm/fv149tlnq9c5dOgQCxYsYPbs2VxyySX8x3/8B2vXrmX16tXVF2UVFRUxevRoli1bRkJCAnfeeSeffPIJ77zzDnff7RswITU1lU8++YTly5czd+5cfvnLX1a/RtW1GTt37jzuKu7OnTuzc+fOE95HZmYmb7/9NgBLlixh+/bt5ObmAr7/3wsvvJChQ4c2yEB+xsfaIPz0SGnFkq0hG/LJGFd89dVXTJo0qXpYhksuuQSANWvWcOedd5Kfn8+RI0eOGw7ikksuQUTIyMjgjDPOICMjA4ABAwawbds2srKyiImJYfx43328MjIyiI2NJTo6moyMjOqxiMrKypg1axYrVqzA6/Xyww8/VL9GVaIJdC1WoJ6Ed9xxB7feeitZWVlkZGQwePDg6uGsv/76azp27Mi+ffu44IIL6Nu3L6NGjTrdXRfxLEH46Zkaz7srdlFUWk6rWNs1pnmo7WLYGTNm8O6775KZmckLL7zAokWLqpdVjSvk8XiOGzfJ4/FUjzMUHR1d/UXuX8+/zuzZsznjjDNYuXIllZWVAQfG69y5c/WRAEBubi4dO3Y8oV5iYmL11duqSnp6Ounp6QDV9VNTU5k8eTJLliyxBNEA7BSTn6qeTFvy6h4vxpimZOTIkfz973+npKSEI0eO8P777wO+MYY6dOhAWVkZr7zySkheu6CggA4dOuDxeHj55ZepqKg4oU6HDh1ISEjgm2++QVV56aWXmDRp0gn18vPzOXbM1w39mWeeYdSoUSQmJlJUVFQ9flNRUREff/xxde8mc3rsZ7KfqgSxcV8hGZ2TXI7GmIYxbNgwLr30UjIzM+nWrRvZ2dkkJSVx//33c+aZZ9KtWzcyMjICDpJ3uv71X/+VK664gjfeeIMxY8YcN9JoVlZW9WmmJ598srqb64QJE5gwYQJA9fhCN910E+vXr+e6667D6/XSv3//6jaTvXv3MnnyZMB3F7Wrr766+tSXOT02FpOfsopKBtz9EdefncZvJzat8XNMeAqXsZiOHDlCfHw8xcXFjBo1ijlz5jBkyBC3wzKNrL5jMdkRhJ9or4ceqfF8v6fhf0kZ46aZM2eybt06SkpKmD59uiUHExRLEDX0bZ/A4s0H3A7DmAb16quvuh2CaYKskbqGPu0T2HO4hILiMrdDMc1EczmNa5q2U/kcWoKooU97301Dvt/TsKNumsgUFxfHgQMHLEkYV6kqBw4cCNjNuC52iqmGvk6C2LC3kDO71/+G6Mb4q+rjn5eX53YoJsLFxcXVet/x2liCqKF9YhyJcVHWUG0aRHR0dPXFXMY0NXaKqQYRoW/7RDZYgjDGRDhLEAH0aZ/AD3sK7byxMSaiWYIIoE/7BApLy9mZf9TtUIwxxjWWIAKobqi200zGmAhmCSKAXmdUdXW1BGGMiVyWIAJIahFNx6Q4O4IwxkQ0SxC16NM+wRKEMSaiWYKoRZ/2iWzOO8Kx8kq3QzHGGFdYgqhF3/YJlFcqW/bbPaqNMZHJEkQt+lhPJmNMhAtpghCR8SKyQUQ2icgdAZbHishcZ/m3IpLmlEeLyIsislpE1ovIb0MZZyA9UuKJ8XpYt8sG7TPGRKaQJQgR8QJPABOA/sA0Eelfo9qNwCFV7QnMBh5yyn8KxKpqBjAU+EVV8mgsMVEe+rRPYM2ugsZ8WWOMCRuhPIIYDmxS1S2qegx4Dah5J/JJwIvO9JvAWBERQIFWIhIFtACOAY3+U35gp0TW7jpsQ24YYyJSKBNEJ2CH33yuUxawjqqWAwVAW3zJogjYDfwI/FFVD9Z8ARGZKSI5IpITiuGUB3RMIr+4zIbcMMZEpFAmCAlQVvOneG11hgMVQEcgHfiViHQ/oaLqHFXNVtXslJSU0433BAM7JQGwZqe1QxhjIk8oE0Qu0MVvvjOwq7Y6zumkJOAgcDXwoaqWqeo+4GsgO4SxBtS3fQJej7DW2iGMMREolAliKdBLRNJFJAaYCsyrUWceMN2ZngIsUN8J/x+B88SnFTAC+D6EsQYUF+2lZ0o8a3ZagjDGRJ6QJQinTWEW8BGwHnhdVdeKyO9F5FKn2rNAWxHZBNwGVHWFfQKIB9bgSzTPq+qqUMValwGdElljXV2NMREopLccVdX5wPwaZXf7TZfg69Jac70jgcrdMLBjEm8v38m+wyWkJtbvht/GGNOU2ZXUJ1HVUL3WjiKMMRHGEsRJ9O+YCGDtEMaYiGMJ4iTiY6Po3q6VXVFtjIk4liCCMKBTkl0LYYyJOJYggjCwYyI7849yqOiY26EYY0yjsQQRhAEdraHaGBN5LEEEYYDTUL3aGqqNMRHEEkQQ2rSKoVvblqzcke92KMYY02gsQQQpq0trvttxyO0wjDGm0ViCCFJWl9bsPVzK7gIb+tsYExksQQRpcNc2AKz40U4zGWMigyWIIPXrkECM18N31g5hjIkQliCCFBvlpX/HRDuCMMZEDEsQ9TC4a2tW7cynvKLS7VCMMSbkLEHUQ1aX1pSUVfL9nkK3QzHGmJCzBFEPg7s4DdXWDmGMiQCWIOqhS3ILklvFWIIwxkQESxD1ICIM7tKa7360C+aMMc2fJYh6yurSms15RRQcLXM7FGOMCSlLEPWU1bU1AKty7TSTMaZ5swRRT5ldWiMCy7dbgjDGNG+WIOopMS6a3qkJ5Gw/6HYoxhgTUidNECLSUkTuEpGnnfleInJx6EMLX8PS27B8+yG7YM4Y06wFcwTxPFAKnOXM5wIPhCyiJmB4eluKjlWwbrfdYc4Y03wFkyB6qOrDQBmAqh4FJKRRhbnhackALNlqp5mMMc1XMAnimIi0ABRARHrgO6KIWO2T4uiS3IKl2yxBGGOar6gg6twLfAh0EZFXgLOB60MZVFMwLC2ZRRvyUFVEIvqAyhjTTJ30CEJVPwYuB2YAfwOyVXVhiOMKe8PTkjlYdIzNeUfcDsUYY0IimF5Mn6nqAVV9X1X/T1X3i8hnjRFcOBueXtUOYcNuGGOap1oThIjEiUgy0E5E2ohIsvNIAzo2VoDhKr1dK9rFx1g7hDGm2arrCOIXwDKgr/Nc9XgPeCKYjYvIeBHZICKbROSOAMtjRWSus/xbJ/lULRskIotFZK2IrBaRuODfVuiJCMPSkq0nkzGm2ao1Qajqo6qaDvxaVburarrzyFTVx0+2YRHx4kskE4D+wDQR6V+j2o3AIVXtCcwGHnLWjQL+CtykqgOA0TjdbMPJsLRkduYfZWf+UbdDMcaYBnfSXkyq+mcRGYjvSz7Or/ylk6w6HNikqlsAROQ1YBKwzq/OJHy9pADeBB4XX5egC4FVqrrSea0DQb2bRlbVDrF060E6De7kcjTGGNOwgmmkvgf4s/MYAzwMXBrEtjsBO/zmc52ygHVUtRwoANoCvQEVkY9EZLmI/Gctsc0UkRwRycnLywsipIbVr0MiCbFRfLs1LPOXMcaclmAulJsCjAX2qOr1QCYQG8R6gS4O0CDrRAEjgWuc58kiMvaEiqpzVDVbVbNTUlKCCKlheT3Cmd2T+cdmSxDGmOYnmARxVFUrgXIRSQT2Ad2DWC8X6OI33xnYVVsdp90hCTjolH+uqvtVtRiYDwwJ4jUb3dk927H9QDE7Dha7HYoxxjSoYBJEjoi0Bp7G14tpObAkiPWWAr1EJF1EYoCpwLwadeYB053pKcACVVXgI2CQM5JsFHAux7ddhI2RPdsB8PWm/S5HYowxDavORmqnwfi/VTUfeEpEPgQSVXXVyTasquUiMgvfl70XeE5V14rI74EcVZ0HPAu8LCKb8B05THXWPSQif8KXZBSYr6rvn/rbDJ2eqfGkJsTy1ab9TB3e1e1wjDGmwdSZIFRVReRdYKgzv60+G1fV+fhOD/mX3e03XQL8tJZ1/4qvq2tYExFG9mzHoh/yqKxUPB4bl8kY0zwEc4rpGxEZFvJImrCze7bjYNEx1u+x+0MYY5qPYBLEGGCxiGwWkVXOVc0nPcUUSc62dghjTDMUzHDfE0IeRRPXPimOnqnxfLXpADNH9XA7HGOMaRDBXEm9vTECaepG9mzH3KU7KC2vIDbK63Y4xhhz2oI5xWSCcHbPdhwtq+C7H/PdDsUYYxqEJYgGcmb3ZLwe4auN1g5hjGkeLEE0kMS4aLK6tObzHxp/TChjjAmFYAbrKxSRwzUeO0TkHREJZsiNiHFe31RW7yxg3+ESt0MxxpjTFswRxJ+A3+AbebUz8Gt8w268BjwXutCanjF9UgFYtMGOIowxTV8wCWK8qv5FVQtV9bCqzgEmqupcoE2I42tS+nVIoENSHAu+3+d2KMYYc9qCSRCVInKliHicx5V+y2oO3x3RRITRfVL5cmMex8or3Q7HGGNOSzAJ4hrgZ/iG+d7rTF8rIi2AWSGMrUk6r28qRccq7F7VxpgmL5gL5bYAl9Sy+KuGDafpO7tnW2KiPCz4fh8je7VzOxxjjDllJ00QIpIC/AuQ5l9fVW8IXVhNV8uYKEZ0b8vCDfu4+5L+bodjjDGnLJhTTO/hu9Pbp8D7fg9Ti7F9U9m6v4it+4vcDsUYY05ZMAmiparerqqvq+pbVY+QR9aEndfX193VejMZY5qyYBLE/4nIxJBH0ox0SW5Jz9R4Plu/1+1QjDHmlAWTIG7FlySOOldRF4qI3RnnJMYNOINvtx7kYNExt0MxxphTctIEoaoJqupR1RaqmujMJzZGcE3ZhIEdqKhUPlm3x+1QjDHmlNSaIESkr/M8JNCj8UJsmgZ0TKRLcgvmr7YEYYxpmurq5nobMBP43wDLFDgvJBE1EyLChIEdeP7rrRQcLSOpRbTbIRljTL3UegShqjOd5zEBHpYcgjBhYHvKKtQaq40xTVIw96RGRH7CiRfKvRSimJqNzM6t6ZAUxwdr9nD5kM5uh2OMMfUSzJXULwM9gBVAhVOsgCWIk/B4hHED2vPqkh85UlpOfGxQ+dgYY8JCMN9Y2UB/VbWRW0/BxIwOvPCPbSz8fh+XZHZ0OxxjjAlaMNdBrAHahzqQ5mpotza0i4/lgzW73Q7FGGPqJZgjiHbAOhFZApRWFarqpSGLqhnxeoSLMtrz2tIdHC4pIzHOejMZY5qGYBLEvaEOorm7bHAnXly8nQ/X7OHK7C5uh2OMMUGpM0GIiBe4S1XPb6R4mqWsLq1Ja9uSd7/baQnCGNNk1NkGoaoVQLGIJDVSPM2SiHDZ4E4s3nKA3QVH3Q7HGGOCEkwjdQmwWkSeFZHHqh6hDqy5mTy4E6rw3opdbodijDFBCSZBvA/cBXwBLPN7nJSIjBeRDSKySUTuCLA8VkTmOsu/FZG0Gsu7isgREfl1MK8Xzrq1bcWQrq1597udbodijDFBCeae1C+eyoad9osngAuAXGCpiMxT1XV+1W4EDqlqTxGZCjwEXOW3fDbwwam8fjiaPLgTd723lnW7DtO/ow2Ia4wJbyc9ghCRXiLypoisE5EtVY8gtj0c2KSqW1T1GPAaMKlGnUlAVQJ6ExgrIuK87mXAFmBtsG8m3F08qCNRHuHdFXYUYYwJf8GcYnoeeBIoB8bgG2Lj5SDW6wTs8JvPdcoC1lHVcqAAaCsirYDbgfvqegERmSkiOSKSk5eXF0RI7mrTKobRfVJ557udlFVUuh2OMcbUKZgE0UJVPwNEVber6r0EN9S3BCirOVxHbXXuA2ar6pG6XkBV56hqtqpmp6SkBBGS+6YO60JeYSmfrbf7VRtjwlswF8qViIgH2Cgis4CdQGoQ6+UC/p3+OwM1u/BU1ckVkSggCTgInAlMEZGHgdZApYiUqOrjQbxuWBvdJ4UOSXG8uuRHxg+0EUyMMeErmCOIfwdaAr8EhgLXAtODWG8p0EtE0kUkBpgKzKtRZ57ftqYAC9TnHFVNU9U04BHgweaQHACivB6uzO7Clxvz2HGw2O1wjDGmVsHck3qpc6rnkKper6pXqOo3QaxXDswCPgLWA6+r6loR+b2IVI3j9Cy+NodN+O5gd0JX2OboqmFdEGDu0h0nrWuMMW6Rk43iLSJn4fsij1fVriKSCfxCVf+1MQIMVnZ2tubk5LgdRtBufGEpq3YW8I87ziPaG8yBnDHGNDwRWaaq2YGWBfPN9AgwDjgAoKorgVENF15kmja8qzVWG2PCWlA/XVW15rmQioAVTdD8G6uNMSYcBZMgdjj3pFYRiXGGvVgf4riavSivh6nDuvLFD3ls2ldnb15jjHFFMAniJuAWfBe15QJZQFi1PzRV14zoSkyUh+e/3up2KMYYc4JgejHtV9VrVPUMVU1V1WuB6xohtmavXXwsk7M68dbyXA4VHXM7HGOMOc6pdp+5rUGjiGA3jEynpKzS2iKMMWHnVBNEoCEyzCno0z6Bc3q148V/bONYuY3PZIwJH6eaIOq+eMLUy40j09lXWMr7q+1mQsaY8FFrghCRQhE5HOBRCHRsxBibvXN7p9AzNZ5nvtzKyS5cNMaYxlJrglDVBFVNDPBIUNVgBvkzQRIRZp7TnbW7DrPoh/AfttwYExlsjIcwcdngTnRq3YJHP91oRxHGmLBgCSJMxER5uHl0D1bsyOerTfvdDscYYyxBhJOfZnemfWIcj31mRxHGGPdZgggjsVFebjq3O0u3HeKbLQfdDscYE+EsQYSZqcO7kpIQy2OfbXQ7FGNMhLMEEWbior3cdG4PFm85wFcbrS3CGOMeSxBh6NoRXenUugX//cF6KiutLcIY4w5LEGEoNsrLr8f1Zu2uw/x9lV1dbYxxhyWIMDUpsxP9OyTyPx9toLTc7s9kjGl8liDClMcj/HZiX3IPHeXlxdvdDscYE4EsQYSxc3qlcE6vdjy+cBP5xXa/CGNM47IEEeb+a2I/CkvK+ePHG9wOxRgTYSxBhLl+HRK57qxuvPLtj6zOLXA7HGNMBLEE0QT8xwW9adsqljvfW2PdXo0xjcYSRBOQGBfN7y7qy8od+byes8PtcIwxEcISRBNxWVYnhqcl89CH33PgSKnb4RhjIoAliCZCRHhg8kCKSiu4e95at8MxxkQASxBNSO8zErj1/F68v2o381fvdjscY5uRwBwAABDuSURBVEwzZwmiifnFqO4M6pzEXe+usVNNxpiQsgTRxER5PfzPlEwKS8q5+z071WSMCZ2QJggRGS8iG0Rkk4jcEWB5rIjMdZZ/KyJpTvkFIrJMRFY7z+eFMs6mpk9751TT6t28vTzX7XCMMc1UyBKEiHiBJ4AJQH9gmoj0r1HtRuCQqvYEZgMPOeX7gUtUNQOYDrwcqjibql+M6s7w9GTufHcNm/OOuB2OMaYZCuURxHBgk6puUdVjwGvApBp1JgEvOtNvAmNFRFT1O1WtGud6LRAnIrEhjLXJifJ6eGzqYGKjPNzyynJKymzEV2NMwwplgugE+F/VleuUBayjquVAAdC2Rp0rgO9U9YQWWRGZKSI5IpKTl5fXYIE3Fe2T4vjTlVl8v6eQ//f+erfDMcY0M6FMEBKgrOY4EXXWEZEB+E47/SLQC6jqHFXNVtXslJSUUw60KRvTN5V/OSedl7/Zzrvf7XQ7HGNMMxLKBJELdPGb7wzUvD1adR0RiQKSgIPOfGfgHeA6Vd0cwjibvP8c35fh6cnc/tYqVuXmux2OMaaZCGWCWAr0EpF0EYkBpgLzatSZh68RGmAKsEBVVURaA+8Dv1XVr0MYY7MQ7fXw5DVDaBcfy8yXlrHvcInbIRljmoGQJQinTWEW8BGwHnhdVdeKyO9F5FKn2rNAWxHZBNwGVHWFnQX0BO4SkRXOIzVUsTYHbeNjefq6bAqOljHz5WXWaG2MOW2i2jyGj87OztacnBy3w3DdB6t3c/Mry5kwsD2PXz0ErydQM48xxviIyDJVzQ60zK6kbmYmZHTgzov68cGaPdz93hqayw8AY0zji3I7ANPwfn5Od/YfOcZTn28mJSGWfz+/t9shGWOaIEsQzdTt4/uQV1jKI59uJCEumhtHprsdkjGmibEE0UyJCH+4IoPiY+Xc/3/rUFV+fk53t8MyxjQh1gbRjEV7PTw2bTATM9rzwPvreebLLW6HZIxpQuwIopmL9np4dOpghBU88P56io9V8G/n9UTEejcZY+pmCSIC+JJEFrFRHv70yQ/sKyzhvksHWhdYY0ydLEFEiCivhz/+NJOUxFj+8vkW9hce45GpWcRFe90OzRgTpqwNIoJ4PMJvJ/Tj7ov789G6PVw15xv2FNiwHMaYwCxBRKAbRqbz1LVD2bS3kEse/4pl2w+5HZIxJgxZgohQ4wa0551bzqZljJepcxbz8jfb7aprY8xxLEFEsN5nJDDvlpGc3bMdd727hpv/upyC4jK3wzLGhAlLEBEuqWU0z00fxu8m9uPT9XuZ8OgXfLvlgNthGWPCgCUIg8cj/Muo7rx180+IjvJw1ZxvuOe9NRSVlrsdmjHGRZYgTLXMLq354NZzmPGTNF76ZjvjH/2CLzdG3r2+jTE+liDMcVrGRHHvpQN4/RdnEeXx8LNnl3DzX5exM/+o26EZYxqZJQgT0LC0ZD649Rx+dUFvFm7Yx9j/XcTsT36gsMQasY2JFJYgTK3ior3829hefHrbuZzXN5VHP9vIqIcX8vQXW+yWpsZEALvlqAnayh35/PHjDXy5cT9nJMbyy7G9mDK0M7FRNlyHMU1VXbcctQRh6m3x5gP8z0ffs/zHfFITYpn+kzSuObMrrVvGuB2aMaaeLEGYBqeqfLlxP09/uYUvN+6nRbSXK7M7c/3Z6aS1a+V2eMaYIFmCMCG1fvdhnvlyK/NW7qSsQhnRPZmrhnVh/IAOtIix00/GhDNLEKZR7Dtcwus5O3g9J5cfDxaTEBvFpVkdmZTViexubfDY/SeMCTuWIEyjqqxUvt16kNdzdjB/9W5KyytJTYhlwsD2TMjowLC0ZLtZkTFhwhKEcc2R0nI+W7+X+at3s2hDHqXllbSLj2FU7xTO7Z3CqF4ptGlljdvGuMUShAkLRaXlLPh+Hx+v28uXG/PILy5DBDI7t2Zkz3YMS09mSNfWJMRFux2qMRHDEoQJOxWVyqrcfD7/IY/Pf8hjVW4BFZWKR6Bfh0SGpSUzLC2ZId1a0z4xDhE7JWVMKFiCMGGvqLScFTvyWbL1IEu3HeS7H/M56lytndwqhgEdExnQMYkBHRMZ2CmJbsktrdHbmAZQV4KIauxgjAmkVWwUZ/dsx9k92wFQVlHJ2l2HWZWbz5qdBazddZhnv9pCWYXvB02LaC/dU1rRIyWeHinx9EyNp0dqK9LatiIu2rrWGtMQLEGYsBTt9ZDVpTVZXVpXl5WWV7Bx7xHW7irg+z2FbMkrYvmPh5i3ctdx66YmxNK5TQs6t2lZ47kFHZJa2LUZxgTJEoRpMmKjvAzslMTATknHlR89VsGW/UfYnFfE1rwiduYXk3voKCt25DN/9W7KK48/jZoQG0VKQiwpCbGkJsaREh9LamIsKfGxJLeKIallNG1axtCmZTSJcdF2KstErJAmCBEZDzwKeIFnVPUPNZbHAi8BQ4EDwFWqus1Z9lvgRqAC+KWqfhTKWE3T1SLG67RPJJ2wrKJS2Xu4hNxDR9lxsJg9h0vIKywlr7CUfYUlrM7NZ19hKcXHAo9OKwJJLXwJw/ccTUJcNPFxUcTHRtEqJsqZ9hIfG02rWC8JcVG0cpbFRXuJi/YQF+0l2muDJ5umJWQJQkS8wBPABUAusFRE5qnqOr9qNwKHVLWniEwFHgKuEpH+wFRgANAR+FREequqjTFt6sXrETq2bkHH1i0Ynp5ca70jpeXkFZZyqPgY+cXHyC8u41Bxmd+073lfYSlb9hdRVFpOYUk5peWV9YolLsqXLGKrnqsSSNQ/E0lslIcor4dorxDt9RDl8Zt2nqO94iuP8hDtkRr1hSiv4BHB6xG8Ing8VfNUl3vEr45TXr2sej2nvrMN33ogCFUdy8RvXgARcZ6x3mdNXCiPIIYDm1R1C4CIvAZMAvwTxCTgXmf6TeBx8X2iJgGvqWopsFVENjnbWxzCeE0Ei4/1HRGkU7+BBssqKikuraCwtIyi0gqOlJZzpLScIue5tKyCkrJKSsoqKCn3TZeW+5VVz1ew/0g5JWUVlJZXUl5RSVmlUlZRSXmF77msopLKJtrpsCp5eESqEwrVCSVwgsF/vsb6HFff2V6A1zxu/oTlJ09eJ2zjhG02zOueUHKSbdTczujeKdx5cf8AtU5PKBNEJ2CH33wucGZtdVS1XEQKgLZO+Tc11u1U8wVEZCYwE6Br164NFrgxwYr2ekhq6SGpZeNc3FdZqZRVVlJWob4kUuGXRCorq6fLK5WKSqVS//lcWQkVqlT6LfMt/2e5f/2KSvzq+K8DqqAoVb3kVdUpO36Z+hYeV+6/PtXrBF6/etsB1qfqNQIkTd8afvNaczl1Lg+0jZPMVsda/9ep3zYCFXZo3SJQrdMWygQRKOnVfK+11QlmXVR1DjAHfNdB1DdAY5oaj0eI9XiJte4lphGEstUsF+jiN98Z2FVbHRGJApKAg0Gua4wxJoRCmSCWAr1EJF1EYvA1Os+rUWceMN2ZngIsUN/x1TxgqojEikg60AtYEsJYjTHG1BCyA1WnTWEW8BG+bq7PqepaEfk9kKOq84BngZedRuiD+JIITr3X8TVolwO3WA8mY4xpXDYWkzHGRLC6xmKyK3eMMcYEZAnCGGNMQJYgjDHGBGQJwhhjTEDNppFaRPKA7aexiXbA/gYKpyFZXPVjcdVfuMZmcdXPqcbVTVVTAi1oNgnidIlITm0t+W6yuOrH4qq/cI3N4qqfUMRlp5iMMcYEZAnCGGNMQJYg/mmO2wHUwuKqH4ur/sI1Nourfho8LmuDMMYYE5AdQRhjjAnIEoQxxpiAIj5BiMh4EdkgIptE5A4X4+giIgtFZL2IrBWRW53ye0Vkp4iscB4TXYpvm4isdmLIccqSReQTEdnoPLdp5Jj6+O2XFSJyWET+3Y19JiLPicg+EVnjVxZw/4jPY85nbpWIDGnkuP5HRL53XvsdEWntlKeJyFG//fZUqOKqI7Za/3Yi8ltnn20QkXGNHNdcv5i2icgKp7zR9lkd3xGh+5z5bvUXmQ98w5BvBroDMcBKoL9LsXQAhjjTCcAPQH989+z+dRjsq21AuxplDwN3ONN3AA+5/LfcA3RzY58Bo4AhwJqT7R9gIvABvjsnjgC+beS4LgSinOmH/OJK86/n0j4L+Ldz/hdWArFAuvN/622suGos/1/g7sbeZ3V8R4TscxbpRxDDgU2qukVVjwGvAZPcCERVd6vqcme6EFhPgPtwh5lJwIvO9IvAZS7GMhbYrKqnczX9KVPVL/Dd08RfbftnEvCS+nwDtBaRDo0Vl6p+rKrlzuw3+O7Y2Ohq2We1mQS8pqqlqroV2ITv/7dR4xIRAa4E/haK165LHd8RIfucRXqC6ATs8JvPJQy+lEUkDRgMfOsUzXIOEZ9r7NM4fhT4WESWichMp+wMVd0Nvg8vkOpSbOC72ZT/P2047LPa9k84fe5uwPcrs0q6iHwnIp+LyDkuxRTobxcu++wcYK+qbvQra/R9VuM7ImSfs0hPEBKgzNV+vyISD7wF/LuqHgaeBHoAWcBufIe3bjhbVYcAE4BbRGSUS3GcQHy3tL0UeMMpCpd9Vpuw+NyJyO/w3bHxFadoN9BVVQcDtwGvikhiI4dV298uLPYZMI3jf4g0+j4L8B1Ra9UAZfXaZ5GeIHKBLn7znYFdLsWCiETj+8O/oqpvA6jqXlWtUNVK4GlCdFh9Mqq6y3neB7zjxLG36pDVed7nRmz4ktZyVd3rxBgW+4za94/rnzsRmQ5cDFyjzglr5/TNAWd6Gb7z/L0bM646/nbhsM+igMuBuVVljb3PAn1HEMLPWaQniKVALxFJd36FTgXmuRGIc27zWWC9qv7Jr9z/nOFkYE3NdRshtlYiklA1ja+Rcw2+fTXdqTYdeK+xY3Mc96suHPaZo7b9Mw+4zullMgIoqDpF0BhEZDxwO3Cpqhb7laeIiNeZ7g70ArY0VlzO69b2t5sHTBWRWBFJd2Jb0pixAecD36tqblVBY+6z2r4jCOXnrDFa38P5ga+l/wd8mf93LsYxEt/h3ypghfOYCLwMrHbK5wEdXIitO74eJCuBtVX7CWgLfAZsdJ6TXYitJXAASPIra/R9hi9B7QbK8P1yu7G2/YPv0P8J5zO3Gshu5Lg24Ts3XfU5e8qpe4Xz910JLAcucWGf1fq3A37n7LMNwITGjMspfwG4qUbdRttndXxHhOxzZkNtGGOMCSjSTzEZY4yphSUIY4wxAVmCMMYYE5AlCGOMMQFZgjDGGBOQJQhj6kFEKuT4EWQbbARgZ2RQt67ZMOYEUW4HYEwTc1RVs9wOwpjGYEcQxjQA5x4BD4nIEufR0ynvJiKfOYPPfSYiXZ3yM8R3L4aVzuMnzqa8IvK0M97/xyLSwrU3ZSKeJQhj6qdFjVNMV/ktO6yqw4HHgUecssfxDbk8CN+geI855Y8Bn6tqJr57D6x1ynsBT6jqACAf35W6xrjCrqQ2ph5E5Iiqxgco3wacp6pbnAHV9qhqWxHZj2+4iDKnfLeqthORPKCzqpb6bSMN+ERVeznztwPRqvpA6N+ZMSeyIwhjGo7WMl1bnUBK/aYrsHZC4yJLEMY0nKv8nhc70//AN0owwDXAV870Z8DNACLideG+C8aclP06MaZ+Wohzw3rHh6pa1dU1VkS+xffDa5pT9kvgORH5DZAHXO+U3wrMEZEb8R0p3IxvBFFjwoa1QRjTAJw2iGxV3e92LMY0FDvFZIwxJiA7gjDGGBOQHUEYY4wJyBKEMcaYgCxBGGOMCcgShDHGmIAsQRhjjAno/wNsALjSvbD5ZQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "torch.manual_seed(1)\n",
    "\n",
    "LR = 0.1\n",
    "iteration = 10\n",
    "max_epoch = 200\n",
    "# ------------------------------ fake data and optimizer  ------------------------------\n",
    "\n",
    "weights = torch.randn((1), requires_grad=True)\n",
    "target = torch.zeros((1))\n",
    "\n",
    "optimizer = optim.SGD([weights], lr=LR, momentum=0.9)\n",
    "\n",
    "# ------------------------------ 3 Exponential LR ------------------------------\n",
    "gamma = 0.95\n",
    "scheduler_lr = optim.lr_scheduler.ExponentialLR(optimizer, gamma=gamma)\n",
    "\n",
    "lr_list, epoch_list = list(), list()\n",
    "for epoch in range(max_epoch):\n",
    "\n",
    "    lr_list.append(scheduler_lr.get_lr())\n",
    "    epoch_list.append(epoch)\n",
    "\n",
    "    for i in range(iteration):\n",
    "\n",
    "        loss = torch.pow((weights - target), 2)\n",
    "        loss.backward()\n",
    "\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "    scheduler_lr.step()\n",
    "\n",
    "plt.plot(epoch_list, lr_list, label=\"Exponential LR Scheduler\\ngamma:{}\".format(gamma))\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q:如何按余弦周期调整学习率？\n",
    "- `torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max, eta_min=0, last_epoch=-1)`\n",
    "- T_max：下降周期\n",
    "- eta_min：学习率下限\n",
    "- 调整方式：$$\\eta_{t}=\\eta_{\\min }+\\frac{1}{2}\\left(\\eta_{\\max }-\\eta_{\\min }\\right)\\left(1+\\cos \\left(\\frac{T_{c u r}}{T_{\\max }} \\pi\\right)\\right)$$\n",
    "\n",
    "Q:CosineAnnealingLR代码示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXxU9bn48c+TyUoCCVlZEiAJgUCAhD2A7ApoVcC6Ua/lXrdatbX21mrbq1Vv7+3Veq1tb39tba1aq4C7Vi0ogrJIIOxrICEJECAkmex7JvP9/TGTEEICk5CZM8v3/XrNi2TmzDlPhjPzzPk+30WUUmiapmlaZ35GB6Bpmqa5J50gNE3TtC7pBKFpmqZ1SScITdM0rUs6QWiapmld8jc6gL4SHR2tRowYYXQYmqZpHmXXrl1lSqmYrh7zmgQxYsQIdu7caXQYmqZpHkVETnT3mG5i0jRN07qkE4SmaZrWJZ0gNE3TtC7pBKFpmqZ1SScITdM0rUtOTRAiskREjopInog83sXjc0Rkt4hYROTmTo+tFJFc+22lM+PUNE3TLua0BCEiJuD3wLXAWGCFiIzttNlJ4F+BNzs9NxL4OTAdmAb8XEQGOitWTdM07WLOHAcxDchTSuUDiMhqYClwuG0DpVSh/TFrp+cuBj5XSpXbH/8cWAKs6usgy2qbeGVrAfED+zF+aDhpQwYgIn19GM1HHT5Tzf6iSspqm4jpH0R6QgSpgwYYHZbmJZRS7CuqQinFxGF9/x3amQliKHCqw+9F2K4IevvcoZ03EpH7gPsAhg0b1qsgT5jr+eNX+bRabetipA7qz6OLR7NwTFyv9qdpAF8dK+XZf+Zw+Gz1RY+NHxrOY0tSuSol2oDING/x2aFinv/sKMfO1TJ/dAyv/Nu0Pj+GMxNEV1/DHV2dyKHnKqVeAl4CmDJlSq9WPpo8fCBH/3MJZ6sa2ZRbyitbC7n7tZ0syxjC/3xzAsEBpt7sVvNRTZZW/uP9g7y9q4ik6FCeWZrG/NGxxPQPoqS6iY1HS3hlawH/8vJ2VkxL4OkbxxHor/uKaI5raG7l0Xf28fH+s4yKC+OXN43n+gmDnXIsZyaIIiChw+/xwJkePHdep+d+2SdRdcHf5EdCZD/umD6cWyYn8Icvj/PiF8coqmjg5ZVTCe8X4KxDa16kurGF+/62k6z8ch6cn8z3F6YQ5H/+C8awqH6snDmC26Ym8Ov1x/jTV/kUVTTwh3+ZTFiQ18x6ozlRRV0zd72Wzd5Tlfxo0Si+MzeZAJPzvmA486tLNpAiIokiEgjcDnzk4HPXAYtEZKC9OL3Ifp/TBfr78fDVKfzfiknsL6ri7teyaWxpdcWhNQ/WZGnlO3/bxc7CCl68LYNHF6dekBw6Cg4w8ZNrx/Crmyfw9XEz3/37LlpaO5fhNO1CDc2t/Nur2Rw6U80f7pjMQwtSnJocwIkJQillAR7C9sF+BHhLKXVIRJ4RkRsBRGSqiBQBtwB/EpFD9ueWA/+JLclkA8+0Faxd5RsTBvPr2zLYeaKCf39rH3rtbq07Sil+8u4BtuWb+dUtE1g28aJyWZdumZLAL5ePZ3NuGT97/4CTo9Q8mdWqeHj1HvYVVfK7FRNZMm6QS47r1OtapdSnwKed7nuyw8/Z2JqPunruX4G/OjO+y/nGhMEUVaTyy3/mMG1bJCtnjjAyHM1Nrck+xXt7TvPI1aNYPrHL07lbt05N4FRFPb/bkMf0xCi+Oblnz9d8w1+3FvDZ4XM8cf1YFqe5JjmAHkl9WffNSWJBaiz/9ekRjnTRI0XzbXkltTz1j0NcNTKa7y0Y2at9/ODqUUxLjOSJDw9SUFbXxxFqnu7g6SqeXZvDorFx3DVrhEuPrRPEZYgIv7p5AgOCA3j83f3t3WE1zWpVPP7ufoIDTLxwazp+fr0bP2PyE35zewb+fsJP3zugmzO1dpZWKz9+Zz+RoYE8d/MEl4/R0gnCAVFhQTxx/Rj2FVXxxvZu19bQfMxbO0+x80QFP71uDLEDgq9oX4PDQ3js2lS25Zt5f8/pPopQ83Svfl3I4bPVPHVDGhH9Al1+fJ0gHHRj+hCuGhnNr9Yepbyu2ehwNINV1bfwP2tzmDYiklv6qG6wYuowJg6L4L8+OUJNY0uf7FPzXCU1jfz682PMHx3jsqJ0ZzpBOEhEeOrGsdQ1W/jdhlyjw9EM9v++yqOqoYWf3zi2zy77/fyEp29Mw1zXzEub8vtkn5rn+s36XJosVp68Ic2w6X90guiBkbH9uW1qAn/POsFJc73R4WgGOVPZwCtbC1meMZS0IeF9uu8J8RFcP2Ewf9lcQEl1Y5/uW/Mcx0trWZ19im9NH0ZidKhhcegE0UM/uHoUJj/hxS+OGR2KZpDfbcgDBT9cNMop+3908WhaWq3838Y8p+xfc38vrs8lyN+P7y9MMTQOnSB6KG5AMHdMH86He89wqlxfRfias1UNvLPrFLdOjSd+YD+nHGN4VCjfnBTP6uxTlNToqwhfc7y0lo/3n+HbM0YQHRZkaCw6QfTCfXOSMInwh6+OGx2K5mJ/+iofq4LvzEl26nG+Oy8ZS6uVv2wucOpxNPfzhy+PE2jy4+6rEo0ORSeI3ogbEMwtU+J5Z2eRbif2IeV1zazOPsmyjKEkRDrn6qHNiOhQbkgfwt+zTlBVr3s0+YozlQ18sOc0K6YNI6a/sVcPoBNEr907O4kWq5W/Z+lxEb5i1Y6TNLZYuX9ukkuOd9+cJOqbW1mz86RLjqcZ72/bTmBVintmG3/1ADpB9NqI6FAWpsbx9+0n9WyvPqDZYuW1rwuZnRJNSlx/lxwzbUg4mUmRvPb1CSx6tlevV99sYdWOkywZN8hp9a2e0gniCtx11QjK65r5cK8e+ertPj1wlpKaJu5ycbvwXbMSOV3ZwLpD51x6XM313t19mqqGFu6a5R5XD6ATxBWZkRTF6Lj+vK6bmbze61knSIwOZW5KjEuPu3BMHAmRIbyeVejS42qupZTi79tOMG7oACYP7/u1pXtLJ4grICJ8a/owDp6u5kBRldHhaE5ytLiGXScq+Na0Yb2ekK+3TH7C7VOHkZVfzvHSWpceW3Od3ScrOHquhjumDzds1HRXdIK4QssnDSUkwMSbO/RVhLd6c/sJAk1+hq3VcMuUePz9hFXbdbHaW72x/SRhQf7cmD7E6FAuoBPEFRoQHMAN6YP5cO8ZapssRoej9bHGllbe23Oaa8cPIjLU9bNpAsT2D2ZRWhzv7C6iyaI7RHibqoYWPtl/lqUZQwh1s7XJdYLoA7dNHUZ9cyvrDhYbHYrWxz4/fI6aRgu3TUkwNI7bpg6jsr6FjTmlhsah9b1PD5ylyWLltqnGnmNd0QmiD0waFsGwyH58oHszeZ0P9pxmcHgwmUlRhsYxKzmK6LAgPtBrRXid9/ecJjkmlPFD+3bix76gE0QfEBGWTRzK1rwyzumR1V7DXNvEV8dKuTFjiMuL0535m/y4MX0IG3JK9MhqL1JUUc+OgnKWTxzqVsXpNjpB9JFlGUOwKvho7xmjQ9H6yMf7z2KxKpZPHGp0KAAsnziU5lYrnx48a3QoWh/50P55sTTDPc6xznSC6CNJMWGkJ0To5SK9yPt7TjNm8ABSBw0wOhQAxg0dwMjYMN7frc8xb6CU4v09p5k2ItLpc3v1lk4QfWh5xhAOn63maHGN0aFoV6igrI69pypZPtF9uh2KCMsnDmVHYbmeat4LHDpTTV5JLcvc5Aq1KzpB9KHr04dg8hN9FeEF3t9zGhG4Md293rxt/eQ/2qebMj3d+3tOE2jy4xvjBxsdSrd0guhD0WFBzEmJ5sO9p7FaldHhaL2klOKDPaeZmRzFoPBgo8O5QEJkP6aNiOS93UUopc8xT2VptfLRvjPMT40hvF+A0eF0SyeIPrZs4lDOVjWy80SF0aFovbSvqIqT5fVuWzhcOnEIx0vrOHJWN2V6qu0F5ZTWNLHMTc+xNjpB9LGFY+IINPmxVg+a81hrDxbj7ycsGhtndChdWpw2CD+BtYf0Oeap1h4sJiTAxLzRsUaHckk6QfSxsCB/ZqdEs+5QsW4C8EBKKdYdKmZGchQR/YyZWuNyosOCmDoiUo/c91BWq+0cmzc6hpBAk9HhXJJOEE6weNwgTlc2cOhMtdGhaD2UW1JLQVkdi9MGGR3KJS0ZN4ij52ooKKszOhSth/YWVVJS08SSce59joFOEE5x9Zg4TH6im5k80NqDxYjgts1LbdoS2DrdzORx1h0sJsAkzE917+Yl0AnCKSJDA5meGKnbiD3Q2oPFTBo2kNgB7tV7qbMhESFMiA/XX0I8jFKKtYeKmZkczYBg9+291MapCUJElojIURHJE5HHu3g8SETW2B/fLiIj7PcHiMhrInJARI6IyE+cGaczLBk3iLySWvJKdE8TT3GqvJ7DZ6tZ4ubNS20Wpw1i76lKzlY1GB2K5qCc4hpOmOs9onkJnJggRMQE/B64FhgLrBCRsZ02uxuoUEqNBH4NPGu//xYgSCk1HpgMfKcteXiKRWPbmgD0WsKeoq25xt3rD23aPmQ+0+eYx2hrwrzGzZsw2zjzCmIakKeUyldKNQOrgaWdtlkKvGb/+R1godimNFRAqIj4AyFAM+BRFd9B4cFMHBahmwA8yNqDxYwdPIBhUe45L05nyTFhpMSG6XPMg6w7VMzUEZFEhwUZHYpDnJkghgKnOvxeZL+vy22UUhagCojClizqgLPASeB5pVR55wOIyH0islNEdpaWut9CKkvSBnHgdBVFFXreHHdXUtPIrpMVHnPp32bJuEFsLzBTXtdsdCjaZRSW1ZFTXOMxTZjg3ATR1eTmnQcGdLfNNKAVGAIkAv8uIkkXbajUS0qpKUqpKTExMVcab59ru4zckFNicCTa5WzMKUEpz7n0b3PN2DisCr48qs8xd7f+iK0p0JPOMWcmiCKg4xp68UDnGcbat7E3J4UD5cC3gLVKqRalVAmwFZjixFidIikmjBFR/dioE4Tb25BTwuDwYFIH9Tc6lB4ZNySc6LAgNh51vyto7UJfHi0lJTbMbaf27oozE0Q2kCIiiSISCNwOfNRpm4+AlfafbwY2KNvw45PAArEJBTKBHCfG6jTzU2P5+riZhma92Ly7arZY2ZJbxvzUWLdc1etS/PyE+aNj+OpoCZZWq9HhaN2obbKwvcDMAg8Y+9CR0xKEvabwELAOOAK8pZQ6JCLPiMiN9s1eBqJEJA/4IdDWFfb3QBhwEFuieUUptd9ZsTrT/NGxNFmsbMsvMzoUrRvZheXUNbeywM3nxenOgtRYqhst7D5ZaXQoWje25JbR0qo8YnBcR/7O3LlS6lPg0073Pdnh50ZsXVo7P6+2q/s90fSkSEICTGzMKWVBque0PfqSDTklBPr7MXNklNGh9MqslGj8/YSNR0uYlhhpdDhaF748WkL/YH8mDx9odCg9okdSO1mQv4lZI6PZkFOiJ+9zUxtzSshMiqJfoFO/LznNgOAApo6I1LUuN6WUYuPREuakxBBg8qyPXM+K1kMtSI3ldGUDuSW1RoeidVJYVkd+WR0LRrtfL7ieWJAaS05xDacr9ahqd3PoTDXnqps8rnkJdIJwifmptg8f3d3V/Wy0dw/1xDdvR23nmL6KcD9t/ydzR3nelxCdIFxgcHgIqYP66zevG9qQU0JSTCjDo0KNDuWKJMeEkRAZosdDuKGNR0tIjw8npr9njJ7uSCcIF1mQGsvOExVUNbQYHYpmV9dkYXt+ucf2XupIRFgwOpateWYaW3SXandRXtfMnlOVHnuFqhOEiyxIjaXVqtiSq7u7uottx800t1o99s3b2fzUWBpaWtlecNGsNJpBNueWopStu7sn0gnCRTISIugf5M+WPD3i1V1szi0lJMDElBGe1fWwO9MTowg0+bElV59j7mLTsTIG9gtg/NBwo0PpFZ0gXMTf5MeM5Cg2HSvT3V3dxObcMqYnRRLk797rAjsqJNDE1MSBbNZXqW5BKcXm3FJmjYzGz8+zRui30QnChWaPiuF0ZQOFZj27q9GKKurJL6tjdorn9Sy5lNkpMeQU11BS3Wh0KD4vt6SWkpom5njwOaYThAvNSYkGbE0bmrHaakGz7f8n3uKqkba/Z0uevoow2qZjtvf5VR58jukE4ULDo0IZFtmPTcf0m9dom3PLiBsQREpsmNGh9KmxgwcQFRqom5ncwObcMpJjQhkSEWJ0KL2mE4SLXZUSTVa+mRY986ZhWq2KLXllzE6J8bjZWy/Hz0+4KiWazbm61mWkxpZWtheYPb4JUycIF5uTEk1tk4W9p/TMm0Y5eLqKqoYWr2teajM7JYay2iZyimuMDsVn7T5RQWOLlTmjPPsc0wnCxWYkR+MnsPmYrkMYpa0GNGukZ795u9NWh9C1LuNsyi0jwCRMT/TMGYLb6AThYuEhAWQkRLBJtxEbZlNuGWlDBnjMwvE9NSg8mFFxYboOYaDNuaVMGjaQ0CDPnCG4jU4QBpidEsP+okqq6vW0G65W22Rhz8kKj28bvpzZKTFsLyjX024YwFzbxKEz1czxwMn5OtMJwgCzU6KxKvj6uP6G52o7Csy0tCqvrT+0uSolmmaLlexCPe2Gq209bgbON/V5Mp0gDJCeEEFooImv7SeS5jrbjpsJ9PfzuJW9emp6YiT+fqLPMQNsO26mf7A/4zx0eo2OdIIwQIDJj6mJkfoKwgBZ+eVMTIggOMA7ptfoTr9AfzISInSCMMD2fDPTEyMxeej0Gh3pBGGQmclRHC+t45yeEsFlqhpaOHSmiswkz+5Z4qiZyVEcKKqkulHXulzlXHUj+WV1XnOO6QRhkJnJtvbJrHz9Dc9VsgvKsSq85s17OTOSbbWubD39t8u0vZ+95RzTCcIgYwYPIDwkgK/zdIJwlax8W/1h4rAIo0NxiYnDIgjy99PNTC6UlW9mQLA/YwYPMDqUPnHZBCEi/UTkCRH5s/33FBG53vmheTeTnzA9MZKv83UdwlWyCsxMGub99Yc2wQEmJg8fqBOEC2XllzMtMcor6g/g2BXEK0ATMMP+exHwC6dF5ENmJkdxqryBU+V6+m9ns9Ufqr3m0t9RM5OjOHK2mvK6ZqND8XrFVY0UlNWRmRRpdCh9xpEEkayUeg5oAVBKNQDekR4NNtPeT3qbrkM43Y6CcpQP1R/azLDXurbrc8zpvK3+AI4liGYRCQEUgIgkY7ui0K5QSmwY0WGBbNNNAE7XVn/ISPCN+kObCfHhesyNi3hb/QHAkYlCngLWAgki8gYwC/g3ZwblK0SEzKQovj5um5rZ26aedidZ+b5Vf2ijx9y4Tla+2avqD+DAFYRS6jPgJuBfgVXAFKXURifH5TNmJkdzrrqJ/LI6o0PxWlX1LRw+W82MJM+f+qA39Jgb5ztbZVtKeEay9zQvgWO9mL5QSpmVUp8opT5WSpWJyBeuCM4XzLSfULqZyXl2FLbVH7yneNgTesyN852vP3jXOdZtghCRYBGJBKJFZKCIRNpvI4AhrgrQ2w2P6seQ8GCdIJwoK99MkL8f6T5Wf2ijx9w4X9bxcsJDAhgzyHvqD3DpK4jvALuAVPu/bbcPgd87snMRWSIiR0UkT0Qe7+LxIBFZY398uz35tD02QUS2icghETkgIsGO/1meQ0TITI5iW74Zq1UvEekM246bmTRsoM/VH9roMTfOl1VgZlpiJH5eVH+ASyQIpdRvlFKJwI+UUklKqUT7LV0p9X+X27GImLAlkmuBscAKERnbabO7gQql1Ejg18Cz9uf6A38H7ldKpQHzsHez9UYzk6Mpr2vm6Dm9RGRfq6xv5kix741/6EyPuXGeM5UNnDDXe+U55kiR+nciMk5EbhWRb7fdHNj3NCBPKZWvlGoGVgNLO22zFHjN/vM7wEKxdeVZBOxXSu2zx2BWSnntyiczdB3CadrGP3hb8bCn9Jgb59le4J31B3CsSP1z4Hf223zgOeBGB/Y9FDjV4fci+31dbqOUsgBVQBQwClAisk5EdovIj7uJ7T4R2SkiO0tLPXf93aERIQyP6qf7qjtBVn65vf7g+XPzXwk95sZ5th03e2X9ARwbKHczsBAoVkr9G5AOOLKYb1eNcZ0b2bvbxh+4CrjD/u9yEVl40YZKvaSUmqKUmhIT49nL+81MjmJ7gZlWXYfoU1n5ZiYPH0iQv2/WH9p0HnOj9Z2s/HKme2H9ARxLEA1KKStgEZEBQAmQ5MDzioCEDr/HA2e628ZedwgHyu33f6WUKlNK1QOfApMcOKbHykyKoqbRwuEz1UaH4jV0/eFCM5KjOFfdRKFZ1yH6yunKBk6We2f9ARxLEDtFJAL4M7ZeTLuBHQ48LxtIEZFEEQkEbgc+6rTNR8BK+883AxuU7evNOmCCfSZZf2AucNiBY3qsthNM91XvO9t9dP6l7uhzrO9t98L5lzq6ZIKwF4x/qZSqVEr9EbgGWGlvaroke03hIWwf9keAt5RSh0TkGRFpq2G8DESJSB7wQ+Bx+3MrgBewJZm9wG6l1Ce9+gs9RNyAYJKiQ/Wbtw9l5ZsJDtD1hzZJ0aHE9A/S51gfysq31R9SB/U3OhSnkMu1R4rILqXUZBfF02tTpkxRO3fuNDqMK/KT9w7w8b4z7P35Iq+az8Uo1/5mM5GhAbxxT6ZD27e0tFBUVERjo/dOSVFe10yTxcrgcK8cVuRyxVWNBJiEqDBHyrLGCg4OJj4+noCAgAvut3/GT+nqOY5M1pclIlOVUtl9EaTWvRnJUazacZJDZ6qYEO+bo377SmV9MznF1fzw6lEOP6eoqIj+/fszYsQIr5040VzbxOnKBpLi+hPkowMH+0qzpZWW4hqGRIQQ7eYJQimF2WymqKiIxMREh5/nSA1iPrBNRI6LyH77qOb9vY5U61Zmoq0ftW4CuHLt9YcejH9obGwkKirKa5MDQFiQ7TthbZPF4Eg8X12TbWhWaKAj37ONJSJERUX1+OrYkb/s2t6FpPVU7IBgkmJCycov5745yUaH49G2HbfVHybE96z+4M3JASDQ348Akx91Ta1EhRkdjWerbbJg8hOCAxz5nm283pzbl00QSqkTvYpG65XMpCj+sfcMllYr/ibPOPHcUVa+mSnDI31+/ENnIkJooD91zRa9BskVqmu2EBbk79Wvof4EcjOZSVHUNFk4fFaPh+itirpmcoprPHLqg+LiYm6//XaSk5MZO3Ys1113HceOHevxfq677joqKyu7fCw0yERLq5Vmi/WS+0hPT2fFihU9PvaV+td//VfeeecdAO655x4OH+5dD/fCwkLGjRvX5f4TExPJyMggPT2dL77oevWCrKwspk+fTkZGBmPGjOGpp55qf6zZ0kqzxUpo0Pnv2GFhV3ZJNmLECMrKLj2hYsfXxhXcv/HMx7TVIbYdN+tCdS9tLygHPK9vulKK5cuXs3LlSlavXg3A3r17OXfuHKNGOV5sB/j000+7faxjHaK7QvWRI0ewWq1s2rSJuro6QkNDe3T8vvKXv/zFKfv91a9+xc0338zGjRu57777yM3NvWiblStX8tZbb5Genk5raytHjx5tf6zWQ+oPFosFf//ex6ivINxM7IBgkmP0eIgr0Tb+wdMS7MaNGwkICOD+++9vvy8jI4PZs2ejlOLRRx9l3LhxjB8/njVr1gBw9uxZ5syZQ0ZGBuPGjWPz5s3A+W+jhYWFjBkzhnvvvZe0tDQWLVpEa0sTASY/DufksmTJEiZPnszs2bPJyclpP+6bb77JnXfeyaJFi/joo/PjW+fNm8djjz3GtGnTGDVqVPvxXn31VW666SaWLFlCSkoKP/7x+enTPvvsM2bMmMGkSZO45ZZbqK2tBeCZZ55h6tSpjBs3jvvuu6/LKUDmzZtHW/f1sLAwfvazn5Genk5mZibnzp0D4Pjx42RmZjJ16lSefPLJHn2TnzFjBqdPn+7ysZKSEgYPHgyAyWRi7FjbZNS1tbXcf+/d3HzNLKZPmci7777b/pyu4istLeWb3/wmU6dOZerUqWzduhUAs9nMokWLmDhxIt/5znfa//7OVz7PP//8BVcvbXbt2sXcuXOZPHkyixcv5uzZs+2v2U9/+lPmzp3Lb37zG4dfi644MllfjYhUd7qdEpH3RcSRKTe0HspMiiK7sAJL66WbALSutdUfAv096/vPwYMHmTy56yFH7733Hnv37mXfvn2sX7+eRx99lLNnz/Lmm2+yePHi9scyMjIuem5ubi4PPvgghw4dIiIigvfee4/QIH8efeRBfvvb37Jr1y6ef/55HnjggfbnrFmzhttuu40VK1awatWqC/ZnsVjYsWMHL774Ik8//XT7/Xv37mXNmjUcOHCANWvWcOrUKcrKyvjFL37B+vXr2b17N1OmTOGFF14A4KGHHiI7O5uDBw/S0NDAxx9/fMnXp66ujszMTPbt28ecOXP485//DMDDDz/Mww8/THZ2NkOG9Gwts7Vr17Js2bIuH3vkkUcYPXo0y5cv509/+lN7D6D//M//JDi0P59v2cH+/ftZsGDBZeN75JFHyM7O5t133+Wee+4B4Omnn+aqq65iz5493HjjjZw8edLhuFtaWvje977HO++8w65du7jrrrv42c9+1v54ZWUlX331Ff/+7//eo9ejM0euPV7ANofSm9gm17sdGAQcBf6Kba0GrQ9lJkXxxvaTHDpT7bOroPVWub3+8Ohi71r0cMuWLaxYsQKTyURcXBxz584lOzubqVOnctddd9HS0sKyZcu6TBBt7e0AkydPprCwkNktDezN3sEtt9xKW421qakJgOzsbGJiYhg+fDjx8fHcddddVFRUMHDgQABuuummC/bVZuHChYSH23qNjR07lhMnTlBZWcnhw4eZNWsWAM3NzcyYMQOwXTE999xz1NfXU15eTlpaGjfccEO3r0FgYCDXX399+7E///xzALZt28YHH3wAwLe+9S1+9KMfXfb1fPTRR/nxj39MSUkJWVlZXW7z5JNPcscdd/DZZ5/x5ptvsmrVKr788ks+/3w9T//mpfb6Q9vr0l1869evv6COUl1dTU1NDZs2beK9994D4Bvf+Eb7fhxx9OhRDh48yDXXXANAa2tr+9UOwG233Yz42aQAACAASURBVObwvi7FkQSxRCk1vcPvL4lIllLqGRH5aZ9EoV1getL58RA6QfTMDg+emz8tLa3bAmR3Mx7MmTOHTZs28cknn3DnnXfy6KOP8u1vX7hcS1DQ+UFcJpOJhoYG+vn70T88nA1bt180CnjVqlXk5OQwYsQIwPaB1vGbb9v+TCYTFsv58RSdj2Ox2HpKXXPNNRddhTQ2NvLAAw+wc+dOEhISeOqppy7bRz8gIKC9x1DnY/fUr371K2666SZ++9vfsnLlSnbt2tXldsnJyXz3u9/l3nvvJSYmBrPZTKvViiAXFKgvFZ/VamXbtm2EhIRctP+uekD5+/tjtZ5vPejqdVFKkZaWxrZt27qMu69qRo5cg1vtiwX52W+3doyzT6LQLhDbX9cheisrv5yQABPjh3peYl2wYAFNTU3tTRNg+zb/1VdfMWfOHNasWUNrayulpaVs2rSJadOmceLECWJjY7n33nu5++672b17t0PHio6MID5hOG+99TZg+8DZt28fVquVt99+m/3791NYWEhhYSEffvjhRR/wjsrMzGTr1q3k5eUBUF9fz7Fjx9o/9KKjo6mtrb2injmZmZntdYC24r4j/Pz8ePjhh7Faraxbt+6ixz/55JP2xJybm4vJZCIiIoJZcxew5rU/E2xvwqyoqLjkcRYtWsT//d/5RTj37t0L2JL7G2+8AcA///nP9v3ExcVRUlKC2Wymqampy6a30aNHU1pa2p4gWlpaOHTokMN/u6McSRB3AHdim+b7nP3nfxGREGyT8WlOMCNZ1yF6IyvfzJQRAz2u/gC2b5Pvv/8+n3/+OcnJyaSlpfHUU08xZMgQli9fzoQJE0hPT2fBggU899xzDBo0iC+//JKMjAwmTrQVSx9++GGHj/W7P7/CG6+/Snp6OmlpaXz44Yds2rSJoUOHMnTo+bW95syZw+HDh9uLoD0RExPDq6++yooVK5gwYQKZmZnk5OQQERHBvffey/jx41m2bBlTp07t8b7bvPjii7zwwgtMmzaNs2fPtjdzga0pJj4+vv329ttvX/Q6/Md//AfPPffcRft9/fXXGT16NBkZGdx555288cYbmEwm7v7ev1NfW8348eNJT09n48aNl4zvt7/9LTt37mTChAmMHTuWP/7xjwD8/Oc/Z9OmTUyaNInPPvuMYcOGAbYrkSeffJLp06dz/fXXk5qaetE+AwMDeeedd3jsscdIT08nIyODr7/+usev3eVcdrI+T+ENk/V19PH+Mzz05h4+eHAWGbqZySHldc1M+s/PeXTxaB6cP7LHzz9y5AhjxoxxQmTuqbyuiaKKBkbF9SfYg+dlqq+vJyQkBBFh9erVrFq1ig8//NApx2q2tJLjIfMvdaWrc/yKJusTkRjgXmBEx+2VUnddUaTaJU1PPD93v04Qjjk/N7/n1R+M0NaHv67J4tEJYteuXTz00EMopYiIiOCvf/2r047VNv4hLMi9xz/0FUf+yg+BzcB6oNW54WhtYvoHMTI2jKx8M/fP1fMyOSIr30xIgKlPxj88/Y9Dfb6639ghA/j5DWl9us8rcX5eJotHTFfdndmzZ7Nv3z6XHKuuyYK/nx9BHtiE2RuOJIh+SqnHnB6JdpHMpEje331az8vkoKz8cqaMGEiAfq0cImLriVPbqOdlcoRSiromC6FBJp95rRxJEB+LyHVKqe7H7mtOkZkUxd+zTnLgdBUThzneR9oXmWubOHquhhsz+mb8gxHf9M1mMwsXLgRsczKZTCZiYmIA2LFjB4GBgX1+zLAgE5X1tkWEumpmMplMjB8/HoBhw4a1j6ouKCjg9ttvp7y8nEmTJvH66687JT530tJqpbnVSnSQ515t9ZQjX7UexpYkGuyjqGtERM8k5wLn1xAuNzgS97fDQ+df6igqKoq9e/eyd+9e7r//fh555JH235314dvWl7+um/UhQkJC2mPoOOXGY489xiOPPEJubi4DBw7k5Zdfdkp87sTX6g/gQIJQSvVXSvkppUKUUgPsvw9wRXC+LjosiBR7HUK7tG35ZvoFmnq8/oOnKiwsJDU1lXvuuYdx48Zxxx13sH79embNmkVKSgo7duwAbFceM2fOZOLEicycObN9wrkXXniBu+66i0CTHwXHjjBz6kTq6+sdOrZSig0bNnDzzTcDtknt2kYyezNfqz/AJRKEiKTa/53U1c11Ifq2zKQodhaW06LHQ1ySbfxDpE/VH/Ly8nj44YfZv38/OTk5vPnmm2zZsoXnn3+e//7v/wYgNTWVTZs2sWfPHp555hl++lPb5Ac/+MEPyMvL44MPPuCJHz7IE//zIiEhIezcubN9xDTYRvFOmTKFzMzM9iRgNpuJiIhonyU0Pj6+2wnvvIVSilofqz/ApWsQPwTuA/63i8cUsMApEWkXyEyK4vWsExzUdYhuldU2cexcLcsmDr38xl4kMTGxvT6QlpbGwoULERHGjx/fPkdSVVUVK1euJDc3FxGhpaUFsI0ifvXVV5kwYQIr77qHCZOn0WSxMmXKlAum2D558iRDhgwhPz+fBQsWMH78eAYMuLgBwds/NJtbrbS0Wgn1ofoDXOIKQil1n/3f+V3cdHJwkbZ5mbbpZqZueUP9oTc6zn3k5+fX/rufn1/7PEBPPPEE8+fP5+DBg/zjH/+4YF6f3NxcwsLCKC0pBrquQ7TNjpqUlMS8efPYs2cP0dHRVFZWth+jqKiox7Ooepo6H6w/gIPrQYjITBH5loh8u+3m7MA0m+iwIEbFhelC9SVk2esP44f6Rv2hJ6qqqtqnzXj11VcvuP/hhx9m06ZNVJaXs+GfH1HbKUFUVFS0z/BaVlbG1q1bGTt2LCLC/Pnz2+dPeu2111i6dKlr/iCD+GL9ARxbD+J14HngKmCq/dblsGzNOXQd4tJ8sf7gqB//+Mf85Cc/YdasWbS2nh/n+sgjj/DAAw8watQoXn75ZV785dOcPF1MdnZ2ew3iyJEjTJkyhfT0dObPn8/jjz/evmjOs88+ywsvvMDIkSMxm83cfffdhvx9ruCr9QdwYC4mETkCjFVuPmmTt83F1NGnB87ywBu7ee+BmUzSdYgLlNU2MeUX63lsSSrfnXdlI859bS6mjsrrmimqqPf4eZmcocnSytHiGoZGhHj0iHPo+VxMjnzlOohtgSDNINMSz68PoV1oe35b/UHPv3QlQoNsSaG78RC+rO016bz+gy9w5C+OBg6LyA6gqe1OpdSNTotKu0BbHWLbcTMPzOv5LKXeLCvfTGigiXFeWn/oOLq6oy+++IKoqL4rygeabPMy1Xr4vEzOUNfU6pP1B3AsQTzl7CC0y8tMiuLtnUW0tFp1W3sHfV1/cLc5idpGVzubiBAW5E+NnpfpAt5Uf+hNleCS7yoRMQFPKKW+6nzrbZBa78xIiqKhpZX9RVVGh+I2ymqbyC2p7bPurcHBwZjN5l69kbxBaJA/FquVJovuDNGmbfyDp3dvVUphNpsJDg7u0fMu+VcrpVpFpF5EwpVS+pPJQB3rEJOH60I1nK/JzEjumwQRHx9PUVERpaWlfbI/T2OxWjlX1URTWYDHfyD2lbomCxX1LVAZRImHX7kHBwcTHx/fo+c4chY0AgdE5HOgru1OpdT3exaediWiwoIYHdefrHxzr1ZL80bt9YchfTM1WEBAAImJiX2yL0+klOL+ZzeSnhDO/7tjvNHhuIUfrN7Dljwz2T+72uObmHrDkZT4CfAEsAnY1eF2WSKyRESOikieiDzexeNBIrLG/vh2ERnR6fFhIlIrIj9y5HjeLjMpkp2FFTTrJgDANsvt1MRIvVZGHxERpidFkpVfjtXqm81sHSmlyMovZ3pSlE8mB3BsNtfXurpd7nn2+sXvgWuBscAKERnbabO7gQql1Ejg18CznR7/NfBPR/4QX5Bpr0McOF1pdCiGK61pIq+ktn1pVq1vZCZFUV7XTG5JrdGhGK7QXE9xdSOZib7bhdqRkdQpIvKOiBwWkfy2mwP7ngbkKaXylVLNwGqg83j8pUBbsnkHWCj2VC0iy4B84JCjf4y3m67Xh2jXNjfVzD6qP2g2M5LOr4Xu67Ydb6txRRsciXEcuTZ/BfgDYAHmA38DXnfgeUOBUx1+L7Lf1+U2SikLUAVEiUgo8Bjw9KUOICL3ichOEdnpC4XFyNBAUgf1129eYNvxMvoH+5PWR/UHzSYhsh9DI0L0OQZ8fbyM2P5BJMeEGh2KYRxJECFKqS+wTctxQin1FI5N9d1Vo13nhs3utnka+LVS6pLXuUqpl5RSU5RSU9qWZvR2tnmZdB1i23Ez0xOjdP3BCTKTothe4Nt1CFv9wczMZN+tP4BjCaJRRPyAXBF5SESWA7EOPK8ISOjwezxwprttRMQfCAfKgenAcyJSCPwA+KmIPOTAMb1eZlKkz9chTlc2UGiu181LTpKZFOnzdYjcklrKapuZ6cPNS+BYgvgB0A/4PjAZ+BdgpQPPywZSRCRRRAKB24GPOm3zUYd93QxsUDazlVIjlFIjgBeB/1ZK/Z8Dx/R60+xF2bb2UV90vm1YJwhnaBt4uO14mcGRGOfrPNvf7uvnmCO9mLLtTT0VSql/U0p9UymV5cDzLMBDwDrgCPCWUuqQiDwjIm3zOL2MreaQh20Fu4u6wmoXOl+H8N1C9dfHy4gMDWR0XH+jQ/FK5+sQvnyOmYkfGEJCZD+jQzHUZQfKicgMbB/kYcAwEUkHvqOUeuByz1VKfQp82um+Jzv83Ajccpl9PHW54/iazKQoVmefpNliJdDHJhBTSpF13MyMpCj8/Hy3bdjZZiRH8cWRc1ityude51arYntBOYvT4owOxXCOfLq8CCwGzABKqX3AHGcGpV1aZlIUjS1W9hf5Xh3ihLmeM1WNPn/p72yZSVFU1LdwrKTG6FBc7sjZaqoaWny+/gAOLjmqlDrV6a7WLjfUXGK6D68P8bWuP7hE+znmg7Wur4/r+kMbRxLEKRGZCSgRCbRPe3HEyXFplzDQXofY5pMJooy4AUEkRftu33RXSIjsR/zAEB89x8wkx4QSN6BnM596I0cSxP3Ag9gGtRUBGcBl6w+ac2UmRbHrRAVNFt+5mDvfNz3ap/umu4ovjodoabWSXVCurx7sHOnFVKaUukMpFaeUilVK/QvwbRfEpl3CjOS2OoTvzMLe1jddv3ldY0ZSFJX1LRw95zt1iP1FVdQ1t+r6g11vu8D8sE+j0HpsemIkIr7VRtzeN72PFgjSLm16ku/VutrGfvTVIlSerrcJQl/fGyyiXyCpgwaQVeA7b96vj5tJiNR9010lfmA/EiJ9a16mr4+bGTN4AJGhgUaH4hZ6myB8p1HSjbWtD+ELdYi2vukzk/SlvytlJvpOHaKxpZVdJyr0FC4ddJsgRKRGRKq7uNUAQ1wYo9aNzKQomixW9p3y/jpEe9/0kfrN60qZ9jpETrH31yH2nKykyWLVCaKDbhOEUqq/UmpAF7f+Sim9YK0baK9D+EATQHvfdN027FKZyb6zPsS242X4CUz14QWCOvOteRq8TES/QMYMGtD+4enNtubZ+qbH6r7pLjU0IoRhkf3aByh6s63HzYyPj2BAcIDRobgNnSA83FUp0ew6UUF9s8XoUJymydLK9gIzs1N8Y80Pd3NVSjRZ+WZaWr13DZLqxhb2nqpk9khd4+pIJwgPNzslmpZWxXYvnnlzV2EFjS1WZqfoN68RZo+MprbJwt5T3jv317bjZlqtSp9jnegE4eGmjogkyN+PTbneu+TqptwyAkyi+6YbZGZyNH4Cm4957zm2ObeUfoEmJg4baHQobkUnCA8XHGBiWmIkW3K9tw6xJa+UicMGEhqk+0YYIbxfAOkJEWzO8+JzLLeMGUlRPjd9/uXoV8MLzE6JJreklrNVDUaH0ufMtU0cPF3NHH3pb6jZKTHsO1VJVX2L0aH0uZPmegrN9bp5qQs6QXiBtuLtZi+8ithi/9aqC9TGmpMSjVXhlT3mNufZms6u0ufYRXSC8AKpg/oTHRbklc1MW3LLiOgXwLih4UaH4tPSEyIIC/L3ymamLbllDAkPJjlGTyHfmU4QXkBEmJ0SzZa8Mq+aEkEpxebcMmYlR2PysWUv3U2AyY8ZyVFsOlaKUt5zjllarWzNK2N2SoyeQr4LOkF4idkp0ZTXNXP4bLXRofSZvJJaiqsbdduwm5iTEk1RRQMnzPVGh9Jn9p+uorrRwlX6HOuSThBe4ir7AB9vqkO0/S36zese2mtdXtTMtCW3DBGYpQfIdUknCC8ROyCY1EH92eRFfdU35ZaSFB1K/EA9vbc7GB5lm/7bq86xY6WMGxKup/fuhk4QXmTu6BiyC8upafT8rogNza1sO25m7mjds8RdiAhzR8WwNa/MK6aYr6xvZvfJCubpc6xbOkF4kQWjY7FYlVf0ZtqWX0aTxcqC1FijQ9E6WJAaS31zKzsKPH9ql6+OlWJVMF+fY93SCcKLTBo+kP7B/mzIKTE6lCu2IaeEfoG2UeKa+5iRFE2Qv59XnGMbc0qIDA0kPT7C6FDclk4QXiTA5MecUTF8eazUo7u7KqXYmFPKrJHRBPmbjA5H6yAk0MSM5Ci+POrZdYhWq+KrY6XMHRWju1Bfgk4QXmbB6FhKa5o4dMZzu7vmltRyurJBNy+5qQWpsRSU1VFQVmd0KL2291QlFfUtunnpMnSC8DJzR8cggkc3AbTFPn+0fvO6o7b/F08+xzbmlGDyE+bq6TUuSScILxMdFsSE+Ag2HPXcN++GnBLGDB7AoHC9epw7Sojsx8jYMDZ6cILYkFPC5GEDCe+nV4+7FJ0gvNCC0bHsL6qkrLbJ6FB6rKqhhV0nKliQqr/ZubMFqbFsLzBT1+R5KxkWVzVy+Gy1bl5ygFMThIgsEZGjIpInIo938XiQiKyxP75dREbY779GRHaJyAH7vwucGae3WZAai1LwlQcWEjfnltJqVbr+4Obmj46lpVW1z7brSb60X13P119CLstpCUJETMDvgWuBscAKERnbabO7gQql1Ejg18Cz9vvLgBuUUuOBlcDrzorTG6UNGUB0WJBHNjNtyCkhol8AGQl6ZS93NmXEQPoH+XtkM9OGnBKGhAczOq6/0aG4PWdeQUwD8pRS+UqpZmA1sLTTNkuB1+w/vwMsFBFRSu1RSp2x338ICBaRICfG6lX8/IQFqTFsOlrqUSNeLa1WNuaUME93PXR7bV2q1x8podWDulQ3trSyObeMBWNi9eytDnBmghgKnOrwe5H9vi63UUpZgCqg88LD3wT2KKUualAXkftEZKeI7Cwt9bzmFGdaMm4QNU0Wvj5uNjoUh+0oLKeivoUl4wYZHYrmgMXjBlFW28SekxVGh+KwTcdKaWhpZUnaYKND8QjOTBBdpefOXzUuuY2IpGFrdvpOVwdQSr2klJqilJoSE6PbEzuamRxNWJA/6w4WGx2Kw9YdLCY4wPbNVHN/80fHEGjyY60HnWNrDxUTHhLA9CQ9Qt8RzkwQRUBCh9/jgTPdbSMi/kA4UG7/PR54H/i2Uuq4E+P0SsEBJuanxvLZ4XMe0QRgtSrWHTrH3FEx9Av0NzoczQH9gwO4KiWatYeKPWIRoZZWK+sPn+PqMXEEmHQHTkc481XKBlJEJFFEAoHbgY86bfMRtiI0wM3ABqWUEpEI4BPgJ0qprU6M0astSRtEeV0z2YXuP7HavqJKiqsbdfOSh1mSNoiiigaPGLmflW+mutGiz7EecFqCsNcUHgLWAUeAt5RSh0TkGRG50b7Zy0CUiOQBPwTausI+BIwEnhCRvfab7vfYQ/NGxxDo78e6Q+7fBLD2UDH+fsKC1DijQ9F64OqxcfgJfOYB59i6Q8X0CzTpFQp7wKnXWUqpT5VSo5RSyUqp/7Lf96RS6iP7z41KqVuUUiOVUtOUUvn2+3+hlApVSmV0uHlefzqDhQb5MyclmnUH3bsJQCnFuoPFzEiOIjxEj2z1JJGhgUxLjGStmyeItibMeaNjCA7QE0A6SjfEebnFaYM4U9XIgdNVRofSraPnaig01+tLfw+1JG0Qx87Vcry01uhQurXnVAWlNU0sTtPnWE/oBOHlrh4Th8lP3LqnydqDxYjANWN185InWmT/0HXnpsy1B4sJNPnpEfo9pBOElxsYGkhmUiT/dONmprUHi5k8bCCx/fXkfJ5oSEQI6QkR/POAeyYIpRT/PFjMzJFR9A/WTZg9oROED7h+whAKyurcspnpyNlqcopruCF9iNGhaFfghgmDOXC6yi2bmXafrKCoooEbJuhzrKd0gvAB140fTKDJj/f3nDY6lIt8sPc0/n7C9RP0yFZPdmP6EPwEPnTDc+z9PacJDvBjsa5x9ZhOED4gPCSAhWNi+ce+M1harUaH085qVXy45wxzR8UQFaan2vJksQOCmTUymvf3nnarpsxmi5WP959lcdogwoL0AMye0gnCRyybOJSy2mY2u9H0zFkFZoqrG1k2sfMUXZonWj5xKKfKG9h1wn3mZvryaAmV9S36HOslnSB8xLzRMYSHBPCBGzUBfLDnNGFB/lw9Rvde8gaL0wYREmByq6bMD/aeJio0kNkj9eC43tAJwkcE+Zu4fsJg1h0qptYNVgFrbGnlnweKWTJuECGBeuCSNwgN8mdRWhwf7z9Ls8X4psyqhhbWHynhhvQh+Ou5l3pFv2o+ZPnEoTS2WN1iWoQvjpRQ02Rhub709yrLJg6lqqGlfdU2I609aEtU+hzrPZ0gfMjk4QOJHxjiFk0A7+85TdyAIDKTOi//oXmy2SOjiQoNdJtzLCk6lAnx4UaH4rF0gvAhIsJNk+LZkldGUUW9YXGUVDey8WgJyzKG6pXjvIy/yY+lGUNZf+Qc5tqL1vhymcKyOrLyy1k+caheOe4K6AThY26dEg/AW9mnLrOl87y9q4hWq+L2acMMi0FznhXTEmhpVby7u8iwGFZnn8LkJ9w6NeHyG2vd0gnCx8QP7Me8UTGs2XnKkDERVqti1Y6TzEyOIjE61OXH15wvJa4/U0cMZNWOU4aMiWi2WHln1ykWpsYSN0BP33IldILwQd+aPpxz1U2sP+L6QuKm3FKKKhr41nR99eDNvjV9GAVldYasif7Z4WLKapv1OdYHdILwQfNHxzAkPJjXvi50+bFf/bqQ6LAgFo3V0x54s2vHDWZgvwBeNeIc21pIQmQIs1P02uZXSicIH+Rv8uPbM0ewLd/MYRcuFZlXUsuXR0u5M3M4gf761PNmwQEm7pg+nPVHznHCXOey4+47VcnOExX868xE3QGiD+h3qY9aMXUYIQEmXtla4LJjvvp1AYH+ftyRqS/9fcGdM4bj7ycuvYp4ZWsBYUH+7Z0xtCujE4SPCu8XwM2T4/lw7xnOVTc6/Xjm2ibe2VXE0vQhROuJ+XxC3IBgrp8whLeyT1FZ3+z0452ubODj/We5ZUq8Xvehj+gE4cPunZ1Eq1K8tCnf6cd6eUsBTRYr35mb7PRjae7jO3OTqGtu5ZWthU4/1p++Oo4I3DM7yenH8hU6QfiwYVH9WJo+hDe2n3DqoKaq+hb+tu0E140fzMjYMKcdR3M/qYMGcM3YOF7ZWkBNY4vTjlNS3cjq7FPcNDGeoREhTjuOr9EJwsc9MD+ZJouVlzY77yri5a0F1DZZeGj+SKcdQ3NfD80fSXWjxam95v74VT6WVivfnaevUPuSThA+bmRsf5ZlDOXVrYWcqWzo8/2X1DTyl835XDd+EGMGD+jz/WvuLz0hgqvHxPGnr/Ipr+v7WsSp8nr+nnWCmyfHM0IPvuxTOkFo/PCaUSgFv/78WJ/v+7df5NJssfLo4tQ+37fmOR5bMpq6Zgu/25Db5/v+38+OIgKPXDOqz/ft63SC0EiI7Me3Zwznnd1F7C+q7LP95hRXs2rHKVZMG6an1fBxKXH9uXVKAn/POkFeSU2f7Xf3yQo+2HuGu65KZHC4rj30NZ0gNAC+f3UK0WFB/Oz9g7Rar3z+HKtV8dP3DhAeEsAP9Tc7DfjR4tH0C/TnZ+8f7JM5mlparfz0vQMMDg/mQV3fcgqdIDQABgQH8OT1YzlwuqpPBja9ueMku09W8tPrxjAwNPDKA9Q8XnRYEI9fm8r2gnLe2nnlswm/vKWAnOIafn5DGmFB/n0QodaZThBau+snDGZhaizPrs3hyNneT8GRV1LDLz45zFUjo/nmJL2al3bebVMSmJ4YydP/OEx+aW2v93OgqIr//ewoS9IGsThNr2nuLDpBaO1EhOdunkB4SADfW7WnV2tX1zdbeOjNPYQG+vPCrel6sRbtAn5+wou3ZxDo78f3V++hsaW1x/uoamjh+6v3EB0WxP98c7w+x5xIJwjtAlFhQfzmtgwKyur47t930dKDNSMsrVa+9+Yejp2r4X9vTSdWz8WvdWFweAjP35zOoTPVPLx6T49qXk2WVu5/fRdFFfX85vaJRPTTzZfOpBOEdpGZI6P55fLxbM4t4wer99Jkufy3vGaLlUff2c8XOSU8s3Qc80bHuiBSzVNdPTaOJ74xlnWHzvH4u/sd+iLS2NLK91ftYVu+mWe/OYFpiZEuiNS3OTVBiMgSETkqInki8ngXjweJyBr749tFZESHx35iv/+oiCx2ZpzaxW6dmsB/fGMMnxw4y8q/7rjkhH4lNY3c/Vo27+85zaOLR/MvmcNdGKnmqe66KpEfXJ3C27uKuOe1nZRdYrqXM5UN3Pnydj47fI6nbhjLTZP0bK2uIM5aElBETMAx4BqgCMgGViilDnfY5gFgglLqfhG5HViulLpNRMYCq4BpwBBgPTBKKdXtV9kpU6aonTt3OuVv8WUf7j3Nj9/ZT6DJj/vnJXPL5Pj2pqPSmibe213E7zfm0dhi5RfLx3HrFL0GsNYzq3ac5MkPDxIa5M+D80Zy06ShRNln/D1X3cjbO0/xhy+P06oUz9+SzvUThhgcsXcRkV1KqSldAkePTAAACFBJREFUPubEBDEDeEoptdj++08AlFK/7LDNOvs220TEHygGYoDHO27bcbvujqcThPOcMNfx5IeH+OpYKQDRYYGAtH/jmzUyimeWjiM5Rk/Ep/VOXkkN//HBQbLyywGI6R+EUoqyWtvUHPNHx/DM0nEkRPYzMkyvdKkE4czOw0OBjp2di4Dp3W2jlLKISBUQZb8/q9NzL+ovKSL3AfcBDBumF6FxluFRobx21zTyS2tZe6iYk+Z6lLLNBrs4LY6Rsf2NDlHzcCNj+7P6vhkcO1fD54dtq9D5iTA8KpQl4wbpkfgGcWaC6KrvWefLle62ceS5KKVeAl4C2xVETwPUeiYpJowH5ukRq5rzjIrrz6g4/YXDXTizSF0EdGyQjgfOdLeNvYkpHCh38LmapmmaEzkzQWQDKSKSKCKBwO3AR522+QhYaf/5ZmCDshVFPgJut/dySgRSgB1OjFXTNE3rxGlNTPaawkPAOsAE/FUpdUhEngF2KqU+Al4GXheRPGxXDrfbn3tIRN4CDgMW4MFL9WDSNE3T+p7TejG5mu7FpGma1nOX6sWkR1JrmqZpXdIJQtM0TeuSThCapmlal3SC0DRN07rkNUVqESkFTlzBLqKBsj4Kpy/puHpGx9Vz7hqbjqtnehvXcKVUTFcPeE2CuFIisrO7Sr6RdFw9o+PqOXeNTcfVM86ISzcxaZqmaV3SCULTNE3rkk4Q571kdADd0HH1jI6r59w1Nh1Xz/R5XLoGoWmapnVJX0FomqZpXdIJQtM0TeuSzycIEVkiIkdFJE9EHjcwjgQR2SgiR0TkkIg8bL//KRE5LSJ77bfrDIqvUEQO2GPYab8vUkQ+F5Fc+78DXRzT6A6vy14RqRaRHxjxmonIX0WkREQOdrivy9dHbH5rP+f2i8gkF8f1KxHJsR/7fRGJsN8/QkQaOrxuf3RWXJeIrdv/OxH5if01Oyoii10c15oOMRWKyF77/S57zS7xGeG880wp5bM3bNOQHweSgEBgHzDWoFgGA5PsP/cHjgFjgaeAH7nBa1UIRHe67zngcfvPjwPPGvx/WQwMN+I1A+YAk4CDl3t9gOuAf2JbOTET2O7iuBYB/vafn+0Q14iO2xn0mnX5f2d/L+wDgoBE+/vW5Kq4Oj3+v8CTrn7NLvEZ4bTzzNevIKYBeUqpfKVUM7AaWGpEIEqps0qp3fafa4AjdLEOt5tZCrxm//k1YJmBsSwEjiulrmQ0fa8ppTZhW9Oko+5en6XA35RNFhAhIoNdFZdS6jOllMX+axa2FRtdrpvXrDtLgdVKqSalVAGQh+3969K4RESAW4FVzjj2pVziM8Jp55mvJ4ihwKkOvxfhBh/KIjICmAhst9/1kP0S8a+ubsbpQAGficguEbnPfl+cUuos2E5eINag2MC22FTHN607vGbdvT7udN7dhe1bZptEEdkjIl+JyGyDYurq/85dXrPZwDmlVG6H+1z+mnX6jHDaeebrCUK6uM/Qfr8iEga8C/xAKVUN/AFIBjKAs9gub40wSyk1CbgWeFBE5hgUx0XEtqTtjcDb9rvc5TXrjlucdyLyM2wrNr5hv+ssMEwpNRH4IfCmiAxwcVjd/d+5xWsGrODCLyIuf826+IzodtMu7uvRa+brCaIISOjwezxwxqBYEJEAbP/xbyil3gNQSp1TSrUqpazAn3HSZfXlKKXO2P8tAd63x3Gu7ZLV/m+JEbFhS1q7lVLn7DG6xWtG96+P4eediKwErgfuUPYGa3vzjdn+8y5s7fyjXBnXJf7v3OE18wduAta03efq16yrzwiceJ75eoLIBlJEJNH+LfR24CMjArG3bb4MHFHq/7d3Ny9VhFEcx78HC7EXgiwiiF4kV0G1kIhoFS0qKIgWFkESbnJTq3Dhtk2bCDGIggj6C3IVxV0EUeQiMhV6Q1oEBhpURCEip8VzLow6UwjXGcPfB4Y7Hsfruc8M98wzL8/4jUw8e8zwNDA6/29LyG2tma2vz5NOco6S2qorFusCHpadW5izV7cc2iwUtc8gcCGuMjkIfK8fIiiDmR0DeoFT7v4rE99sZk0x3wa0A+Nl5RX/t2jdDQJnzazZzHZFbkNl5gYcBd66++d6oMw2K/qOYCm3szLOvi/niXSm/z2p8vdVmMdhUvfvDfA6phPAA2Ak4oPA1gpyayNdQTIMjNXbCWgFasCHeN1YQW5rgK/Ahkys9DYjFagJYIa059Zd1D6krv+t2OZGgI6S8/pIOjZd385ux7JnYv0OA6+AkxW0WeG6A/qizd4Bx8vMK+L3gUvzli2tzf7yHbFk25mG2hARkVwr/RCTiIgUUIEQEZFcKhAiIpJLBUJERHKpQIiISC4VCJFFMLNZmzuCbMNGAI6RQau6Z0NkgVVVJyDyn/nt7vurTkKkDOpBiDRAPCPgupkNxbQ74jvMrBaDz9XMbHvEt1h6FsNwTIfirZrM7G6M9//YzFoq+1Cy4qlAiCxOy7xDTJ2Z3/1w9wPAAHAzYgOkIZf3kgbF6494P/DU3feRnj0wFvF24Ja77wG+ke7UFamE7qQWWQQz++nu63Lin4Aj7j4eA6p9cfdWM5siDRcxE/EJd99kZpPANnefzrzHTuCJu7fHz73Aane/tvSfTGQh9SBEGscL5ouWyTOdmZ9F5wmlQioQIo3TmXl9EfPPSaMEA5wHnsV8DegBMLOmCp67IPJP2jsRWZwWiwfWh0fuXr/UtdnMXpJ2vM5F7DJwz8yuApPAxYhfAe6YWTepp9BDGkFUZNnQOQiRBohzEB3uPlV1LiKNokNMIiKSSz0IERHJpR6EiIjkUoEQEZFcKhAiIpJLBUJERHKpQIiISK4/2c1Hy9fCMQ4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "torch.manual_seed(1)\n",
    "\n",
    "LR = 0.1\n",
    "iteration = 10\n",
    "max_epoch = 200\n",
    "# ------------------------------ fake data and optimizer  ------------------------------\n",
    "\n",
    "weights = torch.randn((1), requires_grad=True)\n",
    "target = torch.zeros((1))\n",
    "\n",
    "optimizer = optim.SGD([weights], lr=LR, momentum=0.9)\n",
    "\n",
    "# ------------------------------ 4 Cosine Annealing LR ------------------------------\n",
    "t_max = 50\n",
    "scheduler_lr = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=t_max, eta_min=0.)\n",
    "\n",
    "lr_list, epoch_list = list(), list()\n",
    "for epoch in range(max_epoch):\n",
    "\n",
    "    lr_list.append(scheduler_lr.get_lr())\n",
    "    epoch_list.append(epoch)\n",
    "\n",
    "    for i in range(iteration):\n",
    "\n",
    "        loss = torch.pow((weights - target), 2)\n",
    "        loss.backward()\n",
    "\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "    scheduler_lr.step()\n",
    "\n",
    "plt.plot(epoch_list, lr_list, label=\"CosineAnnealingLR Scheduler\\nT_max:{}\".format(t_max))\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning rate\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q:如何监控指标， 当指标不再变化则调整学习率？\n",
    "- `torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, verbose=False, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)`\n",
    "- mode：min/max两种模式：如min代表不下降就调整\n",
    "- factor：调整系数\n",
    "- patience：“耐心”，接受几次不变化\n",
    "- cooldown：“冷却时间”，停止监控一段时间\n",
    "- verbose：是否打印日志\n",
    "- min_lr：学习率下限\n",
    "- eps：学习率衰减最小值\n",
    "\n",
    "Q:ReduceLROnPlateau代码示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    16: reducing learning rate of group 0 to 1.0000e-02.\n",
      "Epoch    37: reducing learning rate of group 0 to 1.0000e-03.\n",
      "Epoch    58: reducing learning rate of group 0 to 1.0000e-04.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "torch.manual_seed(1)\n",
    "\n",
    "LR = 0.1\n",
    "iteration = 10\n",
    "max_epoch = 200\n",
    "# ------------------------------ fake data and optimizer  ------------------------------\n",
    "\n",
    "weights = torch.randn((1), requires_grad=True)\n",
    "target = torch.zeros((1))\n",
    "\n",
    "optimizer = optim.SGD([weights], lr=LR, momentum=0.9)\n",
    "\n",
    "# ------------------------------ 5 Reduce LR On Plateau ------------------------------\n",
    "loss_value = 0.5\n",
    "accuray = 0.9\n",
    "\n",
    "factor = 0.1\n",
    "mode = \"min\"\n",
    "patience = 10\n",
    "cooldown = 10\n",
    "min_lr = 1e-4\n",
    "verbose = True\n",
    "\n",
    "scheduler_lr = optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=factor, mode=mode, patience=patience,\n",
    "                                                    cooldown=cooldown, min_lr=min_lr, verbose=verbose)\n",
    "\n",
    "for epoch in range(max_epoch):\n",
    "    for i in range(iteration):\n",
    "\n",
    "        # train(...)\n",
    "\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "    if epoch == 5:\n",
    "        loss_value = 0.4\n",
    "\n",
    "    scheduler_lr.step(loss_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q:如何自定义学习率调整策略？\n",
    "- `torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda, last_epoch=-1)`\n",
    "- lr_lambda：function or list，若为list，里面每个元素须为function\n",
    "\n",
    "Q:LambdaLR的代码示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:    0, lr:[0.1, 0.095]\n",
      "epoch:    1, lr:[0.1, 0.09025]\n",
      "epoch:    2, lr:[0.1, 0.0857375]\n",
      "epoch:    3, lr:[0.1, 0.081450625]\n",
      "epoch:    4, lr:[0.1, 0.07737809374999999]\n",
      "epoch:    5, lr:[0.1, 0.07350918906249998]\n",
      "epoch:    6, lr:[0.1, 0.06983372960937498]\n",
      "epoch:    7, lr:[0.1, 0.06634204312890622]\n",
      "epoch:    8, lr:[0.1, 0.0630249409724609]\n",
      "epoch:    9, lr:[0.1, 0.05987369392383787]\n",
      "epoch:   10, lr:[0.1, 0.05688000922764597]\n",
      "epoch:   11, lr:[0.1, 0.05403600876626367]\n",
      "epoch:   12, lr:[0.1, 0.051334208327950485]\n",
      "epoch:   13, lr:[0.1, 0.04876749791155296]\n",
      "epoch:   14, lr:[0.1, 0.046329123015975304]\n",
      "epoch:   15, lr:[0.1, 0.04401266686517654]\n",
      "epoch:   16, lr:[0.1, 0.04181203352191771]\n",
      "epoch:   17, lr:[0.1, 0.039721431845821824]\n",
      "epoch:   18, lr:[0.1, 0.03773536025353073]\n",
      "epoch:   19, lr:[0.010000000000000002, 0.03584859224085419]\n",
      "epoch:   20, lr:[0.010000000000000002, 0.03405616262881148]\n",
      "epoch:   21, lr:[0.010000000000000002, 0.0323533544973709]\n",
      "epoch:   22, lr:[0.010000000000000002, 0.03073568677250236]\n",
      "epoch:   23, lr:[0.010000000000000002, 0.02919890243387724]\n",
      "epoch:   24, lr:[0.010000000000000002, 0.027738957312183378]\n",
      "epoch:   25, lr:[0.010000000000000002, 0.026352009446574204]\n",
      "epoch:   26, lr:[0.010000000000000002, 0.025034408974245494]\n",
      "epoch:   27, lr:[0.010000000000000002, 0.023782688525533217]\n",
      "epoch:   28, lr:[0.010000000000000002, 0.022593554099256556]\n",
      "epoch:   29, lr:[0.010000000000000002, 0.02146387639429373]\n",
      "epoch:   30, lr:[0.010000000000000002, 0.02039068257457904]\n",
      "epoch:   31, lr:[0.010000000000000002, 0.019371148445850087]\n",
      "epoch:   32, lr:[0.010000000000000002, 0.018402591023557582]\n",
      "epoch:   33, lr:[0.010000000000000002, 0.017482461472379703]\n",
      "epoch:   34, lr:[0.010000000000000002, 0.016608338398760716]\n",
      "epoch:   35, lr:[0.010000000000000002, 0.01577792147882268]\n",
      "epoch:   36, lr:[0.010000000000000002, 0.014989025404881546]\n",
      "epoch:   37, lr:[0.010000000000000002, 0.014239574134637467]\n",
      "epoch:   38, lr:[0.010000000000000002, 0.013527595427905593]\n",
      "epoch:   39, lr:[0.0010000000000000002, 0.012851215656510312]\n",
      "epoch:   40, lr:[0.0010000000000000002, 0.012208654873684797]\n",
      "epoch:   41, lr:[0.0010000000000000002, 0.011598222130000557]\n",
      "epoch:   42, lr:[0.0010000000000000002, 0.011018311023500529]\n",
      "epoch:   43, lr:[0.0010000000000000002, 0.010467395472325502]\n",
      "epoch:   44, lr:[0.0010000000000000002, 0.009944025698709225]\n",
      "epoch:   45, lr:[0.0010000000000000002, 0.009446824413773765]\n",
      "epoch:   46, lr:[0.0010000000000000002, 0.008974483193085076]\n",
      "epoch:   47, lr:[0.0010000000000000002, 0.00852575903343082]\n",
      "epoch:   48, lr:[0.0010000000000000002, 0.00809947108175928]\n",
      "epoch:   49, lr:[0.0010000000000000002, 0.007694497527671315]\n",
      "epoch:   50, lr:[0.0010000000000000002, 0.007309772651287749]\n",
      "epoch:   51, lr:[0.0010000000000000002, 0.006944284018723362]\n",
      "epoch:   52, lr:[0.0010000000000000002, 0.0065970698177871935]\n",
      "epoch:   53, lr:[0.0010000000000000002, 0.006267216326897833]\n",
      "epoch:   54, lr:[0.0010000000000000002, 0.005953855510552941]\n",
      "epoch:   55, lr:[0.0010000000000000002, 0.005656162735025293]\n",
      "epoch:   56, lr:[0.0010000000000000002, 0.005373354598274029]\n",
      "epoch:   57, lr:[0.0010000000000000002, 0.005104686868360327]\n",
      "epoch:   58, lr:[0.0010000000000000002, 0.004849452524942311]\n",
      "epoch:   59, lr:[0.00010000000000000003, 0.004606979898695194]\n",
      "epoch:   60, lr:[0.00010000000000000003, 0.004376630903760435]\n",
      "epoch:   61, lr:[0.00010000000000000003, 0.004157799358572413]\n",
      "epoch:   62, lr:[0.00010000000000000003, 0.003949909390643792]\n",
      "epoch:   63, lr:[0.00010000000000000003, 0.003752413921111602]\n",
      "epoch:   64, lr:[0.00010000000000000003, 0.003564793225056022]\n",
      "epoch:   65, lr:[0.00010000000000000003, 0.0033865535638032207]\n",
      "epoch:   66, lr:[0.00010000000000000003, 0.0032172258856130592]\n",
      "epoch:   67, lr:[0.00010000000000000003, 0.0030563645913324064]\n",
      "epoch:   68, lr:[0.00010000000000000003, 0.002903546361765786]\n",
      "epoch:   69, lr:[0.00010000000000000003, 0.0027583690436774966]\n",
      "epoch:   70, lr:[0.00010000000000000003, 0.0026204505914936217]\n",
      "epoch:   71, lr:[0.00010000000000000003, 0.0024894280619189406]\n",
      "epoch:   72, lr:[0.00010000000000000003, 0.0023649566588229936]\n",
      "epoch:   73, lr:[0.00010000000000000003, 0.0022467088258818434]\n",
      "epoch:   74, lr:[0.00010000000000000003, 0.002134373384587751]\n",
      "epoch:   75, lr:[0.00010000000000000003, 0.0020276547153583635]\n",
      "epoch:   76, lr:[0.00010000000000000003, 0.0019262719795904452]\n",
      "epoch:   77, lr:[0.00010000000000000003, 0.001829958380610923]\n",
      "epoch:   78, lr:[0.00010000000000000003, 0.0017384604615803768]\n",
      "epoch:   79, lr:[1.0000000000000003e-05, 0.001651537438501358]\n",
      "epoch:   80, lr:[1.0000000000000003e-05, 0.00156896056657629]\n",
      "epoch:   81, lr:[1.0000000000000003e-05, 0.0014905125382474755]\n",
      "epoch:   82, lr:[1.0000000000000003e-05, 0.0014159869113351015]\n",
      "epoch:   83, lr:[1.0000000000000003e-05, 0.0013451875657683465]\n",
      "epoch:   84, lr:[1.0000000000000003e-05, 0.001277928187479929]\n",
      "epoch:   85, lr:[1.0000000000000003e-05, 0.0012140317781059325]\n",
      "epoch:   86, lr:[1.0000000000000003e-05, 0.0011533301892006358]\n",
      "epoch:   87, lr:[1.0000000000000003e-05, 0.001095663679740604]\n",
      "epoch:   88, lr:[1.0000000000000003e-05, 0.0010408804957535737]\n",
      "epoch:   89, lr:[1.0000000000000003e-05, 0.000988836470965895]\n",
      "epoch:   90, lr:[1.0000000000000003e-05, 0.0009393946474176001]\n",
      "epoch:   91, lr:[1.0000000000000003e-05, 0.0008924249150467202]\n",
      "epoch:   92, lr:[1.0000000000000003e-05, 0.0008478036692943841]\n",
      "epoch:   93, lr:[1.0000000000000003e-05, 0.0008054134858296649]\n",
      "epoch:   94, lr:[1.0000000000000003e-05, 0.0007651428115381816]\n",
      "epoch:   95, lr:[1.0000000000000003e-05, 0.0007268856709612725]\n",
      "epoch:   96, lr:[1.0000000000000003e-05, 0.0006905413874132089]\n",
      "epoch:   97, lr:[1.0000000000000003e-05, 0.0006560143180425484]\n",
      "epoch:   98, lr:[1.0000000000000003e-05, 0.0006232136021404209]\n",
      "epoch:   99, lr:[1.0000000000000004e-06, 0.0005920529220333997]\n",
      "epoch:  100, lr:[1.0000000000000004e-06, 0.0005624502759317298]\n",
      "epoch:  101, lr:[1.0000000000000004e-06, 0.0005343277621351433]\n",
      "epoch:  102, lr:[1.0000000000000004e-06, 0.0005076113740283861]\n",
      "epoch:  103, lr:[1.0000000000000004e-06, 0.00048223080532696673]\n",
      "epoch:  104, lr:[1.0000000000000004e-06, 0.0004581192650606184]\n",
      "epoch:  105, lr:[1.0000000000000004e-06, 0.00043521330180758743]\n",
      "epoch:  106, lr:[1.0000000000000004e-06, 0.00041345263671720806]\n",
      "epoch:  107, lr:[1.0000000000000004e-06, 0.0003927800048813476]\n",
      "epoch:  108, lr:[1.0000000000000004e-06, 0.00037314100463728026]\n",
      "epoch:  109, lr:[1.0000000000000004e-06, 0.00035448395440541624]\n",
      "epoch:  110, lr:[1.0000000000000004e-06, 0.0003367597566851454]\n",
      "epoch:  111, lr:[1.0000000000000004e-06, 0.0003199217688508881]\n",
      "epoch:  112, lr:[1.0000000000000004e-06, 0.0003039256804083437]\n",
      "epoch:  113, lr:[1.0000000000000004e-06, 0.0002887293963879265]\n",
      "epoch:  114, lr:[1.0000000000000004e-06, 0.00027429292656853016]\n",
      "epoch:  115, lr:[1.0000000000000004e-06, 0.00026057828024010366]\n",
      "epoch:  116, lr:[1.0000000000000004e-06, 0.0002475493662280985]\n",
      "epoch:  117, lr:[1.0000000000000004e-06, 0.00023517189791669353]\n",
      "epoch:  118, lr:[1.0000000000000004e-06, 0.0002234133030208588]\n",
      "epoch:  119, lr:[1.0000000000000005e-07, 0.00021224263786981585]\n",
      "epoch:  120, lr:[1.0000000000000005e-07, 0.00020163050597632508]\n",
      "epoch:  121, lr:[1.0000000000000005e-07, 0.0001915489806775088]\n",
      "epoch:  122, lr:[1.0000000000000005e-07, 0.00018197153164363337]\n",
      "epoch:  123, lr:[1.0000000000000005e-07, 0.00017287295506145168]\n",
      "epoch:  124, lr:[1.0000000000000005e-07, 0.00016422930730837908]\n",
      "epoch:  125, lr:[1.0000000000000005e-07, 0.00015601784194296014]\n",
      "epoch:  126, lr:[1.0000000000000005e-07, 0.00014821694984581212]\n",
      "epoch:  127, lr:[1.0000000000000005e-07, 0.0001408061023535215]\n",
      "epoch:  128, lr:[1.0000000000000005e-07, 0.00013376579723584542]\n",
      "epoch:  129, lr:[1.0000000000000005e-07, 0.00012707750737405313]\n",
      "epoch:  130, lr:[1.0000000000000005e-07, 0.00012072363200535048]\n",
      "epoch:  131, lr:[1.0000000000000005e-07, 0.00011468745040508295]\n",
      "epoch:  132, lr:[1.0000000000000005e-07, 0.0001089530778848288]\n",
      "epoch:  133, lr:[1.0000000000000005e-07, 0.00010350542399058736]\n",
      "epoch:  134, lr:[1.0000000000000005e-07, 9.833015279105799e-05]\n",
      "epoch:  135, lr:[1.0000000000000005e-07, 9.341364515150508e-05]\n",
      "epoch:  136, lr:[1.0000000000000005e-07, 8.874296289392982e-05]\n",
      "epoch:  137, lr:[1.0000000000000005e-07, 8.430581474923332e-05]\n",
      "epoch:  138, lr:[1.0000000000000005e-07, 8.009052401177165e-05]\n",
      "epoch:  139, lr:[1.0000000000000004e-08, 7.608599781118307e-05]\n",
      "epoch:  140, lr:[1.0000000000000004e-08, 7.228169792062392e-05]\n",
      "epoch:  141, lr:[1.0000000000000004e-08, 6.866761302459272e-05]\n",
      "epoch:  142, lr:[1.0000000000000004e-08, 6.523423237336307e-05]\n",
      "epoch:  143, lr:[1.0000000000000004e-08, 6.197252075469492e-05]\n",
      "epoch:  144, lr:[1.0000000000000004e-08, 5.8873894716960165e-05]\n",
      "epoch:  145, lr:[1.0000000000000004e-08, 5.593019998111216e-05]\n",
      "epoch:  146, lr:[1.0000000000000004e-08, 5.313368998205655e-05]\n",
      "epoch:  147, lr:[1.0000000000000004e-08, 5.0477005482953716e-05]\n",
      "epoch:  148, lr:[1.0000000000000004e-08, 4.795315520880603e-05]\n",
      "epoch:  149, lr:[1.0000000000000004e-08, 4.555549744836572e-05]\n",
      "epoch:  150, lr:[1.0000000000000004e-08, 4.327772257594744e-05]\n",
      "epoch:  151, lr:[1.0000000000000004e-08, 4.1113836447150066e-05]\n",
      "epoch:  152, lr:[1.0000000000000004e-08, 3.905814462479256e-05]\n",
      "epoch:  153, lr:[1.0000000000000004e-08, 3.710523739355293e-05]\n",
      "epoch:  154, lr:[1.0000000000000004e-08, 3.524997552387528e-05]\n",
      "epoch:  155, lr:[1.0000000000000004e-08, 3.3487476747681514e-05]\n",
      "epoch:  156, lr:[1.0000000000000004e-08, 3.181310291029744e-05]\n",
      "epoch:  157, lr:[1.0000000000000004e-08, 3.0222447764782564e-05]\n",
      "epoch:  158, lr:[1.0000000000000004e-08, 2.8711325376543437e-05]\n",
      "epoch:  159, lr:[1.0000000000000005e-09, 2.7275759107716264e-05]\n",
      "epoch:  160, lr:[1.0000000000000005e-09, 2.5911971152330445e-05]\n",
      "epoch:  161, lr:[1.0000000000000005e-09, 2.4616372594713925e-05]\n",
      "epoch:  162, lr:[1.0000000000000005e-09, 2.3385553964978226e-05]\n",
      "epoch:  163, lr:[1.0000000000000005e-09, 2.2216276266729317e-05]\n",
      "epoch:  164, lr:[1.0000000000000005e-09, 2.110546245339285e-05]\n",
      "epoch:  165, lr:[1.0000000000000005e-09, 2.0050189330723204e-05]\n",
      "epoch:  166, lr:[1.0000000000000005e-09, 1.9047679864187045e-05]\n",
      "epoch:  167, lr:[1.0000000000000005e-09, 1.809529587097769e-05]\n",
      "epoch:  168, lr:[1.0000000000000005e-09, 1.7190531077428805e-05]\n",
      "epoch:  169, lr:[1.0000000000000005e-09, 1.6331004523557364e-05]\n",
      "epoch:  170, lr:[1.0000000000000005e-09, 1.5514454297379498e-05]\n",
      "epoch:  171, lr:[1.0000000000000005e-09, 1.4738731582510519e-05]\n",
      "epoch:  172, lr:[1.0000000000000005e-09, 1.4001795003384993e-05]\n",
      "epoch:  173, lr:[1.0000000000000005e-09, 1.3301705253215743e-05]\n",
      "epoch:  174, lr:[1.0000000000000005e-09, 1.2636619990554954e-05]\n",
      "epoch:  175, lr:[1.0000000000000005e-09, 1.2004788991027206e-05]\n",
      "epoch:  176, lr:[1.0000000000000005e-09, 1.1404549541475845e-05]\n",
      "epoch:  177, lr:[1.0000000000000005e-09, 1.0834322064402054e-05]\n",
      "epoch:  178, lr:[1.0000000000000005e-09, 1.029260596118195e-05]\n",
      "epoch:  179, lr:[1.0000000000000006e-10, 9.777975663122852e-06]\n",
      "epoch:  180, lr:[1.0000000000000006e-10, 9.28907687996671e-06]\n",
      "epoch:  181, lr:[1.0000000000000006e-10, 8.824623035968373e-06]\n",
      "epoch:  182, lr:[1.0000000000000006e-10, 8.383391884169954e-06]\n",
      "epoch:  183, lr:[1.0000000000000006e-10, 7.964222289961456e-06]\n",
      "epoch:  184, lr:[1.0000000000000006e-10, 7.566011175463383e-06]\n",
      "epoch:  185, lr:[1.0000000000000006e-10, 7.187710616690214e-06]\n",
      "epoch:  186, lr:[1.0000000000000006e-10, 6.828325085855702e-06]\n",
      "epoch:  187, lr:[1.0000000000000006e-10, 6.486908831562916e-06]\n",
      "epoch:  188, lr:[1.0000000000000006e-10, 6.16256338998477e-06]\n",
      "epoch:  189, lr:[1.0000000000000006e-10, 5.854435220485532e-06]\n",
      "epoch:  190, lr:[1.0000000000000006e-10, 5.5617134594612554e-06]\n",
      "epoch:  191, lr:[1.0000000000000006e-10, 5.283627786488193e-06]\n",
      "epoch:  192, lr:[1.0000000000000006e-10, 5.0194463971637825e-06]\n",
      "epoch:  193, lr:[1.0000000000000006e-10, 4.768474077305593e-06]\n",
      "epoch:  194, lr:[1.0000000000000006e-10, 4.5300503734403135e-06]\n",
      "epoch:  195, lr:[1.0000000000000006e-10, 4.3035478547682975e-06]\n",
      "epoch:  196, lr:[1.0000000000000006e-10, 4.088370462029883e-06]\n",
      "epoch:  197, lr:[1.0000000000000006e-10, 3.883951938928388e-06]\n",
      "epoch:  198, lr:[1.0000000000000006e-10, 3.6897543419819688e-06]\n",
      "epoch:  199, lr:[1.0000000000000006e-11, 3.5052666248828703e-06]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5xVdb3/8ddnzwwzwHCR4SY3ZxQUEVBoQDPFFDXwRhamZqlpWhmn/Fmm1UnN0+VYnqyOnorStNLQLBNveQM1SxG8gYjoiCgjCMNFYJDr8Pn9sdYM283eM3tm9tqXmffz8ViPvfa6fvZisz/z/X7X+n7N3REREUkUy3UAIiKSn5QgREQkKSUIERFJSglCRESSUoIQEZGklCBERCQpJQiRDDIzN7PhaW5bGW5fHHVcIm2hBCGdhpktN7Pjcx1HW5jZx82sNsW6W81sh5nVm9l6M3vUzEZmO0bpeJQgRDqGn7h7OTAYeBe4OcfxSAegBCGdmpntY2b3m1mdmW0I54fErX/CzH5gZv8O/0K/z8wqzOx2M9tkZvPNrDLhsCeZ2TIzW2tmPzWzWHisIjO7Ply+DDg5IZYvmNkSM9sc7v+l1n4ed98K3AUc1tp9RRIpQUhnFwN+D+wHDAO2AjcmbHMW8HmCv84PAJ4J9+kDLAGuTtj+dKAaGA9MAy4Il18EnAKMC9dPT9hvTbi+J/AF4AYzG9+aD2Nm3YGzgZrW7CeSjBKEdGruvs7d/+ruH7j7ZuCHwDEJm/3e3d90943AQ8Cb7v6Yu+8C/kLwgx/vOndf7+7vAD8n+MEG+Azwc3df4e7rgR8nxPJAeB539yeBR4Cj0/wo3zSz94HNwFEECU2kXZQgpFMzs25m9hsze9vMNgFPAb3NrChus9Vx81uTvC9POOyKuPm3gUHh/KAk6+JjmWpmz4YNze8DJwF90/wo17t7b6AyjOmgNPcTSUkJQjq7bxD8mB7u7j2BSeFya8cxh8bNDwNWhvOrkqwLTmZWCvwVuB4YEP7YP9jaOMJSy9eBX5hZ19aHLrKHEoR0NiVmVtY4AfsQ/MX9vpn1Ye/2hLa4PGz8HkrwY31nuPwu4GtmNsTM9gGujNunC1AK1AG7zGwqcGLigeNjD6e9Eoi7P0qQlC7OwGeRTkwJQjqbBwkSQuPUG+gKrAWeBf6RgXPcCzwPvAQ8wJ5bTn8LPAy8DLwA/K1xh7D942sESWQD8FlgdsJxByfEvpWg0TyZnwLfCksmIm1iGjBIRESSUQlCRESSUoIQEZGklCBERCQpJQgREUmqw3Qz3LdvX6+srMx1GCIiBeX5559f6+79kq3rMAmisrKSBQsW5DoMEZGCYmZvp1qnKiYREUlKCUJERJJSghARkaQ6TBuEiHRMO3fupLa2lm3btuU6lIJWVlbGkCFDKCkpSXsfJQgRyWu1tbX06NGDyspKkvRNKGlwd9atW0dtbS1VVVVp7xdpFZOZTTGzpWZWY2ZXJlk/ycxeMLNdZjY9Yd15ZvZGOJ0XZZwikr+2bdtGRUWFkkM7mBkVFRWtLoVFliDCAVduAqYCo4CzzWxUwmbvAOcDdyTs29jt8uHARODqsHtkEemElBzary3XMMoqpolAjbsvAzCzWQTj877auIG7Lw/X7U7Y9xPAo+GwjJjZo8AU4M+ZDvK9jdu4Y17K24AjMbGqgqNGpDtQmIhIbkSZIAbz4eEVawlKBG3dd3DiRmZ2MeGgKMOGDUtcnZbVm7bxv3OzN767O4wcuJp/XDqp5Y1FJC+Ul5dTX1/f7uNcc801lJeX881vfrPZ7c4//3xOOeUUpk+f3ux2jZ566ikuvfRSFi5cyKxZs9LeryVRJohk5Zl0B59Ia193nwnMBKiurm7TwBaHDu3NWz8+uS27tskltz/PG6vb/0UTEWk0bNgwbr31Vq6//vqMHjfKRupaPjz+7hD2jM0b5b55zcxo0CBNIgWpvr6eyZMnM378eMaMGcO9994LwPLlyxk5ciRf/OIXGT16NOeccw6PPfYYH/vYxxgxYgTPPfdc0zFefvlljjvuOEaMGMFvf/tbILjLaMaMGYwaNYqTTz6ZNWvWNG1/7bXXMmHCBEaPHs3FF19MskHeKisrGTt2LLFYZn/SoyxBzAdGmFkV8C5wFsEwiul4GPhRXMP0icC3Mx9i9sXMUH4QaZvv37eYV1duyugxRw3qydWnHpLWtmVlZdxzzz307NmTtWvXcsQRR3DaaacBUFNTw1/+8hdmzpzJhAkTuOOOO3j66aeZPXs2P/rRj/j73/8OwMKFC3n22WfZsmUL48aN4+STT+bZZ59l6dKlLFq0iNWrVzNq1CguuOACAGbMmMFVV10FwOc//3nuv/9+Tj311Ixeg1QiK0G4+y5gBsGP/RLgLndfbGbXmtlpAGY2wcxqgTOA35jZ4nDf9cB/ESSZ+cC1jQ3WhS5msFsZQqQguTvf+c53GDt2LMcffzzvvvsuq1evBqCqqooxY8YQi8U45JBDmDx5MmbGmDFjWL58edMxpk2bRteuXenbty/HHnsszz33HE899RRnn302RUVFDBo0iOOOO65p+7lz53L44YczZswY5syZw+LFi7P2eSN9UM7dHyQYJD5+2VVx8/MJqo+S7XsLcEuU8eVCzEwJQqSN0v1LPyq33347dXV1PP/885SUlFBZWdn0bEFpaWnTdrFYrOl9LBZj165dTesSbzdtfJ/sNtRt27ZxySWXsGDBAoYOHco111yT1SfK1RdTlsXM2J14U6+IFISNGzfSv39/SkpKmDt3Lm+/3fpb5O+99162bdvGunXreOKJJ5gwYQKTJk1i1qxZNDQ0sGrVKubOnQvQlAz69u1LfX09d999d0Y/T0vU1UaWqYpJpHCdc845nHrqqVRXV3PYYYcxcuTIVh9j4sSJnHzyybzzzjt873vfY9CgQZx++unMmTOHMWPGcOCBB3LMMccA0Lt3by666CLGjBlDZWUlEyZMSHrM+fPnc/rpp7Nhwwbuu+8+rr766oxURVmyFvFCVF1d7YUwYNAVdy/kidfXMO87x+c6FJGCsGTJEg4++OBch9EhJLuWZva8u1cn215VTGtr4J4vw5rXsnK6WAx2d4ycLCIdnBKEN8DLf4aVL2bldGaW9D5mEZF8owTRZ3+IlUBddkoQRWYqQYhIQVCCKCqBviOyliBiBg3KECJSAJQgAPqNhDVLsnIq03MQIlIglCAgSBDvvw07tkR+KnW1ISKFQgkCoH94L/Pa1yM/lZ6DECk85eXlGTnONddck1aPq+eff36rHor72c9+xqhRoxg7diyTJ09u0wN8yShBAPQL7wvOwq2uRTFVMYlIZo0bN44FCxawcOFCpk+fzre+9a2MHFcJAoI7mYq6QF307RCmrjZECla+dvd97LHH0q1bNwCOOOIIamtrM/J51dUGQFExVIzISglCVUwi7fDQlfDeoswec+AYmPrfaW1aCN1933zzzUydOrU9V6SJEkSjfgfBu89Hfhr15ipSuBq7+37qqaeIxWJJu/sG0uruu2vXrml39/2Tn/yEDz74gPXr13PIIYekTBB/+tOfWLBgAU8++WRGPq8SRKP+B8PivwV3MnXpHtlpghJEZIcX6djS/Es/Kvnc3fdjjz3GD3/4Q5588skPxdIeaoNo1C+8k6luaaSnicWCL4G62xApPPna3feLL77Il770JWbPnk3//v3b/gETqATRqH94J1PdazB4fGSniYV/JTTsdoqL9v6LQUTyV75293355ZdTX1/PGWecAcCwYcOYPXt22z9oSN19N2rYBT/aF474CpxwbeYCS3DjnDe4/pHXef0HU+lSrAKcSEvU3XfmqLvvtsrSnUyN9YxqqBaRfKcEEa//yMifhWisYlJ+EJF8pwQRr9/B8P47sL0+slMUhVdcJQiR9HWUqvBcass1VIKIl4U+mZoaqfWFF0lLWVkZ69atU5JoB3dn3bp1lJWVtWo/3cUUr6lPpiWR3cnU2Abh6m5DJC1DhgyhtraWurq6XIdS0MrKyhgyZEir9lGCiNenCoq7wupXIjtF+BiEqphE0lRSUkJVVVWuw+iUVMUUL1YEAw7JfF8v8afQXUwiUiCUIBINHB0kiIh+wBufpFZ3GyKS75QgEg0cA9veh42Z6S43kaqYRKRQKEEkGhD0xhhVO4SqmESkUChBJBowKnh9L6oEEbyqiklE8p0SRKLSHsEIc+8tjOTwTV1tKEOISJ5TgkhmwOjIqpiK1NWGiBSISBOEmU0xs6VmVmNmVyZZX2pmd4br55lZZbi8xMxuM7NFZrbEzL4dZZx7GTgW1i+D7ZszfuhYeMX1JLWI5LvIEoSZFQE3AVOBUcDZZjYqYbMLgQ3uPhy4AbguXH4GUOruY4CPAF9qTB5ZMXB08Lr61YwfWo3UIlIooixBTARq3H2Zu+8AZgHTEraZBtwWzt8NTLagkt6B7mZWDHQFdgCbIoz1wwaECSKCdoimrjaUIEQkz0WZIAYDK+Le14bLkm7j7ruAjUAFQbLYAqwC3gGud/f1iScws4vNbIGZLchoPy29hkBZ70jaIXQXk4gUiigTRLLxNBN/FlNtMxFoAAYBVcA3zGz/vTZ0n+nu1e5e3a9fv/bGGxeVBQ/MRXCra5GqmESkQESZIGqBoXHvhwArU20TVif1AtYDnwX+4e473X0N8C8g6ZB4kRk4BlYvht0NGT2sxY1JLSKSz6JMEPOBEWZWZWZdgLOAxFG0ZwPnhfPTgTkeVM6/Axxnge7AEUC0Y4EmGjAadm0N7mbKoMYqJhUgRCTfRZYgwjaFGcDDwBLgLndfbGbXmtlp4WY3AxVmVgNcBjTeCnsTUA68QpBofu/u0Ty5lsrAsMuNDDdU6y4mESkUkY4H4e4PAg8mLLsqbn4bwS2tifvVJ1ueVf1GQlEXWPkSjP50xg4baxpyNGOHFBGJhJ6kTqW4SzA2xMoXM3pYlSBEpFAoQTRn0HhY9TLsztz4oDH1xSQiBUIJojmDx8P2TbD+zYwdck8JImOHFBGJhBJEcwaNC17ffSFjh9SAQSJSKJQgmtP3ICjpltF2CFMbhIgUCCWI5hQVBz27rsxcCaIopu6+RaQwKEG0ZPB4WLUQGnZl5HCNVUx6klpE8p0SREsGjQueqK7LzIPcqmISkUKhBNGSQeOD1wy1Q6irDREpFEoQLemzP5T2zFg7hB6UE5FCoQTRklgMBh2WsRJEYyO1miBEJN8pQaRj0LhgbIhd29t9KFMjtYgUCCWIdAwaD7t3BuNDtFNMQ46KSIFQgkhH4xPVGWiHUFcbIlIolCDS0XsYdO8PtQvafSh1tSEihUIJIh1mMHQirJjX7kPFYrqLSUQKgxJEuoYeHgw/Wr+mXYfRba4iUiiUINI19PDgdcVz7TpMUxVT5oaYEBGJhBJEuvY9NBiCtJ3VTCpBiEihUIJIV0lZcDdTO0sQpq42RKRApJUgzOwoM/tCON/PzKqiDStPDZ0YPFHdjgfmitRILSIFosUEYWZXA1cA3w4XlQB/ijKovDX0cGjYHnT/3UaNVUwNShAikufSKUGcDpwGbAFw95VAjyiDyltDJgavK55t8yGs6TmIDMQjIhKhdBLEDg/6hXAAM+sebUh5rMcA2KeyXQ3V6mpDRApFOgniLjP7DdDbzC4CHgN+F21YeWzo4UFDdRt/4JvuYlIRQkTyXIsJwt2vB+4G/gocBFzl7r+MOrC8NfRwqF8N77/dpt2L1BeTiBSI4pY2MLPr3P0K4NEkyzqf+Afm9qls9e4WpmTdxSQi+S6dKqYTkiybmulACkb/g4MR5t7+d5t214NyIlIoUpYgzOwrwCXA/mYWf19nD+BfUQeWt2JFMOyjsPzptu2uu5hEpEA0V8V0B/AQ8GPgyrjlm919faRR5buqo+GNh2HTKui5b6t2VQlCRApFyiomd9/o7svd/Wx3fxvYSnCra7mZDUvn4GY2xcyWmlmNmV2ZZH2pmd0Zrp9nZpVx68aa2TNmttjMFplZWas/XVQqjwpe3259QWrPba6ZDEhEJPPSeZL6VDN7A3gLeBJYTlCyaGm/IuAmgvaKUcDZZjYqYbMLgQ3uPhy4Abgu3LeY4GntL7v7IcDHgZ3pfaQsGDgWSnvBW0+1eteYxqQWkQKRTiP1D4AjgNfdvQqYTHptEBOBGndf5u47gFnAtIRtpgG3hfN3A5PNzIATgYXu/jKAu69z94Y0zpkdsSLY78g2tUOoiklECkU6CWKnu68DYmYWc/e5wGFp7DcYWBH3vjZclnQbd98FbAQqgAMBN7OHzewFM/tWshOY2cVmtsDMFtTV1aURUgZVHgXr34RNK1u1m7raEJFCkU6CeN/MyoGngNvN7BfArjT2syTLEn8WU21TDBwFnBO+nm5mk/fa0H2mu1e7e3W/fv3SCCmDqo4OXltZijAzzNTVhojkv3QSxDTgA+D/Af8A3gROTWO/WmBo3PshQOKf203bhO0OvYD14fIn3X2tu38APAiMT+Oc2TNgNJT1guX/bPWuRWaqYhKRvJdOVxtb3H23u+9y99sIGp6npHHs+cAIM6sysy7AWcDshG1mA+eF89OBOWHHgA8DY82sW5g4jgFeTe8jZUmsCPY7Ct5qfYKImdGgIUdFJM+lTBBm1tPMvm1mN5rZiRaYASwDPtPSgcM2hRkEP/ZLgLvcfbGZXWtmp4Wb3QxUmFkNcBnh8xbuvgH4GUGSeQl4wd0faPvHjEjlUbDhLdhY26rdVMUkIoWguQfl/ghsAJ4BvghcDnQBprn7S+kc3N0fJKgeil92Vdz8NuCMFPv+iXwfmCi+HeLQs9LeLaYqJhEpAM0liP3dfQyAmf0OWAsMc/fNWYmsEPQ/BLruA8uebGWC0F1MIpL/mmuDaHowLXwG4S0lhwSxGOz/cXhzTqsejY7FVIIQkfzXXII41Mw2hdNmgkbjTWa22cw2ZSvAvDf8eKh/D1YvTnuXmJkGDBKRvJeyisndi7IZSME64Ljg9c3HYeDotHZRFZOIFIJ0noOQ5vQcFLRF1DyW9i5qpBaRQqAEkQnDj4N3noXt9WltbmYqQYhI3lOCyIQDJkPDjrS73SiKoTYIEcl7ShCZMOyjUNItaIdIg6qYRKQQNPccBADhHUyJv2YbgQXAN9x9WRSBFZSSsuCp6jTbIWKqYhKRAtBigiDo8mIlwRCkRtCn0kBgKXALwWA+Mvx4eOMRWL8M+uzf7KbqakNECkE6VUxT3P037r7Z3Te5+0zgJHe/E9gn4vgKxwFhb+Q1LVczqYpJRApBOglit5l9xsxi4RTfUZ9+5RpVHAC994M3Hm1x06KY0aArJyJ5Lp0EcQ7weWANsDqc/5yZdSXorVUgqDc6aCq89STs2NLipipBiEi+S2c8iGXufqq793X3fuF8jbtvdffWD8rckR10EuzaFvTN1IyYmdogRCTvpXMXUz/gIqAyfnt3vyC6sArUfkcGo8y99gAcnHrQvZjBbg0YJCJ5Lp27mO4F/gk8BjREG06BKyqBA6fA6/+Ahl1QlPzyqpFaRApBOgmim7tfEXkkHcVBJ8HCO2HFs8GzEUkoQYhIIUinkfp+Mzsp8kg6iuGToagLvPZgyk1iMfXmKiL5L50E8XWCJLFV40GkobQHVB0DSx9IOYiQShAiUgjSuYuph7vH3L2ru/cM3/fMRnAFa+RJsGE5rFmSdLV6cxWRQpAyQZjZyPB1fLIpeyEWoAOnBq9LH0i6OqauNkSkADTXSH0ZcDHwP0nWOXBcJBF1BD33hcHV8OpsmHT5XquLzGhQEUJE8lxzQ45eHL4em71wOpDRn4KHvwNra6Dv8A+tUhuEiBSCtMaDMLMjzeyzZnZu4xR1YAXvkNMBg8V/22uVaUxqESkALSYIM/sjcD1wFDAhnKojjqvw9RwUDCT0yt4JQl1tiEghSOdBuWpglOsXrfVGfwoe/CasfhUGjGpaHIvBbj2TLiJ5Lp0qplcIBgiS1hr1SbAYvPLXDy2OqZFaRApAOgmiL/CqmT1sZrMbp6gD6xDK+0HVpKAdIq4ApiomESkE6VQxXRN1EB3a6E/D7P+AVS/BoHFA2Jur8oOI5LlmE4SZFQHfc/fjsxRPxzPyFLj/sqCaqSlB6DZXEcl/zVYxuXsD8IGZ9cpSPB1Ptz4w/HhYdHdTy7S62hCRQpBOG8Q2YJGZ3Wxmv2yc0jm4mU0xs6VmVmNmVyZZX2pmd4br55lZZcL6YWZWb2bfTOd8eeuws2HzqqaR5opisFsZQkTyXDptEA+EU6uE1VM3AScAtcB8M5vt7q/GbXYhsMHdh5vZWcB1wJlx628AHmrtufPOgVOhWwW8+CcYcYKqmESkILSYINz9tjYeeyJQ4+7LAMxsFjANiE8Q09jTCH43cKOZmbu7mX0SWAZsaeP580dxFxjzGVhwM3ywXglCRApCOk9SjzCzu83sVTNb1jilcezBwIq497XhsqTbuPsuYCNQYWbdgSuA77cQ28VmtsDMFtTV1aURUg6NOwcadsCiuzFLOVSEiEjeSKcN4vfAr4BdwLHAH4A/prGfJVmW+LOYapvvAze4e31zJ3D3me5e7e7V/fr1SyOkHBo4BgaOhZf+pBKEiBSEdBJEV3d/HDB3f9vdryG9rr5rgaFx74cAK1NtY2bFQC9gPXA48BMzWw5cCnzHzGakcc78Nu7zsOplhu54kwYlCBHJc2ndxWRmMeANM5thZqcD/dPYbz4wwsyqzKwLcBaQ+AT2bOC8cH46MMcDR7t7pbtXAj8HfuTuN6bzgfLamOlQ1IXDNz7E7t25DkZEpHnpJIhLgW7A14CPAJ9jz496SmGbwgzgYWAJcJe7Lzaza83stHCzmwnaHGoIBija61bYDqVbHxh5Mh/Z+Aglu7flOhoRkWalcxfTfIDw5qIvtObg7v4g8GDCsqvi5rcBZ7RwjGtac868N+GLdF98D5N5Gjgp19GIiKSUzl1MHzWzVwlKAZjZoWb2f5FH1lHt9zHeK61iesNDupVJRPJaOlVMPwc+AawDcPeXgUlRBtWhmfFs309zMMugdkGuoxERSSmtIUfdfUXCIg130w4L+5zIZrrBczNzHYqISErpJIgVZnYk4GbWJewXaUnEcXVou4q7cx/HwOJ7oH5NrsMREUkqnQTxZeCrBE891wKHAZdEGVRHFzNjFp+A3Tvhhbb2ZCIiEq0WE4S7r3X3c9x9gLv3d/fPAedmIbYOywze8kGw/7Ew/2bYtSPXIYmI7CWtNogkLstoFJ1MkVnwJPWRM4JuwBfdleuQRET20tYEkawPJUlTLBb2xXTA5KCPpn/9Aj1aLSL5pq0JQjfwt4M1jkltBh+7FNa+DksfbHE/EZFsSpkgzGyzmW1KMm0GBmUxxg4nZoY3PiQ36pPQez/418/14JyI5JWUCcLde7h7zyRTD3dPZyQ6SSHWWIIAKCqGI/8DaufD2//OaVwiIvHaWsUk7VBkRkP8mNTjPgfd+sLTN+QuKBGRBEoQOWAWtPE3VTOVdIWPXgI1j0Lt8zmMTERkDyWIHIiFCSK+EMHEL0G3CpjzX7kJSkQkgRJEDsTCm4Q/NOxoaTkcdRksmwvLn85NYCIicZQgciAWayxBJNy1NOFCKB8Ic36oO5pEJOeUIHKgqYop8dm4kq4w6Zvwzr/hzTnZD0xEJI4SRA4krWJqNP5c6DUsaItQKUJEckgJIgf2NFInSQDFpfDxK2Hli7Do7ixHJiKyhxJEDlhTCSLFBoeeDfseCo9dDTu2ZC0uEZF4ShA5UBRLeA4iUSwGU/4bNr0L//7fLEYmIrKHEkQONFYxNaQsQgD7HQmHnA5P/xw21mYpMhGRPZQgciDWUhVTo+O/D74bHvt+5DGJiCRSgsiBvbraSGWf/YKO/BbdBW/9MwuRiYjsoQSRA0m72kjl6G/APpVw/6Wwc1ukcYmIxFOCyIGi8Konvc01UZducMoNsK4G/vk/0QYmIhJHCSIHLJ1G6ngHHAdjzwq6A1+zJMLIRET2UILIgVhTG0QrdvrED6G0B9z3dY1fLSJZoQSRA812tZFK974w5cewYh48c2M0gYmIxFGCyIFmu9poztgz4eBT4fFr4b1FEUQmIrJHpAnCzKaY2VIzqzGzK5OsLzWzO8P188ysMlx+gpk9b2aLwtfjoowz2/Z0993KHc3glF8EAwv99SLYuTXzwYmIhCJLEGZWBNwETAVGAWeb2aiEzS4ENrj7cOAG4Lpw+VrgVHcfA5wH/DGqOHOhTVVMjbpXwCdvgroleoBORCIVZQliIlDj7svcfQcwC5iWsM004LZw/m5gspmZu7/o7ivD5YuBMjMrjTDWrGpzFVOj4cfD4V+Geb+CpQ9lMDIRkT2iTBCDgRVx72vDZUm3cfddwEagImGbTwMvuvv2xBOY2cVmtsDMFtTV1WUs8Kg1lSDaczPS8d8Penz925dg/bKMxCUiEi/KBGFJliX+ydzsNmZ2CEG105eSncDdZ7p7tbtX9+vXr82BZpu1twQBUFIGn/lD0C5x57mw44MMRSciEogyQdQCQ+PeDwFWptrGzIqBXsD68P0Q4B7gXHd/M8I4s66oLc9BJLNPJXz6d7D6FXjgMo1AJyIZFWWCmA+MMLMqM+sCnAXMTthmNkEjNMB0YI67u5n1Bh4Avu3u/4owxpyIhVe9IRM/6CNOgGOugJf/DM/c1P7jiYiEIksQYZvCDOBhYAlwl7svNrNrzey0cLObgQozqwEuAxpvhZ0BDAe+Z2YvhVP/qGLNtoxUMcU75go4+DR45D9hyX2ZOaaIdHrFUR7c3R8EHkxYdlXc/DbgjCT7/QD4QZSx5VIs3e6+0z5gDD41E249JXg+4vwHYMhHMnNsEem09CR1DqQ9YFBrlHSFs2dBeT/485mw/q0MHlxEOiMliBxobKTendEMQZAczrkbdu+CP5wGG9/N7PFFpFNRgsiBpu6+o7jrqN9B8Lm/wQcbgiRRvybz5xCRTkEJIgcaq5giuyt18Hg45y+waSX84ZPwwfqITiQiHZkSRA7s6awvwucW9vsonHVHMBLd70+Cze9Fdy4R6ZCUIHIgkkbqZA44NihJvP8O3DIFNrwd8QlFpCNRgsiBdnfW1xr7H82GOXMAAA5pSURBVAPn3gtb18Pvp8Ka16I/p4h0CEoQORCL6i6mVIZOCJ6NaNgJN58Ib87NznlFpKApQeTAnhJEFk86cAxc9Dj0Ggx/+jQ8f2sWTy4ihUgJIgesPQMGtUfvYXDBw0HbxH1fhwe/Bbt2ZDcGESkYShA5kPGuNlqjrCecfScccQk89xu49SQ9UCciSSlB5EBRW8ekzlgAxTDlx3DGrbBmCfzmaHjj0RwFIyL5SgkiBxpvc23IWYYIHXI6XPwElA+A26fD/ZfBji25jUlE8kakvblKco1dbdwx7x3+/ebarJ23OBbjyx8/gMG9u+5Z2HcEXDQX5vxXMJ7Esifg9F/D0IlZi0tE8pMSRA4M7FXGyIE9eLOunjfr6rNyTgfqNm9nv4pufPHo/T+8sqQMPvFDOPAT8PdLglthqy+AyVdB195ZiU9E8o8SRA6Ulxbzj0snZfWc7s6B//kQa+ubuWupahJc8gzM/RHM+zW8dj+c+EMYM33PrVci0mmoDaKTMDMqupeyrn578xuW9ggasC+aAz32hb99EW4+AVY8l51ARSRvKEF0IhXlXVi3Jc3nHgaNC5LEtJuCvpxuPgHuOhdWvxptkCKSN5QgOpGK8jRKEPFiRTDuc/AfL8Ckb0HNHPjVR8NEsTi6QEUkLyhBdCJ9u3dpvg0ildJyOO67cOlCmHR5mCiOhDs/F1Q95eKBPxGJnBJEJ1JR3oX16VYxJdOtDxz3n2Gi+BYseyqoepr5cXjpz7CrFaUTEcl7ShCdSEV5KVt3NvDBjl3tO1C3PkGJ4rJX4aTrYedW+PuX4Wej4LFroO71jMQrIrmlBNGJVHTvAsC6tlQzJVNaDhMvgq/Og8//PXi47l+/gJsmwMxjYd5M2LIuM+cSkazTcxCdSN/yUgDW1m9naJ9umTuwWdBD7AHHwubVsOgvsHAWPHQ5PPxtqDoGDj4FDjoJegzM3HlFJFJKEJ1IRXmGSxDJ9BgAR84IpvdegUV3wZL74P7/F0xDJgSJ4oDjYOBYiKkQK5KvlCA6kYqwBLFuS5YakweODqbjvw91rwVPZr/2ADz+/WDqVhGULvb/OFQdDftU6YltkTyiBNGJNLZBtOlW1/Ywg/4HB9Oky2Hze7DsSVg2Nxj+dPHfgu269YWhhwdtGUMnBg/rlXRt/tgiEhkliE6krKSI8tLiaKuY0tFjIBx6ZjC5B6WLd56BFfNhxTxY+kCwXawY+o8KhksdEJZGBowO7qISkcgpQXQyQXcbefS8QnzpovqCYNmWtVAbJouVL8Ebj8BLt+/Zp+fgYPuKEVBxAFQMD7ot7zFIbRoiGaQE0clUdO+S+xJES7r3hYOmBlOj+jXw3iJY/UrQ+F23BN5+BnbGDXBU3BX6VEGvodB7KPQaEsz3Cud7DAy6DxGRtChBdDIV5aWsWP9BrsNovfL+MHxyMDVyh82rYF1NMK2tgfXLYGNtUPrY9v6HjxErDkbP694vOF73/lDeL3ztv2d5twoo6w3FXbL7GUXyTKQJwsymAL8AioDfuft/J6wvBf4AfARYB5zp7svDdd8GLgQagK+5+8NRxtpZ9C3vwhNL13DSL/6Z1fPuV9GNn55xKOWlGfzKmUHPQcFUlWR8je2bg2SxsRY2roD3V0D96qA0Ur86KIlsqYPdO5Mfv7grlPUKBk0q6xUkjbJeH17WpRy6dIeSbtClW/C+cb6k+55XVX1JAYosQZhZEXATcAJQC8w3s9nuHt9f9IXABncfbmZnAdcBZ5rZKOAs4BBgEPCYmR3o7g1RxdtZfGr8ENbW78hq/3ruziOvrmbrHS/wf+eMJ5atW1lj3WCfAyntN7JpmNckwcHWDUG7x5Y1QfL4YB1s2xiUQLZthK3ha/17sHbpnve04iIWdw2SRXHXoGRSXAZFXaC4NO61NFhXVBq8T7YuVhw3Fe393pIsS/m+CCwGWJBsLRY3Jb5vy3a6ZbnQRVmCmAjUuPsyADObBUwD4hPENOCacP5u4EYL/idPA2a5+3bgLTOrCY/3TITxdgoTKvswoTL7dwHdMe8dvnPPIkZdlf2CYFlJjP49yiiKpfOD1SecmlEM1n033dhKV99GV7ZS5tspYxtlvp2ubKPMtzW9Nq7rumMbpTt2UMJOSnwnXdhJCVsp8c3h/E66+A5K2NW0TQnBdkXszsSlyKrdGB5OwXwMB5zGf4fgtTHNetP75tc3LbdU+yVfviedJ1++93lbWh8ubzERtvy9S4w9+Taprex3NEd85TctHqO1okwQg4EVce9rgcNTbePuu8xsI1ARLn82Yd/BiScws4uBiwGGDRuWscAl8z57+DD6dO/CsrXZGYO7kTts2LKDuvrtEZSa9tlrybZwSkdrwjFvoNh3EmM3MW+gyBuI0bDXfIzgvYWv8cuSbW8Q/Hx78FOO7276OQ+WNf60B9sQ/uTHmvYJElf8+8R9g+PT9B5oem28Cpbwj7PX+oSrtWf7D6/f8zOb6jzJ1+91/BTxpT5+Cml86dIrZ7VwnF5D0zpKa0WZIJJ97sRPmWqbdPbF3WcCMwGqq6s1KEGemzJa/TCJFJIoW85qgfi0NgRYmWobMysGegHr09xXREQiFGWCmA+MMLMqM+tC0Og8O2Gb2cB54fx0YI67e7j8LDMrNbMqYATwXISxiohIgsiqmMI2hRnAwwS3ud7i7ovN7FpggbvPBm4G/hg2Qq8nSCKE291F0KC9C/iq7mASEcku8w4ynnB1dbUvWLAg12GIiBQUM3ve3auTrdPTOyIikpQShIiIJKUEISIiSSlBiIhIUh2mkdrM6oC323GIvsDaDIWTSYqrdRRX6+VrbIqrddoa137u3i/Zig6TINrLzBakasnPJcXVOoqr9fI1NsXVOlHEpSomERFJSglCRESSUoLYY2auA0hBcbWO4mq9fI1NcbVOxuNSG4SIiCSlEoSIiCSlBCEiIkl1+gRhZlPMbKmZ1ZjZlTmMY6iZzTWzJWa22My+Hi6/xszeNbOXwumkHMW33MwWhTEsCJf1MbNHzeyN8HXvIdaijemguOvykpltMrNLc3HNzOwWM1tjZq/ELUt6fSzwy/A7t9DMxmc5rp+a2Wvhue8xs97h8koz2xp33X4dVVzNxJby387Mvh1es6Vm9oksx3VnXEzLzeylcHnWrlkzvxHRfc/cvdNOBN2QvwnsD3QBXgZG5SiWfYHx4XwP4HVgFMGY3d/Mg2u1HOibsOwnwJXh/JXAdTn+t3wP2C8X1wyYBIwHXmnp+gAnAQ8RjJx4BDAvy3GdCBSH89fFxVUZv12OrlnSf7vw/8LLQClQFf6/LcpWXAnr/we4KtvXrJnfiMi+Z529BDERqHH3Ze6+A5gFTMtFIO6+yt1fCOc3A0tIMg53npkG3BbO3wZ8MoexTAbedPf2PE3fZu7+FMGYJvFSXZ9pwB888CzQ28z2zVZc7v6Iu+8K3z5LMGJj1qW4ZqlMA2a5+3Z3fwuoIfj/m9W4zMyAzwB/juLczWnmNyKy71lnTxCDgRVx72vJgx9lM6sExgHzwkUzwiLiLdmuxonjwCNm9ryZXRwuG+DuqyD48gL9cxQbBINNxf+nzYdrlur65NP37gKCvzIbVZnZi2b2pJkdnaOYkv3b5cs1OxpY7e5vxC3L+jVL+I2I7HvW2ROEJVmW0/t+zawc+CtwqbtvAn4FHAAcBqwiKN7mwsfcfTwwFfiqmU3KURx7sWBI29OAv4SL8uWapZIX3zsz+y7BiI23h4tWAcPcfRxwGXCHmfXMclip/u3y4poBZ/PhP0Syfs2S/Eak3DTJslZds86eIGqBoXHvhwArcxQLZlZC8A9/u7v/DcDdV7t7g7vvBn5LRMXqlrj7yvB1DXBPGMfqxiJr+LomF7ERJK0X3H11GGNeXDNSX5+cf+/M7DzgFOAcDyusw+qbdeH88wT1/AdmM65m/u3y4ZoVA58C7mxclu1rluw3ggi/Z509QcwHRphZVfhX6FnA7FwEEtZt3gwscfefxS2PrzM8HXglcd8sxNbdzHo0zhM0cr5CcK3OCzc7D7g327GFPvRXXT5cs1Cq6zMbODe8y+QIYGNjFUE2mNkU4ArgNHf/IG55PzMrCuf3B0YAy7IVV3jeVP92s4GzzKzUzKrC2J7LZmzA8cBr7l7buCCb1yzVbwRRfs+y0fqezxNBS//rBJn/uzmM4yiC4t9C4KVwOgn4I7AoXD4b2DcHse1PcAfJy8DixusEVACPA2+Er31yEFs3YB3QK25Z1q8ZQYJaBewk+MvtwlTXh6Dof1P4nVsEVGc5rhqCuunG79mvw20/Hf77vgy8AJyag2uW8t8O+G54zZYCU7MZV7j8VuDLCdtm7Zo18xsR2fdMXW2IiEhSnb2KSUREUlCCEBGRpJQgREQkKSUIERFJSglCRESSUoIQaQUza7AP9yCbsR6Aw55Bc/XMhsheinMdgEiB2eruh+U6CJFsUAlCJAPCMQKuM7Pnwml4uHw/M3s87HzucTMbFi4fYMFYDC+H05HhoYrM7Ldhf/+PmFnXnH0o6fSUIERap2tCFdOZces2uftE4Ebg5+GyGwm6XB5L0CneL8PlvwSedPdDCcYeWBwuHwHc5O6HAO8TPKkrkhN6klqkFcys3t3LkyxfDhzn7svCDtXec/cKM1tL0F3EznD5Knfva2Z1wBB33x53jErgUXcfEb6/Aihx9x9E/8lE9qYShEjmeIr5VNsksz1uvgG1E0oOKUGIZM6Zca/PhPP/JuglGOAc4Olw/nHgKwBmVpSDcRdEWqS/TkRap6uFA9aH/uHujbe6lprZPII/vM4Ol30NuMXMLgfqgC+Ey78OzDSzCwlKCl8h6EFUJG+oDUIkA8I2iGp3X5vrWEQyRVVMIiKSlEoQIiKSlEoQIiKSlBKEiIgkpQQhIiJJKUGIiEhSShAiIpLU/weofm1OtStv/QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "torch.manual_seed(1)\n",
    "\n",
    "LR = 0.1\n",
    "iteration = 10\n",
    "max_epoch = 200\n",
    "# ------------------------------ fake data and optimizer  ------------------------------\n",
    "\n",
    "weights = torch.randn((1), requires_grad=True)\n",
    "target = torch.zeros((1))\n",
    "\n",
    "optimizer = optim.SGD([weights], lr=LR, momentum=0.9)\n",
    "\n",
    "# ------------------------------ 6 lambda ------------------------------\n",
    "lr_init = 0.1\n",
    "\n",
    "weights_1 = torch.randn((6, 3, 5, 5))\n",
    "weights_2 = torch.ones((5, 5))\n",
    "\n",
    "optimizer = optim.SGD([\n",
    "    {'params': [weights_1]},\n",
    "    {'params': [weights_2]}], lr=lr_init)\n",
    "\n",
    "lambda1 = lambda epoch: 0.1 ** (epoch // 20)\n",
    "lambda2 = lambda epoch: 0.95 ** epoch\n",
    "\n",
    "scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])\n",
    "\n",
    "lr_list, epoch_list = list(), list()\n",
    "for epoch in range(max_epoch):\n",
    "    for i in range(iteration):\n",
    "\n",
    "        # train(...)\n",
    "\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "    scheduler.step()\n",
    "\n",
    "    lr_list.append(scheduler.get_lr())\n",
    "    epoch_list.append(epoch)\n",
    "\n",
    "    print('epoch:{:5d}, lr:{}'.format(epoch, scheduler.get_lr()))\n",
    "\n",
    "plt.plot(epoch_list, [i[0] for i in lr_list], label=\"lambda 1\")\n",
    "plt.plot(epoch_list, [i[1] for i in lr_list], label=\"lambda 2\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"LambdaLR\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.TensorBoard简介与安装\n",
    "\n",
    "Q:TensorBoard安装和启动命令？\n",
    "- 安装：`conda install tensorboard typing-extensions`\n",
    "- 启动：`tensorboard --logdir=./runs`\n",
    "\n",
    "Q:TensorBoard测试代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "\n",
    "\n",
    "writer = SummaryWriter(comment='test_tensorboard')\n",
    "\n",
    "for x in range(100):\n",
    "\n",
    "    writer.add_scalar('y=2x', x * 2, x)\n",
    "    writer.add_scalar('y=pow(2, x)',  2 ** x, x)\n",
    "    \n",
    "    writer.add_scalars('data/scalar_group', {\"xsinx\": x * np.sin(x),\n",
    "                                             \"xcosx\": x * np.cos(x),\n",
    "                                             \"arctanx\": np.arctan(x)}, x)\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.TensorBoard使用（一）\n",
    "\n",
    "Q:SummaryWriter的功能和属性是什么？\n",
    "- 功能：提供创建event file的高级接口\n",
    "- ```python\n",
    "class SummaryWriter(object):\n",
    "    def __init__(self, log_dir=None, comment='', purge_step=None, max_queue=10, flush_secs=120, filename_suffix=’’)\n",
    "```\n",
    "- log_dir：event file输出文件夹\n",
    "- comment：不指定log_dir时，文件夹后缀\n",
    "- filename_suffix：event file文件名后缀\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}