{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 3.14 Forward Propagation, Back Propagation and Computational Graphs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- In the previous secitons...\n", " - we calculated the forward propagation of the model\n", " - we calculated the model output for the input\n", " - Then, execute the auto-generated `backward` function to calculate the gradient\n", " - based on back-propagation\n", "- In this section...\n", " - we will use both mathematical and computational graphs to describe ***forward and back propagation***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.14.1 Forward Propagation\n", "- Forward Propagation\n", " - It refers to ***the calculation and storage of intermediate variables*** (including outputs) for the neural network in the order from input layer to output layer.\n", " - Let’s assume that the input example is $\\mathbf{x}\\in \\mathbb{R}^d$ and there is no bias term. \n", " - Here the intermediate variable is \n", "\n", " $$\\mathbf{z}= \\mathbf{W}^{(1)} \\mathbf{x}$$\n", " where $\\mathbf{W}^{(1)} \\in \\mathbb{R}^{h \\times d}$ is the weight parameter of the hidden layer.\n", " - After entering the intermediate variable $\\mathbf{z}\\in \\mathbb{R}^h$ into the activation function $\\phi$ operated by the basic elements \n", "\n", "$$\\mathbf{h}= \\phi (\\mathbf{z}).$$\n", " - The hidden variable $\\mathbf{h}$ is also an intermediate variable. \n", " - Assuming the parameters of the output layer only possess a weight of $\\mathbf{W}^{(2)} \\in \\mathbb{R}^{q \\times h}$, we can obtain an output layer variable with a vector length of $q$, \n", " \n", " $$\\mathbf{o}= \\mathbf{W}^{(2)} \\mathbf{h}.$$
\n", " \n", " - Assuming the loss function is $l$ and the example label is $y$, we can then calculate the loss term for a single data example, \n", " \n", " $$L = l(\\mathbf{o}, y).$$
\n", " - According to the definition of $\\ell_2$ norm regularization, given the hyper-parameter $\\lambda$, the regularization term is \n", " \n", " $$s = \\frac{\\lambda}{2} \\left(|\\mathbf{W}^{(1)}|_F^2 + |\\mathbf{W}^{(2)}|_F^2\\right),$$
\n", " where the Frobenius norm of the matrix is equivalent to the calculation of the $L_2$ norm after flattening the matrix to a vector. \n", " - https://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm\n", " - The model's regularized loss on a given data example \n", " \n", " $$J = L + s.$$\n", " - We refer to $J$ as the ***objective function*** of a given data exampl" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.14.2 Computational Graph of Forward Propagation\n", "- Plotting computational graphs helps us visualize the dependencies of operators and variables within the calculation.\n", "![](https://github.com/d2l-ai/d2l-en/raw/master/img/forward.svg?sanitize=true![image.png](attachment:image.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.14.3 Back Propagation\n", "- Back propagation refers to the method of ***calculating the gradient of neural network parameters*** in the order of the output layer to the input layer according to the ‘chain rule ’in calculus.\n", "- Chain rule\n", " - Assume that we have functions $\\mathsf{Y}=f(\\mathsf{X})$ and $\\mathsf{Z}=g(\\mathsf{Y}) = g \\circ f(\\mathsf{X})$, in which the input and the output $\\mathsf{X}, \\mathsf{Y}, \\mathsf{Z}$ are tensors of arbitrary shapes.\n", " - By using the chain rule we can compute the derivative of $\\mathsf{Z}$ wrt. $\\mathsf{X}$ via\n", "\n", "$$\\frac{\\partial \\mathsf{Z}}{\\partial \\mathsf{X}} = \\text{prod}\\left(\\frac{\\partial \\mathsf{Z}}{\\partial \\mathsf{Y}}, \\frac{\\partial \\mathsf{Y}}{\\partial \\mathsf{X}}\\right).$$\n", "\n", "- The parameters of the simple network with one hidden layer are $\\mathbf{W}^{(1)}$ and $\\mathbf{W}^{(2)}$ ==> The objective of back propagation is to calculate the gradients $\\partial J/\\partial \\mathbf{W}^{(1)}$ and $\\partial J/\\partial \\mathbf{W}^{(2)}$.\n", "- The first step is to calculate the gradients of the objective function $J=L+s$ with respect to the loss term $L$ and the regularization term $s$.\n", "\n", "$$\\frac{\\partial J}{\\partial L} = 1 \\text{ and } \\frac{\\partial J}{\\partial s} = 1$$\n", "- Next we compute the gradient of the objective function with respect to variable of the output layer $\\mathbf{o}$ according to the chain rule.\n", "\n", "$$ \\frac{\\partial J}{\\partial \\mathbf{o}} = \\text{prod}\\left(\\frac{\\partial J}{\\partial L}, \\frac{\\partial L}{\\partial \\mathbf{o}}\\right) = \\frac{\\partial L}{\\partial \\mathbf{o}} \\in \\mathbb{R}^q $$
\n", "\n", "- Next we calculate the gradients of the regularization term with respect to both parameters.\n", "\n", "$$\\frac{\\partial s}{\\partial \\mathbf{W}^{(1)}} = \\lambda \\mathbf{W}^{(1)}, \\frac{\\partial s}{\\partial \\mathbf{W}^{(2)}} = \\lambda \\mathbf{W}^{(2)} \\text{ and } \\frac{\\partial \\mathbf{o}}{\\partial \\mathbf{W}^{(2)}} = \\mathbf{h}^\\top $$
\n", "\n", "- Now we are able calculate the gradient $\\partial J/\\partial \\mathbf{W}^{(2)} \\in \\mathbb{R}^{q \\times h}$ of the model parameters closest to the output layer. Using the chain rule yields:\n", "\n", "$$ \\frac{\\partial J}{\\partial \\mathbf{W}^{(2)}} = \\text{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{o}}, \\frac{\\partial \\mathbf{o}}{\\partial \\mathbf{W}^{(2)}}\\right) + \\text{prod}\\left(\\frac{\\partial J}{\\partial s}, \\frac{\\partial s}{\\partial \\mathbf{W}^{(2)}}\\right) = \\frac{\\partial L}{\\partial \\mathbf{o}} \\mathbf{h}^\\top + \\lambda \\mathbf{W}^{(2)} $$
\n", "\n", "- and so forth." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.14.4 Training a Model\n", "- When training networks, forward and backward propagation depend on each other. \n", " - For forward propagation we traverse the compute graph in the direction of dependencies and compute all the variables on its path. \n", " - For backpropagation, the variables are used to compute gradients on the graph in reverse order. \n", "- Why backpropagation requires significantly more memory?\n", " - 1) We need to retain the intermediate values until backpropagation is complete. \n", " - 2) we typically train with minibatches containing more than one variable" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.15 Numerical Stability and Initialization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.15.1 Vanishing and Exploding Gradients\n", "- Consider a deep network with $d$ layers, input $\\mathbf{x}$ and output $\\mathbf{o}$. Each layer satisfies:\n", "\n", "$$\\mathbf{h}^{t+1} = f_t (\\mathbf{h}^t) \\text{ and thus } \\mathbf{o} = f_d \\circ \\ldots \\circ f_1(\\mathbf{x})$$\n", "- The gradient of $\\mathbf{o}$ with respect to any set of parameters $\\mathbf{W}_t$ associated with the function $f_t$ at layer $t$ simply as\n", "\n", "$$\\partial_{\\mathbf{W}t} \\mathbf{o} = \\underbrace{\\partial_{\\mathbf{h}^{d-1}} \\mathbf{h}^d}_{:= \\mathbf{M}_d} \\cdot \\ldots \\cdot \\underbrace{\\partial_{\\mathbf{h}^{t+1}} \\mathbf{h}^{t+2}}_{:= \\mathbf{M}_{t+1}} \\cdot \\underbrace{\\partial_{\\mathbf{h}^{t}} \\mathbf{h}^{t+1}}_{:= \\mathbf{M}_t} \\cdot \\underbrace{\\partial_{\\mathbf{W}t} \\mathbf{h}^t}_{:= \\mathbf{v}_t}.$$\n", "\n", "- The matrices $M_t$ may well have a wide variety of eigenvalues. \n", " - They might be small, they might be large, and in particular, their product might well be very large or very small.\n", " - http://setosa.io/ev/eigenvectors-and-eigenvalues/\n", "- This means that the optimization algorithm is bound to fail. \n", " - It either receives gradients with excessively large or excessively small steps. \n", " - In the former case, the parameters explode\n", " - In the latter case, we end up with vanishing gradients and no meanigful progress." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Exploding Gradients\n", " - we draw 100 Gaussain random matrices and multiply them with some initial matrix." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "A single matrix \n", "[[-5.5410951e-01 -2.5639790e-01 -7.8186011e-01 -1.5031214e+00]\n", " [-1.0607985e+00 -2.5132334e+00 6.5148938e-01 -6.0702908e-01]\n", " [ 5.2347612e-01 -2.0822671e-03 6.1475140e-01 1.4645545e+00]\n", " [ 5.7122988e-01 1.1327640e+00 -5.2853018e-01 -2.0936570e-01]]\n", "\n", "\n", "[-1.705154 +0.j -0.38180867+0.74472266j -0.38180867-0.74472266j\n", " -0.19318593+0.j ]\n", "After multiplying 100 matrices \n", "[[ 4.1391511e+23 3.2834950e+22 -7.0896819e+21 -1.4974215e+22]\n", " [ 1.6037556e+24 1.2722228e+23 -2.7469440e+22 -5.8018901e+22]\n", " [-1.9988672e+23 -1.5856566e+22 3.4237518e+21 7.2313263e+21]\n", " [-3.6506759e+23 -2.8959967e+22 6.2529075e+21 1.3206971e+22]]\n", "\n" ] } ], "source": [ "%matplotlib inline\n", "import mxnet as mx\n", "from mxnet import nd, autograd\n", "from matplotlib import pyplot as plt\n", "from numpy import linalg as LA\n", "\n", "M = nd.random.normal(shape=(4,4))\n", "print('A single matrix', M)\n", "print()\n", "\n", "w, v = LA.eig(M.asnumpy())\n", "print(w)\n", "\n", "for i in range(100):\n", " M = nd.dot(M, nd.random.normal(shape=(4,4)))\n", "\n", "print('After multiplying 100 matrices', M)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Vanishing Gradients\n", " - Picking 'sigmoid' as nonlinear activation function might be problematic.\n", " - If the activations are not in the range of $[-4, 4]$, the gradients of the overall product may vanish. \n", " - Before ReLu $\\max(0,x)$ was proposed, this problem used to be the bane of deep network training. \n", " - As a consequence ReLu has become the default choice when designing activation functions in deep networks." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAD8CAYAAABNa2y4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VOXdxvHvbyY7WYAQEMIui6CIQlgExB1XsPat26ut1gVtq7W12qqt1qVvW5dW27oi7gvWqlUUFNwXlCXIIjvIGoIQCATIPjPP+8eJGDDAQJYzSe7Pdc01Z3ky8zsJwz3nnOc8x5xziIiISOwI+F2AiIiI7E7hLCIiEmMUziIiIjFG4SwiIhJjFM4iIiIxRuEsIiISYxTOIiIiMUbhLCIiEmMUziIiIjEmzq83btOmjevatatfby8iItKgZs+evdk5lxVNW9/CuWvXruTm5vr19iIiIg3KzNZE21aHtUVERGKMwllERCTGKJxFRERijG/nnGtSWVlJXl4eZWVlfpfSaCQlJdGxY0fi4+P9LkVEROpITIVzXl4eaWlpdO3aFTPzu5yY55xjy5Yt5OXl0a1bN7/LERGROrLfw9pm9qSZbTKzBXtZb2b2TzNbYWbzzWzAwRZTVlZGZmamgjlKZkZmZqaONIiINDHRnHN+GjhtH+tPB3pWPcYCj9SmIAXzgdHvS0Sk6dnvYW3n3Cdm1nUfTc4GnnXOOWC6mbU0s/bOuQ11VKOIiMQ45xzhiCNU9QiHHaFIpOb5sNe2MhLxfqbaOucczkHEQcR5899ORxzV1ruo23jrq7f/rmbnwEHV83fzOEdifJBfnNDDl99nXZxzzgbWVZvPq1r2vXA2s7F4e9d07ty5Dt66YVxxxRVcf/319O3bt97e44wzzuDFF1+kZcuWuy2//fbbSU1N5YYbbqi39xaRpss5R2llmG0llRSXhyiuCFPy7XNFiOLyPZ4rQpRVRigPRagIhSkPRSivjFARjlAeCn83XenNV4S8tqGI83tT61xGcnyjDueajqvW+Fdyzo0DxgHk5OQ0mr/k+PHj6/09Jk+eXO/vISKNX1llmIId5WzaUcam7eVs3lnOtpJKtpVWsq2kkqLSiu/NV4aj++82MS5ASkKQ5PggifFBEuMCJMYFSIgLkBwfpGVyPInxARLjgiQEA1XT3vq4QIC4gBEX9J6DASMuaLuWV58PBqyqbbX5oNcmYEbAIGCGVT1/u8x2W1ZzG2r4mUC1ZWbw7dlAo2qeqteGqvX+ny6si3DOAzpVm+8I5NfB6/qiuLiY8847j7y8PMLhMLfeeiuPPPII9913Hzk5OTzxxBPcfffddOjQgZ49e5KYmMiDDz7IpZdeSnJyMkuWLGHNmjU89dRTPPPMM3zxxRcMGTKEp59+GoAJEybw5z//GeccZ555JnfffTfw3XCmbdq04f/+7/949tln6dSpE1lZWQwcONDH34iINJRIxPHN9jLWFZawbmspawtLyCss4ZvtZWzaUc6m7WVsLwvV+LMtEoK0TEkgIzmelinx9GqXSkZyPBnJCbRMiScjOZ7UxDhaJAZJSYijRUIcKYnBXc8p8UHighr6IlbURThPBK4xs5eAIUBRXZxvvuPNhSzK317r4qrr2yGdP44+fJ9t3nnnHTp06MCkSZMAKCoq4pFHvD5u+fn53HXXXXz55ZekpaVx4okn0r9//10/u3XrVj744AMmTpzI6NGjmTZtGuPHj2fQoEHMnTuXtm3b8rvf/Y7Zs2fTqlUrRo0axeuvv84PfvCDXa8xe/ZsXnrpJebMmUMoFGLAgAEKZ5EmZltJBcs27mTZxh2s2LSTlZuLWVdYwvqtpVSEI7vamUH79CQOyUiiR1Yqww7NpG1aIm3Tkmib7j1npSXSMiWeeAVrk7LfcDazCcDxQBszywP+CMQDOOceBSYDZwArgBLgp/VVbEPo168fN9xwA7/73e8466yzOPbYY3etmzlzJscddxytW7cG4Nxzz2XZsmW71o8ePRozo1+/frRr145+/foBcPjhh7N69WrWrFnD8ccfT1aWd1OSiy66iE8++WS3cP70008555xzSElJAWDMmDH1vs0iUj+cc+RtLWXB+iLmry9iwfoiFm/Ywead5bvatEgI0i2rBX3bpzPq8HZ0apVC59YpdGqdQoeWSSTGBX3cAvFLNL21L9zPegf8os4qqrK/Pdz60qtXL2bPns3kyZO5+eabGTVq1K51zu37vE1iYiIAgUBg1/S386FQiLi46A5UxML5DhE5cOWhMF/lFTFjVSEzVxUyL28b20oqAYgLGL0PSeOE3ln0apdGj3ap9GqXRoeMJH3m5XtiaoSwWJCfn0/r1q25+OKLSU1N3XWuGGDw4MH8+te/ZuvWraSlpfHqq6/u2juOxpAhQ7juuuvYvHkzrVq1YsKECVx77bW7tRk5ciSXXnopN910E6FQiDfffJOrrrqqrjZPROpQJOL4an0RHy0t4IuVm5mzdhvlIe+wdK92qZza9xD6dczgyI4Z9D4kTXvBEjWF8x6++uorbrzxRgKBAPHx8TzyyCO7LmPKzs7mlltuYciQIXTo0IG+ffuSkZER9Wu3b9+ev/zlL5xwwgk45zjjjDM4++yzd2szYMAAzj//fI466ii6dOmy22F1EfHfjrJKPl2+mQ+WbOKjpQVs3lmOGRzeIZ2LhnRhSPfWDOramtYtEvwuVRox29+h2vqSk5PjcnNzd1u2ePFi+vTp40s90dq5cyepqamEQiHOOeccLrvsMs455xxfa2oMvzeRxqy0Isz7Szbyxtx8Plq6icqwIyM5npG9sjjxsCyO69VWYSz7ZWaznXM50bTVnvMBuv3223nvvfcoKytj1KhRu3XmEpGmIxxxfLK8gIlz85m68BuKK8K0S0/kkmO6cuoRh3B0p5a69EjqjcL5AN13331+lyAi9aiwuIJ/z1rH89PXsH5bKRnJ8Yw5qgNj+mczuFtrggF13pL6p3AWEQHm523jmc/X8Ob8fCpCEYZ2b83vz+zDSX3aqiOXNDiFs4g0a7NWF/LAe8uYtmILKQlBzsvpyE+O6Uqvdml+lybNmMJZRJqlmasK+cf7Xii3SU3g92f04fzBnUhPive7NBGFs4g0L1/lFfHXdxbvCuU/nNmHi4Z0ITlBh64ldqirYT3r2rUrmzdvBmDYsGEH/TpPP/00+fmN9n4iIr4rLK7g5tfmM+ahz1iyYQd/OLMPn/72RK44truCWWKO9pwPwoEMxVnd559/ftDv+fTTT3PEEUfQoUOHg34NkeYoHHG8OGMN901dxs7yEJcN78Z1J/fU4WuJaQrnGtx111288MILdOrUiTZt2jBw4EDeeusthg0bxrRp0xgzZgy9evXiT3/6ExUVFWRmZvLCCy/Qrl07tmzZwoUXXkhBQQGDBw/ebTzu1NRUdu7cCcC9997Lyy+/THl5Oeeccw533HEHq1ev5vTTT2fEiBF8/vnnZGdn88YbbzBp0iRyc3O56KKLSE5O5osvviA5OdmvX49IozFn7VZ+/98FLNqwnWGHZnL7mMPV0UsahdgN57dvgm++qtvXPKQfnP7XfTbJzc3l1VdfrfGWjdu2bePjjz8GvNtDTp8+HTNj/Pjx3HPPPfztb3/jjjvuYMSIEdx2221MmjSJcePGfe89pk6dyvLly5k5cybOOcaMGcMnn3xC586dWb58ORMmTODxxx/nvPPO49VXX+Xiiy/mwQcf3HVPaRHZt7LKMPe/t4zHP1lJu/QkHr5oAKcfcYhuMCGNRuyGs08+++wzzj777F17pqNHj9617vzzz981nZeXx/nnn8+GDRuoqKigW7duAHzyySe89tprAJx55pm0atXqe+8xdepUpk6dytFHHw14Q4IuX76czp07061bN4466igABg4cyOrVq+tlO0WaqoX5RVz30lxWbNrJhYM7ccsZfUjTIWxpZGI3nPezh1tf9jXWeIsWLXZNX3vttVx//fWMGTOGjz76iNtvv33Xuv19O3fOcfPNN3/vblOrV6/e7VaTwWCQ0tLSA9wCkebJOcdT01bz17eX0DIlnmcuG8xxvbL8LkvkoKi39h5GjBjBm2++SVlZGTt37mTSpEk1tisqKiI7OxuAZ555ZtfykSNH8sILLwDw9ttvs3Xr1u/97KmnnsqTTz656/zz+vXr2bRp0z7rSktLY8eOHQe1TSJNXVFpJVc+O5s731rEsT3b8M6vRiqYpVGL3T1nnwwaNIgxY8bQv39/unTpQk5OTo23hbz99ts599xzyc7OZujQoaxatQqAP/7xj1x44YUMGDCA4447js6dO3/vZ0eNGsXixYs55phjAK+j2PPPP08wuPfLOS699FKuvvpqdQgT2cOSb7Zz9XOzydtayq1n9eWy4V11blkaPd0ysgbf3haypKSEkSNHMm7cOAYMGOB3WXsVK783kYb2zoJv+PW/55KaFMfDFw1gUNfWfpcksle6ZWQtjR07lkWLFlFWVsYll1wS08Es0hw553j045Xc/c4SjurUknE/Hkjb9CS/yxKpMwrnGrz44ot+lyAiexEKR/jD6wt4adY6zjqyPfed25+keI3wJU1LzIWzc07niw6AX6clRPxQVhnmmhfn8N7ijVxzQg+uP6UXAd1fWZqgmArnpKQktmzZQmZmpgI6Cs45tmzZQlKSDudJ01dUWskVz8wid81W7jz7cH5yTFe/SxKpNzEVzh07diQvL4+CggK/S2k0kpKS6Nixo99liNSrbSUV/OTJmSzesJ1/XnA0o/trjHlp2mIqnOPj43eNtCUiAt7dpC4eP4MVm3by6MUDOalPO79LEql3MRXOIiLVbSup4H8fn86qzcU8fkmOBhaRZkPhLCIxaUdZJZc8OZOVBcU8cWkOx/ZUMEvzoeE7RSTmlFaEufzpXBbmb+fhiwYomKXZUTiLSEwJhSNc8+KX5K4p5P7zj+LkvjrHLM2PDmuLSMxwzvGH1xfw/pJN/OkHR6hXtjRb2nMWkZjxj/eX89KsdVxzQg8uHtrF73JEfBNVOJvZaWa21MxWmNlNNazvbGYfmtkcM5tvZmfUfaki0pS9MXc9D7y3nB8N7MhvRvXyuxwRX+03nM0sCDwEnA70BS40s757NPsD8LJz7mjgAuDhui5URJqu2Wu2cuMr8xnSrTV/PqefRgiUZi+aPefBwArn3ErnXAXwEnD2Hm0ckF41nQHk112JItKUrd9WylXP5dI+I4lHLx5IQpzOtolE0yEsG1hXbT4PGLJHm9uBqWZ2LdACOLlOqhORJq2sMsxVz+VSXhnhpbE5tGqR4HdJIjEhmq+oNR1f2vNWSBcCTzvnOgJnAM+Z2fde28zGmlmumeVq/GyR5u3bntkL1m/n/vOPokfbNL9LEokZ0YRzHtCp2nxHvn/Y+nLgZQDn3BdAEtBmzxdyzo1zzuU453KysjSogEhz9vyMtbwyO49fntRT1zKL7CGacJ4F9DSzbmaWgNfha+IebdYCJwGYWR+8cNausYjU6Ku8Iu58cyEn9M7iVyf19LsckZiz33B2zoWAa4ApwGK8XtkLzexOMxtT1ew3wJVmNg+YAFzqnNvz0LeICDvLQ1w74UvapCZy//lHEQioZ7bInqIaIcw5NxmYvMey26pNLwKG121pItIU/fGNhawtLOHFK4fSMkUdwERqomsWRKTBvDF3Pa9+mcc1J/ZkaPdMv8sRiVkKZxFpEGu3lPD7/y4gp0srfnliD7/LEYlpCmcRqXeV4QjXvjQHM3jggqOIC+q/HpF90V2pRKTe/f3dZcxbt42H/ncAHVul+F2OSMzT11cRqVezVhfy6Mdfc8GgTpx5ZHu/yxFpFBTOIlJvyirD/PaV+XRslcytZ+15vxwR2Rsd1haRenP/u8tYtbmYF64YQotE/XcjEi3tOYtIvZi7bhuPf7qSCwd3ZniP743mKyL7oHAWkTpXHgrz21fm0S49iZvPOMzvckQaHR1nEpE699AHK1i2cSdPXTqI9KR4v8sRaXS05ywidWphfhEPf/Q1PxyQzQmHtfW7HJFGSeEsInWmMhzht6/Mp2VKArepd7bIQdNhbRGpM09PW83C/O08evEA3dRCpBa05ywideKbojIeeG8ZJx3WltOO0GAjIrWhcBaROvGnSYsIRRx/HH2436WINHoKZxGptWkrNvPW/A38/PgedM7U2NkitaVwFpFaqQhFuO2NBXRuncJVx3X3uxyRJkEdwkSkVp74bBVfFxTz1KWDSIoP+l2OSJOgPWcROWjrt5Xyz/eXM6pvO13TLFKHFM4ictD+9NYiHI7bRuuaZpG6pHAWkYPy6fIC3l7wDdee2JOOrdQJTKQuKZxF5ICFwhHuemsRnVuncMWx3fwuR6TJUTiLyAH7d+46lm3cyc2nH0ZinDqBidQ1hbOIHJAdZZX8feoyBndtzWlHHOJ3OSJNki6lEpED8tCHX7OluIKnftoHM/O7HJEmSXvOIhK1dYUlPPnZKn44IJsjO7b0uxyRJkvhLCJR++s7SwgE4MZTe/tdikiTpnAWkajMXlPIpPkbuGrkobTPSPa7HJEmTeEsIvsViTjufGsx7dITNX62SANQOIvIfr05P59567Zx46mHkZKgfqQi9U3hLCL7VBGKcN/UpfRtn84Pj872uxyRZiGqcDaz08xsqZmtMLOb9tLmPDNbZGYLzezFui1TRPzy4ow1rCss5XenH0YgoEunRBrCfo9PmVkQeAg4BcgDZpnZROfcomptegI3A8Odc1vNTLenEWkCdpaH+NcHKxh2aCYje7bxuxyRZiOaPefBwArn3ErnXAXwEnD2Hm2uBB5yzm0FcM5tqtsyRcQPj3+yki3FFfzutMM04IhIA4omnLOBddXm86qWVdcL6GVm08xsupmdVtMLmdlYM8s1s9yCgoKDq1hEGkTBjnIe/3QlZ/ZrT/9OGnBEpCFFE841fV12e8zHAT2B44ELgfFm9r1Ps3NunHMuxzmXk5WVdaC1ikgD+tcHyykPRbhBA46INLhowjkP6FRtviOQX0ObN5xzlc65VcBSvLAWkUZozZZiXpyxlgsGdaJbmxZ+lyPS7EQTzrOAnmbWzcwSgAuAiXu0eR04AcDM2uAd5l5Zl4WKSMO5b+oy4oMBrjtJ37FF/LDfcHbOhYBrgCnAYuBl59xCM7vTzMZUNZsCbDGzRcCHwI3OuS31VbSI1J+v8op4c14+l4/oRtv0JL/LEWmWohrqxzk3GZi8x7Lbqk074Pqqh4g0YvdOXUrLlHjGaphOEd9ohDAR2WXmqkI+WVbAz447lPSkeL/LEWm2FM4iAoBzjvumLCUrLZGfHNPV73JEmjWFs4gA8MnyzcxcXci1J/YgOSHodzkizZrCWURwzvG3qUvJbpnMBYM6+12OSLOncBYRpi7ayPy8Iq47uScJcfpvQcRv+hSKNHPhiOPvU5fRPauFbgkpEiMUziLN3Fvz81m6cQe/PrkXcUH9lyASC/RJFGnGKsMR7n93GX3ap3Nmv/Z+lyMiVRTOIs3Yq7PzWL2lhN+c0otAQLeEFIkVCmeRZqo8FOaf7y+nf6eWnNSnrd/liEg1CmeRZmrCjLXkF5Vx46jemGmvWSSWKJxFmqGSihAPfvg1Q7u3ZniPTL/LEZE9KJxFmqFnPl/D5p3l3Hiq9ppFYpHCWaSZ2V5WyaMff80JvbMY2KW13+WISA0UziLNzPhPV1FUWslvRvX2uxQR2QuFs0gzUlhcwROfruSMfodwRHaG3+WIyF4onEWakcc+/prSyjDXn9LL71JEZB8UziLNxKbtZTzzxWp+cFQ2Pdqm+V2OiOyDwlmkmXjwwxWEwo5fnay9ZpFYp3AWaQbWFZYwYeZazhvUic6ZKX6XIyL7oXAWaQbuf28ZATN+eWJPv0sRkSgonEWauGUbd/DfOeu5ZFhXDslI8rscEYmCwlmkifvb1KW0SIjjZ8cd6ncpIhIlhbNIEzZv3TamLNzIlcd2p1WLBL/LEZEoKZxFmrB7pyyldYsELj+2m9+liMgBUDiLNFGfr9jMZys28/PjDyU1Mc7vckTkACicRZog5xz3TFlK+4wkLh7axe9yROQAKZxFmqD3Fm9i7rptXHdST5Lig36XIyIHSOEs0sSEI477piylW5sW/GhgR7/LEZGDoHAWaWLenJfP0o07uP6UXsQF9REXaYyi+uSa2WlmttTMVpjZTfto9yMzc2aWU3cliki0KkIR/v7uMvq2T+fMfu39LkdEDtJ+w9nMgsBDwOlAX+BCM+tbQ7s04JfAjLouUkSi83LuOtYWlnDjqb0JBMzvckTkIEWz5zwYWOGcW+mcqwBeAs6uod1dwD1AWR3WJyJRKq0I88/3l5PTpRXH987yuxwRqYVowjkbWFdtPq9q2S5mdjTQyTn3Vh3WJiIH4OnPV7NpRzk3ntobM+01izRm0YRzTZ9yt2ulWQC4H/jNfl/IbKyZ5ZpZbkFBQfRVisg+FRZX8PCHKzjxsLYM6Z7pdzkiUkvRhHMe0KnafEcgv9p8GnAE8JGZrQaGAhNr6hTmnBvnnMtxzuVkZemwm0hd+dcHyymuCHHT6Yf5XYqI1IFownkW0NPMuplZAnABMPHblc65IudcG+dcV+dcV2A6MMY5l1svFYvIbtZsKeb56Ws4L6cTvdql+V2OiNSB/Yazcy4EXANMARYDLzvnFprZnWY2pr4LFJF9u2fKUuICAa4/pZffpYhIHYlqNHzn3GRg8h7LbttL2+NrX5aIRGPO2q1Mmr+BX57Uk7bpSX6XIyJ1RMMHiTRSzjn+MnkJbVITGTuyu9/liEgdUjiLNFLvLtrIzNWF/OrknrolpEgTo3AWaYRC4Qh/fWcJh2a14IJBnfb/AyLSqCicRRqhCbPWsbKgmJtO76ObW4g0QfpUizQyRSWV/H3qUoZ0a83Jfdr6XY6I1AOFs0gjc/97yygqreT2MYdrmE6RJkrhLNKILP1mB89NX8P/DulMn/bpfpcjIvVE4SzSSDjnuPOthaQmxvGbU3r7XY6I1COFs0gjMWXhN0xbsYXfjOpFqxYJfpcjIvVI4SzSCJRVhvnTpMUcdkga/zu4s9/liEg9UziLNAKPf7KSvK2l3Da6ry6dEmkG9CkXiXEbikp5+KOvOf2IQxh2aBu/yxGRBqBwFolxf5q0mIhz3HJGH79LEZEGonAWiWEfLtnEpPkb+MUJPejUOsXvckSkgSicRWJUSUWIP7y+gB5tU7n6uEP9LkdEGpBuZSMSox54bznrt5Xyn6uPISFO36NFmhN94kVi0ML8Ip74bBUXDu7EoK6t/S5HRBqYwlkkxoQjjlte+4pWKfHcdJo6gYk0RwpnkRjz3BermZdXxK1n9SUjJd7vckTEBwpnkRiyoaiUe6csZWSvLMb07+B3OSLiE4WzSIxwznHr6wsJO8efzj5Ct4MUacYUziIx4j+z83hv8UZuGNWbzpm6plmkOVM4i8SAdYUl3PnmIoZ0a81lw7v5XY6I+EzXOYv4LBJx3PCfeQDcd25/AoF6OpxdUQLFBVBRDJWlEJ8ECS0gJRMS0+rnPUXkoCicRXz25LRVzFhVyD0/OrLuhugsK4K102HtF5CXC1tWwI4Ne2/foi1kHgrZA6HzUOg8DFpk1k0tInLAFM4iPlq2cQf3TFnKKX3bce7AjrV7sbIiWPSG91j5MUQqIRAH7ftD9xOgdXdIOwQSUyE+xdt7riiGnRuh8GvYvBxmPg5fPAgWgC7Doc8YOOKH0EJ3wxJpSApnEZ9UhCL8+t9zSUuM4y8/7HfwvbM3zPNCdcGrUFkCLbvA0Kuh5yjIzoGEA9gbD5VD/lxY8S4smghv3whTboG+Z8Ogy6HzMaBe5CL1TuEs4pN/vr+chfnbeezHA2mTmnjgL7Dmc/jkPvj6fYhLhn4/gpyfQocBBx+gcYnQeYj3OPEPsHERfPkMzJ0AC16BTkNh5A3Q42SFtEg9MuecL2+ck5PjcnNzfXlvEb99vmIzFz0xg/8Z0JH7zu1/YD+8YR68+0dY+SGktIFjfgE5l0Fyy/opFrzD33NegGn/gO153h75qLugy7D6e0+RJsbMZjvncqJqq3AWaVgFO8o545+fkp4Ux8RrRtAiMcoDWDs3wbu3wbyXvCA+9gYvlA/ksHVthSpg3gT46C9eB7PeZ8Jpf4ZWXRuuBpFG6kDCWYe1RRpQJOK4/uW5bC+t5LnLB0cXzJEIzHnWC+bKUhj+Sxhxff3uKe9NXAIMvAT6nQvTH4ZP/w4PDYUTboahv4Cg/ksRqQtRDUJiZqeZ2VIzW2FmN9Ww/nozW2Rm883sfTPrUvelijR+//pgBZ8u38ztYw7nsEPS9/8DBUvh6TPgzeugXT+4ehqccqc/wVxdQop37vmamXDoCd4Xh3HHw/rZ/tYl0kTsN5zNLAg8BJwO9AUuNLO+ezSbA+Q4544EXgHuqetCRRq7D5Zs5IH3l/HDAdlcMKjTvhuHKuDDP8Mjw2HTYhjzIFz6FmT1aphio5XRES54Ec57Dko2w+MnweTfQvlOvysTadSi2XMeDKxwzq10zlUALwFnV2/gnPvQOVdSNTsdqOUFmyJNy+rNxfzqpbn0OSSdP5+zn8umtnwNT46Cj++Gw8+Ba3JhwI9jt3e0GfQdA7+YAYOugJnj4LFjvcFPROSgRBPO2cC6avN5Vcv25nLg7ZpWmNlYM8s1s9yCgoLoqxRpxHaWh7j6+dkEAsZjPx5IUnxw743nvwyPjYTCVXD+8/A/j0NqVsMVWxtJGXDmfd4efrgSnhgFH90N4ZDflYk0OtGEc01f12vs4m1mFwM5wL01rXfOjXPO5TjncrKyGsl/OCK1EI44rpswh+WbdvKvC4/e+/Cc5Tvgv1fDa1fCIf3g6s+gz+iGLbaudB0BP5vmXXf90Z/hqdOgcKXfVYk0KtGEcx5Q/QRZRyB/z0ZmdjLwe2CMc668bsoTadzufmcJ7y/ZxO2j+3Jsz718Ic2fA48dB/P/DcfdBJe8BS33c0461iVlwA/Hwf88AZuXwSMjvOukfbp0U6SxiSacZwE9zaybmSUAFwATqzcws6OBx/CCeVPdlynS+EyYuZZxn6zkkmO68ONjun6/QSQCnz+jZkW5AAAS1klEQVQI40+BUJkXyifc3LQuR+r3I/jZ55A9AN74Obx6BZRt97sqkZi333B2zoWAa4ApwGLgZefcQjO708zGVDW7F0gF/mNmc81s4l5eTqRZeH/xRv7w+gKO65XFrWfteXED3oAiL54LU38PvU71DmN3Hd7whTaEjI7wkze84UAX/hceHaHOYiL7oRHCROrYnLVbufDx6fRql8aEK4d+f6CRrz+A167y7iJ12p8h5/LY7Yld19bO8Paed+TDCb+H4b+CQFTDLYg0egcyQpg+FSJ1aPnGHVz29CzapSfx5KWDdg/mcKU3WMdz50BKaxj7oXfpUXMJZvBuqHH1p3DYWfD+HfDcD2DHN35XJRJzFM4idWT15mIuGj+D+GCAZy8bvPudpgpXwpOnejeOGPhTuPJDaHe4f8X6KbklnPs0jPkXrJsJjwyDZVP9rkokpiicRerA+m2lXDR+BpXhCC9cMYQumS2+Wzn/P/DoSNiyAs57FkY/0LA3q4hFZjDgJ3DVx5DW3jv//vZN3v2kRUThLFJb6wpLuGDcF2wvq+S5y4fQs12at6J8J7z+c3jtCm8v+epp0Pfsfb9Yc5PVG654H4ZcDTMegfEnweblflcl4juFs0gtrNlSzAXjplNUUskLVwzhiOwMb0X+XG+kr3kT4LjfwaWTGv+1y/UlPglOvxsufAmK1nu/tznP65poadYUziIHacWmHZz/2HSKK0K8eOVQjuzY0rt2+YuHYfzJ3u0dL3kTTrilaV27XF96n+6NLJY9EN74Bbx6udejXaQZUjiLHIQv127lR49+QSjimHDlUG+PeXs+PH8OTLkZeo7ygqbrCL9LbVzSO1RdE30rLHzduyZ63Sy/qxJpcApnkQP04ZJNXPT4DDKS43ntZ8Po0z4dFrwGDx/j9T4e/Q+44AXvcik5cIGgd6/oy97xRvF/8lT49G8QCftdmUiDUTiLHIBnPl/N5c/M4tC2LXjl6mF0blEJr42FV34KmT28kb4GXtq8rl2uL50Ge9dE9x0D79/pXRO9fYPfVYk0CIWzSBRC4Qi3vbGAP05cyImHtePfY48ha+On8Mhw+OoVOP4WuGwKZB7qd6lNS3JL+NFT3jXRebneNdELXlVnMWny1EtFZD8KdpRzzYtfMmNVIVeN7M5vj2tHcPK1MO9FaNMLLn8XOg70u8ym69trojsNhf+OhVcu874Qnfk37xy1SBOkcBbZh9lrtvLzF2ZTVFrJ/ecdyTmJufDwaCjdCiNv9B5xift/Iam9rF5w+Xsw/WH48P/goSEw6i4YcIlOI0iTo8PaIjUIRxwPfbiC8x77gqT4IG/+pBvnLLsJ/nMppGfD2I+8uywpmBtWMA6G/9K7DWX7/vDmdfDMaA1cIk2OwllkD/nbSrl4/AzunbKU0X1bM2XADHr++3hY8R6cfIc3otUh/fwus3nLPBR+MtHrGb9hntdTfuqtUL7D78pE6oQOa4tUcc7xn9w87nprEaFIhOdHbGH417dgK1ZDnzEw6k/QqovfZcq3AgGvZ3zvM+C9O+Dzf8L8f8Mpd8KR5+tQtzRq2nMWAVZtLuaSp2bx21fnc2pWIV92e4wRuddiwUT48etw/nMK5liV2hZ+8JB3RCM9G/57lXdtdJ7uFy+Nl/acpVkrLg/x4IcreOLTVXSPK+D9bu/QfcNkLDENTv0zDB4LwXi/y5RodMzxAnruC969osefBL3P9PoGtOvrd3UiB0ThLM2Sc46J8/L5y+QlhLdv4Kn27zKs6C2sIA6GXQsjfq0RvhqjQAAG/BgO/wFMf9Q71P3IMDjyPDj+Zmjdze8KRaJizqeL+XNyclxurg47ScObvnILf5u6lPzVy/hty/cZXTmVgAt5l+SMvBHS2/tdotSVkkKY9gDMGAeRSuh/IQy/Dtr09LsyaYbMbLZzLieqtgpnaS6mr9zCA+8to2jVHH6Z9DanMg0zw/qdB8fdCK27+12i1JftG+Czv8OXz0Ko3OtENvw66DzE78qkGVE4i1RxzjF9ZSH/fG8xCWs+4erEKRzj5uISUrGBl8LQn0FGR7/LlIZSvBlmjvMepVu9UceO+bkX1upbIPVM4SzNXklFiDfm5vPmZ7MZsGUS/xv/ER0owLVoiw29GnIug+RWfpcpfqkohjnPw+cPQtFaaNEWjr4YBl4Crbr6XZ00UQpnabZWbS5mwufLyf9yEqPDH3By8EuCRAh3PY7goJ96vXfjEvwuU2JFJAzL34XZT8PyKeAicOiJXlD3Oh0SUvyuUJqQAwln9daWRm/zznLenreOlbPe4bDN7/KL4CwyrJjKFpkEBlwLAy8hqLtFSU0CQeh9mvcoWu/tTX/5rHdzjfgW0Pt0OOJ/oMdJGqpVGpT2nKVR2lZSwUdfrWb1rMm02/gJpwRm0ca2UxFsQbj3mSQffR50P17nEeXARcKw5nPv1pSL3oDSQkjMgMPOgJ6jvD3r5JZ+VymNkA5rS5PjnGPJhu3MnpNLxZKp9CiaxhBbTKKFKA+mUN71RNJzLoAep0B8kt/lSlMRroSVH8OCV2DZO14nMgtC52Og5yneI6uPd321yH4onKXRc86xZnMxixfMZueyj8nYOJP+4a9oZ9sAKEzuQujQU8g6ejTWZZjOI0v9C4dgfS4smwLLp8LGBd7ylEzoMgy6HgtdhkPbvgprqZHCWRqd4vIQS1euomDpF4TWzSaj8Ct6R5aTZdsB2BbMpKjdEFr1PYH0Pid5dyUS8VPRelj5IayeBms+g21rveVJLSF7AHQY8N2zBrYRFM4Sw5xz5BduZ/3yeexc+xWRjQtpUbSMzpWryLbNAEQwNiZ0oSSrP2k9h5N1xElY5qG6y5DEtm1rvaBe+wXkfwkbF4ELe+tSD4F2h3tjfLetemT1hvhkf2uWBqVwFl9FIo6NhYUUrFnKjm9WECpYQdy21aQUryWzIp8OFBBnEQBCBPkmvhPFGb0IZB9Nm15DadVjECSm+bwVIrVUWQrffAXrZ0P+XNi0CAqWQri8qoF5A+C07gatunkj1LWuem7VDRJTfS1f6l6dX0plZqcB/wCCwHjn3F/3WJ8IPAsMBLYA5zvnVh9I0dI4lFeUs3nTRrZt3kDJ5rVUbFuPK8onWPwNiaUbSS0voFV4M+2tiOoH8raTSkF8B4pa96OodXeSs48gq/tRZHTsQ0edL5amKD4ZOg32Ht8Kh2DrKti4EAqWQOFK77FkEpRs3v3nU9p4h8PTOkDaIZDeAdLaf/ec2haSW0NQV8Q2Rfv9q5pZEHgIOAXIA2aZ2UTn3KJqzS4HtjrnepjZBcDdwPn1UbDUTiQcoaxsJyU7iigvLqK8eDvlJUWU7yyivKSIUPFWXEkhgbKtxJVvI7FiK8mhIlLC20l3O0inmGwge4/X3UYaW4NtKElqy/oWh5PfsjMJWT3IyO5Fm86HkZ6WSbofGywSS4Jx3k03arrxRtl2L7gLV0LhKti2BnZ8A9vzvcPkxQU1v2ZShhfkKZnVHq285YnpVY80SKp6rr5MVzbErGi+cg0GVjjnVgKY2UvA2UD1cD4buL1q+hXgQTMz59cx8wbiIhGcczjniETCVdMRXCRCJBImFAoRCVUSClUQDlUSrpoPh0NEwpVEQpVEwiHCoQoiVctcOEQkHNrt2UUqcaFKXKgMV+k9CJVDuBwLlWHhcixUTiBSTjBcRjBSUfUoJy5SQVykgiRXSrIrpQWlpJhjf+Me7SSZHZZGcTCd0rgMtqd0Ji+xJZbcmri0TJIy2pGc2ZH0dp1p1bYzLROS0ZWfIrWQlA7t+3uPmoQqYOc33wV2cYF3162SLd5ed8kW2J4H38z3pkNl+3/PYIK3hx+X7D3HJ0NcEsSneMG923S1NsF4CMR7z7umE/aY33NdnDcdiPMGf7GA99g1HYxiebDZ9ISPJpyzgXXV5vOAPW/lsquNcy5kZkVAJrDHcZr6MWfKM7SecS9GBKv6PmBEMJz3cO676X08AjjAEaih/a51VfNBcxjwbRelYENs6B4qXJAKEqiweCpIoNISqLR4Ki2BUCCB8kAKpfGtcMFEwnEpROJTiSSkQkIqlpRKIDGNYFI68SlpJLdoSUpaBqktM0lt2ZbU+ER0xkskhsQlQMvO3iMaoQoo3wHl26seO7y98z2XVZZCZQlUlnnPoTJvuqSwarpqXajUaxuuqN/tjMa3Qb0rxL8Nb6vqOFr9OVDDsurr2Ms6844+XPmBL5sYTTjX1EV2zz3iaNpgZmOBsQCdO0f5DywK8S1asSWlO67qF+r47o/hzKBq3lX7xbvd/mABvv1juF3TVd/OLLDba3iPAFb9NXa1/e61zAJV3xrjsGAcVjUdCMZBMI5AwFtm1aYDcfEEg177QFw8gWA8wTjvZ4Jx8cQnppCQmEx8YjKJSSkkxMejs7UiUqO4BIjLhBaZdfu6kYh3b+xwhTdIS7iy2nxo9+lwRdV81XS4wuvB7pw3EpsLe+OZ7zYdOcDlVa/37bzzdqR2e3aRPZaxj3XVnuP9G1s9mnDOAzpVm+8I5O+lTZ6ZxQEZQOGeL+ScGweMA6+39sEUXJMjRoyBEWPq6uVERGRvAgEIJGqs8XoWzcH7WUBPM+tmZgnABcDEPdpMBC6pmv4R8EFTP98sIiJSX/a751x1DvkaYAreqdUnnXMLzexOINc5NxF4AnjOzFbg7TFfUJ9Fi4iINGVRXSDnnJsMTN5j2W3VpsuAc+u2NBERkeapefRJFxERaUQUziIiIjFG4SwiIhJjFM4iIiIxRuEsIiISY3y7ZaSZFQBr6vAl29BAw4U2AG1L7GpK26NtiU3althUF9vSxTmXFU1D38K5rplZbrT3yYx12pbY1ZS2R9sSm7Qtsamht0WHtUVERGKMwllERCTGNKVwHud3AXVI2xK7mtL2aFtik7YlNjXotjSZc84iIiJNRVPacxYREWkSmlQ4m9lRZjbdzOaaWa6ZDfa7ptows2vNbKmZLTSze/yup7bM7AYzc2bWxu9aDpaZ3WtmS8xsvpn918xa+l3TgTKz06r+Xa0ws5v8rudgmVknM/vQzBZXfUau87um2jKzoJnNMbO3/K6lNsyspZm9UvVZWWxmx/hd08Eys19X/ftaYGYTzCypId63SYUzcA9wh3PuKOC2qvlGycxOAM4GjnTOHQ7c53NJtWJmnYBTgLV+11JL7wJHOOeOBJYBN/tczwExsyDwEHA60Be40Mz6+lvVQQsBv3HO9QGGAr9oxNvyreuAxX4XUQf+AbzjnDsM6E8j3SYzywZ+CeQ4547Au21yg9wSuamFswPSq6YzgHwfa6mtnwF/dc6VAzjnNvlcT23dD/wW72/UaDnnpjrnQlWz04GOftZzEAYDK5xzK51zFcBLeF8CGx3n3Abn3JdV0zvwAiDb36oOnpl1BM4ExvtdS22YWTowEngCwDlX4Zzb5m9VtRIHJJtZHJBCA+VKUwvnXwH3mtk6vD3NRrVXs4dewLFmNsPMPjazQX4XdLDMbAyw3jk3z+9a6thlwNt+F3GAsoF11ebzaMSB9i0z6wocDczwt5JaeQDvC2zE70JqqTtQADxVdYh+vJm18Luog+GcW4+XJWuBDUCRc25qQ7x3XEO8SV0ys/eAQ2pY9XvgJODXzrlXzew8vG9uJzdkfQdiP9sSB7TCO1w3CHjZzLq7GO1ev59tuQUY1bAVHbx9bYtz7o2qNr/HO6z6QkPWVgeshmUx+W8qWmaWCrwK/Mo5t93veg6GmZ0FbHLOzTaz4/2up5bigAHAtc65GWb2D+Am4FZ/yzpwZtYK78hSN2Ab8B8zu9g593x9v3ejC2fn3F7D1syexTtnA/AfYvzw0H625WfAa1VhPNPMInhjuxY0VH0HYm/bYmb98P5hzzMz8A4Df2lmg51z3zRgiVHb198FwMwuAc4CTorVL0v7kAd0qjbfkUZ8+sfM4vGC+QXn3Gt+11MLw4ExZnYGkASkm9nzzrmLfa7rYOQBec65b49ivIIXzo3RycAq51wBgJm9BgwD6j2cm9ph7XzguKrpE4HlPtZSW6/jbQNm1gtIoBEOIO+c+8o519Y519U51xXvgzsgVoN5f8zsNOB3wBjnXInf9RyEWUBPM+tmZgl4nVsm+lzTQTHv294TwGLn3N/9rqc2nHM3O+c6Vn1GLgA+aKTBTNVne52Z9a5adBKwyMeSamMtMNTMUqr+vZ1EA3Vua3R7zvtxJfCPqhP3ZcBYn+upjSeBJ81sAVABXNII99KaogeBRODdqiMB051zV/tbUvSccyEzuwaYgtfz9Enn3EKfyzpYw4EfA1+Z2dyqZbc45yb7WJN4rgVeqPoCuBL4qc/1HJSqw/KvAF/incaaQwONFKYRwkRERGJMUzusLSIi0ugpnEVERGKMwllERCTGKJxFRERijMJZREQkxiicRUREYozCWUREJMYonEVERGLM/wPOZrZeVnBqTgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "x = nd.arange(-8.0, 8.0, 0.1)\n", "x.attach_grad()\n", "with autograd.record():\n", " y = x.sigmoid()\n", "y.backward()\n", "\n", "plt.figure(figsize=(8, 4))\n", "plt.plot(x.asnumpy(), y.asnumpy())\n", "plt.plot(x.asnumpy(), x.grad.asnumpy())\n", "plt.legend(['sigmoid', 'gradient'])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Symmetry\n", " - Assume that we initialize the parameters of a layer as $\\mathbf{W}_l = 0$ or all entries of $\\mathbf{W}_l$ are identical. \n", " - In this case the gradients for all units in the layer are identical.\n", " - We will never be able to use the expressive power inherent in the layer. \n", " - In fact, the hidden layer behaves as if it had only a single unit." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.15.2 Parameter Initialization\n", "- We can ensure that at least initially the gradients do not vanish and that they are within a reasonable scale where the network weights do not diverge. \n", "- Basic initialization\n", " - `net.initialize()`\n", " - It use the default random initialization method\n", " - Each element of the weight parameter is randomly sampled with an uniform distribution $U[-0.07, 0.07]$ \n", " - Bias parameters are all set to $0$. \n", " \n", " - `net.initialize(init.Normal(sigma=0.01))` \n", " - Pick normally distributed random numbers as initial values for the weights and bias.\n", " - Both choices tend to work quite well in practice for moderate problem sizes.\n", "\n", "- Xavier Initialization\n", " - http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\n", " - Assume that hidden units of a layer $$h_{i} = \\sum_{j=1}^{n_\\mathrm{in}} W_{ij} x_j$$\n", " - The weights $W_{ij}$ are all drawn independently from the same distribution. \n", " - Let's assume that this distribution has zero mean and variance $\\sigma^2$ \n", " - Let's assume that the inputs also have zero mean and variance $\\gamma^2$ and that they're independent of $\\mathbf{W}$. \n", " - In this case we can compute mean and variance of $h_i$ as follows: $$ \\begin{aligned} \\mathbf{E}[h_i] & = \\sum_{j=1}^{n_\\mathrm{in}} \\mathbf{E}[W_{ij} x_j] = 0 \\\\ \\mathbf{E}[h_i^2] & = \\sum_{j=1}^{n_\\mathrm{in}} \\mathbf{E}[W^2_{ij} x^2_j] \\\\ & = \\sum_{j=1}^{n_\\mathrm{in}} \\mathbf{E}[W^2_{ij}] \\mathbf{E}[x^2_j] \\\\ & = n_\\mathrm{in} \\sigma^2 \\gamma^2 \\end{aligned} $$\n", "\n", " - One way to keep the variance fixed is to set $n_\\mathrm{in} \\sigma^2 = 1$. \n", " - When backpropagation is executed, instead of $\\mathbf{W} \\mathbf{w}$, we need to deal with $\\mathbf{W}^\\top \\mathbf{g}$, where $\\mathbf{g}$ is the incoming gradient from the layer above. \n", " - Using the same reasoning, one way to keep the gradients' variance fixed is to set $n_\\mathrm{out} \\sigma^2 = 1$. \n", " - So, we simply try to satisfy $$ \\begin{aligned} \\frac{1}{2} (n_\\mathrm{in} + n_\\mathrm{out}) \\sigma^2 = 1 \\text{ or equivalently } \\sigma = \\sqrt{\\frac{2}{n_\\mathrm{in} + n_\\mathrm{out}}} \\end{aligned} $$\n", " - For Gaussian random variables...\n", " - the Xavier initialization picks a normal distribution with zero mean and variance $\\sigma^2 = 2/(n_\\mathrm{in} + n_\\mathrm{out})$. \n", " - For uniformly distributed random variables $U[-a, a]$... \n", " - What do we choose $a$ value?\n", " - Variance $\\sigma^2$ is given by $a^2/3$. \n", " - Plugging $a^2/3$ into $\\sigma^2$ yields that $a=\\sqrt{6/(n_\\mathrm{in} + n_\\mathrm{out})}$\n", " \n", "- Beyond\n", " - The `mxnet.initializer` package\n", " - https://mxnet.incubator.apache.org/api/python/optimization/optimization.html#the-mxnet-initializer-package" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3.16 Environment\n", "- Two important things\n", " - 1) where the data came from \n", " - 2) how the built models get deployed" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.16.1 Covariate Shift" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- ***Covariate Shift***\n", " - the situation where the distribution over the covariates (aka training data) is shifted on test data relative to the training case\n", " - Mathematically speaking, we are referring the case where $p(x)$ changes but $p(y|x)$ remains unchanged" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" } }, "nbformat": 4, "nbformat_minor": 2 }