{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "$$\n", "\\newcommand{\\mat}[1]{\\boldsymbol {#1}}\n", "\\newcommand{\\mattr}[1]{\\boldsymbol {#1}^\\top}\n", "\\newcommand{\\matinv}[1]{\\boldsymbol {#1}^{-1}}\n", "\\newcommand{\\vec}[1]{\\boldsymbol {#1}}\n", "\\newcommand{\\vectr}[1]{\\boldsymbol {#1}^\\top}\n", "\\newcommand{\\rvar}[1]{\\mathrm {#1}}\n", "\\newcommand{\\rvec}[1]{\\boldsymbol{\\mathrm{#1}}}\n", "\\newcommand{\\diag}{\\mathop{\\mathrm {diag}}}\n", "\\newcommand{\\set}[1]{\\mathbb {#1}}\n", "\\newcommand{\\norm}[1]{\\left\\lVert#1\\right\\rVert}\n", "\\newcommand{\\pderiv}[2]{\\frac{\\partial #1}{\\partial #2}}\n", "\\newcommand{\\bb}[1]{\\boldsymbol{#1}}\n", "\\newcommand{\\ip}[3]{\\left<#1,#2\\right>_{#3}}\n", "\\newcommand{\\E}[2][]{\\mathbb{E}_{#1}\\left[#2\\right]}\n", "$$\n", "\n", "# CS236781: Deep Learning\n", "# Tutorial 5: Optimization" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Introduction\n", "\n", "In this tutorial, we will cover:\n", "\n", "- Descent-based optimization\n", "- Back-propagation\n", "- Automatic differentiation\n", "- PyTorch backward functions\n", "- Bi-level differentiable optimization\n", "- Time-series prediction with CNNs" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:31.338632Z", "iopub.status.busy": "2020-11-24T17:30:31.337989Z", "iopub.status.idle": "2020-11-24T17:30:31.962948Z", "shell.execute_reply": "2020-11-24T17:30:31.963542Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# Setup\n", "%matplotlib inline\n", "import os\n", "import sys\n", "import time\n", "import torch\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:31.966723Z", "iopub.status.busy": "2020-11-24T17:30:31.966239Z", "iopub.status.idle": "2020-11-24T17:30:31.985956Z", "shell.execute_reply": "2020-11-24T17:30:31.986488Z" }, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "plt.rcParams['font.size'] = 14\n", "data_dir = os.path.expanduser('~/.pytorch-datasets')\n", "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Theory Reminders" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Descent-based optimization" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "As we have seen, training deep neural network is performed iteratively using descent-based optimization." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "The general scheme is,\n", "\n", "1. Initialize parameters to some $\\vec{\\Theta}^0 \\in \\set{R}^P$, and set $k\\leftarrow 0$.\n", "2. While not converged:\n", " 1. Choose a direction $\\vec{d}^k\\in\\set{R}^P$\n", " 2. Choose a step size $\\eta_k\\in\\set{R}$\n", " 3. Update: $\\vec{\\Theta}^{k+1} \\leftarrow \\vec{\\Theta}^k + \\eta_k \\vec{d}^k$\n", " 4. $k\\leftarrow k+1$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Which descent direction to choose?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The one which maximally decreases the loss function $L(\\vec{\\Theta})$:\n", "\n", "$$\n", "\\vec{d} =\\arg\\min_{\\vec{d'}} L(\\vec{\\Theta}+\\vec{d'})-L(\\vec{\\Theta})\n", "\\approx\n", "\\arg\\min_{\\vec{d'}}\\nabla L(\\vec{\\Theta})^\\top\\vec{d'}, \\\n", "\\mathrm{s.t.} \\norm{\\vec{d}}_p=1\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Choice of norm determines $\\vec{d}$. For example,\n", "- $p=1$: Coordinate descent: direction of the largest gradient component.\n", "- $p=2$: Gradient descent: $\\vec{d}=-\\nabla L(\\vec{\\Theta})$.\n", "\n", "|$p=1$|$p=2$|\n", "|---|---|\n", "| | | " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Drawbacks and mitigations?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Susceptible to initialization**\n", "\n", "Initializing near local minima can prevent finding better ones.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Can use stochastic gradient to get a different loss surface every iteration.\n", "\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Sensitive to learning rate**\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Line search (1D minimization):\n", "$$\n", "\\eta_k = \\arg\\min_{\\eta'} L(\\vec{\\Theta}^k+\\eta'\\vec{d}^k)\n", "$$\n", "\n", "- Adaptive LR optimizers, e.g. Adam\n", "\n", "- LR scheduling\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Zig-zags in narrow \"ravines\"**\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Momentum: Use previous gradients to build \"speed\" in the common direction and cancel-out oscillations in opposite directions.\n", "\n", "- BatchNorm: Normalizes activations to zero-mean and unit variance (reduces curvature)\n", "\n", "- Second-order methods: Use quadratic local approximation of the loss surface, instead of linear.\n", " - Newton's method: $\\vec{d}_k=\\mat{H}_k^{-1}\\vec{g}_k = \\nabla^2 L(\\vec{\\Theta}_k)^{-1}\\nabla L(\\vec{\\Theta}_k)$.\n", " - Quasi-Newton methods which use some estimate of the Hessian based on first-order information (e.g. BFGS)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The back-propagation algorithm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All the above optimization methods have a crucial thing in common: They require calculation of gradients of the loss w.r.t. to the parameters.\n", "\n", "In practical settings when training neural networks we have many different parameters tensors we would like to update separately. Thus, we require the gradient of the loss w.r.t. each of them." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Back-propagation is an efficient way to calculate these gradients using the chain rule.\n", "\n", "We represent the application of a model as a **computation graph**.\n", "For example, a simple linear regression model can be represented as:\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Imagine that in this graph we have $N$ variables $\\vec{v}^i,\\ 1\\leq i \\leq N$ and functions $f_i$ which compute them from other variables.\n", "\n", "The graph is directional, thus assume $\\vec{v}^1, \\vec{v}^2,\\dots,\\vec{v}^N$ represents a topological order of the graph (parents before children).\n", "\n", "Define also the notation $\\delta\\vec{v}\\triangleq \\pderiv{L}{\\vec{v}}$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The forward pass can therefore we written as:\n", "\n", "1. For $i=1,2,\\dots,N$:\n", " 1. Graph parents of current node: $$\\mathcal{P}_i \\leftarrow \\left\\{\\vec{v}^j ~\\middle\\vert~ \\vec{v}^j \\text{ parent of } \\vec{v}^i\\right\\}$$ \n", " 2. Evaluate function at current node: $$\\vec{v}^i\\leftarrow f_i(\\mathcal{P}_i)$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And in the backward pass we traverse the graph in reverse and apply the chain rule:\n", "\n", "1. Set $\\delta\\vec{v}^N=1$.\n", "2. For $i=N,N-1,\\dots,1$:\n", " 1. Graph children of current node: $$\\mathcal{C}_i \\leftarrow \\left\\{\\vec{v}^j ~\\middle\\vert~ \\vec{v}^j \\text{ child of } \\vec{v}^i\\right\\}$$ \n", " 2. Chain rule: $$\\delta\\vec{v}^i\\leftarrow \\sum_{\\vec{v}^j\\in\\mathcal{C}_i} \\delta\\vec{v}^j\\pderiv{\\vec{v}^j}{\\vec{v}^i}$$\n", " \n", "Notes:\n", "1. The expression $\\delta\\vec{v}^j\\pderiv{\\vec{v}^j}{\\vec{v}^i}$ is a \"vector\"-Jacobian product (VJP).\n", "2. When a computation node's output is used by more than one other node (more than one child in the graph), we sum the incoming gradients from these children. This again arises directly from the chain rule." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Backpropagation easily lends itself to a modular and efficient implementation.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Modularity:\n", "- Nodes in the computation graph only need to know how to calculate their own derivatives.\n", "- This is then passed to the parent nodes, which can do the same.\n", "\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Efficiency:\n", "\n", "- Only need to compute each $\\delta\\vec{v}^i$ once.\n", "- No need to construct the Jacobian, instead calculate the VJP directly since that's what we actually need." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Modern automatic-differentiation packages such as PyTorch's `autograd` utilize exactly these tricks to implement backprop in an extremely powerful way." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: Custom automatic differentiation with PyTorch" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll now learn how to extend PyTorch's `autograd` by defining our own custom nodes in the computation graph.\n", "\n", "Lets first introduce a cousin of ReLU, the Exponential-Linear Unit (ELU) activation function:\n", "\n", "$$\n", "f(z) =\n", "\\begin{cases}\n", "z, & z > 0\\\\\n", "\\alpha \\left(e^{z}-1\\right) & z \\leq 0\n", "\\end{cases}\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll pretend PyTorch does not include this activation function and implement a custom version ourselves." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:31.991084Z", "iopub.status.busy": "2020-11-24T17:30:31.990585Z", "iopub.status.idle": "2020-11-24T17:30:32.013783Z", "shell.execute_reply": "2020-11-24T17:30:32.014346Z" } }, "outputs": [], "source": [ "import torch\n", "import torch.autograd as autograd\n", "import torchviz\n", "\n", "from torch import Tensor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we'll implement just the actual computation as a standalone function so that we can reuse it later." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.017761Z", "iopub.status.busy": "2020-11-24T17:30:32.017282Z", "iopub.status.idle": "2020-11-24T17:30:32.037961Z", "shell.execute_reply": "2020-11-24T17:30:32.037375Z" } }, "outputs": [], "source": [ "def elu_forward(z: Tensor, alpha: float):\n", " elu_positive = z\n", " elu_negative = alpha * (torch.exp(z) - 1)\n", " elu_output = torch.where(z>0, elu_positive, elu_negative)\n", " return elu_output" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A quick visualization to see what it looks like:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.041753Z", "iopub.status.busy": "2020-11-24T17:30:32.041265Z", "iopub.status.idle": "2020-11-24T17:30:32.189164Z", "shell.execute_reply": "2020-11-24T17:30:32.189625Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAD8CAYAAAC8TPVwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqG0lEQVR4nO3deXiU1fnG8e/JZBKWsAeDBhABxRZRMMF9Ca3W9deq2LohqEBYBDegitpqK7VVEESoRQQElwruigtYtXGlCkFEFFRAVtkhkGCWWc7vj5kMSQhkBmbN3J/rmmuSd2beeQ6TPLw5c+Z+jbUWERFJDCmxLkBERIKnpi0ikkDUtEVEEoiatohIAlHTFhFJIKmR3HlmZqbt0KFDJJ8iIvbu3Uvjxo1jXUZUJduYk228oDEnksLCwu3W2ta13RbRpt2hQwcWLVoUyaeIiIKCAvLy8mJdRlQl25iTbbygMScSY8zaA92m6RERkQSipi0ikkDUtEVEEoiatohIAlHTFhEJsx0l5RHbd52rR4wx9wP31di8xVrb5nCffM+ePWzduhWXy3W4uwqrZs2asXz58liXEVWVY3Y6nRxxxBE0bdo01iWJJKT532zm9jlLGPf7k7i425Fh33+wS/6+A/KqfO853Cfes2cPW7ZsITs7m4YNG2KMOdxdhk1xcTFNmjSJdRlRVVxcTEZGBqWlpWzcuBFAjVskRMs27ua22UvwuMoY+txiRl3QhaF5ncLa34KdHnFbazdXuWw73CfeunUr2dnZNGrUKK4adjIzxtCoUSOys7PZunVrrMsRSSibd5fRf9ZCWro280H6CH6X8glj53/HyBeXUuH2hu15gj3S7miM2QhUAJ8Dd1trV9d2R2NMPpAPkJWVRUFBQa07bNasGW63m+Li4pCLjjSPxxOXdUVS1TFbaykuLj7ga1cflJSU1Ovx1UZjjpxyt+XBL8rYu6eEl9LG0dZs50rHR7zhPYOV6zfxycc7SQnTwWkwTftz4AZgBXAEcC/wmTGmq7V2R807W2unAlMBcnNz7YE+jbR8+fK4/fM7WadHqo65QYMG9OjRI4YVRVaiflLucGjMkeH1WgY/W8iGPcU86ZzM8SnrWek9iptdt9KlTTOeG3IGGenh+/B5nXuy1r5T9XtjzP+A1UA/YHzYKhERSUAPz/+Od7/dwn2pz/IrxxJ22gxuco0iLaMl02/oGdaGDYeQPWKtLTHGfAMcG9ZKREQSzAuL1jPlw1X0cfyHG1PnU2EdDKq4gy2OI5nTL5fs5g3D/pwhr9M2xjQAjgc2hb0aCckPP/xAVlYWu3fvrvO+5eXltG/fPiEDvETi0YJVO7j7la85J+Ur7k+dBcBdroEstMcz/g/d6d6ueUSet86mbYwZZ4w51xhzjDHmVOAloDEwKyIVxbkbbrgBYwzGGFJTU2nfvj1Dhgxh165dQe+jQ4cOjBs3rtbbjDG89NJLtT7vpZdeWm3b3XffzdChQ2nWrFmdz5mens6oUaO48847g65TRGr34/a9DH62kGPseiY7HyPVeJnkvoxXvOcw8jfHccmJ4V+fXSmYI+22wPP41mq/ApQDp1lrDxgdWN+dd955bNq0iTVr1jBt2jTmzp3L0KFDo1rD+vXree2117jxxhuDfsx1113HJ598wjfffBPBykTqt6KfK7hp5kIcpTuY4RxLU1PKW55TGO++kit6ZHNzr84Rff5g3oi8OqIV+HW4661oPM1BrfnHJUHdLz09nTZtfB8Ibdu2LVdddRUzZ84M3P7UU08xduxYVq9eHTgSv/XWW0lJCV9qwJw5c+jWrRvt27cPbOvQoQNr1+7/f+mPP/5Ihw4daNmyJWeeeSbPP/88Y8aMCVstIsmiwu1l8LOF/LR9F8+ljaddyjaWeDsywjWE3A6t+HvvbhH/3ElET4KQDFavXs28efNwOp0APPnkk/z5z39m0qRJ5OTksGzZMgYOHIjT6WTYsGFhe96PP/6Y3NzcatsWLlyIx7Pvw6oDBw5k5cqVZGVlBbadcsopfPjhh2GrQyRZWGu597Wv+d/qHTzqnEpuyvdstK0YWDGSI1q24Inrc0lPdUS8DjXtQzBv3jwyMjLweDyUlZUBMH68b/XjAw88wMMPP8yVV14JwDHHHMNdd93F448/HtamvXbtWrp3715tW+vW+85O9NBDD7FgwQI+//xzGjbc9w72UUcdxZo1a8JWh0iymPrRal5YtIHhjle5zPEZJbYBAypGUtYgk+dvyKVl47So1KGmfQjOOeccpk6dSmlpKU8++SSrVq3illtuYdu2baxfv55BgwYxZMiQwP3dbjfW2rDWUFpaSoMGDWq9be7cudx3333Mnz+fTp06VbutYcOGlJaWhrUWkfpu/jeb+ce8FVyasoARzpfwWMMtrmF8bzow67ocOh8RvQ/jqWkfgkaNGtG5s+/Nhscee4xevXrxwAMPBBr1lClTOOOMMw5p302aNKl1CV9RUVG1VSKZmZm1rlhZtmwZ1113HZMnT+bcc8/d7/adO3dWOyIXkYOrDIHqzg884pwCwN/cffjAezJjLuvKWcdmRrWeuGnawb4JGI/uu+8+LrroIvLz88nOzmbVqlX07dv3kPbVpUsXCgsL6d+/f2Cbx+Phq6++qrZSpEePHnz77bfVHrt9+3Z++9vfMnDgQAYMGFDr/pctW8bJJ598SLWJJJuqIVBT0x8h3bh4zv1rZngu5KYzj6HPaUdHvaa4adqJLC8vj65duzJmzBjuv/9+hg8fTvPmzbn44otxuVwsXryYjRs3Mnr06MBjfvrpJ5YsWVJtP23btuWOO+7gxhtvpGvXrpx//vn8/PPPTJo0iZ07d5Kfnx+47wUXXMCNN96I2+0mNdX3Mvbu3ZujjjqKESNGsHnz5sB9W7dujcPhe4Pk448/5oEHHojgv4ZI/fBzhZv+sxayd88uXkobR2uzh4883bjP3Y9fH5/FPZf8IiZ1qWmHSWWzvfPOO5kxYwZjx45l9OjRNGzYkK5du+73JuSECROYMGFCtW2TJk0K3O+RRx5h9OjRNGrUiJycHD7++OPAMkOAiy++mIYNGzJ//nwuucT3V8pHH30EQHZ2drX9Vi75W7BgAbt37w68SSoitfN6LbfNXsKKn3YxzTkpEAI1zHULndu0YOI1PXCkxCZSWk07RFXXY1d17bXXcu211wJw9NFHc8011xxwH3Wt3rjmmmsO+ngAh8PBPffcw/jx4wNNu643O8ePH8+oUaOqrSYRkf09NH9FIASql+OriIdAhUJNO4ENHDiQnTt3snv37jo/yl5eXs5JJ53E7bffHqXqRBLTnIXreOLD1YEQqHKbSn6EQ6BCoaadwBwOB3fffXdQ901PT+fee++NcEUiiW3Bqh3c8+qy/UKgFtnj+WcEQ6BCobOxi4gAq7eV1BoC9ar37IiHQIVCTVtEkl7RzxX0n7WI1NLtgRCoNz2n+kKgTo58CFQoND0iIkmtagjUvwMhUJ32hUBdEfkQqFCoaYtI0qoZApWT8oM/BGoEWS2bRy0EKhRq2iKStJ7wh0DdUiUEqn/FKH8IVM+ohUCFQk1bRJLSvGWbeWjeCv4v5TPu8IdADXcN5wdztD8EKiPWJdZKTVtEks6yjbu5fY4vBGqc8wnAFwL1X28P/nZ59EOgQqHVI3FKJ+0ViYzKEKhW7s1MTfOFQD3rD4Hqf9YxXHdq9EOgQqGmHaKqJ/atejnttNMAnbRXJJ5VDYGa5twXAnW/PwTq7otjEwIVCjXtQ1B5Yt+ql7fffjts+9dJe0XCr2oI1CR/CNQP3uy4CIEKhZr2Iag8sW/VS8uWLcO2/wOdtLe2I/zK8KmqJ+0Vkf09NM8XAnWvPwRqh23CTa6RpDdpyYwYh0CFIr6qvL/uqYDIPG/d88bRpJP2ioTXnIXreOKj1VzveDcQAjWo4na2Oo7khb65HBXjEKhQ6Ej7EFSe2LfqJZzzyWvXruXII6vnHLRu3TpwVD9r1iwWLFjAm2++qZP2itThs1XbuefVZZxbJQTqTlc+i+zxTLiqOyfFQQhUKOLsSDu+jngPpPLEvlU1b948bPvXSXtFwmPzXi9/f3Yxx9j1TEp7DIexPOa+jNe8ZzHqgi5c3C0+QqBCEV9NO0FUPbFvKHTSXpHo2bW3ggmFZaSWFjEjrTIE6jQm+EOghuZ1qnsncSjk6RFjzN3GGGuMmRyJguqzypP2VlV50t4uXboEtumkvSKHpzIEqujncqZWC4EaTM8OmXEXAhWKkI60jTGnAQOBpZEpJzGUl5dXO3Eu+E5IUHmUq5P2isROZQjU5z/uYKI/BGqDzWRgxQjatGrOlOtz4i4EKhRBN21jTDPgOaA/8OeIVZQA3nvvvf3eKMzOzmbDhg2ATtorEkuVIVC3Ol7hd/4QqAEVI30hUP3iMwQqFKauk8EG7mjMHGCNtfZOY0wBsMxaO6yW++UD+QBZWVk5s2fPrnV/zZo1O6R54WjweDyBo9dYmT59Oq+99hpz584N6v59+/blxBNPZOTIkYf0fDXHvHLlyqA+Qp+oSkpKyMiIz0CgSEmGMRducTP5y3L+L+UzHkubjMcaBrhG8qHtwYicBnTNTIwj7F69ehVaa3Nruy2oI21jzECgM3B9Xfe11k4FpgLk5ubavLy8Wu+3fPlymjRpEszTR11xcXHMaxs+fDg///wzXq83qJP25uTkMGLEiEM+03rNMTdo0IAePXoc0r4SQUFBAQf62ayv6vuYv96wmyff/4we5nvG+kOgxvhDoB68vBvXntq+jj0khjqbtjGmC/AgcLa1tiLyJQnopL0iodi8u4wBTy8k072FqWnjSTcunnGfx1OeCxlw1jH1pmFDcEfapwOZwLIq77Y6gHOMMYOBxtba8gjVJyJyUHvL94VAvZw2lkx/CNRf3H3p3jqV0QkQAhWKYJr2a0DNzM+ngB/wHYHr6FtEYsLrtdw2xxcCNd05iS4pGwIhUMce2ZLBXd0JEQIVijqbtrW2CCiqus0YsxfYaa1ddjhPbq1N2LWS9Vmwb06LxNpD81bwn2+3cH/qM+TVCIGa3i+X75d8HusSwy5m2SNOp1MfuY5TpaWlOJ3OWJchclCzv/CFQPV1zOeG1Hcpt6nkV9zBttQjmZZgIVChOKSPsVtr8w73iY844gg2btxIdnY2DRs21BF3HLDWUlpaysaNG6ulB4rEm89Wbefe13whUPelPg3AH135FNouPP6HxAuBCkXMskeaNm0K+D496HK5YlVGrcrKyg4Y2FRfVY7Z6XSSlZUVeH1E4s3qbSUMeXYxHe06JvtDoCa6L+f1BA6BCkVMA6OaNm0al82hoKCgXq9Rrk0yjlkSz669Fdw0cyGppduZkT6WJv4QqEfdvel9ctuEDYEKhVL+RCQhVIZAbdpRxL/TxtPWbOdLb+dACNSDV5yQFNOsatoiEvestdzz6v4hUPkVd9SLEKhQqGmLSNyb8uFqXizcPwSqvJ6EQIVCTVtE4tq8ZZt4aN4KfpvyGbc7X8ZjDcNcw/nBHM3TfXLofET9DsGqSU1bROLW1xt2c9ucJZxcJQTqAff1FHh78ODlJ3Bm58wYVxh9OrGviMSlTbtL6T9r/xComZ4L6l0IVCh0pC0icWdvuZv+MxdRWryLZ6qEQN3v7sd5v8iqdyFQoVDTFpG44vGHQH23aRcznI/RJWUD33uzudl1K8cd2YKJV/eodyFQodD0iIjElcoQqD+lPsO5jqXssE3o7xpJwyYtmN4vl8bpyX2smdyjF5G4MvuLdUw9QAjUC/3qbwhUKNS0RSQufLby4CFQJ7ZtHtsC44SmR0Qk5lZtK2Hws4W+EChn8oVAhUJNW0RiatfeCvrPXEha2Q5mpPlCoOZ6TmOC+8qkCYEKhaZHRCRmqoZAPZ/2SCAEaqRrMKd0aJU0IVChUNMWkZiw1nJ3lRCok1NWssFmMrBiRNKFQIVCTVtEYmLKh6t5qXADt6W+zO8cn1FsG9K/YiQVDVoxO8lCoEKhpi0iUbcvBOpTbkt9BY81DHcNY5U5mllJGAIVCjVtEYmqpRuKqoRATQX2hUD9/YrkDIEKhVaPiEjUbNpdyoBZi8h0bw6EQD3tPp+ZngsYePYxXHNKcoZAhUJH2iISFVVDoJ6tEgL1F3dfzvtFG+66KHlDoEKhpi0iEefxWm6dvS8E6riUjYEQqC5HtmDi1d2TOgQqFJoeEZGIe2jeCt5bvoU/pz7NuY6lbLdNuck1yhcCdYNCoEKhfykRiaiqIVD9Uv8TCIHantqGF/rlcmQzhUCFos4jbWPMzcaYpcaYPf7LAmPMJdEoTkQS26f+EKi8lCXVQqAW2+N49CqFQB2KYKZHNgB3AicDucAHwGvGmBMjWZiIJLZV20oY4g+BmuSc5A+BuoLXvWfxxwu7cOEJCoE6FHVOj1hrX6+x6R5jzBDgdGBpRKoSkYS2a28FN1WGQKX7QqDe8JzOBHdvep/cliHnKgTqUBlrbfB3NsYB/B54Gsix1n5dy33ygXyArKysnNmzZ4ep1OgpKSkhIyO5PpGVbGNOtvFC9Mbs8lrGLSxjza4ynk8bw8kpK1ns7cw1FffSoUUDRvVsQGqUVook6uvcq1evQmttbm23BfVGpDGmG7AAaACUAJfX1rABrLVTgakAubm5Ni8v71BqjqmCggISse7DkWxjTrbxQnTGbK1l5ItL+W7Xeh5zPhEIgcr3h0DNHnomLaKYKVIfX+dgV498B3QHmgO9gVnGmDxr7bII1SUiCehfH67i5cW+EKjfOhZQbBtyU8UoKhq0Ys4NPaPasOuroJq2tbYCWOn/dpExpidwO9A/UoWJSGJ55+tNPDzvuxohUMNZbdrzdJ8cOrVOvGmKeHSo67RTgPRwFiIiiWvphiJuf6EyBOoJAP7q7kuBtzv/uOIEzlAIVNjU2bSNMf8A3gLWA02Aa4E8QGu1RaSWECg3s9znM8sfAnW1QqDCKpgj7TbAs/7r3fiW+V1krZ0fycJEJP7VFgL1oedE/uruy3m/yFIIVAQEs077hijUISIJpmoI1FPOiYEQqGGuWxQCFUEKjBKRQ/KPd5YHQqDOcXytEKgo0b+qiITs+S/W8eTHP9IvEALlDIRAvdivp0KgIkhNW0RC8unK7fzptWXkpXzJn/0hUKP8IVBTrupOt7bNYlxh/abpEREJ2sqtvhCoTnYtk5yTcRjLo+4reMN7pkKgokRNW0SCsmtvBf1n+UKgpqeNC4RAPeruze9zFAIVLZoeEZE6lbs9DHq2kM07ipid9ghtzXYWezszyjWIU45pxd8u74YxWikSDWraInJQ1lrufmUZX/y4g0nOKfSoEgJ1ZKvmPNEnh7RU/dEeLfqXFpGDqgyBuj31Zf7P8b9qIVDTFQIVdTrSFpEDqgyB+l3KJ9yqEKi4oKYtIrWqDIHKMd/xsHMqoBCoeKDpERHZz09FpfT3h0A9kTahWghU/jkdFQIVQzrSFpFq9pa7GTBrEWXFu3iuRgjU+b/M4s4Lj491iUlNTVtEAnwhUF8eMATq0asUAhVratoiElAZAvVXhUDFLb0CIgLsC4G6wTGfvgqBiltq2iJSLQTqT6nPAAqBildaPSKS5FZuLWHwAUKg7rzweIVAxRk1bZEkttMfAtWgbHsgBOp1zxmBEKjB53aMdYlSg6ZHRJJUudvD4GcqQ6DGB0Kg/ujK51SFQMUtNW2RJFQZArVwzXYeqyUEaopCoOKWXhWRJPR4gS8E6jaFQCUcHWmLJJm3v97E2PmVIVCv4rGGYa5bfCFQ1ysEKt6paYskka/WF3FHjRCov7j78qH3JF8IVCeFQMU7TY+IJIkdpV4GPL2I1u5NTE0bT7pxM9P9G55WCFRCqbNpG2NGG2MWGmP2GGO2GWPmGmNOiEZxIhIee8vdPLq4nLLiXcxwjqOVKabAcxIPuK9XCFSCCeZIOw94HDgD+BXgBt4zxrSMYF0iEiaVIVA/Fbv4p3Mix6Zs5DtvW4a7hnP8US2YeLVCoBJJnXPa1toLqn5vjLke2A2cCcyNUF0iEiZ/f9sXAvVA6qxACFR/1ygaNW3BtH65NErTW1uJ5FBerSb4jtB3hbkWEQmzf3++jmmf+EKgrk99LxACtSO1DS/0VQhUIjLW2tAeYMwLwLFArrXWU8vt+UA+QFZWVs7s2bPDUWdUlZSUkJGRXMuekm3MyTDeb7Z7eKSwjHPNl0xzjsNhLLdUDOMN7xkM75FOTlb9P8JO1Ne5V69ehdba3NpuC6lpG2PGA1cDZ1lrV9d1/9zcXLto0aKg9x8vCgoKyMvLi3UZUZVsY67v4125tYTLH/+U7PLVvJR2PxmmjAmu3kz09ObOC49nSF6nWJcYFYn6OhtjDti0g/6v1hgzAV/D7hVMwxaR2Ni5t4KbZvpCoKaljyPDlPG65wwmeq7gD7kKgUp0QTVtY8xEfA07z1q7IrIlicihqgyB2rJzXwhUofdY/ujKp0sLB2MuUwhUoquzaRtj/glcD1wG7DLGtPHfVGKtLYlgbSISAmsto1/5moVrtjPJHwK13tua/Io7OCqzBcNPtAqBqgeCeQWH4lsx8j6wqcplZATrEpEQPV6wilcWb+S21Je51B8C1d81EnfDTKb3yyUjTUfY9UEw67T1SovEucoQqMtqC4HqczIdW2ewLtZFSljU/zU/IvXcV+uLuH3OEnLNCh7yh0Dd7+7Hh96TeKi3QqDqG01wiSSwjUWlDHh6EUd4NvFE2oRACNQznt8w6JyOXNVTIVD1jY60RRJUSbmb/jMXUl68k3+n+UKg/usPgfqNQqDqLTVtkQTk8Vpuff5LVm7exQznY/uFQD16dXdSFAJVL6lpiySgB99ezvsrtjDGHwK1zR8C1bhpC6b366kQqHpMr6xIgnnu87VM/+RHbnTMo0/q+5RbJ4P8IVAv9utJm2YNYl2iRJCatkgC+eSH7fz59W/olfIl96Y+C8BI1yC+5Dj+dVV3TshuFuMKJdK0ekQkQazcWsyQ5wo51q5lknMSDmOZ4OrNXO8Z3Hnh8Vx4Qpu6dyIJT01bJAH4QqAW0aBsO9PTxpJhynitSgjUoHMUApUsND0iEufK3R4GPbOILTuLmJP2CNlmB4XeY7nTlc9pHVspBCrJqGmLxDFrLaNf/ppFa3YwyfkvuqesqhYCNaVPjkKgkoxebZE49njBKl75ciO3p77EpY7P2WMbcpNrVCAEqnmjtFiXKFGmI22ROPXWUl8I1OUpH3NL6muBEKgfTbtACJQkHzVtkTj01foi7njBFwL1D+eTgC8E6iPvSTzcu5tCoJKYpkdE4kxtIVBPuS/whUCd25E/9GwX6xIlhnSkLRJHqoZAPZ82NhACNcbdxxcCdYFCoJKdmrZInKgZAtU55SdWeNsx3DWcX2QrBEp81LRF4kStIVAVI2nctAXT+ioESnz0UyASBypDoG6qEgKVXzGCnU6FQEl1atoiMVYZAvWrlMXVQqCWcCxTrlYIlFSn1SMiMVQ1BOox52RSjGW860rmes/grguP54KuCoGS6tS0RWJkXwjUtmohUI95LucPuW3JVwiU1ELTIyIxUFsI1CLvcdzpyuf0jpkKgZIDUtMWibL9Q6BWs97bmkEVt3NUZgv+1edkhUDJAeknQyTK/vnflbzy5UbuqCUEasYNPRUCJQcVVNM2xpxjjHnDGLPRGGONMTdEuC6ReumtpZsY9+73XJ7yMcNTX8NtUwIhUFP65HBMZuNYlyhxLtgj7QxgGXArUBq5ckTqryUHCYF68PJunN6pVYwrlEQQ1Jy2tfZt4G0AY8zMSBYkUh9tLCplwKxFZHk2MTVtfCAE6lnP+QqBkpDojUiRCKsMgaoo2cnstLG0NCUKgZJDZqy1oT3AmBJgmLV25gFuzwfyAbKysnJmz559uDVGXUlJCRkZyRUwn2xjjtZ4vdYycXE532wr5ynnw5ztWMYKbzuurLiPVk0zuPuUBqSnRmdpX7K9xpC4Y+7Vq1ehtTa3ttvCfqRtrZ0KTAXIzc21eXl54X6KiCsoKCAR6z4cyTbmaI33r3O/5attq/lb6izOdiyrFgI15+azopopkmyvMdTPMWt6RCRCnv3fWmZ86guBuk4hUBImatoiEfDxD9u4743qIVAjXIMVAiWHLaimbYzJADr7v00B2htjugM7rbXrIlSbSEJaubWYoc8t5ji7hklpk0gxlkdcV/Km93RGX6QQKDk8wa7TzgW+9F8aAn/xf/3XCNUlkpB2lJRz48yFNCjbxrS0cTQ25bzqOZNJnsu5KredQqDksAW7TrsAUHqNyEGUuz0MfraQbTuLmF0lBOou10BO75jJA5edoBAoOWya0xYJA2std/lDoCb7Q6DWeVuTX3EH2QqBkjDST5FIGPzzvyt51R8CdYnji0AIlKdhK6YrBErCSEfaIofpzaU/Me7d77ki5aNACNTNrltZm9KOpxUCJWGmpi1yGL5ct4sRL3xFzxohUB97T+ThKxUCJeGn6RGRQ7SxqJSBTxeS5dnEE2njSTOeQAjU4HM78YdchUBJ+OlIW+QQ1BYC9YGnOw+4r+eCrln88YIusS5R6ik1bZEQebyWW57/kpWbd/GUcyKdU35iubcdw13D+WV2cyZc1Z2UFC3tk8hQ0xYJ0d/eWs4HK7ZUCYFqxoCKkTRp2oLp/XrSKE2/VhI5+ukSCUFlCFR/xztcl/o+ZdbJwEAIVC5ZTRUCJZGlpi0SpI++94VA/TqlkHtSnwNghGsIX9GZJxQCJVGi1SMiQfhhSzE3+0OgHnNODoRAveU9jdEXHc9vFAIlUaKmLVKHHSXl3DRrIQ3KtzE9bSyNTTmveM4KhEANPFshUBI9mh4ROYhyt4dBz+wLgTrK7GShQqAkhtS0RQ6gMgSqcG31EKhBFXfQNrM5U/rkKARKok4/cSIHMPkDXwjUiNQXq4VAeRv5QqCaNXLGukRJQjrSFqnFm0t/4pH/+EKghqW+Xi0E6hmFQEkMqWmL1FBXCNRpHRUCJbGj6RGRKjbs+pmBTxfSxvNTIARqhvtChUBJ3NCRtohfcZmLAbMWUVGygzn+EKj3PT0Y4+7DhV3bKARK4oKatgjVQ6BmOifSKWUTy73tuMU1jK7ZLRh/1UkKgZK4oKYtAox561v++91WHkydyVmOb9hmm9G/YhRNmrZgWr9chUBJ3NBPoiS9Z/63lqc+XUN/xztcm/pBIARqlzNLIVASd9S0Jal99P027lcIlCQQrR6RpFUZAtXF/hgIgRrn+r1CoCSuqWlLUtpTYQMhUNPSxgVCoCZ7LuPqngqBkvgVdNM2xgw1xvxojCkzxhQaY86OZGEikVLu9jBpcRnbdhYxrUYI1BmdFAIl8S2oOW1jzFXARGAo8In/+h1jzC+ttevCVczW4jLeWPJTuHZ3yFb+6GKlY3Wsy4iqZBmztTDzszX8VOTmn85/cVLKatZ6jwiEQP3ruhycDv0BKvEr2Dci7wBmWmuf9H8/3BhzITAEGB2uYjYVlTHmreXh2t3h+S5O6oimJBrzqNQXuNjxBXtso0AI1AyFQEkCqLNpG2PSgBxgXI2b3gXOiERRIpHSlBJuT32ZG1Pn47YpDHXdyrqUtjzTJ4cOCoGSBBDMkXYm4AC21Ni+BTiv5p2NMflAPkBWVhYFBQVBF7N6tyfo+4qEwomb6xzvcWvqK7QwJXit4S73QD7xdqP/CU7K1n1NQdgm+uJTSUlJSL+P9UF9HHMo67Rtje9NLduw1k4FpgLk5ubavLy8oJ+gxfoiWPBpCCWJHFwqbi53fMLNjtfpkOI77vif9xeMcV3HMtuRO84/jlt+fWyMq4yOgoICQvl9rA/q45iDadrbAQ9Qc9HqEex/9H1YMpuk0/+sY8K5y0OyYf162rZLrjS3+jbmFK+Lk3a8zZmbZtGiYhMA2xu05/22w/i+2Vkcu2Ujf7vgFE5q1zy2hYqEqM6mba2tMMYUAucDL1a56Xzg5XAWk928IX+69Jfh3OUhKSjYSl5e7OuIpnoz5r07YPFM+GIaFPtXIrU6Fs4ZReYJvbnK4fuRLyjYpoYtCSnY6ZHxwDPGmC+AT4HBwFHAlEgVJhKSzV/D51Ng6YvgKfdta308nDMKul4OKY7Y1icSJkE1bWvtHGNMK+Be4EhgGXCxtXZtJIsTOaifd8LXL8GS52DTkn3bj/0NnDoYOvaCFK25lvol6DcirbWPA49HsBaRupWXwMr/wLKX4bt54HX5tqc3g5OuhlMHQatOsa1RJIKU8ifxr7QIvp8Py9+Ale+Bu8y33aRA5/Og+7XQ5RJwKkJV6j81bYk/Xg/8tARWvQ8r34cNC8FWWcPftif84rfQ7UpoelTMyhSJBTVtiT2PG7Z+A+v+B+sWwOoCKN2173bjgKPPgl/+Fo6/FJplx6xUkVhT05boK94Cm5fCxkJfk96wCCpKqt+n+dHQ+dfQ6VdwzDnQQCcjEAE1bYkkjwt2rYGt38Kmr2DTUl+zLqnlM1ktOkD706Hdqb4m3bIjKB5VZD9q2nJ4rIW923zNefsPsP172LHSd71zNXjd+z8mvSm06QZHdod2p0D706CJzhIjEgw1bTm48hLfkXHJFti9EYrWwu71ULQOitb7vq5czVGb5u0hswsceSK0OdF33byD1k+LHCI17WRirW/uuHTXfpf2axfB22/5G/TWfdc155pr06C5vzkfC5nH7btu2QnSGkV8WCLJRE07XlnrmxN2l4GnwnftLoeKvVUuxb7r8hJfc63Yu++6vHjfdWVzLiuqfboC6AjwYy03pDaAjCzIOAKaZkPzdr43CZu1833drB00aBrBfwgRqSo+m3Zpka9RWa//Yqt8XWMbtd1WeaH27fs9xlb7OnPbUvh2j+97r9u3btjr8n/t9i1Rq/za6/Ld7qlye9WLp+r9/Ptyl+9rwge73j/59vA5G0PDFv5L88DX67YV0/6XPaFJlr9J+xt1elO9ISgSR+Kzab88wPdR5Rg5AeCbmD39PilO35Fuavq+a2dDSMuAtMa+S3qTfV+nZfgu6Rn7vk9vsq9JN2gOqWm1PtXqggLan54X1eGJSOjis2k3aAaNW/s+pozxXQcupsZ1zYup5TFBPt7/uO07dpLZurXvPimpvuaZkgqOVP/3ldsc4HBW/z4ltcq2KhdHldsrG7AjvXpDrnmtZDoRqSE+m/aV02P69Mvq4dkuRKR+0LorEZEEoqYtIpJA1LRFRBKImraISAJR0xYRSSBq2iIiCURNW0Qkgahpi4gkEGNtBPItKnduzDZgbcSeIHIyge2xLiLKkm3MyTZe0JgTydHW2ta13RDRpp2ojDGLrLW5sa4jmpJtzMk2XtCY6wtNj4iIJBA1bRGRBKKmXbupsS4gBpJtzMk2XtCY6wXNaYuIJBAdaYuIJBA1bRGRBKKmLSKSQNS0g2B85hljrDHmyljXEynGmJbGmEnGmBXGmFJjzHpjzL+MMa1iXVs4GWOGGmN+NMaUGWMKjTFnx7qmSDHGjDbGLDTG7DHGbDPGzDXGnBDruqLFGHO3//d2cqxrCRc17eCMADyxLiIKjgKygT8C3YA+wDnA87EsKpyMMVcBE4EHgR7AZ8A7xpj2MS0scvKAx4EzgF8BbuA9Y0zLWBYVDcaY04CBwNJY1xJOWj1SB2NMLvAqkANsAX5vrX0ptlVFjzHmYuBNoLm1dk+s6zlcxpjPgaXW2oFVtv0AvGStHR27yqLDGJMB7AYus9bOjXU9kWKMaQYsxte0/wwss9YOi21V4aEj7YMwxjTBd5Q5yFq7Ndb1xEhToBz4OdaFHC5jTBq+/3zfrXHTu/iORJNBE3y/97tiXUiETcX3H/EHsS4k3NS0D24KMM9a+3asC4kFY0xz4AHgSWutO8blhEMm4MD3F1NVW4A20S8nJiYCS4AFMa4jYowxA4HOwJ9iXUskJF3TNsaM8b8xcbBLnjHmeuAkYFSsaz5cwY65xmMaA3OBjfjmuOuTmnOCppZt9Y4xZjxwFtDbWlsv36MxxnTB937FddbailjXEwlJN6dtjMnEd8R1MOvwvXnTF/BW2e7wf7/AWntWZCoMv2DHbK392X//DOBtfM3sImttSYRLjAr/9MjPwDXW2herbP8ncIK19tyYFRdhxpgJwNVAL2vtiljXEynGmBuAp6i+cMCB7z9lL9DYWlseg9LCJumadrCMMdlAixqbvwbuAF631q6OflWR55/Hfwdfw77QWlsc45LCyv9G5FfW2vwq274HXq6vb0QaYybia9h51trlsa4nkvxTem1rbH4K+AHfEfg3NsGbXmqsC4hX1tqN+KYGAowxAOvrecN+F9+bj5cBjf3TJAA768mfm+OBZ4wxXwCfAoPxLXWcEtOqIsT/V8T1+F7PXcaYyrn7kvryF1RV1toioKjqNmPMXnw/v8tiUVO4qWlLVTnAaf6vv69xWy+gIKrVRIC1do7/w0L3AkcCy4CLrbWJeIalYAz1X79fY/tfgPujW4qEg6ZHREQSSNKtHhERSWRq2iIiCURNW0Qkgahpi4gkEDVtEZEEoqYtIpJA1LRFRBKImraISAL5f8LVv75fLP2UAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "z = torch.linspace(-5, 5, steps=1000)\n", "plt.plot(z.numpy(), torch.relu(z).numpy(), label='ReLU(z)', linewidth=5);\n", "plt.plot(z.numpy(), elu_forward(z, alpha=0.5).numpy(), label='ELU(z)', linewidth=2); plt.legend(); plt.grid();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we'll wrap it as an `nn.Module` so that we can use it as a layer in a model." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.194035Z", "iopub.status.busy": "2020-11-24T17:30:32.193525Z", "iopub.status.idle": "2020-11-24T17:30:32.213572Z", "shell.execute_reply": "2020-11-24T17:30:32.214100Z" } }, "outputs": [], "source": [ "class ELU(torch.nn.Module):\n", " \"\"\" ELU Activation layer \"\"\"\n", " \n", " def __init__(self, alpha: float = 0.1):\n", " super().__init__()\n", " self.alpha = alpha\n", " \n", " def forward(self, z: Tensor):\n", " return elu_forward(z, self.alpha)\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And as usual, we can look at the resulting computation graph." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.217644Z", "iopub.status.busy": "2020-11-24T17:30:32.217164Z", "iopub.status.idle": "2020-11-24T17:30:32.268048Z", "shell.execute_reply": "2020-11-24T17:30:32.268840Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "%3\n", "\n", "\n", "\n", "140227913363360\n", "\n", "SWhereBackward\n", "\n", "\n", "\n", "140227913363120\n", "\n", "z\n", " (6)\n", "\n", "\n", "\n", "140227913363120->140227913363360\n", "\n", "\n", "\n", "\n", "\n", "140227912065280\n", "\n", "ExpBackward\n", "\n", "\n", "\n", "140227913363120->140227912065280\n", "\n", "\n", "\n", "\n", "\n", "140227913363264\n", "\n", "MulBackward0\n", "\n", "\n", "\n", "140227913363264->140227913363360\n", "\n", "\n", "\n", "\n", "\n", "140227912065184\n", "\n", "SubBackward0\n", "\n", "\n", "\n", "140227912065184->140227913363264\n", "\n", "\n", "\n", "\n", "\n", "140227912065280->140227912065184\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "elu = ELU(alpha=0.5)\n", "z = torch.tensor([-2., -1, 0, 1, 2, 3], requires_grad=True)\n", "torchviz.make_dot(elu(z), params=dict(z=z))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the computation graph accurately represents the various basic mathematical operations performed bby our `elu_forward` function.\n", "\n", "But what if we want to define the entire ELU operarion as one node in the graph?\n", "This can be useful e.g. for performance reasons.\n", "\n", "But how can we accomplish this?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The solution is to use a lower-level PyTorch API, `autograd.Function`\n", "which allows us to define a function in terms of both it's forwards pass\n", "(the regular output computation), and it's **backward** pass\n", "(the gradient w.r.t. all it's inputs).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From the PyTorch docs:\n", " \n", " Every operation performed on Tensor s creates a new Function object, that performs the computation, and records that it happened. The history is retained in the form of a DAG of functions, with edges denoting data dependencies (input <- output). Then, when backward is called, the graph is processed in the topological ordering, by calling backward() methods of each Function object, and passing returned gradients on to next Function s.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll first calculate the simple analytic derivative of the ELU function:\n", "$$\n", "\\pderiv{f(z)}{z} = f'(z) = \n", "\\begin{cases}\n", "1, & z > 0\\\\\n", "\\alpha e^{z} & z \\leq 0\n", "\\end{cases}\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we need to figure out how to compute the vector-Jacobian product efficiently.\n", "Note that for any **elementwise** operation, $\\vec{y}=f(\\vec{x}),\\ f:\\set{R}^n\\rightarrow\\set{R}^n$, we can write the Jacobian as\n", "\n", "$$\n", "\\pderiv{\\vec{y}}{\\vec{x}} = \\pmatrix{\n", "\\ddots & \\vdots & \\\\\n", "\\cdots & \\pderiv{y_i}{x_j} & \\cdots \\\\\n", "& \\vdots & \\ddots\\\\\n", "}\n", "=\n", "\\pmatrix{\n", "f'(x_1) & & \\\\\n", " & f'(x_i) & \\\\\n", "& & f'(x_n)\\\\\n", "}\n", "= \\diag\\{{f'(\\vec{x})}\\}\n", "$$\n", "\n", "And it follows that the VJP can be computed simply:\n", "$$\n", "\\delta \\vec{x} = \\delta{\\vec{y}}\\pderiv{\\vec{y}}{\\vec{x}} = \\delta{\\vec{y}} \\odot f'(\\vec{x}).\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, equipped with the expression for the VJP, we can proceed to implement the Function object representing ELU." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.274575Z", "iopub.status.busy": "2020-11-24T17:30:32.274033Z", "iopub.status.idle": "2020-11-24T17:30:32.296611Z", "shell.execute_reply": "2020-11-24T17:30:32.297273Z" } }, "outputs": [], "source": [ "class ELUFunction(autograd.Function):\n", " \n", " @staticmethod\n", " def forward(ctx, z: Tensor, alpha: float):\n", " elu = elu_forward(z, alpha) # Regular forward pass computation from before\n", " ctx.save_for_backward(z) # Tensors should be saved using this method\n", " ctx.alpha = alpha # other properties can bbe saved like so\n", " return elu\n", " \n", " @staticmethod\n", " def backward(ctx, grad_output):\n", " z, = ctx.saved_tensors\n", " alpha = ctx.alpha\n", " \n", " # Calculate diagonal of d(elu(z))/dz\n", " grad_positive = torch.ones_like(z)\n", " grad_negative = alpha * torch.exp(z)\n", " \n", " # Note: This is not the full Jacobian\n", " grad_elu = torch.where(z>0, grad_positive, grad_negative)\n", " \n", " # Gradient of the loss w.r.t. our output\n", " δ_elu = grad_output\n", " \n", " # Calcualte δz = d(elu(z))/dz * δ_elu\n", " # Note: elementwise multiplication equivalant to vector-Jacobian product\n", "# print(f'{grad_elu.shape=}, {δ_elu.shape=}')\n", " δz = grad_elu * δ_elu\n", " return δz, None" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now use this custom `Function` either directly or as part of a layer.\n", "\n", "For example, here's an ELU layer using our custom backward:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.300981Z", "iopub.status.busy": "2020-11-24T17:30:32.300497Z", "iopub.status.idle": "2020-11-24T17:30:32.321371Z", "shell.execute_reply": "2020-11-24T17:30:32.321741Z" } }, "outputs": [], "source": [ "class ELUCustom(torch.nn.Module):\n", " \"\"\" ELU Layer with a custom backward pass \"\"\"\n", " \n", " def __init__(self, alpha: float = 0.1):\n", " super().__init__()\n", " self.alpha = alpha\n", " \n", " def forward(self, z: Tensor):\n", " # Function.apply() invokes the forward pass\n", " return ELUFunction.apply(z, self.alpha)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.325307Z", "iopub.status.busy": "2020-11-24T17:30:32.324799Z", "iopub.status.idle": "2020-11-24T17:30:32.376217Z", "shell.execute_reply": "2020-11-24T17:30:32.376831Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "%3\n", "\n", "\n", "\n", "140227913299024\n", "\n", "ELUFunctionBackward\n", "\n", "\n", "\n", "140227913494976\n", "\n", "z\n", " (6)\n", "\n", "\n", "\n", "140227913494976->140227913299024\n", "\n", "\n", "\n", "\n", "\n", "140227913442240\n", "\n", "(6)\n", "\n", "\n", "\n", "140227913442240->140227913299024\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "elu_custom = ELUCustom(alpha=0.5)\n", "z = torch.tensor([-2., -1, 0, 1, 2, 3], requires_grad=True)\n", "torchviz.make_dot(elu_custom(z), params=dict(z=z),)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This only tested the forward pass. Let's now put our custom layer in the context of a larger model and see that we can backprop through it." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.380989Z", "iopub.status.busy": "2020-11-24T17:30:32.380493Z", "iopub.status.idle": "2020-11-24T17:30:32.413097Z", "shell.execute_reply": "2020-11-24T17:30:32.413678Z" } }, "outputs": [], "source": [ "elu_mlp = torch.nn.Sequential(\n", " torch.nn.Linear(in_features=512, out_features=1024),\n", " ELUCustom(alpha=0.01),\n", " torch.nn.Linear(in_features=1024, out_features=1024),\n", " ELUCustom(alpha=0.01),\n", " torch.nn.Linear(in_features=1024, out_features=10),\n", " torch.nn.Softmax(dim=1)\n", ")" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.416976Z", "iopub.status.busy": "2020-11-24T17:30:32.416501Z", "iopub.status.idle": "2020-11-24T17:30:32.470751Z", "shell.execute_reply": "2020-11-24T17:30:32.471264Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "%3\n", "\n", "\n", "\n", "140227913495408\n", "\n", "MeanBackward0\n", "\n", "\n", "\n", "140227913496032\n", "\n", "SoftmaxBackward\n", "\n", "\n", "\n", "140227913496032->140227913495408\n", "\n", "\n", "\n", "\n", "\n", "140227913496992\n", "\n", "AddmmBackward\n", "\n", "\n", "\n", "140227913496992->140227913496032\n", "\n", "\n", "\n", "\n", "\n", "140227913496848\n", "\n", "4.bias\n", " (10)\n", "\n", "\n", "\n", "140227913496848->140227913496992\n", "\n", "\n", "\n", "\n", "\n", "140227913299024\n", "\n", "ELUFunctionBackward\n", "\n", "\n", "\n", "140227913299024->140227913496992\n", "\n", "\n", "\n", "\n", "\n", "140227913497040\n", "\n", "AddmmBackward\n", "\n", "\n", "\n", "140227913497040->140227913299024\n", "\n", "\n", "\n", "\n", "\n", "140227913495600\n", "\n", "2.bias\n", " (1024)\n", "\n", "\n", "\n", "140227913495600->140227913497040\n", "\n", "\n", "\n", "\n", "\n", "140227913298608\n", "\n", "ELUFunctionBackward\n", "\n", "\n", "\n", "140227913298608->140227913497040\n", "\n", "\n", "\n", "\n", "\n", "140227913494832\n", "\n", "AddmmBackward\n", "\n", "\n", "\n", "140227913494832->140227913298608\n", "\n", "\n", "\n", "\n", "\n", "140227913494976\n", "\n", "0.bias\n", " (1024)\n", "\n", "\n", "\n", "140227913494976->140227913494832\n", "\n", "\n", "\n", "\n", "\n", "140227913494640\n", "\n", "x\n", " (10, 512)\n", "\n", "\n", "\n", "140227913494640->140227913494832\n", "\n", "\n", "\n", "\n", "\n", "140227913494592\n", "\n", "TBackward\n", "\n", "\n", "\n", "140227913494592->140227913494832\n", "\n", "\n", "\n", "\n", "\n", "140227913496464\n", "\n", "0.weight\n", " (1024, 512)\n", "\n", "\n", "\n", "140227913496464->140227913494592\n", "\n", "\n", "\n", "\n", "\n", "140227913514688\n", "\n", "(10, 1024)\n", "\n", "\n", "\n", "140227913514688->140227913298608\n", "\n", "\n", "\n", "\n", "\n", "140227913494736\n", "\n", "TBackward\n", "\n", "\n", "\n", "140227913494736->140227913497040\n", "\n", "\n", "\n", "\n", "\n", "140227913494928\n", "\n", "2.weight\n", " (1024, 1024)\n", "\n", "\n", "\n", "140227913494928->140227913494736\n", "\n", "\n", "\n", "\n", "\n", "140227913443264\n", "\n", "(10, 1024)\n", "\n", "\n", "\n", "140227913443264->140227913299024\n", "\n", "\n", "\n", "\n", "\n", "140227913495168\n", "\n", "TBackward\n", "\n", "\n", "\n", "140227913495168->140227913496992\n", "\n", "\n", "\n", "\n", "\n", "140227913494784\n", "\n", "4.weight\n", " (10, 1024)\n", "\n", "\n", "\n", "140227913494784->140227913495168\n", "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = torch.randn(10, 512, requires_grad=True)\n", "torchviz.make_dot(elu_mlp(x).mean(), params=dict(list(elu_mlp.named_parameters()) + [('x', x)]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's run the backward pass and make sure we have gradients on all parameter tensors." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.474930Z", "iopub.status.busy": "2020-11-24T17:30:32.474286Z", "iopub.status.idle": "2020-11-24T17:30:32.508795Z", "shell.execute_reply": "2020-11-24T17:30:32.509350Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.weight 3.0498824798996793e-07\n", "0.bias 1.9646263282879772e-08\n", "2.weight 5.69895860280667e-07\n", "2.bias 5.812371739466471e-08\n", "4.weight 7.324277930820244e-07\n", "4.bias 1.7035910104823415e-07\n" ] } ], "source": [ "l = torch.sum(elu_mlp(x))\n", "l.backward()\n", "\n", "for name, param in elu_mlp.named_parameters():\n", " print(f\"{name} {torch.norm(param.grad).item()}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Differentiable Optimization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we'll tackle a more interesting use-case for defining our custom backward functions: differentiating though an inner (unconstrained) optimization problem." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What if we want to solve an inner optimization problem as part of our model, while the parameters of the inner problem are also optimized by the end-to-end optimization of the entire model?\n", "\n", "
\n", "\n", "Training such a network end-to-end means we're trying to find:\n", "\n", "$$\n", "\\begin{align}\n", "\\vec{\\Theta}^\\ast\n", "&=\n", "\\arg\\min_{\\vec{\\Theta}} \\E[(\\vec{x},\\vec{y})\\sim D]{\\mathcal{L}(\\vec{y}, \\hat{\\vec{y}})}\\\\\n", "&=\n", "\\arg\\min_{\\vec{\\Theta}} \\E[(\\vec{x},\\vec{y})\\sim D]{\n", "\\mathcal{L}(\\vec{y}, \\arg\\min_{\\vec{y}} f(\\vec{y}, h_{\\vec{\\Theta}}(\\vec{x}) )\n", "}\n", "\\end{align}\n", "$$\n", "\n", "This type of setting is also known as a bi-level optimization problem." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From the perspective of the inner problem, $\\vec{z}$ is a \"fixed\" parameter. \n", "\n", "However, from the perspective of end-to-end training, we're optimizing $\\vec{\\Theta}$ in order to reduce the final loss.\n", "\n", "Therefore, we can view this as learning to parameterize the inner task." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What do we need in order to train such a model end-to-end?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As usual, we must find a way to calculate a VJP, in this case:\n", "$$\n", "\\delta \\vec{z} = \\pderiv{\\hat{\\vec{y}}}{\\vec{z}}\\ \\delta\\hat{\\vec{y}}.\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Assume that $\\vec{y}=\\arg\\min_{\\vec{y}'}f(\\vec{y}', \\vec{z})$.\n", "\n", "Since $\\vec{y}$ is a minimizer of the function $f$, the necessary optimality condition\n", "must hold: $$\\nabla_{y}f(y, z)=0.$$\n", "\n", "If we then perturb $\\vec{z}$ by $d\\vec{z}$, we'll get a slightly different minimizer, $\\vec{y}+d\\vec{y}$. Thus also,\n", "\n", "$$\\nabla_{y}f(y+dy, z+dz)=0.$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can take a first-order Taylor expansion of $\\nabla_{y}f$ around the point $(y, z)$:\n", "\n", "$$\n", "\\nabla_{y}f(y+dy, z+dz) \\approx \\nabla_{y}f(y, z) + \\nabla^{2}_{yy}f(y,z)dy + \\nabla^{2}_{yz}f(y, z)dz = 0.\n", "$$\n", "\n", "Since $\\nabla_{y}f(y, z)=0$, we rearrange to obtain:\n", "\n", "$$\n", "\\nabla^{2}_{yy}f(y,z)dy = -\\nabla^{2}_{yz}f(y, z)dz.\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll denote the Hessians as $\\mat{K}=\\nabla^{2}_{yy}f(y,z)$ and $\\mat{R}=\\nabla^{2}_{yz}f(y, z)$. We then obtain,\n", "\n", "$$\n", "\\mat{K}d\\vec{y}=-\\mat{R}d\\vec{z}\\Longrightarrow d\\vec{y}=-\\mat{K}^{-1}\\mat{R}d\\vec{z}.\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above equation means that we have found a linear relationship between the change in the function value, $d\\vec{y}$ and the change in the argument value, $d\\vec{z}$.\n", "This linear relationship must be, by definition, related to the gradient of $\\vec{y}$ w.r.t. $\\vec{z}$.\n", "\n", "Writing the above as an inner product, we have $$d\\vec{y}=\\ip{\\mattr{R}\\mat{K^{-T}}}{d\\vec{z}}{}.$$\n", "\n", "Note that $\\mat{K}$ is a Hessian, therefore symmetric and we can drop the transpose.\n", "\n", "Finally, by the \"outer\" definition of the gradient we see that $$\\pderiv{\\vec{y}}{\\vec{z}}=\\mattr{R}\\mat{K^{-1}}.$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now back to our VJP: we need \n", "$$\n", "\\delta \\vec{z} = \\pderiv{\\vec{y}}{\\vec{z}} \\delta\\vec{y} =\\mattr{R}\\mat{K^{-1}}\\delta\\vec{y}.\n", "$$\n", "\n", "We'll do the calculation in two steps:\n", "1. Calculate $\\delta\\vec{u}=\\mat{K}^{-1}\\delta\\vec{y}$: Equivalent to solving the linear system $\\mat{K}\\delta\\vec{u}=\\delta\\vec{y}$.\n", "2. Calculate $\\delta\\vec{z} = -\\mat{R}^\\top \\delta\\vec{u}$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, based on this, we have a way to implement such an inner-optimization layer:\n", "\n", "**Forward pass**: Compute the optimal solution of the inner problem, either with a some solver or even a closed-form expression.\n", "\n", "**Backward pass**: Calculate $\\delta\\vec{z}$ using the two-step procedure described above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that it's also possible to develop these expression for the constrained optimization case, by using the KKT conditions where we used the unconstrained optimality condition." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Implementation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before implementing the argmin layer, we need some required helpers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, let's implement a helper to calculate an approximated least-squares solution to a linear system of equations $\\mat{A}\\vec{x}=\\vec{b}$.\n", "The built-in `torch.solve()` only supports a square matrix A." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.514478Z", "iopub.status.busy": "2020-11-24T17:30:32.513838Z", "iopub.status.idle": "2020-11-24T17:30:32.535041Z", "shell.execute_reply": "2020-11-24T17:30:32.535570Z" } }, "outputs": [], "source": [ "def solve_ls(A: Tensor, b: Tensor, abs: float = 1e-6, rel: float = 1e-6) -> Tensor:\n", " # Solves the system A x = b in a least-squares sense using SVD, and returns x\n", " U, S, V = torch.svd(A)\n", " th = max(rel * S[0].item(), abs)\n", " # Clip singular values\n", " Sinv = torch.where(S >= th, 1.0 / S, torch.zeros_like(S))\n", " return V @ torch.diag(Sinv) @ (U.transpose(1, 0) @ b)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Quick test for solve:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.539913Z", "iopub.status.busy": "2020-11-24T17:30:32.539312Z", "iopub.status.idle": "2020-11-24T17:30:32.985541Z", "shell.execute_reply": "2020-11-24T17:30:32.986103Z" } }, "outputs": [], "source": [ "from sklearn.datasets import make_regression\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "dtype = torch.float64\n", "torch.set_default_dtype(dtype)\n", "\n", "# Create a simple regression problem\n", "N, D = 1000, 20\n", "X, y, w_gt = make_regression(\n", " n_samples=N, n_features=D, coef=True, random_state=42, bias=10, noise=1,\n", ")\n", "X = StandardScaler().fit_transform(X)\n", "X, y, w_gt = [ torch.from_numpy(t).to(dtype) for t in [X, y, w_gt] ]\n", "\n", "# Solve it and check the solution is close to the ground-truth\n", "w_hat = solve_ls(X, y)\n", "assert torch.allclose(w_hat, w_gt, rtol=0.1, atol=0.1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that in the formulation we used, we considered only one variables vector, $\\vec{y}$, one parameters vector, $\\vec{z}$, and moreover we treated them both as 1d vectors.\n", "\n", "In practice, however, when implementing deep neural networks, we deal with many different parameters tensors, virtually all of which high-dimensional.\n", "\n", "In this tutorial, we'll deal with the case of one variable tensor (of any shape) and any number of parameter tensors (of any shape). It's simple to use the approach here to extend to multiple variables as well." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:32.990636Z", "iopub.status.busy": "2020-11-24T17:30:32.989995Z", "iopub.status.idle": "2020-11-24T17:30:33.016217Z", "shell.execute_reply": "2020-11-24T17:30:33.016786Z" } }, "outputs": [], "source": [ "def flatten(*z: Tensor):\n", " # Flattens a sequence of tensors into one \"long\" tensor of shape (N,)\n", " # Note: cat & reshape maintain differentiability!\n", " flat_z = torch.cat([z_.reshape(-1) for z_ in z], dim=0)\n", " return flat_z\n", "\n", "def unflatten_like(t_flat: Tensor, *z: Tensor):\n", " # Un-flattens a \"long\" tensor into a sequence of multiple tensors of arbitrary shape\n", " t_flat = t_flat.reshape(-1) # make sure it's 1d\n", " ts = []\n", " offset = 0\n", " for z_ in z:\n", " numel = z_.numel()\n", " ts.append(\n", " t_flat[offset:offset+numel].reshape_as(z_)\n", " )\n", " offset += numel\n", " assert offset == t_flat.numel()\n", " \n", " return tuple(ts)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Quick test for `flatten`/`unflatten`:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:33.020518Z", "iopub.status.busy": "2020-11-24T17:30:33.019933Z", "iopub.status.idle": "2020-11-24T17:30:33.047517Z", "shell.execute_reply": "2020-11-24T17:30:33.048054Z" } }, "outputs": [], "source": [ "t1, t2 = torch.randn(3, 5), torch.randn(2, 4)\n", "t_flat = flatten(t1, t2)\n", "assert t_flat.shape == (t1.numel() + t2.numel(),)\n", "\n", "t1_, t2_ = unflatten_like(t_flat, t1, t2)\n", "assert torch.allclose(t1, t1_)\n", "assert torch.allclose(t2, t2_)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, finally, we're equipped to write an `autograd.Function`\n", "which implements differentiable optimization!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's start with the forward pass.\n", "Here we simply need to solve an unconstrained optimization problem specified in terms of an *objective function*, *variables* (just one supported) and *parameters*.\n", "\n", "We'll use the LBFGS algorithm (a quasi-Newton method as seen in the lectures) as a general purpose solver." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:33.053003Z", "iopub.status.busy": "2020-11-24T17:30:33.052386Z", "iopub.status.idle": "2020-11-24T17:30:33.082881Z", "shell.execute_reply": "2020-11-24T17:30:33.083527Z" } }, "outputs": [], "source": [ "from torch.optim import LBFGS\n", "\n", "def argmin_forward(ctx, obj_fun, y, *z):\n", " \n", " # Note: solving for y, treating the z's as constants\n", " optimizer = LBFGS(params=(y,), lr=0.9, max_iter=1000)\n", "\n", " # Closure for LBFGS which evaluates the loss and calcualtes\n", " # gradients of the variables.\n", " def _optimizer_step():\n", " # zero gradients\n", " y.grad = torch.zeros_like(y)\n", " \n", " # evaluate loss\n", " f = obj_fun(y, *z)\n", " \n", " # Calculate gradients\n", " # Note: not calling backward() because we don't want to compute\n", " # gradients for anything except y\n", " δy = autograd.grad(f, (y,), create_graph=False)[0]\n", " y.grad += δy\n", " \n", " return f\n", "\n", " # Solve the optimization problem - will evaluate closure multiple times\n", " f_min = optimizer.step(_optimizer_step,)\n", " y_argmin = y # Note: y was modified in place\n", "\n", " ctx.save_for_backward(y_argmin, *z)\n", " ctx.obj_fun = obj_fun\n", "\n", " return y_argmin" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now the backward pass.\n", "Here we use the two-step procedure shown above, to calculate $\\delta\\vec{z}$.\n", "\n", "We'll need to calculate the Hessians\n", "$\\mat{K}=\\nabla^{2}_{yy}f(y,z)$\n", "and $\\mat{R}=\\nabla^{2}_{yz}f(y, z)$,\n", "but luckily autograd can calculate this (by a second automatic differentiation).\n", "\n", "The only complication is the shapes of $\\vec{y}$ and $\\vec{z}$, but we'll flatten them with our helpers." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:33.089274Z", "iopub.status.busy": "2020-11-24T17:30:33.088627Z", "iopub.status.idle": "2020-11-24T17:30:33.115633Z", "shell.execute_reply": "2020-11-24T17:30:33.116265Z" } }, "outputs": [], "source": [ "from torch.autograd.functional import hessian\n", "\n", "def argmin_backward(ctx, grad_output):\n", " y_argmin, *z = ctx.saved_tensors\n", " obj_fun = ctx.obj_fun\n", " \n", " flat_y = flatten(y_argmin)\n", " flat_z = flatten(*z)\n", " \n", " # Wrap objective function so that we can call it with flat tensors\n", " def obj_fun_flat(flat_y, flat_z):\n", " y = unflatten_like(flat_y, y_argmin)\n", " zs = unflatten_like(flat_z, *z)\n", " return obj_fun(*y, *zs)\n", " \n", " # Compute the Hessians on flattened y and z\n", " H = hessian(obj_fun_flat, inputs=(flat_y, flat_z), create_graph=False)\n", " Hyy = K = H[0][0]\n", " Hyz = R = H[0][1]\n", "\n", " # Now we need to calculate δz = -R^T K^-1 δy\n", " # 1. Solve system for δu: K δu = δy\n", " δy = grad_output\n", " δy = torch.reshape(δy, (-1, 1))\n", " δu = solve_ls(K, δy) # solve_ls(A, B) solves A X = B\n", "\n", " # 2. Calculate δz = -R^T δu\n", " δz_flat = -R.transpose(0, 1) @ δu\n", " \n", " # Extract gradient of each individual z\n", " δz = unflatten_like(δz_flat, *z)\n", " δy = torch.reshape(δy, y_argmin.shape)\n", " return None, δy, *δz" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now wrap these functions in a `autograd.Function` class:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:33.120912Z", "iopub.status.busy": "2020-11-24T17:30:33.120305Z", "iopub.status.idle": "2020-11-24T17:30:33.148780Z", "shell.execute_reply": "2020-11-24T17:30:33.149355Z" } }, "outputs": [], "source": [ "class ArgMinFunction(autograd.Function):\n", " @staticmethod\n", " def forward(ctx, obj_fun, y, *z):\n", " return argmin_forward(ctx, obj_fun, y, *z)\n", " \n", " @staticmethod\n", " def backward(ctx, grad_output):\n", " return argmin_backward(ctx, grad_output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's run a quick test for `argmin_forward`: Can we solve the simple regression problem from before?" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:33.153834Z", "iopub.status.busy": "2020-11-24T17:30:33.153228Z", "iopub.status.idle": "2020-11-24T17:30:33.193959Z", "shell.execute_reply": "2020-11-24T17:30:33.193309Z" } }, "outputs": [ { "data": { "text/plain": [ "tensor([ 3.3349, 9.3753, 0.1935, -0.2837, 4.7919, 0.1489, 2.9468, 0.0318,\n", " 0.3177, -0.0361, 0.6610, 1.7553, -0.0704, 0.5406, 0.1594, 1.7986,\n", " 0.5532, 5.1842, -0.1253, -0.0440], grad_fn=)" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Define a simple linear regression objective with l1 and l2 regularization\n", "# (just to test more than one z)\n", "def obj_fun(w: Tensor, l1: Tensor, l2: Tensor):\n", " loss = torch.mean((X @ w - y)**2)\n", " reg1 = l1 * torch.mean(torch.abs(w))\n", " reg2 = l2 * torch.mean(w ** 2)\n", " return torch.sum(loss + reg1 + reg2)\n", "\n", "w = torch.randn_like(w_gt, requires_grad=True)\n", "l1 = torch.randn(1, 1, requires_grad=True)\n", "l2 = torch.randn(1, 1, requires_grad=True)\n", "\n", "# Function.apply() invokes the forward pass\n", "w_hat_argmin = ArgMinFunction.apply(obj_fun, w, l1, l2)\n", "\n", "assert torch.allclose(w_hat_argmin, w_gt, rtol=0.2, atol=3)\n", "w_gt-w_hat_argmin" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Quick test for `argmin_backward`: Do we get gradients on our $\\vec{z}$s ?" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:33.198453Z", "iopub.status.busy": "2020-11-24T17:30:33.197679Z", "iopub.status.idle": "2020-11-24T17:30:33.232872Z", "shell.execute_reply": "2020-11-24T17:30:33.233389Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "loss=tensor(24.1708, grad_fn=)\n", "\n", "w.grad=tensor([-6.5928e-06, 5.7268e-06, -4.9958e-06, 3.5084e-06, 9.8241e-06,\n", " -3.1058e-06, 6.1233e-07, -2.9807e-06, 3.5734e-06, -3.0825e-06,\n", " 4.5685e-06, 4.8643e-07, -2.8262e-07, -4.6700e-07, -2.2415e-08,\n", " 1.5638e-06, -6.6559e-06, 6.6045e-07, 4.7814e-06, 8.2119e-08])\n", "l1.grad=None\n", "l2.grad=None\n", "\n", "w.grad=tensor([0.0500, 0.0500, 0.0500, 0.0500, 0.0500, 0.0500, 0.0500, 0.0500, 0.0500,\n", " 0.0500, 0.0500, 0.0500, 0.0500, 0.0500, 0.0500, 0.0500, 0.0500, 0.0500,\n", " 0.0500, 0.0500])\n", "l1.grad=tensor([[0.]])\n", "l2.grad=tensor([[0.]])\n" ] } ], "source": [ "loss = torch.mean(w_hat_argmin)\n", "print(f'{loss=}\\n')\n", "\n", "# Before backward: z (l1 & l2) gradients should be None\n", "print(f'{w.grad=}')\n", "print(f'{l1.grad=}')\n", "print(f'{l2.grad=}\\n')\n", "\n", "loss.backward()\n", "print(f'{w.grad=}')\n", "print(f'{l1.grad=}')\n", "print(f'{l2.grad=}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's demonstrate how to use our `ArgMinFunction` in the context of a model.\n", "\n", "We'll tackle the task of **time-series prediction**, by applying linear regression on a learned, non-linear representation of the inputs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll implement the following model:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notes:\n", "\n", "1. The encoder and decoder will be simple 1D CNNs.\n", "2. The encoder calculates some non-linear embedding of the input while the decoder maps an embedding back to the input space.\n", "3. The predictor applies linear regression to try fit optimal weights for predict the last part of the embedding, $\\vec{Y}_e$ from the first part, $\\vec{X}_e$ (\"post-diction\"):\n", " 1. Postdiction: $\\vec{w}^\\ast=\\arg\\min_{\\vec{w}} \\norm{\\vec{X}_e\\vec{w}-\\vec{Y}_e}^2+\\lambda\\norm{\\vec{w}}^2$\n", " 2. Prediction: $\\hat{\\vec{Y}}_e=\\vec{Z}_e\\vec{w}^\\ast$ where $\\vec{Z}_e$ is the last part of the embedding with the same length as $\\vec{X}_e$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's load some data: We'll use a open dataset from Kaggle, containing 12 years of daily stock price data from equities included in the Dow Jones Industrial Average (DJIA).\n", "\n", "You can obtain this dataset [here](https://www.kaggle.com/szrlee/stock-time-series-20050101-to-20171231)." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:33.237121Z", "iopub.status.busy": "2020-11-24T17:30:33.236395Z", "iopub.status.idle": "2020-11-24T17:30:33.570391Z", "shell.execute_reply": "2020-11-24T17:30:33.571012Z" } }, "outputs": [ { "data": { "text/plain": [ "(93612, 7)" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "import numpy as np\n", "import matplotlib.dates as mdates\n", "from datetime import datetime\n", "\n", "df = pd.read_csv(\"DJIA_30/all_stocks_2006-01-01_to_2018-01-01.csv.gz\", compression='gzip')\n", "df.shape" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:33.576835Z", "iopub.status.busy": "2020-11-24T17:30:33.576143Z", "iopub.status.idle": "2020-11-24T17:30:34.587212Z", "shell.execute_reply": "2020-11-24T17:30:34.587776Z" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAugAAAFpCAYAAADOXt1wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAADCjUlEQVR4nOzdd3hURRfA4d/sZtMbJHQIvaqAgIoiKqKo2MGKogJWFHuvWFGxYFcEsXxib9iQZkEQFARE6R0CBAKk9935/pit2U2y6e28z5Pn3jt37t3ZCHh29twzSmuNEEIIIYQQom6w1PYAhBBCCCGEEB4SoAshhBBCCFGHSIAuhBBCCCFEHSIBuhBCCCGEEHWIBOhCCCGEEELUIRKgCyGEEEIIUYeE1PYA6prExETdoUOH2h6GEEIIIYRowJYvX56qtW4W6JwE6MV06NCBZcuW1fYwhBBCCCFEA6aU2l7SOUlxEUIIIYQQog6RAF0IIYQQQog6RAJ0IYQQQggh6hAJ0IUQQgghhKhDJEAXQgghhBCiDpEqLuXgcDhITU0lLS0Nu91e28Np9MLDw2nbti02m622hyKEEEIIUWUkQC+HXbt2oZSiQ4cO2Gw2lFK1PaRGS2vNgQMH2LVrFx07dqzt4QghhBBCVJkaTXFRSp2glJqllEpWSmml1FVe52xKqWeUUv8opbKVUnuUUjOVUknF7hGmlHpFKZXq7DdLKdW2WJ8mSqkPlFLpzp8PlFLxlR1/dnY2bdq0ITQ0VILzWqaUIiEhgby8vNoeihBCCCFElarpHPRo4F/gFiC32LlIoB/wpHN7LtAOmK2U8p7pnwKMBC4FBgOxwHdKKatXn5nOe5wBnO7c/6Aq3oDFImn7dYV8SBJCCCFEQ1SjKS5a6x+AHwCUUu8WO5cOnOrdppS6DvgP6AmsVkrFAeOAMVrruc4+o4HtwCnAT0qpnpig/Hit9WKv+yxUSnXXWq+vvncohBBCCCFE5dT16eBY5/aQc9sfsAFzXB201juBtcBxzqZjgSxgsdd9FgHZXn2EEEIIIYSok+psgK6UCgWeB77VWu9yNrcE7EBqse4pznOuPvu11tp10rm/z6tP8de6Vim1TCm1bP/+/VX4LoQQQgghRF2gHQ7yN26s7WEEpU4G6M6c8/8B8cCYYC4BtNexDqKPp7PWU7XWA7TWA5o1a1bO0dYfK1aswGq1MmjQoAr3UUq5f2JiYhgwYABffvml+/zEiRM5/PDDq3zsQgghhBCVceijj9hy9jnk/L2itodSpjoXoDuD84+A3sBQrfUBr9N7ASuQWOyy5phZdFef5srrCULnfjOvPo3S22+/zfjx4/n3339Zu3Ztpfrs2bOHv/76iz59+nDhhRfyxx9/VOfQhRBCCCEqJevnXwAoStlL5oIFrD/6GOxZ2bU7qBLUqQBdKWUDPsEE50O01nuLdVkOFOL1MKmzxGJPPDnnf2CqxRzrdd2xQBS+eemNSm5uLjNnzuSaa67hggsuYPr06RXqAxAfH0/Lli3p0aMHb775JuHh4cyaNau634IQQgghRIUVbNtmdpSFfc+/gCMjg00nnED2kiW1Oq5AarSKi1IqGujiPLQASUqpvsBBYDfwGXAUcDaglVKunPF0rXWu1jpdKTUdmKyU2gccAF4A/gHmAWit1yqlZgNvKaWuwaS2vAV8V9UVXB799j/W7M6oyluWqVfrWB45+7ByX/f555/Tvn17evfuzejRo7nooouYNGmSzyqcwfQpzmazERISQmFhYYXejxBCCCFETbCnpwOg7UXoXFPt25GTw+677qbrwt9qc2h+anoGfQCwwvkTATzq3H8MaIupfd4aM1O+x+vnYq973AZ8iZlpX4Sp2HK21tru1ecyYBWm2stPzv3R1fWm6oNp06YxerT5FZx44olERkb6zXoH08dbfn4+TzzxBBkZGQwdOrT6Bi+EEEIIUUm6qAiAgs1bsGd7UlusCQm1NaQS1XQd9F8wM9olKXPlGa11HjDB+VNSn4PA5eUdX3lVZCa7NmzatIlFixbx0UcfAeZBz8suu4xp06YxcuTIoPu4jB49mquuuorc3Fzi4uJ47rnnOOOMM2r2TQkhhBBClIMrQE99/XWscXHudlvLgEX+alWNBuiidkybNg273U5SUpK7zVWFcufOnbRr1y6oPi6TJ0/m9NNPJzY2lubNm9fQuxBCCCGEqASvdFxXugtASIsWtTGaUtWph0RF1SsqKuK9995j0qRJrFy50v2zatUqevfuzYwZM4Lq461ly5Z06dJFgnMhhBBC1Hu2lnUvQJcZ9Abu+++/JzU1lWuuuYaEYjlWl1xyCW+88QZ9+vQps8+DDz6IxRLc57m8vDxWrlzp0xYZGUm3bt0q9V6EEEIIIapKq0mTiDvvXLwqc9cZMoPewE2fPp0hQ4b4Bd4AF154Idu3b+fSSy8ts8+8efOCfs3Nmzdz5JFH+vyMGjWqUu9DCCGEEKKitMPhcxxz6il1NjgHmUFv8EqrwtKpUyd3nnmwfcrqP3HiRCZOnFiuMQohhBBCVCedn+9zHNKiZZ0NzkFm0IUQQgghRAPnyMvzObY2bVJLIwmOBOhCCCGEEKJB08UC9JAmEqALIYQQQghRaxy5xWbQ4+NrZyBBkgBdCCGEEEI0WFprtgwf7tOmwsJqaTTBkQBdCCGEEEI0WHvuvc+9b4mOBup+gC5VXIQQQgghRIOkHQ7Sv/nGfdzqySfJW7eWqIEDa3FUZZMAXQghhBBCNEiO7GyfY1vLFsSeNqyWRhM8SXERQgghhBANkj0tzb2fcO21hB9+eO0NphxkBl0IIYQQQjRI+Rs2AND+o5lEHnlkLY8meDKDLoQQQgghGqS8//4Di4Xw7t1reyjlIgF6I5GSksJtt91G165dCQ8Pp3nz5hx33HG88sorZGVlufstXbqUc845h6ZNmxIWFkaPHj149NFHyStW4L+8fVeuXMmll15K69atCQsLIykpieHDh/PVV1/hcDgA2LZtG0opli1bVn2/CCGEEEI0CrmrV5P6+hvgcGCJjKzt4ZSLBOiNwLZt2+jXrx+zZ8/m8ccf5++//2bBggXceeedzJ8/n1mzZgEwa9YsBg8eTEJCAvPmzWPDhg088sgjTJ06lWHDhlFQUOC+Z3n6fvfddxxzzDGkp6czY8YM1qxZwxdffMEFF1zAk08+ye7du2v8dyKEEEKIhi3zp59qewgVJjnojcANN9yAxWJh2bJlREVFudsPP/xwRowYgdaanJwcxo0bx/Dhw5kxY4a7T/v27enevTsDBgzgpZde4q677ipX3+zsbMaMGcOZZ57Jl19+6e7buXNnjjrqKMaOHYvWumZ+EUIIIYRoNOzp6bU9hAqTAL0yfrwX9q6u2ddseQSc8XTQ3Q8ePMhPP/3EU0895ROce1NK8dNPP5Gamsrdd9/td75fv34MHTqUmTNnctddd5Wr75w5c0rs6/36QgghhBBVyZGTW9tDqDBJcWngNm7ciNaa7sUejmjbti3R0dFER0dz/fXXs8H5lHPPnj0D3qdXr16sX78eoEJ9vV9/9erV7teOjo7mww8/rMQ7FEIIIYQIoB5/Qy8z6JVRjpnsumbhwoXY7XauvfbagA91Fqe1Dnqmu6y+3bt3Z+XKlQD06dOHwsLCoO4rhBBCCBEsrU0Rig6fflLLIyk/mUFv4Lp06YJSinXr1vm0d+zYkS5duhDpfKq5W7duAKxZsybgfdauXUvXrl0r3Nf79UNDQ+nSpYt7bEIIIYQQVU5DaKdORPTu7X/u0Hb4+SlwVpKrayRAb+ASEhIYNmwYr776qk85xeKGDRtGQkICkydP9jv3999/M3/+fC677LIK9500aVIVvSMhhBBCiCBoDSVNBH4xDn59Bg5srNkxBUkC9Ebg9ddfx+Fw0L9/fz766CPWrFnDhg0b+Oijj1i1ahVWq5WoqCjefvttvv/+e8aOHcuKFSvYsWMHH3/8Meeccw7HH388t9xyC0C5+06fPp3Zs2dz+umnM3v2bDZv3szq1at54YUXyMvLw2q1+ox3w4YNrFy50ucnmDQcIYQQQgg3raGkL+qz95ttVkqNDac8JAe9EejUqRMrVqxg0qRJPPTQQ+zcuRObzUbPnj0ZP348N910EwDnn38+v/32G08++SQnn3wyOTk5dOjQgauvvpp7772X0NBQ9z3L0/fcc89lyZIlPPPMM4wZM4bU1FRiY2Pp168fM2bMcM+2uxQ/BvNg6eGHH15NvyEhhBBCNDjFn4mzF8G2hdB5iCe1JX1X7YytDEpqUPsaMGCALmkly7Vr15ZYuUTUDvlvIoQQQohAdk2YQMG27XT61izIyLyJ8PuLMHYOfHUtHNoGx98OpzxSK+NTSi3XWg8IdE5SXIQQQgghRIOji+egJ/9ttvkZ4LCb/T+nQta+mh9cGSRAF0IIIYQQDY8GLF6hbkG22TqKoDAHmveCgizYs6pWhlcaCdCFEEIIIUTD43CYGfTJXWDJG5BzwLTbC6AwD1o4n207tK3WhlgSCdCFEEIIIUSdlrduHemzZpXvIq0BbSq2zL4XDm017Vn7oDAbErtCSAQc3Frl460sqeIihBBCCCHqtK3nnQ9A3DnnBH+R1qjMvf7tP9xptkV50O00iG5eBSOsWhKgCyGEEEKIesGRm4slIiKovhoN2aU8ANr/KohPqpqBVTFJcRFCCCGEEPWCozwLF5a2UFH/MXU2OAcJ0IUQQgghRD2hCwuD7+woZa2fiPhKj6U6SYAuhBBCCCHqh6Ki4Pum7zTbYU/4nwuPr5LhVBcJ0IUQQgghRJ2Vv2mTe79cM+g5h8w6RR0Gw7074aL3wRpqzoXHVe0gq5gE6I3AVVddhVKKq6++2u/c3XffjVKKs846C4CcnBzuv/9+unTpQnh4OImJiQwaNIiPPvrI737Ff6ZMmRKw3fvn3Xffram3LYQQQogGYPtll7v3dXlm0DVgi4DWfSE8Fnqd6zkXFlNl46sOUsWlkWjXrh2ffPIJL730ElFRUQAUFRXxwQcfkJTkeUji+uuvZ9GiRbz00kscfvjhHDx4kKVLl3Lw4EGf+51yyil88MEHPm1NmzblkksucR/ff//9rFu3ji+//NLdFhdXtz+xCiGEEKJuCe3ahdxly4HyzaDrQA+JWsPMQkVhsVU4wqonAXoj0bt3b3bv3s2nn37KmDFjAPj+++8JDw/nhBNO4MABs7rWrFmzeO6559wz6h06dKBfv35+9wsLC6Nly5Z+7d5tkZGRhIaGBuwnhBBCCBEMZfWEq7qgIPgLtTYriXqLbAIFmRAWXUWjqx41GqArpU4A7gT6A62BMVrrd73OK+AR4FqgCbAUuFFr/Z9XnzDgOeBSIAKYD4zXWu/y6tMEeBlwVbOfBUzQWqdV5ft55s9nWHdwXVXeskw9mvbgnqPvqdC148aN45133nEH6K79LVu2uPu0bNmS2bNnc+GFF8pstxBCCCFqjD0zE52fT0hiok+7zstDhYWh8/NxZGcHd7OVMyE/Ayxhvu0RTSBtB1jq9hx1TeegRwP/ArcAuQHO3w3cAUwAjgL2AXOVUt6JQlOAkZgAfTAQC3ynlLJ69ZkJ9APOAE537vvmYzRCo0aNYtmyZWzcuJG9e/cye/ZsrrrqKp8+U6dOZenSpSQmJtKvXz9uuukm5s6d63ev2bNnEx0d7f4544wzauhdCCGEEKIh2nDU0Ww8fjAFO3f6tDtyctxBezABut63nn2P3k3+QY1fjsvR15ltXNuqGHK1qdGPD1rrH4AfAJRS73qfc86e3wo8rbX+wtl2JSZIHwW8pZSKA8ZhZt7nOvuMBrYDpwA/KaV6YoLy47XWi519rgMWKqW6a63XV9X7qehMdm1p0qQJ559/Pu+88w7x8fGcdNJJPvnnACeccAJbtmxhyZIlLFq0iAULFjBs2DCuvfZa3nrrLZ9+U6dOdR9HBLmqlxBCCCFEcfbMTPd+3n//UbR3LxFHHsn+l14if+NGYk49hcLkZOxZWWXeK+/10RxYa+Z2VfEUlyMvg94Xg1Vm0IPVEWgJzHE1aK1zgd+A45xN/QFbsT47gbVefY4FsoDFXvdeBGR79Wm0xo4dy/vvv88777zD2LFjA/ax2WwMHjyYe++9lzlz5vD4448zdepUtm3b5u4TGRlJly5d3D9t2rSpoXcghBBCiIbGfuiQe//g68+zffQVHHj7bQ68PQ2A8MMOAwLPoBfu3UthSor7OG1Vhudk8QAd6nxwDnUrQHc9SZhSrD3F61xLwA6kltFnv9bavXyUc3+fVx8fSqlrlVLLlFLL9u/fX/F3UA8MHTqU0NBQUlNTOe+884K6plevXgBkBfGpVQghhBCivNI//Z97P3/HPgAKd+92t8XtfxWA7EWLKW7TSUPYdOJJnutTvVYQDRCf1wd18SNE8XVZVYC24or3CdS/xPtoracCUwEGDBhQ1mvVa0op/vnnH7TWhIWF+Z0/6aSTuPTSSxkwYAAJCQmsWbOG+++/n+7du9OzZ89aGLEQQgghGrLC3btIneZ5VNCRZyq1FO3zTJqGFO0EWpO1YAH5W7cS1rFj4Jv99zUFGV6hnKMcddPrkLo0g77XuS0+y90cz6z6XsAKJJbRp7nySjpy7jfDf3a+UYqJiSE2NnD9z9NOO40PPviA0047jR49ejB+/HgGDx7M3LlzsVqtAa8RQgghhKiowv/MrHhk83yf9qxffwUgNDHCJ1Nl+xVXlHivtPdex57vFa/Y80vsW5fVpRn0rZjg+lTgLwClVDimUstdzj7LgUJnn5nOPm2Bnnhyzv/AVIs51qvtWCAK37z0RqOs1Tu9z993333cd999lbqfy6uvvhpUPyGEEEI0XjozDYDY9rnk7PP/dj+ubzNgs/vYvj8VR3Y2FufCi+77FBSQsyPPp01FNq3y8daEGp1BV0pFK6X6KqX6Ol87yXmc5MwTnwLcq5QaoZQ6HHgX88DnTACtdTowHZislDpFKXUkpnziP8A8Z5+1wGxM1ZeBSqljgbeA76qygosQQgghhKg8R5ap4GI9bFjA89Yws3pot7u6Y3NWn9t26Sjy1m/g4Aee3PXCvXvR9iKwQMypQ01js27VOPLqU9Mz6AOAn72OH3X+vAdcBTyLWXzoNTwLFQ3TWmd6XXMbUAR8gmehoiu01navPpdhFipyVXuZBdxUxe9FCCGEEEJUkj3NVHCxduwN/Ol3PiTjX4gBa4iD8B49KNyxg/wNG9gxZgz2gwfd/Qp27sSRV0hYQgihHUyOuiOvsEbeQ1Wr6Trov1DK87TOWfSJzp+S+uRhFjKaUEqfg8DlFRymEEIIIYSoIQW7doPSRBxxRMDzFpvzoU9HEUUHD7jbdZ5vOsuBF56CjANYrBEoWygAuStXVsuYq1tdekhUCCGEEEI0MoW792KLsmNp7qnMEts+x72vLBpi24C9EHuqJ0APad7c5z45/23BUaSwkIMKD6/+gVcjCdCFEEIIIUStSP9oBhnLtmKLApp0cLdbQjylErXdCi17Q/Z+7Onp7vYCrwUUzUWgixSWEI2tTWsAmt1+ezWOvvrUpSouQgghhBCikdB2O7sffRYAa4QCq424Djmkb4skvlMOaZtNlZbItmHQtCNs+x1d4L/mZPTQoVgiI8n49lvy022EDTyD2OHDwaGJPf20Gn1PVUVm0IUQQgghRI3TX45374e2aAJA64fvpOclu4lI8DzcqcKjYM8/UJCJI9+3rnn0iYNo99qrWGNi3G25/65FKUXc2WehbLZqfhfVQwJ0IYQQQghRszL3kr/wC/dhSEvnzPhxE+CKWb59beGw/XcAkm49jfDevd2nrNvngsOBJc6zAGPrZ56uvnHXEAnQhRBCCCFEjSpYv5Jtc5q5j1WE1wrnnU4EoO3gA7Q7w/mA6PlTAYiKP0jHTz9xd7WEaHisCaFrpwFgiw8lcsCAGngH1UsCdCGEEEIIUaP2PPOGb0NopO9xWBwxbfKJjtsDLQ6D3heZ9n+/gG9vpfm4882xMg+T2tReAOy5dhoCCdAbkRUrVmC1Whk0aJDfOaUUSil+//13n3a73U7r1q1RSvH5558DMHHiRHf/QD+PPvooAL/88gtKKXr06EFRUZHPfTt06MBzzz1XTe9UCCGEEHVZ/rZd7v3wpgXEDhns2+G81zz7zXuC8lpGZ/kMYva+SVzHHJp0zQYgNNoE5o58CdBFPfP2228zfvx4/v33X9auXet3vl27dkyfPt2n7ccffyQkxLfYz5133smePXv8fq666iri4+MZNWqUT//t27f73VcIIYQQjZclPNS93/H1J7AOvMy3Q0iEZz++vdmOmOZuCo2x0/qYNMJi7c7uZhve0nPf+kwC9EYiNzeXmTNncs0113DBBRcEDJivuuoqPvvsM7Kystxt06dPZ8yYMT79oqOjadmypc/P/Pnz+eCDD/j444/p2rWrT/+bb76ZiRMnkp2dXT1vTgghhBD1iiXMzIiHduoIfS/1nSEH82CoS2i02fa+EDqf7Nuvz6Uw9BGUBToM20+7cUdW46hrjtRBr4S9Tz1F/tp1NfqaYT170PL++8t93eeff0779u3p3bs3o0eP5qKLLmLSpEnYvMoP9e7dm549e/LJJ58wbtw49u3bxw8//MCLL77IY489VuK9ly9fzjXXXMPTTz/Naaf51xudMGECH330ES+88AIPPfRQuccuhBBCiIZF5+WgQiDpnXcCd7BFePf27B7Y5Nsv5wC0Pw6AiEQNF75ctQOtJTKD3khMmzaN0aNHA3DiiScSGRnJrFmz/PqNHTuWd5x/Wd5//30GDx5Mhw4dSrzvvn37OP/88xkxYgR33nlnwD7h4eE8/vjjTJ48mf3791f+zQghhBCiXrNn5RPXtxm2lv4LDwEQ09qzH9vGs59zyLff4SOh3TFw+tNwyz8QHktDIDPolVCRmezasGnTJhYtWsRHH30EmAdCL7vsMqZNm8bIkSN9+o4aNYo77riD9evX884775Q6411YWMgFF1xAixYtmDZtWon9AEaPHs3zzz/P448/zssvN4xPt0IIIYQoP+1wYM/XWJs0LblTjFfgHucVoPe/Ev541ezHJ0GfS8z+wBuqfqC1SAL0RmDatGnY7XaSkpLcbVqbr4t27txJu3bt3O1xcXGMGDGC66+/nj179nD++eeXeN+bb76ZDRs2sGzZMsLDw0vsB2CxWHj66ac577zzuOWWWyr5joQQQghRXzmys0ErrNERJXdSCmyRENPKt33YEzD0YfjhTuh1XrWOszZJgN7AFRUV8d577zFp0iTOOussn3OjR49mxowZPPzwwz7t48aN4+STT+bGG28sMfCeOnUq77zzDj///DNt27YNaizDhw9n0KBBPPDAAxV7M0IIIYSo9+yHTJpKqQE6wN1bQRXLxlYKQsLgnFeqaXR1gwToDdz3339Pamoq11xzDQkJCT7nLrnkEt544w0efPBBn/YhQ4awf/9+YmJiAt5z0aJFTJgwgYcffphOnTqxd+9en/OhoaE0bRr4a6tnn32WgQMH+jycKoQQQoiGzZGdTfLd99D0yitQVhN0lxmg20r/dr4hk4dEG7jp06czZMgQv+Ac4MILL2T79u3MmzfP71xiYiJhYWEB7zlt2jQKCgp48MEHadWqld/PiBEjShzPUUcdxQUXXEB+fn7F35QQQggh6pWM2T+RNX8+O664kt333AtAePtmtTyquktm0Bu4QJVaXDp16uTORXdtS+J9fsaMGcyYMaPM1z7ppJMC3veTTz7hk08+KfN6IYQQQjQMKsTq3i/clQyArVmT2hpOnScz6EIIIYQQouZZZJ64JBKgCyGEEEKIauXI801t7XreXgnQSyEBuhBCCCGEqFaFu3f7HIeEOyA0qpZGU/dJgC6EEEIIIapV1m+/Eda9uzmwOMPPhK61N6A6Tr5bKCetNUqp2h6GoOwHW4UQQghRN9gPHCBq8PGEdupIkwGJsPFZiGlR28Oqs2QGvRxsNhu5ubm1PQzhVFhYSEiIfMYUQggh6jpHVhbWmFjavvgiUUnO+udRzWt3UHWYRDfl0Lx5c5KTk2nTpg0REREyk16LHA4HKSkpxMXF1fZQhBBCCFGSdd9TlGPHkZNDSHNnQJ61D0JjIDSydsdWh0mAXg6xsbEA7N69m8LCwloejYiKiiIxMbG2hyGEEEKIQLSGj0dhz7ACLQhp5lyYKHsfRMvseWkkQC+n2NhYd6AuhBBCCCFKsGcVANphMg5UWKhpz5IAvSySgy6EEEIIISrOXggOu3/7ktcB0HYToFvCwkz7toVmdl2USGbQhRBCCCFExU3uAq2PhCu+RtvtpL72GvFt9mFb/Qm5h6JJXmxyzdU/H0JP58x59r5aHHDdJwG6EEIIIYSoGK0hLw22/AxA3urVpL7+BqkAtPbpqtZ/CTP/MAdnTanBQdY/kuIihBBCCCEqJi/d5zB32aISu1ptGtJ2mIOOJ1TnqOo9CdCFEEIIIRqxgl3JHHz//YotAJi93+cwb/6nPsdhXTq4963hzjz1zieDlKoulQToQgghhBCN2J6HHiTlqUnsGn9j+S/OSqEw10LmrnBy//2X7PW7TbtF0ezqS2h+74MAxHfKJiTM+QGgqKCKRt5wSQ66EEIIIUQj5cjJIeePJQBk/fwzW846A1uzRNrN+CC4G2TtY9M3Lc3+7xcCISQenkGzz5PdXbqev8cE58fdDItfht4XVfG7aHgkQBdCCCGEaKTyt271Pd60jfxN24K7uCAbkpf7NYed/4DPcUiTpqAsMOxx8yPKJAG6EEIIIUQjVbh7d8UvfspVpcWrWouC6AvG+fa7ZRUoa8VfpxGSHHQhhBBCiEaqcOcuAFo99VT5Lvzv64DNoS3jPQsSuYTFQGhkBUbXeEmALoQQQgjRSDnycgGI7N/Ptz03p/QLd/5p+vXyzSe3tWxRdYNrxCRAF0IIIYRojIry0VuWQkgItjZtfE45Fr/j3//r8fDiEZBzEAqzyTrUgh1fZft0sXXsUZ0jbjTqVICulLIqpR5XSm1VSuU5t08opUK8+iil1ESl1G6lVK5S6hel1GHF7hOmlHpFKZWqlMpWSs1SSrWt+XckhBBCCFFHbZyDY8MCLFaNmtLL55Teuhh2LPHtv/JDSN8BU0+Cgmx2zbeSu2IFALaoIgBC2rSriZE3eHUqQAfuAW4EbgZ6ALc4j+/z6nM3cAcwATgK2AfMVUrFePWZAowELgUGA7HAd0rJEwpCCCGEaGT2/AOzboZD283xoW2w+BUozEMXKZQqgKwULCEO9yV63Wx45zTPPbYv9uynbYfVnxGW4Kk1Yu3QG4CQhITqfCeNRl0L0I8DvtVaf6u13qa1ngXMAo4BM3sO3Ao8rbX+Qmv9L3AlEAOMcvaJA8YBd2mt52qt/wZGA72BU2r6DQkhhBBC1Kq3BsPf78FLvSF9F3x3G8x5EPvmJaRticKeZ+Yvu52/l/jDI8w1xRcVLT6bDlgjPKuBNrvtdgAi+vSplrfQ2NS1AP13YIhSqgeAUqoXcDLwg/N8R6AlMMd1gdY6F/gNE9wD9AdsxfrsBNZ69RFCCCGEaPi2/e57/MfrsOVXAAp/edfnlLJC5LljAdDFA3SH3WwfSoVYk6/usDUncuBAeqz5j+jjB9FjzX+E9+xZ1e+gUaprAfozwAfAGqVUIfAf8J7W+nXneedSVaQUuy7F61xLwA6kltLHh1LqWqXUMqXUsv3791fyLQghhBBC1BGznVnC7Qaa7Z9vgTbBtqOwWBh462pUYmezr5XvucIcsNjAaoOCLAqyrORuTMaekY6ymPu4tqLyyvWbVEpZlFLVWcjyYuAKTLpKP+f+eKVUsYr3fl+8qABtxZXYR2s9VWs9QGs9oFmzZuUftRBCCCFEaX57DibGBZiarmbWULMd8Rac+QI4ityn7AUmDGw/80OYmA7xSWA16S7uYbp2CnPA5gwBC3JI22z2m152WbW/hcaozABdKXWGUup9pdR2oADIdFZGWaiUekAp1bqse5TDZOA5rfXHWuvVWusPgBfwPCS617ktPhPeHM+s+l7ACiSW0kcIIYQQoub8/KTZZu2r2PVrv4O89PJfl7Yd+l0BTTpA5yEA5B6wsXd5LPYCM0se4jU5qZwB+ra5zcg9aIOifPjyWlj6JoTHmk6Xf44jviuWmBjiR46s2PsRpSoxQFdKnaeU2gC8AxQCTwHnA6cBY4CfMQ9dblFKvamUqoqp50hMeoo3u9c4t2IC8FO9xhmOqdTierx4uXO83n3aAj29+gghhBBCVJ/cQ/DfV55j10x2fkb575X8N3xyGcx9pPzXFhV4Zr6bdoK7trD9t9Yc2hhNvt3MsVpjvArhOdNUtF2xbU4zUi7oyv6Z35tzYc4AvdNJ6KQTUaGh5R+PCEpIKefuA24HftBaOwKc/xRAKdUGUw7xCuD5So7nW+BepdRWTP75kc4xvA+gtdZKqSnAA0qpdcAG4EEgC5jp7JOulJoOTFZK7QMOYGbh/wHmVXJ8QgghhBBl+/pGWP89NO0MrXp7HrIsyPL0+fdLU1Vl0M2l32v332a74afyj8NeYPLGXaIS0EUmrMvc2xRIxuIVoDuyfVcQPbg+GoDEXlmoME8/XVCACpMAvbqUGKBrrY8J5gZa62RMbfKqMAF4HHgdk5KyB3gbeMyrz7NABPAa0ARYCgzTWmd69bkNKAI+cfadD1yhtS4+Oy+EEEIIUbW+u90E52BKHB5zPTgKzXG+V4D++RizLStAP7jVbDN3l38sjkLP7L2L3YRDhcnJgCetBaAoZS+B5KeHELZtCRQVoUJC0Pn5WGwSoFeXOvW4rdY6U2t9q9a6vdY6QmvdSWt9v9Y6z6uP1lpP1Fq30lqHa61PdNZD975PntZ6gtY6QWsdqbU+21lqUQghhBCiei2b7nu89E3PfmbgALhUFbkGzAOejiKwhrL/lVfJWb6cwpTSH8eLPevsgO0FmSGs+6w1G44+hrQvviRr0SJJcalGpaW4AKCUigJCtNbpzmMLZiXP44G/gUla64JqHaUQQgghRH3gCJQV7CV9R/D3+vYWWP6u5zg6YLXoktkLKcy1kPb9P6R+u4LU117z69LmpZd8jm0tmhN79tlkfPutT3vy4qYAOHJy2PPAAwCodu3KNx4RtGBm0N/HpIy43A48ABwErqHyeedCCCGEEPVf1n54dUDpfdKcX+jbC0vvZy/0Dc4BQsLKNx57PtvnJ5L67YoSu9haNPdrKx6cl0Rm0KtPMAH6kcBcr+OrgBu11mOAkc4fIYQQQojG7bkucHBz6X2WzzCpJwdK6VeYC48XrxaN50HTYK37nsIs/2SJplde6d63xMb6nY8eOjSo28tDotWntDKLPyulfgbaAs8rpRY4j3sBE5RSCzArf7ZwnltQM0MWQgghhKhjCnN9j4c9AVf9ELhvdqopw1iSl/oEbi9PrYv8TPjqOvdh3MgR7v2Q5mbW3NauHaFJSX6XhrZt43Ps6l+czKBXn9KquAwBcC5QdLfW+jel1JlAG631QOe5OGCH1vrkGhmtEEIIIURddGCT73G7gRDvlaN9/e+wdzV8fQMUZoM9v+R7ZXk9yHnEhdCsO+xaZuqhByH9u+/ZfdeddBuh3G2RRx2FrU0b8lb/S5PLRpEwbmyJ10cecwwH33vffWxt2pSifftIuPZawnv1IufPpRya+REhiQFm+UWVKPMhUeAX4G2l1P8wtc4/9jrXB9hYDeMSQgghhKg/9q/3PQ4JhfA4s9/tdGh5hCeIL8z1zUHXGpTy7Me1g8Su0PEE6HUeNO0I398BO//0fY2U/2D+4zD4Dmh3lKf5ySdBm8orALFnnUXcueeilCIYMSefTNMrr+Tge++B1eouwxjWtSuxp59G1sLfAIg759yg7ifKL5gA/XZgCnARpp74U17nzgP+V+WjEkIIIYSoL7SGL8b5tllCwBYBNyw2K3iCZ0XP1wf69rUXmoDe4YBtCyF9J5z8EPS52Pd+hb6LCPHrM7DhRzMbf9H74FxIyH7IpM9sm2MWeY844vCgg3MXa1NTtcUaG4suMMX6QpqZGfNmt9xCzNBTiDrm6HLdUwSvzABda30AGF3CudurfERCCCGEEPVJdqp/W4vDfLdgAvZA7AUmQH/taDjgTEzoeIJvnwOboCgP/vsKopqZGXnX625eAP+7AMaZlUZVWBg635NCoyIjy/2WXPnlIYmJaGfpSNeKo7bmzbGdHDgvXVSNYGbQhRBCCCFESdK2e/a7ngadhwTuZyshUHatMnrAK2s4vFh1FddiRdsXw59TocNg2L7Ic37nEnOrnByf4ByAoqIy3kCAobZpbS49eJCkaW+z/6WXCevatdz3ERVTYoCulJoGPK613l5SH2c/BYwCLFrrD6p4fEIIIYQQdduhbWZ7/lvQ55KS+5U0g773X8g54DlWFv9g/sJ3TY11ZfLB2bbQ/z7Jf1OYH+/XHDmgjNrsAYQkmvQYe0YG4T170u7NN8p9D1Fxpc2g7wL+UUotBWYBy4A9QB7QBFNu8XjgYmAbcF3g2wghhBBCNGCHtpptz3NK7xfVLHD7e2f5HmuH56FRl4QugIKMXT7N6fuTyNqYQeLhWYStnIm91WUAtDvhAFHHD4LRX5Y7/xwgJMHkoFNYxoJKolqUVmZxolLqNcxqodcBLxXrkgnMA8ZoredU3xCFEEIIIeqwQ9shugWElpHrHV2JvG2lIDQK1vqu8rl7fhEQScaOSHoMtXBg+nSUzUpYXBFqxFv+gX6QXA+JitpRag661no/pmrLU0qpJkASEAGkApu11rr6hyiEEEIIUYcd2gbx7av/dUKjoCDLfZifYfU5ve4OE7zHHp6Ara21Uh8ILNHRFb5WVF7QD4lqrQ8BpSx7JYQQQgjRCGXu9a3WUpq+l8PKClaodualb5nTFkUeMe3yAnazqGxnSkzFudJiok4YXKn7iIqx1PYAhBBCCCHqNUchWINc9v7oqz37fS/zPXfERaVf68x1zz/oIO9gKPtXmUovze+606ebzs8pOd+9HLqvWkm7N+Th0NogAboQQgghRGU47GYhoWDYosz2hLvgsBGe9tOegpFvw0n3wWVflOvlE8b5LpIU1SQdDm4p1z0CsYSFuVcRFTVL6qALIYQQQlSGww6WIAPZZt3ghj+geU/Y+qunPcm5uuhJ95brpRPHj/drUzbtW7ZR1Dsygy6EEEIIAZC1D9bMKt81mSmQuTv4AB2gRS9TXaVpZ09bm/5lX3f5F+iR7/g0WRMTAOgyeZS7zRKi4ZoFwY9H1DnlCtCVUolKqWOUUmHVNSAhhBBCiFrx3W3w6WjY8JOnLT8LSita984ws3Wt9FkKv+J38e3g8i9g5PTgxtflFOhRrNa63QGA7eyH3E2WgWMgPC64e4o6KagAXSkVo5T6FNgHLAbaONvfVEpNrL7hCSGEEELUENcs+Eznw5pL3oRJbeCxhJKvca0iai99QZ+UZ55lXc9e/kF6l1PgiAuCHqIjv8DnOO5szyJHIZEmWFfRUsO8vgt2Bv0ZTFDeD8j1av8OOL+qByWEEEIIUeOaO0slhsWaWfPZ95hjbQ/c3zvYthcE7uN0cMYM0y0trVJD1PmmtGKLBx6g57q1WOPj3eea904HwNYmqVKvIWpfsA+JngOcr7VeqZTy/ui3FuhU9cMSQgghhKhhrlU3m3YyD366xJUQ8Bble/bLCNAtkZE4cnJwZGZCkyYVGp7WmgPTTQ66JSLc73xch1xi2+eiWkmAXt8FO4PeBAj0OHAMUMLHSiGEEEKIekSbFBEcRaa2uUtC58D9V3/q2S/yLBpkz8wkd9Uqcleu9Jx3lit05HonIpRP9uLFHHzHBOgqzD9Ap3kv8xkjIr7CryHqhmBn0P/CzKJPcR67ZtGvw+SkCyGEEELUb66UFXuBb055UeAVO7F61cxQnjnP7aMuI3/jRvdxi/vuBYs578jJqfDwHJmZ7v2QxAB58aO/hr/fg2Y9K/waom4INkC/H/hJKXWY85rbnftHAydU1+CEEEIIIWqMawa9KB8+u8rT7ijy7G/5xazSGdsGvrrW026xUbhvH1vPPgd7errPbVMmPe25VXbFA3Rd6PnQENGvn3+HmBZw4t0Vvr+oO4JKcdFaLwaOA0KBzcBQYDdwrNb67+obnhBCCCFETfGaQd8839O86y/Y4lxU6P1z4Y3jYNVHnvOhMTDkfrJ/+80vOC/OkVH6+VJHl+/JebeEScXrhizolUS11quBK6txLEIIIYQQtcc1g565x//c++fA/V7t6380296XwIi3AFBrvi3zJewZGRUenj0js+xOokEItg76hUqpcwO0n6uUCr54pxBCCCFEXRVoQSJblGc/fZdn/+BWCIuD8153N+X8+afPpaHt2/vdzp5emQC94rPvon4JtorLRCDQExLZznNCCCGEEPVcsQB9zGzoNsxzvGelZz99h6nu4lzcSGtN2mefu0+3mTKFdm+96T7uPG8eKiysUkG2K3/d2lQWImrogg3QOwHrA7RvQuqgCyGEEKIhcKW4AJzzKrQ/Fixe2cBfXuPbP9ITKOti1VlsrVuhbDb3cWjbNlhjY3FUIsUFux1LbCxdf19Y8XuIeiHYHPRDQFdgW7H2boAkRAkhhBCi/tMaLDa4c4Mn+LaUEiqFeGqRO/J8Ew1UaCgqNNSnzRIXW6kUF60dqJAQlCXY+VVRXwX7X/gb4EWlVDdXg1KqO/AC8HU1jEsIIYQQomZpB1htPjPjrhQWt8NGQKu+Zt8W6bm0WICui+w+M+gA1pjYSj0kit3hrqcuGrZg/yvfDaQDa5RSO5VSO4H/gAzgruoanBBCCCFEjVLFQqOQiGLHYWBzttlKnkG3tWoJIc4APcTMwltjYyv3oKd2yOx5IxFsHfRMrfUg4AzgZeAV4HRgkNa6Eh8FhRBCCCHqCO0AlG9bRBPf46Ov8bSFxbqbXQF6i/vvo/vyZYQkJGCJjCDm1FNJensqYFJcHM4UF0deHhlz5gQ9tIy5c016jATojULQddABtNZzgbnVNBYhhBBCiNqjtf8MevEAPaELxLY2+14BuivFJbRTZyxRpjSjslho+8rL7j7W+HiKUlPRBQWkPPMMaR99TMjMmUT2O7LUYeWtW0fyhJsBsLVuXZF3JuqZEgN0pdTtwOta6zznfom01i9U+ciEEEIIIWqSdvhNoPsF6OFxEJlg9h2F7mZXiUVLmO+Dod5C27dH5+dTlJZGYXKyuUVm2YkIBVu3eg5U8QGKhqi0GfQJwHuY+ucTSumnMQ+LCiGEEELUYxq/CD08zrN/1ovOtnizzT1krioqIv3rr81+oSdoL84S7sxZL6VPIPleAborsBcNW4kButa6Y6B9IYQQQogGKVCKS7gzjSXpWBgw1uy3O8Zs2x4FgD3TVJxWkZFEDhxY4u1dVV2KP1BaFnvqgXL1F/VfmU8aKKVsSqmlzrKKQgghhBANk3b4p5C4ZtDzvFJR2vaHO9ZD74sBcDgD9JYPPlhqlRVXgL7lzLM8L6l1Sd3d7GlpQQxeNCRlBuha60KgI37r31YPpVQrpdR7Sqn9Sqk8pdQapdSJXueVUmqiUmq3UipXKfWLUuqwYvcIU0q9opRKVUplK6VmKaXa1sT4hRBCiDovZQ3sXw9Z+2t7JHVMgBl014OgRbm+7TEt3cG8K+3E1rpVqXf3qYtejlxye1oa4X16B91f1H/B1up5D7imzF6VpJSKBxZhEsDOBHpi8t/3eXW7G7jD2X6U89xcpVSMV58pwEjgUmAwEAt8p5QqttqAEEII0cjkpsHUE+G1o2HqSZC1Dxz22h5V3RCozGJ8Egy8ES7+X4mXuXLEQzuWnhEczGx5IPa0NKzx8RW6VtRPwZZZjAIuU0qdCiwHsr1Paq1vrqLx3A3s0Vpf4dXmfjJCKaWAW4GntdZfONuuxATpo4C3lFJxwDhgjLMsJEqp0cB24BTgpyoaqxBCCFG/5B6CZzp4jjN2wXNd4ejrYPizFbtnUQE80czs37PNv+qJ1pC8HKYNhZtXQtM6/FhboBx0peD0p0q9rGDLViyRkYQ0b15qv7AOHQK/Zikc+fkU7t9HWNcupfYTDUuwM+g9gb+BQ0An4Aivn8OrcDznAUuVUp8opfYppVYqpW5yBuZgUm1aAu7K/lrrXOA34DhnU3/AVqzPTmCtVx8hhBCi8fn7/fK1ByMn1bO//D3fc1rDM+1NcA6w9M2Kv05NCJSDHoSCrVsJ7dgRVca1YV270nTsWAgJIfvX38q+765k1vfpi31/qsygNzJBzaBrrYdU90CcOgHjgReBp4G+mFVLAV7FBOcAKcWuSwHaOPdbAnYgNUCflgSglLoWuBYgKSmpwoMXQggh6rSfJ5ntVd+bmeIZZ5jj4vnV5ZGf6dmf9wgcfysseRO2L4LjJkCe19L2S9+EM56p+GtVN3shWEuuY16Swt27CevWLai+lvAwKCoKqu/mU05x71vj4+ny6y+llnEUDUeZAbpS6kLMzLYNmKe1nlqN47EAy7TW9zmPVyilugI3YgJ0l+LfB6kAbcWV2Mf5nqYCDBgwoEYehhVCCCFqTF4G7F7hCcQ7HF/5e6b8B28E+GI6MwVm32P2I+Ir/zo1qSgXQsLLfZkuKkKVskCRt/wtW30bgow6IgcMwNaiRTlHJuqrUlNcnDPLnwADgO7AG0qpSdU4nj3AmmJtawHXtPZe57b4THhzPLPqewErkFhKHyGEEKLx+PoGeP8cs9/l1MB9yvsA4+JXfY9jnBVM8r3KEQZKnXn9ONj7b/leq6YU5UNIWLkv0/YilDW4x/oyZ88O/sYhnntGHnVUeYcl6rGyctAnAE9qrbtrrftgHr68qRrHswjzQcBbN8wDnmAeGN0LuP91UUqFYyq1LHY2LQcKi/Vpi8mjd/URQgghGofUjbDuO8+xs3Y3AH0u9ew7V8UMyq5lsGqmb1ukc15s6VulX7vvP3h3ePCvVZM2zC7/BxWAIjsqpGKF4nRBfuBb7t/vToUpqzqMaHjKCtA7ATO8jj8AwpRSAXO5q8CLwECl1ANKqS7O9JqbgdcAtKlPNAW4Vyk1Qil1OPAukAXMdPZJB6YDk5VSpyiljnSO+x9gXjWNWwghhKh78jLg1QG+bb0v9Oyf9wZc5Jzl3r4o+Pse2ubf5lpx86+3zTbROd926mNmG9HUa1xeeemVlbUPJsbB2u/K7lsWZS3fBxUnbbeDNbgA3ZqQ4HNcmLzbvZ+5YAEHpr8DQMqkpwEzc97hk4/LPSZRv5X1fUwEJvgFQGttV0rlA5HVMRit9V9KqfOAp4CHgB3O7ete3Z51jus1oAmwFBimtfZ6SoXbgCJMek4EMB+4QmsthV6FEEI0Hk+3K/28UtDyCLOfuqHkfg47bFsInU4yxwXO0KBVH9izCnqda0o1es+MXzITFj5n2pOOhegWZib/p/sr/HYC+mu62S56CdJ3mfz3B1LA5swl1zr4yiwh4XDYeeUfQ1HwKS5tnpvMjjFj3ccFu3a693eNvxGA/M2byfjhBwASrr0Wa2xs+cck6rVg/jRdr5TK8joOAcYppQ64GrTWL1TVgLTW3wPfl3JeAxOdPyX1ycOk50yoqnEJIYQQ9UrqpuD6xXcw243zoPNQaN3Xv8/CF+DnJ+DEe6D7cPj2FtN+0Qem6klsK9i3zveaxC5wvrOsYrujzfbYG6s+QHd9WNj1J6TtMPvZ+yG+nXlg9fluMOJt6H1R6fdxOKAwG0KjyvXy6d98gz09PejqKta4OPe+ioykaM9evz7pX34JmNSW6MFV8ECvqHfKSnHZAYzBE+xOwOSAj/I6rs6cdCGEEEJUxKv9g+tnsUBIBOxYbFYYzT7g3yd5udn++ozp4xLXzgTnAM17QN/LzH7/q0p+vcPO9+ynboTfX6xY3reLd7pN7kHn1pmmkulMH5n/WNn3yUsz2/C4UrsVt/ueewEoSi1e3TkwS4xn4fOwLl3QBQUl942IKNdYRMNR6gy61rpDDY1DCCGEENXttBIKsXnXQZ/cCW5abmbAXQLljB9/mwnuvZ33uvkpTUIXU4NdazObvnGOWX20tKC+NN4But0Z7OY4P2Q4nJmt6Tsp0fbFZtbc4gyJYltXaBiOnJyg+oW2a0fHb74hrGsXdlw1xidAt7VuTeFuT066JU5SWxqrYFcSFUIIIUR9cGCzeWgykGPHB3cP1wJGLoe2+vdx5a6Xly3CrNhpL4CDW0zbt7fAH69V7H5pAYLvD84z20KvDx5FAWaqtTbv9a0TIMMZGMdULEDX+YGrsQQS3r0bymJB2Wzu1JisX3/1Cc4BrHHxFRqLqP8kQBdCCCEaklf6efa7nQ4Xvlf+e3jXMtcacg7694lM8G8Lhs1ZZ6IwBw545clXJDfdXgj5pVSEKcrz7OdnQPLfMP00U/Fl0zzf8pMZyWbrStkpr2AfRPW+JDQUR6H54LDzuuv9zluiqqUmh6gHJEAXQgghGooM3xlYDhsBbfoF7utt/BLfY+VVMrAwF+z50KSDbx/vsonl4crxnnmJ/7mC7ODvs20R7F5p9o++1tPe1rmgT1E+5KZ52uc+DL88DTuXwCeXwf9GwieXe87v+stsI4uvc1gybfcUh2tx913Bj91J2Wx+OehhPXuSOME83meJLN8Dq6LhCK4mkBBCCCHqPtcKnZd9bsomHj7S5HsDHHNDydc17wnxSZ4qKN6zwa4HLo+7GZq0h13L4ZenIKaCS6IkdDXbnc4PBfHtIc25HuFTrU0u+MMBHlQtzrukY9ujoVVfSBoIW381wfYTzU3ZRJeVH5Z+vxX/M1tb6Q9mFqbsY98zT9PyscdI//IrAGKHn0FE375lj7kYFRqKLizE4ZUe0+Lee7HGxpC3Zi2J111bytWiIZMAXQghhGgI0nbAll/MfkJn6Hqq59zEIBYGshd59guyYNN86DLUE6BHJUKXU0wpxmOuNQ92VkSzbr7HNyw26S7POQN3R5H5oNHycHPsXce8MBdQprSjt4gm0PUUs79vjafdO8UlWGWkqux75hkyfviRiAEDSHnqKTPknNxSrynxpWw2dEEh9nTz36flIw8TdYwpSdnutVcrdE/RMASd4qKUCldKXaCUukcpFe9s66yUquB3XEIIIYSoMlOOgCWvmeC1Ig86ti224ugvk0xt9DcHmWNXSotSFQ/OwbeMYftBEBYN0c0h1FN+0P2aLxwGrx3jaX+pj/k5UKzGu/d4opr7v2aY12veuQnO9XogtbRvFgIo2LULgJTHHne3tXz00XLdw8U9g55p1lq0yIJEwimoGXSlVBdgLhADxAOfAWnADc7jq6tldEIIIYQoW57XQ532As8qmuVxzsuwdpbneNdf8OFIz3Fc24qPr7geZ5kHNGPbeNru3wX5WTDJ2bZxHmSYYJgDm823Alkp5vi1o3zvF+k1V9h2AJz9Muxfbz6wRLcw93WJSoQjL4eup3mObeEm776PyYs/+OGHpDz+BPGXXkLEYYcRc+qp7gWGCnf6V42xtQjwoSAIRXv3Yk9NpSjFvC9ZMVS4BDuDPgUToLcAvL/HmQUMqeIxCSGEECJYe1bBjOFl9ytLRBO4dTVM+Dvw+fj2lX8NF1dueEJn3/awaM++94eDrb8Gvo9rYaS4dp42ixX6X+kOtmnWHU66x+zft8uTwhLdzPwoBadMhKEPQWJXtN1OyuNPAJD20cfsefAh9j33nNf9q66+Rtav5n3tGDsOkABdeAT7p+w44Dmttb1Y+w6gYgVDhRBCCFF5i16GlNWe4zE/Vvxe8Un+QbNLFQam7rz2hC7+5wJ9EFj7nX8bwLAnTH69NUBCQIvD4aT7YeR0GHSL6RcW49+vmEMffezX5soxT779DuwHfB9g7fzT7DLvWRJb+ySfY0uMBOjCKM/fNluAtiQgiCdPhBBCCFEt7F5l+h4+BO2Pq/rXuPDdqr1fTqrZxif5nxv2uO9x74th99+eVUFdmnb2TW0pzmIxM+fR5Us/sR/0ryCji8wDtBk//ODTHnP66YS2r/g3C81uvtnn2BJRgdQk0SAFG6DPAW73OtZKqVjgUeD7Kh+VEEIIIQLLyzBVTrQ2y9zv/Qe6n2lmiKtqlrvHWZ798Dg47Pyqua/LgLFmm9jN/1yvc+HwC8x+dAtofaSZcXetOupy07KqHZOTtjsASLh6nLutcO8eilJT/fpaIiu3kJByzfzbzByorVUFF0kSDU6wf5NvB45XSq0HwoFPgG1AS+De6hmaEEIIIXyk74Kn25kqJ0vfNBVNDm3zX0Sosi7+H9zmLFfoHaxXlf5XmQ8UEfGBz7sqyhTlQ5OOZn/fWs/5mNZVm3LjzWFH2Ww0u+02uv31JxF9+pC36h82Hj/Y3SX2LPM7sUSUXjO9LCrEuSBUYSHRpwyt1L1EwxJUFRet9W6lVF/gUqAfJrCfCnyota5Y8U8hhBBClM/PT3n2Z3vNj1X1DLdSENcGrl8EzXpU7b2D0cYZoOelQahzNc3dKzzny1hMqDK03QFWK8pqxRoTgwr1rbne7Lbb3HXLLZGVHEeIJwwL61RC7r9olIJeqMgZiL/j/BFCCCFETfNeut6l00kmDaQ6uBYLqmmuADwkHEKdaSS/v2C2zQ+DC6oxFLHbUV6z8yoszOd09IknkPXzz86TlZvFV14Pt4Z17lSpe4mGJag/WUqpi5RSw7yOH1ZK7VJK/aSUkoQpIYQQoroVFcD6AI99XfFN4Com9ZmrDKMtEiITfc9d/zs0r75Zfe0wM+guxWfQba1aocLM+HR+BVYq9eJOcQFs7QI8MCsarWA/+k107Sil+gH3Ay9jKrs8X/XDEkIIIYQPe75n/96d5mHK6xbW3niqk2sGOzQKmnhVSTniourLPXfxm0H3DdAtsbHuNkdePpWhbJ4Cedb4+ErdSzQswX7kbg+sd+6fD3yttX5WKTUH+KlaRiaEEEIID22qi3DsTRAeCxe9X7vjqU6hznrlPc82285DYfN8OPqaan9p7bD7zKBbis2gK6WwhDtn0AsLK/VaIc2aufetcVIDXXgEG6DnAa7q/kPx5KGne7ULIYQQorq46oDHta3dcdSE6GZwyz8Q28YcXzgD/vnU8/BodbI7wOo1gx7qyUHvPG8eALFnnEH24j9odsvNfpeXR0iLFu59WUVUeAs2QF8IPK+U+h0YADgLlNIN2FkdAxNCCCGEF63NtpIPJtYb3qkt4XE1M3tut5P22Wc+bZboaDOcUaMIbWs+MFgiI2nz/HOVfj3XTDz4prsIEezf8puAAkxgfr3Werez/QwkxUUIIYSofq4Ul8YSoNeCnGXL/dosUc4yj9Wd+y6El2DroO8Czg7QfmtVD0gIIYQQAUiAXu1yV/zt12aJNgG6LiioltdU4eHovMpVgxENT7nqMimlTgZ6ARpYo7X+uVpGJYQQQghf2pmDLgF6tXFkZQHQdfEid5vVmeKi8ytXsaUkXX/9pdIPm4qGJ6gAXSnVBvgK6A+40ltaK6WWAed7pbwIIYQQojq4ZtAt1tL7iQqzZ2djjY8npGlTd5sl0iyU5CiongDdGhdXLfcV9VuwH8NfBuxAF611O611O6Crs+3l6hqcEEIIIZwkxaXaOTIyUZERPm3KVVKxkjXPhSiPYFNcTgVO0lpvdTVorbcopW4G5lfLyIQQQoiGKi8dtvwCPc4O/uFDh6S4VLeM7wOs1OqsnqPtRTU8GtGYVfZvuaNKRiGEEEI0JJl7IaOU7M+PRsGnV8CGH4O/p8yg14qII4/EmphIswkTansoohEJ9m/5fOBlpVQ7V4NSKgl4CZlBF0IIIXw93x1e6Ak5BwOfP7TNbD8eBdkHoCC77Hu666BLDnp10A7zASj6xBN92kOaNqXb7wuJOOKI2hiWaKSCDdBvBiKBLUqp7UqpbcBmZ1vlltESQgghGqpfJnn2N86Dg1vNT8YuT/vkTvDiYZCd6nttfhZMjIN/nAvnuGfQVfWOuZGyHzQfpqIGDarlkQgRfB30nUA/pdSpQA9AYcoszqvOwQkhhBD1Wppzse38TPhwJLQ4HA4f6d8v9xCs+RqOutpz/EwHs//l1dD7Qtgw2xwf2FTdo25U7OnpYA2hYIf5b2VLalfGFUJUv3Ilsmmt52qtX9FavyzBuRBCCFGCpp3NNisFDm6BSW3Nccq/MP9Rs3T92GILcX9/h2d/U7Hs0Q9GwKqPzf6BzdUz5gbInpbG2h49yZxfcjbuhmMGsunkkyncuQOA0KT2NTU8IUpU4gy6Uur2YG+itX6haoYjhBBC1HO7lsNBZxC9+294/1z/Ps0Pg6SBkHQs7PjD0578N7Q+EhY84dt/83xo1sPsD3u8esbdABXu3QvAnocfIWboUL/zBTvNrLkjI4P8jRsBsLVqWXMDFKIEpaW4BPu4sgYkQBdCCCEApp1strYoKMyGtB3+fS7/wmxHfw3bF0F8e3i1P7w9BI4cDYe2wpGXw4n3wBTnw4n715l+MRJABktZzQO19gMHsGdmYomORilF2hdfkLXwd/LWrHH3PTBturnGWfdciNpUYoCute5YkwMRQgghGpSb/zbVXFxOuAt+mwy9zoNQszoltnDoMtS3isuKD8y2+3CIT4L7kuGl3pBzAI65rsaG3xC4KrMAbDjqaACihwwh6+efS7xGyUO4og6QYqpCCCFEVSnM9ezHtISHUsEaBq36wskPwri5cM4r/teFRvm3Ne1ktmHRJu0FIKFLlQ+5QXP4L9dSWnAuRF1RahUXpdQZwBtAH611erFzccAq4Fqt9ZzqG6IQQghRTxzabraDnQ98Wm1w8woIjzXH7Y4u+drLv4Atv8Lil82xK+cc4NibIH0XtD2q6sfcgGl7cOspdv5pNmiNCg2t5hEJEZyyZtBvAiYXD84BnG3PALdUx8CEEEKIOmf7H7D41ZIXFjq01Wy7D/e0xbWBsJiy793lFDj5IQiJgGFP+NY77zwEblwKkU0rPvbGSAcO0Dt89hmxw4eDxUKbKS8S2r49oR06YGvduoYHKERgZdVB7w2UVs1lAfBA1Q1HCCGEqMNmnG62W3+Fyz7zP+9aIbRJBR/jCgmFB/dW7Frhz24HIKRlS5TVSvuZM9GFBYS2bUubF56nzQvP1/IAhQisrBn0ZkBp3w9pIKHqhuNLKXW/UkorpV71alNKqYlKqd1KqVyl1C9KqcOKXRemlHpFKZWqlMpWSs1SSrWtrnEKIYRoJKKdFVQ2zoGifP/zs+81W5nprhNcD4m2euIJusyfh61Fc0LbSjgg6r6yAvRdmFn0kvQGkqtuOB5KqYHANcA/xU7dDdyBKQN5FLAPmKuU8v7+cAowErgUGAzEAt8ppazVMVYhhBCNhHfg/Up/WPWJqXsOsGeV51wdqgSS/ccf5K1dW9vDqB3OAF1Z6s5/DyGCUVaA/j3wuFIqovgJpVQk8JizT5VyPoD6ITAOOOTVroBbgae11l9orf8FrgRigFFe144D7nKufPo3MBrzYeKUqh6rEEKIRmD7YpjzEOxbC/2vMm3pO+Graz11z986wWzPeTXgLWqCdjjY+9jjbD7zLOyZmRSm7GPHmLFsPX8EWb8vqrVx1Rb3Q6IWmZ8T9UtZAfqTQBywUSl1j1LqXOfPvcAG57mnqmFcU4HPtdYLirV3BFoC7qoxWutc4DfgOGdTf8BWrM9OYK1XHyGEECJ4M85wVlfR0PNsOPtl3/P71nn2vauv1LCcJUs4NHMmBZs3s+Goo9l04onuczuvvrrWxlVrnA+JKqtUlRb1S6kPiWqt9ymljsOUWnwKcH1HpIGfgPFa65SqHJBS6hqgC2bWuzjX8mnFXzMFaOPVxw6kBugTcPk1pdS1wLUASUlJ5R+0EEKIhiv3kO9x+0Fgi4DcgzBvoml7/Riz7XEWtB1Qo8MDcGRnoyIjSf9mVo2/dl2mnQ+JYpEAXdQvZVVxQWu9HRiulGqCCZwVsFFrfaj0K8tPKdUd80FgsNa6oLRhFb80QJvf7Uvqo7Weipm1Z8CAAWXdRwghRGORnwXPdPAcH3GhCc4Bjr4O0nbCsume88Ofq/H884Lt29l82uk0veoqCvdKBRhvBZs2mR0J0EU9E/SfWK31Ia31X1rrP6sjOHc6FkgE/lVKFSmlioATgfHO/QPOfsVnwpvjmVXfC1id9ympjxBCCFE6hwPeHOQ5vn8PjJzmOQ6NhIHjfa8JCauZsXnJW2vSaw6++y45S5cC0OGLz2k3fRrKZiOkVasaH1NNsmdlsfueeyjYscPvXMqkpwGwRAZYqVWIOqyufaT8GjgC6Ov1swz42Lm/AROAn+q6QCkVjqnUstjZtBwoLNanLdDTq48QQoiG7tdn4bvbKn59TqqnrvnNK0xAXlxiFxjktV5fSHjFX6+CCnft9Dm2xsURcdhhRA8aRI/V/9D25ZdqfEw1oejQIXbf/wAbBhxF+jezSL7Ns2yLIyeHtT16uo/Du3erjSEKUWFlprjUJK11GpDm3aaUygYOOiu2oJSaAjyglFqHCdgfBLKAmc57pCulpgOTlVL7MLPuL2DKNc6rkTcihBCidv33Nfz8pNnvdS607gfhseW7R84Bz358h5L7nfoYtD4Sfn+xVgL0gl27fI7bf/yRz3HEEUcQ0qIFRSmBv0TO+vVXIvr0wRofX11DrBYbj/Wt+6BCQshZtoyw7t3J+esvd7slOrqmhyZEpdWpAD1IzwIRwGtAE2ApMExrnenV5zagCPjE2Xc+cIXW2l7DYxVCCFHTtIbPrvQcv38utBkA18wP/h5Z++D1gWa/w+Cyc5gPO9/81IKCbdsJP+ww2r//Htl//klYR/9VTONHjiT19dfRWqO8cuQLdu5k53XXE3XCYJKmTq3JYVdK9tI//dpyV61i++X+9SVsbdr4tQlR19W1FBc/WuuTtNY3eR1rrfVErXUrrXW41vpE1+y6V588rfUErXWC1jpSa322s9SiEIbDDr9PgfzMMrsKIeqZ3SvMtu3RnrbkZcFdW5gLS96AXc7+Uc3hirpbGaXowAFy/vqLyIHHYImKImbIkMAdQ5x1wO2eeaqUSU+z+dRhADiyc6p7qFVGOxzsuNLzAazFww8Rf8nFAftGDRpE66cn1dTQhKgy9XEGXYjK2zgX5j0CB7fAOS+X3V8IUX8kO1f2HDkNvr8DNs2FZj1Lv8blSWcNgpAIzz3qaAWQokOH2DjoeACiBw0qta9yLtSjHQ53veRDH3lSYUKaN6uWMVYHR3a2e7/NKy8TM2QIKc8869evwxefE3HYYTU5NCGqTN38V0eI6ubKdsqUkmRCNDhbfoGoZhCfBJd/DondoVn30q/JS4eifM9xUa7Zhtat6h9aa3bfex9re/Qk9++/3e0hLcuo1OJcqKcoJQVHXh72rCx0QQHWZonY2rXDvr/40iF1V/biPwAzcx576qmokBBihpwEQNs33yC8d2/az5wpwbmo12QGXTROGbvNNjLB07b+R9i5FIY+UuN1jIUQVWjXX5B0rOfvcUgo2EtZWmPtt/DJ5eZh0uLqWIBuT0sj/euvAdh1ozv7E1vr0gN01wz65lOHEXvmmdgPHQSgySWXkL9pE/lr1lZqXPuee44D06bT459VqNDQSt2rLNm//w5A9AknuNuijjuO7itXYAkPJ+akk6r19YWoCTKDLhqn/74y22jn17r2IvjoElOF4dBW+OYmyDlYe+MTQpSf1rD9D8hKgZa9Pe0h4VCUV/J1mxeY7ZpvzPasKZ5ztgClFWtTUZFfU5f587CEl149RkV4zmd8/717Fjr29NMJbd+egu3bKdq/v0JDSp36NgemmcWacrxm9atL0f79hHXtSmjbtj7tZf0OhKhPJEAXjY+9ELYvMvuur7Tz0jznFzwJKz6Ar66r8aEJISph6Vsw43Sz33Gwpz0k3Dd9xSU/y7Qve8e3vec5nv06NoOunQF6WHdPyo41IaGk7m6hASqZNLv1VsI6d8YaFw/Ajmsr9m/e/hde8Bpf9RdLK0zeha19UrW/jhC1SQJ00fh41zYuyjOzbpM7e9r+/dxsdy6t2XEJISpny8+e/aSBnv2QsMAz6P8bCU80N/u9zgVbFPQ4C6IS4IzJcORoiGhavWMuJ+2qwuLwBMLBzBzb2rXzawt35mjHjxzhPO5VoTFZ4+I8B3bfGX5dUED2H3+gta7QvQPRhUVYqjmNRojaJgG6aHyyvR6GKsqH/IzA/fLS4eDWmhmTEKJyZt8HG2ab/fbH+56zhpm/67Pvg29vMR/KAXYu8fQ54W64ewtc9L45PuZaOPfVOlfBRRcWAuAoMDn1MWecHtR1gWqBRw08BgBrbCzW+HjSv/6GotTyPSzqyM7Gnp7uHof2KuOYv3kzGwYdz44xY0n/+pty3bc02uEAZ069EA1V3fqXR4iasMprlb2iPPNwaHF9LzPbrb/WzJiEEBWTc9DMhC953RyHRHiCbJfQSEj51/RZ/i5smm/SW7wldAZbeN0P/JwBcNPLRxN13LG0fPDBoC6zhIXRdfEievz3Lz3XraXnurUom8193pGdDUVFPg+eBqNw3z4ArLFmFl175chvv+oqHJlmrYmsX4P7t7Rw925ylpVRs95uR1klfBENm/wJF41LejL88arZj28PhXmeXPOB4yHM+VVtj7PMtqD+LN4hRK3Z+y8c2l47r/3FONg0z3N81fcmRcXb8bf5Hu/7D6afavZHvA0PHQBbRPWOs4q4AuCQFs1JeucdQoLIP3cJadoUZQ38ASSkRQsACpKTAVjboyfJd9zp//oFBaS+NZXMn3+mYFcyW84YDoAl0vkwrfMDRMYPP/iUbtR5pTyk6+TIzmbTyUPZfvnoUlNitN0OVilCJxo2CdBF45J7yLmjILo5bPCaPT/mOjh7CjTp6HnArFACdCFKpTW8OQheO7rsvtXBVYEF4ML3oG1//z4tDoP7kuHOTeZ47sOwb43Zj0+qV8Fe+qxvAVBVnHrT4ZOPAbCnppL9h6nwkvH993799r/6GvtffJFdN4xn78MPudtdAbouspOzfDnJt98BQPgRRwCQ9csv7Hv+BXdqTiDeCyfZ09JKHqzMoItGQP6Ei8bFFXBf9pmplexy+Eho0gEOHwG3rITQaNO+4HFY8WFNj1KI+uPAZrMtrYyht/xM+GAE7N8AaTtNVaWKcqWpHH8bTEyHw84ruW9YtKesqouyQLMeFX/9GqaLijj4jqk4o+2OKr13SEICYT3M72LHmLEl9jswdap731WqEaBg2zbnuIrI32T+TIR160brp570XPv226w/sl/A++YsW8a+5553H+ev31DiGCQHXTQGEqCLxmX/erMNjwerVxWAnmf79vNeqOib8dU+LCHqre2/l6//pLaweT68dhRMORzmPlLx194012zbBJg1D8YjhyAivuKvX4O01j5L3Ef07VP1r1FU8Q9Lceea0pRZ8+e7F0Hq8NmnhHXtShfv/HO7ne1XXOmXk777vvsBsDgrwmQvWlTyi9ntJabqCNFQSIAuGo/dK2HWTZDYzfwPvUlHz7nwOP/+3guUFAY5OyhEea340KRc1EdF+aYqisv8x2HJGyX3D7T4V0UexHY4YOVM+Owqc9yiHEu6T0yHk+6Ds18u/+vWktQ33mBdz17uoLbFgw9ic+aMVyVLuG8efsSRR/ocuyrIxI0YQUQf8wEh4ZpraHLFaKIGmrKWmXPnkbdmLZbISCxhYQDYWjQn6rjj3PfJ+fNPdl53vc+9w7p3A6Ddq684X6zkbwhMDroE6KJhkwBdNB5TTzTbNv1N6bQWXjV/w+P9+yd28+znlK/0mKgmWsOhbbU9iuAU5kLGnrL7fTMeFr3kOd68AL65MfDCOnVFfhYkL4cXeprjds6a4wufg9n3lnzd8nfNtuOJnjbvD8IFObDqY08ZxJJsmgtf32D2T5sETTuVa/icdC/0v7J819SSbZddzv6XzIeJ3XffA3g9kFnFmlxysc9xSKLvA6h5G0zaSXiP7nT45GN6rltL8ztup+X996NCQ4m/9BIAMufMIaKfbypL0jvT6bH6H59vJ9O++pqcv0yqoQqxEdqxI5FHHQXAwQ/+V/JA7fYqz8EXoq6RP+Gi8SnMNduzXvS0RQaohHDxB57lwrMrtgS2qEJ7VsGj8fBSH7NfF2ltFpBJ/htePQpePhLSd5XcP22HZ3/XMtj5F3xwPqz4H6z/oWrHtvdf+OHuyuV8A2TuhUlt4O2TPYt+nfZk6df8Ohk2zoO1s8zx6K+hg/NB7PBYT7+Fz5uqSuv8H070kZXi2T/m+pL71QOHPvmUA9OmBTxXdOAAucuX+7WralqkJ/6CC9z7lpgYHHn55K7+F4A9jz7KtpHmfEkfEGKHDfPca8T5fueVzUbXhb+5V0Hdc999bB99BQCZs2ejvRZf0vn5ZC5YQMG2bWy75FLyN24EIOWZZ3Hk5MgMumjw6s+j60JURqAUlYgm8OA+2PEHNGnvfz4+CYY/B+8Mg+wD/udFzXrPa/n1/euhVdXn4FbaH6/BnAd827b+Bn1H+ffN2ANTjvAcTxvqe/6zq8wDjM17Vs3Y3hxktr0vDlzppDT71sI3N5nqRtHFUivOfsmkmLRyfnBq6XxPBzabeuNbf4V133n6H3GR+Qbrym/hyZZm1vzjy8z1C58zfRa/DD3PCjwWhx1mTTD7d2+tcwsJBSvr11990jziL7kUa3SUT5/8jZs8BxaLSe0BbK1bVdu4Wj09CfvBQ2TMnk32woVkL1xIl19+Ju2jj919SnpA1Ttwjxo0KGCfkMREmt1yC7vGe57tyVu7FgCFmV2Pv/QS0j76mF3jb3T32ffc87R7600OzpgBgD09rWJvUIh6on7+yyZEee03/wOgVR84/WlPe0gYdDqp5OuiEs1WUlxql8MOBVnQaYg5Lm1WujZ555InHWu2mcXSXDL3wu8vwvzHzHHx9Kq4JM/+6wPLTvcIhvfy69NOLv/1rw+E5GVm3Jl7TduJ98CEv6H/VaaG+HW/mYet7UXmA9Qr/eDHu3yDc4DDnDOrSkFEU9ix2PT52WsWfudSs2ZBIFt+8eyHxQbuUw8cmDbd53jDgAF+fewZ6QB0/Por4keOdLeH96yiD20BxJ93Hgljx7jrmQNsOsn8vYvo1w8VEUFE374Brw3t3IXQjh1Jeu89rHEBnutxihp0nM/x1vNHANBk9GgAmo7y/0CbvXSpT4lGR0YJK0AL0UBIgC4ah2Tn18QX/w9iyzH75ArQvb9SL4/0XXU7l7g+cDhMqoijCHo5Z9HnPwrZ5fjQpDWk/Fc94/MW61xO/fZ1MHa2+Zbm18mwd7Vpz8swVUvmTYRVM03b3VvhnFc897htNVw933Ocsbvy43r9GN/j3SuCu+6vaTCxWKCVsdu8zyH3m9U3vVls4CiEf7/wbW/V17PfwWtm9bgJJb928YdHiwpg2yLP7+OEu+pV/fLidIB64Dl/e/675CxfTvLN5gFca1yc+5uCFg88gCWi+hdVCjS+JpeNoseKvwnv3i3AFWCNjqLzjz8QdUzpNfEtYWG0eOABv/bIo8yHlLCuXUma8Y67PWrwYHReHnsf8nwAbn7XXUG9DyHqKwnQReOwZ5XJM49rV77rwmJNqsuOpeV/zR/ughcPgx/Ng104HCZAc+XAi9IVFZjg9pn2kOZcpbL7mZ7zsyYEP7v851R44zjYsaTqx+nNYjEpJK4Pgc17QVEuvHm8mX1+pgP840kVoOMJnmuGPgxj55j2tgM83/R8fQMUZJu89A9GQG5a+cd1YJPv8ZYyKqdkp5rA/Ps7/M+t/hRUCfm/Vpv5M57yHyR0NRVTHkmD636FSz6Cy7/wrZh0zPXm71eLw03qzLAnoNe55tzXN7hTOgD49mZ4d7iZyQcY5FU9ph6yp6URc9pphHbsSOzwMwDYPmoURYfMYmr7X3nV3dcaG0vMya5Z7CP9b1YNwg/zr4wTWezBz8poOvpyEq6/zqctrJsn8Le1aePpe6V5oDf9m28AaHHfvYQmJSFEQ1Z/px+ECFZRPvz9vtn3rm8eDKWg81BY/ZlJsyjP4hh/Ohf0WD4D9v4DrfvBX2+btonp5RtHY1NUAE8UW1Rm/BKIaQFX/WACtfU/wIInYOhDvv0cDlg23Sw+FdnUtP39gdkWZFOt7EVg8fpnNaELbHfWc5430dM+Zrb5RiDJWf0kJAwGFwuGB95gxr1nJcwYbrZgyhqe+yqExQQ/rvA4U5XozBfgrcGeb4ZKEuiDzJXfwXvOvPD0Hf7nwaS6ZO8zKSudnak0rr9zPYb797dYTJqMsnpyyR0OUx0ma69JrTnnZfNN1CrnKpPL3zW/Y9diYvWQLiykcP9+Io87ls4/moeB7WlpZC/+g4xvvyWiTx9yliwhcuBAIvsdiSUqiugTT6T7yhVYwsNrZIwtHnyAkGbNOPC2+Tcr5tRTsbWq2tx3ZbO599u88jLK69/nkBYtsERF0XLiRL+UGmt8fJWOQ4i6SGbQRcO388/KXR/bxuQ/lzfvOcTra+jk5Z7gHOCv6dUfLNZXu5b5BufDnoCHD3oeluwwyBOcbZzjf33KavjhTvjK+QBezkHPsu6Wap6TcBQL0AffDu2P9xxHtzAfztofax64tNr87+Gtz8WQl+4JzgHWfG0W+ykPZYHWR0KU8/f6zY2l9w80rg5e76PbGYGvO+7msvsEei3vBz0tFrjQPAhI6np45zT4YpzvNVHNy/9huw7J/OUXdE4O0cd7fqdtXjZpTilPTWLbxaZcYbObbqTZzZ7faU0F5wDW6Gia3exMQbJaaftK1deNL0rZB0DizROIPfVUn3OWsDC6L19G3NlnYY2OotMPnso+IS2r7yFZIeoKCdBFw5aX7pn1u62COciuXNiXeptZ9GCVtrrh97eblAXhS2vfaib3bDd5ysW/ubhnu0k/ahcg19WV/7zxJ1O279mOoJ3/3ez+ebVVylHoG9w26QBjvjffwkD5SwL2Os+zH6hWf7BcM/vez1/sXllyf1ca1g2L4a4tcOcmExC70lNKyh33zkkfUPJy8WUqqa55oinPx8AbKn7vIGmt2X3PveQsW1Y197PbKUxOxlFQwMHp7xDSsiXRJ3rqwRev4AIQ3quXX1tNUjYbrZ58go5ffVkt9485bRhhvXrS9LLLyuwb1qkTEX36EHPqKUQefVS1jEeIukRSXET95XpIsLSv65c5Z+IGjIW4cs46unjXSP9rOhxzbdnXZO03Na67D4dLP4IXD4f0nb599jkry2Ttg8jEelsurkrYC021E9cDtYnd4YLpJS/DbnWmOBQVK59ZmOe76I+rUorLwS1VNuSAiqe4uLgquXQZ6n+uNN7lP6/9xeTifz2+/LXMi8/sA/xvJNy9OXB/13+HkHCI8vrzf9TV5kNPaQsDXfiu+fBUmQc4Y1rC9Yvg6+s9D9gCXPaZ+W/erHvF7x0kR2Ym6d98Q/o339Bz3dpK3atg+3Y2n3Y6YGqY64ICWk58BBXi+zvq8PFHbL9qDDrP/LmurgWJysO7ekxVix40iOgSyjEG0uGTj8vuJEQD0YgjAlGvpayByZ3hzcGl99v9NzTt7LsoUWUEW83luS4mTzempTl2BednvuDps3yGyad9rqup+9xYHdgMjyeamuCvOkvNXfapp552SWzh/vXtXauMNvd6wO3yL0zeOsCPd5tvVaqLozBwgO4qLdikY/nvOXC8Selo2tGUBO08pOzUGG/ZqeZB1eLjKl46NHWjp0JKpnNb/MPvkAfh1tWlV0I67PzyfxAJpOXhMOpT6Hu52Y6bZz6w1EBwDuDIDpyClrXwdzafdjoH3//A71xhyj4yf/7Zrz191rfufV1QADYbceee69cvom9fuv+5lKZXXkmn77/zOy+EaDwkQBf1T9pOeMNVY3p34EWI3H13BF6EqDz6jfbsuwLu0njPbnY5xWxdi7v0KvY/5W+dlSjWfF3h4dV7/xX7+jw+yaSGlCUkHP79HH6bDAe3mqojrnKC575iShdGNDW1070XNapIRZ5gHNoOhTmQusH/3OA74f49vqtmBuv0SXDXRs+xNdSk6vw1HT68CL6/0/NtTHH2IvNBFgI/4JyZYu6Tvst8OHqhp/mQk7rJfCjwrrgC5lue+BqsnhHbGs57DbqdBu1qLq3hwPR32DTEUy++YFcyhcnJFOxKZuc111CwfTspTz1lVrTElEdc26Mnm048kV03jEdrjSMvj7wNG9AOB4XJvjXdY08/vcRSiSo0lBb33UtY584BzwshGgcJ0EX9UpgL391m9l0PChbmlNz/0HaIr2SA3uUUuMlZRz17vyk9V1ptc+9lyruZr7UZNxdGTjczkmPn+Na5BlOXel0VL+1eXyws9u1Gu2MC9yvOleO84Al4ua/vuaadoN8VcM9WE5iGRcMDe021kFk3QX5mpYftpjVsnGueUYDAC19ZLBBaRekKrgD9+9tNnv1fb5tqJ64FhLx5f+OzxpSo455t0M+UreO9s8x9XvT6xuGlPnBgIyR2rZrx1jPp337HvsmTfdo2n3IKm4aewuZTTgGvyiMFO3eSMXcumfPm+d6ksJD1fY9k6znnsq7XYaR//TUATS67DGt8PPEXXFDdb0MIUc9JgC7qD3sRvH4sbJprVmk8fZJpL8wxy6mv9fpKePb9MLkL5B6EuDaB71cernza36eYxVtWf1ZyX1cKxXULPbOWTdrDEc7/KScdY+pcF18F8eNLKz/O+qjQmUpw41+m4snR15Xe3+Woq0s+F9HEv80WYT7UZaWYdBrvGtuV8ePd8KFXwHVsGRVSKis8LnCazvPdzbcIn43xtK33+tDnqp8e0QTaO1dyDDTbD6byUUKXKhluTdN2OwU7dmDPLN+HsKKDB9FFRex2LoAT0rqENJ7CQmLPNPX4Ux5/guQJN3PwnXd8uuT+6/9Aesxpp9HyoQfptuSPMhfyEUIICdBF3bdpPrx/HvzzCRzaatoGjgebc0ZyzTfw3tnwyWWQfQDWfgtLXjOz3QAdTqj8GKyhZmt3zpx7P7hWnKtiSFSzkvuAWdUUoGXvyo2tPnM9xNvrXGjWzVQ8CTaVQSnzzUab/uYbiQdSoMdZcNnnJV9zqXP1ztxDppZ6VXDVuwdTq726DfAqOTh2Dhw52vf8f1/CBmf5yX8+NdumneD63z19An2A8aHr5Qx6/pYtrO/Xn83DTmPDUUeT8uxk1vboWWawnvvvf2w8bhDrDj8CW1IS1sREOs+eTWgJaSbxI83S9CVVeEm+7Taf43bTp9Hm+ecq8I6EEI2VBOiiYlZ/Dlt+Kfl8sCs8BuP3F2HLz/DNeHN8yUyz5HtbZyD30/2evpM7wSeXe45PfdzMWFdWaLESaEvfNDP6gbhKMZZVc7vTiaYm9oXv+p9b8T9PkNWQfXer2Q5/vmLXJ3aBaxaYbyRs4XDJh9D11JL7dzjek/K04w+Ta6112eUz92+Avf/6t2c6U0iU1VRZcdVqr04xLWDMj6b8YdIxMPQRz0OxA51/R2ZeZBZ7OrgF+o+Bm1eY61y8SzaOmws3/mk+4Bx1jae9hf9KknXdwRkz0Pme9DPXzHbWLyWvnJq3Zg1pn37qPi7csYOEceOwhIaS9PZUood6Hnht88LzxF90EZHHHut3n5AWnt9vUUoKoV060+OfVXSe8xPRgwb5VWwRQojSyL8Yovy09iwccl+yye/19ttk+PVZOOMZE/h0GmICqYqwF8Gef3zbXAugNGkPzXrC/hIekLt+kVk9sSoUf1gOYM4D5j0W5w7Qg1x1NKGzmUV3PUiasduzkExDXXE0Y49JP4pqZhZ0ii7j24aqlLbds//T/WbGfsETMPYn81Bx74v8r3nN+WHw0o+hu9cCPLv+MtsLppuFgGqKK0UFzO9u/GLP8c6lZmGs31+EnAOeP1fevMtXtunv+bM67HHPglodyqiQVAdpu0lbanLZZRz68EN3u/3QoYD9d15/A1m//OLXHn28Kf1na92adq+9SuHevTiysgjr0oXY4eabl8gBA8hZtowmV4ymcFcyrR6dSO6//7LrBvMhqeWDD6FCQ2VJeiFEhUiALsrvr2me/f3rzOylty2/mofYXA9zxraB29dU7LVWvA/56dD7ErOE+EXv+9YLH/+HGcPSN03JQoC+l5n89EBBdVW44Q9TRaakWVdXiosqxxdUIeGwf70pOei9JPyh7ZWvQlPXZO6FF3p4jg+v4Qfmjrne/HkBU71k409m/53TzLbFYeZn8wJTMcjm9XDngiehy6meZxKSl5lvSlwPA9cFl38Bz3SAX54yx9HN/ft4z6B7f5C0mcoi2gFpn31J3PnnYQkLq7ahVhVHQQE7x45zp5y0fOhBnwA95dlnCeve3Sf3WzscAYNzgNBOvnXebS39qzclvTuD/I0bCe/p+dYkymtmPWpgFXxzJ4RotCTFRQRn03yY/7jZd1WDALPq42MJJtABWDMLti30vTYsJrjXyDnonzay1Xmv896A0V/530spk1Zw9ksm/xjgjGerJzi//As493Vo0cvMzLsWnymuvDPoLuk74JV+sHaWp+2lBpifvsz3gbqAM7zV6ehrwRZlZoj3BVhddqMzteiD8+Hbm+FLr4dRU1bDz096jrf+ZoJ5W+CSebWieH550wC11119XGVAAXtaGkWpqXDhu2R3e4C9Eyeyvk/f6htnFXEUFLBj9BXu4DzyaBOEd130O91cOeJFRey48kqf63KWmnKbTUaPJvLoo+nsrMQS0rIlylr2310VEuITnANYwsNpM2UKbV6oYMqWEEI4SYAugvO/EbDwOZPfvW2hWTzExVFkyrQ5HLDkdf9rC5xVOjYvMKkNgRQVmCXZH0/wVGPZvcI88NbrvOBW2bzwPbh/t3/KTVXpcgoc6VySOqYVrP/Rv8/6H81MOJi85GDlBv4KHjAPvjYkKf+ZlVMv/dh8kOpbw9VrEjrDA7tLzlXfODdwhZeTHzTb3X+b7eJXTCpJbAVXqK1Oh43w7CcGWNgnJBRu+w8u/QQAXVjIhoHHsu3SUXDY+ejWnm/F7FmBF+ypbUUHDwJw4K2p5K5aBUCHzz4l6V3z4HFIQgLW6CgSrvHk1eetW0fh7t1orTk08yMAEsaNpf377xHatg0dv/majl9+UalxxZ5+mjsNRgghKkpSXOqKf7+AX54xs7Tx7Wp7NL6KCjz7a50r4h0xEoY+ZGbM5z1iSh0ufdMsctL6SFNdYs4DplzbnpXwSn84sMnUiL7CawZ+1zIzC+/tk8t8j48JsuyeNaRyy4uXx1bnQ2fbfjcPHoIJPD+6xNOnPDPoUYmm9rTL4DvNSpqfXWlmcEd/Vfkx16Y/XjMrWvY613yI6XOJyeW+Z7v5FqQ2xAQoo3fsTfDHq/Dbs542a5ipoW6xwOovTKnGtd/CHGfAPvShmhlveVw4Azb8ZNJvSiozGuf5YFGwy3wDVrjTrHib/tXX7nO777iDdm+9WW1DrYjsxYvZMXacT1vC9dcRcYT/6rPNbr8NlOLA1KlsPc+s6Jo44SYy584l4epxPukr4d1rZpVSIYQoi8yg1wUHNsPnYyF1vSkXWNe4Shu69L7EPPgZ0xKOuRaudAbtP91n0j5aH2lm6IZPhhPvMecObDLbLb+YhyC1hq9v9A/OvWf+wMz+tRtY5W+pyuz802y3LoQ3jvM9V54Z9JOLBXk9z4Z2znzZzQsCVxCpDw5th/fOMQ9j/v4CTD3R5OifMtGcr63gHMwHhBPvhdvXwZ2bYMLfcNwEc+6XSZ5+pz3p+QYnPA7WfeepFHTSfTVTuaUibl0Nt5VSDtSLa0VMgOylf7pnpwFyV66s6pH5yfhpDlvOOx9tL7majj0jgyLnw55F+/f7nIseOpTmt94a8DqlFFHH+f7dzJw9G4CI/v0rMWohhKg+MoNeF6R6zZzmHiy5X2055Kx6MeZHU22keAqJ9zLqACd5lT3sMdwE3d7Lub/QE2JaQ+ZuT1u/K+CUR00A1HUYfH29aR//R3DpLTXt+t/hzePNNwBgVmQsrjwz6B0GmYot+9aZGdzmPSEkDI6/3QS2ycug5eFVM/aasuYb+PQK//YWRwR+cLGmhcXAkPs8x65KMvFJppqLS6/zPPvtj4OdXrXOvcsS1jVRCUF31bm57v0dV16JCg0lpHUrinbvIbRL9S9YlHzLLQCsP7If7d9/j4i+fX3O52/dypYzhmNtlkjS9Onsvudec8JmI/H664g5tZTSmkDk0b619fM3mgmD0A4dqmT8QghR1epg5NMIZSSbbc+zzQqBgZbsrk2ulR7D4wPnd1ttcO2vMOhWuHurf8m8EW/DuHmmjJ2LKzgf8oAph3jOKxDZ1AS1fS+F+3aZILi8D1rWlJbOr9LXfw/b/wjcpyKzw817wLmvmuAcYMj9gDIP4U6M86RV1AVbfoH0ZP/2A5vh0yv9g/MT7zWrU17xdU2MruIiE822xRGmjKj3n+fjvRagOfmhcgXBtc2Rl8fOG8aT/Yf/n9eiVN/nHHRBAc1umkD0SSf5BO/VTRcUsO2SS7FnZbnbDn3yKVvOMDnd9v2pbD3nXPe5rr/9SrMbbyS8W+nlVJXXh3zXQ6QgAboQou6SGfS6IGO3SYdo6izt9Xz34Otfb1sEX11nFmgpPpNdVYqcC3+ElFJurXVf8xOINcTUmg704F2/K0yqTHFhMZ4guK4KCYeiPJjhVWKv3UDoegpk7aua17DaTK70b5PN8eJXTIpRbc+mFxXA+85AyfvP6vrZ8NHFnuOup5kVXQ9uhpPu9Z2xrqvi25kHQWNb+38gDY+Fm1eaWup1OfXKS+7q1VhjY8n4cTZZP/9M1s8/03OdZ+2AvA0bSA6QHhJ++GFk/byA/C1bOPTxxxQmJxM7fLhf5ZLK0gH+XTj0vw9petWVHPrf/9j3XMkVUSyhoUG/TsxppxHaoQPNJtzEusOPQIWFoWozxUoIIUohAXpdkJFsgjDv0oCTu8Bdm8q+9tdnIH0n7F5ZPQH6xrmw9C2zX9lSchYLDH8O8tIgLskEmYGC8/rijGfg21s8x816wBlPV/2CNU06+KYDLXgcRn1Sta9RXqnrPfu7lpv3rBRs/92334Xvmj839sLazTcvjx5nm/Sckj4gNu0YuHRhHaTtdrZd6L/wki4oQIWGUrR/v8+MdIdPPmbbxeZB59C2bclZ/jc6L4+9Ex8FIHvxH3T84vOqHWORKa3a7NZbiT55CFvPOZf9U6awf8oUn36JN91ExuwfKdi02d2mIiMJVtuXPPdLemc6ttatKzVuIYSoTnUqQFdK3QeMALoD+cAS4D6t9b9efRTwCHAt0ARYCtyotf7Pq08Y8BxwKRABzAfGa6131dBbKZ9TJppZRu9Uiez9JXb34Qqad6+A/leW3G/DHJh5odm/Y4Pvst8lyc+CD70WkamKWs9H1+Gc3fLqNMT3+Mal1fM6xetax9aBwOI/r6oyn11pvjFoOwAinSkfFhvcvRlCnQFUSPAznbWu94Xm2Qlb8MFfXZXxww8B2x05OVhDQ9nnFQRbExII80oVsURGYj/o+0xM0aFqeEamsBAAZQvB1tq/4oyKjKTz999ha9WKmJOHsHXESFo98TjhRxxR4Rnw4g+NCiFEXVPXctBPAl4HjgNOBoqAeUqppl597gbuACYARwH7gLlKKe8VbKYAIzEB+mAgFvhOqfKU1ahBMS3NbJ13EBNbQmk0l5Q1ZoGg7c4lvpOXldzXYYfPx3iOi89yFrfkTbOYzILHPW22KN/VB4XvCp+dTqq+14lK9D3eNL/6XitY/3xqHuY9bIT5BseeD9sXmUWWug6De7ZW30quNSE0qt7M+Odv3UrBrgDPAgAHpk1H2Wzu4/hLTPpRzooVgKl/7pL0zjtYIiJoM+VF2rz4gs99wnv1Mjv2AGlqFVC41/Ocjbtyi9WKNTqKrn8shhDP3FH35cuwtWrlHkfPdWuJv+ACKYkohGjQ6lSArrU+TWs9Q2v9r9Z6NTAaaAYMAvfs+a3A01rrL5wz61cCMcAoZ584YBxwl9Z6rtb6b+d9egOnFH/NOqXAU+qszBJ9bxxrFgjKzzDHe1ebhwhXB/j6ecnrUJDlKSH3+djA+eAA636A2ffAd7d5lkO/fS3c/l+9CVhq1ICxZgGb0V9X32sMfQSSjjU/YPKfN8ypvtcLRGt46wTzk7rJBOUdTwhcA7z3xcGvHisqpejgQbacMZzNp53md86Rl0fB9u3EX3oJPdetpee6tTS5xKSv7LphPAfemUFh8m7Cunenx9o1hHc3s+exp59O7Bln+Nyr/f8+oOnYsdjT0tBaV2rMaV9+xaaThpB8992AJ8VFhZgPEiFNmtD97+XEnnkmXRf9LnniQohGqU4F6AHEYMboWmaxI9AScEcnWutc4DfMrDtAf8BWrM9OYK1XnzrK6398+ekmKAokM8X3uOswz/7ch33PZR/wVP7o4lWKLNAS5wD/FMttbtbTpFQUT7MQxlkvwgl3Ve+Hl6gEGDvb/LhklrAia3X55kbYs8r8vOqsHd3zHPNg8xXfQMcTPX1b9q7ZsTVSuf/+x8bjBpkDu53MBQt8gueshQvReXlEn+j5bxPSzFORZt+zz5L3779EDhhQZhCsIiKwNolH5+fjyM7GkV3x1UUzvv/ebGd9iyM7G13oCtA9s+aW0FDaPP8cIQn1p0qOEEJUpboeoL8ErARcydmuJwqLRaikeJ1rCdiB1FL6+FBKXauUWqaUWrZ/f5C539Wh/xgYdAsMedCUW3Qt//7L05562wDPO/NEj7raVJQ47w3POVellcI8M6M+2VkZpsdZ0OlEs8InmBre+9aaahxbfvF8GHCl1igLnP40XPR+dbxTUVmWGnp85Ie74NnOsPJD3/aErp4Un04nwZWzzAPAlhBI6FwzY2vktl1wgc/xrvE3sq5nLzLnz0drTfKEmwGIPMpTA7x4wKvz8wntWPIDr02vNM+1KKUISTCpVlvOGM76/gPYcZ3/Cr95a9aQ+uabbLv4EuzpgStRaYdnMaLM+fOhyJmDHlI3MxCFEKI21NkAXSn1AnA8MFJrXXx5ueJTyypAm98tS+qjtZ6qtR6gtR7QzGuGqcaFRsKpj5kybwA/3mMWCfplErxzun//3peYahJRiXCDMxc9oQvMfQSeLPYQ6Al3mW0zr7zN1wfCu8NNubx3zzQB/ZLXzD0eOQQDb4BmpdcXFrVkw4+w8iNTc7yiDm411VVK8+dUyHF+1j3RuThM815m1ry4o6+Bhw/U3dr1dVTGT3M4+MH/ANBak/nLLxTs2FHGVR7NipVI3HXjTex50FMvv3gpQhUe7nOs8/NKvHeL++51l2QMc6bAuFbxzP71NwpTfOdKto4Yyf4pL5G7ahW7br6FXbfc6tOnYPt2cv5YQuQxxwCw++572DpiJAAhLVuV+V6FEKKxqJMBulLqRcwDnidrrbd4nXI9WVR8Jrw5nln1vYAVSCylT93WxplCkLrekwfuKDSL1YB5+O6oa0xtcZcWh0HrfrBxDiya4nu/MbM9Ncoj4uGWfzzndv1lttsXedq6D6+iNyKq3NBHzHbtt2a11Vf6Vew+e1bBy309f76Ksxf6lpDsNASOvRHu3Wk+DMaV8RCzKJUuLCRvzRp2jLua5FtuIeXJJwHIWbKEXdffwOZhp7lzs70VHTqEIyeHwj0mxanJ6NEkXD2Olo89SstHPOlt6V+YlXsTrr3W7x7d/1xKu7enuo/ji83ElyTQYkAZ3weuEgOQs3QpmT/9RPpXpuJPyuTJ7H/pJTPuiy+iyahRAO6Z9vDDegU1DiGEaAzqVJlFAKXUS8AlwEla63XFTm/FBOCnAn85+4djKrU4p4hZDhQ6+8x09mkL9AQWV/f4q0RiV7N15fy6rP0W2h5l0l8C1WHuMdwssOLt/j2eUncuTdqbGXXX4jcuR18H3c+A9oMq/x5E9Rh8OxzY5JtyMjHOrHB5ysTg77PGOQPu+oBW3Dc3ep5H6HclnPNyhYYr/KW+NZX9L77o1762R0/iRoxwH2ctXEjMEE8pT11QwMZjfR+jyfrtV1o+cD9NLjK1znVhISlPTXKfbzbhJr/XUaGhhLb3VCCyxgVXbUfZbGC1gt1OiwcfJOWJJzj47rvEnDIUtCZv3fqA1+2f8hKZC34m7x/PxEDksccS2rkzh2bOBKDVU08R0kSecxFCCJc6FaArpV7DVFw5DziklHLNlGdprbO01lopNQV4QCm1DtgAPAhk4QzGtdbpSqnpwGSl1D7gAPAC8A8wrybfT5Xb9jvkOJfk7jvK//wJd0G3001ZxcRu/oG5t+MmmDrniV0gsTt0HFw9YxZVLypAGtbvL5YvQM9NM9s1AVJVtPZ9WHj4ZP8+Imhaa9I+/Yy8f1fT5PLL/YLz8F69yFuzBoCsn38mrGsXCvfsZf8LL/gE6Ptffc3v3k0uudTnuOkVVxB79tmkffwxEX37+pRY9GaNja3Qe2nz/HMk33obcWefRcoTT1C0bx+bh/lWkGl2661+iwx5B+cJN1xPSJMmplrLP6so2reP0LZtKzQeIYRoqOpUgA6Md26LF3p+FJjo3H8Ws/jQa3gWKhqmtc706n8bpob6J3gWKroiQC573XXx/+CTy83+nRvN/rrvzA+UXFWlpNUPiwuPM6teivonIj5we2Ee2MIDn/OWvNysEOvybCe4a7OpRGMvhMe9ssNuXul58FiUW2HKPgqTk9n7iElNSvvMlEFtfucd5Cz/m5aPTqQoJcW92qf90CEiB/THkV9A/sZN5K1dS3jPnoAJ3qMGDSKsR3cOTn+H1pMnE3f2WX6vGdKkCYk33FDquCzOAD2kefNyvZ/Y008ndp15HqbNlCkkF8t/B2hy+eU4cnOJPPooivbvJ+XxJ3BkZxN75pm0fuZpv2otEpwLIapTXqGdq2b8SZ+28USFhRAXYeOKY9vz1Ypk7vniH964rD+n9Api8cYapipb07ahGTBggF62rJRFf2qK1vBovNmfmA7f3Q7LpnvOTwxcIUE0AstmwHe3+rcHu0LsRGdKQ1Qzz4q1fUbB+W/A+h/hI1MrmwvegcNHVsmQGxN7Who5K1Zw6OOPyf71N8J79/aZQVYREXRduBBrdBRgZtiTb72NzJ9+AiDp3RlYIiPZdpFZVKjrH4sp2LKF7ZddTvyFF9Lq8cfQWle6PrgjJwd7Zha2FuUL0r1tOfsc8jduBKD5XXdia9OW2NP9a7ILIURtuefzf/hk2c5S+8y7/QS6NK/59TuUUsu11gMCnatrM+jCRSk4+yVIc/6h8s45dz1EKhqnkr49yUktOUBf+ZG5rrtXNaAB4+BX57coq2bC2VM8wXlCF+h8cpUNuTHZfPoZ2NPS3Meu4LzLb78S0qwZOi8PS0SE+7xSipYPPkDmTz8Rc9ppRA0c6HO/jcceR1jXLgCEdurkvqayLJGRWCJLSYMLQqdvZ1F06BD2AwcI69Kl0mMSQoiqtD8zv9Tg/NVRR3LTzBW8PH8TL196ZA2OrGwSoNdl/a/y7Pe7AqxhZvVG10OkonGKdgbhR1wIqz/ztG/73VTz8T4uyjeLCH19vWm708x2knQcnHQvdDsN3nbmOf/6rOfaCcurb/wNjLbbwWIBh4NdN9/iE5x7sznTSZRXcO4S0qwZ3ZYtQ1k9hbW6LvqdjYOOByB/4yYAmoy61O/a2ubKJxdCiJpQaHewNz2PuEgb36xI5vPlu7jllK6c1K05Fovv5MU3K5MBGHd8R87t25qE6DAcDs32Azn0bBVDQnQYETYrHROjauOtlEoC9PoiPA6O8S+ZJhqhpIEwcjr0PBuGPgwF2TD1JNi72tPHXmRq24N5nsFl6Vtme8RI8y1Nm35w5bfw3tmw8Dlz7pRHa+RtNBTJd95J5o+zfdpUeLiplKI1+Rs20OaF58u8jyvlxSUkIYGe69ay/YoryfnzT5qMuhRLmDwPIIRovArtDo6dtIDUrHyf9rHvmtTkH28ZTM9WsezLzOOSqUvYsj+bzs2ieOgs3zKu7Zp6vj0c2rPu5Z+DBOhC1D9KwRHO2tXxSWYb3RxWfGBWok3s6lvffPPPnn1XED5gnKet4wn4rOM1YEx1jbxBypwz16+t/XvvEtGnDxk//EDy7XcQFqCGeLDavPA8hXv2EnHE4ZUZphBC1Dk5BUX0njiHw1rHMuywlixYt4828RE8e0Fvwm2+i97lF9k546WFPsF5XISNh8/qxR2fmZLUZ768EEexRyujw+pnqFs/Ry2E8JVhFq5h/mNw8Qcw5wHPuWXTIaErHHCmt0Q1N0G+D+e/aEMeMN/WNEC6qIjsRYuIOuGEKsnhBijYuRPsvsWh2rzwPBF9+gAQO3w4USec6Dc7Xh4hiYmEJBZfd00IIeq/x75dQ5FDs2pXOqt2meIXy7cf4vwj2zCkh+cB9l/W7+OqGZ51O0b0a8OF/dtxeJtYYsJtjOzflls/XsHXK3e7+1x/YmcWb07l8fPq5+SGBOhCNAThceYh0bWzYNFL/udb9DIPfm74ESwB/tr3GWUeFM3P9D/XAGiHgx1XX0POkiU0GTWKlg8/VPl7au1epj5pxjuEH9Gb/PXriOzv+xB3ZYJzIYRoSIrsDkKcz9qs3JnGx395HuC8+viONIkKZfJP6xnz7l8c3bEpYwd1IDbC5hOcD+qSwAsX9fW799jjO/L7plQuPqod15/YmZjwwOtA1BcSoAvREPS+CJa8bvbnOpd8P+NZaDMAUjdAp5PMCrQbfoTM3f7X97/KBOit+9bQgIOT/ccf2NolEdq2TaXuk/rqq+QsWQLAoZkzqyRAz5g1C0dmJk1Gjybq2GMB/IJzIYRo7AqKHMxatZvJP61jf2Y+HRKjCA+xsmZPBgCfXX8sA9o3QSmF1prJP5lVif/cepA/tx4EICrUym93D+Gf5HS6NIsO+Dq928az7MFTa+ZN1QAJ0IVoCIY9YSq2eNfKb3kEtO1vfgBinAvzthvof33SMXD7Ok+fOkDb7ewYMxYVGkqPf1ZxYMa75Pz5Jy0ffABbm+AD9tz//iP19Td82rKXLCVq4DEVHlvO33+z+557sTZLpPmdd1T4PkII0RClZOQxcNJ8EqPD2J/p+0Dnlv3Z7v3Prz+WAR2auo+VUvxw82CGv7yQsBAL+UUOIkOtfHb9cSREhzGke8XXbahvJEAXoiGwWKFJe9+2+GLHSsHdWyGkhNVGY1tVz9jKSdvtaLudtE8+NccFBex/5VVSXzNL3W/65Rc6/zSb0KSkEu+Rt349oe3bs+Pqq8ldZkpGxp17LuF9epPy2OPsfewx2jz/nHuVzvLa86CZgW82frxUVhGiFmmtySt0EBFqLbGPw6FRyrN+QFUs9CUCczg0Ez5ewff/mOeivIPzyRf05oL+bfnx371k5hUysl9bd7qLt16tY9n29Jk1Nua6SlYSLabOrCQqRHn9/iLMm+g5rqerze6+517Sv/mmzH49/vsXZfX/n3LGjz+SfNvtfu3dV61EhYayrqen3Fb3v5ebuuRaoyz+/6Pwlv3nn1giIghNSmLLeedjjYmh49dflXmdEKL6vDBnPS8v2MT0KwcELJdXZHcw8o3FRIWF8P7Yo3n/j+089t0ac+1FfTizdyvCQjz/jmTmFXL59D+ZMKRLnVz+vS5LyyngkqlLWLc3k8PbxHLlsR04p29rn9+v8CUriQrRGDT3WqTItSBRPZO/datPcJ5wzTXkrlhBjvNDc/fly1jf3/xbtu6ww0kcfwNNRo1yVzkp3LfPLzhveuWV2Nq0cc90q4gIdG4uANsvH03B9u04srNp9fQkHOnpNL3ySr9x7XvuOQ5Mm+7TFnf22RKcC1GL3vx1My8vMIt4jXtvGe2aRrDzYC6dEqNo2zSS3zbs9+nf5YEffY5v/3QVt39qyvNFhlq5ZWhXViens2pnGlPmb5AAvZzOe20R2w7kcMvQrtx6Slf5lqKSJEAXoqHoNgyu/RXCY01d9HrEnpHBhqM9OeGtnnySyP79CO3QAYA9Dz9CaIcOWKKi6PrHYjYeexwAqa+/Qfo3s+gyfx4AGd9+C0CzW28l/sILSPviSxKuHucTSLd++mmSb7kFgLw1a9zte+69DwBLTCzxI84n5elnCO3QgSaXXEza51/4jTkkMaEKfwNCiPJIzylk8k/riQkLoWfrWP7cepCdB80H7y2p2WxJzS7x2sX3nozdYR5GnLXKPDSfU2Bn0o/r3H3+Tc7gs2U7uXBAu6DHlJVfxOfLdnL+kW2Ji6zfFUTKa+XONLYdyGFgp6bcdmrF130QHpLiUoykuAhR8zLmzCH5ZhM0Y7PRc/U/ZV6ztodv/nho+/YUbN+ONTGRbr8vLPE6rTW5y5aR8txz5K0K/DodPv+cbRdc4NPW9MoriB46lKxffqVg2zZaP/Uk1vj4MscphAhOfpGd06cs5Ny+rbn1lJKDvImz/uPdxdsA+OKG4+jfvgmrd6Vz9qu/M7JfW774exen9GzOUyOO4J+d6RzbOYH/dmcQYlX0aRuP1eI/szt/bQrj3jP/7//42oFcMtVUfereIoYHz+rJ8V0SS50R/uCPbTz0zX+AWTznqA5NScnIw6LgimM7MLJ/24r+WmqMw6FZvuMQKRl5tI6PoF9Sk4D9MvMK2Z+ZT5jNyhPfreGfXekkp5kPR4+fdzijB7YPeJ3wV1qKiwToxUiALkTN2//aa6S+8ioAoR060Hn2j2VcYR4E3Xn9DRTt2ePTnnDD9TR3zpCXpmDHDtK++JJmt9zMjjFjyVu3jrhzzuHQBx/49bVERdH1t1+xRElNcyGqQ3JaLoOeXuA+jgq1svjeoWzan0lqVgHDerVAKcWh7AKOfNys3juiXxueu6APlgABd0Ws3ZPBhpRMzu3bhp0Hcxj8rGcV5vuH9+DM3q05mFXA2a/+DsAzI4/g4qOSsDs0ne//odR7zxhzVJ2rQHIou4DbP11JaIiFYzomsHTrAX76L8V9/s8HhtI8xlNUIDu/iNHTl/JvcgYFdoe7fUD7JnRrGcNxnRMY1qsloSGS+hcsCdDLQQJ0IWrevilTODD1bTrPmYMlPCzolTO11qS++hr29HSsTZsQ2a8fUQMDlJEs6z52u3lQNCTEPTMf0bcvbd94nYLNm7HExhLeTb62FaI6rN+byb1f/sOKHWm0jA1nb0ZewH7RYSFk5RcBcM3gjjxwZq+A/aqC1przXl/Mqp1p5bpu/h0nMvT5XwGYdsUAnpuznnV7zQJwqycOI9xmxRagcklJNu3L5JQXfgPgvL6tmXLJkeUaj4vWmvTcQnYdyqVdk0jiIm38sHoP4z/826ffncO6sXZPJt+vNhMftwztyrq9GSzefIDMPPO7jwq1cuGAdliU4szerejfPvBMuyibPCQqhKjTdGEhymYr94JESimaTbip0q8fqBpM1KBBhDRpQsiAgP92CiEqqdDu4LoPlrNg3T4ALh+YxBPnHUFBkYNuD/p/i+YKzgGOLCH9oqoopfjmxkHsy8zj/i9XM2/tPve5Ucck0btNHPd+udrd1iI2jEX3nEyI1cJbo/vTLCaMfklNOKVXC+778h8++nMnR0ycA8Cie0/m/+2dd5hdVbn/P+v0OnPOmZ6pmSSTBNJ7QoAkGErwioINkaaCiIINREGuXvTi9SeigOi1XhClhiZIhwRISCE9pM1Mpvd+et/798eZnExLmWRqWJ/nOc/MXnvttdb37L3Pevfa73pXrsN83Por2/w8t6OOh7onwQK8sKuB4gwbt6yaPKgJmG2+MCvvW580sIUAo05DKJoYBf/GiknsrOnkO58oYUlxYm6N6endPLujjgfeTgQcOOIWNDXLzovfOgeTXkZmGW6kgS752KFGo6DTyRnmY4lYDKEfG5OqCh75P2KtraRcKuPwSiSniqqqvLm/mZf2NDIrN5V0u4GtlZ2smJrB+SUZhGMKf36vImmc/+qzs1gzM7EWg0GnYefdq7nxsW3cduFUInGFrz26jXBMYUZuCj//9Exm56WOiI5Mu4kC11HXthe/eQ6z8x0ATHCYmZptJ9Ws72WwXnR27wXffnzpWbx1oCUZE/z6/9vKG989n9JmL5MzbL1cdBrdQe57vZRnd9Ql0yZlWLl2WRG/X3eY+98sxWHRc83SouT+48V1D8fiLPh5YhJ9ntPMtUuL+N93D9PujyTz3HHxtH7H/eqzs5iabePeVw7y6TkT+NXnZrOhvI2FRS5pnI8Q0sWlD9LF5cwm2tRE+YqVZP/sHpyf+9xoN0cCxNrbafqvewh8+CElmz4Y7eZIxilxJdGXbSxv456X9+MPx/jbdQuZnpMyyi37eKAoKn/bWImiqnzlnIl8+8ldSTeJ41GSZeOFb56DxXD88UJFUVFJGKMDLW4znDR7Qvznix/x3dUlTMs+tevpyGJJ1/xtK++XtfXad/jeNTR5QpQ2e/nDusNsrUosb3/rBVM4vySDs3JSMBu0ROMK8372JpfMyOb/fXY2ANf+bSsfVnWQlWLikhnZ3HheMQ6LgUhM4VevH+TP71cCkOsw8/4PVqLRCFRVxR+Jc/szu7l6aSHLJh3bpdAbimLUaaVf+TAhfdAHgTTQzzxi7e341r9L19q1WM9dTtuDD2FZsoTCR/4vmUcJhai75VbSb/o6lvnzkytXTnzheUzT+o8uSI6Nqij4N23CumzZMUd1IlVVKKEQoX37abzrrmT69IMHRqqZkjMERVGZ+7M3cQejA+6/c800rlpciNUoXxgPBe2+MG/ub+az8/OIKSr3vLyfl3Y14O3hfnKE80symJ2XyoPvlJPnNOMNxfqdp+duXnbMaCFnIgP5ffdl2aQ0fv7pGRRn2Prtu+zhjeyu7eLez8zk4XXlyegpJ+LlW5YzI3dk3jpITh5poA8CaaCfWXjffpv6734PNRLpt2/qnt1oDAZUVeXQ7DkD5nF99Stk3X77SDT1jKHj0Udp/sX/kPvQg6SsXt1vf3DPHqo+/4V+6frc3GQ8c8mZTTSucPM/dzAzN5WbV0w65RFRdyDKg++U8dcNiVFCjQBFhXsuO5u/b6qmvMWXzGvQadj9nxced0l4ybGJxRVu+Ps21h06uviPSX/Uj/kIq6Zlsr/Bw9fOncjXzi0GEm83jnhxROMqBp2G8hYfZc1eLul2a/m4EFdUXt7TQFcgyp/eq+hnYK+YmsEvLp9JTurAPuo/f3k/f+m+3iExMv7SLcsx6TX88tWDPLqpGoBMu5FbVk3m0lkT+LCqIxkFRzK2kAb6IJAG+vgiUlVFrKMDy7x5vdLD5eU0/vSnBLdtR5+bS8qaS2j/81/6HV/8SiI0VsWaNcesY+rOHWjMx5/QM5T4t26l5pprSf/mN3Fdfz1a2+iG9jvyG9Hzx11VVSqvuAKdw0HeQw8lww+qsRgHZ8xM5ktZcwmpV1yBsbgYXXo6Qq+n7tZv433jjaN5PvlJsu66E+Lxk47eIhmfuINRfrB2d69QbgCXzszhBxdPpTDt2Nd6abOXL/15M22+CHPyHRSlWXhhV2KRmcmZNl659dx+r+FbPCEW3ft2cnt6Tgprb1oqR9NPkpr2ABVtPlxWA3e/8BG769wAzM5LJRxTaPaE+OmnzuZTsyewo6aLGbkpcln3QRCJKXT4I2TajXQFo7ishhMeo6oqu2q7+PJftpBmM7LuthW94rrXtAd480Az1ywtHFS0mDOdqBJFr0nMc/JGvJh0Ju7dci+F9kIKUgrIseYwPW36CUoZeqSBPgikgT6+OLRgIYrPR9HTT2GeNQtI/IAdnH40/FbRM89gnjmDWGcn8a4uwocOUf+d7wJgW7UK+4Wrafzhj8j/y1/oeuYZvK+/zuR31+P+179o/fX92FauJP8Pvx9Uu5RQCN/6d7EsXoTW4cDz8suYZsyg5ppr0bpchA8dwrZiBfn/+4d+xzb+50/oevrp5HbG975H6mWfQmOxoDGbEbqRNS7qbv02kaoqXNdcTcoll6CxWvG+9RZ137qlVz7znDmkfuYzNP3kJwOWo8vOJuNb36Txx3djnDaNic8/R2DTJkxnnSUX/PkY4A1Fk1EsIDFhra6z9+jhA1+cw5qZOaw/1Mr7Za1Utweo7QxQ0XrsVSEfunIul8zIPuYofCgaZ+32On78wkfJtA13rGR7dSfffnIX03NSePrrS7CbxsYk5VMlHItT1xkk12GmzRfGpNeSbjOecnnPbq/j+8/s7pX27QumcON5xfIBRzJueOLgE9y75d4T5vvM5M9wzzn3jECLeiMN9EEgDfTxxeFPfpJI+WEAch96kGh1NS33/Tq5v/iVVzAWT+x3XNztpnRx73jZUza8j8ZmI1JTg6mkhLjPR+mChQBYFi4k7cYbsCxcSOjAASxz+8eijfv8BHfuQI1Gqbv5m8n0gValPILzmqvJvvNOANwv/xv3Cy/g37DhmHrtF19M3m9/c8z9Q0XowAEqP/8FdGlpxJqakunCaCTz+9+j+d5fAOC67jo6Hnmk17G6nBxyfvoTOp9+BsXtJjDA/ZT7m/tJueSSYdUgGTvsqOnk9X1N/PHdClZOzeCey2aQlWLinYPN3PSP4/vj9uTxGxZzVk4Kf91QyWObq3n12+ce0xWgL6qqcvvaPazdXtdv343nFfOjS6YRjin8c0sNX1iYj20cGaFtvnAyUkdPbEYdq8/K4vmd9eQ5zTz21cUYdRo2lLWxbHIaeU4LsbiCRgg2V7ajKDAzL5VUs57zf7WOus4ga2bmkGrW8dXlxUxMlwt1ScYHXaEuXq16NWmcXzHlCl6tfJVALJDMMzdzLncsvAMhBE6jkxzbyLtbSQN9EEgDfewR6+wk9NE+rMvPSbpZxNraaH3wITyvvYbi8Qx43LQD+4/rc1d70zfwrV8PQN7vHsL+iU/0yxPYto3qL1/dLz3rrrtI/Y9PonU4UCIRGr5/G9433zyhliPL0ffEedVVpH3lesovOFq/YeJE8h7+HbGWVmquu65X/iO+88NBtKWF6i9fTbSm5oR5nVdfTfZdd9J49910PbM2mZ55xx2kXX/d0TKbWxAGPV1PPUXnU0+T98Bvk287JGc24VicP71bwa/fLAWgwGXhte+c2ytiR017gLcPNnNWTgqPb63hxW63lWuXFjK3wMn8QicpJj3VHX5m5TmSx8UVdcAl209E0Q//nfz/2W8s44o/DBw56N+3LkerEaccteNUURT1uCtjbq5oTy5DDzAzN5W99e7k9tQsO4eavQMe+4npWRxq9lDbcfTNhVGnIRzr7UdenG6los3P5XNzuf8Lc05RiURyagwUNrLWU8vWpq1YDVZmp8/msPswAkEoFuK9+vdoD7bT6G+kwF5AV7iLbc0JO04jNHxrzre4YdYNybIUVaG0s5Spzqmj7pcvDfRBIA30sYMai+F7733qbr45mWaeN4/M226j+ktf6pXXPHs2wd2J17G2lSvJvP32AUfOe5UfjRLcvRvLCRai8a5bR903bu6Xbpg4kUmvvkLFpz9D+ODBfvtLtm6h+qqrCJeVY1/9CfIeegiAwI4dqLEYwR07af3tb/sdZ1m0iNzf/gady5Vop6JQc931BLZuTeaZsukDdM6hjXwQbWykfOWqXmn6/HzSb7oJxxWXE2tro2z5uQBk3vZ9XNddh9DpUFUVxecjWleH+18vkfmD20f9R08y+jz9YS0/eHYPkIjjvHRSGl8/bxL5LssxjwlG4vy/1w/y5SWFTBoggsVQsL/Bw2/fKuX8qRlctbiQB98u4/7uB4iBeOXWc5meYwcY1us6Fle45m9b+eBwOwBOi55Us56/XbeQp7fVkWY1sL26k3WHWvoZ1AA3nT+JH16SiDgVisYx6jT8a3cD26s7uXRmDt99ahcN7hBWg5aLzs7muZ31GHQa0qwGdFrB/AIn6TZjcgKiQafhH19dzKKJrmHTLPl4E41HqfJUkWJIoSvcRY23hrs33o0/6md+1nwOtB/goqKLSDen89eP/oqi9r/ue2LUGgnHwxi1RhZlL+L6GdczN3MuOs3YfRsmDfRBIA30sUPLfffR/pe/njCf1umkZNMHBPftQw2FsMyfP+RtUVWVpv/6L7qefArzgvkEt20HEg8MwR1HX9Fn3nEHiteLbdUqzDPOJlpfT2DnLlIuXdOvc1dVlcYf3Yn7hReSaVl3/xjXVVf1q18JBKi++hpC+/YBnHYc93BlJZ2PP4Hr2mtRvB40djuHP3E04oouO5uJzz2bfEg4QtzrRRiNwzaCLzkz2FzRztce3YYvHONryydy20VTx/TiJodbfVgMWnZUd7F8cjq/f7ecP75bAcD15xTx0u4GVBX+dM185hcm7glvKMoTW2tYPDEtuXDNEZ7YWoNRp+FTsyecVISaLRXt/ODZPVS3B06Y9/ySDH71uVmkmPS0+cLkOsx4gjFSLcf3oX9lbyMv72ngy4sLWTY5HaU7bvxAo/Wn+nZC8vGlyd9ErbeWBVkLCMaCBGNBdBodUSWKP+onFAtR76snEAtg09voCHVw37b78EYGftvTl6nOqfzsnJ/R5G9ib9teNtRvYIpzCrm2XM7PP588Wx6pxlSq3FVkWjKx6I89EDCWkAb6IJAG+uihxuM0/uQnuNc+ezRRCLJ+fBepl30aodPSfO8viNTWkPaVr2Bdvpx4VxdCq0WbMvyvoZVAAP+mTdgvuKCf60v6rbeQ9rWvnZLh6tu4kdqbvoH9ggvI+dk9aO32Y+Y9MO3oLPOiJ5/APGfOoOvrW05fJr35Bob8/FMqV3LmcGRhlZMdNVZVlTZfhC/+aROHW/0YdBqeuGEJ8wvHX4xrRVHZUN7GNX/b2m+fEAlXkTf3H41Ec+eaaaTbjKyZmcN/vbSPJ7bWJvflpJpodIfQawVXLirg+6unkmrR4w5E2Vvv5oG3S/mwqhMAu1HHhjtWEVdVOvxhPnH/ewCY9Vo+PTeX5ZPTWTMzW76hkgwrgWiAqBLFbkj0RY3+RnwRHwatgT/s/gNN/ibiSpwpzil0hjrRCA07WnbQEeoYVD2zMmZx5bQrOdx1mNcqX+N7C77H4pzFmLQmFFUhEAvw4I4HybBkcMPMGzBoz7yBIWmgDwJpoI8eoUOlVF52Wa+0gkcfxbp40Si16Nio8TiHL76EaG0tOb/4BY7PfHpE6vVv3kzNddcnt6cd2I/7xRfRGAykHCdUZKSuHsXnxTRtGsG9e6n63OcHzCcnb565+MMxDjZ5jxkK77WPmnjw7TJ+96W5KKrKmgc2oNMKrl1WhEbAmpk5nD2h90Innf4Ir3zUyLqDLaw/1EpMOdqfbP/xJ0g7jSgio000rnDL4zup6wqwoNDF9ecUcf0jH/aKKHPZnAm8ureJSPzYr95n5aWi0wh21HQBML/QiS8U6+UnbtJrePrrS3FZDeQ5j478NbqD7Klz91s6XjI+UVWVmBJDrz21iEEDhbw9EdF4lPZQOx+1fUSdtw4hBDqNDrPOjEVnwWlycqD9AOF4mHRzOi2BFn6/+/foNDpiSv/FpwD0Gj0lzhL2tSfe6Jp1ZiZYJ5BpyURFZZprGhNsE3j8wOOcm3cuhfZCvFEvM9JnkGnJJBANEIgGmJs1Nxn68OOKNNAHgTTQR48jo9Kua69Bn5uHobgY2/JzRrtZY47WBx+k7feJ8IwZ3/lO0o+976RYJRxG8XpBq6Vs6TIASjZvouMf/6Ttd7+j+NVX0GVkAipqNEpg64fYL1g14mEcJcNPmy/Mub9cRzAaB2DNzGzyXRbWHWwhFFXISjEmR3GPx4VnZaEC6w+1IIQg0u0LbdRpWDk1E71OQ6s3xG+/MJfsVNNwSho1Xt7TwM6aLq5aXEBxho2KVh8by9u4+8WEsWI1aNl85wXE4ioxRSXDnnhI8YSiPLzuqOtMht3IqqmZ6LSC764uOa2QiJLRp95Xz6P7HiWmxEg3p9Pga8CkM/HUoad65bPqrdw691aWTlhKhjmDe7fcy6XFl7JswjJiagxFVQjHw2xq2MR7de/hjXjZWL+RSY5JVLgrOD/vfH694mikMlVV6Qh14DK5ev/+qwqvVr7KD9//4SnpWZqzlI/aP8Ib8ZJiSOGKKVewt20vl0y8hM9P/XyyDo2QsdZPB2mgD4LRMNDVWAy02o/9a0vvO+uou/nmZNxyycCo8ThlK1YQb23rlT5l4wZ0aWkAtDzwAO1/+N/jljP94IFha6Pk5HAHo2yr6mBWnoNmT4j1h1p4eU8jxRlWbEYd6w614rIY+OGaaRxo9DAnz8Hi4jQONnl4dns92alGclLNLJrowmHRs62qkylZNjLtCeO4xRPi12+U8vzOeiJxBbNemzTS+zK/0Mk3zp/ETf/YTkxRuWpxAWa9lqJ0KxvK2nht39Fwm+k2A/kuC+UtPr61cjJfXFRAqvnjPRJW3xXkiS01XLOsMPn9D8TG8jZUFZZPkYtyHUFRFco6y2jwNbA4Z3GyLzTrhmaBuEA0QEughcKUQsLxMCbd0D88fv6lz3Og4/i/qdNd06nyVBGMBfvtm+aaxsGO/sEGjqARmuQkyQVZC3AYHbxVczS0plZomeaaRoW7AlVVMevMdIYTD90r8lewunA1K/JX0BHswG6wE4gGWF+3Hl/ER5Y1izRTGnn2PBp8DbjMLs5OOztZ9kBRVSRDgzTQB8FoGOitv3uYjkceIeO738H5xS8SOngQY3HxiK5eOdqoikLrb35D+5//wqTXX8NQWDjaTRrTdD7zDE13/2evNI3VytTt2wZ0YUm97FO4X/xXrzRpoA89nf4I9V1BZuSmsqu2C28oyuKJaexv9NDkDuIPx+kMROjwR+gMRHlpdwO+8MCvkU+HwjRLrwmHLquBh66cyzmT0/ndO2Xc90Ypiya6sBi0/M/ls0gx6zDrE4ME7kCUzZXtrJ6e1WsCoaqqPPJBFXvq3PzkP85KGuSy45YAuMNuIvEIGZYMIDFpsMnfRDAWZE/rHrY2baUj1EFcjaPX6Lmo6CIuLLyQh3c9zGtVr/UrT6/RE1Wi6ISOotQiyrvKmeaaRllnGZcWX8pPl/30pNwjPBEPF669EH/0qGuSw+igMKWQdHM6WZYsjFoj5+WdR4mrhJ3NO3l418O0BFqY6prKtqZtTHJM4jvzv8OyCYk3kUcM1sNdh3lgxwO0B9vZ07aHfHs+vzr/V8SUGG2BNpZMWIJFZ+l1j4RiIX688ce8XvU6AAaNgbgaRyCIqUd/CxZmL+SuxXdh1VvJMGeg1Wg51HGIuzbcxaHOQ700XlBwARqh4c3qo6F+pzin8LmSzzHdNZ05mXNO4gxKRgNpoA+C0TDQu559ltYHHyLWfHTSkenssyla+8yY7Px8GzfieelldJmZpN1440ktRa9GowT37EE/YQL6nKOLAaiKgm/9etzPP4/3zcRoQMmWzWhTU49VlARQIhGCO3agsdrQWC1UrLkUAMvixQS2bAESccrTvvoVNGYzGquVSG0t/vc3IExGrEuXyomgQ8RH9W6q2wNE4wp3v/gR3lAMrUYQV47/26oRcOFZ2Sya6GLt9jomZlj55MwcLp6RzeaKDrQawfxCJw+8Vcra7XX89+Uz8QSjPLapGncwyn2fm43VqKWmI8D/bawiw25kWradLRUdvH2wBUgsVPP4DYt7xQ+XSAYirsQ57D5MliULu8GOQOCNJtwbIGGUeqNeDncdpiPYQSAWYH/7fgpSCvBFfPxxzx8Jx8NMdyUmoPcdTU41puKL+Iir/d/gOIwOvjTtSwRjQao91dgMNva07qHKU0WaKY1MS2a/8pbkLCEYC1LnrSMUD+E0Osm0ZOIwOihIKeDiiRdT1lnGkwefZF/7Psw6M3EljlajJc+eR1lnGVqhHbA9J0PPEe2ilCKyrFncvuB2prqmDrqsjlAHWqEl1Xjifu/IedAJHRa9pdfotjvsJqpE2dmyM2m0S8Y20kAfBKPlgx73+XA/9zyh0kN4X38DxevFUFiILjsbodXguPJKzLNmo7Fa0FitBLZsId7ZiWXJErRWK+I40UPiXi+d//wnaixO+tdvROgH/yo62tCA59XXiDY30fn3x5LpwmJBm5JCvL0dVVUxFBYm4oqrKuZZMwnu20do9x6i9fXE3UcX0zAvmI/i8yfjZwOk3Xgj1mXLsC5ZPOj2DZa2YBsxJUa2tf/kK1VV8Uf92AzDE4d5OKi//Qd4Xnopua11OCjZvGkUWzT28IVj/PLVgxxo9PD9C6eSbjMwJctOeYsXTyiGRgh+/vJ+puekJFeSzLAbWX+oFa1GkGYzMCffgV6rIRJT2FzRzt83VfHWgZZkHULAvAIn6TYDWo1gZq6Df26pZna+g+WT01lSnEaqWY/TMryjz4FIjMo2f79JnZIzB1/Ex0M7HyLLmoVFZ6E12Mprla9x0+ybmJE+gzpvHVnWLIpTi/FFfPiiPircFWRZsjjUeYjOUCe13lqyLFk8uPPBfuUfGcEuTCmk2lM9QAt6k2PN4YopV7C+dj0aoWFO5hyWTliKUWskw5xBUWoRkHBnafQ38sCOB9jfvp+bZt/EpRMvHfBeaPA1kGnJRKfRJSOLeCIe1jyXmBBvN9iZmT6TXFsur1S+0muUvCfTXNN49OJH+4Xe80f9xNU4iqKwvm497rCbbGs2yyYsoy3YRo41B5POxEuHX+LODXcmj8u35zPNNQ2LzsLXZ32d/BQ52CE5NaSBPgjGwiTR0P79VH3xStRIZMD9WoeDeFfX0e3UVAwTJ2IsKcE4eTLN996LMBpxXHEFsdaW5Mg0JGKGm+fNwzJvHmlf/cqA5SuhEMJoBEXBt24daLW4n3suWY7GYqHwySeId3XR/Iv/IdbUhNblQp+dTbS5iUj54X5lGqdMxrZiBe1//kuvdNOsWTg+91nsK1eiSx8Zn8yXK17mR+//CEj8cM/JmIPD5CAQDVDjreFA+wGaA83Y9DZ8UR93L7k7Ofs83Tw2/UZVReHw6guJ1teT9rWvknr55RiLi0e7WaNGrDuqxuFWP7tru9hT38UTW2tPOKp9Mpj0Gkx6LV2BKACfnZ/HOZPT2FfvGfPxvkebrlBX0v/XoDWgERriShwhxBkz2ucOu9ncuJlwPMyGug3otXrKOssIxoIUpxZTlFqEXqPnUMchtjdv55Z5t3DltCsJxUL4o34+aPiAzY2b+dacb6HX6jnQfgCtRkuzv5lGfyPVnmpeqXwlaUAPFZmWTBZkLeBQxyECsQDLJiwjrsbxR/1E4hFSDCnk2/PRa/WkmdKYnjadNFMaQghiSoxMS+aIncOXDr/EBNsE5mf1XvNif/t+sq3ZVLmrqPZUM8U5BZfJRY4157Qfht+re4+z0s5KapZIhgJpoA+CsWCgQ2JEHQRam5VoSwvBbdsIl5cTbW4m3tmFPjsbjdWK96230KamEi4vR/H5BizLWFKC4wufR2i1NP38vyGW8HOb/O569FlZRGpqQAhqv/GNAY3rI4g5ZxP98Tfxp5lpD7YTiUdIM7noCHXQEe4k1ZBKTInRUXGAsvZDFB/owmpzkjFrIVkLlhOKhUhXrEwMWtFPmIDGYgGNhpgawx124wl72FC/IRm6KRALoEGDy+zCqrMmRjpUhRJnCd6Il1pvLRXuCmZnzCbDkkFxajELshfQGmhFIzTElBi+qI+4Eqe0s5RALEB5VzlrSxPL0qcYUvBEPP10FtgLmJc5n39VvNhv5bJb5t7CmolryLXloqLii/po8DVQ660lx5qDJ+zBaXKytnQtu1p38YWpX8CsMxNTYlxUdFFypbMKdwXFqcVDuphCpKqKlvt/w4Rf3IvGemK3o7FEiydEXZeXWblONtbuZUPNLjQC0uwGzAaFQx2HaPR4afEGuyc76ghGY8xwzeHbS64gzZxBWWs7LYF2/u+DcnbU1hPXuBH6LtRoCiZrG0ZzOzZLEIveTHlVEWa1ECWcyaKiLJwWA+5ghJ/8x9mUNnt5YmstGpEItTe/0ElJlp0tlR2UNnvZWtmB3aTnumWFXLOsiBTT4N9IxZQYWqGlLdiGVW9FRSUYC477zr/J38SHTR8SiUco6yrj3dp3cZqcVHuqUVQFX/Tob5RVb8Wqs9IabEUrtKSZ0yhxljAjfQZTnVMx68y4I24q3ZW8WP4iqwpWEVfj2PQ2okqUbGs2WqGlJdBCS6AFb8RLo7+RDEsG0XiUFGMKC7IWoNVoSTMlyvZH/XSGO5mTMeeUJwqqqoqiKmg1WuJKnFA8RHlXOaWdpWys38jbNW8n85p1ZgxaAymGFIKxIG3BoxO7jwwAADiNTrwRby8f5GNx5Ddkums6FxRcwMLshcSUGDMzZqITOt6qeYtGfyPTnNM42HmQxw88Tp49jwJ7AVqNlhJnCUtylqATOqwGK1qhRSM0yZjXEolk5JAG+iAYDQP9yYNPsqlhE8FYEIPWQHlXOQAz02di0BqIxCN4o17CsTBtwTbMOjMaoUGn0WHVJwzXT+Su4oq0VbQ9/DCpl1+OZe5cIjU1CKMRXWZmstMPl5fT9dzzdPztbwAYV5xH6N33ET2ug7BFT1wrCOtU9KEYj5+fOHZricBjPTnjId2cnnzt2nfGer49H4vOQo23hpiSCCvV1w8w15aLWWemLdiGVmjpCnf1m0RzqugwUyK+hRIs4purnczOKeKBja/zYWUnFTW56DRavOEYdqOGklyVLy6zsL75Gd6re++06+5Lvq2IyZbz6YgfYm/HVlYXrqbB10CVuwa9xkhcjVLinEw4HsZusNMV7sIddnPPOffgMDrIt+cPS0SC0yEQDbC3bS+Lshclr7tANIA77CYcVdheX8u6skrcYTf1nSGE3kNDZDs6a8Vxy1UVLWrcCqgITRShDQ2qXS6TC5fJRaW7Mnm9GbVGluQsSRp7NoONS4svZWLKREo7S6n0VBKIBugMdeIyuViYvZAsSxZ6rT4ZeaI92M4kxyTMejMH2g9Q1llGR6iDnS07mWCbwFlpZxFX4lS4K6jyVFHprjzmgh5mnRmXycXE1Ink2nLJs+WRZk744LYH2wnHw/iiPpwmJ83+ZrrCXUx2TGaaaxqZlkw6Qh1MTJ2IQNAeamdb0za2N2+n3lfPR20f0RnuZGLqRFbmr2Rl/sqkm4Mv6qO8qxyT1sTinMWUdpZyoOMAgWiAcDxMliWLjlAHny35LMtzl1PWWUZzoBl/1I+iKuxu3c3mxs297nWdRocGDZmWTApTC3EYHUywTsCgNRBTYjT6G3mn5h3SzelMdkzGG/FS2lmajDzRl2PFZdYIDamGhCvPkWPPTjub5kBzL4O4J0dcLnwRH0WpRYRjYdpD7cklx4/Eaq711lLnqyPXlosn4iEaj6LX6glEA2iEZsAR7AVZC7h+xvWE42HOmXBOr4fwqBKl1ltLni0PvUbP2zVvs7Z0LQ6TA71Gj8vkwh12MyN9BrtadqHX6pmYMpFMayaTUyejoDDFMeW0YmlLJJKxgzTQB8FoGOgP7niQ9XXrMWvNNAcSE0WnOKdQ1lmGiopBY8BusGPWmTFqjQRigeSobCQeYU/bHiAxCWe6a3pyVHhW+izy7HlUuCvoDHXSHGjGoDHQeGgn9/49jj0ICtDigE3TBe/N0NDkBK3eSGFqIWmmNJxGJ2nmNIodxVh1VmwGGxadBYfRgVFrpMHfgM1go9BeiCfiwaA1EI0nRreEEKiqyuGuw+xt24s77Ka0s5SucBctgRamuaah1+pxGp3YDXayLFmkm9OZlzUPnaZ3LO4j8Vab/c1ElSgaocGgNaBV7JR3lRNU26n2VOONeLHqrWgwYNbpsRtS2F3nprnVwbPbO0ExgnrsON+ZdiPhmMKnZk/gsc1H/S7v/uRZfH5ROk8deor2YDsaoWFL4xbcETcTrBOYnTGHDn+UhnYdvkiIXWUuYoqC1thEPJSLRt+FxlSL3lKPMNYQ9cxFaMJoreVodP3ffMR8U0DVojG0odF3gebYDyaTHZMJxoLcufhOluQsIa7GCUQDmHQmrPrESLo77E6O5g0WX8RHKB4i3ZyOoiqEYiE8EQ/vVL/Pxsb32Nu6G5POjEYIcqwT2Nb8YeJAVYNQUhMrUuq7TliPiKeg+mexKGcOq4rnEIsLKto72FYuMOkF503O58uLi3FZDTR7wrxXdYBHdr5MV7SebLuDvNQ0tFiYmetgkjOPbGs2GqFBIDDrzeTacgEIxoI8X/Y8Kiq/3PpLVAb/G2jWmQcMldaTSamTaA229npLM901nelp08myZBFVoglj0JxJIBagzltHijGFJn8TnaFO6rx1eKMntwx2T/q6Pug1elKNqXSEOiiwF1DlqTqpcmalzyKmxtjfvv+Eee16OyvyVySM8ZRCZmXMoiilCK1m8O4+1Z5q6n31tAXb6Ap1sShnESXOEgC8ES+7WnaRZ8/DHXZTkFKA0+gcsJ64Eqcl0II74qbeW8+hzkOYdCYyzBm8ePhFmv3NZFgyaPA1IBBkWjLpDHdS6a5kYupEbHobLpMLg9ZAXImTackkFE88FLpMLlRUzFozZp05aSxPd01nXta8QWuWSCQfT6SBPgjGiovLYCjtLOWPu/9IIBbAE/EQiUcwaU1Jtw6zzkyWJYtsazb1vnpqvbV8ZvJnyLPlYjXYSDenMzN9JhqhQVVV0sxpY2ZJ3UhMYUtlO63eMDqthlhcwROMsrWqg921buq7jhpJq6ZlEldU3i1tBSDFpCOuqPgjcbQawbwCB4VpVlZNy2TNzBz+8n4FP//3AeYWOPjG+ZOYW+BMLioC0OINce+/D/DCrgYAZuc7aPGEKMmys2ZmNnWdQcIxhVZvmBd21dP3Vrp6SSHXLitkcqadUDSOPxwjzWaktiPArtoutlZ28O+99ZwzHQKREO/s94GIARq+ff5iZuc5qO0M8K89lWyrcqOzHcKuzeKbF9l4YM9/ndT3V5hSiEFroKyzLJm2JGcJubZcNjVsYrJzMpdPuZzzcs+jLdhGo7+Rfxz4R+LhUNVR7S1PHqfDjIpCnHAyTY3Z0cayiUR1CKFBY6om5i9Gb63CKvKx6VPxRUI4jKlkmQqIqmHOSp/CooJ8Mu1mGnxNFDuKmOKYckrG3OkSioWo9lRj1plp9DdSYC/grx/9lfZgOwuzF5Jnz6MopQir3kppZymHOg7hjXpp8jfhMDrIs+dRmFJIraeWjnAHs9JnYdFbcJlcFKYUoqoqdb46WgOtZFoyybPnnXTbjkxWbvA30OBrIN+en3xrVtpRSrYtG6fRSUeog9LOUpr8TQhEYlnuqI8SZwmTHJNYmrO013db0VXBfdvuY17WPPxRP1dMuQKt0JJuSWdf2z48EQ9ZlqxkNApVVVFRUVWVLU1bqPZUU+IsId2cjgYNTpMTo874sV8RUCKRSAaLNNAHwXg00I9FXInTGmwly5I15H6toWicDn8innOqWU+KSU+KOTEyvaminRZPmBm5qeQ6zJj0GoQQ+MMx4qpKNKbQ5ovgj8RwB6LEFBWtBnQaDQcaPTS6Q+xv9FDd7qfZEx6wfqdFz8w8B6qq8n5ZGykmHZ7Q0VHmfJeZnFQzkzJsLCxycumsnAGXNz+ZBRjcwSg3/H0bFa0+2nwDT9wFuHJRAReenYWqqswvcJFqOTWDJRpX0Gt7T7Z6v6yVq/+6NbldmK6lui1GTrqPcNiIK62eZvMfAdALE1E1hEM7kaDayJzM+bT6O1GJU+k9Gj/XpElFJUZYGTjygRJxIHSJuRDxQBEi7kRRoyixVKy6FC6ZdB76eC6N7hBWow6X1YAnGGXltEwuOjsbrWb8+lJLJBKJRDLcfGwNdCHEzcDtQA6wD/iOqqrvH++YUfFB31rDv/c2srWyg1yHmTkFDlLNetJtRuJKYrnoApeFdJsBg05DrsPMgUYvdZ0B0m1GDDoNnmCUrmCUCQ4z2SkmXFYDdpMOs0GLzaDrteBIpz/CgSYPdZ1BBIlQb+FYHEWFNm+YFm8IXzhOMBLDatThDkZp9oQJRGKEowreUBR/pH/sWJ1GEBsgSoYQIICTDaBhM+qYlGkj32km3WYkzWpg9dlZeIIxUs16fOEos/Ic/YzYhq4glW1+lk0avol2Te4Qeq2grjOIVpP43gpc1l4j78PJUx/WcMeze4+dQRMA5TgTTzVhtMYGVMWMEs4G4mit5eidWxAiQsx3FjblbCakpHLxtBLOmpCC02pgcqaNFJOeA40eXFYDDot+wAceiUQikUgkJ8fH0kAXQnwB+AdwM7Ch++/1wFmqqtYc67jRMND/59WDvLGviSlZNpo8YWra/XQGhi58FiSMXqtRi1mvpboj0M8d4whCQJrViN2kw6TX4gtHcVoMZNiMWI06THoNNqOeNJuBtG5DzR2M4gnGqOkIYNRpcFoNzMhNpa4zgDsYpc0bQa8VpJj1hKNxnFYD6TYjVqOWVLMBg1aDLxxDIyDfZWGC4+Ozguqp0BWIsLG8neVT0ilr9lKQZsGk1/LYpmqq2vyUZNkpzrCS6zSzu7YLbyhGgctCmy/Clsp2SrLszMxNZW6Bg65AlC2VHQQjMZo8IS6bk0tJlozmIJFIJBLJcPNxNdC3AHtUVb2hR1oZsFZV1R8d67ix4uLiC8c40Ohhek4KZr2WyjY/7b4wkbhCdXsAh0XPnHwH4ZhCJKaQYtZjM+po6ArS5gvT4Y/Q7ovgCUUTftjhhA+0JxQl1axnxdRMzspJQQhQVBWjTotGgNNq6DcyLZFIJBKJRCIZWo5noB87nMU4RghhAOYD9/XZ9QawbORbNHhsRh0Li1zJ7cmZNiZnJla2PHfKsY9LNcuJWhKJRCKRSCTjmTN1qDQd0ALNfdKbgX5ruwshbhRCbBNCbGttbR2J9kkkEolEIpFIJANyphroR+jrvyMGSENV1T+pqrpAVdUFGRkZI9MyiUQikUgkEolkAM5UA70NiNN/tDyT/qPqEolEIpFIJBLJmOGMNNBVVY0A24HVfXatBj4Y+RZJJBKJRCKRSCQnxxk5SbSb+4HHhBBbgY3ATcAE4H9HtVUSiUQikUgkEslxOGMNdFVVnxJCpAE/JrFQ0UfAGlVVq0e3ZRKJRCKRSCQSybE5Yw10AFVVfw/8frTbIZFIJBKJRCKRnCxnpA+6RCKRSCQSiUQyXpEGukQikUgkEolEMoaQBrpEIpFIJBKJRDKGkAa6RCKRSCQSiUQyhpAGukQikUgkEolEMoaQBrpEIpFIJBKJRDKGEKqqjnYbxhRCCDdQ1iMpFXD3ydY3bbDbA6WlA21DXOZolXG6WsaKtr46hqIdo6XtRFrGynd+MnkGe32dSjs+TvfKUNU7FrTI+37075XhKkP2kUPbrqGqdyxoGc995BRVVVMZCFVV5afHB/jT8bZPJs8plrFtGMocrTJOS8sY0rZtqNsxitqOq2WsfOenomUkvuMz+V45k7TI+37075VhLEP2kUPYrjNJy3DcK33zjNT11PMjXVz689IJtk8mz6mUcbp1DFe7TqWMsdqusaBltLQNxEvH+H8k2zUUWkbiOz6T75Whqnc4ypT3/emVcSr5x6q2061juNo1Fq6vkTpmrGo5k/rIJNLFZYwghNimquqC0W7HUHCmaDlTdIDUMlaRWsYeZ4oOkFrGKlLL2GMs6pAj6GOHP412A4aQM0XLmaIDpJaxitQy9jhTdIDUMlaRWsYeY06HHEGXSCQSiUQikUjGEHIEXSKRSCQSiUQiGUNIA10ikUgkEolEIhlDSAN9CBBC/EgI8aEQwiOEaBVCvCSEmNEnjxBC/FQI0SCECAoh1gshzu6TxyiEeEgI0SaE8Ash/iWEyBugvouEEJuEEAEhRJcQ4u3xpkUIsUIIoR7j87nxpKU7T4kQ4oXuPF4hxGYhxMXjUMc8IcSb3ddVuxDiT0II21DoGGItNwoh1nW3UxVCFA1Ql1MI8ZgQwt39eUwI4RinWu4SQmzsPm9D7pc4UlqEEEVCiL8KISq6y6gQQvxCCGEeh1o03fdQjRAiJIRoFEL8QwiRO5509MlrEkLs7s43ZBPmRvheqRL9+5T/GY9auvONh/7+RPfKeOrvT+YaG7b+vifSQB8aVgC/B5YBq4AY8JYQwtUjzw+A7wO3AAuBFuBNIYS9R57fAlcAVwLnAinAy0II7ZEMQohPA08CjwFzgaXA38ahlg+AnD6fXwA+4NVxpgXgZcAEXEDivGwAXhRCTBovOoQQE4C3gApgMXAxcDbwyBBoGGotFuAN4KfHqetxYB5wCQkt80jcN0PFCkZOixF4jsQ5HA5WMDJapgFa4Bskrq1bgGuAB4ZIB4zseXkH+DwwlcS9VQw8PxQiGFkdR7gPqDvdhg/ACkZWyz307lt+ftoKjrKCEdIyjvr7E2kZT/39yVxjw9nfH+VYAdLl59Q/gA2IA//RvS2ARuCuHnnMgBf4evd2KhABruqRJx9QgIu6t7VADXDDeNdyjLoOcZyg/WNVC4kVyFRgZY88uu66PjuOdNxIYiU1bY88M7u1TR4r56TP8Qu621fUJ316d/o5PdKWd6dNHU9a+uT5LKAO1z0yklp65L0ZaD9DtHyqO69pvOkALgP29bh3FozHcwJUAbcNV9tHSgvjpL8fzDXWJ++Y6+9P8ryMWH8vR9CHBzuJtxOd3dsTgWwST2UAqKoaBN4j8bQHMB/Q98lTCxzokycfCAshdgghmoQQbwgh5o5DLb0QQqwAShjeUEfDpaW9e/tqIYSte0T6RhI3/sZxpMMIRFVVjfeoK9j9d/nQSkhyKlpOhqUkRmc+6JG2EfAPspzBMFxaRoOR1JLSo57hYES0dI/UXQVsUVU1dMqtPTbDpkMkXN3+QKL9wRNkHwqG+5zcJhIuertEwj3McLoNPg7DpWW89PeDZgz39yfDiPX30kAfHh4AdgGburezu/8298nX3GNfNoknsLbj5Cnu/vsz4F7gUhKvI9/tdk8YDoZLS19uBHarqrrtdBp7AoZFi5p4hF4NzAA8QJjE67FLVFVtHLLWH2W4zsk7QLoQ4odCCIMQwgkc8d3MGZqm9+NUtJwM2UBr97kBkuepZZDlDIbh0jIajIgWIUQBcBuJV9PDxbBqEUL8UgjhJ9FxFwCfPLVmnpBh0dFtYPwT+LWqqrtOr4knzXCekwdJuPGtBH4HfJfxeX2Nl/7+VBir/f0JGcn+XhroQ4wQ4n4So41X9BmJhMRrkV7ZB0jrV2SPPEfO13+rqrpWVdXtJC70LuDqU270sSoeXi0963EBlzOMT9PDqUUIIUh0AO0kfLsXAWuBZ8UQTRhLVjqMOlRV3QdcC3wHCABNQCWJH7C+dZ02w6ClLwPlP5VyTsgIaBkxRkqLECILeB14E/jNqZRxEnWMhJZfkfBDvZDEffKP7t+EIWOYddwJRIH7T72FJ89wnxNVVe9XVXWdqqp7VFX9CwkXqq8KIdJOudHHYJi1jPf+/lj1jMf+vmfZI9bfSwN9CBFC/IbEk/sqVVUreuxq6v7b9yktk6NPc00kfM7Sj5PnyNPZ/iM7VVWNAWUkRm6GjBHQ0pNrSfhC//N02nwsRkDLKuA/gCtVVd2oquoOVVVvJuFOcf3QqBiZc6Kq6uOqqmYDuUAaiZGBDBKG+pBxmlpOhiYgs6eh1P1/xiDLOSEjoGXEGCktQohsYB3wEXB1zzcdQ8VIaVFVtU1V1VJVVd8EvghcxBC6hI2AjgtIjDZHhRAxoLw7fbMQYkh/k0fpXtnS/XfyaZbTixHQMl76+8Eylvv7k2FE+nuQBvqQIYR4APgSiYviYJ/dlSQujtU98ptIPH0d8ZHdTmIUo2eePBITdnrmCZOIGHAkjwaYBFSPMy09+RrwtKqq7qHS0KPekdBi6f6r9ClfYYjusZE+J6qqNquq6gO+AIRIjHIOCUOg5WTYRGKS0NIeaUsB6yDLOS4jpGVEGCktQogcYD0JP84ru42OIWUUz8uR+914muUAI6bjemA2MKf7s6Y7/SrgjlNp90CM4jmZ0/13yNwPRkjLeOnvB8tY7u9PhmHv75MM5YzTj+sHeJiEL9IqEk9nRz62Hnnu6M5zOQnfpSeBBsDeI88fgHrgEyRema4j4UPVM6rGb0n4oV1E4sZ9CHADE8ablu58RyJrnDMU7R8NLSRGpduAZ0l0dCUkXntHgXnjRUd3nm+RCEdYAnyThKvLrWPwnGST6Hi/1H39rOnedvXI8yqwF1hCwjjfC7w0TrUUdKfd1p1nTvfHNp60ABOAUhIGen6furTjTMtSEvfIbKCwu76NJAyB047iMpLXV596ixjiKC4jfE6+2502kUQIzHrgxfGmpTvPbxkf/f1JXWOMj/7+RNfYsPb3vTQN9Zf0cfx0n8SBPj/tkUeQcBdoJDEi+S4wo085pu4bsJ2EYfQSkN8njx74fySeBD0kOrohuyhGUkt3vkeB/WfAeVlAwp+2vfu8bAEuHYc6/t69PwzsJuF+MBbPyU+PUc51PfK4gH90nw9P9/+OcarlkWPkWTGetADXHaeuonGmZQ6Jh9wj90sliYfgvPGkY4B6ixh6A32kzsk8YDMJP+0gcLD7GMt409KdZ7z09yd1jTE++vuTOS/D1t/3/IjuyiQSiUQikUgkEskYQPqgSyQSiUQikUgkYwhpoEskEolEIpFIJGMIaaBLJBKJRCKRSCRjCGmgSyQSiUQikUgkYwhpoEskEolEIpFIJGMIaaBLJBKJRCKRSCRjCGmgSyQSiUQikUgkYwhpoEskEolEIpFIJGMIaaBLJBKJRCKRSCRjiP8P5PdSTj0Cdu4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Plot some stocks\n", "stock_names = [\"AAPL\", \"GOOGL\", \"MSFT\", \"AMZN\"]\n", "\n", "fig, ax = plt.subplots(1, 1, figsize=(12, 6))\n", "ax.xaxis.set_major_locator(mdates.YearLocator())\n", "ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y'))\n", "ax.xaxis.set_minor_locator(mdates.MonthLocator())\n", "for stock_name in stock_names:\n", " df_stock = df[df['Name'] == stock_name]\n", " df_stock_dates = [datetime.strptime(d,'%Y-%m-%d').date() for d in df_stock['Date']]\n", " ax.plot(df_stock_dates, df_stock['Close'], label=stock_name)\n", "ax.set_ylabel('Close Price ($)'); ax.legend();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's do some minimal processing to make the data useable:\n", "\n", "1. Split the data into segments of a fixed number of days.\n", "2. Split each secment into a BASE ($\\vec{X}$) and a TARGET ($\\vec{Y}$).\n", "3. Split all segments into a training and test set." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:34.642811Z", "iopub.status.busy": "2020-11-24T17:30:34.642033Z", "iopub.status.idle": "2020-11-24T17:30:34.685074Z", "shell.execute_reply": "2020-11-24T17:30:34.685658Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "X_train.shape=torch.Size([210, 1, 30])\n", "Y_train.shape=torch.Size([210, 1, 10])\n", "\n", "\n", "X_test.shape=torch.Size([91, 1, 30])\n", "Y_test.shape=torch.Size([91, 1, 10])\n" ] } ], "source": [ "SEG_LEN = 40\n", "SEG_BASE = 30\n", "SEG_TARGET = SEG_LEN - SEG_BASE\n", "\n", "# Filter out only selected stocks\n", "df = df[df['Name'].isin(stock_names)]\n", "# Split into segments of SEG_LEN days\n", "X = torch.tensor(df['Close'].values, dtype=dtype)\n", "X = X[0:SEG_LEN*(X.shape[0]//SEG_LEN)]\n", "X = torch.reshape(X, (-1, 1, SEG_LEN)) # adding channel dimension\n", "\n", "# Train-test split\n", "test_ratio = 0.3\n", "N = X.shape[0]\n", "N_train = int(N * (1-test_ratio))\n", "idxs = torch.randperm(X.shape[0],)\n", "X_train, X_test = X[idxs[:N_train]], X[idxs[N_train:]]\n", "\n", "# Split out target segment at the end\n", "X_train, Y_train = X_train[..., :SEG_BASE], X_train[..., SEG_BASE:]\n", "X_test, Y_test = X_test[..., :SEG_BASE], X_test[..., SEG_BASE:]\n", "\n", "print(f\"{X_train.shape=}\\n{Y_train.shape=}\\n\\n\")\n", "print(f\"{X_test.shape=}\\n{Y_test.shape=}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can plot some random BASE and TARGET pairs." ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:34.690186Z", "iopub.status.busy": "2020-11-24T17:30:34.689648Z", "iopub.status.idle": "2020-11-24T17:30:34.876107Z", "shell.execute_reply": "2020-11-24T17:30:34.876607Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs0AAAFpCAYAAABnKhXkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAB4XUlEQVR4nO3deZwcx13//1d1z7X3anWsTuuWLF+SLfm2YynECbkDIV+I+eYC4oQEQ0IgX0ICtn/wg/wSCDn5QgK5gMQmBEiwczmx13Z8y7EsX5Kt07pvrfaco7t+f3RPT8/srnZ3tLuzK72fevSju6trp2tqWjOfrq6uNtZaRERERERkaE6tCyAiIiIiMtkpaBYRERERGYaCZhERERGRYShoFhEREREZhoJmEREREZFhKGgWERERERlGotYFGIkZM2bYRYsWTfh+e3p6aGhomPD9TnWqt+qo3qqjequO6q06qrfqqe6qo3qrTrX19uSTTx611s4cbNuUCJoXLVrExo0bJ3y/HR0drF+/fsL3O9Wp3qqjequO6q06qrfqqN6qp7qrjuqtOtXWmzFm91Db1D1DRERERGQYU6KlWURERKam2267DYBdu3bR0dFRliYylShoFhERkXFz++23D0hT0CxTkbpniIiIiIgMQ0GziIiIiMgwFDSLiIiIiAxj2D7NxphdwMJBNv3AWvt6Y4wBbgVuBqYBjwEftNY+F3uNNPA3wNuBOuBnwAestXvP+B2IiIjIpHXrrbcCwY2AtXjmgshYGcmNgJcDbmx9DvAk8O/h+keBjwDvBrYCfw7cY4xZaa3tCvN8FngzQdB8DPgMcJcxZq211jvD9yAiIiKTVPGmP403LFPdsN0zrLVHrLUHixPwOuAU8J2wlflDwCettd+11j4LvAtoAm4CMMa0AL8N/LG19h5r7S+AdwCXAK8ajzclIiIiIjKWRtWnOQySfxv4V2ttL7AYmA38pJjHWtsHPABcEyatBZIVefYAL8TyiIiIiIhMWsZaO/LMxrwa+DFwqbV2kzHmGuAhYKG19uVYvq8C86y1rzHG3AR8E0ja2M6MMfcCL1lr3zfEvm4m6CdNe3v72jvuuGP07+4MdXd309jYOOH7nepUb9VRvVVH9VYd1Vt1VG/VU91VR/VWnWrrbcOGDU9aa9cNtm20Dzd5L/CEtXZTRXpl5G0GSat02jzW2i8DXwZYt26dneh+UIdO9fMfP3mID75hYvd7NlC/teqo3qqjequO6q06qrfqqe6qo3qrznjU24iDZmPMLIKb+T4YSz4YzmcDe2Lps4BDsTwuMAM4UpHngVGWd8J85YEd/NPGfjb1buTjr1vFohkNtS6SiExB1lqOdud4+XgvLx/v4eVjfdFyT9Zj8YwGlswMpxmNLJnZQFMmOeb733m0h51Hu9lxpIcdR3vYtb+Pbe4O3rZ2AS31Y7c/EZGz1Whamt8NZIF4P4mdBEHxjcATAMaYDHA98MdhnieBfJjnW2Ge+cAq4OHqiz6+/ug1Kzl5eC8/3HaUV//dA7znukX83oZlY/pjJiJwrDvLgc5+sgWfbN4jW/DpH+E8W/A5dKifR/peYNH0BhZOr2fR9AZmN2dwHDNh7yFb8Nh7IgiG9xzvZfex3mj55eO99ObKBwma05JhQVs9s5rTPLe/kx8+ewA/dt1tZlOaJTMaWDKzkaWxgHr+tDoS7uC3ovTmCmFg3MOOI8V5NzuO9tDVX4jypVyHhdPryXrwl3e/wN/8ZCtvvGQu77h6IZfMbx2P6hEROSuMKGgObwD8HeCO2DByWGutMeazwMeNMVuAF4FPAN2EAbK1ttMY88/Ap40xhykNObcZ+OkYvpcxlUm6vGFJio/+2lV86sdb+cf7d/DdJ/fx0des5NfWzp/QH2SRs9WdT7zMn33vOXIFf0T5E44hk3RJJ5xo3tPrs+nnu8h5pddIJRzOa6tn0fR6Fk5viM0bmNuaGTLwjMsVfI735KLpWE+W4z05TvTkOFZM686x90QvB071E789JJMM9n9eWwPXLJ3BeW11LJzewIK2euZPqyOTdAfs6+XjPWw/EgS8xWD3R88e4ERvPsqXdA0LpzewZEYDi2c20JMtREHygc7+stec25JhycxG3rJmXqk1e0Yj86bV4TqGjo4OZq64lH999GW+t2kf33lyL5fMb+F/X7mQN66eS12qvIwi1ero6ABg06ZNUZq6G8hUNNKW5vXAcuB/D7LtUwQPLPkSpYebvDoeXAMfBgrAnZQebvLOqTBG86zmDH/zttW846qF3P4/z/HR727mm4/u4tY3Xsjli9pqXTyRKakv5/Hn33uW7zy5l+uWzeCdVy+MguB00iWTdEgnyucp1xk02O3o6OD6V9zAgc4+dh/rZdexHnYf62V3OP/5tqP050sBdcIxLGirj1ql0wmHYxXB8ImeHF3ZwoB9ARgD0+pTtDWkaKtPcdWS6Zw3vT4Mkus5b3o9MxvTBG0NI5NKOCyb1cSyWU0Dtp3oybHjaPeAgPq+rYepS7osmdnI1Uumh4FxI4tnNLB4RsOIgt4L57bw1796MR973fn81y/28a+P7uaj393MX979PL+2dgH/+6rzWDJTNyDJmdmwYcOAtNEMQiAyWYwoaLbW3kdw495g2yxwWzgN9ff9wC3hNCWtXtDKd3/3Gr7/9H4++cMtvO0fHuGNq+fyJ689n3mtdbUunsiUsetoD+//1yfZcrCL33/lMv7gVStwz/DKjesY5k+rZ/60eq5dNqNsm7WWw11Zdh3tKQuqdx3rYeOuE+Q8n+kNYRDckGLh9PooIG5rTDG9IcW0+hTTG1O0NaRpqUuecXlHY1pDirUNbaxdWH6S7vsWYxhVcD6U5kySd12ziHdevZDHdh7nXx/dzTcf2cVXH9rJtcum846rFvKqVe0jaqEXETlbjXb0jHOaMYY3r5nHjRe084/37+Af7t/OPc8f5OZXLOX9NyyhPqXqFDmdHz17kD/+ztO4ruFr77mcDStnjfs+jTG0N2dob85w5ZLpA7Zba8ck8Jxo49FFzBjDVUumc9WS6Rzu6uffn9jDtx57mff/6y9ob07z9ivO4+1XnEd7c2bM9y0iMtmp2aAK9akEH75xBff+0XpuvGA2n//ZS/zS397P9zbt0yUnkUHkPZ+/+sELvP9fn2TJzAbuuuW6CQmYR2IqBswTYVZTht975XIe/D+v5CvvXMf5s5v53M9e4ppP3sv7/+VJnt5zstZFFBGZUGoaPQPzWuv4wtsv5Z1XB/2d/+COTXzzkd3c+sYLdBe6SOjQqX5u+dZTPL7rOO+4aiGfeMMq0gndZDZVuI7hxgvaufGCdnYf6+Fbj73Mv2/cw31bD/OVd67jFStm1rqIMsndcMMNAJw8eZLW1tbaFkbkDChoHgOXL2rj+x+8jv94ci+f+vFW3vTFh/i1tfP56GtWMkuXMeUc9vD2o/z+t5+iJ+vxud9Yw5vXzKt1keQMLJzewMdet4r33bCU3/ynx3jvNzfyT+9ax/XLFTjL0IqjZ+ghHTLVKWgeI45j+F+XL+C1F8/mi/dt42s/38UPnjnA796wlPe+YsmAIaZEio50ZfnRswf40eYsD3Q9z4ymFDMa0sG8Mc30xjQzGlNTqnXW9y3/9/7t/O1PtrJ4RgPffu9VLG8fODKETE1tDSn+7Xeu5KavPMrvfGMj//yuy7lu+Yzh/1BEZApT0DzGmjJJPvbaVdx0xXn89Q+28Lf3vMi3H3+Zj/7y+bxp9VyN7ywAHO/J8aNnD3LX5v08uuMYvoWWtGHT0ZfpyQ0+EmNTJsHMxjTTG4NgekbF8vxpdSyd2Vjz8XU7e/P84b9v4mdbDvOGS+bwybdeQmNaXzVnm7aGFN9671Xc9JVH+e1vPKHAWUTOevolGycLpzfwD+9Yy2M7jvGXd7/Ah+7cxNce3sWfvX4V6zS+8zmpsy/Pj587yF2bD/DQtqN4vmXxjAY+uGEZb7hkLvtf2MiGDRvoy3kc7c6GUy6Yd2U51pPjSLj84qEuHtlxjJOxB19AMIbw/Gl1LJvZyPL2JpbNaoym5gl4muWz+zr53X97koOd/dz+pgt559ULdaPdWUyB88gVPJ8tB7vIFnwumtc8pa4ciUhAQfM4u3LJdL73wWv5r6f28akfb+HX/uERXn/JHP7kl89nQVt9rYsn46yrP89PXzjEXU8f4IGXjpD3LPOn1fHe65fwhkvmcOHc5iioPLAlmNelXBa01Y/o+Cg+te5od5aXj/fy0qFuXjrcxbbD3Ty07VjZU/JmN2dY3t7I0pmNLG9vZPmsIKhua0id8fu01vLtx/dw2/88x4yGFHe+72ouO2/aGb+uTH6VgfNX3335gLGyz0U92QKb9pzkiV3H2bjrBE+9fCK6ipRKOKye38K6RW1cvmgaa89ro6V+/E9qRSZUIQdeDqwHvgfWD+Z+oSKtEC6HacXlVCO0X1Drd1FGQfMEcBzDW9fO57UXz+bLD+zgH+/fwT3PHeK3rlvMBzcspWkCWgAnA2stp/oLUcvp0e7g0cRHu7Ic6c5xrDtoTW2pS3LRvBYuDqf25tE9XW2kfN+y81gPz+ztZPPeTjbvPcnx3hwzG9PMas4wqykdTM1pZjUV1zM01yVOW57eXIGfvXCYuzbv576tR8gVfOa0ZHjX1Yt4w+q5rJ7fMmbvJ5VwmN2SYXZLhovmtcDFpW0Fz2fPiT62HQ4D6UPdbDvSzb9v3ENvrAvI9IYUS2Y20JxJBk/lSwaPqM6ET+LLJF3qik/pS7rhtjBPmP71h3fxn7/Yx/XLZ/C537h0TAJxmTqKfZx/858eCwLnd13ONedY4HzoVD8bd53giV3HeXL3CZ4/cAovfADNyvYmfuWyeVy+qI10wgny7T7BVx7Ywf/tCIYpXdnexLpF07h8URvrFk1jXmudrtLI1NN7HJ77T9j8Hdjz6Jm91uJXwLv+Z2zKNUYUNE+g+lSCD71qBb9x+Xl8+sdb+Yf7t/OdjXv4w1ev4NfXLZgST9uy1pIt+PRkC/RkPbqzBXpyBbqzBXqzHj3ZAr/YmefRvi1RF4NjYReDY925spbPImOgrT4V9dHde6KXjq2H8cMhr2c0prlkfssZBdLWWvae6OPpvSejIPnZfZ3Ro5IzSYcL57Zw/uwmjnRl2bz3JIdPZenLD+xfnEo4pYC6KcOs5jQzG9O0NqR4dMcx7n3hMH15j5lNaW664jzecMkcLjtv2oT3Z0+4TvRI5RsvaI/Sfd+yvzMIpovTjiM9HOjsp7/gkc379Oe9YCr4eP7wY48bAx961XJueeXyCX1ankwe0xvT4c2Bj/FbZ3ng7PuWlw53s3F30Iq8cfdx9hzvA4LvkjULWvndG5aybtE0Lj1vGi115Q0jv3zRHCB4nPymPSfZuOs4T+w+wfc27effHnsZCK4MxYPo82c36/+WTE65Xnjxh0GgvO2eoOV4xkq4/iOQbgYnAY4LxgXHCdaNG0uLL8fy1k++rqwKmmtgdkuGv/1fq3n3NYv4i7uf5+P/9SzfeHgXn3j9BWMy5qm1lr68R1/Ooy8MfvpyfpCW9+jLFcK5H9vuRdv7c6VguCcMhHuyhTDNG1EQldy2o+xmtZWzm8L1gTextTWkBvwY9OYKvHDgFJv3dvLMviDArQykL57XzMXzWwcE0tZaDp3KlgLkfZ08s/ckJ8L+vynXYdWcJt586VwumdfKJQtaWDazccBJi7WW7myBw11ZDp/KcrirnyNdWY50ZYO0rn62H+nmkR3H6OwLXnt6Q4q3rp3HGy6Zy+WL2iblj5wTe+T0+hE8YCTvFYPoYJ4tlJaL87mtdVwwt3kCSi+T2fTGNN9679kbOB/o7ONzv+jn9zt+wqn+4IR7RmOKdQvbeNfVi1i3qI0L5zaTHGEDSF3K5eql07l6afCkSs+3bDl4KgzET/DEzuPctfkAAI3pBJctnMYvnT+L1140W8OZSm35Huy8PwiUX/g+5LqhaQ5c+X645H/B7EuC1pSzjILmGrp4fgt33nwVP37uIH/1gy2886uPs37lTD7+ulWDDs/l+5YTvTkOhQFcEMz1R0Hdoa5+Dp8KgrrBWnRPxzFBS3gm6VKXcmhIJWhIJ2iuSzKnJUNDOkFjOkFD2qUhnYi2NxbXozSXpzc+xutetf6MLi3WpxKsXdjG2oWlM81iIF0MhJ/d18n9Lx4pC6SXzGxg59EejnRlgeDBDCvam3jNhbO5eH4Ll8xrZeXsJlKJ4X/UjDE0ZZI0ZZIsndl42rz9eY/jPTlmNaWnxBWD0Ui6DknXoUm/0TIC0xvT/Nt7g+Hofivs43zN0qkfOOc9n9/71lM8f8zjVy5bwNqFQSvwwun1Y9aNwnUMF85t4cK5LbzrmkVYa9l3si/q9vHojmPc+v3nuO1/nuOKRW284ZI5/PJFc5jZlB6T/Y+XwepHT8+dgqyFA5tg87/Ds9+F7kNBS/KFb4GL/xcsui5oJT6LKWiuMWMMv3zRHDacP4tvPrybz9/7Er/8uQd5y5p5ZJJO2KIZBMdHurIUBmnlbalLRn1vr1zcxszmNNPqU9Sn3Kg/al3SpS4VTrH14vaka8bsi39bcuxeK264QPqZfafYcbSb65fP4JJ5LVw8v5UL5zZPyBjZmaTL3Na6cd+PyFQwozEd3Rz4W19/gq+9+4qoNXWq+pufbOXJ3Sd4/+o0f/LWSyZkn8aUrgq95dLgwUAvHeri7mcOcNfmA/zZ957j1u8/x5WLp/P6S+bwyxfNZkbj5A6gZZxku6FzL3TuCaaTe4L1XA/UT4P6GVA/HRrCef2MoPtDw4zghrvT/WYf3wnPfCcIlo+9BE4SVrwGLn5bME+eO799CponiXTC5b2vWMKvXjaPz/3sJe54fA8NaZf25gwzm9IsmzmD9ubijWmZcDnYdi4/OGWwQFpEaq8YOL/9y4/ynq8/PqUD5/u2HOYf79/BTVeex1XTjtW0LMvbm/hQexMfetUKXjzUxV2bD3DX5v184r+f5c+/9yxXL53O6y+ey2subGe6Auizg/Wh61AYFL8czItBcXG970T53zgJaJ4LqSbY/wvoOQp+fvDXd1OloLoYSNdPD1qRdz4Aex8P8i28Fq75PbjgzVB3bo6OpKB5kpnemOb/efNF3P6mC3XntIhMaTMa03z75iBw/q2vP8HX3nM5Vy2ZWoHzgc4+/vDfN3H+7Cb+/A0X8OhDD9a6SJEV7U384Y1NfPhVy9l6qIu7Nx/g7s0H+NP/eoY/+96zXLN0Oq+/eA6vuXA20zSizdRQyMGhZ2DvxmDa/xSvOL4L7q8IeFNN0LoAWubD/CvC5eI0H5pml3eVsBayXdB7rDT1HA2Xw3lPmL7/qWDe3wmzLoBX3QYX/Vqwj3OcguZJSgGziJwN4l013vO1qRU4Fzyf3//2U+QKPl/6zcsm7VU9Ywznz27m/NnN/OGNK3jhQBd3P7Ofuzcf4E/+8xk+/t/Pcu2yGbxp9VzetHruiO7pkAlgLZx8GfZtLAXJB54GL7gnh8bZMH8de+sv5LyLri0FyS0LINMyuhvtjIFMczC1LR7Z3/jeWd9HebQUNIuIyLia2RR21ZhigfNn7nmRJ3ad4HO/sWbYm4EnC2MMF8xt5oK5zfzRq1fy3P5T/CDsA/1H33maz/3sRT70Syt4y6XzJmx0H5vPwsnddDyzl/UbNkzIPielbFfQirv3Cdj7ZDDvORxsS2Rg7qVwxXth/uUwfx00zwNj2NHRwXlXrp/48ipgHkBBs4iIjLuZTWm+HQucv/D2S3lVbPzwyaZj62H+vmM7b79iAW9eM6/WxamKMYaL5gVj3P/xa1bSsfUIf3vPVj7ynaf5+45tfPjGFbzuojnjP4b8/l/AV1/DtYkm2H910J1g/jqYtzZo+Zyscr3w7H/A9nuD9fj4woONK1xMj/IlgnGJT+wKguQjLwT9kwGmL4NlvxTUwfzLof1CcM+NB51NZQqaRURkQgQtzlfynq89we98cyPvuXYRf/La80knJleL1sHOfv7w35/m/NlN3PrGC2tdnDFhjGHD+bNYv3ImP37uIH/7kxf5vW89xao52/nIjSv4pVWzxq9bYNtSeNMXOPr495hz8mV46SfFUgV9ZhdcHgTSC64Igslad088vhM2/jP84l+g/2TQHSJZV3rcc9kjnwvlj38urtvYsK+Z1uAkYdUbgwB53mWT8sEdMjwFzSIiMmFmNWX47u9ewyd/uIWvPbSLx3ce54s3XcbiGQ21LhpQ6sfcn/f44k2Ttx9ztYrDnN54wWzu2ryfv7vnRX7nmxtZvaCVP3r1Cq5bNmPsg+fGmXDZO9l66jzmrF8PfSeDfrx7nghGZnj2v+DJrwd566aF3ROuCILpeWshPfC5BWPO92HHvfD4P8GLPwLjBEHuFTfDwmtGH8hbWwqiE+nanwjImFDQLCIiEyqTdLntTRdyzdLpfPS7m3nD5x/kL3/lIn7l0vm1Lhqf/elLPL7rOH/366tZNmtq9GOuhusY3rxmHq+7eA7/+Yu9fP5n23jHPz/OFYvb+KNXr+SKxePYElrXCsteFUwQBKxHt8Kex4Mgeu/GQVqjr4AFV8J5V8K0xWMXhPZ3wqZvweNfgePboWEmvOKPYd17giHbqmUMuIlgkrOGPk0REamJV4dP6vyDb2/iw3c+zYMvHeUv3nwRDena/DQ98OIRvtSxjV9ft2BSBPATIek6/Prl5/GWS+dx5xN7+MK92/hf//gIr1gxk4/cuILVC1rHvxCOA7NWBdPadwVpA1qjvwtPfi3Y1jArCKLPuwoWXAVzLglac0fj0PPwxFfg6Tsh3xO0bK//GFzwptG/lpwzFDSLiEjNzGmp41vvvZIv3LuNL9z7EptePsnn334pF81rmdByHDrVz4fv3MSKWU3c9qazox/zaKQTLu+8ehFvW7uAf3l0F/+3Yztv/tJD3HhBO3944wpWzZngG/YGtEZ7cGQLvPxo0CK951HYclewzU0H/YQXXBEE0QuuCB7QUckrwNa7g1blXQ8Gf3fx2+CK3wlGrhAZhoJmERGpqYTr8OEbV3DVkul86M6n+NW/f5iPve583n3NogkZs77Yj7k35/Gl37yUutTZ1Y95NOpSLje/Yik3XbmQr/18J19+cAev+/yDvOGSuXzoVctrN/Se4wYjTLRfCJf/dpDWdTAMoB8Lpkf+Hh76XLBt+rJSAD3nEtj2M9j4VTi1D1rOg1fdDpe+Axom/9CHMnkoaBYRkUnh6qXT+eEfvII//s7T3P4/z/PQtqN8+tdWj/vT7D7/s5d4bOdx/vZtq1k2awJuOpsCGtMJbvml5bzz6kV8+cHtfO2hXfTlPP7pXetG/Vrr168H4OTJk7S2tgLQ0dFx5oVsmh10p7jgTcF6vg/2bwpaoV9+DLb+ADb9ayn/kvXwuk/Dil/WGMRSFQXNIiIyabQ1pPind63j6w/v4q9/sIXXfu5BPvcba7hynB6G8vOXjvKF+7bxa2vn89a150Y/5tFoqU/yx685n/dcu5i+nFfVa9x///1jXKohJOtg4dXBBMEIFse2BYH0nEtg5sqJKYectfQsTRERmVSMMbzn2sX85weuoS7l8vavPMpnf/oinm/HdD+HT/XzoTufYtnMRv6fN597/ZhHY0ZjmgVt9bUuxugYAzOWwyVvU8AsY0JBs4iITEoXzWvhf265jresmcdnf/oSN33lUQ509o3Ja3u+5Q/u2ER3tsCXfvMy6lO68Coip6egWUREJq3GdILP/Poa/vZtq3lmXyev/dyD/PdT+zjQ2Ye11bc8f/5nL/HIjmP8xZsvYkW7+jGLyPB0ai0iIpPeW9fO59LzWrnl20/xoTs3AUFAvXRmA0tnNrJ0ViPLwmlhWz0Jd+g2oYe3HeXz977Er142j7etWzBB7+Dcdd999wGwadMm1qxZU9vCiJwBBc0iIjIlLJnZyH994Fo27j7O9sPdbD/Sw7bD3Ty8/Rj/+dS+KF/SNSya3sCyWY0snVkKppfMbKA7W+D379jE0pmN/OVbLqrhuzl3FEfPqFwWmWoUNIuIyJSRSjhcs3QG1ywtf3hFV3+e7Ud62H64m21Hutl2uJutB7v4yfOHym4gbEi5eNbyb79zpfoxi8io6BtDRESmvKZMkjULWllT8djnXMFn97GgRXrb4W52HO3hNRe2s3K2+jGLyOgoaBYRkbNWKuGwvL2J5brZT0TO0IhGzzDGzDHGfMMYc8QY02+Med4Yc0NsuzHG3GaM2W+M6TPGdBhjLqx4jbQx5gvGmKPGmB5jzPeNMRpJXkREREQmvWGDZmNMK/AQYIDXA6uAW4DDsWwfBT4Spl8ebrvHGBM/tf8s8Fbg7cD1QDNwlzFGz7IUERERkUltJN0zPgocsNa+M5a2s7hgjDHAh4BPWmu/G6a9iyBwvgn4R2NMC/DbwHustfeEed4B7AZeBfz4zN+KiIiIiMj4MMMNDm+MeR74ETAP2ADsB/4J+JK11hpjlgDbgSustU/E/u5u4Ki19l3GmFcCPwNmWWuPxPI8B/yHtfbWQfZ7M3AzQHt7+9o77rjjzN5pFbq7u2lsbJzw/U51qrfqqN6qo3qrjuqtOqq30fv6178OQC6XI5VKAfDud7+7dgWaYnTMVafaetuwYcOT1tp1g20bSdDcHy7+HfDvwBrgC8CfWGu/aIy5hqD7xkJr7cuxv/sqMM9a+xpjzE3AN4Gkje3QGHMv8JK19n2nK8O6devsxo0bh3mbY6+jo0NjSlZB9VYd1Vt1VG/VUb1VR/U2esEF6XJn8jTHc42OuepUW2/GmCGD5pF0z3CAjdbaj4XrTxljlgMfBL4Yy1f5P8AMkjagbCPIIyIiIiJSUyMZPeMA8HxF2gvAeeHywXA+uyLPLOBQLI8LzDhNHhERERGRSWkkQfNDwMqKtBUEN/FBcFPgQeDG4kZjTIZghIyHw6QngXxFnvkEI3EU84iIiIiITEoj6Z7xd8DDxpiPA3cClwK/D/wpQHgz4GeBjxtjtgAvAp8AuoFvhXk6jTH/DHzaGHMYOAZ8BtgM/HRM35GIiIhMGrfeGtzrv2vXLhYtWlTbwoicgWGDZmvtE8aYtwB/BfwZ8HI4//tYtk8BdcCXgGnAY8CrrbVdsTwfBgoEgXcdwWga77TWemf+NkRERGQyuu222wDd0CZT34geo22tvRu4+zTbLXBbOA2Vp5/g4Se3jKqEIiIiIiI1NqLHaIuIiIiInMsUNIuIiIiIDENBs4iIiIjIMBQ0i4iIiIgMQ0GziIiIiMgwRjR6hoiIiEg1Ojo6ANi0aVOUpqHnZCpS0CwiIiLjZsOGDQPSgpFqRaYWdc8QERERERmGgmYRERERkWEoaBYRERERGYb6NIuIiMi4ueGGGwA4efIkra2ttS2MyBlQ0CwiIiLjpjh6RkdHh0bNkClN3TNERERERIahoFlEREREZBgKmkVEREREhqGgWURERERkGAqaRURERESGoaBZRERERGQYCppFRERERIahcZpFRERk3BhjBqRZa2tQEpEzo5ZmEREREZFhKGgWERERERmGgmYRERERkWEoaBYRERERGYaCZhERERk31lqstdx3333RsshUpKBZRERERGQYCppFRERERIahoFlEREREZBgKmkVEREREhqGgWURERERkGAqaRURERESGoaBZRERERGQYiVoXQERERM5e69evB+DkyZO0trYC0NHRUbPyiFRr2KDZGHMbcGtF8iFr7exwuwm33wxMAx4DPmitfS72Gmngb4C3A3XAz4APWGv3jsF7EBERkUnq/vvvr3URRMbESLtnbAXmxKaLY9s+CnwEuAW4HDgM3GOMaYrl+SzwVoKg+XqgGbjLGOOeSeFFRERERCbCSLtnFKy1BysTw1bmDwGftNZ+N0x7F0HgfBPwj8aYFuC3gfdYa+8J87wD2A28Cvjxmb4JEREREZHxNNKW5iXGmH3GmJ3GmDuMMUvC9MXAbOAnxYzW2j7gAeCaMGktkKzIswd4IZZHRERERGTSMtba02cw5rVAE7AFmAV8AjgfuBBYCTwELLTWvhz7m68C86y1rzHG3AR8E0ja2M6MMfcCL1lr3zfEfm8m6CdNe3v72jvuuKPqN1mt7u5uGhsbJ3y/U53qrTqqt+qo3qqjequO6m30Nm3aBEBfXx91dXUArFmzpnYFmmJ0zFWn2nrbsGHDk9badYNtG7Z7hrX2h/F1Y8yjwA7gXcCjxWwVf2YGSat02jzW2i8DXwZYt26dLd59O5E6OjqoxX6nOtVbdVRv1VG9VUf1Vh3V2+gV60t1Vx3VW3XGo95GPU6ztbYbeA5YDhT7Oc+uyDYLOBQuHwRcYMZp8oiIiIiITFqjDpqNMRmC7hkHgJ0EQfGNFduvBx4Ok54E8hV55gOrYnlERERERCatkYzT/DfA/wAvE7QO/xnQAHzDWmuNMZ8FPm6M2QK8SNDnuRv4FoC1ttMY88/Ap40xh4FjwGeAzcBPx/wdiYiIiIiMsZEMOTcf+DZB94ojBP2Yr7LW7g63f4rggSVfovRwk1dba7tir/FhoADcSenhJu+01npj8SZERERERMbTSG4E/I1htlvgtnAaKk8/wcNPbhld8UREREREam/UfZpFRERERM41I30ioIiIiMio3XbbbQDs2rWLjo6OsjSRqURBs4iIiIyb22+/fUCagmaZitQ9Q0RERERkGAqaRURERESGoaBZRERERGQY6tMsIiIi4+bWW28FghsBFy1aVNvCiJwBBc0iIiIyboo3/XV0dLB+/fqalkXkTKh7hoiIiIjIMBQ0i4iIiIgMQ0GziIiIiMgwFDSLiIiIiAxDQbOIiIiIyDAUNIuIiIiIDENDzomIiMi46ejoAGDTpk1Rmoaek6lIQbOIiIiMmw0bNgxIs9bWoCQiZ0bdM0REREREhqGgWURERERkGAqaRURERESGoT7NIiIiMm5uuOEGAE6ePElra2ttCyNyBhQ0i4iIyLgpjp7R0dGhUTNkSlP3DBERERGRYShoFhEREREZhoJmEREREZFhKGgWERERERmGgmYRERERkWEoaBYRERERGYaCZhERERGRYWicZhERERk3xpgBadbaGpRE5MyopVlEREREZBgKmkVEREREhqGgWURERERkGAqaRURERESGMeqg2Rjzp8YYa4z5YizNGGNuM8bsN8b0GWM6jDEXVvxd2hjzBWPMUWNMjzHm+8aY+WPxJkRERGRystZireW+++6LlkWmolEFzcaYq4D3ApsrNn0U+AhwC3A5cBi4xxjTFMvzWeCtwNuB64Fm4C5jjFtVyUVEREREJsiIg2ZjTAvwb8BvAydi6Qb4EPBJa+13rbXPAu8CmoCbYn/728AfW2vvsdb+AngHcAnwqrF5KyIiIiIi42M0Lc1fBv7DWntvRfpiYDbwk2KCtbYPeAC4JkxaCyQr8uwBXojlERERERGZlMxI+hYZY94LvB+42lqbM8Z0AM9aa3/PGHMN8BCw0Fr7cuxvvgrMs9a+xhhzE/BNIGljOzTG3Au8ZK193yD7vBm4GaC9vX3tHXfccSbvsyrd3d00NjZO+H6nOtVbdVRv1VG9VUf1Vh3VW/VUd9VRvVWn2nrbsGHDk9badYNtG/aJgMaYlcBfAddba3OnyVoZfZtB0ga8/FB5rLVfJmjdZt26dXb9+vXDFXXMdXR0UIv9TnWqt+qo3qqjequO6q06qrfqqe6qo3qrznjU20i6Z1wNzACeNcYUjDEF4AbgA+HysTDf7Iq/mwUcCpcPAm74OkPlERERERGZlEYSNP83cDGwJjZtBO4Il18kCIpvLP6BMSZDMELGw2HSk0C+Is98YFUsj4iIiIjIpDRs9wxr7UngZDzNGNMDHA9HysAY81ng48aYLQRB9CeAbuBb4Wt0GmP+Gfi0MeYwQev0ZwiGrvvpGL0XERERmWSKl8hPnjxJa2srEFw6F5lqhg2aR+hTQB3wJWAa8BjwamttVyzPh4ECcGeY92fAO6213hiVQURERCaZ+++/v9ZFkBGy1pLfu5f+518gu3UrfrYf4yYwiQQk3HDZBXeI5UQiWjbpDMl5c0nOm4eTTtf6rY2JqoJma+36inUL3BZOQ/1NP8HDT26pZp8iIiIik5G1Fu/ECfJ79pDbs5f83r3k9u4hv2cvWEti1iwSM2fG5jNJzJxJctYsnIaG2pS5UCC7YwfZF16g//kX6H/hBfq3bME/dSrI4DiYVArreZDPn9G+ErNmkZw/n9SC+STnL4gtzycxaxbGGfUDqmtirFqaRUREREak+8Gfk165gsTMmQTPSJv8/P5+8vv2kduzh/zefUGAvHcv+T17yO/di9/bW5bfnTmD1Lz54Dj0Pf00hcOHsdnsgNd16usHCarD+cwZJHbvJrt9O059PU5dHaa+HpNMjqre/P5+si++SP/zz0cBcvbFF6PymHSa9MqVNL/2tWRWrSJz4QWkly/HyWSA4KQA3w8C6EIB63nYQgHCuS144JWn+7295PfvL6uvnsefoPD9/4HYcMcmmSQ5bx7JBQuCQHrefJIL5pNesoT0smXVfFTjRkGziIiITKg9730vAG5bG+mVK8isWEn6/PPJrFxBaunSml3Ot55Hft8+stu3k9uxk+yO7eR27Sa/Zw+Fw4fL8pq6OlLz55NcsID6q64kNX8ByQXzSS1YEHRJqKsrf21r8bu6KBw+TOHIkdL8yBHy4XLfM88EwXV/f/R304EdlQV1XZy6urJAOr7u1Ndh6urwe3vJvvAC2R07wQt6wzrNzWRWrWLa299O5oJVZFatIrV4cdAFYwjGmLAbhgup1MgrdO3aAUl+Lkdh/35ye/eR37unLKju3LwZv7MTgPorr2ThN74+8n1NAAXNIiIiZzlrLX2FPrpyXXTnu4Mp101XvoueXA/d+e7Stlx3+Tzctn7Bem6/5vZR7/u+++4DYNOmTaxZswavu5vzZs4iu3Ur/S9uJbtlKyfuuKPUCuu6pBYvIrPyfNIrVwaB9MKFOE1NwTSaoG0Ifn8/uV27yO3YQXb7jiA43rGT3M6d2FzpkRTujBmkFy2i4brrYl0L5pFasAB3+vRRtfYaY3Cbm3Gbm0/bgmqtxe/uDgPqozz92KNcuHQpfm8vtq8Pv7cPv68Pv7cXv68XG1v3Tp2icOggfk8vfl8fJpUic/75NL7qVUEL8gUXkJw3r6at+04qRWrRIlKLFg263evsJLd37/BP+qgBBc0iInLWs9ay89RONh7cyLH+Y/jWjybPevh+MLdYPN8rpYdzay2e9Th89DBPPPEEs+pn0V7fzqz6Wcysn8ms+lmk3fFrHS34BbpyXaUpH8y7c92cyp2KAtviVLnek+/BG+a+e4OhIdlAY6qRxmQwTctM47ym82hMNXLJzEuqKnv8ARPx5YYrr4iWreeR2/0y2a1b6N+6lezWF+l96hecuvvugeVMpXCamnAbG8NAuhG3MQio3aZGnIbG0nJjE05DA4VDB8lu3xEEyTt2kN+7t9RFwBiS84PuAA3XXkt6yWJSS5aSXrIYNxztYyIZY3CbmnCbmkgvWUKur5eWc+jhJm5LC3UtLbUuxqAUNIuIyFlpf/d+HjvwGI8dfIzHDzzOkb4jZdsNBte4OMbBdYJ5cSqmVy73Znt5duuz9Hv9A/bXmm5lVv2ssoC6cj3tpjmVO0VntnPAvGw518mp7KlovTvfPez7bUw20pRqojHVSFOyifb6dpa2LqUp2RSlR3mSjQPW65P1OKY2N2QZ1yW9ZDHpJYtpfu1ro3Tv1CmyW7eS378fr6sbv7sLr6sLP1ruxu/qInvkSJDW1TWgb3G0j7CFM3PRhbS86U2kly4htWQJqUWLor67IqejoFlERM4KR/uO8viBx3n84OM8duAx9nbvBaAt08aVs6/kijlXcMXsK5jXOA/HOFVdou7o6OCGG27gVO4Uh3sPc6T3CId6D3G493A0Heo9xAvHXuB4/3HsKK4xJ5wEzalmWtIttKRamFk/k2Wty2hJt9CcaqY53UxzqjkKdItTY6qRhkQDruOO+v1Mdm5zM/WXXz6qv7Geh9/Tg9/Vhdfdjd/dHYxUMW9e0CdXpEoKmkVEZErqzHay8dDGKFDednIbAE3JJtbNXsf/vuB/c8XsK1jWumxM+3AaY4LANt3C8mnLh8yX9/Mc7T1aFlTn/BwtqRaa0820pFqigLgl3UJdom7KjCQxmRnXjfoOJ2tdGDmrKGgWEZER863PC8df4GDPQVJOiqSbJOWkSLkpkk5ywHrKTZFyUiScxKABYcEvkPNy9Hv9ZAtZ+r3+svWsF0zx9YM9B3n84OO8cPwFfOuTcTNc1n4Zb1jyBq6acxXnt50/KVpdk06SOY1zmNM4p9ZFEZExoKBZRERO61jfMR7e/zAP7X+IR/Y/wvH+41W9TjGITjgJCn6BbCFLwRZG/ToJJ8Hqmat5/yXv54o5V3DxjItJuWc+ooKIyOkoaBYRkTIFv8DmI5v5+b6f89D+h3j+2PNA0Df4mrnXcO28a1naspS8nyfn5cj7+Wi5uJ7zg+ViS3LOz5H38tE86SZJu2nSbpqMmyHlpsgkMlFa2k0Pup5yUzQmGxUki8iEU9AsIiIc7DnIQ/se4qH9D/Ho/kfpynfhGpfVM1dzy6W3cO28a1nVtqpmoyvI1HXbbbcBsGvXLjo6OsrSRKYSBc0iIuegvM3zyP5HokC5eBNde307r170aq6ddy1XzrmS5lRzjUsqU93ttw98IIqCZpmKFDSLiJxjHtj7AB/b8zGyL2dJOknWtq/lLcvewrVzr2Vp61KN4CAiMggFzSIi55BtJ7bx0Qc+yozEDP70FX/KuvZ11Cfra10sEZFJT0GziMg54kT/CW659xbqEnW8r+19vGL+K2pdJBGRKUNBs4jIOSDv5fnDjj/kcO9hvvbLX+P4c9UNGycyWrfeeisQ3Ai4aNGi2hZG5AwoaBYROctZa/nrx/+ajYc28lfX/RWXzLyEDjpqXSw5RxRv+uvo6GD9+vU1LYvImdDYQSIiZ7k7tt7Bd178Dr910W/xxqVvrHVxRESmJAXNIiJnsUf2P8L/9/j/x/r56/n9S3+/1sUREZmyFDSLiJyldp/azUfu/wiLWxbzyVd8Etdxa10kEZEpS0GziMhZ6FTuFL/3s9/DNS5feOUXaEg21LpIIiJTmm4EFBE5yxT8Ah+9/6Ps7drLl1/9ZeY3za91kUREpjy1NIuInGU+8+RneGj/Q3z8qo9z+ezLa10cEZGzglqaRUTOIv/50n/yL8//C7+56jf5tRW/VuviiNDR0QHApk2bojQNPSdTkYJmEZGzxC8O/YK/ePQvuHrO1fzRuj+qdXFEANiwYcOANGttDUoicmbUPUNE5Cywr3sfH+74MPMb5/PpGz5NwlGbiIjIWFLQLCIyxfXme7nl3lvIe3k+/8rP05JuqXWRRETOOmqKEBGZwnzr8ycP/gnbT27n//7S/2Vxy+JaF0lE5KykoFlEZAr74lNf5L499/EnV/wJ18y7ptbFERnghhtuAODkyZO0trbWtjAiZ0BBs4jIFHX3jrv5yjNf4a3L38pN599U6+KIDKo4ekZHR4dGzZApTUGziEwJ1lr6Cn105brozndH86d7n6bhYAOt6VamZabRkm4h6SRrXdxx98yRZ/jzh/6cy2Zdxsev/DjGmFoXSUTkrKageQjPHHmGB7oe4NT2UzQkGqhP1tOQbKAx2Rgt1yfqcR231kUVmbJ68708sPcB9vfspztXCoS7c9105bvoznVHAXJPvgfPeoO+zj/9+J/K1huTjVEQHQ+mp6Wn0ZppDebp1mDKBPPJPNqEtZYDPQd4+sjTwXT4abYc30J7Qzt/t+HvSLpn/0mCiEitTd5fiRp7YN8DfOf4d/jOz79z2nx1iToakg1REF1cbkg2MLdxLiumrWDFtBUsbF44qX+URSaKb302HtzI97Z/j3t230NfoQ8A17g0phppTDbSlGqiMdnInMY5NCWbytNTjWVpz2x6hhUXr+BE9gSd/Z2cyJ7gZPYkJ/qD+bH+Y2w/uZ0T2RPRvioZTBBUZ6YxLT2NtkxbsJwJl9Ox5TDPeAaq/YV+nj/2fBQkbz6ymSN9R4DgO+fC6RfyzgvfydtWvI22TNu4lUNEREqGjeKMMR8E3gcsCpOeA/7SWnt3uN0AtwI3A9OAx4APWmufi71GGvgb4O1AHfAz4APW2r1j9k7G2M0X38x5x85jzeVr6Cn00JMfOPXme+nOd0fLPYUeunPdHOo9RHeum5/s+gkFWwAg5aRY2rqU5dOWs2Laimg+o25Gjd+pyMTY1bmL72//PnftuIsDPQdoTDbyusWv4w1L3sAF0y+gLlFXVReDk6mTXDnnyhHl7S/0czJ7ks5sEFyf6A+ncPl4/3FO9J9gZ+dOfnH4F5zMnsS3/qCv1ZRqoi3TFrVkFwPraF6xXJ+oH/T9WWs52HMwCpA3Hd7ElhNbKPjBd8f8xvlcMecKVs9czeqZq1k+bfk50f1ERGSyGUnT517g/wAvEYzr/C7gv40xa621m4GPAh8B3g1sBf4cuMcYs9Ja2xW+xmeBNxMEzceAzwB3ha8x+PXWGku6SZrcJhY0L6j6NfJenh2dO3jxxIu8dOIlXjzxIo/sf4Tvb/9+lKct0xa1RhenJa1LSLvpsXgbIjXVme3kRzt/xPd3fJ/NRzbjGIer517Nh9d+mA0LNpBJZCa0PJlEhtmJ2cxumD2i/J7vcSp3qhRQVwTXxYD7QPcBnj/6PMezx6Ngt1LKSdGaaS0LtPNens1HNnO473BQPjfDhTMu5J0XvJPVM1dzycxLdGItIjJJDBs0W2u/V5H0cWPM7wJXG2OeAT4EfNJa+10AY8y7gMPATcA/GmNagN8G3mOtvSfM8w5gN/Aq4Mdj9F4mnaSbZGXbSla2rSxLP9F/Igqii9OdW+8k62WB4DL1wuaFrJq+ilVtq7hg+gWsbFtJc6q5Fm9jVPJenqyfrXUxpIbyfp6H9j3E97d/n449HeT9PMtal/GRtR/hdUtex6z6WbUu4oi5jhu1FC9hybD5rbX05Hui4Ppk9mQpwC6m9Z/kePY4+4/uB2Dd7HVBK/Ks1ayYtkKtyCIik9SoOtkaY1zgbUAj8DCwGJgN/KSYx1rbZ4x5ALgG+EdgLZCsyLPHGPNCmOesDZqHMi0zjSvmXMEVc66I0jzfY0/XniiI3np8KxsPbuTuHXdHeRY0LWBV2ypWTV/FBW0XcP7088e0P6O1lq58F8f7jnMqdyq6ISs+UkFxvXijVk+up2x7MfD/83/7c2bWz2Rm3cxoPqt+FjPqZpTNG5INY1Z+qR1rLVuOb+H727/PD3b+gOP9x2nLtPHrK3+dNy59I6vaVp0TozsYY4K+1qlGFjRVf5VKREQmH2OtHT6TMRcDjwAZoBv4TWvt3caYa4CHgIXW2pdj+b8KzLPWvsYYcxPwTSBpYzszxtwLvGStfd8Q+7yZoJ807e3ta++4445q32PVuru7aWxsnPD9xnV5XezN7WVPbg97cnvYm9vL0cLRaPs0dxrzU/OZn5rPgtQCFqQW0OK2RAGKb316/B66vC5Oeafo8rro8ruCecVyt9dNgcEvLRdlTIaMk6HOqSPjZMiYYDm+7uU9+t1+Tnmn6PQ6o3nO5ga8XsqkaHFbaHFbaHabaXFbmJ6YTnuynfZkO61u6zkRbMHkON5Go8vrYnd2NzuzO3m271n25/eTIMFF9RdxRcMVXFB3Aa4Z/9Flplq9TRaqt+qo3kZvw4YNA9Luu+++GpRkatIxV51q623Dhg1PWmvXDbZtpC3NW4E1QCvwVuAbxpj1se2VkbcZJK3SafNYa78MfBlg3bp1thYDok/Wgdg7s51sPb6VF46/wPPHnmfL8S38qPNH2LA62zJttGXaON5/fMgbmRJOgumZ6bTVtbGwbmG0PD0znbZMGy3pFhqTjdFIBQ2pBhoSDSMaYm+werPW0p3v5kjfEY70HhkwP9x7mKN9R3mh74WyEQ7qEnUsal7E4pbFZdPC5oVn1O877+c51neMY33HONp3NJr6vX4yboa6RB11yToybob6RD11iToyiTC9YjnpJMcksJ+sxxtAwS/w4okX2XxkczSaw8tdwXmya1wumnER71nyHl67+LW0pFsmtGyTud4mM9VbdVRvY0N1OHI65qozHvU2oqDZWpsDtoWrG40xlwMfBv7fMG02sCf2J7OAQ+HyQcAFZgBHKvI8UF2xz20t6ZYB3Tt68728eOJFnj/2PM8fe56uXBeXzrqUtkwb0+umR8Hw9Lpg3pxqntAWXGMMTakmmlJNLGkZum+otZZj/cfY2bmzbNp0eBM/2PmD0uthmNc4b0AwPa9xHt25bo72B0FwZVBcTDuRPTHo/l3jDjkW8FBc40YB9Pym+WU3dS5rXUZjauq1EBztO1oWID937LnoZGZG3QxWz1zNW1e8ldUzV0cjX4iIiJzNqh042AHSwE6CoPhG4AkAY0wGuB744zDvk0A+zPOtMM98YBVBv2gZA/XJetbMWsOaWWtqXZQzYoxhRt0MZtTN4PLZl5dt6yv0sfvU7iiQ3tW5i52ndvLEwSfo9/qHfM2Uk2Jm/Uym103nvKbzuGzWZcyom8H0uunMrJsZ7W963XRSboq8n6ev0Ed/oZ++Ql+03FvoHTS9uNyT72H3qd38YMcPuDN/Z7T/eY3zWD5tOctbl7OiLQimz2s6b9TjdvvW50T/CQ71HuJw72EO9x7mYM9BDvce5lDvIfJ+nkwiQ8bNkHbT1CXqSLvpUloiTcbNDLpuMLxw/IUoUN7XvQ8IrkisalvFry7/1Wg0h7kNc8+ZLjMiIiJFIxmn+ZPA3QQtyU0Eo2KsB15vrbXGmM8SjKixBXgR+ARBv+dvAVhrO40x/wx82hhzmNKQc5uBn471G5KzV12ijvPbzuf8tvPL0n3rc7DnIDs7d7Kvex9NqaYoEJ5RN4PGZOOogrykkySZSlY9Wklx3N0XT7zISydf4sXjwc2dD+59MGrFTrtplrQsKbVKt63gSP4ITx1+ikO9hzjUUwqMi0Hyod5DA4Yzc43LjLoZtNe3k3JTnMqe4rB3mP5CP9lClj6vj2whS84f2J98MLPqZrF61mrefv7bWT1zNee3nT/hw8KJiIhMRiNp6poN/Gs47yQIdl9rrS2OevEpggeWfInSw01eHRujGYKuHAXgTkoPN3nnZB2jWaYWxzjMbZzL3Ma5tS4KELSWz2mcw5zGOdyw4IYoPetl2dm5MxghJQykH9r/EN/bHhvVcX9pMeNmaG9oZ1b9LNbMWkN7fbA8u342s+pn0d7QzvTM9BH1M/d8j6yXJetl6S/00+/1ly0X/ALLWpeNePxiEZGRKo4BoL65MtWNZJzmdw+z3QK3hdNQefqBW8JJ5JyUdtOllvKlpfRjfcd46eRLdDzZwXWXXhcExPXtY9rv3HVc6p166pP1Y/J6IiIi55pq+zSLyBiZXjed6XXT6W/s57p519W6OCIiIjIIp9YFEBERERGZ7BQ0i4iIiIgMQ0GziIiIiMgwFDSLiIiIiAxDQbOIiIiIyDA0eoaIiIiMm+LYzCdPnqS1tRUIxmwWmWoUNIuIiMi4uf/++2tdBJExoe4ZIiIiIiLDUNAsIiIiIjIMBc0iIiIiIsNQn2YREREZN/fddx8AmzZtYs2aNbUtjMgZUNAsIiIi46Y4ekblsshUo+4ZIiIiIiLDUNAsIiIiIjIMBc0iIiIiIsNQ0CwiIiIiMgwFzSIiIiIiw1DQLCIiIiIyDA05JyIiIuPmtttuA2DXrl10dHSUpYlMJQqaRUREZNzcfvvtA9IUNMtUpO4ZIiIiIiLDUNAsIiIiIjIMBc0iIiIiIsNQn2YREREZN7feeisQ3Ai4aNGi2hZG5AwoaBYREZFxU7zpr6Ojg/Xr19e0LCJnQt0zRERERESGoaBZRERERGQYCppFRERERIahoFlEREREZBgKmkVEREREhqGgWURERERkGBpyTkRERMZNR0cHAJs2bYrSNPScTEXDBs3GmI8BvwqsBLLAo8DHrLXPxvIY4FbgZmAa8BjwQWvtc7E8aeBvgLcDdcDPgA9Ya/eO2bsRERGRSWXDhg0D0qy1NSiJyJkZSfeM9cDfA9cArwQKwE+NMW2xPB8FPgLcAlwOHAbuMcY0xfJ8FngrQdB8PdAM3GWMcc/sLYiIiIiIjK9hW5qtta+Jrxtj3gF0AtcC/xO2Mn8I+KS19rthnncRBM43Af9ojGkBfht4j7X2ntjr7AZeBfx4rN6QiIiIiMhYq+ZGwKbw706E64uB2cBPihmstX3AAwSt0wBrgWRFnj3AC7E8IiIiIiKTUjU3An4O2AQ8Eq7PDueHKvIdAubF8njA0UHyzGYQxpibCfpI097eHt1IMJG6u7trst+pTvVWHdVbdVRv1VG9VUf1NnqrV68GwPM8XDfokak6HDkdc9UZj3obVdBsjPkMcB1wnbXWq9hc2avfDJI24CWHymOt/TLwZYB169bZWtxp29HRoTt8q6B6q47qrTqqt+qo3qqjehu94qgZqrvqqN6qMx71NuLuGcaYvyO4ie+V1todsU0Hw3lli/EsSq3PBwEXmHGaPCIiIiIik9KIgmZjzOcIbup7pbV2S8XmnQRB8Y2x/BmCETIeDpOeBPIVeeYDq2J5REREREQmpZGM0/wl4B3AW4ATxphii3K3tbbbWmuNMZ8FPm6M2QK8CHwC6Aa+BWCt7TTG/DPwaWPMYeAY8BlgM/DTsX1LIiIiIlINz1o6Cx6deQ/HQL3r0OC61DmGYMC0c9dI+jR/IJz/rCL9duC2cPlTBA8s+RKlh5u82lrbFcv/YYIxnu+k9HCTdw7SN1pERERERsFai2chZy0Fa8n5lj7f52S+QGfB40Teo7PgcTJf4GTBC9MKYZrHyXBbl+cP+voGqHMd6h2HBtehPpxKy27ZtjrXwQFcY3BMOAccY3ANOJSnF/M5BNtnJBNc0do4gTU4vJGM0zzsaYUNHu1zG6UgerA8/QQPP7ll5MUTERGZ2qy15HJH6endRjZ7CN/rw/P68PxgHqz34vn9eF5vsB5ui7b7fXheP8FAVNVySSQaSSSaSSSaSCaao+Vgai7Nk80k3CYSyeYwXxOu24Ax1YxUOzGstfT5llMFj5OFAqfCQLCr4GGBhDGlyTEkTCktaQyuCdOc8rS0Y2hNJEg649fK2uf57Mvm2NufY19/nr39OfZmcxzM5jlkG6nbuJW8b8nbcPJL81xsfaTPWUwaQ2vSpTXh0ppI0J5OsrIhQ2vSpSXhMi2ZoDnh4ltLr+fT4/n0ej69vk9ffN3z6fE8juYK9Prl6Wf6zMfrWhv5j0uXneGrjK1qhpwTERGRCtb69Pfvp6d3G7092+np2UZP7zZ6erZTKHQO+jfGJHDdOhynDtcNJ6cOx60jmWzDdTK4bj2OW4frZDiTh+ha61HwuijkT0Xz/uwhCoVTFApd+H7faf9+1szXcvHFXxz1fjd39fJ7z79MzjYy48kXSTsOKceQcRzSjqlYDuYZxyFlDGk3WLcWTha8ICDOF8LAOFgvtpSeKnjkxvHx3C0Jl7akS1syEU3Tki7To/Xyba1JF9cYrLUcz3tRUBwFxtF6nmP5Qtm+HGB2OsncdJIUlunJBCnHkDQOSScI6FNhcJ8yJkqr3FbnOGWBcEvCpTUZtAiPZ1cLay1Z3+JhsTbo8uEBvgXfBul+mG4BL1z2CbdbS707+R4YraBZZBKz1uJ5PRQKp8gXTlHId4bLnRQKXRTyneHyKRwnTSY9l0xmDunMXDLpOaTTc3DddK3fxqRlrQ/YMwpEZGqx1sdaH2PcqoMG38/T1/dyGBCHAXLvNnp6dpQFnsnkdBoaltHe/noa6pfR0LCMTGZuGADX47oZHCc1Vm/tjPl+LvheCYPoQqEr+N4J1+vrzqvqddOOw4qGNPt7u6h3HXK+5UTeJ+vnyfmWft8n61tyNphn/aED34SB5rB1NJi7zMukaE24NCeC4LAl4dKSLC4naE44OBjyYTCWD7svFHxLwRIsx6Z82M0h7wf5gy4OHsfzhXDyOJjN83x3H8fyBfqHKK8BpiVd+rzgNeLqHMP8TIp56RQXz6hnfibJvEwqTEsyJ52KWrY7OjpYv/ryquq+VowxZNyzr/+zgmaRM9TXt48TJx7hZOcTeIWe0obT/iCb2FKw7PkHeOqpr0VBcHEartt/8XKq72fJ5SqfHxT8cGcyc4MpPYdMZi7pzJwowE6lZg56ydX3C/h+P76fjSavuOxly7fZwoC/L3+3p//yNCaB4yQxJonjpKJ5KS2JMcF6aXuwzVob/LjnT1EonCQfP7HId4Y/+p3k8ycp5IsnHJ1h/lOAjzEpXDeN42TCVr5w2a3DcdK4TiZq6XPK8mXC4Ks4JTA4wdw4UTrGxQnnxrgYwrmTjFoWi62MjhPs81y/4WY4vp8LTyRPlj7rcDn4rDvJF05WLHeGn3kxyHEqPr/gsxsq3RgXzz9Fx/3HsTYflSWdnkNDwzLmzbsiCo4bGpaSTE6rRdVUzXFSpFLTSaWmj+nrnt9YNyDNnqZF2Nqgy0EQQAeBtAFaEy717vi2kFaj1/M5ni9wIgyoj+cLHAsD7GO5AmnHYX4mGQTEmRTz0ynaktWftEntKGgWGaVs9hAnTjzK8ROPcOLEo/T37wEgmWyLfmwG/iDYESz3U/Bmk0q1UV+/iESihWSiiUSyhUSimWSiJeprGC0nGstaST0vSzZ7gP7sAbL9++nv309/9kBwybhnO8ePP4jn9ZaVzJgkqdQMwOL7uSgYnjr36Bruf2DoH2Bj3KAuky0kEi2kktNi9duMMQl8vx/P78f3+sOTgz58rx/Pz1IonCLrHQrqxcsG+fw+fD83ru9pqEv28SAbwLd5fD+PtQWsn8O3BayfD+Y2F20L5qW8np/lgQcbYsF6Jnzt+AlCprTf4klEsZuAkwxPABwwJjxZcAETnoQ5wdw4YZ5gPbjlxwR17vVGU8Hria33hP15w+VCmO734hV6KXjdeF7Paeuv+Jknk60kk63U1S+KjgHHJMIW5wIWH2u9aKKYbr1wWyHMG2zv6TnCeef9Cg31S2loWEZ9/VISiYZxPBbOPcYY0saQdiB4xMPkFtwEF7QSy9lNQbPIMHK5Y5w4+RgnTjzKiROP0NsbPNsnkWhmWuuVnLfgPUybdhUNDSvOqOWgo6ODy9etP6Oyum6a+vpF1NcvGnR70Cp7iv7+/UFwHQbW2dyhsLU3bFV1UjhOGsfNBPMoPV2a3FJ6MVga3PB9DEuBXh7fz4XBXz5Ks34Q7Pm2YpufZ9euHSxddlFwIpFsCU8oikFyc3jz0ti36FjrhycXhVIAVjEHP2iFjwdisTy+zQXBudcbCyL78fzeMGjvi9L88EaxXO5otI4xZa3ujklgnCSOSZJwMzgmiXES4bZksM1JYkyCfXsPMGvWzPBGs/7oxrN8oRMve7Aszff7JuwkypgkrlsfTg0kwuV0ur2UnmgkmQgC4uAzb40C5ESihUSiadxuWOvo6GDZ0vXj8toiMrkpaB7CkSM/xff/lV27nidTt4C6uvOoyywgmZymSyqTiOdl6e3bGfYp3EF/3x4cNxO2zBbvBm8e5G7wJhxn8L6++fwpTsaC5O6erQC4bgOtrZczd+6vM23aVTQ1rppyfWGNMWFw0UJT06paF2dMvPxyBwvPWz/h+zXGiVp7p6ID+zs4f+X6Eef3/XwY2Jda2ot9wq31sYSttNjwJMErLeOHaT4QzB03Q8JtKAuQgxZvtdaJyOSkoHkI/f17sWxm+44Hy9Jdt5G6ugXBlAmD6TCozmTmDhmIyZnJ50/Q07Odnt7tYYC8nd6eHfT176HUkmlIp2bh29yI+gI7TrosqE4mmsjlj9PV9Tzg4zgZWlvWsrT9jUGQ3HQRjpMc77cqMik5YSt1ItFU66KIiNSEguYhLFjwbrZvX8T1119BX99e+vr30Nf3Mn19e+jv20Nv706OHbsf38/G/sqQTs+OAumG+qU0NV1IU9MFU+6GkPFgrYfvF8JL0wWszcfSwn6XtkA2e5Denh3hsE076OndTj5/PHodx0lRX7+EpuaLmT37LdQ3LKGhfhn19Yuilj9rLb7fF94odKrsbvDiDXbxu8KLN4i5bgOLF9/CtNaraGlZrZMgEZEzVLzHo6Ojg/Xr19e2MCJnQEHzMFy3nsbGFTQ2rhiwzVqfXO5oFEzHA+tjxx7gwIH/iPKm03PCAPpCmhovoKnpAtLpOVO6q4e1lmz2AD09L9Hd81IwJmnPNjx/Bw886IQ3H5WC5JH0bY1LJtuor1/CzJk30lC/lPr6JdGQTcN1izDGRJd9Sc8+bV4RERGR4ShoPgPGOKTTs0inZ9Haum7A9lzuON3dL9DV9Rxd3c/T1fU8R4/+jGLwmExOo6nxAhqbLggD6Qupr1806frJBgP27wuD4hej4Lind3vZHezFMUkNFzNr1sLgxrLwxqRguKbieqJ8PbpZycUJR3Kor19CKtVWw3ctIiIiUqKgeRylUm20tV1LW9u1UZrn9dLdvYWurufDYPo59uz5BtYGw1c5Th1NjefT0LiSRHizmhuNWFAcvSAVjVxQvr2UxxgnvFO/EN2170d378fmvhd1kyim+X6Ovr499PS8RE/vS/T0bMf3+2PvaxYNDcuYM+etNDQsD6b6pVGQ29ExuhuMRERERCY7Bc0TzHXraWm5jJaWy6I038/T07ONru7n6Op6nu6u5zly5MfhUE9ZRtutYayk07NpaFjOvHlXhoP1L6OhfhnJZEtNyiMiIiJSKwqaJwHHSdLUtCoYAmxO+TZrbfhAgvBpbF7p6WyVkxc9oS2HtYVY9wc36AJRfGJZ/OllxbnjxvInyGTm6i55ERERkZCC5knOGBM+PjhFAgWxIiIiIrUwPo9MEhERERE5i6ilWURERMZNcWzmkydP0traCgQ3jItMNQqaRUREZNzcf//9tS6CyJhQ9wwRERERkWEoaBYRERERGYaCZhERERGRYahPs4iIiIyb++67D4BNmzaxZs2a2hZG5AwoaBYREZFxUxw9o3JZZKpR9wwRERERkWEoaBYRERERGYaCZhERERGRYShoFhEREREZhoJmEREREZFhKGgWERERERmGhpwTERGRcXPbbbcBsGvXLjo6OsrSRKYSBc0iIiIybm6//fYBaQqaZSpS9wwRERERkWEoaBYRERERGYaCZhERERGRYYyoT7Mx5hXAHwFrgbnAe6y1X49tN8CtwM3ANOAx4IPW2udiedLA3wBvB+qAnwEfsNbuHZN3IiIiIpPOrbfeCgQ3Ai5atKi2hRE5AyO9EbAReBb4ZjhV+ijwEeDdwFbgz4F7jDErrbVdYZ7PAm8mCJqPAZ8B7jLGrLXWetW+AREREZm8ijf9dXR0sH79+pqWReRMjKh7hrX2B9baP7XW/gfgx7eFrcwfAj5prf2utfZZ4F1AE3BTmKcF+G3gj62191hrfwG8A7gEeNVYvRkRERERkfEwFn2aFwOzgZ8UE6y1fcADwDVh0logWZFnD/BCLI+IiIiIyKRkrLWj+wNjuoHfK/ZpNsZcAzwELLTWvhzL91VgnrX2NcaYmwi6dSRtbIfGmHuBl6y17xtkPzcT9JGmvb197R133DHa93bGuru7aWxsnPD9TnWqt+qo3qqjequO6q06qrfqqe6qo3qrTrX1tmHDhiettesG2zaWDzepjL7NIGmVhsxjrf0y8GWAdevW2Vr0g1L/q+qo3qqjequO6q06qrfqqN6qp7qrjuqtOuNRb2PRPeNgOJ9dkT4LOBTL4wIzTpNHRERERGRSGougeSdBUHxjMcEYkwGuBx4Ok54E8hV55gOrYnlERERERCalkY7T3AgsC1cd4DxjzBrguLX2ZWPMZ4GPG2O2AC8CnwC6gW8BWGs7jTH/DHzaGHOY0pBzm4Gfjt3bERERkcmko6MDgE2bNkVp6m4gU9FI+zSvA+6Lrd8eTt8gGJv5UwQPLPkSpYebvDo2RjPAh4ECcCelh5u8U2M0i4iInL02bNgwIG20gxCITAYjCpqttR0EN+0Ntd0Ct4XTUHn6gVvCSURERETOZbke6D4MPUeh50j51DIfrplcIeNYjp4hIiIiIuci34dsJ/R3Qt9J6DsxeDAcTUch3zv4a6WbYdkvTWjxR0JBs4iIiMi5xvfBy0IhC16uYp6FQg6yp8Ig+AT0nywFxIMt959iyJGGnQQ0zISGGcF8+vLScsNMaJxVWq+fAcnMRNXCqChoFhEROcv4vqW/4NGX8+jLe/TnPfrzPtmCR7bgkwunaNmLp3nB3CvPs3p+C++4etGoy3LD1evg+HYKhQIJNww7/vnVYC1gwzlDLIfrTgJSDZBqCucNkG6EVGO4HltOD5LmJsG4wes4Lpghe5yOH98LuiPke4N5rhty8eXitnA5XF+5bw+cuAP8QvAa1gvm8eXB0or5C/2lIDg+9wujfw+JOqhrhUwrZFqgeS7MWhWs14Vp0XJrKVCum1abOh9jCppFRERGwfMteS8IJvNeMOUKxXmwLef55GPBaN6zUV7Pt3jWBvP4ZC2eN/i2gm/xraU/79GX9+nLeeGyV74crmcL/hm/z1TCIe06wTzh0JiuLmTo+M+vwY/+hBMnTzJt2jTAxAKo4nK4PtiyMeDlgyCy92XIdZUFldUxYfDsxuZOxXosHcoDeioX7eB5rA1ab3M9UOgbXflSjZCsY1rBg76GoDzxspUtJ2JlTpZODpwEuClIZCCRAjcNiXSYNsg8kSlPSzcHgXAxIE6kq6nss4aCZhGRCZT3/KjlL5svb+nL5r2yFr+c55e1Cua8Uktg3rMkXUN9KkFDyqU+naAhlaA+7dKYTlCfcqP1hlSCuqSL41TX0lMMEj3fUvAsBd/HP4PBDyyW/pxPVzZPT9ajJ1ugO1uI5qVlb0BaT7ZAf97HMWCMwXHAMQbHGAxBfBWtF5fDPMYYHAO+DVpiC74N58H7Kfg+vh/MPR883y8LaH0/+Pzsj35Q/ZsfIWMg4QTvI+EYHMfgOoZMwqUu5ZJJutQlHepSLtPqk+F6sK0uGW4Pl+uSLpmUSybhkE66pGKBcHxeSndJukF9jYnZF8G77+Lp8XiyXbH1Nmqd7YZsbLmY7uVP30Jr/UHS/dJ6PMiHilbTyjRTNsNNQ6o+1ireAMmG0nLZ1AjJekjWRa/3qJ4IOGkoaBaRs1q24NHZl+dUX57O+NSbp7OvQLZQGvUyHgfassakwVqWgsVsvnj5249dBh963TuTaDPkGEi6DoUwoBsJY6A+WQyuXQrZPup+cX8QEPt+GAxbCp4fzoNgsuBbajE6WCYZtGw2pBPRvL05Q0M6QSbhYAHfBmWz1gaBcLjuWxtOxLaX8phiIBrO3crJGFw3nFds2/vyyyxfupik65B0DemEEy47JKPA05B0g+ViWjIMSIv7iwLhwfZlTNUnOOccx4VMczCJjDMFzSJSc4WwRTUbtqIWW2CL/S+zsb6Yxe39+WD9hW057u96bvDAuC9Pf/70l6kTjilrNDKUrQy2WJY/k3Sj1r90wola/GY0JoIWwUSxlc+lLuWU8iaDlr/ylj43mJe1+sW2h+kJN7hcbK0lW/DpzQUtsr05j55csUXWozdXoCfn0ZutmOcK7NmfpX1WIwk3COQSjiktu6X1pGNwHac8bQxaIeuSLo1pl4YwIG4K5w1hUF98j5NNR8dB1q9fXutiiEgNKGgWibHWDt3fMEwveEGrVenSbml73JCBWMW2ot2nPJ7b34mhdFnZGKJLzhBfL12KLua3lqjFMLqU7gd9KaNWQ6+UVrzkHm9dDNIsXixPWetjeLm6cls+9vrx1yxLj7aHZSj45MN8Z9r42rhnLy11SZrrkrTUJVg8o4GWumTZ1FyxXkxLTtLgbCSMMUHQnnRpa0iN6m87OjpYv37tOJVMROTso6BZqub7lmM9OQ6d6udgZz8HT/Vz6FQ/z72U5Wcnny31OQz7EbpOqU9hMS2+3QkvlwaXWG3Qp9AGgalvbbTs+aXLr164zY/lzfs2CMiKN+N4pZtwcoXy9XzFTTqFMbh0fkYe/nlt91/BMQzeEukEl5STjhNcag5bH4t5MkmHRDoRpgWtlMmoFdOJ0pNu0KqZTrhRi2om6ZJOOrG04nqwnIlvSzo8+tCDvHKQJ46JiIiMJQXNQ3h853G+vz3Hc3YbUN7qV1wekO6E7YlhEJh0nOjHvnhzRfGHP7rsGgsAiv3dKi97FrziTULxuRddzh44dJCH71N+2TfhlO278kaQYlpx3/15j8Onshw8FQbDnf0c6AyC4oNhkHy4q5+8Vx5kOgYaErDp+IEwmC31MfSiPobl/Q+HEw+4i/3+ikF2se+fGwu8U1H/QhP1NaxLujRnEmX9DpNuaXsqEay7jlPWxzHq9+iW35BT1g8y1hex+NGV9YeteI/l/WZLa888+ywXXngREPbTJNZnM5Y3WC/25Szlc03xsnoxSA3eT7IY7BaD2ihYLV2Kd50gAI7//VTpV+mcBcMYiYjI5KegeQiPbD/Gf76Uh5e2Tuh+HUN053Les2QL3hlfuh6NVCIIsnpy3oBt9SmX2c0Z2pszXLG4jfbmDHNagvXZLRlmN2eY0Zji5w8+MOI7fa2tCKr90t3vxeB4zO7gnuRSR7aw/qLZtS6GiIiIDEJB8xBueeUyLnT2cv0rXhG1FMZb/aJW0mKLKaUW1GIrYN4rtQAHNzN5UUtx8YamXMXNT/EhpgZrEY5aiuNpbtBiXbyJKJ1wMIayfVcOV1U2pFV+4NBW0+qTtIeB8OyWYGpKJ8Y8gDXFYaEwOhhFRM5Cg/1u2JFcZhSZZBSnDKF4CT6dcGtdFBERERGpMQXNMoD1bfgQpsnXLcJGHXwpDsCK9SHW8Tf2QKbBx9YNlm3pgU3x7UFfkaAOgksKwXI8zQtuVKSYHqZhbdC3xDUY14ATzI3rRMu4BhN00q7IY8rLKDIIay0ULNbzsXm/dIyXzwZ5Sln8RYJZohcKnVlM5bHpOOBMzv//IiK1pKB5CDbv4eTA68qVB05hIIW1WC8WWEVBVBh02vBvChYbDO2AjS97FjwfW7EcbAuXK/ZZLMOogjpL+d/ZoIxR+eLLxW1FDkFQ55iy+cA0yrbN73I49OxTUfmiQLZYnlIflwHbyoPgWPmjIHnijoGJtgyXvT95MAyoiYJtXINJOGGwHQThQZDjlIId10DCwUk4mKQDSRcnaTBJF5IOTtLBpNzgdVIOxnFKAw+HY9dFrx8L6kvBlBPtJwj2g8AKiI77cqUPatirsMXj2Cv937DBECnYgo22xf9/xPM17TV0P3og+D/k2/J88XXfYguxtOIcMBXHenSSc7rjPxyreMDxHP+/Wfw/VVyO/58tLhf80pQP5hQq0gul9LGyCJeDDzw+dAY39t4rj4Pw/330+Q32mQ4mfjCYYl1SqldTrGPK670ib3SMJsKTUjc2L0sr3178fxI82jh2rFV+JwePBxx4PHmWWfsMxw9vLX1fFd9v5Yl75XdWmG4Mwf/DpAPh3CSdKG3wuRu8r6SLSbs4dQmcTCJ4Dak53/fxfR/P8wadCoXCgOWRpnmex/79++nu7g67MwZf3IPNh9pWVNklJr4+1Lbi6wZP33RGtR4vUzWamppYuXJl1X8/HhQ0D6Hr/r0sudflwL2PTcwOiz8cJvaDFA7LYUyYbigO3xH8TbS9tEw8byIIjAwGjCVcCf82SLelVQgbOw2UB+KepfLEoRh8+GXBAmAtaR/y3T2nf79myJUYW57FxFtj4z9Gp99VVQYrUvnAy4Msm+jkpGqW4Me6AHZgJCqDaMfh5LPbhs5Q/H8T+z8S/Z+Jjw4Sb7WNAp/YCVu0bQSFiv6fmbL9D/i/GgaHpiJQxzHByU3GDYNGBgaWJrYfW1o2NlqIvbdwe5RkOHbsGG2t08qDxtj/+bLvgDDNzxdKJwhDvedofZD/RIPlicoe+zay8fWYigA0frI9rifUse/NRmvoPXgk+tI08c+h+NlS/N5mwLEHDPxuLTaSjJZDKZBOBXOTcnAyCZy0iwmDa6fODYLsVKLUcFM8GS3EyhCeeMa3x4+F1JwGml6xoPp6jLn//vsHBIzFyfM88vl82dzzgpvTbTgk6WinwYK5kU7AaYNi3x/77+piAOo4Dp7nceTIkdgISoMHu6cLiuPBa7E+RqKWfc+nT5+uoHmqsBZydZa6TCb8gmPw1l5/jL6sw9en/IG9QVnG4OVPqzJAh1grTtAyY1yDk3IHtNiUWiPD5YTD/gP7mTuvylEgoqaYYNVU/tg4UPqhqggcYkMBVr7kqPYfLVOxbMvSy3/Xg+OgrC4Gbfkauu42/uJJ1l1+eXWFt+GPSd6HnI+f97A5Dxsuk/exOQ8/52MLQbrN+9h8uFwIv/SLJ1lOPA6LnWxV7DNeTlssbyyYsfFgJn41oTIgjQWCg7bsVrZ6OqU6PXT4EO3t7QOurMSDvcrWQ+Lr0dWX+FUNSldeqgnIYsFdPHGwlxmz/9+DnugNshLO6n3InerCJEypFTn8P2+SpWO47Hh1SsvRE3Xib6Ky5bViW3BMxNYrr975pfTo+ClewStefSrGJ5XfPWVXSmINEIOdMMWrZIgvXBs/Pv1S2Y4dOUbbtJaBLdXRsRYPSm3YncYbu9+KOB9s1sNmvQk5xc7tPFVV0Pzoo4/ywx/+sCztvvvuG/XrFIPIoaZEIoHjOLiui+u6ZcsQfKa+7w86P91yZVqxZbmYPh6KgXjxZKEyMB+udTe+LT4/3fJg24r1drq6GKy+KteHep3B0otpra2t41K3Z0JB8xASbRkKdZCYUR8Fg1HQUwyCEpU/KhWXCxOxL+7TdGcothwFP0SU8kEpWIwHh2EQY4rpFS3IUa7iPoZq5YoHyWPo6Y59XLh+2Zi/7tkutw1ScxpqXYwp55mOg1yw/vxx3Ue8W1NlcFcWBEZ/EJ/b0oWRypMvS/gdwOCt0sWuCcX/qxX5zuT/b/BEwGuq/vtz1bMdHaxcf3FVfxudyEHZsWDLjpdwofKkvRhMVJ4IxrrzlboiBYG7n/Px+wvY/gJ+v4fNe6WuLQO63pROQMrSiycmxuA2Jat638uWLeOGG25g9+7dLF26dECwG9TB6VuKi63OuVzutPO+vj7y+TyFQmHU5RwqGHddl3Q6TTqdJpVKkUqlouWh5vHlRCIItU7XteJ085///OfccMMNY9LlQc6MguYhNKxtZ3/XC6xYf1GtizIuPN8j5+WiKetlS3M/F12+cQjPYqNA3eAYJ3osdHFblGbgUP4Q209ux7d+aSI4M/fxy9MHmYqvC4SPiA7KUdx/cZ9ly7G5a1xcxyVhEuXLjjvoesIJ8jnGGdcvI9/65LwceT8/6HxndidNh5oo+IWyKW/zA9KiyRbI+8F2z/fwrY9nh5gPsr2Y5sfaqaLPtqJlsjJ9wGdknOh4cY0bzR3jRJPB4DpuMI9tS7pJ0m46mlJuioybIeWmBl1Pu2nSiWBerIPhyhhfHy1jDLilyii+SjDMpBd9Hp71os+j4Bfw8AZ8XtE238Niy95T9H6dUlrSSZ61P5LF4zLv5ynYoE4GrStbXo++9cuOK9eErYrFYwqnbL3yOHRMEKgNdbwUDTh+MPT5fXTnuoHStYP4NYTBWh4r0xwnLKOJtewR+38yhp93cIz62PBfaUNxFiv7IO+j9B7BmupGk2rO5bh42zYadu5iqSW8X8MNG4jcMIgP0oJg3g3zBJNx3eBk0U1AfSJqTCp2XSzrIxOeVFqg4HnkfJ+C74NxcBMujpsonyeTQf0nEhjXDfYV7b9UDuPEyhNfNgbC8o3X/9Ni4C61p6B5CFkvS6/Xy4n+EwMDu1gA6FkvOAseZJ71svQV+sqm/kL/gLS+Qh+9hV768mEer59sIVsWkEZfrmFQEg9CKrc7xsG3PlkvS97Pk/WywbKXj4Ljgh39WfiofG98X368VP7IFoPsyjTHOIOmF2yBvBcEwTk/Fhh7+ZHV+Y+qL3s8+B90XhFExNOKge+APnFRS1jlD2hFurX4+NGxP9jJkGXwbcWA6Iz8S3V/Fp0YDGHwDhUTb6gTh+J3QOWJbVkaDDy5xHDi5An+7Sf/VrafyvoYEETGtsc/u+IJWPGkIb4eD3o93ysFx7YQnSRPOd8e/10MdhLqGCdoeQ2PS9/60boNW6eL/w/H49h97aLX8qkbPjXqv8vv3cuRv/0MTcDhMS/VJGLM8EF1tM1gzGmWo9cytPX2seNznw9vxHVLrxk/6TDhtuLrF9OMAeuHN0b7WD/sKuR7sbSKbZ4XpWGLXdhsuF68qTlYt9YPu+QVt/lRd6voEoqJn9CUrowH31lm4PZgI/WXXsa8v/2bCf0Ih6OgeQhfffar/P3ev4c7x+f1M26GukRd2ZRJZJhRNyNaBkoBRxiURMth8D7YsrUWDDSnm6Mf25STKmu1i7doVaannFQUQFnswPnp0qzl+Ree56ILLioL5staesJWoHjLSmULUOXrFusivs94vuI2LMEPdvijXfyBrvxxj6cP9sPvWz9q0YoHepWttJWtua5xy+o76STL5mXLToqkm4zmW57bwmWrLyPhJEg6SRJOIphMorQcTpXbXWdqt0L41i+d4BWyZVc+4lPOy9Hv9ZddHdmybQuLFy8eMrCvbFGrPHaGC5yHaz0qXq0ofibF9aE+v3h+YMCVnsHe+1DbPOtFx33l/8coLZZe7BdpseRtnv5Cf/Q+KoOsAUHXIDFY8T2lTXrQKzjF91uWZoJ+pwkT1IHruGXHcfzYLqvLWD0WGwbK/m/Grpp4Nrg5q/Ikrbge/14Z8L4rj6OK9W3bt7Fsaan72WCt1MNd2Rjse32o7/piQ0yx/INd3SueCEUt6LGTpLKrdJhRlbNy++LmxYPmG07d2rWsfOoXPHj//Vx37bVBUGbD4MwLArPB0oK+4MWArhiERZ3OS92lyvrPx4K1+PayQDE2L75+uC/reeENkeXz4LX8Utl8v/SaxeVYcBrkCd9PWSDpl+ePXscv5atY9g4fJtnWVh7UWr9UvkIBGwt0o4DXD+rUOG55MF8RZJtEImhRN+XbBgbzJshT7FIaC+4xhKMyxbfHGmLi9wgU17EDt8c+t9TSJVUdb+NJQfMQrp17LYd2H2L58uWDXtobLACMX5I2xpB201EAXAyM6xP1ZBKZ6MvtbFT/cj3rF6+vdTGmHHeHy9Vzr651MWrCMU7UekpqdH/bcaSD9ZesH5dync2CPs3ra12MKafjSAfrL1xf62JMKcZxMHV12EwGt6mp1sWZcrZ1dHCp/q9OCgqah3DJzEs43nyc9avW17ooIiIiIlJjCpqH0NOZpe+E5dj+blzXwXENTjQ3uInS8tl6k46ML9/zKeR9/IKlkPfJdVs6j/RGVyCD7i6loa9seOnRFoffIriqGV2KL7vbvvJSe4UhujsWR3EodTErjtwQu4Q7yHrxNf1wWC0/fJhHsay2OL73UOnxtNjoFL4ff/+D5z+6zfJU7uVBb3Iqe8uD3KBV/t5j74/S+45vM/EhDwHjGBzH4CRM6Xsi4eCG3w1O+D3huk5ZnuL3h4mPnBGryniPkcp6jufxPYvvWbyCH829QnBMeV7FvLgtzHd8m+U5d1/0Por7Kt1XVSxb+d38URqmrJwD6/P0CU74sBLHKX2PGteUrRfryAlHFHLC7dFIIvGXNmbo+ovXXfh+inWv728RGSkFzUN4/K6d7HjQsuPHp3liVshEX8JEDyqIfkxjaZVf/ME8HAM5PoU/FlGXrWKgMUQQUhaoFMenhegH23Gd0v4qfoSCLkomeMZD7Ac8+gk3sR+a2I9k+Q+nJR5IHNjn8+ChF8EE77/4SF4nCsBKc8cpLUfdpTDB38ReMwpgTHhbtCEKFONBVakOyutjsHTf2mjYpmi5shtn1N2qGMkWu80VPwui9KB7WxCU+AVLoeDjez5eGLD4YdDiFXy8fNR5C/CCOjQ+L/3g/uDN2dgbt2FlF9eH6YN7Ljq06TQPN5EhHdi4tdZFqLnid2D59zRRcG4q1vv6ffZ3PFr2AvH7mMKlspOuYNvp/98OdVJXmRx0Ia34Hi9+x0YnYrHfm4rfoNLvT8VvT+w3yan8TQrzN03PMHtJy+gqGLj+uuvo7zpFb28vmUwa61s+94n/g5fPUcjlKeRyFPK5YD2fp5DL4uXyFGLr1vPCug77zxon/O0trRdvtiud5DrBt2U0lnHsb5zieulvgnG9K9Ni99kUxx72veh32Y/1Py4ul/KF/YtNbDxlJ+hH7IR9jCuXy9ZNsHz4yGFOPflw7Bix0W9PdFNe8RiyseaDMN04Do7rRq9nwjGso/1VrsfKaAHf87Cehx/23/Y9Dz/sT+1H60E/62jZ90v5w3qoXLZhfflDbJt3/gW8+Y8+MerjbTwpaB5Cwd9L6TmxpxcFbj4M2YR3jjn+0t7x34nxMMYP5o4Pxg/WB3wGsabQAWkDXhTrO2AdrI3Px+tGOwMkwhskxmkXE8HEbuJQUH9axtjwBDt28mkGWTe2tO4U04onnbYU7EXbTdnrOlEwRSndKaWfOHmc6W1tFJ8caEx4EJrSchB7BCd0QaZinvj2Yn7K04uvYwn+3kD0DNLoHDB+gh7+kTWY6LvXYItpxmBtsGyjv3LC9GKAGmwzxfxhLhOebFqCgMl6wXe2F57kWo/gxNm3QTAQrvteeHJcDJA8yNNNIuMTncBawjICONHNpdaW9o814fDMJroZKop1bDwAKj2WvaxBIH41qVilxdqMvW5pPXZlKnZ+PvDqVXhVZ4TaFzfxa//n8uEzVvj5Qw8NSHv8v78TBoZh8Bq70czEA9riCBTGYMKbJA2nv3JkrS22qxRTBslUvhjkt4NkDeswdmYUfa6m8qXMgD+3sS0WYg/qLP/ATNQAE/8AgzyeV6Bnf+Vv6hDvf7DyR+nF4Dq+XJEWP96K/9NMbHSeaDneyBZrECvebBr+37Ym+P9hAWtM+BwpEwX30VQR+IPlxaPHBn+PNaSgeQiLr9qBP/2rJBJJgoPAIToYytaDptTowArXofgF74ctm+EXL6XWTqyJtRSb8Eub4GD1AZPAkMSYBOBiTBITziGBIYHBBeNiCNOMCzY4ozSuj3EsxgRzxw2CS+PYYG4sxnhYgsAT/PDH0Qt/ZNzwvTgY62IJgsggLdxmg/caBJUuWIejR48xffrM8H044fEf/ogUf/RiacV1aw3WD7/8rQlvQA7z+gbfD1/DK+UN6tLEthN8LuHnUTR4I48ZsBq0SoR14wRBuDE2TKtYdrxgPVzG+OGPvVtRP6W6KdZnUF9OKY91OH7iBDOmTwPHhvuPfU6OLfsso7I5NswfnjQUP8vo8ywMMi9uD+aWIM0WDJ4HvmfwCyaYe044j60XHKzvhHncYNlzKq4+lNfxgC4I8c+GYkDnxd5vxft0i2l+eR04lp7eLurrM0FrT9jKEw3FZb0ozdrKyaPUoj+UYVoHMeA7WOti/WAitmytC9GyM/j24olZ+DqE+cpO2nwXP7ZsbTFv6UqELV6RsCbIB+E2J/p/Fm0P/8927+sM/1/H0024XxM7VqeK8Tr7jF/haab3yGD7BfDOYBeF8P9AOBkPnIo0p/goaRN9bgM/9/D3p/i5Ruulpm8T/79YLP2A/wux9WIcl9wNjD5oHsyRBcURSGy0p3gJjC0lBOdupRZUa8sDwfDcI1qO8sYD6zAQM+EJgwkDVANhWhjaxbdZgu9UC9Y4WGOGnjBYxwnmFduK+4/2FZajGCibYotwvAzhsrF24H7CoLS0r6CWyvcZ23e0/4HvPZpXlqmy/qLPpTzNxA6igUeQpWAcrHHwHRPVoe+4+Ca+Hs6Ng3WcaFvzvPkD9l9rCpqHsGz5zezdt4L169fXuihTTkdHB1er3kZNoxlU50zqzdpw6KYw2Cp2wxmY5pelR/kojmMaBuHEl234+uXLwWsEeWy8ZSkW8JWGy4v/QNmK9HhUUWxNpSwtWIsHfKXAafPTm1m9ek34t0EzdNSCRBhshctBMcIgrTgvnvjHW1zDk+EgiA/fix/bf/Hk2BJexvbwfRte2rXhuodvLYSXd4P+8V7YwFC8/F26LB0NQVnsukYs3S+WoTT0HtaGLZhBi7/jFk/abDSErnHAuLHlWPq27S+xfPmy8LOPlaO4XnxQUDQMZnxbsO4kwHEsjhtMlX3b4+HHgM8ViyU8GQyPTRvtyysdk2HrS7xc1vrRZxyd0JYdLxXHTFkaNDaO3ZM3j737Q9ERHcaxwf+oaDlI92MnQQ6GhAMuQZdC1xhcEzZVGINrIGGCroBumFbMF79uXBacV7SomCGW46/jhq2o8deP1ovbTamcEI4Uh43eU/AQx+Bo8a0Nt4dXPAAvSrPs3rWbhYsWRmU5TSP7ALFvifD9lr+3sm+OAdvKP4v4Z0TZtlKrsW9L31QGyDhOMLmGjONQ54brjqHOcci4wXImtlwX/k3CKf9sJgMFzSJyzjLGDa7OnIOMKdDWpsdoj9aegx0sWbG+1sWY8r5y0aJaF2HK6Ni9lfWL59S6GIKCZhERERlH9913HwCbNm1izZo1tS2MyBlQ0CwiIiLjJt59Sl3QZCo7ex9LJyIiIiIyRhQ0i4iIiIgMY8KDZmPMB4wxO40x/caYJ40x1090GURERERERmNCg2ZjzK8DnwP+CrgUeBj4oTHmvIksh4iIiIjIaEx0S/MfAl+31n7FWvuCtfYW4ADwuxNcDhERERGREZuwoNkYkwLWAj+p2PQTQIOFioiIiMikZU73/PYx3ZExc4F9wA3W2gdi6X8O/Ka1dmVF/puBmwHa29vX3nHHHRNSzrju7m4aGxsnfL9TneqtOqq36qjeqqN6q47qbfS+/vWvA5DL5UilUgC8+93vrl2Bphgdc9Wptt42bNjwpLV23WDbahE0v8Ja+2As/Vbg7dbaIZ/PuW7dOrtx48YJKGU5Pda4Oqq36qjeqqN6q47qrTqqt9GrfFQ1wETFHmcDHXPVqbbejDFDBs0T2af5KMGj1WdXpM8CDk1gOURERERERmXCgmZrbQ54ErixYtONBKNoiIiIiIhMShP9GO3PAP9ijHkceAh4PzAX+IcJLoeIiIiIyIhNWJ/maIfGfAD4KDAHeBb4cPzGwCH+5giwewKKV2kGQbcSGR3VW3VUb9VRvVVH9VYd1Vv1VHfVUb1Vp9p6W2itnTnYhgkPmqcSY8zGoTqDy9BUb9VRvVVH9VYd1Vt1VG/VU91VR/VWnfGotwl/jLaIiIiIyFSjoFlEREREZBgKmk/vy7UuwBSlequO6q06qrfqqN6qo3qrnuquOqq36ox5valPs4iIiIjIMNTSLCIiIiIyDAXNIiIiIiLDUNA8BGPMB4wxO40x/caYJ40x19e6TJOZMeY2Y4ytmA7WulyTjTHmFcaY7xtj9oV19O6K7Sasy/3GmD5jTIcx5sIaFXfSGEG9fX2Q4+/RGhV30jDGfMwY84Qx5pQx5ogx5n+MMRdV5NExV2GE9aZjroIx5oPGmM1hvZ0yxjxijHl9bLuOtUGMoN50rA3DGPOnYb18MZY25sebguZBGGN+Hfgc8FfApQSP+f6hMea8mhZs8ttK8NCa4nRxbYszKTUSPNTnD4C+QbZ/FPgIcAtwOXAYuMcY0zRhJZychqs3gJ9Sfvy9bmKKNqmtB/4euAZ4JVAAfmqMaYvl0TE30HqGrzfQMVdpL/B/gMuAdcC9wH8bYy4Jt+tYG9xw9QY61oZkjLkKeC+wuWLT2B9v1lpNFRPwGPCVirSXgL+uddkm6wTcBjxb63JMpQnoBt4dWzfAAeDjsbQ6oAt4X63LO1mmynoL074O3FXrsk32ieDkwwPeGK7rmKui3sI0HXMjq7vjwPt0rFVXb+GyjrWh66kF2E5wctsBfDFMH5fjTS3NFYwxKWAt8JOKTT8haHWQoS0JL5/vNMbcYYxZUusCTTGLgdnEjj1rbR/wADr2RuI6Y8xhY8yLxpivGGNm1bpAk1ATwRXGE+G6jrmRqay3Ih1zQzDGuMaY3yA44XgYHWsjMki9FelYG9yXgf+w1t5bkT4ux1ui2j88i80AXOBQRfoh4FUTX5wp4zHg3cAWYBbwCeBhY8yF1tpjtSzYFDI7nA927M2b4LJMNT8C/hPYCSwC/hK41xiz1lqbrWXBJpnPAZuAR8J1HXMjU1lvoGNuUMaYiwnqKUNwVehXrLXPGGOKgYqOtUEMVW/hZh1rgzDGvBdYBrxjkM3j8t2moHlolQNYm0HSJGSt/WF8PbxJYQfwLuAzNSnU1KVjb5SstXfEVp8xxjwJ7AZeT/Bjc84zxnwGuA64zlrrVWzWMTeEoepNx9yQtgJrgFbgrcA3jDHrY9t1rA1u0Hqz1j6rY20gY8xKgvvOrrfW5k6TdUyPN3XPGOgoQd+12RXpsxh4xiJDsNZ2A88By2tdlimkONqIjr0zZK3dT3BzjY4/wBjzd8DbgVdaa3fENumYO43T1NsAOuYC1tqctXabtXajtfZjBC30H0bH2mmdpt4Gy6tjDa4m6BnwrDGmYIwpADcAHwiXi1e4x/R4U9BcITxjeRK4sWLTjZT3L5LTMMZkgPMJOuLLyOwk+GGJjr2wHq9Hx96oGGNmEFyCO+ePP2PM54CbCAK/LRWbdcwNYZh6Gyy/jrnBOUAaHWujVay3AXSsAfDfBCN0rYlNG4E7wuUXGYfjTd0zBvcZ4F+MMY8DDwHvB+YC/1DTUk1ixpi/Af4HeJngTO7PgAbgG7Us12RjjGkk6IMFwZfiecaYNcBxa+3LxpjPAh83xmwh+E//CYL+bd+qQXEnjdPVWzjdBnyX4EdkEfDXBMML/dcEF3VSMcZ8iaC/31uAE8aYYqtLt7W221prdcwNNFy9hcfjbeiYK2OM+SRwN7CH4ObJmwiG73u9jrWhna7edKwNzlp7EjgZTzPG9BD8lj4brn+WsT7eaj1cyGSdgA8Au4AsQcvzK2pdpsk8EZzd7QdywD6C/+AX1Lpck20i+CK0g0xfD7cbgi/IA0A/cD9wUa3LXevpdPVGMIzQjwl+RHIEff2+DiyodblrPQ1RZxa4LZZHx9wo603H3JD19vWwLrJh3fwUeE1su461UdabjrVR1WMH4ZBz4fqYH28mfGERERERERmC+jSLiIiIiAxDQbOIiIiIyDAUNIuIiIiIDENBs4iIiIjIMBQ0i4iIiIgMQ0GziIiIiMgwFDSLiIiIiAxDQbOIiIiIyDAUNIuIiIiIDOP/B3oh6W8kZjYpAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "np.random.seed(42)\n", "fig, ax = plt.subplots(1, 1, figsize=(12, 6))\n", "idx_perm = np.random.permutation(range(N_train))\n", "for i in range(10):\n", " ax.plot(range(SEG_BASE), X_train[idx_perm[i], 0, :])\n", " ax.plot(range(SEG_BASE, SEG_LEN), Y_train[idx_perm[i], 0, :])\n", "ax.axvline(x=SEG_BASE, color='k', linestyle=\":\", linewidth=\"5\")\n", "ax.grid();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll expose the processed data to PyTorch `DalaLoader`s via the `TensorDataset` class." ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:34.880432Z", "iopub.status.busy": "2020-11-24T17:30:34.879960Z", "iopub.status.idle": "2020-11-24T17:30:34.909774Z", "shell.execute_reply": "2020-11-24T17:30:34.910335Z" } }, "outputs": [], "source": [ "from torch.utils.data import TensorDataset, DataLoader\n", "\n", "BATCH_SIZE = 8\n", "\n", "dl_train, dl_test = [\n", " DataLoader(TensorDataset(X, Y), batch_size=BATCH_SIZE, shuffle=True) \n", " for X, Y in [(X_train, Y_train), (X_test, Y_test)]\n", "]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Both the encoder and decoder will use the same model, a 1D CNN." ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:34.915049Z", "iopub.status.busy": "2020-11-24T17:30:34.914553Z", "iopub.status.idle": "2020-11-24T17:30:34.944725Z", "shell.execute_reply": "2020-11-24T17:30:34.945087Z" } }, "outputs": [], "source": [ "class EncDec(torch.nn.Module):\n", " def __init__(self, channels=[1, 4, 8], out_nl=True):\n", " super().__init__()\n", " layers = []\n", " channel_pairs = zip(channels[:-1], channels[1:])\n", " for in_channels, out_channels in channel_pairs:\n", " layers.extend([\n", " \n", " torch.nn.Conv1d(\n", " in_channels, out_channels, kernel_size=5, padding=2, bias=True\n", " ),\n", " \n", " ELUCustom(alpha=0.5),\n", " ])\n", " if not out_nl:\n", " layers = layers[:-1]\n", " \n", " self.layers = torch.nn.Sequential(*layers)\n", " \n", " def forward(self, x):\n", " return self.layers(x)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:34.948253Z", "iopub.status.busy": "2020-11-24T17:30:34.947730Z", "iopub.status.idle": "2020-11-24T17:30:34.977738Z", "shell.execute_reply": "2020-11-24T17:30:34.978247Z" } }, "outputs": [ { "data": { "text/plain": [ "EncDec(\n", " (layers): Sequential(\n", " (0): Conv1d(1, 4, kernel_size=(5,), stride=(1,), padding=(2,))\n", " (1): ELUCustom()\n", " (2): Conv1d(4, 8, kernel_size=(5,), stride=(1,), padding=(2,))\n", " (3): ELUCustom()\n", " )\n", ")" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "enc = EncDec(channels=[1, 4, 8], out_nl=True)\n", "enc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Test encoder forward pass:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:34.981297Z", "iopub.status.busy": "2020-11-24T17:30:34.980809Z", "iopub.status.idle": "2020-11-24T17:30:35.011304Z", "shell.execute_reply": "2020-11-24T17:30:35.011815Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "x0.shape=torch.Size([8, 1, 30])\n", "\n", "enc(x0).shape=torch.Size([8, 8, 30])\n", "\n" ] } ], "source": [ "x0, y0 = next(iter(dl_train))\n", "print(f\"{x0.shape=}\\n\")\n", "print(f\"{enc(x0).shape=}\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, our regression layer will use the custom `ArgMinFunction` to solve an optimization problem during the forward pass." ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:35.017391Z", "iopub.status.busy": "2020-11-24T17:30:35.016881Z", "iopub.status.idle": "2020-11-24T17:30:35.047405Z", "shell.execute_reply": "2020-11-24T17:30:35.047970Z" } }, "outputs": [], "source": [ "class PredictorArgMinLayer(torch.nn.Module):\n", " def __init__(self, in_features: int, out_features: int):\n", " super().__init__()\n", " self.prediction_len = in_features - out_features\n", " self.prediction_target_len = out_features\n", " \n", " # We'll train both W and lambda\n", " self.w = torch.nn.Parameter(torch.randn(\n", " self.prediction_target_len, self.prediction_len, requires_grad=True,\n", " ))\n", " self.reg_lambda = torch.nn.Parameter(torch.tensor([1.], requires_grad=True))\n", " \n", " @staticmethod\n", " def obj_fun(w: Tensor, x: Tensor, y: Tensor, reg_lambda: Tensor):\n", " # Objective function performing linear regression\n", " xw = torch.matmul(x, w.T)\n", " loss = torch.mean((xw - y)**2)\n", " reg = reg_lambda * torch.mean(w ** 2)\n", " return torch.sum(loss + reg)\n", " \n", " def forward(self, x):\n", " # Postdiction\n", " # X = | ------ X_e ------ | -- Y_e -- |\n", " x_post = x[..., :self.prediction_len] # X_e\n", " y_post = x[..., self.prediction_len:] # Y_e\n", " w_opt = ArgMinFunction.apply(self.obj_fun, self.w, x_post, y_post, self.reg_lambda)\n", " \n", " # Prediction\n", " # X = | --------- | ------ Z_e ------ |\n", " x_pred = x[..., -self.prediction_len:] # Z_e in the text\n", " \n", " return torch.matmul(x_pred, w_opt.T)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we'll create a model containing the full architecture of encoder, predictor and decoder." ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:35.052623Z", "iopub.status.busy": "2020-11-24T17:30:35.052124Z", "iopub.status.idle": "2020-11-24T17:30:35.081938Z", "shell.execute_reply": "2020-11-24T17:30:35.082503Z" } }, "outputs": [], "source": [ "from typing import List\n", "\n", "class EncPredictorDec(torch.nn.Module):\n", " def __init__(\n", " self, in_features: int, postdiction_length: int,\n", " encoder_channels: List[int], decoder_channels: List[int]=None\n", " ):\n", " super().__init__()\n", " \n", " if decoder_channels is None:\n", " decoder_channels = list(reversed(encoder_channels))\n", " \n", " self.enc = EncDec(encoder_channels, out_nl=True)\n", " self.dec = EncDec(decoder_channels, out_nl=False)\n", " self.pred = PredictorArgMinLayer(in_features, postdiction_length)\n", " self.postdiction_length = postdiction_length\n", " \n", " def forward(self, x: Tensor):\n", " # Calculate embeding\n", " x_emb = self.enc(x)\n", " \n", " # Postdict then predict\n", " y_hat_emb = self.pred(x_emb)\n", " \n", " # Decode back to input space\n", " y_hat = self.dec(y_hat_emb)\n", " return y_hat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Time to train! Notes:\n", "\n", "1. Here we define the \"outer\" optimizer which performs the end-to-end optimization.\n", "2. We'll demonstrate how to employ simple a simple decaying learning rate schedule." ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:35.086191Z", "iopub.status.busy": "2020-11-24T17:30:35.085699Z", "iopub.status.idle": "2020-11-24T17:30:35.117967Z", "shell.execute_reply": "2020-11-24T17:30:35.118481Z" } }, "outputs": [], "source": [ "torch.manual_seed(42)\n", "\n", "model = EncPredictorDec(\n", " in_features=SEG_BASE, postdiction_length=SEG_TARGET,\n", " encoder_channels=[1, 4, 8, 16]\n", ")\n", "\n", "loss_fn = torch.nn.MSELoss()\n", "\n", "# End-to-end optimizer\n", "optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=0.01, eps=1e-6)\n", "\n", "# Decay learning rate each epoch\n", "scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.9)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:35.122905Z", "iopub.status.busy": "2020-11-24T17:30:35.122414Z", "iopub.status.idle": "2020-11-24T17:30:35.152216Z", "shell.execute_reply": "2020-11-24T17:30:35.152777Z" } }, "outputs": [], "source": [ "from tqdm import tqdm\n", "\n", "def run_epoch(model, dl, epoch_idx, max_batches, train=True):\n", " desc = f'Epoch #{epoch_idx:02d}: {\"Training\" if train else \"Evaluating\"} '\n", " losses = []\n", " pbar = tqdm(dl, desc=desc)\n", " for i, (x, y) in enumerate(pbar):\n", " y_pred = model(x)\n", " loss = loss_fn(y, y_pred)\n", " \n", " if train:\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", " \n", " losses.append(loss.item())\n", " pbar.desc = desc + f\"[loss={loss.item():.3f}]\"\n", " if max_batches and i >= max_batches:\n", " break\n", " pbar.desc = desc + f\"avg. loss = {np.mean(losses)}\"\n", " pbar.update()" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "execution": { "iopub.execute_input": "2020-11-24T17:30:35.156240Z", "iopub.status.busy": "2020-11-24T17:30:35.155752Z", "iopub.status.idle": "2020-11-24T17:30:58.566357Z", "shell.execute_reply": "2020-11-24T17:30:58.566947Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Epoch #00: Training [loss=103.402]: 100%|██████████| 27/27 [00:16<00:00, 1.68it/s]\n", "Epoch #00: Evaluating [loss=326759.975]: 100%|██████████| 12/12 [00:07<00:00, 1.65it/s]\n" ] } ], "source": [ "num_epochs = 1\n", "max_batches = None\n", "\n", "for epoch in range(num_epochs):\n", " run_epoch(model, dl_train, epoch, max_batches, train=True)\n", " with torch.no_grad():\n", " run_epoch(model, dl_test, epoch, max_batches, train=False)\n", " scheduler.step()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Thanks!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "**Image credits**\n", "\n", "Some images in this tutorial were taken and/or adapted from:\n", "\n", "- Dr. Roger Grosse, UToronto, cs321\n", "- Fundamentals of Deep Learning, Nikhil Buduma, Oreilly 2017\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.6" }, "toc-autonumbering": false, "toc-showcode": false, "toc-showmarkdowntxt": false }, "nbformat": 4, "nbformat_minor": 4 }