{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Optimization\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"In this tutorial, we will cover:\n",
"\n",
"* Continuous Optimization\n",
"* Reminder: multivariate calculus\n",
"* Gradient Descent\n",
" + Why does GD work?\n",
" + Selecting the learning rate\n",
" + What can go wrong?\n",
"* Stochastic gradient descent\n",
"* Advanced optimizers\n",
"* Working example\n",
"* PyTorch's optimization API - *torch.optim*\n",
"* Learning rate scheduling\n",
"* Projected Gradient Descent (PGD)\n",
" + Use case: adversarial attacks"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Continuous Optimization\n",
"\n",
"Continious optimization problems are fundumental in Computer Science.\n",
"\n",
"May be either unconstrained:\n",
"$$ \\min_x f(x) $$\n",
"$$ f: \\mathbb{R}^d \\rightarrow \\mathbb{R} $$\n",
"Or constrained:\n",
"$$ \\min_x f(x) \\text{ subject to } x \\in \\mathcal{K} $$\n",
"$$ f: \\mathbb{R}^d \\rightarrow \\mathbb{R} \\text{, } \\mathcal{K} \\subseteq \\mathbb{R}^d \\text{ is closed and convex} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Many problems in CS can be written as a continous optimization problems:\n",
"* Linear programs (LPs)\n",
"\n",
"
\n",
"\n",
"* Linear Regression:\n",
"\n",
"$$ \\min_w \\| Xw - y \\|^2 $$\n",
"$$ \\text{where } X \\in \\mathbb{R}^{n \\times d}, y \\in \\mathbb{R}^n $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Hard SVMs:\n",
"\n",
"$$ \\min_{w,b} \\|w\\|^2 $$\n",
"$$ \\text{subject to } y_i (w^T x_i-b) \\geq 1 $$\n",
"\n",
"* **Empirical risk minimization of deep models** \n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Solving Continious Optimization Problems\n",
"\n",
"In some cases, continious optimization problems may be solved analytically:\n",
"* For unconstrained problems, search for stationary points.\n",
"* For constrained problems, try applying Lagrange multipliers or KKT conditions.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Modern deep architectures include millions (sometimes billions) of parameters...\n",
"the loss function is summed over all the dataset (**memory burden**) and the loss surface is often very noisy!\n",
"\n",
"Therefore, efficient iterative optimization algorithms are required!\n",
"\n",
"
\n",
"\n",
"\n",
"\"GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models\""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Reminder: multivariate calculus\n",
"\n",
"We will be mainly interested in functions $f: \\mathbb{R}^d \\rightarrow \\mathbb{R}$.\n",
"\n",
"The generalization of the derivative in the multivariate case is denoted as the **gradient**, which is composed of the **partial derivatives**:\n",
"$$ \\nabla_x f = (\\frac{\\partial f}{\\partial x_1},...,\\frac{\\partial f}{\\partial x_d}) \\in \\mathbb{R}^d $$\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The gradient gives us local information about the direction of the **largest ascent**:\n",
"\n",
"
\n",
"\n",
"If the gradient at some point $x \\in \\mathbb{R}^d$ is $\\vec{0}$ then $x$ is called a **stationary point**.\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The second derivative of a function $f: \\mathbb{R}^d \\rightarrow \\mathbb{R}$ is defined by computing the gradient of each of the partial derivatives.\n",
"\n",
"The resulting matrix is defined as the **Hessian** of $f$:\n",
"$$\n",
"\\nabla^2_x f = \n",
"\\begin{pmatrix}\n",
" \\frac{\\partial^2 f}{\\partial x_1 \\partial x_1} & \\cdots & \\frac{\\partial^2 f}{\\partial x_1 \\partial x_d} \\\\\n",
" \\vdots & \\ddots & \\vdots \\\\\n",
" \\frac{\\partial^2 f}{\\partial x_d \\partial x_1} & \\cdots & \\frac{\\partial^2 f}{\\partial x_d \\partial x_d} \\\\\n",
" \\end{pmatrix}\n",
"\\in \\mathbb{R}^{d \\times d} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Gradient Descent\n",
"\n",
"* Iterative algorithm for solving continious optimization problems.\n",
"* Exploit local information from the current guess to produce the next guess.\n",
"* Idea: move along the anti-gradient direction of the currrent guess:\n",
"\n",
"$$ x_{k+1} = x_k - \\eta \\nabla_x f (x_k) $$\n",
"\n",
"We denote $ \\eta $, which determines the step size as the **learning rate**."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Why does GD work?\n",
"\n",
"By using first order Taylor's approximation around $x_k$:\n",
"$$ f(x_k + \\delta) = f(x_k) + \\nabla_x f(x_k)^T \\delta + o(\\| \\delta\\|)$$\n",
"Substituting $\\delta = - \\eta \\nabla_x f (x_k)$:\n",
"$$ f(x_{k+1}) = f(x_k) - \\eta \\| \\nabla_x f(x_k) \\|^2 + o(\\| \\delta\\|)$$\n",
"If $x_k$ is not a stationary point, then for a small enough $\\eta > 0 $ we have that $f$ strictly decreases.\n",
"This however **does not prove** that GD converges to a local minimum, but rather gives a motivation.\n",
"The convergence analysis of GD is given in: https://courses.cs.washington.edu/courses/cse546/15au/lectures/lecture09_optimization.pdf."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Selecting the learning rate\n",
"\n",
"* Selecting the right learning rate is very important!\n",
"* Selecting too small learning rate would yield to a very slow optimization process (\"**under-damped**\").\n",
"* Selecting too large learning rate would yield to a jumpy process (\"**over-damped**\").\n",
"* Selecting a very large learning rate would cause the optimization process to diverge!\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* What is the optimal learning rate?\n",
"* For quadratic objectives, $\\eta_{opt} = \\frac{1}{\\lambda_{max}}$ where $\\lambda_{max}$ is the largest eigenvalue of the (constant) hessian matrix.\n",
"* For general objectives, computing $\\lambda_{max}$ in every iteration is hard.\n",
"* In practice: perform manual or black-box tuning.\n",
"* Check out [optuna](https://optuna.org/)."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### What can go wrong?\n",
"\n",
"
\n",
"\n",
"* The loss surface of DNNs is highly non-convex!\n",
"* GD depends on initialization. May converge to a **local minimum** rather than a **global minimum**!\n",
"* Another issue with GD is that it considers all the samples together (memory and computation burdens)!"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Stochastic Gradient Descent\n",
"\n",
"* In our case the optimization objective can be decomposed as a sum (mean) of objectives on each sample:\n",
"$$ f(x) = \\frac{1}{n} \\sum_{i=1}^n f_i(x) $$\n",
"* Recall that $n$ is very large.\n",
"* Idea: sample an index, and compute the gradient on a single datum:\n",
"$$ i \\leftarrow Uniform(\\{1,...,n\\}) $$\n",
"$$ x_{k+1} \\leftarrow x_k - \\eta \\nabla f_i(x_k)$$\n",
"* In expectation the gradient is exact! However, the variance is very high!\n",
"* Optimization process becomes very noisy!"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Idea: instead of sampling a single datum, sample a **batch(mini-batch)** of samples.\n",
"* In practice: shuffle the dataset and split it into **mini-batches**. Each iteration over the whole dataset is called an **epoch**.\n",
"\n",
"\n",
"
\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Advanced optimizers\n",
"\n",
"* Heavy ball momentum\n",
" + Idea: accumulate velocity from prior iterations!\n",
" + Models the physics of a ball that is rolling downhill.\n",
"
\n",
" \n",
"
\n",
" \n",
" + Momentum is modeled by an exponential moving average of the gradients in the prior steps:\n",
" \n",
" $$ v_{k+1} \\leftarrow \\gamma v_k + (1-\\gamma) g_k $$\n",
" $$ x_{k+1} \\leftarrow x_k - \\eta v_{k+1} $$\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* AdaGrad\n",
" + Stands for Adaptive Gradient.\n",
" + Idea: the Hessian matrix may be very unbalanced, so use different effective learning rate for each parameter.\n",
"
\n",
" \n",
"
\n",
" \n",
" + Mathematically: \n",
" $$ G_{k+1} \\leftarrow G_k + g_k \\cdot g_k $$\n",
" $$ x_{k+1} \\leftarrow x_k - \\frac{\\eta}{\\sqrt{G_{k+1} + \\epsilon}} \\cdot g_k $$\n",
"\n",
" + Note that in the above formulation $\\cdot$ multiplication and the division is done **elementwise**.\n",
" + $\\epsilon$ is added to the denominator for numerical stability.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Rmsprop\n",
" + The problem of Adagrad is that the denominator keeps growing, and hence becomes very slow.\n",
" + The solution is to use an EMA of the squared gradients instead:\n",
"\n",
" $$ v_{k+1} \\leftarrow \\beta v_k + (1-\\beta) g_k \\cdot g_k $$\n",
" $$ x_{k+1} \\leftarrow x_k - \\frac{\\eta}{\\sqrt{v_{k+1} + \\epsilon}} \\cdot g_k $$\n",
"\n",
"
\n",
" \n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Adam\n",
" + Stands for Adaptive Moment Estimation.\n",
" + Essentially a combination of momentum and rmsprop:\n",
" $$ m_{k+1} \\leftarrow \\beta_1 m_k + (1-\\beta_1) g_k $$\n",
" $$ v_{k+1} \\leftarrow \\beta_2 v_k + (1-\\beta_2) g_k \\cdot g_k $$\n",
" $$ \\hat{m}_{k+1} \\leftarrow \\frac{m_{k+1}}{1-\\beta_1^{k+1}}, \\quad \\hat{v}_{k+1} \\leftarrow \\frac{v_{k+1}}{1-\\beta_2^{k+1}} $$\n",
" $$ x_{k+1} \\leftarrow x_k - \\frac{\\eta}{\\sqrt{\\hat{v}_{k+1} + \\epsilon}} \\cdot \\hat{m}_{k+1} $$\n",
" + The most common optimizer today."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Which optimizer to use?\n",
" + Adam would be a good place to start.\n",
" + However, **for some tasks it is better to use other optimizers**.\n",
" + For instance, simple SGD with momentum works the best for optimizing ResNet!"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Working example\n",
"\n",
"Let's demonstrate SGD for training a simple MLP architecture for performing hand-written digit recognition."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# Imports\n",
"import torch\n",
"from torchvision import datasets, transforms\n",
"import torch.nn as nn\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Define an MLP architecture\n",
"class Net(nn.Module):\n",
" \n",
" def __init__(self):\n",
" super(Net, self).__init__()\n",
" self.in_dim = 784\n",
" self.hidden_dim = 120\n",
" self.out_dim = 10\n",
"\n",
" self.flatten = nn.Flatten() # (B,H,W) -> (B,D)\n",
" self.linear = nn.Linear(self.in_dim, self.hidden_dim)\n",
" self.activation = nn.ReLU()\n",
" self.classifier = nn.Linear(self.hidden_dim, self.out_dim)\n",
"\n",
" def forward(self, x):\n",
" x = self.flatten(x)\n",
" x = self.linear(x)\n",
" x = self.activation(x)\n",
" x = self.classifier(x)\n",
" return x\n",
" \n",
"model = Net() # Instantiate model"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Define the training dataset\n",
"transform = transforms.Compose([\n",
" transforms.ToTensor(), # Convert to tensor\n",
" transforms.Normalize((0.1307,), (0.3081,)) # Subtract from values 0.13 then divide by 0.31\n",
" ])\n",
"\n",
"dataset = datasets.MNIST('./data', train=True, download=True, transform=transform) # MNIST train set"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# Define dataloader \n",
"batch_size = 64\n",
"loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True) # Different order in each epoch"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"On each batch, optimization can be summarized as follows:\n",
"* Loss computation on the current batch.\n",
"* Loss gradient computation w.r.t each of the model params.\n",
"* perfrom SGD step."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": true,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# Actual Training loop\n",
"num_epochs = 1\n",
"lr = 1e-1\n",
"loss_fn = nn.CrossEntropyLoss()\n",
"losses = [] # For plotting\n",
"\n",
"model.train() # Training mode\n",
"for epoch in range(num_epochs):\n",
" for batch_idx, (x, y) in enumerate(loader):\n",
"\n",
" # 1. Compute loss \n",
" logits = model(x)\n",
" loss = loss_fn(logits, y)\n",
"\n",
" # 2. Magically compute gradient\n",
" grad = torch.autograd.grad(loss, model.parameters())\n",
"\n",
" # 3. Perform optimization step\n",
" for param, g in zip(model.parameters(), grad):\n",
" param.grad = g\n",
" param.data -= lr * param.grad\n",
"\n",
" losses.append(loss.item())"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Lets plot the loss over time!"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAABibUlEQVR4nO3deXwU9fkH8M9eOUnCHRJulEtAjoAKcikKBevRUo+2itXaFhVQ+akVrfeBbbW1tApVUap4YAseKCqoXAJyh1sOORIh4SYJuXd3fn8ks/nO7MzszGZ3J2E/79eLl2Z3dncyu9l55vk+3+frkCRJAhEREZFNnHbvABEREcU3BiNERERkKwYjREREZCsGI0RERGQrBiNERERkKwYjREREZCsGI0RERGQrBiNERERkK7fdO2CG3+/HkSNHkJaWBofDYffuEBERkQmSJKGkpATZ2dlwOvXzH40iGDly5Ajat29v924QERFRGPLz89GuXTvd+xtFMJKWlgag5pdJT0+3eW+IiIjIjOLiYrRv3z5wHtfTKIIReWgmPT2dwQgREVEjE6rEggWsREREZCsGI0RERGQrBiNERERkKwYjREREZCsGI0RERGQrBiNERERkKwYjREREZCsGI0RERGQrBiNERERkKwYjREREZCsGI0RERGQrBiNERERkq0axUF60zN/4I7YdLsJPerfBJV1a2L07REREcSmuMyPL9xzHnNUHsfNIsd27QkREFLfiOhhJdNf8+pVev817QkREFL/iOxjxyMGIz+Y9ISIiil/xHYy4XQCYGSEiIrJTnAcjNb9+RTUzI0RERHaJ82CEmREiIiK7xXcwIteMVDMYISIiskt8ByNuFrASERHZLc6DEQ7TEBER2S3OgxH2GSEiIrJbfAcjgZoRDtMQERHZJb6DEQ7TEBER2S7OgxEO0xAREdmNwQg4m4aIiMhO8R2MeGqHadhnhIiIyDbxHYwwM0JERGS7uA5GPK6aX9/rl2zeEyIiovgV18GI2+kAAPh8DEaIiIjsEtfBiKs2GKn2s2aEiIjILnEdjMjDND4O0xAREdkmroMROTPCmhEiIiL7xHUwIteMSBKzI0RERHaJ72DE5Qj8v5d1I0RERLaI72DEWffrMzNCRERkj7gORuSaEQCo5vReIiIiW8R1MOIWghFmRoiIiOwR18GI0+mAHI+wZoSIiMgecR2MAHV1I14O0xAREdki7oMRuW6EwzRERET2iPtgRJ7ey8ZnRERE9mAwIndh9bFmhIiIyA5xH4y45JoRZkaIiIhsEffBiMfFmhEiIiI7xX0wIhewVnOYhoiIyBZxH4y4OZuGiIjIVgxGXKwZISIishODkcBsGgYjREREdoj7YESuGWE7eCIiInvEfTAiD9OwZoSIiMgeDEac7MBKRERkp7gPRlysGSEiIrJV3AcjHhdrRoiIiOwU98GI3A6eNSNERET2iPtghFN7iYiI7MVghAWsREREtrIUjEyfPh2DBg1CWloaWrdujeuuuw67d+8O+bjly5cjJycHSUlJ6NKlC2bNmhX2DkeaO7BQHmtGiIiI7GApGFm+fDnuvvtufPfdd1iyZAm8Xi9Gjx6N0tJS3cccOHAA48aNw7Bhw7B582Y8/PDDmDJlCubPn1/vnY8EuWakmsM0REREtnBb2fiLL75Q/Pzmm2+idevW2LhxI4YPH675mFmzZqFDhw546aWXAAA9e/bEhg0b8MILL2D8+PHh7XUEebhQHhERka3qVTNSVFQEAGjevLnuNmvWrMHo0aMVt40ZMwYbNmxAdXV1fV4+IlysGSEiIrKVpcyISJIkTJ06FUOHDkXv3r11tyssLERmZqbitszMTHi9Xpw4cQJZWVlBj6msrERlZWXg5+Li4nB3MyS5ZsTrY80IERGRHcLOjEyaNAlbt27Fe++9F3Jbh8Oh+FmSJM3bZdOnT0dGRkbgX/v27cPdzZCYGSEiIrJXWMHI5MmT8cknn2Dp0qVo166d4bZt2rRBYWGh4rZjx47B7XajRYsWmo+ZNm0aioqKAv/y8/PD2U1T3Gx6RkREZCtLwzSSJGHy5Mn48MMPsWzZMnTu3DnkYwYPHoyFCxcqblu8eDEGDhwIj8ej+ZjExEQkJiZa2bWwyX1Gqjm1l4iIyBaWMiN333035s6di3fffRdpaWkoLCxEYWEhysvLA9tMmzYNEyZMCPw8ceJEHDp0CFOnTsWuXbvwxhtvYPbs2bj//vsj91vUg0vuM8KpvURERLawFIzMnDkTRUVFGDlyJLKysgL/5s2bF9imoKAAeXl5gZ87d+6MRYsWYdmyZejXrx+efvppzJgxo0FM6wUAT+0wDWtGiIiI7GF5mCaUOXPmBN02YsQIbNq0ycpLxUxdASuHaYiIiOzAtWnY9IyIiMhWDEZctcM0rBkhIiKyBYMR9hkhIiKyVdwHI2x6RkREZK+4D0Y88tReFrASERHZIu6DEZeTNSNERER2ivtghDUjRERE9mIw4mIwQkREZKe4D0ZcTtaMEBER2SnugxF51d5q1owQERHZgsGIix1YiYiI7MRghAWsREREtor7YCTQ9MzHmhEiIiI7xH0w4qldm4bDNERERPaI+2CE7eCJiIjsFffBiJvDNERERLZiMFI7TMPMCBERkT0YjDg5tZeIiMhOcR+MyDUjbHpGRERkj7gPRtxsB09ERGQrBiOsGSEiIrIVg5HAbBoGI0RERHaI+2DExQJWIiIiW8V9MCIvlOdlzQgREZEtGIw4aw6BXwL8zI4QERHFXNwHI/IwDcAiViIiIjvEfTDicdUFI6wbISIiir24D0bEzEg160aIiIhiLu6DEblmBAB8nN5LREQUc3EfjLicDjhqkyOsGSEiIoq9uA9GAC6WR0REZCcGIxAXy2PNCBERUawxGAHgqa0bYWaEiIgo9hiMAHAFurAyGCEiIoo1BiMQFsvj1F4iIqKYYzCCuum9J0qqbN4TIiKi+MNgBEBhcQUA4ObZa23eEyIiovjDYISIiIhsxWCEiIiIbMVghIiIiGzFYISIiIhsxWCEiIiIbMVghIiIiGzFYISIiIhsxWCEiIiIbMVghIiIiGzFYATA/DsHAwBSElw27wkREVH8YTACoHVakt27QEREFLcYjABw1CzaC59fsndHiIiI4hCDEQAuZ0004pcYjBAREcUagxEALoccjNi8I0RERHGIwQgAR20wwmEaIiKi2GMwgrphGgDwMyAhIiKKKQYjqBumAQAf60aIiIhiisEIAKdwFFjESkREFFsMRgA4HeIwjY07QkREFIcYjEBZM8JhGiIiothiMAJlZoQzaoiIiGKLwQiUmRGJmREiIqKYYjACQIhFmBkhIiKKMQYjqGl6FlifhpkRIiKimGIwUkvuNcJYhIiIKLYsByMrVqzA1VdfjezsbDgcDnz00UeG2y9btqw286D89/3334e7z1HhZEt4IiIiW7itPqC0tBR9+/bFbbfdhvHjx5t+3O7du5Genh74uVWrVlZfOqqcTgA+BiNERESxZjkYGTt2LMaOHWv5hVq3bo2mTZtaflys1K3cy2CEiIgolmJWM9K/f39kZWVh1KhRWLp0qeG2lZWVKC4uVvyLNqdTDkai/lJEREQkiHowkpWVhVdffRXz58/HggUL0L17d4waNQorVqzQfcz06dORkZER+Ne+ffto7yZrRoiIiGxieZjGqu7du6N79+6BnwcPHoz8/Hy88MILGD58uOZjpk2bhqlTpwZ+Li4ujnpA4nJymIaIiMgOtkztveSSS7B3717d+xMTE5Genq74F23MjBAREdnDlmBk8+bNyMrKsuOldblqjwQzI0RERLFleZjm7Nmz2LdvX+DnAwcOIDc3F82bN0eHDh0wbdo0HD58GG+99RYA4KWXXkKnTp3Qq1cvVFVVYe7cuZg/fz7mz58fud8iAuTMiN9v844QERHFGcvByIYNG3DZZZcFfpZrO2699VbMmTMHBQUFyMvLC9xfVVWF+++/H4cPH0ZycjJ69eqFzz77DOPGjYvA7kdOYJiGmREiIqKYckiNYJna4uJiZGRkoKioKGr1I8P/shR5p8qw4K4hGNChWVReg4iIKJ6YPX9zbZpa8sq9fhawEhERxRSDkVpy0zPOpiEiIootBiO1XKwZISIisgWDkVpy0zPGIkRERLHFYKSWg03PiIiIbMFgpJbc9Ky4otreHSEiIoozDEZqbT9cszLwpHc327wnRERE8YXBCBEREdmKwQgRERHZisFIrZsGtQcApCdZ7pBPRERE9cBgpNYNtcFIRorH5j0hIiKKLwxGarlr+4x4fZzaS0REFEsMRmq5nTWHwss+I0RERDHFYKSW2yVnRvw27wkREVF8YTBSS24Hz8wIERFRbDEYqeWpHaZhO3giIqLYYjBSy+ViASsREZEdGIzU8gSGaVgzQkREFEsMRmrJNSN+CfBzqIaIiChmGIzUkqf2AixiJSIiiiUGI7Xkqb0Ai1iJiIhiicFILXmYBgCqWTdCREQUMwxGanlcdYfCxxk1REREMcNgpJaQGGHNCBERUQwxGKnlcDjqFsvjMA0REVHMMBgRuNn4jIiIKOYYjAi4ci8REVHsMRgRnK30AgAe/2SHzXtCREQUPxiMaFix57jdu0BERBQ3GIwQERGRrRiMEBERka0YjBAREZGtGIwQERGRrRiMEBERka0YjBAREZGtGIwI7hp5HgCgRWqCzXtCREQUPxiMCK66MAtAXVt4IiIiij4GIwK5HbyP7eCJiIhihsGIwBVYtZfBCBERUawwGBG4a4MRH1ftJSIiihkGIwJmRoiIiGKPwYhALlxlzQgREVHsMBgR1GVG/DbvCRERUfxgMCKQZ9P4JcDP7AgREVFMMBgRyJkRAPBJDEaIiIhigcGIwC0GI8yMEBERxQSDEYGYGeGMGiIiothgMCJQZEbYa4SIiCgmGIwIlJkRzqghIiKKBQYjAofDEQhIWDNCREQUGwxGVFwOdmElIiKKJQYjKsyMEBERxRaDERU316chIiKKKQYjKq7A+jQsYCUiIooFBiMqzIwQERHFFoMRlcBieewzQkREFBMMRlTqFstjMEJERBQLDEZUXBymISIiiikGIypuTu0lIiKKKQYjKgnumkPyzKc7saug2Oa9ISIiOvcxGFFJSXABALb8WISx/1hp894QERGd+xiMqKQmuu3eBSIiorhiORhZsWIFrr76amRnZ8PhcOCjjz4K+Zjly5cjJycHSUlJ6NKlC2bNmhXOvsZEakLoYOSD9fkY+del+OH42RjsERER0bnNcjBSWlqKvn374l//+pep7Q8cOIBx48Zh2LBh2Lx5Mx5++GFMmTIF8+fPt7yzsZCS6Aq5zYPzt+LgyTL86cPtMdgjIiKic5vlMYmxY8di7NixprefNWsWOnTogJdeegkA0LNnT2zYsAEvvPACxo8fb/Xlo85MZkRW4fVFcU+IiIjiQ9RrRtasWYPRo0crbhszZgw2bNiA6upqzcdUVlaiuLhY8S9W1JmRimoGHERERNEU9WCksLAQmZmZitsyMzPh9Xpx4sQJzcdMnz4dGRkZgX/t27eP9m4GqDMjxeXaARMRERFFRkxm0zgcDsXPUm2rdfXtsmnTpqGoqCjwLz8/P+r7KJOn9sqKDIIR7b0nIiIiK6I+j7VNmzYoLCxU3Hbs2DG43W60aNFC8zGJiYlITEyM9q5pUk/tLa3iMA0REVE0RT0zMnjwYCxZskRx2+LFizFw4EB4PJ5ov7xlmenKIKisyqu7rV5mh4iIiMyzHIycPXsWubm5yM3NBVAzdTc3Nxd5eXkAaoZYJkyYENh+4sSJOHToEKZOnYpdu3bhjTfewOzZs3H//fdH5jeIsMFdWip+LmdmhIiIKKosByMbNmxA//790b9/fwDA1KlT0b9/fzz22GMAgIKCgkBgAgCdO3fGokWLsGzZMvTr1w9PP/00ZsyY0SCn9QJAcoILL1zfN/BzGYMRIiKiqLJcMzJy5MhAAaqWOXPmBN02YsQIbNq0yepL2eYXOe2waFsBvvn+mOEwDREREdUf16bRkVw7q4aZESIiouhiMKIjxRM6GGH5KhERUf0xGNEh9xthASsREVF0MRjRkVLbb4TDNERERNHFYERH3TANC1iJiIiiicGIDhawEhERxQaDER0pCaGHadiAlYiIqP4YjOhITawtYK3mMA0REVE0MRjRkVxbM1JayWEaIiKiaGIwokMepjGa2utgpxEiIqJ6YzCiI1DAymEaIiKiqGIwooNNz4iIiGKDwYiO1NphGtaMEBERRReDER3yME15tQ9+f90qxeL/ExERUf0xGNEhD9MAQIW3Ljvik4RghPWrRERE9cZgRIc8tRdQNj7zMTNCREQUUQxGdDidjkBAUlbJYISIiChaGIwYSNGY3usVghGO0hAREdUfgxEDWovlsYCViIgoshiMGNDqNSJmRhiWEBER1R+DEQNaK/f6JU7zJSIiiiQGIwYCNSNVdTUjYgGrYpovERERhYXBiIEUjZoRHxugERERRRSDEQPJGsM0zIwQERFFFoMRA6mBAlbtqb0+f8x3iYiI6JzDYMSAPLW3VKeA1ednNEJERFRfDEYMaE3trRbSIZHoxrrx0Gl8uvVIvZ+HiIiosXLbvQMNWd3U3rphmopqMUtS/9cYP3M1AOC8Vk3QMyu9/k9IRETUyDAzYiCpdm2aimo/Dp4oxVMLd+LgibLA/ZFcp+bQybLQGxEREZ2DmBkxIC+UV17twy9mrcGJs5WK+70RrGCVODOHiIjiFDMjBpITag5PRbUvKBABgOoIZkYaeihy4EQpnvhkB46cKbd7V4iI6BzDzIiBJLc8TOPTvL86gpkRfwPPjFw/azVOnK3C5rzT+HjSULt3h4iIziHMjBhISqgbptFS5Q0ORsqqvDhdWmX5tRp4LIITZ2t+py0/Ftm8J0REdK5hMGKgLjOinQHRyoz0f2oJ+j+9BEVl1ZZeq6FnRoiIiKKFwYiBZI0+I6JqnxRUeFpZmy3ZUcAMAhERkRkMRgzIs2kqvdrBCFATkGhxOhyWXouJESIiilcMRgwkeWoOj15mBNAvYrUWinCYhoiI4heDEQNyZqTUIBh5a80h3DJ7Lc5WeuEXpvo6naHDEXGIh7GItmqfH/mntBvCbT9chI2HTsV4j4iIKNIYjBiQZ9MY+fMX32Pl3hN4bcV+xYq+ZjIjYpsSZka0TZi9DsP+shQr9hxX3C5JEn76z28xfuYanCmzPnuJiIgaDgYjBuTZNGacLquyHFCI2zMU0bZm/0kAwDtrDyluF1vxnwxjKjURETUcDEYMeFzmKz+8fklxgnSYKGBVBCPMjBhSl+aIR4uHjoiocWMwYsDhcOBn/dsG3Z6S4ELz1ATFbV6fXzlMYyKOEU+iPKEaUwdrPF5EROcOBiMh/OUXF6Jr6yaK2zwuJxJcykPn9UnKAlaLmZEILnNzTlIfHvHYWZxFTUREDQyDkRA8Lieu7ZetuM3tdMDjVp4BvX4JPuEEOW99fsjnlljAapr6+DCrRER07mAwYkJKgnI9QafTEZwZ8fsVNSPvrcsLucIta0bMU2eOJJb8EhGdMxiMmJCimuLrdjrgUQUj1T5lASsAFJUbr08jbs5TqzF1sMZhLSKicweDEROSVcGIy+lAglt56Hz+4GAkVLJDPMH6bT67Vnn9eObTnUH9POxQVFaNz7cVKNrwBw/TMBohIjpXMBgxQT1M49LMjPiDgpFQdSCxyIwUFJXjplfX4IvthYbbvbP2EF7/9gAmvLEuSnti3q1vrsOd72zCX7/YHbhNfSiZGSEiOncwGDFBPUxTE4yoClh9ygJWM8RgRR3IRMrjH+/Ad/tPYeLcjYbb5em0XLdDbv4ZAMD8TT8GbjPOjDAyISJqzBiMmBA0TONwIFHVnVVrmCYU5dTe6JxQTzXi7qRen3h8lPdxlIaI6NzBYMSERFV9iMvpCCyiJ6v2Bw/ThK4Zqft/b5QyI425B0e1v67tanABK6MRIqJzBYMRE9T1IS6nA0me4KZnQcFIiOEDfwwKWB2mluxrmAwzI+L/My4hImrUGIyYoO4p4nY6goZu1GvTAKGLLMX71WuvkDJbpM6EsHstEdG5g8GICR2ap6BVWmLgZ6czuGak2ucPKmA9eKLU8HnFbIjPH6VopPEmRhSMakY4ZENE1LgxGDHB6XTgv38YHPhZKzNS6fUFDbXcOy8XFdU+6BHPoTO+2ReZnVUxG4s09PO50UJ5DX3fiYjIGIMRk1zOutO60xFcwFpZ7ceSXUeDHne6TH82i/qKfs/RknruZbDGXMAqCu4zEv2ZSEREFBsMRkwSi1jdruAC1mMllfj38v1BjzOqZ1CfRKPRa6QxF7CKgvqMiP/fCGORimofXluxH/uORSYAnbc+D0t2BgfDRESNAYMRk8QmZ1qZET1eg8pUdeyR5HHhxNlKvP3dIZRU1K1r4/dLeHdtHvaGkTk5VzIj6mPlNyhubQxeWboPzy7ahSv+tqLez3XwRCn+OH8bfvfWhgjsGRFR7DEYMckj9BrxSxISTQYjVV79YESrd8atb6zDox9tx8Mfbg/c/v76fDz84TZc+ff6n7gaK6OaETuDEb9fwsItR/DjaWsdbDccOh2xfThZWhmx5yIisgODEZPE6b0+v2Q6M1JpFIyofvb7Jew4UgwA+GJ7QeD27/afNL+jKpHIjJwxqHuJleBhGkn4f/v8d2M+Jr+3GUP/vNTS4yIbQNW9yVxAkIgaIwYjJok1I34JEQlGgntnaG9XbjAjJ5T61oy8vy4P/Z5aUq/niASjhfLsPAGv2hdeoBjJXRYDzmitcUREFE0MRkwSZ9P4/VLQ1F49lV79QELdWkTvRFJeVY9gxGQsondCf2jBtrBfO1xa+6zu4SI1kKZn4b50RIMR4f+jtawAEVE0hRWMvPLKK+jcuTOSkpKQk5ODlStX6m67bNkyOByOoH/ff/992DttN58koVd2uqltK71+xYmzyuvHth+LUFRebdhVVFSfzIiosaTw1e33AePMSLRa6UdTJIdpnEL0xmCEiBojy8HIvHnzcO+99+KRRx7B5s2bMWzYMIwdOxZ5eXmGj9u9ezcKCgoC/7p27Rr2TtvN75fQNCUBbdKTQm777tqaYY6vdh7Fn7/4Ht3+9Dmu/te3GPePlYa9M0T1yYyIqn3mTlTlVT7MWv4Dfjh+NiKvq+X1lfvxq9e+0/zdPM7g1EhwAWvDqBkJVyT3WQxGfCbfYyKihsRt9QF/+9vf8Nvf/hZ33HEHAOCll17Cl19+iZkzZ2L69Om6j2vdujWaNm0a9o42JPKQQaIndCwn9364QzXt8vCZctM1I0ZdXENxKK6a/UjQiT/Fl/7rl7vxxqoDeP5z7eyVJEmK5w3HM5/tAgB8sCEftw7ppLjP7XICUP7O8rE5XVqFl77agwvbNRXus+8EHG62KZL7LL4V3mgtK0BEFEWWMiNVVVXYuHEjRo8erbh99OjRWL16teFj+/fvj6ysLIwaNQpLlxrPPKisrERxcbHiX0Mitw7RGk6wwqjpmXhX/QpY65jNjISavRPJIskyjcyIVpwjH6snF+7Af9Ycwv/9d0vgvkYy+qQQyX32KdY4aoQHg4jinqWz6YkTJ+Dz+ZCZmam4PTMzE4WFhZqPycrKwquvvor58+djwYIF6N69O0aNGoUVK/R7ZkyfPh0ZGRmBf+3bt7eym1EnXw2rV/O1KqiRV5RrRqpVDdj0rupDXbWri0ljQX7J7wuDG7/ZmhkJ93ER3Gfx/WDNCBE1RpaHaQAEpeiN0vbdu3dH9+7dAz8PHjwY+fn5eOGFFzB8+HDNx0ybNg1Tp04N/FxcXNygAhL56jPBXb9gJKjpmc6JxGsyo1FQVI6UBDdmf3sA/Ts0xfHiSuw7Vlf3IT7Pwi1H8OTCHZh5cw4GdWquuFIPdXUdyZEAs6M9RgFHY8yMRDJmED838ntc5fXjX0v3YUS3Vsjp2CxyL0ZEFAWWgpGWLVvC5XIFZUGOHTsWlC0xcskll2Du3Lm69ycmJiIxMdHKrsWUL0qZEb2Mg5nU+7HiCgye/o3hNmJmZPJ7mwEAv52zHlufGGNqP8zeX19a8YnRS9raDj7Ml47kPoufD7lm5K01BzHj672Y8fVeHHz+qoi9FhFRNFg6myYkJCAnJwdLliibYC1ZsgRDhgwx/TybN29GVlaWlZduUPwRyowEdRXVOT+ZOflvzj8Tchv1MA0AVGncFir4saMuQT4GTo1Uir2xSHgvHsl99mtktbSGs4iIGirLwzRTp07FLbfcgoEDB2Lw4MF49dVXkZeXh4kTJwKoGWI5fPgw3nrrLQA1s206deqEXr16oaqqCnPnzsX8+fMxf/78yP4mMSR/+Uc6GJlSm60AlBfcfkVhq/aQmNZJWk2rnkArrgg9TBPBmSAmt5OHtIyKWxuTSO6zX6NmpDEeEyKKX5aDkRtvvBEnT57EU089hYKCAvTu3RuLFi1Cx44dAQAFBQWKniNVVVW4//77cfjwYSQnJ6NXr1747LPPMG7cuMj9FjEWqBkRhmkcDutXu+rtT5ZqrwEjnlh8fgnFFdUorfSiffOUwO0arTmCaC7aJwW/Rqhgw84CVq1gxNbMSAM452vNpmkI+0VEZFZYBax33XUX7rrrLs375syZo/j5wQcfxIMPPhjOyzRY8olbzIykJbpRXOG19Dyfbi0IvRGU2QuvX8KAp2uGydY/cgVapdXU1pgpBNXOjNTcpjihhTiTRTQzYrGAVWutneDhrvr3QYm2iNaMaGRGOMWXiBoTrk0TBvmLXuwzkp7ssfw8760z7lqr5WxlXcCzX+iQaubk69WoD5FPWcqra+PniXoBq8bv4jfIjIjn3TmrDuCS6V9HtXtsONQBXPRm09S8eXZkr4iIwsVgJAxamZFhXVtF9DV8fgllVd6gK9y8U2WB/09JqEtsmakZ0SpW1cyMhJi7a3aqsZ5wemzUZUaMn++JhTtxtLgSTy7ciZKK6nB30TQzv8rb3x1C3ycXY3PeaeFxdQ98feV+/Hi6TOuhpihn08jDNAxGiKjxYDASBvm7/45hnQEAV/fNxiNX9cSUUZFdb+fFxXuC6jzyhWCkWggazNSMaAURklTTYn3B5sOB20IWsNbzRCc+v9awi+HUXq3ZNBrbr9hzHH2eWKzos6L2ce5hDJn+NbYfLgqxx/Xz6EfbUVLpxf06XWOf+WwXfjFzTdjPr64pEv9LRNQYMBixYFyfNgDqgpDzWjXBjifHYMZN/dAk0Y2pV3ZDRhjDNXqW7T4WFIz83wd1JzTxPq2TuprW1F4AeFc1XBTqPKY+0f3z67249uVVOHm2MuQ+AKG7hGrdKxlkRoyCo3fX6g+F3fN+Lo4UVWDK+5t1tzHy+bYCfLFDu/OwFpcQMar3uLC4Iqx9OFpcgSNn6h5bVzMS1tMREdkirALWePXSjf0xcUQxemdnBG5LTVQeQnV6/N07LsavXl8b1utlJHtQ6VO2ghdP5GJwYSZbobc2jTpICbXYmvq1XlyyBwDw6MfbMaBDMxQWVeCRq3rq1rGEk1kxUzOiNTTRJNEV8rkrwlwV+c53NlnaXhxKq0926Z9f78WuwmL87YZ+uPi5rxX3yUNsHKYhosaEmRELEtxOXNiuKZxmxkQA3Dq4I4ac3xJt0pMCt/3rV/1Nv15GsgeV1fqBgRhEmClY1Asy1A8N1e5d76p70bZCPPPZLrz+7QEcOFFqsB/CMI3GoTTKfujVjPx3Q35glpGoSVJk4u0PNuTjw80/1us53K7IBCMvLtmDRdsKAytCi+ShOPYZIaLGhJmRKHrs6l4AlIWjgzo1N/34jGSPZtGprMrrxz++2guHA+iVnR7y+TYcPI2fXpgddLv6KjpUZkQepimr8ioKakVlVT5IkoTjZyvROi1JcZ8vjALYuj4jweHIzGU/6HYcbZIYethMQk1g53Y6NJ//TFkVHvzfVgDA2N5ZSPKEzrZocYmZEZ1DLEkSDp8pR9umySFnSGkFHIGaEcYiRNSIMDMSYeI5QK4RqBRW3U1ymz+RNUlyG2ZGCooq8Pev9uBvS/agxESPkzmrD2rO2lDXcISqGZFPgj9/ZTV+8tJKzW1KK734+1d7cdGzX2Pud4d0X8/sBbxRZsSo9bnbRBbrZGkVcp5egvv/u1Xz/lJhGKc+q+Kayag9t2gXhv55KT7bpt2DRpye7XYG//lWyx1Yz8EC1re/O4TfvbUBld7IrGJNRA0Hg5EYqBQKTTXOH7p8fskwM3JCKBjV7K6qQWv4ROxdYna/AOMgoLTKixlf7wUA/Omj7Yr7FLM/NKIRrSt+g8k0hqpNLDFc5fWjuMKL+Zu0h2HM1l/sLizBqBeX4dOtRzTvd4WoGfH5Jby28gAA4J3vtAtvK4T3WWudRrlm5FycTfPoR9uxZOdRfLjpcOiNG6DyKh9umb0Wc1YdUNxeVuU1XfxNdK5iMBJpGucA8Wpa62pWT6XXr8iqqJ0pq+ujUWHyalHr9YvLrfXjMFOfUlpZtz/qAMKr6GlS8//bDxfhhllrsPHQKcP1cszMGlK8VoTHK4xO8vfNy8UPx0sx6V3t2Tlypuzj3MMoKAqePXO6rG45gAt0ht3KQxTbxkPNSLnqbyL/VBle+HK3IjhviN5Zewgr957AEwt3Km7v99QS5DzzFc4I7//3hcWYOi8XH+c2zsCLyCrWjERYqFOAmVikW2YT7Dl6FtU+48yIePLakm+uV0aCO/hkbmaIR2RmCKDUINsi1ozIJ/db31iHk6VVGB+q34bFzEh9hlVk4nnd6HcvqzI+jnIwcs/7uUH3ORzKDJpeMFEhnIi1ZkfJx/NcC0bE7JS6ZuemV7/D4TPl2JR3Gu/+7pKo7sdXO4+iWWoCcjo2s/xYvQyknNXcfrgYQ7u2RFFZdWD4c8Hmw7i2X9vwd5iokWBmJMbMZEY6NE8FAFR5fSgzuBI+cbYuGNEbYjDz+uEO0xh5deX+wP9LEvDMp3VXg2KBrPxceosEqlldcUarBb4R9ZDMMVX/D6PgJlTBqcugZiTZ41JkwfSG3SpCbFO3aq/hrjQ64t9Bkkf5GT58phwAsPqHk1F7/UqvD//b+CPueGsDxs9cXe+ZVUaON/AMD1E0MBiJMTOzgts3TwZQc+VrNIQSzjiz1hXzfoNpuFrMDNPsP658zte/rRsn1+oYaua41CyAZ3Ina1nNjIjZif9t/BEXPfc1nv1sV+A2o4xDqF0zCkYS3E7Fa+s1qKsQCpq1smYNuQNrpdeHaQu24pMt2jU1RsTs3exvD8S8j8rDC7YrOujeN2+LwdbaQg0xSrV51VCz2YjORQxGIszoS7JJotvUgnbtmqUAAL75/hgeqJ1SOrxb8No3ZrMJIq2T1PESa0FNfb8rFTUjtcfLo1WNqbL2wClFLYoZeid1PWLm4YlPdgCAosuq0Uk+1OnRZfDeOx0ORTCilxkpD5EZkX9f8XPYUBqgrdp3Au+ty8eU9zabLriWiesMbT9cjFX7opcFOXCiFCP/uhTvC52JzWYerdJ6b8Q6J3H9K6JzGT/pMZSSEHpa71PX9lI0SZNlJHuQqnq8WMBqViRqKMJdEVb+4hW/bOUaDDPByE2vfodtFteRsVrAKg4HGGUetIQ66RtN7XVAOQVcr1uusmZEKxiR+4wEZ58iZfW+E7jrnY04VmKthb3YgXbDoVOWHlusWvTQTPv8cKc3P/bxdhw8WYaHFmwL6/FWaL034vtqNMy48dAprNx7PCr7RRRrDEYiTOvrL622ZfwIjeyG2oTBnTSvhpI9TnwwcXB9d8/wxDSgQ1NTzxHul7xWcaUcHIndSSNJDL5W7j2OhSGGCOTMw96jJZpX71YLQ8VjZZQZAZRDRJUmMiNa28g9OMRzWLjBo55fvb4Wi7YV4ulPd4XeWCDuhtVAulhVZO0J8Xn5yxffI+eZJYF6Eius1lCF48PNP+KP/9uKIc9/E7hNPj7i36hf0v57kyQJ42euwS2z11nObNK572ylt9H1GmIwEmFa3/sfT7oUU6/shseuvsDUc2gFI4luF5qnJtR39wInZ636hcmXm1t1ONwrbfmqXWtqr5nMSDjEK8tbZq/D5Pc2I+9kGVbvO6G5fXmVD59tLcCVf1+heb/Z3/36Waux4eApRXbF5XLoXumqZ9Po14yECkZqbhO/iKJVgpCv031Xj966SmaoZ2eFKgR/ZdkPOF1WjX/W9rqxwuyXuNXhLzEWvW/eFszbkI9jQiAhz45TZ8W0euWIu3iytOY5yqt8DbJWiGIr/1QZej/+JW59c53du2IJg5EY6NKqCaaM6oq0JHMr+mpd9fkkKWhRvnD4/RIkSdL80mpmMtgJ90q7WqMhVyAYMbnej1XyCVA8cRQUlesuXlhe7cPDH+qn5+XMiNaJSLxl/cHT+MWsNYpg5LOtBej75GLd5xY7i5qZTTND40Qrd+xVZp+snfiPnCnH+oOhh1HE7raSJGHnkWLF/omqfX5FIGZ1+KxC1YnY43LgaHEF/vzF95aexwyz5/NIn/jveT8XryzbF/R+aR0rdTBXVuXFJdO/xjX/+hafbj2C7wuLI7pvjYEkSXj2s50RmelU5fVj0baCRtmMTq5vWrlX+4KroWIwEmEv3tAXAPDQ2B4ht+3RJk2zXXmiRmbE55OQmlD/YMTrl7D2gPaJpoXJYOQPb28M67W/3nUU42euxvyNdV8WPkmC3y/BE6VCPflL26e4Ktc/iZRX+VBkMINJPgdonYg0C0pVt5UaTNUWW//rZQ5CFX7KAY0YjFjNjAx5/htcP2sNtv1oXJ8jZte+2F6IcTNWan425qw6gF6Pf6mYems1QFIHOb9/eyMufu5rzFz2g6XnMcNskBGJ+iu1v3yxOyj40ApG1Pu4Jb8IReXV2HGkGJPe3ay7TIMZZVVey1kvO5RWevHR5sOBv9fle47jtZUHwprppPbmqgO4651NGD9zdb2fK1Y+WJ+P5XuOm1oCoyFiMBJh4/pkYedTYzBxxHkht83p2Ay7nv5JUBCgNWRR7fcbTg01q7i8Gje9+l3Q7Qlup6XMy0GD6cDqQlvZffO2YOOh03h/fX7gtrX7T+LCJxfj0MnofPm9szYPs5b/YHqIoKzKhyYGx0E+CWidiLS6qho1rVNyaA7TSJKEt9ccDAQGRoEUIAzTCJuZyWT9cPws5n53SHFsNuefDtpOzAiJdT7v1s48Wb4nuKDyiYU7UeX1421hjSKrJ3K9jEs0mK0LsjrUZPavV/286s/Q6yv34755ucLzOpCRbC7rasawPy/FsL8sxb5jZy0/9nhJJfYft/64cDz28Q7cOy8Xd71TEwCfCmN2oZ6vvz8GADgYpe+lSNt5pBgPzt+KW99YB5eVNUcaEHZgjYIUkxkMp8MBj8sZVPSqVTMSqZSwXsvs1AQXki2sRjvyhWWKnxPcTsy5bRAOny7H298dwtYQV9WyH45b63ESjuc//x5NhS9rowDB6/cjJcGlW8Qon6jMnkytTGEVh2nkoOLTrQV49OOaKcYHn78qZEZBPmlrNZYzMurF5TWPE2tcNIJfcbaR+KUnzgDz+6WQiwJaHabRK+iNBrPBSLTqM9SfLfV7/sxn1gqHrZJbBizbfQznt25i6bGDnv0KALDu4VForTErMJLk4ZhV+07irnc24soLMgP35eafQb/2TcN+7pZN6l+fF0vi6umhirsbqsYZQp0j5C97df2BVmZE/oJ66/aLMKxrS93nHNixGX54bpzu/afKtK8emqUmaA4PmeV2OjDkvJa4fmD7qBWj1oc4TdPoKtvnlwxPRvIJaI3Jbp9m+6LoFbDuOKIc+w8VBMnPUe0VZ2SYP2nm5p8J/L9H4wrrjDCEJRZ6Nhe+vM1Mu61P/5doEw/x05/u1C1UDZWlCpf62IQK3Pw6NWD1ZaYnkp5dOotort1/EoUaGcRwiBd9i7YVKoY5r3t5lannOHG2EpvyTge9xy1SEwP/r/f++/1Sg1kPSbzAUtdyNRYN76wRR+S/dfX3SILGyfyavtkAapqfTb2ym+5ztkpLDLqi7dgiJdA07dRZ7WBk/IB2ppa41yO+ptb+NyRHDU6WuflnFG321eQhj9+9tcHUa5WGWK9GVKnRXVVS5c1CnZgqvX5IkhSYYQFYu4IXe4FoTbc+K0yxFbNHYgbIqObG7D59sb0QV//z20DKP9zMyPvr8/HfDfmhNxSIwdvsbw9gU94Zze2i1SlV/R6HCtx8fqnBdW1VnwRX/3AC/1l9EDe++h0umf510PYbDp7C0t3HLL1Gsmo42Go8VlJRjeF/WYqfv7IaS3YeVdwnzlzUm4Z+1zubMPCZr7DRYs+caBB7FInfxdGoa4qWhn3WOMfJX/zqP1xxmKZNehI+vvtSjBZSkM1S9FOIWldrr/x6QCBaVo+rzr51IJ65rjd+P7yL9V9AIGZDolWMGilyX4YkjxO/yGmnuO/NVQcNH2t17v5Zk4sQOgBU+YSGZnJmQ/Vyodbaqaz24b55uYrPgZVgpFpcYVojqBSHksTptmVCBkh8vZe+2qP5OqG+JCfO3Yhth4vwf7Ut2M1mRrS2k7sYm/Ht3hNBSxnoDdlZHWoym2hQv1+hjtW89fn4ONd6i/1QIpXs33fsLH712lo8XtvRWMsvZq3BbW+uN7xQUFMPK/ssBmSFRRWBYcctP55R3CdmF9SdruXgUO7M/PrKA5ZeNxrEzIj4aYllRrG+WDNiI/nzHlQzIpwEMpI96Ksa+zQKRuQrpP/cfhG+2nkU913ZDc1TEwLRsvoPa1TPzKDnCEdjyox8tavmCiwlwW253bbVdPhtc9ab2q6kwotth+uGZOQvF/U6PmaGaT5SnZis7HOoL3QxQyEGI2IGyKsIRrT7fOhd7ZdX+RTTik+VVsHr85v+Uj1eUom2TZODbt946BRyOjbXfdzR4gpsyT+D32vMBtJLGFq96jQ77KHuKyIeK633UiwMDsep0ips+fEMhndtpfg7tjpKoxeohyqEFS/GjhVXItNkrYm6o7XV90M8gf94WtkcT3wuMQCfs+oApn/+Pf5z+0WB2xrCCtliVlV8Hyqq/UiLbulOxDTss8Y5To7s1Z9l8QSp9YWQlqQfQ8pXayO6tcLT1/UOpBu1MiNX1w79RIJ4JZHgtq+A6jdDOoXc5kDtTKBkj8tynUyku5nKyqt9WCHMRKnWmBUz8JklOBKio6jWcIaVfRav9rWyMOKXXoXwWuVVYmYk9BWqXlZh6ge5mPBGXbOmQyfLcOGTi7H9cLHm9mrHSio0C5THz1xjmNUa9eJyzUAEAN75Lk/z9jv+sx5vrTmIl5fuCzkN2ooyVZ2ReKys1tpoyTtZhic+2YHDZ8rx3ro8DHh6CW57cz0+UA1nmf0rLqmoxsq9x/Hdgbo6KnmPT5VWYeFW46yNGGCphyWNqIdp1J+pUPUS4vbqYMSnCEbqjvkTC3ei0uvHA/+rmz4cqZGQAydK8eD/tgS+n6wQP/PKTs6NJzPCYMQG08b2QO+26fjtsJqhETMFrCKn06FbxKo3U8SlCkaSPS5M/3mfoO2u6xdegCJeUdlZwGol05GS4EKi2/wMIiB63UzVKjUyI6fLqoOyHkGP0/jykQsG806WYcp7m7Fdtb6PeJIWv4S1AgbFjB8hW6HIjJgYvtC7iv18e2HQbWVVPuwsMBmMFFfq/g3sM5hyatQCXlwoUfTD8VI89vEO/PXL3bj6X9+a2j8zSlTr8Ij1IFau/vVOxr987TvMWX0Qo15chmlCYbe6bsJsJmfcjJW4ZfY6/Oq14EaCt81Zj8+2Fhg+XgyWQ/16e46WYHHt+xEqMxKqwFgM7ArO6GdGtGbEid8b2w8X4d21efUuFr1l9lp8sOFH3KzTkNGI4iJB5//VTpytxP82/qi4kLATgxEb/GHEefh08rBAbwD1R1icmqX3+b5r5Pmat+vVFMiZC/lLd+KI8zT7afz9xn7Y+dSYevU0sXOYxsprpyS4rA/TxCglW+2rKUS1+nLqegcA+PXra3HoZCnuencjPtlyJGimgbjejaIfi0bkJV516WVGzJwwvT4/jhZX4HdvbcCVf1uO37y5LiIzQo6VVAY1mpPFYs2ZSKjwqodphPfEQiGv+n3438Yf8fLSfYH1etQnqnA7POef0s/WbRFmZ+lRZEZCfOBH/30Ffv/2Rmw8dBrJHuX+qjNyobICetkE9XPJ94n7JtarFBRV4OEPt2HRNu2g1Sw5O2NmPaUqr18xTCr+DSvXr9I/Br9+bS3u/+8WTP88ulPFzWIw0gCo//7MXJEkebTfOr0TgXqmTOv0RM3tHA4HUhLcihStmWyJeHVttYA1kvPirQQXyQnWh2mivfiUfCwkqea9jNTUvCU7j2JPYU1mwOuXFKngPUfrpmGKV4FaJz7xfp9fClxdlloNRvwSnvhkB5bsPIq9x85i2e7jplrQh/rTKK3y6mZGKlUn3zdXHcAnIRZOtIP6SlUxTGMhNef1Sdh46FRgdtP9/92Cv365W3f7JokuxectWo0IAeWJ3asYpjFn++Gi4GEa1ecu1Aws8bjKn5mi8mpMfm9zoK4MqMkALt5RiJxnvgrcpvX9u6s2e/fy0n147OPtUZ1We9kLy9Dr8S8DWTQxMBEDEKPMyO7av/tQmatYYTDSAIRTAJWk06BMLzWpbhHcOk07GJGJUzyn//xCzL51oOH29ZlCZqXZWijWhmnc1mtGohyMNBWKk6t9fsXsFiPjB7QLuY14Ir/shWWo9Ppw8EQpfvZKXctrMXug9Z6qr7TkL/wy4XFmakaqff6gjrVmilSbhGgoWOX16zaaE/d9249FeHLhTkx5b3PI1wzX22sOYt76vMB6UKdNdggtVx0HMQCxMoNn4dYjGD9zDX728ipTQXSTRLdimOSNVQewfM9xvL8uD/9e/oOlk6tfCFS1iN9TfguZEZnXLwUVFqtfL1QwIm4vH9cZX+/Fwi1HFMF6lc+P37+9UVFvp/X9K6Hmff7rl7vx1ppD2BtGB1uz5OzJlvyaIVexGaE4I7DSxN9USaUX4/6xEn/9MvLrPFnB2TQNwJPX9MJDC7bhzpGhW8jL9E7gel8A6hbBrUIEI2JqJDnBhVE9M9G3fVPdtKv4JbZbp+GRnpQEd9AS8VoS3M6QHU211mVoluJBSYU36OSa7HEh0WIgFO1hmoxkT2DqcZXX/CySqaO7BRbI0uJwOBQBJlDTkG2hKjMgBiNaga36C76iuqZ9vthbxMwJ0+eXgjJiZtp5pyS6UGIw3FLl9Zta12fb4cgVnMqqff5AvVRReXWgc+6Li/coVucNRR2MhFvAKq8Btf9Eqak+LamJ7qB+JbcKxcSpiW7cfElH+P0Sjp81nvVy8GQZ5r2zSff+S//8DRZNGYZWaYmqVbxD7iaAmmUt1NOZ1ZmvUCdi8VhW+/zw+SXM/jZ4mq76eQHo1pqJmQirU7/DIV/I6n0uzLzvVV4/dhYUY2dBMR4YE3pNtWhhZqQBuOmiDlj3yCg8OKa76cfoZUZC1YzIWoeY76VVMvLmbwbhsu6tNLf364ynmqEuRNPTLCX0+htaHU8f/EkPzRqY5AQXEi3Wt4STGRl6vn7HXDW30xHIYEx+bzMWbDps6nFJITI8Tkdwe3evz4+jJcrsRKkiGBF6F0gS/vD2Bjz2sbJXxA2z1uBYSYVimMbMMXp/fT6qVF/WUz8IvcBZqLqGap9f9wtYvD3/dN0QRKSG3kqEgFocarESiADBJ9GyKuMAUc9+4erezKwKBxyGBdrybK+HFmzFxc99jW8NVoV9+tOdWKwqiBUdL6nEW2sOAlAXTWvvwLGSClw1o27xv39orVjttZoZUQ4V9X9Ke0VtrefRK6kTg3mtpoF1z+nDMQs9VURaf196Te84m4Ysa52WZKn9sl7NiN6XlfpE1DTEiV19FQ3UdCW8fmB7ze3FL5HHrr4Al3VvFXJoR6Ye+9Vj1F9FptX9s6SiWrPWICXBhUSd46jHV5tyt9KDwco6FyUV3sDVtZUlwENlePxS8Bdopdcf1JpbTPeK7+mRogp8uSP45LL/RCke+XC74jaztS5mihvVjBYxBEIN0wi/j1AkaKUOw8jBk6X4oXbGTn2aTanH+e95Pzfw/1Y6rR4XgiAzV8jVPr/h88uzsD7YUJNxeXnpPtP7okU+qYon1z99tF1zxeC/L9kTtDSCmvqYq3/nLflnMOrFZfjm+5rPsTqboJedrdI4oesFWmUmZ5X95o31uOi5ry1nkQHVRULgNu3XkrNOJ89W4qmFO/F9oblZaXZgMNJIiZmRUT1aB9q936fTKt6takqml1mRaQUjgP7sngQhbdktMw1v3nYRRvXMxOf3DEOv7HTD1zKbGQkVQAHA1X2zkJLgwqgerQO3tW2aovn7JCe4LM8a8ktS7clWf5t/3NRP8bOV6cMnSystZ2tqXsP4MRXVvqAi5kqvL6h/hzhFV6xXKTdoa796nzJomvzeZlz3yuqo1NekhqoZ8fl1v5jFIEUx+ydC6fSfv7Iao15cjtvnrA9aSNIK9TCNSFxzyAozwVG132/4nh0pqkCxMO24Y4uUemWV5KBbfM39J0oVfWZkZpYYCMqMqH7nO97agB+Ol+L2ORsgSVJQEG32efVIkmqJBFWwU1Htw2/eXIc5qw5gzf6anixzLTSs8/klLN5RqLiAkIN+vYySfGwf/Xg73lh1AD95aaXmdg0Ba0YaqUS3E33aZqC00ot/35IDp8OBI0XlaNcsRXN7l5AyNGqaJtM7ResV22ZlaA/79MxKx3mtmgRd1Sy4awh+Xls4mWxileOMZA/+dNUF+Ok/jfs5tE5PwqZHr0Si24lNeaex/uBpjO3dBg/+L3gIYGS31pbaTwPKGSR61CdMrezLF/cO0/xiqKj2IyXVDVhcf0urVkZUVuUNCsh+OF4atKid+PYeOVOOAydK0bllquGCf6UafQq25J8JzC6IpFDDNFVeyVQBq3iCifTY/jffW1tjRc2o70O4WRwzJ1SvL3SH3wufqBvKyExPMlwBOxQ5sFEHQFpNv8wEtuqZUXmnynBxlxaBn88Ii4TuPlpiGPSJzAYjZyu9ir+TjzYfRqcWKYGi9LnfHcKy3cexbHddc0P5789MLdC76/Lw6EfblQvh1f5X732TA3OzTQPtxGCkkXI4HPj47kvhl6TAGiJ6gQigzHSUmWhyY7UVtFExm7oJ2uxbBypO2Ckmakw2PXqlqf1wORyBrE9Ox+aBFuBamZHB57XA/zbqF31q8fn1T3ay5qphGTFr8dmUoeiVnaH72PQkd1h9WkIN8ZVV+YKGadSNntQ+zj2CL7YX4oaB7U1Nu1WLRpvs1ETjz0qVz69Y40d0tLgShUUVaJORpAhMIjVMEylyFqNr6yZBMzLCDZzMZEa8Pu3MyANjumtOCfZLUr2CEblg2cxMvHBe5oH/bUXPrHT0yk6Hw+FQBNpaRal6Qv29y95acwgXZNVlgeesPog5qw/iL+MvxA2D2muuVizfNvrvK0I+/1e1Q0Pi8ZIzU3rBjJwxCXWx0hBwmKaBM2qP7HQ6NBcz0yJeFZi5ItBbwVdvb/q11z/BirMmnrmuN0b1zFRMwQ01TPPyrwbA5XToDqn86uIOgf/XWPW+huqhcpt8q4tr+SUTwYiqtkX8XUPVvcy5/SJ4otBO/81VB4NWI1avU6Sl0uvH298dwvf1HNuOlJSQU3t9qKodysjp2Exx36zlP+CS6V+juKJakX2Ixn7WhzxUNuS8mqt6MZgNd19N1YzorH1092XaDRaN6nPMkIORUFkPSZIUtRhW/PSf32L4X5fiTFmVIji2MjxrpQhU6+Lmwfk1CzVqreB9tLgCkiSF1QIeqMt86AWp8vtp9veNdusCIwxG4sTRYmt5f72akcFC2lP264s74HcGq/6KmZHubdJqb6t7fqMCVo/LgasuzAr8nKaRpu8nLCTo0tlv8dbrc9rhvxMHAwC6tGqi+9paqn2SoqBQi9EwmNF9/7ipHwZ0aKaZGbGaqTLDTDBSH2YycFY1CZEZqfbVXa0nuJyYMLhj0DZ5J8sUQ0uPfmSudiBW5JWe02s7NFf5/Hh95X5c9/KqsN8zM5mAd9fm4aPN5mZvATUBTn2CkeIKLyqqfZonwOc/r+t58fCH27H6h5NB25iVf6ocC7cWKC6krAR1ZodpAO0hS1mJRoHsmfLqemWX5N9Drx+RnBkxG4zYGZgzGIkTVmsj9D66rdISMWVU18DPD4zpjmd/1sewSFMMRuR0oZgtMJoKrH7eNQ+PwqqHLlfcJqYg9f7oxEzPX6/vi/Nqg5BBnZrjxev7YlyfNrr7IFq593ig+EyPOlslFlSKw1PqPhvu2rSO1to+8lVyJJ08a7EwxSKjOpNwhZo2Ll6te9xOPHVtb/xBFSiXV/sUTdrEbpuy5Q+MRKrJwupIk09o6Um1y0VIwDOf7UJu/hm8XTsdVk/fdtoZSq2rci1GHVrV5qw+aGnasnom1Jb8M+j31GIc1/gczlr+Q2AI4r112gsVWuFSDdPkaczY0WMl4DLK4GgF5zV9ccLPRvx7xQ84XlKpW8BqNTNSn8CovhiMxIkOzfXrSbQY1SCI/T70Urgi8aQrn2gTXXVf9EZdU9X3NUl0By0RL/6h6Q0vGf0pjs9phwvbNTXYoo7W1Y2aOrMhfpk5DQIn+WFaM2PSEkPPJFL75v9GIPcx/Vqbk2fDz4z8+uIOuPmSDvhJL/0g7nhJeH0UjIQalhSbnsnvg/p4nimrNryCHdMrEx1bpKKNTlF2rKQnB2fR1L1Z1PTuLTaYjWJ2NpuWpxbuCL1RrYs7Nw+6raLaj890VvW1kpEIRR0khMpuitRF3ka0anPkv3O9tZHEIXSrth8uxp1zN+oO08j7I16wGRZIR/CYW8VgpIEa0ysTAPD74ea7shp5/OpeaFFbJ/G8xmq9akaBtNURA/FK36WRGTEqrtKbsio+RDyp6w3T6A07ycyOleqlMcVVlBPdTsXvNKJ7KyS4nIrhJKAuEyJzGWRGmpiYASU6v3UTdGnVBE1TEnRXeJbH7K/pm43fDets6fkv79Eaz1zXx/CEfTAKa5uoA7i+qmNaWuUN9G6QPzvqHixnyqoMr2DlbFx9ljiIhLSk4AA0VG8Wvc/x1xrZH5nRDKWf929r+Hq7CszXEukdT71VvsOtE9FSnyFJcfZLKFoBVE1WRr8u5PDp4ELya//1LXYKMxC/+f4olu/R3o8Nh07rFmG/sHgPpi/apQjib3p1DV5crJ0Bq0+Wpr4YjDRQ//rVAHw1dTjGDzD+MjCrTUYSNj56JXY99RPcdFGHkNsbnbytNGcDlF82cpZEzJaoW9WL9LIm4v6ZGaZ5+rreAIApl2tncsyuibH+4Omg2zKSPYorIqfTgZZN6trtt0hNwJbHR2PBnUMUj9PLjGh1bjQzHVskHpPROtkL+Qu6WYoH3TLTLD2/nHI3akIWiYXWJqveL3Xg2qmFMuO340gx5qw+CKDuM6buhXH8bKXhl678mbPzKhGoCRKs1grpxU9f7NBfUdboPXzh+r6Gr2d2eixQEyhd3Td40U39YMSHz7dFZhG3aA9JyrRqc5zOmqnEx3WGtLRW6d3yYxFuenVN4Ofb52wwfF2jWVb/XrFfcZG25cci/PMb7YZ1rBmhIB6XE+e3TrN84g/FbLdTo5cd2KmZ/p0axJOrnA0QI3Wtk+/5rWtqOq7V+PICgBG1Td46NFc2NNMLosb1ycKWx0Zj6mjtlvvhXAS3TkvER3dfimX3j8SVF9Rksto3rxlCEldFdjkdSE5wBQ0hqWtG5KBMa2xZq3DXiHgcfnVRB/zzl/2xWTU9Wm4k1STJrXlCEKcpBu1P7VW7Ucbm0EntK8E/j9fOzM2+dSDuGKrM0KiDBnUA166ZcshOJAcVGapmeUdCTGmWMypmFymMlmSPy7C5npZwmpAZHUO9Yc9w+PwS/nFjP8z7/SWK2+XgUa282oc7Dda3MZKR7FF8VsyseyQL1UDwygsy8eotOZiukWHWqrlwORya2Q+ZXt2NmfW6ZHo1I4F9aAQFrOwzQpqMMiO9sjMw/84hyG5qbkw9QWOYRqQ1TDPv95fgu/2nAid5tReu74t31+XhZ/3bKlocG/3RqU9KonB6YozrkxUYernt0s5o3ywFg2rHxcVVkfWu/IIyI7XHXOsKyuqCfm6XMlukdUUqS010BwWEfxl/IXJ/PIOdOo3L5EyN0VW13qqlTXWmN4/qmYlRPTOR07FZ4CSknnqt/qz0aas/pVwORq7r3xb5p8pw8GQZluw8irnfGRdEysM008b2wNQPtiDJ4zRcij1a9JZ8MBLOQo792ze1tPRAuHx+CU6nw3Qtzqf1WNr+vFapeP3WQVi59zjueT/XVK2XLCsjyXCI8YaB7XHlBZkoKq/GtAXbFPdpDZM5nQ7D2YzHLM50VEtwO0MGzkbr5Ig4TEMNTqiETE7HZsjK0L+iEmkVsIq0AogWTRJx1YVZusM0zVITcPdl5yO7abIqM2Jql4JYvaJMcDvx4E/qsiwelxNj+2QFhmdaCQsR6tXEqGtG5JWUCzSaI5lphCaugWOlj0LzlISgK/AEtxNNk/WDN3mmh9XhI6Cux4se8bPXtXXd8FFGsgcu1XFIT/Lg2z9ehtcmBK+DJH/W0pM8eOSqCzBaJ7BVkzvm/nxAO6x9eBSeva7uCnjRlGEY29vczKtPJl0acpaW0ewgvSUbBnbUz0yGkxnpanGILlxyoGR2eYQZGovhmeVwONA8NSHw+bQyzbyFMMR6+6XBtVRywa/Z5oQupwPHDIq5je4zU8vWJNEdMjMSqmZOxmEaanDMfnjNCDUkU9/ugOJJPdxhLavf4U9e08uwAZcYGKiDDpkYpD360wsCPVi0mLmy+WTS0LrnNqjDUWvbLBmnVRX9CW4nbh/aGb++uAPeuv2ioMfInVDF2hizQi94WPe7Xt03G4/+9AI8MKY7lkwdHtxnxFHTeVgeHlP/DqLzWpvrKSOeZDLTkxRBQWqiC0N1CoLVnA4H/nZDzbDEZ1OGBt0/ccR5hustJemctJ+8thd66gyhWcnw3TSoPT6bMjTksAQAvK8aWjFLbDwnF7AazZ6LFPkTJP+Nbj9SZPqxLYRg+bIerYKCZ3moWz3MqscdKjNiMD3aTBFvsscVsjOv2U8Fp/ZSg3NR7XCDmS+qUMT4QCvwGHyeuS93PRbOu7q0vsTn3zkYM389QHP7UAWfGUJWQS+QELMXvxVqJd78zSD0aZuBkd1bCc9R90v+85f9Fc/zSu0+ZjdNxi8vqllV+d4rusKs7KbJQVM/E1xOtGySiGd/1kezRkjen94GwyR6QrV0Fz8iHpcDvx3aGXdfdj5apyWheaoy+HHUnnZSPMGBoXrBwfNMNrhTryUk/g04HQ5TV8SXnt8CF2SlI8njwsVdWgQtAXBVnyw8NLaH4cw0vWGa81s3wb9vztG8zydJePzqC0Lun8vpwPPjL0Sv7AzdYUTRJRrNDkPp0jIV/6ttLgjUDbnFIhiR/+bkLIZejNZMIxhsIWYYHY6gbIHcK8hs9rGsyqdZpCrTK2wFgF++9h3+vfwHw+dPS3KHXNIgVDG2XJNmZ9E2a0ZI02NXX4D2zVJwdd+s0BuHoJj5InzxrX/kChwvqUT3Nmno0SYN3xeWoFVaIl66sZ+l52/X1FoPFS1amZGcjs2xOS949gwAdMs0PrGJV/96wYhexuSyHq1xWY/WqPT6MH3R98jp2AylQo8CdV+IcX3q3qPnftYHD47pgWYhhkJE2RnJuGFQe7yweE/gNvGEnJLgxnu/uwRulwNnK72KdvcZBkM5erQCB5GY3VJ/4atb7cubigXDMvVJNiPZg7tGnodXlhl/uTtUIYJ4LFxOh6mrzHfuMM4kmMl06dUJJbpd6NAiBdf0zcYnW47gos7Nse5AzdpBfn9N/dKTC3caPreY/jcTjAA19TnbDpvPMOw/Uap4L+Vzen0vcIae3xIXd26OF5fs0d3mmdrZc6GWD9DKKLQQAl6n0xG0jfz3ZzYLW1blwwqdabkAUFCkH6hsP1wccpG79GRP0D6Knwkg9PBLy7RElFR6ba0ZYTBCmtKTPLjHwtW1EfFPVsyMtEpLDNRJfDZlGPySZPqLUdShRQpm3ZyjeZVjlt7UXjGoWDRlGKZ+kIsurVI1e0CI0sXMiE7Qcen5LbH7aInuFXCi24UnrukFQLnmhdGMKIfDYRiIjOjWKqhfQXKCC8kJLjw8rgeeW1TThlsdZAw26AC7/IGR2HfsLP7+1R5Tq4OGmtElxh/qL/xmqcr9kq9SteortK7AH/xJj5DBiPqLW6xxcDkdYdVl6DE6oYmfixapCRjQsRmu61c31f/Zn/XG0PNbYnSvTPR7agmA8NYWOVtZlxXrnpmG3Ue1e4e8edsgDHzmK9PPq663kDMj9RmWffd3F6NXdgaOl1QGgpFxfdpg0ba6qct/uqonzq+tNVJn4a7rl42PcuuarGn1PhEzI15f8Crd4TaJczi0MzT1LY6u9vmDfg91vVeoAle5toY1IxQ39L6IXE5HWIGI7Ce92yiWC7dKL3UsjumnJ7vx+T3D8MqvtVPkInGWid5V8ANjuuNPV/XEl/cOD/l8YirX7HCDln/c1A9PX9tL8z6xU6xYOBpKxxapGNUzU/MqU+v9DpWmN7rgFK9ae2Wno3db/enH4Q4HqNt/J6iGaSIRi8gB6n1XdgMAjB/QLmgbcTgou2kyXpswULlOU5IHNwxqr5idFM6sMHF9Jq0Mk6xlk0Sseuhy/Omqnri6bzZuu7ST5nb/uKkf5v72YkWBN1B34q9Pu4Ih57VERrJHUa9x92Xn44Exda8lfubUmZFf5LRX/NylVSpe/pVyKFYMxEurvEEn+lDZFlm/9k0Vs9g6t0wN2saoZsgsrWBGPeW+0qAfzKgerQPfvawZoXOa0yDt3lDcMbSLZl+N9CQP0pPcSPa40DotyfQXqTjLRC8AS05w4Y5hXdCxRfCXlNrFXWpqeNo2TUZmevhtypumJOCWwZ007xNTtGb70SgfH/xFplfg+ltVPxFRgkv/tcX9emhsD8X70Stb+f6FcV4GELxKq0c1TVocovvP7RcpToRm/d/omiDkFzntsPLBy/DXX1wYtI34uxlNoRadb7JIV9QtMw3z7xyC76aNCnlB0LZpMu4Y1gX//GV//EGnO/S1/dpiaNeWQdmqSGaUFEXxTieyhOnCYtZSncXIEtoR5HRshpd/NQBXXZilWC5DDD61OqqamXJ9fU47vPGbQeiZVRfUaxUd9862XnOl3o9yjSJXdV8io1W3X791YOAzzswIndsM0u4NRUaKB4vuGYbLhKJRoGbMeO3DV2DDn66wdKUtDpXoDdNYMaBDM3x41xB8fu+wej+Xnpsv6Yg+bTPwhIkCSC3tmgXX7uToTEOdNraH7vMMPq8FhndrFdQATTZhcEf079AUF3dWZsL+c/tFmCEU91pZxE2kHjdX9MlxODCwU3PM+GV/fDp5KEZ0a4XfDu2M+2uDCzPG9WmDbGF9pfbNU+B0OnBtv+BeMJf3aA0A+L3BqtgA8OnkofhFTju8eIN+x1Sj58jp2AxtMpLwmyGdQux9HatX9dlNzbUC+GTSpUHrT6kpuy4rp0iLs1/UwZUYHM++dSA61WYr3Kr2A7+6uAPOa5WKK3sqp4N/838jTH2H3TqkU83UYiEo6KkxW85oBl0o8ow5rZ4oRi3+1RwOB7plpqF/h6aBKft2YM0IUQjhZAnaNk3G5MvPR4LLGbHZA/07WOt8a2TK5edjxjf7FF0kW6UlYuHk4CmoZk3/eR8Mef6bwM+rH7ocuwtL8JlGS2+jBe9cTofmdGLZU9f21ry9ZZNEXNM3G1Pe2wwAKK7QXxzOyM9U67GI++qo/d9rhPR7kseFSZd3xcItBbr1FiK9aZjTf94HH+cqF4379y05OHy6PHDS1NO7bUbI1u3TxvbAqyv2G24zvFsrdGyRYqqVv14fFLV5v78Er397IFD/FEpKbQ2T7Np+2fh8WyEeFYJkMRhxOBxIErY3qpnKSPbg/tHd4JeUzffEduktmyTiuZ/1gSRJcDgcGNChKTblncF5rVIVQ1pG5KyFOFyiDtaX3T8SK/fW1W+N7d0Gn2/Xb9uvZlQEbXYtq6+m1gwR6/1NxRKDEYq6SPYsibZIZm7+T6f1fENw35XdcP3A9oatwK3KbpqMJonuwOqk2U2TsUd1ctZra2+2Z4MZ913RDW+uPqCbWXn1lhxMW7AtsDZPgssZGCtf9dDlQVflihOfweuardfQ206rFsHjcoYMRIzIs9RGdm8Fh8OB939/Cf44f6vhySc7Izki6wrJLu7SwlI9l9vpVBQxP3Vtb/zjJuV0djFAlCQgRcyMhOhjM+ny4ML8/cIidhe2qxk6kb8LXvl1DuasPoibLwm9ppdMzsA0EVbbbiV0Zb5z5Hno1DIVW348E7jN6kVLTX8g7YBbb1hvTK9MfLnjaODn8y3UhkUbgxGKusYTisQPh8OB9s3rPyU66HlD3P+y0Lelc8tUHDhRiuv6ZWOSzgKG4bjniq6YfPn5uuuqjO7VBldekInSKh82HjqNAR2a4u53N2NEt1aawwNi2t/ohBEqFHnwJ93x6or9eGhsT1O/R328PmEgFm0vwDPX9caRM+Xo0LwmoLmkSwssf+Ayw8dKpltkhe8v4y/Eg/O3at7ncTsVQ2VaAawyeFXur3rGlRlGU53bZCThIYNhRS1y1kUMClqnJeKt2y/Cp1uPYNJlNZ93cVjEbEdXWccWqbqz1/SKbP901QWKYKQhYTBCUdehReRPetFyw8D2+Ob7Y4Gro4bK6Qhvcb9oe+zqC/DA/7YG6hPETNOWx0Yr1gf68t7hKKmoVrTfjpRQC7w5HA40SXQHFlw0GhZKTXRj4aShcDiMW5mHWvn5rpHnY+Lw8yK6+JyeKy7IxBW17e+tXv1aKfx98fq++GxbAY4WV2DHEe0To5YbBrXHzoJizUXyPE5H0CrYamIdll9SBhPqrEDz1ATTC+WpV4EOZVjXlobr+og9alqlJaJrZhqGd6urS0tPrttXdafXN28bhOFdW8HldODbvSdw8+y1ivv7tM3AZ8L6PRMGd8Rbaw4BgOL4AUDvtun43bAuUbkAiRQGIxR1gzo1x7M/6605ta2hGdMrE1/cOwydTMxwsZPL6YDfxgZFeq4f2B4jurVSpKRl6oUKE9zOqAQi0dDHRHBq5t2IRSBSX1Y+VeNz2mF8TjvcMGtN6I1V7h/THelJbsxQLWfvcDg012cSiUNnfknCwE7N0bttOrpnpgcNtQ7v2hIf5R4x7G8iZ+n0ZprpeW3CQLy3Lg9upwOPfrwDAPDIuLrMV6qQodBqECjO/BH7mwDAZd1bB/5/aNeW+M/tF2HdgZNo2zQFe46W4Ce92uD5z2v6Ak0Y3BH92jcNBCNFqo7KI7u1xrX9lLVQDQ2DEYqJX1/c0e5dMMXhcKBHG/3eFQ3FzZd0xJurDgau7BuS1sLU44Z/6o2ccKcSi9KS3Cip8Ibs8BtVYfwe4fQ3aZLoxtTR3XG0uBLzNuQHbk/0OPGH4V3w7xX7dYdHnE4HhndrhZNnK9G1dRpcTgc+naw90+zJa3qjaUoCfpET3MtF9s4dF2P9wVO4+kL91a21JHlcuK22udu1/dsi2eNSDPN0b5OGB8Z0R1aGdlsAcZhGvdSB2ohurRR/72Ln1qyMZPStXUEcQNDyDmbXU7ITgxGiRuihsT0wrGvLoOmtZJ9QwzRmfPCHwZi1/AdMvdL8VOFI69+xKdYdPBV6Q8HYPlnYcOi05oKFoTx+zQXI6dQMTocDaUlupCd5cO8V3fDTC7MNM1L/uW0QgNBF5xkpnpAzebKbJtc7c6A3Lfbuy/TrocR+RKkWZ+15FE3xknBeqyb48K4haJWWiKLyavx7xX60bJKIv9/YN6y1hWKNwQhRI5ToduHyHpmhN7RZOAvpNVZDu7bEwZN5Ya3XI+uZlR40cyTW7h3VDRnJHoy+wPzn6zdDOqFTi5Swpp+nJLhxw0BlZ9TkBFfIobGG2rPICrEpm9XZNOLK3K1qhzvl49+uWc3U+pZNEoOed85tgzBtwTb89RfGU8FjjcEIEUVN89QErH14VNjreTQm08b2RKcWqRjTq43du1IvyQku3DXS2uwml9OBUT0bfnDc0NQMC6dh37GzGFS7UrpZYp+Rlho1WnpN5kZ2b40100ZZ29EYCKsb0yuvvILOnTsjKSkJOTk5WLlypeH2y5cvR05ODpKSktClSxfMmjUrrJ0losYnMz0p5MKC54LURDfuaOAzFqjh+WTSUOQ+PhrpSR4M6NAUAExlpVISXOiZlY6OLVIaxeSAUCxnRubNm4d7770Xr7zyCi699FL8+9//xtixY7Fz50506BDcFObAgQMYN24cfve732Hu3LlYtWoV7rrrLrRq1Qrjx4+PyC9BRETUGCW467o0vzphID7bWoDr+oeuX3E4HPhk0qVwwLijcWPhkCxWXV188cUYMGAAZs6cGbitZ8+euO666zB9+vSg7f/4xz/ik08+wa5duwK3TZw4EVu2bMGaNeamgxUXFyMjIwNFRUVIT2/4Mx2IiIjI/PnbUjhVVVWFjRs3YvTo0YrbR48ejdWrV2s+Zs2aNUHbjxkzBhs2bEB1dXhrRxAREdG5w9IwzYkTJ+Dz+ZCZqRzPyszMRGGh9gI/hYWFmtt7vV6cOHECWVlZQY+prKxEZWXdipvFxeY7+xEREVHjEtZAk3pKlby6oZXttW6XTZ8+HRkZGYF/7du319yOiIiIGj9LwUjLli3hcrmCsiDHjh0Lyn7I2rRpo7m92+1GixbajVimTZuGoqKiwL/8/HzN7YiIiKjxsxSMJCQkICcnB0uWLFHcvmTJEgwZMkTzMYMHDw7afvHixRg4cCA8Hu3pfomJiUhPT1f8IyIionOT5WGaqVOn4vXXX8cbb7yBXbt24b777kNeXh4mTpwIoCarMWHChMD2EydOxKFDhzB16lTs2rULb7zxBmbPno37778/cr8FERERNVqW+4zceOONOHnyJJ566ikUFBSgd+/eWLRoETp2rFkIraCgAHl5eYHtO3fujEWLFuG+++7Dyy+/jOzsbMyYMYM9RoiIiAhAGH1G7MA+I0RERI1PVPqMEBEREUUagxEiIiKyFYMRIiIishWDESIiIrIVgxEiIiKyleWpvXaQJ/xwjRoiIqLGQz5vh5q42yiCkZKSEgDgGjVERESNUElJCTIyMnTvbxR9Rvx+P44cOYK0tDTDBfmsKi4uRvv27ZGfn8/+JTbhe2AvHn978fjbi8c/+iRJQklJCbKzs+F06leGNIrMiNPpRLt27aL2/Fz/xn58D+zF428vHn978fhHl1FGRMYCViIiIrIVgxEiIiKyVVwHI4mJiXj88ceRmJho967ELb4H9uLxtxePv714/BuORlHASkREROeuuM6MEBERkf0YjBAREZGtGIwQERGRrRiMEBERka3iOhh55ZVX0LlzZyQlJSEnJwcrV660e5cavenTp2PQoEFIS0tD69atcd1112H37t2KbSRJwhNPPIHs7GwkJydj5MiR2LFjh2KbyspKTJ48GS1btkRqaiquueYa/Pjjj7H8Vc4J06dPh8PhwL333hu4jcc/+g4fPoybb74ZLVq0QEpKCvr164eNGzcG7ud7ED1erxd/+tOf0LlzZyQnJ6NLly546qmn4Pf7A9vw+DdAUpx6//33JY/HI7322mvSzp07pXvuuUdKTU2VDh06ZPeuNWpjxoyR3nzzTWn79u1Sbm6udNVVV0kdOnSQzp49G9jm+eefl9LS0qT58+dL27Ztk2688UYpKytLKi4uDmwzceJEqW3bttKSJUukTZs2SZdddpnUt29fyev12vFrNUrr1q2TOnXqJF144YXSPffcE7idxz+6Tp06JXXs2FH6zW9+I61du1Y6cOCA9NVXX0n79u0LbMP3IHqeeeYZqUWLFtKnn34qHThwQPrvf/8rNWnSRHrppZcC2/D4NzxxG4xcdNFF0sSJExW39ejRQ3rooYds2qNz07FjxyQA0vLlyyVJkiS/3y+1adNGev755wPbVFRUSBkZGdKsWbMkSZKkM2fOSB6PR3r//fcD2xw+fFhyOp3SF198EdtfoJEqKSmRunbtKi1ZskQaMWJEIBjh8Y++P/7xj9LQoUN17+d7EF1XXXWVdPvttytu+/nPfy7dfPPNkiTx+DdUcTlMU1VVhY0bN2L06NGK20ePHo3Vq1fbtFfnpqKiIgBA8+bNAQAHDhxAYWGh4tgnJiZixIgRgWO/ceNGVFdXK7bJzs5G7969+f6YdPfdd+Oqq67CFVdcobidxz/6PvnkEwwcOBDXX389Wrdujf79++O1114L3M/3ILqGDh2Kr7/+Gnv27AEAbNmyBd9++y3GjRsHgMe/oWoUC+VF2okTJ+Dz+ZCZmam4PTMzE4WFhTbt1blHkiRMnToVQ4cORe/evQEgcHy1jv2hQ4cC2yQkJKBZs2ZB2/D9Ce3999/Hpk2bsH79+qD7ePyjb//+/Zg5cyamTp2Khx9+GOvWrcOUKVOQmJiICRMm8D2Isj/+8Y8oKipCjx494HK54PP58Oyzz+KXv/wlAP4NNFRxGYzIHA6H4mdJkoJuo/BNmjQJW7duxbfffht0XzjHnu9PaPn5+bjnnnuwePFiJCUl6W7H4x89fr8fAwcOxHPPPQcA6N+/P3bs2IGZM2diwoQJge34HkTHvHnzMHfuXLz77rvo1asXcnNzce+99yI7Oxu33nprYDse/4YlLodpWrZsCZfLFRThHjt2LChapvBMnjwZn3zyCZYuXYp27doFbm/Tpg0AGB77Nm3aoKqqCqdPn9bdhrRt3LgRx44dQ05ODtxuN9xuN5YvX44ZM2bA7XYHjh+Pf/RkZWXhggsuUNzWs2dP5OXlAeDfQLQ98MADeOihh3DTTTehT58+uOWWW3Dfffdh+vTpAHj8G6q4DEYSEhKQk5ODJUuWKG5fsmQJhgwZYtNenRskScKkSZOwYMECfPPNN+jcubPi/s6dO6NNmzaKY19VVYXly5cHjn1OTg48Ho9im4KCAmzfvp3vTwijRo3Ctm3bkJubG/g3cOBA/PrXv0Zubi66dOnC4x9ll156adB09j179qBjx44A+DcQbWVlZXA6lac2l8sVmNrL499A2VQ4azt5au/s2bOlnTt3Svfee6+UmpoqHTx40O5da9TuvPNOKSMjQ1q2bJlUUFAQ+FdWVhbY5vnnn5cyMjKkBQsWSNu2bZN++ctfak6ra9eunfTVV19JmzZtki6//HJOqwuTOJtGknj8o23dunWS2+2Wnn32WWnv3r3SO++8I6WkpEhz584NbMP3IHpuvfVWqW3btoGpvQsWLJBatmwpPfjgg4FtePwbnrgNRiRJkl5++WWpY8eOUkJCgjRgwIDA9FMKHwDNf2+++WZgG7/fLz3++ONSmzZtpMTERGn48OHStm3bFM9TXl4uTZo0SWrevLmUnJws/fSnP5Xy8vJi/NucG9TBCI9/9C1cuFDq3bu3lJiYKPXo0UN69dVXFffzPYie4uJi6Z577pE6dOggJSUlSV26dJEeeeQRqbKyMrANj3/D45AkSbIzM0NERETxLS5rRoiIiKjhYDBCREREtmIwQkRERLZiMEJERES2YjBCREREtmIwQkRERLZiMEJERES2YjBCREREtmIwQkRERLZiMEJERES2YjBCREREtmIwQkRERLb6f94IYPMkQkX8AAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(losses)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Let's see what happens when we decrease the batch size!"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# This time, let's try with a smaller batch size!\n",
"\n",
"model = Net() # re-initialize net\n",
"\n",
"# re-define dataloader \n",
"batch_size = 16\n",
"loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True)\n",
"\n",
"# Actual Training loop\n",
"num_epochs = 1\n",
"lr = 1e-1\n",
"loss_fn = nn.CrossEntropyLoss()\n",
"losses = [] # For plotting"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"model.train() # Training mode\n",
"for epoch in range(num_epochs):\n",
" for batch_idx, (x, y) in enumerate(loader):\n",
" \n",
" # 1. Compute loss \n",
" logits = model(x)\n",
" loss = loss_fn(logits, y)\n",
" # 2. Magically compute gradient\n",
" grad = torch.autograd.grad(loss, model.parameters())\n",
" # 3. Perform optimization step\n",
" for param, g in zip(model.parameters(), grad):\n",
" param.grad = g\n",
" param.data -= lr * param.grad\n",
" \n",
" losses.append(loss.item())"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGgCAYAAAB45mdaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAABUGUlEQVR4nO3dd3xUVf438E+AEIohSkmhB1eQJmBAwQVEWRFQd139qeu6yrq6+7AqFmRdQVfsuIouVhCliIg1oCgoBE2hBDAhoRNaSCMhENJD6pznj5BhJrkzc+/MvXPuzHzer1eUmbnlnFu/99xTgoQQAkRERESStJKdACIiIgpsDEaIiIhIKgYjREREJBWDESIiIpKKwQgRERFJxWCEiIiIpGIwQkRERFIxGCEiIiKpGIwQERGRVAxGiIiISCpNwci8efMwatQohIaGIjw8HLfeeisyMjKczpOQkICgoKAWf4cOHfIo4UREROQf2miZODExEQ8//DBGjRqF+vp6PPPMM5g0aRIOHDiAjh07Op03IyMDnTp1sn7u1q2b6vVaLBacPHkSoaGhCAoK0pJkIiIikkQIgfLycnTv3h2tWjku/wjyZKC806dPIzw8HImJiRg/frziNAkJCbjuuutQXFyMiy++2K315ObmolevXu4mk4iIiCTKyclBz549Hf6uqWSkudLSUgBA586dXU47YsQIVFdXY9CgQXj22Wdx3XXXOZy2pqYGNTU11s9N8VJOTo5d6QoRERGZV1lZGXr16oXQ0FCn07kdjAghMHPmTIwdOxZDhgxxOF1UVBQWL16MmJgY1NTU4NNPP8XEiRORkJDgsDRl3rx5eOGFF1p836lTJwYjREREPsZVFQu3X9M8/PDDWLduHbZs2eK06EXJLbfcgqCgIKxdu1bx9+YlI02RVWlpKYMRIiIiH1FWVoawsDCX92+3mvbOmDEDa9euRXx8vOZABABGjx6NI0eOOPw9JCTEWgrC0hAiIiL/puk1jRACM2bMwJo1a5CQkIDo6Gi3VpqWloaoqCi35iUiIiL/oikYefjhh7Fq1Sp89913CA0NRUFBAQAgLCwM7du3BwDMnj0beXl5WLFiBQBgwYIF6Nu3LwYPHoza2lqsXLkSsbGxiI2N1TkrRERE5Is0BSMLFy4EAEyYMMHu+2XLluGvf/0rACA/Px/Z2dnW32prazFr1izk5eWhffv2GDx4MNatW4epU6d6lnIiIiLyCx71M+ItaivAEBERkXkYWoGViIiISC8MRoiIiEgqBiNEREQkFYMRIiIikorBCBEREUnFYISIiIik8mjUXl/3TWou9uWVYvKQSIzu10V2coiIiAJSQJeMJB4+jeXbTuDAyTLZSSEiIgpYAR2MNDF9r29ERER+LKCDkSDZCSAiIqLADkaa+ECP+ERERH4roIORIBaNEBERSRfQwQgRERHJF9DBCAtGiIiI5AvoYISIiIjkC+hgJOh8pRHWXyUiIpInoIMRIiIiki+gg5GmOiOC3Z4RERFJE9DBCBEREckX2MHI+aIR1hkhIiKSJ7CDESIiIpIuoIORoopaAMB78Uclp4SIiChwBXQwknj4NACgvLpeckqIiIgCV0AHI0RERCQfgxEiIiKSisEIERERScVghIiIiKRiMEJERERSMRghIiIiqRiMEBERkVQMRoiIiEgqBiNEREQkFYMRIiIikorBCBEREUnFYISIiIikYjBCREREUjEYISIiIqkYjBAREZFUDEaIiIhIKgYjREREJBWDESIiIpKKwQgRERFJxWCEiIiIpGIwQkRERFIxGCEiIiKpGIwQERGRVAxGiIiISCoGI0RERCQVgxEiIiKSisEIERERScVghIiIiKRiMEJERERSMRghIiIiqRiMEBERkVQMRoiIiEgqBiNEREQkFYMRIiIikorBCBEREUnFYISIiIik0hSMzJs3D6NGjUJoaCjCw8Nx6623IiMjw+V8iYmJiImJQbt27dCvXz8sWrTI7QQTERGRf9EUjCQmJuLhhx/G9u3bERcXh/r6ekyaNAmVlZUO58nMzMTUqVMxbtw4pKWlYc6cOXj00UcRGxvrceKJiIjI97XRMvFPP/1k93nZsmUIDw9Hamoqxo8frzjPokWL0Lt3byxYsAAAMHDgQKSkpGD+/Pm4/fbb3Us1ERER+Q2P6oyUlpYCADp37uxwmuTkZEyaNMnuuxtvvBEpKSmoq6tTnKempgZlZWV2f0REROSf3A5GhBCYOXMmxo4diyFDhjicrqCgABEREXbfRUREoL6+HmfOnFGcZ968eQgLC7P+9erVy91kEhERkcm5HYw88sgj2LNnDz7//HOX0wYFBdl9FkIoft9k9uzZKC0ttf7l5OS4m0wiIiIyOU11RprMmDEDa9euRVJSEnr27Ol02sjISBQUFNh9V1hYiDZt2qBLly6K84SEhCAkJMSdpBEREZGP0VQyIoTAI488gtWrV+OXX35BdHS0y3nGjBmDuLg4u+82btyIkSNHIjg4WFtqiYiIyO9oCkYefvhhrFy5EqtWrUJoaCgKCgpQUFCAc+fOWaeZPXs27rvvPuvn6dOnIysrCzNnzsTBgwexdOlSLFmyBLNmzdIvF0REROSzNAUjCxcuRGlpKSZMmICoqCjr35dffmmdJj8/H9nZ2dbP0dHRWL9+PRISEjB8+HC89NJLeOedd9isl4iIiABorDPSVPHUmeXLl7f47tprr8WuXbu0rIqIiIgCBMemOa+qtl52EoiIiAISg5HzLK4LfYiIiMgADEbOa+2gzxMiIiIyVkAHI9FdO8pOAhERUcAL6GBk5g39rf8W4HsaIiIiGQI6GLENP1Q0FCIiIiIDBHQw0mCxWP+98UCBkymJiIjIKAEdjNj6NDlLdhKIiIgCUkAHI9dfHmH9d8cQt8YMJCIiIg8FdDAS0iags09ERGQKAX03ZtciRERE8gV0MEJERETyBXQwEgQWjRAFOotFYO53+/BVSo7spBAFLNbaJKKAlnC4EJ+cb01358heklNDFJgCu2TEpmCEnZ4RBaaSqjrZSSAKeAEdjBAREZF8AR2MsMYIERGRfAEdjBAREZF8AR2MBNlUGuGovURERHIEdjAiOwFEREQU2MEIERFb0hHJF9DBCLuDJyIiki+ggxEiIj6UEMkX0MGIXQVWFtUSERFJEdDBCBEREcnHYISIiIikYjBCREREUjEYOY91RoiIiORgMHIea9QTBSY+iBDJx2DkPF6QiIiI5GAwQkQBjaWiRPIxGDmPA+URERHJwWCEiIiIpGIwQkRERFIxGCEiIiKpGIwQERGRVAxGiCigsVk/kXwMRs7jBYmIiEgOBiNEREQkFYMRIgpo7PSMSD4GI+fxLQ0REZEcDEbO48MRERGRHAxGzmPJCBERkRwMRogooLElHZF8DEaa8IJEREQkBYMRIiIikorBCBEREUnFYISIAhr7GSGSj8HIeWk5xbKTQEREFJAYjJxX18AarERERDIwGCEiIiKpGIwQUUBjPyNE8jEYISIiIqkYjBAREZFUDEZsPPp5GtbtyZedDCIiooDCYMTG2t0n8fCqXbKTQURexH5GiORjMEJERERSMRghIiIiqRiMEFFAY9NeIvk0ByNJSUm45ZZb0L17dwQFBeHbb791On1CQgKCgoJa/B06dMjdNBMREZEfaaN1hsrKSgwbNgz3338/br/9dtXzZWRkoFOnTtbP3bp107pqIiIi8kOag5EpU6ZgypQpmlcUHh6Oiy++WPN8RERE5N+8VmdkxIgRiIqKwsSJExEfH+902pqaGpSVldn9ERERkX8yPBiJiorC4sWLERsbi9WrV2PAgAGYOHEikpKSHM4zb948hIWFWf969epldDKJKECxnxEi+TS/ptFqwIABGDBggPXzmDFjkJOTg/nz52P8+PGK88yePRszZ860fi4rK2NAQkRE5KekNO0dPXo0jhw54vD3kJAQdOrUye6PiIiI/JOUYCQtLQ1RUVEyVk1EZIf9jBDJp/k1TUVFBY4ePWr9nJmZifT0dHTu3Bm9e/fG7NmzkZeXhxUrVgAAFixYgL59+2Lw4MGora3FypUrERsbi9jYWP1yQURERD5LczCSkpKC6667zvq5qW7HtGnTsHz5cuTn5yM7O9v6e21tLWbNmoW8vDy0b98egwcPxrp16zB16lQdkk9ERES+TnMwMmHCBAgn5ZrLly+3+/zUU0/hqaee0pwwIiIiCgwcm4aIiIikYjBCREREUjEY0ehoYQXeijuMsuo62UkhIh2w0zMi+Qzv9Mzf/O6tRABAbnEV3rpzuNzEEJHH2LSXSD6WjLgpPbtEdhKIiIj8AoMRIiIikorBCBEREUnFYISIiIikYjBCREREUjEYISIiIqkYjBBRQGM/I0TyMRghooDGfkaI5GMwQkRERFIxGCEiIiKpGIy4i++ZiYiIdMFgxF18z0xERKQLBiNEREQkFYMRBRU19bKTQEREFDAYjCioqWuQnQQi8hL2M0IkH4MRd/ECRuQX2M8IkXwMRhTw2kREROQ9DEbcxYiFiIhIFwxGiIiISCoGIwpuemczEg+fhhACD36SgufX7pedJCIiIr/FYETBqbIaTFu6E3vzSrHp4Cks33ai5USswEpERKQLBiNO1DWwYggREZHRGIw45SQYYZxCRESkCwYjREREJBWDESIiIpKKwYi7WIGViIhIFwxGyCssFsEBCImISBGDESc4ZoV+7vl4B4bM3YCcs1Wyk0JERCbDYMRdDFQ0ST5eBAD4Lj1PckqIiMhsGIwQERGRVAxG3MUKrG4JCuKGIyIiewEfjEweHCk7CeQDBCsQEREZJuCDkeA2Ab8JyIWKmnpcNz8Bc7/bJzspRER+KeDvxHxpQK58k5KDE0VV+CQ5S3ZSiIj8UsAHI86wYJ4AwMIDgYjIUAEfjLA+JRGRcVjfitQI+GCE54l3MfjzD+dqGxB34BTO1TbITgqZWINF4PfvbcXfV6TITgqZXMAHI+RdQayl4xf+HbsHf1+Rgllf73Y4jRACOWerTP9kbO7U+bb9J0uxN68UcQdOyU4KmRyDESLSbO3ukwCAdXvzHU7zQcIxjHs9Hv/9KcNbySKTMXkcSibCYMQJnkhE7ntjQ2MQsijxmN33pVV1MpLjEMvqiORjMBIg0nNK8MshFpWSXO/9cgTDXtyIL3/Nlp0U0lmDRZj+lRyZF4ORAHHr+1vxt+UpyCqqlJoOVmANbPM3HgYA/Dt2r+SUkJ5q6y0Y/3o87v5ou+ykkI9iMBJg8krOyU6Cz2EAReTcntwS5JWcw/bjZ2UnhXwUg5FAI7kU1Rfv6yx5JiIyFoMRJ/zx/af/5YiIXKlrsMhOApFTDEbc5ItP+ETUkr8H6CuST+CyZ35EfEahYevw921IxmMw4kSQk8oCep988RmFuHNRMk6ckVvB1Gisf0HkXc99tx8A8OiqNMkpIXKMwYgT3nxNc/+yX7HzxFk8/mW6oevxwzdPhmMA5d+4ez3HbUieYjBiMmcra2UnwVDsDp7I//AZhzzFYMRkhMGntdHL90csTSIiMhaDETf56vM9b6xERGQ2DEaIXGCdESIiYzEYccJZIQILGNzDGzsRETXHYISIAhofLIjkYzCikr/0xqqUCyEEXl1/ECu3Z3k9Pb7AT3Y9BToDSyV5jpCnNAcjSUlJuOWWW9C9e3cEBQXh22+/dTlPYmIiYmJi0K5dO/Tr1w+LFi1yJ62m4qtvG2yDqvoGC7YcOYMtR89gcdJxPPvtPokpI5LDV89lzRgwkIlpDkYqKysxbNgwvPfee6qmz8zMxNSpUzFu3DikpaVhzpw5ePTRRxEbG6s5sYHAm08YHyYdx1+W7MC9S3Z6b6U+SFY9l+LKWo4pQj6BdcHM7cSZSty9eDs2HzktOykOtdE6w5QpUzBlyhTV0y9atAi9e/fGggULAAADBw5ESkoK5s+fj9tvv13r6r3K34seY1Nzvb5OZ13s0wXZRVUY/0Y8Lo8MxU+Pj5edHCK3eHq678ouRnCrVhjaM0yfBAWox75Mx+6cEiQfL8KJ126SnRxFhtcZSU5OxqRJk+y+u/HGG5GSkoK6ujrFeWpqalBWVmb3J0OD5UI00jww8dU4xVfTHWh+3JcPADhUUC45JeQJi8VEZ5yEOiOePNCVVdfhtg+24Zb3tthdi0m7M+U1spPgkuHBSEFBASIiIuy+i4iIQH19Pc6cOaM4z7x58xAWFmb969Wrl9HJVPSXJTus/16UdExKGgwloZCC5SIUKPbllWLES3H4NPmE7KT4pGKboTHqLXxd6e+80pqmedF8UyVKR0X2s2fPRmlpqfUvJyfH8DS68vpPGTh2usL6Oa/4nMTUeEA4+DdRgDLqNHjyq90oPVeH/5wfNZeIHNNcZ0SryMhIFBQU2H1XWFiINm3aoEuXLorzhISEICQkxOikAdB2IXpmzV7rv2vqGamTcRgnElEgMbxkZMyYMYiLi7P7buPGjRg5ciSCg4ONXr2u6hr87BbBdyZEZCDbwm9P+mry98YE5EYwUlFRgfT0dKSnpwNobLqbnp6O7OxsAI2vWO677z7r9NOnT0dWVhZmzpyJgwcPYunSpViyZAlmzZqlTw48ZLb7sdEnnexRe9mYRp1A2Ez1DRYUVZi/Yh25j0EEqaU5GElJScGIESMwYsQIAMDMmTMxYsQIPPfccwCA/Px8a2ACANHR0Vi/fj0SEhIwfPhwvPTSS3jnnXdM36xXjZyzVS6nEULg+90ncdymvolMthcHGTe8QLjJkjq3L0pGzMubcPiU3BZDPCY9czC/DHd+mGzoOvgQ4/801xmZMGGC0+K25cuXt/ju2muvxa5du7Suyis8CdzHvR7vss32hv0FmPF5GgCYtn03mU8gPFDuzikBAKxJy8O/J18uNzHktqbrG5EnODaNwXZll8hOAhGRYc7VNshOAvmBgA9GjC7982T5iYdPS+klleQLpFJp1ivwX/YVWOWlg8zP8Ka9Zmf0+eHJ8qctbRwzZnjvi3Fpt4v0SY/kC0KgdQf/zJq92Jl5Ft/PGIt2wa1lJ4cUmPUeWVpVhwYh0LljW9lJcZvs6w35joAvGfEFp3XsypfXBu08CZ8+25GNI4UV1u7d1Qqk/RRg8akqFovAsBc34sqX4lBdZ/8apLKmHvcv24mvUrR1BsnNTGbGYMRgvADY88Ubjx6BAZ8QHeO2aanWZrTm5dtOYO53+1B//ruPN2ciPuM0nvpmj6ZlcjOTmQX8axot90ZPOu0xo0B7ZSKT1kOHe4aavPbjIQDA0J4X4/9ieqL0nPIAo/7Mzy69pCDgS0YC7Rj3t4CKyFO+Evh5+rpWSr9CthVYdVzutmNn7MYKI98X8CUjWrhTkrBqZ7briQKIr1z4bemRZhZCkV5k96KshV7PPrbnz+FT5fjzR40jqrPvJv8R8CUjRjpX24Dy6nrZybDjO5cx/6L1omw7+bKtmbjzw2RU1JjrWCLfYpZzPz2nBL+eOOv2/Afzyxz+9vnObMz9bh9LgJvxhYehgA9GjKwz4k4XyUafRLK7g/eJs6IZ2Ze1F74/gJ2ZZ7F8a6bklBhD9pO+7P2rVZAOZ67FImCxeD/nDRaBW9/fijsWJaO0Sv+6L7NX78UnyVnYerRI92X7Ml+IzQI+GDFyH+3NKzVw6eRLtMZgSpOfq5PX0+WZihp8lZKjqbdNH4w7/VrT7hBC4A/vb8Uf3t/q9RKEesuFVkJnq2pVz2f3EKXiwCqvDrxKvr6OdUY00FJnRMZTB5mXLzyZOPOnxdtxtLAC6TklePWPQ2Unx+85O148LUk6U1FrfVA6W1mLLheFeLQ8Z+x7YPXeSeDjp5vh9uWVIjw0BOGd2slOilXAl4xo4epk+ijpOGZ+lQ6LReC73XmGpqW8ug53LNqGJVucF91X1NTz/amHZDzgm22PHS1sbLmwcX+B5JSQnrx9nPFSJIdtUHjkVDlufncLrnr1Z3kJUsCSER29sv4gAODW4T2QdPiMrsuuqKnHRSEXdtfyrSfw64li/HqiGA+MjVacJ+XEWfzfomT8aVQvm2/lXg1Ycu/beDPxDmeFsHrUGfEWHi/msyu7WHYSFLFkxABVtQ26Xi5eWXcQQ+ZuQOLh0xfWoaL+wNs/HwEAfPHrhW6j7d+9Ks+XUVCOl384gLOV6t/p+jMZ11Pfud3oQPZ4SXJXH1D0qEfE/eWfGIxooL7OiL5X16b3u/POl7wAxp6QNy5IwsdbMvHst3t1XzYrNZrTudoGnDhTqesyuavdZ2SdEb3PQTO+BjZhkkzDrCVrDEY0UDrprn51E+atP2gdN8LKzf1t9Dn03Nr9mLNGXZCxL89xe/5AomZXFlfWthjQzBPevpbeuCAJE+YnyCnCNee1kdxke51sHvgwSCBHGIx46FRZDT5MOo6lNn1AeHrCpWadxeQFSS2+12MsmdPlNVi1IxtnKvQbCTjQFVXUYMRLcRg9z1wVwrTIPlsFAFi3x/nowloObdXHq043qLLqOkx5ezPeOf96Ui0z3h+NqDNi1FhUapdrxu1M5sFgRANnJ926vfatDDwpCrvrw+04VFDu9vxqWCxCSnGdWYsIPfHricbShBInnTid1hj8+d9WMt6KbSdwML8Mb8Udlp0UQ7n7msao1ynOltv8J3fjITV13ci3MRjRwNlJ5875UVJVi0+TT6C4WUXRehV9lKg5IVkkah5NI6+anS8fM7UNxib+qW92419f7zZ0HU18eT844618ye7VV28Wi8Cpsmp9FmbSYI5Ne3XiTrT+8Kpd2Hq0CD+4KBp3lxlPSP98qtF/O5tvzwW2kqpafJWSCwB4esrlhnYU5ogR547MoMfIS4G/BXNPfr0ba9Ly8ME9V2Lq0CjZyTEES0Z00vzEUnPhaBo/YUfmhUGj8ksdR79+eR8nn6KlqN+fjtcGm9JKb3SurHT98PQG6606I85eqdj1yGpIavzTmrTGTjS11ofyJQxGNHB2Mtv+xpOMfJWM0jTZ54s/BU0yqA1QhfC/Egtf4Qsl0gxGNNBSZ8Tofe9xRdAg3zhA9bTt6BnM/W6fpsHeZJG1a/S8WQTa8aUnI27aTdcvb+4WZ/nQNGK69JDVHNwt3ZI+WrsKrDOiE/sBoYxfh6+SmYU/f7wDAHBxh7Z44ob+ElPiWiBdeqWM/SOE9cLuK9vaH85/b/GVfUoXsGREJ81LKvzhwmFEHsywXXKKq2QngWx4+8axelcuRr68ybRjdDii9JAjhFD9mkR2PyPNubvf/bF7AKPUN1jw7Ld7kVt8TnZSXGIwohcvnx9s2uu+5hezLUfO4E+Lk3H8dIVby1O7nb9OyXE9ERlu5le7UVRZi3+uTJWdFI898EkKbn1/q10FW29rHgzZfjLDw4c/0bo5v0nNxcrt2fbLMOlOYTCiE+++hxV495ejKqbzQmL8wF+W7MD242cx4/M0Q9fzr2/2qJ5WXp0R5weNph5YvZ0LjQd88xEczERtHYlfDhVid24pDp8ytpNEPTTPk6Y6I7yWuaWw3HVniyVVtaYYX4jBiE5aNGEz4ELctA6je2dtklVUharaeq+sywzc7SJf6UFj3vqD+J8HPYHKvzSQ2TjtIt7tsbD880gzw83VF2zcX4DhL8bhhe8PyE4KgxG92AYfy7Zm4ksDi+Rr6733SPf6Txm6Ls92O70ffxT/WJHi9SJmT0opDxWUYf6GDJRXX+j6vfl1L7/0HD5MOo63fbBPAJ++hJu0+FkvzlummDPvzdOsR4zg57vZq5p6hl6+7YTchICtaXRje4KkZPlWxThn0gys5PfGhsZA55dDhbhhUIRh69HT5AWbAQDFVbV45Y9DFafxZrCoN1c3CyObnJL3HSusRHhoO12X2bg/GTEYQY9AzKx7JuBLRlp5cc+s3X3So/m1Pv2YsghWIQs19ebv96O55ONFhi7fyMOyvLoO1XVe2Obevuq5GdSY8eKslBUjSgTu/mg7ftxrzHAUgH5pNuGVzKnCsmrkstWeJgEfjPzrxgG6LKeootblNI9+noY6HWrN6XWCu1Oruqq2HpsOnPKJjsMc0WPzWQx+tWTU0qtq6zH0+Y244vmNBq1BO3m1+4XNf32bJ5tw1c5s1xPpQAhtD0hmffXkihACV736M8b+Nx4VNYFT585TAR+M9LykA9qoLB7ZlV3i8LcMlbXZZTbBs3Wq1L3KmrO+3o0HV6Rg9mrnLUOEEEg+VtRiRGKlLW3UReeXQ6fw0GepKKlyHShq5az7fzNfRI8VVgIAah0ExXqWpqndCt58TePLFbL12kxmadppEQIPfZaK//6kPKK1o2PRHKlXp6BU3/49TLLrDBHwwYi3efP1uLN1/f79LW4tc/3eAgDAt+nOXzl9vycfd3+0HTf8L8nlMo06wf62PAXr9xZY66a4S+niHaj1HEydbxUH0oxVxjbf9gXe3IfOVpWWXYL1ewuwMOGY19JDzU4TEwU3DEbg3WjT4sGFQM90qk2Gu6ndsL8xaDlTUePy4mf05j9VZj8Sst7724j0G7VNXOXdzLGGHn4+VOj098OnyvFp8gnpJZhKa/eVp2K1W85R6Zyv8/dzyChsTeNlnhynPnItckpGEbGnFwc9OwJTvU6N06/fm49Ficfw7t0j0KdLRwNS5AN0uAtMairJCwrCvaP7eLw8IyhWblUxn4xgpvk63d1FWktztK7HdqwiMzPzK2BPsWQE3n514v7KdueWNp40Oh2Q3jqsXWXZ29cAfzyhH/psF/bkluLp2L0eLUfPU0Htfj1+uhJvxR1G6bk61xN7yZ6cEtlJ0F3TeejrD+6Ojquzle7VDZv5ZTpuXJDkk6363GHWmIvBiJd5WvqrZXAvPS463gnUjD07PM2CEU9MQgj8ddlO3L9sp67v8MtrPLuhu+xnxKOlK/v5UCHe+fkIXli734Clu2/zkdO45+PtyCqqlJ0UO2a9mcgkhMCVL8W5Ne/qtDwcPlWBxIzT+qRFl6U0mrNmL+75eLuOSzQvvqbxMk9vPNV1FrRt3Vqn1HiH7Ccxb1e6VHOzKKqsRcL5i19JVR0u6dhWn3VrCOzMVjSd6mkHexrz0nRYOJrr3iU7AQCPf5mOj+8b6UHCtNN6zKrJetM0Rp8Otstv0QOrbXqMTYZfWLXDO82uzYAlI15mkpa9unCnmaRi096gC8vzRuCg9/3XnSTbXbBVzlOv88Fjtop2stKjtFrb706X10gPqG25u51KqszzGkwrV1luGfSYaY/px0TPDrpjMALg4g76PJWq4UlrGqDxpFN9QBpcV2Pl9ix1yXCRZ4tF4PCpcgx6bgMe+yLds0Qprd/T+Q24S7qz7T9MPI7Zqz2rE+J6vXr2M+LlHoNd7Kd3PRgryD549M8bnRGcHW+utqLj48fcd2R3rhfZRVU4mF9mQGpaMmudOQYjAAZGhXptXRYhYLEIj25wZnmirVLZC6td0azCefDPz3ZZWzGo6TI/t7hKsQfUugYLPtuRhcwzxr/jl7ULPnfRW6aWIMetPBiYcaOP6zc9GEXZU7uyi3G0sMLANWh4PSfp6BXC/kbtKsVa0qlYuiUEsouqzN03znnj34jHlLc3uz1yuD9gMALv3tzrGwSmvL0Zt76/1Xsr1aB5jXK3L1wGbdOvUnIw9r/x+Hdsyx5gl23NxDNr9uG6+Qn2SWmWFkc3bEfHgdH1Ksx0sZSZFBNtBufF/grprK23ON2Pt32wDb97K1H9+p385s9F9XqavzED49+IVz16thnqT+WcdT6ejd4plJ/jCxiMeFnO2SpknCrH7txS2UlRtGCT/sPe63mT+d/5p9uvU3Ntli/w+k+H8Op6R91Kq1NYXoNX1x9EYbNO0gqafdaDYZ2aufzdpht7hR1jpoDAaGqzKgSQfMzxwIil5+ow/MWNmLbsV30SpiI9ni1Al2Q4WfyFFThLq5ZkuMrz1yk5LaZ/P76xZ1cjrmlqpeeUot5PO3fTG4MRAF0u8l6dkQZvdsXs4nTff7LlO8qf9hWoX77TxzcHXxtwF07NKsYHOnUpvTjpOB5etcvuO6XuqvXMxuS3N2Nfnr7BaVl1HX7cm+90dF5f7uXTm/JKztnVZWq+3eIOnEJVbQOSDuvTNJQaOarboHSMPu1hXSo92R4fs77e7fAhSYnL0hkdTlCznuMMRgA8e9Mgr63L495AIVQfTO618rCfaV9emcPi57d/PoLXHQxyZV+q7Vmmz1TUOB0l2FUrgZbpd74BU7M8bGKqgu1F53R5Df6q81P19E9T8c/PduGF7+X13aH1omeq11Va6iuYIN1mvcG06IHV9jcX8/pLReGlWzNlJ8EnMBgB0C00xGvrkj3mhStKqduX57iWt9YSiaanHbUX8MKyaox8eROufnWTpvUYTe+9WKzXyMLnr/7bzr9W+Col1+GkJriH2jFZchzyZLs1H8XaiHW4XLZxi3axXl/ZwyQDgxEvM/vpqHQRdNVNcvOB6ADYPfYoLVPtxXZH5lkAQFl1ffPFXliWukWZWlOQunF/AWJTHQcQnhBC4EhhudNpXA6kp2F9Wh/WzRYcGWHES3GorTd/HQKl1mpqOdqPG/efcjjPvPUH8X78UbfXqTYN3uDJut0p4Eo8fBrXv5nglRJdIzEY8TIjinRvX7gN3+h0A3Pn6eXqV3/GzwebXWg86Fb8ue/2YdbXuzWnw+G6VLamcXv5bmwzR0n4x6epePLr3SgoVV9ptkRlqcrnO3Oa1X1ome4DJ8swZ81eFJbrX2nXpziteCmafdamvNq9zsecHbZ6HNLl1XW468NkvPvzEVw972fMXt2yxZonHv8yHXtylOtGfZh0HG9syHAcBNl8Lavyt1lNW7oTx09X4i8f75CdFI8wGPEyIwL21KxizPp6t0dPM02UYqUN+wtcNjn7MOm4pvU46/xtRXIWvknNRV7JOX3G1zFgq3saUyoFRLaBapmGQeMWJl54VdZ8sbbLXLLleLPfWi5rd24pVu3IxlPf6HsjUsNXivE9r/dl/Hx1DRbc8u4WPPlVy6DeUfqXbMnEjsyzeDPuME6X1+DznTnKE3rg8CnnJXO2zNo5l0zOtsg5J5XVHS7PRJWNGIx42TEPOz5ydiH8yxL7yDjFjWI7peV/tDkT416Pd/6+20kdUdtlahkfw2IRug1eZYRztQ04flq/jqzstomGa0RN3YVifyFEi6bJ7jiUr/6moRdvF63rVkppRLo9XOb240XYm1eK2F3qS0zVdmJoJGH3b/c3gpqertfvzcd9S3e6nK64slbTsaIl3UUVNXj22wstgbzQmMa0GIx42cvrDno0f1ZRJT7arFwKsc2mL4Q0TwcdU+CsZcbOE2exdItNrXGXr2nUnbBaLqYO19X8NY3HS2xM/40LknD9m4kt3tW6e8Gwj0XcW8ju3FJc9erP7iXALi3K+0fLRVnrU5eZykWcpcXjbj4MyGjzbe2soryjfeuN+5zdMW7gCv+lomTvoc92uWyOveN4EUa8FIcZn6fplTQ7//luH1ZuN24wPOWxwMwZ0TAY8TH/+W4/vkt33WX6PoU+RNRwdrM56qIU4MUfDigvU+Hip8cF2VlzX73XpbTM7POvrtbvzdc8v1KwYfs0543rhdkqjJotPUZx94nfnLcQbVwFs45+t91m3ryZLjr/CvSHPdrPcTUyCowtgfSlU4rBCNnR7eB11SpDxYpeXqcc3DRR282zGaXnlrT4zu51lk7rkXkxss2DupYS6lO7IvlEy0rTXtL8humtui6y9uW6Pfm4d8kOFHlh3BQ9Sgdl0xJUm7WUQgYGI2RHt6dT207PlJr2qri0bnDSFBAA9p903Gup7Q2j+bp0b03jxjZbo/D6yf7pz9n6hOpSIWec7QO9Syne2JChaXohBPJLzyn+ti+vFM99tx8PfJKiR9Icrl+WV9YdwH3LXNdlcMZpz7sas/bwql3YfOQMXvtRfU+i7pI5OrIZ4gJXAZjWJJogS6oxGCE7zi4AejypND0J6HmtV7pxvPTDQZvf9VuX0jKbv55X87SjlCT7khHHy7hv6U4MfO4nTc1/6xssOHZa/WjGjjaZkbcH2/y/vO4gxsz7BZ9uz2oxnV7Njt3NS2G5sSUEH23OxO6ckhbfa2na69Z4LC4O2yKVHbZ5QmsAYobebz1hdLCgtHXMGqAwGPFT7h5wunUQa9uaRnE9xjRDbmL2LphddQTnLJ7ZfOQMAOC79DzV6/sypWUzTW9fx2d+le70d9tjYsn5ytCvKlT4ll18/+yafXafNW9HB9OfLDmHH/a4rg/mzOYjp3HgZBkOGVAXQe05a3Twb6TKmnrlThwNIrs0xkyBSRvZCSBjLHPzZmzEaxpbQc5/dsmdEXQFgNSsszZpcFEUKukKobUCq5pt2LTI3OKWrzycthhx8KOm5p/N8rB6l/rgSTZn2yan2HmfO+4u+9o34lHX4KwVjHOZZypx7xLXr3fcPfcSMk6jvLoOoe2C3VyCa2pKB22/FULfG/qIF+NQ22DBzjkTEd6pnX4LVsl1017XmV2yJROnyqoxZ+pAl9MaXcqnhVslIx988AGio6PRrl07xMTEYPPmzQ6nTUhIQFBQUIu/Q4eMf/8YyLQUydszooMwfVrT/LQv367kRm2R7r68Uty+MFn1etQU/XpaPKz4msbuk74BkTfDq4+SjmPTAe2VSxWLlBXbJrpe1uYjp+0CUF2p3PVaXzk4C0RcCQqCrn3eOPLC984rlfuyoCCgtqGxz55dBnSN4C0v/XAAi5OO42B+mctTpVRD54pG01wy8uWXX+Lxxx/HBx98gN/+9rf48MMPMWXKFBw4cAC9e/d2OF9GRgY6depk/dytWzf3UkyGcnaPPaOlNr2D1zQXOj3TfuF9+2f7FhlqF2F8Z076BHDeb9qrf+CZmlWMV9Y3vloJbaft8qJncppKCE68dpN+C3XArqMuIfBh0nFcFn4RDp/SNzjQpX8cIVq2BhJNy3e9hpQTngV4rvax1gqsZqwx4slxXK/jQKqVNfXqA3wT0Fwy8tZbb+GBBx7Agw8+iIEDB2LBggXo1asXFi5c6HS+8PBwREZGWv9at27tdqLJOM5OhXwNFSZddnrmxjmn181Tj5PR846vnJcW6X29UOx+3ukc7uXQk95flbZJEIDHvkjDLe9uQd35p1bZ11JnW+bHfQV47cdDTlv6GNLpmZe2iuymqN6rsOql7dlsPSu2nXAxvXomHyC+BU3BSG1tLVJTUzFp0iS77ydNmoRt27Y5nXfEiBGIiorCxIkTER8f73TampoalJWV2f2Rd5z1sMa80sVi6/kKl3bTubHsvGb1HvS4Lrkac8cRI66JW2y2k+ynF3fzZ5tuvbLwXfpJ7M0rxa/nR3DWu38ZrTc4Z9M/9NkuT5PjeL2ufleRDaVpPD3Wmi/SaZNxlyGw8u8CjSP7jp73M87aDAxpXHAibP5rnObb3rYXbU8JIaQH7lpoCkbOnDmDhoYGRERE2H0fERGBgoICxXmioqKwePFixMbGYvXq1RgwYAAmTpyIpKQkh+uZN28ewsLCrH+9evXSksyA5u7NVS8rks83xbQ5C/5pc4FuehJwpzVNeU294veeXDD++Vlqi+80d2PuTimPwncPr2q5nfSitDxn6fag9oL1X2XVyvtLyzpt90XD+QSnZZe4k7CW6xONF2ylrsONfAC3CIGnvtmNjx0M62AkgZZ509RJl45pcXVMNv/9w6TjOFVWgyWbL1TOf31DhtOu72Uwy4CPAuZ8jeWIW61pml+shRAOL+ADBgzAgAEDrJ/HjBmDnJwczJ8/H+PHj1ecZ/bs2Zg5c6b1c1lZGQMSlZKPFaFX5w7S1j937X7cObKXIa9pWixDh261s860DN5UVWB12mGY+S4BWp9+3c2DR0/ZLlbp6qbjTop3ZHpeyVXrptp2rAhfpTR2evfguH6q53O1aT096tzdd0qzfbLtBLYdO4N/jHecP8VhIlz83tzipOMYFNXJ5XRavfD9Adw4ONLrJQt6log6euCT3TTeEU3BSNeuXdG6desWpSCFhYUtSkucGT16NFauXOnw95CQEISEhGhJGpnIchfvPQHzPD24e17aV7TTn54Xpdp6i/IPThIu42FTj9cQLedx/KAEOB7fyGmz5xaftSWs0kEJnxpVtfX4VkP/Ms0ZESgrNbefu7ZxUM2IZs1jy845z7s76dNUl80J25t0fmk19uWZr3qAlutCfkm1g4HydEuOrjS9pmnbti1iYmIQFxdn931cXByuueYa1ctJS0tDVFSUllWTSma4yZdV1+kyNo0reizDiFEtVdW7MSjtSsa/rlxHy9mx4m6ndB4VjLhYp6uSEZNeY3X1/Nr9KKlSbo6prkRPf813i20yKpoFXu/ZjFGk5QndNm/euplW1roXNJplbJqX1x0wwd1APc2vaWbOnIl7770XI0eOxJgxY7B48WJkZ2dj+vTpABpfseTl5WHFihUAgAULFqBv374YPHgwamtrsXLlSsTGxiI2NlbfnHjonbtH4I0Nh5BzVnk8DNLIwVlwoWmvYatwyfYCoPfFID6jEPcv+1XXZXrKrY7iTFIyYnszsyg0S3W5TGGOsYhseZKeb1JbjmkEePEhxOBAwK4bAB1XVnquDmHttXfWZngFVhefPWF8lwb60hyM3HXXXSgqKsKLL76I/Px8DBkyBOvXr0efPn0AAPn5+cjOzrZOX1tbi1mzZiEvLw/t27fH4MGDsW7dOkydOlW/XOjg98O64/fDuqPv0+tkJ8UjTZXyzGpvXimuvzxc14unJ9lt5eTszygoR58uyvVvHK3yHZUtPVy2KtB5Fypl02kFVjcT0MrAJz09hhCw5SyPWvLv8DWYA55sIU9fn+3KKsZ/vtun+JsZSpaMunRd9comZLw8xZiFe5HWAM0M+1QttyqwPvTQQ3jooYcUf1u+fLnd56eeegpPPfWUO6shN8mOReIOnMLRQuUOnxYmHENqVjH+d9dwz1ekx6sOhZunEAIb9xfgH5+mYnB35cpx9gPlaU+Iy86fJHf45O5Nz5NYxNU2abBobwHkMhtupLf5vtHcK6nNRrr1/a24d3QfjLusq/aEtEiXa3ct3t7iuy1Hz6gefNDwm9v5TBRX1uJf3+y2fp2jMJyBFjX1FiQePo3kY0WYNak/2rQ2blg2szwKmrVuiCMcKM/PCOj/BKmVo0Ckyc7Ms6YpvXFUMtJUHL7/pOtKbD7RiZVS0OVk8nN1DcgvVb4BaBmgTwsBgcqaerwff1Tx9waTHDN6JiM9pwRPfr0buSWub7ZG5v7md7YYuHTtXvh+v11Lpy9/bTnQo1bTlu7EosRj+GxHtuuJJbF9OPLGsBRmwmDED5ms2b2iDxKOebwMt5v22t2XW96k1dQjcdQ1vtprgx4lIwBQ12Bx+4Lkar7fv7dV8fvHvkh3OI+nJSOv/XgIb2zIUPz90c/T8POhQsfrdn/VLdOi47Kac/Zq0F2Nr2fdn1/tgGlqzg1Ptl3TcZ9ZZFx/SXtyS1VPa3ThQvPNWW9x8cqv2fTOevpVM9CgmTAY8TNCmKNFjSurTPJ04jCo8HI6WqxfRQJKquow7IWN+KSpozkn3LkAnS6vweKkY157+hIAfnUx9snfV9hffF2lzWXaHfyceqJY6yyqae2ATss0ntASSBZV1KC6zrMKkor9jHjhUNNScuxOcpSOuT9+sFXx2G6+zU+V1WDt7pOq1/WLs+DcrFGHAwxG/IyAkF5nxFvMkk93Rr7UI2D8fvdJ1TXmtY9N0+jV9YcQn+H4gtdiPZ48d7nVj4j7q3PGnRZIarl7kzD6IUNtS8I9uSWIeXkTrn71Z8Xf/63Qo61aTTncn6e+9MIXpGWX4I5FF0YOd9S/DdBYAgg4PrZPl9fgqW92Y3dOSYvffPm1DYMRPyOE/Doj3pKSVazpRtmk6YZZXu394bPnO3gFIYPaw0RTc3ePYhE3KgI7+Lez7zxmwoUKHZah5olcCGF9fVd6rg57FV55rNubr2p9yiVEjXnQMnrtKY2Bo8MbtoclCRaVaf7y12wMfO4nvLUxQ3PHaunZJZi9eg++SsnFH95v+RrVNmv9unXUtGzZGIz4IV+oM6KXxj493Mtw0xNIcw0W4VZphxpNnT55O15UuvB/su0EJrwRj892uH7No3493uXJk6Cs08TdbVSgU0+jnmh+bbnlPecVX52VlOlV0qOmx2ejFVXUYNQrm/Dcd/tc5urfsXsBAO/8olxRG3B8XNc2WFw2EGhyw8BIxe/N+vqGwYifMUNrGl8Rn3Ha4W87dRizxBPe2IPvxR/FiaIqPLNGud+JJlkGVia0pdeAg7Z2HD+LY6eVL97l1fU4Uljuxjo92zuKr8xULNJVgGw3yJxBNxyti3U+hpPS9I6X1ORQgfZ95i61+f1k2wkUVdZeGCjUQzszz7pXX8Xu38pb31WX/LK41c8ImZvQ1gdTwDHDk4GrC43e8eSG/cqjaquxdGum6mk96dHWvcqCjf+vb7Ao3jj+smQHAODEazcpzv/q+kNurNUzTU/GRgqCQQGtjufOj/taHpNme46SlRytI14rEQKK++upWPfr9BiJJSPNbHxCeSRhnyEES0aohQP53hn0y8ixaRTngUDKibO47Nkf8WHScQ/WrmGdBpxeni5SCPun4JA2rT1covESD7csmfSFloCuGDVmlrp1X1j52z8fwRGVr3TMgMFIM5eFXyQ7CR5jMCKPq8HcmvjrLvKk1MkiXI+n0S7Y/pIlBPDv2D0QorGXTW+Qteu0HDP3/7avYenwdUbsP72XGRSkT8uY7zU0E5aNr2n8TL1FoNjBqJ7+6nSFilFybSzZkomuF4UYkpa9qpskunxR42lSvK6uweJxz7HZZ53XT6muM8c7yBNnKvGvb3bjoQm/0WV5qvoZ0bC80HbaB4UzgubjwUEmvRG8a60nlnO2Cjszz+IPw7vrnpbPd2bjzY2HFX874aQOl+9dNS5gMOJnNI+T4QfcqWz635+8X1dAC18rOYlNzcWTX+/GvaP7eHW9UkYXFgKPf5mO9JwS3L/cHCM0C3inAqvRZB72CxV6hbbdjPvySjGkR5j187jX4wEApytq8NFmm1eEOmRi00HtXRYAvnfdsMXXNM3oPaQ8Gc+XO/rxB9V1DXjy68ZBzT7drl8zYTVk1TEoqlTXfbpaL/3g+iHCDMe52teQ7nLYBYiky7JtclIc9A782o+HTFNi58sYjBB52cebj/v0E0xzmw6ekrZuKSUj0NgRnApqXu+5frF3YQqj7t1GN/N2FFx+vtPzgfK0Kq+ub1bapG6rZhZVGpQi18okdOSoFwYjCrxd1EyB5eV1B11Oo+Yma5aKyjKTYY4t4B3r9jju2bT5PvjBybRabDt6Rpfl+KK/r0hRbPHjyq0KPaN6y5sbzdPDs1YMRhS8dOsQzLttqOxkkEq+eENyNvosoO71Q74JeuQEgPxSfUsJtJDx6sIkMaBT6itSO+dpB2NaX694Y9v6wv5zV26xvHPRUwxGyOdt9KBDL/KcN3vEbM6P7yuaGXGT9XZdDW/tz5p6z0YcPlTgnX57Aglb0zjAaqy+o67B/25JnjaRDRT+/JRrBp4ehVr3jzdKutbuPqlqUEBHthw5Y+3Z12w2H/Hd12oMRohMyB96ovQKbqbzzHnExO7K1TT9ml15uCq6s0Gp8UxTKZEvdSTmSxiMEJmQLz3xyyzFUTtcPbknv8y79ZLejFPu6MssftqXjy0BXKnXSKwzQkQeyS32zqi+SuasMX7QOV9hxCuODxO9M96PLyitqsP0lbuQV+K7lUTNjMGIA+z7jEidHW70gOtt1XWeVVg0O18qSfNVVX5+DMnGYMSB3/6mq+wkUADjvUVfCzYdkZ0EQ93wvyQUaRyjichMGIw40POSDtgxZ6LsZFCA4pOuvrYc1d55la95P/6o7CT4NZ6TxmIw4kREp3ayk0BEOtC7+3Yzqmvg+CjkPqPHHXKFwQgR+b3Sc747ZodafHA3ljkbT+vnX9/slrp+BiNEJiSzi3UiasnfX9Os3pUndf0MRohMaNuxItlJIF/j5zdL8m8MRoiIiFyQMShjIGEwQkQ+7djpStlJICIPMRghIvIDfG43VkmV55Wg4w6c0iEl/onBCBERkQtfp2ob9E/J31ek6JAS/8RghIiIiKRiMKJSRKcQ2UkgIiLySwxGiIiISCoGIyoJAfy/8f1kJ4OISFFFTb3sJBC5jcGIBm3bcHMRERHpjXdXDdjnDRER+SuZg+UxGFFJQN+BktoHt0aPi9vrtjwiIiJPHMwvk7ZuBiMuXHNpFwDA3Vf11nW53S9uh5uviNJ1mURERL6ojewEmN3H00YiPbsEV0V3xltxh3Vddr3EIjEiIiJbrYKCpK2bwYgLHdq2wTW/6QpA/+6WLayEQkREJtFK4rsSvqaRyMKSESIiMgmZJSMMRjS4d3QfNO2rsedLSzzB1zRERGQW8kIRvqbRpPvF7ZHx0hQEtw7Cqp3Z2HL0jEfL42saIiIyiyCWjPiOtm1aISgoyGmfIwOjOqlalsw23URERLYkxiIMRtwlnEQjjvbn7waG231usOiYICIiIg/ILKxnMOImZ/U9lKLL3c9Nwkf3jbT7rsHCaISIiMwht7hK2roZjLjJ2RuW5sHIJ3+7CmEdglu8j2vgWxoiIjKJ1KxiaetmMOImZ81yg5q9qLm2fzfNywCAiE4h2hNmgN8NjJCdBCIiMhgrsPogZ69pIsPaqVqGqwqsMtt825JZdEdERN7RihVYfY+zZrkv3zpE1TIaXNQWkhmKjLvsQj8qNw6OlJgSIiLyhtYsGfE9zl6xRHRqh/f+PAJt27TC4ntj7H67b0wfAMBTky9Hu+DWTtehtshM70H8gMZRhZvcMIivaYiI/F0riUUj7PTMTa56T735iu6YMiQKrZvt3Bd+PxhP/K4/LunYFllFlfh+90mP0/LguGh8vjPb4+XYso2DglszZiUi8nfsZ8QHdb2oreL3m2Zea/1380AEaCztuKRj47z3/zYaNw2N8jgtoe30jylt66v07txB9+UTEZG58DWND7prVG/cNbIXPrjnSut3118ejt+EX6R6GcGtW2HmpP4Of597yyCXyxjSoxM6tQtWvU4AuHV4d5fT2BbXtW/bGsN6XaxpHURE5FtklozwNY2b2rZphf/+3xV2310WoT4QaeKo7sm+F27ERSGud8+ah36ruaKrO823Lm6vLeAhIiLfcllEqLR1s2REB989/FtMv/ZSPDbxMs3z2raoiex0oUmwo0Bkw+Pj8ccRPQAAKx+4GsGtW6FN61a4/cqeDtcxqu8ldp9H9L7YZbqahyt//W1fu8+bZo53uQyj/DBjrLR1ExH5qwkO+sTyBgYjOhjW62I8PeVydGirvaDJtq+RLf++Dq/8cQh+fvJah9MPiAzFW3cOQ9p/bsBYm+a3d4x0HIz8YXgP67+/+MdojOzT2WW6mpeeXDcg3O610W/CQzHx8vDms3nFkB5hUtZ7Rc8w/GlUL6+s6/9iHO9PIiIjsNOzAHZ5ZCcMjOqEcZd1RZvWrXDP1X1waTfnr3tsK8E2uTracYBh20x3VN/OGBgViv+L6YkHx0ajT5cOuCOmJ9oFuz4UosLa231edG8M4p4Yj39PvhzXXNoF947uY/3Ntq8V2z5L9PLA2Gj0d+O1mCeG9gjDa7df4XRb62X+HcMMX4e3RDnoBHCAxCJhIjIXt4KRDz74ANHR0WjXrh1iYmKwefNmp9MnJiYiJiYG7dq1Q79+/bBo0SK3EuuPWrcKwroZY7Hib1cp/m77+uWLf4x2uJygoCDsfX6S3XePTbwMt1/Z064796Dz086/YxievXkQEmZNwBt3DMPmp67H01Mud5rWCQO6IbprR9wyrLECbHDrVrgsIhT/nHApVv19NDrbBEh/tun7RKk7/PWPjsMH91yJwd07oUtH5ZZJSm47/4rqPzcPwsYnrnXakkjv/lGagjqlHnb/oKJSsBFuu7KH64kkc9S3X58u/ttKy9daoOnVIm/77ImK31+p4tUwyXXzFZ637PSE5mDkyy+/xOOPP45nnnkGaWlpGDduHKZMmYLsbOV+LjIzMzF16lSMGzcOaWlpmDNnDh599FHExsZ6nHh/0apVkMPisVf+OAQf3zcSB1+cjNH9ujhdTmi7YLvmxE/c0B9v3jkMYR2Cseaha7Du0bEtOrVpWm+30BBrXRQAGK7QeqZdcGv8PPNavHv3CMX1PzAuGgMiQvGP8f3s1tO82/seF7fHoO6dMHVoFNY9Og6v3X5F80VZ/fLktRjWs/G1zJh+XfBSs95tR/S2rw8T3PrCehffG4M1D12D/S/caP3u6+ljnLYm6nlJexx+eYpiR3Ktzy/7mZsGtvit20X6jSM0y0kLK1sXhbRBr0uc3/Sav0rrFuo6nasevNrp70M1viYTEBjSo1OL75sHdVOHquvpt3fnDk4Dc2+beUPL/XWVAaVnl2loqadV3y4dXU7TWcVDQ2RYOzzfrBXg5MGRWPrXUW6nzV2XR7LkzZVONkHoPydcKjElbrSmeeutt/DAAw/gwQcfBAAsWLAAGzZswMKFCzFv3rwW0y9atAi9e/fGggULAAADBw5ESkoK5s+fj9tvv92z1AeAdsGt8TsNT/hThkTihz35GNzd/uLf/KatJKJTOyy+NwYXhbTBqOjOaB0E/PY39q9YnPXQ16ldMDY8caFi67QxfbBh/yn8aVRvTBkShdhduSg9V4cHxkbbzXfDoAhsnz0RK5JPYNuxItTUW3AwvwzP3jQQ/bpdhO8eGYvqugbFHmvfvGMYbn53M06V1eD6y8Px7E0D0SooCBe1a4OgoCBrvnfMmYjss1UY1bczRvXtjAV/GoGiihrE7spFh7ZtsP9kKT7fmYP/d+2laNumFebdNhSxqbmobbC0WKdS4NG2TSv8MGMsbn53C4DGICruwCnM+/FQi2mPvToVyceKMLRHGJ5evQc/7isAAPTr1hEL74mxvn5aeM+V+Odnu+zmfe7mQXjxhwMAgLiZ4/FtWstO8+6+qhc+35mDsPbBWPCn4Rj6/Ebrb1v+fR0OF1QgPacYfxndB//6Zg++Sc21m3/MpcpB7+TBkfjz1b0x7rKueOmHg1i6NdP62xf/GI2ITu1w3fyEFvPNvWUwhve6GIuTjmP5thPW7x+57jdYkZxl/fzBPTHILqrC+DfiFdff5J27R2B4r4ux5/lJqKu3oMtFISgsq8bbPx/B6H5dMOPzNKfzW/PZrwve/fMIxB8qxL++2dPi935dO+L4mUrr59CQNvj7+H54K+6w9bvdz01CWIdgu+8Ax4Nc3n1Vb5RX1+GHPfku0xfdtSMybdb/zwmXYuZXuwEAkwZF4LXbr8CVL8U5XcZTkwfg9Z8yXK6rTuE4b275/aMwIDIU+SXVmNBsP/ft0sF63Pz1t9E4UVRl3dev3jYUF3dQX/rprkV/icH0lakAgNlTLsf/u/ZS9H16neHr9WWfPnA1Lo8KRVFFLbpf3N71DAYKEsLFACk2amtr0aFDB3z99df44x//aP3+scceQ3p6OhITE1vMM378eIwYMQJvv/229bs1a9bgzjvvRFVVFYKDWzYZrampQU1NjfVzWVkZevXqhdLSUnTq1PIJiy4or67D97vzMWlwBLrq+LTuLiGE1EpRagkhsP9kGS6PDEWb8z3OHj5VjlU7srE3rxSpWcVI+td16H3+1UJFTT0aLAL//mYPfskoxC9PXouel3TAsq2ZKK6qw8wb+qO6rgFPfr0bG/YVoN4iGm/ifxiCvl0vPIVaLAJFlbUOSyx2Zp7F06v34FRpNV69bahdZeSmdNz1YTImXh6ODiFtMO6yrujTpSO+2JmNyUMi0fOSDrji+Q0oq65Hv24d8cuTE+zmr6qtR9yBU5jQPxzbM4vQPrg1xvfvhl9PnMW3aXm45+o++O9Ph/CP8f3sAlMhBMrO1SOsg/35e7LkHPJLq1F2rg5fpeTgrlG9MGHAhdKZxMOnMW3pTrz/5ytx0xVR+DDxGD7afBxf/GM0fhPe+CS7IvkEvt99Eh/fNwoJhwvxTWou7rm6D345dAoj+3bGnSNdVyL+aV8+ys7VIz6j0BrsNQVyF4W0wcK/XIkx/bpY9/X+k6W46Z3GQPLNO4bh2gHd0PWiEGw+chqfbMvCK38cgvDQEAQFBeHT7Vn4z7f78O7dI6yvLD/efByvrD+I/905HJOHROJEUSUmL2h8fX37lT0Ru6sx4MucNxVnK2vx8rqD6NyxLZZsaQzonryhP9q0boX//tQYvE4aFIHF943E97tP4smvd+M/Nw/CX67ujfd+OYrFSccR+9A16B8Rii92ZuOV9Qex5qFr8Lu3kgA01tHafOQMAGDr09cjLbsYj32RjluH98CcqZcj5uVNABpLAScM6IaV27Pxyd+uwrSlO63br1+3jpg9ZSAuC78ID6/ahVuH98Dfx/ez/r4vrxQv/XAAU4dGYcKAbujduYPdeS6EwFPf7MHIvpfgrlG9rcfqi9/vxx+G90BtgwX3L/sVAHBJh2CkPTcJ2UVV6BraFjO/3I2f9hfY7c9/jO+HxUnHATT255SQUQjbAtcbBkXgo/tGIjXrLIQARvZtLJka9comnC6vwcoHrsbqXblYnZZnnadvlw4Y1L0ThvQIQ+/OHfDIKvsgdkBEKCpr6zG+fzeUVtUht7gKu3NLnRx1ji24azge/zLd7rtP/nYV3t50GO2CW+Oq6M4YEBGK2Wv2oqSqTnEZt1/ZE6//3xUoKKvG+/FH0bZ1K2QUlCP5eJF1mvhZE2ARAhPfvHAvfvtPw/HYFxfWfduIHrAIgRdvHaK5jyp3lJWVISwszPX9W2iQl5cnAIitW7faff/KK6+I/v37K85z2WWXiVdeecXuu61btwoA4uTJk4rzzJ07VwBo8VdaWqoluUS6sFgsoqqmXvG3+gaLKD1X6+UUaVdeXScaGiyykyFFRkGZKKqocTmdo32sVvPtm3O2Upyr1bbMc7X1YuP+AlFT1+Bwmnon+7GuvsH6/5Iq18elxWIRJZW11nWfOFMhTpWd05Rmd1XV1Itv03Kt67dVU9cgqmrqRUlVrcg8XWFNa87ZSlHfYBHl1XWioPScyC6qFDuOFwmLxfWxbbFYRGrWWVFSVSvKq+ta/H4wv1R8m5YrDheUiW/TchWXefx0hfjl0ClRdq5WrNtzUmzYly82HSgQp0rPib25JeKJL9JEdlFjGmvqGkRJZa3YdvSMaGiwiFOl50RCRqF4c8MhkZ5drJjGhgaLOFNeLX4+WCCyiypFXX2DyDxdIVJOnBW19S2PifoGizh+uqJFWsur68TqXTnWa1NVTb345eApzcejHkpLS1XdvzWVjJw8eRI9evTAtm3bMGbMGOv3r7zyCj799FMcOtSySLp///64//77MXv2bOt3W7duxdixY5Gfn4/IyJbviVkyQkRE5PvUloxoqjPStWtXtG7dGgUF9kVohYWFiIhQrtcQGRmpOH2bNm3QpYvyu+mQkBCEhMh/xUBERETG09Sapm3btoiJiUFcnH2lqbi4OFxzzTWK84wZM6bF9Bs3bsTIkSMV64sQERFRYNHctHfmzJn4+OOPsXTpUhw8eBBPPPEEsrOzMX36dADA7Nmzcd9991mnnz59OrKysjBz5kwcPHgQS5cuxZIlSzBr1iz9ckFEREQ+S3PT3rvuugtFRUV48cUXkZ+fjyFDhmD9+vXo06ex9838/Hy7Pkeio6Oxfv16PPHEE3j//ffRvXt3vPPOO2zWS0RERAA0Nu2VRXXTICIiIjINtfdvjk1DREREUjEYISIiIqkYjBAREZFUDEaIiIhIKgYjREREJBWDESIiIpKKwQgRERFJxWCEiIiIpNLcA6sMTf2ylZWVSU4JERERqdV033bVv6pPBCPl5eUAgF69eklOCREREWlVXl6OsLAwh7/7RHfwFosFJ0+eRGhoKIKCgnRbbllZGXr16oWcnJyA6WY+0PLM/Po35te/Mb++TwiB8vJydO/eHa1aOa4Z4hMlI61atULPnj0NW36nTp38ZserFWh5Zn79G/Pr35hf3+asRKQJK7ASERGRVAxGiIiISKqADkZCQkIwd+5chISEyE6K1wRanplf/8b8+jfmN3D4RAVWIiIi8l8BXTJCRERE8jEYISIiIqkYjBAREZFUDEaIiIhIqoAORj744ANER0ejXbt2iImJwebNm2UnSbPnn38eQUFBdn+RkZHW34UQeP7559G9e3e0b98eEyZMwP79++2WUVNTgxkzZqBr167o2LEjfv/73yM3N9fbWVGUlJSEW265Bd27d0dQUBC+/fZbu9/1yl9xcTHuvfdehIWFISwsDPfeey9KSkoMzp0yV3n+61//2mKfjx492m4aX8nzvHnzMGrUKISGhiI8PBy33norMjIy7Kbxp32sJr/+tH8BYOHChbjiiiusHXmNGTMGP/74o/V3f9q/gOv8+tv+1Y0IUF988YUIDg4WH330kThw4IB47LHHRMeOHUVWVpbspGkyd+5cMXjwYJGfn2/9KywstP7+2muvidDQUBEbGyv27t0r7rrrLhEVFSXKysqs00yfPl306NFDxMXFiV27donrrrtODBs2TNTX18vIkp3169eLZ555RsTGxgoAYs2aNXa/65W/yZMniyFDhoht27aJbdu2iSFDhoibb77ZW9m04yrP06ZNE5MnT7bb50VFRXbT+Eqeb7zxRrFs2TKxb98+kZ6eLm666SbRu3dvUVFRYZ3Gn/axmvz60/4VQoi1a9eKdevWiYyMDJGRkSHmzJkjgoODxb59+4QQ/rV/1eTX3/avXgI2GLnqqqvE9OnT7b67/PLLxdNPPy0pRe6ZO3euGDZsmOJvFotFREZGitdee836XXV1tQgLCxOLFi0SQghRUlIigoODxRdffGGdJi8vT7Rq1Ur89NNPhqZdq+Y3Zr3yd+DAAQFAbN++3TpNcnKyACAOHTpkcK6ccxSM/OEPf3A4jy/nubCwUAAQiYmJQgj/38fN8yuEf+/fJpdccon4+OOP/X7/NmnKrxCBsX/dEZCvaWpra5GamopJkybZfT9p0iRs27ZNUqrcd+TIEXTv3h3R0dH405/+hOPHjwMAMjMzUVBQYJfPkJAQXHvttdZ8pqamoq6uzm6a7t27Y8iQIabfFnrlLzk5GWFhYbj66qut04wePRphYWGm3QYJCQkIDw9H//798fe//x2FhYXW33w5z6WlpQCAzp07A/D/fdw8v038df82NDTgiy++QGVlJcaMGeP3+7d5fpv46/71hE8MlKe3M2fOoKGhAREREXbfR0REoKCgQFKq3HP11VdjxYoV6N+/P06dOoWXX34Z11xzDfbv32/Ni1I+s7KyAAAFBQVo27YtLrnkkhbTmH1b6JW/goIChIeHt1h+eHi4KbfBlClTcMcdd6BPnz7IzMzEf/7zH1x//fVITU1FSEiIz+ZZCIGZM2di7NixGDJkCAD/3sdK+QX8c//u3bsXY8aMQXV1NS666CKsWbMGgwYNst44/W3/Osov4J/7Vw8BGYw0CQoKsvsshGjxndlNmTLF+u+hQ4dizJgxuPTSS/HJJ59YK0W5k09f2hZ65E9perNug7vuusv67yFDhmDkyJHo06cP1q1bh9tuu83hfGbP8yOPPII9e/Zgy5YtLX7zx33sKL/+uH8HDBiA9PR0lJSUIDY2FtOmTUNiYqL1d3/bv47yO2jQIL/cv3oIyNc0Xbt2RevWrVtEkIWFhS0idF/TsWNHDB06FEeOHLG2qnGWz8jISNTW1qK4uNjhNGalV/4iIyNx6tSpFss/ffq06bcBAERFRaFPnz44cuQIAN/M84wZM7B27VrEx8ejZ8+e1u/9dR87yq8Sf9i/bdu2xW9+8xuMHDkS8+bNw7Bhw/D222/77f51lF8l/rB/9RCQwUjbtm0RExODuLg4u+/j4uJwzTXXSEqVPmpqanDw4EFERUUhOjoakZGRdvmsra1FYmKiNZ8xMTEIDg62myY/Px/79u0z/bbQK39jxoxBaWkpdu7caZ1mx44dKC0tNf02AICioiLk5OQgKioKgG/lWQiBRx55BKtXr8Yvv/yC6Ohou9/9bR+7yq8SX96/jgghUFNT43f715Gm/Crxx/3rFu/VlTWXpqa9S5YsEQcOHBCPP/646Nixozhx4oTspGny5JNPioSEBHH8+HGxfft2cfPNN4vQ0FBrPl577TURFhYmVq9eLfbu3SvuvvtuxWZzPXv2FJs2bRK7du0S119/vWma9paXl4u0tDSRlpYmAIi33npLpKWlWZtg65W/yZMniyuuuEIkJyeL5ORkMXToUGnN5Jzluby8XDz55JNi27ZtIjMzU8THx4sxY8aIHj16+GSe//nPf4qwsDCRkJBg19SxqqrKOo0/7WNX+fW3/SuEELNnzxZJSUkiMzNT7NmzR8yZM0e0atVKbNy4UQjhX/vXVX79cf/qJWCDESGEeP/990WfPn1E27ZtxZVXXmnXvM5XNLXJDw4OFt27dxe33Xab2L9/v/V3i8Ui5s6dKyIjI0VISIgYP3682Lt3r90yzp07Jx555BHRuXNn0b59e3HzzTeL7Oxsb2dFUXx8vADQ4m/atGlCCP3yV1RUJO655x4RGhoqQkNDxT333COKi4u9lEt7zvJcVVUlJk2aJLp16yaCg4NF7969xbRp01rkx1fyrJRPAGLZsmXWafxpH7vKr7/tXyGE+Nvf/ma9znbr1k1MnDjRGogI4V/7Vwjn+fXH/auXICGE8F45DBEREZG9gKwzQkRERObBYISIiIikYjBCREREUjEYISIiIqkYjBAREZFUDEaIiIhIKgYjREREJBWDESIiIpKKwQgRERFJxWCEiIiIpGIwQkRERFIxGCEiIiKp/j92G+ShV7GuzgAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(losses)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"As we can observe, smaller batch yields to a more noisy optimization process.\n",
"This is due to high gradient variance!"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## PyTorch's optimization API - *torch.optim*\n",
"\n",
"* For performing optimization with ease, PyTorch includes an optimization interface named torch.optim.\n",
"* Supports numerous optimization algorithms!\n",
"* We will demonstrate the API by replacing the above training procedure. "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from torch.optim import SGD"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"model = Net() # re-initialize net\n",
"batch_size = 64\n",
"loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True) # re-define dataloader \n",
"\n",
"# define the optimizer\n",
"optimizer = SGD(model.parameters(), lr=lr)\n",
"\n",
"model.train()\n",
"for epoch in range(num_epochs):\n",
" for batch_idx, (x, y) in enumerate(loader):\n",
"\n",
" # compute loss \n",
" logits = model(x)\n",
" loss = loss_fn(logits, y)\n",
"\n",
" # The three Musketeers!\n",
" optimizer.zero_grad() # sets p.grad = 0 for all params\n",
" loss.backward() # sets p.grad += dloss/dp\n",
" optimizer.step() # performs actual optimization step"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Learning rate scheduling\n",
"\n",
"* Observation: loss surface drastically changes over time and so is the hessian.\n",
"* Idea: change the learning rate over time.\n",
"* The most common practice is to reduce the learning rate after few epochs.\n",
"* Very useful in practice.\n",
"* Schedulers are also supported by torch.optim library!\n",
"\n",
"
\n",
"\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from torch.optim.lr_scheduler import MultiStepLR"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch [1/2] | Batch 300 | loss: 0.4164 | lr: 0.1000\n",
"Epoch [1/2] | Batch 600 | loss: 0.1513 | lr: 0.1000\n",
"Epoch [1/2] | Batch 900 | loss: 0.1877 | lr: 0.1000\n",
"Epoch [2/2] | Batch 300 | loss: 0.1112 | lr: 0.0100\n",
"Epoch [2/2] | Batch 600 | loss: 0.1343 | lr: 0.0100\n",
"Epoch [2/2] | Batch 900 | loss: 0.1085 | lr: 0.0100\n"
]
}
],
"source": [
"model = Net()\n",
"num_epochs = 2\n",
"\n",
"# define the optimizer and the scheduler\n",
"optimizer = SGD(model.parameters(), lr=lr)\n",
"scheduler = MultiStepLR(optimizer, milestones=[1], gamma=0.1) # reduce lr by 0.1 after 1 epoch\n",
"\n",
"model.train()\n",
"for epoch in range(num_epochs):\n",
" for batch_idx, (x, y) in enumerate(loader):\n",
"\n",
" # compute loss \n",
" logits = model(x)\n",
" loss = loss_fn(logits, y)\n",
"\n",
" # The three Musketeers!\n",
" optimizer.zero_grad()\n",
" loss.backward()\n",
" optimizer.step()\n",
" if (batch_idx + 1) % 300 == 0:\n",
" print(f'Epoch [{epoch+1}/{num_epochs}] | Batch {batch_idx+1} | \\\n",
"loss: {loss.item():.4f} | lr: {optimizer.param_groups[0][\"lr\"]:.4f}')\n",
" losses.append(loss.item())\n",
"\n",
" # Inform the scheduler an epoch was done!\n",
" scheduler.step()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Additional learning rate scheduling strategies include:\n",
"* Cosine annealing:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Learning rate warmup:\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Projected Gradient Descent (PGD)\n",
"\n",
"* So far, we have been concerned with **unconstrained** optimization problems.\n",
"* However, all of the above optimization algorithms may be generalized to **constrained** optimization problem of the following form:\n",
"\n",
"$$ \\min_x f(x) \\text{ subject to } x \\in \\mathcal{K} $$\n",
"$$ f: \\mathbb{R}^d \\rightarrow \\mathbb{R} \\text{, } \\mathcal{K} \\subseteq \\mathbb{R}^d \\text{ is closed and convex} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* This is done by a simple-greedy agorithm named PGD.\n",
"* The idea is to project $x$ onto $\\mathcal{K}$ after each iteration:\n",
"$$ \\tilde{x}_{k+1} = x_k - \\eta \\nabla_x f (x_k) $$\n",
"$$ x_{k+1} = \\Pi_\\mathcal{K}(\\tilde{x}_{k+1})$$\n",
"* The algorithm can be proved to converge under the same conditions required for GD to converge!\n",
"\n",
"
\n",
"\n",
"
\n",
"\n",
"* Mathematically, the projection of a point onto a set is defined as the closest point to the original point within the set:\n",
"$$ \\Pi_{\\mathcal{K}}(x) := \\arg \\min_y \\| y-x \\| \\text{ subject to } y \\in \\mathcal{K}$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Common projections:\n",
" + A canonical sphere with radius $R$:\n",
" $$ \\Pi_{\\mathcal{B}(R)}(x) = \\min\\{\\frac{R}{\\| x \\|}, 1\\} \\cdot x $$\n",
"
\n",
" \n",
"
\n",
" \n",
" + A linear subspace $W$:\n",
" $$ \\Pi_{W}(x) = \\sum_{i=1}^m \\langle x , \\; w_i \\rangle w_i $$\n",
" where $\\{ w_1, ..., w_m\\}$ is an orthonormal basis for $W$.\n",
"\n",
"
\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Use case: adversarial attacks\n",
"\n",
"* The goal is to find a small perturbation on a certain input, in a way which would cause the model to generate a wrong prediction.\n",
"\n",
"
![](\"https://upload.wikimedia.org/wikipedia/commons/thumb/e/ef/3dpoly.svg/220px-3dpoly.svg.png\"/)
![](\"https://www.ruder.io/content/images/2016/09/saddle_point_evaluation_optimizers.gif\"/)
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"In this tutorial, we will cover:\n",
"\n",
"* Continuous Optimization\n",
"* Reminder: multivariate calculus\n",
"* Gradient Descent\n",
" + Why does GD work?\n",
" + Selecting the learning rate\n",
" + What can go wrong?\n",
"* Stochastic gradient descent\n",
"* Advanced optimizers\n",
"* Working example\n",
"* PyTorch's optimization API - *torch.optim*\n",
"* Learning rate scheduling\n",
"* Projected Gradient Descent (PGD)\n",
" + Use case: adversarial attacks"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Continuous Optimization\n",
"\n",
"Continious optimization problems are fundumental in Computer Science.\n",
"\n",
"May be either unconstrained:\n",
"$$ \\min_x f(x) $$\n",
"$$ f: \\mathbb{R}^d \\rightarrow \\mathbb{R} $$\n",
"Or constrained:\n",
"$$ \\min_x f(x) \\text{ subject to } x \\in \\mathcal{K} $$\n",
"$$ f: \\mathbb{R}^d \\rightarrow \\mathbb{R} \\text{, } \\mathcal{K} \\subseteq \\mathbb{R}^d \\text{ is closed and convex} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Many problems in CS can be written as a continous optimization problems:\n",
"* Linear programs (LPs)\n",
"\n",
"![](\"https://www.fromthegenesis.com/wp-content/uploads/2018/06/Gradie_Desce.jpg\"/)
![](\"https://www.researchgate.net/publication/371855560/figure/fig1/AS:11431281170405999@1687749079905/Number-of-parameters-in-LLMs-over-the-years.png\"/)
![](\"imgs/paraboloid.png\"/)
![](\"imgs/lr.gif\")
![](\"https://nbviewer.org/github/vistalab-technion/cs236781-tutorials/blob/master/t01%20-%20linear%20models/imgs/sgd2d_2.png\")
![](\"https://www.baeldung.com/wp-content/uploads/sites/4/2022/01/batch-1-1024x670.png\")
\n",
"![](\"https://miro.medium.com/v2/resize:fit:640/1*zVi4ayX9u0MQQwa90CnxVg.gif\")
\n",
" ![](\"https://miro.medium.com/v2/resize:fit:1400/1*WRtvrr9Z0QcokiKlgU7xEw.gif\")
\n",
" ![](\"https://miro.medium.com/v2/resize:fit:640/1*_4zyVpZazh4OSIzprmYQEw.gif\")
\n",
" ![](\"imgs/scheduler.png\")
\n",
"![](\"https://miro.medium.com/v2/resize:fit:1400/1*9hiywbEuoVXz4iX2-BDGZQ.png\")
\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Learning rate warmup:\n",
"![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/warmup_cosine_schedule.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Projected Gradient Descent (PGD)\n",
"\n",
"* So far, we have been concerned with **unconstrained** optimization problems.\n",
"* However, all of the above optimization algorithms may be generalized to **constrained** optimization problem of the following form:\n",
"\n",
"$$ \\min_x f(x) \\text{ subject to } x \\in \\mathcal{K} $$\n",
"$$ f: \\mathbb{R}^d \\rightarrow \\mathbb{R} \\text{, } \\mathcal{K} \\subseteq \\mathbb{R}^d \\text{ is closed and convex} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* This is done by a simple-greedy agorithm named PGD.\n",
"* The idea is to project $x$ onto $\\mathcal{K}$ after each iteration:\n",
"$$ \\tilde{x}_{k+1} = x_k - \\eta \\nabla_x f (x_k) $$\n",
"$$ x_{k+1} = \\Pi_\\mathcal{K}(\\tilde{x}_{k+1})$$\n",
"* The algorithm can be proved to converge under the same conditions required for GD to converge!\n",
"\n",
"![](\"https://www.researchgate.net/profile/Steven-Pollock-2/publication/280104933/figure/fig4/AS:316868284960771@1452558694114/An-example-of-a-canonical-exam-problem-targeting-spherical-SOV-Variations-on-this.png\")
\n",
" ![](\"https://upload.wikimedia.org/wikipedia/commons/thumb/1/14/Linalg_projection_onto_plane.png/223px-Linalg_projection_onto_plane.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Use case: adversarial attacks\n",
"\n",
"* The goal is to find a small perturbation on a certain input, in a way which would cause the model to generate a wrong prediction.\n",
"\n",
"![](\"https://i.stack.imgur.com/5rfe9.png\")
\n",
"