{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Feedforward Nerual Networks\n", "\n", "In this notebook, we show how neural networks described in chapter 5 of PRML can be implemented. \n", "\n", "Because the notation of the book has ambiguity, we first describe the formulation in detail, and then move to the corresponding code. " ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib as mpl\n", "from matplotlib import pyplot as plt\n", "import time\n", "\n", "%matplotlib inline\n", "plt.rcParams['axes.labelsize'] = 14\n", "plt.rcParams['xtick.labelsize'] = 12\n", "plt.rcParams['ytick.labelsize'] = 12\n", "\n", "#to make this notebook's output stable across runs\n", "np.random.seed(42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1 Setting\n", "\n", "Although we consider multiclass classification problem in our implementation, in Sections 1 and 2, we formulate the problem from broader perspectives, where we include regression, binary classification, and multiclass classification problems.\n", "\n", "Let us define symbols as follows: \n", "* $N \\in \\mathbb{N}$ : data size, \n", "* $d \\in \\mathbb{N}$ : the dimension of input, \n", "* Denote input data by $x_0, x_1, \\dots , x_{N-1} \\in \\mathbb{R}^{d}$. \n", "* For a regression problems, denote target data by $t_0, t_1, \\dots, t_{N-1} \\in \\mathbb{R}$. \n", "* For a binary classification problem, denote target data by $t_0, t_1, \\dots, t_{N-1} \\in \\{ 0, 1 \\}$.\n", "* For a multiclass classification problem, let $\\mathcal{C} = \\left\\{ 0, 1, \\dots, C-1 \\right\\}$ be the set of class labels, $t_0, t_1, \\dots, t_{N-1} \\in \\left\\{0,1\\right\\}^{C}$ be the 1-of-$C$ coding of target labels, i.e., if $n$-th label is $c$, $t_n$ is a vector with its $c$ th component being 1 and other components being zero. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2 Theory\n", "\n", "## 2.1 Model : the representation of neural networks\n", "\n", "We consider $L$-layer feedforward neural networks (feedforward neural networks with $L-1$ hidden layers) such as one shown in the following figure: \n", "\"Drawing\"\n", "\n", "For simplicity, we assume that there is no skip-layer connection, i.e., each element in a layer is connected only to elements in its adjacent layer. \n", "\n", "In this section, we write down the definition of functions representing a neural network. Specifically, we decompose the function representing the neural network, so that we can extract contributions of parameters from a specified layer, for calculation of gradient in the next section, where we want derivative with respect to parameters from each layer. \n", "\n", "### 2.1.1 The number of elements and the dimension\n", "\n", "Let $n_l \\in \\mathbb{N}$ be the number of elements in the $l$-th layer. \n", "It follows that $n_0 = d$, and $n_L$ is the dimension of the output, which will be specified later.\n", "\n", "\n", "### 2.1.2 Activation functions\n", "\n", "Again for somplicity, we assume that activation functions for $l$-th layer with $l=1,2,\\dots,L-1$ are all \n", "$h : \\mathbb{R} \\rightarrow \\mathbb{R}$. $h$ can be, for example, logstic sigmoid function, tanh, ReLU and so on.\n", "Also, we define $h^{(l)}$ by\n", "\n", "$$\n", "\\begin{align}\n", " h^{(l)} : \\mathbb{R}^{n_l} \\rightarrow \\mathbb{R}^{n_l}, \\ \\ \n", " h^{(l)}(a) = (h(a_1), h(a_2), \\dots, h(a_{n_l}))^T \\ \\in \\mathbb{R}^{n_l}.\n", "\\end{align}\n", "$$\n", "\n", "Let us denote the activation function for the output layer by\n", "$g : \\mathbb{R}^{n_L} \\rightarrow \\mathbb{R}^{n_L}$. \n", "\n", "Concretely, $g$ can be chosen as below:\n", "* For binary classification, $g$ can be logistic sigmoid function, and $n_L = 1$. \n", "* For multiclass classification, $g$ can be softmax function, and $n_L = C$, where $C$ is the number of class labels.\n", "* For regression, $g$ can be an identity map, and $n_L = 1$.\n", "\n", "### 2.1.3 The input and output of each layer\n", "\n", "Let us define\n", "* $z^{(l)} \\in \\mathbb{R}^{n_l}$ : the output from the $l$-th layer ($l=0,1, \\dots, L-1$)\n", "* $a^{(l)} \\in \\mathbb{R}^{n_{l+1}}$ : the input to the $(l+1)$-th layer ($l=0,1, \\dots, L-1$)\n", "* $y \\in \\mathbb{R}^{n_L}$ : the output of the final layer\n", "\n", "Then, \n", "$$\n", "\\begin{align}\n", " z^{(l)} &= h^{(l)}(a^{(l-1)}) \\ \\ (l = 0,1, \\dots, L-1) \\\\\n", " y &= g(a^{(L-1)})\n", "\\end{align}\n", "$$\n", "follows. \n", "\n", "Let us also define parameters as follows\n", "* $w^{(l)}_{i,j}$ : the weight of $j$-th element of $l$-th layer on the $i$-th element of$(l+1)$-th layer. \n", "* $b^{(l)}_{i}$ : the bias of $i$-th element of $(l+1)$-th layer.\n", "\n", "If we define \n", "$w^{(l)} = (w^{(l)}_{i,j})_{i,j}$ ($n_{l+1} \\times n_{l}$ matrix) and \n", "$b^{(l)} = (b^{(l)}_{0}, b^{(l)}_{1}, \\dots, b^{(l)}_{n_{l+1}-1})^T$, \n", "then we have\n", "\n", "$$\n", "\\begin{align}\n", " a^{(l)} = w^{(l)} z^{(l)} + b^{(l)}\n", "\\end{align}\n", "$$\n", "\n", "For later convenience, we denote this affine mapping by \n", "$$\n", "\\begin{align}\n", " A^{(l)}_{\\theta^{(l)}} : \\mathbb{R}^{n_{l}} \\rightarrow \\mathbb{R}^{n_{l+1}} , \\ \\ A^{(l)}_{\\theta^{(l)} }(z) := w^{(l)}z + b^{(l)}\n", "\\end{align}, \n", "$$\n", "where $\\theta^{(l)} := (w^{(l)}, b^{(l)})$.\n", "\n", "### 2.1.4 The whole network\n", "\n", "Let a function $f_{\\theta} : \\mathbb{R}^{n_0} \\rightarrow \\mathbb{R}^{n_L}$ be the function that represents the whole neural network we defined above. \n", "\n", "From the definitions above, it can be written as \n", "\n", "$$\n", "\\begin{align}\n", " f_{\\theta} = g \\circ A^{(L-1)}_{\\theta^{(L-1)}} \\circ h^{(L-1)} \\circ A^{(L-2)}_{\\theta^{(L-2)}} \\circ \\cdots \n", " A^{(l+1)}_{\\theta^{(l+1)}} \\circ h^{(l+1)} \\circ A^{(l)}_{\\theta^{(l)}} \\circ h^{(l)} \\circ A^{(l-1)}_{\\theta^{(l-1)}} \n", " \\circ \\cdots \\circ h^{(1)} \\circ A^{(0)}_{\\theta^{(0)}} . \n", "\\end{align}\n", "$$\n", "\n", "\n", "If we focus on parameters $\\theta^{(l)}$, we can decompose the function as \n", "$$\n", "\\begin{align}\n", " f_{\\theta} = F^{(l)} \\circ A^{(l)}_{\\theta^{(l)}} \\circ Z^{(l)} \n", " \\ \\ (l = 0, \\dots, L-1), \n", "\\end{align}\n", "$$\n", "\n", "where\n", "\n", "$$\n", "\\begin{align}\n", " F^{(l)} &:= g \\circ A^{(L-1)}_{\\theta^{(L-1)}} \\circ h^{(L-1)} \\circ A^{(L-2)}_{\\theta^{(L-2)}} \\circ \\cdots \n", " A^{(l+1)}_{\\theta^{(l+1)}} \\circ h^{(l+1)} \\ \\ (l = 0,1, \\dots, L-2) \\\\\n", " F^{(L-1)} &:= g \\\\\n", " Z^{(l)} &:= h^{(l)} \\circ A^{(l-1)}_{\\theta^{(l-1)}} \\circ \\cdots \\circ h^{(1)} \\circ A^{(0)}_{\\theta^{(0)}} \\ \\ (l = 1, \\dots, L-1)\\\\\n", " Z^{(0)} &:= I\n", "\\end{align}\n", "$$\n", "\n", "Note that in the representation $f_{\\theta} = F^{(l)} \\circ A^{(l)}_{\\theta^{(l)}} \\circ Z^{(l)}$, \n", "$A^{(l)}_{\\theta^{(l)}}$ is the only part that contains $\\theta^{(l)}$, \n", "which will simplify our calculation of gradients in the next section. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.2 Cost functions and its gradient\n", "\n", "### 2.2.1 Cost functions\n", "\n", "We assume that, in training the neural network, we minimize the following cost function\n", "$$\n", "\\begin{align}\n", " E_{tot}(\\theta) = \\frac{1}{N} \\sum_{n=1}^{N} E \\left( f_{\\theta}(x_n), t_n \\right) + \\frac{\\lambda}{2N} \\|\\theta \\|^2 , \n", "\\end{align}\n", "$$\n", "where $\\lambda$ is the regularization parameter, \n", "and $E$ is chosen appropriately depending on the problem: \n", "* For regression problem, $E$ can be $E(y,t) = \\frac{1}{2} \\| y - t \\|^2$ and so on.\n", "* For multiclass classification problem with $g$ being softmax functions, E can be $E(y,t) = -\\sum_{k=0}^{d_{out}-1} t_k \\log y_k$ (negative log likelihood).\n", "\n", "### 2.2.2 Gradient \n", "\n", "Here, we calculate the gradient of the cost function with respect to parameters $w, b$. \n", "We consider the regularization term separately, and concentrate on the first term. \n", "We can treat each data point separately because this term has the form of summation over data.\n", "\n", "The gradient for each term can be calculated as follows\n", "($l = 0,1,\\dots, L-1$, $i=0, 1, \\dots, n_{l+1}-1$, $j=0, 1, \\dots, n_{l}-1$):\n", "\n", "$$\n", "\\begin{align}\n", " \\delta^{(l)}_{i} &:= \\sum_{k=0}^{n_L-1} \\left. \\frac{\\partial E}{\\partial y_k} \\right|_{y=f_{\\theta}(x)} \\cdot \n", " \\left. \\frac{\\partial F^{(l)}_{k}}{\\partial a^{(l)}_{i}} \\right|_{a^{(l)}= (A^{(l)} \\circ Z^{(l)})(x) } \\\\\n", " \\frac{\\partial}{\\partial w^{(l)}_{i,j}} E \\left( f_{\\theta}(x), t \\right) &= \\delta^{(l)}_{i} Z^{(l)}_{j}(x) \\\\\n", " \\frac{\\partial}{\\partial b^{(l)}_{i}} E \\left( f_{\\theta}(x), t \\right) &= \\delta^{(l)}_{i} \n", "\\end{align}\n", "$$\n", "which can be derived straightforwardly from the definition of $f_{\\theta}$ and the cost function given above.\n", "\n", "Thus, to obtain the gradient, it is sufficient to calculate the quantities $\\delta^{(l)}_{i}$. \n", "Here, $\\delta^{(l)}_{i}$ represents how much the cost function changes when $a^{(l)}_{i}$, the input to the $i$-th element of $(l+1)$-th layer, changes slightly. We may denote $\\delta^{(l)}_{i}$ ($i=0,1, \\dots, n_{l+1}-1$) collectively by $\\delta^{(l)}$.\n", "\n", "In the next section, we discuss back propagation, which is an algorithm to calculate these quantities recursively. \n", "\n", "\n", "### 2.2.3 Back propagation\n", "\n", "In back propagation procedure, we calculate $\\delta^{(l)}_{i}$, begining from $l=L-1$, and all the way down to $l=0$. \n", "\n", "First, for $l=L-1$, we have $F^{(L-1)} = g$, and hence\n", "\n", "$$\n", "\\begin{align}\n", " \\delta^{(L-1)}_{i} = \\sum_{k=0}^{n_L-1} \\left. \\frac{\\partial E}{\\partial y_k} \\right|_{y=f_{\\theta}(x)} \\cdot \n", " \\left. \\frac{\\partial g_{k}}{\\partial a^{(L-1)}_{i}} \\right|_{a^{(L-1)}= (A^{(L-1)} \\circ Z^{(L-1)})(x) }\n", "\\end{align}\n", "$$\n", "\n", "This can be easily calculated for various problems : \n", "* For least square regression, we have $E(y,t) = \\frac{1}{2} \\| y - t \\|^2$, $g(a) = a$, and hence $\\delta^{(L-1)}_{i} = (f_{\\theta}(x) - t)_{i}$. \n", "* For multiclass classification with $E(y,t) = -\\sum_{k=0}^{C-1} t_k \\log y_k$, $g_k(a) = \\frac{e^{a_k}}{\\sum_{m} e^{a_m}}$, we obtain $\\delta^{(L-1)}_{i} = (f_{\\theta}(x) - t)_{i}$. \n", "\n", "Next, assume that we have obtained $\\delta^{(l+1)}$ for a specific $l \\leq L - 2$. \n", "We can derive a recursion equation where $\\delta^{(l)}$ can be calculated from $\\delta^{(l+1)}$ as follows:\n", "$$\n", "\\begin{align}\n", " \\delta^{(l)}_{i} = h'(a^{(l)}_{i}) \\sum_{j=0}^{n_{l+2}-1} \\delta^{(l+1)}_{j} w^{(l+1)}_{j,i}\n", "\\end{align}\n", "$$\n", "This equation can be derived easily by noting the relation $F^{(l)} = F^{(l+1)} \\circ A^{(l+1)}_{\\theta^{(l+1)}} \\circ h^{(l+1)}$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.3 More on backpropagation\n", "\n", "Having described the theory of back propagation in the previous section, we move a step forward to implement the idea.\n", "\n", "### 2.3.1 Change of notations\n", "\n", "From now on, to make the implementation simple, we treat weight $w$ and bias $b$ equally by defining \n", "$$\n", "\\begin{align}\n", " W^{(l)}_{i,j} = \n", " \\begin{cases}\n", " b^{(l)}_{i} & (j=0) \\\\\n", " w^{(l)}_{i,j-1} & (j = 1,2, \\dots, n_{l})\n", " \\end{cases}\n", " \\ \\ (i = 0,1, \\dots,n_{l+1} -1 ) , \n", "\\end{align}\n", "$$\n", "where $W^{(l)}$ is $n_{l+1} \\times (n_{l}+1)$ matrix, and \n", "$$\n", "\\begin{align}\n", " \\tilde{z}^{(l)}_{i} = \n", " \\begin{cases}\n", " 1 & (i=0) \\\\\n", " z^{(l)}_{i-1} & (i = 1, \\dots, n_l)\n", " \\end{cases}\n", "\\end{align}\n", "$$\n", "With these notations, we have\n", "$$\n", "\\begin{align}\n", " a^{(l)} &= W^{(l)} \\tilde{z}^{(l)} \\\\\n", " \\tilde{z}^{(l)} &= \n", " \\begin{pmatrix}\n", " 1 \\\\\n", " z^{(l)}\n", " \\end{pmatrix} \\\\\n", " \\frac{\\partial}{\\partial W^{(l)}_{i,j}} E(f_{\\theta}(x),t) &= \\delta^{(l)}_{i} \\tilde{z}^{(l)}_{j}\n", "\\end{align}\n", "$$\n", "\n", "### 2.3.2 Gradient for a single data point\n", "\n", "For a data point $(x, t)$ we perform the following calculation : \n", "\n", "* Calculate $a, z$ for each layer (forward propagation) by \n", "$$\n", "\\begin{align}\n", " \\tilde{z}^{(0)} &= (1,x^T)^T) \\\\\n", " a^{(l)} &= W^{(l)} \\tilde{z}^{(l)} \\ \\ (l=0, 1, \\dots, L-1) ,\\\\\n", " \\tilde{z}^{(l)} &=\n", " \\begin{pmatrix}\n", " 1 \\\\\n", " h^{(l)}(a^{(l-1)})\n", " \\end{pmatrix}\\ \\ (l=1,\\dots, L-1) \\\\\n", " y &= g(a^{(L-1)})\n", "\\end{align}\n", "$$\n", "* Calculate $\\delta^{(L-1)}$. For examples described in section 3.3, we have\n", "$$\n", "\\begin{align}\n", " \\delta^{(L-1)}_i = (y - t)_i . \n", "\\end{align}\n", "$$\n", "* Recursively calculate $\\delta^{(l)}_{i}$ ($l= L-2, L-1, \\dots, 1, 0$) by \n", "$$\n", "\\begin{align}\n", " \\delta^{(l)}_{i} = h'(a^{(l)}_{i}) \\sum_{j=0}^{n_{l+2}-1} \\delta^{(l+1)}_{j} W^{(l+1)}_{j,i+1}\n", "\\end{align}\n", "$$\n", "where $i = 0,1, \\dots, n_{l+1} - 1$\n", "* Calculate the gradient using $\\delta$\n", "$$\n", "\\begin{align}\n", " \\frac{\\partial}{\\partial W^{(l)}_{i,j}} E(f_{\\theta}(x),t) = \\delta^{(l)}_{i} \\tilde{z}^{(l)}_{j}\n", "\\end{align}\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.4 Optimization by gradient descent\n", "\n", "To numerically minimize the cost function, here we use gradient descent method.\n", "\n", "In gradient descent method, we iteratively update our variable in the direction of gradient, i.e., if we denote the function to be minimized by $f$ and its variable by $x$, then in the $n$-th step of iteration, we have\n", "$$\n", "\\begin{align}\n", " x_{n+1} = x_n - \\alpha \\nabla f(x_n)\n", "\\end{align}\n", "$$\n", "where $\\alpha > 0$ is learning rate, which should be specified by the user." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3 From math to code\n", "\n", "Now, we translate our equations described above into codes. \n", "To perform computation efficiently, we utilize numpy array operation so that we do not have to code iteration over samples explicitly.\n", "\n", "\n", "## 3.1 Overview \n", "\n", "Before delving into the detail, let me give an overview: \n", "\n", "* First, we list up some variables, and prepare some utility functions in Section 3.2.\n", "* Next, we define functions performing forward propagation (`fprop`) and backward propagation (`bprop`) in Sections 3.3 and 3.4 respectively. Section 3.5 deals with the cost function and its gradient.\n", "* Then, we will define a function performing minimization using gradient descent method in Section 3.6.\n", "* Finally, in Section 3.7, we write `NeuralNet` class, which represents our neural network." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2 Variables\n", "\n", "Let us define some variable as follows.\n", "\n", "* `num_neurons` : ($L+1$,) array, where `num_neurons[l]` = $n_{l}$ with $n_{0} = d$.\n", "* `act_func` : activation function $h$.\n", "* `act_func_deriv` : the derivative of the activation function, $h'$.\n", "* `out_act_func` : the activation function for the output layer, $g$.\n", "* `E` : a function representing the first term of the cost function, or more concretely $\\frac{1}{N} \\sum_{n=0}^{N-1}E(y_n, t_n)$\n", "* `lam` : the regularization constant $\\lambda$\n", "* `W_matrices` : ($L$,) array, with `dtype='object'`, `W_matrices[l][i, j]` = $W^{(l)}_{i,j}$, \n", "\n", "The variables shown above mainly concerns the structure and the contents of neural networks.\n", "\n", "Let us define some auxiliary matrices which depends on input data as follows:\n", "* `X` : ($N, d_{in}$) array. The input data, where `X[n, i]` $= X_{n, i} :=(x_{n})_{i}$\n", "* `C` : int, the number of categories we deal with.\n", "* `y` : ($N$,) array. The label data. Each element of `y` should be one of `0,1, ... , C-1`.\n", "* `T` : ($N, d_{out}$) array. The 1-of-C coding of the label data, where `T[n, i]` $= T_{n, i} := (t_{n})_{i}$\n", "* `Y` : ($N, d_{out}$) array. `Y[n, i]` = $Y_{n, i} := $the $i$-th element of the output from the neural network given the input $x_{n}$.\n", "* `A_matrices` : ($L$,) array, with `dtype='object'`, `A_matrices[l][n, i]` = $A^{(l)}_{n, i}$ := ($a^{(l)}_{i}$ for the $n$-th data point $x_n$). Note that $A^{(l)}$ is a $(N, n_{l+1})$ array.\n", "* `Z_matrices` : ($L$,) array, with `dtype='object'`, `z_matrices[l][n, i]` = $Z^{(l)}_{n, i}$ := ($\\tilde{z}^{(l)}_{i}$ for the $n$-th data point $x_n$). Note that $Z^{(l)}$ is a $(N, n_l+1)$ array.\n", "* `D_matrices` : ($L$,) array, with `dtype='object'`, `D_matrices[l][n, i]` = $\\Delta^{(l)}_{n,i} := (\\delta^{(l)}_{i}$ for the $n$-th data point $(x_n, t_n)$). Note that $\\Delta^{(l)}$ is a $(N, n_{l+1})$ array.\n", "\n", "\n", "We included the subscript $n$ in each matrix, because we want to iterate over data points by taking advantage of numpy array operation (For detail, see the next section.).\n", "\n", "Frist, we define a function which calculate 1-of-C encoding of the target variable." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def calcTmat(y, C):\n", " '''\n", " This function generates the matrix T from the training label y and the number of classes C\n", " \n", " Parameters\n", " ----------\n", " y : 1-D numpy array\n", " The elements of y should be integers in [0, C-1]\n", " C : int\n", " The number of classes\n", " \n", " Returns\n", " ----------\n", " T : (len(y), C) numpy array\n", " T[n, c] = 1 if y[n] == c else 0\n", " '''\n", " N = len(y)\n", " T = np.zeros((N, C))\n", " for c in range(C):\n", " T[:, c] = (y == c)\n", " return T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "We defined `W_matrices` as a ($L$,) array, whose element `W_matrices[l]` is a $(n_{l+1}, n_l + 1)$ array.\n", "However, for minimization process, it is easier to handle vector variables. \n", "Thus, for later convenience, here we define a function which transforms the 1-D array of arrays into a single vector and vice versa." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def reshape_vec2mat(vec, num_neurons):\n", " '''\n", " vec : 1-D numpy array, dtype float\n", " Vector, which we will convert into an array of matrices\n", " num_neurons : 1-D array, dtype int\n", " An array representing the structure of a neural network\n", " '''\n", " L = len(num_neurons) - 1\n", " matrices = np.zeros(L, dtype='object')\n", " ind = 0\n", " for l in range(L):\n", " mat_size = num_neurons[l+1] * (num_neurons[l] + 1)\n", " matrices[l] = np.reshape(vec[ind: ind + mat_size], ( num_neurons[l+1], num_neurons[l] + 1) )\n", " ind += mat_size \n", " return matrices\n", "\n", "def reshape_mat2vec(matrices, num_neurons):\n", " '''\n", " matrices : 1-D numpy array, dtype object\n", " An array of matrices (2-D numpy array), which we will convert into a flattened 1-D array of float.\n", " num_neurons : 1-D array, dtype int\n", " An array representing the structure of a neural network\n", " '''\n", " L = len(num_neurons) - 1\n", " vec = np.zeros( np.sum(num_neurons[1:]*(num_neurons[:L] + 1)) )\n", " ind = 0\n", " for l in range(L):\n", " mat_size = num_neurons[l+1]*(num_neurons[l] + 1)\n", " vec[ind:ind+mat_size] = np.reshape(matrices[l], mat_size)\n", " ind += mat_size\n", " return vec" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.3 forward propagation\n", "\n", "In forward propagation, we calculate $A^{(l)}$ and $Z^{(l)}$ ($l=0,1,\\dots, L-1$) by \n", "\n", "$$\n", "\\begin{align}\n", " Z^{(l)} &= \\begin{cases}\n", " \\left(\\boldsymbol{1}_{N}, X \\right) & (l=0) \\\\ \n", " \\left(\\boldsymbol{1}_{N} , h\\left(A^{(l-1)} \\right)\\right) & (l = 1, \\dots, L-1) \n", " \\end{cases}\\\\\n", " A^{(l)} &= Z^{(l)} {W^{(l)}}^{T}\n", "\\end{align}\n", "$$\n", "and the final output $Y$ by \n", "$$\n", "\\begin{align}\n", " Y = g(A^{(L-1)}), \n", "\\end{align}\n", "$$\n", "where $\\boldsymbol{1}_N := (1, 1, \\dots, 1)^T \\in \\mathbb{R}^{N}$, \n", "and we slightly abuse notations so that we regard $h$ and $g$ as acting elementwisely on $A^{(l)}$." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def fprop(W_matrices, act_func, out_act_func, X):\n", " '''\n", " This function performs forward propagation, and calculate input/output for each layer of a neural network.\n", " \n", " Parameters:\n", " ----------\n", " W_matrices : 1-D array, dtype='object'\n", " (L,) array, where W_matrices[l] corresponds to the parameters of l-th layer.\n", " act_func : callable\n", " activation function for hidden layers\n", " out_act_func : callable\n", " activation function for the final layer\n", " X : 2-D array\n", " (N,d) numpy array, with X[n, i] = i-th element of x_n\n", " \n", " Returns:\n", " ----------\n", " Z_matrices : 1-D array, dtype='object'\n", " Z_matrices[l] is 1-D array, which represents the output from the l-th layer.\n", " A_matrices : 1-D array, dtype='object'\n", " A_matrices[l] is 1-D array, which represents the input to the (l+1)-th layer.\n", " Y : 2-D array\n", " The final output of the neural network.\n", " '''\n", " L = len(W_matrices)\n", " N = len(X)\n", " Z_matrices = np.zeros(L, dtype='object')\n", " A_matrices = np.zeros(L, dtype='object')\n", "\n", " Z_matrices[0] = np.concatenate((np.ones((N, 1)), X), axis=1)\n", " A_matrices[0] = Z_matrices[0] @ (W_matrices[0].T)\n", " \n", " for l in range(1, L):\n", " Z_matrices[l] = np.concatenate((np.ones((N, 1)), act_func(A_matrices[l-1])), axis=1)\n", " A_matrices[l] = Z_matrices[l] @ (W_matrices[l].T)\n", " Y = out_act_func(A_matrices[L-1])\n", " \n", " return Z_matrices, A_matrices, Y" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.4 Backward propagation\n", "\n", "In backward propagation, we calculate $\\Delta^{(l)}$ as follows: \n", "\n", "\n", "* First, calculate $\\Delta^{(L-1)}$. For examples described in section 3.3 we have $\\Delta^{(L-1)} = Y - T$. However, note that in general $\\Delta^{(L-1)}$ may depend on $A^{(l)}$ and $Z^{(l)}$. In our implementation, `d_final_layer` takes care of the relation between $\\Delta^{(L-1)}$ and $A^{(l)}, Z^{(l)}, Y$ and $T$.\n", "* Recursively calculate $\\Delta^{(l)}$ ($l= L-2, L-1, \\dots, 1, 0$) by \n", "$$\n", "\\begin{align}\n", " \\Delta^{(l)} = h'(A^{(l)}) \\ast \\left( \\Delta^{(l+1)} (W^{(l+1)}\\mathrm{[:, 1:]})\\right), \n", "\\end{align}\n", "$$\n", "where we again slightly abused the definition of $h$, $\\ast$ stands for elementwise multiplication, and $(W^{(l+1)}\\mathrm{[:, 1:]})$ stands for a matrix obtained by deleting the first column of $W^{(l)}$." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def bprop(W_matrices, act_func, act_func_deriv, out_act_func, Z_matrices, A_matrices, Y, T, d_final_layer):\n", " '''\n", " This function performs backward propagation and returns delta.\n", " \n", " Parameters\n", " ----------\n", " W_matrices : 1-D array, dtype='object'\n", " (L,) array, where W_matrices[l] corresponds to the parameters of l-th layer.\n", " act_func : callable\n", " activation function for hidden layers\n", " act_func_deriv : callable\n", " The derivative of act_func\n", " out_act_func : callable\n", " activation function for the final layer\n", " Z_matrices : 1-D array, dtype='object'\n", " Z_matrices[l] is 1-D array, which represents the output from the l-th layer.\n", " A_matrices : 1-D array, dtype='object'\n", " A_matrices[l] is 1-D array, which represents the input to the (l+1)-th layer.\n", " Y : 2-D array\n", " The final output of the neural network.\n", " T : 2-D array\n", " Array for the label data.\n", " d_final_layer : callable\n", " A function of Z_matrices, A_matrices, Y and T, which returns $delta^{(L-1)}$\n", " \n", " Returns\n", " ----------\n", " D_matrices : 1-D array\n", " D_matrices[l] is a 2-D array, where D_matrices[l][n, i] = $\\Delta^{(l)}_{n, i}$\n", "\n", " '''\n", " L = len(W_matrices)\n", " D_matrices = np.zeros(L, dtype='object')\n", " \n", " D_matrices[L-1] = d_final_layer(Z_matrices, A_matrices, Y, T)\n", " for l in range(L-2, -1, -1):\n", " D_matrices[l] = act_func_deriv(A_matrices[l]) * ( D_matrices[l+1] @ W_matrices[l+1][:, 1:] )\n", "\n", " return D_matrices" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.5 Cost function and its gradient\n", "\n", "Here we implement the function giving the value of the cost function and its gradient.\n", "\n", "Recall that the cost function is given by\n", "$$\n", "\\begin{align}\n", " E_{tot}(\\theta) = \\frac{1}{N} \\sum_{n=1}^{N} E \\left( f_{\\theta}(x_n), t_n \\right) + \\frac{\\lambda}{2N} \\|\\theta \\|^2. \n", "\\end{align}\n", "$$\n", "\n", "The gradient of the first term can be obtained by \n", "$$\n", "\\begin{align}\n", " \\frac{\\partial}{\\partial W^{(l)}_{i,j}} \\frac{1}{N}\\sum_{n=0}^{N-1} E(f_{\\theta}(x_n),t_n) \n", " &= \\frac{1}{N}\\sum_{n=0}^{N-1} \\Delta^{(l)}_{n,i} Z^{(l)}_{n,j} \\\\\n", " &= \\frac{1}{N} \\left( \\Delta^{(l)T} Z^{(l)} \\right)_{i,j}\n", "\\end{align}\n", "$$" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def cost_and_grad(W_matrices, X, T, act_func, act_func_deriv, out_act_func, E, d_final_layer, lam):\n", " '''\n", " This function performs backward propagation and returns delta.\n", " \n", " Parameters\n", " ----------\n", " W_matrices : 1-D array, dtype='object'\n", " (L,) array, where W_matrices[l] corresponds to the parameters of l-th layer.\n", " X : 2-D array\n", " (N,d) numpy array, with X[n, i] = i-th element of x_n \n", " T : 2-D array\n", " Array for the label data.\n", " act_func : callable\n", " activation function for hidden layers\n", " act_func_deriv : callable\n", " The derivative of act_func\n", " out_act_func : callable\n", " activation function for the final layer\n", " E : callable\n", " A function representing the first term of the cost function\n", " d_final_layer : callable\n", " A function of Z_matrices, A_matrices, Y and T, which returns $delta^{(L-1)}$\n", " lam : float\n", " Regularization constant\n", " \n", " Returns\n", " ----------\n", " cost_val : float\n", " The value of the cost function\n", " grad_matrices : 1-D array, dtype object\n", " Array of 2-D matrices corresponding the gradient\n", " '''\n", " \n", " N = len(X)\n", " L = len(W_matrices)\n", " \n", " Z_matrices, A_matrices, Y = fprop(\n", " W_matrices=W_matrices, \n", " act_func=act_func, \n", " out_act_func=out_act_func, \n", " X=X)\n", " \n", " cost_val = E(Y, T) + lam/(2*N)*( np.sum( np.linalg.norm(W)**2 for W in W_matrices ) )\n", " \n", " D_matrices = bprop(\n", " W_matrices=W_matrices, \n", " act_func=act_func, \n", " act_func_deriv=act_func_deriv, \n", " out_act_func=out_act_func, \n", " Z_matrices=Z_matrices, \n", " A_matrices=A_matrices, \n", " Y=Y, \n", " T=T, \n", " d_final_layer=d_final_layer)\n", " \n", " grad_matrices = np.zeros(L, dtype='object')\n", " for l in range(L):\n", " grad_matrices[l] = D_matrices[l].T @ Z_matrices[l] / N + lam/N * W_matrices[l]\n", " \n", " return cost_val, grad_matrices" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.6 Minimization by gradient descent\n", "\n", "In gradient descent, we obtain an approximate optimal solution by the update\n", "$$\n", "\\begin{align}\n", " x_{n+1} = x_n - \\alpha \\nabla f(x_n).\n", "\\end{align}\n", "$$\n", "\n", "Here, as to the stopping criterion, we examine the change of function value, and if it does not decrease more than the specified value `ftol`, we terminate the iteration." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def minimize_GD(func_and_grad, x0, alpha=0.01, maxiter=1e4, ftol=1e-5):\n", " '''\n", " This function minimizes the given function using gradient descent method\n", " \n", " Parameters\n", " ----------\n", " func_and_grad : callable\n", " A function which returns the tuple (value of function to be minimized (real-valued), gradient of the function)\n", " x0 : 1-D array\n", " Initial value of the variable\n", " alpha : float\n", " Learning rate\n", " maxiter : int\n", " Maximum number of iteration\n", " ftol : float\n", " The threshold for stopping criterion. If the change of the value of function is smaller than this value, the iteration stops.\n", " \n", " Returns\n", " ----------\n", " result : dictionary\n", " result['x'] ... variable, result['nit'] ... the number of iteration, result['func']...the value of the function, result['success']... whether the minimization is successful or not\n", " '''\n", " nit = 0\n", " x = x0\n", " val, grad = func_and_grad(x)\n", " while nit < maxiter:\n", " xold = x\n", " valold = val\n", " x = x - alpha*grad\n", " val, grad = func_and_grad(x)\n", " nit += 1\n", " if abs(val - valold) < ftol:\n", " break\n", " success = (nit < maxiter)\n", " return {'x': x, 'nit':nit, 'func':val, 'success':success}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.7 Neural Network\n", "\n", "Because we consider multiclass classification problem here, we use conventional choice of activation functions, where activation function for hidden layers are logistic sigmoid function, the activation function for the output layer is softmax, and the error function is cross-entropy." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def E(Y,T):\n", " N = len(Y)\n", " return -np.sum( T*np.log(Y) )/N\n", "\n", "def softmax(x):\n", " x0 = np.max(x,axis=1)\n", " x0 = np.reshape(x0, (len(x0),1))\n", " tmp = np.sum(np.exp(x-x0),axis=1)\n", " return np.exp(x-x0)/np.reshape(tmp,(len(tmp),1))\n", "\n", "def sigmoid(x):\n", " return 1.0/(1 + np.exp(-x))\n", "\n", "def sigmoid_deriv(x):\n", " return np.exp(-x)/((1+np.exp(-x))**2)\n", "\n", "def d_final_layer(Z, A, Y, T):\n", " return Y - T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With all the preparations, here we define our neural network, `NeuralNet` class." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": true }, "outputs": [], "source": [ "class NeuralNet:\n", " def __init__(self, C, lam, num_neurons, act_func=sigmoid, act_func_deriv=sigmoid_deriv, out_act_func=softmax, E=E, d_final_layer=d_final_layer):\n", " self.C = C\n", " self.num_neurons = num_neurons\n", " self.L = len(num_neurons) - 1\n", " self.act_func = act_func\n", " self.act_func_deriv = act_func_deriv\n", " self.out_act_func = out_act_func\n", " self.E = E\n", " self.d_final_layer = d_final_layer\n", " self.lam = lam\n", " self.W_matrices = np.zeros(self.L,dtype='object') \n", " \n", " def fit(self, X, y, ep=0.01, alpha=0.01, maxiter=1e4, ftol=1e-5, show_message=False): \n", " '''\n", " Parameters\n", " ----------\n", " X : 2-D array\n", " (N,d) numpy array, with X[n, i] = i-th element of x_n \n", " y : 1-D numpy array\n", " The elements of y should be integers in [0, C-1]\n", " ep : float\n", " The maximum value of the initial weight parameter, which will be used for random initialization\n", " alpha : float\n", " Learning rate\n", " maxiter : int\n", " Maximum number of iteration\n", " ftol : float\n", " The threshold for stopping criterion. If the change of the value of function is smaller than this value, the iteration stops.\n", " show_message : bool\n", " If True, the result of the optimization is shown.\n", " '''\n", " T = calcTmat(y, self.C)\n", " def cost_and_grad_vec(x):\n", " val, grad_mat = cost_and_grad(\n", " W_matrices=reshape_vec2mat(x, num_neurons=self.num_neurons),\n", " X=X, \n", " T=T,\n", " act_func=self.act_func, \n", " act_func_deriv=self.act_func_deriv,\n", " out_act_func=self.out_act_func,\n", " E=self.E,\n", " d_final_layer=self.d_final_layer,\n", " lam=self.lam\n", " )\n", " grad_vec = reshape_mat2vec(matrices=grad_mat, num_neurons=self.num_neurons)\n", " return val, grad_vec\n", " tht0 = 2*ep*(np.random.random(np.sum(self.num_neurons[1:]*(self.num_neurons[:self.L]+1))) - 0.5)\n", " time_start = time.time()\n", " result = minimize_GD(\n", " func_and_grad=cost_and_grad_vec,\n", " x0=tht0,\n", " alpha=alpha,\n", " maxiter=maxiter,\n", " ftol=ftol\n", " )\n", " time_end = time.time()\n", " if show_message:\n", " print(f\"success : {result['success']}\")\n", " print(f\"nit : {result['nit']}\")\n", " print(f\"func value : {result['func']}\")\n", " print(f\"calcualtion time : {time_end - time_start}seconds\")\n", " self.W_matrices=reshape_vec2mat(vec=result['x'], num_neurons=self.num_neurons)\n", " \n", " def predict_proba(self, X):\n", " '''\n", " Parameters\n", " ----------\n", " X : 2-D array\n", " (N,d) numpy array, with X[n, i] = i-th element of x_n\n", " \n", " Returns\n", " ----------\n", " Y : 2-D array\n", " (len(X), self.C) array, where Y[n, c] represents the probability that the n-th instance belongs to c-th class\n", " '''\n", " Z_matrices, A_matrices, Y = fprop(\n", " W_matrices=self.W_matrices, \n", " act_func=self.act_func, \n", " out_act_func=self.out_act_func, \n", " X=X)\n", " return Y\n", " \n", " def predict(self, X):\n", " '''\n", " Parameters\n", " ----------\n", " X : 2-D array\n", " (N,d) numpy array, with X[n, i] = i-th element of x_n\n", " \n", " Returns\n", " ----------\n", " classes : 1-D numpy array\n", " (len(X), ) array, where classes[n] represents the predicted class to which the n-th instance belongs\n", " '''\n", " Y = self.predict_proba(X)\n", " return np.argmax(Y, axis=1) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4 Experiment " ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from sklearn import datasets\n", "from sklearn.metrics import accuracy_score\n", "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.1 Gradient check\n", "\n", "Because the implementation of the backward propagation is complicated, it is desirable to check whether the implementation is correct before we use the code. \n", "\n", "In this section, we compare the gradient obtained by the backward propagation and the gradient calculated from direct numerical differentiation. \n", "\n", "The numerical differentiation is obtained as follows\n", "$$\n", "\\begin{align}\n", " \\frac{\\partial E_{tot} }{\\partial W^{(l)}_{i,j}} \\simeq \n", " \\frac{ E_{tot}(W^{(l)}_{i,j}+\\varepsilon) - E_{tot}(W^{(l)}_{i,j}-\\varepsilon) }{2\\varepsilon}, \n", "\\end{align}\n", "$$\n", "where we slightly abused the notation so that $E_{tot}(W^{(l)}_{i,j}\\pm\\varepsilon)$ means the value of the function, where all the elements, except for the $(l,i,j)$ element, of $W$ is fixed to $W^{(l')}_{i',j'}$, and the $(l,i,j)$ element is perturbed by $\\varepsilon$.\n", "The quantity $\\varepsilon$ is assumed to be small, and the error is $\\mathcal{O}(\\varepsilon^2)$, " ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def grad_check(nn, W_vec, X, T, ep):\n", " num_params = len(W_vec)\n", " W_matrices = reshape_vec2mat(W_vec, nn.num_neurons)\n", " cost, grad_mat_bp = cost_and_grad(W_matrices, X, T, act_func=nn.act_func, act_func_deriv=nn.act_func_deriv, out_act_func=nn.out_act_func, E=nn.E, d_final_layer=nn.d_final_layer, lam=nn.lam)\n", " grad_vec_bp = reshape_mat2vec(grad_mat_bp, nn.num_neurons)\n", " \n", " grad_vec_num = np.zeros(num_params)\n", " for cnt in range(num_params):\n", " W_p = np.copy(W_vec)\n", " W_m = np.copy(W_vec)\n", " W_p[cnt] += ep\n", " W_m[cnt] -= ep\n", " cost_p, grad_p = cost_and_grad(reshape_vec2mat(W_p, nn.num_neurons), X, T, act_func=nn.act_func, act_func_deriv=nn.act_func_deriv, out_act_func=nn.out_act_func, E=nn.E, d_final_layer=nn.d_final_layer, lam=nn.lam)\n", " cost_m, grad_m = cost_and_grad(reshape_vec2mat(W_m, nn.num_neurons), X, T, act_func=nn.act_func, act_func_deriv=nn.act_func_deriv, out_act_func=nn.out_act_func, E=nn.E, d_final_layer=nn.d_final_layer, lam=nn.lam)\n", " grad_vec_num[cnt] = (cost_p - cost_m )/(2*ep)\n", " return np.linalg.norm(grad_vec_bp - grad_vec_num), grad_vec_bp, grad_vec_num" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the code, we perform gradient check for a small neural network." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "958.749997375419\n", "958.7499973563445\n", "1.641231604721956e-09\n" ] } ], "source": [ "num_neurons = np.array([4, 3, 2])\n", "\n", "L = len(num_neurons) - 1\n", "dim_W = np.sum(num_neurons[1:] * (num_neurons[:L]+1))\n", "nn = NeuralNet(C=3, lam=1.0, num_neurons=num_neurons)\n", "W_vec = np.arange(0.0, dim_W, 1)\n", "X = np.reshape(np.arange(0.0, 2*num_neurons[0], 1.0), (2, num_neurons[0]))\n", "T = np.array([[1.0,0],[0,1]])\n", "error, grad_vec_bp, grad_vec_num = grad_check(nn, W_vec, X, T, ep=0.0001)\n", "\n", "print(np.linalg.norm(grad_vec_bp)**2)\n", "print(np.linalg.norm(grad_vec_num)**2)\n", "print(error)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the difference between the two gradients, where one is obtained by backward propagation and the other by numerical differentiation, is sufficiently small." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.2 Toy data \n", "\n", "Having checked the validity of our implementation, here we apply our code to simple toy data." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def get_meshgrid(x, y, nx, ny, margin=0.1):\n", " x_min, x_max = (1 + margin) * x.min() - margin * x.max(), (1 + margin) * x.max() - margin * x.min()\n", " y_min, y_max = (1 + margin) * y.min() - margin * y.max(), (1 + margin) * y.max() - margin * y.min()\n", " xx, yy = np.meshgrid(np.linspace(x_min, x_max, nx),\n", " np.linspace(y_min, y_max, ny))\n", " return xx, yy\n", "\n", "def plot_result(ax, clf, xx, yy, X, t):\n", " Z = (clf.predict(np.c_[xx.ravel(), yy.ravel()])).reshape(xx.shape)\n", " ax.contourf(xx, yy, Z, alpha=0.7)\n", " ax.scatter(X[:,0], X[:,1], c=t, edgecolor='k')" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEACAYAAACznAEdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd8lEX+wPHPbG/pPYEkBAKhdwhV\nOgoe4IENCwoKtlOxe56iooh6ljvr8TsRwQZKU6ogIEjvPaEH0nvbJLvZ3fn9kdzKUlQ0EMR5v177\net3OzvPs93lyMjvPzHxHSClRFEVRFE19B6AoiqJcHlSDoCiKogCqQVAURVFqqQZBURRFAVSDoCiK\notRSDYKiKIoCqAZBURRFqXVBDYIQ4gEhxDYhhEMIMeO08mQhxAohRKEQIk8I8ZUQIupnzrNGCFEl\nhCivfaX+jmtQFEVR6sCF9hAygZeA6WeUBwHTgHggDigDPv6Fcz0gpbTVvppdYByKoihKHdNdSGUp\n5TwAIUQnoMFp5UtPryeEeBf4oS4CVBRFUS6NC2oQLkBvYP8v1HlFCDEVSAWekVKuOVclIcR4YDyA\n1WrtmJSUVJdxKoqiXPG2b9+eL6UM+6V6dd4gCCHaAM8Bw3+m2pPAAcAJ3AR8K4RoJ6U8emZFKeU0\nah5H0alTJ7lt27a6DllRFOWKJoRI+zX16nSWkRCiCbAUeEhKue589aSUm6WUZVJKh5TyE2A9MKQu\nY1EURVEuTJ01CEKIOGAlMFlKOesCD5eAqKtYFEVRlAt3odNOdUIIE6AFtEIIU21ZDLAKeE9K+eEv\nnCNQCDH4tGNvoWbMYflvvQhFURTl97vQMYR/AJNOe38r8AI1v/ATgElCCO/nUkobgBDi70AvKeU1\ngJ6aqatJgBtIAUZIKdVaBEVRlHok/kgb5KhBZeXXqqiooKioiMjISLRabX2Hoyj1SgixXUrZ6Zfq\nqdQVyhXF4XAw/q4JhIWE0TyxBdGRMcyadaFDWory53Sx1iEoSr24d8J9rJizik5V/TAIEyWVhfzt\nngeJjIxk4MCB9R2eolzWVA9BuWKUlJQw+8svSahsjUGYAAgQwcRUJDL1paln1a+qqmL27Nm8/vrr\nrFmzhj/S41NFuRhUD0G5YuTk5GDSmzE4jT7lfgRw4sRxn7IjR47Qu8dVaCv16KvMvGb8Jy3aNmf5\nymWYTKZLGbaiXDZUD0G5YsTFxeHGhV2W+pQXanNI7t7Vp+zWm2/DPz+cpPLONHa1om15L47uSOO1\n1167lCErymVFNQjKZc/pdLJ//35ycnJ+tp7RaOSFyS+QYtlOrszELstIE4fIMZ/k2UnPeuvl5eWx\nZ89uYjwJ3jIhBFGVjZj18acX7ToU5XKnGgTlsjZjxgwiw6O4qltfEuIac+0111JcXHze+g89/BDT\nPvkQQzsnaeH7aD0skY2bN3B6UkSPxwPi7IXxAoHb7b4o16EofwRqHYJy2frhhx8YNmQEzSs64ScC\ncUsXxw0HaNqrEctXLvtd5+7QpiNV+wQxNAJASskh005uffhGprwypS7CV5TLxq9dh6AaBOWyUVpa\nyr59+4iKiqJRo0YMGzqcw0vTacBPj3bc0s0W0woOHjpAw4YNfY6XUrJlyxYOHjxIUlISXbt2RZzR\nE8jPz8flcpGfn0+f3n2xOgPQ2Y3Y/YqJSYjihx/XYLPZLsn1Ksql8msbBDXLSLksvPzSy7zy8iv4\nGwIpqy6lS5cu5OflY5EBPmkPtUKLzeBPVlaWT4NQVlbGkMFD2L/nIIEihBJZQLOWzVi2Yin+/v6k\npaVx2+jb2bptKxqhISEhgXkL5pKSksLJkyfp3LkzQ4cORadT/0kof16qh6DUu7lz5zJ+zD20sHfB\nJCx4pIdjhv1oo1x4snQ0rm7trVshy9ljWU92bhZWq9Vbfu+E+1jyyXckOtohhEBKyWHjbgbf2pf3\nP3yfJgmJGDP9aeBujEBDNifJ8DvM4WOHCQ0N9Z7n2LFjbN68mZiYGHr27IlGo4bZlD8+lbpC+cN4\n6/W3iLE3wSQsAGiEhkbO5mRmZVJmK+Sobh8lsoBseZKD1q1MemGST2MA8NlnnxLraOZ9RCSEINbR\nlM8+/5zly5fjLHYR62mKRmhrZhSJOAJdYcycOROoGWged+ddtGnZlr9PeI5RQ2+gRbOWpKenX9qb\noSj1SDUISr3Lzc3DhMWnTCt0mA0W5s7/mqF3D6C0cRYhPSx88sXHPPbYo2edw+F0oD3jCagOPU6n\ngxMnTmB2Wc86Rl9p5tjRYwB89NFHLP5qCZ2r+tOkvB1tynviOaHnputvrsMrVZTLm2oQlHo3cPAA\nCvSZPmUlshCtQUP37t155713SDlykBWrv8PtdvPWW2+xevVqn1QTgwYMIlPjuxo5U3OcAf0G0rVr\nVwo1eXikx/uZlBK7rYjuPboD8OG7/yHKnoBO6IGaHkZDVyI7d+0kM9M3NkW5Ul3oBjkPCCG2CSEc\nQogZZ3zWXwiRIoSoEEKsrt1B7Xznia+tU1F7zIDfGL9yBXjm2WewBxRz1LCXApnNKY6QatnO2++8\n7R3kPXXqFE0bN+W+2//Gu09N44ZhN9GzW0/sdjsAjz7xKOm6I+yWG0iXR9mv3Uq6/iih4aFUVFTQ\noVM7tuq+Z4dcx0G5nRTDNgKi/Rg5ciQAdrsdHQafuAQCvUbPpk2bSE1V23UoV74L7SFkUrO5zfTT\nC4UQocA84FkgGNgGzP6Z83wB7ARCgGeAr4UQYRcYi1LH8vPzSU1Npbq6+pJ+b3R0NLv27uSGh0Zg\n6OiizXVNWbZyKaNHj/bWGTtmHLosGy3Lk0morkk1cWp3Fi++8CK5ublcP/IGIl3x2AggnWPkuNIJ\nczRg0+c7GXb1CDZs3EggoUQRixSSQpHHxzOnYzTW5D0aMXI4ucZTPr2OAnKwV9q554776NIhmXat\n23PixIlLem8U5VL6TbOMhBAvAQ2klHfUvh8P3CGl7F773grkA+2llClnHNsU2AuESinLasvWAZ/9\n0vabapbRxVFSUsKYW+9gxYrvMOnNCJ3gzX+9we23317foQFQXl5OSHAoPaqvQSt+Gicok8WcijjI\n+HvuZsarn9PE0YYKWc5WVpHMIIy1GU+rpZONfEc7uuMvggE4xRFi+4bz3fc1O7cWFRXRtVMylTlO\n/OwhVOntnKw+TBNa01A0wS7LOCL2QLCLrdu3Ehfn2wFOTU1l2ofTyMzIZODVAxk9erRKkqdcNi71\nLKOWwO7/vZFS2oGjteXnqnvsf41Brd3nqatcAqNvvIVdK/bTxTGIDva+NC5uy4P3PsTatWvrOzTK\ny8trZ/pIBL6LzDRocLvdbNm4BVtVEAD5ZBFOjLcxANALA1HEkUeWtyxKxrHqh1Xe90FBQezcs4N/\nvPEUba9PJLCFhQaaxjQUTUiTh9jGagzSjK7AQvOmzfm/af/nPfabb76hc4cuLHxvGbu/OsRzD75A\n107JlJeXX6S7oigXR101CDag5IyyEsDvd9ZFCDG+dtxiW15e3u8OVPGVnp7ODz/8QIKjFbraX9/+\nIoioygTeeO3NeouroqKC228dQ3hoOF06dEUjtWSInwaNpZRkGo4z6vqRNG/ZnEp9TYZTDVrcuM46\nnxsXGn7aSrMaJ1aT78wmq9XKhAkT+GLOFzRs0BCbJ4ByWUoaqXRlIM1FB1qKzrR39mbiQxPJyMig\nurqasXeMo1lFBxq5WhAjEkiyd6bwWCnvvffeRbo7inJx1FWDUA74n1HmD5T9zrpIKadJKTtJKTuF\nhalhhrqWnZ2NzeCHVvjuO2yRNk6mnbyo3+1wOHj77bfp0qEr3bv24KOPPvIml7vj9jtZM/dHOjsG\n0KayB8IlOCz3slOu47g8yBZWUWLM5/kXn+eBBx8g15DBKXmUUKLIJ5tSWeT9HrssI4s0wokBwCM9\npJlSGHPHmPPGdvXQwRRbcskhnUhiMQmz9zOL8CNME83ChQvZu3cvGpeWQPHT4jYhBKGVMcydPa+u\nb5miXFR1tU5/P+D9r6t2DKFxbfm56iYIIfxOe2zUFvi8jmJRLkBSUhL26jIqZDkW8VMOn2JDLn/p\nd/VF+97Zs2dzz133Iuw6YmUiRZTx5N6nWb5kOe+8/w6LFy+mS9VANGjYxmpiSCCaRuSSTgG5VFCK\nvzOQF154gflfL6Ciws4R9nCEvaCR7BTriDTHoEFDniubpEZJ7E/bRLAujILqXHr07M6rr7963vi6\ndu2KCHCT78gkyH2OHyKyZjW01WrF4XYgpfTJm+TCSZCfyomk/LFc6LRTnRDCBGgBrRDCJITQAfOB\nVkKIkbWfPwfsOXNAGUBKeQjYBUyqPf46oA0w9/dejHLhbDYbz056lhTrNnLkKUplIce0+ym1FfD4\nE4/V+fcVFRXx/KTnuWvMeFzlHmJlIinsJIs07JV2FsxfwMyZM/Ez+KMTOgrJRYueeJGEQRhpIBrT\nVnQjkjgcDgcfvvsftNkWesu/0IcRJNEBm9WPPft2M/U/LzH5/UmkHDrIPfdNoHnL5kQmhfLWO2+y\n9LulmM3ms+JzuVzcOOpG+vTsi6bUiEs4yOQEDlnlrVMhy8mVGQwfPpymTZsSFx9Luuaod4ZStXSS\nbU3jnvsn1Pn9U5SLSkr5q1/A84A84/V87WcDgBSgElgDxJ923IfAh6e9j6+tUwmkAgN+zfd37NhR\nKhfHvHnzZM/kXrJJfKIcf/cEefLkyTo9//bt22W71u2lUW+UAo204iejaST1GGRHrpIDxCjZn5Ey\nifYywBYorWab7ME1sjkdZSSxcoAY5fOKpanUopNW/GV/Rvp81kCbIF988UUppZSVlZWyQ9uOMsYS\nJ1uTLJNEBxloCZZTXp7ijc3j8cgvvvhCdu7QRYYGhkqr1ia7MdgbU4iIkAaNQcbrm8l4Q1NpNdnk\ne++95z3+yJEjMj62kYzwi5Lxfk2k1WSVDz/4sHQ6nfL1116XTROayYZRDeX9990vc3Nz6/S+Ksqv\nAWyTv+LfWJXcTrkoUlNTeeG5F9i4YRMRkRHs37+P2MokIonFg5tjHCCTNKKIpZlo53PsLtM6howa\nzOJ5Swm1x3CMA/RkqHecQ0rJBpZhwoIRM61EF5/j0+VROt/SmpmffsJHH33Ecw+9SHN7Z+8jnSpZ\nyQ7TGtJOnSA0NJQXX3iRf7/+bk0+JSzkcIpsTtGFfhiFGbd0sV6/lCeefBx/f39GjRpFo0aNfL7T\n4/Hwww8/kJOTQ48ePWjYsCE3jLyBH5dtJLqiMTr05BhOIiKr2bN/j0qxrVxSKrmdctFt2bKFAX0H\nEuQfRMukVsyaNQuAlJQUunZOZtvcfUSkN+bEtnT8KkOIFvFohAad0JNIG3ToONfPEZvGnz59+/Dv\naW8jmlThwcMOfiBfZlMs89nLZhxUIhAUkYtH+u5ylkuGdw/lRQsXY7MHkclxTsojVMhyTMJMqCGc\n9evXU1JSwqtTX6O5vTPhIgZ/EUSiaEM4MZziCFAzc0mv1XHvvffy+OOPn9UYAGg0Gvr27ctNN91E\nw4YNOXjwIMuWLiepojNBIgw/EUiT6jZUF8Cnn6ptOpXLk0r+rvwmO3bsYEDfATSoSKQ1PShPLeHh\nex8hLy+frZu3Em5vSJxsBgJ0Uo8fgT7HCyHwl0Hkk4mUbb2/3l2ymjx3Fr1796Zx48asWrkK11Ed\nOqnnGPtxUIUWHSa9mRJXIQEymF1sIEE2R4eeUxyhXFviXVRXXlHGUfYTShRadBznALEykSpZSXBw\nMPv37yfAGITJ4TsFNZQoTlAzBJZLBg0bxBIVFfWr78+2bdsI0UacNXvLZg/ixx9+5J577rnge64o\nF5vqISi/yfPPPk90ZRNiRAJGYSZERNLM3oHJz7/Ihh83EOKJ9Nb1I5ACcnyO90gPReThEtXsZB15\nMpNseZL91k2Mun4UDocDu91OTlYuGqmjikrslGHBDws2HNVVdOnamSLyqaSc3WxkK6vJ5iShoaE0\nik2ge3IP1q5dS0euorXoSgvRkWQGcYojaCyCHj16EB0dTamj+KxeRjklePBwzLCXNNtBps/86Kzd\n18rLy7n/3vsJsAVgMpoYfu1wjh+vWSsRFxeHnVLOfCTrNFbQpGmTuvxTKEqdUQ2C8pvs2LGTYOk7\nHdMq/JEeCA8Pw06ptzySOMopIUXuxC7LKJGFpJi30W9AP7bv2sbTrz+OuZMksJuR0AYhfP7Z57Ru\n2RqbzcaSZUs4wh5KKMCEBQeVNKUtXRjAnl17+Gj6f4mKj8CNCy1ajJiJyEmgaVFH8jaX4q72oEPv\njcUoTMTQiMGDB6PRaIiPj6dnzx4cM+7DJWtyOBXLfNINR0ge0Ilbn7iR/Qf30a1bN59rlVIyZPAQ\nFn28nNb2HnR1DuLgsuMkd+lGcXExvXr1IiwmlBO6g7ilCyklOfIU+fos7h5/90X8yyjKb6caBOU3\nSUhIoJRin7IqWYnLU82TzzzJKcthymXNgnQPbkxYyCKNbZrVnAo7yD1P38U3ixfSpk0bHnvsMTZu\n3UBVVRUZh7II8ITQnatpQzeMmOnJUDqKq2hLN0yY2c4aCsgm1BVNSkoK+fn5tKQzHjx04CpCRAQW\nYSNeJNGAxpzksE+cAi1rVq/B6XQCMPvr2bQb1IrNxu/Yavmek6EH+eTTGSxbsYwXXnyBBg0anHX9\nW7duZf/uAyQ62qHHgBsXDd2JmCv8+PjjjxFC8P2alSRc1YCNhmVsNC7D1bSMZd8tJSYm5iL9VRTl\n91FjCMpv8o9Jz3D9dTdgqjATSChVVHDEtJv4uEa8/+8P6NWvB2t/WIejwonT7SCCBrSlB0WeXI7Z\n9zFy5EgMhpp00wcOHGDy85PZs3svbumiA73RCT2pcheNqFl/UChz2csmoogjkljyyKLEVcDGjRsJ\n8oRjxooJq8+KYoBQIjnEHu97l6wmixNYcqy8+cabPPX0UwQEBDD/m3kUFhZSVFREfHw8Wq3vs/8z\nHThwAD+CSGEnOZyq3ZxHElQRzu4dNWm9IiMj+W7lcoqLi6mqqiIiIuKsx06KcjlRPQTlNxk0aBDT\npv+HnKhjrDcsZofxB4pdhThSBQXrKtmzMgWzyYTQQk+uoaXojE7oCBPRRDrjeOuNtwBYtGgRyZ27\nsXXeXhp4EjBh8W5S48aFHgNSSlLZRQs60VS0JVrE01Z0I4woCvML0VTqMGKmCrv3sc//lFFMJeUc\nlNvYJlezjiW4qEbj1PPfaR/51A0ODqZx48a/2BhAzQrvHEcmTqroztX0EkNpR08KyUVn9P2dFRgY\nSGRkpGoMlMueahCU3+zGG2/kZEYa6Znp+NlstHYlE08zwkUMiY62GApteJwSg/BNA21x+XE49TBu\nt5u7xt5N04r2NPI0pwGNcVCJUzoACCGSTE5QRQXVOAjFd5ZPDAlkZ2ZTaslHj54wojnANhyyCikl\nhTKHE6TQmFZkcwotejrSm070xYiRU6dOUVlZ+ZuuPSkpiWq3kxZ0wiBq9lTwE4E0pQ0H9561QF9R\n/hBUg6D8LkII8vPzqXa4fRK8AUS4G+LGRbV0+pSXGgrp2r0rhw8fxllZTVDt3kgGYSSaeHaxnhJZ\nQDjRlFPCHjbhxoUH35lA1TgpL7Xj9DjZZ9lEMBG4cLGeJaxmAfvZRhg1C9t0GGhLd/xFEFbhR5Lo\nQIAmiDlz5njP5/F4WLt2LXPnziUrK4ufk5+fj5/FD73w3WXNiv8vHqsolyvVICi/W2BgII7qKtxn\nTN10UIUeI7tYT5HMo0KWc0zuJ0+XwUMPP4Sfnx9Ol8NnymcibbDixw7WsZHvqKKSMooxmywc1xz0\nTuN0SRfHOEC8TCLCEYt/pI3wXjaadal55BNLIjYCcFFNEKEEEYZG+P7fPbA6nC2btgBw9OhREhOa\ncv21N/LI2Cdo0qgJf3/672dNG/2f2NhY0OIdOP+fQm2Od1GcovzRqAZB+d0iIiLo3qM7afoU70b2\nDlnFEfaRQAsiaUgqu9jOD+SRxZg7byc6OpqYmBjat2/PSd2hn/6hp5pSCokijiaa1viZ/Xj/vfc4\ncuIwAS1MbNAsZZdcz3qWYCOABjQm1tOUzMxM3v3wHTp27oDJYCJDHEMgCCECl9GBXVt6VtxOSwWJ\nzRKRUjL82hHoT/nRuqwHTcs60MnRn/+88398880357xmg8HAlKkvk2rZTo48RbksIU2kkms5xaQX\nJl28m60oF5HKZaTUifz8fIYNHc6BfQfw0weQX5mL8GhIdg3yyUG0x7aemV99zNVXX82RI0f48MMP\nmTljJlUVToL0weQ7cknuloxeqyc8Mpx777/HuwZASklcTByGLH8iicUsrN7yreaV2AKslObYkdKD\nEQse4UYXIJj6z1d4dcpreE7qaehKRIOGLHGCHP8THD52mOzsbHp06UlHez+cVJFDOkbMuKgmtn8o\ny1YuO+91f/PNN7z68mukp6fTrUc3nn9xEklJSRf/hivKBfi1uYzUtFOlToSGhrJh83oOHDhARkYG\n7dq147bRt7NnwzYiK+IQCHLMJ0lq05RBgwYxffp0HnrgYcJcMfi5wnAaM2nWOZENn6w77zx9IQSj\nbxvNF/+a65NqIp8sdAYd2dk5NKYFQYRTQgHH5EFMVQZatGjBDz+uYdwd41i1eikAbVu3ZfaMNQQH\nB3P48GGMWhP72UIuGRgx48SBQCDSm/7sdQ8bNoxhw4bV3Y1UlHqkegjKReN0Onn//ff55KOZeDwe\nbr3jFv72t79ht9uJbRBH26qeWEXNzqkuWc1uyzpuGTuadu3acd111xEcHHzWOYuLi0nu3I3yrEr8\n7ME4jZXk6TIxGgyEF8UTLeK9dXNlBofEbt5493Xuu+8+AOx2Oy6Xi4CAAG89h8NBUEAQboekE1dh\nEX64pZtUdpGvyaS8qgy9Xs+ZNm3axPz58zGZTIwePZpmzZrV7Q1UlDqisp0q9c5gMPDwww+zc+8O\ndu/fxeOPP47JZGL58uWE6SO9jQGATugJqYhm5gefMfmhqcQ2iGPRokVnnfP48eNERUWR58wi3XyE\nxr0asnf/HopLS7xbZP5PGNFUyQpmfPwJDkfNVFar1erTGAAYjUasZhuJtMFSG5NWaGlGW9weN3Pn\n+u7dJKXk/nsf4JoBQ/n6n4v4dMocOrXvzAcffFAn901R6kudNQhCiPIzXm4hxDvnqXtH7een1+9T\nV7Eolze9Xo9HeM4q9+Ah0B1KYkU7WlR24eYbR1NW9tNW26mpqfTp3ZfsdcUkVw+meWVn9mzYx9tv\n/ovw0DDsZ2zLXUE5GrSk783m2X88+7MxuVwuLFh9yrRChx4D6enpPuXr16/ni1lf0M7eiwTZggR3\nK9pU9uCxRx4nJ8c3iZ+i/JHUWYMgpbT97wVEULMb2lc/c8jG04+RUq6pq1iUy9vVV19NsTufElng\nLXPIKjI4TgQ1eYMCRDBB2lCWLl3qrTN1yquEV8XSQDTGIIwEiGCaVXRk2rRp3PvAvRw37adKVnrP\nt5+tNCCBeEcS0//7cU25w4HHc3Zj1LVbF7I55VNWJotx4uCmm27yKf96zteEVER5V1QDWISNcF0U\nS5Ys+Z13R1Hqz8V6ZDQKyAXWXaTzK39gfn5+fDH7Cw5atnLYupODmm1sZDkNSCBAhHjrCTS4XC7v\n+53bdxLg8h1XMAgT/sYABg0axLgH72QDS9kgl7GR5QQRSiJt0GOkzF5K25ZtsVqsBPgFMPGhid7H\nSAAfTvuQXF06KXIHRTKPdHmMHaxj+IhhZyW30xv0SM3ZY28e5DnHGhTlj+JiNQhjgJny50es2wsh\n8oUQh4QQzwohzjnjSQgxXgixTQixLS8v7+JEq1xyQ4cO5WT6SSb9+xlG3j8Mq9lKQ37aJ8Auy8iv\nzmbw4MHesuYtm1Om8c2w6pLVlDqKadSoEVNffYXkrslE0JCeDCFRtAFgP1twV7vJPlBAF88A2lb0\n4qv/zmfcHeO854mPjyfl8EFaDkzkmG0v5ZF5vPbWVObNn3dW7KNvGU2eIcPbGwEolUUUunO49tpr\n6+weKcol92s2Xr6QFxALuIFGP1MnAWhETYPUGjgAPP1L5+7YseNv32VauWx5PB45/q7xMtAaLBM0\nLWQjQ5K0mm1y+vTpPvV27twp/Sz+sg3dZH9Gyh5cI6PNsXLMbWO8dXbv3i0DbAEyUBMijZilDp00\nY5NNaC1jSZR6DLI1ybIPw6XFZJVZWVm/KeZXprwirSarjDc3lXG2JtJqtsn58+f/ntugKBcNsE3+\nin+/63zaqRDiH8BAKeVVF3DMTcDjUsqOP1dPTTu9ckkp2bx5M98s/AaL1cLNN99M48aNz6q3evVq\n/nbfg6QcOojZZGbChAlMmTrFm0rb6XTSrnU7So5WYHRbKCCHTvTxpq0olUXsZB09GUpKwFbmLp1z\n1uY3v1ZaWhqLFy/GZDIxYsSIc06TVZTLQX0uTLsdmHqBx0hA5Qb+ExNCkJycTHJy8s/W69u3L/sO\n7sXhcKDX69FofJ96zp8/n+LMMlq4u7KXTTSksU8OI38RhEX6kU8WxVWFJCYm/uaY4+LivOsbFOVK\nUKdjCEKI7kAMPz+7CCHENUKIiNr/nQQ8Cyysy1iUK5vRaDyrMQDYsH4D1vIghBBo0OI+I0MqgJtq\n0o2HGTv2TkJDQ8/6XFH+rOp6UHkMME9K6TMhXAgRW7vWILa2qD+wRwhhB5YA84ApdRyL8ifUKKER\n1eaawd5IGnKSwz7pt3NlBpWigidfeJx/v/vv+gpTUS5LKnWFckUpKCggMSGR6NImRNCQo+wjg+ME\nE44TB+WimOUrl9OvX7/6DlVRLhmVukL5UwoJCWHl6pWQVMlm03KydCcwGAw4/MrpOrAjh44eUo2B\nopyHynaqXHE6dOjAvoN7yczMRKfTER4eXt8hKcofgmoQlCtWdHR0fYegKH8o6pGRoiiKAqgGQVEU\nRamlGgRFUZRzSE9PZ//+/T4JFq90qkFQFEU5TXZ2Nr3796dpy5b0GDSQyIYNz9ok6UqlBpUVRVFq\nSSkZMnw4JwP9iZj0d4ROR9WJNMbcfTdNmjShbdu29R3iRaV6CIqiKLX27NnD0bQT+F0zCKGr+b1s\nio/D1KMb/3rvvXqO7uJTDYJPYik9AAAgAElEQVSiKEqtrKwsTOHhiDPyZGnDQjiZkX6eo64cqkFQ\nFEWp1bFjR8pOnMBd5rs/t3t/CgOv6lM/QV1CagxBURSlVlhYGA89+BAfTJuOsV8ftP7+OHfsxJJf\nwITx4+s7vItO9RAURVFOM2XyZKa9/k+anDhFwA/rGNezNzs2byYwMLC+Q7voVLZTRVGUK1y9ZDsV\nQqwRQlTV7n1QLoRIPU89IYR4VQhRUPt6TQihdkxTFEWpRxfjkdEDUkpb7avZeeqMB0YAbYE2wLXA\nhIsQi6IoivIr1dcYwhjgDSllupQyA3gDuKOeYlEU5QqVn5/PgQMHqKqqqu9Q/hAuRoPwihAiXwix\nXgjR5zx1WgK7T3u/u7bsLEKI8UKIbUKIbXl5eXUcqqIoV6KKigpuGD2aho0a0WPwIMKjo/jXO+/U\nd1iXvbqedvokcABwAjcB3woh2kkpj55RzwaUnPa+BLAJIYQ8Y5RbSjkNmAY1g8p1HK+iKFegcfdM\n4Pujh4l49ik0JhPOnFyemzKF+NhYhg8fXt/hXbbqtIcgpdwspSyTUjqklJ8A64Eh56haDvif9t4f\nKD+zMVAURblQxcXFLJi/ANt1w9GYTAAYIsIxXj2AV996q56ju7xd7DEECZxr9tB+agaU/6dtbZmi\nKMrvUlBQgMFqQWsx+5Trw8PJzMysp6j+GOqsQRBCBAohBgshTEIInRDiFqA3sPwc1WcCjwghYoQQ\n0cCjwIy6ikVRlD+vuLg4dBIc6Rk+5Y69++nVo0c9RfXHUJdjCHrgJSAJcAMpwAgpZaoQohewVEpp\nq637HyAB2Fv7/r+1ZYqiKL+LTqfjn1On8uCTT2Ia2A9DZASOAym4tm7n+c2b6zu8y5paqawoyhVp\nxYoVvPrmm5xMP0Wvbt155qmnSEhIqO+w6sWvXamsktspinJFGjhwIAMHDqzz8544cQKXy0Xjxo25\n0hIsqOR2iqIov0JKSgqtO3agVYcOtO+WTJPmzdl8hT2CUj0ERVGUX1BVVcVV/fvj6dmNsNtuBiEo\n37WHQUOGcOzQIUJCQuo7xDqhegiKoii/4Ntvv8UTHIRfj24IjQYhBNb2bTE0S+TTTz+t7/DqjGoQ\nFEVRfkFmZiaEnt0LcAcHcSr9ytlaUzUIiqL8aezbt48R119PdHwcXXr2ZOHChb/quG7duuE8mIp0\nubxl0uOBlEP06tnzYoV7yalpp4qi/Cns27eP7r17Y7iqJ6YWzanOyqJyyXf87e67iQgLIzw8nOHD\nh2M2m886VkrJX/56HRtSUzFc1Quh0+Jcv4kEk5mNa9ei013ew7G/dtqpahAURflT+OuNN7LWUYF/\nvz7eMkdGJllvv0tw186IklJkTi6rvvuONm3anHV8dXU1H3zwAR99OguXy8XoUdcz8eGHsVgsl+4i\nfiPVICiK8qeza9cuTpw4Qbt27YiPj/f5LKZRPOLmGzBERviUn5w0maiH7kcfHEz51u34b9vJ4QMH\nWLFiBW++8w5ZOTkM7tePxx99lLCwsEt3MXWoXrbQVBRFqQ+FhYUk9+pF78GDmTD5BVq0bcvtY+/E\n7XZ768TGxuHMyvI5zl1ahnQ40VqtAFg7dSCvqIi/P/MMo26/je1+FnK7dmT6hh9p26kTZ+7Jkp+f\nz+1jx2L198fiZ+Pm224jOzv7N19HSUkJL06eTIfkZPpdfTXz58/nUv5oVz0ERVEuG0ePHuW9Dz7g\n8PHj9OnRg7vGjSMgIOBnj3G73Yy6+WbW5WTjd91fEBoNHoeD0ukzeWb8BB6dOBGARYsWMfqucfjd\nfgvGBjG4S8vI+/xL9GFhhIwcAdSMFRROfYNqu52Qh+5HHxbq/Z7Sr+Yx/qo+THnpZQBcLhet27cn\nNyQIa78+oBFUrFmH38lTpOzdh9FovKBrLy8vp0OXLhT42TB0bI+7vBzn6h+47447mTJ58gWd60yq\nh6Aoyh/KmjVraNepE7P27mKzxcjU2V/Qsl07cnJyOHXqFDNnzmTBggXe7TC3bNlC5+7d0RsMLJw3\nD9uQQQhNzT9pGqMR06ABvP3uu7hqZwZde+21vPHSy1TN/Jy8F18he8preAoKsfXsTu4nn5I+9Z9k\nvf0urvJy3BazT2MAoG/diuWrVnvfL126lFxHFf7XDUMXGIDO3x//YUMpMxiYP3/+BV//9OnTKTQb\n8R99A+akptg6dSBgwl28/a+3ycnJ+a239YKoBkFRlHonpWTshAlYr78Ov6HXYOvcEb9bbqIyPpZr\nR4ygWcuWPPbeu9z9j2eIiY1l3rx59Bs0iFSdIPT2W0CvQ5zxi1xjs5KRkY41MIC2HTsya9Ys7hgz\nhpz0dA7s3ElWejrNGzQk6/W30NhshIy6DmunDlS53VQXl/hMMQVwFRQQExXpfb9v3z48UZFwxlMW\nd8MY9u+/8O1dlq1ahaZlC5/8SFp/P/wSEtiyZcsFn++3uLznSimK8qeQkZFBTm4O4S1b+JSLiHB2\nLl1OzBOPoPX3A8C+Zx833XYr1S43luISild8D0JQsWcv1nY/7btVtnkrwmDA79oh7F/6Hfc99RQf\nzZzJyqVLadiwIQDZ+fn49+5J8LChAJibNMYUF0vWe/+hYOFiQoYPReh0OLOzKV/xPRPnzgPgo+nT\nee2tNykpLqF05y4C+vQmoH9fhBDoMrJoflvzC74HMZGRbDmVRum69ZRt3IzbXoGpSQIiO5uIiIhf\nPkEdqLMGQQhhBN4HBgDBwBHg71LKpeeoewfwEVB5WvG1Uso1dRWPoih/HFarFXe1C+l0+vzSr9i7\nj4D+/byNAYC1TSuKvl1M+NBrsLarmR6aP28heZ99SVXaSYwxMVSmpFJ19BjG+HiERkPIdcMoXfsj\ne9JOMH/+fK6//npycnLIzcsj8saRPrEYYxsidFqqDh3m5KTJ6Pz9qS4sZOjAQfTt25c5c+Yw8Zm/\n43fbaILiYnHm5JI363NkdTVaCeZyO3/9619/1XWXlpayatUq9Ho9d48dyyf9+iFCggn+63B0gYGU\nb9tO2b79xMTE1MFd/mV1+chIB5wCrgICgGeBOUKI+PPU3yiltJ32WlOHsSiK8gcSFBREn759KV68\nrGYFMOCpqsKZkYnGdPbgrMZqRej13vch1w1DGI147BVU7DuAPiqSqIkP4jx1CkNMNIaYaFxFRYhW\nLZj37TcAlJWVIZCUrFpD2ZateBwO7/dKlwt3eTmho28i5MZRaG027ho3DoDnp0zBNHggFQdTyHr3\nQ4qXfod/z+6UfL+GHjZ/Nq1bh6l2L+dzkVKyePFiuvbsSUhUFGMefogxjz3KgKsHg9tN5D13YW7S\nGH1oCEFXDyIwuStv//vfdXavf06d9RCklHbg+dOKFgkhjgMdgRN19T2KolyZZk2fTmyTJqTv248+\nMpKqw0cwREdRtnEzts4dEVotAM6sbJxZ2Zia/LTZjRACjdmMrK4m6C9D8FRVUTD7a4yxDTHGRFO4\naAlSQuE3i5kjBHZ7BSUlJQijCa3Nhn3XXoq+XUr4uDGUbdiEqUkTPA4HhfMXIrQacFZz/PhxAE4e\nP44jIx1TQiMC+vfBVVhE0bLvEMBnM2Zgs9nOdXle99x/P198sxBD92QCGgyh7McNeAIC0DdLRGzb\n4Z0C+z+6pols+KOPIQghIoCmwPlGV9oLIfKBQmAW8IqU0nWeuoqi/EFIKfn+++9Ztnw5QUFB3Hbr\nrcTGxp71+arVqwkLDWX06NFEREQQFhbGjddfz4KD+3A5q7G0bknYLTeRO30mmW/+G1unjrhLSynf\nuBmjvx95sz6nOi8fQ2QE5tYtobSMoUkt+Oatf+NwVmNunkTQ4AEUrfie0jXrCLp2CH49kpHVLr6f\n/gnS46HBc097G5qyzVvJ/mAahuhoqnNysXVqj75DO+w7duFyuli4dCkPP/wwAUFBlIaFEnbLTd5r\nMic2IfP1N39xw5zdu3fz+Zw5hDwxEU1tL8LWqQMZU9/A3CIJV2Eh0uVCnJYKw52ZRdMmTeryT3Re\nF2WWkRBCD3wGfCKlTDlHlbVAKyAcGAncDDx+nnONF0JsE0JsO3NRiKIolxe3282IUaO4fuydTD+w\nl7e+W0bz1q1ZsGABUDN3f+jw4Vw/biwf7NjKlLlf0aRZM1auXAnAs08/DalH8JSXo7XZEFot4ePG\nEDhoAOXbtlOy9ke0RgOOklLMLZoTPuZWjI0TKJg9F4mHzVs246ysQuN2o0vPIOtf71H2/WrMzZsR\n0KcXGr0ercWM2+XCf1B/b2MAYOvcEY3BgLu4hLBbbiLkryPw755M5P0TMDaKJ6926qdfQADWTh18\nrlsfHoYtMpI9e/b87P1ZsWIFxtYtvY0BgMZgwNq+LdVZWej9/Cj68ivc5Xakx0PFvgNU/riBRx96\nqC7+PL+oznsIQggNNb/4ncAD56ojpTx22tu9QogXqWkQXjlH3WnANKhZmFbX8SqKUne++uor1u3e\nReDDD3h/5Ro7tuf2sWPJGTyYL7/8kk2HUgl86H7v57r2bbnp1lvJTk+ncePGbF6/ngcnTmTlmjUE\nDuqP1mbDVViI0OuJffFZst+fRvj1f8XSqiUAhugotBYLBfMWUNQghgZjbwcEpd+vpuGJk7Ru1YoN\n4ux/OgRn/5qXUiIdDswtkn6qJwT+vbpT+m3N/JimTRqzoajI9zi3m8rCQkbdfDMtWrbguaeeplev\nXmedPyAgAFFRcVa5u6wcXUgQmioHfWLjWT7lVSSCmIYN+PSrr2jVqtWv/Av8PnXaQxA1/aWPgAhg\npJSy+lceKuEcfx1FUf5QPpszB22XTj6PPIxxsRjDw/nxxx+ZNXs2uuQuPp+bE5sgLWa2bt0KQLNm\nzVi+ZAlPPvoohW+9Q8mS5ZSsWkPIdcPQmM04MzIwt/Cd1mlp1QJPlYOQ64ahtVrRWi0EDRtKiU5L\nTGQkmhNpvvXbtKJ4xffI01Jb2HfsAo8Hj9MJp5UDeCorCQ4JBuDh+x/AuXot1bk1Tyyk203RoiVg\nsyFuHMkufxtDRoxg0aJFZ92fkSNHUpl6mMrDR7xlVSfSsO/eTfWW7Tz84IMs+PprCvPyyUhL48iB\ngwwYMOCC/ga/R133ED4AmgMDpJSV56skhLgG2CGlzBFCJFEzI+mrOo5FUZRLzGAwIN1nDwV6XC70\nej1arRZqZxGdTrrdNZ+dZsrkyYwcMYI333qLLyoq0YUEU3EwBaHTU52biyHyp0VizuwctOfIOipj\nGxIaGoq5tIySbxZj6dENWe2EUxkYS0rJnvoGxratqM7JperYcfTBwVQXFlH8/WoCBw1ACIHH4aBi\nxSrufeJJAPr168eU557jqWeewRQWSklWNsLfj6j770Hr74chJhptgD8PP/441157rU88wcHBLPj6\na66/6SaqQ0Nwu93YMzLo26s3zz3zDFdddRUAZrP5nGm4L7Y66yEIIeKACUA7IFsIUV77ukUIEVv7\nv/83stQf2COEsANLgHnAlLqKRVGU+nHnrbfi2rAJd8VPvwcrDhxElJfTo0cPxt56K9XrN3qneALY\n9+7DBHTs2BGoGWdYuHAhL7/8MidOnKCsshJ9ZATl23dS8NU8rJ06kP/l17iKSwCoLigk//M5oNX4\nJIKTUuI6cpR27dqxZf16hsbGU/bef3DO+Iyx/QeQfuwYC2bOpKtGjzWvgGD/AGRBIeFjb6di1x4y\nX3uT3BmzOPX8SxirHNx9993ecz9w//3kZGTwzScz0QFRf7vXZ62EOakZJ44c8abZON2AAQPIzsjg\ni3fe5ct336OkoJDvv/vO2xjUJ5XcTlGUOiOl5MGJD/PxzFmYW7VAlNtxpKWx5Jtv6dGjBx6PhzHj\nxrFg0SKMLZsjSstwpp1k+eLFJCcnk5+fT88+fcivduKJa4gmPZPyU+lY+l1F8YpVaIxGGk76O0VL\nllO2fgMasxl3RQV4JMJkwtauDQH9+wCCkhXfYzuZzonDh9GftmbhfPLy8ohr0pjIyZOQHg9Vx47j\nLi5BGxiAc85cCrLPnU+oUbOmOK4eiKnxT9Ngq3PzKHl/GsX5+Wg09Z8hSO2HoChKvUlJSWHlypUE\nBgYyYsSIs+bm79ixg8mTJ/P9mtWUl5bRID6eVyZPZvnKlSw5drQma2ntFM6ihYuoTEkl6Lph5E6f\nSeyUF7wZTd2lZTjS0iicuxC9EGCz4igsAukhqUVL1qxYQXh4+K+K2e12ExYVhWXs7Riio7zlZVu2\nkZSZw7pVq8553IcffsjTr07Fdvto9CEhuEpKKf9iDveNup6XXnzxN97BuvVrGwSVy0hRlDqXlJRE\nUlLSeT9fs3Ytq3fuwO/usYTERFN19Bj3TpxItd1O2BOP+Mzn9x/Yj+I1a3GmHgKNoOSHHwns2xuN\n0Qg2DyWr16ITsG7NGj7//HOys7O55ZZbGDp06Dm/2+l0snTpUvLy8ujduzdNmzYFQKvVMunZZ3nu\ntVexXDcMY0w0FSmpVC5ZxkvzF5z3WiZMmEBufj6vv/FPtGYzznI7d999Fy9MmvQb7179UT0ERVEu\nKY/HQ1h0NOYxt2CIifaWV+w7QN6MWcQ8+xS60/ZAcFdUkvnci9wxdiyz58zB7nSgCwpCHxZK5aHD\nGOPiCHdWIwSUejzoIsKxp6Ry7TXX8MyTT7Fx40YiIiK45pprOHToEP0HD8YTEIAICsR+4CA333AD\nSU2b8tlXc9BoNDRPaMyGrVvJTE+nRevWvP7yy/Tv3/8Xr6uyspJFixbxn+nTOXjwIM2aNeP5Z56h\nd+/eF+U+XgjVQ1AU5bJUWVlJaVERAac1BgDG+Dg0Oh321WvxH36tt5dgX72WocOHM+6OO0jPzGT1\nmtXgdCIlBDRNxJ12Er/YOLIbxhAwoC8ApioHC157gwXffot/69bIwiK099+PTqdD9umFX9fOAJgr\nKpg55XXMDWMw9+4FHjfH122ge4vmHE1JweVysXz5cv7v//6P7t2707Jly3Ne0969e3n0ySdZtXYt\n+pgoAq4ZyN6SUoZedx2zZ81iyJAhF/GO1p36H+1QFOVPxWKxEBIejiPtpLfMkZ5B3uez8fP3R3/o\nMKUf/pfiJcsomzYdy5GjjBo+nIFDh7DVqCN47Bj0sQ2pTj3EHX36sf6HtRw5ehRbn58WglXs3oO0\nWIj8x1PYbvgrfveMw9OzG9l5eVg6d/zpe9NOgtVC8F13YmneDEvLFgTcdQfrNm/mq6++Ir5JE+54\n7FH+8ckMuvbuzejbb/fZlhNg7dq1dOvdm20aCLvzNkyNGpH36RcYGsRgHTmCR5588uLf1DqiegiK\n8ieUlpbGq/98nXUbNtIoPp4nJk6kZ8+el+S7hRC89PzzPPLsP/BcNwxnZhbFK1fj3yMZkdQU587d\nxPsH8Jeu3WlxZwuGDx9Oq/btsY76K5aWNQvSTAmNKP1xA7sPHsBqtaLV6+G0dQzl23cSOLB/zThD\nLWtyFwoXLcWVl4/WasGZlU3FvgNY2rbxSWEhdDr0zZvx0COP4ErujH/vmvtidjpZNm0606dP95mC\nev/EiVhGXIutfbuaeolN0PrZKFqyjIi77uTQfz/GfY51Fpcj1UNQlD+ZY8eO0a5TJ+akHqSgZzIb\ntXDNiBHMnj37ksVw17hxfPjPN7CsXEPRoqVEP/I3goZcTUDvngTcP4ET+Xm0adOGG2+8EY/HQ9qx\nY5ibN/M5h7VtazZv3ER8fDzhoaFU7Pspj6Z0uRAGg++XCoHQaSmc/w3pL02laOlyyrdsozor6+wA\n8wspKi7C1rO7t0hjMGDo05v/zPjYW+Z0OjmwezfWNq19Y+vQjqojx6jOziEoLOwP0RiA6iEoyp/O\ncy++iLZzR/yvHgiAqXEC+phoHnz0EUaNGnXJ/vEaPXo0BoOB+6a8jD4kxFsudDo0Hdox95uFjBw5\nEqPRiNFsxlVU5FOvOjeP8IgIhBDM/OgjrvnLX/AcPooMDUFrr6BszVrMzRK9+yxXph5CL6G6oIAG\n/3garc2Kq7SU9Jdexb5rN5a2bUBK7Nt24MrIxGC1wRnZS4XR4LPYTK/XY7ZacRUX+8TmKihEa7Vg\nn7uAxydOvFi3sM6pHoKi/Mn8sHYtpja+ydJMjeKpqHKQnp5+SWOxWCzIc6zmdVdUsGjRYl57/XWk\nlNwzfjz2ed/gttsBcBUVU/ntEh6rzQLas2dPUvbt44F+A/hLUCiTJk7ElZ5O5hv/omTVGvK//Irc\n/87A43ETOuqvaG01ew7o/P0JH3s7eZ/P5tSzL3LqH89j3bKdNStX4m+xUHngoDcmKSXOjZu5eeQo\nb5kQgvHjx1OxcBGe2utw2+3kz5mLprKKe2+6maeeeOKi3b+6pnoIivInEx4RQXpBoc/iK3dFBa6q\nKoKCgi5pLP3798eTX0DF/gNYavdTdhUVU/bjRgKHDGbqx9PZe+AA06dNo7S8nE9f+SfGgAAcJSU8\nMnEi9957r/dcMTExTHruOQDGjBuHtXMn9PGxVB09ji4kmMBrBlO8/Dt0QYE+MZibNUV6JGF33ErV\nnv20CQ2jXbt2fDZjBteOGIG7zSE8wUGQcohYmx8PPfigz/FTX36ZvPx85r70KtbICMqzsrlx1Che\nnTKF//vvf+mQnIy/vz+D+vThx82bSEk9RKuWLZn097/TuXPni3yHL4xah6AofzJffvklEx5/DP9x\nY9AHB+NxOCibu5BBiU35YtasSx7Ppk2bGDLsL7j8/XEZ9DhOnCTw6kEE9OmFx+Eg96VXObB7N9HR\n0Tz+1FN8NP0jKsrK6dClC++8+SbdunU765zBERFYJ4xDH/rTYxzpdpP21LP49+1N8JCrveUVB1PJ\nnTGTuKkv4SkrJ+/VN6goKwMgIyODj2fM4FRGBl07dWL06NHn3R4zOzub48ePk5iYiMVioWPXruSa\nTRg6d8Btr6Bo8VL0YaEE/2UoVceOU7VyNYsXLLgkOYzUOgRFUc7pxhtv5OiJ40x5ZSqmkGAq8wsY\nNHgw//3ww3qJJzk5maxT6XS/6ioOG3WE3XITWr+aRHEaoxG/xgns2bOHZ194gcXbtxH0t/sICwrk\n5M7dDBoyhC0bNtC8uW86bLPZ7H2E8z8ehxOBpGztejwVlViaN8ORkUXJqtWYExMRQuCurMR8WtbU\n6OhojEYjX339/+3dd3gUVffA8e9N7wUInQRCUTqSqHQQK2D7gYX6oiCgiI2OlY4o6IvSlZcuIkhR\nQQRRVBSRJgSQIp2EQAJJSM9m9/z+yBISIJCQhQQ9n+eZ52Fn79x7ZtjN2Zk7c+9S5syZzbC332Lk\n2+/Qp3fvy/ajbNmylLWPwDp9+nTOOBv8ujyd/TyF5201ODl6HE7eXvg1b4qTtxcDhg5l66ZNDj2e\nhaF9CEr9yxhjeGPoME5HRrJ2yVIO7d/PiiVL8L5kLt+byd3dnZbNmuHu7pGdDADEZiM1MhJvb2++\n/PJL/Lp2xDWoFMbFBZ87w3Bv2pj3Pvjgsvp6dO9O6rr1SGbWUNwiQvK672nTth13hoWRvmMn55au\n4PwPG3ArW4agbp2yJrn5bh3Pdu+eXc+HkyYxZvJkvHr8hwrvjsal45MMHjGcBQsWXHV/1v74I061\nauYagsPZ2wuPqqGk2+dm8KpTm53bthXquDmaniEo9S/l4+OTPeR0cdCvb18+CQvDVCiHd/162NLS\nSF79HXVr1sLDwwPvcmVzTT0J4BpSiYg/d19W15uvv87W7dvZOG4CXtWrkXEykkpBQcxZvJRSpUqx\nY8cOtm/fzkdTp3I0KorUL1eQcugIje68k1EjRgBZSWTs+PF4d+uU3d/iHlwJ2+OPMmLsWLp27Zrn\nvlSqUAH5K3dcIoIl9mx2wrOcPkOpHHM6XEl8fDzz5s1j5549NKxXj27duuHn53ftg3mdNCEopYqF\n0NBQ1nzzDb1e7MuhxV9ijOGRxx7lk6nTyMjIIDnqFF4pqTh7XZw4xnL4KA3r1busLnd3d779+mt2\n7tzJzp07CQ0NpWnTptm/2O+44w7uuOMOevTowbZt2zhw4AB16tShXo66LBYLcTExBF86xEZwRU4e\nP87VPN+rF7OaNMHt9tvwqFIZsVpJ+GED2Gy4V6lM5rk4UpZ/xVtXuSX10KFDNG7eHKlUEVulCnw1\ndzajxo1j86+/EhIScu0Deh0c2qlsjClB1hSaDwCxwDAR+ewK5QzwLvCcfdUsYIhcIxjtVFbq3yEu\nLg4PD49cs4Y99/zzLPvlJ7weaYdLiUCStv9Jxtr1bNu8merVq9+QOEKqVSOjzf14Vq+WvS75z12U\n2/0Xf/7xx1W3XblyJT379MHq6oolJRkfL28S4uJw9/PFkpzCq6+8wugRI3JdVsrpwYcfZpsz+N17\nT/a682vW0czLhxVLlxZoP4pkPgRjzCKy+iV6kjVz2iqgiYjsuaRcH6A/WTOnCbAO+EhErtqrpQlB\nqX+vzMxMxowbx5Tp04g/e45GzZry3/fep2HDhjeszUWff07vl1/G67GH8QgJJvXvQ6R8vZplixbx\nwAMP5Cvm3bt34+vrS9WqVUlOTiYqKooKFSrgdYUpPy8QEVzd3Kg4Zniuy2TWpGROj36XtJSUAu3H\nTU8IxhhvIA6oIyIH7OvmA5EiMvSSsr8Bc0Rkpv11T6CXiDS6WhuaEJRSN9vKlSt5e/Rojhz6m9tq\n1mLciBE3fOJ7EcHLx4egYQNxydFnYDl7jviPppIYF1eg+orittMagPVCMrDbCVzpJtva9vdylrvi\nuLLGmN5Ab4Dg4OArFVFKqRvmscce47HHHrupbRpjeKpjR7757nv8nvg/jDGIzUbK2u/p3KnTDWvX\nkbed+gAJl6xLAHzzUTYB8DFXuJgmIjNFJFxEwoOCghwWrFJKFWeTJk6kssVK/IRJJC9ZRtyE/1LN\n2ZUJ48ffsDYdeYaQBFx6P5QfkJiPsn5A0rU6lZVS6t8iICCArZs2sXHjRvbt20ft2rVp3Lhxnp3Q\njuDIhHAAcDHGVBeRg6+eX/AAACAASURBVPZ19YE9Vyi7x/7eH9cop5RS/1rGGJo3b07z5s2vXdgB\nHHbJSESSgWXASGOMtzGmKfAYcKXBUeYB/Y0xFYwx5YEBwBxHxaKUUqrgHD10RV/AEzgDLAJeEJE9\nxpjmxpikHOVmAF8DEcBusm5PneHgWJRSShWAQ59UFpFzwONXWP8LWR3JF14LMNi+KKWUKgZ0cDul\nlFKAJgSllFJ2mhCUUkoBmhCUUkrZaUJQSikFaEJQSillpwlBKaUUoAlBKaWUnSYEpZRSgCYEpZRS\ndpoQlFJKAZoQlFJK2WlCUEopBWhCUEopZVfohGCMcTfGzDLGHDPGJBpjdhhj2lyl/DPGGKsxJinH\n0qqwcSillCocR8yH4AKcAFoCx4G2wBfGmLoicjSPbTaJSDMHtK2UUspBCp0Q7FNnDs+x6htjzBEg\nDDha2PqVUkrdHA7vQzDGlAFqAHuuUuwOY0ysMeaAMeYtY0yeickY09sYs9UYszUmJsbR4SqllLJz\naEIwxrgCC4G5IrIvj2I/A3WA0kAHoBMwKK86RWSmiISLSHhQUJAjw1VKKZXDNROCMWaDMUbyWDbm\nKOcEzAcygH551Scih0XkiIjYRCQCGAk84YB9UUopVQjX7EMQkVbXKmOMMcAsoAzQVkQsBYhBAFOA\n8koppW4AR10ymgbUBB4RkdSrFTTGtLH3M2CMuR14C1jpoDiUUkpdJ0c8hxAC9AEaANE5ni3oYn8/\n2P462L7JvcAuY0wysBpYBowtbBxKKaUKxxG3nR7jKpd8ROQ44JPj9UBgYGHbVUop5Vg6dIVSSilA\nE4JSSik7TQhKKaUATQhKKaXsNCEopZQCNCEopZSy04SglFIK0ISglFLKThOCUkopQBOCUkopO00I\nSimlAE0IDiMiZGZmFnUYSil13TQhFJLVamX06OGUK1sCd3c3wsNq8t133xV1WEopVWCaEArp9dcH\nsXbVx6xf4k/6iaq8+XIC/+nWgU2bNhV1aEopVSAOSwj2qTbTcsyHsP8qZY0xZrwx5qx9ec8+61qx\nIiJs3ryZcePGMWPGDOLi4nK9n5SUxMyZM1g03Z+aNdxwcjI8+qAPwwd6M3HCyEK3v2zZMho3qkvp\nIH/uu7cxGzZsKHSdSimVF0efIfQTER/7cttVyvUGHgfqA/WAh8maZKfYsNlsPNO9I52efoDY4xP5\ncc3b1KgRws8//5xdJioqipKBrpQrk3taiUZhHuzfv69Q7c+fP48Brz3D0L5n+XN9Sbq3P8LTTz3M\njz/+WKh6lVIqL4WeIOc6dQcmishJAGPMRKAXML2I4rnM4sWL2Ruxjogfg/D0zMqba35woWuXJzh8\nJAoXFxcqVqxIXEImx09aCK7omr3tz5vSqFOn0XW3LSKMGD6UBVP8aRzuCUCXDn44ORlGjxrGPff8\nXridU0qpK3D0GcI4Y0ysMeZXY0yrq5SrDezM8XqnfV2xsXTJHF7q6ZadDAAeau1NiQArf/zxBwBe\nXl68/PJrPPFcPL9vSyXhvJX5S84z+r/JDBr8znW3nZSURNSpWBqFeZCZKdhsAsB9zT3ZsmU7b731\nBgcOHCjcDiql1CUcmRCGAKFABWAm8LUxpmoeZX2AhByvEwCfK/UjGGN6G2O2GmO2xsTEODDcqxMR\nrtSrYUzWexe8/fZIuvcczrOvQaWGkcxbXpkVK7+jYcOG1922t7c3Xp4etHr8JN5V/iawxiH6DjnD\nz7+nElQSUmI/oWmThnz22cLrbkMppS4jItdcgA2A5LFszGObNcBLebyXANyV43UYkHitOMLCwsTR\nvvjiC7mn1Z1Sq2aw9OrVXQ4fPiwiIgsWLJC7GgZK8pGqYj1VXaynqsuaz8tLxQqlxGKxODyOnKKi\noiQgwEM+HhskyUeqSuTOKtKlg6+UDHSSRTPKivVUdfnzh2AJDPSWxMTEGxqLUurWB2yVfPytz9cZ\ngoi0EhGTx9Isr82AvO4c2kNWh/IF9e3rbqoJE8bz1uu96Nv1JAs+hjI+q2naJIzjx4/TsWNHqt/e\nmnqtYxg6+hzd+sXT9cVE5s3/AheXwne9JCQk8N5742nXtgXPPtOR3377Lfu9Tz6ZwdOP+tL32QA8\nPJwoW9qF2ZPK4O5uqBHqBkDdmu40qO2dq5NbKaUKwyGXjIwxAcaYB40xHsYYF2NMF6AFkNcTWvOA\n/saYCsaY8sAAYI4jYsmv5ORkxo0bxaqFgbRv50P92u6MGBxAl/YufPDBeJydnZm/YAnzFnyLb5lX\naNp6OPv2H+Gee+4pdNt79+6lfv0abNk4np5P/k39qj/y1JMPMmvWJwAc3L+Lu8Ny/9c4OxsahXly\n4HBG9rqkZCseHh6FjkcppcBxdxm5AqOB2wErsA94XET2AxhjmgPfioiPvfwMsvobIuyvP7Wvu2n2\n799PxXIeVAl2zbX+kQfcGfpu1q9uYwxNmjShSZMmDmv3nXdeZ+KE92nT2oPFM8tlr7+vhQet2ven\nU6cu1KoTxs+bNtL9qYvbZWQIm7amMnxQCQC+WZfEqTOG5s2bX1ccNpsNJyd9LlEpdZFD/iKISIyI\n3CkiviISICKNRGRdjvd/yZEMsF/WGiwiJezLYPt1rpumXLlynIhMITnFlmv9nv0ZVKwYckPaXL16\nNYsWTiGsngvPdPTL9V6t29ypXMmdnTt30qtXH9b9bOPdj+KJic1k38EMnnzuDCmpMH5yOg91SqD3\nwFS+WPIVrq6uebR2ZcuWLaN+vWo4OzsTWqUcU6dO5iYfeqVUMfWv/YlYrlw5HnzwAfoOTSA+wQrA\nHzvSGP3fFPq9NLhQdW/fvp1HHm5NYKA3tWqG8PHHkxAR5s+bQf8+HpQOciEyOvdAeFarEB2TTokS\nJShVqhQ//byZiMNNqNE0mvufTqRueB82/7GHex4aQ+8Xp3DkaBSNGzcmLS2NuXPn0rdvT8aOHU1U\nVFR2nfHx8SxevJjFixeTkJDAqlWrePml7ox/PZWMk9VYNM2V6VPe5KOPPizU/iql/hnMrfTrMDw8\nXLZu3eqw+pKSknixbw9WfvU1fj6uYNx5d/x/6dy5y3XXuXfvXlq2uJuRg73o0M6Hg0cyeO3tZB5s\n15tdf26h0yN7CfBzou+QM6xfWpGQSq7YbMLoD+NZvymYFSvXYrFYKFu27DXbio+Pp/U9jSnhH8uj\n9xv2HoAvV6Wx8qs1nDhxnBde6EnTu3wQgd+2JFO2TBAjB6bTvl32yRoRf6XTtksSJ07G6CUkpf6h\njDHbRCT8WuWK6knlYsHHx4e5874gPj6ec+fOERISgrOzc6HqnDhxLK/19qLPf/wBOHrCEBpsYeqU\niYTf2YIps9P48ctSvNQzgLAHjlOzuhsHD1uoFFwdZ+cMQoLL4uzsRLWqVZjxyQLCw3P/H8bFxfHh\nhxP4ft1XnDoVw913pLJwamkuPMJxb/NEnuvZmVOnzrBheRB1a7oD8OduD5o9coxqVSrmqq9uTXfO\nJ0aTmJiIv79/ofZdKXVr05+EQEBAAKGhoQVOBqdOnWL58uW8//77fPrpp8THxxOxaxutmmbdGrp8\ndRKP/ieK8AYezJ9cmirldrJrTzJ3tYnGzd3Q7Qk/du5OJyFROHToEA+1iOLMnhBi/wrm1efieejB\nVpw5cya7vcTERJo3C+fY/pmMHhSHsznLq739yfk83/+19eH06VM80MojOxkANKjjzqMP+jLpk/hc\n+7AjIo0Af198fX2v59Appf5BNCFch9jYWB55+F5uvy2Y53o+yd8RY1i9YgjVqlXCz68kW3ZkYLUK\nA4fHsHhmOfo/H8hDrb2Z+m4Jenfz58DfqXz/UzKR0Zm0bu6Fs7ONUiVg+KCSeHg44exs6NLBjwdb\nOTF79qzsdmfN+pTbQs8ze1IgrZp4UTLQmcSk3J3iFouQnp5JgN+lUUPJEq58+U06X32XRGqqjV9+\nT6Xri+cZ9vpwvVyklNKEcD06Pv0oTplbqVjeiUObQ5j2XmmWflqSRdP82bVrF2MmJTN7UQKWTKF5\nI89c23Zu74W3F7i7O3HgUAbN7vbkxWcDOBtnZfbnCbnK3tnAna1bLw5k9+vGtbRve/EqX5cOfoz9\n7znS0i4mhQlT43B2NnzzvYW4eGv2+rPnrCxbnca74z9k3JQSlKx5jBeGOTNo6ET69u3n6EOklLoF\n/av7EK7Hvn372LcvghZ3Q78eAfj5XrzM1CjMA3/fGJq36sKUeb8Qey6WxCQbvj4X8+6xkxYADvxt\nYfO3lXB3z3qvZ2d/Grc7wf+18SHA3xkR4ctvkml5f93sbcuWC+bQkW3Zr5/v7s/m7WlUvOMIrZt5\nsu+ghdhzVkqXcsLDqwx3tYnhuc5uiBg+WZjOM88+T9++fenbt++NPkxKqVuQniEU0KlTpwgN9kSA\nCyNYWK3C62NiCW54hMQkK4s+m0Oz5q3x8/XktbdiSE/P+gV/6nQmg0acJTUNenbxy04GADWqulG/\nljuT/xfPjog0ur90ml1/WRgyZEh2md69+zFlTiqbtqYCYLNB5WAX0tNtbNmRxm3VXKlexQUXZ0hL\nPUO/l4cTdb4D0UkdmL9wFePGTbhpx0kpdevRhFBA9evXJ2JfEi0bezFtdgKpqTbGfxzHb1tS2fVj\nCFG7Qtn9cwg/rptLYADEnLNSOfwozR89Qe0Wx0hNc6Jy5VASEm2X1Z2aZmPO5+dp0ymKZasSCQoq\nzdixI4mNjQWgbt26TJs+l6f7pFKjSTSlax9m8qwE2tzrzeEtVVjyaXl+WhnM42188PXKICrqBB9/\nPJ0BA4YyZ84MKoeU5o4G1fn444+w2S5vXyn176YJoYBKlChB//6D+OjTNAICnKjT8igTpsUx/f0y\nVCiXdcpQJdiVmRODSEpKZ+Xc8vz6TSXGvl6KQ5sr076dH63ueYhpc9I4dfriw2lrNySz/5AFFxdI\nT7cxakgJPh4Nv/8yndq1a7B7924A2rdvz9Fj0Sxf+Stbt/2Fk5Mb494oxflEG8dOWDgZZaHJnR5E\n7EvH29uXM2fO0KxpOGV81rDmMy8+eDuVz+cP55VXXiiS46eUKr7+1Q+mXS8RYenSpUyb8j5/bNlK\napqQfqIaTk4Xb/+MT7BSvv4RUo5Wy7Vt00fO8cY7c4iI2MGoke/w6EPenIuzsvXPdDo+7sv/tfXm\ngaej6PuMH5+vTKL53Z64uRm++zGdHj2fJykpnsiTR7jr7la8+OLL1KpZlZaNbKz7ORVXF0jPECqW\nd+HYyUzatWtHrVr1iTo8kxnvB+SKrVqjU+ze8zfly5e/acdNKVU08vtgmp4hXAdjDE8++SQ/bPiD\n51/oR4kAZ779ISVXma++S8bTwzB2UjwpKTYSzlsZNiae5LRA2rRpw7Bhb+Hr64OXhyEu3oavj2H/\noQwOHc0kwM+J/y06z1fzyrNsdnk+n1GOD0YGMuvTKYSWXkOfToc4fmAqt9UIxphM3NycOLKlMmf2\nVmXVwgrEJ9j4am55kuJ+Y/myhTzYKve9AwH+zoQ18GPXrl0387AppYo5TQiF1KRJK5JToesL0Xy6\nMIGIv9KZPCueQSNiqB7qzeKvAyhZ8yjl65/g6Om7Wbvul+wH4Fq2eoAlXyfRo7Mf3y+tSK9u/rz9\n3lmcnIXat7tnz6dsswljPjzHyrnlGPJSAG3v9abJnS74+2WSmprGJxNLE+CfVWezuz0Z0i+QOYvP\n8+EIX06ciGLnntzjJlkswt79yVSuXPmmHiulVPH2j7/tNCEhgfnz57N//27q1m1Ily5d8Pb2dkjd\nqamp9OnzDJNGlWTitDhWf5/MpJnx1L7djdWfVeDYyUz+t7Qs27bvxhiDq6sr8fHxDBz4CiuWL+F8\nwjk+HBVEj05ZQ0ZUrexG+TIuPNw1kkrlct6qmkmGBVo0zkoQvQeeYf/fGfTp5s/nKxJzzfsMUKem\nOyu+TaZSeRcSEjL4cEYsYfVdePh+bxLO2xg65jz16jXk9ttvd8hxUEr9M/yjzxAOHjxIndrV+OX7\nkYQGfcmq5W9Sv95tREZGXlZ2y5YtdGjfhhrVK9C2TUt++OGHa9b/22+/EVLRmU1bUzlz1sqxExb+\n85Qv8z4uS1h9DxISrXh5eePm5oarqysWi4X77m3K2cjP+PJTV2w2C21a505OTe/yIMMCq9an8sKQ\n0/R4NZpv1iZxPtFKcooQ8Vc6azcks3ZxBfr8x59jJzM5GWXJVceqdcmEN3Dns+WJNLvbne5P+/Hs\nK6fxDT1E6F1HOR3fkM8Xf1W4g6uU+sdx1IxpSZcsVmPMx3mUfcb+fs7yrRwRx6X6v/Y8L/d0YtH0\nAF7pHcjy2QE82S6dN14fkKvcr7/+Sru2rWl993ZW/M+Vp9vup2uXx1i+fPlV68/MzOTQkfNkWmHN\nogpMGlOaH39NpfMLpzgXZ+WD6Rl07dYnu/zKlSvxdDvNpx8EULemO6Ehruzam56jPmHdTynYrIKH\nuyElxUbEXxkMG3MWgPufPMlPv6bwUGtvvLyc8PdzZsDzgbTrEsXXa5PYvS+dN8bFsvDLRFLShFff\njKFieVd2RKSzamF5Dm+pzPBBJfn9903Zt7Lml9VqZfv27UREROj8CUr9U+Vn4uWCLIA3kAS0yOP9\nZ4CN11N3WFhYvieVtlgs4urqLImHqor1VPXs5cSOKhIY6J2r7P33NZHZk8rkKrfm8/JSp3YVsdls\nebYxZMgQCa/vLplR1bK3Sz1WTcqWdhYvT2cZPPi1XNsPHTpYRg4umV12wdSyUqOqq/z5Q7As+bSc\nVCjrLOVKO4uXp5E7G7hLyUAnmTwuSE7vDpXfv60kDeu6i5+vkfAG7tl1ZEZVk/lTykhIRRcpW9pZ\nGtR2k6Z3echLPQOkRSMP8fUxEh1RJde+vfFqkLz4Yq98H8sffvhBKlUsJZWDPaRsaRcpVdJTJk+e\nnO/tlVJFC9gq+fgbeyP6EJ4AzgC/3IC6880Yg4uLM6lpgpfXxfXJKTbc3XLPMvbHlj9ZOCn3/AP3\ntfDiwMEjpKWl4emZezyiCyIidvDUY765Rht1czPc39KL3X9XZvz4D3KVr1KlKt99ffGkrNP/+RIT\nm0mjNsdxc3Vi9aLyNA73JC3NxhvjYok/b+OFZ7JuFy1V0pnVn5WnRpNj7DuYwYy58fTq5o8xEOjv\nTFy8jT2/hFC+7MX/0j4DTxOXYCOoVO7/5nuauTHyv/m7fTc6OponOjzMwqkBPNCqEiLComWJPD/4\nZVxcnOnT5/l81aOUKv5uRB9Cd2CePSvl5Q5jTKwx5oAx5i1jTJ6JyRjT2xiz1RizNSYmJt9BODs7\n89STjzNi4vnsSxw2mzDygyQ6de6aq2xwpbLs3peea93BwxYC/H1wd3cnL+Hhd7Hlz7TL1kfszaBV\nq/suW9+xY0c2b7cxdXYCGRlCaqqNY5EWXF0NQ14KzL6ryMPDiffeDiI5xcae/RfjCirlQrUqrtjE\nhWkL/KjUMJJqjaJ5fkg6JUt6Ubb0xXGVRITft6Vx9LjlshFRt/5poWr1/HUoL1iwgHb3ufJAq6y+\nDmMMnTv40eROd4YNG0RmZuY1alBK3SocmhCMMcFAS2DuVYr9DNQBSgMdgE7AoLwKi8hMEQkXkfCg\noKACxTPxg6n8sassDe8/S59B56nTKpZTZ6szcuS4XOVeeXUYL7+ZxIFDGQCciLTQa0Ai/V565arD\nQr/00ius3WBh/pLzWK1CerqNsZPOcfSkMGrUqMvK+/n5se77X1i2tjJBtY9TsuYRZn92HldXw23V\n3HKVdXY2hFR0JSrHVJsJ5638dTCDl156mZ27DvLTz3/y9apNHDkaTXBILbq+GMeuven8dSCDPoPi\nybCWoWTJkjzV6xQnIi1YrcLy1UlMnJ7CK68MuTS8KzpzJpoaoZfn69AQN8Rm4cSJE/mqRyl1C7jW\nNSVgAyB5LBsvKfsm8FN+rlXl2KYjsC0/ZQvSh3CB1WqV9evXy7Rp02Tjxo1X7BOw2WwyYcJ4CSrl\nJxUr+EpgoLe8/vogyczMvGKdhw4dkmee6SSVQ0pLndqVpWyZQPHxdhIvTyepU7uK7N2795pxxcXF\nSVApD/lsell5vI23PPmoT67r/FG7qoiHh5FPPigtGSeryaE/KkvrZp5yW42QK9aXmJgoQ4YMkNAq\nZSUkuLS8+uqLcu7cOUlLS5P+/ftJQICXuLu7SFjD22X9+vUiIrJp0yZ5/vlnpVvXDvLZZ5+JxWK5\nrN5Vq1bJbdXcJf3ExX6SxENVpUI5Z/HxcZeEhIRr7qtSqmiRzz4Ehw5dYYw5ALwrIv8rwDZPA0NE\npOG1yt7ooSvS09OJjo6mdOnSeHp6Zh2gHP0DAJGRkdx1Zz2e6+xEp//z5kRUJq+PTaZRs6d4550x\nlCpVKt/t+fq4cvD3Srg4Gxq0Pkabe715tpMfkacyGTLqLCUCDcnJwt9HLXh7OSE489tvO6hdu3aB\n9y0zM5P09PTsZzA+/HACH0wYSb8eHgQGGOYszqREUANWrPwOF5eLZwRWq5XGjRrg5vQ3A/sGkp4h\nTJwaR/x5aHXv08ycebWTQaVUcZDfoSsceXdREyAZ8L1GuTZAGfu/bwd2A+/kp43rOUMoKKvVKu++\nO0YqlC8pxhhpdHcd+f7777PfHzJkgLz0XFCuX/PREVUkIMBTYmJiCtRWvbpVZeW8cmI9VV1ORVSR\nQS8GSp3b3STQ31neeOMNKR3kLy0aB8l9LYPE399T5s2b65B9PH36tPj7e8iRLZWz9yH9RDVpFF5C\nFi9efFn59PR0efrpJ6VkoKsElXIVT09X6dGji6SkpDgkHqXUjUU+zxAc2YfQHVgmIomXZKZg+7MG\nwfZV9wK7jDHJwGpgGTDWgXEUyptvDuGrLyfwzXxvUo9VZUCvs3Tu9DibN28GYPvWX3mwVe67lIJK\nuVDndl/27NlToLbeGf4e/YYl8OsfqQSVdKbrE75gXBj2xmhGjx7N8ROnGfLmHF58dSbHj0fTrdt/\nHLKPP/30Ey0a+xNc8eJ+uLgYuj7hzLffXv7shZubG59//gVR0Uls/HU3UVFnmDVrQZ53Xymlbk0O\nu+1URPrksf444JPj9UBgoKPadaTk5GSmT5/KzvUXh7Ju386H07FWJrw/kiVLVxFSpTq79h6mzb0X\nt0tLs7H/7ySCg4PzqPnK2rdvT0pKMj36D+XEyaOULOHHgIHDee21rMPj7u5O27ZtHbZ/F/j6+nL2\nnPWy9WfPgq9vwBW2yOLm5kaNGjUcHo9Sqnj4Rw9dUVCRkZGUCHDNTgYXNAn3YN++vQD06zeAD2cm\ns+aHZESEc3FWXhiSQLNmzalSpUqB2+zatRsHDp7k3LkETkbG0r//oMv6LRytdevWHI+EL7+5eDJ3\n+JiF6fNT6d79uRvatlKq+NKEkEPFihU5F2/hRGTusYE2bk6jVq06QNaMafPmL2XgKFdK144k9O4o\nnLzuY+68JdfdrjEGLy+vG54ILnBzc2PFyjUMHGmjycNxtOt6njsfOs3b74wnLCzspsSglCp+dIKc\nS7z99jC+WzWDyWN9qFndja/WJvPqW4ms/nYD4eEXO+lFhDNnzuDj4+Ow0VNvtszMTDZs2EBSUhIt\nW7YkMDDwiuUiIyOZMmUSu3ZuJrRqTfr166+XjpS6heT3LqN//PDXBTV8+BgCA0vR8YUJnIw8RaO7\n67Nk6cRcyQCyftWXKVOmiKJ0DBcXF+677/InqnM6cOAALVvczZOPuPLcUy5s2/kXTZssZPmKb2nW\nrNlNilQpdTPoGYK6qk4dH6d+td8Y3O9iZ/PilYlM+l8pft8cUYSRKaXyS6fQVA6xfv2PdOngk2vd\nEw/7sHPXPpKTk4soKqXUjaAJQV1ViRJ+REbnHsDuTKwVV1eXqw78p5S69WhCUFfVo+eLDB2dNWMb\nQHq6jYEjztO1a+dcQ1wopW59mhDUVQ0YMIia9R4n9K5T3P9UApXvjMZi7uL99z8q6tCUUg6mncoq\nX6KiotizZw9Vq1YlNDS0qMNRShWA3naqHKp8+fKUL1++qMNQSt1AeslIKaUUoAlBKaWUnSYEpZRS\ngCYEpZRSdpoQlFJKAbfYbafGmBjgWFHHYVcKiC3qIK5A4yoYjatgNK6CKS5xhYhI0LUK3VIJoTgx\nxmzNz329N5vGVTAaV8FoXAVTXOPKi14yUkopBWhCUEopZacJ4frNLOoA8qBxFYzGVTAaV8EU17iu\nSPsQlFJKAXqGoJRSyk4TglJKKUATglJKKTtNCIVgjKlsjFltjIkzxkQbYyYbY4rFkOLGmI7GmL+M\nMcnGmEPGmOZFHdMFxpjqxpg0Y8yCYhCLuzFmljHmmDEm0RizwxjTpgjjKWGMWW7/fztmjOlcVLHk\niKlYHaMrKU6fqQuK83cwL5oQCmcqcAYoBzQAWgJ9izQiwBhzPzAeeBbwBVoAh4s0qNymAFuKOgg7\nF+AEWf93/sBbwBfGmMpFFM8UIAMoA3QBphljahdRLBcUt2N0JcXpM3UrfAevSBNC4VQBvhCRNBGJ\nBtYARf3lBRgBjBSR30XEJiKRIhJZ1EFB1q8mIB5YX9SxAIhIsogMF5Gj9mP1DXAECLvZsRhjvIEO\nwFsikiQiG4GvgG43O5acitMxupLi9pmyK7bfwavRhFA4k4COxhgvY0wFoA1ZSaHIGGOcgXAgyBjz\ntzHmpP1SlmdRxmWPzQ8YCQwo6ljyYowpA9QA9hRB8zUAq4gcyLFuJ8XjR0a2Ij5Gl8ZS7D5Txfk7\neC2aEArnJ7K+rOeBk8BWYEWRRpR1qcEVeAJoTtalrDuAN4syKLtRwCwROVHUgVyJMcYVWAjMFZF9\nRRCCD5BwyboEsi45FAvF4Bhdqjh+porzd/CqNCHkwRizwRgjeSwbjTFOwHfAMsCbrFENA8m6blhk\ncQGp9qIfi8gpEYkFPgDaFmVcxpgGwH3AhzcyjoLGlaOcEzCfrOv3/W5mjDkkAX6XrPMDEosglssU\nk2OUrag+U/lQ3aoX1gAAAW5JREFUJN9BRygWd8QURyLS6mrvG2NKAZWAySKSDqQbY2YDo4HBRRWX\nPbaTwE19BD0fx+tVoDJw3BgDWb+GnY0xtUSkYVHFZY/NALPI+mXXVkQsNyqeazgAuBhjqovIQfu6\n+hSPSzPF5Rjl1Ioi+Exdi4jEFcV30BF06IpCMMYcJmuskglkfRhnAyki0qWI4xpJVn9GO8BCVsfk\nBhF5qwhj8iL3r9+BZH2ZXxCRmCIJys4YM52s0/r7RCSpiGP5nKw/JM/ZY1oNNBGRIk0KxekYXVDM\nP1PF7juYH3rJqHDaAw8BMcDfQCbwWpFGlGUUWbfgHQD+AnYAY4oyIBFJEZHoCwtZl0fSisEXNwTo\nQ9Yfu2hjTJJ9Kaqk3hfwJOt25kVk/XEr6mRQ3I4RUHw/U3bF7juYH3qGoJRSCtAzBKWUUnaaEJRS\nSgGaEJRSStlpQlBKKQVoQlBKKWWnCUEppRSgCUEppZSdJgSllFIA/D/s6cSg9vSSAgAAAABJRU5E\nrkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "X, t = datasets.make_blobs(n_samples=200, n_features=2, centers=3, random_state=42)\n", "plt.scatter(X[:,0], X[:,1], c=t, edgecolor='k')\n", "plt.show()\n", "\n", "xx, yy = get_meshgrid(X[:, 0], X[:, 1], nx=100, ny=100, margin=0.1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we construct our neural network with $L=2$ and $n_0 = 2, n_1 = 2, n_2 = 3$. Note that $n_0$ and $n_L$ is fixed once we fix the input dimension and the class numbers." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "success : True\n", "nit : 1493\n", "func value : 0.14345242871226377\n", "calcualtion time : 0.7380423545837402seconds\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD/CAYAAAD4xAEfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Wd4HNW9+PHvbC+SdtV7lyxZxZZ7\nxRgXbGzTEkggpP4vgQAJ5CY3Nx0IpF5SKQFCgFBCAoRqbDC4F9ybLNmWXGRbvfeVtsyc/wtZa61l\nY4NlSbbO53nyQrOzM2cW5/xOm99RhBBIkiRJI49uqAsgSZIkDQ0ZACRJkkYoGQAkSZJGKBkAJEmS\nRigZACRJkkYoGQAkSZJGKBkAJEmSRigZACRJkkYoGQAkSZJGKMNQF+CT2CxBwmkPH+piSJIkXVKq\nm040CCEiz3XesA4ATns431z8o6EuhiRJ0iXloZfuPn4+58khIEmSpBFKBgBJkqQRSgYASZKkEUoG\nAEmSpBFKBgBJkqQRSgYASZKkEUoGAEmSpBFKBgBJkqQRSgYASZKkEUoGAEmSpBFKBgBJkqQRSgYA\nSZKkEWpYJ4OTpM9CCA1QUBRlqIsiSYOquL6D2rDzb9fLACBdNmqaKlix7T+cqD+EUW9ibNpU5k28\nEaPBNNRFk6SLrri+g5IxegwG/Xl/Rw4BSZeFts5mXlrxF6z1QczmBiarcyk/WsZ/1j471EWTpIuq\nuL6D1aqLkjF6nE4bU/KTz/u7sgcgXRa2l6wnUosnQUkDQI+NHHUiH9d9QENrDRGOGP+5VY3HKS7b\niSY0cpLHkRiVPlTFlqQL0rfV7wwyk5sac+4v9SEDgHRZqG+qJkQLhT7D/jpFh0MXTmNbrT8ArN+7\nnK3Fa4jVkgGFVw/9jbGZU5g/6XNDU3BJOk/F9R0Bf9eG6egYb/pMFX8vGQCky0J0eDzHaw8RIxL9\nx1Sh0qo1EOGIBaCxrY7NxauYrM7FrFgASFTT2XZoFXnpk4gNSzzjtSVpKPVO7LacYXw/6AIqf5AB\nQBrm2jqb2XtkC67uDlJis8iMz0On6z91NTFrFjtLNmD2lRInUvDQzRH9flJjswkPiQLgUMU+okSc\nv/IHMComorUESsr3ygAgDTsXOsRzLjIASMPW4ar9vLH2WaJEPGbNQsmRQraFreFL8+5Brw/8pxts\nc/D1a77HR9vfZFPt+5j0ZgoypzO7YLH/HL3egKbTQA28j6qoGPVypZA0fPS2+i90iOdcZACQhiVV\nU3lnwwvkqZMJVSJBgSTfKAobP2b34Y+ZmDWr33ciHDHcOu/us15zdNI4Vu18hwTRQrDiBKBTtFOn\nq+CG5K9etGeRpE9jteryD/dc6BDPucgAIA0LjW21rNu9jPK6owRZQ8hMyscgjD2V/0k6RUecmsr+\nst0BAcDjdbNq59sUHt2KV/OQFj2aqyffRIQjGiEERcd2sKd0Mz7VS3ZyAbuPbyBMiUZBoVHUsHDS\nzYQGRwzFY0uSn7/Vn2K4qK3+vmQAkIZcU3s9zy3/PfG+VHLERFzdHexoW4dX8yKECHijV0XFoAv8\nZ/vamqfpauhionoVRkxU1h7jHx/8gbuu/zlrdi3l6PEDJPgy0KOnquUYkY44JmTNBAUy4/OwW4Jp\nd7WgKApBVsdgP74kDWqrvy8ZAKQht6lwBTG+ZFLIBgXshGBXQ9jCR1RzjDhSAfAJLxWGI8zNvN7/\n3ZqmcmoaKpiqXo1O6ZkcTiaTLrWdjftWsP/YLqaq8zEoRgAi1Fh2ta3HbLQwOnkctc0VvPzhYzS3\nNwCCSEcsN8z6GuEh0YP+O0gjz1C0+vuSAUAacpX1x0gWWQFr+G1KEHZ9MGUcoF6pxqxZaaCa0cnj\nyE2Z4D+vobUGpy4cnRa4MihEDaOi9igRSoy/8gdQFIUIXyxl1SWkxWbz0oePkuzJIo8pCAQVLUd5\n8YM/853PP4RBb0SSLobTl3Z+mrd3B5IMANKQcwSF09HehoNw/zGf8OLWurjr+p9T2XiMzq52LCYb\n7V0tFB7dyuikAkxGCxGOWFq0RjSh+XsAAG36JsIckVR3lPe7X7eui3hbEkXHdhKihRGv9PQwFBSS\nyKBZrePgib3kpU68+A8vjThD3erva0BzASmK8m1FUXYoiuJWFOUfp302V1GUg4qiuBRFWaMoytCE\nPGnYmZY/l2P6g7SIRgA8wk2JfjdZiWNxBIWRnTiWsqoSPtr6BqV7itmybTWPvnE/1U3lxIQlEBzk\nZB9b6RKd+ISXY6KEGq0CmzmITtrZLTawU6zjgNhFpSijTqlgbPpUWjubsPrs/cpjU4M4VFnE0eqD\naJra73NJ+iz65uzpSDEwJT95SCt/GPgeQBXwS2ABYO09qChKBPAmcDuwFHgYeBWYOsD3lwZAt8fF\nkaoD6BQd6XGjMRkt5/7SBUiJHsU1077IR9vfwOvzogofuckTuWbKFwDYdWgT9bXVTPbNRafoQYUa\ncYI31z7HrIJFtLe3EISDraxExYeVINJENpWHTuDWunESQQyJtNJEKXu5auy1OOxhxEekUGTYgfCd\nmmjWhEYN5bSfaKaiogxV7+NL8+8hyhl3UX8D6fI2nFr9fQ1oABBCvAmgKMpEIKHPR58DioUQr5/8\n/EGgQVGUbCHEwYEsg3RhCo9uY/mWfxGqi0QgeFd7mRtnfZ1RCfkX9b55qRPJSR5Pe1cLVpMtIOjs\nO7KdBF96T+V/UjSJHHUfYPXud8nWxhGmRLNTrCOGJP+QDgIs2GijmXAlhnBisIsQ9h3exvTc+WTG\n57ExZAXFrdtJUNMBwRH2YyOIPHUqFepR6rwVPL/sD9w853bSYkf773/wxB42FX5Ia2czcRFJXDlu\niXyTWOqn71i/02ljyjCp+HsNVjroXGBv7x9CiE7gyMnj0jDR0tHI+1v+zXh1Fvm+qYzxTWOMOo03\n1z+Py91x7gtcIJ1Oh8Me5q/861ur2VT0IR1dbSj039xFQaHV1YSTCDSh0UIDsQSOLMaRQiM1/r+j\niKe+vQavz4NOp+MrV9/LqJw8yuz7KTZsR4+eXCazg7V00UEGeaRo2byx5jm2H1wHwI7SDSzb+G/C\nm2MZ45mGUqXnpRV/pqap/3yDNHIU13f0+1/vcI/TaRs2rf6+BmsSOAioP+1YKxB8+omKotwB3AHg\nsIdd/JJJfkXHdhAlEghSTq2FdyhhRCjRHDyxh/GZMwf8nkIIapsr6fa6iAtLxmQ0A7B2zzK27V9N\nlEhA01TKOEiYiPZP9NZThdFkIswcSUtnA6FEoUOPFzfmU6OPeOjGwKnVPG66MegM6HU9vQmT0czs\ngiXMLljCKx89gaUmiCqO4SCMXGWS/3vhahSrd71Dfuok1u5aSr461f82sY0MhCpYt2c5X5xz54D/\nRtLw13eIp6/hNNxzJoMVADqAkNOOhQDtp58ohPgb8DeAuPBkcfGLJvXy+jzoNQOnN7b1wojH5xnw\n+x2uKOa9za/g9Xqx6mx0am1cPenzxIQlsn3/Wiarc/Hho5rj6DGwhQ+JEgl00kYjtcxMX0BzRwP7\nyrbiIJxQoihlL7liMjpFhypUSthLDD1DM6rwcUhfyNi0qeh0/XdNykoew9aGtag+H8mMCvjMpgRj\n0wVzrLYUTdP8lX+vcBFDceO2Af+NpOFtuA/xnMtgBYBi4Gu9fyiKYgfSTx6XhonM+Dx27t9Iiprl\nXzvvEW7qqOS6+NsG7D6apvLGuuc5VLGPFEahoKNWrcCAife3vE5W8hhitCRMioUDYjNJZJJCNk3U\n0kIjAjBjZdO+DzFhJo0cNFSOU4qGxibdcpz6CFq0BuzmECpcZbTqmnCJDjLjcs+Y+7+2uZL9Zbto\n9tWjx0AXrsAyC41uzUVocAQaKm7RhVk51dPooFX2WEeYi52pczAMaABQFMVw8pp6QK8oigXwAW8B\njyiK8nlgGXA/UCgngIeX+IgURqcWsOPYWmJ9SQgFqg3HmJQ1a0DejK1vqWbTvg85Vl2K292FHj2t\nNKOikkE+OvScoJRD5UXEiRRQoJFqcpiAoiiE0zOR2yna2MJKLFiZynz0Ss8/41iRwlbdR9w0+3Z8\nmo/G1lrWFy4nzBCBR3hAgbGjpvlf8FI1lZLyvRwqL6L42E6SxSims5DDFHGU/YSLKGxKMJrQKFMO\nEOGMJjo0gXEZMzhweBfZ6ngsipU20cQRfTFL8r5E8bGdlFWVYLcGU5AxTeYYugyd3uq/FCv+XgPd\nA/gZ8ECfv78M/EII8eDJyv9x4GVgK3DLAN9bukCKorBo6i2MThnH/rJd6BQdM9PnkxSV8Zmu5/Z2\ns+fwZsprjqA3GCgtLyRRy2CUKKCVRo6yn1aamMki/9i+Q0xlm7aaKl0ZSVoGOgz48GLkVLrmKo5j\nJ5hI4vyVP4BJMROmRdPYXkdGfC5vb/gHE9QrsSs9o48tooE31v6d+276JTpFx4sr/kJ3exdhvmgi\niOMEhwglklxlIsdFKVv4iBB9GG7RRYQzmpuv+iYA8yfeiKIobD+0ClAwGkxcVXAtm4s+orO1gwhf\nLI26OrYdWDsoK6ikwXM5tPr7GuhloA8CD57ls5VA9kDeTxp4iqKQFptNWuz5/acSQnDwxB62H1hH\nl9tFevxopuXNQ0Hh2eWPYO62EqZGcYQDpJLTs2evAk7CsYogStgd8AavoihEi3i6Q11sa1mFTtVx\niELyxVQURUEIQQPVmLHiw9uvPG66MRstFB3dTpSWQDcu2kQzTiJwKhGEKlEcLN9LW0czWpvGOPUK\n/zsAdaKSA+xiqphPsjKKJn0NEwuuJCM+J6AHpNPpuXrS55kz/jq63C7slmC2l6yju6X71PUERGpx\nLN30Mt+9+df+SWfp0jRY+fkHm0wFIV2Q9XuXs+vAJpJ9WUQQR3lHGcXHfkdmQh72rhCyxThQoFTs\nJYr4gO9GEkshm1GFir7PGn+XvoO81InEhCXwzw+fwIuXjSwnWDhpowkfXlRU2mkmXqT6Vy01i3pa\naCArcSwf7XiDGlFOK01YsVPKXmJFMnpNj9fn4cCxPSSo6QGZRiOJo4Q9dNGJWVjoEG1kJ43tN7Yv\nhEAIDYPeSLCt594Hj+0hTk0JuJ5TicAgjNQ0lRMfkTLQP700SIYqU+dgkAFA+sy63J1sLl7JFG2e\nf0LUKSI44NlJ8bFd5GmT/SuKTFjoogMT5lPfpxMdCic4RLIYhYJCLeU0UotJb6K5rQG7LpgoLZ6j\nFNNKIyE4aaUZLx4sWNjOGkJEKD58dNKKouh47M37UVUfKYwiWckCwCs87GQd3cJFRtw32FO6BfW0\nrcEEAg3t5HaSRWTE5QZU/kIINhevZHPxSlyeDsKCopg74Xqykwow6I348AVeTwh8wodBL/9vdika\nrm/vDiT5L1P6zKqbygnRh2IW1oDj4WoMTdTjxe0/lkAaJexhrJiOWbHiFR5KdLsJs0XSoFZT7j6E\ngoKqqSiqwvtbXwN6KuUajhNBHE3UYsbGFUxnN5vwGLvBq9BKTw6hRDJJEhl4vR5KKaSFJv9rYUbF\nRIrIotJaRmhwBGMzp7B91zpC1Uh/76OCI2io7NNvYVzmDOaMvy7guTYUvs+e4i3kqVMIwkFTRy1L\nN/4T42wzBaOmsbLxLSJ9sf4VVFXKMSwWK1HOwJ6PNPxdzq3+vmQAkD6zIGsILq2j36YtXUon4c4o\njrWU4FQjMChGEkinlgo28T4mYUHV+chPncw1U7+IXqenpaOBvy39LToNUsginjS2sZJksolTkukQ\nrah4qaOSVhqJJpE6XSU2vZ0INY4uOshUeiZbzVgpEDPYyHI6RKt/iMiAkc6udtpdLUwcdQUnag6z\npeojIojGpevEq3dz5/yfEO6IDngeAFX1sWX/Ksaps/DhpZKjWLCTpuawce8HfHXhdzlRc5jNRz8k\nQomhW3Hh0bu57apv97uWNHyNhFZ/XzIASJ9ZlDOOMEcUR5qLSBM56BQ9raKJMg7ga/Ri0pnZwDKC\nRShuXBgxMYV5tNDIEYqZPW4Jep2ePYc3s2bXu3T7XATjJEnJpEO0oqERSxKdoo2drCeFLDLIp5M2\nSinE43aTRQHN1BFGVEDZdIoOhwijgzbsIoQWGinjIDYRxOpd73L9zK9y0+zbqWkqp6K+jGCbg4z4\nvLNO1nZ5OtE0wSEK6aSNMKKo4ljPRHSrgqIoXDP1i0zJuYoTdUewWYJIj8uRk7/DWHF9YHqT4ZCf\nf7DJACBdkC/OuZM31z/Ppsb3MWCkW3WRQAYZ5NGhtbJft4NurZN8phBCGIqiYCeENqWZ4mM7MOpN\nrN2xjCy1gHZa6KAVAA0NHToUReGYKCGJTJKVnrdzbQRhE8FsYyV69Fix00YTcaT4yyWEoJVGNFRK\n2YsePTr0eOjmwPE9XD+zZxP4mLBEYs4jiZvVHIRAQyCYxgL/yqUjoog6Kv3nhYVEERYSdbbLSMNE\n3yGevkZCq78vGQCkC2K3BvOVBffS5mrh1VVPkdKSRaySAkAwTvK1qWxlJcGEBgyFGFUTru5O9hxa\nyWh1Ag4lDL0wcJxSVKESjBMVlUZRSzstJBL4LoJdCUYvjFQoR8kVE9nGKoKEkziS8eHlMEVYsNNC\nAzlMIlrpSU7bJprYqa2jub3hU72kpdfpMenNpGk5ActWU8im3HcYt6cLs8n6CVeQhoORNsRzLjIA\nSAMixOak3dVK2mkJXm1KEAjowoWdIABUoVKvr2JmzHw2FX+IQ+lZaROihOIUEexmI6lkkUgGhXyM\nCQsdtBJCqP+6HuFGxUe7aKFY2U68SOU4pRxkNwoKDsLw4saOw1/599wjjDhS2XN4M1eNu5YTdYdZ\ns+s9apvLcdjCmDFmwVl3AtPrDei9gS1GBR3KybxD0vA1XLZgHG4GKx20NAJEOmNpPi3pa6doA6CQ\nzVSKMqrEMbazGkdIGCnRWdhMQbSJZv/5uUzCThD72MoxDiIQuHFzWNnn3zHMLbrZzw5iSWIq82ij\nGWOcCYNFjxkLTsKxYCOEUCz0b5VbhI0udyfldUf498onsdc7mOSbS2xbKh9u/g87Sjec8fmyk8ZS\noTuKEKdyFFZznEhHLDZz0AX/ftLFEdDqd9pk5d+H7AFIA2ZWwSJeXfUUelVPODF00Mp+dpBGDhZs\n1FGBQKDX65mScyWKopCXNok9JR+TruUSQxLN1FNHJU5dOJpeoBlVvrrgPsrrj/Lepn/6K99Ykskk\nH52ix6kP53jdIYw+I1bsaKjUU4VFZ6Vb68IrPBiVnlQSmtCoN1QxPn46a3cvI1XNIU5Jplu4sGJj\ntDqRdbuXMT5jBjqdrt/z/aPqjxR2b8bpi8Clb6dJV8dXpt876L+1dG6y1X9uMgBIAyY5OoObrrqd\nNTvf5UDrLuymYDxuNxEiFrsSTDQJNIt6ipRtJEZl8OyyR2hvb8FJOKXs5QC7cNrDmZm5EIvJSogt\nlMz4XHQ6Pc6gcMoqD9Jc1kQaOf61+5rQaPU1IRDEkYwRM9Ucx4qdoOAQ8mMns+vweuJ9aRgwUGU4\nTlh4JJnxebz38T/JII+NYhluugEwYELzqHR5OrFbArersJmDuOO6H1N8bCdV9cdJCc5kbPoUbBbZ\n+h9u5Fj/+ZEBQBpQabHZpC05lUdoV+lGPtzxBmG6qJ70DaKFm2d/k5U73sLQamCyNhdFUdDQ2Kfb\ngsMRSrDVQVbiWKxmW8C1p+TO4YUTfyRUjSRcRPdM9uqKEJrGZOZhV3oq7ASRzg7WUNNawX8t/l9S\nYkex99AW3D4XM9OuJj91MjqdjhBrKEWe7aQwiiRGIYDjlHKcEjq72voFAFd3B0XHttPuaiUjMY+M\nuNEoihxFHU4u9fz8g00GAOmiGj9qJtnJBRytOohBbyA9LgdFUfj36ieZoV3jXxmkU3SkaqPZXbWR\nzrp2Vmx7nRtnfSMgk6bZaCEyNI59DVtOHhHEhSZjbwzxV/6914oTqXSwl+N1hxiVkH/GjJyhzkhc\nrZ2kKjn+Y+nk0CRqWb17KbfM+Zb/eHndEf616knCRRRm1UZRyU5CQ8P50vx7/OmlpaF1uWXqHAyy\n+SJddDZzEHmpE8lOKsBoMKFpKkII9Ke1PwwYUVDIVSczRp3GW+v/gdvbMzTj9nTx/Pt/wNhoZjoL\nGc8sIpUEuj1dCKX/xnE+PJix8J+1f8ft6TpjuYTQCCWy33En4bR0NPQ5T/DWhhcY5RvLaG0iaUoO\nE3xX0tnUcdYJY2nwFNd3sFp1UdLb6s9PlpX/eZIBQBp0JqOFGGcC1RwPOF5JGRHEAuBQwnEo4Ryu\n7Nk0rrBsG3ZfCGnkYFYsOJQwcsVEuru68Ord1Isq/3XcoovjHCKLApyEU3h0G6UV+yivD1zBkxqT\nRQPVAcd60k3XkBCV5j9W31qN1+Mhkjj/MZ2iI15No/jIjoH9caRP5VLYeH04k0NA0pBYNP1WXv7w\nUTq0VoJUBw1U00ErE5jtP0en6NC0nvX1dU1VhPhCA/YrVhQFB+EkZaWz/cA6rNpBzFhooo5UsolQ\nYilTD/LRjjcJNUTiFl0YLSZunXsXYSFRjM+cwdrd77Hfu4NUkY0AjnEAj+Jm3rgbAsoh0Po9g0BD\np8hUD0Phcs3PP9hkD0AaErFhidx1/c9IzxuNO9pFq9LEBK7EcjKttEu006TVkh7fMz4fGRpDu6El\n4BpCCNpoIjM+ly9c9U08um4iiGUM0zBg4mOxgnbRDAJMXjNjfTOI6Izl1dVPI4RAp9Nz9w33Y4o0\nsY3VbGcVOAV3XvcTLH0moMNDorFZgqjmhP+YJlQqDEfIz5g0CL+W1FfvcE9HiuGyztQ5GGQPQBoy\nQVYHs8Zcg8gXLNv8L3Yf30iULwFN56NGV86CSTf5X7AakzaVjYUrOKaWkCDSUFEp0x0gKNjh37Iy\nOXYURyqLEPSkkY4mnlSuAOAYJexmA5PEHCq7jlLTXEFsWCI2SxBfW/jfn1hORVH4/Oz/x8sfPkaD\nVoVFs9GoryEpJoPxmTMu6m800p0pYZtc2jlwZACQhpyiKCyeditjMqZQWr4Po8HEktQvEd4nqZrF\nZOUb1/wPK7a+zvqapegVA7kpE7h60uf75BgSRCuJOEUE5Rwim/H+z7JEATtYSyM1mBTLWSeGzyY6\nNIF7P/8wB8v30tHVRlLUjXKXr4uo7xDP6WSrf+AM6wDQ5VP7tQByI+VLN5cjRVFIikonKSr9rOeE\nBkdwy7y7/JO2fZPLdXtcHK0+wAyxiBOU4iQy4HNFUQgVETRTT6fWRlz4p38r1GgwkZ8qh3wutpGy\nGctwMKwDgCdSpfxbp/LENJQ7qd3sYo7e9gnfki53Z9pgxePzoFP06NFjI5hKjvY7p5l6unSdLJh0\nEyajud/n0tCSb+8OvmEdABKsrfwmf7n/b0+OygMJ1/LmZoWsQlX2BiS/YKsDuyWIxs4aoojnKPs5\nLIpIpmcPgTIO4lI6uG3+dz6xlyENDdnqHxrDOgCY9VZGBY/x/13VVcrDeUvRcgX3rPiq7A1Ifoqi\nsGjarby+5hnitBSSRIY/rQOA3RjMnYt+SlhI/xe/pKEjW/1DS+n7EsxwM6YgXCz/aHG/46XthXhU\nlWVNeax9c5LsDUh+Da21bD+4jpa2BqxWO+GOaBIj00mOzpB78w4j/pw9CYrM1HkRPH3rjTuFEGfe\n2KKPYd0DOJtRwWOo6irl2vAiFt++jweKr6P2dZ3sDUhEOKK5ZsoXhroY0ieQrf7h45IMAABx1lHE\nWXt6Aw/lLuW92DzelL0BSRq2ZH7+4eeSDQC9AnsDRTxQfC0lmwNfz5dBQZKGlmz1D0+XfACA/r2B\nvtvSCoQcIpKkITLU+fk1VeXYjm1UFBdiDQ4ha9ZVhETL4NPrsggAvfquGOolh4gkaXCc/tImMKT5\n+VWvl2X/90taO9qwFIxBbaph389+wJw7v0PKxMmDWpbh6rIKAGdypiEi2RuQpIHVd4inr6Ec7jm4\ndhVt3m6i7rsb5eT+zrZxY1n797/ylYJx6A1yI5/LPgDAmSeMX94+jrg+cwXRTZrsGUjSpzTUQzyf\n5OjObdhnTPNX/gCWlGT0ISHUHzlMTNboISzd8DAiAkCvgN7AgiIeTLrO/1m5JijZrJdDRJJ0nob7\nFox6oxGPuzvgmBACzeNGb+qfZG4kGlEBAAJ7A7/OW+Y/7tNUtFzkEJEkncPprf7hVvH3Gj3rKtb/\n+0VseXnobT37THTs2IVRbyAiJe0c3x4ZRlwA6HWuCeOXt49j3DJF9gYkqY++OXuGY6u/r5SJU6g6\nuJ+SX/0O6+hs1JYWtMYmFv/vz+Vb4SddkqkgLqaqrlJaPJ1ooqc34NocweeqrYNaBkkabvpO8l5q\nydpaa6up3l+MJTiExBEy+TssU0EoirIWmAr4Th6qFEJkDWYZzqXfhHFcHi9vk70BaeS61DN1OqJj\ncUTHDnUxhqWhGAL6thDi70Nw30+ld4hIUXazeEERDyT0vGEsewPSSCHf3r38jdg5gPOV4xhHVVdp\nQG/AVh74s8mgIF1OTs/ZM1xa/a6WZo5u3YzP3U1iwQTCk2QuoQs1FAHgN4qi/BYoAX4qhFjb90NF\nUe4A7gCIT7APfunOoHdYyKgvZMmCooDP3muWQ0TS5WO4tvrLtm1h9dOPY8vLQbFa2LX8XbJnzmba\nbV+TE7oXYLADwA+B/YAHuAVYqihKgRDiSO8JQoi/AX+DnkngQS7fJzrTyiE5RCRdDoZzpk5PVxdr\nnn6cqLu/iTkxAQDHwvmU/vExksdNID43f4hLeOnSnfuUgSOE2CqEaBdCuIUQLwCbgEWDWYaBluMY\nR5jZzkO5S5n9+e28PLmd1aprqIslSectoNXvtA2ryh+gYt8eLClJ/sofQG+zYZs2mcNbNg1hyS59\nQz0HIIBLvv/Wd4iotzfw5mYFZ8WpDoxMNSENB6cnbBuurf4AAs5UTcihnws3aAFAURQnMAVYR88y\n0C8Cs4DvDlYZLrbeVBMP5S5F5MKDB/qnmpBDRNJQ6G3lE6YjyNHzb7Am1AsMbcK285GQP5buvz2B\nu6ISc0I8AKqri87N28j45j1DXLpL22D2AIzAL4FsQAUOAjcIIUoGsQwX3dlSTXhV1b+KKG6zXqaa\nkAbN6S9x9fYBghjeFX8vk8080YonAAAgAElEQVTG7DvuYe1fn8CWnwcWC1179pI5bSZxcvz/ggxa\nABBC1AOTBut+Q+1ME8anDxHJxHPSxTScM3V+WmlTphGdmcXRrZvwdneT9MPriUhJvaBrdrW14vN4\nCAqPGLHDSUM9BzCiBA4RCe5Z8VVqN7tkb0AacMM9U+dnYQ8LI/+aay/4Op1NTax++jFqD5WgMxqx\nBocw+/a7iM3OGYBSXloGdRWQ1DNElOsch0lv4IkFL+G5uZE3Y7vOuJuSJH1axfUdrFZdlPS2+vOT\nL4vKf6AITeO93z1EV2wkCQ/fT/xDP8dyzTze/8NvaG+oH+riDToZAIbIqOAxhJntPJy3lEduf47d\ni4VcPipdkN5Wf+9yTlnx91ddcgC36sVxzdXojEYURcGen4d9wjgOrFk51MUbdHIIaAj1nTD+64KX\neG9Sz77FfTkrhBwikj6Rf5J3vOmyGe45X0IIWmuqUFAIiYk951h+Z2MDxujofufpY6Jprxl5PQAZ\nAIaBvjuVLbn9VKoJQU9KajlhLJ3NpZ6p80LUHznMyqf+QndnJwiBNcTB/LvvIzz57JPDkekZdL34\nLJrHg67PrmCe4oPETJkxGMUeVmQAGCZ6ewN9VXWV8nDeUjQ5YSydZrjm7Bks7s5Olv3fLwm5cQlh\n4wpACDq27+KtX/wUk82O1ekgf+5CsmbPDWjtO2PjSRk/icqnniVkwVx0VisdH29BaW5h1MzZQ/dA\nQ0QGgGHsbENEsjcwso3kVn+vI5s3YkpPJWj8uJ4DikLwlIl07tqNJTMdY3w82957h7aGBibffAsd\njQ3UlpZgCQlh1u13cnDVSg6sWIXP7Sa5YCLjv3E3RotlaB9qCMgAcAkI2Mz+9n08UHwdta8Hzt/L\nVBOXpzOlbhgprf7ju3aw4+3/0FpVQUhsPJNuvInk8T1zZK6WZvSREf2+Y4yNAQG20VmY4mIo/M3v\n8Xa7OLh2FbbMDHxNzeh9Phb/78/JW3AqDZnP4+bgmpVUlRzAHhrG6NlzCYn+bL9vW20N5YV7MFos\npEyYjMk2fHvtMgBcIvrvVJbP9pqesU4hBLvLnXKI6DLSO8QTlHoqJXpv6oZhm7PnEzRXlFNTcgCr\nw3le2zIe27GVNc8+hfNz1xObnkr30WOs/vuTXPl1H2mTpxEzKpv9LzyDWDgfRa8HQPh8dBXvJ+KW\nmwEwOBzog4I4tHMbcT/7IXp7z2/Ztm4DKx79PTf/6vcoioLH5eLth3+G12LGPCaX+vp6in/2A+Z/\n5/skjin4VM+5/Y1/U/j+Muz5OWiuLja++CxX3/cDEvL6vxg6HMgAcInp7Q3cGLGfGyP2+497c1Q5\nRHSZ6DvEQ9Cpnt6lkrqhL6FprH32Scp2bMM2OhtfYxPihb+z5Ef3ExQeTuPxY1gdThwxsQghOLRh\nLcVrV9J44jgRX74FW85oAOxj8lBMRrb959+kTZ5GfN4YQiOiqH/meYJmXwFC0LpyNYbICLrKjtO6\nYROm6Ci87W1E3PQ5f+UPEHzFDKrWbqClsoLQhEQK31+KLzyUiK/c6p8vsGRnsfbvT/LlPz+Joju/\n1fLVB/dTvGYlcT/+PvrgYAC6Dh/ho0d/z1cefwaDyTzAv+6FkwHgEhRnHdXv2JmGiGRv4NJyOU7s\nlm5YS8WRQ8T97IfozD0VYNumzSz9zYN4u7oxR0XgaWomPCkFZ0wsZft2Yx1XgFpyEOuozIBrWTMz\nqC1/hhe/cwcJuXlMu/WrVO3fR+nKdaheL91Vlej0eoTXi31MPt1lx0DVUF2B79coOh16mw1vdxcA\nR3dtx7ZgTsBksSUrk2ah0VJVSWhC4vk966b12GZM9Vf+ANaMdEyxMVTsKyRlwvDLhCMDwGWi3xBR\nrOwNXCqG82YsF+rAxrUEzZ3tr/wB9E4HHq+H2O/fizEiHKGqNC9dRu3GtejtdroPH0YxGnEfL8eS\nfmpJp7u8AsViwVSQx+GNmynbsY0ZX/4GNz30OwBe+PbtGEdnEXHz5wAIuWIGbVu20bL8A0JmzfRX\n8O7jJ9Da2wlLTqHwg/dorapEe+rvGKMicS68mqDxBaBpaG43BvP5t9pVnw8MFto3b6V98xbUTheW\njHSEEGg+30D8nANOBoDLTOCEcREPFF/L7j77EtjKDTIoDCOXY6u/L83rDVhvD9CxeSuhSxZhjAgH\nQNHrCV2yiPYt24m5+06MEeG0rFpD3UuvEP1fX8OcmIC7opKGf72GOTEBfXAIjlkz8TY1s+ml50id\nNBWfx427o4PwWTMD7hU8aQKNr79J9aN/JXjyRLxNTbSt28Csr99B8UcfsHfNh8R85y5M8XF0HzlK\n/cv/QjHo8dU34IyLJzgy6hOfT/V5Ob5rBx0NDYTFxlG2/F10wUGEXrsIQ2goHTt20bZr93n3Igab\nDACXob69gYfzlkLeqc80IeQQ0TBwObf6+0qbMJmiDZuwZmX6x9K9jU0YwkIDzlMMBvROB1pXz7CM\nY85sPFXV1DzxNMLnQ2e345w/h45duzGEOlGio3AdOowxMoKyHVsIS0gCodH09rsYIyIImjIJc0I8\nwutFURTcFRUYnA4UkwkFhcwZs3j53jsJvvYaWt5fgbu8AkNYGEETxtH46hvYQhws/NH9Z32u5soK\ntr7+CuX79qIzGTElJuCrqkF1uYj7wXf9w0BhixciXC5K1q9h6q1fuTg/8gWQAeAydqaU1KcPEfUl\n004Mjsu91d9X7tWLKPzwfar+9ChB48fhqanF19hIx87dWDMz/Od5qqpR29vRWSz4WloxOB0EjR+H\nu6qauHvvRtHrad/4MWpLG5aMNKofexK1pRXCw9jw/DOYg4IxxcViHzsGX2srtU//ndAli3CXl6MP\nC0XrdNFddhyh+hBC0NXWirujHc+b7xB27SLCb/oc7opKmt56B+H1cusjj5518re5opy3H/optium\nE3PPnXiqa2h+732CJo6HAwcD5gAArLmjqdm07aL+zp+VDAAjzNnSTgC81yTnDS6myyE//5Etm9jz\n/lI6mxqJSs9k0o1fIDw5BeipGHe89Tq1h0uxh4dTcM21pE6ayqQbb2bbsrfpLNqPcLuJve8e6p57\nkfp/vYa9YCzehgZalq9AMZmo+tNjgMAYEYFiMkJrG+U/fwihaZjiYom64/9R9/yLGKOjiPv+feiM\nRlpWraVr/wFi77nTX2nbx+ZT9fu/gF5P0PgCgmdOR+vqonnZ+6iNzbTWVKMzm3EuuYbgaVMAMIQ6\nMTgd1Dz+FAJx1r1qt7/1GvarZuGcexUA5oR4TFGR1Pz9eVBVhM+HYjhVtXqrqgmP+OShpKEiA8AI\ndLa0E/JFs4vncsjPv2/FcnYufwfHdYsJj42mvWg/7/7qfq7/+S9RdApvP/RTgq6ahXP21/DW1LLu\npefobG4me/Yc9ix7B48QGGKiMcfHE/ff99L0zlLqX/wnumA7mtdL+HWLCZ4yCYSgffNWmt55D+Hz\nYU5JQmc00X3kKFW//T3o9ST94ufojD3vEriPlhEyc3pAi90UE4MxOhr0OiK+eJP/uPnuOyn/+UP4\nPG50Oh3W7MAVdebEBBQFjmz5mJTxEzFa+m/hWnu4lNDZXw/8XnISigBDVBT1r7xG+E03oLNa6S45\nRPvaDVz1k18M4H+JgSMDgASc+UWzUwTLt42TL5p9Bqe3+i/Fih96Jjt3vPkqkffcgSmm5xkcs2eB\nqrLz3TdQUAi68gocJ1vFpugojFGR7PjrM4yeM48bHvgVH//zBY5s3ohn/lyMkRF0Hy0j8qtfQm1p\npXNfESEnW+IAITOn07FrD5aMNMIWLQTAW1dPzZ8eQ/V60dlOVcyKxYyvI/CNaSEEamdHz4qePnRG\nI5bMDNzt7Tjj4vFUVGEMC/N/7m1sQvOpbH3/HTY89zRXfONOMmdcEXANe2gY3ro6TNGnWvVqWzua\n14shJARjQxOVv/g1isGAJSiYeXfdd8G7l10sMgBIAXqHiL4Sc9R/rMPXzZIFRf4XzZx9VhUBMiic\ndKZNfS71Vn+vjsYGFKPBX/n3so7Opu6lf/WkY/7yFwM+M8XGgNFAR2MDjuhY5t51Lwk5eWz88+OY\n01MRXi/W7Cwa/vlvTHFx/e5pTkz0t/IBjFGR2CaNx72niK79B7Dl5QIQPHki9a+8in1MHgaHA4D2\nj7egubrwdXQGXFMIgdbQQFBEJOMW38C6F57B4AjBnJyEt7GR+pf+hWPOlYQuWoinsooNf/0bUWnp\nOGJPla/gmmtZ/8oLGKOiMEVHoXZ2Uv+vV9FbrZjbO7nu/l+iKDq8XV3YQkOH9XaTMgBI/Xzyi2ZF\nPHjgOv9xoQmZrpo+qRscp1qmNaHeS77i72UNcaK6ulA7OtAHnfrv7KmqJjgiChB4qmswxZ+qKNWO\nTlSXC2twT6Xs87hxxsWz6H9+yrpnn0R4vfiamuksLMIQFkroogX+YRyhqriKivxpHXrpLFbic/Io\n//d/8MyqxhQfj7ukFDxeKn/1fxgT4tE6O9FcLoSm0VW0n45de7AXjEH4fDR/8BFGdMRm56DodHhc\nnWx94RU8rk40VcVx1ZU4F84HwBQfh33yREo2rGXyF77kL0PalOl0Njez47EnUcxmvO3tOOMSmH7b\n10mdMh39yfF/k7X/8NFwIwOAdF76DhH9Om+Z/7hPUwPSVUc3aWe9xuUYIE4f4unbB7gUUzecjclq\nJfOK2ZT/8zVCb7kJgyOE7mPHaX3vfeZ9616EprHqmScwRkdhTkxAbW+n6dU3yJgxC5PNxoE1K9n8\nyosYQ5342tvxdXejDwul8T9vYR9fgK+hgbrnXsRx1SyEELSsWIna1o4hMtJfBs3tpmPrdmbd9wMm\n3HgThSuW07plB0kpaYx55FF0ej37V31IdekBOpub0XKisU+bQuPrb9L4+psIVcXgdDB6wlR/oMm6\ncg6jrphN0Ypl7CveQ+iiBQHPrQsOwt3Zv2eXv3AxOXOvpr2hDqvDidlm73fOpUARQpz7rCEypiBc\nLP9o8VAXQzqH0vZCvKrKe815bKs+y1inANfmCD5XPfxbReer78TuSEjLrPq8bH7lRUrWrUbR6zGY\nzEz94m2MumI2ACXrVrPl1X/i83pQ3W50JhMp4yeSNmEy6/7xDJF33Y4pJqbnzd/3P8RVWITa3o5z\n4XyCp02hbe0GOvcWgqIgVBVvfQN6m5XgaVNRjAba1m8iMTuHq+/7wTmHVYo/+oDCfTsJ//ItPfMB\nbe0oRgOt768gKy6Vcdd/PuD89vo6Xvvx94j7yf+iD+qpzIWqUvvHx7jy1q+RPH7iRflNL5anb71x\npxDinIWWAUAaEFVdpXT4us/6eW+AWL5tHHGb9Zf0vEHfdfwjoeI/nc/jxt3ZidXhQKfTB3y29pm/\ncvxIKY5FCzCGh9G5bQdtmzbjuHoejtmnJlOFplH+84ew5GTjrakl7nv3+it1ze2m/Be/xmy14m5t\nQ9NUTHY746+9kTGLrw+o/IUQ1B0upfHEcRwxscTl5KEoCq6WZv71/e8Q/e07/cNS3rp6av78BDf9\n6v9wRMf2e66tr/2TAxvXYZ81A53JhGvLNsLDIrnmez8674Rww8X5BgA5BCQNiDPNG5zOqC9k8YIi\nHki4lpcr1PO67nBLXSE3YwGDyXzGzJYdjQ0c2fox8ff/GN3JzVWcixfSvnsPhvDT3vzV6TA4nXTu\nKURRoO65FwmZOR3N7e7J6hnqAJcbU3gY5qREuo8e5fje3VgcTjrr6wlNTCQuJ58Vf3mEppoqLBlp\neN6vwGwwEpedS0VxIRaHg+o/P449ZzSKXo/rwEGm3fb1M1b+AFO+cBvx2bkUrf6IphPH8HV20OTq\nZsdbrzH+us+jN35yCutLkQwA0qDpXWF0enqKTzJcUleMpLd3P6vG48ewJif5K/9e5qREOrfvxJ5/\n6j+6t7EJX2MjeXPmc2D9arrLyvA2NKAzm9EHB9N9+Ajm/DzCb/siiqLgbWyi6g9/ocXVjjktFe/O\nLajP/w1DQjyxP/kBik6H6nZT+cvfcaKplpAv3IjW3Y22YhVWt5es6dNg3BSayk+w+503GDXrKuyh\nYQHl9HZ3caJwN+WFu0Gvx5afS9DE8ZSu3UDdn4+w+Ac/HZTfcTDJACANqjO9hPZJPil1xcXWd7nr\npdrq97rdCFUdlF2pQqKicVdVI1QVRa/HW1dPy0er6D50BOHzUfPYk4TMuwq1pZX2lasJTUyirOwQ\nUXfdQffhI7Rv3Iy3qYnYsRM4rqqE3nitf7ineekygqdPJWzJNUDP0E/ja/9B63L7h2dcu/Zgio8j\n4mTQADCnplD18G8pWvUBbp2CJT8X9cRhdv/v2yz83o+IG92zlFRoGkt/+xBdNgsx37kLBWhds47m\nd5cR8527qP71I9QfOUxkeka/576UyQAgDWuflLriYnuvuWfXNffWqEuu1e9qaWbdc09TsXc3ABHp\nGcz6+h2EJ128pHOhCYmEJybT9J+3sE+bQu3TzxIyawYxd30Tb10dzW+/R+c7ywlLTCL/C19hwwvP\nEP/AT9CZzViSEnHOmU3TG2/jcEaCpqEYe7KICk2js7CI5F896L+Xoig458+j8pE/4dp/ANf+A7iP\nnSB46uSAOQKdyYQ5LRVXYyOx993rDxaW3BzWPP04X/rjEyg6HeWFe+hwdRL9rf/nPyfitluo/tOj\ndB0sxToqk4bjZTIASNJg+7S9hoHQu+vatWH7WBabz8cHpw9uAS6A0DSW/uYXkJlG4sP3oxiNtG/d\nztJfP8AtjzyKJTjkot17wXd/wIbnn+Ho40/hmH0FoQtOrqmPi8UYE0P9409x9be/R2XxPqyJiQH7\nBACYM9Jp2F1ETE4e7Vu24jgtvXO/Z/V6aXp3WU+q5+oa3BWVgZ8LgbuykqDpUwMmcq25o2l58x1a\nqisJjU+k8VgZ5lEZAecoioI1KwtPRQWeikqC5y7icnNpTW1L0iCJs45iVPAYnCY714fvZ1bOFoqC\nd7LTWzHURTunyuJ9uIWG89pF6CwWFL2ekOlTMWdlUrJ+7UW9t9lmZ9493yU4MgpbfuBEjykmGoxG\nqg8U44iOobuqCnHaRimeExWExsRxxdf+i85Va2n856u0b9qMwemkZeVq/3lCCFo+WoViMhL3/ftw\nzJlN5Je/hGvvPjoLi3re+PV6aV2xEuHqwnj6BvJC9KSZPvnSVnBkFL6q6n7P466oxHOiHLNOT3zO\neU5cXUJkD0CSPsGp1U2FLHDsZVlzPjvr0slrnzCk5fokbXW1mBLi+q2VN8TH01ZXMyhlCI6IxF1T\nizkxwX9MdXXh7ehgxaO/Z8w1S4gZlU3jK6/hvH4J+iA7nbv34tq6jbyHf4cjOpZb/u8vlKxbQ1N1\nJbrwCBo/3or72AnMqcl0HzqMp7oGx9w5/nQRhlAn0bd/ndpnX4B/vYbweQmOjmX8dZ+jaPV6rFlZ\n6Ew953Z8vIWg8Aj/iqCUSVPY/OpLtK5aQ/DJ3cNa12+k+/BhYrJySJ09g0Ob1hOXm0/1/iJcLS3E\nZI8mKj1zWKd6OBcZACTpPPTORVwfvh+DTsdWwN0UzQRjwjm/O9giUlLpfvs1/2RsL0/pISJnXDUo\nZRh7zbWsfOpRTDHRPW8Gd3bS8Nob2McXELbkGor/+BjzvnUvh7dtpvTh34Beh8lqo2DJDYRE9cy1\nWIJDGLvkejwuFy99+3bif/R9usuO4a2rxzF3Dg2vvwFa4JvnlvQ0DBFhWFJSsI7OouPNdxm75Hoa\nK05Q/qvfYh2dja++EdHSyrU/fsD/PYPJxPU/fYi1zz5FxYoHAQhPS2fckhsp/OA9Oo0Kwuuj45kn\nMMfEYElLZfeK94jNGMX8b38PnT7wfYhLhQwAknSeLpXeQGRaBhEJyTQ8/xIhC+ahM5tp37AJpamF\n9OlXnPsCAyBxTAHTv/hlPn76WXyaBpqKfVwBYTdej85kxD59Kke2b6G7rQ1rfDy2mdPA56Nw3Uo6\nGhu44ht3+K/V1dqC3m7H4HQSNO5Uds/2LVtp27CJ4KmTMYQ6e84tOYS3qprIr9yGMdRJXUMDOp2e\neXd/l4p9e9j59hu0trQQFBlJw/EyHLGnekoh0TFc95MHcbt6Esh1t7byxv0/JOZ79/qHkIIrKql5\n4imivvVNnNctpu7JZziw+iNy5y8clN91oMkAIEmf0nDvDSiKwsL//iG73nmD0pf+jer1kDphMpMe\n+BXGT7HJ+YXKurJneGbriqVE/NfXAt8PMBjoqKigobaamP+5z7+Bin3cWA79+hFy5y0kLDGp51h4\nBFq3G29dPcaoU7mBTLExdB86QsVvHsGanYXmcuE+fgJLZiamyAg69xURejINs6ulhdVPP455TB6h\n18zF19zCptdfoaWmiok3fiGg3L15fYo+WIZtwriA+QNzQjzW7Cxc+4oInjaF4KuupHTjBhkAJGkk\n6e0NBBlKA3oDwyUQGEwmJt98K5NvvnVIy5E0djzrn30KX0srppieAKC53bi2bsORnIalID9g9yyd\nxYItL5fK4kJ/ADCYTIy/4fPsffYFHDcswRgdRVfRflxbtjP/nu+yb+UKaoqKEIqCdXQWYdctpn37\nTlrfXca8u+4DYN+KZZiyswi94Vqg5+U0c3ISe3/3B/KvXozZ3j+Zm+r1opy2oT2AYjIhvN6eP/R6\nNPXsCRA7Ghso3bCW7o52EvLGkDhm3LBKKzGoAUBRlDDgWeBqoAH4sRDilcEsgyQNpJ5A0NMbWBy6\nj4dKbmBnE8MiCAwHZrudmV+7nU2PPYltwjgUq4WunXtIyh1DaEwcTeWH+31Ha2vDbA9M/VGw+Hps\nIQ72fvAerc1NRKVnMvunDxGRkkrq5GlAz+qnne+8QdNTz+KMS2DhfT8g7uTKnerDJVhmTA64psHp\nwBwdTVP5cWKzc/qVI2XCJIr/9DtC5sxGf3IDGl9rG659RTivnotQVTrXbyR/yrQzPvvxXTtY9eRf\nsBWMRecI4fDLzxMe/QEL//uH/pTRQ22wS/EE4AGigQJgmaIoe4UQxYNcDkkaML29gaquUu7Pejug\nN9DXSA0KWbOuIiYrm0ObNuBzu0m++7vEZI2ms6mR3T98E9ukCVhSUwBwFe/HffwEqd+b0u86o66Y\n7c88eibxufnE5+af8bOg0DCaa+shZ7T/mPD58DQ29ksJ0SsqYxSZU2Zw+A9/xjZ5EsLno23DJowR\n4XRs24m7aD/O0DBy5vUf/lF9XtY8/TgRt3/d/2xi7mzqnvgbhzauI3v23LM+x2AatGygiqLYgWYg\nTwhRevLYS0ClEOJHZ/qOzAYqXWp6s6L6NJU/Hz+1Q1ZbtxsYXnMFw8HxXTtY/fRjGMPCEKqK6HRx\n9X3/Q8yo7AG9T93hUpY98ivC/+trWFKS0dxuWt5djr3dxbU/uv+s3xNCUFt6kCPbt6DT6UgaM47m\ninI6W5qJzR591iGd6oP7WfmPp4n+3r0Bxzt27ca8r+Si5xUajtlARwFqb+V/0l7gykEsgyRdVH17\nA/+T+p+Az3yaKoeITpM8fiJfffzv1JQeRKfXE52ZdVGWVEZljGLW1+9g0wvPoSkKalcXcXljmPPt\n733i9xRFISZrNDFZp3oO8Xljznk/nV6P8Pr6HRde37BaMjqYASAIaD3tWCsQ3PeAoih3AHcAxCdc\nmrvsSNLZttU8fYhIBgLQG41nHboZSOnTZpA6ZSrttbWY7EFYQy5eSozI9Ax0Xh+de/dhH9vzbFp3\nN53rNjLuC7ddtPt+WoMZADqA03/xEKC97wEhxN/4/+3deXRURdrA4V91Z09nXwhbQgDDLoggiMii\nyI6AMIKIArKoiKI4Cp8oKuIoOqIOAwroqCOizgiiCKg4iIhsAoLse8ISkpCdLJ2ku+v7o0PohAQS\nyNJJ3uecPofce/tWUTT3Tb1VXQWLwZ4CqpyqCVHxZMC46hkMxkIbvFdkOb2n/pU1b75K9tbfMfj5\nkrX/ADd0vo3Ijp0rvPzSqswAcARwUUrdoLU+mn+sLSADwKLWcEwRzW6xim+SWkpvoIYKbXID97+7\niJid28nJyKDesFEE1G9Y1dUqpNICgNY6Uym1ApitlJqAfRbQYKD6LLMoRDmp5xlV8GWyAQF7WZ1i\nX3FUgkDN4uruTtNK+vb1tajsaaCTgX8BCUAS8KhMARW1lWNvYHDQAWiO9AZEparUAKC1TgaGVGaZ\nQjg76Q2IquIcX0cTopYr2htwaWlgW1ykfJlMVCgJAEI4kaIrjv6Q1rbgnMVmkxSRKFcSAIRwQhdX\nHB0RcmmtnAyLWVJEolxJABDCSZX0ZbKiKSIJBOJaSQAQohqpLpvSiOpBAoAQ1ZCzb0ojqgcJAEJU\nU9IbENdLAoAQ1VxxvYGiJCiI4kgAEKIGcNyickDAvkLnVqe0lhSRKJYEACFqkOJmDkmKSJREAoAQ\nNZwMGIuSSAAQohZwTBE59gYkENRul29mKYSosep5RuHv5s3goAPMarYS98B4duadqepqiSoiPQAh\nahnHheeKblHpSHoGNZ8EACFqqaJbVL4TM6LgXLo5h33ES4qohpMAIEQt5tgb+GvkV4XOWWxW2be4\nhpMAIIQoceG5oikiCQQ1iwwCCyGKJQPGNZ/0AIQQJXJMEc1usYpvklqyM6FJoWukZ1B9SQAQQlyV\n477Fg4MOFBy32KySIqrGJAAIIUrlShvUDAjYKwPG1ZAEACHENSspRSS9gepBBoGFENetnmcUJheP\nggHjLs03y4BxNSA9ACFEuXDsDQwOOgDNkd6Ak5MAIIQoV44DxgMC9rI6pQ2bD3W57DoJClVPAoAQ\notwV7Q24tDRwMLV5wfl0c44MGDsBCQBCiArjuAx1RpGdymTAuOpJABBCVLiiU0iLSxFJEKh8EgCE\nEJWuuBTRtrhI6Q1UMgkAQogqc6lnUHjfYkcSFCqOBAAhRJVz3Le4pKUmZDP78icBQAjhFK601IRs\nZl8xJAAIIZxWSSki6bIFrFUAAB9ISURBVA2UDwkAFSg7y8KCf+zlu29OYsmz0btfOE883RZ/f/eq\nrpoQ1Ypjikh6A+VHAkAF0VozYcz/CDJl8+V7gbi5Kd5dEsd998TxzfcDcHMzVnUVhahWSuoNSCC4\ndpUSAJRSG4DOgCX/0FmtdbPKKLsinTuXxfIvj5OUmM0tncO4q28DXFzs6+tt25pA/Nl0ftrQAKNR\nAbDozRDuGB7H2tWnGDw0sszlaa35aMlBPv7wIPFxZm5qH8jTM26mY6fQcv17CeHMHHsDjstQO5KA\nUDqVuRroFK21Kf9V7R/+mzaeo2/Pb0k5E80NdZJYPP93Hrj3R8xmKwD79yZzZzePgoc/gFKKPt3d\n2b83uaTbXtE7f9/Dt18dZNmCIM7uacSkkUYmjlnP3j+TyuXvJER1Uc8ziiifG/F382ZWs5V0a7m1\n0Gufz05ZjbQUJAV0DaxWG88+9RufvxdKr25eADz1sD93j4njs0+OMP7hFjQMN/Hjt3mXvfeP/Xnc\ndJupzGVmZebxryUH2f1TAxrWdwVg9HBfklKsLF6wj/mLul/fX0qIasieFjrCiJBjBccyLGZJEZVS\nZfYAXlNKJSqlflNK9SjpIqXUJKXUDqXUjuSknEqsXukdPJCKp7suePgDGI2KyWN9Wfd9NAB39KrP\n+RTFK/OSycqykZuree+TVDZtz2HIsLKnf2JjswgKdMFqg4++SGPl2gxycmx0v9WLX9af4YERP7Jt\na3x5/RWFqDbqeUYVel3sGchm9ldXWQFgOtAYqA8sBlYppZoUd6HWerHWuoPWukNgkHPMljl0MIV1\nP5zmzOkMANzcDJjNGq11oeuyzTbc3O2Duy4uBpb+pzebdrsR1uYkwS1O8Pkq+Oy/vfH1dStzHcLC\nPIlPyKVjn1Os/zWbf36YStPO0fznm3S6dPTgwSHw6EM/s3WLBAEhiqaIujTfLGmhYlx3Cih/gLek\n/MNvWuuuWuttDsc+UUrdB/QH5l9v+RUpLS2XyRN+5vjRVNq08GDbriwGDIrgldc7YfJ15+Mv0xk3\n0g+AzCwbby5MZ/T4mwreX6++Nx991ouMjDxsNl3qB//xY2nEns2iZasAgoI9APh9+3mCAoz8/n1D\nggLtQeY/315g4rR41iyrx223eGE0wPy3dtP5qz7l3BJCVE8XU0SOA8b7KPxLUm1OEV13ANBa97iW\ntwHqqldVsZdmbiOqYS7rlobj4qK4kGFj0INxfLTkMP94vztj7lvHp19l0TjChTU/ZdKrd0OGDr88\nvWMyuZaqvJQUM1Mm/sKRwylENXFnz34zD4yJ4tmZ7Vn51TGmTwkoePgD3Hu3D6+8lYxS9qbs08OL\np16U33CEcFR032JHFpu1Vm9mX+GDwEopf6AT8Av2aaAjgG7AkxVd9vXIzrLw/erTRO+IwMXF/oD1\nMRl4dUYAD08/wsRHW/LL1ntY/9NZEs+b+eyRUJo197/m8r7+7wlmz9rG3X28+HFpBK6uivOJFvre\nd5LGTf0xmy2YvC+PmQF+BrLN9lTUrr05hId7X3MdhKjJSlpqYlazlbV2wLgyZgG5AnOA5oAVOAQM\n0VofroSyr5k5x4pS4OdbeJikToiR9Av22T3u7kb6DQi/7rLWrIrhjVe3k5Nj5e3ZIbi62h/0IcEu\nzJ4ewKv/PMS99zdn8dI9jBjsUxCQ/thr5sDRXLp08GDXn2Yen5nEtP+7pdTlJsRn88Gi/WzfEkdw\nsAf3j21BzzvrX/ffR4jqorgUUW3qDVR4ANBanwc6VnQ55c3f343GTUysWJPB8IE+Bcc//vIC3XvU\nu6Z7Wiw2ln5yhO9WHicvT9OnfwTjJrTgvfl/8tIzAcyam4S3V+Hf8hvUdSElJZehwyJZ8+0Jbh0Y\ny/1DvTgbb+XDz9Kx2RT1b4rBZHJh6rR2DBocwR+7Evnt13P4+bkzcHAEAQH2wfRz57I4dCCFBg1N\n+Pu7MaT/agbd5c5bz5s4eSqPF2f8RszDbRg7ocW1N5wQ1UzRFFFt2qlMFZ3J4kxubBek16wbUGXl\nb9+awKSx6xk/yod2rd348Rcz6zaaWb6qP/UblD3V8tjEDaSeT2bGFD/c3BT/+CCd8+nuHNifyuFN\nDblt0BkW/T2Unrddml76zMuJJGQFMvmJGwmt48HP/4tl08ZY/PzcGT6iCQ3DTVxIz8PXzxWlFM8+\n+Rtbf4vlngFexMbbWPdLFgsWd2fd2hhWLD9Jh7ae7D9sxtvkRteOLnw479K3iI+dzKXzgFi27hqG\nl3fpxi2EqElis4+QYTEXLENdXXcqW3Tf0J1a6w5Xu06+CHYFt3QOZeXaAXz278MsW51Oy9YRrJkV\nRWCQR5nvtfuPRPbsSuDgrw1wdzewZ38O/r6wbWcKHu4Gli5PZ97sYEY9EscTE/xp1dyN5d9l8N26\nLHJyUlm3NgYbBqbPvJk5r3cudO/fNsXx2ccHOXY0HX8fzb4NDfHysqeuftqYxX2TfqFppAsntoXj\n52vEYtFMejqe2HOFv6jWNNKN8PquHD6cxk3tg6+94YSoporuVEZzCnoDjqpjUCiOBICraBTpw8wX\nrxpIC7FYbKz/6Sz/W3eG7KxcGkX6gYJ+d3rh7m5g07Zshj10jmenBDBxtB8bt2Qz641kZk4N5KsP\nw3hjQQrvLk7hQpaNLh08+WJRXYICjezel8PA0b8TWsezIFf/3vy9LP/8EHNmBLBkqZGRQ30KHv4A\nvbp5ERwAo4aY8PO9+B0FxbzZITS86SSZWTa886/PybFx9lweISFlD3BC1CT1PKMKrTf0Q1rbgnMW\nm63GpIgkAJSzg/tTGHf/T2Rl5dK+jTvDB/mw92A6y1ZkEBlhT6vM/Fsi78wJ4b6h9rGFLh09qRvm\nwtMvnmfO2zb8fY3YNLgYFS/+Nahg6me71u68MSuIf767h5531icjI4+F8/fxx7oGhDdw5YuVF0qc\nWxvgX3gw29fH/vPOP8106+yF2Wzj2TnJtLspiAYNy75UhRA1jWNvoOhSEzVlM3sJAOXIarXx8EPr\nuam1gdBgE0veutRt7NXNk3FT41n4UTKbd5hZ99/CD9nhA01MnBZPUICR554MpFN7D/73axbDHjrH\nqqX1uOUm+2/lrZq5c+ZUKgBHD6fSOMKN8Ab2wDJsoIm3Fqbwl0EmPD3tD/j1m7I4HWth/6HcQuV9\nty6TkFAPho+PJzTElfjzeXToGMK8BbdXWPsIUR1daacyxxRRdQwEEgDK0a4diXh5aGJOW3n+qaCC\n4/sO5bD9DzMWi2b22xdwd4Pj0Xm0iLr0zeBjJ/Pw8TawZF4oA++yB4eb23oQFGjkpTeTWLPMnvJZ\n9UMGIaH2QeI6YV7EnM7FbLbh4WHgL4NMfL8+k+ZdY/jLIBPRp/P4cUMWbZq7seDjNBJTNH17erDn\nQB7vf5LOwg960L5DCMePphEY7EHdul4IIa6uaIqouvYGKnMxuBovM9NCoL8RL08DqWn2ZaEXfpRK\n73vPYrNpnnksAKOy4uJi4JFn40hKtl9zPtHCQ0/GcSHTRv87C88uGtLXxKZt2ew7lMOct5OY+88U\nHsyfplmvvje3dK7DlJmJpF+wYjAoHn7Qj9Q0K1t2ZNO6uTvzXgkh/ryVJhGunEn25l8rFGdSA5g8\ntQ0nT6QTdy6LVm0C5eEvRBkVXXju4jLUji9nX3tIegDlqMMtITx+0MxjY32ZMy+ZJhGuvDA3iZ3r\nwmnU0J6meXJSAC1vj6ZOiCtRXaKJaOBCzBkLzZq64uamOBGTR9PISz2Dw8dz8fRQDLg/ltQ0Kw3q\nuvDyzG0c2pfE8y935K35XXnur1uI6BCDh5t9TaJhA0189G5YwT1ubOHO3Q/E0u4WT4YOb8KksT/T\n/kZ3QoONvPm3nYx6IIpnnmtfsKSEEKL0HHcqGxCwr9A5Z/9OgQSAcmQyuTJrdkdem/07EfWNtLvz\nFHd19yp4+IN9OQn7b+k2jm5pRPTpPCLDXVm24gJmax7jn0rgqw/CCAl24UxsHo88k4C7m8JsthEW\n6kKzG9w4HZvFRx8eZvvWeJ6ffQudb6vLzbfUoX2HYMbct45e3by4Z1wsB4/mkpWtyc62YbFCbp5m\n8sRf+Nc7wQU9jaTkQLoMOk7n2+rS7Rq/4CaEsO9U5qg6pIgkBVTO/jKyKUv/04eIqDBsGrKybJdd\nk5FpIyHRQmCAkfY3euBjMvDp8kwmTW5DjvaiSadomnWJpuXtMeRZNI895Mf9w+w5/Q43uhO/vzEp\nh5sQHprLgyPW8efWQ+z89TAjh/6IxQLTX0lkYG9vPv5HGKOH25eOmPFEAGtWnSQsRBVKMwUFGnl8\nvC/frDhemc0kRI1XUorImdJC0gOoAC1bB/LOgm4knf+RX7YksG2XmU7t7bN4Tp3J46PP08k2a95d\nkkKgv5EPlmVg8veh/8BwwsI8GT96Hbl5mlnTgvD2Vrz/SRrn4i24u8Hz0+yDy3sP5rB9dw4Hfo2g\nYX1XTp/N4+W/a778JoMfPm9Ah3b28jq198DdTXHsRB5jR/iw5ffLN9nx9FDk5lgrr4GEqEVK2sy+\n9YWbq7ReIAEAgLi4LFYuP0lqSg633V6Xrt3CrjsffvxYGvv2JtO+jRt9R5yhZ1cvvL0MrF2fyaxp\ngSz85AKrN7rg5eXCPaPa0rFTCC/N3M6vG86iUfzxUzj+fvb5/w/e60vUrdHUrXMplfTlyguMHeFL\nw/qu/L7bzKDRsfTv5YWfr6Hg4X/RkH4mRj0Sx/J/hbH436fZsz+Htq3s6wPl5NhYvDSDcY9W+22a\nhXBqjpvZuxgMbKPq9yKo9QHg5/VneXLyrwzt5039ugZeef44kU2DWLCkBy4ulzJkmzae4/1//smJ\nY+nc0MyPR59oS+db65R435XLT9LvDk82b8/C01Nx4HAu3bp48ufPEdQLc2HFWjPjJraie896nIvN\nZHC/1TwwzIvBvd3IzHItePgDeHsZGHOvL4v+ncLjzyUQf95CUrKVzjfbH/TTZp3n7y8FM6i3N406\nRJOSaiXA/9L7j57IpW4dIweP5hEabODOYWcw52oC/AyAgfYdQxkw6PpXNRVCXFlJvYGqWmqiVgeA\n3Fwrz0z9jRUf1uH2zp4A/N/jgdwxPJavvzrJX0bad6386ccz/N/Tm3jjhUBu7RDGxq3ZPDbhZ95Z\n2J3bu9ct9t6HD6Ww+dcMnpsawJB+Jg4fz+PZl8/z5coL9O/lzd6DZm7uGALAh4sOMOJuL16bGcyH\ny9JY90tWwX1ycmys35TNqh8zsFghIdHC9j9yiIu3sG1XDjm5sOtPMyOHNMDFRXHPABNTnkvgvbmh\n+PoYOXoil5mvJTF2hC8Tp8WjNbz2fDDDBpg4ciKXyTMSiWzsh9FYuuGgtLRcNqy3T2vteWd9/P2d\nY9tOIaqTor0Bwk4WnLu41ERlpIhqdQDYvSuJenWMBQ9/ADc3xWPjfFm6KrogAMybu4slb4UUDJ42\njnDF5K14681d3N69+NVKjxxO4clJ/vx1ciBgX2itWRNXOvY+xev/TGHmrI4FO4X9sTOB16fb5+Hf\n09/EjDmJbNySjU1rRj0SR2S4KyZvA0opft6UzX8/qEu3Wz05dDSP+x4+h1KK9As2AgOMzH81hMkz\nEmh0czS+PgYSk60oA3zwWRquroonJvgzcbR9G8vON3uyemldWnU/wqNPtMHH58pbVn73TTTTp23m\n5rbuaJvm2ac207d/OO8uvB2DQaaQClEWF3sDJpcjhY5nWMyVliKq1QHAaFTk5l2+HHZuni7YdMVm\n0+zdm0afHiGFrul3hzcPTkko8d7ZGXn0vzOw0LGmkW6YTAb6DGrKyNE3FByvE+bFwaM5dO/iRYC/\nkWXvhTHg/jOAYuUn9bjzdntw2Hswhx5Dz9Cwvn3p5xZRbiz/qC7t7jjF0y8m8MHbYXh5GXhvbijH\nY87SvrU7c2YEYdP26adNO0XTs2vhL3zVreNC/bqunI7JoGXrwvV1FBeXxfSnN7Ph6/oF4wd/Hsjh\ntoGnmD3rd16aU/qNaIQQlxS31ERlDRjX6mmg7doHcSEDvv0ho+BYRqaNd5akM2Cw/bd/g0FRv74H\new8WXktnz/4cGjb0pCR16nrx58HCM25SUq0kp9gY5fDwBxgzoSWvvpvGjt1mwP6buZuroltnz4KH\nP0CbFu6MHu7DsuXpBceaNHLDzU2xal02kR1jGDD6HA3bRxNz2sKbLwbj42MsWAU022xj687sQmUn\nJVs5ey6PelfZ32DNqhgG3eVV8PAHuLGlO/febWLZp0fJzrZc8f1CiNJznD7aKexkhU0freU9AAPz\nF3VnwoPr+WBZBg3qGvn2h0x69YkoNCg64eFWTHz6AJ+/H0rTSDcOHc1l8owkxj/cptj7Rp+8QIdO\ndZn2wmEiw13p0cWT+PNWxk2Np1lzP5q3CCh0fafOdZjxQkeGPLQDo7KRmGTB20sRHGS87N7BgUYu\nZF7qtezZn0NurmbBku7UCfPkVEwGf33Rj6ef+JVHZyQy7WE/rFZ4fX4qyujCi3OTiGjgyqDe3kSf\nzuPR6UncMzwSf393YqIvkJtro0lT38tSOuZsa8EKoo78fI24uSrOJ2QTHuFz2XkhxLVxTBE59gYc\nXW+KSHYEAzIz8vhh7WlSUnLo0jWMFi0LP6C11ix8dy8fLDqAfXao4pHHWzPxkZaFpotqrXn15R18\n9cUx+vT05uSpPPbsz0YBNg1du4ax8MMeeHgUH3ctFhtPTN6EOSWBmU8GMnB0LMe3RRYEguxsGy26\nxnD/MB+eejiA3ftzmPBUPFEtQ/h4Wa9C90pLy2X+vD18vzoGZVAMvLsRU568kV07E5k753f270/H\n19eFB8ZEcfc9jXlm6ibOnsnAw92A0dWF1+fdxq1dLs1MOHQwhXsHr+XI5ggCA+z1SU6x0qZHDFlm\nxc599+LhcXnAEkJcv4s7lTm6uGtZcSmi0u4IJgGgDHJzrSQnm9m7J5m130Vjtdjo078RfQeEYzAo\nvl9zirdf28bGlfUKpmF+tjydF9/KYO36QXiXYpvFN1/7g5yk0yycG8qTzyfw9ZpMpk7yx8tT8fai\nVLTW+JgMRJ+2EOhvIClVsX132bdwzM214upqwGLR9OzyNdMmefPoGD8MBljzvyzGTT3P9xvuJizs\nUgpq2uOb2Lj+FI+Pt29ws/jTdKxWeGBcKx578sYrlCaEKG8Xg8LqlNZsi4ss1BsobQCo1WMAZeXm\nZmTJwv3MfXkr3dtl0ruzmYVv/860x39Fa803y48x7RHfQnPwR93jg6vRyrGj6Ve48yX9Bkbw9dpM\n+1jEnFA++kcdDhzOYdbcJLr0bExAsA8xZ20og4HgOr6s+K7vNe3f6+ZmRCnFhvVnqV9HMeUhf4xG\nhVKKAb28GTbQm6++KLw8xLz5XXlsWnsWfZbF395JwapdeHTqTUyeWnwqTAhRcS4uNTEi5Bizmq2k\nS/PNZR4rqNVjAGV1+FAq36w4wYGNDQu+qDV6mA/tep1h+7YEcsxWfEyFY6pSCpO3AbO5dIOkrdsE\n0rtfBLcOOMtTD/tiNCp27c2jy+31mfNaR5RSxMXZvyfg+Nv5tUqIz+aGxpcHkKhIF/afybrs+Ljx\nzRk3vvl1lyuEKB/2sYJLC8/NPjyk1O+VHkAZbNp4jsF9vQt9S9fT08CIwd78sj6WO3tH8N4nGVit\nl9Jq23aZORNrod1Npd9k/W9v3sqT0zux6hd3vv6fKxOf6MD8Rd0KxhvCwrzK5eEP0OGWUH74ObPQ\nonVaa77+PpsOnUr+prMQwnk4Ljw3q9nKUr9PegBlYPJxJSHx8tU9ExJthEa6MnxkE9aujqbLoFhG\n3u3F6XNWln6Vwdy3b8PdvfQDpEop+g0Mp9/Ail+eoVlzf7r3bMBdI88xY4of3l4GFnyUTq7Nnb79\nG1Z4+UKI8nOxN1BaMghcBmlpuXTrtIIv3g8tmJ+/Y7eZvqPO8cPPd1O3njcWi411359h86ZYAgI9\nGHZvEyIaOff0SKvVxn+WHWPl8uPk5lrp1SeCcROaXza2EBN9gUUL9rF7VwJ163ozdmKrEpfCEEJU\nnYahn8osoIqweVMcjz+ykchwF1xdFAeO5DB3Xhf69q/Zi6mdPJHO8EFrmTjaxKDe3hw4ksusN1KY\nNv1m/jKyaVVXTwjhoLQBQFJAZdSlaxhbdg1j25YELFYbnW+tg6dnzW/G9/7xJ4+O9WHWNPtyER3b\nedCmuRuDxuxi6PDGhVZOFUJUDzX/yVUB3NyMtS71sWvneZ4dX3itoPY3euBq1Jw9k+n0aS4hxOXk\n1zZRKqGhnhw5UXg9pJRUK2npVgICZUloIaojCQCiVB4c35Ln/pbC4WP2IJCWbuXRGYn0HxiOr++V\nl5EWQjgnSQGJUunbP5y4c5l0G7KHgAAjCect9O7bgNmvda7qqgkhrpEEAFFqY8e3YOT9UZyKuUBI\niKekfoSo5iQAiDLx8DAS1cy/qqshhCgHMgYghBC1VLkEAKXUFKXUDqVUjlLq42LO36mUOqSUylJK\n/ayUiiiPcoUQQly78uoBxAJzgH8VPaGUCgZWAC8AgcAO4MtyKlcIIcQ1KpcxAK31CgClVAeg6P5k\n9wD7tdb/zb/mJSBRKdVca32oPMoXQghRdpUxBtAK2HPxB611JnA8/7gQQogqUhkBwASkFTmWBhS7\ndoBSalL+eMKO5KScCq+cEELUVlcNAEqpDUopXcJrUynKyAB8ixzzBS4Ud7HWerHWuoPWukNgkMwz\nF0KIinLVMQCtdY/rLGM/MObiD0opb6BJ/nEhhBBVpLymgboopTwAI2BUSnkopS4Gl6+B1kqpYfnX\nzAL+lAFgIYSoWuU1BvA8kA3MAEbn//l5AK31eWAY8CqQAnQCRpZTuUIIIa5ReU0DfQl46QrnfwKa\nl0dZQgghyocsBSGEELWUBAAhhKilnHpTeKXUeSCmlJcHA4kVWJ3yVp3qW53qClLfilSd6gq1t74R\nWuuQq13k1AGgLJRSO7TWHaq6HqVVnepbneoKUt+KVJ3qClLfq5EUkBBC1FISAIQQopaqSQFgcVVX\noIyqU32rU11B6luRqlNdQep7RTVmDEAIIUTZ1KQegBBCiDKQACCEELVUtQkA5bnvsFKqUf41Wfnv\n6VXBdc8o8rIqpeaXcO3Y/POO1/eoyPoVU4cNSimzQ/mHr3CtUkrNVUol5b/eUEqpSqyru1LqQ6VU\njFLqglLqD6VUvytcX+ntq5QKVEp9rZTKzK/nqBKuq9K2zK9DqdvTGT6r+fUo1ee1qtvXGZ8D5bIW\nUCW5uO9wH8DT8YS6tO/wBGAV8Ar2fYc7l3Cvz4EtQP/811dKqRvyF64rd1prk0NdvYF44L9XeMsW\nrXXXiqhLGUzRWn9QiusmAUOAtoAG1gEngPcrsG6OXIDTQHfgFPZ/z/8opdporaNLeE9lt+8CIBeo\nA7QDViul9mitiy6JXtVtCWVvT2f4rELpPq9V2r7O+ByoNj0ArfUKrfVKIKmY0wX7DmutzdgXpmur\nlLpsATqlVBTQHnhRa52ttV4O7MW+YmllGA4kAL9WUnkVbQzwltb6jNb6LPAWMLayCtdaZ2qtX9Ja\nR2utbVrr74CTwM2VVYcryf+PPgx4QWudobXeBHwLPFDM5VXaluD87Xmdqrx9HTjFc6DaBICrKMu+\nw62AE1prxx3J9pRwbUUYA/xbX3n61U1KqUSl1BGl1Avq0t4Klem1/Dr8dpWuZ6G2p3Lb8jJKqTpA\nFFfecKgy2zcKsGqtjzgcK6mNnKotoVTt6QyfVSjd59WZ2tcpngPVKQV0JSagaPqmpH2HS9qjuH4F\n1KsQpVQ49q71+CtcthFojX0NpFbYU1kW4LWKrp+D6cAB7GmLkcAqpVQ7rfXxYq4t2p5pgEkppa7y\n4S53SilX4DPgkytsOFTZ7VuWPbGdpi2hVO3pDJ9VKP3n1Sna15meA07RA1CVu+9wmfYovpoy1v1B\nYJPW+mRJ99Nan9Ban8zvfu8FZmPvLpaL0tRXa71Na31Ba52jtf4E+A17Lrg4RdvTF8gor/9QpW1f\npZQB+BT7Q2BKSfer6PYtxvV8Nsu1LcuiNO1ZBW1ZrDJ8Xp2lfav8OXCRU/QAKnnf4f1AY6WUj0Ma\nqC2w7FoKLmPdHwReL2sRQLnNVLjGtr5SHfZjb7/t+T+3pRz3ey5NffNncnyIfZC1v9Y6ryxFUI7t\nW4wjgEv+JIOj+cdKaqMKbcvSuo72rOi2LK2S6uEU7YsTPAcu3VXravHCHqw8sHeBPs3/s0v+uRDs\n3blh+cfnAluvcK+twN/zrx0KpAIhFVz/LkAm4HOV6/oBdfL/3BzYh33AurLa2R/7TCuP/Da/P7/e\nzUq4/hHgIPYUWj3s/6EeqeTPxvv5/6amUlxb6e0LfIF95pk3cFv+Z7WVM7ZlWdqzqj+r+eWW+vPq\nDO3rbM+BSv1gXWfDvYQ9Cjq+XnI43ws4hH0/4g1AI4dz7wPvO/zcKP+abOAw0KsS6r8I+LSY4+HY\nu6bh+T//Hfv0sEzsU9RmA66V2M4hwO/YUxSp+Q+CuxzO346923zxZwW8ASTnv94gf4mRSqpvRP5n\nwZzfjhdf9ztL+wKBwMr8Mk8Bo5yxLa/Wns7QlmX5vDpp+zrVc0DWAhJCiFrKKQaBhRBCVD4JAEII\nUUtJABBCiFpKAoAQQtRSEgCEEKKWkgAghBC1lAQAIYSopSQACCFELSUBQAghaqn/B2TK9kWHc0tx\nAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "clf = NeuralNet(C=3, lam=0.5, num_neurons=np.array([2, 2, 3]))\n", "clf.fit(X, t, show_message=True, ep=0.1, alpha=0.1, ftol=1e-5)\n", "ax = plt.subplot(111)\n", "plot_result(ax, clf, xx, yy, X, t)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.3 Hand-written digits" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training data : 1347\n", "Test data : 450\n", "Input dimension : 64\n" ] } ], "source": [ "digits = datasets.load_digits()\n", "dat_train, dat_test, label_train, label_test = train_test_split(digits.data, digits.target, test_size=0.25)\n", "\n", "print(f\"Training data : {len(dat_train)}\")\n", "print(f\"Test data : {len(dat_test)}\")\n", "print(f\"Input dimension : {dat_train.shape[1]}\")" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def show_result_digit(clf):\n", " label_test_pred = clf.predict(dat_test)\n", " print(f\"train accuracy score: {accuracy_score(label_train, clf.predict(dat_train))}\")\n", " print(f\"test accuracy score: {accuracy_score(label_test, label_test_pred)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, we try out $L=2$ neural network. We have to choose $n_0=64, n_2 = 10$. The choice of $n_1$ is arbitrary, but here we try $n_1 = 35$." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "success : True\n", "nit : 2205\n", "func value : 0.052883086715949365\n", "calcualtion time : 11.59360384941101seconds\n", "train accuracy score: 0.9985152190051967\n", "test accuracy score: 0.98\n" ] } ], "source": [ "num_neurons = np.array([64,35,10])\n", "nn_mnist = NeuralNet(C=10, lam=0.5, num_neurons=num_neurons)\n", "nn_mnist.fit(dat_train, label_train, show_message=True, ep=0.01, alpha=0.1, ftol=1e-5)\n", "show_result_digit(nn_mnist)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.6" } }, "nbformat": 4, "nbformat_minor": 2 }