{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## 分类" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### 逻辑回归" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "Logistic回归与多重线性回归实际上有很多相同之处，最大的区别就在于它们的因变量不同，其他的基本都差不多。正是因为如此，这两种回归可以归于同一个家族，即广义线性模型（generalizedlinear model）。\n", "\n", "这一家族中的模型形式基本上都差不多，不同的就是因变量不同。\n", "\n", "- 如果是连续的，就是多重线性回归；\n", "- 如果是二项分布，就是Logistic回归；\n", "- 如果是Poisson分布，就是Poisson回归；\n", "- 如果是负二项分布，就是负二项回归。\n", "\n", "Logistic回归的因变量可以是二分类的，也可以是多分类的，但是二分类的更为常用，也更加容易解释。所以实际中最常用的就是二分类的Logistic回归。\n", "\n", "Logistic回归的主要用途：\n", "\n", "- 寻找危险因素：寻找某一疾病的危险因素等；\n", "- 预测：根据模型，预测在不同的自变量情况下，发生某病或某种情况的概率有多大；\n", "- 判别：实际上跟预测有些类似，也是根据模型，判断某人属于某病或属于某种情况的概率有多大，也就是看一下这个人有多大的可能性是属于某病。\n", "\n", "Logistic回归主要在流行病学中应用较多，比较常用的情形是探索某疾病的危险因素，根据危险因素预测某疾病发生的概率，等等。\n", "\n", "例如，想探讨胃癌发生的危险因素，可以选择两组人群，一组是胃癌组，一组是非胃癌组，两组人群肯定有不同的体征和生活方式等。\n", "\n", "这里的因变量就是是否胃癌，即“是”或“否”，自变量就可以包括很多了，例如年龄、性别、饮食习惯、幽门螺杆菌感染等。\n", "\n", "自变量既可以是连续的，也可以是分类的。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "#### 基本原理\n", "Logistic Regression和Linear Regression的原理是相似的，按照我自己的理解，可以简单的描述为这样的过程：\n", "\n", "（1）找一个`玄学的预测函数`（Andrew Ng的公开课中称为hypothesis），一般表示为h函数，该函数就是我们需要找的分类函数，它用来预测输入数据的判断结果。这个过程时非常关键的，需要对数据有一定的了解或分析，知道或者猜测预测函数的“大概”形式，比如是线性函数还是非线性函数。\n", "\n", "（2）`构造损失函数`（也称Cost函数），该函数表示预测的输出（h）与训练数据类别（y）之间的偏差，可以是二者之间的差（h-y）或者是其他的形式。综合考虑所有训练数据的“损失”，将Cost求和或者求平均，记为`J(θ)函数`，表示所有训练数据预测值与实际类别的偏差。\n", "\n", "（3）显然，J(θ)函数的值越小表示预测函数越准确（即h函数越准确），所以这一步需要做的是`找到J(θ)函数的最小值`。找函数的最小值有不同的方法，Logistic Regression实现时有的是梯度下降法（Gradient Descent）。\n", "\n", "- 流程：玄学预测函数h --> 构造损失函数J(θ) --> 找到J(θ)函数的最小值(梯度下降法)" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "#### 具体过程" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### 构造预测函数\n", "\n", "Logistic Regression虽然名字里带“回归”，但是它实际上是一种分类方法，用于两分类问题（即输出只有两种）。\n", "\n", "根据第二章中的步骤，需要先找到一个预测函数（h），显然，该函数的输出必须是两个值（分别代表两个类别），所以利用了Logistic函数（或称为Sigmoid函数），函数形式为：\n", "$$ g(z)=\\frac{1}{1+e^{-z}} $$\n", "\n", "![sigmoid(x)](sigmoid.jpg)\n", "\n", "接下来需要确定数据划分的边界类型，对于图2和图3中的两种数据分布，显然图2需要一个线性的边界，而图3需要一个非线性的边界。\n", "\n", "接下来我们只讨论线性边界的情况。\n", "\n", "![线性边界](decisionBoundary.jpg)\n", "\n", "![非线性边界](Non-DecisionBoundary.jpg)\n", "\n", "对于线性边界的情况，边界形式如下：\n", "\n", "$$ \\theta_0+\\theta_1 x_1+...+\\theta_n x_n=\\sum_{i=0}^{n}\\theta_i x_i=\\theta^T x$$\n", "\n", "构造预测函数为：\n", "\n", "$$ h_{\\theta}(x)=g(\\theta^T x)=\\frac{1}{1+e^{-\\theta^T x}} $$\n", "\n", "$ h_θ(x) $函数的值有特殊的含义，它表示结果取1的概率，因此对于输入x分类结果为类别1和类别0的概率分别为：\n", "\n", "$$ P(y=1|x;\\theta)=h_{\\theta}(x) $$\n", "$$ P(y=0|x;\\theta)=1-h_{\\theta}(x) $$\n", "\n", "##### 构造Cost函数\n", "\n", "Andrew Ng在课程中直接给出了Cost函数及J(θ)函数如式（5）和（6），但是并没有给出具体的解释，只是说明了这个函数来衡量h函数预测的好坏是合理的。\n", "\n", "$$ Cost(h_{\\theta}(x), y)=\n", "\\left\\{ \n", "\\begin{array}{ll}\n", "-log(h_\\theta(x))&, y=1\\\\\n", "-log(1-h_\\theta(x))&,y=0\n", "\\end{array}\n", "\\right. \n", "$$\n", "$$J(\\theta)=\\frac{1}{m}\\sum_{i=1}^{m}Cost(h_{\\theta}(x^{(i)}, y^{(i)})) $$\n", "\n", "实际上这里的Cost函数和J(θ)函数是基于`最大似然估计`推导得到的。下面详细说明推导的过程。（4）式综合起来可以写成：\n", "\n", "$$ P(y|x;\\theta)=(h_{\\theta}(x))^y (1-h_{\\theta}(x))^{1-y} $$\n", "\n", "取似然函数为：\n", "\n", "$$ L(\\theta)=\\prod_{i=1}^{m}P(y^{i}|x^{(i)};\\theta)=\\prod_{i=1}^{m}(h_{\\theta}(x))^y (1-h_{\\theta}(x))^{1-y}$$\n", "\n", "对数似然函数为：\n", "\n", "$$ l(\\theta)=logL(\\theta)=\\sum_{i=1}^{m}(y^{(i)}logh_{\\theta}(x^{(i)})+(1-y^{(i)})log(1-h_{\\theta}(x^{(i)}))) $$\n", "\n", "最大似然估计就是要求得使l(θ)取最大值时的θ，其实这里可以使用梯度上升法求解，求得的θ就是要求的最佳参数。但是，在Andrew Ng的课程中将J(θ)取为（6）式，即：\n", "\n", "$$ J(\\theta)=-\\frac{1}{m}l(\\theta) $$\n", "\n", "因为乘了一个负的系数-1/m，所以J(θ)取最小值时的θ为要求的最佳参数。\n", "\n", "##### 梯度下降法求J(θ)的最小值\n", "\n", "求J(θ)的最小值可以使用梯度下降法，根据梯度下降法可得θ的更新过程：\n", "\n", "$$ \\theta_j := \\theta_j-\\alpha \\frac{\\partial}{\\partial \\theta_j}J(\\theta),(j=0...n) $$\n", "\n", "式中为α学习步长，下面来求偏导：\n", "\n", "$$\n", "\\begin{align}\n", "\\frac{\\partial}{\\partial \\theta_j} J(\\theta) \n", "&=-\\frac{1}{m} \\sum_{i=1}^{m} (y^{(i)}\\frac{1}{h_{\\theta}(x^{(i)})}\\frac{\\partial}{\\partial\\theta_j}h_{\\theta}(x^{(i)})-(１-y^{(i)})\\frac{1}{1-h_{\\theta}(x^{(i)})}\\frac{\\partial}{\\partial\\theta_j}h_{\\theta}(x^{(i)}) \\\\\n", "&=-\\frac{1}{m}\\sum_{i=1}^{m}(y^{(i)}\\frac{1}{g(\\theta^T x^{(i)})}-(１-y^{(i)})\\frac{1}{1-g(\\theta^T x^{(i)})})\\frac{\\partial}{\\partial\\theta_j}g(\\theta^T x^{(i)}) \\\\\n", "&=-\\frac{1}{m}\\sum_{i=1}^{m}(y^{(i)}\\frac{1}{g(\\theta^T x^{(i)})}-(１-y^{(i)})\\frac{1}{1-g(\\theta^T x^{(i)})})g(\\theta^T x^{(i)})(1-g(\\theta^T x^{(i)}))\\frac{\\partial}{\\partial\\theta_j}\\theta^T x^{(i)} \\\\\n", "&=-\\frac{1}{m}\\sum_{i=1}^{m}(y^{(i)}(1-g(\\theta^T x^{(i)}))-(１-y^{(i)})g(\\theta^T x^{(i)}))x_{j}^{(i)} \\\\\n", "&=-\\frac{1}{m}\\sum_{i=1}^{m}(y^{(i)}-g(\\theta^T x^{(i)}))x_{j}^{(i)} \\\\\n", "&=-\\frac{1}{m}\\sum_{i=1}^{m}(y^{(i)}-h_{\\theta}(x^{(i)}))x_{j}^{(i)} \\\\\n", "&=\\frac{1}{m}\\sum_{i=1}^{m}(h_{\\theta}(x^{(i)})-y^{(i)})x_{j}^{(i)} \n", "\\end{align}\n", "$$\n", "\n", "上式求解过程中用到如下的公式：\n", "\n", "$$ f(x)=\\frac{1}{1+e^{g(x)}} $$\n", "$$\n", "\\begin{align}\n", "\\frac{\\partial}{\\partial x}f(x)\n", "&=\\frac{1}{(1+e^{g(x)})^2}e^{g(x)} \\frac{\\partial}{\\partial x}g(x) \\\\\n", "&=\\frac{1}{1+e^{g(x)}}e^{g(x)} \\frac{e^{g(x)}}{1+e^{g(x)}}e^{g(x)} \\frac{\\partial}{\\partial x}g(x) \\\\\n", "&=f(x)(1-f(x)) \\frac{\\partial}{\\partial x}g(x) \n", "\\end{align}\n", "$$\n", "\n", "因此，（11）式的更新过程可以写成：\n", "\n", "$$ \\theta_j := \\theta_j-\\alpha \\frac{1}{m}\\sum_{i=1}^{m}(h_{\\theta}(x^{(i)})-y^{(i)})x_{j}^{(i)},(j=0...n) $$\n", "\n", "因为式中α本来为一常量，所以 $ \\frac{1}{m} $ 一般将省略，所以最终的θ更新过程为：\n", "\n", "$$ \\theta_j := \\theta_j-\\alpha \\sum_{i=1}^{m} ( h_{\\theta}(x^{(i)}) -y^{(i)} ) x_{j}^{(i)} , (j=0...n) $$\n", "\n", "另外，补充一下，3.2节中提到求得l(θ)取最大值时的θ也是一样的，用梯度上升法求（9）式的最大值，可得：\n", "\n", "$$\n", "\\begin{align}\n", "\\theta_j & := \\theta_j+\\alpha \\frac{\\partial}{\\partial \\theta_j}l(\\theta)\\\\\n", "&=\\theta_j+\\alpha \\sum_{i=1}^{m} ( y^{(i)} - h_{\\theta}(x^{(i)}) ) x_{j}^{(i)},(j=0...n) \n", "\\end{align}\n", "$$\n", "\n", "观察上式发现跟（14）是一样的，所以，采用梯度上升发和梯度下降法是完全一样的，这也是《机器学习实战》中采用梯度上升法的原因。\n", "\n", "##### 梯度下降过程向量化\n", "\n", "关于θ更新过程的vectorization，Andrew Ng的课程中只是一带而过，没有具体的讲解。\n", "\n", "《机器学习实战》连Cost函数及求梯度等都没有说明，所以更不可能说明vectorization了。\n", "\n", "但是，其中给出的实现代码确是实现了vectorization的，图4所示代码的32行中weights（也就是θ）的更新只用了一行代码，直接通过矩阵或者向量计算更新，没有用for循环，说明确实实现了vectorization，具体代码下一章分析。\n", "\n", "文献[3]中也提到了vectorization，但是也是比较粗略，很简单的给出vectorization的结果为：\n", "\n", "$$ \\theta_j := \\theta_j-\\alpha \\sum_{i=1}^{m} ( h_{\\theta}(x^{(i)}) -y^{(i)} ) x_{j}^{(i)} , (j=0...n) $$\n", "\n", "且不论该更新公式正确与否，这里的Σ(...)是一个求和的过程，显然需要一个for语句循环m次，所以根本没有完全的实现vectorization，不像《机器学习实战》的代码中一条语句就可以完成θ的更新。\n", "\n", "下面说明一下我理解《机器学习实战》中代码实现的vectorization过程。\n", "\n", "约定训练数据的矩阵形式如下，x的每一行为一条训练样本，而每一列为不同的特称取值：\n", "\n", "$$ x=\n", "\\begin{bmatrix}\n", "x^{(1)} \\\\\n", "x^{(2)} \\\\\n", "... \\\\\n", "x^{(m)} \n", "\\end{bmatrix}\n", "=\n", "\\begin{bmatrix}\n", "x_0^{(1)} & x_1^{(1)} & ... & x_n^{(1)}\\\\\n", "x_0^{(2)} & x_1^{(2)} & ... & x_n^{(2)}\\\\\n", "... & ... & ... & ... \\\\\n", "x_0^{(m)} & x_1^{(m)} & ... & x_n^{(m)}\\\\\n", "\\end{bmatrix},\n", "y=\n", "\\begin{bmatrix}\n", "y^{(1)} \\\\\n", "y^{(2)} \\\\\n", "... \\\\\n", "y^{(m)} \n", "\\end{bmatrix}\n", "$$\n", "\n", "约定待求的参数θ的矩阵形式为：\n", "\n", "$$ \\theta=\n", "\\begin{bmatrix}\n", "\\theta_0 \\\\\n", "\\theta_1 \\\\\n", "... \\\\\n", "\\theta_n \n", "\\end{bmatrix}\n", "$$ \n", "\n", "先求x.θ并记为A：\n", "\n", "$$ A=x\\cdot\\theta =\n", "\\begin{bmatrix}\n", "x_0^{(1)} & x_1^{(1)} & ... & x_n^{(1)}\\\\\n", "x_0^{(2)} & x_1^{(2)} & ... & x_n^{(2)}\\\\\n", "... & ... & ... & ... \\\\\n", "x_0^{(m)} & x_1^{(m)} & ... & x_n^{(m)}\\\\\n", "\\end{bmatrix}\n", "\\cdot\n", "\\begin{bmatrix}\n", "\\theta_0 \\\\\n", "\\theta_1 \\\\\n", "... \\\\\n", "\\theta_n \n", "\\end{bmatrix}\n", "=\n", "\\begin{bmatrix}\n", "\\theta_0 x_0^{(1)} + \\theta_1 x_1^{(1)} + ... + \\theta_n x_n^{(1)}\\\\\n", "\\theta_0 x_0^{(2)} + \\theta_1 x_1^{(2)} + ... + \\theta_n x_n^{(2)}\\\\\n", "... \\\\\n", "\\theta_0 x_0^{(m)} + \\theta_1 x_1^{(m)} + ... + \\theta_n x_n^{(m)}\\\\\n", "\\end{bmatrix}\n", "$$\n", "求hθ(x)-y并记为E：\n", "$$ E=h_{\\theta}(x)-y= \n", "\\begin{bmatrix}\n", "g(A^{(1)})-y^{(1)} \\\\\n", "g(A^{(2)})-y^{(2)} \\\\\n", "... \\\\\n", "g(A^{(m)})-y^{(m)} \n", "\\end{bmatrix}\n", "=\n", "\\begin{bmatrix}\n", "e^{(1)} \\\\\n", "e^{(2)} \\\\\n", "... \\\\\n", "e^{(m)} \n", "\\end{bmatrix}\n", "=g(A)-y$$\n", "\n", "g(A)的参数A为一列向量，所以实现g函数时要支持列向量作为参数，并返回列向量。由上式可知hθ(x)-y可以由g(A)-y一次计算求得。\n", "\n", "再来看一下（15）式的θ更新过程，当j=0时：\n", "\n", "$$ \n", "\\begin{align}\n", "\\theta_0 \n", "&:=\\theta_0 - \\alpha \\sum_{i=1}^{m} ( h_{\\theta}(x^{(i)}) -y^{(i)} ) x_{0}^{(i)} \\\\\n", "&=\\theta_0 - \\alpha \\sum_{i=1}^{m} e^{(i)} x_{0}^{(i)} \\\\\n", "&=\\theta_0 - \\alpha \\cdot (x_0^{(1)}, x_0^{(2)}, ... , x_0^{(m)}) \\cdot E \n", "\\end{align}\n", "$$\n", "\n", "同样的可以写出θj，\n", "\n", "$$ \\theta_j :=\\theta_j - \\alpha \\cdot (x_j^{(1)}, x_j^{(2)}, ... , x_j^{(m)}) \\cdot E $$\n", "\n", "综合起来就是：\n", "\n", "$$ \n", "\\begin{bmatrix}\n", "\\theta_0 \\\\\n", "\\theta_1 \\\\\n", "... \\\\\n", "\\theta_n \n", "\\end{bmatrix} :=\n", "\\begin{bmatrix}\n", "\\theta_0 \\\\\n", "\\theta_1 \\\\\n", "... \\\\\n", "\\theta_n \n", "\\end{bmatrix} - \\alpha \\cdot \n", "\\begin{bmatrix}\n", "x_0^{(1)} & x_0^{(2)} & ... & x_0^{(m)}\\\\\n", "x_1^{(1)} & x_1^{(2)} & ... & x_1^{(m)}\\\\\n", "... & ... & ... & ... \\\\\n", "x_n^{(1)} & x_n^{(2)} & ... & x_n^{(m)}\\\\\n", "\\end{bmatrix} \n", "\\cdot E = \\theta-\\alpha \\cdot x^T \\cdot E $$\n", "\n", "综上所述，vectorization后θ更新的步骤如下：\n", "\n", "（1）求$ A=x\\cdot θ $；\n", "\n", "（2）求$ E=g(A)-y $；\n", "\n", "（3）求$ \\theta := \\theta-\\alpha \\cdot x^T \\cdot E $,x'表示矩阵x的转置。\n", "\n", "也可以综合起来写成：\n", "\n", "$$ \\theta := \\theta-\\alpha \\cdot (\\frac{1}{m})\\cdot x^T \\cdot (g(x\\cdot\\theta)-y) $$\n", "\n", "前面已经提到过：$\\frac{1}{m}$是可以省略的。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "hidden": true }, "outputs": [], "source": [ "# 不用scikit-learn库实现\n", "def sigmoid(x):\n", " return 1.0/(1+exp(-x))\n", "\n", "def gradAscent(dataMatIn, classLabels):\n", " dataMatrix = mat(dataMatIn)\n", " labelMatrix= mat(classLabels).transpose()\n", " m, n = shape(dataMatrix)\n", " alpha = 0.001\n", " maxCycles = 500\n", " weights = ones((n, 1))\n", " for k in range(maxCycles):\n", " h = sigmoid(dataMatrix*weights)\n", " error = (labelMatrix - h)\n", " weights = weights + alpha * dataMatrix.transpose() * error\n", " return weights" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "#### 正则化Regularization" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### 过拟合问题\n", "\n", "对于线性回归或逻辑回归的损失函数构成的模型，可能会有些权重很大，有些权重很小，导致过拟合（就是过分拟合了训练数据），使得模型的复杂度提高，泛化能力较差（对未知数据的预测能力）。\n", "\n", "下面左图即为欠拟合，中图为合适的拟合，右图为过拟合。\n", "\n", "![overfitting](overfitting.png)\n", "\n", "##### 问题的主因\n", "\n", "过拟合问题往往源自过多的特征。\n", "\n", "##### 解决方法\n", "\n", "1）减少特征数量（减少特征会失去一些信息，即使特征选的很好）\n", "\n", "- 可用人工选择要保留的特征；\n", "\n", "- 模型选择算法；\n", "\n", "2）正则化（特征较多时比较有效）\n", "\n", "- 保留所有特征，但减少θ的大小\n", "\n", "##### 正则化方法\n", "\n", "正则化是结构风险最小化策略的实现，是在经验风险上加一个正则化项或惩罚项。\n", "\n", "正则化项一般是模型复杂度的单调递增函数，模型越复杂，正则化项就越大。\n", "\n", "从房价预测问题开始，这次采用的是多项式回归。左图是适当拟合，右图是过拟合。\n", "\n", "![fitting](fitting.png)\n", "\n", "直观来看，如果我们想解决这个例子中的过拟合问题，最好能将$ x^3,x^4 $的影响消除，也就是让$ \\theta_3\\to 0, \\theta_4\\to 0 $。假设我们对$ \\theta_3,\\theta_4 $进行惩罚，并且令其很小，一个简单的办法就是给原有的Cost函数加上两个略大惩罚项，例如：\n", "\n", "$$ \\min_{\\theta}^{ } \\frac{1}{2m} \\sum_{i=1}^{n} (h_{\\theta}(x_i) - y_i)^2 + 1000 \\theta_3 ^2 + 1000 \\theta_4 ^2 ) $$\n", "\n", "这样在最小化Cost函数的时候，$ \\theta_3\\to 0,\\theta_4\\to 0 $\n", "\n", "正则项可以取不同的形式，在回归问题中取平方损失，就是参数的L2范数，也可以取L1范数。取平方损失时，模型的损失函数变为：\n", "\n", "$$ J(\\theta) = \\frac{1}{2m} \\sum_{i=1}^{n} (h_{\\theta}(x_i) - y_i)^2 + \\lambda \\sum_{j=1}^{n} \\theta_j^2 $$\n", "\n", "lambda是正则项系数：\n", "\n", "- 如果它的值很大，说明对模型的复杂度惩罚大，对拟合数据的损失惩罚小，这样它就不会过分拟合数据，在训练数据上的偏差较大，在未知数据上的方差较小，但是可能出现欠拟合的现象；\n", "\n", "- 如果它的值很小，说明比较注重对训练数据的拟合，在训练数据上的偏差会小，但是可能会导致过拟合。\n", "\n", "正则化后的梯度下降算法θ的更新变为：\n", "\n", "$$ \\theta_j := \\theta_j \\frac{\\alpha}{m} \\sum_{i=1}^{n} (h_{\\theta} (x_i) - y_i) x_{i}^{j} - \\frac{\\lambda}{m} \\theta_j $$\n", "\n", "正则化后的线性回归的Normal Equation的公式为：\n", "\n", "$$ \\theta=(X^T X+\\lambda \n", "\\begin{bmatrix}\n", "0 & & & \\\\\n", "& 1 & & \\\\\n", "& & ... & \\\\\n", "& & & 1 \\\\\n", "\\end{bmatrix})^{-1} X_T Y $$\n", "\n", "##### 其他优化算法\n", "\n", "- Conjugate gradient method(共轭梯度法)\n", "\n", "- Quasi-Newton method(拟牛顿法)\n", "\n", "- BFGS method\n", "\n", "- L-BFGS(Limited-memory BFGS)\n", "\n", "后二者由拟牛顿法引申出来，与梯度下降算法相比，这些算法的优点是：\n", "\n", "- 第一，不需要手动的选择步长；\n", "\n", "- 第二，通常比梯度下降算法快；\n", "\n", "但是缺点是更复杂。\n", "\n", "#### 多类分类问题\n", "\n", "对于多类分类问题，可以将其看做成二类分类问题：保留其中的一类，剩下的作为另一类。\n", "\n", "对于每一个类 i 训练一个逻辑回归模型的分类器$ h_{\\theta}^{(i)}(x) $，并且预测y = i时的概率；对于一个新的输入变量x, 分别对每一个类进行预测，取概率最大的那个类作为分类结果：$$ \\max_{i} h_{\\theta}^{(i)}(x) $$" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "C=100.00\n", "Sparsity with L1 penalty: 6.25%\n", "score with L1 penalty: 0.9098\n", "Sparsity with L2 penalty: 4.69%\n", "score with L2 penalty: 0.9098\n", "C=1.00\n", "Sparsity with L1 penalty: 9.38%\n", "score with L1 penalty: 0.9093\n", "Sparsity with L2 penalty: 4.69%\n", "score with L2 penalty: 0.9093\n", "C=0.01\n", "Sparsity with L1 penalty: 85.94%\n", "score with L1 penalty: 0.8609\n", "Sparsity with L2 penalty: 4.69%\n", "score with L2 penalty: 0.8915\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQsAAAD7CAYAAAB9sLH/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFPhJREFUeJzt3XuQVOWZBvDnocHhEkBgULnIjFwMIkEMJJhixaAQ0KqV\nkETKpLISXaWMm8hGsSpaYU1FyssfKbesbOWikABZ1w0SYypAFEXCRSEwXMUFQRGDCCJEGBguM/Du\nH3PYbZnu97xNDww0z69qypnznO/0Geubh9Pdp8+hmUFEJE2zpt4BETk3qCxEJERlISIhKgsRCVFZ\niEiIykJEQs67siD5G5JTmno/RBrb6Z7bBZcFyfdIjsix/AKSzye5kfxyo+zhaUTyyyS3N/V+yNnB\nmdvXkJxPci/J3SRnkezSFPsYdTrmdmMfWSwB8G0AOxt5uyJNqQOAXwGoBFABoBrAr5tyh5pCo5WF\nmR01s383syUAjqWtT3IhycdI/pXkfpIvkuyYlV9D8nWSn5Bcm32kkox9hORSktUkXyZZnpXPIrmT\n5D6Si0hemePx2wCYB6AryQPJV1eSNSQ7Za33+eRfkxan/n9HzmVmNs/MZpnZfjOrAfAzAEPzrV+q\nc7upX7O4DcAdALoAqAPwFACQ7AZgDoApADoCmARgNsnOWWO/BeB2ABcBuCBZ54R5APok2SoA/3ny\nA5vZQQA3AthhZp9JvnYAWAhgXNaq/wTgOTOrLfaXlZIxDMCGlHVKbm43dVnMNLM3k19uMoBxJDOo\nfyoz18zmmtlxM5sPYCWAm7LG/trM3jazQwB+B2DgicDMpplZtZkdAfBjAFeRbB/cp+nJ4yPZl28C\nmFncrymlguQAAP8G4IGUVUtubjd1Wfwt6/ttAFoAKEf988JbksO0T0h+AuAfUN/SJ2S/LlID4DNA\n/f8Eko+TfIfkfgDvJeuUI+ZFAP1IXgZgJIB9ZvbXAn8vKUEke6P+X/aJZrY4ZfWSm9vNoyueJpdm\nfd8DQC2Aj1H/P3qmmd11Ctv8FoAxAEag/n9mewB/B8Ac6zb4yK2ZHSb5O9Q3cF/oqEIAkKwA8AqA\nR8wsMidKbm6f6pFFC5Its76aAwDJMpItk3UuSLJcv8gJ3ybZj2RrAD8B8LyZHQPwWwD/SHJU0qYt\nk7eCugf2rS2AIwD2AGgN4FFn3V0AOuU4jJsB4DsAbobK4nzTYG4nrzMsAPAzM/tFcDslN7dPtSzm\nAjiU9fXjZPmm5OduAF5Kvq9wtjMTwG9Qf9jVEsC9AGBmf0N9gz4EYDfq2/iB4P7OQP1h3wcA3gKw\nLN+KZrYRwH8BeDc5JOyaLF8K4DiAVWa2LfCYUjpyze07AfQE8OOsdxcOpGyn5OY2m+riNyQXAvit\nmT3TJDuQguQCAM+erfsnZ69SndtN/ZrFWYnkFwB8HvX/AoiUjGLmdlO/G3LWITkd9S9k/auZVTf1\n/og0lmLndpM9DRGRc4uOLEQkRGUhIiEFvcBZXl5ulZWVefMDB/x3kzKZjJuXlZW5+b59+9y8utp/\nGta6dWs3T9u/d99992Mz6+yuJOectHmdNu8uuOACN0+b15988ombp/1dpc3rtP3bvHlzaF4XVBaV\nlZVYuXJl3nzJkiXu+LZt27p5nz593HzOnDluvmjRIje/6qqr3Lx9e/8U+3HjxumcixKUNq//9Kc/\nueN79Ojh5r1793bzF1980c2XLl3q5mnzuqLCO9UJGDVqVGhe62mIiISoLEQkRGUhIiEqCxEJUVmI\nSIjKQkRCCnrrtKamBlVVVXnzp59+2h1fW+tf6m/s2LFufsUVV7h52nkWI0eOdPMLL7zQzaU0HTx4\nECtWrMibT5s2zR3fvLn/Z/T1r3/dzfv37+/mR48edfPrr7/ezdPO84jSkYWIhKgsRCREZSEiISoL\nEQlRWYhIiMpCREJUFiISUtB5Fi1btsTll1+eNx81apQ7fvLkyW7++OOPu3naR23TPuq7YMECN+/V\nq5ebS2lKm9c33XRT3gwApkyZ4uY//elP3TztI+7PPvusm7/66qtu3rNnTzeP0pGFiISoLEQkRGUh\nIiEqC8mJ5CUkn0vu2F1Fci7J/E/sY9scRnIVyTqS3zgpG09yc/I1Pmv5IJLrSW4h+VSue+ey3lPJ\nOutIfj5tu1I4lYU0kPxBvgBgoZn1MrNBAB4EcHGRm34f9Tfl/dQrdiQ7AngYwBAAXwTwMMkOSfxz\nAHcB6JN8jc6x3Ruz8gnJmLTtSoFUFpLLcAC12XcMN7O1Zra4mI2a2Xtmtg71N+bNNgrAfDPba2Z/\nBzAfwGiSXQC0M7NlVn83rBkAvppj02MAzLB6ywBcmIzNud1ifofzme51Krn0B5D/WgRZSC4GkOuy\n7ZPM7JXg43VD/d3ET9ieLOuWfH/y8kLG51oup6CgsqiursbChQvz5mvWrHHHz5o1y83TzqN44403\n3DztPI8777zTzf/yl7+4uTRkZtc29T4U69ChQ1i/fn3efN26de74559/3s3TzqNYvNg/YLvuuuvc\n/N5773XzrVu3unmUnoZILhsADIqsSHIxyTU5vkYU8HgfALg06+fuybIPku9PXl7I+FzL5RSoLCSX\nBQDKSE44sYDkAJINjiLM7FozG5jjK/oUBABeAvAVkh2SFyC/AuAlM/sQwH6S1yQvut4GINdpun8E\ncFvyrsg1APYlY3Nut4D9kiwqC2kgeTFxLIARyVunGwA8BmBnMdsl+QWS2wHcAuCXyXZhZnsBPAJg\nRfL1k2QZANwD4BkAWwC8A2Besq27Sd6drDMXwLvJOk8nY9K2KwXSC5ySk5ntADCukbe5Ap9+WpGd\nTQPQ4GKXZrYS9S+4nrw8+50aA/AvhWxXCqcjCxEJUVmISIjKQkRCWP90L2bw4MHm3Zq+WTO/e3bt\n2uXmF110kZvfd999bn7//fe7+e7du9087db1JKvMbLC7kpxz0uZ1q1at3PEHDx5080wm4+a33367\nm6dd5+Wjjz5y87T7kkTntY4sRCREZSEiISoLEQlRWYhIiMpCREJUFiISorIQkZCCPhuyZ88eTJ8+\nPW9++PBhd/y8efPcfNGiRW7+wAMPuLl37wcg/b4gadfLkNJ06NAhvPnmm3nztPMoZsyY4eZ/+MMf\n3HzixIlFje/ataubF3IulUdHFiISorIQkRCVhYiEqCxEJERlISIhKgsRCVFZiEhIQedZdOrUCePH\n579d5P79+93xffr0cfO08xyWL1/u5k8++aSbV1X598357ne/6+ZSurxzEWpqatyxQ4cOdfO0eb1t\n2zY3v/rqq9389ddfd/O0++lE6chCREJUFiISorIQkRCVhYiEqCxEJERlISIhKgsRCSnoPIvjx4+7\nn+3//e9/744fNmyYm6ddjyJt/Pvvv+/mAwYMcPMxY8a4uZSmFi1aoEuXLnnzadP8W6WOHDnSzbt1\n6+bmafPau9YGAFx55ZVunnZfkigdWYhIiMpCREJUFiISorIQkRCVhYiEqCxEJERlISIhLOSeAiR3\nA/A/fF/aKsysc1PvhDQuzevYvC6oLETk/KWnISISorIQkRCVhYiEqCxEJERlISIhKgsRCVFZiEiI\nykJEQlQWIhKishCREJWFiIQUdMHe8vJyq6yszJsfPny4qJ0pKytz87QbL1dXVxe1/datW7v5xo0b\nP9YHyUpP2rw+cOCAOz6Tybh52rzbt2+fm6c9fqtWrdy8ZcuWbr5p06bQvC6oLCorK7Fy5cq8+caN\nG93xzZr5BzKXXXaZm//5z3928yVLlri5NyEAYNCgQW4+ZMiQ8/mTiSUrbV4vWrTIHd+xY0c379mz\np5vPmTPHzdPuwn7FFVe4ed++fd182LBhoXmtpyEiEqKyEJEQlYWIhKgsRCREZSEiISoLEQkp6K3T\nQ4cOYd26dXnzmTNnuuP37Nnj5l/72tfcvE+fPm6edp7H9ddf7+bHjh1zcylNNTU1WLVqVd586tSp\n7vi0eTN27Fg3/+xnP+vmtbW1bn7ddde5edp5IFE6shCREJWFiISoLEQkRGUhIiEqCxEJUVmISIjK\nQkRCCjrPoqysDL169cqbDx8+3B1///33u/mUKVPcvFu3bm4+f/58N1+4cKGb9+jRw82lNKXN61Gj\nRrnjH3nkETd/9NFH3byiosLNZ8+e7eavvfaam6ddmiFKRxYiEqKyEJEQlYWIhKgsJCeSl5B8juQ7\nJKtIziV5eZHbHEZyFck6kt9w1htEcj3JLSSfIslkeUeS80luTv7boZj9kcKoLKSB5I/zBQALzayX\nmQ0C8CCAi4vc9PsAvgPg2ZT1fg7gLgB9kq/RyfIfAnjVzPoAeDX5Wc4QlYXkMhxArZn94sQCM1tr\nZouL2aiZvWdm6wAcz7cOyS4A2pnZMjMzADMAfDWJxwCYnnw/PWu5nAEFvXUq543+AKoiK5JcDKBt\njmiSmb1yCo/dDcD2rJ+3J8sA4GIz+zD5fieKP9KRAhRUFkeOHMHWrVvz5t41AQBg1qxZbp52HsXa\ntWvdfMyYMW5+6623uvmHH37o5tKQmV3bRI9rJK0xtnX48GG8/fbbefPVq1e746dPn+7maedRLFu2\nzM1Hjx7t5hMmTHDzXbt2uXmUjiwklw0A8r4Ame00HFl8AKB71s/dk2UAsItkFzP7MHm68tEpbF9O\nkV6zkFwWACgj+X//ZJEcQLLBUYSZXWtmA3N8nUpRIHmasZ/kNckLrbcBeDGJ/whgfPL9+Kzlcgao\nLKSB5IXFsQBGJG+dbgDwGOpfJzhlJL9AcjuAWwD8MtnuiWxN1qr3AHgGwBYA7wCYlyx/HMBIkpsB\njEh+ljNET0MkJzPbAWBcI29zBT79FCM7G5j1/UrUv8h68jp7ANzQmPskcTqyEJEQlYWIhKgsRCSE\n9a9lxQwePNi8W9O3bt3aHb9+/Xo37927t5s/8cQTbn7XXXe5+c6d/utzabeuJ1llZoPdleSckzav\n27Vr547fsWOHm7dtm+ud5f/3gx/8wM0ffPBBN9+9e7eb9+vXz82j81pHFiISorIQkRCVhYiEqCxE\nJERlISIhKgsRCVFZiEhIQZ8Nqaurw969e/PmBw4ccMcvXbrUzTdu3Ojmae83p933o3Pnzm6eyWTc\nXErT4cOHsWnTprz5vn373PFp12l5+eWX3fyhhx5y88997nNu3rVrVzevra118ygdWYhIiMpCREJU\nFiISorIQkRCVhYiEqCxEJERlISIhBZ1n0bx5c3Ts2DFvXlNT445PO88h7TyMF154wc0ffvhhN3/r\nrbfc/IYbdHnH81GzZs3QqlWrvPnBgwfd8VdffbWbL1++3M29a2kA6edZpI0fNWqUm0fpyEJEQlQW\nIhKishCREJWFiISoLEQkRGUhIiEqCxEJadR7nS5atMjN+/dvcPvKT0l7v3rYsGFunnb/hr59+7r5\nhAkT3FxKUyaTQfv27fPms2fPdscPHTrUzSsqKtx8+PDhbr5582Y3T7svyN133+3mUTqyEJEQlYWI\nhKgsRCREZSEiISoLEQlRWYhIiMpCREJoZvGVyd0Atp2+3TnrVZiZf1EOOedoXsfmdUFlISLnLz0N\nEZEQlYWIhKgsRCREZSEiISoLEQlRWYhIiMpCREJUFiISorIQkRCVhYiEFHQNzvLycqusrDxNu1K8\ntHutZjIZNy8rK3Pzqqqqj/XZkNKTNq/TPhKRljdr5v+bfPToUTc/cuSIm6dp06aNm69evTo0rwsq\ni8rKytSbsDalNWvWuHnbtm3dvFevXm5O8nz+sFHJSpvXaX+sdXV1bp72x7p9+3Y337Rpk5unldGQ\nIUPcvE2bNqF5rachIhKishCREJWFiISoLEQkRGUhIiEqCxEJadR7nTa1gQMHNvUuyDmorq4Oe/fu\nzZvv2rXLHb9161Y3HzRokJunbT9tXh87dszNa2tr3TxKRxYiEqKyEJEQlYWIhKgsRCREZSEiISoL\nEQlRWYhIyBk9z+K1115z8+HDh7v5lClT3PxHP/pRwfskkslk3I+Rd+/e3R2/fPnyoh5/9uzZbj5x\n4kQ3r66udvO0j8hH6chCREJUFiISorIQkRCVhYiEqCwkJ5KXkHyO5Dskq0jOJXl5kdssI/nfJLeQ\nXE6yMs96o0luStb7Ydby7yXLjGR5MfsihVNZSAMkCeAFAAvNrJeZDQLwIICLi9z0PwP4u5n1BvAk\ngCdyPHYGwH8AuBFAPwDfJNkviZcCGAFAF05uAioLyWU4gFoz+8WJBWa21swWF7ndMQCmJ98/D+CG\npJiyfRHAFjN718yOAnguGQczW21m7xW5D3KKzuh5FmnnUcycOdPNdR7FGdMfQFVkRZKLAeS6x8Ik\nM3vlpGXdAPwNAMysjuQ+AJ0AfJxrncR2AP617It0/Phx954zmzdvdsen3Utn6tSpbj5p0iQ3T7vF\nRXm5/4ws7X45USV18Rs588zs2qbeBzkzVBaSywYA34isWOCRxQcALgWwnWRzAO0B7Mmzzgndk2XS\nxFQWkssCAI+SnGBmvwIAkgMAtD/5dYsCjyz+CGA8gDdQX0YLrOG9/1YA6EPyMtSXxK0AvnVqv4Y0\nJr3AKQ0kf8BjAYxI3jrdAOAxADuL3PRUAJ1IbgFwH4AfAgDJriTnJo9dB+B7AF4C8D8AfmdmG5L1\n7iW5HfVHG+tIPlPk/kgBdGQhOZnZDgDjGnmbhwHckuexbsr6eS6AuTnWewrAU425TxKnIwsRCVFZ\niEhIoz4N8W5bDwCDBw9280suuaQxd0ckJJPJoEOHDnnzpUuXuuNHjhzp5q1bt3bzdu3aufmXvvQl\nN2/RooWbN2/eOH/mOrIQkRCVhYiEqCxEJERlISIhKgsRCVFZiEiIykJEQhr1PIu08yjSpL1fLXI6\nmBnq6ury5jfffLM7vlOnTm5+zz33uLl3LQ0AOHjwYFF527a5PhRcOB1ZiEiIykJEQlQWIhKishCR\nEJWFiISoLEQkRGUhIiHn1GX17rjjDjefNm3aGdoTKSUk0axZ/n83L7300rwZANTW1rr5jh073Hzy\n5Mlunna/nLTrWaRdTyNKRxYiEqKyEJEQlYWIhKgsRCREZSEiISoLEQlRWYhIyDl1nsX3v//9pt4F\nKUFmhuPHj+fN086jyGQybl5eXu7mN954Y1Hj02zZsqWo8SfoyEJEQlQWIhKishCREJWFiISoLEQk\nRGUhIiEqCxEJoZnFVyZ3A9h2+nbnrFdhZp2beiekcWlex+Z1QWUhIucvPQ0RkRCVhYiEqCxEJERl\nISIhKgsRCVFZiEiIykJEQlQWIhKishCRkP8FOtfWGX9eDtUAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn import datasets\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "digits = datasets.load_digits()\n", "\n", "X, y = digits.data, digits.target\n", "X = StandardScaler().fit_transform(X)\n", "\n", "# classify small against large digits\n", "y = (y > 4).astype(np.int)\n", "\n", "\n", "# Set regularization parameter\n", "for i, C in enumerate((100, 1, 0.01)):\n", " # turn down tolerance for short training time\n", " clf_l1_LR = LogisticRegression(C=C, penalty='l1', tol=0.01)\n", " clf_l2_LR = LogisticRegression(C=C, penalty='l2', tol=0.01)\n", " clf_l1_LR.fit(X, y)\n", " clf_l2_LR.fit(X, y)\n", "\n", " coef_l1_LR = clf_l1_LR.coef_.ravel()\n", " coef_l2_LR = clf_l2_LR.coef_.ravel()\n", "\n", " # coef_l1_LR contains zeros due to the\n", " # L1 sparsity inducing norm\n", "\n", " sparsity_l1_LR = np.mean(coef_l1_LR == 0) * 100\n", " sparsity_l2_LR = np.mean(coef_l2_LR == 0) * 100\n", "\n", " print(\"C=%.2f\" % C)\n", " print(\"Sparsity with L1 penalty: %.2f%%\" % sparsity_l1_LR)\n", " print(\"score with L1 penalty: %.4f\" % clf_l1_LR.score(X, y))\n", " print(\"Sparsity with L2 penalty: %.2f%%\" % sparsity_l2_LR)\n", " print(\"score with L2 penalty: %.4f\" % clf_l2_LR.score(X, y))\n", "\n", " l1_plot = plt.subplot(3, 2, 2 * i + 1)\n", " l2_plot = plt.subplot(3, 2, 2 * (i + 1))\n", " if i == 0:\n", " l1_plot.set_title(\"L1 penalty\")\n", " l2_plot.set_title(\"L2 penalty\")\n", "\n", " l1_plot.imshow(np.abs(coef_l1_LR.reshape(8, 8)), interpolation='nearest',\n", " cmap='binary', vmax=1, vmin=0)\n", " l2_plot.imshow(np.abs(coef_l2_LR.reshape(8, 8)), interpolation='nearest',\n", " cmap='binary', vmax=1, vmin=0)\n", " plt.text(-8, 3, \"C = %.2f\" % C)\n", "\n", " l1_plot.set_xticks(())\n", " l1_plot.set_yticks(())\n", " l2_plot.set_xticks(())\n", " l2_plot.set_yticks(())\n", "\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "hidden": true }, "outputs": [ { "ename": "ValueError", "evalue": "Logistic Regression supports only liblinear, newton-cg, lbfgs and sag solvers, got saga", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 33\u001b[0m \u001b[0mmulti_class\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'multinomial'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 34\u001b[0m penalty='l1', solver='saga', tol=0.1)\n\u001b[0;32m---> 35\u001b[0;31m \u001b[0mclf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 36\u001b[0m \u001b[0msparsity\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcoef_\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m100\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 37\u001b[0m \u001b[0mscore\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscore\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/more-space/pyworks/venv/lib/python3.5/site-packages/sklearn/linear_model/logistic.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[1;32m 1177\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1178\u001b[0m _check_solver_option(self.solver, self.multi_class, self.penalty,\n\u001b[0;32m-> 1179\u001b[0;31m self.dual)\n\u001b[0m\u001b[1;32m 1180\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1181\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msolver\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'liblinear'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/more-space/pyworks/venv/lib/python3.5/site-packages/sklearn/linear_model/logistic.py\u001b[0m in \u001b[0;36m_check_solver_option\u001b[0;34m(solver, multi_class, penalty, dual)\u001b[0m\n\u001b[1;32m 425\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0msolver\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'liblinear'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'newton-cg'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'lbfgs'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'sag'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 426\u001b[0m raise ValueError(\"Logistic Regression supports only liblinear,\"\n\u001b[0;32m--> 427\u001b[0;31m \" newton-cg, lbfgs and sag solvers, got %s\" % solver)\n\u001b[0m\u001b[1;32m 428\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 429\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mmulti_class\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'multinomial'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'ovr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mValueError\u001b[0m: Logistic Regression supports only liblinear, newton-cg, lbfgs and sag solvers, got saga" ] } ], "source": [ "import time\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "from sklearn.datasets import fetch_mldata\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.utils import check_random_state\n", "\n", "# Turn down for faster convergence\n", "t0 = time.time()\n", "train_samples = 5000\n", "\n", "mnist = fetch_mldata('MNIST original')\n", "X = mnist.data.astype('float64')\n", "y = mnist.target\n", "random_state = check_random_state(0)\n", "permutation = random_state.permutation(X.shape[0])\n", "X = X[permutation]\n", "y = y[permutation]\n", "X = X.reshape((X.shape[0], -1))\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, train_size=train_samples, test_size=10000)\n", "\n", "scaler = StandardScaler()\n", "X_train = scaler.fit_transform(X_train)\n", "X_test = scaler.transform(X_test)\n", "\n", "# Turn up tolerance for faster convergence\n", "clf = LogisticRegression(C=50 / train_samples,\n", " multi_class='multinomial',\n", " penalty='l1', solver='saga', tol=0.1)\n", "clf.fit(X_train, y_train)\n", "sparsity = np.mean(clf.coef_ == 0) * 100\n", "score = clf.score(X_test, y_test)\n", "# print('Best C % .4f' % clf.C_)\n", "print(\"Sparsity with L1 penalty: %.2f%%\" % sparsity)\n", "print(\"Test score with L1 penalty: %.4f\" % score)\n", "\n", "coef = clf.coef_.copy()\n", "plt.figure(figsize=(10, 5))\n", "scale = np.abs(coef).max()\n", "for i in range(10):\n", " l1_plot = plt.subplot(2, 5, i + 1)\n", " l1_plot.imshow(coef[i].reshape(28, 28), interpolation='nearest',\n", " cmap=plt.cm.RdBu, vmin=-scale, vmax=scale)\n", " l1_plot.set_xticks(())\n", " l1_plot.set_yticks(())\n", " l1_plot.set_xlabel('Class %i' % i)\n", "plt.suptitle('Classification vector for...')\n", "\n", "run_time = time.time() - t0\n", "print('Example run in %.3f s' % run_time)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### 支持向量机SVM(support vector machine)" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "#### 感知机(Perceptron)\n", "\n", "感知机是一个非常简单的模型，在它的基础上发展出支持向量机（修改一下损失函数）和神经网络（经过简单的堆叠）\n", "\n", "我们先从感知机开始．\n", "\n", "为了便于理解，我们统一讨论二分类问题，并将两个类别的样本分别称为正｜负样本．\n", "\n", "当然，由二分类问题的推广之后会提到．" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### 感知机能做什么\n", "\n", "感知机能且一定能将线性可分的数据集合分开\n", "\n", "`线性可分`:\n", "- 在二维平面上，线性可分指存在一条线，将正负样本分开；\n", "- 在三维空间中，线性可分指存在一个平面，将正负样本分开；\n", "- 在高维空间$R^n$中，线性可分指存在一个n-1维的超平面，将正负样本分开\n", "\n", "感知机 在`线性可分`的数据集的表现\n", "\n", "![在线性可分的数据集的表现0](svm1.gif)\n", "![在线性可分的数据集的表现1](svm2.gif)\n", "\n", "\n", "感知机 在`线性不可分`的数据集的表现\n", "\n", "![在线性可分的数据集的表现0](svm3.gif)\n", "![在线性可分的数据集的表现1](svm4.gif)\n", "\n", "数学定义线性可分的概念如下：\n", "\n", "1.先定义超平面\n", "\n", "玄学地设有一个＂超平面＂\n", "\n", "$$ \\Pi : w\\cdot x +b=0 $$\n", "\n", "其中，$w$是n维行向量，$x$是n维列向量，b是常数，则$\\Pi$是$R^n$的超平面．\n", "\n", "对二维平面来说n=2，此时\n", "\n", "$$ w=\\begin{bmatrix}w_1 w_2\\end{bmatrix}, x=\\begin{bmatrix} x_1 \\\\ x_2 \\end{bmatrix} \\\\ \\Pi:w_1 x_1 + w_2 x_2 + b = 0$$\n", "\n", "这是平面坐标系中的直线的一般方程\n", "\n", "2.再定义线性可分\n", "\n", "对于一个数据集 $ D={(x_1,y_1),...,(x_N,y_N)} $ ($x_i$为输入，$y_i$为标签且只取$\\pm 1$)\n", "\n", "若存在一个超平面$\\Pi$能将$D$中的正负样本分开（对某个样本$(x_i,y_i)$，若$y_i=1$，则称其为正样本；若$y_i=-1$，则称其为负样本）\n", "\n", "则称$D$是线性可分的，否则就是线性不可分的．\n", "\n", "- 事实上，感知机模型只有$w,b$两个参数（x是输入），我们要做的是根据一定的算法，挖掘样本信息，逐步更新$w,b$，从而使对应的超平面$\\Pi$将$D$分开\n", "\n", "##### 感知机模型\n", "\n", "反正要更新$w,b$，不妨初始化分别设$w,b$为零向量，0\n", "\n", "广泛地，用梯度下降法来更新．梯度下降法包含两步：\n", "\n", "- 弄出一个损失函数，求损失函数的梯度（求导）\n", "- 让$w,b$往梯度反方向更新（梯度是函数增长最快的方向，要最小化损失函数，就是要让函数值减少得最快，即往梯度反方向更新）\n", "\n", "##### 损失函数\n", "\n", "注意到我们的感知机对应的超平面$ \\Pi : w\\cdot x +b=0 $，一个自然的想法就是：\n", "\n", "$$(x,y)是正样本\\Leftrightarrow w\\cdot x +b>0 \\Leftrightarrow y=1 \\Leftrightarrow (x,1)在\\Pi 的上方 $$\n", "$$(x,y)是负样本\\Leftrightarrow w\\cdot x +b<0 \\Leftrightarrow y=-1 \\Leftrightarrow (x,-1)在\\Pi 的下方$$\n", "\n", "设第i步更新的$w,b$分别为$w_i,b_i$，$w,b$最终稳定在$w_N,b_N$，则第i步的损失函数$L(x,y)$可以定义为：\n", "\n", "$$ 若w_i\\cdot x +b_i>0，则L(x,1)=0,\\,\\, L(x,-1)=w_i\\cdot x +b_i - (w_N\\cdot x + b_N) = w_i\\cdot x +b_i - 0 = w_i\\cdot x +b_i $$\n", "$$ 若w_i\\cdot x +b_i<0，则L(x,-1)=0,\\,\\, L(x,1)=-w_i\\cdot x +b_i - (w_N\\cdot x + b_N) = -w_i\\cdot x +b_i - 0 = -w_i\\cdot x +b_i $$\n", "\n", "为了照顾数学家的强迫症，设$　y=1　当 w_i\\cdot x +b_i>0;　y=-1　当 w_i\\cdot x +b_i<0 (即　y　和　w_i\\cdot x +b_i　同号)$\n", "\n", "就可以简化为：\n", "\n", "- 损失函数：$ L(x,y)=max(-y(w_i\\cdot x +b_i),0) $\n", "- (x,y)被正确分类$ \\Leftrightarrow y(w_i\\cdot x +b_i )>0 $\n", "\n", "从而易知：只有错分类的点才会给L(x,y)贡献梯度　(　分类正确的点的L(x,y)=0，梯度为0　)\n", "\n", "所以训练感知机时,我们只需选择使损失函数 $ L(x,y) $ 最大的一个样本 $ (x_i,y_i) $ ，用它来计算梯度，然后梯度下降即可\n", "\n", "有意思的是，如果所有的$(x_i,y_i)$都被正确分类了，则损失函数 $ L(x,y)==0 $，此时模型训练不动了，$w_i,b_i$不更新了\n", "\n", "##### 梯度下降\n", "\n", "- 求梯度\n", "\n", "$$ L(x_i,y_i)=-y_i(w_{i-1}\\cdot x_i+b_{i-1})=-w_{i-1}\\cdot x_i y_i - y_ib_{i-1} 　（y_i=\\pm 1）$$\n", "$$ \\frac{\\partial L}{\\partial w_{i-1}}=-x_i y_i \\,\\, , \\,\\, \\frac{\\partial L}{\\partial b_{i-1}}=-y_i $$\n", "\n", "- 更新$w_i,b_i$\n", "\n", "$$ w_i := w_{i-1}+x_i y_i $$\n", "$$ b_i := b_{i-1}+y_i $$\n", "\n", "下面给出感知机模型的一个实现并进行测试" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "###### 感知机的实现．测试．可视化" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "import numpy as np\n", "# 感知机模型\n", "class Perceptron:\n", " def __init__(self):\n", " self._w = self._b = None\n", " \n", " def fit(self, x, y, lr=0.01, epoch=1000):\n", " x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)\n", " self._w = np.zeros(x.shape[1])\n", " self._b = 0.\n", " # 训练感知机　**关键代码**\n", " for _ in range(epoch):\n", " # 计算 w·x+b\n", " y_pred = x.dot(self._w) + self._b\n", " # 选出使得损失函数最大的样本\n", " idx = np.argmax(np.maximum(0, -y_pred * y))\n", " # 若该样本被正确分类，则结束训练\n", " if y[idx] * y_pred[idx] > 0:\n", " break\n", " # 否则，让参数沿着负梯度方向走一步\n", " delta = lr * y[idx]\n", " self._w += delta * x[idx]\n", " self._b += delta\n", " \n", " def predict(self, x, raw=False):\n", " x = np.asarray(x, np.float32)\n", " y_pred = x.dot(self._w) + self._b\n", " if raw:\n", " return y_pred\n", " return np.sign(y_pred).astype(np.float32)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "from Util import gen_two_clusters\n", "\n", "x, y = gen_two_clusters()\n", "perceptron = Perceptron()\n", "perceptron.fit(x, y)\n", "print(\"准确率：{:8.6} %\".format((perceptron.predict(x) == y).mean() * 100))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "from Util import visualize2d\n", "\n", "visualize2d(perceptron, x, y)\n", "visualize2d(perceptron, x, y, True)" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### *线性可分性相关数学理论\n", "\n", "从数学的角度来说，线性可分性还有一个比较直观的等价定义：正负样本点集的凸包彼此不交。\n", "\n", "所谓凸包的定义如下：若集合$ S\\subset\\mathbb{R}^n $ 由N个点组成：\n", "\n", "$$ S={x_1,...,x_N}\\\\ (x_i\\in\\mathbb{R}^n,\\forall i=1,...,N) $$\n", "\n", "那么S的凸包 $ \\text{conv}(S) $ 即为：\n", "\n", " $$ \\text{conv}(S)=\\left\\{ x=\\sum_{i=1}^N\\lambda_ix_i|\\sum_{i=1}^N\\lambda_i=1,\\lambda_i\\ge0\\ (i=1,...,N) \\right\\}\n", " $$ \n", "比如，上文给出过的两个二维数据集的凸包将如下图所示：\n", "\n", "![凸包](svm5.jpg)\n", "\n", "左图正负样本点集的凸包不交、所以数据集线性可分，右图的橙色区域即为正负样本点集凸包的相交处、所以数据集线性不可分\n", "\n", "该等价性的证明可以用反证法得出：\n", "\n", "###### 1）线性可分 $ \\Rightarrow $ 凸包不交：\n", "\n", "线性可分意味着存在 $ w^* $ 和 $ b^* $ ，使得 $ y(w^*\\cdot x+b^*)>0 $ 对任意 $ (x,y)\\in D $ 成立。\n", "\n", "如果凸包相交的话，就意味着存在某个样本 $ (x^*,y^*) $ 、使得 $ x^* $ 既是正样本输入数据的线性组合、又是负样本输入数据的线性组合：\n", "\n", " $$ x^*=\\sum_{y_i=1}{\\lambda_ix_i}=\\sum_{y_i=-1}{\\mu_ix_i} $$ \n", "\n", "从而\n", "\n", " $$ y^*(w^*\\cdot x^*+b^*)=y^*\\left(w^*\\cdot\\sum_{y_i=1}{\\lambda_ix_i+b^*}\\right)=y^*\\left(w^*\\cdot\\sum_{y_i=-1}{\\mu_ix_i+b^*}\\right)>0（式 1） $$ \n", "\n", "注意到\n", "\n", " $$ y_i=1时，w^*\\cdot x_i+b^*>0 $$ \n", " $$ y_i=-1时，w^*\\cdot x_i+b^*<0 $$ \n", " \n", "所以（注意由凸包的定义我们有 $ \\sum_i\\lambda_i=\\sum_i\\mu_i=1且\\lambda_i\\ge0,\\mu_i\\ge0 $ ）\n", "\n", " $$ w^*\\cdot\\sum_{y_i=1}{\\lambda_ix_i+b^*}=\\sum_{y_i=1}{\\lambda_i\\left(w^*\\cdot x_i+b^*\\right)}>0 $$ \n", "\n", " $$ w^*\\cdot\\sum_{y_i=-1}{\\mu_ix_i+b^*}=\\sum_{y_i=-1}{\\mu_i\\left(w^*\\cdot x_i+b^*\\right)}<0 $$ \n", "\n", "这与式 1 矛盾\n", "\n", "###### 2）凸包不交 $ \\Rightarrow $ 线性可分：\n", "\n", "严谨证明需要用到一些奇怪的东西，这里就只提供一个（非常）不严谨的直观说明（欢迎观众老爷们提供更好的证明，现在这个说明我看上去觉得很像是错的）（喂）：\n", "\n", "在正样本点集凸包的边界上取一个离负样本点集凸包“最近”的点 $ x^{*(1)} $ 并假设负样本点集凸包边界上离 $ x^{*(1)} $ “最近”的点为 $ x^{*(2)} $ 。\n", "\n", "过 $ x^{*(1)} $ 画一个超平面 $ \\Pi:w^*\\cdot x+b^*=0 $ 、使得 $ \\Pi $ 与 $ x^{*(1)} $ 、 $ x^{*(2)} $ 的连线垂直。\n", "\n", "由凸包的几何性质可知此时（除了 $ x^{*(1)}外 $ ）正样本点集都被分到了 $ \\Pi $ 的同一侧、且 $ x^{*(2)} $ 是离 $ \\Pi $ “最近”的点，这样只需把 $ \\Pi $ 稍微往负样本点集那边挪一点（什么鬼！）就行了\n", "\n", "然后是前文遗留下来的、感知机模型收敛性的证明。\n", "\n", "我们知道感知机对应的超平面为：\n", "\n", " $$ \\Pi:w\\cdot x+b=0 $$ \n", "\n", "将其展开的话、就是\n", "\n", " $$ \\Pi:w_1x_1+...+w_nx_n+b=0 $$ \n", "\n", "所以我们可以将其改写为\n", "\n", " $$ \\Pi:\\hat w\\cdot \\hat x=0 $$ \n", "\n", "其中\n", "\n", " $$ \\hat w=(w_1,...,w_n,b),\\hat x=(x_1,...,x_n,1) $$ \n", "\n", "如果数据集线性可分的话，就意味着存在 $ \\hat w^* $ 、使得对任意 $ (x,y)\\in D $ 、都有 $ y(\\hat w^*\\cdot \\hat x)>0 $ ；注意到 $ \\hat w^* $ 的 $ scale $ 不影响超平面、所以我们不妨假设 $ \\|w^*\\|=1 $ 。同时由于数据集D中的样本是有限的，所以这又意味着 $ \\exists\\gamma,\\Delta $ 、使得总有 $ y(\\hat w^*\\cdot\\hat x^*)\\ge\\gamma,\\|x_i\\|^2\\le\\Delta $ \n", "\n", "现在我们初始化 $ \\hat w $ 为 0 向量（ $ \\hat w_0 = \\textbf{0} $ ），并开始感知机模型的训练（假设现在是第k步）：\n", "\n", "\n", "\n", "1）假设 $ \\Pi_k:\\hat w_k\\cdot\\hat x=0 $ 已经将所有样本正确分类，则已证毕\n", "\n", "2）否则，取被 $ \\Pi_k $ 误分类的样本 $ (x_i,y_i) $ ，进行参数的更新： $ \\hat w_{k+1}=\\hat w_k+\\eta y_i\\cdot\\hat x_i $ 。由此易知（注意 $ \\hat w_0 = \\textbf{0} $ ）：\n", "\n", " $$ \\begin{align} \\hat w^*\\cdot \\hat w_{k+1}&=\\hat w^*\\cdot \\hat w_k+\\eta y_i(\\hat w^*\\cdot \\hat x_i)\\ge\\hat w^*\\cdot \\hat w_k+\\eta\\gamma \\\\ &\\ge...\\ge\\hat w^*\\cdot \\hat w_0+(k+1)\\eta\\gamma=(k+1)\\eta\\gamma \\end{align}\n", " $$ \n", "且\n", "\n", " $$ \\|\\hat w^*\\|^2\\cdot\\|\\hat w_{k+1}\\|^2=\\|\\hat w_{k+1}\\|^2=\\|\\hat w_k\\|^2+2\\eta y_i(\\hat w_k\\cdot\\hat x_i)+\\eta^2y_i^2(\\hat x_i\\cdot\\hat x_i)（式 2） $$ \n", "\n", "注意 $ (x_i,y_i) $ 是被误分类的、且 $ y_i $ 只能取 $ \\pm 1 $ ，所以 $ y_i(\\hat w_k\\cdot\\hat x_i)\\le0、y_i^2=1 $ ，从而由 $ 式 2 $ 可以推出：\n", "\n", " $$ \\begin{align} \\|\\hat w^*\\|^2\\cdot\\|\\hat w_{k+1}\\|^2&\\le\\|\\hat w_k\\|^2+\\eta^2\\|x_i\\|^2\\le\\|\\hat w_k\\|^2+\\eta^2\\Delta \\\\ &\\le...\\le\\|\\hat w_0\\|^2+(k+1)\\eta^2\\Delta=(k+1)\\eta^2\\Delta \\end{align} $$ \n", "\n", "从而\n", "\n", " $$ \\begin{align} &\\left[(k+1)\\eta\\gamma\\right]^2\\le(\\hat w^*\\cdot\\hat w_{k+1})^2\\le\\|\\hat w^*\\|^2\\cdot\\|\\hat w_{k+1}\\|^2\\le(k+1)\\eta^2\\Delta \\\\ &\\Rightarrow k+1\\le\\frac\\Delta{\\gamma^2} \\end{align} $$ \n", "\n", "亦即训练步数k是有上界的，这意味着收敛性。而且 $ \\frac\\Delta{\\gamma^2} $ 中不含学习速率 $ \\eta $ ，这说明对感知机模型来说、学习速率不会影响收敛性\n", "\n", "##### *拉格朗日对偶性（Lagrange Duality）\n", "\n", "最后简单介绍一个非常重要的概念：拉格朗日对偶性（Lagrange Duality）。\n", "\n", "我们在前三小节介绍的感知机算法，其实可以称为“感知机的原始算法”；而利用拉格朗日对偶性，我们可以得到感知机算法的对偶形式。\n", "\n", "鉴于拉格朗日对偶性的原始形式太过纯数学，所以我打算结合具体的算法来介绍、而不打算叙述其原始形式，感兴趣的观众老爷可以参见[这里](https://en.wikipedia.org/wiki/Duality_(optimization))\n", "\n", "在有约束的最优化问题中，为了便于求解、我们常常会利用它来将比较原始问题转化为更好解决的对偶问题。\n", "\n", "对于特定的问题，原始算法的对偶形式也常常会有一些共性存在。\n", "\n", "比如对于感知机和后文会介绍的支持向量机来说，它们的对偶算法都会将模型的参数表示为样本点的某种线性组合、并把问题转化为求解线性组合中的各个系数\n", "\n", "虽说感知机算法的原始形式已经非常简单，但是通过将它转化为对偶形式、我们可以比较清晰地感受到转化的过程，这有助于理解和记忆后文介绍的、较为复杂的支持向量机的对偶形式\n", "\n", "考虑到原始算法的核心步骤为：\n", "\n", " $$ w\\leftarrow w+\\eta y_ix_i $$\n", "\n", " $$ b\\leftarrow b+\\eta y_i $$ \n", "\n", "其中 $ (x_i,y_i)\\in E、E $ 是当前被误分类的样本点的集合；可以看见、参数的更新是完全基于样本点的。考虑到我们要将参数$w和b$表示为样本点的线性组合，一个自然的想法就是记录下在核心步骤中、各个样本点分别被利用了多少次、然后利用这个次数来将$w和b$表示出来。比如说，若设样本点$(x_i,y_i)$一共在上述核心步骤中被利用了$ n_i $次、那么就有（假设初始化参数时$ w=(0,...,0)^T\\in \\mathbb{R}^n,b=0 $ ）：\n", "\n", " $$ w = \\eta\\sum_{i=1}^N{n_iy_ix_i} $$ \n", "\n", " $$ b=\\eta\\sum_{i=1}^N{n_iy_i} $$ \n", "\n", "如果进一步设$ \\alpha_i=\\eta n_i $，则有：\n", "\n", "$$ w = \\sum_{i=1}^N{\\alpha_iy_ix_i}$$\n", "\n", "$$ b=\\sum_{i=1}^N{\\alpha_iy_i}$$\n", "\n", "此即感知机模型的对偶形式。\n", "\n", "需要指出的是，在对偶形式中、样本点里面的$x$仅以内积的形式$（x_i\\cdot x_j）$出现；这是一个非常重要且深刻的性质，利用它和后文将进行介绍的核技巧、能够将许多算法从线性算法“升级”成为非线性算法\n", "\n", "注意到对偶形式的训练过程常常会重复用到大量的、样本点之间的内积，我们通常会提前将样本点两两之间的内积计算出来并存储在一个矩阵中；这个矩阵就是著名的 Gram 矩阵、其数学定义即为：\n", "\n", "$$G=(x_i\\cdot x_j)_{N\\times N}$$\n", "\n", "从而在训练过程中如果要用到相应的内积、只需从 Gram 矩阵中提取即可，这样在大多数情况下都能大大提高效率" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "#### LinearSVM\n", "\n", "很多人（包括我）第一次听说 SVM 时都觉得它是个非常厉害的东西，但其实 SVM 本身“只是”一个线性模型。只有在应用了核方法后，SVM 才会“升级”成为一个非线性模型\n", "\n", "不过由于普遍说起 SVM 时我们都默认它带核方法，所以我们还是随大流、称 SVM 的原始版本为 LinearSVM。不过即使“只是”线性模型，这个“只是”也是要打双引号的——它依旧强大，且在许许多多的问题上甚至要比带核方法的 SVM 要好（比如文本分类）\n", "\n", "LinearSVM 往简单来说其实就只是改了感知机的损失函数而已，而且改完之后还很像" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### 感知机的问题与 LinearSVM 的解决方案\n", "\n", "由感知机损失函数的形式可知，感知机只要求样本被正确分类，而不要求样本被“很好地正确分类”。这就导致感知机弄出来的超平面（通常又称“决策面”）经常会“看上去很不舒服”：\n", "\n", "![](svm8.png)\n", "\n", "之所以看上去很不舒服，是因为决策面离两坨样本都太近了。从直观上来说，我们希望得到的是这样的决策面：\n", "\n", "![](svm9.png)\n", "\n", "（之所以画风突变是因为 matplotlib 的默认画风变了，然后我懒得改了……）（喂\n", "\n", "那么应该如何将这种理想的决策面的状态翻译成机器能够学习的东西呢？直观来说，就是让决策面离正负样本点的间隔都尽可能大；而这个“间隔”翻译成数学语言，其实就是简单的：\n", "\n", " $$ d((x,y),\\Pi)=\\frac {1}{\\|w\\|}y(w\\cdot x+b)（文末会给出相应说明） $$ \n", "\n", "在有了样本点到决策面的间隔后，数据集到决策面的间隔也就好定义了：\n", "\n", " $$ d(D, \\Pi)=\\min_{(x,y)\\in D}d((x,y),\\Pi) $$\n", "所以我们现在的目的就转化为：\n", "\n", "- 让所有样本点都被正确分类： $ y(w\\cdot x+b)>0,\\forall(x,y)\\in D $ \n", "\n", "- 让决策面离正负样本点的间隔都尽可能大： $ \\max\\min_{(x,y)\\in D}\\frac {1}{\\|w\\|}y(w\\cdot x+b) $ \n", "\n", "注意到 $ y(w\\cdot x+b)>0 $ 的性质和 $ \\frac {1}{\\|w\\|}y(w\\cdot x+b) $ 的值在 $ w $ 和 $ b $ 同时扩大 $ k $ 倍时不会改变，所以我们完全可以假设：\n", "\n", "$$\n", "若 (x^*,y^*)=\\arg\\min_{(x,y)\\in D}\\frac {1}{\\|w\\|}y(w\\cdot x+b) ，则 y^*(w\\cdot x^*+b)=1\n", "$$\n", "$$\n", "（否则假设 y^*(w\\cdot x^*+b)=c ，令 w\\leftarrow\\frac{ w}{c},b\\leftarrow\\frac{ b}{c} 即可）\n", "$$\n", "\n", "注意由于 $ (x^*,y^*)=\\arg\\min_{(x,y)\\in D}\\frac {1}{\\|w\\|}y(w\\cdot x+b) $ 这个最小化过程中$w$是固定的，所以我们可以把 $ \\frac1{\\|w\\|} $ 这一项拿掉，从而：\n", "\n", "$$ (x^*,y^*)=\\arg\\min_{(x,y)\\in D}y(w\\cdot x+b) $$\n", "\n", "所以 $ y^*(w\\cdot x^*+b)=1\\Rightarrow y(w\\cdot x+b)\\ge1,\\forall(x,y)\\in D $\n", "\n", "于是优化问题转为：\n", "\n", "$$ \\max_{w,b}\\frac {1}{\\|w\\|}，使得y_i(w\\cdot x_i+b)\\ge1,\\forall(x_i,y_i)\\in D $$\n", "\n", "亦即\n", "\n", "$$ \\min_{w,b}\\frac {\\|w\\|^2}2，使得y_i(w\\cdot x_i+b)\\ge1,\\forall(x_i,y_i)\\in D $$\n", "\n", "但是这会导致另一个问题：当数据集线性不可分时，上述优化问题是必定无解的，这就会导致模型震荡（换句话说， $ y_i(w\\cdot x_i+b)\\ge1 $ 这个约束太“硬”了）。\n", "\n", "所以为了让模型在线性不可分的数据上仍有不错的表现，从直观来说，我们应该“放松”对我们模型的限制（让我们模型的约束“软”一点）：\n", "\n", "$$ \\min_{w,b}\\frac {\\|w\\|^2}2，使得y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D $$\n", "其中$\\xi_i\\ge0$。当然仅仅放松限制会使模型变得怠惰（咦），所以我们还需要让这种放松受到惩罚：\n", "\n", "$$\\min_{w,b}\\left[\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i\\right]，使得y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D（\\xi_i\\ge0）$$\n", "\n", "其中$C$是一个常数，可以把它理解为“惩罚力度”（这样做的合理性会在文末给出）。\n", "\n", "若假设数据集为 $ D=\\left\\{(x_1,y_1),...,(x_N,y_N)\\right\\} $ 的话，那么经过数学变换后，可知上述优化问题等价于（文末会给出过程）：\n", "\n", "$$\\min_{w,b}{\\left[\\frac{\\|w\\|^2}2 + C\\sum_{i=1}^N[1-y_i(w\\cdot x_i+b)]_+\\right]}$$\n", "\n", "其中“ $ [ \\cdot ]_+ $ ”其实就是 ReLU 函数：\n", "$$\n", "[x]_+=\\left\\{\n", "\\begin{aligned}\n", "0&\\ ,\\ x\\le0 \\\\\n", "x&\\ ,\\ x>0\n", "\\end{aligned}\n", "\\right.\n", "$$\n", "\n", "注意我们感知机的损失函数为 $ \\sum_{i=1}^N[-y(w\\cdot x+b)]_+ $ ，于是综上所述可以看出，LinearSVM 在形式上和感知机的差别只在于损失函数、且这两个损失函数确实长得很像\n", "\n", "##### LinearSVM 的训练\n", "\n", "【虽然比较简单，但是调优 LinearSVM 的训练这个过程是相当有启发性的事情。仍然是那句老话：麻雀虽小，五脏俱全。我们会先展示“极大梯度下降法”的有效性，然后会展示极大梯度下降法存在的问题，最后则会介绍如何应用 Mini-Batch 梯度下降法（MBGD）来进行训练】\n", "\n", "为了使用梯度下降法，我们需要先求导。我们已知：\n", "\n", "$$L(D)=\\frac{\\|w\\|^2}2 + C\\sum_{i=1}^N[1-y_i(w\\cdot x_i+b)]_+$$\n", "所以我们可以认为：\n", "\n", "$$L(x,y)=\\frac{\\|w\\|^2}2+C[1-y(w\\cdot x+b)]_+$$\n", "于是：\n", "\n", "- $$当 y(w\\cdot x+b)\\ge1 时： \\frac{\\partial L(x,y)}{\\partial w} = w、\\frac{\\partial L(x,y)}{\\partial b}=0 $$\n", "\n", "\n", "- $$当 y(w\\cdot x+b)<1 时： \\frac{\\partial L(x,y)}{\\partial w} = w-Cyx、\\frac{\\partial L(x,y)}{\\partial b}=-Cy $$ \n", "\n", "\n", "所以我们可以把极大梯度下降的形式写成（假设学习速率为 $ \\eta $ ）：\n", "\n", "$$w\\leftarrow (1-\\eta)w$$\n", "\n", "若 $ y(w\\cdot x+b)<1 $ ，则选出某个被错分的样本(x,y)，然后：\n", "\n", " $$ w\\leftarrow w+\\eta Cyx $$ \n", " $$ b\\leftarrow b+\\eta Cy $$ \n", "\n", "我们完全可以照搬感知机里的代码来完成实现（由于思路基本一致，这里就略去注释了）：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "import numpy as np\n", "\n", "class LinearSVM:\n", " def __init__(self):\n", " self._w = self._b = None\n", " \n", " def fit(self, x, y, c=1, lr=0.01, epoch=10000):\n", " x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)\n", " self._w = np.zeros(x.shape[1])\n", " self._b = 0.\n", " for _ in range(epoch):\n", " self._w *= 1 - lr\n", " err = 1 - y * self.predict(x, True)\n", " idx = np.argmax(err)\n", " # 注意即使所有 x, y 都满足 w·x + b >= 1\n", " # 由于损失里面有一个 w 的模长平方\n", " # 所以仍然不能终止训练，只能截断当前的梯度下降\n", " if err[idx] <= 0:\n", " break\n", " delta = lr * c * y[idx]\n", " self._w += delta * x[idx]\n", " self._b += delta\n", " \n", " def predict(self, x, raw=False):\n", " x = np.asarray(x, np.float32)\n", " y_pred = x.dot(self._w) + self._b\n", " if raw:\n", " return y_pred\n", " return np.sign(y_pred).astype(np.float32)\n" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "下面这张动图是该 LinearSVM 的训练过程：\n", "![LinearSVM](https://cdn.rawgit.com/carefree0910/Resources/f0978da2/Lines/LinearSVM.gif)\n", "\n", "![LinearSVM](https://cdn.rawgit.com/carefree0910/Resources/f0978da2/Backgrounds/LinearSVM.gif)" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "虽然看上去不错，但仍然存在着问题：\n", "\n", "- 训练过程其实非常不稳定\n", "- 从直观上来说，由于 LinearSVM 的损失函数比感知机要更复杂，所以相应的函数形状也会更复杂。这意味着当数据集稍微差一点的时候，直接单纯地应用极大梯度下降法可能会导致一些问题——比如说模型会卡在某个很奇怪的地方无法自拔（什么鬼）\n", "\n", "通过将正负样本点的“中心”从原点 (0, 0)（默认值）挪到 (5, 5)（亦即破坏了一定的对称性）并将正负样本点之间的距离拉近一点，我们可以复现这个问题：\n", "\n", "![Failure of LinearSVM][1]\n", "\n", "原理我不敢乱说，这里只提供一个牵强附会的直观解释：\n", "\n", "- 每次只取使得损失函数极大的一个样本进行梯度下降\\rightarrow模型在某个地方可能来来回回都只受那么几个样本的影响\\rightarrow死循环（什么鬼！）\n", "\n", "专业的理论就留待专业的观众老爷补充吧 ( σ'ω')σ\n", "\n", "然后解决方案的话，主要还是从改进随机梯度下降（SGD）的思路入手（因为极大梯度下降法其实就是 SGD 的特殊形式）。\n", "\n", "我们知道 SGD 的“升级版”是 MBGD、亦即拿随机 Mini-Batch 代替随机抽样，我们这里也完全可以依样画葫芦。\n", "\n", "以下是对应代码（只显示出了核心部分）：\n", "\n", "[1]: https://cdn.rawgit.com/carefree0910/Resources/83441596/Lines/LinearSVM(Failed1).gif\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "self._w *= 1 - lr\n", "# 随机选取 batch_size 个样本\n", "batch = np.random.choice(len(x), batch_size)\n", "x_batch, y_batch = x[batch], y[batch]\n", "err = 1 - y_batch * self.predict(x_batch, True)\n", "if np.max(err) <= 0:\n", " continue\n", "# 注意这里我们只能利用误分类的样本做梯度下降\n", "# 因为被正确分类的样本处、这一部分的梯度为 0\n", "mask = err > 0\n", "delta = lr * c * y_batch[mask]\n", "# 取各梯度平均并做一步梯度下降\n", "self._w += np.mean(delta[..., None] * x_batch[mask], axis=0)\n", "self._b += np.mean(delta)" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "这样的话，通常而言会比 SGD 要好\n", "\n", "但是问题仍然是存在的：那就是它们所运用的梯度下降法都只是朴素的 Vanilla Update，这会导致当数据的 scale 很大时模型对参数极为敏感、从而导致持续的震荡（所谓的 scale 比较大，可以理解为“规模很大”，或者直白一点——以二维数据为例的话——就是横纵坐标的数值很大）。\n", "\n", "下面这张动图或许能提供一些直观：\n", "\n", "![Failure of LinearSVM][2]\n", "\n", "可以看到，模型确实一直在持续震荡\n", "\n", "[2]: https://cdn.rawgit.com/carefree0910/Resources/83441596/Lines/LinearSVM(Failed2).gif\n", "\n", "Again，原理我不敢乱说，所以只提供一个有可能对（更有可能错）（喂）的直观解释：\n", "\n", "scale太大$\\rightarrow$梯度很大$\\rightarrow$蹦跶得很欢（什么鬼！）\n", "\n", "专业的理论就留待专业的观众老爷补充吧 ( σ'ω')σ\n", "\n", "解决方案的话，一个很直接的想法就是进行数据归一化：$X\\leftarrow \\frac{X-\\bar X}{\\sqrt{Var(X)}}$。\n", "\n", "事实证明这样做了之后，最基本的极大梯度下降法也能解决上文出现过的所有问题了\n", "\n", "然后一个稍微“偷懒”一点的做法就是，用更好的梯度下降算法来代替朴素的 Vanilla Update。\n", "\n", "比如说 Adam 的训练过程将如下（这张动图被知乎弄得有点崩……将就着看吧 ( σ'ω')σ）：\n", "\n", "![Adam](svm10.gif)\n", "\n", "关于各种梯度下降算法的定义、性质等等可以参见[这篇文章](http://www.carefree0910.com/posts/55a23cf0/)，实现和在 LinearSVM 上的应用则可以参见[这里](https://github.com/carefree0910/MachineLearning/blob/master/NN/Basic/Optimizers.py)和[这里](https://github.com/carefree0910/MachineLearning/blob/master/e_SVM/LinearSVM.py#L59)\n", "\n", "##### 相关数学理论\n", "\n", "我们尚未解决的问题有三个，但这些问题基本都挺直观的，所以大体上不深究也没问题（趴：\n", "\n", "- 为什么被正确分类的样本(x,y)到决策面$\\Pi:w\\cdot x+b=0$的间隔可以写成$d((x,y),\\Pi)=\\frac {1}{\\|w\\|}y(w\\cdot x+b)$\n", "\n", "- 为什么把优化问题从$* \\min_{w,b}\\frac {\\|w\\|^2}2，使得y_i(w\\cdot x_i+b)\\ge1,\\forall(x_i,y_i)\\in D\n", "$转化成$* \\min_{w,b}\\left[\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i\\right]，使得y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D（\\xi_i\\ge0）$是合理的\n", "\n", "- 为什么上面这 个优化问题$* \\min_{w,b}\\left[\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i\\right]，使得y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D（\\xi_i\\ge0）\n", "$等价于$* \\min_{w,b}{\\left[\\frac{\\|w\\|^2}2 + C\\sum_{i=1}^N[1-y_i(w\\cdot x_i+b)]_+\\right]}$\n", "\n", "这三个问题有一定递进关系，我们一个个来看\n", "\n", "###### 1）间隔的定义\n", "\n", "我们在定义点(x,y)到平面（超平面）\\Pi的间隔时，一般都是这样做的：\n", "\n", "- 将(x,y)（垂直）投影到$\\Pi$上\n", "- $设投影点为(x^*,y^*)，则定义d((x,y),\\Pi)=\\left\\{\n", "\\begin{aligned}\n", "\\|x-x^*\\|^2,&\\ \\ y(w\\cdot x + b) \\ge0 \\\\\n", "-\\|x-x^*\\|^2,&\\ \\ y(w\\cdot x + b) <0\n", "\\end{aligned}\n", "\\right.\n", "$\n", "\n", "![](svm-math1.png)\n", "\n", "注意这里我们允许（当样本被错分类时的）间隔为负数，所以间隔其实严格来说并不是一般意义上的距离\n", "\n", "那么为了找到垂直投影，我们得先找到垂直于超平面$\\Pi$的方向。不难看出$w$就是垂直于$\\Pi$的，因为对$\\forall x_1,x_2\\in\\Pi$，由$\\left\\{\n", "\\begin{aligned}\n", "&w\\cdot x_1+b=0 \\\\\n", "&w\\cdot x_2+b=0\n", "\\end{aligned}\n", "\\right.$知$w\\cdot(x_1-x_2)=0$（两式相减即可），从而$w$垂直于向量$x_1-x_2$，从而也就垂直于$\\Pi$：\n", "\n", "![](svm-math2.png)\n", "\n", "那么结合之前那张图，不难得知我们可以设$x-x^*=\\lambda w（这里的\\lambda可正可负）$，于是就有$（注意由x^*\\in\\Pi知w\\cdot x^*+b=0）$\n", "$$\n", "\\begin{align}\n", "\\|x-x^*\\|^2&=(x-x^*)\\cdot(x-x^*)=\\lambda w\\cdot(x-x^*) \\\\\n", "&=\\lambda \\left[w\\cdot(x-x^*)+(b-b)\\right]\\\\\n", "&=\\lambda\\left[ w\\cdot x+b - (w\\cdot x^* + b)\\right] \\\\\n", "&=\\lambda(w\\cdot x+b)\n", "\\end{align}\n", "$$\n", "从而\n", "$$\n", "d((x,y),\\Pi)=\\left\\{\n", "\\begin{aligned}\n", "\\lambda(w\\cdot x+b),&\\ \\ y(w\\cdot x + b) \\ge0 \\\\\n", "-\\lambda(w\\cdot x+b),&\\ \\ y(w\\cdot x + b) <0\n", "\\end{aligned}\n", "\\right.\n", "$$\n", "\n", "注意这么定义的间隔有一个大问题：当$w$和$b$同时增大$k$倍时，新得到的超平面$\\tilde\\Pi:(kw)\\cdot x+(kb)$其实等价于原超平面$\\Pi$：\n", "\n", "$$x\\in\\tilde\\Pi\\Leftrightarrow(kw)\\cdot x+(kb)=0\\Leftrightarrow w\\cdot x+b=0\\Leftrightarrow x\\in\\Pi$$\n", "\n", "但此时$d((x,y),\\Pi)$却会直接增大$k$倍。极端的情况就是，当$w$和$b$同时增大无穷倍时，超平面没变，间隔却也跟着增大了无穷倍，这当然是不合理的\n", "\n", "所以我们需要把 scale 的影响给抹去，常见的做法就是做某种意义上的归一化：\n", "$$\n", "d((x,y),\\Pi)=\\left\\{\n", "\\begin{aligned}\n", "\\frac1{\\|w\\|}|w\\cdot x+b|,&\\ \\ y(w\\cdot x + b) \\ge0 \\\\\n", "-\\frac1{\\|w\\|}|w\\cdot x+b|,&\\ \\ y(w\\cdot x + b) <0\n", "\\end{aligned}\n", "\\right.\n", "$$\n", "（注意：由于 scale 的影响已被抹去，所以$\\lambda$也就跟着被抹去了；同时由$0\\le\\|x-x^*\\|^2=\\lambda(w\\cdot x+b)$知，我们需要在抹去$\\lambda$的同时、给$w\\cdot x+b$套一个绝对值)\n", "\n", "不难看出上式可改写为：\n", "$$\n", "d((x,y),\\Pi)=\\frac1{\\|w\\|}y(w\\cdot x+b)\n", "$$\n", "\n", "这正是我们想要的结果\n", "\n", "###### 2）优化问题的转化的合理性\n", "\n", "我们已知原问题为\n", "$$\n", "\\min_{w,b}\\frac {\\|w\\|^2}2，使得y_i(w\\cdot x_i+b)\\ge1,\\forall(x_i,y_i)\\in D\n", "$$\n", "且由 1）知，式中的$y_i(w\\cdot x_i+b)$其实就是（没有抹去 scale 的影响的）间隔。\n", "\n", "所以想要放松对模型的限制的话，很自然的想法就是让这个间隔不必一定要不小于 1、而是只要不小于$1-\\xi_i$就行，其中$\\xi_i$是个不小于 0 的数。正如前文所说，只放松限制的话肯定不行、还得给这个放松一些惩罚，所以就在损失函数中加一个$C\\xi_i$即可，其中$C$是个大于 0 的常数、可以理解为对放松的惩罚力度\n", "\n", "综上所述，优化问题即可合理地转化为：\n", "$$\n", "\\min_{w,b}\\left[\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i\\right]，使得y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D（\\xi_i\\ge0）\n", "$$\n", "\n", "###### 3）优化问题的等价性\n", "\n", "为方便，称优化问题：\n", "$$\n", "\\min_{w,b}\\left[\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i\\right]，使得y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D（\\xi_i\\ge0）\n", "$$\n", "为问题一；\n", "\n", "称：\n", "$$\n", "\\min_{w,b}{\\left[\\frac{\\|w\\|^2}2 + C\\sum_{i=1}^N[1-y_i(w\\cdot x_i+b)]_+\\right]}\n", "$$\n", "为问题二，则我们需要证明问题一与问题二等价\n", "\n", "先来看问题一怎么转为问题二。\n", "\n", "事实上不难得知：\n", "$$\n", "y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D\\Rightarrow\\xi_i\\ge1-y_i(w\\cdot x_i+b)\n", "$$\n", "\n", "注意问题一是针对$w$和$b$进行优化的，且当$w$和$b$固定时，为使$\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i$最小，必有：\n", "\n", "\n", "- $1-y_i(w\\cdot x_i+b)\\ge0时，\\xi_i=1-y_i(w\\cdot x_i+b) $\n", "\n", "- $1-y_i(w\\cdot x_i+b)<0时，\\xi_i=0（因为我们要求\\xi_i\\ge0）$\n", "\n", "亦即$ \\xi_i=[1-y_i(w\\cdot x_i+b)]_+ $。\n", "\n", "此时损失函数即为$\\frac{\\|w\\|^2}2 + C\\sum_{i=1}^N[1-y_i(w\\cdot x_i+b)]_+$，换句话说，我们就把问题一转为了问题二\n", "\n", "再来看问题二怎么转为问题一。\n", "\n", "事实上，直接令$\\xi_i=[1-y_i(w\\cdot x_i+b)]_+$，就有：\n", "\n", "- 模型的损失为$\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i$\n", "\n", "- 模型的约束为$\\xi_i\\ge 1-y_i(w\\cdot x_i+b)且\\xi_i\\ge0$\n", "\n", "亦即转为了问题一\n", "\n", "###### 4）LinearSVM 的对偶问题\n", "\n", "原始问题\n", "$$\n", "\\min_{w,b}\\left[\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i\\right]，使得y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D（\\xi_i\\ge0）\n", "$$\n", "的对偶问题为\n", "$$\n", "\\min_{\\alpha}\\left[ \\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jy_iy_j(x_i\\cdot x_j)-\\sum_{i=1}^N\\alpha_i\\right]，使得\\sum_{i=1}^N\\alpha_iy_i=0、0\\le\\alpha_i\\le C\n", "$$\n", "通过拉格朗日乘子法可以比较简单地完成证明。不难得知原始问题相应的拉格朗日函数为：\n", "$$\n", "L=\\frac{\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i-\\sum_{i=1}^N\\alpha_i[y_i(w\\cdot x_i+b)-1+\\xi_i]-\\sum_{i=1}^N\\beta_i\\xi_i\n", "$$\n", "其中$\\alpha_i\\ge0、\\beta_i\\ge0$，于是原始问题为\n", "$$\n", "\\min_{w,b,\\xi}\\max_{\\alpha,\\beta} L\n", "$$\n", "从而对偶问题为\n", "$$\n", "\\max_{\\alpha,\\beta}\\min_{w,b,\\xi}L\n", "$$\n", "于是我们需要求偏导并令它们为 0：\n", "\n", "- 对w求偏导：$\\frac{\\partial L}{\\partial w}=w-\\sum_{i=1}^N\\alpha_iy_ix_i=0\\Rightarrow w=\\sum_{i=1}^N\\alpha_iy_ix_i$\n", "\n", "- 对b求偏导：$\\frac{\\partial L}{\\partial b}=-\\sum_{i=1}^N\\alpha_iy_i=0\\Rightarrow\\sum_{i=1}^N\\alpha_iy_i=0$\n", "\n", "- 对$\\xi_i$求偏导：$\\frac{\\partial L}{\\partial\\xi_i}=C-\\alpha_i-\\beta_i=0\\Rightarrow\\alpha_i+\\beta_i=C$\n", "\n", "注意这些约束中$\\beta_i除了\\beta_i\\ge0$之外没有其它约束，$\\alpha_i+\\beta_i=C$的约束可以转为$\\alpha_i\\le C$。然后把这些东西代入拉格朗日函数L、即可得到：\n", "$$\n", "\\begin{align}\n", "L&=\\frac{\\|\\sum_{i=1}^N\\alpha_iy_ix_i\\|^2}2+\\sum_{i=1}^N(C-\\alpha_i-\\beta_i)\\xi_i-\\sum_{i=1}^N\\alpha_iy_i\\left(\\sum_{j=1}^N\\alpha_jy_jx_j\\right)\\cdot x_i-b\\sum_{i=1}^N\\alpha_iy_i+\\sum_{i=1}^N\\alpha_i \\\\\n", "&=-\\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jy_iy_j(x_i\\cdot x_j)+\\sum_{i=1}^N\\alpha_i\n", "\\end{align}\n", "$$\n", "于是对偶问题为\n", "$$\n", "\\max_{\\alpha}\\left[ -\\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jy_iy_j(x_i\\cdot x_j)+\\sum_{i=1}^N\\alpha_i\\right]，使得\\sum_{i=1}^N\\alpha_iy_i=0、0\\le\\alpha_i\\le C\n", "$$\n", "亦即\n", "$$\n", "\\min_{\\alpha}\\left[ \\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jy_iy_j(x_i\\cdot x_j)-\\sum_{i=1}^N\\alpha_i\\right]，使得\\sum_{i=1}^N\\alpha_iy_i=0、0\\le\\alpha_i\\le C\n", "$$\n", "可以看到在对偶形式中，样本仅以内积的形式$（x_i\\cdot x_j）$出现，这就使得核方法的引入变得简单而自然\n", "\n", "###### 5）Extra\n", "\n", "作为结尾，我来叙述一些上文用到过的、但是没有给出具体名字的概念$（假设样本为(x,y)，超平面为\\Pi:w\\cdot x+b=0）$\n", "\n", "- 样本到超平面的函数间隔为：$y(w\\cdot x+b)$\n", "\n", "- 样本到超平面的几何间隔为：$\\frac1{\\|w\\|}y(w\\cdot x+b)$\n", "\n", "- 优化问题$* \\min_{w,b}\\frac {\\|w\\|^2}2，使得y_i(w\\cdot x_i+b)\\ge1,\\forall(x_i,y_i)\\in D的求解过程常称为硬间隔最大化，求解出来的超平面则常称为最大硬间隔分离超平面$\n", "\n", "- 优化问题$* \\min_{w,b}\\left[\\frac {\\|w\\|^2}2+C\\sum_{i=1}^N\\xi_i\\right]$，使得$y_i(w\\cdot x_i+b)\\ge1-\\xi_i,\\forall(x_i,y_i)\\in D（\\xi_i\\ge0）$的求解过程常称为软间隔最大化，求解出来的超平面则常称为最大软间隔分离超平面\n", "\n", "然后最后的最后，请允许我不加证明地给出两个结论（因为结论直观且证明太长……）：\n", "\n", "- 若数据集线性可分，则最大硬间隔分离超平面存在且唯一\n", "\n", "- 若数据集线性不可分，则最大软间隔分离超平面的解存在但不唯一，其中：\n", "\n", " - 法向量（w）唯一\n", "\n", " - 偏置量（b）可能不唯一\n", "\n", "下一篇文章我们则会介绍核方法，并会介绍如何将它应用到感知机和 SVM 上\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "# 极大梯度下降法训练 LinearSVM\n", "import numpy as np\n", "\n", "class LinearSVM:\n", " def __init__(self):\n", " self._w = self._b = None\n", " \n", " def fit(self, x, y, c=1, lr=0.01, epoch=10000):\n", " x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)\n", " self._w = np.zeros(x.shape[1])\n", " self._b = 0.\n", " for _ in range(epoch):\n", " self._w *= 1 - lr\n", " err = 1 - y * self.predict(x, True)\n", " idx = np.argmax(err)\n", " # 注意即使所有 x, y 都满足 w·x + b >= 1\n", " # 由于损失里面有一个 w 的模长平方\n", " # 所以仍然不能终止训练，只能截断当前的梯度下降\n", " if err[idx] <= 0:\n", " continue\n", " delta = lr * c * y[idx]\n", " self._w += delta * x[idx]\n", " self._b += delta\n", " \n", " def predict(self, x, raw=False):\n", " x = np.asarray(x, np.float32)\n", " y_pred = x.dot(self._w) + self._b\n", " if raw:\n", " return y_pred\n", " return np.sign(y_pred).astype(np.float32)\n", "\n", "# 测试\n", "from Util import gen_two_clusters\n", "\n", "x, y = gen_two_clusters()\n", "svm = LinearSVM()\n", "svm.fit(x, y)\n", "print(\"准确率：{:8.6} %\".format((svm.predict(x) == y).mean() * 100))\n", "\n", "# 可视化\n", "from Util import visualize2d\n", "\n", "visualize2d(svm, x, y)\n", "visualize2d(svm, x, y, True)" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "可视化训练过程\n", "![LinearSVM](https://cdn.rawgit.com/carefree0910/Resources/f0978da2/Lines/LinearSVM.gif)\n", "\n", "![LinearSVM](https://cdn.rawgit.com/carefree0910/Resources/f0978da2/Backgrounds/LinearSVM.gif)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "## 存在的问题\n", "'''\n", "+ 由上述可视化其实已经可以看出，用极大梯度下降法训练 LinearSVM 会非常不稳定\n", "+ 从直观上来说，由于 LinearSVM 的损失函数比感知机要更复杂，所以相应的函数形状也会更复杂。\n", " 这意味着当数据集稍微差一点的时候，直接单纯地应用极大梯度下降法可能会导致一些问题\n", " ——比如说模型会卡在某个很奇怪的地方无法自拔（什么鬼）\n", "\n", "可以通过下面这个栗子来直观感受一下 LinearSVM 存在的这些问题：\n", "'''\n", "# 注意我们只是把 center 参数（亦即正负样本点的“中心”）\n", "# 从原点(0, 0)（默认值）挪到(5, 5)（亦即破坏了一定的对称性）、\n", "# 并将正负样本点之间的距离（dis 参数）稍微拉近了一点而已，\n", "# 结果就已经惨不忍睹了\n", "x, y = gen_two_clusters(center=5, dis=1)\n", "svm = LinearSVM()\n", "svm.fit(x, y)\n", "print(\"准确率：{:8.6} %\".format((svm.predict(x) == y).mean() * 100))\n", "visualize2d(svm, x, y)\n", "visualize2d(svm, x, y, True)" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "![Failure of LinearSVM][1]\n", "\n", "可以看到，LinearSVM 确实卡在了奇怪的地方\n", "\n", "原理我不敢乱说，这里只提供一个牵强附会的直观解释：\n", "\n", "+ 每次只取使得损失函数极大的一个样本进行梯度下降$\\rightarrow$模型在某个地方可能来来回回都只受那么几个样本的影响$\\rightarrow$死循环（什么鬼！）\n", "\n", "专业的理论就留待专业的观众老爷补充吧 ( σ'ω')σ\n", "\n", "[1]: https://cdn.rawgit.com/carefree0910/Resources/83441596/Lines/LinearSVM(Failed1).gif\n", "\n", "###### 解决方案\n", "\n", "极大梯度下降法的最大问题很有可能在于它每次都只根据**使得损失函数最大的一个样本点**来进行梯度下降，这会导致两个问题：\n", "\n", "+ 模型的训练将会很不稳定（这点和随机梯度下降类似）\n", "+ 模型对噪声或“不太好的点”极为敏感（因为它们往往会使损失函数最大）\n", "\n", "按部就班、我们先解决第一个问题，为此我们只需要多选出几个样本点（比如选出使得损失函数最大的 top n 个样本）、然后取它们梯度的平均即可\n", "\n", "+ 注：该名字同样只是我瞎编的一个名字（喂）" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "# Top n 梯度下降法\n", "# 继承上一个 LinearSVM 以重复利用代码\n", "class LinearSVM2(LinearSVM): \n", " # 用参数 batch_size 表示 Top n 中的 n\n", " def fit(self, x, y, c=1, lr=0.01, batch_size=128, epoch=10000):\n", " x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)\n", " # 如果 batch_size 设得比样本总数还多、则将其改为样本总数\n", " batch_size = min(batch_size, len(y))\n", " self._w = np.zeros(x.shape[1])\n", " self._b = 0.\n", " for _ in range(epoch):\n", " self._w *= 1 - lr\n", " err = 1 - y * self.predict(x, True)\n", " # 利用 argsort 函数直接取出 Top n\n", " # 注意 argsort 的结果是从小到大的，所以要用 [::-1] 把结果翻转一下\n", " batch = np.argsort(err)[-batch_size:][::-1]\n", " err = err[batch]\n", " if err[0] <= 0:\n", " continue\n", " # 注意这里我们只能利用误分类的样本做梯度下降\n", " # 因为被正确分类的样本处、这一部分的梯度为 0\n", " mask = err > 0\n", " batch = batch[mask]\n", " # 取各梯度平均并做一步梯度下降\n", " delta = lr * c * y[batch]\n", " self._w += np.mean(delta[..., None] * x[batch], axis=0)\n", " self._b += np.mean(delta)\n", "\n", "# 测试 \n", "x, y = gen_two_clusters(center=5, dis=1)\n", "svm = LinearSVM2()\n", "svm.fit(x, y)\n", "print(\"准确率：{:8.6} %\".format((svm.predict(x) == y).mean() * 100))\n", "\n", "# 可视化\n", "visualize2d(svm, x, y)\n", "visualize2d(svm, x, y, True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "'''\n", "+ 上述解决方案已经不错，但我们还是有些太“激进”了\n", " ——我们每次进行梯度下降时，选取的样本点都是使得损失函数最大的样本点，\n", " 但一般而言使损失函数最大的样本点如果不是关键的样本点（支持向量）的话、通常而言会是噪声。\n", " 当数据集比较差时，噪声所带来的副作用很有可能就会盖过支持向量带来的正效应\n", "+ 为此，我们应该引入一定的随机性。\n", " 神经网络的训练中所用的 MBGD 就是很好的方法：\n", " 每次都从数据集中抽样出一个小 Batch，然后用这个 Batch 来做梯度下降\n", "'''\n", "### Mini-Batch 梯度下降法（MBGD）\n", "class LinearSVM3(LinearSVM):\n", " def fit(self, x, y, c=1, lr=0.01, batch_size=128, epoch=10000):\n", " x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)\n", " batch_size = min(batch_size, len(y))\n", " self._w = np.zeros(x.shape[1])\n", " self._b = 0.\n", " for _ in range(epoch):\n", " self._w *= 1 - lr\n", " # 随机选取 batch_size 个样本\n", " batch = np.random.choice(len(x), batch_size)\n", " x_batch, y_batch = x[batch], y[batch]\n", " err = 1 - y_batch * self.predict(x_batch, True)\n", " if np.max(err) <= 0:\n", " continue\n", " mask = err > 0\n", " delta = lr * c * y_batch[mask]\n", " self._w += np.mean(delta[..., None] * x_batch[mask], axis=0)\n", " self._b += np.mean(delta)\n", "\n", "# 进一步拉近正负样本点间的距离以观察性能\n", "x, y = gen_two_clusters(center=5, dis=0.5)\n", "top_n_svm = LinearSVM2()\n", "top_n_svm.fit(x, y)\n", "print(\"Top n LinearSVM 准确率：{:8.6} %\".format((top_n_svm.predict(x) == y).mean() * 100))\n", "mbgd_svm = LinearSVM3()\n", "mbgd_svm.fit(x, y)\n", "print(\"MBGD LinearSVM 准确率：{:8.6} %\".format((mbgd_svm.predict(x) == y).mean() * 100))\n", "\n", "visualize2d(top_n_svm, x, y)\n", "visualize2d(mbgd_svm, x, y)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "## 存在的问题\n", "'''\n", "Top n LinearSVM 和 MBGD LinearSVM 各有优劣，很难直接说谁好谁坏；\n", "但它们都有一个共同的问题，那就是它们所运用的梯度下降法都只是朴素的`Vanilla Update`，\n", "这会导致当数据的 scale 很大时模型对参数极为敏感、从而导致持续的震荡\n", "（所谓的 scale 比较大，可以理解为“规模很大”，或者直白一点——以二维数据为例的话——就是横纵坐标的数值很大）\n", "\n", "可以通过下面这个栗子来直观感受一下 scale 很大的数据所带来的问题：\n", "'''\n", "# 将 scale 从 1（默认）调成 5\n", "x, y = gen_two_clusters(center=5, scale=5)\n", "top_n_svm = LinearSVM2()\n", "top_n_svm.fit(x, y)\n", "print(\"Top n LinearSVM 准确率：{:8.6} %\".format((top_n_svm.predict(x) == y).mean() * 100))\n", "mbgd_svm = LinearSVM3()\n", "mbgd_svm.fit(x, y)\n", "print(\"MBGD LinearSVM 准确率：{:8.6} %\".format((mbgd_svm.predict(x) == y).mean() * 100))\n", "\n", "visualize2d(top_n_svm, x, y)\n", "visualize2d(mbgd_svm, x, y)" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "通过下面这张动图，我们能够直观地感受数据的 scale 很大时 LinearSVM 的训练过程：\n", "\n", "![Failure of LinearSVM][2]\n", "\n", "可以看到，模型确实一直在持续震荡\n", "\n", "[2]: https://cdn.rawgit.com/carefree0910/Resources/83441596/Lines/LinearSVM(Failed2).gif\n", "\n", "###### 解决方案\n", "\n", "+ 采用更好的梯度下降法，比如`Adam`之类的\n", "+ 进行数据预处理、把数据的 scale 弄回 1\n", "\n", "关于`Adam`等梯度下降算法的实现和在 LinearSVM 上的应用可以参见[这里](https://github.com/carefree0910/MachineLearning/blob/master/NN/Basic/Optimizers.py)和[这里](https://github.com/carefree0910/MachineLearning/blob/master/e_SVM/LinearSVM.py)，下面我们就仅展示进行数据预处理后的结果" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "'''\n", "可以看到在归一化处理后，\n", "即使是简单地采用极大梯度下降法，也能够在比较苛刻的数据（`center=5`、`dis=1`、`scale=5`）上表现得不错\n", "'''\n", "x, y = gen_two_clusters(center=5, dis=1, scale=5)\n", "# 进行归一化处理\n", "x -= x.mean(axis=0)\n", "x /= x.std(axis=0)\n", "# Top 1 梯度下降法即为极大梯度下降法\n", "top_1_svm = LinearSVM()\n", "top_1_svm.fit(x, y)\n", "print(\"Top 1 LinearSVM 准确率：{:8.6} %\".format((top_1_svm.predict(x) == y).mean() * 100))\n", "top_n_svm = LinearSVM2()\n", "top_n_svm.fit(x, y)\n", "print(\"Top n LinearSVM 准确率：{:8.6} %\".format((top_n_svm.predict(x) == y).mean() * 100))\n", "mbgd_svm = LinearSVM3()\n", "mbgd_svm.fit(x, y)\n", "print(\"MBGD LinearSVM 准确率：{:8.6} %\".format((mbgd_svm.predict(x) == y).mean() * 100))\n", "\n", "visualize2d(top_1_svm, x, y)\n", "visualize2d(top_n_svm, x, y)\n", "visualize2d(mbgd_svm, x, y)\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "#### 核感知机" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### 什么是核方法？\n", "\n", "往简单里说，核方法是将一个低维的线性不可分的数据映射到一个高维的空间、并期望映射后的数据在高维空间里是线性可分的。我们以异或数据集为例：在二维空间中、异或数据集是线性不可分的；但是通过将其映射到三维空间、我们可以非常简单地让其在三维空间中变得线性可分。比如定义映射：\n", "$$\n", "\\phi(x,y)=\\left\\{\n", "\\begin{aligned}\n", "&(x,y,1),\\ \\ xy>0 \\\\\n", "&(x,y,0),\\ \\ xy\\le0\n", "\\end{aligned}\n", "\\right.\n", "$$\n", "该映射的效果如下图所示：\n", "\n", "![](svm-kernel1.png)\n", " \n", "可以看到，虽然左图的数据集线性不可分、但显然右图的数据集是线性可分的，这就是核工作原理的一个不太严谨但仍然合理的解释\n", "\n", "从直观上来说，确实容易想象、同一份数据在越高维的空间中越有可能线性可分，但从理论上是否确实如此呢？1965 年提出的 Cover 定理从理论上解决了这个问题，我们会在文末附上相应的公式，这里暂时按下不表\n", "\n", "至此，似乎问题就转化为了如何寻找合适的映射$\\phi$、使得数据集在被它映射到高维空间后变得线性可分。不过可以想象的是，现实任务中的数据集要比上文我们拿来举例的异或数据集要复杂得多、直接构造一个恰当的$\\phi$的难度甚至可能高于解决问题本身。而核方法的巧妙之处就在于，它能将构造映射这个过程再次进行转化、从而使得问题变得简易：它通过核函数来避免显式定义映射$\\phi$。往简单里说，核方法会通过用能够表示成$K(x_i,x_j)=\\phi(x_i)\\cdot\\phi(x_j)$的核函数$K(x_i,x_j)$替换各算式中出现的内积$x_i\\cdot x_j$来完成将数据从低维映射到高维的过程。换句话说、核方法的思想如下：\n", "\n", "- 将算法表述成样本点内积的组合（这经常能通过算法的对偶形式实现）\n", "\n", "- 设法找到核函数$K(x_i,x_j)$，它能返回样本点$x_i、x_j$被$\\phi$作用后的内积\n", "\n", "- 用$K(x_i,x_j)$替换$x_i\\cdot x_j$、完成低维到高维的映射（同时也完成了从线性算法到非线性算法的转换）\n", "\n", "当然了，不难想象的是，并不是所有的函数$K$都能够对应一个映射（亦即不是所有的$K(x_i,x_j)$都能拆成$\\phi(x_i)\\cdot\\phi(x_j)$；比如说，显然$K(x_i,x_j)$至少需要是一个对称函数）。幸运的是，1909 年提出的 Mercer 定理解决了这个问题，它的具体叙述会在文末给出\n", "\n", "Mercer 定理为寻找核函数带来了极大的便利。可以证明如下两族函数都是核函数：\n", "\n", "- 多项式核$K(x_i,x_j)=(x_i\\cdot x_j+1)^p$\n", "- 径向基（Radial Basis Function，常简称为 RBF）核：$K(x_i,x_j)=\\exp\\left(-\\gamma\\|x_i-x_j\\|^2\\right)$\n", "\n", "那么核方法的应用场景有哪些呢？在 2002 年由 Scholkopf 和 Smola 证明的表示定理告诉我们它的应用场景非常广泛。定理的具体内容同样会附在文末，这里就暂时按下不表\n", "\n", "##### 核模型的表现\n", "\n", "还是用 GIF 来说明问题最为形象。当我们对感知机应用核方法后，它就能对非线性数据集（比如螺旋线数据集）进行分类了，训练过程将如下：\n", "\n", "![](svm-kernel2.gif)\n", "\n", "##### 怎么应用核方法？\n", "\n", "简单来说，就是把算法中涉及到样本$（x_i）$的地方都通过某种变换、弄成样本的内积形式$（x_i\\cdot x_j）$。\n", "\n", "以感知机为例，感知机的原始损失函数为$L(D) = \\sum_{i=1}^N\\left[ -y_i(w\\cdot x_i+b)\\right]_+$\n", "\n", "为了让损失函数中的样本都变成内积形式，考虑令\n", "$$\n", "w = \\sum_{i=1}^N\\alpha_ix_i（也有令w = \\sum_{i=1}^N\\alpha_iy_ix_i的）\n", "$$\n", "则\n", "$$\n", "\\begin{align}\n", "L(D) &= \\sum_{i=1}^N\\left[ -y_i\\left[\\left(\\sum_{j=1}^N\\alpha_jx_j\\right)\\cdot x_i+b\\right]\\right]_+ \\\\\n", "&= \\sum_{i=1}^N\\left[ -y_i\\left(\\sum_{j=1}^N\\alpha_j(x_i\\cdot x_j)+b\\right)\\right]_+\n", "\\end{align}\n", "$$\n", "在此之上应用核方法是平凡的：设核函数为$K$，只需把所有的$x_i\\cdot x_j$换成$K(x_i,x_j)$即可：\n", "$$\n", "L(D) = \\sum_{i=1}^N\\left[ -y_i\\left(\\sum_{j=1}^N\\alpha_jK(x_i,x_j)+b\\right)\\right]_+\n", "$$\n", "于是优化问题变为\n", "$$\n", "\\min_{\\alpha}\\sum_{i=1}^N\\left[ -y_i\\left(\\sum_{j=1}^N\\alpha_jK(x_i,x_j)+b\\right)\\right]_+\n", "$$\n", "预测步骤则变为\n", "$$\n", "y_{\\text{pred}}=w\\cdot x+b=\\sum_{i=1}^N\\alpha_iK(x_i, x)+b\n", "$$\n", "对于 LinearSVM 而言，用同样的手法不难得出其核形式：\n", "$$\n", "L(D)=\\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jK(x_i,x_j)+C\\sum_{i=1}^N\\left[ 1-y_i\\left(\\sum_{j=1}^N\\alpha_jK(x_i,x_j)+b\\right)\\right]_+\n", "$$\n", "预测步骤则仍然是\n", "$$\n", "y_{\\text{pred}}=w\\cdot x+b=\\sum_{i=1}^N\\alpha_iK(x_i, x)+b\n", "$$\n", "（有没有发现核形式和对偶形式很像？( σ'ω')σ）\n", "\n", "##### 如何训练核模型？\n", "\n", "【注意：为简洁，从此往后的推导和实现均以核感知机为例，核 SVM 的相关讨论会放在下一章介绍 SMO 算法时进行】\n", "\n", "简洁起见，我们还是用梯度下降法来进行训练，为此我们需要进行求导工作。假设当前模型参数为$\\alpha=(\\alpha_1,\\alpha_2,...,\\alpha_N)^T$，$x_i$在参数$\\alpha$下的预测值为$\\hat y_i$，则：\n", "\n", "- $\\frac{\\partial L}{\\partial\\alpha_i}=-\\sum_{y_j\\hat y_j<0}y_jK(x_j, x_i)$\n", "- $\\frac{\\partial L}{\\partial b}=-\\sum_{y_j\\hat y_j<0}y_j$\n", "\n", "为了加速训练，我们需要将该算式向量化，为此我们需要定义核矩阵。假设现在我们有两组样本：$(x^{(1)}_1,x^{(2)}_2,...,x^{(1)}_M)^T$和$(x^{(2)}_1, x^{(2)}_2,...,x^{(2)}_N)^T$，那么它们的核矩阵即为\n", "$$\n", "\\textbf{ K} = \\left[\\begin{matrix}\n", "K(x^{(1)}_1,x^{(2)}_1) & \\ldots & K(x^{(1)}_1,x^{(2)}_N) \\\\\n", "\\vdots & \\ddots & \\vdots \\\\\n", "K(x^{(1)}_M,x^{(2)}_1) & \\ldots & K(x^{(1)}_M,x^{(2)}_N)\n", "\\end{matrix}\\right]_{N\\times N}\n", "$$\n", "对于训练过程而言，我们关心的是训练样本之间的核矩阵\n", "$$\n", "\\textbf{ K} = \\left[\\begin{matrix}\n", "K(x_1,x_1) & \\ldots & K(x_1,x_N) \\\\\n", "\\vdots & \\ddots & \\vdots \\\\\n", "K(x_N,x_1) & \\ldots & K(x_N,x_N)\n", "\\end{matrix}\\right]_{N\\times N}\n", "$$\n", "利用它，不难写出相应的向量化代码：\n", "```python\n", "# 假设 k_mat 存储着原样本之间的核矩阵\n", "# 1、计算损失\n", "err = -y * (k_mat.dot(alpha) + b)\n", "# 2、找出使得损失不小于 0 的样本\n", "mask = err >= 0\n", "# 3、进行相应梯度下降，lr 是学习速率\n", "delta = lr * y[mask]\n", "alpha += np.sum(delta[..., None] * k_mat[mask], axis=0)\n", "b += np.sum(delta)\n", "```" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "对于预测过程，我们关心的是原样本和新样本之间的核矩阵。\n", "\n", "假设新样本为$(\\tilde x_1,...,\\tilde x_n)^T$，则\n", "$$\n", "\\textbf{K} = \\left[\\begin{matrix}\n", "K(x_1,\\tilde x_1) & \\ldots & K(x_1,\\tilde x_n) \\\\\n", "\\vdots & \\ddots & \\vdots \\\\\n", "K(x_N,\\tilde x_1) & \\ldots & K(x_N,\\tilde x_n)\n", "\\end{matrix}\\right]_{N\\times n}\n", "$$\n", "\n", "那么预测过程即为\n", "\n", "$$\n", "y_{\\text{pred}}=\\sum_{i=1}^N\\alpha_iK(x_i,x)+b=\\alpha^T\\textbf{K}+b\n", "$$\n", "\n", "于是关键就在于如何定义计算核矩阵的核函数了。对于多项式核来说，核函数的实现是直观的：\n", "\n", "```python\n", "@staticmethod\n", "def _poly(x, y, p):\n", " return (x.dot(y.T) + 1) ** p\n", "```\n", "但对于 RBF 来说就没那么直观了，用到了 Numpy 的高级实用技巧之一——升维：\n", "```python\n", "@staticmethod\n", "def _rbf(x, y, gamma):\n", " return np.exp(-gamma * np.sum((x[..., None, :] - y) ** 2, axis=2))\n", "```\n", "当然直接用 for 来实现也是可以的，不过那将会非常非常慢……" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### 核模型的实现\n", "\n", "如果思路能够整理清楚，那么核模型相比原模型来说只有如下两点改变：\n", "\n", "- 需要定义核函数并计算出核矩阵\n", "- 计算预测值时不是$w\\cdot x+b=w^Tx+b$，而是$\\alpha^T\\textbf{K}+b$，其中\n", " - 在训练时，\\textbf{K}为原样本之间的核矩阵\n", " - 在测试时，\\textbf{K}为原样本和新样本的核矩阵\n", "\n", "所以实现起来的话会有许多重复代码，这里就只展现其中最核心的部分（仍以核感知机为例）：\n", "```python\n", "# 训练代码\n", "def fit(...):\n", " ...\n", " self._alpha = np.zeros(len(x))\n", " self._b = 0.\n", " self._x = x\n", " # self._kernel 即为核函数，能够计算两组样本的核矩阵\n", " k_mat = self._kernel(x, x)\n", " for _ in range(epoch):\n", " err = -y * (self._alpha.dot(k_mat) + self._b)\n", " if np.max(err) < 0:\n", " continue\n", " mask = err >= 0\n", " delta = lr * y[mask]\n", " self._alpha += np.sum(delta[..., None] * k_mat[mask], axis=0)\n", " self._b += np.sum(delta)\n", "\n", "# 预测代码\n", "def predict(self, x, raw=False):\n", " x = np.atleast_2d(x).astype(np.float32)\n", " # 计算原样本与新样本的核矩阵并根据它来计算预测值\n", " k_mat = self._kernel(self._x, x)\n", " y_pred = self._alpha.dot(k_mat) + self._b\n", " if raw:\n", " return y_pred\n", " return np.sign(y_pred).astype(np.float32)\n", "```" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### 相关数学理论\n", "\n", "###### 1）Cover 定理\n", "\n", "若设 d 维空间中 N 个点线性可分的概率为$p(d,N)$，那么就有：\n", "$$\n", "p(d,N)=\\frac{2\\sum_{i=0}^mC_{N-1}^i}{2^N}=\\left\\{\n", "\\begin{aligned}\n", "&\\frac{\\sum_{i=1}^dC^i_{N-1}}{2^{N-1}},\\ \\ &N>d+1 \n", "\\\\\n", "&1,\\ \\ &N\\le d+1\n", "\\end{aligned}\n", "\\right.\n", "$$\n", "其中$m=\\min(d,N-1)$\n", "\n", "证明从略（也就是说我不会）（喂），但是不难从中看出，它证明了当空间的维数 d 越大时、其中的 N 个点线性可分的概率就越大，这构成了核方法的理论基础之一\n", "\n", "###### 2）Mercer 定理\n", "\n", "若$K(x_i,x_j)$是对称函数（亦即$K(x_i,x_j)=K(x_j,x_i)）$的话，那么它具有 Hilbert 空间中内积形式的充要条件有以下两个：\n", "\n", "对任何平方可积的函数g、满足\n", "$$\\int{K(x_i,x_j)g(x_i)g(x_j)dx_idx_j}\\ge0$$\n", "对含任意 N 个样本的数据集$D={x_1,x_2,...,x_N}$，核矩阵：\n", "$$\n", "\\textbf{K} = \\left[\\begin{matrix}\n", "K(x_1,x_1) & \\ldots & K(x_1,x_N) \\\\\n", "\\vdots & \\ddots & \\vdots \\\\\n", "K(x_N,x_1) & \\ldots & K(x_N,x_N)\n", "\\end{matrix}\\right]_{N\\times N}\n", "$$\n", "是半正定矩阵\n", "\n", "【注意：通常我们会称满足这两个充要条件之一的函数为 Mercer 核函数而把核函数定义得更宽泛。不过如果不打算在理论上深入太多的话，将 Mercer 核函数简称为核函数是可以的。此外，虽说 Mercer 核函数确实具有 Hilbert 空间中的内积形式、但此时的 Hilbert 空间并不一定具有“维度”这么好的概念（或说、可以认为此时 Hilbert 空间的维度为无穷大；比如说 RBF 核，它映射后的空间就是无穷维的）】\n", "\n", "###### 3）表示定理\n", "\n", "设$\\mathcal{H}$为核函数K对应的映射后的空间（RKHS），$\\|h\\|_\\mathcal{H}$表示$\\mathcal{H}$中h的范数，那么对于任意单调递增的函数C和任意非负损失函数L、优化问题\n", "\n", "$$\\min_{h\\in\\mathcal{H}}L\\left(h(x_1),...,h(x_N)\\right)+C(\\|h\\|_{\\mathcal{H}})$$\n", "\n", "的解总可以表述为核函数K的线性组合\n", "$$\n", "h^*(x)=\\sum_{i=1}^N\\alpha_iK(x,x_i)\n", "$$\n", "这意味着对于任意一个损失函数和一个单调递增的正则化项组成的优化问题、我们都能够对其应用核方法" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "hidden": true }, "outputs": [], "source": [ "# 核感知机\n", "import numpy as np\n", "\n", "class KP:\n", " def __init__(self):\n", " self._x = None\n", " self._alpha = self._b = self._kernel = None\n", " \n", " # 定义多项式核函数\n", " @staticmethod\n", " def _poly(x, y, p=4):\n", " return (x.dot(y.T) + 1) ** p\n", " \n", " # 定义 rbf 核函数\n", " @staticmethod\n", " def _rbf(x, y, gamma):\n", " return np.exp(-gamma * np.sum((x[..., None, :] - y) ** 2, axis=2))\n", " \n", " def fit(self, x, y, kernel=\"poly\", p=None, gamma=None, lr=0.001, batch_size=128, epoch=10000):\n", " x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)\n", " if kernel == \"poly\":\n", " p = 4 if p is None else p\n", " self._kernel = lambda x_, y_: self._poly(x_, y_, p)\n", " elif kernel == \"rbf\":\n", " gamma = 1 / x.shape[1] if gamma is None else gamma\n", " self._kernel = lambda x_, y_: self._rbf(x_, y_, gamma)\n", " else:\n", " raise NotImplementedError(\"Kernel '{}' has not defined\".format(kernel))\n", " self._alpha = np.zeros(len(x))\n", " self._b = 0.\n", " self._x = x\n", " k_mat = self._kernel(x, x)\n", " for _ in range(epoch):\n", " indices = np.random.permutation(len(y))[:batch_size]\n", " k_mat_batch, y_batch = k_mat[indices], y[indices]\n", " err = -y_batch * (k_mat_batch.dot(self._alpha) + self._b)\n", " if np.max(err) < 0:\n", " continue\n", " mask = err >= 0\n", " delta = lr * y_batch[mask]\n", " self._alpha += np.sum(delta[..., None] * k_mat_batch[mask], axis=0)\n", " self._b += np.sum(delta)\n", " \n", " def predict(self, x, raw=False):\n", " x = np.atleast_2d(x).astype(np.float32)\n", " k_mat = self._kernel(self._x, x)\n", " y_pred = self._alpha.dot(k_mat) + self._b\n", " if raw:\n", " return y_pred\n", " return np.sign(y_pred).astype(np.float32)\n", "\n", "# 在线性数据集上进行测试 \n", "from Util import gen_two_clusters, visualize2d\n", "\n", "xc, yc = gen_two_clusters()\n", "kp = KP()\n", "kp.fit(xc, yc, p=1)\n", "print(\"准确率：{:8.6} %\".format((kp.predict(xc) == yc).mean() * 100))\n", "visualize2d(kp, xc, yc, True)\n", "# 在非线性数据集上进行测试\n", "from Util import gen_spiral\n", "\n", "xs, ys = gen_spiral()\n", "kp = KP()\n", "kp.fit(xs, ys, p=8)\n", "print(\"准确率：{:8.6} %\".format((kp.predict(xs) == ys).mean() * 100))\n", "visualize2d(kp, xs, ys, True)" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "#### 核SVM" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "###### SMO 算法概述\n", "\n", "SMO 是由 Platt 在 1998 年提出的、针对软间隔最大化 SVM 对偶问题求解的一个算法，其基本思想很简单：在每一步优化中，挑选出诸多参数$（\\alpha_k(k=1,2,...,N)）$中的两个参数$（\\alpha_i、\\alpha_j）$作为“真正的参数”，其余参数都视为常数，从而问题就变成了类似于二次方程求最大值的问题，从而我们就能求出解析解\n", "\n", "具体而言，SMO 要解决的是如下对偶问题：\n", "$$\n", "\\max_\\alpha L(\\alpha)=-\\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jy_iy_j(x_i\\cdot x_j) + \\sum_{i=1}^N\\alpha_i\n", "$$\n", "使得对$i=1,...,N$、都有\n", "$$\n", "\\sum_{i=1}^N\\alpha_iy_i=0、0\\le\\alpha_i\\le C\n", "$$\n", "其大致求解步骤则可以概括如下：\n", "\n", "选出$\\alpha_1,\\alpha_2,...,\\alpha_N$中“最不好的”两个参数$\\alpha_i、\\alpha_j$\n", "只把$\\alpha_i、\\alpha_j$视为参数并把其余的$\\alpha_k$视为常数，于是最大化$L(\\alpha)$就变成了以$\\alpha_i、\\alpha_j$为参数的二次规划问题，从而可以直接对其进行求解；但是，注意到$\\alpha_i、\\alpha_j$需满足$\\sum_{i=1}^N\\alpha_iy_i=0$和$0\\le\\alpha_i、\\alpha_j\\le C$，所以求完解后需要检查是否满足约束；如不满足，则进行调整\n", "\n", "##### KKT 条件\n", "\n", "先来看如何选取参数。在 SMO 算法中，我们是依次选取参数的：\n", "\n", "- 选出违反 KKT 条件最严重的样本点、以其对应的参数作为第一个参数\n", "- 第二个参数的选取有一种比较繁复且高效的方法，但对于一个朴素的实现而言、第二个参数即使随机选取也无不可\n", "\n", "这里就有了一个叫 KKT 条件的东西，其详细的陈列会放在文末，这里就仅简要的说明一下。\n", "\n", "具体而言，对于已有的模型$f(x)=w\\cdot x + b$来说，$\\alpha_i$及其对应样本$(x_i,y_i)$的 KKT 条件为：\n", "$$\n", "\\begin{align}\n", "\\alpha_i=0&\\Leftrightarrow y_if(x_i)>1 \\\\\n", "0<\\alpha_i1\\Leftrightarrow c^{(1)}_i>0$而言，满足以下两种情况的$c^{(1)}_i$将应该置为 0：\n", " - $\\alpha_i>0且c^{(1)}_i\\le0$\n", " - $\\alpha_i=0且c^{(1)}_i>0$\n", " - 对第二个 KKT 条件$0<\\alpha_i 0 和 alpha < C 的 mask\n", "con1 = alpha > 0\n", "con2 = alpha < C\n", "# 算出“差异向量”并拷贝成三份\n", "err1 = y * y_pred - 1\n", "err2 = err1.copy()\n", "err3 = err1.copy()\n", "# 依次根据三个 KKT 条件，将差异向量的某些位置设为 0\n", "# 不难看出为了直观、我做了不少重复的运算，所以这一步是可以优化的\n", "err1[(con1 & (err1 <= 0)) | (~con1 & (err1 > 0))] = 0\n", "err2[((~con1 | ~con2) & (err2 != 0)) | ((con1 & con2) & (err2 == 0))] = 0\n", "err3[(con2 & (err3 >= 0)) | (~con2 & (err3 < 0))] = 0\n", "# 算出平方和并取出使得“损失”最大的 idx\n", "err = err1 ** 2 + err2 ** 2 + err3 ** 2\n", "idx = np.argmax(err)\n", "\n", "```\n", "第二个参数则可以简单地随机选取，虽然这不是特别好，但效果已然不错，而且不仅实现起来更简便、运行起来也更快（其实就是我太懒）（喂）。具体代码如下：\n", "\n", "```python\n", "idx = np.random.randint(len(self._y))\n", "# 这里的 idx1 是第一个参数对应的 idx\n", "while idx == idx1:\n", " idx = np.random.randint(len(self._y))\n", "return idx\n", "```\n", "至于 SMO 算法的第二步，正如前文所说，它的本质就是一个带约束的二次规划，虽然求解过程可能会比较折腾，但其实难度不大。具体步骤会放在文末，这里就暂时按下" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "##### SMO 的效果\n", "\n", "![](svm-kernel4.gif)\n", "\n", "略显纠结，不过还是不错的\n", "\n", "接下来看看蘑菇数据集上的表现；单就这个数据集而言，我们实现的朴素 SVM 和 sklearn 中的 SVM 表现几乎是一致的（在使用 RBF 核时），比较具有代表性的训练曲线则如下图所示：\n", "\n", "![](svm-kernel5.png)\n", "\n", "也算是符合 SMO 这种每次只取两个参数进行更新的训练方法的直观\n", "\n", "##### 相关数学理论\n", "\n", "###### 1）KKT 条件的详细陈列\n", "\n", "注意到原始问题为\n", "$$\n", "\\min_{w,b,\\xi}L(w,b,\\xi)=\\frac12\\|w\\|^2+C\\sum_{i=1}^N\\xi_i，使得\\xi^*\\ge0、y_i(w\\cdot x_i+b)\\ge1-\\xi_i（不妨称这两个约束为原始约束）\n", "$$\n", "所以其拉格朗日算子法对应的式子为\n", "$$\n", "L=L(w,b,\\xi,\\alpha,\\beta)=\\frac12\\|w\\|^2+C\\sum_{i=1}^N\\xi_i-\\sum_{i=1}^N\\alpha_i[y_i(w\\cdot x_i+b)-1+\\xi_i]-\\sum_{i=1}^N\\beta_i\\xi_i\n", "$$\n", "于是 KKT 条件的其中四个约束即为（不妨设最优解为$w^*、b^*、\\xi^*、\\alpha^*和\\beta^*$）：\n", "\n", "- $ \\alpha_i^*\\ge0,\\beta_i^*\\ge0 $（这是拉格朗日乘子法自身的要求）\n", "- $ \\xi_i^*\\ge0、y_i(w^*\\cdot x_i+b^*)-1+\\xi_i^*\\ge0 $（此即原始约束）\n", "- $ \\alpha_i^*[y_i(w^*\\cdot x_i+b^*)-1+\\xi_i^*]=0 $（换句话说，$ \\alpha_i^*和y_i(w^*\\cdot x_i+b)-1+\\xi_i^* $中必有一个为 0）\n", " - 该等式有着很好的直观：设想它们同时不为 0，则必有$ y_i(w^*\\cdot x_i+b)-1+\\xi_i^*>0 $（注意原始约束）、从而$\\alpha_i^*[y_i(w^*\\cdot x_i+b^*)-1+\\xi_i^*]\\ge0 $，等号当且仅当$ \\alpha_i=0 $时取得。然而由于$\\alpha_i^*\\ne0$，所以若将$\\alpha_i$取为 0、则上述L将会变大。换句话说，将参数$\\alpha_i$取为 0 将会使得目标函数比参数取$\\alpha_i^*$时的目标函数要大，这与$\\alpha_i^*$的最优性矛盾\n", "- $\\beta_i^*\\xi_i^*=0$ （换句话说，$\\beta_i^*$ 和$ \\xi_i^*$ 中必有一个为 0，理由同上）\n", "\n", "从而原始问题转为$\\min_{w,b}\\max_\\alpha L$；而对偶问题的实质，其实就是将原始问题$\\min_{w,b}\\max_\\alpha L转为\\max_\\alpha\\min_{w,b} L$。在求解$\\nabla_wL=\\nabla_bL=\\nabla_\\xi L=0$后，可以得到如下对偶问题：\n", "$$\n", "\\max_\\alpha L(\\alpha)=-\\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jy_iy_j(x_i\\cdot x_j) + \\sum_{i=1}^N\\alpha_i，使得对i=1,...,N、都有\\sum_{i=1}^N\\alpha_iy_i=0、0\\le\\alpha_i\\le C\n", "$$\n", "（虽然这些在 Python · SVM（二）· LinearSVM 中介绍过，不过为了连贯性，这里还是再介绍一遍）\n", "\n", "于是，最优解自然需要满足这么个条件：\n", "$$\n", "\\nabla_wL(w^*,b^*,\\xi^*,\\alpha^*,\\beta^*)=\\nabla_bL(w^*,b^*,\\xi^*,\\alpha^*,\\beta^*)=\\nabla_\\xi L(w^*,b^*,\\xi^*,\\alpha^*,\\beta^*)=0\n", "$$\n", "这个条件即是最后一个 KKT 条件\n", "\n", "###### 2）何谓“支持向量”\n", "\n", "为方便说明，这里再次放出上文给出过的图：\n", "\n", "![](svm-kernel3.png)\n", "\n", "图中带黑圈的样本点即是支持向量，数学上来说的话，就是$\\alpha_i>0$对应的样本点即是支持向量。从图中不难看出，支持向量从直观上来说，就是比较难分的样本点\n", "\n", "此外，支持向量之所以称之为“支持”向量，是因为在理想情况下，仅利用支持向量训练出来的模型和利用所有样本训练出来的模型是一致的。这从直观上是好理解的，粗略的证明则可以利用其定义来完成：非支持向量的样本对应着$\\alpha_i=0$，亦即它对最终模型——$f(x)=\\sum_{i=1}^N\\alpha_iy_i(x_i\\cdot x)+b$没有丝毫贡献，所以可以直接扔掉\n", "\n", "###### 3）带约束的二次规划求解方法\n", "\n", "不妨设我们选取出来的两个参数就是$\\alpha_1和\\alpha_2$，那么问题的关键就在于如何把$\\alpha_1和\\alpha_2$相关的东西抽取出来并把其它东西扔掉\n", "\n", "注意到我们的对偶问题为\n", "$$\n", "\\max_\\alpha L(\\alpha)=-\\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jy_iy_j(x_i\\cdot x_j) + \\sum_{i=1}^N\\alpha_i，使得对i=1,...,N、都有\\sum_{i=1}^N\\alpha_iy_i=0、0\\le\\alpha_i\\le C\n", "$$\n", "且我们在 Python · SVM（一）· 感知机 的最后介绍过 Gram 矩阵：\n", "$$\n", "G=(x_i\\cdot x_j)_{N\\times N}\n", "$$\n", "所以L就可以改写为$L(\\alpha)=-\\frac12\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_i\\alpha_jy_iy_jG_{ij}+\\sum_{i=1}^N\\alpha_i$\n", "\n", "把和$\\alpha_1、\\alpha_2$无关的东西扔掉之后，L可以化简为：\n", "$$\n", "L(\\alpha)=-\\frac12(G_{11}\\alpha_1^2+2y_1y_2G_{12}\\alpha_1\\alpha_2+G_{22}\\alpha_2^2)-\\left(y_1\\alpha_1\\sum_{i=3}^Ny_i\\alpha_iG_{i1}+y_2\\alpha_2\\sum_{i=3}^Ny_i\\alpha_iG_{i2}\\right)+(\\alpha_1+\\alpha_2)\n", "$$\n", "约束条件则可以化简为对i=1和i=2，都有$y_1\\alpha_1+y_2\\alpha_2=-\\sum_{i=3}^Ny_i\\alpha_i=c、0\\le\\alpha_i\\le C$，其中c是某个常数\n", "\n", "而带约束的二次规划求解过程也算简单：只需先求出无约束下的最优解，然后根据约束“裁剪”该最优解即可\n", "\n", "无约束下的求解过程其实就是求偏导并令其为 0。以\\alpha_1为例，注意到\n", "$$\n", "y_1\\alpha_1+y_2\\alpha_2=c\\Rightarrow\\alpha_2=\\frac c{y_2}-\\frac{y_1}{y_2}\\alpha_1\n", "$$\n", "令$c^*=\\frac c{y_2},\\ \\ s=y_1y_2$，则$c^*$亦是常数，且由于$y_1、y_2$都只能取正负 1，故不难发现$\\frac{y_2}{y_1}=\\frac{y_1}{y_2}=s$，从而$\\alpha_2=c^*-s\\alpha_1\\Rightarrow\\frac{\\partial\\alpha_2}{\\partial\\alpha_1}=-s$\n", "\n", "于是\n", "$$\n", "\\begin{align}\n", "\\frac{\\partial L}{\\partial\\alpha_1}=&-G_{11}\\alpha_1-y_1y_2G_{12}(\\alpha_2+\\alpha_1\\frac{\\partial\\alpha_2}{\\partial\\alpha_1})-G_{22}\\alpha_2\\frac{\\partial\\alpha_2}{\\partial\\alpha_1} \\\\\n", "&-y_1\\sum_{i=3}^Ny_i\\alpha_iG_{i1}-y_2\\frac{\\partial\\alpha_2}{\\partial\\alpha_1}\\sum_{i=3}^Ny_i\\alpha_iG_{i2}+1 \\\\\n", "=&-G_{11}\\alpha_1-sG_{12}(c^*-s\\alpha_1-\\alpha_1\\cdot s)-G_{22}(c^*-s\\alpha_1)\\cdot(-s) \\\\\n", "&-y_1\\sum_{i=3}^Ny_i\\alpha_iG_{i1}+sy_2\\sum_{i=3}^Ny_i\\alpha_iG_{i2}+\\left(\\frac{\\partial\\alpha_2}{\\partial\\alpha_1}+1\\right)\n", "\\end{align}\n", "$$\n", "考虑到$s^2=1、sy_2=y_1$、Gram 矩阵是对称阵、且模型在第k个样本$x_k$处的输出为$f(x_k)=\\sum_{i=1}^N\\alpha_iy_i(x_i\\cdot x_k)+b=\\sum_{i=1}^N\\alpha_iy_iG_{ik}+b$，从而可知\n", "$$\n", "\\begin{align}\n", "\\frac{\\partial L}{\\partial\\alpha_1}=&-G_{11}\\alpha_1-sG_{12}c^*+2G_{12}\\alpha_1+sG_{22}c^*-G_{22}\\alpha_1 \\\\\n", "&-y_1[f(x_1)-y_1\\alpha_1G_{11}-y_2\\alpha_2G_{21}] \\\\\n", "&+y_1[f(x_2)-y_1\\alpha_1G_{12}-y_2\\alpha_2G_{22}] +(1-s)\n", "\\end{align}\n", "$$\n", "令$v_i=(f(x_i)-b)-y_1\\alpha_1G_{1i}-y_2\\alpha_2G_{2i}\\ \\ (i=1,2)$，则\n", "$$\n", "\\frac{\\partial L}{\\partial\\alpha_1}=-(G_{11}-2G_{12}+G_{22})\\alpha_1-sc^*(G_{12}-G_{22})-y_1(v_1-v_2)+(1-s)\n", "$$\n", "于是\n", "$$\n", "\\begin{align}\n", "\\frac{\\partial L}{\\partial\\alpha_1}=0\\Rightarrow\\alpha_1&=-\\frac{sc^*(G_{12}-G_{22})+y_1(v_1-v_2)-(1-s)}{G_{11}-2G_{12}+G_{22}} \\\\\n", "&=-\\frac{y_1[y_2c^*(G_{12}-G_{22})+(v_1-v_2)-(y_1-y_2)]}{G_{11}-2G_{12}+G_{22}}\n", "\\end{align}\n", "$$\n", "注意到$c^*=s\\alpha_1+\\alpha_2$，从而\n", "$$\n", "y_2c^*(G_{12}-G_{22})=y_2(s\\alpha_1+\\alpha_2)(G_{12}-G_{22})=(y_1\\alpha_1+y_2\\alpha_2)(G_{12}-G_{22})\n", "$$\n", "令$dG=G_{11}-2G_{12}+G_{22}、e_i=f(x_i)-y_i\\ \\ (i=1,2)$，则\n", "$$\n", "y_2c^*(G_{12}-G_{22})+(v_1-v_2)-(y_2+y_1)=... =e_1 - e_2 - y_1\\alpha_1dG\n", "$$\n", "从而\n", "$$\n", "\\alpha_1^{new,raw}=\\alpha_1^{old}-\\frac{y_1(e_1-e_2)}{dG}\n", "$$\n", "接下来就要对其进行裁剪了。注意到我们的约束为\n", "$$\n", "0\\le\\alpha_i\\le C、\\alpha_1y_1+\\alpha_2y_2为常数\n", "$$\n", "所以我们需要分情况讨论$\\alpha_1$的下、上界\n", "\n", "- 当$y_1,y_2$异号$（y_1y_2=-1）$时，可知$\\alpha_1-\\alpha_2$为常数、亦即$\\alpha_1^{new}-\\alpha_2^{new}=\\alpha_1^{old}-\\alpha_2^{old}\\Rightarrow\\alpha_2^{new}=\\alpha_1^{new}-(\\alpha_1^{old}-\\alpha_2^{old})$\n", " 结合0\\le\\alpha_2\\le C，可知：\n", " - $\\alpha_1^{new}不应小于\\alpha_1^{old}-\\alpha_2^{old}$，否则$\\alpha_2$将小于 0\n", " - $\\alpha_1^{new}不应大于C+\\alpha_1^{old}-\\alpha_2^{old}$，否则$\\alpha_2$将大于 C\n", "- 当$y_1,y_2$同号$（y_1y_2=1）$时，可知$\\alpha_1+\\alpha_2$为常数、亦即$\\alpha_1^{new}+\\alpha_2^{new}=\\alpha_1^{old}+\\alpha_2^{old}\\Rightarrow\\alpha_2^{new}=(\\alpha_1^{old}+\\alpha_2^{old}) - \\alpha_1^{new}$结合0\\le\\alpha_2\\le C，可知：\n", " - $\\alpha_1^{new}$不应小于$\\alpha_1^{old}+\\alpha_2^{old}-C$，否则$\\alpha_2$将大于 C\n", " - $\\alpha_1^{new}$不应大于$\\alpha_1^{old}+\\alpha_2^{old}$，否则$\\alpha_2$将小于 0\n", "\n", "综上可知\n", "\n", "- $\\alpha_1^{new}$的下界为$U=\\left\\{\\begin{aligned}\n", "\\max\\{0,\\alpha_1^{old}-\\alpha_2^{old}\\}\\ \\ &y_1y_2=-1 \\\\\n", "\\max\\{0,\\alpha_1^{old}+\\alpha_2^{old}-C\\}\\ \\ &y_1y_2=1\n", "\\end{aligned}\n", "\\right.$\n", "\n", "- $\\alpha_1^{new}$的上界为$V=\\left\\{\\begin{aligned}\n", "\\min\\{C,C+\\alpha_1^{old}-\\alpha_2^{old}\\}\\ \\ &y_1y_2=-1 \\\\\n", "\\max\\{C,\\alpha_1^{old}+\\alpha_2^{old}\\}\\ \\ &y_1y_2=1\n", "\\end{aligned}\n", "\\right.$\n", "\n", "那么直接做一个 clip 即可得到更新后的$\\alpha_1$：\n", "```python\n", "alpha1_new = np.clip(alpha1_new_raw, u, v)\n", "```\n", "注意由于我们要保持$\\alpha_1y_1+\\alpha_2y_2$为常数，所以（注意$\\frac{y_1}{y_2}=y_1y_2$）\n", "$$\n", "\\begin{align}\n", "\\alpha_2^{new}&=\\frac1{y_2}(\\alpha_1^{old}y_1+\\alpha_2^{old}y_2-\\alpha_1^{new}y_1) \\\\\n", "&=\\alpha_2^{old}+y_1y_2(\\alpha_1^{old}-\\alpha_1^{new})\n", "\\end{align}$$\n", "综上所述，我们就完成了一次参数的更新，之后就不断地更新直至满足停机条件即可\n", "\n", "此外，我在 Python · SVM（三）· 核方法 这篇文章中提到过，对 SVM 的对偶形式应用核方法会非常自然。表现在 SMO 上的话就是，我们可以通过简单粗暴地将核矩阵K代替 Gram 矩阵G来完成核方法的应用。直观地说，我们只需将上面所有出现过的G都换成K就行了\n", "\n", "至此，SVM 算法的介绍就大致告一段落了。我们从感知机出发，依次介绍了“极大梯度法”、MBGD（Batch 梯度下降）法、核方法和 SMO 算法；虽然都有点浅尝辄止的味道，但覆盖的东西……大概还是挺多的" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "hidden": true }, "outputs": [ { "ename": "NameError", "evalue": "name 'KP' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# 核SVM\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mclass\u001b[0m \u001b[0mSVM\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mKP\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"rbf\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgamma\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mc\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.0001\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m128\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat32\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat32\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mkernel\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"poly\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNameError\u001b[0m: name 'KP' is not defined" ] } ], "source": [ "# 核SVM\n", "class SVM(KP): \n", " def fit(self, x, y, kernel=\"rbf\", p=None, gamma=None, c=1, lr=0.0001, batch_size=128, epoch=10000):\n", " x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)\n", " if kernel == \"poly\":\n", " p = 4 if p is None else p\n", " self._kernel = lambda x_, y_: self._poly(x_, y_, p)\n", " elif kernel == \"rbf\":\n", " gamma = 1 / x.shape[1] if gamma is None else gamma\n", " self._kernel = lambda x_, y_: self._rbf(x_, y_, gamma)\n", " else:\n", " raise NotImplementedError(\"Kernel '{}' has not defined\".format(kernel))\n", " self._alpha = np.zeros(len(x))\n", " self._b = 0.\n", " self._x = x\n", " k_mat = self._kernel(x, x)\n", " k_mat_diag = np.diag(k_mat)\n", " for _ in range(epoch):\n", " self._alpha -= lr * (np.sum(self._alpha * k_mat, axis=1) + self._alpha * k_mat_diag) * 0.5\n", " indices = np.random.permutation(len(y))[:batch_size]\n", " k_mat_batch, y_batch = k_mat[indices], y[indices]\n", " err = 1 - y_batch * (k_mat_batch.dot(self._alpha) + self._b)\n", " if np.max(err) <= 0:\n", " continue\n", " mask = err > 0\n", " delta = c * lr * y_batch[mask]\n", " self._alpha += np.sum(delta[..., None] * k_mat_batch[mask], axis=0)\n", " self._b += np.sum(delta)\n", "\n", "# 在线性数据集上进行测试\n", "svm = SVM()\n", "svm.fit(xc, yc)\n", "print(\"准确率：{:8.6} %\".format((svm.predict(xc) == yc).mean() * 100))\n", "visualize2d(svm, xc, yc, True)\n", "# 在非线性数据集上进行测试\n", "svm = SVM()\n", "svm.fit(xs, ys)\n", "print(\"准确率：{:8.6} %\".format((svm.predict(xs) == ys).mean() * 100))\n", "visualize2d(svm, xs, ys, True)\n", "#和 sklearn 的 SVM 进行比较\n", "from sklearn.svm import SVC\n", "svm = SVC()\n", "svm.fit(xc, yc)\n", "visualize2d(svm, xc, yc, True)\n", "print(\"准确率：{:8.6} %\".format((svm.predict(xc) == yc).mean() * 100))\n", "svm.fit(xs, ys)\n", "visualize2d(svm, xs, ys, True)\n", "print(\"准确率：{:8.6} %\".format((svm.predict(xs) == ys).mean() * 100))" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n", " decision_function_shape=None, degree=3, gamma='auto', kernel='rbf',\n", " max_iter=-1, probability=False, random_state=None, shrinking=True,\n", " tol=0.001, verbose=False)\n", "[1]\n", "[[ 0. 0.]\n", " [ 1. 1.]]\n", "[0 1]\n", "[1 1]\n" ] } ], "source": [ "from sklearn import svm\n", "X = [[0, 0], [1, 1]]\n", "y = [0, 1]\n", "clf = svm.SVC()\n", "print(clf.fit(X, y)) \n", "print(clf.predict([[2., 2.]]))\n", "# get support vectors\n", "print(clf.support_vectors_)\n", "# get indices of support vectors\n", "print(clf.support_)\n", "# get number of support vectors for each class\n", "print(clf.n_support_) " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "hidden": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbwAAAEICAYAAAAk60G8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd0E0fXh5+RZEtyr4Axvffeey8JLSGU9AohvZHeSA9J\nvvRCSIWQ3hPS8yYkgRAIhI7pGIwxuPcuzffHLkayJFvGTZh5zvGxdPfqzt2d3f3tzM7OCiklCoVC\noVA0dAz1nYBCoVAoFHWBEjyFQqFQnBEowVMoFArFGYESPIVCoVCcESjBUygUCsUZgRI8hUKhUJwR\n1LngCSEWCSFW1HW5dZ2HEGKHEGKU/lkIId4RQmQIIdYLIYYLIXbXQpkthBC5QghjTcc+VfR82tR3\nHorTByGEFEK087DsQiHEz3Wdk0P5FR67QohWev6musyrvvCh87lX271SwRNCrBBCJAkhsoUQe4QQ\nV3nxmwuEEBv0k12SEOIHIcSwqqzA6Y6UsquUcpX+dRgwHmgmpRwgpfxLStmxumUIIeKFEOMcyjws\npQySUtqqG7um0PM5UNNxhRBhQoi3hRDHhBA5+r55V02XUxOUr6dyy2KFEKVCiLZuln0phHimGuXW\n+MlXCDFKj/llOXtP3b6qpspyh5TyfSnlhNoso5LynY7diuq2PjjTBLeqeNPCewJoJaUMAaYBjwoh\n+npyFkLcCjwPPA40BloArwLTq5/uaUtLIF5KmVffiTQgngOCgM5AKNq+ua9eMyqHNycdKWUi8D/g\n4nK/jQDOApbVTnaVU0H+KcBgIUSkg+1SYE/tZ6VoiNSZQEspvf4DOgJJwGwPy0OBXGBWBTEWASsc\nvn8KHAOygD+Brg7LzgJ2AjlAIrBQt0cBK4FMIB34CzB4KK8r8Ivudxy4py7yAOKBccCVQCFg07fN\nQ8Ao4IhD/ObAF2gnkjTgZd3eFvhNt6UC7wNh+rL3ADtQoMe9A2gFSMCk+zQFvtFz2wfMK1cPnwDL\n9fXaAfTzsA2d4uq2VcBV+ud2wB/6tksFPnbwk0A7/fO7wCvAd3qZ64C2Dr4TgN16nFf1mFd5yGk7\nMOMU870MWAO8rJe1CxhbzvcJYD2QDXwNRDgsn6Zvr0zdt7PDsnjgTmArUAR8WL6e3OR7AbC/nO1a\nYJPD906c3I9343AMAlbg/4BD+vqs1m2H9e2Qq/8NRrvIvU/3TdbrP7TcdrtS/+2fbnIdBRwBlgDX\n6TYj2nHxALDKwfcFIEHfhhuB4Q7LjMA9wH59X9gINHfYZxYAe/Vt/AogHOpudbn9y62vvvwKIA7I\nAH4CWnrYZ5YBt+mfY/W4J9avrb7dDTgcu1R8DF6qb8NU4N4Kzoee6q6snHL71jj98wBgg75tjwPP\n6vZTrfPL9brK0Ldnf7R9OBP9fFTZ+RzwQ9vfPwf89XLv0us4De18E+FpX6ts23kZz+QpVymld4KH\ndvLJ1wP+BwR58JsElFZUKK5CcwUQDJjRWoabHZYloR8kQDjQR//8BNoB56f/DcdhJ3f4fbAe4zbA\non8fWBd54LxzXobzQTqKkweNEdiC1mIJ1PMcpi9rh9YVagai9Z3ieXcHgLtK1/1f1WP2QhPUMQ7r\nX4gm5kZ9Xf7xUGcuOxPOAvIhcC/aDlmWv8MJyVHw0tAOVhOagH+kL4tCO3jP1ZfdBJTgWfDeRBOd\ny4H2Vcz3MrT99Ba93uagnWwiHHwTgW56nXzOyYO6A5Cn14sf2kluH+DvUCeb0S5irO7qycNJL6vc\ndlsL3Kx/DkQ7GV2ub5veaCeDLvryV/ScY/W6HIK2z7jbDlfo+bZBayF/AbxXbrst18u0usl1FJrg\nDQHW6baz0MTkKpwF7yIgUs/5NrQLSou+7HZgG9pFtAB6ApEO+8xKIAythygFmOThWKrId7q+rp31\nHO4D/vZQB1cA3+qfL0A7qX7ssOzr8sduJcfgG3q99kS78OnsoVxPdedUjptzylrgYv1zEDCogn3f\nmzpfgnbsTkA7L3wFNNLzSgZGVnQ+19f1O7Rj3Kgvuwn4B2imr9PrwIee9rXKtp2X8aoveA4n5mH6\nTuPnwedC4FglcRbhIDTlloXpSZ+4+jgMXA2ElPN7GO2qu10lZZ2Pw1VyXeaB94I3GO0grbCidN8Z\nOF/1l5VRvtLRTrg2INhh+RPAuw7r/6vDsi5AgYdyXXYmnAVkObAU7R5l+d+WF7w3HZadBezSP18C\nrHVYJtBO8p4Ez4rWQtiIJoz7gMle5nsZcBTnlsB6Tp5AVgFPlts2xWjHwP3AJw7LDGjiOMqhTq7w\ntC9UULdvAkv1z+318hrp3+cAf5Xzfx14UC+/AOjpZb39D7jW4XtHffuZHPzbVJDnKE7uu3v133+E\nduw7CZ6b32acyBOtlTrdg5/EWfw/Ae7ycCxV5PsDcGW5usrHTSsPrRWXofssQTveT6znMuDW8utf\nyTHYzMG2HpjrpsyK6s6pnPJloV3MPgRE1VCdxzosTwPmOHz/HP3iy02ei9B6kf4AXsT5mIrDueck\npqJ9rbJt52W8Cs+jXo/SlFLapJSr0dT1GgB9MEqu/nehvqGivO2PFUIYhRBPCiH2CyGy0SoUtKt9\ngJloJ8VDQog/hBCDdfvTaCe4n4UQByoYrNAc7UqtvvOoiObAISllqZu8GgshPhJCJOp5rXDIqTKa\nAulSyhwH2yG0K7YTHHP4nA9YTrEv/Q40gVqvj069ogLf8mUGOeSbcGKB1PboI56CSCkLpJSPSyn7\norUiPgE+1e99eUOiXsYJDuk5nCCh3DI/tG3fVP9+Ig+77hvr4bfesgyYJYSwoN3P+0lKmawvawkM\nFEJknvhDE5gmek4WvNjPdZzy1z+b0O63VzX/94DrgdHAl+UXCiEWCiHihBBZes6hnNx/Kzs2Pe0n\nVfFtCbzgsM3S0fbT2HK/R0q5H63l3gutp2YlcFQI0REYiXZCrwre5F/VunPkSrTehl1CiH+FEFMq\n8PWmzo87fC5w872i7T8I6IF2keh4TLUEvnTY/nFoF+GV7WsV1Wdl8SrkVB5LMKFdDSGlnCy1UXhB\nUsr30ZrZRWgtEW+4AK3bYRzawdBKtws9/r9SyuloTeuv0E5qSClzpJS3SSnboN1PuVUIMdZN/AS0\nZnx951ERCUALD0LzONpVS3epDRq66EROOtLNb05wFIgQQgQ72FqgtUaqyonBNgEOtiZlSUh5TEo5\nT0rZFO3K+FVPw8orIAntYgrQHuVw/F4RUspstG0VCLSuLF+dWL2ME7RA22YnaF5uWQlaN+JRtAPP\nMc/mOG/X8vVSUT2dYDXaCXk6Wj07DlZJAP6QUoY5/AVJKa/RcypEPya9KNcpf33dSnE+wXmTL2iC\ndy3wvZQy33GBEGI42oXQbCBcShmG1m17YpsneMi5JkkAri633axSyr89+P8BnIfWPZ2of78U7TbG\nZg+/8XZbuaOiusvDYf/VHzWKLitUyr1SyvPRzkmLgc+EEIEe8vGmzqvDz2i9R/8TQpQXs8nltr9F\n37Zlq1KFcryJVyEVCp4QopEQYq4QIkhvBU1E6yb8nzt/KWUW2o3rV4QQM4QQAUIIPyHEZCHEU25+\nEowmkGlolfu4Q9n+QnvmJlRKWYJ2f8euL5sihGinn2yy0FTe7ib+SiBGCHGzEMIshAgWQgyshzwq\nYj3ayf5JIUSgEMIihBjqkFcukCWEiEW77+HIcTwIupQyAfgbeEKP2QPtqrDKz8xIKVPQTugX6fvB\nFTgcpEKIWUKIE+KUgbYTV3U7fAd01/cbE3AdriJVhhDifiFEf71+LGj9+5nA7sry1WkE3Kjvn7PQ\n7vN877D8IiFEFyFEAFrX9WdSe9zjE+BsIcRYIYQf2r2pIrRt7QmP9XQC/cp4OdrJKwz41mHxSqCD\nEOJiPV8/fd076y3Mt4FnhRBN9fUdLIQwo3WV28uV/SFwixCitRAiCG1f/9hdD0NlSCkPorV+7nWz\nOBjtpJoCmIQQDwAhDsvfBB4RQrQXGj2E86jPmmAJcLcQoiuAECJUr2tP/IHWYv1T/75K/75aen7U\np9K69UQldbcHrcflbH0/uw/tvhX6ulwkhIjWY2TqZju1XOcVrMtTwAdooneiFb8EeEwI0VLPOVoI\nUZ3R+tWOV1kLT6J1Xx5BO5E9g9aX+43HH0j5f8CtaBWUgqbK16O1jMqzHK15nYg2CvKfcssvBuKF\n1p23AK0bB7R7HL+iicFa4FUp5e9ucslBG1wwFa2ZvBet+6VO86gI/UCaijZA5TDatp6jL34I6IMm\npt+h3Wx25AngPr2Jv9BN+PPRWqtH0bqcHpRS/lqV/ByYhya4aWgjXx1P8P2BdUKIXLT+/JtkFZ+9\nk1KmArOAp/QyuqCNQivy9BPgHU62usYDZ0spc73IF7QRou313z8GnCelTHNY/h7aPcdjaN1ON+p5\n7kZrgb2k/3YqMFVKWVzB6lVWTydYjnb1/bGUsmy99f14AjBXX9djaMJ44gS4EG0AyL9orcTFaKOF\n8/V1W6OXPQjtBPse2kn9IFoL44YKcqoQKeVqKeVRN4t+An5EO3Ef0stx7L56Fu3i4We0i8i30O7L\n1hhSyi/RtsVH+rG7HZhcwU/+QBPqE4K3Gu0C+E+Pv/C+bj3hqe6y0FrPb6Kdl/Jw7uKfBOzQj7kX\n0O5zFdRFnXtCSvkI2nn+V6HdWngB7XzwsxAiB+286q7B4S3VjndiRKFC4VMIIQxoB/iFVb2I8CL2\nZWgDWNxOhiC0h6dXSCnfrMlyFQpF/aLm0lT4DEKIiUKbQcWMNgJT4NraVigUilNCCZ7ClxiMNmLt\nRFfhDCllQf2mpFAoGgqqS1OhUCgUZwSqhadQKBSKM4IGPaN2cFiEjG7q1aNcCoXiNMBwfG99p3BG\nsD+9MFVKGV255+lFgxa86KbNeOz97yt3VCgUpwWBT4+v7xTOCKZ/uOtQ5V6nH6pLU6FQKBRnBErw\nFArFaYFq3SmqixI8hULh8yixU9QESvAUCoVPo8ROUVMowVMoFD6LEjtFTaIET6FQKBRnBA36sQSF\nQnF6olp2itpAtfAUCoVPocROUVsowVMoFD6DEjtFbaIET6FQ+ARK7BS1jRI8hUKhUJwRKMFTKBT1\njmrdKeoCNUpToVDUG0roFHWJauEpFIp6QYmdoq7xGcETQsQLIbYJITYLITa4WS6EEC8KIfYJIbYK\nIfrUR54KhaL6KLFT1Ae+1qU5WkqZ6mHZZKC9/jcQeE3/r1AoFApFpfhMC88LpgPLpcY/QJgQIqa+\nk1IoFFVDte4U9YUvCZ4EfhZCbBRCzHezPBZIcPh+RLc5IYSYL4TYIITYkJORXkupKhSKU0GJnaI+\n8aUuzWFSykQhRCPgFyHELinln1UNIqVcCiwFaNOlh6zpJBUKRdVRQqfwBXymhSelTNT/JwNfAgPK\nuSQCzR2+N9NtCoXCh1Fip/AVfELwhBCBQojgE5+BCcD2cm7fAJfoozUHAVlSyqQ6TlWhUCgUpym+\n0qXZGPhSCAFaTh9IKX8UQiwAkFIuAb4HzgL2AfnA5fWUq0Kh8BLVulP4Ej4heFLKA0BPN/YlDp8l\ncF1d5qVQKE4dJXYKX8MnujQVCkXDQomdwhfxiRaeQqFoGCihU/gyqoWnUCgUijMCJXgKhaJGUK07\nha+jBE+hUFQbJXaK0wEleAqFoloosVOcLijBUygUp4wSO8XphBI8hUKhUJwRqMcSFApFlVEtO8Xp\niGrhKRSKKqHETnG6ogRPoVB4jRI7xemMEjyFQuEVSuwUpztK8BQKhUJxRqAET6FQVIpq3SkaAmqU\npkKh8IgSOkVDQrXwFAqFW5TYKRoaPiN4QgijEGKTEGKlm2WXCSFShBCb9b+r6iNHheJMQYmdoiHi\nS12aNwFxQIiH5R9LKa+vw3wUCoVC0YDwiRaeEKIZcDbwZn3nolCc6ajWnaKh4hOCBzwP3AHYK/CZ\nKYTYKoT4TAjR3JOTEGK+EGKDEGJDTkZ6jSeqUDRklNgpGjL13qUphJgCJEspNwohRnlw+xb4UEpZ\nJIS4GlgGjHHnKKVcCiwFaNOlh6yFlBUNnMN749iyZhVmq5WB46cQGhFV3ynVOkroFGcCvtDCGwpM\nE0LEAx8BY4QQKxwdpJRpUsoi/eubQN+6TVFxJiCl5P3nHuHJ6y4iPeUY+7ZvZuG5o9j4x8/1nVqt\nosROcaZQ7y08KeXdwN0AegtvoZTyIkcfIUSMlDJJ/zoNbXCLQlGj7Fi/hg2rfubpz38jMDgUgP07\nNrP4+ot58ft1WKwB9ZyhQqGoDr7QwnOLEOJhIcQ0/euNQogdQogtwI3AZfWXmaKhsu5/3zHuvIvK\nxA6gbddetO7cnW3//FmPmdUeqnWnOJOo9xaeI1LKVcAq/fMDDvayVqBCUWtIEML1GlAIA8iGdztY\niZ3iTMNnW3gKRV0zYOxk/vf5CvJzc8ps8bu2c2DnFroNHF6PmdU8SuwUZyI+1cJTKOqTbgOH02PI\nKO6YNZbBE6aRl53JhlU/cdV9T2ENDKrv9GoEJXSKMxkleAqFjhCCS29/iBFTzmPzmt+IaNSEWdcu\nJDy6SX2nViMosVOc6SjBUyjK0bpzd1p37l7fadQoSuwUCnUPT6Fo8CixUyg0VAtP0aCRUrLrv3/Y\nrM+cMmTidJq0aF3fadUZSuwUipOoFp6iwSKl5I2Hb+eNR+7E32whNzOTBy6bzurvvqjv1BQKRT2g\nWniKBsvmNb+zf8dmnvjwJ8xWKwBjZl7AosvPpc+IcQQEe3oT1emPatkpFK6oFp6iwbLh958Yfc4F\nZWIH0KxNBzr07Mv29avrMbPaRYmdQuEeJXiKBovJZKKkqMjFXlJUhNHUMDs3lNgpFJ5RgqfwSex2\nO1vW/M6ypx/ko5ee5Mj+3VWOMXjiNH797D2yM9LKbLv+W0fCvl10b2Azp4ASO4WiMhrmZa7itMZu\nt/PqfTeSsG8Xw84+j7zsTB69ei5zrr+T0TPmeh2nU5+BjJg6i9tnjqHPiHHk5WSxa9N6bnjiFfwt\n1soDKBSKBoUSPIXPsemvX0k8sJdH3luJv9kCwMjpc7j/4qn0HzOJoJAwr2PNvPoWhp19LlvX/oHZ\nGsCCRc82yMEqqnWnUFSOEjyFz/Hfn78yasb5ZWIHENOiNR179Wf7utUMGj+lSvEaN2vJ+FmX1HSa\nPoESOoXCe9Q9PIXP4edvpqggz8VemJ/rJIJnOkrsFIqqoQRP4XMMmTSDXz5dTkbKsTLbtnV/cTR+\nP90GDquXnOw2G//+9gNvPHIn7z2ziAM7t9ZLHidQYqdQVB2f6tIUQhiBDUCilHJKuWVmYDnQF0gD\n5kgp4+s8SUWt06FnXybMvozbzxtHzyEjycvOIn7Xdm56akm9tPDsNhsv3HkNqUlHGDF1Nnk5mTxz\n8+Wcc9WNjJ99aZ3no1AoTg2fEjzgJiAOcDeq4EogQ0rZTggxF1gMzKnL5BR1x9TLrmHoWTPY9s9f\nmK0B9B421ukB8rpkw6qfSU06wkPvfoXJzx+A4WfP5O7zJzNowjSCw8LrNB/VulMoTg2f6dIUQjQD\nzgbe9OAyHVimf/4MGCuEEHWRm6J+iGgUw8hpsxk0fkq9iR3A5jW/MXLa7DKxA4hu2pxOvfuzo45n\nbFFip1CcOr7UwnseuAMI9rA8FkgAkFKWCiGygEgg1dFJCDEfmA8Q1SS21pJVnDmYrQHk52S72PNz\nsjEHBNZJDkroFIrq4xMtPCHEFCBZSrmxurGklEullP2klP2CwyNqIDvF6UBhQT7vPHkvt507ijtn\nj+eXT5fXWOxhZ53DL58uJ+XokTLbpr/+x7GEeLoNGFpj5XhCiZ1CUTP4SgtvKDBNCHEWYAFChBAr\npJQXOfgkAs2BI0IIExCKNnhFcYZTXFjIwnNHERIWyZRLFpCXlcFnr/0fW9f+wW3PvlXt+G279mLq\nZddy99yJdO43iLzsLI4dPsgtz7yBn7+5BtZAoVDUBT4heFLKu4G7AYQQo4CF5cQO4BvgUmAtcB7w\nm5RS1mWeCt/ko5efJCgkjEfe+7ZsUujBk2Zw64wRJOzfQ/O2HapdxqTzr2DIpOnsWL8GS0Ag3QYO\nqxOxU607haLm8IkuTU8IIR4WQkzTv74FRAoh9gG3AnfVX2YKX2L7ur8Ye95FTm9AiGwcQ6feA/j9\nyw9qrJyQ8EgGT5xG7+FjldgpFKchPtHCc0RKuQpYpX9+wMFeCMyqn6wUvoy/2UJWWqqLPTs9lU69\nB9RDRtVHiZ1CUfP4nOApziyy0lN59rZ5HE+IRwB9Rk5g3v2LqxTjrIvm8fbj9zDsrHNo0qI1AOt+\n/Y5jCfFMvmhelWKVFBex+vsv2bz6N8zWAEZMOa9OZ3dRQle7lNjsrIrPZuPRXMwmA2Nah9KzieeR\ntlX1V/g2SvAU9UZuTha3zRhJbJv2XHzbA+RmZ/H1my+y8Nz1PPPF717HGTJpBptX/86ds8fTuksP\n8rIzSU1K5Ip7HsdiDfA6TmlJCU/deBlIyagZc8jLzuLNx+5k1PS5zLjyhlNYQ4UvUWqXPPyHNtJ2\nXJtQcovtvPrvMca1CWVW16hq+yt8HyV4inrjxTsW0Lh5Sx5863MMRiMAA8eexU1Th7J9/Wq6DfC+\nZXXtoy8w69qF/PblhwQFhzJ+9qX4W6o2Ddm6X1dSWlzE/W98WpbPgLFnsXDmGEZNn0NYVKMqxasq\nqnVXu6w5nE2pXfLomBYYDdqcFUOaB3PddwcY1yaMcKupWv4K38enB60oGjYJ+/Yw+pzzy8QFICyq\nEZ37DGTl8terHC+6aXPmXHcHZ19ydZXFDmD7utUMPescl3y69htM3MZ/qhyvKiixq322HMtnRMuQ\nMvECCLea6N44gO3J+dX2V/g+SvAU9YbRaCQ9+ZiLPT35GOG13JpyR2BIKJkpyS72jJTjBIaE1l65\nSuzqhCB/AxmFpS729IJSgvyN1fZX+D5K8BT1xqQLruTHD97iyP7dAEgp+XPlZ6QkHuaShQ+5/U1x\nYQE/ffQui2+4hOfvWMDGP35BSunRXhVGTJnFr5+vcMrnr5Wfk5WeStf+tTOjihK7umNM61B+3JvJ\n4awiQKvf3w9mkVloo0dj13u9VfVX+D6qE1pRb0ycexlfv/MK9154NrFt2pOXnUV2RhodevXDGhTk\n4l9SXMST112EJTCIMeecT152Fh+88Bj7tv3H7k3rXewHdm5h1jULvc6nRYfOXHjzfTx81Xk0a9OB\nvJwsSoqLue25t5ye8VOcnrQKt3B570bc8+shmoeaySu2U2K3c++IWKduy1P1V/g+6ihW1Btrf/6W\nlu07cd3jL/PXt58SGBJG7xHjuHPWWNKTk4hoFOPibzAaWfj8OxgMWudEn5HjuWXacJq1be9iv+2c\nkYydeaFLnIoYPmUm/cdOZu+WDZitgbTr3rssZk2iWnb1w+jWoQxuHsyulAIsfgY6RFowVPDSlar6\nK3wbJXiKeiNuw1oGT5xOeFQjpl1+XZm9S78h7Nq0niETp7v1dxSgkPBIOvbqR1h0Exe7pziVYbEG\n0H3QiFNcq8pRYle/WEwGesV4/yxdVf0VvosSPEW9ERQaTtqxRBd72rFEgsNc33Th0f94EgHBru8M\n9hSnNpiT9UzZ549DPXejKrFTKOoPJXgKAArz8/jlk+Vs+ft3zJYAhk+ZycDxU6ipd+y6iz9i2iwe\nmz+H/mMm07pzd6SU/PbFB+RlZ9G13xCXGCOnz+bRebNd/Avyctj2z18cjNvmVZyaxlHsKkKJ3UkK\nSuz8sDeDjUl5WEyC0a1DGdo8uMb2N4XCHUrwFBQXFfLYgrlENIph6qXXkJeTxZdvvsjBuG2cf9M9\ntRa/19DRXHbXoyy+/mKiYmLJy8nG32xh4QvvOj0Ld4JmbTq49b/z5RUk7NvldZyaxJ3Yzcl6psJW\n3plOsc3O/b8fJspqYmbnCHKL7XyyPY396YVc2qvuH0dRnDkowVPw949fYw0M5uanXy+7wu4xeCS3\nzhjB+NmXEhVTvTfHVxb/pR/WsX/HFiwBgbTs0KXCq/xB46fQd+R4F//Y1u3c2msTb1t2oFp3jvx5\nKJsAk4E7h8WW1VHvmECuWbmfs9qHEx3oV88ZKhoqSvAU7PpvHYPKdV8GhYbTpd8Q9mzdUG3Bqyz+\nkInTq/RWAz9/s1t/T/baoDKxO9HKU0Lnyo7kAoa1CHHaH4LNRro1DiQutUAJnqLWUIKnIDQymuTE\nwy725MTDhEZE+3z8ukS16qpPmMXIsdxiF/vx3GLCLGoGE0XtoQRPwchps3noinPpM2IcHXr2w26z\n8dPH71JcWEjnvoNqJP6Dl01n2z9/kn78GAaTkcjGTSnMzye6aXMenT+HxAN7MZqM9Bw6msvvfpzi\ngnx++OAttqz5HbNVG+Qy7OyZVX4mLj8nu0biQNXETuGZsW1CufvXw/SPDaJzdAA2u+T7vRkU2yTd\nGqkZTBS1h6jq9Eu1koQQFuBPwIwmwp9JKR8s53MZ8DRwYlz6y1LKNyuK26ZLD/nY+9/XfMINkI1/\n/Mzbj99DYEgo+bk5hEc35vrHXqJx81bVjp2dkc6t04fTffAIJsy+hNzsLD555WmE0Oap7DZweJn9\n01efxmjyByTN2nRgzLnna68Neusl2vfsx6W3u59yzB1FBQUsuuKcaseBUxe7lUt/OqXfNXTWHclh\nyYbjBPkbyCuxE2E1cdvgpsQE+9d3agpg+oe7Nkop+9V3HjWNr7TwioAxUspcIYQfsFoI8YOUsvwU\n9R9LKa+vh/waPH1HTqDnkNEc3huHJSCQpq3a1ljsFf/3EK06deXGJ18tu2/Ttd8QrpvUnyYtWrnY\nF4zrTed+g7j20Rec7DdPH8bkC66kUWwLr8pd88OXhEU1qnYc1bKreQY2C6Zv0yDiM4uwmATNQsz1\nnZLiDMAnJo+WGrn6Vz/9r/6bnmcYJj8/2nTpUaNiB3Bw13aGTj7HaZBCQHAInfsOIjA41MUe2TiG\nQeOmuNi79BvCvm3/eV3uni0bGDBmcrXjVIcp8ycyZf7EOinrdMNkELSLsCixU9QZPiF4AEIIoxBi\nM5AM/CJtpSuaAAAgAElEQVSlXOfGbaYQYqsQ4jMhRHMPceYLITYIITbkZKTXas4K7wgMCSHxwF4n\nm5SSo/H7Kd+jLqUkPzeHxIOu/scOHSA8urHX5YZFNSLp0IFqxVGtO4Wi4eArXZpIKW1ALyFEGPCl\nEKKblHK7g8u3wIdSyiIhxNXAMmCMmzhLgaWg3cOrg9QbBDmZGaxcvsRpcMeYcy/0OLgj6dBBljx4\nC0mHD2A0mujafwgLHn4ek5u3Csy8+laeveVKeg0bQ9cBQ7GVlrJy+RJyMtNJP57ES3dfT+KBPfhb\nrIRERFJcXMSf33xKRkpymT0sMhq73UbH3gO9zueceTfz8JUz6TF4ZFm5369YipR2j3FOUF2hS8sq\n4JmPN/LjukMEWv3oabUzsV2Yx4mHs4tsfBmXxn9JeZhNBka3CmFiuzByi+1u7WoCY4Wi6vjEoJXy\nCCEeAPKllG7POkIII5AupazwrZxq0Ip3FObncf8l0+jQsy+j9dfrfPnGCzRv14kr733CxT8zNZmF\n546i3+hJjJt1MXnZWXz88mLsNhtPfvyzW/+bzh6M0c8Ps8VKUWEhBqOBwOBQSkuK6TZgGOPnXEpe\ndhYfvfQkkY1jSNi/m069BzB+9qVl8WNbt+e6x16sUj7n33QPbz9xLwBF+XnEtmnPgoeeI7ppM4/b\no7pil1tQzOBrPmZwtxiuOrsbGTlFPLp8HWGlRVzdx3UmkYISO7f/HE+naCsT2oaRW2zj4+1pNAvx\nZ3dqgYu9ZZiZa/s3qVaOCkVFqEErtYgQIhookVJmCiGswHhgcTmfGCllkv51GhBXx2k2WFZ//wVN\nmrdk3v1Pldk69urPzVOHMuWSq11Gaq549mHaduvNgoeedfK/dkI/dqxfQ9cBQ138O/UdxO0vLmPr\n36sICg2nRftOzB/dk24Dh3HNI887xbnhrEG07tSVax52tt88dSjHE+KrlI/J5MdzX/9FUvx+zNaA\nSh+ir4kuzBW/7KJtbBhLF44rsw3tHkPr2W8xrX2Yy0jEVfFZxAT7c/2Ak68x6hwVwFXf7KNNuMXF\nfvW3+zmnU4Qa0ahQVBFfuYcXA/wuhNgK/It2D2+lEOJhIcQ03edGIcQOIcQW4EbgsnrKtcGxb9sm\n+oyc4GSzBATSud9g9u/Y7OJ/aE8cgyZMdfXvO4h/f//Bo7/JZNKf9euLJSCQyCYx9B8zySVOh579\niG7WssbyMRgMxLZpXydiB7A+7hhTh7R2sgVZ/RnTtyV70wpd/PekFTIg1vmFt1Y/A52jAmgS5Odi\n79YowG0chUJRMT4heFLKrVLK3lLKHlLKblLKh3X7A1LKb/TPd0spu0ope0opR0spd9Vv1g2HiMYx\nJB7Y42STUpJ4YI/bl6cGh4VzeG+ci/+R/buJaek6wtOTf152Fgl7d7nYj8bvoyA3u9bycUdNDk5p\nFh1M3CHnAVNSSuIOpRMZ4NqpEhVgIiGryMX/SHYR+cU2F3tCdpHbOAqFomJ8QvAU9cvoGXP5a+Xn\n/Pfnr0gpKS4q5PMlz2Ly86ejm7kpZ193B79/+aGT/6evPU1+bg7jZ1/q1v+3Lz7g/265kltnjOCu\nuRN58roLKS0tYfX3X5SL8wx+/ma2r1tTa/k4MifrmRofiXnFWV157+ddrFx7ACklhcWlPPTuP/j7\nGbjj3hku/uPahPJ7fDb/JuZq+dvsfLg9FT+jYPOxPCf7+1tTMQpBl2hrjeR6KKOQW388yAWf7eHi\nz/fy+J9HKC6110hshcLX8MlBKzWFGrTiPTs3rOWdJ+4lLyeL4qIi2nfvzfwHnyY82v3giO/ff5Mv\nlj4PSEqLiwkKC+e2596idafuLr4Febncds5I+o6awPhZl5CXncWHLz5OcGg48Xt2Im02JNprhCzW\nQDr06sf4WZfUWj4nqM1HDlZtPsL1z/9OZm4RBUWlDOzShDdvH0fTqCC3s69sO57Hkg3HySuxU1xq\np2OUlQu7R3HP/w5j9dOuS4ttErNRYPUz8tqUNtXOMS2/hOu+O8ioViFMah9GXrGdZVuSyS+28/LZ\n1Y+vOH1pqINWlOApypBSkpp0BH+LldCIqEr97XY7e7ZsIDAkjOZtO3j0++mjd4nbuJabn369zFZc\nWMD1kwfQuksP7np5RVm51oBAbpwyhAff+pwmLVrXSj5QN8/XSSk5dDyHALOJRuEn54j0NN2YlJLk\nvBLMJgNhFhOLVydSYpfcOzy2zG41Gbjsq33cPqQpfZoGuY3jLYtXJ2rvpht58pHWolJ7jcVXnL40\nVMFTXZqKMoQQRDdt7pW4ABgMBjr1HlCpuByM20LPoaOdbP4WKx169icqpplTuf4WK136DuJg3NZa\ny6euHiYXQtCqSYiT2AEeZ14RQtA4yJ8wi3Z/7khOMQNjg5zsZpOBbo0D+PdortsYVSEhq4hBzYKd\nbGaTga7R1hqJr1D4GkrwFLVOVJNmHgeV5GVnudgP791FVIzn5+TOFEL8DRzIdB3McjCjiJah1Z+O\nK9RiZH+682hPKSUHM2smvkLha6ihXg2U9OQkPn/9ebb8rc+ccvZMplxyNSY/989uefI/vDeO1xct\nJOVoAgajkbZde3HLM69jCfC+u2vUjLncc8FkOvbqz4CxZ1FcWMBnS57FGhTMzn//5p9fVpbZv1j6\nPAFBwbTv0bemNoUTNd26S0zJ5eFl6/hxfTyBFj8untCJ2+b0xd+v4ve6nWjlVfQ2hYt7RPPA7wl0\ni7YyuHkwRaWSj7anUlxqp02YPzf+cJDk3BIMBugQaeGOIc0oKLXx0fZUNiblYTEaGN06hBmdIvEz\nus7McnGPaO7/PYHujQPK4n+4LYVim6RvTACvrE9yiZNdVOo2fnZRKZ/EpbPpWD4Wk4ERzYOY3jHC\nbbmg3T/0Nk+FoqZQ9/AaIHk5WdxzwVkMGj+FMeecT15OFp+++gyBIaFc//jLXvub/PzZ8e8axp53\nEeNmXkReThYfPP846clJPPvVn1XK5/aZYzEajRTk52IrLSU4NJxWnbsz5ZKreXfx/aQcTcBWWkr3\ngcO54t4nvO7G9Jba6MbMzC2i3/wPmDWqvTajSm4RD7y9lvAgM+/fP9mrGJW9PujHvRm8vy2VolI7\nNgmRASau6duYxWuOMql9GBP1GViWb0khJb8Eux2Gtggum5nlg22pBPkbuW1IU4/xV2zVRM4mIdJq\n5NbBTXl2bZJLHItJsD+9yK09PruECyZ1Zf7U7mTkFnHv0tWUZuVyywDXQUa5xTZu/TG+Snkq6paG\neg9PCV4D5IcP3mLv1o3c+OSrZbaS4iJumjKEe5d8RGyb9l75XzuhH606duXe1z9ysl8zvg/XPvIC\nfUaMwxtOxL/hiVfITk/Fz9+Mn9lclk/T1u3K7AHBIdVce1dq657di59tYu3OY3z4wElxKyoupc35\n7/DrszPp3DLCqziViZ7dbichq5gAfwPRgf488dcREIK7h518kL7EZueSL/fRMdLCotEtnOzzvtnP\nI2Na0NxDN2X5+N/uTmdXagG3D3WOf8XX+2kXYebBUS1c7AO7xfDjMzOdtkPLWW/x4LAYl3I9xa8s\nT0Xd0VAFT93Da4Ac2r2dHoNGONn8/M106j2AQ3t2eO3fvkcfGjdv6WLv1HsAW9euqnI+QghCI6MJ\nCA5xysfRXtPU5gCVTftSGN/P+Z16Zn8Tw3vEsnlfSo2VYzAYaBluITpQ644+mltC/5hAJx8/o4Eu\n0VYaBfq52hsFcDDD+V5gRfEPZBTSu4lr/I5RFqIDXON3irLSMibMyW72NzGse4zbcj3FryxPhaK6\nKMFrgDSKbcnBXdudbHa7nfjdO2gU29Jr/8N748hKT3OxH9y1nZYdu9VaPjVFbY/GbBMTyqa9yU42\nu12yeV8KbWIqnNe8WoSZjezJcB5sYpeSAxlFZBfZXOwHMwppXG6KsopoHOTPfjfx4zOKyCl2jR+f\nWUhqZr6z3S7ZvC/Vbbme4lc1T4WiqqhBKw2QUTPmcM/5k2nbrRdDJ82gsCCPz177P0IjomjbrZdb\n/7vnTnTy//jlpwgOi2Dr2lX8ufIzB/tibKWldO0/hFfvu4nNf/+OxRLAsCkzmXHlDWz663+88+R9\nFOTmYDAaCYmI5KanlvLU9Rd5nU9NUBePHlxxVlf6zv+A/p2acMG4juQWlPDg22tpHB7AgM7ev7dv\nyvyJlXZrOnJpz2ju/vUQCZlFJGQVYzYJwq0mSu12diTn89vBLEa2DKGg1M4H21IJs5joEGlxG+vv\nw9ks/S+ZvGIbBiEIsxi5Y2gMD61KpH2k1SlOuNXEpqRcrvx6H9lFNowCAv0MBJuNrPrvMMt/iivb\nDve/uYYgg3Rb7vg2odzyY7xL/IryVChqAnUPr4FyYOcWlj/zEId2bQch6DtqApfe/jDBYeEuvrlZ\nGdx+3hgsAUFkpBxHAKFR0cS2bs+g8VP44IXHyc3KBCA6tjnXPfIiz98xn1HT5zLmXO11Qh+/8hQF\nubns276JKZdczdiZF5KXncX7zz3K/p1buOvl97zOpzrU9QtbN+w+zq0v/8GmfSkIBNOGtuGFG0YS\nGVq1qb+qIng5RTau++4Ak9qFMbFdGHkldt7+LxmQXNgjmrf+S+ZARiFCwIDYYOb1bUyI2XXU6L60\nAu7532HO6RzJhLah5JXYeXdzMnvTC3lgRDOXOONah/DYX4nM6BRRVu47m5LZm1bAg6Oas2xbGvvS\nCgAY1DyEK3pGuS0XYG9agdd5KuqehnoPTwleAyc/JxuTnx/+Fs8n4O/ee51Du3dy7aMvlPkbjEZu\nmjKEu15ZQfN2nUhNSsQSEEBQaLiT/wlKS4q5ckRX+gwfx01PveZkv2Z8H86ddzOTL7zKq3xOlfp8\nO3lWbhH+fkas5lPvNPFW9L6KS+NgZhG3DD45orHEJpn/7X4WjWpOyzAzecU2TAaB2eT5rsUN3x+g\nRZiF24c4x7nsq73M6RrFtE4RTnGu/+4ALULN3OE0WMazvzdU1V9RNzRUwVN7WQMnIDikUnFJ2LeL\nLv0GO/mb/Pzp2Ks/Cfu0txlExcQSFBru4n8Ck58/1sAgeg4Z6WLv0Ks/m9f85nU+p0J9ih1AaJC5\nWmJXFQ5lFdG9kfPsLX5GQedoK4f0B9UD/Y2VikhuiZ3ejV3jdGkUwMakXJc4ucU2+rgMlhF0jrK6\n9feGqvorFNVB7WkKmjRvzf4dW5xsdrudA3FbadKitdf+hQV57N6ywTXOji106Fl7F4v1LXZ1TUyw\nP3vSXQd97E8vJCbY+0EfFqMgLs01zt7UQjpFul6UWE0G4lILXP3T3fsrFL6GTwxaEUJYgD8BM1pO\nn0kpHyznYwaWA32BNGCOlDK+jlP1OY4dPshHLz3J5jW/YbYEMGLqeZy3YCEZKcfc2s1W1xPTqBlz\nufv8SbTq2JUR02ZRkJfLxy8vJqpJM1p37uHW/47zxhK/axsJ+3bjb7ES0SiG6KYtWPvj17Tt2pOR\n02ZTkJfLB88/RklxEYMmTuf526/2Kp9TYV9iJncvXcMP67QZTy6Z2JmHLh9EgMW9APxv42EW/N9v\nHE7Owc9ooEfbSL55fBqZecVVjjNv8c8cSc3DaBD0aBvNd4unk5lXzB2v/MlPGw4TYDZyycTOPHLV\nUNbuSHLrHxUW4PXglfFtwrj5x4O0CTcztnUo+SV23tuaQnSAH4czi3j4jwTyiu0YDIJQfyNPjmtB\nqYT3d6Tz75FsLH5GRrcK4cpejXhyTSKHMgo5lFWE2WggKsBEsc3O8BZBPLk6kY1Hc7GYDIxpHcqV\nvRux+O+jtI+wMLaNVu67m5MpsUlmd3X/zGFSTjHLtqQ4xbmgexTpBaVu7TXV2quvchW+jU/cwxNC\nCCBQSpkrhPADVgM3SSn/cfC5FughpVwghJgLnCOlnFNR3IZ+Dy87I517zp/IxLlXMPrE4JGXF5OX\nnUXigT0u9qKCAm5/4R23sQ7viWPFc48Qt/EfTH5+DJ44jYtuud/ts3FH4/dz7wWTmTHvppODU559\nhP07t1CYl4vBYCQ/LweDwYDZGkDTVu1IP360Svl4y5ysZ0jNKqDPVR9ww7k9uWpKNzJyirj3zb/J\nKyzhm8enufxmd0IG/ed/wH0XD2TeVM3/9iV/sT7uGAJRpTh9rljB7K6RTGynzRjy7uYU4nNLkcDE\nlsFMaBuqzSSyPQ0ZFMD6uOMu/ofybCR8Pg/w/j5efEYhb29KZkdKPiaDYFiLEIa3COaxvxKZ2y2S\nie3CyS228c6mZLYfzyfAYuKWuf2YN7U7GTlF3P36X+zdc4xdKfnM7hrJpPaa/9v/aYNWBDClQzgT\n9Dzf25JCUamdro0C+GxnGvkldgQQYjby0OjmtAxzHV2ZXVTKzT/Eu8TJLbKRkF3sNv59Dm9uOFXq\nq9yGREO9h+cTgueIECIATfCukVKuc7D/BCySUq4VQpiAY0C0rGAFGrrgffvuayQe3MuCh54ts9lK\nS1kwrhd9Rozjmoefd7LfPG0oC59/h5YduniMWVJchNFowmD0PFruqRsuJSA4hOsff8kp/rUT+hLb\npgMPvPkpWempWK2BGP38qpWPJxy7MZ/+cANxh9N5+84JZbZSm512F7zD149No2e7aKffTr3ra0KD\nzKy4b5KTf+TUJcwY1pZl90z0Ks6UO78iPzmTWweffAu7zS657Kt9dIoO4N7hsU72K77eT9sICw+M\nbObiv/TOCcwaffItD94KX4nNjkEIjAbBdSv30ybCwm1DnMu99Kt9tGsWzoY3L3Rar/CzXqF/0yBu\ndRj8ciKfjpEWJxGw2bVBMfeNaEbrcAtZBSX4mwxYK5gz9IudaSRkF3PTIOftc8kXe+kfG8TN5cp1\njF8d6qvchkRDFTyfaccLIYxCiM1AMvCLo9jpxAIJAFLKUiALiKzbLH2LxIN76FTuDeBGk4mQiCg6\n9xnkYm/foy+JB/ZUGNPP31yh2AGkJCXQbcBQt/ENRm2XCo2Iwt9qrXY+7ih/z27noXSGdY91spmM\nBgZ3bcrOQ+kuv48/ls2YPs1d/JtEBDCyVzMXu6c4+xMz6NHIuUvWaBB0jLISbja42NtHWoiyGl3s\nnRsF8OfWRA9rWzF+RgNGgzbhcl6JnZ6NnQeVGA2CTlFWisq9xdxkNBBgNtGj3OAXo0HQIdJKhNXk\nYu8UZSUhqxiAUKtfhWIHkJBdRNdo1+0TajHSzU25jvGrQ32Vq/B9fEbwpJQ2KWUvoBkwQAjh/VQe\nDggh5gshNgghNuRkuJ6kGhJNW7Vjz9aNTja7zUZ2ehq7N//rYt+3bRNNW7WrdrlRTWLZufEfJ5sW\n/z+kXbrYazIfdwNUOrUIZ+2OJCebzWZn3c4kOrVwfc6vReNg/thyxMX/eEY+q3XhsdnsSCkrjNO6\naRjbU5wHcdjskj2pBWQW213s+9ILSSuwudh3peQzpFuMU7nl35lns0vcdWY42gP8DGxLzndZvie1\nALOf86Fus9nJLyp1m//etAIy3OS5J62AZiHu37bhjtgQs8sgF5tdkl1kY6e77eYQ39P61na5ioaN\nzwjeCaSUmcDvwKRyixKB5gB6l2Yo2uCV8r9fKqXsJ6XsFxzu3eS9pysjp89h8+rf+OH9NyksyCft\neBJLHryVlh06s2n1b3y/4o0y+8v33kCT5q1o1emUriOcuOCW+/n3f9+7xDeaTCQd2u82ny1/r3Kx\nN23Vtkr5eBqNefnkrvywLp4XPttEXkEJR1JyuHzxL3RsEUHv9o1c/J9eMJwv/tjHc5/+V+Z/0WM/\nEmDx47t/4ul91fsETHyF0LNfo9dV79M2NsxtnGeuGcHfCTl8vSudwlI7qfklPPdPEmaLH/szi/l6\nV5qTvX2LcLYl57v4W8x+9GwbzZS7viZg4iuEnf0a1z//O/klNg5naYNQZn2ym7mf7WXJhmNl9sfX\nHGX2p3u44It9LN2UzJV9Gun5nCz32bVHKbVLDiVlO63vhY/+SMswC38fznbJx2gQ7M0o5BsH+4vr\nkogNNtMmwvtuv3FtQtmYlOcSp2Wohf+Oudpjg82YjMLt+laFUym3KuulOH3xiXt4QohooERKmSmE\nsAI/A4ullCsdfK4DujsMWjlXSjm7orgN/R4eQOLBfXz4wmNs+XsVZksAw84+l8kXXsUDl0wjonEM\nR/btxt9qJapJU4LDI7nv9Y9rpNxNf/3GO0/eQ9qxJIwmE83aduCOF5eRl5Ptks/cG+4i7XiSW7sl\nILCyooDKHz3YdTidO19fzY/rDhFoMXHRhM48Pm8IQVbXK/fj6Xl0uXQ5Fj8TyZkF+JkMhAebadE4\nmPikbO6+aADzpnQjM7eIO5b8RVJaHr8+O9NNqfD9PwdZ8PSvHE3XBo90aRXBsnsnMu7WL4gJsxKX\nkInF30iLRkFEhFjZHp+GSUBaTjEmgyA82EzrpqEcOp7D3Rf258qztXLveWMNm7ceJiGjkPO6RjK+\njTazyXtbkknOKyEp38b9lw3mKj3Pu5b8RdyuowyIsbJiSyq5xXaMBkGIv4G7hsXy+N9JtGoaxvaD\naQRYTDSPDsKvuJip7UN5fUMyKXklGA2ClmFm7h/RjLwSbUDNf/poxlGtQri4ZyOsflW7Rj6SXeQ2\nTlpBiYt9WscI7vr1kMv6pheU8tDoFpUXdorlnsp6NXQa6j08XxG8HsAywIjW6vxESvmwEOJhYIOU\n8hv90YX3gN5AOjBXSnmgorhnguCd4EQ9CiH4+u2XSU48zLz7nyqzS7udW6YP5+anX6d15+41Vq7d\nrnXdGQzOJwzHfLyxe6Kqz9h5E//J9//lQFIWSxeOK8tfSog59w2mDW3Dm3eML/O12ex0uGgZnz50\nNn06uLbyTuC4HRzjn8jHbpe0nvs2/To15otHpnpVbpNzltIn2sINA50HX8z7Zj9DejXjq8enO/m3\nnfs2t/SNpm2EBZtNaxUZjUY+25mGsUkkb9010SkfR/+q1mNV8WZ/+GxHGsfyirl+gPP6Llh5gLuG\nxdL2FFphNbUfnmk0VMHziefwpJRb0YSsvP0Bh8+FwKy6zOt0wvHATTq0n069BzrZhdFI2269SDq0\nv0YFr/wJ0l0+3tjdcSoPlHsTf3dCBsN7aINcHPOPCrUypJvzC0iNRgMDOjfRHkGoQPAc4zjGP5GP\n0Sjo26Ex0WEBXpcbEWSmc5Tr4Iu2ERYaRwa5+Pfv1IjE7CLaRlgwOgw8OpZfyuwezVzycfSvaj1W\nFW/2h8ScIrpGuxtEYyExu/iUBK8m9kNFw0G14xsgsW06smuT8yDX0pIS9mzZQLM2Hespq6pRm7On\ndG0VyV/64JSColJKSm2UlNo4npHPH5uPuNj/3n6Urq0inOzexj9BSamNdXHHOJ6eB0B6diH5hcUV\nlpuaXchOfSaUolI7pXZJqT7I4mhyjkuctTuO0SLU38kfIDbQj1X/HXbJx9HfE45xapvmoWZ2pDgP\nuim1S+JSC9yuV3Woy/VS+A4+0aVZW5xJXZqO5GRmcPf5Exl33sWMOfdC8nK0B71tJcXc9tzb9Z1e\npdT2VGFpWQV0u/w9gs0mElLzMApBTGQArWLC2LI/hchQC4eO5WA0CmIiAmkdE8LiBcO56fnfWL87\nGZNBcO6Idrxw02jCglzfzp2WVUCfeR+wYFp3/UHvQu0B9oIS1sUdw2gQZOYWIYQgLMhM5xbh7ErI\ncJvP1n3JmLGTnFeKQUCk1URUoB8Hs4oxCMgp1OwhFj9ahlu4pHsky7alsfO4NpPLsJYhzOwYzgN/\nJHLDrN7Mn9aDjJxC7l6ymmNH0rhrSIybLaS9pLV8nMt6RBHkX3tvM8gusnHzjweZrL8FIrfYzntb\nUii1S87vHsVb/yWzO7UAowGGNA/hyj6NqpzPgYzCGonT0GmoXZpK8BooxxPi+fiVp9i8+jcsAYEM\nP3smM6++pVYmbq5palvwktLy6H7pci7oEs7o1mEUlGon1mRpZNeRTJ6/fiSXTupCTn4xdyxZzcq1\nB5F2O3M7hzO6Vaj2/rbtqeT4mVn10my33WP7EzO57621fP/PQYKsflw8oTMzhrdl/K1f8Nz1I7hk\nohb/rqWr+XbNQUpLbS75pBv82H80y8m+bHMy8fpUYPP7Nnay/3s0FyEEi689Gf+epavZsDmBq3tH\n8VFcBv8eycHqb2RUy2DmdIl0O6VWekEpt/1ymCcWDHOJs2hE01rtDkzKKWbF1hQ2HM3DahKMbh3K\nxLZh3PHLIS7qGV22/VdsTSExu5hHxzT3Op/0glJu/uFgteOcCSjBOw05kwXvdKUuJoJ+ZNk61q/d\nw9V9Tt6Ts0vJVd8c0AaDPHZyGjG7XRI59TUGxwZxbb/GTv43/HSIzx6fTv9OTbwqd+LCL2jROJg3\nbj85OKWi+Jd/vZ9BzYO5pq+z/cqv99M23Ow0E4pdSq74ah892jfmtxdnOcVvd/7b3NArkvZeTvD8\n6Y40rM2iWVIuz6rGqSk+2Z5KWkEp1/Q/uZ3tUnLNygMsHNLU63xqKs6ZQEMVPHUPT+ETzMl6ps7e\nerD3cDpty927MghB6zAzjSOcH5MwGAQRQWY6hJtd/NtGWNmXmOV1uYmpeQzu6jw4paL4Qf5G2oe5\n2luHm11mQjEIQbsIC4ZyLTaDQdCnfTRJOSVe53m8oJQBbvKsapyaIim3mPblBqxo299SpXxqKo7i\n9EUJnqLeqUjoMnIKySuo2ZNRrw6N2JFW5GQrsUn2pBdw6JizgBWX2DieVcBWfYaO3GIbhaV2SmyS\nncl59GgTVWFZjvm3jw3lx/XxFcZ3zCezsJRtqYUu9j2phRzPK3Gxx6UWUFxS6hL/7x1JtAp3vdfo\nieZBfvz6r2ueVY1TU7QKs7D1uPNglhKbJC6loEr51FQcxemLcdGiRfWdQ63xwiuvLRo788LKHRX1\nhiexWx93jFkPfse9b/7NMx//x9YDqYzq1czja3qqQqeWESz6YAOZ+SU0DfYnOa+El/89RusWkazb\nlae/L50AABmdSURBVExeQQntm4URn5TNVU/9QmGJjQOpefywN4NPd6TxZVw6vx7IpH3LCO68cIDb\nMtzlf9eF/Xng7bVu46cV2MjKL6ZJkJbP0k0pdGrXiK1HspzyfGldEuEWE7vSCigslWX2F9clUVBi\nJzOniIy84rL4Vz/zC8H2Uia3C/N6+zQL8WfpmoRqx6kpmoX4s2JrCjlFNmL07fPahuM0DvTj7A6u\nU77VdpwzgY+2pyYtWrRoaX3nUdOoe3iKesOT2B1JyaHf/A/5v2tHMHdMB/IKS1j07j/8G3ecP1+a\nVe3BBUdScuh91ft0bRHBjvg0AswmmjUKBgRPXzuM+c/8RvyxbPyMBvp0iOa8ke259eU/uG5AE4a3\nCKHIpg0q+f1gNpk/XIPJZHKJ7yn/xQuGusT/8pGpZOYVcf+ba/jhn0MEWExcOrkLl0zqQv95H9A8\nyMThrGIsJgMRViMlNklSThGB/iayimwYhSDI30CoxcidQ2P5ZFcGG47mYvUzMrJ5EDM7R+BnrFpn\nTkpeCR/HpVc7Tk2RklfCB9tSy95jN7p1COd1iTyl9aqJOA2dhnoPTwmeol6oqBvz4WX/kJpVyIs3\njiqzSSnpculy3rlrAoO6uB9K7y1Vjd9o2hL6NbZyrcNgB6kPHpk8vB3L7nae6Lmm8l/0zlo2r9/H\nVb1PDq6RUjL/2wO0Czdz5/BmTvZrvzvIzYNi6BilBl8oqkdDFTx1WaOoU7wZnHIwKZte5d49J4Sg\nZ9toDiZlVzuHqsYvKS6lQ7nBDkII2oSb2bw3udbyP3Akk1blZvEXQtAyzJ9gs9HF3jrczPFcNfhC\nofCEEjxFneHtKMw+7Rvx87+HnGyFxaX8uSWR3u2jPfzKM1JKjqbmkp5d6FX88v5BQWbWJeY6+Rfb\n7OxIKeDcEa6vN6pqfE/069KEreUGrRTb7OxJKyI5v8TFviM5nzYRavBFZUgpScsvIaeoam9hUJz+\nqC5NRZ1QlUcOsnKL6H/1h0wf1pZ5U7qRnlPIonf+ITrMynv3ln9rVMWs3pbIDc+v4mhaHsWlNkb2\nbMZTC4Yx5a6v3ca/elp3F/95U7oy6/6VTGgbxuT24eQW21i2OZlDWUVk/XRDlfK/elp3rnn6V5LS\n8iixSUb0aMobd02gSYTrWyOycovoc+UKekeZGd86lNxiGx/HpdOyVTR/b05gULNgJrQNI7fYxgfb\nUgk1G7l1SFOXOIqT7EzJ5/UNx0kvKKXULunWKIBr+zch3OoT0wr7DA21S1MJnqLWOZXn65LS8nh0\n+Tq+XxdPkMWPiyd25pZZvfEzeT8F1KFj2QxY8BGv3TqGGcPaUlhcyqPvree3/xL44pEpPPbeeqf4\n5w7///buPDqqKtvj+HeHBEII8xgSJmUQARGIgKAIioJIq6jtPGCrIAINrT7b5rX69NmtT3m22rhE\nlEFEQQRUGkGwBcUB0RDmQR+TQICEIYTMA7XfHykwIQMJVOXeqtqftbJMbk7duysu8ss599xz2tJ3\nzNwS7Zes3s3u/cdpEB1B8rFswsOERvWjqBddg5+m3FnqJJrS6r+p3/n0HjmHkd0a0ysumvwTytwt\nR9mRo/z09l1lnue56av4bNUuakWGM3xIZx69rTuH03J46Ml5JJyafFGXGy5oQHiYrRZSlpTMfB5b\nuptHLml26uf/4abDbEjO4uVrWtlKK0VY4AUgCzx3qKoHyk/3zLRVpGfn8croK04dU1W63D+Ltx67\nir5dmleofezNb3PrgA68OrZi5ynLU+98z+bEnfzh4t+GZVWV8V/sYeYzQyt8npMWTVlaqfah7v0N\nh8gu8PBg999WrlFVxizexZiezeh42k4NoSxYA8/u4Rm/cirsAPakpNOpdcNix0SETq0bsse720BF\n2teNrkGX8yp+nrL8eiCNFrWLP0coIrSsG1mp85w0dMSgMzcypxzOyqdV3eL3OEWEVnVrcCizoIxX\nmWBigWd8zuPxEL/lr/Tb+WwF2ys7ko5x4EimT+u4pENTFq/eXexYVk4+X6/fR3yHJiWue7J90eNZ\nOfnsP5zJolW7yjxPRfXqHMPalOIrquQWeNiYnFHueQoKPKxI3MO67YcqfK3K8KhyID2Po9nB/Uu/\nbYOaJOwvPvkot8DDppQs2jas/F57JvC44k6tiLQAZgJNAQWmqOprp7XpD3wKnPzNs0BVn6vKOs2Z\nbVr9LXOeHUFmVh7Z+Sfo0b4JM/5zMHGNa5fa/ss1exjz2gqycgrIzMmnR4emTH1iYJntK+OeQR2Z\n+OEaRk78N6Nu7Fq4Tc/b33NltxbsSU7n+gkLi133tbFX8D+zE2h2w2TCgOx8D1E1qnF1fAu27E5l\nzKsreHBoZ1LTc/jvd1cz9NI2tIur+Aod9w66kNc/WsuUxGQGeiehzNuWyu/6nF/meSbOSeC5GT8Q\nBuQVeKgXXYNPX6z4gtVnsv5gJpMTksk94SG3wEPbBoU7rDeKOvcVbdxmQJs6fPZLKpMTDp6a7PPh\npsNcEhtN89rl7wtogoMr7uGJSAwQo6qJIlIbWAPcqKpbirTpDzyuqkMrel67h1e1UpL28OztAxgb\n34TuMbUo8CgLtqWyIa2AddPvKTEpYNeBNHqP+pAZf7mGwT1bkZd/gpdmr+GTb3eQMOWOc55EsOtA\nGj1HzqF/t1i27D5KVI1wmjaI4teDx0lOzebdCYOKXffDFb9w4FA6f7yk6an65209yqbjJ/jiHzfz\n0uw1LFm9m1qRhdv9PHLjRVSr7Eofx7J44b0f+WzVTqIiI7h/SGdGD+ta6nlWrt/H4McX8Oe+scXq\nWbz9GEcWjSq2S/nZ3M9Lzsjj8WW/Mr53zKnzz996lB/2pvOPwa2DchJHWk4B87ccYc2BTGqEhzGg\ndR2GtKtPNZvsU0yw3sNzReCdTkQ+BSap6hdFjvXHAs/VNkwcxs/rdjG8a/FJGY/+ey/TnhrC5RfF\nFmv/1NTvyck7wcujLi/W/uIH3mfS+AEl2ldWWefveO9MurVrwuynry12PPamt7k0pib3X1x8ZZOy\n6ve3fmPm0rAglwe6F6/n4UU7mfCHPowedvGp42cTeLM2HCL/hHL/aSu5jFuym5HxTenUxCZxhKpg\nDTzX3cMTkdZAN2B1Kd++VETWi8gSEelUxutHiEiCiCSkpx71Y6WmqNvSJrIv+TjNo0tOyoirU4N9\nhzJKvCbpcAYdWtYv0f6Clg1KbV9ZZZ2/Q4v61I4qWWd0ZDixtUuubFJW/f52+FgWLeqWrKdlvUi2\n7UktdvxsJrAcySogtpSVXOLqVOdIVnDfzzOhyVWBJyLRwHxgvKqevgZTItBKVbsC/wQ+Ke0cqjpF\nVeNVNb52/Qb+LdgUWyqsz0WxJCZnU3TUICv/BBsPZtCrY8l7Tr06xvCv73cWa5+elcdX6/aV2r6y\nyjr/dxv3s/NAWonjB4/lkHAwq8L1+1v3Dk35fm96KfVkcvMV7c75/B0aRfJjUkbJ86dk0b6RTeIw\nwcc1Q5oiEgEsApaq6isVaL8biFfVw2W1sSFN/zr9kYPM7Hy63DeT1rXCGOJdkeT9DYfp1a0ls566\ntsTrM7Pz6TP6Q+I7ND01GeRv7/3IxW0b88afrjzn+jKz8+l477v06xrH6GG/TVppHVObnfuPl7hu\np9YN+X5jEnE1hIFt6pCRd4L521LpG9+ayY8PPOd6KislNYv2d04nPqYWg9sWTrKYteEQ0XVrsW76\nPaW+pjJDmzkFHp5Y9ittG0Ryjff8czcd5rz6kTx8SdUHvHGPYB3SdMssTQGmAlvLCjsRaQYkq6qK\nSE8Ke6dHqrBMU0Rpz9ftP5JBem4BcRe14r1fUoiqEc7FnWNZv+MwHo8SdtrEgFo1I/jqtVt4ZW4i\nY19bQa3ICIZf24kHhpQ6Wl1p32/eT2p6DtXChLGvrSCqRjjnx9ZleeJefn7vPl5fsK7EddMyc5k4\nZw3vfbeDWpER/PGuXjxwXWef1FNZTepHsXba3dz3t6W88N1+wqsJg3q1Ydqfr/HJ+SPDw/j7wJZ8\nsu0obyUcJDI8jIHn1WPg+XV9cn5j3MYVPTwRuQz4BtgIeLyHJwAtAVR1soiMAUYBBUA28Kiqfl/e\nea2H5x9lPUw+4e3v8HiUF0deduqYqtLjoQ94ZcwV9L84rtTX+cvlY+fS+8KYEpNWLrjnXUbd2JXx\nt3Sr0nqqgq2+YnzBenh+pKrfAuXOC1bVScCkqqnInI2DRzPp07n48lgiQtu4ej5/qLwijmXkljpp\npX2L+uxIOlbl9VSFoSMGWegZUwZXTVoxVaMgP48dm9exb+cvVLaHX95SYX06N2fByu3FzpmWkctX\na/dxaacY8vJP8NO2g2zZfaTS1z0bPdo34YN/bytRz9frkrjtyvZ+v76vVPXPzZhg5YohTX+xIc2S\nflq+hOkv/pU69RuSlX6c2vUbMOaFN4hp2abc11VkTczs3AL6jf2Idi3q8eB1hZNBXnj/J/p1jeXy\ni2IZ8+oKmtSL4lhmLo3r1uT9pwZXaqWSyjp6PId2d81gQLc4HvGutPLU1FXUiAhj7dS7/XZdX/r4\nm+1n9XOzXp45FzakaQJe0q7tvPP8k/zHazNo26UbHo+HL+bOZOK44bw8bzlh1UrfeqeiC0DXrBHO\nl/+4iUkfr+eZaauIqhnB+N93p0f7xvQfN5+Ff7+eXhc2w+NR3vx0A9dPWMim6fdUerWSiko5loUI\nJB3K4J6/fU54tTBaNa1NcmomJ054/HZdX9m25ygP/+/yKv+5GROs7F9MCFm5cC4Dht1B2y6FkzXC\nwsIYdPtwatSsxdY1P5T6msrudlCnVg0m3N2TbybdytKXh3H31Rfw7udbefC6TvS6sJn3usLoYV2J\njqzO1+uTzu1NlWPGki2M/F0XVr15O0nzH+LXuQ+w8p+3Uicq0q/X9ZUZS7Y48nMzJlhZ4IWQtKOH\naRLbssTxJrEtSEst+Tijr7b2STmWReuYklPdW8fUIeVYlk+u4abr+sq51D90xCDbPsiY01jghZCO\nPXqzatm/ik18yEhLZfNP39Gh6yXF2vpyH7t+XeP4aEXxCTJHj+ewYu1e+nau3KangXDdotd6fd5a\n3lu2FY/Hc8b22bkFfL1uHwk/J6OqjtdvTLCxSSshJC83h+dH3Ebdho24ctidZB5PY+GMN+je72pu\nH/sk4J8NW3PyCrjqT/NpUj+qcDJLRi4vfZDA0D5t+PtDfX1+PaevCzDu9RVMWbiRmNrVycw7Qc4J\nZdZT1/K7vueX2n7O8p8Z9/rXnN+8LqkZOYSHhTHzPwcVTlg5x/ptAouprGCdtGKBF2JysrP4cv4s\n1n27gsioKC6/7mYuufJaRMSvu5NnZucz5V8bC7fXqRnB3VdfwE392vp9Cxonrvvptzu467nFvHBV\nS9rUj0RV+WJnGtPXpnBk0SiqVy8+V2zzriMMfGw+n780jK5tG6OqTFu8mRc/SGDNW3cwdfHmc6rf\nAs9UlgVeALLAqzh/hl2o6f3wbFqHn+DuixoXOz528S5G3RbPE3cU/z3yxORvqFk9nGf/cGnx84ya\nw/MP9mFgj5L3XSvLQs9URrAGnt3DMxZ2PnY8I5dGUSWf+GlcK4KkUrYZOpKWQ/NG0SWOxzaO5nBa\ntl9qNCYUWeAZ42NX9GjJl7uO4ykyenIsp4CNyZncN7hjifYDurfgw+U/4/H81j4lNYuv1u6jXxVv\nOmtMMLPAC3HWu/O9l0dexpFcD0+v2Mt3e46zbPsxHl26mz6dm9O9fdMS7W/t3w5VuO7JT5n39f/x\n9qJN9PvjR4y75eJSe35nwx5RMMZWWglZFnT+Ex1VnV8+GM4jryzng/VJRERUY8zve/D08N6ltq8e\nUY0lL9/I9CVbmLl0K9E1I3hldD+G9C5/uTdjTOXYpJUQZGEXumzyiqkIm7RigoKFnTEmVNmQZghx\nOuxS03P4at0+akVGMKBbHBHhpS9Wbfzn5L086+mZUOSKHp6ItBCRFSKyRUQ2i8i4UtqIiLwuIttF\nZIOIdHei1kDldNhNXriBtnfO4J3PNvHsjB9oe+cMEn5OdrQmY0xocUsPrwB4TFUTRaQ2sEZEvlDV\nLUXaXAu08370At70/teUw+mgA0j8JYXnZ/7IT2/dwXnNCxdD/vib7dz010Vs/2A41SOsp2eM8T9X\n9PBU9YCqJno/Twe2Aqc/gHQDMFML/QDUE5GYKi41oLgh7ADeW7aVh6/vcirsAIZd3pbWzWrzZeJe\nBysLXfaYgglFrgi8okSkNdANWH3at2KBor8d91EyFBGRESKSICIJ6alH/VWm67kl7ADSs/JoWCey\nxPGGdWuSnpXnQEXGmFDkqsATkWhgPjBeVY+fzTlUdYqqxqtqfO36DXxbYIBwU9gBDOrZmplLt5Jf\ncOLUsb0p6axcn8SAbnEOVhbarJdnQo1b7uEhIhEUht37qrqglCZJQIsiX8d5jxmXG3b5+cz6Yiv9\nx83jvsEXciwjlzc+Xs8z9/Wicb0op8szxoQIVwSeFO51MhXYqqqvlNFsITBGROZQOFklTVUPVFWN\ngcJtvTuA8GphzH9uKAtWbmfxD7uJrhnBh/81hN4X2i1YY0zVcUXgAX2Be4CNIrLOe2wC0BJAVScD\ni4EhwHYgC7jfgTpdy41BV1R4tTBuHdCeWwe0d7oUU8TQEYPsmTwTMlwReKr6LVDujpZauAba6Kqp\nKLC4PeyMMcYNXDVpxVSehZ05VzZ5xYQKC7wAZmFnjDEVZ4EXoCzsjC8NHTHIenom6LniHp6pOAs6\nY4w5O9bDCyAWdsYYc/Ys8AKEhZ2pCjasaYKZBV4AsLAzxphzZ4FnjCnGenkmWFnguZz17owxxjds\nlqZLWdAZY4xvWQ/PhSzsjNNsWNMEIws8l7GwM8YY/7DAcxELO+Mm1sszwcYCzyUs7Iwxxr9s0orD\nLOiMm53s5dmeeSYYWA/PQRZ2xhhTdVwReCIyTURSRGRTGd/vLyJpIrLO+/F0VdfoS7elTbSwMwHF\n7ueZYOCWIc0ZwCRgZjltvlHVoVVTjjHGmGDjih6eqq4EjjpdR1Wwnp0JVNbLM4HOLT28irhURNYD\n+4HHVXVzaY1EZAQwAqBRs9gqLK98FnTGGOMsV/TwKiARaKWqXYF/Ap+U1VBVp6hqvKrG167foMoK\nLI+FnQkW1sszgSwgAk9Vj6tqhvfzxUCEiDRyuKwKsbAzxhh3CIjAE5FmIiLez3tSWPcRZ6s6Mws7\nY4xxD1cEnojMBlYBHURkn4g8ICIPi8jD3ia3AJu89/BeB25XVXWq3oqwsDPByoY1TaByxaQVVb3j\nDN+fROFjC65nQWeMMe7kih5esLCwM6Fi6IhB1tMzAccCz0cs7Iwxxt0s8HzAws4YY9zPAs8Yc9Zs\nWNMEEgu8c2S9O2OMCQwWeGfJdjwwppD18kygsMA7CxZ0xhgTeCzwKsnCzpiSrJdnAoEFXiVY2Blj\nTOCywKsgCztjjAlsFnhnYJNTjKkYG9Y0bmeBVw4LOmOMCR4WeGWwsDOm8myNTeNmFnilsLAzxpjg\nY4FnjPE56+UZN7LAO4317owxJji5YgNYEZkGDAVSVLVzKd8X4DVgCJAFDFfVRF/WYEFnjDHBzS09\nvBnA4HK+fy3QzvsxAnjTlxe3sDPG92xY07iNKwJPVVcCR8tpcgMwUwv9ANQTkRhfXNvCzhhjQoMr\nhjQrIBbYW+Trfd5jB05vKCIjKOwFAmTc2b3Fz+Wd+E5fVXhuGgGHnS6iCtn7DW72fgNfK6cL8IdA\nCbwKU9UpwBSn66gMEUlQ1Xin66gq9n6Dm71f41auGNKsgCSgRZGv47zHjDHGmAoJlMBbCNwrhXoD\naapaYjjTGGOMKYsrhjRFZDbQH2gkIvuAZ4AIAFWdDCym8JGE7RQ+lnC/M5X6TUANwfqAvd/gZu/X\nuJKoqtM1GGOMMX4XKEOaxhhjzDmxwDPGGBMSLPAcJiLVRGStiCxyuhZ/E5HdIrJRRNaJSILT9fib\niNQTkXkisk1EtorIpU7X5C8i0sH7//Xkx3ERGe90Xf4kIn8Skc0isklEZotIpNM1mfLZPTyHicij\nQDxQR1WHOl2PP4nIbiBeVYPtId1Sici7wDeq+o6IVAeiVPWY03X5m4hUo/CxoV6q+qvT9fiDiMQC\n3wIXqmq2iMwFFqvqDGcrM+WxHp6DRCQOuA54x+lajG+JSF2gHzAVQFXzQiHsvK4CdgRr2BURDtQU\nkXAgCtjvcD3mDCzwnPUq8ATgcbqQKqLAMhFZ410CLpi1AQ4B071D1u+ISC2ni6oitwOznS7Cn1Q1\nCZgI7KFwicM0VV3mbFXmTCzwHCIiJ7dDWuN0LVXoMlXtTuHuF6NFpJ/TBflRONAdeFNVuwGZwJPO\nluR/3qHb64GPnK7Fn0SkPoWL2rcBmgO1RORuZ6syZ2KB55y+wPXe+1pzgCtFZJazJfmX969iVDUF\n+Bjo6WxFfrUP2Keqq71fz6MwAIPdtUCiqiY7XYifDQR2qeohVc0HFgB9HK7JnIEFnkNU9S+qGqeq\nrSkcAlquqkH7F6KI1BKR2ic/B64BNjlblf+o6kFgr4h08B66CtjiYElV5Q6CfDjTaw/QW0SivBtU\nXwVsdbgmcwauWFrMhISmwMeFvxsIBz5Q1c+dLcnvxgLve4f5dhJ8S+IV4/1D5mpgpNO1+JuqrhaR\neUAiUACsxZYYcz17LMEYY0xIsCFNY4wxIcECzxhjTEiwwDPGGBMSLPCMMcaEBAs8Y4wxIcECzxhj\nTEiwwDPGGBMS/h8gUfid7pgp0AAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn import svm, datasets\n", "\n", "# import some data to play with\n", "iris = datasets.load_iris()\n", "X = iris.data[:, :2] # we only take the first two features. We could\n", " # avoid this ugly slicing by using a two-dim dataset\n", "Y = iris.target\n", "\n", "\n", "def my_kernel(X, Y):\n", " \"\"\"\n", " We create a custom kernel:\n", "\n", " (2 0)\n", " k(X, Y) = X ( ) Y.T\n", " (0 1)\n", " \"\"\"\n", " M = np.array([[2, 0], [0, 1.0]])\n", " return np.dot(np.dot(X, M), Y.T)\n", "\n", "\n", "h = .02 # step size in the mesh\n", "\n", "# we create an instance of SVM and fit out data.\n", "clf = svm.SVC(kernel=my_kernel)\n", "clf.fit(X, Y)\n", "\n", "# Plot the decision boundary. For that, we will assign a color to each\n", "# point in the mesh [x_min, x_max]x[y_min, y_max].\n", "x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n", "y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n", "xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n", "Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n", "\n", "# Put the result into a color plot\n", "Z = Z.reshape(xx.shape)\n", "plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired)\n", "\n", "# Plot also the training points\n", "plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired, edgecolors='k')\n", "plt.title('3-Class classification using Support Vector Machine with custom kernel')\n", "plt.axis('tight')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "heading_collapsed": true }, "source": [ "### 随机森林" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "hidden": true }, "source": [ "#### 信息熵\n", "\n", "先抛出信息熵公式如下：\n", "\n", "$$ H(X)=-\\sum_{i=1}^{n}p(x_i)log p(x_i) $$\n", "\n", "其中代表$p(x_i)$随机事件X为$x_i$的概率\n", "\n", "\n", "###### 信息量\n", "\n", "信息量是对信息的度量，就跟时间的度量是秒一样，当我们考虑一个离散的随机变量x的时候，当我们观察到的这个变量的一个具体值的时候，我们接收到了多少信息呢？\n", "\n", "多少信息用信息量来衡量，我们接受到的信息量跟具体发生的事件有关。\n", "\n", "信息的大小跟随机事件的概率有关。\n", "\n", "越小概率的事情发生了产生的信息量越大，如湖南产生的地震了；\n", "\n", "越大概率的事情发生了产生的信息量越小，如太阳从东边升起来了（肯定发生嘛，没什么信息量）。\n", "\n", "所以一个具体事件的信息量应该是随着其发生概率而递减的，且不能为负。\n", "\n", "但是这个表示信息量的函数形式怎么找呢？\n", "\n", "随着概率增大而减少的函数形式太多了！不要着急，我们还有下面这条性质\n", "\n", "- 如果我们有俩个不相关的事件x和y，那么我们观察到的俩个事件同时发生时获得的信息应该等于观察到的事件各自发生时获得的信息之和，即：\n", "\n", "$$ h(x,y) = h(x)+h(y) $$\n", "\n", "- 由于x，y是俩个不相关的事件，那么满足\n", "\n", "$$ p(x,y) = p(x)*p(y) $$\n", "\n", "根据上面推导，我们很容易看出h(x)一定与p(x)的对数有关（因为只有对数形式的真数相乘之后，能够对应对数的相加形式，可以试试）。因此我们有信息量公式如下：\n", "\n", "$$ h(x)=-log_2 p(x) $$\n", "\n", "\n", "下面解决俩个疑问？\n", "\n", "- （1）为什么有一个负号\n", "\n", " 其中，负号是为了确保信息量一定是正数或者是0，总不能为负数吧！\n", "\n", "- （2）为什么底数为2\n", "\n", " 这是因为，我们只需要信息量满足低概率事件x对应于高的信息量。那么对数的选择是任意的。\n", " 我们只是遵循信息论的普遍传统，使用2作为对数的底！\n", "\n", "下面我们正式引出信息熵。\n", "\n", "信息量度量的是一个具体事件发生了所带来的信息，而熵则是在结果出来之前对可能产生的信息量的期望\n", "\n", "考虑该随机变量的所有可能取值，即所有可能发生事件所带来的信息量的期望。即\n", "\n", "$$ H(X)=-\\sum_{i=1}^{n}p(x_i)log p(x_i) $$\n", "\n", "最终我们的公式来源推导完成了。\n", "\n", "###### 额外理解\n", "\n", "这里我再说一个对信息熵的理解。信息熵还可以作为一个系统复杂程度的度量，如果系统越复杂，出现不同情况的种类越多，那么他的信息熵是比较大的。\n", "\n", "如果一个系统越简单，出现情况种类很少（极端情况为1种情况，那么对应概率为1，那么对应的信息熵为0），此时的信息熵较小。\n", "\n", "#### 条件熵\n", "\n", "条件熵的定义是：定义为X给定条件下，Y的条件概率分布的熵对X的数学期望\n", "\n", "这个还是比较抽象，下面我们解释一下：\n", "\n", "设有随机变量（X,Y），其联合概率分布为 \n", "\n", "$$ p(X-x_i,Y=y_i)=p_i i=1,2,..,n;j=1,2,...,n $$\n", "\n", "条件熵$H（Y|X）$表示在已知随机变量X的条件下随机变量Y的不确定性。\n", "\n", "在随机变量X给定的条件下，随机变量$Y$的条件熵$H(Y|X)$\n", "\n", "下面推导一下条件熵的公式：\n", "\n", "$$\n", "\\begin{align}\n", "H(Y|X)&=\\sum_{x\\in X}p(x)H(Y|X=x)\\\\\n", "&=-\\sum_{x\\in X}p(x)\\sum_{y\\in Y}p(y|x)log p(y|x)\\\\\n", "&=-\\sum_{x\\in X}\\sum_{y\\in Y}p(x,y)log p(y|x)\n", "\\end{align}\n", "$$\n", "\n", "> 注意:这个条件熵，不是指在给定某个数（某个变量为某个值）的情况下，另一个变量的熵是多少，变量的不确定性是多少，而是期望！\n", "\n", "> 因为条件熵中X也是一个变量，意思是在一个变量X的条件下（变量X的每个值都会取），另一个变量Y熵对X的期望。\n", "\n", "> 这是最容易错的！\n", "\n", "###### 例子\n", "\n", "下面通过例子来解释一下：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW4JgTic2cTicmAAG58IRRP6cbZ3rB424gfRzdYXpMrdEUOln0fh7VW5iaSKohMOQtBTERrhyGrseYFUQ/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "假如我们有上面数据：\n", "\n", "设随机变量$Y={嫁，不嫁}$\n", "\n", "我们可以统计出，嫁的个数为$\\frac{6}{12} = \\frac{1}{2}$，不嫁的个数为$\\frac{6}{12} = \\frac{1}{2}$\n", "\n", "那么Y的熵，根据熵的公式来算，可以得到$H（Y） = -\\frac{1}{2}log\\frac{1}{2} -\\frac{1}{2}log\\frac{1}{2}$\n", "\n", "为了引出条件熵，我们现在还有一个变量X，代表长相是帅还是帅，当长相是不帅的时候，统计如下红色所示：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW4JgTic2cTicmAAG58IRRP6cbyagkdhbuynAuxp00jFT3aibW4oMcRhUqTaEHDPBMGSnhIicYxvozvkEg/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "可以得出，当已知不帅的条件下，满足条件的只有4个数据了，这四个数据中，不嫁的个数为1个，占$\\frac{1}{4}$\n", "\n", "嫁的个数为3个，占$\\frac{3}{4}$\n", "\n", "那么此时的$H（Y|X = 不帅） = -\\frac{1}{4}log\\frac{1}{4}-\\frac{3}{4}log\\frac{3}{4}$\n", "\n", "$p(X = 不帅) = \\frac{4}{12} = \\frac{1}{3}$\n", "\n", "同理我们可以得到：当已知帅的条件下，满足条件的有8个数据了，这八个数据中，不嫁的个数为5个，占$\\frac{5}{8}$\n", "\n", "嫁的个数为3个，占$\\frac{3}{8}$\n", "\n", "那么此时的$H（Y|X = 帅） = -\\frac{5}{8}log\\frac{5}{8}-\\frac{3}{8}log\\frac{3}{8}$\n", "\n", "$p(X = 帅) = \\frac{8}{12} = \\frac{2}{3}$\n", "\n", "5\n", " 计算结果\n", "\n", "有了上面的铺垫之后，我们终于可以计算我们的条件熵了，我们现在需要求：$H（Y|X = 长相）$\n", "\n", "也就是说，我们想要求出当已知长相的条件下的条件熵。\n", "\n", "根据公式我们可以知道，长相可以取帅与不帅俩种\n", "\n", "条件熵是另一个变量Y熵对X（条件）的期望。\n", "公式为：\n", "\n", "$$ H(Y|X)=\\sum_{x\\in X}p(x)H(Y|X=x) $$\n", "\n", "$$H（Y|X=长相） = p(X =帅)*H（Y|X=帅）+p(X =不帅)*H（Y|X=不帅）$$\n", "\n", "然后将上面已经求得的答案带入即可求出条件熵！\n", "\n", "这里比较容易错误就是忽略了X也是可以取多个值，然后对其求期望！！\n", "\n", "###### 总结\n", "\n", "其实条件熵意思是按一个新的变量的每个值对原变量进行分类.\n", "\n", "比如上面这个题把嫁与不嫁，帅与不帅分成了俩类。\n", "\n", "然后在每一个小类里面，都计算一个小熵，然后每一个小熵乘以各个类别的概率，然后求和。\n", "\n", "我们用另一个变量对原变量分类后，原变量的不确定性就会减小了，因为新增了Y的信息，可以感受一下。\n", "\n", "不确定程度减少了多少就是信息的增益。\n", "\n", "#### 决策树（decision tree）\n", "\n", "决策树（decision tree）是一个树结构（可以是二叉树或非二叉树）。\n", "\n", "其每个非叶节点表示一个特征属性上的测试，每个分支代表这个特征属性在某个值域上的输出，而每个叶节点存放一个类别。\n", "\n", "使用决策树进行决策的过程就是从根节点开始，测试待分类项中相应的特征属性，并按照其值选择输出分支，直到到达叶子节点，将叶子节点存放的类别作为决策结果。\n", "\n", "总结来说,决策树模型核心是下面几部分：\n", "- 结点和有向边组成\n", "- 结点有内部结点和叶结点俩种类型\n", "- 内部结点表示一个特征，叶节点表示一个类\n", "\n", "决策树表示如下：\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW58E4B2icMtDt1qRxwxwcFlj54lCoavXcBJex9AibsBwb5EUg7KGAk3Nic3RbMK9UpCwjUAfbxMcymXg/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "决策树代表实例属性值约束的合取的析取式。\n", "\n", "从树根到树叶的每一条路对应一组属性测试的合取，树本身对应这些合取的析取。\n", "\n", "理解这个式子，比如上图的决策树对应表达式为：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW44oWxtJibIpS5YbPkOOiabCPsbLYkonY6VG8SMicSrWKia6vIibvY6FGrLoUlnubSX32kGI6K3cXOVia5g/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "###### 决策实例\n", "\n", "假如我现在告诉你，我买了一个西瓜，它的特点是纹理是清晰，根蒂是硬挺的瓜，你来给我判断一下是好瓜还是坏瓜，恰好，你构建了一颗决策树，告诉他，没问题，我马上告诉你是好瓜，还是坏瓜？\n", "\n", "判断步骤如下：\n", "\n", "根据纹理特征，已知是清晰，那么走下面这条路，红色标记：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_jpg/nJZZib3qIQW58E4B2icMtDt1qRxwxwcFljibXdFA6zubP3RjjoSFv6ZnI1mo5ypOicibEiaCdPRROlSMfgOAr5bdVaMg/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "\n", "好的，现在咋们到了第二层了，这个时候，由决策树图，我们看到，我们需要知道根蒂的特征是什么了？很好，他也告诉我了，是硬挺，于是，我们继续走，如下面蓝色所示：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_jpg/nJZZib3qIQW58E4B2icMtDt1qRxwxwcFljiaiaeyIzuXYIHa7ibW0Q3BBPr4njY84aTP8QIekXpZibGYBibZVK9zQA3iaQ/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "此时，我们到达叶子结点了，根据上面总结的点，可知，叶子结点代表一种类别，我们从如上决策树中，可以知道，这是一个坏瓜！\n", "\n", "于是我们可以很牛的告诉他，你买的这个纹理清晰，根蒂硬挺的瓜是坏瓜，orz！\n", "\n", "###### 回归源头\n", "\n", "根据上面例子，非常容易直观的得到了一个实例的类别判断，只要你告诉我各个特征的具体值，决策树的判定过程就相当于树中从根结点到某一个叶子结点的遍历。每一步如何遍历是由数据各个特征的具体特征属性决定。\n", "\n", "好的，可能有人要问了，说了这么多，给你训练数据，你的决策树是怎么构建的呢？没有树，谈何遍历，谈何分类？\n", "\n", "于是构建决策树也就成为了最重要的工作！\n", "\n", "比如，给我下面训练数据，我如何构建出决策树\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW58E4B2icMtDt1qRxwxwcFljY9M7qR6bBG10CFic2AOyXH8vibMTTESnoIjpDdys7H8pRyavNJkDr5WQ/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "我们可以从上面决策树看出，每一次子结点的产生，是由于我在当前层数选择了不同的特征来作为我的分裂因素造成的。\n", "\n", "比如下图用红色三角形表示选择的特征：\n", "![](http://mmbiz.qpic.cn/mmbiz_jpg/nJZZib3qIQW58E4B2icMtDt1qRxwxwcFljBQTwMfmL4oaR5vEkCs6OCkegicbBmEMTVJYfnMh2tDA4s4ibCzonrg3g/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "每一层选择了指定的特征之后，我们就可以继续由该特征的不同属性值进行划分，依次一直到叶子结点。\n", "\n", "看起来一切很顺利！但是细心的小伙伴可能会问了，为什么在第一次选择特征分裂的时候，不选择触感呢？而是选择纹理，比如如下：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_jpg/nJZZib3qIQW58E4B2icMtDt1qRxwxwcFljfl5Xia8eycRtsmYlnYO7BfjabMoEFfWtdkcctaDBOJGiblp9duliaghog/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "不换成触感，或者其它特征为什么也不行呢？为什么选择的是纹理，这是以什么标准来选择特征的？这就是我们要说的决策树的关键步骤是分裂属性。\n", "\n", "所谓分裂属性就是在某个节点处按照某一特征属性的不同划分构造不同的分支，其目标是让各个分裂子集尽可能地“纯”。尽可能“纯”就是尽量让一个分裂子集中待分类项属于同一类别。\n", "\n", "而判断“纯”的方法不同引出了我们的ID3算法，C4.5算法以及CART算法，这些后面会详细介绍！\n", "\n", "#### ID3\n", "\n", "我们既然希望划分之后结点的“纯度”越来越高，那么如何度量纯度呢？\n", "\n", "“信息熵”是度量样本集合不确定度（纯度）的最常用的指标。\n", "\n", "在我们的ID3算法中，我们采取`信息增益`这个量来作为纯度的度量。\n", "\n", "我们选取使得信息增益最大的特征进行分裂！那么信息增益又是什么概念呢？\n", "\n", "我们前面说了，信息熵是代表随机变量的复杂度（不确定度），条件熵代表在某一个条件下，随机变量的复杂度（不确定度）。\n", "\n", "而我们这里说的的信息增益恰好是：信息熵-条件熵。\n", "\n", "我们看如下定义：\n", "\n", "$$ Gain(D,a)=Ent(D)-\\sum_{v=1}^{V}\\frac{|D_v|}{|D|}Ent(D^v) $$\n", "\n", "当前样本集合D 中第 k 类样本所占的比例为 $p_k$，则 D 的信息熵定义为\n", "\n", "$$ Ent(D)=-\\sum_{k=1}^{|y|}p_k log_2 p_k $$\n", "\n", "离散属性a 有 V 个可能的取值$ {a1,a2,…,aV}$；样本集合中，属性 a 上取值为 $a_v$ 的样本集合，记为$ D_v $。\n", "\n", "用属性a 对样本集 D 进行划分所获得的“信息增益”\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_jpg/nJZZib3qIQW4vnPfz1ftnuLWcEG12oNVNuiapssk3nr8grMSd4wvvYkzEFdibrHpTy6qC73XHZibCbRea52Uia0yW3g/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "信息增益表示得知属性 a 的信息而使得样本集合不确定度减少的程度\n", "\n", "那么我们现在也很好理解了，在决策树算法中，我们的关键就是每次选择一个特征，特征有多个，那么到底按照什么标准来选择哪一个特征。\n", "\n", "对于ID3算法来说，这个问题就可以用信息增益来度量。\n", "\n", "如果选择一个特征后，信息增益最大（信息不确定性减少的程度最大），那么我们就选取这个特征。\n", "\n", "好的，我们现在已经知道了选择指标了，就是在所有的特征中，选择信息增益最大的特征。那么如何计算呢？看下面例子：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW4vnPfz1ftnuLWcEG12oNVNn0ibggH0BSue92Jxdm0u8HLlOdZXBeKNvUE3UVOVgNibL2GeDBLV5RicA/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "正例(好瓜)占 $\\frac{8}{17}$，反例占 $\\frac{9}{17}$ ，根结点的信息熵为\n", "\n", "$$ Ent(D)=-\\sum_{k=1}^{2}p_k log_2 p_k=-(\\frac{8}{17}log_2\\frac{8}{17}+\\frac{9}{17}log_2\\frac{9}{17})=0.998 $$\n", "\n", "计算当前属性集合{色泽，根蒂，敲声，纹理，脐部，触感}中每个属性的信息增益\n", "\n", "色泽有3个可能的取值：{青绿，乌黑，浅白}\n", "D1(色泽=青绿) = {1, 4, 6, 10, 13, 17}，正例 3/6，反例 3/6\n", "D2(色泽=乌黑) = {2, 3, 7, 8, 9, 15}，正例 4/6，反例 2/6\n", "D3(色泽=浅白) = {5, 11, 12, 14, 16}，正例 1/5，反例 4/5\n", "\n", "3 个分支结点的信息熵\n", "\n", "$$ Ent(D^1)=-(\\frac{3}{6}log_2\\frac{3}{6}+\\frac{3}{6}log_2\\frac{3}{6})=1.000 $$\n", "$$ Ent(D^2)=-(\\frac{4}{6}log_2\\frac{4}{6}+\\frac{2}{6}log_2\\frac{2}{6})=0.918 $$\n", "$$ Ent(D^3)=-(\\frac{1}{5}log_2\\frac{1}{5}+\\frac{4}{5}log_2\\frac{4}{5})=0.722 $$\n", "\n", "那么我们可以知道属性色泽的信息增益是：\n", "\n", "$$\n", "\\begin{align}\n", "Gain(D,色泽)&=Ent(D)-\\sum_{v=1}^{3}\\frac{|D_v|}{|D|}Ent(D^v)\\\\\n", "&=0.998-(\\frac{6}{17}\\mu1.000+\\frac{6}{17}\\0.918+\\frac{5}{17}\\0.722)\n", "&=0.109\n", "\\end{align}\n", "$$\n", "\n", "同理，我们可以求出其它属性的信息增益，分别如下：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW4vnPfz1ftnuLWcEG12oNVNSA9za9TnGQjiaF6Gv2ibx8XycV9O86leRJOkcqyZeYlXyfE9mAKAahSQ/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "于是我们找到了信息增益最大的属性纹理，它的Gain(D，纹理) = 0.381最大。\n", "\n", "所以我们选择的划分属性为“纹理”\n", "如下：\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW4vnPfz1ftnuLWcEG12oNVNRibVrblfz2CYKMq0Ih4nxGsSgwtfVxtQLOQG13ibAlEFyrJicnPNazekw/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "根据纹理属性划分后，我们可以得到了三个子结点。\n", "对于这三个子节点，我们可以递归的使用刚刚找信息增益最大的方法进行选择特征属性，\n", "\n", "比如：D1(纹理=清晰) = {1, 2, 3, 4, 5, 6, 8, 10, 15}，第一个分支结点可用属性集合{色泽、根蒂、敲声、脐部、触感}，基于 D1各属性的信息增益，分别求的如下：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW4vnPfz1ftnuLWcEG12oNVN8muO54AFECJxFtwRXISoN46DdguU0l0TFiclKBlicAzrvTImibFQESnHA/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "于是我们可以选择特征属性为根蒂，脐部，触感三个特征属性中任选一个（因为他们三个相等并最大）。\n", "其它俩个子结点同理，然后得到新一层的结点，再递归的由信息增益进行构建树即可\n", "我们最终的决策树如下：\n", "\n", "![](http://mmbiz.qpic.cn/mmbiz_png/nJZZib3qIQW4vnPfz1ftnuLWcEG12oNVNjre21sX22udBF57mEoofee4Rv4g1jvdGYqGyibD1qVuOKvic5nau7ibcA/640?tp=webp&wxfrom=5&wx_lazy=1)\n", "\n", "啊，那到这里为止，我们已经知道了构建树的算法，上面也说了有了树，我们直接遍历决策树就能得到我们预测样例的类别。\n", "\n", "那么是不是大功告成了呢？\n", "\n", "结果是：不是的\n", "\n", "我们从上面求解信息增益的公式中，其实可以看出，信息增益准则其实是对可取值数目较多的属性有所偏好！\n", "\n", "现在假如我们把数据集中的“编号”也作为一个候选划分属性。我们可以算出“编号”的信息增益是0.998\n", "\n", "因为每一个样本的编号都是不同的（由于编号独特唯一，条件熵为0了，每一个结点中只有一类，纯度非常高啊）。\n", "也就是说，来了一个预测样本，你只要告诉我编号，其它特征就没有用了，这样生成的决策树显然不具有泛化能力。\n", "\n", "于是我们就引入了信息增益率来选择最优划分属性！\n", "而信息增益率也是C4.5算法的核心思想。下面就讲解C4.5算法\n", "\n", "#### C4.5算法\n", "\n", "这次我们每次进行选取特征属性的时候，不再使用ID3算法的信息增益，而是使用了信息增益率这个概念。\n", "\n", "首先我们来看信息增益率的公式：\n", "\n", "$$ Gain_{4.5}(D,a)=\\frac{Gain(D,a)}{TV(a)} $$\n", "\n", "由上图我们可以看出，信息增益率=信息增益/IV(a),说明信息增益率是信息增益除了一个属性a的固有值得来的。\n", "\n", "我们一开始分析到，信息增益准则其实是对可取值数目较多的属性有所偏好！（比如上面提到的编号，如果选取编号属性，每一个子节点只有一个实例，可取值数目是最多，而且子节点纯度最高《只有一个类别》，导致信息增益最大，所以我们会倾向于选他，但是已经分析了这种树是不具备泛化能力的）。\n", "\n", "但是刚刚我们分析到了，信息增益并不是一个很好的特征选择度量。于是我们引出了信息增益率。\n", "\n", "我们来看IV(a)的公式：\n", "属性a的固有值：\n", "\n", "$$IV(a)=-\\sum_{v=1}^{V}\\frac{|D^v|}{|D|}log_2\\frac{|D^v|}{|D|}$$\n", "\n", "IV(触感) = 0.874 ( V = 2 )\n", "IV(色泽) = 1.580 ( V = 3 )\n", "IV(编号) = 4.088 ( V = 17 )\n", "\n", "由上面的计算例子，可以看出IV(a)其实能够反映出，当选取该属性，分成的V类别数越大，IV(a)就越大，如果仅仅只用信息增益来选择属性的话，那么我们偏向于选择分成子节点类别大的那个特征。\n", "\n", "但是在前面分析了，并不是很好，所以我们需要除以一个属性的固定值，这个值要求随着分成的类别数越大而越小。于是让它做了分母。\n", "\n", "这样可以避免信息增益的缺点。\n", "因为一开始我仅仅用信息增益作为我的选择目标，但是会出现“编号”这些使得类别数目多的属性选择，但是又不具有泛化能力，所以我给他除以一个值（这个值）随着你分的类别越多，我就越大，一定程度上缓解了信息增益的缺点\n", "\n", "那么信息增益率就是完美无瑕的吗？\n", "\n", "当然不是，有了这个分母之后，我们可以看到增益率准则其实对可取类别数目较少的特征有所偏好！\n", "\n", "毕竟分母越小，整体越大。\n", "\n", "所以C4.5算法不直接选择增益率最大的候选划分属性，候选划分属性中找出信息增益高于平均水平的属性（这样保证了大部分好的的特征），再从中选择增益率最高的（又保证了不会出现编号特征这种极端的情况）\n", "\n", "#### 随机森林\n", "\n", "通过两个随机性，构造不同的次优树\n", "\n", "1.随机选择样本，通过有放回的采样，重复的选择部分样本来构造树 \n", "2.构造树的过程中，每次随机考察部分特征，不对树进行裁剪\n", "\n", "> 单树采用CART树\n", "\n", "对于森林中的单棵树，分类强度越大越好；对于森林中的多棵树，树之间的相关度越小越好。\n", "\n", "在生成一定数目的次优树之后，随机森林的输出采用简单的多数投票机制（针对分类）或采用单棵树输出结果的简单平均（针对回归）得到。\n", "\n", "当然可以使用决策树作为基本分类器，但也可以使用SVM、Logistic回归等其他分类器，习惯上，这些分类器组成的“总分类器”，仍然叫做随机森林。\n", "\n", "###### GBDT(Gradient Boost Regression Tree)\n", "\n", "假设输入数据x可能属于5个分类（分别为1,2,3,4,5），训练数据中，x属于类别3，则y = (0, 0, 1, 0, 0)，\n", "\n", "假设模型估计得到的F(x) = (0, 0.3, 0.6, 0, 0)，\n", "\n", "则经过Logistic变换后的数据p(x) = (0.16,0.21,0.29,0.16,0.16)，y - p得到梯度g：(-0.16, -0.21, 0.71, -0.16, -0.16)。\n", "\n", "观察这里可以得到一个比较有意思的结论：\n", "\n", "假设gk为样本当某一维（某一个分类）上的梯度:\n", "\n", "- $gk>0$时，越大表示其在这一维上的概率p(x)越应该提高，比如说上面的第三维的概率为0.29，就应该提高，属于应该往“正确的方向”前进\n", " 越小表示这个估计越“准确”\n", "- $gk<0$时，越小，负得越多表示在这一维上的概率应该降低，比如说第二维0.21就应该得到降低。属于应该朝着“错误的反方向”前进\n", " 越大，负得越少表示这个估计越“不错误 ”\n", "\n", "总的来说，对于一个样本，最理想的梯度是越接近0的梯度。\n", "\n", "所以，我们要能够让函数的估计值能够使得梯度往反方向移动（>0的维度上，往负方向移动，$<0$的维度上，往正方向移动）最终使得梯度尽量=0，并且该算法在会严重关注那些梯度比较大的样本，跟Boost的意思类似。\n", "\n", "得到梯度之后，就是如何让梯度减少了。\n", "\n", "这里是用的一个迭代+决策树的方法，当初始化的时候，随便给出一个估计函数F(x)（可以让F(x)是一个随机的值，也可以让F(x)=0），然后之后每迭代一步就根据当前每一个样本的梯度的情况，建立一棵决策树。\n", "\n", "就让函数往梯度的反方向前进，最终使得迭代N步后，梯度越小。\n", "\n", "这里建立的决策树和普通的决策树不太一样，首先，这个决策树是一个叶子节点数J固定的，当生成了J个节点后，就不再生成新的节点了。\n", "\n", "算法的流程如下\n", "\n", "![](RandomForest0.jpg)\n", "\n", "0. 表示给定一个初始值\n", "1. 表示建立M棵决策树（迭代M次）\n", "2. 表示对函数估计值F(x)进行Logistic变换\n", "3. 表示对于K个分类进行下面的操作（其实这个for循环也可以理解为向量的操作，每一个样本点xi都对应了K种可能的分类yi，所以yi, F(xi), p(xi)都是一个K维的向量，这样或许容易理解一点）\n", "4. 表示求得残差减少的梯度方向\n", "5. 表示根据每一个样本点x，与其残差减少的梯度方向，得到一棵由J个叶子节点组成的决策树\n", "6. 为当决策树建立完成后，通过这个公式，可以得到每一个叶子节点的增益（这个增益在预测的时候用的）\n", " 每个增益的组成其实也是一个K维的向量，表示如果在决策树预测的过程中，如果某一个样本点掉入了这个叶子节点，则其对应的K个分类的值是多少。比如说，GBDT得到了三棵决策树，一个样本点在预测的时候，也会掉入3个叶子节点上，其增益分别为（假设为3分类的问题）：\n", "(0.5, 0.8, 0.1), (0.2, 0.6, 0.3), (0.4, 0.3, 0.3)，那么这样最终得到的分类为第二个，因为选择分类2的决策树是最多的。\n", "7. 的意思为，将当前得到的决策树与之前的那些决策树合并起来，作为新的一个模型(跟6中所举的例子差不多)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "hidden": true, "scrolled": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnYAAAI4CAYAAADnBqTlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FNUawOHfN5st6YTQi4CKCkhRsRdAAXsB1GvFeu29\ni4pcFbvX3rteKyoKiCJ2EUHAAqIiSm8BQknZvnPuHzOBTbK7SWCzm3Le5/GR7LRvy5z55swpopRC\n0zRN0zRNa/yMdAegaZqmaZqmJYdO7DRN0zRN05oIndhpmqZpmqY1ETqx0zRN0zRNayJ0YqdpmqZp\nmtZE6MRO0zRN0zStidCJXZqJyCgReSHZ69ZiX0pEdk7GvuqLiNwjIldt47Zni8g0+99uEflTRFon\nN0JNi01EVojIwHTHkSwisrOI1MvYWCKyQEQOrqd9XyoiD27jtoNFZIn9bxGR2SKy23bEcqL9uygT\nkd7bup/6ICLtRWSaiJSKyH3pjqequpxPInK+iHxdvxFVOl5b+/rirod9b3kvIpJpnyuFNW2nE7sk\nspOJeSLiFZE1IvK0iLRItI1S6m6l1Pm12X9d1t0eIvK1iPjtk7xEROaIyE11+eFub+JoJ2EjgWft\nv/cTkakiskFE1onIOBFpX5t9KaUCwEvATdsaj7b97AtaxX+miPii/j493fFViL6g19P+dxCR8SKy\nXkQ222XGmSKSZZ9vh8TY5nERedv+94qoz26TiHwvIheIiNRXzMkiIv8TkTHRrymldlVKfVcPx3ID\no4AH7b97iMhEu/zYICKfiEj32uxLWQO+/hf4z3aE9BBwoVIqRyk1r0qsGXaZ2XU79r89LgJWAXlK\nqRvTFEPKichdIvLKdu5mFPCCfZ2puFGJLuvCIjI+xrHPtb/zs2tzEKWUD3gVuKGmdXVilyQici1w\nH3A9kA/sB3QBpoqIK842GamLsM4uU0rlAu2Ba4FTgMkpvHicDUy2f8wABcBzQFesz7UUeLkO+3sT\nOKs+7qq02rEvaDlKqRxgGXBs1GtvVF2/gZ8f2+MNYBGwA1AInAWsVUp5gXFYNzRbiIgT6/x7Nerl\nI+3PsSvwANbF5bnaHFxEDBFpDmX/cGCuUmqN/Xc+8AGwK9AW+AWodsFN4ENgqIi0qWsg9ufdGZhf\n122j9lGf50MX4He1DTMWNOHztEYikgmciXVOA1tuVCrKuTyshHlcle0KsRK0P+p4yDeAc+wyIT6l\nlP5vO/+zv7wy4OQqr+cA64Bz7b/HAO8B/wNKgPPt1/4Xtc1IYClQDNwGLAEGR23/P/vfXQGFdVFY\nBqwHbonazz7AD8AmYDXwBOCKWq6AneO8n6+B86u8tgPgBY6paf/At/b+y+3P5V9Yidkk+/PYaP+7\nU4LP9EvgjATL9wRKo/4uBCbYn+uPwJ3AtCrbLAQGpPv3ov9TRP+uo167C3gHeAsrcT/bPlfGRK0z\nGFgS9fcK4BpgHrDZ3tYdtXw41gW8BPgbGGq/fj5WoVoK/FPxe8e6+PsA0/7tlgFtsG6CR9nrrgfe\nBgqijnO2fd6ux6oZXgEMjPPe/cDucZYdYp9TnqjXjrPPMUfUex5YZbv97Zh3i7PfafY58YP9/roC\nLbBujlbb+7wDMOz1HcDDWOXQIuAy7IqrWDHY390rVd7HDPs7WY518bsECAFB+3MdX3VfgAd4zI5p\nJVYtWUW5Mtj+3dyAVY6sAkYm+I29BtyUYHkbrHIq3/47C3gdq3yaD9wY/Vuz1/kKOD3O/gxgtP07\nWAu8gnVtyLbfb0WZuCDGttOpXGaOiHq/o4A19ndVCExmazk6EehY5Xv+j72/UuBToGXU+3vT/k43\nYZWTrez3HP29DKzl9xAdV8VrN0d9N8cCx2CVuxuAG6p8Vsk6n1pjXU9KsH5zY4Gvo5Y/YW9fAswC\nDrBfP8Z+zyH7fc9JVDbEOfahwJ8Jlh+GdQ5kVnn9BeAC+/s6u7bvxV5nMXBgovK1Ody1pcIBWCfC\nB9EvKqXKsE7CIVEvH4+V3LUgKssHEJGewFPA6Vg1ZflAxxqOfRDWHehhwGgR6WG/HgGuxjpx97eX\nX1LH9xX9XpYBs4GKtjBx96+UqniU1FdZdy7vYJ3IL2PdGe6AdXF5IsEhewMLEiw/hMp3v09iXTDb\nA+fa/1X1B9A3wT619BuGdfHJx0ryauNkrHNsR2AvrCQCETkA6xH8tVjn2yCsiwVAEXA01oX338Dj\nItJHKbUZ64K0TG2tTVyL9Vs/Gut31wnrQvCYfZzeWL/l07DO1w5AuwTxzgCeFpF/iUjnKsu+w7oI\nnhD12pnAG0qpSLwdKqV+wLrIJmqrdibWeZGHdaF7Hes83AnrczsaOMde92JgKNb5sjfWZ1wrItIN\nq9z7L1YisgcwTyn1FNZ3erf9uQ6LsflooD/Qx97uQKxkoUInIBPrM74I63PMixNKbcqQFfZ3DlZi\n2xnrd3QU1k1zVYnKkPOBM7ASo52wbmYfVUqVY/3+AHoppXaNE0vF8hyl1Pv2352wKgh2wCpfDeB5\n++8uWEnJo1X2dZode1uspPIa+/VzsJK7TljfyyWAXyl1JpW/l6+p3fcQHVfFawbWd3Mn8CJWTfMe\n9mdyh4jsYK+bzPPpaawkrB1WslS17J9pv4+WWNfecSLiVkpNAu7HOrdylFJ72evHLBviHLum39hZ\nwDi19clTRbnUGyu5q+t7gVpcx3RilxytgPVKqXCMZavt5RV+UEp9qJQyo79s24nARKXUNKVUEOvk\nqqlq/D9KKZ9S6lfgV+wvXCk1Ryk1QykVVkotwWqrNqDub62SVVgnR533r5QqVkq9r5TyKqVKse5E\nEsXTAusHXo19ko3GeuyNiDiw7nBHK6XKlVK/UfmxVYVSthawWsM0TSk1Mc75Ec8jSqk1SqlirLvd\nfvbr5wHPK6W+sPe3XCm1AMA+xiJl+RL4gsRJ0UXAKKXUSqWUH6tW5CT7EdtJwIdKqe+V1c5mFJCo\nycJwrJqz24GlIvKTiOxlx6WwappGAthtdI8l9u+5qi3nZxwvKaX+UEqFsC76g4Gr7XOyCHgE60IM\nViL3sFJqhf253luL41c4A/hEKfWuXT6sV0r9UsttT8eqoV1nJ9R3YCfqNj9wl1IqpJSaAASAXeLs\nK1EZsgNWInFN1Msn2/veqJRaSuwbz0RlyOnAg0qpxXYZNwo4bTsfe4exPo+gXc6vU0qNt/9dAtxN\n9XL0RaXUQrX10X7F+RDCuhbtrJSKKKVm25UP8d5Lou+hUlz2a37gXvv39TZW7dPDSqkypdRcrASo\nIkFKyvlkP5I8AbjN/h3Pxbph2UIp9bpSaoN9fb4fK2GL2/67jmVDot9YDta5/krUaxlYlRCXKaXM\nur4XW43XMZ3YJcd6oFWctgbt7eUVlifYT4fo5faJWVzDsddE/duLdReFiOwiIpPsThwVBUCrWDuo\ng45YtQl13r/dMPxZEVlqr/8t0MJOymLZCOTG2M/OwCfAlWprg+vWQAaVP9ulVbe197cp4TvU0i3R\n+RFPzHMAq/bln1gbiMgxIjLTbkS/Cat2KtH5sQMw0e6ssAnr0S9Yj/Oqnrdl2OdJLPZF5galVE+s\nBGs+ldt6vQYMEZG2WMnGH6pKY/s4tpyfcUR/tl0AN1AU9Z6etOOh6nsi9vkUT9zPvRY6VDnWUio/\ntVhfpeYy+vuuKl4Z0gb4DKs2LbrtU3u2rwyJFbsLq3zaVkX2TT5gJQsi8oKILLPL0S+p/ruNdz68\nAnwOvCsiK0Xk3gTt42r6HirFZYv+biqSvaKo5b6oWJJ1PrXFajYQ93sTkRvE6rW6Ges3kU3ia1Vd\nyoaYvzHbicAapdS0qNcuB2YppWZty3ux1Xgd04ldcvyAdec4PPpFO2M/Eivjr5CoBm41VrV0xfaZ\nWFXm2+Jp4E+gu1Iqj5prERKyHxnthfWoaFv2fy3WI+N97fUrHj3E22YuVe7ERaQLVsF0p1Iq+k5m\nHdYdZPRjrR2orgdWrabWcFU9P8qxHh9VSPRIpqrlWI/EKrHPq/eAe4C2SqkWWBf6it9irHN0BTBE\nKdUi6j+Pshrmrybqt2ef94lqzrZQSq3D6i3ZWUTy7dcWYZUpp2PVktRYWyci+2FdGKYlWC36fS3H\nuui3jHo/eUqpihqVSu+J6udTou8l5uceI4ZYVmElndHHXVnDNvHEKkMKscqQ95RSVYf1WMP2lSGx\nYg9ilU81ife5VH39eqAbsI9djh5ai31bO7Jq18YopXpgNeEZhvUbi6Wm72F7h75J1vlUhNW2NOb3\nJiKDsGplR2DVchVgPfaNea7XomyoqtpvLMpZWDdp0Q4DTrQrRNZgtVV/VEQerem9RKnxOqYTuyRQ\nVhuN/2A9iz9CRJx2t/V32dqWpTbeA44VkQPsnrRj2PZkLBerAWaZWGMvXbwtO7Fr2gYAH2E1tp1c\ny/0XYbVViY7HB2wSkZZYj6ESmUzUIwYR6Yh1d/qEUuqZ6BXtu8QPgDF2vD2p0j7G3r4lVvsmrfH4\nBThaRArEGt7mijps+yJwvogMsnuCdhKRXbFqqlxYF9yIiByDVeBWKMKqgY++E38GuLuijZCItBGR\n4+xl44DjRWR/u9f1XSS48InI/SLSS0Qcdvuwi7EaYG+OWu1V4EpgX6w2h/H2lW/H8SZW54Va9bJT\nSi0HvgEeFJE8+/PZWbYOtfIucJWIdLSToapDYPwCnCLWMB37UPmm9n/AESIywl7eSkQq2gRVLReq\negurrXArsYY8us3e37aoWobkY12kv1RK3Rpj/XeBUSLSwv6eL4teaF/0+2ElhvFiv0ZEutq/nbHA\nW1UfucVil2HFJP5swCpHvcBG+3sZXdO+o+I/VER2tx93lmA9mo0XWzK/h1iScj7Zj30/BP4j1jhv\nu1P5kXEu1k3/esCJdU3NjlpeBHQV2TLaQ01lQ1U/AK1FpNINp10JcTDVE7szgJ5Yv6N+wM9Y3+Ho\nWryXiiYEOVidQOLSiV2SKKXuZ+uYSSVYDTaXA4fZ7QRqs4/5WFW1b2PdtZRh9a6q1fZVXIfV+LQU\nq7FtbRuiV3hCREqxfviPAO8DR0QVUjXtfwzwql3VfrK9j0ysE2wGVm+tRF4DjrILU7AaJu+Ilbxt\nGSMoav3LsH7wa7AeOVQdCuU04NXafhdag/EKVmPhpVi/mbdru6FSajpW4+fHsHqmfQV0Vkptwmq8\nPR7rEc+JWG3zKrb7Dev3vsT+/bbB6gjwKfCFfV5Mx+pUgN0W5kqsxGAl1m8w+nFYVTlYN0qbsR5Z\ndqByZwmwLm6tgCl2G6eqPrF//8uweg0+gHWO1MUZWBe537EeKY1ja83b01hPGuZhXUTeq7LtLcBu\nWI+EbiMq+VRKLcZqF3gj1uf7E1ZjcbAajPcVkY0iUnWfYN0g/wr8hlUbMhOr9mRbfAj0ibronojV\nm/58qTzOWAd7+e1Y5e4SrOYeVS/KJwBT7faIsVSUg99h9SQuxfpd1NbtwJv2b254nHX+i9WxqBjr\nN/hJHfbfAesGuATr8f/nxL9pSOb3EEsyz6eLsWriirBu5qLL/slY73Mh1vdagvUdV3gHK5HbICI/\n1lQ2VGVfT16nes3nSOA7ZbU/j15/k7LaA6+xaydDwOaom7pE7wX7OC/HeAxeiShVL4OJa0lgV0Fv\nwnrcuTjd8aSaiNyNNb7XI9u5HzdWIXVInIukpmlNkIhcAuyolLpuO/cjWAnumbWtFdWaB7Hawn4N\n9KvPigO7kuMXrKFO1idcVyd2DYuIHIt1pyxYbW/2BfZU+ovSNE3TNK0G+lFsw3M8VsPVVUB34BSd\n1GmapmmaVhu6xk7TNE3TNK2J0DV2mqZpmqZpTUSdJu9t1aqV6tq1az2FomlaOs2ZM2e9Ump7BlNt\nknS5p2lNV1Ms9+qU2HXt2pXZs2fXVyyapqWRiNRldoFmQ5d7mtZ0NcVyTz+K1TRN0zRNayJ0Yqdp\nmqZpmtZE6MRO0zRN0zStidCJnaZpmqZpWhOhEztN0zRN07QmQid2mqZpmqZpTYRO7DRN0zRN05qI\nOo1jpzUPS39fzj+/LqVluxb0PqQHDocj3SFpmtYIhENhxj04gfGPTWbz+lJady7klJuGcfS/ByMi\n6Q5P05oFndhpW5RsKOX2E+5n4U+LMBwOQOHJcjNm/A303G+XdIenaVoKKKUo3+zF6c7Anemu03a3\nD7ufn7/4jVAgBEDRknU8efmLLJzzD1c/e1F9haxpWhT9KFbbYvRx9/HnjwsJeIP4Sn34Sv1sLNrM\nTYffycaiTekOT9O0evbd+zMYufNlnNT2PI7PP4sbh97Jir9W1Wrbed/9wS9fbk3qKoRDET554QuW\n/L68PkLWNK0Kndg1YCUbSpn6+jdMfv5zVv2zpl6PtXjeUv7+ZTHhYKTaskgowqRnp9br8TVNS6/P\n3/iW+856nDWL1xIORYiEI/z8xTwu2+9m1i5fX+P2X787naA/FHOZUvDq6HeSHbKmaTHoR7EN1LsP\nTuDV0W/jyHBgmgplmhxw/N7c+NrlZDiT/7X9/fMSxIid5wf9IeZ//2fMZUop5n33B5Oencqmok30\nGdCToy8cSkGb/KTHqGla/YhEIjxzzasEvMFKryul8JcHePve8Vzx5L8T7mPjmsS1+n/M+Gu749Q0\nrWa6xq4BmjFpDq+NeZegP4SvzE/AGyDoD/HDhNm8cPMb9XLM/NZ5GEbsxs1iCIUdWlZ7XSnF45e9\nyC1H383Xb0/j5y9/4617xnP2Lpez8KdF9RKnpmnJt2LBKgK+YMxlkVCE6R/NqnEfPfdP3A43w6k7\nYWlaKujErgF6/Y5xBLyBaq8HfEEmPTOVoD92Abw99hzcG0dG7ILX5XFxzEVDq70+Z+pcpr72Nf7y\nAEpZrwX9IbwlPsYMfwBV8aKmaQ2a4TBQpplweU2OumAwEu/mUISDhu+3zfFpmlZ7OrFrgBI1VhZD\nKF61MenHzHBmcPt71+HJduN0WY96RQR3lpthlx9Jj327V9vmoyc+wV9ePQEFKN1Qxh8zFyY9Tk3T\nkm/1orVx28dluDIYdMpBNe5j7dL15LfKjbksM9fDidccs10xappWO7qNXQOU3yoPb4kv5rJIKExe\nYU69HLfvwF68OP9hPnpyCn/M/ItWHQs57pLD2f3A3WKunyjBFEPYtHZzvcSpaVryLJ63lDtOeghl\nxqhhF8htmcNJ1x2bcB9//7KYS/rfGHMfHXZux3/G30CrjoXJClnTtAR0YtcADbvyKF68+Y1qDZkd\nTgf9D+9Hdn52vR27zQ6t+fd9Z9Rq3R7778KieUuJhKr3pA0Hw3TrvUOyw9M0Lcnevv+jakOUVDAM\ngwe/HEOL1ok7Q4099ZHYiSGwsWgzXXt13u44NU2rHf0otgE67uLD6TdodzzZWwcHzczx0LpTIVc/\n13AG+Rx+5VExe+g63U76Dtqd9t3aJty+ZEMpHz35KS/e8iZfvjWNYJyLi6Zp9ef36QswI7Hb12Xm\neCgpLq1xHysWxG8+4iv1sW5F8TbHp2la3egauwbIkeHgzgk38fMX85j6+jcEygPsf/zeDDhpf1we\nV7rD26Ljzu35z/jruetfD2OaJspUmBGTngfsyi1vXZVw22njZ3LvGY8BVqeQzBwPT131Mg99NYYu\nPfXdvaalSl5hDmsWr425LByKkFcYu91cXXhLYzctSQalImCuA8lCjLx6O46mNRY6saujoqXr+Pi5\nqSyZv5wuPTtxzIVDadulddKPIyLsObgPew7uk/R9J9NeQ/oyrugFfvp8HiXFpezSfyd22K1jwm3W\nLl/PvWc8Vml4BV+ZH1+ZnxuH3skbS5/W89NqWoocd8kRPHH5izE7QrXt0qrG8xkgpyCbso3lMZeJ\nCJ137bDdcVallEJ534Cyx0H5gAjKuQeSPxbJ6JL042laY6EfxdbBd+/P4LyeVzHuoYn8MGE27/13\nEuf1vIrv3p+RlnhCwRBfv/M9/73gGZ674TUWzV2aljgynBnsc+QeDD7jkFpdBCY/NxUzztAK3lIf\nP30+L9khapoWx+AzDqH3wT3wZHu2vOZ0O8nKy2TUm4lr3iscfs6guMtErF7yyabKn4eyB0BtBPxA\nCEKzUcUjUJHYNZD1SakQSvlTflxNq0rX2NVSSXEp9418vFItUzgYJgzcN/Jx+g7slZRHFrW1ftUG\nrjrwVkqKS/GV+TEcBhOenMJR/x7MxQ+fjUjs8aQagiXzVxAKhGMui4RNVv9TlOKINK35cmQ4uHPi\nTXw//kc+fu5zyjaV039oX4679AgK2xfUah+b1pbEXebKcjPz458YetbAJEWMlUCVP2nX1EUzQflR\n3teQ3OuSdryEsYSXokrGQvA7QKEc3ZC8GxH3wJQcX9Oq0oldLX351jTiDber7OUnXHZkyuK5+7RH\nWbeieEujZzNiEvAF+eTFL+g3aHcOOH7vpBxnxcLVTHpmCisWrKZbnx2S8ui5S89O/Dj5J0LB6smd\nI8NB+50Sd7rQNC25HA4Hh5y4P4ecuP82bR+vVy2AMhXhGOf6dgnNB+I11wiC/3NIQWKnIqtRxSNA\nlQH2U4jIP6iNV6DyH8DIPLzeY9C0qvSj2FoqXrmBYJwpd4K+IMUrN6QslrXL17Pgx4Uxe7L5ywOM\ne2hCUo4z5dWvuLDfdXz4xKfMnPzTlkfPP0ycvV37PfqCwXFHss/M8bDn4N7btf9QMETx6o31MkOH\npmnVHXjCPmTmeGIuU6bJnkOS3FZYnBD3Vrtief1TZc+A8rIlqdvCD6V3olT82Tw0rb7oxK6WuvXe\nIW7BlZnjSemYbetXbsDpjl9wrV22PgnHKOaxi58n6AtuGacuHAwT8AUZe+ojlJd4t3nfbXZozY2v\nXY4704Ur0+rlm5njIa9VLvdOuXWbO06EgiGeve41hheew8idL2NYy7N58Nwn67VHnqZpcPCIfSns\n2LLafLDuLBeHnLQ/7bq2Se4BM3oB8UYI8EDm8OQeL57A50Cc2kizFCLLUhOHpkXRiV0tHTRivy1J\nSDQRcGW6OGhE6uZBbL9j27jT/wDs0KPmDgw1mfr6N8Sb6lUEvh33w3bt/+AR+/HG0qf5971n8K8b\njueKp/7NW8ueodvu254g3/Wvh5n49BT85QGCviBBf4gv35rGtQNvj9tZQ9O07ed0OXn0+7s4eMR+\nON1O3JkusnIzGXH1MVz30iVJP56IA8m/C6h6s+0CR3sk8+SkH7PuFNBw2zprTZduY1dLLreTh74a\nw42H34W3xIcZjmBkGGTlZXHflFtxJahBS7aCNvnsfUQ/Zn36c7VOCO4sN6fcOGy7j7F2WXHcdjMB\nX5ANazZt9zHyW+VxwuXJaZe4eN5S5nz2a6XOLQChQJiVC1cz69Nf2PeoPZNyLE3TqstrmcuoN6/C\n7w1QtqmcFq3zYg5gniziGQItX0SVPgrheUAmZI1Asi9CjPqbnacS9xHge4uYtXZGATj07Dta6unE\nrg669OzMm0uf5ucv5rHqnyI67NSWPQ7rjWGkvuLzhlcu5eYjx7J43jJCwTAZTgfKVJw79lT6Duy1\n3fvftf9OfJ7jwV9Wvfu+J9vd4KYLm/3ZXCJxRs/3lfmZMXG2Tuw0LQU8WW48We6aV0wCce2NFP4v\nJceKefycC1H+SaA2U7mdnQfJG9OgRyfQmi6d2NWRYRjsNaQvew1JbxzZ+dk8+v1Y/pi5kN+++4PM\n3EwOGr4vBW0Sz+lYWwP+dQDP3fA6VdM6MYTs/KwGlyRlOB0YRuxCVETIcOmfuqZti7JN5Ux9/RsW\nzV1Khx3bMvTsQbUeBqWpE0cbaDUeVfoA+D8DwpCxO5J7PeKuv+Y5pumHskfBPwFUBNwDIe8mDKNF\nvR1TazxExWtIFUP//v3V7Nnb1yNSazwWzV3KzUeOxVfmQ0UUYggt2uRz75Rb6bBTu3SHV0nR0nWc\n0+NKQjHaHnqy3dw75TZ6HbBrtWW+cj8b12yioG0+mTmZqQi1wRKROUqp/umOo6FpzuXeb9//yaij\nxmJGFAFvAJfHiYhww6uXbfPQKE2VdS1ViNTvExzTLIF1A0BVnenDAa0mYWTsVK/Hb2qaYrmnqzG0\nuHbs04U3lz3NL1/+RtHS9XTs3o4+h/RskI8X2nZpzXEXD2XSs58T8G6dGsmd5WKvIX3puf8uldb3\nlft58oqX+OqtaTgyHITDEQ4ZsR9XPPVvsnKbd4KnaQBBf5Bbj7kHX6k/6jXrxun+s56g5wG70qpD\ny3SF1+BY5WIKysaNl8ZI6gAisOFsaPNd/cegNWg6sdMScjgc7DWkb7rDqJULHzyLnfp24827P2Dt\nsnW0aJPP8KuO5oTLj6yUjCqlGHXkWP6a/Y99obIuVt++9wPLF6zkiZn3NsjkVdNSafpHs2KOlQnW\nOfTZK19z2qgUDSuibRWaFX+ZWYRpbsAwdMLdnOnETmsyRIQhIwcwZOSAhOvNn76Av39eXG3ImFAg\nzPI/V/HzF/PYc3CSB1TVtEZm7bL1cYdVCvpDrPx7dYoj0iw1DN0UXgMundg1ZzqxqwclxaV8+94M\nSopL6b7Xjuw1pE9aes5qsf38xbxqw6JU8JX5mTXlF53Yac1ep1064Mp04iuNVFvmznSxYwPrGd98\nuIFA/MUZO9a4B2VuBP9kMDdBRk9wH4LItg0MrzU8OrFLsimvfsVjFz+PGELQH8KT5aZl+wIe/GqM\nbo/SQLg8LhwOg7BZ/YJlOAzcMQai1rTmZt+j98STZQ15VLWPnTgMhpw1MC1xNXtZI8H7fOxlzv0w\njDhTu4WXo8qfAd8nQBnW5T8CkmWNudfyTcTRsDrFadtGVyMl0aK5S3n8khcI+kMEvEGUqfCV+Vm9\nqIjbT7g/3eFptoOG74MRZ9oypyuDAScfkOKINK3hcWQ4eOCL0bRok09mrgdHhkFmjoesvExGvXEl\nU17+intHPsarY95h7bJ16Q632TDyrgfXwOoLHDtBwUsxt1Ghv1DFx4PvfaykDqxBlZXVESOyGrXx\ngnqKWEu1ZlNjV765HETIzsuqt2N88MgkQsHqI5CbEZOlvy9nyfzldO3Vud6Or9VOfqs8euzXnd+m\n/UkkvLU6Cv2kAAAgAElEQVTWzp3l5rDTD9quac00rSnp0rMzby57hpkf/8TyBato07mQFm3zGTPs\nAcyIScAXxOnK4N0HJnD1cxcy+PRD0h1ys2C0fA4zvBLKXwFCkHkShiv+wPSqZLTdkzbe8GYRCC9F\nhX5HnD3rIWItlZp8Yjf329954oqXWPb7ChBrCI/LHjuXnvtXH9Nsey3+bXncXmSODAcrF67WiV2a\nzZn6K2OGP4BpKiLhCCKCUop2XVtz5u0n19jxQtOamwxnBgeesA9gDYFycvt/44uakabiZvbhC56l\nz8E9aLND67TE2dwYGR0h/5Ya11NmCYTmET+ps4kDIktBJ3aNXpN+FDvvuz8YdeRYFs9dSiQcIRKK\nsHDOIm4YcgcLZv2d9ON17N4eiTP7gRkxadtFF3jpVLKhlDHDH8BfHiBod56oGKC7dGM5A07eXw9z\nomkJ/DBxTvwhUCImn7z0ZYoj0mqk/NTuUm+Co2N9R6OlQJNO7J659pWYvR8D3iDP35j8+QWHXXEU\nLo+z2utiCG12aM1O/bom/Zha7X3xxnfEm2nFjJh89/7MFEekaY3L+hXFMZubgFVzt/qfohRHpNXI\naA1GTVNNGmC0g4zeKQlJq19NNrELBkL8/fOSuMvnfvt73Iv8tuqxb3fO+s+/cHmcZLisxvmZOR5a\ntM7nzgk36tqgNFv+50oC3tjDnPjLA6xZvDbFEWla49J5t4443bFb8LgzXezYt2tqA9JqJCKQcz0Q\nu7cseMBoiRQ8q69RTUSTbWNX0+8z3oTx2+uka4/joOH78vnr37KxaBM999+VQ07cD5dHD6GRbjv0\n6IQ7yxUzufNku+mwU9s0RKVpjcdeQ/uQnZ+FvyxQ7cZYHAZHnDMoTZEl16Z1m/nwiU/57r0fEMNg\n8BmHcOxFQ8jOz053aNvEyDoekyCUPgAEQUVA3ODaB/EcAZ4jENHXqKZC6lJr1dgmw77usDH8+tX8\naq+LCPsesxd3fnRjGqLS0qV0YxmndbkYf1TD7wo5LbJ5Z9VzzToBb4qTYSdDYyv36tuKhau5/rAx\nlG/2EgqEcLqdGIbBnRNuovfBPdId3nZbu2wdl+x9E94SH6GANfOGy+OkoG0Lnpx1L/mt8tIc4bZT\nKgLhRSBOcHTRNXQ0zXKvyT6KBbjk4XPw5Hgq1d6JIWTmerjg/jPSF5iWFrkFOdz50Y1k5njwZLsB\n61F5dn4W93x6S7NO6jSttjp1b8//Fj/FLW9dzXl3n861z1/Mu6ufbxJJHcATV7xEaXHplqQOrCnU\nildt4JXRb6cxsu0n4kCc3ZGMrjqpa8KadI0dwJL5y3npljeZPeVXRKzR1M8dexqddumQ7tC0NPGW\n+vj6nekULVlLp106cPCJ++HJcqc7rLRrineuydAYyz1t2wQDIY7PO5NwqPqsNGDdCE4oeT3FUWn1\nqSmWe022jV2Frr06c8eH+pGrtlVWbiZHnX9YusPQNK2BCfljd66qEG+OaU1rSJp8YqdpmqY1D6Fg\niOkfzWbxvKUUti9g4CkHkluQU+vts/KyaNmugLXL18dcvmPfLskKVdPqjU7sNE3TtEZv2Z8ruW7Q\n7fi9AXylftxZLp697jVufuPKLTNn1EREOGfsKTxy0fMEvIFKy9xZLs4de1p9hK5pSdVkErtwKMyP\nk39mzZK1dNy5Hf0P74cjI/ZE75pWW+UlXt5/eBJTXv6KgC/IHofuzhm3nUiXnnpqOE1rKEzT5KbD\n72TT2s1UNBuvGNbontMf5aU/Hqn1VGeDzxhA+WYvL416CwRQYGQYXPb4eex9eL96egcNkwrMQJU9\nAqH5IB7IHIbkXIrUOOCxlk5NIrH7a84/3HzkWEKBEOFgmAxXBpnZHu6bOlrPzaptM2+pj8v2uYmi\nZesJ+a0ect+O+4EZk+Zw75Tb6HVA8ucb1jSt7n758jfKNpYTqy+gGTGZ9OzUOtW2HX/pkRx5/mAW\nzlmE4TDYZa8dm11Fgen7BDbfCNjDQ6kAeN9EBb6Ewg8Ro/aPuLXUavTDnfjKfNw45E5K1pfiK/UT\nCoTxlfrZsGYT1x86hlAwVPNONC2GDx6ZxNqopA7ANBX+8gAPnPNk0mcu0TRt26z6pyjuHLahYJgl\nvy2v8z5dbie9DtiVHvt2b3ZJnVJhKLmdLUndFkGIFKG876QjLK2WGn1i99Xb0wmHYs9dGPAHmf6R\nHqZA2zZTXvmaoD/2jcH6lcWsXqTnxdQav0gkwurFRWxcuzndoWyzdt3aYMRJvjJcDjrvpie3r5PQ\nfCBepUgA/ONTGY1WR43+UeyiuUvxlwdiLvOV+Vn2x4oUR6Q1FfGSOgDDYVRrXK1pjc2Ep6fw6uh3\nCPiCmJEIO/btyrXPX0S33o2r9+ceh+1OVq4HX6mv2jKHw8ExFw1JQ1SNWRirgWEcSj8Ja8gafY1d\n266tcWXGnjHAk+WmVceWKY5Iayr2OKw3hiP2KWIYBp121YNca43XB499zHPXv05JcSkBb4BQIMyC\nH//mqoNvY82StekOr04cDgf3fHILeYW5ZOZYk927Ml24Ml1c+9IltO+m54GuE+fuQOxH2+ACz+E1\n7sIM/oO59gjMNbta/609HDP4V1LD1GJrdDV2a5ev5/2HJ/HjJz+Tme1mwMkHJGzrNODkA1IYndaU\nnHHrCL4f/yP+8srtTNxZbkaOOQmny5mmyDRt+4SCIaumLkatc8Ab5J37P+TKpy5IQ2TbrlvvLry5\n7Gm+HTeDf+YuoXWnQg497WAK2ugenHUl4kblXAGljwLRtaAGSBaSNTLh9mZwDmw4tcqLi2HDMZgF\nb2C49056zNpWjSqxWzR3KVcffBvBQJBw0JryZfmfqyhsX0Dx6o2gFKFAGJfHiRgGo9+7jqzczDRH\nrTVWnXbpwANfjObB855m9aIiHA4DR4aDM8ecxLDLj0p3eJq2zRJ1JoiEI8z8+KcURpM87kw3Q0YO\nYAgD0h1Ko2dkn4sp2VD2KJglgAmufZG8/yCOVok33nB+/GUbz4d2vyY1Vq2yRpXY3XfW43irtKHw\newNsXLOJk649DkeGwYoFq+i6e2eOOPdQWrYrSFOkWlOx2z7deWHefylaug6/N0DHnduR4WxUp42m\nVeN0O+P2IgVweXRttAZG1r9QmSeBuRHEgxjZNW5jRiJAeYI1fJiRIIYjdhMqbfs1mivU2mXrWLFg\nVcxlAV+Qb979nlcWPJ7iqLTmom2X2g1uqmmNQZeenchtmROz45nL42To2YPSEJXWEIkY4CiswxbV\nO7BUVwbo9u/1pdF0nijf7MWRoKbEW1KbH1PTp0K/o8qeR5W/jArXfewmTdOaPhHhuhcvwZ3lQqI6\nPzrdGbTqVMjxlx6RvuC0Rs1w1DxwseHQSV19ajQ1dh27t4/bSUIModeBu6U4ooZFqSBq46UQnInV\nVd2A0v+isk5Fcm9GJEHXdS1pyku8/DBhNmWbyumx3y7s2n+ndIekaTHtObgP//3mDl4b8y7zvvsT\nd5aLw88eyCk3nkB2Xla6w9MaM9ehEPwy9jLnwamNpRlqNImdy+PipGuP5d0HJlTryeXyODn91hFp\niix1KhLbWEmaKn3ATuqqjBTufQecPSBzWAoibN4+f+NbHrngWQyHQSQcQQyDHft24Z7Jo8jOr7lt\niqal2i577cRdE29OdxhaE2O0fAZz3TCIzK+8wNEDo/DF9ATVjDSaR7EAZ9x2IsOvPAp3povs/Cwy\nczy0bN+CMR/cwM79uqU7vHqjQgsxN5yPKuqJKuqBuWEkKrT1hFEqCN53qT79C4APVfZMymJtrhb+\ntIhHLniWgC+Ir8xP0B8i4A3w95xF3H36o+kOT9M0LaWM1uOh9ffgOR08p0Lr7zFaf5TusJqFRlNj\nB9agsOeOPY1TbhrG3z8vxp3lpvue3TCMRpWf1okKLURtOAmUD7AfRQdnoIpPg8LXEWcfMDdsXRZL\nJHanEy153n3gI4KB6qOxh4Jhfv7yN9YuX0+bzjUMEaBpmtaEGI7W0OL2dIfR7DTKjCgrN5M+h/Rk\n1/47NemkDkCV3l85qdvChyq52/qn0SLG8igO3aOzvv3z61KUGfs7cLmdrPhrdYoj0jRN05qjpp0V\nNXJKKQh+T9ykLfQLSgUQ8UDm0UCscYEyIeuceoxSA2jTOf5wAOFQmMIOekxFTdOaLjMSwSx5AHP9\ncMwNF2NGNqQ7pGarUT2KbZ4S1MRFTdIsubeiQgsgshiUF2vqFze4BiBZp2370ZWCyFIgDI5uiDi2\neV9N2Yirj2H+9AXVxgUTQ+jYvT1denSqto2vzMc342aw8u/VdNixLQNOPkDPlKJpWqNjBmbCxpFs\nvV79Buu+wPSchNFibDpDa5Z0YtcAKdOL8o0D3/uAE4jEXtHZCxE3AGLkQOH7EJyG8n8F4kI8R4Gz\nzzYPdaIC36M232q14RMB3KjcGzCymn4P5Lrqf3g/jr5gMJOenUrQH0KZCk+2G0+Wm9vfu67a+vOn\nL2DUUWMxIyb+8gCebDdPX/MKd028mT6H9EzDO9A0rTZM02TaBzP56IlP2Vi0iR777cLJ1x9Hl56d\n0x1a+lRK6qL4x2H6DsXIPCzlITVnEm9suFj69++vZs+eXY/haMosQxWfDJEVxO7lWsGDtHwFce1Z\nP3EEf0VtODNGDB7Iuwsj67h6OW5jt/CnRUx55Ss2ryuh36DdOfT0g8nM9lRax1fu55SOF8QcVDsz\n18PbK55LS82diMxRSvVP+YEbOF3uaRWUUtx9+qPMmDh7S+284TBwup2MHnct+xy5R5ojTD2z7Cko\neyT+CtIao+33qQuojppiuadr7BoYVf4CRJYBwThrGODsbQ06nMSkTillddIQJyJOVNnDxE4s/VD2\nICrz2FrVBCqlmtXgyN333JHue+6YcJ1v3v0BM05HCxVRfPXWNI6+YEh9hKdp2naYPeWXSkkdgBkx\nCXgD3HP6o4wreqH5zSUdmJ54udJt7VJNd55oaHzjiJ/UOaH1NxiF45Ka1Jm+yaj1h6HW7oUq6ou5\n8TIIzkmwwQYwixPuUwW+xVx/PKpoN8w1fTA3j0JFEm/TXKz+Zw3+sti1sX5vgBULdQ9aTWuIPn7+\n85jz64L1iPbnL39LcUQNgKNLDSvodsOp1mBvLcKhMAt/WgxA9z27NZ+7IJVgzlvJQFT1sdK2h+l9\nB0rGUql2LvA5iTttKKtjRtVXI+sgstQaPLn0oah9+sH3ISowDVpNRIz85L2BRqhNl1YYGQZm2Ky2\nzJ3lpuPO7dMQlaZpNSlZXxp/oYLyTeWpC6ahyL0V/OPiL885P3WxaEADTeymvPoVz1z9KqZpXfhE\nhAsfGsmR5zaDBpjOPhCMV7XtAke7pB1KqSCU3k/1R64mWytzqyZ4As6+iJG7dT9mGWrzTRD4BsQF\nKlbhFwZzI8r7BpJzSdLeQ2O0eN6ymEkdgDJNBp16YIoj0jStNvY4rDcLZv1N0F/9BjscCrPbvt3T\nEFV6GY5MzOxLofzJGAu7YDTz8j4dGtyj2OkTZvH4pS9Qtqkcb4kPb4mP8s1enrziJaaNn5nu8Oqd\n5FwJeGIsyYScSxFJYi4e+gMriYulIrmLHhvPAZKF5I2ptKbaeCEEvgYCcZK6CgHwTdj2eJuAgC/A\npy99FXd5225t9ATsmtZAHXPRUJxuZ7XXXR4nex+xB+26tklDVOln5F4JLd8DoxvWNaMF5IzCaDM1\n3aE1Sw0usXtx1JsEvNXbmAW8QV68+Y00RJRa4toDafEIGC1BskFyQDIh5yIk66wkH6yGTg2OTpB9\nDhjtwGgNmSOQwgmIcxcAVHgZZvmbEPqV+O0Cq4qXSDYP61duwDDif+7FK3VDY01rqAra5PPfb+6g\nY/f2eLLdZOdn4XQ7OXDYvtz8xhXpDi+tDFcfjDZTMNr9htHuR4ycs9MdUrPVYB7FhkNhSjaUsfyP\nFXHXWfn3GkLBEE5X9TumpkQ8h4L7ewjPBxUA5+6I1EMD1IyexP8JuCHzBIycSyH32kpLVGQ9atPl\nEKpoKFzbpM4FnqMq78sss96j0bJZ9J7Nb5VHOBSOu7xF6+bd/lDbduFQmC/e+I6Jz3xG2cZy+gzo\nycnXH0+n7rrNZjLt2KcLL//5KEt+W8amdSV07dWZgrYt0h2Wpm2R9sTOV+7n2WtfZerr32JGTBIN\nq+dwOHBkNP6ZD5RZBpGVYBQijtgTw4s4rPZ29UgkA5V7G5TcSuV2dhlgtECyzqi2jVIR1IbT7HH2\n4ico1RkgOUj2SGs/4b9Rm2+H0M/WMqMQlXsdRuax2/GOGr6cFtnsNaQvs6b8QiRUeeBpd5abYVce\nFWdLTYsvEo5w8xFj+fPHhVt6ba5ZXMRXb03j7sm30PvgHmmOsGkREbr1rqk3qKalR1ofxZqmyfWH\njuGzV78h6AsSDsZPFBwZBgeN2BfDaHBPj2tNqSDm5tGotfujNpyKWjcQs/hMVGRN2mIyso5DCh6D\njN2wpijzQObxSOH42L1XA9+CuY7aJXWC9RPLAPdhSKvxiNESFV5hDcIcmm3vJwjmath8C6b3/eS9\nuQbqmucvolXHlnhytral9GR76DuwF8deNDSNkWmN1ZdvTquU1AFEwtasJnef/gh1GYhe07TGLa01\ndnM++5Vlf6wkFEg8hIfL7SSnIJsLHxyZosjqh9p0rdVzlID1+BEgNBtVfCK0+gwx0tNoXtwDEffA\nWg0mrEK/gKpFl37JBM/xSO6N1vRmsvXxuSp/xh7WperFxg+l96MyT2jSc9IWtG3BS78/wtfvTGf6\nhFl4stwMGTmAPQ7r3ahvXLT0mfTsZ3HHVyvf5GXhT4vYZa+dUhyVpmnpkNbEbvqE2fjiDNRqGEJe\nqzxyW+Yw8F8HcMJlR5JXmBtz3cZAhZdu7TlaSQTMtah1QyDvOvCckLa2ZrU5rhh5KFzEblcnIHmQ\n0R3JPgfcg2PvM/AVcee/JQCRxZCxcx0ib3xcHhdDzxrI0LMGpjsUrQko2+SNu8xwGJRvjr9c07Sm\nJa2JnSPDQERiPibIcGVw2qjhDLuiibQ5Cs4GccQf91etQ20eA8G5SP7tqYysbjxHQ2m8eQFdSKuP\nEEeHGnaS4GenVMzlyiyxHgMTAte+tTiGpjUf/Qb1YuXfq6u12wQIBULs1K9r6oPSNC0t0vrcZ+DJ\nB+DOcsVdvt+xe6Uwmu2nlGl1Cggvqp6sihurzVkiPvC9hwovrq8Qt5s42kHu1VjTxFS8H7H+zrmy\ndglX5rFAnJ7NRstqU9SY5S+j1h6IKrkNVfIf1LrDMTffjFLxav00rXk58dpjY46v5s5yMfTsQeS1\nbLxPOzRNq5u0Jna9DtyNvoN2r5bcubPcHHX+YbTv1jZNkdWd6fsYte5AVPGJqPXDUOsGoPxRA9G6\nB4CqTYeDCPg/q7c4k8HIPhdp+SK4DwPHjuAehLR8AaOWU8dI9rlgFFC9Zs6D5N9Z6fGt8n9p1xAG\nrLZ9ymv92/cxquyxZL0lTWvU2ndry32f3Ua7rq23jK/m8jg5/JxBXPbYuekOT9O0FJK69Jbq37+/\nmj17dlIDiIQjvP/wJN5/5GM2ryuhzQ6FnHLTcI4879BGM66Z8n+B2nQ11afm8iAFzyPufQEwy9+G\n0nuABPPBIpB9KUZu0x7sUkWKrcTMPwlUEFx7IDnXIK5+ldYz1w+zxvOLRbKRNjMRiV/rq9WeiMxR\nSvVPdxwNTX2Ue/VFKcXiecso21TOjn26kNMiO90hNVvFqzey5LdlLPjxb3789BdM02TwGYdwzIVD\ndCepBqQplntpT+yaAnPdUIgsib3Q2RejcOsEySowA1V6D4T/iL2+ZCEFLyKuxvUYur6Ya/oSNxGW\nTKTVJ7q9XZI0xQIuGXS5p9VFwBfggXOeZNr4H4mEI9XaVecUZPP6P0+Q0yInPQFqlTTFck/fNmwn\nZZZDZHn8FULzKv0p7v2Qwg/B2ZfK87ACuCGjBzj3THqcjYVSYVR4CSpSZL1gJBjRXUWsXriapmkN\nxD2nP8YPE2ZbHVli1JuUbSzn+sF3pD4wrdlIS6/Y8s3lzJk6l3AoQr9BvWjZriAdYSSH1DS9WfWP\nWESg4BVU6R3gmwxigDIh81gk99ZG8wg6mZRSKO+bUPYIEAIVQWV0A/dg8I2j+mPuDHAPQAx916tp\nWsOwZslaZn36M0F/4rFZ//5pMUF/EJdHNyPRki/lid17D0/k5VvexuF0AIpwMMJR/z6MSx45p1G2\nOxBxoVwHQvA7Yo9lEsFcfwzS4gkko+vW7YxsJP8+VO7tYBaDo1X9zAfbSCjv61D6EJUeu4b/hMgy\ncO4Gob8AeywuybLmls3Td72apjUcC2b9Q4Yro8bEDqw2eI2pg6DWeKQ0k/r+wx955bZ3CPqD+Ep9\n+Er9hAIhPn3pK96578NUhpJUkncbSC6x8+QIhP9GFZ9iPbatuq2RhWR0bt5JnQpB2aPEbEunAuDo\nhrR4GNxHWD1wc0cjrSYjjsKUx6ppmhZPbkHtO6sUdmxZj5FozVlKE7vXxrxLwFt92puAN8C7D0yw\nGpo2QpLRBWk10Rr+IyYTlB/lm5jSuBqN8D+AGWdhBILfI55BGAWPYRQ8i2QNR8QTZ31N07T06Duw\nFw5HzdMh7rr3TrhcNTXj0bRtk9LEbvmClXGXBf1BNq8vSWE0ySWO9ohrb8AdZw0vhGakMqTGQ9xW\nG8O4y6u3Q1FKoQIzMUsfxCx9DBX6qx4D1DRNq5kjw8Got67Ck+VG4lxd81vlcf8XDXh2Ia3RS2kb\nu+wW2Wwq2hxzmVKKrLysVIaTfEY+EO9uzQDRjw5jcnQFR6s4vYvdkDkcFV6G8v4PwgvA6Ajh36z2\nd8oLOFDlL6Ayj0HyxjbLzieapjUM/Yf25ak59zHuoYn88cNf1hTahkFuQTaHn3MoQ0cOSHeIWhOX\n0sTu2IuG8s79HxL0VW5YmuF0sN+x/fFkxavtavhUZDUKE4g3u4QLyRqRypAaDRGB/HtRG84DAmzt\nhOICR1swdkCtPwbrs431+Uas/3wfo5z9kKyTUxS5pmladZ137cg1z12U7jC0Ziqlj2JPufEEdurb\nDU/O1vZRnmw3hR1bcsWTtZuOqqFRKoi56WrUuqFQcieV50+t+GcmZJ2GOHumI8RGQVx7I4XvWsOb\nSD4YbSH7PCh4HUpuxRrupKYp2XxQ/kIKotU0TdO0himlNXYuj4uHv72D6R/N4rPXviYUCDPgpP0Z\ndOpBjba2TpWMAf8XWHOZRncMETA6QUY3JPtsxH1QmiJsPMS5G1LwZKXXlPd9VF3uPyoGNtY0TdO0\nZijl49g5MhwcPGI/Dh6xX6oPnXTK3AS+iViPD2Nw9cZo8UhKY2pyzGLifr6xONrXWyip5PcGmDFx\nNpvXl7LzHt3ouf8uuu2gpmmaVqO0zDzRZIQXWT02VazEw4Tg3JSH1OQ4dwPxgKo+BmB1GUj2BfUe\nUn2bMWkOY099GDGESCiC4TDosHM77p1yGwVt8tMdnqZpmtaANb6pHhoIpRQYBaASjDDuaMRTpTUU\nroOsz7k2P1VHJ8gcVu8h1adV/6zhrlP+i788gK/UT9Afwl8eYOnvKxh9/H3pDk/TNE1r4HRiV0em\ndwLmusGoot1QxSOAOAPlSiaSNTKlsTVFIgbS8n/g6AYkmlfRCZ5jG/3jyvGPTbYmD68iEoqweN5S\nFv+2LA1RaZqmaY2FTuzqwCx7Bkpus8ZPQ4EqA8qxPsaopEOywHUgeI5JT6BNjDg6IK0mQ8ErQLyp\n1wwk68QURlU/Fv60iHCMxA7A4XCw7PcVKY5I0zRNa0x0YldLyiyBsiepPp9pCHCD+1Bw7gWuQUj+\nf5EWTyBS89QyWu2ICIa7P9LyFZBstiZ4LsAN+fciTaDjRLuubRAjdq2jUkrPL6lpmqYlpDtP1Fbw\nB5CMOB0lfKDKMQrfSnlYzY249oDW36B8H0H4T3B0QjKHg9EGFfwFgj9anS08Qxplonf8ZUcybfxM\nAt5gtWW5LXPodcCuaYhK0zRNayx0YldrCeYyrdVyLVnEyEOyz9zytzK9qA2nQ2g+EAQyoPR+VPYF\nGLlXpC3ObdFj3+6cetMw3rpnPKFgGDNi4s5y43RlcMdHNzb6NoSapmla/dKJXW259o3fA1YyEd2e\nLm1Uye0QmsfW8e7sNmreF1HOXojnsHSFtk1Ov/VEDhy2L5Nf+JwNqzbS68BdGXrWQLLzs9MdmqZp\nmtbA6cSulsRoico6G7yvU7mdndOa/irz6DRF1rwpsxT8nxJzEGPlQ5U/0+gSO4CuvTpzycPnpDsM\nTdOaKRX6HeX7EFQp4joAPIcjkmhkAq2h0IldHUjutShHOyh/CsyNQAZkHo3k3oxInGFPtPoVWZOg\n7SMQXpraeDRN0xoxpRSqdCx438Vq2mKi/J9A6YNQ+C7iaJv0Y5rBebDx36A2AW7IvQ0ju/GPcpAu\nOrGrAxFBss9AZZ0Oygvi0T1f083RuoZBotulLpY0U0oxf/oCZkyag2EIB56wD7vuvXO6w9I0rTEJ\nfA6+cYB/62vKCyqA2nQ1UvhmUg9nbrwaAh9HveKD0lGYpY9jtPsmqcdqLtKS2JUUl/LDxNkE/SH6\nDOhJlx6d0hHGNhMRe8gNLd3EaIFyHwKBb7CGnomWiWSfn46wUi4YCHHrsffwxw9/4fcGEBE+eHQy\n/Yf25bZ3r8GRoW9ANE2rmSp/CVTVYb0AIhCah4qsQhwdknIsM7S+SlIXbTVm6VMYuZck5VjNScoT\nu/censjLt7yFkeFARUwUsNeQPtz6zjW43M5Uh6M1AZI/FlV8GpirrTvLigGjM49FZfSA4CzI2Akx\nmu4YcK+Ofpv53y8g6LOGSVFKEfAGmP3Zr4x7cAKn3NS4p1rTtPoSiURQpiLDqR9gARBZFX+ZuCCy\nFpKU2LHxrMTLy58AndjVWUoHKJ716c+8cts71vyXZX4CviBBX5A5U+fy9FUvpzKU7abCKzA3XY9Z\ntCBD7bMAACAASURBVAdmUT/MjVegwovSHVazJEYB0moSkv8QZJ4G2edB/v0QnAPFI1AbL0StPQRz\n0w2oeG3xGqBN6zYz99vfWb5gZcL1TNNk4jOfbUnqogW8Ad5/NN4dsaY1X6v+WcPo4+/j6MzTOMpz\nGhfucR0/fT433WGlX0aC5hsqYM3JnSxqTQ0rhJN3rGYkpbcob4x9n4C3+oU16Asy9bVvuOCBM8nM\niTdlVMOhwstQxcNAlbNl/LrAZ6jgd9DyHcS5S1rja45EHOA5DPEchjI3oNYNAVVqLVT2Sv5PUMqH\nFDyetjhrI+AL8ND5zzBt/ExcbifhUJj2O7bl1neuidlswV8eIOiP385w89oSlFJ6DLwGZv2qDXz4\n2GRmfvwT7iw3R553KEPOGqifXKTA2mXruHSfmyjf7EWZVgGx6NeljD7+Pm595xr2O2avNEeYPpJz\nIWrDbKrPsuQC9wDE0Sp5BzM6Q+T3BCvoc2FbpLTGbtkf8WsejAwHa5cXpzCabadKH6ic1IH1b1WO\nKr0rXWFpNuV9C1T12isIQOBrVCRxDVi6jT3lEb4fP5OQP0T5Zi8Bb5Cl85dz9cG3UbKhtNr6nmw3\n7sz4wxAUtGuhk7oGZunvyzm/19V88OjHLJm/nAWz/uaZa1/lqoNuJeBrPLXKjdUbYz/AV+rbktRV\nCPiCPH75Cyil4mzZ9IlrH8i9DnBjTd3otP7v7IXk31enfSkVQMVsr2dr8WriHeTcVKfjaZaUJnYF\nbfPjLgsHwxS0ib+8oVBKQeBL4s40EZyFiplUaCkT+J6Y49qBNTRKqOE+bln592rmTP21Wg2cUhD0\nB/n0pS+rbWMYBidcdmTM5M6d5eaka4+tt3i1bXP/WU/gLfESCmx91OQvD7D09xV89MSnaYysefh+\n/I9EwrHL8M1rS1izZG2KI2pYjOwzkTbfIHk3I7nXIC1fQ1q+jRg5tdpehf7ALD4dVdQPVbQn5vrj\nUIEZ1Y/jzAfPiNg7ceyCkXNm7GVaQvWS2M399nduOuIu/tXxAi7a83qmvvYNpmky4qpj8GS7q63v\nyHDQb9Du5BXm1kc49aCm5/6RlEShxWHUcIMgtSuc0uHPmX9jxOnBGvAGmfNZ7KT0zNtPYo/BvXFn\nuTAcBg6HgSvTxYEn7M3wq/Tg2Q3J+pXFLP5tObEqhYK+IJOenZr6oJqbZlKBrZSJCnyLKnscVf4q\nKlL7hFWMlkjWKUj2eYirb61r/VX4b9SGUyE0C+taGIHwn6iNF6AC0yuta5a/Df5JVG4V5oD8pzBa\nT6p1rFplSW9jN/mFz3nqqpe3TGK+YfVGHrv0eWZO/omb/nc5sz/7lVmf/kzAG0ApyMzxkNcql+tf\nbiQ9X8ILsUqFOFX1Gd0RafjtBJsyyTwZFZwep8u+YU0P10Bl5WViJChAcwtjJ6UZzgzu/OgmFv60\niB8n/4wYwv7H9ef/7N13dBTl18Dx7zPbU0nognREUYpIUVFERLChYkMFu9j4KfqKvaMoYO8NGypg\nBWygKCggFkAFFAURRJAeICTZvvO8f2xAQnaXlC3Jcj/neDxkdmZuIJm9+5R7Wx7SLFGhiioqKXRj\ntVsI+CKviywpdCc5on3PUQN7MP3VmYSC5T+E12mQS6MWDVIQVXzp0JZwD21zU+nSIQcUPYzOvhkj\n88LE3bfo8SjPXi96xwOo+p+FX+dfBEUPUn52xQDP2+Dqm7AY011cE7uSHW6eHf5aud153hIfP3y6\nkN/mLuOud/+PJXN+Z8b4r/EU++hxcheOOfsI7M7a0apE77iPqNOwGKhsWROQco5jwN4L/LN3e8CE\nS6Co3IdrdFucw47viI7yocGZ6eCky2K3R2vbpRVtu7RKRGgiThq3bhT1c6FS0P7IdskNaB90/h1n\n8M1783AXujF3W2fncNm59tnL02JNqt5+LYTW8N8MU2kCVfQI2tYBZT+0etfXZmmhflfZQv2+OUT9\nAQ+tRpvbUEYeumQckZfMBMC/IK718vY1cZ2K/eHTn7BYI1/S5/bx+euzUErRsVd7bhx3DXdOuoHj\nLzim9iR12gOBn2O8wopyHJm0eERkShmoOk+ickaBrQtYmoHzZFTd91DOPqkOLya7085Nrw4LT6ka\n/725ODMdHHFqNw49rkMKoxPxYHfYOHvEqTgyyi9LsbvsDLlLWiklWoP96/Hc/DEceVp3LFYLSina\ndGnJ/R/fSo+TuqQ6vGrTwX8g8CuRlw35SpOqKl5bBzGLnkZv6obe1B296TDMHQ+idWmniphJsWZX\n2hFcTvRPOHYI/l3lGPd1cR2x85b4MEOR/6G0hpLttXeKQWs/2oyxuweAEGbBeeFfKOUC10BU1jCU\nkZOUGMV/lDLAdQrKdUqqQ6m0o888nMatG/LO2KksX/AXeQ3rMPDaE+l19hFpMZIgYPCdZ+Lz+Jny\n1GdY7Va0qbE5bdz06jW069o61eHtExq3asg9H4zANE20qdOrO0toTTg5ili3U0M1aq7q7SNKNxCW\nJnI6CO6J6MASyJ8Ajj7gnUbEmS1rW9TONdCWphBaFeUmgfgVQd4HxTWx63D0gWgz8jSlM8tJ95Oq\nN/SbCtq/AL3jIQj+RnhtnZ3omydMCCwsPdEH7rfRvq+g7pQK7yYSAqBN55bcMeH6VIchEsQwDC5/\naDDn3TaQ5Qv+wuGy0657GyyWNEouagnDMJJcHyIJLE1i9NBWYG1epcvqwJ/g+4ryU6g+CP4O/u9R\nWTegfbMjlARzonLu+S+KzEvR/kj18oxwAmhtUaUYRZx/nPdv14QufTti36PsgmExyMh20Wfw0fG8\nXcJp/4/orZdCcAnhH9AQ5X8Iy5yxx5/9ENqEdr+TsBiFELVXZk4Gh/bpQPsj2klSJ+JGWVuA9QAg\n0s+UE5V5WdUu7J9N1DXm2o32zkBZm6HqfgiO4wnXwLOA/XBU3bdR9v+muZWjJ2ReTLheXukYk8oE\nowEq75mqxSeABOyKvevd/+OpYeOYOXEuNruVgC9Iu+5tuO3Na3FlOuN9u4TSO0aya7i5HCfhRE6D\nUR/Mgiiv9YJnMmRV8RdJCCGEqCSV90y47Ii5vXT0zAYYkHUNyt6tqlcleq0YBaWbKJS1+a4OP7G6\n3hjZN6BdA9CeqWBuQ9l7gLN/jd7gVhvEPbGzO+2MeOUarnr0Ijas2kSdBjnUa1I33rdJOG0W7mUd\ngkbVnwEYEFqL3nZ51HWgEL3dkxBCCBFvytII6n0Bvplo/09g5KKcA1DW/at+UUcfKHo8yg1dKGf/\n8l8uTeq0WYL2TAHfl+HXus4Ax7EoaxtU9o1Vj0mUk7BesVl1MmlzaMtEXb4GUOFfHEAbuUQvgWIH\nZ7+kRSWEEEIAKGULj4BFSLiqdD1rC7TrVPB8QtllSU6wdQVb5B67OrQBXXAWmEW7ztO+eWDvDHkv\nh+MUcZNuS0bjRhm5YG0T5agBjt7/vVY5IOs6wn319nidcqEyElcMUuydDvyJuXUo5ob24f+2XoYO\nLEt1WEIIUeuonPsh+0YwGhKefs2DrCtRec9HnXLVhbeBuYWyyaAb/D+h3W8nI+x9iiR2Maicuwmv\npSvzVVAZqOwbynzVyLwUcm4Hox7hxaC20gWjH6As9ZMUsdiTDixHbz27dNFvMPyffw566yB0YGmq\nwxNCiFpFKQMj80KMBnNQDf/AaPgDRtawqKNu2twG/vlEntXygvvNhMa7L5LELgZl74rKH186vKwI\n7+7pHS50ay0/zWxkDELVn4uqPxPV4AeM/NdRVmnplEq66KFwdfQ9F0Bqd/iYEEKIKqlQXU1zO6gY\nq77M7fELSAAJXGOXLpS9M6ruxHD7FNRef5CVMkBG6GoErU3wfxf9Bf75aB1ExXroCCGEqDrLfsTY\nWQhWaaEXbzJiV0FKGVL1Py3FeOAIIYSoFqUc4BpC+WVNAE5U1v+SHVLak8ROpC2ljKi7tACwdaqV\nu7FKdrhZ9M1vLF/4F2aUTi9CCFFTqOwbwHUy4c5NmaCyABfk3CH91RNA5qDiQOtguACkykIpqR5f\nU+hQQXiYP/AT4a4hu3Oism9LRVhVZpomr9z2NlOeno7NYcUMmWTkuLj5jWvpclyHVIcnhBARKWVF\n5T6Ezhoe3kihHGDviTIyUx1aWkpaYvf9Jwt5e9QHrF2+jvxGdTjzhgGccOmx4T59tZQ23eiih8Hz\nIRAE5URnXIjKGibrtlJM+2ajt18LemcrOEV42tUA2yGo7DtQ9s6pDbKSXr9rElOf/Ry/14/f6wfA\nU+zl7tPG8PT3D9LyENmoU5P5fQHmTfmRlYtXU3e/fI49ryc5+dmpDkuIpFGWRuAakOow0p7SuuJr\njLp27aoXLFhQ6ZtMHD2Ztx/4AJ/7v8bBzkwHhw/oyu1vD6+Va9e0NtFbB0Hgd8C/2xEnOHpj5D2V\nqtD2edrcgd50NJH7+jpRDeaijJxkh1UtXrePsxpcVuZ3aCfDUPQ6+wjumHhDhDMrTim1UGvdtVoX\nSUNVfe7t7p8//uXG3vfgc/vwFHtxlPbTvuXN6zj6jB7xCFMIUQXp+NxL+HDZto3beWvke+XekLwl\nPr7/eAFLv1ue6BASwz8Pgn9SNqkD8ILva3RwRSqiEgDeT6O3MySE9kxOZjRxseaPf7FYI/+6mqZm\nyZw/khyR2JPWml9m/cqDg5/kthMf4L1HP6JoWzGmaXJrv/sp3FyIpzjcT9rn8ePz+BlzwVNsXL05\nxZGLaMzgZkzfj5hSkkPUIgmfL/zuowWoKNOtPrefr96ezcFH1r7tzto7o7Q+WiQh8M2O0blCJJIO\nrQMdabQOIABFj6AdPVG16N8nMzeDUGDPdYK7Hc/Zs+uJSCatNWMvfoa5H/6AtyT8IXbJnN+Z+NBk\nrhh7AcWFJUSaHAmFTD59aQaXjjo/yRGLWMzgOth6Ppjr/vuapS3UnYhRy0b7xb4n4SN2Po8fMxR5\n557WGk9J+aml2sFC9GGh0mLGIiWUtTWojBiv8KG3Xk5lliGk2n6tG9GoZYOIxxwZdk656vgkRyR2\nN+eD78skdRD+4Fq8rYTX756EGYz8DAz6g/z965pkhSkqwDT9sKV/maQOgNCfsFl+z0TNl/DErvOx\nB2MYkRMgV5aTHicemugQEkK5TgQVbZREgePYpMYjduM8AdhLGRNdCIHqrZtKtlvfvA5Xtgur7b+B\ndkeGg5YdmnPylf1SGJmY/PRnZZK6nbTW7CgoivoZ0Gq30qx9kwRHJyql5DkgyoCD3obp+SSp4QhR\nWQlP7Fp2aE6nYw/BXrpYeCerzUJeozr0HNg90SEkhq0r2LpRvuiiC1xnSiuxBNHaj9axR3mVcqLy\nXyf2SgMNobXxDC3h2hzaknFLHmXA1f1o2m4/DjisFVc/dhGPfn0fdkftq8eXTrauj74Gy+6048hw\nRDxmsRicIkl5ymhzG9r3Ldq/qLS7EOD9IvZJ3o8SH5gQ1ZCUmhz3vH8jz93wGl+On41hNQj6Q3Tt\n34kbx12NzV4735CUUpD3HLrkFXCPB3MrGI0g80pUxrmpDi/t6MBi9I5REFgU/rOtQ8ySJcp2MNp1\nHngmUL6GXSlLi8QEm0ANmtXnmicuSXUYYg8HHNaKDSs3Yprlp/f93gCjp97CyLMexe/14yn27vqg\ne9Pr/6NRi8hT7CJxtA6gd4wEzxRQttKySEE0QSI3q99d5CRdiJoiKeVOdvKUeNmytoA6DXLJzsuq\n8nXEvkUHfkUXDKZ8+RInKn981OROB1ejtwwAvHscMcDSDFXv81pZaidR0nHbfzxU5Lm34pdVXH/U\nnfjcZXfJ2502jjitG3dOvAG/18/s979n5eLV1G9alz7nH0VuPVmInwpm4T3gmUz5Z0MF5L+HYe8U\n95hEaqTjcy+pVXRdmU72byfrSUTl6KIxRK5J50UXjUbVnRTxPGVtjs4dC4U3gTJAe8ObKlQOKu8V\nSepE3LTp3JKbX/8fj1z6HMpQmKbGDIbo3OcQRrxyDRCeku07pFeKI02uYCDIX4tWoxS07tQCizX1\nm8q0WVhaVL4KG/esnSSpEzWetEdIAB38G7zT0dqLsh8O9h6SRFSR1ma4BU00gV/QOgAYaPckcL8B\nZgFYW6GyhmG4TkA7jgDvtNKvHwiOY6QziIi7XmcdQY+Tu7Dwi8WUFLo56PC2ND1gv1SHlTLTX5vJ\nizeOJxQyAY3NbmXYU5fS57yjUxtYcHnp9GtlEjs7uAZB9h0JCyvdaa0hsADtfj+8dMnRE+U6o9YV\ni68N5N0tjrTW6KLR4N65riuIdr8OllaQPx5lyPRz5e09IdZaQ+Ew8H3HrpG9wCL0tuHo7OEYmZeC\nrHsUSeBwOTjytG6pDiPl5nz4A89c+0qZqWkP8NjQF8jOy6LbCSmshqCyibruNhLXYIzcexIWzr5A\na43ecSd4PiE8/a3B/yO6+DmoOzFcokrETdIatf42bxl3nTqaC1oP48Zj72HeR/NrVR2xCvFNB88k\nwkP8wfDXtBuCy8M/1KLSlFJgPzL6C2xdUYH54Pue8tO1Hih6DC1V44VIqnG3vlVuvSGEa/uNu+3t\nFES0G2s7MOpW/PWugYmLZV/h+wq8nxB+Ru983/eALkRvuy6FgaWnpCR2n7w0g1v6jeSHTxeyYdUm\nFn+zlIcGP8lzw19Nxu2TRhe/FKXjgR+8X6LNoqTHlA5U9q2RCw4rFyrndrTnIyBaFxAr+L5OYHRC\niN35PD42rNoU9fiqxf9gmnvbeZo4SilU7uOlz5S9TFpZmmHYOyYlrnSm3eOjvDdqCK2RFpxxlvDE\nbsfWIp6//jV8bn+ZljreEh/TXp3Jip9XJTqE5An9G/2YsoG5MXmxpBFlOwCV/y7YexHu6GEB+1Go\n/EkoW/sYrd0AzEqupRFCVIfVZkVFKUoP4aLMqV5zrOydUHU/gYzzwNI2vPaWPQrOWztAXSlGHBeh\nGO99ygahLcmLZR+Q8DV286bMx7BEzh8D3gBfvDGLNoe2THQYyWHZD4JRpv10AAypV1UZOvgPuvjZ\n8DA+Cpx9od4MlKVJmTcG5eyD9s+JkuBpsB+etJiF2NdZrBaOGNCVeVPnl2snabFa6D3oyJQndgDK\n2hSVc1eZr5nBv8Mf0G0dpCdsPNkOhtBqItYI1H6wtkp6SOks4SN27iIPoWDkhaqmqSnaWpLoEJJG\nZV4Rpc2YHRzHVWn3j9YBdHBFuLH9PkQHV6ILBoJ3Kugd4RZgnilQcDqE9uit6TwJjHzKf05xguNY\nlLV5ssIWQgBXP3YR2flZ2Bz//U7aHTZy6+dw+ejBKYwsNsPaAsPRU5K6OFOZQwF7hCN2cPRGWWTQ\nI54Snti1P+IADEvk2kWuLCdd+tbs9Qtam2jfXMzCuzAL7wq3n9FR1oc4TwTXOYQrk5c+0FQGWNug\nch+o5H01Zsmr6E2HowvOQm/uj7nlFHTg12p9PzWdDizFLLwVveUM0EWU/YQXAl2MLhpb5hylHKVT\ntUcD9tK1My7IGISq80gSoxdCQLhDystLHuOs/xvAfm0a0aRtY8655TReXvwo+Y3yUh2eSDJlOwhy\nx5bWEc0kPO3tBHt3VO7oVIeXdhLeeUJrzfVH38WfC/8i4Avu+rphMchrVIfxfz6N3Rkpk089rb3o\nrZdA8Pf/pvlUBljbo/JfQ6nIrWV0cCXaMx3whHvKajf454DKQDkHoCpQ4NIsHgfFT1Nup6fKQNWd\nmpajUGbJ21A0hvCu4lg/l1ZUw98iTudosxDMbWBphFJ79vEVsaRjBfZ4qG7HHSGqQmsTfF+gS94M\nP9Psh6EyL0NZW6Q6tCrT2gu+b8AsBnsnlLVNqkNKy+dewtfYKaV48LPbGXvRMyz4/BdsDhsBf5DW\nnZpz56QbamxSB6CLnoDAr5SpUK7dEFiCLn4alT0i4nnK2gqVfQ3a3IouOBfMTaWJoYF2v4d29kfl\njkapyAOmWvuh5FkidlvQPnTJS6jcUdX+/moSHdoARaOpWDX4EOHEr3xip4xcMHLjHF38eYo9zJ38\nI9s3FtKyY3O69O2AYVR8AD0YCGKxWmrEWiUhRHxpbaK3DwffHHbt+Pf8Ha4AkPcCynFESuOrKqWc\n4Oyf6jDSXlIKFGfmZHDf5JvZumEb61ZsoG6TfBq3bJiMW1eZ1ho87xA50fCFixBHSex2XWP7bRBa\ny66adpiAB3yfg/eI6PWRYm79DpX+sqcX7ZlK7FG63dg6Rk2Ka4MfPvuJBwY9hlIKvy+A3WGjToNc\nHpl5Dw2a1Y957udvzOLNe99j0z9bsDlt9B3Si6FjhpBVJzNJ0QshEs43MzzLU6aMUxAIorffAA2+\nRanUt2cTNVNS3x3zG+VxyFEH1fikLiwQpe5OKV2C1sHoh81t4P+W/5K63Q960CUxavgpB0RbxweQ\njlOMZgFQvqBpeQYq++ZER5MwG1dv5v5zHsVb4sNT7CUUCOEp9rLx703cesIDMYt2vz3qA54e9gob\nV29Ga43f4+eLN77m2sNvx+eRki5CpAvtnhijjJMPAj8lNR5Ru9TeYY+Es4GqE/2wUTd2v9HQlnB9\nnqjHY9T1sbQCS7TK6A5wnRn93FpK2TqWLqqN+SpwDUbZa2/Lpo9f+BwzWD5pN03NlrVbWfrd8ojn\nlRSWMGHUB/jcZRO4oD/Iln8LmDlhbkLiFUKkQMxuOQrMwqSFImofSeyiUEpB5mWUK1oJ4a9lDo19\nAUtjiDGiR4wFsOHK6GNK7737P5EDLE1QGUNi37s2cvYr3c0a60eyDir7hmRFlBB//bKagD/yz4XW\nmjXLIpe1WfT1Uqz2yB8kvCU+vpqQftPzomYLBUN8/c633Hz8SIZ1v5U37n2HbRulfV9cOA4ncnkQ\nwnXfbIckNRxRu0hiF4PKvAxcJxEuX7Lbf65TUBkXxT7XyALXKaXn7MmFyrw69vn2bqi674KjH6h8\nMJpA5hWouu+jjPRbT6WUHZU/ESwtSmsB7lw/YgCO8E7keu+F/15rsf3aNMJijfxrZxgGDfavRA9L\nIVIkGAhy24mjePTy5/n5qyUsX/AX746dyiUHDWf10jV7v4CISWVcEGXGxwHO41GWRkmPSdQeSdk8\nUVspZaByH0JnXv1fv1FHb5S1WcXOz7kbHVoP/p8I7+S0ACZkXYVyHrv3823tUHlPVTX8WkdZm0G9\naRD8FUJr0eSgDDsY9Wv1Fv/dnXp1P6a/OpNQsPx6QmeWg07HHhzxvE692xOMMtLnzHRw3PlHxzVO\nIWKZ9spMln63vMzSAL83QMAX4KHBT/HCzw+nMLraT1kaQf4b6G3DQW8DLOHWiM7+aVcRQcSfJHYV\noKzNwHph5c9TLlT+6+jAUvB/Hx6JcvRFWWLvfNyXKaXA1gFsHSIUM6n9mrffnysfuZAXb3wD0zQJ\n+kM4Mx1YbVZGfXI7lijFvDNzMzn/jjOZ+NDkMm+mVruVek3q0uf8o5L1LQjBR899Xm69J4DWsGbZ\nv6xftbGWbJKruZStI9SfCcGl4TV31nYoS71UhyVqAUnsqklrDd6P0MUvgrku3A8243JUxlm7SnIo\nW3uwtU9xpKKmOPXq/nQ7oTOfvzaLLWsLOKBrG44bcjSZORkxzxt8x5nUb1qX8fe+u6vcyfEXHMPl\nowfjcEUuli1EIhRvK456zGq3UrS1WBK7OAh/0I08ii9ENJLYVZMuehDc77KrmHDobygahQ4uLN0A\nIUR5jVs25OKR51b6vH4X9abfRb2lQLFIqQO7t+XbKT9GLM8TDIRoesB+KYhKCAGyeaJadPAfcE+i\nfIcID3imoQO/pyIssQ+w2qyS1ImUOf+OM7C7yi/ud2TYOeWKvmRkR6omIIRIBknsqsP3BWWb1O/O\nj/Z+lsxoRC2ktWb5wr/4cdrPbFqzJdXhCFEhbbu04o6JN5CVl0lGjouMHBc2p43jBvfiiocrvx5Z\nCBE/MhVbHTpAeLdrJGa43pAQUaz4ZRUjz3qUbZsKsVgMAr4Ah/XrxG1vXYcrS0Y8RM12xICuvLdh\nHL/O/QNPsZcDe7Qlr0HN79MsRLqTEbvqsB9J5Dp1gMpAOXolNRxRe2zfXMiNve9h/cqNeIu9lBS6\n8XsDLPh8ESPPfjTV4QlRIVablc7HHsIRA7pKUidEDSEjdtVh6wi2ThD4Gdh967893BbMfkSqIqsQ\nrTUEf4PQBrC2RFlbV+58sxjtmRJuWK1cKNeZ4Tp/Sj4v7M1nL39JMFB+tDfgC7D4m6WsXb5OFqCL\nGsfn8TFr4rd8/c63GFYLfYf04ugze2Czx2ifKIRIKknsqkEpBfkvoXeMAs8UUAZoE5wnoXLuqtEJ\njg6uRG+7GsyNgAE6iLYdhKrzHCpqn9rdzg+tRxecBWYxOzePaN9csB2ArvMCRgWusS/7ZdZv+D2R\np+otVgt//LhCEjuREisXr2bS6Mn8/v2f5NTN4tRhJ9D3gl6UbHdz7eG3sXXDdrwl4Q+yS+b8znuP\nfMRjs0fiynSmOHIhBEhiV21KOVG596Nz7oDQFjDyUUbsemSppk03uuA80NuB3coVBJagt10MdT/a\n645LXXgLmFspu8bQA4FFsPkoTOcAVO59KCVrxSLJrZ+DUuGCrntShiI7v3a3ThO10w+f/cT95zxK\nwBvANDUb/t7EM9e+wjfvfUd2fiab/tlSZqTZW+zln9/X8uZ973HF2AtSGLkQYqekDSm5izz4vZXf\nTDBv6nz+d/htnFn/Eq7sPIIv35odsXZSqinlRFmbxj2p0zoQHl0LbYjfNT2fhNvTsOffYxBCayCw\nIPb5oYLd2qRFEgLvNPS2/8W+jrkVs/g5zIILMbddh/Z9WyP/bRPh5KF9cWREXp+plKJL3w5Jjkjs\n60LBEGMueAqf249p/vd76C3xsfib3/jm3e8iLh/wewNMG/dVMkMVQsSQ8BG7eVPn89LNb7Jh1UZA\ncWifQxj29GU0bdt4r+e+ce+7vPfIR7ta1+woKObJq19i0Te/cePLVyc48uTTgaUQXA5GfbStmy2S\nhQAAIABJREFUB3jehOJngSDoENraCpU7GmU7qHo3CiwA3NGCgMASsHeLEei2cIPqmLt+feCfjw4s\nR9kOiHCbZeit55fuLPaGv+b/Bhz9IXdM2tdo69T7YI499yhmTZq7a1rLYjWw2m3cPuF6WbMkkm7x\n7KWEQpHLN/ncsT+Uu4v2rOUphEiVhCZ2MyfN5bHLny/zUFg4YzH/63ErL/3yCA2aRe+ZunltAe+M\nnULAGyjzdW+Jj1kT53LaNSfQ5tCWCYs9mXSoAL1tKAT/AqUABWjQIcpsygj+ji4YiLa2BUcfVMZg\nlKVB5W9oqUf4nz5SU3kjXHg58BtY20dOsCxNwmsJKyKwAPZI7LTW6O3Xgi4q+1rtCdcG9B0PzuMr\ndv1aSinFDS9dSc/TuzHlmWkUrNvGQYe35az/G8D+7ZqkOjyxD/IUeat87n5tGsUxEiFEdSQssTNN\nk+evf73cJz2tNd5iH5PGTOG6Z4dGPX/uhz9EbQIf8AWZNWluWiR2Wmv0tksh+Cfhkbm9nWFCcBkE\nV6Hdb0L+BJTtwErdU7nORJe8SeTEzgeeqWjvZLDsD3kvoSxlF/Er5UJnDCrtuhHrzcAAFWFqOris\ndNNGBNqNdr+JSvPEDsLJXY+TD6PHyYelOhQhaNe9DQFfpGdCbI4MBxfde04CIhJCVEXC1titXb4e\nb0nkN/1QMMS3U+bHPN/n8RMKRh4VMkMmnhJfxGO1TmAxBP8mcpIVix90MXr7DZW+pbK2hqyrARdE\nTJ9LwqNnwRXogsFoXX5djcq+CZz9gVhThiFw9Cn/ZbOAmJ8pQptjxi+EiG39qo3M+eB7fpn1K6FQ\ntLWwZdVtnMcxg47E4bJX6PUZOS7sTjsX3H0WvQf1rE64Qog4StiInWGomKNPhhF7DVXnYw/GZrcS\nCpZ/KLmynHQ9vlN1Q6wZgr9TgWG66EL/ooMrUdZWFT5Faz/KeiA68yLwL4bQP2BuoHxyaYZ3zvq+\nAWfZBE0pG6rOw5jBG6DwxvC6PHYfnXVC9q0oI6d8ANa2MdbnWcDescLfixDiP163jwfPf4KFXyzC\nareCBpvDyl3v3kin3gfv9fz/e+lKHE4b01+dFfHZu7sRr1xDl74dyMzNjFf4Qog4SNiIXZO2jaOW\nbLDarRwz6MiY57fr1oYDurXG7iw7ImS1W6i/f116nNIlbrGmlFEXlKXq5ysbmNsr/HLt+xa96Qh0\n4QgoeQMCPxFexxdlxFCXhIsYR2FY90PlT0DlPgTWg8FoAPaeqPxxGJnnRw7Z0qB0JC/SrlAbKjP6\nFL0QIrrRQ55i4ReL8HsDuHd4cBd5KNxSxJ0DHmL9yijLH3Zjs9u4/oUreeq7UbFf57Rx9JmHS1In\nRA2UsMROKcXw54eWG9a3WA2y6mQy6KbT9nr+A5/cRq9zjsTutJGRHW4yfVCPAxh08+ms/2vvD6la\nwXEMkadDK0j7wdqmYi8NrkFvuya8aUHvLCzsLZ0ajRaDE1RezOsqpVCuARj1JmM0mIuR/xrK3j32\nOXVGg6O0JZvKBJUFKgtV50lUBb8fIcR/Nq3ZwvzpP+PfY8MZQNAf5IMnPqnwtQ44rDV1GkZvEdbv\nwmOqFKMQIvESWseux8mHMeqz2zno8LYYFgOHy86x5x3F8wvHkNewzl7Pd2U6ueX1//HuhnHc+MrV\n5NbNZsXPq3h2+Ktc2XkEI469hx1bi/Z6nZpMKTuqzjOgXMDOJNgAnOAaDI6+QHaUs53gGhh5ujMC\n7Y62YcIk+nSwBtdJFbp+ZSjlwsh7EVXvE1TOSFSdx1ANvkc5j437vYTYF6xavDo8/RpBMBDit2+X\nVep6j319H1Zb+dmE/do05Lrn9s1RdW26MUsmYW4bhll4G9q/YJ+pvSlqj4TXset0zME8Ne/Bal0j\nFAjx2NAXKCksW3vtt++Wc9eA0Tz5bexpg5pOOY6EetPR7gnhzRSW/cKlTGyH7HqN9n2D3n4TEABU\neKTOeSIq586K3yjwS+n5kTgIJ3ih0v9bABvk3I8y8qv2jVWAsjYHa/OEXV+IfUV23Wy0GT3JyGsU\nfQQukv3bNWFq4Xhev/sdvvt4AY4MB2fdcDJ9h+ybo3U69C+64Gww3YTrgCq05zNwngC5o9O+9qao\nPWpFS7Fpr3xF0F9+pCnoD/LXotWs+HlVrS99oiyNUdk3Rj/uOAYazAP/wvC6N1sHlCV6HcCIjMbA\nIiKOzikrZN8BgYUQXAnWA1CZF8m0qBC1xIHd25BZJwNPcflqBM5MB6defUKlr2l32rli7AXSLgzC\nFQjMrYQ/+EL4OeoB7/TwkpoEzGxEjEOb4P8eQuvB2gJsXSSpFGXUisRuydw/8EVpmK4ULF+4stYn\ndhWhlBUcPap+fuYQtP/rcCmT8gdRrjNQGWdV+rra3IEueRk8k0F7wd4dlXVt9TtkCCEqzDAM7n5v\nBLf0G0nQHyLgC4/OOzMdHH3m4fQ4OU02nKWADv0Lgd/5L6nbnQftfh2VhMROB5aht11euka69AO6\n0QDyXkVZmyb8/qJ2qBWJXX6jOhiGKtO/cCfDMMitF20NmtidsndDuwaD+23CO2FNwpsjrKi851Gq\n8ksutVmELhgIoY3sKnfi+wrt/xbyxqFitSYTQsTVQT3a8urSJ/jouc9Z9PVv1GmQyylX9aNrv04y\nqlMdoc2g7KU9tiMdT/xmPm260VuHgC7c497/hL9e/ytUdSosiLRRKxK7k4f2ZeaEOZH7FSroduKh\nyQ+qljJybkY7+6M9k8IPK3sXVMa5VV5Hp0teh9Amytaw06A96MLboN4MeUMRIonqNanLpaMilxoS\nVWRtHqP2pgJr5br/VIn309Le2nsyw8mebzbI5jNBgnfFxku7bm04bdgJODMd7MwRLDYLjgwHd0y8\nAbtDGqZXhrJ3wsh9CCN/HEbWNdXbHOGZTJl+trsLbQoXPxZCiFpMGXng7Efk2psOVNZVCY9BBxYT\n3rQR6aA33KpRCGrJiB3A0DEX0OPkw5jy9DQ2r9lCu+5tGHjdSTRp0zjVoe3jYrR2U5bwA0cIIWo5\nlfsA2twO/tJ2mMoCOgg596DsSZg1sjQiXBIr0sihA4x6iY9B1Aq1JrED6NirPR17tU91GGJ3jl7g\nmUK4TMqeLFCJVmdCCFFTKeVC5b+CDvwZrh6gssDRG2VE7rAU9/u7zkAXvxDlqA6XXRGCWpbYiZpH\nZV6F9k4DvecUgQuyrkcpmSbf3b8r1vPO2KksmvUrrmwXJ19xPCdceiw2u/w9CVEbKFtbsLVN/n0t\njdE5d8GOBwjXIw0BNsASLvCepART1HyS2IlqUdbmkD8BXXg7BFcAFlBOyBoetVfsvmrpd8u4pd8D\nBHx+QsFw2YQXR4znyze/4eGZ98paUZFw2zYVsvibpRgWgy59O5CZk5HqkEQlGBnnoO2HoUsmQGgV\n2A5Cuc6TUieijLRL7Nav3Mh7j37EwhmLych2cdLQvvS/5Fh500wgZWuPqjcFHdoUXlNnaSLb7veg\nteahIU/hLSm75tDn9vHXotV8/tosBlzVL0XRiXSntebFm8bz8XOfY7FZUCiCwRCXPHAuZ90wINXh\niUpQ1tao3LtSHYaowWrFrtiKWjZ/BVd2HsFnL3/FuhUbWPHzKl4cMZ4be9+D3xetlZaIF2VpgLI2\nk6Qugr9/W8P2TYURj/ncPj59cUaSIxL7kvcf+5hPXpiB3xvAU+TFXeTB7/Hz+l3vMOfDH1IdnhAi\njtImsds5IuIp9hIK/reQ3+f2sWrJaqaN+zKF0Yl9nafIg2GJ/utWsiNKGQMhqsk0TSaNnozPXX4H\nu8/tY/y976QgKiFEoqRNYrdm2Tq2/Ls14jGf288nL8iIiKiagvXbWDhjEX/+tBKtozdZj6XFIc0I\nBSLtHAaL1aBzn0OqE6IQUe0oKMJTHL0s0Zpl65IYjRAi0dJmjZ2nyIPFGj1PdRdF6I8qRAw+j49H\nLnuebyf/iN1pIxQMUadBLne/dyNtu1SujEtGtotTh/Xno+e+KDdyYnPYGHTz6fEMXYhdMrJd7Oor\nGkFWrmygECKdpM2IXbP2TaOOiBgWg47HSP07UTmjhzzNvCk/EvAFKCl04y3xsWHVJoYfeQfzp/9S\n6etd9tBgTr2mPw6XnczcDJyZDhq2qM9D0++kaVsptC0Sw+60c+Rp3bHYyq99tTttnHzF8SmISgiR\nKGkzYufKdDLwupOY/PS0iCMi5912RooiE7XRxtWb+XHaT/i95TfdBPxB7jp1NEee1o07Jl2PxVKx\nzSIWi4Urxl7AkLvOYuXi1WTmuGhxSDPppSsS7n9PX8qyH1ewfXMh3pLw89GZ6aDZQU0573Z5Nora\nTZvF4P0UHfwTLE1RrlOr1yqzlkubxA7gklHnYWrN1KenYbFbMUMhsvOyuPXN62h2YJNUhydqkWXz\nV2C1WyMmdgChYIgfp/3EB49/yjkjTq3UtTOyXRzSMwlNw4UoVad+Li//+hgz357D7Pe/w2Kz0ndI\nL44+swdWW1q9DYgItA6g3W+B+00wt4O1NSrrfyjHMakOrdq0fxF62yWgQ4AHcKKLHkPnPobh6pvq\n8FJCVWYxeNeuXfWCBQsSGE58uIs8rFz0N65sF606Nq/UiIhpmnhLfDgzHRhG2sxUi0r66asl3Hfm\nw7h3xF6bmd+oDu+sezlJUSWWUmqh1rprquOoaWrLc0+ISLQ20dsuBf9PwO51NJ2QfRNG5gWpCq3a\ntPajN/UEHamUlBNV/yuUpX7Ma6Tjcy8tM5eMbBeHHHUQrTu1qHBSF/AHGHfb2wzMu5gz6l3CwLyL\neeX2CQT8Uv9uX9TpmPZYrHufYt0WpTadEELUCL5vIPALZZM6wn8uGhuexqytfDOBYJSDGu35IJnR\n1BhpmdhVltaae04fy5SnPsNd5CEUCOEu8jD5yU+5Z+DDqQ5PpIDFauGOidfjyHDEfF3dxnlJikgI\nISpPeyZH6OVdSlnBNzu5AcVT6F/Q0Ur5+CC4Kqnh1BSS2AF//LiCJXN+x+fxl/m6z+Nn8TdLWTZ/\nRYoiE6l02PGdeH7hGFp1ak6kgV9HhoOzK7m+TgghkkrHWk6igeg1Dms8SzNQ0T58O8HaNqnh1BSS\n2AHff7IAn9sf8Zjf6+f7TxYmOSJRU+zfrgnP/PAQ3U/qgiPDjmEoLFYLDpedo8/swenXnpjqEIUQ\nNdzfv/7DyLMe4d4zH2blkr+Te3PH8YAr8jEdAvvhSQ0nrhy9gSiJnVKojH1zx7dshwIMw0ApiLSP\nRCkVsxWUSH82u40HPr6N5Qv/4sfPfsawGBx5WjdaHLx/qkMTQtRwF7e7ln//3LDrz99O/pFGLerz\n+p9PY7FYME2T2e99x9Rnp1O4pYiDjziAs286LW6VHFTGAHTJs2D6gd1rvTrBeSLKUntraCplg/zX\n0VsvAvygS4AMUKDqPLvPljyRxA448rRuvPfoRxFH7WwOK0ecmlYbZkQVHXBYaw44rHWqwxBC1BIj\njru3TFK304a/N/N/x9zN47PvZ+TZj7Lwi0W76gv+++d6Zr0zj5FTbqZL347VjkEpF9R9H114O/i/\nC6+rQ4NrCCr7hmpfP9WU7UBoMAe8X0BoFRiNwXkCyshKdWgpI4kd0LZLK3qcdBg/fLawTHLnyLBz\n+CmH0aZzyxRGJ4QQojZaNOu3qMeWzlvO9x8vZOEXi3cldQBmyMTn9jHq/Cd4d/3LFS6AHouy1Efl\nv4w2d4Tr2FkaoqKuTasZtLkNdACM+nutbqGUHVynJCmymk/mGEvdPnE4F9x9NnkNc0FBXsM6XHjv\nIG57e3iqQxNCCFHLeDx735Tw4ZOf4C3ZswxJWNAX5Nc5f8Q1JmXkoKzNanRSpwO/Ym45A73pKPTm\nPujNvTE901MdVq0iI3alLBYLg24+nUE3n47WWto8CSGEqDK7fe9vryWxCqArKCmMUqYkTengCvTW\nIWXLs5jrofBmTEwM10mpC64WkRG7CCqS1Gmt8Xv9mKaZhIiEEEJUxa9zf+eegWO5tP313H36GJbM\n+T0p97VYLGTlZUY9npHjolv/ztgctojHA74gB3RtlajwaiRd9CToSCOYXigaTWU6Ze3LZMSukrTW\nTH91JuPve4+t67dhtVnoc/5RDB17ATn52akOTwgh4qZwyw5mvPkN//65nmYHNaXvkF5k59WeRenv\nP/Yxr989Cb/Hj9awdtm//PTlEi6452wG3XRawu//+DcjGdrxxojHHv36Puo0yGXqs9MJ+Mp2OLK7\n7PQ8vTv1mtRNeIw1iv9bIMpgibkdzA1Qi3fxJouM2FXSW/e/z7PDX2PL2gLMkInfG+DLt+Zw3RF3\n4KvAmgohhKgN5n/+C4NbXMPrd07ikxdm8Mptb3N+s6v4ZdavqQ6tQjavLeC1Oyfic/t3lbLSGnxu\nH+PveYdN/2xOeAwtDmnGhDUvcHDPdhhWA8NqcNDhB/DW38/RpnNL6u2XzyMz76Vx64Y4Mx1k5mZg\nc9roddbhjHj1moTHV/PEGmsy93Jc7CR/S5VQvL2ESaMn4/eW/XQV9AcpWLeVmRPmcuJlx6UoOiGE\niI+ibcWMPOsRfO7/PqzurBhw92ljmPTvS2RkRyl6W0N8PenbqFN3oZDJrEnfMujm0xMeR/0mdXli\nzgNRj7ft0oo3lj/NysWr2VFQRMsOzahTPzfhcdVIzpPA8y4R+79aW6Is9ZMeUm0kI3aVsOjr37Da\nIufC3hIfMyfOTXJEQggRfzMnzI2aFGmt+ebdeUmOqPKKthcT8EVuEB8KhJjyzDRW/fpPkqOKTClF\n604tOLRPh303qQNU1jAwcik75qQAFypnZIqiqn0ksasEpVT4ZyzWcSGEqOXWr9wQtc2it8THxtWJ\nn8asrvaHt8OV7Yx6fMvarQzveSfr/ipfQLgmC4VCzH7/O27pdz/Dut/K63dNZOuGbUm59z9//MuY\ni57m/OZXcdnBNzD5qU/xeyP/nFSFstRH1Z0KrrNA5YBygeNYVN1JKPuhcbtPupOp2Ero1Ls9QX8o\n4jFnpoM+5x+d5IhERa1fuZH3Hv2In7/6lYwcF6dceTx9L+iFzR55R5oQ+7JmBzbFmekoUzh3J1eW\nkyZta/4C9m4ndiavYR187o2Yocijjz63j7cf+ICbXhuW5OiqJhQMcdepo1ky5/dd/zarlvzDlGem\n8/jskbTs0Dxh914y53duP2kUfm8AMxTe4PDK7RP46u05PDb7fuxRdvdWlrI0QOWOhFwZoasqGbGr\nhMzcTIbcfRaOjLLFHa12Kw2a1aPPeT1TFFn62765kEljpnD/oMcYd+tblfqU/cePf3JF5xF8Nu4r\n1i5fx/IFf/Hc8Ne46bj7CPgDe7+AEPuY3uf2jNoj22Kz0Ousmt843mKx8PjskbTqFL1zkBky+f7T\nhUmMqnpmvDm7TFIHEPAFKCl0M+q8JxJ2X601oy94Cm+Jb1dSB+F1l3//tobpr8xM2L1F5UliV0nn\n3TqQ61+4gsatGgLhkbqTLj+Op+aNwu60o7Vm7uQfuOHouxjc8mruOPlBfp2bnLpJ6erXub9zYev/\n8ebI95j93nd88PgnDO14I5+N+3Kv52qteXDwk3iLvYQC/422et0+Vvz8N9NfnZXI0IWolTKyXTw0\n/U4yczNwZTlRhsKV7SQ7L5MxX9yFw5W8zgWmafLTV0t4Z+xUpr3yFcXbSyp8bn6jPO79YAQ2Z/TR\nJIu1+i27EmHVr//w4PlPcG7TK7m0/XA+eOITJj/9acRRVIDVv69lzIVP4y6Of1HjVUv+YUdBUcRj\nPrefz16eEfd7iqpTlSn417VrV71gwYIEhlO7hEIhDMMos7bu6f+N44s3vi7zy+fIsHPlIxcy4Kr+\nqQiznKXfL2figx+y4pe/yWuYyxnDT6bP+UdhGDUvz/f7AgxqPDTiw9zutPHyksfYr3WjqOevXrqG\nYd1vK7O7b3etO7XghZ8fjlu8tZlSaqHWumuq46hp9uXnns/jY+6HP7Jh1Sb2a9OIngO7x23KrSIK\n1m/jpuPuY8vaAvzeADaHFdPU3DjuKvqcV7GlL1prLm53HetWlB/lt9osnDrsBK5+7OIqxbdy8Wpe\nv3sSv8z8FavNSu9zj2TIXWeR3yivStfb6ZdZv3LngNEEvH5MM/we7XDZCYVMgv7IG0J21//SPowY\nd3W1Ytjdb/OWcftJo3BH6ZTRuFVDxq94Jm73S6Z0fO7VvHfyWsRisZRJ6v78aSWfvz6r3Ccqn9vP\nC//3Bju2Rv7Ek0xfvj2bm4+7jx8+XciWtQX8uXAlT179Eg+e/0SNrOr9w6c/YYYir2s0Q5rPXo49\naldS6MZijf5jXrJj32rZI0RlOFwOjht8NIPvPJNjz+2Z1KQO4J7Tx7BuxXo8xV5CwRDeEh9+j5/H\nLn+Bv39bU6FrKKW44cUrcWTYy3zdYrWQnZ/FubdUreTJ0u+Wcd2Rd/D9xwvwFHsp2lbMZ+O+4qpD\nb2Lbxu1VuiaEE9ExFz6Nz+3bldQB+Dz+MtOgsXz+6kwmPPRhlWPYU6uOzcrMeOzOYrVwWL+OcbuX\nqD5J7OJoxvivCXgjr9kyLAbfTv4xyRGV5Snx8sSVL+Hz/FewE8K73H749Cd+/mpJ6oKLouDfrQSj\nPFCCgSAbVm2KeX6LQ6I/kAyLQafeB1c7RiFE/P392xr+/m0NoWD5ZCbgD/LBE59U+Fqdjz2ER2be\ny6HHdcDmsJGR7aLfxb15/qeHyWtYp0rxPX7li/jcvjLP0lAgRNHWYt4e9UGVrgnw1y9/R+0Ra4bM\nCldfmFCNGPbkynJxxvUnl1tfDmBz2jhnROK7eIiKk12xcVS03V3mE9buQsEQnqJIPfCSZ8H0X6KO\nXnlLfEx/bRZd+tasT17N2jctXQNTPmG2WC20PrRFzPMzsl2ccnU/PnlhRrnpWLvDxqCb5YEkRDzs\nKCjio+emM/uD77FYLBx/4TGceFkfXFlVK2S8dvm6qOvfzJDJqsWrK3W9A7u3ZeyMu6sUy562btjG\nv3+uj3gsGAgxa9K3/O+py6p0bW+JF2VET97yG9fB7w1QtLU45nWilaupqovvPxdlMfjw8U8wLAah\nYIj6Tetx65vX7lpzLmoGSeziqMtxHfj2wx/wFJdP4CxWCwcfdWAKovqPp9gbNfEEKN4W+0GRCp2P\nPZisOpkR/05DwRDbt+x9envomCFgaj5+4QtsDhuhkEl2nUxufes69m/XJBFhC7FP2bRmC8O63Yp7\nh3tXZ541f/zL1Gen88wPD1Wpv2yDZvWiTj0qQ7Ffm9SVXAn6gzHXJEebJaiIVp1aRD3farPQc2AP\nrnn8Yi5oNYzNawuqfJ/KMgyDS0aey3m3DmT10rVk5rhoesB+Sbu/qDiZio2jY84+gsw6GeXKBNgc\nNtoc2pJ2XVunKLKwg3u2wwxGfmA4Mhx0O7HmFYA0DIM2XaKXK/jsxRl4o2yM2MlisXDVYxfz7vqX\nGTn1Fh7/ZiRvr36ejr3axztcIfZJT//vFXYUFJVpt+jz+Nn8zxbG3/tula7Ztksr6jWpG3Hq0e60\nMfC6k6ocb1W5izy8ce87DD/qznKtJXdShuKwfp2qfI+MbBcDh5+EM8q059k3DsBitXDZQ4NjXqdl\nh2ZVjiEWZ4aDdl1bS1JXg0liF0d2p52n5j3IQYe3xe60hxs6O2z0OLkLoz69vdrXXzZ/Ba/dOZFX\n75jA0u+WVXqzQ5M2jenavzN2V9lFxIahcGU66HdR72rHmAh//rQq6jHDYrDmj38rdJ3M3Ew69mpP\nm0NbSpcQIeLE5/Gx4PNfIo6uBfxBZoz/pkrXVUpx/0e3kFsvG1dWuIOExWbB7rRxwT3ncFCPttWK\nu7I8xR7+1+M23hkzlS1rt0Z9/jpcdi685+xq3euSB87jzBtPwZnpJCPHhcNlp3n7pjw66z4atWgA\nQJ/zj+KYc46IeL7FanDPhyOqFYOovaTcSYJsXL2ZgnVbady6EXkNqtf7LxQMcf85j7Lgi8X4vX60\n1jgzHBx8ZDvu//jWSnVP8Hv9PDVsHDMnzMXutBHwBWjZsTl3TLi+xq6TuLT99VGTN0eGnRd/eYQm\nKZyWSRfpuO0/HuS5F9uOgiIGNbkiahkOi9XCdP+kKl/f5/Hx9Tvz+G3eMvIa5tLvot4p+X1/Z+wU\nxt/3Hn5P5LVrVruVlh2acekD57FtYyGmaXJYv07U2y+/yvf0eXys+WMdGTmuqGWdFs5YxJNXv8TG\n1VswLAZd+3Vi+AtXVOu++5J0fO5JYlcLTHjoQyaM+qDcYli7y87A607i8r0MyUdStK2YdSs2UKdB\nLg2b149XqAnx4ZOf8urtE/BFeKDuf+B+vLr0yRRElX7S8QEXD/Lci01rzblNr2Tr+sj9Stse1orn\n5o9JclTxd+lBw1mzbF3EYw6XnQen38H8aT/zwROfYrVaAE0wYHLyFX255olLZJaghkrH555MxdYC\nHz7xacQdTn6Pn4+f+xzTrFhto91l52XRrlubGp/UAZxy5fE0P3j/MnWoLDYLriwnt4y/LoWRCSGU\nUlw8clDEUhiODDuXPHBeCqKKv2hr6iA8KvnTjMVMeXoaAW8AT7EXT7GPgC/AtFdm8uETnyYx0tTQ\nwdXokvHokjfRwbWpDmefJoldDRYMBJn9/ncUbt4R9TU+jz9qV4V0YXfaeXz2SK4YeyGtO7Vgv9YN\nOfGyPry46JGUb0gRQsCJlx3HhfeegzPTQUaOi4wcF5m5GQx//gq69e+c6vDiotsJnaOWX9FaM2P8\nNxHbffncPiaOnlwjC8DHg9YhzO03o7ecgi56GF00Fr3lRMzCe9P2e67ppNxJDbWjoIjhR91Jwb9b\nY77OmemI+Ek53diddk69pj+nXlMz2rIJIco6Z8SpnHpNf5b9uALDYnBgjzaVWv9b0w265XS+mjAX\nT1HZtlqODAeD7zyTcbe+HfXc4m3FeN0+XJnORIeZdLrkZfBOB/ZIaj2T0da2qMzKLxUXsIeiAAAg\nAElEQVQS1SMjdjXUo5c/z4aVGyPWb9vJ4bJz+rUn1sger0KIfY8zw0Gn3gfT4eiD0iqpA2jUogGP\nzx5Jmy4tsTlsOLOcZOVlcumocznnptPIzM2Ieq7FZsXuTK+/DwiPVFLyChDpfcoDJS8mOySBjNjV\nSEXbipk//ZeorbQg/CmxS98ODLnrrCRGJoQQtcu8j+bz2p2TWL10Dc4MB8dfeAwX339ulYomt+7U\ngucXjGXLuq14i700btVw1/TsSUP7MvWZaeXW4tkcVo6/oBcWS+Rp3FpNu0HHKGxvbkBrLRtHkkwS\nuxpo28ZCrHYLAV/kxbo2p43HZ4+kbZdWcb3vjoIivnjja/7+bQ1N2zam38W9yW+UF9d7CCFEsnw2\n7kueu/61XZvPPMVePhv3FQs+X8QLP4+tcruzSKVELrz3HJbMXsrqpWt3zbS4spw0atmAoWMvqPo3\nUZMpFyg7aE+U43UqlNSZ3u2w4xwwd0Dm1RjZF8U50H2LJHY1UP2m+YQC0Xe6turYPO5J3aKvf+PO\nU0ejQyY+jx+708ab97/PbW9dx1EDe8T1XkIIkWh+r58X/u+NchUFgv4gBeu3Mv21WQy8Nn7dK5wZ\nDp6Y+wDff7KQmRPmoE3NMeccSc+B3bHa0vOtVikD7ToL3O9Sbo0dTsgYEvVcbbrRno+haCRleoGX\njMIsGQV1ZmM4I9fuE7Gl509bLefKctF3yNF8+dbscsP6zkwH599+Rlzv5ynxctdpo/Hutp5v531H\nD3mKN1c9V+0iy0IIkUxLv1uOMiKPFvncfr56a05cEzsIlz3peXp3ep7ePa7XrclU9gh0YDEE/wxP\nzQKoDLB2RGVdFfEcHVqHLjgbzM3RL7y9FzRanoCI05+suq+hhj11KR2PaY8jw47Nbg3/32Hj3FtO\n58hTu8X1XnM/+AGi7EoPb+P/Oq73E0KIRIvU4qzs8ehrmEXFKeVC5b+DqvMUuM4B1yBwXQ6Bn9Ab\nD8HccADmhnaYhWN3naO33wBmwV6vbZZ8mMjQ05aM2NVQdqedh6bdyV+L/uaXmb9id9npeXq3hKx5\n27RmS9RaeH5vgPUrN8b9nkIIkUgHHXEAoWDk5M7hsnPMoCOTHFH6UsoARy+Uoxem50sovGaPV2jw\njMM0t6JyhkNgKVCBwvrFT0NmfGeo9gWS2NVwrTu1oHWnFgm9x/7t9sOR6cBTVH7LujPTQctDmif0\n/kIIEW+uTCcX3nsO4+99t8wHV4vVICsvk5OHHp/C6NJY4fXRj/k+BPN8UDbQFSisb5E1dlUhU7GC\nI07tit1pj3jMMAyOG3J0kiMSQojqO2fEqVz7zGXUa1oXi9WC1W6l58AePDt/DFl1MlMdXpoq3/5y\nd9q/HHTs1+yS9Voc4tn3yIidwGa38fCXd3Pz8SPxefz4PQHsLhuGYTDq09vJzIleeFMIIWqy/hcf\nS7+LeuMu8mB32tKucHLt4wfnCeD9nPI7aXdjtMRwpl+njmSQxE4A0LJDcyaueZEfPv2JtcvX07B5\nPY48rVvUkbw9aa1ZNn8FX741G3eRh279D+WoM7rLQ1QIkXJKKfmAmjQGMdfPOc5AZQxEm9vB/yMQ\nLP1vN/aTMfIfT2CM6U0SO7GL1Wat0jZ9rTWPXPoc37z3HX6vH21q5n7wA6/eMYGn5o0ir2GdBEQr\nhNjXBINBRg16nHkfLcAMmRiG4vABh3HHO/+HXT5E1gyuoeCJ0krMaIthDY/Cqfxx6OAK8C8AlQmO\nY1FG5buBiPJkjZ2otq/ensPs97/D5/ahzXDdFE+xl81rChh70TMpjk4IkS4uOXA4cyf/uKuUiWlq\n5k1dwCUHXJfiyMRORu6N4Iywk9Voi9Hg0zJfUtY2qIxzUa4BktTFkSR2otref+xjvCXl10qEgiEW\nfbOU7ZsLUxCVECKdzPnwezas3BTx2KZ/tjBr4twkRySiMeqMxmi0HHKfgex7od7ickmdSBxJ7ES1\nFazbGvWYzWFl24btSYxGCJGOPnru85jHP37hiyRFIirKcPXDyDx/1/SrSA5J7ES1NW3XJOqxgD9I\ng2b1khiNECId7a2TRCgonSSEAEnsRBycf/sZODIc5b5ud9o45pwjycyVelFCiOo5/sJj9nK8d3IC\nEaKGk8ROVFu3/p254J6zsTttODMc2Bw2HBl2Du55IMOfG5rq8IQQaaDfRb3JqZsd8VhWnUxOGnpc\nkiMSomaSciciLgbddBr9LurNd1Pn43X76NirPW0ObZnqsIQQacIwDMb/9Qw3H3cfyxeu3PX11p1b\n8MjMezAMGacQAiSxE3GU1yCXk4b2TXUYQog0lZmTwbPzx+D3+tm8poD6+9etcBF1IfYVktgJIYSo\nVexOO03aNk51GELUSDJ2LYQQQgiRJiSxE0IIIYRIE5LYCSGEEEKkCUnshBBCCCHShCR2QgghhBBp\nQhI7IYQQQog0IYmdEEIIIUSaUFrrir9Yqc3A6sSFI4RIoeZa6/qpDqKmkeeeEGkt7Z57lUrshBBC\nCCFEzSVTsUIIIYQQaUISOyGEEEKINCGJnRBCCCFEmpDETgghhBAiTUhiJ/6fvfuOk+QqD37/e05V\n554cNmijNkirnJAQQiCTJGQMxryvMRhjMMGkaxuM7de8tsG+trk2TmB8wdfYgJHBYMAmR71WQgIJ\nERRX0q4255nZ6UmdqurcP6pmp2emuydP98w8X31G293VVXWquvvUUycqpZRSapXQwE4ppZRSapXQ\nwE4ppZRSapXQwE4ppZRSapXQwE4ppZRSapXQwE4ppZRSapXQwE4ppZRSapXQwE4ppZRSapXQwE4p\npZRSapVYFYGdiLxHRD7W6HSohRORtIh8TURyIvKZRqdnKhG5R0ReN8v3vkBEDi5tipRSK5GIvFFE\n7ogeOyIyIiJb5rGdXxWRbyx6AtWK1RSBnYgcFJHTIpKpeO3cl34m1to/t9a+cQnSdYeIFKIfXE5E\n7hKRSxd7P40mIh+NjnFEREoiUq54vtwZxiuBTqDLWvuqZd53w8zl+67UYovy4HzF735ERD48i/Vu\nEpGjC9jvlin7tCIyWvH8xvlueyWx1vrW2qy19nC994nIThGxU9b9pLX2xUubQrWSNEVgF3GA32x0\nIqp4h7U2Sxhs3AF8qrHJWXzW2rdEmUoW+HPgs+PPq2UYIuIuYXK2Ak9Ya725rrjE6VJqtfu5it99\n1lr7jsXYaL3fpbX2cOU+o5cvr3jt7irbcxYjXYtJ8x7VTJopsPsA8G4Raa+2UEQ+KCJHRGRIRB6s\nvJMTkfeJyG3R42+IyDumrPtTEfmF6PGFIvIdERkQkSdE5BdnkzhrrQ/8O3BRxXavFZH7RGRQRE6I\nyIdFJB4t+wcR+esp6fiyiLwzerxRRL4gImdE5ICI/MaU7f4wOtZTIvI3tdIlIm8SkX3R8XxZRDZW\nLLMi8hYReSpK4z+IiMzmeKfsY2e0rdeLyGHg2yJiROTzInIy2vYdIrKnYp3bRORD0ecxHJ2n7dEy\nEy07HZWEPiQiF4nInwHvAX45ulv/1ei9fyQih6L3f0JEWuuka/y114nI0ei8vElErhORh6O0fnDK\n8b1RRPaKyNkovZsrlt0SfU9y0Xo1z5+E1cifirbzKHD1lOV/ICJPR+fjURF5afT6pcCHgRuj4+6L\nXn+piPwk+h4cFpE/nOtnp9RCichHROQLFc//QkRul7CG5RvARpkoYdsY5cefj/KAIeB19fLKWez/\ntijv+qaIjBL+TpIi8jcSXhNOicj/KyLJinVeKmG+Pyhh84lLKpa9R0SOR7+rvSJy0wz7vT36zf73\neN4gIm6Uz7xNRPYBe6PXLxKR70b5zl4ReUXF9npE5KvRfr8PbK9YNr69bdHztIj8bfS7H68tSgB3\nRcvHz/czZEppv4g8W8LrR05E7heR6yqW3SMifywi90bH9E0R6azY56dFpD86b/eLSPdsPiPVZKy1\nDf8DDgIvAL4I/Gn02huBOyre8xqgC3CB3wZOAslo2fuA26LHrwW+V7HeRcAgkAAywBHg9dF2rgT6\ngItqpOsO4I3R4zjwZ8BdFcuvBp4ZbWsb8DjwW9Gya4HjgImedwNjwDrCgPpB4I+i7Z4PPA3cHL33\nPuBXosdZ4Jk10ve8KP1XRcf391PSZ4GvAu3AFuAMcMsMn8W5c1nx2s5oWx8H0kAqOobXAS1AkjAw\n+WHFOrdFabsGiAGfrfiMfha4H2iLtnMRsD5a9qfAJyq282bgScJMsAX4EvDxOukaf+3D0Tm5FcgD\n/wn0AJuAfuCGaBuvAJ4ALog+x/cBd0fLeoER4OXRMfwO4AGvq3Hu/ir6znQQljw+BhysWP6LwIbo\nmF8dbXtdte97xed7cfT+y6Pz+ZJG/171b/X9EeXBNZalo9/g64Abo+/hpmjZTcDRKe9/H1AGfj76\n7qaok1dOWdcCO6e8dhtwFrg+2t54Xvef0W+tFfg68H9H738GcCr61wF+DdhPmNdeDByqyG+2A+fX\nOO7bgBxwQ7TPfxj/jUbHYYFvRmlIEebVxwivQW50zP3ABdE6nwc+E53Py4ATVba3LXr+j8DtUX7h\nAM+O8qCdgJ2SzjdWbKc7SvOrom3+SpSGjmj5PcBTwK4oHXczcc19O/Bf0bE4hHl3ttHfTf2bx++5\n0QmwdlJgd0n0peyhyoVuyjpnCYvsxzOS8aChBRgFtkbP/wz4l+jxK4ku2hXb+UfgvTX2cQdhMDYI\nFKO0Pb9Omn4L+M+K548DL4wevwP4evT4OuDwlHV/n4mA5S7gj4HuGc7bPwN/WfE8S5ihboueW+DZ\nFcs/B/yvGbZ57lxWvDYeLG2ps1539J5M9Pw24KMVy18KPBI9fhHhHe51RIFvxfumBnZ3Am+ueH5x\n9FmYaumqeG1dxWs54BUVz79EWMUO8B3gVyuWudH2zyO8INxTscwQZsavq3EODlNxcQTeRkVgV+X9\njwA/Gz2u+32P3vNh4ANL8RvUv7X9R5gHj0R53fjfmyqWXwcMEAZFr6p4/SaqB3Z3zbC/SXllxeu1\nArt/qXhugAJRHh+9diPwVPT4n5iSpxMGdjcQ3sCdAp4PuDOk8bbKvJDwRjQgDLbGA7HnVCz/ZeC/\np2zjn4H/TRiUeZXHBvwlVQI7wqCqCFxcJU0zBXavB+6dsvwB4DXR43uouAYAvwF8NXr85mj5pY3+\nPurfwv6aqSoWa+0jhCVM/2vqMhF5t4g8HhUvDxL+yKYVE1trh4GvAb8UvfQq4N+ix1uB66Ji5sFo\nO78MrK+TrN+w1rYT3sW8BPi8iFwWpWl3VLR+Mqpy+PMpafokYUkj0b/j7fO2ElZfVKbjPYSleQBv\nAHYDe0XkARF5SY20bSTMaMePfYTw7uy8ivecrHg8Rhj8zdeR8QcS9uL6y6hqcQjYFy2qPP6q+7bW\nfhv4KPAR4JSEnTdaauxz0jFGj+OEwf+0dI2z1p6qeJonzMwrn4+fh63AP1R8Dn2EmfemaN/ntm2t\nDYB6DcU3TElLZbqRsHr4pxX7upAq3+GK918vYRX3GRHJEWbgWjWilsrPW2vbK/7+aXyBtfYHhLUK\nQniDOJNJv8lZ5JVz2d56whK0yt/SVwlL2CH8Tf/elPx1A3CetfYJwhqfPwFOi8hnRKRe/l/5+88R\n3iRurLY82u8NU/b7ymjf6wgDtpr5Q4V1hHnc/jrpqmVqfjm+n9lcEz4BfBf4nIgcE5H/R7Tt4IrU\nVIFd5L3Am6j4IkrYnu53CauyOqJAK0ft9k6fAV4lItcTVhP+d/T6EeDOKZlX1lr71pkSZa0NbNiQ\ndx9hiROEgcleYJe1tpUwOKtM023Ay0TkcmAPYTH3eDoOTElHi7X21mhfT9mwR2gv8BeEwWSG6Y4T\nZibj5ylDWF19bKbjmQ9rw9u6yGsJqzmfRxhk7xxPxiy39XfW2qsIS2kvAt5V462TjpGwSrlEWK1c\nLV1zdQR4w5TPIhVdyE4Ale3tDGHAV8vJyvdHaR1f93zC78tbCXv8thN+d8bPV7Vj+HfgC8Bma20b\n8DFmeX6VWkwi8nbCYOo4YV48rtZvb+rrM+WVM6nc3inCPOCCit9sW/QbgfA3/cdTftNpa+3nAKy1\nt1lrbyCshnWA99fZb+Xvv40wrzteI11HgNurXF/eEaU5oEb+MMX48e2Y4TxUMzW/HN/PjNcEa23J\nWvs+a+0ewqrflxMWfKgVpukCO2vtPsL2WL9R8XILYTH2GcAVkT8ibFdRy9cJv9x/QtjDM4he/yqw\nW0R+RURi0d8zpKLRfz1RoHgR8GhFuoaAERG5kPCiXXksRwmLwT8FfMFam48W3Q8Mi8jviUgqKv26\nRESeEe3nNSLSE6V7MFonYLrPAK8XkSuihrV/DvzAWntwNsezQC2E1QX9hG01/my2K0rYkPra6G5w\nlDATq3Z8EB7ju0RkW1Sq92fAZyo+04X6KPC/x78DItIuIv8jWvZV4AoReZmIxIB3MrmkcKrPAe+J\ntrGFsPp9XJYwUz4T7kbeRFhiN+4UsCnaz7gWYMBaWxCRZzJRCq3UshGR3YRNJF5D2Gbrd0Xkimjx\nKaArCnrqqZtXzoUNO7J9DPg7CTskiIhsEpHxG+5/At4e5e0iIlkR+TkRyYjIHhH5mSi/zEd/9fKS\nn4tKzhOE5+Bua+2JGu/9MnCxiLy64vpyrYhcYK0tE97Y/3GU519CeC5rHd8nouNbH10fbojyhtOA\njW4Uq/lqlIZXStgh49WEN91fq3OMAIjI86LrkCH8rMrUPzeqSTVdYBf5E8KODuO+RdhI9UnCYuUC\nVarfxllri4QdMV4AfLri9WHC0rZfIryzOUlYIpaok5YPS9QDiTBA+wNr7fjYbu8mbAQ/TJiZfLbK\n+p8ELqVimJToh/sS4ArgAGH138cI7wYBbgEejfb5QeCXKoLCyuP8LvCHhKU6Jwjv8Jbr4v9xwnN4\nnDDQvXcO67YTtj0ZJGzbcwKo1fN3/LzeTVgVNMwiDotjrf2PaN//EVURPQTcHC07RViV8gHCz2gL\n8IM6m3sv4bEcJOwt+K8V+3mIsMH3/dF7Lpiyre8QNmo+JSLjVSVvBd4vIsOEJRyzqQJTar6+IpPH\nlPvP6ObrNuAvrLU/tdY+Rfhd/JSIJKy1ewlvvp6Oqh831tj2bPLKufhtwmvB/YS1N98m7BCAtfb7\nhL+djxC2xX6SiSYxCcK2bX2E+X8HYRu4Wm4jDOj6CDs8vLbWG6Oq2pujfZ2Itv9+Jq4vb432d4ow\n//t4nf2+k7CN9oOEbRv/HJDoGvZ+4AfR+b5mShrOELZn/j3Cm+53Ena4OltnX+M2El43hwjz9O9S\ncf1UK4csrBZLzUREnkOYOWxdYJWhUkqpZSLhEFr7rLXva3RalJqLZi2xWxWiovPfBD6mQZ1SSiml\nlpoGdkskarM13hvr7xqcHKWUUkqtAVoVq5RSSim1SmiJnVJKKaXUKjGnwQfjkrBJqg2nppRa6YY5\n22etrTecy5q0kvI9cV2IL92YskHcmfE9XtQH1JQnXrMzr1ZV5TaCWO33LbmoYksHkZzOlBev1k+8\n5a9BHB49vuryvTnlAEkyXCfPX6q0KKUa6Lv287VGwl/TVlK+53b3EmzqnfmN8zS2aeYAd3BHGMWl\nT09cpMuZ+YVE6dP+ucej62tHhzaqezLevHYzs2g0N9GWS9NkTyzeSU+cmTaq15L7zg/eu+ryPa2K\nVUoppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0op\npZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRa\nJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSw\nU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJTSwU0oppZRaJdxGJ0AppZRqGn5A/EA/iaf6ACju7Ka0\nrQscLQdRK4MGdkoppRRAyaftSw/jDBUQLwAgdjyH/+NjDL30EojpJVM1P70FUUoptbIEFin5YO2i\nbjZ9/yGcwfy5oA5AvAAnlyf9wKFF3ZdSS0VvP5RSSq0Iki+T+d4BEk/3g4Ug6TJ29WYKl64DkYVt\n3FqST5xGgunBogSWxJNnGLv+/Mn7sRb3xBDOYJ4gG6e8qQPMAtOh1AJpYKeUUqr5lX3av/AQZrR0\nLvhyxspk7zuIyZcYu27LwrYfWKgoqZvGC8L3OGHgZoYKtHz9UUyhDBasAYxh+JaL8HuyC0uLUgug\nVbFKKaWaXvLJM5h8eVqJmngB6R8fQ4rewnbgGIJMvObiIBOf6EARWFq+9ghmpIh4AeIHmHKAKXq0\nfP1RpLTAtCi1AFpip5RSdYjr4nb1NDoZs2K72yl1Jpds+4X2mcsCSm3hv05xokqynJ7f/kxpYn/p\nB/smtX2rZB2DDOQo7uia344iwzdsovX/HMRM2U/gGoaftYlSa/g88fRZpORRtdLVWuJPnaF48YYF\npUWp+dISO6WUUivADG3XFqFt29hl6xi9egPWEYK4IYgZrCOMXr2B/KXrzr3P7c8jfvUg03gBTv/o\ngtOi1HxpiZ1SStUTcwk2rIwSu3J3irHe2JJtv9A1c/BU7PIBEN8595qXmV/vVbEVpX7P7Kb75DCm\nVCWgspahK1oX4YomFF++lYEXbSS1LwfWkt/Vjk1F5zQ6DGd9nIxrkCppsY5Q6k1QWFjh4YphvPBz\nTp/xG5wSNU4DO6WUUk1v9Ipu2u46TqyvgPEmAsUgZhi4eTM26SCL1LQtyMQYvbz73HOZEr+NXdRJ\n15f2V19ZhLFLe6e/7gdkfniCzI9OYPIeXlea4WdvprijY3ESrVREq2KVUkotvsASPzZM/PAQUq7T\n23SWbMxw/O2XMnT9eoKEgwVKPUnOvHIHQzdtXHh655KWuMPpV19IEDMEbliqaB0hcA2DLzyfoDUx\neYXA0vW5x2i55wjuUAlTDoifHKHjv54g/cPjy5p2tfppiZ1SSqlFlf1RP72fORi2QxMBaxl40WaG\nb1xYAGaTDgMv3cbAS7eFgxNXGbvOPZMndiaP3xantDGz8PHtaijsaufYu64i+8Ap4idH8TqTDF+7\nHhOb3nklsf8ssRMj0zplGC+g7c7D5C/txSb0cqwWh36TlFJKLZrUkznWfeppzJRSus5vHcEmHUae\nsa7GmnM0JWAzI2XWf+IJ4kdHsY4g1uK1JTj12gvxelKLs88p/NY4uedvnvRafGj6+9KPnJ52PsZZ\nA4kDgxQu7K66XKm50qpYpZRSi6brK0erBjGmHNDxrSOLPg0YANay4R8fI3F4OOyVWvQxpYBYX56N\nH304nH5snuLHR+n4+gG6vriP9EN99QcxrqHWMC1h2mdYrtQcaYmdUkqpRZM4UnuoD2fMw4x6BNnF\n7bmbPDBMrL8wrZODWJByQOanfXMvKbSWri8dIPvg6XAQYguZh/rwvx3j5K9fit9SezDjqQq7Ookf\nGaoa8EpgKW5pm3Pa3L48WIvXndZpzNQkGtgppZRaNDZmwKtdQmbji19RlDgyAn71kkBTCkgeGJpz\nYJd+ZIDsg5OrUE0pQLwi3Z97ilNvuHjW28pf3EP2vqOIX5oUfAauIb+ne3pnizoS+wZo/+b+sBRS\nAGPIPW8b+Wo9cdWapFWxSimlFs3Qdd0EzvQSJCuQ39GGjTtV1loYP+2em8N12n4N+PMoIWy7+1iN\nEjZIHB7CGSrNels25tD3K5dR3N4R9p6NhYMfj16zgdwtO2a9nfjhHB1fehJntBxOYVYKMAWPtm8/\nTXJv36y3o1Y3LbFTSim1aAZu3UTmkUHcwfK5XqCBK9i4Q//Lty/JPkcv6aT7iweqLrPGMHLN3Euz\n3ME6gZtjcIaK+K2zr44NsnEG/scepOhhCh5+tmLu2VlqvfPQtJ61EPaubb3jEIULupasF7BaObTE\nTiml1KIJ0i5Hfu8Shm7aTKknTbkrydCNGzn6rsvxlmgeW5tyOfM/zw9LwqKrmiUcvDh300bKvXOf\nrLbcW6cnrR/gdczvWGzCxW9LzjmoA4idGKm5zBkuIUWd/UFpiZ1SSqkqxnqnl/zkZ1HwFespAHDm\nZRvgZRvCFzPjU0KU55SGvExUofo9ddYVy9DNbRQuvJD2b50icTRPuTvB4At7KexuAWZfbTqu/+fX\nsfFD06cwC1xh9PJ2Ctuh2vH4yaW7rNpY9WnMxuU3gI0tQa/jeqJp3wJn4cftx+YegKvpNLBTSim1\nKpQ2pTn9hsWp7s1f1ErfyzfS/cVjYScF34JjKG5OcfJ1S1OlPJORK7toeeDMtN6/VmBsVxs2FrZf\nlKJP253Haf3BaUwpoLC9hYEXbaK0KduAVKvlpoGdUkopVUXu5vWMPLOLzIODmKJPfmcLhe1LN5vF\nTAZu2UL68UHMqIeJegFbRwjihr5f2AaEQd15H3yYWP/EnLrpx86SejLHyddfQP6C9oakXS0fDeyU\nUkqpGvy2GLmbojroZa7lnJaW1jhH3n057XeeIPujPrCW0Us6GXzeRvy2cMiU1u+fIjZQPBfUwcR4\nfr2f3c+hP7xKO1ischrYKaWUWlaxUwVa7zhDrK9IcUuaoef04Lct7qDFq1WQiTFw6xYGbt1SdXnL\n/bWnLzN5j/iJsXAOXbVqaWCnlFJq2bTeeYaeTx9GAov4kHkoR+fXT3L8N3eSv7C10clb8WoFdQBW\nBKmzXK0OOtyJUkqpZeGeLtLz6cOYchjUAZiyxRQDNvz9fg06FsHonnZsjTGgxVpKG7Xn6WqnJXZK\nKaUm83xSjw4QP5LDT7kULu7F61549V3b3WcgqNFQzVoyPxlk5BmdC97PmuXbcEq3KsPZBTHD2eef\nd67nrFq9NLBTSil1jgwXyHz7EfB9TDnACmQeOsXolRvIv2LrgrbtniliaoyhK8Ys+dQAACAASURB\nVJ7FHZzbOHdqsnWfepL03kEqu0ZYACMM3LyJ3E0bG5QytZy0KlYppdQ56bueQArlc221xIJ4Aekf\nnyC5b3BB2y5uyxDEa/TIdIXixjqzPai64kdGSO8dnNbGTgjny81f0K69YdcIDeyUUkoBYHJjmOEC\n1S7/4gW03X18QdsfenY31kzfuhXwsi75PS0L2v5alnlkAKkyjyyA+JbMIwPLnCLVKBrYKaWUAkDG\nSlUDLwhLftyzxQVtP8i6HH/XbvyMg580BDEhSBjK3QmO/e4FUGPfapbqjbPX4DH41PLRNnZKKaUA\nCFqSSFC91McKlDYsvANFYWeWp//2cjIP53DPliltSJK/sEWrCRdo7KIO2u86UXUuWesaRi/uaECq\nVCNoYKeUUnVY11DuXhltvwodMYrtCwiQ2lMkNrQSOz6ETOm9al3DwK3r8dpq9H6IrG8dBeBUcaL3\nZSI9vVOE/9z0uc6bSUpVt1UoT1QqpVvzNffpmDCtZW9penwG0UT3syn1KjYoPh27NMnY7hbSTwxh\nyhMJDWLC2EUt5C9OAt6i77dUCAeWNouwaQm0EnEx6FlUSil1ztDNu/G60gQxE85DGjMErmHgZ8+n\nuEVnLGhaIhx/x07OvmAdftJgDfgph7M3r+f4W3c2OnVqGWmJ3QrgZLPnHo+88OKa7/OSy3er6BQn\n37oGcaH9vmOUN3dNe2/siaO1x66aws8NLUr6xlkb7ldqVPNYT4dXUPX5cWFkY7zRyZiVYrtQ6F7o\nVmKMvf0yZCRH8sAIQdpl5IoOgrRLqnd0xrUv6wo7WDwsE7/57tTM61XztDORn+zuPlPzfa6E1Y8F\nf2kuaZ4Ny0DOldzVcTLZ2A4g/lvb6f/1NihZiAsYoZ35nf/ZGCQ8Xusu/Nz7tXpMqznRwE6tarUC\nOqVUHSIUdrRQ2KG9VFckI7CMN/qquWhgp5RSa5zbn8eMeZS7U9iUXhbWAudAgdR3c5hBj9IlaYo3\ntWFT2jprNdBfsFqzxqtplVqr3DNjdP/Hk7j9BXAE/IDRy3s5e+v2RidtxepMjzU6CTOSfxnEfGEY\nyhYJIPHACC3/dgb/g+tgc2xO2xrJJAHwCgsPCk1JpztbDBqeqzXJWgv+4vcQU2qlMKNl1n3sYWKn\nxjBegCn6GM+SeegMnV/a1+jkqSUiPylgvjiMFMOgDkAKFoYDnPeeAb3hXfE0sFNrlGZeam3LPnAS\n8YJps0yYckDmsf4FD0asmpN8cRgK0/M/scBpH57WDmUrnQZ2am3SuE6tccn9gxiv+g/BOobkgaXr\nSakaR056VaeMA8AB6as/TqFqfhrYqbVJe8uqNUryHrHjI1i3fvYfJPXysBrZnXFsrY+2DHazNr1f\n6fQTXAYmkWx0EtQUIqKFdmpNkbJPx1eeJv1oHzgGKQdYiargqhjb3bq8CaziivXHzj2+vm1/zfeZ\n6CDGgqUZb7Bsw0b9Qc2IaEIhmNz54FSx8eexUul1AcfvAluYssCF1GXChgtzc9pewQuP91Sw8Jvl\nEitjvMhmp7dkau1y9L5GrR3d//4E6Uf7MJ7FFP1wyjA7uVWCFQhihv5f2AUzlOiplSm+09D9hzEk\nAZIG4iApiO8Set+faHTy1CLQK5tas7TUTq0VsVOjJA4OTWtTJ0DgCF42hgDFTS0M3Xge5Q1ZQNta\nrVbZF8ZIP8tl7C4ff8iSuMiQuMTogO6rhAZ2a4CI5bJtJ+ltH+XAyQ72nZg+7ZdSavVKHBqmVo8h\n41vym1ro/8ULljdRyyAhPpvdAgIc9pIUrY6TNs5khOyLNQRYjfRTXeV2bOjnA2/4Bi2pEhC2RTl4\nup3f+ecXMzCcbnDqlFLLwcZNOM1UleDOCgSJ1RfwXJs4y3Wp3Llpqh2BBwqt3FfogNr9QlUFG1gK\nDwSUjwS4Gwyp6wzi6rlrdhrYrRJdrWM4WZ+TZ7PnGvhmkkX+4W1fJpssYSqay+za2M+H3vJVXvOB\n/4lmcEqtfmMXdNLxleqdD6xrGLuid5lTtDgMlqzx8YBCRWncnvgw16VyxMROyuKuTg4xHLg8Umqu\nDg3NqHQo4OQ7igTDNqyVd0CSsP5DSRK7tf1lM9PAbgbi1LiTlZm/2H2vfwYA573m6QWl4Yr2o+ce\n/07XhyYtMwgJiWEqcq+i9fDwieEQF3dauwnXsWxfN8B3P/D3BIvYymyfBxud6bM53FtYR9nO7qv2\nF0/evGjpmY3eXzo27TV/ZGRZ06DUUrMpl8Gbt9H+7UNIeWJQ4iBmGNvTRXFLS0PTN3eWKxPDXJUc\nRggDvDN+nG+M9jIYxLg+ORgGdVPExXJ9alADuxlYz3LyLQX8ASYV8toxOPnWApu/msKktFCgWWnY\nvYIJQkriOBI2eh3/S4iLi4OJntfi6Mev1Joxcu0GzvzyHgrnt+FlYxQ3ZBh46Q4GfmHnihvX8drE\nENckh0mIJS4WV2CdU+LVLcfJiEebqT1dYEZ8HIJlTO3KM3aXT5CnarNM68Hod7RjTTPTErsVLC7V\nSxNFhAQuZXystTWDO6t9QpVaU4rb2zizva3RyViQGAFXJIeZ2tTLCLgEXJ7IUUaI18jfAgRfm6DU\nVXraYvPVl9k8FJ8MWGllvGuJFtmsYA5O3RI5z9a/K/X0rlUptcL0uqWagZkrsCOW59FilmqzpXkW\nHitl0LbF9bndgtQaVz8B7ropr1mLe7qI21cCqwUGjaYldiuUQWbMmiyWkvWIRx+ziGCjH13Bzm2i\nZxeHmDgIgk9A2XqL2j5PKaVm4hKwyS3i1Ml7PAzfy3eyOVag1XjEo7Z2JSuMBA535ztnvb82U+ba\n5CBb3QIe8FChjQeL7ZRW+bApmRc49P9N9WUCtNw6MbuGc88IiY/0kR4OwILXFafv1zaTv1jL9BpF\nA7sVKCXxc50lalW1BlgsljI+vg3CoMwKAZay9edUDZuUGA4Tg1eKNbgSp2DL+Frqp5RaBhudIrdm\n+wA7rRp2XNkKjxSzlDD829B57I6PsCcedobaW8ryRCmDP8uKqh6nyCtbTuBiw5FigBvSA1yWHOLj\nuS2rOrgz2XAWitO/X8T6QBmIhX0Gu/8whtMVnpCxuz2SHziNFCeuJ/GTRdb/1X6Ov2cXxV2ZxhzA\nGqeB3QqTEDcsravVbi4qkStWlMgFWIq2dmPielzMpKAOOPc4SYxRW5zXdpVSarbS4nFrtu9c6Vs1\nnoWzfozHSlkAfITHSy08XppLyZEljqWMcHN6+v5iYmk1Htclz3J3vrvuduTc/1em9A0Omz6fZPi/\nPEr7AmJbDS0vd4ltDANjay39f1eeFNSNMyVL12ePc/wPdi13shUa2K0ocfFx67Srs9biE1BaQDVp\nteFT6rXjM0jVfaUkRlpiOGK4NT3KU6UEj5cTKzqjU0o1xvWZU4ilatM4a6FohYdKWb6f75x1idyU\nrXBNIsczkjniEmARTI081BXLZYmhqoFdiynzksxJ9iSGcbCc8JJ8bnA7e4vt80hT47m9ho43x6su\nC0bAO1H7OpN8cnSpkqVmoIHdCtJSpwv/uLm2natkouFT5jJfoDAxmn0MSIlLAhdXwuFWAOICF8SL\ndDsedxW04bJSyyF2apTk07lwAOILOwlaql+gV4KNsTFipnYQ8XApy4+KrXjz7A/4gnQfe+KjFWPf\n2bp9AGJS2QTFstXNs8kd49rUWWJicWQ83QXe1vUE/zywk58WVtdUjuJCvZY41tF8vlE0sJuB013j\nx9iSnXHdF7ztXgDe1nXPgtLQYibacojUngZMRMjW7Mq0+ETCQHDcRbGJUd6ndu1wBXpdj1dkBzkb\n1OhHD3Tu+cKSpLWWv9rxi9NffGjv3DYyQ+9jpZaVF9D92SdIPp0DLIjQ8Y0D5J59HkPP29Lo1M3L\nCS9F2UrVQYfLCMPB/C9lrabMRfFR3CnbrnV/G1g4XE4BkDVlXt16lKx4xCSsj5i6XtwEvKrjAA+d\n6FxVNRa2QK3ph7HAyLUre1idlUyHO1GLRkQY/6/me1ZRxgZo137VdNq/dZDk04MYL8B4FlMOEN/S\neu9xUo/1Nzp583Lv6NTxNSZ7ujz/G9qtbn5OXcA8hHvy3YDllS3HaDdlEibsYFErGEyJT69bmHca\nm9HwV8p1i4byF2mv2EbREjullKrDCnjJlXFD4huf7I9PY6oM4mbKAa13H2HkqlkO95GcPrtAOlma\ncbXOeNi2qjUxEci0x2uX0tfTHptY795CihuSY0BYA+DZsGTonnyaDjfs+ToWJOa8j6xTROq0SfZt\n2BFDsOStw91jreyID/LyxCgdxpv1pB3tsTHKZvWU7ucOAOUaB+9AolCilJhbp718UkOSxaBncQZ+\nX4073P6zM677nf/vWQDs/+Weactu2/6NWafBrJJSrvHSui5Tuwv8z6Tm30ZwPr71icnVro++qJNA\nq1ZVBRNAYrjJvxN+gHuoj5YDpxCvdlpjfUXc0dnlJ/7w9MvDUDo143rH8mFHgTOjE81VAhvuc0dL\n36z2PS7lTASSIzbg9nyC81yftFhGAuG47+DjkzHzzzcGAluz6qps4f5iktHAxbcwZB2emxym2/Fq\nDrlSdTsIJQKuT53m8qiDxr5ShgcL7YzNch7tpXRX3845rxP0DkFsOBwKZeqymCGfaKFwpnbToWri\nA6t3CJnl1PhvlJpGCGeVGA/oAiyBjhenlKqm7JO6/VHMcAHx6+cTfjZWd3ljWa5N9XNzyzG6nBJD\nQYzvjXXx/Xz3ubZpHsIhb3EvWwVr2FeOsSNWnhSseRZygcNJ3yWIxqzb6pbnHtRZ4fbRbl7depR1\nbvHcECpdTolrkoP869Bm+v25lzQ2mrw4g/30cPWFRhi5bGX2BF4NtI1dkxEgxsRYdSKCQXDROxml\n1HTxx45hhvIzBnVBzDD8zA3LlKq5e0XrYV7VfoANsQJxE9DtFnlx9gS/1HZoyff9SDnBT0oJRgOJ\nhk+Bp8px7i6kqOzFvzNWnFVQZ23YyWLQd/iv4Q2kTcD6iqAOwjHx4hLwc5mTS3BES096HPj9DoK4\nEMSiQoi4wU86HHvHheBqeNEoWmLXZMYDuKkDAltrNbhTSk0Te/o0EtRuI2YBGzMUdnYwevX65UvY\nHHQ6RW7Mnpo2IHDcWC5L5HDaDnDSWop2qZqlCIe8OIe8+kPCVOuVW8mP2v39qJDhiXIaH+FEOcub\nMweqrmsEetwSLabMcNDMpanVmeemObJxPdk7zyKHA0rnpRm+tosgpaFFI+nZn4Fpa636uqRmbmuy\n+zVPAPDeTV+Ztiwhc5tqZS5jy60mYZNl7XmqGidwIN/ZvKUPGW96J4dxVmD46h6Grl9HcVs2Knya\n5e+pY3pHid72GlVvFS5qOQ7AiDdRvbghlQuXpY9XXeeyxHDNAYFF4OJEjt0IdxcSFJYsuJvZaS9G\nNlak2hBtQVTK92gpO234laTULk31bbh8/My2mjIXx4fIGp9jXpInStl5Drq8PIL2GEMv66XUP/M1\nUS0PDexmUCuA2/+39aaTCf1428cASM0xiFvLTJUMbCmHSPmjnvsmPb/uze9ky1+NLGibQXF1DWuw\n1lkHSk3cXKjckyZ+svoo/37G4dRbtoRFQ8yth2J7+9i017a1Dsy43iXJowCczU7ke1sSYSe0y5NH\n6TKQFMhb6A/njWejU79dkIgQs5brEiMMBNN/X+OB1liwNHmFPz43NwZIMnWQ9cBaBgMPX8a4MDE4\nadnxWCujNiBtqw+H4ohle+IoWxE2Oz67YjZ6Ha6wcAvwQNEhv4QBbV/bzOOy1lKM2jxWD9nnpmSW\nbxzW1ax5bwOUUkrNaOi5WwiqtGcK4oaBl2zg3Az2DbbeKXJdAnbGYKsLu2JwXQIyAoPBzOWIIkJa\nGlsWUcZyyC9SsgG+tfjWEljL2cDjRFC7Z+6Bsqna/c2zcMQTAoQWseyKhbNWjAeqroQz+lwZ95l1\nSata8zSwU0qpFay4o4Pci7aHDdcTDn7KEMSEszevI/fC+gP7LpcYAS/J9uFKGKxI9G9M4NI4jFnI\nBWG1ZD3NEKLmbcBTXoEDXoHDfpEnvDwn6wR1AENWeLhkKNswmCvb8FiPe8I+L7wMb3GDqsdnBBIC\nLc1w8GpF0KpYpZRa4cauWMfYJT2YgRxBpkxhV5Yg3TzZ++74WM1BgAXoNvBYGba7sNGxVdsUW2sp\n2NrtCZdbETunmWfOBIY7C0KbAQfLUCCUK0K5tNiahasWSIlluIHtC9XK0Ty/fKWUUnNiRko4o2W8\n1gQ25ZLf3Y7fvbyDfM9GhykTqz1JASkJg5enPegwJVLEpwV3FsgFxaVO6pKyCIMBgBDHstv1We+G\no/SVoyFSqgV3AoxpUKdmSQM7pZRaYcxIiY4vP0XiyBDWMYgfkL+wi1OvOr/RSasqF7iULTWDu04H\nTvjhJAYeAXlbIkkMIxOthc74eUqrZKD2OJZnJn1iTARytQY7CWxYVT2igZ2aJW1jp5RSK4kf0PPJ\nh0kcziG+xZR8xLck9/az7uN7Z16/AZ4sp6nVQk4k7EBxRWLiguQRMGInl84VaZ5q2IXaEQsmBXUw\n0WPWRu3wxv8tWPhJUccwVbOngZ1SSq0gqb39mLEyU4dGM74leXiE+OHpw5Q0WtE6fGu0MxzAt0qz\nNCNh9VHvGolf1jvV29NJVCW9vyzs9wwPlwzfKzoUm6LbiFoptCpWKaVWkMT+QUy5RpWktaT3DlPa\nMrfJ15fDYS/F/jLsiFF1Dh1Xwk4Ua8FMYdoJ3+ABFkuvsWx2feICuUA45BlGtVpW1aGBnVJKrSA2\n4WAFqs5uJUKQbN7oKAxWaguoHvStNoMBdJrqAxZ7wHUJn6RMnKvx0r20WNY7Pj8tGfqD5v2cVWPp\nN0MppVaQsUt7sLUmWLeW0Suac5qMzW6BDU7twM2zcHr1NKOra1/ZqdoNJLBRL+Eo6DMyuR2eiQYv\nvjQe1Bw+RikN7JRSagUpb2whv6ebIDY5+w5ihoFbtuC3Nt9k8s9I5Lgl00+HU72UyrcwaqFvdXR6\nndGQFX5cMoxGgzJ7ForRsVebh7aaDqOBnapOq2KVUmqFGfzZHRS3t5H9wXGc4RJeV4rhGzYxdGUb\n4aAhzaPTKXJlchi3SsBibZja4x4cXSOldePOBoZ7i0JSwjZ3WQm4JG5nVdoi1B46ppmYYQ/3RJGg\nzcVbl2h0ctYMDeyUUmqlESF/cQ/5i3umLGi+UpzLEoM1OwsEwFEPjq2xoG6CUIg+sqwIs/38hLAj\nRbOSUkDnPx4l8/0cNiZQtpQ3J+n7rS146zXAW2paFauUUmpJOARkTLlm9aJh9lWPq93ZQGY1qImN\n5tUtNHHP2O6/OUT6+zmkbDFjAaZsiR/Is/4P9iFjazaKXzZaYrfE3DXRx0sppSakxPLatgNckBhC\nCIORqm3rgKE10q5uJh7CAU/Y5tqq1dbjLLCv3LxlMu7xIsmHRzDlyaWPYkGKAdk7Bhi+dWpJs1pM\nGtgtoRgOjgZ2Sq1oEkBspNGpmJ3AFYLE4mTrw6nktNeOp1pnXO9EexevbjlBQoqTSuOmBne+heHA\n4cFCJ5Uju7WYw+cet5vhc48PerX37URVmKM2PmP65sO3YSAVzKOS61i5Y9bv3V+yXBAb41nJHHGx\nVYPhkhV+XFyPXcRBi4+OzT6NU/WPZgAojYTnPvHQUNSVd3q1sila4j8ao/Sc6p+TM6zXy8XQvGH/\nCndeYgQHZ9pE1koptZpdnhgiLsG0KlaRMLgrRb1AT3hxvjnWwczD9a4lwhPlDJ8eWc+oNfhTYqOy\nFe4ptC1qULfYbDIcZ7HqMiBIa/C21LTEbok8q/1Eo5OglFLLblcsX7Mq0QPuL7RwzIuTt3qBr6Vo\nDV8Y6eW6ZI6dsTwOMBC4HCinuDwxynNTOQrW8HAxwyOlLEETBXqjl7ZOm+5unE0Yhm7sXt4ErUEa\n2M0g6K4+2Odztu2vu97O9JmlSI5aZGkzuUrgRb9wP09+dsuCthnsPxg+sJbavdykeqOjRmq+DpVN\nIXAhv4KuRTa+OB9kLDa9kXs2VppxvZkGzi3i45oSLTWWtxjv3OM2MxH8bXaHq719krFgaYLF0gKa\n1JiqQxHPzn4P9ntJwHK+63FFYmLYmJj4XJscYld8mPuLcRZS8tmTnPnc1nIiHX6SY/koL03BmTdu\noudjR5CyPTdDSpAwjF7dSunKFI54Vbfla2neotDAbok8Uph/mwW1SozP6D3tQteEQZ1Si2R/OcFl\nZqxqqV0ADOpUWHMWA3bH/GnV266EAxX3mIAzSxTUzsfojZ14GxK0fek0iYNj+O0xci/uYfSZ7Zr3\nLQMN7JZILogTYDEWbWe3lomgbYjUWrK3lOaCeAEIJgV3YaP/hZUsrVW9TlCzHNQBNro+Z0rNE9gB\nFHdmOP3b2xudjDVJA7uZPHWw6ssP3HbFjKsee/e3yIhLp5noXRYQEGCJiZ76ZjASFCc9v/0z13Le\nkR8tbKNWx29YTYwHqdONTsXsySK1XSvY9LTXnmLmYSruSO3mwZESz0n08aL2I8SNz4FiK//at5th\nZ+Zq4lxmYr+dzkR35AE/O+O6o8HSDH5bXsA5PVla+Ny9z0j2syd2BFemnz8RGPBauHP4/Hlv/6f9\n58173eOno+MbWFiP5PixUTp/cAwJLPndnZQ2ZbV0b540ulhio9ajFR2QUSm1uKTokXiqDydXwO9I\nUdzZjY03R6nNcBDn/Ueu4f1HrgGgJx0GaDtbtO3xfOwrZcMRRKooBoZHiwsPHhsmsKy7bR+Zh84i\nXgAWWr5/guKmFs68eg/EtOp+rjSwW2aznzRGKaWqix3N0fa1x8GC8QIC15C9+yCDL90DPTOPNbe8\nLJtiIySMT4yAso6yNWdngwQPFdq5NDFI3ExcQTwLZYQup8ieeI69pdamHgqlmra7TpJ56CymPFHT\nIeWAxJEh2m8/xOAtWp07VxrYLRMHw+wmjFFKqdqk6NH2tccnXQiNFz5u/8rjjF54TdOU3F2a7uOP\ntj5ATyyPbw2uBNw+so7vjG5A29rNzX8MbSWXjfGsVB8CuBIOchLD5wWZk5SsIW8dPjqwi7NLVCU9\nJ4El/dggmUfOYh1h5KouCue3TKtebf8/JyZ9l8cZz5J98BSDL9ym887NkQZ2y2A8qNNOFEqphUo+\ncaZ2sb+1pB/rZ/SK3mVNUzVbEsN8aMfdpJzxpijhxfv52VOARMGdmq0A4Zsj5/GdkQ1scsd4U+dT\nxGUi5klKQMwGvLFjPx/o30MjA2cpeJz3oceI9xUwxQAr0Hr/GcZ2t3Ly1y6YFKi5w7WH0JHAYooe\nQTq2HMleNTSwm4Hp7qz6un3+2RnX7XHiCIIRrXpoVimZnGFkn38a86mFVWUFpwoLWl81l8CFQhNN\nbZks5s+V0E0l5QCKeYrdC2/XG+/JT3tte3f/jOs9szUc4/OCjoC4mZ6OhAl4YctxRs0YfpXg44rE\n0XOPOyuqHQeCmXuwjNqlCQCCWlMpzMKx+OK3f7s4Vqw6sp4j0OEUeUnrXgbmMPzJUHn69HGzNX5u\nTpmJfLPnHw8RP5XHeOHnJxakFJB+coj2Hx0l9+KJGw+vPUZsoFx129YV8ucJONpOfS404lhiRk+x\nUmoB3NNjtH33EB1ffZrUY/14nUmCGg3KbdxQ7m6CajhgTyJXswbNAq1Vgj41O61m+pRt4yyQNY3r\nmS+lgOx9Z88FdZVMKaDtG5MD9MGX9BIkph9MEBNyN/RqNew8aIndEptpFHallKrKWtq/fZDs/aeQ\nIEACyPz0NH46Rs26WBFGrqxey7DcYrXmlSIsUfAWUAq21o1YQ48NavaUHWvgINBmtH7A7gxNnnVi\n6IU9JPePkXlgEPEsBOENSn57lv6f27yUSV21NLBbYo6eYqXUPKSePEv2gVOTql1NKUC8IqXeDLGB\nfNgrthwQxA2IcPpXLmqKjhPrnRKpOiVyeSsMW63NmK8D5Rjb3OnTclkbBsx9DQzs/BaXMOKsfvPh\ndU8Z784Ip9+2jfjhPOl7hsG3jF7SQXFrRsexmyeNOpaQ9oNVSs1Xyz3HqvYWlABifWOcfOsVJA4N\n4Q4U8LpTjF3cFQV1ja/ivCg+hqlxYbcW9haTxLGEzeY1l5yrEWt4qBTnsnh4Bh0Jhz7xgXsLSRp6\nTl0h94Ju2r5zBlOa/B0IEoazL1tfdbXSlhSFF7ctRwpXPQ3slsiWxJC2r1NKzZs7WKy90DGYvMfo\n1euWL0Gz4BCwzinRYbya1YQWuDqZ5yry5K3hJ8UkR/3maBe4khzyYpz2HTa5HimxDAaGY55btUPK\nchv4xY3EThVJPzQUdoY2gIXc87sZebbOo77UNLBbZL2xMf565z3sSuVwq/ZbUkqpmZW7U7hD1YeC\nED/A62iuYOg5Lcd5e++jxI2Pg8Xa6jVpwsTrWQl4ZnKMB4pwyGuu41kJ8tbwVHlhU3ktCVc49c7z\niR3Nk354GOsIo1e34Xc1YVpXIQ3sFpFDwMf33E5vbIyYsWgVg1JqvoZuPI/EkeFp1bGBIxR2dhBk\nm+cieVmqn99c9zDJGXpjVgv2XIEr43kOe/EVN2uCqq+8KUVuU6rRyVhztK5wET2n/TjtbjEK6pRS\nav6K57eT+5nNWFcIXMEKBHFDeV2a/p/f2ejkTfKa7idrBnXWQtlKzRI8AEcs2Tq9aJVSs6cldjOw\nA4NVX/fv3TLttcuf/TgZZ3pPJQBrLTZqTKyzUDSPop38eQ3e10vb0LEFbdOkVugd6lijE9CkBIIG\nDXyfu+k8Rq7qIf1oP6boU9jaSnHb9GmZKtnY4txYxtzpnTDSbvWBZHcmhmpux7fw/UIHVyUGyTjV\ngzcDZJ08jhXaK26MW8zEiS9Te4aCcU4w83vmw19Aft1imn/A8tbY/NOYjL4TW9fNPHj1TE4ktPPE\nYtDAbgaSSVd9fXTP9AzkdItLyTfEq2VeGsc1pYRM/gmUL8wjqfmPwr6iLbvEawAAIABJREFUaWBX\nlRUI3MaVwgedMXI3Tu1JWDs94i5OyVe1wC7pVA/sitYhQY39irC3lCRr0lxmRqqON1uyYCiTEUjL\nRNvklEy0u0vLLHr7LtGgx76d/+efMXU6wSyStMBGx5ARoYjlpG8ZDGaf5pSZf0CcrDLsynxV+86p\nudOq2EX05ZM7CDSCU0qtMXeOrKcUTM/7fAtPlzKUMfy42IIPTI03fAv7Fi82WHO6jHBxzKHTCCkj\ntBvDLtew1dHL+1qlJXYzcar3bHXi0+8sTgVpPnjwCv6vbT8haXyMhFWwIsL4f6q5xeIeonP7KjUn\nX85t4apUP91ugUTU1q5shaI1/OfQZlpiwxSsw4+KsNWFHicsVRgK4KAHw9oseV4McL5rcKZUFTsi\n9DrQF8Conts1RwO7RfaxI5fyk6FeXr/pUZ7ffZAAS0yHPVFKrWIF6/JHJ6/ixsxJfr59P0bgB2O9\n3DfWzZh1aYneVyYsndMSusXRUWuwQMLWPz3GMOprp5S1RgO7JfDD3Dp+mFvHj5/zUQBiooGdUmp1\nK1mH20fOoyd1FIC7R6vPMKAWT70LuBEhJlpctxZpYKdUA1jfJygUsJ6HuC4mmURqVPsrpVQ1I3U6\ndfjWkptDB4rlYH2LvbeA/XYeyhZ5bgp5XgpJaDOlxaSBnVLLzC8UsKOj555bz8MvFDDZLCaho++r\n+qTok3lkAGekTGljmvyONmrO39Uk2kyZPfExNrlhe7rTPmht7MII0GXCiSvH23KPC6wlAPqaKLCz\nJYv/2/2wrwz5MF32JyX412Gcj3Qj7Xpju1g0sFNqGdkgmBTUVQpGRpBYDDHaeUNVl35sgN5/ewoE\nxLdYR/Ba4pz49Yvw25vzpuBZqTPc2nIKIZysvsuGHSgeLsFI88QdK84FrqHFTB4T1UYleHkLT3l+\nrQFoGiL49xF4osSk4QgLYZQf/F0O532dDUvbaqNXEKWWUVCoPxCoLS3NAKtq5XMHCvT+21OYcoAp\nBYhvMaUA92yB9f/yeDjFQ5NZ5+S5peUErnBu/DpHwmnELm6eGdFWnIxAi5FpvWFFhAA44PkUmuzr\nYP9rlKpjTPtgv1fAFpopDF3ZNLBTajnNdPENNHNT1bXeexKpUrVmAogNFEkcrV4S3EjXpftxagym\nbIB2vQLNS7uRmhdvA2x1m7Bac7hO3mbQcVkWkf6slFpG4s7Q+kE7UKga4sdGEb/2xS92Jr+MqZmd\nLqdUdaYJCNuIaZv5xSciZAQ21DrxjbK5Tt7nCrRpOLJY9EwqtYwkHq89z6dIuFypKryuJEGtHFvA\na2u+786xcoqyrR1gjGkB9bwMBLZu+zkjwsYmm3nCvL4FklW+CwmQV2YRt8kC0RWsuT55pVY5EcFp\nbYWpHSSMwWlrm9QQWqlKQzeshyoXawsESZfC9tblT9QMvp/vqtr6ILBhu3mdcaI6B+g0QpcRYlWW\n520Y3Nk6TTuc6K9ZmBtTyK9mIU44uW1KIA7yojTmNdlGJ29V0V6xSi0zcV2c9nas54Hvh+PXua4G\ndaqu0oYMAy/eQufXDyE27BUbxA3WEU7+2oVNOeTJUBDnX3PbeW37AQAcLBYoWXhE+wlVtd4RNjvm\nXMtEA5zyLYemzCCx3wvojDt1g7dmKxB1XtWCfUkGe38RfItcmUB6min8XB00sFOqAUQEicUgVu1+\nXKnqhp69gbE9HbQ8cBonV6K4JcvIlT3YZPNeHPeVWvh4biPbY3l2xAcYtZBrtoijSXSYMKib2tu1\n14ESwokpbSxP+JaNTlj1Wimwlv7A1ui20ljSYpDnpxqdjFVNAzullFpBvK4kZ2/Z0uhkzImPsK+c\nJuUMnHstKeEFKG/Bb1zSmsqmKkEdgCPCeY7hhO+TEeg0BgFygWXMQCp6D4QzTpQsHPQ0el6rNLBT\nSim1bNICF8TCJlaWsHfsSR8OeDRlCdNyStWpTTfALtdMGupknSOMWcvTXkCXEQToD5q3tE4tDw3s\nlFJqBUkcHKb1vpO4uRKFrS0MPWsdfltzzjoxVQy4PB426q8smFrvhK89tcbnGfOp3aNRCMevqyzR\nc4A00CLwpJbQqYgGdkoptUJ0fO0QbfeeRLwAsZA4PEzb905w8g17mrJX7FQbnDBAmVrb6EjYjuzg\nGg/sTvkBG6pUxwY2LIGrVU3b48AhX0s8VUiHO1FKqRUgcWiYtntPYsphUAdgvHBasXWffALqDF7c\nLDocag5YHAAta/yKdNy3FKzFrxjGZLzNXL0+z4KW0qgJa/xnpJRSK0PrfWFJXTXiB6T255Y5RXM3\nU+y5VjtRCLDVMVwdd0hK2FauGFgGg4CDXsBDZZ9SnXNngTVe2KkqaGC3ZJr/7lkptXI4udK5krqp\nrAVnpLy8CapBsJga+d9Jv3ZwZ4GhWTYTczGkJEZSYjh1y7KanwH2uIZ1Tth+zhHBiBCLeg2fiWaZ\nOO4Hk0ryxvnWcsrXzhJqgpbeLjJHAt6y5SFev+kxMrIyGjSrCYXRONZfq+UGqpkVt7aQPDSM8aZf\nwsVCcWOmAamasM4d4zUd+7ksVUSAK+N7+frwRp4sTbT96wtgXRBOCzpeJRvYMKg7WK5f3TiuRRI4\nFWUSSXEp4TFmmyOwnS0BtjhhQBe2O5x89EaEFNAmQs5aTgeWjG/pHm+nSFh9PRRYjvjacUJN0MBu\nkf3Nnju5qesoacdndtmUWi2stRAEYIzOIqEW3dD162m75wRTawOsIxTPy1Ben25MwoAup8Afr/8x\nKeOdmwBjvVvkNe0H+ffcFkYq3vtoGdY5sNEJL0ACxAS2x+B8wMOlaKtXLKYlhsP031fcungEjK2g\nytyd40OX1MkrDGFP2FxUzHnADzgRhAMZj49jN9rgojprLZyJAssezfuagQZ2i2h35iw/03WUlLNy\nMhe1cDYICMbGwKu4GCUSmGSyaiZnrQXPIygWw0DQccL3Os07e4BqPL8tzok37mH9J59AvABro5K6\n8zKcet0FDU3by9oOkagI6sbFxfLSluN8eriXyhvdUz6c9uHqOMQlnA1tvAzOWBcjQr5KCVyc6lPv\niQhJYmBXxjxlSZk+dEktU8viCpZpM1A0ytm7Bf9PT8NAdM3rdjDvasNck6z6fjsSEHx1FHtHAQzI\nC1OYW9JISluFLSYN7BbRTZ1HcUWLxNcSay3ByEgYoFUqFgmCACczvXosyOehVHEBCgKCchmTyYTT\njClVQ3FbK4f+4BpST+cwI2VKGxpbUjfuqnQ/bo0YJW08Wo3PUDD5ctNtwpK6qcGgiOBaBzOlO4CZ\noQZkpuXNpHWWpVoWGJiatzSJ3PeFp95toFBRkHHcJ/jfA8hfdiGXT26KZPt9/LecgSELxTAwtfvL\n+F8YxflID7LWu0QvIj2Ti8gCdgVlLmrhbLk8PagbVy5Pa69nPW9yUFchGB0NS/OUqscR8rvaGb2y\npymCutmo9q3uNNQMBoFJ7egAghm6B8y0vJlYZu5eZ61l1Da+qrWWQ39t/v/27jQ4suu6D/j/3vd6\n7wYaywCzD2eG4pBDMqJIkSNLoihZYliSqpRYTslJWOUlTlmRrcqnRFE5ikqVVFz6EDupkhelZO1W\nFLsUi3HJoUQvlEUt1EpRHA7J4QxFDgEM9rW3t92TDw97v16AaQC9/H9TKAz6dTdeN4D3zjv3nnNh\nKhE/QAcIPrlcdbP5wyVg3qwHdWv3xVQA85nq+9PuMbBrob+fO4FAGNj1EvHqT9gWf2vWwThO/Sds\n8HxE7eiHxWFE1HQAAArGxoqpnmZgEFbz7oQDP/LiR0RQ6aDiiQUjDU++AuB5rz2zdcYDSlfq3OF5\nD2I2fk7iC+Tbleh+Nh4g3yi3fB97GQO7FrpayuPrMzehFHCuFK3aPuTS4EzGjB11or9aPoWysata\nmbii8MjycUQVkk2b+n3r/IitZfHgw0BEtnw48OF2UOGED+DVGu1LZLVB8Uu+adtXpNbKcmvR27Y7\nUj9FWeZxr5UY2LXYf3j+zfjvP78b005q/aBD3UvH43W3K9uu+3Wj+xN1goUggf90/R78oHQobF8i\nwDU3hc8unNnS7mSzJQMsmuq+diICR7yacUBBHKyIg4p4KIuHZalEFlq0u+uB4EXfYMWEAV4gAk8E\nC0bwnBdg1rTvuUPZQP8bVqt3qjYC6o3bCsfSCsjXCTdO8bjXSnw3W0yg8Lmx2/G5sdvx1Fs+CQDI\nqugKIeoCth1++NXtGVQyCaW3HsxUPA5xnOjMnWWxMpY61lyQxB/Pnsf8oScBAE+s3LJluw2DQ6vz\n6lYMUBDgOQ84bAHHrLA6tiRA2JWu/hBkAIOgwX06waIRLLZxAFfPTR8yuPiwhaCEjdJdDSCtoD+w\nNZhXSkH9Rg7yiSWgsu2JEgr6N3P7sMe9g4Ed0Q1QSkFnMhDHgbju1vYlERWuSmvobBamWNxadGHb\n0BEVtESdp3qM7kyshLenF7B5BK8owLNuuBrF5KYxx9fGOz9g6wWpm4A7/zzA05/IQb5TCTN1b05B\n/1oWarQ6tLDenUGwaCBfWAmj+9UKEvX+Puj7U/u+/92MgR3RDVJKQSWTQLK5zKyyLOhcDgiCMHOn\nNTN11LVG7TLenl5AbNuwXRbArTHgYueNotKq5HHA+vBA0/e3Hs5BfikDuegCWkHdGYdKsOCw1RjY\nEbXQ+uoTQN0VKJRS4RAuUZe7PzUTuXasVkCfDpv1VjpzNJI2kZIBJgJgQEMN1b5QVWkNdR+nJ+0l\nnlmIWkQ8L1yBYm3+3GomTye4ZjD1rqOx8vq6sNsJgBQDu44mnsD84RLk0VI4xOoLcFsc1u/mI4dk\nae+xKpaoBcT3w3lzm4siRCDlMkyNhsREvWAxiNfs8qMAuAzqOpr5rwthUOcirH5xATzjIvjAbJjF\no33HcLoBqWwv4Vm9fapxKtlbLRUyq5/V6j9qH9u71duT8bAIYqfPU+P3BEAY3IlwcWzaHy36NdMR\nrSyihlQb+U55GK9JrCC27bGyurJUsysrWE28sGbuEyWpbNjQMBCUI1qtBBGvO6Ns5HQcFhRcMVgW\nF460a+e5vVEZA+S7lTCY28wAKBmYx0qw/mn2IHatpzFjt49k0z/qMvXWc2QvQ+phV90cnnEy8ESt\n/ykEEjbpvXTAhRMWFA7pNPpUAlkdR1bFMaIzSKr6OY9hncSATiKuLFhKI6VtjOgUBnWypy7dV36i\nUHOcvQLgyQYr7dCeYMZun60FdczcEdF+0AUf6WfDtThL53Mwueo2PHvtyUoeL3pp3J+aRgzAogAz\nQf2VJ/bDoE5BQ61n0/Xq534k4EkQmalLKAtJZa/fd41SChnYSFtZzARlODVenQUgsToVrdMnaegk\n6meI0zzPHQQGdkStYFlh+5Iodapjqf0pAbTXBj8/3yBzcQ7pS/MwMY3i6w6hcra/atm6wF0diBHB\n4NfGMfCN6xAr7BumAsHi2w9j7peOVy93t03FrT49FPzGhUArJuxJVggSm25LYsUkkbYKDR+/aFbW\n/683dbNdNGETcAUgq2JIKxsCoCAeShJuK0nzg1BpFa57FfW3KQA0YpgzYUox2DS41afjNWMZpcJL\n9mErhct+GQbhaw/3W3BrzOCIJTAIh8tWDHDRs1A+4DXGl/3dVanqCwIJTOT7IUkF/8E+uF7zFxKO\nx5CkFfguErWAisUgNQI71WDZMaJGdNHDkU/+DFbBg3bDmaGZS3OonO7H9MO3Rg6H5b47i4HHJqE9\nAbyNzFP+8Sl4Q3EsPzC6j69g92KwEFM2MlZiNX8mANR6xiwuFvIiWBYHFTFNr0cRg64ZoGmlkFDR\nQaIGmrpQ61MWFjfNubsjZnDIElgqzNoBYbuX+xIBvlOx4HfgKI6VUQh+ZwDWHy0Ajqy/AkkqyN1J\nyD1sa3IQOMeOqAWU1lCpVNibbu2gb9uRy4oR7dTQV6/AXnSg3bVCLEC7BsmXlpD7/vXox/z1+Pr9\nN9OuwdBfT3TE3M+EshFfHfZUKgzm1KagDgiDMEsp5HUSN9tp9Knmmn17VaVTG0QEGWXhtJ1azext\nKJgAQYP3zlIK8U2PSypZD+o20yo8CR+xOrd6VN6ZQ/B7I2Egl9eQ0zEEvz2A4KOHGmaFaW8wY9dA\nsLAUefstn1lo+Ng/e+hWAMCv5C5VbRuxWCnUDp5ytx54bv7cHIJy7QrX5qjwgBYYSOCyVKbDKR+I\nLx7g93d8pC8vQkWc+7Vn0P+d63DOH12/TWwLMAJ7rvYMLmvZQ2xWQ2K1LzpK8eplnq4l8g3399nc\nMQDA1ZXh9duKTQzhrompcFg1rQTHLLcqOxaVLVsbAgWAESuJi+UslkyjAE/wrvQK0pCq+GPteySg\ncNRK4VuVLKaDcEjRguDd6SUkIdA14hZPgKteP172E5jw8rg1XoAvs7AiKo1tBaRUAs9UjjTY373z\nUmG48Z1qmFzqA071Ab87snVD41H3Ks4Se362AlMJRERtTJd9SK0IAoAu+RE3Kki89uFdLB3Ou2tj\nh3T03K1GNIBbYs1UYyr81EnCl7BKtxZbAXfFy+tfB1D423IfZgOrZtJTALzqb0zB8BrMoXN2MDeQ\nqBH+NhERtbEgG0e9tK8/ED2PaeXeEZiI4M1YCoV7hlEz3dThtALyun69rQXBW5IFXEiW19+GeqOr\neR1s6eFXEo2/r/ThiXIGa1MYZfWzK8A/lHMINoWlr3ipmqOSrihcdHJNvz6iRhjYERG1M1uj9NpR\nGLv6cG1sjZU3noh82MJDJ+GNpGA2Ze5EK8BWCLJx6OIBN5FrYNbsph1yGGAVGmTAXhcvY8gKYK+2\nYbNU4+lgUfsyYeJ4pJjHj500nvWS+LGTxv8t5jFnts5y8qHxt6UheKJgNj2RKwrXvBRe9quHvYl2\ni3PsiIja3PIDp6ALLlJX5yFQ4TROI1h50wk4Nw9GPkYSFiY+eCcyF+fR/3djiM+UARFoR9D/D+Po\nf2ICk//6PJxT7ZktKorCvBEM6rAYYY2Rra3TtgdkAYAX3dpztWwITsa8mn11txMBpgMbUmNgOIDC\ny03MIXzezWEpiOFCchEjtouSsfCU04dLbhYtWy6ECAzsiIjan6Wx+J5zWFkoI3FtCWJpVM4OQFIN\neoTZGu7hNGILDjbP29e+ABCMfu55XPvIPYDVnoM3V32DkqVw1NKwoeADeMW38HPPwmHL4Fzch5Zw\nHpyRcCWrF9wEZk3tU1tKhS1RapVWiGwEi0bCQPEpN92S13M9SOKR4uGWPBdRLQzsiIg6RDCQQmlg\nZ8N2ue9PQvnR7TRUYJC6vITybQOt2L09cT0QXA8CLAQZbDR7Aa4FGmNlC0csgyEdwIHCFTeFFalf\nDVsRXXcOkoewKbUCcN23cdFNYqnOc2ZVgNMxB0kIZo2Na358y/w6ov3GwK4RiT4gmksvNnzoVz78\nEADgT371LVXbfvCGP234+LTe+8a2avXfZrLeBrT5GS4Kq53tlYKszkIOYBDUaBfq1Vgs+5Vgb/o5\nHbaiX8sH/uDfbfl65Pkna/7MqTfpAEjNdVDTmm1FEfEpFxFdNkIGSE56MKMRRRZ2dTZwvok2TRdz\nYeuVa/MbweJyJizwcBu2IAG8Tdm2fru0/v9XnaGGjy0Fza1ycNq6jNuTS7C3vTGO0fiLpVP4fvnQ\nlttNjarWd2Qm8I/7x6AhsBVwzPg4H3Px+7PnMb06b27mgAsj3KDxe/785O6bVftTrZsfmFxoz8xx\np+G72ONk07+1rzd/bsZaYLjW+0mtNhO1wlUYW7zHRLQT3uFsZHXs+vah3pu4/6XFM5j1E6iY8BRo\nJAzqflIZxPfLzfV0uylWwLv7xhFXYVAHAEltkNE+/s3gZdQtZSbaQ8zYEQQCBbUlwNupWgGcBQ3/\nwJf6JupdpbtGkf3BeFWzNlFA0JeAd6w9iyf2Ukls/N7MnbgjuYjbEotwjYUfVYbwqpdp+jnelr2O\nWETXaK2AvOXiZKyIax4b0dP+Y2BHAHaWoWuWUgqqA5YtIupmJhvH/D+7DYN/+XyYRFotKw1yCcz9\nyvmeXfbJQOFnlQH8rLK7+YXDtlOzFaCBwoDl4lp7d5ShLsXAjvaMiNRZjZGI9ot7sh+TH7wXyZcW\noIse/OE03OO5ng3qWmHcTeNkrBjZNsWCYNqPbhzdTZRrEJsqI0jbMLnm5jfS3mNgRy2xNpy7nalR\nPEFE+8zWqNzSuACBmvN48TDuS8/B2jYcGwgw4aVw3W9Ni5S2ZAT5L19H3zfmwq8DQeV0DtMPn4U/\nyPVeDxqLJ+iGrRdciGz58BEwX0fUKXyDxJV5pC5Ow54rNb5/j7vup/HFxdNwjUbFaPgCVIzGtJ/E\n/5y/5aB3b08NfWoMfV+fhXZM+OELUleWcfz3L0KXI9Yupn3FjB21hEBQFg8JZUPAIViiTpK4Oo+B\nv3oRgITz8ATInslh8tfPQRKN22X0qh+Xh3GpksddqXlktY9X3Awuu33o5pUkrAUP2ScWoLytx3gl\ngHYC5J6cwdLbjhzQ3hHAjB210FrfOgZ1RJ3Dni1h4JHL0G4A7Rpoz0D7BsmXljH6pcb9OntdWWx8\nrzSCvykcxWW3H90c1AFA4vkipEb7HO0aZC4u7PMe0XbM2FHPsxbKUI6PYIcd/Ym6QeaHE1ARjcG1\nL0i9sAhrwUEwwHlT3UYcgbzkAXEFpKXpQhqJq7qxq4kzX3TQGNhRz3rlUhqDf/YUrBUHolV4cmN7\nFuox8YlCzZUpxNaIT5VQZmDXNUQE/v8qIvhiMQzQDHA8tYzZD5xA5bWNexpW7sihVk2ciWusvGGk\ntTtMO8bAbpckaNx0N/vjMQBAZeBU1bYXXt/4quY1thM+j+ZBtdUWJmP4z798B6yVMhS6ffCEeklq\nZocXJ4nabSpUILC9OJIz4V9IpYlj0ZVcuHJDaW6jKtQph9/D8RufciqblgV769ALDe9POxN8pYTg\nCwWgsnGbXfEw+vGXMPcbx1B4cKhu9k4SGnP/6hiGPj0G7W78rpmYRuVUFsU723fd4V7BwI560jc+\newSewwXPiCrnj8CeWoHyt6ZhBIDJxOGNNr8aA7U38QX+57cGdesMMPTZcSQvFzH7OyfrBnfFtw4i\nOBRH/1cmEX+5giBtY+n+w1h6y2jVWsW0/xjYUU/62bfy8F1W+xH5x/Nwzw4jfnUW8A0UALEUxNaY\ne+85NjHuIjIZADVWw1AAYID0D5aRvncJpQv5us9VuT2Lyu03w5/i3OR2w8COelImx15LRAAApVB6\n4xm4Z4YRvzwNXfbgHenH0ptGYdK1h2l1yUf2hzOIT5XgDSexct8ITJarD7QzlVJVawZvpx2Dvkdn\nGwZ21L4Y2FFPetvDU7jy0xycErN2RFAK/pF++Ef6128y6dqZuuTVZRz51HOAAbRnYGyFwa+PYepX\nXwO5iaeVdqWGLKgzNuSF+he21gIXue1krEumnnTh3XO45fUrEI4yEe2IcgMc+dTz4YoDXjgvT/sC\n7RmMfuFF6CVmw9tZ7MP9QKpOcYQCnLNdvBxaD2BgRz3JsoEPfeESincfYztloh3IPD1fpy2QIPXN\nxX3dH9oZfSaGxOeHgWPRoxUSU1h6D1uWdDIGdtSzLBsovfEUlh+8GcbWMDH+ORA1Yi84UF50IzPt\nCexJd5/3iHZKjVpIfGYY+hcSQBwwSQ2T0jBJjdl/ewreTSyI6GScDEE9z7l1BM7ZIcSvLSL//y4D\nzOER1eQdSkLiGsqpDu5MXMM7wb6bnUAlFeIfH4AZ8zH+vTRMSqPyj3IQrhzR8fgTJAKAmAX3bP3G\nnEQEFO8chNg6+vJHAeUHWE3ZSfRxG8X7B1B+fT+Dui7BnyIRETXP1pj4wHmYjI0goSEKCBIaQdLC\n9d+6DZJhpTnRQeJQLBER7Yh7LIOXP3YPMhcXEJspwx9MhJm8uIUUlg9694h6GgO7faCXy4i/MAVr\nuYJgIA3n3OhB7xIR0Y2xNYp3DdXebgSZZxaR+8EcNAyK9/YDDyaAGKc7EO0lBnZ7yB+fAAD0fXVi\n/bbY2AKSz4zjI4+9HToRr/v44NQRAMDlX9/oKZQea89hDhMHTn59BYvnspHbtb91Rk56MroBZuLS\nqy3fNwCQkcHI20ee+e6efD+inuYZHP9vzyH5agl6tcgi/fQyvK/GMfaRW2HStU89FXdjW9reqLCd\nLOUaflsn2JtTWmB2P2tpuZxs4Z7snDGNA2lvevdVsMmZ1s3oSiy07Kl6GgO7A2KWlxHdMGCDmg9/\ny299qW/9Nqk4e7hXu6e0himXMfDT5gJPCYLI2/0at9+w6dm9eV7qetoTpGf26PeyjRmr+vQgduMg\nodifwuAj40i+XNxyQaddg9i0g5HPjGPmX9xc8/GlTYHIi8lD6/9fKjUOkAJ/b6aNyw0EdkHlYC/G\nJWj8M0vO7/71Jed2/dAqqflGZ0VqBosniIiopfofn6nK0gOADgSZp2cBnydwor3CwI6IiFrKKtZf\nVkw7vZcBJdovHIolIqKW8kYSiE9GTxuRmIZJ8dTTSrrkY/CrE+j77iy0Y+AcS2HuvcdQei17CvYi\n/nVRbxBh82GifTL/nqMY+ewr0NuWHjMxjcUHjgJawVp2YS848PMJBP31C8l6SerSMoYeGUfiWhkm\npbH0wCEsvPMwJBE9V085AU78l+dgz7nrw9/JV8s48icvYfrhE1h+I9d97TUM7AgAIMYAxoTBj9ZQ\nDIKIaJdWLgwiccVD/7fGAa0AAZQRFO4exsobRjH66eeQenEJYmso36Byug/TD78GJhvb1/1UrkHy\n2QKUa+DckkEwsL/ff7vcd2Yx8sVr0G4YEGvXYODRSWSeWsTYR26DRKxn3ffELOwFt2pOo3YNDn35\nVazcOxz5OOpeDOx6nBgDcZwwo7VGKSCRgNLddDAQAAxWifaFUlh88CSWf+EI0pcXYJUFzuk8yidj\nOPYHP0NsugwdCOCHc+1SV5dw9I8uYuzf37Vvu5h5Yh7Dnx7fOCz2kCWsAAAJ6UlEQVT4gsL9A5j7\nzeOAtf/HCuUZHPrSRlC3RnuC+JSD3JPzWL5/uOpxfd+bg3Zrr2+dfKmA8rm+mtup+3TTmZt2SESq\ng7pwA6RSgWy/nYhoB0w2hsLdIyjdNYqgP4HklWXE5iphULeJMoC97CL93P40Mks8V8Dwp8agKwa6\nvPrhCbLfXsDA/76+L/uwXfLFQs1t2jXIfbtGy6ZGdSgsQO45DOx6mTHVQd1me9VT7kAwW0d00JI/\nX4ZyoyMN7Rgkr+7PcmT5v5yCishyaVfQ99hczX3cS8qXuocpXaNFTOH1eZgaq3koA1TOZFqxe9RB\nGNj1MlP/4FWriTAR0W5IwoLUGOYUDZjk/jTzTbxcrh1DKcCedmtt3TOVmzNQXvSFtgBwh6ILTJbe\nNgKTtiHbzuYmrjH/niM1iy6oezGw62WNCiS6qYCim14LUYcqvHaoZlZKtELhddVzyPZCkKkT7ASC\nILv/wZBJ26jclEZUaKcAZJ5djmzsbDI2rn30NhTvykMsBbEU/HwMM//yBBbedWTP95vaD4snuoCI\nQHx/Y+jUsqBsu3Flq1X/4KVs/noQUesE+QQW33Ec+b8b39IKxcQ1lt50GP6hFID6zY03s8cqyH1t\nDvGXy/BH41h66BCcWxoPPS4/NIzBL1+vKjoQBThn0zD5g6mOtZe92plEARKvluGcrn59wUAc1z94\nM5RnoFwDk7Z4MdvDeObucLJa6LBlrpwxYaCXTNYN7pRSkFgM8LzqjbFYl1XFElE7WHzHcTgnsuh/\nfByx2Qr8wSQW33oU5fMDO3qe1PcWMfSJMShfoAwQv1pG6kfLWPwno1h672jdx648OIzMj5aQuFKG\ndsIA08QVJKkx+9sndv3abtgNBmMS02xtQgzsOp24bnQBhAjEdaESibqP17EYxLIgnrfexFcxqCOi\nPVQ+l0f53O5XRVDlAEOfGNuScVMCKEeQf2QKpQv98I4laz+BrTD5H88i/cMl5B6fh3IMSnf3ofCL\nQzD1hmn32Mp9gxh4dDJynV1YCs7J9P7vFHUcBnYdTETqV642WfygtG4YABIRtYvU95ZqblO+IPvN\neSw8fLT+k2iF0oU8ShfaZ9mtxQdH0f+tGagVH2rTdDoTV5j55ycOpL8edR6mZTqUGBMOwTa6H3vR\nEVEXyfzNHAb/dCKyXQkQtviwFiOml3QAk7Vx7aPnsXLvYFgIAcA5ksTk+89i5U37U1hCnY8Zuw60\nPq+uEaW4NBgRdY30txYw8LnqoofNTELBOde5vduCgTim3n8GU78lYXNhZulohxjYdSDxm6saU7GD\nXfeQiKhlRJD/0hS0UzuoE4QFBIU376wQoy0pBbAFHe0CA7uDohqPgqvVdiRqYGMOiCoUYRYaL7uj\nshlYmd1ftUoQwJTKgO8DMRs6lVrfn0iWBeW6UKlU9PbtzZBrBKdScXa5x/XV2nep0c2diNqLLgSw\nlmpf1AoAfzSOqQ+dhuy20bERpJ5eQebbC1C+oHRfP4oX+gEWk1EHYWDXgZTSkU0s17f39cFK1akI\nayAolyHLKxs3uC6CYgk63w/NIgsiOgAS03WXQJSEwtj/uHX3LUN8g8Mf/zkSV0rQlfCCL/30CvL/\nZwoTH7sFJsvTJXUGXoZ0IJVK1j54KQWd3H3wJUGwNajbxCwuQRosQ0ZEtBckqVG5PQuJOPSJBRQf\nGLihPnD9X5tB4nJxPagDAF0xsKcdDH1+bNfPS7TfeAlyUKRxgGS81WGHicmNG4OgfqWr1pDFpboZ\nvbrfs8H8PTO/UHNY03g+VFDY5Xde1cT7squn5bK3tEvKFySnmyhW6jLGrp5WYWKNhzjd/nBub2J+\n47ZYMfysTOPAy5GN09JsMrv+/6AYw+T7zuDklWehHAMdyOp+KgS5GKYfOgUzH72eajP6Hp2LLMrQ\nPpD5/iKmftmGxHeeC7EqB5s/UU0c+zb/rHYqNde6Y3ZqtjOrmdsNA7sOpJQCbDtcYcKYjcbClsXG\nwkTUtbyRJF7+2J0YeOw6sk8tAJbC8oUhLP7iYZiMDdX8amRVdLHOgxWgKz6C+O4DR6L9wsCuQyml\nwoKFBuu97uZ562b72D6FiA5QkI9j9n2nMPu+Uy19Xu9QEonr5chtYmkEGXYZoM7A9A5t1Sjjx8CO\niLrQ/LuOwUQMtZq4xuLbDrOfHHUMZuwOSDOZNrWa9levuWnjtqUbnMPWBHFdBFMzqxVognAcQsMa\nPVS7N55lQSYmoWwbEvjhsHAisTE0bJqb9RcsLbfkNWxXq6AkKOz9+0lE7a9wzxBiUxUMfn08DOIE\ngBGs3D2I+Xceq/vY2GQZ2afmoD1B6Vwfyrf08SKYDgwDO6qi4nFYx4+Gq1v4AWDbUMlE3VUspFIJ\nP9a+BoBCEbq/D5rzUog6TnqielhSVI0+lZt42fCiNT29cTEXXwkn2FtuExe03sZxphzbuCBLFJro\n/XmDRVKle0+icv4IUi8uAIGgcqYfwUCydnGBCPKP/hzZn0xDGQMYIP/NSXjDKUz/2h2QhAW7dGP7\ndKOaeU8yU7svgMhMtK73aGw6uiMD7QyHYimSUgo6lYLOZaFTyfpBnTEIJqcit5mlZbZIIaKOYTIx\nFO8aQfGeUQQD9fuBpp+ZRfapaWjfQBlAAdCuQWyqhIGvXd2fHSbahoEd3TAplVGv4sLs0WoSREQH\nqe/b49Be9YWrDgTp5+agKjdQpku0SxyKPSASNM6PSzkcClGXruzocftufS5ejc3FIvxSdLVZ9Z33\nJrsXFNgfiehGZcYb/x27uXApw+zExt9cbNkFANhO4xVxNg/Xir2Re1jrhVfPjbQ7qfu8NQ5v9mKd\ni1alkH3Zg0oc7IKvupmh2HFefHcTZuyIiIh2weRqr/KjjMBkOL+Y9h8DO9oHrA4jou5Tet2xLVnF\nNaIVnDNDkPjBZuuoNzGwoxunVJ3+d4pl/0TUldybh1G5bRRiaYgOJ6RITMMfzqD4lrMHvXvUozjH\njlpiY8UKhfXed+GGA9snIqI9pRSKbz6N8h2Hkbg6BxUE8I7l4R1lHzs6OAzsqLWUAodeiaiXmHwK\n5XuOH/RuEAHgUCwRERFR12BgR0RERNQlGNgRERERdQkGdkRERERdgoEdERERUZdgYEdERETUJRjY\nEREREXUJBnZEREREXYKBHREREVGXYGBHRERE1CUY2BERERF1CQZ2RERERF2CgR0RERFRl2BgR0RE\nRNQlGNgRERERdQkGdkRERERdQolI83dWagbAK3u3O0R0gE6JyKGD3ol2w+MeUVfruuPejgI7IiIi\nImpfHIolIiIi6hIM7IiIiIi6BAM7IiIioi7BwI6IiIioSzCwIyIiIuoSDOyIiIiIugQDOyIiIqIu\nwcCOiIiIqEswsCMiIiLqEv8fcR7HHDFaoH0AAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "from sklearn.datasets import make_circles\n", "from sklearn.ensemble import RandomTreesEmbedding, ExtraTreesClassifier\n", "from sklearn.decomposition import TruncatedSVD\n", "from sklearn.naive_bayes import BernoulliNB\n", "\n", "# make a synthetic dataset\n", "X, y = make_circles(factor=0.5, random_state=0, noise=0.05)\n", "\n", "# use RandomTreesEmbedding to transform data\n", "hasher = RandomTreesEmbedding(n_estimators=10, random_state=0, max_depth=3)\n", "X_transformed = hasher.fit_transform(X)\n", "\n", "# Visualize result after dimensionality reduction using truncated SVD\n", "svd = TruncatedSVD(n_components=2)\n", "X_reduced = svd.fit_transform(X_transformed)\n", "\n", "# Learn a Naive Bayes classifier on the transformed data\n", "nb = BernoulliNB()\n", "nb.fit(X_transformed, y)\n", "\n", "\n", "# Learn an ExtraTreesClassifier for comparison\n", "trees = ExtraTreesClassifier(max_depth=3, n_estimators=10, random_state=0)\n", "trees.fit(X, y)\n", "\n", "\n", "# scatter plot of original and reduced data\n", "fig = plt.figure(figsize=(9, 8))\n", "\n", "ax = plt.subplot(221)\n", "ax.scatter(X[:, 0], X[:, 1], c=y, s=50)\n", "ax.set_title(\"Original Data (2d)\")\n", "ax.set_xticks(())\n", "ax.set_yticks(())\n", "\n", "ax = plt.subplot(222)\n", "ax.scatter(X_reduced[:, 0], X_reduced[:, 1], c=y, s=50)\n", "ax.set_title(\"Truncated SVD reduction (2d) of transformed data (%dd)\" %\n", " X_transformed.shape[1])\n", "ax.set_xticks(())\n", "ax.set_yticks(())\n", "\n", "# Plot the decision in original space. For that, we will assign a color to each\n", "# point in the mesh [x_min, x_max]x[y_min, y_max].\n", "h = .01\n", "x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n", "y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n", "xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n", "\n", "# transform grid using RandomTreesEmbedding\n", "transformed_grid = hasher.transform(np.c_[xx.ravel(), yy.ravel()])\n", "y_grid_pred = nb.predict_proba(transformed_grid)[:, 1]\n", "\n", "ax = plt.subplot(223)\n", "ax.set_title(\"Naive Bayes on Transformed data\")\n", "ax.pcolormesh(xx, yy, y_grid_pred.reshape(xx.shape))\n", "ax.scatter(X[:, 0], X[:, 1], c=y, s=50)\n", "ax.set_ylim(-1.4, 1.4)\n", "ax.set_xlim(-1.4, 1.4)\n", "ax.set_xticks(())\n", "ax.set_yticks(())\n", "\n", "# transform grid using ExtraTreesClassifier\n", "y_grid_pred = trees.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]\n", "\n", "ax = plt.subplot(224)\n", "ax.set_title(\"ExtraTrees predictions\")\n", "ax.pcolormesh(xx, yy, y_grid_pred.reshape(xx.shape))\n", "ax.scatter(X[:, 0], X[:, 1], c=y, s=50)\n", "ax.set_ylim(-1.4, 1.4)\n", "ax.set_xlim(-1.4, 1.4)\n", "ax.set_xticks(())\n", "ax.set_yticks(())\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "hidden": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEWCAYAAACwtjr+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvXl8HMWZ8P+tPmZ6Dml0y7bkQz6wjQ8ZMBBgDVnAcbhy\nsOs3CSRcm03IRXg3bJaFZUNY2N28CT/YTQgENuEKIQRIyCYknI4DBHNjGzC+LVu2ZVnnSHN0Tx/1\n+6NH45Es2ZItI9v09/PRRzNT1VVPV1fV01X11FNCSklAQEBAQMChRBlrAQICAgICjn4CZRMQEBAQ\ncMgJlE1AQEBAwCEnUDYBAQEBAYecQNkEBAQEBBxyAmUTEBAQEHDI+dArGyHExUKIZ8ZajgNBCHGj\nEOLnH1BeXxFCtAohUkKIyg8iz8MJIcR9QoibD1X8DwohxGVCiJf2Eb5ICLHug5QpYGQcqc9o1JSN\nEOIiIcQb+c6oRQjxRyHEX41W+ocKKeVDUsqPjbUchzNCCB34/4CPSSnjUsqODzBvKYSY/kHldyDs\nrwM/nBlYvlLKF6WUMw9RXqOqgD9ohS6EaBJCnP1B5VeU7xH7jIoZFWUjhPgH4Hbg34FaYBLwY+CT\no5H+oUIIoR2NeR0CagEDeG+kFwqfD/0IOiDgw4AQQh0yUEp5UH9AAkgBS/cRJ4yvjHbm/24Hwvmw\njwLbgW8Du4EW4FPAucB6oBO4riitG4HHgEeAXuAtoLEo/FpgUz5sDfDporDLgL8AtwEdwM35314q\niiOBK4ENQDdwByDyYSpwK9AObAG+no+vDXHfTcA/AasBC9CGId9LwA+Arnwe5xSFNwB/zl/7LPAj\n4OdF4Z/AVwjdwHJg9gBZ/jEvSxr4Kb4S+WM+veeA8kHu4Zh8fJl/zsvyv58KvA4k8/9PLbpmOXBL\nvqyzwPR8Pflp/vnuyJe9mo8/PX9fyXzZPpL//YV8vul83p8BqoDf5++xE3gRUIYo//8CmoEe4E1g\n0YB69Cvggfz9vwcsLAo/Dr9u9eLXtV8CNw+Sx2zABNy8jN353+/DrztP5tN4FZhWdN2s/DPsBNYB\n/2cf7Wd5vrxezufxO6ASeCh/b68DU/JxpzCgTuav/2JxHdtH+X4U2H6AbbmQ9oD2NB34EmADub57\nKKqX/4zfFrqAewHjQNMbRF4JfBW/PfcC/wZMy5dlT74OhIrinw+sxK9fLwPz878/CHj49TkFfDv/\n+6PALvy6+wIwZz995Wi0gX7PiBG27aFk3sczmo1fh7rx28knitK6D7gT+EM+77OHvP9RUDYfBxyG\n6HDzcW4CXgFqgOr8Q/y3ImXjAP8K6MDfA23AL4ASYE7+ATcUdRI28Lf5+Nfgd8p6PnwpMAF/1PaZ\nfAGML6q8DvAN/I4/wuDK5vdAGf4IrQ34eD7sSvxGUQ+U5x/i/pTNSmAiEBmmfHa+DFTgK/gNuk/Z\nrcCfzgoDp+cr0s8HKIXF+XL5NrCRfEPKy/IKfiWsw1fsb+F3qgawDPjOEPcxpfg+gQr8juEL+XL8\nXP57ZVHnti3/7LS8PL8BfgLE8OvBa8CX8/EfBq7Pl4kB/NXAzqXo+38Ad+XT1IFFfeUziNyfx++U\nNeBb+A2sryO7EV9JnJsv6/8AXsmHhYCtwP/N5/G3+eeyl7LZR6d4H/4LzUn5/B8CfpkPi+Erwcvz\nYcfhdzDH7kPZbMTvJBP4dXA9cHb++geAe0eqbIYo34+yb2Wzr7Y8WDkU0s+Xyc0DwpuAd/HbSAX+\nC8rNB5reIPJK4LdAKX59tIDngalFZXlpPu5x+O3i5HyduDQvX7hI1rMHpH8Ffj/Vp4RX7kOW0WoD\n/Z4RI2zb+5J5YJni1/+NwHX47eJM/H5nZlH8JHBan+xD3v++HtRw/oCLgV37ibMJOLfo+xKgqajg\nsuzR8CX5wj25KP6bwKeKOolXisIU/DeFRUPkvRL4ZFHl3bavjiKfd/GD/hVwbf7zsr7Kkf9+NvtX\nNlfsp2wGyrexKCyaT38cvuJzgFhR+C/Yo2xuAH41oFx2AB8tkuXiovDHgTuLvn8DeGIIGafQX9l8\nAXhtQJwVwGX5z8uBm4rCavEbeaTot88Bf8p/fgC4G6gforMobmg34Xce0weTdT9l3UV+FJyvR88V\nhR0LZPOfT6dIyed/e5mRK5v/Kfp+LrA2//kzwIsD4v+EoZX9cuD6ou+3An8s+n4B+Q5j4LMqun60\nlM2+2vJg5TAcZXPlgHLadKDpDVF/Tiv6/ibwTwPK8vb85zvJK86i8HXAGUWyDv3m7r+gSiAxSNho\ntoF+z4iDa9v9ZB5Ypvgvc7somj3AV4w3FsV/YDjtbzTm0juAqv2sSUzAf1PsY2v+t0IaUko3/zmb\n/99aFJ4F4kXfm/s+SCk9/Gm4CQBCiEuEECuFEN1CiG5gLv7Uy17X7oNdRZ8zRXlPGHD9cNLqF2cY\n8hXyllJm8h/j+by7pJTporjFZdqvjPPl0oz/ptPHwDLdVxnvi4HPs0+W4ryK73sy/htSS9F9/wT/\n7Q78UZgAXhNCvCeEuGIfeX8f/03rGSHEZiHEtUNFFEJcI4R4XwiRzOeZYIiyxn/ORr4eTwB2yHxr\nKrq/kTJUPZoMnNxXFnnZLsZ/qRiK0Xp2wyZvqZnK//0x//P+2vKBUFxXRiO9gQy37CYD3xrwXCYO\nJY8QQhVC/KcQYpMQoge/04f+dayP0WwDB3yPI5QZ8n1evj/pY19tfUhGY9F6Bb7G/hT+Wspg7MQv\n7L4F5kn53w6UiX0f8ovP9cBOIcRk4B7gLGCFlNIVQqzEf4h9SA6clnxee8mxDwr5DVO+feVdLoSI\nFSmcSUXp7wTmFeUl8vLtGEbaI6XveRYzCXiq6HtxOTfj15EqKaUzMDEp5S78qUPyFozPCSFekFJu\nHCRuL/6U2LeEEHOBZUKI16WUzxfHE0Iswm/AZwHvSSk9IUQXwy/rOiGEKFI4k/Df6gdjpHWqGfiz\nlHLxCK8bDn11I4q/JgH7VmJDIqV8CH/6r5h9teV0Pl8AhBAD8x2qnIrb0Wikd6A0A7dIKW8ZInxg\nfhfhG0Gdjd9pJ/BHz4PVsVFrAwfJ/mQeeI87gYlCCKVI4UzCn8ZliGsG5aBHNlLKJP56yx1CiE8J\nIaJCCF0IcY4Q4v/loz0M/IsQoloIUZWPfzD7Q04QQlyYfwu9Gv8hvoI/Fyrx11kQQlyOP3IYLX4F\nfFMIUSeEKMNf/B8JByyflHIr8AbwXSFEKF8hLxgg23lCiLPypsrfwi+Xl0co43D4A3BM3txdE0J8\nBn8a6vdDyN4CPAPcKoQoFUIoQohpQogzAIQQS4UQfUq8C7+M+ip2K/78Ovm45wshpueVaRJ/Yb74\nrauPEvxpxzZAE0L8K/68/XBYkb/2qnxdvhB/7WUoWoF6IURomOn/Hr/8vpBPXxdCnCiEmD3M64dE\nStmG/4Lx+fxb7BX4az1D0a98h8G+2vIqYI4QYoEQwsCfqhxOXl8TQtQLISrw1y0eOcj0DpR7gCuF\nECfnrShjQojzhBAlQ+RXgt/GOvCV4r8PlfBotoGDZH8yD8zrVfxR+bfz9fSj+P3OL0ea8aiYpEop\nbwX+AfgX/MbdjG+p9UQ+ys34HeVq4B38xauDseX+Lf68d98i9YVSSltKuQZ/DnYFfqHNw19wHC3u\nwa8wq4G38TtdB7/D2y+jIN9F+IuXncB38Od5+9Jeh78g/kP8xeYLgAuklLkRpD8spL/P5nx8hdaB\nP4I4X0rZvo/LLsFfYOyzOnoMGJ8POxF4VQiRAv4X+KaUcnM+7Ebg/vzUw/8BZuAbZqTwy/HHUso/\nDZLf0/gjrfX4w36TYQ7382V2If6aQSd+Xfv1Pi5Zhv+mv0sIsa8y6Eu/F/gY8Fn8N8ddwPfwF2xH\ng7/Ht07qwF8U39cLx430L9/9MWRbllKux19Tew7f+mvg3qOfAsfm83qi6Pdf4Lerzfijx4NN74CQ\nUr6BX3Y/wq+jG/HrQB//ga9ou4UQ1+C3v634yn0N/gvvvhitNnAw7E/mfmWabwsXAOfg9ys/Bi6R\nUq4dacZCytEeiR5ahBA34i+Wff4wkOUc4C4p5cAppYCAgGEghGjCN154bqxlCTi0BJvtRoAQIiKE\nODc/dVSHP7r4zVjLFRAQEHC4M6bKRghRJoR4TAixVvhWQ6eMpTzDQADfxR8Cvw28jz9nHRAQEBCw\nD8Z0Gk0IcT/+foP/yS+uRqWU3WMmUEBAQEDAIWHMlI0QIoG/oXGqPNIWjgICAgICRsRYOodswLdc\nu1cI0Yi/s/ebAzYtIoT4Er7PHmKx2AmzZs36wAUNCAgIOJJ5880326WU1WMpw1iObBbim92dJqV8\nVQjxX0CPlPKGoa5ZuHChfOONNz4wGQMCAgKOBoQQb0opF46lDGNpILAd37/Pq/nvjwHHj6E8AQEB\nAQGHiDFTNnn3DM1CiL5DgM7C32QUEBAQEHCUMdYHen0DeChvibYZ3+V6QEBAQMBRxpgqGynlSmBM\n5xEDAo4kbNtm+/btmKY51qIEHIYYhkF9fT26ro+1KHsx1iObgICAEbB9+3ZKSkqYMmUKvi/SgAAf\nKSUdHR1s376dhoaGsRZnLwJ3NQEBRxCmaVJZWRkomoC9EEJQWVl52I56A2UTEHCEESiagKE4nOtG\noGwCAgICAg45gbIJCAgYEaqqsmDBAubMmUNjYyO33nornjfY+XV7aGpq4he/+MUHJGHA4UhgIBAQ\ncBRy++2v0NaWHjK8ujrG1Vd/5IDSjkQirFy5EoDdu3dz0UUX0dPTw3e/+90hr+lTNhdddNEB5Rlw\n5BOMbAICjkLa2tJMnlw25N++FNFIqKmp4e677+ZHP/oRUkqamppYtGgRxx9/PMcffzwvv+wfEnrt\ntdfy4osvsmDBAm677bYh4wUcvQQjm4CAgINi6tSpuK7L7t27qamp4dlnn8UwDDZs2MDnPvc53njj\nDf7zP/+TH/zgB/z+978HIJPJDBov4OglUDYBAQGjhm3bfP3rX2flypWoqsr69esPKl7A0UOgbAIC\nAg6KzZs3o6oqNTU1fPe736W2tpZVq1bheR6GYQx6zW233TaseAFHD8GaTUBAwAHT1tbGlVdeyde/\n/nWEECSTScaPH4+iKDz44IO4rgtASUkJvb29heuGihdw9BIom4CAgBGRzWYLps9nn302H/vYx/jO\nd74DwFe/+lXuv/9+GhsbWbt2LbFYDID58+ejqiqNjY3cdtttQ8YLOHoJptECAgJGxL5GITNmzGD1\n6tWF79/73vcA0HWdZcuW9Ys7WLyAo5dA2QQEHIVUV8fYurV7n+EBAR8kgbIJCDgKOdANmwEBh4pg\nzSYgICAg4JATKJuAgICAgENOoGwCAgICAg45gbIJCAgICDjkBMomICBgRAgh+PznP1/47jgO1dXV\nnH/++fu9Nh6PA3sfOfDGG29w1VVX7RV/+fLlJBIJFixYwKxZs7jmmmsKYffddx/V1dUsWLCABQsW\ncMkllxzMbX0g3H///cyYMYMZM2Zw//33Dxpn5cqVfOQjH2HBggUsXLiQ1157DYDf/va3zJ8/v/D7\nSy+99EGKfvBIKY+YvxNOOEEGBHyYWbNmzYji244nd3Y6cmNLTu7sdKTteActQywWk42NjTKTyUgp\npfzDH/4gGxsb5XnnnTesa6WU8k9/+tOw4hfHy2QycubMmfKll16SUkp57733yq997WsHehsHhOM4\nB3xtR0eHbGhokB0dHbKzs1M2NDTIzs7OveItXrxY/uEPf5BSSvnkk0/KM844Q0opZW9vr/Q8//mt\nWrVKzpw5c9B8BqsjwBtyjPvvYGQTEHCUksx4PLvKZMU6k3eabFasM3l2lUkys++DzobDueeey5NP\nPgnAww8/zOc+97lC2I033sgPfvCDwve5c+fS1NTU7/qBRw4sX758vyOjSCTCggUL2LFjx7DlfPTR\nR5k7dy6NjY2cfvrpgL8p9ZprrmHu3LnMnz+fH/7whwA8//zzHHfcccybN48rrrgCy7IAmDJlCv/0\nT//E8ccfz6OPPsqmTZv4+Mc/zgknnMCiRYtYu3btsGR5+umnWbx4MRUVFZSXl7N48WKeeuqpveIJ\nIejp6QF8tz4TJkwA/FFh37HP6XT6sD4CejCCfTYBAUchjit5ea0FSKpK1MLvacvj5bUWixsNNPXA\nO6vPfvaz3HTTTZx//vmsXr2aK664ghdffHHY1w88cmD58uX7vaarq4sNGzYUlAbAI488UphO+uY3\nv8nll1/e75qbbrqJp59+mrq6Orq7/U2ud999N01NTaxcuRJN0+js7MQ0TS677DKef/55jjnmGC65\n5BLuvPNOrr76agAqKyt56623ADjrrLO46667mDFjBq+++ipf/epXWbZsGQ899BDf//7395J7+vTp\nPPbYY+zYsYOJEycWfq+vrx9Ucd5+++0sWbKEa665Bs/z+p3185vf/IZ//ud/Zvfu3QVlf6QQjGwC\nAo5C2no8sjmPWLh/E4+FFbI5j7aegxvdzJ8/n6amJh5++GHOPffcg0prf7z44os0NjZSV1fHkiVL\nGDduXCHsM5/5DCtXrmTlypV7KRqA0047jcsuu4x77rmn4Gbnueee48tf/jKa5r9rV1RUsG7dOhoa\nGjjmmGMAuPTSS3nhhRf65QOQSqV4+eWXWbp0KQsWLODLX/4yLS0tAFx88cUFWYr/HnvssRHd7513\n3sltt91Gc3Mzt912G3/3d39XCPv0pz/N2rVreeKJJ7jhhhtGlO5YEyibgICjkIzloTD4yEUgyOYO\nfirtE5/4BNdcc02/KTQATdPwvD3pm6Z5UPksWrSIVatW8d577/HTn/60cCT1cLjrrru4+eabaW5u\n5oQTTqCjo+OAZOhzFOp5HmVlZf2Uyfvvvw/AQw89VDBWKP7727/9WwDq6upobm4upLl9+3bq6ur2\nyuv+++/nwgsvBGDp0qUFA4FiTj/9dDZv3kx7e/sB3c9YECibgICjkGhYwUMOGiaRREIH3/SvuOIK\nvvOd7zBv3rx+v0+ZMqUw5fTWW2+xZcuWva4deOTAcGhoaODaa68dkdPOTZs2cfLJJ3PTTTdRXV1N\nc3Mzixcv5ic/+QmO4wDQ2dnJzJkzaWpqYuPGjQA8+OCDnHHGGXulV1paSkNDA48++ijgG1itWrUK\n2P/IZsmSJTzzzDN0dXXR1dXFM888w5IlS/bKY8KECfz5z38GYNmyZcyYMQOAjRs34q/1++VqWRaV\nlZXDLouxJlA2AQFHIdWlCpGQQtrqP4JJWx6RkEJ16cE3/fr6+kHNlf/mb/6Gzs5O5syZw49+9KPC\n1FQxA48cGC5XXnklL7zwwl4GB0Pxj//4j8ybN4+5c+dy6qmn0tjYyBe/+EUmTZrE/PnzaWxs5Be/\n+AWGYXDvvfeydOlS5s2bh6IoXHnllYOm+dBDD/HTn/6UxsZG5syZw29/+9thyVJRUcENN9zAiSee\nyIknnsi//uu/UlFRAcAXv/jFwrHY99xzD9/61rdobGzkuuuu4+677wbg8ccfZ+7cuSxYsICvfe1r\nPPLII0eUkYDo05RjJoAQKvAGsENKuU9zlIULF8rgnPKADzPvv/8+s2fPHlbcZMY3BsjmPASiMKI5\ndVaYRDR4zzxaGayOCCHelFIuHCORgMPDGu2bwPtA6VgLEhBwNJGIKixuNArGAn0jmoOxQgsIOFDG\n9PVGCFEPnAf8z1jKERBwtKKpgvHlKlNrdcaXq4GiCRgzxnosfTvwbeDgTWMCAgICAg5bxkzZCCHO\nB3ZLKd/cT7wvCSHeEEK80dbW9gFJFxAQEBAwmozlyOY04BNCiCbgl8CZQoifD4wkpbxbSrlQSrmw\nurr6g5YxICAgIGAUGDNlI6X8ZyllvZRyCvBZYJmU8vP7uSwgICAg4AhkrNdsAgICjjBUVWXBggXM\nnTuXCy64oOBzrKmpqeAss+8vl8uNsbT75s0332TevHlMnz6dq666isG2giSTSS644ILCvpp7770X\n8I8COOWUU5gzZw7z58/nkUce+aDFP6I4HEyfkVIuB5aPsRgBAWPG7be/Qltbesjw6uoYV1/9kQNK\nW0rJ+myWYyKRUdkEGIlECi5jLr30Uu644w6uv/56AKZNmzYidzIHi+M4BR9nB8JXvvIV7rnnHk4+\n+WTOPfdcnnrqKc4555x+ce644w6OPfZYfve739HW1sbMmTO5+OKLiUajPPDAA8yYMYOdO3dywgkn\nsGTJEsrKyg72to5KDgtlExBwpHCgSmF/17322g6WLp0zZPjWrd0jE7SILabJfbt28aXx42mIRA44\nncE45ZRTWL169bDjv/fee1x++eXkcjk8z+Pxxx9nxowZPPDAA/zgBz9ACMH8+fN58MEHaWpq4oor\nrqC9vZ3q6mruvfdeJk2axGWXXYZhGLz99tucdtpp/Nu//Rvf+MY3ePfdd7FtmxtvvJFPfvKT+5Wl\npaWFnp4ePvIR/3ldcsklPPHEE3spGyEEvb29SClJpVJUVFSgaVo/zwgTJkygpqaGtra2QNkMQaBs\nAgJGQFtbmsmTh+5MhlIK+7tu+fImnntuM6nU4NNOyaTJ7be/wpIliRHJK6Xk2c5OXCl5pquLLxnG\nqLk4cV2X559/vp9X4k2bNrFgwQLA97h8xx139Lvmrrvu4pvf/CYXX3wxuVwO13V57733uPnmm3n5\n5Zepqqqis7MTgG984xtceumlXHrppfzsZz/jqquu4oknngB8J5Yvv/wyqqpy3XXXceaZZ/Kzn/2M\n7u5uTjrpJM4++2y2b99e8NY8kOXLl7Njxw7q6+sLvw3l8v/rX/86n/jEJ5gwYQK9vb088sgjKEr/\nFYjXXnuNXC7HtGnTDqAkPxwEyibgiOJgpptsx+OVtjTjCRMz1APaTS+BrNRxpEATkjA2ygH23cXK\nZevWJLt2pYhE9EK4YWjMmlVV+O7f98iUzRbTpMmymB6J+J9N86BHN9lstnCI2ezZs1m8eHEhbH/T\naKeccgq33HIL27dv58ILL2TGjBksW7aMpUuXUlXl32ufv7AVK1bw61//GoAvfOELfPvb3y6ks3Tp\nUlTVP6fnmWee4X//938LB7aZpsm2bduYPXv2qEzpPf300yxYsIBly5axadMmFi9ezKJFiygt9Z2e\ntLS08IUvfIH7779/LyUUsIdA2QQcURzoyCKZ8Xh8TZLfZ1tZotZSrYRG7CcsmfHIxGto92K+pzEp\nUPGoVlLowh3xvaRSOSoq/I4/HFbJZGyyWacQbtsupul/z2ZtbNvl85+fMOz0+0Y1iXynnFDVURnd\n9K3ZZDIZlixZwh133DGoQ87BuOiiizj55JN58sknOffcc/nJT35yQDL0ufwH/z4ff/xxZs6c2S/O\nunXr9jmyqaurY/v27YXfhnL5f++993LttdcihGD69Ok0NDSwdu1aTjrpJHp6ejjvvPO45ZZbCtNx\nAYMTqOGAox7HlfzlfZO3nSShkGBDqIfKuAL4p1k67v6d0RZOvpQQFg4h4RIWviJo8+J4o+DP1nE8\nwmGt8KfrKrFYiFgsRCSik8nYI0qvb1RTofujpQpdpyk/uhkNotEo//3f/82tt95acNe/PzZv3szU\nqVO56qqr+OQnP8nq1as588wzefTRRwtnzfRNo5166qn88pe/BHxPy4sWLRo0zSVLlvDDH/6wYEn2\n9ttvAzBz5sxBXf6vXLmSsrIyxo8fT2lpKa+88gpSSh544IFB13omTZrE888/D0Brayvr1q1j6tSp\n5HI5Pv3pT3PJJZcUzqwJGJpgZBNw1NPW49FkZelSLepVg12uyS7PYnzYoL3Xpa3HY3y5ut80sjmP\nrVs62L6t/zuaGonw1vYdOOk0tj3yEc6hYOCopo/RGt30cdxxxzF//nwefvjhIZVBMb/61a948MEH\n0XWdcePGcd1111FRUcH111/PGWecgaqqHHfccdx333388Ic/5PLLL+f73/9+wUBgMG644Qauvvpq\n5s+fj+d5NDQ0FI6b3h8//vGPueyyy8hms5xzzjkF44C77roL8I80uOGGG7jsssuYN28eUkq+973v\nUVVVxc9//nNeeOEFOjo6uO+++wC47777CmtWAf0Z8yMGRkJwxMCHj9tvf4Unn1xfeKvfujVJOLyn\nAw2FNObNq+Hss6fmw7u55Zaz+qWxsSXHrZt24oZcShWNHs+hVNG4wKihMyVpbNCZWqszEMeVtPV4\nZCyP9l6P7e0uP7trBVVVUSSSnpikNC1QQgZuVwcy1cuWLV385S9/t1da11///F7Tf088sbYwjbZi\nRTM9PRbxeLgQblkOuq5i2y627RIOazz66F8zadKeRWhNU6itje+V35Zsltu2b99L2QAkXZf/W18/\n6pZpAYcHwREDAQEjoK+j39EtiZSVUjNO0lsq6E5axGOhQrx0OjekBVcf7SJHqzSZqvida6miFUY3\nOvqgp1YWnwWjIOgxXVq7PRTdbzKpKGya6FH2pkm826Vt3S7M7l66u02uv/75QjrD3R8TCmm4nokW\nj6GFdRzLJp3pwrZddF0lGg2RSlnkci6ZjI1luUgp8TxJV1cWAEURxPJlk8XlM9XVdHaZuO4AP7cC\nUm1Zdqj2kMoqIGC0CZRNwGFHcUefM0qJ1GtkSz02VXZjv6vDCJYcpJS85SRJaDqW7ZFWXSoUnajQ\neCnTzRK1GteTbNplEw3vOcHy5bUWIKkqUfE8CVKw2fQoa6gnu7uNLSUWlgO7xykYm2ysnhQAJSXh\nfiOYPoOF6urYXsYLyeSeG2k8oZ5WJ0ZJWRSkBCEw0yZta5sIKb6yWL/eRFEEqqoghIuqKnieJBTy\nm7HreoRCeWOAHBxbVsqOdE/ht8HI5Q6Pab+Ao59A2QQcVhQW4vMdveraONkMO6fqEI2Rnh2n7O00\nw11taDJN1mYzVFWorG3Pst5KM02JERUqWSy2ezlyG0BB4OVPspw2TiOb86gqUclYHi+0pKBbY7cw\nCZfG8GpCqGWdTLcFW+gm1tFL4/xaADo7s/3yl0BLl8sFnz2hoMz6zK37ptY8Cbu8BO0vbSXV2Vu4\n1hMqoePxGgwHAAAgAElEQVQnkV6xATyJqipIyaAuVUYLz5OYtsT1QFUgrAtcDzwpUYRAVzmijiL+\nsHE4L4sEyibgA2G4+2P6FuKrSva8jadj0OpkibVpZMrDbEy1ou72p84cx6O728S2XU46qY7q6li/\ndEtUlc/kvYX/WmlHS3lUqyrnlFaxcZdDmaZQFSnKy/J4bYNFNCTwPMmrrWn+7LUzI1LKFqOHCd0G\nrbEsYU9BInHbumipUZnexV4K0JYqmXgNK9aZ/ZRZn7l132jH0cKYkQhhDSIle9ZR7HKF1gaYFZlK\nrD3NihXNtLVZVFb2IGWEvXM8MKSUWJ6H8ATdaYmX77B8xeYrHFUB8BVOzCA4hO0wREpJR0cHhmGM\ntSiDEiibgA+EofbH9G1sTCZN2trS5PQoOaMU1fUNAjY3ddM6AxTTQ1UVyowwnDme6td6EfhrNpMm\nJTjuuHF7GQYAVIVCVIVCbM5mCakK59RWsMU0mWAYSOinaABWvLSVTlNBAMLNsXKCJGV4/EVrIdqr\nkoxZhHRI2CHAI+LqpMsF6bhCPLVnbcSTvkl0c3MrLz2/ofC7KxR++WePCe0dKPhK9oLPnsA7TTaJ\nqCDVawEgkbxTm8O2LTaVSxLtNnpI5amndgOQSKgoikBK0AwVxSO/fuMbOriuR0+PQXe3iaoOvcPB\ndT06k2F6HQfhKChS0DdwKV7qMXQBSEwpCSMwQgrFA5yeHsufbhwCRRGUloaHDA8YHQzD6OcV4XAi\nUDYBY0rxxsbJk8vISp12L0JY+J3m2zs7yMRU9DaBExG4vZJUXCeTM1FbLWzbJZu1C+5c+hbji70F\nRMMKT2X6b258tquDObJyL3l6e0282giluRBJXZCJZol4Kq1GlsqUwbZ4inrP7zQ9AFVBy3m01GlM\nX7fHUMFCx0XBTJtUVO45h6lTd9iYyDKptIIqR2V7S5q0JXE8ydlnNRTi7XRN2rKtyPdSbK016S6J\nwYZOVqzYyYoVO2ltTaHrKma5RvjCek5uU8ltS/OpT80C9ljlLV78ALq+jzUb22XpfWexojVN906F\n88K1CATZnEdbj4ehC3qzHvMmh2grSXF3upnPinq+OLm6n7n4YNZ2xQxmJRjw4SJQNgGHFWFsVDwc\nqaAKl7YJEHIV9KoypIRoWZxQGJRPJuCxjaQ6fRcviYRRmKYb6C3AxePPpDh1gm91VaHrvONlqJQW\nlUT75Z8Me6wrS3NG0qQpItA9hY5wjpCnstswyXkOrQq4Ngjp0a06aI5Hd1Qh4uaoifvWYI4UCPa8\n6UsJNgprIzY2gtcjKguTMXKGyvqdObZ3eKiKoLJEQUrJ67ludE9FSpce4aBW69CYILoqgwB0XSUU\n1kgfV4btSVrqNCq27V2emYxNQ8PQ1mbvd/fSZFlUeWG2qhnayVFNGNctmqQTkM7Z/C7biislz3q7\n+ZxZDux7b1JAQDGBsgk4LJDALt2j1hZUKynavDitmsCsDWFoEVKWQ6qzB4kLFsiExoTjp5B6/j0M\nY081HugtYL2SRCKJ2ArvNeeYWKlihBRqIhprenuYbIaJGyrPPb+F3l6T19Ve7HaFx3Z34YQU4olK\nUrWCiKVgx1xqrRDRCoM54QRzKiK8syPJuHF+Zz7urFrinj9lpQnfnQ2AKwUpadCp27TrkmguSrtu\n0hzNUNHpUFOqoQqXrW0OrqfShsUW16RejxCK2OR6JbotscaF6F6XJtThu65xakLI2jCh9hzp2TEi\niZF1/mveb2fLnDCvPNeEjkEmGuNpu5k57VEQGhgGdeNiIGGrlmWXZzFJM9iay/KOnWaaDLE+m2Wy\nGsYKl9DqxAgrHiVk2cfMXcCHlEDZBBwWZBMqK0ptFiV1qhyXcUoSHZWFqRDT59ezbs1OOuwSwiEF\nHAfSksSUCHPOPRaZzRSswAZ6C9jopEm5HnpGw7RtWl0VXQVdFRhxSWsuh9kbIpnxcGtjOHYvic4M\nzVMMEp0evaE0mh1BoqC7GjlVMtEOEymFv59XxX2vxGhb55s9d+T/wFeembjAiBmkpIHEY3s8jZty\nyUmbtGXyTInLzBYT8cRaABw1RI9ho3y8nFmEqY+pdJaWU/OORSyqs1uCforKxNUZHFey+/hSnPFh\nUqt60FvTuBN1mrZ2I6BgKBGN6ntZyPXRrrkoEyLUJSJIDzyp0RN3cFyFspxKS5fJVkcicflzqBlT\n99jm2Ji6x7/seJs3ug26T6+loilBOj4OFxXPE6iUMIkuosrI3OsEHN0EyibgkNK3OTOnR8lKfVAv\nyRJomxymTkjWxBwWJXUUIaj2XOp6BImeCDpxxrsRQo6G9By6ZQ9hGzxVZe3adrq6siSTJpb+Km9N\nCpFTbHY5Cj26h2eEWZSowANqNIWymIppe2AKPjYnipLTePmpJC1xDXddF1ZUxw5pdFYr2DqobhZX\nZinRIpjCZVxdAtXI0aXkCmtEUkrWpDKU22GyOUk0rBDW4cY71pNDJaXbdGkeuq0gVQcrKnFC0DE7\nROuGFJbpoUYM3mjbiVXZTchVMFWX3TGPTNYiFNMoGRejqiLOomPC/JWARypNNkRdptYm+OpZs9hi\nmvz9gDNrTjqpbtC1FIlk6+ZmjIj/MIQCyU3bsWsqWSVTTO+IYZkeQlNplm20aw6JrEooplIiJB1l\nOs9aJvEdOcJaBw2uICR8KyhHqmzzypkud2MrvodsRwvjuDKwYvsQEyibgEPGwM2Z7V5kUC/JqRIF\ns0ylxlbo0CQdmqTKEbhSsjsRQVNM1lX0UNelUelIUlGPzeMEM7ZCpN0lWapgmP66TWRSBKvUo8o2\nIAwGCjs9lxo1TMTRmRbWSYQEhKC910XJaYwvV8loWXr1CCHTQwlLqlsdUnGV2h05wpYkk86xYEGM\nzg6LpadWUqZplBS5glndleGWtdv5a6oZrxgFM2fd6iVUYrAh3kOJJ0mmM3hlBpYmieYU2ipdOja7\n1MbCCCNMbX0dCScGeKwzTCoUh17VptxWiHoCTcLj23aRMh2aTonjOoL3oy6PP7GWnAYrnS0sEAb/\nfuWpCOGvGv3p5V1kchJpOziZDEhJJqHSPFVFuvBWcydTGsroxCKSaqM7FqFpaysdq9vQ7DQ7LqnD\n7dTo7LExDRWQ2AmNZHWY4x3BeqMHIx6lBF/ZaMIlK3W2yUp0z0MgMSMqz64yR+RlO+DoIlA2AaPC\nwH00/jRSDUhQpEtrSy+qW4InVLqFSjTVgQC6kybJRJRYfn3D8CiMbnbqKmtiGbqNJMJU2JGwiO4S\nbKu2cBBsLbWIpZO0nVBC/co0EtgUNYm4YTwEChJPCkKe4B2vm9PUauLGnjfrPqsrKRW2VPh5A+g2\n6LYkZLmELcn0dTm6OrOcMTXM1p1Zzqvac8YM+JZv/7O+DQ/JplAvxxoRhFBIWx52uARLt+kM2xiu\noFe3SZUKXEWieAJb89gxPcS43QlEOIK2o4XplkaX7iBVD1OVaOEwXZqkJCeQAnZVKuQqoriGQsgG\nK67RqmtMsFW6NMnTSYsvmSYVMkwmXoNXblJuqKDpICVuRxtRuxfxbhvWcWW0v7ab0rVpstuSeGG/\nS3j/vS7MVpNJ50/CrNFQLYmbEOhR30owbQAaNIk0KoL3qtJMb4+hoIAEBwUHQTzvGVt1c/R52V7c\naAQjnA8hgbIJGBUG7qPxTZhjBTf8QMEst73X5ZSZxzK+XOVffvwyyyJZyrKCzg5/baE1IilthY01\nYdzSEGuy7UzMRGktEbQ7kIlBqanRIZP0zDIIA/HFleTWZmgJeYQ9i6SiI1HwpIPtKOxUTOJVbr+F\na5kffTSZJu1xQbUi8SpDtPe5cHEoWJnZSZOtW7v32jQK8GZ7hh22ydRIZI9HadUgFlaIRjU6enJM\nyqoo0iXZ4WLGbRIiglBUEqqBUw5pN0ppLkS0bgI5zyHkCaZkQphxEzoExniFelOhzFVoWZtlw9kJ\n9JxvOOA5sMvwOLNb5d2YIAc80txO3c4yXEXFs21ERQlCUfGAzIw6oq1JFCcFnTkyKvS+0k5ySxdK\nfo7Tslxs22PD67ugxEV050CCVhnDiat4kw0SpTHsOsk4wmyNpng7lGRmp29c4OhgWVk6bZMtTb6b\nnmf/uB5XDbH8t51ojlUov+H6jws4sgmUTcABUzyaWb68iURiz85lvSxBSf04Zk0t3eu64hHFxE9O\n5hOOQ4lQWdNpEXFUbM2j4nSByNiYXSbtnUm0qEu4w2Z9tUUiKXEcBylcemeUUbZRkkxoVEwtZXwy\nRqnI4CIwpT/N1ZmGk6dWUR7aU93Tlkck5LuP6XZVjt0lqR2nMbemai95x51VS8eWnkH3iUgpebar\nk1i+KUWFxuu5JBcYYYQQnHxiHdPGq+zo8MhYLne8v5byiErUFWTxzaQ7etPsMLqJZ8uQUtJFjCq7\nh0xcUm3pKFmXSIfHNsukpD2OV5nACSkYSV+RxxQFMwY7QgKkoMwu4aWdKY7tCqPG4hjjItg5E88x\nSUVcNlXkqM1IIhW1WG82w4QIepdA2dpNONznZ03iOB4aIKI6xstdOM0ZJh4raDu5BKXJxJusUFpf\nieNJSgydNVOynDmpDtmt0ZGSzKwbT0lEQ7KBykrfxNySKuVlCeLKnj1JQx14F3B0ESibgANCSsna\ndJrjJicQCBIJw9+cKQTCiCCjBs1VkhnSQxX95+hzrsOLLSZPbLd4g14qQiord1o0iwwT3CiGVNiu\nZzihOkpGuIRdhbaQQ3VlmI64RU3YwHAUrKiKo7kQMakvi6CXKozvCpOhlFJsyoREIjBSWaaLEpy0\npAOvMKI5dVYYTRVUqSHqkzC5bOjm0DHE71tMk91ejpjwFVuxR+nxqoFEUl2qcWy9wpttGbo3ZahS\nQ2QUhZQUqMLDUVy6DEilXSI58BDs0gXduku5reKVGZyysJpXWtLI8Qla9C0YqiRSFiWEgyIkadfj\nufIcjd0lCDwSikaT0ctfNVST3dBLOGRgYLGqvJeIrrN7bo7IBh2nPI6ZzZGcEUH+ZcAzBrwFZagK\n5OYnEM0ZcuUamZoQtuNhl2lUlSu0dDkkPYe04vA7q5WFyfFoimDrbg8pc2RjVeSkRUi4NDUlWbV5\nG056z5RrMrnHU3Ywyjl6CZTNh4jh+icbDjfcuYLf2z2s/VMb0aTLunXtlFaUUD1rCqqn0x2XbDgm\nx7guhWPNPfPzOzsdXtiR4c3SNhzhMSNXSmuviqiwGBcKU6ooTCZGj5fj/d0W9dUqqRZBKuSRNSQh\nV2F3yKEOyKiSiCfoKYOop9Clu7TqCnFbQRWyYISgeB4hFRqnhMm5sjCiKV43GMwr88CyGUjfAWXj\nohqtvYKcIwlpojC6OVPo/fKaVqEzb6dN7bgwna5BL1F0XFrbJbquoktACCSSTTGLsCvwECA91u+w\nMYTC6yUd9EoHTSqk8lOUmvBwgawKXSGPSulRpumsU7Lsdkz/CGugU/dI6g5hV2VX1CWiAIpAdtv0\nTgnjVIVwuvw0PU/CeANZa6DusnBrwii1YRTLo2J1it3TDaKtFgsXxemI2LzW7aJ4Gq1OlpIaGyMb\nAXzrsy6gU0aplGlatnfRuq4FWeTaJpu1efvtXYB/FHagbI5OAmXzIWIo/2R9DHc6Q0rJm2TRIyrZ\nY+PUrcsRjYWom+cf6mVbFpsngetorCpLMq5lHFLNsStps3qbza6yFL3hHAKwpUvUDmPlBDP0EjoV\ni93SoloJs8FL0WMJbExcRaNL8yhzVHo0l4ym4qL602VRyKgeuquyJZZlfreGKwV6Xpco0sVyJKoq\nmFo1eJU/kA6uyTR5L5MhoaqEy2F7h4OdlQhgBxZzIqVcODtRUGpVoRDHhfx9OWVhgR4vQ3UtrDYF\nPaQgFBM1HiOpO7SHc8Q8gaU4WFoOT1rEdJWMcGloDxNT/YV6B4UoWZpiNuU5BcNRAN/NTEyorFV6\nqSCERNIUy6C7vkcEw4VcjYrRrKKVhqkpj7HxI5WUvdKDADq7suQWliPTNp4nkSkbe34Ca72J67k4\nMwxKc4KcKdmUNkmEdIQr6BY51kR7WKxH6egBy/aViouChcaO1Rspje59UF2fy6ItW7pG/BwCjgwC\nZRMwYraYJskIxJpd0rU66bhCuKwUo7IMz3HpSphkE1nKUpCqztEayVCT9AjrKpaRodOwUIQACc0i\ni1aaI+QomDZ4IVgrepgooyS8EHGpEMuBHoYQgllpgRA6m8IS1dVQ8DCzCp1hCwUNS3fZ7epUpnJk\nPP8tPV4SLqwTjaaLlWKP0gBuDXSnJZbtEdYVTqqKkTD6TyH2KTUr53Hn0ylAYhhdmKa/AdJTPGTG\n45jeLJr0UFwbtbwSw06QCKnMlQnaWrqpMvzOOSs1NpV0kSzrZEoqjCs1uqPQ4ljESmBnykQPqai6\nJKm7GJ4gq+fQPEHacJHlEiWs4WgCb3KUzJoUWlsOp0pHjDPQOmxChgY5sCdFmVFThWZLdnZC/bET\n0bpC4KlUCZ2U6THJDtFr2PRoNuPKQ1g2bHi/Fy0cZlfTNpqrPOLrUhSXymHsFT9gFAmUTcCI6Js6\nCtv+27qWk7RMiTG1ehaRkhiO67BzWjtRRWC7FmTg1Wg3x7Up/PHPu3i3NkuT2kMEhXhJiG7VIqO7\nTOkppS2dxhU2MuKgdttMdg2MbCdtq3Zz5pm+k8pxOb+bOkb36PT8qa10SmXR8VNIm5Kmdpsp06Oc\nVG8ULKvAt4Ab7ETOg6HPo3Q/yod3bTik8DenRHl8RYbKCWX+nhjpT/H9zSlRJlZNLcRdvTXHk29k\nqJZ+c3U0k3QqhwSskMbaSC96zqPHs5jREUXXDcYnS2ioUSEMa9Z38W44h5Z1wPOocFSElKRaesmk\nHcZtsphgh0lvyFBeEUMzIpizwuSyLrquEI35IxHTgY1lkonvZJk6axqVvRrS8DjGi1IiVDbZORQT\n4mFYI5KcTjWRkMDOWBiRMOk6sE6oJ/7rVsIb9kznplIWAUc/gbIJGBFbTJNHX9zMtjXt9CQtwt0m\nsYUNJLI5SGu0xFIkdYsSU0dNxJG2TW/EJlWlUptwaC0x6Ym4hNIQRqVVtbAVj5zuUaIoeJpCrWfg\nxDRKN0naXm1i18oW3t3tr7+8WyRLSUWc2afNoSsrmGDFcKUkhMvkcrWfoim2PDucmFil8ZUlcda1\nOPSkXUpjKjPHa75LniJmjtdYFlJImS5xQ2XWLN9irqPXYbXdQ7hW40Qtzk7X5LRJVUTbYyTTLoke\njZghiJ2s40TbiBHFdiW6KoiEBW+8bvOO7TI+JNHWplDW9ZCL6KQqdWiIEuqwMKIaFbUlAGRNm5rZ\nlZx4TBRpJdiZ7iLpZIkKlTbXYoOewdDDOLZKMuTQLn2nnkpYx3VtNtRZSCD7sVqqe5N4OZtsd+/A\nYgk4SgmUTcCwkVJy7WNvsmFNK5mkhWk6hMtK0RyVHfEU8R6drRMtQq7fWQpFIRQJIVWblmMleqaH\nrlAOISEdkli42MIjp3jsjKQpszU6jRzVZhhFs2DrdsaVhSkpGfwclO1b2pk2eSsTqstobPAX48M6\nvLo+R3uvi8gvjRdbnh1uhEMK8yeH9hunbxTU1uMURkGu9Nhc00VM8ZtxRGg8bbdx7YRSdnYpNNQo\nTBsfQkR0jjH3vvfzPl7BI796D3WmSdT2F+czGRs7pBBel6W9LQOAqPbNlMvDGhO1GB22QFVDlJRE\nmWqXo6nwttLNeDVMKK5R31OCyEHGgzbPQVUUNrSvp3tqmLLuCLl6FeUjMSpbBY6Vo3nlxlEu1YDD\nkTFTNkKIicADQC2+leXdUsr/Git5AvZPk2myRebQSkLgeSi2jhgfQ4YE2XLYZHWRjLnoroKruQhV\nguvhZXN0hKBVszA1gSpVUrrLLs9CRUHBJY5FVDgotiAhLU5MwaquNETCNDSUFTaEFjPUGSmLG43C\niZ+DWZ4diQw2Cnrf66Fzm8VkxV+/KVc0droma7xe6ow408aH8mfOqIwzBldop33ptBHLsn6nzW9f\ny6Cp0Ja0UICusGROaSnNepZLFiYodcMFObvf2snGvy6nUlHp7c0Sykl2HysZ1+YRTxhET50NJIPF\nm6OcsRzZOMC3pJRvCSFKgDeFEM9KKdeMoUwB+6BEVZm9S7Kq3UZss6mZVkc4FyOcKUV6Hk7Exd3Y\nC0gUVSVSGiXd3kUuZePokKqQ1IfjqJ6K6VhMnRDl5JJSXn9xE7WVEXaEJMdYgg7No9LTCWU9iOxX\nrL3QVFHoZI8mikdBnufxH++3Eld0HBe0/K1GhMav061cY8QPybSh40o27LTRVcHyd7fyflUvmiNQ\nFYUON03aTHHp4+9Q+3oPtTUxfvObz/L/yqBNl8RzKr3SQ8tCulSwu9KlvMVFNQx6sgInnSE6iKVa\nwNHBmCkbKWUL0JL/3CuEeB+oAwJlc4g4kL0kxVSFQsRzkKoJMTFXib7TQ+xMUTN/PEIIHNtlvCjF\nTGVxLIuS8jhdmzpxLUGmBDaFYEbG30nudKVZcEKcvxtfi/fIJrZVCUrzXmL6/KNJ/CFvqkLD3555\nZI9ORpOXkknWZDJEIyqtGQvX8Q87k0CHyCHqLTR138/zQGjr8bAcyaw6jQe3ZpExjW7DYXJHCKEp\naGmP1JQ4NabO7nc68DyPt+sg5PpPLxrVcT2JlNC+QKXBAk/ozJhaS8jO7LcOBhy5HBZrNkKIKcBx\nwKuDhH0J+BLApEmTPlC5jjYOdrOclJItFSBMHWu2waTNEjVRDjkTJRrF8xQURSFaGsHJ6ex6Zx29\nx5ZQ2xZmR30OI7/JMi4skvjHMz/T1UVXBDo0SaXjK5OYJ+jQJNmESqpEYcekSMETdD95gJYul4zl\nEQ0fHdNlw6U2FOLK8eMB/3jqVNZ3CKprCvEITBtinetgyVgeCoJuLUeX2wsyjCM9TJnFMFXA8y0U\n6zTUd3yl2B4XxBRJVnEQlRoCMABLkVSckiDWovIPX57Z75jpgKOPMVc2Qog48DhwtZSyZ2C4lPJu\n4G6AhQsXBpO6Y0hhf007pBIe6VKdUkUFO4eXtOlJOYSiBm46i2poZGcnWDfLxA2HSJdDensvrq6T\nFNAThrTrstOyWFMLFV7/vAwPtk0Ok4lpeIpX8ATdN7qxpUomXsOKdSYKouDS/3B2Ye95Hk93dbGk\nvBxFOTgZZ8ZizIx98KOAaFjBlR6v57oBQSbkIlxJa6lNZRqkJ5G2wIyqxMo1akMhFnXrpHpz5MIl\nIEHgv3RIoN2yqIkemim/gMOLMVU2QggdX9E8JKX89VjKErBvivfXpF3v/2fvzaMku+77vs+9b6+9\nq5fpnn3FQggDAgRJkABJiaRISpQsxtQx48hychSLlmQ7oZI4SsxEJ5HDOJIjG1aUoyPJkqwcbT4S\nbVMLJYogCILYCIDCMgBmAWame7qnt+qu7lrfeu/NHzU90z1rz94zUx8cnDP9qvq9269e3d/9/e7v\n9/3hKsH0kKK4pE+FbzSLtKm2Fa7rgDAc3dqrNh+vdtl+UFKIfYaORQhgb8XnU9UqM0nCS5VZas2I\n2lnXnB+2CASMNMWaPjfaQE0XwIQMFc+shjux3tAS9k83GvzC+Dh5KfnwwDoLcjYYwyVJw0l5/lgN\nVU9wVW8KSQsScULRPjCF0eC5FlasuTuf50/+/oeAtf2NBIIZHfIUC3zu3q0b8vPqc225mdloAvgt\n4KAx5l/drHHcKVytLtrxKGI8jgkycKRGhNAoQMPVBF1FO1DM3gWFWZtcatEYTskqkqGOpFXVDA04\nBNOCX/zJ968JlywkCWPfv/+c6xlj+Pm/PkDSzij5EC5GPJOFvPskKNsjCgLKhbWPb96TLLQUtabe\ncCEZrTW/OT2NAn59ZobHyuWr9m5uJCsdVzuRYirXQoSanOvjKgvbc4mVIbUbDM0bhNZ0OiFO2Ttv\nnyNleyghOTCWkVYEL+wssr+Sozcl9LlduZmezaPAjwMHhBCvnjr2z4wxX72JY7ptuRxdtJUJ4sUX\nT9Ltphhg7r1FskAS1VPKY8PklcbFZ3HMcJc3wGS1iW0Ui5tSNrVLHB6bxTnVgt5J4e0diu85Kc+R\njDlvFT5wLAx5/3u3sss/07Zgpe2xbtgcGE+pFs+drK+HLM2FWJmA17Nn9HSjwbEo4q4g4GgY8kyj\ncct4N42u5tmDEUfjLo6RfEe3SaWhJSPKQ0MoKTDakO2pMvwDY7SePUSnswBc+LmrORpZAn8uYjzq\n/b+6nfVqrqWAbJ+bx83MRnsG+ulFG5GVCeKVV2bZtatAuyCp7/UIUijvLpJmmvKQRxgm5LYHtB1F\n4htKM9DNK47JJk0noRg6ZCrGDmE5r2mO2uuSjFkJ2ZWttQZjJaHgh/0RNOffvlsp4rzerA4JXWrP\naMWrWfl7ypZ1y3g3mep115w1Ed+x6nzEq/KDZphvLkWUg+2o+Ridpqffb2eCsYfuY3nu2xc8p8Fw\nMJfha4jp3Y+f+5PvsvdQfN4J4amnxtm6tcTHPrb7PK/2++HcKtz0BIE+GxNtwM7nEYU8tq0YnM/I\npw5W4hI2u+wLPFoNw4P7Bnm2vEzV2KiijaVCDo82CJSF41p4tocwGmEbFu6XDBYufe3Vasodpda8\nNp0kfLScELi9tst578xkfaNkaVYmYDDr2jNa8Wr2nFq5D7nuLePd9Dw3xZtWk26qeTlp8WlvhNEw\nRyDytOZSsvjM/VbE6EoBWR5gpHpmQfDEE8dot3tKBN2yxeQDeYJGRhj2POcDUYcH9mxiODvXIy2X\nfVqt5JzjfW4t+samzzkoaTOrywRbNZZvYQJDc3PIyJGM8rJNe0myY5tHva0oGptQaqrGY2Cny7Lt\ncTCqEWQ2sQbXtrGkoCABz/BCu3nJCfZsNeWzGXBsPniPzXOHYhZaCgycNCF7vByP3nv9ZWlW1AlW\nGxo4/57R2V7NCreKd9ONNVNpzBvtkLLyOCYiXmp0ydw8ec9lZG8VW6xNJYyNzZbvv4vH/5cH+eIX\nv886X0UAACAASURBVIE2EBmHwe0VjFIs7lAUpcA75QG6Iz7NZsRLxYwfWJL9eqrblL6xuQMxGOYc\nw6ZUnPPFzpQhylXxARWGaGlzclSjBczuDChNrT3Pm7TwjIUBLCnISYv77CK+I9ma5hktWVSLNoUA\nJL36kEtxoX2cNbhnZGkOt7t8s7XMp3bkbkja80qtyfk4e8/omUaDN7pdSpZFJ4rWvHem09nw3o1r\nw7daS3i2hW8LStgct1qkbkaJcz+j2dk22vaZe32KL36xzrdfmGG0ERAGAySJJC0LmlvAanTIbE3X\nsnjHzzAdwcFcxntb9nm9mz63Pn1jc4eh0Txfyph1FB9peOcUStaaGiOs06vVdg46OU2pLWkXFO3A\nRjq9x6bpaZZkSCIECTGetBAG9jg5GlnG9xaq/PgDlevmadiWYLQi+dNuk8ATPNlYZm8uuO5ZTTlP\nrnvPaHXx5flYj/G9mZxIYpZkypjoFYnmsTkhutSGu+xv5MilEpszno1C4giNHbbZtn0z7jgsNSKW\nai0sSxIvGfKygMBh8Z0J4rxk5L4qpYmIgUqRruyX0t2u9I3NHcaRQPEXAwm7YnlOoST0Vu2cmkgN\nhukhTbyUkHYNOnN4tVNnW83nwOEGTZ1QeNlik2fzrncNY5Q5rbTsOZKP7Mtd95DWSkr23iDo/fsi\nWU3XiuGSPL1nlHMFkypim+XTTcw5e0Y3q/jyWmCM4clGr+01MURZT0CobiW0fcV0p0apVSU5LZQj\nyFJN49BbmM0eETbCsrF1imVJHNuCRFGd0ziFHH6Y49g2yWDksNS1GI0lE4Fme9KXJrod6RubO4Th\n4TzHJ5b5s/0QGcFJpbC6CeU5KEfA1gBjDDlPspIk2Mn3vBq5ZLAtiaUt6kWYmR1nW3ErbjvjsXcV\nmRuf45/+xH03XGn57Ky1lWy1z/v+dfVubEvwwXs8njsU82azyzfUAh+3htnpBxu2lcGVMB5FjGcR\nqTQEJYPKoKlTEqOoBg5+xeHHd1fXKDz/4W+8zevFjCO7HNK/mSKOPMJmhyRWxDJDa8OigaFcHuf+\nbbg7UoT2kcUSU5OKZdVk6pUWucaZxJDDhxdw3X5o7Vanb2zuEL7whUf4Zr3Ovz98mH22TTPL2H/v\nAPtLJT5eqfBvZ2cZjyK2lXwKOYtaPWJ22JAsK9JEYVxJ2k3oLrRI9wYsnFykVHQJRIrg5igtr3g1\nK7U4Vce5Iu/mcuplVijnJB/f7/Gvji9SDQWtXJeP76zg2Bt3s389xInm8EzGclthAs2PjQ7zSpgC\nBi8neCapc5cJcLWFY0sO0OSntm9GiF448I+A8SGJl3No77KoZC4FDzoxOL5Nw8/INw0jO4c4XKmT\nTxMsNHEnolIqUXIqhPfBllVp0AMDAbOz7QumOPfFO28N+sbmDkFrzb+emsIArpT4UvJau82AZfH7\ncUIjVPz+eI2f2LSZ//4n9vKVQ03eCW1yWExMNPA9SU6k7BHbiSR8aMllKLt5E+ulanHW691cTr3M\n2UymMctWwnuHChyPIqbSmF329Q3hXS+01vzexAILb7jECUgh0MYQuBY/+O484zXF8ahLogSbRQ7X\nluwZtTmRxGuM+3IAkW0z1NS0i+DmDWMjYzhRRljQNLZFbJor0vENy36C0dCwFWlJsiQSpLBZKtmU\nyzbF1EPHMSOjMQMDPl/60sd4/PEX+Iu/OEK3m64Z/x//8ZtAT1X605++q1/kuQHpG5s7hG8tL/NG\np0PF7n3kBdtmIUl4uxsz0WzxMBVeIeQrS012+gGP7gkY7IwRp5qvvDjH2JCHEJJeTllPKPNmcqla\nnPV4N5dbL7OamxXCu148sbjEzx8d54fkZt5dOlPx344UX3s14ic/nuP1qZA9icOI51DwBZYUlNMz\nfzfA8SpYNQVSYqeGxVEYaWk6RcV8NUUDJysRW2sG9+0O3TBlwenipwrP1/jVAqMDYxS3uiwVUra2\nAnZV3oU9dwLoFRw7jsWuXecv2KrXw4uqDfS5efSNzR2A1prHp6ZQQGYM2anJWQPfXFqmikPXUVRt\nm6NWix3G4+hxyQ+eyiR7tZ6yo7ixQhWXqsUpWpcO511OvczZXE0IzxjDkTDkruD6Z86tB601jx+f\nIdOGl/1F9usS8tSiouBb1JoZ3zrZ5SQh5cCiSUJzVY3linE3QCMANzKQA187tIpwoj7P3C4LgSBf\nMyzlI/ZVSrw7GebNY1OUyi7NZorlKHZv3UI432G22OStzRHxGx2G2Uq7tJk4uckrnD5XRd/Y3Kas\nntCeaTQ4GoaULQtjIM4MaWbophBJTSn1aEagpKBZiGgVUpyusyEFLVdYVy3OJbicepnVXG0I73gU\n8e9mZ/n82NgVZc5da2P1dKPBiSRk2PjUiHmbDndTPP26EAIdwee2Xti4F6TkPywsUPVc5sJeiMsy\nFkjDxD5D4mfYKYhORpYYjo002bScEMeKTichSzXlsQpCCMJ2hxP7U5AwviPlg2GKERaHZ7Kr/lv7\n3Dz6xuY25XgU8dvTM/ztwihxLPnx6hZSpXn1eEqSaaLEMOG0KQiHMcenbNukyhC1BS/ayzxqhm+Y\noOXN4nLqZVZzNSG8FUOljLnisNvVGqvVnFE4sOkaQ4DNt+U8+3T+tHdjjGF7MWB/5cLG/XgY8ma3\ny4OPbGE5TClVPDJjEbUVYRncBCwbTE5hNTXLdCn6kuJgkVzg4CQK1/eIWiHNMUm3AKWuQ6OQ8JbT\nYVtYoNlRF7x+n41P39jchhhj+MrcIkfnM35ndp5PWiPsMmVeO57xSF7iOPANWaNkOYhYspRpKMRI\nS5Aow5Gky14ZE7i9ENHVtpPeqKyul7kcjbWrCeFdbV3QtTBWq1nRbdtZDHi7nWFpQU2e8W7akSJw\nJXePXXyqWH1PTiweZ5Njo42hnXeJHPAsgWUE7S0QZIq8Jalva9L42iSVSp76QovB7SPc/dEHObmn\niWsspGNhWzYvVJcpLXkUfElme9iVMs0Rh2KtizD9ItBbhQs+QUKI+4HfBLYAfwn8nDFm6dRrLxpj\n3ndjhtjncnm7E/LCdIfN0qNuxSRBilrueS7dGGatFi/nFtm3kKfc9QGJNRcjdYojJFJlvBpOEr14\nhJ/9wiO3bWbP6nqZhZY6XZC6ko12oeSAKw3hXYukgmtZxLpat82SsH3I4sSCwsokXzdzVDoeedfm\nsx/I4V1CSXv1PXnJzVM73GYpgHCnhXZsEiMwQLcoEZsA02Fxu0PxwSpi3sKNDCJX4ODIEm0/Jd+1\nyFAo1xAJzaulJX7r34dM1iSLlQIzuwRbQof2y+NInXHvvUNXdA/63Dgutlz5NeB/A14A/gHwjBDi\nbxljjgLODRhbnyvAGMN/nKljK4nrCnLa5qWkwd1xBVsIEqN43l1AANN+yP2NChkuNj4eGZmxKMiI\nzaUm0ydu/6yeck6e1li73gWpV1sXdK0z4M6n2+YNQD6FVhYztlPxY7srlzQ0Z9cp/eN/8n4sCb8x\nPc1gHNNINBNzGUmmmVxoklopllOgZGwKH6qw/ZiP63ksjCQ8u2WBQuJgWRaZZdC2ws0kU+WI55/t\nEL/ToP5um7RpcWJAMrxpE4efPkAcZ3iezaZNt6aXfSdwMWNTNMb81al//99CiO8CfyWE+HG4QKC7\nz03neBRxPAwpi956oCRtZlXEVjfDIJl0OtSthKHUZ9brMhNEjIYOeRIsFIFI2SKWuE2K4NfFjShI\nvRZ1QauNldIGK7V4pdHhRdnlPcOXLw10Kd22Rwcu7dFcqE5p8y7Nm90uBWlRq2tyjqDs29RPGmZt\nRZDLsbnrM62XqKcp5TTj4IMh2oWWjjBxSuwCQiCNQHmG+kMuY/lR5qsN7MWUzgAMDrsElQJhmLK8\nHFGt+rdsWPd256KBWCFE2RjTADDGfFMI8Vngy0D1Rgyuz+WxMqFVXZv6Ka0qgJywmQha5OwiL3kL\nBMbCcwROLDhQaTAc5nBQ2EIzLNt3lKG5UVxtXdBqYxUmhqOzPU+hYwS/05xn8eQmHr3XvyzV66vV\nbbtYndLBo4of3TPMcsdQMDHlXO/1eGmZdtHFTi201rjaYmFEEccJkaMod2y6gFtTWFULNxZIy0JK\nQSQS5ndCVZZIwyaWNmT3eowdrvCP/qt7mZhY5ktf+tgV/z19ri8XMza/CNxLL4wGgDHmdSHEx4D/\n9XoPrM/lszKhFRyLhsxoJAbnlOWom4SxMUEjThhWHqkwBMqm6SeooMZwIvBIkX1Dc1242rqg05+t\ntHhtrjfBO05PrnLBpMyZmOcOiYsWol5rLlanFLYs9pGna2u05VE95SF9N7UoFh2C1AffUFU+J/wm\n9UCTUy6BcnBMTLJZsaWeg2Yv3dmxJXUroz6YsW3Rx7gOKkwIh0AW+oGWW4ELGhtjzB9c4PgJ4Cev\n24j6XDGrJ7ROwXBgIiVONQKB0oq/sua5a8glyCyaoaHbTClpzWsDLd4z751Ode1z7bnauqCVz3ax\nvdZTWGHQtgk7+obWRq2nTml1erkxhrlREKFCiZ66uA6glTOkgWBTU4AFDoJGTrOkUoqWxdLJOoWB\nPJ0KZDaEUqM0GCHImgntrb1U9T4bm37q8y3A44+/cFEJjuHhPF/4wiPnTGgfHept3LYjxbcbDRbq\nMX5sMdFJUQYSH4SwWPIUb3iS/XGvHXSMQ2YEme2RKXPbqBjfyqx8tkejdI2nsJpF9Lpro7TWfG1p\niU8ODFxxp9D11CmtTi9v2gnB3hy7sZiYzzBAO6eIFAih0Q6QKlQU41mwFMQMH3OpjBt0vY18IEc+\nsVA5SVaxSUJFd6GOGfNZtPvGZqPTNza3ALVahx07Khd8/UI1MLYlyHmCV48rOpHgQ+kwJxYUYwLK\necnMQgPPs1FI2mlApFrUKaCQCAxRYPH116J1iVL2uTFcaSEqrM0aezVp8i9r4+SlvOJOoWfXKRlj\nmFQR1cxdk9W3kl4eduA9uorBcFcF3mlFPK9CxKn/2m6KSTKED6nd82Kag4bNr/qoVGE/H2E7No6r\nmDk0QX22gZQGy5I8eSgjnO3yxS9+48z4Ti3C+mwMLmlshBCPGmOevdSxPhuP1Ru4D5QLDNYDDnQT\nAlfAsmBL2yJe6EmLZJbDEZXDTkOk6W1gl4seYC4pStnnxnGlhairs8bQhn+TnaSO5lcnp3msXL4i\n7+bsOqU5HfN1Nc8PB5v42/eWTz8vZ9LLXX7n/ztOo95BZjG1LVAuuDS7KSLvYDoxQx0f8hYLIsVt\nWsiiR/5dFfymZsQIkijhzT9/kXBuCdNNsT2LQsFjdK9HvazXLMouVojc58azHs/m/wEeWsexPhuM\nszdwO5Gi7kRstwKS1LBz92DP8ADTSxnloMyDu71zznMpUco+N44rKUQ9O2vslaRBvROzFZ8Dy12e\nWm7w0erAmvevt7/PiiGZbyh+e36RUWURlbuUgrWe+Ep6eWe+zp4dFWqOhyil3JNBc6LLYDpIt5Jn\nZAqavsZZdnHxiW3NdKFN5a0WSTdDKcXWe3fwznKTKMqwbUml4lGvhxSLG7vF9p3OxRQEPgB8EBgW\nQvx3q14qcTsLZt1GnL2B2/FSXs8tUcws8rhk6kx8XymN752/VvdiopR9bjyXW4i6etGhtebPwjkC\nYWNL8DKLXz0xzfdWet7NlfT3sS1BFKR03Ix3+/lLFqoaDAdzGb6GtgVLWxzyrRhX56jtUnglh+HU\no5sYkqU2UcFQ2hkQLPUy06wgYMjZx9tvTLFr1wCf+cw91+bG9rmuXMyzcYHCqfcUVx1vAj96PQfV\n59pwdibQhN9CCMNB2eQ9DGKfSrdtR4qcZ3Hs8CyHXzo3EUFZLi/8ZZ2xAbsfA98gXE4h6upFxytJ\ni8k0ZpPwSDSUpMN4FPJMo8EHS5Ur6u9zucoGi7Zh2tV42nDC1ajAppaTVLsRc4WEPaKAjSLvSjJX\nIoTA3O/wQNNHIIiNxcDdPlnn9le4uJ24WOrzt4BvCSH+nTFmQgiRM8Z0b+DY+lwlq+P7DTthXsfc\nU8zx5nKXaRXjh9COBIEr+dxjAY//VkSlWsAWZ/qGZKa3mh0dCJg8ceEY+JW0Vu5zY1hZdHQixR/U\nZxFIQnoehtaGvGfx6zMz7NGFK+rvcykZntXPRmZ7uBoebtksWDBtweauTcfT+MrH1Zo4S1g0Gcry\nSfICy5YseCnLjkMlsZmY6vLmzDyTtYRwucV/+k+H1oynWHT52Md2X7f72efKWM+ezWYhxF/S83K2\nCyEeAP6hMeZnru/Q+lwtK/H9Zw9GfH2pzlIXPKWoujbt4Q4VE/DgLo+Hdrl4rsTv1qGyhdjYcEqB\nwKKnKnCxYs+raa3c5/ozXJJ4tuAvZpeY9kNy2CyToI3BSEMcSxqyw9PLDQJy5z3HhUKpl5Lh+Zx2\nef5wwtPPTtHtJEzUJMcXUzrT08w+ugntGrzI4AQC7St2HvXwfAfbcbA8D20AA8l8xMtHamwZHsFY\nDk6pyOi9u7GloSrakJ7p5lavh9f6Fva5BqzH2DwOfBL4UwBjzGtCiA9fi4sLIT4F/Bt6T/C/Ncb8\nX9fivLcbVyPxX85Jdu0zLL6aMBZ4+K7EdxzmdIQnMp5Y7vAesQkAS2eMygYxDsoILGEuqSpwNa2V\n+9wYbEuwb7PDt6ZtPpAMI+gtJWwJwyWLONPsqbhszXnULjOt+mIyPFNxjDsdMCo94maH4cEcJ8cz\ngkKAemgXSSWm0NbgQhwqOlaXgTDPjuImTKYg0QjbIurGQB5r7wDJ0jKduXk8WxA1O/hFn0WrwKBY\ngn67gQ3NuupsjDGTZ8Ver7qLkRDCAv5f4PuBKeAlIcSfGmPeutpz325cap8kU4apxYzpeoZBsGXA\nYnTAwrYExhj+dG6JvLAYKJyZLHLC5mu6xitRg++Z8fnPtvfk7qSAgJQLFIafw9W0Vu5z47AkPDJc\n4EN2kSQzuLag6AukFCy2NA+UHLYP2Xx9JrqstOqLyfAstg31RUH+rN8zKmV62MXVFs1mF6UMKtMk\nEbxT1YwmCWkzJI0S0BlaCupbBDvSAUQS4dmCfN4hl3dI4xR8n2YkyDq9KH+jETExsdwX5NxgrMfY\nTAohPggYIYQD/LfAwWtw7fcB7xhjjgEIIf4I+BGgb2wug0ZX8+TrEe/Mpr3YuBUzise+MZeP3u9T\nFzFvddqkCOZVfPr3jDE8nywhEfzuwjTH/8MhnnpqnHLZP+91CgWXffvO1V+90tbKfW4sOU+C4Lxh\nzRWv5UrSqleUDc7XqvpolHJApGve7/sWNVIWXXCUpOVoZCDQmSDzBe2tgvn5iPJiQieI8RqCxUHF\noR0J3ozPkHVmyhodLQAQaYv3312lIHuhtL4g58ZkPcbmp+iFurYAJ4G/Bv7RNbj2FmBy1c9TwPvP\nfpMQ4vPA5wG2b99+DS57+5ApwzNvRYzP90JdHTflDa/O8jGH8aOSp57sIOJ5VNHBeCVildDyDCOW\nR/mBMjYwKnym0ohXkpBy2adaPX+66mI9ZDHXM1KrvdyrqWjvc+NYbzHolfb3OV+r6vM9G/fcM0zH\nUoxgwIITYYrvCLphxnQRRFuwmIuQbsKJXZpd0wEndrYwGN4ZblI9dr6r90K+fTY2lzQ2xpgF4Mdu\nwFgudP3fAH4D4OGHH+4/UauoNTUzS4rFlkFIw6tBg3aseScX8ZAuEwRFRi3DPaTM6gJLfsZMpc2m\n8Yw3sxZV4dFajOlkHf7K6VA4vEA+cFhY6KJULyPNti0GBwOaecGBNIf1fzzJvuIZGZArrWjvc2O5\nHK9lPWnVqzPMAlfwV91zW1UPlySOhKOzGbGT58iJDt16E+HYCAPd6WlwK5D3yDzINgnKJxXdLTFz\nd3lYtmZ8V0ZYFFTiHE035rDuUpxNcNze2CzPIUs1z80s87GP7rret7HPVbAeuZpfOc/hBvCyMeYr\nV3Htk8C2VT9vPXWszzppher0Pk3kpbTclBHlM+G2qDuazSkoI5AShmSLl/OClmV4dUzRThSjIqCb\ndBj0fZZGUpwHB+BQG6U0jtP7MqepwgDhfQUGyx6NzTnmXz9T33ClrZX73HiuVVfSs7MPp3XIt2jz\nwc0FxlelPM8sKQ5PZ8w3FalXQFZzlAeHyOqLqOlJ3Jxk4vW3qWwaYPqBgO6kIu0kFLYUmCtEbGvk\nODTUwM96CxYrgdl7LTY18ni2DUKgk5SFQ+MEw+cqX/TZWKwnjOYD9wB/fOrnzwLHgQeEEN9njPnC\nFV77JWCfEGIXPSPznwP/xRWe644kSntZRZY0vOM08XTvS+koOFRqMbJYOB1eaLgZTT8mdAyzFcUY\nBYQAI21iY+FLj/QHRnnYb/P6q7Pk8z3pj04nYc+HNhPtsHh48wCLtuHsbZ3LncSyLONfTE3xP2/d\nim33tWBvJFfblfTs7ENjDN+OWuRSi6OzGZs2Sf56aYn/Urp8+fkutgX7Rh1enV+m20oQnoMMcijV\nUwMw2tC0I1qOhZiNcEerhJ4iSVNqliKVCksbonYX17FZzmWc1A3yb2eoJCVutul2EnKuOZ2x2U8M\n2Jis55u+H3jUmJ46oxDi14BvA48BB670wsaYTAjxj4Gv0Xvyf9sY8+aVnu9OxHcg50pOpCGLMmFQ\nn1rdCcmJQpuZrmRvkmIwvJVLadqGUGq6DiyTYAlD08lwHE2WwvKQR200WnMNA8xutrHi3uTgazg+\neO7ezeVMYr86Pc0vnThBRUr+SX8f7pbi7OzDGR0zp2JGHZ92pHGVw3gU8eRkhzDRDJd6U0zgSeZn\nI3SaYgU5OiIgri/QDVOi7S5eCtq2MAWb2NGgYbaSIGJDIgy21Xuy3EzQ+WCB/WkXiQvkqddDHnxw\ntJ8UsMFZj7EZoFfQ2Tj1cx6oGmOUECK+8K9dGmPMV4GvXs057jRW97bJbI9ObpADmxLiTFJXMULC\nUpAgjeaVwSUem3VYdARTniYTUMwsWkYzErg8bJcZry2Qy3tMTaW4OZdArxUzTAcdOkULd66XVZTX\ngrmA81aHr0c5IMsyHp+aQgG/PDXFT2/e3PdubiFWZx8aY3gpWSYnep+fQJBkhnJg8WS9zpA4I8Z5\nzz1DRFFKvuASG5u7B3cwYg/xB998m9n7y9gJRMMadW8ZLQwd15BaBicVaGFoSshniqSlmfYNR3OK\nwamkl36wNei3TrsFWM+3/JeAV4UQT9Grvvgw8H8KIfLAE9dxbH3Ow+reNtrAQUsQFhM8JVG2oKlT\n2naKh2QqSPluXhDa0LAMjgGpJQOppDRgU7Ec9s/7DA7m8Y61KQx4qO4Zz8QA7XvyjCSGbNUYvFSs\nqQ6/HOWAX52eZiHL2OQ41NKUX5ue5vOjWzk8k7HcVlQKFneP2Xj9LLYNyeoMs1kdM5GF5IRFpBRd\nY/C1TUEJpkWCEDFD55liDAJX9hJQKq5N9GoLEHjbhmkkCu2AlYFUAjeTdAONbwTbGz72yQbddsyn\nH9nBaGZRczTPlFIab/VVAzY6FzU2ohcn+Wt63sf7Th3+Z8aY6VP//qfXcWx9LsJKR82Kzri3uRKj\nNhxwEwY7gqyoKGXwfCmjoKFlQaAF2tIoLVjSGW9mbbZ4mkHA8yzCSBEudpidbSMEqBGXTjFPONvE\nsiXPH631rq00B9rt09Xh61UOWPFq/FPhN18IfunEFPFrBdJEIoVAG8OTruSzH8ixbajv8Ww0Vmcf\nBo7F93lDAESpBk/w3jEXS0BaMjwzY2hHioJ/5vnIjIWFokjPOPzgo70MstA4vNmSbNlV4Q+PThAk\nZVp2QsdRoBR+Jtj13UVyyzH1esjobuu0erQS5ryh3T4bi4t+m40xRgjxVWPM/cDVZJ71uYakxqKm\nz3TU3JYJEJA6bQ6nMaOu5HCYUklhIacwHSgpgadgZqaFN1BGxIJalpAdCWnITi8dVaUEVka57DE0\nlCfNSZYOR71izgxY6K1oZ2bafCI3TH1REBQEja4mSQ2u06tKr3f0eZUDVryaYafXyqBg28x0E561\nF/i0v/n0+9qR4svPd/npTxb6Hs4GY3X2YdaRbCa/JvtwtUe755GMLz/fpdbMEEKgLA+DYbtcwjr1\ntm984xitVkKtK8lvHuW5eI6uSlFpl2irw/Jm2FR3qWUtWl68RrltwTEs2oaRVHJsVWi3z8ZkPUvH\nvxFCvNcY89J1H02fS2KAmu5VTnviVHBLQGoEr+Y1XktRfd8w+5Uib1mciCLm0pRPDQ4RJfDCi9MU\n8i6RsYg9KIw5WE0waUr35EmW5ptobVhY6O0L+a6Frdbu44ymmvvsAi9lCW9NZSSZOa235dqCakGe\noxxwtlfTOwaWETzt1vikHsE+9TgWfItaM+PwTMb+Hf2GWBuN9WYfbhuy+elPFjg8k9HsKNITy3RP\n1qites/UVJNy2SfqRBQ3OYgRTe6kRrqS+YrCqJ43brUUb+6ysA62aLUSDs9mvDJmSBsGoohqxb1o\nW4M+N5/1GJv3Az8mhJgAOvT2bYwxZv91HVmf86JsD4XEExmHDtWIop5MXTQS8HZBE2aGo4dm8TNw\nPYstO8pobWjMS7aagK3tAnnjYWHwRczIVklsGbalAmv/7tPpo6vb657NxMQyniOYXMzIuZKCf2Y1\nG6eGycWMx+y1dQ+/Nj3NXJriCMFieqoVtQYlDE2R8RQLfJzR0+8XQtDsXLUEX5/rxMWyD1dL13iu\nPL1geOxd504ZX/ziN9ixo8J//MphlnZLHCVIgbAMiQ9OCl1XUcgg2F1k7+dG0QtV5O4cxq8zpBwc\nW/LQbndNjU+fjcd6jM0nr/so+qwbLSzEqQ3aKFLkB/JY1SE8B/a0EyZKNhYwJBPCesgD9xVwOy5H\nxut445LukTomcBCOTcvAm/VJJu8PuOdoyo+8b+f6B3Iq/UeIXisCg2GOmAHhrHl9hX25HJ8ZHFxz\nbLmjObmU4dmSEdYaJ2MMpXxfU+1W5HzSNbA2k3KFFT2+NxebJO0FRoaqxFWHhW306sCEJhEZP35w\nJgAAIABJREFUlgOegidKmo8sZMwUO1SxyTsWUab5m2MJ998l+97NBmY9cjUTAEKIEXoFnn1uItIo\nzCrhS6s6BAa8SJMXEqUziplPBxtbh6hE4GWCg17I/aMVgkmHXN6mmTeUUp9w6wBeoJgYUpjLSCCN\nM8O2IZt6S9OONYsi5nl7gQ+LYe4aDEjU2nN9amiITw0NrT1Hovm1r7UhNWs2kduRInAld4/1EwRu\nNVb625wtXQNrMymht18zO9um0YjpdCOyP5+kPbRIYzQgGq0iOxqNwfdt4rxFV0rafsrRcpNjQnGX\nKRCjwYZOophsONRU2vduNijrkav5W8AvA5uBeWAHPdXn+67v0Pqcj9EBm4nlDi0D2vXRRqLTBIRA\nxJqdNRfQGMul/k7IXGOGxYpHrZzwnakFrNkujDos3+OzZUnRyVsU6ykLFZtF+4yBeOKJY7TbyXnH\n0GhElIYr7HvkHu7d6tAKNQeTRUpIlt0OjvLXJcDpnco6W72JbExvs/mzH8j1kwNuQVa6du4NgjXd\nOh9//IVzVMUPH16g2YwJw4ysk5KPDMOZRzhTJ+uEKNUzNKObCoRhxs77SmRunoVizKQI2W3yvJsy\nAMupYr9weGDYpWj1PeKNyHqWjv8ceAR4whjzoBDi+4C/d32H1edC/OwXHjmtTfWnT85CzsXGRsCp\nRmcuk5NNvKJPIc7z3nyB16oJdiaZ2S0YOmnTGIX2Upe5ckry1iJqvkMaSP714WXcr9cQQJIoPvGJ\nPRccR6O2TOBKolTT9RM6JmG35TOZRDTdlOHS+iRDzt5ELuX7dTa3Kmd37Vzp1vl536dW61Au+wxU\nA1plSbGhCQKHMMzwPJturNj27n2MbB2k2ulSPzZFpxmxfXuZz3xgBxMTy3zuk+/hKy83ebxyGInh\nFbHMj+ot2NjUsozHBnLsr/QTSjYq6zE2qTFmUQghhRDSGPNNIcTj131kfS7ISjbQX//lMmnOw0Mx\nP9ckS3uFcs1GjKttDr42xQsnj9L6+ztxlwwLwzaNR2zctsZqCKKSwdIKz7OxUo3eXsC5G+y5mNo7\nczz3/CTpJg9nLkYArmuza1eFYtFFwOmW0081l8BYtIUmZ9k0hjpY8sIJBmezehO5z63Lilezy+95\nL1XHOe3drNAuSo7uc9l36IzXXNk2wns/+l4cz8WSAiFg6L59vPPUS8CZ9909ZvNybok2GUVcWqQ8\nxQKPRMP9sOstwHo+nWUhRAF4Gvh9IcQ8vay0PjcR2xL43TpiYAiDIEs13inZdS/vk8YJjdka8d/d\njWyn+PkiiafpjAm8pocZALXcIB1xyXUMg1tHUQUL7yMV7pp2Gdy9hdEdmnf2CPYdSii0NfV6yGc+\ncw/Qy0gr5yR77wZnQnGv5eHagoIvOJHE/bj5HcbZXg2A0gaRCH5/vEZquxghmN1sY4RgZouNAfy8\ny90fez9Ka+JOhO30xD2lJdnx2HtYfO750103Lal5uVjDjgVKGywEX2OOh6nyuQ+U+t7wBmc9xuZH\ngBD4WXp9bcrA/349B9VnfUhgu1zihB7ACnJg9/SpCLscfvK7xPsC4gGXivIJsxhlWWjb0PUV0haU\nvDLtnEVxaBAdaVCKuGDRnA572WV3DWBMg5ktNnsPn7t/Y4zhm40lthQcqs6ZSWZ1+KSfFXRnMB5F\nvNntUrYsOkrx3IvTtFIbhCSxNMNdSWvzCONJC+utJvURj9Kgw91b9+HlXLQy2I7ED1Z01iBNDT/1\ny5/le+/reUqPnzhBQ2cM5xyyDAIjWNYZcl+bbUPndpHts7FYj7H5eWPMzwEa+F0AIcQvAj93PQfW\n59IMD+epTdbwqDF3pEuxUkQnMW+8cIR6PST6O3txMgG2IClbxI5BKohsTa4D5cTF13mkUZSOZQjA\ncTR2BmnVQhUtSosB7WJIpyChvvb6Z08wq5lOkr53cwdRtCw+NzwMgDJwuLbMzoKHQKNTSU02OfKQ\nQzkcIG2kdNsZ7bvzSHxaOUXQAqN7CyjoZc4bIXh9ucVHjIdS6nRRsAAcG0CQywS/Mj3Ff7O1L+i6\n0VnPp/P9nGtYfuA8x/rcYFa6ZcJKcVwRKBLXFjh4T0K4M4fRknYewrxBCTBWr34h8SAKM5zEIkRB\nPaWS2gR5m2B4gPmxCFe4yLyFi8PM9hbVE2ujp6snmPNxo7KC4kT3hTxvElprvra0xCcHBni00tun\nm1lSjHV8qrZD2/hoBIe3xczvTNg2GyBmfPRCg3jQZU4o4sEGu6McVXMmU00ALTfjW6bND0cF/rxW\nO6coeIW5U4Ku/XYVG5sLGhshxE8DPwPsFkK8vuqlIvDs9R5Yn6vDaipGXmjgFPJ0HiqSpgmWEWSW\nJghtnKbCExajdYlRAdqKIBPkhqu0A0XbSRhLPMhSPCNojxQIKo011xhyXYbcm7uxP7nQ098KE90X\n8rwJPN1o8Avj4+Sl5MMDA0CvDYHB0D5lPIRIOLSljcBQH8zYu3sTOVsjSzazQylloZkfjBlp5Egz\nDQa0gOnBFvvyvULNvUFwTlHwavblchd8rc/G4GLfxj8A/hL4F8D/tOp4yxhTP/+v9NkIFAou5vAC\nxcWU8mPDHJGafOxiKVCWxjaSkvHJZIazlJHvSrrGQQYWwrKYrXZxtIXKMtKwt1cjpM3Mjhxv1QSZ\n9KgOj9GNFDl/rfeyWqrkeu/XxInmy893AXO6SRf0hTxvFFprfnN6GgX8+swMj5XLSCl7bQikjUZg\nC8VbxS7dwOCmko6TMVswmILN8OYcWUVSaGnCnKYZGzSKcurTsmM6osPJ73T4XS/lwSl4fzG/xpvv\nc2txQWNjjGnQa5j2d2/ccPpcCz7+8d202wmDWwZZ2lqk0A1xlItlSYyB0M0Y6LhE4x3qk13e/M7b\nbNu/l8HNg7QCQ7NkcFKJNwDBUK9eJjYWaiRPJx6loly6Br705SZ/78N57t12xru5kFTJ9eDwTLam\nG+QKfSHPG8PTjQbHooi7goCjYcgzjQYfHhhguCQRxrC4FBJ1urx6f4sgFHhKYmeGrpuxZUJR2mvz\nQ/tGqJ2wyIzmeFojc+H+5QqL+Q45y2Go7JCzbRp3CeZf7yfB3sr04wy3CcPD+dMimgBpqjg512Xx\nrjzObEa8VKe6eRBpWdiRJLUzHigOokY0933iLo4dmuGDH9vKuLEpdDykUoiwpyiggTBfYVRoRioO\nedPzZjqx4vee7vDFz1rkfOuiUiXXkhXvaakF8gLn7wt5Xl9WvJqSlDSzjJKUp70b25L40RIzUZn5\nsqLjGXJtG600IowIByxOtFocfynEWYLULSBdh9lCREU7vFNqkErNYGqRSYu8VizahnJfLOuWpm9s\nbhPOF174zmyHX3h7gkHX5pvfHKceLWPZNsK2iMoek8dinK6NEDbO2FZ+5r++jxffTgBD3jsTfjow\nkXLwZMpYWSLMmZbAjiVZ7mi+fiDm0w8FnEjOL1VyrVnxnj7kD6HN+fXc+kKe6+Ny23qvsOLVjLgu\n74Qh+4KAY6u8GyfpUhvvcPSTLnakSbopaI0lLawYmh+tkH+tSakcEGKxFLdRZCghmcnFbI4dpJBo\n0+vV5Gv6DdJucfrG5jZmR8Xho94QYPiboxn5ogsiIz9SxnQMHdWikvQma6UNL76d8P67XL5zJGGh\npRCn1Jw7scazYW6uQ5r2hEC1ZdNTexZ89dstnnyiweTWNu/78Bbg+tXarPaejrtNfLd4TjfIvpDn\n+liRPbqctt6w1quZjWOMMczEMWUp+ZeTkzxaKjEynOfIq4doF7djZxJVBIGFNoo4jlGbPLJRj4aj\n6JqMaU8hDTS9FCUM82RgJKlQ2KKnjFEriH46/S1M/9t4GzPqu/yD7xniuUMxI90cBelg+R5eK2Cu\nMcv8HvC6kGtDFiWEiSZOOacxViWf8NXvhqSpwvFsMiwkvfTUDEHFhqjkMikk32/1HqnVUiXXcnI4\nW+jxh95j8dJ3dV/I8zLJlOG5QzFg1t3We4VnGg3e6HZxheDtxTYi00xK8FPouvCjTz7Fu2qgZ9sM\nPDnP6PYq8/MhjoQ0ilEupIUIfajJA9u2Mi4dFr2IXCyxdW8skWUYixx26gwhbF5+aZpoocOvHHuO\nXHrumIaH+8kDG52+sbnNWdFR+x8OHaNTzuEVc+SGKkxvS8mMYHbUMPaWJh6SYCBMNLZlMVySzDcM\nbzTb7BnycW1oCQtzqpuOpLeXIzDkiDiYD3ETi3ZkKOfEGqmSn9i0mZGyta7wzMU4n9DjGzT5qU9s\n4sis6gt5XoJupHjxaMpiI8N2JGGs2DzgrHlP3pMstNR523qvsMl1+YejozzTbNJYbjLgO0TC0LAN\nQxmoLTbbcx6FZUXwrTp3vdslfH2OgU0DSNdBJylxq8VCrc3ej1lMlkKqoY2vbPyk5zHHMsN2QvY2\nLQSCP3viJEmiONw4/zOUpqpvbDY4fWNzB2Bbgopv2LXZRQQu3W153DGodiXtLRrPsWgUYuZMTOD6\np8Mrx6Mu31ALfNwa5u4tLs8uhCRYGM4YmkE6ND3FvJehlWYijAiUZGoxI1WGCMXgUpOdfnDJ8Myl\nuJDQ44xO2L+jH1q5GAcnE37v6Q5xahCi15JbG/jgPfIcoyIQ57T1Xs3d+TyOlEymKcuLUB20aVuG\nA3bG3aFNLFnTrgLHZeyBu/HzPhgDQqCTlOZLh1i0DfNeBp2YrrSQtocQ4KKYs2DRlgxlgiRRBIFD\ntXr+z/n48aVrdav6XCf6xuYOQ0chJwdiHGUBBjuFI9sV6YLmTw6+w8RftIgKIxhjeHMkZCIOObI8\nxb6THllm8H0Hb3gQ1Q2ZPT6LzhTFzTn8okFJi6W5aebdAhgoBhbf9/BmxqTFy9ES+q0yn3wwd0Ue\nzvmEHqGvw3YhMmWYbyjebHXY7fr83re6IAzVQu/+hVnGs6IORyr80IP5NZ6gObV3c77Omr3X4ZUt\ncKLeIVqICAKH+haXJCd5MdTkF1ImQ8NyIyaXd7GGN2FmWqTt7ulzWK7D6H17cDU83LKZW4woV2wa\nx9der9aN6KQQx4pKpb+guJXpG5s7hFzOoV4P6ZYtFsIlKkmJ1LLIgGZOUYrBKivcPVWcOE/HjUjK\nFuVJm3jMJW5DIXJw23UWj0akStNcDgGQsxJnyUGkGS3dobyrwN27CtTnu9zvFnklafBH6TRkkoea\n/gXDMxfjRumwrZZfkfLWDMWd7Zm+q1GlEVqMlM7c90NBkxdzc9hK8j0LPndv7v2tnbi3Tzdckud0\n1lxhwdbEAwmNuQbeiE9SckiHbNzYoCo23fEuSzZ0ipJGCNOzXRbnmkjZWwxYUpIvKNx8Dkd57I1S\nnAZ86X989IJ/01NPjbNr18A1vlN9biR9Y3OH8L73bWH7jjLfLqcMSENOCVIs3vEzhi3DoheSMwFv\nBSn3xIojuYR2w2AsD5MYTo7B7pMBU1MdJg4dYedDd+EVCzi2oFD20UnKwqFxtr5nG0mcnb6u1po/\nC+dQxvB1Pc/f6fYmr8tNtb1ROmznk1+5lVjZ+DdGc9Rt4Sg4ZDXYxADtSFPKSYxQPOfPgzC8Vlng\nw40yQ0ULg+G7L06harM88+XsnM6a0FOneN8ndnJ/06cxa+E7gqUcVBYVTmpIPNHLBgxs0g8OMTBu\nU60GpN0QlfWyyrJM4wc2asxmcqaNl4YMD6+v2V6fW5e+sblDGB7O8/pcg8O+wDuVzRPZMO+BF0JW\n6W3MNhzNhJ9Ss8FJBJHKcGNBy1dEBcPQrs2MHxhn+tXDRMamPJAnq3rURjVWGIFSPaXPU7yWtZjV\nMdttn+NxyB8eW2a/U7ysVNtMGdKOxWicvywDdblcSH7lZnClsj8rWYRJLmUujNlq+bzpdsnZKb7x\nyBS8lWvQslMGtEfdSVje3OWBXWUC9/9v786D7Lqvw85/z13f/l7v3WgAxEKCK7jINE1KtkWFlCwr\njmWV84ddmXFiJ1HJcZTIVsoli5maVDlOObET2VOeKZfG1kzNRGUnlqw1tiyRtGSbEneRAomFxA50\nN3p/+93vL3/c10ADRIMA2I8PAH+fKlax+3X3+91uvHvebznnGDz/tdPs6s1mqtXcG/ZI6p2EdjRE\nJTLYagTEVZOVaoTb6BBaisaQQTJiY8SCeUuZMdfGcS1GR8/VLvO8iIm9NX4wmdA6FbC1lgWaxx57\nYsPrmp1t6ZnNdW4gwUZEfgf4B2Rt+I4Av6iUql/6u7S34hOfeJClMORQ99y6+ZeXlmglCa+9NI93\nqs7BOCAxhdd2RQy1CqSdiCRNSQ3BsQ3mql1KKwF2uYjfaFOvt4habfxtY5x53zjb/Rjleyg3IlYG\nKdmsJi8WUaIwI5Nvmws8XKycvYm/2VHbq80FuRoblV8ZhKst+9MNUkTBc2GdgmQv76mCzclSi2rD\nJkoTnnIXsFODJFU4YvBta4H/ODqN2ZsdesomVoJVLIKobFMfQITCdJZH5UpM3PUwbIsdcw6KCt/3\nZwm7JsFqgLzUIIlTXpwLuOUuG8OANIwplhzCMGHO6NJtK54PG4y0CwhcdMluTdTrQqtdvwY1s/kW\n8BtKqbjXG+c30C0L+mKjTd56Dp69SRhybMJOTN6ysEo2qSM0Swl5L4WaQZpkAUAAvxzSrVnUTgzj\nzpjEScrWWyZZ/fAWimUIPzBB+p0O3ZkZuHOcUzWDU1HAhLjECkZtm8PNJv/X919la+vckdvEdPj2\nV1aw4uC8fIm3kgtypdZmNeuPVV84u7nabPsrdaVlf9bPggquwZzymU8CJs1zp/aWxz2anYjXjTZ1\nI6SQZi/9bWWH2TjkC4uLnPzTU7wyb3OkG4BSePkhfJWweOg4cRAwsnOaLdPF7Ph7L/5UDQv/ZIQ3\n4pIaJsUjXUIXzHqMMethWSYLB45x0317oJCjVHZo5BKiilBd6LDkQsOAmr/R1WVs22Blxdvw8ULB\n3vAx7dowkGCjlPrmug+fBv7hIMbxTrDRJu+IkVIkZf5kG4D4qE+loggdodYxKFt5WnMeUaxQSYIh\nKcHeHIkDq9tNppddbn5oL/HtedpVoeqbNEdiFqcT7H2rjBurHBjqMJ6vMuaYxImi0VGYgeLZWod4\nPxi9VllmPod/pk7UaNBoZHedT3ziwbNLQusDDVxeLsiVWpvV7O7NIkYd57zZzdoMq+PH/EC1uFvK\nFHNWX2ZYFyauvtnhh/WzoK1llwNGCzs1zzu5POxYOLvaHGWFYmoQk1CyIWeblCLhP58+zY94Nm4+\nwg9TlGGSIIR+QuXmm0ijBGeoQigO+094OJbQaMfk8wkKxZmxmFK3iDORJ40C8u+1KTy5yMqyx9KZ\nOvXHXyBXK5PbU6W5zcTaL5idhDhncOzWAvfOXPp3cvPNw9x33+SGj+s9n2vftbBn80vAf9voQRH5\nKPBRgO26OdKmKaUGN/sGdq9FzeJsxLDf++cw08GcGmJ2PqW5ks2KjCmHSAwqTWGlmJK7Zxudkx4n\ndkTYkYVKwI6EI3vL7HmlztF8ynIhZcRSLKQBYaJopIrIgE4JOlttJlcNFIpWzaDctsDMbqhrM7Fu\nkC2dXcyb5YJcyZ7HhbOaNWuzmwdLFb57MKsZN5Pv8sXOLGOFbdysyps+w7pY4uqFs5v1M6y8I3yj\ne24W9P5ajbAS4tfhuOcjZEeVbVM4Veyy0A3ABD9JaEYxrTSrvLAYRthTRXYzhasEEaE4kSJAmqQ0\n55borLZhuITX9ilNlBjZNk7RDGhVLeIJm9FmEZSiEKf4+RjzeEI8fxLTNDANoWMHdB5wiZOUiSbg\nWETNgHoeGm9SZPOBB6b5rd96ZFN+x9pg9C3YiMjjwMXeijymlPpK72seA2Lg8xv9HKXUZ4HPAtx/\n//0Xr7qovWXlsnPeMoXRPk7sDpGvlgijhPouG6ub0llcRUKD+Skh7MZ4ZYeaBzEplgeNsrC8u8rK\nXMy7yfHou8bPVhs4cCpm/+ICq6td8lmbHNolk8NTPrtnA0oXjKngGqRsUGizt3ezkSvZ81grv1Ix\nTTr++es5c50O/2OugQpdhovC11rZybqvegs8Vi6z0kk3dYa1UeLq2uzmwj2s2dTjO7R595YSx32f\n1TjmF7aMk0xBvaMIohRDoOWlvNJ2yLkmc6rLTBTRjGMcySpC3OGWmS2ljLsRTh0sy8SwbUzbxLRs\nIj9ETIs4TsAwCPwIJ2fjeQGzW13MMCFoe4ghRGGMaTnYH7gZee40RNlfMbyniozZKC9FNaOzbyPc\nSDg2orgbhWzw5kK7/vUt2CilHr3U4yLyT4CfAh5RaoPSvdqme+KJo7Ra4dmP15at1o64lssOjzyy\nC4Avf+UQI1uGmZOE4p4S47FCJkY5dKxNa8igMVVgKOdQsLKbvkJQptB+eIyfmrwDDCG/cG4zvzGc\n8r9/u8XcgpDPmSgHZkd9Eq/D3BaTmw+dnz8zVjHIOwadID2vCvX6XJCLudI9jwnH4WNTUxs+XkpM\nOsh5J+tmE599cYutlC85w7oSGyWu7nt6hl+pH+OeGfBK46DAUNny1b6JkKRoccTIMTFh8Hy7zUen\nprLrHTq/k+mo1LCcGNsGp9RmT8Hhdd9nm+syH0bkEoe5SpftjTymm/UBEpVVHHBdkzRnUSg6JHkY\n21JBIbTsAp1qi1ICYU0wTYPYExw7pVM2KN89zcJ3j5FOuCQ35TFEUFULLwTTV8R2SmRCOycsx4rR\nWAebG9WgTqN9EPh14L1Kqe6bfb22eVqt8KIlP9Y+t352UyraLM0sc2yPzfaVPKu99JnhmotXs/Bq\nKeOF3o1cQTdQmErhO4qZfId7nOp5m/nVgkGhvYB3uoUzUqSVT2knKeVWSqds0ikZsK4HrGUK777N\n5bsHg/OqUK+dRtto6epK9zxuLRa5tbjxmv/casJTM12+2p3HiE2agcIwTb7SnedjUiDvuJf6lV+2\njRJX616EM5XDzhWwgiKuZH+IFTsmrCY49ZgwTnESm+PrrvfCTqaLBPhGyFzaJfTAN1PiNKURx4SJ\nYlocOnlFN6fIiZAmCYZk+TdJokjiFLtcpJG0mESRYGClFrc2i+RImF9KsW2DKLKYGC8ShibHq2W6\nZZfwoVFSX1FejElcAyNKqb7WhbrP7sksfyo3vCm/Ru0aNag9mz8AXOBbvXecTyulPjagsWgbePTR\nXSxZKUtWix/90R1nP58oMOc8jNTnh+0KVcPGD2EuiHEtYSpvMmRk74wv3MwXIO50SN2UuWkHK8xm\nJ1aomJu2GD55/hjWComur0J9qVNgl7PncaXGKgYvR20O+h5DsUubFDA4YHnsy3X4mcqFC4BXZ6PE\n1ZPLx5iwLazEIO4tKyoUhwsBbpItgwlCGCuq+XPXu76TqULxqjToqISOmZBLDE62fBQpC3GAGxvM\nuB22eXkWxkKGVwXDMrFNyfZhCjZuvkZSUxwx20wleWqRTTFJsb0yZXzSRojrmARhwnQlh69sjq14\nGFvyBDULmfeRMLvphMM2ybyPcaTFfR/KAv3isSbLG/0N9AGA696gTqPdPIjn1S6uVDq3X9No+Od1\n/Oza8NCWHB8cPv9t58Mlxb4TESOJQyWxOeB32RnZ3L3FJe8Inp/w0umQVjfGMk1ungyZGjo3u2iX\nDTplk2Iry59wA0WnbJKvvnE5yjKltyeSPZamKX+5vHLRkjJvtudxNaIo4ev+PG5qYqyLV25q8jX/\nDJ+KRrHMt/5SGnUcRp03trH+iwbcVLPwlLDU29Oo21nx01wieE7KogrIpRalRM6W76m3z3UyXSJk\nhi6H6RCFCaEkBEaKJIrUEnwzJbUjOolJJw1ZVXlyLUVkm4hhELY97HKB41aLBDhR7DLdcDAUtMkR\nYGFZBkGYEIUJrW4CpCyfmCP8sSGcWOGWnLMz6DAnOA+Nst1xdbXmd4hr4TSa1kcXtote26NZUyo5\nPProrrMfnzhRv+xTP39vNDsVdajd5ehSnXcVhsk7wvGFmGdeC0lUlp+TqoQ/etzjF94rjI0VqTd8\nGtUCUSMA/9x2XZgKrVuLqAv6lVx4smyjkjKf+b3v8U2ni28JL5+rmINnwcvxMd4fFvjVTzx0Wde2\n3p8cX+WM4VO0LDoqygoXkxVKmEl9PntgmV+5e7wvOTfruUSYpMTKwEmFvc0cSkG9Y3NHUuW+osNI\nxcCUbJZUK3G2k6kZC67v4JcSjFTo2AkoA2VkJ86UAXEas1SLmJh3mB/22R2VkFSxcvgkYllY1exE\nGUc6+HeatByDociinHp4OAxNDiEoAj9hzy01fvahAtK9kwN35chdJCfTN+DWYqWvvzPt2qGDzQ3u\nwneNjz32xCUzta+EZQqTNYOvdpvUSsLBoMVUw+aZ17I7fc7Oytiv3Zj/6990+c2P3s+yk/KNko8T\ngWdDPuLsGaTABrN9/jnYtZNlvzQxScG3+S/HZ/BjxR/Onp90ebjVJbgrR/6CG1ue7Mb2+itdDnW7\nV1wCxglMHgxGKeay50mUouUpUhRRpJgPFd9Sfl9ybtYzBMaMNotpCSt2GYtcPFycMOIOq4yag85y\n70CGY1CeSnnSMVhtJSw0hUmzxA/Fwpm0zbGhNmaYEDkGSiUYKWALXTNkrtukbecorLaZSCyqQ3nE\ntjk2HKBWOpDEpO2Il1WLvcsOqenieDMo0yE2bLaUTH75J7biOgZTtQLt1zfelp2qFTZ8TLux6GCj\nvSVry1a3Fgv8yQtHWVkMKcgwqIQwS01BVIqgSE2bz/11l1333M4/LsTUxhL+qrnCh0dHmXIcTgYB\n212XWwvnbkBrezDdMOX/2LfAaOxwKPGYEJfvBx2+sVDnQ5PZEp+TZOXqn39+lm73je0cn2oEvPSn\nT/OIn+M3/8W7L/sa766VOHXYYMy1UCo7MAAK04QTkcedY0VAbXrOzcXYkjBpNPCVzYIqUyQAr8mW\noSzLYP2BDNcx+MiDeX77vy0RKwNbGeyUMiVlkl9NaYc+C8MRQRRRMoXEAiOG9NkF7pghYxKSAAAc\nMklEQVQeJj25yLvffweJElbsmJlqwnhUwSu43HvXJGcSn7vvHWfSyPH+e26/6HX/249f+UxSuzHp\nYKNdNaUUn/rCCyx6IS/HcPTUCu1phzuTCokfg1KoNMWyDTBMMBK+sDLLnsMlLNPk+HQTtTVmqRJy\nX6nE11dWeLBSOW/f4pjvc9TzUXWbM0mXv1qaJUgTFuOQrq34tb85wN+87GMCz377OFu3Vsi1QrZc\ncOJOAS/ttinVXF4446GUuuzZza1TFk86Bm0/wTQMYqXIWcJs4vNqaZX3lnMUXfuyqhpsVD5ozaXa\nGz/++FHa7ezYulUskt86TeJ5eF7El798kFLZ5dFHdvIXT57ki9+c5/gTx0gsl9xNOyiP1oiihCRO\n6NQ7DFtCcSiHT0LnQAsjjXAtk/ItFZrHfcbthJVuQl4ilCheLEUMpaCwUIZFx1coZfCi3eA391b6\nvoSoXf90sHmHuXAP52KPX65jvs/pOGBXtcDBA0t0z3gsTXZpWSFOlCLSq+GYpDi5HEeqdV4YX4Ru\nzEjL4XDaZnHZ49Rri/xJFwJLeNVa4pP33cVQ2WLPpMm3VlYwY4MwTvGNhHk3YkfggiM4wHIpoXNr\njb1+liu0PodovXbZwK+ZjEcGR/Nc0YEB1zH42YcKfPF7XZaaSVZVOYJj5RZjNeG7foP3eTZeBG0/\nYe0gw8VqqW1UPihVEGBzeq7D3Gpy3om7tb/ZzEzzbD6U2Bb1VY9uvY1pCSdPNfG8iHYrYN9qh+Nb\nuqSez7vvmmJVFJLG+I0OpmlgCFiWwcx4hK1MSBWuYxGEMVao8O4onbdvtmwpZp0023cxOnT9GLNa\nZdw0iKyQVSOkhm5spl2aDjbvMJt18mdtecuNshuiH8TYjkl8ZoXZ28bZ3Smj4hRQWLZNKin7R5ZR\nKA5MN9i1XKaeD1ACS8MOaQWmWzUO2wH/43CLocTlv7sRqzu6bLFyvHB8kZcrbUKVsNjysXppKHHe\n4POrq4z94THi8FxuSj5/rjCj7VokPz+N0cnuoG4kV3wcetuoxS//RInvvR7y8lEfVUk4asZYXZdD\nymMk8sgHNt8/KoxXs5fVxapVJ8YbX3JP/M1pVG0MsVN8X/Hv/+gYqUDea+D4dYbGavzSx3+c2HLZ\nuSWPIVll5u8eDJgcOb8A5dBInpViF8c2Wbglj0oS0jQijRNM24I029DyCopOVUErwsvFWOUcUQSx\nJYRbcnSWz+U85XrdNNfML/r8/LbRsx9vVi8h7camg412VdYSECMTVqwUr2wSpRaYcHp1hknnFqpW\nDqUUhm1xpFynY4WUQpuWE3JsqE1kpuQTi5YdgjIpOzENJ+T1QoOfVJN8P27QPa3YuhMWJaKbB/EN\nWjXIB72BGII3ZhHuKVI8ci4htVg8txS3MGXS2uYwfCg7uJCPOS/5cSMf+cifsrBwwZKXCMO37eb4\nzQprtMht20aomhaHafIec4yiK/zd/uzEn2HwhmrVfmGYVIVnj1CnClRtjHzBBtMgN7EFijl8L8bL\n51DOKK+dWGT2/32NE4sGx5YDujMzFF1BjW6Bgg3JuaN3S5ai68ZUlgNmRhzanYC47eDXW+DmsfMO\nyjAR12LylMeJZ45hNLu4wwUIYkrbwTm8QnukynAp+x2u1dFbYzfgPbXNOWSivXPoYKNdlbUExMby\nSSZsi5WjPul8h3zegtc6PHvwEHvuuh27kKc0nOeVnR5WmhXeVChW8iHlxCZBkaYpbYEjZhPHN/ie\nv8Dq8RazkzEuNovzC6x4MXmBwIsR12BL08SNTdqtmM6rs9SPNWnPB2fHZ9vZTd60TZo/exNuomiP\nZMmNcHnJngsLHYIgwfezVMpw3MFZCFg9dYLg3h1UA4PFxS7DowWWzYDSaEI553B8MQKEHWPnv7yK\nroESEx+Lph0yEQkBDmKnkMSYY1uI4xSLlMQPsAsFlEB5YpSRSshMGpMv5cnfsoPFV1/HCGbI37ID\ncXKAwkhNjhR91JEmKMHwE+a2mARfPc7WvbvpdJq4BZc4BcOHxfmTTLUsGgsJe6oOjU7KwxMjTC2n\n3HFbllx6sSVXnWCpXQ0dbLSrspaAuJZw+MpsRPukd3ZGUV8KOP69VwiCmNw/2ErXmaIU2SSiwBBS\nQxGlKaGZLTPFlsIrKEoNk9CB1WmDmxtVlFjUckWs/a9TmRxhpZEFhp1uCddP+cFfv0Z1ZhUAu+DQ\n7Z6/Z9O8yaEzZBMteqiixd+eXiZtRjQfP0Jgww9OHOCW8sU35WdnW7TbIbZtEo87dO8pkn/SIxxT\nuCeWsWuKWh7GKmWqpsM+1WSHyhHHILJRuT/FgpXyt7WQD67a5EJBqZTGSI5hw0ClEatOCNKbSaQp\nYpj49JbLkhhxcliFAnGnQzJ3GsnlwTRZjjxsexSrE0HBwewkdMomYSFhcf9hVjqKyliF6vQEatwi\n3TvJjnoJcyHgvXsLzJ5c1pWVtb7RwUbrqwSYf08Z1QnBsUGy/BQUhGaCFYApQiKKFIhsBRgsFH1u\nWShQilwmVUKSDuMcnGfxaAvTsbF310h8j8g7l6Q6OVmi08mCTbHooIAT95ZQpzy2jhSJQsGyhMqM\nzw/tqpKkwuSkwezpNkfORG9oiBZFKbZt4rgWwb01TMfE+Ilp3LECrjJplRVtK6BlZjOq43HIF/72\nCLRsFLAvemOzr5l5n84uxZFczN+WDN63YtApwqktIXYrIlYJp6odtrcKDMHZLpkx6/N3FGL1XrpK\nobwuCjizVdhzQYwzQ8XSw0NUvlGnu9pk+vYdBF2fes0iThSzVY/pRWExLaE2LBajaW+dDjbapsnl\nrLM3+yjKNuu791fwh2wkSOhIQGJCYmTdhmMDLAVGqjBMAMH2TXKege8m1KMIafk89cxhlOkws2+e\n5ZkVtm+voracX9ZFDMGtlJAimKJoLNZp35zHdwTzWBcjsnABf8hhy+RWKt0hUgXLuATVGs+9HuLY\nkHcMHrrV4Qw+4ZSLOeMTjzmoqTxTxRGi6Tyjjey554djqu0iO70KRdcg7wqvtGaZLOVAFMuOy0Rk\nnC2bHyuD1dU2hwtC1zDYX1DsapgsbbdJjIiZSpckilAizFV8as1itqFvWFisr4gtqHhdiQQRuiMF\nvGmhIwZe2WK57dPKCcfTgNa9ZQ59f5XiGZcwVaykHk07T3EZVqZixksWCQaJtTkFRTXtYnSw0d6S\ntWO5jYbP+Pi5tfwkUYRhjB0Ko692CYKE6e0ucxGE5WyVKJ+3sE/HhE5I4hjYuTLl0MVNDMIopV6N\nCV+cp5S3aQSKwkiNIFY0A8XTz5xGpYqVFQ9xHHa961aKExXKhjA2WqThBPz19AocbgGwvNQlSRSl\naoX9VpfFbx1kbMcWCkWbKEx59iXhpx8ex4sUn/7/D/K94jyrDw9jf2kWduYYLlRIh3KYrYRG0MV2\nLHKBjeeYdE/YuEUL34JcajFutlm0El6swN56leHIxERhSsLszpSGAWasaBopT+eXaTVjhrtllocC\nJCdMhTkWXY+W0aVsOag4IUeE65p0QwVBl9NRi1wjxHAdCtPTKNfi5lbKllIF0pRDr7yOvdQmvLOM\n2U1If3SU4sk6CHT3OFiholTOUSy6dHYbNJa6TA5nf78raTynaZdLBxvtLVnb67gwWfG8Fr6tXrLi\nLz7IZ37vac7MxKRiYqiEx188yfLDw7hBiunEdHIlOkqBp+gmARNGQNkuMTI+TL6Upzo+THUox/hI\njmRxnpdfOEX1ll0kScp4LfvnnIYex3co0uEcanuCxIrYNsiX8lglm6AU424vYdoWjikoE/wwpeUr\nKnnhe94qp/MxyYRL+p5hcqMlrLzL6nBMZQ46VUWuZFBtGrSHA7pE7CrZDBWF0yolUYrnCyapJJws\ndqnWy5gCvt2hOWZQKplUDItGEjFfgskTNkOmxWquQ9cwUC0HO4KZUpdbVxzC1QZRtcSu26YwSRG3\nwcy7pvl7fo52o3y2vw0KmA2pGWVy4ztp7V7h5K4hRmODpbtSPvRr21FhGavWxKkn3Htv9jc6k/jc\nnozxM7dXgStrPKdpl0sHG21TXG7+zq9e8HW/+u8C7LGsRL9S8PiLLZI4hSSlCBgRlG4ao90JWZ1Z\nIo4SzDSiVDCR4jCSW8J0bPyVFp1eHk27qGihGJoxaERQPBPitBJqEy4FT4j9FCs1Ob9lnyKMFXNp\nSL2UYtsWKgV1Xw3mUrq17Bxbq5qgHAPshGE7h6lMXrNbTHdclLJIDZtDlkvDbjIRmaw6IYnrYYY2\n3ymndOyUMck2+20RltOIXNnF7KS0JdsDqotBOtdmqZjgvHSISbvA0M7tmKJwCHmqGJF2ofKB7dy3\nWGOs8saX8Z997TDfvd0kl2bLb7kUnql2GesaOIkB65bl7NRkv9Hkn5VrV9x4TtMulw422kAVIrjJ\nP/fP8OtPHmFkzw5yxTykCqk4iGnSOL1AIW/jGzAxWeKeO0cJlEXseVTGa7SrFvfeO4lC8Vy1y04j\nZXUuoJDYmNvLqK/PUfY9apZJ1AYrnyLnvWkXbBO+G9Txcil2nGbVQcdcAivBKEJOgVfNiooaphDY\nKdXEoVMKCc0EpSwiJ8+xQoNckt2g3UQ4XAjYRcrRkk9sZLf5OE1opylxCo1yQr6YkHNMDBG8omJn\nUsHvNii8Z5hdM7B8agGA1TwcdWC75XK061FVRcYu8jJ+OWhyzPPILQesAp12hD9u49RXGZMKYlnU\nDy1knThViu3GnL5nAhVxRY3nNO1y6WCjDdSF5XNaqx0aT7+KWyliuTaloQoWCX7XxzQN0lSRy60l\nSiryORPPT/C8iJXlLnU34UwppNIxiOOEUiKs1CzMEZug1SENI0zHJvZD7DgBw0LM7IbbdkIOdjqk\nBljdBCdSxO2ENC84KxEl0yYeSkGEkcDFTA2SOCVuJnw7PM2dSwXmDWjYCRNxdnqskBqs2gnzjpBI\nSj4WXGUQiyIhxkoFXxLmxQcFjhgsqpDJbRXeNXYTiZ3yz7duZWc+j1KKz87Osj2OGbZtjjcDvp/W\nuU2dP/tI05QjN0WkrYhub7YXBDHimRjDRSovgWEmjAUm48M5hqXDyvEOJcPgz5eWNrXxnKat0cFG\nG6gLl9++/e3j7NyZ9ag5cGAJucjxYd9PeOmlM+SrJX547zB1VaC+muPDP30zX/XnuTdNyKUmr766\nyGTRYt+xZeZ25bCfaxHuO8LYbTswHYfOaotiJY/tWhhJwHfaq6yoiKTZBcvESBSWnxCPOnjzbcqW\niTlik1pCCsQ5RWjF2KS08glhbDKTtsilApxbo3MT4XQuYiiyMDsBk9UcXpKNMUgUkZkwbNvkTIO8\nYeClKYXE4qFclelh62w5mAsbw20rORxZaXPU99m9bvbxrNekVVHEKynmaHbCLA1N8pUiiQOJ72Md\nCyi2hZ33VACHfKNJK003vfGcpq3RwUa7ZgVBTNEOMVVKrpiDOCFJUopFG0wLr+OTE5cxaVMXk/1N\nj4NJlzwmtpngSYdV22brniGaDtzilim1U6CBGHma7Zgfmsiad71ab9Mo1YjimDAntF0DmcohVRvD\nMjC2CMGRVZKXBHu0iDXiMLWUY3goD73GzL6Rsmq0qUUOvkqRdZtCHQd2zlncbFj8w7u2Ue8o5usx\nJ5ZirFLCK3GLHda51gpHPY+bCnl+pHbuhNj6dtcApiHcPu7y4lKDStPGkKxCQ810mHhqhXbLx3Wz\nr++EMLytTOwF2N1zDX8sSbPlSMt5w88HPbvRNo8ONto1ZXy8yLFjWUWAet3H8yKazx2ksnM7biFH\nuZRHnBwqienOzGDcvQuDhEJ7mR+75RZG2xO4tkGtKPzx351iwsreka8c6GKpXhZ+LxEyangUjGyZ\nqeR7fGTrHj73Fwd4/cl5rLyDUjHJPVUkTPFaEUEjIvVijKUOubttSgebDOVKYJo4FkzeXOG+sMAv\nfmgn+09FBFGKICgUrm3wkVur7Kr2clmGIJ5SfPMlj7+MFhg2zuUNhbGiatm8GNd5QBUQkbO16Kqm\nSSdZl3NjgBqO2VoRxsQm7xiMVQr82afbHD26erYgqW3nmWym+M1ehQXj/CTR1bx58Z8PZ9tM69mN\n9lboYKNdU770pZ87+//ru4o+/sQx/CRAxGLlcIO426VR987u94yPFblrLMddY+e6fL7oFFk81Aag\nfMJnrts677kKBfvs9980VuRDo6M8dSjk1efbmKZB2PXhtew4d7cbYRgGgR/juBbVjkMuN0a+6AIK\nz08opia7Flr85NQQ7x/P2gt4YdoLAMYber5YpjC9S3Fyv4cdGayS1WCzTWHriMUBr3v2Jr9Wi24j\nuws2o865CtAPPDCNbZsM9/r6HDzeoVrNUTCzWU0Qnp8kmouTS/58XdlZe6t0sNGuC48+svMNnztx\non7JWl5X205h794J2u0Q71BEfjlbCpuf97Ftk3rdZ3SsSEFaLHsrbPuRnWCaeMsdJm8zOJVmmf2W\nKb0mape+SU8XbT55xxT1jiKI0rOzsrW4tHaTX6tFd7WCZvvs4YgkPNesJlYGJillL9SVnLW+0sFG\n0y7w6KO7gPM7Y3peRD5v43kR5bJDpxOSy1korwtA3PHOtg24EqOOw487DlkhtP5RqWL1yEmGdm/H\nLhVQYYKZz2aBY0ab2f4+vabpYKNpG1kLOgBf/vJBhofzeF50NvP+WjY2VqTROFekNElSzpxeYX52\nlXytTK7gEhRMdhdGmUW3DdD6TwcbTbsCjmOysvLG49iNhs+JE/Vr5qa9toS4VkLovPJBPWNjF2+t\noGn9oIONds26MOHzYo+/3XbuHOJnfua2N3z+zfaPBkEHEu1aooONds3SN0tNu3HoYKNp62w0m4qi\nhGPHVs87Ln3h92matrGBBhsR+STwu8CYUmppkGPRNNCzKU3rF+PNv6Q/RGQb8AHg5KDGoGmapr09\nBhZsgM8Av876ioWapmnaDWkgwUZEPgzMKKVeHsTza5qmaW+vvu3ZiMjjwMWy3x4DPk22hHY5P+ej\nwEcBtm/fvmnj0zRN094+otTbu4olInuBJ4Bu71NbgVngAaXUmUt97/3336+ef/75Po9Q0zTtxiIi\nLyil7h/kGN7202hKqX3A+NrHInIcuF+fRtM0TbtxDfKAgKZpmvYOMfCkTqXUjkGPQdM0TesvPbPR\nNE3T+k4HG03TNK3vdLDRNE3T+k4HG03TNK3vdLDRNE3T+k4HG03TNK3vdLDRNE3T+k4HG03TNK3v\ndLDRNE3T+k4HG03TNK3vdLDRNE3T+k4HG03TNK3vdLDRNE3T+k4HG03TNK3vdLDRNE3T+k4HG03T\nNK3vdLDRNE3T+k4HG03TNK3vdLDRNE3T+k4HG03TNK3vdLDRNE3T+k4HG03TNK3vdLDRNE3T+k4H\nG03TNK3vdLDRNE3T+k4HG03TNK3vdLDRNE3T+k4HG03TNK3vBhZsROTjInJQRF4Vkf80qHFomqZp\n/WcN4klF5H3Ah4F7lFKBiIwPYhyapmna22NQM5tfBn5bKRUAKKUWBjQOTdM07W0wkJkNsAf4MRH5\nLcAH/o1S6rmLfaGIfBT4aO/DQEReeZvGOAijwNKgB9FHN/L13cjXBvr6rne3DnoAfQs2IvI4MHmR\nhx7rPe8w8CDww8B/F5FdSil14RcrpT4LfLb3M59XSt3frzEPmr6+69eNfG2gr+96JyLPD3oMfQs2\nSqlHN3pMRH4Z+PNecHlWRFKydxaL/RqPpmmaNjiD2rP5MvA+ABHZAzjc2FNYTdO0d7RB7dl8Dvhc\nb/8lBP7xxZbQLuKz/R3WwOnru37dyNcG+vqudwO/Prm8e7ymaZqmXT1dQUDTNE3rOx1sNE3TtL67\nLoPNO6HUjYh8UkSUiIwOeiybRUR+p/d3+4GIfElEaoMe02YQkQ+KyCEROSwinxr0eDaTiGwTkb8W\nkf2919u/HvSYNpuImCLyfRH5+qDHstlEpCYiX+i97g6IyEODGst1F2wuKHVzJ/C7Ax7SphORbcAH\ngJODHssm+xZwl1LqbuA14DcGPJ63TERM4P8EfhK4A/h5EbljsKPaVDHwSaXUHWR5cb9yg10fwL8G\nDgx6EH3y+8A3lFK3AfcwwOu87oIN74xSN58Bfh24oU5vKKW+qZSKex8+DWwd5Hg2yQPAYaXUUaVU\nCPwp2ZuhG4JSak4p9WLv/1tkN6vpwY5q84jIVuDvA3806LFsNhGpAj8O/DGAUipUStUHNZ7rMdis\nlbp5RkS+IyI/POgBbSYR+TAwo5R6edBj6bNfAv5y0IPYBNPAqXUfn+YGuhmvJyI7gPuAZwY7kk31\ne2Rv7NJBD6QPdpIlyv8/vWXCPxKR4qAGM6g8m0varFI316o3ub5Pky2hXZcudW1Kqa/0vuYxsuWZ\nz7+dY9OunoiUgC8Cn1BKNQc9ns0gIj8FLCilXhCRhwc9nj6wgHcBH1dKPSMivw98CvjfBjWYa86N\nXupmo+sTkb1k70ZeFhHIlpleFJEHlFJn3sYhXrVL/e0AROSfAD8FPHI9vUG4hBlg27qPt/Y+d8MQ\nEZss0HxeKfXngx7PJnoP8NMi8iEgB1RE5L8qpf6XAY9rs5wGTiul1maiXyALNgNxPS6j3bClbpRS\n+5RS40qpHUqpHWT/WN51vQSaNyMiHyRbsvhppVR30OPZJM8Bt4jIThFxgJ8DvjrgMW0ayd71/DFw\nQCn1XwY9ns2klPoNpdTW3mvt54Anb6BAQ+++cUpE1io+PwLsH9R4rsmZzZu42lI32uD9AeAC3+rN\n3J5WSn1ssEN6a5RSsYj8S+CvABP4nFLq1QEPazO9B/hfgX0i8lLvc59WSv3FAMekXb6PA5/vvRE6\nCvzioAaiy9VomqZpfXc9LqNpmqZp1xkdbDRN07S+08FG0zRN6zsdbDRN07S+08FG0zRN6zsdbLR3\npF413H/xNj3XwyLy7g0eu01EvicigYj8m7djPJo2CDrYaO9UNeCKgo1kruY18zBw0WADrAD/ihuw\nermmraeDjfZO9dvAbhF5qddnpyQiT4jIiyKyr1cQFRHZ0etV8/8BrwDbROSfishrIvKsiPzfIvIH\nva8dE5Evishzvf/e0yte+THgV3vP9WPrB6GUWlBKPQdEb+fFa9rb7XqsIKBpm+FTZL117gUQEQv4\niFKq2WtY97SIrJWduYWsUsXTIrKFrJDhu4AW8CSwVqH794HPKKX+TkS2A3+llLpdRP4QaCul9OxF\ne8fSwUbTMgL8BxH5cbJy89PARO+xE0qpp3v//wDwHaXUCoCI/BlZ2wuAR4E7eqV4ICvsWHo7Bq9p\n1zodbDQt84+AMeCHlFKRiBwnqwQM0LnMn2EADyql/PWfXBd8NO0dS+/ZaO9ULaC87uMqWW+TqNd6\n/KYNvu854L0iMtRbevvZdY99k6zwIQAicu8Gz6Vp7zg62GjvSEqpZeApEXlFRH6HrJHb/SKyD/gF\n4OAG3zcD/AfgWeAp4DjQ6D38r3o/4wcisp/sYADA14CPXOyAgIhMishp4NeAfysip0WkspnXqmnX\nAl31WdOukIiUlFLt3szmS2RtBb406HFp2rVMz2w07cr9u15vl1eAY2QN/TRNuwQ9s9E0TdP6Ts9s\nNE3TtL7TwUbTNE3rOx1sNE3TtL7TwUbTNE3rOx1sNE3TtL77nyDVP+X7WJhuAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.ensemble import RandomForestRegressor\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.multioutput import MultiOutputRegressor\n", "\n", "\n", "# Create a random dataset\n", "rng = np.random.RandomState(1)\n", "X = np.sort(200 * rng.rand(600, 1) - 100, axis=0)\n", "y = np.array([np.pi * np.sin(X).ravel(), np.pi * np.cos(X).ravel()]).T\n", "y += (0.5 - rng.rand(*y.shape))\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y,\n", " train_size=400,\n", " random_state=4)\n", "\n", "max_depth = 30\n", "regr_multirf = MultiOutputRegressor(RandomForestRegressor(max_depth=max_depth,\n", " random_state=0))\n", "regr_multirf.fit(X_train, y_train)\n", "\n", "regr_rf = RandomForestRegressor(max_depth=max_depth, random_state=2)\n", "regr_rf.fit(X_train, y_train)\n", "\n", "# Predict on new data\n", "y_multirf = regr_multirf.predict(X_test)\n", "y_rf = regr_rf.predict(X_test)\n", "\n", "# Plot the results\n", "plt.figure()\n", "s = 50\n", "a = 0.4\n", "plt.scatter(y_test[:, 0], y_test[:, 1],\n", " c=\"navy\", s=s, marker=\"s\", alpha=a, label=\"Data\")\n", "plt.scatter(y_multirf[:, 0], y_multirf[:, 1],\n", " c=\"cornflowerblue\", s=s, alpha=a,\n", " label=\"Multi RF score=%.2f\" % regr_multirf.score(X_test, y_test))\n", "plt.scatter(y_rf[:, 0], y_rf[:, 1],\n", " c=\"c\", s=s, marker=\"^\", alpha=a,\n", " label=\"RF score=%.2f\" % regr_rf.score(X_test, y_test))\n", "plt.xlim([-6, 6])\n", "plt.ylim([-6, 6])\n", "plt.xlabel(\"target 1\")\n", "plt.ylabel(\"target 2\")\n", "plt.title(\"Comparing random forests and the multi-output meta estimator\")\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### 朴素贝叶斯\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of mislabeled points out of a total 150 points : 6\n" ] } ], "source": [ "from sklearn import datasets\n", "iris = datasets.load_iris()\n", "from sklearn.naive_bayes import GaussianNB\n", "gnb = GaussianNB()\n", "y_pred = gnb.fit(iris.data, iris.target).predict(iris.data)\n", "print(\"Number of mislabeled points out of a total %d points : %d\"\n", "% (iris.data.shape[0],(iris.target != y_pred).sum()))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" }, "toc": { "colors": { "hover_highlight": "#DAA520", "navigate_num": "#000000", "navigate_text": "#333333", "running_highlight": "#FF0000", "selected_highlight": "#FFD700", "sidebar_border": "#EEEEEE", "wrapper_background": "#FFFFFF" }, "moveMenuLeft": true, "nav_menu": { "height": "12px", "width": "252px" }, "navigate_menu": true, "number_sections": true, "sideBar": true, "threshold": 4, "toc_cell": false, "toc_section_display": "block", "toc_window_display": true, "widenNotebook": false } }, "nbformat": 4, "nbformat_minor": 2 }