{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# The Kernel Trick - Revisited\n", "\n", "The kernel trick is a powerful tool to convert a linear method to a non-linear one. In the following, we will try to explain how and why it works. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Linear Models\n", "\n", "A linear model for an input space $\\mathcal{X}$ and an output space $\\mathcal{Y}$ is a linear function $f:\\mathcal{X}\\rightarrow\\mathcal{Y}$. If $\\mathcal{X}\\subseteq\\mathbb{R}^d$, then we can describe $f$ using a vector of coefficients $\\beta\\in\\mathbb{R}^d$, i.e., $f(x)=\\beta^\\top x = \\langle\\beta,x\\rangle$ (here we are ignoring the intercept).\n", "\n", "## The Dual Problem\n", "\n", "### An Example: Classification with the Hinge Loss\n", "\n", "Let us assume that we want to solve a classification problem, i.e., $\\mathcal{Y}=\\{-1,1\\}$, and we measure the error our model makes by the hinge loss\n", "$$\\ell(f(x),y) = \\max\\{0, 1 - f(x)y\\} = \\max\\{0, 1 - \\langle\\beta,x\\rangle y\\}$$\n", "That is, our model outputs a score $\\beta^\\top x$ and we interpret this score as class $1$ if $\\langle\\beta,x\\rangle\\geq 0$ and as $-1$, otherwise. Thus, the model makes a correct prediction, whenever $sgn(\\langle\\beta,x\\rangle) = y \\Leftrightarrow \\langle\\beta,x\\rangle y \\geq 0$. However, the hinge loss demands not only that the prediction is right, but also that the score is larger than $1$. \n", "\n", "Given a dataset $X\\subset\\mathcal{X}$ of size $n\\in\\mathbb{N}$ and corresponding labels $y\\subset\\mathcal{Y}$, we want to find model parameters that lead to correct predictions, but like in Ridge regression, we also would like to have parameters of small squared norm $\\|\\beta\\|_2^2$. For simplicity, let us assume we could predict $y$ perfectly from $x$. Then we are trying to find the optimal parameters $\\beta$, i.e.,\n", "$$\\min_{\\beta\\in\\mathbb{R}^d}\\frac{1}{2}\\|\\beta\\|_2^2\\ \\ s.t.\\ \\forall i\\in [n]:\\langle\\beta,x_i\\rangle y_i\\geq 1$$\n", "\n", "### The Lagrangian\n", "\n", "We can solve the above optimization problem using the Lagrangian\n", "$$L(\\beta,\\alpha)=\\frac{1}{2}\\beta^\\top\\beta - \\sum_{i=1}^n\\alpha_i\\left(\\langle\\beta,x_i\\rangle y_i - 1\\right)$$\n", "and maximize\n", "$$\\min_{\\beta\\in\\mathbb{R}^d}\\max_{\\alpha\\geq 0}\\frac{1}{2}\\beta^\\top\\beta - \\sum_{i=1}^n\\alpha_i\\left(\\langle\\beta,x_i\\rangle y_i - 1\\right)\\enspace .$$\n", "Now, under some conditions (Slater's condition ensures the Karush-Kuhn-Tucker conditions) we can swap the min and the max and get the dual problem\n", "$$\\max_{\\alpha\\geq 0}\\min_{\\beta\\in\\mathbb{R}^d}\\frac{1}{2}\\beta^\\top\\beta - \\sum_{i=1}^n\\alpha_i\\left(\\langle\\beta,x_i\\rangle y_i - 1\\right)\\enspace .$$\n", "In this form, we can solve this for the optimal $\\beta$ in terms of $\\alpha$ by setting the gradient to zero, i.e.,\n", "$$\\frac{\\partial L}{\\partial\\beta}=\\beta - \\sum_{i=1}^n\\alpha_ix_iy_i \\stackrel{!}{=}0 \\Leftrightarrow \\beta = \\sum_{i=1}^n\\alpha_ix_iy_i\\enspace ,$$\n", "which we can substitue back into our dual problem and get\n", "\\begin{equation*}\n", "\\begin{split}\n", "&\\max_{\\alpha\\geq 0}\\frac{1}{2}\\left(\\sum_{i=1}^n\\alpha_ix_iy_i\\right)^\\top\\left(\\sum_{j=1}^n\\alpha_jx_jy_j\\right) - \\sum_{i=1}^n\\alpha_i\\left(\\langle\\sum_{j=1}^n\\alpha_jx_jy_j,x_i\\rangle y_i - 1\\right)\\\\\n", "&=\\max_{\\alpha\\geq 0}\\frac{1}{2}\\sum_{i,j=1}^n\\alpha_i\\alpha_jy_iy_j\\langle x_i,x_j\\rangle - \\sum_{i,j=1}^n\\alpha_i\\alpha_jy_iy_j\\langle x_i,x_j\\rangle + \\sum_{i=1}^n\\alpha_i\\\\\n", "&=\\max_{\\alpha\\geq 0}\\sum_{i=1}^n\\alpha_i-\\frac{1}{2}\\sum_{i,j=1}^n\\alpha_i\\alpha_jy_iy_j\\langle x_i,x_j\\rangle\\enspace .\n", "\\end{split}\n", "\\end{equation*}\n", "This is a quadratic optimization problem which can be solved using some standard solver. If we found the optimal parameters $\\alpha$, we can then compute the optimal parameters $$\\beta=\\sum_{i=1}^n\\alpha_ix_iy_i\\enspace .$$\n", "Now, how would our linear model then look like? Well, we can just substitute the formula for $\\beta$ directly and get\n", "$$f(x)=\\sum_{i=1}^n\\alpha_iy_i\\langle x_i, x\\rangle\\enspace .$$\n", "If we now rewrite our $\\alpha$'s a bit and say $\\alpha'_i = \\alpha_i y_i$ we get the dual representation of our model $f$, i.e.,\n", "$$f(x)=\\sum_{i=1}^n\\alpha'_i\\langle x_i, x\\rangle\\enspace .$$\n", "\n", "Now, we found that nice representation for this special case. However, the representer theorem (Schölkopf, Herbrich, Smola) guarantees that we can always find such a representation. In a simplified form it says that if we have a loss function $\\ell$, a training set $X,y$ of size $n$, a strictly increasing real-valued function $g$, and an inner product $\\langle\\cdot,\\cdot\\rangle$, then the linear function minimizing the regularized loss, i.e., \n", "$$f^* = \\arg\\min_{f\\in\\mathcal{H}}\\sum_{i=1}^n\\ell(f(x_i),y_i) + g(\\|f\\|)$$\n", "can be represented as\n", "$$f^*(\\cdot)=\\sum_{i=1}^n\\alpha_i\\langle x_i, \\cdot\\rangle\\enspace .$$\n", "Here, $f(\\cdot)$ means we can insert any $x\\in\\mathcal{X}$. Moreover, $\\mathcal{H}$ is the Hilbert of linear functions from $\\mathcal{X}$ to $\\mathcal{Y}$. Now, this is a simplified version only for the standard dot product. We will discuss the full version below.\n", "\n", "### Excursion: Max-Margin Classifiers\n", "Before we go to the kernel trick, let us just do a small excercise to show why using the hinge loss for classification makes sense. For that let's assume a simple 2d binary classification example." ] }, { "cell_type": "code", "execution_count": 283, "metadata": {}, "outputs": [], "source": [ "#some general imports\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import math\n", "%matplotlib inline\n", "import warnings\n", "warnings.filterwarnings('ignore')" ] }, { "cell_type": "code", "execution_count": 284, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VdW5//HPc8YMkDCFOSHMozIFHBAFASecq9ZWrW2ttNaprf6s1tZq7cuqVatWb5Uq3vZWW8c6C4I4K0KCzDMyj2EIGc+41++PBEzIgYSck+xzdp7369X7Ivske39vTJ6ss/bazxJjDEoppZzDZXcApZRSiaWFXSmlHEYLu1JKOYwWdqWUchgt7Eop5TBa2JVSymG0sCullMNoYVdKKYfRwq6UUg7jseOinTp1Mvn5+XZcWimlUlZRUdEeY0xOQ59nS2HPz8+nsLDQjksrpVTKEpFNjfk8nYpRSimH0cKulFIOo4VdKaUcRgu7Uko5jBZ2pRzEGEO4ogorGrU7irKRLatilFKJt3XmfObd8Djlm3fj8nkYOO1cxjwwDZdXf81bG/0vrpQDFM9fxdxL7iZaGQQgGomy+um3CZWUM37GbTanUy1Np2KUcoDF9z1PtCpU51i0Ksg3/55LcF+pTamUXbSwK+UAB1Zthhj7F7v9Xiq2FNuQSNkpIYVdRNqJyCsiskpEVorISYk4r1KqcToVDEBc9X+drVCEtn262ZBI2SlRI/bHgJnGmEHAcGBlgs6rlGqE4XdeiTvdV+eYJyONITdfjLdthk2plF3iLuwikgWcCjwLYIwJGWNK4j2vUk5kjGHHR4tY9vBLbHzlY6KhcELO225wL8755FG6nT4ST0YamXmdGf3AtYy+7ycJOb9KLWJizMsd0wlERgDTgRVUj9aLgJuNMRWHfd40YBpAXl7e6E2bGtXLRinHiFQGmHXGbexb8g1WMIw7zYenTRpTP3uctr11ukQ1TESKjDEFDX1eIqZiPMAo4G/GmJFABXD74Z9kjJlujCkwxhTk5DTYdVIpx1l83wvsXbiWSHkVVjhCuKySwK4SPrnqT3ZHUw6TiMK+FdhqjPmq5uNXqC70Sqla1v3zfaKBuksSjWWxZ8FqgiXlNqVSThR3YTfG7AS2iMjAmkOTqJ6WUUrVYqJW7BcEjLYAUAmUqFUxNwLPi8gSYARwX4LOq5Rj9Ll8Ii6/t+5BEdoN7U1ax2x7QilHSkhLAWPMIqDBCX2lWrMRd13FtlkLKN+8m0h5FZ7MNFw+L6f9X71bUkrFRXvFKNVCfNltuODr6Wx+60v2zF9F2z7d6H35RHxZmXZHUw6jhV2pFuTyesi/eDz5F4+3O4pyMC3syjb7l21g1+fLyOjWkR5njcHt8zb8RUqpBmlhVy3Oikb5+Ir72PLWlyDgcrtxp/s55+O/kD0w1+54SqU87e6oWtzaGTPZ+s48olVBopXB6gd1ikuY+53f2x1NKUfQwq4azQpHCJdXxX2e1U+/RaQiUPegMZRt2Enp+u1xn1+p1k6nYlSDIlVB5t30V9b/aw4matG2TzdOfuqXdJswoknnO1LjK3EJVjAU8zWlVOPpiF016KPL7+Wb5z/ACoYxkSila7Yy59w72b98Y5PO1+d7p+NO89U77s3OJHtQXpxplVJa2NVRlW/exfbZRfV6nESDIZY99GKTzjnkpovJHpyHp006AC6/F09mGhOevzPmZhFKqWOTMlMxZRt2sOyRl9lbuIYOI/ox7JZLyerXw+5Yjle+YScuv7d+86qoRcmKprVe9mamc968J9n0+ufs+PBrMnM70//qM8jo3ikRkZVq9VKisO9bvJ53xt9MNBjChKPsKVrD+n/N5qy5D5MzZpDd8Rwte3BezHlv8XrIOXFIk8/r8nrofelp9L70tHjiKaViSIn3vfNu+iuR8ipMuLoDnolEiVQEmHfD4zYnc770zu3pd/VZuDP83x4UwZPuY9gtl9oXTCl1RCkxYi+eF7sL8J7CNRjL0nnZZnbSkzeR1b8HKx57lVBJBV1PO56CB39Km7wudkdTSsWQEoXd0yaD0P6y+scz/FrUW4C4XAz71aUM+5WO0JVKBSlRFQdddz7udH+dY+50HwOvnWpTIqWUSl4pUdhH/v4H9LroFNx+L97sTNxpPnpOPZHRf9Id2JVS6nBijGnxixYUFJjCwsJj/rqKbcUcWL2VrP49aJPbuRmSKaVU8hKRImNMg5sapcQc+0GZPXLI7JFjdwyllEpqKTEVo5RSqvESVthFxC0iX4vI24k6p1IqORljWP+v2bxR8DNe7nsF825+gqrd++2OpWokcirmZmAlkJXAcyqlktCC255m9VPftl9e/dRbbHzlYy5aNgN/+7Y2p1MJGbGLSE9gKvBMIs6nlEpeVbv3s+rJ1+v01LfCEUL7y1n11Js2JlMHJWoq5lHgNsBK0PmUajbGstg2u5BlD7/Epjc+xwpH7I6UUvYuXIvLX7/tcjQQYvvsIhsSqcPFPRUjIucCu40xRSIy4SifNw2YBpCXpz23lT3CZZW8O+GXlK7dhhUK4/J78XfI4tzPH09od8nyTbso37yL9kPz8Xdw1uxkRo9Oh/o21SZuF217d7MhkTpcIubYxwHni8g5QBqQJSL/MsZcWfuTjDHTgelQvY49AddNesH9ZWx8+WMCew7QdcIIOp80BBGxO1arVnTns5Ss2IQVrN7FyQpFiFYG+eyahzjjvfvjPn+4rJIPL7uHnR8vweX3YgXDDLr+AsY8+FPH/LfvcFwfsgfnsW/xekzk2wLv8nsZcvPFNiZTB8Vd2I0xdwB3ANSM2G89vKi3Rrs+W8r759wBlkU0GMad5qP7lNFMfPn3uNxuu+O1Wutf+OBQUT/IRC22f7CQaDCEO8YUw7H4fNrD7PhoMVYwfKiH/eq/vUX2wDwG/uScuM6dTM549098ePm9FH+xHPG48WSkMe7vt9Dh+L52R1Ok2ANKqcKKRpn7nbuJ1Nr4OVIRYPvsIr55YS79rppiY7rWzUSPfBso3oeww+VVbHr983p/OCKVAZY/8pKjCntaTjvO/uBhqnbtI3SggrZ9u+uAJYkk9AElY8xHxphzE3nOVLS3aC3RQLDe8UhFgLUz3rMhkToo/5JTcXkPG8+4hM7jhuGJsQ/rsQiXVR5xuiW4r353UidI79KB7AG5WtSTjD552hyOOvRrFbcXklbB/deSmdf50H6rnsw0/B2zOOWZW+M+d3rXDvg71r9RKi4X3U4fGff5lWosnYppBh0LBuBO8xEuq6pz3JOZRv8fnW1TKgWQ1jGbi5bPYNNrn7Jv0TqyBuTS+7sT8Gamx31uEeHkp37Jh9/9Q/X8umVw+Tx4MtMY/ccfJyC9Uo2TUt0dU8nOT5cwe+pvMJZFNBDCk+6n++RRTHzlbn3b6nB7v17L0j+/SOnabXQ99XiG/uoSbV6nEqKx3R21sDej4L5SNrxUvdyx28QRdD55qGOWvCWTaDBE0W+eZc0z7xCuCNBl3DBOfOImOhzXx+5oSiWUFnbVasy54Ldsn1NEtCp06Ji3bQYXLntW+/YrR2lsYdebp8p2xhh2frqEVU+/xY6PFnEsg43S9dvZPrtuUYfqUfyKx19LdFSlUoLePFW2CpVWMHPSLRxYvQUTNdWPpfftxtlzH2lUl8ADq7fg8nsPPQx0kBWKsHfh2uaKrVRS0xG7stWCW59i/9INRMoDRKuCRMqrOLByM1/d/ESjvj57YG69B4IAXD4PHUf1T3RcpVKCFnZlq2/+PRcrVLe7ohWKsOHljxs1JZPVtzvdp4zGnV734SK338eQm7RviWqdtLArWx2pZW7t5lINmfjSXQz66Xl426aDS+gy/jjO+fRRvXGqWi2dY1e26nHWWLa8PQ+sb3u4iMtF98mjGr001O33MfaRnzP2kZ83V0ylUoqO2JWtTnz8BtI6ZeHJTAPAnZGGr0NbTnryZpuTKZW6dMSubNUmrwuXrP0/1v1rDvsWr6PD8X3pe+VkfFmZjfr68k272PjKx0SDYfLOP5n2w3o3c2Klkp8+oKRS1prn3mPe9Y9jLIOJWrh8HobceCEF90+zO5pSzUIfUFKOVrV7P/Ouf5xoIIQVCmOiUaJVQVY88TrFC1bZHU8pW2lhVylp6ztfIZ76P77RQIgNL37U8oGUSiJa2FVqkkP/pz6XNlpTrZsWdpWScs89KeZad3eajz7fnWhDIqWShxZ2lZLSOmUz7u+34E7z4U7z4fJ5cKf5GPqrS+g0eoDd8ZSylS53VCmr7xWT6TphOJte/ZRoKELeeSeRPTDX7lhK2S7uwi4iucA/ga6ABUw3xjwW73mVaozMHjkp1RMmWFLOplc+JrCnlG4TR9Bp7CDdfEUlXCJG7BHgFmPMQhFpCxSJyGxjzIoEnFuphDCWxbZZC9j+4SIyunag7xWTSO/SoUUz7PpiOe+f/WuwDNFgGLffS48zxzDhxd/pdokqoeIu7MaYHcCOmn+XichKoAeghV0lhWgozKwzbmPvwrVEyqtwp/n4+vf/y+Q3/0i3iSNbJIMVjTL3O78nUmuD80gkyrZZC/jmhbn0u2pKi+RQrUNCb56KSD4wEvgqxmvTRKRQRAqLi4sTeVmljmrNM++wp3A1kfLqohoNhIhUBPjo8j9iRRvfRTIee4vWEqkM1DseqQiwdsZ7LZJBtR4JK+wi0gZ4FfiFMab08NeNMdONMQXGmIKcHN2xXbWcdf+cTbQyWO94JBBk3+L1LRPCmCOtuj+mrQCVaoyEFHYR8VJd1J83xuhGkyqpuLxHmL82BpenZea2OxYMwOX31TvuyUyj/w/PbJEMqvWIu7BL9S39Z4GVxphH4o+kVGINvPbcQ22Ba/N3yKL9cX1aJIPL7eb0V+7G0yYNd7ofRPBkptFt4gj6Xqnz6yqxErEqZhxwFbBURBbVHPuNMebdBJxbqbj1vXIyW9+bz+a3vqjuAun14PK4mfTfP7ToUsOupx7PpRteYMOLHxHcc4CuE0fQ5ZTjmi1DJBDC5XG32LsS9a2yjTvZ8cFCfNmZ9Jx6Ip50f4teX9v2qlZj79dr2fnJEtI7tyfvgpPxZNQfxTvB7nkr+OKnj1CyYhPicdP3ismc8Nj1eDPTm+2akaog294vJFoZpPvkUaTltGu2ayW7Bb+ezsq//hdxuxCXC1zCmTMfIOeEwXGfu7Fte7WwK+Ugpeu388aIa4lUfLsCx53mpetpwznjvQea5Zo7P13CnPPuBAMGgwlHGX3/Txh603ea5XrJbNv7hdXLWivqroDyd8ri8u2vxP3uSfuxK9UKLX/0FaLBcJ1j0UCYnZ8spXTdtoRfL1IVZM55dxIurSRcVkmkrIpoIETRHc+yd9G6hF8v2a155p16RR3ACkXY9dnSFsuhhV0pB9m/dEPMrpcun6dZCvu2WQsgxpv+aDDM2udmJvx6yS5SFTria9HAkV9LNC3sSjlIzgmDcfm89Y5bwTDthuYn/HqRigAmVmW3LMKllQm/XrLr873TY67AMhGLLuOPa7EcWtiVcpChN1+MO90HtVbauNP99Lp4PG1yOyf8et0nj8KE679D8LRJI/+SUxN+vUQzxrDp9c+YOflW3hxzHUse/A/h8qqGv/AIel82gc7jhuFpU13cxevBne5n3DO3NOvN68PpzVOlHObA6i3Mv/Vv7PxwMZ626Qy+/gKOv/37zbbscdmjr7DwzhlYwTDGsqrX508axaTX7qleFZLECm+fzson3zg0L+5O99G2dzfOW/C3Ji9RNJbF1pkL2PLWF/g7ZtHv6jPJ7t8zIXl1VYxSqsXsKVrD2udmEimvIv+SU+l5zglJX9Qrt+/h5b5XYh12s9mTkcYJj/6cAT+ZalOyI2tsYdeNNpRSces0ekDK7Vy1+8sVuH3eeoU9UhlgyzvzkrKwN1Zy/0lVSqlmkpbTLuaNX3G7yOjRyYZEiaOFXSnVKnU5ZRhpHbLAVbelg8vvZdDPzrcpVWJoYY+hbMMOlv/lFZY/+iplG3faHUcp1QzE5eKsDx4ie2Aunsw0vFkZeLMzGf/cr2k/rLfd8eKiN08Ps/yxVym645lDPbJFhIIHpzHkhotsThbb/uUbWfKnF9i3eD0dhvfl+Du+T/tmWK+slFMZYziwajPh0ko6jOyHO8ZzAMlCV8U0Qen67bx+3DX1nhBzp/m4aMVztM3valOy2Iq/WsnMSbcSDYQwloW4XLjTfJw5+0E6nzTU7nhKqQTTXjFNsOm1TzGWVe+4MYZNr31qQ6Kjm3fTX4lUBg5lNpZFpDLAvJuesDmZUspOWthrMyZm34vq48m3fdmeojUxj+/9eq1ut6ZUK6aFvZa8C8ch7vrfEnG5yLvwFBsSHZ0vu03s41mZLbqBhFIquWhhryV7QC4j7r4ad7oP8bpxeT2403yMuvdHZPXtbne8eobcdBHujLqPPbsz/Ay+8UKbEimlkoE+eXqY42+7nF4XjmPTa5+BQK+Lxyesz0OiDf/tlVRu38v6/5uNy1/9BF3f709ixO9+YHc0pZSNdFWMAwT2HKDsmx207dONtE7ZdsdRSjWTFl0VIyJnichqEVknIrcn4pyq8dI6ZZMzdpAWdaUUkIDCLiJu4EngbGAI8D0RGRLveZVSSjVNIkbsY4F1xphvjDEh4D/ABQk4r1JKqSZIRGHvAWyp9fHWmmNKKaVskIjCHmvBdL07siIyTUQKRaSwuLg4AZdVSikVSyIK+1Ygt9bHPYHth3+SMWa6MabAGFOQk5OTgMsqpZSKJRGFfQHQX0R6i4gPuBx4MwHnVUop1QRxP6BkjImIyA3ALMANzDDGLI87mVJKqSZJyJOnxph3gXcTcS6llFLx0V4xSinlMFrYlVLKYbSwK6WUw2hhV0oph9HCrpRSDqOFXSmlHEYLu1JKOYwWdqWUchgt7Eop5TBa2JVSymG0sCullMNoYVdKKYfRwq6UUg6jhV0ppRxGC7tSSjmMFnallHIYLexKKeUwWtiVUsphtLArpZTDaGFXSimHiauwi8ifRWSViCwRkf+KSLtEBVNKKdU08Y7YZwPDjDHHA2uAO+KPpJRSKh5xFXZjzPvGmEjNh/OAnvFHUkopFY9EzrH/GHgvgedTSinVBJ6GPkFE5gBdY7x0pzHmjZrPuROIAM8f5TzTgGkAeXl5TQqrlFKqYQ0WdmPM5KO9LiJXA+cCk4wx5ijnmQ5MBygoKDji5ymllIpPg4X9aETkLODXwGnGmMrERFJKKRWPeOfYnwDaArNFZJGIPJWATEoppeIQ14jdGNMvUUGUUkolhj55qpRSDqOFXSmlHEYLu1JKOYwWdqWUchgt7Eop5TBa2JVSymG0sCullMNoYVdKKYfRwq6UUg6jhV0ppRwmrpYCyh6WZfhmzR7C4Sh9B+bg87ntjqSUSiJa2FPMpm/28cgf5xKoCiMiWJbhmhtO4oRT8u2OplKYMYb1q/dQUlJFvwGdaNchw+5IKg5a2FNIOBzlgbtmU1EeqnP8mce/oFefDnTtnmVTssYJh6NYUQt/mtfuKKqWvcUVPHDXbPbtrSQStjDGkJHp48qfFDBuYl+746km0MKeQpZ+vZ1o1Kp3PBq1+GTOOi77wSgbUjWsrDTAjCfnsbhwG8YY8nq355obTyYvv73d0RTw6H0fsntnGbW3yamsCDHjyXlUVoSZcu4g+8KpJtGbpymkoixErD2qolFD2YFAywdqBGMM9/9uNosLtxGNWliWYeP6fdz3m1mUllTZHa/V27WjjJ3bSmP+XEUiFq/9e1HMwYRKblrYU8igYV2wYvyS+dM8DC/oaUOihq1dWUzxrvJ6xSEStvh4zjqbUqmDAlVhXG454uvhsEVZabAFE6lE0MKeQnK6tGHS1EH4/d/OoPn9Hnr16cDIsclZ2HfvLIt5PByOsn3LgRZOow7XI68dLteRy4BLhMw2vhZMpBJB59hTzOVXj2LIsK58OGsNgWCEk0/tzcmn9cbtTs6/0bn57Ym1x7nP76bPgE42JFK1eTwurrnhRP728GdEInXfVfl8bqacOxCvV5fTphot7ClGRBhe0IPhBT3sjtIovfp0oP+gzqxZuZtwKAqAyyVkZPg4ZWIfm9M1jmUZwuEoPp8bkSNPW6SqgpN6cc8j2Tz/zALWrNxNNGLh83s46/zBXHj5cLvjqSaQWKOp5lZQUGAKCwtb/LrKHqFQlDdfWsLHc9YRDkUZOaYnl109ivZJvlbailq8/uISZr21klAwSrsO6VxxTQEFJ/WyO1qzMcYQCETw+9y4kvRdYGsmIkXGmIIGPy8RhV1EbgX+DOQYY/Y09Pla2FUqePEfRcx5dzWhYPTQMZ/PzS/unMjQ4d1sTKZaq8YW9rj/JItILjAF2BzvuZRKFqFQlDnv1C3qB4+/9u/FNqVSqnES8V7rL8BtQMvP6SjVTMoOBOAI0+m7d8Re6aNUsojr5qmInA9sM8YsbuimkohMA6YB5OXlxXNZR9lbXEHRvM1YlmHUCbl07trW7kgpaf++SnZsPUDnrm3p1LlN3OfLbpdWswwwWu+1XH1iViW5Bgu7iMwBusZ46U7gN8AZjbmQMWY6MB2q59iPIaNjfThrDc8/WwjGYIBX/rWIi68YzjkXDrU7WsqIRi2e/euXzP98Ix6vm0jEYtiIbvz81lPj6nrp8bq54LLj+O9/FtedY/e7+c4VIxIRXalm02BhN8ZMjnVcRI4DegMHR+s9gYUiMtYYszOhKR1o354Knn+mkHC47ojwtRcWM3JMT7r1yLYpWWp5+9VlLPhiE+GwRThcvQ572aId/HtGIVf/7IS4zn32hUNo09bPmy8v5cD+KnLz23P5j0bTV9ffqyTX5KkYY8xSoPPBj0VkI1DQmFUxCoq+2hJzDteKWsz/fBMXXHZ8y4dKQXPeWU0oVPePYzgU5dO567lq2lhcrqavOxcRTp3cj1Mn94s3plItSheq2sVArM5LNjxWkNKqqsIxj0dqWgQr1RolrLAbY/J1tN54o07IJdaQ3e1xUXCSs28ul+yrZOe20oQU3gGDc2Ie79mrPR59FF61UtpSwCYdczK57OqRvPTPr7GiBjC43C7Ou2QYPXLb2R2vWZTsq+SJP3/CxnV7cblc+NM8/OTGk+Nqj/D9a8Zw76/fIxyKEo0aXC7B63Xzwzjn15VKZdpSwGa7dpRR+OUmrKhh9El5dO/pzJumxhjuvOktdmwrxbK+/Znz+d3c8/DUuP7/3rO7nJlvrmTD2j307NWOsy8YStceyb2blFJN0dgnT3XEbrMu3doy9eJhdsdodhvW7WVPcUWdog7Vfdk/eHcVV01r+gi7U+c2XPmTMfFGVMoxtLCrFlGyrypmZ0TLMuzeVWFDItWSQsEIb7+2jM/mfoMxhhPH9+b8y44jPV33v20OWthVi8jv15FopP5TnD6fWxtqOZwxhgd+P4dN6/cdem7j/bdXsvTrbdzz8NSk3Usglel3VLWIDh0zOHVKf3z+b1equD0u2malcdrkvjYmS147t5fy6guLeOHZBaxcujPmhiWpYNWyXWzZuL/Ow3iRsEXxznIWF26zMZlz6YhdtZirrh1Dn/4def+tVVRVhhl9Yi7nXjyM9Azdeu1wn36wjn88PR8rahG1DB+9v44RY3py3S2npNxmHxvW7SUSrv9uLRCIsH7tnpqlvyqRtLCrFiMinDKxL6dM1BH60VSUh/jH0/MP7TgFEAxGWFS4lSVF21Nm96yDOuZk4vW6iUYjdY77/R5yEtCwTdWnUzFKJZkVS3bEnHcOBiLM+3SDDYniM3JsLj6/hzpvNAQ8XhcnnOLc3ajspIVdqSTj9rhitoIXISWfpvX53Pz2/jPJ79sRj8eFx+sit1d7fnPfmToN10x0KkapBNqx7QDFu8rJzW/f5D1dhw3vhomxb43X52b8pNScxurSLYu7HzqHstIAxjJktUu3O5KjaWFXKgEqK0I8dt9HfLN2D26Pi0g4yskT+vDD60485g6TPr+Hm26fwGN/+ggBLGMwFpx9wRAGDO7c4Ncns7ZZaXZHaBW0sMdQVhrgP/9bROGXWxCBE07J57IfjCKzjb5tVLHN+J95rFtTTCRsQc1Nzy8/2UDPvHaccd7gYz7f0OHdeGzGd1g4fyuhYIRhI7qT00VvNKrG0cJ+mEg4yh9ue4+9xZVEa7oPfjZ3PWtX7uaPj56LSx+mcBwrarFk4Xa2bi6ha48sRhT0xONp/H/nYDDC119tIRKp260yFIzy/turmlTYAdIzfIyb0KdJX6taNy3sh1k4fyulJYFDRR0gErHYW1zB0kU7GD46tZaaqaMrLwvyx9tnsn9vJaFQFJ/fTZu2fn53/1m0a+QceSgYOeJrVZWhREVVqtF0+HmYLRv3EwjU/0UNhaNs3bTfhkSqOf3nuSKKd5UTCESwLEOgKsK+PZX871NfNfocbdr66dCp/h8BERg2snsi4yrVKFrYD9Ole1v8afXfyPh8brp011awTjP/i031plAsy7C4cFujNwIREX58/Un4/O5DN0o9XhcZbXxccsXIhGdWqiE6FXOYsSf34qV/LCQcih5qMetyCZmZfkYU9LQ5nUq0RPVfGXxcV/7wyFRmvbmSHdtK6T+4M1OmDiRbl/UpG2hhP4zP7+F3D5zNc//zJSuX7gKBYSO68+PrTzymG2oqNYw+IY/5n28kGv22wLtcMGxEt2O+Ud6tRzY/vO7EREdU6pjFXdhF5EbgBiACvGOMuS3uVDbL6dKG2+6ZcugtuhZ05/r+NQWsW11M6YEAwUAEf5qH9AyvFmiV0uIq7CIyEbgAON4YExSR1H564jBa0J0vKzuN+584n4Xzt7Jl43669cyi4KRe+Hyp9+i+UgfFO2K/DrjfGBMEMMbsjj+SUi3L43Uzdlwvxo7ThlTKGeIdkg4AxovIVyLysYjoxpOq0XbtKGXFkh2UHgjYHUUpR2lwxC4ic4CuMV66s+br2wMnAmOAl0Skj4mx1EBEpgHTAPLy8uLJrFJcZUWIx//0EevW7MHjcREORzn9rAF8/8cFKbeJhFLJqMHCboyZfKTXROQ64LWaQj5fRCygE1Ac4zzTgekABQUFqbnHl0qIZ5/4krWrq/uqHNxM4qM52c/gAAAKFUlEQVT319IjN5sJZwywOZ1SqS/eqZjXgdMBRGQA4AP2xBtKOVdVVZhFhVurm2XVEgpGmfnGSptSNV3J/ioWfLGJlUt3NvqBJqWaW7w3T2cAM0RkGRACro41DaPUQYGq8BGnWyrKU6uvyqvPf827r684tHoqPd3Lr++dQrce2TYnU61dXCN2Y0zIGHOlMWaYMWaUMWZuooIpZ2rXPp22bf31jrtccFwK9VVZXLSNWW+uIhK2CFRFCFRF2L+/iof/MDdhT7Mq1VS6ULuVKdlfxUv/XMi9t8/kmb9+wdbNJS16fRHhR9efiM/vRg72VfG4SM/0cfH3h7dolnh88O5qgod3dTRQeiDA5g3aLE7ZS1sKtCLFu8r5/S3vEAxGiIQtvlmzh68+28jNd0xg2IiWGy0fP6oHv3vgbGa+sYJd20sZNKwLU84dTLv2qdNXpfII7XhdLqGqKtzCaZSqSwt7K/LKv76msiLEwZkCyzKEglFmPDmPh6df1KJLDfPy2zPt5nEtdr1EGzsun03r9xGqWdVzkGUZ+vTvZFMqparpVEwrsnzxDmJN/x4oqaJMHxI6JhOm9KNrjyz8/urWAy4X+PxufnjdCdqOQNlOR+ytSEamj7LSYP0XDDF70LdmwWCEwi83U7Kvkr4Dcxg4pHOddzQ+v4e7HjybeZ9s5OsFW2jXPp2JZw4gN7+9jamVqqa/za3IGecN5sV/FBEKfjt94PG6GDk2F3+a18ZkyWXr5hL+dOcswjUPUHl9bvr078gtd03C6/12NO71uhk/qS/jJ/W1Ma1KFaUlVVRVRcjp0ubQhizNRQt7K3L6WQPYvqWET+asw+N1E4lY9BuYwzXXa4va2p588BPKy0NQM20VDERYt3oP77+9iqkXDbU3nEo5pSVVPPnQp6xbXYzLJaSne7nmxpObdf9ksWPNbUFBgSksLGzx66pqB0qq2LqphE6dM+nSzTnb/QWqwnw4cw0LvtxMZhsfk88ZxPCCY/vl2bO7nNtvePNQq4PauvXM4v4nLkhUXNUKGGO465fvsHVLCVatzVx8fjf3PDSV7rnH9jCbiBQZYwoa+jwdsbdC2e3SHbdlWzAY4Z7b3qN4V/mhorxq+S7OuWgoF13e+PXxxsCR3iQb7RigjtGmb/axa0dZnaIOEAlbzH5nFVf/7IRmua6uilGO8PmH69mzu7zOSDsUjPLOq8soLalq9Hk6dc6kfceMese9PjfjJvZJSFbVeuzfW4nLXX+oYFmG3TvLmu26WtiVIywq3FbnpvBBHq+bdWsa35dORLj+/51KeoYXX81SxrQ0D3n57Tnr/MEJy6tah/x+HYmE6/9cen1uhg7v1mzX1akY5QjZ7dIRl2Csum95jWVoE6M3zdH06tOBR/5+MV99tpH9eyvpNyiHYSO6N/tKBuU87TtkcOqU/nz2wfpDLSjcHhdt2vg4bUr/ZruuFnblCJPPGci8TzbUeRJUBNpk+ek3MOeYz5eR6WPimdobXsXvqmvH0LtvR2a9tZKqyhAjx+Zy/iXDyGzja7ZramFXjtCrTwd+eN2J/OPprzCWIRyO4na7GDmmJ4GqMBmZzfdLpNTRiEiLP++gc+zKMcZN7MOEM/tjTPXqlkjE4qP313L3re8S0MZcqhXRwq4co2R/FR+8u5pwrZtV4bDF/n2VfDp3vY3JlGpZWtiVY3yzZg9eT/0GXKFglCULt9uQSCl7aGFXjpHVLg0rxpPULpfQIcbadKWcSgu7coy+AzrRrn16vWWJHo+LyecMtCmVUi1PC7tyDBHh13+YQm5+O3w+N2npHjIyfVz7i3HaTle1KnEtdxSREcBTQBoQAX5ujJmfiGBKNUXHnEz+8Mi57NpRRlVliJ692uPx6PhFtS7xrmN/ELjHGPOeiJxT8/GEuFMpFacu3draHUEp28Q7lDHAwb6v2YAuPVBKKZvFO2L/BTBLRB6i+o/EyUf6RBGZBkwDyMvLi/OySimljqTBwi4ic4CuMV66E5gE/NIY86qIXAY8C0yOdR5jzHRgOlRvtNHkxEoppY6qwcJujIlZqAFE5J/AzTUfvgw8k6BcSimlmijeOfbtwGk1/z4dWBvn+ZRSSsUprj1PReQU4DGqR/4Bqpc7FjXi64qBTU2+cNN1Ahq/60LL0mxNo9maRrM1jd3ZehljGuxDbctm1nYRkcLGbARrB83WNJqtaTRb0yRzttr0yQ2llHIYLexKKeUwra2wT7c7wFFotqbRbE2j2ZommbMd0qrm2JVSqjVobSN2pZRyvFZR2EXkUhFZLiKWiBTUOt5RRD4UkXIReSKZstW8doeIrBOR1SJyph35amUZLiJfishSEXlLRLIa/qqWISIjRGSeiCwSkUIRGWt3poNE5MWaXItEZKOILLI7U20icmPNz9dyEXnQ7jwAInK3iGyr9X07x+5MhxORW0XEiEgnu7PEEm+vmFSxDLgYePqw4wHgd8Cwmv/ZIWY2ERkCXA4MBboDc0RkgDEmWv8ULeIZ4FZjzMci8mPg/1H9vUsGSdtl1Bjz3YP/FpGHgQM2xqlDRCYCFwDHG2OCItLZ7ky1/MUY85DdIWIRkVxgCrDZ7ixH0ipG7MaYlcaY1TGOVxhjPqO6wNviSNmo/oX7jzEmaIzZAKwD7ByJDgQ+qfn3bOA7NmY5XNJ3GRURAS4D/m13llquA+43xgQBjDG7bc6TKv4C3Eb1z11SahWFPUX1ALbU+nhrzTG7LAPOr/n3pUCujVkO9wvgzyKyBXgIuMPmPLGMB3YZY5Kp7cYAYLyIfCUiH4vIGLsD1XKDiCwRkRkikjTbX4nI+cA2Y8xiu7McjWOmYo7WhdIY80ZL56mtidkkxrFmHSE00Mnzx8DjInIX8CYQas4sx5it0V1GWzpbrf++38OG0XoD3zcP0B44ERgDvCQifUwLLJVrINffgHup/nm/F3iY6p+/FtFAtt8AZ7RUlqZyTGE/WhdKuzUx21bqjop70sxTDI3IeQaAiAwApjZnlsMlc5fRhr5vIuKh+j7K6JZJ9K0Gvm/XAa/VFPL5ImJR3Qul2M5ctYnI34G3mzlOHUfKJiLHAb2BxdUza/QEForIWGPMzhaM2CCdiklebwKXi4hfRHoD/QHb9pM9eGNNRFzAb6ne6zZZJHuX0cnAKmPMVruDHOZ1qr9fB/9Y+0iC5lsi0q3WhxdRPQ1oO2PMUmNMZ2NMvjEmn+rB16hkK+rgoBH70YjIRcBfgRzgHRFZZIw5s+a1jVTfePOJyIXAGcaYFXZnM8YsF5GXgBVUbxR+vY0rYgC+JyLX1/z7NeA5G7Mc7lrgsZqRcYCanbqSyOUk103Tg2YAM0RkGdVTa1e3xDRMIzwoIiOonorZCPzU3jipR588VUoph9GpGKWUchgt7Eop5TBa2JVSymG0sCullMNoYVdKKYfRwq6UUg6jhV0ppRxGC7tSSjnM/wf4CH3U3V52MQAAAABJRU5ErkJggg==\n", "text/plain": [ "