{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# MATH80629A\n", "# Week \\#3 - Supervised Learning - Exercices\n", "\n", "In this practical session we will explore several classification models." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cloning into '80-629'...\n", "remote: Enumerating objects: 797, done.\u001b[K\n", "remote: Counting objects: 100% (23/23), done.\u001b[K\n", "remote: Compressing objects: 100% (12/12), done.\u001b[K\n", "remote: Total 797 (delta 12), reused 20 (delta 11), pack-reused 774\u001b[K\n", "Receiving objects: 100% (797/797), 101.99 MiB | 2.58 MiB/s, done.\n", "Resolving deltas: 100% (462/462), done.\n" ] } ], "source": [ "import pandas as pd\n", "import numpy as np \n", "import matplotlib.pyplot as plt\n", "import sklearn as sk\n", "%matplotlib inline\n", "from sklearn.datasets import make_classification\n", "\n", "# Code to obtain utils.py\n", "!rm -rf 80-629\n", "!git clone https://github.com/lcharlin/80-629/\n", "import sys\n", "sys.path += ['80-629/week3-Supervised/']\n", "\n", "\n", "from utils import generate_data, plot_predictions, plot_svc_decision_function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### First let's generate three datasets and plot the training and testing splits\n", "\n", "We will use 3 distinct binary classification datasets. Each datum is in two dimensions. \n", "\n", "Each dataset contains a training set and a test set. Each of the datasets was generated by a distinct data generative process, resulting in distinct class separations.\n", "\n", "We start by visualizing the three datsets." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAy8AAAE/CAYAAABPZPyCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5QcdZnv8c8ziQRWMAkkkCEwk7AbQiBI1DbCCiYG1OCugb0oovESveFkA4vcRfQYjYf1ZtclevFXRHbNYhRWIEpYMHsXT0Aw2R+CMlkTQ1AgxERifpsQyE8zzHP/6GrodLpnerqrq+vb9X6dM2e6u6qrn6npfr71fL/fqjZ3FwAAAACkXVuzAwAAAACAalC8AAAAAAgCxQsAAACAIFC8AAAAAAgCxQsAAACAIFC8AAAAAAgCxQsQCDObYWY/anYcAAAAzULxgtiZ2QYzO2BmL5vZi2b2UzObbWZVvd/MbJSZuZkNbHCcfb6OmY03s2VmttPMqv5SJDObbmZ7o58DZtZTdH9vLfG6+53ufmktzwWA3pC3G5O3o+2eZWbdtT4fwJEoXtAo73P3EyR1Spov6dOSvt3ckGpyWNIPJM3sz5Pc/W53P97dj5d0qaTNhfvRY0dodIMPAFUgb/cjbwNoDooXNJS773H3pZI+KGmGmY2XJDP7MzP7hZm9ZGYvmNnni57279HvF6MerwvM7I/N7DEz+33Um3a3mQ0pPMHMPm1mv4t6DZ8xs4ujx9vMbI6ZPR899wdmdmKl1ykT/zPu/m1Ja+PdM5KZbTKzT5nZGkn7o8c+Z2bro79jrZlNK1r/GjNbHt0eGPU+/qWZrTOz3Wa2IO4YAWQPebsyMzvdzH4Y/T3rzWx20bK3F+2frWZ2S1HMA4pGcd4Ud1xAllC8IBHu/nNJmyRdFD20T9LVkoZI+jNJ15rZ5dGyd0S/h0Q9Xo9LMkm3SDpV0jhJp0v6vCSZ2VhJ10t6a9Rr+B5JG6Jt3CDpckmToufulvTNXl4naVcp38M3OLr/rKS3R/e/IOkeMzull+e/V9JbJL1J0kfM7JIGxgogQ8jbRzKzAZIekvTTKK6pkj5rZpOiVW6T9Pfu/gZJYyQ9WBTzK0WjOL9IKmagFVG8IEmbJZ0oSe6+3N3XuHuPu/9S0r3KN1Rlufs6d3/E3Q+5+w5JXyla/xVJgySdbWavc/cN7v58tOwvJc11903ufkj5hvP9KZqm9fUotgOS5O4/cPct0X65R/nGPNfL82+Jekk3SFouaUKjAwaQKeTt11wo6Vh3/6K7/8Hdn5X0HeU7oaT8dLUzzewkd3/Z3X/WtEiBFkbxgiSNlLRLkszsbWb2EzPbYWZ7JM2WNKzSE83sZDNbHE0xeEnS9wrru/s6SX+tfAO3PVrv1OipnZIesPwJqC9K+pXyjWZvoxlJeqH4jpl91MxWF8V7lnrZL5K2Ft3eL4l52QDiRN5+TaekUYW4otg+IWlEtHyGpDdKetbMfmZm72lWoEAro3hBIszsrco3gv8ZPXSPpKWSTnf3wZL+UfkpBpJU7uowt0SPvzEakv9I0fpy93vc/ULlGxeX9MVo0QuSLnX3IUU/x7r77yq8TtJejcHMzpD0D5KulXSSuw+R9GsV/Z0AkBTy9lFekPTrkrhOcPe/kCR3/5W7f1DSyZIWSPoXMzumyTEDLYfiBQ1lZm8wsz+XtFjS99x9TbToBEm73P2gmU2U9OGip+2Q1CPpjKLHTpC0V/mTNEdK+lTRa4w1sylmNkjSQUkHlO+lk/KN6xfMrDNad7iZXdbL65TGb2Z2rKRjovvHRq9TWP5dM/tu9XukV8cr38jtiF76GuVHXgAgMeTtiv4zev5fR9scaGZvNLM3R49fHU0Ze0XSHuXzeY+k7cqfsN9Rw2sCKEHxgkb5VzN7WfmeqrnKz3X+WNHy6yTNi9a5WfnLWkqS3H2/8ier/1c0NH++pP8j6c3KNwj/JulfirY1SPnLeu5UfhrVyZI+Gy37uvI9hQ9Hr/WEpLf18jqlOpVvVAtXrTkg6Zmi5adL+q8q90mvojnkCyT9XNIW5QsX5kwDSAp5uxfuflj5i6T8qaSNyhdS/6DXpuv+uaRnophvkXSlu3e7+25JX5K0MoqZcxOBOpg7o5lALaLpAKuVnxJxuNnxAAB6R94GwkfxAgAAACAITBsDAAAAEASKFwAAAABBoHgBAAAAEASKFwAAAABBGNjsAGoxbNgwHzVqVLPDAIB+W7ly5U53H97sOJJEzgYQqizm7LQLsngZNWqUurq6mh0GAPSbmW1sdgxJI2cDCFUWc3baMW0MAAAAQBAoXgAAAAAEgeIFAAAAQBAoXgAAAAAEgeIFAAAAQBAoXgAAAAAEgeIFAAAAQBAoXgAAAAAEgeIFAAD025zJd2nO5LuaHQaAjKF4AQAAABCEgc0OAAAAhKMw2vLUio1H3J+//OqmxtOs1weQLEZeAABArJhSBqBRGHkBAKAFNWpEorC9Rm2/3HZ7eywtI0AAkkHxAtSARhIAjkZBAaDRKF4QLBpFADhaUgVEo0ZciuNev2qrzpgwouzf0ugRIADpRPEC9AO9igBQGQUFgEajeEFwKCAAoLJQC4je4u7tbwnl7wMQD4oXNEVojWpBqAcFAJAkciOARqF4QXAoIADgaKU5MdTcyOgKgN5QvCBRjZ7ylVRBQ0MKAACQPIoXBIsCAgA4DxBAtlC8IFGNmvJF4w0AjUE+BZAmFC8AAASM8wABZAnFC5qiUV+WRuMNAPFgRBtAGlG8AADQAigqAGQBxQtaCo03AMSDEW0AadTW7ACAJM2ZfNerDTEAAADCEkvxYmaLzGy7mT1VYbmZ2QIzW2dmvzSzNxctm2Fmz0U/M+KIB0gjCiekBTkb/TF/+dWMugBIjbimjX1X0m2SKh2ZXSppTPTzNkn/IOltZnaipL+RlJPkklaa2VJ33x1TXKhTq0wX4MRT4AjfFTkbABCgWIoXd/93MxvVyyqXSbrL3V3SE2Y2xMzaJU2W9Ii775IkM3tE0lRJ98YRF5qrrwIhKwUEhRPShpwNAAhVUifsj5T0QtH9TdFjlR5Hk7XaATcnngL9Qs5OGLkJAKqTVPFiZR7zXh4/egNmsyTNkqSOjo74IkPs+ip8Wq0w6guFEwJEzkZDkQ8B1Cqp4mWTpNOL7p8maXP0+OSSx5eX24C7L5S0UJJyuVzZxhLxadUD7lb5O4AGI2cnJGudOY3APgOyJaniZamk681ssfInf+5x9y1mtkzS35vZ0Gi9d0v6TEIxoUH6KnxatTDqS1b+TrQEcnYLSkPOpVgDUK9Yihczu1f53rhhZrZJ+avRvE6S3P0fJT0k6b2S1knaL+lj0bJdZva3kp6MNjWvcCIo0oEGBWg95Oz0yGpnThwohIBsiutqYx/qY7lL+qsKyxZJWhRHHEiXvhqQZjQwNG4AOTtrKh3kFySZD4uLtfWrtib++gDCl9S0MaRANQfuIRzchxAjAPQH+az/GLUCsoniBZnA9AIAWVV6kF/QrHxYeL39ew7pqRUbyccA+oXiJQOqOXAP4eA+hBgBAOWVFk9xoQ0AsoXiBZnA9AIAWVea95qVD8nHAOpB8ZIB1TQU5daZM/kuzZl8V2oalv42eOXWK5wgCgBIRl8XDACA/qB4QaacMWFEs0MAgFRodsdUs18fQJgsf0XMsORyOe/q6mp2GC2rtJds/KROSeE0NOXiX79qq86YMCK4v4lpFa3HzFa6e67ZcSQpCzmbz2rf2EcIURZzdtq1NTsAAACaoTA1FtnD/x4IF9PGcJTQT6bsLf5Q/iaurAaEIeTPatKxhrBPAKQfxQsAIFNCLjhQH/73QPgoXlBR6Mm8t6uqpV3oo19AVoT4WeUAHkDIKF4AAJkSYsGBePC/B8JH8QKkGA0rEIaQPqscwAMIGcULACCTOGjPLv73QLgoXiCJHjgAyJpG5nvaFACNwve8AAAAAAgCIy8Bi6Nni6vOAEA6hZiPaVMANBojLwAAAHWaM/muV4s1AI3DyEuA4uzZ4qozAJAuIY9e0KYAaDSKFwAAgBqFXGwCIaJ4CVAjerZIsgCQDq0wehFizADCQPECAABQo1YoNoGQULwErJoESTIFgDCRtwHgaBQvAAAAdaLYBJIRS/FiZlMlfV3SAEl3uPv8kuVflfTO6O4fSTrZ3YdEy16RtCZa9lt3nxZHTFnHCYTpwv5HmpCzkRXkXqD11F28mNkASd+U9C5JmyQ9aWZL3f3pwjrufmPR+h+X9KaiTRxw9wn1xgEA6Bs5GwAQsjhGXiZKWufu6yXJzBZLukzS0xXW/5Ckv4nhddGLUE4gTHt89WIEDClEzkbLI/cCrasthm2MlPRC0f1N0WNHMbNOSaMlPVb08LFm1mVmT5jZ5THEAwCojJwNAAhWHCMvVuYxr7DuVZKWuPsrRY91uPtmMztD0mNmtsbdnz/qRcxmSZolSR0dHfXGnBlp7WXKSq9YKCNgyBRyNloeuRdoXXGMvGySdHrR/dMkba6w7lWS7i1+wN03R7/XS1quI+dWF6+30N1z7p4bPnx4vTEDQFaRszNizuS7Xj14B4BWEcfIy5OSxpjZaEm/U76x+3DpSmY2VtJQSY8XPTZU0n53P2RmwyS9XdKXYogJKZeVXrFW//sQJHI2MoPcC7SeuosXd+82s+slLVP+spuL3H2tmc2T1OXuS6NVPyRpsbsXT08YJ+lbZtaj/CjQ/OIr3uA1HAQDiAM5u/VlZVougGyK5Xte3P0hSQ+VPHZzyf3Pl3neTyWdG0cMCFOrNqYcPCDNyNkAgrFxcv535/JmRoEUiaV4QeNwEAwA6I+sTMsFkE0UL0AN+joo4OABAIA6FEZcDqw48j4jMJlH8ZJyHAQDAGpBewGgFVG8AP3Q32l8HDwAAFCDwggLIy4oQfESCA6CAaAxGNkGgHBQvAD9EOc0Pg6YAADoAyMuKEHxAgDIJK7mCADhoXgBahDHiAsHTAAAAP1D8QIAyCSu5ogk8P4C4kXxAiSMAyagNfAZBoDkUbwAADKN4gONcOWQL0mS9u85JIliF4gLxUuA6k2AJNB0YP8DYeK8NSAhfMcLyqB4AQAAiEmhmC2MuPzR4EGSKG6BuFC8BKTe3j56CwGgfpy3BjRYYcTlwIoj7zMCA1G8AAAAxKav4paiF6gPxUtA6u3to7cQAOLT31Fvci5QpcIICyMuKIPiBQAAIGaVRlyYuh0DippMo3gJUL2JjkTZGmj4gHTjYBWoE8UJyqB4ARLGAQyATKGXXBJTt2PBifwQxQsQHHpzgTBwsAoA8aN4ARJC0QEgU+glL4ucXwdO5IcoXoDg0JsLhIXPKADEh+IFSAhFB4BMoZccjcJ7KdMoXoBAUfwAQLrRWQXEry2OjZjZVDN7xszWmdmcMss/amY7zGxV9HNN0bIZZvZc9DMjjniANJu//OpYG7I5k+96tYEEqkXeRmI6l9NTDiA2dY+8mNkASd+U9C5JmyQ9aWZL3f3pklW/7+7Xlzz3REl/IyknySWtjJ67u964AADlkbfzWrVXvFX/rpBwgRagceKYNjZR0jp3Xy9JZrZY0mWSShvBct4j6RF33xU99xFJUyXdG0NcQEujcUQdyNsAgCDFUbyMlPRC0f1Nkt5WZr0rzOwdkp6VdKO7v1DhuSNjiCkYHHACaIJM5+2QC//eYg3572o1XKAFaJw4ihcr85iX3P9XSfe6+yEzmy3pTklTqnxu/kXMZkmaJUkdHR21R4swcHWaPtE4og4Nz9vkbABAI8RRvGySdHrR/dMkbS5ewd1/X3T3nyR9sei5k0ueu7zci7j7QkkLJSmXy5UtcEJSrods/aqtOmPCCA5CM4TCA03S8Lxdbc5uxmcgxMK/mlGVEP+utGjUPuN/AMQvjuLlSUljzGy0pN9JukrSh4tXMLN2d98S3Z0m6VfR7WWS/t7Mhkb33y3pMzHEhFDxjcz9RuOIGpC3gRgUFz0UjUAy6i5e3L3bzK5XvkEbIGmRu681s3mSutx9qaQbzGyapG5JuyR9NHruLjP7W+UbUkmaVzgJtNUV95CtX7VVkrR/zyE9tWIjCTADmJuOZkpD3k7DZyCkz1t/RlVC+ruaLQ3vQwD9E8uXVLr7Q5IeKnns5qLbn1GFnjl3XyRpURxxoAXwjcxAIsjbQO1Ki54rh3xJ+/ccOmIZBRDQGLEUL6hdcXIj4WUHc9MDQAHdUHwGasN+ihfvQyA8FC9ovnIHiRwwAgBSqlzRQwEEJIPiJUVIeNnD/zyFuGhEokL7DHCA2pr4fwLhoHhB83CQCAAIWLlLVQNoLIoXJK9QpABpxEUjUAZXpQKAdKB4QfNwkIik8B4DAKAlULwgOZWmiQFpRKGDIlyVCgDSgeIFzdfXQSK95qgV51UBANBSKF6QHKaJ1YzeXiAd+AwCQHNRvCSAA88a0WuOelEwAwDQUihekDwOIKvGFY4AAABeQ/HSQBx41olec8SF9w4AAC2B4gVImeIilyscAdXhMwIA2UDx0kAceMaEXnMAAACI4gVIjd6mGVL4AuUxPTcFyk3tZbovgAaheEkAjSgQKA7AAABIFYoXICWYZgj0H5+bJip3OftDq6RBE7jEfT/x/gWqR/ECAKX4jiEAAFKJ4qUJ6GFBb3hfNAHFSfD43DRBb5ez5zNVFc7ZAvqP4qXBSEQ1oNHrN95nMeM7hoDaFU8fA4CYUbwkiB4WIGVKp4c9OyR/wEWxAlSv3OeFz1FVOGerAjqO0AuKlwYpV6isX7VVZ0wY0cyw0o3zDPqNgrhKtb6XeO8BlZV+rsjhABJA8ZKgMyaM0PzlV3OACSSptyksncvzy9sGSz178j+F53DABSAhHA9EKIBRBYqXOvRWhDAUXAPOM+g33md9KBQuPXvyjSHvLaB+fR1g8jkD0EAUL03AASZqQYHST8WFS0G5ERgOuAAgHcjHqEIsxYuZTZX0dUkDJN3h7vNLln9C0jWSuiXtkPS/3H1jtOwVSWuiVX/r7tPiiKmR+nOeAQeaNSBZ9RvvswqKvyyvbTAnEUeylrMRs74OMPmMAWiguosXMxsg6ZuS3iVpk6QnzWypuz9dtNovJOXcfb+ZXSvpS5I+GC074O6ZvZ4ivenoCyfl16j4AKsw4tLbQVVGDrjI2QBSLyP5GLWJY+RloqR17r5eksxssaTLJL3aELr7T4rWf0LSR2J43abhPAMgMIy4FMtczkaD8JkC0ARxFC8jJb1QdH+TpLf1sv5MST8qun+smXUpPz1hvrs/WO5JZjZL0ixJ6ujoqCvgNKA3HdWiWK4TB1ilMp+z+SwBQLjiKF6szGNedkWzj0jKSZpU9HCHu282szMkPWZma9z9+aM26L5Q0kJJyuVyZbefhOJGj4YPQIAylbMBAK0ljuJlk6TTi+6fJmlz6UpmdomkuZImufuhwuPuvjn6vd7Mlkt6k6SjGsJWQ286+ov3CGKS2ZzNiDcAhC+O4uVJSWPMbLSk30m6StKHi1cwszdJ+pakqe6+vejxoZL2u/shMxsm6e3KnxiaOjR6AFpEJnI2kAXBH4twSWTUoO7ixd27zex6ScuUv+zmIndfa2bzJHW5+1JJ/1fS8ZLuMzPptctrjpP0LTPrkdSm/Pzpp8u+UIsKNuEACFKWczYj3kDAKHQQieV7Xtz9IUkPlTx2c9HtSyo876eSzo0jhkaj0QPQKrKQs4FWFvxskEIhUvgeLgoT9EMsxQsAAKEI5gAvFBx4opEodFCC4qWfaPQAAEAzBT8bpPhLhIvvNwoFT0uheAEABCfYg7ZWQo84kpB0oYPUo3hBWEheAABIqq94T0UHQFIjLikrsFeuXHnywIED75A0XvmLn+A1PZKe6u7uvuYtb3nL9nIrULwAAIIR/InKrYQecSSphd5fAwcOvGPEiBHjhg8fvrutrY0v8S3S09NjO3bsOHvr1q13SJpWbh2KF4Qhpb0nSeNADQBQj0x1AKS3wB5P4VJeW1ubDx8+fM/WrVvHV1qH4gUAEIzgT1RuRek5IAwW7+cEpaOQaaNwqSzaNxWn01G8IAzp7T1JRKZ6ygAADZPJDoCMHTPU6hOf+MSpxx9//Cvz5s3bFve2/+M//uOPZs6cOergwYNtU6ZM2bNo0aIX2tpqO92H4qWJMpU4ACBG5E20AjqmEsT086a67rrrOm+//faNU6ZM2Td58uQxS5YsecOVV175Ui3bonhBWDKaZDLZUwYAaBjakbDcOPHbYyXpqz+f+Uwc27vttttOWrBgwSlmpnHjxh148MEHf1O8/Mtf/vKw73znO8MPHz5so0aNOrRkyZLfnHDCCT2LFi0aesstt5za1tbmJ5xwwitdXV3PdHV1Hfuxj31s9OHDh62np0f333//8+eee+6hwrY2btz4ur1797Zdcskl+yRp+vTpv3/wwQeHUrwEhJ6WPtAbAgDIADqmEpTx6efFurq6jr311lvbH3/88V+3t7d3b9u2bUDpOtOnT99900037ZSkG2644dQFCxYMmzt37vb58+e3P/zww8+OHj368M6dOwdI0je+8Y3h11133bZrr71218GDB627u/uIbW3cuPF17e3thwv3Ozs7/7Bly5bX1Ro/xQsQEBo2AACyozDi8tyTm48vvl/PCMyyZcve8L73vW93e3t7tySdcsopr5Sus3LlyuNuvvnmkS+//PKAffv2DZg0adIeScrlcnunT58+6oorrtg9ffr03ZJ0wQUX7Lv11lvbN23adMxVV121u3jURZLcj742gZnVGj7FSzPQ01IB81EBABnEcUCCOKaQu8vMer3a2axZs0YvWbJk3QUXXHBgwYIFJ61YseIESbrnnnt++9hjj71+6dKlgydMmHDOqlWr1s6ePXvXRRddtO+BBx4YfOmll555++23b5g2bdrLhW2NGjXqcPFIy8aNG48ZMWLE4XKvWw2+1RMAAADZs3Hyax2lKfXVn8985qs/n/nMmLeeunfMW0/dW7hfzzanTp360tKlS0/cunXrAEkqN21s//79bR0dHYcPHTpkixcvPrHw+Nq1awdNmTJl39e+9rXNQ4cO7V6/fv0xTz/99DHjxo079LnPfW77u9/97hdXrVp1XPG2Ojs7D7/+9a/vefTRR1/f09Oju++++6TLLrvsxVrjZ+SliehpKcF8VAAAEBeOJ8rK5XIHb7rppi0XXXTRWW1tbT5+/Pj9999//4bidebMmbN54sSJ40aOHPmHcePG7d+7d+8ASbrxxhtP27BhwyB3twsvvPCl888//8DcuXNH3HfffScNHDjQhw8ffviWW27ZXPqat99++8aZM2eOPnjwoL3zne986QMf+MCeWuO3cvPQ0i6Xy3lXV1ezw0CjkGzQwsxspbvnmh1HksjZAJqi0vFE6TT14yaVX0+NydmrV6/ecN555+2Mc5utZvXq1cPOO++8UeWWMfKC9KFoAYDWR0cVGoVzaFsaxQsQMC76AABAPzFNPWgULwAAIDn0iqPRKE5aGsULECC+6BQAgDpR1ASJ4gUAACQnsF7xajuH6ERKoZS/t1AbipcGIpGhUeL8olPep0BlfD4AIF0oXgAAQPJS3ite7fRcpvGiVXziE5849fjjj39l3rx52+Le9sc//vGR991330kvvfTSgP379/+inm1RvDQAiQxJiWPEhfcp8JrC56CAzwcA1O/yyy9/8ZOf/OT2cePGja93W21xBGRmU83sGTNbZ2ZzyiwfZGbfj5b/zMxGFS37TPT4M2b2njjiAdCiNk5+bZ486kLeBno3f/nVmr/8ao2f1KnxkzpfvV/rekBdNkwcqw0Tx8a1udtuu+2kM8888+yxY8eeffnll48uXf7lL3952Pjx48eNHTv27Pe85z1//PLLL7dJ0qJFi4aOGTPmnLFjx56dy+XGSlJXV9ex55577rizzjrr7DPPPPPsNWvWDCrd3sUXX7yvs7PzcByx1z3yYmYDJH1T0rskbZL0pJktdfeni1abKWm3u/+JmV0l6YuSPmhmZ0u6StI5kk6V9GMzO9PdX6k3rmaK83wEoFF4n2YXeftopSOR4yd1HvGbzwfQIIFcuKGVdHV1HXvrrbe2P/74479ub2/v3rZt24DSdaZPn777pptu2ilJN9xww6kLFiwYNnfu3O3z589vf/jhh58dPXr04Z07dw6QpG984xvDr7vuum3XXnvtroMHD1p3d3dD449j2thESevcfb0kmdliSZdJKm4EL5P0+ej2Ekm3mZlFjy9290OSfmNm66LtPR5DXABaBd8LETfyNlClagtXClw0RGG05eCTxx9xf9TPn6l1k8uWLXvD+973vt3t7e3dknTKKacc1fm0cuXK426++eaRL7/88oB9+/YNmDRp0h5JyuVye6dPnz7qiiuu2D19+vTdknTBBRfsu/XWW9s3bdp0zFVXXbX73HPPPVRrbNWIo3gZKemFovubJL2t0jru3m1meySdFD3+RMlzR8YQUyqQyBLAQWzdeJ9mEnm7BCORQMLolGoad5eZeW/rzJo1a/SSJUvWXXDBBQcWLFhw0ooVK06QpHvuuee3jz322OuXLl06eMKECeesWrVq7ezZs3dddNFF+x544IHBl1566Zm33377hmnTpr3cqPjjKF6szGOlO6TSOtU8N78Bs1mSZklSR0dHf+IDELrAvhciAA3P2+RsAIhBYYQlhhGXgqlTp770/ve//08++9nPbhsxYsQr27ZtG1A6+rJ///62jo6Ow4cOHbLFixef2N7efliS1q5dO2jKlCn7pkyZsm/ZsmVD1q9ff8yuXbteGTdu3KFzzjln+/r16wetWrXquLQXL5sknV50/zRJmyuss8nMBkoaLGlXlc+VJLn7QkkLJSmXy/VaLSID6LFBX3hP9KbheTvUnM2IC5AQOqWaJpfLHbzpppu2XHTRRWe1tbX5+PHj999///0biteZM2fO5okTJ44bOXLkH8aNG7d/7969AyTpxhtvPG3Dhg2D3N0uvPDCl84///wDc+fOHXHfffedNHDgQB8+fPjhW2655ag2Yfbs2ac98MADJx48eLDtlFNOeeP06dN3fuUrXyl7zN8Xc6+vTYkatWclXSzpd5KelPRhd19btM5fSbVphHAAABJnSURBVDrX3WdHJ37+D3e/0szOkXSP8vOlT5X0qKQxfZ34mcvlvKurq664EbjS4uW4SfnfJD8UpLRBNLOV7p5rcgyJ5m1yNoCKUpqrCxqRs1evXr3hvPPO2xnnNlvN6tWrh5133nmjyi2re+Qlmgt9vaRlkgZIWuTua81snqQud18q6duS/jk6sXOX8leqUbTeD5Q/SbRb0l+FfsUaJIQeG1TCqFyfyNsAUoPcjH6K5Usq3f0hSQ+VPHZz0e2Dkj5Q4blfkPSFOOIAgKpQ0JC3AQBBiqV4AZomwwefqIBROQAAWhbFC4DsYEoZ0Fh8poBq9PT09FhbW1swFzNJUk9Pj0nqqbS8LcFYACA5ncs5gAIApNFTO3bsGBwdpKNIT0+P7dixY7Ckpyqtw8hLgvjysxjQq4d6MKUMaAxGNYGqdXd3X7N169Y7tm7dOl4MJJTqkfRUd3f3NZVWoHhBmGgYAaD1VJPbyf8I3Fve8pbtkqY1O45QUbwkoDDi8tSKjUfcZwSmH0p79doGNy0UtAAOeoB4MaoJICEUL0iH/jZ4PXtqex4AIH2qmXbG1DQAonhJRGGEhRGXOhQap2eH5H8Xipf+oKEDgMYivwJoMIoXNFd/e9IGTTjyPg0lAISvmmlnldahYwrIFIqXBDHiEoPSxqoaTDUAgHiRRwE0CcULmqvWnjMaTABoPdXk9tJ2g44pIFMoXtD6mFoAAPGgYGh9/E+RchQvSAeSJACgP+iYaiz2K1KK4gXZQQIGgPpQMLSu/n6fGu8BNAnFC4Dw0GgCKCAPNAbfp4aUonhBdpCAWx//YyAZfMZaT7Xfp8Z5T2gyihcA4aDRBMKSps9ommJJM75PDSlH8YLWxwFv6+N/DADx6Ov71DjvCU1G8YLsIvGmT1//k76+YRtAOqSpQyFNsYSE/YOUonhB6+OAt/XREwgAySLPokkoXpA99MKlT3//J/yvgHRLU4dCmmIBUDeKF2QHIy6tj4MSIBsoRIDMonhB9lDEpA89o0BrStNnOU2xAKgZxQsAAAhD6RTTZ4fkL+1LYQJkRls9TzazE83sETN7Lvo9tMw6E8zscTNba2a/NLMPFi37rpn9xsxWRT8TSp8PNNyBFfmfjZMZjWm2zuUchDQYeRsAELK6ihdJcyQ96u5jJD0a3S+1X9LV7n6OpKmSvmZmQ4qWf8rdJ0Q/q+qMBwDQO/I2wlXo3GgbnP9d+BZ4Op6AzKh32thlkiZHt++UtFzSp4tXcPdni25vNrPtkoZLerHO1wbqw3kWyCbyNgAgWPWOvJzi7lskKfp9cm8rm9lEScdIer7o4S9E0xK+amaD6owHqB3TxpAN5G2ErXO5dOaL0nGT8j9MNwUypc+RFzP7saQRZRbN7c8LmVm7pH+WNMPde6KHPyNpq/IN40Lle//mVXj+LEmzJKmjo6M/Lw30jquPhYERsqqlIW+TswEAjdBn8eLul1RaZmbbzKzd3bdEjdz2Cuu9QdK/Sfqcuz9RtO0t0c1DZvYdSZ/sJY6FyjeUyuVy3lfcQNX40kq0mDTkbXI2Go4cDWRSvdPGlkqaEd2eIemHpSuY2TGSHpB0l7vfV7KsPfptki6X9FSd8QBoNYXpfFwVLi7kbQBAsOo9YX++pB+Y2UxJv5X0AUkys5yk2e5+jaQrJb1D0klm9tHoeR+NrlBzt5kNl2SSVkmaXWc8QP9x4n628H8mbwMAglVX8eLuv5d0cZnHuyRdE93+nqTvVXj+lHpeH0AGUFzGirwNNBB5Cmi4ekdegNZBYxOvtDXinNsEAEDwKF4AhIEiA0Ba0TkCJIbiBagXjdSR0tqIM/0MAIDgUbwAAADUg84RIDEUL0Ct+hphyEIjVu5vTPvfn7Z4AABA1SheAAAA4kDnCNBwFC9ArSqNMKT1nI84VfM3ttLfCwBoPa3YPmdAW7MDAJAyfIM9AABIKUZegHqV9tik/ZyPOGThbwQAtKYszJBoYRQvAPJI5gAAIOUoXoBaVHNgn4WD/iz8jQAQIjqgKmP2QNAoXgDkkcwBAEDKUbwA/cHUKgBAmqW9nUpTPGmIAf1G8QLgSCRzAK0qTQfOAGpC8QL0B1OrACAdGpGHK31vV0i5Pq2xp31ECMGgeAHQXDRgABqNA2egZVC8ALWgwQOA5mhEIVK6zWeH5H/37InvNZKWtljTOiKE4FC8AGgOekIBJCVLB85Z+BuRaRQvAAAgHI0oRCptk0IgfuxL1IniBUBzcHAAIGmtnGcqTX0788WmhAM0CsULAAAITyMKkdJthlbsHFrVuG3T0YSUaGt2AECmbZyc7x0rNApZ1LmcxhBA2DZObn4e71wuDZogaYDUNjh/sYGePemIDYgRIy9AOfQwAQBCUTplrGdv47ZN+4gmo3gBmmHj5PzwfuEynAdW5EdgBk1oXoNAgwQA/ZPWA/vjLsy3Mc1sU4AGqat4MbMTJX1f0ihJGyRd6e67y6z3iqQ10d3fuvu06PHRkhZLOlHSf0v6n+7+h3piAuqS1oYIiAl5G2hB5S6AEtdUMS6ugpSp95yXOZIedfcxkh6N7pdzwN0nRD/Tih7/oqSvRs/fLWlmnfEAYehcnr8CzHGT8nOTj5uUv9+MRqEwH/rAivwP86NbHXkbqEdxjiycs3fcpPxPms7hS1MsQIzqLV4uk3RndPtOSZdX+0QzM0lTJC2p5flAQ6S5IQoVxVDakLeBVtXINov2EClR7zkvp7j7Fkly9y1mdnKF9Y41sy5J3ZLmu/uDkk6S9KK7d0frbJI0ss54gGTVO4xeb0MQxzA+UwKyhrwN1KK3acXkTSAxfRYvZvZjSSPKLJrbj9fpcPfNZnaGpMfMbI2kl8qs573EMUvSLEnq6Ojox0sDNaAhqh/nDzVNGvI2ORtoMHIqMqrP4sXdL6m0zMy2mVl71HvXLml7hW1sjn6vN7Plkt4k6X5JQ8xsYNSLd5qkzb3EsVDSQknK5XIVixwgEc0+MG/E69MAtow05G1yNloOo9RAKtR7zstSSTOi2zMk/bB0BTMbamaDotvDJL1d0tPu7pJ+Iun9vT0fQBUOrWrsNyvXgvOH0oq8DYSMi6wg4+o952W+pB+Y2UxJv5X0AUkys5yk2e5+jaRxkr5lZj3KF0vz3f3p6PmflrTYzP5O0i8kfbvOeIBkNLsHrvT1geqRt4F60AkDNJXlO9LCksvlvKurq9lhAM2fPvDskPzvwpddHjcp/5vGNbXMbKW755odR5LI2UADNLv9yYgs5uy0q3fkBci2ZjcagybkfxfOfQEAAGhhFC9AyJKevkZPHwCkQ1x5mLyOwNR7wj6AUHBSJwAACBwjL0BoyvWSJTXiwne2AAgJuaoy8joCRfECtDoaKAAA0CIoXoBQNLMIafaloQGgP+i06Rt5HYGieAFaHQ0UAABoERQvQCjSUIRQ+AAIQRryZSjYNwgMxQuQFTRQAAAgcBQvQGgoQgCgOuRLoOXwPS8AyuN7YQAAQMpQvAAAAAAIAtPGAByJS4wCAICUYuQFAAAAQBAYeQFwJC4xCgAAUoqRFwAAAABBYOQFQHmMuAAAgJRh5AUAAKCVcel7tBCKFwAAgGagqAD6jWljAAAAaVXPxVO49D1aEMULAABAkigqgJpRvAAAAKRNHAUOl75HC6J4AQAASBJFBVAzihcAAIC0ibPAoThCC6nramNmdqKZPWJmz0W/h5ZZ551mtqro56CZXR4t+66Z/aZo2YR64gEA9I68DaRI53IKC6Cf6r1U8hxJj7r7GEmPRveP4O4/cfcJ7j5B0hRJ+yU9XLTKpwrL3X1VnfEAAHpH3gZCQoEDHKHe4uUySXdGt++UdHkf679f0o/cfX+drwsAqA15GwAQrHqLl1PcfYskRb9P7mP9qyTdW/LYF8zsl2b2VTMbVGc8AIDekbcBAMHq84R9M/uxpBFlFs3tzwuZWbukcyUtK3r4M5K2SjpG0kJJn5Y0r8LzZ0maJUkdHR39eWkAyJQ05G1yNgCgEfosXtz9kkrLzGybmbW7+5aokdvey6aulPSAux8u2vaW6OYhM/uOpE/2EsdC5RtK5XI57ytuAMiqNORtcjYAoBHqnTa2VNKM6PYMST/sZd0PqWTqQdRwysxM+XnXT9UZDwCgd+RtAECw6i1e5kt6l5k9J+ld0X2ZWc7M7iisZGajJJ0uaUXJ8+82szWS1kgaJunv6owHANA78jYAIFh1fUmlu/9e0sVlHu+SdE3R/Q2SRpZZb0o9rw8A6B/yNgAgZPWOvAAAAABAIiheAAAAAATB3MO7CIyZvSzpmWbH0YthknY2O4heEF/t0hybRHz1SiK+Tncf3uDXSBUz2yFpn9L7v+d9WR/iq12aY5OIT8pgzk67UIuXLnfPNTuOSoivPmmOL82xScRXr7THF7I079s0xyYRX73SHF+aY5OID+nEtDEAAAAAQaB4AQAAABCEUIuXhc0OoA/EV580x5fm2CTiq1fa4wtZmvdtmmOTiK9eaY4vzbFJxIcUCvKcFwAAAADZE+rICwAAAICMCaJ4MbMPmNlaM+sxs4pXlTCzDWa2xsxWmVlXCuObambPmNk6M5uTYHwnmtkjZvZc9HtohfVeifbdKjNb2uCYet0XZjbIzL4fLf+ZmY1qZDw1xPdRM9tRtL+uKbedBsW2yMy2m9lTFZabmS2IYv+lmb05qdiqjG+yme0p2nc3Jxzf6Wb2EzP7VfS5/d9l1mnqPgwdObvu+FKXs6PXS23eTnPOjl4/tXmbnI3guHvqfySNkzRW0nJJuV7W2yBpWBrjkzRA0vOSzpB0jKTVks5OKL4vSZoT3Z4j6YsV1tubUDx97gtJ10n6x+j2VZK+n+D/s5r4PirptqTfa9Frv0PSmyU9VWH5eyX9SJJJOl/Sz1IW32RJ/68Z+y56/XZJb45unyDp2TL/36buw9B/yNl1x5eqnF3t/mhW3k57zo5eP7V5m5zNT2g/QYy8uPuv3D21X0pZZXwTJa1z9/Xu/gdJiyVd1vjopOh17oxu3ynp8oRet5Jq9kVxzEskXWxmlqL4msbd/13Srl5WuUzSXZ73hKQhZtaeTHRVxddU7r7F3f87uv2ypF9JGlmyWlP3YejI2XVLW86W0p23U52zpXTnbXI2QhNE8dIPLulhM1tpZrOaHUyJkZJeKLq/SUd/+BrlFHffIuWTgKSTK6x3rJl1mdkTZtbIxrKaffHqOu7eLWmPpJMaGFPZ145U+l9dEQ1PLzGz05MJrSrNfK9V6wIzW21mPzKzc5oVRDSt5U2SflayKIR92ArI2eWlLWdL6c7boedsKf05h5yN1BjY7AAKzOzHkkaUWTTX3X9Y5Wbe7u6bzexkSY+Y2a+jHoU0xFeu9ym2S731Fl8/NtMR7b8zJD1mZmvc/fl4IjxCNfuiofurD9W89r9KutfdD5nZbOV7G6c0PLLqNHPfVeO/JXW6+14ze6+kByWNSToIMzte0v2S/trdXypdXOYpadqHTUfOrk9gOVtKd94OPWdL6c455GykSmqKF3e/JIZtbI5+bzezB5QfSo6lIYwhvk2Sint6TpO0uc5tvqq3+Mxsm5m1u/uWaBh1e4VtFPbfejNbrnzvRiMawmr2RWGdTWY2UNJgJTes3Wd87v77orv/JOmLCcRVrYa+1+pV3Oi4+0NmdruZDXP3nUnFYGavU74RvNvd/6XMKqneh2lAzq5PYDlbSnfeDj1nSynOOeRspE3LTBszs9eb2QmF25LeLanslTOa5ElJY8xstJkdo/zJjA2/OkxkqaQZ0e0Zko7qdTSzoWY2KLo9TNLbJT3doHiq2RfFMb9f0mPunlQvSp/xlcylnab8HNy0WCrp6ujqK+dL2lOYgpIGZjaiMA/ezCYqn4d+3/uzYn19k/RtSb9y969UWC3V+7AVkLN7lbacLaU7b4ees6UU5xxyNlKn3jP+k/iR9BfKV9WHJG2TtCx6/FRJD0W3z1D+CiOrJa1VfmpAauKL7r9X+atkPJ9wfCdJelTSc9HvE6PHc5LuiG7/qaQ10f5bI2lmg2M6al9ImidpWnT7WEn3SVon6eeSzkj4PddXfLdE77PVkn4i6awEY7tX0hZJh6P33UxJsyXNjpabpG9Gsa9RL1d7alJ81xftuyck/WnC8V2o/HSCX0paFf28N037MPQfcnbd8aUuZ1faH2nJ22nO2dHrpzZvk7P5Ce3Hon86AAAAAKRay0wbAwAAANDaKF4AAAAABIHiBQAAAEAQKF4AAAAABIHiBQAAAEAQKF4AAAAABIHiBQAAAEAQKF4AAAAABOH/A6NIVCHPyh2KAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAykAAAE/CAYAAABCeozFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5RV9Xnv8c8zEJFGo1EIjCg/bBRRjCROiTZaCNolphVtNQktXZosXRRN6qqaLqnc6811pYX0knQVSdJaQyK3MSRqo2TVu7BJSpK2JnFYhSomUkIhEH4HgvyQCcM894+zB46HMzNnztk/vnvv92utWTPnnD17f/f58Xz3832+ex9zdwEAAABAKNqybgAAAAAAVCNJAQAAABAUkhQAAAAAQSFJAQAAABAUkhQAAAAAQSFJAQAAABAUkhQgMGZ2h5n9v6zbAQAAkBWSFDTNzDab2RtmdtDMfmlm/25m88ysofeVmY03MzezoQm3c8DtRInBGjN73cy2mdlfNdIuM5tjZoeinzfMrKfq9qFm2uvuT7j7jc38LwD0h7idTNyO1nuJmXU3+/8A3owkBa26yd3PlDRO0iJJD0r6YrZNasqvSfpTSSMkvVfSdZI+MdA/uftX3P0Mdz9D0o2Stvfeju57k6Q7dgBoAHF7EHEbQDZIUhALdz/g7islfVjSHWY2WZLM7HfM7D+ika6tZvbJqn/7XvT7l9EI1tVm9utm9h0z+4WZ7TWzr5jZ2b3/YGYPmtnPo1HA18zsuuj+NjObb2Y/jf7362Z2Tl/bqdP+L7j79939V+7+c0lfkfS+OJ6baITvz8zsZUlHovv+h5ltivZjvZnNqlr+LjNbHf09NBpN/GMz22hm+81sSRztAlBuxO2+mdkFZvZctD+bzGxe1WPvq3p+dprZwqo2D6mqyrw7jrYAZUWSgli5+48kbZN0bXTXYUm3Szpb0u9IutvMboke+63o99nRCNaLkkzSQknnSZok6QJJn5QkM5so6eOSfiMaBbxB0uZoHfdKukXStOh/90v6XD/bGchvSVrf8I4PbLYqI3ZnRbc3qNKZniXpLyQ9aWaj+vn/D0i6UtK7Jf2RmV0fY9sAlBhx+83MbIik5yX9e9SumZIeMrNp0SJLJf2lu79N0kWSnq3a/vGqqsx/tNoWoMxIUpCE7ZLOkSR3X+3uL7t7j7v/p6SvqtIh1eXuG939n929y933SPps1fLHJQ2TdKmZvcXdN7v7T6PH/ljSAnff5u5dqnSQtzUzvcrMPiqpQ9Liwf5vP/4matsbkuTuX3f3HdHz8qQqnXZHP/+/MBr13CxptaQpMbYNAIjbJ10j6XR3/3RUpdkg6UuqDDZJ0jFJF5vZue5+0N1/GMM2AdQgSUESxkjaJ0lm9l4z+xcz22NmByTNU2X+cF1m9g4zWxFNDXhd0j/0Lu/uG1WZf/xJSbuj5c6L/nWcpG9Y5UTQX0r6sSqdY3/ViXrbv0WVOdo3uvvewfzvALbWbOcjZrauqr2XqJ/nRdLOqr+PSGLeNIA4EbdPGidpfG+7orbdL2l09Pgdkt4laYOZ/dDMbohhmwBqkKQgVmb2G6p0dv8a3fWkpJWSLnD3syT9rSpTAyTJ66xiYXT/u6JS+h9VLS93f9Ldr1GlE3FJn44e2qpKB3V21c/p0Tzletup1/aZkv5elZNKX254pxtzog1mdqGkL0i6W9K57n62pJ+oaj8BIC3E7VNslfSTmnad6e6/F+3Pj939w5LeIWmJpH80s9MabTOAxpCkIBZm9jYz+11JKyT9Q1Vncaakfe5+1MymSvrDqn/bI6lH0oVV950p6ZAqJ0uOkfRnVduYaGYzzGyYpKOS3lBl1E2qdKJ/YWbjomVHmtnN/Wyntv0zVDnp8tZofnbt4182sy838FQ04gxVOrM9lVXbXapUUgAgNcTtPv1r9P9/amanW+UCJu8ys/dE998eTfU6LumAKvG8R9JuVU6cH9vENgHUIElBq75pZgdVGXlaoMpc5I9WPX6PpEeiZR6W9PXeB9z9iConjf9bVFK/StL/lvQeVQL/P0n6x6p1DVOlpL9XlelP75D0UPTY36gy8vdCtK0fqHJJyr62U+t/qnIS+/NVV2ap/kLFCyT926CemT5Ec7yXSPqRpB2qJCjMaQaQFuJ2P9z9mCoXK/lNSVtUSZi+oJPTbH9X0mtRmxdK+pC7d7v7fkl/JWlN1GbOHQRaYO5UJ4H+RGX8dapMZTiWdXsAAP0jbgP5R5ICAAAAIChM9wIAAAAQFJIUAAAAAEEhSQEAAAAQFJIUAAAAAEEZmnUD+jJixAgfP3581s0AgKasWbNmr7uPzLodaSJuA8irMsbs0AWbpIwfP16dnZ1ZNwMAmmJmW7JuQ9qI2wDyqowxO3RM9wIAAAAQFJIUAAAAAEEhSQEAAAAQFJIUAAAAAEEhSQEAAAAQFJIUAAAAAEEhSQEAIEfmT1+u+dOXZ90MAEgUSQoAAACAoAT7ZY4AAOCk3urJK9/d8qbbi1bfnlmbACApVFIAAAAABIVKCgAAOdBbMaGCAqAMqKQAAAAACAqVFAAAcoQKCoAyoJICAAAAICgkKQAAJITvNAGA5pCkAAAAAAgK56QAABAzvtMEAFpDJQUAAASFaXIAqKQAABAzvtMEAFpDkgIAAILANDkAvUhSAABICAfXANAckhQAABAEpskB6MWJ80A/OHkTQJ4RwwDkVSxJipktM7PdZvZKH4+bmS0xs41m9p9m9p44tgukjQ4fRUDMRugWrb6dKgpQcnFN9/qypKWS+jp6u1HSRdHPeyV9IfoNBKmvkzeBgviyiNmFxgnoAPIuliTF3b9nZuP7WeRmScvd3SX9wMzONrN2d98Rx/aBpG1au1OSdORAlyQ6fOQbMRsAELq0TpwfI2lr1e1t0X1v6vDMbK6kuZI0duzYlJoGnKr25M1evaOSQME1FLOl4sTtog08cAI6gLxLK0mxOvf5KXe4PybpMUnq6Og45XEgK3T4KJmGYrZE3AYAJCOtJGWbpAuqbp8vaXtK2waaRjKCkipNzC76uRtF2Q8A5ZNWkrJS0sfNbIUqJ18eYG4z8ogOHyVBzEahFS0ZBYooliTFzL4qabqkEWa2TdL/kvQWSXL3v5X0vKQPSNoo6Yikj8axXQDA4BGzT2IqJwCEKa6re/3BAI+7pI/FsS0AQGuI2SiDeoln0af3AUWS1nQvAACCxUEqAISFJAUAABRGf9USpvcB+dGWdQMAAADQvPnTl5/yvV5A3lFJAQYpixE4Rv0AoDGNVEuIpUD4SFKAGiQEAIA84EIAKDKSFKBBWXQGdEAA0Jy44iRxF8gGSQqCEEInQEIAAMgTLgSAIiNJKTkCW+Oy6AzogAAgGwxcAdkiSUGmQuoESAgAAHlEf4UiIkkpqZCSg7zJMoECAKSDgSsgWyQpyFSInUAIbQAAZIvvHQGyRZJSUkknByElHQCQV8TS7PHcA9kgSUEQ6AQAACFIajo0CScwOCQpJZdUBYVzXfKF1wkIC7EUQNmRpAAAAETing5Nwgk0hyQFsQrxRHj0jc4TCBOxFEDZkaQUEJ0aAACtiasPJeEEmkOSgkQQhPOBzhMIG59JAGVFklIgTN0BgNYQN9GMRt43vKeAwSFJAUDnCQAAgkKSUiBM3QGA5lCJRjN43wDJacu6AQAAAABQjUpKAeVhBIfRJgAhoRKNZvC+AZJDJQUAAABAUKikIFXM3wUQMmIRmsH7BogflZQ65k9ffuLgGQAAIAscj6DMqKQgVczfBQAAwEBIUqowFQkAAGSN4xGAJCUYZQtAZdlPAAAADB5JShWmIgEAkIAt0yu/x63OshW5wfEIQJKSOUq6AAAAwJuRpNRBgoAyICEGkLjeCsob333zbSoqDSE+o8xIUjJGSRd5wvsUAACkgSQFKBmmGAJITW/FhAoKgEEiSQkEB4gIGYkNAABIE0lKyji4Q9aYYgggdQNVUKi0AKhBkgJgQCQ2yBLvOwAoH5KUlJR2ugyjY8FK+71Xmvc8kFdZxGuu/gWgDyQpABpGgoE0lXZwBwBAkpKW0k2XYXQMEQ40gcBlGa/7uvoXfQZQeiQpAIAglW5wBwBwAklKykrTyTIahggHmkDgQojXtW2gCg+UHkkKACBoJLYAUD6xJClmNlPS30gaIulxd19U8/hHJP0fST+P7lrq7o/HsW0EjtEvRDjQDAcxG3WFEK9DqOoACELLSYqZDZH0OUm/LWmbpJfMbKW7v1qz6Nfc/eOtbi9NTE8BUDRFjtkoCRIYoBTiqKRMlbTR3TdJkpmtkHSzpNoODwCQPWI2wkcCApReHEnKGElbq25vk/TeOsvdama/JWmDpPvcfWudZYLAJVNbxCgXELLCxmxidMFxUj1QKm0xrMPq3Oc1t78paby7v0vStyQ9UXdFZnPNrNPMOvfs2RND0wAANWKL2RJxGwCQDHOv7ZsGuQKzqyV90t1viG7/uSS5+8I+lh8iaZ+7n9Xfejs6Oryzs7OltrUqztG5Uoz01Y5yDZ9W+c0oF0rIzNa4e0fW7aiVVMyW0o/btVXvydPGSSp4nAUVFCQi1JhdZnFUUl6SdJGZTTCz0yTNlrSyegEza6+6OUvSj2PYLgBg8IjZyJ8t008mJwBKoeVzUty928w+LmmVKpezXObu683sEUmd7r5S0r1mNktSt6R9kj7S6nbTEGcFpRTntzRy6chmRsAYNQNiU6SYzReFllTSfQF9DhCEWL4nxd2fl/R8zX0PV/3955L+PI5tAUGgE0OOEbORG5wsn3sMIqBZfON8wko50tdfBWUwHQ2dE4AGlCKuInlJ9Tn0XUBTSFKAwYi7E6PzAoC+8Q30uVWq6e5IBElKSkr/oWymo8lj55SntgIATuqvz2nlfEpmAwBNIUkBBiOuxInOCwAaR2zMnVJOd0esSFKQrmY6mjx0TiQdAJLGORLpqFdBaSa253E2ABAQkhQ0JqQgG0JbWt02nRcwaIzIAvnD5xXNIklBcaWZAJB0AEhKXJVaKr6DF0ds5/kFmkKSgoq+AnDanVojXwQZV0cdQscRQhuAwHGVoIIJKQYDCBZJCoonq9FCOl4ASYirUkvFt3k8V0DqSFLKbqAD+rQ6tUYSi1bbwlSH1DDSjTgV4ipBxBtiMIBBIUlB8fSVzPTejhsdL4A0xBVTiE2DQ0wHMkGSUnaNVieSDs6DqZIw1SFYnDuAJDX6PgrqfccgxknEYACDQJKC4kqrA6TjBYDiIcEEMlXYJCWokbQ8CCXopnm5YMSuEOcOILeCrOQxiHEqngMADShskgLEKslpaACA8JBgApkqXJIS5Eha2kILqKG1B4kr1ecNwQi6kkf8Ky76OCARhUtSgFgxJxlIVZAJBsqNeA9konBJStAjaUkL7YA6tPYAKIVSxX1khz4OSFThkhQgVsxJBlLBVF0AQLXCJiml7NhCO6AOrT0AgPJJqg+ijwMSVdgkpR5G5tA0Oh8gUaWeqgsAOEWpkpTSCO2AOrT2YNA4cASQK1umS11rpWFTkj9nhD4uSPRb+VeKJIW5zgCQD8TlKkwjAlBipUhSADSHBB9ArtRecUuS2s6qVFRKkOwRo+m3iqQUSQpznREkRkkB1IsDXNoWAMqRpABoDgk+gFwp6RW3qB6cRL9VHKVKUnijIgiMkgLoLw6U9EAbAKqVKkkB0BwSfAC5UrLEjurBqXgO8o8kBUgbo6QAGokDxIbSI+lAmZGkAKEiiQHSl+Tnjs80EkYygyIhSQGywoEKAOIA6sjDifAhtgnFQpISKkbcyosT64H0Jfm54zOdf7xmQOpIUgAAAAIS8onweajyoBhIUkLDiBs4sR5IX5KfOz7T+UWfDGSGJAUAACBAIVYnQq7yoFhIUkLDiBt68doD6Uvyc8dnOn/ok4HMkKQACA4jdAAQNuIzkkaSEipGawAACAN9MpA6khSgUZT7E8dVYwAAgCS1Zd0AAC3YMv1k8gQAAFAQVFKAgXAJytRw1RgAwSDWA5kiSSkrgm++kTgBAIACI0kBBsIlKFNHBQVAZhgEAoJAktKAXE89qQ2uBN9iIHECgGwQd4FUxHLivJnNNLPXzGyjmc2v8/gwM/ta9PgPzWx8HNsFUjVuNZ0SCoO4DfShN9YPn1b5IfYDmWi5kmJmQyR9TtJvS9om6SUzW+nur1Ytdqek/e7+TjObLenTkj7c6raTluvLoQ5UMWEkqBh4/dCEIsdtIDHMRABSFUclZaqkje6+yd1/JWmFpJtrlrlZ0hPR309Lus7MLIZtAyiA+dOXnxgEQCqI28BAqKAAmYrjnJQxkrZW3d4m6b19LePu3WZ2QNK5kvbGsP3E5PpyqANVTAi8yWBkDflQ2LgNJIaZCECq4khS6o2seRPLyMzmSporSWPHjm29ZQCClusplflG3C4LDqhTRQwD4hNHkrJN0gVVt8+XtL2PZbaZ2VBJZ0naV7sid39M0mOS1NHRcUpnmJVcBxs6pnQwVxn5Uvi4DSSGuA6kIo4k5SVJF5nZBEk/lzRb0h/WLLNS0h2SXpR0m6TvuDudGdCIAic8uZ5SmW/E7aJj4CRVVIWB+LWcpERzlT8uaZWkIZKWuft6M3tEUqe7r5T0RUn/18w2qjISN7vV7aIFdFbxY64ycoS4DQAInYU6MNbR0eGdnZ1ZN6OYOJBOTpzPbe1I6PBp8a0biTOzNe7ekXU70kTcDhDxPnlVzzEVlPwqY8wOHd84XyaU/5PHcwkEjwNJAAgfSUqekWQUWwGnkHFwCMSk0bhQgLgRrDoDf4ueEM85EBOSlEHI/QFWAQ96AaBRnNyM2HWtzboFCNSaNWveMXTo0MclTVY8X55eND2SXunu7r7ryiuv3F1vAZKUPGLaVrk0+7oG9L7g4BCICfE/HONWV57/rrXSsCm8BlWI8dLQoUMfHz169KSRI0fub2trC/ME8Az19PTYnj17Lt25c+fjkmbVW4YkpQG5PcDi2+YB4AQueY3Y1CaLXWsr99G/4qTJJCh9a2tr85EjRx7YuXPn5L6WIUnJI6ZtoT8BjrRycAig0IZNyboFQcjtoG4y2khQ+hc9P31OhSNJaUDuDrACPEgFgFAEH8MRPgYLkVP333//eWecccbxRx55ZFfc6/7+97//a3feeef4o0ePts2YMePAsmXLtra1NX86DklKnhEUUU/AnScHh0CLAv58A7kb1EWs7rnnnnGf//znt8yYMePw9OnTL3r66aff9qEPfej1ZtfH1QYGYdHq2/PxgRu3uvIzfFrlp/e2VOnYejs3AADQvOr+FWjRfVO/OPG+qV+cGNf6li5deu7FF1986cSJEy+95ZZbJtQ+/pnPfGbE5MmTJ02cOPHSG2644dcPHjzYJknLli17+0UXXXTZxIkTL+3o6JgoSZ2dnadffvnlky655JJLL7744ktffvnlYdXr2rJly1sOHTrUdv311x9ua2vTnDlzfvHss8++vZX2U0kBkpTlaCcdJ1BcfL7zp0TVr1wM6BZcZ2fn6YsXL25/8cUXf9Le3t69a9euIbXLzJkzZ/8DDzywV5Luvffe85YsWTJiwYIFuxctWtT+wgsvbJgwYcKxvXv3DpGkRx99dOQ999yz6+6779539OhR6+7uftO6tmzZ8pb29vZjvbfHjRv3qx07dryllX0gSSmy6kDIeSoAAsFUEAA4qbd68l8vbT+j+vZf/+jO15pd56pVq95200037W9vb++WpFGjRh2vXWbNmjXDH3744TEHDx4ccvjw4SHTpk07IEkdHR2H5syZM/7WW2/dP2fOnP2SdPXVVx9evHhx+7Zt206bPXv2/ssvv7yrel3up14jwMyabb4kpnsByeidVvfGdys/TLMDgHKiP0AG3F1m1u/VxebOnTth6dKlP9uwYcOrDz744Paurq42SXryySd/9qlPfWr71q1bT5syZcplO3fuHDJv3rx9zz333Mbhw4f33HjjjRevXLnyzOp1jR8//lh15WTLli2njR49+ljtNgeDSkpZ9HeyJVUVAClI4/KkVGkA5E1vxSSOCkqvmTNnvn7bbbe986GHHto1evTo47t27RpSW005cuRI29ixY491dXXZihUrzumdrrV+/fphM2bMODxjxozDq1atOnvTpk2n7du37/ikSZO6Lrvsst2bNm0atnbt2uGzZs062LuucePGHXvrW9/a8+1vf/ut73//+w9/5StfOfdjH/tY3W+SbxRJSlGReGSLK/AAiBvxJJ/oD5CBjo6Oow888MCOa6+99pK2tjafPHnykWeeeWZz9TLz58/fPnXq1Eljxoz51aRJk44cOnRoiCTdd99952/evHmYu9s111zz+lVXXfXGggULRj/11FPnDh061EeOHHls4cKF22u3+fnPf37LnXfeOeHo0aP2/ve///UPfvCDB1rZB6s3hywEHR0d3tnZmXUz8quRYFh7nsrwaQP/DwaHTqm0zGyNu3dk3Y40NRq3k6yg9FZpJk8bF/s2Mkc8yTdev6DFHbPXrVu3+Yorrtgb1/qKat26dSOuuOKK8fUeo5JSNJwgHxaedwCtIq4XA68XMCgkKWVGCRpABpKobvAlcjEKqU8IqS0AUkWSUjQkHgBQDLVxnLjeOp5DIDdIUkCwDgWdJ9AyKigtCGlaWUhtAZAJkpSiGuwlhukAACAMHKDHj+cUyB2SFCBrdJ4oMc4hCUhI08qybEsI+w+AJKXQGjn45QAZAMKS5AF6WWN8SAkYkKH777//vDPOOOP4I488sivudf/Jn/zJmKeeeurc119/fciRI0f+o9X1kaQAWeuv86RDRUGl8e3zaFJI8SaLCgqDdkBTbrnlll9+4hOf2D1p0qTJcayPJKXIGhk5YnQJAMKURAWl7AfgZdvfBDCgkKDNUydKksb/6LU4Vrd06dJzlyxZMsrMNGnSpDeeffbZ/65+/DOf+cyIL33pSyOPHTtm48eP73r66af/+8wzz+xZtmzZ2xcuXHheW1ubn3nmmcc7Oztf6+zsPP2jH/3ohGPHjllPT4+eeeaZn15++eVd1eu77rrrDsfR7l4kKUAomIaHEuF7TRAcBu1QIJ2dnacvXry4/cUXX/xJe3t7965du4bULjNnzpz9DzzwwF5Juvfee89bsmTJiAULFuxetGhR+wsvvLBhwoQJx/bu3TtEkh599NGR99xzz667775739GjR627uzvxfSBJKYNGAi3BGACKiwNwtIgpmgnqraAcfemMN91uoaKyatWqt910003729vbuyVp1KhRx2uXWbNmzfCHH354zMGDB4ccPnx4yLRp0w5IUkdHx6E5c+aMv/XWW/fPmTNnvyRdffXVhxcvXty+bdu202bPnr2/toqShLakNwCgCeNWV36GT6v89N7OkfnTl5/oxIC+LFp9Owc5CEsO4y1Qy91lZt7fMnPnzp2wdOnSn23YsOHVBx98cHtXV1ebJD355JM/+9SnPrV969atp02ZMuWynTt3Dpk3b96+5557buPw4cN7brzxxotXrlx5ZtL7QCUlwogAmsKoJIA8IVahSUzRTFBvxSTGc1Jmzpz5+m233fbOhx56aNfo0aOP79q1a0htNeXIkSNtY8eOPdbV1WUrVqw4p729/ZgkrV+/ftiMGTMOz5gx4/CqVavO3rRp02n79u07PmnSpK7LLrts96ZNm4atXbt2+KxZsw622s7+kKQAIUvxgCKujocpAUAgGERpHs8dcq6jo+PoAw88sOPaa6+9pK2tzSdPnnzkmWee2Vy9zPz587dPnTp10pgxY341adKkI4cOHRoiSffdd9/5mzdvHubuds0117x+1VVXvbFgwYLRTz311LlDhw71kSNHHlu4cOH22m3Omzfv/G984xvnHD16tG3UqFHvmjNnzt7PfvazpyzXKHPvtxKUmY6ODu/s7Ex8O7UHVJOnjZPEARUGUHti+/Bpld857tCSSlLK+pkyszXu3pF1O9KUVtxGgzjQbh7PXenEHbPXrVu3+Yorrtgb1/qKat26dSOuuOKK8fUeo5IClFzclQ+mBAAZ4+qAzeO5A4JR+iSFA6oCS7Jz4Uo5AIBG0VcAg1b6JAUou6QSdRJ+oEmtHtAyiNK8NJ87Xh+gXyQpEQ6oCiTNcj2dCwCgL0wfK7Oenp4ea2trC/Pk7wD09PSYpJ6+HidJASCJRB3IXNwHtLX/xwFy45J6jrrWVn5vmU7iUnyv7Nmz59KRI0ceIFE5VU9Pj+3Zs+csSa/0tQxJSh2cn5JzTHUA0ARiP2JX3R91rZWGTcmyNUhRd3f3XTt37nx8586dk8WXp9fTI+mV7u7uu/pagCSlrDiAbw7PG4CkJDXAwpSjbPUmKD0HKq9B7yXrC3DpevTtyiuv3C1pVtbtyDOSlCp8CV3BEPgBNIDYj8QNm3IySQTQEJKUsmFErTk8bwCSUhtP4o4rTIHNVgbPP4k2ioAkpQrfmRIQOlMAKSH2A0B4SFLKhmvAN4eRSABxS7tCS9zKVooVFKYuoghIUurgw5yy6o6ZaVUAMkLszwH6BKA0SFLKKo0KSsiJxmDbFOI+AMi3Viq0xCTUwdRFFAlJCrLTXzJDBwwA6JWHwS8AsWopSTGzcyR9TdJ4SZslfcjd99dZ7rikl6ObP3N3rhtdZCGfvzHYjo6OEQVD3A5QMxUUYhL6QQUFRdBqJWW+pG+7+yIzmx/dfrDOcm+4O1+zijfrL5mhwwWSQtxG/oQ8+AUgEa0mKTdLmh79/YSk1arf2aGMQuxEBtvR0TGieIjbeUZMGjTOzwDyqdUkZZS775Akd99hZu/oY7nTzaxTUrekRe7+bL2FzGyupLmSNHbs2BabhtzIupOls0e5ELeRX8RpoDQGTFLM7FuSRtd5aMEgtjPW3beb2YWSvmNmL7v7T2sXcvfHJD0mSR0dHT6I9QODM9iOjo4ROULcLgFi0oD4zhAg3wZMUtz9+r4eM7NdZtYejca1S9rdxzq2R783mdlqSe+WdEpnB6SKE1BRUMRtAEDetTrda6WkOyQtin4/V7uAmb1d0hF37zKzEZLeJ+mvWtxu0xhJKRGSDqCe3MVtoBl8ZwiQb60mKYskfd3M7pT0M0kflCQz65A0z93vkjRJ0kzkPgMAAAosSURBVN+ZWY+kNlXmNr/a4naB1nEC6gl04qVC3K7Cex8AwtRSkuLuv5B0XZ37OyXdFf3975Iub2U7cWBuaokwjQvoU57iNhAH+nkgn/jGeaDEyQvJO8qK9z4AhK00SQpzU0uEaVwn8H4H0CdiJICAlSZJAXBq0kLyjrLivQ8AYStdkkJHVCIlHh3sayoLAHDeHoA8KF2SkhRG4xCyTWt3SpKOHOiS1HdFBSgb3vsAECaSFKCAaqey9OqtrAAoMc7bA5ADJCkt4goxyIM459/zHgcAAEkjSQEKjEQCQJ+ooMSKARwgXiQpLeIKMTnAlIYT4qigUDUEAABJI0kBAABoUn8DOAzmAM0jSYkJAShAXGYzVlQNAQBAWkhSAAAAmlRvAGf+9OWaP30502OBFpCkoLi4zGYi6GSBdHGAC6CMSFIGQOcAAEBMCjxoVH2cwPRYoHUkKSi+AnaGAIqPK+oBKDOSlD7QOQAAEJOSXsgkrmMGjkFQRiQpAAAEiClDAMqMJKUPdA4AgKLJrE/jQiZNYVYHyowkBQCAgHFAmiMkYUBsSFIGQOcAAMi7YEbkOXgfFGZ1oMxIUgAAAFpR0gsDAEkiSQEAoOAYkW9N1s8brxfKiCSlwLIOqgAAlAIXBgBiR5JSMCQmAIC+0DcMTjDn8gAlRJJSQJvW7tT86csJqgAApIkKChAbkpSCqB3t2bR2Z5bNQT9IGgEgHziXB8gOSUpBXThltDat3akLp4wmqAJAiXGADSCPSFIKot5oT+/fCANzmwEgn4jTQPpIUgoszqDKAXW+8HoBYGAEQJ6RpBQMnU+4mNsMAADQGJIU9IuRuHzh9QLQi4ERAHlGkgKkjAMFAACA/pGkoF+MxOULrxeQD2l+RokDAPKoLesGAAAAAEA1KiloSKMjcYzgh4HnHwgT540BQGOopCCX5k9fzvfAAAAAFBSVFMQiz6ODeWorgHwr43ljRdzXIu4TEBqSFORKnpMhAAAANIYkBbHI4+ggCQ+ArJQhzhQxxhZxn4BQkaQgV/KYDAEAAGBwSFIQqzwlDSQ8AJCcIsbYIu4TECqSFOQSHQMAAEBxkaSg9OJIeBhVA4D6ihgXi7hPQGha+p4UM/ugma03sx4z6+hnuZlm9pqZbTSz+a1sEwDQPOI2ACAPWq2kvCLp9yX9XV8LmNkQSZ+T9NuStkl6ycxWuvurLW4byBxXekEOEbfRMGIagKy0lKS4+48lycz6W2yqpI3uviladoWkmyXR2QWOzgkoHuI2ACAP0jgnZYykrVW3t0l6bwrbBRLHlV5QUMTtkkuiSkycBDAYAyYpZvYtSaPrPLTA3Z9rYBv1huu8j23NlTRXksaOHdvAqpEEpjAB+UbcBgDk3YBJirtf3+I2tkm6oOr2+ZK297GtxyQ9JkkdHR11O0QgRCRwCAlxG62Ks0rMwBeAZqQx3eslSReZ2QRJP5c0W9IfprBdNIkpTEDpEbcBAJlqKUkxs9+T9KikkZL+yczWuvsNZnaepMfd/QPu3m1mH5e0StIQScvcfX3LLQcADBpxG4MRxyAVA18AmtHq1b2+Iekbde7fLukDVbefl/R8K9tC85rtGOhIgOIhbgMA8oBvnAcAAIlj4AvAYJCkFBgnKwIAACCP2rJuAAAAeTR/+vITgz8AgHhRSSkwTlYEAABAHpGkAAAwCEylBYDkkaSUAB0nAAAA8oQkBQCAQWAqLQAkjxPnAQAAAASFSgoAAE2gggIAyaGSAgAAACAoJCkAAAAAgkKSAgAAACAoJCkAAAAAgkKSAgAAACAoJCkAACDX5k9ffuJ7awAUA0kKAAAAgKDwPSkAACCXeqsnr3x3y5tu8x02QP5RSQEAAAAQFCopAAAgl3orJlRQgOKhkgIAAAAgKFRSAABArlFBAYqHSgoAAACAoJCkAAAAAAgKSQoAAACAoJCkAAAAAAgKSQoAAACAoJCkAAAAAAgKSQoAAACAoJi7Z92Gusxsj6QtCW5ihKS9Ca4/a0XfP6n4+1j0/ZOKvY/j3H1k1o1IU8Jxu8jvFan4+yexj0VQ5P0rXcwOXbBJStLMrNPdO7JuR1KKvn9S8fex6PsnlWMfEY+iv1eKvn8S+1gERd8/hIXpXgAAAACCQpICAAAAIChlTlIey7oBCSv6/knF38ei759Ujn1EPIr+Xin6/knsYxEUff8QkNKekwIAAAAgTGWupAAAAAAIUGmSFDP7oJmtN7MeM+vzyhRmNtPMXjOzjWY2P802tsLMzjGzfzaz/4p+v72P5Y6b2droZ2Xa7WzGQK+JmQ0zs69Fj//QzMan38rmNbB/HzGzPVWv211ZtLNZZrbMzHab2St9PG5mtiTa//80s/ek3UaEh5h9YjlidmCI2cRspKM0SYqkVyT9vqTv9bWAmQ2R9DlJN0q6VNIfmNml6TSvZfMlfdvdL5L07eh2PW+4+5ToZ1Z6zWtOg6/JnZL2u/s7Jf21pE+n28rmDeI997Wq1+3xVBvZui9LmtnP4zdKuij6mSvpCym0CeEjZlcQswNCzJZEzEZKSpOkuPuP3f21ARabKmmju29y919JWiHp5uRbF4ubJT0R/f2EpFsybEucGnlNqvf9aUnXmZml2MZW5Pk91xB3/56kff0scrOk5V7xA0lnm1l7Oq1DqIjZuUXMzjliNkJRmiSlQWMkba26vS26Lw9GufsOSYp+v6OP5U43s04z+4GZ5aFTbOQ1ObGMu3dLOiDp3FRa17pG33O3RmX1p83sgnSalpo8f+6QrTy/d4jZImbnVJ4/d8iRoVk3IE5m9i1Jo+s8tMDdn2tkFXXuC+byZ/3t3yBWM9bdt5vZhZK+Y2Yvu/tP42lhIhp5TYJ+3QbQSNu/Kemr7t5lZvNUGYGckXjL0pPn1w8tIGY3hJgdFmJ2vl8/5EihkhR3v77FVWyTVD3icb6k7S2uMzb97Z+Z7TKzdnffEZVdd/exju3R701mtlrSuyWF3OE18pr0LrPNzIZKOkv9l6pDMuD+ufsvqm7+vXI0f7tBQX/ukBxiNjGbmJ1LQX/uUBxM93qzlyRdZGYTzOw0SbMl5eJqKqq0847o7zsknTIKaWZvN7Nh0d8jJL1P0quptbA5jbwm1ft+m6TveH6+AGjA/auZ6ztL0o9TbF8aVkq6PbpizFWSDvROgwEGQMwODzGbmA3Ew91L8SPp91TJ/rsk7ZK0Krr/PEnPVy33AUkbVBmpWpB1uwexf+eqcoWY/4p+nxPd3yHp8ejv35T0sqR10e87s253g/t2ymsi6RFJs6K/T5f0lKSNkn4k6cKs2xzz/i2UtD563f5F0iVZt3mQ+/dVSTskHYs+g3dKmidpXvS4qXK1nJ9G78uOrNvMT/Y/xGxidqg/xGxiNj/p/PCN8wAAAACCwnQvAAAAAEEhSQEAAAAQFJIUAAAAAEEhSQEAAAAQFJIUAAAAAEEhSQEAAAAQFJIUAAAAAEEhSQEAAAAQlP8PNniXTAF1ndMAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxcAAAE/CAYAAAA0QD5OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5TU9Znn8c/ToICKkijS7QUaJqKtqDj2MLjGgMQzo4kymdU4zpLRVbMEyWgiZAaUPc4kx2yTLLoJEmeXk2BCovEkGi87mQm4maBxxolpJpBgVDQdOhBsLgGRi3S66Wf/qCoomr7U5fer3+39OocD1V31q6er6Of7fb7P7/ctc3cBAAAAQLXqog4AAAAAQDpQXAAAAAAIBMUFAAAAgEBQXAAAAAAIBMUFAAAAgEBQXAAAAAAIBMUFEBAzu8XM/jnqOAAAAKJCcZFBZrbJzN41s71m9raZ/ZuZzTGzkv4/mFmjmbmZDQ05zkGfx8xuMrPXzWyPmW03s2+Y2cklHHuWme3L/3nXzHqKbu+rJF53/4a7X1PJYwFgIOTtcPJ2/rjnmVl3pY8HcDSKi+y6zt1HShonabGkBZK+Fm1IFflXSZe7+ymSJkgaKun+wR7k7o+6+0nufpKkayRtLdzOf+0oYQ/IAFAC8nYZeRtANCguMs7d97j7s5L+QtItZjZJkszsw2b2MzN7x8w2m9nfFz3shfzfb+dXjC4zsz8ws38xs9+Z2U4ze9TMRhUeYGYLzOy3+VW3183sg/mv15nZQjP7Vf6x3zGz9/b3PH3Ev9nddxZ96ZCk9wXx2pjZFjP7GzP7haQD+a/9dzNry/8cr5jZzKL7f9zM1uT/PTS/evcJM3vTzHab2dIg4gKQbeTt/pnZ2Wb2TP7naTOzOUXfu7zo9ekws5aimIcUdUEuCSIWIKsoLiBJcveXJW2RdEX+S/sl3SxplKQPS7rDzD6S/94H8n+Pyq8YvSTJJLVIOkNSk6SzJf29JJnZuZL+WtIf5Vfd/lTSpvwx7pL0EUnT8o/dLekrAzzPMczs/Wa2R9JeSddL+lJlr0KfblJuheyU/O2Nki7P3/68pMfMbMwAj/+QpEslXSLpY2Z2VYCxAcgw8vYxxxwi6Z8k/Vs+rqsl3Wtm0/J3WSbpf7j7yZLOkfR0UcyHirogP6s2FiDLKC5QbKuk90qSu69x91+4e4+7/1zSt5UbSPrk7m+6+3Pu3unuOyQ9WHT/Q5KGSTrfzI5z903u/qv89z4haZG7b3H3TuUGthvKOQ3J3V/Mt9fPkvQ/dWQADMKX87G9m3+u77j7W/nX5bH8czUP8PiW/CrjJklrJE0OMDYAIG8f8X5Jw939C+7+e3ffKOkR5RaJJKlL0kQzO9Xd97r7TwJ4TgC9UFyg2JmSdkmSmf2xmf3IzHbkV5fmSDqtvwea2elm9ni+hf6OpG8V7u/ub0r6tHID0Pb8/c7IP3ScpKcsd4Hi25JeVW5QG6gb0Cd3/62kH0h6vNzHDmBz8Q0z+69mtr4o3vM0wOsiqaPo3wckcV4wgCCRt48YJ6mxEFc+tnmS6vPfv0XSRZI2mtlPzOxPA3hOAL1QXECSZGZ/pNwg9WL+S49JelbS2fnVpf+tXAtdkryPQ7Tkv35RvuX8saL7y90fc/f3K5f8XdIX8t/aLOkadx9V9Gd4fsDp63kGM1TSH1TwuP4cjsHMJkj6B0l3SDrV3UdJek1FPycA1Ap5+xibJb3WK66R7v7n+Z/nVXf/C0mnS1oq6XtmdnyFMQPoB8VFxpnZyWZ2rXKrRt9y91/kvzVS0i53P2hmUyT9l6KH7ZDUo9wuHyq6/z7lLuI7U9LfFD3HuWY2w8yGSToo6V3lVrmk3OD3eTMbl7/vaDP7swGep3f8s8xsrOWMU+46iB8Wff/rZvb1Ml6SgZyk3CC0I3do+7hynQsAqBnydr9ezD/+02Y23HIba1xkZn+Y//rN+VOiDknao1w+75G0XbkLusdW8JwAeqG4yK7/a2Z7lVvpWaTcuba3Fn1/rqTP5e9zn6TvFL7h7geUGwz+Nd96nirps5L+ULmE/X1J3ys61jDltk3cqdxpQqdLujf/vS8rt9K2Ov9c/y7pjwd4nt7OV+7ivX3KbW/4uqT/VvT9s/Nfr1r+HOalkl6W9JZyhQXn7AKoFfL2ANy9S7lNNP6TpHblCp1/0JHTUa+V9Ho+5hZJN7p7t7vvlvRFSWvzMXNtHFAFc6cbiHTKt7vXK9fy74o6HgDAwMjbQPJRXAAAAAAIBKdFAQAAAAgExQUAAACAQFBcAAAAAAgExQUAAACAQAwN46CnnXaaNzY2hnFoAAjV2rVrd7r76KjjqCVyNoCkymLOjrtQiovGxka1traGcWgACJWZtUcdQ62RswEkVRZzdtxxWhSQUgunr9TC6SujDgMAAGQIxQUAAACAQIRyWhSA6BS6FRuebz/q9uI1N0cWEwAAyAY6FwAAAAACQecCSJlCh4KOBQAAqDU6FwAAAAACQecCSCk6FgAAoNboXAAAAAAIBMUFAAAAgEBQXAAAAAAIBMUFAAAAgEBQXAAAAAAIBMUFAAAAgEBQXAAAAAAIBMUFAAAAgEBQXABlWjh9pRZOXxl1GAAAALFDcQEAAAAgEEOjDgBIikK3YsPz7UfdXrzm5shiAgAAiBM6FwAAAAACQecCKFGhQ0HHAgBQC4w3SCI6FwAAAAACQecCKBMrSACAMHGNH5KMzgUAAACAQNC5AAAAiBGu8UOS0bkAAAAAEAg6FwAAADFExwJJROcCAAAAQCAoLgAAAAAEguICAAAAQCAoLgAAAAAEguICAAAAQCAoLoAKLZy+8vAe5AAAAKC4AAAAABAQPucCKFOhW7Hh+fajbrMfOQAAyDo6FwAAAAACQecCKFOhQ0HHAgAA4Gh0LgAAAAAEgs4FUKGkdSzotAAAgLDRuQAAAJnEluKV47VDf0rqXJjZJkl7JR2S1O3uzWEGhSLt03N/j1sTZRRIsCTsbhXHmJKMnA0AiEo5p0Vd6e47Q4sEABAkcjbQjyQsusQVrx0GwzUXcVXoWLz7/NG36WCgTHHe3YpBCgCAdCm1uHBJq83MJf0fd18eYkwAgOqQs4EBxHnRJe547TCYUouLy919q5mdLuk5M3vN3V8ovoOZzZY0W5LGjh0bcJgZVOhQ0LFAQOI4ADBIhYacDQCIREnFhbtvzf+93cyekjRF0gu97rNc0nJJam5u9oDjBACKkBKRs4HSkEsqx2uH/gxaXJjZiZLq3H1v/t9/IulzoUeGHDoWyAAGqeCQs5EodOeB1CmlczFG0lNmVrj/Y+7+g1CjAoAiXPhdFnI2ACAygxYX7t4m6eIaxAIAqBI5uwir4vHFjohAarEVLYDY48JvAACSgeICAJAuaVgVT2LM5WBHRCC1KC4AJAYdCyQOk2ekGN1k9IXiAkiaCicrDALIjCSviqeh61KOKH6utL+mQMQoLgAACFrWioS44HWuCXbww0AoLoCkqHCywiCAzEriBDPJXZe4o+ALTdu6jqhDQIxQXAAAEDSKhNqicKip3jv4AcUoLoCkqHCywjauQAIxKQ4eBV/gCh2LA3s6JTHOIIfiAgCAsDCBrY0wCgeKkEFNmFwv6chpt4BEcQEkT4UDHStJACB2qAoQnXH0heICAACkQ5AdC67fACpCcQEAABCGjBQqdCxQjOICABC+lE6qkEJc+A1UheICAAAgDBQqyCCKCwBAeDJyWghSiP+jQEUoLgbDQAgAAKrBHAIZQnEBAAgPp4UAQKZQXPSHVj4AAABQFooLAED4WJgBgEyguOgPrXwASDbyNwDUXF3UAQAAAABIBzoXg2HFCwCShWvmACAydC4AAAAABILOBdKFFUoAXDMXmIXTV0qSFq+5OeJIACQFnQsAAJB4C6evPFwMAYgOnQukA+dYA+iN3/+KFSbpG55vP+o2HQwAg6G4AAAAiUUhFB5eS1SC4gLpwDnWABCYwmSSySWAclFcADXGYA3EGAsUiUMhFLwgu0G8L9lDcYF0YUIAAIFhQgigXObugR+0ubnZW1tbAz8ukGS9V4ImTRsnicE7bsxsrbs3Rx1HLZGzdeymECOm5f5mwQIZFkTHIuwxL4s5O+7YihYAAACRYAvh9OG0KKBGOC8YiDE2hQCOUc04xZiXXRQXAAAAqCm2EE4vigugxqpOnKysAuEJ4/eK31lkGMVC9lBcAAAAoKY4bSq9KC6ApOi9mw2roUC88TsLIIMoLlAbDKoAAKAXOhbpQ3EBJAW72QDJwu8saoxTjBAHFBcIF6cFVIQBAqgCeQYAIlNycWFmQyS1Svqtu18bXkgABsSECSUgZ8dIFL+zFFjRq+F7wLauiJNyOhefkvSqpJNDigVpxGkBZWGAQICyl7PplAKRYKxCsZKKCzM7S9KHJX1e0rxQIwJQFpI6eiNnZxgFVvQieA/Y1hVxUmrn4kuS/lbSyBBjQZoxsEkaPPEzQCAg2czZdEqBmqLbjr4MWlyY2bWStrv7WjObPsD9ZkuaLUljx44NLEAAfSOpoy/k7IyjwIpehO8B+R9xUErn4nJJM83sQ5KGSzrZzL7l7h8rvpO7L5e0XJKam5s98EiTIu4JPe7xpVS5hQADBKpAzia/ATVBtx19GbS4cPd7JN0jSflVsM/0HqQA1B5JHX0hZ0MSBVYc8B4go/ici6DE/SK6uMeXchQCAIC0YkxDsbKKC3dfI2lNKJEAqAhJHf0hZwPZk9RFrKTGjWPRuQhK3C+ii3t8GUHSBAAAaUZxAQAA0idji2lJ3UEwqXGjfxQXQYt7Eot7fEgMBgBUJGMTvlThvQNQAooLAAD6w4Q6eTK6gUlSNw5JatzoH8UFEISMDF4SLWxUKKMTvlTgvUuEQi4GokZxAeQxSQZwGBPq5Mr4BiZJHcOSGjeORXEBFJk9r0VqX1H6YJTBCQgtbFQk4xO+ROO9izW6yYgbigtkXnFi3n/bQbWt36blt6wkMQNZxoQ6+XjPgEhQXADKdSz233ZQF07ZdPh2SR2MDE9AKL5QkQz9jqQO710s0U1G3FBcoKbimPwWr7lZal+htvXbDn9twsVjIowIQGwwoQaAslBcAJI0bo2W37JSs+e15AqLcicUTEAAABGK06Idso3iAjWRhAvOCh0MJEcc/x8hBTJ4miMABIXiImwMUsnC+wQAAFAxigvUBBecIUhJ6IQhgTK4tTQABI3iIiwMUgAAAMgYioskSUGBwsoygkAnDKHI8NbSABAUiouwMEgBAAAgYygukoBTrCpTyuvEa5lodCwQCvIBAFSM4iJsDFIAACDFOEUVxSgukoBTrMpTSqeHbhAADCjqCWPUzw+gMhQXAAAAKBvbgqMvFBdxMtjqOavqpSml00M3CAD6FNSEsdrHMWEFkoniAgAAAGVjW3D0heIiDjj/PxylvH68xgBwlGonjNV2Hpiwph/vbbpRXAAAgNKw+IU+UCSgmLl74Adtbm721tbWwI+beiRtIHJmttbdm6OOo5bI2ShZGeMUq9PZUsr73burNWnauEEfM5gs5uy4o3MBZBiDP4CScPougBJRXMQJSRoAkBJJXLRgwaV85Vxjw/U02UBxAWQQWz0CNZCm1X227wZQIooLxAYTXABAFFhwqVwl3Qhe13SjuEAsLJy+Um3rOjRhcn3UoWQCrWkgRGm+PiENPwOAUFFcIHKFwuLAnk5teL594AlvmgZpAEAssOBSPV4zFFBcIFLFhUUBHYzaYTAAQsD1CQAyjOICkZswuf7wea4nnDJMEybXHzvpjfI0g/bpalu/TcsfvIfJOACkGDkeqB7FBSJV3IoudCxI7gBSoZYdC7okAGKiLuoAgIIBC4txa3J/RkzL/SncDlP7dLU92yS9+7wmTHxNs277O7U923T4nNxqjnt4IgAAAJAidC4QC3QrAKACad6ZCkAiUVwgWWo5YI5bownjdPiai0dX5K65WDyzwuMxCQAAAClHcQEEgO0LAUSCnakAxAzFBTCYfAej4o5F0XEkMQkAAACpRXGRdExUI1XoWBS20qWDASASjAEAYmLQ4sLMhkt6QdKw/P2fcPe/CzswIBK1KNaYBCBE5GyEigUtAIMopXPRKWmGu+8zs+MkvWhm/+zu/x5ybBhIxB8qV7Pnirniz+kovg1EiJyNVCPfpl+U7/HatWtPHzp06FclTRIf2dCXHkkburu7P37ppZdu7+sOgxYX7u6S9uVvHpf/44GFCMQBOzkhJcjZCAU5siwUQMk1dOjQr9bX1zeNHj16d11dHbmzl56eHtuxY8f5HR0dX5XU59WoJV1zYWZDJK2V9D5JX3H3nwQXJioSxcXBDC79YgCJp6wO8ORspBHXuKVfTN7jSRQW/aurq/PRo0fv6ejomNTffUoqLtz9kKTJZjZK0lNmNsndNxTfx8xmS5otSWPHjq0ibCAC7OSEFCFnI3DkyJLEZHKM6tRRWAws//r0e8pYWbtFufvbZrZG0tWSNvT63nJJyyWpubmZN6VWavyhcpISNbiQ2LOJAT6HnI004Rq39OM97t+8efPOOOmkkw597nOf2xb0sX/84x+fcPvttzcePHiwbsaMGXtWrFixua6u8stNStktarSkrvwgNULSVZK+UPEzAnGWgIIJGAg5G4OqZoGIHDkgJsdIorlz5457+OGH22fMmLF/+vTp5zzxxBMn33jjje9UerxSOhcNkr6RP4e3TtJ33P0fK31CpEACBhdWrrMt4wM8ORuBiOvvT9ziQfCS9h7fPeVr50rS/3r59teDON6yZctOXbp06RgzU1NT07tPP/30r4u//8ADD5z2yCOPjO7q6rLGxsbOJ5544tcjR47sWbFixXtaWlrOqKur85EjRx5qbW19vbW1dfitt946vqury3p6evTkk0/+6sILL+wsHKu9vf24ffv21V111VX7JWnWrFm/e/rpp98TanHh7j+XdEmlTwBkTVwHZGQDORv9YlOOmiH/o1Ktra3DlyxZ0vDSSy+91tDQ0L1t27Yhve8za9as3fPnz98pSXfdddcZS5cuPW3RokXbFy9e3LB69eqN48eP79q5c+cQSXrooYdGz507d9sdd9yx6+DBg9bd3X3Usdrb249raGjoKtweN27c7996663jqvkZ+IRuVCbmg1LGV66Rx/sOlI/OL1CaQsfijZ9uPan4djUdjFWrVp183XXX7W5oaOiWpDFjxhzqfZ+1a9eOuO+++87cu3fvkP379w+ZNm3aHklqbm7eN2vWrMbrr79+96xZs3ZL0mWXXbZ/yZIlDVu2bDn+pptu2l3ctZCk3O7lRzOzSsOXxIeDAIFZOH2lFk5fqQ3Pt2vD8+2Hb5fyGAAI3bg1uT8jpuX+FG4nBPkSWeDuMrMBN9mYPXv2+GXLlv1m48aNv1ywYMHWzs7OOkl67LHHfnP//fdv3bx58/GTJ0++oKOjY8icOXN2PfPMM2+OGDGi55prrpn47LPPjiw+VmNjY1dxp6K9vf34+vr6rt7PWQ46FyhPwtrqrLQBQHno/AKlKXQogrzm4uqrr37nhhtueN+99967rb6+/tC2bduG9O5eHDhwoG7s2LFdnZ2d9vjjj7+3cFrTK6+8MmzGjBn7Z8yYsX/VqlWj2trajt+1a9ehpqamzgsuuGB7W1vbsHXr1o2YOXPm3sKxxo0b13XiiSf2/PCHPzzxyiuv3P/oo4+e+slPfrLPT94uFcUFEJByBmROOwAQmZguBvWHfFk6Xpvka25uPjh//vy3rrjiivPq6up80qRJB5588slNxfdZuHDh1ilTpjSdeeaZv29qajqwb9++IZJ09913n7Vp06Zh7m7vf//735k6deq7ixYtqv/ud7976tChQ3306NFdLS0tW3s/58MPP9x+++23jz948KBdeeWV73z0ox/dU83PQHGB8iTwsy4AAOVjggqUJqhdogruvPPO3915552/K/7agw8+eLgoWLBgwY4FCxbs6P241atX/6r311paWjpaWlo6Bnq+D3zgAwfeeOONV6qJuRjFRdiYhGdOKQMypx2Uh9cJyC7y5eDo7iBOKC5wrFIKIoolRIABE2ULY4Ena4tGWft5AVSF4iIsCbvwGdFgkjyw3qtxJ5wyLMpwAESIfNk/ujuIE4oLHEFBhJg7sCe3PTcDKAYVRj7LWo7M2s8LIBAUF2HhwmegaoXi4cZRX5R0pLgAAByLBRfEAcUFjqAgirUsr9ZPmFx/1O0svgYoUxj5LGs5Mms/L4BA8AndYUvYJ6ACcbR4zc2hFBR84i8C0z79yCQcAAI2b968M+67774xYRz7zjvvPLO+vv6iE0444ZIgjkfnAseiGIoVthg8Ios/M6oURj7LWo4c6OelqwEk3kc+8pG3P/OZz2xvamqaFMTxKC5QPgYTJBwFGwKTwYue+X0Betk05VxJUuPLgXyY3rJly05dunTpGDNTU1PTu08//fSvi7//wAMPnPbII4+M7urqssbGxs4nnnji1yNHjuxZsWLFe1paWs6oq6vzkSNHHmptbX29tbV1+K233jq+q6vLenp69OSTT/7qwgsvPOoCxg9+8IP7g4i7gOICiDm2GAQQN7PntUjtKzJVVAG10NraOnzJkiUNL7300msNDQ3d27ZtG9L7PrNmzdo9f/78nZJ01113nbF06dLTFi1atH3x4sUNq1ev3jh+/PiunTt3DpGkhx56aPTcuXO33XHHHbsOHjxo3d3dof8MFBdJEYfEnaAVOibiGAgFGwKToYueizt++287qLb12zRhYsRBAVEqdCwO/vSko25X0cFYtWrVydddd93uhoaGbkkaM2bMod73Wbt27Yj77rvvzL179w7Zv3//kGnTpu2RpObm5n2zZs1qvP7663fPmjVrtyRddtll+5csWdKwZcuW42+66abdvbsWYeCCbiAhwrqoGQDKdc8tt2n5g/eobeN50ohpbF4CBMTdZWY+0H1mz549ftmyZb/ZuHHjLxcsWLC1s7OzTpIee+yx39x///1bN2/efPzkyZMv6OjoGDJnzpxdzzzzzJsjRozoueaaayY+++yzI8P+GehcxF2cugW1WKGr8ticS49y8P8CgUn6xLqE3Ntnx699RciBxVAGulQoUaFDEeA1F1dfffU7N9xww/vuvffebfX19Ye2bds2pHf34sCBA3Vjx47t6uzstMcff/y9DQ0NXZL0yiuvDJsxY8b+GTNm7F+1atWotra243ft2nWoqamp84ILLtje1tY2bN26dSNmzpy5t9o4B0LnApVj60UAyDY6FkCgmpubD86fP/+tK6644rxzzz33/Llz557d+z4LFy7cOmXKlKYrrrhi4jnnnHOw8PW77777rIkTJ55/zjnnXDB16tS9U6dOffeb3/zmeydOnHjBeeedd/4bb7wx/BOf+MTveh9vzpw5Z40ZM+aigwcP1o0ZM+aiefPmnVHNz2DuA3ZeKtLc3Oytra2BHzfT4rhSEmRMvTs0I6ZVdWw6FqiUma119+ao46glcnaGBZx7U43XKpaCztnr16/fdPHFF+8M6nhptX79+tMuvvjixr6+x2lRKF+cTtXKCIolAAgXeRYIBsVFUqR94h7w9RwMDkCFWCzIlgztdlU1XiugJBQXKF+KEmzcV6q4QB3AgFKQh6NGngWCRXGBeGGABKLB6Y7ZxvtcOl6rtOvp6emxurq64C9KTomenh6T1NPf9ykuULlqEmzEE5darVRVe1w+7A1AnygGA0OeRS8bduzYcf7o0aP3UGAcq6enx3bs2HGKpA393YfiAkDoGLQTIEWnOwJApbq7uz/e0dHx1Y6OjkniIxv60iNpQ3d398f7uwPFBWorJqttYa9UBd0ZYVIOlCBLhRHFYODIs5CkSy+9dLukmVHHkWQUFwBCw4WSCcQkFQBQBYoL1FbMVtvCmuRyDi9QQzHpiEai3J8xS69NCMjpwOAoLsJEEkfGUWQBAJAtFBeIRkYKLibTQA3ErCMaS1nu7gSAUzyB0lFchIEkDhyFARgAgGyguAAApEPSF3DCXIiiu1MVTvEESkdxEYZC0t446ujbAAAkBBNpAJWguAAAIEq1PJU2xMWuqIqRWj4vhVZtUNgmG8VFGAoDQ8+eo2/XooNByxsAUAUuXgZQDYoLAACiFNT1EBEtLkVVjFAEpQ/vaTpQXIQhigvn2KEKABAALl4GUA2KCwAA4qDajkVEi0tRFSMUQenDe5oOFBdhqmXXgG0GAQABYmIXLCbMyIpoi4tSJ8JMmNEHEjUAaPDFpRqNoVHl4kqel/Ej3nhfkm3Q4sLMzpa0UlK9pB5Jy939y2EHhgpRgAGZRs4G4oWLlJE1pXQuuiXNd/f/MLORktaa2XPu/suKn7XU80O5SBl9SG2i5v83ghF8zkYyMIYOKrXjBxAjgxYX7v6WpLfy/95rZq9KOlMSAxWAwxik44GcDcQLFykja8q65sLMGiVdIuknVT1rqRcf1/oiZVZ1EiF1iZrVRYQksJyNZGKjj2OkbvwAYqjk4sLMTpL0pKRPu/s7fXx/tqTZkjR27NjAAgQQb5xmEE/kbCBeyInIipKKCzM7TrlB6lF3/15f93H35ZKWS1Jzc7OX9OylrqLUqmPBynGipCZRs7qIgIWWs5FM5JRjxGn8YEEGaVPKblEm6WuSXnX3B8MPCUCScJpBvJCzAQBRKqVzcbmkv5L0CzNbl//ave7+T+GFVWOsHCMO+H+HYKQ/ZwMVitMiCKeUIq1K2S3qRUlWg1gAJBgDYjyQs4skdcEoqXGHidcESIxoP6E7bkhaAACkThy7BJxSirSiuAAApEtSN+lIatxh4jUBEofiAgCAUjG5TaQ4dwniFAsQBIoLBIMBF0BcJHWTjqTGHSZeEyBxKC4AABgMp+ekQta7BHHs3CB9KC5QHQbcmmFQAMqU1DyU1LjDxGsSiLZ1HVGHgAyguEAyULQAiBKn5yDBCotTB/Z0HnWbxSqEgeICgxowCTHghi6OWygCAJKjd8eCDgbCRHGBeCv3tCuKnEBQwAD9ILcggSZMrpd0ZJGqcBsIA8UF+lXWinlYA27nunCOmyBx3kIRABB/hXHjxlFfPOo2EAaKC8TbsMlH3x6sY8GF5VXhFJTpYvoAAAjhSURBVCwASC86FqgFigv0K9IV897FQt0ptXvumGKCDwCoBuMIaoHiAsnQu4PRWwIvLI9jV4BTsAAAQDUoLjCoSCaYCSwWAAAAso7iAumSgCIkCdc1xCkWAACQHBQXiLeAi4U4TuQBAADSguICqDGuawAApBHjGiSKi3TjeoXDknAqEgBkEfkYSBeKCyAiDKSDY9IBAPHHAh6KUVykER8od4yBTkUiCQJA7TEhBdKJ4gJAJAaaSDDpAIDk4FpCFKO4SCM+I6JffXUsmMACQHj6y62xm5AyZgKBoLgAUFOlFHWxm3QAKF9hsq7boowCNUSuhkRxkW6svgyICSwAhKdt/TZJg3eHI8+9XKcIBIriAugPA0woyinqIp90AChfPndOmPiaJKnlGyskSY+u+GxUEQGooWiLCyZviAEmsAAQnhNHDZcU41zLdYpAoOhcAL3RIq+J2E40gIwJ/NTQXpP15Q/mrrlYPDOYwwOIt2iKCyZvAABkwuGiJe5jfVzjAhKGzgXQGy1yABkQ+nbc5E4gk6IpLpi8IaXYeQoAeknY2QrkcaA62ehcxDyRIab4/wIgxdiOG0AYoi0umLwhJfi0bwBZNmDOS8jZCuRxIBjp7lwkrBULAECtMXkGEKR0FxdAjXB6AYAsKmu1P+YLe+RxIBjpLi4S0opF/DHYABEhfwNAoqS7uIgCA2GmlVJ8UKgAqKkQx6U0rvan4WcAopSN4oKJPirEBX5ARLhmDgASKRvFRS0wEGIQFCoAaqqG4xJ5DEABxQUwgDS2/IFE4Jo5AEgkiougMBBiEBQqAGqKcQlABAYtLsxshaRrJW1390nhh4TEycDARSGAJElV3k5xXsHAWIgBkqmUzsXXJS2TtDLcUFKCgRCDYKBEDXxd5G0UMC4BqKFBiwt3f8HMGsMPBbFRaicigxexV7KSxuobao28jSRj8wsg2eqiDgAJ0z79SBGByC2cvvLwwAsAscA4AWRaYBd0m9lsSbMlaezYsUEdFrVUbiciQxcLVrKSxuob4oycjbhi8wsg2QIrLtx9uaTlktTc3OxBHRcxkcFToOJs0MKF9weDIGcjcGWOExQPQDqxFS2OqLQTkYEJbCUraay+AUDlyJlAMpWyFe23JU2XdJqZbZH0d+7+tbADQ8xk6BSoJOi3cKHDBJG3YykLv4sljhOcMgqkWym7Rf1lLQJBjKR58KtSJYMfAyZqjbwNAIiKuQd/qm1zc7O3trYGflwAJcrCKmlIzGytuzdHHUctkbND1LubOGJa7m9+N+lYIBBZzNlxx1a0AAAAAALBBd1AGrEqCsQD16v1i44FkE50LpA6fLAcAABANOhcAAAQNjoWADKC4gKpUfb2hpymAAAAEChOiwIAAAAQCDoXSI2SPxGbD5oDAAAIBZ0LAAAAAIGgc4HUGXR7Q7aGBAAACAWdCwAAAACBoHOB7KJjAQAAECg6FwAAAAACQXEBAAAAIBAUFwAAAAACQXEBAAAAIBAUFwAAACjJwukrD39YLdAXigsAAAAAgWArWgAAAAyo0K3Y8Hz7UbcH/eBaZA6dCwAAAACBoHMBAACAARU6FHQsMBg6FwAAAAACQecCAAAAJaFjgcHQuQAAAAAQCHP34A9qtkNSe+AHDtZpknZGHUSJkhJrUuKUiDUMSYlTGjjWce4+upbBRI2cHThiDV5S4pSSE2tS4pTI2YkSSnGRBGbW6u7NUcdRiqTEmpQ4JWINQ1LilJIVK3KS9J4Ra/CSEqeUnFiTEqeUrFjBaVEAAAAAAkJxAQAAACAQWS4ulkcdQBmSEmtS4pSINQxJiVNKVqzISdJ7RqzBS0qcUnJiTUqcUrJizbzMXnMBAAAAIFhZ7lwAAAAACFCmiwsz+6iZvWJmPWYWu10IzOxqM3vdzN40s4VRx9MfM1thZtvNbEPUsQzGzM42sx+Z2av59/5TUcfUFzMbbmYvm9n6fJyfjTqmgZjZEDP7mZn9Y9SxDMTMNpnZL8xsnZm1Rh0PyhP3nC2Rt4NGzg4PeRthyXRxIWmDpP8s6YWoA+nNzIZI+oqkaySdL+kvzez8aKPq19clXR11ECXqljTf3ZskTZX0yZi+rp2SZrj7xZImS7razKZGHNNAPiXp1aiDKNGV7j6ZbQ0TKbY5WyJvh4ScHR7yNkKR6eLC3V9199ejjqMfUyS96e5t7v57SY9L+rOIY+qTu78gaVfUcZTC3d9y9//I/3uvcon1zGijOpbn7MvfPC7/J5YXSJnZWZI+LOmrUceCdIt5zpbI24EjZ4eDvI0wZbq4iLkzJW0uur1FMUyoSWZmjZIukfSTaCPpW75lvU7SdknPuXss45T0JUl/K6kn6kBK4JJWm9laM5sddTBIHfJ2iMjZgSJvIzRDow4gbGb2/yTV9/GtRe7+TK3jKYP18bXYroIkjZmdJOlJSZ9293eijqcv7n5I0mQzGyXpKTOb5O6xOj/azK6VtN3d15rZ9KjjKcHl7r7VzE6X9JyZvZZfwUVMJDhnS+Tt0JCzg0PeRthSX1y4+1VRx1ChLZLOLrp9lqStEcWSKmZ2nHKD1KPu/r2o4xmMu79tZmuUOz86bgPV5ZJmmtmHJA2XdLKZfcvdPxZxXH1y9635v7eb2VPKncbCIBUjCc7ZEnk7FOTswJG3ESpOi4qvn0o6x8zGm9nxkm6S9GzEMSWemZmkr0l61d0fjDqe/pjZ6Pzql8xshKSrJL0WbVTHcvd73P0sd29U7v/ov8R1gDKzE81sZOHfkv5E8Rz4kVzk7YCRs4NH3kbYMl1cmNmfm9kWSZdJ+r6ZrYo6pgJ375b015JWKXcB23fc/ZVoo+qbmX1b0kuSzjWzLWZ2e9QxDeBySX8laUZ+W7t1+dWbuGmQ9CMz+7lyE5bn3D3W2wUmwBhJL5rZekkvS/q+u/8g4phQhjjnbIm8HRJydraRtxOIT+gGAAAAEIhMdy4AAAAABIfiAgAAAEAgKC4AAAAABILiAgAAAEAgKC4AAAAABILiAgAAAEAgKC4AAAAABILiAgAAAEAg/j+kzt7jJKUhXQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "datasets_train, datasets_test = generate_data()\n", "\n", "for i, (ds_train, ds_test) in enumerate(zip(datasets_train, datasets_test)):\n", " ds_train = datasets_train[i]\n", " ds_test = datasets_test[i]\n", "\n", " X = ds_train[0]\n", " Y = ds_train[1]\n", " \n", " X_test = ds_test[0]\n", " Y_test = ds_test[1]\n", "\n", " i_c0 = (Y == 0)\n", " i_c1 = (Y == 1)\n", " \n", " #TRAIN: true and false predictions \n", " i_c0_t = (Y[i_c0]==0); i_c0_f = (Y[i_c0]==1)\n", " i_c1_t = (Y[i_c1]==1); i_c1_f = (Y[i_c1]==0)\n", " \n", " plot_predictions(i+1, X, Y, X_test, Y_test, pred_train=Y, pred_test=Y_test )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Linear Classification" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1. Linear least squares for classification" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the previous practical session, we explored how to tune a linear regression using a least squares loss function. We saw that minimization of the least square loss function led to a simple closed-form solution (see also Week #2 Slide 31). \n", "\n", "As discussed in the capsules, a similar idea can also be usef for classification tasks, with the difference that now we additionally require a decision rule for classification. The decision rule will allow us to classify the predicted values (*real numbers*) in a class (for example *binary*). \n", "\n", "We express the decision rule as $sign(y(x))$, where $y(x) = W^\\top x$ is our model. \n", "\n", "Where we define: \n", "\n", "$$ sign(a) = \\left\\{\\begin{array}{} \n", "+1, &\\text{if } a > 0\\\\\n", "-1, &\\text{otherwise}\n", "\\end{array}\\right. $$\n", "\n", "Thus, such a classifier will return class '+1' for all the points lying on one side of the decision boundary and '-1' for the ones lying on the other side.\n", "\n", "Let's implement this simple classifier." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# least squares for classification\n", "def train_LSC(X, Y):\n", " #Note: we transform the labels from {0,1} to {-1,+1}\n", " Y = (ds_train[1]*2)-1\n", " Y = Y.reshape(Y.size,1)\n", " \n", " # We learn the parameters of the model \n", " # (this is the same estimation as the one last week for OLS)\n", " A = np.linalg.inv(np.dot(X.T, X))\n", " B = np.dot(X.T, Y)\n", " \n", " return np.dot(A, B)\n", "\n", "# for visualizing the decision boundary \n", "def calculate_decision_boundary(W):\n", " x_1 = np.linspace(-10,10) # <- this would give x1;\n", " \n", " #the goal is to calculate x2 given x1 and the weights \n", " x_2 = (-W[0] - W[1]*x_1) / W[2] \n", " return x_1, x_2" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i, (ds_train, ds_test) in enumerate(zip(datasets_train, datasets_test)):\n", " ds_train = datasets_train[i]\n", " ds_test = datasets_test[i]\n", "\n", " X = ds_train[0]\n", " #Add a column of ones to account for the bias term\n", " X_b = np.array([np.ones(len(X)), X[:,0], X[:,1]]).T\n", " ####\n", " Y = ds_train[1]\n", " X_test = ds_test[0]\n", "\n", " #Add a column of ones to account for the bias term\n", " X_test_b = np.array([np.ones(len(X_test)), X_test[:,0], X_test[:,1]]).T\n", " ###\n", "\n", " Y_test = ds_test[1]\n", "\n", " # 1) train weights\n", " #let k be the number of casses (2), dim - dimentionality of the data (2)\n", " W = train_LSC(X_b, Y) # dim x k\n", "\n", " # 2) Calculate the discriminant function and predict \n", " # a) for test data\n", " y_x = np.dot(W.T, X_test_b.T)\n", " pred_test = 1*(y_x>0)[0]\n", "\n", " # b) for training data\n", " y_x = np.dot(W.T, X_b.T)\n", " pred_train = 1*(y_x>0)[0]\n", "\n", " \n", " # 3) Calculate decision boundary\n", " line_x, line_y = calculate_decision_boundary(W)\n", " \n", " \n", " # 4) plot predictions\n", " plot_predictions(i+1 ,X,Y, X_test, Y_test, pred_train, pred_test, line_x, line_y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 1 (Lin. Classifier):** for which dataset does the model achieve the best performance? Explain in your own words, why this is the case." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2. SVM" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we will use the sklearn library for the first time. We will train and evaluate a Suport Vector Machine (SVM) model, which was brefly introduced in the capsules." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, let's generate some linearly separable data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#let's load a new linearly separable dataset\n", "X_l,Y_l = make_classification(n_features=2, n_redundant=0, n_informative=2,\n", " random_state=1, n_clusters_per_class=1)\n", "fig = plt.figure(figsize = (10,5))\n", "xfit = np.linspace(-2, 2)\n", "plt.scatter(X_l[:, 0], X_l[:, 1], c=Y_l)\n", "\n", "for m, b, d in [(1.5, 1, 0.2), (10, 1, 2), (-1.5, 1.2, 0.2)]:\n", " yfit = m * xfit + b\n", " plt.plot(xfit, yfit, '-k')\n", " plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none',\n", " color='#AAAAAA', alpha=0.4)\n", "plt.xlim(-2,2)\n", "plt.ylim(-0.5,2.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that there exist many decision boundaries can be used to separate the given dataset.\n", "\n", "SVM choses the line that maximizes the margin to the closest point. Let's use sklearn to train an SVM model and visualize the results." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#import the SVM classifier\n", "from sklearn.svm import SVC \n", "model = SVC(kernel='linear', C=1E10)\n", "\n", "#let us fit the model (learn its parameters)\n", "model.fit(X_l, Y_l)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = plt.figure(figsize = (10,5))\n", "ax = fig.add_subplot(111)\n", "ax.scatter(X_l[:, 0], X_l[:, 1], c = Y_l)\n", "ax = plot_svc_decision_function(model,ax)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that the model has chosen the decision boundary with the maximal margin. Here, the pivot elements that touch the (dotted) margins are called support vectors." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now try to classify Dataset 3 from our innitial set using SVM." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X, Y = datasets_train[2]\n", "X_test, Y_test = datasets_test[2]\n", "\n", "# DEFINE SVM model\n", "model = SVC(kernel='linear', C=1E10)\n", "\n", "#let's fit the model (learn its parameters)\n", "model.fit(X, Y)\n", "\n", "\n", "fig = plt.figure(figsize = (10,5))\n", "ax = fig.add_subplot(111)\n", "\n", "plt.scatter(X[:, 0], X[:, 1], c=Y)\n", "plot_svc_decision_function(model,ax)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 1 (SVM)**: complete the cell below to calculate the training and the test accuracy of the above model.\n", "\n", "Hint: check the documentation of the sklearn SVM classifier [here](https://scikit-learn.org/stable/modules/svm.html#svc). Find the function that would return the predictions for the trained model." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "acc_train = None #QUESTION - complete this line\n", "acc_test = None #QUESTION - complete this line\n", "\n", "print(\"Accuracy train: \", acc_train, \"%\")\n", "print(\"Accuracy test: \", acc_test, \"%\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that our dataset is noisy, i.e. the data points from the two classes overlap and there is no decision boundary for our SVM model that can perfectly separate all the points.\n", "\n", "While a large amount of research has been conducted on SVMs providing a solid theoretical foundation for this algorithm, here we will solely try to gain some intuition on how the algorithm works. Intuitively, the algorithm tries to optimize the trade-off between maximizing the margin around the decision boundary and minimizing the training error. We see in the above graph that the margin around the decision boundary is so tiny that it is almost not visible (no dotted lines). One of the possible reasons for this is that the trade-off between the error minimization and maximization of the margin around the decision boundary is miscalibrated. \n", "\n", "As mentioned in the previous tutorial, hyperparameter tuning is very important in machine learning and (almost) any result can be improved through hyperparameter tuning." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2 (SVM): [Advanced question that goes beyond the material covered in the capsule]** check the documentation of the sklearn SVM classifier [here](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html). Try to find the parameter that controls the regularization strength and change it in the definition of the SVM model above to allow for a \"softer-margin\". Measure the performance of your model. Can you observe any improvement?\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's run the SVM classifier on all the 3 datasets (*Note: please complete the previous question first, otherwise this next cell will run for a long time*)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i, (ds_train, ds_test) in enumerate(zip(datasets_train, datasets_test)):\n", " ds_train = datasets_train[i]\n", " ds_test = datasets_test[i]\n", "\n", " X = ds_train[0]\n", " Y = ds_train[1]\n", " \n", " \n", " X_test = ds_test[0]\n", " Y_test = ds_test[1]\n", " # 1) train weights\n", " #let's fit the model (learn its parameters)\n", " #we use the model form the cell above\n", " model.fit(X, Y)\n", "\n", " # 2) Calculate the discriminant function and predict \n", " # a) for test data\n", " pred_test = model.predict(X_test)\n", "\n", " # b) for training data\n", " y_x = np.dot(W.T, X_b.T)\n", " pred_train = model.predict(X)\n", "\n", " \n", " # 4) plot predictions\n", " plot_predictions(i,X,Y, X_test, Y_test, pred_train, pred_test, plot_svm=model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Probabilistic Models" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1. Naive Bayes " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Up to now we used linear classifiers, that is classifiers with a linear decision boundary. Here, we will implement the Naive Bayes classifier. It was introduced in the capsules and we will see if this classifier can come up with better decision boundaries." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In our toy datasets, the target variables $y \\in \\{0,1 \\}$ are simple binary variables, and hence, can be modelled with a *Bernoulli* distribution $p(y=k|\\pi) = \\pi^k(1-\\pi)^{1-k}$. Our feature variables $x \\in \\mathbb{R}^2$ are continuous variables. As discussed in the class, we can model continuous variables with a Gaussian distribution $\\mathcal{N}(x_j | \\mu_{jk}, \\sigma^2_{jk})$.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The fitting procedure for a Naive Bayes classifier is described below. Fill in the gaps with missing text (double-click on this cell to be able to edit in the text).\n", " \n", " 1. Calculate maximum likelihood estimate (MLE) for all the parameters of our distributions: \n", " $\\theta \\in \\{$ $ \\sigma^2_{jk}, \\mu_{jk}, \\pi $ \\} using the training set.\n", " 2. Use the estimated parameters and the Bayes theorem to make predictions. The corresponding formula is\n", " \n", " $posterior \\propto joint \\space distribution = class\\space conditional * class\\space prior$\n", " \n", " $P(y = k|x) \\propto$ P(y = k, x) = P(x | y = k) P(y) ." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's start by building the Naive Bayes model for our second dataset. After, we can easily train and test our implementation on all datasets." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X, Y = datasets_train[1] # we use the 2 circles dataset\n", "X_test, Y_test = datasets_test[1] # we use the 2 circles dataset\n", "\n", "#class indices\n", "i_c0 = (Y == 0)\n", "i_c1 = (Y == 1)\n", "\n", "X_0 = X[i_c0] #data of class 0\n", "X_1 = X[i_c1] #data of class 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Training the model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's start by calculating the prior probability for each class. The MLE (maximum likelihood estimate) of a Bernoulli random variable is simply given by the sample mean:\n", "\\begin{align}\n", " \\hat{\\pi} = \\frac{\\sum_{i=0}^n y_i}{n}, \n", "\\end{align}\n", "where $n$ is the total number of samples." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 2 (NB):** Fill in missing code in the following cell to calculate the prior probabilities for each class." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def calculate_prios(X_0, X_1, X):\n", " # TODO: calculate prior class 1\n", " prior_k0 = 0 # TODO: fix this line\n", "\n", " # TODO: calculate prior class 0\n", " prior_k1 = 0 # TODO: fix this line\n", "\n", " #let's store the priors in a dictionary\n", " prior_dict = {\"class 0\":prior_k0 , \"class 1\":prior_k1 }\n", " \n", " return prior_dict" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "prior_dict = calculate_prios(X_0, X_1, X)\n", "print(prior_dict)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's calculate the MLE for the parameters of a Gaussian. Note that although the Gaussian is embedded into a larger NB model, taking the log of the likelihood separates the different terms. We can call each dimension of our input dataset X a *feature*.\n", "\n", "As explained e.g. [here](https://www.statlect.com/fundamentals-of-statistics/normal-distribution-maximum-likelihood), the maximum likelihood estimates of the **mean** is simply the empirical mean:\n", "\n", "$\\hat{\\mu} = \\frac{1}{n} \\sum_{i=0}^{n} x^{(i)}$. For our algorithm we need to calculate this quantity for each class ($k$) and for each feature ($j$):\n", "\\begin{align}\n", " \\hat{\\mu}_{jk} = \\frac{\\sum_{i=0}^{n} \\mathbb{1}(y^{(i)}=k) x_{j}^{(i)}}{ \\sum_{i=0}^{n} \\mathbb{1}(y^{(i)}=k)} \n", "\\end{align}\n", "(*Note, $\\mathbb{1}(y^{(i)}=k)$ evaluates to 1 when $y^{(i)}=k$ and to 0 otherwise.*)\n", "\n", "\n", "Further, the MLE of the *variance* of a Gaussian distribution is simply the empirical variance: \n", "\n", "$\\hat{\\sigma}^2 = \\frac{1}{n} \\sum_{j=0}^{n} (x^{(j)} - \\mu)^2$\n", "\n", "(*Note the variance also needs to be calculated per class per feature using our training data.*)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 3 (NB)**: complete the implementation of the MLE estimates in the cell below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def calculate_MLE(X_0,X_1):\n", " #Calculate statistics for features per class\n", " class_summaries = dict()\n", "\n", " feat_0_class_0 = np.mean(X_0[:,0]), np.std(X_0[:,0])\n", " feat_1_class_0 = np.mean(X_0[:,1]), np.std(X_0[:,1])\n", "\n", " #put them in the summaries dictionary\n", " class_summaries['class 0'] = [feat_0_class_0, feat_1_class_0]\n", "\n", " feat_0_class_1 = 0,0 #QUESTION <- complete this line by calculating the statistics for class 1 feature 0\n", " feat_1_class_1 = 0,0 #QUESTION <- complete this line by calculating the statistics for class 1 feature 1\n", " \n", " #put them in the summaries dictionary\n", " class_summaries['class 1'] = [feat_0_class_1, feat_1_class_1]\n", " return class_summaries" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class_summaries = calculate_MLE(X_0, X_1)\n", "#lets print the result\n", "for c, class_summary in class_summaries.items():\n", " print(c)\n", " print(class_summary)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To summarize:\n", "\n", "- Our *class_summaries* dictionary (printed in the previous cell) contains a list with empirical mean and variance of each features for each of the two classes calculated on our training data .\n", "\n", "- Our *prior_dict* dictionary contains the prior probabilities for each class.\n", "\n", "Congratulations, we have completed the training procedure of our Naive Bayes classifier. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Now let's make predictions using our trained model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we assumed that our features are drawn from Gaussian distributions, we need to implement the Gaussian probability density function (pdf) that will return the probability of the input for a given mean and variance parameters (we will use the per class MLE parameters estimated earlier)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$ \\mathcal{N}(x,\\mu,\\sigma) = \\frac{1}{ \\sigma \\sqrt{2 \\pi}} e^{\\left(-\\frac{{\\left(\\mu - x\\right)}^{2}}{2 \\, \\sigma^{2}}\\right)} $\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def Gaussian_PDF(x, mean, stdev):\n", " exponent = np.exp(-((x-mean)**2 / (2 * stdev**2 )))\n", " return (1 / (np.sqrt(2 * np.pi) * stdev)) * exponent\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's apply our model to a single test point. \n", "\n", "Remember, all we need to do is to apply the Bayes theorem using the parameters we estimated earlier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#test point\n", "test_point = X_test[2]\n", "print('test point: ', test_point)\n", "print(f'value of feature 1: {test_point[0]}, value of feature 2: {test_point[1]}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 4 (NB):** complete the function in the cell below that calculates the posterior probability for a given test point." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def calculate_probability(x_test, class_summaries, prior_dict):\n", " # we will store our resulting probabilities in this dictionary:\n", " resulting_probabilities = dict() \n", " \n", " \n", " for c, class_stats in class_summaries.items():\n", "\n", " prior_c= prior_dict[c]\n", " resulting_probabilities[c] = prior_c # put P(y) in the dictionary containing the result\n", "\n", " for i, feature_stat in enumerate(class_stats):\n", " mean, stdev = feature_stat\n", " resulting_probabilities[c] *= 0 #QUESTION <- complete this line, here the posterior for the test sample should be calculated\n", " \n", " return resulting_probabilities\n", " \n", " " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "probs = calculate_probability(test_point, class_summaries,prior_dict)\n", "print(probs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output *'probs'* is a dictionary containing quantities proportional to the posterior porobability of each class given the input data sample. Our decision rule will be simply to pick the class with the highest unnormalized probability." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 5 (NB)**: describe why the Naive Bayes classifier is called 'naive'? State the line from the *'calculate_probability'* function that implements the 'naive' assumption." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**(Bonus) Question 6 (NB)**: why the output probabilities of our Naive Bayes classifier don't sum to 1?\n", "\n", "Hint: have a look on the formula we used for the posterior calculation defined at the beginning of the section." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's run our Naive Bayes mode on all the 3 datasets:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def train_NB(X,Y):\n", " #class indices\n", " i_c0 = (Y == 0)\n", " i_c1 = (Y == 1)\n", "\n", " X_0 = X[i_c0] #data of class 0\n", " X_1 = X[i_c1] #data of class 1\n", " \n", " prior_dict = calculate_prios(X_0, X_1, Y)\n", " \n", " #Calculate statistics for features per class\n", " class_summaries = calculate_MLE(X_0,X_1)\n", " \n", " return prior_dict, class_summaries\n", "\n", "\n", "def test_NB(X, class_summaries, prior_dict):\n", " result = []\n", " for x_test in X:\n", " probs = calculate_probability(x_test, class_summaries, prior_dict)\n", " result.append(list(probs.values()))\n", " return np.array(result)\n", " \n", " " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i, (ds_train, ds_test) in enumerate(zip(datasets_train, datasets_test)):\n", " ds_train = datasets_train[i]\n", " ds_test = datasets_test[i]\n", "\n", " X = ds_train[0]\n", " Y = ds_train[1]\n", " \n", " \n", " X_test = ds_test[0]\n", " Y_test = ds_test[1]\n", " # 1) train model\n", " prior_dict, class_summaries = train_NB(X,Y)\n", " \n", " # 2) Predict \n", " # a) for test data\n", " posterior = test_NB(X_test,class_summaries, prior_dict )\n", " pred_test = np.argmax(posterior,1)\n", "\n", " # b) for training data\n", " posterior = test_NB(X, class_summaries, prior_dict )\n", " pred_train = np.argmax(posterior,1)\n", "\n", " \n", " #4) plot predictions\n", " plot_predictions(i,X,Y, X_test, Y_test, pred_train, pred_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question 7 (NB):** for which datasets did the Naive Bayes classifier perform well? Describe in your own words why this is the case. Why did the Naive Bayes classifier work so badly for the first dataset?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course sklearn package offers an implementation of the Naive Bayes classifier. Let's try it out." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.naive_bayes import GaussianNB\n", "\n", "for i, (ds_train, ds_test) in enumerate(zip(datasets_train, datasets_test)):\n", " ds_train = datasets_train[i]\n", " ds_test = datasets_test[i]\n", "\n", " X = ds_train[0]\n", " Y = ds_train[1]\n", " \n", " \n", " X_test = ds_test[0]\n", " Y_test = ds_test[1]\n", " # 1) train weights\n", " model = GaussianNB()\n", " #let's fit the model (learn its parameters)\n", " model.fit(X, Y)\n", "\n", " # 2) Calculate the discriminant function and predict \n", " # a) for test data\n", " pred_test = model.predict(X_test)\n", "\n", " # b) for training data\n", " y_x = np.dot(W.T, X_b.T)\n", " pred_train = model.predict(X)\n", "\n", " \n", " # 4) plot predictions\n", " plot_predictions(i,X,Y, X_test, Y_test, pred_train, pred_test, plot_nb=model )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2. Example of Naive Bayes for text classification." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have understood how Naive Bayes works, let's walk through a text classification example taken from the exercises notebook accompanying the 'Python Data Science Handbook' [book](https://www.oreilly.com/library/view/python-data-science/9781491912126/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now apply the Naive Bayes classifier to the text classification task. As discussed in class, in text classification the features are related to word counts or frequencies within the documents to be classified.\n", "\n", "Let's download the 20 Newsgroups corpus and take a look at some target names." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import fetch_20newsgroups\n", "\n", "data = fetch_20newsgroups()\n", "data.target_names" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For simplicity here, we will select just a few of these categories, and download the training and testing set:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "categories = ['talk.religion.misc', 'soc.religion.christian',\n", " 'sci.space', 'comp.graphics']\n", "train = fetch_20newsgroups(subset='train', categories=categories)\n", "test = fetch_20newsgroups(subset='test', categories=categories)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is an example exntry from the data:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(train.data[5])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's convert the content of each string into a vector of numbers using the [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) vectorizer and create a pipeline that attaches it to a multinomial Naive Bayes classifier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.naive_bayes import MultinomialNB\n", "from sklearn.pipeline import make_pipeline\n", "\n", "model = make_pipeline(TfidfVectorizer(), MultinomialNB())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now fit the model on the training data and predict the labels for the unseen test data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model.fit(train.data, train.target)\n", "labels = model.predict(test.data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's evaluate the performance of our learned classifier using a [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import seaborn as sns\n", "from sklearn.metrics import confusion_matrix\n", "mat = confusion_matrix(test.target, labels)\n", "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False,\n", " xticklabels=train.target_names, yticklabels=train.target_names)\n", "plt.xlabel('true label')\n", "plt.ylabel('predicted label');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, this simple classifier is able to relatively well separate the space talk from the computer talk, but it gets confused between talks about religion and Christianity." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The very cool thing here is that we now have the tools to determine the category for *any* string, using the ``predict()`` method of this pipeline.\n", "Here's a quick utility function that will return the prediction for a single string:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def predict_category(s, train=train, model=model):\n", " pred = model.predict([s])\n", " return train.target_names[pred[0]]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predict_category('sending a payload to the ISS')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predict_category('discussing islam vs atheism')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predict_category('determining the screen resolution')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remember that this is nothing more sophisticated than a simple probability model for the (weighted) frequency of each word in the string; nevertheless, the result is striking.\n", "Even a very naive algorithm, when used carefully and trained on a large set of high-dimensional data, can be surprisingly effective." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 4 }