{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Generative Classification"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Preliminaries\n",
"\n",
"- Goal \n",
" - Introduction to linear generative classification with multinomial-Gaussian generative model\n",
" \n",
"- Materials \n",
" - Mandatory\n",
" - These lecture notes\n",
" - Optional\n",
" - Bishop pp. 196-202 "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Example Problem: an apple or a peach?\n",
"\n",
"You're given numerical values for the skin features roughness and color for 200 pieces of fruit, where for each piece of fruit you also know if it is an apple or a peach. Now you receive the roughness and color values for a new piece of fruit but you don't get its class label (apple or peach). What is the probability that the new piece is an apple? "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkEAAAG2CAYAAAB4e1KRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xt4FNX9P/D3JpAEhAS5CIkJBB6RcCliCGIQJKCgIChSi1SgYhXRr3gplVotISGJRavWS1tpiApeiy23n5dKtUAQihcIIhQQLWKJZi0XIQGUhCTn98cwu7Ob2ftcd96v58mz7Oxk5sxuwnzyOZ9zjksIIUBERETkMAlmN4CIiIjIDAyCiIiIyJEYBBEREZEjMQgiIiIiR2IQRERERI7EIIiIiIgciUEQERERORKDICIiInIkBkFERETkSAyCiIiIyJFsGwQtWrQILpcL9913n9lNISIiIhuyZRC0detWLFmyBAMHDjS7KURERGRTtguCTp48iWnTpqGiogLnnnuu2c0hIiIim2pldgMiddddd+Gaa67BlVdeibKysqD71tfXo76+3vO8ubkZ3333HTp16gSXy6V3U4mIiEgDQgicOHECGRkZSEjQLn9jqyBo+fLl2L59O7Zu3RrW/osWLcLChQt1bhUREREZobq6GpmZmZodzzZBUHV1Ne699168++67SElJCet7HnzwQcydO9fzvLa2Ft27d0d1dTVSU1P1aioRERFpqK6uDllZWWjfvr2mx3UJIYSmR9TJmjVrcP311yMxMdGzrampCS6XCwkJCaivr/d5TU1dXR3S0tJQW1vLIIiIiMgm9Lp/2yYTdMUVV2DXrl0+22655Rbk5OTggQceCBkAERERESnZJghq3749BgwY4LPtnHPOQadOnVpsJyIiIgrFdkPkiYiIiLRgm0yQmsrKSs2PKYRAY2MjmpqaND82USitW7dm1y4RkUFsHQRpraGhAW63G99//73ZTSGHcrlcyMzMRLt27cxuChFR3GMQdFZzczMOHDiAxMREZGRkICkpiRMqkqGEEDh8+DC+/vpr9O7dmxkhIiKdMQg6q6GhAc3NzcjKykLbtm3Nbg45VJcuXfDVV1/hzJkzDIKIiHTGwmg/Wk7HTRQpZh+JiIzDOz4RERE5EoMgIiIiciQGQRS24uJiDBo0yOxmEBERaYJBkF7cbqC4WHo0yJYtW5CYmIirr77asHMSERHZFYMgvbjdwMKFhgZBL7zwAu6++25s3rwZBw8eNOy8REREdsQgKE6cOnUKf/3rX3HnnXdiwoQJWLZsmee1yspKuFwuvP3227jooouQkpKCoUOH+ixIu2zZMnTo0AFr1qzBhRdeiJSUFIwZMwbV1dVBz7t06VL07dsXKSkpyMnJwbPPPut5raGhAXPmzEF6ejpSUlKQnZ2NRYsWaX7tRERE0WAQpCW3G9i+3fsF+D7XMSv0+uuvo0+fPujTpw+mT5+OpUuXQgjhs8+8efPw+OOPY+vWrTjvvPNw7bXX4syZM57Xv//+ezz88MN48cUX8a9//Qt1dXWYOnVqwHNWVFTgN7/5DR5++GHs3bsXv/3tb1FYWIgXX3wRAPDMM8/gjTfewF//+lfs27cPr7zyCrKzs3W5fiIiokhxskQtlZdLXWBKs2Z5/11UJNUJ6eD555/H9OnTAQBXX301Tp48iXXr1uHKK69UnL4IY8aMAQC8+OKLyMzMxOrVqzFlyhQAwJkzZ/DHP/4RQ4cO9ezTt29ffPzxx7jkkktanLO0tBRPPPEEJk+eDADo2bMn9uzZg/Lyctx88804ePAgevfujeHDh8PlcqFHjx66XDsREVE0mAnS0uzZQFWV9FVRIW2rqPBumz1bl9Pu27cPH3/8sSdr06pVK9x444144YUXfPbLz8/3/Ltjx47o06cP9u7d69nWqlUr5OXleZ7n5OSgQ4cOPvvIDh8+jOrqatx6661o166d56usrAz79+8HAMycORM7duxAnz59cM899+Ddd9/V9LqJiIhiwUyQltLTpS+l3FzpS0fPP/88Ghsbcf7553u2CSHQunVrHDt2LOj3+s9QrDZjsdq25uZmAFKXmJw5ksnLPeTm5uLAgQN455138M9//hNTpkzBlVdeiRUrVoR3YURERDpiEGRzjY2NeOmll/DEE09g7NixPq/9+Mc/xquvvooBAwYAAD788EN0794dAHDs2DF8/vnnyMnJ8TnWtm3bPF1f+/btw/Hjx332kXXt2hXnn38+vvzyS0ybNi1g+1JTU3HjjTfixhtvxA033ICrr74a3333HTp27BjztRMREcWCQZBe0tOlGiD/zJDG3nrrLRw7dgy33nor0tLSfF674YYb8Pzzz+PJJ58EAJSUlKBTp07o2rUrfvOb36Bz586YNGmSZ//WrVvj7rvvxjPPPIPWrVtjzpw5uPTSS1XrgQBp8sR77rkHqampGDduHOrr67Ft2zYcO3YMc+fOxZNPPon09HQMGjQICQkJ+Nvf/oZu3bqhQ4cO+r0hREREYWJNkF7S06UiaJ2DoOeffx5XXnlliwAIkDJBO3bswPazI9UeeeQR3HvvvRg8eDDcbjfeeOMNJCUlefZv27YtHnjgAdx0003Iz89HmzZtsHz58oDnvu222/Dcc89h2bJl+NGPfoSRI0di2bJl6NmzJwCgXbt2ePTRR5GXl4chQ4bgq6++wt///ncuUktERJbgEv7jqONYXV0d0tLSUFtbi9TUVJ/XTp8+jQMHDqBnz55ISUkxqYX6qKysxKhRo3Ds2LGAWZhly5bhvvvuw/Hjxw1uHSnF888hEVG0gt2/Y8E/yYmIiMiRGAQRERGRIzEIcoCCggIIIYIWJM+cOZNdYURE5CgMgoiIiMiRGAQRERGRIzEIIiIiIkdiEERERESOxCCIiIiIHIlBEBERETkSgyCKWXFxMQYNGmR2M4jIZoqLgdJS9ddKS6XXzTwexT8GQRoy6xdw5syZcLlccLlcaN26NXr16oX7778fp06d0ueEREQaSEwEFixo+f9maam0PTHR3ONR/OMq8hqSfwEBoLDQu13+BSwp0e/cV199NZYuXYozZ85g06ZNuO2223Dq1CksXrxYv5MSEcVA/n9S+f+m8v9L5f+jZhyP4h8zQRoqLJR+0ZR/iRj1C5icnIxu3bohKysLN910E6ZNm4Y1a9YAAPbs2YPx48ejXbt26Nq1K2bMmIEjR454vnft2rUYPnw4OnTogE6dOmHChAnYv3+/z/G//vprTJ06FR07dsQ555yDvLw8fPTRRz77vPzyy8jOzkZaWhqmTp2KEydOeF4TQuB3v/sdevXqhTZt2uCiiy7CihUrPK8fO3YM06ZNQ5cuXdCmTRv07t0bS5cu1eOtIiILUf6/mZwc+/+XWh+P4huDII1Z5RewTZs2OHPmDNxuN0aOHIlBgwZh27ZtWLt2Lf73v/9hypQpnn1PnTqFuXPnYuvWrVi3bh0SEhJw/fXXo7m5GQBw8uRJjBw5EjU1NXjjjTfw6aef4le/+pXndQDYv38/1qxZg7feegtvvfUWNm7ciEceecTz+vz587F06VIsXrwYu3fvxi9+8QtMnz4dGzduBAAUFhZiz549eOedd7B3714sXrwYnTt3NujdIiIzFRYCSUlAQ4P0GOv/l1ofj+KYcJDa2loBQNTW1rZ47YcffhB79uwRP/zwgybnSkoSApAe9XbzzTeL6667zvP8o48+Ep06dRJTpkwRhYWFYuzYsT77V1dXCwBi3759qsc7dOiQACB27dolhBCivLxctG/fXhw9elR1/6KiItG2bVtRV1fn2TZv3jwxdOhQIYQQJ0+eFCkpKWLLli0+33frrbeKn/70p0IIISZOnChuueWWCK88/mj9c0hkByUl3v8vAem5lY5H5gt2/44FM0E6KC31/gXS0BC4WFpLb731Ftq1a4eUlBTk5+fj8ssvxx/+8AdUVVVhw4YNaNeunecrJycHADxdXvv378dNN92EXr16ITU1FT179gQAHDx4EACwY8cOXHzxxejYsWPA82dnZ6N9+/ae5+np6Th06BAAqTvu9OnTGDNmjE87XnrpJU8b7rzzTixfvhyDBg3Cr371K2zZskX7N4mILEdZMlBf37KkwOzjUXxjYbTG/GuA5OeAvinZUaNGYfHixWjdujUyMjLQunVrAEBzczMmTpyIRx99tMX3pKenAwAmTpyIrKwsVFRUICMjA83NzRgwYAAaGhoASF1rocjnk7lcLk93mfz49ttv4/zzz/fZLzk5GQAwbtw4/Pe//8Xbb7+Nf/7zn7jiiitw11134fHHH4/kbSCKO8XF0qALtf8/SkuBpib7Dv1Wq5lUK24263gU/xgEacjMX8BzzjkHF1xwQYvtubm5WLlyJbKzs9GqVcuP++jRo9i7dy/Ky8sxYsQIAMDmzZt99hk4cCCee+45fPfdd0GzQYH069cPycnJOHjwIEaOHBlwvy5dumDmzJmYOXMmRowYgXnz5jEIIsczc9Sp3pqa1Gsm5edNTeYej+IfgyANWfEX8K677kJFRQV++tOfYt68eejcuTP+85//YPny5aioqMC5556LTp06YcmSJUhPT8fBgwfx61//2ucYP/3pT/Hb3/4WkyZNwqJFi5Ceno5PPvkEGRkZyM/PD9mG9u3b4/7778cvfvELNDc3Y/jw4airq8OWLVvQrl073HzzzViwYAEGDx6M/v37o76+Hm+99Rb69u2r19tCZBvxPOw7WAYrmuvS+ngU/xgEaciKv4AZGRn417/+hQceeABXXXUV6uvr0aNHD1x99dVISEiAy+XC8uXLcc8992DAgAHo06cPnnnmGRQUFHiOkZSUhHfffRe//OUvMX78eDQ2NqJfv37405/+FHY7SktLcd5552HRokX48ssv0aFDB+Tm5uKhhx7ynOPBBx/EV199hTZt2mDEiBFYvny51m8HkS0pA6GyMqnW0CoBkF276+zabtKYpmXWFmfk6DCiaPDnkIIxctRpuOSRWP4jsAJttwq7ttup9BodxkwQEZENqI06tUImKNbuOrMyMvHczUjhYxBERGRxZo06DVcs3XVmFn5buZuRDKJpXsni2B1GVsefQ/Jnp26baLvr/K/F6GuzYjcj+WJ3GBGRA1lx1KmaWLrrzMzIWLWbkYzBGaOJiCysuDjwTbmw0BojmLSYpVm53lewGiEtr5ezSxMzQUREFDWtJomVMzKJiVJ264orgHXr1M9jpXaTvTEIIiKiqGnRXecfkFxxBbB+vTcQ0mPUll26GUlfLiGEMLsRRqmrq0NaWhpqa2uRmprq89rp06dx4MAB9OzZEykpKSa1kJyOP4fkNIECHDkQkjNDHLXlbMHu37FgTRAREZkmUICzbp03AEpKYgBE+mAQpIOmpiZUVlbiL3/5CyorK9EUR3nV4uJiDBo0yOxmWArfE6LoBSr8lidKVI7a0uv8gY6tdSE2WQ+DII2tWrUK2dnZGDVqFG666SaMGjUK2dnZWLVqle7n3rJlCxITE3H11Vfrfi6KzCOPPIL+/fujbdu2uPDCC/Haa6+Z3STSGW+u0TNy1JY8WaP/seU2JCZqf06yDgZBGlq1ahVuuOEGfP311z7bv/nmG9xwww26B0IvvPAC7r77bmzevBkHDx7U9VxGEkKgsbHR7GbEZNOmTXjyySfx73//G9OnT8fPfvYzfPnll2Y3i3Rk95urWUFcoFFbegVCasfm8hnOwSBII01NTbj33nuhVmcub7vvvvt06xo7deoU/vrXv+LOO+/EhAkTsGzZMp/XKysr4XK58Pbbb+Oiiy5CSkoKhg4dil27dnn2WbZsGTp06IA1a9bgwgsvREpKCsaMGYPq6uqg5166dCn69u2LlJQU5OTk4Nlnnw26f319Pe655x6cd955SElJwfDhw7F169YWbf3HP/6BvLw8JCcnY9OmTS2O89VXX8HlcmH58uUYNmwYUlJS0L9/f1RWVvrst2fPHowfPx7t2rVD165dMWPGDBw5csTz+tq1azF8+HB06NABnTp1woQJE7B//36fY3z99deYOnUqOnbsiHPOOQd5eXn46KOPfPZ5+eWXkZ2djbS0NEydOhUnTpzwvPb2229j7Nix6NWrF+bMmYOmpibU1NQEfZ/I3ux+czUriAs2aqukRJ9RW8rPKjnZPp8RaUDT+actTs9lMzZs2CAAhPzasGFDjFeh7vnnnxd5eXlCCCHefPNNkZ2dLZqbm1u0r2/fvuLdd98VO3fuFBMmTBDZ2dmioaFBCCHE0qVLRevWrUVeXp7YsmWL2LZtm7jkkkvEsGHDPMcpKioSF110kef5kiVLRHp6uli5cqX48ssvxcqVK0XHjh3FsmXLArb1nnvuERkZGeLvf/+72L17t7j55pvFueeeK44ePerT1oEDB4p3331X/Oc//xFHjhxpcZwDBw4IACIzM1OsWLFC7NmzR9x2222iffv2nv1rampE586dxYMPPij27t0rtm/fLsaMGSNGjRrlOc6KFSvEypUrxeeffy4++eQTMXHiRPGjH/1INDU1CSGEOHHihOjVq5cYMWKE2LRpk/jiiy/E66+/LrZs2eJ5T9q1aycmT54sdu3aJd5//33RrVs38dBDD7Voc3Nzs5g5c6YYMGCAqK+vb/E6l82IP/ISEPLSDFovBVFUFPiYJSXS69EyezkLo3H5DOvSa9kMBkFnxXrzee2118IKgl577bVYL0PVsGHDxFNPPSWEEOLMmTOic+fO4r333vO8LgcWy5cv92w7evSoaNOmjXj99deFEFIQBEB8+OGHnn327t0rAIiPPvpICNEyCMrKympxTaWlpSI/P1+1nSdPnhStW7cWr776qmdbQ0ODyMjIEL/73e982rpmzZqg1ywHQY888ohn25kzZ0RmZqZ49NFHhRBCFBYWirFjx/p8X3V1tQAg9u3bp3rcQ4cOCQBi165dQgghysvLRfv27T1Bmr+ioiLRtm1bUVdX59k2b948MXTo0Bb7/vznPxcXXnih+Prrr1WPxSAoPul5c9V7bTG9gzircMp12pVeQRC7wzSSnp6u6X6R2LdvHz7++GNMnToVANCqVSvceOONeOGFF1rsm5+f7/l3x44d0adPH+zdu9ezrVWrVsjLy/M8z8nJQYcOHXz2kR0+fBjV1dW49dZb0a5dO89XWVlZi+4k2f79+3HmzBlcdtllnm2tW7fGJZdc0uIcynYEo7wmuf3ysaqqqrBhwwaf9uXk5HjaIj/edNNN6NWrF1JTU9GzZ08A8NRV7dixAxdffDE6duwYsA3Z2dlo376953l6ejoOHTrks8/OnTvxwgsv4I033sD5558f1rWR/amtTaUlvbvdlMtZxOtQdS6f4VycMVojI0aMQGZmJr755hvVuiCXy4XMzEyMGDFC83M///zzaGxs9LmxCiHQunVrHDt2DOeee27Q73e5XEGfB9rW3NwMAKioqMDQoUN9XksMUDAgvzf+xxNCtNh2zjnnBG13MPKxmpubMXHiRDz66KMt9pED0okTJyIrKwsVFRXIyMhAc3MzBgwYgIaGBgBAmzZtQp6vdevWLc4vvz+yAwcOAAD69OkT+QWRLfkHI/JzQNtgQs8FSON9gVEun+FszARpJDExEU8//TSAwEHFU089FTA4iFZjYyNeeuklPPHEE9ixY4fn69NPP0WPHj3w6quv+uz/4Ycfev597NgxfP75557MiHy8bdu2eZ7v27cPx48f99lH1rVrV5x//vn48ssvccEFF/h8ydkUfxdccAGSkpKwefNmz7YzZ85g27Zt6Nu3b1TvgfKaGhsbUVVV5Wlvbm4udu/ejezs7BZtPOecc3D06FHs3bsX8+fPxxVXXIG+ffvi2LFjPscfOHAgduzYge+++y6q9slGjhzpUwBO8c2MUU5aZ2yckCExoxCbLETTzjWL07MmSLZy5UqRmZnpUweUlZUlVq5cGdNxA1m9erVISkoSx48fb/HaQw89JAYNGiSE8NbZ9O/fX/zzn/8Uu3btEtdee63o3r27p0BXLoy+5JJLxIcffiiqqqpEfn6+uPTSSz3H9K8JqqioEG3atBFPPfWU2Ldvn9i5c6d44YUXxBNPPBGwzffee6/IyMgQ77zzjk9h9HfffefT1mPHjgW9drkmqHv37mLVqlVi79694vbbbxft2rUThw8fFkII8c0334guXbqIG264QXz00Udi//794h//+Ie45ZZbRGNjo2hqahKdOnUS06dPF1988YVYt26dGDJkiAAgVq9eLYQQor6+Xlx44YVixIgRYvPmzWL//v1ixYoVPoXRyvdECCGefPJJ0aNHD59tq1atEn369Al6TawJsqZoio/1LFgOdEwta1r0rjUiigQLozVgRBAkhBCNjY1iw4YN4rXXXhMbNmwQjY2NMR8zkAkTJojx48ervlZVVSUAiKqqKk9g8eabb4r+/fuLpKQkMWTIELFjxw7P/kuXLhVpaWli5cqVolevXiIpKUmMHj1afPXVV5591G74r776qhg0aJBISkoS5557rrj88svFqlWrArb5hx9+EHfffbfo3LmzSE5OFpdddpn4+OOPPa9HGgS99tprYujQoSIpKUn07dtXrFu3zme/zz//XFx//fWiQ4cOok2bNiInJ0fcd999ntFz7733nujbt69ITk4WAwcOFJWVlT5BkBBCfPXVV+LHP/6xSE1NFW3bthV5eXkBi8WFUA+C5MLzYBgEWZPVAwI9RnEZHcQRBcMgSANGBUFWFE5gIQdBdiEHQZ988onZTdFMvP8c2plVh4tbPUAj0oJeQRALo4mIwqBn8XEsgtW0yK8bpbhYmkQx2FpgXC6ErIRBEBFRmAoLvQGQVYaLBwsqjG6fPMu0/7mVBdZEVsLRYQ5RUFAAIQQ6dOgQcJ+ZM2fi+PHjBrYqNtnZ2RBCcAV3Mozec/7Ynd2XCiHnsVUQtHjxYgwcOBCpqalITU1Ffn4+3nnnHbObRUQO4ITh4lrgOlxkJ7YKgjIzM/HII49g27Zt2LZtG0aPHo3rrrsOu3fv1uwcQmWiQyKj8OfPmoye88funDDLNMUHW9UETZw40ef5ww8/jMWLF+PDDz9E//79Yzq2POPv999/H9YMwUR6kGep1npSTYqNlYqPjRZNsXO8zzJN8cNWQZBSU1MT/va3v+HUqVM+a0cp1dfXo76+3vO8rq4u4PESExPRoUMHz3pPbdu2VV0qgkgvzc3NOHz4MNq2bYtWrWz7qxmXrFR8bLRIi52NWiqESAu2+592165dyM/Px+nTp9GuXTusXr0a/fr1U9130aJFWLhwYdjH7tatGwC0WPiSyCgJCQno3r07A3CyDLV1tAIVO3MdLrIbl7BZEUJDQwMOHjyI48ePY+XKlXjuueewceNG1UBILROUlZWF2tpapKamBjxHU1MTzpw5o0v7iYJJSkpCQoKtSvXIIeQAR+7iUuseVHad+XejKbvOrDpnEOc5sq66ujqkpaWFvH9HynaZoKSkJFxwwQUAgLy8PGzduhVPP/00ysvLW+ybnJyM5OTkiM+RmJjImgwiIoVw5khSBgj+3WjKYMiqcwZxniPnsV0Q5E8I4ZPtISIi7UVa7BxJN5pV2LHNFBtbBUEPPfQQxo0bh6ysLJw4cQLLly9HZWUl1q5da3bTiIjiVrTFzlZdaiQYO7aZomermqBbb70V69atg9vtRlpaGgYOHIgHHngAY8aMCev79epTJCKKV5EUQQeSnOzNIgVL3FupJifcNpMxWBME4Pnnnze7CUREjhLrHEmRdKNZpSaH8xw5h62CICIiMlYscyRF2o0WrCanoCD4ebTKEnGeI2dhEERERJqLds6gQDU5gb5PyywR5zlyHgZBRESkuVi60YINx9dr5FZxMbBxY+BjFRTE9/IoTsUgiIiINBdrN5paTY6eI7cSE4HKSmD06JZt4RD5+GWr0WGx4ugwIiJrkkeGAeo1OQUFwIYN0ut6jdwKVA/EAMh8HB1GRERxSzkyTC3oqKyUghJAv5FbnCPIebhIERERma6wsOUIMGUmpqQEWL/e+7y+XnpcsMAbHGnVDjnACrQ8CMUPBkFERGQJGzZ4A5vk5JZdUZWVLUduaR0IqdUjUfxidxgRUQSsNKtxPAo0MizWSRv9qX2OcuZp9GhgxIjAkzdS/GAQREQUAavMahyvAo0Mi2W0mRr/z1EZAK1fL3XNcY6g+McgiIgoAvG60rgVMlxGztbs/zk2NXkDILXJEjlHUJwSDlJbWysAiNraWrObQkQ2V1IiBCBEUpL0WFJidotiI1+P/3UE2h4v54+3zzFe6XX/5jxBRERRireVxs2cJ0fLTFSkx4q3zzEe6XX/5ugwIqIoxOMoIuVoK7XRWbpxu1GMYhTe5g7Yrki64uR6H//PRA7q5EkZ5W3x9jlSBDTNK1kcu8OISAv+XTRGdRkZRe4aSkoy6IRVVdIJq6o0O2Q4n1G8f47xRK/7NwujiRzOCgWxdhLvK40HGp1lN6Fmf473z5HCwyCIyOE45DsyWs9XYyV6jM4KGGS73SgtEWhqdqF4yHZp2/bt3tfT06WvGARbjT6eP0eKgKZ5JYtjdxiROnYLkF6jswIet2CdtB3zpR38v4qKojuhyrk58sv+2B1GRLrhwpH6sFNXo16ZkYDzKlWORskdNSicdT2wvQcwaxZQUQHk5ko7xpgFMnLOIbIvBkFEBCB41wFJIg1q7NTVqPWMzGrf3zLIzgCQ4d0xN9cbBEVB/nzkc/kHdQUFDITIF4fIExEADhUORyRDrwH1BT7jYXbpaBixOrv8+fjP+qxcEqOkBFi3LvDPd2mpdbJzZABNO9csjjVBROpYExS+aN4r1qaEeA9qaqQaoJqagN9fVBT4fSsp8ZYQyecZPdr3ufLzKigwd3Zsipxe928GQUQOZ/ZyCXYUTVBj+Nw7FlFUJAUkaoHj6NHh1z/7Bzdq2+VjyedLTAwcsAYKZv2PrzyPBrXaFCUGQRpgEETUUrh/YZOvSIIaJ2eC5IDEP7gItD2SYykDF//3VQ6AEhN991WbLFH+XNSOE+h7yVgMgjTAIIiItBBJUOP0rkatMkEy/yyPWuAiH18ZCAV6z/1AsxJaAAAgAElEQVSDWad/XlbFIEgDDIKIKFaR3CT17Gq0WwZPy2yYHNTIX8GWwvDPCIXTJidn7qyKQZAGGAQRUSwiDWrUAhV5m1qgEknwYmYtl/K6/K/Rv0hZeT1a1EXJ1yd/KYObQAGqMmsUal/5uVNruKyKQZAGGAQRUSy0yL5oGbyY1XUTqMA4nMLjWLIr/jVA/sGN8vPxP7+y2yzUZxCosNp/Xytl2+yWGYwUgyANMAgiIi3EesPRMngxq+tGLeAJFQDFcr2BiqDVCqxDBTkFBYHPrRYshepyswIzM4NGYBCkAQZBRKQFLW44WgYvZnXd+F9DsPqaWG/OagXWMv/t0Qapam0KFeBZiVmZQSPodf92CSGE0RM0mqWurg5paWmora1Famqq2c0hIhsLtDZVJDNBJyd7Z1Cur4+tHfJszEbPRK28BqDl9Wi5flqwY40aJT2OHNlyH/k8iYnBzxfo+PJ7nJAANDdbe7Zvs38e9KLb/VvTkMrimAkiIi3Fks3RskZGr7/8Q2VU5JmXg2WCjBKouyzQ9kjZqVDaTm0NF7vDNMAgiIi05n/DCacrRssaGT1rQEKdI9yaoEC0Lub1D3j8H6N9T8yqu4qGndoaCQZBGmAQRERakm84ylFE/gXDamtaaVUjY8RooEABW6Sjw8I5dqjtkbQ3nIkSIzmeHeps7NTWSDEI0gCDICJtxfuw3GACZR1CZUb0es/0/Cz8swvK0VWRzBMU7Nha3riV3XOxdAvZacSVndoaDQZBGmAQRKSteP+PN5BAN27/gMjIbolQK6MXFMR2fD3rTLTswtEyE2SnIN9ObY0GgyANMAgi0l48p+ADUbvhqHWNhRM4aHXz8s8+BdoWzfmNqDPRcjZprWuCyHwMgjTAIIhIH/FajBkp5Y083Pck2mxasEAMECIhIXQAFM75A4220vIz1uLnR7W9NTWipGAdA6E4wCBIAwyCiPQTj8NyI+F/I48kcIgmmxZoH2VXXLg3/VDde3p2d2qVSZSDQp/gsKpKCEAU5NV56phiWa+NzMMgSAMMgoj04fRMkPLGHagLKtxAKJL3MFAAEWkQFOj8eteZ6F5TdjYIKrnjG0fWrsUTBkEaYBBEpD0n1gQp+V+vcpV4tUAoWOAQTTZNLQMVKCMVitHZPF2CrJoaKfipqhKiokK6oIoKbyA0r85zfCf9nNodgyANMAgi0pZTR4cpaV3YHE02zT8AClUYHW5hty0VFbVMh539KsF8zTKW8T4ay2oYBGmAQRCRtngj0EYs2TR5X5fLW8ej9rpyiHyoGiBbFxEHyATJ25KSmjXJdvEPAGMxCNIAgyAisppYbqbKfYKtsh6oIDhY4BPpzTyagFj3IPpsTZCoqvIcU8vaNad3BRuJQZAGGAQRkdVEGwhokYkI1QUWTXdeJO0xqjBaVFWFFbBE81k4fVCAURgEaYBBEBHFC62yKFoWQ0eTGdE1m1JTI0RRkSiZVxdWsBVtUOb06SGMwCBIAwyCiIi89MhiRHNMvbMpkQSMkQZlzAQZQ6/7t0sIIeAQdXV1SEtLQ21tLVJTU81uDhGRYYqLgcREoLBQel5aCixYAJSUSM/XrwcqK6Xn8j7RSk4GGhqApCSgvl6/79GL/N4kJUltCvSeKN/DwsKWz0k7et2/W2l2JCIisqzEROkGLVMGQPK/R4/27hPtTby01BvMNDRIz0MdK5rv0VNhIVBW5m1TOAGQ/H1A7O8hGYdBEBGRhvwzLkqlpUBTk7SP0ZQ36IKClgGQsr1NTdGdI1BmRHl+Lb5Hb+EEZU1N6hkf+Xm07yEZTNPONYtjTRAR6c3q88foVcNiydFhUeCwd2vS6/7NTBARkYbUukSsVCsSTldPNKLJjFgtm8IuLudhYTQRkQ7CLa4Nh5ZdbFq2K95YtSuT9Lt/J2h2JCIi8igs9AYasWZc5KLm0lLf7XJAk5gY3nGUmY76eulR7bhGKi4OfP7SUmODjuLiwJ9TYSEDoHjEIIiILMtKN8hIqRXXRquwsGXAEmkXW6CuHrMDIa0CPKJosCaIiCxLOaxbeaP3n+PGavQY8aSsTZFreiLpyrJa/Y3/+a1aQ0VxTtMya4vj6DAi+7HbaB29RzzF6xINnHmZguHoMCJypFgzIEZrapLm4fEnt3f9+ugLbK02qaCW9Bq1Fs9YyB071gQRkeVpWWSst+Ji78zLanU2lZXR1blYsahZS1rWUDkF66k0oGleyeLYHUZkT3bsKtGyG8+KkwpqyW5dnlbilPeOq8hrgEEQkf3Y+T95rYK3SFZBj1hNjXSAmpoYDhK9eA/wjGDHPxIixVXkNcDJEonsJdAoITNGD0Vbf2Gl1dFVbd8ODB4MVFUBubmGn551Ldqw/M9ZjDhZIhE5TrBh3SUl+gzrDjQ3kVx/MWqU7/Zg9Rd2qHMpLk9HKearvmbEXEycoDB2dvg5sypbBUGLFi3CkCFD0L59e5x33nmYNGkS9u3bZ3aziCiOBCo2lVVWhjdhoaULmd1uKQO0fTsS//cNFqAUpSXNnm1wu1lcaxOW/jmzA00713R21VVXiaVLl4p///vfYseOHeKaa64R3bt3FydPngzr+1kTRGQvZtWLBKtDCqf+Qut2a14TVFQkNeTsVwnmS+3CfOl5wbq4rS2JJ06qp2JhtIpDhw4JAGLjxo1h7c8giMh+YiqMjqHoN1iwE2rCQq2DFs1vdjU1QlRVSV8VFVLgc91W6ZpaN8XdDTRe6VowbzEMglR88cUXAoDYtWuX6uunT58WtbW1nq/q6moGQUQ2FPXol6oq6RuqqqI6r1qwY9ZIHN1GySneo3idjZrsT68gyFY1QUpCCMydOxfDhw/HgAEDVPdZtGgR0tLSPF9ZWVkGt5KItGDGZIlqxaZm1l8oFztNTtZ+dFxpRTcW15LzaBpSGej//u//RI8ePUR1dXXAfZgJIooPEWVfVLp6REWFd1sYXWOBsi5WqL/QPFtTU9OiBigea0rI3tgdpjBnzhyRmZkpvvzyy4i+jzVBRPYTcTeQX9Fvi68QhRKBjl9QELwQ2oj6Cz264pxUXEv2xQVUIXWB3X333Vi9ejUqKyvRs2dPs5tE5Gh6T3SnNgRduaCq8rnH7NnAtddK/96+HZg1C6io8E4EmJ4e9JyB5ibasMF7Tf6M6p5Tvhfy81jPH2wuJvl1W3G7gfJy6ecgxGcd122g8GgaUunszjvvFGlpaaKyslK43W7P1/fffx/W9zMTRKQtvbMIMY9+ibEw2iqYrYlAuJ+5nsuFxMnPnZUwEwRg8eLFAICCggKf7UuXLsXMmTONbxCRw6llZbRc0iJYFsnKK8lrzY7ZGssvh+F2AwsXSllDZmscy1ZBkHDOMmdExoohfa8MhMrKpJFFRq7pFVR6OlBUZP2bXIj3347BoDzzNhB43TfNuN3SFyB1gSofAek91ftnwAptoMhpmleyOHaHEQWgQfqec8zEIE67T3Sb28hfuMXwMY4cDNo9W7BOFCFIO+Jp5kITsDuMiCxLbU4dq2YoyDhqWUK/agYfUXeThVsMX14udYEpzZrl/XdRUdCTB81uVY5GyR05wKxroyrIJ3MwCCJyKo3S93qNWop7Duk+KSz0BkBJScDo0Tp0k6m9V7m53gBEFuPIwdA1cBkAMoK3gaxF07ySxbE7jEghxvl0hNBp1JKeo3asRIP33w7U5jbStZss3K7FGLogQ87XFKfdm2bS6/7tEsI51cZ1dXVIS0tDbW0tUlNTzW4Okbn8MxFqfxWH+MtYlxFA27cDgwcDVVWW/Ctas2vW4P23ukBZQjnbs2CBtwtVs2L6cIv8Y/w5S072Zrfq66NsA4VNt/u3piGVxTETRBSAlf5ytVJbVOiS/bL4NUcjnPfJ1GL6GDKOZi2i62QsjCai+GWj+hi950bSQotslSIzUfpcuiFz9ISa22j9epOL6dPTo3oTWAMXZzQNqSyOmSCiAMyuw7FhfUw42YCwZ7zW+P1vkYU5m2kqueMbS2QuDBs6rzHO3G0eLqCqAQZBRBYlz9/yyivewCfCld9DiXkJDhWhunPMvGn6nKOqSpRgviVu1HYOJPT4GaLwsDuMiOJXuEOcY6D1DMbhzI1kZtdZ4W1uoEZgwYIMlJVchAbkouS6bSi8JgHYDtO6GO24BIjMjjN3UwiahlQWx0wQkQWpzeILSFkhjbJAMq26YSI9Touus3l1+nc/nu1iTMJp6dw4rV8Xo9ndqRT32B2mAQZBRBZkcD1QrCN7ou3O8ek6M2I0WE2NpwYoqVWj1LbrtmraxegRh6PbyFrYHUZE8SnGWXwj5T+DcaTdGNF057ToOqvoBr17T0qfS8eCP59t6zWfonTwaiz4f6XAYHbdEMkYBBGRuQyoB1KKdZ2zSOtCPDVA806gcOoXKK3ohgV/zgAwH4U6TQPQou5oO1CIMuCOO7FgQUbAtkbERtMaEAWSYHYDiIiMogwO6uulxwULpO16n6+w7RPA4MEo/PP5KEEhFqAUpbP+K81aPHiwNI+PRlpkq9LTgaIiFC5woaREo+Lj8nJv2+VFSGfN0uV6iPTCZTOIyDrcbuDxx6V/33+/ppmEQKOy9Byt5TNpoV/mpHTWf9E08XpvZslumRMHLPtB1qHX/ZvdYURkHenpwLRpUiZh2jRNb6JmDM326TrzCwqa8DUSz+8G5Ga0+L7SX51A08dVKP5LH+sGEgZ3YxLpgUEQETmC1eZ4SUSTVBuUoZKZeqw9SrAOcKdaNwgiigMMgojIfE4rsk1PR2FRIvD9CSxY0B6A30SKd9Sg8M9lAK43t53hSkgARo6UHolshEEQEZmvvBxYuNB3m1xsCwBFRfqv+Gmks4t3FgJAeynwKSsTaGhwSQHQ4L9L+9klEGxuBjZulB6JbISF0URkPocX2SYnnx22j3rUI0V9JysHgtu3S3VcVVWsCSJdsDCaiOKXEUW2breUcZo921IBlXfeIoGGhmSU3vGNlAmKcNJIn5FoKudoatI4hnJaFybFJXbgEpF53G7pzizfTPU+18KFxpwrTL7zFklz+Cz4cwZKq8ZLO8iBYG5uyIBCXiDWf84j+RyJiRo3nvMEURxgJoiIzCMHJtde673Jn53YL96zCGrzE3lXnT87o3QEx4tpxfposmQGL3dCpIeogqAffvgB3333Hc4//3yf7bt370b//v01aRgROdTZomFNWLjLJui8RSdOoOnjKyJumzIQktdHC2sSSLVgNBTOE0RxIOLC6BUrVuAXv/gFOnbsCCEEKioqMHToUABAbm4utiv/g7EYFkYTWYCRRdDFxS1HnSlZudg4Bp5C6yRpeZCQYi1sVvl+w2uUTNbU1IRNmzbB7XYjPT0dI0aMQKLmfZDOpdf9O+KaoLKyMmzfvh2ffvopXnjhBfz85z/Ha6+9BgBw0EAzIoqWkbUks2dLN+aqKinQAqRHedvs2dqdyyLUFohV5XZLwYv8Bfg+j6R2SqULU7VGye1G6aj1+tQomWjVqlXIzs7GqFGjcNNNN2HUqFHIzs7GqlWrzG4ahSIi1K9fP5/nR44cEZdffrlYuHChuPjiiyM9nKFqa2sFAFFbW2t2U4icq6ZGiKoq6auiQghAepS31dToc96qKulcVVX6HN8CSkqkSywpUX/uo6hIejHQV1GR9u254xvp+R3fxHxsq1i5cqVwuVwCgM+Xy+USLpdLrFy50uwmxgW97t8RB0EFBQXi008/9dlWX18vpk6dKhITEzVrmB4YBBFZjJGBSZwHQYECnoCBkEHBqHz+pKSz7cD8uPkMGhsbRWZmZosASBkIZWVlicbGRrObant63b/DLow+ceIE2rdvj5dffhmtWvl+W1JSEv7yl79gzpw5GuSmiIh0EOejziJeIDaSwuYY5lgqvM2NstKuaGhIQFKrJhQ2lgHbewRvh01s2rQJX3/9dcDXhRCorq7Gpk2bUFBQYFzDKGxhB0EjRozA2rVrkZmZGXCfyy67TJNGEZFDGBmYaDnqzIJ0XSA2mtFjZ5XetBcNZ9KRhHo0NCajFPNRGCdLorjDrJsKdz8yXtiF0Xl5eRg6dCg+++wzn+2ffPIJxo8fr3nDiMgkRk5gKAcmNs0ExA2dgtHSUmBB5WiU3FGD+qrdKLluGxagFKXXbY2L4vT0MN+vcPcj44UdBD333HP4+c9/juHDh2Pz5s34/PPPMWXKFOTl5SE5OVnPNhI5k5HBiP95LTazMulMLRiNcfSYz0SNizOA3FwULkhACQqx4P/lofTt8GbCtrIRI0YgMzMTLpdL9XWXy4WsrCyMGDHC4JZRuCIaIl9UVIRf/vKXGDNmDAYMGIAffvgBW7duxerVq/VqH5FzMRiRmBUMOl2MUxkErFFCGUruqGlZo2RDiYmJePrppwGgRSAkP3/qqac4X5CFhV0T5Ha7sWjRIjz33HPo168fPvvsM0ydOhW5nB2UyP4sPLOyTzBo1a6zHTuA++4DnnoKGDQo8H4WXcRVVYzLYqiW+Zztdiuc7QIsfvnhmjx5MlasWIF7773Xp0g6MzMTTz31FCZPnmxi6yiUsIOgXr16IScnB3/7299wzTXX4B//+AemTJmCr7/+Gg888ICebSRyDrOCkfLyljMrW614dckS6wYPu3cDGzdKj8GCoJ07pfc5P9+a16Gkx7IYcVqcPnnyZFx33XWcMdqGwg6Cli5diqlTp3qeX3XVVdiwYQMmTJiA//73v3j22Wd1aSCRo5gVjFhtMUy1YBAA9u71tseoNmmZvTlyxPeR4kZiYiKHwdtQ2EGQMgCS5ebmYsuWLRwdRqQVs4IRqy2GqRYMAsD06dLj7bcb1zUWaHj4jh1S5gcA3n3X9xEA+veXskLKgE4eXbt1K/D++8DkycDAgeZkhSIJ7oKMHnPKGmFOuU7H0WLGxe+++06Lw+iOM0aTrZg1w7EVZlauqRHi9tt1X9YhLIHej5Ejg7dv5Ehpv7lzg+83d64x1+FPo8854pmqbcop12lVps8YHcy5556rxWGIyAqsMLOyXDsye7bUBSZngIzqolNmb9av932Uz/3UU1Im6NgxqV07dwI/+xkwdqy0T//++rXPQuTMyIIF3uc+w+NjnajRIpxynY6jaUhlccwEka3U1EjZDr0WFLULOWNhZHYqksVFle175ZWWx5LX6Fq7Vohbb5X2u+Ya6XH+fGm7ngvHqrVFh/XCWqwRFqeZEadcp9VYZgFVO2MQRGQQLQM4ZdeYWhCkR7CoDBbmz/cGLGrBQqggSGbAqu2q16F8b3RugxwYJCXF2nBrc8p1Wole9++IJkskIgqLlhM9yl1jgbrozJhU8n//azmb8oUXAgkJgWdTnj0bKCuT/n3bbdJjRYW+y0f4vzezZ3vPV1GhaRtKS4GGBiApSXosLdWg/RbklOt0DE1DKotjJogsJ167vIwsrtbjXKEyJqGKogNlVNaulV7/wx+MeX+CvTcavm/+xcHxWizslOu0IksXRhNRlGJYndtyIp3oMZb5d/SeVFI5VcH69cC8ecBjjwGjR0vbEhKA5mbvecOdyqBLF+lRz8EkBk+4qVYcrFZEbHdOuU6nYRBERNqIdKLHWAJAvSeVVAsURo8OPm9SoHmV1IKSF18Epk0DDh+WXtMyAA73vdFoFGDANcIKAZw4gaZ1VcBtfWwf5Ae9zrOvk/24hBDC7EYYpa6uDmlpaaitrUVqaqrZzSGn8r8pqmUR7HjDULsuAHjlFaBv35bXtX27tBBnVVXkkzKqnWv+fKnm5pVXpIBFq/cwVDvVXldmuQJN/CjTehZwK/18xfIZEynodf9mJojIaHZYpysagW6uffv6BgdadNWo7ZeT4z1fOMcItzsuVMZE7XVllkttFnDANzjUktVm/yayMAZBREaz2jpdWpKDHHmNL8A3yHn1VeD3v/f9nlgCQOW55CUpwg2qwu2Ok0enyavY+wdNoRYFlfd1u4E2bbzbf/jBu125n92ZtQgwURQYBBEZLZ7/Ug+V5Zo7V+oaAbQJAJXnk4ef65VVCxU0Bbv5l5cDS5b47m9E9s+M2b/jNdNJcYlBEBFpJ5wsl5YBYKRZNT2zFKFu/rffLi2Y+qc/AW++GX7wF8soulBZKj3Ec6aT4g6DICIzWWGdLi0ZneWK9HzRjGALN2gKNwDs0kUKgsJ9X+w2jUI8Zzop7jAIIjKTGX+pW4WWAaCcLQkl0ixFJEETb/5EtsMgiIj0Ec6oKi3rdZYskbqcQhU5RxKo6NG1E+h9UXZ7yc8PH5a6z+TzB7sOK4q3TCfFHQZBRKQPI7Nchw9Lj5Mn+47GiraWRhZtdifYzT/Q+6Ls9nrjjfgoLnZyppNsgUEQEelHi0Ak2LHlep3Nm72P8tIUhw8Hr6XRM0sR681fzkDt3QtMny5t07u4WM/PisiiuIo8EeknkhXe5Xl4wl0Nvrxcmo148GDv8PiyMu+2VauCH18OVMK94esRNLndLVejX79e+tq71/e9kOcY0qsrLJLPiihOMBNERNYQ6SioSZOA3r2lf7/7LvDSS8ANNwAXXSRta2iQHuXgYu/e2EZZpad7l8HQKluiVng9b576vnJGKJquMGZ5iFQxCCIibRk1Y/CaNS0DiBUrpC8lZS1NrLQerq5WeP3YY751TfPmARdeKAU/OTnRnde/3XJQNGkS0NzsPb/yEbBPATZRlBgEEZG2IhlWHkvApAwgVq+WusLuvRcYMsR77HnzpIVVc3KkZTXKyqx1kw+1Wr3c1s8/l65Bq+H2yq4vM2ayJrIIBkFEpK1IhpXHssSCMoCQR4eNGwdcdZX33IC3XijS48vsuhZWsHbL661Nnuwdkq/H7M7shiOLYxBERNqKZFi5VvPwyCPC5EclebX2aI9v1FpYysJrOYA5fBiYOFGaYTrSwCtUuwGgutr7nmVlSY9aTvBot9muyXEYBBGRebSaZVlt5Ja8bfRo3+25udJz5cSEwRi1FpZyWH1xcejAK1SRtrLd69erF1z7r21G5DC2CoLef/99PPbYY6iqqoLb7cbq1asxadIks5tFRIEYNWOw2rw8webqiSRDYcZyGOEEXqGuQdluufurtBQYP179mAkJ2nTt2bX7kBzJVkHQqVOncNFFF+GWW27Bj3/8Y7ObQ0ShRDJpoN4Bk393UzBG1rKonSucwCua+Xx69vQ9hv8xBw2K/Jj+jOo+JNKArYKgcePGYdy4cWHvX19fj/r6es/zuro6PZpFZB1WKkT1XwsrVLuMWGLh2mu9ExQCgTMUcpYlP9+33XpNmBhuViqcLAvQcvi7/D3ytcuZIT0Y1X1IpAVhUwDE6tWrg+5TVFQkALT4qq2tNaiVRAarqhICkB7NpmyLFdpVVCS1IdBXUZF3X7m9r7yif7tDvTc1NVLb5MdQ1yAf7/bbg+87cqR0TD1Z4XOnuFBbW6vL/dtWmaBIPfjgg5g7d67neV1dHbLkERBEZK3Mkd5CZSiOHpX2mTxZGjUFSHMLAVLmRMtalkjqZpQZsnBrhYDwhr/H+2dOFEJcB0HJyclITk42uxlE+oqlEFXrIczKtqxfLz2uXu19Xd4Wql16CFVnM3u2NHGgcvJAeY6h6dOl0VPKbrFYRFs3E+galMPq5c9e7+Hv4TCqMJ4oWprmlQyEMLrD/OmVTiMyVSTdPP607q4I1ZZw26U3teteu1badv31+re7psbbTVhRIR23osK7LZxuKuU1hHrf5a4xdkuRTbE7jIjURVqIqucQZrW5aebPl56XlUnrYo0erd4uI8kZioQE4B//kFaclxdjHTZMeu8KC6VlOJ5+WvvCXi2G3SuzLKF+BrQa/k4UZ2wVBJ08eRL/+c9/PM8PHDiAHTt2oGPHjujevbuJLSMyUaQ3VD2HMKu15frrpceyMt91scwk19moTUqonFRQrg0yuhspHP6j6UL9DGgx/J0oziSY3YBIbNu2DRdffDEuvvhiAMDcuXNx8cUXY8GCBSa3zCHcbuk/3WjmJyHrmD0bqKqSvioqpG0VFd5t4cyiHA55PS/50Ypmz5aW1QC8GauKCmDtWqkG6IYb9G9DqLoZ/t4R6cZWmaCCggIIIcxuhnNxHSDrC6cQVYuumEhHlQVrlxkj1JRdgj/84PtamzbAwIHSQqxut/6FvaHmR4r0947FyERhs1UmiIhCkG+oRsxyvHBh8OyEclHTYO0K51haKy8HBg+WvuSuQOVIsPJy6d9GvZ9asmObiUxiq0wQmYDrAMW3hARg5EjpUcntBh5/XPr3/feH/xlb+edFmXFSKyR+7DHgiy+k+XUGDjSnjcq2WvV91JOT5q0iS2AQRMFxHaD41twMbNzoXV5B5nYDv/+99O9p03znoQEC35jD/Xkx4yav7FaS59ZRGj1aCvj8v8eMm7JTf+/Y5U4GYxBEwXEdIOsw+6/kcG7Ms2dLQ82nT5cKjcvK1H9e7HKTN+umzN87IkMwCKLgtCiiJW1odUMOlIU5fFjqDgKA//3Pu//q1dKyERdeKI2a6tIl+DIMfftKz3NypEe1nxejbvLhZpysVkjspN87p3b9kSUwCCKys2iyQ6GyMP7kgmGgZYZGvjHLNzLlsg3B1t0y6iYfbsZJeU28KRvLLllBiksMgih8VvyLOd653cDOnb4zGitvyIcPR54dCpSFycryzQTJwc/8+VJWp3PnwAXDajcy5Wgrs25k8rUePgz86U/Am2+GzjjpcVOOpStTz987s7tYAXb9kbk0XYTD4rh2GFlaTY20BpRy3Si914QKtHaYvD3Qsf3bGstaWGrXrbVQ1+PfHrnd8+dL3zN/fmTregU6v5lrd6m9z1Zol5LV2kOWwbXDiOKdWs2PWqHxY495X/fvtgGM6a6R56KRZzOePbtlV1a43VuhJgs0mvL927tXeszJsX89DkdeEbXAIIjIioLNaPzBB1L3mFK03TWBulrS04G5c73/DtZOq95Y1Wp7Jk6Uusa2bw8cLCq/T9DDsuUAABw6SURBVK5r+uwz7zHCDTKtWlskX7/V2iWfm13uZCRN80oWx+4wspxA3UhyN1ewbrBIu5706HZS674wonsrHKG6EouKYvu+UNcZ7fm1pPbzNXGi+e0iipBe92+XEM5ZjKuurg5paWmora1Famqq2c0hUl/FXOn224EhQ9RnNE5Pl/6CHzxYWvg0VHdNJPsG45/hCDRUXu37/Itw9SzMVWvnxInAXXd5l/II1M7166UuyMsvB95/X/36Qr2f0b5PWgr18zVxonqxODMxZDF63b/ZHUZkpnBGxsg3UrUZjc0Q7egpta4z5bphWq93pXYzf/NN6TzBgkDlXEfvvy89RjN83wpz/YT6+Tp8WHpP4nUOIqIQGAQRmSmcG2WwhUXl2p1XX1U/lh51KXoMaV6yJPpskFbZJLX3SibPdSTvp9zHKvU0akL9fPlfJ5HDMAgiUrLCvCn+ghWLpqdLa3sNHuxd40tJjzlvIslwqAUW69d7R10pAzx5W6SBRLDibPn8n30mzXj9+eeBgxa190o2fbrUNdmunXdNNVmo99PtloLUuXOt8zMlYyEyOZ2mFUYWx8JoCsnMeUoiKShW7huszbHM3xOOUO9XqOLgQEXfkbQrWBsiKU6uqQldkD53buTvp1XmvrFKwTpRFDhPEFG8i2S+nJ07paxF797eIfRqGQ6961JCdcepdZ099pj6MH/ZkiWh34twu/nk8+/dK2VzgMBdd+npwJ13SsXQBw4AhYXS9p/9DBg7Vvp3//7AoEG+bbFLPY3V5mMisgAGQURWnc8lGDmAkG/sgDnrLYXqjlN770aPlvb9zW/CC07UhLuivdpcS23aBG7bmjUtj/vSS9KXfFz/IEiNHX+miByIQRCRXRZwVN5Y5XXE5s+XHuWZpEePlp4Hqh+ySv1HrBmqcIqzA9X4yEGX2ueqXGvst7+VRofNnw9cf733uMprCPR+2uVnisjhGAQR2WUBx2CLlALSHELBhtBr2R0STaZDLWhIT5cKjpcsiez84QRRap+r/zxBoY77/vvA8OHqwVmw99MuP1NEDscgiMgK87mEI9CNtU0bKbsxebJxbYkm06EWNMjb9OgeUjtmqDmClLp08X2M9dxW/JkicjgGQUR2EejGKmdYBg40ri1aZjqCZVTCmbJAi24+tfNYqfuQiHTBIIhIyY43PjNG/RiV6QhngdZwrj/U56p2Hq3eVzv+TBE5RILZDSCiKPDGGhllt5tVzi0vFRJsRnAi0hUzQURK4WQerMBKc75oHZAZNbzc7GHsdvlZI4pjzAQRWZVapsCK2QOtsyzl5dK8Q4MHe4utZ83ybisvt9d5iMiymAkiMjsjEKxdgVZdj+fsQSSzPGtxHsC4YexW/VkjcigGQUSc2M5aC8caVXRtxjB2/qwRWQqDICIrTWwX7qrrWmcPrJJhUrt+IPoV5q3GSj9rRMQgiMhSE9upZQrmzWu5X7xmD6JZ6kILes01pHYeq/ysWSn7R2QSBkFEVhJo1XVlTdC8edpkD8KtTzHyZql2/fPnS8uDvPKKd200rWkx2s4q2bRw2a29RDpgEESkZPb8O4FWXZcDHjlI0SJ7EG59ipE3S7Xrz8mRHvv2ja+btdk/a0TEIIjIh5Xm39GbletT3G5gyxbp31u3So9WHEUVy2gvM37WODqNyAeDICKrCrTqulbZg2D1KW43sHOndJPu3Vt6TYubZbhda8os1dNPS49WrIOy22gvu7WXSGcuIYQwuxFGqaurQ1paGmpra5Gammp2c4isY/t2aYLAqiopCCouVi9QlkV7s/Q/TyButzQqbvp0b02Qf5bKChkL/8yKWjbNCu2U2a29RGfpdf9mJojIKYJlYfwzTLNnSxmgYEFINA4f9n1Ua6N8k/7hB9/X2rTR/iYda9G3lUZ7hcNu7SXSGZfNIPuz4lISViQXOKu9T8r6lO3bpX0CBSG5uZEFDG63dMzt24HNm6Vtmzd7tynbo7aURVmZ9Dh9uu9SFlp87sHeEyKKewyCyP7i6Ua2YwdQUCA9miGSICSaY8rHKitTX6Nr9mypq6yqSso8AdIUAbffDqxdK70us9rnbrfRXnZrL5EO2B1GFA25G2XSJGDNGu3m0Nm9G9i4UXocNEibdkYyGijQPEVffAFMngwMHBh5GyZN8hZXv/su8NJLwM9+BowdK23r39/bVrWuqdGjgfvvj/y8geg1QspuIwvt1l4iHTAIInsye6ivnIXo3dvaE85FOhoo0DxFsQQha9a0bMNLL0lfchsGDZJGoy1cCOTnB34vtfjcOUKKiM5iEET2FE83sh07pMwPIGVKlI+AlCmJNiukzOysXi11Q82fD1x/vbRN70BRzpaF04YjR3wf1bprtPjcrTw/EhEZikEQ2ZMZNzI5C3H4sLfAVw5WVq+WtnfpEnkW6r77pC4wJWWmZORIoLIyujYr2yIvQpqT03I0kFpXVKw1I8qZpuXzyaPChg/3zkckf332mfTaZ595szv+XWPy5753r3c9sUg/d46QIqKzGASRPZlxI1PLQsiBilzwC0SehXrqKd9MUKCamWgou4/Uggzl+mD+3Xp61Ix06eL7+PjjwO9/77tPWZn3/Zw7F3jiCe9rDGCISEMMgojCJWch5ExQWZkUrLz0ktS9M3y4NxMUiUGDfLu7XnpJCoCmTYu9zWqBmzLI0LrbMFTNTkJC9NkltWMD3gxXNHVgHCFF5GgMgsj+jLqRKW+yXbpIgcTYsVLQcv31kWcjjFidPVi3odw1Jc/XI/9bFk1QEWnNzv33e4O9UDVLascGvN1i4QR0/u85R0gRORqDILI/u97I1Lqg+veXaoBi6QJTCtZ9pLY0RqzF5ZHWaoVbsxTNsdWovedE5FgMgoiiIWef+vfXNgs1aFD0RdCR0qO4PJaanc6dfR+1PDYRkQoGQUTRUGafIhm+bub8Rv7dhlYLKgYOlNoXzYSMwYTzngP6d00SkeVwFXkiI+m1Onuswl3dPRJ61jxFcuxw3vNrr9X++olIM3rdvxkEERnJPyuh1gUVbLZkKwQVdqP2ngPAK68Afft6pwhgEERkWXrdv9kdRmSkWLqg9CzqtWtxeTgCBZZy7ZG8yj1g7NIrRGQ6BkFEZAwzs01yNkgegQYAf/oT8OabvvvZdekVIooKgyAis4Qzv5HZC8Vqyczh6WpzDCkDoIkTpedcQ4zIURgEEZklnC6oeFoo1kyhpgM4fFgKgjjknshRGAQRWZndVzy3SiZLOQw+P1/6tzLgUbaJiByDQRCRlVltLp9IWSmTJXfHvfJKy9e4hhiRIyWY3QCiiLnd0o1TzjCYdQwKbfZsadh5VZWUwQKkR3nb7NnGt6lz55YBj9w1ySCIyFGYCSL70aLA1o5rSGmdrTBitJbZmSy17rjqaulzl7fb5fMnIs0xE0RkF1pnK+RAMJ6zYeXl0iSIgwd7u+FmzfJuKy83t31EZCpmgsgetCiwtUqRrt1olTGKJJOl1TntXlhORLpiEET2oEWBrZWKdM0STSCoVddhJLNSa3lOOxeWE5GuGASRPWjxF300x4i3NbUYCBIReTAIInvQ4i/6aI5hxwLqYMINBLXqOowkiNS7u5LD4InIjy0Lo5999ln07NkTKSkpGDx4MDZt2mR2k4jsIT3dG/jJgY/yuXJSQS0KiiMpvta7iJnD4InIj+0yQa+//jruu+8+PPvss7jssstQXl6OcePGYc+ePejevbvZzaNwxNrFpMVf9MGOwQJqcwqKWcRMRAZzCSGE2Y2IxNChQ5Gbm4vFixd7tvXt2xeTJk3CokWLfPatr69HfX2953ldXR2ysrJQW1uL1NRUw9pMfrZvl/6yr6qyZoFqcXHLuhmleKmbCTcYjfTz8g8i1YKZUAGN1X9GiMhQdXV1SEtL0/z+batMUENDA6qqqvDrX//aZ/vYsWOxZcuWFvsvWrQIC4PdzIjUOCUjEclorUiw+JqIbMJWQdCRI0fQ1NSErl27+mzv2rUrvv322xb7P/jgg5g7d67nuZwJIhPYqYuJw6p9Rdr9qEUQySJmIjKArYIgmcvl8nkuhGixDQCSk5ORnJxsVLMoGGYH7CvSjJFWI/n480BEOrPV6LDOnTsjMTGxRdbn0KFDLbJDZDHRLKRphUVOtchIWOE67IjvGxHpzFZBUFJSEgYPHoz33nvPZ/t7772HYcOGmdQqCku4Q7OVrLC2lRbDqq1wHWaJJYh08vtGRIawXXfY3LlzMWPGDOTl5SE/Px9LlizBwYMHcccdd5jdNCLyx24tIrIw2wVBN954I44ePYqSkhK43W4MGDAAf//739GjRw+zm0bhcsIcPfFyHUbj+0ZEBrLdPEGx0GueAdJQvMzREy/XYTS+b0SkQq/7N4MgshYtJtqzgni5DqPxfSMiFZwskZwhXuboiZfrMBrfNyIykK1GhxGRBXDoOhHFCQZBZF3xMmtwvFyHzKih6/H2vhGR5bAmiIgiw8VNichgrAkiIvNw6DoRxSEGQUQUGtd+I6I4xCCIiELTYmV4IiKLYRBERKFx6DoRxSGODiMiIiJHYhBERJHh0HUiihPsDiOiyHBleCKKE8wEEZEXZ4MmIgdhEEREXkbNBk1EZAEMgoiIiMiRWBNE5HScDZqIHIpBEJHTcTZoInIoBkFETsfZoInIoRgEETkdZ4MmIodiYTQRERE5EoMgIvLibNBE5CDsDiMiL84GTUQOwkwQERERORKDICIiInIkBkFERETkSAyCiIiIyJEYBBEREZEjMQgiIiIiR2IQRERERI7EIIiIiIgciUEQERERORKDICIiInIkBkFERETkSAyCiIiIyJEYBBEREZEjMQgiIiIiR2IQRERERI7EIIiIiIgciUEQERERORKDICIiInIkBkFERETkSAyCiIiIyJEYBBEREZEjMQgiIiIiR2IQRERERI7EIIiIiIgciUEQERERORKDICIiInIkBkFERETkSAyCiIiIyJEYBBEREZEjMQgiIiIiR2IQRERERI7EIIiIiIgciUEQERERORKDICIiInIkBkFERETkSAyCiIiIyJEYBBEREZEjMQgiIiIiR2IQRERERI7EIIiIiIgciUEQEREROZJtgqCHH34Yw4YNQ9u2bdGhQwezm0NEREQ2Z5sgqKGhAT/5yU9w5513mt0UIiIiigOtzG5AuBYuXAgAWLZsWdjfU19fj/r6es/z2tpaAEBdXZ2mbSMiIiL9yPdtIYSmx7VNEBSNRYsWeYInpaysLBNaQ0RERLE4evQo0tLSNDteXAdBDz74IObOnet5fvz4cfTo0QMHDx7U9E20urq6OmRlZaG6uhqpqalmN8cwvG5etxPwunndTlBbW4vu3bujY8eOmh7X1CCouLhYNVOjtHXrVuTl5UV1/OTkZCQnJ7fYnpaW5qgfHllqaiqv20F43c7C63YWp153QoK2pcymBkFz5szB1KlTg+6TnZ1tTGOIiIjIUUwNgjp37ozOnTub2QQiIiJyqMTi4uJisxsRjoMHD+LAgQP4+OOPsXnzZowfPx7ffvst2rVrh6SkpLCPk5iYiIKCArRqFdflUC3wunndTsDr5nU7Aa9bu+t2Ca3Hm+lk5syZePHFF1ts37BhAwoKCoxvEBEREdmabYIgIiIiIi3ZZsZoIiIiIi0xCCIiIiJHYhBEREREjsQgiIiIiBwp7oOghx9+GMOGDUPbtm3RoUOHsL5HCIHi4mJkZGSgTZs2KCgowO7du3VuqbaOHTuGGTNmIC0tDWlpaZgxYwaOHz8e9HsKCgrgcrl8vkJNZmm2Z599Fj179kRKSgoGDx6MTZs2Bd1/5cqV6NevH5KTk9GvXz+sXr3aoJZqK5LrXrZsWYvP1eVy4fTp0wa2OHbvv/8+Jk6ciIyMDLhcLqxZsybk92zcuBGDBw9GSkoKevXqhT//+c8GtFRbkV53ZWWl6uf92WefGdTi2C1atAhDhgxB+/btcd5552HSpEnYt29fyO+z++93NNcdD7/fixcvxsCBAz2zYOfn5+Odd94J+j1afdZxHwQ1NDTgJz/5Ce68886wv+d3v/sdfv/73+OPf/wjtm7dim7dumHMmDE4ceKEji3V1k033YQdO3Zg7dq1WLt2LXbs2IEZM2aE/L5Zs2bB7XZ7vsrLyw1obXRef/113HffffjNb36DTz75BCNGjMC4ceNw8OBB1f0/+OAD3HjjjZgxYwY+/fRTzJgxA1OmTMFHH31kcMtjE+l1A9IU+8rP1e12IyUlxcBWx+7UqVO46KKL8Mc//jGs/Q8cOIDx48djxIgR+OSTT/DQQw/hnnvuwcqVK3VuqbYivW7Zvn37fD7v3r1769RC7W3cuBF33XUXPvzwQ7z33ntobGzE2LFjcerUqYDfEw+/39FcN2D/3+/MzEw88sgj2LZtG7Zt24bRo0fjuuuuC5h80PSzFg6xdOlSkZaWFnK/5uZm0a1bN/HII494tp0+fVqkpaWJP//5z3o2UTN79uwRAMSHH37o2fbBBx8IAOKzzz4L+H0jR44U9957rxFN1MQll1wi7rjjDp9tOTk54te//rXq/lOmTBFXX321z7arrrpKTJ06Vbc26iHS6w73Z99OAIjVq1cH3edXv/qVyMnJ8dk2e/Zscemll+rZNF2Fc90bNmwQAMSxY8cMapX+Dh06JACIjRs3BtwnXn6/lcK57nj8/RZCiHPPPVc899xzqq9p+VnHfSYoUgcOHMC3336LsWPHerYlJydj5MiR2LJli4ktC98HH3yAtLQ0DB061LPt0ksvRVpaWshrePXVV9G5c2f0798f999/v2WzXw0NDaiqqvL5nABg7NixAa/xgw8+aLH/VVddZZvPFYjuugHg5MmT6NGjBzIzMzFhwgR88sknejfVdIE+723btuHMmTMmtco4F198MdLT03HFFVdgw4YNZjcnJrW1tQAQdAXxePj99hfOdQPx9fvd1NSE5cuX49SpU8jPz1fdR8vP2llzbofh22+/BQB07drVZ3vXrl3x3//+14wmRezbb7/Feeed12L7eeed57k+NdOmTUPPnj3RrVs3/Pvf/8aDDz6ITz/9FO+9956ezY3KkSNH0NTUpPo5BbrGb7/9NqL9rSia687JycGyZcvwox/9CHV1dXj66adx2WWX4dNPP7VVF0mkAn3ejY2NOHLkCNLT001qmb7S09OxZMkSDB48GPX19Xj55ZdxxRVXoLKyEpdffrnZzYuYEAJz587F8OHDMWDAgID7xcPvt1K41x0vv9+7du1Cfn4+Tp8+jXbt2mH16tXo16+f6r5afta2DIKKi4uxcOHCoPts3boVeXl5UZ/D5XL5PBdCtNhmtHCvG2jZfiD0NcyaNcvz7wEDBqB3797Iy8vD9u3bkZubG2Wr9RXp52TFzzUakVzHpZdeiksvvdTz/LLLLkNubi7+8Ic/4JlnntG1nWZTe5/UtseTPn36oE+fPp7n+fn5qK6uxuOPP27LIGjOnDnYuXMnNm/eHHLfePn9BsK/7nj5/e7Tpw927NiB48ePY+XKlbj55puxcePGgIGQVp+1LYOgOXPmhBy1lJ2dHdWxu3XrBkCKNJV/KR46dKhF5Gm0cK97586d+N///tfitcOHD0d0Dbm5uWjdujW++OILywVBnTt3RmJiYovIP9jn1K1bt4j2t6JorttfQkIChgwZgi+++EKPJlpGoM+7VatW6NSpk0mtMsell16KV155xexmROzuu+/GG2+8gffffx+ZmZlB942H329ZJNftz66/30lJSbjgggsAAHl5edi6dSuefvpp1cE5Wn7WtqwJ6ty5M3JycoJ+RVsZL3cHKbuAGhoasHHjRgwbNkyrS4hKuNedn5+P2tpa/P/27iW0iTUM4/irNrGooRgo3sALXaQIFtvYhaANUrCKFxB0ISUExY0gUnexIpqFEDeKoIhgEVcK1hQXBY1gg6K1ih2xthVR04riqkKtCuLlOZtjOPVyzkm9jNP5/2AWmflS3rcfX/owmY/eunWr8N6uri4bHh4uqofe3l57//79H/m1QTAYtGg0+tVXdZcvX/5uj0uXLv1qfDabdX1eizGWvr8kye7evftHzuvP9L35XrJkiQUCAZeqcofjOJ6ab0m2Y8cOy2QyduXKFVuwYMF/vmc8rO+x9P2tnzEe1rcke/fu3Tev/dS5LvpRao8ZHByU4zhKpVKaNm2aHMeR4zgaGRkpjIlEIspkMoXX6XRaZWVlymQy6unp0ebNmzVr1iy9evXKjRbGZNWqVaqqqlJnZ6c6Ozu1aNEirV27tnD92bNnikQi6urqkiQ9evRIqVRKt2/fVj6fV3t7uyorK1VdXa0PHz641ca/Onv2rAKBgFpaWtTX16empiZNnTpVAwMDkqR4PD5qx9T169c1adIkpdNp9ff3K51Oq6SkZNQuOi8otu/9+/fr4sWLevz4sRzH0ZYtW1RSUlKYe68YGRkprF8z06FDh+Q4jgYHByVJyWRS8Xi8MP7JkyeaMmWKdu3apb6+PrW0tCgQCKi1tdWtFsak2L4PHz6strY2PXz4UPfv31cymZSZ6fz58261ULTt27errKxMuVxOL168KBxv374tjBmP63ssfY+H9b17925dvXpV+Xxe9+7dU3NzsyZOnKhsNivp1871uA9BiURCZvbV0dHRURhjZjp16lTh9adPn7Rv3z7NnDlTkydPVl1dnXp6en5/8T9gaGhIjY2NCoVCCoVCamxsHLVlNp/Pj/o9PH36VHV1dQqHwwoGg6qoqNDOnTs1NDTkUgf/z7FjxzRv3jwFg0HV1NSM2koai8WUSCRGjT937pwikYgCgYAqKys99Yfhn4rpu6mpSXPnzlUwGFR5eblWrlypGzduuFD1j/m89fvL43OviURCsVhs1HtyuZyqq6sVDAY1f/58HT9+/PcX/oOK7fvgwYOqqKhQaWmppk+frmXLlqm9vd2d4sfoW/1++Tk9Htf3WPoeD+t769athc+z8vJy1dfXFwKQ9GvneoL095OCAAAAPuLJZ4IAAAB+FCEIAAD4EiEIAAD4EiEIAAD4EiEIAAD4EiEIAAD4EiEIAAD4EiEIAAD4EiEIAAD4EiEIgOecOXPGSktL7fnz54Vz27Zts6qqKhseHnaxMgBewr/NAOA5kmzx4sW2fPlyO3r0qKVSKTt58qTdvHnT5syZ43Z5ADyixO0CAKBYEyZMsAMHDtjGjRtt9uzZduTIEbt27VohAG3YsMFyuZzV19dba2ury9UC+FNxJwiAZ9XU1Fhvb69ls1mLxWKF8x0dHfb69Ws7ffo0IQjAd/FMEABPunTpkj148MA+fvxoM2bMGHVtxYoVFgqFXKoMgFcQggB4Tnd3t23atMlOnDhhDQ0NtnfvXrdLAuBBPBMEwFMGBgZszZo1lkwmLR6P28KFC622ttbu3Llj0WjU7fIAeAh3ggB4xsuXL2316tW2fv16a25uNjOzaDRq69atsz179rhcHQCv4U4QAM8Ih8PW39//1fkLFy64UA0Ar2N3GIBxp6Ghwbq7u+3NmzcWDoetra3Namtr3S4LwB+GEAQAAHyJZ4IAAIAvEYIAAIAvEYIAAIAvEYIAAIAvEYIAAIAvEYIAAIAvEYIAAIAvEYIAAIAvEYIAAIAvEYIAAIAvEYIAAIAv/QWCOHv2lLEAbwAAAABJRU5ErkJggg==",
"text/plain": [
"Figure(PyObject )"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"using Distributions, PyPlot\n",
"N = 250; p_apple = 0.7; Σ = [0.2 0.1; 0.1 0.3]\n",
"p_given_apple = MvNormal([1.0, 1.0], Σ) # p(X|y=apple)\n",
"p_given_peach = MvNormal([1.7, 2.5], Σ) # p(X|y=peach)\n",
"X = Matrix{Float64}(undef,2,N); y = Vector{Bool}(undef,N) # true corresponds to apple\n",
"for n=1:N\n",
" y[n] = (rand() < p_apple) # Apple or peach?\n",
" X[:,n] = y[n] ? rand(p_given_apple) : rand(p_given_peach) # Sample features\n",
"end\n",
"X_apples = X[:,findall(y)]'; X_peaches = X[:,findall(.!y)]' # Sort features on class\n",
"x_test = [2.3; 1.5] # Features of 'new' data point\n",
"\n",
"function plot_fruit_dataset()\n",
" # Plot the data set and x_test\n",
" plot(X_apples[:,1], X_apples[:,2], \"r+\") # apples\n",
" plot(X_peaches[:,1], X_peaches[:,2], \"bx\") # peaches\n",
" plot(x_test[1], x_test[2], \"ko\") # 'new' unlabelled data point\n",
" legend([\"Apples\"; \"Peaches\"; \"Apple or peach?\"], loc=2)\n",
" xlabel(L\"x_1\"); ylabel(L\"x_2\"); xlim([-1,3]); ylim([-1,4])\n",
"end\n",
"plot_fruit_dataset();"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Generative Classification Problem Statement\n",
"\n",
"- Given is a data set $D = \\{(x_1,y_1),\\dotsc,(x_N,y_N)\\}$\n",
" - inputs $x_n \\in \\mathbb{R}^D$ are called **features**.\n",
" - outputs $y_n \\in \\mathcal{C}_k$, with $k=1,\\ldots,K$; The **discrete** targets $\\mathcal{C}_k$ are called **classes**."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- We will again use the 1-of-$K$ notation for the discrete classes. Define the binary **class selection variable**\n",
"$$\n",
"y_{nk} = \\begin{cases} 1 & \\text{if } \\, y_n \\in \\mathcal{C}_k\\\\\n",
"0 & \\text{otherwise} \\end{cases}\n",
"$$\n",
" - (Hence, the notations $y_{nk}=1$ and $y_n \\in \\mathcal{C}_k$ mean the same thing.)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- The plan for generative classification: build a model for the joint pdf $p(x,y)= p(x|y)p(y)$ and use Bayes to infer the posterior class probabilities \n",
"\n",
"$$\n",
"p(y|x) = \\frac{p(x|y) p(y)}{\\sum_{y} p(x|y) p(y)} \\propto p(x|y)\\,p(y)\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### 1 - Model specification \n",
"\n",
"##### Likelihood\n",
"\n",
"- Assume Gaussian **class-conditional distributions** with **constant covariance matrix** across the classes,\n",
" $$\n",
" p(x_n|\\mathcal{C}_{k}) = \\mathcal{N}(x_n|\\mu_k,\\Sigma)\n",
" $$\n",
"with notational shorthand: $\\mathcal{C}_{k} \\triangleq (y_n \\in \\mathcal{C}_{k})$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"##### Prior\n",
"\n",
"- We use a categorical distribution for the class labels $y_{nk}$: \n",
"$$p(\\mathcal{C}_{k}) = \\pi_k$$\n",
"\n",
"- This leads to\n",
"$$\n",
" p(x_n,\\mathcal{C}_{k}) = \\pi_k \\cdot \\mathcal{N}(x_n|\\mu_k,\\Sigma)\n",
"$$ "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- The log-likelihood for the full data set is then\n",
"$$\\begin{align*}\n",
"\\log\\, &p(D|\\theta) \\stackrel{\\text{IID}}{=} \\sum_n \\log p(x_n,\\mathcal{C}_{1},\\ldots,\\mathcal{C}_{K} \\,|\\,\\theta) \\\\\n",
" &= \\sum_n \\log \\prod_k p(x_n,\\mathcal{C}_{k}|\\theta)^{y_{nk}} \\;\\;\\text{(use 1-of-K coding)} \\\\\n",
" &= \\sum_{n,k} y_{nk} \\log p(x_n,\\mathcal{C}_{k}|\\theta) \\\\\n",
" &= \\sum_{n,k} y_{nk} \\log\\mathcal{N}(x_n|\\mu_k,\\Sigma) + \\sum_{n,k} y_{nk} \\log \\pi_k \\\\\n",
" &= \\sum_{n,k} y_{nk} \\underbrace{ \\log\\mathcal{N}(x_n|\\mu_k,\\Sigma) }_{ \\text{see Gaussian est.} } + \\underbrace{ \\sum_k m_k \\log \\pi_k }_{ \\text{see multinomial est.} } \n",
"\\end{align*}$$\n",
"where we used $m_k \\triangleq \\sum_n y_{nk}$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- As usual, the rest (inference for parameters and model prediction) through straight probability theory."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### 2 - Parameter Inference for Classification\n",
"\n",
"- We'll do ML estimation for $\\theta = \\{ \\pi_k, \\mu_k, \\Sigma \\}$ from data $D$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- Recall (from the previous slide) the log-likelihood (LLH)\n",
"\n",
"$$\n",
"\\log\\, p(D|\\theta) = \\sum_{n,k} y_{nk} \\underbrace{ \\log\\mathcal{N}(x_n|\\mu_k,\\Sigma) }_{ \\text{Gaussian} } + \\underbrace{ \\sum_k m_k \\log \\pi_k }_{ \\text{multinomial} } \n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- Maximization of the LLH breaks down into\n",
" - **Gaussian density estimation** for parameters $\\mu_k, \\Sigma$, since the first term contains exactly the LLH for MVG density estimation (see lesson on Density Est., Eq.1) \n",
" - **Multinomial density estimation** for class priors $\\pi_k$, since the second term holds exactly the LLH for multinomial density estimation (see lesson on Density Estimation, Eq.2). \n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
" - The ML for multinomial class prior (we've done this before!)\n",
"$$\\begin{align*} \n",
"\\hat \\pi_k = m_k/N \n",
"\\end{align*}$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- Now group the data into separate classes and do MVG ML estimation for class-conditional parameters (we've done this before as well):\n",
"$$\\begin{align*}\n",
" \\hat \\mu_k &= \\frac{ \\sum_n y_{nk} x_n} { \\sum_n y_{nk} } = \\frac{1}{m_k} \\sum_n y_{nk} x_n \\\\\n",
" \\hat \\Sigma &= \\frac{1}{N} \\sum_{n,k} y_{nk} (x_n-\\hat \\mu_k)(x_n-\\hat \\mu_k)^T \\\\\n",
" &= \\sum_k \\hat \\pi_k \\cdot \\underbrace{ \\left( \\frac{1}{m_k} \\sum_{n} y_{nk} (x_n-\\hat \\mu_k)(x_n-\\hat \\mu_k)^T \\right) }_{ \\text{class-cond. variance} } \\\\\n",
" &= \\sum_k \\hat \\pi_k \\cdot \\hat \\Sigma_k\n",
"\\end{align*}$$\n",
"where $\\hat \\pi_k$, $\\hat{\\mu}_k$ and $\\hat{\\Sigma}_k$ are the sample proportion, sample mean and sample variance for the $k$th class, respectively."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- Note that the binary class selection variable $y_{nk}$ groups data from the same class."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### 3 - Application: Class prediction for new Data\n",
"\n",
"- Let's apply the trained model: given a 'new' input $x_\\bullet$, use Bayes rule to get posterior class probability\n",
"$$\\begin{align*}\n",
" p(\\mathcal{C}_k|x_\\bullet,\\hat{\\theta} ) &\\propto p(\\mathcal{C}_k) \\,p(x_\\bullet|\\mathcal{C}_k) \\\\\n",
" &\\propto \\hat{\\pi}_k \\exp \\left\\{ { - {\\frac{1}{2}}(x_\\bullet - \\hat{\\mu}_k )^T \\hat{\\Sigma}^{ - 1} (x_\\bullet - \\hat{\\mu}_k )} \\right\\}\\\\\n",
" &\\propto \\exp \\left\\{ {\\hat{\\mu}_k^T \\hat{\\Sigma}^{-1} x_\\bullet - {\\frac{1}{2}}\\hat{\\mu}_k^T \\hat{\\Sigma}^{ - 1} \\hat{\\mu}_k + \\log \\hat{\\pi}_k } \\right\\} \\\\\n",
" &= \\exp\\{\\beta_k^T x + \\gamma_k\\}\n",
"\\end{align*}$$\n",
"where \n",
"$$\\begin{align*}\n",
"\\beta_k &= \\hat{\\Sigma}^{-1} \\hat{\\mu}_k \\\\\n",
"\\gamma_k &= - \\frac{1}{2} \\hat{\\mu}_k^T \\hat{\\Sigma}^{-1} \\hat{\\mu}_k + \\log \\hat{\\pi}_k \\,.\n",
"\\end{align*}$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- The class posterior function $$\\phi(a_k) \\triangleq \\frac{\\exp(a_k)}{\\sum_{k^\\prime}\\exp(a_{k^\\prime})}$$ is called a **softmax** function. Note that the softmax function is per definition properly normalized in the sense that $\\sum_k \\phi(a_k) = 1$. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Discrimination Boundaries\n",
"\n",
"- The class log-posterior $\\log p(\\mathcal{C}_k|x) \\propto \\beta_k^T x + \\gamma_k$ is a linear function of the input features."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- Thus, the contours of equal probability (**discriminant functions**) are lines (hyperplanes) in feature space\"\n",
"$$\n",
"\\log \\frac{{p(\\mathcal{C}_k|x,\\theta )}}{{p(\\mathcal{C}_j|x,\\theta )}} = \\beta_{kj}^T x + \\gamma_{kj} = 0\n",
"$$\n",
"where we defined $\\beta_{kj} \\triangleq \\beta_k - \\beta_j$ and similarly for $\\gamma_{kj}$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- (homework). What happens if we had not assumed class-independent variances $\\Sigma_k=\\Sigma$? Are the discrimination functions still linear? quadratic?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- How to classify a new input $x_\\bullet$? The Bayesian answer is a posterior distribution $ p(\\mathcal{C}_k|x_\\bullet)$. If you must choose, then the class with maximum posterior class probability\n",
"$$\\begin{align*}\n",
"k^* &= \\arg\\max_k p(\\mathcal{C}_k|x_\\bullet) \\\\\n",
" &= \\arg\\max_k \\left( \\beta _k^T x_\\bullet + \\gamma_k \\right)\n",
"\\end{align*}$$\n",
"is an appealing decision. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### CODE EXAMPLE\n",
"\n",
"We'll apply the above results to solve the \"apple or peach\" example problem."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkEAAAG2CAYAAAB4e1KRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xl8FOX9B/DP5iQBEiDhCJeA5RApQjg0KuWwWPFEQ0ULKEr4gS+0KioWIXIEBUVabKkhJogCXigiViqtWgEVrCGAF4IXlMOgRkMSkuyRzf7+mMzu7O7sPbMzs/t5v155bXZ3MvPM7Op8+T7f53lMDofDASIiIqI4k6B1A4iIiIi0wCCIiIiI4hKDICIiIopLDIKIiIgoLjEIIiIiorjEIIiIiIjiEoMgIiIiiksMgoiIiCguMQgiIiKiuMQgiIiIiOKSYYOg5cuXw2Qy4Z577tG6KURERGRAhgyCysvL8fTTT2Pw4MFaN4WIiIgMynBB0NmzZzFlyhSUlpaiffv2WjeHiIiIDCpJ6waEas6cObjqqqvw29/+FsuWLfO7rcVigcVicT5vbm7GL7/8gqysLJhMJrWbSkRERApwOByoq6tD165dkZCgXP7GUEHQSy+9hP3796O8vDyo7ZcvX44lS5ao3CoiIiKKhhMnTqB79+6K7c8wQdCJEydw991349///jdatWoV1N/Mnz8fc+fOdT6vqalBz5498eKLJ5CTk6FWU4mIiEhBZ8/W4uqre6Bt27aK7tcwQVBFRQV+/PFHDBs2zPma3W7H7t27sWbNGlgsFiQmJrr9TWpqKlJTU732lZ6egTZtGAQREREZgcMhPCpdymKYIOiyyy7DZ5995vbabbfdhgEDBuDBBx/0CoCIiIiI/DFMENS2bVsMGjTI7bXWrVsjKyvL63UiIiKiQAw3RJ6IiIhICYbJBMnZuXOnCnt1AGgCYFdh30SBJANg1y4RUTQYOghSnhVAJRITGwAAnEqIosnhAJqbTXA4ugNoo3VziIhiHoMgp2aYTEeRlpaIDh26Ijk5BQCjIIoeh8OBM2d+Qk3NSTgcfcGMEBGRuhgEOVmRkNCMTp16oFWrdK0bQ3GqXbuOqKs7BrvdBgZBRETqYmG0hMkEmEy8JKQdk8nEblgioijhHZ+IiIjiEoMgIiIiiksMgihojzyyGHl5Q7RuBhERkSIYBKnEdLoSyY8shul0ZdSO+dFHe5CRkYiJE6+I2jGJiIiMikGQSkynK5GyfElUg6CNG5/B7Nl3Ye/eD3DixPGoHZeIiMiIGATFiPr6erz22mYUFNyBK664Gps2Pet8b/funWjTxoQdO7bjoosuQFZWK4wZcyE+/9y1IO2mTc+iW7d2+Mc/XseQIf2QldUK11wzHidPnvB73I0b1yM39zxkZbXC0KED8PTTTznfs1qtmDv3Tpx7bg6yslph4MBeeOKJ5YqfOxERUTgYBCnIdLoSCQf3O38AuD1XMyu0ZcvL6Nu3P/r164+bbpqKTZvWw+FwuG2zYMEDePTRJ7BrVzk6duyEyZOvhc1mc77f0NCAlSsfQUnJc3jnnQ9RV1eL6dNv8nnM9etLsWTJAixa9AgqKr7E4sWPYtmyQjz//HMAgOLiv+Kf/3wDGzZsxoEDR1BWtgk9e/ZS5fyJiIhCxckSFZS0rgQpy5e4vZZ650zn79b5i2BbsFiVY2/YsA6TJ08FAIwffwXOnj2LnTvfxdixv3VuM3/+IowbNx4AUFLyHPr374433tiK/PwbAQA2mw2rVq3BiBEXOrcZNuw87Nv3MYYPH+l1zMceK8Kjj67CddfdAADo1as3Dh8+hGeeKcGUKbfixInjOPfcvrj44kthMpnQs+c5qpw7ERFROBgEKahpxizYr7oWgJABSr1zJixrStE8JBcA4OiSo8pxv/rqCPbt+xjPP/8aACApKQn5+ZOxYcMzbkHQhRfmOX/v0KED+vbtjyNHvnS+lpSUhNzc4c7n/fsPQLt27XDkyJdeQdBPP/2EkydPYM6cGbjrLleg19TUhIyMTADA1KnTce214zF0aH/89rdXYMKEq3HZZZcre/JERERhYhCkIEeXHK9Ap3lIrjMIUsuGDevQ1NSEfv26udricCA5ORnV1dV+/9bkMT2x53NfrzU3NwMA1qwpxfDhF7q9l5goLPcwZEguPv/8KP7977ewc+c7uOWWGzFmzG/x/POvBndiREREKmIQZHBNTU144YUNWL58FcaNc8+yTJ2aj5dffh4DBw4CAHz88Ufo0aMnAKC6uhrffPMV+vUb4Lav/fv3ObM+X311BGfOnHHbRtS5c2d07doNR49+h8mTp/hsX0ZGBiZNmoxJkyZj4sRJmDjxCvzyyy/o0KFDxOdOREQUCQZBKnF0yYF1/iLVusBEb731Js6cqcYtt8xAZmam23vXXTcJGzasw4oVfwEArFixFB06ZKFTp85YunQBsrKycc01E53bJycn4/7778LKlX9FcnIy7rvvTowceZFsPRAAPPTQYjzwwB/Rtm0GLr98AiwWCw4c2IczZ6px111zsWbNX9C5cw4GDx6ChIQEbN36Cjp37oJ27dqpd0GIiIiCxCBIJY4uOaoVQUtt2LAOY8f+1isAAoCJE/PxxBOP4pNPhJFqS5euwLx5d+Pbb7/Gr399ATZvfgMpKSnO7dPT03HvvQ/i9tv/gFOnTiIv71IUFz/j89jTpxcgPT0dq1evRGHhPLRu3RoDB/4ac+bcAwBo3boN/vKXx/Dtt18jMTERubkj8Npr/0RCAgclEhGR9kwOz3HUMay2thaZmZnYtq0G3bpleLxrRlLSUfTo0Rupqa00aZ9adu/eiSuvHIuTJ6t9ZmE2bXoWDz54D06dOhPl1pGUxWLGiRNH0dTUG0BsfQ+JiMJVV1eLsWMzUVNTg4wMz/t3+PhPciIiIopLDIKIiIgoLjEIigO/+c0YnD3r8FuQPHXqdHaFERFRXGEQRERERHGJQRARERHFJQZBREREFJcYBBEREVFcYhBEREREcYlBEBEREcUlBkEUsUceWYy8vCFaN4OIDKakBCgrk3+vrEx4X8v9UexjEKSgRx4BVqyQf2/FCuF9NcyaNR1t2pjQpo0J7dolY9CgPnjooftRX1+vzgGJiBSQmAisXesduJSVCa8nJmq7P4p9XEBVQYmJwLJlwu9/+pPr9RUrhNcXLlTv2OPHX4G1a9fDZrNhz573MWdOAerr6/Hkk8XqHZSIKAIFBcLj2rWu52LAMnu2632t9kexj5kgBf3pT0Kgs2yZKyMkDYCkgZHSUlNT0blzF3Tv3gM33vgHTJ48BW+++ToA4MsvD+GGG65E585t0Lt3ZxQUTENVVZXzb99+ewfGj78U3bq1Q8+eWZg06Wp89923bvs/deokbr31JvTo0QGdOrXGqFHDUV7+X7dtXnxxIwYO7IWuXTNx6603oa6uzvmew+HAX/7yOAYN6oPs7DRcdNEF2Lr1Vef71dXVuP32KTjnnI7Izk7DBRf0xcaN69W4VESkIwUFQoCydi2Qlxd5wKL0/ii2MQhSmDQQ6tAhOgGQnFat0mCz2XD6dCWuuGI0Bg8egt279+H113fgxx9/wC233Ojctr6+HnfeORe7dpXjzTffRUJCAm6++Xo0NzcDAM6ePYsrrhiNysrv8fLLb2Dv3k9wzz3znO8DwNGj3+If/3gdr7zyJl555U188MEurFrl6htcsmQhNm5cj9Wri1Fe/gXuvPNeFBRMxfvv7wIAFBUV4vDhQ9i69S1UVHyJ1auLkZWVHaWrRURaKigAkpMBm014jDRgUXp/FLvYHaaCP/0JePxxwGoFUlKiHwDt2/cxNm9+AWPGXIbS0mIMGZKLxYsfdb5fXPwM+vfvga+//gp9+/bDxIn5bn//97+vQ+/enfDll4dw/vmDsHnzC6iq+gm7dpWjQ4cOAIBzz/2V2980NzejpORZtG3bFgBw883TsGvXuwAeQX19Pdas+TO2b/8PLrwwDwDQu3cf7N37AZ55pgSjRo3GyZPHccEFQ5GbOxwAcM45vVS6OkSkN2VlroDFZhOeRxK4KL0/il3MBKlgxQpXAGS1+i6WVtJbb72Jzp3bICurFcaNy8Mll/wGTzzxNxw8WIHdu99D585tnD+5uQMACNkbAPjuu29x221/wKBBfZCTk4FBg3oDAE6ePA4A+PTTgxg8eKgzAJLTs2cvZwAEAF265OCnn34EABw+fAhmsxnXXjverR0vvLDB2YaCgjvw6qsvIS9vCBYunIePPtqj/EUiIt2R1uzs3evqyvI1yiva+6PYxkyQwjxrgMTngLoZod/8ZixWry5GcnIycnK6Ijk5GYCQoZkw4RoUFT3m9TdduuQAAH7/+2vQvXsP/O1vpcjJ6Yrm5maMHDkIVqsVAJCWlhbw+OLxRCaTydldJj6++up2dO3azW271NRUAMDll0/AoUP/w7/+tR3vvfcOrr76Mvzf/83Bo48+EcplIIo5JSXCoAu5TEZZGWC3A7NmRb9dSpArWpYrbtZqfxT7GAQpSK4IWnxUOxBq3bq1VxcVAFxwQS62bduCc87phaQk74/7559/xpEjX+Kvfy3BJZeMAgDs2fOB2zaDBg3Gc8+V4ZdffvGbDfJlwICBSE1NxcmTxzFq1Gif23Xs2BFTp07H1KnTcfHFo7Bw4QMMgijuicO+AfcbuPSGb1R2u3zRsvjcbtd2fxT7GAQpyG6XL4IWn2vxH+CsWXPw7LOlmD79ZtxzzwPIysrGd999g1dffQlr1pSiffv26NAhC+vXP40uXXJw4sRxLFrkfgK///3NeOKJR3HTTROxZMlydOmSg08+OYCcnK7OGh9/2rZtiz/+8X48+OC9aG5uRl7epairq8VHH+1BmzZtMGXKrSgqehhDhw7DeeedD4vFgh073kT//uepdVmIDCOWh337y2CFc15K749iH4MgBS1Y4Pu9aBdHi3JyuuKddz5EYeGDmDjxd7BYLOjR4xyMH38FEhISYDKZ8NxzL+H++/+IkSMHoW/f/li58q+YMGGMcx8pKSnYtu3fmD//PuTnX4mmpiYMGDAQf/7z34Nux8MPF6Fjx0544onlOHbsO2RmtsOQIbm4//6HnMdYtGg+jh8/hrS0NFx88Sg8++xLSl8OIkOSBkLr1gnFvnoJgIzaXWfUdpOyTA6Hw6F1I6KltrYWmZmZ2LatBt26ZXi8a0ZS0lH06NEbqamtNGkfkcVixokTR9HU1BsAv4fkLi/PNepp716tWyPwlZXSe7bKqO2OV3V1tRg7NhM1NTXIyPC8f4ePmSAiIgPQ67DvSLvrtMrIxHI3IwWPQRARkc553pzF54A+btaRdNdpWfit525Gig4GQUREOmaUYd8FBa5AIpRZmrXOyITbbooNDIKIiHTMKMO+I+mu0zIjo9duRooOBkFERDpmhGHfSnTXSTMyCQnRqRHSezcjqY9BEBERhU2p7joxI5OQADQ3A3fcARQXyx9HT+0mY2MQREREYVOiu84zILnjDqC83BUIqVEjZJRuRlIX5wly4jxBpD3OE0TxxleAIwZCYmaIo7bim1rzBHEVeSIi0oyvjExxsSsA4qgtUguDIBXY7Xbs3r0Tmze/iN27d8IeQ3nVRx5ZjLy8IVo3Q1d4TYjCN2uW7yJoMQASR22poaTE977LyoT3KXYxCFLYtm2vYeDAXrjyyrG4/fY/4Morx2LgwF7Ytu011Y/90Ud7kJGRiIkTr1D9WBSaVatWYPjw89GxYzqGDOmHzZtf0LpJpDLeXMMn7SLbu1d4XLtWnUBInKzRc99iGxITlT8m6QeDIAVt2/Yapk6dhFOnTrq9/v33pzB16iTVA6GNG5/B7Nl3Ye/eD3DixHFVjxVNDocDTU1NWjcjInv2vI/HHvsLPv74c0yePBUzZ96Co0e/07pZpCKj31y1CuJ8jdpSKxCS2zeXz4gfDIIUYrfbMW/e3ZCrMxdfe/DBe1TrGquvr8drr21GQcEduOKKq7Fp07Nu7+/evRNt2piwY8d2XHTRBcjKaoUxYy7E559/5txm06Zn0a1bO/zjH69jyJB+yMpqhWuuGY+TJ0/4PfbGjeuRm3sesrJaYejQAXj66af8bm+xWHD//X9Er16dkJXVCuPHX4qKinKvtr7zzr8watRwdOiQig8/fN9rP//73zG0aWPCK6+8hMsuuxhZWa0wfPj52L17p9t2X355CDfccCU6d26D3r07o6BgGqqqqpzvv/32Dowffym6dWuHnj2zMGnS1fjuu2/d9nHq1EnceutN6NGjAzp1ao1Ro4ajvPy/btu8+OJGDBzYC127ZuLWW29CXV2d870tW7bjsssuR+/efTBr1p2w2+2orPze73UiYzP6zVWrIM7fqK3Zs9UZtSX9rPLyjPMZUeQYBCnkww/f98oASTkcDpw8eUL2Zq6ELVteRt++/dGvX3/cdNNUbNq0XjYgW7DgATz66BPYtascHTt2wuTJ18Jmsznfb2howMqVj6Ck5Dm8886HqKurxfTpN/k87vr1pViyZAEWLXoEFRVfYvHiR7FsWSGef/45n3+zcOE8bNu2BU8//Rw++GA/+vT5FSZO/B1++eUXr+2WLFmOioovMWjQYD/7ewB33XUfPvzwAC666GJMnnwtfv75ZwDA6dOVuOKK0Rg8eAh2796H11/fgR9//AG33HKj8+/r6+tx551zsWtXOd58810kJCTg5puvR3NzMwDg7NmzuOKK0ais/B4vv/wG9u79BPfcM8/5PgAcPfot/vGP1/HKK2/ilVfexAcf7MKqVSu82upwOPDQQ/dh4MBBGD58pM9zotgQjZurWhkbrYI4XzVCYpvUWExV3LdYf8RC7PjBeYIUcvp0paLbhWrDhnWYPHkqAGD8+Ctw9uxZ7Nz5LsaO/a3bdvPnL8K4ceMBACUlz6F//+54442tyM8XggKbzYZVq9ZgxIgLndsMG3Ye9u37WPam/dhjRXj00VW47robAAC9evXG4cOH8MwzJZgy5Vav7evr61FWVoySkmdx+eUTAABr1pRi4MC3sWHDOtxzzwPObRcuXOpsqz+zZt2JiRPzAQCrVxfj7bd3YMOGdbj33nkoLS3GkCG5WLz4Uef2xcXPoH//Hvj666/Qt28/59+K/v73dejduxO+/PIQzj9/EDZvfgFVVT9h165ydOjQAQBw7rm/cvub5uZmlJQ8i7Zt2wIAbr55GnbtehfAI27bzZlTgP/+dw+2b/8PUlJSAp4bGZ/aa1OpuQBpPC0wyuUz4hMzQQrp0iVH0e1C8dVXR7Bv38eYNEnI2CQlJSE/fzI2bHjGa9sLL8xz/t6hQwf07dsfR4586XwtKSkJubnDnc/79x+Adu3auW0j+umnn3Dy5AnMmTMDnTu3cf48/vgyr+4k0dGj38Jms+Giiy5xvpacnIxhw0Z6HUPaDn9GjnSdU1JSEoYOHe7c18GDFdi9+z239uXmDnC2BQC+++5b3HbbHzBoUB/k5GRg0KDeAICTJ4W6qk8/PYjBg4c6AyA5PXv2cgZAgPA5//TTj27bfP75p9iw4Rm8/PIb6Nq1W1DnRsYnd3NVktoZm3jIkESzEJv0hZkghVxyySh069Yd339/SrYbymQyoVu37rjkklGKH3vDhnVoampCv36uG6vD4UBycjKqq6vRvn17v39vMpn8Pvf1mtgdtGZNKYYPv9DtvUQfBQPitfHen8PrtfT01n7b7Y+4r+bmZkyYcA2Kih7z2kYMSH//+2vQvXsP/O1vpcjJ6Yrm5maMHDkIVqsVAJCWlhbweMnJyV7Hl3aXAcCxY0cBAP369Q/9hMiQorU2lZoZm1jPkHD5jPjGTJBCEhMT8fjjTwLwHVQ89thqn8FBuJqamvDCCxuwfPkq7Nlz0Pmzd+8n6NnzHLz88vNu23/88UfO36urq/HNN1+hX78Bbvvbv3+f8/lXXx3BmTNn3LYRde7cGV27dsPRo9/h3HN/5fbTq1dv2fb26fMrpKSkYO/eD5yv2Ww27N+/D/37nxfWNSgvd51TU1MTDh6scLb3ggty8eWXX+Ccc3p5tbF169b4+eefceTIl5g3byHGjr0MAwachzNnqt32P2jQYHz22UGvmqVQXXrpaOzeXR54Q4oJWoxyUjpjEw8ZEi0KsUk/mAlS0HXX3YBNm17FvHl3uxVJd+vWHY89ttpZN6Okt956E2fOVOOWW2YgMzPToz2TsGHDOsyefafztRUrlqJDhyx06tQZS5cuQFZWNq65ZqLz/eTkZNx//11YufKvSE5Oxn333YmRIy/yWcT70EOL8cADf0Tbthm4/PIJsFgsOHBgH86cqcZdd8312r5169YoKLgDCxY8gPbtO6B7955YvfpxNDY24JZbZoR1DZ5++u8499y+6N//PKxZ8xecOVONadNuBwDMmjUHzz5biunTb8Y99zyArKxsfPfdN3j11ZewZk0p2rdvjw4dsrB+/dPo0iUHJ04cx6JFf3Lb/+9/fzOeeOJR3HTTRCxZshxduuTgk08OICenq1v3YiC7d7+HRYvm48CBw2GdJ2mnpESovQllZfNor02ldMYmXjIk/gqtY+H8yD8GQQq77robcPXV1+HDD9/H6dOV6NIlB5dcMkrxDJBow4Z1GDv2t14BEABMnJiPJ554FAcP7ne+tnTpCsybdze+/fZr/PrXF2Dz5jfcCnTT09Nx770P4vbb/4BTp04iL+9SFBd71xaJpk8vQHp6OlavXonCwnlo3bo1Bg78NebMucfn3yxdugLNzc0oKJiGs2frkJs7HK+//q+A3Xa+LFmyAn/5y2P45JMD6N37XLz00jZkZ2cDAHJyuuKddz5EYeGDmDjxd7BYLOjR4xyMH38FEhISYDKZ8NxzL+H++/+IkSMHoW/f/li58q+YMGGMc/8pKSnYtu3fmD//PuTnX4mmpiYMGDAQf/7z30NqZ21tDb7++khY50jaCqf4OJo3VzW63bjAKMUDLqDqFNsLqO7evRNXXjkWJ09Wo127drLbbNr0LB588B6cOnUmyq0Lz//+dwznn98be/YcwODBsbFsBRdQ1S9fgYbWo6V8tUMv7SNSgloLqDITREQUBL0OF9dTxiacbkMiLTEIIiIKktpz/oRDTzUtas5ZRKQGjg6LE7/5zRicPevw2RUGAFOnTjdMVxgAnHNOL5w964iZrjDSP7Xn/DE6oy8VQvHHUEFQcXExBg8ejIyMDGRkZCAvLw9vvfWW1s0iojgQD8PFlcB1uMhIDBUEde/eHStWrMC+ffuwb98+jBs3Dtdddx2++OILRfbvcEB2okOi6HGAX0H9ifacP0YXD7NMU2wwVE3QNddc4/b8kUceQXFxMT766COcf/75Ee49GQ4HYLE0oFWrwDMEE6nBZrO2/KbSEt0UFj0VH0dbOMXOsT7LNMUOQwVBUna7Ha+88grq6+uRlyc/YZ3FYoHFYnE+r62t9bPHRDQ3t0NVlbDeU2pquuxSEURqcTia8csvP8FuT4eB/9OMSXoqPo62UIudo7VUCJESDPd/2s8++wx5eXkwm81o06YNtm7dioEDB8puu3z5cixZsiSEvXeBzQb88MOPMJkAxkAUTQ4H0NycAKAnAH75SB/kZokOZW6iWJxlmmKH4SZLtFqtOH78OM6cOYMtW7agrKwMu3btkg2E5DJBPXr08DFZopQdgE35xhMFlAKDlepRnBADHLGLS657UNp15tmNJu060+ucQZznSL84WWKLlJQU/OpXvwIADB8+HOXl5XjyySdRUlLitW1qaipSU1PDOEoiWJNBROQSzBxJ0gDBsxtNGgzpdc4gznMUfwwXBHlyOBxu2R4iIlJeqMXOoXSj6YUR20yRMVQQ9NBDD2HChAno0aMH6urq8NJLL2Hnzp3YsWOH1k0jIopZ4RY763WpEX+M2GYKn6GCoB9++AHTpk1DZWUlMjMzMXjwYOzYsQPjx4/XumlERDEp0mLnUJYa0UtNjh6XRyF1GCoIWrdundZNICKKK5HOkRRKN5peanI4z1H8MFQQRERE0RXJHEmhdqP5q8kZNsz/cZTKEnGeo/jCIIiIiBQXbjear5ocX3+nZJaI8xzFHwZBRESkuEi60fzV5Kg1cqukBNi/3/e+hg2L7eVR4hWDICIiUlyk3WhyNTlqjtxKTAQqKoARI7zbwiHysYtBEBERaU4cGQbI1+SUlwvbqDVyi3MExScGQUREpDnpyDC5oKOiQghKAPVGbnGOoPjDRYqIiEhzBQXeI8CkmZjZs4VskPh8717hce1aV3CkVDvEAItzBMU+BkFERKQLJSWuwCYvz7srqqLCe+SW0oGQXD0SxS52hxERhUAvsxrHKl81P5FO2uhJ7nMUM08jRgBDhvievJFiB4MgIqIQ6GVW41jla2RYJKPN5Hh+jtIAqLxc6JrjHEGxj0EQEVEIYnUUkR4yXNGcrdnzc7TbXQGQ3GSJnCMoNjEIIiIKUSyOItI6w6XFbM3S/YuZJ39dbhR7GAQREYUh1lYa1zrDpWTNTyhZrVj7HCk0HB1GRBSGWBxFJB1tJTc6Sy1JVZVYjMWYPbHSZ7tC6YoTs1qen4kY1ImTMoqvxdrnSMFjEEREFCJphkSt+Wq0osU8OclVlehaugTJVfJBUKjkhs7LZbVi+XOk4LA7jCjO6aEg1khifaVxX6OzjCZQ3Vasf44UHAZBRHFO64JYo1F6vho9UWN0lq8gO6mqEmWlDjQ3m/DA+fsBAOmH9zvft2XnoCk7J7yDtvBX7xPLnyMFj0EQUZzTuiDWaJSer0Yv1MqM+AqyNy34EmsqxmEpCtFr6zIAQK9lM53vfz9zESpnLQ79gBL+slqx+jlSaBgEEVFMDvnWAyN1NaqVGfEZZFeMw5353yP/+utx7PA56LVsJo4tLEXDgFwAQiYoEtGcc4iMi0EQEQHgUOFghBrUGKmrUc3MiK8ge3pBVzSiq3O7hgG5aGwJgsIhfj7isTyDumHDGAiRO44OIyIAHCocjFCGXgPBj1KKB9EYdSZ+Pp6zPkuXxJg9G/j4Y9/f77IyIZii+MBMEBGx6yBI4dRPsatR4K8+x5adg+9nLvLbBRZKFm7tWiAhQf7zKSsDTCbjZOhIXQzsIYyDAAAgAElEQVSCiOIchwqHJpygJp67GktKgIMH3bMz4neuokJYrX3WrJyARdBilqeiAigudr0uzfKUlAiBUEWFcLyRI4HmZu/jzp4tbO9rAVUj1HCRMtgdRhTn/BXEzp7NocJyQu3aieeuRjEAkgYXBQWuxUoPHgxuP9K/ueMO4TXPld/F7sjiYiET1NzsOyMkNzu2uJ9guzvJ+JgJIopzHCoculAmFIz3rsYhQ4RHMbgQr4EYGInvB6O4WAiApFkeuZXfy8pcAVBzs3dGSOSZoSsu9v584rWGK14wCCIiCkEoQY2aXY1GGX4vtkG8FpHWRRUXu4IawHcRtPiauK2YEZLyF8zGew1XvGB3GBFRkHwFNb7WnJLrahRHHsl1NYYyMinUkWpKKilxHVf6u3h88Rykvys1OkzM8oikwY1cgCrNCIndaJ7beq4bpsX6aaQNBkFEREEKtX5q1izvbaVzB0kzNaEGL1oOv5cGYNLfpefgeT5K1EVJa4AA7+BG+vlIr8XHH7vX+wQKZu+4Q2hjQoLvtuptKL1nMCqlt7bqCbvDiIiCJAYtcl1R0myEv64oJZcp0arrRnrc2bNdwQPgGl4u12UYSV2UZxG0uC+xRuiOO1yjxuSup7TeZ9gw38GsOLJMeh6ebdXjUHojTcypJwyCiIhCFOkNR8ngRavh99JzSE52ve55PkrVRdnt8kXQ0mJpsSsrmCVAfNVSee5fbKfYVvG53uqElAyu44nJ4XA4tG5EtNTW1iIzMxPbttWgW7cMrZtDRAbmK7sRyg0nL88VvOzdG1k7xG6maN/wpOcAeJ+PkgXc/vYl7iM313sb8TiJif6P52v/4jUWu9/0HFRo/X1QS11dLcaOzURNTQ0yMpS7fzMIIiIKUyQ3HCVuVkoEYv4ECmDKy4XuI/EcAO1uvr66y3y9HiolAtZoMVJbg6VWEMTuMCKiMMl1RQWT+RC705SokVFzpu9A3X6Ae+0MAMyYEXwblMwSSc9dnA1arO+JNAAKZV4orRmprXrAIIiIKEziDUc6ikgaOACuG7m/jES4NTKB6l4i5a/OBPAugpb+Li2W9nU+ShfzStubkCBcZ/ExkgDIKJNdGqmtesEgiIjCZpQJ+9TgGdSI2Qe50VLSm5PdLoxOijR48byu0s9CrqYl3M9Croh72DDXMhglJd4BhvRY/s5HjWJeaXYOEGp4wi0YN9K6ekZqq54wCCKisMXrsFxf/+IWAyFxHhsgtNFfkdyk9u8Xun889yMdFh4uz24/6ZwznoGV3LQBgfYNKDfMX5qdEydKDLdbKBrZNqUYqa16wiCIiMIWr8NyPW84ct0ws2cHN3RdqWzaiBFCEOSr60oamIV6/MREdetMlBrmL5edk2bpxGMFy0jr6hmprXrCGaOJKCIFBd6rccdyAATIzwRdUCDcwMXuFyC4GZLDXf7Cc4Zg8XMAhL8bOdK9Sy5QXY6v4x886Ht5CaUoPZu0GISWFlXi/mH/cQuElGw3GR8zQUQUMa0m7NMTzxt5sAWq4WbT5LoiCwpcI6LE9bUCBaT+jq9UEbc/ShXzitk5ac1V8uFKrKy4DLvPq4W9ua3P9dpiuXaN/GMQREQRi/dhuZ41UNLRYUDgwCGcuhhfwUt5eejt93V8pYq4fVGymNdfEDNyYB3WbGmLESPk12uL1do1CoxBEBFFJN6H5XqevzhaCnC/DoECh3CyaXLBi0iakZJu629foRxfic9WjWLepKpKJFdVAgDSD+8HANx33j+B/CuxZm1XJNTX4fa728ZF7RoFxiCIiMLGYbneN3LPjIT0Ru7vWoSbTfMcEg54B6Sen4VcMbTcnEdqf3ZqFPN23FKCrqVL3F7rtWwm/gagExbi4Y1FKH0p8lFo8Tw9RCxhEEREYeOwXGVu5JFk08TgxWQCHA7X/D3Sv127VugmE5971hP5mvMolHPQi5/yZ+HM6GsBCJmgXstm4tjCUjQMyEU+gKLbHLDZTBHXrsXr9BCxhkEQEYWNw3IjF0k2zXMSxoMH3VdTl/6tXEbK39ISoXZrhpMZUSOb0pSdg6bsHLfXGgbkonFArqK1a/E6PUSsYRBERKShcLNpvm64csGL3A3Z19xG/oInf8LJjEQzmxJMti3UoEzpiR4p+hgEERFpKNxsmhJdkYGKoUO5mYeTGVE7m2LLzsH3Mxfh7//qi7UbA2fbwgnKOD2EsTEIIiIyIKVqkZSc2iDSof5KZ1OasnNQOWsxbDLrm0mPLQaM4QRl8T49hNGZHA6HQ+tGREttbS0yMzOxbVsNunXL0Lo5RERR49nV45ndKC8X6oOUCEDy8lyBwd696v2NWsRrIwY2/gIguS42dokpr66uFmPHZqKmpgYZGcrdv5kJIiKKA9KuHsB7csfZs5UZFRZOZkRv2ZRgurg4PURsYBBERKQgvc4fI71BDxvmHQBJ2xvu1AbhDPXX42SbwQRlnB4iNjAIIiJSkJ7nj5EGQp9+Kt/VE0kGKNTMiB6zKcEGZZweIjYwCCIiUpDe549RazRTOJkRvWVT9BiUkboYBBERKUzpEU9KdrGpVX8TTmZEb9kUvQVlpL4ErRtARBSLCgpcgYZSSzSUlbm/LmYuEhOD248007F3r/Aot99oKinxffyyMuH9aJk1y/fnVFDAtcBiETNBRKRbei0yDobelmjQa1ePnmuoKPYxCCIi3TLqDVKNEU+RdrHptatH7zVUFNsYBBGRbhnxBqlmxiWSoma91d/IHZ9rcFG0MQgiIl0z2g3Sbhfm4fEktre8PPxuPL1NKqgkrsEVOiN3F+sFC6OJSPeULDJW26xZrpmX5Qp+KyqCL2SW0mNRs5LkAjzyT6mC+XjGTBAR6Z7RMiBKd+PptahZKXqcNdoIjNhdrDcMgohI14x6g1SyG0/Nouakqkp03FKCn/JnoSk7J/wdhSnWAzy1Ga27WG+4ijwR6Zavf9Vq8a/dcOsv9LQ6upy0w/sxcOowHNpUgcYBuVE/PutalKH371mk1FpFnjVBRKRb/jIgs2erM6zb1+R9Yv2F5w3ZX/2FEepc/r4lB0VYKPteNCYr5ASFkTPC90yvDBUELV++HCNGjEDbtm3RqVMnTJw4EUeOHNG6WUQUQ3wVm4oqKlzv+ctI6bmQOamqEmmH9yPt8H6k/nIKD6MIZaXNzteSqipZXGsQev6eGYGhaoJ27dqFOXPmYMSIEWhqasKCBQtw+eWX49ChQ2jdurXWzSMihWkxWWKgYlPxPX/1F0rXuSjdZdRxSwm6li4BAPwZQHssxMO7itB+VyEKsQwPDHsXaytyWFuic6ynipyhgqAdO3a4PV+/fj06deqEiooK/OY3v9GoVUSklkhHv4Rb9Buo2DTQfDZKFzIrHQz+lD8LZ0ZfCwBIP7wfhctmonr0dXh4VxGKkpbAVpHAAMgA9DoLuJEYKgjyVFNTAwDo0KGD7PsWiwUWi8X5vLa2NirtIiLlRDL6JbmqEl1Ll+DM6GtDHvnka/K+YIbrKz07s9JDoZuyc7yuR8HMBKzZA9hsCbqfi4kEep4F3CgMVRMk5XA4MHfuXFx66aUYNGiQ7DbLly9HZmam86dHjx5RbiURKUGLyRLlgh0t6y/EYvC1a4WRQEqPjive2oXFtRR3DJsJuvPOO/Hpp5/igw8+8LnN/PnzMXfuXOfz2tpaBkJEBhTKZIlJVZVIrqoEIHT1SB8BwCaTBZE7nq+5ibSsv1BjaQlbdg4eGPYu1mzpari5mIgiZcgg6K677sIbb7yB3bt3o3v37j63S01NRWpqahRbRkRKC3WyRGnRr6jXspnO37+fuQiVsxYHfTzxOOXlwsgwT9Gsv1Bj5uy1r+d4FUGzuJbihaGCIIfDgbvuugtbt27Fzp070bt377D2M316P6Snt0arVulITU1Dq1ZpLY/Cc/FH+rxVq3Tndt7veW+bnJwCk8mk8BUg0he1J7oLZ/SLZ9Fvr2UzcWxhKRpaJgK0BcgC+So2FecPkgt2otU9p8bM2bFWXKv1DNh6aQMFx1BB0Jw5c/DCCy9g27ZtaNu2LU6fPg0AyMzMRFpaWtD7qa7+AdXVarVSYDKZvAImuWDLM5hKTfUXbPl+LynJUB8lxQi1h7CHc4OWK/ptGJAb9GzIeiw2VXMotB7PNxLBFsOrGahEUpBP0WWoO2dxcTEAYMyYMW6vr1+/HtOnTw96P9sXrURCq3Q0mC2ob2xEo9mCBrMFjRYLGsxmmK3C740WCxosFjRaLWi0WtFotcBstcJss8Jis8BstcBis8DSZIHFZoXFZkazoxmAkLUymxtgNjegpuZnpS6BT4mJSW5BUVqav6yW70AscNYrHamprZjlIgDqL+AYazfocBkxW6P35TAYqBBgsCBIqWXOLs3th4xu3RTZl6PZAUezA81NzbDbHbCYrahvtKChoRFnG8xCINVoRn1DoxBQmYVAq8FshtliRYPFDLMYbDkDLltLwCUEXY02C8w2CyxWCyxNQrBlsVlgbbI622G3N6G+vhb19dGZBiAlpRVapaYhNUDA5Os9z/f9vZeUlMygS2WR/KtYzws42rJz8P3MRQG7wLQW6PobMRiM5kSXShTDx0IbKHSGCoL0yJRgginBhISkBCQBSG2djAyoN3u1GHA1NwNN1iY0mq0429CIxkazEGyZLahvaHQFW41mV0bLbHEGXULAZUVDS5ZLzH6ZbVYh4LJZYbaaW4IuC6xNFjTZm5ztsFrNsFrNQJ3K/YoAEhIS0Cq1JUjyymIF7kIMVPfl+V5CgmFnjghbpP8qVmPUkhKasnP8FkHrRSxmJdTOEkoFWwwfaaDiL7u1acGXaF2xG4vhakcoBfmkDQZBBmNKMCExJRGJAJJbJSItIxUd0Fa14zU3NTuDLqvFKnQhNgjdiA0tgVdDoxn1jRY0WswtgZYQgDWaxW5Fs5Dhslid3YqNViHYMlvFoEv4MdsssLZ0MYqZv+bmZjQ0nkVD41nVzlMqOTlFCIpS05HaKvguxEB1X3LvpaSkxkSWS41RS2R8clnCYcN8bx9uN1mwxfCRjhz0m92qGIc78wfg0PXXhlWQT9pgEER+JSQlICFJyIykpCehDdJVO5Y0y9Vsb4a50YKzjWY0NJhR32gWArD6RlewJelaFOq5LK56LqvQ1ShkuayubkWrtGuxpZaryQKbpGvRZrPCZrPi7Nka1c5VJC2gV6ILMVAgJi2gVyp9r9aopVgXL90nnlnCESOU7yYLthg+0pGDgbJb0wu6ohFd/baB9IVBEOmGNMsFJCK1dTIy0Ua14zU3NQuBVzNgs9jQ0GgRgq1GM+rFeq6GRtSbhbquRrMZDRZrSz2XEGCJAZiY6WqwtnQptnQxmsV6LptYyyUEXc3NQiWrtIA+GhITk5xBUWurBW3OnkEagHQAaQDSls10/t48cDjsuaP9BlvvvpuGf/wjHZMmpWH8+DT88IPwu82WhrVrWwFICDkQipfhxZFmJYzCM0sIuGa+BtTtJvMU6chBQN81cBQ6k0OpamMDqK2tRWZmJmq2bVOsMJooGNIC+uZmwGK24myDWQiyWuq26htauhctFjSaheyXuSXQEgIvzyyXTch82awwW8SAy7uWy2KzBG6gShITW6F1epqPei75zFWbujPo+VoJam6Zh4Se/by6In39bbQK6JUa9eSZCZLLShg9CPSVJRSzPWvXuoIjpQKJYIPotMP7MXDqMBzaVBFWtiYvzxXc7d0bXhsoeHV1tRg7NhM1NTXIyMhQbL/MBBFFgbSAHhC6Ftt2iE7Xot1mF7JcDUI2q16S3WpoFAIsy/+OIvnFp3H62j/gbOt2LZktIRgT6rhaslySaSIarS2ZLWk9l0cBvd1uRm1dmAX0Gx4PafOEhARXLVdL0BXclA/ydV+ewZb4e3NzGkpL0wAkRtSdo0RWQs8CzW00e7Y668EFWwwfycjBQDVwRinIJwZBRDHJs4C+VdsU/wX0hw8DLz4N3HglMGBAyMfzLKCvb7SgodEiKZxvKaQ3S6aJkEwZYamphrmuBmabDZbqn2D65jP80qMfzialotFmRaMDaGy2S4rorS01XWa3AvrGxno0NtaHedVCs3ZtCp55Jg1t26bBak1DXV06OnZMw3//m4ZPPgmtXivjp1P4EcDpbz6DKSnZ629DLaD3zFZJMxNrX8+Jyhw9geY2Ki/Xtpg+3ECFNXCxhUEQEQHZ2cDMmcJjGLwK6NuHmOUqKQFKN7u/duIr1+8zZzrv2p5ZLovFKmS26huFQnqzq5bLbG55bragwdzYMlrR6jVqsUHMcEkK6C0t3YtipsuzgB6wwmq14uefXQX0P/0k/IRt8XTZl00mkzA3lxhMyXQvSoOrb75Jw2efpaOiIg0XX5yGNmeqcO5zj+GlTzrj3x93xVVXpeHgQd8ZsaSk5AhOQhAoyKqoMF4goebM3aQN1gQRkfaqqoSfo0eBwkLhtYULXVmp7OywAzTR4pIcJCY6UFhw2uu9orIusNtNWDyr0us9McvlgAlN1ibUi6MVG8244LaBsNnNSEqsx+bCcjSIGS+LFY0WM3bsS8euz1vjwn7fY1DPH4RaLosVn59IxjeViejW4WdkZdTAYrWi0ebqXrQ2Cd2LYgF9tIkF9GLA1SotuC7EYGq+3ngjDS+9lI7p09NQUJCGlJRWSEhIiFpxdCT0Pgt2LGNNEBHFLrkgZ8CAsLrmfElMdODhtcI/fqSBUFFZFzy8thuWzj4l+3fSLJc4N1c2MlFU1gU2ezekJDfDakvA56d7eQVYf5Ls/6pxp1BYcBpFZV3w+sfC8Ty3lyugd81A39gyA70FDY1CpqtRzHQ1ml11XC21XOa6GlR81wqfnW4LE+rhgBk90k+hXdtamJtsMDfbYbbbYbaZvQro7fYmNDTUoaGhTolLL+vZZ4UfwDUDfXp6Gl54IR3vvOO/LkurGeiNOHM3+ccgiIi0JWaBAKE2SXT0qPCoQBYIcAU+0kBIGgDJZYh88fw78bn0OHLHXbYuB1ZbApZO+xqF9pVAVb7buckW0CMdQPvQT7ikBNhTilSYYUUqUmDB8YZWgDgbw8yZcMz8P7euxUazBWfrzc7RiWLGq7El0KpvGaXoXGvRIoxstNSegf3YV6jJ7op6mIRuRed0ES0z0Lcs9yMu/SM7Az2EAvpaFVf/SUxIdA+Q0twzW9IZ6JWYFDUeZ6A3EnaHEZG2SkqA0lLf70vqgZQgBixiBifSACjQ66LUvKGw2hKQktwMy/oXgalTgU2bFM12uamqQlFpFzy85QKkJDbBak/C0tHvonBmS9sUCi4BCMFrEOcjLaC3WWxoaJkawjk9hLllclRx/i2zRfhdMvt8g1noKhRmoG95vWVurkarxWvEoqVJmIVeXNw62uRnoA88KWowIxZjdQZ6OewOI6LYlJ8PjB4t/H74MLBsmXc9kIIKC047MzIpyc0hBUAAYLebZAMd8bnd7n0TKirr4jye1ZaAoq2DUBj+KQSl6PVBeHhLS1B26U4UTT2Mh3cVAeeFFvQpybOAvjXS0FGlY3nOQG8xW4XC+fpG1DcKwVWDuM5iy0LXQoZLnIVemH+r0Wpx1nKJy/6Iy/00BlFAr9cZ6INZVzFQICadgd6ojH8GRGRsUagHkvIKSMq6hBQUyBVPi3wVXT+8tpvQBfa7j1G0dRAe3nIBgIUolHb/KZiZ8cpKHQYKsQzIn4SH117gs60hkevGVOl8wiE3A72ai1v7K6BvkIxabLBYWxa5dhXQO+fjEtddlJuBXrrsjw5noA8ncxXKPF3Nzepk8hgEEVHcCKWWR/Hj2VcCU0tbMkAL8TCKgGWFKMRUYWMFu/28slUtUyAU5p8GOmbLZqtCtmWLdzfmsmWu3xXuxtQ7XwX0avA3A/1ZccmfRtcM9GJXYzAz0JttQjBmlmS4xPm5tCigVxtrgohIP6qqgI0bhd+nTVM0kxBuLU8k3Ible2ROipYB9lFjXZkljTMnIfPMBMl1YxrpfMinYGegbzRbUN/Y6KzhEhe5Fmu3zBah67HRanEubu2vgN5qs8Bmt7m1hTVBRBS7srOBCROEItsJExS9iYZTyxMpt64zj6DAjq+R2CkLGOCdLSh6MgP2Lw5j8aMp+g0kotyNSdoJeQb6CMkV0J/+6WcMvm2y4sdiEEREcSHUWh61JcIu1AZ1lMlMbeyGpdgAVA3QbxBEpBK5AvrU1omqHItBEBFpT+dFtorLzkbhzB8A89d4eG1fAB7zFuV/gsItywBs0radwTKZgNxc4ZHIQBgEEZH24q3INjsbmDULhagFWp9yn0gx/xMUnveqsJ1RAkGHA9i/X3gkMhAGQUSkvSjPFaQnbvMWwYLCLUNcb8ZyIEikAwyCiEh70SiyraoSMk75+boKqtznLUpFUf5BIRMUYiAY7gKxYYu3LkyKSVzUhIi0U1UlLJsh3kzVPlZpaXSOFSTp8HzL3gNYOvsUHt5yAYq+nCRsIAaCAwIXSIsLxBaVdZE9RmKiwl1VW7YIo/imTnVlrJYtc722ZYuyxyNSATNBRKQdMTAZPdp1k2+Z2C/Wswhy8xO5FlttmVE6hP1FtEBsOFmyOO7CpNgRVhDU2NiIX375Bd08Jhz84osvcP755yvSMCKKUy1Fw4rQcZeN33mL6htg/2IkkJ0S0j5lV6wPZhJIuWA0EM4TRDEg5CDo1Vdfxb333osOHTrA4XCgtLQUF154IQBg2rRp2L9/v+KNJKIYEs3ARMejzvzOW3R3LYCuYe030gVilRL1GiWN2e12vH/gACqrqpCTnY1RQ4ciMVGduW1IOSEHQcuWLcP+/fvRsWNH7Nu3D7feeisWLFiAP/zhD4ijFTiIKFzRDEzisMsm6AVilQxGZbowxRolQDIZZVUVihaY8XDFMCydfSrUU9Ot1/7zH9z9xBM4+eOPzte6d+qEJ++/HzeMG6dhyyiQkIMgm82Gjh07AgCGDx+O3bt344YbbsA333wDEyfKIqJAohmYxFmXTUgLxCoZjMp0YcrWKJV2wcMVFwhzIRU0BX9iOvbaf/6DSfPmwTMFcOrHHzFp3jy8+vjjDIR0LOQgqFOnTvj0008xePBgAEBWVhbefvtt3Hrrrfj0008VbyARxZg4C0yixX+htUwgFIVgVLZGCYUovH4AAON/3na7HXc/8YRXAAQADgAmAPesWoXrRo9m15hOBT1Evq6uDgCwceNGdOrUye29lJQUvPjii9i1a5eyrSMiUkqMjzrzV2i9dPYp7wVis7Pdh+ADvofkRzCVQeHEz5GSZBe66BKbUIhlQtAl/uhoyoJQvX/ggFsXmCcHgBM//ID3DxyIXqMoJEFngkaNGoUdO3age/fuPre55JJLFGkUEcWJaAYmSo460yFVF4gNZ/RYi6IFZlibEpECC6z2VBRhIQp1UpweqcogA7hgt6PoCzoTNHz4cFx44YU4LC2cA3DgwAFceeWVijeMiDQSzQkMxcAkRrMzhqFSMFpU1gUPV1yHpfmfwLLpVSwd/S4eRhGKRr8DbNok/OTnK3rMaMoJ8noFux1FX9BBUFlZGW6//XZceuml+OCDD/DVV1/hxhtvxPDhw5GamqpmG4niUzSDEc/j6mxmZVKZXDBaVeXebQWE1I3lVqM0vwkYMACFM09jKQrx8K7LUPTBmKBmwtazUUOHonunTvA1JMgEoEfnzhg1dGg0m0UhCKkwetGiRUhJScH48eNht9vxu9/9DuXl5cjNzVWrfUTxK4IuiJii0zW/Yl6Eo8d81ihhGZA/CXZ7RyVbq4nExEQ8ef/9mDRvHkyAW4G0GBitvu8+FkXrWNBBUGVlJZYvX46ysjIMHDgQhw8fxk033cQAiCgW6HhmZbfMlF67zo4cAVatAu67D+jf3/d2RgroIhw9Jluj1NLtVph/GsiOjSHyN4wbh1cff9x7nqDOnbH6vvs4PF7ngg6C+vTpgwEDBuCVV17BVVddhX/961+48cYbcfLkSTz44INqtpEofmgVjOh4ZmWnrVv1Gzx89x2wf7/w6C8I+vpr4ToPHqzP85BSYyqDGC1Ov2HcOFw3ejRnjDagoIOg9evX46abbnI+/93vfof33nsPV199Nf73v//hqaeeUqWBRHFFq2BEbzMrywWDAHD0qKs90WqTktmbM2fcHylmJCYmYszw4Vo3g0IUdBAkDYBEubm52LNnD0eHESlFq2BEbxMYygWDAFDYsq769ddHr2vMV23WkSNC5gcAPvrI/REA+vQRskLSgO7YMeHx0CEhczRuHNC3rzZZoVCCOz+jx+JljbB4Oc94E9Yq8lK9evXChx9+qERbiEhvwYhW8vOFm/TWrfLvb92qfdfKqlVCICO1fbvwAwC5ucDTTwMbNwLPP+++3YsvCo9btwJTpgD33qt+ez2FUnjv51rLrhEG99FhsSBezjPeRBwEAUD79u2V2A0R6YEeZlYWb7r5+UIXmJgBilYXnTR7U17u/ige+777hExQXZ0QzHz9NXDVVcBFFwnb9OmjXvt0RHaNMJklPIwuXs4z3igSBBGRCrQKRrTOsEjboVVWTK477sknXb+LtVn9+wvdlo8/Lrx+0UXAhAnufzdtmvBadTXwzjvAtm3ApZcCH3wAzJgBDBki7CMadU4qFd7LrhEWg4FBvJxnPDE5HA65td9iUm1tLTIzM1GzbRsyunXTujlEsUvJYmJx0sitW4UZhj2DIDWGnUuDhffeA9atEwKWsWOF16TBwuHDwNSpwu9FRd5BkKikRL7OSaRG0bvntVG5Dal5Q4U1wpKbYdkbu+tlxct56kltXR0yx45FTU0NMjIyFNsvM0FEpDwlJ3qUzmYsty8tJpX8+WfvjErPnoDJ5Durk58PJCcDTz0FTJwIvP66+t17ntdGxcL7orIuzsDAaktAUVmXmMyQxMt5xgsGQURaMtLkeVqKdhedXHfYunXCDyAUPXsWRR8/LgQUgHxGJTsbOOP8w9oAACAASURBVO884fd+/YTHaBe9q9TF6FkbIz4HFFi8VUfi5TzjCYMgIi3F0tIYodabRBIAqj2ppDRjUl4u1APdfTcwYoTwmskEiJUEoWRUxEEkbduG37ZAojzhplxxsFwRsdHFy3nGGwZBRKSMUCd6jCQAVHtSSblAYcQI/xkTXxkVuaDkzTddxdJVVcoGwMFeG4UK732uEVZwGqhvgP3jw8DEFMMH+X7Ps+V9Mh4GQUTRpud1uiIhV28CCMXCvXsre05yx5oxQ+iuKipyZWy0Is1yyQUl//2v8PjWW8oXRAdb96NQF6O/CQILf/cxsHEqULXJmN9pCb/nyQyQYTEIIoo2I6zTFQ5fwVvv3q4bsFIBoNx2vXq5jhfMPoLtjguUMZF7X5rlimZwKLaHE24SBYVBEFG06W2dLiWJQY64xhfgHuS89Zb37MmRBIDSY4lLUgQbVAXbHSdmTMSh+p5BU6CMirhtVRWQmup63WJxvS7dzuhiNdNJMYlBEFG0xfK/1ANluaZMEeb6AZQJAKXHE0duqZVVCxQ0+bv5b9nivQRINLJ/Wky4GauZTopJDIKISDnBZLmUDABDzaqpmaUIdPO//nphwdTNm4H33w8++ItkFJ0Ws3/HcqaTYg6DICIt6WGdLiVFO8sV6vHCGcEWbNAUbADYvr0QBAV7XYw2jUIsZzop5jAIItKSXtbp0oKSAaCYLQkk1CxFKEETb/5EhsMgiIjUEcyoKiXrdbZuFbqcAhU5hxKoqNG14+u6SLu9xOfV1UL3mXh8f+ehR7GW6aSYwyCIiNQRzSxXdbXwOG6c+2isSJckCTe74+/m7+u6SLu9du2KjeLieM50kiEwCCIi9ai5Npq0XufgQdejuDRFdbX/Who1sxSR3vzFDNTRo0BhofBaNBZb5Tp2FGcYBBGRekIp6g31JhxokdPrr/e//1ADFTWCJrnC6/Jy92yWSJxjSK2uMKMVYBMpgEEQEelDqDfhMWOAnj2F3z/6CNi+HbjsMtcK7Tab8CgGF0ePRnaTz852LYOhVLZELpB78kn5bcWMUDhdYczyEMliEEREyorWjME7d3oHEO++K/xISWtpIqV0tkSu8Pruu90zQU8+KQR7//d/wtIg4RzXs91iUDRmDOBwuI4vfQSMU4BNFCYGQUSkrFCGlUcSMEkDiPfeE7rBbr4ZGDjQte8nnxQWVu3VS1hWY906fd3kA61WL7b1+HHhHJQabi8GReKoOikjFmAThYlBEBEpK5Rh5ZEssSANIMTRYRdfDOTluY4NuGqEQt2/yKhrYflrt7je2rhxriH5aszuzG440jkGQUSkrFCGlSs1D484Ikx8lBJXaw93/9FaC0taeC0GMNXVwKhRwgzToQZegdoNAD/84LpmnTsLj0pO8Mhia9I5BkFEpB2lZlmWG7klvjZihPvrAwYIz6UTE/oTrbWwpKPVSkoCB16BirSl7S4vly+49lzbjCjOGCoI2r17N1auXImKigpUVlZi69atmDhxotbNIiJfojVjsNxwd39D4EPJUGixHEYwgVegc5C2W+z+uuMO4JJL5PdpMinTtWfU7kOKS4YKgurr63HBBRfgtttuQ34w/4IjIm2FMheP2gGTZ3eTP9GsZZE7VjCBV6BzkNO1q/s+PPfZv3/o+/QUre5DIgUYKgiaMGECJkyYEPT2FosFFovF+by2tlaNZhHph54KUT3XwgrUrmgssTB6tNCuQBkKMcsyeLD3BItqTJgYyoSSgbIsgPfwd/FvxHMXM0NqiFb3IZECDBUEhWr58uVYsmSJ1s0gih49FaJK2wJo365wMhRnzri3W4u1sKSBVzDnMHq07+Hv0rqg3Fz1lgyJdvchUZhiOgiaP38+5s6d63xeW1uLHj16aNgiIp3RU+ZIbYEyFGfOAI88Igwb/+EH4bVjx4THo0eVrWUJpW5GGngFWysEBDf8PdY/c6IAYjoISk1NRaq43g5RrIqkEFXpzJG0LeXlwuN777neF18L1C41BMpQPPKIkDmRZk/EOYYKC4XRU9JusUiEWzfj6xykw+rFz17t4e/BiFZhPFGYYjoIIooLeipE9bWoqUjaHaO3Atlx44QAaOxY98BNJAZISrRb6bqZQN8BrYa/a9F9SBQCBkFERhfqDVXNIcxyc9PMmCE8X7dOWBdrxAj5dkWTmKEwmYC9e4H//Me1GOvgwcK1Ky4WluF48UXlC3uVqJuRZlkCfQeUGv5OFGMMFQSdPXsW33zzjfP50aNHcfDgQXTo0AE9xf+BEcWbUG+oamaO5NoydqzwuG6d+7pYWhIzFHKTEkqzVadPC496LOz1zLIE+g4oMfydKMYYKgjat28fxor/QwWcRc+33nornn32WY1aFUfiqYg2lkVrCLO4nld1tfxyFnqQny9kgAoLhYzVunXCtejcWcgODRsm3zWmpEB1M/zvjkg1hgqCxowZA4c47wVFn56GX5O8YApRleiKCfXG7K9dWtzkpV2CkrnEAACpqUDfvsJCrFVV6hf2BqqbCfW/OxYjEwXNUEEQEQUQrULUYG7M0kVNlVrCQin+CrgLC11dgkYs7DVim4k0wiCI/OM6QLHNZBImzTOZ3F+vqgI2bhR+nzYt+M9Yz98XacZJrkvw7ruB48eFUWJ9+2rTRmlb9Xod1cSuP4oyBkHkn56GX5PyHA5g/37X8gqiqirg+eeF3ydMcJ+HBvB9Yw72+6LFTV6acRLn1pEaMUII+Dz/Roubcrz+d8cud4oyBkHkH9cB0g+t/5UczI3ZV6Gx5/fFKDd5rW7K/O+OKCoYBJF/XAdIP5S6IfvKwlRXAydOCM9//tm1/XvvCctGnHMO8Le/CTU+/pZh6N1beN6rl/Ao932J1k0+2IyT3gqJ4+m/u3jt+iNdYBBEZGThZIcCZWE8SWd89szQiDdm8UYmXbbB37pb0brJB5txkp4Tb8rRZZSsIMUkBkEUPD3+iznWVVUBX3/tPqOx9IZcXR16dshXFqZzZ/dMkBj8zJghZHXatfNdMBzsaKtoE8+1uhrYvBl4//3AGSc1bsqRdGWq+d+d1l2sALv+SFMMgih4HHqrLrkbkhprQvnLwuTlCc8PH3YFMWPHemdoPG/MkdzI1LzJi+d6+LAQAAGBM07Sc3nvPeE6zJjhmvk6nHZG0pWp1H93ct8vPRQix1PXH+kOgyAivZC7IckVGt99t/tNDIh+d414Y66qEpaeyM/3vmkFeyPTW3AtvX5HjwqPvXoZ/6ash4CHSGcYBBHpkb8ZjT/91Hsph3C7a3xlYbKzgSlTXL/7a6deb6xytT2jRgldY4cP+w4WpX8n1jUdO+baR7BBpl5ri8Tz11u7xGOzy52iyOSIo3UoamtrkZmZiZpt25DRrZvWzSHyvlGK3UhffAFs3er7766/Xsi++BulJXcspes/Dh8Gpk4FNm1yHV8PdSaA/OKoUr6CxWD/LtB5hnt8Jcl9v0aNcnUNatUuohDV1tUhc+xY1NTUICMjQ7H9MhNEpKVgan7OP19+RmPpjTeYrie1h9iLfHVv+apJUStgkqtTGjUKuPFG11Ievv5O7IIcOhQ4cEC+vinQ9dRDwa/c90saAIkBEQuRKU4xCCLSUjA3SjHgkJvRWAvhjp6SCxrE16qqXGt1KUUuI/b++8Jx/AWM0rmODhwQHsMp1NVDwW+g71d1tXBNWIhMcYpBEJGWgrlRikGQr7+fMgV46y35falRl6JGhmPr1vCzQUplk+SulUic60jcTrqNXupp5AT6fnmeJ1GcYRBEJKWXehYpf8Wi2dnC2l5Tp7rW+JJSY86bUDIccoFFeblr1JU0wBNfCzWQ8NctJR7/2DGhi+v4cd9Bi9y1EhUWCl2T6emuNdVEga5nVZUQpE6Zop/vlIiFyBTnGAQRSWk52snfSC25G6sYrPmjdV2KXGDx5JPy2xYWCo/XX69c11goQWB+vnBdfRWkb90qBDKbNgnPg72e4mK0mzZpG2zIfb/0Nj0BUZQxCCLSi1BuSF9/Ldzce/Z0DaGXy3CoXZcSqDtOLgi7+275Yf6irVsDX4tgu/nE4x896gqyfAUt2dnApElAbi7w/fdAcbHw+lVXARddJPzepw/Qv797W4xST8OAh8gLgyAivc7n4s9//iM8ijd2QJv1lgJ1x8lduxEjhG1nzAguOJET7Ir2cnMtpab6btvOnd773b5d+BH36xkEyTHid4ooDjEIIjLKAo7SG6u4jtiMGcKjOJP0iBHCc1/1Q3qp/4g0QxVMN5+vGh8x6JL7XKVrjT3zjDA6zNeSGf6up1G+U0RxjkEQkdZ1M8Hyt0gpIBT9+htCr2R3SDiZDl81Kddf739iSDnBBFHhzBPkud+77gKGDJEPzvxdT6N8p4jiHIMgIj3M5xIMXzfW1FQhuzFuXPTaEk6mQy5oEF9To3tIbp+B5giSat/e/THSY+vxO0UU5xgEERmFrxurmGHp2zd6bVEy0+EvoxLMlAVKdPPJHUdP3YdEpAoGQURSRrzxaTHqJ1qZjmCmLAjm/AN9rnLHUeq6GvE7RRQnErRuABGFgTfW0Ei73fRy7KoqYZFVfzOCE5GqmAkiktJyssRQ6GnOF6UDsmgNL9d6GLtRvmtEMYxBEJFeRXvV9XApHZBFa3g5h7ETxT0GQURaZwT8tcvXquuxnD0IZZZnJY4DRG8Yu16/a0RxikEQETMC+sowRavoWoth7PyuEekKgyAiPU1sF+yq60pnD/SSYZI7fyD8Feb1Rk/fNSJiEESkq4ntgl11PVazB+EsdaEEteYakjuOXr5resr+EWmEQRCRnvhadV1aE/Tkk8pkD4KtT4nmzVLu/GfMEJYHKSpyrY2mNCWKu/WSTQuW0dpLpAIGQURSWs+/42vVdTHgEYMUJbIHwdanRPNmKXf+vXoJj717x9bNWuvvGhExCCJyo6f5d9Sm5/qUqirg00+F3w8dEh71OIoqktFeWnzXODqNyA2DICK98rXqulLZA3/1KVVVwNdfCzMa9+wpvKfEzTLYrjVplurFF4VHPdZBGW20l9HaS6QyBkFEeuVv1XW1qXWzDLZrLT9fCL4KC101QXrJUknpOZsmx2jtJVIZgyCieOEvC+OZYVIrCKmudn+Ua6PYXWOxuL+Xmqp8d02kRd96Gu0VDKO1l0hlXECVjI8LUQZHzMLIXSdphunwYWEbX0HIgAGhBQxVVcI+Dx8GDh4UXjt40PWatD1btgBTpwo/YuZp3TrhsbBQeF+630g/d3/XhIhiHjNBZHyxNNT3yBFg1SrgvvuA/v2jf3y5bjBpEBJON5ivfYr7le7T1xQBx48D48YBffu69qG3z91oo72M1l4iFTAIIgqH2I0yZgywc6dyc+h89x2wf7/wqEQQFOpooFCCkGCNGeMqrv7oI2D7duCqq4CLLhJe69PH1Va5rqkRI4Bp00I/ri9qjZAy2shCo7WXSAUMgsiYtB7qK2YhevbUVzbCU6gFzr7mKYokCNm507sN27cLP2Ib+vcXRqOVlgKDB/u+lkp87hwhRUQtGASRMcXSjezIESHzAwiZEukjIGRKws0KSTM7770ndEHNmAGMHSu8pnagKGbLgmnDmTPuj3LdNUp87hwhRUQtGASRMWlxIxOzENXVrgJfMVh57z3h9fbtQ89CrVoldIFJSTMlubnA00+H12ZpW8RFSHv18h4NJNcVFWnNiLRmRzyeOCpsyBDXfETiz7FjwnvHjrmyO55dY+LnfvSoaz2xUD93jpAiohYMgsiYtLiRyWUhxEBFLPIFQs9C3XefeybIV81MOKTdR3JBhnR9MM9uPTVqRtq3d3/cuBF4/nn3baRF01OmAPfe63qPAQwRKYhBEFGwxCyEmAlat04IVrZvF7p3hgxxZYJC0b+/e3fX9u1CADRhQuRtDmVklhIC1eyYTOFnl+T2DbgyXOHUgXGEFFFcYxBExhetG5n0Jtu+vRBIXHSRELSMHRt6NiIaq7P76zYUu6bE+XrE30XhBBWh1uxMm+YK9gLVLMntG3B1iwUT0Hlec46QIoprDILI+Ix6I5PrgurTR6gBiqQLTMpf91FJifLF5aHWagVbsxTOvuXobW4hItIUgyCicIjZpz59lM1C9e8ffhF0qNQoLo+kZqddO/dHJfdNRCSDQRBROKTZp1CGr2s5v5Fnt6Hegoq+fYX2hTMhoz/BXHNA/a5JItIdBkFE0aTl/EbR7jYMtVYrlPaFsu9grvno0ewmI4pDDIKIoimSLig1C6nVKC5XM+gKZd9y1xwAioqA3r1dUwQQUdxhEEQUTZF0QalZ1GvU4vJg+OpiFGuPxFXugeguvUJEmmMQRETREY0pAfwdu6rKNQINADZvBt5/3307oy69QkRhYRBEpJVguqC0XihWSVoOT5erC5IGQKNGCc+5hhhRXGEQRKSVYLqgYmmhWC0FqsWqrhaCIA65J4orDIKI9MzoK57rJZMlHQY/eLDwuzTgkbaJiOIGgyAiPdPbXD6h0lMmS+yOKyryfo9riBHFJQZBZDxKFNhqWaQbT/SYyWrXzjvgieXRcUTkE4MgMh4lCmyNuIaU0tmKaASCWmey5LrjfvhB+NzF143y+ROR4hgEERmF0tkKIwaCodJTdxwR6Q6DIDIGJQps9VKkazRKZYxCyWQpdUw9dscRkW4wCCJjUOJf9MwKhBcIKpUxCiWTpeQxjVxYTkSqYhBExqDEv+jD2UesFVAzECQicmIQRMagxL/ow9lHrNXNBBsIKtV1GEoQqXZ3JYfBE5EHQwZBTz31FFauXInKykqcf/75WL16NUaNGqV1s4j0L9hAUKmMUShBpNpZKg6DJyIPhguCXn75Zdxzzz146qmncMkll6CkpAQTJkzAoUOH0LNnT62bR8GItItJiX/R+9sHC6i1KShmETMRRZnhgqA///nPmDFjBgoKCgAAq1evxr/+9S8UFxdj+fLlbttaLBZYLBbn89ra2qi2lXyItItJiX/R+9tHvNTN+AsEI+l+DDeIZBEzEUWZoYIgq9WKiooK/OlPf3J7/fLLL8eePXu8tl++fDmWLFkSreZRrIiXjIRa3UPxEkQSkeEZKgiqqqqC3W5H586d3V7v3LkzTp8+7bX9/PnzMXfuXOfz2tpa9OjRQ/V2kgwjdTExI+Eu1O5HJYJIFjETURQYKggSmUwmt+cOh8PrNQBITU1FampqtJpF/jA7YFyhZoyUGsnH7wMRqcxQQVB2djYSExO9sj4//vijV3aIdMaoc/QokZHQw3kYEa8bEaksQesGhCIlJQXDhg3D22+/7fb622+/jYsvvlijVlFQsrNd2QAx8JE+9xUElZa6utG0IGYkIg2CtD4PrUQSRMbzdSOiqDBUJggA5s6di2nTpmH48OHIy8vD008/jePHj2P27NlaN42IPLFbi4h0zHBB0OTJk/Hzzz9j6dKlqKysxKBBg/DPf/4T55xzjtZNo2DFwxw9sXIe0cbrRkRRZHI4HA6tGxEttbW1yMzMRM22bcjo1k3r5pCckhLvAmopoxRQx8p5RBuvGxHJqK2rQ+bYsaipqcH/t3fnsVGVXRzHT+kqUgZqgRaRRSRtKqB00Zat1cYWEFwSQZBMqkaixIpIjCkQY/sHoRqDQVHAQICoESOlakJESGwLSIECU3YQtVA0VAShhRpalvP+4ctI6TrDLJ37fD/JJMyd5w7nzMMDP+7ce9u9e3ePvW/AHQmCxVnlHj1W6cPX+NwA+BAhCJ2LVe7RY5U+fI3PDYAPBdTVYQA6gbNn//3aiqu2AAQ4QhA6L6vcNdgqfdzgq0vXrfa5Aeh0+DoMnZdVLq+2Sh++xucGwMsIQQDax6XrACyIEASgffzsNwAWRAgC0D4uXQdgQYQgAO3j0nUAFsTVYQAAwEiEIACu4dJ1ABbB12EAXMOl6wAsgiNBAP7D3aABGIQQBOA/vrobNAB0AoQgAABgJM4JAkzH3aABGIoQBJiOu0EDMBQhCDAdd4MGYChCEGA67gYNwFCcGA0AAIxECALwH+4GDcAgfB0G4D/cDRqAQTgSBAAAjEQIAgAARiIEAQAAIxGCAACAkQhBAADASIQgAABgJEIQAAAwEiEIAAAYiRAEAACMRAgCAABGIgQBAAAjEYIAAICRCEEAAMBIhCAAAGAkQhAAADASIQgAABiJEAQAAIxECAIAAEYiBAEAACMRggAAgJEIQQAAwEiEIAAAYCRCEAAAMBIhCAAAGIkQBAAAjEQIAgAARiIEAQAAIxGCAACAkQhBAADASIQgAABgJEIQAAAwEiEIAAAYiRAEAACMRAgCAABGIgQBAAAjEYIAAICRCEEAAMBIhCAAAGAkQhAAADASIQgAABiJEAQAAIwUMCFowYIFMnLkSOnatav06NHD3+UAAIAAFzAhqLGxUSZPniwzZ870dykAAMACQvxdQEcVFBSIiMjq1as7vE9DQ4M0NDQ4n9fW1oqISN0//4hcuuTR+gAAgHfU1deLiIiqevR9AyYEuWPhwoXO8HSze6ZN80M1AADgdpw7d05sNpvH3s/SIWju3LkyZ84c5/MLFy7IgAEDpLq62qMfYmdXV1cn99xzj5w6dUq6d+/u73J8hr7p2wT0Td8mqK2tlf79+0tUVJRH39evISg/P7/FIzU3q6iokOTkZLfePzw8XMLDw5ttt9lsRv3huaF79+70bRD6Ngt9m8XUvrt08eypzH4NQbm5uTJ16tQ2xwwcONA3xQAAAKP4NQRFR0dLdHS0P0sAAACGCs7Pz8/3dxEdUV1dLVVVVbJr1y7Ztm2bTJgwQWpqaqRbt24SFhbW4fcJDg6WjIwMCQmx9OlQzdA3fZuAvunbBPTtub6D1NPXm3nJ888/L2vWrGm2vaSkRDIyMnxfEAAACGgBE4IAAAA8KWDuGA0AAOBJhCAAAGAkQhAAADASIQgAABjJ8iFowYIFMnLkSOnatav06NGjQ/uoquTn50vfvn3ljjvukIyMDDl06JCXK/Ws8+fPi91uF5vNJjabTex2u1y4cKHNfTIyMiQoKKjJo72bWfrbJ598IoMGDZKIiAhJSkqSrVu3tjm+qKhIEhISJDw8XBISEqS4uNhHlXqWK32vXr262bwGBQXJ5cuXfVjx7duyZYtMmjRJ+vbtK0FBQfLNN9+0u09ZWZkkJSVJRESE3HvvvbJs2TIfVOpZrvZdWlra4nwfPXrURxXfvoULF0pKSopERkZK79695amnnpJjx461u1+gr293+rbC+l66dKkMHz7ceRfstLQ0+f7779vcx1NzbfkQ1NjYKJMnT5aZM2d2eJ/33ntPFi1aJEuWLJGKigqJiYmRxx57TC5evOjFSj3rueeek8rKStm4caNs3LhRKisrxW63t7vfjBkz5PTp087H8uXLfVCte7766iuZPXu2zJ8/XxwOh4wZM0bGjx8v1dXVLY4vLy+XZ599Vux2u+zbt0/sdrtMmTJFdu7c6ePKb4+rfYv8e4v9m+f19OnTEhER4cOqb199fb088MADsmTJkg6Nr6qqkgkTJsiYMWPE4XDIvHnzZNasWVJUVOTlSj3L1b5vOHbsWJP5HjJkiJcq9LyysjJ59dVXZceOHbJ582a5evWqZGVlSf3/f5J4S6ywvt3pWyTw13e/fv2ksLBQdu/eLbt375ZHH31UnnzyyVYPPnh0rtUQq1atUpvN1u6469eva0xMjBYWFjq3Xb58WW02my5btsybJXrM4cOHVUR0x44dzm3l5eUqInr06NFW90tPT9fXX3/dFyV6xEMPPaSvvPJKk23x8fGal5fX4vgpU6bouHHjmmzLzs7WqVOneq1Gb3C1747+2Q8kIqLFxcVtjnnrrbc0Pj6+ybaXX35ZU1NTvVmaV3Wk75KSEhURPX/+vI+q8r4zZ86oiGhZWVmrY6yyvm/Wkb6tuL5VVXv27KkrVqxo8TVPzrXljwS5qqqqSmpqaiQrK8u5LTw8XNLT02X79u1+rKzjysvLxWazycMPP+zclpqaKjabrd0evvjiC4mOjpb7779f3nzzzU579KuxsVH27NnTZJ5ERLKyslrtsby8vNn47OzsgJlXEff6FhG5dOmSDBgwQPr16ycTJ04Uh8Ph7VL9rrX53r17t1y5csVPVfnOiBEjJDY2VjIzM6WkpMTf5dyW2tpaEZE2f4K4Fdb3rTrSt4i11ve1a9dk7dq1Ul9fL2lpaS2O8eRcm3XP7Q6oqakREZE+ffo02d6nTx85efKkP0pyWU1NjfTu3bvZ9t69ezv7a8n06dNl0KBBEhMTIwcPHpS5c+fKvn37ZPPmzd4s1y1nz56Va9eutThPrfVYU1Pj0vjOyJ2+4+PjZfXq1TJs2DCpq6uTxYsXy6hRo2Tfvn0B9RWJq1qb76tXr8rZs2clNjbWT5V5V2xsrHz66aeSlJQkDQ0N8tlnn0lmZqaUlpbK2LFj/V2ey1RV5syZI6NHj5ahQ4e2Os4K6/tmHe3bKuv7wIEDkpaWJpcvX5Zu3bpJcXGxJCQktDjWk3MdkCEoPz9fCgoK2hxTUVEhycnJbv8eQUFBTZ6rarNtvtbRvkWa1y/Sfg8zZsxw/nro0KEyZMgQSU5Olr1790piYqKbVXuXq/PUGefVHa70kZqaKqmpqc7no0aNksTERPnoo4/kww8/9Gqd/tbS59TSdiuJi4uTuLg45/O0tDQ5deqUvP/++wEZgnJzc2X//v2ybdu2dsdaZX2LdLxvq6zvuLg4qayslAsXLkhRUZHk5ORIWVlZq0HIU3MdkCEoNze33auWBg4c6NZ7x8TEiMi/SfPm/ymeOXOmWfL0tY72vX//fvnzzz+bvfbXX3+51ENiYqKEhobK8ePHO10Iio6OluDg4GbJv615iomJcWl8Z+RO37fq0qWLpKSkyPHjx71RYqfR2nyHhITIXXfd5aeq/CM1NVU+//xzf5fhstdee02+++472bJli/Tr16/NsVZY0/4XqAAABWNJREFU3ze40vetAnV9h4WFyX333SciIsnJyVJRUSGLFy9u8eIcT851QJ4TFB0dLfHx8W0+3D0z/sbXQTd/BdTY2ChlZWUycuRIT7Xglo72nZaWJrW1tbJr1y7nvjt37pTa2lqXejh06JBcuXKlU35tEBYWJklJSc2+qtu8eXOrPaalpTUbv2nTJr/Pqyvc6ftWqiqVlZWdcl49qbX5Tk5OltDQUD9V5R8OhyOg5ltVJTc3V9avXy8//vijDBo0qN19rLC+3em7pfewwvpWVWloaGjxNY/OtcunUgeYkydPqsPh0IKCAu3WrZs6HA51OBx68eJF55i4uDhdv36983lhYaHabDZdv369HjhwQKdNm6axsbFaV1fnjxbcMm7cOB0+fLiWl5dreXm5Dhs2TCdOnOh8/ffff9e4uDjduXOnqqr+8ssvWlBQoBUVFVpVVaUbNmzQ+Ph4HTFihF69etVfbbRp7dq1GhoaqitXrtTDhw/r7Nmz9c4779QTJ06oqqrdbm9yxdRPP/2kwcHBWlhYqEeOHNHCwkINCQlpchVdIHC17/z8fN24caP++uuv6nA49IUXXtCQkBDn3AeKixcvOteviOiiRYvU4XDoyZMnVVU1Ly9P7Xa7c/xvv/2mXbt21TfeeEMPHz6sK1eu1NDQUF23bp2/WnCLq31/8MEHWlxcrD///LMePHhQ8/LyVES0qKjIXy24bObMmWqz2bS0tFRPnz7tfPzzzz/OMVZc3+70bYX1PXfuXN2yZYtWVVXp/v37dd68edqlSxfdtGmTqnp3ri0fgnJyclREmj1KSkqcY0REV61a5Xx+/fp1feeddzQmJkbDw8N17NixeuDAAd8XfxvOnTun06dP18jISI2MjNTp06c3uWS2qqqqyedQXV2tY8eO1aioKA0LC9PBgwfrrFmz9Ny5c37qoGM+/vhjHTBggIaFhWliYmKTS0nT09M1Jyenyfivv/5a4+LiNDQ0VOPj4wPqH4abudL37NmztX///hoWFqa9evXSrKws3b59ux+qvj03Lv2+9XGj15ycHE1PT2+yT2lpqY4YMULDwsJ04MCBunTpUt8Xfptc7fvdd9/VwYMHa0REhPbs2VNHjx6tGzZs8E/xbmqp31v/nrbi+nanbyus7xdffNH591mvXr00MzPTGYBUvTvXQar/P1MQAADAIAF5ThAAAMDtIgQBAAAjEYIAAICRCEEAAMBIhCAAAGAkQhAAADASIQgAABiJEAQAAIxECAIAAEYiBAEIOF9++aVERETIH3/84dz20ksvyfDhw6W2ttaPlQEIJPzYDAABR1XlwQcflDFjxsiSJUukoKBAVqxYITt27JC7777b3+UBCBAh/i4AAFwVFBQkCxYskGeeeUb69u0rixcvlq1btzoD0NNPPy2lpaWSmZkp69at83O1ADorjgQBCFiJiYly6NAh2bRpk6Snpzu3l5SUyKVLl2TNmjWEIACt4pwgAAHphx9+kKNHj8q1a9ekT58+TV575JFHJDIy0k+VAQgUhCAAAWfv3r0yefJkWb58uWRnZ8vbb7/t75IABCDOCQIQUE6cOCGPP/645OXlid1ul4SEBElJSZE9e/ZIUlKSv8sDEEA4EgQgYPz9998yfvx4eeKJJ2TevHkiIpKUlCSTJk2S+fPn+7k6AIGGI0EAAkZUVJQcOXKk2fZvv/3WD9UACHRcHQbAcrKzs2Xv3r1SX18vUVFRUlxcLCkpKf4uC0AnQwgCAABG4pwgAABgJEIQAAAwEiEIAAAYiRAEAACMRAgCAABGIgQBAAAjEYIAAICRCEEAAMBIhCAAAGAkQhAAADASIQgAABjpf+E5dTP+mVcCAAAAAElFTkSuQmCC",
"text/plain": [
"Figure(PyObject )"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"p(apple|x=x∙) = 0.7842698441755891\n"
]
}
],
"source": [
"# Make sure you run the data-generating code cell first\n",
"\n",
"# Multinomial (in this case binomial) density estimation\n",
"p_apple_est = sum(y.==true) / length(y)\n",
"π_hat = [p_apple_est; 1-p_apple_est]\n",
"\n",
"# Estimate class-conditional multivariate Gaussian densities\n",
"d1 = fit_mle(FullNormal, X_apples') # MLE density estimation d1 = N(μ₁, Σ₁)\n",
"d2 = fit_mle(FullNormal, X_peaches') # MLE density estimation d2 = N(μ₂, Σ₂)\n",
"Σ = π_hat[1]*cov(d1) + π_hat[2]*cov(d2) # Combine Σ₁ and Σ₂ into Σ\n",
"conditionals = [MvNormal(mean(d1), Σ); MvNormal(mean(d2), Σ)] # p(x|C)\n",
"\n",
"# Calculate posterior class probability of x∙ (prediction)\n",
"function predict_class(k, X) # calculate p(Ck|X)\n",
" norm = π_hat[1]*pdf(conditionals[1],X) + π_hat[2]*pdf(conditionals[2],X)\n",
" return π_hat[k]*pdf(conditionals[k], X) ./ norm\n",
"end\n",
"println(\"p(apple|x=x∙) = $(predict_class(1,x_test))\")\n",
"\n",
"# Discrimination boundary of the posterior (p(apple|x;D) = p(peach|x;D) = 0.5)\n",
"β(k) = inv(Σ)*mean(conditionals[k])\n",
"γ(k) = -0.5 * mean(conditionals[k])' * inv(Σ) * mean(conditionals[k]) + log(π_hat[k])\n",
"function discriminant_x2(x1)\n",
" # Solve discriminant equation for x2\n",
" β12 = β(1) .- β(2)\n",
" γ12 = (γ(1) .- γ(2))[1,1]\n",
" return -1*(β12[1]*x1 .+ γ12) ./ β12[2]\n",
"end\n",
"\n",
"plot_fruit_dataset() # Plot dataset\n",
"x1 = range(-1,length=10,stop=3)\n",
"plot(x1, discriminant_x2(x1), \"k-\") # Plot discrimination boundary\n",
"fill_between(x1, -1, discriminant_x2(x1), color=\"r\", alpha=0.2)\n",
"fill_between(x1, discriminant_x2(x1), 4, color=\"b\", alpha=0.2);"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Recap Generative Classification\n",
"\n",
"- Model speccification: $p(x,\\mathcal{C}_k|\\,\\theta) = \\pi_k \\cdot \\mathcal{N}(x|\\mu_k,\\Sigma)$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- If the class-conditional distributions are Gaussian with equal covariance matrices across classes ($\\Sigma_k = \\Sigma$), then\n",
" the discriminant functions are hyperplanes in feature space."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- ML estimation for $\\{\\pi_k,\\mu_k,\\Sigma\\}$ breaks down to simple density estimation for Gaussian and multinomial."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- Posterior class probability is a softmax function\n",
"$$ p(\\mathcal{C}_k|x,\\theta ) \\propto \\exp\\{\\beta_k^T x + \\gamma_k\\}$$\n",
"where $\\beta _k= \\Sigma^{-1} \\mu_k$ and $\\gamma_k=- \\frac{1}{2} \\mu_k^T \\Sigma^{-1} \\mu_k + \\log \\pi_k$."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"open(\"../../styles/aipstyle.html\") do f\n",
" display(\"text/html\", read(f,String))\n",
"end"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"anaconda-cloud": {},
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Julia 1.1.0",
"language": "julia",
"name": "julia-1.1"
},
"language_info": {
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
"version": "1.1.0"
}
},
"nbformat": 4,
"nbformat_minor": 1
}