{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Article outline\n", "\n", "1. [Ensembles](#1.-Ensembles)\n", "2. [Bootstrapping](#2.-Bootstrapping)\n", "3. [Bagging](#3.-Bagging)\n", "4. [Out-of-bag error](#4.-Out-of-bag-error)\n", "5. [Demo assignment](#5.-Demo-assignment)\n", "6. [Useful resources](#6.-Useful-resources)\n", "\n", "$\\DeclareMathOperator{\\Var}{Var}$\n", "$\\DeclareMathOperator{\\Cov}{Cov}$\n", "$\\DeclareMathOperator{\\Corr}{Corr}$\n", "$\\DeclareMathOperator{\\Err}{Err}$\n", "$\\DeclareMathOperator{\\Bias}{Bias}$\n", "$\\DeclareMathOperator{\\E}{\\mathbb{E}}$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In previous articles, you explored different classification algorithms as well as techniques that can be used to properly validate and evaluate the quality of your models.\n", "\n", "Now, suppose that you have chosen the best possible model for a particular problem and are struggling to further improve its accuracy. In this case, you would need to apply some more advanced machine learning techniques that are collectively referred to as *ensembles*.\n", "\n", "An *ensemble* is a set of elements that collectively contribute to a whole. A familiar example is a musical ensemble, which blends the sounds of several musical instruments to create harmony, or architectural ensembles, which are a set of buildings designed as a unit. In ensembles, the (whole) harmonious outcome is more important than the performance of any individual part." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Ensembles\n", "\n", "[Condorcet's jury theorem](https://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem) (1784) is about an ensemble in some sense. It states that, if each member of the jury makes an independent judgement and the probability of the correct decision by each juror is more than 0.5, then the probability of the correct decision by the whole jury increases with the total number of jurors and tends to one. On the other hand, if the probability of being right is less than 0.5 for each juror, then the probability of the correct decision by the whole jury decreases with the number of jurors and tends to zero. \n", "\n", "Let's write an analytic expression for this theorem:\n", "\n", "- $\\large N$ is the total number of jurors;\n", "- $\\large m$ is a minimal number of jurors that would make a majority, that is $\\large m = floor(N/2) + 1$;\n", "- $\\large {N \\choose i}$ is the number of $\\large i$-combinations from a set with $\\large N$ elements.\n", "- $\\large p$ is the probability of the correct decision by a juror;\n", "- $\\large \\mu$ is the probability of the correct decision by the whole jury.\n", "\n", "Then:\n", "\n", "$$\\large \\mu = \\sum_{i=m}^{N}{N\\choose i}p^i(1-p)^{N-i}$$\n", "\n", "It can be seen that if $\\large p > 0.5$, then $\\large \\mu > p$. In addition, if $\\large N \\rightarrow \\infty$, then $\\large \\mu \\rightarrow 1$.\n", "\n", "Let's look at another example of ensembles: an observation known as [Wisdom of the crowd](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd). In 1906, [Francis Galton](https://en.wikipedia.org/wiki/Francis_Galton) visited a country fair in Plymouth where he saw a contest being held for farmers. 800 participants tried to estimate the weight of a slaughtered bull. The real weight of the bull was 1198 pounds. Although none of the farmers could guess the exact weight of the animal, the average of their predictions was 1197 pounds.\n", "\n", "\n", "A similar idea for error reduction was adopted in the field of Machine Learning." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Bootstrapping\n", "\n", "*Bagging* (also known as [Bootstrap aggregation](https://en.wikipedia.org/wiki/Bootstrap_aggregating)) is one of the first and most basic ensemble techniques. It was proposed by [Leo Breiman](https://en.wikipedia.org/wiki/Leo_Breiman) in 1994. Bagging is based on the statistical method of [bootstrapping](https://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29), which makes the evaluation of many statistics of complex models feasible.\n", "\n", "The bootstrap method goes as follows. Let there be a sample $\\large X$ of size $\\large N$. We can make a new sample from the original sample by drawing $\\large N$ elements from the latter randomly and uniformly, with replacement. In other words, we select a random element from the original sample of size $\\large N$ and do this $\\large N$ times. All elements are equally likely to be selected, thus each element is drawn with the equal probability $\\large \\frac{1}{N}$.\n", "\n", "Let's say we are drawing balls from a bag one at a time. At each step, the selected ball is put back into the bag so that the next selection is made equiprobably i.e. from the same number of balls $\\large N$. Note that, because we put the balls back, there may be duplicates in the new sample. Let's call this new sample $\\large X_1$.\n", "\n", "By repeating this procedure $\\large M$ times, we create $\\large M$ *bootstrap samples* $\\large X_1, \\dots, X_M$. In the end, we have a sufficient number of samples and can compute various statistics of the original distribution.\n", "\n", "![image](../../img/bootstrap_eng.png)\n", "\n", "For our example, we'll use the familiar telecom_churn dataset. Previously, when we discussed feature importance, we saw that one of the most important features in this dataset is the number of calls to customer service. Let's visualize the data and look at the distribution of this feature." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAESCAYAAAAv0qjVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHFdJREFUeJzt3XmYHVWd//H37e5skCaQplmykAiYLxhRHoQhDMiggsIok0E2I0kQZpAoiwqiYjCgEoYt4mjwIT8lmAGMRLYYZX6DDCLDJmCIIwhfUEJWME0nkDR00um+d/6o6qQ76eXe2/fcusvn9Tw86a5zq+pbRdKfPlWnTqUymQwiIiIh1CRdgIiIVC6FjIiIBKOQERGRYBQyIiISjEJGRESCUciIiEgwChkREQlGISMiIsEoZEREJBiFjIiIBKOQERGRYOqSLiAhQ4AjgNeBjoRrEREpF7XAvsAzwJZsVqjWkDkC+J+kixARKVMfBh7L5oPVGjKvA2zY8A7pdO6zUDc0DKe5uaXgRZUrnY/tdC660/nYrhLORU1Nij322BXin6HZqNaQ6QBIpzN5hUznurKdzsd2Ohfd6XxsV0HnIuvbDLrxLyIiwShkREQkmGq9XCYiVSyTybBhQxNtbZuB4lzCWreuhnQ6XZR9DUyKwYOHsscejaRSqQFvTSEjIlWnpeVtUqkUe+89hlSqOBd06upqaG8v/ZDJZNK89dabtLS8TX397gPeni6XiUjVaW1tob5+96IFTDlJpWqor9+D1tbCjITTGRaRqpNOd1Bbqws5vamtrSOdLsxz6goZEalKhbjfUKkKeW4U5SWufrdhDB1SnP9Nm7e0s2lja1H2JVJqQv1by/bf1Wmnncz119/E/vsfWPAairmPHSlkStzQIXWcfOniouxryZzJbCrKnkRKT6h/a9X+70qXy0REStSLL77A+eefw9lnf4bzzz+HF198AYAbb7yWn/3s9m2fe/nll5gy5dNkMhkefPD/c955Z3POOZ/lnHM+y7PPPp1U+YB6MiIiJWnr1q3MnPk1Lr98FkcccSTPPvs0M2d+jbvuup/TTjuTr3/9K0yZMpVUKsU99yzilFNOJ5VKceSRkzjhhE+QSqVYufI1vvSlL3LffQ8kdhwKGRGRErRy5QoGDRrEEUccCcDhh/8dgwYNYuXKFRxwwIGMGjWap556gokTD+Hxxx/loosuAWDNmtVcddVMmpqaqKurY/36Zpqb36ShYc9EjkMhIyJSgjKZTI+jvDoXnXbaZ7jvvrt57bXlHHvsRxg+fDgAV101kwsv/ArHHnsc6XSa448/hra2tmKW3o3uyYiIlKBx48bT1tbG0qXPArB06bO0t7czduw4AI466mhWrlzBXXfdySmnnL5tvZaWFvbddxQAv/rV4kQDBtSTEREpGV/+8gXU1tZu+/6aa27g+9+/kc2bWxk6dBhXX30dgwYNAqCmpoaTTvokTz31BO9974Rt61x88SV885tfZc89Gzn00MMYMWJE0Y+jq6KEjJndCJwKjAcOcffn4+UTgAVAA9AMTHf3VwbSJiKSj81b2lkyZ3KQ7Wbj7ruX9Lh83rzbel3nuef+wOmnT+m27MQTP8mJJ35y2/fnn39Bv/sIqVg9mfuBf2fnVx7fAtzs7neY2VRgHvDRAbaJiORs08bWsnme5aWX/sysWZczYYJx3HGl/aOvKCHj7o8BmNm2ZWa2F3AYcEK8aCEw18wagVQ+be7eFPhQREQSd9BB72PRouI8pD1QSd74HwuscfcOgPjPtfHyfNtERKSEVPWN/4aG4Xmv29hYX8BKSke+x1Wp5yMfOhfdleL5WLeuhrq64v+OncQ+81VTU1OQ/3dJhswqYLSZ1bp7h5nVAqPi5ak823LS3NxCOp37W/EaG+tpairO1dti/wPN57iKeT5Knc5Fd6V6PtLpdNFfIFYuLy3rlE6nd/p/V1OTyvmX88Ri1d3XAcuAzqERU4Dn3L0p37biVS8iItko1hDmHwCfBvYBHjKzZnefCMwAFpjZLGADML3Lavm2iYhIiSjW6LKLgYt7WP4ScGQv6+TVJiKSjz1GDKZu8JCCb7e9bQsb3u7/qfv29nZ++tOf8NBDD1JXV0cmk2bSpKMZN248Tz/9JFdffX3BayuGqr7xLyLSqW7wEF6dfWrBt7v/zHuA/kPmmmu+zZYtm5k//3Z22WVX2tvb+fWvf8nWrQOfFiadTpNKpRJ5G6hCRkQkYatWreTRR3/Lvfc+wC677ApAXV0dkyd/mgceWMI777zDrFmX8+qrf6W+fjhXX309DQ17cuut82htbeXCC78M0O37W2+dx5o1q2ltfZc1a1Yzd+6POffcszjxxE/yzDO/p7n5TaZMmcqpp54Z9NgUMiIiCXv5ZWfMmP3Ybbfdemx/8cU/s2DBQvbeex+uu+5q7r77rm7TxfRm2bKlzJ9/J7vvvvu2ZZs3b2bevNt4/fW1TJ9+JieddDK77LJLwY5lR+UzaFtEpGL1/SjFBz7wQfbeex8AJk58P2vXrs5qq0cddXS3gAE4/viPA7DvvqOor9+NpqZ1edSbPYWMiEjCJkw4iNWrV7Jx48Ye2wcPHrzt65qaWjo6OgCora0lk9n+7M2O0/oPG7ZzD6X7tmro6MhuAs98KWRERBI2dux+HH30sdxwwzW8++47AHR0dLBo0UJaW9/tdb3Ro8fg/hLpdJp3332HJ57YcQ7i5OmejIgI0VDjaCRY4bebjSuu+Dbz5/8/zj13GoMG1ZHJZJg06Wj2229cr+scd9zHePjhh5g27QxGjx6L2cGFKrtgUplM7tOqVIDxwPJymVbm5EuLM9vqkjmTNa3MAOlcdFeq5+ONN1awzz69//AOodymlenpHHWZVuY9wGvZbEeXy0REJBiFjIiIBKOQEZGqVKW3CrJSyHOjkBGRqhMNAw47dLecdXS0U1NTW5BtKWREpOoMGzacTZve6vaMiUQymTSbNm1g2LD8X+rYlYYwi0jVGT58BBs2NPG3v62mv6ftC6WmpoZ0uhxCLcXgwUMZPnxEQbamkBGRqpNKpRg5cq+i7rNUh3OHpstlIiISjEJGRESCUciIiEgwChkREQlGISMiIsEoZEREJBiFjIiIBKOQERGRYBQyIiISjEJGRESCUciIiEgwChkREQlGISMiIsEoZEREJBiFjIiIBFMS75Mxs08B3wVSRMF3lbvfa2YTgAVAA9AMTHf3V+J1em0TEZHSkHhPxsxSwO3ANHc/FJgKLDCzGuAW4GZ3nwDcDMzrsmpfbSIiUgISD5lYGuh81+fuwOvAnsBhwMJ4+ULgMDNrNLO9emsrXskiItKfxEPG3TPAGcBiM1sB3A+cDYwF1rh7R/y5DmBtvLyvNhERKRGJ35MxszrgcmCyuz9uZkcDdwHTQu+7oWF43us2NtYXsJLSke9xVer5yIfORXc6H9tV47lIPGSAQ4FR7v44QBw07wCbgdFmVuvuHWZWC4wCVhENEOitLWvNzS2k05mcC25srKepaVPO6+Wj2H8p8zmuYp6PUqdz0Z3Ox3aVcC5qalI5/3Ke+OUyYDUwxswMwMwOBvYBXgGWAVPiz00BnnP3Jndf11tbUSsXEZE+JR4y7v4G8AXgbjP7I/Bz4Bx3Xw/MAC4ys5eBi+LvO/XVJiIiJaAULpfh7ncCd/aw/CXgyF7W6bVNRERKQ+I9GRERqVwKGRERCUYhIyIiwShkREQkGIWMiIgEo5AREZFgFDIiIhKMQkZERIJRyIiISDAKGRERCUYhIyIiwShkREQkGIWMiIgEo5AREZFgFDIiIhKMQkZERIJRyIiISDAKGRERCUYhIyIiwShkREQkmLqkC5DS0ba1g8bG+rzWzXW9zVva2bSxNa99iUj5UMjINoMH1XLypYuLsq8lcyazqSh7EpEk6XKZiIgEk3XImNk/mZl6PiIikrVcejLfBV43s7lmdmSogkREpHJkHTLu/kHgeKAVuMfM3MyuMLPxoYoTEZHyltM9GXf/o7tfBowFLgBOB/5qZo+a2Vlmpns8IiKyTc73WMzsAGBq/F8amAWsBC4ETgU+XcgCRUSkfGUdMmZ2ATANOBBYBExz96e6tN8DrCt4hSIiUrZy6cmcBMwBFrt7246N7v6umakXIyIi2+QSMo+4+y92XGhml7j79wDc/cF8ijCzocBNRAMLNgNPuvvnzWwCsABoAJqB6e7+SrxOr20iIlIacrlRP6uX5VcUoI7ricJlgrsfAnwrXn4LcLO7TwBuBuZ1WaevNhERKQH99mTM7KOdnzWzjwCpLs37w8BmBzGz4cB0YIy7ZwDc/W9mthdwGHBC/NGFwFwza4xr6LHN3ZsGUo+IiBRONpfLbo3/HALM77I8A7wBXDTAGg4gutx1ZRxiLUS9o1Zgjbt3ALh7h5mtJRo+neqjLXjIDGQiSRGRatJvyLj7ewDM7D/cfXqgGvYHnnP3y+LZBJYQPYMTVEPD8LzXLeZEkpWqUoO6Uo8rXzof21Xjucj6xn+ggAFYAbQTXfLC3X9vZm8S9WRGm1lt3FOpBUYBq4h6Mr21Za25uYV0OpNzwdX4FyWEpqbKm4e5sbG+Io8rXzof21XCuaipSeX8y3mfIWNmL7r7wfHXq4guke3E3ffLaa/d133TzH5LdH/lwXjU2F7Ay8AyYApwR/znc533XMys1zYRESkN/fVkzuvy9dSAdcwA5pvZHGAr0YOeb5nZDGCBmc0CNhANEOi6Tm9tIiJSAvoMGXd/rMvXvwtVhLu/ChzXw/KXgB5nfO6rTURESkMu08pcAjzs7svMbBLR1DLtwFnu/mSoAkVEpHzl8jDmV4Dl8df/BnwPmA18v9BFiYhIZcglZEa4+9tmVg98EPihu98KWJjSRESk3OUyd9kqM/t7YCLwaDx0eDegI0xpIiJS7nIJmcuAu4E2ovfGAHwKeLrQRYmISGXI5WHMB4geeOzqF/F/IiIiO8npzZhmNoLoHsyOj3w+XLCKRESkYuQyhPlzRFPqtwDvdmnKEM09JiIi0k0uPZnZwGnu/p+hihERkcqSyxDmOiCvN1+KiEh1yiVkrgOuMLNc1hERkSqWy+WyrwD7AF8zs+auDQOZhVlERCpXLiETchZmERGpQLk8JxNsFmYREalMuQxhHgLMInpBWIO7jzCzjwMT3H1uqAJFRKR85XIT/ybg/cBZbH9D5gvAFwpdlIiIVIZcQuYU4LPxu2PSAO6+BhgdojARESl/uYRMGztcXjOzRqC554+LiEi1yyVkfgEsMLP3AJjZvsBc4OchChMRkfKXS8h8k+jNmH8CdgdeAV4Hvh2gLhERqQC5PCdzIPAScA1QC9zv7n8KUpWIiFSEfkPGzFLArcDZwGpgLdHN/ivN7HbgXHfP9LEJERGpUtn0ZD4PHAdMcvdnOhea2RHAQuB84JYg1YmISFnL5p7MNODirgEDEH//5bhdRERkJ9mEzPuA3qaU+V3cLiIispNsQqbW3Tf11BAv19T/IiLSo2zuyQwys48AqQFsQ0REqlA2AbEOmN9Pu4iIyE76DRl3H1+EOkREpALpfoqIiARTUvdTzOxK4CrgEHd/3swmAfOAYcBrwFR3Xxd/ttc2EREpDSXTkzGzw4BJwMr4+xRwB3CBu08AHgWu7a9NRERKR0mETPzWzZuBL7L9hWiHA5vd/bH4+1uAM7JoExGRElESIQN8B7jD3Zd3WbYfsKLzG3d/E6gxs5H9tImISIlI/J6MmR0FHAF8o9j7bmgYXuxdSheNjfVJlxBEpR5XvnQ+tqvGc5F4yAD/ABwELDczgDHAfwE/AMZ1fsjM9gQy7r7ezFb21pbLjpubW0inc59Auhr/ooTQ1NTjRBJlrbGxviKPK186H9tVwrmoqUnl/Mt54pfL3P1adx/l7uPjZ3JWA58AbgCGmdkx8UdnAIvir//QR5uIiJSIUujJ9Mjd02Y2DZhnZkOJhyn31ybloW1rR9F6hJu3tLNpY2tR9iUi3ZVcyHSdYcDdnwAO6eVzvbZJ6Rs8qJaTL11clH0tmTOZ8r5IIVK+Er9cJiIilUshIyIiwShkREQkGIWMiIgEo5AREZFgFDIiIhKMQkZERIJRyIiISDAKGRERCUYhIyIiwShkREQkGIWMiIgEo5AREZFgFDIiIhKMQkZERIJRyIiISDAKGRERCUYhIyIiwShkREQkGIWMiIgEo5AREZFgFDIiIhKMQkZERIJRyIiISDAKGRERCUYhIyIiwShkREQkGIWMiIgEo5AREZFgFDIiIhKMQkZERIKpS7oAM2sAbgcOALYAfwHOd/cmM5sEzAOGAa8BU919Xbxer20iIlIaSqEnkwGud3dz9w8AfwWuNbMUcAdwgbtPAB4FrgXoq01EREpH4iHj7uvd/ZEui54CxgGHA5vd/bF4+S3AGfHXfbWJiEiJSPxyWVdmVgN8AfglsB+worPN3d80sxozG9lXm7uvz3Z/DQ3DC1e8lLTGxvqK3Fc50PnYrhrPRUmFDPBDoAWYC5wSemfNzS2k05mc16vGvyjlrqlpU1H209hYX7R9lQOdj+0q4VzU1KRy/uU88ctlnczsRuC9wJnungZWEl0262zfE8jEPZW+2kREpESURMiY2WzgQ8A/u/uWePEfgGFmdkz8/QxgURZtIiJSIhK/XGZmE4FvAi8DT5gZwHJ3P8XMpgHzzGwo8TBlAHdP99YmIiKlI/GQcfcXgFQvbU8Ah+TaJiIipaEkLpeJiEhlUsiIiEgwChkREQlGISMiIsEoZEREJBiFjIiIBKOQERGRYBQyIiISjEJGRESCSfyJf8negpkfZeTIsDNAL5kzucfl69dv4uzZDwfdt4hUHoVMGRk5sp5XZ5+ayL73n3lPIvsVkfKmy2UiIhKMQkZERIJRyIiISDAKGRERCUY3/vOQbm/rdRRWCMXcl4hIISlk8lBTNziRUV5JjvAKEazZbK/chk63be2gsTHsMPNOm7e0s2lja1H2JZIvhYxkpRqDNR+DB9Vy8qWLi7KvJXMms6koexLJn0JGpBd7jBhM3eAhOa830B5fufXeRPqikBHpRd3gIeq9iQyQRpeJiEgwChkREQlGISMiIsEoZEREJBiFjIiIBKPRZVLSCvUQaLEekCyEXI650MfV3raFDW+3FXSbUt0UMlLSknoIFJIbSpz8MStkpHB0uUxERIJRT0ZEuunpEpzmSZN8KWREpJue5l7TPGmSr7IOGTObACwAGoBmYLq7v5JsVSIi0qmsQwa4BbjZ3e8ws6nAPOCjCdckUrb6GtmW70i2bNbTqLbKVbYhY2Z7AYcBJ8SLFgJzzazR3Zv6Wb0WoKYmlff+60Y05r3uQCS13yT3rWMunpq6waycO6Po+x0z4weJDDNvb9vC25u2Fm1/A/mZUwq61F+b7TqpTCYTpprAzOxDwH+4+8Quy/4MTHX3pf2sfgzwPyHrExGpYB8GHsvmg2XbkxmgZ4hO0utAR8K1iIiUi1pgX6KfoVkp55BZBYw2s1p37zCzWmBUvLw/W8gyhUVEpJu/5vLhsn0Y093XAcuAKfGiKcBzWdyPERGRIinbezIAZnYQ0RDmPYANREOYPdmqRESkU1mHjIiIlLayvVwmIiKlTyEjIiLBKGRERCQYhYyIiARTzs/JFJ0m5NzOzBqA24EDiJ47+gtwfrUPITezK4GrgEPc/fmEy0mMmQ0FbgKOBzYDT7r755OtKjlm9ingu0CK6Jf7q9z93mSrKg71ZHLTOSHnBOBmogk5q1UGuN7dzd0/QPSA1rUJ15QoMzsMmASsTLqWEnA9UbhMcPdDgG8lXE9izCxF9AvZNHc/FJgKLDCzqvj5WxUHWQhdJuRcGC9aCBxmZsnN3pggd1/v7o90WfQUMC6hchJnZkOIfvH4IlEAVy0zGw5MB77l7hkAd/9bslUlLg2MiL/eHXjd3dMJ1lM0CpnsjQXWuHsHQPzn2nh5VYt/I/sC8Muka0nQd4A73H150oWUgAOILidfaWbPmtkjZnZM0kUlJQ7aM4DFZrYCuB84O9mqikchI4XwQ6AFmJt0IUkws6OAI4AfJV1LiagD9iea5ulw4OvAvWa2W7JlJcPM6oDLgcnuPg44Gbgr7vFVPIVM9rZNyAmQ44ScFcvMbgTeC5xZLd3/HvwDcBCw3MxeA8YA/2VmH0+yqAStANqJLy27+++BN4EJSRaVoEOBUe7+OED85zvAwYlWVSQKmSxpQs6dmdls4EPAP7v7lqTrSYq7X+vuo9x9vLuPB1YDn3D3BxMuLRHu/ibwW+IXCsajMvciGoFYjVYDY8zMAMzsYGAfcpzNuFxp7rIcaELO7cxsIvA88DLQGi9e7u6nJFdVaYh7M5+q8iHM+wPziYb7bwVmuvt/JltVcszsLOAbRAMAAK509/sTLKloFDIiIhKMLpeJiEgwChkREQlGISMiIsEoZEREJBiFjIiIBKOQESkQM/upmV2d0L5TZnabmW0ws6cLuN3PmdljXb7PmNmBhdq+VD5N9S8VK35eZRiwv7u/Ey/7V2Cqux+XXGVBHEP08OOYzmMVKQXqyUilqwO+lHQRueqcvigH44DXFDBSatSTkUp3A/A1M/uRu7/VtcHMxgPLgUHu3h4ve4RoNuWfmNnngPOAp4FzgPVE7wKZQPQCqiHAZe6+oMtm9zSz3xC9V2Yp0awQK+JtH0Q0meiHgCaiqfAXxW0/JZo5YRzRXGiTgYd2qHcU0TuNjolruc7df2xm/0L0moFBZtYCzHH3K3c8EWZ2HnAJ0dxqq4h6dEvN7Bvxce4VL5/p7vf1d2LN7B+BG4lmIt8I3OTuN/a3nlQX9WSk0j0LPAJ8Nc/1jwT+l2h6lJ8BPyeacflAosCZu8NsumcRBdCeRHPd3QlgZrsCv4m3sRfR3Hc/iqfn6fRZYDZQDzzGzhYSzYM1CjgNuMbMPubutwIziN4+ObyXgDmd6I2d04HdgH8imo4fojm0Pkz0vpNvA3eY2b5ZnJtbid6GWg+8H3g4i3WkyqgnI9VgFvC4mf17Husud/fbAMzsLmAm8J14QtAHzayNKHCWxZ//tbs/Gn9+JvC2mY0F/p7octZt8eeWmtk9RGHxQrxscedMvURvldwm3sYxRHOibQaWmdlPgGnAf2dxHP9K9CbTZ+Lvt01W6e6/6PK5u8zscuDvgMX9bHMr8D4z+6O7byCaz0+kG4WMVDx3f97MfkU0QeGLOa7e9Y2OrfH2dlzWtSez7dUP7t5iZuuJeh7jgCPNrOsluzqi1/LutG4PRgHr3X1Tl2UrgMOzOQiiS1o9zvprZtOJLqONjxcNJ+qJ9edU4ArgWjP7X+Ab7v5klvVIlVDISLW4kugeyZwuyzpvku9CdE8BoinYB2Lbm1Ljy2gjid6gugr4nbuf0Me6fc1WuxYYaWb1XYJmP2BNlnWtInpjZTdmNg74MfAxosttHWa2DEj1t8G4VzTZzAYBFwKL0JtiZQcKGakK7v6X+HLXxcCf4mVNZrYGmGpm84heibvTD+Ic/WP8quGnie7N/N7dV8U9qWvNbBrRfR2IXmbV4u799q7ibTwB/JuZfZVo8MG/EN0XysZPgO/Fz7wsJTrOrcCuROHWBGBm5xDdX+mTmQ0GTgd+5e5vm9lGoCPLWqSK6Ma/VJPvEP1Q7eo84DKim+ATgScGuI+fEfWa1hONIjsLIO59fBz4DFGv5A3gOqIRatmaQnRJay1wH9E7SX6TzYrxfZfZcX2biN4zP9Ld/0zUu3uS6NLgIcDjvW1nB9OA1+KAmUH2gSdVRO+TERGRYNSTERGRYBQyIiISjEJGRESCUciIiEgwChkREQlGISMiIsEoZEREJBiFjIiIBKOQERGRYP4PmhBG41BBfCAAAAAASUVORK5CYII=\n", "text/plain": [ "