{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lesson 6 - Model interpretability\n",
"\n",
"> How to interpret the predictions from Random Forest models and use these insights to prune the feature space."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/lewtun/dslectures/master?urlpath=lab/tree/notebooks%2Flesson06_model-interpretation.ipynb) \n",
"[![slides](https://img.shields.io/static/v1?label=slides&message=lesson06_model-interpretation.pdf&color=blue&logo=Google-drive)](https://drive.google.com/open?id=15IIYC_MksmXI6VfSL3ee-rLm7TRF0v2U)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning objectives"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Understand how to interpret feature importance plots for Random Forest models.\n",
"* Know how to drop uninformative features to build simpler models"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This lesson is adapted (with permission) from Jeremy Howard's fantastic online course [_Introduction to Machine Learning for Coders_](https://course18.fast.ai/ml), in particular:\n",
"\n",
"* [3 — Performance, validation and model interpretation](https://course18.fast.ai/lessonsml1/lesson3.html)\n",
"\n",
"Below are a few relevant articles that may be of general interest:\n",
"\n",
"* [Explaining Feature Importance by example of a Random Forest](https://towardsdatascience.com/explaining-feature-importance-by-example-of-a-random-forest-d9166011959e)\n",
"* [Beware Default Random Forest Importances](https://explained.ai/rf-importance/index.html)\n",
"* [Explainable AI won’t deliver. Here’s why.](https://hackernoon.com/explainable-ai-wont-deliver-here-s-why-6738f54216be)\n",
"* [Confidence Intervals](https://dfrieds.com/math/confidence-intervals.html)\n",
"* [Reading and Writing Files in Python (Guide)](https://realpython.com/read-write-files-python/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Homework\n",
"\n",
"* Solve the exercises included in this notebook"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this lesson we will analyse the preprocessed table of clean housing data and their addresses that we prepared in lesson 3:\n",
"\n",
"* `housing_processed.csv`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What is model interpretability?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"\n",
"
Figure reference: https://bit.ly/3djjWc6
\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A nice explanation for what it means to interpret a model's predictions is given in the _Beware Default Random Forest Importances_ article:\n",
"\n",
"> Training a model that accurately predicts outcomes is great, but most of the time you don't just need predictions, you want to be able to interpret your model. For example, if you build a model of house prices, knowing which features are most predictive of price tells us which features people are willing to pay for.\n",
"\n",
"In this lesson we will focus on one specific aspect of interpretability for Random Forests, namely _feature importance_ which is a technique that (with care) can be used to identify the most informative features in a dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# reload modules before executing user code\n",
"%load_ext autoreload\n",
"# reload all modules every time before executing Python code\n",
"%autoreload 2\n",
"# render plots in notebook\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# uncomment to update the library if working locally\n",
"# !pip install dslectures --upgrade"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# data wrangling\n",
"import pandas as pd\n",
"import numpy as np\n",
"from dslectures.core import get_dataset, convert_strings_to_categories, rmse, fill_missing_values_with_median\n",
"from pathlib import Path\n",
"\n",
"# data viz\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"\n",
"sns.set(color_codes=True)\n",
"sns.set_palette(sns.color_palette(\"muted\"))\n",
"\n",
"# ml magic\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.ensemble import RandomForestRegressor\n",
"from sklearn.metrics import r2_score\n",
"import scipy\n",
"from scipy.cluster import hierarchy as hc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load the data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As usual we can download the dataset with our helper function:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Download of housing_processed.csv dataset complete.\n"
]
}
],
"source": [
"get_dataset('housing_processed.csv')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We also make use of the `pathlib` library to handle our filepaths:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"housing.csv imdb.csv\n",
"housing_addresses.csv keep_cols.npy\n",
"housing_columns_to_keep.npy submission.csv\n",
"housing_gmaps_data_raw.csv test.csv\n",
"housing_merged.csv train.csv\n",
"housing_model.pkl uc\n",
"housing_processed.csv word2vec-google-news-300.pkl\n"
]
}
],
"source": [
"DATA = Path('../data/')\n",
"!ls {DATA}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
longitude
\n",
"
latitude
\n",
"
housing_median_age
\n",
"
total_rooms
\n",
"
total_bedrooms
\n",
"
population
\n",
"
households
\n",
"
median_income
\n",
"
median_house_value
\n",
"
city
\n",
"
postal_code
\n",
"
rooms_per_household
\n",
"
bedrooms_per_household
\n",
"
bedrooms_per_room
\n",
"
population_per_household
\n",
"
ocean_proximity_INLAND
\n",
"
ocean_proximity_<1H OCEAN
\n",
"
ocean_proximity_NEAR BAY
\n",
"
ocean_proximity_NEAR OCEAN
\n",
"
ocean_proximity_ISLAND
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
-122.23
\n",
"
37.88
\n",
"
41.0
\n",
"
880.0
\n",
"
129.0
\n",
"
322.0
\n",
"
126.0
\n",
"
8.3252
\n",
"
452600.0
\n",
"
69
\n",
"
94705
\n",
"
6.984127
\n",
"
1.023810
\n",
"
0.146591
\n",
"
2.555556
\n",
"
0
\n",
"
0
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
\n",
"
\n",
"
1
\n",
"
-122.22
\n",
"
37.86
\n",
"
21.0
\n",
"
7099.0
\n",
"
1106.0
\n",
"
2401.0
\n",
"
1138.0
\n",
"
8.3014
\n",
"
358500.0
\n",
"
620
\n",
"
94611
\n",
"
6.238137
\n",
"
0.971880
\n",
"
0.155797
\n",
"
2.109842
\n",
"
0
\n",
"
0
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
\n",
"
\n",
"
2
\n",
"
-122.24
\n",
"
37.85
\n",
"
52.0
\n",
"
1467.0
\n",
"
190.0
\n",
"
496.0
\n",
"
177.0
\n",
"
7.2574
\n",
"
352100.0
\n",
"
620
\n",
"
94618
\n",
"
8.288136
\n",
"
1.073446
\n",
"
0.129516
\n",
"
2.802260
\n",
"
0
\n",
"
0
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
\n",
"
\n",
"
3
\n",
"
-122.25
\n",
"
37.85
\n",
"
52.0
\n",
"
1274.0
\n",
"
235.0
\n",
"
558.0
\n",
"
219.0
\n",
"
5.6431
\n",
"
341300.0
\n",
"
620
\n",
"
94618
\n",
"
5.817352
\n",
"
1.073059
\n",
"
0.184458
\n",
"
2.547945
\n",
"
0
\n",
"
0
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
\n",
"
\n",
"
4
\n",
"
-122.25
\n",
"
37.85
\n",
"
52.0
\n",
"
1627.0
\n",
"
280.0
\n",
"
565.0
\n",
"
259.0
\n",
"
3.8462
\n",
"
342200.0
\n",
"
620
\n",
"
94618
\n",
"
6.281853
\n",
"
1.081081
\n",
"
0.172096
\n",
"
2.181467
\n",
"
0
\n",
"
0
\n",
"
1
\n",
"
0
\n",
"
0
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" longitude latitude housing_median_age total_rooms total_bedrooms \\\n",
"0 -122.23 37.88 41.0 880.0 129.0 \n",
"1 -122.22 37.86 21.0 7099.0 1106.0 \n",
"2 -122.24 37.85 52.0 1467.0 190.0 \n",
"3 -122.25 37.85 52.0 1274.0 235.0 \n",
"4 -122.25 37.85 52.0 1627.0 280.0 \n",
"\n",
" population households median_income median_house_value city \\\n",
"0 322.0 126.0 8.3252 452600.0 69 \n",
"1 2401.0 1138.0 8.3014 358500.0 620 \n",
"2 496.0 177.0 7.2574 352100.0 620 \n",
"3 558.0 219.0 5.6431 341300.0 620 \n",
"4 565.0 259.0 3.8462 342200.0 620 \n",
"\n",
" postal_code rooms_per_household bedrooms_per_household \\\n",
"0 94705 6.984127 1.023810 \n",
"1 94611 6.238137 0.971880 \n",
"2 94618 8.288136 1.073446 \n",
"3 94618 5.817352 1.073059 \n",
"4 94618 6.281853 1.081081 \n",
"\n",
" bedrooms_per_room population_per_household ocean_proximity_INLAND \\\n",
"0 0.146591 2.555556 0 \n",
"1 0.155797 2.109842 0 \n",
"2 0.129516 2.802260 0 \n",
"3 0.184458 2.547945 0 \n",
"4 0.172096 2.181467 0 \n",
"\n",
" ocean_proximity_<1H OCEAN ocean_proximity_NEAR BAY \\\n",
"0 0 1 \n",
"1 0 1 \n",
"2 0 1 \n",
"3 0 1 \n",
"4 0 1 \n",
"\n",
" ocean_proximity_NEAR OCEAN ocean_proximity_ISLAND \n",
"0 0 0 \n",
"1 0 0 \n",
"2 0 0 \n",
"3 0 0 \n",
"4 0 0 "
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"housing_data = pd.read_csv(DATA/'housing_processed.csv'); housing_data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With the data loaded, we can recreate our train/validation splits as before:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X = housing_data.drop('median_house_value', axis=1)\n",
"y = housing_data['median_house_value']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"15554 train rows + 3889 valid rows\n"
]
}
],
"source": [
"X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.2, random_state=42)\n",
"print(f'{len(X_train)} train rows + {len(X_valid)} valid rows')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To simplify the evaluation of our models, we'll reuse our scoring function from lesson 5:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def print_rf_scores(fitted_model):\n",
" \"\"\"Generates RMSE and R^2 scores from fitted Random Forest model.\"\"\"\n",
"\n",
" yhat_train = fitted_model.predict(X_train)\n",
" R2_train = fitted_model.score(X_train, y_train)\n",
" yhat_valid = fitted_model.predict(X_valid)\n",
" R2_valid = fitted_model.score(X_valid, y_valid)\n",
"\n",
" scores = {\n",
" \"RMSE on train:\": rmse(y_train, yhat_train),\n",
" \"R^2 on train:\": R2_train,\n",
" \"RMSE on valid:\": rmse(y_valid, yhat_valid),\n",
" \"R^2 on valid:\": R2_valid,\n",
" }\n",
" if hasattr(fitted_model, \"oob_score_\"):\n",
" scores[\"OOB R^2:\"] = fitted_model.oob_score_\n",
"\n",
" for score_name, score_value in scores.items():\n",
" print(score_name, round(score_value, 3))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Confidence intervals"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Recall that to make predictions with our Random Forest models, we take the _average value_ in each leaf node as we pass each row in the validation set through the tree. However, we would also like to estimate our _**confidence**_ in these predictions - how can we achieve this?\n",
"\n",
"One way to do this is to calculate the _**standard deviation of the predictions**_ of the trees. Conceptually, the idea is that if the standard deviation is high, each tree is generating very different predictions and may indicate the model has not learnt the most important features of the data.\n",
"\n",
"To get started, let's use our baseline model from the previous lesson:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"RMSE on train: 16520.37\n",
"R^2 on train: 0.971\n",
"RMSE on valid: 42727.043\n",
"R^2 on valid: 0.81\n",
"OOB R^2: 0.791\n"
]
}
],
"source": [
"model = RandomForestRegressor(n_estimators=40, max_features='sqrt', n_jobs=-1, oob_score=True, random_state=42)\n",
"model.fit(X_train, y_train)\n",
"print_rf_scores(model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As before, we concatenate all the predictions from each tree into a single array:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(137125.0, 41845.23120978064)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds = np.stack([t.predict(X_valid) for t in model.estimators_])\n",
"# calculate mean and standard deviation for single observation\n",
"np.mean(preds[:,0]), np.std(preds[:,0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(40, 3889)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, let's create a copy of the validation dataset and add the mean predictions and their standard deviation as new columns."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"#### Exercise #1\n",
"\n",
"* Combine `X_valid` and `y_valid` into a single array called `valid_copy`. You may find the `DataFrame.join(DataFrame)` method from pandas useful here.\n",
"* Create two new columns `preds_mean` and `preds_std` that are the mean and standard deviation of the predictions for each tree in `preds`\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can use these new columns to drill-down into the predictions of each individual, categorical feature. Let's examine `ocean_proximity_INLAND` which denotes whether a housing district is inland or not:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEJCAYAAACaFuz/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAXi0lEQVR4nO3de1AV5+HG8ecgSCQxVQwY6yWt8TYx463emyqYAVRERyUxagRLTdTqpGE0RkeJxpjYMUYbtSaxNhMsaLUqMRirGI31AhmLbbRx1DqJdy1EBBUVwuHs749O+JUIxxV24bB8P3+5u+zheXdnnllf9uy6DMMwBABwLL/aDgAAsBdFDwAOR9EDgMNR9ADgcBQ9ADgcRQ8ADkfRA4DD+dd2gMrk59+Sx1N/bvFv1uwh5eUV1naMGsWY6wfGXDP8/Fxq2vTBCrf5bNF7PEa9KnpJ9W68EmOuLxhz7WLqBgAcjqIHAIej6AHA4Sh6AHA4ih4AHI6iBwCHo+gBwOEoegBwOBdvmAIA31BU7NbNG3eqtK+fn0vNmj1U4Taf/WbsS8u+0NWCotqOAQA1Zv3CMN204XOZugEAh6PoAcDhKHoAcDiKHgAcjqIHAIej6AHA4Sh6AHA4ih4AHI6iBwCHo+gBwOEoegBwOIoeAByOogcAh6PoAcDhKHoAcDiKHgAcjqIHAIej6AHA4Sh6AHA4ih4AHI6iBwCHo+gBwOEoegBwOIoeAByOogcAh6PoAcDhKHoAcDiKHgAcjqIHAIej6AHA4fzv9QPHjh1TcnKy/v3vf+uBBx5Qhw4dFB8frw4dOtREPgBANXm9ot+/f79+/etfq0OHDpo5c6amT5+uFi1aKCEhQYcPH66pjACAavB6Rb969WqtXbtWnTp1Kls3cOBADRgwQO+884569+5te0AAQPV4vaK/detWuZL/XpcuXXTz5k3bQgEArOO16P39K7/gNwzD8jAAAOtx1w0AOJzXOfpvvvlGMTExFW67cOGCLYEAANbyWvR/+MMfaioHAMAmXou+unfVpKen67333lNJSYkmTpyo8ePHV+vzAAD3z2vRDxo0SC6Xq8JtLpdLn332WaX75uTkaPny5dq6dasaNmyo5557Tn369FG7du2qlxgAcF+8Fv2KFSvuWpedna1ly5YpKirK6wdnZmaqb9++atKkiSQpKipKO3fu1PTp06sRFwBwv7wW/ZNPPln2b4/HoxUrVmjDhg1atGiRhg8f7vWDc3NzFRISUrYcGhqqY8eOVTMuAOB+3fNZN5J06dIlzZgxQ5K0efNmtW7d+p77VHSffWXTQAAA+9zzPvpPPvlEo0aNUt++fZWammqq5CWpefPmunr1atlybm6uQkNDq54UAFAlXq/oX3nlFWVkZCgxMVG9evXSyZMny23v3Llzpfv2799fK1eu1LVr19SoUSNlZGTojTfesCY1AMA0r0V/5MgRNWvWTOvWrdO6devKbXO5XNqzZ0+l+zZv3lyJiYmKi4tTSUmJYmNj1aVLF2tSAwBM81r0e/furXRbQUHBPT88Jiam0m/WAgBqxn0/6+brr7/Wa6+9prCwMBviAACsZuquG0k6cOCAPvroI2VmZupnP/uZ3nvvPTtzAQAs4rXoi4uLlZaWpnXr1ikvL09Dhw7V6dOnlZKSUlP5AADV5LXow8LC1LVrV7388ssKCwtTw4YNtX///prKBgCwgNei79mzp7Kzs/Xggw8qMDBQv/jFL2oqFwDAIl6LfuXKlcrNzdWmTZuUlJQkj8ej4uJiXbx4Ua1ataqpjACAarjnXTehoaGaPn26Pv/8cyUlJenJJ5/U4MGD9fLLL9dEPgBANZm+66ZBgwaKiopSVFSUzpw5o40bN9qZCwBgkSq9M/anP/2pjh49anUWAIANqvxy8FOnTlmZAwBgkyoXPQCgbqDoAcDhvP4xdtGiRRWuNwxDJSUltgQCAFjLa9F//77XikyePNnyMAAA63ktel7kDQB1n9einzNnTqXbXC6X3nrrLcsDAQCs5bXo27dvf9e6/Px8JScnq2XLlraFAgBYx2vRJyQklFvOzMzUq6++qpiYGM2bN8/WYAAAa5h6BILb7dY777yjtLQ0LViwQIMHD7Y7FwDAIvcs+nPnzikxMVFBQUFKS0tTixYtaiIXAMAiXr8wtXnzZj3zzDOKiIhQSkoKJQ8AdZDLMAyjso2dOnWSn5+fAgMD5XK5ytYbhiGXy6V//OMftgV7adkXulpQZNvnA4CvWb8wTN9+e7NK+/r5udSs2UMVbvM6dbNnz54q/UIAgO/wWvTcQgkAdZ/Xou/evXu5KZvv1cTUDQDAGl6Lfvv27TWVAwBgk2pP3Tz//PNKSUmxLBAAwFrVfh59YWGhFTkAADapdtFXNIcPAPAdvGEKAByOogcAh6PoAcDhql30Xp6gAADwAaaKfuTIkfrLX/6iO3fu3LUtNTXV8lAAAOuYKvqkpCRlZ2crIiJCCxcu1OnTp8u2Pfjgg7aFAwBUn6kXj/To0UM9evTQjRs3lJ6erqlTpyo0NFQTJkzQkCFD7M4IAKgG03P0N27c0LZt27Rp0yY1btxYQ4YM0bZt2zRr1iw78wEAqsnUFf2MGTO0f/9+hYWFacGCBerevbskaezYserfv7+tAQEA1eP1xSPfe//99/Xss88qODj4rm1ff/21Hn/8cVvCAUB9UlTs1s0bd9/0YkaVXzzyvezsbE2ZMqXcumeffVabNm2yreTz8grl8dSfWzdDQhpX+c0ydRVjrh8Yc+3zWvQvvfSSzpw5owsXLigmJqZsvdvtlp8f37UCgLrAa9HPmjVLly5dUlJSkpKSksrWN2jQQO3bt7c9HACg+rwWfatWrdSqVSvt2rWLp1QCQB3ltejHjh2rDRs2qEePHuWKnlcJAkDd4bXo3333XUm8UhAA6jKvRR8aGirpv68U/PLLL3X9+vVy2828ahAAULtM3V6ZmJio7OzssuKX/vtmqYEDB9oWDABgDVNF/9VXX2nPnj1q2LCh3XkAABYzdTN827Zt5Xa77c4CALCBqSv6MWPGaPjw4erevbv8/f9/l8WLF9sWDABgDVNFv2TJEj311FNq06aN3XkAABYzVfT+/v5asGCBzVEAAHYwNUffrVs37du3z+YoAAA7mLqi/+KLL7R582YFBAQoICCAb8YCQB1iquiTk5PtzgEAsInXos/KylK/fv10/PjxCrfzzVgA8H1ei/7TTz9Vv3799Kc//emubS6XS5GRkbYFAwBYw2vRL1q0SJK0du1aBQYGltuWk5NjXyoAgGVM3XXzzDPP6Ny5c2XLe/fu1ahRo2wLBQCwjqk/xsbFxWn8+PGaNWuWjh49qoMHD2rVqlW2BqvsJbdOFhLSuLYj1DhfGLP7u2LlX/+utmMAtjFV9LGxsQoJCdGUKVP0yCOP6JNPPlHTpk1tDfbNmply37hq6+8AJKnDzI8kUfRwLlNTNxs2bNDs2bM1d+5chYeHa9y4cfrXv/5ldzYAgAVMXdGnpqYqOTlZHTp0kCRlZGRoypQpOnTokK3hAADVZ6rot2zZosDAQLndbhmGocjISHXp0sXubAAAC5iauiksLNSkSZPUrVs3denSRRMmTCj3snAAgO8yVfQLFy5Ut27dlJmZqczMTPXq1YunWQJAHWGq6M+ePavp06fr4YcfVtOmTfXSSy/p/PnzdmcDAFjAVNG73W4VFxeXLd+5c4epGwCoI0z9MXbo0KGaOHFi2bdht27dqqioKFuDAQCsYarop02bpkcffVQHDhyQx+PRqFGjFBsba3c2AIAFTBV9fHy8kpOTNXr0aLvzAAAsZmqO/ubNm7p9+7bdWQAANjB1Rd+oUSOFh4erY8eOCgoKKlv//vvv2xYMAGAN0w81AwDUTaaKfuTIkcrPz1d2drb8/PzUu3dvNW5c+4+XBQDcm6k5+t27dysyMlLJyclau3atIiIi9MUXX9idDQBgAVNX9MuXL1dKSoo6duwoSTp+/LjmzZuntLQ0W8MBAKrP1BX9Aw88UFbyktS5c2e+GQsAdYSpou/Tp4/WrFmj27dvq7i4WBs3blT79u11/fp1FRQU2J0RAFANpqZu1q1bp9LSUi1btqzc+m3btsnlcunEiRO2hAMAVJ+poj9+/Hil29LT0y0LAwCwnqmpG28+/PBDK3IAAGxS7aI3DMOKHAAAm1S76Ln7BgB8W7WLHgDg2yh6AHA45ugBwOGqXfQxMTFW5AAA2MTUffSHDx/WypUrdf369XJX8Onp6frVr35lWzgAQPWZKvqFCxdq9OjReuKJJ7jLBgDqGFNFHxAQoF/+8pd2ZwEA2MDUHH379u116tQpu7MAAGxg6or+woULGj16tH784x8rMDCwbD3PuQEA32eq6BMTE+3OAQCwiami7927twoKCnTnzh0ZhqHS0lKdP3/e7mwAAAuYKvp3331Xa9askSQ1aNBAJSUlateuHVM3AFAHmPpj7LZt2/T5558rKipKGRkZ+u1vf6t27drZnQ0AYAFTRR8cHKzQ0FC1bdtWJ0+e1IgRI3Tu3Dm7swEALGCq6P39/XX+/Hm1bdtW2dnZcrvdunHjht3ZAAAWMFX0kydPVlJSksLCwrR7926FhYWpb9++99yvsLBQw4YN08WLF6sdFABQNab+GBseHq7w8HBJ0scff6xz586pY8eOXvc5evSo5s2bp7Nnz1Y7JACg6kxd0d+6dUuvv/664uPjVVxcrPXr1+v27dte99m0aZPmz5+v0NBQS4ICAKrGVNEvWrRIDz/8sPLy8hQYGKjCwkK99tprXvd588031bNnT0tCAgCqzlTRnzhxQomJifL391ejRo20dOlSnThxwu5sAAALmCp6P7/yP1ZaWnrXOgCAbzL1x9hevXrp7bffVlFRkQ4cOKCUlBT16dPH7mwAAAuYuiyfOXOmgoKC1LhxYy1fvlydOnXSrFmz7M4GALCA6ReP9O7dW9OmTVNBQYGys7PLPa7Ym71791YrIACgekxd0S9fvlwrVqyQJBUVFWnNmjVavXq1rcEAANYwVfR79uzRhx9+KEl69NFHlZKSoh07dtgaDABgDVNFX1JSooCAgLLlgIAAXhIOAHWEqTn6Hj16aMaMGYqNjZXL5VJaWpq6du1qdzYAgAVMXdEnJSXpkUce0fTp0/Wb3/xGISEhmjt3rt3ZAAAWMFX0ubm5OnjwoFwul9xut3bu3KnLly/bnQ0AYAFTRf/GG29o0qRJys7O1pEjRzR16lS9/vrrdmcDAFjAVNHn5eVp5MiRZcujR49Wfn6+baEAANYxVfSlpaUqKCgoW7527ZptgQAA1jJ1183zzz+vMWPGaMiQIZKkv/71r4qPj7c1GADAGqaKfsyYMWrTpo0OHjwoj8ej+fPnq3///nZnAwBYwFTRS1K/fv3Ur18/O7MAAGzAQ+UBwOEoegBwOIoeAByOogcAh6PoAcDhKHoAcDiKHgAcjqIHAIej6AHA4Sh6AHA4ih4AHI6iBwCHo+gBwOEoegBwOIoeAByOogcAh6PoAcDhKHoAcDiKHgAcjqIHAIej6AHA4Sh6AHA4ih4AHI6iBwCHo+gBwOEoegBwOP/aDlCZti8ure0IqCfc3xXXdgTAVj5b9Hl5hfJ4jNqOUWNCQhrr229v1naMGlUfxwzUBqZuAMDhKHoAcDiKHgAcjqIHAIej6AHA4Sh6AHA4ih4AHI6iBwCH89kvTPn5uWo7Qo1jzPUDY64fanrM3n6fyzCM+vP1UwCoh5i6AQCHo+gBwOEoegBwOIoeAByOogcAh6PoAcDhKHoAcDiKHgAcjqIHAIfzqaJPT0/X0KFDFRERodTU1NqOY6m4uDhFR0drxIgRGjFihI4ePVrpeDMzMxUTE6PIyEgtX768FlNXTWFhoYYNG6aLFy9Kqnw8J06c0OjRoxUVFaW5c+fK7XZLki5fvqzx48dr8ODBmjp1qm7dulUr47gfPxzznDlzFBkZWXa+d+/eLen+j4WvWrVqlaKjoxUdHa0lS5ZIcv55rmjMdeY8Gz7iP//5jxEeHm7k5+cbt27dMmJiYozTp0/XdixLeDwe4+c//7lRUlJStq6y8d65c8cYOHCgcf78eaOkpMRISEgw9u3bV4vp78+XX35pDBs2zOjcubNx4cIFr+OJjo42/vnPfxqGYRhz5swxUlNTDcMwjBdffNHYvn27YRiGsWrVKmPJkiW1MxiTfjhmwzCMYcOGGTk5OeV+rirHwhcdOnTIGDNmjFFcXGx89913RlxcnJGenu7o81zRmDMyMurMefaZK/rMzEz17dtXTZo0UVBQkKKiorRz587ajmWJb775Ri6XSy+88IKGDx+ulJSUSsd77NgxPfbYY2rdurX8/f0VExNTp47Dpk2bNH/+fIWGhkpSpeO5dOmSioqK1K1bN0nSqFGjtHPnTpWUlOjvf/+7oqKiyq33ZT8c8+3bt3X58mUlJSUpJiZGK1askMfjue9j4atCQkI0e/ZsNWzYUAEBAXr88cd19uxZR5/nisZ8+fLlOnOefebplbm5uQoJCSlbDg0N1bFjx2oxkXVu3Lihfv36acGCBSoqKlJcXJyGDBlS4XgrOg45OTm1EbtK3nzzzXLLlY3nh+tDQkKUk5Oj/Px8PfTQQ/L39y+33pf9cMx5eXnq27evFi5cqKCgIE2ePFmbN29WUFDQfR0LX9W+ffuyf589e1Y7duzQhAkTHH2eKxrz+vXrdfjw4Tpxnn3mit6o4CGaLpczHm3avXt3LVmyREFBQQoODlZsbKxWrFhx18+5XC7HHYfKxnO/6+uS1q1b6/e//72aNWumRo0aacKECfrb3/7muDGfPn1aCQkJevXVV9WmTZu7tjvxPP/vmNu2bVtnzrPPFH3z5s119erVsuXc3Nyy/wrXddnZ2crKyipbNgxDLVu2rHC8TjsOlY3nh+u//fZbhYaGKjg4WIWFhSotLS23vi45deqUdu3aVbZsGIb8/f3v+1j4siNHjmjixImaMWOGRo4cWS/O8w/HXJfOs88Uff/+/ZWVlaVr167pzp07ysjI0IABA2o7liVu3rypJUuWqLi4WIWFhUpLS9Pbb79d4Xi7du2qM2fO6Ny5cyotLdX27dvr9HGobDwtW7ZUYGCgjhw5Ikn6+OOPNWDAAAUEBKhnz57asWNHufV1iWEYeuutt3T9+nWVlJRo48aNioiIuO9j4auuXLmiadOmaenSpYqOjpbk/PNc0Zjr0nn2qRePpKen64MPPlBJSYliY2P1wgsv1HYky/zud7/Trl275PF4NG7cOMXHx1c63qysLC1evFjFxcUaOHCg5syZUyf+W/u/Bg0apHXr1qlVq1aVjufkyZOaN2+ebt26pSeeeEKLFy9Ww4YNdenSJc2ePVt5eXlq0aKFli1bph/96Ee1PaR7+t8xp6amKjU1VW63W5GRkZo5c6akys9tZcfCFy1atEhbtmwpN13z3HPP6Sc/+Yljz3NlY/Z4PHXiPPtU0QMArOczUzcAAHtQ9ADgcBQ9ADgcRQ8ADkfRA4DDUfSAjRISEnTt2rXajoF6jqIHbHTo0KHajgBQ9Ki/Nm/erOjoaMXExCguLk5XrlzRxo0bNWzYMA0fPlwJCQk6c+aMJGn27Nn64x//WLbv/y4PGjRIK1eu1Lhx4xQeHl7uWeWSFB8frytXrtTw6ID/5zNPrwRq0smTJ7V06VKlpaWpRYsW+uijjzRx4kR5PB5t3LhRwcHB2rp1q6ZNm6ZPP/30np93+/ZtrV+/Xjk5OYqIiNDYsWO1ePFibd26VcnJyQoODq6BUQEV44oe9VJWVpaeeuoptWjRQpI0ceJEPf300xo6dGhZKY8aNUo5OTllb43y5umnn5b034e4NWvWTNevX7cvPHCfKHrUSw0aNCj3/KCioqIKC90wDLnd7rseM1tSUlLu5wIDA8v+XdkjaYHaQtGjXurTp4+ysrKUm5srSfrzn/+sffv2aceOHWV3yWzZskVNmjTRY489pqZNm+qrr76SJF27dk3Z2dmmfk+DBg18/v2vcD7m6FEvdezYUa+88oomTZok6b9v+9m9e7c+++wzxcfHy+PxKDg4WB988IH8/Pw0YcIEzZw5U1FRUWrVqpV69+5t6vdERERo3LhxWr16tTp06GDnkIBK8fRKAHA4pm4AwOEoegBwOIoeAByOogcAh6PoAcDhKHoAcDiKHgAcjqIHAIf7Pzwbt48Rn6E8AAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"sns.countplot(y='ocean_proximity_INLAND', data=valid_copy);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can calculate the predictions and standard deviation per category by applying a _**group by**_ operation in pandas, followed by taking the mean."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
ocean_proximity_INLAND
\n",
"
median_house_value
\n",
"
preds_mean
\n",
"
preds_std
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
0
\n",
"
227863.194178
\n",
"
227319.699349
\n",
"
51913.127856
\n",
"
\n",
"
\n",
"
1
\n",
"
1
\n",
"
123654.224570
\n",
"
124241.488165
\n",
"
35858.872619
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" ocean_proximity_INLAND median_house_value preds_mean preds_std\n",
"0 0 227863.194178 227319.699349 51913.127856\n",
"1 1 123654.224570 124241.488165 35858.872619"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cols = ['ocean_proximity_INLAND', 'median_house_value', 'preds_mean', 'preds_std']\n",
"preds_quality = valid_copy[cols].groupby('ocean_proximity_INLAND', as_index=False).mean()\n",
"preds_quality"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From the table, we can see that the predictions and ground truth are close to each other on average, while the standard deviation varies somewhat for each category. We can visualise this table in terms of bar plots as follows:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA18AAAGeCAYAAABrbbtRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3daWAUdb718dNJOmGLLCFhxxFkGVHZ1+uFoMMmCV42EUFwYBzkwqhcFFGIIKIoIsiiKPB4CSaCyGIMKqAgAgJq4ArKIDCyBRgBCZCELCTpel4wZoyQTkG6qknl+3llV6erT+dnC8d/LS7DMAwBAAAAACwV4O8AAAAAAFAaUL4AAAAAwAaULwAAAACwAeULAAAAAGxA+QIAAAAAG1C+AAAAAMAGlC8AAAAAsEGQvwMU5ty5i/J4uAVZSRYWVkFnz6b7OwaKiTk6A3N0DmbpDMzR/wICXKpcuby/Y6CUuWHLl8djUL4cgBk6A3N0BuboHMzSGZgjUPpw2CEAAAAA2IDyBQAAAAA2oHwBAAAAgA0oXwAAAABgA8oXAAAAANiA8gUAAAAANqB8AQAAAIANKF8AAAAAYAPKFwAAAADYgPIFAAAAADagfAEAAACADShfAAAAAGADyhcAAAAA2IDyBQAAAAA2oHwBAAAAgA0oXwAAAABgA8oXAAAAANggqKgf2LNnj2JjY3XgwAGVKVNGDRs21NChQ9WwYUM78gEAAACAI3gtX5s3b9azzz6rhx56SL169ZIkff/99xo2bJhmzpypNm3aWBYsLKyCZfuGfcLDQ/0dAT7gxDlmZecpLTXD3zEAAEAp4rV8vfnmm1q0aJEaN26cv61Tp07q2LGjXnvtNUvL19BX/q7T53Ms2z+A0u3TaU2V5u8QAACgVPF6ztfFixcLFK9f3XnnnUpL468tAAAAAGCW1/IVFFT4wphhGD4PAwAAAABOxdUOAQAAAMAGXs/5OnTokKKjo6/6XHJysiWBAAAAAMCJvJavhQsX2pUDAAAAABzNa/my8mqGAAAAAFCaeC1fd999t1wu11Wfc7lc+vzzzy0JBQAAAABO47V8zZkz54ptSUlJmjlzprp162ZZKAAAAABwGq/l6/bbb8//Z4/Hozlz5mjp0qWaOnWqevXqZXk4AAAAAHAKr+XrVydOnNDYsWMlSStWrFCdOnUsDQUAAAAATlPkfb4++ugj9enTR+3atVN8fDzFCwAAAACug9eVr6eeekrr16/XmDFj1Lp1a/34448Fnm/SpIml4QAAAADAKbyWr507dyosLExLlizRkiVLCjzncrm0YcMGS8MBAAAAgFN4LV8bN24s9Lnz58/7PAwAAAAAOFWR53z93k8//aTnnntOkZGRFsQBAAAAAGcydbVDSdqyZYsWL16sbdu2qWXLlpo/f76VuQAAAADAUbyufGVnZ2vZsmW699579eSTT6pu3boKDw9XXFyc2rdvX+TO09PTFRUVpePHj/ssMAAAAACURF5XviIjI9W0aVM98cQTioyMVHBwsDZv3mxqx7t379bEiRN15MgRX+QEAAAAgBLN68pXq1attHv3bq1bt07bt2+Xx+MxvePly5dr0qRJioiIKHZIAAAAACjpvK58zZ07V6dPn9by5csVExMjj8ej7OxsHT9+XLVr1/a64xdffNGnQQEAAACgJCvyghsREREaPXq0Ro4cqc8//1zLli1T9+7d9ac//Umvv/66HRkBwBLh4aH+jmCr0vZ5nYxZOgNzBEof01c7DAwMVLdu3dStWzcdPnxY77//vpW5AMByZ86k+TuCbcLDQ0vV53UyZukMzNH/AgJcCgur4O8YKGWu+T5fknTLLbdo9+7dvs4CAAAAAI51XeVLkvbv3+/LHAAAAADgaKYPO7xeGzdutPotAAAAAOCGd90rXwAAAAAA87yufE2dOvWq2w3DUE5OjiWBAAAAAMCJvJavSpUqFfrciBEjfB4GAAAAAJzKa/kaPXq0XTkAAAAAwNG8lq9nnnmm0OdcLpdeeuklnwcCAAAAACfyWr4aNGhwxbZz584pNjZWtWrVsiwUAAAAADiN1/I1bNiwAo+3bdump59+WtHR0Zo4caKlwQAAAADASUzd5ys3N1evvfaaVq9ercmTJ6t79+5W5wIAAAAARymyfB09elRjxoxRuXLltHr1atWoUcOOXAAAAADgKF5vsrxixQr1799fXbp0UVxcHMULAAAAAK6T15WviRMnKiAgQAsWLNDChQvztxuGIZfLpV27dlkeEAAAAACcwGv52rBhg105AAAAAMDRvJYvLicPAAAAAL7htXw1b95cLpfriu0cdggAAAAA18Zr+VqzZo1dOQAAAADA0Yp92OHgwYMVFxfns0AAAAAA4EReLzVvRnp6ui9yAAAAAICjFXmT5aJc7ZwwX4h9+jZL9gsAkpSVnefvCAAAoJQpdvmyytmz6fJ4DH/HQDGEh4fqzJk0f8dAMTFHAAAA3yj2YYcAAAAAgKJRvgAAAADABsUuX4bBoYEAAAAAUBRT5at379764IMPlJmZecVz8fHxPg8FAAAAAE5jqnzFxMQoKSlJXbp00ZQpU3Tw4MH858qXL29ZOAAAAABwClNXO2zRooVatGih1NRUJSYmauTIkYqIiNBDDz2kHj16WJ0RAAAAAEo80+d8paamKiEhQcuXL1doaKh69OihhIQEjRs3zsp8AAAAAOAIpla+xo4dq82bNysyMlKTJ09W8+bNJUkDBw5Uhw4dLA0IAAAAAE5gqnw1aNBAEyZMUJUqVQq+OChIS5cutSQYAAAAADiJqcMOk5KSrihe999/vySpfv36vk8FAAAAAA7jdeXrscce0+HDh5WcnKzo6Oj87bm5uQoI4P7MAAAAAGCW1/I1btw4nThxQjExMYqJicnfHhgYqAYNGlgeDgAAAACcwmv5ql27tmrXrq1169bJ5XLZlQkAAAAAHMdr+Ro4cKCWLl2qFi1aFChfhmHI5XJp165dlgcEAAAAACfwWr5mz54tSVqzZo0tYQAAAADAqbyWr4iICElSrVq19N133+nChQsFnq9Vq5Z1yQAAAADAQUzd52vMmDFKSkrKL2OS5HK51KlTJ8uCAQAAAICTmCpfP/zwgzZs2KDg4GCr8wAAAACAI5m6WVe9evWUm5trdRYAAAAAcCxTK18DBgxQr1691Lx5cwUF/fsl06ZNsywYAAAAADiJqfI1ffp03XXXXapbt67VeQAAAADAkUyVr6CgIE2ePNniKAAAAADgXKbO+WrWrJk2bdpkcRQAAAAAcC5TK187duzQihUr5Ha75Xa7ZRiGXC6Xdu3aZXU+AAAAAHAEU+UrNjbW6hwAAAAA4Ghey9f27dvVvn177d2796rP16pVy5JQAAAAgL/k5OQoOTlZmZlZ/o6CEqZs2TKqU6eO3G73VZ/3Wr4+/vhjtW/fXu++++4Vz7lcLnXt2tU3KQEAAIAbRHJysoKCQlSjRoRcLpe/46CEMAxDaWkXlJycrHr16l31Z7yWr6lTp0qSFi1apJCQkALPnTp1ykcxAQAAgBtHZmYWxQvXzOVyKTS0ov75z2OF/oypqx32799fR48ezX+8ceNG9enTp/gJAQAAgBsQxQvXo6h/b0xdcGPIkCEaNGiQxo0bp927d2vr1q2aN2+eTwICAAAAN7Ky5cuoTHCgz/ebdSlPmRc5r6w0MVW++vXrp/DwcD366KOqWrWqPvroI1WuXNnqbAAAAIDflQkOVI9ndvt8v59Oa6rMiz7fbaF27kzSokVva/78hXrxxSnq06ef/vjH23y2///6r556882Fqlmzps/2aZff/m6sZOqww6VLl2r8+PGaMGGCOnfurAcffFDff/+9pcEAAAAAWGPChOd8WrxgjqmVr/j4eMXGxqphw4aSpPXr1+vRRx/VV199ZWk4AAAAoLTbuTNJixf/PxmGoRMnjuvuu+9R+fKh2rz5CxmGoZkz5+rAgR+1cOFbys3NVY0aNfXsszGqWLGSvv56u15//TUFBwfr5ptvyd/nyJGP6C9/GaGmTZtp+vRpOnToH0pJSVHdujfr5ZdnKCUlRePHj1W9evV14MB+ValSRS++OF0VK1b0mvWddxbowIH9ysrK0nPPTdHtt9+hY8eOatq0qUpNvaCyZcvqf/5nnG67rYmmTJmkFi1aKiqqlySpXbsW2rFjl7799mvNmzf7XxewuEkvvPCSKlWqrE8+WaNly96TYXjUuPEf9eST46+4KOCvNm/+UgkJK/Xaa3MkSR98sEzHjh3To4/+t158cYpOnz6lX345o2bNWmjSpBcKvPbX303Llq108uRJ/fd/P6IPP/xYZ8+e1SuvvKhTp35WQECARo78m9q0aXtNszS18rVy5Uo1bNhQubm5ysnJUdeuXbVy5cpreiMAAAAA12fv3h8UEzNZS5d+oFWrVqhy5UpavDhet97aQKtXr9Cbb87V66+/oSVLlqpdu/aaN2+OLl26pClTJumll6YrNva9qxaV77/fLbc7SIsWxWrFigRlZ2dr27bLCywHDx7QwIGD9d57H6hChVCtW/dpkTlvuaWelixZqv79B+i99y7frmry5Im6//4HFB+/XE88MVbPPvuULl26VOg+/vd//5+efnqCFi+O1113ddT+/T/q0KGflJCwSgsX/q/efXeZKleuovj4K2+H9asOHTpo//4flZqaKklav36dune/V199tVUNGjTUokWx+uCDBP3wwx7t37+vyM8lSbNmvaqoqPsUG/ueXn11ll555UVdvHhtx42aWvlKT0/XqFGjtGPHDuXl5alVq1aaMWPGNb0RAAAAgOtTv359VatWXZJUsWIltWrVRpJUvXoNbd26WadO/axRo/4qSfJ4PLrpppv000//UNWqVXXLLZfvOdWzZ5Tefnt+gf02b95SFStW0ooV7+vIkSM6fvyYMjMzJEmVK1dRo0aN898/NfVCkTk7doyUJNWrV19ffLFRGRkZOn48WZ073yNJuv32O3XTTRV19OiRQvfxn//ZUU8/PVadOkXqP/8zUm3bttMHHyxTcnKy/vKXoZIu3wj712xXExTkVmTk3friiw1q06atUlMvqEmT29Wkye3au/cHLVsWryNHDuvChQvKyMgs8nNJ0rfffqOjR49o4cLLv8Pc3FydOHFcDRs2MvV6yWT5mjJlipo1a6aZM2cqLy9P7777riZPnqz58+cX/WIAAAAAxRIU5C7wODDw33+N93g8uvPOZpox43VJUnZ2tjIyMnTq1D9lGMZvXnPlFRs3b/5SCxfO14ABAxUV1UsXLpzPf01wcPBvftJVYF+F+Xeuyz9vGJ4rXmcYhvLy8nT5quyXn8vNzcl/fuDAwbrrro766qsteuON2fr7339QuXLldM89XTR27DhJUkZGhvLy8rxm6d79Xr399nylpaWqa9fukqTly5fpiy8+13339VH//m31008/XZHv8uXif82Vm7/d48nTvHlv5x96eebMGVWpUqXI38lvmTrs8MiRIxo9erRuuukmVa5cWY899piOHSv85mEAAAAA7NGkye364Yc9Onbs8n1533lnoebOfV233tpA586l6ODBA5IuH3r3e99++7XuuaeLoqLuU1hYVX333S55PB6fZStfvoJq1aqtL77YIEn64Yc9Onv2rOrXv1WVKlXSoUOHJElffrkp/zXDhg1RRkaGHnhgkB54YJD27/9RLVq00pdffqGUlBQZhqHp01/SsmXxXt/79tvv1C+/nNGnn36s7t3vlSR9880O/dd/9VX37vfK5XLp4MH98ngKlrjLuX6SJG3e/EX+9pYtW2vlyg8kSYcPH9LgwQOUlXVttwowtfKVm5ur7Ozs/ONEMzMzufEcAAAASoWsS3n6dFpTS/brC1WqhGnChEmaMOFpeTwehYdH6PnnpyooyK0pU17S5MkTFRgYdNXD9O67r7cmTZqgjRs/l9vtVpMmd+jkyRM+yfWryZOn6pVXXtKiRW/L7Xbr5ZdnyO12q0+f/po4cbwGDbpfrVq1VtWqVSVJI0eO0gsvTFJgYKBCQsro6aefVf36t2r48L9q9OgR8ng8atiwsYYM+XOR7/2nP3XVjh3bVKtWbUnSAw88qOnTpyk+/l2VL19Od9zRVCdPnlTt2nXyXzN48FBNmfKcEhM/UqdOkfnbx459Wi+//IIGDbpfkjRp0gsqX778Nf0uXIaJ9cM33nhDW7duVZ8+fSRJq1atUocOHfS3v/3tmt7sWpw9my6Pp+ilTdy4wsNDdeZMmr9joJiYozMwR+dgls7AHP0vIMClsLAKV31u796/q2bNm21OBKc4efKomjS5+mX8Ta18jRo1StWrV9eWLVvk8XjUp08f9evXz6chAQAAANzY/vu//6q0tNQrtvfu3U99+tjfD260PEUxVb6GDh2q2NhY9e3b1+o8AAAAAG5Qb765wN8RCrjR8hTFVPlKS0tTRkaGypUrZ3WefIUtA6NkCQ8P9XcE+ABzdAbm6ByldZZZ2XlKS83wdwyUEoZhcI0DXLOizugyVb7Kli2rzp07q1GjRgUK2FtvvVW8dF4MfeXvOn0+p+gfBAAApcKn05qKs6Rgh8DAQOXl5V5xeXegKJf/vbnykv6/MlW+OL8LAAAApUWVKpWVmnpOlStXlctl6s5MgAzDowsXzqly5cqF/oyp8tW7d2+dO3dOSUlJCggIUJs2bRQaWjoPeQAAAICzVa1aVZmZmfr552SZuK8wIElyuaTy5cvnXzL/akyVr88++0zPPvusGjVqpLy8PE2YMEGvv/662rVr57OwAAAAwI0gICBAN9/Mpebhe6bK16xZsxQXF6dGjRpJkvbu3auJEydq9erVloYDAAAAAKcwdRBrmTJl8ouXJDVp0oSrvwAAAADANTBVvtq2basFCxYoIyND2dnZev/999WgQQNduHBB58+ftzojAAAAAJR4pg47XLJkifLy8jRz5swC2xMSEuRyubRv3z5LwgEAAACAU5gqX3v37i30ucTERJ+FAQAAAACnKvaNC9555x1f5AAAAAAARyt2+TK4+QEAAAAAFKnY5YurHgIAAABA0YpdvgAAAAAARaN8AQAAAIANOOcLAAAAAGxQ7PIVHR3tixwAAAAA4Gim7vP1zTffaO7cubpw4UKBla7ExEQNHz7csnAAAAAA4BSmyteUKVPUt29f3Xbbbdd0dcPExETNnz9fOTk5evjhhzVo0KDrDgoAAAAAJZmp8uV2u/XnP//5mnZ86tQpzZo1S6tWrVJwcLAeeOABtW3bVrfeeut1BQUAAACAkszUOV8NGjTQ/v37r2nH27ZtU7t27VSpUiWVK1dO3bp109q1a68rJAAAAACUdKZWvpKTk9W3b1/VrFlTISEh+dsTExMLfc3p06cVHh6e/zgiIkJ79uwpRlQAAAAAKLlMla8xY8Zc846vdgn6azlfDAAA4PfCw0P9HcFnnPRZAJhjqny1adNG58+fV2ZmpgzDUF5eno4dO+b1NdWqVVNSUlL+49OnTysiIqJ4aQEAQKl25kyavyP4RHh4qGM+S0kVEOBSWFgFf8dAKWOqfM2ePVsLFiyQJAUGBionJ0e33nqr18MOO3TooLlz5yolJUVly5bV+vXr9cILL/gmNQAAAACUMKbKV0JCgr744gu9/PLLGjdunL7++mtt2rTJ62uqVaumMWPGaMiQIcrJyVG/fv105513+iIzAAAAAJQ4pspXlSpVFBERoXr16unHH3/Ufffdp9jY2CJfFx0drejo6GKHBAAAAICSztSl5oOCgnTs2DHVq1dPSUlJys3NVWpqqtXZAAAAAMAxTJWvESNGKCYmRpGRkfrss88UGRmpdu3aWZ0NAAAAABzD1GGHnTt3VufOnSVJH374oY4ePapGjRpZGgwAAAAAnMTUytfFixf1/PPPa+jQocrOztZ7772njIwMq7MBAAAAgGOYKl9Tp07VTTfdpLNnzyokJETp6el67rnnrM4GAAAAAI5hqnzt27dPY8aMUVBQkMqWLasZM2Zo3759VmcDAAAAAMcwVb4CAgr+WF5e3hXbAAAAAACFM3XBjdatW+vVV19VVlaWtmzZori4OLVt29bqbAAAAADgGKaWr5588kmVK1dOoaGhmjVrlho3bqxx48ZZnQ0AAAAAHMPUypfb7VabNm00atQonT9/XklJSQoJCbE6GwAAAAA4hqmVr1mzZmnOnDmSpKysLC1YsEBvvvmmpcEAAAAAwElMla8NGzbonXfekSRVr15dcXFx+uSTTywNBgAAAABOYqp85eTkyO125z92u91yuVyWhQIAAAAApzF1zleLFi00duxY9evXTy6XS6tXr1bTpk2tzgYAAAAAjmFq5SsmJkZVq1bV6NGj9fjjjys8PFwTJkywOhsAAAAAOIap8nX69Glt3bpVLpdLubm5Wrt2rU6ePGl1NgAAAABwDJdhGEZRPzR8+HBFRUWpd+/ekqSVK1cqISFBS5YssTwgAACAJGVl5yktNcPfMXwiPDxUZ86k+TtGqRYQ4FJYWAV/x0ApY+qcr7Nnz+YXL0nq27evFi9ebFWmf71nujyeInshbmD8weIMzNEZmKNzMEsAKLlMHXaYl5en8+fP5z9OSUmxLBAAAAAAOJGpla/BgwdrwIAB6tGjhyTp008/1dChQy0NBgAAAABOYqp8DRgwQHXr1tXWrVvl8Xg0adIkdejQwepsAAAAAOAYpsqXJLVv317t27e3MgsAAAAAOJapc74AAAAAAMVD+QIAAAAAG1C+AAAAAMAGlC8AAAAAsAHlCwAAAABsQPkCAAAAABtQvgAAAADABpQvAAAAALAB5QsAAAAAbED5AgAAAAAbUL4AAAAAwAaULwAAAACwAeULAAAAAGxA+QIAAAAAG1C+AAAAAMAGQf4OUJiAAJe/I8AHmKMzMEdnYI7OwSydgTn6F79/+IPLMAzD3yEAAAAAwOk47BAAAAAAbED5AgAAAAAbUL4AAAAAwAaULwAAAACwAeULAAAAAGxA+QIAAAAAG1C+AAAAAMAGlC8AAAAAsAHlCwAAAABsQPkCAAAAABtQvgAAAADABpQvAAAAALAB5QsAAAAAbED5AgAAAAAbUL4AAAAAwAaULwAAAACwAeULAAAAAGxA+QIAAAAAG1C+AAAAAMAGlC8AAAAAsAHlCwAAAABsQPkCAAAAABtQvgAAAADABpQvAAAAALAB5QsAAAAAbED5AgAAAAAbUL4AAAAAwAaULwAAAACwAeULAAAAAGxA+QIAAAAAG1C+AAAAAMAGlC8AAAAAsAHlCwAAAABsQPkCAAAAABsE+TtAYc6duyiPx/B3DBRDWFgFnT2b7u8YKCbm6AzM0TmYpTMwR/8LCHCpcuXy/o6BUuaGLV8ej0H5cgBm6AzM0RmYo3MwS2dgjkDpw2GHAAAAAGADyhcAAAAA2IDyBQAAAAA2oHwBAAAAgA0oXwAAAABgA8oXAAAAANiA8gUAAAAANqB8AQAAAIANKF8AAAAAYAPKFwAAAADYgPIFAAAAADagfAEAAACADShfAAAAAGADyhcAAAAA2IDyBQAAAAA2oHwBAAAAgA0oXwAAAABgA8oXAAAAANggqKgf2LNnj2JjY3XgwAGVKVNGDRs21NChQ9WwYUM78gEAAACAI3gtX5s3b9azzz6rhx56SL169ZIkff/99xo2bJhmzpypNm3aWBYs2xOoPI9h2f5hvVPnsnXJE+jvGCgmp88xJEhyefL8HQMAAJQCXsvXm2++qUWLFqlx48b52zp16qSOHTvqtddes7R8fbTtjFIv5lq2f1ivTFm3sjJz/B0DxeT0OQ68p7rKcAA2AACwgde/cly8eLFA8frVnXfeqbS0NMtCAQAAAIDTeC1fQUGFL4wZBocEAgAAAIBZHGwDAAAAADbwes7XoUOHFB0dfdXnkpOTLQkEAAAAAE7ktXwtXLjQrhwAAAAA4Ghey5eVVzMEAAAAgNLEa/m6++675XK5rvqcy+XS559/bkkoAAAAAHAar+Vrzpw5V2xLSkrSzJkz1a1bN8tCAQAAAIDTeC1ft99+e/4/ezwezZkzR0uXLtXUqVPVq1cvy8MBAAAAgFN4LV+/OnHihMaOHStJWrFiherUqWNpKAAAAABwmiLv8/XRRx+pT58+ateuneLj4yleAAAAAHAdvK58PfXUU1q/fr3GjBmj1q1b68cffyzwfJMmTSwNBwAAAABO4bV87dy5U2FhYVqyZImWLFlS4DmXy6UNGzZYGg4AAAAAnMJr+dq4cWOhz50/f97nYQAAAADAqYo85+v3fvrpJz333HOKjIy0IA4AAAAAOJOpqx1K0pYtW7R48WJt27ZNLVu21Pz5863MBQAAAACO4nXlKzs7W8uWLdO9996rJ598UnXr1lV4eLji4uLUvn37Ineenp6uqKgoHT9+3GeBAQAAAKAk8rryFRkZqaZNm+qJJ55QZGSkgoODtXnzZlM73r17tyZOnKgjR474IicAAAAAlGheV75atWql3bt3a926ddq+fbs8Ho/pHS9fvlyTJk1SREREsUMCAAAAQEnndeVr7ty5On36tJYvX66YmBh5PB5lZ2fr+PHjql27ttcdv/jiiz4NCqCgjxePs+V9AgJc8ngMW97LH75NCFaAnPv5fhUcHKRLl3It2/+8eQss2zcAAE5R5AU3IiIiNHr0aI0cOVKff/65li1bpu7du+tPf/qTXn/9dcuChZQJUhmPy7L9wx5lyrr9HcGxAgLs+37Y+V52C3C5FOwO9HcMWwQHm77G0jULDw+1bN+4Er9vZ2COQOlj+k/iwMBAdevWTd26ddPhw4f1/vvvW5lL2Vm5ysq07v/SwnplyrqVlZnj7xiO1WPIK7a8j9PnOPCe6ioTkOfvGJYLDw/VmTNplu3fyn2jIKtnCXswR/8LCHApLKyCv2OglLnm+3xJ0i233KLdu3f7OgsAAAAAONZ1lS9J2r9/vy9zAAAAAICjWXcCwL9s3LjR6rcAAAAAgBveda98AQAAAADM87ryNXXq1KtuNwxDOTnOPQEfAAAAAHzNa/mqVKlSoc+NGDHC52EAAAAAwKm8lq/Ro0fblQMAAAAAHM1r+XrmmWcKfc7lciAxsAQAABkiSURBVOmll17yeSAAAAAAcCKv5atBgwZXbDt37pxiY2NVq1Yty0IBAAAAgNN4LV/Dhg0r8Hjbtm16+umnFR0drYkTJ1oaDAAAAACcxNR9vnJzc/Xaa69p9erVmjx5srp37251LgAAAABwlCLL19GjRzVmzBiVK1dOq1evVo0aNezIBQAAAACO4vUmyytWrFD//v3VpUsXxcXFUbwAAAAA4Dp5XfmaOHGiAgICtGDBAi1cuDB/u2EYcrlc2rVrl+UBAQAAAMAJvJavDRs22JUDAAAAABzNa/nicvIAAAAA4Btey1fz5s3lcrmu2M5hhwAAAABwbbyWrzVr1tiVAwAAAAAcrdiHHQ4ePFhxcXE+CwQAAAAATmTqJsvepKen+yLHFXp1CFeex7Bk37BHcHCgLl3K83cMFJPT5xgSJMnj7xQAAKA0KHb5uto5Yb4QEpAnjyhfJVl45XI6cybN3zFQTI6fI8ULAADYxOtNlgEAAAAAvkH5AgAAAAAbUL4AAAAAwAbFLl+GwXlZAAAAAFAUU+Wrd+/e+uCDD5SZmXnFc/Hx8T4PBQAAAABOY6p8xcTEKCkpSV26dNGUKVN08ODB/OfKly9vWTgAAAAAcApTl5pv0aKFWrRoodTUVCUmJmrkyJGKiIjQQw89pB49elidEQAAAABKPNPnfKWmpiohIUHLly9XaGioevTooYSEBI0bN87KfAAAAADgCKZWvsaOHavNmzcrMjJSkydPVvPmzSVJAwcOVIcOHSwNCAAAAABOYKp8NWjQQBMmTFCVKlUKvjgoSEuXLrUkGAAAAAA4ianDDpOSkq4oXvfff78kqX79+r5PBQAAAAAO43Xl67HHHtPhw4eVnJys6Ojo/O25ubkKCOD+zAAAAABgltfyNW7cOJ04cUIxMTGKiYnJ3x4YGKgGDRpYHg4AAAAAnMJr+apdu7Zq166tdevWyeVy2ZUJAAAAABzHa/kaOHCgli5dqhYtWhQoX4ZhyOVyadeuXZYHBAAAAAAn8Fq+Zs+eLUlas2aNLWEAAAAAwKm8lq+IiAhJUq1atfTdd9/pwoULBZ6vVauWdckAAAAAwEFM3edrzJgxSkpKyi9jkuRyudSpUyfLggEAAACAk5gqXz/88IM2bNig4OBgq/MAAAAAgCOZullXvXr1lJuba3UWAAAAAHAsUytfAwYMUK9evdS8eXMFBf37JdOmTbMsGAAAAAA4ianyNX36dN11112qW7eu1XkAAAAAwJFMla+goCBNnjzZ4igAAAAA4Fymzvlq1qyZNm3aZHEUAAAAAHAuUytfO3bs0IoVK+R2u+V2u2UYhlwul3bt2mV1PgAAAABwBFPlKzY21uocAAAAwA3B4/EoOTlZFy9elGH4O03p5nJJ5cuXV506dRQQYOqgvRua1/K1fft2tW/fXnv37r3q87Vq1bIkFAAAAOAvv/zyi3JzPapevY5crpL/F/6SzDA8Skn5Rb/88osiIiL8HafYvJavjz/+WO3bt9e77757xXMul0tdu3a1LBgAAADgDykp5xQWVo3idQNwuQJUsWJlpaSccn75mjp1qiRp0aJFCgkJKfDcqVOnrEsFAAAA+EleXp4CA02dnQMbBAYGKTc3z98xfMJUne/fv7+OHj2a/3jjxo3q06ePZaEAAAAAf3K5XP6OgH9x0ixMVfohQ4Zo0KBBGjdunHbv3q2tW7dq3rx5VmcDAAAA/C5PAcrK8f1+y7ilQHl8v2MvpkyZpBYtWioqqpet74vLTJWvfv36KTw8XI8++qiqVq2qjz76SJUrV7Y6GwAAAOB3WTnS0o0/+3y/A++urvJun+8WNzBT5Wvp0qWaM2eOJkyYoAMHDujBBx/U9OnTdccdd1idDwAAACjVdu5M0qJFbykwMEinT5/Sbbc10Z///BeNG/c/qlixkoKDgzV79huaO/d17dq1Ux5Pnnr2jNbAgYNlGIZmz56pr77aoqpVw+Xx5KlFi5a6eDFdMTHP6uzZXyRJw4ePUMeOnQrNMGXKJJUtW0a7d3+n9PQ0PfHEk/r004/1j38cVMeOkXr88f9RXl7eVTPk5uZq+vRpOnToH0pJSVHdujfr5ZdnKCUlRePHj1W9evV14MB+ValSRS++OF0VK1a061drO1PlKz4+XrGxsWrYsKEkaf369Xr00Uf11VdfWRoOAAAAgPT3v+/VkiVLVbfuzZow4Wl99dVWHT16RKtWrVHNmjW1atUKSdKSJe/p0qVLevzxUfrjH29TSkqKDhzYr6VLP1BaWroGDx4gSdq06QvVqFFDM2fO0eHDh7RmTYLX8iVdvgR/XNz7+vjjRE2dOlnLl3+okJAQRUd31/Dhj2j9+nVXzWAYhtzuIC1aFCuPx6NRo0Zo27av1LjxH3Xw4AFNmDBJjRo11vjxT2rduk91//0PWPmr9CtT5WvlypUKCQlRbm6uDMNQ165ddeedd1qdDQAAAICkZs2a6+ab/yBJ6t69pxISVqpy5SqqWbOmJOnbb7/WwYP7tXPnt5KkzMwM/eMf/9CRI4cUGXm3goLcqly5sjp0+A9J0h13NNVbb83TmTNn1KHDXRo27JEiM7Rv30GSVL16DdWrd6uqVKkiSbrpppuUmppWaIZ+/e5XxYqVtGLF+zpy5IiOHz+mzMwMSVLlylXUqFFjSVL9+vWVmnrBR7+xG5Op8pWenq5Ro0Zpx44dysvLU6tWrTRjxgyrswEAAACQClz63jA8CgwMKnArqMsrSo+rc+d7JEnnz59TmTJlNW/ebBmG5zf7CZQk1a1bV8uWrdKOHdu0detmLV0ap2XLVnq9smBQ0L9PUPt1P79VWIbNm7/UwoXzNWDAQEVF9dKFC+dlGIYkKTg4+Dd7cOVvdypTl5qfMmWKmjVrpm3btmnbtm1q3bq1Jk+ebHE0AAAAAJK0e/d3On36tDwejz75ZE3+KtSvWrZsrYSE1crNzVFGRoZGjBiuvXt/UJs2bbVhw+e6dOmSUlNTtWPHNknSBx8s08KFb+mee7roqaee0blzKUpPTy9WxsIyfPvt17rnni6KirpPYWFV9d13u+Tx2HuVxxuFqZWvI0eOaPbs2fmPH3vsMfXs2dOyUAAAAAD+rWrVqpoyJUZnzpxR69Zt1bp1W8XG/m/+83369FVy8jENGfKg8vLy1LNnL7Vs2UrS5fPFHnywv8LCwvSHP9STJN17b5RiYp7VoEH3KzAwSMOHj1BoaGixMhaWoWLFipo0aYI2bvxcbrdbTZrcoZMnTxTrvUoql2Fiba9nz55atWpV/tJmZmam+vfvrzVr1lgW7OzZdHk8zl52dLrw8FCdOZPm7xgoJuboDMzROZilMzBH/wsIcCksrMJVn9u79++qWfPm/Mf+vs/X5asdvq358xf6PkQJcfLkUTVpcpu/YxSbqZWve++9Vw8//LD69OkjSVq1apW6detmaTAAAADgRhAoT6m4H9fcubP0zTdfX7G9cePbNGHCc35I5DymyteoUaNUvXp1bdmyRR6PR3369FG/fv2szgYAAACUei1btso/hNBKf/vbGMvfo7QzVb6GDh2q2NhY9e3b1+o8+bI9gcrjsMMS7dS5bF3yXHklHJQszNEZmKNzlNZZhgRJLk+ev2MAQLGYKl9paWnKyMhQuXLlrM6T76NtZ5R6Mde294PvlSnrVlamBQdIw1bM0RmYo3OU1lkOvKe6ypi6RjPgG4ZheL3sOuzjpMvPmypfZcuWVefOndWoUaMCBeytt96yLBgAAADgD2XLllFa2gWFhlakgPmZYRhKS7ugsmXL+DuKT5gqX5zfBQAAgNKiTp06Sk5O1j//eczfUaDLZbhOnTr+juETpspX7969de7cOSUlJSkgIEBt2rQp9n0AAAAAgBuR2+1WvXr1/B0DDmTq6OnPPvtMXbt2VWxsrBYtWqQuXbpox44dVmcDAAAAAMcwtfI1a9YsxcXFqVGjRpKkvXv3auLEiVq9erWl4QAAAADAKUytfJUpUya/eElSkyZNOPkQAAAAAK6BqfLVtm1bLViwQBkZGcrOztb777+vBg0a6MKFCzp//rzVGQEAAACgxDN12OGSJUuUl5enmTNnFtiekJAgl8ulffv2WRIOAAAAAJzCVPnau3dvoc8lJib6LAwAAAAAOFWx7xX/zjvv+CIHAAAAADhascuXYRi+yAEAAAAAjlbs8sVVDwEAAACgaMUuXwAAAACAolG+AAAAAMAGnPMFAAAAADYodvmKjo72RQ4AAAAAcDRT9/n65ptvNHfuXF24cKHASldiYqKGDx9e6OsSExM1f/585eTk6OGHH9agQYOKnxgAAAAASiBT5WvKlCnq27evbrvtNtNXNzx16pRmzZqlVatWKTg4WA888IDatm2rW2+9tViBAQAAAKAkMlW+3G63/vznP1/Tjrdt26Z27dqpUqVKkqRu3bpp7dq1Gj169LWnBAAAAIASzlT5atCggfbv369GjRqZ3vHp06cVHh6e/zgiIkJ79uy59oQAAJj08eJx/o5guYAAlzye0nexq28TghUg53zu4OAgXbqU6+8YN4R58xb4OwJgG1PlKzk5WX379lXNmjUVEhKSvz0xMbHQ11ztKojXckPmkDJBKuPhBs4lXZmybn9HgA8wR2coDXMMCCgdf26Uls/5WwEul4Ldgf6O4VPBwab+GuZ44eGh/o4A2MbUt37MmDHXvONq1aopKSkp//Hp06cVERFh+vXZWbnKyuT/CJVkZcq6lZWZ4+8YKCbm6AylZY49hrzi7wiWKy2z/L2B91RXmYA8f8fwmfDwUJ05k+bvGDcEf/0eAgJcCgur4Jf3Rull6lLzbdq0UcOGDVWnTh3Vrl1bNWrUUE6O9//wd+jQQdu3b1dKSooyMzO1fv16dezY0SehAQAAAKCkMbXyNXv2bC1YcPl43MDAQOXk5OjWW2/1ethhtWrVNGbMGA0ZMkQ5OTnq16+f7rzzTt+kBgAAAIASxlT5SkhI0BdffKGXX35Z48aN09dff61NmzYV+bro6GhuwgwAAAAAMnnYYZUqVRQREaF69erpxx9/1H333aejR49anQ0AAAAAHMNU+QoKCtKxY8dUr149JSUlKTc3V6mpqVZnAwAAAADHMFW+RowYoZiYGEVGRuqzzz5TZGSk2rVrZ3U2AAAAAHAMU+d8de7cWZ07d5Ykffjhhzp69Og13XAZAAAAAEo7UytfFy9e1PPPP6+hQ4cqOztb7733njIyMqzOBgAAAACOYap8TZ06VTfddJPOnj2rkJAQpaen67nnnrM6GwAAAAA4hqnytW/fPo0ZM0ZBQUEqW7asZsyYoX379lmdDQAAAAAcw1T5Cggo+GN5eXlXbAMAAAAAFM7UBTdat26tV199VVlZWdqyZYvi4uLUtm1bq7MBAAAAgGOYWr568sknVa5cOYWGhmrWrFlq3Lixxo0bZ3U2AAAAAHAMUytfbrdbbdq00ahRo3T+/HklJSUpJCTE6mwAAAAA4BimVr5mzZqlOXPmSJKysrK0YMECvfnmm5YGAwAAAAAnMVW+NmzYoHfeeUeSVL16dcXFxemTTz6xNBgAAAAAOImp8pWTkyO3253/2O12y+VyWRYKAAAAAJzG1DlfLVq00NixY9WvXz+5XC6tXr1aTZs2tTobAAAAADiGqZWvmJgYVa1aVaNHj9bjjz+u8PBwTZgwwepsAAAAAOAYpsrX6dOntXXrVrlcLuXm5mrt2rU6efKk1dkAAAAAwDFchmEYRf3Q8OHDFRUVpd69e0uSVq5cqYSEBC1ZssSyYCfPZCrPU2Q03MCCgwN16VKev2OgmJijMzBH5yitswwJklwe53zu8PBQnTmT5u8YpVpAgEthYRX8HQOljKlzvs6ePZtfvCSpb9++Wrx4sVWZJEkhAXnyiPJVkoVXLscfLA7AHJ2BOTpHqZ2lx98BAKD4TB12mJeXp/Pnz+c/TklJsSwQAAAAADiRqZWvwYMHa8CAAerRo4ck6dNPP9XQoUMtDQYAAAAATmKqfA0YMEB169bV1q1b5fF4NGnSJHXo0MHqbAAAAADgGKbKlyS1b99e7du3tzILAAAAADiWqXO+AAAAAADFQ/kCAAAAABtQvgAAAADABpQvAAAAALAB5QsAAAAAbED5AgAAAAAbUL4AAAAAwAaULwAAAACwAeULAAAAAGxA+QIAAAAAG1C+AAAAAMAGlC8AAAAAsAHlCwAAAABsQPkCAAAAABsE+TtAYQICXP6OAB9gjs7AHJ2BOToHs3QG5uhf/P7hDy7DMAx/hwAAAAAAp+OwQwAAAACwAeULAAAAAGxA+QIAAAAAG1C+AAAAAMAGlC8AAAAAsAHlCwAAAABsQPkCAAAAABtQvgAAAADABpQvAAAAALDBDVW+EhMTde+996pLly6Kj4/3dxz8y5AhQ9SzZ0/dd999uu+++7R79+5CZ7Vt2zZFR0era9eumjVrVv72ffv2qW/fvurWrZsmTJig3NxcSdLJkyc1aNAgde/eXSNHjtTFixdt/3xOl56erqioKB0/flyS72aUmpqqv/71r+rRo4cGDRqkM2fOSJIuXbqkp556Sj169FDv3r31008/2fyJnen3c3zmmWfUtWvX/O/lZ599Jsn6+aJ45s2bp549e6pnz56aPn26JL6TJdHV5sh3EoApxg3i559/Njp37mycO3fOuHjxohEdHW0cPHjQ37FKPY/HY/zHf/yHkZOTk7+tsFllZmYanTp1Mo4dO2bk5OQYw4YNMzZt2mQYhmH07NnT+L//+z/DMAzjmWeeMeLj4w3DMIy//vWvxpo1awzDMIx58+YZ06dPt/kTOtt3331nREVFGU2aNDGSk5N9OqPnn3/eePvttw3DMIzVq1cbjz/+uGEYhrFo0SIjJibGMAzD+Oabb4x+/frZ94Ed6vdzNAzDiIqKMk6dOlXg5+yYL67fV199ZQwYMMDIzs42Ll26ZAwZMsRITEzkO1nCXG2O69ev5zsJwJQbZuVr27ZtateunSpVqqRy5cqpW7duWrt2rb9jlXqHDh2Sy+XSI488ol69eikuLq7QWe3Zs0c333yz6tSpo6CgIEVHR2vt2rU6ceKEsrKy1KxZM0lSnz59tHbtWuXk5Ojbb79Vt27dCmyH7yxfvlyTJk1SRESEJPl0Rps2bVJ0dLQkKSoqSps3b1ZOTo42bdqkXr16SZJat26tc+fO6eTJk3Z/dEf5/RwzMjJ08uRJxcTEKDo6WnPmzJHH47Flvrh+4eHhGj9+vIKDg+V2u1W/fn0dOXKE72QJc7U5njx5ku8kAFOC/B3gV6dPn1Z4eHj+44iICO3Zs8ePiSBdPsyhffv2mjx5srKysjRkyBD16NHjqrO62gxPnTp1xfbw8HCdOnVK586dU4UKFRQUFFRgO3znxRdfLPDYlzP67WuCgoJUoUIFpaSkXHVfP//8s2rWrGnZ53S638/x7NmzateunaZMmaJy5cppxIgRWrFihcqVK2f5fKtVq2b1x3WsBg0a5P/zkSNH9Mknn+ihhx7iO1nCXG2O7733nr755hu+kwCKdMOsfBmGccU2l8vlhyT4rebNm2v69OkqV66cqlSpon79+mnOnDlX/JzL5Sp0hte6HdaxekYBAVf/T0ph23F96tSpozfeeENhYWEqW7asHnroIX355Zd+my+uzcGDBzVs2DA9/fTTqlu37hXP850sGX47x3r16vGdBGDKDfOtrVatmn755Zf8x6dPn84/xAb+k5SUpO3bt+c/NgxDtWrVuuqsCpvh77efOXNGERERqlKlitLT05WXl1dgO6zjyxlFRETkvyY3N1fp6emqVKmSIiIiCpwIzlx9b//+/Vq3bl3+Y8MwFBQUZMt8UTw7d+7Uww8/rLFjx6p37958J0uo38+R7yQAs26Y8tWhQwdt375dKSkpyszM1Pr169WxY0d/xyr10tLSNH36dGVnZys9PV2rV6/Wq6++etVZNW3aVIcPH9bRo0eVl5enNWvWqGPHjqpVq5ZCQkK0c+dOSdKHH36ojh07yu12q1WrVvrkk08KbId1fDmjTp066cMPP5QkffLJJ2rVqpXcbrc6deqkhIQESZfLe0hICIc3+ZhhGHrppZd04cIF5eTk6P3331eXLl1smS+u3z//+U+NGjVKM2bMUM+ePSXxnSyJrjZHvpMAzHIZV1vj9pPExES9/fbbysnJUb9+/fTII4/4OxIkvf7661q3bp08Ho8efPBBDR06tNBZbd++XdOmTVN2drY6deqkZ555Ri6XSz/++KMmTpyoixcv6rbbbtO0adMUHBysEydOaPz48Tp79qxq1KihmTNnqmLFin7+xM5z9913a8mSJapdu7bPZnT+/HmNHz9eycnJCg0N1YwZM1S7dm1lZ2frueee0w8//KDg4GBNnTpVTZo08fevwBF+O8f4+HjFx8crNzdXXbt21ZNPPinJd9/BwuaL6zd16lStXLmywKGGDzzwgP7whz/wnSxBCpujx+PhOwmgSDdU+QIAAAAAp7phDjsEAAAAACejfAEAAACADShfAAAAAGADyhcAAAAA2IDyBQAAAAA2oHwBAAAAgA0oXwAAAABgA8oXAAAAANjg/wMtTshTR9BrwgAAAABJRU5ErkJggg==\n",
"text/plain": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# prepare figure \n",
"fig, (ax0, ax1) = plt.subplots(nrows=2, ncols=1, figsize=(12,7), sharex=True)\n",
"\n",
"# plot ground truth\n",
"preds_quality.plot('ocean_proximity_INLAND', 'median_house_value', 'barh', ax=ax0)\n",
"# put legend outside plot\n",
"ax0.legend(loc='upper left', bbox_to_anchor=(1.0, 0.5))\n",
"\n",
"# plot preds\n",
"preds_quality.plot('ocean_proximity_INLAND', 'preds_mean', 'barh', xerr='preds_std', alpha=0.6, ax=ax1)\n",
"# put legend outside plot\n",
"ax1.legend(loc='upper left', bbox_to_anchor=(1.0, 0.5))\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The error bars in the plot indicate how confident the model is at predicting each category. Alternatively, we can compare the _distribution_ of values to inspect how close the predictions match the ground truth. For example, for housing districts where `ocean_proximity_INLAND` is 0 we have:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhAAAAEJCAYAAADFMR5HAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de1yUVf4H8M8wMwwgjCgOKoi2XhLRUlMS1CRbRRQRb1teVn7mppaVpa13LLM0Ym0tV90sK7fUzUuKRmZZplthimSaipdCEEHlIldhhrmc3x/EyMgA8yAMt8/79fIF88z3PM95Do/Dl+ec5xyZEEKAiIiISAKH+q4AERERNT5MIIiIiEgyJhBEREQkGRMIIiIikowJBBEREUnGBIKIiIgkYwJBREREkinquwJ3y8m5DZOp8U9N4eHhiuzswvquRoNQvi0+PLsNyflXcZ+6I2b0mmrzPhYtmo9ffz2NBx7ojTff/Kek43896wvcTLiOtv3aI/i9UEllaxuvC0tsjztq2hYODjK0atWiDmpEVLUGl0CYTKJJJBAAmsx51IaytsjV5iOz6BbcHd0ltU9GRgauXbuGtm3bS27XwhuFyEvJQwtvtwbxM2kIdWhI2B53sC2oMWEXBhEREUnGBIKIiIgka3BdGNS0DWjfH93cO6O1c2tJ5caMGYf+/R+Gl5e35GP6TuoJr4EdoO7YUnJZoqbCZDIhNTUVt2/fBldAourIZECLFi3g4+MDBwfr9xqYQJBdBbbvX6Ny4eHja3xM30k9a1yWqKnIysqCwWBCu3Y+kMl485mqJoQJt25lISsrC56enlZjeBURETUDt27lQK1uxeSBbCKTOaBly1bIycmpNIZXEhFRM2A0GiGX86Yz2U4uV8BgMFb6Pq8msqtj10/iVvEttHZuLak7Y9++PUhPT4OXl7fk7owLn55D/tU8qDu2ZHcGNWsymay+q0CNSHXXC+9ANAMGIUNhCWz6ZxB1+wFz/PpJHEj+Bsevn5RUbv/+vdi0aQP2798r+ZgXPj2Hk2t+woVPz0kuS0S1Lz09HQEBDyEq6nWL7ZcuXURAwEOIjd1v9zrdvl2I6dOnYtq0Sbh6NaXOjvPMMzORkHASiYnnsWrVykrjCgsLsHDhfABAZmYm5s17vs7qVFO8A9EMaPUC8RfybYr191XD1bGOK0REzV7Llu746ae4P7pW5ACAb775Gq1ataqX+ly6dBFKpRJbtmyzy/F69PDDsmUvV/p+fn4BLl26CADQaDRYu/ZfdqmXFEwgiIjI7pydnXH//d3xyy8/o18/fwDA8ePH4O8/wBxz7NiPeP/9d2EwGNC+vReWLl2Oli3d8e23h7B9+yfQ6XTQ6XRYunQ5+vbth2eemQk/v544ffoUcnNzMH/+IgwcOMjiuNnZ2Vi9+lXcuHEDcrkCzzzzLLp374FVq15FdnY2/v73F7Fmzdvm+NjY/Thy5DDy8/Nx61Y2Bg8eghdemI+ff07Ahg3vwGg0okuXrvj73xdjzZoo/P77bzCZTJg2bTqCg0NQUlKC1atXIjHxPNq390JeXi4AICHhJDZv3oR///t9XLp0EVFRq6DTaaFWq/Hqq6vwz39GIysrE4sWvYQXXngJc+bMREzMF1brHxg4CO+//y4yMzORmnoVN25cx5gxY/Hkk0/h8uVLiIp6HUajEY6OKkRGrkDHjh1r5WfIBIKIqBmKidmDmJg9Vcb4+vbA4sXLzK8vXDiPqKjVVZYZO3Y8xo61bZzSn/88HIcPf4N+/fxx/vw5dO3azTxHRU5ODjZu/Bc2bHgParUae/fuxvr167BkSST27t2Nt956B+7urfD55zHYuvVj9O3bDwBgMOixefN/8P33R7Fp04YKCcQ//xmNfv0expQpf0Va2jXMnj0D//nPf7FkycvYvHmTRfJQJjHxPD7++L9Qq9WYM2cmjhw5DLW6Ja5eTUFMzBdwdXXDhg3r0L17D7z88krcvl2ImTOfRM+evXDkyGEAwI4de3D16lVMm/ZEhf2/8soyPPvsXAwePASffbYLO3Zsx/z5CzFnzky8+eZbSE9Pr7b+APDbb5exadMHKCgowMSJYzBx4hP49NNtmDJlGv785+E4dOgrnDt3hgkEERHVXFraNZw8eUJSmfz8gmrL+Ps/bPP+Bg8egk2bNsJkMuGbb77GsGHBOHToawDAuXO/4ubNG3j22VkASifCUqvVcHBwwJtvvoXvv/8frl5Nwc8/n4SDg9y8z4CAgQCALl26Ij+/YtftyZPxWLIkEgDg7d0BPXs+gHPnzqJFi8oXJHvkkSHw8PAAAAwbNgIJCfEYOnQYOna8D66ubgCA+Pjj0Gq1iI3dBwAoLi5GUtLv+Pnnkxg7dgIAoGPHjnjggQct9p2bm4Ps7CwMHjwEADBhwl8AwCJpsKX+ANCvX38olUq0bt0aarUahYUFGDToEfzjH1H46ac4DBr0CB57bFil5ykVEwgiombI27sD+vev+pe9r28Pi9dqtVu1Zby9O9hchxYtWqBbt/tx+vQpnDwZjzlznjcnECaTCQ8+2Md8R0Cn06GoqAhFRUV48slpCAkJRZ8+D6Fr127YtWuHeZ+Ojqo/vpNBWJlyUwjTXa8FjEZDlfUsG6NRVr7stUqlMm83mUxYseJ1c5tlZ2ejZUs1YmL2WNSj/L4AQKGw/DWs0+mQlZVZ6XwdVdXf0bH8ADYZhAAee2wYevV6ED/++D/s2LEdcXE/YunS5VWer62YQBARNUNSuhrK+Pr6YcuWrbVajz//eTg2bvwXevToYfHLtGfPXli9eiWuXk1Bx46d8OGH7yMzMxOPPz4JMpkM06fPAAC88cZrMJkqn6vgbv36+WP//n3mLoAzZ37BwoVLkJJS+ZMXx47FobCwAEqlI77++ivMmvW01f3u2bMbS5cuR1ZWJqZNm4z33/8IDz88AF999SUGDx6Cmzdv4Ndfz1iUc3V1g6dnWxw//hMGDAjAl19+gVOnEvDss3NhNFY8r8rq/9tvl63WfdmyRRg+fATGjZuI++77E95++y2b26o6TCCIiKjeDB48BKtWrcSsWc9YbPfwaINly17BsmWLYDKZoNF44tVXX4erqxvuv787nnhiPJycnNC3bz/cuHHD5uPNn78QUVGv4YsvSh8VXbr0ZbRpo6kygWjVqhXmzZuLvLxchISEIiBgIBISLB9Ff+qpWYiOfgNTpvwFRqMRzz33Ajp08MGECX/B77//jkmTJqBdu/bo3LlLhf2vWPE6oqPfwPr1b6NlS3esWPEa3N3d0a5dO8yZMwuRkSuqrX9lpk//G1avXokPP9wMhUKOF154yea2qo5MWLvHc5f169fjyy+/BAAEBQVh4cKFWLJkCRISEuDs7AwAeO655zB8+HDExcXhjTfegE6nw8iRIzFv3jxJFcrOLoTJ1PhXetFo3JCZWVDf1QBQOr9DfT7GWb4tdl/aj2uF6ejg6oWJ94+xeR/R0atx8WIiunfvgYULl0o6/g+R3yHrbCba9NJg8OtDJZWtbQ3pumgI2B531LQtHBxk8PBwrTbu3Lnz8PLqVJOqNWuxsfvx888JePnlV+u7KvUiPT0FPXv6WX2v2jsQcXFx+OGHH7B3717IZDI89dRTOHToEM6ePYutW7daLLKh1WqxdOlSfPLJJ2jfvj1mz56No0ePIigoqPbOhho1KUlDeVKThvLqO2kgImqKqp2JUqPRYPHixXB0dIRSqUSXLl2Qnp6O9PR0LF++HGFhYVi3bh1MJhPOnDmDTp06wcfHBwqFAmFhYTh48KA9zoOIiKjWjR49ptnefahOtXcgunXrZv4+OTkZBw4cwPbt23HixAmsXLkSLi4umD17Nnbv3g0XFxdoNHf6Yjw9PXHz5s26qTkRERHVG5sHUV6+fBmzZ8/GokWL0LlzZ2zYsMH83rRp0xATE4OQkJAK5aQu3mJLX15jodG41XcVAACmHB1c3ZxsinVxcYSmlar6QInK2iI5JxW39cVooXTGfa18bC5/9uxZ5OfnQ61Wo1evXpKOfeOXG9DmauHk7oR2fdpJKlsXGsp10VCwPe5gW1BjYlMCkZCQgLlz52Lp0qUIDQ3FxYsXkZycjBEjRgAofQ5VoVCgbdu2yMrKMpfLyMiwGCNhCw6irH1FJUBhgdam2OJiFa4Uldi8byelDApZ1T+v8m3x/s+f4nJuErq5d8aLD1V8FKoyS5dGIiEhHv36+eODDz6xuRwAfP7sF0iPuwavgR0wNuZxSWVrW0O6LhoCtscddT2Ikqi2VZtAXL9+Hc8++yzWrl2LwMBAAKUJw+rVqxEQEAAXFxfs2LED48aNQ+/evXHlyhWkpKSgQ4cOiI2NxYQJE+r8JKj2lBgEfrls2xMbABffIiJqrqpNID744APodDpERUWZt02aNAmzZs3C5MmTYTAYEBwcjNGjRwMAoqKi8Pzzz0On0yEoKMhqtwYRERE1btUmEJGRkYiMjLT63tSpUytsCwwMxP799l/LnYiIGofyK1E2pzoEBDyEn3762WKbEAIffbQZR44chlarxZNP/g0jR5b+Qb51638QG1v6+3TMmHGYMuWvAIDly5fg999/w+DBQzBnzvMAgE8+2YIuXbpi4MDBVo+dlZWJf/3rbVy6dBFyuRxt27bF/PkL4e3dAenp6ebVPqWo9jFOIiIiqhsHDx7AiRM/4YMPPsa///0+1q17GwUFBUhNvYrPPtuFLVu24aOPtmLnzu1ITb2Ky5cvobCwENu378KPP36PwsIC5OXl4ezZXytNHoqLi/HMMzPRt+9D2L59F7Zu3YHhw0Mwd+4cGAz6GtedU1kTETVDcWnxOJYeX2VMBzdvPOEbbn6dmp+GnRf3VVkm0MsfA739a1yvLVs+wMGDByCXy/HwwwF47rkXsHDhfEyY8DgGDhyEf/97PS5evIC3316PrKxMzJ07B9u378KBA7H49NPtEMIEX98e+PvfF0OlUiEk5DH4+vZAdnY2PvroEygUSvOx8vJy8eKLzyIzMxN+fr2wYEHpnEc//PA/8yqh3t4dsGjRMnh4eGDs2FBs3Pg+vLy8LO5gPPPMTPj59cTp06eQm5uD+fMXYeDAQUhPT8eKFZEoLi5Cr14PWD3fb775GlOmRECpVMLDow02bfoQKpUKQggYDHqUlJRACAEhShfeUigU0OtLYDAYYDAYIJcr8OGH7+L//m9GpW166NBXaNNGY14VFABCQkbB0dERJSVMIIiISILs4lu4lJMkqUyRobjaMve3qrjWg63i4n7A998fxZYt26BQKLBkyQLs3bsbgwY9gpMnT2DgwEH45ZefcfPmTRiNRvz0UxwCAwchKel37Nu3B++//xFUKhU2bvwXtm37BDNmPIXc3FxMm/Yk+vXrX+F46elpiIp6Cz4+PoiMXIy9e3dj+PAQvPnmKmza9BG8vLywdet/8NZbb2L16ugq624w6LF583/w/fdHsWnTBgwcOAhvvRWF0NAwhIePw5dfxmLv3s8qlLt2LRXJyVfw8ccfoqREj4iI6ejYsSM6duyE4cNDMHZsKACBMWPGon17LwBA167dMH36VIwfPxE5OTnIybkFP7+eldbt0qUL6Nmz4uPvZUt75+bmVnlulWECQUTUDHk4t8b9rTpXGdPBzdvitYvCudoyHs6ta1ynkyfjERwcAien0nlrwsLC8cUXn2P+/AVYsGAebt++DQDo1u1+XLx4AceOxWHixMeRkBCP1NRUPPXU/wEA9Ho9unf3Ne/X2i9PAOjT5yF07NgRADBixCh88cV+eHt3gJ9fL3h5lf6yHjt2PD7++KNq6x4QMBAA0KVLV+Tnlz7J9vPPCVi58g3z/letWlmhnNFoxG+/Xca7725GdnY2Zs2agfvv90VaWiouXEhEbOxXEMKEefPm4ptvvsawYcF48cW/m8uvXPkKZsyYid27d+LYsR/Qs+eDmDHjKYtjyGQOVpc2v1dMIIiImqGB3tK7GnzU3njJf04d1QgwmUwWr4UQMBqNaNu2HUwmE7777ls8+GAftG7dGidPnsCFC4l48ME+uHTpIv785+F46aWFAICioiKLpbDLEpK7yeXy8keDQqGoMA+REIDBULqv0okRS983GAwWcY6OZRPwycy/rGUyGYQwmb93cKg47NDDwwOPPfZnKBRKtG3bDr16PYBLly7g1KkEDB36GFxcXAAAwcEjcOpUAoYNCzaXvXDhPFq0aIHWrT2wZ88ubNu2E/PmPWdeAr1Mjx5+5tU7y1u1aiUmT54KJydnq+1THQ6iJLt68aGnseGxaEmTSAHABx98gl9+uSB5EikAGBvzOOZkzK/3SaSIqGr9+/vj668PQqvVwmAwIDZ2v7nrITBwELZs2YyHHuqH/v0fxq5dn6Jnz16Qy+V46KH+OHr0O9y6dQtCCERHr8ann26r9nhnzvyCGzeuw2Qy4YsvPoe//8Po2bMXzp79Fenp6QCAmJjPzHVwd3dHUtLvAIDvvz9S7f79/Qfg4MEDAIDvvjuMkpKKk/QNHjwE3357CEII5OXl4ty5X3H//d3Rrdv9+PHH72E0GmEw6HHs2I/o0cOym+LDDzdjxoyZFkmKTOYAnU5nEffYY8Nw/fp17N8fY94WG7sPp06dRIcOts8IfDfegSAiIrs7ffoUhg4dZH4dEjIKixYtw6VLl/Dkk3+F0WjEgAGB+MtfJgEABg0ajO3bt6J3775wdnaGXq/HoEGPACjt0vjb32bhuedmw2Qy4f77fRER8WS1dfjTn7rg9ddfRXZ2Fvr180dY2FjI5XIsXrwMixe/BL1ej3bt2mPZspcBADNnPo233orGBx+8hwEDAqvd/0svLcKrr0Zi797P4OfXEy4uLSrETJ48FevXv4MpU/4Ck8mEJ5+ciY4dO6FDBx8kJydjypS/QC6XY+DAwQgNDTOXi4v7Ab6+PdCqVSsAwMMPB2Dy5Ino0cMP3brdb3EMJycn/Otf/8bbb7+F//53K2QyGby8vPHOOxvh6FjzmQBloi46Ru4Bp7KufYUlQPwF22aX7NNNXeszUTaktqhvbAtLbI876noq63PnzsPLq1O1cUTlpaenoGdPP6vvsQuDiIiIJGMCQXb19s/v4tnDC/H2z+9KKve3v01Dnz6++Nvfpkk+ZszYndjo+U/EjN0puSwREVnHBIKIiIgkYwJBREREkjGBICIiIsmYQBAREZFkTCCIiMiu0tPT/1jjwVJAwEN2q0NCwkk888xMux0PsH5+Qgh8+OH7iIiYjMcfL10zo8zWrf/BpEkTMGnSBGzfvtW8ffnyJZgy5S/YuPFf5m2ffLIFcXE/VHrsrKxMvPLKMkyePBF//esTeOmluUhLuwag8p9HdTiRFNmFClrI9Fo4iNLpXx2EAU76igu4CKUTdLA+7SwRUVNTfjnv/Pw8/PWvkzB4cBByc3Pw2We78N//7oYQApMnT8AjjwyBVqs1L+c9derjiIiYDqPRhLNnf8W0adOtHqNsOe+pU6dhxYrXIZPJcPDgAcydOwc7dlRc4MtWTCDILmR6LXS/nYSpuHSiHFNxAXS/nawQp+raH1AygSBqzmJj9+PIkcPIz8/HrVvZGDx4CF54YT5+/jkBmze/C7lcgYyMm/Dz64mlS1+Go6Mjl/Pmct5ERGQP57efReJ/z1YZo+nliSFvPGZ+nflrBv639HCVZXpM7gW/KdZXv5QiMfE8Pv74v1Cr1ZgzZyaOHDkMtbolzp8/h48//i86duyEZcsWYffunQgICORy3lzOm5q6ieruKDYZ4Owg7dJbsGApCgry4eamlnzMwa89Cl2+Diq1qvpgomYi/2oe0n68JqmMLk9bbRnvQdUvzuTgIKuwTQjxx2qXpR55ZAg8PDwAAMOGjUBCQjyGDh2GPn36olOn+wAAISGh2LfvMyiVCi7nzeW8qSHp7CHgJNNVGaMWJihh+7gFH6X0BAAAfH171KgcALR5wLPGZYmaKnXHlvAe1KHKGE0vy/87qpZO1ZZRd2xZ7bHd3NS4fbvQYtutW7cs/kAov9S2ECbza7lccdd2BUwmE5fzroflvJlAUKWcZDrknT9RZUyrdk5w7M5xC0SNjd8U6V0Nmgc8MeHzSfd87BYtWsDHpyMOH/4Wjz32ZwDAvn174O//sDnm2LE4FBYWQKl0xNdff4VZs54GAJw+/QsyMjLQpk0bHDgQi8DAgejV60Fs2/YJnnzyKbRq1QrR0avh7d0BM2c+XWU9ypbz9vRsiy+++ByBgQPRs2cvvPnmKqSnp8PLy8vqct5eXt6SlvOeOPGJapfzDgoaivz8PJw79ytmz56DwsICHDlyGOPH/wVCmHDs2I8YOnSYRdkPP9yMJUuW27Sc9+bNm7B/fwzGjBkL4M5y3gsWLEZWVla152INEwgiIrK7FSteR3T0anz44XvQ6/Xo2rUbFixYbH6/VatWmDdvLvLychESEoqAgIFISDiJNm3aYOXK5cjMzIS//wCMGTMOcrmcy3lzOW8u510Xarqct18bbbV3IO5r5wR19/7QKt0rjdFo3FCQngrdbyexK+8CrhkK0EHhhr+09K0Qq+pqfV/R0atx8WIiunfvgYULl9p0LmV+iPwOWWcz0aaXBoNfHyqpbG1rSNdFQ8D2uIPLed8RG7sfP/+cgJdfftVie/knH8g+qlrOm3cgyK6uGQpwuSRHcrmLFxORkBBfo2Nmnc1Eepy0wWJERFQ1JhBERNSgjB49BqNHj6mwvV+//lYfx6T6wQSC7pncQVidVbKMMbcIchgrfZ+IiBofJhB0z2QGHXRJlU9I4+jmBJmmqx1rRETW3D3XAlFVqhsiycW0iIiaAWdnJxQU5NXJhELU9AghUFCQB2fnyh/R5x0IIqJmwMfHB6mpqbh+/Wp9V4UaCWdnJ/j4VD6zKBMIIqJmQKlUonPnzvVdDWpCbOrCWL9+PUJDQxEaGoro6NIFReLi4hAWFobg4GCsXbvWHJuYmIgJEyZgxIgRWLZsWYXpPomIiKjxqzaBiIuLww8//IC9e/ciJiYG586dQ2xsLJYuXYqNGzfiwIEDOHv2LI4ePQoAWLBgAZYvX46vvvoKQgjs3Lmzzk+CGo8AZy+Mcu2MAGcvSeXGjBmH2bOfxZgx4yQf03dST/T/ewB8J1W+Wh0REUlTbReGRqPB4sWLzdNddunSBcnJyejUqZO5byQsLAwHDx5E165dodVq0adPHwDA+PHjsW7dOkyZMqUOT4Eak0AX7xqVCw8fX+NjMnEgIqp91d6B6NatmzkhSE5OxoEDByCTyaDRaMwxnp6euHnzJjIyMiy2azQa3Lx5sw6qTURERPXJ5kGUly9fxuzZs7Fo0SIoFApcuXLF4v3SZUsrPh4k9ZljW+Z0byw0Grf6rgIAwJSjg6ubbatlKpRyc6yjox4qlbLKeEdHBZRKOdyq2b8tMQCgcHGEm3vDaLe60lCui4aC7XEH24IaE5sSiISEBMydOxdLly5FaGgoTpw4YbH8Z0ZGBjw9PdG2bVuL7ZmZmfD09LS2y0pxMa3aV1QCFBZobYo16B3NsSUqI3Q6fZXxJSVy6PVG3K5i/25uTuaYY0VpyDYWw0PubLU7Q1VUAq2+Yrvt27cH6elp8PLyltydceHTc8i/mgd1x5b13p3RkK6LhoDtcUddL6ZFVNuq7cK4fv06nn32WaxZswahoaEAgN69e+PKlStISUmB0WhEbGwshgwZAm9vb6hUKiQkJAAAYmJiMGTIkLo9A2pUfipOx4HCJPxUnC6p3P79e7Fp0wbs379X8jEvfHoOJ9f8hAufnpNcloiIrKv2DsQHH3wAnU6HqKgo87ZJkyYhKioKzz//PHQ6HYKCghASEgIAWLNmDSIjI3H79m34+fkhIiKi7mpPRERE9aLaBCIyMhKRkZFW39u/f3+Fbb6+vti9e/e914yIiIgaLM5E2Qx19hBwkumsvtcKJvi1KQEAuDsL5NmzYkRE1GgwgWiGnGQ65J0/YfU9tacT8jJKB0S26dfXntWqwCBk0OpLB9Qa/xhXaxRAYYn1eCelDApZ4x+AS0TUGDCBoAZLqxeIv5APACgoMpq/lm27m7+vGq6OdqseEVGzxgSC7pEMRgHojJVHiCIDnP+I4f0BIqKmgQkE3ROjSaCw2IjkG5XPA6FSKdFBUxojVFXvT+4g4KTPBQA4CJjHY7RQmsxf/dpooRUqJGVLm6SMiIhqDxMIsitvhRscZEAHhfUZ92QGHXRJZwGU3rHI+yMx6eDuBkPnzujg7oa88yfQ0u9hALbNrtmml8biKxER3TsmEGRX49x8oZJLLzc9PLzGxxz8+tAalyUiIuuYQFCj5N7CAX6w7DZRCxMc75p5WyidoLPxTgUREdmOCQQ1SgqTDnnnT1lsa9XOCeKuuxuqrv0BJRMIIqLaxgSC7CpNnw+DwQBnBwV8lGqbyyWnpeF2cTFaODvjPu+Ki3BVJevXDOjydVCpVWjzgLTF3YiIyDomEGRXewsv4nd9Dro5tsI8D3+by23Ztw/nk5Lg17kzVsyZI+mYPyw/gvS4a/Aa2AFjYx6XWmUiIrKi2tU4iYiIiO7GBIKIiIgkYwJBREREkjGBICIiIsmYQBAREZFkTCCIiIhIMj7GSfXCVMkKnk7ltgvBxbKIiBoqJhBkV+KP9by1JSarK3h28b6zsqePJ2eQJCJqqJhAkF3NcfdHakblS39XRurkUeVx8igiotrHMRBEREQkGRMIIiIikoxdGI2QQcig1Qub400cjEhERLWMCUQjpNULxF/Itzm+TzfbV72saxtz45HkkANv0RITRW+by63YuLHGi2nFjN3JxbSIiGoZuzCIiIhIMt6BoCZEBp3RsmtHZgKMptLvjSagsOTOe05KGRQy27uCiIjoDiYQ1GQYTaLCI6ItW5egoMgAACgoMlh0/fj7quHqaNcqEhE1GUwgqElzb+EAF6UJRQBclCb4tbmTYKiFCY760u+F0gk6cOIqIiJbMYGgJk1h0sFQVAAAMBQVIO/8CfN7rdo5QchLv1d17Q8omUAQEdmKgyiJiIhIMpsTiMLCQowePRrXrl0DACxZsgTBwcEIDw9HeHg4Dh06BACIi4tDWFgYgoODsXbt2rqpNREREdUrm9Vn7VkAAB4nSURBVLowTp8+jcjISCQnJ5u3nT17Flu3boWnp6d5m1arxdKlS/HJJ5+gffv2mD17No4ePYqgoKBarzgRERHVH5sSiJ07d+KVV17BwoULAQBFRUVIT0/H8uXLkZ6ejuHDh+O5557DmTNn0KlTJ/j4+AAAwsLCcPDgQSYQZBbu2h1XbxVBJXH4zfTwcNwuLkYLZ2fJx+w+TQN9kQlKF/bYERHVFps+xVetWmXxOjs7GwEBAVi5ciVcXFwwe/Zs7N69Gy4uLtBoNOY4T09P3Lx5s3ZrTI2at0INE6Q/O3mft3eNj+l2HwdHEhHVtho9heHj44MNGzaYX0+bNg0xMTEICQmpECuTSVuHwcPDtSZVapA0Grc62a8pRwdXN9t/KSqUcot4R0c9VCql1VgHuYP5PXm57yvjIHewKa4sxqGa2PL7qirW2jGtxVdVN4VCDuFQen0KuRwmReWJjYtKDjeX2nloqa6ui8aK7XEH24Iakxp9Il68eBHJyckYMWIEAEAIAYVCgbZt2yIrK8scl5GRYTFGwhbZ2YUwmRr/7IAajRsyMwvqZN9FJUBhgbb6wD8Y9I4W8SUqI3Q6vdVYk1Fufs9oNFUaVz6+ujiVSmmOKb9/a8rvq6pYa8e0Fl9V3XQlcvPEUy1di3G+3LV7N39fNbS3K33bZnV5XTRGbI87atoWDg6yJvWHFzUeNUoghBBYvXo1AgIC4OLigh07dmDcuHHo3bs3rly5gpSUFHTo0AGxsbGYMGFCbdeZKtHZQ8BJpquwvRVM8GtzZw5nd2eBPHtWrJx9hReQJMuDBq4IEl1sLrdl3z4kp6XhPm9vTA8Pl3TMix9noCBFB7dOKnSPkJbQEhGRdTVKIHx9fTFr1ixMnjwZBoMBwcHBGD16NAAgKioKzz//PHQ6HYKCgqx2a1DdcJLpLCZKKqP2dEJeuSme2/Tra89qWUgzFCBNlgdIvMmUnJaG80lJNTpmQYoOuYnFNSpLRETWSUogDh8+bP5+6tSpmDp1aoWYwMBA7N+//95rRkRERA0Wn2sjIiIiyZhAEBERkWRMIIiIiEgyJhBEREQkGRMIIiIikowJBBEREUlWO3PzEtnI38kLmgI3qIW09Ske9feHX5cu8GzdWvIxvYao0aqHM5w1VU+3TUREtmMCQXbl7+SNdvm2T8Nd5lF//xof0yuoZY3LEhGRdezCICIiIsl4B4IIgHsLB/ih8jsjamGCox4QSifowOXBiYiYQJBdxWvTkCwrgFo4wQ/tbC53JD4eGbduwbN1a8ndGelH81CcqYezRllpd4bCpEPe+VOV7qNVOycIOaDq2h9QMoEgImIXBtlVvDYdx2VXcV52U1K5I/Hx2H3oEI7Ex0s+Zvr/8nFlzy2k/y9fclkiIrKOCQQRERFJxi4MIgnkDgJO+txq4zhWgoiaOiYQRBLIDDroks5WG8exEkTU1LELg4iIiCRjAkFERESSMYEgIiIiyTgGogEwCBm0emFzvEnI6rA2RERE1WMC0QBo9QLxF2yfo6BPN3Ud1qZueSvcoNOboIGrpHL3eXtbfJXCrZPK4isREd07JhBkV+Guvkgtkr6Y1vTw8Bofs3uEZ43LEhGRdRwDQURERJLxDgSRTWTQGQWcBKAzVh+tBMepEFHTxgSC7CrNkI9rKIIKCknjIJLT0nC7uBgtnJ0lj4MoSNZCX2SC0sUBbvfVbHIno0kgNUOLLt5GJN+ovgvGt5vtg2KJiBojJhBkV/sKLyLJIQfeoiUmit42l9uybx/OJyXBr3NnrJgzR9IxL36SidzEYrj3cEb/5T5Sq0xERFZwDAQRERFJxgSCiIiIJGMCQURERJIxgSAiIiLJmEAQERGRZDYlEIWFhRg9ejSuXbsGAIiLi0NYWBiCg4Oxdu1ac1xiYiImTJiAESNGYNmyZTAYDHVTayIiIqpX1SYQp0+fxuTJk5GcnAwA0Gq1WLp0KTZu3IgDBw7g7NmzOHr0KABgwYIFWL58Ob766isIIbBz5846rTwRERHVj2oTiJ07d+KVV16Bp2fpegJnzpxBp06d4OPjA4VCgbCwMBw8eBBpaWnQarXo06cPAGD8+PE4ePBg3daeiIiI6kW1E0mtWrXK4nVGRgY0Go35taenJ27evFlhu0ajwc2bN2uxqtQUzHH3R2qG9MW0pE4eVR4njyIiqn2SZ6IUouIUvTKZrNLtUnl4SFvmuSHTaNxsijPl6ODqZvsUywql3Gq8o6MeKpWywnYHuYPFdvldryuLrSqufLwtcWUxd9elsjhr9a4szlrdq4qzFl/dOZTF2nKuAKBUyqGu5Odv63XRXLA97mBbUGMiOYFo27YtsrKyzK8zMjLg6elZYXtmZqa520OK7OxCmEyNfx0BjcYNmZkFNsUWlQCFBZX/Vd7ZQ8BJpjO/dtMXoaOqpEKci1wgU6evsN1klENXbrvRaLJ4XVlsVXHl46uLU6mU5pi763K38vuqKtbaMa3F19a5lsXa0iYAoNcbrf78pVwXzQHb446atoWDg6xJ/eFFjYfkBKJ37964cuUKUlJS0KFDB8TGxmLChAnw9vaGSqVCQkIC+vXrh5iYGAwZMqQu6tzgGYQMN3N0KKr4O94qk6j6To2TTIe88yfMr9WeTsiz0g3Qpl9fSfWkuuMoF4A+t8J2Y24RnPR3LgyhdIIONVvgi4ioPklOIFQqFaKiovD8889Dp9MhKCgIISEhAIA1a9YgMjISt2/fhp+fHyIiImq9wo2BVi+QmFxY5V2F8vp0U9dxjRqOjbnxNVpMa8XGjTVeTOvka6l2X0xLZtBBe+Vshe2Obk7QlbsuVF37A0omEETU+NicQBw+fNj8fWBgIPbv318hxtfXF7t3766dmjUwBiGDVm9b10p1dxSIiIgaOy7nbSOtXiD+Qr5Nsc3pjgIRETVPnMqaiIiIJGMCQURERJIxgSAiIiLJOAaCqImTMgAYAJyUMihkjX8uFiKqW0wgiOqEDDpjxa2iyICScttlJqCwpG5/aUsZAAwA/r5quDrWSVWIqAlhAkFUB4QQSL5RcR4QlUppMZNly9YlOJ+Vz1/aRNToMIEguwp37Y6rt4qgknjpTQ8Px+3iYrRwdpZ8zO7TNNAXmaB0afpDflTQQqa3TFwcBODX5s7sl1qhQlI25yohonvDBKIela1x0Qomiw/4u7k7C+TZsV51yVuhhgnS/9S+z9u7xsd0u6/hzvTo3sIBftBCLUxwrGKJDVunvJbptdD9dtJim84I5JW7G9LS72GA02cT0T1iAlGPyta4qGxtizJc46LpUph0yDt/Cq3aOUHIK4/jlNdE1NAwgSBqhCp7ssJBoMLgTcGp1YmoDjCBILvaV3gBSbI8aOCKINHF5nJb9u1Dcloa7vP2xvTwcEnHvPhxBgpSdHDrpEL3COlLzNuHDDpj5U9hlD2tUcYkgISLFZ+s8GtTYtFdAQA+nrxzQUS1jwkE2VWaoQBpsjxA4hOLyWlpOJ+UVKNjFqTokJtYXKOy9mI0CaRW0Y1V9rRGGa63QkT1rekPSyciIqJaxwSCiIiIJGMXBlEjUPa4Z5nKHv1tSo/8ElHDxgSCqBEoe9yzTGWP/vKRXyKyF3ZhEBERkWRMIIiIiEgyJhBEREQkGcdAkF35O3lBU+AGtZA2udGj/v7w69IFnq1bSz6m1xA1WvVwhrNGKbksERFZxwSC7MrfyRvt8iufMKkyj/r71/iYXkEta1yWiIisYxcGERERScYEgoiIiCRjF0YdUCu06Op2GyUqY5VxzXHSn3htGpJlBVALJ/ihnc3ljsTHI+PWLXi2bi25OyP9aB6KM/Vw1ijZnUFEVEuYQNQBuUGL/MST0On0VcY1x0l/4rXpSJLlwBst4SekJRDnk5Lg17mz9ATif/nITSyGew9nJhBERLWECQQRWZDJZCgssX25VCelDAqZxOVViajRYwJBRBZKDAK/XM6vPvAP/r5quDrWYYWIqEHiIEoiIiKSjHcgiMiqzh4CTjJdtXGuckcA0iYGI6LGjwkEUTNz99LgdytbKtzdWSAlIb7a/bX3GAyTnAkEUXNzTwlEREQEsrOzoVCU7mblypW4evUq/v3vf0Ov12P69OmYOnVqrVSUiGrH3UuD361sqfDm+JQQEdmuxgmEEAJJSUk4cuSIOYG4efMm5s2bhz179sDR0RGTJk3CgAED0LVr11qrMBEREdW/GicQSUlJkMlkmDlzJrKzs/H444+jRYsWCAgIgLu7OwBgxIgROHjwIJ577rlaqzA1bt4KN+j0JmjgKqncfd7eFl+lcOuksvhKRET3rsYJRH5+PgIDA7FixQpotVpERERg5MiR0Gg05hhPT0+cOXNG0n49PKT9YrEXU44Orm629fM6OBQDAFSqqld/lMsdoFIp4fDH1+rizPuvJP7uuMriK4u7O7aquPLxtsSVxYxr6Ycbxson2Cq/r/J1mf34xErjrNW9fNyDM60nHVLOtSzWlnMFSudSqCzO2s+iLq8BKT//8vG2nqtSKYda41ZtXGU091C2qWFbUGNS4wSib9++6Nu3tI/UxcUFEydOxBtvvIGnn37aIk4mk0nab3Z2IUymhjcpTVEJUFhg2yqSZfWvbiZKo9EEnU4Pk1FeZWxZnHn/lcTfHVdZfGVxd8dWFVc+vro4lUpZo3OtKtbaMa3F19a5lsXa0iZAaReftTiVSmn1Z1GX14CUn3/5eFvPVa83IjOzoNo4azQatxqXbWpq2hYODrIG+4cXNW01ngfi5MmTOHbsmPm1EALe3t7Iysoyb8vIyICnp+e91ZCIiIganBonEAUFBYiOjoZOp0NhYSH27t2Lf/zjHzh27Bhu3bqF4uJifP311xgyZEht1pcauTRDPq4hF5kolFQuOS0N5377DclpaZKPWZCsxa3zRShItu0OEhERVa/GXRhDhw7F6dOnMXbsWJhMJkyZMgX9+vXDvHnzEBERAb1ej4kTJ+LBBx+szfpSI7ev8CKSHHLgLVpiouhtc7kt+/aZF9NaMWeOpGNe/CTTvJhW/+U+UqtMRERW3NM8EC+++CJefPFFi21hYWEICwu7p0oRERFRw8a1MIiIiEgyJhBEREQkGRMIIiIikowJBBEREUnG1TiJqMEyCBm0etsnlnNSyqCQNbyJ6IiaIiYQEnT2EHCS6aqNUznI7VAboqZPqxeIv5Bvc7y/rxqujnVYISIyYwIhgZNMh7zzJ6qNkw3qb4faEBER1Z9mm0BIvTVqEtLW9CDr5rj7IzVD+oyQUiePKo+TR9U1GQpLbItkFwNR09FsEwipt0b7dFPXYW2IGi+lgwm3MzOqjdMKFTw0LdnFQNRENNsEgohqh8ygQ975k9XGtfR7uO7rIpOhsISDLonsgQkEETUZJQaBXy5z0CWRPTCBILvamBtfo8W0VmzcWOPFtE6+lsrFtBqIu+8QmHJ0KKpi/ATHHhE1XEwgiMhu7r5D4OrmhMKCygfVcuwRUcPFBIKImi0pYyY4XoLIEhMIIrIL9xYOUCEXfm3u9Fk4OupRojJaxGmFCknZ9um6kDJmguMliCwxgSAiu1CYdCi+/Avyys0DolIpodPpLeJKn9ZwsnPtiEgqLqZFREREkjGBICIiIsnYhUFEVAckryRaZKjD2hDVPiYQREQ2kDrLpUkACRdtn9Qq6CFH3hKmRoUJBNlVuGt3XL1VBJXES296eDhuFxejhbOz5GN2n6aBvsgEpQs/nqnmpM5yyTksqKljAkF25a1QwwTpz8Ld5+1d42O63ccR/UREtY0JBIDOHgJOMl2VMa1ggs5ZIM9OdSIiImrImEAAcJLpkHf+RJUxak8nOPr0sFONiIiIGjYmEGRX+wovIEmWBw1cESS62Fxuy759SE5Lw33e3pgeHi7pmBc/zkBBig5unVToHuEptcpERGQFEwiyqzRDAdJkeYDEJQWS09JwPimpRscsSNEhN7G4RmXJ/txbOMAPpbNVtoLJYurr8uw55TURVcQEgogaFIVJh7zzpwCUdh2Wn/q6PE55TVS/mkwCIXXSFpPgXy5EjVn5OxVlrN2x4J0KorrRZBIIrV4g/gKf0SZqLsrfqShj7Y4F71QQ1Q3OrENERESS1UkC8fnnn2PUqFEYPnw4tm3bVheHICIionpU610YN2/exNq1a7Fnzx44Ojpi0qRJGDBgALp27VrbhyIiIqJ6UusJRFxcHAICAuDu7g4AGDFiBA4ePIjnnnvOpvIODjUb7KSQAy5OcgnxMnO8UqmAqkWLKuPlTirIFcpq4wBAJlfA0cUFUFS9ul7Z/uROKqhaVF73u49bWXxl9bs7vqrzKB9ry/na0i6OjgpzTCulOzQGGVpBDZWwUtdy+ypfl7bt2iG/pARt27UrbTMrx7TWLnKFEm5eaogiR7h5OVXajtWdQ1mslGvAWpyjo8LiurDHNSDl518+/l7P1Vrd5E4uFnW5uz3url9V7WLrNaBUKir9bCj/OVAdKbE1iXdwqNnnX00/M4nulUwIIfGJ/Kpt2rQJRUVFmDdvHgBg165dOHPmDF577bXaPAwRERHVo1ofA2EtH5HJmCETERE1JbWeQLRt2xZZWVnm1xkZGfD05PTBRERETUmtJxADBw7EsWPHcOvWLRQXF+Prr7/GkCFDavswREREVI9qfRBl27ZtMW/ePERERECv12PixIl48MEHa/swREREVI9qfRAlERERNX2ciZKIiIgkYwJBREREkjGBICIiIsmYQBAREZFkTCCsKCwsxOjRo3Ht2jUApdNzh4WFITg4GGvXrjXHJSYmYsKECRgxYgSWLVsGg6F0St709HRMnToVISEheOaZZ3D79m0AQH5+PmbNmoWRI0di6tSpyMzMBACUlJRgwYIFGDlyJMaNG4fff//dzmdcufXr1yM0NBShoaGIjo4G0Lzb45133sGoUaMQGhqKjz76CEDzbg8AePPNN7F48WIAdX/OQgi8+eabCAkJwahRo5CQkFAPZ1xRREQEQkNDER4ejvDwcJw+fbrSRQXr+nohshtBFn755RcxevRo0bNnT5GamiqKi4tFUFCQuHr1qtDr9WLGjBniyJEjQgghQkNDxalTp4QQQixZskRs27ZNCCHErFmzRGxsrBBCiPXr14vo6GghhBCvvvqq2LRpkxBCiL1794oXXnhBCCHE5s2bxfLly4UQQpw4cUJMnDjRfidchR9//FE88cQTQqfTiZKSEhERESE+//zzZtsex48fF5MmTRJ6vV4UFxeLoUOHisTExGbbHkIIERcXJwYMGCAWLVokhKj7c/7yyy/FzJkzhdFoFElJSWLYsGFCr9fb6WytM5lMYtCgQRb1uHHjhhg6dKjIyckRt2/fFmFhYeLy5ct2+TwhshfegbjLzp078corr5hnzzxz5gw6deoEHx8fKBQKhIWF4eDBg0hLS4NWq0WfPn0AAOPHj8fBgweh1+sRHx+PESNGWGwHgCNHjiAsLAwAMHr0aPzvf/+DXq/HkSNHMGbMGACAv78/cnJykJ6ebu9Tr0Cj0WDx4sVwdHSEUqlEly5dkJyc3Gzb4+GHH8bHH38MhUKB7OxsGI1G5OfnN9v2yM3Nxdq1a/H0008DgF3O+ejRoxg1ahQcHBzwpz/9CV5eXjh16pS9T91CUlISZDIZZs6ciTFjxmDr1q0Wiwq6uLiYFxW0x+cJkb0wgbjLqlWr0L9/f/PrjIwMaDQa82tPT0/cvHmzwnaNRoObN28iJycHrq6uUCgUFtvv3pdCoYCrqytu3bpldV83btyo0/O0Rbdu3cwfaMnJyThw4ABkMlmzbQ8AUCqVWLduHUJDQxEYGNisr4+XX34Z8+bNg1qtBlDx/0pdnPPdU+M3hLbIz89HYGAgNmzYgC1btuDTTz9Fenq6TddFXVwvRPbCBKIaopLFwaRur4yDg/UfQWXb68Ply5cxY8YMLFq0CB07dqzwfnNrj7lz5+LYsWO4fv06kpOTK7zfHNpj165daN++PQIDA83b7HHO1vZV323Rt29fREdHw8XFBa1bt8bEiROxbt26CnE1uS5qq+2I6gKvtmpUtjjY3dszMzPh6emJ1q1bo7CwEEaj0WI7UPrXRlkZg8GAwsJCuLu7w9PT02IAVPky9S0hIQHTp0/HSy+9hHHjxjXr9vj999+RmJgIAHB2dkZwcDCOHz/eLNvjwIED+PHHHxEeHo5169bh8OHD2LVrV52fc9u2bRtcW5w8eRLHjh0zvxZCwNvb26broi6uFyJ7YQJRjd69e+PKlStISUmB0WhEbGwshgwZAm9vb6hUKvMo8JiYGAwZMgRKpRL9+/fHgQMHLLYDQFBQEGJiYgCUfgD3798fSqUSQUFB2LdvH4DSDyOVSgUvL696OFtL169fx7PPPos1a9YgNDQUQPNuj2vXriEyMhIlJSUoKSnBt99+i0mTJjXL9vjoo48QGxuLffv2Ye7cuXjsscfwxhtv1Pk5DxkyBJ9//jmMRiNSUlKQnJyMBx54oB5a4I6CggJER0dDp9OhsLAQe/fuxT/+8Q+riwra4/8Pkd3Ux8jNxmDo0KEiNTVVCFE60jwsLEwEBweLVatWCZPJJIQQIjExUUyYMEGEhISI+fPnC51OJ4QQ4tq1a+Kvf/2rGDlypJgxY4bIzc0VQgiRk5MjZs+eLUaNGiWeeOIJ8/61Wq1YuHChGDVqlBg7dqw4e/ZsPZxxRa+99pro06ePGDNmjPnf9u3bm217CCHEO++8I0aOHClGjx4t1q1bJ4RovtdHmc8++8z8FEZdn7PJZBJRUVFi1KhRYtSoUeL777+vhzOuaO3atSIkJEQEBweLLVu2CCGE2L9/vwgNDRXBwcHivffeM8fW9fVCZC9cTIuIiIgkYxcGERERScYEgoiIiCRjAkFERESSMYEgIiIiyZhAEBERkWRMIKjZ6tu3r3nFVSIikoYJBBEREUmmqO8KENni+PHjiI6ORtu2bZGamgonJydERUXh/fffR25uLlJTU/Hoo4/ihRdewJo1axAfHw+j0Qg/Pz9ERkbC1dUVJ0+exGuvvQaZTIYHHngAJpMJAHD79m0sWbIEKSkpcHBwQM+ePbFy5coq1xVYvHgxVCoVfv31V2RlZWHkyJFo3bo1vvvuO2RmZuL1119HYGAgSkpKKq3Pd999h02bNqGkpAS3bt3C2LFj8eKLL+L48eNYu3YtfHx8cPnyZZSUlODll19GQECAvZqbiKhavANBjcb58+cxY8YMfP755xg/fjwWLFgAANBqtfjiiy+wYMECvPfee5DL5dizZw/2798PT09PrFmzBiUlJXjhhRewePFixMTEYMCAAdBqtQCAQ4cO4fbt29i3bx92794NAEhNTa22PomJidixYwc+++wzbNmyBS4uLvj0008RERGB999/HwAqrY8QAh9++CGioqKwZ88e7NixA++99555NcUzZ85gxowZiImJwcSJE7F+/fq6aFIiohrjHQhqNHx9fc1LrU+YMAErV66Ep6cn+vXrZ445cuQICgoKEBcXBwDQ6/Xw8PDApUuXoFAozKtHjh49Gi+//DIAoF+/fli7di2mTZuGgQMH4v/+7//QqVOnauszdOhQKJVKaDQauLi44JFHHgEAdOzYEbm5uVXWRyaT4d1338WRI0cQGxuL33//HUIIFBcXAwC8vLzQo0cPAICfnx/27t17z+1HRFSbmEBQoyGXyy1eCyHg4OAAFxcX8zaTyYSlS5ciKCgIQGn3hE6nw/Xr1yssjaxQlF7+Pj4+OHToEI4fP46ffvoJTz75JCIjIxESElJlfRwdHa3ur7zK6lNUVIRx48Zh2LBh6N+/PyZMmIBvvvnGXEcnJyfzPipb1pmIqD6xC4MajQsXLuDChQsAgB07duChhx6CWq22iBk8eDC2bduGkpISmEwmLF++HP/85z9x//33QwiBo0ePAgC+/fZb5OXlAQC2b9+OJUuWYPDgwViwYAEGDx6My5cv10qdK6tPSkoKCgsL8eKLL+Kxxx7DiRMnzDFERI0BEwhqNNq0aYO3334bYWFh+OabbxAdHV0hZs6cOfD29sa4ceMwatQoCCGwePFiKJVKbNiwAe+88w7Cw8Nx6NAheHh4AADGjh0Lo9GIUaNGYfz48SgsLERERESt1Lmy+nTv3h2PPvooRo4ciXHjxuHw4cPo2rUrUlJSauW4RER1jatxUqNw/PhxvPbaa4iNja3vqhARETgGgsiqpKQkzJs3z+p7f/rTn/D222/buUZERA0L70AQERGRZBwDQURERJIxgSAiIiLJmEAQERGRZEwgiIiISDImEERERCQZEwgiIiKS7P8BeWJjZeV7+nMAAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"sample = valid_copy.copy().loc[valid_copy[\"ocean_proximity_INLAND\"] == 0]\n",
"sample_mean = sample['preds_mean'].mean()\n",
"sample_std = sample['preds_std'].std()\n",
"lower_bound = sample_mean - sample_std\n",
"upper_bound = sample_mean + sample_std\n",
"\n",
"sns.distplot(\n",
" sample[\"median_house_value\"], kde=False,\n",
")\n",
"sns.distplot(sample[\"preds_mean\"], kde=False)\n",
"plt.axvline(\n",
" x=sample_mean,\n",
" linestyle=\"--\",\n",
" linewidth=2.5,\n",
" label=\"Mean of predictions\",\n",
" c=\"k\",\n",
")\n",
"plt.axvline(\n",
" x=lower_bound,\n",
" linestyle=\"--\",\n",
" linewidth=2.5,\n",
" label=\"Lower bound 68% CI\",\n",
" c=\"g\",\n",
")\n",
"plt.axvline(\n",
" x=upper_bound,\n",
" linestyle=\"--\",\n",
" linewidth=2.5,\n",
" label=\"Upper bound 68% CI\",\n",
" c=\"purple\",\n",
")\n",
"plt.legend(bbox_to_anchor=(1.01, 1), loc=\"upper left\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In general, we expect our models to perform best on categories that are most frequent in the data. One way to validate this hypothesis is by calculating the ratio of the standard deviation of the predictions to the predictions themselves:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ocean_proximity_INLAND\n",
"1 0.288622\n",
"0 0.228371\n",
"dtype: float64"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds_quality = valid_copy[cols].groupby(\"ocean_proximity_INLAND\", as_index=True).mean()\n",
"(preds_quality[\"preds_std\"] / preds_quality[\"preds_mean\"]).sort_values(ascending=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What the above tells us is that our predictions are less confident (i.e. higher variance) for housing districts that are inland - indeed looking at our bar plot we see these categories are under-represented in the data!\n",
"\n",
"In general, confidence intervals serve two main purposes:\n",
"\n",
"* We can identify which categories the model is less confident about and investigate further\n",
"* We can identify which rows in the data the model is not confident about. This is particularly important when deploying models to production, where e.g. we need to decide how to evaluate the model's predictions for a _single_ housing district."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Feature importance"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One drawback with the confidence interval analysis is that we need to drill-down into each feature to see where the model is making mistakes. In practice, we can get a global view by ranking each feature in terms of its importance to the model's predictions. In scikit-learn, the Random Forest model has an attribute called `feature_importances_` that we can use to rank each feature:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def rf_feature_importance(fitted_model, df):\n",
" return pd.DataFrame(\n",
" {\"Column\": df.columns, \"Importance\": fitted_model.feature_importances_}\n",
" ).sort_values(\"Importance\", ascending=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's use this function to calculate the feature importance for our fitted model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
Column
\n",
"
Importance
\n",
"
\n",
" \n",
" \n",
"
\n",
"
7
\n",
"
median_income
\n",
"
0.248165
\n",
"
\n",
"
\n",
"
14
\n",
"
ocean_proximity_INLAND
\n",
"
0.135266
\n",
"
\n",
"
\n",
"
13
\n",
"
population_per_household
\n",
"
0.088856
\n",
"
\n",
"
\n",
"
9
\n",
"
postal_code
\n",
"
0.088172
\n",
"
\n",
"
\n",
"
0
\n",
"
longitude
\n",
"
0.080466
\n",
"
\n",
"
\n",
"
12
\n",
"
bedrooms_per_room
\n",
"
0.070059
\n",
"
\n",
"
\n",
"
1
\n",
"
latitude
\n",
"
0.066439
\n",
"
\n",
"
\n",
"
10
\n",
"
rooms_per_household
\n",
"
0.048305
\n",
"
\n",
"
\n",
"
2
\n",
"
housing_median_age
\n",
"
0.030146
\n",
"
\n",
"
\n",
"
8
\n",
"
city
\n",
"
0.022991
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Column Importance\n",
"7 median_income 0.248165\n",
"14 ocean_proximity_INLAND 0.135266\n",
"13 population_per_household 0.088856\n",
"9 postal_code 0.088172\n",
"0 longitude 0.080466\n",
"12 bedrooms_per_room 0.070059\n",
"1 latitude 0.066439\n",
"10 rooms_per_household 0.048305\n",
"2 housing_median_age 0.030146\n",
"8 city 0.022991"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# expected shape - (n_features, 2)\n",
"feature_importance = rf_feature_importance(model, X)\n",
"\n",
"# peek at top 10 features\n",
"feature_importance[:10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From the table we see that `median_income`, `ocean_proximity_INLAND`, and `population_per_household` are the most important features - this is not entirely surprising since income and house location seem to be good indicators of house value. We can also plot the feature importance to gain a visual understanding:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_feature_importance(feature_importance):\n",
" return sns.barplot(y=\"Column\", x=\"Importance\", data=feature_importance, color='b')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhQAAAEJCAYAAADM2r49AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdfVzPd////9u7UzmraDEn25zPyWKOIadbMtaUVKw4FL5OR04O0xRJJJMYKnR0fMw2YzlLJ7IMGRspMyczJ2sHSc5aJKTzd78/+vU6pBNvKmc9rpfLLpf1fr/ez9fz+bLp+X6+nq/7Q1VYWFiIEEIIIUQlaD3vDgghhBDi5ScTCiGEEEJUmkwohBBCCFFpMqEQQgghRKXJhEIIIYQQlSYTCiGEEEJUmkwohBBCCFFpOs+7A0IIIYR4NnJy8rh/P4eGDety69b9J/qslpYKY+M65b4vEwpRY03/Ipy09Mzn3Q0hhHhmNi/7J3fvZgOgVldtrqXc8qgB4uPjcXZ2BmDevHn8/vvvVdr+hAkTuHnzZpW2KYQQ4uUiKxQ1jK+vb5W3+Z///KfK2xRCCPFykQnFCyw+Pp7g4GAKCwtJTk5m0KBB1KtXj3379gEQEhLC2bNnCQgIID8/n2bNmuHj44OxsTG//PILX3zxBfr6+rRo0UJp09nZGVdXV/7xj3/g7e1NYmIiaWlptGjRgqCgINLS0nB1daVNmzacO3eOhg0bsnr1aoyMjMrtZ//+/fn2229JSEjg559/JiMjgytXrtC7d2+8vb0pLCxk+fLl7Nu3D21tbRwdHRk9ejSXLl3Cy8uLO3fuULt2bebNm4eZmRnu7u4YGBhw/Phx7t27x9y5c4mIiOD8+fMMGDAAd3d3CgoKWLZsGQkJCRQUFGBvb8+YMWOq+49ECCFEOeSWxwvu1KlTfPHFF0RHRxMaGkqDBg0ICwujXbt2hIaGsmLFCtavX094eDh9+vRh+fLl5Obm4u7uTkBAAGFhYdSqVatUuydOnEBXV5ctW7awd+9ecnJyOHjwIADnz59n7Nix7Nq1i/r16xMVFaVxf0+cOEFAQACRkZEcOHCACxcuEBMTw2+//UZUVBTbtm0jLCyMv//+Gzc3N5ydnYmKisLDw4MZM2aQm5sLQGpqKpGRkUyfPh0PDw8WLlxIeHg4W7du5d69e2zduhWAnTt3sn37dvbv38+vv/5aBVdcCCHE05AVihdc27Ztef311wEwNjamZ8+eADRp0oTY2FiuX7+Oi4sLAGq1GkNDQy5cuICpqSmtWrUCwM7OjtWrV5dot1u3bhgZGbFp0yYuXrxIUlISDx48AKBhw4Z06NABgDZt2pCRkaFxf999913q1q0LQPPmzcnIyODYsWNYWVmhp6eHnp4eERERZGZmkpyczMCBAwHo0qULhoaGXLx4EYB+/fop42zTpg0NGzYEwMjIiIyMDOLi4jh37hxHjx4F4MGDB1y4cIH33nvvSS6vEEKIKiITihecrq5uiZ+1tbWVf1er1XTt2pXg4GAAcnJyyMzM5Nq1a6jV6jI/U2z//v0EBATg4uKCvb096enpFFey19fXV45TqVQ8SYX7sj6ro1PyP7OUlBQMDQ1LtVtYWEhBQUGpcT/6eYCCggLc3NyUCcnt27epXbu2xv0UQghRteSWx0vMzMyMkydPcunSJQDWrl3LsmXLaNeuHbdu3eL8+fMAREdHl/psXFwcVlZWODg4YGJiwrFjx5Rf5lWtW7du7N27l7y8PLKyshg/fjxpaWk0b96cH3/8EYCTJ0+SlpZGmzZtNGrT3NycrVu3kpeXR2ZmJiNHjuTUqVPV0n8hhBCPJysUL7HXXnuNJUuWMHPmTNRqNY0aNcLf3x9dXV2+/PJL3Nzc0NHRUW5fPGz48OHMnj2bmJgY9PT06NKlCykpKdXSzw8//JAzZ85gb2+PWq3GxcWFFi1a4O/vj7e3N4GBgejq6hIYGIienp5GbTo5OXH58mXs7OzIz8/H3t6eHj16PFG/AjyGPs1whBDipZWdk1dtbasKn2Q9W4hXyK1b96s82OVl8dpr9fj773vPuxvPVU2/BjJ+Gf+Tjl9LS0XDhnXLfV9WKIRGnJ2duXv3bqnXnZycGDFixHPoUeVV9D9GTfDaa/Wedxeeu5ftGmTn5HHv/085FOJFIxOKl9jq1avp1KkTlpaWGh0/YcIEFi9eTG5uLuvWrWPJkiUan2vjxo0AhIWFkZCQwNKlSwkMDGTDhg3Y2dkpj6bGx8cTFBTExo0bcXd3p3v37tjb25fZpp+fHzt37uTQoUPKrY6UlBQsLS356quv6N27t3JscdYFwEcffaQ8wZKdnU27du3w8vLCxMRE4/GARG+Ll8/mZf/kHjKhEC8m2ZT5EpsxY4bGkwkoSrRs1KgR165d48qVK1XSh2vXrvHll18+8efy8/P54YcfePfdd4mJiSnxnq6uLvPnz+f+/bIL15iamhIREUFERAQxMTG8+eabTJ8+/an6L4QQomq80hOK4OBgPv74Y2xsbFi6dCkFBQV8/fXXDBo0iI8//hh/f38A0tLSmDJlCvb29jg4OHDkyBEAbt68ybhx4/jkk0+wsLBg+fLlQNG39H/961/8v//3//jwww/x9vausB8pKSnY2NgwZcoUBg8ezIQJE7hz5w5Q9LTCuHHjsLW1JS8vr8w+79+/n4EDB5KTk8Ply5fp168fN27cwN3dnbCwMFJSUrC1tcXV1ZWBAwcya9YsQkNDcXR05KOPPuK///0vUPQtPyUlhcWLF3PmzBkWLlyIm5sbW7ZsUfrq7Oz8RE9LODk5sXv37icOlTp48CDNmzdn6NChJc4PRROGXr164efn99h2VCoV06ZNIzExUXmqRQghxLP3yk4oDh48SGxsLGFhYezcuZPLly+zbt06Nm/ezPbt24mMjOSPP/7gzJkz+Pr64uDgQFhYGOvWrcPLy4v79++za9curK2t2bp1K5GRkWzevJnbt28DZSdCVuTPP/9k9OjRREdH06pVK4KCggBIT09n4sSJREREcOTIkVJ9Dg0NxdLSknfffZd169bh4eHBnDlzaNy4cYn2L1y4wJQpU4iJieH333/n6tWrbNmyBWtr61K/sD09PenUqRMLFizAwcGByMhIAK5evcrt27fp3LmzxtfZ0NAQb29v5s2bR3a25kuxYWFhfPTRR7z//vucO3eOv/76q8T77u7u/PLLLxw+fPixbenp6fHmm28qoVhCCCGevVd2QnH06FEGDx5MrVq10NHRwcHBgaCgICwsLKhXrx46Ojp8/fXXdOrUiSNHjhAQEICtrS0TJkwgPz+fK1euMG7cOF5//XXWr1+Pr6+vkqMA/0uENDAwUBIhK/LWW28pjzUOHTpUSXgElF/gZfU5Li4OKKoSun37dkxMTBg8eHCp9k1MTOjQoQNaWlo0bty4RKJmWZspi/Xo0YPU1FRSUlIIDw/H1tb2Ca5ykQEDBtCpUyeNb33cvn2bX375BSsrK2rVqoWFhQWhoaEljqlbty4+Pj4V3vp4mEqlKjNiXAghxLPxym7KfDgpstijT8jevHkTAwMD1Go133zzjVIA6+bNm5iYmLB06VKuXLmCtbU1AwYM4MiRI0+dJvlw2mNhYWGJ9MriX4Rl9Tk/Px8oui2jra3NpUuXyM3NLZXX8OjPZaVjlkWlUjF06FCio6OJiYnh//7v/zT63KPmz5+PtbV1hUXEikVGRlJYWMiwYcOAoo2VeXl5zJ49u8Rxffr00ejWR25uLpcuXaJ169ZP1XchhBCV98quUJibmxMdHU12djb5+fns2LGDOXPmcOjQITIzM8nPz+ezzz7jzJkzmJubs3nzZgD++usvhgwZQlZWFocPH2bcuHFYWVlx/fp1bt68WeYvfU1cunSJc+fOAbBjxw6lVsXj+mxubk5BQQEeHh7MmzePbt26sWrVqqe/MBRNNoonKgD29vaEhobSuHFjGjVq9FRtGhkZ4e3tzdq1ax977I4dO1i6dCmxsbHExsbyyy+/YGhoyO7du0sdW3zrIzU1tcy21Go1gYGBdO7cmTfeeOOp+i6EEKLyXtkVCgsLC86dO4eDgwP5+fn07dsXFxcX9PX1cXJyQq1W8+GHH9KrVy9atWqFl5cXNjY2ACxbtoy6desyadIkPv/8c+rXr0/Dhg3p1KnTU6dJGhoaEhAQQHJyMu3atWPx4sUa9XnUqFF89dVXNGzYkIEDB9KrVy+sra2VGhZPo1WrVty7dw83Nzf8/f15/fXXef3117Gzs3vqNqHo1segQYNK/PJfsGABPj4+ys///ve/SU9P58MPP1Re09LSYvTo0YSGhtK9e/cSbRbf+hg3bpzyWmpqqnJrRq1W0759e1asWFGpvgshhKgcScp8BlJSUnBxcSE2NvZ5d6WUwsJCUlNTcXZ2ZteuXRpHX78KJIdCvGw2L/tnlaU7SlKkjF+SMl9QycnJTJs2rcz3ylqNeFHs2bMHb29vvL29lcmEn5+f8ujswzp16oSvr++z7mK1kVoe4mVTnXUYhKgsWaEQNZbU8qi5385AroGMX8YvKxRCVBGp5fFy1bEoi9S2EOLFIROKJ+Ds7Iyrq2uFZbKdnZ2Vuhe2trZEREQ8q+5VSrt27R4bzlXVnmZvSXFNj2bNmpV4XZM/m0fJHoqXn9S2EOLF8co+Nvq8JCQkKP/+skwmhBBCiMp6pSYU8fHxjBo1ijFjxjBo0CDc3NzIzc1lx44dWFtbY2Njg7u7O5mZRd9Kzc3N8fDwwMbGBicnJ+WR0OKaF8VtOjs7lzhPfn4+np6eODo6Ymlpyfjx48nOzlY2Xw4fPhwo+tYPkJWVxWeffab0ITw8HKjamiCHDh1i2LBhDB06FFdXV9LT05WxzJw5k0GDBnHr1q0K2/fy8mLIkCEMGTKEy5cvA3Dy5EmGDx/OkCFDGD16tPK6s7Mz8fHxSr/69+8PQFRUFLa2ttjb2zN9+nRycnIACAkJwc7OjiFDhrBs2TIlCCw7O5t//etfWFtbM3LkSKXfBw4cwNbWVhlvWlpaib7m5ubi5uaGlZUV48ePVz4nhBDi+XilJhQAp0+fxsvLi5iYGHJycggJCSE4OJiNGzcSFRWFgYFBiToa3bt3JyoqisGDB2v8NMaJEyfQ1dVly5Yt7N27l5ycHA4ePIinpycA27ZtK3F8YGAgxsbG7Nq1i2+++YbAwEClkFVV1AS5ffs2K1asYP369YSHh9OnTx+lkBlAv3792LNnDw0bNqyw7V69ehEZGUnv3r0JDQ0lNzeXWbNmMX/+fCIjI3FycmLWrFkVtrFq1Sq++uorwsLCaNGiBRcvXuTQoUOcOXOG7du3Ex4ezs2bN5X6Ibdv32bs2LHs2rULExMTdu/eza1bt/Dy8mLNmjVERUXRtWtXFi1aVOI8xbeVfvjhBzw9PUlOTq6wX0IIIarXKzeh6NatGy1btkSlUmFra8vatWuxsLDA2NgYAEdHR6WOhr6+PkOHFj06aGdnp3zj1uQcI0eOZNOmTfj6+pKUlMSDBw/KPf7o0aNKzHSDBg2wtLRUbo1URU2QU6dOcf36dVxcXLC1tWXTpk3KSgKgcbGvAQMGANC6dWvu3LlDUlIS9evXx8zMDAArKyuSk5O5d6/8ncEWFhaMGDECPz8/LCwsaN++PXFxcZw+fRp7e3vs7Ow4c+aMUgzM1NRUab9169akp6dz+vRpzMzMlH0SD/+ZFUtISMDKykq5Ju+++65GYxRCCFE9XrlNmQ/XsCgsLCwVlV1YWKjETmtpaaFSqYCixMVHPwuUiKgutn//fgICAnBxccHe3p709PQKa3k8+l5hYSEFBQVA1dQEKSgooGvXrgQHBwOQk5Oj3NZ59ByatF3cj/LqoRT3vaxr5Onpyfnz5zl48CBubm64urpSUFDA6NGjGTt2LAB3795FW1ub9PT0EuMp77wP/5k9fOzDxz3cjhBCiGfvlVuhOH78uFJzIzw8HA8PD2JjY5W9Blu3blW+4WdlZSlPGISFhSn1NYyNjZVv0Pv37y91jri4OKysrHBwcMDExIRjx44pv2QfrZMBRXs1tm/fDhQt8e/fv79UxLSmyqoJ0rlzZ06ePMmlS5cAWLt2LcuWLXuq9h/WsmVL7ty5w+nTpwHYvXs3TZo0wcjIqMQ12rdvH1A0sRg4cCDGxsZMmjQJW1tbzp07h7m5OREREUoNlalTp7Jnz55yz9u5c2dOnTql7GPZsmVLqac3evbsya5du1Cr1Vy9epXffvut0uMVQgjx9F65r3WmpqZ8/vnn3Lx5k969ezNq1Chq166Ns7MzeXl5dOzYkYULFyrHx8TEsHLlSkxNTZWqltOnT8fHx4egoCD69OlT6hzDhw9n9uzZxMTEoKenR5cuXZRffpaWltja2hIWFqYcP3XqVLy9vbGxsaGgoIDJkyfTsWPHp3pMs6yaILVr12bJkiXMnDkTtVpNo0aN8Pf3f+K2H6Wnp8fKlSvx8fEhKysLQ0NDVq5cCcD48eNxd3dnx44dWFpaAkWrBNOnT2fs2LHUqlWL+vXr4+fnR6NGjTh//jyffPIJBQUF9O3bFzs7O65evVrmeU1MTFi0aBGurq7k5eXRpEmTUgmdI0eOJDExESsrK5o2bUrbtm0rPV4hhBBP75VKyoyPjycoKEjZsPc4zyN7oTJe5JogQjwPlQm2kqREGb+MX5IyX2nVVRMkOzsbR0fHMt+bPn26sspQk0j0ds39y1QIUfVeqRUKIUTNUdnY7Zo+qZLxy/hlhUI8N7GxsVy+fFl5WqMsYWFhJCQksHTp0mrpQ2BgIEC5qzhPQqK3X24Suy3Ei0UmFEJjf/zxx/PughBCiBeUTChqiPj4eAIDA9HR0eH69euYmZnh6+tLVFQUGzZsQKVS0bFjR+bPn4+enh5z584lMTERKHqiomvXroSGhgLQpEkT+vTpw9y5c7l37x5///03gwcPZvbs2Rr15dy5c3h5eZGdnY2hoSHLly+ncePGBAcHExkZiba2Nr1798bNzQ1tbW3+7//+j61bt2JsbFwiaOvQoUMEBASQn59Ps2bN8PHxUQLMhBBCPFuvXA6FKJ+mseQnTpwgIyOD8PBwNmzYwG+//Ubr1q1xcnLCyckJBwcHdu3ahbW1NVu3biUyMpLNmzdz+/Ztjfoxe/ZspkyZQlRUFB9//DHffPMNBw8eJDY2lrCwMHbu3Mnly5cJDQ3l999/Z8eOHezcuZMNGzZw48YNgMfGjQshhHi2ZIWiBimOJYei0urTpk1j1KhRJWLJPTw8mDhxIpcuXWLcuHH069evzJWHcePGcfToUdavX09iYiJ5eXlkZWU9tg+3b9/m77//xsLCAiha/QDw8/Nj8ODB1KpVCwAHBwfCw8PJzs7m/fffp06dOgB89NFHqNXqEnHjUJR0amhoWMkrJIQQ4mnJhKIG0TSW3NjYmOjoaA4fPszBgwexs7MjOjq6xLFLly7lypUrWFtbM2DAAI4cOfLY2HAAXV3dEj/n5OSQmppaZsx3fn5+mRHbubm5j40bF0II8WzJLY8aRNNY8v379zN79mw++OADPD09qV27NtevXy8RK3748GHGjRuHlZUV169fV9p9nHr16tG4cWMOHz4MQEREBKtXr8bc3Jzo6Giys7PJz89nx44dmJub07NnT3766Sfu3btHTk4Oe/fuBai2uHEhhBBPR1YoahBNY8n19fXZs2cPgwcPRl9fn4EDB9KuXTvu3r3LnDlzMDExYdKkSXz++efUr1+fhg0b0qlTJyV+/HH8/f3x9vZm2bJlGBsbs2zZMkxNTTl37hwODg7k5+fTt29fRo0ahY6ODqNHj2bYsGHUr1+fJk2aAPDaa69VS9y4EEKIpyPBVjXEk8aSC/Gik2CrypHxy/gl2Eq8FD777DOlGunD+vfvz4wZM55Dj0qT6O2a+5epEKLqyQqFqLFkQlGzJxQ1/RrI+GX8skIhRBWp6H+MmuC11+o97y6UqbK3MoQQz4dMKESFqnrvxf79+zlz5gwzZswgICCAXr168d5772n8+aosOS+1PF5MUqNDiJeTPDYqnilLS0tlD8WxY8coKCh4zj0SQghRFWSFQmjk0qVLeHl5cefOHWrXrs28efMwMzPD3d2dunXr8scff3Dz5k2mTp2Kg4MD9+7d4/PPPyc5OZnmzZtz48YNgoKCSEhIICEhAXNzc86cOYOnpydBQUEsXrwYV1dXevToQUpKCi4uLsTGxpKSkoKbmxsPHjygc+fOSn8yMzNZtGgRiYmJFBQUMGHCBKytrZ/jFRJCiJpNViiERtzc3HB2diYqKgoPDw9mzJhBbm4uADdu3GDz5s2sW7dOCZdas2YNLVq0IDo6mqlTp5a6TTF06FA6derE4sWLadeuXbnn9fHxwd7enoiICLp27aq8vm7dOjp27EhYWBibNm0iODiYK1euVMPIhRBCaEImFOKxMjMzSU5OZuDAgQB06dIFQ0NDLl68CEDv3r1RqVS0bdtWSd08fPgwtra2ALzzzjsVThoqkpCQgJWVFQBDhgxRoruPHDlCaGgotra2/POf/+TBgwdKdVQhhBDPntzyEI9VWFhYqk5HYWGhsv9BX18fAJVKpbyvra2tUW2PR9sElHjvR19XqVTKOdRqNf7+/nTs2BGAtLQ0KQ4mhBDPkaxQiMeqW7cuzZs358cffwTg5MmTpKWl0aZNm3I/06tXL6KiogC4cOECiYmJJSYcUDTpKJ6UGBsbK0FY+/btK9FOZGQkAD/++KNym8Xc3Jzvv/8egNTUVIYMGcL169erYrhCCCGegkwohEb8/f3ZuHEjNjY2LFq0iMDAQPT09Mo9fsqUKSQnJ2NjY0NAQAAmJiZKafJiffv2ZcGCBfz222+MHz+ezZs3Y2dnR3b2/x4Z9PLyYs+ePdjY2HDw4EGljLmrqyvZ2dlYW1szevRo3NzceOONN6pn8EIIIR5LkjJFtYiIiKBZs2b84x//4Nq1a4waNYp9+/ahpSVzWFGxZxVsJUmJMn4ZvyRlipdAy5YtWbBgAWq1Gi0tLRYtWvTCTSYkervm/mUqhKh6MqEQ1eKdd94hLCzseXejQhK9Xf3R2xKjLUTNIROKSqhMLLWUE3/+JHq7+kmMthA1x4u1Bi2EEEKIl5KsUFRSeno648aNIzU1FTMzMxYsWMDRo0cJCAggPz+fZs2a4ePjg7GxMb/88gtffPEF+vr6tGjRQmnD2dkZQ0NDEhMTWbVqFTdu3GDVqlWo1WqaN2/OokWLMDEx4eTJk/j6+pKTk4OxsTGLFi3izTffxNnZmfbt2xMXF0d2djaenp5s3LiRv/76izFjxjBmzBji4uLw9/cHwNDQkBUrVtCgQYNyx2Vubo6FhQVnzpyhTp06LF++nGbNmnH69Gm++OILsrOzMTY2ZuHChTRv3rzUGNq3b19mu/3798fMzIxz586xefNmfvrpJzZs2IBKpaJjx47Mnz+fOnXqcODAgTKvQf/+/bGysuKnn35CW1ubWbNm8dVXX3H58mXmzJnDxx9/XLV/wEIIITQiKxSVlJKSwvz584mMjCQzM5OQkBBWrFjB+vXrCQ8Pp0+fPixfvpzc3Fzc3d0JCAggLCys1COU7dq1Y8+ePZiamuLl5cWaNWuIioqia9euLFq0iNzcXGbNmqWcy8nJiVmzZpVoIyoqCltbWxYvXkxgYCCbNm1izZo1AKxduxZvb2/CwsKwsLDg7NmzFY4rPT2d7t27ExUVxeDBg1m8eDG5ubl4enqyYsUKdu7cydixY5k/f36pMZQ3mSjWr18/9uzZQ1paGsHBwWzcuJGoqCgMDAwICgri1q1bZV6DYqampkRHR9OxY0dCQkL46quv8Pf3JyQkRKM/MyGEEFVPVigq6b333uOtt94CwMbGBnd3d1QqFS4uLkBRoqOhoSEXLlzA1NSUVq1aAWBnZ8fq1auVdszMzAA4ffo0ZmZmNGvWDABHR0dCQkJISkqifv36ynFWVlZ4eXlx717RTv1+/foB0KRJEzp37oyBgQFNmzbl7t27QFGVT1dXVwYMGIClpSW9e/eucFz6+voMHTpU6euXX35JUlISV65c4dNPP1WOu3//fqkxPE5xka9jx45hYWGBsbGxMlYPDw+6d+9e5jUo9vBYTU1N0dHRoUmTJspYhRBCPHsyoagkHZ3/XcLiSI+uXbsSHBwMQE5ODpmZmVy7dg21Wq0cq62tXaKd4hWLh48pbjM/P7/U68XvFSdNFte4eLRPxcaMGYOFhQUHDhzA39+f06dPl5gYPEpLS6tEzLW2tjZqtZpmzZoREREBQEFBAWlpaaXG8DjFUd2ajrX49WKPG6sQQohnT255VNLx48eVyUJ4eDijR4/m5MmTXLp0CSi61bBs2TLatWvHrVu3OH/+PADR0dFltte5c2dOnTpFSkoKAFu2bKFHjx60bNmSO3fucPr0aQB2795NkyZNMDIy0qifw4cPJzMzU9lT8bhbHllZWcTGxgIQFhZGv379aNmyJRkZGfz6668A7Nixg9mzZ2t0/rJ0796d2NhYpaDY1q1b6dGjR7nXQAghxItLvt5VUuvWrZk7dy5///035ubmfPrpp3To0IGZM2eiVqtp1KgR/v7+6Orq8uWXX+Lm5oaOjg4dOnQosz0TExMWLVqEq6sreXl5NGnSBF9fX/T09Fi5ciU+Pj5kZWVhaGjIypUrNe7nrFmzcHd3R0dHB319fRYuXPjYz8TExLBy5UpMTU3x8/NDT0+P1atXKxtD69ati5+fn8Z9eNTbb7/NpEmTcHZ2Ji8vj44dO7Jw4ULq1q1b5jUQQgjx4pLobVGmdu3aceHChefdDfGSe5GDrWp6WqiMX8Yv0duiSmRnZ+Po6Fjme9OnT69U287OzmVukHRycmLEiBGVarsqSfR2zf3LVAhR9WRCUUPVqlVL2VxZlsqsTkj6pxBC1DwyoRA1ltTyqNpaHi/y7Q0hRPWTCYWoEo+rTRIbG8vly5cZO3Ys33//PQAjRozAw8MDV1dXmjZtqtF5UlJScHFxUZ5AqQyp5VG1pG6HEDWbPDYqnok//vhDCcEaMWKEspciPj4e2RcshBAvP1mhEFUqISGBlStXkhidMLkAACAASURBVJ2dTUZGBm5ubrRp04bQ0FCgKN3y2rVrQFHAVWpqKhMnTmTTpk04ODjw7bff0qxZsxIrHmfPnmXevHlA0aOmxdLS0vDy8uLGjRuoVCo+++wzevXq9ewHLYQQQlYoRNX67rvvWLx4MTt37sTX15e1a9fSunVrnJyccHJywsHBQTl24sSJmJqaEhISosRvl2XOnDm4ubmxc+dOJY4bwNfXFwcHB8LCwli3bh1eXl4losCFEEI8O7JCIaqUv78/Bw4cICYmhlOnTpGZWbk9Crdv3yY1NVVZebC3t2fHjh0AHDlyhIsXLxIQEABAfn4+V65ceWxxMiGEEFVPJhSiSo0cOZIePXrQo0cPevbs+cTR3MX7KYprd6hUqhJ7LB6ugaJWq/nmm2+U+PGbN29iYmJS2SEIIYR4CnLLQ1SZO3fukJSUxIwZM3j//fc5fPiwUrxMW1u7RIGvYtra2soxxsbG/PXXXwDs379fea1Jkyb89NNPAOzatUv5rLm5OZs3bwbgr7/+YsiQIWRlZVXb+IQQQpRPJhSiyhgZGTF8+HAGDx7M0KFDuXXrFtnZ2Tx48IBu3boRFRVV6rHSDz74gIkTJ3LlyhWmT5+u7IuoV+9/GQn+/v4EBQUxdOhQkpOTldc9PT05deoUNjY2/Otf/2LZsmXUrVuzsyWEEOJ5kVoeQogq8bIFW9X0+HEZv4xfankIUUWklkfN/ctUCFH1ZEIhaqyaGL39sq0iCCFeHjKhEACEhYWRkJDA0qVLn+l5AwMDAZg2bZpGx5cX8f00kdw1MXpb4rGFENVFNmUKIYQQotJkheIx4uPj8ff3R61W06xZM3R1dblw4QIqlYpx48YxdOhQ1Go1S5YsIS4uDpVKxZAhQ5g4cSLx8fEEBwdTWFhIcnIygwYNol69euzbtw+AkJAQDA0NmTt3LomJiUBRjsMnn3xSbn8CAwNJSkoiOTmZO3fu4OjoyPjx4ykoKGDZsmUkJCRQUFCAvb09Y8aMKdH/Nm3a4OfnV27bly9fxtnZmWvXrtGzZ08WL14MQHBwMJGRkWhra9O7d2/c3Ny4fv16iRWB4pWGyZMnlzmeimKyT58+jZOTEzdv3sTe3p5p06aVe00fVl4ktxBCiGdPJhQaSEpK4sCBA6xbt47c3Fx27drF7du3GT58OG+//TbHjx/n+vXrREZGkpubi7OzM23btsXAwIBTp04RHR2NkZERvXr1Ys6cOYSFheHh4UF0dDTt27cnIyOD8PBw0tPT8fPzq3BCAfDnn38SGhqKWq3G3t6enj17cvr0aQB27txJbm4u48aNo1OnTiX6//CjmGW5fv064eHh1K5dmwEDBpCYmMi1a9eIjY0lLCwMHR0dpk2bRmhoKO+//36ZbZw4caLM8RQ/DmppaUlqaiojR44kPDwcgFu3bhEaGsr9+/fp378/Y8eOJSIiotxrWmzOnDl4eHjQq1cv1qxZQ3x8vMZ/pkIIIaqW3PLQQIsWLahXrx5Hjx5l2LBhADRo0ABLS0sSEhKIj4/Hzs4ObW1tDAwMsLGxIS4uDoC2bdvy+uuvY2BggLGxMT179gSKimTdvXuXNm3acOnSJcaNG0dkZKRGyZLW1tbUqVOHevXq0b9/f44ePUpcXByxsbHY2toyfPhwbty4wYULF0r0/3Hee+89jIyM0NPT44033iA9PZ2jR48yePBgatWqhY6ODg4ODsrYylLeeI4cOUJAQAC2trZMmDBBickG6Nu3L3p6ejRo0ABjY2MyMjIqvKZQdiS3EEKI50dWKDRQq1YtgFJltgsLCykoKECtVpf5OoCurm6J9x6OjoaiJMjo6GgOHz7MwYMHsbOzIzo6mvr165fbn0fjp4vTJt3c3Bg4cCBQ9Au3du3anDp1Sun/4+jo/O8/h+LI60fHBkWx2I9GYufn56Ojo1PueMqLyd63b59G5334mj58XFnXRAghxLMnKxRPwNzcnO3btwNFv7D3799P9+7dMTc3Jzw8nIKCArKysoiKiqJHjx4atbl//35mz57NBx98gKenJ7Vr1+b69esVfmbfvn3k5uaSkZHBgQMH6NOnD+bm5mzdupW8vDwyMzMZOXIkp06dqpIxR0dHk52dTX5+Pjt27MDc3Jz69euTkZHB7du3yc3N5eeff65wPE8ak/24a1pRJLcQQohnT1YonsDUqVPx9vbGxsaGgoICJk+eTMeOHWnbti1JSUnY2tqSl5fHkCFD+PDDDzW6p9+vXz/27NnD4MGD0dfXZ+DAgbRr167Cz+jr6zNy5Eju37/PpEmTaN26NW+++SaXL1/Gzs6O/Px87O3t6dGjR6X3FVhYWHDu3DkcHBzIz8+nb9++jBo1Ch0dHcaNG8ewYcNo3Lgx77zzToXj8fT0xMvLCxsbG4DHxmQ7Ojo+9pr6+/vj4eHBqlWr6NKlS6XGKYQQonIkevsl86S5DUI8rDjYSpIyJS1Uxi/jfy7R21lZWcTExJCRkVHivvXYsWOfqDNCM19//TU7d+4s9bqpqSlmZmZP3e7u3bv597//XeZ7ERERT93uy6omR28LIURV02iFYvLkyaSmptK2bVtUKpXy+hdffFGtnROiOtXkCUVN/3YGcg1k/DL+57JCcfHiRXbv3l1iN35NVV70c1X4/vvvARgxYkSVt10V3N3d6d69O71798bT05P//Oc/z7tLlfKi1vKQehtCiJeRRjOExo0bV3c/BC/uROJRjRo1euknE/Di1vKQehtCiJeRRhOKtm3b4uLiQt++fUtkGtTUPRS3b99mwoQJJCcn06JFCwICAoiKimLDhg2oVCo6duzI/PnzqVOnDu3atVMCph4uwOXn58fhw4fR1tbG0tISV1fXEhsu+/Tpw6BBgzh+/Dja2tqsWrWK5s2bEx8fz+LFi9HW1qZLly7897//rXC1xN3dHQMDA44fP869e/eYO3cuERERnD9/ngEDBuDu7l5ubHdhYSFLly7lp59+wtTUlIKCArp3716iENeff/6Jj48PDx484Pbt24wdOxYXFxcCAwO5efMmly9f5urVqwwfPpxPP/203H7m5+fj7e1NYmIiaWlptGjRgqCgIGrVqsW3337Ld999R7169WjZsiVvvPEG06ZN49ChQwQEBJCfn0+zZs3w8fHB2Ni4av+whRBCaESjHIrMzEzefPNNkpOT+fPPP5V/aqpr167h5eXFDz/8QFpaGt9//z3BwcFs3LiRqKgoDAwMCAoKKvfzV69e5dChQ0RGRhIaGkpSUhI5OTkljvn777/p2bMn4eHhdOvWjU2bNpGXl8fnn3+Ov78/4eHhGt+CSk1NJTIykunTp+Ph4cHChQsJDw9n69at3Lt3j61btwJFsd3bt29n//79/Prrr+zZs4ezZ8+ya9cuVq9eTXJycqm2t23bxpQpU9ixYwfffvstK1euVN67cOEC69evZ9u2bYSEhHD37t1y+3jixAl0dXXZsmULe/fuJScnh4MHD3L+/Hk2bdpEWFgYmzdv5vLly0DRpG7FihWsX7+e8PBw+vTpw/LlyzW6HkIIIaqeRr+RZPNlSW+//TbNmzcHoFWrVty7dw8LCwvl27GjoyMeHh7lfr5Ro0bo6+vj5OSEhYUFM2fORF9fv9Rxffv2BYrirH/99Vf+/PNPGjZsqBTCGjZsGL6+vo/tb79+/YCiuO82bdrQsGFDAIyMjMjIyCAuLo5z585x9OhRAB48eMCFCxf473//y8CBA9HV1aVBgwZKOw9zd3fn559/5t///jcXLlzgwYMHyns9evRAT0+Phg0bYmRkxL1798pNAO3WrRtGRkZs2rSJixcvkpSUxIMHD4iLi8PCwkLJrBg8eDB3797l1KlTSoEyKEoMNTQ0fOy1EEIIUT00mlDEx8cTEhJCRkZGideLUyNrmkejouvXr1/i23dhYSH5+fklflapVMprOjo6bNu2jYSEBA4dOoSTk1OZty2KJxnFMdPa2tplRmE/zsPx32WtapQX211cpbSiz86cOZP69etjYWHBxx9/THR0dKn+PzyG8uzfv5+AgABcXFywt7cnPT2dwsJCtLS0yhxzQUEBXbt2JTg4GICcnBwyM1+8/RBCCFFTaHTLw9PTk/fffx9XV9cS/4j/iY2N5c6dOwBs3bpViYk2NjYmMTGRwsJCpdT32bNnGTVqFN26dWPOnDm0atWKS5cuPfYcLVu25O7du8qejKioqCrpe3mx3T179iQmJkaJ+S6O137Y4cOHmT59OgMGDODYsWMAJWpuaCouLg4rKyscHBwwMTHh2LFjFBQU0LNnTw4ePMj9+/fJzc3lxx9/RKVS0blzZ06ePKlct7Vr17Js2bLKXQghhBBPTaMVioYNGypLy6K0unXrMmnSJJydncnLy6Njx44sXLgQgM8++4zJkydjYmLCP/7xD9LT0+nQoQNdunTB2toaAwMD2rdvT79+/fjjjz8qPI+enh7Lli1jzpw5aGlp0aJFC40Lf1XEycmpzNhugN9//x1ra2tMTExo1apVqc9OmzaNkSNHUr9+fVq0aEHTpk1JSUl54j4MHz6c2bNnExMTg56eHl26dCElJYXhw4fj4uKCo6MjtWvXxtjYGH19fV577TWWLFnCzJkzUavVNGrUCH9//0pfCyGEEE9Ho2CrkJAQ6tSpQ9++fUssezdp0qRaOydKUqvVLF++HFdXV2rXrs2GDRu4efMm7u7uz7tr1ebSpUscPHiQMWPGAPDpp58yfPhw+vfvX+m2X+THRqs7cKemh/qAXAMZv4z/uQRb3b59my+//BIDAwPlNZVKxW+//fZEnRGVo6WlhZGREcOGDUNXV5emTZvi6+uLn58fR44cKXV8p06dNNq0+az8+uuv+Pj4lPleSEgIjRo1KvV606ZNlVUSlUpFnz59sLCwqJL+BHgMrZJ2qlp2Tt7z7oIQQjwxjVYoPvjgA7Zv346Jicmz6JMQz4REb9fcb2cg10DGL+N/LisUDRs2pEGDBk90YiFedM8jeltitYUQryqNJhTvvPMOI0eOxMLCAj09PeX1mpqUKZ6tefPm4eTkxFtvvcWcOXNYu3ZtlbT7PPZQSKy2EOJVpdGEIicnhxYtWpCUlFTN3RGitOJ9ICkpKZw/f/4590YIIURZJClTvFAKCwtZvnw5+/btQ1tbG0dHR/bt24erqysbNmwgNTWVqVOn0qZNG9RqNbNmzQLAw8ODvn378vHHHz/nEQghRM2k0YRi8uTJZb5enFIoRFWJiYnht99+Iyoqiry8PEaOHKnUOfH09MTFxYU1a9Zw5coVRo8ezb/+9S+ysrKIi4tTsj+EEEI8expNKAYNGqT8e15eHrGxsbRr167aOiVqrmPHjmFlZYWenh56enpERETg7Oxc6rjmzZvTtGlTjh07xrVr13j//fdL7O8RQgjxbGk0obCzsyvxs729PaNGjaqWDoma7dF6ISkpKSUKjj3MwcGBXbt2ce3aNaZNm/YsuieEEKIcGtXyeJRarSY1NbWq+yIE3bp1Y+/eveTl5ZGVlcX48eO5efMmUDTZeLjo2kcffURcXBxpaWl07tz5eXVZCCEET7mH4s8//6R79+7V0iFRs3344YecOXMGe3t71Go1Li4u/PDDD0BRHkqTJk1wdnZm48aN1KpViy5dutC2bdvn3GshhBAaJWXu3Lnzfx9QqTA2NqZPnz5oa2tXa+eEKE9hYSGZmZk4Ojry9ddf89prrz3vLmnkRQm2qukpgSDXQMYv43+mSZnF5bjLqp1w7949jIyMnqgzQlSV33//nfHjxzN16tSnnkzU5OhtIYSoahVOKMzNzVGpVEDRN8KHqVQqzp07V309E6ICZmZmJCQkVKqNZxW9/aKsSgghRHWqcEIhqYTiVfasorclblsIURNo9JSHWq3mP//5D87OzowYMYKgoKASu+3Fy+HevXtMmTKlwmM8PDy4evVqhcc4OzsTHx9flV0TQgjxktNoQrFixQqOHj3K6NGjGTt2LCdOnMDPz6+6+yaqWEZGxmNXneLj40vd3hJCCCEeR6PHRn/++Wd27NiBrq4uAB988AFDhgyp1o6Jqrd48WKlFkb//v3ZsGEDKpWKjh07Mn/+fDZt2kRqaioTJ05k06ZNHD16lA0bNpCdnU1OTg6LFy+mW7dujz1PSkoK48ePx9jYGH19fb766iuWLFlCXFwcKpWKIUOGMHHiRKAovj0yMhJtbW169+6Nm5sb169fZ+rUqTRv3pw///yTTp060b17d3bu3ElGRgZr1qyhVatW+Pn5cfjwYbS1tbG0tMTV1bW6L6EQQohyaLRCUVhYqEwmAPT09Er8LF4Onp6emJqaMn36dIKDg9m4cSNRUVEYGBgQFBTExIkTMTU1JSQkBENDQ0JDQ5Vf+BMmTGD9+vUan+vSpUv4+/vz9ddf8/3333P9+nUiIyPZtm0bP/74Iz/99BMHDx4kNjaWsLAwdu7cyeXLlwkNDQXgwoULTJkyhZiYGH7//XeuXr3Kli1bsLa2ZsuWLVy9epVDhw4RGRlJaGgoSUlJSs0PIYQQz55GKxRvv/02S5YsUeK2v/vuOwkTeokdO3YMCwsLjI2NAXB0dMTDw6PEMVpaWqxZs4bY2FguXbpEQkICWlqaB6s2bNiQZs2aAUW3Uezs7NDW1sbAwAAbGxvi4uLQ0tJi8ODB1KpVCyiK0g4PD+f999/HxMSEDh06ANC4cWN69uwJQJMmTUhJSaFRo0bo6+vj5OSEhYUFM2fORF9fv9LXRgghxNPR6DfEggULuHv3Lk5OTnzyySekp6czf/786u6bqCZqtbrEz4WFhaU22WZmZuLg4EBKSgrdunUrs0BXRYonCeWdr6CgoNTrgNKPRwt9PRqipqOjw7Zt25gxYwZ37tzBycmJS5cuPVEfhRBCVJ0KJxS5ubnMmTOHo0ePsnTpUo4cOYKZmRna2trUrftsnuEXVae4Fkb37t2JjY1Vgsu2bt1Kjx49gKJf3AUFBSQlJaGlpcXkyZMxNzfn0KFDFBQUPNV5zc3NCQ8Pp6CggKysLKKioujRowfm5uZER0eTnZ1Nfn4+O3bswNzcXKM2z549y6hRo+jWrRtz5syhVatWMqEQQojnqMIJRUBAAPfv3+fdd99VXvPx8eHu3bsEBgZWe+dE1SquheHr68ukSZNwdnbmo48+4u7du8ycORMo2nA7ceJE6tWrR/v27bGyssLOzo7atWtz7dq1pzqvo6MjjRs3xtbWlqFDh9K/f38+/PBDLCws+OCDD3BwcGDw4ME0bdpU4yq2HTp0oEuXLlhbW2NnZ0fTpk3p16/fU/VPCCFE5VVYy8Pa2prt27eXWL4GuH//Po6OjkRHR1d7B4V42b2ISZk1vY4ByDWQ8cv4n2ktD11d3VKTCYC6deuWusctapbk5GSmTZtW5nuLFy/mnXfeecY9enJSy0MIIapOhRMKLS0t7t+/X2q/xP379yUps4Z74403iIiIeN7dqJTqruXxIq5MCCFEdalwQmFtbY2npydLliyhdu3aADx48ABPT08GDhz4TDr4NOLj4wkKCmLjxo3P9LMvs/79+/Ptt98qj3o+K+3atePChQsaH+/s7Iyrq6uyibSYu7s73bt3x97eXuO2qruWh9TwEELUJBVuyhw9ejT16tWjd+/efPLJJwwbNozevXtTv359pk6d+qz6KIQQQogX3GNvefj4+DBp0iTOnj2LlpYW77zzDo0aNXpW/Xtq6enpjBs3jtTUVMzMzFiwYAFHjx4lICCA/Px8mjVrho+PD8bGxvzyyy988cUX6Ovr06JFC6UNZ2dnDA0NSUxMZNWqVdy4cYNVq1ahVqtp3rw5ixYtwsTEhJMnT+Lr60tOTg7GxsYsWrSIN998E2dnZ9q3b09cXBzZ2dl4enqyceNG/vrrL8aMGcOYMWOIi4vD398fAENDQ1asWEGDBg3KHZe5uTkWFhacOXOGOnXqsHz5cpo1a8bp06f54osvyM7OxtjYmIULF9K8efNSY2jfvn25ba9Zs4Zz586RlZXFsmXL6Ny5M5cuXcLLy4s7d+5Qu3Zt5s2bh5mZWakVgeKVhvLGEx4ezjfffINaraZjx44sWLBACaLy8vLi5MmTAAQGBvLmm2+We02LFRYWsnTpUn766SdMTU0pKCige/fuT/lfixBCiMrSKNiqWbNmDBw4kAEDBrwUkwkoqicxf/58IiMjyczMJCQkhBUrVrB+/XrCw8Pp06cPy5cvJzc3F3d3dwICAggLCyu1CbVdu3bs2bMHU1NTvLy8WLNmDVFRUXTt2pVFixaRm5vLrFmzlHM5OTkxa9asEm1ERUVha2vL4sWLCQwMZNOmTaxZswaAtWvX4u3tTVhYGBYWFpw9e7bCcaWnp9O9e3eioqIYPHgwixcvJjc3F09PT1asWMHOnTsZO3ZsieCx4jFUNJkAaN26NeHh4Tg7Oysx225ubjg7OxMVFYWHhwczZswgNze33DbKGk9iYiJbt24lNDSUiIgIGjZsWCLGu1evXkRGRtK7d29CQ0M1uqZ79uzh7Nmz7Nq1i9WrV5OcnFzh2IQQQlQvjaK3X0bvvfceb731FgA2Nja4u7ujUqlwcXEBitIbDQ0NuXDhAqamprRq1QoAOzs7Vq9erbRjZmYGwOnTpzEzM1P2GDg6OhISEkJSUhL169dXjrOyssLLy4t794oexynORmjSpAmdO3fGwMCApk2bcvfuXQClqNWAAQOwtLSkd+/eFY5LX1+foUOHKn398ssvSUpK4sqVK3z66afKcffv3y81hscZMGAAUDSx2LNnD5mZmSQnJyv7Zbp06YKhoSEXL14st42yxvPdd99x+fJlPvnkEwDy8vKUWO1Hz/vrr78+9poCJCQkMHDgQHR1dWnQoIFkUAghxHP2yk4odHT+N7TiqI2uXbsSHBwMQE5ODpmZmVy7dq1EBPSjEc/FKxblxVWXFR9dHC0NlCii9nCfio0ZMwYLCwsOHDiAv78/p0+fLjExeJSWlhYqlUrpk7a2Nmq1mmbNmilPXRQUFJCWllZqDI9TPPbi9gsLC0uVMi8em0qlUt7Ly8urcDy1a9fGysoKT09PoCjW++HUzeLrUtzm465p8bEPH1fWtRVCCPHsaF7t6SVz/PhxZbIQHh7O6NGjOXnypBLPvHbtWpYtW0a7du24desW58+fByg3rKtz586cOnWKlJQUALZs2UKPHj1o2bIld+7c4fTp0wDs3r2bJk2aYGRkpFE/hw8fTmZmprKn4nG3PLKysoiNjQUgLCyMfv360bJlSzIyMvj1118B2LFjB7Nnz9bo/BWpW7cuzZs358cffwTg5MmTpKWl0aZNG4yMjPjrr78A2LdvX4Xj6dGjB3v37uXWrVsUFhbi7e3NN998U+55NbmmPXv2JCYmhtzcXDIyMvj5558rPV4hhBBP75X9Wte6dWvmzp3L33//jbm5OZ9++ikdOnRg5syZqNVqGjVqhL+/P7q6unz55Ze4ubmho6NTYin+YSYmJixatAhXV1fy8vKUCGs9PT1WrlyJj48PWVlZGBoasnLlSo37OWvWLNzd3dHR0UFfX5+FCxc+9jMxMTGsXLkSU1NT/Pz80NPTY/Xq1comxrp16+Ln56dxHyri7++Pt7c3gYGB6OrqEhgYiJ6eHiNHjmTmzJnY2Nhgbm7Oa6+9Vu542rZti6urK6NHj0atVtO+fXsmTpxY7jk1uaYDBgzg999/x9raGhMTE+WWlRBCiOejwuht8eJ50twG8fy8yMFWNT12GOQayPhl/M80els8e9nZ2Tg6Opb53vTp0yvVtrOzs7IZ9GFOTk6MGDGiUm2/jCR6Wwghqo6sUAhRTWSF4sVW06+BjF/GLysU4qVUXmT2o8cUR57b2tpWe60Qid4WQoiq88o+5SFePgkJCcq/v+yFx4QQoqaRFQoBFBVFCwwMREdHh+vXr2NmZoavry9RUVFs2LABlUpFx44dmT9/PnXq1Ck3AvzhImNlFVrLz8/H29ubxMRE0tLSaNGiBUFBQSxfvhwoeux027ZtyubTrKwsPD09uXDhAiqVinHjxjF06FDCwsL4+eefycjI4MqVK/Tu3Rtvb+/ndPWEEELICoVQnD59Gi8vL2JiYsjJySEkJITg4GA2btxIVFQUBgYGBAUFAWVHgGvixIkT6OrqsmXLFvbu3UtOTg4HDx5UQq+2bdtW4vjAwECMjY3ZtWsX33zzDYGBgUpmyIkTJwgICCAyMpIDBw7I0y9CCPEcyYRCKLp160bLli1RqVTY2tqydu1aLCwsMDY2Borixo8ePQqUjgCPj4/X+BwjR45k06ZN+Pr6kpSUxIMHD8o9/ujRowwbNgyABg0aYGlpqdwaeffdd6lbty4GBgY0b96cjIyMpx67EEKIypEJhVA8HDteVgR2cdw4lB0B/vBxgHLsw/bv38/s2bOpVasW9vb2dOvWrVS896PnfPTn4gju4mqlQIkocCGEEM+eTCiE4vjx49y8eVOJK/fw8CA2NpY7d+4AsHXrVuUpjbIiwAGMjY2VSO79+/eXOkdcXBxWVlY4ODhgYmLCsWPHlAmCtrZ2qUmIubk527dvB+D27dvs379fypQLIcQLSCYUQmFqasrnn3/Oxx9/TKNGjRg1ahSTJk3C2dmZjz76iLt37zJz5kzl+JiYGGxsbPj555+ZO3cuUBS+5evri4ODA/Xq1St1juHDhxMdHc3QoUOZNm0aXbp0UeqjWFpaYmtrS05OjnL81KlTuXPnDjY2NowaNYrJkyfTsWPHar4SQgghnpQEWwmAMp/IqIhEgD+eBFu92Gr6NZDxy/gl2EqIKiLR20IIUXVkQiEA6NGjR4Uplo+S1QkhhBAPkwmFqLEqWrqrrBf5docQQlQHmVDUQE+6X6IqhIWFkZCQwNKlSzU6PiUlBRcXF+VJkodV1f6N6qzlIXU8hBA1jTzlIYQQQohKkxWKunqtzAAAIABJREFUGur27dtMmDCB5ORkWrRoQUBAQLl1Ox5eEXh4pcHPz4/Dhw+jra2NpaUlrq6uZGZmsmjRIhITEykoKGDChAlYW1sDcPnyZZydnbl27Ro9e/ZU4rqDg4OJjIxEW1ub3r174+bmVqKvKSkpuLm58eDBAzp37qy8HhcXh7+/PwCGhoasWLGCBg0aPIvLJ4QQ4hGyQlFDXbt2DS8vL3744QfS0tL4/vvvy63bUZarV69y6NAhIiMjCQ0NJSkpiZycHNatW0fHjh0JCwtj06ZNBAcHc+XKFQCuX79OYGAgP/zwA4cOHSIxMZGDBw8SGxtLWFgYO3fu5PLly4SGhpY4l4+PD/b29kRERNC1a1fl9bVr1+Lt7U1YWBgWFhacPXu2ei6WEEKIx5IJRQ319ttv07x5c7S0tGjVqhX37t0rt25HWRo1aoS+vj5OTk58/fXXzJw5E319fY4cOUJoaCi2trb885//5MGDByQmJgLw3nvvYWRkhN7/196dR2VVdQ8c/z6AA4TibJJDikkaOfzUIlMLQiZBTNJ4VZwhxBGUVwglRDGcokDTTMskjVRIQU1dBYqzluaQYzliCqGoiCLT/f3B4r4+MogMorA/a7GW3PHse596Dufcu3fNmrRs2ZLU1FT2799P3759qV27Nnp6ejg7O7Nv3z6tcx08eBA7OzsA+vXrR40aNQDUUZGgoCBMTEzo2bNnRVwqIYQQJSBTHtWUnt7/br1Go6Fu3brcuXNHXfZw3Y783zUajbpMT0+PdevWcfDgQRISEnBxcSEiIoLc3Fzmz5+vZrNMSUnByMiI2NjYAucsrF4IFF4DJD//mkajUWuIjBgxAgsLC+Lj45k/fz7Hjh1j7NixZbksQgghSklGKISqqLod9evX59y5cyiKor51cfLkSYYOHUr37t2ZNm0aJiYmXLhwAXNzc3744QcAkpOT6devH9euXSvynObm5mzevJmMjAyys7OJiorC3Nxca5sePXoQExMDwPbt28nMzATy0ninp6czYsQIRowYIVMeQghRiWSEQgBgaGio1u3IysritddeY+bMmQBMmTIFDw8PGjVqRNeuXUlNTaVDhw507twZBwcH9PX1ad++Pb179+aNN94gMDAQBwcHcnJy8PHxoWXLlvz222+FntfCwoJTp07h7OxMdnY2vXr1YujQoVy/fl3dJiAgAB8fHyIjI3n99dd54YUXAPD29sbX1xc9PT1q1aqltlcIIcTTJ7U8hKgAz3piq+pexwDkGkj8Er/U8hCinEgtDyGEKD/SoRDVVkWk3n7WRyaEEKKiSIdCVFsVkXpbUm4LIaorecvjGZeWloanp2ex2/j5+XH16tVit3F1deXAgQNFrk9MTMTS0rJUbSyP/YUQQjzfpEPxjLt9+zanT58udpsDBw4gz9YKIYSoTDLl8YybPXs2ycnJjBs3DktLywK1NlavXk1ycjLu7u6sXr2a/fv38+2335KRkcGDBw+YPXs23bt3L9G5Hjx4wKRJk7hw4QItW7YkODgYIyMjjh07xqeffkpGRgb169dn5syZtGjRgpMnT+Lv7w/kZd7M5+vry61bt7h06RI+Pj40aNCA4OBgHjx4QP369QkKCqJVq1ZcuHCBgIAAbt26hYGBAf7+/nTs2BFfX1/09fX5/fffSUtL4+OPP2bjxo2cPn0aKysrfH19OX36NAEBAWRnZ1OrVi0+/fRTXn755Yq4BUIIIUpARiiecdOnT6dJkyZMnDix0Fob7u7uNGnShGXLlmFkZERkZKRabMvNzY0VK1aU+Fw3btzA1dWVmJgYWrZsyeLFi8nMzGT69OksXLiQn376iZEjRzJjxgwApk2bho+PDz/99BPNmzfXOla9evX4+eef6dmzJ97e3syYMYOYmBhcXFzw9vYGwMfHB1dXV2JjY/Hz82PSpElq0qrk5GRiYmKYOHEifn5+zJw5kw0bNrB27VrS0tL47rvvGDlyJNHR0bi6uvLHH3+U0xUXQghRGjJC8Zw4dOhQgVobfn5+Wtvo6OiwePFi4uLiuHDhAgcPHkRHp+R9xtatW9OtWzcgr2aGr68vFy9e5MqVK1opre/evcvNmzdJTk6mR48eAAwYMICoqCh1m44dOwJw8eJF6tatq/5uZ2dHQEAAaWlpXL58GWtrawA6d+6MkZER58+fB6B3794AGBsb88orr9CwYUMgr6Ny+/Zt3nnnHYKCgti1axcWFhbY2NiUOE4hhBDlTzoUz4lHa148WmsDID09HWdnZ5ycnOjevTumpqasXr26xOd4uNZG/u+5ubk0b96cjRs3ApCTk0NKSopaiyOfrq6u1r61a9cutN35bU9LSyvw3IeiKOTk5ACoBcAKaxeAra0tXbp0IT4+nu+++46dO3eq5dCFEEI8fTLl8YzT09MjOzubN954o8haG7q6uuTk5HDx4kV0dHTw8PDA3NychIQE9Qu6JP7++2+1Hsb69evp0aMHbdq04fbt22rq7KioKKZOnUr9+vUxNjZmx44dAGzatKnQY7Zp04Zbt25x7NgxALZs2YKxsTHGxsa0aNGC7du3A/DHH3+QkpLCK6+8UqK2Tp48mWPHjuHi4sKkSZOkjocQQlQyGaF4xjVs2BBjY2OCg4OLrLXx7rvv4u7uztdff0379u2xs7Ojdu3adO/enX/++afE58p/buLy5cu0a9cOLy8vatasyRdffKE+VGloaMjcuXMBmD9/Pn5+fnz++ed07ty50GPWrFmT0NBQZs2axf379zEyMiI0NFTdPzAwkPDwcGrUqEF4eDg1a9YsUVs9PDzw9/fnyy+/RFdXF19f3xLHmS/Mr/8T7/M4GQ+yyv2YQgjxPJBaHqLaqs6pt6t7HQOQayDxS/xSy0OU2uXLl5kwYUKh62bPns3rr7/+lFskhBCiqpAORTXSsmVL9eFKUT61PKR2hxBC5JEOxVPwxRdfYGZmxnvvvVei7d3c3Jg9ezaZmZksWbKEOXPmlHub7t69i4uLC0uXLlVzSJiamnLmzBmt7SwtLVm1alWBPBPp6eksWLCA3bt3o6+vj6GhIRMmTOCtt95St9mxYwdLly7l3r175ObmYmVlxcSJE9HR0cHX15f9+/djZGSkddzo6Gj1jZHvv/+ekJAQ4uPjady4sbqNqakps2bNYtCgQeoyV1dXxo8frz6oWhLlUctDancIIUQe6VA8BZMmTXqi7b/++msgL6X2lStXynz+nJwcfvnlFwwMDOjVqxdHjx5l+vTpXLx4sVTHUxQFDw8P2rdvz+bNm6lZsyYnT57E3d2dhQsX8uabb5KQkEBQUBArVqygdevWZGRkMHnyZMLCwpg8eTIAEydOZMCAAUWeJzo6GktLS9avX6+VBwMgNDSUXr160axZs1LFIIQQonxV+mujS5cuxd7eHkdHR0JCQtTXHFeuXImNjQ329vbMnz8fgJSUFDw9PRkwYADOzs7s3bsXgKSkJEaPHs2gQYOwsLBgwYIFQN4XkpeXF6NGjaJPnz4EBgYW25bExEQcHR3x9PSkb9++uLm5qa9pmpubM3r0aJycnMjKyiq03b/++ivW1tY8ePCAS5cu0bt3b65fv46vry/R0dEkJibi5OTE+PHjsba2xtvbm8jISD788ENsbW35+++/gbxRgcTERGbPns2JEyeYOXMmPj4+/Pjjj2pbXV1dOXr0aLHxpKWl8c0332Bvb8/27dvVUYa1a9fyySef0KRJkye8W3kOHjzIP//8g5+fn/pWRocOHRg7dixffvklkHdfx48fT+vWrYG8vBSBgYG88cYbJTrH6dOnuXXrFu7u7qxbt65APothw4Yxffr0UrVfCCFE+avUDsXOnTuJi4sjOjqan376iUuXLhEZGcmxY8dYs2YN69evJyYmhj///JMTJ04QHByMs7Mz0dHRLFmyhICAAO7evcumTZtwcHBg7dq1xMTEsGbNGm7evAnAkSNHCAsLIyYmhvj4+AJD+o86e/Ysw4cPZ/PmzZiYmLBo0SIAUlNTcXd3Z+PGjezdu7fQdr/33nt06dKFJUuW4Ofnx7Rp03jxxRe1jn/mzBk8PT3ZunUrx48f5+rVq/z44484ODhodRggL+22mZkZn3zyCc7OzsTExABw9epVbt68SadOnYqM46uvvsLZ2Zn09HQiIiJYuHCh+uUeHBysZsR8lJOTk9ZPcnJygW2OHz+OmZkZGo1Ga3n37t05fvw4AKdOnSrQvhdffFHNrAkQFhamda7812AhrzNoa2uLmZkZurq67Nq1S+tYbm5upKamsnbt2iKvgRBCiKenUqc89u/fT9++fdWsis7OzmzYsIGMjAwsLCyoU6cOkDdaAbB3717Onz9PWFgYANnZ2Vy5coXRo0ezf/9+VqxYwblz58jKyuL+/fsAdOnSBUPDvIfvWrRowe3bt4tt08svv6zOw/fv35+pU6eq6/K/IItq95AhQ/D398fe3p7/+7//o2/fvgWO36hRIzp06ADkfcHmP3NgbGxMYmJike168803mTFjBomJiWzcuBEnJ6di49BoNOrPk6TffvShzcJKkms0mkITZmVlZWlt87g3koua8sjKyiI2NlatQ2Jvb09kZCTvvPOOuo2enh4hISEMHz6cXr16FR+UEEKIClepHYrC0jJnZ2cXSLWclJSEvr4+ubm5fPfdd9SrV09d3qhRI0JCQrhy5QoODg5YWVmxd+9e9cusVq1a6nFK8iX38LkVRdFKKV1cOun8NNgpKSno6upy4cIFMjMzCyRqevT3R1NWF0Wj0dC/f382b97M1q1bWb58ebHbu7u785///Id169YxZMgQOnXqxLhx42jVqlWJzlecTp06ERERQVZWllaK7D/++EN99dTMzIwTJ07Qtm1bdf2FCxdYsmQJ8+bNK/b4O3bs4M6dO4wfPx7I62DcuHGD69eva434tGvXTqY+hBDiGVGpUx7m5uZs3ryZjIwMsrOziYqKwtzcnG7dupGQkEB6ejrZ2dlMmTKFEydOYG5uzpo1awD466+/6NevH/fv32fPnj2MHj0aOzs7rl27RlJSUqFf+iVx4cIFTp06BeSlmc4vUlWSdufk5ODn54e/vz/du3fn888/L/3FIa+z8XC9jgEDBhAZGcmLL75I06ZNH7t/nTp1GDVqFFu2bOHdd98t9UOYj+rWrRtt27Zlzpw56qjEiRMnWLJkCZ6engCMGTOGRYsWqedMT08nJCSkRA9RRkVFMWnSJOLi4oiLi2PXrl107dqVdevWFdg2f+rjyJEj5RKbEEKI0qnUEQoLCwtOnTqFs7Mz2dnZ9OrVi6FDh6Knp8fQoUNxcXEhNzeXPn360KNHD0xMTAgICMDR0RGAefPmYWhoyEcffcR///tf6tatS8OGDTEzMyt2+qA4RkZGhIWFcfnyZUxNTQstOFVUu7/55hsaNmyItbU1PXr0wMHBQa2mWRomJiakpaXh4+PD/PnzadasGc2aNeP9999/ouPo6upib29f6nYUZtGiRYSGhuLg4ICuri5GRkbMnz9fnS7q3bs3Xl5eeHl5kZOTQ3Z2Nra2tuqoA+Q9Q/Hdd99pHdff358DBw4UeFV25MiRBAYGqh2WfPlTH8W9LSKEEKLiSerthyQmJjJs2DDi4uIquykFKIpCcnIyrq6ubNq0qcQ1L0TFel4TW1X3tMMg10Dil/gl9XYZPS799LNq27ZtBAYGEhgYqHYm5s6dq746+zAzMzOCg4OfdhOfO9W5locQQpQ3GaEQogjP6+hDSVT3v85AroHEL/HLCMUzpDJSaoeHhxMbG0tMTIz61smBAwdYtGgRERERhIeHExkZSaNGjbT2W7p0qfpAZHx8PB4eHkRFRWFmZqZuY2lpSe3atdU3N9LS0jAzMyMkJAQDA4MCbVm5ciU//vgjurq66OrqMmjQIIYMGaKuP3/+PPPmzePq1atA3lsZ/v7+NGjQgOjo6EIf0gwKClJfzz179iyOjo6EhYVhY2OjbuPq6kqTJk1YuHCh1nUBihx9KszjUm9LWm0hhCg56VCUQWWl1P7nn3/47LPP+Pjjjwtd7+LiUuwXa3R0NDY2NkRGRhaY5lm2bJmaUTMzM5PBgwezYcMGBg8erLVdeHg4hw4dIiIigkaNGnHz5k08PT25desW48aNIykpiWHDhhEUFISlpSWKovDVV18xfvx49U0dS0tLQkJCStTOhzsUkDcFZGdnh5WVVdEXSgghxFNTYa+NSkrtikup7eLiwpYtW/jtt9+e+L7cvHmTffv28d///petW7dy9+7dIrdNS0sjLS1NzfuR7/79+6xYsYI5c+aoIyENGjRg9uzZLF++nPv37/PDDz/Qs2dPNTGWRqPBzc2NwYMHa70KW5Ts7GxiYmLw8vLi5MmTXL58WWv92LFjmTlzpnofhRBCVK4K6VBISu2KS6kNea+2BgYG4u/vT0ZGwSH5yMhIrZTW48aNU9fFxsby9ttv07x5c8zMzApkxnR3d8fR0ZEePXrg5ubG0KFDsbOz09rm3Llz6OvrF6hA2rZtW2rWrMn58+c5deoUHTt21Fqvq6uLg4ODmjwsLi5Oq50DBw5Ut92xYwfGxsa0bt0aKysrIiMjtY7VrVs3bG1tmTVrVrHXSgghxNNRIVMeklK74lJq57OysuLnn3/ms88+K/AMR3FTHtHR0WouCHt7e77//nut5x7ypzy2bdvGp59+iqWlZYGaHUWl3ob/ZQwtSVbS4qY8oqOjcXBwUNs5depUJk+erPW6rLe3N05OTvzyyy/FnkcIIUTFq5AOhaTUrriU2g+bMWMGDg4OBaYkinLy5EnOnj1LcHAwn376KTk5OSQnJ3PkyBG6dOmita2NjQ179uwhICBAramRr23btmRlZXH+/HnatGmjLj937hy5ubm0adNGTb39sNzcXCZOnPjYKaobN26QkJDAiRMnWLVqFYqicOfOHbZv3652MgD09fWZM2cOXl5e2NjYYGRkVKLrIIQQovxVyJSHpNQuXllTauerV68egYGBasnwx4mOjmbQoEHs2LGDuLg4du7ciZOTU4EpmXyTJk3iyJEjxMfHay3X19dn7Nix+Pv7c+PGDSCvEzBjxgzGjBmDvr4+H374ITt37mTnzp1AXifuyy+/5MaNGwXeQHlUTEwM5ubmJCQkEBcXp76VUlg786c+Hp0SEUII8XRVyAiFpNQuXnml1Ia8qQ8bGxutMuORkZEFpgF8fHyIjY1l1apVWstHjBjBhx9+iJ+fX4FjN2zYEDc3N+bNm0evXr20Rnnc3d2pU6cOI0aMQFEUNBoNLi4u6vRJ48aN+frrr5k3bx4LFiwgJyeHDh06sHjxYvUY+c9QPGzkyJHqQ7cPGzx4MMuXL1cfcH2Yt7e32nERQghROapFYitJqS1KQxJbVW3V/RpI/BK/JLYqgqTUFk9KUm8LIUT5qRYjFEIUpjp3KKr7X2cg10Dil/jLe4SiwhJbiZL54osv+PXXX0u8vZubG0lJSVy5cqXITJmPEx4ejrW1tVYOiwMHDuDq6qquf/vtt7VyRDg5OXHt2jV1+/j4eExNTQu8yWFpaYm9vb26j6WlJRMnTuTevXsF2mFqaqpuZ29vz5QpUwrk1Zg7dy7m5uZkZmaqy4KCgpg4caLWdrt37+a9994rNlHXoxo2NKRx4zpaP3Xq1i7x/kIIIf6nykx5PK+qc/puQE2spSgKEyZMICoqSn2wMzs7m59//pkuXbqwdetW+vXrB8CUKVNwdHQkLi4OS0tL7t27R2BgIHPmzFFzk5REYbU8pH6HEEKUzjM9QiHpu6tu+u5H5Scte/iV0p07d9KiRQv69++vFd8LL7zA7NmzCQoK4t69e4SFhWFpaakmLhNCCPH0PbMdCknfXbXTd+fLP36vXr34999/1QyjkNfps7W15Z133uHUqVP89ddf6roePXrQs2dP/Pz82LNnD97e3sXGK4QQomI9sx2Kh9Ng6+np4ezszL59+zh06JCavltPT4+VK1diZmbG3r17CQsLw8nJCTc3N6303c2aNWPFihUEBwcXmr5bX1+/VOm79+/fr64rLH33w+0G8Pf3Z/369TRq1KjY9N06OjoF0nffuXOnyHa9+eabJCcnk5iYyIYNG54ofbeZmRmfffZZgXUuLi5s3LhR/Xk4f8SjabEf7ewsW7aM2NhYPvnkE27evFlo+u58+cfft28fvXr1UvNP3Lx5k927d2NnZ0ft2rWxsLAokLzK19eXPXv2MH36dDXbqRBCiMrxzD5DIem7q3b67kfp6Ojg6OioZkyNiYlBURQ++OADADIyMsjKymLq1KnqtTY0NKRu3bq89NJLJY5XCCFExXhmRygkfXfxnvf03YXZt2+fWmAtKiqKkJAQ4uLiiIuLY/fu3RgZGbFly5YSxyaEEOLpeWZHKCR9d/GqQvpuQJ2eyc7Opn79+gQFBXHixAlSU1Pp06ePup2Ojg7Dhw8nMjKSAQMGPHGMQgghKpYktiohSd9d9RT12mh1SHZT3ZP6gFwDiV/il9TbFUjSd1cvYX79CyzLeJBVCS0RQojnn4xQiGpLUm9X37/OQK6BxC/xS+ptIYQQQjxzpEPxnKuMWiDR0dH4+voCeam1Z86ciYODA46OjgwZMoRjx44B2vVBCnP27FlMTU3Ztm2b1nJXV1emTJmitSw8PJzw8HB1fZ8+fXBycsLBwYEBAwawc+fOUsUihBCifMgzFM+5yqoFkm/lypXk5uYSGxuLRqPh999/x9PTs0SviT5cD8TGxkZr3bZt27Czs8PKyqrQfWfPnq0mGTt+/Dhjxoxh9erVtG3btswxCSGEeHJVfoSisLoaUguk7LVA8qWkpJCVlUVWVt7DjF27dmXOnDmPzfWRnZ1NTEwMXl5enDx5ksuXL2utHzt2LDNnzlSvUXFef/117OzsWLduXYnaLIQQovxV6Q5FYfVAlixZIrVAKHstkHzDhg3j6NGjvPXWW4wdO5ZVq1bRpUsXrSykhdmxYwfGxsa0bt0aKyurAmm1u3Xrhq2tLbNmzSpRO1555RXOnz9fom2FEEKUvyrdoSisrsaiRYukFsgjSlsLBKB58+Zs2rSJb7/9lk6dOqn7F3c+KFgP5KeffiIzM1NrG29vb44fP14gwVZhNBqN1PMQQohKVKWfoShs2P3Rt2SlFkjZaoF89tlnDBkyhI4dO9KxY0c8PDxwcXFhz549NGjQoNB9bty4QUJCAidOnGDVqlUoisKdO3fYvn272skA0NfXZ86cOXh5eWFjY4ORkVGR7Thz5gwmJiYlbrcQQojyVaVHKAqrqzFt2jSpBUL51QJJSkpi8eLF6ujCrVu3SE1NpV27dkXuExMTg7m5OQkJCcTFxREfH4+Hh0eh9UDypz4enRJ52LFjx9i2bZtaSEwIIcTTV6VHKAqrqzFs2DBq1aoltUDKqRbIjBkzmDt3Lra2tujr61OjRg2mTJmCiYkJKSkp/Pbbb1pVSB0dHTly5Ihapjzf4MGDWb58ufrw6MO8vb0LvBY6ffp0DAwM0Gg06OvrExoaSvPmzZ+o7UIIIcqPZMp8SqQWyLMnNTW92mbKbNjQkBs37lZ2MypVdb8GEr/E/6Tx6+hoqF//hSLXV+kRiqdNaoE8X4r7D6M6KC6FbnVR3a+BxC/xlycZoRBCCCFEmVXphzKFEEII8XRIh0IIIYQQZSYdCiGEEEKUmXQohBBCCFFm0qEQQgghRJlJh0IIIYQQZSYdCiGEEEKUmXQohBBCCFFm0qEQVU5sbCz29vb06dOH1atXF1ifXyvFxsYGf39/tUjaP//8w5AhQ7C1tWXs2LGkp6c/7aaXi9LGv2HDBnr27ImTkxNOTk6EhoY+7aaXi8fFn2/atGlER0erv1eV+w+lvwbV5TPwyy+/4OTkRL9+/fD09OT27dtA1fkMlDb+Mt9/RYgq5Pr164qFhYWSmpqqpKenK46Ojsq5c+e0tunbt69y5MgRRVEUxc/PT1m9erWiKIri7u6ubNq0SVEURVm0aJEyb968p9v4clCW+IOCgpTY2Nin3ubyVJL4r1+/rnz00UdKx44dlaioKHV5Vbj/ilK2a1AdPgNpaWnK22+/rVy/fl1RFEX5/PPPlVmzZimKUjU+A2WJv6z3X0YoRJWyd+9ezM3NqVevHgYGBtjY2LB161Z1/dWrV8nIyKBz585AXtn2rVu3kpWVxaFDh7CxsdFa/rwpbfwAx48fZ8OGDfTr14+pU6eqf7U8Tx4XP+T99fbee+9hZ2enLqsq9x9Kfw2genwGsrKyCAwMpGnTpgCYmppy7dq1KvMZKG38UPb7Lx0KUaUkJyfTuHFj9fcmTZqQlJRU5PrGjRuTlJREamoqhoaG6OnpaS1/3pQ2/vx/T5gwgY0bN9KsWTOCgoKeXsPLyePiBxgzZgwDBw7UWlZV7j+U/hpA9fgM1K9fHysrKwAyMjJYtmwZVlZWVeYzUNr4oez3X6qNiipFKaTWnUajeez6x+33vCht/ACLFy9Wl40ZM0b9n8zzpLT3sarcfyhbLNXpM5CWloanpyevvvoq77//fqGdh+fxM1Da+KHs919GKESV0rRpU1JSUtTfk5OTadKkSZHr//33X5o0aUKDBg24e/cuOTk5WsufN6WNPy0tjZUrV6rLFUVR/1J7njwu/qJUlfsPpb8G1ekzkJyczODBg3n11VcJDg4Gqs5noLTxl8f9lw6FqFJ69OjBvn37uHnzJvfv32f79u307t1bXf/SSy9Rq1Ytfv/9dyDvqebevXtTo0YNunXrxpYtW7SWP29KG7+BgQHLly/n6NGjAHz//ff06dOnUmIoi8fFX5Sqcv+h9NegunwGcnJy8PDwwM7ODn9/f/Wv96ryGSht/OVy/0v9OKcQz6iYmBilb9++irW1tbJs2TJFURRlzJgxyrFjxxRFUZRTp04pzs7Oiq2treLt7a08ePBAURRFSUxXj5WFAAAEtklEQVRMVIYOHarY2dkpo0aNUm7dulVpMZRFaeM/dOiQ0r9/f8XW1lbx8PBQ7ty5U2kxlMXj4s83bdo0rTccqsr9V5TSX4Pq8BnYvn27YmpqqvTr10/9+fjjjxVFqTqfgdLGX9b7r1GUQiZchBBCCCGegEx5CCGEEKLMpEMhhBBCiDKTDoUQQgghykw6FEIIIYQoM+lQCCGEEKLMpEMhhBDFsLS05Pjx40/1nDt27OCLL754qucUoqykQyGEEM+Y48ePP5eFuUT19vzlVRVCiErw+uuvM2LECHbs2MHdu3fx8fFh69atnD17liZNmrB06VIMDAzo0KEDw4cP58CBA9y7dw9vb2+sra2BvFoJmzdvRldXl9atWzNjxgwaN26Mq6srRkZGnD9/Hnt7eyIjI8nJyaFOnTp89NFHBAYGcvHiRW7fvs0LL7zAggULaNOmDa6urnTu3JnDhw9z7do1unbtyty5c9HR0SE+Pp7PP/+c3NxcDAwMmDlzJq+++iqHDx9mwYIF3L9/H41Gw4QJE7CwsKjkqyuqAulQCCFECWRmZtK4cWNiY2NZtmwZ06dP5+eff6Zx48Z88MEH/Prrrzg6OpKTk4ORkRHR0dGcPn2aoUOH0q1bN+Lj49m1axfr16/HwMCA8PBwfH19WbFiBQB169ZV0z4rikJqaipeXl5s3bqVunXrsnbtWgACAgJYvXo1M2bMAODy5ctERERw79497OzsOHjwIG3btsXHx4eIiAjat2/P9u3bWbBgAQsXLsTPz48VK1bQvHlzkpKSGDRoEKamphgbG1fOhRVVhnQohBCihGxsbABo2bIl7dq1o2nTpgA0b95ca4pi6NChALz66qu0a9eOQ4cOkZCQwIABAzAwMABg2LBhLF26lMzMTAC6detW6DltbW1p0aIFERERXLp0iYMHD9KlSxd1vYWFBTo6OhgaGtKqVStu377N4cOHeeWVV2jfvj0A1tbWWFtbs3PnTv7991/GjRun7q/RaDhz5ox0KESZSYdCCCFKqEaNGoX++1G6urrqv3Nzc9HV1S1QVjo3N5fs7Gz19/yOxqPWrFnD2rVrGTJkCI6OjtSrV4/ExER1fe3atdV/azQatUrko2Xrz5w5Q05ODiYmJqxbt05dl5SURIMGDYoLW4gSkYcyhRCinG3YsAGAP//8kwsXLtC9e3d69uxJdHQ09+7dAyAiIoLu3btTs2bNAvvr6uqqnY3du3fz/vvvM3DgQFq3bk1cXJxaYrsonTp14u+//+bcuXMA/Prrr/j4+NC5c2cuXbrEoUOHADh16hQ2NjYkJyeXW+yi+pIRCiGEKGeHDx9m7dq15ObmEhoaipGRER988AHXrl1j4MCB5Obm0qpVKxYsWFDo/m+99RYTJkygRo0ajBo1ioCAAKKjo9HV1eW1117j7NmzxZ6/UaNGLFiwgGnTppGTk4OhoSGhoaE0aNCAsLAw5s2bx4MHD1AUhXnz5vHSSy9VxGUQ1YxUGxVCiHJkamrKvn37ZBpBVDsy5SGEEEKIMpMRCiGEEEKUmYxQCCGEEKLMpEMhhBBCiDKTDoUQQgghykw6FEIIIYQoM+lQCCGEEKLMpEMhhBBCiDL7f/NTQQWicBsGAAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot_feature_importance(feature_importance);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In nearly every real-world dataset, this is what the feature importance looks like: a handful of columns are very important, while most are not. The powerful aspect of this approach is that is _focuses our attention_ on which features we should investigate further and which ones we can safely ignore."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> Warning: The feature importance analysis above can be biased and has a tendency to inflate the importance of continuous features or categorical features with high cardinality (i.e. many unique categories). See the Beware Default Random Forest Importances article in the references for more information."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Drop uninformative features"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From the feature importance plot above, we can see there are only a handful of informative features - let's use this insight to make a simpler model by dropping uninformative columns from our data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"9"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"feature_importance_threshold = 0.03\n",
"cols_to_keep = feature_importance[\n",
" feature_importance['Importance'] > feature_importance_threshold\n",
"]['Column']\n",
"\n",
"len(cols_to_keep)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# create a copy of the data with selected columns and create new train / test set\n",
"X_keep = X.copy()[cols_to_keep]\n",
"X_train, X_valid = train_test_split(X_keep, test_size=0.2, random_state=42)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"