{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "##
[mlcourse.ai](https://mlcourse.ai) – Open Machine Learning Course \n", "#
Tutorial. Mlxtend.SFS: an easy way to select features\n", "###
Author: Anton Gilmanov, @wicker\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1. Intro" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"Feature engineering and feature selection are one of the most important elements of data analysis and machine learning at all\".
\n", "
\n", "Your could read such phrase in many articles or books, and it's truth. But why do we need to select features?\n", "
\n", "
\n", "\n", "### 1. \"Noisy\" features\n", "\n", "Whatever good data you have, always there are some useful features, that help you to solve the problem and some noisy features - unuseful in your prediction model. Such features are dangerous, because it can lead to overfit. Opposite, quality of your model on hold-out data can be improved by deleting it from the dataset." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Сomputation problem" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If dataset has hundreds or thousands features, fitting estimator, cross-validation or another\n", "computation can take a lot of time. Of course, we can use PCA to reduce dimension of data, but sometimes it's not available for current business-task. It's impossible to explain how business could change \"new PCA feature\" to reach their goals - it's called \"interpretation problem\". So, feature selection is useful in this case." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Feature engineering" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, we have good dataset. But we added many custom features ~~to beat Kaggle's baseline~~ and now we are searching how we can select, which features improve quality metric and which not. Hm... We can use L1 regularization, it will move some weights towards 0. But what if we could fit estimator with different subsets of new features, and add one if quality metric is increasing or remove one if quality metric is decreasing, and make sure that solution takes only a couple of lines of code?

**Mlxtend SequentialFeatureSelector is what we need! **" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

**Everytime we try to select the best features :)**

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Image on web](https://i.giphy.com/media/5yLgoceFO3BdJW1zvFu/giphy.webp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Well, the informal intro is coming to an end. It's time to understand some formalized theory." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2. Introduce to SequentialFeatureSelector" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please, install some libraries, if you haven't it in your system" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#!pip install pandas\n", "#!pip install mlxtend\n", "#!pip install scikit-learn\n", "#!pip install matplotlib" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import pandas as pd\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split, KFold, cross_val_score\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import accuracy_score\n", "from sklearn.feature_selection import RFE\n", "from mlxtend.feature_selection import SequentialFeatureSelector\n", "from mlxtend.plotting import plot_sequential_feature_selection as plot_sfs\n", "from sklearn.decomposition import PCA\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Mlxtend SequentialFeatureSelector is greedy search algorithm that is used to reduce an initial d-dimensional feature space to a k-dimensional feature subspace where k < d." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are 4 different flavors of SFAs available via the *SequentialFeatureSelector*:\n", "\n", "* Sequential Forward Selection (SFS)\n", "* Sequential Backward Selection (SBS)\n", "* Sequential Forward Floating Selection (SFFS)\n", "* Sequential Backward Floating Selection (SBFS)\n", "\n", "In \"forward\" algorithm we start with no features in our subset and add one feature on each iteration, that maximize quality metric. In the contrary, \"backward\" algorithm start with full subset of features and remove one feature on each iteration maximizing quality of our model.\n", "\n", "The floating variants, SFFS and SBFS, can be considered as extensions to the simpler SFS and SBS algorithms. The floating algorithms have an additional exclusion or inclusion step to remove features once they were included (or excluded), so that a larger number of feature subset combinations can be sampled.\n", "\n", "Lets take a look at each of them." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sequential Forward Selection (SFS)\n", "\n", "\n", "### Input: \n", "\n", "Y={y1,y2,...,yd}\n", "\n", "* The SFS algorithm takes the whole d-dimensional feature set as input.\n", "\n", "### Output: \n", "\n", "Xk={xj|j=1,2,...,k;xj∈Y}, where k=(0,1,2,...,d)\n", "\n", "* SFS returns a subset of features; the number of selected features k, where k0=∅, k=0\n", "\n", "* We initialize the algorithm with an empty set ∅ (\"null set\") so that k=0 (where k is the size of the subset).\n", "\n", "\n", "### Step 1 (Inclusion):\n", "\n", "x+ = arg max J(xk+x), where x∈Y−Xk\n", "\n", "Xk+1=Xk+x+\n", "\n", "k=k+1 \n", "\n", "#### Go to Step 1\n", "\n", "* In this step, we add an additional feature, x+, to our feature subset Xk.\n", "* x+ is the feature that maximizes our criterion function, that is, the feature that is associated with the best classifier performance if it is added to Xk.\n", "* We repeat this procedure until the termination criterion is satisfied.\n", "\n", "### Termination: \n", "k=p\n", "\n", "We add features from the feature subset Xk until the feature subset of size k contains the number of desired features p that we specified a priori." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "___" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sequential Backward Selection (SBS)\n", "\n", "### Input: \n", "\n", "The set of all features: Y={y1,y2,...,yd}\n", "\n", "* The SBS algorithm takes the whole feature set as input.\n", "\n", "### Output: \n", "\n", "Xk={xj|j=1,2,...,k;xj∈Y}, where k=(0,1,2,...,d)\n", "\n", "* SBS returns a subset of features; the number of selected features k, where k0=Y, k=d\n", "\n", "* We initialize the algorithm with the given feature set so that the k=d.\n", "\n", "\n", "### Step 1 (Exclusion):\n", "\n", "x-= arg max J(xk-x), where x∈Xk \n", "\n", "Xk-1=Xk-x-\n", "\n", "k=k-1 \n", "\n", "#### Go to Step 1\n", "\n", "* In this step, we remove a feature, x- from our feature subset Xk.\n", "* x- is the feature that maximizes our criterion function upon removal, that is, the feature that is associated with the best classifier performance if it is removed from Xk.\n", "* We repeat this procedure until the termination criterion is satisfied.\n", "\n", "### Termination: \n", "\n", "k=p\n", "\n", "We remove features from the feature subset Xk until the feature subset of size k contains the number of desired features p that we specified a priori" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sequential Forward Floating Selection (SFFS)\n", "\n", "### Input: \n", "\n", "The set of all features: Y={y1,y2,...,yd}\n", "\n", "* The SFFS algorithm takes the whole feature set as input\n", "\n", "### Output: \n", "\n", "Xk={xj|j=1,2,...,k;xj∈Y}, where k=(0,1,2,...,d)\n", "\n", "* The returned output of the algorithm is a subset of the feature space of a specified size.\n", "\n", "### Initialization: \n", "\n", "X0=∅, k=0\n", "\n", "* We initialize the algorithm with an empty set (\"null set\") so that the k = 0 (where k is the size of the subset) \n", "\n", "\n", "### Step 1 (Inclusion):\n", "\n", "x+= arg max J(xk+x), where x∈Y−Xk \n", "\n", "Xk+1=Xk+x+\n", "\n", "k=k+1 \n", "\n", "#### Go to Step 2\n", "\n", "In step 1, we include the feature from the feature space that leads to the best performance increase for our feature subset (assessed by the criterion function). Then, we go over to step 2.\n", "\n", "### Step 2 (Conditional Exclusion):\n", "\n", "x-= arg max J(xk-x), where x∈Xk\n", "\n", "if J(xk - x) > J(xk):\n", "\n", "   Xk-1=Xk-x- \n", "\n", "   k=k-1 \n", " \n", "#### Go to Step 1\n", "\n", "In step 2, we only remove a feature if the resulting subset would gain an increase in performance. If k=2 or an improvement cannot be made (i.e., such feature x- cannot be found), go back to step 1; else, repeat this step.\n", "\n", "Steps 1 and 2 are repeated until the Termination criterion is reached.\n", "\n", "### Termination: \n", "\n", "k=p\n", "\n", "We add features from the feature subset Xk until the feature subset of size k contains the number of desired features p that we specified a priori." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sequential Backward Floating Selection (SBFS)\n", "\n", "### Input: \n", "\n", "The set of all features: Y={y1,y2,...,yd}\n", "\n", "* The SBFS algorithm takes the whole feature set as input.\n", "\n", "### Output: \n", "\n", "Xk={xj|j=1,2,...,k;xj∈Y}, where k=(0,1,2,...,d)\n", "\n", "* SBFS returns a subset of features; the number of selected features k, where k0=Y, k=d\n", "\n", "* We initialize the algorithm with the given feature set so that the k=d.\n", "\n", "\n", "### Step 1 (Exclusion):\n", "\n", "x-= arg max J(xk-x), where x∈Xk \n", "\n", "Xk-1=Xk-x-\n", "\n", "k=k-1 \n", "\n", "#### Go to Step 2\n", "\n", "* In this step, we remove a feature, x- from our feature subset Xk.\n", "* x- is the feature that maximizes our criterion function upon removal, that is, the feature that is associated with the best classifier performance if it is removed from Xk.\n", "\n", "### Step 2 (Conditional Inclusion):\n", "\n", "x+= arg max J(xk+x), where x∈Y−Xk\n", "\n", "if J(xk + x+) > J(xk):\n", "\n", "   Xk+1=Xk+x+ \n", "\n", "   k=k+1 \n", " \n", "#### Go to Step 1\n", "\n", "In Step 2, we search for features that improve the classifier performance if they are added back to the feature subset. If such features exist, we add the feature x+ for which the performance improvement is maximized. If k=2 or an improvement cannot be made (i.e., such feature x+ cannot be found), go back to step 1; else, repeat this step.\n", "\n", "### Termination: \n", "\n", "k=p\n", "\n", "We add features from the feature subset Xk until the feature subset of size k contains the number of desired features p that we specified a priori." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3. SequentialFeatureSelector object" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets take a look at documentation and parameters of SFS object" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**SequentialFeatureSelector**(estimator, k_features=1, forward=True, floating=False, verbose=0, scoring=None, cv=5, n_jobs=1, pre_dispatch='2n_jobs', clone_estimator=True)\n", "\n", "Sequential Feature Selection for Classification and Regression.\n", "\n", "_**Parameters**_\n", "\n", "**estimator** : scikit-learn classifier or regressor\n", "\n", "**k_features** : int or tuple or str (default: 1)\n", "\n", "Number of features to select, where k_features < the full feature set. A tuple containing a min and max value can be provided, and the SFS will consider return any feature combination between min and max that scored highest in cross-validtion. For example, the tuple (1, 4) will return any combination from 1 up to 4 features instead of a fixed number of features k. A string argument \"best\" or \"parsimonious\". If \"best\" is provided, the feature selector will return the feature subset with the best cross-validation performance. If \"parsimonious\" is provided as an argument, the smallest feature subset that is within one standard error of the cross-validation performance will be selected.\n", "\n", "**forward** : bool (default: True)\n", "\n", "Forward selection if True, backward selection otherwise\n", "\n", "**floating** : bool (default: False)\n", "\n", "Adds a conditional exclusion/inclusion if True.\n", "\n", "**verbose** : int (default: 0), level of verbosity to use in logging.\n", "\n", "If 0, no output, if 1 number of features in current set, if 2 detailed logging i ncluding timestamp and cv scores at step.\n", "\n", "**scoring** : str, callable, or None (default: None)\n", "\n", "If None (default), uses 'accuracy' for sklearn classifiers and 'r2' for sklearn regressors. If str, uses a sklearn scoring metric string identifier, for example {accuracy, f1, precision, recall, roc_auc} for classifiers, {'mean_absolute_error', 'mean_squared_error'/'neg_mean_squared_error', 'median_absolute_error', 'r2'} for regressors.\n", "\n", "**cv** : int (default: 5)\n", "\n", "Integer or iterable yielding train, test splits. If cv is an integer and estimator is a classifier (or y consists of integer class labels) stratified k-fold. Otherwise regular k-fold cross-validation is performed. No cross-validation if cv is None, False, or 0.\n", "\n", "**n_jobs** : int (default: 1)\n", "\n", "The number of CPUs to use for evaluating different feature subsets in parallel. -1 means 'all CPUs'.\n", "\n", "**pre_dispatch** : int, or string (default: '2*n_jobs')\n", "\n", "Controls the number of jobs that get dispatched during parallel execution if n_jobs > 1 or n_jobs=-1. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be: None, in which case all the jobs are immediately created and spawned. Use this for lightweight and fast-running jobs, to avoid delays due to on-demand spawning of the jobs An int, giving the exact number of total jobs that are spawned A string, giving an expression as a function of n_jobs, as in 2*n_jobs\n", "\n", "**clone_estimator** : bool (default: True)\n", "\n", "Clones estimator if True; works with the original estimator instance if False. Set to False if the estimator doesn't implement scikit-learn's set_params and get_params methods. In addition, it is required to set cv=0, and n_jobs=1." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Logistic Regression with feature selection by mlxtend.sfs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this article we will use toy sklearn dataset **\"breast_cancer\"** (binary classification task). Lets load the dataset and take a look at the data." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "data = load_breast_cancer()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "df, y = pd.DataFrame(data=data.data, columns = data.feature_names), data.target " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 569 entries, 0 to 568\n", "Data columns (total 30 columns):\n", "mean radius 569 non-null float64\n", "mean texture 569 non-null float64\n", "mean perimeter 569 non-null float64\n", "mean area 569 non-null float64\n", "mean smoothness 569 non-null float64\n", "mean compactness 569 non-null float64\n", "mean concavity 569 non-null float64\n", "mean concave points 569 non-null float64\n", "mean symmetry 569 non-null float64\n", "mean fractal dimension 569 non-null float64\n", "radius error 569 non-null float64\n", "texture error 569 non-null float64\n", "perimeter error 569 non-null float64\n", "area error 569 non-null float64\n", "smoothness error 569 non-null float64\n", "compactness error 569 non-null float64\n", "concavity error 569 non-null float64\n", "concave points error 569 non-null float64\n", "symmetry error 569 non-null float64\n", "fractal dimension error 569 non-null float64\n", "worst radius 569 non-null float64\n", "worst texture 569 non-null float64\n", "worst perimeter 569 non-null float64\n", "worst area 569 non-null float64\n", "worst smoothness 569 non-null float64\n", "worst compactness 569 non-null float64\n", "worst concavity 569 non-null float64\n", "worst concave points 569 non-null float64\n", "worst symmetry 569 non-null float64\n", "worst fractal dimension 569 non-null float64\n", "dtypes: float64(30)\n", "memory usage: 133.4 KB\n" ] } ], "source": [ "df.info()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": true }, "outputs": [], "source": [ "columns = df.columns" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
mean radiusmean texturemean perimetermean areamean smoothnessmean compactnessmean concavitymean concave pointsmean symmetrymean fractal dimension...worst radiusworst textureworst perimeterworst areaworst smoothnessworst compactnessworst concavityworst concave pointsworst symmetryworst fractal dimension
017.9910.38122.801001.00.118400.277600.30010.147100.24190.07871...25.3817.33184.602019.00.16220.66560.71190.26540.46010.11890
120.5717.77132.901326.00.084740.078640.08690.070170.18120.05667...24.9923.41158.801956.00.12380.18660.24160.18600.27500.08902
219.6921.25130.001203.00.109600.159900.19740.127900.20690.05999...23.5725.53152.501709.00.14440.42450.45040.24300.36130.08758
311.4220.3877.58386.10.142500.283900.24140.105200.25970.09744...14.9126.5098.87567.70.20980.86630.68690.25750.66380.17300
420.2914.34135.101297.00.100300.132800.19800.104300.18090.05883...22.5416.67152.201575.00.13740.20500.40000.16250.23640.07678
\n", "

5 rows × 30 columns

\n", "
" ], "text/plain": [ " mean radius mean texture mean perimeter mean area mean smoothness \\\n", "0 17.99 10.38 122.80 1001.0 0.11840 \n", "1 20.57 17.77 132.90 1326.0 0.08474 \n", "2 19.69 21.25 130.00 1203.0 0.10960 \n", "3 11.42 20.38 77.58 386.1 0.14250 \n", "4 20.29 14.34 135.10 1297.0 0.10030 \n", "\n", " mean compactness mean concavity mean concave points mean symmetry \\\n", "0 0.27760 0.3001 0.14710 0.2419 \n", "1 0.07864 0.0869 0.07017 0.1812 \n", "2 0.15990 0.1974 0.12790 0.2069 \n", "3 0.28390 0.2414 0.10520 0.2597 \n", "4 0.13280 0.1980 0.10430 0.1809 \n", "\n", " mean fractal dimension ... worst radius \\\n", "0 0.07871 ... 25.38 \n", "1 0.05667 ... 24.99 \n", "2 0.05999 ... 23.57 \n", "3 0.09744 ... 14.91 \n", "4 0.05883 ... 22.54 \n", "\n", " worst texture worst perimeter worst area worst smoothness \\\n", "0 17.33 184.60 2019.0 0.1622 \n", "1 23.41 158.80 1956.0 0.1238 \n", "2 25.53 152.50 1709.0 0.1444 \n", "3 26.50 98.87 567.7 0.2098 \n", "4 16.67 152.20 1575.0 0.1374 \n", "\n", " worst compactness worst concavity worst concave points worst symmetry \\\n", "0 0.6656 0.7119 0.2654 0.4601 \n", "1 0.1866 0.2416 0.1860 0.2750 \n", "2 0.4245 0.4504 0.2430 0.3613 \n", "3 0.8663 0.6869 0.2575 0.6638 \n", "4 0.2050 0.4000 0.1625 0.2364 \n", "\n", " worst fractal dimension \n", "0 0.11890 \n", "1 0.08902 \n", "2 0.08758 \n", "3 0.17300 \n", "4 0.07678 \n", "\n", "[5 rows x 30 columns]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.62741652021089633" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sum(y) / len(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are 30 float not-null features and 569 examples. 62,7% of examples have class 1 and 37,3% of examples have class 0. Classes are not very skewed, so accuracy metric is suitable for us." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will use LogisticRegression as our base algorithm. Firstly, scaling the data. " ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": true }, "outputs": [], "source": [ "scaler = StandardScaler()\n", "df_scaled = scaler.fit_transform(df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then initiating cross-validation object with 5 folds with fixed random_state " ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true }, "outputs": [], "source": [ "cv = KFold(n_splits=5, shuffle=True, random_state=17)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Checking cv results without parameters tuning" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.98417947523676441" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "logit = LogisticRegression()\n", "cross_val_score(logit, df_scaled, y, cv = cv).mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "CV result is **0.984**\n", "It will be our baseline. Trying to beat it with feature selection\n", "\n", "We will use Sequential Backward Selection, so toggle **forward** and **floating** parameters to **False**\n", "\n", "Sequential Backward Selection means that:\n", "\n", "* We will start with all features K (in our dataset K=30)\n", "* On each iteration n we fit estimator with K-n features and keep on K-n subset of features with best scoring\n", "\n", "Setting parameter k_features to tuple (1, K), so it will be subset of features in range (1, 30) with best scoring on CV as output of fit_transform method." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": true }, "outputs": [], "source": [ "logit = LogisticRegression()\n", "sbs = SequentialFeatureSelector(logit, \n", " k_features=(1, 30), \n", " forward=False, \n", " floating=False, \n", " verbose=2,\n", " scoring='accuracy',\n", " cv=cv)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is information about CV scoring on each iteration in log. The best quality we have with subset with 15 and from 17 to 24 features." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 30 out of 30 | elapsed: 1.3s finished\n", "\n", "[2018-12-12 00:08:17] Features: 29/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 29 out of 29 | elapsed: 1.2s finished\n", "\n", "[2018-12-12 00:08:19] Features: 28/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 28 out of 28 | elapsed: 1.1s finished\n", "\n", "[2018-12-12 00:08:20] Features: 27/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 27 out of 27 | elapsed: 1.1s finished\n", "\n", "[2018-12-12 00:08:21] Features: 26/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 26 out of 26 | elapsed: 1.0s finished\n", "\n", "[2018-12-12 00:08:22] Features: 25/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 25 out of 25 | elapsed: 0.9s finished\n", "\n", "[2018-12-12 00:08:23] Features: 24/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.8s finished\n", "\n", "[2018-12-12 00:08:24] Features: 23/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.8s finished\n", "\n", "[2018-12-12 00:08:25] Features: 22/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 22 out of 22 | elapsed: 0.7s finished\n", "\n", "[2018-12-12 00:08:26] Features: 21/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 21 out of 21 | elapsed: 0.7s finished\n", "\n", "[2018-12-12 00:08:26] Features: 20/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.6s finished\n", "\n", "[2018-12-12 00:08:27] Features: 19/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:28] Features: 18/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 18 out of 18 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:28] Features: 17/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 17 out of 17 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:29] Features: 16/1 -- score: 0.985949386741[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:29] Features: 15/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 15 out of 15 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:30] Features: 14/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 14 out of 14 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:30] Features: 13/1 -- score: 0.982425089272[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 13 out of 13 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:31] Features: 12/1 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 12 out of 12 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:31] Features: 11/1 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 11 out of 11 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:31] Features: 10/1 -- score: 0.982440614811[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:32] Features: 9/1 -- score: 0.978916317342[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 9 out of 9 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:08:32] Features: 8/1 -- score: 0.978900791803[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 8 out of 8 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:08:32] Features: 7/1 -- score: 0.980639652228[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 7 out of 7 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:08:32] Features: 6/1 -- score: 0.978885266263[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:32] Features: 5/1 -- score: 0.977146405838[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:32] Features: 4/1 -- score: 0.975376494333[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 4 out of 4 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:32] Features: 3/1 -- score: 0.963080267039[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:33] Features: 2/1 -- score: 0.954261760596[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:33] Features: 1/1 -- score: 0.920835274026" ] } ], "source": [ "X_sbs = sbs.fit_transform(df_scaled, y, custom_feature_names=columns)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Plotting results:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcJHV9+P/Xu+/uuXdnWfZkORZ0VYK6osYgqFFBDSiIUVFBTYgR48EPEzcYRFRQUb8k0WgIEkWJRgnyXRU5RJT49WIRQe7d5dh7d/aYs6e7q6vevz+qZrenp7q7ZnZ6zvfz8ejHdFd/+lOf7uqpd9fnFFXFGGOMqSc23QUwxhgz81mwMMYY05AFC2OMMQ1ZsDDGGNOQBQtjjDENWbAwxhjTkAULY4wxDVmwMMYY05AFC2OMMQ0lmpWxiFwPvAHYo6rPDXlegH8GXgfkgQtU9ffBc+cDHw+SflpVv9lof93d3bpq1apJKr0xxswP9913315VXdQoXdOCBfAN4MvADTWePwNYHdxeDHwVeLGILAA+AawFFLhPRNar6oF6O1u1ahUbNmyYpKIbY8z8ICLPREnXtGooVb0H2F8nyVnADer7DdApIkuA1wJ3qur+IEDcCZzerHIaY4xpbDrbLJYBWysebwu21dpujDFmmkxnsJCQbVpn+9gMRC4UkQ0isqGnp2dSC2eMMeaQ6QwW24AVFY+XAzvqbB9DVa9V1bWqunbRoobtM8YYYyZoOoPFeuBd4nsJ0KeqO4HbgdeISJeIdAGvCbYZY4yZJs3sOvsd4DSgW0S24fdwSgKo6teAW/G7zW7C7zr77uC5/SLyKeDeIKsrVLVeQ7kxxpgma1qwUNW3NXhegYtqPHc9cH0zymWMMWb8bAS3McaYhpo5KM+YeUNVGS659OWLOK5HV0uGlkyCmIR17ht/noPD5Uks7dwkMehqSdOWTZJKxKe7OHOOBQtjJkhVyZfK9A2V2NM/TMlxicdiiAi7DwwTjwvdbRk6W9O0pJPEY40DR3WeTtkjJkIq4edravNUOTBYRIH2TJLu9gxt2RTppAWOyWDBwphx8FQZLpY5MFRkb3/BP5nHhFw6QUs6OSqt6yn7Bovs7hsmJkJ3e4aukMChquSLZXqHivQMFCg5HvEaeZr6sin/lFZ0XJ7eM4AqtGSTdLdlaM+lyFjgmDALFmZSuZ7H7r5hyq7H8oWth1UNM5MMFhw/QPQVcDyPxMjJPFP7ZB6PCa3B856n7B8ssrt3mHhMWNCWpqslzUDBYW9fgbLrEY8L2ZQFiMmQTsYPXlEUHZctPQMokEsl6O7IkEvNjs84Jv73KB6PkYjJtF5dWrAwk0KDKoAtewcpux6eQqnscfQRbcRjs7cfxVDRYfu+IQ4MFUnG4+TScVpj4z/RxKoCR99Qib39BeKxGLl0nPgE8jTRVAaOUtlla8+gP1XELPgdo1VzVyTjMZKJGKl4nFQyRioRI5WIk0zEaM+mmloWCxbmsA0VHbbsHaQ/79CWSR78td07WGSjqxx7ZDvJ+OwKGIVSmR0H8uztL5BOxlnQmpm0vGMxqXtFYponlYjP6sZv11M8Tyk4ZfIl/0re9fyI8sJjFzX1St6ChZmwUtll54E8u3vzpJMJFrSmRz3f0ZKmP1/iiR29rF7SMSv+SQ+9p2GSiRidLSlrWDYzRjwmxGNCsmrUQ+9Qsen7tmBhxs1TZW9/ga17BwE/KNT6RdOeSzFYcHhsey/HL+2csQ2MjuvR0zfM9v1DxEToaEnNmfYWYyaDBQszLv3DJZ7pGWC46NKeS0Zqj2jNJMkXyzy6bT/HL+2cUQ24ruext7/A9v1DeJ7Snk0Ri9DF1Zj5xoKFqatUdik4LsOlMr1DJXqHirSkk3RVVTk1kksnKDouj2w9wLOWddGWnZ6AoaoUHf89DRYcv/ur69GWjRb4jJmvLFiYg0pll6LjkS85DAw7DA47OK4H6jfKJhMxulrSE67DTyfjiMCj2w6wekk7XZPYaBxGVXFcj0LJZajo0B+8J1cVAeKxGJlU3BqbjYnAgsU8ly+W2XVgiP6RwABIMGI4k0rQMslVMqlEHMkKT+zs49jFSnd7NjSdqlL2lLLr4ZQ9HNejHJSvEUUZLJQZyDuUPQ/Bf0/pZJzWbNLaIoyZAAsW89jAsMNj2w+QjAe/sKeor38yHqMjl2Lz7n5KrkdLOolTdsmXyhRLHgWnTLHs4umhmS4VxnWSTyZiZNOJSFNsGGMas2AxTx0YLLBxZz8tmcS0dGmNx2J05tJs3zcE+L/84zEhEYxWbU/GrcuqMTOIBYt5aO9Agc27+mjLpqZ1sFwsJnS2jK+h3BgzPaz7xzyzuzfP5p19dOSmN1DMRetvTnLai9o4YVk7p72ojfU3H361XtQ8x7Pv+ZynmTi7spgnVJUd+4fYtn+Izpa0jSWYZOtvTvLxj2YpDPuf647twsc/6jfen3m2M6l5eh78xZsO5fnDHyS57B8apxtP2tmeJ0z8czfhRKtnqpql1q5dqxs2bJjuYsxInipbegbY01ewkclNMNAPr3ppG70Hxl6pLV3m8fN7B8adZ18vvPpl4Xmaxib6uc9WvUPFCc8NJSL3qeraRunsymKOcz3l6T397Bso2jxHE7D+5iRfuirDzh3CkqXKxesKnHm2Q38f/PT2JLf/KMkv70nglMI/1x3bha9/LcVrX++wfIXWzXP/PuGu2xPc9uMkv/llAsepdayUD3300FxA/3x1GghLOzrdeNLO9jx3bBcuX5fhta93eNFLXBLBma7WZ18tarrpznMq2ZXFHFZ2PZ7c3U9fvmQNyRNQXQ0EkEwqx6x2eXJjHMcRli7zeO0bHH54c5K9PWOvAhJJpRyc9J97YpmVqzzuuj1JsTg6z6OO9nhqcwzXFZav9Dj99Q633BSeZ/Wv5tNe1MaO7dGuaqKmne15ZrL+wMvhYWHBQo9Xn+HQ0anc8PX0qOOZySqfvnp41Ik47LiHpRtP2mbkWcmuLMyElcoum3b2MVxyLVBM0Jeuyoz6hwVwHGHT43HefWGJ177B4cSTXETgOc9za/6Dn/TCMnf8OMltP05y6/qxaw44jvDUkzH++qIip7/e4dnP9RCBZz0nPM+L1xVGvf7idYVI6caTdrbn+emrh3n16Q733J3gth8l+eHNKfL5sSfSwrDwiY9leeSPh7qP//eNqTHHPSzdeNJORp5fuiozrVcXdmUxBxVKZTbu7MNxddrmYJoLTljWjurYE4yI8vj2/jHbo1QdNCPP8aSbr3kWhuHEY9upVbWVyx16lM8TKd140k5Wng891U8q5LffVFxZWLCYYwaGHTbu7A1WYLMLx4lQhZ/8MMnF78/ieWP/+Q6n8XQ8VTFmcs3E6rLx5AnQ2qa88jUOp7/e4c9OLXPHTw4FyhUr4MorhfPOC31pTVGDhXW1mEP2DhR4dNsB0sm4BYoJ2rNbuOi9OT78vhxLl3uk06N/TNWqNonq4nUFMtnJzdNEE/WzH88xmso83/u+Aqe/3uGenyV4/3taeOGz2vn7D2XZsT2GqrBli3DhhXDjjTU+gMNkVxZzQOUYio5cyqbangBV+MH3klx5eZZiET50SYELLixx6/rJ75UyE3u6zBczvbosSlrHgd/9KsFFf5UjPzT2yveoo+Dpp6N/JlYNNU+4nsfTewbYN1Cou2LdTDPdJ8zK/R+xWOns8nj80QQvPLnMlV8c5uhjo81wa8x0qd3+Bd44vr7WG2oeGOnxlC+Wm742xGRqxmjnw9n/7l3C7l3CG99c4rPXDGMXZmY2WLJU2bF9bLBYubI5+7N/i1kqXyzzyLYDFMseHbOsa2xYl9SRroFT4QtXjt0/CL/7dcIChZk1wto2cjn4zGeasz/715gBXM9f2CdqlWDvUJGHt+4nLkLrLFzlbeeO2qOd774zQakY+vRh6e+DH3w/yfvOz7Grxv5rlcuYmejMsx0+ffUwS5d5iCgrVyrXXsu4e0NF1dRqKBE5HfhnIA5cp6qfrXr+KOB6YBGwH3iHqm4Lnvs88Hr8gHYn8CGdKw0sFUplf11qx/UHYqUTcVLJOOlEnEwqRioRJxGLkYjHSMSFA4NFnt4z4E8vnji8WD/V7QaOA9d9NU2toygCf3N+C23tyqte63DGGxxe9vIyt/14Yg2I7/tQgVQSfvKjJL+6x58+Y8lSj5ZWGBocu/8lS+fc18vMcWee7XDm2U7FOIvm7atpwUJE4sBXgFcD24B7RWS9qj5SkewLwA2q+k0ReSVwFfBOEflT4GXAiUG6XwKnAj9vVnmng6fK03sG8BQ6W9KoKq6nOGV/3ej9gx6e55/AFBAET5WOlvRhrwA31e0GjzwU4x8vzvHIQ3FOfH6ZJx6NUyiMHnV7+VXDLFio3PbDJD+9Pckt30+RzihlB1z3UDn/8f/LsuWpGKe8onzw9f97d4KvfTlNqXgo3WV/nwWEZcs93vmeEmf8hcOJz3f54Q/Cp1Ow7qvG1NbMK4uTgU2q+iSAiHwXOAuoDBZrgI8E9+8GbgnuK5ABUvhDGZPA7iaWdVrs7s3TO1Q82DgtIiTiQjMXrlOFh/8Y4/J14e0GX7xycqcUKJXgq9ek+fcvp+nsUv71uiFe+7py3aua015VplQa5te/TPDBC3MHA8XBPIvCv3wxw798sdHehe4jPH722wEqO4mN7Me6rxoTXTODxTJga8XjbcCLq9I8AJyDX1X1JqBNRBaq6q9F5G5gJ36w+LKqPtrEsk65wYLDlr2DdOYmt3E67CT8hjc6PHh/nNt+lOSOW5Ns2xrDj8dj7dwhXPz+LK99vcMpryiTy028f/jCRUoiDrt2xjjrzSX+8fICXQv8/Y5cPteSSsGpryxTGK6VQvmPb+cPPvrrd+QImyJhX48Q1pu40f6NMaM1M1iET24y2iXAl0XkAuAeYDtQFpHjgGcDy4N0d4rIy1X1nlE7ELkQuBBgZbP6izWB43ps2tVHSzo5qYsQhVUt/cOHslxxaYb+vhjJpPKyU8t84OIC13w+w66dY/edy8H/uyfBj25Jkckox53g8vgj8YPTZderrqre/949AijveV+Bj102sVbrWt0Dly5TTn1ledTjsHTWDmHM5GhmsNgGrKh4vBzYUZlAVXcAZwOISCtwjqr2BUHgN6o6GDz3E+Al+AGl8vXXAteCPyivSe9jUqkqz/QM4LpKS3py65u+GNIl1XWFUhGu/tc8r/hzh/YOf3siSWi9/RWfH+Z1Zzps+G2c23+c5Ds3pMbMj1QYFtZ9JMs3rk2hKqj61Vsbn4gdnI77EOG2H6YmHCyaMaOpMWb8mhks7gVWi8jR+FcMbwXeXplARLqB/arqAevwe0YBbAH+WkSuwr9CORW4pollnTJ7+grsGyiyoHV81U+1qoJKRfjVLxPc8eMkO0N+WQMUi3DWOaOvAhrV27/kZS4veZnLf31z7JTa4PdsWrhIEfHXDRCBRx8O7511OF1So7YvTHc7hKpSdFwKjus3siXiZFLxwxpRr6oUHJdCyW1qL5e5QvHb/XKpxGH3FDRjNXW6DxF5Hf5JPg5cr6qfEZErgA2qul5E3ozfA0rxrxouUtVi0JPq34CXB8/dpqoX19vXbJjuY6jo8PCWA7TnUuPqzRS6CE9Ked6JLk88HmdwQGhtU9yyv9hLtamaJXW+zahaGSBQaG9J0t2WJRGPsW+gwIHBIp6nJBMxsqlEpCrHkQBRdFwAOnIputsyZG1iyIY8VQaGS+zpK/gBNjZ/AsesX/xIVW8Fbq3adlnF/ZuAm0Je5wJ/08yyTbWy6/Hkrn6yqfi4u72GLsJTEu7/fZxz3uLw6tc5vOwUfzzCZFfFNGPBmtksLEAsXdBCWzZJqqIbW0cuhbtIGSo47BsssH+giKdKIu4HjsrvgKdKoeRSclwQ6GpJs7K7lZZMkmR87p/oJlNLOsnijhwFx6UvX2RPX4HBQWdeBY5msZ8rU2TbvkGKjjuuqTkGB+DuO5OhDbcAKFz5pUPdhZpRFTOePKe7KqiZio7LcKlcN0BUi8eE9lyK9lyKld3KUNHhwGCRfQMFXFeJxfxxMxIEiIWLWmnNJElYgDgsIkI2lSCbStQMHKlZEjRSidiMmUXaZp2dAnsHCmze1UdXSxqpuEwMa4c47VUOP7szyW0/TPK/v0jglIRYTCd9ER4TjarSny+RTsVZ2tU4QEThqTJUKDNQKNGSTtKaScyYE8JcNlLF15cvMlgoN37BDNCfL+F5SlsuVbeKadZXQxkYLpV5anc/7dnUmEBR3c317z+YBfzV2Y5c4nHe+SVOf4PD1i0x/unv53b1zkzkeh59+RKLO7Ks6G6dtBN6TIS2bNKWvJ1ilVccs4XjeuzuzbPjQJ5kPEZLOjHqPDKVZs+nNgu5nsfmXf3+/E5VVQth7RCeJ7S0Kv/5nUFOfL57cAbUF7zIRWRuVu/MVEXHJV8sc8zidrrbMtP2D2rmt2Q8xvKFrXS3Zdi2f4h9AwVyqQSZaQh4FiyaaPv+IYZLZTpD2ilqdSfND8FJL3THbLcRx1OnP18iHhees6KLllk4q6+ZezKpBMcd2cHijhzP9AzQO1ikJTu1HSCsorRJevqH2XkgT0cufJxCrZHF823EcX++hFOeGavSeZ5yYLBAezbJc1YssEBhZpy2bJI1K7o45sh2So5L31Dx4GSjzWbBogl29w7z5K5+OnLpmtUXF3+sgEi0BdznqqLjkkzEyJcc8sXpbXB0yh69+SLLu1s5dkmHdVk1M1ZMhIVtGZ531AKWLWyhf7hUa6q3SWXVUJNsV2+eZ3oGGk4jfsSRHqpCZ6dHX9/8bIfIFx1WL+kgm0qwaVc/fUNF2nOpKW8fyBfLlF2PZy3tnHWrDpr5Kx6LsaSrhQWtGfryxcOaLSAKCxaTaOeBIbb0DNLZkm44Wveb16XpWuDxi3sHyGSj78NTbfqXYiqUXY9UIk57zg+qz17eyTM9g/T0F+gY5wj3iVJV+vIlWtIJTli2gEyyiXPDG9Mk6WScIzpyTd+PBYtJoKrs7M2zdW+0QLHl6Rg/uyPB+z5YjBwoXM+jb6hEIi6MVFHGxB9clErEJ3X22qkwWHBY2d16MCjEYzGOPqKNlnSCZ3oGaMkc/niGWjxPGSw4eJ5yRGeW5QtbpyQ4GTObWbA4TKrK9v1DbN83RGdrOtKv/m9dnyIeh7efX4q0j4PdOI/0u3E6rr+SXr5UZiDvMFDwB+4ofgBJBgFkpp4APc8ftbygLTNqu4iwuDNHLp1k485eSmVvUtcYL7segwUHEVjcmWNRe9auJoyJyILFYVBVtu0bYseBfORAMTgAN303xevOdFh8ZONWqf68fzVR2Y0zlYgHVTgpjuz0y+G4HoUgqAzkHQYLDo7nIfiXqenk4c2AOpkGCw6LO3M1G5Hbgt5Im3f30TtUqNtRIIpS2WWoUCYZj7Gyu5UFbRlrwDZmnCxYTJCqsnXfILsODNPZUn8ofqX/+e8UQ4PCu/6q/lWF5yl9+SIL2jIctait7slNRA4FkOyhAFJ0XAYLDr35En1DpWAeIiGTjJNKxKZloJmq4nnKovb69W/pZJwTlnaxbd8guw7k6WhJjXsE9XCpTKFUJptOcOyR7XROwtrlxsxXFiwmwFNlS88Ae/qG6WyJ/qvXdeGGr6d4wdoyJ540duDdiFLZP8kftaiNxR3ZCZ3URYRMMNKzuz2Lp8pwscxgwZ/Mri/vd7eLx2Pk0vEpm5soXyyzoD0TqfonHhOOWuS3Yzy5Z4BELBa6RGo1z1NcT2nPJVm1qIu2bNJGYBtzmCxYjNNIoNjdNzxmYsBGfv7TBFufiXPJPw7VTDNYcABlzfIFkzp3UEyElkySlkySxZ05yq4XtHmU2HEgTyYZn5IpBEpllyM7xtH9C+huz5JNJzgwGG21vZgIHS0pWtI2qM6YyWLBYpy27h1kT19h3IEC/O6yS5Z6vPqMsQPQPFX6hkp05FIcvbitaT2BRiTiMdqzKdqzKbpaMzyxs5fBgjOpDcrVCqUybdnUhEZGt6STdvI3ZhpZK984eKrs7S/Q0TL+gWOPPRLjN/8vwTveXSRRFaKdskffUJFlC3KsXtrR9EBRLZdOsGZ5F+lEjL6hia2VHcVwyWVJV/P7gxtjJp8Fi3EolNwJD4q74bo02axy7tvHjtAeKJQ4fmknyxa2TluPpVQizgnLOulqTXNgsIA3yeucOGWPVDJGe425sowxM5sFi3EYLk1s/qL9+4T1P0jyxnNLdHaNPgkXHZe2bCp0ZtqpFo/FOGZxO0sXtNA7WMT1Jm+Cv6Giw7IFLTOm+64xZnwsWIxD/3BpQlVE3/1WilJReOd7x3aXLZTKdFcNTptOIsLyha0ce2Q7ffkSpXLtXltRuZ5/NdY1AwKiMWZirIF7HPqGSmRS4wsWpRLc+I0Up5zmcNzqsb/UPYX2GbhiWnd7llQyzsYdfbieHtbqYoMFh6VdOVtb2phZzP57IyqVXcquN+7xCD/5YZKePTHO/+uxVxWO65FJTU2X1Yloz6Z4zoouAAaGo01NUs1TRVXpbp85V0/GmPGzYBHRcMllvG2+qvDN61Icc5zLn506tr1juFie8SfRTCrBs5Z1kksn6R0qoOP8EPLBe5zqHl7GmMllwSKiweES8XFWo/z+3jgPPZDgXe8tEXZB4nlKR3bm9w5KJeKsXtLBER05DgwVx9XQ75Q9Fk/B9MnGmOayYBFRX94hnRzfx/XN69K0dyhvPHdsFY7rKYl4jGx6ZlZBVRuZeuO5KxeQTMTYP1hsuBzqcKlMey5Jbpa8R2NMbRYsInA9j6GiE3mm0vU3JznlBW3c9qMEngs/vW1sA/ZwqczCtmgz1c4kLekkz1rayfFL2im5Lr1DRdwaawAXSi5Lu1qmuITGmGawn3wRFEp+99Eoo7bX35zk4x/NUhj20w4Owsc/6s+FVLlkarnszYixFRMhInS1ZmjPpdjTN8y2fUPEYzFaM4mDn1Gp7JJNxSd1fitjzPSxK4sIhorlyNN7fOmqzMFAMaIwLHzpqkMN2Z4qEhNaMrM7Vo+sAXziUQvpbEnRO1SkELRn5Itlli5osdlejZkjLFhE0DdUIpWI9lHt3BF+cqzcXii5dE1gfYaZKp2Mc8zidp69fAEiwv6BAvGYzNorJ2PMWLP7p+0UUFX6CyVaI8542r1I6dkzNmAsWXqoXr9UdlnY1jppZZwp2rJJ1qzoYv9AARGxhYaMmUOa+tNWRE4XkcdFZJOIfCzk+aNE5C4ReVBEfi4iyyueWykid4jIoyLyiIisamZZayk6rj9dRYQTn+dBJqvA6AbfTFa5eF0B8IMPyoSm6Z4NYiJ0t2dZOIOmMDHGHL7IwUJEsiJywjjSx4GvAGcAa4C3iciaqmRfAG5Q1ROBK4CrKp67AbhaVZ8NnAzsibrvyTRccon6+/j730my9Zk4b31niaXLPESUpcs8Pn318MHG7VLZoy2btDWgjTGzSqRqKBH5C/wTewo4WkROAq5Q1TPrvOxkYJOqPhnk8V3gLOCRijRrgI8E9+8GbgnSrgESqnongKoORn5Hk2xguBTpxL5/n/DFKzO86CVlPvnZAiKF0HTDpTKrjmib7GIaY0xTRf15ezn+yb8XQFX/AKxq8JplwNaKx9uCbZUeAM4J7r8JaBORhcDxQK+I3Cwi94vI1cGVypTrzZdIRVgv+gtXZhgcED5x5XDDdaLbZsGobWOMqRQ1WJRVtW+ceYedMqtHb10CnCoi9wOnAtuBMv4VzynB8y8CjgEuGLMDkQtFZIOIbOjp6Rln8RpzXI+i4za8svj9vXFu+k6KC/66xPHPqj2quVR2/YkDIwQfY4yZSaIGi4dE5O1AXERWi8i/Ar9q8JptwIqKx8uBHZUJVHWHqp6tqs8HLg229QWvvV9Vn1TVMn711Auqd6Cq16rqWlVdu2jRoohvJboocyCVy3D5uixHLvG46OLwqqcRhZLLohk+caAxxoSJGiz+DngOUAT+C+gDPtzgNfcCq0XkaBFJAW8F1lcmEJFuERkpwzrg+orXdonISAR4JaPbOqbEUMFpOB3Hjd9I8dgjcS69YpiWBjNbeKpWBWWMmZUiNXCrah7/l/+lUTNW1bKIfAC4HYgD16vqwyJyBbBBVdcDpwFXiYgC9wAXBa91ReQS4C7xhwDfB/xH9Lc1OfrzJdJ1qox27xKu+XyGU17h8JrX1b8KcT2PZDxGboauXWGMMfVE7Q11J3CuqvYGj7uA76rqa+u9TlVvBW6t2nZZxf2bgJtqvPZO4MQo5WsG11MGhv1ZU2v53BUZHAcu+3ShYaN2vuiyqCNj018YY2alqNVQ3SOBAkBVDwBHNKdIM0PBKQNa8+T+q/+N86NbUlx4UZGjjq4/VTdA2fXozNn0F8aY2SlqsPBEZOXIAxE5irE9m+aU4WLtaqVSET75j1lWrnK58KJiw7w8T4nHxNZ1MMbMWlHPXpcCvxSRXwSPXw5c2JwizQz9+RLJGkuBXn9tmqc2x/mPbw+RyTbOq+C4dLWmba4kY8ysFbWB+zYReQHwEvzxEx9R1b1NLdk0UlX6hx0yFY3R629O8qWrMuzcIajCc/+kzKmvjLa8aMlxWbBo7k0caIyZP8YzQVEa2I/fbXaNiLy8OUWafqWyh+N6B68ERhY02rE9hqoAwsbH46y/ufFkgKoKAq1zdOJAY8z8ELU31OeAvwQeBkZac0e6u8451YPxwhY0Khb8BY0qV78LU3Bc2nMpEjZxoDFmFovaZvFG4ARVbdyaOwcMFpxRJ/coCxrVUnRcli2wdaiNMbNb1J+7TwLzph6lt2plvMqFiyrV2j6KWhWUMWb2ixos8sAfROTfReRfRm7NLNh0Kbsew6UyqYqeUBevKwSLGh1SuaBRLaWySy6dqDsK3BhjZoOo1VDrqZrXaa4qOGMXOzrzbId9e+Gqy3OAsnSZHygatVfki2VWdlsvKGPM7Be16+w3m12QmWKo6BA2d8eqY/wri+/cMsQLT3Yj5aUK7TmbONAYM/tF7Q21Gn/J0zXAwTm2VfWYJpVr2vQNlUgnxtbOPbnR33bs6sZTe4BfnZVKxmztCmPMnBC1zeI/ga/iL0z0CvyqMspTAAAbHElEQVT1sb/VrEJNF0+VgWEndGW8zRvjLOz26OyKNstJvljmiPasTRxojJkTogaLrKreBYiqPqOql+OvMTGnFB0Xz9PQNSw2b4xFvqpQVVxPWdhmCx0ZY+aGqMGiECxStFFEPiAib2IOzjpbKLmhi8GqwuZNcY47PlpbxXDJZWFb2npBGWPmjKjB4sNADvgg8ELgncD5zSrUdOnLl0LX297bI/T3CcccF+3KouC4LO7MTXbxjDFm2kTtDXVvcHcQeHfzijO9+mqsjLf5YON24yuLouOSS8dpsenIjTFzSNTeUGvxpyk/qvI1qjptK9lNtlLZPTiIrtrmjX4AidJmMVwqc+zidmvYNsbMKVF//t4IfBT4I4cmEpxTCqXaVw2bN8ZoaVUWH1m/J5TrecRE6GixsRXGmLklarDoUdU5PYJ7sOjUXJxo08Y4x652G66zPVgos2xBjnjMZpg1xswtUYPFJ0TkOuAu4ODMs6p6c1NKNQ36hsLbK8AfkPeyU+svdKSqeKosaLXussaYuSdqsHg38Cz8mWcr17OYE8HC9ZTBgkNHyNQcA/2wZ3eMY4+r37idL5bpbstYd1ljzJwUNVj8iao+r6klmUYFx79qCGuUjtq4XSy7LO6IsCC3McbMQlEr138jImuaWpJp5NWJA5sjzAlVdFxa00labN0KY8wcFfXK4s+A80XkKfw2CwF0LnWdrWXzxjjJlLJ8Ze1gkS86rF7SMYWlMsaYqRU1WJze1FLMYJs3xjj6GI9EjU+q7Hok4jHac+mpLZgxxkyhhsEimBPqx6r63Ckoz4yzeVOM5zyvduP2UMFheXdLzW63xhgzFzRss1BVD3hARFZOQXlmlGIBtm2J1ZwTylNFFesua4yZ86JWQy0BHhaR3wFDIxtV9cymlGqGeOrJGJ4nNRu388Uy3R2ZUet1G2PMXBQ1WHyyqaWYoUa6zdaamtwpuxxh3WWNMfNApK6zqvoL4DGgLbg9GmyrS0ROF5HHRWSTiHws5PmjROQuEXlQRH4uIsurnm8Xke0i8uVob2dybd4YIxZTjj5m7JVFoVSmNZuiJW3dZY0xc1+kYCEibwF+B5wLvAX4rYi8ucFr4sBXgDPw1+5+W8hYjS8ANwRdcK/AX+e70qeAhkGpWTZvjLF8pUc6pEmiUHJZ0mlXFcaY+SFqNdSlwItUdQ+AiCwCfgrcVOc1JwObVPXJ4DXfBc4CHqlIswb4SHD/buCWkSdE5IXAYuA2YG3Eck6qJzfFOTakcbvseiQT1l3WGDN/RB3BHRsJFIF9EV67DNha8XhbsK3SA8A5wf03AW0isjDorvtF/GnRp0W57DdwhzVuDxYclnTlrLusMWbeiBosbhOR20XkAhG5APgxcGuD14SdSasXhLgEOFVE7gdOBbYDZeD9wK2qupU6RORCEdkgIht6enqivI/Itm2JUSrKmNXxPPXfQlerXVUYY+aPutVQIpJW1aKqflREzsaf9kOAa1X1Bw3y3gasqHi8HNhRmUBVdwBnB/tqBc5R1T4ReSlwioi8H2gFUiIyqKofq3r9tcC1AGvXrq2/MtE41ZoTaqhQ5oiOrHWXNcbMK43aLH4NvEBEvqWq72R8U5LfC6wWkaPxrxjeCry9MoGIdAP7g4F/64DrAVT1vIo0FwBrqwNFs9Vad9v1PLrbbRCeMWZ+aRQsUiJyPvCnwZXFKPUWP1LVsoh8ALgdiAPXq+rDInIFsCFYee804CoRUeAe4KIJvo9Jt3lTnCMWe7S1j30uk4zaL8AYY+aGRme99wHnAZ3AX1Q913DxI1W9laq2DVW9rOL+TdTvUYWqfgP4RoNyTrrNG2McU1UF5XpKMh6zhm1jzLxTN1io6i9F5FfANlX9zBSVadqp+qO33/jm0qjtrufZSnjGmHkp6kSCb5iCsswYu3cJQ4Nj54Qqu0o2ZVVQxpj5J2rX2TtE5BwJW3d0DqrVuF12PTIpu7Iwxsw/UX8mXwy0AK6IDHNopbyQ5t/Zr9a6254qaesya4yZhyIFC1Vta3ZBZpInN8Voa1cWHTF26EYyEfVizBhj5o6oEwmKiLxDRP4peLxCRE5ubtGmz+aNcY5b7RJW6ZaIW7Awxsw/Uc98/wa8lEOD6gbxZ5SdkzZvDJ8TCiBpwcIYMw9FbbN4saq+IJjDCVU9ICKpJpZr2vT1wt6e2Ng5oTwlEbMxFsaY+Snqz2QnWJ9C4eAU5eE/vWe5kcbt6gF5Zc96Qhlj5q+oweJfgB8AR4jIZ4BfAlc2rVTTaNMTtbrNqgULY8y8FbU31I0ich/wKvxus29U1UebWrJpsnlTnHRGWbZ8dE8o1/PIWrAwxsxTjaYoz+DPD3Uc8Efg31W1PBUFmy6bN8Y4+hiPeFVccD0lnbDR28aY+alRNdQ38Zc0/SP+WtpfaHqJptmTG+NjqqBGWLdZY8x81ein8hpVfR6AiHwd+F3zizR9hvOwfZtw9l/W6DZrA/KMMfNUo7OfM3Jnrlc/ATy1OYbq2KVURyTj1m3WGDM/Nbqy+BMR6Q/uC5ANHs/JuaFqzgnlKfGYEI/ZlYUxZn5qtJ7FvOr+s3lTjFhMWXV0yBgLWx3PGDOP2U/lCps3xlm5yiOVHr297CrZ9LyKm8YYM4oFiwqbN8Y49rixjduu55GxFfKMMfOYBYtAuQzPPBXj2OPHNm67nlo1lDFmXrNgEdi+JY7jjF1KFQC1MRbGmPnNzoCBpzaH94QCQKzbrDFmfrNgEXh6s1/NdMxxNcZY2IA8Y8w8ZmfAwFObEhy5xKO1dfR2T5WYjbEwxsxzdgYMPLU5fE6osuuRtcZtY8w8Z8EC8Dx45sl4aHuF3xPKPiZjzPxmZ0Fg+3YYzoevu+0vemRXFsaY+c2CBfD4Y35Pp7BqKNeWUzXGGAsWAI8/PhIswqcmtzEWxpj5zs6CwBOPQXunx4KFOuY5AVIWLIwx89y8PwveeCP8140x+nuFV5zcxvqbk6OeV+zKwhhjmnoWFJHTReRxEdkkIh8Lef4oEblLRB4UkZ+LyPJg+0ki8msReTh47i+bUb4bb4QLL4RiUQBhx/YYH/9o9mDAGBljYcHCGDPfNe0sKCJx4Cv4a3evAd4mImuqkn0BuEFVTwSuAK4KtueBd6nqc4DTgWtEpHOyy3jppZDPj95WGBa+dFUGANdVm23WGGNo7pXFycAmVX1SVUvAd4GzqtKsAe4K7t898ryqPqGqG4P7O4A9wKLJLuCWLeHbd+7wG7zLnkfWgoUxxjQ1WCwDtlY83hZsq/QAcE5w/01Am4gsrEwgIicDKWBz9Q5E5EIR2SAiG3p6esZdwJUrw7cvWeo3dNsYC2OM8TUzWIRN01rd3egS4FQRuR84FdgOlA9mILIE+BbwblUd069VVa9V1bWqunbRovFfeHzmM5DLjd6WySoXrysANsbCGGNGNPNn8zZgRcXj5cCOygRBFdPZACLSCpyjqn3B43bgx8DHVfU3zSjgeef5f9etU7Zt868oLl5X4MyznYNprHHbGGOaGyzuBVaLyNH4VwxvBd5emUBEuoH9wVXDOuD6YHsK+AF+4/f3m1hGzjsPzjy7zGPbD9DZMnrxbQGSFiyMMaZ51VCqWgY+ANwOPAp8T1UfFpErROTMINlpwOMi8gSwGPhMsP0twMuBC0TkD8HtpGaVtRbFgoUxxkBzryxQ1VuBW6u2XVZx/ybgppDXfRv4djPL1oiniohVQxljDNgI7pr8MRbWE8oYY8CCRU1l6wlljDEHWbCowfWUrI2xMMYYwIJFTWXXI52wj8cYY8CCRU0CpBJWDWWMMWDBoq6kXVkYYwxgwaImW8fCGGMOsbNhCB0ZYxELm97KGGPmHwsWIcqeP8ZCxIKFMcaABYtQZdfGWBhjTCULFiFcz1bIM8aYShYsQpRdz4KFMcZUsGARRm2MhTHGVLJgEcJmmzXGmNHsjBhCsQF5xhhTyc6IVWyMhTHGjGXBokrZU9KJuI2xMMaYChYsqriuZ4seGWNMFQsWVcqekknZx2KMMZXsrFil7Hq26JExxlSxYBEiaWMsjDFmFAsWIZI2xsIYY0axs2KIRNx6QhljTCULFhVGxljYlYUxxoxmZ8UKro2xMMaYUBYsKpRtjIUxxoSyYFGh7ClpG2NhjDFj2JmxguspORtjYYwxY1iwqKCqNsbCGGNCWLCoYt1mjTFmrKYGCxE5XUQeF5FNIvKxkOePEpG7RORBEfm5iCyveO58EdkY3M5vZjkP7hPrNmuMMWGadmYUkTjwFeAMYA3wNhFZU5XsC8ANqnoicAVwVfDaBcAngBcDJwOfEJGuZpUVAPUXPbIV8owxZqxmnhlPBjap6pOqWgK+C5xVlWYNcFdw/+6K518L3Kmq+1X1AHAncHoTy+qPsUjGidkYC2OMGaOZwWIZsLXi8bZgW6UHgHOC+28C2kRkYcTXIiIXisgGEdnQ09NzWIUtex7ppDVuG2NMmGYGi7Cf6Fr1+BLgVBG5HzgV2A6UI74WVb1WVdeq6tpFixYdVmFdT8mmLFgYY0yYZg4q2AasqHi8HNhRmUBVdwBnA4hIK3COqvaJyDbgtKrX/ryJZUUEG71tjDE1NPPK4l5gtYgcLSIp4K3A+soEItItIiNlWAdcH9y/HXiNiHQFDduvCbY1TVxipBLWuG2MMWGadnZU1TLwAfyT/KPA91T1YRG5QkTODJKdBjwuIk8Ai4HPBK/dD3wKP+DcC1wRbGuaeFxIWrAwxphQojqmKWBWWrt2rW7YsGFCrx0Ydnhs+wH+ZNVCUjaC2xgzj4jIfaq6tlE6+ykdSMRjNsbCGGNqsLNjIGNjLIwxpiYLFgEbY2GMMbVZsMCfPLAtk5zuYhhjzIxlAwuAbCpB1taxMMaYmuzKwhhjTEMWLIwxxjRkwcIYY0xDFiyMMcY0ZMHCGGNMQxYsjDHGNGTBwhhjTEMWLIwxxjQ0Z2adFZEe4JnDyKIb2DuJ6SxPy9PytDxnYp7VjlLVxkuNqqrd/IC5YTLTWZ6Wp+Vpec7EPCd6s2ooY4wxDVmwMMYY05AFi0OuneR0lqflaXlanjMxzwmZMw3cxhhjmseuLIwxxjTW7Bb0mX4Drgf2AA81SLcCuBt4FHgY+FCNdBngd8ADQbpPNsg3DtwP/KhBuqeBPwJ/oEHPB6ATuAl4LCjvS0PSnBDkNXLrBz5cJ8+PBO/nIeA7QKZGug8FaR6uzi/sswYWAHcCG4O/XTXSnRvk6QFrG+R5dfDeHwR+EHweYek+FaT5A3AHsLTRdwK4BFD8ropheV4ObK/4XF9XL0/g74DHg/f2+Rp5/ndFfk8Df6jz3k8CfjPyPQFOrpHuT4BfB9+pHwLt9b7nIcfpuTXSjTpOdfILO0a10lYfp7Vh6Woco1p5Vh+nd9XKM+QY1cqz+jg9XCNd2DGqleeY40SNcwxwNPDb4Bj9N5Ca1HPlZGY2G2/Ay4EX0DhYLAFeENxvA54A1oSkE6A1uJ8MDt5L6uR7MfBfRAsW3RHf0zeBvwrup4DOBunjwC78/tZhzy8DngKywePvAReEpHsufqDI4S+s9VNgdb3POvjn+1hw/2PA52qkezZ+gPs5o4NFWNrXAIng/ufq5Nlecf+DwNfqfSeCf+jb8cfzdNfI83LgkijfM+AVwWeUDh4f0ej7CHwRuKxOnncAZwT3Xxd8XmHp7gVODe6/B/hUve95yHH6co10o45TnfzCjlGttNXH6YawdDWOUa08Rx2nOunCjlHDc0FwnD5fI8+wY1Rr/2OOEzXOMfj/l28Ntn8N+Nso54uot3lfDaWq9wD7I6Tbqaq/D+4P4P8CWBaSTlV1MHiYDG6hDUMishx4PXDdxEofmmc7/snh60F5Sqra2+BlrwI2q2q9QY0JICsiCfxgsCMkzbOB36hqXlXLwC+AN408WeOzPgs/uBH8fWNYOlV9VFUfr95hjbR3BPsH/xfc8hrp+isethAcpzrfif8D/H2EdGPUSPu3wGdVtRik2VMvTxER4C34V3a18lT8X58AHcCOGulOAO4J7t8JnBPkWet7Xn2cXh2Wrvo41cqvxjGqlbb6OA3V+V+sPkZR/29rpQs7RnXzrDhO/1EjXdgxqpXnmONU5xzzSvwahZFj9Mbq93k45n2wmAgRWQU8Hz+ihz0fF5E/4F/636mqoemAa/C/2F6E3Spwh4jcJyIX1kl3DNAD/KeI3C8i14lIS4O830pwAgrdsep24AvAFmAn0Keqd4QkfQh4uYgsFJEc/q+mFQ32vVhVdwb72Yn/y20yvQf4Sa0nReQzIrIVOA+4rE66M4HtqvpAhH1+QEQeFJHrRaSrTrrjgVNE5Lci8gsReVGDfE8BdqvqxjppPgxcHbynLwDraqR7CDgzuH8uIcep6nte8zg1+n+IkG7MMapOW+s4VaZrdIxC9h96nKrS1T1GNd7TmONUla7uMapKG3qcqs8xwGagtyIAbyMkKB4OCxbjJCKtwP/g18f3h6VRVVdVTwKWAyeLyHND8nkDsEdV74u465ep6guAM4CLROTlNdIl8KscvqqqzweG8KsNar2fFP6X8ft10nTh/7I8GlgKtIjIO6rTqeqj+NUJdwK34deplqvTTRURuTTY/4210qjqpaq6IkjzgRr55IBLqRNMKnwVOBa/XnonfnVELQn8NpqXAB8Fvhf8Kq3lbdQJ6oG/BT4SvKePEFxhhngP/vfoPvxqj1Llk1G+55ORLuwYhaUNO06V6YI8ah6jkDxDj1NIuprHqM57H3WcQtLVPEYhaUOPU/U5Bv+qvtrkdnWdzDqt2XoDVtGgzUIP1Q/eDlw8jrw/QXgd9lX40f9p/PaCPPDtiHleHpZn8NyRwNMVj08Bflwnr7OAOxrs71zg6xWP3wX8W4RyXgm8v95njd9wuCS4vwR4vN4xoarNolZa4Hz8hsFclOMMHFVVroNpgefh/4J7OriV8a+yjmyQZ/V7rX58G3BaxePNwKIa7ycB7Mavrqm3jz4OdYkXoD/Cez8e+F2973nYcQpLF3acaqWrcYzq/o+NHKfqdA2OUaM8V4Xl2eAY1XpPo45TjTxrHaNG5Rx1nCq2fwI/kO3lUDvQS4HbG/2PjudmVxYRBb8mvg48qqpfqpNukYh0BvezwJ/j9/oYRVXXqepyVV2FXw30M1Ud82s9yKdFRNpG7uM3Dj4UllZVdwFbReSEYNOrgEfqvLUov1a3AC8RkVzwObwKv041rKxHBH9XAmdHyHs9/kmD4O//bZC+IRE5HfgH4ExVzddJt7ri4ZmEHCcAVf2jqh6hqquC47UNvzFyV0ieSyoevokaxylwC349MyJyPH5nhFqTwf058JiqbquTH/htSacG91+J3zNmjIrjFAM+jt8gWu97Hnacovw/hOYXdozqpA07TqPS1TpG+CfusDzDjlPY+6l1jGq994PHqc5nOeYY1XnvY45TjXPMo/i9qd4cvHRS/pdGmczIMxtv+CeznYCD/wV7b410f4Z/WTfShe9gt8iqdCfid4V9EP8LeFmEMpxGnd5Q+O0QD3Coq9ylDfI7Cb9L3oP4X/auGulywD6gI0IZP4n/T/oQ8C2C3iEh6f4XPzg9ALyq0WcNLATuwj+p3YXfRTMs3ZuC+0X8E8DtdfLcBGytOE5fq5Huf4L38yB+t8RlUb4TBD3TauT5Lfxujg/in2CX1ClnCvh2UIbf4584QvcNfAN4X4TP88+A+4LP/7fAC2uk+xB+j5sngM9y6Jdu6Pc85DidUSNd9XH6bY10Yceo1r6rj9Mbw9LVOEa18qw+TmfVSBd2jGqeCyqPU519hx2jWmnHHCdqnGPwzxO/Cz7b71Pjf3SiNxvBbYwxpiGrhjLGGNOQBQtjjDENWbAwxhjTkAULY4wxDVmwMMYY05AFCzPjiYiKyBcrHl8iIpdPUt7fEJE3N0552Ps5V0QeFZG7q7avEpFhEflDxS01gfxXicjbJ6/ExoxmwcLMBkXgbBHpnu6CVBKR+DiSvxd/NPsrQp7brKonVdxKIWkaWQWMO1iM8z2YecyChZkNyvjLRn6k+onqKwMRGQz+nhZM/PY9EXlCRD4rIueJyO9E5I8icmxFNn8uIv8bpHtD8Pq4iFwtIvcGk839TUW+d4vIf+EP6qouz9uC/B8Skc8F2y7DH3T1NRG5OsobDkbtXx/s/34ROSvYvioo6++D258GL/ks/oR3fxCRj4jIBSLy5Yr8fiQip418RiJyhYj8FnipiLww+KzuE5HbR0Y3i8gHReSR4P1/N0q5zRw23SOo7Wa3RjdgEH9K56fxp3S+BLg8eO4bwJsr0wZ/TwN68ecxSuMvdDOySMyHgGsqXn8b/g+n1fijjzPAhcDHgzRp/BHxRwf5DgFHh5RzKf7UKIvw5wj6Gf6U6xAyp1WwfRUwzKFRu18Jtl8JvCO434k/grcFf9R9Jti+mmAhLKpmAQAuAL5c8fhHBHMc4Y8UfktwPwn8ClgUPP5L4Prg/g4OreNQd00Uu839W6JBLDFmRlDVfhG5AX/xm+GIL7tXg2m1RWQz/qIz4F8RVFYHfU9VPfw5ep4EnoU//9aJFVctHfgn5xL+ZG5PhezvRcDPVbUn2OeN+GuL3NKgnJvVn0G00muAM0XkkuBxBliJfwL/soicBLj4k8uNl4s/hQb46yU8F7jTn56IOP7UIOBPJ3GjiNwS4T2YOc6ChZlNrsGfn+c/K7aVCapTg8nYKhuHixX3vYrHHqO/+9Vz3ij+HDx/p6q3Vz4RVOUM1ShfvenFx0vwF7oZteBT0LC/G3+5zRhQqPH6g59LIFNxv6CqbsV+HlbVl4bk8Xr8YHcm8E8i8hw9tF6CmWeszcLMGqq6H3/pyPdWbH4afyI28CeDS04g63NFJBa0YxyDP/327cDfikgS/BlHpfEiUr8FThWR7qDh+G34qwVOxO3A3wUBEBF5frC9A9gZXAm9E/9KAGAAf72DEU8DJwXvawX+mgdhHgcWichLg/0kReQ5wSynK1T1bvwFujqB1gm+FzMH2JWFmW2+yOhFiv4D+L8i8jv82VBr/eqv53H8k/pi/BlDCyJyHX57wu+DE3YPDZapVNWdIrIOf6poAW5V1YlOE/0p/CupB4P9Pw28Afg34H9E5NxgPyPv90GgLCIP4LfDXIO/bvofOTRjaliZS0FV27+ISAf+OeEa/DaSbwfbBPg/2nh5XjOH2ayzxhhjGrJqKGOMMQ1ZsDDGGNOQBQtjjDENWbAwxhjTkAULY4wxDVmwMMYY05AFC2OMMQ1ZsDDGGNPQ/w9Y2ygMZ7QhKAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_sfs(sbs.get_metric_dict(), kind='std_dev');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SBS returns subset of dataframe with optimal K features" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(569, 24)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_sbs.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is the subset of selected feature names:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('mean radius',\n", " 'mean texture',\n", " 'mean area',\n", " 'mean smoothness',\n", " 'mean concavity',\n", " 'mean concave points',\n", " 'mean symmetry',\n", " 'mean fractal dimension',\n", " 'radius error',\n", " 'texture error',\n", " 'area error',\n", " 'smoothness error',\n", " 'compactness error',\n", " 'concave points error',\n", " 'symmetry error',\n", " 'fractal dimension error',\n", " 'worst radius',\n", " 'worst texture',\n", " 'worst area',\n", " 'worst smoothness',\n", " 'worst concavity',\n", " 'worst concave points',\n", " 'worst symmetry',\n", " 'worst fractal dimension')" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sbs.k_feature_names_" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'The best quality is 0.9877037727061015 with 24 features in dataset'" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'The best quality is {} with {} features in dataset'.format(sbs.k_score_, len(sbs.k_feature_idx_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Quality is increased! ***0.984 -> 0.988***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Saving scores to dict and try another SFS algorithms" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": true }, "outputs": [], "source": [ "sbs_dict = dict()\n", "for i in sbs.subsets_.values():\n", " sbs_dict[len(i['feature_names'])] = i['avg_score']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we try to use Sequential Forward Selection, so toggle forward parameter to **True**

\n", "Sequential Forward Selection means that:\n", "\n", "* We will **start with 0 features**\n", "* On each iteration N we fit estimator with N features and keep on N subset of features with best scoring" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": true }, "outputs": [], "source": [ "logit = LogisticRegression()\n", "sfs = SequentialFeatureSelector(logit, \n", " k_features=(1, 30), \n", " forward=True, \n", " floating=False, \n", " verbose=2,\n", " scoring='accuracy',\n", " cv=cv)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 30 out of 30 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:34] Features: 1/30 -- score: 0.920835274026[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 29 out of 29 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:35] Features: 2/30 -- score: 0.954261760596[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 28 out of 28 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:36] Features: 3/30 -- score: 0.966589038969[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 27 out of 27 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:36] Features: 4/30 -- score: 0.971867722403[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 26 out of 26 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:37] Features: 5/30 -- score: 0.975376494333[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 25 out of 25 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:37] Features: 6/30 -- score: 0.977115354759[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:38] Features: 7/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:39] Features: 8/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 22 out of 22 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:39] Features: 9/30 -- score: 0.978900791803[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 21 out of 21 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:40] Features: 10/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:40] Features: 11/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:41] Features: 12/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 18 out of 18 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:41] Features: 13/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 17 out of 17 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:42] Features: 14/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:42] Features: 15/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 15 out of 15 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:43] Features: 16/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 14 out of 14 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:08:43] Features: 17/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 13 out of 13 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:44] Features: 18/30 -- score: 0.978900791803[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 12 out of 12 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:44] Features: 19/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 11 out of 11 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:45] Features: 20/30 -- score: 0.982409563732[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:45] Features: 21/30 -- score: 0.982409563732[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 9 out of 9 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:45] Features: 22/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 8 out of 8 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:46] Features: 23/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 7 out of 7 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:46] Features: 24/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:46] Features: 25/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:08:46] Features: 26/30 -- score: 0.984163949697[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 4 out of 4 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:08:47] Features: 27/30 -- score: 0.982409563732[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:47] Features: 28/30 -- score: 0.982409563732[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:47] Features: 29/30 -- score: 0.984163949697[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:47] Features: 30/30 -- score: 0.984179475237" ] } ], "source": [ "X_sfs = sfs.fit_transform(df_scaled, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ptotiing results:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcJHV9+P/Xu++ee3ZndtmT5RY0SnRFjAfEEzxAwJt4+yPx6xHli4kIQYMiSUBi8vOIRlGIqFFjEhKRI0TUPIy4ixyKCHuwLLuz7M4ec/ZZVe/vH1Wz29NT3V0zOz3TM/N+Ph79mD4+XfXprp561+cWVcUYY4ypJzbfGTDGGNP6LFgYY4xpyIKFMcaYhixYGGOMaciChTHGmIYsWBhjjGnIgoUxxpiGLFgYY4xpyIKFMcaYhhLN2rCI3Ai8Btinqs8IeV2AvwNeBeSAd6rqr4LX3gFcGST9tKre1Gh/fX19umHDhlnKvTHGLA333XffflXtb5SuacEC+AbweeDmGq+fC5wU3J4HfAl4nogsAz4BbAQUuE9EblXVQ/V2tmHDBjZv3jxLWTfGmKVBRJ6Ikq5p1VCq+lPgYJ0k5wM3q+8XQI+IrAJeCdylqgeDAHEXcE6z8mmMMaax+WyzWAM8WfF4V/BcreeNMcbMk/kMFhLynNZ5fuoGRC4Rkc0isnlwcHBWM2eMMeaI+QwWu4B1FY/XAgN1np9CVb+iqhtVdWN/f8P2GWOMMTM0n8HiVuDt4jsTGFbVPcAdwCtEpFdEeoFXBM8ZY4yZJ83sOvtt4GygT0R24fdwSgKo6j8At+F3m92K33X2XcFrB0XkU8CmYFNXq2q9hnJjjDFN1rRgoapvafC6Au+v8dqNwI3NyJcxxpjpsxHcxhhjGrJgYcwSNJIr8dTQOK7nzXdWzALRzBHcxpgWVHJctu0dpux4PDWUZ0N/Jz3t6fnOlmlxVrIwZglRVXbsG0VV6O3IkIzHeGxgiK17hiiUnPnOnmlhFiyMWUL2DucYGi/SmU0CkErE6e3IMFpweGjnQQYOWdWUCWfVUMYsEWOFMk8MjtHTNrXKqSOTxPOU3QfG2Tec59i+Dnra0/iTQ5u5pqqUXQ/XOzJ5xaQjUXVcYuIH/mayYGHMElB2PbY+NUx7OkksFh4AYjGhpz1NyXHZsmeYnvY06/s6yKTsNNEsE0Gh5HgUSg65ksN4wSFXdPBHF4S8J+S5ZDzGszYsb2pwt1/BIlRyXIplD1VF4fCPThWUivuqwXPRCNDdnmr6FYyZXarKzsFRXFdpTzc+dqlEnFRHnLFCmYd2HmRNbxvZdLRTRSaZoC1i2qXI9ZTh8SJjxbIfFEoO6unh/8FEPEYqEaMjmyQ2jRP/0Hix5sR6s8WO6gI2cVVSKLnkSg6juTJjhTKu56EhvxxhcmAQkWn9uDxVGIRjettY2Z21oLFA7B8tsH+0yLKO6fV4mqiaemoo7x/7CFTh+JWd9HVlZ5LVRW00X2bH4Aj5oksyEQSFzPSCwnyyYLGAuJ4yXigfDgyjhdLhOs2YCMlEjGw6QbxGNcNs8Dxl71Cepw7lWN3bRr8FjZaWKzo8vm+U7rbkjN4fiwldbanI6V3PY/veEcaLDmuXdzT1t7hQlByX3QfGGRzJ05ZO0jvNoN0qLFgsEH7f+BFGc2XiMSGViNWtf26WWEzobkvhecqeQzn2DOVYs6ydvq4sybh1rmslruexbe8wmWSceGxujk08FqOnPc2+4Ry5ksMJK7uW7MWEp8rgcIEnD4wSE1nwHQYsWCwA48UyW/YMo0rLXJXEYkJ3exo36EEzcDDHmuXt9HVmSFjQaAlP7h+jWHLpnuMBdyJCT3uGsUKZh588yEmreujIzKxks1CNFcrs2DdKrujQ1Zacs2DdTBYsWtyhsQJbnxohk4yTacGGw/jhoOHx5P4xBg6Os2ZZ++ERwQpwuKE9aGAPGtUrG9xNfYlYjGwqHvnK9OBogb1D+Xm9uOjIJCk5Lg8/eZAN/V2s6M60xJX1RDufSNCsJyBIxWO/LU/ED3zxmERuVyg5LgMHx9k7nKctlWiZi7vZ0HpnHwP4jddPDeV4YnCMrrZUy1fxTFQ/uJ7Hzv1jPLl//HAgmPg3mwgLInI4gJjo4jGhtyNNb3uatnSiZvVOoeSwbe8InW2peT85pxJxutti7Ng3Qr5UZl1fx7xeZQ+NF3l83wiOo0GQqPhdTrrvPzrcSykWI5mIkU7ESSVjpJMxkvE4iXiMRMwPKKOFMjsHxwDoXeBVTmEsWLQg1/N4YnCM/SMFetvTc94ucTQmgoaZfa6njOTKDA4XECCTirOsI01nW4r2dIJ4LIbrKdv3jpBKxFvmAmMiyO0fKZArOpxwTDfp5Ny2Yziux64DY+wdztORSdKRmd7+PU9xPaVQdsiV/P/Ric4lh4OMQkc2uWirYS1YtJiS47J1zzC5okNP+/xfGZrWEY8Jbekj4xjKjsfe4TwDh3KIQGdwohovOi0XsEX86srxoB3j+JVdkcdjxGOxo+pVNZIvsX3vCK6rM77ij8WEWExILuEZkixYtJDxQtCQDXPeKGkWnmQiRjLhd2tVVUqOx3ihNK2urnOtPWjHeGxgOHI7gAj0d2VY3pWhLZWIfLJ3PY9dB8bZO5SjPZOMNCDR1GbBokUcHC2wde8IbanEnBfRzcInIqST8QXx25kYIR6V6yn7R4vsHc6TScZZ2dNGT4OZBMYKZbbvHaFUdhd8l9VWYcFinqkqAwfH2XVgnK621KKt7zRmpuIxOTxLbslxeWJwlB2D0NueYkV3Gx2Z5OFqKtdT9hwaZ/eBcdozSSuhzyILFvPIcT12DI5ycLRAT0d6wQz7N2a+pBJxUok4qkq+5PLYwBAxEVb0ZOnMJHnywBiFkmv/T01gwWKeFEoOW54aplT26O3IzHd2jFlQRIRsKkE2lcD1lMHhPHsO5sim4i3XuL9YWLCYB0PjRbY+NUwqEW/pxkhjFgK/msr+j5rNgsUcmhhot3P/GJ2ZFMmEtU8YYxYGO1vNkYnZOJ/cP05PW9oChTEt7NYfJDn7uZ2csqaLs5/bya0/mNu5raLufyLd807u57gNcMstzcuTlSzmQKHsD7QrlJxFNVeMMYvRrT9IcuVHsxTyfgP5wG7hyo/663Ocd2G5ZfZfnW7nTrjkEv+1iy+e/XxJraX7FpqNGzfq5s2b5zsbU4zkimzZM0IiHrMVxIxZAM5+bicDu6eW/Fev8bhn02hT910uw4uf08mB/VP3H4spvcv81S0Bhg4Jnje1x9exx8KOHdH3KSL3qerGRumsLqRJVJW9Qzke2T1EJhW3QGEWnOlUxcx3tc1sUfWv5MMM7Ba+9Hdptm+dfNqcbpVRdbpSCe65O8HHL83ygtPDAwWA58HLzy3zyleXOec1ZTwv/DPs3Bnxw06TlSya5NBYgcf2DNPdlrbVwsyCU13FAZDJKp++Lj+lKmY6aVvZ7l3CVX+W5Wf3hJ/skymlXPI/48lPc3nFq8qkM8oX/jbT8LOHfUeplPKM0122/C7O6IjQ0am85BVlfnZPgkMHGpdsapWAmlWysMvdJtk34s9nb4HCLEQ3XDv5BAhQyAtXfzzD3j2Tn//y/58OTXvDtZkFESw8D759c4rrr8mgCq97fYnbf5gMDQBnnOlw54+S3PHDJF/42zSqU/+/w76nsO+oVBLu3xTnwjeWeeVryvzBCx1S6drB99LLC5Pef+nlhSnp2trgmmuO+isJZSWLJiiUXR7accAas82CMz4OP7k7yYf/JMuRlUhmRkR55MkRWnmRuB3bY1xxWZZNv0jwgheX+dR1edauU279QZIbrs2wZ0BYtdo/UVcHvsF9wgtO7+RovicR5dHdI1Oej7L/6nTr1sFnPiPTbtxuiTYLETlHRB4Vka0i8rGQ148VkbtF5CERuUdE1la89jci8rCIPCIify8LaCawofECCyG3R1vXOhf7blY+F9s2p6N6m9+9JcmtP0jy/ve0cebvdfHhP2mreYJftdrjwa3Dk26rVodXnqsKZz23k0//RYZN98Zx3db6Pjee2sm5Z3fw6CNxrr0hx43fzrF2nX/xfN6FZe7ZNMqju0e4Z9No6Im6f4Wyek34xXb191TrO1q1Ovz9UfZfme7exwZ5fEdzekFNaFrJQkTiwGPAy4FdwCbgLar624o03wP+U1VvEpGXAO9S1beJyB8A1wEvDpL+D3C5qt5Ta3+tUrLwVHnoiQOkE/GWnhSwVlH3k5/J8+rzj/wwf/jvST758dmtj46673r7j5rPxbjN2f7u/aV7hBXHeLzyVX4D6sDuGFf9+VG0WWSUC95YYt/eGD+7J0GpKHR2euRyguu2zvcZiylXXJ3nbe+e3d9ylDaL2WzXGRov8pwT+mc0H1bUkkUzg8XzgU+q6iuDx5cDqOq1FWkeBl6pqruCksOwqnYF7/088EL8Mt5Pgbep6iO19tcqwWI0X+aRXQdbfr6nWo1jUa1a7fGTzdPvRuh58MLf72T/YOsG0lY30y6cBw8Ir3xRB8NDU7/7vn6P/7l/dFKJImpVSKO0Y2N+1dbll1YHqdZwtF1iZ1Jl1Oj7nK6FHixeD5yjqu8NHr8NeJ6qfqAizbeAe1X170TkQuBfgD5VPSAi1wPvxQ8Wn1fVK+rtr1WCxeN7RxjOlWjPtG7Xwe1bY5zz4g7C61qV/3t58fCjz16brpnutReUecWry7z4bIdsm/9s2D/Ea15X5leb4tzxQ79h8Kk9tQLF5H032n/UfC7Wbb7y1WU2HO9Xb9Q6Ee3bK9z1oyR33Jbklz+Ph/bLh9p157PplDVdoQ3C8/19zsVnb7aFHizegF9qqAwWZ6jqByvSrMYvQRyHX3q4CHg60A/8HfCmIOldwJ+r6k+r9nEJcAnA+vXrn/PEE0805bNEVXY9Hnh8P11tqZacHtl14etfTvF312coFQn9x43aPa+tTUmllaFDMbJZ5eyXlVnWp/zLt1MUCke2G48rbW3K6GiMVFp50VkO922KM3Qo2qCnqAOkpjOQaqFvM5lUymX/Oz7lNJcNx7nc819JisUj33siqaxb77FjewxV4fgTXc55TZnvfSvF4L75GXDWqt/nXHz2ZpuLYNHMuoBdwLqKx2uBgcoEqjqgqheq6u8DVwTPDQMXAL9Q1TFVHQN+BJxZvQNV/YqqblTVjf39/c36HJGN5EoALRkotj4W403ntfM3n87ywrMc/uLTeTLZyRcKtbrnhaW7+m/y/PzBUW767hgXvLHEpl8kuOXr6UmBAsB1hbIj3PDFHL/49Qhf+kaOKz8Vvs3qfdfbf9R8LsZtXvu3ee755Qgf/8s8HR3KHT+cHCgAnLLw5BMxPvh/i9x2zyi3/3SMD/9ZkT+/Kvr+Z1urfp9z8dkXg2aWLBL4DdwvBXbjN3C/VVUfrkjTBxxUVU9ErgFcVb1KRN4E/H/AOfjlxtuBz6nqf9TaXytUQz288yAKLbW0ZbkMX/1Sms/fkKa9XbnqmgKvPr+MyOzWtbounLq+C0JKK2HF/NmqD7dt1q7eOdpumc2wEL7PhWhBV0MFmXgV8DkgDtyoqteIyNXAZlW9NWjXuBa/O8ZPgferajHoSfVF/N5QCtyuqpfW29d8B4t8yeHXTxyc97EVlf8Mff1KKgW7d8U497UlrrqmwPK+5h3vxVzMb2X2vZu5CBZNHcGtqrcBt1U9d1XF/e8D3w95nwv8cTPzNtsOjhbmfbR2dfe8wX0CKG9/T4ErP1Ws/+ZZEDai1Ir5zWffe/O4ns77/3WrsP6Ls8D1lL3Dedoz8zd7SqkIn/nE1CkaQPiv2+dmFbHzLizz6evyrF7jIaKsXuMtuPmBFiL73pujWHYZyZU4NFak7NSYtW8JsbmhZsFYoYzrKvEmzmsQVtf6snPK/OzHCe74YZIf/1eS8bHwK6A9A3N3ZXTehWU7Sc0D+95nl+spuaLDqWt7KTsuOw+MMV4s05lNLdmShgWLWTA4nCfVxEbtsMVQ/uxDWWKxLI4j9C7zeNV5Ze6+M8HBkOmNa00pYIwJN5IrcWx/B53ZJJCkuz3F4EiBXQfGEISObLIlez02kwWLo1RyXA6NFelub15VT9gMoJ4nZLLKjd8eY+PzXBIJeF7E2SqNMbWN5Eos60yzojt7+Ll4LMYxPW0s60iz51COp4ZyZJKJJbVOzdL5pE0yNF4CgWbNc+g4tRdjyefgzBe4hx9PVEMsxq6BxsyFYtklHheO7e8M/Z9OJeIc299Jf1eWXQfGODhaoD2TbKnu8s1iweIoTKyG16yri0cfiXH5pbWnig6rXrK6a2NmZqKd4unrekk2mAS0LZ3gpFXdjObLPLF/lKGxIh3ZZEtPHnq0Fu8nmwO5okOh7JJKzO5VRakEf399mgvP6WBgV4w/elfRRp4aU4enSr7kcDTjxoZzJY7t74w8r5uI0NWW4unrlrFhZSf5ksNwrnRUeajmeYp7+Obheh6O69/KE7c56qllJYujcGC0MOs9oH79YJyPX5rl0UfinHdhiY//ZYFly5XTn+Na9ZIxVYpll3zJQQTaM0kOjRfpyCSnfQE3kiuxvDPNiu7pzxYdE6G/K0tPe5qBg+M8NZQjm0qQTc3s9KqqjBcdyo5HIi5+Q3pQuSD4QWriL4AILOtIH+VSVY1ZsJgh1/MYHCnQcRSzy1Z2hz1mlfK001x+8t8J+lYoX75pnD98uXM4rVUvGeNzXI/xol+K6MgkOWFlF11tKRLxGEPjRR7fN0Kh5NKZTUZqSyyWXRJ12imiSsZjHNvfSV9nhicGRzk0VqAzm4pcNXUkSLgs78pyTE+W9nTrzF5twWKGRnIlPE+JzbDPdXV32D0Dwp6BGGc83+GLN47T1T2buTVmYfNUyRcdSo5LKhlnzbI2etvTZKqu3nva0zxj/XKe3D/G/pE8HdlU3fYH1/O3e1qEdoqo2jNJnra2l4OjBXbuH0OVuoHLU2W84OC6Hn1dGY7pbZtxqaSZWi9HC8Te4TyZ1MzbKsK6wwLs2hmzQGEWJVXF8RTH9XA9JWrVvuN6xGKwvDNDX2eW9kyi7hiHZDzG8Su76G1P8fi+UQpAZza8a/twrsSGabRTRBUToa8rS1dbioGD4+wbLpBNxScFN0+VsXwZV5UVXRmO6WmbEvxaSevmrIUVyi4jufJRTRpYa1T1XI62Nma2qSqOqzhBQ6zrKf4MZb5MMk5bOkEmGY88qC2djNPdFr06Z0JvR4b2TJKd+8c4MFI4XFU1YSRXom+G7RRRpRJxNqzooq8re7hqqiOTIl9y8DxlZU+WFT1tZBZA11sLFjMwNF7gaIZVqEJbO4yPTX3NRlubVuK4HiO5UuTqVhHIJBN0pBNk0wkyyQSpRIxkIkYiHpvzUc+pRJwTVnbR055mx74R4rEYHZkkhZJDMhE76naKqDoySU5d28uBUX8U+IqeLCu7s7Pek7KZLFhMk6fKU0P5GTdsq/oT/o2PCfG4Tlm8fql1hy07Hl4Tp8lfLBJxaercY2E8VUbyJU5Y1U1ve7RSdKyJA1RnSkTo68zQmUkGV/f+DMxPX987p+MiJnpN9XdlGyduQRYspmm84FAquzPqpeB5cPUVGb51U5p3XlLkGb/ncsNfLc3usJ6nDOdKpJOxlq6nbRX5kkOpXCaViJFN16+zny3D4yVW97bT19m8apq5lE7GOWlVNwdGC4hIS/U0Wgjsv3Sa9o/mZ1R09Dz4xMcy/PM307z3fUU+eqVflXXeRUsjOFTKFR2Kjsu6vnZWdLct2Vk8p8NTZaxQZv9IgQOjBUBpS09/PEFUo/kyPe0p1ixvb8r254sEDc9m+ixYTIOqcnCsSMc0r0g8D668LMv3v5Pijz9Y4NKPFY+qzWOhclyP0XyZzrYkJ6/ubsnuga0qJkJXNkVXNsW6vg6Gx4vsHcpzaKxIPCa0Z5KzFnSLZZdYDDas6FxyM6ua2uy/dRqKZRd3mmMrXBcuvzTLv30vxfs/UuBDly29QKHBVbEqHL+yk+WdmZar115IkvEYfV1Z+rqy5IoOB8cK7BvO47pKOhU/qiDsuB65kj8/0kJqfDXNZ8FiGvIlN9KQ+sqR2ZkM5PPChy4r8IFLm7+06UxEbWCunGIgqpLjMpYvs7wrw/q+DjsBzbK2dIK2dAeretsZK5TZc2icQ2NFutqmv0iPp8pIzm/Qtvp8U82CxTSM5ksNR3lWj8zO5yGRUNZvaM1lGYfHi/4U6xHSKn5vrol+8yJCIiYk4jGS8dikEpenymiuRCIe45Q1PfRE7E1jZiYeE7rbUnRlk+wbLvDE/lEyyemVMobHS6xetngatM3ssmAxDUO5UsMV8cJGZjuOcMO1mZbr6ZQvOaRTcU5b2xu5W6ZTMdPlxCRuE7ey6x0OOp7Cqt42VvW2Leppm1uNiLCyJ0tnNsn2vSMMjfuljEZtD6P5Mt1ti69B28weCxYRlV3/5NjoSm2hjMx2gs/zjPXLptV/PxH3B1eFzZ7gen4QKTke8bh1TZxPbekEp67tZeDQOAMHx+vOxDrRoH3cSmvQNrXZJV9E+ZLTOBG1R2C30shsnRhstbJrVnskxWP+mImutpQFihYQjwnrlndw6tpeyo4/Erua6/kN2iet6rb2JFOXBYuIxgvlSFddF79raiN2q43MHs75g62WWd30ktCVTfH09cvobk9xcLSA4/rtZ6rK8HiJ41d2WXA3DVmwiGh4vBRpnd1dO2PE48oxqzxElNVrPD59Xb5l2ivGC2Xa0wlWL7O66aUkGY9xwsouTljVzVihTK7oMGQN2mYarM0iAtdTxgoOXW31r76GDgn/+t0UF7yxzGc+m5+j3EVXclw8VU44pttGTS9BE3MkdaQTbN87QiZpDdomOgsWERTK/qpcjcYYfO9bSQoF4R3vbb3xFJ7nL7ByyuruSCUks3hlUgmetrYXVaxB20RmwSKCfNFpOOraceCbX09z5gsdTjm19cZUDOWKrOvroNvGOxiYtK6zMVFYm0UEI7kSyQY9Re76UYI9AzHe8Z7WK1WM5kv0tqc5pqdtvrNijFmgLFg04HczLTesurnpH9Os3+By9suidbGdK34ferFJ4YwxR6WpwUJEzhGRR0Vkq4h8LOT1Y0XkbhF5SETuEZG1Fa+tF5E7ReQREfmtiGxoZl5rKTkejqt1G4QfeiDOrzYneNu7S8RbqDnA9ZRc0eHEY6wPvTHm6EQOFiKSFZFTppE+DnwBOBc4DXiLiJxWlex64GZVfSZwNXBtxWs3A9ep6qnAGcC+qPueTfmSg1J/QN3NX03R3qFc9Kapg57mi6q/uNCx/Z0zXtXPGGMmRAoWIvJa4AHg9uDx6SJya4O3nQFsVdXtqloCvgOcX5XmNODu4P6PJ14PgkpCVe8CUNUxVc1FyetsGy2U685ttPcp4Uf/keT1by7R0TmHGatDg9lDm70YvTFm6Yhasvgk/sl/CEBVHwA2NHjPGuDJise7gucqPQhcFNy/AOgUkeXAycCQiPxARO4XkeuCksqcGx4vka5ThfPtm1M4Drzt3fNfqvA8P0gMjZfo6UjP2WL0xpjFL2qwcFR1eJrbDjtLVdfnXAacJSL3A2cBuwEHv0vvi4LXnwscD7xzyg5ELhGRzSKyeXBwcJrZa8xxPfIlh2Qi/GsqFvxg8ZKXO/M6BbnjegyNFxktlFjRneWZxy7jxGO6bbZXY8ysiTrO4jci8lYgLiInAR8Cft7gPbuAdRWP1wIDlQlUdQC4EEBEOoCLVHVYRHYB96vq9uC1fwPOBL5W9f6vAF8B2Lhx46zP1Fco11/s6D/+NcmhgzHe/t55qSGj5LjkCg6JeIz1fR0s68w0XG/DGGNmIuqZ5YPA04Ei8C1gGPhwg/dsAk4SkeNEJAW8GZjUziEifSIykYfLgRsr3tsrIv3B45cAv42Y11kzXijXrMZRhZu+luaUU13OfIE7p/nKlxwOjRWCqTu6eOaG5azsabNAYYxpmkgli6Bx+YrgFomqOiLyAeAOIA7cqKoPi8jVwGZVvRU4G7hWRBT4KfD+4L2uiFwG3C3+2fo+4B+jf6zZMZQrkapRBXXvz+M8+ts411yfm5M1tVWV8aJD2fHobk+xYUUnnZmktUkYY+aEaIT1l0XkLuANqjoUPO4FvqOqr2xy/iLbuHGjbt68eda256ly//b9dGSToYPZ3veuNn61Kc5PNo2Syc7abkPzMV4o47pKX1eGlT1ttKVtlhZjzOwQkftUdWOjdFHPOn0TgQJAVQ+JyIoZ524BKJZdPE9DA8XOHTH++84Ef/KhYtMChecpY4UyniorurOs7M6SmcWFiowxZjqinn08EVmvqjvBH3nN1J5Ni0qu6NScaO2bX08Rj8Nb3zH73WVdTxnLl1DgmN42VnRlbZZYY8y8ixosrgD+R0R+Ejx+MXBJc7LUGkbz5dApMsZG4XvfTnHua8usPGb24qXreYzlyyDC6uX+gjQ2RYcxplVEbeC+XUSejd99VYCPqOr+puZsng3lipMG4936gyQ3XJthYLcAwnEnzN64iqHxEvEYrOvrYHlnxsZHGGNaznQqwdPAweA9p4kIqvrT5mRrfpUcl7LjHV6X+NYfJLnyo1kK+SP1Ul/5Qpr1G7yjXi51eLzIso40G1Z0Eo9ZkDDGtKZIwUJE/hp4E/AwMHFJPdHdddEplCYPxrvh2sykQAFQyAs3XJs5qmAxmi/Tnk1aoDDGtLyoJYvXAaeoauut7NMEY4UysYopyfcMhLd013o+inzJIREXTljZZYHCGNPyop6ltgNLZp7roVxpUg+kVavDG7JrPd/IRDXXSatsnQljzMIQNVjkgAdE5Msi8vcTt2ZmbL64nsd4oTxp6oxLLy+QyU4ODJmscunlhRlu3+Hk1T1kbdyEMWaBiHq2upWqeZ0Wq0LJRWHSNBrnXVhm6CB8+qo2QFm9xg8U022v8IIFiU44ppvO7JIpqBljFoGoXWdvanZGWkWu5ISOxTv+RL9kcfP3xmc0caCqMjxeYl1fB32dtiCRMWZhidob6iT8JU9PAw6f6VT1+Cbla94MV7VXTNi21a+WOvHkmY2vGM6V6O/KsKqn7ajyZ4wx8yFqm8XXgS/hL0z0h/jrY/9C2EAdAAAbQ0lEQVRTszI1XybWrQ5bGW/blhjdPR7L+6bfqD2aL9GZTbHeVq4zxixQUYNFVlXvxp+l9glV/ST+GhOLStHx/MkDY1NP6Nu2xDnhRG/a05Hnig7JeIwTjukiHrJdY4xZCKIGi0KwSNEWEfmAiFwALLpZZwslh1oztm/bEuP4k6ZXBVVyXFzP7yJrCxMZYxayqGewDwNt+MupPgd4G/COZmVqvozkSqHrbR88IBw8EOOEk6I3bDvukS6yNrW4MWahi9obalNwdwx4V/OyM78aNW6fMI2SxVihzIYVnXRkrIusMWbhi9obaiP+NOXHVr5HVZ/ZpHzNubLrUSi59HRM/Uq2bZkIFtFKFmXHI5mIsdy6yBpjFomo9SO3AB8Ffs2RiQQXlXyp9mJH27fEyWSUNWuj9YQaK5Q5fmWnNWgbYxaNqMFiUFUX9QjuXNEJXUIVgsbtEz2izPdXdjxSyRi9HVaqMMYsHlGDxSdE5KvA3cDhmWdV9QdNydU8GB4v1ly+dNuWOM85w4m0nbGCP52HlSqMMYtJ1GDxLuBp+DPPVq5nsSiChafKWMGhI2S+pvFxGNgd440RGrdLjksmlaC3I92MbBpjzLyJGiyepaq/19SczKNCycVTDa2Genxb9MbtXMHhhGO6alZnGWPMQhV1nMUvROS0puZkHrle7YbrbVv8qqlG3Wb9UkWcHitVGGMWoaglixcC7xCRx/HbLATQxdR1tpatj8WIx5X1G+oHi1zB4cRVVqowxixOUYPFOU3NRQvbtiXO+g0eqVTtNBOliu52K1UYYxanhsEimBPqh6r6jDnIT8vZtiXWsApqvOBwkpUqjDGLWMM2C1X1gAdFZP0c5KellEqwc0eME0+u3bhdLLtk03F6rFRhjFnEolZDrQIeFpFfAuMTT6rqeU3JVYvYuSOG60rdkkWu6HDy6m5bp8IYs6hFDRZ/2dRctKhGc0IVyy5t6QTdbXUaNIwxZhGI1HVWVX8C/A7oDG6PBM8tahPdZo8/MbxkkSuWWdfXYaUKY8yiFylYiMgbgV8CbwDeCNwrIq+P8L5zRORREdkqIh8Lef1YEblbRB4SkXtEZG3V610isltEPh/t48yubVtirF7j0RaybHah5NCRSdIVMurbGGMWm6jVUFcAz1XVfQAi0g/8F/D9Wm8QkTjwBeDlwC5gk4jcqqq/rUh2PXCzqt4kIi8BrsVfWGnCp4B5K8Fs2xKvWQWVLzk8bU2vlSqMMUtC1BHcsYlAETgQ4b1nAFtVdbuqloDvAOdXpTkNf3JCgB9Xvi4izwFWAndGzOOs8jzYvjW822y+5NCZTdFppQpjzBIRNVjcLiJ3iMg7ReSdwA+B2xq8Zw3wZMXjXcFzlR4ELgruXwB0isjyYGzHZ/HX0KhJRC4Rkc0isnlwcDDiR4lm9y6hUBCODylZ5Esua5a3W6nCGLNk1A0WIpIGUNWPAl8Gngk8C/iKqv55g22HnUmrJ2G6DDhLRO4HzgJ2Aw7wf4DbVPVJ6lDVr6jqRlXd2N/f3yA70zPRuH1iVckiX3LoakvSaculGmOWkEZtFv8LPFtE/klV38b0piTfBayreLwWGKhMoKoDwIUAItIBXKSqwyLyfOBFIvJ/gA4gJSJjqjqlkbxZjnSbnRwsCiWX41Z0WanCGLOkNAoWKRF5B/AHInJh9YsNFj/aBJwkIsfhlxjeDLy1MoGI9AEHg1HilwM3Btu9uCLNO4GNcxkowC9ZLFvu0btscmFIBNrSUfsFGGPM4tDorPcnwMVAD/DaqtfqLn6kqo6IfAC4A4gDN6rqwyJyNbA5WKb1bOBaEVHgp8D7Z/QpmmB7yJxQnqckYjFbBc8Ys+TUDRaq+j8i8nNgl6peM92Nq+ptVDWEq+pVFfe/T53ut0GabwDfmO6+j4YqbNsa49zXlic973gemVT40qvGGLOYRZ1I8DVzkJeWcWC/MDw0tWThuGrBwhizJEXtOnuniFwkS6RV93DjdtU0H67nkbVgYYxZgqK21F4KtAOuiOQ5slJeV9NyNo+2PjaxlOrkMRaup6QT1rhtjFl6Ip35VLWz2RlpJdu2xGhvV45ZPXVt7kQ8amHMGGMWj6gTCYqI/JGI/EXweJ2InNHcrM2fbVviHH+SS1ilWzJhwcIYs/REPfN9EXg+R8ZJjOFPErgobasxJxRAMr4kmm2MMWaSqBXwz1PVZwfTcqCqh0RkUa74MzoC+56KTWnc9jwlHhPiMStZGGOWnqhnvnIw5bjC4SnKa681uoBt3xreuO14HpmkNW4bY5amqMHi74F/BVaIyDXA/wCfaVqu5lGtOaFcT8mmrdusMWZpitob6hYRuQ94KX632dep6iNNzdk82bYlTjKlrDu2ekCeRyZpwcIYszTVDRYiksGfH+pE4NfAl1XVmYuMzZdtW2JsOM6jejiF66lVQxljlqxG1VA3ARvxA8W5+MugLmpbt8Q48eSQpVTVxlgYY5auRpfKp6nq7wGIyNeAXzY/S/OnWIBdO2O89oLy1BfFus0aY5auRpfKh8+ai736CeDx7TE8T2qPsbABecaYJapRyeJZIjIS3BcgGzxelHNDTSylWt1t1sZYGGOWukbrWSyp7j/btsQQUY47vqonlI2xMMYscXapXGHblhhr13tkspOf93tC2VdljFm67AxYYfvWeGh7heMqWVt32xizhFmwCLiu38BdPScU+IsepW1AnjFmCbNgERjYFadUlPAxFtgYC2PM0mZnwMDjhycQnFqyECBlwcIYs4TZGTDw+Fa/TaK62yz4U+1aycIYs5TZGTCwY1ucFSs9OqtGjniqxGJiwcIYs6TZGTCwY1uC40OqoFxXbbZZY8ySZ8ECUPVLFiecOLUKyvE8shYsjDFLnAULYM8AjI+Hr7vtuEomZWMsjDFLmwUL4NFH/dlkw7rNup5HJmUlC2PM0mbBgiPBotZss9a4bYxZ6uwsCDz6O+js8ujr1ymvCZC0YGGMWeKW/FnwllvgmzfHGB0R/vCMTm79QXLS64oFC2OMaepZUETOEZFHRWSriHws5PVjReRuEXlIRO4RkbXB86eLyP+KyMPBa29qRv5uuQUuuQSKRQGEgd0xrvxo9nDA8FSJiVVDGWNM086CIhIHvoC/dvdpwFtE5LSqZNcDN6vqM4GrgWuD53PA21X16cA5wOdEpGe283jFFZDLTX6ukBduuDYD+GMs0raOhTHGNLVkcQawVVW3q2oJ+A5wflWa04C7g/s/nnhdVR9T1S3B/QFgH9A/2xncuTP8+T0DfoO343lkrSeUMcY0NVisAZ6seLwreK7Sg8BFwf0LgE4RWV6ZQETOAFLAtuodiMglIrJZRDYPDg5OO4Pr14c/v2q139DtuErWxlgYY0xTg4WEPFfd3egy4CwRuR84C9gNOIc3ILIK+CfgXao6pV+rqn5FVTeq6sb+/ukXPK65BtraJj+XySqXXl4AbB0LY4yZ0MzL5l3AuorHa4GBygRBFdOFACLSAVykqsPB4y7gh8CVqvqLZmTw4ov9v5dfruza5ZcoLr28wHkXloGg22zCGreNMaaZwWITcJKIHIdfYngz8NbKBCLSBxwMSg2XAzcGz6eAf8Vv/P5eE/PIxRfDeRc6/G73IXra05Nes26zxhjja9qZUFUd4APAHcAjwHdV9WERuVpEzguSnQ08KiKPASuBa4Ln3wi8GHiniDwQ3E5vVl7rsW6zxhjT3JIFqnobcFvVc1dV3P8+8P2Q930T+GYz89aIqiICiVhY04sxxiwtdtlcg+MpmWQCEQsWxhhjwaIGx7XZZo0xZoIFixpcz1bIM8aYCRYsanBcz4KFMcYELFjUIEAqYcHCGGPAgkVd1m3WGGN8djasQbHR28YYM8HOhiFsjIUxxkxmwSKE4ynpRNzGWBhjTMCCRQjX9cjY1OTGGHOYBYsQjqe26JExxlSwYBHCxlgYY8xkFixqsDEWxhhzhAWLEIKNsTDGmEp2RgyhQCJuPaGMMWaCBYsqE2MsbIU8Y4w5ws6IVVwbY2GMMVNYsKji94SyMRbGGFPJgkUVx1MyKftajDGmkp0VqziuR9ZGbxtjzCQWLEIkbYyFMcZMYsEihPWEMsaYyeysGMLGWBhjzGQWLCrYGAtjjAlnZ8UKrqekbIyFMcZMYcGiguspWRtjYYwxU1iwqFB2PdI2xsIYY6awM2MFK1kYY0w4CxYVVJWULXpkjDFTWLCoYt1mjTFmqqYGCxE5R0QeFZGtIvKxkNePFZG7ReQhEblHRNZWvPYOEdkS3N7RzHwe3ifWbdYYY8I07cwoInHgC8C5wGnAW0TktKpk1wM3q+ozgauBa4P3LgM+ATwPOAP4hIj0NiuvAOjEokcWLIwxplozz4xnAFtVdbuqloDvAOdXpTkNuDu4/+OK118J3KWqB1X1EHAXcE4T8+qvY5GME7MxFsYYM0Uzg8Ua4MmKx7uC5yo9CFwU3L8A6BSR5RHfi4hcIiKbRWTz4ODgUWXW8TzS1rhtjDGhmhkswi7RterxZcBZInI/cBawG3AivhdV/YqqblTVjf39/UeVWddTsikLFsYYE6aZgwp2AesqHq8FBioTqOoAcCGAiHQAF6nqsIjsAs6ueu89TcwrItgKecYYU0MzSxabgJNE5DgRSQFvBm6tTCAifSIykYfLgRuD+3cArxCR3qBh+xXBc00TlxiphDVuG2NMmKadHVXVAT6Af5J/BPiuqj4sIleLyHlBsrOBR0XkMWAlcE3w3oPAp/ADzibg6uC5ponHhaQFC2OMCSWqU5oCFqSNGzfq5s2bZ/Te0XyZ3+0+xLM2LCdlq+QZY5YQEblPVTc2SmeX0oFEPGZjLIwxpgY7OwYyNsbCGGNqsmARsDEWxhhTmwUL/MkDOzPJ+c6GMca0LBtYAGRTCbIp+yqMMaYWK1kYY4xpyIKFMcaYhixYGGOMaciChTHGmIYsWBhjjGnIgoUxxpiGLFgYY4xpyIKFMcaYhhbNrLMiMgg8cRSb6AP2z2I626Zt07Zp22zFbVY7VlUbLzWqqnbzA+bm2Uxn27Rt2jZtm624zZnerBrKGGNMQxYsjDHGNGTB4oivzHI626Zt07Zp22zFbc7IomngNsYY0zxWsjDGGNNYs1vQW/0G3AjsA37TIN064MfAI8DDwJ/WSJcBfgk8GKT7ywbbjQP3A//ZIN0O4NfAAzTo+QD0AN8Hfhfk9/khaU4JtjVxGwE+XGebHwk+z2+AbwOZGun+NEjzcPX2wr5rYBlwF7Al+NtbI90bgm16wMYG27wu+OwPAf8afB9h6T4VpHkAuBNY3eg3AVwGKH5XxbBtfhLYXfG9vqreNoEPAo8Gn+1vamzznyu2twN4oM5nPx34xcTvBDijRrpnAf8b/Kb+A+iq9zsPOU7PqJFu0nGqs72wY1QrbfVx2hiWrsYxqrXN6uP09lrbDDlGtbZZfZwerpEu7BjV2uaU40SNcwxwHHBvcIz+GUjN6rlyNje2EG/Ai4Fn0zhYrAKeHdzvBB4DTgtJJ0BHcD8ZHLwz62z3UuBbRAsWfRE/003Ae4P7KaCnQfo48BR+f+uw19cAjwPZ4PF3gXeGpHsGfqBow19Y67+Ak+p918E/38eC+x8D/rpGulPxA9w9TA4WYWlfASSC+39dZ5tdFfc/BPxDvd9E8A99B/54nr4a2/wkcFmU3xnwh8F3lA4er2j0ewQ+C1xVZ5t3AucG918VfF9h6TYBZwX33w18qt7vPOQ4fb5GuknHqc72wo5RrbTVx+nmsHQ1jlGtbU46TnXShR2jhueC4Dj9TY1thh2jWvufcpyocY7B/798c/D8PwDvi3K+iHpb8tVQqvpT4GCEdHtU9VfB/VH8K4A1IelUVceCh8ngFtowJCJrgVcDX51Z7kO32YV/cvhakJ+Sqg41eNtLgW2qWm9QYwLIikgCPxgMhKQ5FfiFquZU1QF+Alww8WKN7/p8/OBG8Pd1YelU9RFVfbR6hzXS3hnsH/wruLU10o1UPGwnOE51fhN/C/xZhHRT1Ej7PuCvVLUYpNlXb5siIsAb8Ut2tbap+FefAN3AQI10pwA/De7fBVwUbLPW77z6OL08LF31caq1vRrHqFba6uM0Xud/sfoYRf2/rZUu7BjV3WbFcfrHGunCjlGtbU45TnXOMS/Br1GYOEavq/6cR2PJB4uZEJENwO/jR/Sw1+Mi8gB+0f8uVQ1NB3wO/4ftRditAneKyH0ickmddMcDg8DXReR+EfmqiLQ32PabCU5AoTtW3Q1cD+wE9gDDqnpnSNLfAC8WkeUi0oZ/1bSuwb5XquqeYD978K/cZtO7gR/VelFErhGRJ4GLgavqpDsP2K2qD0bY5wdE5CERuVFEeuukOxl4kYjcKyI/EZHnNtjui4C9qrqlTpoPA9cFn+l64PIa6X4DnBfcfwMhx6nqd17zODX6f4iQbsoxqk5b6zhVpmt0jEL2H3qcqtLVPUY1PtOU41SVru4xqkobepyqzzHANmCoIgDvIiQoHg0LFtMkIh3Av+DXx4+EpVFVV1VPB9YCZ4jIM0K28xpgn6reF3HXL1DVZwPnAu8XkRfXSJfAr3L4kqr+PjCOX21Q6/Ok8H+M36uTphf/yvI4YDXQLiJ/VJ1OVR/Br064C7gdv07VqU43V0TkimD/t9RKo6pXqOq6IM0HamynDbiCOsGkwpeAE/DrpffgV0fUksBvozkT+Cjw3eCqtJa3UCeoB94HfCT4TB8hKGGGeDf+7+g+/GqPUuWLUX7ns5Eu7BiFpQ07TpXpgm3UPEYh2ww9TiHpah6jOp990nEKSVfzGIWkDT1O1ecY/FJ9tdnt6jqbdVoL9QZsoEGbhR6pH7wDuHQa2/4E4XXY1+JH/x347QU54JsRt/nJsG0Grx0D7Kh4/CLgh3W2dT5wZ4P9vQH4WsXjtwNfjJDPzwD/p953jd9wuCq4vwp4tN4xoarNolZa4B34DYNtUY4zcGxVvg6nBX4P/wpuR3Bz8EtZxzTYZvVnrX58O3B2xeNtQH+Nz5MA9uJX19TbxzBHusQLMBLhs58M/LLe7zzsOIWlCztOtdLVOEZ1/8cmjlN1ugbHqNE2N4Rts8ExqvWZJh2nGtusdYwa5XPScap4/hP4gWw/R9qBng/c0eh/dDo3K1lEFFxNfA14RFVvqJOuX0R6gvtZ4GX4vT4mUdXLVXWtqm7Arwb6b1WdcrUebKddRDon7uM3Dv4mLK2qPgU8KSKnBE+9FPhtnY8W5Wp1J3CmiLQF38NL8etUw/K6Ivi7HrgwwrZvxT9pEPz99wbpGxKRc4A/B85T1VyddCdVPDyPkOMEoKq/VtUVqrohOF678BsjnwrZ5qqKhxdQ4zgF/g2/nhkRORm/M0KtyeBeBvxOVXfV2R74bUlnBfdfgt8zZoqK4xQDrsRvEK33Ow87TlH+H0K3F3aM6qQNO06T0tU6Rvgn7rBthh2nsM9T6xjV+uyHj1Od73LKMarz2accpxrnmEfwe1O9PnjrrPwvTTKbkWch3vBPZnuAMv4P7D010r0Qv1g30YXvcLfIqnTPxO8K+xD+D/CqCHk4mzq9ofDbIR7kSFe5Kxps73T8LnkP4f/Ye2ukawMOAN0R8viX+P+kvwH+iaB3SEi6n+EHpweBlzb6roHlwN34J7W78btohqW7ILhfxD8B3FFnm1uBJyuO0z/USPcvwed5CL9b4poovwmCnmk1tvlP+N0cH8I/wa6qk88U8M0gD7/CP3GE7hv4BvAnEb7PFwL3Bd//vcBzaqT7U/weN48Bf8WRK93Q33nIcTq3Rrrq43RvjXRhx6jWvquP0+vC0tU4RrW2WX2czq+RLuwY1TwXVB6nOvsOO0a10k45TtQ4x+CfJ34ZfLffo8b/6ExvNoLbGGNMQ1YNZYwxpiELFsYYYxqyYGGMMaYhCxbGGGMasmBhjDGmIQsWpuWJiIrIZyseXyYin5ylbX9DRF7fOOVR7+cNIvKIiPy46vkNIpIXkQcqbqkZbH+DiLx19nJszGQWLMxCUAQuFJG++c5IJRGJTyP5e/BHs/9hyGvbVPX0ilspJE0jG4BpB4tpfgazhFmwMAuBg79s5EeqX6guGYjIWPD37GDit++KyGMi8lcicrGI/FJEfi0iJ1Rs5mUi8rMg3WuC98dF5DoR2RRMNvfHFdv9sYh8C39QV3V+3hJs/zci8tfBc1fhD7r6BxG5LsoHDkbt3xjs/34ROT94fkOQ118Ftz8I3vJX+BPePSAiHxGRd4rI5yu2958icvbEdyQiV4vIvcDzReQ5wXd1n4jcMTG6WUQ+JCK/DT7/d6Lk2yxi8z2C2m52a3QDxvCndN6BP6XzZcAng9e+Aby+Mm3w92xgCH8eozT+QjcTi8T8KfC5ivffjn/hdBL+6OMMcAlwZZAmjT8i/rhgu+PAcSH5XI0/NUo//hxB/40/5TqEzGkVPL8ByHNk1O4Xguc/A/xRcL8HfwRvO/6o+0zw/EkEC2FRNQsA8E7g8xWP/5NgjiP8kcJvDO4ngZ8D/cHjNwE3BvcHOLKOQ901Uey2+G+JBrHEmJagqiMicjP+4jf5iG/bpMG02iKyDX/RGfBLBJXVQd9VVQ9/jp7twNPw5996ZkWppRv/5FzCn8zt8ZD9PRe4R1UHg33egr+2yL81yOc29WcQrfQK4DwRuSx4nAHW45/APy8ipwMu/uRy0+XiT6EB/noJzwDu8qcnIo4/NQj400ncIiL/FuEzmEXOgoVZSD6HPz/P1yuecwiqU4PJ2Cobh4sV972Kxx6Tf/vVc94o/hw8H1TVOypfCKpyxmvkr9704tMl+AvdTFrwKWjY34u/3GYMKNR4/+HvJZCpuF9QVbdiPw+r6vNDtvFq/GB3HvAXIvJ0PbJegllirM3CLBiqehB/6cj3VDy9A38iNvAng0vOYNNvEJFY0I5xPP7023cA7xORJPgzjkrjRaTuBc4Skb6g4fgt+KsFzsQdwAeDAIiI/H7wfDewJygJvQ2/JAAwir/ewYQdwOnB51qHv+ZBmEeBfhF5frCfpIg8PZjldJ2q/hh/ga4eoGOGn8UsAlayMAvNZ5m8SNE/Av8uIr/Enw211lV/PY/in9RX4s8YWhCRr+K3J/wqOGEP0mCZSlXdIyKX408VLcBtqjrTaaI/hV+SeijY/w7gNcAXgX8RkTcE+5n4vA8Bjog8iN8O8zn8ddN/zZEZU8PyXAqq2v5eRLrxzwmfw28j+WbwnAB/q42X5zWLmM06a4wxpiGrhjLGGNOQBQtjjDENWbAwxhjTkAULY4wxDVmwMMYY05AFC2OMMQ1ZsDDGGNOQBQtjjDEN/T+4gkRRFmgavQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_sfs(sfs.get_metric_dict(), kind='std_dev');" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'The best quality is 0.9841794752367644 with 30 features in dataset'" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'The best quality is {} with {} features in dataset'.format(sfs.k_score_, len(sfs.k_feature_idx_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now the quality is equal to our baseline. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Why quality of SFS is worse, than SBS? \n", "We use \"Forward\" algorithm, so on first iteration we select one feature and fit estimator with it. It's obviously, that finding \"the best\" feature fitting one dimensional dataset is not very effective. More than that, in Sequential Forward Selection we can't remove feature once added.

Let's try to find, \"bad feature\" that we add in our dataset once and on what iteration stage this happened." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "We add \"bad feature\" # {5} on 8 iteration stage\n" ] } ], "source": [ "sbs_feat = set(sbs.subsets_[24]['feature_idx']) #best feature set of SBS algorithm\n", "for i in range(1, 30):\n", " sfs_feat = set(sfs.subsets_[i]['feature_idx']) #iterate throw feature set on each iteration of SFS algorithm\n", " if len([x for x in sfs_feat if x not in sbs_feat]) > 0:\n", " print('We add \"bad feature\" # {} on {} iteration stage'.format(sfs_feat - sbs_feat, i))\n", " break" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Save results on each itertaion too" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": true }, "outputs": [], "source": [ "sfs_dict = dict()\n", "for i in sfs.subsets_.values():\n", " sfs_dict[len(i['feature_names'])] = i['avg_score']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we will try to use Sequential Forward Floating Selection, so toggle floating parameter to True. It can help us to remove worst feature at each iteration additional step" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": true }, "outputs": [], "source": [ "logit = LogisticRegression()\n", "sffs = SequentialFeatureSelector(logit, \n", " k_features=(1, 30), \n", " forward=True, \n", " floating=True, \n", " verbose=2,\n", " scoring='accuracy',\n", " cv=cv)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 30 out of 30 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:08:49] Features: 1/30 -- score: 0.920835274026[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 29 out of 29 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:49] Features: 2/30 -- score: 0.954261760596[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 28 out of 28 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:50] Features: 3/30 -- score: 0.966589038969[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 27 out of 27 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:50] Features: 4/30 -- score: 0.971867722403[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 26 out of 26 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:51] Features: 5/30 -- score: 0.975376494333[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 25 out of 25 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:52] Features: 6/30 -- score: 0.977115354759[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:08:53] Features: 7/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 7 out of 7 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:08:53] Features: 8/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 22 out of 22 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 8 out of 8 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:08:54] Features: 9/30 -- score: 0.978900791803[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 21 out of 21 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 9 out of 9 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:08:55] Features: 10/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:56] Features: 11/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 11 out of 11 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:57] Features: 12/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 18 out of 18 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 12 out of 12 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:08:57] Features: 13/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 17 out of 17 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 13 out of 13 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:58] Features: 14/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 14 out of 14 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:08:59] Features: 15/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 15 out of 15 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 15 out of 15 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:00] Features: 16/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 14 out of 14 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:01] Features: 17/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 13 out of 13 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 17 out of 17 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:03] Features: 17/30 -- score: 0.982409563732[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 13 out of 13 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 17 out of 17 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:09:04] Features: 18/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 12 out of 12 | elapsed: 0.3s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 18 out of 18 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:09:05] Features: 19/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 11 out of 11 | elapsed: 0.3s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.6s finished\n", "\n", "[2018-12-12 00:09:06] Features: 20/30 -- score: 0.978900791803[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.3s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.6s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.6s finished\n", "\n", "[2018-12-12 00:09:08] Features: 20/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.3s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.6s finished\n", "\n", "[2018-12-12 00:09:09] Features: 21/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 9 out of 9 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 21 out of 21 | elapsed: 0.7s finished\n", "\n", "[2018-12-12 00:09:10] Features: 22/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 8 out of 8 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 22 out of 22 | elapsed: 0.8s finished\n", "\n", "[2018-12-12 00:09:11] Features: 23/30 -- score: 0.978885266263[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 7 out of 7 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.8s finished\n", "\n", "[2018-12-12 00:09:12] Features: 24/30 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.9s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.8s finished\n", "\n", "[2018-12-12 00:09:14] Features: 24/30 -- score: 0.982425089272[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.9s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.8s finished\n", "\n", "[2018-12-12 00:09:17] Features: 24/30 -- score: 0.984179475237[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.9s finished\n", "\n", "[2018-12-12 00:09:18] Features: 25/30 -- score: 0.984179475237[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.1s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 25 out of 25 | elapsed: 1.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.9s finished\n", "\n", "[2018-12-12 00:09:20] Features: 25/30 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.1s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 25 out of 25 | elapsed: 1.0s finished\n", "\n", "[2018-12-12 00:09:21] Features: 26/30 -- score: 0.984179475237[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 4 out of 4 | elapsed: 0.1s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 26 out of 26 | elapsed: 1.0s finished\n", "\n", "[2018-12-12 00:09:23] Features: 27/30 -- score: 0.984179475237[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 27 out of 27 | elapsed: 1.1s finished\n", "\n", "[2018-12-12 00:09:24] Features: 28/30 -- score: 0.984179475237[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 28 out of 28 | elapsed: 1.2s finished\n", "\n", "[2018-12-12 00:09:25] Features: 29/30 -- score: 0.984179475237[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 29 out of 29 | elapsed: 1.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 28 out of 28 | elapsed: 1.2s finished\n", "\n", "[2018-12-12 00:09:28] Features: 29/30 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 29 out of 29 | elapsed: 1.2s finished\n", "\n", "[2018-12-12 00:09:29] Features: 30/30 -- score: 0.984179475237" ] } ], "source": [ "X_sffs = sffs.fit_transform(df_scaled, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Plotting results:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcXGWZ8P3fVVtX9ZLuzr6ThNWIiBJQ5lFgdEZgYIKAqIgs6gwzjrjx4PPKBwYZkEFFeXzm1RlfRFQERURUFCQwDOjMMy4EWQQxZCGSpAMk6fRa+znX+8c5nVRXn1NVvVR3dff1/XwqqTp1133uqlN9rrrPvYmqYowxxlQSmeoCGGOMaXwWLIwxxlRlwcIYY0xVFiyMMcZUZcHCGGNMVRYsjDHGVGXBwhhjTFV1CxYicpuIvCoiz4Y8LyLyLyKyRUSeEZE3ljx3sYhs9m8X16uMxhhjalPPmsW3gNMqPH86cLh/uxT4NwARmQt8BngTcALwGRHprGM5jTHGVBGrV8aq+ksRWVUhyVnA7eoNIf+1iHSIyBLgFOBhVe0GEJGH8YLO9yrtb/78+bpqVaXdGWOMKffEE0/sVdUF1dLVLVjUYBmwo+TxTn9b2PaKVq1axcaNGye0gMYYM9OJyJ9qSTeVDdwSsE0rbB+ZgcilIrJRRDbu2bNnQgtnjDHmoKkMFjuBFSWPlwNdFbaPoKq3qOo6VV23YEHVWpQxxpgxmspgcR9wkd8r6s1Ar6ruBjYA7xCRTr9h+x3+NmOMMVOkbm0WIvI9vMbq+SKyE6+HUxxAVb8GPAD8FbAFSAMf8J/rFpHrgcf9rK4bauw2xhgzNerZG+r8Ks8r8JGQ524DbqtHuYwxxoyejeA2xhhTlQULY4wxVVmwMMYYU9VUDsozxpgJMZgtsKcvSyIWobkpRiIWpSkeIRqx38MTxYKFMWbaGsgW2NU9SO9gjngsiio4rovgjeRtikdpbYrRkoyTSsRIxKMkYhEiEjT211RiwcIYM+0MZAvs2jdITzpHMh6jszUZmK7ouAzkiuwfzOO6igpEEJriERIx7xaPRWmKRYnHIkQiQiwSIRoRYlGpW83EcV0cVyk6StF1cV2lUHTJOy75okO+4JAvugBeOeNR4tHIpJezlAULY8y0oKoMZIvs6h6gN50nlYgxNyRIDIlFI8SiEVKJ4fk4rlJwlFyxiOMWcFzF683vEQT1Zxma6ErI0G7Er/94DxUR7+QfEfGDgBcAvHJ6ZQwrZyoR5bUr5iJ1rDFZsJihCo6LqqLqT6yl3pdy6HumeE8ETrpVQXNTzKrwk8xxXVyFeHTifj0WHJdcwZmw/MYiIkIk4p0Yh06SQVSVfr8m0ZcpkEpEqwaJSkS8X+OxaPW0pSfmiTSak3ot5ewZzIVOrDdRLFjMAPmiQ67gkskX6cvkGcgUKDju8F9HB/7VAwFi6LpurRRIxaMsn9dCR0tTXX/FGC9I7O3L0bV/AMdVlnQ0s6A9RaKWs0eIguOyty9DV3cat04nwrGKiFcTiEejxKJCPBYhHo3QnynQnymQTESZ29o0qWWy7/hBFiymmbDAAN4XOxGLkExEaYnE67b/zbv7aG6KsWJ+K3NScfuDmmBFx2Vff5ad+wZxVWlNxomIsLsnQ9f+NIs7m1k0yqBRGiRUldZUgmiksY6bq3rgElG+qGTyDq4qsYjQOclBwoxkwWKacFXZtW+Q3T1phNLAEKNlEv/oE7EoidYouYLDpl37aU3GWT6/lTmlF4XNmBQdl739WXb5QaItFR/WcNnenMB1lVd7MrxcY9CYDkFiSEQERJjAq21mAlmwmAaKjsuLr/SxfzBPe0uiIdoMmuJRmuJRsvkiz+/sob05zvJ5rbQm61OjmckKjsu+/gy79g2d0OOhvVsiEWFOadDo9oLGwvYUTfHosDynS5Aw04MFiwaXLThs3t1DvuA2ZFU8mYiRTMTI5Iv8YUc3HS1NLJvXQktTeNDQYY3tjXXdfDI5rrKvP8uu7jRUCRLlDgQNVV7tzfByT5qF7Snmz0nSl85bkDATzoJFA+vPFNi8u4doJMKc5sa+zJNKxEglYqRzRZ57aT+JuH/S04PXot3ZGxcCKd58O+M5oUfkYNDY15/lld7MuPM0JogFiwa1ty/Dtlf6aUnGxtX7ZbI1N8VIJaJ+TxuvJ/lQf76hthYz8SIitFm7kakjCxYNxlWlq3uQXfsGaW9JTMu5bUSEqAUFY2YUCxYNpOi4bN/Tz77+LB2tTQ3RkG2MMWDBomHkCg6bd/eSKzjjGp1qjDH1YMGiAQxkC7zQNT0aso0xs5MFiyk21JDd3BQb1k/eGGMayfRrPZ0hXFV27B1g68t9zGmOW6AwZhLcd2+cU45v48hlczjl+Dbuu7cxB5E2YjmtZjEFCv6I7J7BvDVkGzNJ7rs3ztWfSpHNeH9vXbuEqz+VAmD9OYWpLNowjVpOq1lMsnSuyPM79zOQLdBpgcKYSdHfB5/9x+SBE/CQbEa4+cbG6VDS19u45bRgMYn2D2R5bkc3gA2gMmaChF2y6e2Be++Oc+lFzbz5mDn07A8+3XXtEr53e4K9e6RqnqPZf63pevYLP7wrzt9e2MyJVcp51x1x9u0dWc43HbGA1avgzjsrfVLjI/Va3GOyrVu3Tjdu3DjVxQikquzuSbNjzwBtzYkJXcTGmNms/JINQDyurDncYdvmKIWCsHSZy6lnFvjpvXH27hn5txeNKU5RiESU49/ssHSZywM/jZPLHswzmVI+e1NmxGWgoP0HpQ0t52EOWzdHKRaF5StcTj2jwE9+WL2cJ/yZw9KlLvffN7yczc1wyy1wwQW1f4Yi8oSqrquazoJFfQ0NtOvuz9Le3ETE5usxZsKccnwbXbsCTqxR5QOX5jn1zALHHOsgUuHE/oUMR73W4cGfxXnwZ3G2vBDc2aS1VTn/ovywbd+7PcHAwMi/6fK0YemiUeWDf5/jtDOLHH1M9XIe8RqvnD//aZwXtwaX85BDYPv2wKcCWbBoANmCw5bdvWTzRdpbGm/GWGOmuyOXzUF15ElYRNm0q2/E9vvujXPzjUl2dwlLliqXX5kdUVsIyxOUprJmg1wWghczHZ42LN1Yy6kKRy0Pe+/gugFFClFrsLDeUHWSzRf5w879RCMRCxRmzGo5uc1mHZ3K/u6RJ8wlS4N/BK8/p1D181uyVOnaNTLPpcuUxx7vH7bNq9lUTxuWbqzlFAkv58qVoS8bF7t4Xiev9mYAobnJ4rEZm6HLEV27IqgKXbsiXP2pVEP0uW8EG+6P0bNfEBl+wk2mvKA6VpdfmSWZqi3PWtOOJs/xlLO5GW64YcxZVmTBog4cV9nTl6XFAoUZo+59wvVXN2YXykbw4M9ifOLvm3nDcQ7X35Rh6TIXEWXpMjewIXo01p9T4LM15llr2tHkOdZyrlypo27cHg0LFnXQn8njuG7DN2aPt8vfZOy7XuVsxDz37vG6cF787hb+x7Ft9PaEd6G85SsJXtoeqZrndHnvo/Hzn8b45Iebef0bHW797iDvfl+Bxx7vZ9OuPh57vH9CLtOtP6f2PGtNO5o8R1vO37ywhxe31y9QQJ0buEXkNOD/AFHgVlX9XNnzhwC3AQuAbuD9qrrTf+4LwBl4Ae1h4ONaobCN1MD9QlcP2YJDKtG4NYvxdPkL60Y40fuuVzkbLc94XFm5yuXFrRFcV1i9xuG0vy5wz/cS7Hl1ZMCIx5VCwXv92qMdVq52ePShOLnc9Hvvo/0uPXBfnP/5kRTHHufw9TsGaW2t6WUzXs9gjuMOXTCmQb5T3htKRKLAC8BfAjuBx4HzVfUPJWl+APxMVb8tIm8DPqCqF4rInwE3ASf5Sf8LuFJVHwvbX6MEi1zB4ek/7aOjOdHQq8KFdTlsblHe+a6DXf5+fE+C9GBQA547orFvovddaf+1lnO65BmLKX/30RynnVngiKPcyl0ob8pw3AlFNtzvdfV86ongHyXT5b3X+l26/ydxrrjMAkWQ6R4sTgSuVdVT/cdXAqjqjSVpngNOVdWd4p1Ze1V1jv/arwBvwetv9kvgQlV9Pmx/jRIsXulJs2PvQMP3gDpi6RzCuvx1zj34nfB6mgSn+8nDAxy11jux1eLl3cJD98f57DXJmvZdbf+1lnM65Fmvrp7T4b2DsuE/B1h9aHh/z6FA8YZ1XqBoaQlNOitNRrCo53WSZcCOksc7gTeVpXkaOBfvUtXZQJuIzFPVX4nIo8BuvG/XV4IChYhcClwKsLJe/cVGQVV5pTdDc1Pj9lbZ3+01nAb/0dbe5Q/grL9sY9Uah1PPKHDaGQXWvs7lpz8afnL74N9lcVxhw8/i/G6j93WLxZRisfq+K+2/1nJOlzynS1fPeuQJwqlvbePItQ6nn1ngtDMLrDnMPRAoh16z+lDXAsUUqmcDd/BPiOGuAE4WkSeBk4FdQFFEDgNeAyzHCzpvE5GTyl6Lqt6iqutUdd2CBQsmtvRjkM4VyeYd4rHG7Dfw0M9j/NUprWy4P86pZ+TH1eXv2s9luO4LaZYtd7n1X5s4+7Q2TjymjU9/YnhXz89e08yN16ZIp4VP/K8sP/9FP5/7cmZKuyZOlzxrNdXlHG+e/3h9mquuy9DSonz5C0lOO6mNk9Yd/C55pxJh964Ij2xo3B9iM109axY7gRUlj5cDXaUJVLULOAdARFqBc1W1168x/FpVB/znfg68Ge9yVMPa259tyEAx1A3z/p8kWHu0wzfvGuSotS733Vuseolj6HFYuve+v0D3PuGRDTGuuypFsTjyN8LCRS73/fvAgceHHu5WzHM0+x9tuumUZ62mupwTlefFf5M/cKnyC59NjvguZbNet2EblDg16tlmEcNr4H47Xo3hceB9qvpcSZr5QLequiJyA+Co6jUi8h7gb4HT8H5WPAh8WVV/Gra/qW6zcFyXJ1/cR1syPqVdZsuvcf/F6Xnu/3GCvl7hHz6R49LLcsTr9ONstFMvGBPGvkujMxltFnX7GayqReAyYAPwPHC3qj4nIteJyHo/2SnAJhF5AVgEDI09vAfYCvwer13j6UqBohH0pfOoq1MeKMpH/N5+a5JkEn748wE+8sn6BQoIv+4ett2YMPZdajx1HQigqg8AD5Rtu6bk/j14gaH8dQ7wd/Us20R7tS9DMjG1S6PefOPIEb8ArsJRa0cxs9gYXX5lNrCr53iux5vZyb5LjadxR41NI9mCQ99ggY7Wqekuu2unsOH+eGivpZe7Jqe2U4/r8WZ2qud3KZsvUnSVlqZYQ4+FSueKqCrNDVJOCxYToGcwV/NYg7Eqb4u46G9yFAvw4P1xnn3a75IaV4oBf0uTWXWvpaunMbWox3epP5MnFo3Qloqzry9Lc1OMZIPNtJArOKRzBTpavPVv9vVnaU5MfTkb61OahlSVl3vSNCfr1xgQtID75/7JW8D9mDcU+dRVGU49o8hTv4ta1d2YAKpKbzrPnFSC1YvaSMSiLGpv5k97+ukZyNGSik/5CpaO69KfKZCMRzlqWSdzmr2ll4fKuX8gS2tq6lbatGAxToO5IvmCS0sdB+J96Z+D2yIWLXa55/7BA49Xrqq9S6oxs4XjKr2DORZ1pFgxv42o3wmlLRVn7YpOugdy7NjbTzqrtKUSk95JxVVlIFNABFYtbGNua/JAGRupnBYsxmlvX6auYyt++6sou0PaHF59ZeR2uwxkzEGFokt/Js+qhW0sbE+NuPYfEWF+W5KO5gSv9mbY1T1INBKhNTk57QSD2QKFosvizmYWdzaH1hpKy/lKT5qu/Wli0ciktrtYsBiHouOytz9LWyox4XkPDsJNn03y3W83EY0qjjMyjXUjNCZcJl8kV3A4clkHHVXmaotFIyyd28LctiS7ugfZ15chmYjVbeboXMFhMFeksyXBimWtNe8nFo2wbF4r8+ak2Llv4EC7y2SwYDEOfek8qoxpIEwl//eXUa6+opmuXcIlf5vjiNc4XHeVtUUYU6uhhuyjV84d1Qk/GY9y6KI5LGpP8dKefvYP1KHzikIyEeU1yzoOtEuMVjIe5bDF7Sxqb2bH3v5hl63qxYLFOLzam5nQXx79ffD561PcfWeC1WscvvujNMed4FUpEglrizCmmqGG7LZUgjV+Q/ZYtCbjvGZ5J/3ZAq47sTX4iAitqfiE/MhsS8U5anknmXxxwn+0lrNgMUbZfJH+bKFq9baS0u6wnfO8bq8D/cLffDjHx67IkkwdTGttEcZUFtaQPVYiwpw6XGKeaBGRunawGWLBYoz2D+bG1bBU3h22e6+38PxHL89x2f/MTVQxjZkVXFV607nQhmwzfo03Reo04PrrVrSMo2EpaGoOVeGeuxr/l4wxjaYvnWf53BYWdTRboKgTCxZjMJgtki84xMYxOCasO2zYdjM7Oa6SKzhk8kUct/7ze01H2XyRZCLK4k5bFame7DLUGOzty4y54Qxgz6tCNErganHWHXZ2clyl6Ljki+6woBCPRkg1xYiIMJgrUCgWELxVxKIRIRaNkIhFiEZm5+8+11XS+SJHr5g7KT2CZjMLFqM0NLZirF3eXn1FuOi8FkQgkVDy+dnbHdZ1lf5MfsTyibOOQjzmBYW5rU2kEjES8QiJWHTEIC3HdckVXPJFr7YxmCsykPWCyJDmpti4fsxMJ30Z7/JTSx2n2zEeCxaj1DuOsRWvviJc+K4WXtkd4VvfH2R3V2TWdodN57wBU8vnt9DcYBO5TaZ4LEJTLFrzJc1oJEJzU4TmptiwnniO69VK0rkiXd2D9AzkaE7O7KBhl58m1+z9Kx2jPb2ZMY2YfOVl4aJ3tfDKyxG+fucgx7/JAZxZExyGOK5LXzpPW3OCI5a2122E7GwTjURIJSKkEjE6W5voHcyxc9/MDRp2+Wny2V/qKDiuS3+2QPsoL0G9vNu79LTnlQi33jnIujcFzN0xCwxkCxQdl9WL5jCvLVn3QUSzVUSEztYk7S0zN2j0ZfKsmNdql58mkQWLUcjmHRRG1TXv5S7hwvNa2Lsnwje+O8gbj599gaLguAxk8sxtbWLF/Daa4jPjhNXogoLG/oEcLdM8aGT8y0+LOpqnuiizigWLUUjni9QSJkpHZkciEI3Bd34wyBvWNV6gcF0lW3BQrd7MLCIHeuDUUvVXVfr9qZcPW9xOZ2uT9YGfAmFBI1HH2ZKrERFSieiovw+uq2TyRY5eaZefJpsFi1HoTeer/iouH5ntOBCNKTteijRcsFBVegZzLGhPEavl5I+3hGwmX6RQHN7nPxaNEI9GiEWFaCRCvugwkC2wqD3F0rkt0/qX7ExRHjT6M1PXXpYvunQP5EgmoqNqtzpw+WkSprcww1mwqJGq0p8uVG3cDhqZnc8JN9+YbLjG7L50noXtKVYtbBv1LzzHdSkUXfKOS77gkM4XyeSHAkmBeCzCa0pW+zKNYyhodLYmp7Qcg9nCgRXg2lKJqj3CMvkiqUTMLj9NEQsWNcoVXYqOW3V1qukyMjuTL9KUiLJifuuYLg1FIxGiiQhJgNTw5xx/lk67TGAqaUl6M6Z292d5ae8AqoTOxuq6Sjbv8NqVnfa9miKzc9jnGGTzRWppsAgbgd1II7OLjkuu4HDY4vZxTVkSJhoR+4M2NYmIMH9OitcdMo8Fc5L0DubI5EdObdCbzrN8XotdfppCFixq1OcvplLNGWflR2xrpJHZqkpfOs/qRXNsjINpGPFohJUL2njtyrnEoxH2D+QoOF67WCZfpLnJLj9NNQsWNepN52mqoZH2hU1RWttclixzEVGWLnP57E2Zhmmv6EvnWdSRYn7b1F6vNiZIS1Oco5Z1cOjiOeQKRXoHc2TzDqsXjX99CjM+9tOyBkXHJZt36Gip/HFt3xbhF4/EuezyLB+7ovHWpEjniqSaYqyY3zrVRTEmlIgwry1Je3OC3T1pEtHJWdzHVGbBogaZvFPT+Io7vpkgHlfOv2jkpaipVnBcCo7DkUvnztoZSs30EotGWDHPftg0Cjtr1CCdK1TtMTTQDz/8foLT1xdYsLBxGrPBW6ypP5Pn0EVzSFo7hTFmDCxY1KA3na862vXe7ycYHBAu+lDj1Sp6B/Ms7WyZ8n71xpjpy4JFFa4/ZUWiwsht14XvfDPBG44rcsyxjTVKezBboK05ztK5No2zMWbs6hosROQ0EdkkIltE5NMBzx8iIo+IyDMi8piILC95bqWIPCQiz4vIH0RkVT3LGiZXcHBdrThD6i/+I8afXoxyYYPVKvJFB1eVNQvnWE8SY8y41C1YiEgU+CpwOrAWOF9E1pYl+yJwu6oeA1wH3Fjy3O3ATar6GuAE4NV6lbWSbN6pOhjv9lsTLFzscuoZjdE9Frwa0UC2wKGL5tgsr8aYcas5WIhISkSOHEXeJwBbVHWbquaBu4CzytKsBR7x7z869LwfVGKq+jCAqg6oanoU+54wven8iKUtS215IcL//WWcCy7OE2+Q3n2O69IzmGP53BbaS1ZTM8aYsaopWIjIXwNPAQ/6j48VkfuqvGwZsKPk8U5/W6mngXP9+2cDbSIyDzgC6BGRe0XkSRG5ya+pTLq+TOWZZu/4ZoJEk/Ke90/9Jah80aFnMEc6V+SQBW0ssXYKY8wEqbVmcS1eTaEHQFWfAlZVeU3QxZvyPqVXACeLyJPAycAuoIg3/uOt/vPHA2uAS0bsQORSEdkoIhv37NlT41upXb7okC04odN89PbAj+5O8NfvLDB33tR1l83mi3QP5HBdZc2iObx+1TwWdzTbSnTGmAlTa6f7oqr2jnJ20p3AipLHy4Gu0gSq2gWcAyAircC5/n52Ak+q6jb/uR8Dbwa+Ufb6W4BbANatWzfhZ+tsofJgvHvuSpDJCBd+aPJHa6sq6VyRfNGhLZXgqGVttIXM2GmMMeNVa7B4VkTeB0RF5HDgY8B/V3nN48DhIrIar8bwXuB9pQlEZD7QraoucCVwW8lrO0VkgaruAd4GbKyxrBNmMFsIPfk6Dtz5zSaOf3ORtUe7gWnqwXW9hmvXVebOSbK4PWXrEBtj6q7Wy1AfBV4L5IDvAr3AJyq9QFWLwGXABuB54G5VfU5ErhOR9X6yU4BNIvICsAi4wX+tg3cJ6hER+T3eJa2vj+J9TYjewfD2iv94KMbOHREumqRaheMqvYM5+jLegkWvO2Quhy6aY4HCGDMppJa1l6eDdevW6caNE1f5cFzlyW17mdMcD5zq48J3tbDjTxH+/Vf9xOo4g4bjuvRnvBrO0rnNzGtL2hKlxpgJIyJPqOq6aulq7Q31sIh0lDzuFJEN4ylgo8sVHEADA8Uf/xDhN/8d44JLcnULFEXH6/46mC2ycn4rr181jyWdtpa1MWZq1Hqqm6+qPUMPVHW/iCysU5kaQjpXGNF1a8h3bmsimVTOe9/ED8IrFF0GcwXi0QiHLGhjXluTzRJrjJlytQYLV0RWqupL4E3TwchusDNKX6YQ+Cu+e59w371x3vmuPB2dE/cR5IsOg1lvDqrVC9vobE3aFB3GmIZRa7C4CvgvEfmF//gk4NL6FKkx9KXzJBMHg8V998a5+cYkXbsEEFYcMjE9oFSV3kFvX4ctbqejtcm6vxpjGk5NwUJVHxSRN+KNdRDgk6q6t64lm0K5gkPRcYlGvJ5G990b5+pPpchmDp7Ev3JzksVLdNzLpfYM5lnYnmTlgjYLEsaYhjWai+FNQDdet9m1InJSfYo09bIFZ9g1tptvTA4LFADZjHDzjeNbH6IvnaezJWGBwhjT8GqqWYjI54H3AM8BQ9dfFPhlnco1pfoz+WGNyru7gk/kYdtrkc4VaYpHWb1ojgUKY0zDq7XN4p3Akao6+fNaTIG+dIGm+MFgsWSp+m0Vwy1ZOrYG7lzBwXFdjlw2N3TeKWOMaSS1nqm2AbNiqLDjHuy6OuTyK7MkU8MDQzKlXH5ldtT5Fx2XdL7IkUs7SNo6E8aYaaLWmkUaeEpEHsGb8gMAVf1YXUo1hbJ5b1nU0sF4688psL8bbrimGVCWLvMCxWgbt11X6UvnOWJpu03TYYyZVmoNFvf5txlvMFcM3L7mMK9mcccPBznhxNGvs62q9KTzrFzQSmfr+BrGjTFmstXadfbb9S5Iowhb7GjrZu+y1KGHj218RW86z6L2JIs7msdVPmOMmQq19oY6HG997LXAgZ/FqrqmTuWaEqpKbzpPa9PIS0TbtkRo73DHtMhRXzpPe3OCFfPbAueaMsaYRldrA/c3gX/DW8Xuz4Hbge/Uq1BTJVd0cV0lEjDNxtbNUQ49zGW05/p0rkgiFmH1ojk2fYcxZtqqNVikVPURvCnN/6Sq1+ItSDSjZPNFwmZs37o5wppRXoLKF70usocvaR/Wu8oYY6abWhu4syISATaLyGV4K9/NuFln+9J54rGRJ/X93UL3vgiHHl57w7bjugxmi6xd0UkyUccFL4wxZhLU+nP3E0Az3nKqxwEXAhfXq1BTpTcd3Li9bYv3Ma05rPaaRV+6wKqFbbRaF1ljzAxQa2+ox/27A8AH6lecqVNwXLIFh46AWsDBnlC11SwKjksiFmFem3WRNcbMDLX2hlqHN035IaWvUdVj6lSuSTc0GC/I1i1RmpLKsuW19YQazHq1CmvQNsbMFLVeTL8T+BTwew5OJDijDOYKoRP6bd0cYfUal2gNs3MUHJd4LMJcG3hnjJlBag0We1R1Ro/gDmuvANi2JcoxxwaP7C43mC2w2moVxpgZptZg8RkRuRUonxvq3rqUapK5qgxkCrSmRjZGZzOwa4dw9nnVK1SFolersOk8jDEzTa3B4gPAUXgzz5auZzEjgkWu4KCqgZehtm2NoCo1TfMxkC2wZpHVKowxM0+tweL1qvq6upZkChUdJazpettm79JUtZ5QhaJLIm61CmPMzFTrOItfi8jaupakQW3dEiESUVatrlyzGMgWWD63xWoVxpgZqdaaxVuAi0XkRbw2CwF0JnWdDbNtS4TlK12aKlQYrFZhjJnpag0Wp9W1FA1saALBSgazBdYstokCjTEzV9Vg4c8Jdb+qHj0J5WkojgMvbotw0p+Hd5stFF3i8QgdLU2TWDJjjJlcVdssVNUFnhaRlZNQnoay86UIhbyw5rDwxu2BbJ6miH7zAAAYhklEQVQV81qtVmGMmdFqvQy1BHhORH4LDA5tVNX1dSlVg6i2Ol6+6NAUj9LZarUKY8zMVmuw+Ke6lqJBVZtAcDBb4LDF7aHThBhjzExRU9dZVf0F8Eegzb8972+rSEROE5FNIrJFRD4d8PwhIvKIiDwjIo+JyPKy5+eIyC4R+Uptb2dibdsSZcFClzntI5/LFx2SiRgdVqswxswCNQULEXk38FvgPODdwG9E5F1VXhMFvgqcjrd29/kBYzW+CNzud8G9Dm+d71LXA1WDUr1s3RIJXcNi0B9XYbUKY8xsUOugvKuA41X1YlW9CDgB+McqrzkB2KKq21Q1D9wFnFWWZi3efFMAj5Y+LyLHAYuAh2os44RS9bvNBlyCslqFMWa2qTVYRFT11ZLH+2p47TJgR8njnf62Uk8D5/r3zwbaRGSe3133S3jToocSkUtFZKOIbNyzZ0+19zAqe14V+vsksGaRzhatVmGMmVVqDRYPisgGEblERC4B7gceqPKaoDNp+RRMVwAni8iTwMl4a3sXgX8AHlDVHVSgqreo6jpVXbdgwYJa3kfNhpZSLa9Z5IsOTYmo1SqMMbNKxd5QItKkqjlV/ZSInIM37YcAt6jqj6rkvRNYUfJ4OdBVmkBVu4Bz/H21Aueqaq+InAi8VUT+AWgFEiIyoKojGsnrZevQBIJlNYt0rshhi+dYrcIYM6tU6zr7K+CNIvIdVb2Q0U1J/jhwuIisxqsxvBd4X2kCEZkPdPsD/64EbgNQ1QtK0lwCrJvMQAFet9mWVmXRkpHz0balEpNZFGOMmXLVgkVCRC4G/syvWQxTafEjVS2KyGXABiAK3Kaqz4nIdcBGf+W9U4AbRUSBXwIfGeP7mHBbN0dZc5hDaQXCVSUSEWLRWq/eGWPMzFAtWPw9cAHQAfx12XNVFz9S1Qcoa9tQ1WtK7t8D3FMlj28B36pSzgm3bWuEE98yfE4ox1GaYjUsxG2MMTNMxWChqv8lIv8N7FTVGyapTFNuoB9e2T1yjEXRdWltqnXQuzHGzBy1TiR45iSUpWFs3TLUuD28J1TRUZIJCxbGmNmn1ovvD4nIuSKzowtQ2ASCjuvSFLfLUMaY2afWn8mXAy2AIyIZDq6UN6duJZtC27ZEiMeVFYcMDxYCxGPWuG2MmX1qChaq2lbvgjSSrZujrFzlEo8P365A3HpCGWNmoVonEhQReb+I/KP/eIWInFDfok2dbZsjoWtYWLdZY8xsVOuZ71+BEzk4qG4Ab0bZGSefh5f+FBkxzYeqIgIxWxHPGDML1dpm8SZVfaM/hxOqul9EZuQw5pe2R3AcGVGzKLpKMh5jlrTxG2PMMLXWLAr++hQKICILgODrNNPcUE+o8nW3i45LMmE9oYwxs1OtweJfgB8BC0XkBuC/gH+uW6mm0NAEguUD8hxXSVq3WWPMLFVrb6g7ReQJ4O14PUjfqarP17VkU2TblghLl7k0Nw/fXnRcCxbGmFmr2hTlSbz5oQ4Dfg/8f6parPSa6S5sdTyAhM0LZYyZpapdhvo2sA4vUJyOt2b2jOW6Xs0iaHU8wbrNGmNmr2qXodaq6usAROQbwG/rX6Sps7tLyGRG9oQCr2U/FrWeUMaY2anaT+XC0J2ZfvkJYNvQBIIhYyxs9LYxZraqVrN4vYj0+fcFSPmPZ+TcUOETCHrrWNgYC2PMbFVtPYtZ1aK7dXOEjk6XzrnDl1L1ekLZ1OTGmNnLrquU8JZSdSmvQBRdpSlhH5UxZvayM2CJbVuCJxB0XKXZFj0yxsxiFix8PfuF7n0jJxAEr4E7bmMsjDGzmAUL3/atXs0haIwFWLdZY8zsZsHCt31rcLdZ8FfIs26zxphZzM6Avu3bYiSTyrLlOuI5b0CefVTGmNnLzoC+7VujrD7UJVL2iTiuSyIWIWJjLIwxs5gFC9+LW2Mj1rAAKDpK0npCGWNmOQsWQDoNL+8K7jZbdFxSNjW5MWaWs2ABbNkMqhLYuO24VrMwxhgLFsCmTV57RNiAvETMPiZjzOxmZ0Fg0x+FSERZtTpgHQubbdYYYyxY3Hkn/L//J4Lrwjve0sZ998aHPS+IdZs1xsx6dT0LishpIrJJRLaIyKcDnj9ERB4RkWdE5DERWe5vP1ZEfiUiz/nPvace5bvzTrj0UshkBBC6dkW4+lOpYQFDUeJ2GcoYM8vV7SwoIlHgq3jLsa4FzheRtWXJvgjcrqrHANcBN/rb08BFqvpa4DTgyyLSMdFlvOoqrydUqWxGuPnGJOC1V8RtjIUxxtS1ZnECsEVVt6lqHrgLOKsszVrgEf/+o0PPq+oLqrrZv98FvAosmOgCvvRS8PbdXV5wsG6zxhjjqWewWAbsKHm8099W6mngXP/+2UCbiMwrTSAiJwAJYOtEF3DlyuDtS5Z6U344rtJkix4ZY0xdg0XQtZvyiZeuAE4WkSeBk4FdwIG1vkVkCfAd4AOqOqKrkohcKiIbRWTjnj17Rl3AG26A5ubh25Ip5fIrs4Bfs0hYzcIYY+oZLHYCK0oeLwe6ShOoapeqnqOqbwCu8rf1AojIHOB+4GpV/XXQDlT1FlVdp6rrFiwY/VWqCy6AW26BFSsUEWXpMpfP3pRh/TkFAFxVmuwylDHGVF6De5weBw4XkdV4NYb3Au8rTSAi84Fuv9ZwJXCbvz0B/Aiv8fsHdSwjF1wA688p8sdd++loaRr2nGCzzRpjDNSxZqGqReAyYAPwPHC3qj4nIteJyHo/2SnAJhF5AVgE3OBvfzdwEnCJiDzl346tV1nDKDYgzxhjoL41C1T1AeCBsm3XlNy/B7gn4HV3AHfUs2y1spqFMcbYCO5QjqvEoxGiERtjYYwxFixCOK5rjdvGGOOzYBHCW/TIgoUxxoAFi1CO65KydSyMMQawYBHKcZWmmNUsjDEGLFhUZD2hjDHGY2fDCmxqcmOM8djZsIJ41LrNGmMMWLAI5LpKNCJEI/bxGGMMWLAIVHRdkjY1uTHGHGDBIoDjKqmEfTTGGDPEzogBio5L0sZYGGPMARYsAngr5NkYC2OMGWLBIoSNsTDGmIPsjBhAgIQFC2OMOcDOiAEUq1kYY0wpOyOWcVWJiAULY4wpZWfEMo6jNNkYC2OMGcaCRZmi65KydSyMMWYYCxZlio7aOhbGGFPGgkUZW07VGGNGsmBRRrCpyY0xppydFQPErSeUMcYMY2fFMjbGwhhjRrKzYglVRQRiEVv0yBhjSlmwKFF0laZYFBELFsYYU8qCRQnHpiY3xphAFixKFF21AXnGGBPAgkWJouOStDEWxhgzggWLMomYBQtjjClnwaKUWrdZY4wJUtczo4icJiKbRGSLiHw64PlDROQREXlGRB4TkeUlz10sIpv928X1LOfBAkEsaj2hjDGmXN2ChYhEga8CpwNrgfNFZG1Zsi8Ct6vqMcB1wI3+a+cCnwHeBJwAfEZEOutVVgAURGz0tjHGBKnnmfEEYIuqblPVPHAXcFZZmrXAI/79R0uePxV4WFW7VXU/8DBwWh3LimNjLIwxJlQ9g8UyYEfJ453+tlJPA+f6988G2kRkXo2vRUQuFZGNIrJxz5494ypswXVJ2qJHxhgTqJ7BIugnupY9vgI4WUSeBE4GdgHFGl+Lqt6iqutUdd2CBQvGVVjXVZoSdgnKGGOC1POn9E5gRcnj5UBXaQJV7QLOARCRVuBcVe0VkZ3AKWWvfayOZUUEUlazMMaYQPX8Kf04cLiIrBaRBPBe4L7SBCIyX0SGynAlcJt/fwPwDhHp9Bu23+Fvq5uoREjYgDxjjAlUt2ChqkXgMryT/PPA3ar6nIhcJyLr/WSnAJtE5AVgEXCD/9pu4Hq8gPM4cJ2/rW6iUbFus8YYE0JURzQFTEvr1q3TjRs3jum1/ZkCf9y1n9evmmcjuI0xs4qIPKGq66qlsxZdXywasdHbxhgTws6OvmQ8SsTGWBhjTCALFr4ma9w2xphQFizw5oNqS8anuhjGGNOwbGABkErESNkKecYYE8pqFsYYY6qyYGGMMaYqCxbGGGOqsmBhjDGmKgsWxhhjqrJgYYwxpioLFsYYY6qyYGGMMaaqGTPrrIjsAf40jizmA3snMJ3laXlanpZnI+ZZ7hBVrb7UqKrazQuYGycyneVpeVqelmcj5jnWm12GMsYYU5UFC2OMMVVZsDjolglOZ3lanpan5dmIeY7JjGngNsYYUz9WszDGGFNdvVvQG/0G3Aa8CjxbJd0K4FHgeeA54OMh6ZLAb4Gn/XT/VCXfKPAk8LMq6bYDvweeokrPB6ADuAf4o1/eEwPSHOnnNXTrAz5RIc9P+u/nWeB7QDIk3cf9NM+V5xf0WQNzgYeBzf7/nSHpzvPzdIF1VfK8yX/vzwA/8j+PoHTX+2meAh4Cllb7TgBXAIrXVTEoz2uBXSWf619VyhP4KLDJf29fCMnz+yX5bQeeqvDejwV+PfQ9AU4ISfd64Ff+d+qnwJxK3/OA43R0SLphx6lCfkHHKCxt+XFaF5Qu5BiF5Vl+nC4KyzPgGIXlWX6cngtJF3SMwvIccZwIOccAq4Hf+Mfo+0BiQs+VE5nZdLwBJwFvpHqwWAK80b/fBrwArA1IJ0Crfz/uH7w3V8j3cuC71BYs5tf4nr4N/I1/PwF0VEkfBV7G628d9Pwy4EUg5T++G7gkIN3ReIGiGW9hrX8HDq/0Wft/fJ/2738a+HxIutfgBbjHGB4sgtK+A4j59z9fIc85Jfc/Bnyt0nfC/4PegDeeZ35IntcCV9TyPQP+3P+MmvzHC6t9H4EvAddUyPMh4HT//l/5n1dQuseBk/37HwSur/Q9DzhOXwlJN+w4Vcgv6BiFpS0/TrcHpQs5RmF5DjtOFdIFHaOq5wL/OH0hJM+gYxS2/xHHiZBzDN7f5Xv97V8DPlzL+aLW26y/DKWqvwS6a0i3W1V/59/vx/sFsCwgnarqgP8w7t8CG4ZEZDlwBnDr2EofmOccvJPDN/zy5FW1p8rL3g5sVdVKgxpjQEpEYnjBoCsgzWuAX6tqWlWLwC+As4eeDPmsz8ILbvj/vzMonao+r6qbyncYkvYhf//g/YJbHpKur+RhC/5xqvCd+N/A/6oh3QghaT8MfE5Vc36aVyvlKSICvBuvZheWp+L9+gRoB7pC0h0J/NK//zBwrp9n2Pe8/Dj9ZVC68uMUll/IMQpLW36cBiv8LZYfo1r/bsPSBR2jinmWHKevh6QLOkZheY44ThXOMW/Du6IwdIzeWf4+x2PWB4uxEJFVwBvwInrQ81EReQqv6v+wqgamA76M98V2a9itAg+JyBMicmmFdGuAPcA3ReRJEblVRFqq5P1e/BNQ4I5VdwFfBF4CdgO9qvpQQNJngZNEZJ6INOP9alpRZd+LVHW3v5/deL/cJtIHgZ+HPSkiN4jIDuAC4JoK6dYDu1T16Rr2eZmIPCMit4lIZ4V0RwBvFZHfiMgvROT4Kvm+FXhFVTdXSPMJ4Cb/PX0RuDIk3bPAev/+eQQcp7Lveehxqvb3UEO6EceoPG3YcSpNV+0YBew/8DiVpat4jELe04jjVJau4jEqSxt4nMrPMcBWoKckAO8kICiOhwWLURKRVuCHeNfj+4LSqKqjqscCy4ETROTogHzOBF5V1Sdq3PX/UNU3AqcDHxGRk0LSxfAuOfybqr4BGMS7bBD2fhJ4X8YfVEjTiffLcjWwFGgRkfeXp1PV5/EuJzwMPIh3TbVYnm6yiMhV/v7vDEujqlep6go/zWUh+TQDV1EhmJT4N+BQvOvSu/EuR4SJ4bXRvBn4FHC3/6s0zPlUCOq+DwOf9N/TJ/FrmAE+iPc9egLvske+9MlavucTkS7oGAWlDTpOpen8PEKPUUCegccpIF3oMarw3ocdp4B0occoIG3gcSo/x+DV6stNbFfXibymNV1vwCqqtFnoweuDG4DLR5H3Zwi+hn0jXvTfjtdekAbuqDHPa4Py9J9bDGwvefxW4P4KeZ0FPFRlf+cB3yh5fBHwrzWU85+Bf6j0WeM1HC7x7y8BNlU6JpS1WYSlBS7GaxhsruU4A4eUletAWuB1eL/gtvu3Il4ta3GVPMvfa/njB4FTSh5vBRaEvJ8Y8Are5ZpK++jlYJd4AfpqeO9HAL+t9D0POk5B6YKOU1i6kGNU8W9s6DiVp6tyjKrluSoozyrHKOw9DTtOIXmGHaNq5Rx2nEq2fwYvkO3lYDvQicCGan+jo7lZzaJG/q+JbwDPq+rNFdItEJEO/34K+Au8Xh/DqOqVqrpcVVfhXQb6D1Ud8Wvdz6dFRNqG7uM1Dj4blFZVXwZ2iMiR/qa3A3+o8NZq+bX6EvBmEWn2P4e3411TDSrrQv//lcA5NeR9H95JA///n1RJX5WInAb8P8B6VU1XSHd4ycP1BBwnAFX9vaouVNVV/vHaidcY+XJAnktKHp5NyHHy/RjvOjMicgReZ4SwyeD+Avijqu6skB94bUkn+/ffhtczZoSS4xQBrsZrEK30PQ86TrX8PQTmF3SMKqQNOk7D0oUdI7wTd1CeQccp6P2EHaOw937gOFX4LEccowrvfcRxCjnHPI/Xm+pd/ksn5G9pmImMPNPxhncy2w0U8L5gHwpJ9xa8at1QF74D3SLL0h2D1xX2Gbwv4DU1lOEUKvSGwmuHeJqDXeWuqpLfsXhd8p7B+7J3hqRrBvYB7TWU8Z/w/kifBb6D3zskIN1/4gWnp4G3V/usgXnAI3gntUfwumgGpTvbv5/DOwFsqJDnFmBHyXH6Wki6H/rv5xm8bonLavlO4PdMC8nzO3jdHJ/BO8EuqVDOBHCHX4bf4Z04AvcNfAv4+xo+z7cAT/if/2+A40LSfRyvx80LwOc4+Es38HsecJxOD0lXfpx+E5Iu6BiF7bv8OL0zKF3IMQrLs/w4nRWSLugYhZ4LSo9ThX0HHaOwtCOOEyHnGLzzxG/9z/YHhPyNjvVmI7iNMcZUZZehjDHGVGXBwhhjTFUWLIwxxlRlwcIYY0xVFiyMMcZUZcHCNDwRURH5UsnjK0Tk2gnK+1si8q7qKce9n/NE5HkRebRs+yoRyYjIUyW3xBjyXyUi75u4EhsznAULMx3kgHNEZP5UF6SUiERHkfxDeKPZ/zzgua2qemzJLR+QpppVwKiDxSjfg5nFLFiY6aCIt2zkJ8ufKK8ZiMiA//8p/sRvd4vICyLyORG5QER+KyK/F5FDS7L5CxH5Tz/dmf7royJyk4g87k8293cl+T4qIt/FG9RVXp7z/fyfFZHP+9uuwRt09TURuamWN+yP2r/N3/+TInKWv32VX9bf+bc/81/yObwJ754SkU+KyCUi8pWS/H4mIqcMfUYicp2I/AY4UUSO8z+rJ0Rkw9DoZhH5mIj8wX//d9VSbjODTfUIarvZrdoNGMCb0nk73pTOVwDX+s99C3hXaVr//1OAHrx5jJrwFroZWiTm48CXS17/IN4Pp8PxRh8ngUuBq/00TXgj4lf7+Q4CqwPKuRRvapQFeHME/QfelOsQMKeVv30VkOHgqN2v+tv/GXi/f78DbwRvC96o+6S//XD8hbAomwUAuAT4Ssnjn+HPcYQ3Uvjd/v048N/AAv/xe4Db/PtdHFzHoeKaKHab+bdYlVhiTENQ1T4RuR1v8ZtMjS97XP1ptUVkK96iM+DVCEovB92tqi7eHD3bgKPw5t86pqTW0o53cs7jTeb2YsD+jgceU9U9/j7vxFtb5MdVyrlVvRlES70DWC8iV/iPk8BKvBP4V0TkWMDBm1xutBy8KTTAWy/haOBhb3oionhTg4A3ncSdIvLjGt6DmeEsWJjp5Mt48/N8s2RbEf9yqj8ZW2njcK7kvlvy2GX4d798zhvFm4Pno6q6ofQJ/1LOYEj5Kk0vPlqCt9DNsAWf/Ib9V/CW24wA2ZDXH/hcfMmS+1lVdUr285yqnhiQxxl4wW498I8i8lo9uF6CmWWszcJMG6rajbd05IdKNm/Hm4gNvMng4mPI+jwRifjtGGvwpt/eAHxYROLgzTgq1ReR+g1wsojM9xuOz8dbLXAsNgAf9QMgIvIGf3s7sNuvCV2IVxMA6Mdb72DIduBY/32twFvzIMgmYIGInOjvJy4ir/VnOV2hqo/iLdDVAbSO8b2YGcBqFma6+RLDFyn6OvATEfkt3myoYb/6K9mEd1JfhDdjaFZEbsVrT/idf8LeQ5VlKlV1t4hciTdVtAAPqOpYp4m+Hq8m9Yy//+3AmcC/Aj8UkfP8/Qy932eAoog8jdcO82W8ddN/z8EZU4PKnPcvtf2LiLTjnRO+jNdGcoe/TYD/rdWX5zUzmM06a4wxpiq7DGWMMaYqCxbGGGOqsmBhjDGmKgsWxhhjqrJgYYwxpioLFsYYY6qyYGGMMaYqCxbGGGOq+v8BOudn9R2Ltp8AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_sfs(sffs.get_metric_dict(), kind='std_dev');" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'The best quality is 0.9859338612016767 with 25 features in dataset'" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'The best quality is {} with {} features in dataset'.format(sffs.k_score_, len(sffs.k_feature_idx_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The quality is little higher, that SFS one, but SBS is the best algortihm today. Saving results to dict and lets try the last implementation - Sequential Backward Floating Selection" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": true }, "outputs": [], "source": [ "sffs_dict = dict()\n", "for i in sffs.subsets_.values():\n", " sffs_dict[len(i['feature_names'])] = i['avg_score']" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": true }, "outputs": [], "source": [ "logit = LogisticRegression()\n", "sbfs = SequentialFeatureSelector(logit, \n", " k_features=(1, 30), \n", " forward=False, \n", " floating=True, \n", " verbose=2,\n", " scoring='accuracy',\n", " cv=cv)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 30 out of 30 | elapsed: 1.3s finished\n", "\n", "[2018-12-12 00:09:32] Features: 29/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 29 out of 29 | elapsed: 1.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:09:33] Features: 28/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 28 out of 28 | elapsed: 1.1s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:09:35] Features: 27/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 27 out of 27 | elapsed: 1.1s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s finished\n", "\n", "[2018-12-12 00:09:36] Features: 26/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 26 out of 26 | elapsed: 1.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 4 out of 4 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:09:37] Features: 25/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 25 out of 25 | elapsed: 0.9s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:09:38] Features: 24/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.9s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.1s finished\n", "\n", "[2018-12-12 00:09:40] Features: 23/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.8s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 7 out of 7 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:09:41] Features: 22/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 22 out of 22 | elapsed: 0.7s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 8 out of 8 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:09:42] Features: 21/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 21 out of 21 | elapsed: 0.7s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 9 out of 9 | elapsed: 0.2s finished\n", "\n", "[2018-12-12 00:09:43] Features: 20/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.6s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:09:44] Features: 19/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.6s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 11 out of 11 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:09:45] Features: 18/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 18 out of 18 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 12 out of 12 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:09:46] Features: 17/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 17 out of 17 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 13 out of 13 | elapsed: 0.3s finished\n", "\n", "[2018-12-12 00:09:47] Features: 16/1 -- score: 0.985949386741[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 14 out of 14 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:48] Features: 15/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 15 out of 15 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 15 out of 15 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:49] Features: 14/1 -- score: 0.985933861202[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 14 out of 14 | elapsed: 0.3s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 15 out of 15 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:51] Features: 14/1 -- score: 0.987703772706[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 14 out of 14 | elapsed: 0.3s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:51] Features: 13/1 -- score: 0.982425089272[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 13 out of 13 | elapsed: 0.3s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 17 out of 17 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 16 out of 16 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:53] Features: 13/1 -- score: 0.984179475237[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 13 out of 13 | elapsed: 0.3s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 17 out of 17 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:54] Features: 12/1 -- score: 0.982425089272[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 12 out of 12 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 18 out of 18 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:54] Features: 11/1 -- score: 0.984179475237[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 11 out of 11 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:55] Features: 10/1 -- score: 0.982409563732[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:09:57] Features: 10/1 -- score: 0.984163949697[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.4s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 19 out of 19 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:09:58] Features: 10/1 -- score: 0.984163949697[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 10 out of 10 | elapsed: 0.2s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 20 out of 20 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:09:59] Features: 9/1 -- score: 0.984163949697[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 9 out of 9 | elapsed: 0.1s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 21 out of 21 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:00] Features: 8/1 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 8 out of 8 | elapsed: 0.1s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 22 out of 22 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:01] Features: 7/1 -- score: 0.978885266263[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 7 out of 7 | elapsed: 0.1s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:01] Features: 6/1 -- score: 0.977130880298[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:10:03] Features: 6/1 -- score: 0.977130880298[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.4s finished\n", "\n", "[2018-12-12 00:10:03] Features: 5/1 -- score: 0.975376494333[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 25 out of 25 | elapsed: 0.6s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 23 out of 23 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:05] Features: 6/1 -- score: 0.980655177767[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 6 out of 6 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:06] Features: 5/1 -- score: 0.977146405838[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 25 out of 25 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:07] Features: 4/1 -- score: 0.975376494333[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 4 out of 4 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 26 out of 26 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:07] Features: 3/1 -- score: 0.964834653004[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 27 out of 27 | elapsed: 0.5s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 26 out of 26 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:09] Features: 3/1 -- score: 0.968358950474[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 27 out of 27 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:09] Features: 2/1 -- score: 0.954261760596[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.0s finished\n", "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s\n", "[Parallel(n_jobs=1)]: Done 28 out of 28 | elapsed: 0.5s finished\n", "\n", "[2018-12-12 00:10:10] Features: 1/1 -- score: 0.915572116131" ] } ], "source": [ "X_sbfs = sbfs.fit_transform(df_scaled, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ploting results:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmYZGV58P/vXXv13rPvC6ugIuqAS6KMighRUVBck0iWl58m7i8auVBUFHFBX82L0RAliqKIqAlJDANMQOMbFQZlR5RZmKVn6Vl6qe7azjn3749zeqa6+tTSPV29VN2f66qra3nqqafqVJ+7nl1UFWOMMaaayGwXwBhjzNxnwcIYY0xNFiyMMcbUZMHCGGNMTRYsjDHG1GTBwhhjTE0WLIwxxtRkwcIYY0xNFiyMMcbUFJvtAkyXRYsW6bp162a7GMYYM6888MADB1V1ca10DQsWInIj8BrggKo+K+RxAb4C/AkwClyqqr8JHnsH8NEg6adV9du1Xm/dunVs2bJluopvjDEtQUSeriddI5uhvgWcX+XxC4CTg8tlwNcARGQB8HHgBcDZwMdFpLeB5TTGGFNDw4KFqv4cOFwlyeuAm9T3K6BHRJYDrwLuUtXDqnoEuIvqQccYY0yDzWYH90pgV8nt3cF9le43xhgzS2YzWEjIfVrl/okZiFwmIltEZEt/f/+0Fs4YY8wxsxksdgOrS26vAvqq3D+Bqt6gqhtUdcPixTU7840xxkzRbAaL24E/F98LgUFV3QtsAs4Tkd6gY/u84D5jjDGzpJFDZ78PbAQWichu/BFOcQBV/TrwU/xhs0/hD539i+CxwyLyKeD+IKurVbVaR7kxxpgGa1iwUNW31nhcgb+t8NiNwI2NKJcxxpjJa5oZ3GZq8kWXgZE8ERGiUSEigoggQnAdoiX3qYKniuspnqdHrzueh+sqjuvhuB6uV9/rRyKwYkE76YR9FY2Zy+w/tIW5nrJ13yCZvENEBL+y5xsbkjY2PK38LyWPgx9YIhE5GmAE8CfpV+e4Ho/tOsK6JZ0s6kxNw7uafiP5Ijv7M+Qdt670qViUBZ0putJxUscRBAuOy3C2yJFMnky+OOV8WkVEhJ62BD0dSdqTcaKR2t8/Uz8LFi1s75ERMjmH3o7krJUhGY/iuB5b9w6SyRZYvaiDaGRurG/pesr+wVF2HxwhlYiSitf37+K4Hjv7h1GFVCLKoq4UXW0J2hKxmgE0V3AYyhY5OJwjkysi+J9Rva/dylSVQ5k8+weziAi97QkWdqZoT8WJR+fGd2o+s29gixoYydN3eITu9tkLFGNi0Qi9HUkODuUYyTucuLTruH6RT4fRvMP2A0OM5By62xJEJvErNRqJkoxHASg6Hn2H/YATi0ZY2Jkc98vXUyWbdxgcLXBwOEeu6BIBUokYvXPg2MwvQkcQFDxVMjmHw5lBALrScRZ2puhIJ0gFx8ZMjgWLFlRwXLbtH6IjlSBSR1PRTBARutuTjOYdHtl5mJOWddHbMfPNUp4qBwaz7OzPkEpEj7vWFY9FiMcSALiex6FMnn0DWSIRoSsdJ5Mr4rhKNCKkkzF6re9mWkREaEvGaEvGUFUKjsf2A8MApOJROtKJWS5hfaIRSMSixKMRohEhFvz1L5EZbWqzb2aL8VTZvn8YESEem3tV87ZkjHgswu/3DrKi12HFgvYZ+4fIFRy29w8zPFqkqy0x7a8bjUToSB375ZsvurRZ23rDiQjJ+Pja3tBoYZZLVR/VY4NISh3tPxQ/mLQnY5ywtKuufsKpsmDRYvYPjDI4WpjVfopa4tEIPe1J9g2MMpwtcOKy7qP/6I2gqhwYzLHrUIZ40CTWaBGRWW9qa1V+bW/u/VCairFgcjiTZ/3S8LWSpktzfGKmLsPZIjsPZuhum/tV8IgIPe1JCo7HozsPMzCSb8jr5Isuv+8bZGf/MB0pv9nCmPlCRGZsQIgFixZRcFy27hukIxWfVGftbGtPxUknYvy+b4A9hzLjhvceryOZHI/sPMRowaGnIzlnRmEZMxfZz6gWoKrs7B/GUyURm38jQeKxCN3tSfYcHiFXdFm3pPO4Tuyqyr6BUXb2Z+hsS9iwSmPqYMGiBRwYzHI4M7f7KWqJiNDbkeJIJk++6HLS8u4pBT7XU3YdHGb/YI6e9uS8qmUZM5vsJ1WTG8kVebo/Q9c86KeoR3d7krzj8fjuI4zmnUk9t+C4/GHvIP1DOXrbJzd3wphWZ8GiiTmux9b9Q7QlY001PLMjFScqwuO7j9Td8Z0tOPxuzwCjeYee9mRDhxga04wsWDQpVWXnwQxFx2vosNPZkkr4I5ee3DPA/oHRqh3fQ6N5Htvlr3LfmY7PVBGNaSoWLJpU/1CO/sFsU58c41G/43vHgWGeDjrwy/UPZXlizwDpRMxWtjXmOFiwaDK5gsNTewfYfmCI7vZE0ze3RCNCb0eSA4M5nto7SDFYG91TZdfBDNv2DdHdlpzyKLDbfxxn41mdnLqyi41ndXL7jysH38mkne7Xb0Q5mzFPM3UynePWZ9OGDRt0y5Yts12MWVNwXPYPZNl7ZJR4LEJ7svYKp81maLRAIh7hhCVd7B0Y5fBwnu72qa9/dfuP43z0Q2ly2WPPTyaV916e42WvHN+5fs9dMf7+uhT5/LG0qbTy6S9kufDiqS0vXu/rh732ZMrZbHke7+c+Hw2M5Hn+iYun9F0XkQdUdUPNdBYs5jfXUw4OZdl9aATFb5OfK4sDzobRvEOu4BCLRo57BNjGszrp23N8le/lKzx+tmV4Us9xHHjgvijvvLSdkUzrHsvjsWSZxy9+M7nPfT6biWBhjbjzlKoyMJLn6YMZCkWPrra4zUDGX4gwnYged61q7x6hb0+lPJQvfz077p73vzNN2Mo8e/uEd13axiteVeRlr3RYuEi5/cdxvnRtir19wvIVygevyPGKVxX5xb0x7t4U52ebYwwciXBsa6nqr1/ptSdTzmbL88C+CK89t4Nzzyty7vlFTn+2hwihn31YDaTedJNJ24g8Z5LVLOahTK7IroMZhrJFOlKxeTkre65yHLjpmwn+/gspRkch7ES0YqXHvfeP/9VaqRbS3q50dSt7+yKIKGvWeezZHcEpHss3ElFEwHWF7h6Pjec6nPuqIp/5eJq9fRPzLH/9Sq89mXI2W55d3R6nnubxwH1RPE9YvsLjhJNd7v9ljEKhepNVWPNfpaatetM2Is9S1gw1Cc0WLMaWJfYv3tE9rw9n8vQPZW10TwM89NsoV304zROPRdl4bpEXv6TIlz57/P/gr72oyBOPRrh7U5yv/9/kuEAxpr1D+do/j7DhBS6xWO085+PJbTbyPHxIuPfuGHfdEWfzphhhwT8SURYtOXYePHhA8Lza6SaTdjryDAuqYyxYTMJ8DRaup/QPjTKScyi6HkXHo+D6wSGsISIWndud1/OlSl6a59LlyvoTXX71ixiLlyof+1SW8/7EmVSzRb3lPHVlF6oTj52I8uSeoSm/9/nSbDKbeVb67EG55G3H0v/we3EqNYGVpptM2unK8+rP5Xj5eUWWLPXPDKXvffVq+MxnhLe/PeSpVViwmAdyBYet+4cYzRdJxv1Z1hERIsHf+WauVMmnUk5QXrLR4Sv/OEpH55SyrctkmmLM9JqLzWWTyTMaVVzX/86e8VyHFas87rkzPm4kWFsb3HADkwoYFizmuIPDObbvHyIRizbNHgqVvuTpNuVVrz52Yt/0H3GyoxODYTqtnHu+n07Vv/zXnXGy2clVyWt56YZO9tXRF9AIjQh+pj5zubmsrjw/n+UZz3TZvCnO3ZtiPPJg+Hlj7VrYsaPODwUbDTVnOa7H7kMZ9g9m6UoniDXJ8tiOQ8XRQ9lR2PKr2Ljboemyfr/BWKVKxL8vTN8e4fFHI5z2TO9o+mrNEX27hbs3xdm8Kc6+vvBy7q1w/3QaK89cG+nSCur97CdzjGYjz1Oekedd78tXbFbbuXNSH0vdrGYxg0byRbbuG6JQdOlqa57Z1Q8+EOWqv0vzu8fDR2U1okoe7EDMylUeLz+vSLpNuembyQkT2F768iK7d0Z54jG/bCec5HJgn5DJWFOQmd8q/W80qmbRHD9r5zhVZf/AKI/vPILgL7PdDIFicACu+rsUb76wnYEjwjv+OkcqPf7HRyrt/yoq9cEr6ktXLe3HP5PlmutGOfU0l1u/l+CG61NlfRCQzwt3/WecdJvy4Y9m2fTfw9zx8wyf+Gz9r2/MXBX2v9HWBtdc05jXs2aoBis4Ljv7hzmcKdDVlmiKpcJV/Safaz+RYnBAuPSyAu/53zk6OuDZZ3ozWiW/5G1FRkfhzJO6CBtBIgK3/OvIlF/fmLmq/Hs81dFQ9WpoM5SInA98BYgC31DVz5Y9vha4EVgMHAb+VFV3B499Hng1fu3nLuB9WqWwc7EZajhbZOu+QTxVOtMzs/lQo4eZLl6idHYpW/8Q5TnPc/jkZ7Oc/ixvmko/dTbKyLSymZhn0bBmKBGJAl8FLgBOB94qIqeXJbsOuElVzwCuBq4Nnvti4I+AM4BnAWcB5zSqrI0wMJLn8d2HicciMxooPvqhNH17IqgKfXsifPRD6eNahbM8zwP7I2z9Q4Q3vLnAD24fmROBAibXtGWMmbxG9lmcDTylqttUtQDcAryuLM3pwObg+j0ljyuQAhJAEogD+xtY1mnlev7GQx2p+IwuxfGlaye22+eywuc/lZpSfkcOC5/+2MQ8QfjlL2LMpaWoLry4yKe/kGXFSg8RZcVKz4ajGjONGtlnsRLYVXJ7N/CCsjQPAW/Ab6q6COgUkYWq+ksRuQfYi98Qfb2qPtHAsk6rg0M58kWXnvbkjL1m/4HKC98d2B/h1S/r4Nzzi7ziVQ7Pfo5bcXby88922HxHnLs3xbn/V9Gjk4DKzcQw08m68OKiBQdjGqSRwSJ8vvp4lwPXi8ilwM+BPYAjIicBpwGrgnR3ichLVfXn415A5DLgMoA1a9ZMY9GnruC47D6UmbEd6lwXbvlOgi99tnLtoavbY8FC5Ybrk3ztKymWLvc44SSXB359bFG1vj3Ch96TPjpu++RTXf7X3+a57ZYEBw9MrEIsX9EcQ66NMfVpZLDYDawuub0K6CtNoKp9wMUAItIBvEFVB4Mg8CtVzQSP/SfwQvyAUvr8G4AbwO/gbtD7mJT9A1kUZmS58McejnDVR9I88mCMF/2xw0tfVuQr16UmzPy86hq/k/vIYeHezTHuviPO3XfEJkzoURW6uj1u+48R1p3g90WcdIoXOpvU+gKMaS2NDBb3AyeLyHr8GsNbgLeVJhCRRcBhVfWAK/BHRgHsBP6XiFyLX0M5B/hyA8s6LbIFh31HRulqn/4O7dImo2XLlZNOcfl/P4/Ru0D54vWjvOaiIiKweKlWHA3Vu0C56JIiF11S5NSVXaGvMzwkRwMF2DBTgJFckYLjEalz2LN6CuLvrXE8fVZF1yObd3BVEWiKuTmNpKqgkIhHSSWi83J9tbmsYcFCVR0ReTewCX/o7I2q+piIXA1sUdXbgY3AtSKi+LWGvw2efhvwcuAR/KarO1T13xpV1umy51CGWCwy7V/S8rVi9vYJe/uEF73E4f/eMEpX97G09bbbL1+hoX0cYc1LrdoXkC+6jOSLLOxIsmJBB4lYfbXFousxNFrg4FCOgUweREnGY6TitTdlyhddsgXHH+ERj7K8t43utgTJuO1ZUounymje4XAmz+FMHvWUWCxCOhFrivlNs82W+5gmQ9kCT+w+woKOqY08qqYRC9/ZgnaVFV2PTLZAWzLOmsUddB3H0OeC4zKcLXJoOMfgaAEU4sEJLBIRVJVc0SVfdFGgIxVnUWeKrnSclO1XMmWup4zkiwxk8hwczuF4SiwipBOxplmPrZRtqzpPeKo83T9Me3JyndrVJtDt6xM23+n3LTRi4btmbF5SVYquR8HxiEcjxCdZy3M9ZThbIB6NcMKybhZ0JI+7lpiIRVnYGWVhZwrH9RjJOxzO5Dg8nMdTf4e8nrYkqxa005Ge2aHWzSwaEbrSCbrSCVYt6iCbdxgYydM/nCOTLYYPv5mDopEIHam5sX+N1SymwcGhLNv2D9E7iVpF2C/7ZFJ52SuL7NkdObr88LoTXA4eiJDJTO8y3c3A85S841JwvKP9BO3JOB2pGNmiQybr4AXf70QsSjIeCR14oKpkckVcT1m1sJ0l3emGD1DwVMkWHFLxqO2dPoNUlWzBpejOjcmk1agqAyN5DgxmScSitKcq/xi1msU84Lgeuw5l6JhkU0XYBLp8Xrjj3xM853kO//uKHK84v8iJJ3n820/Cm4xabUSS43rkiy5F1wX8DaK60gmW9SRoS8ZIJ8afeD1VcgWX0XyR4VyRwZECRdevOUUjEZLxCI6r5IoOS7vTLO9tn7G+gYjIpGui5viJyLzaP6anPcmS7jb6Dmc4nMmRmsXtlOfPpzZH7R8YxXWVeGpyvw4rNSGJKD/8d1v4rpTnKYOjeZLxGL0dSbrSCdKJKMkaHcaR4MTQloyxqCsN+H0I2YLLcLbA4EiBVCLKScu77MRt5qy2ZIyTlveQyRXZdSjDkUyOtmR8xgc9WLA4DrmiS9+R0Sl1gC5boeytczQStO6IpILjkskWWbO4g2U9bcfddpuIRUnEonS3JVi1cJoKacwM6EjFecaKHoazRZ4+OMxAJk9b6viGZ0+GNZYeh77DI0QjkbrH35c66RR3wn2t2LRUTSZXJF90OW11L8t72+dEJ58xs0lE6GpL8MzVCzhxWRdF12NgJD8jr201iynK5IocHMpOaf2nn/wwzn/fE+ePzimy/aloSzYtVeOpMjji7/9xwtJOGyFkTJmICAs6U3S3JzmcyXFoON/wAV4WLKZAVdnZP0wqMfkhbQ/9NsrHPpzmhX/kcMNNo8StqXycouMxnC2wcmE7Kxa02yxcY6qIRoTFXWkWB31yjWTBYgqOZPJkcg69HZOrVfQfEN79120sWaJ8+esWKMqN5h2KrsspK7onNQzZGNN4FiwmyfU8dh7y96qYjEIe3vPXbQwNCLfcnmHBwsbNbxkbSz5f1sdRVQZHC7QnY5y6ciEpW9rCmDnHgsUk9Q9lKTrupIZaqsInr0zzmy0xvvKPI5z2zMZNCBprxkknY4zkijO2S99UOcE6Sst621i1sMPW8DFmjrJgMUn7BrJ0pCZ3Av7etxP88HsJ3vneHBe81mlQyWA4WwDglBXdpJNxHn76EJ7qjNcuBkfyEzYuqSQqwonLu1nUac1OxsxlFiwmoeh6FIvepGoV9/0yyjVXpdh4bpH3f7gxQ9yKrl+bWNiZYs2ijqOjh5Z2pzk4lKOrbeZqF5lckbZkjFNX9tgyFsY0EQsWk1AounUtQFa6QKAILFjk7znRiHPncLaIqnLysm56O5LjRmct62njwGB2xmoXuYKDCJy0vNsChTFNxv6jJyFbcGrGirEFAvv2RFAVPE8YHhTuuXt6hz45rsfhTJ6udIxnr13Ags7UhGG8yXiUpd1pf5XNBis6HrmiyynLe2xehDFNyILFJIzmnZpr4VdaIPBL105fm3wmV2QkX+TEZV2cuKy76sl5aU8bnqdHV19thLGlvU9e3j2vFmkzxtTP/rMnYShbrPmrudICgcez98QYx/UYzhbpaU+wdnFnXQuJJeNRlvX6zVGN6Lvwh73mWbeka0qz2Y0x84PVLOrkev5y17Fo9ZN+pYUAK91fD89TBkfyZAsOJyzt5OTl3ZNacXJpTxpPFc+b/trF4GiB5b1tLO1p/AxSY8zssWBRp4LjAlpzeY93vjcHZQNHp7pAoKoyNFpgKFtgxcJ2zli7kEVd6UkvMZKI+bWLTG56+y6GRgv0tCVYtbBjWvM1xsw91gxVp3zRpZ4f5rmcAMLiJR4H+6e+QOBIrkjB8VjSnZqWTXmWdqfZd2QUz9MprZJbbjTvkIxHWb+0a17MEjfGHB8LFnWqp3NbFW75ToIzn+9w67+NVE1bSbbgkC04LOxIsmJBx7R1GI/VLvYPZOk+zr4LP3B6PGP5gpqfiTGmOdh/ep2GswUSseof15ZfR9n2VJS3/Glh0vkXHJfDmTyxiHD6qgWctLxn2kcWLe1Oo8fZd1F0PUYLDqcs75nxnbqMMbPHgkUdVJVM3iFeI1h8/6YEnV3KBa+dXJPTwEgB11NOWd7Faat66Uw3ZjnaRCzKiuPou/DGhsgu66q6ebwxpvlYsKhDwfH8tv4qbfOHDwmbfhrn9W8skG6rP+9swaEjFeOZqxfQ2zFxYt10W9w9tZFRrufvyLV2cYctH25MC7JgUYe8M3EL1HI/uTVOsSC85c/qb4JS9Yfjrlk0c6utJmJRVi5oP7roYD1GckWGs0XWLeliafckIqExpmlYsKhDtuBUrVWowg9uTvD8sx1OPrX+5ceHs0WWdKdmvElnUVcKxJ87Uo3reRzJ5EgnYjx7zQKW9kx+2K4xpjlYsKjDSLZYtb/iV/8vyo5t0UnVKtxgCY4VC9qno4iT4vddtJOpUrsYq02sXdzFKSu6SSVs4JwxrcyCRR2GcsWqI6Fu+U6Cnl6P819df8fxcLbA6pLlxGfa4u50aO1iXG1i7UKrTRhjgAYHCxE5X0SeFJGnROQjIY+vFZHNIvKwiNwrIqtKHlsjIneKyBMi8riIrGtkWStxXI+i41Vccvtgv3DXf8a56E1FknX2+xYcl3gsMiObrFcSj0ZYuWB87aK0b+KUFd22vakx5qiGBQsRiQJfBS4ATgfeKiKnlyW7DrhJVc8ArgauLXnsJuALqnoacDZwoFFlrSZfrN65/aNbEjiO8Oa3T67DeO0MdmpXMrZ0SMFxx9UmlnRbbcIYM14jaxZnA0+p6jZVLQC3AK8rS3M6sDm4fs/Y40FQianqXQCqmlHV0QaWtaJsofI2qJ4Ht34vzgte7HDCSfV1bI/mHbrbknNihdZ4NMKKBW2M5B3WL7XahDGmskYGi5XArpLbu4P7Sj0EvCG4fhHQKSILgVOAARH5sYj8VkS+ENRUZtxo3iFeYUmL//l5jF1PR3lznTO2PVXyRYfVizrmzC/3Jd1tnLluIYunsEChMaZ11B0sRCQtIqdOIu+wM0/5WM3LgXNE5LfAOcAewMFfs+olweNnAScAl4aU6TIR2SIiW/r7+ydRtPpV28Pi+99J0LvA47wL6uvYzmSLLO1pm1MbBEUjYjvbGWNqqitYiMhrgQeBO4LbZ4rI7TWethtYXXJ7FdBXmkBV+1T1YlV9LnBlcN9g8NzfBk1YDvAvwPPKX0BVb1DVDaq6YfHixfW8lUmptofF/n3Cf90Z441vKZKoo0XJ9fxmquW9NqnNGDP/1Fuz+AR+H8QAgKo+CKyr8Zz7gZNFZL2IJIC3AOMCjIgsEpGxMlwB3Fjy3F4RGYsALwcer7Os06bguKiG72Hxo1sSuK7wpjo7toezRVYtnL2hssYYczzqDRZO8Iu/bkGN4N3AJuAJ4FZVfUxErhaRC4NkG4EnReT3wFLgmuC5Ln4T1GYReQS/SeufJvP60yFfdEMb01wXbr05wYtfUmTt+tod2/miSyoe9WdOG2PMPFRv4/mjIvI2ICoiJwPvBf6n1pNU9afAT8vuu6rk+m3AbRWeexdwRp3la4jRvBM6v+K/743RtyfCRz6erSufkVyR01b1zvpQWWOMmap6axbvAZ4J5IHvAYPA+xtVqLliuMLM7VtuSrB4iccrXlV5WO2YkVyR3o4kXce54ZAxxsymumoWwRyHK4NLS1BVMrkiHcnxi/zt3SPcuznGZe/OE6+x/p+nStHxWL3S9qg2xsxv9Y6GuktEekpu94rIpsYVa/YVXQ8t2a/69h/H2XhWJ+ec1YnnwYKFtfsqhkcLLFvQRtoW4TPGzHP1NkMtUtWBsRuqegRY0pgizQ25ont0UsjtP47z0Q+l6dsTwe/xFr702TS3/7hy1cJxPSIRYVmPDZU1xsx/9QYLT0TWjN0QkbVMnGDXVLIF5+iQ2S9dmyKXHd85ncsKX7q28uimkbzDigXtFWd/G2PMfFJv+8iVwC9E5GfB7ZcClzWmSHPDSPZY5/bevvBRTJXuB1BP6bZObWNMk6i3g/sOEXke8EL8dpgPqOrBhpZslg3likcX1Vu+QunbMzEwLF8RXrlyXI94PGKL8hljmsZk2kiSwGH8YbOni8hLG1Ok2Ve+h8UHr8iRSo8PDKm08sErcqHPzxYcltjCfMaYJlJXzUJEPge8GXgMGBsGpMDPG1SuWVW+h8WFFxfJjsLHPtwGKCtW+oHiwovDFxB0rAnKGNNk6u2zeD1wqqrmG1mYuSIXsuHRac/yY+Q/3DjKuedXnoznekosIqTn0MqyxhhzvOpthtoG1JiC1jxGcsUJo5h2bPNvrzux+vyKbMFhYWeKiDVBGWOaSL0/f0eBB0VkM/6SHwCo6nsbUqpZlsk5xGMTg0UkoqxZUz1YOI5Hb8fs74JnjDHTqd5gcTtly4s3K0+V0bxDV9v4itSObRFWrNKqe1d4qkhEaE+2TCXMGNMi6h06++1GF2SuKBRdlIl7WOzYHmHd+ol9GaVyBZfe9oStLmuMaTr1rg11sojcJiKPi8i2sUujCzcbwjq3VWHHtijrTqjeBFVwXBZ22p4VxpjmU28H9z8DX8PfH/tlwE3AdxpVqNnk72ExvmZw6KCQGZaqwULVn4fRnrImKGNM86k3WKRVdTMgqvq0qn4Cf6vTpuPvYTF+5vXRkVBVgkW+6NLVFre1oIwxTaneDu5csFf2H0Tk3cAemnDV2Up7WBwLFpX7LHJFl+W9tsKsMaY51fsz+P1AG/52qs8H/gx4R6MKNVvK97AYs2N7hHjcn7ldkUJn2mZtG2OaU72joe4PrmaAv2hccWZX6R4WpXZsi7J6rUeswqdVcFzakjGStnCgMaZJ1bs21Ab8ZcrXlj5HVc9oULlmRa7ghi7+t2NbhPVV+itG8w5rFtnWqcaY5lVvn8XNwIeARzi2kGDTyWQLR/ewGON58PSOCH98TuX1oBTosoUDjTFNrN5g0a+qTT+DO5N3Joxm2tcn5HPCuhPDO7eLrkcqHrW9K4wxTa3eYPFxEfkGUL421I8bUqpZ4Lhtw2qZAAAZ6UlEQVQe+aJLOjH+I9mxPRgJtT68QpXNOyzvbbO9K4wxTa3eYPEXwDPwV54t3c+iaYJF3qncuQ2V51h4tneFMaYF1BssnqOqz25oSWZZvugSVjfYsS1COq0sXTYxlLieEotGbO8KY0zTq3eexa9E5PSGlmSWZbJFYiGzr3dsj7B2vUdYK5O/d0XS9q4wxjS9en8S/zHwDhHZjt9nIYA209DZTM6ZMBIK/JrFac8M79x2XI8e27vCGNMC6q1ZnA+cDJwHvBZ4TfC3KhE5X0SeFJGnROQjIY+vFZHNIvKwiNwrIqvKHu8SkT0icn2d5ZySsT0sykdCFYuw62m/ZjHhOZ4SEdu7whjTGmrWLII1of5DVZ81mYxFJAp8FXglsBu4X0RuV9XHS5JdB9ykqt8WkZcD1+IvJTLmU8DPJvO6U1FpD4s9uyK4bvhqs7miS29H0vauMMa0hJo1C1X1gIdEZM0k8z4beEpVt6lqAbgFeF1ZmtPxh+MC3FP6uIg8H1gK3DnJ1520oquEDYXaHiwgGDZ7u1B0WWBNUMaYFlFvM9Ry4LGgyej2sUuN56wEdpXc3h3cV+oh4A3B9YuAThFZGNRmvog/a3xmhFQQng7mWJQ3Q6kqCHTY3hXGmBZRbwf3J6eQd1j7TPnv98uB60XkUuDn+EufO8DfAD9V1V3VJruJyGXAZQBr1ky24lPbjm0Runs8eheML3au6NLdlggdPWWMMc2o3lVnfyYiS4GzgrvuU9UDNZ62G1hdcnsV0FeWbx9wMYCIdABvUNVBEXkR8BIR+RugA0iISEZVP1L2/BuAGwA2bNhQZf3wqdm+NRo6bDZfdFi5oH26X84YY+asevfgfhNwH3AJ8Cbg1yLyxhpPux84WUTWi0gCeAswrulKRBYFTU4AVwA3Aqjq21V1jaquw6993FQeKGbC09sjoZ3bitCZtiYoY0zrqLcZ6krgrLHahIgsBu4Gbqv0BFV1gl31NgFR4EZVfUxErga2BAsTbgSuFRHFb4b62ym/k2mWy0LfnolLk+eLLh2p+IStV40xppnVGywiZc1Oh6hvJNVPgZ+W3XdVyfXbqBJwgjTfAr5VZzmnzc6nwzu3cwWHNYs7Z7o4xhgzq+oNFneIyCbg+8HtN1MWBJrNsWGz42dvK1gTlDGm5VQNFiKSVNW8qn5IRC7GX/ZDgBtU9SczUsJZsmOr38y0NqTPwpqgjDGtplbN4pfA80TkO6r6ZzTRkuS1PL09wuIlHh0lu6W6nhKPRmzWtjGm5dQKFgkReQfw4qBmMU4zbX5Ubvu2iSOhXM8jaTviGWNaUK1g8U7g7UAPExcObKrNj8o9vT3CxnPH77vtuEpXm+1dYYxpPVXPfKr6CxH5H2C3ql4zQ2WadcNDcLA/MqFz2/U80gmrWRhjWk+9Cwm+ZgbKMmfs2B7eue26SjJmNQtjTOupd3GjO0XkDVJtoaYm8nSl1WYFWw/KGNOS6v2Z/EGgHXBFJMuxnfK6GlayWbR9WwQRZc3aicNm49GWiJfGGDNOvQsJttSU5ae3R1ixUkmmJj5mNQtjTCuqdyFBEZE/FZGPBbdXi8jZjS3a7PGHzY7v3PZUiVgzlDGmRdV75vsH4EXA24LbGfwtU5uOKuzYFp2wJpTrKsm4dW4bY1pTvWe/F6jq80TktwCqeiRYdrzpHDksDA8J608cHywcz6MjacHCGNOa6q1ZFEUkSrDTXbBE+cTe3yawfav/kZTP3nZcJZWwYGGMaU31Bou/B34CLBGRa4BfAJ9pWKlm0Y5g2Oy68mYoW+rDGNPC6h0NdbOIPAC8An/Y7OtV9YmGlmyW7NgeIRZTVq4eHywEiFvntjGmRdVaojyFvz7UScAjwD+qqlPtOfPdjm1RVq/xCJuobSOhjDGtqtbZ79vABvxAcQFwXcNLNMt2VNx3G+IxCxbGmNZUqxnqdFV9NoCIfBO4r/FFmj2e5y/18eKXjK88qSoiELN9LIwxLarWT+Xi2JVmb34C2L9XyOVkQue24ynJWJQWWRrLGGMmqFWzeI6IDAXXBUgHt5tybagd28eGzZYtTe56pGxCnjGmhdXaz6Klxoru2Oa/3dCaRcL6K4wxrcvOgCV2bI+QSilLl+u4+11PSVvNwhjTwixYlNixNcLaEzwiZZ+KqpKwCXnGmBZmwaLEjm2RCU1Q4A+bjdk+FsaYFmbBIuA4sGvnxKXJwf+QbPa2MaaV2RkwsHdPFMeZOGwWxmoW9lEZY1qXnQEDu3YEI6FOmLiAYDwWIWJzLIwxLayhwUJEzheRJ0XkKRH5SMjja0Vks4g8LCL3isiq4P4zReSXIvJY8NibG1lOgJ3b/WAxYR8LV0lb57YxpsU1LFgE+198FX9NqdOBt4rI6WXJrgNuUtUzgKuBa4P7R4E/V9VnAucDXxaRnkaVFWDnjiidXUrvgonDZm0fC2NMq2tkzeJs4ClV3aaqBeAW4HVlaU4HNgfX7xl7XFV/r6p/CK73AQeAxQ0sK7t2xFh3gkt5a5PjeqQSVrMwxrS2RgaLlcCuktu7g/tKPQS8Ibh+EdApIgtLE4jI2UAC2NqgcgJ+zSKsc9tTf10oY4xpZY0MFmE9wlp2+3LgnGBv73OAPcDRBQtFZDnwHeAvVHXCmVxELhORLSKypb+/f8oFzeVgX1/40uSCjYQyxphGngV3A6tLbq8C+koTqGqfql6sqs8FrgzuGwQQkS7gP4CPquqvwl5AVW9Q1Q2qumHx4qm3Um3fBqrC+gr7WFiwMMa0ukaeBe8HThaR9SKSAN4C3F6aQEQWichYGa4AbgzuT+Dv+X2Tqv6wgWUE4Kk/+JWgtSHBAiBus7eNMS2uYcEi2P/i3cAm4AngVlV9TESuFpELg2QbgSdF5PfAUuCa4P43AS8FLhWRB4PLmY0q69atfjBYt3787G3PU6IRIVq+WJQxxrSYho4JVdWfAj8tu++qkuu3AbeFPO+7wHcbWbZST/1BWLDQo7Nsdw7Hs30sjDEGbAY3N98MP7hFOHxI2HhWJ7f/OH70McdV0kkbCWWMMS39s/nmm+GyyyCf95uh+vYIH/1QGoALLy7ieh4pm71tjDGtXbO48koYHR1/Xy4rfOnaFODP3k5asDDGmNYOFjt3ht+/t+/Y6CcbNmuMMS0eLNasCb9/+Qp/7qAACQsWxhjT2sHimmugrW38fam08sErcgCoWs3CGGOgxYPF298ON9wAq1crIsqKlR6f/kKWCy8u4qkSiViwMMYYaPHRUOAHjAsvdvjdniP0tCeP3u+6StLmWBhjDNDiNYtqHM+WJjfGmDEWLCpwPbU5FsYYE7BgUYHj2oQ8Y4wZY8GiEoWEbXpkjDGABYuKRGwklDHGjLGzYQX+pke2j4UxxoAFi1CqigjErWZhjDGABYtQrqckYlFErGZhjDFgwSKUrTZrjDHjWbAI4bgeaZuQZ4wxR1mwCOF4atupGmNMCQsWIVSVRMw+GmOMGWNnxApsJJQxxhxjZ8QQgtiEPGOMKWFnxBCKErdmKGOMOcrOiGVczw8UEZtjYYwxR1mwKOO4HmmbY2GMMeNYsCjjT8izYbPGGFPKgkUZx7Ud8owxppwFizKeKknbx8IYY8ZpaLAQkfNF5EkReUpEPhLy+FoR2SwiD4vIvSKyquSxd4jIH4LLOxpZznI2EsoYY8Zr2FlRRKLAV4ELgNOBt4rI6WXJrgNuUtUzgKuBa4PnLgA+DrwAOBv4uIj0Nqqspfx9LCxYGGNMqUaeFc8GnlLVbapaAG4BXleW5nRgc3D9npLHXwXcpaqHVfUIcBdwfgPLepQAcdv0yBhjxmlksFgJ7Cq5vTu4r9RDwBuC6xcBnSKysM7nTjvPU6IRIRqxmoUxxpRq5Fkx7Oe5lt2+HDhHRH4LnAPsAZw6n4uIXCYiW0RkS39///GWF8fzbLVZY4wJ0chgsRtYXXJ7FdBXmkBV+1T1YlV9LnBlcN9gPc8N0t6gqhtUdcPixYuPu8Cup6TiVqswxphyjTwz3g+cLCLrRSQBvAW4vTSBiCwSkbEyXAHcGFzfBJwnIr1Bx/Z5wX0N5bhKKmE1C2OMKdewYKGqDvBu/JP8E8CtqvqYiFwtIhcGyTYCT4rI74GlwDXBcw8Dn8IPOPcDVwf3NZTr2YQ8Y4wJI6oTugLmpQ0bNuiWLVum9NzhbJHf7TkCwKkre+hKJ6azaMYYM2eJyAOquqFWOmugL+EPm7WPxBhjytmZsYRNyDPGmHB2ZgyoggjEIjYhzxhjylmwCLiev4Cg2KZHxhgzgQWLgL80uQ2bNcaYMBYsAp4qaRs2a4wxoSxYBEQgZdupGmNMKAsWgahEiNumR8YYE8qCRSAWE2K2NLkxxoSyYBGIRsQm5BljTAV2dgxEIxGbkGeMMRXY2TGQikeJ2BwLY4wJZcEikLSRUMYYU5EFCyAWFTpT8dkuhjHGzFk2ZRlIJ2Kkbfa2McZUZDULY4wxNVmwMMYYU5MFC2OMMTVZsDDGGFOTBQtjjDE1WbAwxhhTkwULY4wxNVmwMMYYU5Oo6myXYVqISD/w9HFksQg4OI3pLE/L0/K0POdinuXWqurimqlU1S5+wNwyneksT8vT8rQ852KeU71YM5QxxpiaLFgYY4ypyYLFMTdMczrL0/K0PC3PuZjnlDRNB7cxxpjGsZqFMcaY2hrdgz7XL8CNwAHg0RrpVgP3AE8AjwHvq5AuBdwHPBSk+2SNfKPAb4F/r5FuB/AI8CA1Rj4APcBtwO+C8r4oJM2pQV5jlyHg/VXy/EDwfh4Fvg+kKqR7X5DmsfL8wj5rYAFwF/CH4G9vhXSXBHl6wIYaeX4heO8PAz8JPo+wdJ8K0jwI3AmsqPWdAC4HFH+oYlienwD2lHyuf1ItT+A9wJPBe/t8hTx/UJLfDuDBKu/9TOBXY98T4OwK6Z4D/DL4Tv0b0FXtex5ynJ5VId2441Qlv7BjVClt+XHaEJauwjGqlGf5cfrzSnmGHKNKeZYfp8cqpAs7RpXynHCcqHCOAdYDvw6O0Q+AxLSeK6czs/l4AV4KPI/awWI58Lzgeifwe+D0kHQCdATX48HBe2GVfD8IfI/6gsWiOt/Tt4G/Dq4ngJ4a6aPAPvzx1mGPrwS2A+ng9q3ApSHpnoUfKNrwN9a6Gzi52mcd/PN9JLj+EeBzFdKdhh/g7mV8sAhLex4QC65/rkqeXSXX3wt8vdp3IviH3oQ/n2dRhTw/AVxez/cMeFnwGSWD20tqfR+BLwJXVcnzTuCC4PqfBJ9XWLr7gXOC638JfKra9zzkOF1fId2441Qlv7BjVClt+XG6KSxdhWNUKc9xx6lKurBjVPNcEBynz1fIM+wYVXr9CceJCucY/P/LtwT3fx14Vz3ni3ovLd8Mpao/Bw7XkW6vqv4muD6M/wtgZUg6VdVMcDMeXEI7hkRkFfBq4BtTK31onl34J4dvBuUpqOpAjae9AtiqqtUmNcaAtIjE8INBX0ia04BfqeqoqjrAz4CLxh6s8Fm/Dj+4Efx9fVg6VX1CVZ8sf8EKae8MXh/8X3CrKqQbKrnZTnCcqnwn/g/w4TrSTVAh7buAz6pqPkhzoFqeIiLAm/BrdpXyVPxfnwDdQF+FdKcCPw+u3wW8Iciz0ve8/Di9Mixd+XGqlF+FY1QpbflxGqnyv1h+jOr9v62ULuwYVc2z5Dj9U4V0YceoUp4TjlOVc8zL8VsUxo7R68vf5/Fo+WAxFSKyDngufkQPezwqIg/iV/3vUtXQdMCX8b/YXh0vq8CdIvKAiFxWJd0JQD/wzyLyWxH5hoi018j7LQQnoNAXVt0DXAfsBPYCg6p6Z0jSR4GXishCEWnD/9W0usZrL1XVvcHr7MX/5Tad/hL4z0oPisg1IrILeDtwVZV0FwJ7VPWhOl7z3SLysIjcKCK9VdKdArxERH4tIj8TkbNq5PsSYL+q/qFKmvcDXwje03XAFRXSPQpcGFy/hJDjVPY9r3icav0/1JFuwjEqT1vpOJWmq3WMQl4/9DiVpat6jCq8pwnHqSxd1WNUljb0OJWfY4CtwEBJAN5NSFA8HhYsJklEOoAf4bfHD4WlUVVXVc8EVgFni8izQvJ5DXBAVR+o86X/SFWfB1wA/K2IvLRCuhh+k8PXVPW5wAh+s0Gl95PA/zL+sEqaXvxfluuBFUC7iPxpeTpVfQK/OeEu4A78NlWnPN1MEZErg9e/uVIaVb1SVVcHad5dIZ824EqqBJMSXwNOxG+X3ovfHFFJDL+P5oXAh4Bbg1+llbyVKkE98C7gA8F7+gBBDTPEX+J/jx7Ab/YolD5Yz/d8OtKFHaOwtGHHqTRdkEfFYxSSZ+hxCklX8RhVee/jjlNIuorHKCRt6HEqP8fg1+rLTe9Q1+ls05qvF2AdNfos9Fj74Cbgg5PI++OEt2Ffix/9d+D3F4wC360zz0+E5Rk8tgzYUXL7JcB/VMnrdcCdNV7vEuCbJbf/HPiHOsr5GeBvqn3W+B2Hy4Pry4Enqx0TyvosKqUF3oHfMdhWz3EG1paV62ha4Nn4v+B2BBcHv5a1rEae5e+1/PYdwMaS21uBxRXeTwzYj99cU+01Bjk2JF6AoTre+ynAfdW+52HHKSxd2HGqlK7CMar6PzZ2nMrT1ThGtfJcF5ZnjWNU6T2NO04V8qx0jGqVc9xxKrn/4/iB7CDH+oFeBGyq9T86mYvVLOoU/Jr4JvCEqn6pSrrFItITXE8D5+KP+hhHVa9Q1VWqug6/Gei/VHXCr/Ugn3YR6Ry7jt85+GhYWlXdB+wSkVODu14BPF7lrdXza3Un8EIRaQs+h1fgt6mGlXVJ8HcNcHEded+Of9Ig+PuvNdLXJCLnA38HXKiqo1XSnVxy80JCjhOAqj6iqktUdV1wvHbjd0buC8lzecnNi6hwnAL/gt/OjIicgj8YodJicOcCv1PV3VXyA78v6Zzg+svxR8ZMUHKcIsBH8TtEq33Pw45TPf8PofmFHaMqacOO07h0lY4R/ok7LM+w4xT2fiodo0rv/ehxqvJZTjhGVd77hONU4RzzBP5oqjcGT52W/6VxpjPyzMcL/slsL1DE/4L9VYV0f4xfrRsbwnd0WGRZujPwh8I+jP8FvKqOMmykymgo/H6Ihzg2VO7KGvmdiT8k72H8L3tvhXRtwCGgu44yfhL/n/RR4DsEo0NC0v03fnB6CHhFrc8aWAhsxj+pbcYfohmW7qLgeh7/BLCpSp5PAbtKjtPXK6T7UfB+HsYflriynu8Ewci0Cnl+B3+Y48P4J9jlVcqZAL4blOE3+CeO0NcGvgW8s47P84+BB4LP/9fA8yukex/+iJvfA5/l2C/d0O95yHG6oEK68uP06wrpwo5RpdcuP06vD0tX4RhVyrP8OL2uQrqwY1TxXFB6nKq8dtgxqpR2wnGiwjkG/zxxX/DZ/pAK/6NTvdgMbmOMMTVZM5QxxpiaLFgYY4ypyYKFMcaYmixYGGOMqcmChTHGmJosWJg5T0RURL5YcvtyEfnENOX9LRF5Y+2Ux/06l4jIEyJyT9n960QkKyIPllwSU8h/nYi8bfpKbMx4FizMfJAHLhaRRbNdkFIiEp1E8r/Cn83+spDHtqrqmSWXQkiaWtYBkw4Wk3wPpoVZsDDzgYO/beQHyh8orxmISCb4uzFY+O1WEfm9iHxWRN4uIveJyCMicmJJNueKyH8H6V4TPD8qIl8QkfuDxeb+v5J87xGR7+FP6iovz1uD/B8Vkc8F912FP+nq6yLyhXrecDBr/8bg9X8rIq8L7l8XlPU3weXFwVM+i7/g3YMi8gERuVREri/J799FZOPYZyQiV4vIr4EXicjzg8/qARHZNDa7WUTeKyKPB+//lnrKbZrYbM+gtotdal2ADP6Szjvwl3S+HPhE8Ni3gDeWpg3+bgQG8NcxSuJvdDO2Scz7gC+XPP8O/B9OJ+PPPk4BlwEfDdIk8WfErw/yHQHWh5RzBf7SKIvx1wj6L/wl1yFkTavg/nVAlmOzdr8a3P8Z4E+D6z34M3jb8Wfdp4L7TybYCIuyVQCAS4HrS27/O8EaR/gzhd8UXI8D/wMsDm6/GbgxuN7HsX0cqu6JYpfmv8RqxBJj5gRVHRKRm/A3v8nW+bT7NVhWW0S24m86A36NoLQ56FZV9fDX6NkGPAN//a0zSmot3fgn5wL+Ym7bQ17vLOBeVe0PXvNm/L1F/qVGObeqv4JoqfOAC0Xk8uB2CliDfwK/XkTOBFz8xeUmy8VfQgP8/RKeBdzlL09EFH9pEPCXk7hZRP6ljvdgmpwFCzOffBl/fZ5/LrnPIWhODRZjK+0czpdc90pue4z/7peveaP4a/C8R1U3lT4QNOWMVChfteXFJ0vwN7oZt+FT0LG/H3+7zQiQq/D8o59LIFVyPaeqbsnrPKaqLwrJ49X4we5C4GMi8kw9tl+CaTHWZ2HmDVU9jL915F+V3L0DfyE28BeDi08h60tEJBL0Y5yAv/z2JuBdIhIHf8VRqb2J1K+Bc0RkUdBx/Fb83QKnYhPwniAAIiLPDe7vBvYGNaE/w68JAAzj73cwZgdwZvC+VuPveRDmSWCxiLwoeJ24iDwzWOV0tareg79BVw/QMcX3YpqA1SzMfPNFxm9S9E/Av4rIffiroVb61V/Nk/gn9aX4K4bmROQb+P0JvwlO2P3U2KZSVfeKyBX4S0UL8FNVneoy0Z/Cr0k9HLz+DuA1wD8APxKRS4LXGXu/DwOOiDyE3w/zZfx90x/h2IqpYWUuBE1tfy8i3fjnhC/j95F8N7hPgP+jtbfnNU3MVp01xhhTkzVDGWOMqcmChTHGmJosWBhjjKnJgoUxxpiaLFgYY4ypyYKFMcaYmixYGGOMqcmChTHGmJr+f0Wv/+64cHjnAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plot_sfs(sbfs.get_metric_dict(), kind='std_dev');" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'The best quality is 0.9877037727061015 with 24 features in dataset'" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'The best quality is {} with {} features in dataset'.format(sbfs.k_score_, len(sbfs.k_feature_idx_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The quality of SBS and SBFS algorithms is equal in our example. But sometimes it increased." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": true }, "outputs": [], "source": [ "sbfs_dict = dict()\n", "for i in sbfs.subsets_.values():\n", " sbfs_dict[len(i['feature_names'])] = i['avg_score']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5. Comparing results with RFE and PCA" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Trying another feature selection and dimensional reducing algorithms" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dict_pca = dict()\n", "for i in range(1, 31):\n", " pca = PCA(n_components = i)\n", " df_pca = pca.fit_transform(df_scaled, y)\n", " logit = LogisticRegression()\n", " score = cross_val_score(logit, df_pca, y, cv = cv).mean()\n", " dict_pca[i] = score" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'The best quality is 0.9841794752367644 with 18 features in dataset'" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'The best quality is {} with {} features in dataset'.format(max(dict_pca.values()), max(dict_pca, key=dict_pca.get))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Accuracy metric is lower on PCA dataset" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dict_rfe = dict()\n", "for i in range(1, 31):\n", " rfe = RFE(logit, n_features_to_select=i)\n", " df_rfe = rfe.fit_transform(df_scaled, y)\n", " logit = LogisticRegression()\n", " score = cross_val_score(logit, df_rfe, y, cv = cv).mean()\n", " dict_rfe[i] = score" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'The best quality is 0.9841794752367644 with 24 features in dataset'" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'The best quality is {} with {} features in dataset'.format(max(dict_rfe.values()), max(dict_rfe, key=dict_rfe.get))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "RFE quality is lower too. RFE is computationally less complex using the feature weight coefficients (e.g., linear models) or feature importance (tree-based algorithms) to eliminate features recursively, whereas SFSs eliminate (or add) features based on a user-defined classifier/regression performance metric." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Comparing CV scores of all algorithms" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
RFEPCASBS
10.9208350.9138180.920835
20.9508150.9507530.954262
30.9490450.9472910.963080
40.9455210.9683740.975376
50.9701440.9735910.977146
60.9683740.9753920.978885
70.9683590.9753920.980640
80.9701290.9788850.978901
90.9666050.9824250.978916
100.9683900.9806550.982441
110.9754080.9789010.980655
120.9736530.9753920.980655
130.9718830.9753760.982425
140.9718830.9824100.985934
150.9753760.9824100.987704
160.9771460.9824100.985949
170.9806550.9806550.987704
180.9806550.9841790.987704
190.9824100.9824250.987704
200.9824100.9824250.987704
210.9824100.9841790.987704
220.9824100.9824250.987704
230.9824250.9824250.987704
240.9841790.9841790.987704
250.9841790.9841790.985934
260.9841790.9841790.985934
270.9841790.9841790.985934
280.9841790.9841790.985934
290.9841790.9841790.985934
300.9841790.9841790.984179
\n", "
" ], "text/plain": [ " RFE PCA SBS\n", "1 0.920835 0.913818 0.920835\n", "2 0.950815 0.950753 0.954262\n", "3 0.949045 0.947291 0.963080\n", "4 0.945521 0.968374 0.975376\n", "5 0.970144 0.973591 0.977146\n", "6 0.968374 0.975392 0.978885\n", "7 0.968359 0.975392 0.980640\n", "8 0.970129 0.978885 0.978901\n", "9 0.966605 0.982425 0.978916\n", "10 0.968390 0.980655 0.982441\n", "11 0.975408 0.978901 0.980655\n", "12 0.973653 0.975392 0.980655\n", "13 0.971883 0.975376 0.982425\n", "14 0.971883 0.982410 0.985934\n", "15 0.975376 0.982410 0.987704\n", "16 0.977146 0.982410 0.985949\n", "17 0.980655 0.980655 0.987704\n", "18 0.980655 0.984179 0.987704\n", "19 0.982410 0.982425 0.987704\n", "20 0.982410 0.982425 0.987704\n", "21 0.982410 0.984179 0.987704\n", "22 0.982410 0.982425 0.987704\n", "23 0.982425 0.982425 0.987704\n", "24 0.984179 0.984179 0.987704\n", "25 0.984179 0.984179 0.985934\n", "26 0.984179 0.984179 0.985934\n", "27 0.984179 0.984179 0.985934\n", "28 0.984179 0.984179 0.985934\n", "29 0.984179 0.984179 0.985934\n", "30 0.984179 0.984179 0.984179" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(data = [pd.Series(dict_rfe),pd.Series(dict_pca), pd.Series(sbs_dict)], index = ['RFE', 'PCA', 'SBS']).T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Maximum score we got with SBS and minimum 15 features in subset. RFE is worse with any number of features. PCA is better only with 9 features in subset. RFE and PCA could not find subset of features with score more than full dataset's score. SBS del with it." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

**How we will choose features after this tutorial :)**

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Of course, RFE, PCA and SBS solve slightly different tasks. It's important to know how and when we should implement one or another instrument. And more important is to have an inquiring mind \n", "and test the craziest hypotheses :)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this tutorial we studied something new about feature selection, understood how SequentialFeatureSelector from Mlxtend library works - it allows very easy selection from new generated features and boost model's quality. Then we compared it with another feature selection and dimension reducing algorithms.

\n", "Beginners data scientists often pay a little attention to feature selection and trying to testing many different models instead. But feature selection can boost model score very much. It's near impossible to get top Kaggle without ~~stacking xgboost~~ careful feature engineering and selecting best features.\n", "\n", "Save the best, delete the rest! That's all, folks!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Featured links\n", "Official Mlxtend Docs https://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }