{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Feature Importance Permutation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A function to estimate the feature importance of classifiers and regressors based on *permutation importance*." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> `from mlxtend.evaluate import feature_importance_permutation` " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The *permutation importance* is an intuitive, model-agnostic method to estimate the feature importance for classifier and regression models. The approach is relatively simple and straight-forward:\n", "\n", "1. Take a model that was fit to the training dataset\n", "2. Estimate the predictive performance of the model on an independent dataset (e.g., validation dataset) and record it as the baseline performance\n", "3. For each feature *i*:\n", " - randomly permute feature column *i* in the original dataset\n", " - record the predictive performance of the model on the dataset with the permuted column \n", " - compute the feature importance as the difference between the baseline performance (step 2) and the performance on the permuted dataset\n", "\n", "Permutation importance is generally considered as a relatively efficient technique that works well in practice [1], while a drawback is that the importance of correlated features may be overestimated [2].\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### References\n", "\n", "- [1] Terence Parr, Kerem Turgutlu, Christopher Csiszar, and Jeremy Howard. *Beware Default Random Forest Importances* (http://parrt.cs.usfca.edu/doc/rf-importance/index.html)\n", "- [2] Strobl, C., Boulesteix, A. L., Kneib, T., Augustin, T., & Zeileis, A. (2008). Conditional variable importance for random forests. BMC bioinformatics, 9(1), 307." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example 1 -- Feature Importance for Classifiers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following example illustrates the feature importance estimation via permutation importance based for classification models." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.svm import SVC\n", "from sklearn.model_selection import train_test_split\n", "from mlxtend.evaluate import feature_importance_permutation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Generate a toy dataset" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import make_classification\n", "from sklearn.ensemble import RandomForestClassifier\n", "\n", "# Build a classification task using 3 informative features\n", "X, y = make_classification(n_samples=10000,\n", " n_features=10,\n", " n_informative=3,\n", " n_redundant=0,\n", " n_repeated=0,\n", " n_classes=2,\n", " random_state=0,\n", " shuffle=False)\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.3, random_state=1, stratify=y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Feature importance via random forest" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we compute the feature importance directly from the random forest via *mean impurity decrease* (described after the code section):" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training accuracy: 100.0\n", "Test accuracy: 95.0666666667\n", "[ 0.283357 0.30846795 0.24204291 0.02229767 0.02364941 0.02390578\n", " 0.02501543 0.0234225 0.02370816 0.0241332 ]\n" ] } ], "source": [ "forest = RandomForestClassifier(n_estimators=250,\n", " random_state=0)\n", "\n", "forest.fit(X_train, y_train)\n", "\n", "print('Training accuracy:', np.mean(forest.predict(X_train) == y_train)*100)\n", "print('Test accuracy:', np.mean(forest.predict(X_test) == y_test)*100)\n", "\n", "importance_vals = forest.feature_importances_\n", "print(importance_vals)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are several strategies for computing the feature importance in random forest. The method implemented in scikit-learn (used in the next code example) is based on the Breiman and Friedman's CART (Breiman, Friedman, \"Classification and regression trees\", 1984), the so-called *mean impurity decrease*. Here, the importance value of a features is computed by averaging the impurity decrease for that feature, when splitting a parent node into two child nodes, across all the trees in the ensemble. Note that the impurity decrease values are weighted by the number of samples that are in the respective nodes. This process is repeated for all features in the dataset, and the feature importance values are then normalized so that they sum up to 1. In CART, the authors also note that this fast way of computing feature importance values is relatively consistent with the permutation importance." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, let's visualize the feature importance values from the random forest including a measure of the *mean impurity decrease* variability (here: standard deviation):" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAFvVJREFUeJzt3X20XXV95/H3l4TwECJYSIuQQNCiELUFmoIdR5oRsEGG4EyxE6xWfGh0BqpWO4pPFJmZNqVdFdpFqwgOFoQMYJ1JEYuMinaNCyQ8WAkBDSGQKwLhWXwAIt/5Y/8iO4ebe/ch95yb/O77tdZZOfvs39nf3z57n8/Z93f22YnMRJJUlx0muwOSpIlnuEtShQx3SaqQ4S5JFTLcJalChrskVchwr1xELIyIkcnux7YiInaJiH+KiMci4vLJ7k+/ImJVRCyc7H5o22e4T4KIWBcRP42IJyLivoi4MCJ2m+x+ba2IyIj4cVmvJyLi0SHX7/JBdiLwK8CemfnGrax3RkRcvDXL6Fdmvjwzrx1mzS0p+/HRk90Pjc5wnzzHZ+ZuwCHAocCHJ7k/E+XXM3O3ctuj3ydHxPRBdKplf+B7mblxwHXGNYR1HYjttd9TTmZ6G/INWAcc3Zo+C/hSa/o44GbgcWA9cEZr3jwggbcC9wAPAh9tzd8FuBB4BLgN+K/ASGv+wcC1wKPAKmBxa96FwN8BXwaeAP4fsDdwdlne7cChY6xXAr+6hXl/CKwBHgZWAPv0PO8U4PvAXeWxg4BrSvs7gN9rtX99WbcfAT8A/gSYCfwUeKb0/Yl2jfK8TwBPAU+X+e8oj78dWF3W8Wpg/9Zzzinb4HHgRuA15fFFPcv6zha27RnAxT3b7h1l232zPP4q4Ftlm3wHWNhl3ynLvhy4uLwW3wVeSnOg8EDp9+taz70W+HPg28BjwP8Bfqk1f3HZJx4tbQ/uqfsh4F+BJ4FLy2v907L+HyztLgfuK8v/JvDynv3rXOBLpb/XAy9pzX95a5vfD3ykPL4DcBpwJ/AQcFm73962sK9Mdgem4q3nDTqnvCnPac1fCLyy7NS/Vnb0N5R5mwLiMzRB/uvlzXZwmb8M+Bfgl4C5wK2UcAd2pAnYjwAzgNeWN9nLyvwLaT4sfgPYGfgacBfwB8A04L8DXx9jvUYN91LnQeAwYCfgbynB1nreNaXPu9AE9XrgbcD08rwHNwUF8EOeDdkXAoe1XreRcV77MyhhW6bfUF6Tg0utjwHfas1/M7BnmfeBElw7j7as3m3b26a17f6hrOMuwL4lsF5ftvcxZXp2h33nDOBnwO+U/v1D2V4fLdv6DykflqX9tTQfhq8o9b/Q6ttLgR+X+jsCHyyvy4xW3Vto9qldRlvX8tjbgVllO58N3NKadyFNcB9e+vt5YHmZN6ts1w/Q7HuzgCPKvPcB19G8V3YCPg1cOtnv4239NukdmIq38qZ4giZYE/gqsMcY7c8GPlnubwqIOa353waWlPtrgUWteUt5NtxfU8Jph9b8Syl/GZQ332da8/4IWN2afiXw6Bj9TJoj3EfL7W/K4xcAZ7Xa7UZzxDuv9bzXtub/J+Bfepb9aeBPy/17gHcBL+hps5D+w/3LlCP4Mr0D8BNaR+89z3+EZujpOctqbdvxwv3FrfkfAi7qWcbVwFvH2Hfa4X5Na97xZb+aVqZnlXp7lOlrgWWt9vNp/vqYBnwcuKzndfgB5a+IUvftY63rKH3do9TfvbV/nd+a/3rg9nL/JODmLSxnNXBUa/pFZf+ZPsj36fZ+c8x98rwhM2fRBNJBwF6bZkTEERHx9YjYEBGPAe9uzy/ua93/CU1gAuxDc9S7yd2t+/sA6zPzmZ75+7am72/d/+ko0+N98XtYZu5Rbu9p1f1FPzLzCZqj03bddp/3B46IiEc33YDfpxkiAvhdmmC4OyK+ERG/NU6fxrI/cE6rzsNAbOpbRHwgIlaXs2seBXbnuduiX73r+saedf23NAHWRe/2eTAzf96ahs23We++sSPN+vRuo2dK2y1to+eIiGkRsSwi7oyIx2nCHzZ/vba0386lGXYZzf7AF1uvz2rg5zRfjGsLDPdJlpnfoDmi+avWw5fQjEvPzczdgU/RBE4XP6R5o2yyX+v+vcDciNihZ/4P+ux2v+6leYMCEBEzaYY62nWzdX898I3Wh8Qe2XxB+58BMvOGzDwB+GXgf9OMwfYuo6v1wLt6au2Smd+KiNfQHFn/HvDCbL4gfoxnt8Vo9X4M7Nqa3nuUNr3relFP/ZmZuex5rEsXvfvG0zRDXr3bKErbLW2j0abfBJwAHE3zIThv0+I69Gs98JIx5h3b8xrtnJmD3m+3a4b7tuFs4JiIOKRMzwIezsyfRcThNG+ari4DPhwRL4yIOTRDK5tcTxM+H4yIHcv50scDy7d6DcZ2CfC2iDgkInYC/gy4PjPXbaH9lcBLI+ItpZ87RsRvRsTBETEjIn4/InbPzKdphoE2HaneD+wZEbv30bdP0bxeLweIiN0jYtMpkrOAjcAGYHpEnA68oPXc+4F5PR+WtwBLSp8X0Jx6OZaLgeMj4nfKke/O5ZTOOX2sQz/eHBHzI2JX4EzginKkfxlwXEQcFRE70ox9P0nzRe+W3A+8uDU9qzznIZoPuD/ro19XAntHxPsiYqeImBURR5R5nwL+R0TsDxARsyPihD6WPSUZ7tuAzNxA82XYx8tD/wU4MyJ+BJzOs0emXXyC5s/ru4CvABe16jxFc0bEsTRHa38H/EFm3r616zCWzPwqzbp9geYvi5cAS8Zo/yPgdaXNvTR/yv8FzZdpAG8B1pU//d9N86UnZT0uBdaWP+H36dC3L5ZlLy/Lu5Xm9YFm7PvLwPdoXtOfsfnQxKYfQT0UETeV+x8v6/cIzba4ZJz662mOdj9C8yGynuYMp0G9Ny+i+UvxPpovLt9T+nEHzev4tzT7xvE0p+s+Ncay/hz4WHmt/4RmH76b5mj/NpovQTsp2/yYUvc+mjOn/l2ZfQ7NX7JfKe+J64AjRluOnhXlCwpJlYuIa2m+3D1/svuiwfPIXZIq1CncI2JRRNwREWsi4rRR5p9czuy4pdzeOfFdlSR1Ne6wTERMoxlzPAYYAW4ATsrM21ptTgYWZOapg+uqJKmrLkfuhwNrMnNt+XJlOc0XQJKkbVSXCwDty+ZnCIww+jfVvxsRR9Ic5f9xOQtgMxGxlOYXk8ycOfM3DjrooP57LElT2I033vhgZs4er12XcB/tBwi9Yzn/RHOthycj4t3A52iuJ7L5kzLPA84DWLBgQa5cubJDeUnSJhFx9/itug3LjLD5r9rm0Jx7/AuZ+VBmPlkmP0Nz4SlJ0iTpEu43AAdGxAERMYPmhyUr2g0ion0djMU0136QJE2ScYdlMnNjRJxK82u9acBnM3NVRJwJrMzMFcB7ImIxzU+1HwZOHmCfJUnjmLRfqDrmLkn9i4gbM3PBeO38haokVchwl6QKGe6SVCHDXZIqZLhLUoUMd0mqkOEuSRUy3CWpQoa7JFXIcJekChnuklQhw12SKmS4S1KFDHdJqpDhLkkVMtwlqUKGuyRVyHCXpAoZ7pJUIcNdkipkuEtShQx3SaqQ4S5JFTLcJalChrskVchwl6QKGe6SVCHDXZIqZLhLUoUMd0mqkOEuSRUy3CWpQoa7JFXIcJekChnuklShTuEeEYsi4o6IWBMRp43R7sSIyIhYMHFdlCT1a9xwj4hpwLnAscB84KSImD9Ku1nAe4DrJ7qTkqT+dDlyPxxYk5lrM/MpYDlwwijt/htwFvCzCeyfJOl56BLu+wLrW9Mj5bFfiIhDgbmZeeVYC4qIpRGxMiJWbtiwoe/OSpK66RLuMcpj+YuZETsAnwQ+MN6CMvO8zFyQmQtmz57dvZeSpL50CfcRYG5reg5wb2t6FvAK4NqIWAe8Cljhl6qSNHm6hPsNwIERcUBEzACWACs2zczMxzJzr8ycl5nzgOuAxZm5ciA9liSNa9xwz8yNwKnA1cBq4LLMXBURZ0bE4kF3UJLUv+ldGmXmVcBVPY+dvoW2C7e+W5KkreEvVCWpQoa7JFXIcJekChnuQ7Jw4UIWLlw42d2QNEUY7pJUIcNdkipkuEtShQx3SaqQ4S5JFTLcJalChrskVchwl6QKGe6SVCHDXZIqZLhLUoUMd0mqkOEuSRWacuHu1RklTQVTLtwlaSow3CWpQoa7JFXIcJekChnuklQhw12SKmS4S1KFDPfKeV6/NDUZ7pJUIcNdkipkuEtShQx3SaqQ4S5JFTLcJalChrskVchwl6QKTe/SKCIWAecA04DzM3NZz/x3A6cAPweeAJZm5m0T3NdRzTvtS321v2/tQ30/b92y4/qqIUmTbdwj94iYBpwLHAvMB06KiPk9zS7JzFdm5iHAWcBfT3hPJUmddRmWORxYk5lrM/MpYDlwQrtBZj7empwJ5MR1UZLUry7DMvsC61vTI8ARvY0i4hTg/cAM4LWjLSgilgJLAfbbb79++ypJ6qjLkXuM8thzjswz89zMfAnwIeBjoy0oM8/LzAWZuWD27Nn99VSS1FmXcB8B5ram5wD3jtF+OfCGremUJGnrdAn3G4ADI+KAiJgBLAFWtBtExIGtyeOA709cFyVJ/Rp3zD0zN0bEqcDVNKdCfjYzV0XEmcDKzFwBnBoRRwNPA48Abx1kpyVJY+t0nntmXgVc1fPY6a37753gfkmStoK/UJWkChnuklShTsMyGl0/lzDwsgeShskjd0mqkOEuSRUy3CWpQoa7JFVoyn2huveblo3fSJK2cx65S1KFDHdJqpDhLkkVMtwlqUKGuyRVyHCXpAoZ7pJUIcNdkipkuEtShQx3SaqQ4S5JFTLcJalCU+7CYTUY9P8ABf4vUNL2ziN3SaqQ4S5JFTLcJalChrskVchwl6QKGe6SVCHDXZIqZLhLUoUMd0mqkOEuSRUy3CWpQoa7JFXIcJekChnuklShTuEeEYsi4o6IWBMRp40y//0RcVtE/GtEfDUi9p/4rkqSuho33CNiGnAucCwwHzgpIub3NLsZWJCZvwZcAZw10R2VJHXX5cj9cGBNZq7NzKeA5cAJ7QaZ+fXM/EmZvA6YM7HdlCT1o0u47wusb02PlMe25B3Al0ebERFLI2JlRKzcsGFD915KkvrSJdxjlMdy1IYRbwYWAH852vzMPC8zF2TmgtmzZ3fvpSSpL13+D9URYG5reg5wb2+jiDga+Cjw25n55MR0T5L0fHQ5cr8BODAiDoiIGcASYEW7QUQcCnwaWJyZD0x8NyVJ/Rg33DNzI3AqcDWwGrgsM1dFxJkRsbg0+0tgN+DyiLglIlZsYXGSpCHoMixDZl4FXNXz2Omt+0dPcL+qs/eblk12FyRNIf5CVZIqZLhLUoUMd0mqkOEuSRUy3CWpQoa7JFXIcJekChnuklQhw12SKmS4S1KFDHdJqpDhLkkVMtwlqUKdrgqp7ZdXo5SmJo/cJalChrskVchwl6QKGe6SVCHDXZIqZLhLUoUMd0mqkOEuSRUy3CWpQoa7JFXIcJekChnuklQhw12SKmS4S1KFDHdJqpDhLkkVMtwlqUKGuyRVyHCXpAoZ7pJUoU7hHhGLIuKOiFgTEaeNMv/IiLgpIjZGxIkT301JUj/GDfeImAacCxwLzAdOioj5Pc3uAU4GLpnoDkqS+je9Q5vDgTWZuRYgIpYDJwC3bWqQmevKvGcG0EdJUp+6DMvsC6xvTY+Ux/oWEUsjYmVErNywYcPzWYQkqYMu4R6jPJbPp1hmnpeZCzJzwezZs5/PIiRJHXQJ9xFgbmt6DnDvYLojSZoIXcL9BuDAiDggImYAS4AVg+2WJGlrjBvumbkROBW4GlgNXJaZqyLizIhYDBARvxkRI8AbgU9HxKpBdlqSNLYuZ8uQmVcBV/U8dnrr/g00wzWSpG2Av1CVpAoZ7pJUIcNdkipkuEtShQx3SaqQ4S5JFTLcJalChrskVchwl6QKGe6SVCHDXZIqZLhLUoUMd0mqkOEuSRUy3CWpQoa7JFXIcJekChnuklQhw12SKmS4S1KFDHdJqpDhLkkVMtwlqUKGuyRVyHDXwCxcuJCFCxdOmbrStsRwlybQVPxAm2rrvL0cPBjuklSh6ZPdAW1f5p32pc5t71v7UN/PWbfsuG2q7mTWnsx11vbPcJe0mX4+IKCOD7RB1x2r9qA4LCNJFfLIXQOz95uWTam6k1nbdVYvw12S+rC9fKg4LCNJFTLcJalChrskVahTuEfEooi4IyLWRMRpo8zfKSL+V5l/fUTMm+iOSpK6GzfcI2IacC5wLDAfOCki5vc0ewfwSGb+KvBJ4C8muqOSpO66HLkfDqzJzLWZ+RSwHDihp80JwOfK/SuAoyIiJq6bkqR+RGaO3SDiRGBRZr6zTL8FOCIzT221ubW0GSnTd5Y2D/YsaymwtEy+DLhjolakT3sBD47byrrbc23XeWrUnmp1AfbPzNnjNepynvtoR+C9nwhd2pCZ5wHndag5UBGxMjMXWLfe2q7z1Kg91er2o8uwzAgwtzU9B7h3S20iYjqwO/DwRHRQktS/LuF+A3BgRBwQETOAJcCKnjYrgLeW+ycCX8vxxnskSQMz7rBMZm6MiFOBq4FpwGczc1VEnAmszMwVwAXARRGxhuaIfckgOz0BJmtoaKrVnczarvPUqD3V6nY27heqkqTtj79QlaQKGe6SVKEpE+4R8dmIeKCckz/s2mNevmGAdedGxNcjYnVErIqI9w6x9h4RcUVE3F7q/9YQa783Im4t6/y+IdZdFxHfjYhbImLlsOqW2n9c1vfWiLg0InYeYu1pEXFzRFw5xJovK6/zptvjw9jWEbFzRHw7Ir5TXu9PDLrm85aZU+IGHAkcBtw65LrTgDuBFwMzgO8A84dU+0XAYeX+LOB7Q6z9OeCd5f4MYI8h1X0FcCuwK80JA/8XOHBItdcBew1r32rV3Re4C9ilTF8GnDzE+u8HLgGuHPa6l/rTgPtoftwz6FoB7Fbu7whcD7xqMtZ7vNuUOXLPzG8yOefed7l8w0Bk5g8z86Zy/0fAapogGKiIeAHNh+kFpfZTmfnooOsWBwPXZeZPMnMj8A3gPwyp9mSaDuxSfmeyK8/9LcpARMQc4Djg/GHU24KjgDsz8+5BF8rGE2Vyx3LbJs9KmTLhPon2Bda3pkcYQsD2KlfqPJTmSGPQXgxsAP5n+XP9/IiYOYS60By1HxkRe0bErsDr2fxHeIOUwFci4sZyqY3hFM38AfBXwD3AD4HHMvMrQyp/NvBB4Jkh1RvNEuDSYRUrw1C3AA8A12TmMN5TfTPcB6/TpRkG2oGI3YAvAO/LzMeHUHI6zRDY32fmocCPgaF815CZq2muSnoN8M80w2Abh1EbeHVmHkZzBdVTIuLIYRSNiBfS/DV4ALAPMDMi3jyEuv8eeCAzbxx0rTH6MANYDFw+rJqZ+fPMPITm1/qHR8QrhlW7H4b74HW5fMPARMSONMH++cz8xyGVHQFGWkc0V9CE/VBk5gWZeVhmHkkzFPf9IdW9t/z7APBFmiG5YTgauCszN2Tm08A/Av9mCHVfDSyOiHU0w42vjYiLh1C37Vjgpsy8f8h1KUON1wKLhl27C8N98LpcvmEgymWXLwBWZ+ZfD6MmQGbeB6yPiJeVh44CbhtW/Yj45fLvfsB/ZAh/skfEzIiYtek+8DqaIaJhuAd4VUTsWrb5UTTfrwxUZn44M+dk5jya/fprmTnwvxh6nMRwh2RmR8Qe5f4uNB+stw+rfj+6XBWyChFxKbAQ2CsiRoA/zcwLBl03t3D5hkHXLV4NvAX4bhkjBPhIZl41hNp/BHy+fKCtBd42hJqbfCEi9gSeBk7JzEeGUPNXgC+W/8ZgOnBJZv7zEOqSmddHxBXATTRDUDezHfw8fmuV71SOAd41xLIvAj5X/hOjHYDLMnNop4D2w8sPSFKFHJaRpAoZ7pJUIcNdkipkuEtShQx3SaqQ4S5JFTLcJalC/x8+bZdLyB+i3wAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n", " axis=0)\n", "indices = np.argsort(importance_vals)[::-1]\n", "\n", "# Plot the feature importances of the forest\n", "plt.figure()\n", "plt.title(\"Random Forest feature importance\")\n", "plt.bar(range(X.shape[1]), importance_vals[indices],\n", " yerr=std[indices], align=\"center\")\n", "plt.xticks(range(X.shape[1]), indices)\n", "plt.xlim([-1, X.shape[1]])\n", "plt.ylim([0, 0.5])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, the features 1, 0, and 2 are estimated to be the most informative ones for the random forest classier. Next, let's compute the feature importance via the permutation importance approach." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Permutation Importance" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([ 0.26833333, 0.26733333, 0.261 , -0.002 , -0.00033333,\n", " 0.00066667, 0.00233333, 0.00066667, 0.00066667, -0.00233333])" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "imp_vals, _ = feature_importance_permutation(\n", " predict_method=forest.predict, \n", " X=X_test,\n", " y=y_test,\n", " metric='accuracy',\n", " num_rounds=1,\n", " seed=1)\n", "\n", "imp_vals" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the `feature_importance_permutation` returns two arrays. The first array (here: `imp_vals`) contains the actual importance values we are interested in. If `num_rounds > 1`, the permutation is repeated multiple times (with different random seeds), and in this case the first array contains the average value of the importance computed from the different runs. The second array (here, assigned to `_`, because we are not using it) then contains all individual values from these runs (more about that later)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's also visualize the importance values in a barplot:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEICAYAAACnL3iHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGeBJREFUeJzt3Hu8XWV95/HPj4RwCQEU4i0JARWVqC3QFOxFZBAsAU2cFhWmXvCGOuK9o3hDZGpLrVO1LR1BcbxCClidFKPoS0V0HJCAWIGAxggkXMNdFIHor388z4GVzT4n5wnn7LNJPu/X67zOXms9ez3Pun7XdUdmIklSi62mugGSpEcew0OS1MzwkCQ1MzwkSc0MD0lSM8NDktRsqMIjIg6MiLVT3Y5hERHbRcS/R8SdEXHWVLenVURcHhEHTnU7HokiYreIuDsipk11W4ZRnTdPnITxus6O00bDIyKujoh76sK6MSI+ExE7DKJxkykiMiJ+Vafr7oi4Y8D1jycojwAeC+ySmS96mPWdEBFfeDjjaJWZT8/M8wZZ52jqenzwVLdjvDLz2szcITN/O9VtmWh123tyQ/nzIuI13X513qye6La5zo7feM88XpCZOwB7A/sA7568Jg3U79eVcIfM3Ln1yxExfTIa1TEf+Glmrp/kejZqANM6KR6p7R4Wzr/Be8TM88wc8w+4Gji40/1h4Kud7sOBHwF3AWuAEzrDdgcSeAVwLXAL8N7O8O2AzwC3A1cA/wNY2xm+F3AecAdwObC4M+wzwL8AXwPuBv4f8DjgY3V8VwL7jDFdCTx5lGGvBVYBtwHLgCf0fO+NwM+AX9R+TwO+WctfBby4U/6wOm2/BK4D/gqYCdwD/K62/e5uHfV7HwTuA+6vw19d+78KWFmn8Vxgfuc7H6/L4C7gYuDZtf+hPeP68SjL9gTgCz3L7tV12Z1f+z8L+EFdJj8GDhzPulPHfRbwhTovfgI8hXIgcnNt9/M63z0P+Fvgh8CdwP8FHt0ZvriuE3fUsnv11Psu4D+Ae4Ez6ry+p07/O2u5s4Ab6/jPB57es36dDHy1tvdC4Emd4U/vLPObgPfU/lsBxwE/B24Fzuy2u2f+rASe3+meTtlG9u3M/+l12Ctr+V8Cq4HXjTHfj6ZsD/9Up+1K4Lmd4TsBpwE3UNbJvwam9Xz3o3Xa/rqn3x21/j+u/dfU5feKnmX3mp72fL9+Pr9O16/qsngJ8CjgHGAdZb0+B5hby38I+C3wm1r+n3u33zo9n6vfvwZ4H7BVt27gI3XcvwAWuc5u2jq7wbxqCQ9gbp2BH+8MPxB4Zm3A79VGvbBnB/RJSlD8fp0xe9XhJwHfAx4NzAMuo4YHsDVlB/4eYAZwUJ0hT+3MqFuAPwC2Bb5dV4yXA9MoK/13WsOj1jOyAW9D2QDP7/neN2ubt6MEwRrKxj29fu+WkYVK2UBHduKPAvbtzLe1G5n3J1B35rX7hXWe7FXreh/wg87wlwK71GHvqCvZtv3G1RAen6vTuB0wp65ch9XlfUjtnj3ODfE3wJ/V9n2uLq/31mX9WmoYdzbE64Bn1Pq/1GnbUyg7n0Pqd99Z58uMTr2XUtap7fpNa+33KmBWXc4fAy7t2RBvA/ar7f0isLQOm1WX6zso694sYP867K3ABZRtZRvgFOCMUebP8cAXO92HA1f2zP/pnWFPAgJ4DvBr6ro0SnisB95W589LKDubR9fhX6ntmgk8hrKze13Pd99Up3u7Tr9X8uC2dS1lR7UN8DzKtrnDxsKj37ZHWWf/Ati+zsuzgK+MFkZ9wuNzlB31rDrffsqDB1tHUw6aXlvb/gbgeiBcZ9vX2U0Jj7vrypHAt4Cdxyj/MeCjPRvA3M7wHwJH1s+rgUM7w47hwfB4NmXnt1Vn+BnUM5s6oz7ZGfYmYGWn+5nAHWO0MylH6HfUv3+s/U8DPtwptwNl5du9872DOsNfAnyvZ9ynAB+on68FXgfs2FPmQNrD42vUjaJ2b0XZicwf5fu3Uy7NPWRc/VZO+ofHEzvD3wV8vmcc59I56tzIhvjNzrAX1PVq5Ih3Vq1v586GeFKn/ALK2dM04P3AmT3z4TrqWVCt91VjTWuftu5c69+ps359qjP8MB7csR8F/GiU8axkw6P8x9f1Z3qfsk+mbFfb1+4vAsf3zP+HfK8O/wrwllGGHU3PDpKy3b2Mcg/tXuoOqjM93+l899o+4/tZz7aVwGM7/W4F9u4su3GHR5/27w3c3uneYHzdcdT14V5gQWfY64DzOnWv6gzbvn73ca6z7ets92+89zxemJmzKDu8pwG7jgyIiP0j4jsRsS4i7gRe3x1e3dj5/GvKDhngCZSj9hHXdD4/AViTmb/rGT6n031T5/M9fbo3dmN/38zcuf69uVPvA+3IzLspG0a33m6b5wP7R8QdI3/AX1IuoUE5ojoMuCYivhsRf7SRNo1lPvDxTj23UY5E5wBExDsiYmV9OusOyul877Jo1TutL+qZ1j+lrGzj0bt8bskHbwjfU/93l1nvurE1ZXp6l9HvatnRltFDRMS0iDgpIn4eEXdRNlTYcH6Ntt7Oo5zi9zMf+HJn/qykXHZ5bG/BzFxVh78gIranXNY4fZT2LoqICyLitjrewxh72V6XdU9QXUOZb/Mp8/GGThtPoZyBjOg373qXHZnZur31FRHbR8QpEXFNXRbnAzuP80mzXSlXJrr7jt79xAPLMTN/XT+Ot62us6NoelQ3M79LSbePdHqfTrkvMC8zdwI+QdmhjccNlIkasVvn8/XAvIjYqmf4dS1t3gTXU2YmABExk3Ja3a23u1GuAb7bCaGds9yAfwNAZl6UmUsoG+dXKNcTe8cxXmsolxe6dW2XmT+IiGdTzgxeDDwqywMAd/LgsuhX368oR2IjHtenTO+0fr6n/pmZedImTMt49K4b91MuCfYuo6hlR1tG/br/G7AEOJgSsruPjG4c7VpDuYQ02rBFPfNo28wcbb09g3JUuAS4ogbKBiJiG8olkI9QjvZ3BpZvpK1z6nwZsRtlvq2hHKnv2mnfjpn59E7ZTVk3u8azXnW9A3gq5TLKjsABtf9Y6+6IWyjrxfxOv0HsJ0azJayzwKa95/Ex4JCI2Lt2zwJuy8zfRMR+lAkcrzOBd0fEoyJiLuXS04gLKSvhOyNi6/rs9QuApZvQ5hanA6+MiL3rRvs3wIWZefUo5c8BnhIRL6vt3Doi/jAi9oqIGRHxlxGxU2beT7lMNnLUchOwS0Ts1NC2T1Dm19MBImKniBh5hHcW5br0OmB6RBwP7Nj57k3A7j1hfClwZG3zQsqjwWP5AuUo+c/qUdC29ZHjuQ3T0OKlEbGgHpWfCJxdj/rOBA6PiOdGxNaUnc+9lBv5o7kJ6L4XMKt+51bKju5vGtp1DvC4iHhrRGwTEbMiYv867BPAhyJiPkBEzI6IJWOMaynlnsEbGOWsg3JkvQ1l2a6PiEX1O2N5DPDmumxfRLlPtjwzbwC+AfyviNgxIraKiCdFxHM2Ptnjdinw5/WM4smUhy66+i2Le4A7IuLRwAc2Uv4BnfXhQ3U5zAfeTllXp8KWsM4CmxAembmOcuPo/bXXfwdOjIhfUm4Anjnad/v4IOVU7heUFfrznXruo5zGL6Ik978AL8/MK1vb3CIzv0WZti9RzoyeBBw5RvlfUjbkIylHFzcCf0fZ2KFcZ766nma+nnJTmzodZwCr6+niE8bRti/XcS+t47uMMn+g3Hv4GuVm4TWUG33d0+CRlwxvjYhL6uf31+m7nbIsRtt5jdS/hnLk8x7KjmwN5Qm5yXrZ9POUM90bKTf53lzbcRVlPv4TZd14AeVx8vvGGNffAu+r8/qvKOvwNZQjvysoNwzHpS7zQ2q9N1KevPsvdfDHKWfi36jbxAXA/v3GU8d1A/D/KU8v/esY9b2Zsm3dTjlAW7aRZl4I7EmZPx8CjsjMW+uwl1MC6Yo6vrMZ/6XH8fgo5Vr/TcBnKfdyuk4APluXxYspB6Tb1bZeAHy9p/zHgSMi4vaI+Mc+9b2JcqC5mvJk1enApydmUppt9uvsiNjwsqg0HCLiPMrN+09NdVseaSLiaMoN5j+d6rZsSba0dXaofp5EkvTI0BQeEXFoRFwVEasi4rg+w4+O8tTVpfXvNf3GI0l6ZBv3Zav62NxPKdfN1gIXAUdl5hWdMkcDCzPz2IlvqiRpWLSceexHedlmdb3Js5Ry81SStIVp+QGuOWz49M5a+t+R/4uIOIBylvK2+oTOBiLiGMrb5MycOfMPnva0pzU0Q5J08cUX35KZs6eq/pbw6PciSu81r3+n/CbKvRHxespjegc95EuZpwKnAixcuDBXrFjR0AxJUkRcs/FSk6flstVaNnx7ci7lvYYHZOatmXlv7fwk5UcLJUmbmZbwuAjYMyL2iIgZlJfiNnhRKSK6LxotpvxGiiRpMzPuy1aZuT4ijqW8yTwN+HRmXh4RJwIrMnMZ5ecQFlN+JuM2yi9aSpI2M1P+hrn3PCSpXURcnJkLp6p+3zCXJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUrOm8IiIQyPiqohYFRHHjVHuiIjIiFj48JsoSRo24w6PiJgGnAwsAhYAR0XEgj7lZgFvBi6cqEZKkoZLy5nHfsCqzFydmfcBS4Elfcr9T+DDwG8moH2SpCHUEh5zgDWd7rW13wMiYh9gXmaeM9aIIuKYiFgRESvWrVvX0ARJ0jBoCY/o0y8fGBixFfBR4B0bG1FmnpqZCzNz4ezZsxuaIEkaBi3hsRaY1+meC1zf6Z4FPAM4LyKuBp4FLPOmuSRtflrC4yJgz4jYIyJmAEcCy0YGZuadmblrZu6embsDFwCLM3PFhLZYkjTlxh0embkeOBY4F1gJnJmZl0fEiRGxeLIaKEkaPtNbCmfmcmB5T7/jRyl74KY3S5I0zHzDXJLUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNWsKj4g4NCKuiohVEXFcn+Gvj4ifRMSlEfH9iFgwcU2VJA2LcYdHREwDTgYWAQuAo/qEw+mZ+czM3Bv4MPAPE9ZSSdLQaDnz2A9YlZmrM/M+YCmwpFsgM+/qdM4E8uE3UZI0bKY3lJ0DrOl0rwX27y0UEW8E3g7MAA7qN6KIOAY4BmC33XZraIIkaRi0nHlEn34PObPIzJMz80nAu4D39RtRZp6amQszc+Hs2bMbmiBJGgYt4bEWmNfpngtcP0b5pcALN6VRkqTh1hIeFwF7RsQeETEDOBJY1i0QEXt2Og8HfvbwmyhJGjbjvueRmesj4ljgXGAa8OnMvDwiTgRWZOYy4NiIOBi4H7gdeMVkNFqSNLVabpiTmcuB5T39ju98fssEtUuSNMR8w1yS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUrOmlwTV3+7HfXVSx3/1SYdP6vglqdVmEx6TvQMHd+KSNMLLVpKkZpvNmceWyLMtSVPFMw9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNWsKj4g4NCKuiohVEXFcn+Fvj4grIuI/IuJbETF/4poqSRoW4w6PiJgGnAwsAhYAR0XEgp5iPwIWZubvAWcDH56ohkqShkfLmcd+wKrMXJ2Z9wFLgSXdApn5ncz8de28AJg7Mc2UJA2TlvCYA6zpdK+t/UbzauBr/QZExDERsSIiVqxbt66hCZKkYdASHtGnX/YtGPFSYCHw9/2GZ+apmbkwMxfOnj27oQmSpGEwvaHsWmBep3sucH1voYg4GHgv8JzMvPfhNU+SNIxazjwuAvaMiD0iYgZwJLCsWyAi9gFOARZn5s0T10xJ0jAZd3hk5nrgWOBcYCVwZmZeHhEnRsTiWuzvgR2AsyLi0ohYNsroJEmPYC2XrcjM5cDynn7Hdz4fPEHtkiQNMd8wlyQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzZrCIyIOjYirImJVRBzXZ/gBEXFJRKyPiCMmrpmSpGEy7vCIiGnAycAiYAFwVEQs6Cl2LXA0cPpENVCSNHymN5TdD1iVmasBImIpsAS4YqRAZl5dh/1uAtsoSRoyLZet5gBrOt1ra79mEXFMRKyIiBXr1q3blFFIkqZQS3hEn365KZVm5qmZuTAzF86ePXtTRiFJmkIt4bEWmNfpngtcP7HNkSQ9ErSEx0XAnhGxR0TMAI4Elk1OsyRJw2zc4ZGZ64FjgXOBlcCZmXl5RJwYEYsBIuIPI2It8CLglIi4fDIaLUmaWi1PW5GZy4HlPf2O73y+iHI5S5K0GfMNc0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1awqPiDg0Iq6KiFURcVyf4dtExL/W4RdGxO4T1VBJ0vAYd3hExDTgZGARsAA4KiIW9BR7NXB7Zj4Z+CjwdxPVUEnS8JjeUHY/YFVmrgaIiKXAEuCKTpklwAn189nAP0dEZGZOQFs1RHY/7quTOv6rTzp8Uscv6eFpCY85wJpO91pg/9HKZOb6iLgT2AW4pVsoIo4Bjqmdd0fEVS2NnkC70tO2scTEnUc9Iuqdyrqncpon0FTV7TRvGXXPn4I6H9ASHtGnX+8ZxXjKkJmnAqc21D0pImJFZi603s23bqd5y6h7S5zmqdZyw3wtMK/TPRe4frQyETEd2Am47eE0UJI0fFrC4yJgz4jYIyJmAEcCy3rKLANeUT8fAXzb+x2StPkZ92Wreg/jWOBcYBrw6cy8PCJOBFZk5jLgNODzEbGKcsZx5GQ0egJN1aWzLa3eqazbad4y6t4Sp3lKhScGkqRWvmEuSWpmeEiSmm2R4RERn46ImyPisimoe8yfeJmkOudFxHciYmVEXB4RbxlEvbXunSPi7Ii4stb/RwOs+y0RcVmd5rcOsN6rI+InEXFpRKwYVL217rfV6b0sIs6IiG0HWPe0iPhRRJwzwDqfWufzyN9dg1jWEbFtRPwwIn5c5/cHJ7vOYbNF3vOIiAOAu4HPZeYzBljvNOCnwCGUx5ovAo7KzCvG/OLDr/fxwOMz85KImAVcDLxwsuutdX8W+F5mfqo+pbd9Zt4xgHqfASyl/DLCfcDXgTdk5s8GUPfVwMLMHOiLYxExB/g+sCAz74mIM4HlmfmZAdX/dmAhsGNmPn8QdfbUPw24Dtg/M6+Z5LoCmJmZd0fE1pT5/pbMvGAy6x0mW+SZR2aez9S8f/LAT7xk5n2UnduSya40M2/IzEvq518CKym/BjCpImJH4ADKU3hk5n2DCI5qL+CCzPx1Zq4Hvgv81wHVPZWmA9vV96y256HvYk2KiJgLHA58ahD1jeK5wM8nOzgAsri7dm5d/7aoI/EtMjymUL+feJn0nXhX/aXjfYALB1DdE4F1wP+plzM+FREzB1AvwGXAARGxS0RsDxzGhi+5TqYEvhERF9ef4hlMpZnXAR8BrgVuAO7MzG8MqPqPAe8Efjeg+vo5EjhjUJXVy3SXAjcD38zMQWxTQ8PwGKxx/XzLpFUesQPwJeCtmXnXAKqcDuwL/O/M3Af4FTCQ+zyZuZLyq87fpFyy+jGwfhB1A3+SmftSfoH6jfUy6aSLiEdRzmT3AJ4AzIyIlw6g3ucDN2fmxZNd1xhtmAEsBs4aVJ2Z+dvM3Jvyaxv71UulWwzDY7DG8xMvk6Jel/0S8MXM/LdB1EmZ3rWdI7KzKWEyEJl5Wmbum5kHUC5TTvr9jlrv9fX/zcCXKZcrB+Fg4BeZuS4z7wf+DfjjAdT7J8Dieq9nKXBQRHxhAPV2LQIuycybBlwv9VLsecChg657KhkegzWen3iZcPXm3mnAysz8h8mub0Rm3gisiYin1l7PZcOf8J9UEfGY+n834M8ZwCWNiJhZH0qgXqJ7HuUS2iBcCzwrIravy/y5lPtbkyoz352ZczNzd8o6/e3MnPQznh5HMdhLVrMjYuf6eTtKcF85qPqHQcuv6m42IuIM4EBg14hYC3wgM0+b7HpH+4mXya6XcmT4MuAn9RotwHsyc/kA6n4T8MUalquBVw6gzhFfiohdgPuBN2bm7QOo87HAl8u+m+nA6Zn59QHUS2ZeGBFnA5dQLtH9iC3gpzPqPa1DgNcNsNrHA5+tT3htBZyZmQN7RHkYbJGP6kqSHh4vW0mSmhkekqRmhockqZnhIUlqZnhIkpoZHpKkZoaHJKnZfwIyRimttlsZ0AAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "indices = np.argsort(imp_vals)[::-1]\n", "plt.figure()\n", "plt.title(\"Random Forest feature importance via permutation importance\")\n", "plt.bar(range(X.shape[1]), imp_vals[indices])\n", "plt.xticks(range(X.shape[1]), indices)\n", "plt.xlim([-1, X.shape[1]])\n", "plt.ylim([0, 0.5])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, also here, features 1, 0, and 2 are predicted to be the most important ones, which is consistent with the feature importance values that we computed via the *mean impurity decrease* method earlier." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(Note that in the context of random forests, the feature importance via permutation importance is typically computed using the out-of-bag samples of a random forest, whereas in this implementation, an independent dataset is used.)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Previously, it was mentioned that the permutation is repeated multiple times if `num_rounds > 1`. In this case, the second array returned by the `feature_importance_permutation` contains the importance values for these individual runs (the array has shape [num_features, num_rounds), which we can use to compute some sort of variability between these runs. " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdgAAAEICAYAAAD85+W2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAH4FJREFUeJzt3Xu8VXWd//HXWxAFRUE5qXH1QiVWo84JZ7oYv7yEWuI0muhYWpblL7tMNY1dRo2pGWv6/dJpbNJJKjUl1Kn4GWb+SrT5ORigVgKSSAhHvKDgLUlDP78/vt+ji+0+h32Q795nc97Px+M82Gt911rf71rru9d73c5BEYGZmZltXdu1ugFmZmbbIgesmZlZAQ5YMzOzAhywZmZmBThgzczMCnDAmpmZFdCvAlbSFEldrW5HfyFpqKT/I+lxSVe3uj19JWmxpCmtbkc7kjRO0lOSBrW6Lf1R3jb7FFiu+2w/J+kDkuZt4bz7SWra76ZuNmAlrZS0IXfoByV9V9LOzWhcSZJC0h/yej0l6bEm19/IycTxwB7A7hFxwsus7zxJV7ycZfRVRBwQEfOaWWdPcj8+vNXtaFRErIqInSPiuVa3ZWvL3739+jD9PEkfqI7L22bF1m6b+2xrvZzw7I8avYJ9Z0TsDBwIHAR8tlyTmurP8hd154gY0deZJQ0u0aiK8cDvImJj4Xo2qwnrWkS7tru/8PZrPm/zbUhE9PoDrAQOrwx/FfhJZfgY4A7gCWA1cF6lbAIQwKnAKuAR4POV8qHAd4H1wBLg74CuSvn+wDzgMWAxcGyl7LvAN4HrgaeA/wfsCVyQl3c3cFAv6xXAfj2UfRBYDqwD5gCvrJnvI8A9wO/zuNcAN+bplwHvrkx/dF63J4H7gU8DOwEbgOdz25+q1pHn+yLwLPCnXH56Hv9+YGlexxuA8ZV5Lsz74AlgEfCWPH5qzbJ+3cO+PQ+4ombfnZ733S15/F8At+Z98mtgSiN9Jy/7auCKvC1+C7yKdLL2cG73kZV55wH/DPwKeBz4MbBbpfzY3Ccey9PuX1Pv3wO/AZ4BrsrbekNe/8/k6a4GHszLvwU4oKZ/XQT8JLf3NmDfSvkBlX3+EPC5PH474GzgXuBRYHa13TXbZynwjsrwYNJ35ODK9h+cy96Xp38SWAF8qJftfhrp+/CNvG53A4dVyncFLgUeIPXJLwGDaub9el63L9WMeyzX/8Y8fnXef6fW7LsP1LTnv/LnW/J6/SHvixOBkcB1wFpSv74OGJOn/zLwHPDHPP2/1X5/8/pclue/D/gCsF21buBredm/B45yn93iPnsz8Nf585vzfjg6Dx8O3Lm5TMnTnp7Xubs/Twdel/fzc3mdH8nTduQ+8QQwP/eJeQ3WM4jUbx/N63cWEJXyEcB3SN+FLmBG3h5Dc32vqUy7Z94fuzdSd0T0LWCBMbmTXVgpn5I3zHbA6/OOO67mIP0fucF/ljvP/rn8fOCXwG7AWOAucsAC25NC7nPAEOBteWe8utKZHgH+HNgR+AXpy/PevFG/BNzUy3rVDdhcT/dBbgfSQeqWmvluzG0eSgrL1aQD4OA83yPkjp93XHfQjQQOrmy3rs1s+/PIgZeHj8vbZP9c1xeAWyvlpwC757JPkb6IO9ZbVu2+rZ2msu8uy+s4FBhN6qhH5/19RB7uaPBg9Ufg7bl9l+X99fm8rz9IPmGpHKzuB16b67+20rZXkQ7QR+R5P5O3y5BKvXeS+tTQeuuax70fGJ738wVUDg6k/rUOmJzb+31gVi4bnvfrp0h9bzhwSC77BOkgMCYv92Lgqh62zznA9yvDxwB312z/wZWyfQEBbwWeJvelOss9DdgI/G3ePieSDsi75fIf5XbtBLyCFAgfqpn3o3m9h1bGvY8Xv1urSAfzHYAjSd/NnSv7rm7A1vvukfrsXwPD8ra8GvhRTV/4QM06VgP2MlKYDc/b7Xe8eEJ6GunE8oO57WcCawC5z25Rn50BfCN//hwptL5SKbuw3nw1y9iF1B8n5uG9gEn58weoCU/gGtIJxzBSxjxQO00vdZ1FOqkZQ+pnt7BpwF5HulAbRgrQRZW+cxnwxcq0Hweua6TeF+ZpoIErSWcTT5I69c+BEb1MfwHw9ZqDxJhK+a+A6fnzCmBqpewMXgzYt5ACYrtK+VXkK+Tcmf6jUvZRYGll+HXAY720M0hnKI/ln3/N4y8FvlqZbmfSF3RCZb63VcpPBH5Zs+yLgXPz51XAh4BdaqaZQt8D9vrunZ+HtyMdaMf3MP960m3wlyyr3heY+gG7T6X874HLa5ZxA5Wrl56Wn5d9Y6XsnblfdV85Dc/1jcjD84DzK9NPIl2FDwL+AZhdsx3uJ19N53rf39u61mnriFz/rpX+9e1K+dG8GH4nAXf0sJylbHq1uFfuP4PrTLsf6Xs1LA9/HzinZvu/ZL5c/iPg4z2UnUZNiJC+d+8hPdN/hnwQr6zPTZV5V9VZ3j01360A9qiMexQ4sLLvGg7YOu0/EFhfGd5kedVl5P7wDPkAncs+RD4A57qXV8qG5Xn3dJ/doj57GPCb/PmnpECcn4dvBt7VU3sry9iFdMz9K/IFQKVsk4AlnYxsZNMTsq/SeMDeUtMXjyYHLOmCYQOwQ6X8Pd37nHTn73eVstuAkxupt/un0Wewx0XEcFIovAYY1V0g6RBJN0laK+lx4MPV8uzByuenSaEF8ErS1V+3+yqfXwmsjojna8pHV4YfqnzeUGd4cy9jHRwRI/LPxyr1vtCOiHiKdPCo1ltt83jgEEmPdf8Af0M6G4J0Zn40cJ+kmyX95Wba1JvxwIWVetaRrmhGA0j6lKSl+a3jx0i3zmr3RV/VrusJNev6ZtIXshG1++eRePElng353+o+q+0b25PWp3YfPZ+n7WkfvYSkQZLOl3SvpCdIBzPYdHv11G/Hks7c6xkP/LCyfZaSbnntUTthRCzP5e+UNIx0C/HKHtp7lKT5ktbl5R5N7/v2/u6jSHYfabuNJ23HByptvJh0Jdut3rar3XdERF+/b3VJGibpYkn35X1xCzCiwTeoR5HucFWPHbXHiRf2Y0Q8nT822lb32U39N/AqSXuQToQuA8ZKGkW6cr6lt3UAiIgnSIH/EeBBSddJelUPk+9BOkHpKSc2p7eMGU+6Yn+osu4X8eJ6/19SP/xzSfuSbrH/uA919+3XdCLiZtJZ0tcqo68kPaccGxG7At8iHfQb8QBpx3cbV/m8hrTjtqspv78vbd4Ca0gbHgBJO5FuLVTrrR64VgM3V4J6RKSXps4EiIgFETGNdAD7Een5Ru0yGrWadCuvWtfQiLhV0ltIV5jvBkZGemnrcV7cF/Xq+wPpjL7bnnWmqV3Xy2vq3ykizt+CdWlEbd/4E+n2e+0+Up62p31Ub/hkYBrpudGupCtGaKzvribdru2p7KiabbRjRPTUb68iHWymAUty6G5C0g6k241fI101jgDmbqato/N26TaOtN1Wk674RlXat0tEHFCZdkv6ZlUj/arqU8CrSbcsdwEOzeN767vdHiH1i/GVcc04TvRkm+6z+QRlEel26V0R8SzpnYxPAvdGxCMNtIWIuD4iDiednC8nneTBS9f5IdKz6J5yYnN6y5jVpJOQ3Wq+C6/PbdxIelxxEmnb/zgi/tCHurfo92AvAI6QdGAeHg6si4g/SpqcG9Ko2cBnJY2UNIZ0m7fbbaQv6mckbZ9/N+2dwKwtaHNfXAm8T9KB+cD2T8BtEbGyh+mvI53RvSe3c3tJb5C0v6Qhkv5G0q4R8SfSLenus9+HgN0l7dqHtn2LtL0OAJC0q6TuX98ZTrqVshYYLOkc0q2Ybg8BE2pOWO4Epuc2d5J+Lag3V5Cutt6ez6Z3zL9uNKYP69AXp0ialK/uZgDX5KuH2cAxkg6TtD3pAP0M6Yvek4eA6u9NDs/zPEoKg3/qQ7uuA/aU9AlJO0gaLumQXPYt4MuSxgNI6pA0rZdlzSI9wzyTHq5eSVdoO5D27UZJR+V5evMK4GN5355Aem4/NyIeAH4G/C9Ju0jaTtK+kt66+dVu2J3Au/KV6X6kF1qq6u2LDcBjknYDzt3M9C+o9Icv5/0wnnSwb+qvpFUMhD57M+nZ5s15eF7NcK8k7SWp+67Ns6TjfPW4OCZvI/Jx80fAF5X+LsBrSbdxGzUb+ISk0ZJ2J12EkJe9Orf5a5Xvwn6SDq3MfyXpMeDJ9Pz97FGfAzYi1pJuC/xDHvU/gRmSniS9tDG7p3nr+CLpkv33pC/95ZV6niXdMjuKdAb4TeC9EXF3X9vcFxHxc9K6XUs6+9mX9IZbT9M/STrYTSedpT4IfIV0QITUGVbmWzofJr2IRF6Pq4AV+fbEKxto2w/zsmfl5d1F2j6QnoVeT3rB4z7SyxnVWyPdf6jiUUm358//kNdvPWlf9NqBcoecRnq5YW1e/t9R7g+WXE66Y/Ig6cWMj+V2LCNtx2+Q+sY7Sb9K9mwvy/pn4At5W3+a1IfvI11BLCG95NGQvM+PyPU+SHqj/H/k4gtJd3R+lr8T84FD6i0nL+sB0m23NwI/6KW+j5G+W+tJX/Y5m2nmbcBE0vb5MnB8RDyay95LCu0leXnX0Pht/kZ8nXTgfAj4HunZctV5wPfyvng36aR9aG7rfNKzvaoLgeMlrZf0r3Xq+yjpIL2C9MbwlcDMrbMqfbbN91lSKA3nxdvBtcPdv+ff098WGEQ6bjxAOll4IymgIb1Aeg/ptm337e4zSS+IPkR6R+Y71YVJWibpxB7q+nfSe0O/BRaQ+nrVKaQX0rq/C1ez6R2XW0kXLh2kjOqucx+lv5/Q63Fbmz6mMesflH7Z/IqI+Har29JuJJ1GerHjza1uy0DiPmu1+tWfSjQzM9tWOGDNzMwK8C1iMzOzAnwFa2ZmVoD/qHQdo0aNigkTJrS6GWZmbWXRokWPRERHq9vRXzhg65gwYQILFy5sdTPMzNqKpL78laVtnm8Rm5mZFeCANTMzK8ABa2ZmVoAD1szMrAAHrJmZWQEOWDMzswIcsGZmZgU4YM3MzApwwPYTU6ZMYcqUKa1uhpmZbSUOWDMzswIcsGZmZgU4YM3MzApwwJqZmRXg/02noAln/6ThaR9c8Wif5wFYef4xfZq+nu6Xq+bNm/eyl2VmZomvYM3MzApwwJqZmRXggLWW8e/+mtm2zAFrZmZWgF9y2kaVfsFqa7xcZWa2LXPA9hN7nnx+q5tgZmZbkW8Rm5mZFeArWNuqfGvazCzxFayZmVkBvoI1P/81MyvAV7BmZmYFtEXASpoqaZmk5ZLOrlP+SUlLJP1G0s8lja+UPSfpzvwzp7ktNzOzgarf3yKWNAi4CDgC6AIWSJoTEUsqk90BdEbE05LOBL4KnJjLNkTEgU1ttJmZDXj9PmCBycDyiFgBIGkWMA14IWAj4qbK9POBU5raQtsifvZrZtuydrhFPBpYXRnuyuN6cjpwfWV4R0kLJc2XdFyJBpqZmdVqhytY1RkXdSeUTgE6gbdWRo+LiDWS9gF+Iem3EXFvnXnPAM4AGDdu3MtvtZmZDWjtcAXbBYytDI8B1tROJOlw4PPAsRHxTPf4iFiT/10BzAMOqldJRFwSEZ0R0dnR0bH1Wm9mZgNSOwTsAmCipL0lDQGmA5u8DSzpIOBiUrg+XBk/UtIO+fMo4E1Unt2amZmV0u9vEUfERklnATcAg4CZEbFY0gxgYUTMAf4F2Bm4WhLAqog4FtgfuFjS86STifNr3j42MzMrot8HLEBEzAXm1ow7p/L58B7muxV4XdnWmZmZvVQ73CI2MzNrOw5YMzOzAhywZmZmBThgzczMCnDAmpmZFeCANTMzK8ABa2ZmVoAD1szMrAAHrJmZWQEOWDMzswIcsGZmZgU4YM3MzApwwJqZmRXggDUzMyvAAWtmZlaAA9bMzKwAB6yZmVkBDlgzM7MCHLBmZmYFOGDNzMwKcMCamZkV4IA1MzMrwAFrZmZWgAPWzMysAAesmZlZAQ5YMzOzAhywZmZmBbRFwEqaKmmZpOWSzq5T/klJSyT9RtLPJY2vlJ0q6Z78c2pzW25mZgNVvw9YSYOAi4CjgEnASZIm1Ux2B9AZEa8HrgG+mufdDTgXOASYDJwraWSz2m5mZgNXvw9YUjAuj4gVEfEsMAuYVp0gIm6KiKfz4HxgTP78duDGiFgXEeuBG4GpTWq3mZkNYO0QsKOB1ZXhrjyuJ6cD1/d1XklnSFooaeHatWtfRnPNzMzaI2BVZ1zUnVA6BegE/qWv80bEJRHRGRGdHR0dW9RQMzOzbu0QsF3A2MrwGGBN7USSDgc+DxwbEc/0ZV4zM7OtrR0CdgEwUdLekoYA04E51QkkHQRcTArXhytFNwBHShqZX246Mo8zMzMranCrG7A5EbFR0lmkYBwEzIyIxZJmAAsjYg7plvDOwNWSAFZFxLERsU7SP5JCGmBGRKxrwWqYmdkA0+8DFiAi5gJza8adU/l8eC/zzgRmlmudmZnZS7XDLWIzM7O244A1MzMrwAFrZmZWgAPWzMysAAesmZlZAQ5YMzOzAhywZmZmBThgzczMCnDAmpmZFeCANTMzK8ABa2ZmVoAD1szMrAAHrJmZWQEOWDMzswIcsGZmZgU4YM3MzApwwJqZmRXggDUzMyvAAWtmZlaAA9bMzKwAB6yZmVkBDlgzM7MCHLBmZmYFOGDNzMwKcMCamZkV4IA1MzMroC0CVtJUScskLZd0dp3yQyXdLmmjpONryp6TdGf+mdO8VpuZ2UA2uNUN2BxJg4CLgCOALmCBpDkRsaQy2SrgNODTdRaxISIOLN5QMzOzin4fsMBkYHlErACQNAuYBrwQsBGxMpc934oGmpmZ1WqHW8SjgdWV4a48rlE7Slooab6k43qaSNIZebqFa9eu3dK2mpmZAe0RsKozLvow/7iI6AROBi6QtG+9iSLikojojIjOjo6OLWmnmZnZC9ohYLuAsZXhMcCaRmeOiDX53xXAPOCgrdk4MzOzetohYBcAEyXtLWkIMB1o6G1gSSMl7ZA/jwLeROXZrZmZWSn9PmAjYiNwFnADsBSYHRGLJc2QdCyApDdI6gJOAC6WtDjPvj+wUNKvgZuA82vePjYzMyuiHd4iJiLmAnNrxp1T+byAdOu4dr5bgdcVb6CZmVmNfn8Fa2Zm1o4csGZmZgU4YM3MzApwwJqZmRXggDUzMyvAAWtmZlaAA9bMzKwAB6yZmVkBDlgzM7MCHLBmZmYFOGDNzMwKcMCamZkV4IA1MzMrwAFrZmZWgAPWzMysAAesmZlZAQ5YMzOzAhywZmZmBThgzczMCnDAmpmZFeCANTMzK8ABa2ZmVoAD1szMrAAHrJmZWQEOWDMzswIcsGZmZgW0RcBKmippmaTlks6uU36opNslbZR0fE3ZqZLuyT+nNq/VZmY2kPX7gJU0CLgIOAqYBJwkaVLNZKuA04Ara+bdDTgXOASYDJwraWTpNpuZmfX7gCUF4/KIWBERzwKzgGnVCSJiZUT8Bni+Zt63AzdGxLqIWA/cCExtRqPNzGxga4eAHQ2srgx35XFbdV5JZ0haKGnh2rVrt6ihZmZm3dohYFVnXGzteSPikojojIjOjo6OhhtnZmZWTzsEbBcwtjI8BljThHnNzMy2WDsE7AJgoqS9JQ0BpgNzGpz3BuBISSPzy01H5nFmZmZF9fuAjYiNwFmkYFwKzI6IxZJmSDoWQNIbJHUBJwAXS1qc510H/CMppBcAM/I4MzOzoga3ugGNiIi5wNyacedUPi8g3f6tN+9MYGbRBpqZmdXo91ewZmZm7cgBa2ZmVoAD1szMrAAHrJmZWQEOWDMzswIcsGZmZgU4YM3MzApwwJqZmRXggDUzMyvAAWtmZlaAA9bMzKwAB6yZmVkBDlgzM7MCHLBmZmYFOGDNzMwKcMCamZkV4IA1MzMrwAFrZmZWgAPWzMysAAesmZlZAQ5YMzOzAhywZmZmBThgzczMCnDAmpmZFeCANTMzK8ABa2ZmVkBbBKykqZKWSVou6ew65TtI+kEuv03ShDx+gqQNku7MP99qdtvNzGxgGtzqBmyOpEHARcARQBewQNKciFhSmex0YH1E7CdpOvAV4MRcdm9EHNjURpuZ2YDXDlewk4HlEbEiIp4FZgHTaqaZBnwvf74GOEySmthGMzOzTbRDwI4GVleGu/K4utNExEbgcWD3XLa3pDsk3SzpLT1VIukMSQslLVy7du3Wa72ZmQ1I7RCw9a5Eo8FpHgDGRcRBwCeBKyXtUq+SiLgkIjojorOjo+NlNdjMzKwdArYLGFsZHgOs6WkaSYOBXYF1EfFMRDwKEBGLgHuBVxVvsZmZDXjtELALgImS9pY0BJgOzKmZZg5wav58PPCLiAhJHfklKSTtA0wEVjSp3WZmNoD1+7eII2KjpLOAG4BBwMyIWCxpBrAwIuYAlwKXS1oOrCOFMMChwAxJG4HngA9HxLrmr4WZmQ00/T5gASJiLjC3Ztw5lc9/BE6oM9+1wLXFG2hmZlajHW4Rm5mZtR0HrJmZWQEOWDMzswIcsGZmZgU4YM3MzApwwJqZmRXggDUzMyvAAWtmZlaAA9bMzKwAB6yZmVkBDlgzM7MCHLBmZmYFOGDNzMwKcMCamZkV4IA1MzMrwAFrZmZWgAPWzMysAAesmZlZAQ5YMzOzAhywZmZmBThgzczMCnDAmpmZFeCANTMzK8ABa2ZmVoAD1szMrAAHrJmZWQFtE7CSpkpaJmm5pLPrlO8g6Qe5/DZJEypln83jl0l6ezPbbWZmA1NbBKykQcBFwFHAJOAkSZNqJjsdWB8R+wFfB76S550ETAcOAKYC38zLMzMzK2ZwqxvQoMnA8ohYASBpFjANWFKZZhpwXv58DfBvkpTHz4qIZ4DfS1qel/ffTWq7NcmEs3/S8LQPXplugux58vkNz7Py/GP63Kb+ZMqUKQDMmzdvQNTbSgNxne2l2iVgRwOrK8NdwCE9TRMRGyU9Duyex8+vmXd0bQWSzgDOABg3btxWaXQrD8itqrtt1nkrtrMvwb4lttY23RYO9qVPoqD/nUiV7l/Q/9Z5W9EuAas646LBaRqZl4i4BLgEoLOz8yXlZj3py8FpIF7ZbM11bdVJVF+1ap0HYv/qz9olYLuAsZXhMcCaHqbpkjQY2BVY1+C8Zk3hA5+V5P7Vv7TFS07AAmCipL0lDSG9tDSnZpo5wKn58/HALyIi8vjp+S3jvYGJwK+a1G4zMxug2uIKNj9TPQu4ARgEzIyIxZJmAAsjYg5wKXB5folpHSmEydPNJr0QtRH4SEQ815IVMTOzAUPpIs+qOjs7Y+HCha1uhplZW5G0KCI6W92O/qJdbhGbmZm1FQesmZlZAQ5YMzOzAhywZmZmBThgzczMCnDAmpmZFeBf06lD0lrgvhZUPQp4pAX1trJur/PAqHug1dvKulu5zuMjoqNFdfc7Dth+RNLCVv0OWavq9joPjLoHWr2trLuV62yb8i1iMzOzAhywZmZmBThg+5dLBmDdXueBUfdAq7eVdbdyna3Cz2DNzMwK8BWsmZlZAQ5YMzOzAhyw/YCkmZIelnRXC+qeKmmZpOWSzm5ivWMl3SRpqaTFkj7exLpHSLpG0t25/r9sYt0fl3RXXudPNLHelZJ+K+lOSU39vxgl/W1e37skXSVpxybWPUjSHZKua2Kdr87bufvniWbsa0k7SvqVpF/n7f3F0nVa7/wMth+QdCjwFHBZRLy2ifUOAn4HHAF0AQuAkyJiSRPq3gvYKyJulzQcWAQc16S6vwf8MiK+LWkIMCwiHmtCva8FZgGTgWeBnwJnRsQ9Tah7JdAZEU39AwSSRgP/BUyKiA2SZgNzI+K7Tar/k0AnsEtEvKMZddbUPwi4HzgkIor+8RpJAnaKiKckbU/a7h+PiPkl67We+Qq2H4iIW4B1Lah6MrA8IlZExLOkg/+0ZlQcEQ9ExO3585PAUmB06Xol7QIcClya6362GeGa7Q/Mj4inI2IjcDPwV02qu5UGA0MlDQaGAWuaUamkMcAxwLebUV8PDgPuLR2uAJE8lQe3zz++gmohB+zANhpYXRnuogkhV0vSBOAg4LYmVLcPsBb4Tr51+G1JOzWhXoC7gEMl7S5pGHA0MLZJdQfwM0mLJJ3RpDqJiPuBrwGrgAeAxyPiZ02q/gLgM8DzTaqvnunAVc2qLN8SvxN4GLgxIprxnbIeOGAHNtUZ19QzXkk7A9cCn4iIJ5pQ5WDgYODfI+Ig4A9AU549R8RS4CvAjaTbw78GNjajbuBNEXEwcBTwkfxYojhJI0l3RfYGXgnsJOmUJtT7DuDhiFhUuq5e2jAEOBa4ull1RsRzEXEgMAaYnB9LWIs4YAe2Lja9ghpDk27fAeTnRNcC34+I/2xStV1AV+XM/hpS4DZFRFwaEQdHxKGkxwLFn7/metfkfx8Gfkh6PNAMhwO/j4i1EfEn4D+BNzah3jcBx+Znz7OAt0m6ogn1Vh0F3B4RDzW5XvJjj3nA1GbXbS9ywA5sC4CJkvbOZ9vTgTnNqDi/kHEpsDQi/ncz6gSIiAeB1ZJenUcdBhR/saqbpFfkf8cB76IJtw8l7ZRfJCPfDj+SdLu6GVYBfyFpWN7nh5GetxcVEZ+NiDERMYHUr38REcWvnGucRHNvD3dIGpE/DyWd3NzdrPrtpQa3ugEGkq4CpgCjJHUB50bEpaXrjYiNks4CbgAGATMjYnHperM3Ae8BfpufGQF8LiLmNqHujwLfzycVK4D3NaHObtdK2h34E/CRiFjfhDr3AH6Y8o3BwJUR8dMm1EtE3CbpGuB20u3wOxgAf8ovP2M/AvhQE6vdC/hefnN5O2B2RDTt15PspfxrOmZmZgX4FrGZmVkBDlgzM7MCHLBmZmYFOGDNzMwKcMCamZkV4IA1MzMrwAFrZmZWwP8HJ2pjcDPm1/gAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "imp_vals, imp_all = feature_importance_permutation(\n", " predict_method=forest.predict, \n", " X=X_test,\n", " y=y_test,\n", " metric='accuracy',\n", " num_rounds=10,\n", " seed=1)\n", "\n", "\n", "std = np.std(imp_all, axis=1)\n", "indices = np.argsort(imp_vals)[::-1]\n", "\n", "plt.figure()\n", "plt.title(\"Random Forest feature importance via permutation importance w. std. dev.\")\n", "plt.bar(range(X.shape[1]), imp_vals[indices],\n", " yerr=std[indices])\n", "plt.xticks(range(X.shape[1]), indices)\n", "plt.xlim([-1, X.shape[1]])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It shall be noted that the feature importance values do not sum up to one, since they are not normalized (you can normalize them if you'd like, by dividing these by the sum of importance values). Here, the main point is to look at the importance values relative to each other and not to over-interpret the absolute values." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Support Vector Machines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While the permutation importance approach yields results that are generally consistent with the *mean impurity decrease* feature importance values from a random forest, it's a method that is model-agnostic and can be used with any kind of classifier or regressor. The example below applies the `feature_importance_permutation` function to a support vector machine:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training accuracy 95.0857142857\n", "Test accuracy 94.9666666667\n" ] } ], "source": [ "from sklearn.svm import SVC\n", "\n", "\n", "svm = SVC(C=1.0, kernel='rbf')\n", "svm.fit(X_train, y_train)\n", "\n", "print('Training accuracy', np.mean(svm.predict(X_train) == y_train)*100)\n", "print('Test accuracy', np.mean(svm.predict(X_test) == y_test)*100)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAG+pJREFUeJzt3XmYXVWd7vHva4UwT5JyIAlJ0DgE8AG6DA4tnStTECV0tzwGmr7QrU1rS7de9bYRvWDHCbXb5t4rrXAlXgRDZFA7F6OAQrRtjKQYRAJGQoikDENJGAQUCPzuH2sV7hxOpfapnDonxXo/z1NP7WHttdbeZ59377P3GRQRmJlZGV7Q7Q6YmVnnOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0G+BpDdKukPSo5KO63Z/WiHpTZJWd7sf45Wk0yV9pdv92BZJ+gtJV41Bvd5nx8C4DH1JfyzpOkkPS9oo6T8lvVbS6yU9JmnXJsvcJOk0SdMlhaQbG+ZPkvSkpHVbaHoh8MWI2CUivr2V67BO0uFbU0crIuI/IuKVnWpvSyTNkTTQ7X60IiI+HRHv6nY/2k3SKZJ+3EL5oefPhKFpEfH1iDiy3X3zPjs2xl3oS9oNuAL438ALgcnAPwFPRMRPgAHgzxuW2R+YBVxcmbxznj7kROCuEZqfBqzaqhVok+qTbjwZr/3eFigZd8/Z8e55t89GxLj6A/qAh7Yw/3TgmoZpnwO+mYenAwF8DPh8pUw/8FFg3TD13gk8A/wOeBTYHtgdOB+4B/g18EmgJ5d/GXAN8ADwG+DrwB553oUNdf0jMAcYaGhzHXB4Hv44cBlwEfAI8C7SQXtB7tsDwCXAC4fp/2b157r/O3AL8FhejxcD3wV+C3wf2LNhm50KbMjr+8FKXdsDZ+d5G/Lw9tV2gQ8D9wKX5vV+Jq/7o8DewGzgJ8BDuf4vAhMrbQTwbuAO4EHgHECV+X8D3J77fhtwcJ6+N3A5MEg6qP/DMNvndbl/PZVpfwrcUtn+F1XmXZrLPwz8CNhvC/vkcuAzwPW5/L9XH6fc9nV53X8GzGlY9lPAf+bt9vI87ZN5mUeB/wfsRdrHHgFWAtMbHrsJDXW+C3g18Hvg6VzPQ3n+McBNua71wMcry96d6xt67F4PnAL8uFLmDbkPD+f/b2ho+xN5fX4LXAVM8j7b+j476gwd65Bu9x+wGyngLgCOHnqQK/OnAk8B++TxF+QH8LiGnWF63qF78s6/GjicYUK/stMdXhn/NnAusDPwItKT+m/zvJcDR+Sdq5cUDGdvoa7NdvDGMqTQeQo4Lq/TjsD7gRXAlNzOucDFLTyBVpCeNJOB+4EbgYNyXdcAZzZss4vzuh6Qd8ihvi3Mdb0or+t1wCcq7W4CPpvr3XGYdf0jUvhNyO3dDry/4Ql0BbAHsE9uf26edzzpoPtaQHnbT8vb6QbgDGAisC+wFjhqmG10J3BEZfxSYEFl+1dD/6+BXflDeNy8hf1mee7f/nn7XT5UV972DwBvyf09Io/3Vpa9G9gvb5vt8rQ1pBOL3UmB8UvS/jsB+Brw1YbH7jmhn4dPoRLYlcfsgNyf1wD38dznT7W+Z+sgvfp+EPjL3JcT8vhelbbvBF6R94XlwFneZ0e3z44qQ8cyoMfqjxTS/5cU5puApcCLK/O/D5yeh48gnWlv17jT5nJHAWeRzvJrh37e8Z4AdqzMPwG4dphljwNualZXsx28SXsfB37UMP924LDK+EtJB4YJTdrfrP5c919Uxi8HvlQZ/3vg2w3b7FWV+Z8Dzs/DdwJvqcw7amg75nafBHbY0ro26e/7gW81PIH+uDJ+CX8I5CuB9zWp4xDg7oZpHyEHYpPynwQW5eFdSWeT0yrb/6Jhltsj92/3YeYvpxJspEuNT5JOOD4MXNhQ/krg5MqyC5vU99HK+L8A362Mv418EGIUod+k/2cD/7qF+p6tgxT21zcs/xPglErbH6vM+zvge8O0u9l+gvfZtuTnuLxWFRG3k3Y0JL2KdMnjbFLoQnoV8FHg06SdcHFEPNWkqq/let4AHArMbKEb00hnXfdIGpr2AtKrByS9CPhfwJtIAfIC0hnP1ljfpA/fkvRMZdrTpAPSr2vUd19l+HdNxnfZQvu/Ip09QXo5+quGeXtXxgcj4vdb6oikVwBfIF2+24l0UL6hodi9leHHK/2bSnoSN5oG7C3pocq0HuA/hunGYuA6Se8B/gy4MSJ+1VhIUg/pksvxpLPEoe0/iXRJo5nGbbddLj8NOF7S2yrztwOuHWbZIa0+drVJOoR0IrQ/6Wxze9Krnjoa9wXy+OTK+HCPYx3eZ7fSuL8pFBG/IJ31V2/KfhOYLOm/kJ68Xxtm8ctJ1y/XNntyj2A96Ux/UkTskf92i4j98vzPkI70r4mI3YCTSC/jnu16Q32PkXYc4Nlg6W0o07jMeuDoSvt7RMQOEVEn8EdjamV4H9K1UPL/acPMg+f2u3Ec4EvAL4CZeXudzubba0vWky51NJt+V8P22TUi3tKskoi4jfTkP5p0Y3/xMO2dCMwjvTLcnXRWyQj9bdx2T5Fega4nnelX+7hzRJxV7doW6h3JY/n/TpVpLxmh7sWkV89TI2J34Mv8Yd1G6kvjvgBpfcdqnxzJ83qfHY1xF/qSXiXpg5Km5PGppDP8FUNlIuIx0k3PrwK/ioj+ZnXlcm8m3dRqSUTcQ7oJ9S+SdpP0Akkvk/Qnuciu5JtjkiaTbkBV3Ue6Xjfkl8AOko6RtB3pRvP2I3Tjy8CnJE0DkNQraV6r69KC/yFpJ0n7AX8FfCNPvxj4WG5/Eul65EVbqOc+YC9Ju1em7Uq6cfhofvX2nhb69RXgQ5L+KL/D5eV5m1wPPCLpw5J2lNQjaX9Jr91CXYuBfyC98hvu7HZX0gH/AVKYfrpGH0+SNEvSTqTryZdFxNOk7fQ2SUfl/u2Q3x44pdaajyAiBkmBe1Ku/6/ZPGzuA6ZImliZtiuwMSJ+L2k26SA3ZJD0yqa671YtA14h6URJEyS9g3Q564p2rM8olLDPtmTchT7pTvchwE8lPUYK+1uBDzaUu4B0JB/uLB+AiOiPiGYvs+r4r6SXv7eRLt1cRrquDultpAeTXu5/h/Tqo+ozpJ3uIUkfioiHSdc3v0J6kj5GumexJf+TdEZ2laTfkrbFIaNclzp+SLqB+APgnyNi6AM5nyS9++kW4Oekm2ufHK6S/OrsYmBtXv+9gQ+RwuW3wP/hD0/OEUXEpaTLLYvz8t8mvTvmadL17QNJ74L4DWn77j5MVeR+zSG9A+w3w5T5GukVwa9Jj/2KYcpVXUh6RXovsAPpwEJErCe9ajidFKjrSScI7Xxu/k2u8wHSDeHrKvOuIb0N+V5JQ+v7d8DCvE+dQboWTe7v4+R3E+XH7nXVhiLiAeCtpOfjA6R3pr11C9tyrJWwz7ZE+UaB2bAkTSftgNtFxKbu9mb8kbScdBPYn+jtEO+zwxuPZ/pmZjZKDn0zs4L48o6ZWUF8pm9mVpBt7sNZkyZNiunTp3e7G2Zm48oNN9zwm4ho/GzPc2xzoT99+nT6+5u+rd7MzIYhqdYHTH15x8ysIA59M7OCOPTNzApSK/QlzZW0WtIaSQuazH+3pJ9LulnSjyXNqsz7SF5utaSj2tl5MzNrzYihn7/t8RzSNw/OAk6ohnq2OCIOiIgDSd9Z/YW87CxgPun7PuYC/5brMzOzLqhzpj8bWBMRayPiSWAJ6QuinhURj1RGd+YPX0M6D1gSEU9ExF2kLz6avfXdNjOz0ajzls3JbP5DBAM0+SZHSe8FPkD61sk3V5atfgPhAJv/mIKZmXVQnTP9Zj8K8JzvboiIcyLiZaSff/tYK8tKOlVSv6T+wcHBGl0yM7PRqBP6A2z+6zNT2PwXZhotIf0ebO1lI+K8iOiLiL7e3hE/UDYm5syZw5w5c7rStplZp9QJ/ZXATEkz8q/rzCf9cMezJFV/W/YY4I48vBSYL2l7STNIv0F7/dZ328zMRmPEa/oRsUnSaaRfb+8BFkXEKkkLgf6IWAqcJulw0u9+PgicnJddJekS0q8LbQLem38ZxszMuqDWd+9ExDLSb19Wp51RGX7fFpb9FOlnwczMrMv8iVwzs4Jsc9+y2U7TF3yndtl71z7Q8jLrzjqm5T6ZmXWTz/TNzAri0C+Y36ZqVh6HvplZQZ7X1/Rb8ZITz+p2F8zMxpzP9M3MCuLQNzMriEPfzKwgDn0zs4L4Ru7zjD+QZmZb4jN9M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzArit2wWzN83ZFYen+mbmRXEoW9mVhCHvplZQRz6ZmYFceh3mX+y0Mw6yaFvZlaQWqEvaa6k1ZLWSFrQZP4HJN0m6RZJP5A0rTLvaUk357+l7ey8mZm1ZsT36UvqAc4BjgAGgJWSlkbEbZViNwF9EfG4pPcAnwPekef9LiIObHO/zcxsFOp8OGs2sCYi1gJIWgLMA54N/Yi4tlJ+BXBSOzs53vg77c1sW1Xn8s5kYH1lfCBPG847ge9WxneQ1C9phaTjmi0g6dRcpn9wcLBGl8zMbDTqnOmrybRoWlA6CegD/qQyeZ+I2CBpX+AaST+PiDs3qyziPOA8gL6+vqZ1m5nZ1qtzpj8ATK2MTwE2NBaSdDjwUeDYiHhiaHpEbMj/1wLLgYO2or9mZrYV6oT+SmCmpBmSJgLzgc3ehSPpIOBcUuDfX5m+p6Tt8/Ak4I1U7gWYmVlnjXh5JyI2SToNuBLoARZFxCpJC4H+iFgKfB7YBbhUEsDdEXEs8GrgXEnPkA4wZzW866d4/qZLM+ukWl+tHBHLgGUN086oDB8+zHLXAQdsTQfNzKx9/IlcM7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysILVCX9JcSaslrZG0oMn8D0i6TdItkn4gaVpl3smS7sh/J7ez82Zm1poRQ19SD3AOcDQwCzhB0qyGYjcBfRHxGuAy4HN52RcCZwKHALOBMyXt2b7um5lZK+qc6c8G1kTE2oh4ElgCzKsWiIhrI+LxPLoCmJKHjwKujoiNEfEgcDUwtz1dNzOzVtUJ/cnA+sr4QJ42nHcC321lWUmnSuqX1D84OFijS2ZmNhp1Ql9NpkXTgtJJQB/w+VaWjYjzIqIvIvp6e3trdMnMzEajTugPAFMr41OADY2FJB0OfBQ4NiKeaGVZMzPrjDqhvxKYKWmGpInAfGBptYCkg4BzSYF/f2XWlcCRkvbMN3CPzNPMzKwLJoxUICI2STqNFNY9wKKIWCVpIdAfEUtJl3N2AS6VBHB3RBwbERslfYJ04ABYGBEbx2RNzMxsRCOGPkBELAOWNUw7ozJ8+BaWXQQsGm0HzcysffyJXDOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCC1Ql/SXEmrJa2RtKDJ/EMl3Shpk6S3N8x7WtLN+W9puzpuZmatmzBSAUk9wDnAEcAAsFLS0oi4rVLsbuAU4ENNqvhdRBzYhr6amdlWGjH0gdnAmohYCyBpCTAPeDb0I2JdnvfMGPTRzMzapM7lncnA+sr4QJ5W1w6S+iWtkHRcswKSTs1l+gcHB1uo2szMWlEn9NVkWrTQxj4R0QecCJwt6WXPqSzivIjoi4i+3t7eFqo2M7NW1An9AWBqZXwKsKFuAxGxIf9fCywHDmqhf2Zm1kZ1Qn8lMFPSDEkTgflArXfhSNpT0vZ5eBLwRir3AszMrLNGDP2I2AScBlwJ3A5cEhGrJC2UdCyApNdKGgCOB86VtCov/mqgX9LPgGuBsxre9WNmZh1U5907RMQyYFnDtDMqwytJl30al7sOOGAr+2hmZm3iT+SamRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQRz6ZmYFceibmRXEoW9mVhCHvplZQWqFvqS5klZLWiNpQZP5h0q6UdImSW9vmHeypDvy38nt6riZmbVuxNCX1AOcAxwNzAJOkDSrodjdwCnA4oZlXwicCRwCzAbOlLTn1nfbzMxGo86Z/mxgTUSsjYgngSXAvGqBiFgXEbcAzzQsexRwdURsjIgHgauBuW3ot5mZjUKd0J8MrK+MD+RpdWzNsmZm1mZ1Ql9NpkXN+mstK+lUSf2S+gcHB2tWbWZmraoT+gPA1Mr4FGBDzfprLRsR50VEX0T09fb21qzazMxaVSf0VwIzJc2QNBGYDyytWf+VwJGS9sw3cI/M08zMrAtGDP2I2AScRgrr24FLImKVpIWSjgWQ9FpJA8DxwLmSVuVlNwKfIB04VgIL8zQzM+uCCXUKRcQyYFnDtDMqwytJl26aLbsIWLQVfTQzszbxJ3LNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OCOPTNzAri0DczK4hD38ysIA59M7OC1Ap9SXMlrZa0RtKCJvO3l/SNPP+nkqbn6dMl/U7Szfnvy+3tvpmZtWLCSAUk9QDnAEcAA8BKSUsj4rZKsXcCD0bEyyXNBz4LvCPPuzMiDmxzv83MbBTqnOnPBtZExNqIeBJYAsxrKDMPuCAPXwYcJknt66aZmbVDndCfDKyvjA/kaU3LRMQm4GFgrzxvhqSbJP1Q0puaNSDpVEn9kvoHBwdbWgEzM6uvTug3O2OPmmXuAfaJiIOADwCLJe32nIIR50VEX0T09fb21uiSmZmNRp3QHwCmVsanABuGKyNpArA7sDEinoiIBwAi4gbgTuAVW9tpMzMbnTqhvxKYKWmGpInAfGBpQ5mlwMl5+O3ANRERknrzjWAk7QvMBNa2p+tmZtaqEd+9ExGbJJ0GXAn0AIsiYpWkhUB/RCwFzgculLQG2Eg6MAAcCiyUtAl4Gnh3RGwcixUxM7ORjRj6ABGxDFjWMO2MyvDvgeObLHc5cPlW9tHMzNrEn8g1MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwKUiv0Jc2VtFrSGkkLmszfXtI38vyfSppemfeRPH21pKPa13UzM2vViKEvqQc4BzgamAWcIGlWQ7F3Ag9GxMuBfwU+m5edBcwH9gPmAv+W6zOzDpkzZw5z5szpdjdsGzGhRpnZwJqIWAsgaQkwD7itUmYe8PE8fBnwRUnK05dExBPAXZLW5Pp+0p7u27Zi+oLv1C577+L0YvElJ55Ve5l1Zx3Tcp+er1rZ1gD3rn2g5eXasb2HDjTLly/f6rrGQ7vjRZ3Qnwysr4wPAIcMVyYiNkl6GNgrT1/RsOzkxgYknQqcCrDPPvvU7fuIuhUU3QyocbHObexjywHYxgNOq223arh2W36MfcDcKs+3E5o6oa8m06JmmTrLEhHnAecB9PX1PWe+2XBafcLMWfF5AJZ3+Ik2mjAY77p1pt3udrt1QjNW6oT+ADC1Mj4F2DBMmQFJE4DdgY01lzXrmHYGQith0K2DjVmjOqG/EpgpaQbwa9KN2RMbyiwFTiZdq387cE1EhKSlwGJJXwD2BmYC17er82bjha8v27ZixNDP1+hPA64EeoBFEbFK0kKgPyKWAucDF+YbtRtJBwZyuUtIN303Ae+NiKfHaF3MzGwEiti2LqH39fVFf39/t7thZjauSLohIvpGKudP5JqZFcShb2ZWEIe+mVlBHPpmZgVx6JuZFcShb2ZWEIe+mVlBtrn36UsaBH7VpeYnAb8pqN1utu11LqPt0trtZtvTIqJ3pELbXOh3k6T+Oh9ueL602822vc5ltF1au91uuw5f3jEzK4hD38ysIA79zZ1XWLvdbNvrXEbbpbXb7bZH5Gv6ZmYF8Zm+mVlBHPpmZgVx6AOSFkm6X9KtXWh7rqTVktZIWtChNqdKulbS7ZJWSXpfJ9qttL+HpMsk/SL34fUdavd9km7N6/z+TrRZaXudpJ9LullSx34wQtJ/y+t7q6SLJe3QwbZ7JN0k6YoOtvnKvI2H/h7pxGMtaQdJ10v6Wd7e/zTWbY5aRBT/BxwKHAzc2uF2e4A7gX2BicDPgFkdaPelwMF5eFfgl51ot9L+BcC78vBEYI8OtLk/cCuwE+kX474PzOzgOq8DJnWqvdzmZOAuYMc8fglwSgfb/wCwGLiik+tdab8HuJf0oaWxbkvALnl4O+CnwOu6sd4j/flMH4iIH5F+5rHTZgNrImJtRDwJLAHmjXWjEXFPRNyYh38L3E4KiDEnaTfSQfb83P6TEfFQB5p+NbAiIh6PiE3AD4E/7UC73TYB2FHSBNIBb0MnGpU0BTgG+Eon2hvGYcCdETHmn/CP5NE8ul3+2ybfJePQ767JwPrK+AAdCt8hkqYDB5HOTDphX2AQ+Gp+6f8VSTt3oN1bgUMl7SVpJ+AtwNQOtDskgKsk3SDp1I40GPFr4J+Bu4F7gIcj4qpOtA2cDfwj8EyH2mtmPnBxpxrLl7NuBu4Hro6ITj2nWuLQ7y41mdaxswNJuwCXA++PiEc61OwE0qW0L0XEQcBjwJjfy4iI24HPAlcD3yNdSts01u1WvDEiDgaOBt4r6dCxblDSnqRXjjOAvYGdJZ3UgXbfCtwfETeMdVtb6MNE4Fjg0k61GRFPR8SBwBRgtqT9O9V2Kxz63TXA5mebU+jcy+/tSIH/9Yj4ZifazAaAgcpZ0GWkg8CYi4jzI+LgiDiUdDnvjk60m9vekP/fD3yLdGlvrB0O3BURgxHxFPBN4A0daPeNwLGS1pEuWb5Z0kUdaLfqaODGiLivw+2SL1cuB+Z2uu06HPrdtRKYKWlGPjOZDywd60YliXRN/faI+MJYt1cVEfcC6yW9Mk86DLitE21LelH+vw/wZ3Topb+knSXtOjQMHEm63DTW7gZeJ2mn/JgfRrp/M6Yi4iMRMSUippP26WsiYsxfYTQ4gc5e2umVtEce3pF0wP1Fp9pvxYRud2BbIOliYA4wSdIAcGZEnD/W7UbEJkmnAVeS3mmwKCJWjXW7pDOxvwR+nq9BApweEcs60DbA3wNfzwe6tcBfdajdyyXtBTwFvDciHuxQuy8GvpVylwnA4oj43lg3GhE/lXQZcCPpUtZNbONfEdAO+Z7NEcDfdrDZlwIXSOohnUxfEhEde6tqK/w1DGZmBfHlHTOzgjj0zcwK4tA3MyuIQ9/MrCAOfTOzgjj0zcwK4tA3MyvI/weSHrz1Fs5p4gAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "imp_vals, imp_all = feature_importance_permutation(\n", " predict_method=svm.predict, \n", " X=X_test,\n", " y=y_test,\n", " metric='accuracy',\n", " num_rounds=10,\n", " seed=1)\n", "\n", "\n", "std = np.std(imp_all, axis=1)\n", "indices = np.argsort(imp_vals)[::-1]\n", "\n", "plt.figure()\n", "plt.title(\"SVM feature importance via permutation importance\")\n", "plt.bar(range(X.shape[1]), imp_vals[indices],\n", " yerr=std[indices])\n", "plt.xticks(range(X.shape[1]), indices)\n", "plt.xlim([-1, X.shape[1]])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example 1 -- Feature Importance for Regressors" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([ 0.43676245, 0.22231268, 0.00146906, 0.01611528, -0.00522067])" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from mlxtend.evaluate import feature_importance_permutation\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.datasets import make_regression\n", "from sklearn.svm import SVR\n", "\n", "\n", "X, y = make_regression(n_samples=1000,\n", " n_features=5,\n", " n_informative=2,\n", " n_targets=1,\n", " random_state=123,\n", " shuffle=False)\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.3, random_state=123) \n", "\n", "svm = SVR(kernel='rbf')\n", "svm.fit(X_train, y_train)\n", "\n", "imp_vals, _ = feature_importance_permutation(\n", " predict_method=svm.predict, \n", " X=X_test,\n", " y=y_test,\n", " metric='r2',\n", " num_rounds=1,\n", " seed=1)\n", "\n", "imp_vals" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADGxJREFUeJzt3XGoXvddx/H3Z8mi0g0Fd/8YSboEDdYwtWXXtDCYUiqkVhLBCilMVqiEwcKqG2hEKRr/qR1s/mH+WNSiqFtWp39c10gYW4cIruZ2q9M0Ri8hmkuEZm5uDrE17usf97Y83t30nic5d0/y7fsFF55zzq/P/R5C3pyce5/TVBWSpF7eMOsBJEnjM+6S1JBxl6SGjLskNWTcJakh4y5JDQ2Ke5L9Sc4nWUpydJ3jDye5kuT51a+fH39USdJQWzdakGQLcBz4CWAZOJNkoapeWLP0E1V1ZBNmlCRNaciV+z5gqaouVNXLwEng4OaOJUm6ERteuQPbgUsT28vA3eus+5kk7wL+CfjFqrq0dkGSw8BhgNtuu+0dd9xxx/QTS9Lr2HPPPfflqprbaN2QuGedfWufWfAXwMer6qUk7wX+ELj3W/6jqhPACYD5+flaXFwc8O0lSa9I8i9D1g25LbMM7JzY3gFcnlxQVf9eVS+tbv4u8I4h31yStDmGxP0MsCfJ7iTbgEPAwuSCJG+d2DwAnBtvREnStDa8LVNVV5McAU4DW4Anq+pskmPAYlUtAO9PcgC4CnwFeHgTZ5YkbSCzeuSv99wlaXpJnquq+Y3W+QlVSWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNbR11gPM2q6jT896hKlcfPyBWY8g6RbglbskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJamhQ3JPsT3I+yVKSo6+x7sEklWR+vBElSdPaMO5JtgDHgfuBvcBDSfaus+7NwPuBZ8ceUpI0nSFX7vuApaq6UFUvAyeBg+us+03gCeC/R5xPknQdhsR9O3BpYnt5dd+rktwF7KyqT73WGyU5nGQxyeKVK1emHlaSNMyQuGedffXqweQNwEeAD270RlV1oqrmq2p+bm5u+JSSpKkMifsysHNiewdweWL7zcDbgc8luQjcAyz4Q1VJmp0hcT8D7EmyO8k24BCw8MrBqvpaVb2lqnZV1S7g88CBqlrclIklSRvaMO5VdRU4ApwGzgFPVdXZJMeSHNjsASVJ0xv0P8iuqlPAqTX7HrvG2h+/8bEkSTfCT6hKUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8ZdkhoaFPck+5OcT7KU5Og6x9+b5O+TPJ/kr5PsHX9USdJQG8Y9yRbgOHA/sBd4aJ14f6yqfqiq7gSeAD48+qSSpMGGXLnvA5aq6kJVvQycBA5OLqiqr09s3gbUeCNKkqa1dcCa7cClie1l4O61i5K8D/gAsA24d703SnIYOAxw++23TzurJGmgIVfuWWfft1yZV9Xxqvo+4JeBX1vvjarqRFXNV9X83NzcdJNKkgYbEvdlYOfE9g7g8musPwn89I0MJUm6MUPifgbYk2R3km3AIWBhckGSPRObDwD/PN6IkqRpbXjPvaquJjkCnAa2AE9W1dkkx4DFqloAjiS5D/gf4KvAezZzaEnSaxvyA1Wq6hRwas2+xyZePzryXJKkG+AnVCWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkNbZz2ANs+uo0/PeoSpXXz8gVmPILXglbskNWTcJakh4y5JDRl3SWrIuEtSQ8ZdkhoaFPck+5OcT7KU5Og6xz+Q5IUkX0rymSRvG39USdJQG8Y9yRbgOHA/sBd4KMneNcu+CMxX1Q8DnwSeGHtQSdJwQ67c9wFLVXWhql4GTgIHJxdU1TNV9V+rm58Hdow7piRpGkPivh24NLG9vLrvWh4B/nK9A0kOJ1lMsnjlypXhU0qSpjIk7llnX627MHk3MA98aL3jVXWiquaran5ubm74lJKkqQx5tswysHNiewdwee2iJPcBvwr8WFW9NM54kqTrMeTK/QywJ8nuJNuAQ8DC5IIkdwEfBQ5U1YvjjylJmsaGca+qq8AR4DRwDniqqs4mOZbkwOqyDwFvAv40yfNJFq7xdpKkb4NBj/ytqlPAqTX7Hpt4fd/Ic0mSboCfUJWkhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNTQo7kn2JzmfZCnJ0XWOvyvJF5JcTfLg+GNKkqaxYdyTbAGOA/cDe4GHkuxds+xfgYeBj409oCRpelsHrNkHLFXVBYAkJ4GDwAuvLKiqi6vHvrkJM0qSpjTktsx24NLE9vLqvqklOZxkMcnilStXructJEkDDIl71tlX1/PNqupEVc1X1fzc3Nz1vIUkaYAhcV8Gdk5s7wAub844kqQxDIn7GWBPkt1JtgGHgIXNHUuSdCM2jHtVXQWOAKeBc8BTVXU2ybEkBwCS/GiSZeBngY8mObuZQ0uSXtuQ35ahqk4Bp9bse2zi9RlWbtdIkm4CfkJVkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJasi4S1JDxl2SGjLuktSQcZekhoy7JDVk3CWpIeMuSQ0Zd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJakh4y5JDRl3SWrIuEtSQ8Zdkhoy7pLUkHGXpIaMuyQ1ZNwlqSHjLkkNGXdJamjrrAeQtL5dR5+e9QhTu/j4A7MeQau8cpekhoy7JDVk3CWpoUFxT7I/yfkkS0mOrnP8O5J8YvX4s0l2jT2oJGm4DeOeZAtwHLgf2As8lGTvmmWPAF+tqu8HPgL81tiDSpKGG/LbMvuApaq6AJDkJHAQeGFizUHg11dffxL4nSSpqhpxVun/8bdJpGsbEvftwKWJ7WXg7mutqaqrSb4GfC/w5clFSQ4Dh1c3v5Hk/PUMfYt4C2vOfwy5ef5N1Pn8NuXcwPP7Ntm087tJvG3IoiFxzzr71l6RD1lDVZ0ATgz4nre8JItVNT/rOTZL5/PrfG7g+b1eDPmB6jKwc2J7B3D5WmuSbAW+G/jKGANKkqY3JO5ngD1JdifZBhwCFtasWQDes/r6QeCz3m+XpNnZ8LbM6j30I8BpYAvwZFWdTXIMWKyqBeD3gT9KssTKFfuhzRz6FtH99lPn8+t8buD5vS7EC2xJ6sdPqEpSQ8Zdkhoy7iPb6FENt7okTyZ5Mck/zHqWsSXZmeSZJOeSnE3y6KxnGlOS70zyt0n+bvX8fmPWM40tyZYkX0zyqVnPMmvGfUQDH9Vwq/sDYP+sh9gkV4EPVtUPAvcA72v25/cScG9V/QhwJ7A/yT0znmlsjwLnZj3EzcC4j+vVRzVU1cvAK49qaKOq/oqmn2Goqn+rqi+svv5PViKxfbZTjadWfGN1842rX21+oyLJDuAB4PdmPcvNwLiPa71HNbSJw+vJ6pNN7wKene0k41q9bfE88CLw6arqdH6/DfwS8M1ZD3IzMO7jGvQYBt3ckrwJ+DPgF6rq67OeZ0xV9b9VdScrnzTfl+Tts55pDEl+Cnixqp6b9Sw3C+M+riGPatBNLMkbWQn7n1TVn896ns1SVf8BfI4+Pz95J3AgyUVWbofem+SPZzvSbBn3cQ15VINuUknCyqetz1XVh2c9z9iSzCX5ntXX3wXcB/zjbKcaR1X9SlXtqKpdrPy9+2xVvXvGY82UcR9RVV0FXnlUwzngqao6O9upxpXk48DfAD+QZDnJI7OeaUTvBH6Olau+51e/fnLWQ43orcAzSb7EyoXIp6vqdf8rg135+AFJasgrd0lqyLhLUkPGXZIaMu6S1JBxl6SGjLskNWTcJamh/wOqoc/oAlft6gAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure()\n", "plt.bar(range(X.shape[1]), imp_vals)\n", "plt.xticks(range(X.shape[1]))\n", "plt.xlim([-1, X.shape[1]])\n", "plt.ylim([0, 0.5])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## API" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "## feature_importance_permutation\n", "\n", "*feature_importance_permutation(X, y, predict_method, metric, num_rounds=1, seed=None)*\n", "\n", "Feature importance imputation via permutation importance\n", "\n", "**Parameters**\n", "\n", "\n", "- `X` : NumPy array, shape = [n_samples, n_features]\n", "\n", " Dataset, where n_samples is the number of samples and\n", " n_features is the number of features.\n", "\n", "\n", "- `y` : NumPy array, shape = [n_samples]\n", "\n", " Target values.\n", "\n", "\n", "- `predict_method` : prediction function\n", "\n", " A callable function that predicts the target values\n", " from X.\n", "\n", "\n", "- `metric` : str, callable\n", "\n", " The metric for evaluating the feature importance through\n", " permutation. By default, the strings 'accuracy' is\n", " recommended for classifiers and the string 'r2' is\n", " recommended for regressors. Optionally, a custom\n", " scoring function (e.g., `metric=scoring_func`) that\n", " accepts two arguments, y_true and y_pred, which have\n", " similar shape to the `y` array.\n", "\n", "\n", "- `num_rounds` : int (default=1)\n", "\n", " Number of rounds the feature columns are permuted to\n", " compute the permutation importance.\n", "\n", "\n", "- `seed` : int or None (default=None)\n", "\n", " Random seed for permuting the feature columns.\n", "\n", "**Returns**\n", "\n", "\n", "- `mean_importance_vals, all_importance_vals` : NumPy arrays.\n", "\n", " The first array, mean_importance_vals has shape [n_features, ] and\n", " contains the importance values for all features.\n", " The shape of the second array is [n_features, num_rounds] and contains\n", " the feature importance for each repetition. If num_rounds=1,\n", " it contains the same values as the first array, mean_importance_vals.\n", "\n", "**Examples**\n", "\n", "For usage examples, please see\n", " [http://rasbt.github.io/mlxtend/user_guide/evaluate/feature_importance_permutation/](http://rasbt.github.io/mlxtend/user_guide/evaluate/feature_importance_permutation/)\n", "\n", "\n" ] } ], "source": [ "with open('../../api_modules/mlxtend.evaluate/feature_importance_permutation.md', 'r') as f:\n", " s = f.read() \n", "print(s)" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" }, "toc": { "nav_menu": {}, "number_sections": false, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": true } }, "nbformat": 4, "nbformat_minor": 1 }