{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext watermark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka 14/01/2015 \n",
      "\n",
      "CPython 3.4.2\n",
      "IPython 2.3.1\n",
      "\n",
      "scikit-learn 0.15.2\n",
      "numpy 1.9.1\n",
      "pandas 0.15.2\n"
     ]
    }
   ],
   "source": [
    "%watermark -d -a 'Sebastian Raschka' -v -p scikit-learn,numpy,pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing a Weighted Majority Rule Ensemble Classifier in scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"width:500px;height:70px;border:1px solid #000;padding:10px;background-color:#e5ffe5;\"><p>If you are interested in using the <code>EnsembleClassifier</code>, please note that it is now also available through scikit learn (>0.17) as <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html\"><code>VotingClassifier</code></a>.</p></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, I want to present a simple and conservative approach of implementing a weighted majority rule ensemble classifier in [scikit-learn](http://scikit-learn.org/stable/) that yielded remarkably good results when I tried it in a [kaggle](http://www.kaggle.com) competition. For me personally, kaggle competitions are just a nice way to try out and compare different approaches and ideas -- basically an opportunity to learn in a controlled environment with nice datasets.  \n",
    "\n",
    "Of course, there are other implementations of more sophisticated [ensemble methods](http://scikit-learn.org/stable/modules/ensemble.html) in scikit-learn, such as [bagging classifiers](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html), [random forests](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), or the famous [AdaBoost](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html) algorithm. However, as far as I am concerned, they all require the usage of a common \"base classifier.\"\n",
    "\n",
    "In contrast, my motivation for the following approach was to combine conceptually different machine learning classifiers and use a majority vote rule. The reason for this was that I had trained a set of equally well performing models, and I wanted to balance out their individual weaknesses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Classifying Iris Flowers Using Different Classification Models](#Classifying-Iris-Flowers-Using-Different-Classification-Models)\n",
    "- [Implementing the Majority Voting Rule Ensemble Classifier](#Implementing-the-Majority-Voting-Rule-Ensemble-Classifier)\n",
    "- [Additional Note About the EnsembleClassifier Implementation: Class Labels vs. Probabilities](#Additional-Note-About-the-EnsembleClassifier-Implementation:-Class-Labels-vs.-Probabilities)\n",
    "- [EnsembleClassifier - Tuning Weights](#EnsembleClassifier---Tuning-Weights)\n",
    "- [EnsembleClassifier - Pipelines](#EnsembleClassifier---Pipelines)\n",
    "- [Some Final Words](#Some-Final-Words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifying Iris Flowers Using Different Classification Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a simple example, let us use three different classification models to classify the samples in the [Iris dataset](http://en.wikipedia.org/wiki/Iris_flower_data_set): Logistic regression, a naive Bayes classifier with a Gaussian kernel, and a random forest classifier -- an ensemble method itself. At this point, let's not worry about preprocessing the data and training and test sets. Also, we will only use 2 feature columns (sepal width and petal height) to make the classification problem harder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X, y = iris.data[:, 1:3], iris.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5-fold cross validation:\n",
      "\n",
      "Accuracy: 0.90 (+/- 0.05) [Logistic Regression]\n",
      "Accuracy: 0.92 (+/- 0.05) [Random Forest]\n",
      "Accuracy: 0.91 (+/- 0.04) [naive Bayes]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import cross_validation\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import GaussianNB \n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "import numpy as np\n",
    "\n",
    "np.random.seed(123)\n",
    "\n",
    "clf1 = LogisticRegression()\n",
    "clf2 = RandomForestClassifier()\n",
    "clf3 = GaussianNB()\n",
    "\n",
    "print('5-fold cross validation:\\n')\n",
    "\n",
    "for clf, label in zip([clf1, clf2, clf3], ['Logistic Regression', 'Random Forest', 'naive Bayes']):\n",
    "\n",
    "    scores = cross_validation.cross_val_score(clf, X, y, cv=5, scoring='accuracy')\n",
    "    print(\"Accuracy: %0.2f (+/- %0.2f) [%s]\" % (scores.mean(), scores.std(), label))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the cross-validation results above, the performance of the three models is almost equal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing the Majority Voting Rule Ensemble Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will implement a simple `EnsembleClassifier` class that allows us to combine the three different classifiers. We define a `predict` method that let's us simply take the majority rule of the predictions by the classifiers.\n",
    "E.g., if the prediction for a sample is\n",
    "\n",
    "- classifier 1 -> class 1\n",
    "- classifier 2 -> class 1\n",
    "- classifier 3 -> class 2\n",
    "\n",
    "we would classify the sample as \"class 1.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Furthermore, we add a `weights` parameter, which let's us assign a specific weight to each classifier. In order to work with the weights, we collect the predicted class probabilities for each classifier, multiply it by the classifier weight, and take the average. Based on these weighted average probabilties, we can then assign the class label.\n",
    "\n",
    "To illustrate this with a simple example, let's assume we have 3 classifiers and a 3-class classification problems where we assign equal weights to all classifiers (the default): w1=1, w2=1, w3=1.\n",
    "\n",
    "The weighted average probabilities for a sample would then be calculated as follows:\n",
    "\n",
    "| classifier      | class 1  | class 2  | class 3  |\n",
    "|-----------------|----------|----------|----------|\n",
    "| classifier 1    | w1 * 0.2 | w1 * 0.5 | w1 * 0.3 |\n",
    "| classifier 2    | w2 * 0.6 | w2 * 0.3 | w2 * 0.1 |\n",
    "| classifier 3    | w3 * 0.3 | w3 * 0.4 | w3 * 0.3 |\n",
    "| weighted average| 0.37     | 0.4      | 0.3      |\n",
    "\n",
    "We can see in the table above that class 2 has the highest weighted average probability, thus we classify the sample as class 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's put it into code and apply it to our Iris classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.base import ClassifierMixin\n",
    "import numpy as np\n",
    "import operator\n",
    "\n",
    "class EnsembleClassifier(BaseEstimator, ClassifierMixin):\n",
    "    \"\"\" \n",
    "    Ensemble classifier for scikit-learn estimators.\n",
    "        \n",
    "    Parameters\n",
    "    ----------\n",
    "    \n",
    "    clf : `iterable`\n",
    "      A list of scikit-learn classifier objects.\n",
    "    weights : `list` (default: `None`)\n",
    "      If `None`, the majority rule voting will be applied to the predicted class labels.\n",
    "        If a list of weights (`float` or `int`) is provided, the averaged raw probabilities (via `predict_proba`)\n",
    "        will be used to determine the most confident class label.\n",
    "      \n",
    "    \"\"\"\n",
    "    def __init__(self, clfs, weights=None):\n",
    "        self.clfs = clfs\n",
    "        self.weights = weights\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        \"\"\" \n",
    "        Fit the scikit-learn estimators.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "\n",
    "        X : numpy array, shape = [n_samples, n_features]\n",
    "            Training data\n",
    "        y : list or numpy array, shape = [n_samples]\n",
    "            Class labels\n",
    "      \n",
    "        \"\"\"\n",
    "        for clf in self.clfs:\n",
    "            clf.fit(X, y)\n",
    "            \n",
    "    def predict(self, X):\n",
    "        \"\"\"\n",
    "        Parameters\n",
    "        ----------\n",
    "\n",
    "        X : numpy array, shape = [n_samples, n_features]\n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        \n",
    "        maj : list or numpy array, shape = [n_samples]\n",
    "            Predicted class labels by majority rule\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        self.classes_ = np.asarray([clf.predict(X) for clf in self.clfs])\n",
    "        if self.weights:\n",
    "            avg = self.predict_proba(X)\n",
    "\n",
    "            maj = np.apply_along_axis(lambda x: max(enumerate(x), key=operator.itemgetter(1))[0], axis=1, arr=avg)\n",
    "        \n",
    "        else:\n",
    "            maj = np.asarray([np.argmax(np.bincount(self.classes_[:,c])) for c in range(self.classes_.shape[1])])\n",
    "        \n",
    "        return maj\n",
    "            \n",
    "    def predict_proba(self, X):\n",
    "        \n",
    "        \"\"\"\n",
    "        Parameters\n",
    "        ----------\n",
    "\n",
    "        X : numpy array, shape = [n_samples, n_features]\n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        \n",
    "        avg : list or numpy array, shape = [n_samples, n_probabilities]\n",
    "            Weighted average probability for each class per sample.\n",
    "        \n",
    "        \"\"\"\n",
    "        self.probas_ = [clf.predict_proba(X) for clf in self.clfs]\n",
    "        avg = np.average(self.probas_, axis=0, weights=self.weights)\n",
    "        \n",
    "        return avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.90 (+/- 0.05) [Logistic Regression]\n",
      "Accuracy: 0.92 (+/- 0.05) [Random Forest]\n",
      "Accuracy: 0.91 (+/- 0.04) [naive Bayes]\n",
      "Accuracy: 0.95 (+/- 0.03) [Ensemble]\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(123)\n",
    "eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[1,1,1])\n",
    "\n",
    "for clf, label in zip([clf1, clf2, clf3, eclf], ['Logistic Regression', 'Random Forest', 'naive Bayes', 'Ensemble']):\n",
    "\n",
    "    scores = cross_validation.cross_val_score(clf, X, y, cv=5, scoring='accuracy')\n",
    "    print(\"Accuracy: %0.2f (+/- %0.2f) [%s]\" % (scores.mean(), scores.std(), label))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Note About the EnsembleClassifier Implementation: Class Labels vs. Probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might be wondering why I implemented the `EnsembleClassifier` class so that it applies the majority voting purely on the class labels if no weights are provided and is the predicted probability values otherwise. \n",
    "\n",
    "Let's consider the following scenario:\n",
    "\n",
    "#### 1) Prediction based on majority class labels:\n",
    "\n",
    "\n",
    "| classifier      | class 1  | class 2  | \n",
    "|-----------------|----------|----------|\n",
    "| classifier 1    |     1 |      0|  \n",
    "| classifier 2    |     0 |      1| \n",
    "| classifier 3    |     0 |      1| \n",
    "| prediction      |     - |       1|\n",
    "\n",
    "To achieve this behavior, initialize the `EnsembleClassifier` like this: \n",
    "\n",
    "    eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3])\n",
    "\n",
    "\n",
    "\n",
    "#### 2) Prediction based on predicted probabilities (equal weights, `weights=[1,1,1]`)\n",
    "\n",
    "| classifier      | class 1  | class 2  | \n",
    "|-----------------|----------|----------|\n",
    "| classifier 1    |     0.99 |      0.01|  \n",
    "| classifier 2    |     0.49 |      0.51| \n",
    "| classifier 3    |     0.49 |      0.51| \n",
    "| weighted average| 0.66     | 0.18     |\n",
    "| prediction      | 1        | -        |\n",
    "\n",
    "To achieve this behavior, initialize the `EnsembleClassifier` like this: \n",
    "\n",
    "    eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[1,1,1])\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the results are different depending on whether we apply a majority vote based on the class labels or take the average of the predicted probabilities. In general, I think it makes more sense to use the predicted probabilities (scenario 2). Here, the \"very confident\" classifier 1 overules the very unconfident classifiers 2 and 3. \n",
    "\n",
    "The reason for the different behaviors is that not all classifiers in scikit-learn support the `predict_proba` method. In this case, the `EnsembleClassifier` can still be used just based on the class labels if no weights are provided as parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EnsembleClassifier - Tuning Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get back to our `weights` parameter. Here, we will use a naive brute-force approach to find the optimal weights for each classifier to increase the prediction accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>w1</th>\n",
       "      <th>w2</th>\n",
       "      <th>w3</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2 </th>\n",
       "      <td> 1</td>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.953333</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td> 3</td>\n",
       "      <td> 1</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0.953333</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td> 3</td>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.045216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td> 3</td>\n",
       "      <td> 2</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.045216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1 </th>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 3</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.040000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6 </th>\n",
       "      <td> 1</td>\n",
       "      <td> 3</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7 </th>\n",
       "      <td> 1</td>\n",
       "      <td> 3</td>\n",
       "      <td> 3</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td> 2</td>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td> 2</td>\n",
       "      <td> 3</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td> 2</td>\n",
       "      <td> 3</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td> 3</td>\n",
       "      <td> 1</td>\n",
       "      <td> 3</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td> 3</td>\n",
       "      <td> 3</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td> 3</td>\n",
       "      <td> 3</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0.946667</td>\n",
       "      <td> 0.033993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td> 3</td>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.057349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5 </th>\n",
       "      <td> 1</td>\n",
       "      <td> 3</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.044222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8 </th>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.044222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9 </th>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.044222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td> 2</td>\n",
       "      <td> 2</td>\n",
       "      <td> 3</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.044222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td> 3</td>\n",
       "      <td> 2</td>\n",
       "      <td> 3</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.044222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4 </th>\n",
       "      <td> 1</td>\n",
       "      <td> 2</td>\n",
       "      <td> 3</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.038873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3 </th>\n",
       "      <td> 1</td>\n",
       "      <td> 2</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.032660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "      <td> 3</td>\n",
       "      <td> 0.940000</td>\n",
       "      <td> 0.032660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0 </th>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0.933333</td>\n",
       "      <td> 0.047140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td> 2</td>\n",
       "      <td> 3</td>\n",
       "      <td> 3</td>\n",
       "      <td> 0.933333</td>\n",
       "      <td> 0.047140</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    w1  w2  w3      mean       std\n",
       "2    1   2   1  0.953333  0.033993\n",
       "17   3   1   2  0.953333  0.033993\n",
       "16   3   1   1  0.946667  0.045216\n",
       "20   3   2   2  0.946667  0.045216\n",
       "1    1   1   3  0.946667  0.040000\n",
       "6    1   3   2  0.946667  0.033993\n",
       "7    1   3   3  0.946667  0.033993\n",
       "11   2   2   1  0.946667  0.033993\n",
       "13   2   3   1  0.946667  0.033993\n",
       "14   2   3   2  0.946667  0.033993\n",
       "18   3   1   3  0.946667  0.033993\n",
       "22   3   3   1  0.946667  0.033993\n",
       "23   3   3   2  0.946667  0.033993\n",
       "19   3   2   1  0.940000  0.057349\n",
       "5    1   3   1  0.940000  0.044222\n",
       "8    2   1   1  0.940000  0.044222\n",
       "9    2   1   2  0.940000  0.044222\n",
       "12   2   2   3  0.940000  0.044222\n",
       "21   3   2   3  0.940000  0.044222\n",
       "4    1   2   3  0.940000  0.038873\n",
       "3    1   2   2  0.940000  0.032660\n",
       "10   2   1   3  0.940000  0.032660\n",
       "0    1   1   2  0.933333  0.047140\n",
       "15   2   3   3  0.933333  0.047140"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "np.random.seed(123)\n",
    "\n",
    "df = pd.DataFrame(columns=('w1', 'w2', 'w3', 'mean', 'std'))\n",
    "\n",
    "i = 0\n",
    "for w1 in range(1,4):\n",
    "    for w2 in range(1,4):\n",
    "        for w3 in range(1,4):\n",
    "            \n",
    "            if len(set((w1,w2,w3))) == 1: # skip if all weights are equal\n",
    "                continue\n",
    "            \n",
    "            eclf = EnsembleClassifier(clfs=[clf1, clf2, clf3], weights=[w1,w2,w3])\n",
    "            scores = cross_validation.cross_val_score(\n",
    "                                            estimator=eclf,\n",
    "                                            X=X, \n",
    "                                            y=y, \n",
    "                                            cv=5, \n",
    "                                            scoring='accuracy',\n",
    "                                            n_jobs=1)\n",
    "            \n",
    "            df.loc[i] = [w1, w2, w3, scores.mean(), scores.std()]\n",
    "            i += 1\n",
    "            \n",
    "df.sort(columns=['mean', 'std'], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EnsembleClassifier - Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, we can also use the `EnsembleClassifier` in `Pipelines`. This is especially useful if a certain classifier does a pretty good job on a certain feature subset or requires different `preprocessing` steps. For demonstration purposes, let us implement a simple `ColumnSelector` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class ColumnSelector(object):\n",
    "    \"\"\" \n",
    "    A feature selector for scikit-learn's Pipeline class that returns\n",
    "    specified columns from a numpy array.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, cols):\n",
    "        self.cols = cols\n",
    "        \n",
    "    def transform(self, X, y=None):\n",
    "        return X[:, self.cols]\n",
    "\n",
    "    def fit(self, X, y=None):\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.95 (+/- 0.03) [Ensemble]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.lda import LDA\n",
    "\n",
    "pipe1 = Pipeline([\n",
    "               ('sel', ColumnSelector([1])),    # use only the 1st feature\n",
    "               ('clf', GaussianNB())])\n",
    "\n",
    "pipe2 = Pipeline([\n",
    "               ('sel', ColumnSelector([0, 1])), # use the 1st and 2nd feature\n",
    "               ('dim', LDA(n_components=1)),    # Dimensionality reduction via LDA\n",
    "               ('clf', LogisticRegression())])\n",
    "\n",
    "eclf = EnsembleClassifier([pipe1, pipe2])\n",
    "scores = cross_validation.cross_val_score(eclf, X, y, cv=5, scoring='accuracy')\n",
    "print(\"Accuracy: %0.2f (+/- %0.2f) [%s]\" % (scores.mean(), scores.std(), label))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pipe1 = Pipeline([\n",
    "               ('sel', ColumnSelector([1])), # use only the 1st feature\n",
    "               ('clf', RandomForestClassifier())])\n",
    "\n",
    "pipe2 = Pipeline([\n",
    "               ('sel', ColumnSelector([0, 1])), # use the 1st and 2nd feature\n",
    "               ('dim', LDA(n_components=1)), # Dimensionality reduction via LDA\n",
    "               ('clf', LogisticRegression())])\n",
    "\n",
    "pipe3 = Pipeline([\n",
    "               ('eclf', EnsembleClassifier([pipe1, pipe2])),\n",
    "])\n",
    "parameters = {\n",
    "'eclf__clfs__dim__n_components':(1,1),\n",
    "}\n",
    "grid_search = GridSearchCV(pipe3, parameters, n_jobs=-1, cv=5, verbose=5, refit=True, scoring=None)\n",
    "grid_search.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Final Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[back to top](#Sections)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we applied the `EnsembleClassifier` to the iris example above, the results surely looked nice. But we have to keep in mind that this is just a toy example. The majority rule voting approach might not always work so well in practice, especially if the ensemble consists of more \"weak\" than \"strong\" classification models. Also, although we used a cross-validation approach to overcome the overfitting challenge, please always keep a spare validation dataset to evaluate the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anyway, if you are interested in those approaches, I added them to my [`mlxtend`](http://rasbt.github.io/mlxtend/) Python module; in `mlxtend` (short for \"machine learning library extensions\"), I collect certain things that I personally find useful but are not available in other packages yet.\n",
    "\n",
    "You can find the most up to date documentation at [http://rasbt.github.io/mlxtend/](http://rasbt.github.io/mlxtend/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}