{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This is one of the 100 recipes of the [IPython Cookbook](http://ipython-books.github.io/), the definitive guide to high-performance scientific computing and data science in Python.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8.5. Using Support Vector Machines for classification tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Let's do the traditional imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn\n",
    "import sklearn.datasets as ds\n",
    "import sklearn.cross_validation as cv\n",
    "import sklearn.grid_search as gs\n",
    "import sklearn.svm as svm\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. We generate 2D points and assign a binary label according to a linear operation on the coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = np.random.randn(200, 2)\n",
    "y = X[:, 0] + X[:, 1] > 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. We now fit a linear **Support Vector Classifier** (SVC). This classifier tries to separate the two groups of points with a linear boundary (a line here, more generally a hyperplane)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We train the classifier.\n",
    "est = svm.LinearSVC()\n",
    "est.fit(X, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. We define a function that displays the boundaries and decision function of a trained classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We generate a grid in the square [-3,3 ]^2.\n",
    "xx, yy = np.meshgrid(np.linspace(-3, 3, 500),\n",
    "                     np.linspace(-3, 3, 500))\n",
    "# This function takes a SVM estimator as input.\n",
    "def plot_decision_function(est):\n",
    "    # We evaluate the decision function on the grid.\n",
    "    Z = est.decision_function(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    cmap = plt.cm.Blues\n",
    "    # We display the decision function on the grid.\n",
    "    plt.figure(figsize=(5,5));\n",
    "    plt.imshow(Z,\n",
    "                extent=(xx.min(), xx.max(), yy.min(), yy.max()),\n",
    "                aspect='auto', origin='lower', cmap=cmap);\n",
    "    # We display the boundaries.\n",
    "    plt.contour(xx, yy, Z, levels=[0], linewidths=2,\n",
    "                colors='k');\n",
    "    # We display the points with their true labels.\n",
    "    plt.scatter(X[:, 0], X[:, 1], s=30, c=.5+.5*y, lw=1, \n",
    "                cmap=cmap, vmin=0, vmax=1);\n",
    "    plt.axhline(0, color='k', ls='--');\n",
    "    plt.axvline(0, color='k', ls='--');\n",
    "    plt.xticks(());\n",
    "    plt.yticks(());\n",
    "    plt.axis([-3, 3, -3, 3]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Let's take a look at the classification results with the linear SVC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot_decision_function(est);\n",
    "plt.title(\"Linearly separable, linear SVC\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The linear SVC tried to separate the points with a line and it did a pretty good job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. We now modify the labels with a *XOR* function. A point's label is 1 if the coordinates have different signs. This classification is not linearly separable. Therefore, a linear SVC fails completely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = np.logical_xor(X[:, 0] > 0, X[:, 1] > 0)\n",
    "# We train the classifier.\n",
    "est = gs.GridSearchCV(svm.LinearSVC(), \n",
    "                      {'C': np.logspace(-3., 3., 10)});\n",
    "est.fit(X, y);\n",
    "print(\"Score: {0:.1f}\".format(\n",
    "      cv.cross_val_score(est, X, y).mean()))\n",
    "# Plot the decision function.\n",
    "plot_decision_function(est);\n",
    "plt.title(\"XOR, linear SVC\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. Fortunately, it is possible to use non-linear SVCs by using non-linear **kernels**. Kernels specify a non-linear transformation of the points into a higher-dimensional space. Transformed points in this space are assumed to be more linearly separable, although they are not necessarily in the original space. By default, the `SVC` classifier in scikit-learn uses the **Radial Basis Function** (RBF) kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = np.logical_xor(X[:, 0] > 0, X[:, 1] > 0)\n",
    "est = gs.GridSearchCV(svm.SVC(), \n",
    "                      {'C': np.logspace(-3., 3., 10),\n",
    "                    'gamma': np.logspace(-3., 3., 10)});\n",
    "est.fit(X, y);\n",
    "print(\"Score: {0:.3f}\".format(\n",
    "      cv.cross_val_score(est, X, y).mean()))\n",
    "plot_decision_function(est.best_estimator_);\n",
    "plt.title(\"XOR, non-linear SVC\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, the non-linear SVC does a pretty good job at classifying these non-linearly separable points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).\n",
    "\n",
    "> [IPython Cookbook](http://ipython-books.github.io/), by [Cyrille Rossant](http://cyrille.rossant.net), Packt Publishing, 2014 (500 pages)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}