{
 "metadata": {
  "name": "02B_feature_extraction"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Feature Extraction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here we will talk about an important piece of machine learning: the extraction of\n",
      "quantitative features from data.  By the end of this section you will\n",
      "\n",
      "- Know how features are extracted from real-world data.\n",
      "- See an example of extracting numerical features from textual data\n",
      "\n",
      "In addition, we will go over several basic tools within scikit-learn which can be used to accomplish the above tasks."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "What Are Features?"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Numerical Features"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Recall that data in scikit-learn is expected to be in two-dimensional arrays, of size\n",
      "**n_samples** $\\times$ **n_features**.\n",
      "\n",
      "Previously, we looked at the iris dataset, which has 150 samples and 4 features"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.datasets import load_iris\n",
      "iris = load_iris()\n",
      "print iris.data.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "These features are:\n",
      "\n",
      "- sepal length in cm\n",
      "- sepal width in cm\n",
      "- petal length in cm\n",
      "- petal width in cm\n",
      "\n",
      "Numerical features such as these are pretty straightforward: each sample contains a list\n",
      "of floating-point numbers corresponding to the features"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Categorical Features"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "What if you have categorical features?  For example, imagine there is data on the color of each\n",
      "iris:\n",
      "\n",
      "    color in [red, blue, purple]\n",
      "\n",
      "You might be tempted to assign numbers to these features, i.e. *red=1, blue=2, purple=3*\n",
      "but in general **this is a bad idea**.  Estimators tend to operate under the assumption that\n",
      "numerical features lie on some continuous scale, so, for example, 1 and 2 are more alike\n",
      "than 1 and 3, and this is often not the case for categorical features.\n",
      "\n",
      "A better strategy is to give each category its own dimension.  \n",
      "The enriched iris feature set would hence be in this case:\n",
      "\n",
      "- sepal length in cm\n",
      "- sepal width in cm\n",
      "- petal length in cm\n",
      "- petal width in cm\n",
      "- color=purple (1.0 or 0.0)\n",
      "- color=blue (1.0 or 0.0)\n",
      "- color=red (1.0 or 0.0)\n",
      "\n",
      "Note that using many of these categorical features may result in data which is better\n",
      "represented as a **sparse matrix**, as we'll see with the text classification example\n",
      "below."
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Using the DictVectorizer to encode categorical features"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When the source data is encoded has a list of dicts where the values are either strings names for categories or numerical values, you can use the `DictVectorizer` class to compute the boolean expansion of the categorical features while leaving the numerical features unimpacted:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "measurements = [\n",
      "    {'city': 'Dubai', 'temperature': 33.},\n",
      "    {'city': 'London', 'temperature': 12.},\n",
      "    {'city': 'San Fransisco', 'temperature': 18.},\n",
      "]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.feature_extraction import DictVectorizer\n",
      "\n",
      "vec = DictVectorizer()\n",
      "vec"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "vec.fit_transform(measurements).toarray()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "vec.get_feature_names()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Derived Features"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Another common feature type are **derived features**, where some pre-processing step is\n",
      "applied to the data to generate features that are somehow more informative.  Derived\n",
      "features may be based in **dimensionality reduction** (such as PCA or manifold learning),\n",
      "may be linear or nonlinear combinations of features (such as in Polynomial regression),\n",
      "or may be some more sophisticated transform of the features.  The latter is often used\n",
      "in image processing.\n",
      "\n",
      "For example, [scikit-image](http://scikit-image.org/) provides a variety of feature\n",
      "extractors designed for image data: see the ``skimage.feature`` submodule.\n",
      "We will see some *dimensionality*-based feature extraction routines later in the tutorial."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Text Feature Extraction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Unstructed content such as text documents require there own feature extraction step. In general we treat words in text documents as individual categorical features. An example on text mining will be introduced later, at the end of this session."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}