{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Supervised Learning: Regression of Housing Data"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here we'll do a short example of a regression problem: learning a continuous value\n",
      "from a set of features.\n",
      "\n",
      "We'll use the simple Boston house prices set, available in scikit-learn.  This\n",
      "records measurements of 13 attributes of housing markets around Boston, as well\n",
      "as the median price.  The question is: can you predict the price of a new\n",
      "market given its attributes?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.datasets import load_boston\n",
      "data = load_boston()\n",
      "print(data.data.shape)\n",
      "print(data.target.shape)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can see that there are just over 500 data points.\n",
      "\n",
      "The ``DESCR`` variable has a long description of the dataset:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(data.DESCR)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It often helps to quickly visualize pieces of the data using histograms, scatter plots,\n",
      "or other plot types.  Here we'll load pylab and show a histogram of the target values:\n",
      "the median price in each neighborhood."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline\n",
      "import matplotlib.pyplot as plt\n",
      "import numpy as np"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.hist(data.target)\n",
      "plt.xlabel('price ($1000s)')\n",
      "plt.ylabel('count')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's have a quick look to see if some features are more relevant than others for our problem"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for index, feature_name in enumerate(data.feature_names):\n",
      "    plt.figure()\n",
      "    plt.scatter(data.data[:, index], data.target)\n",
      "    plt.ylabel('Price')\n",
      "    plt.xlabel(feature_name)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This is a manual version of a technique called **feature selection**.\n",
      "\n",
      "Sometimes, in Machine Learning it is useful to use \n",
      "feature selection to decide which features are most useful for a\n",
      "particular problem.  Automated methods exist which quantify this sort\n",
      "of exercise of choosing the most informative features."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Predicting Home Prices: a Simple Linear Regression"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we'll use ``scikit-learn`` to perform a simple linear regression\n",
      "on the housing data.  There are many possibilities of regressors to\n",
      "use. A particularly simple one is ``LinearRegression``: this is\n",
      "basically a wrapper around an ordinary least squares calculation.\n",
      "\n",
      "We'll set it up like this:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.cross_validation import train_test_split\n",
      "\n",
      "X_train, X_test, y_train, y_test = train_test_split(data.data, data.target)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.linear_model import LinearRegression\n",
      "\n",
      "clf = LinearRegression()\n",
      "clf.fit(X_train, y_train)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "predicted = clf.predict(X_test)\n",
      "expected = y_test"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.scatter(expected, predicted)\n",
      "plt.plot([0, 50], [0, 50], '--k')\n",
      "plt.axis('tight')\n",
      "plt.xlabel('True price ($1000s)')\n",
      "plt.ylabel('Predicted price ($1000s)')\n",
      "print(\"RMS:\", np.sqrt(np.mean((predicted - expected) ** 2)))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The prediction at least correlates with the true price, though there\n",
      "are clearly some biases.  We could imagine evaluating the performance\n",
      "of the regressor by, say, computing the RMS residuals between the\n",
      "true and predicted price.  There are some subtleties in this, however,\n",
      "which we'll cover in a later section."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Exercise: Gradient Boosting Tree Regression"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are many other types of regressors available in scikit-learn:\n",
      "we'll try a more powerful one here.\n",
      "\n",
      "**Use the GradientBoostingRegressor class to fit the housing data**.\n",
      "\n",
      "You can copy and paste some of the above code, replacing `LinearRegression`\n",
      "with `GradientBoostingRegressor`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.ensemble import GradientBoostingRegressor\n",
      "# Instantiate the model, fit the results, and scatter in vs. out"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Solution:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load solutions/05B_houses_regression.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}