\n",
"\n",
" | \n",
" Name | \n",
" Description | \n",
"
\n",
"\n",
" | 1 | Year | 1987-2008 | \n",
"
\n",
" | 2 | Month | 1-12 | \n",
"
\n",
" | 3 | DayofMonth | 1-31 | \n",
"
\n",
" | 4 | DayOfWeek | 1 (Monday) - 7 (Sunday) | \n",
"
\n",
" | 5 | DepTime | actual departure time (local, hhmm) | \n",
"
\n",
" | 6 | CRSDepTime | scheduled departure time (local, hhmm) | \n",
"
\n",
" | 7 | ArrTime | actual arrival time (local, hhmm) | \n",
"
\n",
" | 8 | CRSArrTime | scheduled arrival time (local, hhmm) | \n",
"
\n",
" | 9 | UniqueCarrier | unique carrier code | \n",
"
\n",
" | 10 | FlightNum | flight number | \n",
"
\n",
" | 11 | TailNum | plane tail number | \n",
"
\n",
" | 12 | ActualElapsedTime | in minutes | \n",
"
\n",
" | 13 | CRSElapsedTime | in minutes | \n",
"
\n",
" | 14 | AirTime | in minutes | \n",
"
\n",
" | 15 | ArrDelay | arrival delay, in minutes | \n",
"
\n",
" | 16 | DepDelay | departure delay, in minutes | \n",
"
\n",
" | 17 | Origin | origin - IATA airport code | \n",
"
\n",
" | 18 | Dest | destination - IATA airport code | \n",
"
\n",
" | 19 | Distance | in miles | \n",
"
\n",
" | 20 | TaxiIn | taxi in time, in minutes | \n",
"
\n",
" | 21 | TaxiOut | taxi out time in minutes | \n",
"
\n",
" | 22 | Cancelled | was the flight cancelled? | \n",
"
\n",
" | 23 | CancellationCode | reason for cancellation (A = carrier, B = weather, C = NAS, D = security) | \n",
"
\n",
" | 24 | Diverted | 1 = yes, 0 = no | \n",
"
\n",
" | 25 | CarrierDelay | in minutes | \n",
"
\n",
" | 26 | WeatherDelay | in minutes | \n",
"
\n",
" | 27 | NASDelay | in minutes | \n",
"
\n",
" | 28 | SecurityDelay | in minutes | \n",
"
\n",
" | 29 | LateAircraftDelay | in minutes | \n",
"
\n",
"
\n",
"\n",
"To simplify, we will build a supervised learning model to predict flight delays for flights leaving O'Hare International airport (ORD), where we \"learn\" the model using data from 2007, and evaluate its performance using data from 2008.\n",
"\n",
"But first, let's do some exploration of this dataset.\n",
"\n",
"We start by importing some useful python libraries that we will need later like Pandas, Numpy, Scikit-learn and MatplotLib."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# Python library imports: numpy, random, sklearn, pandas, etc\n",
"\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"import sys\n",
"import random\n",
"import numpy as np\n",
"import scipy as sp\n",
"\n",
"from sklearn import linear_model, cross_validation, metrics, svm\n",
"from sklearn.metrics import confusion_matrix, precision_recall_fscore_support, accuracy_score\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We now define a utility function to read an HDFS file into a Pandas dataframe using Pydoop. Pydoop is a package that provides a Python API for Hadoop MapReduce and HDFS.\n",
"\n",
"Pydoop's *hdfs.open()* function reads a single file from HDFS. However many HDFS output files are actually multi-part files, so our *read_csv_from_hdfs()* function uses *hdfs.ls()* to grab all the needed file names, and then read each one separately. Finally, it concatenates the resulting Pandas dataframes of each file into a Pandas dataframe."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# function to read HDFS file into dataframe using PyDoop\n",
"import pydoop.hdfs as hdfs\n",
"def read_csv_from_hdfs(path, cols, col_types=None):\n",
" files = hdfs.ls(path);\n",
" pieces = []\n",
" for f in files:\n",
" fhandle = hdfs.open(f)\n",
" pieces.append(pd.read_csv(fhandle, names=cols, dtype=col_types, error_bad_lines=False))\n",
" fhandle.close()\n",
" return pd.concat(pieces, ignore_index=True)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 2
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Great. Now we got the logistics out of the way, so let's explore this dataset further. \n",
"\n",
"First, let's read the raw data for 2007 from HDFS into a Pandas dataframe. We use our utility function *read_csv_from_hdfs()* and provide it with column names since this is a raw file, not a HIVE table with meta-data. Let's see how it works:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# read 2007 year file\n",
"cols = ['year', 'month', 'day', 'dow', 'DepTime', 'CRSDepTime', 'ArrTime', 'CRSArrTime', 'Carrier', 'FlightNum', \n",
" 'TailNum', 'ActualElapsedTime', 'CRSElapsedTime', 'AirTime', 'ArrDelay', 'DepDelay', 'Origin', 'Dest', \n",
" 'Distance', 'TaxiIn', 'TaxiOut', 'Cancelled', 'CancellationCode', 'Diverted', 'CarrierDelay', \n",
" 'WeatherDelay', 'NASDelay', 'SecurityDelay', 'LateAircraftDelay'];\n",
"flt_2007 = read_csv_from_hdfs('airline/delay/2007.csv', cols)\n",
"\n",
"flt_2007.shape"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 3,
"text": [
"(7453216, 29)"
]
}
],
"prompt_number": 3
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see 7.4M+ flights in 2007 and 29 variables.\n",
"\n",
"Our \"target\" variable will be *DepDelay* (scheduled departure delay in minutes). To build a classifier, we further refine our target variable into a binary variable by defining a \"delay\" as having 15 mins or more of delay, and \"non-delay\" otherwise. We thus create a new binary variable that we name *'DepDelayed'*.\n",
"\n",
"Let's look at some basic statistics, after limiting ourselves to flights originating from ORD:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"df = flt_2007[flt_2007['Origin']=='ORD'].dropna(subset=['DepDelay'])\n",
"df['DepDelayed'] = df['DepDelay'].apply(lambda x: x>=15)\n",
"print \"total flights: \" + str(df.shape[0])\n",
"print \"total delays: \" + str(df['DepDelayed'].sum())"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"total flights: 359169\n",
"total delays: 109346\n"
]
}
],
"prompt_number": 4
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's see how delayed flights are distributed by month:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# Select a Pandas dataframe with flight originating from ORD\n",
"\n",
"# Compute average number of delayed flights per month\n",
"grouped = df[['DepDelayed', 'month']].groupby('month').mean()\n",
"\n",
"# plot average delays by month\n",
"grouped.plot(kind='bar')"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 5,
"text": [
"
\n",
"\n",
"Let's begin by exploring the airline delay dataset available here: http://stat-computing.org/dataexpo/2009/the-data.html\n",
"This dataset includes details about flights in the US from the years 1987-2008. Every row in the dataset includes 29 variables:\n",
"