{
"metadata": {
"name": ""
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Diving into Open Data with IPython Notebook & Pandas\n",
"\n",
"# I'm Julia Evans\n",
"\n",
"[http://twitter.com/b0rk](http://twitter.com/b0rk)\n",
"\n",
"[http://jvns.ca](http://jvns.ca)\n",
"\n",
"[http://github.com/jvns](http://github.com/jvns)\n",
"\n",
"Data scientist, programmer, co-organize [Montr\u00e9al All-Girl Hack Night](http://mtlallgirlhacknight.ca), [PyLadies MTL](http://meetup.com/pyladiesmtl)\n",
"\n",
"You can follow along with this talk at: \n",
"\n",
"[http://bit.ly/pyconca-pandas](http://bit.ly/pyconca-pandas)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# IPython Notebook\n",
"\n",
"* web-based user interface to IPython\n",
"* pretty graphs\n",
"* literate programming\n",
"* Can make slideshows :) (this presentation)\n",
"* version controlled science!\n",
"\n",
"#Pandas\n",
"\n",
"* \"R for Python\"\n",
"* Provides easy to use data structures & a ton of useful helper functions for data cleanup and transformations\n",
"* Fast! (backed by numpy arrays)\n",
"* integrates well with scikit-learn\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# An installation warning\n",
"\n",
"## Don't: Use the Ubuntu packages\n",
"\n",
"\n",
"sudo apt-get install ipython-notebook
\n",
"sudo apt-get install python-pandas\n",
"\n",
"\n",
"## Do: Use pip or [Anaconda](https://store.continuum.io/)\n",
"\n",
"\n",
"pip install ipython tornado pyzmq
\n",
"pip install pandas\n",
"\n",
"\n",
"## Anaconda is amazing."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# How to run IPython Notebook\n",
"\n",
"\n",
"\n",
"$ ipython notebook --pylab inline \n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## The open data\n",
"\n",
"Taken from [http://donnees.ville.montreal.qc.ca/fiche/velos-comptage/](http://donnees.ville.montreal.qc.ca/) (click \"V\u00e9los - comptage\")\n",
"\n",
"Number of people per day on 7 bike paths (collected using sensors)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Part 1: Import the 2012 bike path data from a CSV\n",
"## Before"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Download and unzip the zip file from \n",
"this page to run this yourself. "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import pandas as pd"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"prompt_number": 1
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# some display options to make figures bigger\n",
"# hide this\n",
"pd.set_option('display.max_columns', 15)\n",
"pd.set_option('display.line_width', 400)\n",
"pd.set_option('display.mpl_style', 'default')\n",
"rcParams['figure.figsize'] = (14, 7)\n",
"import matplotlib\n",
"font = {'family' : 'normal',\n",
" 'weight' : 'bold',\n",
" 'size' : 22}\n",
"\n",
"matplotlib.rc('font', **font)"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"prompt_number": 2
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"bike_data = pd.read_csv(\"./2012.csv\")\n",
"bike_data[:5]"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Date;Berri 1;Br\ufffdbeuf (donn\ufffdes non disponibles);C\ufffdte-Sainte-Catherine;Maisonneuve 1;Maisonneuve 2;du Parc;Pierre-Dupuy;Rachel1;St-Urbain (donn\ufffdes non disponibles) | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 01/01/2012;35;;0;38;51;26;10;16; | \n",
"
\n",
" \n",
" | 1 | \n",
" 02/01/2012;83;;1;68;153;53;6;43; | \n",
"
\n",
" \n",
" | 2 | \n",
" 03/01/2012;135;;2;104;248;89;3;58; | \n",
"
\n",
" \n",
" | 3 | \n",
" 04/01/2012;144;;1;116;318;111;8;61; | \n",
"
\n",
" \n",
" | 4 | \n",
" 05/01/2012;197;;2;124;330;97;13;95; | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Berri 1 | \n",
" C\u00f4te-Sainte-Catherine | \n",
" Maisonneuve 1 | \n",
"
\n",
" \n",
" | Date | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2012-01-01 | \n",
" 35 | \n",
" 0 | \n",
" 38 | \n",
"
\n",
" \n",
" | 2012-01-02 | \n",
" 83 | \n",
" 1 | \n",
" 68 | \n",
"
\n",
" \n",
" | 2012-01-03 | \n",
" 135 | \n",
" 2 | \n",
" 104 | \n",
"
\n",
" \n",
" | 2012-01-04 | \n",
" 144 | \n",
" 1 | \n",
" 116 | \n",
"
\n",
" \n",
" | 2012-01-05 | \n",
" 197 | \n",
" 2 | \n",
" 124 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Berri 1 | \n",
" C\u00f4te-Sainte-Catherine | \n",
" Maisonneuve 1 | \n",
"
\n",
" \n",
" | Date | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2012-01-01 | \n",
" 35 | \n",
" 0 | \n",
" 38 | \n",
"
\n",
" \n",
" | 2012-01-02 | \n",
" 83 | \n",
" 1 | \n",
" 68 | \n",
"
\n",
" \n",
" | 2012-01-03 | \n",
" 135 | \n",
" 2 | \n",
" 104 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Berri 1 | \n",
" Maisonneuve 1 | \n",
"
\n",
" \n",
" | Date | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2012-01-01 | \n",
" 35 | \n",
" 38 | \n",
"
\n",
" \n",
" | 2012-01-02 | \n",
" 83 | \n",
" 68 | \n",
"
\n",
" \n",
" | 2012-01-03 | \n",
" 135 | \n",
" 104 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Berri 1 | \n",
" C\u00f4te-Sainte-Catherine | \n",
" Maisonneuve 1 | \n",
" weekday | \n",
"
\n",
" \n",
" | Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2012-01-01 | \n",
" 35 | \n",
" 0 | \n",
" 38 | \n",
" 6 | \n",
"
\n",
" \n",
" | 2012-01-02 | \n",
" 83 | \n",
" 1 | \n",
" 68 | \n",
" 0 | \n",
"
\n",
" \n",
" | 2012-01-03 | \n",
" 135 | \n",
" 2 | \n",
" 104 | \n",
" 1 | \n",
"
\n",
" \n",
" | 2012-01-04 | \n",
" 144 | \n",
" 1 | \n",
" 116 | \n",
" 2 | \n",
"
\n",
" \n",
" | 2012-01-05 | \n",
" 197 | \n",
" 2 | \n",
" 124 | \n",
" 3 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Berri 1 | \n",
" C\u00f4te-Sainte-Catherine | \n",
" Maisonneuve 1 | \n",
"
\n",
" \n",
" | weekday | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 134298 | \n",
" 60329 | \n",
" 90051 | \n",
"
\n",
" \n",
" | 1 | \n",
" 135305 | \n",
" 58708 | \n",
" 92035 | \n",
"
\n",
" \n",
" | 2 | \n",
" 152972 | \n",
" 67344 | \n",
" 104891 | \n",
"
\n",
" \n",
" | 3 | \n",
" 160131 | \n",
" 69028 | \n",
" 111895 | \n",
"
\n",
" \n",
" | 4 | \n",
" 141771 | \n",
" 56446 | \n",
" 98568 | \n",
"
\n",
" \n",
" | 5 | \n",
" 101578 | \n",
" 34018 | \n",
" 62067 | \n",
"
\n",
" \n",
" | 6 | \n",
" 99310 | \n",
" 36466 | \n",
" 55324 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Dew Point Temp (C) | \n",
" Rel Hum (%) | \n",
" Stn Press (kPa) | \n",
" Temp (C) | \n",
" Visibility (km) | \n",
" Weather | \n",
" Wind Spd (km/h) | \n",
"
\n",
" \n",
" | Date/Time | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2012-01-01 00:00:00 | \n",
" -3.9 | \n",
" 86 | \n",
" 101.24 | \n",
" -1.8 | \n",
" 8.0 | \n",
" Fog | \n",
" 4 | \n",
"
\n",
" \n",
" | 2012-01-01 01:00:00 | \n",
" -3.7 | \n",
" 87 | \n",
" 101.24 | \n",
" -1.8 | \n",
" 8.0 | \n",
" Fog | \n",
" 4 | \n",
"
\n",
" \n",
" | 2012-01-01 02:00:00 | \n",
" -3.4 | \n",
" 89 | \n",
" 101.26 | \n",
" -1.8 | \n",
" 4.0 | \n",
" Freezing Drizzle,Fog | \n",
" 7 | \n",
"
\n",
" \n",
" | 2012-01-01 03:00:00 | \n",
" -3.2 | \n",
" 88 | \n",
" 101.27 | \n",
" -1.5 | \n",
" 4.0 | \n",
" Freezing Drizzle,Fog | \n",
" 6 | \n",
"
\n",
" \n",
" | 2012-01-01 04:00:00 | \n",
" -3.3 | \n",
" 88 | \n",
" 101.23 | \n",
" -1.5 | \n",
" 4.8 | \n",
" Fog | \n",
" 7 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Berri 1 | \n",
" C\u00f4te-Sainte-Catherine | \n",
" Maisonneuve 1 | \n",
" weekday | \n",
" mean temp | \n",
"
\n",
" \n",
" | Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2012-01-01 | \n",
" 35 | \n",
" 0 | \n",
" 38 | \n",
" 6 | \n",
" 0.629167 | \n",
"
\n",
" \n",
" | 2012-01-02 | \n",
" 83 | \n",
" 1 | \n",
" 68 | \n",
" 0 | \n",
" 0.041667 | \n",
"
\n",
" \n",
" | 2012-01-03 | \n",
" 135 | \n",
" 2 | \n",
" 104 | \n",
" 1 | \n",
" -14.416667 | \n",
"
\n",
" \n",
" | 2012-01-04 | \n",
" 144 | \n",
" 1 | \n",
" 116 | \n",
" 2 | \n",
" -13.645833 | \n",
"
\n",
" \n",
" | 2012-01-05 | \n",
" 197 | \n",
" 2 | \n",
" 124 | \n",
" 3 | \n",
" -6.750000 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Berri 1 | \n",
" Rain | \n",
" mean temp | \n",
" weekday | \n",
"
\n",
" \n",
" | Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2012-05-01 | \n",
" 1986 | \n",
" 0.416667 | \n",
" 9.437500 | \n",
" 1 | \n",
"
\n",
" \n",
" | 2012-05-08 | \n",
" 1241 | \n",
" 0.666667 | \n",
" 12.645833 | \n",
" 1 | \n",
"
\n",
" \n",
" | 2012-05-22 | \n",
" 2315 | \n",
" 0.583333 | \n",
" 18.279167 | \n",
" 1 | \n",
"
\n",
" \n",
" | 2012-06-02 | \n",
" 943 | \n",
" 0.583333 | \n",
" 13.566667 | \n",
" 5 | \n",
"
\n",
" \n",
" | 2012-06-25 | \n",
" 2245 | \n",
" 0.208333 | \n",
" 17.270833 | \n",
" 0 | \n",
"
\n",
" \n",
" | 2012-08-05 | \n",
" 1864 | \n",
" 0.166667 | \n",
" 25.783333 | \n",
" 6 | \n",
"
\n",
" \n",
" | 2012-08-10 | \n",
" 2414 | \n",
" 0.458333 | \n",
" 19.841667 | \n",
" 4 | \n",
"
\n",
" \n",
" | 2012-08-11 | \n",
" 2453 | \n",
" 0.125000 | \n",
" 20.891667 | \n",
" 5 | \n",
"
\n",
" \n",
"
\n",
"
Thanks! Questions?
\n",
"\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"print 'Email:', julia['email']\n",
"print 'Twitter:', julia['twitter']\n",
"print 'Slides: http://bit.ly/pyconca-pandas'"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Email: julia@jvns.ca\n",
"Twitter: http://twitter.com/b0rk\n",
"Slides: http://bit.ly/pyconca-pandas\n"
]
}
],
"prompt_number": 29
}
],
"metadata": {}
}
]
}