{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Working with time series data" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "Some imports:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "slideshow": { "slide_type": "-" } }, "outputs": [], "source": [ "%matplotlib inline\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "try:\n", " import seaborn\n", "except:\n", " pass\n", "\n", "pd.options.display.max_rows = 8" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Case study: air quality data of European monitoring stations (AirBase)\n", "\n", "AirBase (The European Air quality dataBase): hourly measurements of all air quality monitoring stations from Europe. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from IPython.display import HTML\n", "HTML('')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I downloaded and preprocessed some of the data ([python-airbase](https://github.com/jorisvandenbossche/python-airbase)): `data/airbase_data.csv`. This file includes the hourly concentrations of NO2 for 4 different measurement stations:\n", "\n", "- FR04037 (PARIS 13eme): urban background site at Square de Choisy\n", "- FR04012 (Paris, Place Victor Basch): urban traffic site at Rue d'Alesia\n", "- BETR802: urban traffic site in Antwerp, Belgium\n", "- BETN029: rural background site in Houtem, Belgium\n", "\n", "See http://www.eea.europa.eu/themes/air/interactive/no2" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Importing the data" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Import the csv file:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "!head -5 data/airbase_data.csv" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the missing values are indicated by `-9999`. This can be recognized by `read_csv` by passing the `na_values` keyword:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data = pd.read_csv('data/airbase_data.csv', index_col=0, parse_dates=True, na_values=[-9999])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Exploring the data - recap of some useful methods" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Some useful methods:\n", "\n", "`head` and `tail`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "slideshow": { "slide_type": "-" } }, "outputs": [], "source": [ "data.head(3)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data.tail()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "`info()`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data.info()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "slideshow": { "slide_type": "subslide" } }, "source": [ "Getting some basic summary statistics about the data with `describe`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data.describe()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Quickly visualizing the data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "slideshow": { "slide_type": "-" } }, "outputs": [], "source": [ "data.plot(kind='box', ylim=[0,250])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "data['BETR801'].plot(kind='hist', bins=50)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "data.plot(figsize=(12,6))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This does not say too much .." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We can select part of the data (eg the latest 500 data points):" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data[-500:].plot(figsize=(12,6))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or we can use some more advanced time series features -> next section!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Working with time series data" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "When we ensure the DataFrame has a `DatetimeIndex`, time-series related functionality becomes available:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data.index" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Indexing a time series works with strings:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data[\"2010-01-01 09:00\":\"2010-01-01 12:00\"]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "A nice feature is **\"partial string\" indexing**, where we can do implicit slicing by providing a partial datetime string.\n", "\n", "E.g. all data of 2012:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [], "source": [ "data['2012']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Normally you would expect this to access a column named '2012', but as for a DatetimeIndex, pandas also tries to interprete it as a datetime slice." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "Or all data of January up to March 2012:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data['2012-01':'2012-03']" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Time and date components can be accessed from the index:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data.index.hour" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data.index.year" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
data.resample('D').mean()
was expressed as data.resample('D', how='mean')
.\n",
"