{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"![](\"figures/PDSH-cover-small.png\")
\n",
"*This notebook contains an excerpt from the [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by Jake VanderPlas; the content is available [on GitHub](https://github.com/jakevdp/PythonDataScienceHandbook).*\n",
"\n",
"*The text is released under the [CC-BY-NC-ND license](https://creativecommons.org/licenses/by-nc-nd/3.0/us/legalcode), and code is released under the [MIT license](https://opensource.org/licenses/MIT). If you find this content useful, please consider supporting the work by [buying the book](http://shop.oreilly.com/product/0636920034919.do)!*\n",
"\n",
"*No changes were made to the contents of this notebook from the original.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"< [Vectorized String Operations](03.10-Working-With-Strings.ipynb) | [Contents](Index.ipynb) | [High-Performance Pandas: eval() and query()](03.12-Performance-Eval-and-Query.ipynb) >"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Working with Time Series"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Pandas was developed in the context of financial modeling, so as you might expect, it contains a fairly extensive set of tools for working with dates, times, and time-indexed data.\n",
"Date and time data comes in a few flavors, which we will discuss here:\n",
"\n",
"- *Time stamps* reference particular moments in time (e.g., July 4th, 2015 at 7:00am).\n",
"- *Time intervals* and *periods* reference a length of time between a particular beginning and end point; for example, the year 2015. Periods usually reference a special case of time intervals in which each interval is of uniform length and does not overlap (e.g., 24 hour-long periods comprising days).\n",
"- *Time deltas* or *durations* reference an exact length of time (e.g., a duration of 22.56 seconds).\n",
"\n",
"In this section, we will introduce how to work with each of these types of date/time data in Pandas.\n",
"This short section is by no means a complete guide to the time series tools available in Python or Pandas, but instead is intended as a broad overview of how you as a user should approach working with time series.\n",
"We will start with a brief discussion of tools for dealing with dates and times in Python, before moving more specifically to a discussion of the tools provided by Pandas.\n",
"After listing some resources that go into more depth, we will review some short examples of working with time series data in Pandas."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dates and Times in Python\n",
"\n",
"The Python world has a number of available representations of dates, times, deltas, and timespans.\n",
"While the time series tools provided by Pandas tend to be the most useful for data science applications, it is helpful to see their relationship to other packages used in Python."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Native Python dates and times: ``datetime`` and ``dateutil``\n",
"\n",
"Python's basic objects for working with dates and times reside in the built-in ``datetime`` module.\n",
"Along with the third-party ``dateutil`` module, you can use it to quickly perform a host of useful functionalities on dates and times.\n",
"For example, you can manually build a date using the ``datetime`` type:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"datetime.datetime(2015, 7, 4, 0, 0)"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datetime import datetime\n",
"datetime(year=2015, month=7, day=4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or, using the ``dateutil`` module, you can parse dates from a variety of string formats:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"datetime.datetime(2015, 7, 4, 0, 0)"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from dateutil import parser\n",
"date = parser.parse(\"4th of July, 2015\")\n",
"date"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"Once you have a ``datetime`` object, you can do things like printing the day of the week:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"'Saturday'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"date.strftime('%A')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the final line, we've used one of the standard string format codes for printing dates (``\"%A\"``), which you can read about in the [strftime section](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior) of Python's [datetime documentation](https://docs.python.org/3/library/datetime.html).\n",
"Documentation of other useful date utilities can be found in [dateutil's online documentation](http://labix.org/python-dateutil).\n",
"A related package to be aware of is [``pytz``](http://pytz.sourceforge.net/), which contains tools for working with the most migrane-inducing piece of time series data: time zones.\n",
"\n",
"The power of ``datetime`` and ``dateutil`` lie in their flexibility and easy syntax: you can use these objects and their built-in methods to easily perform nearly any operation you might be interested in.\n",
"Where they break down is when you wish to work with large arrays of dates and times:\n",
"just as lists of Python numerical variables are suboptimal compared to NumPy-style typed numerical arrays, lists of Python datetime objects are suboptimal compared to typed arrays of encoded dates."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Typed arrays of times: NumPy's ``datetime64``\n",
"\n",
"The weaknesses of Python's datetime format inspired the NumPy team to add a set of native time series data type to NumPy.\n",
"The ``datetime64`` dtype encodes dates as 64-bit integers, and thus allows arrays of dates to be represented very compactly.\n",
"The ``datetime64`` requires a very specific input format:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(datetime.date(2015, 7, 4), dtype='datetime64[D]')"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"date = np.array('2015-07-04', dtype=np.datetime64)\n",
"date"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once we have this date formatted, however, we can quickly do vectorized operations on it:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(['2015-07-04', '2015-07-05', '2015-07-06', '2015-07-07',\n",
" '2015-07-08', '2015-07-09', '2015-07-10', '2015-07-11',\n",
" '2015-07-12', '2015-07-13', '2015-07-14', '2015-07-15'], dtype='datetime64[D]')"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"date + np.arange(12)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because of the uniform type in NumPy ``datetime64`` arrays, this type of operation can be accomplished much more quickly than if we were working directly with Python's ``datetime`` objects, especially as arrays get large\n",
"(we introduced this type of vectorization in [Computation on NumPy Arrays: Universal Functions](02.03-Computation-on-arrays-ufuncs.ipynb)).\n",
"\n",
"One detail of the ``datetime64`` and ``timedelta64`` objects is that they are built on a *fundamental time unit*.\n",
"Because the ``datetime64`` object is limited to 64-bit precision, the range of encodable times is $2^{64}$ times this fundamental unit.\n",
"In other words, ``datetime64`` imposes a trade-off between *time resolution* and *maximum time span*.\n",
"\n",
"For example, if you want a time resolution of one nanosecond, you only have enough information to encode a range of $2^{64}$ nanoseconds, or just under 600 years.\n",
"NumPy will infer the desired unit from the input; for example, here is a day-based datetime:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"numpy.datetime64('2015-07-04')"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.datetime64('2015-07-04')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a minute-based datetime:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"numpy.datetime64('2015-07-04T12:00')"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.datetime64('2015-07-04 12:00')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice that the time zone is automatically set to the local time on the computer executing the code.\n",
"You can force any desired fundamental unit using one of many format codes; for example, here we'll force a nanosecond-based time:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"numpy.datetime64('2015-07-04T12:59:59.500000000')"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.datetime64('2015-07-04 12:59:59.50', 'ns')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following table, drawn from the [NumPy datetime64 documentation](http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html), lists the available format codes along with the relative and absolute timespans that they can encode:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"|Code | Meaning | Time span (relative) | Time span (absolute) |\n",
"|--------|-------------|----------------------|------------------------|\n",
"| ``Y`` | Year\t | ± 9.2e18 years | [9.2e18 BC, 9.2e18 AD] |\n",
"| ``M`` | Month | ± 7.6e17 years | [7.6e17 BC, 7.6e17 AD] |\n",
"| ``W`` | Week\t | ± 1.7e17 years | [1.7e17 BC, 1.7e17 AD] |\n",
"| ``D`` | Day | ± 2.5e16 years | [2.5e16 BC, 2.5e16 AD] |\n",
"| ``h`` | Hour | ± 1.0e15 years | [1.0e15 BC, 1.0e15 AD] |\n",
"| ``m`` | Minute | ± 1.7e13 years | [1.7e13 BC, 1.7e13 AD] |\n",
"| ``s`` | Second | ± 2.9e12 years | [ 2.9e9 BC, 2.9e9 AD] |\n",
"| ``ms`` | Millisecond | ± 2.9e9 years | [ 2.9e6 BC, 2.9e6 AD] |\n",
"| ``us`` | Microsecond | ± 2.9e6 years | [290301 BC, 294241 AD] |\n",
"| ``ns`` | Nanosecond | ± 292 years | [ 1678 AD, 2262 AD] |\n",
"| ``ps`` | Picosecond | ± 106 days | [ 1969 AD, 1970 AD] |\n",
"| ``fs`` | Femtosecond | ± 2.6 hours | [ 1969 AD, 1970 AD] |\n",
"| ``as`` | Attosecond | ± 9.2 seconds | [ 1969 AD, 1970 AD] |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For the types of data we see in the real world, a useful default is ``datetime64[ns]``, as it can encode a useful range of modern dates with a suitably fine precision.\n",
"\n",
"Finally, we will note that while the ``datetime64`` data type addresses some of the deficiencies of the built-in Python ``datetime`` type, it lacks many of the convenient methods and functions provided by ``datetime`` and especially ``dateutil``.\n",
"More information can be found in [NumPy's datetime64 documentation](http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dates and times in pandas: best of both worlds\n",
"\n",
"Pandas builds upon all the tools just discussed to provide a ``Timestamp`` object, which combines the ease-of-use of ``datetime`` and ``dateutil`` with the efficient storage and vectorized interface of ``numpy.datetime64``.\n",
"From a group of these ``Timestamp`` objects, Pandas can construct a ``DatetimeIndex`` that can be used to index data in a ``Series`` or ``DataFrame``; we'll see many examples of this below.\n",
"\n",
"For example, we can use Pandas tools to repeat the demonstration from above.\n",
"We can parse a flexibly formatted string date, and use format codes to output the day of the week:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Timestamp('2015-07-04 00:00:00')"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"date = pd.to_datetime(\"4th of July, 2015\")\n",
"date"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"'Saturday'"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"date.strftime('%A')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Additionally, we can do NumPy-style vectorized operations directly on this same object:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"DatetimeIndex(['2015-07-04', '2015-07-05', '2015-07-06', '2015-07-07',\n",
" '2015-07-08', '2015-07-09', '2015-07-10', '2015-07-11',\n",
" '2015-07-12', '2015-07-13', '2015-07-14', '2015-07-15'],\n",
" dtype='datetime64[ns]', freq=None)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"date + pd.to_timedelta(np.arange(12), 'D')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the next section, we will take a closer look at manipulating time series data with the tools provided by Pandas."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pandas Time Series: Indexing by Time\n",
"\n",
"Where the Pandas time series tools really become useful is when you begin to *index data by timestamps*.\n",
"For example, we can construct a ``Series`` object that has time indexed data:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2014-07-04 0\n",
"2014-08-04 1\n",
"2015-07-04 2\n",
"2015-08-04 3\n",
"dtype: int64"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"index = pd.DatetimeIndex(['2014-07-04', '2014-08-04',\n",
" '2015-07-04', '2015-08-04'])\n",
"data = pd.Series([0, 1, 2, 3], index=index)\n",
"data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have this data in a ``Series``, we can make use of any of the ``Series`` indexing patterns we discussed in previous sections, passing values that can be coerced into dates:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2014-07-04 0\n",
"2014-08-04 1\n",
"2015-07-04 2\n",
"dtype: int64"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data['2014-07-04':'2015-07-04']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are additional special date-only indexing operations, such as passing a year to obtain a slice of all data from that year:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2015-07-04 2\n",
"2015-08-04 3\n",
"dtype: int64"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data['2015']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Later, we will see additional examples of the convenience of dates-as-indices.\n",
"But first, a closer look at the available time series data structures."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pandas Time Series Data Structures\n",
"\n",
"This section will introduce the fundamental Pandas data structures for working with time series data:\n",
"\n",
"- For *time stamps*, Pandas provides the ``Timestamp`` type. As mentioned before, it is essentially a replacement for Python's native ``datetime``, but is based on the more efficient ``numpy.datetime64`` data type. The associated Index structure is ``DatetimeIndex``.\n",
"- For *time Periods*, Pandas provides the ``Period`` type. This encodes a fixed-frequency interval based on ``numpy.datetime64``. The associated index structure is ``PeriodIndex``.\n",
"- For *time deltas* or *durations*, Pandas provides the ``Timedelta`` type. ``Timedelta`` is a more efficient replacement for Python's native ``datetime.timedelta`` type, and is based on ``numpy.timedelta64``. The associated index structure is ``TimedeltaIndex``."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The most fundamental of these date/time objects are the ``Timestamp`` and ``DatetimeIndex`` objects.\n",
"While these class objects can be invoked directly, it is more common to use the ``pd.to_datetime()`` function, which can parse a wide variety of formats.\n",
"Passing a single date to ``pd.to_datetime()`` yields a ``Timestamp``; passing a series of dates by default yields a ``DatetimeIndex``:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"DatetimeIndex(['2015-07-03', '2015-07-04', '2015-07-06', '2015-07-07',\n",
" '2015-07-08'],\n",
" dtype='datetime64[ns]', freq=None)"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dates = pd.to_datetime([datetime(2015, 7, 3), '4th of July, 2015',\n",
" '2015-Jul-6', '07-07-2015', '20150708'])\n",
"dates"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Any ``DatetimeIndex`` can be converted to a ``PeriodIndex`` with the ``to_period()`` function with the addition of a frequency code; here we'll use ``'D'`` to indicate daily frequency:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"PeriodIndex(['2015-07-03', '2015-07-04', '2015-07-06', '2015-07-07',\n",
" '2015-07-08'],\n",
" dtype='int64', freq='D')"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dates.to_period('D')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A ``TimedeltaIndex`` is created, for example, when a date is subtracted from another:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"TimedeltaIndex(['0 days', '1 days', '3 days', '4 days', '5 days'], dtype='timedelta64[ns]', freq=None)"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dates - dates[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Regular sequences: ``pd.date_range()``\n",
"\n",
"To make the creation of regular date sequences more convenient, Pandas offers a few functions for this purpose: ``pd.date_range()`` for timestamps, ``pd.period_range()`` for periods, and ``pd.timedelta_range()`` for time deltas.\n",
"We've seen that Python's ``range()`` and NumPy's ``np.arange()`` turn a startpoint, endpoint, and optional stepsize into a sequence.\n",
"Similarly, ``pd.date_range()`` accepts a start date, an end date, and an optional frequency code to create a regular sequence of dates.\n",
"By default, the frequency is one day:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"DatetimeIndex(['2015-07-03', '2015-07-04', '2015-07-05', '2015-07-06',\n",
" '2015-07-07', '2015-07-08', '2015-07-09', '2015-07-10'],\n",
" dtype='datetime64[ns]', freq='D')"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.date_range('2015-07-03', '2015-07-10')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, the date range can be specified not with a start and endpoint, but with a startpoint and a number of periods:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"DatetimeIndex(['2015-07-03', '2015-07-04', '2015-07-05', '2015-07-06',\n",
" '2015-07-07', '2015-07-08', '2015-07-09', '2015-07-10'],\n",
" dtype='datetime64[ns]', freq='D')"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.date_range('2015-07-03', periods=8)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The spacing can be modified by altering the ``freq`` argument, which defaults to ``D``.\n",
"For example, here we will construct a range of hourly timestamps:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"DatetimeIndex(['2015-07-03 00:00:00', '2015-07-03 01:00:00',\n",
" '2015-07-03 02:00:00', '2015-07-03 03:00:00',\n",
" '2015-07-03 04:00:00', '2015-07-03 05:00:00',\n",
" '2015-07-03 06:00:00', '2015-07-03 07:00:00'],\n",
" dtype='datetime64[ns]', freq='H')"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.date_range('2015-07-03', periods=8, freq='H')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To create regular sequences of ``Period`` or ``Timedelta`` values, the very similar ``pd.period_range()`` and ``pd.timedelta_range()`` functions are useful.\n",
"Here are some monthly periods:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"PeriodIndex(['2015-07', '2015-08', '2015-09', '2015-10', '2015-11', '2015-12',\n",
" '2016-01', '2016-02'],\n",
" dtype='int64', freq='M')"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.period_range('2015-07', periods=8, freq='M')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And a sequence of durations increasing by an hour:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"TimedeltaIndex(['00:00:00', '01:00:00', '02:00:00', '03:00:00', '04:00:00',\n",
" '05:00:00', '06:00:00', '07:00:00', '08:00:00', '09:00:00'],\n",
" dtype='timedelta64[ns]', freq='H')"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.timedelta_range(0, periods=10, freq='H')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All of these require an understanding of Pandas frequency codes, which we'll summarize in the next section."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Frequencies and Offsets\n",
"\n",
"Fundamental to these Pandas time series tools is the concept of a frequency or date offset.\n",
"Just as we saw the ``D`` (day) and ``H`` (hour) codes above, we can use such codes to specify any desired frequency spacing.\n",
"The following table summarizes the main codes available:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Code | Description | Code | Description |\n",
"|--------|---------------------|--------|----------------------|\n",
"| ``D`` | Calendar day | ``B`` | Business day |\n",
"| ``W`` | Weekly | | |\n",
"| ``M`` | Month end | ``BM`` | Business month end |\n",
"| ``Q`` | Quarter end | ``BQ`` | Business quarter end |\n",
"| ``A`` | Year end | ``BA`` | Business year end |\n",
"| ``H`` | Hours | ``BH`` | Business hours |\n",
"| ``T`` | Minutes | | |\n",
"| ``S`` | Seconds | | |\n",
"| ``L`` | Milliseonds | | |\n",
"| ``U`` | Microseconds | | |\n",
"| ``N`` | nanoseconds | | |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The monthly, quarterly, and annual frequencies are all marked at the end of the specified period.\n",
"By adding an ``S`` suffix to any of these, they instead will be marked at the beginning:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Code | Description || Code | Description |\n",
"|---------|------------------------||---------|------------------------|\n",
"| ``MS`` | Month start ||``BMS`` | Business month start |\n",
"| ``QS`` | Quarter start ||``BQS`` | Business quarter start |\n",
"| ``AS`` | Year start ||``BAS`` | Business year start |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Additionally, you can change the month used to mark any quarterly or annual code by adding a three-letter month code as a suffix:\n",
"\n",
"- ``Q-JAN``, ``BQ-FEB``, ``QS-MAR``, ``BQS-APR``, etc.\n",
"- ``A-JAN``, ``BA-FEB``, ``AS-MAR``, ``BAS-APR``, etc.\n",
"\n",
"In the same way, the split-point of the weekly frequency can be modified by adding a three-letter weekday code:\n",
"\n",
"- ``W-SUN``, ``W-MON``, ``W-TUE``, ``W-WED``, etc.\n",
"\n",
"On top of this, codes can be combined with numbers to specify other frequencies.\n",
"For example, for a frequency of 2 hours 30 minutes, we can combine the hour (``H``) and minute (``T``) codes as follows:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"TimedeltaIndex(['00:00:00', '02:30:00', '05:00:00', '07:30:00', '10:00:00',\n",
" '12:30:00', '15:00:00', '17:30:00', '20:00:00'],\n",
" dtype='timedelta64[ns]', freq='150T')"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.timedelta_range(0, periods=9, freq=\"2H30T\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All of these short codes refer to specific instances of Pandas time series offsets, which can be found in the ``pd.tseries.offsets`` module.\n",
"For example, we can create a business day offset directly as follows:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"DatetimeIndex(['2015-07-01', '2015-07-02', '2015-07-03', '2015-07-06',\n",
" '2015-07-07'],\n",
" dtype='datetime64[ns]', freq='B')"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from pandas.tseries.offsets import BDay\n",
"pd.date_range('2015-07-01', periods=5, freq=BDay())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For more discussion of the use of frequencies and offsets, see the [\"DateOffset\" section](http://pandas.pydata.org/pandas-docs/stable/timeseries.html#dateoffset-objects) of the Pandas documentation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Resampling, Shifting, and Windowing\n",
"\n",
"The ability to use dates and times as indices to intuitively organize and access data is an important piece of the Pandas time series tools.\n",
"The benefits of indexed data in general (automatic alignment during operations, intuitive data slicing and access, etc.) still apply, and Pandas provides several additional time series-specific operations.\n",
"\n",
"We will take a look at a few of those here, using some stock price data as an example.\n",
"Because Pandas was developed largely in a finance context, it includes some very specific tools for financial data.\n",
"For example, the accompanying ``pandas-datareader`` package (installable via ``conda install pandas-datareader``), knows how to import financial data from a number of available sources, including Yahoo finance, Google Finance, and others.\n",
"Here we will load Google's closing price history:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Open | \n",
" High | \n",
" Low | \n",
" Close | \n",
" Volume | \n",
"
\n",
" \n",
" | Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2004-08-19 | \n",
" 49.96 | \n",
" 51.98 | \n",
" 47.93 | \n",
" 50.12 | \n",
" NaN | \n",
"
\n",
" \n",
" | 2004-08-20 | \n",
" 50.69 | \n",
" 54.49 | \n",
" 50.20 | \n",
" 54.10 | \n",
" NaN | \n",
"
\n",
" \n",
" | 2004-08-23 | \n",
" 55.32 | \n",
" 56.68 | \n",
" 54.47 | \n",
" 54.65 | \n",
" NaN | \n",
"
\n",
" \n",
" | 2004-08-24 | \n",
" 55.56 | \n",
" 55.74 | \n",
" 51.73 | \n",
" 52.38 | \n",
" NaN | \n",
"
\n",
" \n",
" | 2004-08-25 | \n",
" 52.43 | \n",
" 53.95 | \n",
" 51.89 | \n",
" 52.95 | \n",
" NaN | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Fremont Bridge West Sidewalk | \n",
" Fremont Bridge East Sidewalk | \n",
"
\n",
" \n",
" | Date | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2012-10-03 00:00:00 | \n",
" 4.0 | \n",
" 9.0 | \n",
"
\n",
" \n",
" | 2012-10-03 01:00:00 | \n",
" 4.0 | \n",
" 6.0 | \n",
"
\n",
" \n",
" | 2012-10-03 02:00:00 | \n",
" 1.0 | \n",
" 1.0 | \n",
"
\n",
" \n",
" | 2012-10-03 03:00:00 | \n",
" 2.0 | \n",
" 3.0 | \n",
"
\n",
" \n",
" | 2012-10-03 04:00:00 | \n",
" 6.0 | \n",
" 1.0 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" West | \n",
" East | \n",
" Total | \n",
"
\n",
" \n",
" \n",
" \n",
" | count | \n",
" 35752.000000 | \n",
" 35752.000000 | \n",
" 35752.000000 | \n",
"
\n",
" \n",
" | mean | \n",
" 61.470267 | \n",
" 54.410774 | \n",
" 115.881042 | \n",
"
\n",
" \n",
" | std | \n",
" 82.588484 | \n",
" 77.659796 | \n",
" 145.392385 | \n",
"
\n",
" \n",
" | min | \n",
" 0.000000 | \n",
" 0.000000 | \n",
" 0.000000 | \n",
"
\n",
" \n",
" | 25% | \n",
" 8.000000 | \n",
" 7.000000 | \n",
" 16.000000 | \n",
"
\n",
" \n",
" | 50% | \n",
" 33.000000 | \n",
" 28.000000 | \n",
" 65.000000 | \n",
"
\n",
" \n",
" | 75% | \n",
" 79.000000 | \n",
" 67.000000 | \n",
" 151.000000 | \n",
"
\n",
" \n",
" | max | \n",
" 825.000000 | \n",
" 717.000000 | \n",
" 1186.000000 | \n",
"
\n",
" \n",
"
\n",
"