{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "*This notebook contains an excerpt from the [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by Jake VanderPlas; the content is available [on GitHub](https://github.com/jakevdp/PythonDataScienceHandbook).*\n", "\n", "*The text is released under the [CC-BY-NC-ND license](https://creativecommons.org/licenses/by-nc-nd/3.0/us/legalcode), and code is released under the [MIT license](https://opensource.org/licenses/MIT). If you find this content useful, please consider supporting the work by [buying the book](http://shop.oreilly.com/product/0636920034919.do)!*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "< [Sorting Arrays](02.08-Sorting.ipynb) | [Contents](Index.ipynb) | [Data Manipulation with Pandas](03.00-Introduction-to-Pandas.ipynb) >\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Structured Data: NumPy's Structured Arrays" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While often our data can be well represented by a homogeneous array of values, sometimes this is not the case. This section demonstrates the use of NumPy's *structured arrays* and *record arrays*, which provide efficient storage for compound, heterogeneous data. While the patterns shown here are useful for simple operations, scenarios like this often lend themselves to the use of Pandas Dataframes, which we'll explore in [Chapter 3](03.00-Introduction-to-Pandas.ipynb)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Imagine that we have several categories of data on a number of people (say, name, age, and weight), and we'd like to store these values for use in a Python program.\n", "It would be possible to store these in three separate arrays:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [], "source": [ "name = ['Alice', 'Bob', 'Cathy', 'Doug']\n", "age = [25, 45, 37, 19]\n", "weight = [55.0, 85.5, 68.0, 61.5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But this is a bit clumsy. There's nothing here that tells us that the three arrays are related; it would be more natural if we could use a single structure to store all of this data.\n", "NumPy can handle this through structured arrays, which are arrays with compound data types.\n", "\n", "Recall that previously we created a simple array using an expression like this:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [ "x = np.zeros(4, dtype=int)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can similarly create a structured array using a compound data type specification:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[('name', ', which means \"little endian\" or \"big endian,\" respectively, and specifies the ordering convention for significant bits.\n", "The next character specifies the type of data: characters, bytes, ints, floating points, and so on (see the table below).\n", "The last character or characters represents the size of the object in bytes.\n", "\n", "| Character | Description | Example |\n", "| --------- | ----------- | ------- | \n", "| 'b' | Byte | np.dtype('b') |\n", "| 'i' | Signed integer | np.dtype('i4') == np.int32 |\n", "| 'u' | Unsigned integer | np.dtype('u1') == np.uint8 |\n", "| 'f' | Floating point | np.dtype('f8') == np.int64 |\n", "| 'c' | Complex floating point| np.dtype('c16') == np.complex128|\n", "| 'S', 'a' | String | np.dtype('S5') |\n", "| 'U' | Unicode string | np.dtype('U') == np.str_ |\n", "| 'V' | Raw data (void) | np.dtype('V') == np.void |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## More Advanced Compound Types\n", "\n", "It is possible to define even more advanced compound types.\n", "For example, you can create a type where each element contains an array or matrix of values.\n", "Here, we'll create a data type with a mat component consisting of a $3\\times 3$ floating-point matrix:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(0, [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]])\n", "[[ 0. 0. 0.]\n", " [ 0. 0. 0.]\n", " [ 0. 0. 0.]]\n" ] } ], "source": [ "tp = np.dtype([('id', 'i8'), ('mat', 'f8', (3, 3))])\n", "X = np.zeros(1, dtype=tp)\n", "print(X[0])\n", "print(X['mat'][0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now each element in the X array consists of an id and a $3\\times 3$ matrix.\n", "Why would you use this rather than a simple multidimensional array, or perhaps a Python dictionary?\n", "The reason is that this NumPy dtype directly maps onto a C structure definition, so the buffer containing the array content can be accessed directly within an appropriately written C program.\n", "If you find yourself writing a Python interface to a legacy C or Fortran library that manipulates structured data, you'll probably find structured arrays quite useful!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## RecordArrays: Structured Arrays with a Twist\n", "\n", "NumPy also provides the np.recarray class, which is almost identical to the structured arrays just described, but with one additional feature: fields can be accessed as attributes rather than as dictionary keys.\n", "Recall that we previously accessed the ages by writing:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([25, 45, 37, 19], dtype=int32)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['age']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we view our data as a record array instead, we can access this with slightly fewer keystrokes:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([25, 45, 37, 19], dtype=int32)" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_rec = data.view(np.recarray)\n", "data_rec.age" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The downside is that for record arrays, there is some extra overhead involved in accessing the fields, even when using the same syntax. We can see this here:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1000000 loops, best of 3: 241 ns per loop\n", "100000 loops, best of 3: 4.61 µs per loop\n", "100000 loops, best of 3: 7.27 µs per loop\n" ] } ], "source": [ "%timeit data['age']\n", "%timeit data_rec['age']\n", "%timeit data_rec.age" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Whether the more convenient notation is worth the additional overhead will depend on your own application." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## On to Pandas\n", "\n", "This section on structured and record arrays is purposely at the end of this chapter, because it leads so well into the next package we will cover: Pandas.\n", "Structured arrays like the ones discussed here are good to know about for certain situations, especially in case you're using NumPy arrays to map onto binary data formats in C, Fortran, or another language.\n", "For day-to-day use of structured data, the Pandas package is a much better choice, and we'll dive into a full discussion of it in the chapter that follows." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "< [Sorting Arrays](02.08-Sorting.ipynb) | [Contents](Index.ipynb) | [Data Manipulation with Pandas](03.00-Introduction-to-Pandas.ipynb) >\n", "\n", "\n" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.1" } }, "nbformat": 4, "nbformat_minor": 0 }