{ "metadata": { "name": "", "signature": "sha256:29cedbf1d5d462a0ee4e706b735e7bf20cbc3685de903c5be20682cd5df72b06" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "*Extracts from an early draft of a series of notebooks developed for the forthcoming Open University course TM351, codename \"The Data Course\", due for release in October 2015.*" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Warning suppression - for python 3 users...\n", "# pandas is throwing quite a lot of warnings at the moment becuase not everything currently works in python 3.4\n", "# Uncomment the following two lines and run this block to suppress warnings\n", "\n", "#import warnings\n", "#warnings.filterwarnings('ignore')" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "pandas Series and Dataframes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Whilst Python is a general purpose scripting language, when working in a particular domain - such as the world of data - it can be useful to make use of code libraries that provide higher level data structures and operations that are suited to that domain.\n", "\n", "A library we will be drawing on heavily in this course is *pandas*. If data is represented in an appropriate format, it can make it much easier to work with. *panads* provides just such a medium. In this notebook, you will learn about two key *pandas* data structures: *lists* and *dataframes*.\n", "\n", "To start with we need to load in the *pandas* library. By convention we associate this with the convenience label *pd*." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import pandas as pd" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Python Recap - Lists and Dicts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python lists are flexible, mutable data structures that can be used to repreent an ordered list of objects." ] }, { "cell_type": "code", "collapsed": false, "input": [ "simple_list=['apples', 'oranges', 'bananas','pears']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Associated with each list is a numerical *index* value, with a count starting at zero, that identifies the position of each list member. The *N*'th list member has index value *N-1*." ] }, { "cell_type": "code", "collapsed": false, "input": [ "print(\"First list item (index value 1-1 = 0): simple_list[0] -> {0}\".format(simple_list[0]))\n", "print(\"Third list item (index value 3-1 = 2): simple_list[2] -> {0}\".format(simple_list[2]))" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python also supports unordered *associative arrays* in the form of *dicts* that allow you to index a value by name:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "simple_dict={1:'one', 2:'two', 4:'four', 3:'three'}\n", "print( \"Item with key (index) '4' has value: {0}\".format(simple_dict[4]))" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can inspect the keys and the values contained within a dict directly:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print( list(simple_dict.keys()) )\n", "print( list(simple_dict.values()) )" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With a simple list or dict, we can use a list or dict comprehension to filter the contents of an object according to a test condition. For example, we can test against the value of the keys in a dict and generate a list containing associated dict values." ] }, { "cell_type": "code", "collapsed": false, "input": [ "[simple_dict[k] for k in simple_dict if k>2]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "#The following statement swaps the keys and values contained in simple_dict\n", "alternative_dict= dict(zip(simple_dict.values(),simple_dict.keys()))\n", "\n", "#How would you generate a list containing the keys of alternative_dict for dict values < 3?\n", "\n", "\n", "##YOUR CODE HERE" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "pandas Series and Dataframes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the following sections, you will be introduced to two powerful data representations supported by the pandas library: *Series* and *DataFrames*. The introduction provides a quick overview of some of the operations that are possible using these dataframes. We will be revisiting many of the operations in more depth later in the course, so for now just try to get a feel for what's possible..." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Series" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A pandas *Series* combines the idea of a list with an additional index column:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "pd.Series( ['one', 'two', 'three', 'four'] )" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can simplify the calling of Series by loading the Series method in from the pandas library explicitly." ] }, { "cell_type": "code", "collapsed": false, "input": [ "from pandas import Series\n", "simple_series = Series( [1, 2, 3, 4] )\n", "simple_series" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can index into a Series using the corresponding index value:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "simple_series[1]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also grab several values at once if we pass the desired index values in as a list in the order we want them to be displayed:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "simple_series[ [1,0,3] ]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In much the same way that we can inspect the keys used in a Python dict, we can inspect the index values used within a Series." ] }, { "cell_type": "code", "collapsed": false, "input": [ "simple_series.index" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also define our own index values:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "myindex_series = Series( [1, 2, 3, 4], index=['one', 'two', 'three', 'four'] )\n", "print( myindex_series.index )\n", "myindex_series" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, it's easy enough to pull out several values from the Series by providing several of our own index values in a list:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "myindex_series[ ['two','four'] ]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Series can also be created from a simple dict, where the unique key values become index values:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "Series( {'Q1':'Spring', 'Q2':'Summer','Q3':'Autumn', 'Q4':'Winter'} )" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "Series( {'Q1':'Spring', 'Q2':'Summer','Q3':'Autumn', 'Q4':'Winter'}, index=['Q4','Q3','Q2','Q1'] )" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "####Exercise\n", "\n", "Construct a series containing the names of the days of the week and a day number index, ordering the days Monday to Sunday with day count 1 to 7." ] }, { "cell_type": "code", "collapsed": false, "input": [ "#YOUR CODE HERE" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----" ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "\"Vector\" Operations with Series" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In certain respects, we can think of a Series as a vector. For example, if we multiply a Series by a scalar value, each member of the series is multiplied by that value:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "5 * Series([1,1,2,3,4])" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "Series([1,1,2,3,4]) - 7.1" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "Series([1,1,2,3,4]) + Series([10,10,15,-15,-20])" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Filtering Series" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A very useful feature of Series is that we can filter their values by value. The values in the Series (*not* the index values) are tested against the condition and the series elements that pass the test are returned, along with their index values:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "myindex_series[ myindex_series>2 ]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This takes a little bit of thinking about... Let's see what the index expression is returning:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "myindex_series>2" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Graphically, we might think of this operation in the following terms:\n", "\n", "![Filtering a series](files/images/series_filter2.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "####Exercise\n", "\n", "Assign your \"days of the week\" Series to a variable *daysOfWeek*. Find a way of creating a new Series containing just the weekdays by filtering *daysOfWeek*." ] }, { "cell_type": "code", "collapsed": false, "input": [ "#YOUR CODE HERE" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Aligning Data in Series" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In many situtations, it can be useful to be able add Series together that share some common index values, even if the Series are presented in a different order. For example, imagine you have one series that represents the total amount of expenditure spent with a range of companies across several projects, and separate series containing the value of payments made to just those companies who worked on a particular project:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "totalSpend=Series({\"Company A\":0,\"Company B\":0, \"Company C\":0, \"Company D\":0})\n", "totalSpend" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "project1=Series([1000, 2000, 500], index=[\"Company A\",\"Company B\", \"Company C\"])\n", "project2=Series([800, 2000 ], index=[\"Company D\", \"Company A\"])\n", "project2" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What happens if we try adding various combinations of *totalSpend*, *project1* and *project2*?" ] }, { "cell_type": "code", "collapsed": false, "input": [ "totalSpend+project1" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "project1+project2" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If any index values aren't matched across the Series, the sum for that index value returns *NaN* - not a number. However, using the Series *add* method we can force missing values to be treated as a particular value, such as 0, using a *fill_value* parameter:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "totalSpend.add(project1,fill_value=0)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As the result is itself a Series, we can also chain the add expression:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "totalSpend.add(project1,fill_value=0).add(project2,fill_value=0)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "DataFrames" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "DataFrames are two dimensional data tables in which rows of data have values spread across one or more columns, much like a sheet in a spreadsheet. Each column behaves as if it is a Series; a DataFrame can thus be thought of as a dict of Series, where dict keys correspond to column names. That said, the spreadsheet idea is probably easier to desrcibe to non-Python programmers!" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from pandas import DataFrame" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ " courseData= { 'courseCode': ['TM351','TU100','M269'],\n", " 'points':[30,60,30],\n", " 'level':['3','1','2']\n", " }" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "DataFrame( courseData )" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also create a DataFrame from a list of tuples." ] }, { "cell_type": "code", "collapsed": false, "input": [ "DataFrame([('a',1,2),('b',2,3)], columns=[\"alpha\",'num1','num2'])" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can pull out any required column as a Series by using the column name as a key value:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df=DataFrame( courseData )\n", "course_df['courseCode']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df.courseCode" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To pull out several columns, provide a list of the column names you want to extract:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df[ ['courseCode','level'] ]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can force the ordering of columns in the DataFrame by means of the *columns* variable. If a column name is specified that does not have a match in the keys of the source data dict, an 'empty' column is created with all the values set to NaN. " ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df=DataFrame( courseData, columns=['level','courseCode','title','points'] )\n", "course_df" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can populate a column with the same value in each cell using a 'scalar' assignment:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df.title='Unknown'\n", "course_df" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also set column values from a Series:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df.title = Series( ['The data course','The foundation course','The algorithms course'] )\n", "course_df" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can pull out the values of a column in a similar way:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df.title" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "EXERCISE:\n", "What problems can you foresee with setting the values within a column from a Series? Does the original assignment of data to the dataframe from the *coursedata* dict suffer from similar problems and if so how? How might you get around such problems?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
*Your answer here...*
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here's what I thought..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*If the Series elements are in a different order to the DataFrame index order, the titles will be incorrectly assigned to the course code indexed rows.\n", "A similar problem exists in the original dict - we rely on the order of items in each independent list to be in synch with the order of all the other lists.\n", "One way round the problem would be to put the values for each row in a tuple and label the columns explictly.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One of the reasons for using a library such as pandas is that it offers data structures that make working with data as easy as possible.\n", "\n", "Sometimes, this requires a little careful thinking about how best to organise the data within a DataFrame.\n", "\n", "For example, for the course information dataframe, it may be most useful to use the course codes as the index values." ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df=course_df.set_index('courseCode')\n", "course_df" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The visual presentation of this is slightly misleading - it looks as if there may be a blank row with index value *courseCode*, but the size of the trasnformed table is correctly shown; one of the columns in the original DataFrame has moved from being a column in its own right to the index column." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can pull out one or more rows by referencing the appropriate index element(s) and the columns we wish to extract:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df.ix[ ['TU100','TM351'], ['title','level'] ]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we identify a particular column, we can use an index value to pull out the value from the correspondingly indexed row and the chosen column:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df.level['TM351']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df[ 'title' ]['TM351']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also filter the rows of a DataFrame based on the vlaues of one or more columns. We will cover this powerful feature in more depth later in the course." ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df[course_df.points==30]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You may recall that passing a column name in as the DataFrame key returns a Series containing the values of the column." ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df['title']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Paraphrasing the author of pandas, Wes McKinney, writing in his book Python for Data Analysis, \"[this apparent inconsistency] in syntax arose out of practicalityand nothing more\". Pragmatic programming, as you might, FTW! [FTW is slang for \"for the win\", a positive exclamation in gamer culture.]" ] }, { "cell_type": "heading", "level": 4, "metadata": {}, "source": [ "Unique Items in a Column" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can pull out the unique values contained with a column very easily by applying the `unique()` function to the appropriate column:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df['points'].unique()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can easily iterate through the items:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "for item in course_df['points'].unique():\n", " print(item)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get a list rather than an array, make use of the `tolist()` helper function:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df['points'].unique().tolist()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 4, "metadata": {}, "source": [ "Column Typing in DataFrames" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to perform certain operations on a column, the column needs to be correctly typed - that is, all elements need to be of a particular type.\n", "\n", "You can check the type of each column in a dataframe using the `.dtypes` attribute:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "course_df.dtypes" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we see that the level and title are classed as object types, but the points values have been identified as integers." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Further Information" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*pandas'* Series and DataFrame objects provide several more useful, and powerful, data manipulation methods than have been described here. You will meet many of them later in the course, as and when they are required.\n", "\n", "If you would like to learn about *pandas* in more depth, a copy of the book written by pandas' original developer, Wes McKinney, is available to OU students as an ebook via the OU Library: [Python for Data Analysis](http://proquestcombo.safaribooksonline.com.libezproxy.open.ac.uk/book/programming/python/9781449323592)" ] } ], "metadata": {} } ] }