{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction to data frames\n", "\n", "[Data set download](https://s3.amazonaws.com/bebi103.caltech.edu/data/gfmt_sleep.csv)\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Colab setup ------------------\n", "import os, sys, subprocess\n", "if \"google.colab\" in sys.modules:\n", " cmd = \"pip install --upgrade watermark\"\n", " process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n", " stdout, stderr = process.communicate()\n", " data_path = \"https://s3.amazonaws.com/bebi103.caltech.edu/data/\"\n", "else:\n", " data_path = \"../data/\"\n", "# ------------------------------\n", "\n", "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "Throughout your research career, you will undoubtedly need to handle data, possibly lots of data. Once in a usable form, you are empowered to rapidly make graphics and perform statistical inference. **Tidy data** is an important format and we will discuss that in subsequent sections of this lesson. In an ideal world, data sets would be stored in tidy format and be ready to use. The data comes in lots of formats, and you will spend much of your time **wrangling** the data to get it into a usable format. Wrangling is the topic of the next lesson; for now all data sets will be in tidy format from the get-go." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The data set\n", "\n", "We will explore using Pandas with a real data set. We will use a data set published in [Beattie, et al., Perceptual impairment in face identification with poor sleep, *Royal Society Open Science*, **3**, 160321, 2016](https://doi.org/10.1098/rsos.160321). In this paper, researchers used the [Glasgow Facial Matching Test](https://doi.org/10.3758/BRM.42.1.286) (GMFT) to investigate how sleep deprivation affects a subject's ability to match faces, as well as the confidence the subject has in those matches. Briefly, the test works by having subjects look at a pair of faces. Two such pairs are shown below.\n", "\n", "
\n", "\n", "![GFMT faces](gfmt_faces.png)\n", "\n", "
\n", "\n", "\n", "The top two pictures are the same person, the bottom two pictures are different people. For each pair of faces, the subject gets as much time as he or she needs and then says whether or not they are the same person. The subject then rates his or her confidence in the choice.\n", "\n", "In this study, subjects also took surveys to determine properties about their sleep. The Sleep Condition Indicator (SCI) is a measure of insomnia disorder over the past month (scores of 16 and below indicate insomnia). The Pittsburgh Sleep Quality Index (PSQI) quantifies how well a subject sleeps in terms of interruptions, latency, etc. A higher score indicates poorer sleep. The Epworth Sleepiness Scale (ESS) assesses daytime drowsiness.\n", "\n", "The data set can be downloaded [here](https://s3.amazonaws.com/bebi103.caltech.edu/data/gfmt_sleep.csv). The contents of this file were adapted from [the Excel file posted on the public Dryad repository](https://doi.org/10.5061/dryad.r620r). (*Note this: if you want other people to use and explore your data, make it publicly available.*) I'll say it more boldly.\n", "\n", "
\n", " \n", "If at all possible, share your data freely.\n", " \n", "
\n", "\n", "The data file is a **CSV file**, where CSV stands for comma-separated value. This is a text file that is easily read into data structures in many programming languages. You should generally always store your data in such a format, not necessarily CSV, but a format that is open, has a well-defined specification, and is readable in many contexts. Excel files do not meet these criteria. Neither do `.mat` files. There are other good ways to store data, such as [JSON](http://json.org), but we will almost exclusively use CSV files in this class.\n", "\n", "Let's take a look at the CSV file. We will use the command line program `head` to look at the first 20 lines of the file." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "participant number,gender,age,correct hit percentage,correct reject percentage,percent correct,confidence when correct hit,confidence incorrect hit,confidence correct reject,confidence incorrect reject,confidence when correct,confidence when incorrect,sci,psqi,ess\n", "8,f,39,65,80,72.5,91,90,93,83.5,93,90,9,13,2\n", "16,m,42,90,90,90,75.5,55.5,70.5,50,75,50,4,11,7\n", "18,f,31,90,95,92.5,89.5,90,86,81,89,88,10,9,3\n", "22,f,35,100,75,87.5,89.5,*,71,80,88,80,13,8,20\n", "27,f,74,60,65,62.5,68.5,49,61,49,65,49,13,9,12\n", "28,f,61,80,20,50,71,63,31,72.5,64.5,70.5,15,14,2\n", "30,m,32,90,75,82.5,67,56.5,66,65,66,64,16,9,3\n", "33,m,62,45,90,67.5,54,37,65,81.5,62,61,14,9,9\n", "34,f,33,80,100,90,70.5,76.5,64.5,*,68,76.5,14,12,10\n" ] } ], "source": [ "fname = os.path.join(data_path, \"gfmt_sleep.csv\")\n", "\n", "# This will not work in Colab because the file is not local\n", "!head {fname}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first line contains the **headers** for each column. They are participant number, gender, age, etc. The data follow. There are two important things to note here. First, notice that the `gender` column has string data (`m` or `f`), while the rest of the data are numeric. Note also that there are some **missing data**, denoted by the `*`s in the file.\n", "\n", "Given the file I/O skills you recently learned, you could write some functions to parse this file and extract the data you want. You can imagine that this might be kind of painful. However, if the file format is nice and clean, like we more or less have here, we can use pre-built tools to read in the data from the file and put it in a convenient data structure. This is one of the many uses of **Pandas**, a _**very**_ powerful tool for handling data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pandas\n", "\n", "[Pandas](https://pandas.pydata.org) is the primary tool in the Python ecosystem for handling data. Its primary object, the `DataFrame` is extremely useful in handling data. We will use Pandas to read in CSV (comma separated value) files and store the results in the very handy Pandas `DataFrame`. This incredibly flexible and powerful data structure will be a centerpiece in the rest of this course and beyond. In this lesson, we will learn about what a data frame is and how to use it.\n", "\n", "Traditionally, Pandas is imported as `pd`, which we did in the first cell of this notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using pandas.read_csv() to read in data\n", "\n", "We will use `pd.read_csv()` to load the data set. The data are stored in a **data frame** (data type `DataFrame`), which is one of the data types that makes Pandas so convenient for use in data analysis. Data frames offer mixed data types, including incomplete columns, and convenient slicing, among many, many other convenient features. We will use the data frame to look at the data, at the same time demonstrating some of the power of data frames. They are like spreadsheets, only a lot better." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "df = pd.read_csv(fname, na_values=\"*\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that we used the kwarg `na_values=*` to specify that entries marked with a `*` are missing. The resulting data frame is populated with a **NaN**, or not-a-number, wherever this character is present in the file. In this case, we want `na_values='*'`. So, let's load in the data set.\n", "\n", "If you check out the doc string for `pd.read_csv()`, you will see there are *lots* of options for reading in the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exploring the DataFrame\n", "\n", "Let's jump right in and look at the contents of the `DataFrame`. We can look at the first several rows using the `df.head()` method." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numbergenderagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectscipsqiess
08f39658072.591.090.093.083.593.090.09132
116m42909090.075.555.570.550.075.050.04117
218f31909592.589.590.086.081.089.088.01093
322f351007587.589.5NaN71.080.088.080.013820
427f74606562.568.549.061.049.065.049.013912
\n", "
" ], "text/plain": [ " participant number gender age correct hit percentage \\\n", "0 8 f 39 65 \n", "1 16 m 42 90 \n", "2 18 f 31 90 \n", "3 22 f 35 100 \n", "4 27 f 74 60 \n", "\n", " correct reject percentage percent correct confidence when correct hit \\\n", "0 80 72.5 91.0 \n", "1 90 90.0 75.5 \n", "2 95 92.5 89.5 \n", "3 75 87.5 89.5 \n", "4 65 62.5 68.5 \n", "\n", " confidence incorrect hit confidence correct reject \\\n", "0 90.0 93.0 \n", "1 55.5 70.5 \n", "2 90.0 86.0 \n", "3 NaN 71.0 \n", "4 49.0 61.0 \n", "\n", " confidence incorrect reject confidence when correct \\\n", "0 83.5 93.0 \n", "1 50.0 75.0 \n", "2 81.0 89.0 \n", "3 80.0 88.0 \n", "4 49.0 65.0 \n", "\n", " confidence when incorrect sci psqi ess \n", "0 90.0 9 13 2 \n", "1 50.0 4 11 7 \n", "2 88.0 10 9 3 \n", "3 80.0 13 8 20 \n", "4 49.0 13 9 12 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Look at the contents (first 5 rows)\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that the column headings were automatically assigned. Pandas also automatically defined the indices (names of the rows) as integers going up from zero. We could have defined the indices to be any of the columns of data, but since we did not specify anything for the `index_col` kwarg of `pd.read_csv()`, the default integer indexing is used.\n", "\n", "Also note (in row 3) that the missing data are denoted as NaN." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Indexing data frames\n", "\n", "The data frame is a convenient data structure for many reasons that will become clear as we start exploring. Let's start by looking at how data frames are indexed. We could try to look at the first row like this:\n", "\n", "```python\n", "df[0]\n", "```\n", "\n", "but that would be a mistake and we would get an error message (try it if you like). The problem is that we tried to index numerically by row. **We index `DataFrame`s, by columns.** And there is no column that has the name `0` in this `DataFrame`, though there could be. Instead, a might want to look at the column with the percentage of correct face matching tasks. (To restrict the length of the output, I will use the `.head()` method to just show the first five rows. I'll do this for much of the display of data frames and slices thereof in these lessons.)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 72.5\n", "1 90.0\n", "2 92.5\n", "3 87.5\n", "4 62.5\n", "Name: percent correct, dtype: float64" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"percent correct\"].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This gave us the numbers we were after. Notice that when it was printed, the index of the rows came along with it. If we wanted to pull out a single percentage correct, say corresponding to index `4`, we can do that." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "62.5" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"percent correct\"][4]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, this is **not** the preferred way to do this. It is better to use **`.loc`**. This give the location in the `DataFrame` we want." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "62.5" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[4, \"percent correct\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is also important to note that **row indices need not be integers**. And you should not count on them being integers. In practice you will almost never use row indices, but rather use **Boolean indexing**. (Though see the caveat about speed, below.)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Boolean indexing of data frames\n", "\n", "Let's say I wanted the percent correct of participant number 42. I can use Boolean indexing to specify the row. Specifically, I want the row for which `df['participant number'] == 42`. You can essentially plop this syntax directly when using `.loc`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "54 85.0\n", "Name: percent correct, dtype: float64" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[df[\"participant number\"] == 42, \"percent correct\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If I want to pull the whole record for that participant, I can use `:` for the column index." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numbergenderagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectscipsqiess
5442m291007085.075.0NaN64.543.074.043.03216
\n", "
" ], "text/plain": [ " participant number gender age correct hit percentage \\\n", "54 42 m 29 100 \n", "\n", " correct reject percentage percent correct confidence when correct hit \\\n", "54 70 85.0 75.0 \n", "\n", " confidence incorrect hit confidence correct reject \\\n", "54 NaN 64.5 \n", "\n", " confidence incorrect reject confidence when correct \\\n", "54 43.0 74.0 \n", "\n", " confidence when incorrect sci psqi ess \n", "54 43.0 32 1 6 " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[df[\"participant number\"] == 42, :]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that the index, `54`, comes along for the ride, but we do not need it.\n", "\n", "Now, let's pull out all records of females under the age of 21. We can again use Boolean indexing, but we need to use an `&` operator, taken to mean logical AND. We did not cover this bitwise operator before, but the syntax is self-explanatory in the example below. Note that it is important that each Boolean operation you are doing is in parentheses because of the precedence of the operators involved. The other bitwise operators you may wish to use for Boolean indexing in data frames are `|` for OR and `~` for NOT." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numbergenderagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectscipsqiess
273f16708075.070.057.054.053.057.054.52313
295f189010095.076.583.080.0NaN80.083.02175
6658f16858585.055.030.050.040.052.535.029211
7972f18807577.567.551.566.057.067.053.02946
8885f18858585.093.092.091.089.091.591.025421
\n", "
" ], "text/plain": [ " participant number gender age correct hit percentage \\\n", "27 3 f 16 70 \n", "29 5 f 18 90 \n", "66 58 f 16 85 \n", "79 72 f 18 80 \n", "88 85 f 18 85 \n", "\n", " correct reject percentage percent correct confidence when correct hit \\\n", "27 80 75.0 70.0 \n", "29 100 95.0 76.5 \n", "66 85 85.0 55.0 \n", "79 75 77.5 67.5 \n", "88 85 85.0 93.0 \n", "\n", " confidence incorrect hit confidence correct reject \\\n", "27 57.0 54.0 \n", "29 83.0 80.0 \n", "66 30.0 50.0 \n", "79 51.5 66.0 \n", "88 92.0 91.0 \n", "\n", " confidence incorrect reject confidence when correct \\\n", "27 53.0 57.0 \n", "29 NaN 80.0 \n", "66 40.0 52.5 \n", "79 57.0 67.0 \n", "88 89.0 91.5 \n", "\n", " confidence when incorrect sci psqi ess \n", "27 54.5 23 1 3 \n", "29 83.0 21 7 5 \n", "66 35.0 29 2 11 \n", "79 53.0 29 4 6 \n", "88 91.0 25 4 21 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[(df[\"age\"] < 21) & (df[\"gender\"] == \"f\"), :]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can do something even more complicated, like pull out all females under 30 who got more than 85% of the face matching tasks correct. The code is clearer if we set up our Boolean indexing first, as follows." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 False\n", "1 False\n", "2 False\n", "3 False\n", "4 False\n", "dtype: bool" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "inds = (df[\"age\"] < 30) & (df[\"gender\"] == \"f\") & (df[\"percent correct\"] > 85)\n", "\n", "# Take a look\n", "inds.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that `inds` is an array (actually a Pandas `Series`, essentially a `DataFrame` with one column) of `True`s and `False`s. When we index with it using `.loc`, we get back rows where `inds` is `True`." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numbergenderagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectscipsqiess
2293f281007587.589.5NaN67.060.080.060.01674
295f189010095.076.583.080.0NaN80.083.02175
306f28958087.5100.085.094.061.099.065.019712
3310f25100100100.090.0NaN85.0NaN90.0NaN171011
5644f21859087.566.029.070.029.067.029.026718
5848f23908587.567.047.069.040.067.040.01868
6051f24859590.097.041.074.073.083.055.52917
7567f25100100100.061.5NaN58.5NaN60.5NaN2889
\n", "
" ], "text/plain": [ " participant number gender age correct hit percentage \\\n", "22 93 f 28 100 \n", "29 5 f 18 90 \n", "30 6 f 28 95 \n", "33 10 f 25 100 \n", "56 44 f 21 85 \n", "58 48 f 23 90 \n", "60 51 f 24 85 \n", "75 67 f 25 100 \n", "\n", " correct reject percentage percent correct confidence when correct hit \\\n", "22 75 87.5 89.5 \n", "29 100 95.0 76.5 \n", "30 80 87.5 100.0 \n", "33 100 100.0 90.0 \n", "56 90 87.5 66.0 \n", "58 85 87.5 67.0 \n", "60 95 90.0 97.0 \n", "75 100 100.0 61.5 \n", "\n", " confidence incorrect hit confidence correct reject \\\n", "22 NaN 67.0 \n", "29 83.0 80.0 \n", "30 85.0 94.0 \n", "33 NaN 85.0 \n", "56 29.0 70.0 \n", "58 47.0 69.0 \n", "60 41.0 74.0 \n", "75 NaN 58.5 \n", "\n", " confidence incorrect reject confidence when correct \\\n", "22 60.0 80.0 \n", "29 NaN 80.0 \n", "30 61.0 99.0 \n", "33 NaN 90.0 \n", "56 29.0 67.0 \n", "58 40.0 67.0 \n", "60 73.0 83.0 \n", "75 NaN 60.5 \n", "\n", " confidence when incorrect sci psqi ess \n", "22 60.0 16 7 4 \n", "29 83.0 21 7 5 \n", "30 65.0 19 7 12 \n", "33 NaN 17 10 11 \n", "56 29.0 26 7 18 \n", "58 40.0 18 6 8 \n", "60 55.5 29 1 7 \n", "75 NaN 28 8 9 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.loc[inds, :]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of interest in this exercise in Boolean indexing is that we never had to write a loop. To produce our indices, we could have done the following." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numbergenderagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectscipsqiess
2293f281007587.589.5NaN67.060.080.060.01674
295f189010095.076.583.080.0NaN80.083.02175
306f28958087.5100.085.094.061.099.065.019712
3310f25100100100.090.0NaN85.0NaN90.0NaN171011
5644f21859087.566.029.070.029.067.029.026718
5848f23908587.567.047.069.040.067.040.01868
6051f24859590.097.041.074.073.083.055.52917
7567f25100100100.061.5NaN58.5NaN60.5NaN2889
\n", "
" ], "text/plain": [ " participant number gender age correct hit percentage \\\n", "22 93 f 28 100 \n", "29 5 f 18 90 \n", "30 6 f 28 95 \n", "33 10 f 25 100 \n", "56 44 f 21 85 \n", "58 48 f 23 90 \n", "60 51 f 24 85 \n", "75 67 f 25 100 \n", "\n", " correct reject percentage percent correct confidence when correct hit \\\n", "22 75 87.5 89.5 \n", "29 100 95.0 76.5 \n", "30 80 87.5 100.0 \n", "33 100 100.0 90.0 \n", "56 90 87.5 66.0 \n", "58 85 87.5 67.0 \n", "60 95 90.0 97.0 \n", "75 100 100.0 61.5 \n", "\n", " confidence incorrect hit confidence correct reject \\\n", "22 NaN 67.0 \n", "29 83.0 80.0 \n", "30 85.0 94.0 \n", "33 NaN 85.0 \n", "56 29.0 70.0 \n", "58 47.0 69.0 \n", "60 41.0 74.0 \n", "75 NaN 58.5 \n", "\n", " confidence incorrect reject confidence when correct \\\n", "22 60.0 80.0 \n", "29 NaN 80.0 \n", "30 61.0 99.0 \n", "33 NaN 90.0 \n", "56 29.0 67.0 \n", "58 40.0 67.0 \n", "60 73.0 83.0 \n", "75 NaN 60.5 \n", "\n", " confidence when incorrect sci psqi ess \n", "22 60.0 16 7 4 \n", "29 83.0 21 7 5 \n", "30 65.0 19 7 12 \n", "33 NaN 17 10 11 \n", "56 29.0 26 7 18 \n", "58 40.0 18 6 8 \n", "60 55.5 29 1 7 \n", "75 NaN 28 8 9 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Initialize array of Boolean indices\n", "inds = [False] * len(df)\n", "\n", "# Iterate over the rows of the DataFrame to check if the row should be included\n", "for i, r in df.iterrows():\n", " if r[\"age\"] < 30 and r[\"gender\"] == \"f\" and r[\"percent correct\"] > 85:\n", " inds[i] = True\n", "\n", "# Make our seleciton with Boolean indexing\n", "df.loc[inds, :]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This feature, where the looping is done automatically on Pandas objects like `DataFrame`s, is very powerful and saves us writing lots of lines of code. This example also showed how to use the `iterrows()` method of a `DataFrame` to iterate over the rows of a `DataFrame`. It is actually rare that you will need to do that, as we'll show next when computing with `DataFrames`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Calculating with data frames\n", "\n", "Recall that a subject is said to suffer from insomnia if he or she has an SCI of 16 or below. We might like to add a column to the data frame that specifies whether or not the subject suffers from insomnia. We can conveniently compute with columns. This is done elementwise." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 True\n", "1 True\n", "2 True\n", "3 True\n", "4 True\n", "Name: sci, dtype: bool" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(df[\"sci\"] <= 16).head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This tells use who is an insomniac. We can simply add this to the `DataFrame`." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numbergenderagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectscipsqiessinsomnia
08f39658072.591.090.093.083.593.090.09132True
116m42909090.075.555.570.550.075.050.04117True
218f31909592.589.590.086.081.089.088.01093True
322f351007587.589.5NaN71.080.088.080.013820True
427f74606562.568.549.061.049.065.049.013912True
\n", "
" ], "text/plain": [ " participant number gender age correct hit percentage \\\n", "0 8 f 39 65 \n", "1 16 m 42 90 \n", "2 18 f 31 90 \n", "3 22 f 35 100 \n", "4 27 f 74 60 \n", "\n", " correct reject percentage percent correct confidence when correct hit \\\n", "0 80 72.5 91.0 \n", "1 90 90.0 75.5 \n", "2 95 92.5 89.5 \n", "3 75 87.5 89.5 \n", "4 65 62.5 68.5 \n", "\n", " confidence incorrect hit confidence correct reject \\\n", "0 90.0 93.0 \n", "1 55.5 70.5 \n", "2 90.0 86.0 \n", "3 NaN 71.0 \n", "4 49.0 61.0 \n", "\n", " confidence incorrect reject confidence when correct \\\n", "0 83.5 93.0 \n", "1 50.0 75.0 \n", "2 81.0 89.0 \n", "3 80.0 88.0 \n", "4 49.0 65.0 \n", "\n", " confidence when incorrect sci psqi ess insomnia \n", "0 90.0 9 13 2 True \n", "1 50.0 4 11 7 True \n", "2 88.0 10 9 3 True \n", "3 80.0 13 8 20 True \n", "4 49.0 13 9 12 True " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Add the column to the DataFrame\n", "df[\"insomnia\"] = df[\"sci\"] <= 16\n", "\n", "# Take a look\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A note about vectorization\n", "\n", "Notice how applying the `<=` operator to a `Series` resulted in **elementwise** application. This is called `vectorization`. It means that we do not have to write a `for` loop to do operations on the elements of a `Series` or other array-like object. Imagine if we had to do that with a `for` loop." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "insomnia = []\n", "for sci in df[\"sci\"]:\n", " insomnia.append(sci <= 16)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is cumbersome. The vectorization allows for much more convenient calculation. Beyond that, the vectorized code is almost always faster when using Pandas and Numpy because the looping is done with compiled code under the hood. This can be done with many operators, including those you've already seen, like `+`, `-`, `*`, `/`, `**`, etc., just as with Numpy arrays." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Applying functions to Pandas objects\n", "\n", "Remember when we saw the `np.mean()` function? We can compute with that as well. Let's compare the mean percent correct for insomniacs versus those who are not." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Insomniacs: 76.1\n", "Control: 81.46103896103897\n" ] } ], "source": [ "print(\"Insomniacs:\", np.mean(df.loc[df[\"insomnia\"], \"percent correct\"]))\n", "print(\"Control: \", np.mean(df.loc[~df[\"insomnia\"], \"percent correct\"]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that I used the `~` operator, which is a bit switcher. It changes all `True`s to `False`s and vice versa. In this case, it functions like NOT.\n", "\n", "We will do a lot more computing with Pandas data frames as the course goes on. For our last demonstration in this lesson, we can quickly compute summary statistics about each column of a data frame using its `describe()` method." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numberagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectscipsqiess
count102.000000102.000000102.000000102.000000102.000000102.00000084.000000102.00000093.000000102.00000099.000000102.000000102.000000102.000000
mean52.04902037.92156983.08823577.20588280.14705974.99019658.56547671.13725561.22043074.64215761.97979822.2450985.2745107.294118
std30.02090914.02945015.09121017.56985412.04788114.16591619.56065314.98747917.67128313.61972515.9216707.5471283.4040074.426715
min1.00000016.00000035.00000020.00000040.00000029.5000007.00000019.00000017.00000024.00000024.5000000.0000000.0000000.000000
25%26.25000026.50000075.00000070.00000072.50000066.00000046.37500064.62500050.00000066.00000051.00000017.0000003.0000004.000000
50%52.50000036.50000090.00000080.00000083.75000075.00000056.25000071.25000061.00000075.75000061.50000023.5000005.0000007.000000
75%77.75000045.00000095.00000090.00000087.50000086.50000073.50000080.00000074.00000082.37500073.00000029.0000007.00000010.000000
max103.00000074.000000100.000000100.000000100.000000100.00000092.000000100.000000100.000000100.000000100.00000032.00000015.00000021.000000
\n", "
" ], "text/plain": [ " participant number age correct hit percentage \\\n", "count 102.000000 102.000000 102.000000 \n", "mean 52.049020 37.921569 83.088235 \n", "std 30.020909 14.029450 15.091210 \n", "min 1.000000 16.000000 35.000000 \n", "25% 26.250000 26.500000 75.000000 \n", "50% 52.500000 36.500000 90.000000 \n", "75% 77.750000 45.000000 95.000000 \n", "max 103.000000 74.000000 100.000000 \n", "\n", " correct reject percentage percent correct \\\n", "count 102.000000 102.000000 \n", "mean 77.205882 80.147059 \n", "std 17.569854 12.047881 \n", "min 20.000000 40.000000 \n", "25% 70.000000 72.500000 \n", "50% 80.000000 83.750000 \n", "75% 90.000000 87.500000 \n", "max 100.000000 100.000000 \n", "\n", " confidence when correct hit confidence incorrect hit \\\n", "count 102.000000 84.000000 \n", "mean 74.990196 58.565476 \n", "std 14.165916 19.560653 \n", "min 29.500000 7.000000 \n", "25% 66.000000 46.375000 \n", "50% 75.000000 56.250000 \n", "75% 86.500000 73.500000 \n", "max 100.000000 92.000000 \n", "\n", " confidence correct reject confidence incorrect reject \\\n", "count 102.000000 93.000000 \n", "mean 71.137255 61.220430 \n", "std 14.987479 17.671283 \n", "min 19.000000 17.000000 \n", "25% 64.625000 50.000000 \n", "50% 71.250000 61.000000 \n", "75% 80.000000 74.000000 \n", "max 100.000000 100.000000 \n", "\n", " confidence when correct confidence when incorrect sci \\\n", "count 102.000000 99.000000 102.000000 \n", "mean 74.642157 61.979798 22.245098 \n", "std 13.619725 15.921670 7.547128 \n", "min 24.000000 24.500000 0.000000 \n", "25% 66.000000 51.000000 17.000000 \n", "50% 75.750000 61.500000 23.500000 \n", "75% 82.375000 73.000000 29.000000 \n", "max 100.000000 100.000000 32.000000 \n", "\n", " psqi ess \n", "count 102.000000 102.000000 \n", "mean 5.274510 7.294118 \n", "std 3.404007 4.426715 \n", "min 0.000000 0.000000 \n", "25% 3.000000 4.000000 \n", "50% 5.000000 7.000000 \n", "75% 7.000000 10.000000 \n", "max 15.000000 21.000000 " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This gives us a data frame with summary statistics. Note that in this data frame, the row indices are not integers, but are the names of the summary statistics. If we wanted to extract the median value of each entry, we could do that with `.loc`." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "participant number 52.50\n", "age 36.50\n", "correct hit percentage 90.00\n", "correct reject percentage 80.00\n", "percent correct 83.75\n", "confidence when correct hit 75.00\n", "confidence incorrect hit 56.25\n", "confidence correct reject 71.25\n", "confidence incorrect reject 61.00\n", "confidence when correct 75.75\n", "confidence when incorrect 61.50\n", "sci 23.50\n", "psqi 5.00\n", "ess 7.00\n", "Name: 50%, dtype: float64" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.describe().loc[\"50%\", :]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Outputting a new CSV file\n", "\n", "Now that we added the insomniac column, we might like to save our data frame as a new CSV that we can reload later. We use `df.to_csv()` for this with the `index` kwarg to ask Pandas not to explicitly write the indices to the file." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "df.to_csv(\"gfmt_sleep_with_insomnia.csv\", index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at what this file looks like." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "participant number,gender,age,correct hit percentage,correct reject percentage,percent correct,confidence when correct hit,confidence incorrect hit,confidence correct reject,confidence incorrect reject,confidence when correct,confidence when incorrect,sci,psqi,ess,insomnia\n", "8,f,39,65,80,72.5,91.0,90.0,93.0,83.5,93.0,90.0,9,13,2,True\n", "16,m,42,90,90,90.0,75.5,55.5,70.5,50.0,75.0,50.0,4,11,7,True\n", "18,f,31,90,95,92.5,89.5,90.0,86.0,81.0,89.0,88.0,10,9,3,True\n", "22,f,35,100,75,87.5,89.5,,71.0,80.0,88.0,80.0,13,8,20,True\n", "27,f,74,60,65,62.5,68.5,49.0,61.0,49.0,65.0,49.0,13,9,12,True\n", "28,f,61,80,20,50.0,71.0,63.0,31.0,72.5,64.5,70.5,15,14,2,True\n", "30,m,32,90,75,82.5,67.0,56.5,66.0,65.0,66.0,64.0,16,9,3,True\n", "33,m,62,45,90,67.5,54.0,37.0,65.0,81.5,62.0,61.0,14,9,9,True\n", "34,f,33,80,100,90.0,70.5,76.5,64.5,,68.0,76.5,14,12,10,True\n" ] } ], "source": [ "!head gfmt_sleep_with_insomnia.csv" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Very nice. Notice that by default Pandas leaves an empty field for NaNs, and we do not need the `na_values` kwarg when we load in this CSV file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Slicing with iloc\n", "\n", "Let's say we wanted to consider only the males in the study. We can slice them out and put them in a new data frame (actually a view into the existing data frame `df`) using Boolean indexing." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numbergenderagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectscipsqiessinsomnia
116m42909090.075.555.570.550.075.050.04117True
630m32907582.567.056.566.065.066.064.01693True
733m62459067.554.037.065.081.562.061.01499True
1878m311007085.092.0NaN81.060.087.560.014611True
1980m281005075.0100.0NaN100.0100.0100.0100.012712True
\n", "
" ], "text/plain": [ " participant number gender age correct hit percentage \\\n", "1 16 m 42 90 \n", "6 30 m 32 90 \n", "7 33 m 62 45 \n", "18 78 m 31 100 \n", "19 80 m 28 100 \n", "\n", " correct reject percentage percent correct confidence when correct hit \\\n", "1 90 90.0 75.5 \n", "6 75 82.5 67.0 \n", "7 90 67.5 54.0 \n", "18 70 85.0 92.0 \n", "19 50 75.0 100.0 \n", "\n", " confidence incorrect hit confidence correct reject \\\n", "1 55.5 70.5 \n", "6 56.5 66.0 \n", "7 37.0 65.0 \n", "18 NaN 81.0 \n", "19 NaN 100.0 \n", "\n", " confidence incorrect reject confidence when correct \\\n", "1 50.0 75.0 \n", "6 65.0 66.0 \n", "7 81.5 62.0 \n", "18 60.0 87.5 \n", "19 100.0 100.0 \n", "\n", " confidence when incorrect sci psqi ess insomnia \n", "1 50.0 4 11 7 True \n", "6 64.0 16 9 3 True \n", "7 61.0 14 9 9 True \n", "18 60.0 14 6 11 True \n", "19 100.0 12 7 12 True " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_male = df.loc[df[\"gender\"] == \"m\", :]\n", "\n", "df_male.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the indices of the original data frame came along for the ride. The third row, for example, has index 7. This might seem counterintuitive, but it is actually a good idea. Remember, indices of data frames do not have to be integers!\n", "\n", "If we do for some reason want to index by the order of the rows and columns in the data frame, we can use `iloc`." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "78" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Using iloc enables indexing by the corresponding sequence of integers\n", "df_male.iloc[3, 0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While this works, I warn that you really should not need to use `iloc`. If you find yourself using it, remember that you are relying on the sorting of the data frame, which may not be relevant nor reliable. So, **use `iloc` at your own risk, the consequences of which could be dire!**\n", "\n", "The one instance where I use `iloc` is when I don't care which row I get, I just want *a* row, so I will use something like `df.iloc[0, :]`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Renaming columns\n", "\n", "You may be annoyed with the rather lengthy syntax of access column names and wish to change them. Actually, you probably do not want to do this. Explicit is better than implicit! And furthermore, high level plotting libraries, as we will soon see, often automatically use column names for axis labels. So, let's instead lengthen a column name. Say we keep forgetting what ESS stands for an want to rename the ess column to \"Epworth Sleepiness Scale.\"\n", "\n", "Data frames have a nice `rename` method to do this. To rename the columns, we provide a dictionary where the keys are current column names and the corresponding values are the names we which to check them to. While we are at it, we will choose descriptive names for all three of the sleep quality indices." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
participant numbergenderagecorrect hit percentagecorrect reject percentagepercent correctconfidence when correct hitconfidence incorrect hitconfidence correct rejectconfidence incorrect rejectconfidence when correctconfidence when incorrectSleep Condition IndicatorPittsburgh Sleep Quality IndexEpworth Sleepiness Scaleinsomnia
08f39658072.591.090.093.083.593.090.09132True
116m42909090.075.555.570.550.075.050.04117True
218f31909592.589.590.086.081.089.088.01093True
322f351007587.589.5NaN71.080.088.080.013820True
427f74606562.568.549.061.049.065.049.013912True
\n", "
" ], "text/plain": [ " participant number gender age correct hit percentage \\\n", "0 8 f 39 65 \n", "1 16 m 42 90 \n", "2 18 f 31 90 \n", "3 22 f 35 100 \n", "4 27 f 74 60 \n", "\n", " correct reject percentage percent correct confidence when correct hit \\\n", "0 80 72.5 91.0 \n", "1 90 90.0 75.5 \n", "2 95 92.5 89.5 \n", "3 75 87.5 89.5 \n", "4 65 62.5 68.5 \n", "\n", " confidence incorrect hit confidence correct reject \\\n", "0 90.0 93.0 \n", "1 55.5 70.5 \n", "2 90.0 86.0 \n", "3 NaN 71.0 \n", "4 49.0 61.0 \n", "\n", " confidence incorrect reject confidence when correct \\\n", "0 83.5 93.0 \n", "1 50.0 75.0 \n", "2 81.0 89.0 \n", "3 80.0 88.0 \n", "4 49.0 65.0 \n", "\n", " confidence when incorrect Sleep Condition Indicator \\\n", "0 90.0 9 \n", "1 50.0 4 \n", "2 88.0 10 \n", "3 80.0 13 \n", "4 49.0 13 \n", "\n", " Pittsburgh Sleep Quality Index Epworth Sleepiness Scale insomnia \n", "0 13 2 True \n", "1 11 7 True \n", "2 9 3 True \n", "3 8 20 True \n", "4 9 12 True " ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Make a dictionary to rename columns\n", "rename_dict = {\n", " \"ess\": \"Epworth Sleepiness Scale\",\n", " \"sci\": \"Sleep Condition Indicator\",\n", " \"psqi\": \"Pittsburgh Sleep Quality Index\",\n", "}\n", "\n", "# Rename the columns\n", "df = df.rename(columns=rename_dict)\n", "\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can go on and on about indexing Pandas `DataFrame`s, because there are *many* ways to do it. For much more on all of the clever ways you can access data and subsets thereof in DataFrames, see the [Pandas docs on indexing](http://pandas.pydata.org/pandas-docs/stable/indexing.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A note on indexing and speed\n", "\n", "As we have seen Boolean indexing is very convenient and fits nicely into a logical framework which allows us to extract data according to criteria we want. The trade-off is speed. Slicing by Boolean indexing is essentially doing a reverse lookup in a dictionary. We have to loop through all values to find keys that match. This is much slower than directly indexing. Compare the difference in speed for indexing the percent correct by participant number 42 by Boolean indexing versus direct indexing (it's row 54)." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Boolean indexing:\n", "230 µs ± 18.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n", "\n", "Direct indexing:\n", "5.9 µs ± 197 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n" ] } ], "source": [ "print(\"Boolean indexing:\")\n", "%timeit df.loc[df['participant number'] == 42, 'percent correct']\n", "\n", "print(\"\\nDirect indexing:\")\n", "%timeit df.loc[54, 'percent correct']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The speed difference is stark, differing be almost two orders of magnitude. For larger data sets, or for analyses that require repeated indexing, this speed consideration may be important." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Computing environment" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPython 3.8.5\n", "IPython 7.18.1\n", "\n", "numpy 1.19.1\n", "pandas 1.1.1\n", "jupyterlab 2.2.6\n" ] } ], "source": [ "%load_ext watermark\n", "\n", "%watermark -v -p numpy,pandas,jupyterlab" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }