{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook will cover the assumed knowledge of pandas.\n",
"Here's a few questions to check if you already know the material in this notebook.\n",
"\n",
"1. Does a NumPy array have a single dtype or multiple dtypes?\n",
"2. Why is broadcasting useful?\n",
"3. How do you slice a DataFrame by row label?\n",
"4. How do you select a column of a DataFrame?\n",
"5. Is the Index a column in the DataFrame?\n",
"\n",
"If you feel pretty comfortable with those, go ahead and skip this notebook.\n",
"[Answers](#Answers) are at the end. We'll meet up at the next notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Aside: IPython Notebook\n",
"\n",
"- two modes command and edit\n",
" - command -> edit: `Enter`\n",
" - edit -> command: `Esc`\n",
"- `h` : Keyboard Shortcuts: (from command mode)\n",
"- `j` / `k` : navigate cells\n",
"- `shift+Enter` executes a cell"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Outline:\n",
"\n",
"- [NumPy Foundation](#NumPy-Foundation)\n",
"- [Pandas](#Pandas)\n",
"- [Data Structures](#Data-Structures)\n",
"\n",
"## Numpy Foundation\n",
"\n",
"pandas is built atop NumPy, historically and in the actual library.\n",
"It's helpful to have a good understanding of some NumPyisms. [Speak the vernacular](https://www.youtube.com/watch?v=u2yvNw49AX4).\n",
"\n",
"### ndarray\n",
"\n",
"The core of numpy is the `ndarray`, N-dimensional array. These are homogenously-typed, fixed-length data containers.\n",
"NumPy also provides many convenient and fast methods implemented on the `ndarray`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([1, 2, 3])"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from __future__ import print_function\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"x = np.array([1, 2, 3])\n",
"x"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"dtype('int64')"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x.dtype"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ True, False],\n",
" [False, True]], dtype=bool)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y = np.array([[True, False], [False, True]])\n",
"y"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(2, 2)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### dtypes\n",
"\n",
"Unlike python lists, NumPy arrays care about the type of data stored within.\n",
"The full list of NumPy dtypes can be found in the [NumPy documentation](http://docs.scipy.org/doc/numpy/user/basics.types.html).\n",
"\n",
"\n",
"\n",
"We sacrifice the convinience of mixing bools and ints and floats within an array for much better performance.\n",
"However, an unexpected `dtype` change will probably bite you at some point in the future.\n",
"\n",
"The two biggest things to remember are\n",
"\n",
"- Missing values (NaN) cast integer or boolean arrays to floats\n",
"- the object dtype is the fallback\n",
"\n",
"You'll want to avoid object dtypes. It's typically slow."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Broadcasting\n",
"\n",
"It's super cool and super useful. The one-line explanation is that when doing elementwise operations, things expand to the \"correct\" shape."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"x: [0 1 2 3 4]\n",
"x+1: [1 2 3 4 5]\n",
"\n",
"y: \n",
"[[ 0.27612315 0.18577684 0.9610967 0.92516769 0.43423458]\n",
" [ 0.41534299 0.58181509 0.49158344 0.43734366 0.09782122]]\n",
"y+1:\n",
"[[ 1.27612315 1.18577684 1.9610967 1.92516769 1.43423458]\n",
" [ 1.41534299 1.58181509 1.49158344 1.43734366 1.09782122]]\n"
]
}
],
"source": [
"# add a scalar to a 1-d array\n",
"x = np.arange(5)\n",
"print('x: ', x)\n",
"print('x+1:', x + 1, end='\\n\\n')\n",
"\n",
"y = np.random.uniform(size=(2, 5))\n",
"print('y: ', y, sep='\\n')\n",
"print('y+1:', y + 1, sep='\\n')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since `x` is shaped `(5,)` and `y` is shaped `(2,5)` we can do operations between them."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0. , 0.18577684, 1.9221934 , 2.77550307, 1.73693832],\n",
" [ 0. , 0.58181509, 0.98316688, 1.31203099, 0.3912849 ]])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x * y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Without broadcasting we'd have to manually reshape our arrays, which quickly gets annoying."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0. , 0.18577684, 1.9221934 , 2.77550307, 1.73693832],\n",
" [ 0. , 0.58181509, 0.98316688, 1.31203099, 0.3912849 ]])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x.reshape(1, 5).repeat(2, axis=0) * y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Pandas\n",
"\n",
"We'll breeze through the basics here, and get onto some interesting applications in a bit. I want to provide the *barest* of intuition so things stick down the road.\n",
"\n",
"## Why pandas\n",
"\n",
"NumPy is great. But it lacks a few things that are conducive to doing statisitcal analysis. By building on top of NumPy, pandas provides\n",
"\n",
"- labeled arrays\n",
"- heterogenous data types within a table\n",
"- better missing data handling\n",
"- convenient methods \n",
"- more data types (Categorical, Datetime)\n",
"\n",
"## Data Structures\n",
"\n",
"This is the typical starting point for any intro to pandas.\n",
"We'll follow suit.\n",
"\n",
"### The DataFrame\n",
"\n",
"Here we have the workhorse data structure for pandas.\n",
"It's an in-memory table holding your data, and provides a few conviniences over lists of lists or NumPy arrays."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" A | \n",
" B | \n",
" C | \n",
"
\n",
" \n",
" \n",
" \n",
" | a | \n",
" 1 | \n",
" True | \n",
" 0.496714 | \n",
"
\n",
" \n",
" | b | \n",
" 2 | \n",
" True | \n",
" -0.138264 | \n",
"
\n",
" \n",
" | c | \n",
" 3 | \n",
" False | \n",
" 0.647689 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" A | \n",
" C | \n",
"
\n",
" \n",
" \n",
" \n",
" | a | \n",
" 1 | \n",
" 0.496714 | \n",
"
\n",
" \n",
" | b | \n",
" 2 | \n",
" -0.138264 | \n",
"
\n",
" \n",
" | c | \n",
" 3 | \n",
" 0.647689 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" A | \n",
" B | \n",
" C | \n",
"
\n",
" \n",
" \n",
" \n",
" | a | \n",
" 1 | \n",
" True | \n",
" 0.496714 | \n",
"
\n",
" \n",
" | b | \n",
" 2 | \n",
" True | \n",
" -0.138264 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" A | \n",
" B | \n",
" C | \n",
"
\n",
" \n",
" \n",
" \n",
" | a | \n",
" 1 | \n",
" True | \n",
" 0.496714 | \n",
"
\n",
" \n",
" | b | \n",
" 2 | \n",
" True | \n",
" -0.138264 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" A | \n",
" B | \n",
" C | \n",
"
\n",
" \n",
" \n",
" \n",
" | a | \n",
" 1 | \n",
" True | \n",
" 0.496714 | \n",
"
\n",
" \n",
" | b | \n",
" 2 | \n",
" True | \n",
" -0.138264 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" A | \n",
" C | \n",
"
\n",
" \n",
" \n",
" \n",
" | a | \n",
" 1 | \n",
" 0.496714 | \n",
"
\n",
" \n",
" | b | \n",
" 2 | \n",
" -0.138264 | \n",
"
\n",
" \n",
"
\n",
"