{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook is part of my [Python data science curriculum](http://www.terran.us/articles/python_curriculum.html)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Series indexing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this table, the columns indicate that type of the index. Indexes behave differently depending on whether the values are on monotonic (M), non-monotonic order (NM), or contain duplicates (D), so multiple cases have been included. \n",
"\n",
"The rows indicate the arguments to the bracket operator. Slices can behave differently depending on whether the endpoints are present in the index or not. [] indicates both endpoints are included; \\[) indicates the right endpoint of the slice is not present in the index. When the index is nonmonotonic, slices always return rows in the order they are present in the Series - they never skip around based on the values of the index.\n",
"\n",
"| | Int M | Int NM | Float M | Float NM | Str M | Str NM | Str D | Tup M | Tup NM |\n",
"|------------- |-------|--------|-------- | -------- | ----- | ------ | ----- | ------| ------ |\n",
"| single str | - | - | - | - |Idx | Idx | Idx*1 | - | - |\n",
"| str list | - | - | - | - |Idx {} |Idx {} | Idx {}| - | - |\n",
"| str slc [] | - | - | - | - |Idx \\[]|Idx \\[] |Idx \\[]| - | - |\n",
"| str slc [) | - | - | - | - |Idx \\[)| - |Idx \\[)| - | - |\n",
"| single int | Idx | Idx | Idx | Idx |Row | Row | - | Row | Row |\n",
"| int list |Idx {} | Idx {} | Idx {} | Idx {} |Row {} | Row {} |Row {}| Row {}|Row {} |\n",
"| int slc [] |Row \\[)| Row \\[)| Idx \\[] | Idx \\[] |Row \\[)| Row \\[)|Row \\[)|Row \\[)|Row \\[) |\n",
"| int slc [) |Row \\[)| Row \\[)| Idx \\[) | - |Row \\[)| Row \\[)|Row \\[)|Row \\[)|Row \\[) |\n",
"| single float | Idx | Idx | Idx | Idx | - | - | - |- | - |\n",
"| float list | Idx {}| Idx {} | Idx {} | Idx {} | - | - | - |- | - |\n",
"| float slc [] | - | - | Idx \\[] | Idx \\[] | - | - | - |- | - |\n",
"| float slc [) | - | - | Idx \\[) | - | - | - | - |- | - |\n",
"| single tuple | - | - | - | - | - | - | - |Idx | Idx |\n",
"| tuple slc [] | - | - | - | - | - | - | - |Idx \\[]| Idx \\[]|\n",
"| tuple slc [) | - | - | - | - | - | - | - |- | - |\n",
"\n",
"\\[) indicates that the right endpoint is not included \n",
"\\[] indicates that the right endpoint is included \n",
"\\- indicates that combination doesn't work and you get an exception\n",
"\n",
"*1 Returns a scalar if the key matched one value, or a Series if it matched more than one.\n",
"\n",
"Slices always return a Series (possibly empty or with a single element). \n",
"Single indexes always return a scalar."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Series Construction"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"RangeIndex(start=0, stop=5, step=1)"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# If you do not provide an index, a RangeIndex is created for you:\n",
"sn = pd.Series([0,11,22,33,44])\n",
"sn.index"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"RangeIndex(start=10, stop=15, step=1)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# You can also explicitly create a RangeIndex starting somewhere other than zero:\n",
"sr = pd.Series([0,11,22,33,44], index=pd.RangeIndex(10,15))\n",
"sr.index\n",
"\n",
"# As far as I can tell, a RangeIndex is functionally equivalent to an IntegerIndex,\n",
"# but there might be corner cases I haven't found yet!"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"Index(['a', 'b', 'c', 'd', 'e'], dtype='object')"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Strings get dtype object\n",
"scm = pd.Series([0,11,22,33,44], index=['a','b','c','d','e'])\n",
"scn = pd.Series([0,11,22,33,44], index=['a','b','c','f','e'])\n",
"scd = pd.Series([0,11,22,33,44], index=['a','b','c','c','e'])\n",
"scm.index"
]
},
{
"cell_type": "code",
"execution_count": 103,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"Int64Index([10, 11, 12, 14, 13], dtype='int64')"
]
},
"execution_count": 103,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sim = pd.Series([0,11,22,33,44], index=[10,11,12,14,13])\n",
"sin = pd.Series([0,11,22,33,44], index=[10,11,12,15,13])\n",
"sid = pd.Series([0,11,22,33,44], index=[10,11,12,12,13])\n",
"sim.index"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"Float64Index([10.0, 11.0, 12.0, 13.0, 14.0], dtype='float64')"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sfm = pd.Series([0,11,22,33,44], index=[10.0,11,12,13,14])\n",
"sfn = pd.Series([0,11,22,33,44], index=[10.0,11,12,15,14])\n",
"sfm.index"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(10, 10) 0\n",
"(11, 11) 11\n",
"(12, 12) 22\n",
"(14, 14) 33\n",
"(13, 13) 44\n",
"dtype: int64"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Finally, let's create one with complex objects (e.g. tuples) for an index:\n",
"stm = pd.Series([0,11,22,33,44], index=[(10,10), (11,11), (12,12), (14,14), (13,13)])\n",
"stn = pd.Series([0,11,22,33,44], index=[(10,10), (11,11), (12,12), (15,15), (13,13)])\n",
"stm"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Examples of the Most Counterintuitive Cases"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"22"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# With an integer index, a single element uses the index\n",
"sim[12]"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"12 22\n",
"dtype: int64"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# but a slice uses the row numbers:\n",
"sim[2:3]"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"12.0 22\n",
"15.0 33\n",
"14.0 44\n",
"dtype: int64"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# if your index were a float instead of int, the slice _would_ use the index,\n",
"# and it would include both endpoints:\n",
"sfn[12:14]"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"11"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# If you have an integer index, using a single float works and uses the index:\n",
"sim[11.0]"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TypeError(\"cannot do slice indexing on <class 'pandas.core.indexes.numeric.Int64Index'> with these indexers [11.0] of <class 'float'>\")\n"
]
}
],
"source": [
"# but a float slice is an error:\n",
"try:\n",
" sim[11.0:12.0]\n",
"except Exception as e:\n",
" print(repr(e))"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"e 44\n",
"dtype: int64\n",
"KeyError('q')\n"
]
}
],
"source": [
"# With an Ojbect dtype that's actually characters, you can slice with values that are not\n",
"# in the index, if and only if the index is monotonic:\n",
"print(scm['e':'q'])\n",
"try:\n",
" scn['e':'q']\n",
"except Exception as e:\n",
" print(repr(e))"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"True\n",
"(10, 10) 0\n",
"(11, 11) 11\n",
"dtype: int64\n",
"KeyError((17, 17))\n"
]
}
],
"source": [
"# but with other objects this doesn't work, even if comparison is defined - only elements\n",
"# present in the index can be used even if the index is monotonic:\n",
"print((10,10) < (12,13))\n",
"print(stm[(10,10):(11,11)])\n",
"try:\n",
" stm[(10,10):(17,17)]\n",
"except Exception as e:\n",
" print(repr(e))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Even More Problems"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {
"hidden": true
},
"outputs": [
{
"ename": "IndexError",
"evalue": "1",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m~/.pyenv/versions/anaconda3-5.3.0/lib/python3.7/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m 3117\u001b[0m return self._engine.get_value(s, k,\n\u001b[0;32m-> 3118\u001b[0;31m tz=getattr(series.dtype, 'tz', None))\n\u001b[0m\u001b[1;32m 3119\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine._get_loc_duplicates\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mTypeError\u001b[0m: '<' not supported between instances of 'str' and 'int'",
"\nDuring handling of the above exception, another exception occurred:\n",
"\u001b[0;31mIndexError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-82-ba3497aaf88f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mscd\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSeries\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m11\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m22\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m33\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m44\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'a'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'b'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'c'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'c'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'e'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mscd\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m~/.pyenv/versions/anaconda3-5.3.0/lib/python3.7/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m 765\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply_if_callable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 766\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 767\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 768\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 769\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mis_scalar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/.pyenv/versions/anaconda3-5.3.0/lib/python3.7/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m 3136\u001b[0m \u001b[0;31m# python 3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3137\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mis_scalar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pragma: no cover\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3138\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mIndexError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3139\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mInvalidIndexError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3140\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mIndexError\u001b[0m: 1"
]
}
],
"source": [
"# Beyond the chart above, there are _even more_ cases. Consider what happens when we have a DUPLICATE label - now\n",
"# we can't index by row anymore\n",
"\n",
"scd = pd.Series([0,11,22,33,44], index=['a','b','c','c','e'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## loc and iloc"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"As an alternative, always use .loc (by Index value) and .iloc(by integer starting at 0). You will still have to remember that monotonic and nonmonotonic indices behave differently when you try to slice with an endpoint that is not present, but most of the other heuristics are simplified away.\n",
"\n",
"Note that these are not functions - they expose a special attribute which you then access with [] as usual, so you can get slices."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"11 11\n",
"dtype: int64"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sin[1:2]"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"11 11\n",
"dtype: int64"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sin.iloc[1:2]"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"11 11\n",
"12 22\n",
"dtype: int64"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sin.loc[11:12]"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"11.0 11\n",
"12.0 22\n",
"dtype: int64"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sfn[11:12]"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"11.0 11\n",
"dtype: int64"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sfn.iloc[1:2]"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"11.0 11\n",
"12.0 22\n",
"dtype: int64"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sfn.loc[11:12]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Dataframe Indexing"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Dataframe Construction"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"RangeIndex(start=0, stop=1, step=1)\n",
"RangeIndex(start=0, stop=6, step=1)\n"
]
}
],
"source": [
"# If you don't supply column or row names, you get an implicit RangeIndex:\n",
"dn = pd.DataFrame(np.arange(0,6))\n",
"print(dn.columns)\n",
"print(dn.index)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1 2\n",
"0 0 1 2\n",
"1 1 2 3"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# If you construct a dataframe from a list, the elements are taken as ROWS, not columns:\n",
"pd.DataFrame([np.arange(0,3), np.arange(1,4)])"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>x1</th>\n",
" <th>x2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" x1 x2\n",
"0 0 1\n",
"1 1 2\n",
"2 2 3"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# To construct it from columns, pass them as a dict with names:\n",
"pd.DataFrame({'x1':np.arange(0,3), 'x2':np.arange(1,4)})"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 1\n",
"0 0 1\n",
"1 1 2\n",
"2 2 3"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# The only way I know of to construct it from columns without supplying names is to go via a\n",
"# Numpy 2d array:\n",
"pd.DataFrame(np.hstack([np.arange(0,3).reshape(3,1), np.arange(1,4).reshape(3,1)]))"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>0</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 0 0\n",
"0 0 1\n",
"1 1 2\n",
"2 2 3"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# If you make two dataframes and concatenate them, note that both columns are\n",
"# called 0:\n",
"pd.concat([pd.DataFrame(np.arange(0,3)),pd.DataFrame(np.arange(1,4))], axis=1)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>w</th>\n",
" <th>v</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>11</td>\n",
" <td>22</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>22</td>\n",
" <td>33</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>33</td>\n",
" <td>44</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>44</td>\n",
" <td>55</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>55</td>\n",
" <td>66</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" w v\n",
"0 0 11\n",
"1 11 22\n",
"2 22 33\n",
"3 33 44\n",
"4 44 55\n",
"5 55 66"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# In practice, columns will essentially always have names, so we consider only two cases here,\n",
"# with and without named rows.\n",
"dn = pd.DataFrame({'w':np.arange(0,66,11),'v':np.arange(11,77,11)})\n",
"dn"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>w</th>\n",
" <th>v</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>a</th>\n",
" <td>0</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>11</td>\n",
" <td>22</td>\n",
" </tr>\n",
" <tr>\n",
" <th>c</th>\n",
" <td>22</td>\n",
" <td>33</td>\n",
" </tr>\n",
" <tr>\n",
" <th>d</th>\n",
" <td>33</td>\n",
" <td>44</td>\n",
" </tr>\n",
" <tr>\n",
" <th>e</th>\n",
" <td>44</td>\n",
" <td>55</td>\n",
" </tr>\n",
" <tr>\n",
" <th>f</th>\n",
" <td>55</td>\n",
" <td>66</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" w v\n",
"a 0 11\n",
"b 11 22\n",
"c 22 33\n",
"d 33 44\n",
"e 44 55\n",
"f 55 66"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds = dn.copy()\n",
"ds.index=['a','b','c','d','e','f']\n",
"ds"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>w</th>\n",
" <th>v</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>0</td>\n",
" <td>11</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>11</td>\n",
" <td>22</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>22</td>\n",
" <td>33</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>33</td>\n",
" <td>44</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>44</td>\n",
" <td>55</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15</th>\n",
" <td>55</td>\n",
" <td>66</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" w v\n",
"10 0 11\n",
"11 11 22\n",
"12 22 33\n",
"13 33 44\n",
"14 44 55\n",
"15 55 66"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# We also need one with an integer index, where the integers don't start at 0.\n",
"di = dn.copy()\n",
"di.index=[10,11,12,13,14,15]\n",
"di"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## DataFrame single indexing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this table, we need to record whether it tries to index the rows or columns, whether it\n",
"tries to use the Index (named) or Row (integers starting at 0) naming, and whether it returns the result as a DataFrame (2d) or Series (1d).\n",
"\n",
"| Row Idx | Arg | Axis | Lookup | Result | Right Endpt |\n",
"| --------|------- | -----| -------| -------|-------|\n",
"| any | scalar | Col | Index | Series | |\n",
"| any | list | Col | Index | DF | |\n",
"| str | str slc| Row | Index | DF | incl |\n",
"| str | int slc| Row | Row | DF | excl |\n",
"| int | int slc| Row | Row | DF | excl |"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Dataframe Double Indexing"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Since ix is such a disaster, we will consider only loc and iloc. Any time you use a slice (even if it has 1 element) or a list, that dimension is preserved; any time you use a scalar, that dimension is dropped."
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"22"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# This returns a single number\n",
"ds.loc['b','v']"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>w</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>11</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" w\n",
"b 11"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# This returns a 1x1 data frame\n",
"ds.loc['b':'b','w':'w']"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>w</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>11</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" w\n",
"b 11"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# As does this\n",
"ds.loc[['b'],['w']]"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>w</th>\n",
" <th>v</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>11</td>\n",
" <td>22</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" w v\n",
"b 11 22"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Since the column names are monotonically decreasing, I can refer to a value which isn't present as part of the slice:\n",
"ds.loc['b':'b','z':'v']"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"w 11\n",
"v 22\n",
"Name: b, dtype: int64"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# If I want to drop one dimension and keep the other, I can do so like this. When a dimension is dropped, I always\n",
"# get back a Series, which will print as a column. Note the metadata:\n",
"ds.loc['b','w':'v']"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"22"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# All the same ideas apply with iloc\n",
"ds.iloc[1,1]"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>v</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>22</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" v\n",
"b 22"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Note that iloc integer slices exclude the right endpoint:\n",
"ds.iloc[1:2,1:2]"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>v</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>22</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" v\n",
"b 22"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds.iloc[[1],[1]]"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>w</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>b</th>\n",
" <td>11</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" w\n",
"b 11"
]
},
"execution_count": 64,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# The only trick part is when you want to index rows by name and columns by number or vice versa.\n",
"# Some approaches are shown here:\n",
"# http://pandas-docs.github.io/pandas-docs-travis/indexing.html#ix-indexer-is-deprecated\n",
"# but I find those approaches expose a disgusting amount of implementation detail, and are just as unusable in their\n",
"# own way as .ix was.\n",
"#\n",
"# I prefer to simply index in two steps:\n",
"ds.iloc[[1],:].loc[:,['w']]"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.38 ms ± 23.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n",
"1.06 ms ± 48.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/terran/.pyenv/versions/anaconda3-5.3.0/lib/python3.7/site-packages/ipykernel_launcher.py:1: DeprecationWarning: \n",
".ix is deprecated. Please use\n",
".loc for label based indexing or\n",
".iloc for positional indexing\n",
"\n",
"See the documentation here:\n",
"http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated\n",
" \"\"\"Entry point for launching an IPython kernel.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.02 ms ± 57.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
}
],
"source": [
"# The difference in time is probably not important for most purposes\n",
"%timeit ds.iloc[[1],:].loc[:,['w']]\n",
"%timeit ds.iloc[[1], ds.columns.get_indexer(['w'])]\n",
"%timeit ds.ix[[1],['w']]"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.5 ms ± 64.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
}
],
"source": [
"# Alternatively, we could convert the numbers to names and use .loc:\n",
"%timeit ds.loc[[ds.index[1]], ['w']]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"hidden": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 2
}