{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Combining Datasets: merge and join"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"One important feature offered by Pandas is its high-performance, in-memory join and merge operations, which you may be familiar with if you have ever worked with databases.\n",
"The main interface for this is the `pd.merge` function, and we'll see a few examples of how this can work in practice.\n",
"\n",
"For convenience, we will again define the `display` function from the previous chapter after the usual imports:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"class display(object):\n",
" \"\"\"Display HTML representation of multiple objects\"\"\"\n",
" template = \"\"\"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>{0}</p>{1}\n",
" </div>\"\"\"\n",
" def __init__(self, *args):\n",
" self.args = args\n",
" \n",
" def _repr_html_(self):\n",
" return '\\n'.join(self.template.format(a, eval(a)._repr_html_())\n",
" for a in self.args)\n",
" \n",
" def __repr__(self):\n",
" return '\\n\\n'.join(a + '\\n' + repr(eval(a))\n",
" for a in self.args)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Relational Algebra\n",
"\n",
"The behavior implemented in `pd.merge` is a subset of what is known as *relational algebra*, which is a formal set of rules for manipulating relational data that forms the conceptual foundation of operations available in most databases.\n",
"The strength of the relational algebra approach is that it proposes several fundamental operations, which become the building blocks of more complicated operations on any dataset.\n",
"With this lexicon of fundamental operations implemented efficiently in a database or other program, a wide range of fairly complicated composite operations can be performed.\n",
"\n",
"Pandas implements several of these fundamental building blocks in the `pd.merge` function and the related `join` method of `Series` and `DataFrame` objects.\n",
"As you will see, these let you efficiently link data from different sources."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Categories of Joins\n",
"\n",
"The `pd.merge` function implements a number of types of joins: *one-to-one*, *many-to-one*, and *many-to-many*.\n",
"All three types of joins are accessed via an identical call to the `pd.merge` interface; the type of join performed depends on the form of the input data.\n",
"We'll start with some simple examples of the three types of merges, and discuss detailed options a bit later."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### One-to-One Joins\n",
"\n",
"Perhaps the simplest type of merge is the one-to-one join, which is in many ways similar to the column-wise concatenation you saw in [Combining Datasets: Concat & Append](03.06-Concat-And-Append.ipynb).\n",
"As a concrete example, consider the following two `DataFrame` objects, which contain information on several employees in a company:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df1</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df2</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Lisa</td>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Bob</td>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Jake</td>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df1\n",
" employee group\n",
"0 Bob Accounting\n",
"1 Jake Engineering\n",
"2 Lisa Engineering\n",
"3 Sue HR\n",
"\n",
"df2\n",
" employee hire_date\n",
"0 Lisa 2004\n",
"1 Bob 2008\n",
"2 Jake 2012\n",
"3 Sue 2014"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1 = pd.DataFrame({'employee': ['Bob', 'Jake', 'Lisa', 'Sue'],\n",
" 'group': ['Accounting', 'Engineering',\n",
" 'Engineering', 'HR']})\n",
"df2 = pd.DataFrame({'employee': ['Lisa', 'Bob', 'Jake', 'Sue'],\n",
" 'hire_date': [2004, 2008, 2012, 2014]})\n",
"display('df1', 'df2')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To combine this information into a single `DataFrame`, we can use the `pd.merge` function:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" employee group hire_date\n",
"0 Bob Accounting 2008\n",
"1 Jake Engineering 2012\n",
"2 Lisa Engineering 2004\n",
"3 Sue HR 2014"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df3 = pd.merge(df1, df2)\n",
"df3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `pd.merge` function recognizes that each `DataFrame` has an `employee` column, and automatically joins using this column as a key.\n",
"The result of the merge is a new `DataFrame` that combines the information from the two inputs.\n",
"Notice that the order of entries in each column is not necessarily maintained: in this case, the order of the `employee` column differs between `df1` and `df2`, and the `pd.merge` function correctly accounts for this.\n",
"Additionally, keep in mind that the merge in general discards the index, except in the special case of merges by index (see the `left_index` and `right_index` keywords, discussed momentarily)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Many-to-One Joins"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Many-to-one joins are joins in which one of the two key columns contains duplicate entries.\n",
"For the many-to-one case, the resulting `DataFrame` will preserve those duplicate entries as appropriate.\n",
"Consider the following example of a many-to-one join:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df3</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df4</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>group</th>\n",
" <th>supervisor</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Accounting</td>\n",
" <td>Carly</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Engineering</td>\n",
" <td>Guido</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>HR</td>\n",
" <td>Steve</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df3, df4)</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" <th>hire_date</th>\n",
" <th>supervisor</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" <td>2008</td>\n",
" <td>Carly</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" <td>2012</td>\n",
" <td>Guido</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" <td>2004</td>\n",
" <td>Guido</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" <td>2014</td>\n",
" <td>Steve</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df3\n",
" employee group hire_date\n",
"0 Bob Accounting 2008\n",
"1 Jake Engineering 2012\n",
"2 Lisa Engineering 2004\n",
"3 Sue HR 2014\n",
"\n",
"df4\n",
" group supervisor\n",
"0 Accounting Carly\n",
"1 Engineering Guido\n",
"2 HR Steve\n",
"\n",
"pd.merge(df3, df4)\n",
" employee group hire_date supervisor\n",
"0 Bob Accounting 2008 Carly\n",
"1 Jake Engineering 2012 Guido\n",
"2 Lisa Engineering 2004 Guido\n",
"3 Sue HR 2014 Steve"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df4 = pd.DataFrame({'group': ['Accounting', 'Engineering', 'HR'],\n",
" 'supervisor': ['Carly', 'Guido', 'Steve']})\n",
"display('df3', 'df4', 'pd.merge(df3, df4)')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The resulting `DataFrame` has an additional column with the \"supervisor\" information, where the information is repeated in one or more locations as required by the inputs."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Many-to-Many Joins"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Many-to-many joins may be a bit confusing conceptually, but are nevertheless well defined.\n",
"If the key column in both the left and right arrays contains duplicates, then the result is a many-to-many merge.\n",
"This will be perhaps most clear with a concrete example.\n",
"Consider the following, where we have a `DataFrame` showing one or more skills associated with a particular group.\n",
"By performing a many-to-many join, we can recover the skills associated with any individual person:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df1</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df5</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>group</th>\n",
" <th>skills</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Accounting</td>\n",
" <td>math</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Accounting</td>\n",
" <td>spreadsheets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Engineering</td>\n",
" <td>software</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Engineering</td>\n",
" <td>math</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>HR</td>\n",
" <td>spreadsheets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>HR</td>\n",
" <td>organization</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df1, df5)</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" <th>skills</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" <td>math</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" <td>spreadsheets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" <td>software</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" <td>math</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" <td>software</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" <td>math</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" <td>spreadsheets</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" <td>organization</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df1\n",
" employee group\n",
"0 Bob Accounting\n",
"1 Jake Engineering\n",
"2 Lisa Engineering\n",
"3 Sue HR\n",
"\n",
"df5\n",
" group skills\n",
"0 Accounting math\n",
"1 Accounting spreadsheets\n",
"2 Engineering software\n",
"3 Engineering math\n",
"4 HR spreadsheets\n",
"5 HR organization\n",
"\n",
"pd.merge(df1, df5)\n",
" employee group skills\n",
"0 Bob Accounting math\n",
"1 Bob Accounting spreadsheets\n",
"2 Jake Engineering software\n",
"3 Jake Engineering math\n",
"4 Lisa Engineering software\n",
"5 Lisa Engineering math\n",
"6 Sue HR spreadsheets\n",
"7 Sue HR organization"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df5 = pd.DataFrame({'group': ['Accounting', 'Accounting',\n",
" 'Engineering', 'Engineering', 'HR', 'HR'],\n",
" 'skills': ['math', 'spreadsheets', 'software', 'math',\n",
" 'spreadsheets', 'organization']})\n",
"display('df1', 'df5', \"pd.merge(df1, df5)\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These three types of joins can be used with other Pandas tools to implement a wide array of functionality.\n",
"But in practice, datasets are rarely as clean as the one we're working with here.\n",
"In the following section we'll consider some of the options provided by `pd.merge` that enable you to tune how the join operations work."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Specification of the Merge Key"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We've already seen the default behavior of `pd.merge`: it looks for one or more matching column names between the two inputs, and uses this as the key.\n",
"However, often the column names will not match so nicely, and `pd.merge` provides a variety of options for handling this."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The on Keyword\n",
"\n",
"Most simply, you can explicitly specify the name of the key column using the `on` keyword, which takes a column name or a list of column names:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df1</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df2</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Lisa</td>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Bob</td>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Jake</td>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df1, df2, on='employee')</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df1\n",
" employee group\n",
"0 Bob Accounting\n",
"1 Jake Engineering\n",
"2 Lisa Engineering\n",
"3 Sue HR\n",
"\n",
"df2\n",
" employee hire_date\n",
"0 Lisa 2004\n",
"1 Bob 2008\n",
"2 Jake 2012\n",
"3 Sue 2014\n",
"\n",
"pd.merge(df1, df2, on='employee')\n",
" employee group hire_date\n",
"0 Bob Accounting 2008\n",
"1 Jake Engineering 2012\n",
"2 Lisa Engineering 2004\n",
"3 Sue HR 2014"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"display('df1', 'df2', \"pd.merge(df1, df2, on='employee')\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This option works only if both the left and right ``DataFrame``s have the specified column name."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The left_on and right_on Keywords\n",
"\n",
"At times you may wish to merge two datasets with different column names; for example, we may have a dataset in which the employee name is labeled as \"name\" rather than \"employee\".\n",
"In this case, we can use the `left_on` and `right_on` keywords to specify the two column names:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df1</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df3</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>salary</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>70000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>80000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>120000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>90000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df1, df3, left_on=\"employee\", right_on=\"name\")</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" <th>name</th>\n",
" <th>salary</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" <td>Bob</td>\n",
" <td>70000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" <td>Jake</td>\n",
" <td>80000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" <td>Lisa</td>\n",
" <td>120000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" <td>Sue</td>\n",
" <td>90000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df1\n",
" employee group\n",
"0 Bob Accounting\n",
"1 Jake Engineering\n",
"2 Lisa Engineering\n",
"3 Sue HR\n",
"\n",
"df3\n",
" name salary\n",
"0 Bob 70000\n",
"1 Jake 80000\n",
"2 Lisa 120000\n",
"3 Sue 90000\n",
"\n",
"pd.merge(df1, df3, left_on=\"employee\", right_on=\"name\")\n",
" employee group name salary\n",
"0 Bob Accounting Bob 70000\n",
"1 Jake Engineering Jake 80000\n",
"2 Lisa Engineering Lisa 120000\n",
"3 Sue HR Sue 90000"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df3 = pd.DataFrame({'name': ['Bob', 'Jake', 'Lisa', 'Sue'],\n",
" 'salary': [70000, 80000, 120000, 90000]})\n",
"display('df1', 'df3', 'pd.merge(df1, df3, left_on=\"employee\", right_on=\"name\")')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The result has a redundant column that we can drop if desired—for example, by using the `DataFrame.drop()` method:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee</th>\n",
" <th>group</th>\n",
" <th>salary</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>Accounting</td>\n",
" <td>70000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>Engineering</td>\n",
" <td>80000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>Engineering</td>\n",
" <td>120000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>HR</td>\n",
" <td>90000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" employee group salary\n",
"0 Bob Accounting 70000\n",
"1 Jake Engineering 80000\n",
"2 Lisa Engineering 120000\n",
"3 Sue HR 90000"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.merge(df1, df3, left_on=\"employee\", right_on=\"name\").drop('name', axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The left_index and right_index Keywords\n",
"\n",
"Sometimes, rather than merging on a column, you would instead like to merge on an index.\n",
"For example, your data might look like this:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df1a</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>group</th>\n",
" </tr>\n",
" <tr>\n",
" <th>employee</th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Bob</th>\n",
" <td>Accounting</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Jake</th>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Lisa</th>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Sue</th>\n",
" <td>HR</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df2a</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" <tr>\n",
" <th>employee</th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Lisa</th>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Bob</th>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Jake</th>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Sue</th>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df1a\n",
" group\n",
"employee \n",
"Bob Accounting\n",
"Jake Engineering\n",
"Lisa Engineering\n",
"Sue HR\n",
"\n",
"df2a\n",
" hire_date\n",
"employee \n",
"Lisa 2004\n",
"Bob 2008\n",
"Jake 2012\n",
"Sue 2014"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1a = df1.set_index('employee')\n",
"df2a = df2.set_index('employee')\n",
"display('df1a', 'df2a')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use the index as the key for merging by specifying the `left_index` and/or `right_index` flags in `pd.merge()`:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df1a</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>group</th>\n",
" </tr>\n",
" <tr>\n",
" <th>employee</th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Bob</th>\n",
" <td>Accounting</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Jake</th>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Lisa</th>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Sue</th>\n",
" <td>HR</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df2a</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" <tr>\n",
" <th>employee</th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Lisa</th>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Bob</th>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Jake</th>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Sue</th>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df1a, df2a, left_index=True, right_index=True)</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>group</th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" <tr>\n",
" <th>employee</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Bob</th>\n",
" <td>Accounting</td>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Jake</th>\n",
" <td>Engineering</td>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Lisa</th>\n",
" <td>Engineering</td>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Sue</th>\n",
" <td>HR</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df1a\n",
" group\n",
"employee \n",
"Bob Accounting\n",
"Jake Engineering\n",
"Lisa Engineering\n",
"Sue HR\n",
"\n",
"df2a\n",
" hire_date\n",
"employee \n",
"Lisa 2004\n",
"Bob 2008\n",
"Jake 2012\n",
"Sue 2014\n",
"\n",
"pd.merge(df1a, df2a, left_index=True, right_index=True)\n",
" group hire_date\n",
"employee \n",
"Bob Accounting 2008\n",
"Jake Engineering 2012\n",
"Lisa Engineering 2004\n",
"Sue HR 2014"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"display('df1a', 'df2a',\n",
" \"pd.merge(df1a, df2a, left_index=True, right_index=True)\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For convenience, Pandas includes the `DataFrame.join()` method, which performs an index-based merge without extra keywords:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>group</th>\n",
" <th>hire_date</th>\n",
" </tr>\n",
" <tr>\n",
" <th>employee</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Bob</th>\n",
" <td>Accounting</td>\n",
" <td>2008</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Jake</th>\n",
" <td>Engineering</td>\n",
" <td>2012</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Lisa</th>\n",
" <td>Engineering</td>\n",
" <td>2004</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Sue</th>\n",
" <td>HR</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" group hire_date\n",
"employee \n",
"Bob Accounting 2008\n",
"Jake Engineering 2012\n",
"Lisa Engineering 2004\n",
"Sue HR 2014"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df1a.join(df2a)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you'd like to mix indices and columns, you can combine `left_index` with `right_on` or `left_on` with `right_index` to get the desired behavior:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df1a</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>group</th>\n",
" </tr>\n",
" <tr>\n",
" <th>employee</th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Bob</th>\n",
" <td>Accounting</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Jake</th>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Lisa</th>\n",
" <td>Engineering</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Sue</th>\n",
" <td>HR</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df3</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>salary</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>70000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>80000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>120000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>90000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df1a, df3, left_index=True, right_on='name')</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>group</th>\n",
" <th>name</th>\n",
" <th>salary</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Accounting</td>\n",
" <td>Bob</td>\n",
" <td>70000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Engineering</td>\n",
" <td>Jake</td>\n",
" <td>80000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Engineering</td>\n",
" <td>Lisa</td>\n",
" <td>120000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>HR</td>\n",
" <td>Sue</td>\n",
" <td>90000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df1a\n",
" group\n",
"employee \n",
"Bob Accounting\n",
"Jake Engineering\n",
"Lisa Engineering\n",
"Sue HR\n",
"\n",
"df3\n",
" name salary\n",
"0 Bob 70000\n",
"1 Jake 80000\n",
"2 Lisa 120000\n",
"3 Sue 90000\n",
"\n",
"pd.merge(df1a, df3, left_index=True, right_on='name')\n",
" group name salary\n",
"0 Accounting Bob 70000\n",
"1 Engineering Jake 80000\n",
"2 Engineering Lisa 120000\n",
"3 HR Sue 90000"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"display('df1a', 'df3', \"pd.merge(df1a, df3, left_index=True, right_on='name')\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All of these options also work with multiple indices and/or multiple columns; the interface for this behavior is very intuitive.\n",
"For more information on this, see the [\"Merge, Join, and Concatenate\" section](http://pandas.pydata.org/pandas-docs/stable/merging.html) of the Pandas documentation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Specifying Set Arithmetic for Joins"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In all the preceding examples we have glossed over one important consideration in performing a join: the type of set arithmetic used in the join.\n",
"This comes up when a value appears in one key column but not the other. Consider this example:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df6</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>food</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Peter</td>\n",
" <td>fish</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Paul</td>\n",
" <td>beans</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Mary</td>\n",
" <td>bread</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df7</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>drink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Mary</td>\n",
" <td>wine</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joseph</td>\n",
" <td>beer</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df6, df7)</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>food</th>\n",
" <th>drink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Mary</td>\n",
" <td>bread</td>\n",
" <td>wine</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df6\n",
" name food\n",
"0 Peter fish\n",
"1 Paul beans\n",
"2 Mary bread\n",
"\n",
"df7\n",
" name drink\n",
"0 Mary wine\n",
"1 Joseph beer\n",
"\n",
"pd.merge(df6, df7)\n",
" name food drink\n",
"0 Mary bread wine"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df6 = pd.DataFrame({'name': ['Peter', 'Paul', 'Mary'],\n",
" 'food': ['fish', 'beans', 'bread']},\n",
" columns=['name', 'food'])\n",
"df7 = pd.DataFrame({'name': ['Mary', 'Joseph'],\n",
" 'drink': ['wine', 'beer']},\n",
" columns=['name', 'drink'])\n",
"display('df6', 'df7', 'pd.merge(df6, df7)')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we have merged two datasets that have only a single \"name\" entry in common: Mary.\n",
"By default, the result contains the *intersection* of the two sets of inputs; this is what is known as an *inner join*.\n",
"We can specify this explicitly using the `how` keyword, which defaults to `\"inner\"`:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>food</th>\n",
" <th>drink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Mary</td>\n",
" <td>bread</td>\n",
" <td>wine</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" name food drink\n",
"0 Mary bread wine"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.merge(df6, df7, how='inner')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other options for the `how` keyword are `'outer'`, `'left'`, and `'right'`.\n",
"An *outer join* returns a join over the union of the input columns, and fills in all missing values with NAs:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df6</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>food</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Peter</td>\n",
" <td>fish</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Paul</td>\n",
" <td>beans</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Mary</td>\n",
" <td>bread</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df7</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>drink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Mary</td>\n",
" <td>wine</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joseph</td>\n",
" <td>beer</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df6, df7, how='outer')</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>food</th>\n",
" <th>drink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Peter</td>\n",
" <td>fish</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Paul</td>\n",
" <td>beans</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Mary</td>\n",
" <td>bread</td>\n",
" <td>wine</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Joseph</td>\n",
" <td>NaN</td>\n",
" <td>beer</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df6\n",
" name food\n",
"0 Peter fish\n",
"1 Paul beans\n",
"2 Mary bread\n",
"\n",
"df7\n",
" name drink\n",
"0 Mary wine\n",
"1 Joseph beer\n",
"\n",
"pd.merge(df6, df7, how='outer')\n",
" name food drink\n",
"0 Peter fish NaN\n",
"1 Paul beans NaN\n",
"2 Mary bread wine\n",
"3 Joseph NaN beer"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"display('df6', 'df7', \"pd.merge(df6, df7, how='outer')\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The *left join* and *right join* return joins over the left entries and right entries, respectively.\n",
"For example:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df6</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>food</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Peter</td>\n",
" <td>fish</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Paul</td>\n",
" <td>beans</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Mary</td>\n",
" <td>bread</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df7</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>drink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Mary</td>\n",
" <td>wine</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Joseph</td>\n",
" <td>beer</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df6, df7, how='left')</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>food</th>\n",
" <th>drink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Peter</td>\n",
" <td>fish</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Paul</td>\n",
" <td>beans</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Mary</td>\n",
" <td>bread</td>\n",
" <td>wine</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df6\n",
" name food\n",
"0 Peter fish\n",
"1 Paul beans\n",
"2 Mary bread\n",
"\n",
"df7\n",
" name drink\n",
"0 Mary wine\n",
"1 Joseph beer\n",
"\n",
"pd.merge(df6, df7, how='left')\n",
" name food drink\n",
"0 Peter fish NaN\n",
"1 Paul beans NaN\n",
"2 Mary bread wine"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"display('df6', 'df7', \"pd.merge(df6, df7, how='left')\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The output rows now correspond to the entries in the left input. Using\n",
"`how='right'` works in a similar manner.\n",
"\n",
"All of these options can be applied straightforwardly to any of the preceding join types."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Overlapping Column Names: The suffixes Keyword"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Last, you may end up in a case where your two input ``DataFrame``s have conflicting column names.\n",
"Consider this example:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df8</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>rank</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>df9</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>rank</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pd.merge(df8, df9, on=\"name\")</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>rank_x</th>\n",
" <th>rank_y</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>1</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"df8\n",
" name rank\n",
"0 Bob 1\n",
"1 Jake 2\n",
"2 Lisa 3\n",
"3 Sue 4\n",
"\n",
"df9\n",
" name rank\n",
"0 Bob 3\n",
"1 Jake 1\n",
"2 Lisa 4\n",
"3 Sue 2\n",
"\n",
"pd.merge(df8, df9, on=\"name\")\n",
" name rank_x rank_y\n",
"0 Bob 1 3\n",
"1 Jake 2 1\n",
"2 Lisa 3 4\n",
"3 Sue 4 2"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df8 = pd.DataFrame({'name': ['Bob', 'Jake', 'Lisa', 'Sue'],\n",
" 'rank': [1, 2, 3, 4]})\n",
"df9 = pd.DataFrame({'name': ['Bob', 'Jake', 'Lisa', 'Sue'],\n",
" 'rank': [3, 1, 4, 2]})\n",
"display('df8', 'df9', 'pd.merge(df8, df9, on=\"name\")')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Because the output would have two conflicting column names, the `merge` function automatically appends the suffixes ``_x`` and ``_y`` to make the output columns unique.\n",
"If these defaults are inappropriate, it is possible to specify a custom suffix using the ``suffixes`` keyword:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>rank_L</th>\n",
" <th>rank_R</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Bob</td>\n",
" <td>1</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Jake</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lisa</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Sue</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" name rank_L rank_R\n",
"0 Bob 1 3\n",
"1 Jake 2 1\n",
"2 Lisa 3 4\n",
"3 Sue 4 2"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.merge(df8, df9, on=\"name\", suffixes=[\"_L\", \"_R\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These suffixes work in any of the possible join patterns, and also work if there are multiple overlapping columns."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For more information on these patterns, see [Aggregation and Grouping](03.08-Aggregation-and-Grouping.ipynb), where we dive a bit deeper into relational algebra.\n",
"Also see the [\"Merge, Join, Concatenate and Compare\" section](http://pandas.pydata.org/pandas-docs/stable/merging.html) of the Pandas documentation for further discussion of these topics."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example: US States Data\n",
"\n",
"Merge and join operations come up most often when combining data from different sources.\n",
"Here we will consider an example of some data about US states and their populations.\n",
"The data files can be found at [http://github.com/jakevdp/data-USstates](http://github.com/jakevdp/data-USstates):"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"# Following are commands to download the data\n",
"# repo = \"https://raw.githubusercontent.com/jakevdp/data-USstates/master\"\n",
"# !cd data && curl -O {repo}/state-population.csv\n",
"# !cd data && curl -O {repo}/state-areas.csv\n",
"# !cd data && curl -O {repo}/state-abbrevs.csv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take a look at the three datasets, using the Pandas ``read_csv`` function:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>pop.head()</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>state/region</th>\n",
" <th>ages</th>\n",
" <th>year</th>\n",
" <th>population</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2012</td>\n",
" <td>1117489.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2012</td>\n",
" <td>4817528.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2010</td>\n",
" <td>1130966.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>4785570.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2011</td>\n",
" <td>1125763.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>areas.head()</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>state</th>\n",
" <th>area (sq. mi)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Alabama</td>\n",
" <td>52423</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Alaska</td>\n",
" <td>656425</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Arizona</td>\n",
" <td>114006</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arkansas</td>\n",
" <td>53182</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>California</td>\n",
" <td>163707</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>\n",
"<div style=\"float: left; padding: 10px;\">\n",
" <p style='font-family:\"Courier New\", Courier, monospace'>abbrevs.head()</p><div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>state</th>\n",
" <th>abbreviation</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Alabama</td>\n",
" <td>AL</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Alaska</td>\n",
" <td>AK</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Arizona</td>\n",
" <td>AZ</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Arkansas</td>\n",
" <td>AR</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>California</td>\n",
" <td>CA</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" </div>"
],
"text/plain": [
"pop.head()\n",
" state/region ages year population\n",
"0 AL under18 2012 1117489.0\n",
"1 AL total 2012 4817528.0\n",
"2 AL under18 2010 1130966.0\n",
"3 AL total 2010 4785570.0\n",
"4 AL under18 2011 1125763.0\n",
"\n",
"areas.head()\n",
" state area (sq. mi)\n",
"0 Alabama 52423\n",
"1 Alaska 656425\n",
"2 Arizona 114006\n",
"3 Arkansas 53182\n",
"4 California 163707\n",
"\n",
"abbrevs.head()\n",
" state abbreviation\n",
"0 Alabama AL\n",
"1 Alaska AK\n",
"2 Arizona AZ\n",
"3 Arkansas AR\n",
"4 California CA"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pop = pd.read_csv('data/state-population.csv')\n",
"areas = pd.read_csv('data/state-areas.csv')\n",
"abbrevs = pd.read_csv('data/state-abbrevs.csv')\n",
"\n",
"display('pop.head()', 'areas.head()', 'abbrevs.head()')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given this information, say we want to compute a relatively straightforward result: rank US states and territories by their 2010 population density.\n",
"We clearly have the data here to find this result, but we'll have to combine the datasets to do so.\n",
"\n",
"We'll start with a many-to-one merge that will give us the full state names within the population `DataFrame`.\n",
"We want to merge based on the `state/region` column of `pop` and the `abbreviation` column of `abbrevs`.\n",
"We'll use `how='outer'` to make sure no data is thrown away due to mismatched labels:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>state/region</th>\n",
" <th>ages</th>\n",
" <th>year</th>\n",
" <th>population</th>\n",
" <th>state</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2012</td>\n",
" <td>1117489.0</td>\n",
" <td>Alabama</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2012</td>\n",
" <td>4817528.0</td>\n",
" <td>Alabama</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2010</td>\n",
" <td>1130966.0</td>\n",
" <td>Alabama</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>4785570.0</td>\n",
" <td>Alabama</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2011</td>\n",
" <td>1125763.0</td>\n",
" <td>Alabama</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" state/region ages year population state\n",
"0 AL under18 2012 1117489.0 Alabama\n",
"1 AL total 2012 4817528.0 Alabama\n",
"2 AL under18 2010 1130966.0 Alabama\n",
"3 AL total 2010 4785570.0 Alabama\n",
"4 AL under18 2011 1125763.0 Alabama"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"merged = pd.merge(pop, abbrevs, how='outer',\n",
" left_on='state/region', right_on='abbreviation')\n",
"merged = merged.drop('abbreviation', axis=1) # drop duplicate info\n",
"merged.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's double-check whether there were any mismatches here, which we can do by looking for rows with nulls:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"state/region False\n",
"ages False\n",
"year False\n",
"population True\n",
"state True\n",
"dtype: bool"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"merged.isnull().any()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some of the ``population`` values are null; let's figure out which these are!"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>state/region</th>\n",
" <th>ages</th>\n",
" <th>year</th>\n",
" <th>population</th>\n",
" <th>state</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2448</th>\n",
" <td>PR</td>\n",
" <td>under18</td>\n",
" <td>1990</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2449</th>\n",
" <td>PR</td>\n",
" <td>total</td>\n",
" <td>1990</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2450</th>\n",
" <td>PR</td>\n",
" <td>total</td>\n",
" <td>1991</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2451</th>\n",
" <td>PR</td>\n",
" <td>under18</td>\n",
" <td>1991</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2452</th>\n",
" <td>PR</td>\n",
" <td>total</td>\n",
" <td>1993</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" state/region ages year population state\n",
"2448 PR under18 1990 NaN NaN\n",
"2449 PR total 1990 NaN NaN\n",
"2450 PR total 1991 NaN NaN\n",
"2451 PR under18 1991 NaN NaN\n",
"2452 PR total 1993 NaN NaN"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"merged[merged['population'].isnull()].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It appears that all the null population values are from Puerto Rico prior to the year 2000; this is likely due to this data not being available in the original source.\n",
"\n",
"More importantly, we see that some of the new `state` entries are also null, which means that there was no corresponding entry in the `abbrevs` key!\n",
"Let's figure out which regions lack this match:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"array(['PR', 'USA'], dtype=object)"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"merged.loc[merged['state'].isnull(), 'state/region'].unique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can quickly infer the issue: our population data includes entries for Puerto Rico (PR) and the United States as a whole (USA), while these entries do not appear in the state abbreviation key.\n",
"We can fix these quickly by filling in appropriate entries:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"state/region False\n",
"ages False\n",
"year False\n",
"population True\n",
"state False\n",
"dtype: bool"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"merged.loc[merged['state/region'] == 'PR', 'state'] = 'Puerto Rico'\n",
"merged.loc[merged['state/region'] == 'USA', 'state'] = 'United States'\n",
"merged.isnull().any()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"No more nulls in the `state` column: we're all set!\n",
"\n",
"Now we can merge the result with the area data using a similar procedure.\n",
"Examining our results, we will want to join on the `state` column in both:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>state/region</th>\n",
" <th>ages</th>\n",
" <th>year</th>\n",
" <th>population</th>\n",
" <th>state</th>\n",
" <th>area (sq. mi)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2012</td>\n",
" <td>1117489.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2012</td>\n",
" <td>4817528.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2010</td>\n",
" <td>1130966.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>4785570.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2011</td>\n",
" <td>1125763.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" state/region ages year population state area (sq. mi)\n",
"0 AL under18 2012 1117489.0 Alabama 52423.0\n",
"1 AL total 2012 4817528.0 Alabama 52423.0\n",
"2 AL under18 2010 1130966.0 Alabama 52423.0\n",
"3 AL total 2010 4785570.0 Alabama 52423.0\n",
"4 AL under18 2011 1125763.0 Alabama 52423.0"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"final = pd.merge(merged, areas, on='state', how='left')\n",
"final.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Again, let's check for nulls to see if there were any mismatches:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"state/region False\n",
"ages False\n",
"year False\n",
"population True\n",
"state False\n",
"area (sq. mi) True\n",
"dtype: bool"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"final.isnull().any()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are nulls in the ``area`` column; we can take a look to see which regions were ignored here:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"array(['United States'], dtype=object)"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"final['state'][final['area (sq. mi)'].isnull()].unique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that our ``areas`` ``DataFrame`` does not contain the area of the United States as a whole.\n",
"We could insert the appropriate value (using the sum of all state areas, for instance), but in this case we'll just drop the null values because the population density of the entire United States is not relevant to our current discussion:"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>state/region</th>\n",
" <th>ages</th>\n",
" <th>year</th>\n",
" <th>population</th>\n",
" <th>state</th>\n",
" <th>area (sq. mi)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2012</td>\n",
" <td>1117489.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2012</td>\n",
" <td>4817528.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2010</td>\n",
" <td>1130966.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>4785570.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AL</td>\n",
" <td>under18</td>\n",
" <td>2011</td>\n",
" <td>1125763.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" state/region ages year population state area (sq. mi)\n",
"0 AL under18 2012 1117489.0 Alabama 52423.0\n",
"1 AL total 2012 4817528.0 Alabama 52423.0\n",
"2 AL under18 2010 1130966.0 Alabama 52423.0\n",
"3 AL total 2010 4785570.0 Alabama 52423.0\n",
"4 AL under18 2011 1125763.0 Alabama 52423.0"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"final.dropna(inplace=True)\n",
"final.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we have all the data we need. To answer the question of interest, let's first select the portion of the data corresponding with the year 2010, and the total population.\n",
"We'll use the `query` function to do this quickly (this requires the NumExpr package to be installed; see [High-Performance Pandas: `eval()` and `query()`](03.12-Performance-Eval-and-Query.ipynb)):"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>state/region</th>\n",
" <th>ages</th>\n",
" <th>year</th>\n",
" <th>population</th>\n",
" <th>state</th>\n",
" <th>area (sq. mi)</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AL</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>4785570.0</td>\n",
" <td>Alabama</td>\n",
" <td>52423.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>91</th>\n",
" <td>AK</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>713868.0</td>\n",
" <td>Alaska</td>\n",
" <td>656425.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>101</th>\n",
" <td>AZ</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>6408790.0</td>\n",
" <td>Arizona</td>\n",
" <td>114006.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>189</th>\n",
" <td>AR</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>2922280.0</td>\n",
" <td>Arkansas</td>\n",
" <td>53182.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>197</th>\n",
" <td>CA</td>\n",
" <td>total</td>\n",
" <td>2010</td>\n",
" <td>37333601.0</td>\n",
" <td>California</td>\n",
" <td>163707.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" state/region ages year population state area (sq. mi)\n",
"3 AL total 2010 4785570.0 Alabama 52423.0\n",
"91 AK total 2010 713868.0 Alaska 656425.0\n",
"101 AZ total 2010 6408790.0 Arizona 114006.0\n",
"189 AR total 2010 2922280.0 Arkansas 53182.0\n",
"197 CA total 2010 37333601.0 California 163707.0"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data2010 = final.query(\"year == 2010 & ages == 'total'\")\n",
"data2010.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's compute the population density and display it in order.\n",
"We'll start by re-indexing our data on the state, and then compute the result:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"data2010.set_index('state', inplace=True)\n",
"density = data2010['population'] / data2010['area (sq. mi)']"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"state\n",
"District of Columbia 8898.897059\n",
"Puerto Rico 1058.665149\n",
"New Jersey 1009.253268\n",
"Rhode Island 681.339159\n",
"Connecticut 645.600649\n",
"dtype: float64"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"density.sort_values(ascending=False, inplace=True)\n",
"density.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The result is a ranking of US states, plus Washington, DC, and Puerto Rico, in order of their 2010 population density, in residents per square mile.\n",
"We can see that by far the densest region in this dataset is Washington, DC (i.e., the District of Columbia); among states, the densest is New Jersey.\n",
"\n",
"We can also check the end of the list:"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"state\n",
"South Dakota 10.583512\n",
"North Dakota 9.537565\n",
"Montana 6.736171\n",
"Wyoming 5.768079\n",
"Alaska 1.087509\n",
"dtype: float64"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"density.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that the least dense state, by far, is Alaska, averaging slightly over one resident per square mile.\n",
"\n",
"This type of data merging is a common task when trying to answer questions using real-world data sources.\n",
"I hope that this example has given you an idea of some of the ways you can combine the tools we've covered in order to gain insight from your data!"
]
}
],
"metadata": {
"anaconda-cloud": {},
"jupytext": {
"formats": "ipynb,md"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.2"
}
},
"nbformat": 4,
"nbformat_minor": 4
}