{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Pandas\n",
    "\n",
    "**pandas** is a Python package providing fast, flexible, and expressive data structures designed to work with *relational* or *labeled* data both. It is a fundamental high-level building block for doing practical, real world data analysis in Python. \n",
    "\n",
    "pandas is well suited for:\n",
    "\n",
    "- Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet\n",
    "- Ordered and unordered (not necessarily fixed-frequency) time series data.\n",
    "- Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels\n",
    "- Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure\n",
    "\n",
    "\n",
    "Key features:\n",
    "    \n",
    "- Easy handling of **missing data**\n",
    "- **Size mutability**: columns can be inserted and deleted from DataFrame and higher dimensional objects\n",
    "- Automatic and explicit **data alignment**: objects can be explicitly aligned to a set of labels, or the data can be aligned automatically\n",
    "- Powerful, flexible **group by functionality** to perform split-apply-combine operations on data sets\n",
    "- Intelligent label-based **slicing, fancy indexing, and subsetting** of large data sets\n",
    "- Intuitive **merging and joining** data sets\n",
    "- Flexible **reshaping and pivoting** of data sets\n",
    "- **Hierarchical labeling** of axes\n",
    "- Robust **IO tools** for loading data from flat files, Excel files, databases, and HDF5\n",
    "- **Time series functionality**: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Data Structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series\n",
    "\n",
    "A **Series** is a single vector of data (like a NumPy array) with an *index* that labels each element in the vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     632\n",
       "1    1638\n",
       "2     569\n",
       "3     115\n",
       "dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts = pd.Series([632, 1638, 569, 115])\n",
    "counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If an index is not specified, a default sequence of integers is assigned as the index. A NumPy array comprises the values of the `Series`, while the index is a pandas `Index` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 632, 1638,  569,  115])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=4, step=1)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can assign meaningful labels to the index, if they are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Firmicutes         632\n",
       "Proteobacteria    1638\n",
       "Actinobacteria     569\n",
       "Bacteroidetes      115\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria = pd.Series([632, 1638, 569, 115], \n",
    "    index=['Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes'])\n",
    "\n",
    "bacteria"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These labels can be used to refer to the values in the `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "569"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria['Actinobacteria']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Proteobacteria    1638\n",
       "Actinobacteria     569\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria[[name.endswith('bacteria') for name in bacteria.index]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[False, True, True, False]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[name.endswith('bacteria') for name in bacteria.index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the indexing operation preserved the association between the values and the corresponding indices.\n",
    "\n",
    "We can still use positional indexing if we wish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "632"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can give both the array of values and the index meaningful labels themselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Firmicutes         632\n",
       "Proteobacteria    1638\n",
       "Actinobacteria     569\n",
       "Bacteroidetes      115\n",
       "Name: counts, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria.name = 'counts'\n",
    "bacteria.index.name = 'phylum'\n",
    "bacteria"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy's math functions and other operations can be applied to Series without losing the data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Firmicutes        6.448889\n",
       "Proteobacteria    7.401231\n",
       "Actinobacteria    6.343880\n",
       "Bacteroidetes     4.744932\n",
       "Name: counts, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.log(bacteria)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also filter according to the values in the `Series`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Proteobacteria    1638\n",
       "Name: counts, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria[bacteria>1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `Series` can be thought of as an ordered key-value store. In fact, we can create one from a `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Proteobacteria    1638\n",
       "Actinobacteria     569\n",
       "Firmicutes         632\n",
       "Bacteroidetes      115\n",
       "dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria_dict = {'Proteobacteria': 1638, 'Actinobacteria': 569,'Firmicutes': 632, \n",
    "                 'Bacteroidetes': 115}\n",
    "pd.Series(bacteria_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we pass a custom index to `Series`, it will select the corresponding values from the dict, and treat indices without corrsponding values as missing. Pandas uses the `NaN` (not a number) type for missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cyanobacteria        NaN\n",
       "Firmicutes         632.0\n",
       "Proteobacteria    1638.0\n",
       "Actinobacteria     569.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2 = pd.Series(bacteria_dict, \n",
    "                      index=['Cyanobacteria','Firmicutes',\n",
    "                             'Proteobacteria','Actinobacteria'])\n",
    "bacteria2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cyanobacteria      True\n",
       "Firmicutes        False\n",
       "Proteobacteria    False\n",
       "Actinobacteria    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2.isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Critically, the labels are used to **align data** when used in operations with other Series objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Actinobacteria    1138.0\n",
       "Bacteroidetes        NaN\n",
       "Cyanobacteria        NaN\n",
       "Firmicutes        1264.0\n",
       "Proteobacteria    3276.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria + bacteria2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrast this with NumPy arrays, where arrays of the same length will combine values element-wise; adding Series combined values with the same label in the resulting series. Notice also that the missing values were propogated by addition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrame\n",
    "\n",
    "Inevitably, we want to be able to store, view and manipulate data that is *multivariate*, where for every index there are multiple fields or columns of data, often of varying data type.\n",
    "\n",
    "A `DataFrame` is a tabular data structure, encapsulating multiple series like columns in a spreadsheet. Data are stored internally as a 2-dimensional object, but the `DataFrame` allows us to represent and manipulate higher-dimensional data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>value</th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>632</td>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1638</td>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>569</td>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>115</td>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>433</td>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1130</td>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>754</td>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>555</td>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   value  patient          phylum\n",
       "0    632        1      Firmicutes\n",
       "1   1638        1  Proteobacteria\n",
       "2    569        1  Actinobacteria\n",
       "3    115        1   Bacteroidetes\n",
       "4    433        2      Firmicutes\n",
       "5   1130        2  Proteobacteria\n",
       "6    754        2  Actinobacteria\n",
       "7    555        2   Bacteroidetes"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({'value':[632, 1638, 569, 115, 433, 1130, 754, 555],\n",
    "                     'patient':[1, 1, 1, 1, 2, 2, 2, 2],\n",
    "                     'phylum':['Firmicutes', 'Proteobacteria', 'Actinobacteria', \n",
    "    'Bacteroidetes', 'Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes']})\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can change the order by indexing them in the order we desire:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>patient</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1130</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           phylum  value  patient\n",
       "0      Firmicutes    632        1\n",
       "1  Proteobacteria   1638        1\n",
       "2  Actinobacteria    569        1\n",
       "3   Bacteroidetes    115        1\n",
       "4      Firmicutes    433        2\n",
       "5  Proteobacteria   1130        2\n",
       "6  Actinobacteria    754        2\n",
       "7   Bacteroidetes    555        2"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[['phylum','value','patient']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `DataFrame` has a second index, representing the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['value', 'patient', 'phylum'], dtype='object')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `dtypes` attribute reveals the data type for each column in our DataFrame. \n",
    "\n",
    "- `int64` is numeric integer values \n",
    "- `object` strings (letters and numbers)\n",
    "- `float64` floating-point values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "value       int64\n",
       "patient     int64\n",
       "phylum     object\n",
       "dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wish to access columns, we can do so either by dict-like indexing or by attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "3    1\n",
       "4    2\n",
       "5    2\n",
       "6    2\n",
       "7    2\n",
       "Name: patient, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['patient']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "3    1\n",
       "4    2\n",
       "5    2\n",
       "6    2\n",
       "7    2\n",
       "Name: patient, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.patient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   value\n",
       "0    632\n",
       "1   1638\n",
       "2    569\n",
       "3    115\n",
       "4    433\n",
       "5   1130\n",
       "6    754\n",
       "7    555"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[['value']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice this is different than with `Series`, where dict-like indexing retrieved a particular element (row). \n",
    "\n",
    "If we want access to a row in a `DataFrame`, we index its `loc` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "value                1638\n",
       "patient                 1\n",
       "phylum     Proteobacteria\n",
       "Name: 1, dtype: object"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Try out these commands to see what they return:\n",
    "\n",
    "- `data.head()`\n",
    "- `data.tail(3)`\n",
    "- `data.shape`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative way of initializing a `DataFrame` is with a list of dicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame([{'patient': 1, 'phylum': 'Firmicutes', 'value': 632},\n",
    "                    {'patient': 1, 'phylum': 'Proteobacteria', 'value': 1638},\n",
    "                    {'patient': 1, 'phylum': 'Actinobacteria', 'value': 569},\n",
    "                    {'patient': 1, 'phylum': 'Bacteroidetes', 'value': 115},\n",
    "                    {'patient': 2, 'phylum': 'Firmicutes', 'value': 433},\n",
    "                    {'patient': 2, 'phylum': 'Proteobacteria', 'value': 1130},\n",
    "                    {'patient': 2, 'phylum': 'Actinobacteria', 'value': 754},\n",
    "                    {'patient': 2, 'phylum': 'Bacteroidetes', 'value': 555}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes    115\n",
       "4        2      Firmicutes    433\n",
       "5        2  Proteobacteria   1130\n",
       "6        2  Actinobacteria    754\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its important to note that the Series returned when a DataFrame is indexed is merely a **view** on the DataFrame, and not a copy of the data itself. So you must be cautious when manipulating this data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     632\n",
       "1    1638\n",
       "2     569\n",
       "3     115\n",
       "4     433\n",
       "5    1130\n",
       "6     754\n",
       "7     555\n",
       "Name: value, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vals = data.value\n",
    "vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fonnesbeck/anaconda3/envs/dev/lib/python3.6/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0     632\n",
       "1    1638\n",
       "2     569\n",
       "3     115\n",
       "4     433\n",
       "5       0\n",
       "6     754\n",
       "7     555\n",
       "Name: value, dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vals[5] = 0\n",
    "vals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now look at the original `DataFrame`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes    115\n",
       "4        2      Firmicutes    433\n",
       "5        2  Proteobacteria      0\n",
       "6        2  Actinobacteria    754\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we plan on modifying an extracted Series, its a good idea to make a copy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes    115\n",
       "4        2      Firmicutes    433\n",
       "5        2  Proteobacteria      0\n",
       "6        2  Actinobacteria    754\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vals = data.value.copy()\n",
    "vals[5] = 1000\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create or modify columns by assignment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fonnesbeck/anaconda3/envs/dev/lib/python3.6/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value\n",
       "0        1      Firmicutes    632\n",
       "1        1  Proteobacteria   1638\n",
       "2        1  Actinobacteria    569\n",
       "3        1   Bacteroidetes     14\n",
       "4        2      Firmicutes     21\n",
       "5        2  Proteobacteria      0\n",
       "6        2  Actinobacteria      5\n",
       "7        2   Bacteroidetes    555"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.value[[3,4,6]] = [14, 21, 5]\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year\n",
       "0        1      Firmicutes    632  2013\n",
       "1        1  Proteobacteria   1638  2013\n",
       "2        1  Actinobacteria    569  2013\n",
       "3        1   Bacteroidetes     14  2013\n",
       "4        2      Firmicutes     21  2013\n",
       "5        2  Proteobacteria      0  2013\n",
       "6        2  Actinobacteria      5  2013\n",
       "7        2   Bacteroidetes    555  2013"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['year'] = 2013\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But note, we cannot use the attribute indexing method to add a new column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year\n",
       "0        1      Firmicutes    632  2013\n",
       "1        1  Proteobacteria   1638  2013\n",
       "2        1  Actinobacteria    569  2013\n",
       "3        1   Bacteroidetes     14  2013\n",
       "4        2      Firmicutes     21  2013\n",
       "5        2  Proteobacteria      0  2013\n",
       "6        2  Actinobacteria      5  2013\n",
       "7        2   Bacteroidetes    555  2013"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.treatment = 1\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.treatment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "From the `data` table above, create an index to return all rows for which the phylum name ends in \"bacteria\" and the value is greater than 1000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year\n",
       "1        1  Proteobacteria   1638  2013"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data.phylum.str.endswith('bacteria') & (data.value > 1000)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specifying a `Series` as a new columns cause its values to be added according to the `DataFrame`'s index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    0\n",
       "2    0\n",
       "3    0\n",
       "4    1\n",
       "5    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "treatment = pd.Series([0]*4 + [1]*2)\n",
    "treatment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year  treatment\n",
       "0        1      Firmicutes    632  2013        0.0\n",
       "1        1  Proteobacteria   1638  2013        0.0\n",
       "2        1  Actinobacteria    569  2013        0.0\n",
       "3        1   Bacteroidetes     14  2013        0.0\n",
       "4        2      Firmicutes     21  2013        1.0\n",
       "5        2  Proteobacteria      0  2013        1.0\n",
       "6        2  Actinobacteria      5  2013        NaN\n",
       "7        2   Bacteroidetes    555  2013        NaN"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['treatment'] = treatment\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other Python data structures (ones without an index) need to be the same length as the `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Length of values does not match length of index",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-39-1fbc7bd65338>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mmonth\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Jan'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Feb'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Mar'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Apr'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'month'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmonth\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/dev/lib/python3.6/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m__setitem__\u001b[0;34m(self, key, value)\u001b[0m\n\u001b[1;32m   3117\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3118\u001b[0m             \u001b[0;31m# set column\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3119\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_set_item\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3120\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3121\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_setitem_slice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/dev/lib/python3.6/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m_set_item\u001b[0;34m(self, key, value)\u001b[0m\n\u001b[1;32m   3192\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3193\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ensure_valid_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3194\u001b[0;31m         \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sanitize_column\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3195\u001b[0m         \u001b[0mNDFrame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_set_item\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3196\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/dev/lib/python3.6/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m_sanitize_column\u001b[0;34m(self, key, value, broadcast)\u001b[0m\n\u001b[1;32m   3389\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3390\u001b[0m             \u001b[0;31m# turn me into an ndarray\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3391\u001b[0;31m             \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_sanitize_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3392\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndarray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mIndex\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3393\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/dev/lib/python3.6/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m_sanitize_index\u001b[0;34m(data, index, copy)\u001b[0m\n\u001b[1;32m   3999\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4000\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4001\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Length of values does not match length of '\u001b[0m \u001b[0;34m'index'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4002\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4003\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mABCIndexClass\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Length of values does not match length of index"
     ]
    }
   ],
   "source": [
    "month = ['Jan', 'Feb', 'Mar', 'Apr']\n",
    "data['month'] = month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jan</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year  treatment month\n",
       "0        1      Firmicutes    632  2013        0.0   Jan\n",
       "1        1  Proteobacteria   1638  2013        0.0   Jan\n",
       "2        1  Actinobacteria    569  2013        0.0   Jan\n",
       "3        1   Bacteroidetes     14  2013        0.0   Jan\n",
       "4        2      Firmicutes     21  2013        1.0   Jan\n",
       "5        2  Proteobacteria      0  2013        1.0   Jan\n",
       "6        2  Actinobacteria      5  2013        NaN   Jan\n",
       "7        2   Bacteroidetes    555  2013        NaN   Jan"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['month'] = ['Jan']*len(data)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `drop` method to remove rows or columns, which by default drops rows. We can be explicit by using the `axis` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.drop('month', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>patient</th>\n",
       "      <th>phylum</th>\n",
       "      <th>value</th>\n",
       "      <th>year</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>632</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1638</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>569</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>21</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>0</td>\n",
       "      <td>2013</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>555</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   patient          phylum  value  year  treatment\n",
       "0        1      Firmicutes    632  2013        0.0\n",
       "1        1  Proteobacteria   1638  2013        0.0\n",
       "2        1  Actinobacteria    569  2013        0.0\n",
       "3        1   Bacteroidetes     14  2013        0.0\n",
       "4        2      Firmicutes     21  2013        1.0\n",
       "5        2  Proteobacteria      0  2013        1.0\n",
       "6        2  Actinobacteria      5  2013        NaN\n",
       "7        2   Bacteroidetes    555  2013        NaN"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can extract the underlying data as a simple `ndarray` by accessing the `values` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 'Firmicutes', 632, 2013, 0.0],\n",
       "       [1, 'Proteobacteria', 1638, 2013, 0.0],\n",
       "       [1, 'Actinobacteria', 569, 2013, 0.0],\n",
       "       [1, 'Bacteroidetes', 14, 2013, 0.0],\n",
       "       [2, 'Firmicutes', 21, 2013, 1.0],\n",
       "       [2, 'Proteobacteria', 0, 2013, 1.0],\n",
       "       [2, 'Actinobacteria', 5, 2013, nan],\n",
       "       [2, 'Bacteroidetes', 555, 2013, nan]], dtype=object)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that because of the mix of string and integer (and `NaN`) values, the dtype of the array is `object`. The dtype will automatically be chosen to be as general as needed to accomodate all the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1. ,  0.4],\n",
       "       [ 2. , -1. ],\n",
       "       [ 3. ,  4.5]])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'foo': [1,2,3], 'bar':[0.4, -1.0, 4.5]})\n",
    "df.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas uses a custom data structure to represent the indices of Series and DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=8, step=1)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Index objects are immutable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "Index does not support mutable operations",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-46-90b10fa0e9b5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m15\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/dev/lib/python3.6/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36m__setitem__\u001b[0;34m(self, key, value)\u001b[0m\n\u001b[1;32m   2063\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2064\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__setitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2065\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Index does not support mutable operations\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2066\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2067\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: Index does not support mutable operations"
     ]
    }
   ],
   "source": [
    "data.index[0] = 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is so that Index objects can be shared between data structures without fear that they will be changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "bacteria2.index = bacteria.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phylum\n",
       "Firmicutes           NaN\n",
       "Proteobacteria     632.0\n",
       "Actinobacteria    1638.0\n",
       "Bacteroidetes      569.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bacteria2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A key, but often under-appreciated, step in data analysis is importing the data that we wish to analyze. Though it is easy to load basic data structures into Python using built-in tools or those provided by packages like NumPy, it is non-trivial to import structured data well, and to easily convert this input into a robust data structure:\n",
    "\n",
    "    genes = np.loadtxt(\"genes.csv\", delimiter=\",\", dtype=[('gene', '|S10'), ('value', '<f4')])\n",
    "\n",
    "Pandas provides a convenient set of functions for importing tabular data in a number of formats directly into a `DataFrame` object. These functions include a slew of options to perform type inference, indexing, parsing, iterating and cleaning automatically as data are imported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with some more bacteria data, stored in csv format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taxon,Patient,Group,Tissue,Stool\n",
      "Firmicutes,1,0,136,4182\n",
      "Firmicutes,2,1,1174,703\n",
      "Firmicutes,3,0,408,3946\n",
      "Firmicutes,4,1,831,8605\n",
      "Firmicutes,5,0,693,50\n",
      "Firmicutes,6,1,718,717\n",
      "Firmicutes,7,0,173,33\n",
      "Firmicutes,8,1,228,80\n",
      "Firmicutes,9,0,162,3196\n",
      "Firmicutes,10,1,372,32\n",
      "Firmicutes,11,0,4255,4361\n",
      "Firmicutes,12,1,107,1667\n",
      "Firmicutes,13,0,96,223\n",
      "Firmicutes,14,1,281,2377\n",
      "Proteobacteria,1,0,2469,1821\n",
      "Proteobacteria,2,1,839,661\n",
      "Proteobacteria,3,0,4414,18\n",
      "Proteobacteria,4,1,12044,83\n",
      "Proteobacteria,5,0,2310,12\n",
      "Proteobacteria,6,1,3053,547\n",
      "Proteobacteria,7,0,395,2174\n",
      "Proteobacteria,8,1,2651,767\n",
      "Proteobacteria,9,0,1195,76\n",
      "Proteobacteria,10,1,6857,795\n",
      "Proteobacteria,11,0,483,666\n",
      "Proteobacteria,12,1,2950,3994\n",
      "Proteobacteria,13,0,1541,816\n",
      "Proteobacteria,14,1,1307,53\n",
      "Actinobacteria,1,0,1590,4\n",
      "Actinobacteria,2,1,25,2\n",
      "Actinobacteria,3,0,259,300\n",
      "Actinobacteria,4,1,568,7\n",
      "Actinobacteria,5,0,1102,9\n",
      "Actinobacteria,6,1,678,377\n",
      "Actinobacteria,7,0,260,58\n",
      "Actinobacteria,8,1,424,233\n",
      "Actinobacteria,9,0,548,21\n",
      "Actinobacteria,10,1,201,83\n",
      "Actinobacteria,11,0,42,75\n",
      "Actinobacteria,12,1,109,59\n",
      "Actinobacteria,13,0,51,183\n",
      "Actinobacteria,14,1,310,204\n",
      "Bacteroidetes,1,0,67,0\n",
      "Bacteroidetes,2,1,0,0\n",
      "Bacteroidetes,3,0,85,5\n",
      "Bacteroidetes,4,1,143,7\n",
      "Bacteroidetes,5,0,678,2\n",
      "Bacteroidetes,6,1,4829,209\n",
      "Bacteroidetes,7,0,74,651\n",
      "Bacteroidetes,8,1,169,254\n",
      "Bacteroidetes,9,0,106,10\n",
      "Bacteroidetes,10,1,73,381\n",
      "Bacteroidetes,11,0,30,359\n",
      "Bacteroidetes,12,1,51,51\n",
      "Bacteroidetes,13,0,2473,2314\n",
      "Bacteroidetes,14,1,102,33\n",
      "Other,1,0,195,18\n",
      "Other,2,1,42,2\n",
      "Other,3,0,316,43\n",
      "Other,4,1,202,40\n",
      "Other,5,0,116,0\n",
      "Other,6,1,527,12\n",
      "Other,7,0,357,11\n",
      "Other,8,1,106,11\n",
      "Other,9,0,67,14\n",
      "Other,10,1,203,6\n",
      "Other,11,0,392,6\n",
      "Other,12,1,28,25\n",
      "Other,13,0,12,22\n",
      "Other,14,1,305,32"
     ]
    }
   ],
   "source": [
    "!cat ../data/microbiome.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This table can be read into a DataFrame using `read_csv`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>693</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>718</td>\n",
       "      <td>717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>173</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>228</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>162</td>\n",
       "      <td>3196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>372</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>4255</td>\n",
       "      <td>4361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>107</td>\n",
       "      <td>1667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>96</td>\n",
       "      <td>223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>281</td>\n",
       "      <td>2377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2469</td>\n",
       "      <td>1821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>839</td>\n",
       "      <td>661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4414</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>12044</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2310</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>3053</td>\n",
       "      <td>547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>395</td>\n",
       "      <td>2174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>2651</td>\n",
       "      <td>767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>1195</td>\n",
       "      <td>76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>6857</td>\n",
       "      <td>795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>483</td>\n",
       "      <td>666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>2950</td>\n",
       "      <td>3994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>1541</td>\n",
       "      <td>816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>1307</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1590</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>51</td>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Actinobacteria</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>310</td>\n",
       "      <td>204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>67</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>85</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>143</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>678</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>4829</td>\n",
       "      <td>209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>74</td>\n",
       "      <td>651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>169</td>\n",
       "      <td>254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>106</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>73</td>\n",
       "      <td>381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2473</td>\n",
       "      <td>2314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Bacteroidetes</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>102</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>Other</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>195</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>Other</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>42</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>Other</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>316</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>Other</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>202</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>Other</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>116</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>Other</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>527</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>Other</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>357</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>Other</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>106</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Other</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>67</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>Other</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>203</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>Other</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>392</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>Other</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>28</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>Other</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>Other</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>305</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>70 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Taxon  Patient  Group  Tissue  Stool\n",
       "0       Firmicutes        1      0     136   4182\n",
       "1       Firmicutes        2      1    1174    703\n",
       "2       Firmicutes        3      0     408   3946\n",
       "3       Firmicutes        4      1     831   8605\n",
       "4       Firmicutes        5      0     693     50\n",
       "5       Firmicutes        6      1     718    717\n",
       "6       Firmicutes        7      0     173     33\n",
       "7       Firmicutes        8      1     228     80\n",
       "8       Firmicutes        9      0     162   3196\n",
       "9       Firmicutes       10      1     372     32\n",
       "10      Firmicutes       11      0    4255   4361\n",
       "11      Firmicutes       12      1     107   1667\n",
       "12      Firmicutes       13      0      96    223\n",
       "13      Firmicutes       14      1     281   2377\n",
       "14  Proteobacteria        1      0    2469   1821\n",
       "15  Proteobacteria        2      1     839    661\n",
       "16  Proteobacteria        3      0    4414     18\n",
       "17  Proteobacteria        4      1   12044     83\n",
       "18  Proteobacteria        5      0    2310     12\n",
       "19  Proteobacteria        6      1    3053    547\n",
       "20  Proteobacteria        7      0     395   2174\n",
       "21  Proteobacteria        8      1    2651    767\n",
       "22  Proteobacteria        9      0    1195     76\n",
       "23  Proteobacteria       10      1    6857    795\n",
       "24  Proteobacteria       11      0     483    666\n",
       "25  Proteobacteria       12      1    2950   3994\n",
       "26  Proteobacteria       13      0    1541    816\n",
       "27  Proteobacteria       14      1    1307     53\n",
       "28  Actinobacteria        1      0    1590      4\n",
       "29  Actinobacteria        2      1      25      2\n",
       "..             ...      ...    ...     ...    ...\n",
       "40  Actinobacteria       13      0      51    183\n",
       "41  Actinobacteria       14      1     310    204\n",
       "42   Bacteroidetes        1      0      67      0\n",
       "43   Bacteroidetes        2      1       0      0\n",
       "44   Bacteroidetes        3      0      85      5\n",
       "45   Bacteroidetes        4      1     143      7\n",
       "46   Bacteroidetes        5      0     678      2\n",
       "47   Bacteroidetes        6      1    4829    209\n",
       "48   Bacteroidetes        7      0      74    651\n",
       "49   Bacteroidetes        8      1     169    254\n",
       "50   Bacteroidetes        9      0     106     10\n",
       "51   Bacteroidetes       10      1      73    381\n",
       "52   Bacteroidetes       11      0      30    359\n",
       "53   Bacteroidetes       12      1      51     51\n",
       "54   Bacteroidetes       13      0    2473   2314\n",
       "55   Bacteroidetes       14      1     102     33\n",
       "56           Other        1      0     195     18\n",
       "57           Other        2      1      42      2\n",
       "58           Other        3      0     316     43\n",
       "59           Other        4      1     202     40\n",
       "60           Other        5      0     116      0\n",
       "61           Other        6      1     527     12\n",
       "62           Other        7      0     357     11\n",
       "63           Other        8      1     106     11\n",
       "64           Other        9      0      67     14\n",
       "65           Other       10      1     203      6\n",
       "66           Other       11      0     392      6\n",
       "67           Other       12      1      28     25\n",
       "68           Other       13      0      12     22\n",
       "69           Other       14      1     305     32\n",
       "\n",
       "[70 rows x 5 columns]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb = pd.read_csv(\"../data/microbiome.csv\")\n",
    "mb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `read_csv` automatically considered the first row in the file to be a header row.\n",
    "\n",
    "We can override default behavior by customizing some the arguments, like `header`, `names` or `index_col`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Taxon</td>\n",
       "      <td>Patient</td>\n",
       "      <td>Group</td>\n",
       "      <td>Tissue</td>\n",
       "      <td>Stool</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0        1      2       3      4\n",
       "0       Taxon  Patient  Group  Tissue  Stool\n",
       "1  Firmicutes        1      0     136   4182\n",
       "2  Firmicutes        2      1    1174    703\n",
       "3  Firmicutes        3      0     408   3946\n",
       "4  Firmicutes        4      1     831   8605"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"../data/microbiome.csv\", header=None).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`read_csv` is just a convenience function for `read_table`, since csv is such a common format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "mb = pd.read_table(\"../data/microbiome.csv\", sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sep` argument can be customized as needed to accomodate arbitrary separators. For example, we can use a regular expression to define a variable amount of whitespace, which is unfortunately very common in some data formats: \n",
    "    \n",
    "    sep='\\s+'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a more useful index, we can specify the first two columns, which together provide a unique index to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Patient</th>\n",
       "      <th>Taxon</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>Firmicutes</th>\n",
       "      <td>0</td>\n",
       "      <td>693</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Group  Tissue  Stool\n",
       "Patient Taxon                           \n",
       "1       Firmicutes      0     136   4182\n",
       "2       Firmicutes      1    1174    703\n",
       "3       Firmicutes      0     408   3946\n",
       "4       Firmicutes      1     831   8605\n",
       "5       Firmicutes      0     693     50"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb = pd.read_csv(\"../data/microbiome.csv\", index_col=['Patient','Taxon'])\n",
    "mb.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is called a *hierarchical* index, which we will revisit later in the section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we have sections of data that we do not wish to import (for example, known bad data), we can populate the `skiprows` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>693</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>173</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>228</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Taxon  Patient  Group  Tissue  Stool\n",
       "0  Firmicutes        1      0     136   4182\n",
       "1  Firmicutes        2      1    1174    703\n",
       "2  Firmicutes        5      0     693     50\n",
       "3  Firmicutes        7      0     173     33\n",
       "4  Firmicutes        8      1     228     80"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"../data/microbiome.csv\", skiprows=[3,4,6]).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we only want to import a small number of rows from, say, a very large data file we can use `nrows`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Group</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>4182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1174</td>\n",
       "      <td>703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>408</td>\n",
       "      <td>3946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>831</td>\n",
       "      <td>8605</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Taxon  Patient  Group  Tissue  Stool\n",
       "0  Firmicutes        1      0     136   4182\n",
       "1  Firmicutes        2      1    1174    703\n",
       "2  Firmicutes        3      0     408   3946\n",
       "3  Firmicutes        4      1     831   8605"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"../data/microbiome.csv\", nrows=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternately, if we want to process our data in reasonable chunks, the `chunksize` argument will return an iterable object that can be employed in a data processing loop. For example, our microbiome data are organized by bacterial phylum, with 15 patients represented in each:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Firmicutes         688.142857\n",
       "Proteobacteria    3036.285714\n",
       "Actinobacteria     440.500000\n",
       "Bacteroidetes      634.285714\n",
       "Other              204.857143\n",
       "dtype: float64"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_chunks = pd.read_csv(\"../data/microbiome.csv\", chunksize=14)\n",
    "\n",
    "mean_tissue = pd.Series({chunk.iloc[0].Taxon:chunk.Tissue.mean() for chunk in data_chunks})\n",
    "    \n",
    "mean_tissue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most real-world data is incomplete, with values missing due to incomplete observation, data entry or transcription error, or other reasons. Pandas will automatically recognize and parse common missing data indicators, including `NA` and `NULL`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taxon,Patient,Tissue,Stool\n",
      "Firmicutes,1,632,305\n",
      "Firmicutes,2,136,4182\n",
      "Firmicutes,3,,703\n",
      "Firmicutes,4,408,3946\n",
      "Firmicutes,5,831,8605\n",
      "Firmicutes,6,693,50\n",
      "Firmicutes,7,718,717\n",
      "Firmicutes,8,173,33\n",
      "Firmicutes,9,228,NA\n",
      "Firmicutes,10,162,3196\n",
      "Firmicutes,11,372,-99999\n",
      "Firmicutes,12,4255,4361\n",
      "Firmicutes,13,107,1667\n",
      "Firmicutes,14,?,223\n",
      "Firmicutes,15,281,2377\n",
      "Proteobacteria,1,1638,3886\n",
      "Proteobacteria,2,2469,1821\n",
      "Proteobacteria,3,839,661\n",
      "Proteobacteria,4,4414,18\n",
      "Proteobacteria,5,12044,83\n",
      "Proteobacteria,6,2310,12\n",
      "Proteobacteria,7,3053,547\n",
      "Proteobacteria,8,395,2174\n",
      "Proteobacteria,9,2651,767\n",
      "Proteobacteria,10,1195,76\n",
      "Proteobacteria,11,6857,795\n",
      "Proteobacteria,12,483,666\n",
      "Proteobacteria,13,2950,3994\n",
      "Proteobacteria,14,1541,816\n",
      "Proteobacteria,15,1307,53\n",
      "Actinobacteria,1,569,648\n",
      "Actinobacteria,2,1590,4\n",
      "Actinobacteria,3,25,2\n",
      "Actinobacteria,4,259,300\n",
      "Actinobacteria,5,568,7\n",
      "Actinobacteria,6,1102,9\n",
      "Actinobacteria,7,678,377\n",
      "Actinobacteria,8,260,58\n",
      "Actinobacteria,9,424,233\n",
      "Actinobacteria,10,548,21\n",
      "Actinobacteria,11,201,83\n",
      "Actinobacteria,12,42,75\n",
      "Actinobacteria,13,109,59\n",
      "Actinobacteria,14,51,183\n",
      "Actinobacteria,15,310,204\n",
      "Bacteroidetes,1,115,380\n",
      "Bacteroidetes,2,67,0\n",
      "Bacteroidetes,3,0,0\n",
      "Bacteroidetes,4,85,5\n",
      "Bacteroidetes,5,143,7\n",
      "Bacteroidetes,6,678,2\n",
      "Bacteroidetes,7,4829,209\n",
      "Bacteroidetes,8,74,651\n",
      "Bacteroidetes,9,169,254\n",
      "Bacteroidetes,10,106,10\n",
      "Bacteroidetes,11,73,381\n",
      "Bacteroidetes,12,30,359\n",
      "Bacteroidetes,13,51,51\n",
      "Bacteroidetes,14,2473,2314\n",
      "Bacteroidetes,15,102,33\n",
      "Other,1,114,277\n",
      "Other,2,195,18\n",
      "Other,3,42,2\n",
      "Other,4,316,43\n",
      "Other,5,202,40\n",
      "Other,6,116,0\n",
      "Other,7,527,12\n",
      "Other,8,357,11\n",
      "Other,9,106,11\n",
      "Other,10,67,14\n",
      "Other,11,203,6\n",
      "Other,12,392,6\n",
      "Other,13,28,25\n",
      "Other,14,12,22\n",
      "Other,15,305,32"
     ]
    }
   ],
   "source": [
    "!cat ../data/microbiome_missing.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>632</td>\n",
       "      <td>305.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>136</td>\n",
       "      <td>4182.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>703.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>408</td>\n",
       "      <td>3946.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>5</td>\n",
       "      <td>831</td>\n",
       "      <td>8605.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>6</td>\n",
       "      <td>693</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>7</td>\n",
       "      <td>718</td>\n",
       "      <td>717.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>8</td>\n",
       "      <td>173</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>9</td>\n",
       "      <td>228</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>10</td>\n",
       "      <td>162</td>\n",
       "      <td>3196.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>11</td>\n",
       "      <td>372</td>\n",
       "      <td>-99999.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>12</td>\n",
       "      <td>4255</td>\n",
       "      <td>4361.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>13</td>\n",
       "      <td>107</td>\n",
       "      <td>1667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>14</td>\n",
       "      <td>?</td>\n",
       "      <td>223.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>15</td>\n",
       "      <td>281</td>\n",
       "      <td>2377.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1</td>\n",
       "      <td>1638</td>\n",
       "      <td>3886.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>2</td>\n",
       "      <td>2469</td>\n",
       "      <td>1821.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>3</td>\n",
       "      <td>839</td>\n",
       "      <td>661.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>4</td>\n",
       "      <td>4414</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>12044</td>\n",
       "      <td>83.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Taxon  Patient Tissue    Stool\n",
       "0       Firmicutes        1    632    305.0\n",
       "1       Firmicutes        2    136   4182.0\n",
       "2       Firmicutes        3    NaN    703.0\n",
       "3       Firmicutes        4    408   3946.0\n",
       "4       Firmicutes        5    831   8605.0\n",
       "5       Firmicutes        6    693     50.0\n",
       "6       Firmicutes        7    718    717.0\n",
       "7       Firmicutes        8    173     33.0\n",
       "8       Firmicutes        9    228      NaN\n",
       "9       Firmicutes       10    162   3196.0\n",
       "10      Firmicutes       11    372 -99999.0\n",
       "11      Firmicutes       12   4255   4361.0\n",
       "12      Firmicutes       13    107   1667.0\n",
       "13      Firmicutes       14      ?    223.0\n",
       "14      Firmicutes       15    281   2377.0\n",
       "15  Proteobacteria        1   1638   3886.0\n",
       "16  Proteobacteria        2   2469   1821.0\n",
       "17  Proteobacteria        3    839    661.0\n",
       "18  Proteobacteria        4   4414     18.0\n",
       "19  Proteobacteria        5  12044     83.0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"../data/microbiome_missing.csv\").head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, Pandas recognized `NA` and an empty field as missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Taxon  Patient  Tissue  Stool\n",
       "0   False    False   False  False\n",
       "1   False    False   False  False\n",
       "2   False    False    True  False\n",
       "3   False    False   False  False\n",
       "4   False    False   False  False\n",
       "5   False    False   False  False\n",
       "6   False    False   False  False\n",
       "7   False    False   False  False\n",
       "8   False    False   False   True\n",
       "9   False    False   False  False\n",
       "10  False    False   False  False\n",
       "11  False    False   False  False\n",
       "12  False    False   False  False\n",
       "13  False    False   False  False\n",
       "14  False    False   False  False\n",
       "15  False    False   False  False\n",
       "16  False    False   False  False\n",
       "17  False    False   False  False\n",
       "18  False    False   False  False\n",
       "19  False    False   False  False"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.isnull(pd.read_csv(\"../data/microbiome_missing.csv\")).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, there will sometimes be inconsistency with the conventions for missing data. In this example, there is a question mark \"?\" and a large negative number where there should have been a positive integer. We can specify additional symbols with the `na_values` argument:\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Taxon</th>\n",
       "      <th>Patient</th>\n",
       "      <th>Tissue</th>\n",
       "      <th>Stool</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>1</td>\n",
       "      <td>632.0</td>\n",
       "      <td>305.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>2</td>\n",
       "      <td>136.0</td>\n",
       "      <td>4182.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>703.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>4</td>\n",
       "      <td>408.0</td>\n",
       "      <td>3946.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>5</td>\n",
       "      <td>831.0</td>\n",
       "      <td>8605.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>6</td>\n",
       "      <td>693.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>7</td>\n",
       "      <td>718.0</td>\n",
       "      <td>717.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>8</td>\n",
       "      <td>173.0</td>\n",
       "      <td>33.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>9</td>\n",
       "      <td>228.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>10</td>\n",
       "      <td>162.0</td>\n",
       "      <td>3196.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>11</td>\n",
       "      <td>372.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>12</td>\n",
       "      <td>4255.0</td>\n",
       "      <td>4361.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>13</td>\n",
       "      <td>107.0</td>\n",
       "      <td>1667.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "      <td>223.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Firmicutes</td>\n",
       "      <td>15</td>\n",
       "      <td>281.0</td>\n",
       "      <td>2377.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>1</td>\n",
       "      <td>1638.0</td>\n",
       "      <td>3886.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>2</td>\n",
       "      <td>2469.0</td>\n",
       "      <td>1821.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>3</td>\n",
       "      <td>839.0</td>\n",
       "      <td>661.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>4</td>\n",
       "      <td>4414.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Proteobacteria</td>\n",
       "      <td>5</td>\n",
       "      <td>12044.0</td>\n",
       "      <td>83.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Taxon  Patient   Tissue   Stool\n",
       "0       Firmicutes        1    632.0   305.0\n",
       "1       Firmicutes        2    136.0  4182.0\n",
       "2       Firmicutes        3      NaN   703.0\n",
       "3       Firmicutes        4    408.0  3946.0\n",
       "4       Firmicutes        5    831.0  8605.0\n",
       "5       Firmicutes        6    693.0    50.0\n",
       "6       Firmicutes        7    718.0   717.0\n",
       "7       Firmicutes        8    173.0    33.0\n",
       "8       Firmicutes        9    228.0     NaN\n",
       "9       Firmicutes       10    162.0  3196.0\n",
       "10      Firmicutes       11    372.0     NaN\n",
       "11      Firmicutes       12   4255.0  4361.0\n",
       "12      Firmicutes       13    107.0  1667.0\n",
       "13      Firmicutes       14      NaN   223.0\n",
       "14      Firmicutes       15    281.0  2377.0\n",
       "15  Proteobacteria        1   1638.0  3886.0\n",
       "16  Proteobacteria        2   2469.0  1821.0\n",
       "17  Proteobacteria        3    839.0   661.0\n",
       "18  Proteobacteria        4   4414.0    18.0\n",
       "19  Proteobacteria        5  12044.0    83.0"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\"../data/microbiome_missing.csv\", na_values=['?', -99999]).head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These can be specified on a column-wise basis using an appropriate dict as the argument for `na_values`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Microsoft Excel\n",
    "\n",
    "Since so much financial and scientific data ends up in Excel spreadsheets (regrettably), Pandas' ability to directly import Excel spreadsheets is valuable. This support is contingent on having one or two dependencies (depending on what version of Excel file is being imported) installed: `xlrd` and `openpyxl` (these may be installed with either `pip` or `easy_install`).\n",
    "\n",
    "The read_excel convenience function in pandas imports a specific sheet from an Excel file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fonnesbeck/anaconda3/envs/dev/lib/python3.6/site-packages/pandas/io/excel.py:329: FutureWarning: The `sheetname` keyword is deprecated, use `sheet_name` instead\n",
      "  **kwds)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Archaea \"Crenarchaeota\" Thermoprotei Acidiloba...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Archaea \"Crenarchaeota\" Thermoprotei Acidiloba...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Archaea \"Crenarchaeota\" Thermoprotei Desulfuro...</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Archaea \"Crenarchaeota\" Thermoprotei Desulfuro...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Archaea \"Crenarchaeota\" Thermoprotei Desulfuro...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   0   1\n",
       "0  Archaea \"Crenarchaeota\" Thermoprotei Acidiloba...   2\n",
       "1  Archaea \"Crenarchaeota\" Thermoprotei Acidiloba...  14\n",
       "2  Archaea \"Crenarchaeota\" Thermoprotei Desulfuro...  23\n",
       "3  Archaea \"Crenarchaeota\" Thermoprotei Desulfuro...   1\n",
       "4  Archaea \"Crenarchaeota\" Thermoprotei Desulfuro...   2"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mb = pd.read_excel('../data/microbiome/MID2.xls', sheetname='Sheet 1', header=None)\n",
    "mb.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several other data formats that can be imported into Python and converted into DataFrames, with the help of buitl-in or third-party libraries. These include JSON, XML, HDF5, relational and non-relational databases, and various web APIs. These are beyond the scope of this tutorial, but are covered in [Python for Data Analysis](http://shop.oreilly.com/product/0636920023784.do)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## References\n",
    "\n",
    "[Python for Data Analysis](http://shop.oreilly.com/product/0636920023784.do) Wes McKinney"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "latex_envs": {
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": false,
   "threshold": "3",
   "toc_cell": true,
   "toc_section_display": "none",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}