{ "cells": [ { "cell_type": "markdown", "id": "4e8388ea-3018-4b42-87e5-8bb26323414e", "metadata": {}, "source": [ "# GGIS 4381/6321 - Lab 2 - Data Wrangling and Pandas\n", "\n", "**Authors:** Alexander Michels\n", "\n", "This notebook is a quick crash course on how to perform basic data wrangling tasks using the [Pandas](https://pandas.pydata.org/docs/user_guide/index.html) package." ] }, { "cell_type": "markdown", "id": "5c6fc1b0-a5da-4a67-8a36-8f0785826053", "metadata": {}, "source": [ "
\n", "\n", "## Intro to Pandas\n", "\n", "[Pandas](https://pandas.pydata.org/docs/user_guide/index.html) is a key package for working with tabular (table-like) data in Python. There is also a geospatial version which we will explore in the next lab.\n", "\n", "\n", "Pandas introduces two key classes (Python objects) for working with data:\n", "\n", "* **Series:** a list (or *array*) of data of some type.\n", "* **DataFrame:** a table-like data struction with rows and columns. The data within a DataFrame is sorted as Series." ] }, { "cell_type": "code", "execution_count": 2, "id": "4e92ef5b-db99-4711-8c16-aea19966728a", "metadata": {}, "outputs": [], "source": [ "import pandas as pd # import and save it as an abbreviation" ] }, { "cell_type": "code", "execution_count": 3, "id": "75f03b9c-1376-4b9e-a898-72fd0b4f33ef", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 1\n", "1 2\n", "2 3\n", "3 4\n", "4 5\n", "dtype: int64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s = pd.Series([1, 2, 3, 4, 5]) # create a basic series\n", "s # print out the series" ] }, { "cell_type": "code", "execution_count": 4, "id": "4a4784e6-94ce-4f3d-80b4-0a29b2eddcca", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s[0] # you can *mostly* treat a series like a list, for example accessing by index" ] }, { "cell_type": "markdown", "id": "64dfd473-f405-433b-b4bd-db61d167b32f", "metadata": {}, "source": [ "There are a LOT of ways to create a DataFrame by passing in data and a key benefit of using a well-established open-source package like this is that [there is plenty of documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html).\n", "\n", "One way to do this is to create a *dictionary* of the column labels and data you want to include. For example:" ] }, { "cell_type": "code", "execution_count": 5, "id": "7437aa1b-93be-4d88-aa63-cede2eb8aa95", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'City': ['New York', 'Dallas', 'Los Angeles'],\n", " 'State': ['New York', 'Texas', 'California'],\n", " '# of UTDs': [0, 1, 0]}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = { # the brackets make a \"dictionary\" that saves key and value pairs\n", " 'City': [\"New York\", \"Dallas\", \"Los Angeles\"],\n", " 'State': [\"New York\", \"Texas\", \"California\"],\n", " '# of UTDs': [0, 1, 0]\n", "}\n", "\n", "data" ] }, { "cell_type": "markdown", "id": "84c7115a-be94-4c83-a833-9407e7f65fd0", "metadata": {}, "source": [ "We can access particular parts of the dictionary by giving the key (e.g., 'City')." ] }, { "cell_type": "code", "execution_count": 6, "id": "6c875514-2df1-4762-b3aa-0a0ee9450613", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['New York', 'Dallas', 'Los Angeles']" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[\"City\"]" ] }, { "cell_type": "markdown", "id": "c311e85f-e6d9-4af4-946f-8d483d7d0c7e", "metadata": {}, "source": [ "We can also use pandas to transform the data into a DataFrame (basically a table) using the \"DataFrame\" function from Pandas (which we saved as `pd`):" ] }, { "cell_type": "code", "execution_count": 7, "id": "358e2e88-54c2-4a1a-bfd5-908ff9e50f84", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CityState# of UTDs
0New YorkNew York0
1DallasTexas1
2Los AngelesCalifornia0
\n", "
" ], "text/plain": [ " City State # of UTDs\n", "0 New York New York 0\n", "1 Dallas Texas 1\n", "2 Los Angeles California 0" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame(data)\n", "df" ] }, { "cell_type": "markdown", "id": "80e44af4-534a-43ed-9759-4d2a00a21ac0", "metadata": {}, "source": [ "You'll notice that the DataFrame labeled each row with a number. This is called the `index` which is kind of the like the ID of the row. It allows us to refer to individual rows using the [`iloc`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iloc.html#pandas.DataFrame.iloc) function." ] }, { "cell_type": "code", "execution_count": 8, "id": "09c5934c-ea4d-4ca8-b32e-126717bb0980", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "City Dallas\n", "State Texas\n", "# of UTDs 1\n", "Name: 1, dtype: object" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(type(df.iloc[1])) # this returns a Series\n", "df.iloc[1]" ] }, { "cell_type": "markdown", "id": "fea0a684-f4f5-49d9-ada7-5499f1eef011", "metadata": {}, "source": [ "We can similarly refer to columns using their name within brackets, like so:" ] }, { "cell_type": "code", "execution_count": 9, "id": "4bbbe419-10b8-458e-a3d4-ab2835596019", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 New York\n", "1 Texas\n", "2 California\n", "Name: State, dtype: object" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"State\"]" ] }, { "cell_type": "markdown", "id": "d994ad2b-0606-40ba-89b1-c63becb88d4f", "metadata": {}, "source": [ "If you want to view the whole DataFrame, you can just print it, but it is often more useful to only print the first few rows (especially for LONG dataframes). We can do this by calling the `head` function.\n", "\n", "Here, we use `.()` syntax because the `head` function is a *method* of the DataFrame object and not just a generic Python function. You sometimes need to keep track of:\n", "\n", "* What are generic Python functions like `len` (that get called by `len()`\n", "* What are functions from certain packages like `pd.DataFrame` (that get called by using the package name, dot, function name)\n", "* What are *methods* that come from certain *Objects* (that get called by using `.`)." ] }, { "cell_type": "code", "execution_count": 10, "id": "b3c52523-5610-4a52-a769-309d65292fb9", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CityState# of UTDs
0New YorkNew York0
1DallasTexas1
2Los AngelesCalifornia0
\n", "
" ], "text/plain": [ " City State # of UTDs\n", "0 New York New York 0\n", "1 Dallas Texas 1\n", "2 Los Angeles California 0" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "markdown", "id": "918cec44-cd6b-4fe1-ab77-7fede6b444b4", "metadata": {}, "source": [ "You can also view how long a DataFrame is using `len` just like with lists." ] }, { "cell_type": "code", "execution_count": 11, "id": "5a817702-1d3c-44fe-b463-1c78214cfab9", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(df) # we can get the length using `len` just like with lists!" ] }, { "cell_type": "markdown", "id": "390f9de5-a16e-4575-9c4d-22b6096f6df0", "metadata": {}, "source": [ "For more information on Pandas, check out the [10 minutes to pandas](https://pandas.pydata.org/docs/user_guide/10min.html) guide.\n", "\n", "
\n", "\n", "## Data Wrangling Basics\n", "\n", "To work with real-life data, we can also load SpreadSheet data like Comma-Separated Values (CSV) files. If you look in the File Browser (right) for this folder, you should see a \"TEXAS_COUNTY.csv\" file. This file is the Texas county-level [Social Vulnerability Index (SVI)](https://www.atsdr.cdc.gov/place-health/php/svi/svi-data-documentation-download.html) data for 2022, calculated by the Centers for Disease Control (CDC). This data is used to assess how vulnerable communities are to environmental hazards like natural disasters or pandemics.\n", "\n", "We can load the data using the pandas `read_csv` function. Note that this is a *pandas* function, so we need to use the following syntax:" ] }, { "cell_type": "code", "execution_count": 12, "id": "2e6d14cc-da5f-4509-a7a4-e49c1ea4baa7", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
STSTATEST_ABBRSTCNTYCOUNTYFIPSLOCATIONAREA_SQMIE_TOTPOPM_TOTPOP...EP_ASIANMP_ASIANEP_AIANMP_AIANEP_NHPIMP_NHPIEP_TWOMOREMP_TWOMOREEP_OTHERRACEMP_OTHERRACE
048TexasTX48001Anderson County48001Anderson County, Texas1062.616344580770...0.70.20.70.50.10.14.71.00.20.2
148TexasTX48003Andrews County48003Andrews County, Texas1500.720658183620...0.50.20.00.30.00.30.80.70.10.1
248TexasTX48005Angelina County48005Angelina County, Texas797.864652866080...0.90.10.40.40.00.13.60.90.20.2
348TexasTX48007Aransas County48007Aransas County, Texas252.068582240480...1.71.30.50.30.00.22.21.10.00.2
448TexasTX48009Archer County48009Archer County, Texas903.28879286490...0.20.20.40.20.00.63.10.60.00.1
\n", "

5 rows × 158 columns

\n", "
" ], "text/plain": [ " ST STATE ST_ABBR STCNTY COUNTY FIPS LOCATION \n", "0 48 Texas TX 48001 Anderson County 48001 Anderson County, Texas \\\n", "1 48 Texas TX 48003 Andrews County 48003 Andrews County, Texas \n", "2 48 Texas TX 48005 Angelina County 48005 Angelina County, Texas \n", "3 48 Texas TX 48007 Aransas County 48007 Aransas County, Texas \n", "4 48 Texas TX 48009 Archer County 48009 Archer County, Texas \n", "\n", " AREA_SQMI E_TOTPOP M_TOTPOP ... EP_ASIAN MP_ASIAN EP_AIAN MP_AIAN \n", "0 1062.616344 58077 0 ... 0.7 0.2 0.7 0.5 \\\n", "1 1500.720658 18362 0 ... 0.5 0.2 0.0 0.3 \n", "2 797.864652 86608 0 ... 0.9 0.1 0.4 0.4 \n", "3 252.068582 24048 0 ... 1.7 1.3 0.5 0.3 \n", "4 903.288792 8649 0 ... 0.2 0.2 0.4 0.2 \n", "\n", " EP_NHPI MP_NHPI EP_TWOMORE MP_TWOMORE EP_OTHERRACE MP_OTHERRACE \n", "0 0.1 0.1 4.7 1.0 0.2 0.2 \n", "1 0.0 0.3 0.8 0.7 0.1 0.1 \n", "2 0.0 0.1 3.6 0.9 0.2 0.2 \n", "3 0.0 0.2 2.2 1.1 0.0 0.2 \n", "4 0.0 0.6 3.1 0.6 0.0 0.1 \n", "\n", "[5 rows x 158 columns]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "svi = pd.read_csv(\"TEXAS_COUNTY.csv\") # load the CSV and save it using the svi variable\n", "svi.head() # let's see what we are working with!" ] }, { "cell_type": "markdown", "id": "ded6b75c-826b-47af-93d3-93fddaa4f8a2", "metadata": {}, "source": [ "### Quick Overviews\n", "\n", "That's a lot of data and the ellipses (`...`) in the middle means we aren't seeing it all! Let's explore a few ways to get more info quickly:" ] }, { "cell_type": "code", "execution_count": 13, "id": "6d19c011-9544-494f-b52b-b4c987ca974a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['ST', 'STATE', 'ST_ABBR', 'STCNTY', 'COUNTY', 'FIPS', 'LOCATION', 'AREA_SQMI', 'E_TOTPOP', 'M_TOTPOP', 'E_HU', 'M_HU', 'E_HH', 'M_HH', 'E_POV150', 'M_POV150', 'E_UNEMP', 'M_UNEMP', 'E_HBURD', 'M_HBURD', 'E_NOHSDP', 'M_NOHSDP', 'E_UNINSUR', 'M_UNINSUR', 'E_AGE65', 'M_AGE65', 'E_AGE17', 'M_AGE17', 'E_DISABL', 'M_DISABL', 'E_SNGPNT', 'M_SNGPNT', 'E_LIMENG', 'M_LIMENG', 'E_MINRTY', 'M_MINRTY', 'E_MUNIT', 'M_MUNIT', 'E_MOBILE', 'M_MOBILE', 'E_CROWD', 'M_CROWD', 'E_NOVEH', 'M_NOVEH', 'E_GROUPQ', 'M_GROUPQ', 'EP_POV150', 'MP_POV150', 'EP_UNEMP', 'MP_UNEMP', 'EP_HBURD', 'MP_HBURD', 'EP_NOHSDP', 'MP_NOHSDP', 'EP_UNINSUR', 'MP_UNINSUR', 'EP_AGE65', 'MP_AGE65', 'EP_AGE17', 'MP_AGE17', 'EP_DISABL', 'MP_DISABL', 'EP_SNGPNT', 'MP_SNGPNT', 'EP_LIMENG', 'MP_LIMENG', 'EP_MINRTY', 'MP_MINRTY', 'EP_MUNIT', 'MP_MUNIT', 'EP_MOBILE', 'MP_MOBILE', 'EP_CROWD', 'MP_CROWD', 'EP_NOVEH', 'MP_NOVEH', 'EP_GROUPQ', 'MP_GROUPQ', 'EPL_POV150', 'EPL_UNEMP', 'EPL_HBURD', 'EPL_NOHSDP', 'EPL_UNINSUR', 'SPL_THEME1', 'RPL_THEME1', 'EPL_AGE65', 'EPL_AGE17', 'EPL_DISABL', 'EPL_SNGPNT', 'EPL_LIMENG', 'SPL_THEME2', 'RPL_THEME2', 'EPL_MINRTY', 'SPL_THEME3', 'RPL_THEME3', 'EPL_MUNIT', 'EPL_MOBILE', 'EPL_CROWD', 'EPL_NOVEH', 'EPL_GROUPQ', 'SPL_THEME4', 'RPL_THEME4', 'SPL_THEMES', 'RPL_THEMES', 'F_POV150', 'F_UNEMP', 'F_HBURD', 'F_NOHSDP', 'F_UNINSUR', 'F_THEME1', 'F_AGE65', 'F_AGE17', 'F_DISABL', 'F_SNGPNT', 'F_LIMENG', 'F_THEME2', 'F_MINRTY', 'F_THEME3', 'F_MUNIT', 'F_MOBILE', 'F_CROWD', 'F_NOVEH', 'F_GROUPQ', 'F_THEME4', 'F_TOTAL', 'E_DAYPOP', 'E_NOINT', 'M_NOINT', 'E_AFAM', 'M_AFAM', 'E_HISP', 'M_HISP', 'E_ASIAN', 'M_ASIAN', 'E_AIAN', 'M_AIAN', 'E_NHPI', 'M_NHPI', 'E_TWOMORE', 'M_TWOMORE', 'E_OTHERRACE', 'M_OTHERRACE', 'EP_NOINT', 'MP_NOINT', 'EP_AFAM', 'MP_AFAM', 'EP_HISP', 'MP_HISP', 'EP_ASIAN', 'MP_ASIAN', 'EP_AIAN', 'MP_AIAN', 'EP_NHPI', 'MP_NHPI', 'EP_TWOMORE', 'MP_TWOMORE', 'EP_OTHERRACE', 'MP_OTHERRACE']\n" ] } ], "source": [ "print(list(svi.columns)) # we can print the column as a list to get all of them!" ] }, { "cell_type": "code", "execution_count": 14, "id": "ed1478d1-bd60-4a45-b97c-c1ca549b27fe", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 254 entries, 0 to 253\n", "Columns: 158 entries, ST to MP_OTHERRACE\n", "dtypes: float64(75), int64(79), object(4)\n", "memory usage: 313.7+ KB\n" ] } ], "source": [ "svi.info()" ] }, { "cell_type": "markdown", "id": "7bcc5f3e-9924-4512-a03a-c73f03078aba", "metadata": {}, "source": [ "Another fun functionality is the `describe` method that provides quick statistics on the columns. \n", "\n", "If you aren't entirely sure of what that outputs mean, remember you always refer to the [documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.describe.html). Another option is Google or even AI, but it is always better to verify anything you get from random sites or AI just in case. " ] }, { "cell_type": "code", "execution_count": 15, "id": "ab5414b6-481c-4922-a92a-2adb810391d4", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
STSTCNTYFIPSAREA_SQMIE_TOTPOPM_TOTPOPE_HUM_HUE_HHM_HH...EP_ASIANMP_ASIANEP_AIANMP_AIANEP_NHPIMP_NHPIEP_TWOMOREMP_TWOMOREEP_OTHERRACEMP_OTHERRACE
count254.0254.00000254.00000254.0000002.540000e+02254.0000002.540000e+02254.0000002.540000e+02254.000000...254.000000254.000000254.000000254.000000254.000000254.000000254.000000254.000000254.000000254.000000
mean48.048254.0000048254.000001028.6206551.151313e+0517.4566934.588571e+0470.6771654.130139e+04380.976378...1.1283460.7885830.2531500.8098430.0673231.0519692.3488191.2562990.1811021.048031
std0.0146.93536146.93536658.5650404.106693e+0555.6354961.609053e+0565.8286051.481685e+05435.239745...2.1415671.7348060.4718672.8506050.1786953.5235371.7162503.2389460.2602013.513145
min48.048001.0000048001.00000127.2087059.600000e+010.0000005.500000e+0111.0000003.800000e+0125.000000...0.0000000.1000000.0000000.1000000.0000000.1000000.0000000.1000000.0000000.100000
25%48.048127.5000048127.50000835.6803196.305500e+030.0000002.798000e+0327.2500002.094000e+03149.250000...0.2000000.2000000.1000000.1000000.0000000.1000001.4000000.5000000.0000000.100000
50%48.048254.0000048254.00000908.6837071.838300e+040.0000008.375000e+0350.0000006.594500e+03259.000000...0.6000000.2000000.1000000.2000000.0000000.3000002.2000000.8000000.1000000.300000
75%48.048380.5000048380.500001043.4677045.242425e+040.0000002.136875e+0487.7500001.801875e+04442.750000...1.1000000.6000000.3000000.4750000.1000000.7000003.0750001.1000000.2750000.700000
max48.048507.0000048507.000006183.7605004.726177e+06328.0000001.851489e+06485.0000001.692896e+064042.000000...21.10000020.5000006.10000037.5000001.70000037.50000017.60000037.5000001.70000037.500000
\n", "

8 rows × 154 columns

\n", "
" ], "text/plain": [ " ST STCNTY FIPS AREA_SQMI E_TOTPOP M_TOTPOP \n", "count 254.0 254.00000 254.00000 254.000000 2.540000e+02 254.000000 \\\n", "mean 48.0 48254.00000 48254.00000 1028.620655 1.151313e+05 17.456693 \n", "std 0.0 146.93536 146.93536 658.565040 4.106693e+05 55.635496 \n", "min 48.0 48001.00000 48001.00000 127.208705 9.600000e+01 0.000000 \n", "25% 48.0 48127.50000 48127.50000 835.680319 6.305500e+03 0.000000 \n", "50% 48.0 48254.00000 48254.00000 908.683707 1.838300e+04 0.000000 \n", "75% 48.0 48380.50000 48380.50000 1043.467704 5.242425e+04 0.000000 \n", "max 48.0 48507.00000 48507.00000 6183.760500 4.726177e+06 328.000000 \n", "\n", " E_HU M_HU E_HH M_HH ... EP_ASIAN \n", "count 2.540000e+02 254.000000 2.540000e+02 254.000000 ... 254.000000 \\\n", "mean 4.588571e+04 70.677165 4.130139e+04 380.976378 ... 1.128346 \n", "std 1.609053e+05 65.828605 1.481685e+05 435.239745 ... 2.141567 \n", "min 5.500000e+01 11.000000 3.800000e+01 25.000000 ... 0.000000 \n", "25% 2.798000e+03 27.250000 2.094000e+03 149.250000 ... 0.200000 \n", "50% 8.375000e+03 50.000000 6.594500e+03 259.000000 ... 0.600000 \n", "75% 2.136875e+04 87.750000 1.801875e+04 442.750000 ... 1.100000 \n", "max 1.851489e+06 485.000000 1.692896e+06 4042.000000 ... 21.100000 \n", "\n", " MP_ASIAN EP_AIAN MP_AIAN EP_NHPI MP_NHPI EP_TWOMORE \n", "count 254.000000 254.000000 254.000000 254.000000 254.000000 254.000000 \\\n", "mean 0.788583 0.253150 0.809843 0.067323 1.051969 2.348819 \n", "std 1.734806 0.471867 2.850605 0.178695 3.523537 1.716250 \n", "min 0.100000 0.000000 0.100000 0.000000 0.100000 0.000000 \n", "25% 0.200000 0.100000 0.100000 0.000000 0.100000 1.400000 \n", "50% 0.200000 0.100000 0.200000 0.000000 0.300000 2.200000 \n", "75% 0.600000 0.300000 0.475000 0.100000 0.700000 3.075000 \n", "max 20.500000 6.100000 37.500000 1.700000 37.500000 17.600000 \n", "\n", " MP_TWOMORE EP_OTHERRACE MP_OTHERRACE \n", "count 254.000000 254.000000 254.000000 \n", "mean 1.256299 0.181102 1.048031 \n", "std 3.238946 0.260201 3.513145 \n", "min 0.100000 0.000000 0.100000 \n", "25% 0.500000 0.000000 0.100000 \n", "50% 0.800000 0.100000 0.300000 \n", "75% 1.100000 0.275000 0.700000 \n", "max 37.500000 1.700000 37.500000 \n", "\n", "[8 rows x 154 columns]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "svi.describe()" ] }, { "cell_type": "markdown", "id": "cbb9a256-caab-4d03-a70b-19de79432fbc", "metadata": {}, "source": [ "Pandas also has some basic visualization functionalities built-in including histograms:" ] }, { "cell_type": "code", "execution_count": 16, "id": "d251e1ac-9378-4299-bb27-3aed030afecc", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGvCAYAAAC9yRSTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAlQklEQVR4nO3de3BU9f3/8dc2ITHEZEsSkmWHEFJNrJpoMVgkooJAMCAWsBVrVcDo6AiUNGRUtKPYMgSkBNphoNjSBLwQSwteCipRIBSpU4hGLraIihAkaQTj5iJsIDm/P/ix810CCJsNZ/nwfMycGffs2bPv3WHMc86ePeuwLMsSAACAob5n9wAAAACdidgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYLRwuwcIBW1tbTpw4IBiYmLkcDjsHgcAAJwFy7LU2Ngot9ut733v9MdviB1JBw4cUHJyst1jAACAAFRXV6tnz56nvd/W2CkqKtLKlSv13//+V1FRUcrOztbs2bN1xRVX+LYZP368li5d6ve4fv366f333/fd9nq9Kiws1PLly3X48GENHjxYCxcuPOML/79iYmIkHX+zYmNjg/DKAABAZ2toaFBycrLv7/jp2Bo7FRUVmjhxoq6//nodO3ZMTz31lHJycvTxxx8rOjrat91tt92mkpIS3+2IiAi//eTn5+uNN95QWVmZ4uPjNXXqVN1+++2qrKxUWFjYd85x4qOr2NhYYgcAgAvMd52CYmvsvPXWW363S0pKlJiYqMrKSt18882+9ZGRkXK5XKfch8fj0ZIlS/TCCy9oyJAhkqQXX3xRycnJeueddzRs2LDOewEAACDkhdS3sTwejyQpLi7Ob/2GDRuUmJio9PR0PfTQQ6qrq/PdV1lZqaNHjyonJ8e3zu12KyMjQ5s3bz7l83i9XjU0NPgtAADATCETO5ZlqaCgQAMGDFBGRoZvfW5url566SWtW7dOc+fO1ZYtW3TrrbfK6/VKkmpraxUREaFu3br57S8pKUm1tbWnfK6ioiI5nU7fwsnJAACYK2S+jTVp0iRt27ZNmzZt8ls/duxY339nZGSob9++SklJ0erVqzVmzJjT7s+yrNN+hjdt2jQVFBT4bp84wQkAAJgnJI7sTJ48Wa+//rrWr1//nd+g6tGjh1JSUrR7925JksvlUktLi+rr6/22q6urU1JS0in3ERkZ6TsZmZOSAQAwm62xY1mWJk2apJUrV2rdunVKTU39zsccOnRI1dXV6tGjhyQpKytLXbp0UXl5uW+bmpoa7dixQ9nZ2Z02OwAAuDDY+jHWxIkT9fLLL+u1115TTEyM7xwbp9OpqKgoNTU1afr06brzzjvVo0cPffHFF3ryySeVkJCg0aNH+7bNy8vT1KlTFR8fr7i4OBUWFiozM9P37SwAAHDxsjV2Fi1aJEkaOHCg3/qSkhKNHz9eYWFh2r59u5YtW6ZvvvlGPXr00KBBg/TKK6/4XUBo3rx5Cg8P11133eW7qGBpaelZXWMHAACYzWFZlmX3EHZraGiQ0+mUx+Ph/B0AAC4QZ/v3OyROUAYAAOgsxA4AADAasQMAAIxG7AAAAKMROwAAwGgh83MRpur9xGq7RwjIF7NG2D0CAABBwZEdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNFsjZ2ioiJdf/31iomJUWJiokaNGqVdu3b5bWNZlqZPny63262oqCgNHDhQO3fu9NvG6/Vq8uTJSkhIUHR0tO644w7t37//fL4UAAAQomyNnYqKCk2cOFHvv/++ysvLdezYMeXk5Ki5udm3zXPPPafi4mItWLBAW7Zskcvl0tChQ9XY2OjbJj8/X6tWrVJZWZk2bdqkpqYm3X777WptbbXjZQEAgBDisCzLsnuIE7766islJiaqoqJCN998syzLktvtVn5+vh5//HFJx4/iJCUlafbs2Xr44Yfl8XjUvXt3vfDCCxo7dqwk6cCBA0pOTtaaNWs0bNiw73zehoYGOZ1OeTwexcbGBvU19X5idVD3d758MWuE3SMAAHBGZ/v3O6TO2fF4PJKkuLg4SdKePXtUW1urnJwc3zaRkZG65ZZbtHnzZklSZWWljh496reN2+1WRkaGb5uTeb1eNTQ0+C0AAMBMIRM7lmWpoKBAAwYMUEZGhiSptrZWkpSUlOS3bVJSku++2tpaRUREqFu3bqfd5mRFRUVyOp2+JTk5OdgvBwAAhIiQiZ1JkyZp27ZtWr58ebv7HA6H323LstqtO9mZtpk2bZo8Ho9vqa6uDnxwAAAQ0kIidiZPnqzXX39d69evV8+ePX3rXS6XJLU7QlNXV+c72uNyudTS0qL6+vrTbnOyyMhIxcbG+i0AAMBMtsaOZVmaNGmSVq5cqXXr1ik1NdXv/tTUVLlcLpWXl/vWtbS0qKKiQtnZ2ZKkrKwsdenSxW+bmpoa7dixw7cNAAC4eIXb+eQTJ07Uyy+/rNdee00xMTG+IzhOp1NRUVFyOBzKz8/XzJkzlZaWprS0NM2cOVNdu3bVPffc49s2Ly9PU6dOVXx8vOLi4lRYWKjMzEwNGTLEzpcHAABCgK2xs2jRIknSwIED/daXlJRo/PjxkqTHHntMhw8f1qOPPqr6+nr169dPa9euVUxMjG/7efPmKTw8XHfddZcOHz6swYMHq7S0VGFhYefrpQAAgBAVUtfZsQvX2WmP6+wAAELdBXmdHQAAgGAjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNFsjZ2NGzdq5MiRcrvdcjgcevXVV/3uHz9+vBwOh99yww03+G3j9Xo1efJkJSQkKDo6WnfccYf2799/Hl8FAAAIZbbGTnNzs6699lotWLDgtNvcdtttqqmp8S1r1qzxuz8/P1+rVq1SWVmZNm3apKamJt1+++1qbW3t7PEBAMAFINzOJ8/NzVVubu4Zt4mMjJTL5TrlfR6PR0uWLNELL7ygIUOGSJJefPFFJScn65133tGwYcOCPjMAALiwhPw5Oxs2bFBiYqLS09P10EMPqa6uzndfZWWljh49qpycHN86t9utjIwMbd68+bT79Hq9amho8FsAAICZQjp2cnNz9dJLL2ndunWaO3eutmzZoltvvVVer1eSVFtbq4iICHXr1s3vcUlJSaqtrT3tfouKiuR0On1LcnJyp74OAABgn4BiZ8+ePcGe45TGjh2rESNGKCMjQyNHjtSbb76pTz75RKtXrz7j4yzLksPhOO3906ZNk8fj8S3V1dXBHh0AAISIgGLn8ssv16BBg/Tiiy/qyJEjwZ7ptHr06KGUlBTt3r1bkuRyudTS0qL6+nq/7erq6pSUlHTa/URGRio2NtZvAQAAZgoodj766CP16dNHU6dOlcvl0sMPP6x///vfwZ6tnUOHDqm6ulo9evSQJGVlZalLly4qLy/3bVNTU6MdO3YoOzu70+cBAAChL6DYycjIUHFxsb788kuVlJSotrZWAwYM0NVXX63i4mJ99dVXZ7WfpqYmVVVVqaqqStLxj8eqqqq0b98+NTU1qbCwUP/617/0xRdfaMOGDRo5cqQSEhI0evRoSZLT6VReXp6mTp2qd999Vx9++KHuvfdeZWZm+r6dBQAALm4dOkE5PDxco0eP1l//+lfNnj1bn332mQoLC9WzZ0/df//9qqmpOePjt27dqj59+qhPnz6SpIKCAvXp00dPP/20wsLCtH37dv3kJz9Renq6xo0bp/T0dP3rX/9STEyMbx/z5s3TqFGjdNddd+nGG29U165d9cYbbygsLKwjLw0AABjCYVmWFeiDt27dqr/85S8qKytTdHS0xo0bp7y8PB04cEBPP/20Ghsbz8vHWx3V0NAgp9Mpj8cT9PN3ej9x5pOpQ9UXs0bYPQIAAGd0tn+/A7qoYHFxsUpKSrRr1y4NHz5cy5Yt0/Dhw/W97x0/UJSamqrFixfrhz/8YWDTAwAABElAsbNo0SI98MADmjBhwmmvbtyrVy8tWbKkQ8MBAAB0VECxc+Kr32cSERGhcePGBbJ7AACAoAnoBOWSkhKtWLGi3foVK1Zo6dKlHR4KAAAgWAKKnVmzZikhIaHd+sTERM2cObPDQwEAAARLQLGzd+9epaamtlufkpKiffv2dXgoAACAYAkodhITE7Vt27Z26z/66CPFx8d3eCgAAIBgCSh27r77bv3yl7/U+vXr1draqtbWVq1bt05TpkzR3XffHewZAQAAAhbQt7FmzJihvXv3avDgwQoPP76LtrY23X///ZyzAwAAQkpAsRMREaFXXnlFv/3tb/XRRx8pKipKmZmZSklJCfZ8AAAAHRJQ7JyQnp6u9PT0YM0CAAAQdAHFTmtrq0pLS/Xuu++qrq5ObW1tfvevW7cuKMMBAAB0VECxM2XKFJWWlmrEiBHKyMiQw+EI9lwAAABBEVDslJWV6a9//auGDx8e7HkAAACCKqCvnkdEROjyyy8P9iwAAABBF1DsTJ06Vb///e9lWVaw5wEAAAiqgD7G2rRpk9avX68333xTV199tbp06eJ3/8qVK4MyHAAAQEcFFDvf//73NXr06GDPAgAAEHQBxU5JSUmw5wAAAOgUAZ2zI0nHjh3TO++8o8WLF6uxsVGSdODAATU1NQVtOAAAgI4K6MjO3r17ddttt2nfvn3yer0aOnSoYmJi9Nxzz+nIkSP64x//GOw5AQAAAhLQkZ0pU6aob9++qq+vV1RUlG/96NGj9e677wZtOAAAgI4K+NtY7733niIiIvzWp6Sk6MsvvwzKYAAAAMEQ0JGdtrY2tba2tlu/f/9+xcTEdHgoAACAYAkodoYOHar58+f7bjscDjU1NemZZ57hJyQAAEBICehjrHnz5mnQoEG66qqrdOTIEd1zzz3avXu3EhIStHz58mDPCAAAELCAYsftdquqqkrLly/XBx98oLa2NuXl5ekXv/iF3wnLAAAAdgsodiQpKipKDzzwgB544IFgzgMAABBUAcXOsmXLznj//fffH9AwAAAAwRZQ7EyZMsXv9tGjR/Xtt98qIiJCXbt2JXYAAEDICOjbWPX19X5LU1OTdu3apQEDBnCCMgAACCkB/zbWydLS0jRr1qx2R30AAADsFLTYkaSwsDAdOHAgmLsEAADokIDO2Xn99df9bluWpZqaGi1YsEA33nhjUAYDAAAIhoBiZ9SoUX63HQ6HunfvrltvvVVz584NxlwAAABBEVDstLW1BXsOAACAThHUc3YAAABCTUBHdgoKCs562+Li4kCeAgAAICgCip0PP/xQH3zwgY4dO6YrrrhCkvTJJ58oLCxM1113nW87h8MRnCkBAAACFFDsjBw5UjExMVq6dKm6desm6fiFBidMmKCbbrpJU6dODeqQAAAAgQronJ25c+eqqKjIFzqS1K1bN82YMYNvYwEAgJASUOw0NDTof//7X7v1dXV1amxs7PBQAAAAwRJQ7IwePVoTJkzQ3/72N+3fv1/79+/X3/72N+Xl5WnMmDHBnhEAACBgAZ2z88c//lGFhYW69957dfTo0eM7Cg9XXl6e5syZE9QBAQAAOiKg2OnatasWLlyoOXPm6LPPPpNlWbr88ssVHR0d7PkAAAA6pEMXFaypqVFNTY3S09MVHR0ty7KCNRcAAEBQBBQ7hw4d0uDBg5Wenq7hw4erpqZGkvTggw/ytXMAABBSAoqdX/3qV+rSpYv27dunrl27+taPHTtWb731VtCGAwAA6KiAztlZu3at3n77bfXs2dNvfVpamvbu3RuUwQAAAIIhoCM7zc3Nfkd0Tjh48KAiIyM7PBQAAECwBBQ7N998s5YtW+a77XA41NbWpjlz5mjQoEFBGw4AAKCjAvoYa86cORo4cKC2bt2qlpYWPfbYY9q5c6e+/vprvffee8GeEQAAIGABHdm56qqrtG3bNv34xz/W0KFD1dzcrDFjxujDDz/UZZddFuwZAQAAAnbOR3aOHj2qnJwcLV68WM8++2xnzAQAABA053xkp0uXLtqxY4ccDkdnzAMAABBUAX2Mdf/992vJkiXBngUAACDoAjpBuaWlRX/+859VXl6uvn37tvtNrOLi4qAMBwAA0FHnFDuff/65evfurR07dui6666TJH3yySd+2/DxFgAACCXnFDtpaWmqqanR+vXrJR3/eYg//OEPSkpK6pThAAAAOuqcztk5+VfN33zzTTU3Nwd1IAAAgGAK6ATlE06OHwAAgFBzTrHjcDjanZPDOToAACCUnfPHWOPHj9eYMWM0ZswYHTlyRI888ojv9onlbG3cuFEjR46U2+2Ww+HQq6++2u75pk+fLrfbraioKA0cOFA7d+7028br9Wry5MlKSEhQdHS07rjjDu3fv/9cXhYAADDYOcXOuHHjlJiYKKfTKafTqXvvvVdut9t3+8Rytpqbm3XttddqwYIFp7z/ueeeU3FxsRYsWKAtW7bI5XJp6NChamxs9G2Tn5+vVatWqaysTJs2bVJTU5Nuv/12tba2nstLAwAAhnJYIXLijcPh0KpVqzRq1ChJx4/quN1u5efn6/HHH5d0/ChOUlKSZs+erYcfflgej0fdu3fXCy+8oLFjx0qSDhw4oOTkZK1Zs0bDhg07q+duaGiQ0+mUx+NRbGxsUF9X7ydWB3V/58sXs0bYPQIAAGd0tn+/O3SCcmfas2ePamtrlZOT41sXGRmpW265RZs3b5YkVVZW+n6r6wS3262MjAzfNqfi9XrV0NDgtwAAADOFbOzU1tZKUrtr+CQlJfnuq62tVUREhLp163babU6lqKjI72O35OTkIE8PAABCRcjGzgknf9vLsqzv/AbYd20zbdo0eTwe31JdXR2UWQEAQOgJ2dhxuVyS1O4ITV1dne9oj8vlUktLi+rr60+7zalERkYqNjbWbwEAAGYK2dhJTU2Vy+VSeXm5b11LS4sqKiqUnZ0tScrKylKXLl38tqmpqdGOHTt82wAAgItbQL96HixNTU369NNPfbf37NmjqqoqxcXFqVevXsrPz9fMmTOVlpamtLQ0zZw5U127dtU999wjSXI6ncrLy9PUqVMVHx+vuLg4FRYWKjMzU0OGDLHrZQEAgBBia+xs3bpVgwYN8t0uKCiQdPx6PqWlpXrsscd0+PBhPfroo6qvr1e/fv20du1axcTE+B4zb948hYeH66677tLhw4c1ePBglZaWKiws7Ly/HgAAEHpC5jo7duI6O+1xnR0AQKi74K+zAwAAEAzEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGghHTvTp0+Xw+HwW1wul+9+y7I0ffp0ud1uRUVFaeDAgdq5c6eNEwMAgFAT0rEjSVdffbVqamp8y/bt2333PffccyouLtaCBQu0ZcsWuVwuDR06VI2NjTZODAAAQknIx054eLhcLpdv6d69u6TjR3Xmz5+vp556SmPGjFFGRoaWLl2qb7/9Vi+//LLNUwMAgFAR8rGze/duud1upaam6u6779bnn38uSdqzZ49qa2uVk5Pj2zYyMlK33HKLNm/efMZ9er1eNTQ0+C0AAMBMIR07/fr107Jly/T222/rT3/6k2pra5Wdna1Dhw6ptrZWkpSUlOT3mKSkJN99p1NUVCSn0+lbkpOTO+01AAAAe4V07OTm5urOO+9UZmamhgwZotWrV0uSli5d6tvG4XD4PcayrHbrTjZt2jR5PB7fUl1dHfzhAQBASAjp2DlZdHS0MjMztXv3bt+3sk4+ilNXV9fuaM/JIiMjFRsb67cAAAAzXVCx4/V69Z///Ec9evRQamqqXC6XysvLffe3tLSooqJC2dnZNk4JAABCSbjdA5xJYWGhRo4cqV69eqmurk4zZsxQQ0ODxo0bJ4fDofz8fM2cOVNpaWlKS0vTzJkz1bVrV91zzz12jw4AAEJESMfO/v379fOf/1wHDx5U9+7ddcMNN+j9999XSkqKJOmxxx7T4cOH9eijj6q+vl79+vXT2rVrFRMTY/PkAAAgVDgsy7LsHsJuDQ0Ncjqd8ng8QT9/p/cTq4O6v/Pli1kj7B4BAIAzOtu/3xfUOTsAAADnitgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGC3c7gEQmno/sdruEc7ZF7NG2D0CACAEcWQHAAAYjdgBAABGI3YAAIDRiB0AAGA0Y2Jn4cKFSk1N1SWXXKKsrCz985//tHskAAAQAoz4NtYrr7yi/Px8LVy4UDfeeKMWL16s3Nxcffzxx+rVq5fd4+E84Rtk5wfvM4ALjRFHdoqLi5WXl6cHH3xQV155pebPn6/k5GQtWrTI7tEAAIDNLvgjOy0tLaqsrNQTTzzhtz4nJ0ebN28+5WO8Xq+8Xq/vtsfjkSQ1NDQEfb4277dB3yfM0Rn/5jrbhfhvutevVtg9wjnb8ewwu0dAiMp45m27RzhnnfXv+cT/Qy3LOuN2F3zsHDx4UK2trUpKSvJbn5SUpNra2lM+pqioSM8++2y79cnJyZ0yI3A6zvl2T4BQxb8NmKSz/z03NjbK6XSe9v4LPnZOcDgcfrcty2q37oRp06apoKDAd7utrU1ff/214uPjT/uYQDQ0NCg5OVnV1dWKjY0N2n5xdnj/7cN7by/ef/vw3p9flmWpsbFRbrf7jNtd8LGTkJCgsLCwdkdx6urq2h3tOSEyMlKRkZF+677//e931oiKjY3lH72NeP/tw3tvL95/+/Denz9nOqJzwgV/gnJERISysrJUXl7ut768vFzZ2dk2TQUAAELFBX9kR5IKCgp03333qW/fvurfv7+ef/557du3T4888ojdowEAAJsZETtjx47VoUOH9Jvf/EY1NTXKyMjQmjVrlJKSYutckZGReuaZZ9p9ZIbzg/ffPrz39uL9tw/vfWhyWN/1fS0AAIAL2AV/zg4AAMCZEDsAAMBoxA4AADAasQMAAIxG7HSihQsXKjU1VZdccomysrL0z3/+0+6RLgobN27UyJEj5Xa75XA49Oqrr9o90kWjqKhI119/vWJiYpSYmKhRo0Zp165ddo91UVi0aJGuueYa38Xs+vfvrzfffNPusS5KRUVFcjgcys/Pt3sU/H/ETid55ZVXlJ+fr6eeekoffvihbrrpJuXm5mrfvn12j2a85uZmXXvttVqwYIHdo1x0KioqNHHiRL3//vsqLy/XsWPHlJOTo+bmZrtHM17Pnj01a9Ysbd26VVu3btWtt96qn/zkJ9q5c6fdo11UtmzZoueff17XXHON3aPg/+Cr552kX79+uu6667Ro0SLfuiuvvFKjRo1SUVGRjZNdXBwOh1atWqVRo0bZPcpF6auvvlJiYqIqKip088032z3ORScuLk5z5sxRXl6e3aNcFJqamnTddddp4cKFmjFjhn70ox9p/vz5do8FcWSnU7S0tKiyslI5OTl+63NycrR582abpgLOP4/HI+n4H12cP62trSorK1Nzc7P69+9v9zgXjYkTJ2rEiBEaMmSI3aPgJEZcQTnUHDx4UK2tre1+iDQpKandD5YCprIsSwUFBRowYIAyMjLsHueisH37dvXv319HjhzRpZdeqlWrVumqq66ye6yLQllZmT744ANt2bLF7lFwCsROJ3I4HH63Lctqtw4w1aRJk7Rt2zZt2rTJ7lEuGldccYWqqqr0zTff6O9//7vGjRuniooKgqeTVVdXa8qUKVq7dq0uueQSu8fBKRA7nSAhIUFhYWHtjuLU1dW1O9oDmGjy5Ml6/fXXtXHjRvXs2dPucS4aERERuvzyyyVJffv21ZYtW/T73/9eixcvtnkys1VWVqqurk5ZWVm+da2trdq4caMWLFggr9ersLAwGycE5+x0goiICGVlZam8vNxvfXl5ubKzs22aCuh8lmVp0qRJWrlypdatW6fU1FS7R7qoWZYlr9dr9xjGGzx4sLZv366qqirf0rdvX/3iF79QVVUVoRMCOLLTSQoKCnTfffepb9++6t+/v55//nnt27dPjzzyiN2jGa+pqUmffvqp7/aePXtUVVWluLg49erVy8bJzDdx4kS9/PLLeu211xQTE+M7uul0OhUVFWXzdGZ78sknlZubq+TkZDU2NqqsrEwbNmzQW2+9ZfdoxouJiWl3Xlp0dLTi4+M5Xy1EEDudZOzYsTp06JB+85vfqKamRhkZGVqzZo1SUlLsHs14W7du1aBBg3y3CwoKJEnjxo1TaWmpTVNdHE5camHgwIF+60tKSjR+/PjzP9BF5H//+5/uu+8+1dTUyOl06pprrtFbb72loUOH2j0aYDuuswMAAIzGOTsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAACATrFx40aNHDlSbrdbDodDr7766jnvw7Is/e53v1N6eroiIyOVnJysmTNnntM+uIIyAADoFM3Nzbr22ms1YcIE3XnnnQHt48Qvyv/ud79TZmamPB6PDh48eE774ArKAACg0zkcDq1atUqjRo3yrWtpadGvf/1rvfTSS/rmm2+UkZGh2bNn+35y5j//+Y+uueYa7dixQ1dccUXAz83HWAAAwBYTJkzQe++9p7KyMm3btk0/+9nPdNttt2n37t2SpDfeeEM/+MEP9I9//EOpqanq3bu3HnzwQX399dfn9DzEDgAAOO8+++wzLV++XCtWrNBNN92kyy67TIWFhRowYIBKSkokSZ9//rn27t2rFStWaNmyZSotLVVlZaV++tOfntNzcc4OAAA47z744ANZlqX09HS/9V6vV/Hx8ZKktrY2eb1eLVu2zLfdkiVLlJWVpV27dp31R1vEDgAAOO/a2toUFhamyspKhYWF+d136aWXSpJ69Oih8PBwvyC68sorJUn79u0jdgAAQOjq06ePWltbVVdXp5tuuumU29x44406duyYPvvsM1122WWSpE8++USSlJKSctbPxbexAABAp2hqatKnn34q6XjcFBcXa9CgQYqLi1OvXr1077336r333tPcuXPVp08fHTx4UOvWrVNmZqaGDx+utrY2XX/99br00ks1f/58tbW1aeLEiYqNjdXatWvPeg5iBwAAdIoNGzZo0KBB7daPGzdOpaWlOnr0qGbMmKFly5bpyy+/VHx8vPr3769nn31WmZmZkqQDBw5o8uTJWrt2raKjo5Wbm6u5c+cqLi7urOcgdgAAgNH46jkAADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBo/w+RlTDsyLKkSwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "svi[\"E_TOTPOP\"].plot.hist() # plotting a histogram of the svi dataframe's \"E_TOTPOP\" (estimated population) column" ] }, { "cell_type": "markdown", "id": "67ac821f-eb8c-4f69-a743-6e571ceb6e7c", "metadata": {}, "source": [ "### Basic Data Cleaning/Filtering\n", "\n", "We will walk through a few basic steps to clean and filter data using Pandas. This will be useful when we move onto spatial data (GeoDataFrames) as it all uses DataFrames as a baseline." ] }, { "cell_type": "markdown", "id": "c96cd354-78a4-4146-9278-aba054571ad4", "metadata": {}, "source": [ "We can create new columns or overwrite columns by using that bracket syntax we saw earlier to refer to columns. For example, let's say we want to work with population as thousands. The following code will create a new column called `PopDensity` and save the `E_TOTPOP` variable divided by `AREA_SQMI` variable as the result. It will automatically do this for every row and assign the correct values to each row." ] }, { "cell_type": "code", "execution_count": 17, "id": "3276f6f4-84e7-44ad-bae8-66e0fe111443", "metadata": {}, "outputs": [], "source": [ "svi[\"PopDensity\"] = svi[\"E_TOTPOP\"] / svi[\"AREA_SQMI\"] " ] }, { "cell_type": "code", "execution_count": 18, "id": "78901de3-545e-4172-a719-1b6aaf1cb6e4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGdCAYAAAD0e7I1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAjcklEQVR4nO3deXDU9f3H8ddKjoY0WQkh2ayEmBHQSiJTQTm0yiHhEFRwKp5ERQeqUFJgVHQ6YH8OQRgDdqhILcOhVbzAOgMqUa5isAKCELCIGu6sEYw5ODaYfH5/MKxdwpXNN+zy4fmY2Rn3+/1mee9nvk6e890jLmOMEQAAgKUuCfcAAAAATYnYAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGC1qHAPEAnq6uq0f/9+JSQkyOVyhXscAABwDowxqqqqktfr1SWXnP76DbEjaf/+/UpPTw/3GAAAIAR79uxR69atT7uf2JGUkJAg6fhiJSYmhnkaAABwLiorK5Wenh74PX46xI4UeOkqMTGR2AEA4AJztreg8AZlAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYLSrcA9ju8qeWhHuEkOyccmu4RwAAwBFc2QEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWC2vs5Ofn67rrrlNCQoJSUlJ0xx13aPv27UHHGGM0adIkeb1excXFqUePHtq6dWvQMX6/X6NHj1ZycrLi4+N12223ae/evefzqQAAgAgV1thZtWqVHn/8cX322WcqLCzUzz//rJycHB06dChwzNSpU1VQUKCZM2dq3bp18ng86tOnj6qqqgLH5OXlafHixVq4cKHWrFmj6upqDRw4ULW1teF4WgAAIIK4jDEm3EOc8MMPPyglJUWrVq3STTfdJGOMvF6v8vLy9OSTT0o6fhUnNTVVzz//vEaMGKGKigq1atVKr776qoYOHSpJ2r9/v9LT07V06VL17dv3rP9uZWWl3G63KioqlJiY6OhzuvypJY4+3vmyc8qt4R4BAIAzOtff3xH1np2KigpJUlJSkiSppKREPp9POTk5gWNiY2N18803q6ioSJK0YcMGHTt2LOgYr9errKyswDEn8/v9qqysDLoBAAA7RUzsGGM0duxY3XjjjcrKypIk+Xw+SVJqamrQsampqYF9Pp9PMTExatGixWmPOVl+fr7cbnfglp6e7vTTAQAAESJiYmfUqFHavHmz3njjjXr7XC5X0H1jTL1tJzvTMRMmTFBFRUXgtmfPntAHBwAAES0iYmf06NF6//33tWLFCrVu3Tqw3ePxSFK9KzRlZWWBqz0ej0c1NTUqLy8/7TEni42NVWJiYtANAADYKayxY4zRqFGjtGjRIi1fvlyZmZlB+zMzM+XxeFRYWBjYVlNTo1WrVql79+6SpE6dOik6OjromNLSUhUXFweOAQAAF6+ocP7jjz/+uF5//XX961//UkJCQuAKjtvtVlxcnFwul/Ly8jR58mS1a9dO7dq10+TJk9W8eXPde++9gWOHDx+ucePGqWXLlkpKStL48eOVnZ2tW265JZxPDwAARICwxs6sWbMkST169AjaPnfuXD344IOSpCeeeEJHjhzRY489pvLycnXp0kXLli1TQkJC4Pjp06crKipKd911l44cOaLevXtr3rx5atas2fl6KgAAIEJF1PfshAvfs1Mf37MDAIh0F+T37AAAADiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYLWwxs7q1as1aNAgeb1euVwuvffee0H7H3zwQblcrqBb165dg47x+/0aPXq0kpOTFR8fr9tuu0179+49j88CAABEsrDGzqFDh9SxY0fNnDnztMf069dPpaWlgdvSpUuD9ufl5Wnx4sVauHCh1qxZo+rqag0cOFC1tbVNPT4AALgARIXzH+/fv7/69+9/xmNiY2Pl8XhOua+iokJz5szRq6++qltuuUWS9Nprryk9PV0ff/yx+vbt6/jMAADgwhLx79lZuXKlUlJS1L59ez366KMqKysL7NuwYYOOHTumnJycwDav16usrCwVFRWFY1wAABBhwnpl52z69++v3//+98rIyFBJSYn+/Oc/q1evXtqwYYNiY2Pl8/kUExOjFi1aBP1camqqfD7faR/X7/fL7/cH7ldWVjbZcwAAAOEV0bEzdOjQwH9nZWWpc+fOysjI0JIlSzRkyJDT/pwxRi6X67T78/Pz9eyzzzo6KwAAiEwR/zLW/0pLS1NGRoZ27NghSfJ4PKqpqVF5eXnQcWVlZUpNTT3t40yYMEEVFRWB2549e5p0bgAAED4XVOwcPHhQe/bsUVpamiSpU6dOio6OVmFhYeCY0tJSFRcXq3v37qd9nNjYWCUmJgbdAACAncL6MlZ1dbW++eabwP2SkhJt2rRJSUlJSkpK0qRJk3TnnXcqLS1NO3fu1NNPP63k5GQNHjxYkuR2uzV8+HCNGzdOLVu2VFJSksaPH6/s7OzAp7MAAMDFLayxs379evXs2TNwf+zYsZKk3NxczZo1S1u2bNGCBQv0008/KS0tTT179tSbb76phISEwM9Mnz5dUVFRuuuuu3TkyBH17t1b8+bNU7Nmzc778wEAAJHHZYwx4R4i3CorK+V2u1VRUeH4S1qXP7XE0cc7X3ZOuTXcIwAAcEbn+vv7gnrPDgAAQEMROwAAwGrEDgAAsFpIsVNSUuL0HAAAAE0ipNhp27atevbsqddee01Hjx51eiYAAADHhBQ7X375pX77299q3Lhx8ng8GjFihD7//HOnZwMAAGi0kGInKytLBQUF2rdvn+bOnSufz6cbb7xRHTp0UEFBgX744Qen5wQAAAhJo96gHBUVpcGDB+utt97S888/r2+//Vbjx49X69atNWzYMJWWljo1JwAAQEgaFTvr16/XY489prS0NBUUFGj8+PH69ttvtXz5cu3bt0+33367U3MCAACEJKQ/F1FQUKC5c+dq+/btGjBggBYsWKABAwbokkuOt1NmZqZmz56tq666ytFhAQAAGiqk2Jk1a5YefvhhPfTQQ/J4PKc8pk2bNpozZ06jhgMAAGiskGJnx44dZz0mJiZGubm5oTw8AACAY0J6z87cuXP19ttv19v+9ttva/78+Y0eCgAAwCkhxc6UKVOUnJxcb3tKSoomT57c6KEAAACcElLs7Nq1S5mZmfW2Z2RkaPfu3Y0eCgAAwCkhxU5KSoo2b95cb/uXX36pli1bNnooAAAAp4QUO3fffbf++Mc/asWKFaqtrVVtba2WL1+uMWPG6O6773Z6RgAAgJCF9Gms5557Trt27VLv3r0VFXX8Ierq6jRs2DDeswMAACJKSLETExOjN998U//3f/+nL7/8UnFxccrOzlZGRobT8wEAADRKSLFzQvv27dW+fXunZgEAAHBcSLFTW1urefPm6ZNPPlFZWZnq6uqC9i9fvtyR4QAAABorpNgZM2aM5s2bp1tvvVVZWVlyuVxOzwUAAOCIkGJn4cKFeuuttzRgwACn5wEAAHBUSB89j4mJUdu2bZ2eBQAAwHEhxc64ceP04osvyhjj9DwAAACOCullrDVr1mjFihX64IMP1KFDB0VHRwftX7RokSPDAQAANFZIsXPppZdq8ODBTs8CAADguJBiZ+7cuU7PAQAA0CRCes+OJP3888/6+OOPNXv2bFVVVUmS9u/fr+rqaseGAwAAaKyQruzs2rVL/fr10+7du+X3+9WnTx8lJCRo6tSpOnr0qF5++WWn5wQAAAhJSFd2xowZo86dO6u8vFxxcXGB7YMHD9Ynn3zi2HAAAACNFfKnsT799FPFxMQEbc/IyNC+ffscGQwAAMAJIV3ZqaurU21tbb3te/fuVUJCQqOHAgAAcEpIsdOnTx/NmDEjcN/lcqm6uloTJ07kT0gAAICIEtLLWNOnT1fPnj119dVX6+jRo7r33nu1Y8cOJScn64033nB6RgAAgJCFFDter1ebNm3SG2+8oS+++EJ1dXUaPny47rvvvqA3LAMAAIRbSLEjSXFxcXr44Yf18MMPOzkPAACAo0KKnQULFpxx/7Bhw0IaBgAAwGkhxc6YMWOC7h87dkyHDx9WTEyMmjdvTuwAAICIEdKnscrLy4Nu1dXV2r59u2688UbeoAwAACJKyH8b62Tt2rXTlClT6l31AQAACCfHYkeSmjVrpv379zv5kAAAAI0S0nt23n///aD7xhiVlpZq5syZuuGGGxwZDAAAwAkhxc4dd9wRdN/lcqlVq1bq1auXXnjhBSfmAgAAcERIsVNXV+f0HAAAAE3C0ffsAAAARJqQruyMHTv2nI8tKCgI5Z8AAABwREixs3HjRn3xxRf6+eefdeWVV0qSvv76azVr1kzXXntt4DiXy+XMlAAAACEKKXYGDRqkhIQEzZ8/Xy1atJB0/IsGH3roIf3ud7/TuHHjHB0SAAAgVCG9Z+eFF15Qfn5+IHQkqUWLFnruuef4NBYAAIgoIcVOZWWlvv/++3rby8rKVFVV1eihAAAAnBJS7AwePFgPPfSQ3nnnHe3du1d79+7VO++8o+HDh2vIkCFOzwgAABCykN6z8/LLL2v8+PG6//77dezYseMPFBWl4cOHa9q0aY4OCAAA0BghxU7z5s310ksvadq0afr2229ljFHbtm0VHx/v9HwAAACN0qgvFSwtLVVpaanat2+v+Ph4GWOcmgsAAMARIcXOwYMH1bt3b7Vv314DBgxQaWmpJOmRRx7hY+cAACCihBQ7f/rTnxQdHa3du3erefPmge1Dhw7Vhx9+6NhwAAAAjRXSe3aWLVumjz76SK1btw7a3q5dO+3atcuRwQAAAJwQ0pWdQ4cOBV3ROeHAgQOKjY1t9FAAAABOCSl2brrpJi1YsCBw3+Vyqa6uTtOmTVPPnj0dGw4AAKCxQnoZa9q0aerRo4fWr1+vmpoaPfHEE9q6dat+/PFHffrpp07PCAAAELKQruxcffXV2rx5s66//nr16dNHhw4d0pAhQ7Rx40ZdccUVTs8IAAAQsgZf2Tl27JhycnI0e/ZsPfvss00xEwAAgGMafGUnOjpaxcXFcrlcTTEPAACAo0J6GWvYsGGaM2eO07MAAAA4LqTYqamp0axZs9SpUyeNGDFCY8eODbqdq9WrV2vQoEHyer1yuVx67733gvYbYzRp0iR5vV7FxcWpR48e2rp1a9Axfr9fo0ePVnJysuLj43Xbbbdp7969oTwtAABgoQbFznfffae6ujoVFxfr2muvVWJior7++mtt3LgxcNu0adM5P96hQ4fUsWNHzZw585T7p06dqoKCAs2cOVPr1q2Tx+NRnz59VFVVFTgmLy9Pixcv1sKFC7VmzRpVV1dr4MCBqq2tbchTAwAAlnKZBvz1zmbNmqm0tFQpKSmSjv95iL/+9a9KTU1t/CAulxYvXqw77rhD0vGrOl6vV3l5eXryySclHb+Kk5qaqueff14jRoxQRUWFWrVqpVdffVVDhw6VJO3fv1/p6elaunSp+vbte07/dmVlpdxutyoqKpSYmNjo5/K/Ln9qiaOPd77snHJruEcAAOCMzvX3d4Ou7JzcRR988IEOHToU2oRnUVJSIp/Pp5ycnMC22NhY3XzzzSoqKpIkbdiwIfDpsBO8Xq+ysrICx5yK3+9XZWVl0A0AANgppPfsnNCAi0IN5vP5JKneVaPU1NTAPp/Pp5iYGLVo0eK0x5xKfn6+3G534Jaenu7w9AAAIFI0KHZcLle9j5w39UfQT358Y8xZ/82zHTNhwgRVVFQEbnv27HFkVgAAEHka9KWCxhg9+OCDgT/2efToUY0cOVLx8fFBxy1atKjRg3k8HknHr96kpaUFtpeVlQWu9ng8HtXU1Ki8vDzo6k5ZWZm6d+9+2seOjY3lD5YCAHCRaNCVndzcXKWkpARe/rn//vvl9XqDXhJyu92ODJaZmSmPx6PCwsLAtpqaGq1atSoQMp06dVJ0dHTQMaWlpSouLj5j7AAAgItHg67szJ0719F/vLq6Wt98803gfklJiTZt2qSkpCS1adNGeXl5mjx5stq1a6d27dpp8uTJat68ue69915Jktvt1vDhwzVu3Di1bNlSSUlJGj9+vLKzs3XLLbc4OisAALgwhfRXz52yfv169ezZM3D/xBcS5ubmat68eXriiSd05MgRPfbYYyovL1eXLl20bNkyJSQkBH5m+vTpioqK0l133aUjR46od+/emjdvnpo1a3benw8AAIg8DfqeHVvxPTv18T07AIBI1yTfswMAAHChIXYAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFaL6NiZNGmSXC5X0M3j8QT2G2M0adIkeb1excXFqUePHtq6dWsYJwYAAJEmomNHkjp06KDS0tLAbcuWLYF9U6dOVUFBgWbOnKl169bJ4/GoT58+qqqqCuPEAAAgkkR87ERFRcnj8QRurVq1knT8qs6MGTP0zDPPaMiQIcrKytL8+fN1+PBhvf7662GeGgAARIqIj50dO3bI6/UqMzNTd999t7777jtJUklJiXw+n3JycgLHxsbG6uabb1ZRUdEZH9Pv96uysjLoBgAA7BTRsdOlSxctWLBAH330kV555RX5fD51795dBw8elM/nkySlpqYG/Uxqampg3+nk5+fL7XYHbunp6U32HAAAQHhFdOz0799fd955p7Kzs3XLLbdoyZIlkqT58+cHjnG5XEE/Y4ypt+1kEyZMUEVFReC2Z88e54cHAAARIaJj52Tx8fHKzs7Wjh07Ap/KOvkqTllZWb2rPSeLjY1VYmJi0A0AANjpgoodv9+vr776SmlpacrMzJTH41FhYWFgf01NjVatWqXu3buHcUoAABBJosI9wJmMHz9egwYNUps2bVRWVqbnnntOlZWVys3NlcvlUl5eniZPnqx27dqpXbt2mjx5spo3b65777033KMDAIAIEdGxs3fvXt1zzz06cOCAWrVqpa5du+qzzz5TRkaGJOmJJ57QkSNH9Nhjj6m8vFxdunTRsmXLlJCQEObJAQBApHAZY0y4hwi3yspKud1uVVRUOP7+ncufWuLo450vO6fcGu4RAAA4o3P9/X1BvWcHAACgoYgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFgtKtwDIDJd/tSScI/QYDun3BruEQAAEYgrOwAAwGrEDgAAsBqxAwAArEbsAAAAqxE7AADAasQOAACwGrEDAACsRuwAAACrWfOlgi+99JKmTZum0tJSdejQQTNmzNDvfve7cI+F8+hC/CLECxFf3gjgQmNF7Lz55pvKy8vTSy+9pBtuuEGzZ89W//79tW3bNrVp0ybc4wFWuRCjkkCDTfh/sOGseBmroKBAw4cP1yOPPKLf/OY3mjFjhtLT0zVr1qxwjwYAAMLsgr+yU1NTow0bNuipp54K2p6Tk6OioqJT/ozf75ff7w/cr6iokCRVVlY6Pl+d/7DjjwmgYdr86e1wj9Bgxc/2DfcIF4WsiR+Fe4SLQlP8fv3fxzXGnPG4Cz52Dhw4oNraWqWmpgZtT01Nlc/nO+XP5Ofn69lnn623PT09vUlmBICGcs8I9wSAc5r6fK6qqpLb7T7t/gs+dk5wuVxB940x9badMGHCBI0dOzZwv66uTj/++KNatmx52p8JRWVlpdLT07Vnzx4lJiY69ri2Yr0ahvU6d6xVw7Be5461ahin18sYo6qqKnm93jMed8HHTnJyspo1a1bvKk5ZWVm9qz0nxMbGKjY2NmjbpZde2lQjKjExkf8JGoD1ahjW69yxVg3Dep071qphnFyvM13ROeGCf4NyTEyMOnXqpMLCwqDthYWF6t69e5imAgAAkeKCv7IjSWPHjtUDDzygzp07q1u3bvr73/+u3bt3a+TIkeEeDQAAhJkVsTN06FAdPHhQf/nLX1RaWqqsrCwtXbpUGRkZYZ0rNjZWEydOrPeSGU6N9WoY1uvcsVYNw3qdO9aqYcK1Xi5zts9rAQAAXMAu+PfsAAAAnAmxAwAArEbsAAAAqxE7AADAasROE3rppZeUmZmpX/3qV+rUqZP+/e9/h3uk827SpElyuVxBN4/HE9hvjNGkSZPk9XoVFxenHj16aOvWrUGP4ff7NXr0aCUnJys+Pl633Xab9u7de76fiuNWr16tQYMGyev1yuVy6b333gva79TalJeX64EHHpDb7Zbb7dYDDzygn376qYmfnfPOtl4PPvhgvXOta9euQcdcLOuVn5+v6667TgkJCUpJSdEdd9yh7du3Bx3D+fWLc1kvzq/jZs2apWuuuSbwpYDdunXTBx98ENgfseeVQZNYuHChiY6ONq+88orZtm2bGTNmjImPjze7du0K92jn1cSJE02HDh1MaWlp4FZWVhbYP2XKFJOQkGDeffdds2XLFjN06FCTlpZmKisrA8eMHDnSXHbZZaawsNB88cUXpmfPnqZjx47m559/DsdTcszSpUvNM888Y959910jySxevDhov1Nr069fP5OVlWWKiopMUVGRycrKMgMHDjxfT9MxZ1uv3Nxc069fv6Bz7eDBg0HHXCzr1bdvXzN37lxTXFxsNm3aZG699VbTpk0bU11dHTiG8+sX57JenF/Hvf/++2bJkiVm+/btZvv27ebpp5820dHRpri42BgTuecVsdNErr/+ejNy5MigbVdddZV56qmnwjRReEycONF07NjxlPvq6uqMx+MxU6ZMCWw7evSocbvd5uWXXzbGGPPTTz+Z6Ohos3DhwsAx+/btM5dccon58MMPm3T28+nkX95Orc22bduMJPPZZ58Fjlm7dq2RZP773/828bNqOqeLndtvv/20P3Mxr1dZWZmRZFatWmWM4fw6m5PXyxjOrzNp0aKF+cc//hHR5xUvYzWBmpoabdiwQTk5OUHbc3JyVFRUFKapwmfHjh3yer3KzMzU3Xffre+++06SVFJSIp/PF7ROsbGxuvnmmwPrtGHDBh07dizoGK/Xq6ysLKvX0qm1Wbt2rdxut7p06RI4pmvXrnK73Vau38qVK5WSkqL27dvr0UcfVVlZWWDfxbxeFRUVkqSkpCRJnF9nc/J6ncD5Fay2tlYLFy7UoUOH1K1bt4g+r4idJnDgwAHV1tbW+0Okqamp9f5gqe26dOmiBQsW6KOPPtIrr7win8+n7t276+DBg4G1ONM6+Xw+xcTEqEWLFqc9xkZOrY3P51NKSkq9x09JSbFu/fr3769//vOfWr58uV544QWtW7dOvXr1kt/vl3TxrpcxRmPHjtWNN96orKwsSZxfZ3Kq9ZI4v/7Xli1b9Otf/1qxsbEaOXKkFi9erKuvvjqizysr/lxEpHK5XEH3jTH1ttmuf//+gf/Ozs5Wt27ddMUVV2j+/PmBN/eFsk4Xy1o6sTanOt7G9Rs6dGjgv7OystS5c2dlZGRoyZIlGjJkyGl/zvb1GjVqlDZv3qw1a9bU28f5Vd/p1ovz6xdXXnmlNm3apJ9++knvvvuucnNztWrVqsD+SDyvuLLTBJKTk9WsWbN6BVpWVlaveC828fHxys7O1o4dOwKfyjrTOnk8HtXU1Ki8vPy0x9jIqbXxeDz6/vvv6z3+Dz/8YPX6SVJaWpoyMjK0Y8cOSRfneo0ePVrvv/++VqxYodatWwe2c36d2unW61Qu5vMrJiZGbdu2VefOnZWfn6+OHTvqxRdfjOjzithpAjExMerUqZMKCwuDthcWFqp79+5hmioy+P1+ffXVV0pLS1NmZqY8Hk/QOtXU1GjVqlWBderUqZOio6ODjiktLVVxcbHVa+nU2nTr1k0VFRX6/PPPA8f85z//UUVFhdXrJ0kHDx7Unj17lJaWJuniWi9jjEaNGqVFixZp+fLlyszMDNrP+RXsbOt1Khfz+XUyY4z8fn9kn1chva0ZZ3Xio+dz5swx27ZtM3l5eSY+Pt7s3Lkz3KOdV+PGjTMrV6403333nfnss8/MwIEDTUJCQmAdpkyZYtxut1m0aJHZsmWLueeee075McXWrVubjz/+2HzxxRemV69eVnz0vKqqymzcuNFs3LjRSDIFBQVm48aNga8ncGpt+vXrZ6655hqzdu1as3btWpOdnX1BfdT1hDOtV1VVlRk3bpwpKioyJSUlZsWKFaZbt27msssuuyjX6w9/+INxu91m5cqVQR+VPnz4cOAYzq9fnG29OL9+MWHCBLN69WpTUlJiNm/ebJ5++mlzySWXmGXLlhljIve8Inaa0N/+9jeTkZFhYmJizLXXXhv0McaLxYnvWIiOjjZer9cMGTLEbN26NbC/rq7OTJw40Xg8HhMbG2tuuukms2XLlqDHOHLkiBk1apRJSkoycXFxZuDAgWb37t3n+6k4bsWKFUZSvVtubq4xxrm1OXjwoLnvvvtMQkKCSUhIMPfdd58pLy8/T8/SOWdar8OHD5ucnBzTqlUrEx0dbdq0aWNyc3PrrcXFsl6nWidJZu7cuYFjOL9+cbb14vz6xcMPPxz4vdaqVSvTu3fvQOgYE7nnlcsYY0K7JgQAABD5eM8OAACwGrEDAACsRuwAAACrETsAAMBqxA4AALAasQMAAKxG7AAAAKsROwAAwGrEDgAAsBqxAwAArEbsAAAAqxE7AADAav8PnQ+crBtLmYsAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "svi[\"PopDensity\"].plot.hist()" ] }, { "cell_type": "markdown", "id": "4b047bd9-0a20-4483-a491-b2b35ab342b9", "metadata": {}, "source": [ "If we want to delete that column, we can use the `drop` function and save the result. Note that just calling `svi.drop()` doesn't change the svi object, you need to assign svi to the result like so:" ] }, { "cell_type": "code", "execution_count": 19, "id": "5dafb3bc-c8d5-4c73-acd0-428ba9507d28", "metadata": {}, "outputs": [], "source": [ "svi = svi.drop(columns=[\"PopDensity\"])" ] }, { "cell_type": "code", "execution_count": 20, "id": "9cfcc09e-3b9b-4ea2-8a44-e089eb4ff966", "metadata": {}, "outputs": [ { "ename": "KeyError", "evalue": "'PopDensity'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", "File \u001b[0;32m/cvmfs/cybergis.illinois.edu/software/conda/cybergisx/python3-0.9.4/lib/python3.8/site-packages/pandas/core/indexes/base.py:3652\u001b[0m, in \u001b[0;36mIndex.get_loc\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m 3651\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 3652\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_engine\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_loc\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcasted_key\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 3653\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n", "File \u001b[0;32m/cvmfs/cybergis.illinois.edu/software/conda/cybergisx/python3-0.9.4/lib/python3.8/site-packages/pandas/_libs/index.pyx:147\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n", "File \u001b[0;32m/cvmfs/cybergis.illinois.edu/software/conda/cybergisx/python3-0.9.4/lib/python3.8/site-packages/pandas/_libs/index.pyx:176\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n", "File \u001b[0;32mpandas/_libs/hashtable_class_helper.pxi:7080\u001b[0m, in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n", "File \u001b[0;32mpandas/_libs/hashtable_class_helper.pxi:7088\u001b[0m, in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n", "\u001b[0;31mKeyError\u001b[0m: 'PopDensity'", "\nThe above exception was the direct cause of the following exception:\n", "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", "Input \u001b[0;32mIn [20]\u001b[0m, in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43msvi\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mPopDensity\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241m.\u001b[39mplot\u001b[38;5;241m.\u001b[39mhist() \u001b[38;5;66;03m# this won't work because the column was dropped. You'll get a \"KeyError\" meaning that column name doesn't exist.\u001b[39;00m\n", "File \u001b[0;32m/cvmfs/cybergis.illinois.edu/software/conda/cybergisx/python3-0.9.4/lib/python3.8/site-packages/pandas/core/frame.py:3760\u001b[0m, in \u001b[0;36mDataFrame.__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m 3758\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcolumns\u001b[38;5;241m.\u001b[39mnlevels \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[1;32m 3759\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_getitem_multilevel(key)\n\u001b[0;32m-> 3760\u001b[0m indexer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcolumns\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_loc\u001b[49m\u001b[43m(\u001b[49m\u001b[43mkey\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 3761\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m is_integer(indexer):\n\u001b[1;32m 3762\u001b[0m indexer \u001b[38;5;241m=\u001b[39m [indexer]\n", "File \u001b[0;32m/cvmfs/cybergis.illinois.edu/software/conda/cybergisx/python3-0.9.4/lib/python3.8/site-packages/pandas/core/indexes/base.py:3654\u001b[0m, in \u001b[0;36mIndex.get_loc\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m 3652\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_engine\u001b[38;5;241m.\u001b[39mget_loc(casted_key)\n\u001b[1;32m 3653\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n\u001b[0;32m-> 3654\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m(key) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01merr\u001b[39;00m\n\u001b[1;32m 3655\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m:\n\u001b[1;32m 3656\u001b[0m \u001b[38;5;66;03m# If we have a listlike key, _check_indexing_error will raise\u001b[39;00m\n\u001b[1;32m 3657\u001b[0m \u001b[38;5;66;03m# InvalidIndexError. Otherwise we fall through and re-raise\u001b[39;00m\n\u001b[1;32m 3658\u001b[0m \u001b[38;5;66;03m# the TypeError.\u001b[39;00m\n\u001b[1;32m 3659\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_indexing_error(key)\n", "\u001b[0;31mKeyError\u001b[0m: 'PopDensity'" ] } ], "source": [ "svi[\"PopDensity\"].plot.hist() # this won't work because the column was dropped. You'll get a \"KeyError\" meaning that column name doesn't exist." ] }, { "cell_type": "markdown", "id": "a7b529bd-c8bf-45be-90ec-2e357ab530b1", "metadata": {}, "source": [ "Another common task is filtering data! We can use the [`loc`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.loc.html#pandas.DataFrame.loc) method to filter our data down based on some criteria.\n", "\n", "For example, below we filter the svi data to look for counties with a population over 1,000,000 (1 million, note you don't use the commas in the code). This will save the result in a new variable, so our `svi` dataset remains unaffected." ] }, { "cell_type": "code", "execution_count": 21, "id": "5767a4d3-9d37-476f-800e-86de4cfe5586", "metadata": {}, "outputs": [], "source": [ "bigCounties = svi.loc[svi[\"E_TOTPOP\"] > 1000000]" ] }, { "cell_type": "code", "execution_count": 22, "id": "f46fb033-4f4c-42e3-9152-2b75f024d02e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There are 6 counties with a population over 1,000,000! The svi dataset still has 254 rows though!\n" ] } ], "source": [ "# this uses a fancy trick called \"f-strings\" to insert variables into the text we want to print!\n", "print(f\"There are {len(bigCounties)} counties with a population over 1,000,000! The svi dataset still has {len(svi)} rows though!\")" ] }, { "cell_type": "markdown", "id": "c7e61364-1c4e-4d46-b12f-725fc4420689", "metadata": {}, "source": [ "## Self-Guided Exercises\n", "\n", "A few exercises to help you test your knowledge and help you explore the documentation!\n", "\n", "* Can you filter the svi dataframe by population (`E_TOTPOP`) above 1,000,000 but below 3,000,000?\n", "* What county has the highest overall social vulnerability (refer to the documentation in this folder to see the variable name).\n", "* What does the histogram look like for `AREA_SQMI`?\n", "* How can you sort a dataframe using a column?\n", "\n", "[Additional exercises for those interested](https://martinfleischmann.net/sds/data_wrangling/exercise.html)" ] }, { "cell_type": "code", "execution_count": null, "id": "656289ee-076c-4488-878e-43c23152302a", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" } }, "nbformat": 4, "nbformat_minor": 5 }