{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Data enrichment\n", "\n", "### Introduction\n", "\n", "We define enrichment as the process of augmenting your data with new variables by means of a spatial join between your data and a `Dataset` aggregated at a given spatial resolution in the CARTO Data Observatory, or in other words:\n", "\n", "\"*Enrichment is the process of adding variables to a geometry, which we call the target, (point, line, polygon…) from a spatial (polygon) dataset, which we call the source*\"\n", "\n", "We recommend you check out the [CARTOframes quickstart](/developers/cartoframes/guides/Quickstart/) since this guide uses some of the generated DataFrames as well as the [Data Discovery guide](/developers/cartoframes/guides/Data-discovery) to learn about exploring the Data Observatory catalog to find variables of interest for your analyses.\n", "\n", "### Choose variables to enrich from the Data Observatory catalog\n", "\n", "Let's follow up with the [Data Discovery guide](/developers/cartoframes/guides/Data-discovery), where we subscribed to the AGS demographics dataset and listed the variables available to enrich our own data." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from cartoframes.auth import set_default_credentials\n", "\n", "set_default_credentials('creds.json')" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[,\n", " ,\n", " ,\n", " ]" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from cartoframes.data.observatory import Catalog, Dataset, Variable, Geography\n", "Catalog().subscriptions().datasets" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[ #'Median Household Income: Age 65-74 (2019A)',\n", " #'Median Household Income: Age 55-64 (2019A)',\n", " #'Median Household Income: Age 45-54 (2019A)',\n", " #'Median Household Income: Age 35-44 (2019A)',\n", " #'Median Household Income: Age 25-34 (2019A)',\n", " #'Median Household Income: Age < 25 (2019A)',\n", " #'Household Income > $200000 (2019A)',\n", " #'Household Income $60000-$74999 (2019A)',\n", " #'Household Income $45000-$49999 (2019A)',\n", " #'Household Income $40000-$44999 (2019A)',\n", " #'Household Income $35000-$39999 (2019A)',\n", " #'Household Income $25000-$29999 (2019A)',\n", " #'Household Income $20000-$24999 (2019A)',\n", " #'Household Income $15000-$19999 (2019A)',\n", " #'Household Income $125000-$149999 (2019A)',\n", " #'Families married couple w children (2019A)',\n", " #'Families male no wife w children (2019A)',\n", " #'Families female no husband children (2019A)',\n", " #'Median Age of Householder (2019A)',\n", " #'Family Households (2019A)',\n", " #'Average Household Size (2019A)',\n", " #'Households (2019A)',\n", " #'Pop 25+ 9th-12th grade no diploma (2019A)',\n", " #'Pop 25+ less than 9th grade (2019A)',\n", " #'Pop 25+ HS graduate (2019A)',\n", " #'Pop 25+ graduate or prof school degree (2019A)',\n", " #'Pop 25+ Bachelors degree (2019A)',\n", " #'Housing units vacant (2019A)',\n", " #'Occupied units renter (2019A)',\n", " #'Occupied units owner (2019A)',\n", " #'Median Age (2019A)',\n", " #'Population age 85+ (2019A)',\n", " #'Population Age 25+ (2019A)',\n", " #'Population Age 15+ (2019A)',\n", " #'Population age 80-84 (2019A)',\n", " #'Population age 75-79 (2019A)',\n", " #'Population age 70-74 (2019A)',\n", " #'Population age 60-64 (2019A)',\n", " #'Population age 55-59 (2019A)',\n", " #'Population age 50-54 (2019A)',\n", " #'Population age 45-49 (2019A)',\n", " #'Population age 40-44 (2019A)',\n", " #'Population age 30-34 (2019A)',\n", " #'Population age 25-29 (2019A)',\n", " #'Population age 15-19 (2019A)',\n", " #'Population age 5-9 (2019A)',\n", " #'Population age 0-4 (2019A)',\n", " #'Pop 25+ college no diploma (2019A)',\n", " #'Now Married (2019A)',\n", " #'Population age 20-24 (2019A)',\n", " #'Population age 10-14 (2019A)',\n", " #'Population age 35-39 (2019A)',\n", " #'Pop 25+ Associate degree (2019A)',\n", " #'Household Income $10000-$14999 (2019A)',\n", " #'Household Income < $10000 (2019A)',\n", " #'Population (2024A)',\n", " #'Median household income (2024A)',\n", " #'Median Age (2024A)',\n", " #'Housing units (2024A)',\n", " #'Average household Income (2024A)',\n", " #'Per capita income (2024A)',\n", " #'Households (2024A)',\n", " #'Households: No Vehicle Available (2019A)',\n", " #'Households: Two or More Vehicles Available (2019A)',\n", " #'Households: One Vehicle Available (2019A)',\n", " #'Unemployment Rate (2019A)',\n", " #'Population male (2019A)',\n", " #'Population female (2019A)',\n", " #'Non Hispanic White (2019A)',\n", " #'Non Hispanic Other Race (2019A)',\n", " #'Non Hispanic Multiple Race (2019A)',\n", " #'Non Hispanic Hawaiian/Pacific Islander (2019A)',\n", " #'Non Hispanic Black (2019A)',\n", " #'Non Hispanic Asian (2019A)',\n", " #'Non Hispanic American Indian (2019A)',\n", " #'Institutional Group Quarters Population (2019A)',\n", " #'Population in Group Quarters (2019A)',\n", " #'Population (2019A)',\n", " #'Widowed (2019A)',\n", " #'Separated (2019A)',\n", " #'Never Married (2019A)',\n", " #'Divorced (2019A)',\n", " #'SPANISH SPEAKING HOUSEHOLDS',\n", " #'LINGUISTICALLY ISOLATED HOUSEHOLDS (NON-ENGLISH SP...',\n", " #'Pop 16+ civilian unemployed (2019A)',\n", " #'Pop 16+ not in labor force (2019A)',\n", " #'Median household income (2019A)',\n", " #'Median family income (2019A)',\n", " #'Average household Income (2019A)',\n", " #'UNITS IN STRUCTURE: 20 OR MORE',\n", " #'UNITS IN STRUCTURE: 1 DETACHED',\n", " #'Median Value of Owner Occupied Housing Units',\n", " #'Population Hispanic (2019A)',\n", " #'Median Household Income: Age 75+ (2019A)',\n", " #'Household Income $150000-$199999 (2019A)',\n", " #'Geographic Identifier',\n", " #'Population In Labor Force (2019A)',\n", " #'Pop 16+ in Armed Forces (2019A)',\n", " #'Housing units (2019A)',\n", " #'Population Age 16+ (2019A)',\n", " #'Pop 16+ civilian employed (2019A)',\n", " #'Per capita income (2019A)',\n", " #'Median Cash Rent',\n", " #'Household Income $30000-$34999 (2019A)',\n", " #'Household Income $50000-$59999 (2019A)',\n", " #'Household Income $100000-$124999 (2019A)',\n", " #'Household Income $75000-$99999 (2019A)',\n", " #'Population age 65-69 (2019A)']" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset = Dataset.get('ags_sociodemogr_e92b1637')\n", "variables = dataset.variables\n", "variables" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we saw in the Data Discovery guide, the `ags_sociodemogr_e92b1637` dataset contains socio-demographic variables aggregated to the Census block group level. \n", "\n", "Let's try and find a variable for total population:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idslugnamedescriptioncolumn_namedb_typedataset_idagg_methodvariable_group_idstarred
55carto-do.ags.demographics_sociodemographic_usa...POPPY_946f4ed6POPPYPopulation (2024A)POPPYFLOATcarto-do.ags.demographics_sociodemographic_usa...SUMNoneFalse
75carto-do.ags.demographics_sociodemographic_usa...POPCYGRPI_147af7a9POPCYGRPIInstitutional Group Quarters Population (2019A)POPCYGRPIINTEGERcarto-do.ags.demographics_sociodemographic_usa...SUMNoneFalse
76carto-do.ags.demographics_sociodemographic_usa...POPCYGRP_74c19673POPCYGRPPopulation in Group Quarters (2019A)POPCYGRPINTEGERcarto-do.ags.demographics_sociodemographic_usa...SUMNoneFalse
77carto-do.ags.demographics_sociodemographic_usa...POPCY_f5800f44POPCYPopulation (2019A)POPCYINTEGERcarto-do.ags.demographics_sociodemographic_usa...SUMNoneFalse
99carto-do.ags.demographics_sociodemographic_usa...LBFCYPOP16_53fa921cLBFCYPOP16Population Age 16+ (2019A)LBFCYPOP16INTEGERcarto-do.ags.demographics_sociodemographic_usa...SUMNoneFalse
\n", "
" ], "text/plain": [ " id slug \\\n", "55 carto-do.ags.demographics_sociodemographic_usa... POPPY_946f4ed6 \n", "75 carto-do.ags.demographics_sociodemographic_usa... POPCYGRPI_147af7a9 \n", "76 carto-do.ags.demographics_sociodemographic_usa... POPCYGRP_74c19673 \n", "77 carto-do.ags.demographics_sociodemographic_usa... POPCY_f5800f44 \n", "99 carto-do.ags.demographics_sociodemographic_usa... LBFCYPOP16_53fa921c \n", "\n", " name description column_name \\\n", "55 POPPY Population (2024A) POPPY \n", "75 POPCYGRPI Institutional Group Quarters Population (2019A) POPCYGRPI \n", "76 POPCYGRP Population in Group Quarters (2019A) POPCYGRP \n", "77 POPCY Population (2019A) POPCY \n", "99 LBFCYPOP16 Population Age 16+ (2019A) LBFCYPOP16 \n", "\n", " db_type dataset_id agg_method \\\n", "55 FLOAT carto-do.ags.demographics_sociodemographic_usa... SUM \n", "75 INTEGER carto-do.ags.demographics_sociodemographic_usa... SUM \n", "76 INTEGER carto-do.ags.demographics_sociodemographic_usa... SUM \n", "77 INTEGER carto-do.ags.demographics_sociodemographic_usa... SUM \n", "99 INTEGER carto-do.ags.demographics_sociodemographic_usa... SUM \n", "\n", " variable_group_id starred \n", "55 None False \n", "75 None False \n", "76 None False \n", "77 None False \n", "99 None False " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vdf = variables.to_dataframe()\n", "vdf[vdf['name'].str.contains('pop', case=False, na=False)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can store the variable instance we need by searching the Catalog by its `slug`, in this case `POPCY_f5800f44`:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'id': 'carto-do.ags.demographics_sociodemographic_usa_blockgroup_2015_yearly_2019.POPCY',\n", " 'slug': 'POPCY_f5800f44',\n", " 'name': 'POPCY',\n", " 'description': 'Population (2019A)',\n", " 'column_name': 'POPCY',\n", " 'db_type': 'INTEGER',\n", " 'dataset_id': 'carto-do.ags.demographics_sociodemographic_usa_blockgroup_2015_yearly_2019',\n", " 'agg_method': 'SUM',\n", " 'variable_group_id': None,\n", " 'starred': False}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "variable = Variable.get('POPCY_f5800f44')\n", "variable.to_dict()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `POPCY` variable contains the `SUM` of the population for blockgroup for the year 2019. Let's enrich our stores DataFrame with that variable." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Enrich a points DataFrame\n", "\n", "In the [CARTOframes Quickstart](/developers/cartoframes/guides/Quickstart/) you learned how to load your own data (in this case Starbucks stores) and geocode the addresses to coordinates for further analysis.\n", "\n", "Let's start by loading those geocoded Starbucks stores:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
cartodb_idfield_1nameaddressrevenuegeometry
010Franklin Ave & Eastern Pkwy341 Eastern Pkwy,Brooklyn, NY 112381321040.772POINT (-73.95901 40.67109)
121607 Brighton Beach Ave607 Brighton Beach Avenue,Brooklyn, NY 112351268080.418POINT (-73.96122 40.57796)
23265th St & 18th Ave6423 18th Avenue,Brooklyn, NY 112041248133.699POINT (-73.98976 40.61912)
343Bay Ridge Pkwy & 3rd Ave7419 3rd Avenue,Brooklyn, NY 112091185702.676POINT (-74.02744 40.63152)
454Caesar's Bay Shopping Center8973 Bay Parkway,Brooklyn, NY 112141148427.411POINT (-74.00098 40.59321)
\n", "
" ], "text/plain": [ " cartodb_id field_1 name \\\n", "0 1 0 Franklin Ave & Eastern Pkwy \n", "1 2 1 607 Brighton Beach Ave \n", "2 3 2 65th St & 18th Ave \n", "3 4 3 Bay Ridge Pkwy & 3rd Ave \n", "4 5 4 Caesar's Bay Shopping Center \n", "\n", " address revenue \\\n", "0 341 Eastern Pkwy,Brooklyn, NY 11238 1321040.772 \n", "1 607 Brighton Beach Avenue,Brooklyn, NY 11235 1268080.418 \n", "2 6423 18th Avenue,Brooklyn, NY 11204 1248133.699 \n", "3 7419 3rd Avenue,Brooklyn, NY 11209 1185702.676 \n", "4 8973 Bay Parkway,Brooklyn, NY 11214 1148427.411 \n", "\n", " geometry \n", "0 POINT (-73.95901 40.67109) \n", "1 POINT (-73.96122 40.57796) \n", "2 POINT (-73.98976 40.61912) \n", "3 POINT (-74.02744 40.63152) \n", "4 POINT (-74.00098 40.59321) " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from geopandas import read_file\n", "stores_gdf = read_file('http://libs.cartocdn.com/cartoframes/files/starbucks_brooklyn_geocoded.geojson')\n", "stores_gdf.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note: Alternatively, you can load data in any geospatial format supported by GeoPandas or CARTO.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, for each store we have its name, address, the total revenue by year and a `geometry` column indicating the location of the store. This is important because for the enrichment service to work, we need a DataFrame with a geometry column encoded as a [shapely](https://pypi.org/project/Shapely/) object.\n", "\n", "We can now create a new `Enrichment` instance, and since the `stores_gdf` dataset represents store locations (points), we can use the `enrich_points` function passing as arguments, the stores DataFrame and a list of `Variables` (that we have a valid subscription from the Data Observatory catalog for).\n", "\n", "In this case we are only enriching one variable (the total population), but we could enrich a list of them." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
cartodb_idfield_1nameaddressrevenuegeometryPOPCYdo_area
010Franklin Ave & Eastern Pkwy341 Eastern Pkwy,Brooklyn, NY 112381321040.772POINT (-73.95901 40.67109)221559840.196748
121607 Brighton Beach Ave607 Brighton Beach Avenue,Brooklyn, NY 112351268080.418POINT (-73.96122 40.57796)183160150.636995
23265th St & 18th Ave6423 18th Avenue,Brooklyn, NY 112041248133.699POINT (-73.98976 40.61912)74538950.618837
343Bay Ridge Pkwy & 3rd Ave7419 3rd Avenue,Brooklyn, NY 112091185702.676POINT (-74.02744 40.63152)117457353.293114
454Caesar's Bay Shopping Center8973 Bay Parkway,Brooklyn, NY 112141148427.411POINT (-74.00098 40.59321)2289188379.242640
\n", "
" ], "text/plain": [ " cartodb_id field_1 name \\\n", "0 1 0 Franklin Ave & Eastern Pkwy \n", "1 2 1 607 Brighton Beach Ave \n", "2 3 2 65th St & 18th Ave \n", "3 4 3 Bay Ridge Pkwy & 3rd Ave \n", "4 5 4 Caesar's Bay Shopping Center \n", "\n", " address revenue \\\n", "0 341 Eastern Pkwy,Brooklyn, NY 11238 1321040.772 \n", "1 607 Brighton Beach Avenue,Brooklyn, NY 11235 1268080.418 \n", "2 6423 18th Avenue,Brooklyn, NY 11204 1248133.699 \n", "3 7419 3rd Avenue,Brooklyn, NY 11209 1185702.676 \n", "4 8973 Bay Parkway,Brooklyn, NY 11214 1148427.411 \n", "\n", " geometry POPCY do_area \n", "0 POINT (-73.95901 40.67109) 2215 59840.196748 \n", "1 POINT (-73.96122 40.57796) 1831 60150.636995 \n", "2 POINT (-73.98976 40.61912) 745 38950.618837 \n", "3 POINT (-74.02744 40.63152) 1174 57353.293114 \n", "4 POINT (-74.00098 40.59321) 2289 188379.242640 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from cartoframes.data.observatory import Enrichment\n", "enriched_stores_gdf = Enrichment().enrich_points(stores_gdf, [variable])\n", "enriched_stores_gdf.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once the enrichment finishes, there is a new column in our DataFrame called `POPCY` with population projected for the year 2019, from the US Census block group which contains each one of our Starbucks stores." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All this information, is available in the `ags_sociodemogr_e92b1637` metadata. Let's take a look:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'id': 'carto-do.ags.demographics_sociodemographic_usa_blockgroup_2015_yearly_2019',\n", " 'slug': 'ags_sociodemogr_e92b1637',\n", " 'name': 'Sociodemographic',\n", " 'description': 'Census and ACS sociodemographic data estimated for the current year and data projected to five years. Projected fields are general aggregates (total population, total households, median age, avg income etc.)',\n", " 'country_id': 'usa',\n", " 'geography_id': 'carto-do-public-data.usa_carto.geography_usa_blockgroup_2015',\n", " 'geography_name': 'Census Block Groups (2015) - shoreline clipped',\n", " 'geography_description': 'Shoreline clipped TIGER/Line boundaries. More info: https://carto.com/blog/tiger-shoreline-clip/',\n", " 'category_id': 'demographics',\n", " 'category_name': 'Demographics',\n", " 'provider_id': 'ags',\n", " 'provider_name': 'Applied Geographic Solutions',\n", " 'data_source_id': 'sociodemographic',\n", " 'lang': 'eng',\n", " 'temporal_aggregation': 'yearly',\n", " 'time_coverage': '[2019-01-01,2020-01-01)',\n", " 'update_frequency': None,\n", " 'version': '2019',\n", " 'is_public_data': False}" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.to_dict()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Enrich a polygon DataFrame\n", "\n", "Next, let's do a second enrichment, but this time using a DataFrame with areas of influence calculated using the [CARTOframes isochrones](/developers/cartoframes/reference/#heading-Isolines) service to obtain the polygon around each store that covers the area within an 8, 17 and 25 minute walk." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
data_rangelower_data_rangerange_labelgeometry
050008 min.MULTIPOLYGON (((-73.95959 40.67571, -73.95971 ...
1100050017 min.POLYGON ((-73.95988 40.68110, -73.95863 40.681...
21500100025 min.POLYGON ((-73.95986 40.68815, -73.95711 40.688...
350008 min.MULTIPOLYGON (((-73.96185 40.58321, -73.96231 ...
4100050017 min.MULTIPOLYGON (((-73.96684 40.57483, -73.96830 ...
\n", "
" ], "text/plain": [ " data_range lower_data_range range_label \\\n", "0 500 0 8 min. \n", "1 1000 500 17 min. \n", "2 1500 1000 25 min. \n", "3 500 0 8 min. \n", "4 1000 500 17 min. \n", "\n", " geometry \n", "0 MULTIPOLYGON (((-73.95959 40.67571, -73.95971 ... \n", "1 POLYGON ((-73.95988 40.68110, -73.95863 40.681... \n", "2 POLYGON ((-73.95986 40.68815, -73.95711 40.688... \n", "3 MULTIPOLYGON (((-73.96185 40.58321, -73.96231 ... \n", "4 MULTIPOLYGON (((-73.96684 40.57483, -73.96830 ... " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "aoi_gdf = read_file('http://libs.cartocdn.com/cartoframes/files/starbucks_brooklyn_isolines.geojson')\n", "aoi_gdf.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this case we have a DataFrame which, for each index in the `stores_gdf`, contains a polygon of the areas of influence around each store at 8, 17 and 25 minute walking intervals. Again the `geometry` is encoded as a `shapely` object.\n", "\n", "In this case, the `Enrichment` service provides an `enrich_polygons` function, which in its basic version, works in the same way as the `enrich_points` function. It just needs a DataFrame with polygon geometries and a list of variables to enrich:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
data_rangelower_data_rangerange_labelgeometryPOPCY
050008 min.MULTIPOLYGON (((-73.95959 40.67571, -73.95971 ...21112.458330
1100050017 min.POLYGON ((-73.95988 40.68110, -73.95863 40.681...60157.083967
21500100025 min.POLYGON ((-73.95986 40.68815, -73.95711 40.688...110657.471723
350008 min.MULTIPOLYGON (((-73.96185 40.58321, -73.96231 ...23505.104589
4100050017 min.MULTIPOLYGON (((-73.96684 40.57483, -73.96830 ...29781.046917
\n", "
" ], "text/plain": [ " data_range lower_data_range range_label \\\n", "0 500 0 8 min. \n", "1 1000 500 17 min. \n", "2 1500 1000 25 min. \n", "3 500 0 8 min. \n", "4 1000 500 17 min. \n", "\n", " geometry POPCY \n", "0 MULTIPOLYGON (((-73.95959 40.67571, -73.95971 ... 21112.458330 \n", "1 POLYGON ((-73.95988 40.68110, -73.95863 40.681... 60157.083967 \n", "2 POLYGON ((-73.95986 40.68815, -73.95711 40.688... 110657.471723 \n", "3 MULTIPOLYGON (((-73.96185 40.58321, -73.96231 ... 23505.104589 \n", "4 MULTIPOLYGON (((-73.96684 40.57483, -73.96830 ... 29781.046917 " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from cartoframes.data.observatory import Enrichment\n", "enriched_aoi_gdf = Enrichment().enrich_polygons(aoi_gdf, [variable])\n", "enriched_aoi_gdf.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now have a new column in our areas of influence DataFrame, `SUM_POPCY` which represents the `SUM` of total population in the Census block groups that instersect with each polygon in our DataFrame." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### How enrichment works" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a deeper look into what happens under the hood when you execute a polygon enrichment.\n", "\n", "Imagine we have polygons representing municipalities, in blue, each of which have a population attribute, and we want to find out the population inside the green circle.\n", "\n", "![Enrichment Schema](img/enrichment/enrichment_01.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We don’t know how the population is distributed inside these municipalities. They are probably concentrated in cities somewhere, but, since we don’t know where they are, our best guess is to assume that the population is evenly distributed in the municipality (i.e. every point inside the municipality has the same population density).\n", "\n", "Population is an extensive property (it grows with area), so we can subset it (a region inside the municipality will always have a smaller population than the whole municipality), and also aggregate it by summing.\n", "\n", "In this case, we’d calculate the population inside each part of the circle that intersects with a municipality." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Default aggregation methods**\n", "\n", "In the Data Observatory, we suggest a default aggregation method for certain fields. However, some fields don’t have a clear best method, and some just can’t be aggregated. In these cases, we leave the `agg_method` field blank and let the user choose the method that best fits their needs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Conclusion\n", "\n", "In this guide you've seen how to use CARTOframes in conjunction with the Data Observatory to enrich a Starbucks dataset with a new population variable for the use case of revenue prediction analysis by:\n", "\n", "- Choosing the total population variable from the Data Observatory catalog\n", "- Calculating the sum of total population for each store\n", "- Calculating the sum of total population around the walking areas of influence around each store\n", "\n", "In addition, you were introduced to some more advanced concepts and further explanation of how the enrichment itself works." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }