{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Spatial exploratory data analysis\n",
"\n",
"In the previous lessons, we learnt about several ways of analyzing data visually such as using bar plots, scatter plots, and categorical plots. When our datsset consists of geospatial information such as zipcodes, states, and geographical coordinates, we can further explore by overlaying the data on top of spatial maps. In this lesson, you will learn to perform simple visual analysis of spatial data with Pandas and `folium` library using on-time performance data of dometics flights. The data forthis exercise was downloaded from Kaggle. We will explore the data to understand if there a geospatial pattern in the data using visualization. \n",
"\n",
"The U.S. Department of Transportation's (DOT) Bureau of Transportation Statistics tracks the on-time performance of domestic flights operated by large air carriers. Summary information on the number of on-time, delayed, canceled, and diverted flights is published in DOT's monthly Air Travel Consumer Report and in this dataset of 2015 flight delays and cancellations."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What is folium\n",
"\n",
"Folium is a Python library that makes it easier to create maps using Leaflet, which is an open source javascript library for creating interactive maps. Folium map can be used for a range of purposes from simple visualization to creating interactive dashboard applications.\n",
"\n",
"To install folium with pip:\n",
"```python \n",
"pip install folium\n",
"```\n",
"or with conda:\n",
"\n",
"```python\n",
"conda install -c conda-forge folium\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Vizualize the number of flight by airports"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Read data into pandas dataframe"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import pandas as pd\n",
"import folium"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"DATA_DIR = \"/home/asimbanskota/t81_577_data_science/weekly_materials/week7/files\"\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"airlines = os.path.join(DATA_DIR, 'airlines.csv')\n",
"airports = os.path.join(DATA_DIR, 'airports.csv')\n",
"flights = os.path.join(DATA_DIR, 'flights.csv')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/asimbanskota/anaconda3/envs/api/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3062: DtypeWarning: Columns (7,8) have mixed types.Specify dtype option on import or set low_memory=False.\n",
" has_raised = await self.run_ast_nodes(code_ast.body, cell_name,\n"
]
}
],
"source": [
"df_air = pd.read_csv(airlines)\n",
"df_ap = pd.read_csv(airports)\n",
"df = pd.read_csv(flights)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lets create a feature that keeps the value of total number of flights by airport."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"count_flights = df['ORIGIN_AIRPORT'].value_counts().reset_index()\n",
"count_flights.rename({'index': 'IATA_CODE', 'ORIGIN_AIRPORT': 'count_flights'}, axis = 1, inplace = True)\n",
"df_ap = df_ap.merge(count_flights, on = 'IATA_CODE')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Maps are defined as a folium.Map object. We start with creating a base map by providing the latitude and logitude of the center of the map. This will instantiate a map object for a given location ( 45 degree lat and 96 degree west). Once the base map is created, other map objects can be incrementally added on top of the folium.Map. \n"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [],
"source": [
"m = folium.Map(location=[45, -96], zoom_start =4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" We can access and display the map object within the notebook simply by referring to its name `m`. "
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since we have geographical latitude and longitude associated with the airports, we can simply plot the locations on top of the map object as follows.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before that, we need to remove NaN values from the location columns. We might also need to filter out airports with relatively lower number of flights to avoid visual clutter.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"df_ap = df_ap.dropna(subset = ['LATITUDE','LONGITUDE'])\n",
"df_ap = df_ap[df_ap['count_flights'] > 5000]"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"for lat, lon, ct in zip(df_ap['LATITUDE'], df_ap['LONGITUDE'], df_ap['count_flights']):\n",
" folium.CircleMarker(\n",
" [lat, lon],\n",
" \n",
" popup = ('Count_flights: ' + str(ct)\n",
" ),\n",
" \n",
" key_on = ct,\n",
" \n",
" ).add_to(m)\n",
"m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above map is neither pretty nor intuitive. May be we can color code the circles with each color representing the some ranges of flights departed from the airport. Lets first use Pandas Cut function to bin the flight counts in four equally sized buckets."
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [],
"source": [
"\n",
"df_ap['count_cl'] = pd.qcut(df_ap['count_flights'],4, labels=False)\n",
"colordict = {0: 'lightblue', 1: 'lightgreen', 2: 'orange', 3: 'red'}"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"for lat, lon, count_cl, count_flights in zip(df_ap['LATITUDE'], df_ap['LONGITUDE'], df_ap['count_cl'], df_ap['count_flights']):\n",
" folium.CircleMarker(\n",
" [lat, lon],\n",
" popup = ('Count_cl: ' + str(count_cl)\n",
" ),\n",
" color='b',\n",
" key_on = count_cl,\n",
" #threshold_scale=[0,1,2,3],\n",
" fill_color=colordict[count_cl],\n",
" fill=True,\n",
" fill_opacity=0.7,\n",
" legend_name='SALE PRICE'\n",
" ).add_to(m)\n",
"m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above shows the values of your interest in terms of categories. You can go one step futher and display the size of each circle relative to the number of flights departed from the respective airports."
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"latitude = 45\n",
"longitude = -96\n",
"m = folium.Map(location=[latitude, longitude], zoom_start=4)\n",
"\n",
"for lat, lon, count_cl, count_flights in zip(df_ap['LATITUDE'], df_ap['LONGITUDE'], df_ap['count_cl'], df_ap['count_flights']):\n",
" folium.CircleMarker(\n",
" [lat, lon],\n",
" radius= count_flights/25000,\n",
" popup = ('Count_cl: ' + str(count_cl)\n",
" ),\n",
" color='b',\n",
" key_on = count_cl,\n",
" threshold_scale=[0,1,2,3],\n",
" fill_color= 'crimson',\n",
" fill=True,\n",
" fill_opacity=0.7\n",
" ).add_to(m)\n",
"m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the similar vain, you can analyze other appropriate features in the data as well."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Choropleth map\n",
"\n",
"Chlopleth map is a thematic map that uses differences in shading and coloring to indicate the corresponding values of interest."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to create a chloropleth map, we need to have a dataset that defines the boundary of geographical units of our interest such as US states, zipcodes etc. Such geographical file can come with various formats, one of them is geoJSON, whihc is nothing other that a JSON formatted data with additional geographical information. The geoJSON file for the US states can be obtained from the URL below."
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"execution_count": 66,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"url = 'https://raw.githubusercontent.com/python-visualization/folium/master/examples/data'\n",
"state_geo = f'{url}/us-states.json'\n",
"\n",
"df_geo = pd.read_json(state_geo)\n",
"#To create a base map, simply pass your starting coordinates to Folium:\n",
"\n",
"m = folium.Map(location=[48, -102], zoom_start=3)\n",
"\n",
"folium.Choropleth(\n",
" geo_data=state_geo,\n",
" name='choropleth',\n",
" data=dfs,\n",
" columns=['STATE', 'count_flights'],\n",
" key_on='feature.id',\n",
" fill_color='YlGn',\n",
" fill_opacity=0.7,\n",
" line_opacity=0.2,\n",
" legend_name='Flight_counts'\n",
").add_to(m)\n",
"\n",
"folium.LayerControl().add_to(m)\n",
"m"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.1"
}
},
"nbformat": 4,
"nbformat_minor": 4
}