{ "cells": [ { "cell_type": "markdown", "id": "b747131b", "metadata": {}, "source": [ "# Getting Started with Interactive Mapping and Data Visualization\n", "#### Prepared for 2021 GIS Day Workshop by Zhiyu Li" ] }, { "cell_type": "markdown", "id": "891d541b", "metadata": {}, "source": [ "## Programming with CyberGIS-Jupyter\n", "\n", "- **Cell**\n", " - Add a cell: single click on a cell --> Menu --> Insert\n", " - Delete a cell: single click on the cell to delete --> Menu --> Edit --> Delete Cells\n", " - Reorder a cell: click on a cell --> up arrow or down arrow\n", " - Change cell type: single click on a cell ---> Menu --> Cell Type\n", " - Edit cell\n", " - Markdown: double click ([basic syntax reference](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Working%20With%20Markdown%20Cells.html))\n", " - Code: single click \n", " - Run a cell: single click on a cell --> Run (or Shift + Enter)\n", " - Run all cells: Menu --> Cell --> Run All\n", " - Clear all cell output: Menu --> Cell --> All Output --> Clear\n", "- **Kernel**\n", " - Change kernel: Menu --> Kernel --> Change kernel\n", " - Choose versioned kernel (eg XXXXX-0.8.0) before sharing\n", " - Restart kernel: Menu --> Kernel --> Restart (& Clear output)\n", "- **See all notebooks (Tree View) on CyberGIS-Jupyter**\n", " - On CyberGIS-Hub page: click on \"Launch CyberGISX\" at the upper-right corner\n", " - On an opened notebook page: remove notebook filename (XXXX.ipynb) from browser address bar \n", "- **Troubleshooting**\n", " - Restart kernel\n", " - Restart CyberGIS-Jupyter (save notebook first!): Control Panel --> Stop My Server --> Start Server\n", " - Bug Report button\n", " - Announcement area (maintenance plan, release notes)\n", "- **More Info**\n", " - [\"Getting Started\" series notebooks on CyberGISX](https://cybergisxhub.cigi.illinois.edu/notebooks/)\n", " - [\"Jupyter Notbooks\" section in Knowledge Base](https://cybergisxhub.cigi.illinois.edu/knowledge-base/)\n", "- **Try it out now**" ] }, { "cell_type": "markdown", "id": "9791cedf", "metadata": {}, "source": [ "#### Task 1: Add one new code cell after this cell\n", "Hint: Single click on this cell --> go to Menu --> Insert --> Insert Cell Below" ] }, { "cell_type": "markdown", "id": "8d27abe2", "metadata": {}, "source": [ "**Task 2: Change the new cell's type to Markdown, and write something in it**\n", "\n", "Hint: Single click on the new cell --> go to Menu --> Cell --> Cell Type --> Markdown; Single click on the new cell ---> write some Markdowns see [basic syntax reference](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Working%20With%20Markdown%20Cells.html); Click on the \"Run\" button on the tool bar;" ] }, { "cell_type": "markdown", "id": "15524fe4", "metadata": {}, "source": [ "#### Task 3: Uncomment the Python codes in the cell below and run it\n", "Hint: Single click on the cell below --> Remove the Pound sign ('#') --> press Shift + Enter keys together (or click on the 'Run\" button on the tool bar)" ] }, { "cell_type": "code", "execution_count": 23, "id": "9416a56a", "metadata": {}, "outputs": [], "source": [ "#print(\"hello world\")" ] }, { "cell_type": "markdown", "id": "2dbe84cd", "metadata": {}, "source": [ "#### Task 4: clean all output of this notebook\n", "Hint: go to Menu --> Cell --> All Output --> Clear" ] }, { "cell_type": "markdown", "id": "334c3f80", "metadata": {}, "source": [ "##### ------------- Below is the main section of the Mapping and Visualization notebok -----------------" ] }, { "cell_type": "markdown", "id": "e20e967c", "metadata": {}, "source": [ "\n", "## Introduction\n", "This notebook will walk you through some basic techniques of conducting Interative Mapping and Data Visualization in the CyberGIS-Jupyter environment. We will retrieve the latest COVID-19 data from the [Illinois Department of Public Health](https://dph.illinois.gov/covid19/data/data-portal.html) (IDPH) website, examine and preprocess the data, make plots of daily new cases by counties using [matplotlib](https://matplotlib.org/), create an interactive map with [ipyleaflet](https://ipyleaflet.readthedocs.io/en/latest/) to visualize weekly channge of new cases across the state, and finally link map and plot together. \n", "\n", "After finishing this notebook, you will have a \"app-style\" notebook like the screenshot below. You are encouraged to tweak the codes a little bit to visualize other COVID-19 indices, such as deaths and testings. \n", "\n", "" ] }, { "cell_type": "markdown", "id": "a14ae327", "metadata": {}, "source": [ "## Notebook Outline:\n", "- [Introduction](#intro)\n", "- [Setup](#setup)\n", "- [Data retrieval and preprocessing](#data_processing)\n", " - [Download data from IDPH website](#download_data)\n", " - [Examine raw data](examine_data)\n", "- [Plot time series data for a county](#time_series)\n", "- [Visualize weekly change of new cases at county level across the state](#visualize_weekly_change_rate)\n", " - [Calcuate weekly change rate](#calculate_weekly_change_rate)\n", " - [Data classification and Colormap](#data_classification)\n", " - [Create a Choropleth Map with ipyLeaflet](#choropleth_map)\n", "- [App-style Interactive Map](#app_style_map)\n", "- [Using this Metric Alone May Mislead You](#mislead)" ] }, { "cell_type": "markdown", "id": "8383290f", "metadata": {}, "source": [ "\n", "## Setup\n", "This cell is to import required modules and libs. A breif description on the purpose of each libs can be found below:\n", "\n", " - [json](https://docs.python.org/3/library/json.html) - standard Python module for JSON format I/O operations\n", " - [wget]() - for downloading files from URLs\n", " - [numpy](https://numpy.org/) - for handling N-dimentional arrays and numerical computing\n", " - [pandas](https://pandas.pydata.org/) - for tabular data analysis and manipulation\n", " - [ipyleaflet](https://ipyleaflet.readthedocs.io/en/latest/) - for interactive mapping in Jupyter Notebook environment\n", " - [branca](https://python-visualization.github.io/branca/colormap.html) - for dealing with colormaps\n", " - [matplotlib]() - for creating plots and figures " ] }, { "cell_type": "code", "execution_count": 24, "id": "3d4d4a60", "metadata": {}, "outputs": [], "source": [ "import json\n", "import wget\n", "import numpy as np\n", "import pandas as pd\n", "import geopandas as gpd\n", "import ipyleaflet\n", "from branca import colormap\n", "import matplotlib.pyplot as plt\n", "\n", "# for interactive charting\n", "%matplotlib notebook" ] }, { "cell_type": "markdown", "id": "b968cfa3", "metadata": {}, "source": [ "\n", "## Data retrieval and preprocessing" ] }, { "cell_type": "markdown", "id": "dd07d62d", "metadata": {}, "source": [ "The [Illinois Department of Public Health](https://dph.illinois.gov/) has a [COVID-19 data portal](https://dph.illinois.gov/covid19/data/data-portal.html) that provides different metrics and data, including \"county-level histrorical cases, deaths and tested\", \"hospitalization data\", \"vaccine administration data\", \"zipcode-level cases and tested data\" and others. As noted on the website, data avaialbility, update frequency, data format and metrics reported are subject to change.\n", "\n", "Since IDPH may choose to stop updating the data from their website or change the format in the future, we downloaded the data as of Nov 16, 2021 and saved it (filename: idph_counties_Nov16_2021.csv) alongside the notebook for archive. You can also programmatically download the latest \"county-level histrorical cases, deaths and tested\" data on the fly by uncommenting some codes in the cell below." ] }, { "cell_type": "markdown", "id": "0f6bb72d", "metadata": {}, "source": [ "\n", "### Download data from IDPH website" ] }, { "cell_type": "code", "execution_count": 25, "id": "116c2241", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using data at ./idph_counties.csv\n" ] } ], "source": [ "# By default, use data downloaded from IDPH as of Nov 16, 2021\n", "idph_counties_csv = \"./data/idph_counties_Nov16_2021.csv\"\n", "\n", "## Uncomment the 2 lines below to download latest data from IDPH on the fly\n", "## IDPH may have removed or changed the data format after this notebook was developed\n", "## So in that case you would need to change codes accordingly\n", "## see: https://dph.illinois.gov/covid19/data/data-portal/all-county-historical-snapshot.html\n", "\n", "idph_counties_url = \"https://idph.illinois.gov/DPHPublicInformation/api/COVIDExport/GetSnapshotHistorical?format=csv\"\n", "idph_counties_csv = wget.download(idph_counties_url, out=\"./idph_counties.csv\")\n", "\n", "print(\"Using data at {}\".format(idph_counties_csv))" ] }, { "cell_type": "markdown", "id": "ec168d80", "metadata": {}, "source": [ "\n", "### Examine raw data\n", "The raw data is a csv file. We load it into a pandas dataframe (the 1st row is ignored as it is the title line). " ] }, { "cell_type": "code", "execution_count": 26, "id": "3b80bb39", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | CountyName | \n", "CumulativeCases | \n", "CasesChange | \n", "TotalTested | \n", "TotalTestedChange | \n", "Deaths | \n", "DeathsChange | \n", "ReportDate | \n", "
---|---|---|---|---|---|---|---|---|
0 | \n", "Adams | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "2020-03-17 | \n", "
1 | \n", "Alexander | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "2020-03-17 | \n", "
2 | \n", "Bond | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "2020-03-17 | \n", "
3 | \n", "Boone | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "2020-03-17 | \n", "
4 | \n", "Brown | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "2020-03-17 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
78827 | \n", "Whiteside | \n", "15376 | \n", "1 | \n", "191353 | \n", "272 | \n", "225 | \n", "0 | \n", "2022-04-14 | \n", "
78828 | \n", "Will | \n", "162799 | \n", "180 | \n", "2371004 | \n", "3814 | \n", "1530 | \n", "0 | \n", "2022-04-14 | \n", "
78829 | \n", "Williamson | \n", "22583 | \n", "7 | \n", "263413 | \n", "360 | \n", "249 | \n", "0 | \n", "2022-04-14 | \n", "
78830 | \n", "Winnebago | \n", "76629 | \n", "32 | \n", "973502 | \n", "1261 | \n", "819 | \n", "0 | \n", "2022-04-14 | \n", "
78831 | \n", "Woodford | \n", "10397 | \n", "11 | \n", "157757 | \n", "179 | \n", "111 | \n", "0 | \n", "2022-04-14 | \n", "
78832 rows × 8 columns
\n", "\n", " | CountyName | \n", "CumulativeCases | \n", "CasesChange | \n", "TotalTested | \n", "TotalTestedChange | \n", "Deaths | \n", "DeathsChange | \n", "
---|---|---|---|---|---|---|---|
ReportDate | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
2020-03-17 | \n", "Chicago | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
2020-03-18 | \n", "Chicago | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
2020-03-19 | \n", "Chicago | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
2020-03-20 | \n", "Chicago | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
2020-03-21 | \n", "Chicago | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
2022-04-10 | \n", "Chicago | \n", "569847 | \n", "234 | \n", "11600646 | \n", "7311 | \n", "7338 | \n", "0 | \n", "
2022-04-11 | \n", "Chicago | \n", "570277 | \n", "430 | \n", "11602904 | \n", "2258 | \n", "7338 | \n", "0 | \n", "
2022-04-12 | \n", "Chicago | \n", "570680 | \n", "403 | \n", "11611165 | \n", "8261 | \n", "7338 | \n", "0 | \n", "
2022-04-13 | \n", "Chicago | \n", "571214 | \n", "534 | \n", "11621370 | \n", "10205 | \n", "7340 | \n", "2 | \n", "
2022-04-14 | \n", "Chicago | \n", "571988 | \n", "774 | \n", "11634853 | \n", "13483 | \n", "7340 | \n", "0 | \n", "
758 rows × 7 columns
\n", "\n", " | Current Week | \n", "Last Week | \n", "
---|---|---|
CountyName | \n", "\n", " | \n", " |
Scott | \n", "4 | \n", "0 | \n", "
Pope | \n", "1 | \n", "3 | \n", "
Gallatin | \n", "3 | \n", "3 | \n", "
Pulaski | \n", "5 | \n", "5 | \n", "
Stark | \n", "15 | \n", "6 | \n", "
... | \n", "... | \n", "... | \n", "
Will | \n", "1510 | \n", "1068 | \n", "
DuPage | \n", "1712 | \n", "1371 | \n", "
Chicago | \n", "3682 | \n", "2717 | \n", "
Cook | \n", "4199 | \n", "2986 | \n", "
Illinois | \n", "24923 | \n", "18559 | \n", "
104 rows × 2 columns
\n", "\n", " | Change_Rate | \n", "
---|---|
CountyName | \n", "\n", " |
Brown | \n", "-0.833333 | \n", "
Hamilton | \n", "-0.666667 | \n", "
Pope | \n", "-0.666667 | \n", "
Henderson | \n", "-0.416667 | \n", "
Mason | \n", "-0.391304 | \n", "
... | \n", "... | \n", "
Menard | \n", "1.259259 | \n", "
Stark | \n", "1.500000 | \n", "
Pike | \n", "1.526316 | \n", "
McDonough | \n", "1.760000 | \n", "
Scott | \n", "inf | \n", "
104 rows × 1 columns
\n", "\n", " | Class | \n", "
---|---|
CountyName | \n", "\n", " |
Adams | \n", "1 | \n", "
Alexander | \n", "2 | \n", "
Bond | \n", "4 | \n", "
Boone | \n", "1 | \n", "
Brown | \n", "5 | \n", "
... | \n", "... | \n", "
Whiteside | \n", "2 | \n", "
Will | \n", "2 | \n", "
Williamson | \n", "2 | \n", "
Winnebago | \n", "2 | \n", "
Woodford | \n", "1 | \n", "
104 rows × 1 columns
\n", "\n", " | id | \n", "geometry | \n", "
---|---|---|
0 | \n", "Chicago | \n", "MULTIPOLYGON (((-87.93514 42.00089, -87.93521 ... | \n", "
1 | \n", "McHenry | \n", "POLYGON ((-88.70742 42.49352, -88.70741 42.493... | \n", "
2 | \n", "Boone | \n", "POLYGON ((-88.70742 42.49352, -88.70750 42.493... | \n", "
3 | \n", "Ogle | \n", "POLYGON ((-89.68809 42.19950, -89.68807 42.184... | \n", "
4 | \n", "Will | \n", "POLYGON ((-88.26146 41.72439, -88.26103 41.708... | \n", "
... | \n", "... | \n", "... | \n", "
98 | \n", "Lawrence | \n", "POLYGON ((-87.90806 38.85013, -87.90819 38.835... | \n", "
99 | \n", "Marion | \n", "POLYGON ((-89.13844 38.73633, -89.13847 38.721... | \n", "
100 | \n", "Union | \n", "POLYGON ((-89.04143 37.59650, -89.06017 37.597... | \n", "
101 | \n", "Pope | \n", "POLYGON ((-88.70860 37.59926, -88.70876 37.584... | \n", "
102 | \n", "Cook | \n", "MULTIPOLYGON (((-87.93993 41.99344, -87.94006 ... | \n", "
103 rows × 2 columns
\n", "