![](\"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/doc/source/images/Harvey.png\")
\n",
"\n",
"NASA TERRA MODIS INFRARED IMAGE OF HARVEY AT 0419 UTC 26 AUGUST 2017 JUST AFTER LANDFALL AS A\n",
"CATEGORY 4 HURRICANE IN TEXAS. IMAGE COURTESY OF UW/CIMSS.\n",
"\n",
"## A Jupyter Notebook used to visualize data from the Houston Flood of 2017 running on IBM Watson Studio\n",
"\n",
"### [Jupyter Notebook]( www.jupyter.org)\n",
"An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.\n",
"\n",
"### [IBM Watson Studio](https://www.ibm.com/cloud/watson-studio)\n",
"Build and train AI & machine learning models, prepare and analyze data – all in a flexible, hybrid cloud environment\u000b",
"IBM Watson Studio provides tools for data scientists, application developers and subject matter experts to collaboratively and easily work with data to build and train models at scale. It gives you the flexibility to build models where your data resides and deploy anywhere in a hybrid environment so you can operationalize data science faster.\n",
"\n",
"### [Python](https://www.python.org/)\n",
"Python is a programming language that lets you work quickly\n",
"and integrate systems more effectively.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Contents\n",
"[1.0 Install dependencies and import packages](#install) \n",
"[2.0 Obtain and curate data](#obtain) \n",
"[3.0 Create Pixie App](#create) \n",
"[4.0 Use Folium for mapping](#folium) \n",
"[5.0 Explore more tools](#more) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## 1.0 Install dependencies and import packages"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.1 Install pixiedust\n",
"\n",
"We install the prerequisites using the `!pip install` syntax here.\n",
"In some cases, running pip install from a notebook may require a one-time kernel restart. Check the output for messages.\n",
"If instructed to restart the kernel, from the notebook toolbar menu: • Go to > Kernel > Restart • Click Restart in the confirmation dialog\n",
"> Note: The status of the kernel briefly flashes near the upper right corner, alerting when it is Not Connected, Restarting, Ready, etc. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install --upgrade pixiedust"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 Import the packages\n",
"Numpy is a package for scientific computing in Python. \n",
"Pandas helps with data structures. \n",
"[Pixiedust](https://pixiedust.github.io/pixiedust/) is a Python helper library for Jupyter notebooks."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"import pixiedust"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## 2.0 Obtain and curate data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to do data science, or data engineering, we'll need some data.\n",
"So, what problem are we trying to solve?\n",
"\n",
"### Where in Houston does flooding occur, and which specific adresses are vulnerable?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.1 Search for data\n",
"We are interested in the flooding in Houston on August, 2017, as a result of Hurricane Harvey.\n",
"Using a search engine, we can try `houston flood 2017 data`. \n",
"That gave many results, mostly news stories. But there are some promising sites, many from US government agencies: \n",
"* https://www.weather.gov\n",
"* https://txpub.usgs.gov/floodwatch/#page-top\n",
"* https://water.weather.gov/ahps2/index.php?wfo=hgx\n",
"* https://www.nhc.noaa.gov/data/tcr/AL092017_Harvey.pdf\n",
"* https://stn.wim.usgs.gov/STNDataPortal/#\n",
"* https://streamstats.usgs.gov/ss/\n",
"\n",
"That's just the beginning, but hopefully it can lead to some data we can use."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2 Download and examine data\n",
"We'll look at some data from stream gauges in Houston, gathering historical data from the dates 8/23/2017 - 8/31/2017. \n",
"I've grabbed the data for one stream gauge in the Houston Beltway area, Hunting Bayou, from the [waterdata.usgs.gov](https://waterdata.usgs.gov/nwis/nwismap/?site_no=08075763&agency_cd=USGS) site."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!wget https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/data/HuntingBayou.csv\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 2.3 First, let's look at the header to the file (which I've peeked at in an editor). This gives us some info on the contents:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! head -n 35 HuntingBayou.csv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 2.4 Look at the pandas dataframe \n",
"I see from the header that I need to skip the first 29 rows (rows 0-28), that the data is tab separated, and the header begins on the 30th line (line 29)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv('HuntingBayou.csv',sep='\\t',skiprows=(0-28),header=(29))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### 2.5 Do some data frame cleanup\n",
"What's with the extra fields on line 0? Let's delete that line."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = df.drop(0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can rename those obscure column names, to give them a name that represents the data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.rename(columns={'140488_00060': 'Discharge(cfs)', '140489_00065': 'GuageHeight(feet)'}, inplace=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's replace the `site_no` with a `site_name` for `HuntingBayou.csv`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.rename(columns={'site_no': 'site_name'}, inplace=True)\n",
"df['site_name'].replace(\"08075770\", \"HuntingBayou.csv\", inplace=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We don't care about the first `agency_cd` column, which is `USGS`, so let's drop it. The same goes for the `140488_00060_cd`, `140489_00065_cd`, and `tz_cd` columns. The `axis=1` denotes that this is a column."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.drop(['agency_cd', '140488_00060_cd', '140489_00065_cd', 'tz_cd'], axis=1, inplace=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our `Discharge` and `GuageHeight` data are strings, but we need them to be floats. We'll convert them"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['GuageHeight(feet)'] = df['GuageHeight(feet)'].convert_objects(convert_numeric=True)\n",
"df['Discharge(cfs)'] = df['Discharge(cfs)'].convert_objects(convert_numeric=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our datetime has the year 2017 for all fields, but our graphs will be cleaner if we skip it"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['datetime'] = df['datetime'].map(lambda x: x.lstrip('2017-'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And finally, we can get the latitude and longitude for this stream gauge at Hunting Bayou, so let's add a couple columnns with that data, to allow this to show up on a map.\n",
"We'll need that lat/long in decimal, so [find a tool](https://www.latlong.net/degrees-minutes-seconds-to-decimal-degrees) for that as well."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['latitude']='29.808611'\n",
"df['longitude']='-95.313056'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can see our results."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.6 Use Matplotlib to visualize data\n",
"[Matplotlib](https://matplotlib.org/) is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. Matplotlib can be used in Python scripts, the Python and IPython shells, the Jupyter notebook, web application servers, and four graphical user interface toolkits.\n",
"\n",
"We'll use [the `inline` backend](https://ipython.readthedocs.io/en/stable/interactive/plotting.html) to plot the graph in the notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"import matplotlib as mpl\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Plot the Discharge against time"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# setup line graph\n",
"plt.plot(df['datetime'],df['Discharge(cfs)'])\n",
"plt.title('Houston Flood discharge at Hunting Bayou stream gauge')\n",
"plt.ylabel('Discharge(cfs)')\n",
"plt.xlabel('datetime')\n",
"ax = plt.gca()\n",
"df.set_index('datetime')\n",
"\n",
"# Only label every 20th value\n",
"ticks_to_use = df.index[::100]\n",
"# label ticks per day\n",
"dr = pd.date_range('2017-08-23', periods=9, freq='D')\n",
"\n",
"## Now set the ticks and labels\n",
"ax.set_xticks(ticks_to_use)\n",
"ax.set_xticklabels(dr)\n",
"plt.xticks(rotation='vertical')\n",
"\n",
"plt.show()\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Plot the Gauge Height against time"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# setup line graph\n",
"plt.plot(df['datetime'],df['GuageHeight(feet)'])\n",
"plt.title('Houston Flood Gauge Height at Hunting Bayou stream gauge')\n",
"plt.ylabel('GuageHeight(feet)')\n",
"plt.xlabel('datetime')\n",
"ax = plt.gca()\n",
"df.set_index('datetime')\n",
"\n",
"# Only label every 20th value\n",
"ticks_to_use = df.index[::100]\n",
"# label ticks per day\n",
"dr = pd.date_range('2017-08-23', periods=9, freq='D')\n",
"\n",
"## Now set the ticks and labels\n",
"ax.set_xticks(ticks_to_use)\n",
"ax.set_xticklabels(dr)\n",
"plt.xticks(rotation='vertical')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Plot the stream Gauge Height against time"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.7 Use pixiedust `display()` to explore the schema and browse the data \n",
"\n",
"With ther pixiedust helper library, we can display charts and graphs more easily, with the one-line `display()` method.\n",
" \n",
"#### 2.7.1 Make sure that the `Renderer` is set to `bokeh`:\n",
"\n",
"![](\"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/doc/source/images/choose-renderer-bokeh.png\")
\n",
"\n",
"\n",
"#### 2.7.2 Select _DataFrame Table_ icon in the display widget to see the data in tabular form\n",
"\n",
"![](\"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/doc/source/images/selectDataFrameTableIcon.png\")
\n",
"\n",
"#### 2.7.3 Select the chart icon to pull down and choose `line chart`. Click the `Options` button, and then for `Keys` drag and drop `datetime` and for `Values` drag and drop `Discharge`. This will display the water discharge at this stream gauge in cubic feet per second.\n",
"\n",
"![](\"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/doc/source/images/selectLineChart.png\")
\n",
"\n",
"![](\"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/doc/source/images/chooseDatetimeDischarge.png\")
\n",
"\n",
"\n",
"#### 2.7.4 Click the `Options` button, and then for `Keys` drag and drop `datetime` and for `Values` drag and drop `Gauge_Height`. This will display the height of the water at this stream gauge, in feet.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pixiedust": {
"displayParams": {
"aggregation": "MAX",
"chartsize": "91",
"filter": "{}",
"handlerId": "lineChart",
"keyFields": "datetime",
"no_margin": "true",
"rendererId": "bokeh",
"rowCount": "843",
"timeseries": "true",
"valueFields": "GuageHeight(feet)"
}
},
"scrolled": true
},
"outputs": [],
"source": [
"hunting = df\n",
"display(hunting)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.8 Gather data for Max stream flows\n",
"#### We have already:\n",
"* Found data we can use\n",
"* Cleaned the data\n",
"* Displayed the data\n",
"\n",
"Next, let's aggregate stream Gauge Maximums for `Gauge Height` and `Discharge` so that we can put them on a map.\n",
"This is done offline, manually using a text editor after downloading from various stream gauges, and then put into a single file."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!wget https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/data/maxFlood.csv\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"maxFlood = pd.read_csv('maxFlood.csv')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"maxFlood"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Insert your [Mapbox token](https://www.mapbox.com/) after running the cell below by clicking the `Options` button and adding it to the form.\n",
"\n",
"![](\"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/doc/source/images/map-view-options-add-mapbox-token.png\")
\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pixiedust": {
"displayParams": {
"coloropacity": "100",
"colorrampname": "Green to Purple",
"handlerId": "mapView",
"keyFields": "latitude,longitude",
"kind": "simple",
"mapboxtoken": "",
"rowCount": "843",
"valueFields": "Discharge,Gage_Height,site_name"
}
},
"scrolled": true
},
"outputs": [],
"source": [
"display(maxFlood)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## 3.0 Create Pixie App\n",
"\n",
"### Building the PixieApp Dashboard\n",
"\n",
"#### What you'll need:\n",
"- Mapbox token: A Mapbox token is rquired for the notebook. To get your own visit [Mapbox](https://www.mapbox.com/)\n",
"- Mapbox layers Documentation: [circle](https://www.mapbox.com/mapbox-gl-js/style-spec/#layers-circle), [fill](https://www.mapbox.com/mapbox-gl-js/style-spec/#layers-fill), [symbols](https://www.mapbox.com/mapbox-gl-js/style-spec/#layers-symbol)\n",
"- Mapbox Maki Icons: [https://www.mapbox.com/maki-icons](https://www.mapbox.com/maki-icons)\n",
"- Browse the data from [Huricane Harvey data archives](https://www.dropbox.com/sh/5757a3ujflzdwxo/AAAFD97LMXCRe0YW1HMJDvQ-a?dl=0) to get the GeoJSON url\n",
"- Some understanding of [Jinja2 template](http://jinja.pocoo.org/docs/dev/templates)\n",
"- A Quick read of [PixieApp documentation](https://pixiedust.github.io/pixiedust/pixieapps.html)\n",
"\n",
"### FAQ about the code below:\n",
"- How do I get the pixiedust options in `self.mapJSONOptions`?\n",
"> - Call `display()` on a new cell \n",
"> - Graphically select the options for your chart \n",
"> - Select \"View\"/\"Cell Toolbar\"/\"Edit Metadata\" menu \n",
"> - Click on the “Edit Metadata” button and copy the pixiedust metadata \n",
"- What's the `self.setLayers` call for?\n",
"> This is a method from the MapboxBase class used to specify the custom layer definitions array. \n",
"> The fields are: \n",
"> - name: Layer name \n",
"> - url: geojson url to download the data from\n",
"> - type: (optional) style type e.g Symbol. If not defined, then default value will be infered from geojson geometry\n",
"> - paint: (optional) paint style, see appropriate documentation e.g. [circle](https://www.mapbox.com/mapbox-gl-js/style-spec/#layers-circle) \n",
"> - layout: (optional) layout style, see appropriate documentation e.g. [fill](https://www.mapbox.com/mapbox-gl-js/style-spec/#layers-fill)\n",
"- How do I find new layer data to add?\n",
"> The data layers use geojson format. If you have a `.csv` file, you can use a [csv to geojson converter](https://csv.togeojson.com/) to format correctly.\n",
"- What does the `mainScreen` method do?\n",
"> This is a PixieApp View associated with the default route. See [PixieApp documentation](https://pixiedust.github.io/pixiedust/pixieapps.html) for more information.\n",
"- What's the {{...}} notation in the mainScreen markup for?\n",
"> This is a Jinja2 template notation to call server side Python code. See [Jinja2 template](http://jinja.pocoo.org/docs/dev/templates) for more info\n",
"- How to turn CSV (comma separated values) into geojson?\n",
"> There's [an app](https://csv.togeojson.com/) for that!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!wget https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/data/streamGauges.geojson\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Add your `mapboxtoken` to the PixieApp:\n",
"\n",
"![](\"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/doc/source/images/pixieapp-add-mapbox-token.png\")
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pixiedust": {
"displayParams": {}
}
},
"outputs": [],
"source": [
"from pixiedust.display.app import *\n",
"from pixiedust.apps.mapboxBase import MapboxBase\n",
"\n",
"@PixieApp\n",
"class HoustonDashboard(MapboxBase):\n",
" def setup(self):\n",
" self.mapJSONOptions = {\n",
" \"colorrampname\": \"Green to Purple\",\n",
" \"coloropacity\": \"100\",\n",
" \"handlerId\": \"mapView\",\n",
" \"kind\": \"simple\",\n",
" \"mapboxtoken\": \"\",\n",
" \"keyFields\": \"latitude,longitude\",\n",
" \"valueFields\": \"Gage_Height(feet),Discharge(cfs),date,time\"\n",
" }\n",
" \n",
"\n",
" self.setLayers([\n",
" {\n",
" \"name\": \"Houston Flooded Streets\",\n",
" \"url\": \"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/data/houston.geojson\",\n",
" \"type\": \"LineString\"\n",
" },\n",
" {\n",
" \"name\": \"Random fictional homes\",\n",
" \"url\": \"https://raw.githubusercontent.com/IBM/visualize-data-with-python/master/data/HoustonFloodedZips250.geojson\",\n",
" \"circle-color\": \"rgb(0, 255, 0)\"\n",
" }\n",
" ])\n",
" def formatOptions(self,options):\n",
" return ';'.join([\"{}={}\".format(key,value) for (key, value) in iteritems(options)])\n",
"\n",
" @route()\n",
" def mainScreen(self):\n",
" return \"\"\"\n",
"