{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Rivulet: U. S. Geological Survey Water Quality eXchange\n", "_by Michelle H Wilkerson_" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "### Purpose of this Notebook\n", "\n", "This notebook focuses on **Water Quality** as a phenomenon. While most students understand that poor water quality can impact health, they may not know what sorts of pollutants are involved, or what kinds of events or conditions lead to reductions in water quality.\n", "\n", "This data tool allows users to connect to the United States Geological Survey (USGS) water data APIs, search for water quality data streams in an area of interest, and then provides a collection of ways to filter and search the data to ensure you find datasets that have patterns worth exploring. \n", "\n", "
\n", " Click here for more information about APIs\n", "\n", "API stands for **Application Programming Interface**. In data science, APIs are incredibly common because they allow different computer programs to access to \"fetch\", or download, datasets from large databases or services. Instead of downloading a massive CSV file that might be out of date by the time you open it, an API lets you request exactly the data you need, right when you need it. \n", "\n", "For educators, APIs are powerful because they allow students to work with **real-world, live data**. However, they do come with some risks: sometimes an API service might be down, or the way you ask for data (the \"request\") might need to change if the service is updated. Learning to use APIs is a core skill in modern science, helping students understand that data isn't just a static table, but a dynamic resource that we can query and explore.\n", "\n", "You are welcome to modify and adapt this script. You may find the USGS' water data APIs documentation [here]('https://code.usgs.gov/water/dataretrieval-python/-/blob/v1.0.2/dataretrieval/wqp.py') and [here](https://doi-usgs.github.io/dataretrieval-python/) helpful.\n", "\n", "This notebook was developed as part of NSF Grant 2445609 to support accessing and processing public data for middle and high school classroom activities. It's written to be relatively accessible to beginners, but if you have not interacted with computational notebooks or python before you may find navigating this tool difficult. (Check out the [Show Your Work](https://github.com/CalCoRE/show-your-work) project for a gentle introduction to computational notebooks for educators!)\n", "\n", "Our project is focused on supporting data analysis and mechanistic reasoning in science education. In other words, we want students to learn how data provides information about _how scientific mechanisms work_, and how understanding scientific mechanisms can help them to _explain and interpret patterns in data_. This builds on a long history of research on complex systems and agent-based modeling, and more closely connects that work to current expansions of data analysis across subjects." ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "# Part I: Setup\n", "\n", "The USGS has developed a Python library (unhelpfully but impressively called `dataretrieval`) to help people access and fetch hydrological data from several different water-related data services." ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "### Connect to NWIS and WQP\n", "\n", "You can sign up for an API key [at this site](https://api.waterdata.usgs.gov/signup). Once you receive it, replace the DEMO_KEY below with your unique API key. This will give you more reliable access to the data, and will help USGS understand more about who is using their services. Do not share your key!" ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [], "source": [ "!pip install dataretrieval\n", "\n", "API_KEY = \"DEMO_KEY\" " ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "Not sure this is important yet, but the docs say that if you want data after March 2024 you want to specify `legacy=False`. See [here](https://github.com/DOI-USGS/dataretrieval-python#:~:text=%E2%9A%A0%EF%B8%8F,the%20wqp%20module.) and [here](https://doi-usgs.github.io/dataRetrieval/articles/Status.html#:~:text=Discrete%20Data,non%2DUSGS%20data) for more information.\n", "\n", "#### **Note for Google CoLab users:**\n", "If you are running this notebook in Google CoLab, uncomment the first line of the code below. This will make it easier for you to download your data files. There will also be lines to uncomment in each \"Data Fetch\" section, once you are ready to get your data." ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "# from google.colab import files ## UNCOMMENT THIS LINE IF USING COLAB\n", "import dataretrieval.nwis as nwis\n", "from dataretrieval import wqp\n", "import pandas as pd\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "### Specify a Location" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "Let's create a bounding box to indicate the region we are interested in. We will then filter our queries to focus only on monitoring sites within the bounding box." ] }, { "cell_type": "code", "execution_count": null, "id": "9", "metadata": {}, "outputs": [], "source": [ "# EDIT HERE: Define a bounding box around your\n", "# target region. If it is densely populated, we suggest\n", "# you start with a bounding box that is only one degree\n", "# in area. \n", "\n", "min_lat = 37.5 # CHANGE TO YOUR MINIMUM LATITUDE\n", "max_lat = 38 # CHANGE TO YOUR MAXIMUM LATITUDE\n", "\n", "min_long = -122.5 # CHANGE TO YOUR MINIMUM LONGITUDE\n", "max_long = -122\n", "\n", "# this is unnecessary but sort of luxurious. let's map the box to\n", "# make sure we're capturing what we want.\n", "\n", "import folium\n", "\n", "bbox = [[min_lat, min_long], [max_lat, max_long]]\n", "\n", "# Calculate the center of the box to position the map\n", "map_center = [(bbox[0][0] + bbox[1][0]) / 2, (bbox[0][1] + bbox[1][1]) / 2]\n", "\n", "# Create a Folium map object\n", "m = folium.Map(location=map_center, zoom_start=8)\n", "\n", "# Add a rectangle for the bounding box to the map\n", "folium.Rectangle(\n", " bounds=bbox,\n", " color=\"#ff0000\", # Red border\n", " fill=True,\n", " fill_color=\"#ff7800\", # Orange fill\n", " fill_opacity=0.2\n", ").add_to(m)\n", "\n", "m" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "The function what_sites() allows us to see which sites satisfy certain criteria, like falling within a specific bounding box (how we use it here) or measuring a particular thing (which we will do later). The code below takes the latitudes and longitudes you entered above, and passes them to NWIS to see what data sites they have within that area." ] }, { "cell_type": "code", "execution_count": null, "id": "11", "metadata": {}, "outputs": [], "source": [ "bbox = str(min_long) + \",\" + str(min_lat) + \",\" + str(max_long) + \",\" + str(max_lat)\n", "\n", "sites, sites_metadata = nwis.what_sites(bBox=bbox)\n", "sites" ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "Whoa! You are likely looking at a long list of sites. Depending on the phenomena we're interested in, we'll filter this to only what we need. Or, you can use the list above to help you identify a specific site if you know of one to use below." ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "### Identify a Target Date\n", "\n", "Next, we want to identify a focus date. This might be a time when we think something interesting might have happened, or a date we're interested in for other reasons. California experienced king tides during Jan 11-13 in 2026, so we'll identify Jan 12 as a target date. You can edit it to whatever date you want." ] }, { "cell_type": "code", "execution_count": null, "id": "14", "metadata": {}, "outputs": [], "source": [ "# EDIT HERE: identify a target date when something interesting\n", "# was happening. Below, I define Jan 12, 2025, during king tides in CA.\n", "\n", "from datetime import datetime, timedelta\n", "\n", "target_date = \"01-12-2024\"\n", "target_datetime = datetime.strptime(target_date, \"%m-%d-%Y\")\n", "\n", "start_date = target_datetime\n", "end_date = target_datetime + timedelta(days=1)" ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "Now that we've identified a general location and time period, let's dive into the data." ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "# Part II: Fetch Data" ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "## Salinity in Estuaries (Exploring Tides)\n", "\n", "This first section helps you select datasets that feature information about **salinity** (saltiness) in bodies of water. There are lots of places where salinity behaves in interesting ways: in estuaries where fresh and salt water meet and move in and out with the tides, and in cold areas during winter months, when salt road treatments run off into freshwater sources. \n", "\n", "Salinity is measured through electrical conductance. In NWIS, there are many salinity measures, but the best one to look for here is \"specific conductance\", which is a measure of salinity that is also adjusted for temperature and that is used by both the USGS and the EPA. You can choose other measures, see the full list in [USGS parameter codes](https://help.waterdata.usgs.gov/parameter_cd?group_cd=PHY). \n", "\n", "Below, we'll look for all the sites within the region you specified that are identified as estuaries and that are measured for specific conductance. A lot of these don't actually have the data we need, so I'm applying an extra filter to make sure we get back only sites with actual measurements." ] }, { "cell_type": "code", "execution_count": null, "id": "18", "metadata": {}, "outputs": [], "source": [ "estuaries, estuaries_metadata = nwis.what_sites(bBox=bbox,\n", " startDt=start_date,\n", " endDt=end_date,\n", " parameterCd='00095',\n", " siteType='ES') #estuaries\n", "\n", "estuaries = estuaries[~estuaries['alt_datum_cd'].isna()] #remove if NaN\n", "\n", "estuaries" ] }, { "cell_type": "markdown", "id": "19", "metadata": {}, "source": [ "To really see how salinity in an estuary changes as the tides flow in and out, you will need a week or two of data around the target date. \n", "\n", "Pick your favorite site from the list above, and plop it into the code below. When you run the code, it will fetch the specific conductance data for two weeks around your target date." ] }, { "cell_type": "code", "execution_count": null, "id": "20", "metadata": {}, "outputs": [], "source": [ "start_date = target_datetime - timedelta(days=7)\n", "end_date = target_datetime + timedelta(days=7)\n", "\n", "sal_data, meta = nwis.get_iv(sites='373015122071000', #site of your choice \n", " parameterCd='00095',\n", " start=start_date.strftime(\"%Y-%m-%d\"),\n", " end=end_date.strftime(\"%Y-%m-%d\"))\n", "\n", "sal_data" ] }, { "cell_type": "markdown", "id": "21", "metadata": {}, "source": [ "NWIS reports this data at different depths, at 4 feet from the sea floor at that location (the \"bed\") and at 25 feet from the floor. Salt water is heavier, so it's expected that there will be a higher conductance deeper in the water. Comparing these measures can also reveal information about whether the water has been mixing a lot in this location.\n", "\n", "Now, let's plot the salinity of water in the shallower part of this site. Will we see the impact of the tides?" ] }, { "cell_type": "code", "execution_count": null, "id": "22", "metadata": {}, "outputs": [], "source": [ "import seaborn as sns\n", "\n", "sns.lineplot(data=sal_data,x='datetime', y='00095_upper: 25 ft from bed')" ] }, { "cell_type": "markdown", "id": "23", "metadata": {}, "source": [ "You should see a daily rise and fall of salinity, representing when the tide pushes ocean water into the estuary and when the outflow of fresh water flushes it back out as the tides recede. \n", "\n", "If this data look interesting and you'd like to use them with another program for deeper analysis, use the code below to save it as a CSV file." ] }, { "cell_type": "code", "execution_count": null, "id": "24", "metadata": {}, "outputs": [], "source": [ "sal_data.to_csv(\"salinity_data.csv\")\n", "\n", "# files.download(\"salinity_data.csv\") ## UNCOMMENT THIS LINE IF USING COLAB" ] }, { "cell_type": "markdown", "id": "25", "metadata": {}, "source": [ "## Fecal Coliform\n", "\n", "Fecal Coliforms like EColi appear in water sources after rains, when sewers release bacteria. Let's check it out. Remember that you defined your location bounding box and your target date in Part 2 setup above. These measurements are not conducted very often, so we will look for any sites that recorded Fecal Coliform levels for any dates within 1 week of your target date. Remember: you can adjust the time period _or_ the location if you need more sites or are looking for something in particular." ] }, { "cell_type": "code", "execution_count": null, "id": "26", "metadata": {}, "outputs": [], "source": [ "start_date = target_datetime - timedelta(weeks=1)\n", "end_date = target_datetime + timedelta(weeks=1)\n", "\n", "# check the parameter here, I went for one that was there and made sense\n", "ecoli_data, ec_meta = wqp.what_sites(bBox=bbox,\n", " startDateLo=start_date.strftime(\"%m-%d-%Y\"),\n", " startDateHi=end_date.strftime(\"%m-%d-%Y\"),\n", " characteristicName=\"Fecal Coliform\") #fecal coliforms\n", " # siteType='ST') #streams\n", "\n", "ecoli_data" ] }, { "cell_type": "markdown", "id": "27", "metadata": {}, "source": [ "Comparing patterns against storms, or looking at different sites can be interesting. Fecal coliform is most likely to be found in sites that represent smaller, fast-reacting bodies of water in watersheds with significant sources of fecal contamination (like streams or creeks, smaller lakes, or drainage structures). These are often called \"flashy\" systems. Take a look at the 'MonitoringLocationDescriptionText' column to see if you can find a site you know, or that is likely to have interesting dynamics.\n", "\n", "If you want to get data from more than one site, add them below with a list ['SITE_1','SITE_2']" ] }, { "cell_type": "code", "execution_count": null, "id": "28", "metadata": {}, "outputs": [], "source": [ "#Look 6 mongths before and after the target date\n", "start_date = target_datetime - timedelta(weeks=24)\n", "end_date = target_datetime + timedelta(weeks=24)\n", "\n", "ec_data, ecmeta = wqp.get_results(siteid=['CABEACH_WQX-BAY#300.1_SL','CABEACH_WQX-Crown Crab Cove'], #site of your choice \n", " startDateLo=start_date.strftime(\"%m-%d-%Y\"),\n", " startDateHi=end_date.strftime(\"%m-%d-%Y\"),\n", " characteristicName=\"Fecal Coliform\")\n", "\n", "# This returns a lot of data, let's look only at the most relevant columns\n", "ec_reduced = ec_data[['MonitoringLocationIdentifier',\n", " 'ActivityStartDate',\n", " 'ActivityStartTime/Time',\n", " 'ResultMeasureValue',\n", " 'ResultMeasure/MeasureUnitCode',\n", " 'ResultValueTypeName']]\n", "\n", "ec_reduced.head()" ] }, { "cell_type": "markdown", "id": "29", "metadata": {}, "source": [ "Let's take a look at the plots for your sites." ] }, { "cell_type": "code", "execution_count": null, "id": "30", "metadata": {}, "outputs": [], "source": [ "sns.lineplot(\n", " data=ec_reduced,\n", " x='ActivityStartDate',\n", " y='ResultMeasureValue',\n", " hue='MonitoringLocationIdentifier',\n", " marker='o'\n", ")\n", "plt.xticks(rotation=45)\n", "plt.xlabel('Activity Start Date')\n", "plt.ylabel('Fecal Coliform (ResultMeasureValue)')\n", "plt.legend(title='Site')\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "id": "31", "metadata": {}, "source": [ "If this looks good, you're ready to download!" ] }, { "cell_type": "code", "execution_count": null, "id": "32", "metadata": {}, "outputs": [], "source": [ "ec_reduced.to_csv(\"coliform_data.csv\")\n", "\n", "# files.download(\"coliform_data.csv\") ## UNCOMMENT THIS LINE IF USING COLAB" ] }, { "cell_type": "markdown", "id": "33", "metadata": {}, "source": [ "## Credits\n", "\n", "We are grateful to Dr. Lex Van Geen for his insights into pedagogically productive datasets.\n", "\n", "Hodson, T.O., Hariharan, J.A., Black, S., and Horsburgh, J.S., 2023, dataretrieval (Python): a Python package for discovering and retrieving water data available from U.S. federal hydrologic web services: U.S. Geological Survey software release, https://doi.org/10.5066/P94I5TX3." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }