{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Intake Tutorial\n", "\n", "\n", "
\n", "\n", "### Overview\n", " \n", "* **teaching:** 20 minutes\n", "* **exercises:** 0\n", "* **questions:**\n", " * How does Intake simplify data discovery, distribution, and loading?\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Table of contents\n", "1. [**Intake primer**](#Intake-primer)\n", "1. [**Build and intake catalog**](#Build-an-intake-catalog)\n", "1. [**Work with an intake catalog**](#Work-with-an-intake-catalog)\n", "1. [**Intake xarray example**](#Intake-xarray-example)\n", "1. [**Intake STAC example**](#Intake-STAC-example)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Intake primer\n", "\n", "\"intake\n", "\n", "\n", "[Intake](https://intake.readthedocs.io/en/latest/index.html) is a lightweight package for finding, investigating, loading and disseminating data. This notebook illutrates the usefulness of intake for a \"Data User\". Intake simplifies loading data from [many formats](https://intake.readthedocs.io/en/latest/plugin-directory.html#plugin-directory) into familiar Python objects like Pandas DataFrames or Xarray Datasets. Intake is especially useful for remote datasets - it allows us to bypass downloading data and instead load directly into a Python object for analysis. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Build an intake catalog\n", "\n", "Let's say we want to save a version of the data from our geopandas.ipynb tutorial for easy sharing and future use. intake has csv support by default but for loading data with geopandas we need to make sure the [intake_geopandas plugin](https://github.com/intake/intake_geopandas) is installed." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import intake\n", "import xarray\n", "\n", "print(intake.__version__)\n", "xarray.set_options(display_style=\"html\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Save data locally from our queries\n", "import pandas as pd\n", "import geopandas as gpd\n", "\n", "server = 'https://webservices.volcano.si.edu/geoserver/GVP-VOTW/ows?'\n", "query = 'service=WFS&version=2.0.0&request=GetFeature&typeName=GVP-VOTW:Smithsonian_VOTW_Holocene_Volcanoes&outputFormat=csv'\n", "df = pd.read_csv(server+query)\n", "df.to_csv('votw.csv', index=False)\n", "\n", "# Or save as geojson\n", "# Now load query results as json directly in geopandas\n", "query = 'service=WFS&version=2.0.0&request=GetFeature&typeName=GVP-VOTW:Smithsonian_VOTW_Holocene_Volcanoes&outputFormat=json'\n", "gf = gpd.read_file(server+query)\n", "gf.to_file('votw.geojson', driver='GeoJSON')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%writefile votw-intake-catalog.yaml\n", "\n", "metadata:\n", " version: 1\n", "\n", "sources:\n", " votw_pandas:\n", " args:\n", " csv_kwargs:\n", " blocksize: null #prevent reading in parallel with dask\n", " #urlpath: 'https://webservices.volcano.si.edu/geoserver/GVP-VOTW/ows?service=WFS&version=2.0.0&request=GetFeature&typeName=GVP-VOTW:Smithsonian_VOTW_Holocene_Volcanoes&outputFormat=csv'\n", " urlpath: './votw.csv'\n", " description: 'Smithsonian_VOTW_Holocene_Volcanoes 4.8.4'\n", " driver: csv\n", " metadata:\n", " citation: 'Global Volcanism Program, 2013. Volcanoes of the World, v. 4.8.4. Venzke, E (ed.). Smithsonian Institution. Downloaded 06 Dec 2019. https://doi.org/10.5479/si.GVP.VOTW4-2013'\n", " plots:\n", " last_eruption_year:\n", " kind: violin\n", " by: 'Region'\n", " y: 'Last_Eruption_Year'\n", " invert: True\n", " width: 700\n", " height: 500\n", " \n", " \n", " votw_geopandas:\n", " args:\n", " #urlpath: 'https://webservices.volcano.si.edu/geoserver/GVP-VOTW/ows?service=WFS&version=2.0.0&request=GetFeature&typeName=GVP-VOTW:Smithsonian_VOTW_Holocene_Volcanoes&outputFormat=json'\n", " urlpath: './votw.geojson'\n", " description: 'Smithsonian_VOTW_Holocene_Volcanoes 4.8.4'\n", " driver: geojson\n", " metadata:\n", " citation: 'Global Volcanism Program, 2013. Volcanoes of the World, v. 4.8.4. Venzke, E (ed.). Smithsonian Institution. Downloaded 06 Dec 2019. https://doi.org/10.5479/si.GVP.VOTW4-2013'" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# put this catalog, votw.csv, and votw.geojson, in a public place like GitHub!\n", "# This facilitates sharing and version controlled analysis\n", "cat = intake.open_catalog('votw-intake-catalog.yaml')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(list(cat))\n", "cat.votw_pandas.description" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Loading the data is now very straightforward:\n", "# We know the data will be read into a Pandas DataFrame because\n", "cat.votw_pandas.container" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = cat.votw_pandas.read()\n", "df.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Notice we also specified some pre-defined plots in the catalog\n", "# This requires hvplot\n", "import hvplot.pandas\n", "source = cat.votw_pandas\n", "source.plot.last_eruption_year()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load a different dataset in the same catalog\n", "source = cat.votw_geopandas\n", "source.description" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "gf = source.read()\n", "test = gf.loc[:,['Last_Eruption_Year', 'Volcano_Name', 'geometry']]\n", "test.hvplot.points(geo=True, hover_cols=['Volcano_Name'], color='Last_Eruption_Year')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Intake xarray example\n", "\n", "We've seen a plugin to load geospatial vector data into geopandas geodataframes, there is also a plugin to facilitate loading geospatial raster data into xarray dataarrays! https://github.com/intake/intake-xarray" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# load a catalog stored on github\n", "xcat = intake.open_catalog('https://raw.githubusercontent.com/intake/intake-xarray/master/examples/catalog.yml')\n", "display(list(xcat))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The use of the intake catalog is much the same as above, except that the data container has switched to xarray objects." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "geotiff = xcat.geotiff\n", "geotiff.plot.band_image()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "da = geotiff.read() # to xarray.DataArray\n", "da.max('band')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Intake STAC example\n", "\n", "Instead of creating your own metadata catalogs from scratch as YAML files, intake plugins exist to read catalogs in different formats. For example, for geospatial data on the web, [SpatioTemporal Asset Catalogs (STAC)](https://stacspec.org/) are emerging as a standard way to descripe data that you want to search for based on georeference location, time, and perhaps other metadata fields. The intake-stac plugin greatly facilitates loading datasets referenced in STAC catalogs into Python Xarray objects for analysis. https://github.com/pangeo-data/intake-stac" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "stac_cat = intake.open_stac_catalog(\n", " 'https://storage.googleapis.com/pdd-stac/disasters/catalog.json',\n", " name='planet-disaster-data'\n", ")\n", "display(list(stac_cat))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(stac_cat['Houston-East-20170831-103f-100d-0f4f-RGB'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Entries in the catalog are accessed just like above. Below we pull the thumbnail image from the Hurricane Harvey composite image." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "da = stac_cat['Houston-East-20170831-103f-100d-0f4f-RGB']['thumbnail'].to_dask()\n", "da" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "da.plot.imshow(rgb='channel')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Related Intake plugins\n", "\n", "- [Intake-ESM](https://intake-esm.readthedocs.io/en/latest/): Intake driver for loading catalogs of climate model data, intake-esm.readthedocs.io\n", "- [Pangeo-Datastore](https://pangeo-data.github.io/pangeo-datastore/): Pangeo's public Intake catalog\n", "- [Sat-Search](https://github.com/sat-utils/sat-search): Search and discovery of STAC catalogs, with plugin support to intake-stac" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }