{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "
\n", "\n", "
\n", "\"Unidata\n", "
\n", "\n", "

THREDDS Catalogs: The Basics

\n", "

Unidata AMS 2021 Student Conference

\n", "\n", "
\n", "
\n", "\n", "---\n", "\n", "
\"HTML
\n", "\n", "### Focuses\n", "* Become familiar with THREDDS Catalogs and the THREDDS Data Server (TDS)\n", "* Browse THREDDS Catalogs using Siphon\n", "* Show metadata and available datasets contained within a THREDDS Catalog\n", "* List the data access methods associated with a dataset\n", "\n", "### Objectives\n", "1. [Read a THREDDS Catalog](#read)\n", "1. [Moving from one THREDDS Catalog to another](#follow)\n", "1. [Working with a TDS Catalog Dataset](#dataset)\n", "
\n", "
\n", "
\n", "---\n", "\n", "### Imports\n", "\n", "The main python package we will use to work with THREDDS Catalogs is called Siphon.\n", "Siphon can read THREDDS Catalogs, which are xml documents that do one or more of the following:\n", "1. reference other THREDDS Catalogs\n", "1. expose metadata about a dataset\n", "1. describe how to access a dataset\n", "\n", "The xml documents themselves can be written by hand, but often they are generated by a server, such as the THREDDS Data Server.\n", "They may be read locally from an xml file, or remotely over HTTP.\n", "Siphon greatly simplifies the process of reading and using the information contained in xml, allowing users to \"siphon off data\" from a variety of sources." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from siphon.catalog import TDSCatalog" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Read a THREDDS Catalog from a TDS \n", "\n", "For this notebook, we will use the Unidata demonstration TDS.\n", "If you visit the server in your browser, you will see something like the image at the top of this notebook.\n", "The page you see is actually a product of the TDS, and is generated by the server (we call this an HTML view of the catalog).\n", "If you change the last part of the URL from `.html` to `.xml` (that is, ), you will see the actual THREDDS Catalog in your browser, which looks similar to this:\n", "\n", "~~~xml\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "~~~\n", "\n", "This catalog tells us that there are other catalogs containing data from forecast models, radar data, satellite data, etc.\n", "In this sense, a catalog can point to other catalogs, creating a tree-like structure in which the datasets are organized.\n", "This will vary from server to server, as needs vary across organizations and groups.\n", "\n", "We can use Siphon to read in this remote catalog programmatically, without the need for a web browser:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "catalog = TDSCatalog('http://thredds.ucar.edu/thredds/catalog/catalog.xml')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have read in the THREDDS Catalog from the THREDDS Data Server, we can investigate what information it holds.\n", "A list of the names of other catalogs it points to is contained within the `catalog_refs` instance attribute, and can be access as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "catalog.catalog_refs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are more things you can do once you have read a THREDDS Catalog using Siphon, but for now we'll leave it at this.\n", "\n", "
\n", "Note: Not all TDS catalogs are intended to be browsed directly: Occasionally, the TDS is used purely as \"middleware\", and the catalogs are not setup for users to easily browse directly.\n", "An example of this would be the catalogs produced by the TDS serving data for the North America component of the Coordinated Regional Downscaling Experiment ([NA-CORDEX](https://na-cordex.org/index.html)).\n", "The intent of the data providers is for users to search for datasets using the NA-CORDEX search page on the NCAR Climate Data Gateway, which allows for one to search for datasets by variable type, experiment, driver, model, etc.\n", "Although the NA-CORDEX datasets are hosted on a TDS, they are all contained in one catalog, and their names are defined using a combination of the parameters used in the NA-CORDEX search page.\n", "For example, one of the over 62,000 NA-CORDEX datasets is named cordex.raw.NAM-44.seas.RCA4.EC-EARTH.rcp26.prec.v20180914.\n", "When you see a THREDDS Catalog in which the datasets have opaque names like this, that's your clue that the catalogs are probably not intended to be browsed directly by users, but rather accessed through another service (such as the NA-CORDEX search interface on the Climate Data Gateway).\n", "
\n", "\n", "Top\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading a referenced catalog \n", "\n", "If we'd like to see what is available in the `'Satellite Data'` catalog, we can use the `.follow()` method to read in the new catalog, and look at the `.catalog_refs` instance attribute of the new catalog:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "satellite_catalog = catalog.catalog_refs['Satellite Data'].follow()\n", "satellite_catalog.catalog_refs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The URL of the new catalog is in the `catalog_url` instance attribute, and can be accessed as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "satellite_catalog.catalog_url" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Any datasets described by the catalog are contained in the `datasets` instance attribute:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "satellite_catalog.datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `[]` indicates there are no datasets contained within the catalog.\n", "We can continue to work our way down through the catalog structure until we reach a catalog that contains a dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "goes_east_grb_catalog = satellite_catalog.catalog_refs['GOES East GOES Rebroadcast (GRB)'].follow()\n", "print(goes_east_grb_catalog.catalog_url)\n", "print(' catalogs: {}'.format(goes_east_grb_catalog.catalog_refs))\n", "print(' datasets: {}\\n'.format(goes_east_grb_catalog.datasets))\n", "\n", "abi_catalog = goes_east_grb_catalog.catalog_refs['ABI'].follow()\n", "print(abi_catalog.catalog_url)\n", "print(' catalogs: {}'.format(abi_catalog.catalog_refs))\n", "print(' datasets: {}\\n'.format(abi_catalog.datasets))\n", "\n", "conus_catalog = abi_catalog.catalog_refs['CONUS'].follow()\n", "print(conus_catalog.catalog_url)\n", "print(' catalogs: {}'.format(conus_catalog.catalog_refs))\n", "print(' datasets: {}\\n'.format(conus_catalog.datasets))\n", "\n", "channel01_catalog = conus_catalog.catalog_refs['Channel01'].follow()\n", "print(channel01_catalog.catalog_url)\n", "print(' catalogs: {}'.format(channel01_catalog.catalog_refs))\n", "print(' datasets: {}\\n'.format(channel01_catalog.datasets))\n", "\n", "date_catalog = channel01_catalog.catalog_refs['20210110'].follow()\n", "print(date_catalog.catalog_url)\n", "print(' catalogs: {}'.format(date_catalog.catalog_refs))\n", "print(' datasets: {}\\n'.format(date_catalog.datasets))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We used the `follow()` method several times before finally reaching a catalog with datasets.\n", "Normally, it is easiest to browse the catalogs of a TDS using a web browser in order to find a dataset collection that you might be interested in using.\n", "Once you have found a dataset you are interested in, you can use the URL from your browser to begin working in python using Siphon.\n", "For this collection of data (CONUS domain of the GOES East satellite Advanced Baseline Imager instrument (channel 1)), the catalog looks like a good place to start, as it points to catalogs named by date (`yyyyMMdd`).\n", "\n", "
\n", "Real-time data availability: In general, the datasets available on the demonstration TDS managed by Unidata are updated in real time.\n", "Data are removed from the server after a certain period of time, typically between three days and one month (depending on the size of the data files).\n", "This collection contains, roughly, the most recent 14 days of data.\n", "
\n", "\n", "As mentioned at the beginning of this notebook, catalogs can expose metadata about a dataset.\n", "The `metadata` instance variable holds any metadata defined by the catalog, such as `dataFormat`, `documentation`, etc.\n", "For example, the metadata associated with `date_catalog` looks like:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "date_catalog.metadata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The amount of metadata contained within a catalog depends on how much effort has been put into currating the collection." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---\n", "\n", "## Working with a TDS Catalog Dataset \n", "Once we have found a catalog with datasets, we can access once of the datasets using its name:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dataset = date_catalog.datasets['OR_ABI-L1b-RadC-M6C01_G16_s20210100156163_e20210100158536_c20210100158591.nc']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have a dataset, we can see in what ways we can access the dataset using the `access_urls` instance variable:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dataset.access_urls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each service provides a unique way of accessing the metadata or actual data contained within the dataset.\n", "Other Siphon notebooks explore ways in which the services can be used, but at this point, you are ready to begin your data analysis journey!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## See also\n", "* [Siphon documentation](https://unidata.github.io/siphon/latest/index.html)\n", "* [Siphon TdsCatalog class documentation](https://unidata.github.io/siphon/latest/api/catalog.html#siphon.catalog.TDSCatalog)\n", "\n", "## Related Notebooks\n", "* [Siphon (catalog filtering)](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/dataAccess/siphon-catalog-filtering.ipynb)\n", "* [Siphon (remote_access)](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/dataAccess/siphon-RemoteOpen.ipynb)\n", "* [Siphon (remote_open)](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/dataAccess/siphon-RemoteAccess.ipynb)\n", "* [Siphon (subset)](https://nbviewer.jupyter.org/github/Unidata/pyaos-ams-2021/blob/master/notebooks/dataAccess/siphon-Subset.ipynb)\n", "\n", "## Example TDS instances\n", "* [Unidata Demonstration TDS](https://thredds.ucar.edu/thredds/catalog/catalog.html)\n", "* [Department of Atmospheric and Oceanic Sciences - University of Wisconsin – Madison](https://thredds.aos.wisc.edu/thredds/catalog/catalog.html)\n", "* [Northwest Knowledge Network (NKN) - University of Idaho](https://www.reacchpna.org/thredds/catalog/catalog.html)\n", "* [Coastal Data Information Program (CDIP) - University of California San Diego](https://thredds.cdip.ucsd.edu/thredds/catalog/catalog.html)\n", "* NOAA Servers\n", " * [National Centers for Environmental Information (NCEI)](https://www.ncei.noaa.gov/thredds/catalog/catalog.html)\n", " * [Center for Satellite Applications and Research (STAR)](https://www.star.nesdis.noaa.gov/thredds/catalog/catalog.html)\n", " * [Environmental Research Division (ERD) - Southwest Fisheries Science Center](https://oceanwatch.pfeg.noaa.gov/thredds/catalog.html)\n", " * [Center for Operational Oceanographic Products and Services (CO-OPS)](https://opendap.co-ops.nos.noaa.gov/thredds/catalog/catalog.html)\n", "* NASA Servers\n", " * [Jet Propulsion Laboratory (JPL) Physical Oceanography Distributed Active Archive Center (PO.DAAC)\n", "](https://thredds.jpl.nasa.gov/thredds/catalog/catalog.html)\n", " * [Oak Ridge National Laboratory (ORNL) DAAC](https://thredds.daac.ornl.gov/thredds/catalogs/ornldaac/ornldaac.html)\n", "* USGS Servers\n", " * [Center for Integrated Data Analytics](https://cida.usgs.gov/thredds/catalog/catalog.html)\n", " * [Woods Hole Coastal and Marine Science Center](https://geoport.whoi.edu/thredds/bathy_catalog.html) (Topography/Bathymetry)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Top\n", "\n", "---" ] } ], "metadata": { "kernelspec": { "display_name": "Python [conda env:pyaos-ams-2021]", "language": "python", "name": "conda-env-pyaos-ams-2021-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 4 }