{ "cells": [ { "cell_type": "markdown", "id": "d2583da9-3f1e-40e6-b81d-700f26e89daa", "metadata": { "tags": [] }, "source": [ "## Accessing GOES Geostationary Lightning Mapper (GLM) with the Planetary Computer STAC API\n", "\n", "The [Geostationary Lightning Mapper (GLM)](https://www.goes-r.gov/spacesegment/glm.html) is a single-channel, near-infrared optical transient detector that can detect the momentary changes in an optical scene, indicating the presence of lightning. GLM measures total lightning (in-cloud, cloud-to-cloud and cloud-to-ground) activity continuously over the Americas and adjacent ocean regions with near-uniform spatial resolution of approximately 10 km. GLM collects information such as the frequency, location and extent of lightning discharges to identify intensifying thunderstorms and tropical cyclones. Trends in total lightning available from the GLM provide critical information to forecasters, allowing them to focus on developing severe storms much earlier and before these storms produce damaging winds, hail or even tornadoes.\n", "\n", "The GLM data product consists of a hierarchy of earth-located lightning radiant energy measures including events, groups, and flashes:\n", "\n", "- Lightning \"events\" are detected by the instrument.\n", "- Lightning \"groups\" are a collection of one or more lightning events that satisfy temporal and spatial coincidence thresholds.\n", "- Similarly, lightning \"flashes\" are a collection of one or more lightning groups that satisfy temporal and spatial coincidence thresholds.\n", "The product includes the relationship among lightning events, groups, and flashes, and the area coverage of lightning groups and flashes. The product also includes processing and data quality metadata, and satellite state and location information.\n", "\n", "The NetCDF files are delivered to Azure as part of the [NOAA Open Data Dissemination (NODD) Program](https://www.noaa.gov/information-technology/open-data-dissemination)." ] }, { "cell_type": "markdown", "id": "459cd92c-22e1-4976-8a7d-65187bd34fd2", "metadata": {}, "source": [ "### Data Access\n", "This notebook works with or without an API key, but you will be given more permissive access to the data with an API key. The [Planetary Computer Hub](https://planetarycomputer.microsoft.com/compute) sets the environment variable \"PC_SDK_SUBSCRIPTION_KEY\" when your server is started. The API key may be manually set via the following code:\n", "\n", "```python\n", "pc.settings.set_subscription_key()\n", "```" ] }, { "cell_type": "markdown", "id": "ecb76ce5-b20f-491a-9033-fc53e2c934a6", "metadata": {}, "source": [ "The datasets hosted by the Planetary Computer are available from [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/). We'll use [pystac-client](https://pystac-client.readthedocs.io/) to search the Planetary Computer's [STAC API](https://planetarycomputer.microsoft.com/api/stac/v1/docs) for the subset of the data that we care about, and then we'll load the data directly from Azure Blob Storage. We'll specify a `modifier` so that we can access the data stored in the Planetary Computer's private Blob Storage Containers. See [Reading from the STAC API](https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/) and [Using tokens for data access](https://planetarycomputer.microsoft.com/docs/concepts/sas/) for more. " ] }, { "cell_type": "code", "execution_count": 1, "id": "f3f030fa-8639-4b4e-b0a1-f3226375c28e", "metadata": {}, "outputs": [], "source": [ "import planetary_computer\n", "import pystac_client\n", "import rich.table\n", "\n", "# Open the Planetary Computer STAC API\n", "catalog = pystac_client.Client.open(\n", " \"https://planetarycomputer.microsoft.com/api/stac/v1/\",\n", " modifier=planetary_computer.sign_inplace,\n", ")" ] }, { "cell_type": "markdown", "id": "97e4f9bb-1313-46b2-916a-05709f1f2d4c", "metadata": {}, "source": [ "The GOES GLM product is typically queried by datetime and the \"GOES-16\" and \"GOES-17\" platforms. Each item has an identical spatial footprint." ] }, { "cell_type": "code", "execution_count": 2, "id": "63ddd01e-2ae3-4b5d-af8f-cdcbb56f0423", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Fetch the collection of interest and display available items\n", "search = catalog.search(\n", " collections=\"goes-glm\",\n", " datetime=\"2022-10-26T00:02:50Z\",\n", " query={\"platform\": {\"eq\": \"GOES-17\"}},\n", ")\n", "items = list(search.get_items())\n", "items" ] }, { "cell_type": "markdown", "id": "24391f86-a70b-457d-ac54-2fdca8737921", "metadata": {}, "source": [ "### STAC Metadata" ] }, { "cell_type": "code", "execution_count": 3, "id": "3a5e122e-49cc-4139-8137-1db756715c64", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃ Property                             ┃ Value                       ┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ constellation                        │ GOES                        │\n",
       "│ datetime                             │ 2022-10-26T00:02:50Z        │\n",
       "│ end_datetime                         │ 2022-10-26T00:03:00.0Z      │\n",
       "│ goes:event_count                     │ 1110                        │\n",
       "│ goes:flash_count                     │ 34                          │\n",
       "│ goes:flash_time_threshold            │ 3.3299999237060547          │\n",
       "│ goes:group_count                     │ 523                         │\n",
       "│ goes:group_time_threshold            │ 0.0                         │\n",
       "│ goes:lightning_wavelength            │ 777.3699951171875           │\n",
       "│ goes:nominal_satellite_height        │ 35786.0234375               │\n",
       "│ goes:nominal_satellite_subpoint_lat  │ 0.0                         │\n",
       "│ goes:nominal_satellite_subpoint_lon  │ -137.1999969482422          │\n",
       "│ goes:orbital_slot                    │ West                        │\n",
       "│ goes:percent_navigated_L1b_events    │ 1.0                         │\n",
       "│ goes:percent_uncorrectable_L0_errors │ 0.0                         │\n",
       "│ goes:system_environment              │ OR                          │\n",
       "│ goes:yaw_flip_flag                   │ 2                           │\n",
       "│ gsd                                  │ 8000                        │\n",
       "│ instruments                          │ ['FM2']                     │\n",
       "│ mission                              │ GOES                        │\n",
       "│ platform                             │ GOES-17                     │\n",
       "│ processing:facility                  │ WCDAS                       │\n",
       "│ processing:level                     │ L2                          │\n",
       "│ proj:centroid                        │ {'lat': 0.0, 'lon': -137.0} │\n",
       "│ proj:epsg                            │ 4326                        │\n",
       "│ start_datetime                       │ 2022-10-26T00:02:40.0Z      │\n",
       "└──────────────────────────────────────┴─────────────────────────────┘\n",
       "
\n" ], "text/plain": [ "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1mProperty \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mValue \u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", "│ constellation │ GOES │\n", "│ datetime │ 2022-10-26T00:02:50Z │\n", "│ end_datetime │ 2022-10-26T00:03:00.0Z │\n", "│ goes:event_count │ 1110 │\n", "│ goes:flash_count │ 34 │\n", "│ goes:flash_time_threshold │ 3.3299999237060547 │\n", "│ goes:group_count │ 523 │\n", "│ goes:group_time_threshold │ 0.0 │\n", "│ goes:lightning_wavelength │ 777.3699951171875 │\n", "│ goes:nominal_satellite_height │ 35786.0234375 │\n", "│ goes:nominal_satellite_subpoint_lat │ 0.0 │\n", "│ goes:nominal_satellite_subpoint_lon │ -137.1999969482422 │\n", "│ goes:orbital_slot │ West │\n", "│ goes:percent_navigated_L1b_events │ 1.0 │\n", "│ goes:percent_uncorrectable_L0_errors │ 0.0 │\n", "│ goes:system_environment │ OR │\n", "│ goes:yaw_flip_flag │ 2 │\n", "│ gsd │ 8000 │\n", "│ instruments │ ['FM2'] │\n", "│ mission │ GOES │\n", "│ platform │ GOES-17 │\n", "│ processing:facility │ WCDAS │\n", "│ processing:level │ L2 │\n", "│ proj:centroid │ {'lat': 0.0, 'lon': -137.0} │\n", "│ proj:epsg │ 4326 │\n", "│ start_datetime │ 2022-10-26T00:02:40.0Z │\n", "└──────────────────────────────────────┴─────────────────────────────┘\n" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "item = items[0]\n", "\n", "table = rich.table.Table(\"Property\", \"Value\")\n", "for key, value in sorted(item.properties.items()):\n", " table.add_row(key, str(value))\n", "\n", "table" ] }, { "cell_type": "markdown", "id": "2fe774d5-a95b-4244-818f-45710774c57d", "metadata": {}, "source": [ "### Data assets\n", "\n", "The single asset is a NetCDF file, which can be opened by libraries like xarray." ] }, { "cell_type": "code", "execution_count": 4, "id": "0814481f-2ca0-4ffe-8f6a-ca05509ac0ec", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:                                 (number_of_events: 1110,\n",
       "                                             number_of_groups: 523,\n",
       "                                             number_of_flashes: 34,\n",
       "                                             number_of_time_bounds: 2,\n",
       "                                             number_of_wavelength_bounds: 2,\n",
       "                                             number_of_field_of_view_bounds: 2)\n",
       "Coordinates: (12/21)\n",
       "    event_id                                (number_of_events) uint32 9759000...\n",
       "    event_time_offset                       (number_of_events) datetime64[ns] ...\n",
       "    event_lat                               (number_of_events) float64 15.14 ...\n",
       "    event_lon                               (number_of_events) float64 -91.25...\n",
       "    event_parent_group_id                   (number_of_events) uint32 4131682...\n",
       "    group_id                                (number_of_groups) uint32 4131682...\n",
       "    ...                                      ...\n",
       "    product_time                            datetime64[ns] 2022-10-26T00:02:40\n",
       "    lightning_wavelength                    float32 777.4\n",
       "    group_time_threshold                    float32 0.0\n",
       "    flash_time_threshold                    float32 3.33\n",
       "    lat_field_of_view                       float32 0.0\n",
       "    lon_field_of_view                       float32 -137.0\n",
       "Dimensions without coordinates: number_of_events, number_of_groups,\n",
       "                                number_of_flashes, number_of_time_bounds,\n",
       "                                number_of_wavelength_bounds,\n",
       "                                number_of_field_of_view_bounds\n",
       "Data variables: (12/27)\n",
       "    event_energy                            (number_of_events) float32 1.655e...\n",
       "    group_frame_time_offset                 (number_of_groups) datetime64[ns] ...\n",
       "    group_area                              (number_of_groups) float32 5.32e+...\n",
       "    group_energy                            (number_of_groups) float32 1.438e...\n",
       "    group_quality_flag                      (number_of_groups) float32 0.0 .....\n",
       "    flash_frame_time_offset_of_first_event  (number_of_flashes) datetime64[ns] ...\n",
       "    ...                                      ...\n",
       "    nominal_satellite_subpoint_lon          float64 -137.2\n",
       "    lon_field_of_view_bounds                (number_of_field_of_view_bounds) float32 ...\n",
       "    percent_uncorrectable_L0_errors         float64 0.0\n",
       "    algorithm_dynamic_input_data_container  int32 -2147483647\n",
       "    processing_parm_version_container       int32 -2147483647\n",
       "    algorithm_product_version_container     int32 -2147483647\n",
       "Attributes: (12/29)\n",
       "    production_site:           WCDAS\n",
       "    featureType:               point\n",
       "    dataset_name:              OR_GLM-L2-LCFA_G17_s20222990002400_e2022299000...\n",
       "    naming_authority:          gov.nesdis.noaa\n",
       "    Conventions:               CF-1.7\n",
       "    institution:               DOC/NOAA/NESDIS> U.S. Department of Commerce, ...\n",
       "    ...                        ...\n",
       "    time_coverage_start:       2022-10-26T00:02:40.0Z\n",
       "    time_coverage_end:         2022-10-26T00:03:00.0Z\n",
       "    production_data_source:    Realtime\n",
       "    production_environment:    OE\n",
       "    LUT_Filenames:             GLM_CALINR_AllFilters(FM2W_CDRL79RevD_PR_09_00...\n",
       "    id:                        6ecd089c-b52d-473b-a63e-f4ed30132de3
" ], "text/plain": [ "\n", "Dimensions: (number_of_events: 1110,\n", " number_of_groups: 523,\n", " number_of_flashes: 34,\n", " number_of_time_bounds: 2,\n", " number_of_wavelength_bounds: 2,\n", " number_of_field_of_view_bounds: 2)\n", "Coordinates: (12/21)\n", " event_id (number_of_events) uint32 ...\n", " event_time_offset (number_of_events) datetime64[ns] ...\n", " event_lat (number_of_events) float64 ...\n", " event_lon (number_of_events) float64 ...\n", " event_parent_group_id (number_of_events) uint32 ...\n", " group_id (number_of_groups) uint32 ...\n", " ... ...\n", " product_time datetime64[ns] ...\n", " lightning_wavelength float32 ...\n", " group_time_threshold float32 ...\n", " flash_time_threshold float32 ...\n", " lat_field_of_view float32 ...\n", " lon_field_of_view float32 ...\n", "Dimensions without coordinates: number_of_events, number_of_groups,\n", " number_of_flashes, number_of_time_bounds,\n", " number_of_wavelength_bounds,\n", " number_of_field_of_view_bounds\n", "Data variables: (12/27)\n", " event_energy (number_of_events) float32 ...\n", " group_frame_time_offset (number_of_groups) datetime64[ns] ...\n", " group_area (number_of_groups) float32 ...\n", " group_energy (number_of_groups) float32 ...\n", " group_quality_flag (number_of_groups) float32 ...\n", " flash_frame_time_offset_of_first_event (number_of_flashes) datetime64[ns] ...\n", " ... ...\n", " nominal_satellite_subpoint_lon float32 ...\n", " lon_field_of_view_bounds (number_of_field_of_view_bounds) float32 ...\n", " percent_uncorrectable_L0_errors float32 ...\n", " algorithm_dynamic_input_data_container int32 ...\n", " processing_parm_version_container int32 ...\n", " algorithm_product_version_container int32 ...\n", "Attributes: (12/29)\n", " production_site: WCDAS\n", " featureType: point\n", " dataset_name: OR_GLM-L2-LCFA_G17_s20222990002400_e2022299000...\n", " naming_authority: gov.nesdis.noaa\n", " Conventions: CF-1.7\n", " institution: DOC/NOAA/NESDIS> U.S. Department of Commerce, ...\n", " ... ...\n", " time_coverage_start: 2022-10-26T00:02:40.0Z\n", " time_coverage_end: 2022-10-26T00:03:00.0Z\n", " production_data_source: Realtime\n", " production_environment: OE\n", " LUT_Filenames: GLM_CALINR_AllFilters(FM2W_CDRL79RevD_PR_09_00...\n", " id: 6ecd089c-b52d-473b-a63e-f4ed30132de3" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import xarray as xr\n", "import fsspec\n", "\n", "ds = xr.open_dataset(fsspec.open(item.assets[\"netcdf\"].href).open())\n", "ds" ] }, { "cell_type": "markdown", "id": "c8280ca8-8c14-405a-8fa4-4bf8924ebab6", "metadata": {}, "source": [ "The three types of data (events, groups, and flashes) are embedded in this single NetCDF file. You might want to convert one of those to a tabular format for analysis or visualization. In this example, we'll extract the `group` variables." ] }, { "cell_type": "code", "execution_count": 5, "id": "f36bbee2-c7ea-4392-b4fd-7ae52063e406", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
energyareatime_offsetframe_time_offsetquality_flagparent_flash_idgeometry
413168201.438346e-145.319701e+082022-10-26 00:02:41.2790884672022-10-26 00:02:41.4072642700.019663POINT (-91.41206 15.12315)
413168214.098026e-141.240653e+092022-10-26 00:02:41.3584353932022-10-26 00:02:41.4869926720.019663POINT (-91.42130 15.11019)
413168243.208133e-141.063330e+092022-10-26 00:02:41.4381637942022-10-26 00:02:41.5663395980.019663POINT (-91.43069 15.12149)
413168262.184922e-158.850908e+072022-10-26 00:02:41.5075923542022-10-26 00:02:41.6361496330.019663POINT (-91.46420 15.12907)
413168276.984346e-151.782390e+082022-10-26 00:02:41.9588779962022-10-26 00:02:42.0874352750.019664POINT (-91.17548 15.04030)
\n", "
" ], "text/plain": [ " energy area time_offset \\\n", "41316820 1.438346e-14 5.319701e+08 2022-10-26 00:02:41.279088467 \n", "41316821 4.098026e-14 1.240653e+09 2022-10-26 00:02:41.358435393 \n", "41316824 3.208133e-14 1.063330e+09 2022-10-26 00:02:41.438163794 \n", "41316826 2.184922e-15 8.850908e+07 2022-10-26 00:02:41.507592354 \n", "41316827 6.984346e-15 1.782390e+08 2022-10-26 00:02:41.958877996 \n", "\n", " frame_time_offset quality_flag parent_flash_id \\\n", "41316820 2022-10-26 00:02:41.407264270 0.0 19663 \n", "41316821 2022-10-26 00:02:41.486992672 0.0 19663 \n", "41316824 2022-10-26 00:02:41.566339598 0.0 19663 \n", "41316826 2022-10-26 00:02:41.636149633 0.0 19663 \n", "41316827 2022-10-26 00:02:42.087435275 0.0 19664 \n", "\n", " geometry \n", "41316820 POINT (-91.41206 15.12315) \n", "41316821 POINT (-91.42130 15.11019) \n", "41316824 POINT (-91.43069 15.12149) \n", "41316826 POINT (-91.46420 15.12907) \n", "41316827 POINT (-91.17548 15.04030) " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import geopandas\n", "\n", "df = geopandas.GeoDataFrame(\n", " {\n", " \"energy\": ds.group_energy,\n", " \"area\": ds.group_area,\n", " \"time_offset\": ds.group_time_offset,\n", " \"frame_time_offset\": ds.group_frame_time_offset,\n", " \"quality_flag\": ds.group_quality_flag,\n", " \"parent_flash_id\": ds.group_parent_flash_id,\n", " },\n", " index=ds.group_id,\n", " geometry=geopandas.points_from_xy(ds.group_lon, ds.group_lat, crs=\"EPSG:4326\"),\n", ")\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 6, "id": "da6357ad-df21-4b26-993e-c0be84aba52d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import cartopy.crs as ccrs\n", "import matplotlib.pyplot as plt\n", "\n", "# Display the GOES GLM \"group\" product centriod location\n", "fig, ax = plt.subplots(\n", " figsize=(12, 12),\n", " subplot_kw=dict(projection=ccrs.Orthographic(central_longitude=-90.0)),\n", ")\n", "\n", "df.geometry.plot(ax=ax, facecolor=\"red\", marker=\"+\", transform=ccrs.PlateCarree())\n", "\n", "ax.coastlines(resolution=\"50m\")\n", "ax.set_global()\n", "ax.gridlines()\n", "ax.set_title(\n", " f\"GOES Geostationary Lightning Mapper (GLM) \\n {item.datetime.isoformat()}\",\n", " fontweight=\"bold\",\n", " fontsize=\"12\",\n", ");" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" }, "vscode": { "interpreter": { "hash": "4ffbd1b0996e3a79afb3d81e30644170a5904f770a23d1ec65a6a61c4dfe8e95" } }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }