{ "cells": [ { "cell_type": "markdown", "id": "88337e22-e47d-465c-84c7-b8a120a136ff", "metadata": {}, "source": [ "## Accessing ECMWF Open Data – Real Time\n", "\n", "The Planetary Computer includes data from the ECMWF Open Data (real time) program. See the [dataset](https://planetarycomputer.microsoft.com/dataset/ecmwf-forecast) page and [ECMWF Uesr Guide](https://confluence.ecmwf.int/display/UDOC/ECMWF+Open+Data+-+Real+Time) for more.\n", "\n", "Each item in this collection includes metadata about the *stream* (or forecasting system) and *type* that produced that particular dataset. Filter on these values to select the item of interest. For example, we can select data from the `wave` stream, with the `fc` type that are forecast `0h` hours out." ] }, { "cell_type": "code", "execution_count": null, "id": "667f654a", "metadata": {}, "outputs": [], "source": [ "import pystac_client\n", "import planetary_computer\n", "\n", "catalog = pystac_client.Client.open(\n", " \"https://planetarycomputer.microsoft.com/api/stac/v1\",\n", " modifier=planetary_computer.sign_inplace,\n", ")\n", "search = catalog.search(\n", " collections=[\"ecmwf-forecast\"],\n", " query={\n", " \"ecmwf:stream\": {\"eq\": \"wave\"},\n", " \"ecmwf:type\": {\"eq\": \"fc\"},\n", " \"ecmwf:step\": {\"eq\": \"0h\"},\n", " },\n", ")\n", "items = search.get_all_items()\n", "len(items)" ] }, { "cell_type": "markdown", "id": "b6cda672-d955-463d-aef3-81aaf84477e5", "metadata": {}, "source": [ "We'll select the most recent item, using the item's datetime." ] }, { "cell_type": "code", "execution_count": 2, "id": "e3ac3d8b-042a-42d0-8a57-383f4562da5e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "item = max(items, key=lambda item: item.datetime)\n", "item" ] }, { "cell_type": "markdown", "id": "faaea556-2f07-4951-abe8-8cbd9a46e5e9", "metadata": {}, "source": [ "This STAC item has two assets. One asset is the actual GRIB2 file with the data. The second asset is the \"index\" file, which contains information about the messages within the GRIB2 file." ] }, { "cell_type": "code", "execution_count": 3, "id": "c34fc8a3-1ecd-4b4b-848e-f506bdf1bcbe", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'https://ai4edataeuwest.blob.core.windows.net/ecmwf/20220310/00z/0p4-beta/wave/20220310000000-0h-wave-fc.grib2'" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "url = item.assets[\"data\"].href\n", "url" ] }, { "cell_type": "markdown", "id": "edde8193-7ba8-49b5-bfb7-707ab0edce8e", "metadata": {}, "source": [ "To open the file with xarray, we can download it locally and open it with `cfgrib`." ] }, { "cell_type": "code", "execution_count": 4, "id": "9dba8491-8e37-435e-abb8-0a73626c11d2", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:     (latitude: 451, longitude: 900)\n",
       "Coordinates:\n",
       "    time        datetime64[ns] 2022-03-10\n",
       "    step        timedelta64[ns] 00:00:00\n",
       "    meanSea     float64 0.0\n",
       "  * latitude    (latitude) float64 90.0 89.6 89.2 88.8 ... -89.2 -89.6 -90.0\n",
       "  * longitude   (longitude) float64 -180.0 -179.6 -179.2 ... 178.8 179.2 179.6\n",
       "    valid_time  datetime64[ns] 2022-03-10\n",
       "Data variables:\n",
       "    swh         (latitude, longitude) float32 ...\n",
       "    mwp         (latitude, longitude) float32 ...\n",
       "    mwd         (latitude, longitude) float32 ...\n",
       "    pp1d        (latitude, longitude) float32 ...\n",
       "    mp2         (latitude, longitude) float32 ...\n",
       "Attributes:\n",
       "    GRIB_edition:            2\n",
       "    GRIB_centre:             ecmf\n",
       "    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts\n",
       "    GRIB_subCentre:          0\n",
       "    Conventions:             CF-1.7\n",
       "    institution:             European Centre for Medium-Range Weather Forecasts\n",
       "    history:                 2022-03-10T19:02 GRIB to CDM+CF via cfgrib-0.9.1...
" ], "text/plain": [ "\n", "Dimensions: (latitude: 451, longitude: 900)\n", "Coordinates:\n", " time datetime64[ns] ...\n", " step timedelta64[ns] ...\n", " meanSea float64 ...\n", " * latitude (latitude) float64 90.0 89.6 89.2 88.8 ... -89.2 -89.6 -90.0\n", " * longitude (longitude) float64 -180.0 -179.6 -179.2 ... 178.8 179.2 179.6\n", " valid_time datetime64[ns] ...\n", "Data variables:\n", " swh (latitude, longitude) float32 ...\n", " mwp (latitude, longitude) float32 ...\n", " mwd (latitude, longitude) float32 ...\n", " pp1d (latitude, longitude) float32 ...\n", " mp2 (latitude, longitude) float32 ...\n", "Attributes:\n", " GRIB_edition: 2\n", " GRIB_centre: ecmf\n", " GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts\n", " GRIB_subCentre: 0\n", " Conventions: CF-1.7\n", " institution: European Centre for Medium-Range Weather Forecasts\n", " history: 2022-03-10T19:02 GRIB to CDM+CF via cfgrib-0.9.1..." ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import urllib.request\n", "import xarray as xr\n", "\n", "filename, _ = urllib.request.urlretrieve(url)\n", "\n", "ds = xr.open_dataset(filename, engine=\"cfgrib\")\n", "ds" ] }, { "cell_type": "markdown", "id": "9e481590-5e18-45ff-a3ae-ed0db41abf40", "metadata": {}, "source": [ "We can plot the various data variables, for example the significant height of combined wind waves and swell." ] }, { "cell_type": "code", "execution_count": 5, "id": "468e3062-f159-44cb-89ce-b2ded4e352d3", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "import cartopy.crs as ccrs\n", "\n", "projection = ccrs.Robinson()\n", "fig, ax = plt.subplots(figsize=(16, 9), subplot_kw=dict(projection=projection))\n", "\n", "ds.swh.plot(ax=ax, transform=ccrs.PlateCarree());" ] }, { "cell_type": "markdown", "id": "cc4f0715-89ec-415d-89f7-520ad7b79a24", "metadata": {}, "source": [ "Or the joint distribution between the wave period and height." ] }, { "cell_type": "code", "execution_count": 6, "id": "70780ca2-840c-42b8-850f-d51e6442dcfe", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import seaborn as sns\n", "\n", "grid = sns.jointplot(\n", " x=ds.mwp.data.ravel(), y=ds.swh.data.ravel(), alpha=0.25, marker=\".\", height=12\n", ")\n", "grid.ax_joint.set(xlabel=\"Mean wave period\", ylabel=\"Significant wave height\");" ] }, { "cell_type": "markdown", "id": "ba157759-147c-4247-ad4b-ad321da571c4", "metadata": {}, "source": [ "### GRIB2 files with multiple datasets \n", "\n", "Some GRIB2 files contain many messages that for separate xarray Datasets. For example, we can find the STAC item containing the atmospheric fields ensemble forecast (`stream=enfo`) from the ensemble forecast model (`type=ef`)." ] }, { "cell_type": "code", "execution_count": 7, "id": "0095f86f-3c76-4ab4-ac4f-4dbdb03d8159", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "189 matched\n" ] } ], "source": [ "search = catalog.search(\n", " collections=[\"ecmwf-forecast\"],\n", " query={\n", " \"ecmwf:stream\": {\"eq\": \"enfo\"},\n", " \"ecmwf:type\": {\"eq\": \"ef\"},\n", " \"ecmwf:step\": {\"eq\": \"0h\"},\n", " },\n", ")\n", "items = search.get_all_items()\n", "print(len(items), \"matched\")\n", "# select the most recent forecast\n", "item = max(items, key=lambda item: item.datetime)" ] }, { "cell_type": "code", "execution_count": 8, "id": "0e2d2cba-6052-405e-a5cd-84b0035488d7", "metadata": {}, "outputs": [], "source": [ "url = item.assets[\"data\"].href\n", "filename, _ = urllib.request.urlretrieve(url)" ] }, { "cell_type": "markdown", "id": "baa890b6-0ca6-4ca9-8b27-6954099a249d", "metadata": {}, "source": [ "If we provided just the `filename` to `xarray.open_dataset`, we'd get an error from `cfgrib` saying it can't form a valid DataArray from the file. That's because the GRIB2 file contains multiple data variables that don't form a neat hypercube. Provide `filter_by_keys` to indicate which subset of the data to read in." ] }, { "cell_type": "code", "execution_count": 9, "id": "cb28e64a-8376-41c7-ac29-a6bd3181d159", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:     (number: 50, latitude: 451, longitude: 900)\n",
       "Coordinates:\n",
       "  * number      (number) int64 1 2 3 4 5 6 7 8 9 ... 42 43 44 45 46 47 48 49 50\n",
       "    time        datetime64[ns] 2022-03-10T06:00:00\n",
       "    step        timedelta64[ns] 00:00:00\n",
       "    surface     float64 0.0\n",
       "  * latitude    (latitude) float64 90.0 89.6 89.2 88.8 ... -89.2 -89.6 -90.0\n",
       "  * longitude   (longitude) float64 -180.0 -179.6 -179.2 ... 178.8 179.2 179.6\n",
       "    valid_time  datetime64[ns] 2022-03-10T06:00:00\n",
       "Data variables:\n",
       "    tp          (number, latitude, longitude) float32 ...\n",
       "    sp          (number, latitude, longitude) float32 ...\n",
       "    skt         (number, latitude, longitude) float32 ...\n",
       "    ro          (number, latitude, longitude) float32 ...\n",
       "Attributes:\n",
       "    GRIB_edition:            2\n",
       "    GRIB_centre:             ecmf\n",
       "    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts\n",
       "    GRIB_subCentre:          0\n",
       "    Conventions:             CF-1.7\n",
       "    institution:             European Centre for Medium-Range Weather Forecasts\n",
       "    history:                 2022-03-10T19:03 GRIB to CDM+CF via cfgrib-0.9.1...
" ], "text/plain": [ "\n", "Dimensions: (number: 50, latitude: 451, longitude: 900)\n", "Coordinates:\n", " * number (number) int64 1 2 3 4 5 6 7 8 9 ... 42 43 44 45 46 47 48 49 50\n", " time datetime64[ns] ...\n", " step timedelta64[ns] ...\n", " surface float64 ...\n", " * latitude (latitude) float64 90.0 89.6 89.2 88.8 ... -89.2 -89.6 -90.0\n", " * longitude (longitude) float64 -180.0 -179.6 -179.2 ... 178.8 179.2 179.6\n", " valid_time datetime64[ns] ...\n", "Data variables:\n", " tp (number, latitude, longitude) float32 ...\n", " sp (number, latitude, longitude) float32 ...\n", " skt (number, latitude, longitude) float32 ...\n", " ro (number, latitude, longitude) float32 ...\n", "Attributes:\n", " GRIB_edition: 2\n", " GRIB_centre: ecmf\n", " GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts\n", " GRIB_subCentre: 0\n", " Conventions: CF-1.7\n", " institution: European Centre for Medium-Range Weather Forecasts\n", " history: 2022-03-10T19:03 GRIB to CDM+CF via cfgrib-0.9.1..." ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = xr.open_dataset(\n", " filename,\n", " engine=\"cfgrib\",\n", " filter_by_keys={\"dataType\": \"pf\", \"typeOfLevel\": \"surface\"},\n", ")\n", "ds" ] }, { "cell_type": "markdown", "id": "9587e2fd-a23f-4790-b250-4442497a60b5", "metadata": {}, "source": [ "Now we can plot the skin temperature, `skt`." ] }, { "cell_type": "code", "execution_count": null, "id": "a6aeb918-8f22-4400-bb31-e5efa3719e42", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import cartopy.crs as ccrs\n", "\n", "projection = projection = ccrs.Robinson()\n", "fig, ax = plt.subplots(figsize=(16, 9), subplot_kw=dict(projection=projection))\n", "\n", "ds.skt[0].plot(ax=ax, transform=ccrs.PlateCarree());" ] }, { "cell_type": "markdown", "id": "a9e7a71e-3bec-4c17-838d-8faa6bce8b37", "metadata": {}, "source": [ "There are many more streams and forecast types available. See the [catalog](http://planetarycomputer.microsoft.com/dataset/ecmwf-forecast) for more." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.10" } }, "nbformat": 4, "nbformat_minor": 5 }