{ "cells": [ { "cell_type": "markdown", "id": "3dbac58d-0638-4e3a-a594-efca35d34a7e", "metadata": {}, "source": [ "## Creating a cross-model ensemble using STAC" ] }, { "cell_type": "markdown", "id": "e0df9241-0c9d-4739-90f5-35b3f3b94092", "metadata": {}, "source": [ "This tutorial builds a cross-collection ensemble of GDPCIR bias corrected and downscaled data, and plots a single variable time series for the ensemble." ] }, { "cell_type": "code", "execution_count": 1, "id": "f6c291ba-5b61-41d9-bde9-3e49afddebaf", "metadata": {}, "outputs": [], "source": [ "# required to locate and authenticate with the stac collection\n", "import planetary_computer\n", "import pystac_client\n", "\n", "# required to load a zarr array using xarray\n", "import xarray as xr\n", "\n", "# optional imports used in this notebook\n", "import pandas as pd\n", "from dask.diagnostics import ProgressBar\n", "from tqdm.auto import tqdm" ] }, { "cell_type": "markdown", "id": "815bee61-c752-465b-ace8-48d7a6bff367", "metadata": {}, "source": [ "### Understanding the GDPCIR collections\n", "\n", "The [CIL-GDPCIR datasets](https://planetarycomputer.microsoft.com/dataset/group/cil-gdpcir) are grouped into two collections, depending on the license the data are provided under.\n", "\n", "- [CIL-GDPCIR-CC0](https://planetarycomputer.microsoft.com/dataset/cil-gdpcir-cc0) - provided in public domain using a [CC 1.0 Universal Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/)\n", "- [CIL-GDPCIR-CC-BY](https://planetarycomputer.microsoft.com/dataset/cil-gdpcir-cc-by) - provided under a [CC Attribution 4.0 License](https://creativecommons.org/licenses/by/4.0/)\n", "\n", "Note that the first group, CC0, places no restrictions on the data. CC-BY 4.0 requires citations of the climate models these datasets are derived from. See the [ClimateImpactLab/downscaleCMIP6 README](github.com/ClimateImpactLab/downscaleCMIP6) for the citation information for each GCM.\n", "\n", "Also, note that none of the descriptions of these licenses on this page, in this repository, and associated with this repository constitute legal advice. We are highlighting some of the key terms of these licenses, but this information should not be considered a replacement for the actual license terms, which are provided on the Creative Commons website at the links above.\n", "\n", "### Structure of the STAC collection\n", "\n", "The data assets in this collection are a set of [Zarr](https://zarr.readthedocs.io/) groups which can be opend by tools like [xarray](https://xarray.pydata.org/). Each Zarr group contains a single data variable (either `pr`, `tasmax`, or `tasmin`). The Planetary Computer provides a single STAC item per experiment, and each STAC item has one asset per data variable.\n", "\n", "Altogether, the collection is just over 21TB, with 247,997 individual files. The STAC collection is here to help search and make sense of this huge archive!\n", "\n", "For example, let's take a look at the CC0 collection:" ] }, { "cell_type": "code", "execution_count": 2, "id": "3dd5d503-3e05-4248-97bf-137cd3e61448", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", " \n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "catalog = pystac_client.Client.open(\n", " \"https://planetarycomputer.microsoft.com/api/stac/v1/\",\n", " modifier=planetary_computer.sign_inplace,\n", ")\n", "collection_cc0 = catalog.get_collection(\"cil-gdpcir-cc0\")\n", "collection_cc0" ] }, { "cell_type": "code", "execution_count": 3, "id": "c664b2c3-4cdc-4c58-b3d8-7415a154deee", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'cmip6:variable': ['pr', 'tasmax', 'tasmin'],\n", " 'cmip6:source_id': ['FGOALS-g3', 'INM-CM4-8', 'INM-CM5-0'],\n", " 'cmip6:experiment_id': ['historical', 'ssp126', 'ssp245', 'ssp370', 'ssp585'],\n", " 'cmip6:institution_id': ['BCC',\n", " 'CAS',\n", " 'CCCma',\n", " 'CMCC',\n", " 'CSIRO',\n", " 'CSIRO-ARCCSS',\n", " 'DKRZ',\n", " 'EC-Earth-Consortium',\n", " 'INM',\n", " 'MIROC',\n", " 'MOHC',\n", " 'MPI-M',\n", " 'NCC',\n", " 'NOAA-GFDL',\n", " 'NUIST']}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "collection_cc0.summaries.to_dict()" ] }, { "cell_type": "markdown", "id": "524f0c97-03ba-42f0-b52c-3904d4555dd8", "metadata": {}, "source": [ "The CC0 (Public Domain) collection has three models, from which a historical scenario and four future simulations are available. \n", "\n", "*Note that not all models provide all simulations. See the [ClimateImpactLab/downscaleCMIP6 README](https://github.com/ClimateImpactLab/downscaleCMIP6) for a list of the available model/scenario/variable combinations.*" ] }, { "cell_type": "markdown", "id": "0ff25073-5178-4b35-bfa1-a9c78c6b6073", "metadata": { "tags": [] }, "source": [ "### Querying the STAC API\n", "\n", "Use the Planetary Computer STAC API to find the exact data you want. You'll most likely want to query on the controlled vocabularies fields, under the `cmip6:` prefix. See the collection summary for the set of allowed values for each of those." ] }, { "cell_type": "markdown", "id": "408a8935-8070-4215-8e55-61828e85d976", "metadata": {}, "source": [ "### Combining collections to form a custom ensemble" ] }, { "cell_type": "markdown", "id": "233982e1-7081-4c8c-b61d-32d9366552bc", "metadata": {}, "source": [ "As an example, if you would like to use both the CC0 and CC-BY collections, you can combine them as follows:" ] }, { "cell_type": "code", "execution_count": 6, "id": "0f90eed5-c378-4853-96d8-21cca319347e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "20" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "search = catalog.search(\n", " collections=[\"cil-gdpcir-cc0\", \"cil-gdpcir-cc-by\"],\n", " query={\"cmip6:experiment_id\": {\"eq\": \"ssp370\"}},\n", ")\n", "ensemble = search.item_collection()\n", "len(ensemble)" ] }, { "cell_type": "code", "execution_count": 7, "id": "2a179418-5e9e-44d5-b899-9512740bad87", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Counter({'cil-gdpcir-cc-by': 17, 'cil-gdpcir-cc0': 3})" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import collections\n", "\n", "collections.Counter(x.collection_id for x in ensemble)" ] }, { "cell_type": "markdown", "id": "0b9600a8-0270-41fe-b689-caf17a47254f", "metadata": {}, "source": [ "### Reading a single variable across models into xarray" ] }, { "cell_type": "code", "execution_count": 8, "id": "58acf247-7e52-488f-818e-258ab54ee8c6", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "aea638d343ca4218b02188648c7469fa", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/20 [00:00\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 3TB\n",
       "Dimensions:  (lat: 720, lon: 1440, model: 20, time: 31390)\n",
       "Coordinates:\n",
       "  * lat      (lat) float64 6kB -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88\n",
       "  * lon      (lon) float64 12kB -179.9 -179.6 -179.4 ... 179.4 179.6 179.9\n",
       "  * time     (time) object 251kB 2015-01-01 12:00:00 ... 2100-12-31 12:00:00\n",
       "  * model    (model) object 160B 'GFDL-ESM4' 'NorESM2-MM' ... 'BCC-CSM2-MR'\n",
       "Data variables:\n",
       "    tasmax   (model, time, lat, lon) float32 3TB dask.array<chunksize=(1, 365, 360, 360), meta=np.ndarray>\n",
       "Attributes: (12/25)\n",
       "    contact:                      climatesci@rhg.com\n",
       "    dc6_bias_correction_method:   Quantile Delta Method (QDM)\n",
       "    dc6_citation:                 Please refer to https://github.com/ClimateI...\n",
       "    dc6_data_version:             v20211231\n",
       "    dc6_dataset_name:             Rhodium Group/Climate Impact Lab Global Dow...\n",
       "    dc6_description:              The prefix dc6 is the project-specific abbr...\n",
       "    ...                           ...\n",
       "    realization_index:            1\n",
       "    realm:                        atmos\n",
       "    sub_experiment:               none\n",
       "    sub_experiment_id:            none\n",
       "    table_id:                     day\n",
       "    variable_id:                  tasmax
" ], "text/plain": [ " Size: 3TB\n", "Dimensions: (lat: 720, lon: 1440, model: 20, time: 31390)\n", "Coordinates:\n", " * lat (lat) float64 6kB -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88\n", " * lon (lon) float64 12kB -179.9 -179.6 -179.4 ... 179.4 179.6 179.9\n", " * time (time) object 251kB 2015-01-01 12:00:00 ... 2100-12-31 12:00:00\n", " * model (model) object 160B 'GFDL-ESM4' 'NorESM2-MM' ... 'BCC-CSM2-MR'\n", "Data variables:\n", " tasmax (model, time, lat, lon) float32 3TB dask.array\n", "Attributes: (12/25)\n", " contact: climatesci@rhg.com\n", " dc6_bias_correction_method: Quantile Delta Method (QDM)\n", " dc6_citation: Please refer to https://github.com/ClimateI...\n", " dc6_data_version: v20211231\n", " dc6_dataset_name: Rhodium Group/Climate Impact Lab Global Dow...\n", " dc6_description: The prefix dc6 is the project-specific abbr...\n", " ... ...\n", " realization_index: 1\n", " realm: atmos\n", " sub_experiment: none\n", " sub_experiment_id: none\n", " table_id: day\n", " variable_id: tasmax" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "all_datasets" ] }, { "cell_type": "markdown", "id": "c0d12471-c337-4eec-adca-3dbd71a6d201", "metadata": {}, "source": [ "### Subsetting the data\n", "\n", "Now that the metadata has been loaded into xarray, you can use xarray's methods for [Indexing and Selecting Data](https://xarray.pydata.org/en/latest/user-guide/indexing.html) to extract the subset the arrays to the portions meaningful to your analysis.\n", "\n", "Note that the data has not been read yet - this is simply working with the coordinates to schedule the task graph using [dask](https://docs.xarray.dev/en/latest/user-guide/dask.html)." ] }, { "cell_type": "code", "execution_count": 10, "id": "cf93fc89-6f1d-424c-a403-88430112b2ef", "metadata": {}, "outputs": [], "source": [ "# let's select a subset of the data for the first five days of 2020 over Japan.\n", "# Thanks to https://gist.github.com/graydon/11198540 for the bounding box!\n", "subset = all_datasets.tasmax.sel(\n", " lon=slice(129.408463169, 145.543137242),\n", " lat=slice(31.0295791692, 45.5514834662),\n", " time=slice(\"2020-01-01\", \"2020-01-05\"),\n", ")" ] }, { "cell_type": "code", "execution_count": 11, "id": "d9032269-cdb4-4a10-91a4-f373b43f527d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[########################################] | 100% Completed | 8.90 ss\n" ] } ], "source": [ "with ProgressBar():\n", " subset = subset.compute()" ] }, { "cell_type": "code", "execution_count": 12, "id": "5b186f7b-5816-4a9e-bd18-6a196492ad42", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray 'tasmax' (model: 20, time: 5, lat: 58, lon: 64)> Size: 1MB\n",
       "array([[[[285.73315, 285.4604 , 285.3989 , ..., 288.46887, 288.45538,\n",
       "          288.3411 ],\n",
       "         [285.4088 , 284.85898, 284.64523, ..., 288.25568, 288.40067,\n",
       "          288.26636],\n",
       "         [285.042  , 284.65338, 284.3192 , ..., 288.09857, 288.33298,\n",
       "          288.17307],\n",
       "         ...,\n",
       "         [255.6336 , 255.90742, 255.76608, ..., 270.9189 , 271.3525 ,\n",
       "          271.49866],\n",
       "         [255.53918, 256.2673 , 257.10635, ..., 271.33408, 270.98483,\n",
       "          271.3151 ],\n",
       "         [254.87746, 255.66376, 256.50848, ..., 271.04498, 271.05704,\n",
       "          271.30338]],\n",
       "\n",
       "        [[285.91162, 285.58484, 286.10092, ..., 292.35254, 292.3309 ,\n",
       "          292.34427],\n",
       "         [285.565  , 284.94022, 285.75223, ..., 292.1405 , 292.2511 ,\n",
       "          292.22943],\n",
       "         [285.25647, 284.79587, 285.2175 , ..., 291.90317, 292.11487,\n",
       "          292.16623],\n",
       "...\n",
       "         [263.99496, 263.98923, 265.9909 , ..., 272.94186, 270.91608,\n",
       "          271.09778],\n",
       "         [263.7273 , 264.36838, 267.27963, ..., 271.01   , 270.0902 ,\n",
       "          270.34082],\n",
       "         [262.84192, 263.58743, 266.7099 , ..., 270.6768 , 270.26273,\n",
       "          270.4232 ]],\n",
       "\n",
       "        [[289.9362 , 289.80225, 288.88925, ..., 291.64294, 291.30453,\n",
       "          291.20703],\n",
       "         [289.503  , 289.25168, 288.7057 , ..., 291.51035, 291.37308,\n",
       "          291.30908],\n",
       "         [289.054  , 288.8851 , 287.89902, ..., 291.3675 , 291.34402,\n",
       "          291.34195],\n",
       "         ...,\n",
       "         [265.43088, 265.4783 , 268.89062, ..., 270.09174, 269.90125,\n",
       "          270.14105],\n",
       "         [264.06845, 264.98584, 268.8448 , ..., 270.29214, 269.33145,\n",
       "          269.47906],\n",
       "         [263.4161 , 264.3746 , 268.24796, ..., 270.58902, 269.15533,\n",
       "          269.333  ]]]], dtype=float32)\n",
       "Coordinates:\n",
       "  * lat      (lat) float64 464B 31.12 31.38 31.62 31.88 ... 44.88 45.12 45.38\n",
       "  * lon      (lon) float64 512B 129.6 129.9 130.1 130.4 ... 144.9 145.1 145.4\n",
       "  * time     (time) object 40B 2020-01-01 12:00:00 ... 2020-01-05 12:00:00\n",
       "  * model    (model) object 160B 'GFDL-ESM4' 'NorESM2-MM' ... 'BCC-CSM2-MR'\n",
       "Attributes:\n",
       "    cell_measures:  area: areacella\n",
       "    interp_method:  conserve_order2\n",
       "    long_name:      Daily Maximum Near-Surface Air Temperature\n",
       "    standard_name:  air_temperature\n",
       "    units:          K\n",
       "    comment:        maximum near-surface (usually, 2 meter) air temperature (...
" ], "text/plain": [ " Size: 1MB\n", "array([[[[285.73315, 285.4604 , 285.3989 , ..., 288.46887, 288.45538,\n", " 288.3411 ],\n", " [285.4088 , 284.85898, 284.64523, ..., 288.25568, 288.40067,\n", " 288.26636],\n", " [285.042 , 284.65338, 284.3192 , ..., 288.09857, 288.33298,\n", " 288.17307],\n", " ...,\n", " [255.6336 , 255.90742, 255.76608, ..., 270.9189 , 271.3525 ,\n", " 271.49866],\n", " [255.53918, 256.2673 , 257.10635, ..., 271.33408, 270.98483,\n", " 271.3151 ],\n", " [254.87746, 255.66376, 256.50848, ..., 271.04498, 271.05704,\n", " 271.30338]],\n", "\n", " [[285.91162, 285.58484, 286.10092, ..., 292.35254, 292.3309 ,\n", " 292.34427],\n", " [285.565 , 284.94022, 285.75223, ..., 292.1405 , 292.2511 ,\n", " 292.22943],\n", " [285.25647, 284.79587, 285.2175 , ..., 291.90317, 292.11487,\n", " 292.16623],\n", "...\n", " [263.99496, 263.98923, 265.9909 , ..., 272.94186, 270.91608,\n", " 271.09778],\n", " [263.7273 , 264.36838, 267.27963, ..., 271.01 , 270.0902 ,\n", " 270.34082],\n", " [262.84192, 263.58743, 266.7099 , ..., 270.6768 , 270.26273,\n", " 270.4232 ]],\n", "\n", " [[289.9362 , 289.80225, 288.88925, ..., 291.64294, 291.30453,\n", " 291.20703],\n", " [289.503 , 289.25168, 288.7057 , ..., 291.51035, 291.37308,\n", " 291.30908],\n", " [289.054 , 288.8851 , 287.89902, ..., 291.3675 , 291.34402,\n", " 291.34195],\n", " ...,\n", " [265.43088, 265.4783 , 268.89062, ..., 270.09174, 269.90125,\n", " 270.14105],\n", " [264.06845, 264.98584, 268.8448 , ..., 270.29214, 269.33145,\n", " 269.47906],\n", " [263.4161 , 264.3746 , 268.24796, ..., 270.58902, 269.15533,\n", " 269.333 ]]]], dtype=float32)\n", "Coordinates:\n", " * lat (lat) float64 464B 31.12 31.38 31.62 31.88 ... 44.88 45.12 45.38\n", " * lon (lon) float64 512B 129.6 129.9 130.1 130.4 ... 144.9 145.1 145.4\n", " * time (time) object 40B 2020-01-01 12:00:00 ... 2020-01-05 12:00:00\n", " * model (model) object 160B 'GFDL-ESM4' 'NorESM2-MM' ... 'BCC-CSM2-MR'\n", "Attributes:\n", " cell_measures: area: areacella\n", " interp_method: conserve_order2\n", " long_name: Daily Maximum Near-Surface Air Temperature\n", " standard_name: air_temperature\n", " units: K\n", " comment: maximum near-surface (usually, 2 meter) air temperature (..." ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset" ] }, { "cell_type": "markdown", "id": "6c5d4731-0bd5-4575-a1fc-7f7af7d1781f", "metadata": {}, "source": [ "At this point, you could do anything you like with the data. See the great [xarray getting started guide](https://xarray.pydata.org/en/latest/getting-started-guide/quick-overview.html#) for more information. For now, we'll plot it all!" ] }, { "cell_type": "code", "execution_count": 13, "id": "36820786-32c3-426c-85d5-448e2aad877b", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "subset.plot(row=\"model\", col=\"time\");" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": { "01a366a69a7244bfb320e08e5b276a1d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": { "description_width": "", "font_size": null, "text_color": null } }, "01f590d0198d45d686d6a055ec9a2380": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "0574dd5ea3ea49a9998e544607eff60d": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "1b28dfd2c25444ebb50cbc4e7e3dee3f": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "2fd960365dc043619ec9aaf846823351": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {} }, "30d9373b281b42c0ae5c7ddc624b7378": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_01f590d0198d45d686d6a055ec9a2380", "max": 20, "style": "IPY_MODEL_d8d16e9be9bf459b9600b7d192e25501", "value": 20 } }, "aea638d343ca4218b02188648c7469fa": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_eef97ce4416a457c9476361f92ebc8d3", "IPY_MODEL_30d9373b281b42c0ae5c7ddc624b7378", "IPY_MODEL_e4f0ff2a28a94979b55bb48bf270e6ab" ], "layout": "IPY_MODEL_1b28dfd2c25444ebb50cbc4e7e3dee3f" } }, "d8d16e9be9bf459b9600b7d192e25501": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "e4f0ff2a28a94979b55bb48bf270e6ab": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_0574dd5ea3ea49a9998e544607eff60d", "style": "IPY_MODEL_fa274a69282043b695857a970bbab01b", "value": " 20/20 [00:32<00:00,  1.36s/it]" } }, "eef97ce4416a457c9476361f92ebc8d3": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_2fd960365dc043619ec9aaf846823351", "style": "IPY_MODEL_01a366a69a7244bfb320e08e5b276a1d", "value": "100%" } }, "fa274a69282043b695857a970bbab01b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": { "description_width": "", "font_size": null, "text_color": null } } }, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }