{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1b5d37f3",
   "metadata": {},
   "source": [
    "# 01 — REF concepts\n",
    "\n",
    "This notebook introduces the core vocabulary of the **Rapid Evaluation Framework (REF)**.\n",
    "By the end you will know what a *diagnostic*, *provider*, *execution*, *metric* and *dataset* are, and how they fit together.\n",
    "\n",
    "**Prerequisites:** none. This is the place to start.\n",
    "\n",
    "**What you need:** an internet connection as we read from the public REF API."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dea4894f",
   "metadata": {},
   "source": [
    "## What is the REF?\n",
    "\n",
    "The REF performs evaluation of climate datasets, much like a CI/CD pipeline runs tests against code. \n",
    "As new climate model output is published, the REF evaluates it against reference data \n",
    "and produces figures and metrics — in near-real time.\n",
    "\n",
    "The public deployment currently evaluates CMIP6 dataset, but will include CMIP7 Assessment Fast Track data\n",
    "as they become available.\n",
    "\n",
    "The REF has a dashboard (<https://dashboard.climate-ref.org>) which offers a curated view of the results from the REF. \n",
    "This dashboard uses an API (<https://api.climate-ref.org>) to provide access to the results in a structured way.\n",
    "\n",
    "We can use this same API to make our own view of the data. As part of the API we also publish a [schema](https://api.climate-ref.org/docs) that describes the different endpoints that can be queried and the shape of the response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1df88e89",
   "metadata": {},
   "source": [
    "Throughout these notebooks we talk to the API via an \"SDK\" which is a small software package generated from the schema of the API.\n",
    "This SDK allows us to make requests to the API as function calls without having to directly make HTTP requests.\n",
    "\n",
    "A small script `scripts/generate_client.sh` shows how to built the SDK.\n",
    "\n",
    "> If you are runnning this on Binder, this has been pre-built for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "05ac27ad",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T02:09:38.290850Z",
     "iopub.status.busy": "2026-10-28T02:09:38.290790Z",
     "iopub.status.idle": "2026-10-28T02:09:38.653608Z",
     "shell.execute_reply": "2026-10-28T02:09:38.653187Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Client(raise_on_unexpected_status=False, _base_url='https://api.climate-ref.org', _cookies={}, _headers={}, _timeout=None, _verify_ssl=True, _follow_redirects=False, _httpx_args={}, _client=None, _async_client=None)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ref_tutorials import get_client\n",
    "\n",
    "client = get_client()\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "871f914b",
   "metadata": {},
   "source": [
    "## Datasets\n",
    "\n",
    "Datasets containing CMORised climate model output, and the reference data it is compared to are a key input to the REF.\n",
    "The REF tracks which datasets are available and which diagnostics have already been run."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19c3c017",
   "metadata": {},
   "source": [
    "## Diagnostics\n",
    "\n",
    "A **diagnostic** is a single, well-defined evaluation to understand a component of the earth system.\n",
    "For example \"the global mean surface temperature timeseries\" or \"the Atlantic overturning circulation strength\".\n",
    "\n",
    "Let's list the diagnostics the REF currently provides:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8196f36e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T02:09:38.654782Z",
     "iopub.status.busy": "2026-10-28T02:09:38.654642Z",
     "iopub.status.idle": "2026-10-28T02:09:43.260142Z",
     "shell.execute_reply": "2026-10-28T02:09:43.259517Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47 diagnostics available. Showing the first 10:\n",
      "\n",
      "  annual-cycle                        Annual Cycle Analysis\n",
      "  sea-ice-area-basic                  Arctic and Antarctic Sea Ice Area Seasonal Cycle\n",
      "  amoc-rapid                          Atlantic Meridional Overturning Circulation (RAPID)\n",
      "  burntfractionall-gfed               Burnt Fraction (GFED)\n",
      "  climate-drivers-for-fire            Climate Drivers for Fire\n",
      "  climate-at-global-warming-levels    Climate at Global Warming Levels\n",
      "  cloud-radiative-effects             Cloud Radiative Effects\n",
      "  cloud-scatterplots-reference        Cloud Scatterplots for Reference dataset\n",
      "  cloud-scatterplots-clwvi-pr         Cloud-Precipitation Scatterplots (clwvi vs pr)\n",
      "  cloud-scatterplots-clivi-lwcre      Cloud-Radiation Scatterplots (clivi vs lwcre)\n"
     ]
    }
   ],
   "source": [
    "from climate_ref_client.api.diagnostics import diagnostics_list\n",
    "\n",
    "diagnostics = diagnostics_list.sync(client=client).data\n",
    "print(f\"{len(diagnostics)} diagnostics available. Showing the first 10:\\n\")\n",
    "for diagnostic in sorted(diagnostics, key=lambda d: d.name)[:10]:\n",
    "    print(f\"  {diagnostic.slug:35s} {diagnostic.name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c06cfd",
   "metadata": {},
   "source": [
    "Every diagnostic has a human-readable `name`, a short machine-friendly `slug`, and a\n",
    "`description`. Let's look at one in full:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3389ca04",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T02:09:43.261876Z",
     "iopub.status.busy": "2026-10-28T02:09:43.261750Z",
     "iopub.status.idle": "2026-10-28T02:09:43.263971Z",
     "shell.execute_reply": "2026-10-28T02:09:43.263548Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name:        Climate at Global Warming Levels\n",
      "slug:        climate-at-global-warming-levels\n",
      "description: Calculate climate variables at global warming levels (e.g. 1.5C, 2C, 3C, 4C above pre-industrial).\n"
     ]
    }
   ],
   "source": [
    "example = diagnostics[0]\n",
    "print(\"name:       \", example.name)\n",
    "print(\"slug:       \", example.slug)\n",
    "print(\"description:\", example.description.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9697babb",
   "metadata": {},
   "source": [
    "## Providers\n",
    "\n",
    "Diagnostics are grouped into **providers**. A **provider** is a package that knows how to compute a family of diagnostics.\n",
    "\n",
    "For the Assessment Fast Track we use:\n",
    "\n",
    "- ESMValTool\n",
    "- PCMDI Metrics Package (PMP)\n",
    "- ILAMB/IOMB\n",
    "\n",
    "The REF orchestrates providers and does not compute diagnostics itself.\n",
    "Each provider is a thin wrapper around the upstream diagnostics package.\n",
    "\n",
    "Each diagnostic tells you which provider it belongs to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "45bfc188",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T02:09:43.265437Z",
     "iopub.status.busy": "2026-10-28T02:09:43.265311Z",
     "iopub.status.idle": "2026-10-28T02:09:43.267845Z",
     "shell.execute_reply": "2026-10-28T02:09:43.267321Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  esmvaltool           ESMValTool  (24 diagnostics)\n",
      "  ilamb                ILAMB  (13 diagnostics)\n",
      "  pmp                  PMP  (10 diagnostics)\n"
     ]
    }
   ],
   "source": [
    "providers = {d.provider.slug: d.provider.name for d in diagnostics}\n",
    "for slug, name in sorted(providers.items()):\n",
    "    count = sum(1 for d in diagnostics if d.provider.slug == slug)\n",
    "    print(f\"  {slug:20s} {name}  ({count} diagnostics)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be0528dc",
   "metadata": {},
   "source": [
    "## Execution groups and Executions\n",
    "\n",
    "An **execution** is one run of a diagnostic against one specific group of datasets.\n",
    "This execution has information about the datasets that were used to run it, the diagnostic and the outputs.\n",
    "\n",
    "A single diagnostic is typically executed many times on different sets of data.\n",
    "This generally depends what is being calculated, but this is often once per model variant.\n",
    "Each of these subsets is an *execution group*.\n",
    "\n",
    "Each execution group has a unique key that describes it.\n",
    "If any datasets in an execution group change, then a new execution is performed.\n",
    "\n",
    "We can list the execution groups available for the Global Warming Level diagnostic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fda2000d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T02:09:43.269048Z",
     "iopub.status.busy": "2026-10-28T02:09:43.268950Z",
     "iopub.status.idle": "2026-10-28T02:09:46.306034Z",
     "shell.execute_reply": "2026-10-28T02:09:46.305373Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Diagnostic: Climate at Global Warming Levels\n",
      "Execution groups: 4\n",
      "Execution group keys: ['cmip6_ssp126', 'cmip6_ssp245', 'cmip6_ssp370', 'cmip6_ssp585']\n"
     ]
    }
   ],
   "source": [
    "from climate_ref_client.api.diagnostics import diagnostics_list_execution_groups\n",
    "\n",
    "execution_groups = diagnostics_list_execution_groups.sync(\n",
    "    client=client, provider_slug=example.provider.slug, diagnostic_slug=example.slug\n",
    ").data\n",
    "\n",
    "first_execution_group = execution_groups[0]\n",
    "\n",
    "print(f\"Diagnostic: {example.name}\")\n",
    "print(f\"Execution groups: {len(execution_groups)}\")\n",
    "print(f\"Execution group keys: {[eg.key for eg in execution_groups]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dce894b0",
   "metadata": {},
   "source": [
    "Within each execution group, there may be multiple executions.\n",
    "We pick the most recent successful one.\n",
    "For this execution we can access the outputs that were generated as well as a URL for downloading them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c0e32afa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T02:09:46.307706Z",
     "iopub.status.busy": "2026-10-28T02:09:46.307529Z",
     "iopub.status.idle": "2026-10-28T02:09:46.310670Z",
     "shell.execute_reply": "2026-10-28T02:09:46.309969Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_mean_1.5.png https://api.climate-ref.org/api/v1/results/2284307\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_mean_2.0.png https://api.climate-ref.org/api/v1/results/2284308\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_mean_3.0.png https://api.climate-ref.org/api/v1/results/2284309\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_stdev_1.5.png https://api.climate-ref.org/api/v1/results/2284310\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_stdev_2.0.png https://api.climate-ref.org/api/v1/results/2284311\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_stdev_3.0.png https://api.climate-ref.org/api/v1/results/2284312\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_mean_1.5.png https://api.climate-ref.org/api/v1/results/2284313\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_mean_2.0.png https://api.climate-ref.org/api/v1/results/2284314\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_mean_3.0.png https://api.climate-ref.org/api/v1/results/2284315\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_stdev_1.5.png https://api.climate-ref.org/api/v1/results/2284316\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_stdev_2.0.png https://api.climate-ref.org/api/v1/results/2284317\n",
      "  executions/recipe_20261026_015959/plots/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_stdev_3.0.png https://api.climate-ref.org/api/v1/results/2284318\n",
      "  executions/recipe_20261026_015959/work/calculate_gwl_exceedance_years/gwl_exceedance_calculation/GWL_exceedance_years.csv https://api.climate-ref.org/api/v1/results/2284319\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_mean_1.5.nc https://api.climate-ref.org/api/v1/results/2284320\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_mean_2.0.nc https://api.climate-ref.org/api/v1/results/2284321\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_mean_3.0.nc https://api.climate-ref.org/api/v1/results/2284322\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_stdev_1.5.nc https://api.climate-ref.org/api/v1/results/2284323\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_stdev_2.0.nc https://api.climate-ref.org/api/v1/results/2284324\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_pr/plot_gwl_stats/CMIP6_mm_stdev_3.0.nc https://api.climate-ref.org/api/v1/results/2284325\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_mean_1.5.nc https://api.climate-ref.org/api/v1/results/2284326\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_mean_2.0.nc https://api.climate-ref.org/api/v1/results/2284327\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_mean_3.0.nc https://api.climate-ref.org/api/v1/results/2284328\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_stdev_1.5.nc https://api.climate-ref.org/api/v1/results/2284329\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_stdev_2.0.nc https://api.climate-ref.org/api/v1/results/2284330\n",
      "  executions/recipe_20261026_015959/work/gwl_mean_plots_tas/plot_gwl_stats/CMIP6_mm_stdev_3.0.nc https://api.climate-ref.org/api/v1/results/2284331\n",
      "  executions/recipe_20261026_015959/index.html https://api.climate-ref.org/api/v1/results/2284332\n"
     ]
    }
   ],
   "source": [
    "from ref_tutorials import latest_successful_execution\n",
    "\n",
    "execution = latest_successful_execution(first_execution_group)\n",
    "if execution is None:\n",
    "    print(\"This execution group has no successful execution yet, so there are no outputs to show.\")\n",
    "else:\n",
    "    for o in execution.outputs:\n",
    "        print(f\"  {o.filename:20s} {o.url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3d3f48d",
   "metadata": {},
   "source": [
    "## Metrics\n",
    "\n",
    "An execution produces output. That output comes in two main shapes:\n",
    "\n",
    "- **Metric values** — numbers that summarise a property of a model. \n",
    "  A *scalar* is a single number (e.g. a root-mean-square error) or a *series* is a number per index point\n",
    "  (e.g. a seasonal cycle).\n",
    "- **Files** — NetCDF data and figures for deeper analysis or custom plotting.\n",
    "\n",
    "The REF ingests all of these outputs into its database so they can be queried by the API.\n",
    "\n",
    "Notebook 02 shows how to retrieve metric values, and notebook 03 turns them into a figure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4fbac3c",
   "metadata": {},
   "source": [
    "## Recap\n",
    "\n",
    "| Term                | Meaning                                                                       |\n",
    "| ------------------- | ----------------------------------------------------------------------------- |\n",
    "| **Provider**        | A package that computes a family of diagnostics (ESMValTool, PMP, ILAMB, ...) |\n",
    "| **Diagnostic**      | A single well-defined evaluation                                              |\n",
    "| **Dataset**         | Climate model output or reference data a diagnostic runs against              |\n",
    "| **Execution**       | One run of a diagnostic against one group of datasets                         |\n",
    "| **Metric value**    | A scalar or series result summarising a model                                 |\n",
    "\n",
    "**Next:** [02 — Querying the REF API](02-querying-the-api.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba1f51af-f582-40de-98ae-662f3523f71a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "climate-ref-tutorials (3.14.4)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}