{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f6c39c84",
   "metadata": {},
   "source": [
    "# Kerchunk, hvPlot, and Datashader: Visualizing datasets on-the-fly"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9eced552",
   "metadata": {},
   "source": [
    "## Overview\n",
    "   \n",
    "This notebook will demonstrate how to use Kerchunk with hvPlot and Datashader to lazily visualize a reference dataset in a streaming fashion.\n",
    "\n",
    "We will be building off the references generated through the notebook content from the[Pangeo_Forge](notebooks/generating_references/Pangeo_Forge.ipynb) notebook, so it's encouraged you first go through that.\n",
    "\n",
    "## Prerequisites\n",
    "| Concepts | Importance | Notes |\n",
    "| --- | --- | --- |\n",
    "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
    "| [Introduction to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Required | IO |\n",
    "| [Introduction to hvPlot](https://hvplot.holoviz.org/) | Required | Data Visualization |\n",
    "| [Introduction to Datashader](https://datashader.org/index.html) | Required | Big Data Visualization |\n",
    "- **Time to learn**: 10 minutes\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4043365b",
   "metadata": {},
   "source": [
    "## Motivation\n",
    "\n",
    "Using Kerchunk, we don't have to create a copy of the data--instead we create a collection of reference files, so that the original data files can be read as if they were Zarr.\n",
    "\n",
    "This enables visualization on-the-fly; simply pass in the URL to the dataset and use hvplot."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8e2c4765",
   "metadata": {},
   "source": [
    "### Getting to Know The Data\n",
    "\n",
    "`gridMET` is a high-resolution daily meteorological dataset covering CONUS from 1979-2023. It is produced by the Climatology Lab at UC Merced. In this example, we are going to look create a virtual Zarr dataset of a derived variable, Burn Index. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8d0f8f1",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "706c1b95",
   "metadata": {},
   "outputs": [],
   "source": [
    "import hvplot.xarray  # noqa\n",
    "import xarray as xr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7818440e",
   "metadata": {},
   "source": [
    "## Opening the Kerchunk Dataset\n",
    "\n",
    "Now, it's a matter of opening the Kerchunk dataset and calling `hvplot` with the `rasterize=True` keyword argument.\n",
    "\n",
    "If you're running this notebook locally, try zooming around the map by hovering over the plot and scrolling; it should update fairly quickly. Note, it will **not** update if you're viewing this on the docs page online as there is no backend server, but don't fret because there's a demo GIF below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce0f1766",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r 1 -n 1\n",
    "\n",
    "\n",
    "storage_options = {\n",
    "    \"remote_protocol\": \"http\",\n",
    "    \"skip_instance_cache\": True,\n",
    "}  # options passed to fsspec\n",
    "open_dataset_options = {\"chunks\": {}, \"decode_coords\": \"all\"}  # opens passed to xarray\n",
    "\n",
    "ds_kerchunk = xr.open_dataset(\n",
    "    \"references/Pangeo_Forge/reference.json\",\n",
    "    engine=\"kerchunk\",\n",
    "    storage_options=storage_options,\n",
    "    **open_dataset_options,\n",
    ")\n",
    "\n",
    "display(ds_kerchunk.hvplot(\"lon\", \"lat\", rasterize=True))  # noqa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e53531df",
   "metadata": {},
   "source": [
    "<img src=\"../images/kerchunk.gif\" width=400 alt=\"Kerchunk Zoom\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2c56945",
   "metadata": {},
   "source": [
    "## Comparing Against THREDDS\n",
    "\n",
    "Now, we will be repeating the previous cell, but with THREDDS.\n",
    "\n",
    "Note how the initial load is longer.\n",
    "\n",
    "If you're running the notebook locally (or a demo GIF below), zooming in/out also takes longer to finish buffering as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "368ac51d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r 1 -n 1\n",
    "\n",
    "\n",
    "def url_gen(year):\n",
    "    return (\n",
    "        f\"http://thredds.northwestknowledge.net:8080/thredds/dodsC/MET/bi/bi_{year}.nc\"\n",
    "    )\n",
    "\n",
    "\n",
    "years = list(range(1979, 1980))\n",
    "urls_list = [url_gen(year) for year in years]\n",
    "netcdf_ds = xr.open_mfdataset(urls_list, engine=\"netcdf4\")\n",
    "display(netcdf_ds.hvplot(\"lon\", \"lat\", rasterize=True))  # noqa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c0dd0cd",
   "metadata": {},
   "source": [
    "<img src=\"../images/thredds.gif\" width=400 alt=\"THREDDS Zoom\"></img>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "b8afa8ad8f3d27e858f1dbdc03ccd45fac432e2a03d4a98c501e197170438b83"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}