{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "09ff1329-d043-41c5-9e76-78eda95f102a",
   "metadata": {},
   "source": [
    "## Fetching supplimentary model input from the Planetary Computer\n",
    "\n",
    "This notebook produces additional input layers for the training data used in the [sentinel 1 flood detection](https://www.drivendata.org/competitions/81/detect-flood-water/) competition run by DrivenData. If fetches JRC Global Surface Water and NASADEM elevation data from the Planetary Computer (PC) STAC API and creates pixel-aligned chips that match what will be used in the evaluation process for the competition.\n",
    "\n",
    "The notebook will iterate through chip paths and query the PC STAC API for the `nasadem` and `jrc-gsw` Collections. It then creates a set of GeoTiffs by \"coregistering\" the raster data with the chip GeoTIFF, so that all of the additional input layers have the same CRS, bounds, and resolution as the chip. These additional layers are then saved alongside the training chip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bfb8028-2d90-4d48-8f35-88cd85f5a5cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataclasses import dataclass\n",
    "import os\n",
    "from tempfile import TemporaryDirectory\n",
    "from typing import List, Any, Dict\n",
    "\n",
    "from shapely.geometry import box, mapping\n",
    "import rasterio\n",
    "from rasterio.warp import reproject, Resampling\n",
    "import pyproj\n",
    "from osgeo import gdal\n",
    "\n",
    "from pystac_client import Client\n",
    "import planetary_computer as pc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fb13cd6-fc7a-45b4-b6b3-3320b70cd4b4",
   "metadata": {},
   "source": [
    "#### Extract training chips\n",
    "\n",
    "Download the `flood-train-images.tgz` file from [competition Data Download page](https://www.drivendata.org/competitions/81/detect-flood-water/data/) and upload it to the Hub in the same directory as this notebook.\n",
    "\n",
    "Then run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c6dfda2-5315-4c17-bfdb-38032fbd48bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xvf flood-train-images.tgz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a5f4ac2-c5a0-40df-bca5-eb5658ba3756",
   "metadata": {},
   "source": [
    "to uncompress this. Afterwards you should see an `train_features` directory containing all of the training chips ending in `.tif`. \n",
    "\n",
    "Use this directory to define the location of the chips, or if you have already uncompressed the chips elsewhere set the location here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dd7579d-ff6d-41dc-ad13-7e1291325c51",
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_DATA_DIR = \"train_features\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ef61164-4847-47f9-9a86-608805a5f60d",
   "metadata": {},
   "source": [
    "#### Gather chip paths\n",
    "\n",
    "These chip paths will be used later in the notebook to process the chips. These paths should be to only one GeoTIFF per chip; for example, if both `VV.tif` and `VH.tif` are available for a chip, use only one of these paths. The GeoTIFFs at these paths will be read to get the bounds, CRS and resolution that will be used to fetch auxiliary input data. These can be relative paths. The auxiliary input data will be saved in the same directory as the GeoTIFF files at these paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c599f97-1b2d-4f31-82b4-3744f3520fb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "chip_paths = []\n",
    "for file_name in os.listdir(TRAINING_DATA_DIR):\n",
    "    if file_name.endswith(\"_vv.tif\"):\n",
    "        chip_paths.append(os.path.join(TRAINING_DATA_DIR, file_name))\n",
    "print(f\"{len(chip_paths)} chips found.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b35bbcf-b8ae-4165-b552-f3c1e7b3209f",
   "metadata": {},
   "source": [
    "#### Create the STAC API client\n",
    "\n",
    "This will be used in the methods below to query the PC STAC API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cc03136-f3d1-4c15-8d7e-6a3f5acbe820",
   "metadata": {},
   "outputs": [],
   "source": [
    "STAC_API = \"https://planetarycomputer.microsoft.com/api/stac/v1\"\n",
    "catalog = Client.open(STAC_API)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49b96853-98ec-4463-a350-59c6bb593856",
   "metadata": {},
   "source": [
    "#### Define functions and classes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "995778c8-eb05-4cd4-a71f-5bf11f6738e1",
   "metadata": {},
   "source": [
    "Define a `ChipInfo` dataclass to encapsulate the required data for the target chip. This includes geospatial information that will be used to coregister the incoming jrc-gsw and nasadem data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a30baf9f-5a39-47ab-8547-7cb6c4ec5cc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class ChipInfo:\n",
    "    \"\"\"\n",
    "    Holds information about a training chip, including geospatial info for coregistration\n",
    "    \"\"\"\n",
    "\n",
    "    path: str\n",
    "    prefix: str\n",
    "    crs: Any\n",
    "    shape: List[int]\n",
    "    transform: List[float]\n",
    "    bounds: rasterio.coords.BoundingBox\n",
    "    footprint: Dict[str, Any]\n",
    "\n",
    "\n",
    "def get_footprint(bounds, crs):\n",
    "    \"\"\"Gets a GeoJSON footprint (in epsg:4326) from rasterio bounds and CRS\"\"\"\n",
    "    transformer = pyproj.Transformer.from_crs(crs, \"epsg:4326\", always_xy=True)\n",
    "    minx, miny = transformer.transform(bounds.left, bounds.bottom)\n",
    "    maxx, maxy = transformer.transform(bounds.right, bounds.top)\n",
    "    return mapping(box(minx, miny, maxx, maxy))\n",
    "\n",
    "\n",
    "def get_chip_info(chip_path):\n",
    "    \"\"\"Gets chip info from a GeoTIFF file\"\"\"\n",
    "    with rasterio.open(chip_path) as ds:\n",
    "        chip_crs = ds.crs\n",
    "        chip_shape = ds.shape\n",
    "        chip_transform = ds.transform\n",
    "        chip_bounds = ds.bounds\n",
    "\n",
    "    # Use the first part of the chip filename as a prefix\n",
    "    prefix = os.path.basename(chip_path).split(\"_\")[0]\n",
    "\n",
    "    return ChipInfo(\n",
    "        path=chip_path,\n",
    "        prefix=prefix,\n",
    "        crs=chip_crs,\n",
    "        shape=chip_shape,\n",
    "        transform=chip_transform,\n",
    "        bounds=chip_bounds,\n",
    "        footprint=get_footprint(chip_bounds, chip_crs),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8497b2da-6935-48b6-934b-63ab852fd0ef",
   "metadata": {},
   "source": [
    "This method reprojects coregisters raster data to the bounds, CRS and resolution described by the ChipInfo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7da8244f-e06e-401c-b493-469b1dcf9b78",
   "metadata": {},
   "outputs": [],
   "source": [
    "def reproject_to_chip(\n",
    "    chip_info, input_path, output_path, resampling=Resampling.nearest\n",
    "):\n",
    "    \"\"\"\n",
    "    Reproject a raster at input_path to chip_info, saving to output_path.\n",
    "\n",
    "    Use Resampling.nearest for classification rasters. Otherwise use something\n",
    "    like Resampling.bilinear for continuous data.\n",
    "    \"\"\"\n",
    "    with rasterio.open(input_path) as src:\n",
    "        kwargs = src.meta.copy()\n",
    "        kwargs.update(\n",
    "            {\n",
    "                \"crs\": chip_info.crs,\n",
    "                \"transform\": chip_info.transform,\n",
    "                \"width\": chip_info.shape[1],\n",
    "                \"height\": chip_info.shape[0],\n",
    "                \"driver\": \"GTiff\",\n",
    "            }\n",
    "        )\n",
    "\n",
    "        with rasterio.open(output_path, \"w\", **kwargs) as dst:\n",
    "            for i in range(1, src.count + 1):\n",
    "                reproject(\n",
    "                    source=rasterio.band(src, i),\n",
    "                    destination=rasterio.band(dst, i),\n",
    "                    src_transform=src.transform,\n",
    "                    src_crs=src.crs,\n",
    "                    dst_transform=chip_info.transform,\n",
    "                    dst_crs=chip_info.crs,\n",
    "                    resampling=Resampling.nearest,\n",
    "                )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1496dd96-5bf2-49b3-8140-3ff3817b07ac",
   "metadata": {},
   "source": [
    "This method will take in a set of items and a asset key and write a [VRT](https://gdal.org/drivers/raster/vrt.html) using signed HREFs. This is useful when there's multiple results from the query, so we can treat the resulting rasters as a single set of raster data. It uses the `planetary_computer.sign` method to sign the HREFs with a SAS token generated by the PC [Data Auth API](https://planetarycomputer.microsoft.com/docs/concepts/sas/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03111f51-3112-4936-bc3c-89c296e72a45",
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_vrt(items, asset_key, dest_path):\n",
    "    \"\"\"Write a VRT with hrefs extracted from a list of items for a specific asset.\"\"\"\n",
    "    hrefs = [pc.sign(item.assets[asset_key].href) for item in items]\n",
    "    vsi_hrefs = [f\"/vsicurl/{href}\" for href in hrefs]\n",
    "    gdal.BuildVRT(dest_path, vsi_hrefs).FlushCache()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26813323-f96d-4a3e-b002-f079fbbb2711",
   "metadata": {},
   "source": [
    "This method ties it all together - for a given `ChipInfo`, Collection, and Asset, write an auxiliary input chip with the given file name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3befe6b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_chip_aux_file(\n",
    "    chip_info, collection_id, asset_key, file_name, resampling=Resampling.nearest\n",
    "):\n",
    "    \"\"\"\n",
    "    Write an auxiliary chip file.\n",
    "\n",
    "    The auxiliary chip file includes chip_info for the Collection and Asset, and is\n",
    "    saved in the same directory as the original chip with the given file_name.\n",
    "    \"\"\"\n",
    "    output_path = os.path.join(\n",
    "        os.path.dirname(chip_info.path), f\"{chip_info.prefix}_{file_name}\"\n",
    "    )\n",
    "    search = catalog.search(collections=[collection_id], intersects=chip_info.footprint)\n",
    "    items = list(search.get_items())\n",
    "    with TemporaryDirectory() as tmp_dir:\n",
    "        vrt_path = os.path.join(tmp_dir, \"source.vrt\")\n",
    "        write_vrt(items, asset_key, vrt_path)\n",
    "        reproject_to_chip(chip_info, vrt_path, output_path, resampling=resampling)\n",
    "    return output_path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2d5fcd7-0343-4da7-a885-1238b09176a4",
   "metadata": {},
   "source": [
    "#### Configurate the auxiliary input files that we will generate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aac63bad-d86a-4ac1-9c35-9a813eb4eff1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a set of parameters to pass into create_chip_aux_file\n",
    "aux_file_params = [\n",
    "    (\"nasadem\", \"elevation\", \"nasadem.tif\", Resampling.bilinear),\n",
    "    (\"jrc-gsw\", \"extent\", \"jrc-gsw-extent.tif\", Resampling.nearest),\n",
    "    (\"jrc-gsw\", \"occurrence\", \"jrc-gsw-occurrence.tif\", Resampling.nearest),\n",
    "    (\"jrc-gsw\", \"recurrence\", \"jrc-gsw-recurrence.tif\", Resampling.nearest),\n",
    "    (\"jrc-gsw\", \"seasonality\", \"jrc-gsw-seasonality.tif\", Resampling.nearest),\n",
    "    (\"jrc-gsw\", \"transitions\", \"jrc-gsw-transitions.tif\", Resampling.nearest),\n",
    "    (\"jrc-gsw\", \"change\", \"jrc-gsw-change.tif\", Resampling.nearest),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f269e1c0-8fa1-4de1-8c4e-238a54de75a8",
   "metadata": {},
   "source": [
    "#### Generate auxiliary input chips for NASADEM and JRC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "407b3859-268d-4dd4-8118-65f7a611a266",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterate over the chips and generate all aux input files.\n",
    "count = len(chip_paths)\n",
    "for i, chip_path in enumerate(chip_paths):\n",
    "    print(f\"({i+1} of {count}) {chip_path}\")\n",
    "    chip_info = get_chip_info(chip_path)\n",
    "    for collection_id, asset_key, file_name, resampling_method in aux_file_params:\n",
    "        print(f\"  ... Creating chip data for {collection_id} {asset_key}\")\n",
    "        create_chip_aux_file(\n",
    "            chip_info, collection_id, asset_key, file_name, resampling=resampling_method\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5039f710-1421-41cd-b05f-52f2b47a8c53",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}