{ "cells": [ { "cell_type": "markdown", "id": "c798da7c", "metadata": {}, "source": [ "## Writing outputs to Azure Blob Storage\n", "\n", "Azure provide [many options](https://azure.microsoft.com/en-us/product-categories/storage/) for storing data. For cloud-native workflows favored by the Planetary Computer, [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) is a good choice. In this example we'll compute NDVI for a single image and write the results to a container in Azure Blob Storage as a single-band COG. Note that your storage and compute should be located in the same [Azure region](https://azure.microsoft.com/en-us/global-infrastructure/geographies/). The Planetary Computer Hub and data reside in the West Europe region.\n", "\n", "### Compute NDVI\n", "\n", "We need something to write. To simulate a workload, let's load in a single [NAIP image](https://planetarycomputer.microsoft.com/dataset/naip#Example-Notebook) from the Planetary Computer's data catalog using [rioxarray](https://corteva.github.io/rioxarray/html/rioxarray.html) and compute the NDVI using [xrspatial](https://xarray-spatial.org/reference/_autosummary/xrspatial.multispectral.ndvi.html)." ] }, { "cell_type": "code", "execution_count": 1, "id": "52a7c344", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "import pystac_client\n", "import planetary_computer\n", "import rioxarray\n", "import xrspatial\n", "\n", "\n", "catalog = pystac_client.Client.open(\n", " \"https://planetarycomputer.microsoft.com/api/stac/v1/\",\n", " modifier=planetary_computer.sign_inplace,\n", ")\n", "item = catalog.get_collection(\"naip\").get_item(\n", " \"fl_m_2608005_nw_17_060_20191215_20200113\"\n", ")\n", "\n", "ds = (\n", " rioxarray.open_rasterio(item.assets[\"image\"].href)\n", " .assign_coords(band=[\"red\", \"green\", \"blue\", \"nir\"])\n", " .load()\n", " .astype(float)\n", ")\n", "\n", "ndvi = xrspatial.ndvi(ds.sel(band=\"red\"), ds.sel(band=\"nir\"))\n", "\n", "fig, ax = plt.subplots(figsize=(15, 10))\n", "ndvi.isel(y=slice(1000), x=slice(1000)).plot.imshow(ax=ax)\n", "ax.set_axis_off()\n", "\n", "ax.set(title=f\"NDVI for {item.id=}\");" ] }, { "cell_type": "markdown", "id": "3f673981", "metadata": {}, "source": [ "### Write to Azure Blob Storage\n", "\n", "Now we'll use the `azure.storage.blob` Python library to write this data to blob storage.\n", "\n", "In this case we'll write to the `pc-scratch` *container* in the `pcstoraccount` *storage account*. This is a private container so we need to somehow authorize the access to the storage account. There are [several ways](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth?toc=/azure/storage/blobs/toc.json) to achieve this, but we'll use a SAS token that was generated in the Azure Portal. Make sure to include the necessary permissions for what you're trying to accomplish (including write new files, perhaps to delete existing ones).\n", "\n", "This SAS token should be considered sensitive and kept secret. We recommend against storing secrets in plain-text on the Planetary Computer Hub. For this interactive example, we'll manually paste it into the Python session using `getpass.getpass`. If you [deploy your own Hub](https://planetarycomputer.microsoft.com/docs/concepts/hub-deployment/), you'd have additional options available, including granting access to an Azure Key Vault that could store the secrets." ] }, { "cell_type": "code", "execution_count": 2, "id": "c8f7ae99", "metadata": {}, "outputs": [ { "name": "stdin", "output_type": "stream", "text": [ " ········\n" ] } ], "source": [ "import getpass\n", "import azure.storage.blob\n", "\n", "sas_token = getpass.getpass() # prompts for the sas_token\n", "container_client = azure.storage.blob.ContainerClient(\n", " \"https://pcstoraccount.blob.core.windows.net\",\n", " container_name=\"pc-scratch\",\n", " credential=sas_token,\n", ")" ] }, { "cell_type": "markdown", "id": "ee70505a", "metadata": {}, "source": [ "Now we're ready to write the `ndvi` data to a single-band COG and upload that to Blob Storage. We write data to an in-memory `io.BytesIO` buffer, which can be uploaded to Azure Blob Storage with `container_client.upload_blob`." ] }, { "cell_type": "code", "execution_count": 3, "id": "80dbab30", "metadata": {}, "outputs": [], "source": [ "import io\n", "\n", "with io.BytesIO() as buffer:\n", " ndvi.rio.to_raster(buffer, driver=\"COG\")\n", " buffer.seek(0)\n", " blob_client = container_client.get_blob_client(\"ndvi-wb.tif\")\n", " blob_client.upload_blob(buffer, overwrite=True)" ] }, { "cell_type": "markdown", "id": "2497ddde", "metadata": {}, "source": [ "Now the data is available in Azure Blob Storage, in our private storage container. To read the data, can use another [SAS token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview) (e.g. a read-only one) or the same one, assuming it has read permissions. The blob is available at `https://.blob.core.windows.net//ndvi-wb.tif`. It can be accessed by appending a SAS token with like `https://.blob.core.windows.net//ndvi-wb.tif?`. The `blob_client.url` already has the SAS token appended, so we can hand that straight to `rioxarray.open_rasterio`." ] }, { "cell_type": "code", "execution_count": 4, "id": "9191e477", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray (band: 1, y: 12250, x: 11050)>\n",
       "[135362500 values with dtype=float32]\n",
       "Coordinates:\n",
       "  * band         (band) int64 1\n",
       "  * x            (x) float64 5.494e+05 5.494e+05 5.494e+05 ... 5.56e+05 5.56e+05\n",
       "  * y            (y) float64 2.987e+06 2.987e+06 ... 2.979e+06 2.979e+06\n",
       "    spatial_ref  int64 0\n",
       "Attributes:\n",
       "    AREA_OR_POINT:             Area\n",
       "    TIFFTAG_IMAGEDESCRIPTION:  OrthoVista\n",
       "    TIFFTAG_RESOLUTIONUNIT:    1 (unitless)\n",
       "    TIFFTAG_SOFTWARE:          Trimble Germany GmbH\n",
       "    TIFFTAG_XRESOLUTION:       1\n",
       "    TIFFTAG_YRESOLUTION:       1\n",
       "    scale_factor:              1.0\n",
       "    add_offset:                0.0\n",
       "    long_name:                 ndvi
" ], "text/plain": [ "\n", "[135362500 values with dtype=float32]\n", "Coordinates:\n", " * band (band) int64 1\n", " * x (x) float64 5.494e+05 5.494e+05 5.494e+05 ... 5.56e+05 5.56e+05\n", " * y (y) float64 2.987e+06 2.987e+06 ... 2.979e+06 2.979e+06\n", " spatial_ref int64 0\n", "Attributes:\n", " AREA_OR_POINT: Area\n", " TIFFTAG_IMAGEDESCRIPTION: OrthoVista\n", " TIFFTAG_RESOLUTIONUNIT: 1 (unitless)\n", " TIFFTAG_SOFTWARE: Trimble Germany GmbH\n", " TIFFTAG_XRESOLUTION: 1\n", " TIFFTAG_YRESOLUTION: 1\n", " scale_factor: 1.0\n", " add_offset: 0.0\n", " long_name: ndvi" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rioxarray.open_rasterio(blob_client.url)" ] }, { "cell_type": "markdown", "id": "04f9d705", "metadata": {}, "source": [ "### Next steps\n", "\n", "For more on Azure Blob Storage, see the [Python quickstart](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python) and the [general introduction to Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction). For more on the tools used in this example, see the documentation for [rioxarray](https://corteva.github.io/rioxarray/html/rioxarray.html), [xarray](https://xarray.pydata.org/en/stable/), and [rasterio](https://rasterio.readthedocs.io/en/latest/), and [xarray-spatial](https://xarray-spatial.org/)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.10" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }