{ "cells": [ { "cell_type": "markdown", "id": "64bd13a2-e1bb-43cd-846c-410346c55d3e", "metadata": {}, "source": [ "## Microsoft Building Footprints Example\n", "\n", "Bing Maps is releasing open building footprints around the world. The building footprints are detected using Bing Maps imagery between 2014 and 2021 including Maxar and Airbus imagery. The data is freely available for download and use under ODbL." ] }, { "cell_type": "code", "execution_count": 1, "id": "05217579-1f22-4c5c-a649-6fcbd2cc2773", "metadata": { "tags": [] }, "outputs": [], "source": [ "import planetary_computer\n", "import pystac_client\n", "import deltalake\n", "import shapely.geometry\n", "import contextily\n", "import mercantile" ] }, { "cell_type": "markdown", "id": "6ceab670-3781-4ca6-9c10-0177f9ac450d", "metadata": {}, "source": [ "### Data access\n", "\n", "The datasets hosted by the Planetary Computer are available from [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/). We'll use [pystac-client](https://pystac-client.readthedocs.io/) to search the Planetary Computer's [STAC API](https://planetarycomputer.microsoft.com/api/stac/v1/docs) to get a link to the assets. We'll specify a `modifier` so that we can access the data stored in the Planetary Computer's private Blob Storage Containers. See [Reading from the STAC API](https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/) and [Using tokens for data access](https://planetarycomputer.microsoft.com/docs/concepts/sas/) for more." ] }, { "cell_type": "code", "execution_count": 2, "id": "76dfc9a9-c787-41c3-aaa8-95787bed362c", "metadata": { "tags": [] }, "outputs": [], "source": [ "catalog = pystac_client.Client.open(\n", " \"https://planetarycomputer.microsoft.com/api/stac/v1\",\n", " modifier=planetary_computer.sign_inplace,\n", ")\n", "collection = catalog.get_collection(\"ms-buildings\")" ] }, { "cell_type": "markdown", "id": "46c937a5-8ceb-46e8-9b27-b154184236fb", "metadata": {}, "source": [ "### Using Delta Table Files\n", "\n", "The assets are a set of [geoparquet](https://geoparquet.org/) files grouped by a processing date. Newer files (since April 25th, 2023) are stored in [Delta Format](https://docs.delta.io/latest/delta-intro.html). This is a layer on top of parquet files offering scalable metadata handling, which is useful for this dataset." ] }, { "cell_type": "code", "execution_count": 3, "id": "aace14f8-5a33-42e1-9651-44e0f1826546", "metadata": { "tags": [] }, "outputs": [], "source": [ "asset = collection.assets[\"delta\"]" ] }, { "cell_type": "markdown", "id": "c78f7f60-1761-4d2e-8e7f-938aad628b87", "metadata": { "tags": [] }, "source": [ "This Delta Table is partitioned by `RegionName` and `quadkey`. Each `(RegionName, quadkey)` pair will contain one or more parquet files (depending on how dense that particular quadkey) is. The quadkeys are at level 9 of the [Bing Maps Tile System](https://learn.microsoft.com/en-us/bingmaps/articles/bing-maps-tile-system)." ] }, { "cell_type": "code", "execution_count": 4, "id": "c7d30272-1914-4c57-b0d6-c93a9db59c60", "metadata": { "tags": [] }, "outputs": [], "source": [ "storage_options = {\n", " \"account_name\": asset.extra_fields[\"table:storage_options\"][\"account_name\"],\n", " \"sas_token\": asset.extra_fields[\"table:storage_options\"][\"credential\"],\n", "}\n", "table = deltalake.DeltaTable(asset.href, storage_options=storage_options)" ] }, { "cell_type": "markdown", "id": "8a247424-0e04-460f-8b77-d333c715b217", "metadata": {}, "source": [ "You can load all the files for a given `RegionName` with a query like this:" ] }, { "cell_type": "code", "execution_count": 5, "id": "51ad77aa-4e95-427d-aaf1-f43259527256", "metadata": {}, "outputs": [], "source": [ "import geopandas\n", "import pandas as pd\n", "\n", "file_uris = table.file_uris([(\"RegionName\", \"=\", \"VaticanCity\")])\n", "df = pd.concat(\n", " [\n", " geopandas.read_parquet(file_uri, storage_options=storage_options)\n", " for file_uri in file_uris\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": 6, "id": "3c235ec0-68f4-4621-875f-9c81a4c0ddf5", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ax = df.plot(figsize=(12, 12), color=\"yellow\")\n", "contextily.add_basemap(\n", " ax,\n", " source=contextily.providers.Esri.WorldGrayCanvas,\n", " crs=df.crs.to_string(),\n", " zoom=16,\n", ")\n", "ax.set_axis_off()" ] }, { "cell_type": "markdown", "id": "421eda4c-1112-4553-9df2-136a727820f7", "metadata": {}, "source": [ "This partitioning, and the Delta Table format, let you quickly query the subset of files that match your area of interest.\n", "\n", "For example, we can visualize the footprints for a small region outside the town of Ueckermünde in Northeast Germany defined by this bounding box." ] }, { "cell_type": "code", "execution_count": 7, "id": "b5908697-1f05-4c17-8a1b-298fcd54455e", "metadata": { "tags": [] }, "outputs": [], "source": [ "area_of_interest = shapely.geometry.box(14.11, 53.73, 14.13, 53.75)" ] }, { "cell_type": "markdown", "id": "72f2dfa3-7476-4fb5-840b-3c4f92161b52", "metadata": {}, "source": [ "You will need to translate your area of interest to a set of intersecting quadkeys, which can be done with [mercantile](https://mercantile.readthedocs.io/en/stable/)." ] }, { "cell_type": "code", "execution_count": 8, "id": "0e3e8ca5-4327-47dc-919b-c8ff99faa43e", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "[120210300, 120210302]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "quadkeys = [\n", " int(mercantile.quadkey(tile))\n", " for tile in mercantile.tiles(*area_of_interest.bounds, zooms=9)\n", "]\n", "quadkeys" ] }, { "cell_type": "markdown", "id": "a95e23ab-b8bb-4609-b316-247ec5fbcf99", "metadata": {}, "source": [ "Now we can provide those quadkeys as a partition filter to get the set of matching URIs." ] }, { "cell_type": "code", "execution_count": 9, "id": "ce9870f3-4baf-4aa2-9921-a076e4b9f797", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "['az://footprints/delta/2023-04-25/ml-buildings.parquet/RegionName=Germany/quadkey=120210300/part-00168-a20a8489-66de-4ede-be14-48423c230226.c000.snappy.parquet',\n", " 'az://footprints/delta/2023-04-25/ml-buildings.parquet/RegionName=Germany/quadkey=120210302/part-00197-b547555b-9f9e-4cf2-8016-b35087e71f2b.c000.snappy.parquet']" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "uris = table.file_uris([(\"RegionName\", \"=\", \"Germany\"), (\"quadkey\", \"in\", quadkeys)])\n", "uris" ] }, { "cell_type": "markdown", "id": "e65796c3-63fd-44b8-823f-f858703821bc", "metadata": {}, "source": [ "And quickly load them into, for example, a `geopandas.GeoDataFrame`:" ] }, { "cell_type": "code", "execution_count": 10, "id": "1d9d8bcd-2549-43ed-98a8-a19757f9b983", "metadata": {}, "outputs": [], "source": [ "df = pd.concat(\n", " [\n", " geopandas.read_parquet(file_uri, storage_options=storage_options)\n", " for file_uri in uris\n", " ]\n", ")" ] }, { "cell_type": "markdown", "id": "0b0babda-2aaa-4780-9cc4-995553397125", "metadata": {}, "source": [ "Which can be visualized:" ] }, { "cell_type": "code", "execution_count": 11, "id": "36a5eb0f-efc1-4976-99c2-7199b2d54fab", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subset = df.clip(area_of_interest)\n", "subset.explore()" ] }, { "cell_type": "markdown", "id": "f8da8ca0-7f4d-4e48-839e-2cab39584848", "metadata": {}, "source": [ "### Using the STAC API\n", "\n", "As an alternative to Delta, you can use the Planetary Computer STAC API to query files matching your area of interest." ] }, { "cell_type": "code", "execution_count": 12, "id": "a4f666ae-76d1-48a4-a12a-7740d614487a", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "2" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "search = catalog.search(\n", " collections=[\"ms-buildings\"],\n", " intersects=area_of_interest,\n", " query={\n", " \"msbuildings:region\": {\"eq\": \"Germany\"},\n", " \"msbuildings:processing-date\": {\"eq\": \"2023-04-25\"},\n", " },\n", ")\n", "\n", "ic = search.item_collection()\n", "len(ic)" ] }, { "cell_type": "markdown", "id": "9c1eccac-d030-4fb2-9a1f-69e0263701bd", "metadata": {}, "source": [ "See the summaries on the collection for a list of valid processing dates.\n", "\n", "Once you have the items, you can load them into a `GeoDataFrame`. Note that each `data` asset has one or more parquet files under it, so we need to do two levels of concatentation." ] }, { "cell_type": "code", "execution_count": 13, "id": "ee3a214f-7b90-43e5-a331-ee9d903f125b", "metadata": { "tags": [] }, "outputs": [], "source": [ "import adlfs\n", "\n", "fs = adlfs.AzureBlobFileSystem(\n", " **ic[0].assets[\"data\"].extra_fields[\"table:storage_options\"]\n", ")\n", "\n", "prefixes = [item.assets[\"data\"].href for item in ic]\n", "parts = []\n", "for item in ic:\n", " parts.extend(fs.ls(item.assets[\"data\"].href))" ] }, { "cell_type": "code", "execution_count": 14, "id": "880fb178-2684-4455-918a-de5967eebafa", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
geometrymeanHeight
0POLYGON ((14.23111 53.66713, 14.23120 53.66714...-1.0
1POLYGON ((14.13342 53.53018, 14.13337 53.53030...-1.0
2POLYGON ((14.34231 53.33267, 14.34236 53.33258...-1.0
3POLYGON ((14.09257 53.49413, 14.09274 53.49409...-1.0
4POLYGON ((14.07604 53.68923, 14.07603 53.68927...-1.0
\n", "
" ], "text/plain": [ " geometry meanHeight\n", "0 POLYGON ((14.23111 53.66713, 14.23120 53.66714... -1.0\n", "1 POLYGON ((14.13342 53.53018, 14.13337 53.53030... -1.0\n", "2 POLYGON ((14.34231 53.33267, 14.34236 53.33258... -1.0\n", "3 POLYGON ((14.09257 53.49413, 14.09274 53.49409... -1.0\n", "4 POLYGON ((14.07604 53.68923, 14.07603 53.68927... -1.0" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.concat(\n", " [\n", " geopandas.read_parquet(f\"az://{part}\", storage_options=storage_options)\n", " for part in parts\n", " ]\n", ")\n", "df.head()" ] }, { "cell_type": "markdown", "id": "63a5ad0c-6865-4161-ad72-7d0e51670412", "metadata": {}, "source": [ "### Next steps\n", "\n", "This example briefly demonstrated searching the STAC catalog and loading the data with geopandas. To learn more, visit the following resources:\n", "\n", "* Learn more about [geopandas](https://geopandas.readthedocs.io).\n", "* Learn more about [Dask](https://dask.org/)\n", "* Learn more about [geoparquet](https://github.com/opengeospatial/geoparquet)\n", "* Learn more about the [Global MLBuildings Footprints](https://github.com/microsoft/GlobalMLBuildingFootprints/) project." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }