{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Share delair.ai map tiles with the Python SDK\n",
    "\n",
    "In this notebook, you will be guided to :\n",
    "- Create a **project** and upload an **orthomosaic** and **DSM** rasters\n",
    "- Generate a tile layer URLs\n",
    "- Visualize an orthomosaic and a DSM in the notebook\n",
    "- Publish it on **arcgisonline**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You may execute this notebook one cell after the other with `Shift + Enter`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more options, explore `Cell` in the top menu bar, or read a [tutorial](https://www.dataquest.io/blog/jupyter-notebook-tutorial/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- delair.ai Python SDK\n",
    "- folium\n",
    "- arcgis\n",
    "- shapely\n",
    "- pyproj\n",
    "\n",
    "⚠️ **Restart the Jupyter kernel after first install**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install python-delairstack folium arcgis shapely pyproj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from delairstack import DelairStackSDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "platform_url = 'https://www.delair.ai'\n",
    "login = input('Enter your email ')\n",
    "password = getpass.getpass('Enter your password ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdk = DelairStackSDK(url=platform_url, user=login, password=password)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download sample files (images, mesh, raster...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An archive `Saint-Papoul.zip` containing sample files will be downloaded (if not found in the current directory)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "import zipfile\n",
    "\n",
    "try: working_dir\n",
    "except NameError: working_dir = os.getcwd()\n",
    "%cd {working_dir}\n",
    "\n",
    "if not os.path.exists('Saint-Papoul'):\n",
    "    print('\"Saint-Papoul\" folder not found')\n",
    "    if not os.path.exists('Saint-Papoul.zip'):\n",
    "        print('\"Saint-Papoul.zip\" not found')\n",
    "        print('Downloading it...', end=' ')\n",
    "        url = 'https://delair-transfer.s3-eu-west-1.amazonaws.com/sdks/sample-data/Saint-Papoul.zip'\n",
    "        filename, _ = urllib.request.urlretrieve(url, 'Saint-Papoul.zip')\n",
    "        print('OK')\n",
    "\n",
    "    print('Extracting \"Saint-Papoul.zip\"...', end=' ')\n",
    "    with zipfile.ZipFile(filename, 'r') as zip_ref:\n",
    "        zip_ref.extractall('./Saint-Papoul')\n",
    "    print('OK')\n",
    "else:\n",
    "    print('\"Saint-Papoul\" folder found. No need to download it again.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_path = './Saint-Papoul'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd {sample_path}\n",
    "!ls ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_project = sdk.projects.create(\n",
    "    name='SDK Map Tiles Tutorial',\n",
    "    geometry={\"coordinates\": [[\n",
    "                [2.0362935018049213, 43.33793077500704],\n",
    "                [2.052168442639523, 43.33793077500704],\n",
    "                [2.052168442639523, 43.34757088135606],\n",
    "                [2.0362935018049213, 43.34757088135606],\n",
    "                [2.0362935018049213, 43.33793077500704]]],\n",
    "              \"type\": \"Polygon\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('We just created the project {!r} with id {!r}'.format(\n",
    "    my_project.name, my_project.id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and upload the orthomosaic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ortho_dataset = sdk.datasets.create_raster_dataset(\n",
    "  name='Orthomosaic',\n",
    "  project=my_project.id,\n",
    "  dataset_format='geotiff',\n",
    "  categories=['orthomosaic'])\n",
    "\n",
    "sdk.datasets.upload_file(\n",
    "  dataset=ortho_dataset.id,\n",
    "  component='raster',\n",
    "  file_path='Orthomosaic.tif')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DSM (Digital Surface Model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsm_dataset = sdk.datasets.create_raster_dataset(\n",
    "  name='DSM',\n",
    "  project=my_project.id,\n",
    "  dataset_format='geotiff',\n",
    "  categories=['dsm'])\n",
    "\n",
    "sdk.datasets.upload_file(\n",
    "  dataset=dsm_dataset.id,\n",
    "  component='raster',\n",
    "  file_path='DSM.tif')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### After the raster uploads, they need to be tiled. It can take up to 5 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import sleep\n",
    "print('Please wait till the rasters are tiled (it can takes up to 5 minutes)...')\n",
    "while True:\n",
    "    ds_list = sdk.datasets.describe([ortho_dataset.id, dsm_dataset.id])\n",
    "    ingestion_statuses = [ds.ingestion.get('status') for ds in ds_list]\n",
    "    if ingestion_statuses == ['completed', 'completed']:\n",
    "        break\n",
    "    else:\n",
    "        sleep(10)\n",
    "\n",
    "print('OK 👍')\n",
    "print('Orthomosaic and DSM have been ingested properly')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tile layer URLs Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**delair.ai** provides a tile service for all raster datasets.\n",
    "\n",
    "It follows the TMS standard (for example: https://delair.ai/tileserver/tiles/DATASET_ID/{z}/{x}/{y}.png).\n",
    "\n",
    "This service is protected using a delair.ai token.\n",
    "\n",
    "The Python SDK provides the `sdk.datasets.share_tiles` function:\n",
    "- to quickly generate a token authorized for a specific dataset\n",
    "- to build a URL for the tile service of this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ortho_tile_url = sdk.datasets.share_tiles(dataset=ortho_dataset.id)\n",
    "ortho_tile_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsm_tile_url = sdk.datasets.share_tiles(dataset=dsm_dataset.id)\n",
    "dsm_tile_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Determine the center of the raster and its bounding box"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request the updated dataset properties (such as its geometry)\n",
    "ortho_dataset = sdk.datasets.describe(ortho_dataset.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapely.geometry import shape\n",
    "\n",
    "def get_center_coords(dataset):\n",
    "    shapely_geometry = shape(dataset.geometry)\n",
    "    return list(shapely_geometry.centroid.coords)[0]\n",
    "\n",
    "center_longitude, center_latitude = get_center_coords(ortho_dataset)\n",
    "map_center = (center_latitude, center_longitude)\n",
    "print('Coordinates of the center: lat={:.4f}, long={:.4f}'.format(center_latitude, center_longitude))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_bbox(dataset):\n",
    "    shapely_geometry = shape(dataset.geometry)\n",
    "    return shapely_geometry.bounds  # (minx, miny, maxx, maxy)\n",
    "\n",
    "bbox = get_bbox(ortho_dataset)\n",
    "print('Bounding box: {}'.format(bbox))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization with Folium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Orthomosaic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import folium\n",
    "\n",
    "m = folium.Map(location=map_center, zoom_start=18)\n",
    "\n",
    "folium.raster_layers.TileLayer(\n",
    "    tiles=ortho_tile_url,\n",
    "    attr='delair.ai',\n",
    "    max_zoom=20,\n",
    "    overlay=False,\n",
    "    control=True,\n",
    "    bounds=[[bbox[1], bbox[0]], [bbox[3], bbox[2]]]\n",
    ").add_to(m)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsm_map = folium.Map(location=map_center, zoom_start=18)\n",
    "\n",
    "folium.raster_layers.TileLayer(\n",
    "    tiles=dsm_tile_url,\n",
    "    attr='delair.ai',\n",
    "    max_zoom=20,\n",
    "    overlay=False,\n",
    "    control=True,\n",
    "    bounds=[[bbox[1], bbox[0]], [bbox[3], bbox[2]]]\n",
    ").add_to(dsm_map)\n",
    "dsm_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Functions to generate a custom tile URL and render it\n",
    "\n",
    "from urllib.parse import urlencode, urlsplit, parse_qs, urlunsplit\n",
    "from ipywidgets import interact, fixed, FloatSlider, IntSlider\n",
    "\n",
    "def add_query_params_to_url(url, param_dict):\n",
    "    scheme, netloc, path, query_string, fragment = urlsplit(url)\n",
    "    params = parse_qs(query_string)\n",
    "    params.update(param_dict)\n",
    "    new_query_string = urlencode(params, doseq=True)\n",
    "    return urlunsplit((scheme, netloc, path, new_query_string, fragment))\n",
    "\n",
    "def generate_custom_tile_url(original_tile_url, **kwargs):\n",
    "    params = kwargs\n",
    "    tile_url = add_query_params_to_url(original_tile_url, params)\n",
    "    return tile_url\n",
    "\n",
    "def render_folium_map(original_tile_url, **kwargs):\n",
    "    tile_url = generate_custom_tile_url(original_tile_url, **kwargs)\n",
    "    m = folium.Map(location=map_center, zoom_start=18)\n",
    "    folium.raster_layers.TileLayer(\n",
    "        tiles=tile_url,\n",
    "        attr='delair.ai',\n",
    "        max_zoom=20,\n",
    "        overlay=False,\n",
    "        control=True,\n",
    "        bounds=[[bbox[1], bbox[0]], [bbox[3], bbox[2]]]\n",
    "    ).add_to(m)\n",
    "    return m\n",
    "\n",
    "def interactive_dsm_map():\n",
    "    original_tile_url = dsm_tile_url\n",
    "    ds = sdk.datasets.describe(dsm_dataset.id)\n",
    "    min_value = ds.bands[0].get('stats').get('min')\n",
    "    max_value = ds.bands[0].get('stats').get('max')\n",
    "\n",
    "    interact(\n",
    "        render_folium_map,\n",
    "        original_tile_url=fixed(original_tile_url),\n",
    "        band_id=fixed(1),\n",
    "        colormap=['spectral', 'RdYlGn', 'GnYlRd', 'greyscale'],\n",
    "        outofrange_type=['clip', 'transparent'],\n",
    "        slope=IntSlider(value=100, max=100, min=0, step=5, continuous_update=False),\n",
    "        min=FloatSlider(min_value, min=min_value, max=max_value, step=1e-1, continuous_update=False),\n",
    "        max=FloatSlider(max_value, min=min_value, max=max_value, step=1e-1, continuous_update=False)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For exemple, you can change :\n",
    "- `slope` = 50\n",
    "- `colormap` = `GnYIRd`\n",
    "- `min_value` around 188\n",
    "- `outofrange_type` = `transparent`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_dsm_map()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## With Argis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_arcgis_url = ''\n",
    "arcgis_url = input('Enter your arcgis url (for example \"https://mycompany.maps.arcgis.com\")')\n",
    "arcgis_username = input('Enter your arcgis username ')\n",
    "arcgis_password = getpass.getpass('Enter your arcgis password ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from arcgis.gis import GIS\n",
    "\n",
    "print('Connecting to {!r} with the supplied credentials...'.format(arcgis_url))\n",
    "gis = GIS(arcgis_url, arcgis_username, arcgis_password)\n",
    "print('OK 👍')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### New map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may refer to [the ArcGIS documentation](https://developers.arcgis.com/python/) and [the ArcGIS python package API](https://developers.arcgis.com/python/api-reference/) to get a deeper understanding of the following concepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyproj import Proj, transform\n",
    "\n",
    "def convert_coords_to_mercator(long, lat):\n",
    "    origin_projection = Proj('epsg:4326')\n",
    "    mercator_projection = Proj('epsg:3857')\n",
    "    return transform(origin_projection, mercator_projection, long, lat)\n",
    "\n",
    "def get_mercator_full_extent(dataset):\n",
    "    xmin_4326, ymin_4326, xmax_4326, ymax_4326 = get_bbox(dataset)\n",
    "    xmin, ymin = convert_coords_to_mercator(ymin_4326, xmin_4326)\n",
    "    xmax, ymax = convert_coords_to_mercator(ymax_4326, xmax_4326)\n",
    "    \n",
    "    full_extent = {\n",
    "        'xmin': xmin,\n",
    "        'ymin': ymin,\n",
    "        'xmax': xmax,\n",
    "        'ymax': ymax,\n",
    "        'spatialReference': {'wkid': 102100}\n",
    "    }\n",
    "    return full_extent\n",
    "\n",
    "def create_arcgis_layer(name, url):\n",
    "    proper_url = url.replace('{z}', '{level}').replace('{x}', '{col}').replace('{y}', '{row}')\n",
    "    return {\n",
    "        \"opacity\": 1,\n",
    "        \"visibility\": True,\n",
    "        \"title\": name,\n",
    "        \"type\": \"WebTiledLayer\",\n",
    "        \"layerType\": \"WebTiledLayer\",\n",
    "        \"templateUrl\": proper_url,\n",
    "        \"copyright\": \"Delair\",\n",
    "        \"fullExtent\": get_mercator_full_extent(ortho_dataset)\n",
    "    }\n",
    "\n",
    "def create_arcgis_basemap(title, layers):\n",
    "    return {\n",
    "        \"baseMap\": {\n",
    "            \"baseMapLayers\": [\n",
    "                {\n",
    "                    \"id\": \"defaultBasemap\",\n",
    "                    \"layerType\": \"ArcGISTiledMapServiceLayer\",\n",
    "                    \"url\": \"https://services.arcgisonline.com/ArcGIS/rest/services/World_Topo_Map/MapServer\",\n",
    "                    \"visibility\": True,\n",
    "                    \"opacity\": 1,\n",
    "                    \"title\": \"Topographic\"\n",
    "                }\n",
    "            ],\n",
    "            \"title\": title\n",
    "        },\n",
    "        \"spatialReference\": {\n",
    "            \"wkid\": 102100,\n",
    "            \"latestWkid\": 3857\n",
    "        },\n",
    "        \"authoringApp\": \"WebMapViewer\",\n",
    "        \"authoringAppVersion\": \"7.1\",\n",
    "        \"version\": \"2.14\",\n",
    "        \"operationalLayers\": layers,\n",
    "    }\n",
    "\n",
    "def publish_map_on_arcgis(title, layers):\n",
    "    arcgis_layers = [create_arcgis_layer(name=layer.get('name'), url=layer.get('url')) for layer in layers]\n",
    "    base_map = create_arcgis_basemap(title, arcgis_layers)\n",
    "    item_prop = {\n",
    "        \"type\": \"Web Map\",\n",
    "        \"title\": title,\n",
    "        \"tags\": [\"WebTiledLayer\", \"delair.ai\"],\n",
    "        \"snippet\": \"\",\n",
    "        \"text\": base_map,\n",
    "        \"extent\": [[bbox[0], bbox[1]],[bbox[2], bbox[3]]]\n",
    "    }\n",
    "    \n",
    "    generated_map = gis.content.add(item_properties=item_prop)\n",
    "    return generated_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's generate a map on Arcgis with both Orthomosaic and DSM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_dsm_tile_url = generate_custom_tile_url(\n",
    "    dsm_tile_url,\n",
    "    min=188,\n",
    "    max=201,\n",
    "    outofrange_type='transparent'\n",
    ")\n",
    "\n",
    "layers = [\n",
    "    {'name': 'Orthomosaic', 'url': ortho_tile_url},\n",
    "    {'name': 'DSM', 'url': custom_dsm_tile_url}\n",
    "]\n",
    "\n",
    "generated_map = publish_map_on_arcgis('Serving delair.ai data on Arcgis', layers)\n",
    "generated_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = gis.map(generated_map)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}