{ "cells": [ { "cell_type": "markdown", "id": "8d750ac6", "metadata": {}, "source": [ "![Geospatial Studio banner](../docs/images/banner.png)\n", "\n", "# 🌍 Geospatial Exploration and Orchestration Studio - Getting Started Notebook\n", "\n", "\n", "\n", " \n", " \n", "\n", "\n", " \n", " \n", "\n", "\n", " \n", " \n", "\n", "\n", " \n", " \n", "\n", "
License\n", " \n", "
TerraStackAI\n", " \n", " \n", " \n", "
Built With\n", " \n", " \n", " \n", "\n", "
Deployment\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "\n", "[![Studio Documentation](https://img.shields.io/badge/Studio_Documentation-526CFE?style=for-the-badge&logo=MaterialForMkDocs&logoColor=white)](https://terrastackai.github.io/geospatial-studio)\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "482879b6", "metadata": {}, "source": [ "### 1.0 Introduction\n", "\n", "Now you have a clean deployment of the studio and it is time to start using it. The steps below will enable you to onboard some initial artefacts, before trying out the functionality.\n" ] }, { "cell_type": "markdown", "id": "d178bd2d", "metadata": {}, "source": [ "### 1.1 Set up and Installation\n", "\n", "#### 1.1.0 Prerequisites\n", "\n", "Create a python environment \n", "```bash\n", "python -m venv venv\n", "source venv/bin/activate\n", "```\n", "Install geostudio sdk\n", "```bash\n", "pip install geostudio\n", "```\n", "\n", "Set the jupyter notebook kernel to point to the above created python environment." ] }, { "cell_type": "markdown", "id": "bcaa9bb6", "metadata": {}, "source": [ "#### 1.1.1 Import required packages" ] }, { "cell_type": "code", "execution_count": null, "id": "66469ebd", "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": null, "id": "d916c3a4", "metadata": {}, "outputs": [], "source": [ "# Import the required packages\n", "import json\n", "import urllib3\n", "urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n", "from geostudio import Client" ] }, { "cell_type": "markdown", "id": "523c1284", "metadata": {}, "source": [ "### 1.2 Connecting to the platform\n", "First, we set up the connection to the platform backend. To do this we need the base url for the studio UI and an API key.\n", "\n", "To get an API Key:\n", "1. Go to your deployed version of the Geospatial Studio UI (e.g. https://localhost:4180/) page and navigate to the Manage your API keys link.\n", "2. This should pop-up a window where you can generate, access and delete your api keys. NB: every user is limited to a maximum of two activate api keys at any one time.\n", "\n", "Store the API key and geostudio ui base url in a credentials file locally, for example in /User/bob/.geostudio_config_file. You can do this by:\n", "\n", "```bash\n", "echo \"GEOSTUDIO_API_KEY=\" > .geostudio_config_file\n", "echo \"BASE_STUDIO_UI_URL=\" >> .geostudio_config_file\n", "```\n", "\n", "Copy and paste the file path to this credentials file in call below. You will need to ensure that the `geostudio_config_file` is accessible and correctly configured. If you encounter any issues, please verify the file path and contents of the `.geostudio_config_file` to ensure they are accurate and up-to-date." ] }, { "cell_type": "code", "execution_count": null, "id": "d48fa4b1", "metadata": {}, "outputs": [], "source": [ "#############################################################\n", "# Initialize Geostudio client using a geostudio config file\n", "#############################################################\n", "gfm_client = Client(geostudio_config_file=\".geostudio_config_file\")" ] }, { "cell_type": "markdown", "id": "16c72538", "metadata": {}, "source": [ "> In the follow-up steps where you utilize the `gfm_client`, in case you don't get successful responses (200 - 299) you can also check the logs for the deployed pods to understand what went wrong, these include `geofm-gateway`, `geofm-gateway-celery-worker`, `postgresql-0`, `pipelines-xxx`, etc.\n", "\n", "> In addition if you need to restart any of the port-forwards you can use the following commands:\n", "\n", "```bash\n", "kubectl port-forward -n $OC_PROJECT svc/keycloak 8080:8080 >> studio-pf.log 2>&1 &\n", "kubectl port-forward -n $OC_PROJECT svc/postgresql 54320:5432 >> studio-pf.log 2>&1 &\n", "kubectl port-forward -n $OC_PROJECT svc/geofm-geoserver 3000:3000 >> studio-pf.log 2>&1 &\n", "kubectl port-forward -n $OC_PROJECT deployment/geofm-ui 4180:4180 >> studio-pf.log 2>&1 &\n", "kubectl port-forward -n $OC_PROJECT deployment/geofm-gateway 4181:4180 >> studio-pf.log 2>&1 &\n", "kubectl port-forward -n $OC_PROJECT deployment/geofm-mlflow 5000:5000 >> studio-pf.log 2>&1 &\n", "kubectl port-forward -n $OC_PROJECT svc/minio 9001:9001 >> studio-pf.log 2>&1 &\n", "kubectl port-forward -n $OC_PROJECT svc/minio 9000:9000 >> studio-pf.log 2>&1 &\n", "```" ] }, { "cell_type": "markdown", "id": "e2d1f4ec", "metadata": {}, "source": [ "### 1.3 Onboard an existing inference output (useful for loading inference precomputed examples)" ] }, { "cell_type": "markdown", "id": "3f2ee081", "metadata": {}, "source": [ "Onboard one of the inferences output. This will start a pipeline to pull the data and set it up in the platform. You should now be able to browser to the inferences page in the UI and view the example/s you have added." ] }, { "cell_type": "code", "execution_count": null, "id": "5bc52db3", "metadata": {}, "outputs": [], "source": [ "# onboard inference example for AGB Data for Karen, Nairobi,kenya\n", "with open('payloads/inferences/inference-agb-karen.json', 'r') as f:\n", " onboard_example = json.load(f)\n", "example_response=gfm_client.submit_inference(data=onboard_example)\n", "display(example_response)" ] }, { "cell_type": "markdown", "id": "e39844b6", "metadata": {}, "source": [ "### 1.4 Onboard an existing tuned models and run inference" ] }, { "cell_type": "markdown", "id": "e6f8c285", "metadata": {}, "source": [ "We will onboard a tuned model from a URL. This is initiated by an API call, which will trigger the onboarding process, starting with a download in the backend. Once the download is completed, it should appear with completed status in the UI models/tunes page. \n", "\n", "First we ensure we have a tuning task templates. These are the outline configurations to make basic tuning tasks easier for users. The tuning task tells the model what type of task it is (segmentation, regression etc), and exposes a range of optional hyperparameters which the user can set. These all have reasonable defaults, but it gives users the possibility to configure the model training how they wish. Below, we will onboard the segmentation task template to the studio, which will make it available to users in the UI." ] }, { "cell_type": "code", "execution_count": null, "id": "a5256ee0", "metadata": {}, "outputs": [], "source": [ "# segmentation template\n", "with open('payloads/templates/template-seg.json', 'r') as f:\n", " template_seg = json.load(f)\n", "template_response=gfm_client.create_task(template_seg)\n", "display(template_response)" ] }, { "cell_type": "markdown", "id": "4d943aa3", "metadata": {}, "source": [ "Then we want to onboard an existing tuned model. This will involve downloading to the cluster the tune checkpoints/weights and terratorch config yaml from a presigned url and also register metadata for this tune in to the database." ] }, { "cell_type": "code", "execution_count": null, "id": "ed142413", "metadata": {}, "outputs": [], "source": [ "# Load a prithvi-eo-flood complete tune to the studio\n", "with open('payloads/tunes/tune-prithvi-eo-flood.json','r') as f:\n", " complete_tune = json.load(f)\n", "tune_response=gfm_client.upload_completed_tunes(complete_tune)\n", "display(json.dumps(tune_response,indent=2))" ] }, { "cell_type": "markdown", "id": "77904b95", "metadata": {}, "source": [ "Below we are polling to check if the onboarding of tune is completed. This essentially takes a few minutes (1 or 2 minutes depending on network connection). In case of any errors or the process taking long kindly check the logs for the deployed pods for `geofm-gateway`, and/or `geofm-gateway-celery-worker`. In some cases you might need to restart the pods. In addition if you need to restart any of the port-forwards you can use the port fowarding commands defined in section 1.2 above." ] }, { "cell_type": "code", "execution_count": null, "id": "8dd2a8c2", "metadata": {}, "outputs": [], "source": [ "gfm_client.poll_finetuning_until_finished(tune_response['tune_id'])" ] }, { "cell_type": "markdown", "id": "ebb78ecb", "metadata": {}, "source": [ "Finally, we submit an inference run to try-out the tune above. The payload below defines which spatial and temporal domain to run for the inference. " ] }, { "cell_type": "code", "execution_count": null, "id": "26222ac4", "metadata": {}, "outputs": [], "source": [ "\n", "tune_id = tune_response['tune_id']\n", "\n", "# Define the inference payload\n", "payload = {\n", " \"model_display_name\": \"geofm-sandbox-models\",\n", " \"fine_tuning_id\":tune_id,\n", " \"location\": \"Dakhin Petbaha, Raha, Nagaon, Assam, India\",\n", " \"description\": \"Flood Assam local with sentinel aws\",\n", " \"spatial_domain\": {\n", " \"bbox\": [\n", " [92.703396, 26.247896, 92.748087, 26.267903]\n", " ],\n", " \"urls\": [],\n", " \"tiles\": [],\n", " \"polygons\": []\n", " },\n", " \"temporal_domain\": [\n", " \"2024-07-25_2024-07-28\"\n", " ]\n", "}\n", "\n", "# Submit the inference request\n", "gfm_client.try_out_tune(tune_id=tune_id, data=payload)" ] }, { "cell_type": "markdown", "id": "fb047f44", "metadata": {}, "source": [ "Navigate to the geospatial studio UI and click on the `Start fine-tuning` card. Under Model & Tunes identify and click on `geofm-sandbox-models` and under `History` click on the entry with name `Flood Assam local with sentinel aws` i.e. the inference you just run. This loads the inference page with status of your inference and if results are available they will be loaded in the UI." ] }, { "cell_type": "markdown", "id": "35284f82", "metadata": {}, "source": [ "### 1.5 Tuning a model from a dataset (requires GPUs)" ] }, { "cell_type": "markdown", "id": "c175231e", "metadata": {}, "source": [ "At the moment for successful tuning, you need access to GPUs in your cluster or in your local machine (still work in progress for leveraging Mac GPUs). \n", "\n", "In order to run a fine-tuning task, you need to select the following items:\n", "\n", "* tuning task type - what type of learning task are you attempting? segmentation, regression etc. This has already been onboarded in 1.4 above. You can also onboard a new task type if you need to.\n", "* base foundation model - which geospatial foundation model will you use as the starting point for your tuning task?\n", "* fine-tuning dataset - what dataset will you use to train the model for your particular application?\n", "\n", "\n", "\n", "To start, let us onboard the base model weights from which we will fine-tune/customize a model for a downstream task of burn-scars identification. The base model is the foundation model (encoder) which has been pre-trained and has the basic understanding of the data. More information can currently be found on the different models in the documentation." ] }, { "cell_type": "code", "execution_count": null, "id": "83813f82", "metadata": {}, "outputs": [], "source": [ "with open(\"payloads/backbones/backbone-Prithvi_EO_V2_300M.json\",\"r\") as f:\n", " backbone= json.load(f)\n", "onboard_backbone_response=gfm_client.create_base_model(backbone)\n", "display( json.dumps(onboard_backbone_response,indent=2))" ] }, { "cell_type": "markdown", "id": "be34e9aa", "metadata": {}, "source": [ "Now that we have a backbone model registered in our studio instance we need to onboard a sample of burscar labelled dataset. Here we are downloading to your cluster this dataset from a presigned link and this process may take a few minutes depending on your network speeds." ] }, { "cell_type": "code", "execution_count": null, "id": "1fddacd9", "metadata": {}, "outputs": [], "source": [ "with open(\"payloads/datasets/dataset-burn_scars.json\",\"r\") as f:\n", " wild_fire_dataset= json.load(f)\n", "onboard_dataset_response=gfm_client.onboard_dataset(data=wild_fire_dataset)\n", "display(json.dumps(onboard_dataset_response,indent=2))" ] }, { "cell_type": "markdown", "id": "a8a66d4f", "metadata": {}, "source": [ "You can then monitor the status of the onboarding process through the polling function. Since dowloading and curating this dataset is resource intensive, in an environment with limited CPUs and RAM you might have other functionality of your deployment dropped to allocate more resources to this tasks. For example port-forwarding may be dropped and the polling will fail with a some errors. In that case, give it some time and later attempt to portforward your services again using the port-fowarding commands defined in section 1.2 above and re-run polling. \n", "\n", "> You may also want to monitor the logs for the onboarding job pod that pops up in your cluster." ] }, { "cell_type": "code", "execution_count": null, "id": "6a9aad8e", "metadata": {}, "outputs": [], "source": [ "gfm_client.poll_onboard_dataset_until_finished(onboard_dataset_response[\"dataset_id\"])" ] }, { "cell_type": "markdown", "id": "15c95b55", "metadata": {}, "source": [ "Note: Currently, for local deployments with access to non-NVIDIA GPUs (for example in Mac), you will need to run the fine-tuning outside of the local cluster, and the resulting model can be onboarded back to the local cluster for inference. This will be addressed in future, and is not an issue for cluster deployments with accessible GPUs. For this case jump to section [1.5.1 - Tuning a model from a dataset using Mac GPUs](#151-tuning-a-model-from-a-dataset-using-mac-gpus)" ] }, { "cell_type": "markdown", "id": "e25de618", "metadata": {}, "source": [ "#### 1.5.0 Tuning a model from a dataset in a cluster deployments with accessible GPUs\n", "\n", "Now that we have a backbone model and a dataset registered in our studio instance we can now trigger a fine tuning job. We will use the Prithvi-EO-V2-300M backbone model and the burn scars dataset we onboarded earlier. We will use the default tuning template for this task. The tuning template is a configuration file that defines the hyperparameters for the fine tuning job. You can find the default tuning template in the payloads/tune_templates directory. You can also create your own tuning template and register it in the studio instance. \n", "\n", "Now we can trigger a fine tuning job, using the payload and script below. " ] }, { "cell_type": "code", "execution_count": null, "id": "e65cb6fe", "metadata": {}, "outputs": [], "source": [ "# Get the dataset id, base model id and tune template id\n", "dataset_id = onboard_dataset_response['dataset_id']\n", "base_model_id = onboard_backbone_response['id']\n", "tune_template_id = template_response['id']" ] }, { "cell_type": "code", "execution_count": null, "id": "7b733a99", "metadata": {}, "outputs": [], "source": [ "payload={\n", " \"name\": \"burn-scars-demo\",\n", " \"description\": \"Segmentation\",\n", " \"dataset_id\": dataset_id,\n", " \"base_model_id\": base_model_id,\n", " \"tune_template_id\": tune_template_id,\n", "}\n", "\n", "tune_submitted=gfm_client.submit_tune( payload,output='json')\n", "display( json.dumps(tune_submitted,indent=2))" ] }, { "cell_type": "markdown", "id": "b4dee1b6", "metadata": {}, "source": [ "You can then monitor the status of your tune polling function, or alternatively monitor progress and view the dataset factory in the UI. Since tuning is resource intensive, in an environment with limited CPUs and RAM you might have other functionality of your deployment dropped to allocate more resources to this tasks. For example port-forwarding may be dropped and the polling will fail with some errors. In that case, give it some time and later attempt to portforward your services again using the port-fowarding commands defined in section 1.2 above and re-run polling. \n", "\n", "> You may also want to monitor the logs for the `geotune-xxx` job pod that pops up in your cluster. In cases where you have a signle GPU in your cluster, you need to scale down the `terratorch-inference` deployment pods to zero to release the GPU for tuning, otherwise, the geotune pod will remain pending if it lacks a GPU to bind to." ] }, { "cell_type": "code", "execution_count": null, "id": "789fee6d", "metadata": {}, "outputs": [], "source": [ "gfm_client.poll_finetuning_until_finished(tune_id=tune_submitted['tune_id'])" ] }, { "cell_type": "markdown", "id": "2c1f6e83", "metadata": {}, "source": [ "After the tune above completes, we can trigger an inference run. This can be run through the SDK as below, where you define the spatial and temporal domain to run the inference.\n", "\n", "> Below, we show an expanded payload for submitting the inference to demonstrate how you can override the different configurations." ] }, { "cell_type": "code", "execution_count": null, "id": "35228921", "metadata": {}, "outputs": [], "source": [ "tune_id = tune_submitted['tune_id']\n", "\n", "# Define the inference payload\n", "payload={\n", " \"model_display_name\":\"geofm-sandbox-models\",\n", " \"location\":\"Red Bluff, California, United States\",\n", " \"description\":\"Park Fire Aug 2024\",\n", " \"spatial_domain\":{\n", " \"bbox\":[\n", " \n", " ],\n", " \"urls\":[\n", " \"https://geospatial-studio-example-data.s3.us-east.cloud-object-storage.appdomain.cloud/examples-for-inference/park_fire_scaled.tif\"\n", " ],\n", " \"tiles\":[\n", " \n", " ],\n", " \"polygons\":[\n", " \n", " ]\n", " },\n", " \"temporal_domain\":[\n", " \"2024-08-12\"\n", " ],\n", " \"pipeline_steps\":[\n", " {\n", " \"status\":\"READY\",\n", " \"process_id\":\"url-connector\",\n", " \"step_number\":0\n", " },\n", " {\n", " \"status\":\"WAITING\",\n", " \"process_id\":\"terratorch-inference\",\n", " \"step_number\":1\n", " },\n", " {\n", " \"status\":\"WAITING\",\n", " \"process_id\":\"postprocess-generic\",\n", " \"step_number\":2\n", " },\n", " {\n", " \"status\":\"WAITING\",\n", " \"process_id\":\"push-to-geoserver\",\n", " \"step_number\":3\n", " }\n", " ],\n", " \"post_processing\":{\n", " \"cloud_masking\":\"False\",\n", " \"ocean_masking\":\"False\",\n", " \"snow_ice_masking\":null,\n", " \"permanent_water_masking\":\"False\"\n", " },\n", " \"model_input_data_spec\":[\n", " {\n", " \"bands\":[\n", " {\n", " \"index\":\"0\",\n", " \"RGB_band\":\"B\",\n", " \"band_name\":\"Blue\",\n", " \"scaling_factor\":\"0.0001\"\n", " },\n", " {\n", " \"index\":\"1\",\n", " \"RGB_band\":\"G\",\n", " \"band_name\":\"Green\",\n", " \"scaling_factor\":\"0.0001\"\n", " },\n", " {\n", " \"index\":\"2\",\n", " \"RGB_band\":\"R\",\n", " \"band_name\":\"Red\",\n", " \"scaling_factor\":\"0.0001\"\n", " },\n", " {\n", " \"index\":\"3\",\n", " \"band_name\":\"NIR_Narrow\",\n", " \"scaling_factor\":\"0.0001\"\n", " },\n", " {\n", " \"index\":\"4\",\n", " \"band_name\":\"SWIR1\",\n", " \"scaling_factor\":\"0.0001\"\n", " },\n", " {\n", " \"index\":\"5\",\n", " \"band_name\":\"SWIR2\",\n", " \"scaling_factor\":\"0.0001\"\n", " }\n", " ],\n", " \"connector\":\"sentinelhub\",\n", " \"collection\":\"hls_l30\",\n", " \"file_suffix\":\"_merged.tif\",\n", " \"modality_tag\":\"HLS_L30\"\n", " }\n", " ],\n", " \"geoserver_push\":[\n", " {\n", " \"z_index\":0,\n", " \"workspace\":\"geofm\",\n", " \"layer_name\":\"input_rgb\",\n", " \"file_suffix\":\"\",\n", " \"display_name\":\"Input image (RGB)\",\n", " \"filepath_key\":\"model_input_original_image_rgb\",\n", " \"geoserver_style\":{\n", " \"rgb\":[\n", " {\n", " \"label\":\"RedChannel\",\n", " \"channel\":1,\n", " \"maxValue\":255,\n", " \"minValue\":0\n", " },\n", " {\n", " \"label\":\"GreenChannel\",\n", " \"channel\":2,\n", " \"maxValue\":255,\n", " \"minValue\":0\n", " },\n", " {\n", " \"label\":\"BlueChannel\",\n", " \"channel\":3,\n", " \"maxValue\":255,\n", " \"minValue\":0\n", " }\n", " ]\n", " },\n", " \"visible_by_default\":\"True\"\n", " },\n", " {\n", " \"z_index\":1,\n", " \"workspace\":\"geofm\",\n", " \"layer_name\":\"pred\",\n", " \"file_suffix\":\"\",\n", " \"display_name\":\"Model prediction\",\n", " \"filepath_key\":\"model_output_image\",\n", " \"geoserver_style\":{\n", " \"segmentation\":[\n", " {\n", " \"color\":\"#000000\",\n", " \"label\":\"ignore\",\n", " \"opacity\":0,\n", " \"quantity\":\"-1\"\n", " },\n", " {\n", " \"color\":\"#000000\",\n", " \"label\":\"no-data\",\n", " \"opacity\":0,\n", " \"quantity\":\"0\"\n", " },\n", " {\n", " \"color\":\"#ab4f4f\",\n", " \"label\":\"fire-scar\",\n", " \"opacity\":1,\n", " \"quantity\":\"1\"\n", " }\n", " ]\n", " },\n", " \"visible_by_default\":\"True\"\n", " }\n", " ]\n", "}\n", "# Submit the inference request\n", "gfm_client.try_out_tune(tune_id=tune_id, data=payload)\n" ] }, { "cell_type": "markdown", "id": "d57cfda1", "metadata": {}, "source": [ "Navigate to the geospatial studio UI and click on the `Start fine-tuning` card. Under `Model & Tunes` identify and click on `geofm-sandbox-models` and under `History` click on the entry with name `Park Fire Aug 2024` i.e. the inference you just run. This loads the inference page with status of your inference and if results are available they will be loaded in the UI." ] }, { "cell_type": "markdown", "id": "59301986", "metadata": {}, "source": [ "#### 1.5.1 Tuning a model from a dataset using Mac GPUs\n", "If you have access to Mac GPUs, you can use the following code to tune a model from a dataset. This is useful for testing and development purposes.\n", "\n", "TBA!!!" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }