{
"cells": [
{
"cell_type": "markdown",
"id": "7e2fa7e2",
"metadata": {},
"source": [
"# Introduction\n",
"\n",
"This tutorial will demonstrate how to use the QFieldCloud API to build custom applications and undertake analysis that use data stored and managed by a QFieldCloud instance. \n",
"\n",
"We'll introduce the QFieldCloud API and demonstrate how to use the qfieldcloud-sdk, a Python package and client to make requests to the QFieldCloud API. We'll use these tools to complete two tasks:\n",
"\n",
"1. Create interactive chart and web map visualisations using QFieldCloud data.\n",
"2. Use QFieldCloud data in an accuracy assessment of the ESA World Cover v200 dataset.\n",
"\n",
"Along the way we'll provide code snippets that illustrate how to use the qfieldcloud-sdk that you can expand upon for your own applications. "
]
},
{
"cell_type": "markdown",
"id": "1baa4bce",
"metadata": {},
"source": [
"## QFieldCloud API\n",
"\n",
"QFieldCloud comes with a REST API which can be used to interact with projects and data stored in QFieldCloud. This supports various use cases including building web apps on top of data collected in-the-field using QField. \n",
"\n",
"The API for the hosted version of QFieldCloud can be found at: https://app.qfield.cloud/swagger/\n",
"\n",
"The API for the QFieldCloud instance that will be used in this workshop can be found at: https://pgc.livelihoods-and-landscapes.com/swagger/\n",
"\n",
"The QFieldCloud API has endpoints for authentication, managing users and teams, querying and managing projects, and querying and downloading project data. \n",
"\n",
"## QFieldCloud SDK\n",
"\n",
"The qfieldcloud-sdk is the official client to connect to the QFieldCloud API. The qfieldcloud-sdk is a Python package and can be installed using `pip`:\n",
"\n",
"```\n",
"pip install qfieldcloud-sdk\n",
"```\n",
"\n",
"This makes it well suited for integrating QFieldCloud data in data analysis workflows that leverage other tools in the Python ecosystem (e.g. GeoPandas, sklearn) or web applications (e.g. Django, FastAPI; QFieldCloud is actually a Django application).\n",
"\n",
"Both QFieldCloud and qfieldcloud-sdk are developed by OPENGIS.ch, the developers of QField. "
]
},
{
"cell_type": "markdown",
"id": "f4c332dc",
"metadata": {},
"source": [
"### Setup for Colab\n",
"\n",
"If you are running this notebook using Google Colab you will need to uncomment the below lines and install qfieldcloud-sdk, geopandas, and rasterio. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "22f55bb1",
"metadata": {},
"outputs": [],
"source": [
"# !pip install qfieldcloud-sdk==0.6.1\n",
"# !pip install geopandas\n",
"# !pip install rasterio"
]
},
{
"cell_type": "markdown",
"id": "2b8f3977",
"metadata": {},
"source": [
"### Import packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e88d0a9e",
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"import rasterio\n",
"import os\n",
"import plotly.express as px\n",
"import geopandas as gpd\n",
"from qfieldcloud_sdk import sdk\n",
"from pathlib import Path\n",
"from sklearn.metrics import accuracy_score\n",
"import plotly.io as pio\n",
"pio.renderers.default = \"jupyterlab\"\n"
]
},
{
"cell_type": "markdown",
"id": "0007c4dc",
"metadata": {},
"source": [
"### Login\n",
"\n",
"Create a `Client` object and login to QFieldCloud. The constructor function for a `Client` takes the URL for the QFieldCloud API as an argument.\n",
"\n",
"Here, we pass in the URL for the api for QFieldCloud instance being used for this workshop: https://pgc.livelihoods-and-landscapes.com/api/v1/"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f42f488d",
"metadata": {},
"outputs": [],
"source": [
"# Create a client object\n",
"client = sdk.Client(\n",
" url=\"https://pgc.livelihoods-and-landscapes.com/api/v1/\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "404b145d",
"metadata": {},
"source": [
"The `Client` object has methods for authentication and querying users, projects, and data stored in QFieldCloud. First, let's login to our QFieldCloud instance. \n",
"\n",
"A successful login returns a token that can be used to make authenticated requests to the QFieldCloud API end points.\n",
"\n",
"**For non-demonstration purposes, don't pass in credentials as clear text!**"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "822cbf91",
"metadata": {},
"outputs": [],
"source": [
"# Authenticate using the client's login() method\n",
"client.login(\n",
" username=\"demo_user\",\n",
" password=\"demo_user\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "4d8b39a9",
"metadata": {},
"source": [
"### Query projects\n",
"\n",
"Call `list_projects()` on the authenticated `Client` object to get a list of the user's projects.\n",
"\n",
"`list_projects()` returns a list of dictionary objects with project metadata including its id, name, owner, description, status, and the user's role on the project."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ffb93162",
"metadata": {},
"outputs": [],
"source": [
"projects = client.list_projects()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "94fd3def",
"metadata": {},
"outputs": [],
"source": [
"projects"
]
},
{
"cell_type": "markdown",
"id": "b25df7d3",
"metadata": {},
"source": [
"### List project files\n",
"\n",
"The `list_remote_files()` method can be used to list files associated with a project. The `list_remote_files()` method takes a `project_id` as an argument and returns a list of dictionary objects describing the project's files and the file versions. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ed5b0619",
"metadata": {},
"outputs": [],
"source": [
"# list project files stored in QFieldCloud\n",
"files = client.list_remote_files(\n",
" project_id=\"2127b0a8-ced6-4129-a56b-4e8edf332d3d\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11d541ce",
"metadata": {},
"outputs": [],
"source": [
"# print the first file object\n",
"files[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7d4180ba",
"metadata": {},
"outputs": [],
"source": [
"# print filenames\n",
"for i in files:\n",
" print(i[\"name\"])"
]
},
{
"cell_type": "markdown",
"id": "861a5d8e",
"metadata": {},
"source": [
"### Download files\n",
"\n",
"We can use the `download_file()` method to download a specified file from QFieldCloud. \n",
"\n",
"The `download_file()` method has `project_id`, `remote_filename`, `local_filename`, `download_type`, and `show_progress` parameters.\n",
"\n",
"The `local_filename` parameter expects a `Path` object from the `pathlib` module. \n",
"\n",
"The qfieldcloud-sdk has a `FileTransferType` class which specifies whether we want the `PROJECT` or `PACKAGE` files. Here, we want the `PROJECT` files. \n",
"\n",
"Let's download `data.gpkg` which stores the ground truth points collected in the field using QField."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2008344f",
"metadata": {},
"outputs": [],
"source": [
"# Create a path object for the file to download\n",
"local_filename = Path(os.path.join(os.getcwd(), \"data.gpkg\"))\n",
"\n",
"# Download the file from QFieldCloud\n",
"client.download_file(\n",
" project_id=\"2127b0a8-ced6-4129-a56b-4e8edf332d3d\",\n",
" remote_filename=\"data.gpkg\",\n",
" local_filename=local_filename,\n",
" download_type=sdk.FileTransferType.PROJECT,\n",
" show_progress=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3804cc89",
"metadata": {},
"outputs": [],
"source": [
"# Check data.gpkg downloaded OK\n",
"print(f\"downloaded data.gpkg successfully: {'data.gpkg' in os.listdir()}\")"
]
},
{
"cell_type": "markdown",
"id": "064193dd",
"metadata": {},
"source": [
"### Visualise data\n",
"\n",
"Now we have downloaded data from the QFieldCloud API we can visualise and analyse it. First, let's explore the data using charts and web map widgets."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8c0202dd",
"metadata": {},
"outputs": [],
"source": [
"# Read the data into a GeoPandas GeoDataFrame\n",
"gdf = gpd.read_file(os.path.join(os.getcwd(), \"data.gpkg\"))"
]
},
{
"cell_type": "markdown",
"id": "a520d584",
"metadata": {},
"source": [
"First, let's inspect the data in data table."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eff1a239",
"metadata": {},
"outputs": [],
"source": [
"display(gdf)"
]
},
{
"cell_type": "markdown",
"id": "dfa4cbf2",
"metadata": {},
"source": [
"Next, let's create interactive visualisations using the data downloaded from QFieldCloud. We'll use Plotly Express to create interactive figures and web maps. \n",
"\n",
"We can use the `px.histogram()` function to create a bar plot of the counts of observations for each land cover class in our QFieldCloud project."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eb6579e9",
"metadata": {},
"outputs": [],
"source": [
"color_discrete_map={\n",
" \"1\": \"#00097B\",\n",
" \"2\": \"#04e3a5\",\n",
" \"3\": \"#8a6d1d\",\n",
" \"4\": \"#ffffff\",\n",
" \"5\": \"#ff9143\",\n",
" \"6\": \"#d0ff14\",\n",
" \"7\": \"#a4c93f\",\n",
" \"8\": \"#377d22\"}\n",
"\n",
"fig = px.histogram(\n",
" gdf, \n",
" x=\"land_cover_class\", \n",
" color=\"land_cover_class\",\n",
" color_discrete_map=color_discrete_map, \n",
" title=\"Number of observations per-land cover class\",\n",
" labels={\"land_cover_class\": \"land cover class\"}\n",
")\n",
"\n",
"fig.update_layout(\n",
" xaxis = dict(\n",
" tickmode = \"array\",\n",
" tickvals = [1, 2, 3, 4, 5, 6, 7, 8],\n",
" ticktext = [\"water\", \"mangrove\", \"bare soil\", \"urban\", \"cropland\", \"grassland\", \"shrubland\", \"trees\"]\n",
" )\n",
")\n",
"\n",
"fig.show()"
]
},
{
"cell_type": "markdown",
"id": "66d5c281",
"metadata": {},
"source": [
"Next, let's visualise the data in our QFieldCloud project on a web map using the `px.scatter_mapbox()` function."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "53df7b92",
"metadata": {},
"outputs": [],
"source": [
"color_discrete_map={\n",
" \"1\": \"#00097B\",\n",
" \"2\": \"#04e3a5\",\n",
" \"3\": \"#8a6d1d\",\n",
" \"4\": \"#ffffff\",\n",
" \"5\": \"#ff9143\",\n",
" \"6\": \"#d0ff14\",\n",
" \"7\": \"#a4c93f\",\n",
" \"8\": \"#377d22\"}\n",
"\n",
"fig = px.scatter_mapbox(\n",
" gdf,\n",
" lat=gdf.geometry.y,\n",
" lon=gdf.geometry.x,\n",
" zoom=12,\n",
" mapbox_style=\"open-street-map\",\n",
" color=\"land_cover_class\",\n",
" color_discrete_map=color_discrete_map\n",
")\n",
"\n",
"fig.show()"
]
},
{
"cell_type": "markdown",
"id": "a7b9ab72",
"metadata": {},
"source": [
"### Accuracy Assessment\n",
"\n",
"Here, we'll use the ground truth data that we've collected in the field using QField to perform a quick accuracy assessment of the ESA World Cover v200 land cover map.\n",
"\n",
"Let's download clip of the ESA World Cover v200 land cover map that covers Suva and the surrounding area."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49faf59e",
"metadata": {},
"outputs": [],
"source": [
"!wget \"https://github.com/livelihoods-and-landscapes/pacific-geo-conf/raw/main/esa-world-cover-v2-suva.tif\""
]
},
{
"cell_type": "markdown",
"id": "6d9af92b",
"metadata": {},
"source": [
"Sample the ESA World Cover v2 land cover class at each location where we've collected a ground truth point. Append the predicted class as a column to our `GeoDataFrame` `gdf`. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b03aa002",
"metadata": {},
"outputs": [],
"source": [
"# based on https://geopandas.org/en/stable/gallery/geopandas_rasterio_sample.html\n",
"coord_list = [(x,y) for x,y in zip(gdf[\"geometry\"].x , gdf[\"geometry\"].y)]\n",
"\n",
"with rasterio.open(os.path.join(os.getcwd(), \"esa-world-cover-v2-suva.tif\")) as src:\n",
" meta = src.meta\n",
" img = src.read(1)\n",
" gdf[\"predicted_land_cover\"] = [str(x[0]) for x in src.sample(coord_list)]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "acab9565",
"metadata": {},
"outputs": [],
"source": [
"display(gdf)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5e322f24",
"metadata": {},
"outputs": [],
"source": [
"# accuracy score\n",
"print(f\"the accuracy score for the ESA World Cover v2 land cover map is {round(accuracy_score(gdf['land_cover_class'], gdf['predicted_land_cover']), 2)}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2c27a65a",
"metadata": {},
"outputs": [],
"source": [
"# quick look at the land cover map to check it seems OK\n",
"# px.imshow(img)"
]
},
{
"cell_type": "markdown",
"id": "e661f779",
"metadata": {},
"source": [
"Finally, we can visualise a confusion matrix as a heatmap. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "41aac12d",
"metadata": {},
"outputs": [],
"source": [
"fig = px.density_heatmap(\n",
" gdf,\n",
" x=\"land_cover_class\",\n",
" y=\"predicted_land_cover\",\n",
" text_auto=True,\n",
" labels={\"land_cover_class\": \"land cover class\",\n",
" \"predicted_land_cover\": \"ESA World Cover prediction\"}\n",
")\n",
"\n",
"fig.update_layout(\n",
" xaxis = dict(\n",
" tickmode = \"array\",\n",
" tickvals = [1, 2, 3, 4, 5, 6, 7, 8],\n",
" ticktext = [\"water\", \"mangrove\", \"bare soil\", \"urban\", \"cropland\", \"grassland\", \"shrubland\", \"trees\"]\n",
" ),\n",
" yaxis = dict(\n",
" tickmode = \"array\",\n",
" tickvals = [1, 2, 3, 4, 6, 8],\n",
" ticktext = [\"water\", \"mangrove\", \"bare soil\", \"urban\", \"grassland\", \"trees\"]\n",
" )\n",
")\n",
"\n",
"fig.show()\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.8 ('base')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
},
"vscode": {
"interpreter": {
"hash": "0a3a540d733e13dd2cfc4c76ea59809fcb1c94b9ada5b1181b5aea89bebb185d"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}