{ "cells": [ { "cell_type": "markdown", "id": "c0b2bc1d", "metadata": {}, "source": [ "# 080 Google Earth Engine in Python\n", "\n", "Draft notes under development" ] }, { "cell_type": "markdown", "id": "182deb6a", "metadata": {}, "source": [ "## GEE\n", "\n", "Google Earth Engine is a powerful cloud resource provided by Google that has found wide application in Earth Observation (EO) and related disciplines. \n", "\n", "The power of the tool comes from provision of the underlying processing tools on the remote cloud resource, **alongside** vast quantities of EO data. \n", "\n", "The purpose of this notebook is to provide a brief introduction to some of the basic GEE functionality, datasets and coding in Python.\n", "\n", "### EE\n", "\n", "The GEE library is imported as `ee`. When you want to use GEE resources, you will need to authenticate your account using a google login. We can set this going using `ee.Authenticate()`. This will pop up another browser window. \n", "\n", "You should follow the instructions on that to obtain an authentification code (e.g. `401AX4XfWj37NbOf5RnY4d4910lyW76yh2B0j9Rvj0aVqs9AVmf8oKM`) and initialise GEE (`ee.Initialize()`). You will only need to do that once per session." ] }, { "cell_type": "code", "execution_count": 11, "id": "ef654a1a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "

To authorize access needed by Earth Engine, open the following\n", " URL in a web browser and follow the instructions:

\n", "

https://accounts.google.com/o/oauth2/auth?client_id=517222506229-vsmmajv00ul0bs7p89v5m89qs8eb9359.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fearthengine+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&code_challenge=2OI00sTloC_svLa7KJPAxJIcwORffBNnOdx4yzBDwvo&code_challenge_method=S256

\n", "

The authorization workflow will generate a code, which you\n", " should paste in the box below

\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Enter verification code: 4/1AX4XfWiEu_p9CW1qMJ4Lv1pIZyYTeCK0IfwFy6N3ohAxZ5JLieVIxPFQT9Q\\\n", "\n", "Successfully saved authorization token.\n" ] } ], "source": [ "import ee\n", "\n", "ee.Authenticate()\n", "ee.Initialize()" ] }, { "cell_type": "markdown", "id": "60c27906", "metadata": {}, "source": [ "The main data types we need to be interested in for GEE are:\n", "\n", "| Class | example | comment | \n", "|---:|---:|---:|\n", "[Image](https://developers.google.com/earth-engine/guides/image_overview) | `image = ee.Image('JAXA/ALOS/AW3D30/V2_2')` | An image, often with multiple bands you can access with `image.bandNames()`\n", "| [ImageCollection](https://developers.google.com/earth-engine/apidocs/ee-imagecollection) |`imageCollection = ee.ImageCollection(\"COPERNICUS/S2_SR\")`| A collection (set) of `Image`s. You can e.g. use `imageCollection.map(lambda )`\n", "\n", "\n", "For a full set of docs (in [Javascript, but very similar to Python](https://giswqs.medium.com/15-converting-earth-engine-javascripts-to-python-code-with-just-a-few-mouse-clicks-6aa02b1268e1)) in the [GEE API docs](https://developers.google.com/earth-engine/apidocs)." ] }, { "cell_type": "markdown", "id": "dab5ddf4", "metadata": {}, "source": [ "If we look at the example for `Image`, we can gain an insight into how GEE works:" ] }, { "cell_type": "code", "execution_count": 12, "id": "65384d46", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ee.List({\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Image.bandNames\",\n", " \"arguments\": {\n", " \"image\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Image.load\",\n", " \"arguments\": {\n", " \"id\": {\n", " \"constantValue\": \"JAXA/ALOS/AW3D30/V2_2\"\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", "})\n" ] } ], "source": [ "image = ee.Image('JAXA/ALOS/AW3D30/V2_2')\n", "print(image.bandNames())" ] }, { "cell_type": "markdown", "id": "417fc1f1", "metadata": {}, "source": [ "The reason this doesn't seem to be run, is there hat all of this code (and the rest resulting list) is server-side code. So all of this he internal prcessing, which is computationally efficient ads the processing happens on the same machines that hold the data (in effect). But we want to print the value of something, we want that information on the Client side, which is slow as it involves sending the data over the internet. You should [read more on the pghilisohpy](https://developers.google.com/earth-engine/guides/client_server) of clibnet-Server architechure and how to write cide in iyt\n" ] }, { "cell_type": "code", "execution_count": 20, "id": "c8c0146d", "metadata": {}, "outputs": [], "source": [ "imageCollection = ee.ImageCollection(\"COPERNICUS/S2_SR\")a]\n", "firstOne = imageCollection.map(lambda x : x.multiply(0.01))cd" ] }, { "cell_type": "code", "execution_count": 24, "id": "206742f9", "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'reducer': , 'geometry': ee.Geometry({\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Feature.geometry\",\n", " \"arguments\": {\n", " \"feature\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Collection.first\",\n", " \"arguments\": {\n", " \"collection\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Collection.filter\",\n", " \"arguments\": {\n", " \"collection\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Collection.loadTable\",\n", " \"arguments\": {\n", " \"tableId\": {\n", " \"constantValue\": \"EPA/Ecoregions/2013/L3\"\n", " }\n", " }\n", " }\n", " },\n", " \"filter\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Filter.equals\",\n", " \"arguments\": {\n", " \"leftField\": {\n", " \"constantValue\": \"us_l3name\"\n", " },\n", " \"rightValue\": {\n", " \"constantValue\": \"Sierra Nevada\"\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", "}), 'scale': 30, 'maxPixels': 1000000000.0}\n", "ee.Dictionary({\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Image.reduceRegion\",\n", " \"arguments\": {\n", " \"geometry\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Feature.geometry\",\n", " \"arguments\": {\n", " \"feature\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Collection.first\",\n", " \"arguments\": {\n", " \"collection\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Collection.filter\",\n", " \"arguments\": {\n", " \"collection\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Collection.loadTable\",\n", " \"arguments\": {\n", " \"tableId\": {\n", " \"constantValue\": \"EPA/Ecoregions/2013/L3\"\n", " }\n", " }\n", " }\n", " },\n", " \"filter\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Filter.equals\",\n", " \"arguments\": {\n", " \"leftField\": {\n", " \"constantValue\": \"us_l3name\"\n", " },\n", " \"rightValue\": {\n", " \"constantValue\": \"Sierra Nevada\"\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " }\n", " },\n", " \"image\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Image.load\",\n", " \"arguments\": {\n", " \"id\": {\n", " \"constantValue\": \"JAXA/ALOS/AW3D30/V2_2\"\n", " }\n", " }\n", " }\n", " },\n", " \"maxPixels\": {\n", " \"constantValue\": 1000000000.0\n", " },\n", " \"reducer\": {\n", " \"functionInvocationValue\": {\n", " \"functionName\": \"Reducer.mean\",\n", " \"arguments\": {}\n", " }\n", " },\n", " \"scale\": {\n", " \"constantValue\": 30\n", " }\n", " }\n", " }\n", "})\n" ] } ], "source": [ "region = ee.Feature(ee.FeatureCollection('EPA/Ecoregions/2013/L3')\\\n", " .filter(ee.Filter.eq('us_l3name', 'Sierra Nevada'))\\\n", " .first());\n", "\n", "\n", "idict = {\n", " 'reducer': ee.Reducer.mean(),\n", " 'geometry': region.geometry(),\n", " 'scale': 30,\n", " 'maxPixels': 1e9\n", "}\n", "print(idict)\n", "# Reduce the region. The region parameter is the Feature geometry.\n", "meanDictionary = image.reduceRegion(**idict)\n", "\n", "# The result is a Dictionary. Print it.\n", "print(meanDictionary);" ] }, { "cell_type": "markdown", "id": "b895a0c6", "metadata": {}, "source": [ "When we print this function, we might think that it would print out the band names for 'JAXA/ALOS/AW3D30/V2_2', but instead it returns data in an `ee.List` type that sets up a dictionary describing the hierachy of command we want to run. " ] }, { "cell_type": "markdown", "id": "27fac31b", "metadata": {}, "source": [ "## GEEmap\n", "\n", "One of the best resources built around the core GEE functionality is the `geemap` library from [https://github.com/giswqs](Qiusheng Wu) to provide a simple mapping front end. You will find a huge number of blog entries and tutorials with examples. \n", "\n", "`Wu, Q., (2020). geemap: A Python package for interactive mapping with Google Earth Engine. The Journal of Open Source Software, 5(51), 2305.` [https://doi.org/10.21105/joss.02305](https://doi.org/10.21105/joss.02305).\n", "\n", "One thing to bear in mind with interactive processing in Jupyter notebooks in Python is that you need to be aware of which active window you are using. Notice that if yiou want to finish processing in one cell and run code in another cell, you need to stop the code running in the first cell (stop button on the control panel). \n" ] }, { "cell_type": "markdown", "id": "00fe2925", "metadata": {}, "source": [ "## Image collections\n", "\n", "First you might want to take a look at the datasets available on GEE by looking in the [dataset catalog](https://developers.google.com/earth-engine/datasets/catalog/). \n", "\n", "Suppose we want to develop a visualisation using Sentinel-2 MSI data. In that case, we could select the dataset `\"COPERNICUS/S2_SR\"`:\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "5c5a31f5", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ee.imagecollection.ImageCollection" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s2 = ee.ImageCollection(\"COPERNICUS/S2_SR\")\n", "\n", "# what type?\n", "type(s2)" ] }, { "cell_type": "markdown", "id": "3e864084", "metadata": {}, "source": [ "The variable `s2` contains an EE `ImageCollection`, which is the core data type for collections of raster spatial data assets. \n", "\n", "An image collection is made up of a group of images. The method `first()`, for example, selects the first image in the collection. " ] }, { "cell_type": "code", "execution_count": 70, "id": "6e63ef21", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ee.image.Image" ] }, "execution_count": 70, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s2_image = s2.first()\n", "\n", "# what type?\n", "type(s2_image)" ] }, { "cell_type": "markdown", "id": "ead95076", "metadata": {}, "source": [ "To do some processing on this collection, the main functionality that processes or filters over the image collection are:\n", "\n", "| Operation | example | comment | \n", "|---:|---:|---:|\n", "| `filterBounds(geometry)` | `s2.filterBounds(ee.Geometry.Point(0.1276,51.5072))` | filter physical extent by defined geometry | \n", "| `filterDate(start, opt_end=None)` | `s2.filterDate('2019-01-01', '2019-12-31')` | filter the collection by date | \n", "| `limit(n)` | `s2.limit(5)` | limit the number of images in the collection to `n` | \n", "| `map(algorithm)` | `s2.map()` | map `algorithm` to each image in the image collection | \n", "\n", "\n", "\n", "\n", "\n", "| Operation | example | comment | \n", "|---:|---:|---:|\n", "| `reduce(Reducer)` | `s2.reduce(ee.Reducer.median())` | Apply the reducer to the image collection. This results in a single |" ] }, { "cell_type": "code", "execution_count": 55, "id": "f233eee1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Help on method map in module ee.collection:\n", "\n", "map(algorithm, opt_dropNulls=None) method of ee.imagecollection.ImageCollection instance\n", " Maps an algorithm over a collection.\n", " \n", " Args:\n", " algorithm: The operation to map over the images or features of the\n", " collection, a Python function that receives an image or features and\n", " returns one. The function is called only once and the result is\n", " captured as a description, so it cannot perform imperative operations\n", " or rely on external state.\n", " opt_dropNulls: If true, the mapped algorithm is allowed to return nulls,\n", " and the elements for which it returns nulls will be dropped.\n", " \n", " Returns:\n", " The mapped collection.\n", " \n", " Raises:\n", " ee_exception.EEException: if algorithm is not a function.\n", "\n" ] } ], "source": [ "help(s2.map)" ] }, { "cell_type": "code", "execution_count": 54, "id": "8f7aa345", "metadata": {}, "outputs": [], "source": [ "# centre longitude,latitude\n", "location = 0.1276, 51.5072\n", "cloud_thresh = 30\n", "\n", "collection = ee.ImageCollection(\"COPERNICUS/S2_SR\")\\\n", ".filterBounds(ee.Geometry.Point(*location)).filterDate('2019-01-01', '2019-12-31') \\\n", " .filter(ee.Filter.lte('CLOUDY_PIXEL_PERCENTAGE', cloud_thresh))" ] }, { "cell_type": "code", "execution_count": 53, "id": "fdf9f237", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "69ee530787d54a5c95799cd60eafb435", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map(center=[51.5072, 0.1276], controls=(WidgetControl(options=['position', 'transparent_bg'], widget=HBox(chil…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import ee\n", "import geemap\n", "\n", "# Create a map centered at (lon, lat).\n", "Map = geemap.Map(center=location, zoom=10)\n", "\n", "# Compute the median of each pixel for each band of the 5 least cloudy scenes.\n", "median = collection.limit(5).reduce(ee.Reducer.median())\n", "\n", "# Define visualization parameters in an object literal.\n", "vizParams = {'bands': ['B4_median', 'B3_median', 'B2_median'],\n", " 'min': 0, 'max': 3000, 'gamma': 1.3}\n", "\n", "Map.setCenter(*location, 10)\n", "Map.addLayer(median, vizParams, 'Median image')\n", "# Display the map.\n", "Map" ] }, { "cell_type": "code", "execution_count": null, "id": "17e809b4", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 7, "id": "a0f2b1b7", "metadata": {}, "outputs": [], "source": [ "import ee" ] }, { "cell_type": "code", "execution_count": 12, "id": "6c73376c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Help on function ee_search in module geemap.common:\n", "\n", "ee_search(asset_limit=100)\n", " Search Earth Engine API and user assets. If you received a warning (IOPub message rate exceeded) in Jupyter notebook, you can relaunch Jupyter notebook using the following command:\n", " jupyter notebook --NotebookApp.iopub_msg_rate_limit=10000\n", " \n", " Args:\n", " asset_limit (int, optional): The number of assets to display for each asset type, i.e., Image, ImageCollection, and FeatureCollection. Defaults to 100.\n", "\n" ] } ], "source": [ "help(geemap.ee_search)" ] }, { "cell_type": "code", "execution_count": null, "id": "5f1e8530", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "conda env:geog0111-geog0111", "language": "python", "name": "conda-env-geog0111-geog0111-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 5 }