{
"cells": [
{
"cell_type": "markdown",
"id": "civil-aaron",
"metadata": {},
"source": [
"# Verbs\n",
"\n",
"In this notebook we will describe the different verbs that semantique offers. Remember that result instructions in query recipes can be formulated by combining basic building blocks into processing chains. These processing chains start with a reference. For a description of those, see the [References notebook](references.ipynb). At the query recipe construction stage, the reference is nothing more than a small piece of text. When executing the recipe, the query processor solves this reference and evaluates it internally into a multi-dimensional array filled with data values. Several actions can then be applied to this array. These actions are all labeled with an action word that should intuitively describe the operation they are performing. That is why we call them *verbs*. The same building blocks can also be used when constructing a set of mapping rules according to semantiques native mapping configuration.\n",
"\n",
"## Content\n",
"\n",
"- [Verbs for single arrays](#Verbs-for-single-arrays)\n",
"- [Verbs for collections of arrays](#Verbs-for-collections-of-arrays)\n",
"- [Split-apply-combine structures](#Split-apply-combine-structures)\n",
"- [Utility verbs](#Utility-verbs)\n",
"\n",
"## Prepare\n",
"\n",
"Import packages:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "mighty-methodology",
"metadata": {},
"outputs": [],
"source": [
"import semantique as sq"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "paperback-indiana",
"metadata": {},
"outputs": [],
"source": [
"import geopandas as gpd\n",
"import pandas as pd\n",
"import numpy as np\n",
"import xarray as xr\n",
"import json\n",
"import copy"
]
},
{
"cell_type": "markdown",
"id": "f13759b9",
"metadata": {},
"source": [
"Set the context of query execution:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "e42db710",
"metadata": {},
"outputs": [],
"source": [
"# Load a mapping.\n",
"with open(\"files/mapping.json\", \"r\") as file:\n",
" mapping = sq.mapping.Semantique(json.load(file))\n",
"\n",
"# Represent an EO data cube.\n",
"with open(\"files/layout_gtiff.json\", \"r\") as file:\n",
" dc = sq.datacube.GeotiffArchive(json.load(file), src = \"files/layers_gtiff.zip\")\n",
"\n",
"# Set the spatio-temporal extent.\n",
"space = sq.SpatialExtent(gpd.read_file(\"files/footprint.geojson\"))\n",
"time = sq.TemporalExtent(\"2019-01-01\", \"2020-12-31\")\n",
"\n",
"# Collect the full context.\n",
"# Including additional configuration parameters.\n",
"context = {\n",
" \"datacube\": dc, \n",
" \"mapping\": mapping,\n",
" \"space\": space,\n",
" \"time\": time,\n",
" \"crs\": 3035, \n",
" \"tz\": \"UTC\", \n",
" \"spatial_resolution\": [-1800, 1800]\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "weighted-settle",
"metadata": {},
"source": [
"## Verbs for single arrays\n",
"\n",
"Most verbs in semantique are verbs that apply an action to a single array. The currently implemented verbs in this category are:\n",
"\n",
"- [Evaluate](#Evaluate): Evaluates an expression for each pixel in an array.\n",
"- [Extract](#Extract): Extracts dimension coordinates as a new one-dimensional array.\n",
"- [Filter](#Filter): Filters values from an array.\n",
"- [Assign](#Assign): Assign a new value to each pixel in an array.\n",
"- [Reduce](#Reduce): Reduces the dimensionality of an array.\n",
"- [Shift](#Shift): Shifts array values a given number of steps along a dimension.\n",
"- [Smooth](#Smooth): Smoothes array values by applying a moving window function.\n",
"- [Trim](#Trim): Trims the dimensions of an array.\n",
"- [Delineate](#Delineate): Delineates spatio-temporal objects in a binary array.\n",
"- [Fill](#Fill): Fills nodata values in an array.\n",
"- [Groupby](#Groupby): Splits an array into multiple groups.\n",
"\n",
"### Evaluate\n",
"\n",
"The evaluate verb evaluates an expression for each pixel in an array. These expressions can take many different forms, but each of them accepts the value of a specific pixel in the input array and applies some operator to it. The result of that operation is the new value of that particular pixel in the output array. That is, the output array always has the *same shape* as the input array. Below, we will show different forms of expressions, and the built-in [operators](https://zgis.github.io/semantique/reference.html#operator-functions) that semantique offers for them. For advanced users, it is also possible to define their own custom operators, which is explained in the notebook on [Internal query processing](processor.ipynb#Adding-custom-operators).\n",
"\n",
"You can specify an operator function simply by its name:\n",
"\n",
"```python\n",
"sq.entity(\"water\").evaluate(\"not\")\n",
"```\n",
"\n",
"To be autocomplete-friendly, you can also use built-in constants that refer to an operator function. These are stored in the [operators module](https://zgis.github.io/semantique/reference.html#operators) of semantique. Hence, the snippet below is equal to the one above:\n",
"\n",
"```python\n",
"sq.entity(\"water\").evaluate(sq.operators.NOT)\n",
"```\n",
"\n",
"When using the evaluate verb, it is important to be aware of the *value type* of the input array(s). For example, they may contain nominal, ordinal, binary or numerical data. Different operators may only support specific (combinations of) value types. For details, see [here](processor.ipynb#Tracking-value-types)."
]
},
{
"cell_type": "markdown",
"id": "fe6292f2",
"metadata": {},
"source": [
"#### Univariate expressions\n",
"\n",
"The simplest form of expressions are the univariate expressions. For each pixel value $x_{i}$ in input array $X$, these expressions *only* consider $x_{i}$, and apply an operator to it. That is, the expression is of the following form.\n",
"\n",
"$$\n",
"expression = operator(x_{i})\n",
"$$\n",
"\n",
"This means that the output array has the same shape as the input array, and that each pixel value in the output array is the result of the univariate expression evaluated on the value of the corresponding pixel in the input array.\n",
"\n",
"\n",
"\n",
"The built-in operators for this purpose include the **numerical univariate operators**, which are intended for usage on numerical arrays:\n",
"\n",
"- `absolute`: Computes the absolute value of $x_{i}$.\n",
"- `floor`: Returns the largest integer $k$ such that $k \\leq x_{i}$.\n",
"- `ceiling`: Returns the smallest integer $k$ such that $k \\geq x_{i}$.\n",
"- `exponential`: Computes the exponential value of $x_{i}$, i.e. $e^{x_{i}}$.\n",
"- `natural_logarithm`: Computes the natural logarithm of $x_{i}$.\n",
"- `square_root`: Computes the square root of $x_{i}$.\n",
"- `cube_root`: Computes the cube root of $x_{i}$.\n",
"\n",
"Next to those there are the **boolean univariate operators**, which are intended for usage on binary arrays:\n",
"\n",
"- `not`: Returns $1$ if $x_{i} = 0$, and $0$ if $x_{i} \\neq 0$.\n",
"\n",
"There are also the **trigonometric univariate operators**, which are intended for usage on numerical arrays in which the numbers are angles, which may for example occur in data layers containing sun zenith angles. All these functions assume the angles are in radians, except the conversion function \"to_radians\", which excepts the angles to be in degrees.\n",
"\n",
"- `cosine`: Computes the cosine of $x_{i}$.\n",
"- `sine`: Computes the sine of $x_{i}$.\n",
"- `tangent`: Computes the tangent of $x_{i}$.\n",
"- `secant`: Computes the secant of $x_{i}$, which is equal to $\\frac{1}{cos(x_{i})}$\n",
"- `cosecant`: Computes the cosecant of $x_{i}$, which is equal to $\\frac{1}{sin(x_{i})}$\n",
"- `cotangent`: Computes the cotangent of $x_{i}$, which is equal to $\\frac{1}{tan(x_{i})}$\n",
"- `to_degrees`: Converts angles in radians to angles in degrees.\n",
"- `to_radians`: Converts angles in degrees to angles in radians.\n",
"\n",
"Finally, there are two operators that allow to locate nodata values in an array:\n",
"\n",
"- `is_missing`: Returns $1$ if $x_{i}$ is a missing observation, and $0$ otherwise.\n",
"- `not_missing`: Returns $1$ if $x_{i}$ is valid observation, and $0$ otherwise.\n",
"\n",
"For example, applying the `not` operator to the translated semantic concept *water* marks all non-water pixels as 1 and all water pixels as 0."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "48a5b097",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "2194e1fd",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"water\"] = sq.entity(\"water\")\n",
"recipe[\"not_water\"] = sq.result(\"water\").evaluate(\"not\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "eaf9290e",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "d30685c7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[1., 1., 1.],\n",
" [1., 1., 1.],\n",
" [1., 1., 1.]],\n",
"\n",
" [[1., 0., 1.],\n",
" [1., 0., 1.],\n",
" [1., 1., 1.]],\n",
"\n",
" [[1., 1., 1.],\n",
" [0., 1., 1.],\n",
" [0., 1., 1.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"not_water\"]"
]
},
{
"cell_type": "markdown",
"id": "fb99db57",
"metadata": {},
"source": [
"#### Bivariate expressions\n",
"\n",
"Bivariate expressions consist of a left-hand side value, a right-hand side value and an operator that in some way combines these values. For each pixel value $x_{i}$ in input cube $X$, the left-hand side value of the expression is $x_{i}$, and the right-hand side value of the expression is another value $y_{i}$.\n",
"\n",
"$$\n",
"expression = x_{i} \\; operator \\; y_{i}\n",
"$$\n",
"\n",
"The right-hand side value $y_{i}$ can be a constant, meaning that the same right-hand side value is used for all pixels in $X$.\n",
"\n",
"\n",
"\n",
"The right-hand side value $y_{i}$ can also be a pixel value from another array $Y$. In that case, $Y$ should have the same shape as $X$ (or able to be aligned to that shape, see [below](#Aligning-two-arrays)), such that each pixel $x_{i} \\in X$ has a *matching* pixel $y_{i} \\in Y$, i.e. a pixel that has exactly the same coordinates for each dimension.\n",
"\n",
"\n",
"\n",
"The built-in operators for bivariate expressions can be subdivided into different categories. The **algebraic operators** are intended for usage on numerical arrays and perform an operation of arithmetic:\n",
"\n",
"- `add`: Adds $y_{i}$ to $x_{i}$..\n",
"- `subtract`: Subtracts $y_{i}$ from $x_{i}$.\n",
"- `multiply`: Multiplies $x_{i}$ by $y_{i}$.\n",
"- `divide`: Divides $x_{i}$ by $y_{i}$.\n",
"- `power`: Raises $x_{i}$ to the $y_{i}$th power.\n",
"- `normalized_difference`: Calculates $\\frac{x_{i} - y_{i}}{x_{i} + y_{i}}$."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "eba9137e",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "f8562882",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"count\"] = sq.entity(\"vegetation\").reduce(\"count\", \"time\")\n",
"recipe[\"twice\"] = sq.result(\"count\").evaluate(\"multiply\", 2)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "98c959a3",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "64893942",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[4., 0., 2.],\n",
" [2., 0., 4.],\n",
" [2., 2., 4.]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: discrete"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"twice\"]"
]
},
{
"cell_type": "markdown",
"id": "4f763153",
"metadata": {},
"source": [
"The **relational operators** evaluate a condition. The result is always binary, i.e. true (1) when the condition holds and false (2) when it doesn't. Some of the conditions test for equality, and hence can be used on any array whenever the value type of the left-hand values is the same as the value type of the right-hand values:\n",
"\n",
"- `equal`: Returns $1$ if $x_{i} = y_{i}$, and $0$ otherwise.\n",
"- `not_equal`: Returns $1$ if $x_{i} \\neq y_{i}$, and $0$ otherwise.\n",
"\n",
"The other conditions imply a fixed order among the values, and hence should not be used on nominal arrays:\n",
"\n",
"- `greater`: Returns $1$ if $x_{i} > y_{i}$, and $0$ otherwise.\n",
"- `less`: Returns $1$ if $x_{i} < y_{i}$, and $0$ otherwise.\n",
"- `greater_equal`: Returns $1$ if $x_{i} \\geq y_{i}$, and $0$ otherwise.\n",
"- `less_equal`:Returns $1$ if $x_{i} \\leq y_{i}$, and $0$ otherwise."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "1f62b9bd",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "623c151c",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"count\"] = sq.entity(\"vegetation\").reduce(\"count\", \"time\")\n",
"recipe[\"high\"] = sq.result(\"count\").evaluate(\"greater_equal\", 2)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "694df2dd",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "1710e9e6",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[0., 0., 0.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]],\n",
"\n",
" [[1., 1., 1.],\n",
" [1., 1., 1.],\n",
" [1., 1., 1.]],\n",
"\n",
" [[1., 0., 0.],\n",
" [1., 0., 1.],\n",
" [1., 0., 1.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"both\"]"
]
},
{
"cell_type": "markdown",
"id": "04c266d8",
"metadata": {},
"source": [
"The **membership operators** are a special category of operators that test for set membership. That is, they test for each pixel value $x_{i}$ in cube $X$ if it is or is not member of a set of values $Y$. $Y$ may be another array, but in that case it will be treated as being a set. That is, for each pixel value $x_{i} \\in X$ it will be tested if it occurs anywhere in array $Y$.\n",
"\n",
"- `in`: Returns $1$ if $x_{i} \\in Y$, and $0$ otherwise. Here, $Y$ is a finite set of values that remains constant for each $x_{i}$.\n",
"- `not_in`: Returns $1$ if $x_{i} \\notin Y$, and $0$ otherwise. Here, $Y$ is a finite set of values that remains constant for each $x_{i}$.\n",
"\n",
"To represent a set, you can use the [set()](https://zgis.github.io/semantique/_generated/semantique.set.html) function of semantique. Alternatively, you can use Pythons built-in set, list or tuple objects."
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "a98681a7",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "33583f01",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"colors\"] = sq.appearance(\"colortype\")\n",
"recipe[\"water\"] = sq.result(\"colors\").evaluate(\"in\", sq.set(21, 22, 23, 24))"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "b8946ae9",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "f0ee7af5",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[0., 0., 0.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]],\n",
"\n",
" [[0., 1., 0.],\n",
" [0., 1., 0.],\n",
" [0., 0., 0.]],\n",
"\n",
" [[0., 0., 0.],\n",
" [1., 0., 0.],\n",
" [1., 0., 0.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"water\"]"
]
},
{
"cell_type": "markdown",
"id": "75ddea76",
"metadata": {},
"source": [
"The **temporal relational operators** are relational operators specifically designed to deal with *time instants* and *time intervals* as operand values. Hence, it requires each pixel value $x_{i}$ in input array $X$ to be a time instant. The right-hand side of the expression can either be a single time instant, a time interval (i.e. a list of two time instants representing the start and end of an interval) or another array $Y$ in which each pixel value $y_{i}$ is a time instant. The latter will in practice be evaluated as being a time interval with the earliest time instant being the start of the interval, and the latest time instant the end of the interval.\n",
"\n",
"The currently implemented temporal relational operators are:\n",
"\n",
"- `after`: When $y$ is a time instant: returns $1$ if $x_{i} > y$, and $0$ otherwise. When $y$ is a time interval: returns $1$ if $x_{i} > max(y)$, and $0$ otherwise.\n",
"- `before`: When $y$ is a time instant: returns $1$ if $x_{i} < y$, and $0$ otherwise. When $y$ is a time interval: returns $1$ if $x_{i} < min(y)$, and $0$ otherwise.\n",
"- `during`: Returns $1$ if $min(y) \\leq x_{i} \\leq max(y)$, and $0$ otherwise. Only intended for time intervals as $y$.\n",
"\n",
"To construct time instants and time intervals for usage as right-hand operand, you can use the [time_instant()](https://zgis.github.io/semantique/_generated/semantique.time_instant.html) and [time_interval()](https://zgis.github.io/semantique/_generated/semantique.time_interval.html) functions that semantique provides. The first expects a single datetime value, while the latter expects two datetime values (i.e. the start and end of the interval, with the interval being closed at both sides). You can provide datetimes in formats as \"2020-12-31\" or \"2020/12/31\", but also complete ISO8601 timestamps such as \"2020-12-31T14:37:22\". As long as the [Timestamp](https://pandas.pydata.org/docs/reference/api/pandas.Timestamp.html) initializer of the [pandas](https://pandas.pydata.org/) package can understand it, it is supported by semantique. Any additional keyword arguments will be forwarded to this initializer."
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "d8dbda5c",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "d58a2839",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"times\"] = sq.entity(\"water\").extract(\"time\")\n",
"recipe[\"early\"] = sq.result(\"times\").evaluate(\"before\", sq.time_instant(\"2019-12-31\"))\n",
"recipe[\"late\"] = sq.result(\"times\").evaluate(\"during\", sq.time_interval(\"2020-01-01\", \"2020-12-31\"))"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "380aaac9",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "ce8d6906",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([0., 1., 1.])\n",
"Coordinates:\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-1...\n",
" temporal_ref int64 0\n",
"Attributes:\n",
" value_type: binary"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"late\"]"
]
},
{
"cell_type": "markdown",
"id": "022310e0",
"metadata": {},
"source": [
"The **spatial relational operators** are relational operators specifically designed to deal with *geometries* as operand values. It requires each pixel value $x_{i}$ in input array $X$ to be a tuple of spatial (x,y) coordinates. The right-hand side of the expression can either be one or more spatial geometries, or another array $Y$ in which each pixel value $y_{i}$ is a coordinate tuple. The latter will in practice be evaluated as being a geometry covering the spatial bounding box of the array.\n",
"\n",
"The currently implemented spatial relational operators are:\n",
"\n",
"- `intersects`: Returns $1$ if the spatial point with the coordinates of $x_{i}$ spatially intersects with geometry $y$, and $0$ otherwise.\n",
"\n",
"To construct a spatial geometry for usage as right-hand operand, you can use the [geometry()](https://zgis.github.io/semantique/_generated/semantique.geometry.html) function that semantique offers. It expects an object that can be read by the [GeoDataFrame](https://geopandas.org/docs/reference/api/geopandas.GeoDataFrame.html) initializer of the [geopandas](https://geopandas.org/en/stable/) package. Any additional keyword arguments will be forwarded to this initializer. In practice, this means you can read any GDAL-supported file format with [geopandas.read_file()](https://geopandas.org/en/stable/docs/reference/api/geopandas.read_file.html), and then use that object to create spatial geometries."
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "aa37fd8f",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "2bf5d7d6",
"metadata": {},
"outputs": [],
"source": [
"parcels = gpd.read_file(\"files/parcels.geojson\")\n",
"recipe[\"coords\"] = sq.entity(\"water\").extract(\"space\")\n",
"recipe[\"in_parcel\"] = sq.result(\"coords\").evaluate(\"intersects\", sq.geometry(parcels))"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "a5d51525",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/luuk/miniconda3/envs/semantique/lib/python3.11/site-packages/numpy/core/numeric.py:407: RuntimeWarning: invalid value encountered in cast\n",
" multiarray.copyto(res, fill_value, casting='unsafe')\n"
]
}
],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "c708d5f7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[0, 1, 0],\n",
" [0, 0, 0],\n",
" [0, 0, 0]])\n",
"Coordinates:\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" spatial_ref int64 0\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: binary"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"in_parcel\"]"
]
},
{
"cell_type": "markdown",
"id": "b3bdacbb",
"metadata": {},
"source": [
"#### Aligning two arrays\n",
"\n",
"In bivariate expressions involving two arrays, the second array $Y$ does not necessarily have to be of the same shape as input array $X$. Instead, it should be possible to *align* it to that shape. This can be done in two ways. \n",
"\n",
"First consider the case where $Y$ has the same dimensions as $X$, but not all coordinate values of $X$ are present in $Y$. In that case, we can align $Y$ with $X$ such that pixel values at position $i$ in both arrays, i.e. $x_{i}$ and $y_{i}$ respectively, belong to pixels with the *same coordinates*. If $y_{i}$ was not originally part of $Y$, we assign it a nodata value. This also works vice-versa, with coordinate values in $Y$ that are not present in $X$.\n",
"\n",
"\n",
"\n",
"Secondly, consider a case where $Y$ has one or more dimensions with exactly the same coordinate values as $X$, but does not have *all* the dimensions that $X$ has. In that case, we can align $Y$ with $X$ by duplicating its values along those dimensions that are missing. This does *not* work vice versa. When cube $Y$ has more dimensions that cube $X$, there is no clear way to define how to subset the values in $Y$ to match the shape of $X$.\n",
"\n",
"\n",
"\n",
"Alignment is something you have to be aware of to understand how bivariate expressions are evaluated. However, it is not something you have to do for yourself. Internally, the query processor takes care of it."
]
},
{
"cell_type": "markdown",
"id": "81c20b59",
"metadata": {},
"source": [
"### Extract\n",
"\n",
"The extract verbs extracts the coordinates of a specified dimension from an array, and returns them as a new, one-dimensional array.\n",
"\n",
"\n",
"\n",
"Dimensions can be referred to by their name. For example:\n",
"\n",
"```python\n",
"sq.entity(\"water\").extract(\"time\")\n",
"```\n",
"\n",
"Semantique also contains built-in constants storing the reserved names for the spatio-temporal dimensions. These include the time dimensions, the spatial x and y dimensions, as well as the space dimension as a whole, which is a multi-indexed dimension containing both the y and x coordinates. These constants are stored in the [dimensions module](https://zgis.github.io/semantique/reference.html#dimensions) can be used to refer to the spatio-temporal dimensions without knowing their exact name. Hence, the snippet below is equivalent to the one above:\n",
"\n",
"```python\n",
"sq.entity(\"water\").extract(sq.dimensions.TIME)\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "ec873117",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "9ba7ae75",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"time\"] = sq.entity(\"water\").extract(\"time\")\n",
"recipe[\"x\"] = sq.entity(\"water\").extract(\"x\")"
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "aac1618e",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "111a705a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[(2695500.0, 4531500.0), (2695500.0, 4533300.0),\n",
" (2695500.0, 4535100.0)],\n",
" [(2693700.0, 4531500.0), (2693700.0, 4533300.0),\n",
" (2693700.0, 4535100.0)],\n",
" [(2691900.0, 4531500.0), (2691900.0, 4533300.0),\n",
" (2691900.0, 4535100.0)]], dtype=object)\n",
"Coordinates:\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" spatial_ref int64 0\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: coords"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"space\"]"
]
},
{
"cell_type": "markdown",
"id": "b119e14b",
"metadata": {},
"source": [
"#### Extracting a component of dimension coordinates\n",
"\n",
"Coordinate values of some dimensions may consist of multiple components. For example, the time coordinates are made up of a year, a month, a day, an hour, etc, while the multi-indexed space coordinates are made up of an x and a y coordinate. If you are only interested in a single component of a dimension, you can specify that through the second, optional argument of the extract verb. A component can be referred to directly by its name. For example:\n",
"\n",
"```python\n",
"sq.entity(\"water\").extract(\"time\", \"year\")\n",
"```\n",
"\n",
"To be autocomplete-friendly, you can also use built-in constants that refer to a dimension component. These are stored in the [components module](https://zgis.github.io/semantique/reference.html#components) of semantique. The snippet below is equal to the snippet above:\n",
"\n",
"```python\n",
"sq.entity(\"water\").extract(sq.dimensions.TIME, sq.components.time.YEAR)\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "64629836",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "52dee97f",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"years\"] = sq.entity(\"water\").extract(\"time\", \"year\")"
]
},
{
"cell_type": "code",
"execution_count": 55,
"id": "896d38b2",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "3583d9b8",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[1., 1., 1.],\n",
" [1., 1., 1.],\n",
" [1., 1., 1.]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: nominal\n",
" value_labels: {1: 'feature_1'}"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"feats\"]"
]
},
{
"cell_type": "markdown",
"id": "bored-gasoline",
"metadata": {},
"source": [
"### Filter\n",
"\n",
"The filter verb filters values from an array. That is, the output array is a *subset* of the input array. Which values in the input array are kept, and which are removed, is defined by a second, binary array which we call the *filterer*. The filterer should have the same shape as the input array (or able to be aligned to that shape, see [below](#Filtering-by-dimension-coordinates)), such that each pixel $x_{i}$ in input cube $X$ has a *matching* pixel $y_{i}$ in filterer $Y$, i.e. a pixel that has exactly the same coordinates for each dimension. Then:\n",
"\n",
"- $x_{i}$ is kept if $y_{i} \\neq 0$.\n",
"- $x_{i}$ is removed if $y_{i} = 0$.\n",
"\n",
"Here, *kept* means that the pixel value remains unchanged, while *removed* means that the pixel gets a nodata value assigned.\n",
"\n",
"\n",
"\n",
"For example, we may filter only those pixels in the translated semantic concept *water* that where not covered by clouds."
]
},
{
"cell_type": "code",
"execution_count": 61,
"id": "ab7cd644",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "5d453d9c",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"water\"] = sq.entity(\"water\")\n",
"recipe[\"cloud\"] = sq.entity(\"cloud\")\n",
"recipe[\"not_cloud\"] = sq.result(\"cloud\").evaluate(\"not\")\n",
"recipe[\"filtered\"] = sq.result(\"water\").filter(sq.result(\"not_cloud\"))"
]
},
{
"cell_type": "code",
"execution_count": 63,
"id": "65548d49",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 64,
"id": "a845ca5a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[ 2., nan, nan],\n",
" [nan, nan, 2.],\n",
" [nan, nan, 2.]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: discrete"
]
},
"execution_count": 71,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"high\"]"
]
},
{
"cell_type": "markdown",
"id": "2eac13aa",
"metadata": {},
"source": [
"#### Filtering by dimension coordinates\n",
"\n",
"The filterer does not have to be of the same shape as the input array, as long as we can align it to that shape. See [this section](#Aligning-two-arrays) for more details on how this works. In practice, it means that we can also filter values in an array based on the coordinates of a dimension. All we have to do is to construct a filterer for that dimension. Hence, a binary, one-dimensional array that specifies for each of the coordinates if values of pixels having that coordinate should be kept (i.e. 1) or removed (i.e. 0).\n",
"\n",
"\n",
"\n",
"For example, when we only want to keep pixel values observed in 2020:"
]
},
{
"cell_type": "code",
"execution_count": 72,
"id": "0e5ca080",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 73,
"id": "a05eeb61",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"2020\"] = sq.entity(\"water\").filter(sq.self().extract(\"time\", \"year\").evaluate(\"equal\", 2020))"
]
},
{
"cell_type": "code",
"execution_count": 74,
"id": "bff21133",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 75,
"id": "65c88852",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[nan, nan, nan],\n",
" [nan, nan, nan],\n",
" [nan, nan, nan]],\n",
"\n",
" [[ 1., nan, 1.],\n",
" [ 1., nan, 1.],\n",
" [ 1., 1., 1.]],\n",
"\n",
" [[ 1., nan, nan],\n",
" [nan, nan, 1.],\n",
" [nan, nan, 1.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 83,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"true_vegetation\"]"
]
},
{
"cell_type": "markdown",
"id": "e8b80100",
"metadata": {},
"source": [
"### Assign\n",
"\n",
"The assign verb assigns each pixel in an array a new value that is not a function of the original value. The new value can be a constant, meaning that the same right-hand side value is used for all pixels in the input array. The new value can also be a pixel value from another array $Y$. In that case, $Y$ should have the same shape as input array $X$ (or able to be aligned to that shape, see [here](#Aligning-two-arrays)), such that each pixel $x_{i} \\in X$ has a *matching* pixel $y_{i} \\in Y$, i.e. a pixel that has exactly the same coordinates for each dimension. Missing observations (i.e. pixels with a nodata value) in the input are preserved: they never get assigned a new value, unless specifically stated through the *at* argument (see below).\n",
"\n",
"\n",
"\n",
"A trivial example:"
]
},
{
"cell_type": "code",
"execution_count": 84,
"id": "485298de",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 85,
"id": "beb34ff7",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"vegetation\"] = sq.entity(\"vegetation\")\n",
"recipe[\"foo\"] = sq.result(\"vegetation\").assign(-99)"
]
},
{
"cell_type": "code",
"execution_count": 86,
"id": "f56c15c0",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 87,
"id": "a5958682",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[2., 0., 0.],\n",
" [0., 0., 2.],\n",
" [0., 0., 2.]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: discrete"
]
},
"execution_count": 93,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"foo\"]"
]
},
{
"cell_type": "markdown",
"id": "abd7963b",
"metadata": {},
"source": [
"#### Assigning dimension coordinates"
]
},
{
"cell_type": "markdown",
"id": "e571bcd5",
"metadata": {},
"source": [
"When the values to be assigned are taken from another array, this array does not have to be of the same shape as the input array, as long as we can align it to that shape. See [this section](#Aligning-two-arrays) for more details on how this works. In practice, it means that we can also assign dimension coordinates as new values (or optionally, a specific [component](#Extracting-a-component-of-dimension-coordinates) of dimension coordinates).\n",
"\n",
"\n",
"\n",
"For example, for each observation, we want to store the month in which the observation was made:"
]
},
{
"cell_type": "code",
"execution_count": 94,
"id": "e285075e",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 95,
"id": "d6e560d3",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"vegetation\"] = sq.entity(\"vegetation\")\n",
"recipe[\"months\"] = sq.result(\"vegetation\").assign(sq.self().extract(\"time\", \"month\"))"
]
},
{
"cell_type": "code",
"execution_count": 96,
"id": "e129cd2e",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 97,
"id": "cfc30bbf",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[ 2., -999., 1.],\n",
" [ 1., -999., 2.],\n",
" [ 1., 1., 2.]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: discrete"
]
},
"execution_count": 106,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"bar\"]"
]
},
{
"cell_type": "markdown",
"id": "norman-transport",
"metadata": {},
"source": [
"### Reduce\n",
"\n",
"The reduce verb applies a reducer function along a dimension and subsequently drops the reduced dimension. That is, the output array always has one dimension less than the input array. Hence, the reduce verb reduces the dimensionality of an array.\n",
"\n",
"To reduce a dimension, the reducer function operates on each slice of values along the axis of the dimension. Such a *slice* contains one value for each coordinate label of the dimension to reduce over, while the coordinate labels of all other dimensions are *constant* within each slice. The reducer function always returns a single value, such that each slice gets reduced from $n$ values to one value.\n",
"\n",
"For example: an array with a spatial and a temporal dimension contains for each location in space $n$ values, where $n$ is the number of timestamps in the temporal dimension. When we reduce the temporal dimension of this array, the reducer function reduces these $n$ values for each location in space to one value. The resulting array has a single value per location in space, and no temporal dimension anymore.\n",
"\n",
"\n",
"\n",
"The are many different types of reducers available in semantique. For advanced users, it is also possible to define their own custom reducers, which is explained in the notebook on [Internal query processing](processor.ipynb#Adding-custom-reducers).\n",
"\n",
"You can specify a reducer function simply by its name:\n",
"\n",
"```python\n",
"sq.entity(\"water\").reducer(\"mean\", \"time\")\n",
"```\n",
"\n",
"To be autocomplete-friendly, you can also use built-in constants that refer to a reducer function. These are stored in the [reducers module](https://zgis.github.io/semantique/reference.html#reducers) of semantique. In the same way, you can use the built-in constants stored in the [dimensions module](https://zgis.github.io/semantique/reference.html#dimensions) to refer to one of the spatio-temporal dimensions, without having to know its exact name. The snippet below is equal to the snippet above:\n",
"\n",
"```python\n",
"sq.entity(\"water\").reduce(sq.reducers.MEAN, sq.dimensions.TIME)\n",
"```\n",
"\n",
"The built-in reducer functions of semantique currently are:\n",
"\n",
"- `mean`: Returns the average value of each slice $S$.\n",
"- `median`: Returns the median value of each slice $S$.\n",
"- `mode`: Returns the most occuring value in each slice $S$.\n",
"- `max`: Returns the largest value in each slice $S$.\n",
"- `min`: Returns the smallest value in each slice $S$.\n",
"- `range`: Returns the difference between the largest and smallest value in each slice $S$.\n",
"- `n`: Returns the number of observations in each slice $S$.\n",
"- `product`: Returns the product of the values in each slice $S$.\n",
"- `sum`: Returns the sum of the values in each slice $S$.\n",
"- `standard_deviation`: Returns the standard deviation of the values in each slice $S$.\n",
"- `variance`: Returns the variance of the values in each slice $S$.\n",
"- `all`: For each slice $S$, returns $1$ if all $x_{i} \\in S \\neq 0$, and $0$ otherwise.\n",
"- `any`: For each slice $S$, returns $1$ if any $x_{i} \\in S \\neq 0$, and $0$ otherwise.\n",
"- `none`: For each slice $S$, returns $1$ if all $x_{i} \\in S \\eq 0$, and $0$ otherwise.\n",
"- `count`: Counts the number of non-zero values in each slice $S$.\n",
"- `percentage`: Calculates the percentage of non-zero values in each slice $S$.\n",
"- `first`: Returns the first value of each slice $S$.\n",
"- `last`: Returns the last value of each slice $S$.\n",
"\n",
"It is important to mention that nodata values are **ignored** by the reducer functions! That is, for example, when a slice has the values `[1, 1, nan, 1]` the `all` reducer will return \"true\" and the `percentage` reducer will return 100. A reducer will only return a nodata value when *all* values in a slice are nodata values.\n",
"\n",
"Also, when using the reduce verb, it is important to be aware of the *value type* of the input array(s). For example, they may contain nominal, ordinal, binary or numerical data. Different reducers may only support specific value types. For details, see [here](processor.ipynb#Tracking-value-types).\n",
"\n",
"Having said that, lets show some examples. The reduce verb takes as first argument the reducer function to be applied, and as second argument the name of the dimension to be reduced over."
]
},
{
"cell_type": "code",
"execution_count": 107,
"id": "96f39e2e",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 108,
"id": "2992bae6",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"vegetation\"] = sq.entity(\"vegetation\")\n",
"recipe[\"map\"] = sq.result(\"vegetation\").reduce(\"count\", \"time\")\n",
"recipe[\"series\"] = sq.result(\"vegetation\").reduce(\"count\", \"space\")"
]
},
{
"cell_type": "code",
"execution_count": 109,
"id": "lucky-territory",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 110,
"id": "4ac84e2b",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array(10.)\n",
"Coordinates:\n",
" spatial_ref int64 0\n",
" temporal_ref int64 0\n",
"Attributes:\n",
" value_type: discrete"
]
},
"execution_count": 115,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"stat\"]"
]
},
{
"cell_type": "markdown",
"id": "d35e8bb3",
"metadata": {},
"source": [
"### Shift\n",
"\n",
"The shift verb shifts values in an array with a given offset along a given dimension. This allows to compare pixel values with their lagging or leading values along a dimension, e.g. using a comparison operator with the [evaluate()](#Evaluate) verb.\n",
"\n",
"\n",
"\n",
"The offset can be specified by an integer, defining how many steps each pixel value should be shifted. A negative offset implies a shift to the left, while a positive offset implies a shift to the right. For the time dimension, this means that an offset of -1 shifts all pixel values one step into the past, while an offset of 1 shifts all pixel values one step into the future. For the spatial dimensions, the direction of the shift depends on the direction given by the CRS. Normally, for the X dimension, an offset of -1 shifts all pixel values one step (i.e. one time the spatial resolution) to the west, while an offset of 1 shifts all pixel values one step to the east. For the Y dimension, a negative offset normally shifts to the north and a positive offset to the south. A shift along the multi-indexed spatial (x,y) dimension follows the pixel order given by the CRS, normally starting in the top-left corner of the spatial extent and moving down each column of distinct y-coordinates.\n",
"\n",
"The dimension to shift along can be referred to by its name. For example:\n",
"\n",
"```python\n",
"sq.entity(\"water\").shift(\"time\", 1)\n",
"```\n",
"\n",
"You can also use the built-in constants stored in the [dimensions module](https://zgis.github.io/semantique/reference.html#dimensions) to refer to one of the spatio-temporal dimensions, without having to know its exact name. The snippet below is equal to the snippet above:\n",
"\n",
"\n",
"```python\n",
"sq.entity(\"water\").shift(sq.dimensions.TIME, 1)\n",
"```\n",
"\n",
"The pixels at the start of the dimension to be shifted along are filled with nodata values, since there are no values preceeding/succeeding them."
]
},
{
"cell_type": "code",
"execution_count": 116,
"id": "4939ca39",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 117,
"id": "82aa1f80",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"water\"] = sq.entity(\"water\")\n",
"recipe[\"shifted\"] = sq.entity(\"water\").shift(\"time\", 1)"
]
},
{
"cell_type": "code",
"execution_count": 118,
"id": "2c43d866",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 119,
"id": "96597faa",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[nan, nan, nan],\n",
" [nan, nan, nan],\n",
" [nan, nan, nan]],\n",
"\n",
" [[ 0., 0., 0.],\n",
" [ 0., 0., 0.],\n",
" [ 0., 0., 0.]],\n",
"\n",
" [[ 0., 1., 0.],\n",
" [ 0., 1., 0.],\n",
" [ 0., 0., 0.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 120,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"shifted\"]"
]
},
{
"cell_type": "markdown",
"id": "c9f84404",
"metadata": {},
"source": [
"### Smooth\n",
"\n",
"The smooth verbs smoothes values in array by applying a window function over the neighbourhood of each pixel (i.e. a moving window function). The window always moves along a given dimension. The size of the neighbourhood of each pixel can be given as an integer, which specifies how many neighbours at each side of the pixel should be considered. Hence, the pixel being smoothed will always be in the center of the window, with k pixels at its left and k pixels at its right. If the dimension to smooth over is the multi-indexed spatial (x,y) dimension, the size will be used for both the X and Y dimension, forming a square window with the smoothed pixel in the middle.\n",
"\n",
"\n",
"\n",
"The moving window function reduces the values in the neighborhood of a pixel (including the smoothed pixel itself) to a single value, which will be the new value of that pixel. This means that the same functions can be used as for the [reduce()](#Reduce) verb. They can be referred to by their name. For example:\n",
"\n",
"```python\n",
"sq.entity(\"water\").smooth(\"mean\", \"time\", size = 1)\n",
"```\n",
"\n",
"To be autocomplete-friendly, you can also use built-in constants that refer to a reducer function. These are stored in the [reducers module](https://zgis.github.io/semantique/reference.html#reducers) of semantique. In the same way, you can use the built-in constants stored in the [dimensions module](https://zgis.github.io/semantique/reference.html#dimensions) to refer to one of the spatio-temporal dimensions, without having to know its exact name. The snippet below is equal to the snippet above:\n",
"\n",
"\n",
"```python\n",
"sq.entity(\"water\").smooth(sq.reducers.MEAN, sq.dimensions.TIME, size = 1)\n",
"```\n",
"\n",
"By default, the smooth verb requires each window to contain at least two valid data values (i.e. nodata values not included). This can be tuned by setting the *limit* parameter, which defined the minimum number of valid data values each window should have. Pixels that do not have enough data values in their window are assigned a nodata value. This effects the pixels towards the edges of the array, as well as those surrounded by many nodata values."
]
},
{
"cell_type": "code",
"execution_count": 121,
"id": "1658a1fa",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 122,
"id": "96854cc7",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"vegetation\"] = sq.entity(\"vegetation\")\n",
"recipe[\"time_smoothed\"] = sq.result(\"vegetation\").smooth(\"sum\", \"time\", size = 1)\n",
"recipe[\"space_smoothed\"] = sq.result(\"vegetation\").smooth(\"sum\", \"space\", size = 1, limit = 9)"
]
},
{
"cell_type": "code",
"execution_count": 123,
"id": "9c6ef9a3",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 124,
"id": "0605df2b",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[nan, 1., nan],\n",
" [nan, 1., nan],\n",
" [nan, nan, nan]],\n",
"\n",
" [[nan, nan, nan],\n",
" [ 1., nan, nan],\n",
" [ 1., nan, nan]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2020-09-05T10:17:43.167942 2020-12-1...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 134,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"time_trimmed\"]"
]
},
{
"cell_type": "markdown",
"id": "ca0d19ee",
"metadata": {},
"source": [
"### Delineate\n",
"\n",
"The delineate verb delineates sets of \"true\" pixels in a binary array that are connected in space-time. These sets are called spatio-temporal objects. In two-dimensional space (i.e. the spatial X and Y dimensions), pixels are considered to be connected when they are neighbours according to the queen criterion. In one-dimensional time, pixels are considered to be connected when they are directly preceeding/succeeding each other. Pixels belonging to the same object get the same integer index assigned, starting from 1. All pixels that do not belong to an object (i.e. those that are \"false\" in the input) get an index of 0. Nodata values in the input are always preserved.\n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": 135,
"id": "b42a4a6f",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 136,
"id": "e9715a8c",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"water\"] = sq.entity(\"water\")\n",
"recipe[\"objects\"] = sq.result(\"water\").delineate()"
]
},
{
"cell_type": "code",
"execution_count": 137,
"id": "5e053db7",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 138,
"id": "52d167f5",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[0., 0., 0.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]],\n",
"\n",
" [[0., 1., 0.],\n",
" [0., 1., 0.],\n",
" [0., 0., 0.]],\n",
"\n",
" [[0., 0., 0.],\n",
" [2., 0., 0.],\n",
" [2., 0., 0.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: ordinal"
]
},
"execution_count": 139,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"objects\"]"
]
},
{
"cell_type": "markdown",
"id": "c571635c",
"metadata": {},
"source": [
"### Fill\n",
"\n",
"The fill verbs fills nodata values in an array by interpolating the valid data values along a given dimension. There are several interpolation methods to choose from:\n",
"\n",
"- `nearest`: Finds the nearest pixel (based on the dimension coordinates) to the pixel with missing data and copies its value.\n",
"- `linear`: Draws straight lines between the valid data points to determine the values of the missing data points.\n",
"- `cubic`: Fits a cubic polynomial function to the valid data points to determine the values of the missing data points.\n",
"\n",
"When interpolation along the stacked space dimensions, the two-dimensional versions of these interpolation methods are used, i.e. 2D nearest neighbour, bilinear and bicubic. See [here](https://en.wikipedia.org/wiki/Nearest-neighbor_interpolation#/media/File:Comparison_of_1D_and_2D_interpolation.svg) for a figure that clearly explains the different methods. By default the one-dimensional interpolation methods also extrapolate to pixels with missing data at the edges, but this behaviour can be changed by setting `extrapolate = False`.\n",
"\n",
"\n",
"\n",
"A trivial example of one-dimensional interpolation using the *nearest* method:"
]
},
{
"cell_type": "code",
"execution_count": 140,
"id": "58578570",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 141,
"id": "c8bae0f7",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"foo\"] = sq.entity(\"vegetation\").filter_time(\"month\", \"equal\", 12)\n",
"recipe[\"bar\"] = sq.result(\"foo\").fill(\"time\", \"nearest\")"
]
},
{
"cell_type": "code",
"execution_count": 142,
"id": "9cdac258",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 143,
"id": "bf9aa790",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[2. , 1.5, 1. ],\n",
" [1. , 1.5, 2. ],\n",
" [1. , 1. , 2. ]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
" spatial_ref int64 0\n",
"Attributes:\n",
" value_type: discrete\n",
" _FillValue: nan"
]
},
"execution_count": 149,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"bar\"]"
]
},
{
"cell_type": "markdown",
"id": "0f85444e",
"metadata": {},
"source": [
"Another way to fill missing data value is by assigning fixed values using the [assign verb](#Assigning-to-nodata-values)."
]
},
{
"cell_type": "markdown",
"id": "sacred-catch",
"metadata": {},
"source": [
"### Groupby\n",
"\n",
"The groupby verb splits an array into multiple smaller subsets, called groups. That is, the output array is a collection of multiple subsets of the input array. Which pixel ends up in which group is defined by a second array which we call the *grouper*. The grouper should have the same shape as the input array (or able to be aligned to that shape, see [below](#Grouping-by-dimension-coordinates)), such that each pixel $x_{i}$ in input cube $X$ has a *matching* pixel $y_{i}$ in grouper $Y$, i.e. a pixel that has exactly the same coordinates for each dimension. Then, pixels in $X$ that have an equal value for their corresponding pixel in $Y$ are grouped together. Hence, given grouper $Y$, pixels $x_{i}$ and $x_{j}$ in input array $X$ are grouped together if and only if $y_{i} = y_{j}$.\n",
"\n",
"\n",
"\n",
"For example, we can group the translated semantic concept *water* such that each group comprises a set of of water pixels that are connected in space-time (i.e. we use the [delineate()](#Delineate) verb to create the grouper). The result of this operation is a collection of multiple arrays, i.e. one for each object. These array collections object have specific verbs to combine their elements into a single array again. For a description of those, see [the next section](#Verbs-for-collections-of-arrays)."
]
},
{
"cell_type": "code",
"execution_count": 150,
"id": "6bb7689c",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 151,
"id": "ef76e72a",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"water\"] = sq.entity(\"water\")\n",
"recipe[\"objects\"] = sq.result(\"water\").delineate()\n",
"recipe[\"groups\"] = sq.result(\"water\").groupby(sq.result(\"objects\"))"
]
},
{
"cell_type": "code",
"execution_count": 152,
"id": "f46eb36a",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 153,
"id": "311fc011",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[0., 0., 0.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]],\n",
"\n",
" [[0., 0., 0.],\n",
" [1., 0., 0.],\n",
" [1., 0., 0.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 2020-12-1...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 164,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"seasons\"][1]"
]
},
{
"cell_type": "markdown",
"id": "631fad91",
"metadata": {},
"source": [
"#### Multiple groupers\n",
"\n",
"It is also possible to provide a collection of groupers to the groupby verb, as long as their dimensions match. In that case, groups are formed as follows: given grouper $Y$ and grouper $Z$ with matching coordinates, pixels $x_{i}$ and $x_{j}$ in input array $X$ are grouped together if and only if $y_{i} = y_{j}$ *and* $z_{i} = z_{j}$.\n",
"\n",
"\n",
"\n",
"That means for example that we can group the input array along the time dimension in a way that two pixels observed in the same month but a different year end up in a different group."
]
},
{
"cell_type": "code",
"execution_count": 165,
"id": "a50bcf18",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 166,
"id": "6582d228",
"metadata": {},
"outputs": [],
"source": [
"multigrouper = sq.collection(sq.self().extract(\"time\", \"year\"), sq.self().extract(\"time\", \"month\"))\n",
"recipe[\"groups\"] = sq.entity(\"water\").groupby(multigrouper)"
]
},
{
"cell_type": "code",
"execution_count": 167,
"id": "9758bc90",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 168,
"id": "8a6e692f",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[0., 0., 0.],\n",
" [1., 0., 0.],\n",
" [1., 0., 0.]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" time datetime64[ns] 2020-12-19T10:17:34.610661\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 170,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"groups\"][2]"
]
},
{
"cell_type": "markdown",
"id": "4d825366",
"metadata": {},
"source": [
"A shorter formulation for the above statement would be:"
]
},
{
"cell_type": "code",
"execution_count": 171,
"id": "c7587f47",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"groups\"] = sq.entity(\"water\").groupby_time([\"year\", \"month\"])"
]
},
{
"cell_type": "markdown",
"id": "de6dfae1",
"metadata": {},
"source": [
"## Verbs for collections of arrays\n",
"\n",
"When constructing a query recipe, you can start a processing chain with a reference to a [collection of multiple arrays](references.ipynb#Referencing-collections). These array collections have specific verbs that all in some way combine the elements of the collections back into a single array. The currently implemented verbs in this category are:\n",
"\n",
"- [Concatenate](#Concatenate): Concatenates multiple arrays over a new or an existing dimension.\n",
"- [Compose](#Compose): Creates a categorical composition of multiple binary arrays.\n",
"- [Merge](#Merge): Merges values of corresponding pixels in multiple arrays into one by applying a reducer function.\n",
"\n",
"It is important to mention that the verbs are intended for arrays that all have the *same* dimensions (but not necessarily the same coordinates)! They will also work on arrays that do not have the same dimensions, as long as they can all be aligned to each other. However, in these cases you should be aware of the pecularities of alignment, see [here](#Aligning-two-arrays) for details. To summarize: When the arrays in the collection have the same dimensions, but don't share all of their coordinate labels, they get aligned to each other by filling the missing pixels in either of them with nodata values. When one of the arrays in the collection (e.g. $C_{1}$) is missing a dimension that is present in another array in the collection (e.g. $C_{2}$), they get aligned to each other by duplicating the values of $C_{1}$ for each coordinate of the missing dimension. In any case, the coordinates of the output array are always the *union* of all coordinates from the input arrays. Only when the input arrays can in no way be aligned to each other, the verb will throw an error."
]
},
{
"cell_type": "markdown",
"id": "persistent-female",
"metadata": {},
"source": [
"### Concatenate\n",
"\n",
"The concatenate verb concatenates multiple arrays along a given dimension. There are two main ways in which you can do this: either you concatenate along a *new* dimension, or you concatenate along an *existing* dimension.\n",
"\n",
"#### Concatenating along a new dimension\n",
"\n",
"Concatenating multiple arrays along a new dimension is a relatively simple process. Each of the input arrays becomes a dimension in the output array. Lets consider a collection with two two-dimensional arrays $A$ and $B$ that have matching coordinates along the dimensions $\\Gamma$ and $\\Delta$. Concatenating them along a new dimension $E$ will result in a new three-dimensional array $C$ with dimensions $\\Gamma$, $\\Delta$ and $E$. A pixel with coordinates $(\\gamma_{i}, \\delta_{i})$ in array $A$ becomes a pixel with coordinates $(\\gamma_{i}, \\delta_{i}, \\epsilon = A)$ in array $C$, while the pixel with the same coordinates $(\\gamma_{i}, \\delta_{i})$ in array $B$ becomes a pixel with coordinates $(\\gamma_{i}, \\delta_{i}, \\epsilon = B)$ in array $C$.\n",
"\n",
"\n",
"\n",
"All you have to provide to the concatenate verb is the name of the new dimension. Note that this cannot be one of the names semantique has reserved for its spatio-temporal dimensions, see [here](https://zgis.github.io/semantique/reference.html#dimensions). The coordinate labels of the new dimension will be the names of the input arrays."
]
},
{
"cell_type": "code",
"execution_count": 172,
"id": "7520be92",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 173,
"id": "a5dcb597",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"concepts\"] = sq.collection(sq.entity(\"water\"), sq.entity(\"snow\"), sq.entity(\"vegetation\")).\\\n",
" concatenate(\"concept\")"
]
},
{
"cell_type": "code",
"execution_count": 174,
"id": "a0104ee0",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 175,
"id": "facb27fe",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[0., 0., 0.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]],\n",
"\n",
" [[0., 1., 0.],\n",
" [0., 1., 0.],\n",
" [0., 0., 0.]],\n",
"\n",
" [[0., 0., 0.],\n",
" [1., 0., 0.],\n",
" [1., 0., 0.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" AREA_OR_POINT: Area\n",
" scale_factor: 1.0\n",
" add_offset: 0.0\n",
" _FillValue: 1.7976931348623157e+308\n",
" value_type: binary"
]
},
"execution_count": 179,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"water\"]"
]
},
{
"cell_type": "markdown",
"id": "5c6c62ad",
"metadata": {},
"source": [
"We can also concatenate arrays that share coordinate labels of the dimension to concatenate over. However, for these coordinates, only the values of the *first* array in the collection that contains that coordinate, will end up in the output array. For the others, these values will simply be dropped. Only if the value of the first array is a missing value, the value of the second array will be considered, et cetera."
]
},
{
"cell_type": "markdown",
"id": "veterinary-riding",
"metadata": {},
"source": [
"### Compose\n",
"\n",
"The compose verb is primarily meant for collections of binary arrays, i.e. arrays that only have \"true\" (i.e. 1) and \"false\" (i.e. 0) values. Then, a pixel in the output array gets a value of 1 when it was \"true\" in the first array of the collection, a value of 2 of it was \"true\" in the second array of the collection, a value of 3 if it was \"true\" in the third array of the collection, et cetera. Hence, with the compose verb you convert a set of binary arrays into one categorical array.\n",
"\n",
"When a pixel is \"true\" in more than one array in the collection, it gets the index of that array that comes first in the collection. Hence, if a pixel is \"true\" in both the second and third array in a collection, it gets a value of 2 in the output array. When a pixel is not \"true\" for any of the arrays in the collection, it gets a nodata value in the output array.\n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": 180,
"id": "21bbc02b",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 181,
"id": "e2f2a7a5",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"land_cover\"] = sq.collection(sq.entity(\"water\"), sq.entity(\"snow\"), sq.entity(\"vegetation\")).\\\n",
" compose()"
]
},
{
"cell_type": "code",
"execution_count": 182,
"id": "2285f1fb",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 183,
"id": "26a616ff",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[ 2., nan, 2.],\n",
" [ 2., nan, 2.],\n",
" [ 2., 2., 2.]],\n",
"\n",
" [[ 3., 1., 3.],\n",
" [ 3., 1., 3.],\n",
" [ 3., 3., 3.]],\n",
"\n",
" [[ 3., nan, nan],\n",
" [ 1., nan, 3.],\n",
" [ 1., nan, 3.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: nominal\n",
" value_labels: {1: 'water', 2: 'snow', 3: 'vegetation'}"
]
},
"execution_count": 183,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"land_cover\"]"
]
},
{
"cell_type": "markdown",
"id": "domestic-catalyst",
"metadata": {},
"source": [
"### Merge\n",
"\n",
"The merge verb is actually a combination of two other verbs. First, it [concatenates](#Concatenate) the arrays in the collection along a new dimension, and then it [reduces](#Reduce) the output of that over this new dimension. In practice, that means that the merge verb applies a reduction function to each set of pixels that have the same dimension coordinates but are stored in different arrays in the collection. For example, if we merge the translated semantic concepts *water*, *snow* and *vegetation* using the `any` reducer, we get an output cube that contains a \"true\" value (i.e. 1) for a pixel if the value of that pixel was \"true\" in at least one of the water, snow or vegetation arrays, and a \"false\" value (i.e. 0) if the value of that pixel was not \"true\" in any of those.\n",
"\n",
"\n",
"\n",
"The only argument you need to provide to the verb is the reducer function. See the [reduce()](#Reduce) verb for an overview of them."
]
},
{
"cell_type": "code",
"execution_count": 184,
"id": "fd2b9663",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 185,
"id": "81e61ce1",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"any_concept\"] = sq.collection(sq.entity(\"water\"), sq.entity(\"snow\"), sq.entity(\"vegetation\")).\\\n",
" merge(\"any\")"
]
},
{
"cell_type": "code",
"execution_count": 186,
"id": "1d73c3c1",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 187,
"id": "630c719a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[[1., 0., 1.],\n",
" [1., 0., 1.],\n",
" [1., 1., 1.]],\n",
"\n",
" [[1., 1., 1.],\n",
" [1., 1., 1.],\n",
" [1., 1., 1.]],\n",
"\n",
" [[1., 0., 0.],\n",
" [1., 0., 1.],\n",
" [1., 0., 1.]]])\n",
"Coordinates:\n",
" * x (x) float64 4.532e+06 4.533e+06 4.535e+06\n",
" * y (y) float64 2.696e+06 2.694e+06 2.692e+06\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-...\n",
" temporal_ref int64 0\n",
" spatial_feats (y, x) float64 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
"Attributes:\n",
" value_type: binary"
]
},
"execution_count": 187,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"any_concept\"]"
]
},
{
"cell_type": "markdown",
"id": "a484e544",
"metadata": {},
"source": [
"Note that the process of merging a collection of two arrays usually can be modelled as well with the [evaluate()](#Evaluate) verb. For example, the following lines produce identical results:\n",
"\n",
"```python\n",
"sq.collection(A, B).merge(\"any\")\n",
"A.evaluate(\"or\", B)\n",
"```\n",
"\n",
"However, where the evaluate verb can \"merge\" one other array into a given input cube, the merge verb allows to combine an unrestricted number of arrays in one go."
]
},
{
"cell_type": "markdown",
"id": "b90bab3e",
"metadata": {},
"source": [
"## Split-apply-combine structures\n",
"\n",
"All [verbs for single arrays](#Verbs-for-single-arrays) (except the groupby verb) can also be applied to array collections. In that case, they will simply be applied to each element of the collection seperately. Hence, the output will again be an array collection, with the same amount of members.\n",
"\n",
"This allows to model well-know \"split-apply-combine\" processes, such as aggregation. You start with a single array, split it with the [groupby()](#Groupby) verb into a collection, apply one of the verbs for single arrays to each of its members, and then combine them back together using one of the dedicated verbs for array collections.\n",
"\n",
"For example: we want to know the average water count over space for each year in our time dimension separately."
]
},
{
"cell_type": "code",
"execution_count": 188,
"id": "4848b99d",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 189,
"id": "78e79f94",
"metadata": {},
"outputs": [],
"source": [
"recipe[\"avg_count\"] = sq.entity(\"water\").\\\n",
" groupby_time(\"year\").\\\n",
" reduce(\"count\", \"space\").\\\n",
" reduce(\"mean\", \"time\").\\\n",
" concatenate(\"year\")"
]
},
{
"cell_type": "code",
"execution_count": 190,
"id": "20a83c4b",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 191,
"id": "4184de98",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
],
"text/plain": [
"\n",
"array([[ 0., 61., 18.],\n",
" [ 0., 102., 67.]])\n",
"Coordinates:\n",
" spatial_ref int64 0\n",
" * time (time) datetime64[ns] 2019-12-15T10:17:33.408715 ... 2020-1...\n",
" temporal_ref int64 0\n",
" * feat (feat) object 'Northern' 'Southern'\n",
"Attributes:\n",
" value_type: discrete"
]
},
"execution_count": 196,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"out[\"count_per_feat\"]"
]
},
{
"cell_type": "markdown",
"id": "eb68383a",
"metadata": {},
"source": [
"## Utility verbs\n",
"\n",
"An additional set of verbs are the utility verbs. These are verbs that do not affect the values of an array themselves, but rather update the attributes of the array. The currently implemented verbs in this category are:\n",
"\n",
"- [Name](#Name): Give a (new) name to an array.\n",
"\n",
"### Name\n",
"\n",
"The name verbs gives a name to an array. In some cases this can be particularly useful. For example, when [concatenating](#Concatenate) multiple arrays together along a new dimension, the names of these arrays will be used as coordinate labels of this new dimension."
]
},
{
"cell_type": "code",
"execution_count": 197,
"id": "42cdf7ba",
"metadata": {},
"outputs": [],
"source": [
"recipe = sq.QueryRecipe()"
]
},
{
"cell_type": "code",
"execution_count": 198,
"id": "19761e88",
"metadata": {},
"outputs": [],
"source": [
"water = sq.entity(\"water\").name(\"W\")\n",
"snow = sq.entity(\"snow\").name(\"S\")\n",
"vegetation = sq.entity(\"vegetation\").name(\"V\")\n",
"\n",
"recipe[\"concepts\"] = sq.collection(water, snow, vegetation).concatenate(\"concept\")"
]
},
{
"cell_type": "code",
"execution_count": 199,
"id": "c4c5fefd",
"metadata": {},
"outputs": [],
"source": [
"out = recipe.execute(**context)"
]
},
{
"cell_type": "code",
"execution_count": 200,
"id": "8ad547b2",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"