## Table of contents
- [Real-world examples](#examples)
- [Quick Start](#quickstart)
- [Operating in production](#operating)
- [Configuration](#configuration)
- [Authoring Task Templates](#templates)
- [Extending µTask with plugins](#plugins)
- [Contributing](#contributing)
- [License](#license)
## Real-world examples
Here are a few real-world examples that can be implemented with µTask:
### Kubernetes ingress TLS certificate provisioning
A new ingress is created on the production kubernetes cluster. A hook triggers a µTask template that:
- generates a private key
- requests a new certificate
- meets the certificate issuer's challenges
- commits the resulting certificate back to the cluster
### New team member bootstrap
A new member joins the team. The team leader starts a task specifying the new member's name, that:
- asks the new team member to generate an SSH key pair and copy the public key in a µTask-generated form
- registers the public SSH key centrally
- creates accounts on internal services (code repository, CI/CD, internal PaaS, ...) for the new team member
- triggers another task to spawn a development VM
- sends a welcome email full of GIFs
### Payments API asynchronous processing
The payments API receives a request that requires an asynchronous antifraud check. It spawns a task on its companion µTask instance that:
- calls a first risk-assessing API which returns a number
- if the risk is low, the task succeeds immediately
- otherwise it calls a SaaS antifraud solution API which returns a score
- if the score is good, the task succeeds
- if the score is very bad, the task fails
- if it is in between, it triggers a human investigation step where an operator can enter a score in a µTask-generated form
- when it is done, the task sends an event to the payments API to notify of the result
The payments API keeps a reference to the running workflow via its task ID. Operators of the payments API can follow the state of current tasks by requesting the µTask instance directly. Depending on the payments API implementation, it may allow its callers to follow a task's state.
## Quick start
### Running with docker-compose
Download our latest install script, setup your environment and launch your own local instance of µTask.
```bash
mkdir utask && cd utask
wget https://github.com/ovh/utask/releases/latest/download/install-utask.sh
sh install-utask.sh
docker-compose up
```
All the configuration for the application is found in the environment variables in docker-compose.yaml. You'll see that basic auth is setup for user `admin` with password `1234`. Try logging in with this user on the graphical dashboard: [http://localhost:8081/ui/dashboard](http://localhost:8081/ui/dashboard).
You can also explore the API schema: [http://localhost:8081/unsecured/spec.json](http://localhost:8081/unsecured/spec.json).
Request a new task:
Get an overview of all tasks:
Get a detailed view of a running task:
Browse available task templates:
### Running with your own postgres service
Alternatively, you can clone this repository and build the µTask binary:
```bash
make all
```
## Operating in production
The folder you created in the previous step is meant to become a git repo where you version your own task templates and plugins. Re-download and run the latest install script to bump your version of µTask.
You'll deploy your version of µTask by building a docker image based on the official µTask image, which will include your extensions. See the Dockerfile generated during installation.
### Architecture
µTask is designed to run a task scheduler and perform the task workloads within a single runtime: work is not delegated to external agents. Multiple instances of the application will coordinate around a single postgres database: each will be able to determine independently which tasks are available. When an instance of µTask decides to execute a task, it will take hold of that task to avoid collisions, then release it at the end of an execution cycle.
A task will keep running as long as its steps are successfully executed. If a task's execution is interrupted before completion, it will become available to be re-collected by one of the active instances of µTask. That means that execution might start in one instance and resume on a different one.
### Maintenance procedures
#### Key rotation
1. Generate a new key with [symmecrypt](https://github.com/ovh/symmecrypt), with the 'storage' label.
2. Add it to your configuration items. The library will take all keys into account and use the latest possible key, falling back to older keys when finding older data.
3. Set your API in maintenance mode (env var or command line arg, see config below): all write actions will be refused when you reboot the API.
4. Reboot API.
5. Make a POST request on the /key-rotate endpoint of the API.
6. All data will be encrypted with the latest key, you can delete older keys.
7. De-activate maintenance mode.
8. Reboot API.
### Dependencies
The only dependency for µTask is a Postgres database server. The minimum version for the Postgres database is 9.5
## Configuration 🔨
### Command line args
The µTask binary accepts the following arguments as binary args or env var.
All are optional and have a default value:
- `init-path`: the directory from where initialization plugins (see "Developing plugins") are loaded in *.so form (default: `./init`)
- `plugins-path`: the directory from where action plugins (see "Developing plugins") are loaded in *.so form (default: `./plugins`)
- `templates-path`: the directories where yaml-formatted task templates are loaded from, can be a colon separated list (default: `./templates`)
- `functions-path`: the directory where yaml-formatted functions templates are loaded from (default: `./functions`)
- `region`: an arbitrary identifier, to aggregate a running group of µTask instances (commonly containers), and differentiate them from another group, in a separate region (default: `default`)
- `http-port`: the port on which the HTTP API listents (default: `8081`)
- `debug`: a boolean flag to activate verbose logs (default: `false`)
- `maintenance-mode`: a boolean to switch API to maintenance mode (default: `false`)
### Config keys and files
Checkout the [µTask config keys and files README](./config/README.md).
### Authentication
The vanilla version of µTask doesn't handle authentication by itself, it is meant to be placed behind a reverse proxy that provides a username through the "x-remote-user" http header. A username found there will be trusted as is, and used for authorization purposes (admin actions, task resolution, etc...).
For development purposes, an optional `basic-auth` configstore item can be provided to define a mapping of usernames and passwords. This is not meant for use in production.
Extending this basic authentication mechanism is possible by developing an "init" plugin, as described [below](#plugins).
### Notification
Every task state change can be notified to a notification backend.
µTask implements three differents notification backends: Slack, Opsgenie, and generic webhooks.
Default payload that will be sent for generic webhooks are:
__task_state_update notifications:__
```json
{
"message": "string",
"notification_type": "task_state_update",
"task_id": "public_task_uuid",
"title": "task title string",
"state": "current task state",
"template": "template_name",
"requester": "optional",
"resolver": "optional",
"steps": "14/20",
"potential_resolvers": "user1,user2,admin",
"resolution_id": "optional,public_resolution_uuid",
"tags": "{\"tag1\":\"value1\"}"
}
```
__task_step_update notifications:__
```json
{
"message": "string",
"notification_type": "task_step_update",
"task_id": "public_task_uuid",
"title": "task title string",
"state": "current task state",
"template": "template_name",
"step_name": "string",
"step_state": "string",
"requester": "string",
"resolver": "string",
"steps": "14/20",
"resolution_id": "public_resolution_uuid",
"tags": "{\"tag1\":\"value1\"}"
}
```
__task_validation notifications:__
```json
{
"message": "string",
"notification_type": "task_validation",
"task_id": "public_task_uuid",
"title": "task title string",
"state": "TODO",
"template": "template_name",
"requester": "optional",
"potential_resolvers": "user1,user2,admin",
"tags": "{\"tag1\":\"value1\"}"
}
```
Notification backends can be configured in the global µTask configuration, as described [here](./config/README.md#utask-cfg).
## Authoring Task Templates
Checkout the [µTask examples directory](./examples).
A process that can be executed by µTask is modelled as a `task template`: it is written in yaml format and describes a sequence of steps, their interdepencies, and additional conditions and constraints to control the flow of execution.
The user that creates a task is called `requester`, and the user that executes it is called `resolver`. Both can be the same user in some scenarios.
A user can be allowed to resolve a task in four ways:
- the user is included in the global configuration's list of `admin_usernames`
- the user is included in the task's template list of `allowed_resolver_usernames`
- the user is in a group that is included in the task's template list of `allowed_resolver_groups`
- the user is included in the task `resolver_usernames` list
### Value Templating
µTask uses the go [templating engine](https://golang.org/pkg/text/template/) in order to introduce dynamic values during a task's execution. As you'll see in the example template below, template handles can be used to access values from different sources. Here's a summary of how you can access values through template handles:
- `.input.[INPUT_NAME]`: the value of an input provided by the task's requester
- `.resolver_input.[INPUT_NAME]`: the value of an input provided by the task's resolver
- `.step.[STEP_NAME].output.foo`: field `foo` from the output of a named step
- `.step.[STEP_NAME].metadata.HTTPStatus`: field `HTTPStatus` from the metadata of a named step
- `.step.[STEP_NAME].children`: the collection of results from a 'foreach' step
- `.step.[STEP_NAME].error`: error message from a failed step
- `.step.[STEP_NAME].state`: current state of the given step
- `.step.[STEP_NAME].max_retries`: max retries of the given step
- `.step.[STEP_NAME].try_count`: try count of the given step
- `.config.[CONFIG_ITEM].bar`: field `bar` from a config item (configstore, see above)
- `.iterator.foo`: field `foo` from the iterator in a loop (see `foreach` steps below)
- `.pre_hook.output.foo`: field `foo` from the output of the step's pre-hook (see [pre-hooks](#pre-hooks))
- `.pre_hook.metadata.HTTPStatus`: field `HTTPStatus` from the metadata of the step's pre-hook (see [pre-hooks](#pre-hooks))
- `.function_args.[ARG_NAME]`: argument that needs to be given in the conifguration section to the function (see `functions` below)
The following templating functions are available:
| Name | Description | Reference |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| **`Golang`** | Builtin functions from Golang text template | [Doc](https://golang.org/pkg/text/template/#hdr-Actions) |
| **`Sprig`** | Extended set of functions from the Sprig project | [Doc](https://masterminds.github.io/sprig/) |
| **`field`** | Equivalent to the dot notation, for entries with forbidden characters | ``{{field `config` `foo.bar`}}`` |
| **`fieldFrom`** | Equivalent to the dot notation, for entries with forbidden characters. It takes the previous template expression as source for the templating values. Example: ```{{ `{"foo.foo":"bar"}` | fromJson | fieldFrom `foo.foo` }}``` | ```{{expr | fieldFrom `config` `foo.bar`}}``` |
| **`eval`** | Evaluates the value of a template variable | ``{{eval `var1`}}`` |
| **`evalCache`** | Evaluates the value of a template variable, and cache for future usage (to avoid further computation) | ``{{evalCache `var1`}}`` |
| **`fromJson`** | Decodes a JSON document into a structure. If the input cannot be decoded as JSON, the function will return an empty string | ``{{fromJson `{"a":"b"}`}}`` |
| **`mustFromJson`** | Similar to **`fromJson`**, but will return an error in case the JSON is invalid. A common usecase consists of returning a JSON stringified data structure from a JavaScript expression (object, array), and use one of its members in the template. Example: ``{{(eval `myExpression` \| fromJson).myArr}}`` or ``{{(eval `myExpression` \| fromJson).myObj}}`` | ``{{mustFromJson `{"a":"b"}`}}`` |
| **`b64RawEnc`** | Encode a string to a b64 raw encoded string as defined in [RFC 4648 section 3.2](https://www.rfc-editor.org/rfc/rfc4648.html#section-3.2). Example: ``{{eval `myString` \| b64RawEnc}}`` | ``{{b64RawEnc `a nice string`}}`` |
| **`b64RawDec`** | Decode a b64 raw encoded string as defined in [RFC 4648 section 3.2](https://www.rfc-editor.org/rfc/rfc4648.html#section-3.2) to a decoded string. Example: ``{{eval `cmF3IG1lc3NhZ2U` \| b64RawDec}}`` | ``{{b64RawDec cmF3IG1lc3NhZ2U`}}`` |
### Basic properties
- `name`: a short unique human-readable identifier
- `description`: sentence-long description of intent
- `long_description`: paragraph-long basic documentation
- `doc_link`: URL for external documentation about the task
- `title_format`: templateable text, generates a title for a task based on this template
- `result_format`: templateable map, used to generate a final result object from data collected during execution
### Advanced properties
- `allowed_resolver_groups`: a list of groups with the right to resolve a task based on this template
- `allowed_resolver_usernames`: a list of usernames with the right to resolve a task based on this template
- `allow_all_resolver_usernames`: boolean (default: false): when true, any user can execute a task based on this template
- `auto_runnable`; boolean (default: false): when true, the task will be executed directly after being created, IF the requester is an accepted resolver or `allow_all_resolver_usernames` is true
- `blocked`: boolean (default: false): no tasks can be created from this template
- `hidden`: boolean (default: false): the template is not listed on the API, it is concealed to regular users
- `retry_max`: int (default: 100): maximum amount of consecutive executions of a task based on this template, before being blocked for manual review
- `tags`: templatable map, used to filter tasks (see [tags](#tags))
### Inputs
When creating a new task, a requester needs to provide parameters described as a list of objects under the `inputs` property of a template. Additional parameters can be requested from a task's resolver user: those are represented under the `resolver_inputs` property of a template.
An input's definition allows to define validation constraints on the values provided for that input. See example template above.
#### Input properties
- `name`: unique name, used to access the value provided by the task's requester
- `description`: human readable description of the input, meant to give context to the task's requester
- `regex`: (optional) a regular expression that the provided value must match
- `legal_values`: (optional) a list of possible values accepted for this input
- `collection`: boolean (default: false) a list of values is accepted, instead of a single value
- `type`: (string|number|bool) (default: string) the type of data accepted
- `optional`: boolean (default: false) the input can be left empty
- `default`: (optional) a value assigned to the input if left empty
### Variables
A template variable is a named holder of either:
- a fixed value
- a JavaScript expression evaluated on the fly.
See the example template above to see variables in action. The expression in a variable can contain template handles to introduce values dynamically (from executed steps, for instance), like a step's configuration.
The JavaScript evaluation is done using [otto](https://github.com/robertkrimen/otto).
### Tags
Tags are a map of strings property of a task. They will be used in the task listing to search for some tasks using filters. With tags, uTask can be used as a task backend by others APIs.
Tags values are expected to be a `string`: it support all uTask templating on values. To remove a tag from a task, use the empty value `""`.
```yaml
tags:
customer: "{{.input.customer_id}}"
type: "billing"
```
In this example, tag `customer` will be templated from the task inputs, and allow others APIs to search all the tasks for a given customer.
Tags can be added to a task:
- from the template definition of the task
- while creating a task, requester can input custom tags
- during the execution, using the [`tag` builtin plugin](./pkg/plugins/builtin/tag/README.md)
### Steps
A step is the smallest unit of work that can be performed within a task. At is's heart, a step defines an **action**: several types of actions are available, and each type requires a different configuration, provided as part of the step definition. The state of a step will change during a task's resolution process, and determine which steps become eligible for execution. Custom states can be defined for a step, to fine-tune execution flow (see below).
A sequence of ordered steps constitutes the entire workload of a task. Steps are ordered by declaring **dependencies** between each other. A step declares its dependencies as a list of step names on which it waits, meaning that a step's execution will be on hold until its dependencies have been resolved. [More details about dependencies](#dependencies).
The flow of this sequence can further be controlled with **conditions** on the steps: a condition is a clause that can be run before or after the step's action. A condition can either be used:
- to skip a step altogether
- to analyze its outcome and override the engine's default behaviour
Several conditions can be specified.
Unless `final` is set to `true`, they are all evaluated in order. If multiple conditions evaluate to `true`, they will be applied sequentially. Once a condition is applied, the next condition is evaluated using the new context (i.e. using the new `state` value of steps that got updated). If multiple conditions are evaluated to `true` and are changing the same step `state` value, then the last condition to evaluate as `true` will be the one that will change the `state` step _for real_.
A condition is composed of:
- a `type` (skip or check)
- a list of `if` assertions (`value`, `operator`, `expected`) which all have to be true (AND on the collection),
- a `then` object to impact the state of steps (`this` refers to the current step)
- a `final` boolean, defaulting to `false`. When set to `true`, it prevents the evaluation of the next conditions if this one is evaluated to `true`
- an optional `message` to convey the intention of the condition, making it easier to inspect tasks
Here's an example of a `skip` condition. The value of an input is evaluated to determine the result: if the value of `runType` is `dry`, the `createUser` step will not be executed, its state will be set directly to `DONE`.
```yaml
inputs:
- name: runType
description: Run this task with/without side effects
legal_values: [dry, wet]
steps:
createUser:
description: Create new user
action:
... etc...
conditions:
- type: skip
if:
- value: '{{.input.runType}}'
operator: EQ
expected: dry
then:
this: DONE
message: Dry run, skip user creation
```
Here's an example of a `check` condition. Here the return of an http call is inspected: a 404 status will put the step in a custom `NOT_FOUND` state. The default behavior would be to consider any 4xx status as a client error, which blocks execution of the task. The check condition allows you to consider this situation as normal, and proceed with other steps that take the `NOT_FOUND` state into account (creating the missing resource, for instance).
```yaml
steps:
getUser:
description: Get user
custom_states: [NOT_FOUND]
action:
type: http
configuration:
url: http://example.org/user/{{.input.id}}
method: GET
conditions:
- type: check
if:
- value: '{{.step.getUser.metadata.HTTPStatus}}'
operator: EQ
expected: '404'
then:
this: NOT_FOUND
message: User {{.input.id}} not found
createUser:
description: Create the user
dependencies: ["getUser:NOT_FOUND"]
action:
type: http
configuration:
url: http://example.org/user
method: POST
body: |-
{"user_id":"{{.input.id}}"}
```
#### Condition Operators
A condition can use one of the following operators:
- `EQ`: equal
- `NE`: not equal
- `GT`: greater than
- `LT`: less than
- `GE`: greater or equal
- `LE`: less than or equal
- `REGEXP`: match a regexp
- `NOTREGEXP`: doesn't match a regexp
- `IN`: found in a list of values
- `NOTIN`: not found in a list of values
Note that the operators `IN` and `NOTIN` expect a list of acceptable values in the field `value`, instead of a single one. You can specify the separator character to use to split the values of the list using the field `list_separator` (default: `,`). Each value of the list will be trimmed of its leading and trailing white spaces before comparison.
#### Basic Step Properties
- `name`: a unique identifier
- `description`: a human readable sentence to convey the step's intent
- `action`: the actual task the step executes, see [Action](#step-action)
- `foreach`: see [Loops](#step-foreach)
- `pre_hook`: an action that can be executed before the actual action of the step
- `dependencies`: a list of step names on which this step waits before running
- `idempotent`: a boolean indicating if this step is safe to be replayed in case of uTask instance crash
- `json_schema`: a JSON-Schema object to validate the step output
- `resources`: a list of resources that will be used by this step to apply some rate-limiting (see [resources](#resources))
- `custom_states`: a list of personnalised allowed state for this step (can be assigned to the state's step using `conditions`)
- `retry_pattern`: (`seconds`, `minutes`, `hours`) define on what temporal order of magnitude the re-runs of this step should be spread (default = `seconds`)
- `resources`: a list of resources that will be used during the step execution, to control and limit the concurrent execution of the step (more information in [the resources section](#resources)).