# Graphic Walker Workflow Computation Specification

This document is the normative specification for Graphic Walker view-data computation. An implementation is conforming when it evaluates every valid `IDataQueryPayload` according to the semantics below, independent of whether the backend is the client-side JavaScript executor or SQL generated by `@kanaries/gw-dsl-parser`.

The workflow DSL is an execution IR, not a user authoring format. UI specs and future shorthand formats compile into this IR before computation starts.

## 1. Request Model

```ts
export interface IDataQueryPayload {
    workflow: IDataQueryWorkflowStep[];
    tag?: string;
    limit?: number;
    offset?: number;
}
```

A payload is evaluated against an ordered array of rows. Each row is a record keyed by field id (`fid`). Workflow steps are evaluated strictly in array order. The output of each step becomes the input to the next step.

`offset` and `limit` are applied after all workflow steps, including `sort`. `offset` defaults to `0`. A missing `limit`, `undefined`, `0`, or a negative UI sentinel means no upper bound. When `limit > 0`, the final slice is `[offset, offset + limit)`. Historical client builds used JavaScript truthiness for `limit`, which made `limit: -1` drop the final row; conforming implementations must use the `limit > 0` rule instead.

Workflow steps may include multiple `view` steps. They cascade exactly like any other step: the rows produced by the first `view` step are the rows consumed by the next step.

## 2. Value And Type Semantics

Missing fields evaluate to `undefined`. `null` and `undefined` are nullish values. Unless a rule below says otherwise, nullish input propagates to a nullish or failed result rather than being coerced to a useful value.

Numeric operators coerce with JavaScript `Number(value)` semantics for strings and booleans when the value is explicitly converted by the workflow builder. Direct executor comparisons use the stored value; relational comparisons with nullish values fail. SQL backends must match these observable results, including numeric strings that participate in numeric arithmetic.

Boolean values used in numeric contexts coerce to `1` for `true` and `0` for `false`. Date strings are parsed by the offset-date model in section 4 when used by temporal operators. ISO strings with explicit timezone offsets are absolute instants. Date-only strings (`YYYY`, `YYYY-MM`, `YYYY-MM-DD`) are interpreted in the supplied parse offset, not as host-local dates.

Floating point comparisons in conformance tests use an absolute tolerance of `1e-9`. This tolerance covers IEEE-754 arithmetic and DuckDB decimal/float conversion differences; it is not large enough to hide different bin boundaries, different timestamps, or different aggregate formulas.

## 3. Filter Step

```ts
export interface IFilterWorkflowStep {
    type: 'filter';
    filters: IVisFilter[];
}
```

All filters in a filter step are combined with logical AND. Input row order is preserved for rows that pass.

### `range`

`{ type: 'range', value: [min, max] }` passes a row when:

```ts
(min ?? -Infinity) <= row[fid] && row[fid] <= (max ?? Infinity);
```

Bounds are inclusive. A nullish row value fails the predicate. A null bound is open on that side.

### `temporal range`

`{ type: 'temporal range', value: [min, max], offset?, format? }` parses the row value with `newOffsetDate(offset)`, compares the resulting epoch milliseconds to inclusive bounds, and passes when the timestamp is inside the range. A nullish or unparseable row value fails. `format` is metadata and does not change runtime parsing.

### `one of`

`{ type: 'one of', value }` passes when the encoded row value is equal to one encoded member of `value`. Encoding is `_unstable_encodeRuleValue`: objects and arrays are compared by their encoded representation, not by reference identity. Nullish values pass only when the set contains the same nullish encoded value.

### `not in`

`{ type: 'not in', value }` passes when the encoded row value is not equal to any encoded member of `value`. Nullish values pass unless the set contains the same nullish encoded value.

### `regexp`

`{ type: 'regexp', value, caseSensitive? }` constructs a JavaScript-compatible regular expression from `value`. It tests the field value with search semantics (`RegExp.prototype.test`), not full-string matching. `caseSensitive: true` uses no flags; otherwise matching is case-insensitive. Invalid patterns compile to a predicate that rejects all rows.

SQL backends that cannot implement JavaScript-compatible regular expressions must reject the payload or mark the case as an unsupported backend capability. They must not silently reinterpret anchors or flags.

## 4. Offset Date Model

Temporal transforms and temporal filters use two offsets:

-   `offset`: the data parse timezone, in minutes, matching `Date#getTimezoneOffset` sign convention.
-   `displayOffset`: the timezone used when extracting or truncating calendar fields.

The effective date constructor is:

```ts
const prepareDate = newOffsetDate(offset);
const toDisplayDate = newOffsetDate(displayOffset);
const date = toDisplayDate(prepareDate(value));
```

For truncation results, the calendar components are computed in display time and then converted back to epoch milliseconds with `toDisplayDate(...)`. `displayOffset` is injected by `toWorkflow()` for `dateTimeDrill` and `dateTimeFeature`. `temporal range` uses `offset` only.

## 5. Transform Step

```ts
export interface ITransformWorkflowStep {
    type: 'transform';
    transform: IFieldTransform[];
}
```

Transforms are evaluated in array order. Each expression appends or replaces the field named by `expression.as` / `key` while preserving all existing columns and row order.

Expression parameters can be fields, constants, literal values, nested expressions, SQL strings, paint maps, `offset`, `displayOffset`, or `format` metadata. Nested expressions are evaluated before their parent expression.

### `bin`

`bin(field, num = 10)` computes `_min` and `_max` over all values of `field`, then `step = (_max - _min) / num`. Each row receives `[lower, upper]`, where `bIndex = floor((value - _min) / step)`, clamped so the maximum value falls into the last bin. Bins are left-closed and right-open except the final bin, which includes `_max`.

If all values are equal or the bin index is `NaN`, the bin index is `0`. Nullish or non-numeric values are not valid inputs.

### `binCount`

`binCount(field, num = 10)` is quantile/rank bucketing. Values are sorted ascending and assigned 1-based bucket numbers by `floor(orderIndex / (rowCount / num)) + 1`, clamped to `num`. Ties keep the JavaScript stable sort order. For an all-equal column the output is still rank buckets, not one equal-width bucket.

### `log`, `log2`, `log10`

`log(field, base = expression.num ?? 10)` computes `Math.log(value) / Math.log(base)`. The normative result for `value <= 0`, nullish, or non-numeric input is `null`. `log2` uses base 2 and `log10` uses base 10.

### `one`

`one()` returns `1` for each input row.

### `dateTimeDrill`

`dateTimeDrill(field, level)` truncates the timestamp to the start of the requested display-time unit and returns epoch milliseconds.

Supported levels:

-   `year`: January 1, 00:00:00.000 of the display year.
-   `quarter`: first day of the calendar quarter.
-   `month`: first day of the month.
-   `week`: Sunday start of week.
-   `day`: day start.
-   `hour`: hour start.
-   `minute`: minute start.
-   `second`: second start.
-   `iso_year`: January 1 of the ISO week-numbering year containing the date.
-   `iso_week`: Monday start of the ISO week.

Nullish input returns `null`.

### `dateTimeFeature`

`dateTimeFeature(field, level)` extracts a display-time calendar component.

Supported levels:

-   `year`: full year.
-   `quarter`: 1 through 4.
-   `month`: 1 through 12.
-   `week`: Sunday-based week number, starting at 1 for the first Sunday-starting week.
-   `weekday`: Sunday = 0 through Saturday = 6.
-   `iso_year`: ISO week-numbering year.
-   `iso_week`: ISO week number.
-   `iso_weekday`: Monday = 1 through Sunday = 7.
-   `day`: day of month.
-   `hour`, `minute`, `second`: clock component.

Nullish input returns `null`.

### `paint`

`paint` maps each row to a paint bucket using either the legacy 2D paint map (`type: 'map'`) or the current facet paint map (`type: 'newmap'`). The result is backend-specific but must be deterministic for the same map and row order. SQL backends may reject `paint` as an unsupported capability.

### `expr`

Transform-level `expr` evaluates the SQL-like expression supplied in the `sql` parameter against each row. If the expression returns an array, it must have exactly one value per input row. If it returns a scalar, the scalar is broadcast to every row. The expression language is the shared expression subset accepted by `pgsql-ast-parser` in the client and the SQL backend. Backends must reject expressions outside their supported subset.

## 6. View Step

```ts
export interface IViewWorkflowStep {
    type: 'view';
    query: IViewQuery[];
}
```

Queries inside a `view` step are evaluated in array order.

### `aggregate`

Rows are grouped by the tuple of `groupBy` field values. Null group key values form ordinary groups; all nullish values for the same group field compare equal for grouping.

For each measure, the output key is `asFieldKey` when present, otherwise `getMeaAggKey(field, agg)`.

Aggregators:

-   `sum`: arithmetic sum of all values.
-   `count`: count of rows in the group, including rows where the measured field is nullish.
-   `max`: maximum by numeric comparison.
-   `min`: minimum by numeric comparison.
-   `mean`: `sum(values) / rowCount`. The current phase-1 semantics keeps nullish rows in the value array; `null` participates through JavaScript numeric coercion, while `undefined` can produce `NaN`.
-   `median`: middle value after ascending numeric sort, or the average of the two middle values. Nullish rows are retained under the same comparator; this means `null` sorts as a numeric zero in the current client implementation.
-   `variance`: population variance, `mean((x - mean(values)) ** 2)`.
-   `stdev`: square root of population variance.
-   `distinctCount`: count of distinct values using SameValueZero semantics; `null` counts as one distinct value and `undefined` counts as one distinct value.
-   `expr`: evaluate the SQL-like aggregate expression over the group. It must return a scalar; returning an array is an error.

Aggregators other than `count`, `distinctCount`, and `expr` require numeric inputs. The current normative null behavior follows the client executor for compatibility: nullish rows are not excluded before aggregation unless an operator explicitly says so. SQL-style null exclusion would be a future breaking semantic revision; current backend differences are recorded in `docs/computation-divergences.md`.

### `fold`

For each input row and each field in `foldBy`, emit one output row. The output row copies the input row, deletes the folded source field, sets `newFoldKeyCol` to the folded field id, and sets `newFoldValueCol` to the original field value. Output order is input row order, and within each input row the order of `foldBy`.

### view-level `bin`

View-level `bin` has the same bin boundary semantics as transform `bin`, but it appends `newBinCol` directly inside the view query sequence.

### `raw`

`raw` projects fields listed in `fields` and preserves row order. Conforming implementations should not expose unlisted fields in raw output.

## 7. Sort Step

```ts
export interface ISortWorkflowStep {
    type: 'sort';
    sort: 'ascending' | 'descending';
    by: string[];
}
```

Sort is lexicographic over `by`. For each key, numbers compare numerically; other values compare by `String(value).localeCompare(String(other))`. Direction applies to every key. The sort is stable: rows with equal sort keys preserve their input relative order.

Nullish values sort by their string representation under the same comparator. Implementations should avoid database-default null ordering and must make null ordering explicit when compiling to SQL.

## 8. Coverage Notes

The phase-1 conformance suite covers filter range, temporal range, one-of/not-in, regexp, transform bin/binCount/log/one/date drills/date features/expr, non-zero `offset`/`displayOffset` combinations, all aggregate operators, aggregate null handling, group-by nulls, raw, sort with limit/offset, special-character fields, numeric strings, empty data, single-row data, null-heavy data, a `toWorkflow()` generated pipeline, and multi-view cascading.

Known gaps are recorded in `docs/computation-divergences.md`. The main not-yet-covered executable areas are `paint` and full expression-language conformance; both require a backend capability contract beyond the current parser behavior.