# Changelog All notable changes to Ciaren will be documented in this file. Ciaren follows the spirit of [Keep a Changelog](https://keepachangelog.com/) and uses semantic versioning. Until the first stable release (`1.0.0`), breaking changes may still happen between `0.x` releases. ## [Unreleased] ## [0.3.1] - 2026-10-12 This release improves Python export parity and the documentation for both engines. Existing flows keep their configuration; corrected exports can change results that previously differed from a Ciaren run. ### Added - **Semi and anti joins** select matching or nonmatching left-hand rows without adding right-hand columns (#198). - **Cumulative product** is available in Window Function (#208). - **Runnable pandas and Polars references** cover input/output formats, SQL, row and column transformations, combining data, windows and ML workflows. The ML guides explain training, model persistence, prediction and evaluation. - **Text-cleaning recipes and troubleshooting** show how to split irregular fields, choose fallback values and diagnose missing generated columns. ### Fixed - **pandas Split Column exports** preserve missing split parts and requested output columns when rows have fewer delimiters (#231). - **Polars exports** match runtime duplicate counts for Assert Unique (#235), preserve aggregation group order (#236), and handle optional/fractional Assert Range bounds without truncating them (#238). - **ML exports** preserve model definitions and preprocessing (#239), explain the model-definition requirement in Cross Validation errors (#240), and honor prediction batch sizes, including nullable features (#241). - **Text input exports** preserve UTF-8, blank lines and boundary whitespace. - **Icon-only editor controls** have accessible labels (#230). - **Documentation checks on Windows** invoke npm tooling correctly (#232). - **Python dependency security floors** apply to pip installs as well as the development lockfile, excluding affected multidict, urllib3, Werkzeug and Pillow releases from the existing MLflow dependency stack. ### Changed - **Release validation** runs the complete backend inventory in disposable module processes, combines coverage and blocks PyPI publication until the full OS/Python matrix and both package builds pass. - **Frontend tooling** uses Vite 8 with verified vendor chunks; installed packages continue to bundle the editor. Tailwind 4 removes an unpatched dependency chain while retaining the purple theme and keyboard focus styles. - **Documentation dependencies** use patched Vite and source-map packages. The docs audit now blocks high-severity advisories, matching the editor. - **Documentation navigation and metadata** use specific descriptions, current capability notes and examples that distinguish runtime safeguards from ordinary exported Python. The roadmap describes alpha limitations. - **Codespaces opens the editor directly.** The dev container builds from a slim image with the expired Yarn apt source removed (the old setup fell back to a recovery container without Ciaren), installs the released package from PyPI onto the `/workspaces` volume, starts Ciaren in the terminal when the editor attaches, and opens it in a new browser tab. The README puts the "Try it in your browser" button right under the tagline. ## [0.3.0] - 2026-09-25 Notebook export, run drift, CSV dialect detection for storage inputs, and a way to try Ciaren in the browser with GitHub Codespaces. There are no breaking changes. The polars `pivot` count fix changes results where the values column has nulls, so the engines now agree. ### Added - **Try Ciaren in GitHub Codespaces.** The repository ships a dev container, and the README's "Open in GitHub Codespaces" badge opens the editor with the Demo project in your browser, with nothing to install. The server trusts exactly the codespace's forwarded URL through `CIAREN_CORS_ORIGINS`, so the cross-site request guard stays on. The installation guide covers it. - **`ciaren serve` asks for a GitHub star.** One static line after the app URL. It makes no network call and records nothing. - **Jupyter notebook export.** The code export dialog can download each engine variant (pandas, polars, lazy polars) as a `.ipynb` notebook, and `POST /api/flows/{flow_id}/export/python?include_notebooks=true` returns them in the new `notebook`, `notebook_polars`, and `notebook_polars_lazy` fields (`null` unless requested). Cells split only between top-level statements, so every cell runs on its own. The notebook exporters are listed in `GET /api/catalog/exporters`, and the Python client's `export_flow_python` takes `include_notebooks`. Thanks to @tusharui. - **Run drift.** The run detail page shows a "Since last run" panel: per-node row-count change and added or removed columns compared with the previous run of the same flow, plus nodes added or removed when the graph changed (#142). Thanks to @tusharui. - **CSV dialect detection for storage inputs.** CSV files read from a local folder, S3, GCS, or Azure Blob connection now get the same delimiter, encoding, and decimal detection as dataset uploads. The node panel shows the detected values, and a delimiter, encoding, or decimal set in the node config still wins (#200). - **Recently used nodes.** The node palette shows the last five node types you placed when the search box is empty (#191). - **`lstrip` and `rstrip`** operations on the String transform node (#188). Thanks to @rashmeetchhabra12. - **Validator example plugin** in `examples/plugins/validator-plugin/`, a data-quality node between the Hello and MLP Classifier examples (#120). Thanks to @tusharui. ### Fixed - **`pivot` with `aggfunc="count"`** now counts non-null values on polars, so the pandas and polars engines and their exported code agree when the values column has nulls (#143). Thanks to @tusharui. - Transformation validation messages follow one documented format (#119). Thanks to @tusharui. - An S3 error without a response object returned HTTP 500 instead of the scrubbed connector error (#200). ### Changed - Contributions no longer need a DCO `Signed-off-by` line. They are licensed under the repository licenses through the GitHub Terms of Service (#202). - The PyPI project links and the app's footer and landing links point at `/docs/latest` instead of the redirecting `/docs` URL. The docs home drops its hero glow and card hover effects to match ciaren.com (#203). - The test suite builds, signs, installs, and runs every example plugin from source (#190). - Pull-request checks also run against release-preparation branches, and routine Dependabot updates for dependencies and GitHub Actions are folded in (#137). ### Security - Patched vulnerable dependencies: aiohttp, anyio, cryptography, gitpython, mlflow, pyasn1, and sqlparse in the backend lock file, and `npm audit fix` for the frontend and docs (#189). ### Documentation - Every docs page has a specific search description; README and client links point at the live `/docs/latest/` URLs; the docs build now fails on dead internal links (#193). - One start path from installation to the first flow, one sidebar home per page, and clearer roles for the plugin tutorial, guide, and reference (#199). - Source installs need Node.js 20 or newer (vitest 4). - The README, PyPI description, keywords, and project URLs are clearer, and the docs have a social card for link previews (#177). - The comparison page compares Ciaren with KNIME, Alteryx, and Flowfile by name (#204). - The docs state what plugins can run today: nodes, connectors, and ML model types work end to end, while engine, exporter, and validator providers only register metadata so far. The ML classification example now imports pandas before using it. - The docs site has a Changelog page with these release notes, checked in and kept identical to `CHANGELOG.md` by a test. ## [0.2.0] - 2026-07-20 A repo-wide correctness and hardening pass from an internal audit. Most of it is straightforward bug fixing, but a few changes alter results or reject input that previously passed — read **Breaking changes** before upgrading. ### Breaking changes - **`groupByAggregate` rejects aggregations with no exact polars equivalent** (`sem`, `skew`, `kurt`, `size`, `mad`). These previously ran on the pandas backend and failed or silently differed elsewhere. *Migration:* switch to a supported aggregation, or compute the statistic downstream in a `pythonTransform` node. - **polars `first` / `last` / `nunique` now skip nulls, matching pandas.** Groups containing nulls produce different (now correct and engine-consistent) values. *Migration:* re-run affected flows; polars results now agree with the pandas backend and with the exported code. - **`concatRows` on polars unions mismatched columns** and null-fills, instead of raising. Flows that previously errored now succeed. *Migration:* none. - **`GET /api/datasets` and `GET /api/flows` return at most 500 rows** by default. *Migration:* none for the UI; API consumers that relied on an unbounded list should expect truncation until `limit`/`offset` query parameters land. - **A Train node whose model fails to save now fails the run** instead of reporting success with an unusable model reference. *Migration:* none — the run was already broken downstream; the failure is now visible. - **Run creation rejects an unknown or cross-project `input_dataset_id`** (404 / 400). *Migration:* pass a dataset that exists in the flow's project, or omit the field. - **"Default" is a reserved project name** — creating or renaming a project to it returns 400. *Migration:* pick another name. - **Plugins are re-verified at load.** An installed plugin whose files were modified on disk after installation is refused (*migration:* reinstall it to re-pin the baseline); an install whose id case-collides with an existing plugin is rejected; a plugin whose metadata id differs from its manifest id is refused; and a node's declared `provider` is forced to the owning plugin's id. - **Hardened connector mode** (`CIAREN_CONNECTOR_BLOCK_PRIVATE_HOSTS=true`) now rejects multi-host / URI-style hostnames (e.g. a comma-separated libpq seed list) and refuses `verify_tls: false`. *Migration:* use a single plain hostname per connection and keep TLS verification on, or leave the guard off. - **`pythonTransform` strict mode** (`CIAREN_PYTHON_TRANSFORM_STRICT=true`) blocks additional capability modules (`io`, `tempfile`, `socket`, `pickle`, …). Imports that are allowed now genuinely work at runtime, where they previously failed with an opaque error. ### Security - Connector SSRF guard now fails closed on multi-host / URI-style hostnames (which drivers could expand into a seed list or full URI) and blocks the NAT64 range that maps to the cloud metadata endpoint. - The REST connector re-validates every HTTP redirect hop against the SSRF guard, rejects non-`http(s)` redirect targets, refuses to disable TLS verification in hardened mode, and no longer forwards `Authorization` / API-key headers when a redirect crosses to a different host. - Marketplace artifact resolution is confined for untrusted (URL-shaped) index sources — `file://`, absolute, and `..` paths are rejected — ahead of any hosted plugin index. - Plugins: code integrity is now verified at load (previously only the manifest was hashed), a planted or altered bytecode cache can no longer execute in place of trusted code, a node's `provider` is bound to its owning plugin, and an install whose id case-collides with an existing plugin is rejected. - SQLite foreign-key enforcement is enabled; the `/api/settings/webhook` auth exemption is scoped to `GET`; `pythonTransform` strict mode blocks additional capability modules and frame/traceback traversal. - CI: all GitHub Actions are pinned to commit SHAs (including the OIDC PyPI publish jobs), and `API_TOKEN` now documents the need for a long, high-entropy value. ### Fixed - Projects: creating or renaming a project to the reserved name "Default", or deleting a project that holds a dataset whose name also exists in the default project, no longer returns a 500. - A `PUT /api/flows` with an explicit `null` project id now moves the flow to the default project instead of erroring; an impossible-but-valid cron expression (e.g. Feb 30) now returns a validation error instead of a 500. - Scheduler: daily schedules no longer fire twice during the DST fall-back hour; a concurrent user edit is no longer overwritten by a just-finished run; a manual "Run now" can no longer start a second concurrent run of a flow the scheduler is already running. - Engine: `groupByAggregate` and `concatRows` now produce identical results on the pandas backend, the polars backend, and the exported code (previously a config valid on one could fail or differ on another). - ML: a Train node whose model fails to save now fails clearly instead of reporting success with an unusable model; prediction aligns feature columns and warns before overwriting an existing column; training hyperparameters are bounded to prevent a runaway fit. - Frontend: file downloads work when an `API_TOKEN` is configured; the flow editor warns before discarding unsaved changes; storage-restricted browser contexts no longer crash the editor; the "not authorized" message points to the real `?api_token=` mechanism. - Database: networked Postgres/MySQL engines use connection health checks (`pool_pre_ping`) so a stale pooled connection no longer surfaces as a 500, and startup no longer runs best-effort column patching on Alembic-managed databases. - Listing datasets no longer loads every version of every dataset into memory — the latest version and version count come from an aggregate query, so a workspace with a long run history stays responsive. ## [0.1.0] - 2026-07-14 First public release of Ciaren — a local-first visual builder for data and ML workflows that exports clean, readable pandas/polars Python. This is alpha software: APIs, data models, and generated code may still change before a stable `1.0`. ### Visual builder - 80 nodes across 9 categories — inputs, cleaning, columns, reshape, analytics, data quality, charts, machine learning, and outputs — including 8 chart nodes (bar, line, area, scatter, pie, histogram, box plot, correlation heatmap) that render instantly from a compact run artifact and support one-click PNG export. - File, SQL, and cloud-storage I/O (S3, GCS, Azure Blob, Snowflake) plus a built-in, read-only REST API connector (auth, custom headers, pagination, SSRF-guarded). - Per-node data preview, undo/redo, duplicate flows, and copy/paste nodes with their connecting edges. - Import options with auto-detection — CSV/TSV dialect, encoding, and multi-sheet Excel workbooks are detected on upload, with an explicit override always available. ### Execution and code export - Dual-engine execution — **polars** (default) or **pandas**, selectable per run. - Exported Python reads like hand-written code (keyword `assign()`, chainable `.loc[lambda _d: …]` filters, fused method chains, reused dataframe variables on straight-line flows) and is verified equivalent to the in-app run by an automated pandas/polars, eager/lazy equivalence harness. - Cancel a running run from the UI or API; optional failure webhook notifications when a run fails or a schedule auto-disables. ### Data quality and Machine Learning - Data contract nodes: assert not-null, unique, value range, row count, and arbitrary expressions. - Built-in ML: split, feature engineering, train, predict, and evaluate nodes, with MLflow tracking and a Models page. `pip install ciaren` alone gives a working ML palette (scikit-learn, MLflow, and joblib are core dependencies); XGBoost and LightGBM are available via the `ml` extra. ### Scheduling and automation - Built-in cron scheduler with retries, catch-up, and auto-disable after repeated failures. - REST, CLI, and webhook-based ways to trigger flows outside the UI. ### Plugin platform - A versioned, contract-first Plugin API/SDK (`backend/app/plugin_api/`, Apache-2.0) — plugins can contribute transformation nodes, ML model types, connectors, and schema-driven config forms without forking the core. - Signed `.ciarenplugin` packaging, local and entry-point discovery, and permission-gated model loading. - `ciaren-client` — a Python SDK for scripting against the server (flows, runs, datasets, plugins). - Two documented example plugins (hello-world node, MLP classifier) covering both extension paths. ### Documentation and project - Full documentation site: getting started, guides, transformation reference, recipes, plugin authoring, and API reference. - Contributing guide, Developer Certificate of Origin (DCO) policy, issue and PR templates, and public GitHub Discussions. - Dual licensing: Ciaren Core is AGPL-3.0-only; the public Plugin API/SDK is Apache-2.0 so plugin authors can choose their own license. - Docker and Docker Compose support for single-container local/self-hosted use. ### Security - pip-audit and npm audit run on every push to `main` and fail CI on new high/critical advisories in pinned dependencies. ### Known limitations - Ciaren is alpha software; recommended for controlled internal workflows while the project matures, not yet for critical production jobs. - Not a distributed or streaming data engine — batch-style local and self-hosted workflows are the product center. - Has not yet completed a formal independent third-party security audit.