The open-source data platform for building, running, and explaining data pipelines.
Website · Documentation · Quick Start · Features · Examples
--- It starts as a single ~30MB binary with a visual editor, parallel DAG execution, built-in data quality, real-time monitoring, and a Python SDK — no infrastructure required beyond the binary itself. The important part is the complete operating loop: define a pipeline, run it, see the physical plan and evidence behind the result, and trace the data that was affected. Start locally, then keep the same pipeline when you need more workers or stronger governance. --- ## Quick Start ### Option A: Install Script ```bash curl -fsSL https://raw.githubusercontent.com/Tnsor-Labs/brokoli/main/install.sh | sh brokoli serve ``` Open `http://localhost:8080`, create your admin account, choose the **Hello World** template, and click **Run**. You should have a successful run, an output CSV, and an execution record in a few minutes. For the complete first-success path, see the [five-minute getting started guide](https://docs-brokoli.orkestri.site/docs/getting-started/get-started). ### Option B: Download Binary ```bash # Linux x86_64 curl -L https://github.com/Tnsor-Labs/brokoli/releases/latest/download/brokoli_Linux_x86_64.tar.gz | tar xz ./brokoli serve ``` ### Option C: Python SDK ```bash pip install brokoli ``` ```python from brokoli import Pipeline, task, source_api, quality_check, sink_file with Pipeline("my_pipeline", schedule="0 6 * * *") as p: data = source_api("Fetch", url="https://api.example.com/data", retries=3) @task("Transform") def clean(raw): return [r for r in raw if r.get("status") == "active"] cleaned = clean(data) quality_check("Validate", cleaned, rules=["not_null(id)", "unique(id)"]) cleaned >> sink_file("Save", path="/tmp/output.csv") ``` ```bash brokoli deploy my_pipeline.py --server http://localhost:8080 ``` TypeScript code nodes are available when **Node.js 20 or newer** is installed on the machine running the Brokoli server. Node is optional: Python nodes and non-code pipelines work without it, and Brokoli never downloads or bundles a runtime. The server resolves the existing `node` executable at startup; when it finds none, `/api/capabilities` stops advertising `typescript`, and the SDKs refuse to deploy a TypeScript node against that host rather than letting it fail at run time. See the [TypeScript example](examples/hello-world-typescript.mjs) for a dependency-free end-to-end run. The smallest working example is also in [`examples/hello-world.py`](examples/hello-world.py). ### From Source ```bash bash build-ui.sh # builds the React UI onto web/dist go build -tags minicore_disabled -o brokoli . && ./brokoli serve ``` `build-ui.sh` is the step that places the built UI on the embed target. Building in `ui/` alone leaves it in `ui/apps/community/dist`, and the binary then starts with no UI and says so on its first page. --- ## Features **Single binary, no infrastructure.** Runs on any Linux/macOS machine with no external dependencies. SQLite by default; swap to PostgreSQL with a connection string. **Visual pipeline editor.** Drag-and-drop canvas with 15 node types, zoom/pan, inline node config, dry-run preview, and YAML import/export. No code required for common ETL patterns. **Built-in data quality.** 10 quality rules (not_null, unique, min/max, range, regex, row_count, type_check, freshness, no_blank), auto-profiling on every node output, and schema drift detection across runs. **Parallel DAG execution.** Wave-based Kahn's algorithm, per-node retry with exponential backoff, resume from failure, cross-pipeline dependencies with trigger-mode chaining, condition branching. **Explainable operations.** Live run status and log streaming over WebSocket, physical execution plans, run evidence, artifacts, profiling, a run calendar heatmap, and cross-pipeline lineage. **Secrets and connections.** AES-256-GCM encryption at rest, typed database, API, file, and cloud-storage connections, typed variables with `${var.key}` resolution in any config field. **Authentication.** httpOnly cookie sessions, JWT auth, API key support, role-based access (admin/editor/viewer). **Pipeline versioning.** Auto-snapshot on every save, full version history, one-click rollback. **Webhook triggers.** Per-pipeline tokens — integrate with GitHub Actions, dbt, Kafka, or any HTTP source. --- ## Brokoli Cloud Brokoli Cloud builds on the open-source core with team and governance features — SSO/OIDC, audit logging, workspaces with RBAC, Git Sync, data contracts, column-level lineage, PII detection, OpenLineage export, and Kubernetes execution. Free tier available. [Try it now](https://brokoli.orkestri.site) --- ## Configuration | Flag | Default | Description | |---|---|---| | `--port` / `-p` | 8080 | HTTP server port | | `--db` | ./brokoli.db | SQLite path or `postgres://...` URI | | `--api-key` | — | Enable API key authentication | | Environment variable | Description | |---|---| | `BROKOLI_APP_URL` | Base URL for Slack deep links | | `BROKOLI_JWT_SECRET` | Persistent JWT signing secret | | `BROKOLI_CORS_ORIGINS` | Allowed CORS origins (comma-separated) | | `BROKOLI_MAX_CONCURRENT_RUNS` | Max parallel pipeline runs (default: 4) | --- ## License Apache 2.0 — see [LICENSE](LICENSE). Enterprise features are available under a separate commercial license. [Contact us](https://brokoli.orkestri.site) for pricing.