# AlchemyLake for Databricks
[](./LICENSE)
[](https://docs.databricks.com/dev-tools/cli/)
[](https://app.alchemylake.com/docs#developers)
Install a **governed creative** surface directly inside your Databricks
workspace. AlchemyLake turns the data your lakehouse already trusts into
finished deliverables **Genie doesn't make**: board-ready **PowerPoint decks
with a read-aloud script and Q&A prep under every slide**, enterprise **PDF
dossiers with a statistical appendix and an Excel evidence workbook**,
**deep-research dossiers** (a planned multi-step investigation — Genie sources
get governed SQL follow-ups in one continuing conversation), designed
**infographics**, animated **video briefings with narration** (six formats),
two-host **data podcasts** (five formats), and **sonified scores** (six
genres) whose tempo follows your growth.
## Get started free — no card required
1. **[Sign up](https://app.alchemylake.com/sign-up)** — 50 free credits land in
your account immediately, nothing to enter but an email.
2. **Grab a developer key** — sign in at [app.alchemylake.com](https://app.alchemylake.com) →
**Studio → Developer · MCP & keys → Forge a new key** (starts with `alk_…`).
3. **Clone this repo and deploy it** into your Databricks workspace:
```bash
git clone https://github.com/zorost/alchemylake-databricks.git
cd alchemylake-databricks
# edit the workspace host in databricks.yml (see Path 1 below), then:
databricks bundle deploy -t prod
databricks bundle run alchemylake_app -t prod
```
Tracking `main` gets you the latest build; `git checkout v0.4.0` first pins
the deploy to a tagged release instead, if you'd rather not move with `main`.
4. Open the App URL the CLI prints, paste your key, then **Load governed
sources → pick one → Transmute.**
That's a governed PDF, deck, infographic, or video briefing — sealed to your
own data, running inside your own workspace — for free, in about five minutes.
No Databricks Apps on your workspace? Skip straight to
[Path 2](#path-2--register-alchemylake-as-an-mcp-server-genie--agent-bricks) —
register one URL and Genie, Claude, or Cursor get the same 50 free credits to
work with, no bundle required.
**Every lane binds to a governed source**, and the numbers discipline holds
throughout: the platform computes an analyst-grade statistical dossier from the
bound rows by code — trend with fit quality, outliers, correlations, segment
shares, concentration, pivots — the model never authors a figure, a **verifier
checks every claim after generation**, and the result is sealed (source · row
count · data sha256 · verification score) with the seal embedded in the file
itself. Genie answers bind as sources too, and one-click **recipes** ship a
board deck or a whole campaign pack from one table. Every credit accounted.
**Where Genie stops, AlchemyLake starts.** Genie answers questions about your
data inside the workspace — tables and charts. AlchemyLake takes the same
governed rows the last mile: *"Ask Genie for Q2 revenue by region, then render
an 8-slide deck titled Q2 Momentum"* leaves the workspace as a .pptx the CFO
can present cold — every figure verified against the rows, provenance sealed
into the file.
This repository **is** a self-contained **Databricks Asset Bundle**. One command
deploys a Databricks App (SSO-authenticated, running on your workspace) that
calls the AlchemyLake platform over MCP. Nothing bypasses governance: the same
credit ledger, provenance seals, and role checks that protect the web app
protect every call made from inside Databricks. Your data never leaves the
lakehouse except as the exact rows you choose to bind.
```
.
├── databricks.yml # the Asset Bundle (App resource + targets)
├── app/
│ ├── app.py # the Streamlit App (thin MCP client)
│ ├── app.yaml # App runtime config (command + env)
│ └── requirements.txt
└── sql/
└── ai_render.sql # optional UC function: call AlchemyLake from SQL/Genie
```
---
## Three ways to use AlchemyLake with Databricks
| Path | What you get | Setup |
|---|---|---|
| **1. The App (this bundle)** | A governed render UI inside your workspace, SSO’d, next to your data | `databricks bundle deploy` |
| **2. MCP for Genie / Agent Bricks** | Every agent gains **13 governed tools** (`list_governed_sources`, `upload_source`, `get_wallet`, `render_governed_chat`, `render_deep_research`, `render_infographic`, `render_report`, `render_presentation`, `render_video_briefing`, `render_music`, `render_podcast`, `list_recipes`, `run_recipe`) — all render tools accept `source_id` for data-bound, verified output, and chat returns a `thread_id` so a Genie conversation continues across agent turns | Register one URL |
| **3. `ai_render()` in SQL** | Sealed narrative from a query or Genie space | Run `sql/ai_render.sql` |
**What agents can ship from a table** (things a Genie answer alone cannot):
- `render_deep_research` — a planned multi-step investigation: the brief is
decomposed into sub-questions, each answered with real evidence (Genie
sources get governed SQL follow-ups in one continuing conversation), then
synthesized into a sealed PDF dossier + Excel evidence workbook.
- `render_presentation` — a .pptx board deck, 5–20 slides, speaker script + Q&A
in every notes pane, AI cover art, real charts from the rows.
- `render_report` — an enterprise PDF dossier (KPI band, chart sections,
statistical appendix, citations, methodology) **plus** an Excel evidence
workbook: raw rows, facts, statistics, pivot, correlations.
- `render_infographic` — a designed KPI poster with the exact figures rendered
in-image and a branded provenance strip.
- `render_video_briefing` — an animated video briefing: platform-drawn charts,
spoken narration, motion and crossfades. Six formats (`style` param):
consultant walkthrough, newsroom segment, executive stand-up, documentary
deep-dive, field report, social recap.
- `render_podcast` — a two-host audio briefing with a sealed transcript. Five
formats: two-host interview, skeptic's debate, executive stand-up, narrative
deep-dive, plain-language walkthrough.
- `render_music` — a sonified score (tempo ↔ momentum, mode ↔ trend) plus the
literal data-motif WAV of the rows. Six genres: cinematic score, corporate
uplift, ambient data fields, electronic pulse, orchestral arc, lo-fi data
study.
Prefer plain REST? The same keys work on the public API:
`https://app.alchemylake.com/api/public/v1` (OpenAPI at `/api/public/v1/openapi.json`).
Prefer a terminal? `npx alchemylake` is a zero-dependency CLI over that API:
```bash
export ALCHEMYLAKE_API_KEY=alk_YOUR_KEY
npx alchemylake upload ./q3-actuals.xlsx # CSV/Excel/PDF/Word/text → governed source
npx alchemylake render report --prompt "Board brief" --source up.1a2b3c
npx alchemylake render deep_research --prompt "Why did Q3 dip?" --source genie:dbx1
```
---
## How it fits together
AlchemyLake never runs *inside* this bundle — the App is a thin, SSO'd client that
calls the governed platform over HTTPS/MCP. Your tables stay in your lakehouse; only
the exact rows you bind for a given render ever leave the workspace, and only to
produce that one sealed deliverable.
```mermaid
flowchart LR
subgraph WS["Your Databricks workspace"]
UC[("Unity Catalog
tables & volumes")]
Genie["Genie space"]
CSV["Uploaded CSV / Excel"]
App["AlchemyLake App
(this bundle · Streamlit · SSO'd)"]
Agents["Genie / Agent Bricks
Claude · Cursor"]
end
AL(["AlchemyLake
governed render platform"])
UC -- "bind rows" --> App
Genie -- "bind an answer
(conversation continues)" --> App
CSV -- "bind (BYO-data)" --> App
App <-- "MCP: render, verify, seal" --> AL
Agents <-- "13 MCP tools, same contract" --> AL
AL -. "no-egress text tier" .-> FM["Your Databricks
Foundation Model endpoint"]
AL --> Out(["Sealed deliverable
PDF · PPTX · image · video · audio"])
```
Every render — from the App, from an agent, or from `ai_render()` in SQL — goes
through the same discipline before it comes back:
```mermaid
flowchart LR
A["Bind
pick a governed source"] --> B["Analyze
compute the facts"]
B --> C["Design
plan the deliverable"]
C --> D["Transmute
generate"]
D --> E["Verify
check every claim"]
E --> F["Seal
embed provenance"]
```
The model is never the source of a number — every figure is computed from your rows
first, and whatever the render says is checked against those figures afterward. The
seal (source · row count · data hash · verification score) rides embedded in the file
itself, in every format, so it survives being forwarded, downloaded, or printed.
### Beyond a Genie answer
| | Genie alone | Genie / Agent Bricks + AlchemyLake |
|---|:---:|:---:|
| Answers a question inside the workspace | ✅ | ✅ |
| Ships a board-ready **.pptx** with a read-aloud script + Q&A prep per slide | | ✅ |
| Ships an enterprise **PDF dossier + Excel evidence workbook** | | ✅ |
| Runs a **deep-research investigation** (planned Genie SQL follow-ups → sealed dossier) | | ✅ |
| Ships a designed **infographic** with figures rendered in-image | | ✅ |
| Ships a narrated **video briefing** or two-host **podcast** | | ✅ |
| Every number checked against the source *after* generation and scored | | ✅ |
| Provenance (source, rows, hash, score — plus Genie reasoning, SQL, and trusted-asset status) embedded in the output | | ✅ |
| Scores your Genie space's curation health and certifies it with live probes | | ✅ |
| Metered by an auditable, per-render credit ledger | | ✅ |
### What people actually ask it for
Every lane binds to a governed source, so the figures in the output are the figures
in your table — not a paraphrase. A sample of real workflows across the eight lanes:
| Lane | Ships | Example asks |
|---|---|---|
| **Analyst** (chat) | A sealed, sourced answer — follow-ups stay in context | *"Draft the board narrative from the certified loan book."* · *"Now break that down by region."* |
| **Deep Research** | A sealed investigation dossier (PDF + Excel) | *"Why did Q3 dip, and what should we do about it?"* · *"Investigate concentration risk across our segments."* |
| **Infographic** | A KPI poster, figures rendered in-image | *"This week's sales milestone, ready for social."* · *"'45 days without a recordable incident' for the plant floor."* |
| **Report** | PDF dossier + Excel evidence workbook | *"Quarterly board-packet narrative, sealed."* · *"Compliance rollup with a hash of the source data baked in."* |
| **Presentation** | .pptx, speaker script + Q&A per slide | *"QBR deck with the CRM's exact numbers."* · *"Steering-committee deck anyone on the team can present cold."* |
| **Video briefing** | A narrated, animated data video | *"All-hands opener with the real ARR on screen."* · *"30-second earnings-day teaser, safe to post at the bell."* |
| **Music** | A score that follows the actual trend | *"A launch-video bed whose tempo rises with the real growth curve."* |
| **Podcast** | A two-host audio briefing + transcript | *"A 3-minute briefing leadership can play on the commute."* · *"An audio version of the same sealed report, for accessibility."* |
Free to try: [sign up](https://app.alchemylake.com/sign-up) for 50 credits, no card
required, and point this bundle at your own key.
---
## Prerequisites
1. **Databricks CLI ≥ 0.230** and authentication to your workspace:
```bash
databricks auth login --host https://.cloud.databricks.com
```
2. **Databricks Apps enabled** on the workspace (Premium/Enterprise).
3. An **AlchemyLake developer key** (`alk_…`): [sign up free](https://app.alchemylake.com/sign-up)
(50 credits, no card required) or sign in at →
**Studio → Developer · MCP & keys → Forge a new key**.
---
## Path 1 — Deploy the App (one command)
First set your workspace host in `databricks.yml` — replace the
`your-workspace…` placeholder under **both** `targets.dev` and `targets.prod`
(run `databricks auth profiles` if you need to find it). Then:
```bash
databricks bundle deploy -t prod # upload source + create the App
databricks bundle run alchemylake_app -t prod # start it; prints the App URL
```
> **CLI note (Terraform pin).** Some Databricks CLI builds download a Terraform
> whose signing key has expired, which breaks `bundle deploy`. If you hit that,
> point the CLI at a local Terraform binary and pin the version to match it:
> ```bash
> export DATABRICKS_TF_EXEC_PATH="$(command -v terraform)"
> export DATABRICKS_TF_VERSION="1.9.8" # set to your local `terraform version`
> ```
> This is the exact, verified path AlchemyLake's own reference App is deployed
> with — it runs live in-workspace as a Streamlit App under SSO.
Open the App URL that the CLI prints. Paste your `alk_…` key in the sidebar
(or bind it from a secret — see below), then **Load governed sources → pick one
→ Transmute**. Every result carries its provenance seal.
The pasted key is remembered in that browser (client-side only, until cleared
or replaced) so it survives a refresh — a workspace-bound secret still always
takes priority. The App also carries a self-contained **Docs** tab and an
expanded **About / Install** tab (how to use it, why the verification model
matters, where the render compute actually runs, and the fastest path to a
key) so it's useful even where the public docs site isn't reachable.
### Bind the key from a secret (recommended for teams)
So analysts never paste a key:
```bash
databricks secrets create-scope alchemylake
databricks secrets put-secret alchemylake api_key # paste alk_...
```
Then uncomment the `ALCHEMYLAKE_API_KEY` / `valueFrom` block in
`app/app.yaml` and `databricks.yml`, and redeploy.
---
## Path 2 — Register AlchemyLake as an MCP server (Genie / Agent Bricks)
No bundle required. In your agent/MCP client configuration add:
```json
{
"mcpServers": {
"alchemylake": {
"url": "https://app.alchemylake.com/api/mcp",
"headers": { "Authorization": "Bearer alk_YOUR_KEY" }
}
}
}
```
Genie / Agent Bricks, Claude, and Cursor will discover the thirteen tools
automatically. Ask, for example: *“Using AlchemyLake, write a sealed 3-bullet
executive summary of `gold_ridership_national_monthly`”* — or go bigger:
*“Render an 8-slide presentation of that table titled Ridership Momentum”*
(a .pptx with a presenter script under every slide), *“Run deep research on
why ridership dipped this quarter”* (`render_deep_research` returns a sealed
dossier + evidence workbook), or *“Run the campaign-pack recipe”*
(`run_recipe` renders the poster, the report, and the copy in one call, all
verified and sealed).
---
## Path 3 — `ai_render()` in SQL / Genie
Edit `sql/ai_render.sql` (set your catalog/schema) and run it in a SQL editor
on **serverless compute with external network access enabled**. Then:
```sql
SELECT ai_render(
'One-sentence executive read of this quarter''s ridership',
'ntd_demo.ntd.gold_ridership_national_monthly'
) AS narrative;
```
See the header of `sql/ai_render.sql` for the network-egress and secret
prerequisites, and the MCP fallback if egress cannot be enabled.
---
## Validate before deploying
```bash
databricks bundle validate -t prod
```
---
## FAQ
**Does my data leave the lakehouse?** Only the exact rows you bind for a given
render are sent to produce that render — nothing else is read, and bound data is
never used to train models. A no-egress **text tier** can route even that narrative
step to your own Databricks Foundation Model endpoint instead; see
[the residency docs](https://app.alchemylake.com/docs#residency).
**Do I need Databricks Apps enabled for this?** Only for Path 1 (the App). Path 2
(MCP) and Path 3 (`ai_render()`) need nothing beyond CLI/SQL access and a developer
key — no App, no extra compute resource.
**Can agents use this without a human in the loop?** Yes — that's Path 2. Register
`/api/mcp` once and Genie, Agent Bricks, Claude, and Cursor all gain the same thirteen
governed tools, metered and sealed exactly like the UI.
**What does a Genie question cost?** Genie compute bills inside *your* Databricks
workspace (serverless DBUs — roughly 1.5 DBU ≈ $0.11 per question; Databricks
gives every user 150 free Genie DBUs a month). AlchemyLake credits meter only the
render itself, and the Studio shows the DBU estimate next to any Genie source.
**I don't have Databricks — can I still use everything?** Yes. Upload CSV, Excel,
PDF, Word, text or Markdown (in the web Studio, this App's Sources tab, the API,
the CLI, or the `upload_source` MCP tool) and every lane — including Deep
Research — runs with the same verification and seals.
**Is AlchemyLake a Databricks product?** No. AlchemyLake is an independent platform
by [Zorost Intelligence](mailto:info@zorost.com) that plugs into Databricks; this
repository is Apache-2.0 licensed and is how it plugs in. The hosted platform it
talks to is a separate, metered service (50 free credits to start, no card
required).
**What does this bundle actually contain?** Only a Streamlit client (`app/app.py`),
its App runtime config, and an optional SQL function — everything under
[`sql/`](./sql) and [`app/`](./app). It holds no model weights, no proprietary
rendering code, and no secrets; it is a thin, auditable client over the same MCP/REST
contract documented at [app.alchemylake.com/docs](https://app.alchemylake.com/docs).
**Something's broken.** Open an issue in this repository, or email
info@zorost.com — see [`SECURITY.md`](./SECURITY.md) for vulnerability reports
specifically.
---
## Documentation
- [`docs/DEVELOPER_GUIDE.md`](./docs/DEVELOPER_GUIDE.md) — **CLI, REST API & MCP**: get a key, every command, VS Code/Cursor/Claude setup, use cases, Python/Node examples, GitHub Actions.
- [`docs/GENIE_GUIDE.md`](./docs/GENIE_GUIDE.md) — Genie integration: connect a space, health score, certification, all eight lanes, sharing model.
- Platform docs: [app.alchemylake.com/docs](https://app.alchemylake.com/docs) (Studio walkthrough, credits, security).
---
## Uninstall
```bash
databricks bundle destroy -t prod
```
---
*AlchemyLake is a product of **Zorost Intelligence**. Platform docs:
. Support: info@zorost.com.*