# 14 — Real compute: PySpark, Delta, and the warehouse

**Status: shipped (R0–R5).** This document laid out the track that turns
fabric-emulator from a *contract emulator* into a *local Fabric runtime* — by
attaching **real engines**, never by faking results. Most of it now exists, each
milestone gated on a real client in CI; the honest engine and protocol
boundaries are marked inline. Tense below is kept as originally written where it still
reads as intent; shipped items carry a ✅ and their `e2e/` witness.

> **Historical terminology:** where this design document says "real Spark" it
> originally meant JVM `apache/spark`. The default engine has since moved to
> **LakeSail's Sail** (Rust Spark-Connect server — same PySpark clients, no
> JVM). Read those references as "Spark-compatible Sail execution" unless a
> section explicitly names the optional `e2e/spark-jvm` oracle. See
> [20-lakesail-engine.md](20-lakesail-engine.md) for the current parity tiers.

## The principle (a refinement, not a reversal)

[03-architecture.md](03-architecture.md) declares "no compute engines" a
non-goal. What that non-goal was always protecting against is *pretend*
compute — a notebook run that "succeeds" without running, a query endpoint
returning canned rows. The refinement, proven by the fabric-cicd and
azure-keyvault-emulator work:

> **Never fake results. Either do it for real (attached real engines) or
> fail honestly (501).**

Real PySpark executing real Delta commits against our OneLake plane is
consistent with that. Emulated Spark would not be.

## Where this lives

**In this repository.** The family's repo boundaries follow service/trust
surfaces (STS / Fabric planes / vault), and real compute is not a new
service — it is:

1. this repo's OneLake plane becoming complete enough for real engines,
2. e2e harnesses in `e2e/` (like `e2e/fabric-cicd/`),
3. compose-level sidecar attachments (Spark, SQL Server, Airflow, kustainer, Kafka).

This held even for the piece once expected to become a separate service: the
**TDS-FedAuth termination** (Track C) shipped **in this repo** as `internal/tds`
(a pure-Go TDS front that validates the login's Entra token and then byte-splices
the session to the SQL Server sidecar) — not a sibling repo after all.

## The junction: OneLake is where engines meet

In real Fabric, Spark and the Warehouse are engines rendezvousing on **Delta
files in OneLake**. Our P3 data plane is therefore the single point of
leverage: make it complete enough for real storage clients, and every real
engine above it inherits the emulator — including its auth (Storage-audience
tokens from entra-emulator, workspace RBAC).

## Track A — storage completeness (the enabler)

What the P3 wire subset lacks for real engines, in dependency order:

| Gap | Needed by | Notes |
|---|---|---|
| **Range reads** (`Range: bytes=a-b`) | every Parquet reader | Parquet is seek-heavy; 206 Partial Content |
| **ETags + conditional requests** (`If-Match`/`If-None-Match`) | **Delta commit atomicity** | Delta's ADLS log store commits `_delta_log/N.json` with put-if-absent; without this, concurrent commits silently corrupt — the one gap that affects *correctness*, not just compatibility |
| **Rename** (`PUT dst` + `x-ms-rename-source`) | Hadoop committers | staged-file rename; delta-rs does not need it, ABFS does |
| List-paging fidelity (continuation, `etag`/`lastModified` per entry) | ABFS `listStatus` | |
| Blob-endpoint alias (`onelake.blob.…`) | some clients | parity doc: both endpoints exist |

**Milestones** (each an e2e harness in `e2e/`):

- **A1 — delta-rs**: the Python `deltalake` package (explicitly listed as
  OneLake-compatible in `onelake-api-parity.md`) writes and reads a Delta
  table through our DFS surface with an entra Storage token. Object-store
  semantics only — needs Range + conditional writes, not rename. Closes the
  SDK half of the "azcopy / ADLS SDK e2e" roadmap item; the azcopy half is now
  also wired (`e2e/azcopy`).
- **A2 — real PySpark**: `pyspark` + `delta-spark`, ABFS driver
  (`abfss://{workspace}@onelake.dfs.fabric.microsoft.com/{item}/…`, the
  documented URI), OAuth client-credentials token provider pointed at
  entra-emulator. Write a Delta table from Spark; read it back with delta-rs
  (A1) — cross-engine interop on our storage.

  > **A2 is delivered inside Track B, not standalone (decided).** A1 (delta-rs)
  > and the ADLS-SDK e2e both redirect cleanly because their clients expose an
  > endpoint knob (`azure_endpoint`, `account_url`). The Hadoop **JVM ABFS
  > driver does not** — it derives its endpoint from the URI authority and
  > takes no host/port override, so pointing it at a localhost emulator has no
  > in-process ("ABFS-in-a-venv") answer. The structural fix is a **container
  > network** where `onelake.dfs.fabric.microsoft.com` resolves to the
  > emulator — which is exactly the Track B sidecar. So A2 rides on B's
  > containerized Spark rather than being fought first as a bare-venv
  > milestone. Bonus: the JVM lives in a Linux container, making this the
  > only cross-OS-honest path for Windows developers (thin client, no
  > `winutils`).

## Track B — Spark execution (jobs become real), with A2 riding on it

Fabric's documented Spark surface is the **Livy API**
(`data-engineering/get-started-api-livy.md`), and its endpoint lives on the
control-plane origin we already serve:

```
https://api.fabric.microsoft.com/v1/workspaces/{ws}/lakehouses/{lh}/livyapi/versions/2023-12-01/{sessions|batches}
```

- **B1 — Livy sessions on real Spark. ✅** Those routes are implemented
  (bearer-validated, RBAC-checked like everything else). Apache Livy itself is
  retired (Attic), so the shipped path is **native**: `--spark-agent-url` points
  the emulator at a Spark **statement-executor agent** (`python/spark_agent/agent.py`) that
  runs each session/statement/batch on real Spark, with the session configured so
  `abfss://` resolves to our OneLake plane — Fabric-shaped URL outside, real Spark
  inside (`internal/api/livy_native.go`, `e2e/livy`). A `--spark-livy-url` mode
  still reverse-proxies an external Livy server if you have one. Fabric's
  **high-concurrency** (5-REPL) session packing is implemented for real on top
  (`internal/api/livy_hc.go`). Without either flag, statements 501 honestly.
- **B2 — jobs integration. ✅** `RunNotebook` job execution parses the notebook
  into cells and runs them on the Spark agent, finalising the job's terminal
  status and exit value from the *actual* run (`e2e/notebook-run`); without an
  agent, the deterministic clock behavior remains the default — CI stays fast.
- **Token passthrough**: the Livy docs define `Code.Access*` scopes for Spark
  code acquiring downstream tokens — including `Code.AccessAzureKeyvault.All`.
  Spark code calling `mssparkutils`-style credential helpers can be pointed
  at entra-emulator's MSI endpoint, and secrets resolve from
  **azure-keyvault-emulator** — the three-emulator chain, from inside a real
  Spark job.

## Track C — the warehouse (two fidelity targets, priced separately)

- **C1 — SQL analytics endpoint *semantics*. ✅** **DuckDB** with its delta
  extension queries the *same Delta files* Spark wrote in Track A — completely
  real SQL over completely real Delta, the actual lakehouse↔warehouse interop
  story, verified in `e2e/duckdb` (aggregation / join / filter over lakehouse
  Delta, agreeing with the TDS engine).
- **C2 — real T-SQL engine. ✅** Sidecar **SQL Server on Linux** (real TDS +
  T-SQL; Babelfish considered and rejected for fidelity — see
  [16-warehouse-tds.md](16-warehouse-tds.md)), each Warehouse/Lakehouse item
  routed to its **own SQL Server database**, with a Lakehouse's OneLake Delta
  reflected into the engine on connect (read-only) and a Warehouse relayed
  read-write. RBAC → SQL permissions, `information_schema` parity, native
  per-column type fidelity, connect-by-name.
- **C3 — Entra FedAuth over TDS. ✅ (in-repo, not a proxy repo).** The original
  "local engines only do SQL auth" compromise was dissolved: `internal/tds`
  terminates the FedAuth login itself (validates the login's Entra token against
  entra-emulator's JWKS, `database.windows.net` audience) and then **byte-splices**
  the client's post-login session straight to the SQL Server sidecar over a SQL
  login — so the engine emits every token natively (transactions, RPCs, prepared
  statements). This restores the production auth story for real clients:
  unmodified `go-mssqldb` **and** the Microsoft ODBC Driver 18 (pyodbc) connect,
  and Microsoft's real **dbt-fabric** adapter passes end-to-end (`e2e/dbt-fabric`).
- **C4 — Fabric SQL Database (OLTP + OneLake mirror). ✅** The `SQLDatabase` item
  reuses the read-write TDS path (its own SQL Server database) and adds
  **mirroring**, the reverse of the lakehouse reflection: `warehouse.Mirror`
  reads the item's SQL tables and writes each as a **Delta** table into OneLake
  `Tables/` (real Parquet + `_delta_log`), triggered by
  `POST …/sqlDatabases/{id}/refreshMirror`. So an operational database becomes
  queryable as Delta by Spark/DuckDB/delta-rs — the OLTP↔analytics story. Proven
  by a gated e2e (write over TDS → mirror → the Delta reads back). *Deferred:*
  continuous/CDC mirroring (the emulator snapshots on the refresh trigger).

## Track D — the notebook developer loop

A Fabric "Notebook" is four layers; three already have designs. This track
adds the fourth and composes them into a complete local environment for a
Fabric data engineer:

| Layer | Status |
|---|---|
| **The item + source format** (`.platform` + `notebook-content.py`, definitions, git/fabric-cicd round-trip) | ✅ shipped (P1) |
| **Execution** (Livy on the documented endpoint → real Spark sidecar) | 📐 Track B |
| **Storage** (Delta in OneLake via real engines) | 📐 Track A |
| **The runtime library — NotebookUtils** | ⬅ this track |

**D1 — a functional `notebookutils` shim** (pip-installable Python package,
`python/notebookutils/` in this repo). `notebook-utilities.md` documents ten
modules; every one maps onto surfaces the family already serves:

| Module | Backed by |
|---|---|
| `notebookutils.fs` (+ mount) | the OneLake DFS plane |
| `notebookutils.credentials.getToken` | entra-emulator (MSI endpoint / workspace identity) |
| `notebookutils.credentials.getSecret` | **azure-keyvault-emulator** — the `Code.AccessAzureKeyvault.All` path, for real |
| `notebookutils.notebook` (run/runMultiple/exit + management) | jobs + Livy (B2) and `/v1` item CRUD |
| `notebookutils.lakehouse` | `/v1/workspaces/{id}/lakehouses` |
| `notebookutils.runtime` (context) | session/workspace context from the emulator |
| `notebookutils.session`, `udf`, `variableLibrary` | session mgmt; 501-honest until their items exist |

Naming note: Microsoft publishes a non-functional `notebookutils` typing stub
on PyPI for IDE support; ours is a *functional* implementation of the same
documented API — distribution name TBD to avoid collision, module name
compatible so notebook code runs unchanged.

**D2 — default-lakehouse session semantics**: the Spark sidecar's sessions
are provisioned with the notebook's attached lakehouse mounted — relative
`Files/`/`Tables/` paths and `spark.sql` over lakehouse tables resolve to our
OneLake plane, matching the attached-lakehouse experience. Spark/Delta
versions pinned to the documented Fabric Runtime (1.3 = Spark 3.5/Delta 3.x).

**D3 — authoring surfaces**, in order of fidelity:

1. **VS Code + the Microsoft Fabric extension. ✅** The shared-backend/MWC
   authoring protocol extracted from Microsoft's pinned 1.18.1 extension is
   served through a DNS alias for `api.powerbi.com`: discovery, item authoring,
   notebook content/resources with ETags, Spark-job actions, and lakehouse table
   discovery. `e2e/vscode-extension` replays that exact route contract with a
   real Power BI-audience token. Interactive Jupyter kernel websockets and table
   preview are not claimed; execution remains on the jobs/Livy/Sail APIs.
2. **Plain Jupyter/VS Code `.ipynb`** + git / fabric-cicd sync — works
   **today**: author anywhere, `fabric-cicd` publishes into the emulator,
   `updateFromGit` pulls edits back.
3. The Svelte portal gets a read-only notebook view, not an editor —
   authoring belongs to real tools.

The complete loop this yields: *author in VS Code → sync via git/fabric-cicd
→ run interactively on the Spark sidecar with the default lakehouse mounted →
`notebookutils` resolves files, tokens, and Key Vault secrets through the
emulator family → Delta lands in OneLake → query it via the DuckDB SQL
endpoint → schedule via the jobs API → CI drives the identical REST surfaces.*

## Track E — pipelines: real orchestration where it exists, real work everywhere

Pipelines are the sharpest test of the never-fake principle, because the
engines split three ways:

**E1 — Apache Airflow: the fully real tier. ✅ shipped.**
Fabric's own code-first orchestrator is **genuine Apache Airflow**
(`apache-airflow-jobs-concepts.md`: "Python-based DAGs", the next generation
of ADF's Workflow Orchestration Manager) — and Airflow is OSS. So the
highest-fidelity local pipeline story attaches a **real Airflow sidecar** as
the runtime behind `ApacheAirflowJob` items:

- DAG files stored as item definitions (the P1 machinery), synced into the
  sidecar's DAG folder;
- operators drive our control plane over REST — trigger notebook jobs (real
  Spark via Track B), poll LROs, move OneLake data;
- Airflow's **Azure Key Vault secrets backend** (documented:
  `apache-airflow-jobs-enable-azure-key-vault.md`) pointed at
  azure-keyvault-emulator — connections/variables resolve from the family's
  vault.

The shipped `ApacheAirflowJob` file API stores DAG sources in OneLake, syncs
them into a shared DAG volume, and drives the upstream Airflow REST API for
discovery, unpause, trigger, and terminal-state polling. Without the optional
backend flags it fails with `AirflowNotConfigured`. `e2e/airflow` witnesses the
real scheduler, executor, PythonOperator task, and Fabric job completion.

Real scheduler, real executor, real DAG semantics — zero orchestration
emulation. And not merely "compatible": Fabric's hosted offering *is*
upstream Apache Airflow (the docs pin exact releases — currently Airflow
**2.10.5** on Python **3.12** — and point at airflow.apache.org's own guides
for custom operators/hooks/plugins). The sidecar pins the same documented
versions, so a DAG that runs locally runs identically hosted. For data
engineers who *can choose*, this is the recommended local pipeline path.

(To be precise about what Airflow is and isn't in Fabric: the default
no-code **Data pipeline** does *not* run on Airflow — that's the proprietary
ADF-lineage engine E2 addresses. Airflow is Fabric's separate code-first
orchestrator, offered side by side.)

**E2 — DataPipeline interpreter: our control flow, real work. ✅ (core).** The
no-code pipeline engine (ADF lineage) is proprietary — Polaris-class
unobtainable. For `DataPipeline` items we implement the **documented control-flow
semantics** ourselves — `dependsOn` conditions
(Succeeded/Failed/Completed/Skipped), ForEach, If / Until (+timeout), Switch,
Filter, Fail, the ADF expression language, and per-activity retry/timeout
policies — with leaf activities delegating to real work where an engine exists:

| Activity | Executes via | Status |
|---|---|---|
| Invoke pipeline | recursive interpretation of the referenced `DataPipeline` | ✅ real |
| Lookup | reads rows from a CSV/JSON/Parquet file, or a lakehouse Delta table, **directly in OneLake** (pure-Go, real bytes) | ✅ real |
| GetMetadata | stats a OneLake path (exists/size/children) | ✅ real |
| Copy | real byte movement, **OneLake→OneLake** (a file or a directory subtree) | ✅ real (in-family) |
| Notebook | Livy → the real Spark agent (Track B) | ✅ real (with `--spark-agent-url`) |
| Script / SqlServerStoredProcedure | real T-SQL against a Warehouse/SQLDatabase item's own SQL Server database (Track C's backend) — the emulator's own scoped `{workspaceId?, itemId}` target reference, not Fabric's linkedService wire shape | ✅ real (with `--warehouse-sql-url`) |
| Web / WebHook | real HTTP call; `FABRIC_WEB_ACTIVITY=stub` records success without calling | ✅ real (default) |
| Custom (Azure Batch) | `command` runs in the Spark agent container; `FABRIC_CUSTOM_ACTIVITY=off` refuses | ✅ real (default) |
| RestSource / RestSink | real HTTP + pagination into a Lakehouse Delta sink | ✅ real |
| Salesforce | Bulk API 2.0 ingest and sink | ✅ real |
| SparkJobDefinition / InvokeCopyJob | run the referenced item; CopyJob moves real OneLake bytes | ✅ real |

Control-flow fidelity is deterministic on the controllable clock: **ForEach**
honors `isSequential` + `batchCount` and reports the right wall-clock (sequential
iterations add, a parallel batch costs its slowest); a per-activity **timeout**
fails an over-running attempt; and **retry with `retryIntervalInSeconds`** folds
the backoff into the activity's duration — all exercised in milliseconds, no real
sleep. Stated plainly: the orchestrator is *ours* — faithful to documented
control-flow semantics — but it is not Microsoft's engine. The *work* a pipeline
does is real: it exercises real Spark, real OneLake bytes, and real data movement.

**E3 — honestly unobtainable → 501.** Dataflow Gen2 (the Power Query M
compute is proprietary), self-hosted integration runtime / gateway scenarios,
and cloud connectors outside the scoped set (REST, Salesforce, ServiceNow via
RestSource, and Custom are in-scope and run). Activities we cannot execute
for real fail with a clear 501 — a pipeline "succeeding" without doing its
work is exactly what this design forbids.

CI/CD for pipelines needs none of this and works **today**: `DataPipeline`
definitions round-trip through git and fabric-cicd like any other item.

## Engine weights — what actually runs, and when

The **binary itself** never gets heavier: engines attach via flags
(`--spark-agent-url`, `--warehouse-sql-url`, `--sql-tds-addr`,
`--airflow-url`/`--airflow-dag-dir`), and with none of
them set it runs nothing but the Go process — clock-derived jobs, milliseconds,
no JVM. That's what CI's fast unit-test job runs, and it's what a bare
`go run`/`fabric-emulator` binary gives you.

The **bundled `docker compose up`** experience is different on purpose: since
[T5](16-warehouse-tds.md) and native Livy shipped, `docker-compose.override.yml`
(auto-loaded alongside `docker-compose.yml`, no flag needed) wires those same
flags to a Spark agent + SQL Server sidecar it also brings up — so the default
containerized experience has real engines attached, not just the contract. Opt
out to the lite pair with `docker compose -f docker-compose.yml up` (the
override is skipped when you name the base file explicitly). CI's heavy e2e jobs
(`livy-native`, `warehouse-tds`, the dbt suites, …) use their own per-suite
compose files under `e2e/`, independent of this one.

| Rung | What runs | Weight |
|---|---|---|
| Bare binary, no flags | the Go binary only; jobs are clock-derived | milliseconds |
| `docker compose -f docker-compose.yml up` (opt-out) | entra + fabric-emulator only, same as the bare binary | milliseconds |
| `docker compose up` (default) | + a Spark agent + a SQL Server sidecar | ~1.5 GB+, seconds to start |
| A1 delta-rs | Rust library in a Python wheel — no JVM | tens of MB |
| A2/B PySpark | **Sail** (default, Spark Connect, no JVM); JVM overlay is `make up-jvm` | hundreds of MB / GBs |
| C1 DuckDB | in-process library — no server, no JVM | a few MB |
| C2 SQL Server | **full SQL Server engine** in a container | ~1.5 GB, x86 (emulated on ARM) |

Honest asterisk on Track C: Spark's JVM *is* what Fabric runs — the weight
buys true fidelity. DuckDB/SQL Server are **real engines standing in for an
unobtainable one** (Fabric's warehouse is Microsoft's proprietary Polaris
engine): the Delta files they query are byte-identical to production, but
T-SQL dialect edges will differ. Same class of documented divergence as the
C2 SQL-auth compromise.

## Phasing

| Phase | Delivers | Status |
|---|---|---|
| **R0** | Track A storage completeness + A1 (delta-rs e2e) + the ADLS-SDK e2e (real `azure-storage-blob`) + azcopy; concurrent-commit race test | ✅ shipped |
| **R1+R2** (merged) | **containerized Spark** — A2 (real PySpark writes Delta via ABFS, cross-engine read with delta-rs) **and** B1+B2 (native Livy sessions/HC on a real Spark agent; real RunNotebook mode). JVM Spark image + Docker network so ABFS resolves to the emulator. | ✅ shipped (`e2e/spark`, `e2e/livy`, `e2e/notebook-run`) |
| **R3** | C1 (DuckDB SQL over the lakehouse) **and** C2/C3 (SQL Server sidecar + in-repo FedAuth-over-TDS splice; two driver witnesses) | ✅ shipped (`e2e/duckdb`, `e2e/dbt-fabric`, gated TDS tests) |
| **R4** | D1 notebookutils, D2 default-lakehouse sessions, and the pinned D3 VS Code extension authoring contract | ✅ shipped |
| **R5** | E2 DataPipeline interpreter and real-engine leaves (Web, Custom, REST, Salesforce, SJD, InvokeCopyJob included); E1 real Airflow sidecar; explicit Dataflow Gen2 engine boundary | ✅ shipped |

## Correctness: how we prove it

Five oracle layers, cheapest first. The theme throughout: **we don't write
the oracles — we borrow them** from the clients and engines that production
Fabric already has to satisfy.

1. **Protocol conformance — unmodified real clients.** The established
   method (fabric-cicd found four real bugs; azsecrets/azkeys proved the
   vault): if delta-rs, the ADLS SDK, the ABFS driver, and azcopy succeed
   *unmodified*, the wire is right. Every A/B/C milestone is gated on a real
   client, never on our own HTTP calls agreeing with ourselves.
2. **Cross-engine round-trips — the data oracle.** Write with engine A, read
   with engine B: Spark writes Delta → delta-rs reads it → row counts and
   content checksums match → DuckDB queries the same table → same results.
   Three independent implementations agreeing on bytes they didn't write is
   the strongest correctness signal available without a formal spec.
3. **Concurrency and atomicity — the Delta commit protocol.** Two writers
   racing to commit `_delta_log/N.json`: exactly one must win the
   put-if-absent (`If-None-Match: *`), the loser must observe the conflict
   and retry as version N+1, and no committed row may vanish. This is the
   corruption class the R0 ETag work exists to prevent, and it gets explicit
   adversarial tests (concurrent writers + fault injection), not just happy
   paths.
4. **Borrowed reference suites.** The ecosystems ship their own conformance
   tests, written by the people who defined the semantics: Hadoop's **ABFS
   filesystem contract tests** (hadoop-azure), delta-rs's object-store
   integration suite, and dbt's **dbt-tests-adapter** acceptance suite.
   Pointing them at the emulator turns thousands of third-party assertions
   into our regression net. These run as opt-in CI jobs (they're heavy),
   like the fabric-cicd job.
5. **Version pinning as a correctness contract.** Engines and libraries pin
   to the documented Fabric Runtime (Spark 3.5.x / Delta 3.x for Runtime
   1.3; Airflow 2.10.5 / Python 3.12) — so "works on the emulator" and
   "works hosted" refer to the same binaries, and drift is a deliberate
   bump, not an accident.

Where a layer can't apply, the surface 501s rather than passing vacuously —
an untestable claim is treated as a missing feature, not a passing test.

## The compatibility matrix: what must run

**Python on PySpark — Tier 1 (touches our surfaces; each gets an e2e):**

| Library | Why it must work | Surface it exercises |
|---|---|---|
| `pyspark` 3.5.x + `delta-spark` 3.x | the engine itself (Runtime 1.3 pin) | ABFS → OneLake DFS (Track A) |
| `notebookutils` (the D1 shim) | every Fabric notebook imports it | fs → DFS; credentials → entra + vault; lakehouse/jobs → `/v1` |
| `deltalake` (delta-rs) | Spark-free Delta access; the A1 milestone | object-store semantics on DFS |
| `pandas` + `pyarrow` | `toPandas()`/`createDataFrame`, `pyspark.pandas`, parquet bridging | in-engine, plus storage when reading `abfss://` |
| `fsspec` + `adlfs` + `azure-storage-file-datalake` + `azure-identity` | what `pandas.read_parquet("abfss://…")` actually uses | the Python storage path incl. the credential chain |
| `mlflow` | Fabric experiments/models *are* MLflow | real MLflow 3 sidecar through the authenticated workspace proxy; typed `MLExperiment`/`MLModel` items and OneLake artifact mirror (`e2e/data-science-loop`) |
| `semantic-link` (`sempy`) | Fabric-native analytics | partial: list/REST paths work; semantic-model *query* needs the unobtainable engine → 501 |

**Tier 2 (compute-local, storage-agnostic — work automatically once Spark
runs; verified by one smoke notebook, not per-library e2e):** numpy,
scikit-learn, matplotlib/seaborn, and the rest of the Runtime's preinstalled
scientific stack.

**dbt — three adapters, three engine targets:**

| Adapter | Speaks | Works against | Status |
|---|---|---|---|
| **`dbt-fabricspark`** (Microsoft) | the **Fabric Livy API** | our native Livy → real Spark agent; models materialize as Delta in OneLake | ✅ shipped (`e2e/dbt-fabricspark`) |
| **`dbt-fabric`** (Microsoft; documented in `tutorial-setup-dbt.md`) | TDS/pyodbc + Microsoft ODBC Driver 18 to the warehouse SQL endpoint | the C2 SQL Server sidecar, FedAuth terminated in-repo and byte-spliced (T-SQL dialect edges per the Polaris asterisk) | ✅ shipped (`e2e/dbt-fabric`, debug→seed→run→test) |
| **`dbt-duckdb`** (+ delta plugin) | DuckDB in-process | the C1 engine over the same OneLake Delta files | ✅ shipped (`e2e/data-science-loop`, build + four tests) |

Acceptance for each: the official **dbt-tests-adapter** suite plus a
jaffle-shop-style project building end-to-end. The documented
Airflow-orchestrated dbt pattern (`apache-airflow-jobs-dbt-fabric.md`)
composes with E1: a real Airflow DAG running real dbt against the emulator.

## Non-goals

Performance parity, autoscaling/capacity behavior, Spark version matrixing, and
emulating engine *internals* in any form. (High-concurrency Livy session packing,
once listed here as out of scope, shipped — `internal/api/livy_hc.go`.) Where a
real engine can't be attached, the surface returns 501 — it never pretends.