# Feature parity: fabric-emulator vs. real Microsoft Fabric

How the emulator's surface maps to real Fabric (as documented at
[learn.microsoft.com/fabric](https://learn.microsoft.com/en-us/fabric/) /
[MicrosoftDocs/fabric-docs](https://github.com/MicrosoftDocs/fabric-docs)), and
— the point of this table — **whether real work happens or just the API shape**.

The emulator's design bet is that the durable, testable surface is
*contracts + storage + identity + orchestration*, and those are done for real
(real signed JWTs, real Delta bytes on disk, real RBAC, a real pipeline
interpreter, real cross-engine SQL, real Livy high-concurrency session packing).
The heavyweight or proprietary **compute engines** run by default: `docker
compose up` auto-loads an override that starts Sail (Spark Connect) and a SQL
Server sidecar, so Livy sessions, notebook cells and the T-SQL warehouse do real
work out of the box. What stays 🟠 is the narrower set that needs a *different*
engine — the JVM overlay, or an opt-in profile (`rti`, `eventstream`) — and what cannot be done
honestly at all is stubbed.

## What the denominator is

**This table measures completeness against a self-selected surface, not against
Fabric.** The rows below are the surfaces the emulator implements, chosen the
way [07](07-control-plane-api.md) describes — an endpoint-frequency scan of
`fabric-docs`, keeping what SDKs, `fabric-cicd`, git integration and
deployment-pipeline automation actually call. That is a deliberate engineering
choice and a good one; the ratio it produces is not a coverage figure.

For scale: Fabric's REST reference publishes on the order of **880 operations
across ~57 workload groups**. A green count here says the chosen surface is
complete and proven, never that Fabric is covered.

That figure is Fabric's published reference, and it is **not** the denominator
`docs/surface-ledger.json` counts — that one reads the swagger vendored in this
tree and totals 1002 operations, because it also counts Power BI's
`/v1.0/myorg` surface and, for Fabric itself, can only count what is committed
here. The two numbers are reconciled with their sources in
[65](65-api-surface-coverage.md), which is also where the machine-written
surface ledgers are explained.

**Fabric-era workloads this map does not grade at all** — absent rather than
red, so probing is the only way to discover them otherwise: external data
shares, managed private endpoints, Digital Twin Builder, Data Agent, Org Apps,
Graph models, Anomaly Detector, Ontology, Operations Agent, Datamart,
paginated reports, Snowflake and Cosmos DB mirroring, Databricks catalog
mirroring, and the Spark settings and Livy-monitoring endpoints.

**Their ITEMS exist; their WORKLOADS do not.** Every one of Fabric's 50
documented `ItemType` values creates, reads, updates, deletes and round-trips
its definition here, through the generic item surface and through the typed
collection the reference prints for it. That is item CRUD and nothing more: a
Data Agent does not answer questions, an Ontology models nothing, and
`sqlEndpoints/{id}/refreshMetadata` is unimplemented. A client doing CI/CD over
these items works; a client doing the workload's own job does not.

The [Scope boundary](#scope-boundary-fabric-not-the-predecessor-azure-products)
section below covers *predecessor* products (Synapse, U-SQL, ADLS Gen1). This
paragraph is its Fabric-era equivalent.

**"Real via our own wire-protocol implementation."** A row is 🟢 **Real** not
only when an external engine/client does the work, but also when the emulator
*itself* implements Fabric's wire protocol and the logic behind it — so a real,
unmodified client gets byte- and behaviour-identical responses. Fabric's
control plane, OneLake's ADLS/Blob surfaces, the Data Pipeline expression
language + control flow, and the Livy **high-concurrency** session-packing layer
are all in this category: no engine is being proxied, yet the observable
contract matches real Fabric because we built the protocol, not a mock of it.
Where a row's *execution* needs an engine that is not the default — JVM-only
Spark surfaces, an opt-in KQL profile — that part is split out as 🟠 or 🔴.

## Legend

| | Meaning |
|---|---|
| 🟢 **Real** | Genuine work: real signed JWTs, real bytes on disk, a real engine/client computes, real logic enforced — no pretending. |
| 🟡 **Emulated** | Faithful API contract + persisted state, but no engine — status is clock-derived / management-only. |
| 🟠 **Non-default engine** | Real, but only on an engine that is *not* the default: the JVM Spark overlay (`docker-compose.spark-jvm.yml`) or an opt-in profile such as `--profile rti`. **Not** "bring your own": `docker compose up` already starts Sail and the SQL Server sidecar, so the everyday Spark and T-SQL surfaces are 🟢. |
| 🔴 **Not implemented** | Honest 501 or absent. |

## Platform / fundamentals

| Fabric feature | Emulator | Type |
|---|---|---|
| Workspaces CRUD | Full. Display names are **unique tenant-wide** — duplicates 409 `WorkspaceNameAlreadyExists` (uniqueness per the REST reference; fabric-docs covers workspace naming portal-side only) | 🟢 Real |
| Items CRUD + typed collections | Full. Display names are **unique per (workspace, type)** — duplicates 409 `ItemDisplayNameAlreadyInUse`; names stay reusable *across* types, which is why OneLake addresses items as `name.Type`. The `type` is validated against the documented **`ItemType` enumeration** (50 values, REST reference) — anything else is `InvalidItemType`, as real Fabric returns — and canonicalised case-insensitively so `notebook` and `Notebook` cannot become two types. 24 typed collections alias the generic surface; each collection segment is taken from its own reference page, since they are not derivable (`GraphQLApis` is capitalised, `variableLibraries` is not) | 🟢 Real |
| Role assignments / workspace RBAC | Enforced from the validated bearer principal | 🟢 Real |
| Folders | Full — list/create plus get, rename, delete (empty only), and move; a cycle is `InfiniteFolderHierarchyLoop` | 🟢 Real |
| **Catalog Search** (`POST /v1/catalog/search`) | Cross-workspace item discovery over workspaces the caller can already see. `search` matches display name, workspace name and description; `filter` supports `Type eq`/`ne` with `or`/`and`/parens; `pageSize` 1–1000 (default 50). Dashboard and Dataflow are excluded as the reference requires. Returns `ItemCatalogEntry` (`catalogEntryType: FabricItem`). Does not grant data-plane access | 🟢 Real |
| **Fabric Core MCP Server** (`POST /v1/mcp/core`) | Streamable HTTP JSON-RPC facade over the existing Core REST — same handlers, so RBAC, audit and LRO stay one path. Tool names match Microsoft's published Core MCP list. Does **not** run notebooks or write lakehouse tables (Microsoft's own limitation). GET is 405 (no SSE). Witnessed by the unmodified Python `mcp` SDK in CI. Distinct from local `Fabric.Mcp.Server` and from pbix-mcp | 🟢 Real |
| **Fabric IQ MCP** (`POST /v1/mcp/fabriciq`) | Read-only MCP server over Power BI reports and semantic models, on the same transport as Core MCP. The six documented tools — `DiscoverArtifacts`, `ResolveFabricItem`, `GetReportMetadata`, `GetSemanticModelSchema`, `ValueSearch`, `ExecuteQuery` — run as the signed-in user: **delegated tokens only**, as Microsoft documents, so a service principal is refused before any tool runs; a Fabric or a Power BI audience token is accepted; `X-Variants: Fabric.Routing.FabricIQ.V1` is served and any other variant refused. Each tool needs Read on the item, not Build, and row- and object-level security apply through the loader `executeQueries` uses, so two users asking the same question get different rows. `ExecuteQuery` runs 1–4 DAX queries, 250 rows by default and at most 1,000, and refuses MDX, DMV and `INFO` functions; the DAX is the bounded subset, now with `ORDER BY`. `GetReportMetadata` reads PBIR and PBIR-Legacy definitions: pages, visuals and the fields they bind, filters at report, page and visual level (simple conditions restated, the rest kept raw), report-level measures, and the bound model. `queries` takes JMESPath, with `regex_match`. **Inferred, not published:** Microsoft does not document the tools' input or output schemas, so the argument names follow its Fabric IQ skill and the response documents follow the paths that skill queries. **Not modelled:** verified answers and AI instructions (present and empty), workspace-app URLs, the embedded CSV resource for large results, OAuth discovery. Witnessed by the unmodified Python `mcp` SDK in CI, as two users whose answers differ by their RLS role ([07](07-control-plane-api.md#fabric-iq-mcp)) | 🟢 Real (tool schemas inferred from Microsoft's skill) |
| **Fabric Data Warehouse MCP** (`POST /v1/mcp/dataPlane/sqlEndpoint`) | Microsoft's MCP server for T-SQL, at both documented endpoints: the global one and one scoped to a single item. Its one tool, `execute_query(workspaceId, itemId, query)` (also answering to the Learn page's `executeSQL`), runs one batch on a Warehouse, or on a lakehouse's SQL analytics endpoint named by its endpoint id, **as the signed-in user**. It takes the TDS wire's own path: route and Read check, the read-only rules (a Viewer's session, the endpoint's data), Fabric's dialect and time travel, a login as the caller so that grants and row-, column- and object-level security are SQL Server's, and lineage and versioning for an accepted write. Returns the last result set as an embedded `text/csv` resource then `Query returned N rows.`, at most 10,000 rows, with a 300-second timeout; a SQL error is `Error -32002: <message>`. **From a live capture, not Microsoft's docs:** the tool name `execute_query` (Learn says `executeSQL`), `serverInfo` `microsoft.fabric.sqlEndpoint`, the title, annotations, result and error shapes, as recorded by a third party (`iemejia/fabio`) and matching Microsoft's skills; the row and time limits are what the skills call observed defaults. **Not modelled:** the observed 20-requests-a-minute limit, OAuth discovery. Witnessed by the unmodified Python `mcp` SDK in CI on a real SQL Server, as two users whose rows differ by a row-level security policy ([07](07-control-plane-api.md#fabric-data-warehouse-mcp)) | 🟢 Real (contract from a third-party live capture and Microsoft's skills) |
| **Eventhouse MCP** (`POST /v1/mcp/dataPlane/kqlEndpoint`) | Fabric's remote MCP server over one KQL database, at both endpoints: global (each call names `workspaceId` and `itemId`) and scoped to a single KQL database. The four tools the live server lists, `executeQuery`, `getSchema`, `getGeneralKQLExamples` and `getSpecificKQLExamples`, run as the signed-in user, who needs Read on the database (share-aware). `executeQuery` runs real KQL on the attached engine and answers a Kusto document whose rows are in `PrimaryResult`, capped at 1,000 silently; a failed query carries the cluster, the database and the engine's error. `getSchema` reads the real schema with row counts and sample rows. The grounding tools refuse an empty database (`Database is empty`); `clusterUrl` and `databaseName` redirect to another of the emulator's own databases. **From live captures, not Microsoft's docs:** Microsoft names no tools; the names, `serverInfo` `KustoMCP`, `executeQuery`'s schema and the result and error shapes are from two third parties' captures. **Ours:** the grounding documents and their ranking, which Fabric produces with Copilot. **Not modelled:** Azure Data Explorer clusters, learned examples. Witnessed by the unmodified Python `mcp` SDK in CI against kustainer, as two users ([07](07-control-plane-api.md#eventhouse-mcp)) | 🟢 Real (contract from two third-party live captures; grounding is ours) |
| Capacities (list, assign / unassign) | Full state, no billing/SKU enforcement | 🟢 Real state |
| **ARM-created capacities** (`FABRIC_ARM_URL`) | Opt-in poll of arm-emulator's `GET /_family/capacities`. ARM-created `Microsoft.Fabric/capacities` appear on Fabric `GET /v1/capacities` under the Fabric REST GUID ARM assigned at create. The seeded default stays. Empty `FABRIC_ARM_URL` is standalone (seed only). No job metering — SKU is a label | 🟢 Real (feed consume) |
| Long-running operations (202 → poll) | Clock-derived | 🟡 Emulated |
| **Asynchronous outcomes forced** (`FABRIC_FORCE_LRO`) | The reference documents **two** outcomes for this API — 200 with the body, and 202 + `Location`/`x-ms-operation-id`/`Retry-After` whose `/result` carries the definition — and **a real tenant answered 202** (measured 2026-08-11 on a VariableLibrary). The emulator only ever produced the 200, so nothing here exercised the half a tenant actually takes, and the failure it hides is the quiet kind: a client that reads the 202's body gets `null` and reports an **empty definition** rather than an error. Exactly that happened when this repo first called the real API. **`createItem` is the sharper case and the reason this became one switch rather than two**: Create Warehouse documents 201 and 202, a real tenant answered 202, and it "does not support create a warehouse with definition" — while the emulator went async only FOR a definition-bearing create, so the one type measured asynchronous was the one type guaranteed synchronous here. A client indexing the 201 body got `None["id"]` against a tenant. So the async outcome is now available as a **testing lever**, the same idea as `FABRIC_LIST_PAGE_SIZE`: set it and every dual-outcome API forces the operation loop on the spot. **Three surfaces audited and covered**: `getDefinition` (200/202), `createItem` (201/202) and `git/initializeConnection` (200/202) — the last carries a real result body, unlike `commitToGit`/`updateFromGit` whose LROs have none. **Scope stated rather than implied**, with the full audit in [49-async-outcome-audit.md](49-async-outcome-audit.md): these are the surfaces checked against the reference, not proof no fourth exists. The durable output of that audit is a searchable rule rather than a list — **every surface that answers 202 says so in its description, in the words "This API supports long running operations (LRO)"**, and every clean one does not, so the next sweep is a grep rather than a reading exercise. Six further surfaces were checked and found clean (`git/connect`, `git/disconnect`, bulk `admin/domains` assignment, and mirroring, whose LRO routes to the already-async `updateDefinition`), which says the three fixed were anomalies rather than the front of a queue; the already-async ones (updateDefinition, commit/update from git, capacity assign, identity provision, deploy, job instances) were confirmed and needed nothing, and two documented LROs are *not implemented at all* (Load Table, `sqlEndpoints/{id}/refreshMetadata`) so they are gaps rather than wrong shapes. **Off by default** — 200 is equally legal and is what most calls see; the point is that the other path is now reachable rather than hypothetical, and both are asserted to return the same bytes. **A real client now drives the forced half in CI** (`ci:fabric-cicd-lro`): the same fabric-cicd publish as the synchronous job, against an emulator started with the toggle, so the poll loop that follows `Location`, honours `Retry-After` and reads the operation's error body is Microsoft's own and knows nothing about this emulator. Four arms, because a green publish alone would not tell them apart — the toggle demonstrably reached THIS server (a raw definition-less `createItem` answers 202, and its `/result` carries the item rather than the operation); an operation that is genuinely `Running` rather than one that succeeds on the first poll, with `Location` staying on the state url until it moves to `/result`; the same delay traversed by fabric-cicd itself; and an **injected failure**, which is the arm that proves the client parses the emulator's own `errorCode` instead of treating any 202 as success. **Mutation-tested rather than asserted**: making a succeeded operation keep pointing at itself is caught by fabric-cicd on the *synchronous* job too (a definition-bearing create was always an LRO here), while making the toggle skip `createItem` is caught ONLY by the new leg — which is what makes the new leg evidence rather than repetition. Nothing diverged: the emulator's 202 satisfied a client that had never seen it | 🟢 Real (both documented outcomes, one witnessed by Microsoft's own client) |
| **Deleted display names held** (`FABRIC_NAME_RESERVATION`) | A tenant does not free a deleted item's display name at once: recreating a Notebook under the name of one just deleted answers **`409 ItemDisplayNameNotAvailableYet`**, *"is not available yet and is expected to become available in the upcoming minutes"*, with **`isRetriable: true`** (measured 2026-08-11). The emulator freed the name instantly, so a provision / teardown / re-provision loop passed here forever and failed on a real deploy — the permissive direction, which is the one that ships. **The distinguishing field is `isRetriable`, not the status**: a name held after a delete and a name taken by a live item are BOTH 409, and a client that treats them alike gives up on the one that would have succeeded seconds later. **A duration rather than a flag, because the tenant disagreed with itself** — the message says "the upcoming minutes" and the name was free again on a retry **20 seconds** later, so nothing observed licenses a constant and the window is the operator's to choose. **Off by default**, for `FABRIC_FORCE_LRO`'s reason: instant reuse is what a local create/delete loop wants, and the point is that the other behaviour is reachable at all before a tenant is the thing that finds out | 🟢 Real (shape and retriability; the window is policy) |
| Item **job execution** (`jobs/instances`) | **Five item types really execute, and their status follows the work rather than the clock.** Each parks `CompleteAt` beyond the clock's reach at submit — so no poll can observe a clock-completed job whose work is still running — and finalises when the work reports: **DataPipeline** runs the interpreter, **CopyJob** moves the bytes, **Notebook** and **SparkJobDefinition** are executed by the Spark agent when one is configured (without an agent they stay open for the documented external callback, which is the honest contract rather than a hang), and **Apache Airflow Job** is decided by a real Airflow sidecar. An item type with **no execution engine** — a Lakehouse, Warehouse, SemanticModel, Reflex and most of the ~50-type enumeration — still accepts the POST, because Fabric's job surface is generic across item types, and its status is then derived from the virtual clock (`NotStarted` → `InProgress` → `Completed` as `CompleteAt` passes) rather than from work. The lifecycle is faithfully shaped; nothing ran. That is the honest grade for a type with nothing to execute — inventing a work-derived status there would be the lie the notebook reconciliation was built to kill. Dataflow's refusal is the one outcome still reached at submit, because a 501 really is instantaneous. **Capacity admission is real state, not a broker:** a per-capacity concurrent-job ceiling (default 999) throttles Manual submits with `430 CapacityNotAvailable` + `Retry-After`, and queues scheduled / event-triggered jobs as `Queued` until a slot frees, FIFO, on the same clock and list levers that fire schedules. Same-item jobs both occupy slots when capacity allows — the Delta collision Fabric allows, not a mutex. [36-capacity-job-queueing.md](36-capacity-job-queueing.md) | 🟢 Real (types with an engine) / 🟡 clock-derived where there is none |
| **List Item Job Instances** (`GET …/jobs/instances`) | An item's runs, paged, newest first, with the same clock-derived status the single-instance read returns. Listing is itself an evaluation point for the scheduler and for capacity-queue drain, so a run that has come due — or a queued job whose slot has freed — appears without touching the control surface | 🟢 Real |
| **Item Job Scheduler** (`…/items/{id}/jobs/{jobType}/schedules`) | Fabric's *own* per-item scheduler — distinct from the `ApacheAirflowJob` item, which delegates to a real Airflow sidecar. All four `ScheduleConfig` members: `Cron` (`interval` 1–5,270,400 min), `Daily`/`Weekly` (`times[]` ≤ 100, `weekdays[]`), `Monthly` (`DayOfMonth` or ordinal weekday, `recurrence` 1–12) — every documented bound enforced at write time, the 20-per-item ceiling as `ScheduleExceedsLimit`, and `localTimeZoneId` (Windows ids or IANA) honoured as real local wall time, so a daily 09:00 stays 09:00 across a DST change. Schedules **really fire**: each due occurrence starts a job instance down the same path a manual run takes, so a scheduled DataPipeline executes the interpreter and only `invokeType: "Scheduled"` tells them apart. Evaluation is driven by the **controllable clock** rather than a background worker — advancing time materialises exactly the occurrences that came due, and a past `startDateTime` triggers instantly with no special case. *Boundary:* catch-up is capped at 100 occurrences per evaluation (newest kept), because a controllable clock can be advanced a year against a one-minute Cron. [07-control-plane-api.md](07-control-plane-api.md) | 🟢 Real |

## Identity & security (`security/`, `admin/`)

| Fabric feature | Emulator | Type |
|---|---|---|
| Entra OAuth2 tokens / JWKS / client-credentials | entra-emulator mints **real signed JWTs** | 🟢 Real |
| Workspace managed identity handshake | Provisioned via entra admin API; the identity's own token passes RBAC | 🟢 Real |
| Key Vault references in connections | Resolved against azure-keyvault-emulator | 🟢 Real |
| **Governance domains** (`/v1/admin/domains`) | Full admin surface: domain/subdomain CRUD (the hierarchy stops at two levels, as documented), workspace assignment (single-valued — re-assigning moves a workspace), `nonEmptyOnly` listing, and bulk `Admins`/`Contributors` role assignment. Deleting a domain cascades to its subdomains, assignments and roles. **Tenant-admin gated**, and graded as Microsoft documents it rather than uniformly: a READ (`admin/workspaces`, `admin/items`) requires *"a Fabric administrator **or** a service principal"*, while a WRITE (`admin/domains/create-domain` and every other mutation) requires *"a Fabric administrator"* with **no service-principal escape** — so a service principal may read the tenant and may not change it. Administrators are declared by the operator (`FABRIC_TENANT_ADMINS`), never inferred from a token claim; with none declared every mutation is refused, because the pre-gate behaviour was that everyone was an admin. Refusals are `InsufficientPrivileges`/403, the documented code | 🟢 Real (mgmt + tenant-admin gate) |
| **Activity log / audit** (`GET /v1.0/myorg/admin/activityevents`) | Real audit trail — the emulator **records events as operations happen**, using the documented audit vocabulary (`CreateWorkspace`, `CreateArtifact`/`UpdateArtifact`/`DeleteArtifact` from admin/operation-list.md; `InsertDataDomainAsAdmin` & co. with their `DataDomainObjectId`/`FoldersToSetCounter` properties from governance/domains-audit-schema.md). Enforces the documented request rules (single-quoted UTC bounds, same-day window) and pages with `continuationToken`/`continuationUri` until the token stops coming back. Nothing is synthesised at read time | 🟢 Real |
| **Tenant settings** (`GET /v1/admin/tenantsettings`) | The documented `TenantSetting` object in full — `settingName`, `title`, `enabled`, `canSpecifySecurityGroups`, `tenantSettingGroup`, the three `delegateTo*` flags, enabled/excluded security groups (`graphId`/`name`), and typed `properties` validated against the documented `TenantSettingPropertyType` enum. Optional arrays are **omitted, not null**, as the reference's sample does; seeded with the setting names that sample uses. `POST /v1/admin/tenantsettings/{name}/update` is the **real** update API (`enabled` required; response wrapped as `{"tenantSettings":[…]}`) | 🟢 Real |
| **Tenant-wide workspace admin** (`GET /v1/admin/workspaces`) | The documented `Workspace` shape — note it differs from the user-facing surface: the envelope key is `workspaces` (not `value`) and the field is `name` (not `displayName`). Filters `type`/`state`/`capacityId`/`name` are enforced, with undocumented enum values returning `BadRequest` as the reference specifies; `domainId` is reported from the real domain assignment. Emulator workspaces are always `Workspace`/`Active` (no soft delete), so `state=Deleted` is legitimately empty. **Tenant-admin gated on the read rule** — an administrator *or* a service principal, as this API's reference states | 🟢 Real (mgmt + tenant-admin gate) |
| **Tenant-wide item admin** (`GET /v1/admin/items`) | The documented `Item` shape across every workspace, with `workspaceId`/`capacityId`/`type`/`state` filters and the reference's own error codes (`InvalidItemType`, `InvalidItemState`). Envelope key is **`itemEntities`** — a third spelling after the user-facing `value` and admin workspaces' `workspaces`, which is why each came from its own reference page. `Active` is the only documented state, so that is all the emulator reports. Fields it does not model (`creatorPrincipal`, `defaultIdentity`, `tags`) are omitted rather than faked | 🟢 Real (mgmt + tenant-admin gate) |
| **Capacity tenant-setting overrides** (`GET /v1/admin/capacities/delegatedTenantSettingOverrides`, `POST …/{capacityId}/delegatedTenantSettingOverrides/{name}/update`) | The documented `CapacityTenantSetting` (a tenant setting plus `delegatedFrom` and `delegateToWorkspace`, and *without* `delegateToCapacity`/`delegateToDomain`). The update body is the documented one — `enabled` required, no `properties`, no `delegatedFrom` — and the response wraps as `{"overrides":[…]}`. An override may only be created for a setting whose `delegateToCapacity` is true. **Note there is no `/v1/admin/capacities` list API in Fabric**: capacities are listed on the Core surface at `/v1/capacities` | 🟢 Real (mgmt + tenant-admin gate) |
| **Sensitivity labels** (`bulkSetLabels` / `bulkRemoveLabels`) | The documented admin bulk APIs, reporting per-item `successfulItems`/`failedItems` rather than failing whole calls. Every change writes the documented `SensitivityLabelEventData` to the audit log — `SensitivityLabelApplied`/`Changed`/`Removed` with `SensitivityLabelId`, `OldSensitivityLabelId`, `ActionSource` 3 (Manual), `ActionSourceDetail` 5 (PublicAPI), `ArtifactType` 12, and a `LabelEventType` genuinely computed from label order (upgraded/downgraded/same-order/removed). **The label taxonomy is emulator-provided** — real Fabric gets labels and their order from Purview, which cannot be attached offline; `GET /v1/admin/labels` exposes it. Gated as writes (administrator only), but note what the gate does **not** model: these two APIs are documented `User: Yes / Service principal: No`, so real Fabric refuses a service principal *even when it is an administrator*. The gate models the administrator requirement, not Microsoft's per-API Entra identity-support matrix | 🟢 Real (label APIs + audit) / 🟡 taxonomy |
| **Sensitivity labels → catalog** | Labels are Purview *Information Protection* objects, not Atlas entities, so the Atlas-API route a Purview → OpenMetadata migration takes cannot carry them — assets, classifications, glossary and lineage cross; labels do not. The optional OpenMetadata profile exports them instead: the taxonomy becomes classification **`FabricSensitivity`** (`mutuallyExclusive`, since an item carries at most one label), each label a tag under it, and an item's `sensitivityLabel` is applied to that item's catalog entity. CI reads the tag back **through OpenMetadata's API**, on two items with different labels, and asserts that clearing the label in Fabric clears the tag in OM ([22-openmetadata.md](22-openmetadata.md)) | 🟢 Real (stored in OM) |
| **Purview Data Map** (`{endpoint}/datamap/api/atlas/v2/…`) — *split out of the old single "Purview scanning / classification" row, because its three parts have different **ceilings**: this one is reachable and moving, scanning is reachable in-family, and the built-in classifiers can never move. One 🔴 made them look equally stuck, and became actively wrong the moment this landed* | The Data Map **is Apache Atlas v2** — the Azure spec's own TypeSpec declares `@route("/atlas/v2/entity")`, `/types`, `/relationship`, `/glossary`, `/lineage` and annotates them *"This is Atlas API"* — so this is Atlas semantics, not a Purview-shaped façade. Implemented: the **type system** (typedef CRUD, per-category reads, `typedefs/headers`) and **entities** (createOrUpdate, read/delete by GUID, `/entity/bulk`, lookup by unique attribute), on the documented `purview.azure.net` audience with the documented `AtlasErrorResponse` codes. What is enforced rather than shaped: an entity naming an unregistered type is refused; required attributes are checked **through the supertype chain**, which is how `qualifiedName` — declared once on `Referenceable` — is required of every type; `qualifiedName` is the per-type identity, so createOrUpdate genuinely updates instead of duplicating; Atlas's **negative-GUID placeholder protocol** is honoured and reported in `guidAssignments`; a batch validates fully before it writes anything; deletes are soft, as the spec's `EntityStatus` states. The four Atlas base types are seeded, as a real account has them. **Scope, explicitly: 96 routes in the spec, and this is the type system and entities only** — glossary, lineage, relationships, classifications, business metadata and search are NOT implemented. 🟡 rather than 🟢 for one reason: the row's evidence is 14 Go tests, and this repo's rule is that green needs a **real-client witness in CI**. `pyapacheatlas` — an off-the-shelf Apache Atlas v2 client, not a client shaped like this implementation — now drives these routes in CI (`e2e/purview-datamap`), which is what moved the grade. **The green covers the type system and entities ONLY**; the witness is deliberately scoped to them, because a suite extended to the unimplemented routes would fail honestly and a suite extended until it passed would not be evidence | 🟢 Real (type system + entities only; witnessed by `pyapacheatlas`) |
| Purview **scanning** (`scan/`) | Not implemented. A scan is an **engine**, not a surface: it connects to a source, enumerates assets, infers schema and applies rules over real values. Against the emulator's own sources — OneLake Delta, the warehouse — that is genuinely computable, so this row has a **reachable ceiling of 🟢 in-family / 🔴 external**, which is why it is not bundled with the row below | 🔴 Not implemented |
| Purview **system classifiers** (the ~200 built-ins) | Not implemented, and **this one can never move**. The built-in classifiers are proprietary detection patterns published in no OpenAPI spec and in no documentation; custom classification rules are implementable, the shipped set is not. Split out from scanning precisely because its ceiling is different: scanning is blocked by work, this is blocked by information nobody outside Microsoft has | 🔴 Not implemented (no reachable ceiling) |
| **Lineage** (catalog graph) | Via the optional OpenMetadata profile: OneLake **shortcut** edges and executed pipeline **Copy** source→sink edges are persisted exactly and witnessed in OM's graph API. Notebook edges come from the engine's own report of what it read and wrote, or from the data plane observing it; Script/stored-procedure code is still not guessed. Catalog SSO can also be pointed at entra-emulator ([22-openmetadata.md](22-openmetadata.md)) | 🟢 Real (shortcuts + Copy) |

## OneLake (`onelake/`)

| Fabric feature | Emulator | Type |
|---|---|---|
| ADLS Gen2 DFS surface (create → append → flush, ranged read, list) | Full, incl. the `x-ms-range` dialect | 🟢 Real (real bytes) |
| Blob surface | Full | 🟢 Real |
| Delta commits (put-if-absent atomicity) | Real; `-race`-tested concurrent-commit race | 🟢 Real |
| **Item permissions** — sharing one item without the workspace | **Granted, validated and reported; not yet enforced.** Effective access is the workspace role's implied permissions unioned with a direct grant — so revoking a grant leaves what the role gives, as the product documents. Semantic models are shared through **Power BI's documented dataset-users API** (`GET/POST/PUT …/datasets/{id}/users`, and the in-group spellings), with its documented rules: Post needs ReadReshare, Get and Put need ReadWriteReshare, Write can be neither added nor removed, an App cannot be a target, `None` removes. Every other item is shared through an **authenticated emulator-native** `…/items/{iid}/_emulator/access`, because Fabric's Core REST has no item-permissions operation and the portal uses calls Microsoft does not document — not under the unauthenticated `/_emulator/` prefix, where anyone could grant themselves ReadAll. The documented **admin list** `…/admin/workspaces/{wid}/items/{iid}/users` reports effective access, which is **inferred**: no sample shows an inherited row. A grantor shares at most what they hold; a UPN is refused by name, since there is no directory to resolve it; a group is stored but matches nobody. **Enforced on OneLake**: a ReadAll grant admits a Viewer, or a principal with no workspace role, to that item's data on DFS and Blob — reading and listing inside it, never enumerating the workspace — and `principalAccess` answers for grantees too. All of it comes from one shared decision, `store.OneLakeReadAccess`, so no two surfaces can disagree about the same principal. When the item has OneLake security roles, the roles decide and ReadAll reads through `DefaultReader`. **Enforced on Direct Lake and `executeQueries`**: Build is required to query, and a grant on a Direct Lake source is read through the same decision. **Enforced at the SQL endpoint**, through the real relay against a real SQL Server: Read connects, ReadData reads, sharing never writes — and a revoke **stops working**, including by three-part name from another granted database, because no access means `REVOKE CONNECT` there, which also defeats an explicit T-SQL `GRANT` an owner once authored. Witnessed grant-and-revoke on every surface, and mutation-checked: with memberships only ever added, or with CONNECT left in place, the SQL witnesses fail. The admin list's inclusion of inherited access remains an **inference**. **A limit a real client found:** Microsoft's own `fab acl ls <item>` (1.2.0, the locked version) reads that admin list and then fails inside fab on `principal.displayName` — a field Microsoft's spec marks *optional* (`Principal` requires only `id` and `type`) and that fab evidently relies on real Fabric sending (unconfirmed without a tenant). This emulator has no directory to source a name from, and omits a field it cannot source rather than inventing one, so the response is spec-conformant and unusable by that client. Asserted as a failure in `e2e/fabric-cli`, so it cannot close unnoticed. [57](57-item-permissions.md) | 🟢 Real (the admin list's inherited rows inferred; `fab acl ls <item>` cannot read it) |
| **OneLake security roles** (`dataAccessRoles`) | Real: roles are stored, evaluated deny-by-default, and a role WIDENS a Viewer without narrowing Contributor and above. `DefaultReader`'s virtual membership (`fabricItemMembers`) is modelled, because an evaluator with only explicit Entra members cannot express it — matched on the principal's **real item access**, and only when the member's `sourcePath` names this item: it was ignored until 2026-09-13, so a role written for holders of ReadAll on *another* item admitted ReadAll holders here. DENY rules are refused rather than silently ignored, as the product refuses them. **Row and column security are read from the documented `constraints` object, keyed by `tablePath`** — and the decision rule is decoded **strictly**: a restriction the parser cannot read with certainty (an unknown field, the old flat `rows`/`columns`, an empty column list, a non-`Read` column action, two constraints on one table) drops the rule rather than granting past it. Until 2026-09-13 flat fields were read instead, so a policy authored the way the REST reference documents it was enforced as **no policy** — and every enforcement witness wrote the flat dialect, so none could see it ([54](54-onelake-security.md#the-authoring-payload-was-the-wrong-shape-and-every-witness-spoke-it)). Dropping a rule is always safe: a constraint narrows only the rule it sits in. Witnessed with the reference's own sample payloads. **The item type is checked on write**: `Lakehouse`, `MirroredDatabase` and `MirroredAzureDatabricksCatalog` carry roles and everything else is `DataAccessRolesNotSupported` — a **Warehouse** most of all, since its data is secured by T-SQL and a role stored there would be one the emulator honours and a tenant ignores. Two enum members the supported-items table does not name by spelling (`MirroredWarehouse`, `MirroredCatalog`) are refused rather than admitted on inference: a wrong refusal comes back as a bug report naming the type, a wrong acceptance as a policy that did not hold in production. The GET is **not** gated, because the docs do not say what a read against an unsupported item returns; that asymmetry is pinned by a test rather than left to drift. **Unsupported role combinations are refused, in the one shared decision.** "OneLake security doesn't support the combination of two or more roles where one contains RLS rules and another contains CLS rules … users … receive query errors": a principal whose row filter and column restriction on a table come from different roles is marked on that table by `onelakesec.Effective`, where the union used to open both and grant everything. Direct Lake and direct DFS/Blob reads raise the error by name; `principalAccess`, whose response has no field for an error, gives an engine **both** restrictions, never more than either role grants; the SQL analytics endpoint shows the reader no rows. Judged per table — inferred from "tables that are part of an unsupported role combination". Both kinds in one role remain supported. | 🟢 Real |
| **Row- and column-level security, applied by the engine** | Real for Spark SQL and the catalog-resolved DataFrame readers: two callers get different rows and different columns from one `SELECT`. **Constraints are per table**: one rule can grant `*` and filter only `Tables/sales`, and each consolidated entry carries its own union, so the most specific entry decides — the Go evaluator and the Spark agent now read it the same way, where they had quietly disagreed on overlapping grants. The filter replaces the RELATION rather than rewriting user SQL, and every qualified registration of a secured table is swept, so `default.sales` cannot walk around it ([measured](54-onelake-security.md)). **Refuses** where the engine's catalog is shared, because reshaping one there would take the table from everyone. Under the two-context split the same filtering is done by the privileged half instead, and witnessed separately. | 🟢 Real |
| **Two-context execution** — the user context holds no service credential | Real. A statement against an item with OneLake security roles runs in a child process carrying a token minted for the CALLER, with the agent's storage bearer and `ENTRA_CLIENT_SECRET` scrubbed from its environment; the privileged half reads the Delta files, applies the row filter and column projection, and hands over only the result. A table no rule grants is never registered there at all. **Enabled per item, not globally** (`FABRIC_TWO_CONTEXT`): the split exists to enforce policy, and where there is no policy there is nothing to enforce. [docs/54](54-onelake-security.md) | 🟢 Real |
| **Direct path access blocked for a narrowed principal** | Real, and enforced by OneLake rather than by the engine. RLS and CLS cannot be applied to bytes, so a principal who may see only part of a table cannot fetch its files — on the DFS and Blob surfaces alike, and **from a notebook cell**: a secured session runs on an engine of its own carrying that caller's token, so `spark.read.load("abfss://…")` arrives as the caller and is refused 403. That is Fabric's own mechanism, not a patched reader. Sharing one engine across callers is what used to defeat it, and Fabric shares one only within a [single-user boundary](54-onelake-security.md) too. | 🟢 Real |
| Shortcuts (OneLake → OneLake) | Symlinks with target-side RBAC (trusted-workspace-access) | 🟢 Real |
| Shortcuts to external targets (S3 / ADLS Gen2 / Dataverse) | **Real Amazon S3 / S3-compatible read-through**: a Connection carrying the Access Key Id + Secret Access Key pair — as `Basic` credentials, because Fabric's S3 connector uses authentication kind "Access Key" while the REST reference's `CredentialType` enum has no `AccessKey` member and `Basic` is its only two-secret type — makes the emulator sign upstream requests with **AWS SigV4** (`internal/awssig`, verified against AWS's own published example signature). Witnessed by `e2e/s3` against a real **SeaweedFS** server started with an identity config — the suite proves an unsigned GET is 403 and that a *wrong* secret is refused, so the pass cannot be vacuous; the object itself is written by **boto3**. **ADLS Gen2 reads are witnessed too**: `e2e/azurite-shortcut` drives a real SAS through the shortcut against **Azurite**, Microsoft's own storage emulator, proving an unauthenticated GET is 403 and a *tampered* SAS is refused. Scope stated honestly — Azurite implements Blob/Queue/Table and **not** ADLS Gen2, so the endpoint is the Blob one; that witnesses the read path (a plain authenticated GET, identical on both endpoints of a real ADLS Gen2 account) but **not** DFS-specific behaviour, which stays unwitnessed offline. **Dataverse is now a real target type**, at a scope worth stating precisely: Microsoft documents the target's four fields (`connectionId`, `deltaLakeFolder`, `environmentDomain`, `tableName` — the only documented target with no `location`) and documents that the tables must already exist in the Dataverse Managed Lake. It does **not** document that lake's byte layout, so the emulator serves ordinary Delta from `environmentDomain/deltaLakeFolder/tableName` and **that composition is ours, not Microsoft's**. This is a target type in the existing shortcut machinery, *not* a Dataverse emulator: no Dataverse Web API, no OData metadata, no table discovery. The **read-only rule is real and enforced**: Fabric states twice that "Dataverse shortcuts are read-only. They don't support write operations regardless of the user's permissions", and both the flush and the delete paths refuse, with `e2e/azurite-shortcut` confirming via the Azure SDK that the refused write never reached the target and the refused delete left it intact. Auth diverges and is disclosed: Fabric's delegated model is organizational account (OAuth2) or service principal, while the emulator presents whatever the Connection carries — the witness uses a SAS, so what is proven is that the credential is really presented and really checked, not that OAuth2 delegation works. **S3 read-only is by design** — Fabric documents S3 shortcuts as read-only regardless of permissions, so a write there is refused locally and never reaches the target. **ADLS Gen2 write-through is real**: a file flushed under an ADLS shortcut is PUT to the storage account and the local buffer dropped (it would otherwise shadow later target changes), and deleting a file within a shortcut deletes it at the target, as documented. Witnessed by `e2e/azurite-shortcut`, which reads the result back **with the Azure SDK straight from Azurite** rather than trusting the emulator's own 200 | 🟢 Real (S3 SigV4 + ADLS reads) / 🟠 Dataverse (opt-in layout) |

## Data Engineering (`data-engineering/`)

> Engine-level claims below are measured, not asserted: the generated
> [Spark engine matrix](engine-matrix.md) probes each capability against both
> Sail and the JVM overlay and is regenerated in CI.

| Fabric feature | Emulator | Type |
|---|---|---|
| Lakehouse item + Tables/Files storage | Full (via OneLake) | 🟢 Real |
| Notebook authoring / definition round-trip | Full | 🟢 Real |
| `notebookutils` / `mssparkutils` — **the whole documented surface** | All **eight** namespaces Microsoft's overview table lists — `fs`, `notebook`, `credentials`, `lakehouse`, `runtime`, `session`, `udf`, `variableLibrary` — with **all 44 documented members** present, each carrying the documented parameter names in the documented order. The module list is Microsoft's, not ours, so it can name a module nobody here thought to look for. **A second Microsoft source now sits beside the transcription**: `third_party/notebookutils-stubs/` pins the `dummy-notebookutils` wheel Microsoft publishes, and `check_notebookutils_surface.py` holds both — the docs decide behaviour, the stub decides what exists, and where they disagree the divergence is COMPUTED rather than declared. Holding them together turned "we transcribed carefully" into a list of 15 gaps, all now closed: the whole `help()` family (Fabric's `fs` page opens by documenting `notebookutils.fs.help()` as the discovery mechanism, and it was on no module — prose on the page rather than a row in the method table, so transcription could not yield it), `getHelpString`, `runtime.getCurrentWorkspaceId`, `fs.nbResPath` and `fs.refreshMounts`, `lakehouse.getDefinition`/`updateDefinition`, and `udf.run`. **Two of those gaps had gone stale before anyone reread them** — `fs.nbResPath` was waived as "Axis C lists it as absent" after Axis C landed it, and `udf.run` as "the UDF item has no engine here" after `getFunctions` began running the item's own code, which is why the check fails on a waiver that is no longer a gap. Graded on every conformance run by contract 2 on both lakehouse backends. Getting there fixed **25 defects**: 17 members absent, and 8 that existed and would still have been declined because the parameter NAMES differed (`dst` for `dest`, `recursive` for `recurse`, `maxBytes` for `max_bytes`) — a framework introspects a signature and refuses to start without ever calling it. Plan and measurements in [56-notebook-capability-parity-plan.md](56-notebook-capability-parity-plan.md). **Shape is half the claim; behaviour is the other half.** All 44 members are also EXERCISED end to end — 42 by `ci:notebookutils`, and `session.stop`/`session.restartPython` by `ci:livy-native`, which is the only harness shaped right for them: a Livy session is a persistent REPL, so one statement can restart the session and the NEXT one can report what survived. Each confirms out of band — the call that acted is never the call that answers. Giving the last members an e2e found **four defects that shape could never have caught**: `fs.rm` deleted a non-empty directory without `recurse=True` (ADLS answers 409 `DirectoryNotEmpty`, so the emulator was more destructive than the thing it emulates); the LRO `Location` header hardcoded `https://`, so a client following it — the documented way to reach a result — could not reach an emulator started with `-disable-tls`; `session.stop`/`restartPython` raised *"running outside a notebook session"* **inside** one, because nothing in the tree ever set the environment variables they read, and in a secured session the identity never crossed into the user-context child; and `restartPython` left the session without `display`, `displayHTML` and `%run`, which are kernel builtins on Fabric and cannot be lost to a Python restart | 🟢 Real |
| **`notebookutils.notebook` management** (`create`, `get`, `getDefinition`, `update`, `updateDefinition`, `delete`, `list`) | The seven documented CRUD methods, with the documented parameter names in the documented order. **Added because framework-conformance contract 2 measured all seven ABSENT** ([38-framework-conformance.md](38-framework-conformance.md#2-the-api-shape-is-the-contract-independent-of-behaviour)) — and absence is what a CI/CD framework reads before it will start, without ever calling one. `create` takes `.ipynb` as Fabric documents it, and the executable `notebook-content.py` is derived **server-side** so the created item can actually run: before that, a documented create returned 201 and its `RunNotebook` job then died on `notebook-content.py is missing`. The 200-or-202 outcome is followed rather than read as a body, which is what `FABRIC_FORCE_LRO` exists to force a client to prove | 🟢 Real |
| `notebookutils.notebook.runMultiple` — DAG structure and ordering | Runs a DAG in dependency order, accepting a bare list or Fabric's `{activities, timeoutInSeconds, concurrency}` shape. A failed activity's dependents are reported skipped with the reason rather than run. A cycle, a dependency naming an activity outside the DAG, a duplicate activity name, or a malformed shape is refused. `validateDAG` exposes those same checks ahead of a run, and `workspace` accepts a name or an id | 🟢 Real |
| `notebookutils.notebook.run` — exit values | Returns the child's exit value, the exact string passed to `notebookutils.notebook.exit(...)`, and `""` when the child never called it. Applies to `runMultiple`'s per-activity `exitVal` too. Until 0.18.0 this returned the terminal job status, so a parent branching on a child's exit value took one path here and the other on Fabric | 🟢 Real |
| **`notebookutils.notebook.run` — exit values OVER REST** | **A shim limitation on the real target, NOT an emulator divergence — the emulator is the side matching Fabric here.** In Fabric you call `run()` inside a notebook kernel, which has an in-process channel to the child's exit value; that contract is the row above and the emulator implements it. This shim also offers `run()` from OUTSIDE, over REST, and there the value is read from `…/jobs/instances/{jid}/notebookRun` — the emulator's own endpoint, paired with the `notebookRunResult` engine callback. **Real Fabric answers it `404 EntityNotFound`** (measured 2026-08-11), so a REST-submitted run cannot carry an exit value at all. Reported as `None` — *could not be obtained* — rather than `""`, which claims the notebook exited with no value and is what this returned until 2026-08-11, so every run on a tenant looked like a deliberate empty exit. Both are falsy, so `or ""`/`or 0` are unaffected. The shim now WARNS on the real target naming the two paths that work: call `run()` inside a notebook, or have the child write a one-row Delta table. Fixing this by making the emulator report `None` would move it AWAY from Fabric's documented contract | 🟡 REST path only; notebook path 🟢 |
| `runMultiple` — failure contract | Raises `RunMultipleFailedException` (importable from `notebookutils.common.exceptions`) when any activity did not complete, carrying every result on `.result`. Each value carries Fabric's `exitVal` and `exception` keys; `status`/`error` ride alongside as emulator extras for local debugging and nothing should depend on them | 🟢 Real |
| **Materialized lake views** | **The refresh is real**: the view's own query runs on the engine the emulator already hosts and the result is written as a **Delta table under `Tables/<name>`**, so a Delta reader, the SQL analytics endpoint, the flow stream and lineage all see a refresh exactly as they see any other write. A refresh is only reported successful if **a Delta commit actually landed** — checked against OneLake rather than inferred from the statement's exit code, because a cheerful statement that wrote nothing is precisely how a view whose rows do not exist would be reported `Materialized`. **Staleness is measured, not assumed**: the declared sources' Delta versions are recorded at refresh time (read **before** the query, so a write landing mid-refresh cannot be counted as data the view read — forced in the witness by advancing a source from inside the engine fake) and compared with what those tables are now; the answer names *which* source moved. A view that has never refreshed is `NeverRefreshed`, not stale — there is nothing there to be out of date. *Emulator-native definition surface, and deliberately:* Fabric defines these with Spark SQL DDL that no capture here has observed, and this repo does not invent a syntax — so the definition hangs off `…/lakehouses/{id}/materializedlakeviews`, on the same precedent as the Reflex trigger binding, and `dependsOn` is **declared** rather than parsed out of the SQL (a wrong parse would not fail, it would silently report a stale view as fresh). Dropping a definition leaves the materialised table: the rows are real data a reader may still be using. When a DDL capture arrives it becomes a second front door onto this model, not a rewrite | 🟢 Real (refresh + staleness) / 🟠 definition surface emulator-native |
| Reference-run lakehouse rule | A referenced child notebook bound to a **different** default lakehouse than its parent is blocked, matching Fabric; a child that declares none inherits, a matching one runs, and `useRootDefaultLakehouse` in the arguments bypasses the check. The failure names the cause and the way out rather than "The job failed." | 🟢 Real |
| `runMultiple` — retry and timeouts | Per-activity `retry`/`retryIntervalInSeconds` (last error reported, no trailing wait after a final attempt) and the DAG-level `timeoutInSeconds`, defaulting to Fabric's 12 hours. `timeoutPerCellInSeconds` is applied per **cell**, multiplied by the notebook's real cell count **where that count is readable** — it comes from `…/notebookRun`, which real Fabric does not serve, so on a tenant the shim falls back to a whole-notebook ceiling instead. It previously assumed **one** cell there, which handed a twelve-cell notebook a one-cell deadline and produced a spurious timeout for a run that was working | 🟢 Real |
| `runMultiple` — concurrency | An explicit `concurrency` is honoured with a bounded pool per dependency level, `0` meaning unlimited; a level boundary stays a barrier and results keep their listed order regardless of completion order. **The default diverges deliberately**: Fabric defaults to 3× the CPU count, the emulator to sequential, because a harness comparing two runs needs the same sequence more than it needs speed | 🟡 Real, sequential by default |
| `runMultiple` — child Spark sessions | Fabric runs children on isolated REPL instances **within the parent's Spark session**, so they share its compute and its session-scoped state. The emulator gives each child its own session, so sibling activities cannot see one another's temp views. Deliberate: the benefit is narrow and the change is large. Scoped in [39-run-multiple-parity-plan.md](39-run-multiple-parity-plan.md) | 🟡 Isolated sessions |
| Spark session / statement / batch via the **Livy API** | **Native termination** (`--spark-agent-url`): the emulator implements the Livy contract and drives a persistent statement-executor agent. The default agent is a PySpark Connect client of Sail, not Apache Spark. An external Livy backend remains configurable with `--spark-livy-url` | 🟢 Real (default engine) |
| Notebook **cell execution** | The emulator parses and records the notebook run, resolves the attached lakehouse and **applies a bound Environment to the session before the first cell runs**, and Sail executes cells by default. The same fixture runs on Spark 3.5 JVM and proves unqualified `saveAsTable`/`spark.table` bind to OneLake `Tables/` | 🟢 Real (default engine; per-capability detail in [engine-matrix.md](engine-matrix.md)) |
| Notebook **cell languages** (`%%sql`, `%%pyspark`, `%%configure`, `%%html`, `%%markdown`, `%%spark`) | A cell's language now decides what happens to it. Before this the parser RECOGNISED the magic and the run loop sent everything that was not `sql` to the **Python executor** — so correct Scala came back as a Python `SyntaxError` pointing at the user's own code, and a `%%configure` block of JSON failed the same way. Four dispositions: run, render, ignore, refuse. `%%html`/`%%markdown` are content and are rendered, never executed. **`%%configure` is accepted and IGNORED, out loud** — the cell records that it changed nothing and that the requested executors, memory and conf were not applied; refusing would be worse, since it must be the first cell on Fabric and the results it would produce are correct, just not on the requested hardware. Scala, R and C# are **named**, with what to do instead and explicitly "Real Fabric runs it", so the message cannot be read as "your cell is invalid". An unknown magic still falls back to Python, because Fabric adds magics and refusing every unheard-of one would break notebooks on upgrade. **Witnessed through a real RunNotebook job** (`ci:conformance-sail`), not only by unit tests on the classifier: the defect was never in `Disposition()` — the parser always classified the magic correctly and the RUN LOOP ignored the answer — so a test on the classifier passes on both sides of the bug. Every cell in the probe is chosen to be INVALID PYTHON (`%%configure`'s `false`, the `<h1>` markup), so a regression cannot pass quietly; reintroducing the old behaviour was measured, and it fails with the original signature, `NameError: name 'false' is not defined`. The Scala leg asserts the refusal NAMES the language and that the run STOPPED — proven by the absence of a marker the next cell would have written, read over DFS rather than taken from the job record | 🟢 Real (Python and SQL execute; other languages refuse by name) |
| **`%run`** | Splices another notebook's code into **this** session, so what it defines is usable on the next line — the whole difference from `notebookutils.notebook.run`, which starts a separate session and returns an exit value. A line magic inside an ordinary Python cell, so it is a source rewrite applied before `ast.parse`: a `%run` inside a string or a comment is left alone, and source with no `%run` comes back byte-identical. Parameters bind before the referenced code runs, nested `%run` expands, and a non-Python cell in the reference is skipped rather than spliced. **Proven through the real agent** — the only harness that can, since the notebookutils e2e runs its notebook as a plain script and `e2e/notebook-run`'s runner is itself the engine | 🟢 Real `ci:conformance-sail` |
| **`display()` / `displayHTML()`** | Notebook BUILTINS, bound in the session namespace — a notebook writes `display(df)` with nothing above it. They were **absent entirely**, so one of the most common lines in any Fabric notebook raised `NameError`. `summary=True` is honoured rather than ignored, because Fabric documents it as column name, type, unique values and missing values — a data QUALITY read a notebook branches on, so there IS something to switch. **Under a kernel it publishes a MIME bundle** — `text/html` plus a `text/plain` alternative, both built from one description of the data so they cannot drift; without one (the agent's RunNotebook path) it still prints, which is what every stdout-reading suite asserts on. "Nothing here to render into" was true of the agent and false of the repository: the `jupyter` profile ships a real JupyterLab against this same stack, and its kernel now binds Fabric's `display` rather than IPython's — the two are not interchangeable, and `display(df, summary=True)` is a TypeError to IPython's. **Witnessed by `ci:notebook-display`**, which runs nbclient against the kernel from that same image and reads the notebook's own outputs: a stdout reader cannot tell a published bundle from a print, which is how this row stayed unevidenced. Removing the bundle was measured — the harness reports the `stream` output it fell back to. Column names and cell values are escaped, so a value holding `<script>` renders as characters. **The interactivity is still NOT emulated**: Fabric's `display` is a proprietary widget with chart views, sorting and an inspect panel, a correct HTML table is not that widget, and no local front end can show it is | 🟡 Data, shape and rich output; not Fabric's widget |
| **Notebook resources** (`notebookutils.nbResPath`, `builtin/`) | The `builtin/` folder that travels with a notebook, materialised locally so ordinary file I/O reaches it — which is how the docs describe using it. **`builtin/` is the ROOT notebook's folder, not the current one**: in a reference run a child resolving to its own would see different files depending on how it was started. A value, not a call, resolved on first read so importing the package costs no round-trip. **Witnessed through a real reference run** (`ci:conformance-sail`), and writing that witness found the root semantics had never worked: **nothing in the tree ever sent a `rootNotebookId`**, so a referenced child resolved `builtin/` to its OWN folder — the exact divergence the module was written to prevent, while its unit test supplied the context the product never produced. `notebookutils.notebook.run`/`runMultiple` now forward the root — **forward, not replace**, so a parent that is itself a child passes on the root it was given rather than becoming one. The probe's negative control is the assertion: both notebooks ship a `builtin/data.txt` with different content, so the child's answer names which folder it resolved against. Removing the fix was measured — the probe reports *the child resolved builtin/ to ITS OWN folder* | 🟢 Real |
| Livy **High-Concurrency** (5-REPL) sessions | Fabric's packing layer is implemented directly: `sessionTag` packing, 5-REPL cap + spill, independent lifecycle and slot reuse. Statements use Sail by default, so engine compatibility is limited to the Spark Connect subset | 🟢 Real (default engine) |
| Environments | **Applied to the session, not just reported.** A session carries an `environmentId`; the item's `requirements.txt`, Spark properties and JAR declarations are resolved and handed to the agent, which **installs the packages** and applies `sparkConfig` before the first statement runs. Witnessed by `e2e/environment` (CI job `environment`), which imports a package the runtime image does not have — and asserts the same import **fails** without the Environment, so a package that happened to be in the image cannot make it pass. **Both binding paths are witnessed, each with that negative half**: a Livy session binding by id, and a **notebook** binding in its own `# META` dependencies, which is how most Fabric data engineering does it. The notebook path was resolved-and-reported but never applied until 0.28.0: `resolveComputeBinding` put the Environment on the run and only the Livy session and the Spark Job Definition driver acted on it, so a notebook's first import failed while the run detail said the environment had been honoured. That is why this row now names the path rather than saying "a session". JARs are **reported as skipped**: a Spark Connect session's classpath is fixed at engine start. One Environment per agent process — a second bind of the same one is a no-op and a **different one is refused with a reason**, because Fabric isolates per container and a single process cannot; letting the last bind win would corrupt a dependency tree. `/opt/wheels` remains the fallback for consumers who model no Environment. [37-runtime-fidelity-gaps.md](37-runtime-fidelity-gaps.md) | 🟢 Real (packages + Spark config; JARs out of scope) |
| Spark Job Definitions | **The emulator runs the job**, as Fabric's pool does: V1 definition/main/arguments/libraries parsing, attached lakehouse + Environment resolution, then execution on the configured Spark agent with `sys.argv` = main file + the definition's arguments, and the real outcome finalising the job and publishing its terminal event. Previously the emulator resolved the job and *waited* — only a client that fetched the source and ran it itself ever executed one. A JAR-bearing Environment is **refused before any user code runs** on an engine with no JVM (asked of the engine, not assumed), and accepted on the JVM overlay. With **no agent configured** the original callback contract stands unchanged: the job stays open for an external engine to report. Both halves are witnessed where each is real: **`e2e/livy`** runs with a Spark agent, so the emulator IS the pool there — it publishes a definition, submits the job and polls, and `sjd-result=5` proves the engine read the bound lakehouse (3 rows) and received the definition's arguments (2). **`e2e/notebook-run`** deliberately runs with **no** agent, so it remains the witness for the callback contract, with its runner playing the external engine | 🟢 Real (emulator-executed) |

### Notebook code on the default engine (LakeSail's Sail)

The engine behind the agent is [Sail](20-lakesail-engine.md) (Rust
Spark-Connect, no JVM). Every row is **probed in CI** (`e2e/sail`), not
inferred — the fidelity deltas a Fabric notebook author actually hits.

A Spark 3.5.5 JVM image (`docker/spark-runtime`, Fabric Runtime 1.3's engine
baseline: Spark 3.5.5 + Delta 3.2 + Java 11 + Scala 2.12) exists as a **CI compatibility oracle** (`e2e/spark-jvm`), and the
statement agent still has its classic-session path. It is also exposed as a
**user-facing overlay** — `docker compose -f docker-compose.yml -f
docker-compose.override.yml -f docker-compose.spark-jvm.yml up` swaps the
statement agent onto it — so the JVM-only rows below are graded 🟠
(🟠 non-default engine): real with that overlay attached, unavailable on the default
engine. Verified live, not inferred: `sc.parallelize([1,2,3,4]).map(x*2)
.sum()` returns 20 through the Livy agent, and `spark._jvm.io.delta.tables
.DeltaTable` resolves.

| Notebook pattern | Emulator (Sail) | Type |
|---|---|---|
| `abfss://…@onelake.dfs.fabric.microsoft.com/…` production paths | Work unmodified (endpoint override routes the Hadoop URL form) | 🟢 Real |
| Delta write/read/append; SQL over temp views | Full | 🟢 Real |
| Time travel — DataFrame option **and** SQL `VERSION AS OF` | Both forms work on Sail. The SQL form was previously graded 🔴 on the strength of an assertion; the generated [engine matrix](engine-matrix.md) probes it against both engines and it passes on each | 🟢 Real |
| `MERGE INTO` | Registered table on `az://` via Sail (`e2e/sail`). Subquery source, `INSERT *`, and path-addressed ``delta.`path` `` targets on the emulator via delta-rs (middle column of [engine-matrix.md](engine-matrix.md)). Bare Sail still cannot plan a local-path MERGE. `WHEN MATCHED THEN DELETE` is not intercepted | 🟢 Real |
| `createDataFrame(local_rows)` | Works. The pinned engine needs nothing preset; on one that serves `localRelationSizeLimit` unparseably the agent rewrites it to the same size, never to a smaller constant | 🟢 Real |
| `sc` / RDD API / `spark._jvm` | **The measured subset now WORKS on the default engine.** Spark Connect exposes no SparkContext or JVM bridge on any engine — Apache's own Connect server included — so the agent binds a facade covering exactly what Fabric code was measured to call ([50-rdd-usage-capture.md](50-rdd-usage-capture.md) swept four corpora and found the whole surface in use is **logging plus one smoke idiom**): `sc.setLogLevel`, `sc.parallelize(...)` with `map`/`filter`/`collect`/`count`/`sum`/`toDF`, and `spark._jvm.org.apache.log4j`. Bound on BOTH spellings — the bare `sc` global and `spark.sparkContext`, which is the one Microsoft's own samples use. **Eager and LOCAL, with a stated ceiling**: `parallelize` refuses above 10,000 elements rather than pretend to distribute, and every unmeasured API (`textFile`, partition-level ops, accumulators, broadcasts, `java.*`) raises a pointer to the JVM overlay. **The expectations are not self-asserted**: `spark_agent/rdd_contract.py` holds the idioms and their answers, the unit tests assert them against the facade, and `e2e/spark-jvm` evaluates the same snippets against a real Apache Spark 3.5 JVM. Full RDD semantics remain overlay-only | 🟡 measured subset / 🟠 full API on the JVM overlay |
| DML row-count envelopes (`INSERT`/`MERGE` counts) | The engine's row count reaches the client: DataFusion reports it as `uint64`, which the Arrow conversion to the Connect client rejects, and the agent used to absorb that failure as an empty envelope — "wrote 3 rows" indistinguishable from "wrote nothing". It now recovers the count from the statement's **cached result relation**, which does not re-run the statement (measured: a 3-row INSERT stays 3 rows through recovery, and the e2e asserts the count-back for exactly that reason) | 🟢 Real (recovered count) |
| **Structured streaming** | `readStream` plans. Durable sinks (`delta` / `parquet` / `memory`) on the Livy path pull one micro-batch via Sail `limit(n).collect()` and batch-write, announced on stderr, no checkpoint — the same class of wrap as CDF / JSON `multiLine`. Bare Sail still cannot land those sinks (`DeltaWriteNode`, listing table, missing memory format). `foreachBatch` on a *rate* (or other engine) stream is rejected at `start()`. Console is liveness-only. Native `format("kafka")` on **Sail** consumes (or produces) on the driver and `createDataFrame`s the Kafka schema into the engine (`CAST(value)` runs on Sail) — `subscribe` / `subscribePattern` / `assign`, JSON offsets, `includeHeaders`, SASL PLAIN, GSSAPI, PEM SSL, JKS/P12; one announced micro-batch, never `rate`. JVM uses OSS `spark-sql-kafka`. The Fabric Eventstream notebook API (`format("kafka")` + `eventstream.itemid` / `eventstream.datasourceid` + `foreachBatch`) works on **Sail** (emulator consume → Kafka-schema LocalRelation, `foreachBatch` in the agent) and on the **JVM overlay**. Opt-in broker for Eventstream items ([51-eventstream-kafka.md](51-eventstream-kafka.md)). The engine matrix asserts rows *reach the sink*, and the kafka source row asserts bytes on the engine. GSSAPI and JKS/P12 fail loud. Native checkpointed streaming is the **JVM overlay** | 🟢 Real (one micro-batch, materialised) / 🟠 JVM overlay |
| `OPTIMIZE` / `VACUUM` (Delta maintenance) | **Partly real on the default engine**, via delta-rs: Sail's planner has no such commands, so the Livy agent recognises those statements and runs them through delta-rs against the same table — verified compacting 3 files into 1 and vacuuming the superseded 3, rows intact. `ZORDER`/`WHERE` are **refused, not ignored** (they change what is compacted); use the JVM overlay for the full syntax. Proven against a real `abfss://…onelake…` table end-to-end through Livy REST (`e2e/livy`), with a negative control: the same statement with credentials forced off is refused, so the pass is not vacuous. delta-rs authenticates on its own account — a Connect client cannot read back the session's bearer — minting from the same issuer with the same Storage audience, resolved per statement so a refreshed token needs no restart. One caveat remains, unchanged and by design: the executor differs from real Fabric (the emulator performs it, not Spark). See [20-lakesail-engine.md](20-lakesail-engine.md) | 🟢 Real (delta-rs) |
| Java/Scala UDFs, `spark.jars` | **Out of scope for Sail by design, not a technical impossibility.** Sail already embeds a foreign runtime — it links CPython via `pyo3` (`sail-python-udf`) for Python UDFs — so a JVM could be embedded too. The real obstacle is that a Spark Java/Scala UDF is compiled against *Spark's own classes* (`InternalRow`, Catalyst encoders), so running one needs Spark's jars loaded — which is JVM Spark, exactly what Sail exists to avoid. Bytecode translation (GraalVM, TeaVM) does not help: the problem is Spark's API surface, not executing bytecode. The **JVM overlay** is the answer | 🔴 default / 🟠 JVM overlay |
| Change Data Feed | **Notebook API is real on the default engine**, via delta-rs: `.option("delta.enableChangeDataFeed", "true")` on a Delta write, and `.option("readChangeFeed", "true")` on a read, are intercepted by the Livy agent, announced on stderr, and executed against the same table. The result is **materialised** (`createDataFrame`) — `.explain()` is a LocalRelation, no predicate pushdown. Bare Sail still cannot enable the feature (loud on write; inert on an unwrapped read — `e2e/sail` does not install the intercept). `spark.delta_change_feed(uri)` remains as the named helper. Proven locally by the engine-matrix CDF probe and on OneLake by `e2e/livy`. Native lazy CDF is the **JVM overlay** | 🟢 Real (delta-rs, materialised) |
| `read.json(multiLine=True)` | **Notebook API is real on the default engine.** Sail's JSON reader is NDJSON-only; the Livy agent wraps the named option, parses the file on the driver (Spark text-writer directories included), announces on stderr, and materialises via `createDataFrame`. Plain `json()`, `multiLine=False`, a list of paths, and a schema fall through. `.explain()` is a LocalRelation; a large file lands on the driver — `from_json` + `explode` stays the distributed advice. Bare Sail (`e2e/sail`) does not install the wrap. Native lazy read is the **JVM overlay** | 🟢 Real (materialised) |
| `spark.jars` | Accepted but inert on Sail: JARs have no classloader. The **JVM overlay** has a real one (verified) | 🔴 default / 🟠 JVM overlay |
| Concurrent Delta overwrite writers | Two independent Connect sessions race at one barrier; one commits and the other receives a transaction failure from the conditional Delta-log create | 🟢 probed conflict rejection |

## Data Warehouse (`data-warehouse/`)

| Fabric feature | Emulator | Type |
|---|---|---|
| SQL-analytics-endpoint semantics over lakehouse Delta | DuckDB runs real SQL (aggregation / join / filter), e2e | 🟢 Real (engine in e2e) |
| Warehouse item management | Full | 🟢 Real |
| **T-SQL over TDS + Entra FedAuth** | Pure-Go TDS front (`internal/tds`) terminates the FedAuth handshake (real Entra token, `database.windows.net` audience), then **byte-splices** the client's post-login session to a real per-item **SQL Server** connection so the engine emits every token itself. Unmodified `go-mssqldb` *and* Microsoft **ODBC Driver 18** (pyodbc) clients connect and run T-SQL — including RPCs, prepared statements, and transactions. Verified against a real SQL Server; Microsoft's real **dbt-fabric** adapter passes `debug`/`seed`/`run`/`test` end-to-end (`e2e/dbt-fabric/`) | 🟢 Real (front + default sidecar) |
| **Lakehouse SQL analytics endpoint — Delta → engine** | The emulator reads the lakehouse's `Tables/<t>` Delta in pure Go and reflects (CREATE+INSERT) it into the sidecar on connect, so `SELECT` hits real OneLake data (matches DuckDB), **read-only** (writes rejected). *Not PolyBase* — SQL Server reading Delta in place is a proven dead-end on the Linux container (a throwaway spike; see [16-warehouse-tds.md](16-warehouse-tds.md)) | 🟢 Real (reflection) |
| **Warehouse — read-write T-SQL** | Client `CREATE`/`INSERT`/`SELECT` relay straight to the sidecar; the warehouse owns its data (no reflection) | 🟢 Real (relay) |
| **Fabric SQL Database (`database/`)** — OLTP + OneLake mirror | Same read-write TDS/FedAuth path (its own SQL Server database), plus **mirroring**: `POST …/sqlDatabases/{id}/refreshMirror` snapshots every table to OneLake as **Delta** (real Parquet + `_delta_log`), so Spark / DuckDB / delta-rs query the operational data. Verified with a `go-mssqldb`-writes → mirror → Delta-reads-back e2e (gated). *Continuous/CDC mirroring and write-back-to-Delta are the deferred edge* | 🟢 Real (snapshot mirror) |
| Per-item isolation (each item = its own SQL Server database) | Lakehouse/Warehouse routed by type; per-item databases so they never collide | 🟢 Real |
| RBAC → SQL permissions | Workspace role enforced on connect: no role → rejected; Viewer → read-only; Contributor+ → read-write (warehouse). **Each caller connects as its own database principal**, not the relay's account, so the engine's own RLS/CLS/masking have somebody to restrict ([55](55-tsql-security.md)); writers get `db_ddladmin`+reader+writer rather than `db_owner`, because ownership carries `UNMASK` **Below Contributor the rung is the item permission** — ReadData reads, Read alone connects with no role, no Read is no access — so a principal shared one warehouse reaches exactly that database. **Memberships are synced on connect, not only added**: until 2026-09-13 a Contributor demoted to Viewer kept `db_datawriter`, since provisioning never removed a role; the sweep across the workspace also fails closed now, because it revokes as well as grants. [57](57-item-permissions.md) | 🟢 Real |
| SQL analytics endpoint **data access modes** (user identity / delegated identity) | **Both modes served, with OneLake table, column and row security synced into the endpoint.** Every analytics endpoint is delegated identity by default — SQL permissions only. The mode is switched through an **authenticated emulator-native** `…/sqlEndpoints/{id}/_emulator/dataAccessMode`, since Fabric switches it only in the portal; Admin or Member only, and a warehouse is refused. A switch applies Fabric's documented effects on a real SQL Server: the workspace's live SQL sessions close; to user identity, SQL security policies are turned off and users' explicit table permissions set aside, and custom roles dropped; back to delegated, those return; unbound functions are dropped either way (a policy's own predicate is kept — a stated divergence). **In user identity mode, OneLake security is synced into the endpoint** as `OLS_<role>` database roles holding SELECT on the tables and columns each role grants — computed by `pkg/onelakesec`, the same decision every OneLake reader uses — with each caller's memberships synced on connect, including into a lakehouse reached by three-part name. Below Contributor, tables are readable only through those roles; a DDL trigger refuses table GRANT/DENY/REVOKE and security policies in T-SQL, as Fabric does. **Row filters** are validated against OneLake's documented grammar, translated into SQL Server predicate functions comparing text case-insensitively as OneLake does, and applied by one security policy per table that ORs the roles' filters — role members filtered, a Contributor+ in no filtering role unfiltered (inferred). A filter outside the grammar grants nothing rather than everything; a row filter or column list naming a column the table no longer has fails the whole sync instead — Fabric's documented "Database enters error state until policy is fixed" — for a role's own constraint and for a OneLake shortcut source's alike. The sync runs in one transaction and only when something changed; reflection drops a table's synced policy before recreating it. **Direct Lake on SQL** over such an endpoint reads the caller's OneLake-filtered rows through it, and under `directLakeOnly` fails every table, since Fabric says it "falls back to DirectQuery 100% of the time". The endpoint also now accepts the views, functions, roles, grants, policies and masks authored on it — refusing data writes past comments and later statements in a batch. Witnessed with Viewers in different roles, one in none, and a Contributor, under both modes, and mutation-checked. Sync is on connect, not within "up to 5 minutes". [60](60-sql-endpoint-access-modes.md) | 🟢 Real (sync on connect; mixed row and column roles shown no rows rather than an error) |
| SQL analytics endpoint **shortcut tables**, and access at the source | **A OneLake, ADLS Gen2, Amazon S3 or Dataverse shortcut under `Tables/` reads as a table, from its target.** In user identity mode a caller the OneLake source refuses is refused; the source's column allow-list and row filters narrow the read on top of the consumer's, and the intersection wins. In delegated mode a shortcut whose OneLake source has row or column security is blocked for every reader, as a view that refuses with the reason. Source-side narrowing and delegated blocking are OneLake-only, correctly — an external target carries no OneLake security to check. Not built: ownership chaining. An Admin or Member is `db_owner` and is not narrowed; where a filter reads a column the source withholds from a reader, that reader is refused the table (SQL Server needs SELECT on a predicate's columns); listing an external target (S3 ListObjectsV2, Blob List Blobs) is this repository's own choice, not Microsoft's. Built to Learn's *OneLake security for SQL analytics endpoints* and enforced by real SQL Server; not verified against a real tenant. [61](61-sql-endpoint-shortcuts.md) | 🟢 Real (ownership chaining not built) |
| **Row-level security (T-SQL)** — `CREATE SECURITY POLICY … FILTER PREDICATE` | **Real, and the engine's own.** Each caller connects as its own database principal, so a filter predicate keyed on `USER_NAME()` gives two callers different rows from one SELECT — witnessed through the relay by a real go-mssqldb client (`ci:warehouse-tds`), with the other caller's row asserted in the same run. **A different mechanism from OneLake security's RLS**, and not interchangeable: this one is defined in SQL, enforced by the T-SQL engine, and "only applies to queries on a Warehouse or SQL analytics endpoint". OneLake security's RLS covers Lakehouse-type items across every engine and [does not cover Warehouse at all](54-onelake-security.md). A warehouse is secured by this and nothing else. | 🟢 Real (SQL Server) |
| **Column-level security (T-SQL)** — `GRANT SELECT ON t(col)` | **Real.** A denied column ERRORS for the denied caller and reads for the other — withholding the value, not blanking it. Same split applies: SQL-defined CLS is the Warehouse/SQL-endpoint mechanism, OneLake security's CLS is the Lakehouse one. | 🟢 Real (SQL Server) |
| **Dynamic data masking** — `ALTER TABLE … MASKED WITH` | **Real**, witnessed as `"aXXX@XXXX.com"` vs `"ada@example.test"` for two callers, the second holding `UNMASK`. **SQL-only by design**: the docs' mode comparison lists DDM as "Not supported in OneLake security", so unlike RLS and CLS it has no cross-engine counterpart — there is no path to it except the T-SQL one. | 🟢 Real (SQL Server) |
| `information_schema` / `sys.*` introspection | Relays natively — reflected/warehouse tables are real SQL Server tables | 🟢 Real (relay) |
| Per-column type fidelity (real SQL types over the wire) | The splice forwards SQL Server's own COLMETADATA, so **every** column carries its true native type over the wire (the re-encode fallback, used only by fake test backends, synthesizes INTN/FLTN/BITN and falls back to NVARCHAR text) | 🟢 Real (native) |
| **Nested CTEs** (`WITH x AS (WITH y AS …)`) | Fabric supports standard, sequential **and nested** CTEs; the SQL Server sidecar rejects the nested form (`Msg 156`), which would make the emulator *stricter* than Fabric. The TDS layer closes the gap on the wire: it parses the `WITH` prefix with a real lexer (string literals, quoted identifiers, nestable block comments) and flattens the nesting into the sequential form the sidecar accepts, then re-encodes the batch. Statements Fabric *itself* refuses are rejected rather than rewritten — non-`SELECT` targets, DML or `OPTION` hints in a nested definition, same-level duplicate names, and references escaping their nesting scope (the last of these verified to otherwise succeed post-flattening, i.e. a real divergence, not a theoretical one). Cross-level name shadowing is refused by name rather than renamed, because rewriting references safely needs a parser this layer deliberately is not. Statements sent as **parameters** (`sp_prepexec`/`sp_executesql`) are rewritten too: the RPC parameter list is walked, the statement parameter re-encoded, and the result **re-parsed and compared with the original** before it is forwarded — every other parameter byte-identical, or the original goes instead. Unmodelled parameter types (TEXT/XML/UDT/`SQL_VARIANT`) forward untouched rather than being guessed at. Witnessed by `examples/medallion-pyspark` running dbt's **native `accepted_values` and `relationships`** — which compile to nested CTEs — with no CTE-free substitute in the project, and by a parameterized-statement probe through the real ODBC driver ([29-tsql-parity.md](29-tsql-parity.md)) | 🟢 Real (rewrite + refusals) |
| **CTAS** (`CREATE TABLE … AS SELECT`) | Fabric/Synapse spell "materialise this query" as CTAS; SQL Server has no such statement and spells it `SELECT … INTO`, so the sidecar rejects valid Fabric T-SQL with `Msg 156`. The TDS layer rewrites it, splicing `INTO` before the first `FROM` at paren depth 0 — including **inside the `EXEC('…')` dynamic SQL dbt-fabric actually ships**, which a statement-level rewrite would miss. Fabric's `WITH (DISTRIBUTION = …)` options are dropped (physical layout, not results). Witnessed by `examples/medallion-pyspark` building gold with dbt's own `+materialized: table` ([29-tsql-parity.md](29-tsql-parity.md)) | 🟢 Real (rewrite) |
| **Class B strict mode** (`-tsql-strict`) | Refuses what real Fabric rejects but SQL Server accepts — recursive CTEs, triggers, synonyms, `CREATE USER`, `SET TRANSACTION ISOLATION LEVEL`/`ROWCOUNT`/`IDENTITY_INSERT`, `FOR XML`, `FOR JSON` inside a subquery, `/` or `\` in a schema or table name, `IDENTITY(seed, increment)`, enforced key constraints, multi-column statistics, `PREDICT`, `sp_showspaceused` — so a locally green build means a Fabric-green build. **A real client now sees the refusal** (`ci:warehouse-tds`): go-mssqldb connects to the TDS endpoint with an entra token and, with the flag on, gets the emulator's own error naming the feature — `recursive-cte`, `identity-seed`, `enforced-constraint`, `set-isolation-level`. The **negative half runs first and is what makes it evidence**: without the flag the same recursive CTE reaches the real SQL Server and returns its 5 rows, so a refusal cannot be confused with the engine rejecting the statement; and an ordinary `select 1` still succeeds with the flag on, so a build that refused everything would not pass. Mutation-tested: dropping the flag on the way to the TDS layer, and removing the feature name from the error, each fail the witness. **Off by default**: it removes capability, which is the operator's call. Indexed views stay unenforced, with the reason recorded; the `FOR JSON` rule is also checked against sqlglot-go's parse tree | 🟢 Real (opt-in) |
| Connection by item *name* (vs GUID) | Workspace read from the server name (`<workspace>.datawarehouse.fabric.microsoft.com`), item resolved by display name; a GUID still resolves by id (back-compat). Verified with a real `go-mssqldb` client | 🟢 Real |

## Data Factory (`data-factory/`)

| Fabric feature | Emulator | Type |
|---|---|---|
| Data Pipeline control flow (If / ForEach / Until / Switch / Filter / Fail, expression language, `dependsOn`) | Pure-Go interpreter that really executes. **Deactivated activities never run**: `state: Inactive` records the reserved `Inactive` placeholder status, branching follows `onInactiveMarkAs` (a `Failed` mark steers `UponFailure` branches without failing the run), and no output fields are invented — previously both fields were dropped on parse and a deactivated activity executed, side effects included | 🟢 Real (orchestration) |
| Per-activity **policy — retry + backoff + timeout** | Applied to every activity type: `policy.retry` re-runs a failed activity (each retry from scratch; only the final outcome is recorded, carrying `retryAttempt`); `policy.retryIntervalInSeconds` is folded into the run's `durationInSeconds` as virtual backoff; `policy.timeout` fails an attempt whose own virtual duration exceeds the limit. No real sleeping — backoff and timeouts are exercised in milliseconds on the controllable clock | 🟢 Real |
| **ForEach** sequential / parallel (`isSequential`, `batchCount`) | Iterations run in array order (deterministic); the mode sets the reported wall-clock — sequential iterations add, a parallel batch costs its slowest — matching how real Fabric overlaps them | 🟢 Real |
| List **pagination** (`continuationToken`) | **On by default**, as in Fabric: a list past the server page size returns an opaque `continuationToken` and an **absolute** `continuationUri`, with no client opt-in — so a client that ignores the token breaks here exactly as it would in production. `?maxPageSize` may narrow a page, never widen it. The page size is a **testing lever** (`-list-page-size` / `FABRIC_LIST_PAGE_SIZE`): set it to 2 and every list forces the token loop on the spot, the same idea as the controllable clock for LROs. Witnessed by Microsoft's **`fab` CLI** — a plain list (no page-size parameter) comes back paged, and paging through is asserted complete: no item twice, none missed, terminal page tokenless | 🟢 Real |
| **Spark Job Definition** and **Invoke copy job** activities (`SparkJobDefinition`, `InvokeCopyJob`) | Both RUN the referenced item rather than reporting it ran. `InvokeCopyJob` moves real bytes through the same executor a direct Copy job run uses, and the witness asserts the destination table's CONTENT, not just a Completed status. `SparkJobDefinition` submits a real job and, with a Spark agent attached, drives it to its actual outcome; with **no** engine the job is submitted and left Pending and the activity says so rather than claiming success — the fabrication these two replaced. Both are SYNCHRONOUS, as Fabric's are: the run happens in the activity's goroutine so the next activity cannot read outputs that do not exist yet. typeProperties are the documented ones (`sparkJobDefinitionId`/`copyJobId` + `workspaceId`), and the zero GUID is honoured as Fabric's "this pipeline's workspace" sentinel. **Not implemented**: the SJD activity's optional overrides (`executableFile`, `mainClass`, `commandLineArguments`, `defaultLakehouse`, `environmentId`) — the item's own definition decides, and an override would be silently ignored, so they are left for when they are needed | 🟢 Real (both run) |
| **Invoke pipeline** (ExecutePipeline) | Resolves the referenced `DataPipeline` (GUID or name, optional other workspace) and runs it through a fresh interpreter — **real recursive interpretation**, one level deeper on the same engines. `waitOnCompletion` (default) gates the parent on the child's terminal status; parameters flow into the child; a cycle or excessive nesting fails loudly | 🟢 Real |
| Pipeline → notebook activity (TridentNotebook) | Resolves the notebook reference and creates a **real RunNotebook job instance** the pipeline gates on — the pipeline→jobs linkage is real; the notebook's **cells** execute on the Spark sidecar, and a job with cells outstanding has **no clock-derived completion**: it stays non-terminal until the engine reports, so a completed job means the cells ran. Cell execution is witnessed end-to-end by `ci:medallion`: the pipeline's Notebook activity starts a genuine run, Sail executes it over Spark Connect, and the example asserts the engine-reported read/write set as the `Notebook` lineage edge — evidence that exists only if the cells ran, not inferable from the binding | 🟢 Real (chain + cells) |
| `queryactivityruns` detail | Full | 🟢 Real |
| Activity-level lineage | Successful Copy execution persists its resolved workspace/item/path source→sink edge, returns it in activity output, and exposes workspace lineage for OpenMetadata ingestion | 🟢 Real (Copy) |
| Copy activity — **OneLake → OneLake** | Really moves the bytes through the storage layer: a file, or a directory subtree preserving structure; source/sink locations `{workspaceId?, itemId, path}` are expression-resolved (GUID or name); returns real `filesWritten` / `dataWritten`. **`fileListPath` is implemented**: a source naming one copies exactly the files it lists — each "the relative path to the path configured in the dataset" (ADF's own wording, `third_party/adf-pipeline-schema/Pipeline.json`) — rather than the whole subtree, and a listed file that does not exist fails the activity; the list itself is addressed the same way `folderPath`/`fileName` are. A sink naming `fileListPath` is refused, since the schema carries it only on `*ReadSettings`. Wildcards, `sqlReaderQuery`, `versionAsOf`/`timestampAsOf`, `partitionOption` and `keyColumns` remain unimplemented and **fail loudly** by name, as do external stores / format transformation | 🟢 Real (in-family) / 🔴 external |
| Lookup activity — **OneLake CSV/JSON/Parquet/Delta** | Reads **real rows** from a CSV, JSON, or standalone Parquet file, or a lakehouse **Delta table** (`Tables/<name>`, auto-detected — no format hint needed) in OneLake; honors `firstRowOnly`; the result flows into `@activity(…).output` for downstream steps. Parquet/Delta reuse the warehouse's own Parquet reader — a real Delta column keeps its native type (int/float/bool), not a stringified cell | 🟢 Real (CSV/JSON/Parquet/Delta) |
| GetMetadata activity — **OneLake path** | Stats a **real** OneLake path: `exists` / `itemType` / `size` / `lastModified` / `childItems`; a missing path honestly returns `exists:false` | 🟢 Real |
| **Delete data** activity | **Really removes the data** through the storage layer, so `FileDeleted` events fire and a Reflex trigger sees a pipeline Delete exactly as it sees an ADLS client's delete; `recursive` removes the subtree, non-recursive removes direct-child files and leaves subdirectories; returns `filesDeleted`. A **missing path fails loudly** naming the path — held to the loud side while the real oracle is unmeasured, per the comment on `deleteActivity` | 🟢 Real (storage layer) |
| **Script** / **SqlServerStoredProcedure** / **SqlPoolStoredProcedure** activities | Run **real T-SQL** against a Warehouse/Fabric-SQL-Database item's own SQL Server database — the same per-item backend the TDS endpoint and the SQLDatabase mirror share. `Script` runs each `scripts[]` entry (`Query` → real rows back, `NonQuery` → rows affected); `SqlServerStoredProcedure` calls a real stored procedure with named parameters. The target is named directly as `{workspaceId?, itemId}` (the emulator's own scoped mapping — real Fabric's linkedService/connection reference isn't modeled), the same shape Copy/Lookup/GetMetadata already use — and `sqlPool` is accepted as a fourth spelling of that key, because `SqlPoolStoredProcedure` is the Synapse-dedicated-pool name for the identical activity (same `storedProcedureName` + `storedProcedureParameters`, differing only in how it names its target) and the Warehouse is the item that owns a real SQL Server database here. That type string used to fall to the dispatch default and report `Succeeded` without executing anything. Honest error without a warehouse SQL backend attached | 🟢 Real (scoped) |
| **HDInsight Spark** activity (`HDInsightSpark`) | **The submission protocol terminated locally, the code executed by the engine the emulator already runs** — the Livy precedent (docs/20) applied to a second protocol: no HDInsight cluster is proxied to, the activity's contract is answered here. A **Python** `entryFilePath` is read from OneLake and **Sail computes**, with `arguments` as `sys.argv` (argv[0] the entry file) and `sparkConfig` carried in. The witness asserts **the entry file's own code reached the engine**, not that the activity returned Succeeded — mutation-checked: an activity that reports success without executing fails it. **`className` NOW RUNS on the JVM overlay**, the same `spark-submit --class` path as a Databricks JAR task: the entry file is the jar, addressed through the lakehouse mount, and the JVM's **exit code decides the activity**. The old refusal ("no path that submits a Java/Scala main class, on either engine") became false when that path landed. Sail still answers `available:false` and is refused by that **probe**, not by the stale sentence. Refused **by name**, each with its cause: `proxyUser` (no impersonation model), and `sparkJobLinkedService` (an external store the emulator does not model). No engine attached → an honest failure naming the missing engine, never a claimed submission. The output carries `executedBy` so a run cannot be misread as a real cluster | 🟢 Real (Python on Sail; className on the JVM overlay) / 🔴 className on Sail |
| **Azure Databricks** activities (`DatabricksNotebook`, `DatabricksSparkPython`, `DatabricksSparkJar`) | Same bet as the HDInsight row: **the submission contract is terminated locally and the engine the emulator already runs executes the code**. A notebook task's file is read from OneLake and run with its **`baseParameters` bound as names** (the way `dbutils.widgets` delivers them, which is what notebook code reads); a Python task's file runs with **`parameters` as `sys.argv`**, argv[0] the file. The two deliveries are opposite and a test pins each against the other's mechanism — mutation-checked. A JAR task's **`parameters` resolve against the pipeline scope** the same way: `@pipeline().parameters.since` evaluates rather than being handed to `spark-submit` as the literal expression, which is what Sprint-the-JSON-value used to do. **`DatabricksSparkJar` NOW RUNS on the JVM overlay.** It was refused on both engines because *the agent runs Python statements and nothing here submits a main class* — the first clause true, the second a **gap rather than a law**: the overlay image is Apache Spark and ships `spark-submit`, and nothing had wired it up. The agent grew a `/submit` route running `spark-submit --class`, and the jar is addressed through the **lakehouse mount** the notebook path already depends on rather than through a second transfer with its own failure modes. The JVM's **exit code decides the activity**: a submit that fails while the HTTP call succeeds is exactly the case a naive implementation reports green. Two refusals survive, both narrower and both **probed rather than assumed** — **Sail** answers `available:false` for want of a `spark-submit` (the witness fails if the emulator refuses without asking the engine), and a **remote workspace** (`FABRIC_DATABRICKS_URL`) stays refused because the emulator does not upload libraries into someone else's DBFS, so a `jobs/create` naming a jar that workspace cannot see would fail there, late, for a reason the definition does not describe. The distinction the old note was careful about still holds: attaching a JAR *library* and executing a main class are different capabilities, and the emulator now has both. Refused too are `libraries` (installing on a cluster whose lifecycle the emulator does not own — an Environment item is the modelled way). A `dbfs:`/`/Workspace`/`/Shared`/`/Repos` path is refused by name **unless** `FABRIC_DATABRICKS_URL` points at a workspace (databricks-emulator): then those paths are submitted as written, and a lakehouse `<itemId>/<path>` is imported there first. Without the URL, files address OneLake as `<lakehouseItemId>/<path>` and reinterpreting a Databricks path as a lakehouse path would invent a mapping nobody wrote. No engine attached (and no URL) → an honest failure; the output carries `executedBy` rather than a fabricated `runPageUrl` | 🟢 Real (notebook + python; JAR task on the JVM overlay) / 🔴 JAR on Sail or a remote workspace |
| **Azure Batch** activity (ADF `Custom`) | **On by default.** A Custom activity's `command` is a shell process on the Spark agent — the same machine a notebook cell can already `subprocess.run` on, behind the same bearer and workspace RBAC. Refusing it by default was the false pass for any pipeline that actually uses Batch, the same class of reversal as the Web activity. The command executes in the **agent's container** rather than the emulator's process (blast radius is the engine, not the API), `extendedProperties` become **environment variables** as Batch documents, and the command's **own exit code decides the activity** — a non-zero exit fails it with the code and the command's stderr, and an unreadable report fails too, because an unknown exit status is not success. `FABRIC_CUSTOM_ACTIVITY=off` restores the refusal so no command reaches the agent (the old opt-in spelling `shell` remains on). Refused even when on, each naming its cause: `resourceLinkedService`/`folderPath` (Batch stages resource files from a storage account; the emulator models no connections, and staging nothing would leave the command referencing files that are not there), `autoUserSpecification` (no user model — certifying a privilege behaviour that does not exist), `referenceObjects` (linked services and datasets serialised onto the node) | 🟢 Real (command in the agent container) / 🔴 Batch-node features |
| **Azure ML** activities (`AzureMLExecutePipeline`, `AzureMLBatchExecution`, `AzureMLUpdateResource`) | **Refused by name — and the refusal is a correction, not a gap.** These three type strings were absent from the dispatch switch, so they fell to its default and were reported **`Succeeded`**: a pipeline that scored a model was recorded as having scored it, and the next step read an output nobody wrote. They now fail loudly instead, each naming its own cause. **Why these do not run when HDInsight, Databricks and Azure Batch do:** every one of those names *a thing to execute* the emulator can get hold of — an entry file, a notebook, a python file, a command — so terminating the external submission protocol locally and letting the engine we already run compute is honest (the Livy precedent, docs/20). `mlPipelineId` is an opaque handle on a pipeline **published in an Azure ML workspace**; its steps are not in the definition, not in OneLake, not in any item held here, so there is nothing to terminate the protocol *onto*, and running some other artifact in its place would invent the published pipeline's behaviour — the same objection that refuses a `dbfs:` path. The MLflow sidecar does not rescue it: AML's run history is MLflow-shaped, so the emulator *could* write the run **record**, but a record marked finished for steps that never ran only relocates the fabrication into the tracking store, where it is harder to see. `AzureMLBatchExecution` and `AzureMLUpdateResource` additionally target **ML Studio (classic), which Azure has retired**. **Skipping one deliberately is supported by a real Fabric feature** rather than an emulator flag — `state: "Inactive"` with `onInactiveMarkAs` — and the witness *runs that advice* rather than only asserting the message contains it | 🔴 Refused by name (was a silent success) |
| **Validation** activity | **Really waits for the data**, which is the only version of this activity worth having: it polls a real OneLake path and passes only when the path is actually there, honouring `minimumSize` (a file must be at least that many bytes) and `childItems` (`true` = the folder holds at least one file, `false` = the folder is empty — kept **tri-state**, since absent and `false` are different assertions). Directories are **implicit** in OneLake, so a folder is judged by what is under it rather than by a row that does not exist; an activity that only looked one up would report a landing folder absent forever. The wait is measured on the **virtual clock** — no real sleeps — and `sleep` between attempts is clamped to the remaining time, so a 100-second sleep cannot overshoot a 3-second deadline. Failing names **which predicate was still unsatisfied**, not just that time ran out. Before this it fell to the dispatch default and reported `Succeeded`, which for a guard is the worst possible bug: the pipeline reads absent data *with the guard's blessing*. The wait loop is unit-tested through a scripted clock that forces the subscribe-before-read interleaving, the same discipline the WebHook park had to learn | 🟢 Real (OneLake + virtual clock) |
| **Azure Data Explorer Command** activity (`AzureDataExplorerCommand`) | **Runs the control command on the real Kusto engine** the emulator already hosts behind Eventhouse — no cluster is proxied to, and the command travels the same relay helpers the KQL data plane uses, so the two cannot drift on database isolation or first-use creation. The witness asserts **the engine received that command against the isolated engine database for that KQL Database item**, not that the activity returned Succeeded, and that the engine's internal database name is **mapped home** to the Fabric display name before the output is handed back. A **query is refused by name**: the schema's contract is control commands (they begin with a dot), and one relayed to `/v1/rest/mgmt` would fail with a message about the engine rather than about the mistake — nothing is relayed when it is refused. Naming a non-KQLDatabase item is refused too, because proceeding would create an engine database from the wrong item's id and report success against it. `commandTimeout` bounds the call. *Connection stand-in:* ADF puts the cluster and database on a linked service; the emulator names the target as `{workspaceId?, itemId}` like the rest of the family. It too used to be reported `Succeeded` while nothing ran — a silently-successful `.drop table` is the version worth picturing | 🟢 Real (protocol terminated, Kusto computes) |
| **HDInsight Hive/Pig/MapReduce/Streaming**, **U-SQL**, **SSIS package** activities | **Refused by name, each with its own cause** — and, like the Azure ML row, this is a correction rather than a gap: all six fell to the dispatch default and were reported `Succeeded`. The default was justified as right for a *connector leaf* and wrong for a *compute activity*, whose whole point is an effect later steps consume. **That distinction has since been retired: the leaf half was a fiction** — no oracle has a connector activity type — and the default now refuses everything that reaches it, so the six below are refused alongside every undocumented string rather than instead of them. HiveQL is refused because the agent routes SQL to **Spark SQL**, a different dialect — the overlap between them is not the same as the guarantee; Pig because nothing here interprets Pig Latin; MapReduce because it is a **mapper/reducer contract executed by Hadoop**, and `spark-submit` is a different execution model — running the jar through it would execute something else and report it under this activity's name (a named Java main class is no longer the obstacle: the Databricks JAR task and HDInsight Spark `className` both submit one on the JVM overlay); Streaming because there is no Hadoop Streaming harness (a mapper/reducer pair is not a Custom-activity command — it is a MapReduce runtime the emulator would have to supply); U-SQL because Azure has **retired Data Lake Analytics**; SSIS because the emulator hosts no integration runtime and a package's work is defined inside the package. **How the set was found is the reusable part:** the 41 discriminators in ADF's published schema were diffed against what the dispatch switch and the pipeline interpreter actually handle | 🔴 Refused by name (were silent successes) |
| **Functions** activity (`AzureFunctionActivity`) | Makes the **real call** to `functionAppUrl` + `/api/` + `functionName` over the same HTTP core as Web (shared `httpActivity`, so the two cannot drift on timeouts, bounds, the non-2xx rule or output shaping); the function key travels as `x-functions-key`, and the witness's stand-in **rejects a wrong key** so the pass cannot be vacuous. The ADF schema's own rules are refused by name: `method` outside its seven-value enum, `body` on GET ("not allowed"), no `body` on POST/PUT ("required"). *Connection stand-in:* in ADF the URL and key live on the `AzureFunctionLinkedService`; the emulator models no connections, so both inline in typeProperties, and the error for a missing `functionAppUrl` says exactly that | 🟢 Real (connection inlined) |
| **Web** and **WebHook** activities | Makes the **real HTTP call**: method, URL, headers and body all resolved as expressions; a JSON response is merged into the activity output at the top level, so `@activity('X').output.field` resolves downstream; `ADFWebActivityResponseHeaders` and `statusCode` carried as Fabric does; a **non-2xx fails the activity**, matching Fabric rather than handing the status back; `policy.timeout` becomes the request deadline. Bounded at 8 MiB — the output lives in the run record, so it is not a download mechanism. `FABRIC_WEB_ACTIVITY=stub` restores the old record-without-calling behaviour for a hermetic CI leg, and labels the output `stubbed: true` so such a run cannot be mistaken for one that called | 🟢 Real (both) — WebHook **parks for real**: the call carries a generated `callBackUri` (a PATH, per the repo's the-path-is-the-contract precedent; the token in it is the credential, as ADF's own), the pipeline goroutine parks until the receiver's callback or a timeout measured on the **virtual clock** (a frozen clock advanced past the deadline expires it with no real sleep — `clock.Changed` exists for exactly this, and the witness fails without it), callback body fields surface in the activity output, and `reportStatusOnCallBack` consumes a reported non-2xx as the activity's failure. ADF schema enforced by name: POST-only enum, object body, D.HH:MM:SS timeout |
| **REST connector** — `RestSource` and `RestSink` in a Copy, with pagination | Makes the **real HTTP requests** and commits the rows: `url` (or the linked service's `url` + dataset's `relativeUrl`), `requestMethod` GET/POST, `requestBody`, `additionalHeaders` and `httpRequestTimeout`, all expression-resolved. The JSON response is shaped into a Delta table in the sink lakehouse — `translator.collectionReference` picks the record array, and a response holding exactly one array needs no hint; **`translator.mappings`** lift fields out of nested records by JSONPath — the shape BMC Helix actually returns (`entries[].values.*`), where auto-flatten finds no scalar columns at all. Witnessed end to end by `e2e/rest-helix`: a Web activity logs in for an AR-JWT, the copy pages `limit`/`offset` through a stand-in Helix that **rejects `Bearer`**, and the incidents land as Delta. **`paginationRules` really page**: `AbsoluteUrl` (absolute or relative) and `QueryParameters.x` / `Headers.x` cursors read out of the body (JSONPath) or the response headers; `{placeholder}` selectors stepped by `RANGE:start:end:step` (an empty end is open-ended) — the ServiceNow/BMC Helix `limit`+`offset` shape; `EndCondition:` with `Empty` / `NonExist` / `Exist` / `Const:v`; `MaxRequestNumber`; and RFC 5988 `Link: …; rel="next"`, **on by default only when no other rule is declared**, as Fabric documents. The loop also stops on HTTP 204 and on any JSONPath resolving to null. A rule the emulator does not implement is refused **by name** rather than ignored. `Accept` is forced to `application/json` as Fabric does; a non-2xx fails the copy; a top-level JSON array with pagination rules is refused, since Fabric does not paginate that shape. Nested fields have no column shape and are reported by name in `skippedColumns`. Bounded at 8 MiB per page, 1M rows and 1000 pages — refused, never truncated, so a `next` that points at itself fails loudly instead of looping. Auth is Anonymous + `additionalHeaders`; the emulator models no connections, so any other `authenticationType` fails by name — a Web activity can fetch a token and pass it as an expression, which is how BMC Helix's `AR-JWT` scheme works here. **`RestSink` writes the other direction**: rows from a Delta table or a Parquet/CSV file are POSTed (or PUT/PATCHed) in batches of `writeBatchSize` (default 10000), each request carrying Fabric's payload — a JSON array of row objects. `httpCompressionType: gzip`, `additionalHeaders`, `httpRequestTimeout` and `requestInterval` (validated against Fabric's [10, 60000]ms band, and virtual like retry backoff) are honoured; a non-2xx fails the copy, zero rows send **no request** rather than an empty array, and a `BinarySource` is refused because opaque bytes have no rows to send | 🟢 Real |
| **Salesforce** — `SalesforceV2Source` and `SalesforceV2Sink` in a Copy | Runs the **real Bulk API 2.0 query lifecycle**: creates a query job (`operation` is `queryAll` when `includeDeletedObjects` is set — a different operation, not a filter), polls it to a terminal state, then downloads CSV result sets paged by the **`Sforce-Locator`** header, which ends on the literal string `"null"`. `objectApiName` or a SOQL `query`; with neither, all the object's data. A `Failed`/`Aborted` job fails the copy naming the **job id** so it is findable in the org's own monitor. `reportId` is **refused by name** — a report is the Analytics REST API, not a Bulk query, and running a query instead would return the object's rows rather than the report's. The org is named directly (`instanceUrl` + `accessToken`, both expression-resolved) because the emulator models no connections; a Web activity can run the OAuth call and pass the token in. Bounded at 8 MiB per page, 1M rows and 1000 pages. Sink (`SalesforceV2Sink`) is scoped, not built — [41-salesforce-connector-plan.md](41-salesforce-connector-plan.md). **`SalesforceV2Sink` writes the other direction** through the ingest lifecycle: create a job, **PUT** the CSV to its `contentUrl`, `PATCH` it to `UploadComplete` (a job left Open uploads rows nothing ever processes), then poll. One job per `writeBatchSize` (default 100,000); `writeBehavior` Insert or Upsert, with `externalIdFieldName` required for upsert and refused without it. `ignoreNullValues` picks between Bulk CSV's two meanings — an **empty field leaves the value unchanged**, the literal **`#N/A` sets it to NULL** — and an empty string stays an empty string rather than becoming either. A job that reaches `JobComplete` with records rejected **fails the copy**, naming `failedResults`, because a partial write reported as a whole one is the worst outcome available. Witnessed end to end by `e2e/salesforce`: a **round trip** — OAuth via a Web activity, five accounts out over three locator pages into Delta, then upserted back in three ingest jobs, with every record compared against what the org originally served | 🟢 Real |
| **ServiceNow via `RestSource`** | **Real, and it is the case Microsoft documents.** `api/now/table/incident?sysparm_limit=…&sysparm_offset=…` is Example 1 of the REST connector's pagination rules, with `"QueryParameters.{offset}": "RANGE:0:10000:1000"` — the emulator runs that example. Both documented paging routes are witnessed in one pipeline (the offset RANGE **and** RFC 5988 `Link` headers), rows land as Delta read back through OneLake, and the negative controls refuse an anonymous and a wrong-credential read so a pass cannot mean the target let anyone in. Records are FLAT under `$.result`, so this carries **no** `translator.mappings` and auto-flattening must infer the columns — the half `e2e/rest-helix` cannot reach, since ARS nests under `values`. **Scope:** this is the Table API's documented shape reached through `RestSource`, *not* Fabric's first-party ServiceNow connector type, which the emulator does not implement; auth is Basic through `additionalHeaders` because a native `authenticationType` is refused by name (no connection model). That is the documented reason a real user reaches for `RestSource` here — the built-in connector is Basic-only, so OAuth needs this route | 🟢 Real (against a modelled Table API) |
| External connectors the emulator cannot run (ServiceNow, …) | **Refused by name, and this row is a correction rather than a gap.** It read *Stubbed success — reached in `dependsOn` order with inputs resolved*, describing a connector leaf that **does not exist**: a connector is not an activity type in either oracle, it is the `type` of a `Copy` source or sink, and `Copy` has always refused the ones it cannot run — `copy "sn" source: source type "ServiceNowSource" is not supported by the emulator`. Two tests had encoded the fiction, running activity types named `ServiceNowLeaf` and `SalesforceSource`, strings no authoring surface emits (Fabric's real names are `SalesforceV2Source` and `SalesforceServiceCloudV2Source`, and they are **source** types). What the stub measurably answered for was type strings no oracle documents — including **typos**: `TridentNotebok` was reported `Succeeded` having run no notebook, and so was an activity with **no `type` at all**. The dispatch default now refuses, names the type, and suggests the nearest accepted spelling. The **REST connector remains the working route** to any connector exposing a REST API — and to BMC Helix, which real Fabric has no connector for at all | 🔴 Refused by name (was a silent success) |
| **Apache Airflow Job** | Typed item + beta file APIs; uploaded Python DAGs sync to an opt-in real Airflow 2.10.5/Python 3.12 sidecar, whose scheduler/executor and REST state determine the Fabric job result (`e2e/airflow`) | 🟢 Real (sidecar) |
| **Dataflow Gen2** (Power Query M engine) | Typed item/definition management round-trips. Refresh, Publish, and in-pipeline execution fail with `DataflowEngineNotImplemented`; no open Power Query M engine exists to attach | 🟡 mgmt / 🔴 exec |
| Connectors / on-prem gateways | — | 🔴 Not implemented |

## CI/CD (`cicd/`)

| Fabric feature | Emulator | Type |
|---|---|---|
| Git integration (connect / status / commit / update / disconnect) | Full, real state | 🟢 Real |
| `fabric-cicd` tool publishing | The real client round-trips definitions (e2e) | 🟢 Real |
| Deployment pipelines — model, assignment, **item pairing**, **Deploy Stage Content** (D0–D2) | Real promotion: definitions really copy, pairs decide (not names), metadata only — a deployed lakehouse arrives empty — and target-only items survive. 202 LRO + `/result` detail. [23-deployment-pipelines.md](23-deployment-pipelines.md) | 🟢 Real |
| Deployment pipelines — role-assignment CRUD (D3) | Add / Delete / List; Admin is the only role a pipeline defines; mutations require Admin, reads require membership | 🟢 Real |
| **Variable Library consumed by a Data Pipeline** (`libraryVariables`, `@pipeline().libraryVariables.<alias>`) | Real resolution, not storage: the library definition is parsed (`variables.json`, `settings.json`, `valueSets/<name>.json`), the **active value set is merged over the defaults as a partial override**, and the pipeline's `libraryVariables` declarations are resolved before the run so an unresolvable reference fails it with `PipelineLibraryVariableUnresolved` instead of silently becoming `""`. `activeValueSetName` is an item property set by PATCH — deliberately *not* in the definition, which is what lets one definition serve every stage. The declaration binds **by name with no GUID**, so a pipeline moves between workspaces with nothing rewritten, and the lookup key is the declaration's **alias**, not the variable name. **The wire shapes were captured from a live tenant**, because the public docs publish neither the expression syntax nor the pipeline-side JSON ([48-variable-libraries.md](48-variable-libraries.md)). **Witnessed by Microsoft's own `fabric-cicd`**, which is what moved this to green: it publishes the Variable Library AND the referencing pipeline from a git-shaped repository (`envLib.VariableLibrary/` with a nested `valueSets/`, `medallion.DataPipeline/`), sets the active value set itself, and the emulator then RUNS that published pipeline — which fails by construction unless the variable resolved to the expected value. The value-set switch is exercised through the same published definitions. **That witness immediately found a real interop defect**: fabric-cicd hardcodes `/VariableLibraries/` (capital V) while the REST reference documents `variableLibraries`; ASP.NET routing is case-insensitive and Go's ServeMux is not, so Microsoft's own tool could publish into a tenant and NOT into the emulator. Typed collections now accept the initial-capital spelling too. **Notebooks too**, via `notebookutils.variableLibrary` (`getLibrary`, `get("$(/**/lib/var)")`) — that API IS published, so the shim follows the reference rather than a capture, resolves through the same active-value-set merge, and is built on the public item APIs so the identical code path runs against real Fabric. It follows `getDefinition`'s **202**, since reading the 202 body yields `null` and presents as an empty library. Two documented asymmetries are preserved rather than smoothed: the `/**/` prefix is required, and **names are case-sensitive for notebooks while the library name is case-INsensitive for pipelines**. **User data functions** get the documented `fn.FabricVariablesClient` shape (`getVariables()`, `["name"]`/`.get("name")`) so a function BODY is testable. **They also execute now** — `notebookutils.udf.getFunctions` reads the item's real definition (`definition.json` for the function list, `resources/functions.json` for parameters and return types, `function_app.py` for the code) and runs it. The divergence that remains is stated rather than implied: Fabric runs a UDF in its own isolated runtime with platform-materialised connection bindings, while this runs `function_app.py` in the CALLER's interpreter and provides none — a function taking `fn.FabricLakehouseClient` raises rather than receiving a stub that fails later, further from the cause. The last three are blocked UPSTREAM rather than unbuilt, which is why they are not merely 'todo': **shortcut** assignment is UI-only (Fabric: "including REST API assignment, aren't supported") so there is no request to implement; **Copy job** parameterises the source/destination CONNECTION ID, which is exactly what this emulator refuses by name for external endpoints; **Dataflow Gen2** has no Power Query M engine to resolve into. Ceilings recorded in [48](48-variable-libraries.md). **`ItemReference` variables included**: a real tenant's definition is a committed fixture, so the `{"itemId","workspaceId"}` value round-trips and `@pipeline().libraryVariables.<alias>.itemId` reaches it — passed through, never synthesised. `Connection reference` (also *preview*) is NOT captured and is not claimed. Note the two type vocabularies differ: the library says `ItemReference`, the pipeline declaration says `Object` **`ItemReference` variables included**: a real tenant's definition is a committed fixture, so the `{"itemId","workspaceId"}` value round-trips and `@pipeline().libraryVariables.<alias>.itemId` reaches it — passed through, never synthesised. `Connection reference` (also *preview*) is NOT captured and is not claimed. Note the two type vocabularies differ: the library says `ItemReference`, the pipeline declaration says `Object` | 🟢 Real (witnessed by fabric-cicd) |## Item types present, engine absent

| Fabric area | Emulator | Type |
|---|---|---|
| Real-Time Intelligence — Eventhouse / KQL Database (`real-time-intelligence/`) | Full item management (including the default child database an eventhouse creates, and `creationPayload.parentEventhouseItemId`), plus the **Kusto REST protocol** on the eventhouse's published `properties.queryServiceUri` — `/v1/rest/mgmt`, `/v1/rest/query`, `/v2/rest/query` — terminated by the emulator (Kusto-audience bearer, workspace RBAC, one isolated engine database per Fabric KQL Database) and **executed by Microsoft's own KQL engine container** (`kustainer`) when the `rti` profile attaches it. No engine attached → honest 501. Witnessed by `e2e/rti` (CI job `rti`) with raw REST and Microsoft's `azure-kusto-data`. The engine needs AVX2, so CI on amd64 is the witness of record — the same opt-in-sidecar shape as Airflow. [25-rti-kusto.md](25-rti-kusto.md) | 🟢 Real (sidecar) |
| Real-Time Intelligence — Eventstream (`real-time-intelligence/event-streams/`) | Item management always. Execution is a **real Apache Kafka KRaft broker** (ASF, not Redpanda/BSL) behind Fabric's notebook API: `readStream.format("kafka")` with `eventstream.itemid` / `eventstream.datasourceid` — **never** `rate`. Create mints a DefaultStream `datasourceId` and topic `{itemId}.{datasourceId}`; unknown IDs fail loudly; Custom HTTP produce writes JSON key/value bytes. **Sail (default):** emulator consume → Kafka-schema LocalRelation, `foreachBatch` in the agent (one micro-batch, announced). **JVM overlay:** OSS `spark-sql-kafka`. No broker → honest 501. **Lakehouse, Reflex, and Eventhouse destinations, and operators, are separate rows.** Witnessed by `e2e/eventstream` (CI job `eventstream-sail`) — the same opt-in-sidecar shape as Eventhouse and Airflow. [51-eventstream-kafka.md](51-eventstream-kafka.md) | 🟢 Real (sidecar) |
| **Eventstream — Lakehouse destination** | Custom HTTP source events land in a real Lakehouse Delta table. Bind is an emulator-native control surface (`POST …/eventstreams/{id}/destinations`, `{type, itemId, table}`) because Fabric's topology has no public REST — the same situation as Reflex triggers. `type` is `Lakehouse`, `Reflex`, or `Eventhouse`. After a successful produce, each event `value` is parsed as a JSON object (else a `value` string column) and appended through `WriteDeltaTableAs` into `Tables/<table>`. No destination bound → produce unchanged. No broker → 501 and no table. Spark-native `format("kafka")` writes are not drained; there is no background consumer. Overwrite/merge and Fabric definition round-trip for topology are out of scope. Operators are a separate row. Witnessed by `e2e/eventstream` (CI job `eventstream-sail`) asserting `_delta_log` content, plus Go tests that `ReadDeltaTable` the five produced rows and a second produce that appends. [51-eventstream-kafka.md](51-eventstream-kafka.md) | 🟢 Real (Custom HTTP source → Delta append) |
| **Eventstream — Reflex destination** | Custom HTTP source events start a real item job. Bind `{type: Reflex, itemId}` on the same emulator-native dest surface; `itemId` is a Reflex. A trigger on that Reflex with `eventType: Microsoft.Fabric.Eventstream.EventReceived` and `source.itemId` the Eventstream names the action. After a successful produce, each event fires that trigger through the existing Activator path (`invokeType: EventTriggered`) with `@pipeline()?.TriggerEvent?.Key` / `.Value` / `.EventType` bound from the produce payload — not `FileName` / `FolderPath`. No dest bound → produce does not fire even if the trigger exists. A `FileCreated` trigger on the same Reflex does not fire. No broker → 501 and no job. Eventhouse dest and operators are separate rows. Witnessed by `e2e/eventstream` (CI job `eventstream-sail`) asserting five `EventTriggered` job instances, plus Go tests that the pipeline reads `TriggerEvent.Value`. [51-eventstream-kafka.md](51-eventstream-kafka.md) | 🟢 Real (Custom HTTP source → job) |
| **Eventstream — Eventhouse destination** | Custom HTTP source events land in the Eventhouse's KQL database through **direct ingest** (`.create-merge` + `.ingest inline`), the path kustainer actually supports. Bind `{type: Eventhouse, itemId, table}` on the same emulator-native dest surface; `itemId` is an Eventhouse, `table` a Kusto identifier, optional `database` the child KQL Database display name (default: the eventhouse's own child). After a successful produce, operator output (or the raw events) is inferred into columns and ingested into the isolated engine database for that KQL Database — the same isolation the Kusto relay uses. No engine attached → produce 502 naming `--kql-url`. A stored dest pointing at a non-Eventhouse fails loud. This is **not** Fabric's streaming-ingest protocol and not `Kusto.Ingest` / the `ingest-` endpoint; those stay refused because the engine does not host them. Same produce trigger as Lakehouse dest: one micro-batch, no background consumer. The table name and every column name are quoted (`['kind']`) because both a bound `table` and a source field can be a name real Kusto refuses bare with `SYN0002`. Witnessed by Go tests that attach the in-process Kusto stand-in and assert `.create-merge` + `.ingest inline` against the isolated database, and by `e2e/rti`, which runs the emitted form against kustainer itself, pins as a set the nine keywords of 36 probed that the engine actually refuses, and shows a quoted name resolving to the same entity a bare one does. [51-eventstream-kafka.md](51-eventstream-kafka.md) | 🟢 Real (Custom HTTP source → Kusto direct ingest) |
| **Eventstream — operators** | Filter, GroupBy, and tumbling Window run on the produce batch, between the Custom HTTP source and destinations. Bind is emulator-native (`POST …/eventstreams/{id}/operators`) because Fabric's topology has no public REST. Filter (`eq`/`ne`/`gt`/`gte`/`lt`/`lte`/`contains`/`exists`) drops events; Window (tumbling, this batch only) stamps `_window_start`; GroupBy (`count`/`sum`/`min`/`max`/`avg`) collapses the batch. Kafka / DefaultStream stays the raw source; dests see the operator output. Join / Union / Expand and hopping/sliding windows are refused by name — they need more than one stream or cross-batch state. Witnessed by Go tests: Filter keeps `n>=3` (2 of 5 rows), GroupBy counts the batch, Window+GroupBy splits two tumbling hours. [51-eventstream-kafka.md](51-eventstream-kafka.md) | 🟢 Real (produce-batch operators) |
| **`CopyJob`** | A Copy Job **copies**. `jobType=Execute` (Microsoft's documented run-on-demand spelling; the readback says `CopyJob`, and both dispatch) parses `copyjob-content.json` and runs each activity through the **same pipeline Copy executor** — real bytes into the sink, real Delta commits for `Tables/` sinks (Append appends, Overwrite replaces), and a lineage edge with producer `Copy`. **The copy's completion decides the status, not the clock** — run inline, a finished copy reported `InProgress` for the whole LRO-delay window, which is what the witness pins (1h delay vs a 5s poll deadline, destination bytes asserted present). Dispatch is async like every other executing type, so the POST does not hold a socket for the copy's duration; that half is true by construction and **not** separately witnessed — a blockable copy would be needed, and OneLake-to-OneLake has no seam to block on. The boundary is refused by name, not skipped: external connections (`CopyJobExternalSourceNotSupported` / `…DestinationNotSupported`, source checked first so a both-external refusal is deterministic), `jobMode: CDC` (`CopyJobCDCNotImplemented`), Merge/Upsert (`CopyJobWriteBehaviorNotSupported`) | 🟢 Real (Lakehouse legs) / 🔴 external, CDC |
| `KQLDashboard`, `KQLQueryset`, `WarehouseSnapshot` | Typed collections over the generic item surface — create/get/list/patch/delete and definition round-trip, with the collection forcing its type. Type names taken from fabric-docs payloads; **no execution engine** for these | 🟡 mgmt / 🔴 exec |
| **Event triggers** — `Reflex` (Data Activator) on OneLake events | The Reflex **triggers**. A trigger subscribes to a source item's OneLake file events (`FileCreated` / `FileDeleted` / `FileRenamed`) under an optional path prefix, and a match starts a real item job — `invokeType: "EventTriggered"` — with the event bound in as `@pipeline()?.TriggerEvent?.FileName` / `.FolderPath` / `.Subject`, reading which needed **safe navigation** (`?.`) in the expression language, so one definition works whether or not a trigger started it. **No broker is needed**: every byte written to OneLake passes through the emulator's own storage layer, so a file event is observable at the source whoever wrote it — an ADLS client, azcopy, delta-rs, a Copy activity, the mirror writer. Dispatch is synchronous and reentrant, so bronze→silver→gold chains work; a runaway is bounded by an **activation count** rather than by identity, matching how Fabric bounds one — Activator documents `Fabric item — Activations/user/minute — 50` and throttles or cancels beyond it, and caps input at 10,000 events/second/rule, with **no** documented loop detection or dedup. Independent events each activate, because Activator "continues monitoring without waiting for the action to complete". *Boundary:* the **binding** is an emulator-native control surface (`…/reflexes/{id}/triggers`) because Fabric has no public REST for it — the Eventstream/Reflex rule is assembled in the portal. What is faithful is everything downstream: the filter, the invocation, the `TriggerEvent` fields, and a real pipeline really running. [07-control-plane-api.md](07-control-plane-api.md) | 🟢 Real (OneLake sources) |
| Mirroring — Mirrored Database (`mirroring/`) | `POST …/mirroredDatabases/{id}/refreshMirror` mirrors an **external** SQL Server source (reached via a Connection with Basic credentials) to OneLake as real Delta — reusing the exact same mirror writer the Fabric SQL Database uses (`warehouse.Mirror`; same code, external source). Proven by a gated e2e: a table seeded directly on an external database (bypassing the emulator's own per-item routing entirely) mirrors and reads back correctly. *Snapshot-on-trigger, not continuous/CDC replication*; other source engines (Snowflake, CosmosDB, on-prem via gateway) are out of scope | 🟢 Real (snapshot mirror, SQL Server sources) |
| Power BI — Semantic Model **query** (`executeQueries`) | Real bounded **DAX engine** — `EVALUATE`, `SUMMARIZECOLUMNS`, measures, `SUM`/`DIVIDE`/`COUNTROWS`/`IF`/`SELECTEDVALUE`, the infix operators (`+ - * / &`, comparisons) with DAX precedence, relationship filter propagation, and a name that does not resolve erroring rather than aggregating to zero — over imported `data.json`. **Direct Lake is no longer graded here.** This row bundled the evaluator with the binding, the source resolution and the authorization behind one verdict, which is the one-witness-several-claims shape [24-parity-completion.md](24-parity-completion.md) exists to prevent — and the authorization half was witnessed by nothing and is not faithful. Each half now has its own row. **Querying needs Read and Build on the dataset**, as the reference states — so a workspace Viewer, who inherits Read only, needs Build granted, while Contributor and above inherit it and a principal with no workspace role can query a model shared with them for both. Proven by the golden DAX/GX suites. | 🟢 Real (DAX subset) |
| Power BI — **Direct Lake** binding (model on OneLake) | Compatibility-level-1604 **entity partitions** over a shared expression naming a OneLake URL: the table's rows *are* the item's current Delta, read at query time rather than copied in, so a Spark write is visible to the next DAX query with no refresh step. The binding is also recorded as a **lineage edge**, because a Direct Lake table's definition names the Delta it reads while an import model's does not. **The expression is matched by regex, not lexed as M** (`internal/api/directlake.go`): an `onelake.dfs.fabric.microsoft.com/<workspace>/<item>` URL is found and anything else is refused — which is why the SQL flavour below fails as a parse error rather than as a missing feature. | 🟢 Real (binding over current Delta) |
| Power BI — Direct Lake over a **Warehouse** source | A model may name a **Warehouse** rather than a Lakehouse, addressed the same OneLake way, and the emulator resolves the item and serves its tables' rows — the binding gold needs, since dbt-fabric builds gold in the warehouse and a `data.json` part is not a thing real Fabric has. **Those rows come over SQL rather than from Delta**, on the pooled service connection (`sqlDBFor` → `be.DB`, the DSN account), where Fabric reads the warehouse's own Delta through OneLake. The test that built this calls it *a backend difference, not a contract one*, which holds for the values and not for the identity they were fetched under — see the next row. An entity name that is not a plain identifier never reaches the `SELECT`, and a missing SQL engine is an error rather than a model with no rows. | 🟢 Real (rows; identity on the next row) |
| Power BI — Direct Lake **security**: table and column | **OneLake security is applied on the query path, for the identity asking.** Direct Lake does not read through a SQL analytics endpoint, so nothing upstream has already filtered — the product is explicit that it "uses the current user (or fixed identity) to figure out OneLake security roles and enforce OLS and RLS on the target Fabric item". A table no role grants cannot be resolved; a column outside the grant's projection is reported missing **by name**, which is the product's own shape for this ("Can't find table", "Column can't be found" — errors "when object permissions are missing after applying OneLake security roles"), not a 403. Contributor and above hold Read and ReadAll and are never narrowed, the same rule the DFS surface applies. Witnessed two callers / one query / different answers, with an unrestricted caller asserted in every run, plus a covering grant that narrows nothing and a projection covering the whole model — a gate that refused every Viewer would pass the refusals alone. An item with *no* roles is **ReadAll's to open**, as Fabric requires ("needs the effective identity to have Read and ReadAll"): a Viewer is refused until ReadAll is granted, and a grant on the source reaches a lakehouse in a workspace the reader has no role in. The decision is `store.OneLakeReadAccess`, the one the storage surface asks, so Direct Lake cannot admit what DFS would refuse. A warehouse source carries no roles, so ReadAll is the whole of it. | 🟢 Real (table and column) |
| Power BI — Direct Lake **row filters** | **Applied, by the same parser the SQL analytics endpoint uses.** Real Fabric applies a OneLake role's row filter and returns the rows it admits; so does the emulator now. The filter is parsed once in `pkg/onelakesec`, against Microsoft's documented grammar (`SELECT * FROM [schema.]table WHERE …`: comparisons, `[NOT] IN`, `IS [NOT] NULL\|BLANK`, `AND`/`OR`/`NOT`, `TRUE`/`FALSE`, 1000 characters), and Direct Lake evaluates it over the Delta rows it read — text case-insensitively as the RLS collation does, trailing spaces ignored, numbers and decimals exactly, dates against ISO-8601 text, and SQL's three-valued logic, so a comparison with NULL admits nothing, nor does its NOT. The SQL analytics endpoint renders the same parse into a SQL Server predicate (docs/60), and a **gated witness runs ten filters through both against a real SQL Server and requires the same rows** — each admitting some rows and not all. Filters from several roles union; a filter reads the whole row before the role's column projection; a filter that excludes everything returns no rows, as documented; one outside the grammar, written for another table, naming a missing column, or comparing a value it cannot convert **blocks the table by name**, never serves it unfiltered. Contributor and above are unrestricted, as OneLake security documents. **Inferred:** text ordering follows the named collation, not the syntax page's contradictory "bitwise comparison" for `>`/`<`; `IS BLANK` is NULL or empty text. Mutation-checked. [54](54-onelake-security.md) | 🟢 Real (OneLake's documented filter grammar) |
| Power BI — Direct Lake **on SQL** (`Sql.Database` over the SQL analytics endpoint) | **Served through the SQL analytics endpoint as the caller.** Fabric has two Direct Lake flavours, told apart by the shared expression; a `Sql.Database("<server>", "<endpoint>")` model used to fail for not containing a OneLake URL. It is now classified as Direct Lake on SQL, and refused by name when it cannot be one: both flavours in one model, more than one SQL source (the flavour "can use the data from a single Fabric data source"), `Sql.Database` options, an expression of neither flavour, or an unknown `directLakeBehavior`. The database argument resolves to a lakehouse's SQL analytics endpoint or a warehouse, by GUID or by name in the model's workspace (the name's workspace is inferred), and **Read on the source is required**, decided before the item is described so a stranger learns nothing about it; the server argument is not checked, a stated divergence. **Rows are read through the endpoint as the caller's own login**, provisioned by the same path the TDS relay uses, selecting the model's columns by name — so SELECT grants, column denials, row-level security and masking are SQL Server's decisions, witnessed against a real SQL Server with two callers and one model. That is Fabric's answer under the default `automatic` behaviour, where endpoint RLS or masking falls back to DirectQuery; `directLakeBehavior` is graded on its own row. Divergences stated: the server argument is unchecked, framing is not modelled (the endpoint's current rows are served), and a denied column fails every query over its table rather than only queries touching it. The datasources endpoint and lineage share the parser, so neither reports a SQL model as a OneLake source. [59](59-direct-lake-on-sql.md) | 🟢 Real (warehouse and lakehouse endpoints; server argument unchecked, framing not modelled) |
| Power BI — Direct Lake **DirectQuery fallback** | **`directLakeBehavior` decides, for Direct Lake on SQL.** Parsed from TMSL and TMDL, an unknown value refused. Under `automatic` (the default) and `directQueryOnly`, a table is served as the endpoint returns it to the caller — the DirectQuery answer, row-level security and masks applied. Under `directLakeOnly`, a table whose endpoint enforces a security policy, masks a column, or is a view fails naming the cause, as Fabric documents ("the query fails with an error"); the message is the emulator's own. The catalog is read through the service connection, since a restricted caller may not see the policy restricting them. Witnessed on a real SQL Server with a policy, a mask, a view and a plain table under both behaviours, and mutation-checked. Direct Lake on OneLake has no fallback, and the setting does not apply to it. **Not modelled:** capacity guardrails and framing as fallback causes; `TMSCHEMA_DELTA_TABLE_METADATA_STORAGES` still refuses `FallbackReason`, whose wire encoding is unverified, and `TABLETRAITS()` is not implemented. [59](59-direct-lake-on-sql.md) | 🟢 Real (Direct Lake on SQL; FallbackReason refused) |
| Power BI — **Report items** | Typed item + definition round-trip. **Rendering is deliberately absent and is not a gap**: a Power BI report is drawn client-side from a definition plus query results, and the emulator serves both halves — the definition here, the DAX through `executeQueries`. The definition is stored byte for byte and also **read**, never rewritten, by Fabric IQ MCP's `GetReportMetadata`. There is no server-side renderer in Fabric for this to be unfaithful to | 🟢 Real (definition round-trip; rendering is client-side) |
| Power BI — **semantic-model roles** (RLS and OLS) | **Row-level and object-level security are applied.** Roles were dropped silently by both parsers, so a model built to restrict a Viewer evaluated unfiltered for them. They are now read from TMSL and TMDL and applied in the loader REST `executeQueries` and every XMLA route share, to the principals the product applies them to — those without Write; Write holders read as before. Membership is by `memberId` or by `memberName` against the token's UPN (`preferred_username`/`upn`), case-insensitively; groups match nobody, since group membership is not modelled. Filters are a **bounded DAX row-predicate subset** — own-table columns, literals, comparisons, `&&`/`\|\|`/`NOT`/`AND`/`OR`, `IN { … }`, `TRUE`/`FALSE`/`BLANK`, `USERPRINCIPALNAME`/`USERNAME` — with DAX's case-insensitive text comparison; anything else (`LOOKUPVALUE`, `RELATED`) is refused by name, as is a comparison DAX would reject. Roles are additive; a principal in no role sees empty tables; filters travel active relationships one → many, transitively. Refused rather than guessed: `bothDirections` security filtering and many-to-many relationships; a service principal below Write, which the product cannot put in a role; `impersonatedUserName` on a secured model; relaying a restricted caller to an attached msmdsrv, whose catalog carries no roles;. **Object-level security** removes what the caller's roles hide from a copy of the model and data that queries and TMSCHEMA rowsets both read, so a hidden table or column is the ordinary not-found error; a hidden key column still joins, as documented; measures reading a hidden object are hidden; row filters and hidden objects from different roles of the caller are the documented query-time error; a secured table between two others is refused for everyone, as the product refuses it at design time. An object is hidden only when **every** role of the caller's hides it — **inferred**, since the product documents no OLS combination rule. The portal runner has no principal and refuses a secured model. Witnessed two callers / one query / different answers with a Write holder and a principal in no role in every run, and mutation-checked. [58](58-semantic-model-roles.md) | 🟢 Real (DAX filter subset; OLS combination across roles inferred) |
| Power BI — **DAX beyond the bounded subset** | `internal/semanticmodel` answers the subset its golden fixtures pin (`EVALUATE`, `SUMMARIZECOLUMNS`, measures, `SUM`/`DIVIDE`/`COUNTROWS`/`IF`/`SELECTEDVALUE`, infix operators with precedence, single-hop relationship propagation) and **errors rather than mis-evaluates** outside it. `ORDER BY` over the result's own columns was added for Fabric IQ MCP, whose skill tells agents to sort every multi-row query: BLANK first, text without regard to case, stable on ties. It is unit-tested but **not yet checked against Desktop**, and any other sort expression is refused. "Full DAX" is not a reachable state; what is reachable is a measured coverage figure against a real engine, and the oracle now exists — `e2e/pbix-desktop` runs **Power BI Desktop itself** on `windows-latest` and agreed with `executeQueries` **bit-identically** (`rel = 0.000e+00`, 5/5 runs). Attaching that `msmdsrv` from a **developer machine** is host-specific (UTM on a Mac you own / Linux+KVM `dockur/windows` / native Windows) and **not** a compose default, and **not** GitHub `macos-latest` / `ubuntu-latest` — those runners are already VMs; nested Windows is not every-push CI. The CI oracle stays `e2e/pbix-desktop` on `windows-latest`. Growing the in-process evaluator is evidence-driven goldens on ubuntu/mac; see [33-pbix-tooling.md](33-pbix-tooling.md) and [52-msmdsrv-hosts.md](52-msmdsrv-hosts.md) | 🟡 bounded subset, oracle available |
| Power BI — **SemPy / semantic-link-labs over XMLA** | **Two independent Microsoft clients** drive this surface in CI, which is the point: they read it differently. `sempy` projects DMV rowsets (`list_workspaces`, `list_datasets`, `evaluate_dax`, `list_tables`, `list_measures`, `list_columns`, `list_partitions`, `list_relationships`); `semantic-link-labs` 0.17.0 goes through the Tabular Object Model (`connect_semantic_model`), so it exercises TOM's ~35 element `<Discover>` batch and the object graph built from it, and both report the SAME seeded model (3 tables, 6 measures, 12 columns, 3 partitions). Neither client is modified; they reach the emulator as `api.fabric.microsoft.com` and the token is minted by `entra-emulator`. Labs also needs this repo's own `notebookutils` shim, which it calls for every REST request; that was measured, not assumed, by watching it fail at `resolve_workspace_name` before XMLA was reached. DAX results declare a type per column, so a caller gets numeric dtypes rather than strings. **Writes work too**: `connect_semantic_model(readonly=False)` adds a measure and it survives a reconnect, which is the assertion, because TOM reports SaveChanges as successful whenever the server answers at all. The write protocol was measured and is NOT the TMSL JSON this repo planned for: TOM sends a `<Batch Transaction="true">` of `<Create>`/`<Alter>` in the 2014/engine namespace carrying ROW DELTAS in the same rowset shape this server emits for Discover, keyed by the object ids it handed out. A field or object type the emulator cannot represent is refused by name rather than accepted and dropped. Scope: measures, plus lineage tags and annotations on existing objects. MDX, the LRO continuation byte, and structural writes (new tables, columns, relationships, partitions, Direct Lake migration) are NOT implemented. Phased in [32-xmla-plan.md](32-xmla-plan.md) | 🟢 Real (read path, two clients; TMSL writes and MDX not implemented) |
| Data Science — ML models / experiments / MLflow (`data-science/`) | Authenticated, workspace-scoped proxy to a real MLflow 3 tracking/model-registry server. Experiment/model creation synchronizes typed Fabric items; experiment/run references are isolated by workspace; successful artifact uploads are mirrored under the experiment item's OneLake `Files/mlflow-artifacts`. | 🟢 Real (sidecar) |
| **MicrosoftPowerBIMgmt 1.3.84 contract** (Microsoft) | **Microsoft's own Power BI PowerShell module connects and reads, with nothing reconfigured** — its stock `Public` environment resolves to `api.powerbi.com` and `login.microsoftonline.com`, and the emulator answers at both. `Connect-PowerBIServiceAccount -ServicePrincipal`, `Get-PowerBIWorkspace`, `Get-PowerBIDataset` and `Get-PowerBIActivityEvent` all round-trip **into the module's own .NET types**, which is the independence `az rest` cannot give: a transport returns whatever the body was, a typed client fails inside Microsoft's code on a wrong shape. The workspace is written through the Fabric API and read back through the module, so the writer never sees what the reader deserialises. **Wiring it up cost three defects, none reachable by any existing witness**: global-service discovery (`/powerbi/globalservice/v201606/environments/discover`) was unimplemented, so the module could not connect at all; `api.powerbi.com` was missing from the TLS certificate although the emulator has served that alias since the VS Code extension contract landed, failing every client that validates; and MSAL rejects an authority with no path segment. **Not served, and asserted as refused rather than skipped**: `New-PowerBIWorkspace` (`POST /v1.0/myorg/groups`), `Get-PowerBIReport` (`…/reports`), `Get-PowerBICapacity`, and `Get-PowerBIWorkspace -Scope Organization` (`GET /v1.0/myorg/admin/groups` — the Power BI spelling; the **Fabric** spelling `/v1/admin/workspaces` is served and is what that row claims). **How they failed was itself a finding, and it is fixed**: an unrouted `/v1.0/myorg` path fell through to the portal and answered **HTML**, so a typed client reported *"Unable to deserialize the response"* instead of a 404 naming the gap. The Power BI roots are now in `apiPrefixes`, and they answer **Power BI's own envelope** rather than Fabric's — measured from the real service, which returns ASP.NET's `{"Message":"No HTTP resource was found…"}` | 🟢 `e2e/powerbi-ps` |
| **Power BI global-service discovery** (`/powerbi/globalservice/v201606/environments/discover`) | The pre-authentication call a Power BI client makes before it has a token, and the reason `MicrosoftPowerBIMgmt` could not connect at all. **The oracle here is the client, not the documentation**: the public endpoint answers 404 to both GET and POST, so there is no response to copy, and the shape is taken from the `GSEnvironments` / `GSEnvironment` / `GSEnvironmentService` types in Microsoft's shipped assemblies. The served cloud is reported **relative to the host the caller reached**, so the same binary is right behind the `api.powerbi.com` alias and on localhost. A client populates **every** cloud it knows about when connecting to any one, and resolves each one's services with a LINQ `First()` — so returning GlobalCloud alone, or the others empty, fails the whole connect. The four national clouds are therefore fully formed and pointed at **RFC 2606 `.invalid`** hosts: naming them for this emulator would make `-Environment USGov` succeed against a deployment that is not USGov, and naming their real endpoints would be four constants nothing here can check. No `appId` is advertised, because the flows that read it are not implemented | 🟢 `e2e/powerbi-ps` |
| Graph (`graph/`), Real-Time Hub, Copilot / IQ (`iq/`), Embed, Workload Dev Kit | — | 🔴 Not implemented |

## Emulator-only (no Fabric equivalent — these exist for testing)

| Capability | Purpose |
|---|---|
| Controllable clock (`/_emulator/clock`) | Advance virtual time to drive LRO / job status transitions deterministically. |
| Fault injection (`/_emulator/faults`, `/_emulator/permissions`) | Force failures / throttling / RBAC denials to test client resilience. |
| **Flow stream** (`/_emulator/events`) | Server-Sent Events for every byte that moves through OneLake — `file` events for writes/renames/deletes whoever made them, and `table` events derived from Delta commits (`version`, `rowsAdded`, `filesAdded`). Plus `job` and `activity` events — a failing activity reaches the stream with its error and job id the moment it fails, rather than being reconstructed from `queryactivityruns` afterwards — and **attribution**: a Copy activity's writes name the activity, a notebook cell's name the cell (including engines that cannot set headers, via the same bearer claims lineage uses). Never inferred; a write with nothing to say carries no attribution. `curl -N` tails a medallion run live; `?kinds=` narrows it and `?since=` replays the ring buffer. Delivery is deliberately lossy — a slow consumer is told what it dropped rather than being allowed to stall a writer. Two further kinds complete a medallion's chain: `lineage` when a movement is recorded (so a graph redraws as a **warehouse** build happens, which has no job to end), and `query` when a semantic model is read — the Power BI hop, which is an event and never an edge because a query moves no data. [31-flow-observability.md](31-flow-observability.md)  **A job's terminal event is checked STRUCTURALLY, not type by type.** `startJob` used to decide the outcome in one place and announce it from a second list, and a disagreement between the two was **silent**: the status converged (the clock, or a `FinalizeJob` call) while nothing reached the stream, so every status-polling test passed and the Flow view showed a job running forever. Writing the structural test found a live instance — **an Apache Airflow job that could not start** (no `dagId`, or no Airflow attached) was recorded Failed and announced to nobody, because the "failed to even start" branch listed Notebook and Spark job definition and not it. The second list is now gone: `startJob` records what it settled *where it settles it*, and the deferred announcement reads that, so the decision and the announcement cannot drift. The test walks every dispatched (item type, job type) pair and pins **three** states rather than one — announced now, left to the clock (a generic item, deliberately unannounced), and awaiting an engine (**not** terminal and must not be announced as though it were, which is the opposite failure and just as silent). |
| **Data-flow lineage beyond OneLake** | A medallion's last hops move no bytes through OneLake, so each is observed on its own terms rather than inferred. **silver → gold**: the TDS front already parses every statement for dialect adaptation, so it also records what the engine *accepted* — CTAS, `SELECT … INTO`, `INSERT … SELECT`, `CREATE VIEW`, `sp_rename`, `DROP`, including inside the `EXEC('…')` dbt-fabric actually ships — resolving three-part names to Fabric items (`producer: Warehouse`). dbt's build-then-swap is followed through `sp_rename`, so the graph names `fct_order_lines` and not its `__dbt_temp` scaffold. A statement whose response carries an error, or whose outcome cannot be read, records nothing. **gold → semantic model**: a Direct Lake table's binding names its source, so the edge is recorded when the definition lands (`producer: DirectLake`); an **import** model gets none, because its rows arrive detached from wherever they were selected and inventing a source would be a guess. **Interactive engines**: `POST /v1/workspaces/{wid}/lineage` is the notebook read/write report without the job, for a Spark session or plain script (`producer: Reported`) — **emulator-native**, as Fabric has no such endpoint. The producer is the point: `Warehouse` is evidence, `Reported` is a claim, and a consumer never has to guess which. [31-flow-observability.md](31-flow-observability.md) | 🟢 Real |
| Svelte management portal | Dashboard, workspaces, operations, clock, and fault controls. |

## Ecosystem conformance: real OSS/vendor clients as witnesses

Parity isn't claimed from our own tests alone — each 🟢 surface is pinned against
the **real, unmodified client** a Fabric user runs, executed against the emulator
in CI (`e2e/<client>/`). If Microsoft's own tool round-trips unchanged, the
contract holds better than any assertion we could write ourselves.

| Real client (pinned) | Surface exercised | Status |
|---|---|---|
| `fabric-cicd` (Microsoft) | Control plane / CI-CD publish | 🟢 `e2e/fabric-cicd` |
| **Fabric CLI `fab`** (Microsoft) | Control plane — SPN auth (MSAL) + workspace/item CRUD (Notebook, SemanticModel, Report, DataPipeline, Lakehouse), `ls`/`get`/`api` | 🟢 `e2e/fabric-cli` |
| **`microsoft/fabric` Terraform provider** (Microsoft) | Control plane — SPN auth (Go azidentity) + workspace / nested folder / Lakehouse / capacity data source / workspace RBAC. Fabric REST, not ARM | 🟢 `e2e/terraform-fabric` |
| **Fabric Data Engineering VS Code extension 1.18.1 contract** (Microsoft) | Shared-backend/MWC authoring routes through `api.powerbi.com`; interactive kernel websocket is not claimed | 🟢 `e2e/vscode-extension` |
| Apache Airflow 2.10.5 | `ApacheAirflowJob` DAG discovery, scheduling, execution, and status | 🟢 `e2e/airflow` |
| MLflow 3 + dbt-duckdb | Workspace-scoped experiment/run/artifact/model lifecycle, followed by dbt's real Delta plugin over the same Spark-written OneLake table | 🟢 `e2e/data-science-loop` |
| `deltalake` (delta-rs) | OneLake Delta write/read | 🟢 `e2e/delta-rs` |
| `azure-storage-file-datalake` + Blob SDK | OneLake ADLS **Gen2 DFS** + Blob | 🟢 `e2e/adls-sdk` |
| `azcopy` (Microsoft) | OneLake Blob multi-block transfer | 🟢 `e2e/azcopy` |
| DuckDB | Lakehouse SQL over Delta/Parquet | 🟢 `e2e/duckdb` |
| PySpark behind the **Livy API** | Spark sessions / statements | 🟢 `e2e/spark`, `e2e/livy`, `e2e/notebook-run` |
| `notebookutils` | Notebook utility shim | 🟢 `e2e/notebookutils` |
| **JupyterLab** (Project Jupyter) — the editor `make up-jupyter` ships | The shim and Spark Connect inside a REAL ipykernel, driven by `nbclient`; then the same `.ipynb` published and run by the emulator through the agent, compared cell by cell | 🟢 `e2e/jupyter` — kernel and agent agree |
| `go-mssqldb` | Warehouse/Lakehouse **TDS + FedAuth** | 🟢 `internal/server`, `internal/tds` |
| **`dbt-fabricspark`** (Microsoft) | Fabric **Spark** via Livy HC sessions | 🟢 `e2e/dbt-fabricspark` — debug→seed→run→test on Sail |
| **`dbt-fabric`** (Microsoft) | Warehouse **TDS via mssql-python** (1.11+) | 🟢 `e2e/dbt-fabric` — debug→seed→run→test through the TDS splice |
| **`azure-kusto-data`** (Microsoft) + raw Kusto REST, over **`kustainer`** (Microsoft's own KQL engine) | Eventhouse / KQL Database: `/v1/rest/mgmt`, `/v1/rest/query`, `/v2/rest/query` on the published `queryServiceUri` — create table, ingest, query values back, per-database isolation | 🟠 `e2e/rti` — witness of record is CI (amd64). The engine needs AVX2, which Rosetta does not provide, so the default Docker setup on Apple silicon cannot run it; a QEMU x86-64 VM with `--cpu-type max` can, and does ([25-rti-kusto.md](25-rti-kusto.md)) |
| Apache Kafka (`apache/kafka` KRaft) + Fabric Eventstream notebook API on **Sail and JVM** | `format("kafka")` + `eventstream.itemid` / `eventstream.datasourceid` + `foreachBatch`, Kafka schema (`key`/`value`/`topic`/`partition`/`offset`), unknown IDs fail, not `rate`. Sail: emulator consume + LocalRelation. JVM: OSS `spark-sql-kafka`. Native OSS kafka on Sail (subscribe / pattern / assign, SASL PLAIN, kafka sink) is the same wrap class — one micro-batch, not checkpointed | 🟠 `e2e/eventstream` — Sail in CI (`eventstream-sail`); JVM weekly/manual with `e2e/spark-jvm` ([51-eventstream-kafka.md](51-eventstream-kafka.md)) |

The TDS surface now has **three independent driver witnesses**: `go-mssqldb`,
Microsoft **ODBC Driver 18** (pyodbc, still used by medallion warmup and
`e2e/type-map`), and **mssql-python** (via `dbt-fabric` 1.11+). The second
driver mattered historically — `go-mssqldb` tolerated a synthesized FedAuth
login, but ODBC Driver 18 took a compatibility path (prepared-statement RPCs +
`sp_reset_connection` under mandatory connection pooling) that desynced against a
re-encoding relay. The fix was to **byte-splice** the post-login session straight
to the real SQL Server (so it emits every token itself), which is exactly the
kind of driver-family gap a single-driver test never surfaces. `dbt-fabricspark`
likewise drives the high-concurrency Livy layer over its real Livy-session
protocol (`method: livy`, service-principal auth via entra-emulator).

## Scope boundary: Fabric, not the predecessor Azure products

The emulator targets **Microsoft Fabric** — the *convergence/successor* product —
not the earlier Azure analytics services Fabric replaced. That boundary is why
some adjacent dbt adapters and Azure surfaces are intentionally **not** built:
they belong to predecessor (often retired) products, and their Fabric-native
successors are what we emulate instead.

| Adjacent product / client | Why out of scope | Fabric-era equivalent (in scope) |
|---|---|---|
| **Azure Synapse** dedicated SQL pool (`dbt-synapse`) | Different product: its own control plane (Synapse workspaces) **and** an MPP T-SQL dialect (`DISTRIBUTION = HASH`, clustered-columnstore / resource-class DDL) that our vanilla SQL Server sidecar rejects. `dbt-synapse` layers on `dbt-fabric`, so the *shared* SQL path is already covered by the `dbt-fabric` witness | **Fabric Warehouse** — 🟢 TDS relay |
| **Azure Data Lake Analytics** — U-SQL / SCOPE (`dbt-scope`) | Retired service (EOL **Feb 2024**), proprietary batch language, no Fabric embodiment. The only overlap (Delta on a lake) is Spark/OneLake, already witnessed | Fabric **Spark** — 🟠 Livy |
| **ADLS Gen1** | Retired (**Feb 2024**), superseded by Gen2 | — |
| **ADLS Gen2** (standalone storage account) | **Not missing — OneLake *is* the Gen2 endpoint**: hierarchical namespace, the `dfs` filesystem API, `onelake.dfs.fabric.microsoft.com`. Fabric has no separate storage account to emulate | **OneLake** — 🟢 `e2e/adls-sdk` |

Rule of thumb: if a capability exists only in a product Fabric replaced, it's out
of scope; its Fabric-native successor is what we build. "We already have the
TDS/SQL Server foundation" makes Synapse *cheaper*, not *done* — the remaining
delta is a whole MPP dialect plus a second control plane, for a superseded
target. So the two dbt adapters we build (`dbt-fabricspark`, `dbt-fabric`) are
exactly the two that hit live Fabric surfaces; the other two (`dbt-synapse`,
`dbt-scope`) target predecessor products outside the emulator's remit.

## Why the boundary sits where it does

Real Fabric's own Livy endpoint is *Microsoft's implementation of the Livy REST
contract* over their Spark platform — they honor the protocol, not the retired
Apache Livy server. And where Fabric adds its *own* layer on top of that
protocol — high-concurrency REPL packing, which a vanilla Livy server has no
concept of — the emulator implements that layer directly rather than proxying,
because there is nothing to proxy it to. That is the same stance throughout: the
**protocol and control plane are the durable, real things** (built, not mocked,
so real clients can't tell the difference), and the compute engine is attached
(Spark; T-SQL on SQL Server; **KQL on Microsoft's own Kusto engine** —
[25-rti-kusto.md](25-rti-kusto.md); **Eventstream on Apache Kafka, Sail and JVM Spark** —
[51-eventstream-kafka.md](51-eventstream-kafka.md)) or deferred when proprietary or without an
implementation to attach at all (Dataflow Gen2's M engine, Power BI rendering,
Eventhouse streaming ingest). Every deferral fails loudly rather than
pretending to succeed. See [13-roadmap.md](13-roadmap.md) for the milestone
history and the deferred-with-cause rationale.