--- name: distilled-sdk description: Build or update a distilled SDK for an API provider — sourcing its OpenAPI/Smithy/GraphQL/discovery description, adding the spec mirror that feeds it, generating packages/, listing it on distilled.cloud with a category and a logo, writing a README with a complete Effect example, opening a GitHub PR whose body includes that same example, and regenerating an existing one. Use for "create a distilled SDK for ", adding a provider, writing or fixing a fetch-specs.ts, giving a provider a catalogue group or brand mark, working on stacks/distilled-submodules or a spec-mirror-* repository, or anything about where a package's specs come from. --- # Building a distilled SDK ## The pipeline ``` upstream API description (a URL, a git repo, a GraphQL endpoint, docs) │ │ stacks/distilled-submodules/spec-repos//fetch-specs.ts ← you write this ▼ distilled-mirror/spec-mirror- one repo per package, refetched daily │ layout: .meta/ (machinery) + specs/ (payload) │ git submodule ▼ packages//specs/spec-mirror-/specs/… │ │ packages//scripts/convert.ts spec dialect → Smithy ▼ packages//.generated-specs/*.json Smithy 2.0 models (committed) │ │ packages//scripts/generate.ts Smithy → Effect SDK ▼ packages//src/services/*.ts (committed) ``` Two facts follow from this and shape everything below: - **The generated output is committed.** CI never runs `generate` and never checks out a submodule. A PR is green with no spec present at all — the specs only matter to whoever regenerates. - **The mirror is the spec source, not the upstream.** A package never submodules the upstream repository. `github/rest-api-description` is 6.7 GB checked out for one 13 MB file; the mirror holds the file. See [`stacks/distilled-submodules/README.md`](../../../stacks/distilled-submodules/README.md). ## Step 1 — identify the upstream shape There is no generic fetch script. Copy the closest existing one and adapt it; this table is the whole decision. | Upstream publishes | Copy | Technique | | --- | --- | --- | | One OpenAPI document at a stable URL | `spec-repos/hetzner` | `fetch()` → validate → write | | A live endpoint serving YAML | `spec-repos/posthog` | `fetch()` → `yaml` parse → write JSON | | A few files inside a big git repo | `spec-repos/github`, `spec-repos/turso` | `raw.githubusercontent.com` per file — never clone | | Whole directories inside a huge git repo | `spec-repos/aws`, `spec-repos/azure` | blobless (`--filter=blob:none`) + `--no-checkout` clone, narrow sparse patterns from the tree, only then fetch blobs | | An index that enumerates many documents | `spec-repos/gcp` | crawl the discovery directory, write `_manifest.json` + one doc per entry | | A GraphQL endpoint | `spec-repos/railway`, `spec-repos/expo-eas` | introspection query → `schema.json` | | Only human docs (markdown/HTML) | `spec-repos/cloudflare` | crawl the docs sidebar, download each page's markdown twin, render the page HTML when that twin is truncated | A YAML spec does not have to be converted in the mirror: `spec-repos/coinbase` mirrors `openapi.yaml` verbatim and `packages/coinbase/scripts/convert.ts` passes `parse` from the `yaml` package to `runOpenApiConvert`. Convert in the mirror only when the upstream is an endpoint rather than a file. **When the user says "like GitHub"** they mean a few files out of a big repo — raw download, no clone. **"Like cloudflare"** means the description has to be scraped out of documentation pages. Copy `spec-repos/cloudflare`: crawl the sidebar for every page URL, download each page's `index.md` twin, and fall back to rendering the page's HTML when the twin comes back truncated — that host cuts the markdown off mid-page on its largest schemas. If you are about to scrape docs, first check whether the provider publishes a real machine-readable description somewhere — that is nearly always the better spec. Rules every `fetch-specs.ts` follows: - It runs from `.meta/` and writes to `../specs/` — nothing else. - It **validates what it got** before writing. A login page, a rate-limit body, or a gutted response is still valid JSON; failing here beats failing three steps later in the generator. The usual check is `typeof spec.openapi === "string" && spec.paths !== undefined`. - It writes deterministically — `JSON.stringify(spec, null, 2) + "\n"` — so a whitespace-only change upstream produces no diff and no daily commit. - It fetches the **subset the generator reads**, not the repository. ## Step 2 — add the mirror ```sh mkdir stacks/distilled-submodules/spec-repos/ # fetch-specs.ts, package.json, readme.md — copy the exemplar from step 1 ``` Add `{ package: "" }` to `SPEC_REPOS` in `stacks/distilled-submodules/SpecRepos.ts`, in alphabetical position. You cannot create the mirror repository — it lives in the `distilled-mirror` org and is created by the `distilled-submodules` stack, which deploys on merge to `main`. That is fine; step 3 replaces it. ## Step 3 — materialise the mirror locally ```sh pnpm specs:local ``` This copies the same scaffold the stack deploys into `packages//specs/.local/` and runs `fetch-specs.ts` from its `.meta` directory, so the result is byte-identical to a checkout of the real mirror: ``` packages//specs/.local/ ├── .meta/ fetch-specs.ts, package.json, tsconfig.json └── specs/ ← what the generator reads ``` The directory is gitignored. Re-run the command to refetch after editing the fetch script (the `pnpm install` only happens once). ## Step 4 — write the package Create `packages//` from the closest existing package. In `convert.ts`, declare the **production** spec path and resolve it: ```ts import { resolveSpecPath } from "@distilled.cloud/core/codegen/spec-path"; const specPath = resolveSpecPath(root, "specs/spec-mirror-/specs/openapi.json"); ``` `runOpenApiConvert` already does this for its `specPath` entries, so an OpenAPI package gets it for free — just declare the mirror path. Then register the package: `pnpm-workspace.yaml` needs nothing (it globs `packages/*`), but add both tsconfig references to the root `tsconfig.json`. Public surface: re-export the generated barrel at the package root (`export * from "./services/index.ts"`) so callers write `Pkg.vms.createVm`, not `Pkg.Services.vms.createVm`. Do not add a `Services` namespace. A single-service package re-exports operations on the root (`Pkg.listX`) the same way. ### Credentials `src/credentials.ts` is hand-written (copy `packages/s2/src/credentials.ts`). Every secret it touches is a `Redacted.Redacted` from `effect/Redacted`: API keys, tokens (access, refresh, bearer, session), passwords, client secrets, private keys, and signing or HMAC secrets. Base URLs, account and org IDs, emails, usernames, client IDs and key IDs stay plain strings. - **Inputs take `Redacted` only.** Type a secret parameter of `fromApiKey`, `credentials`, `fromToken` and the like, and any callback that returns one (an OAuth `load`/`refresh`), as `Redacted.Redacted`. Do not type it `string`, and do not type it `string | Redacted`. A plain string is how a secret ends up in a log or an error, so the caller wraps it. - **The resolved credentials hold `Redacted`.** The value the `Credentials` service yields, and any cache of it, keeps each secret redacted. - **Redact environment secrets on read.** Use `Config.Redacted("")`. - **Unwrap at the point of use.** Call `Redacted.value` only where the header, query string, body or signature is built, and never put a secret in an error message or an error field. Anything minted at runtime (an exchanged OAuth token, a signed JWT) is wrapped as soon as it exists. This must print nothing: ```sh grep -niE 'readonly \w*(key|token|secret|password)\??: string' \ packages//src/credentials.ts ``` ### Errors and response validation Every error class in `src/errors.ts` must be one the protocol can actually raise. A class in the operation error union that nothing constructs tells callers to handle a failure that never happens — that is how ~75 packages ended up declaring a `ParseError` no code path created. `src/errors.ts` declares, and the operation error union (`OpError` / `DefaultErrors`) includes: - `UnknownError` — built by the protocol's `unknownError` fallback. - `ParseError` with `{ body: Schema.Unknown, cause: Schema.Unknown }`, `.pipe(Category.withParseError)` — built by the protocol's `parseError` (copy `packages/s2/src/errors.ts`). - Any status classes the provider needs beyond core's `HTTP_STATUS_MAP`, each wired into the protocol's `statusMap`. The protocol wires every one of them: - **`makeRestProtocol`** requires `parseError`: `parseError: ({ body, cause }) => new ParseError({ body, cause })`. - **A hand-written protocol** calls `validateResponse(outputAst, value, (cause) => new ParseError({ body, cause }))` from `@distilled.cloud/core/response-validation` on every 2xx path that returns the operation output — after wire→TS key mapping, before `wrapSensitive`. `packages/core/src/protocol-rest.ts` is the reference. 2xx responses are validated only in strict mode. `ResponseValidation` (`import { ResponseValidation } from "@distilled.cloud/core"`) is one context reference shared by every SDK: lenient by default, switched with `Effect.provide(ResponseValidation.strict)` — only ever by layer. A new protocol reads the mode through `validateResponse`; it never adds its own flag or environment variable. Lenient mode checks only what the protocol needs in order to transform the body (unwrap an envelope, map keys, wrap sensitive members) and nothing more: a non-JSON body comes back as text, a body missing members comes back as read. Never decode against the output schema outside `validateResponse`. Strict mode surfaces every spec inaccuracy (an undocumented `null`, a new enum member) as a `ParseError`; that is the cost of opting in, and why strict is never the default. Do not add tests to a generated SDK. Generated code is tested once, through the generator and the protocols in `packages/core`. `packages/core/src/sdks.test.ts` runs against every package and checks the glue a new SDK hand-writes: its `OpError` union reaches `ParseError` (so `catchTag` on a strict call typechecks), something outside `src/services/` constructs it, and `errors.ts` exports it. A new package is covered the moment it exists; a package that cannot follow the pattern goes in that file's exemption list with the reason. A package gets a test only for code someone wrote by hand in it — a custom protocol (`packages/fly-io/src/protocol.ts`, everything in `packages/aws`), or credentials logic like `packages/prisma/test/credentials.test.ts` — next to that code. No live tests: calls against real APIs belong to Alchemy's test suite. Before opening the PR, confirm the parse error and the unknown-error fallback are both constructed outside the generated code — this must print two or more lines: ```sh grep -rnE 'new \w+(ParseError|Unknown\w*Error)\(' packages//src \ --include='*.ts' --exclude-dir=services ``` For every other class you add to `errors.ts`, find where the protocol raises it (`statusMap`, a code lookup table, or `new`). A class nothing raises comes out of `errors.ts` and the error union. ## Step 5 — iterate ```sh DISTILLED_SPECS_LOCAL=1 pnpm generate ``` The environment variable re-roots every spec read into `.local`. It prints a warning to stderr on every run so a local-spec generation is never mistaken for a real one, and it lives in one command's environment — **never** in a file. Without it the same command reads the submodule, which is what you want once the mirror exists. Iterate on `convert.ts` (spec → Smithy) and `generate.ts` (Smithy → TS) separately: `pnpm --filter @distilled.cloud/ run convert` stops after the Smithy models, which is usually where the interesting bugs are. Patches go in `packages//patches/` as RFC-6902 `*.json` and apply in **convert** (`finalizeConvert` at the end of every dialect — OpenAPI, GraphQL, discovery, proto, Cloudflare markdown), so `.generated-specs` is the patched Smithy model. OpenAPI pointers (`/paths`, `/components`) run on the spec before conversion; Smithy pointers (`/shapes`, `/metadata`) run on the model after. generate does not patch — it only compiles the committed models. `finalizeConvert` is **not idempotent** — `move` patches and model transforms only make sense on freshly converted models — so it stamps `metadata["distilled.finalized"]` and refuses to run over a model that already carries it. To re-apply patches or naming, re-run the package's `convert`; never post-process a committed `.generated-specs` file. It also fails if any `target` in a model points at a shape that does not exist. **Stale patch pointers fail the run** (they used to warn-and-skip, which is how Fly's whole chain vanished after the spec-mirror prefixed paths with `/v1`). Pass `onStalePatch: "warn"` only if you truly want skip. Operation **names** are convert policy, not patches, and default to **verbNoun** (`listApps`, `getApp`, `createMachine`) in the OpenAPI and GraphQL converters and in `finalizeConvert` (for dialects with no naming step of their own). `toVerbNoun` only reorders ids it can recognise — go-swagger `Apps_list`, REST `ConfigsList`, GraphQL `projectCreate` — and leaves anything already verb-first or ambiguous (`WatchPodList`, `AppGetOrCreate`, `accountById`) unchanged. Irregulars go in `operationNames` (lookup by `"METHOD path"`, then operationId) — PUT vs PATCH that share an upstream id need the path key. Cases live in `packages/core/src/codegen/rewrite-operation-ids.test.ts` (`pnpm vitest run`); add one before changing the heuristic. Do not RFC-6902-patch `/paths/~1foo/get/operationId`; those break when upstream adds a prefix. Patch the spec, not the generated TypeScript. Writing and checking a patch is the `distilled-sdk-patch` skill ([`.agents/skills/distilled-sdk-patch/SKILL.md`](../distilled-sdk-patch/SKILL.md)). ## Step 6 — wire the submodule ```sh pnpm specs:link ``` Everything about the entry is derived from the package name (`packages//specs/spec-mirror-` ← `https://github.com/distilled-mirror/spec-mirror-.git`), so this is a convenience, not a decision. Before the mirror exists it writes the `.gitmodules` stanza alone. That is inert — `git submodule` iterates gitlinks in the index, so a stanza without one is skipped by `pnpm specs:sync` and by every `submodule update` — and it becomes a real submodule when you run the same command again after the mirror is deployed. ## Step 7 — list it on the website The catalogue on distilled.cloud reads `packages/*/package.json` at build time, so a new non-private package appears on its own — in the catch-all group **More**, with a generated monogram where a logo should be. Two files under `website/build/` fix that, and both tolerate a package that is not in the tree yet, so the entries belong in the PR that adds the SDK. **Category** — add `` to a group array in `GROUPS` (`website/build/packages.ts`). Order within a group does not matter; the catalogue sorts each group by package name. While you are there add `SEARCH_HINTS[]` — extra words the catalogue filter matches on top of the npm name and the directory, which it already matches (`neon: "postgres serverless"`). **Logo** — add a `{ viewBox, inner }` entry to `website/build/data/brand-icons.json`, keyed by the same ``: ```json "neon": { "viewBox": "0 0 64 64", "inner": "" } ``` - `inner` is the source SVG's children — no `