# E2B Infrastructure Architecture

This document explains what this repository implements, what each service does, and how the
services interact. It is the fastest way to build a mental model of the codebase before diving
into the code.

> **Keep this document updated.** If a code change alters anything described here (service
> responsibilities, ports, protocols, data stores, flows, deployment topology), update this file
> in the same PR.

## What this repository implements

E2B provides **sandboxes**: isolated Linux VMs that start almost instantly (they resume pre-booted
snapshots instead of cold-booting), run arbitrary code (typically
generated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole
backend: the control-plane REST API, the data-plane VM orchestration built on
**Firecracker microVMs**, the in-VM agent, the edge routing layer, and template building.

Two ideas drive the design:

1. **A sandbox is a resumed snapshot.** Templates are pre-booted VM snapshots (memory + disk +
   VM state) stored in object storage. "Creating" a sandbox means restoring a snapshot, which is
   why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root
   filesystem is a copy-on-write overlay, so only touched data is ever fetched.
2. **Control plane and data plane are separate.** The API decides *where* a sandbox runs and
   tracks *that* it runs (Postgres/Redis); the orchestrator on each node owns *how* it runs
   (Firecracker, networking, storage). Sandbox traffic never passes through the API.

## System overview

```mermaid
flowchart TB
    subgraph clients["Clients"]
        SDK["SDK / CLI"]
        Browser["Browser / HTTP clients"]
    end

    LB["Load balancer<br/>api.* | *.domain wildcard"]

    VC["volume-content API (belt)<br/>api.&lt;domain&gt;"]

    subgraph controlplane["Control plane (API node pool)"]
        API["API<br/>REST :80, gRPC :5009/:5109"]
        DashAPI["dashboard-api :3010"]
        CP["client-proxy<br/>:3002"]
    end

    subgraph datastores["State"]
        PG[("PostgreSQL<br/>teams, templates, builds, snapshots")]
        RD[("Redis<br/>running sandboxes, routing catalog, caches")]
        CH[("ClickHouse<br/>metrics, events, optional logs")]
        OS[("Object storage GCS/S3<br/>template + snapshot artifacts")]
    end

    subgraph clientnode["Sandbox nodes (one orchestrator per node)"]
        ORCH["orchestrator<br/>gRPC :5008, proxy :5007"]
        subgraph vm["Firecracker microVM (per sandbox)"]
            ENVD["envd :49983"]
            USERPROC["user processes"]
        end
    end

    subgraph buildnode["Build nodes"]
        TM["template-manager<br/>(orchestrator binary, gRPC :5008)"]
    end

    SDK -->|REST| LB --> API
    Browser -->|"port-sandboxid.domain"| LB --> CP
    API -.->|"mint content token + domain"| SDK
    SDK -->|"volume content (token-authed)<br/>api.&lt;BYOC or default domain&gt;"| VC
    API -->|"gRPC Create/Delete/Pause"| ORCH
    API -->|"gRPC TemplateCreate"| TM
    CP -->|"lookup sandbox → node"| RD
    CP -->|"forward :5007"| ORCH
    CP -.->|"gRPC auto-resume"| API
    ORCH --> ENVD
    ENVD --> USERPROC
    API --> PG & RD & CH
    DashAPI --> PG & CH
    ORCH --> OS & CH
    TM --> OS
```

## Services

| Service | Package | Runs on | Purpose |
|---|---|---|---|
| API | `packages/api` | API nodes | Public REST API; sandbox lifecycle, placement, auth, quotas |
| Orchestrator | `packages/orchestrator` | every sandbox node | Runs Firecracker VMs; sandbox create/pause/resume/kill |
| Template manager | `packages/orchestrator` (role) | build nodes | Builds templates from Docker images |
| Client proxy | `packages/client-proxy` | API nodes | Edge router: sandbox URL → correct node |
| Envd | `packages/envd` | inside every VM | In-VM agent: process/filesystem API for SDKs |
| Dashboard API | `packages/dashboard-api` | API nodes | Backend for the web dashboard (teams, builds, admin) |

Supporting packages: `packages/shared` (protos, telemetry, storage clients, feature flags),
`packages/auth` (authentication library), `packages/db` (Postgres migrations + sqlc queries),
`packages/clickhouse` (ClickHouse schema + clients), `packages/otel-collector` (collector config),
`packages/nomad-nodepool-apm` (autoscaler plugin), `packages/local-dev` (local stack).

### API (`packages/api`)

The control-plane entry point (Gin, OpenAPI-generated from `spec/openapi.yml`, port 80).

- **Resources**: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs),
  templates and builds, teams, volumes, API keys, secrets, admin operations.
- **Auth** (via `packages/auth`): team API keys (`X-API-Key`, `e2b_` prefix), auth-provider JWTs
  (OIDC), and either an admin token or a service JWT verified from the configured admin JWKS.
  Backed by an auth DB (Postgres) with a Redis team cache.
- **Workload identity**: sandbox create accepts an optional `iam.tokens` map of caller-named
  workload-token definitions (each an exact `audience` and `tokenType`). A non-empty, validated
  map enables workload identity, whose identity the orchestrator derives from the sandbox's
  already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the
  orchestrator in `SandboxConfig.iam`. The API mints no credential and delivers nothing into the
  sandbox; file-based delivery is rejected at admission. Definitions are persisted in the
  running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and
  orchestrator re-sync; a fork starts a new workload and does not inherit them.
- **Placement**: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a
  static list). Chooses a node per sandbox with a **best-of-K** algorithm
  (`internal/orchestrator/placement/`): sample K ready nodes, score by CPU
  commitment/usage, pick the lowest; retry on exhausted nodes. Tunable live via feature flags.
- **State**: writes sandbox records to Redis (source of truth for *running* sandboxes) and the
  sandbox→node **routing catalog** (`sandbox:catalog:{id}`) in Redis that client-proxy reads. This
  API-written record is the default routing source; the orchestrator-written
  `sandbox:routing:{id}` is a flag-gated alternative (see "Sandbox routing records"). Persistent
  entities (templates, builds, snapshots, teams) live in Postgres.
- **Secrets**: `/secrets` is the only public surface for secret management (create, list, get,
  update, delete). The API authenticates the caller with the customer alternatives above, converts
  the authenticated team UUID to the project UUID the backend knows, checks the `customer-secrets`
  feature flag, and forwards a metadata-only request over a unary gRPC contract
  (`e2b.secretsstore.management.v1`) to the secrets store backend named by
  `SECRETS_STORE_BACKEND_GRPC_ADDRESS`. No caller credential, header or client-supplied tenant
  crosses that hop, and no response - here or in a log, a span or an error - carries a secret
  value. What a caller stores is a runtime marker, not a resolved secret; orchestrator-ee resolves
  a marker to a value at sandbox egress, never here. Without a configured address, or with the flag
  off, the routes stay registered and answer 403. Responses are `Cache-Control: no-store`, request bodies are capped at 512 KiB, and values
  at 64 KiB.
- **Rig passthrough**: admin endpoints under `/clusters/{clusterID}/rigs` manage a cluster's
  orchestrator node pools ("rigs"). They list rigs, change rig capacity, list and terminate
  instances, and read recent scaling errors. Every handler delegates to the cluster's edge service
  (`/v1/rigs/...`) through the per-cluster HTTP client that the cluster sync keeps fresh. The API
  holds no cloud credentials and makes no scaling decisions. The caller authenticates with an
  admin credential and never holds the cluster's edge secret. Edge status codes pass through unchanged
  (400, 404, 409, 501). An edge 401 becomes a 500: it means the API is misconfigured against the
  cluster, not that the caller's token is invalid. A cluster with no rig management configured
  returns an empty list. The local cluster has no edge deployment and answers 501 for every rig
  operation.
- **Extra listeners**: internal gRPC :5009 and edge gRPC :5109 expose `ResumeSandbox` so
  client-proxy can wake paused sandboxes on incoming traffic.
- Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to
  Loki, with a LaunchDarkly-gated ClickHouse read path (`logs-read-config`) for local-cluster logs
  during the log storage migration. `LOKI_URL` is optional: without it the api has no Loki client
  and those reads fail until the flag routes them to ClickHouse. `LOGS_READ_CONFIG=true|false` sets
  the value the flag falls back to when LaunchDarkly has none (no LaunchDarkly, or the flag not
  defined there); a LaunchDarkly value wins. LaunchDarkly feature flags also gate placement
  parameters, rate limits, and rollouts.

### Orchestrator (`packages/orchestrator`)

A single Go binary running on every sandbox node (as root). `ORCHESTRATOR_SERVICES` selects its
roles: `orchestrator` (run sandboxes) and/or `template-manager` (build templates). Code lives
under `pkg/`, almost all Linux-only.

gRPC services on :5008 (`pkg/server/`, `pkg/service/`, `pkg/template/server/`, `pkg/volumes/`):

- **SandboxService** — `Create`, `Update`, `List`, `Delete`, `Pause`, `Checkpoint`.
- **TemplateService** — `TemplateCreate`, `TemplateBuildStatus`, `TemplateBuildDelete` (template-manager role only).
- **InfoService** — node identity, roles, capacity, health status (used by API node discovery).
- **ChunkService / VolumeService** — peer-to-peer template chunk serving; persistent volumes.

Process shutdown, triggered by `SIGINT`, `SIGTERM`, `SIGUSR1`, or a service failure, moves
the node to `ShuttingDown` before draining builds and sandboxes. Like `Draining`, this excludes new
placement while existing work stays reachable, and `/health` returns HTTP 200 with status
`draining`. Unlike reversible `Draining`, `ShuttingDown` is terminal for the process;
status overrides cannot enter or leave it. `FORCE_STOP` skips waiting for sandboxes to exit.

After sandbox drain and snapshot uploads complete, sandbox proxy cleanup attempts graceful HTTP
server shutdown with a separate deadline, then forces server closure if needed. Successful forced
closure recovers graceful deadline expiry; other server errors still propagate. Pprof stays
available through service teardown; its final shutdown also has a deadline and forced-close fallback.

`InfoService.ServiceInfo` reports optional `outstanding_work` for sandbox, template-builder, and
mixed-role nodes. It counts overlapping work holds, not distinct sandboxes or builds, and includes
tracked background persistence and cleanup. Reporting nodes send an explicit zero when idle;
an absent field means unknown. The API caches this report and exposes it as optional top-level
`outstandingWork` in admin node list and detail responses, preserving explicit zero and omitting
unknown counts. This observational count does not authorize node deletion.

Key mechanisms (all under `pkg/sandbox/`):

- **Firecracker** (`fc/`): each sandbox is one Firecracker process in its own cgroup and network
  namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots.
  Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.
- **Lazy memory / UFFD** (`uffd/`): on resume, Firecracker restores the VM without loading
  memory; a userfaultfd handler serves page faults directly from the template's memfile, so only
  touched pages are read. An optional prefetcher warms known-hot pages.
- **Copy-on-write rootfs** (`rootfs/`, `nbd/`, `block/`): the template rootfs stays read-only;
  writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by
  an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.
- **Template cache** (`template/`): templates are fetched lazily from object storage and cached
  on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other
  nodes before upload completes).
- **Networking** (`network/`): each sandbox gets a slot — a network namespace with a veth pair
  and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall
  (with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and
  reused; slot indexes are allocated locally against the node's netns state (leftover
  namespaces from a previous run are torn down by startup reclaim).
- **Sandbox proxy** (:5007, `pkg/proxy/`): reverse-proxies incoming traffic from client-proxy to
  the sandbox's slot IP and requested port over HTTP or configured HTTPS, enforcing per-sandbox
  traffic access tokens. HTTPS backends may use self-signed certificates.
- Writes sandbox lifecycle **events** and cgroup **host stats** to ClickHouse; exports metrics via
  OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy
  collector remains the fallback primary destination, and configured shadow destinations can mirror
  writes during collector/storage migrations without changing sandbox behavior.

Detached sandbox-event publication holds node work until the publisher returns. This covers
delivery attempts, not end-to-end delivery: the ClickHouse target enqueues into an in-memory
batcher, whose flushing remains part of shutdown.

### Envd (`packages/envd`)

The agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.

- **Process service** (`spec/process/process.proto`): start/list/connect to processes, stream
  stdout/stderr, stdin, signals, PTYs — this is what SDKs use to "run code".
- **Filesystem service** (`spec/filesystem/filesystem.proto`): stat/list/make/move/remove/watch.
- **REST**: `/health`, `/metrics`, `/envs`, `/files` upload/download, `/init` (orchestrator pushes
  env vars, access token, metadata after boot/resume), `/upgrade` (live self-upgrade, below),
  freeze/thaw hooks used during pause.
- **Public vs. control-plane routes**: the control routes (`/init`, `/upgrade`, and the freeze/thaw
  hooks) are marked `x-internal: true` in `spec/envd.yaml`, and `/upgrade` — which the spec does not
  describe — is listed alongside them in the orchestrator's `pkg/sandbox/envd`. The orchestrator
  reaches them over the host network at the sandbox slot IP; the sandbox proxy refuses them with a
  404, for every method, so they are not reachable through a sandbox URL. Adding a control route
  therefore means marking it in the spec (`go generate` carries the marker into the proxy's
  rejection list) — otherwise it ships reachable from the internet.
- **Auth**: `X-Access-Token` header checked against a token delivered via Firecracker MMDS;
  signed URLs for file endpoints. `/init` is exempt (it is what *delivers* the token), which is the
  main reason the proxy refuses it outright.
- **Live upgrade** (`internal/services/process/upgrade.go`): an authenticated `POST /upgrade` lets
  the orchestrator swap envd inside a *running* sandbox at resume. It streams the new binary in the
  request body and envd `syscall.Exec`s into it **with the same PID**, carrying the workload's
  stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a
  tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade `/init` restores the
  access token (so no re-adopted process runs unauthenticated), and the handover outcome
  (procs/watchers re-adopted, plus any failures) rides back on that `/init`'s `X-Envd-Handover`
  header for fleet visibility.
- Scans guest ports and forwards them so any port a user process opens becomes reachable through
  sandbox URLs. **`pkg/version.go` must be bumped on every behavioral change** — the API and the
  orchestrator gate features on the envd version recorded in each template build.

### Client proxy (`packages/client-proxy`)

The stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates
`https://<port>-<sandboxID>.<domain>` requests (host parsing in `packages/shared/pkg/proxy/host.go`),
looks the sandbox up in the Redis routing record to find the owning node, and reverse-proxies to
that node's orchestrator proxy on :5007 by default. `ORCHESTRATOR_PROXY_PORT` selects a
different downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls
the API's `ResumeSandbox` gRPC and retries — paused sandboxes wake transparently on traffic.
By default the record is the API-owned `sandbox:catalog:{id}`; the `orchestrator-routing-prioritized`
flag switches the read to the orchestrator-owned `sandbox:routing:{id}` (see "Sandbox routing
records" below).

### Dashboard API (`packages/dashboard-api`)

A separate REST service (port 3010, spec `spec/openapi-dashboard.yml`) consumed by the web
dashboard, not the SDK: legacy team management/provisioning, template tags, build listings, admin
bootstrap. Its admin routes accept either a shared admin token or a short-lived service JWT
verified against the configured admin JWKS. The `disable-legacy-team-mutations` LaunchDarkly flag
rejects legacy lifecycle writes with 412 after authentication; it leaves reads and the workspace
API's management projection writes available. Team-scoped template and build read routes accept either dashboard user auth or
team API key auth (`X-API-Key`). Its workspace-agnostic `/v1/management` operations are defined in the
same dashboard OpenAPI contract and registered on the existing router. Their `AdminJWTAuth`
OpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api
`/.well-known/jwks.json` endpoint, with accepted signing methods derived from each JWK's required
`alg` metadata. Issuers and audiences are configured through the JSON `ADMIN_AUTH_PROVIDER_CONFIG` value —
the same config shape as `AUTH_PROVIDER_CONFIG`. Talks to Postgres and ClickHouse; never talks to
orchestrators.

An issuer normally configures at least one accepted audience, binding a JWT to its intended target.
An issuer may omit `audiences` only with no `audienceMatchPolicy`; this deliberately disables
audience matching while issuer, signature, and temporal-claim verification remain required.

The `/v1/management` operations are the cluster's half of a contract the workspace residency owns:
project upsert (a project is a `public.teams` row created from a caller-supplied UUID; the tier is
assigned once at creation from a local default and no push moves it; a changed slug renames the project, and nothing else follows it), per-member projection,
and limit sync (into `project_limits`, which `team_limits` reads in preference to `tiers`). All are
idempotent, because the caller is level-triggered and retries. `PUT
/v1/management/projects/{projectID}/members/{userID}` applies the desired presence for one user,
gated by a monotonic per-project/user revision stored in `projection.project_members`; duplicate or
older revisions succeed without changing target state. `PUT
/v1/management/projects/{projectID}/limits` is gated the same way, by a monotonic per-project
revision in `projection.project_limits`: the caller raises it whenever the limits it resolved for a
project change, and a delivery at or below the recorded revision is dropped and still answers 204.
The ledger and the values in `public.project_limits` advance in one transaction, so a revision is
never recorded without the values it admitted. Both fences are the target's, and both exist for the
same reason — the caller can only fence what it sends, so two deliveries in flight arrive in
whichever order the network gives them and the older one has to be refused where it lands. A present projection includes that User's
OIDC issuer/subject identities. Every projected user has at least one identity, and an identity
already owned by a different user returns 409. A revocation removes only that User's
`users_teams` row; projected Users and identities are retained. Membership writes live in
`internal/management` with their post-commit cache eviction rather than in the handlers: auth
caches member authorization, so each accepted command invalidates that User's authorization for
the Project after commit.

The management surface also owns a replay-safe cluster lifecycle. A caller registers a stable
cluster UUID with immutable connection details, assigns it only to the named project, detaches that
exact assignment before provider cleanup, and deletes the cluster only after no project references
it. Replaying the same registration or assignment succeeds, while a changed descriptor, a different
assignment, an inexact detach, or deletion of a referenced cluster returns a conflict. A new or
replacement assignment requires the project's tier identifier to contain `enterprise`,
case-insensitively; an identical assignment remains replayable after a later tier change.

`DELETE /v1/management/projects/{teamID}` is declared and answers 501. `envs`, `snapshots` and
`volumes` reference `teams` with `ON DELETE NO ACTION` and templates are only soft-deleted, so a
project that ever built one pins its team row — and releasing it needs the API service's
orchestrator connections, which this service does not have. Projects are not deleted from control
planes today.

## Data stores

| Store | Owner packages | What lives there |
|---|---|---|
| **PostgreSQL** | `packages/db` (goose migrations, sqlc) | Durable control-plane state: `teams`, `users`, `tiers` (quota defaults), `project_limits` (per-team quota overrides pushed in by the owning service; the `team_limits` view reads it in preference to `tiers`), `envs` (templates), `env_builds` (build rows: vcpu, ram_mb, status, versions), `env_aliases`, `snapshots` (paused sandboxes), `team_api_keys`, `volumes`, `clusters` |
| **Redis** | API, client-proxy, orchestrator | Ephemeral runtime state: running-sandbox store (source of truth), sandbox→node routing catalog, team/template/snapshot caches, rate limiting, P2P chunk peer registry |
| **ClickHouse** | `packages/clickhouse` | Time-series/analytics: `metrics_gauge`/`metrics_sum` (written by the OTel collector), `sandbox_events`, `sandbox_host_stats` (written by orchestrator), team metrics, and optionally `sandbox_logs` during the log migration. Read by API and dashboard-api |
| **Object storage** (GCS/S3/local, `packages/shared/pkg/storage`) | orchestrator, template-manager | Template & snapshot artifacts, keyed by build ID: `{buildID}/memfile`, `{buildID}/rootfs.ext4`, `{buildID}/snapfile`, `{buildID}/metadata.json` + `.header` index files |

A template and a paused-sandbox snapshot have the **same artifact shape** — a snapshot is just a
new build whose memfile/rootfs are stored as diffs against the template it came from (diff chains
are resolved through the `.header` files).

## Core flows

### Sandbox creation

```mermaid
sequenceDiagram
    autonumber
    participant C as SDK
    participant API as API
    participant R as Redis
    participant O as Orchestrator (chosen node)
    participant FC as Firecracker
    participant E as envd (in VM)

    C->>API: POST /sandboxes {templateID}
    API->>API: auth team, resolve template alias → ready build (Postgres/cache)
    API->>API: best-of-K placement → pick node
    API->>O: gRPC SandboxService.Create(SandboxConfig)
    O->>O: fetch template (local cache / NFS / object storage)
    O->>O: acquire network slot + NBD rootfs overlay + uffd memory
    O->>FC: load snapshot, resume VM
    O->>E: POST /init (env vars, access token) — retried until ready
    E-->>O: 204
    O-->>API: Create OK
    API->>R: store running sandbox + routing catalog entry
    API-->>C: 201 sandbox {sandboxID, domain}
```

The API blocks on the gRPC `Create`, which itself blocks on envd's `/init` — when the client
gets a response, the sandbox is fully usable. Fresh creates are internally a *resume* of the
template's base snapshot (cold boots happen for filesystem-only templates and builds, or when
an explicit resume requests one — see pause and resume below; template creates never do).

### Sandbox traffic

```mermaid
sequenceDiagram
    autonumber
    participant U as Client
    participant CP as client-proxy :3002
    participant R as Redis catalog
    participant API as API
    participant OP as orchestrator proxy :5007
    participant E as envd / user process

    U->>CP: https://3000-i7fa3.domain
    CP->>CP: parse host → port 3000, sandbox i7fa3
    CP->>R: GetSandbox(i7fa3)
    alt running
        R-->>CP: node IP
    else paused / unknown
        CP->>API: gRPC ResumeSandbox(i7fa3)
        API-->>CP: node IP (after resume)
    end
    CP->>OP: forward to http://nodeIP:5007
    OP->>OP: lookup sandbox, check traffic access token
    OP->>E: http://slotIP:3000 (via veth/tap into VM)
    E-->>U: response
```

### Sandbox routing records

client-proxy resolves the node IP of a sandbox from a routing record in Redis. Two records exist
today. Both have the same JSON shape (`sandbox_catalog.SandboxInfo` in
`packages/shared/pkg/sandbox-catalog`): `orchestrator_id`, `orchestrator_ip`, `execution_id`,
`sandbox_started_at`, `sandbox_max_length_in_hours`.

| Record | Key | Writer | Written | Deleted |
|---|---|---|---|---|
| API-owned (default) | `sandbox:catalog:{sandboxID}` | API (cloud) or the cluster edge from gRPC metadata (BYOC) | after `Create` returns | before `Pause`/`Kill` is sent to the node |
| Orchestrator-owned (v1, flag-gated) | `sandbox:routing:{sandboxID}` | orchestrator, `packages/orchestrator/pkg/routing` | on `MarkRunning` (sandbox enters the live map, envd is ready) | on `MarkStopping` (kill, pause, checkpoint, crash) |

**The API-owned record is still the source of truth.** client-proxy reads `sandbox:catalog:{id}`
unless the `orchestrator-routing-prioritized` flag is on. The orchestrator-owned record is a v1
test path. It runs next to the API path and does not replace it yet.

Two feature flags in `packages/shared/pkg/featureflags` control the new path:

- `orchestrator-routing-publish` (orchestrator): write `sandbox:routing:{id}` on `MarkRunning` and
  delete it on `MarkStopping`. A failed write is logged and counted
  (`orchestrator.routing.publish.total{result=error}`); the sandbox keeps running. Build sandboxes
  are skipped. The delete is guarded by `execution_id` in a Lua script, so a stale lifecycle never
  removes the record of a newer execution.
- `orchestrator-routing-prioritized` (client-proxy): resolve the node from `sandbox:routing:{id}`
  instead of `sandbox:catalog:{id}`. There is no fallback to the API-owned record on a miss. A miss
  goes to the auto-resume path (`ResumeSandbox` gRPC to the API), same as today.

Rollout order: turn on `orchestrator-routing-publish` first and wait one maximum sandbox length,
so every live sandbox has a record. Then turn on `orchestrator-routing-prioritized`. To roll back,
turn off `orchestrator-routing-prioritized`; the API path is untouched.

The TTL of both records is `sandbox_max_length_in_hours` from the write time. The record is
deleted earlier in every normal stop path.

### Volume content

Persistent volumes (`packages/orchestrator/pkg/volumes/`) are managed through the control-plane
API (`POST/GET /volumes`), but their **content** — reading and writing files — is served by a
separate volume-content API (belt, `e2b-dev/belt`) that the SDK talks to directly, not through the
control-plane API. The API's role is to mint the credential and tell the SDK where to send content
traffic.

```mermaid
sequenceDiagram
    autonumber
    participant U as SDK
    participant API as API
    participant PG as PostgreSQL
    participant VC as volume-content API (belt)

    U->>API: POST /volumes (create) or GET /volumes/{id}
    API->>PG: persist / load volume row
    API->>API: mint JWT (aud = https://api.#lt;domain#gt;)<br/>resolve domain
    API-->>U: { volumeID, name, token, domain? }
    Note over U: domain is returned only for BYOC teams.<br/>SDK stores it and falls back to api.#lt;E2B_DOMAIN#gt; otherwise
    U->>VC: /volumecontent/{id}/... at api.#lt;domain#gt;<br/>Authorization: Bearer token
    VC->>VC: verify token (audience must match its own origin)
    VC-->>U: file content
```

- **Domain selection.** The token's audience and the content host are the same origin,
  `https://api.<domain>`. For teams on a **custom (BYOC) cluster** (`team.ClusterID` set), the API
  returns that cluster's domain (`cluster.SandboxDomain`, resolved in
  `handlers.volumeContentDomain`) so content traffic goes to the BYOC cluster's edge instead of the
  control-plane host. For teams on the default cluster the response omits `domain` and the SDK uses
  its configured default (`api.<E2B_DOMAIN>`); the audience then uses the deployment's `DOMAIN_NAME`.
- **Token.** A short-lived JWT (`handlers.generateVolumeContentToken`, config in
  `cfg.VolumesTokenConfig`) signed by the API, scoped to the team and volume, presented as a bearer
  token on every content request. Its `aud` claim is `https://api.<domain>`, so a token minted for
  one cluster's origin is not accepted by another.

### Pause and resume

- **Pause**: API records a snapshot row in Postgres, then gRPC `Pause` to the node. The
  orchestrator pauses the VM, snapshots it, diffs memory (dirty-page tracking) and rootfs (COW
  cache) against the template, caches the snapshot locally, and uploads asynchronously to object
  storage (with a retry budget). The sandbox leaves the Redis catalog.
  - **Deferred rootfs export** (gated by the `deferred-rootfs-export` flag in
    `packages/shared/pkg/featureflags`): instead of diffing the rootfs on the pause critical
    path, the orchestrator ejects the writable COW cache during pause and returns, then seals it
    into the rootfs diff (reflink) in the background. This moves the rootfs-diff latency off the
    pause, but the local snapshot's rootfs body isn't materialized until the seal finishes, so the
    async upload — and any origin-node resume/prefetch that reads the rootfs diff — waits on the
    seal. A seal failure is permanent (it never re-runs), so the upload fails fast rather than
    retrying.
- **Resume**: same path as creation, but placement prefers the **origin node** — if the snapshot
  is still in its local cache, resume avoids any object-storage reads. `Checkpoint` is a
  pause+resume in place used to persist state while keeping the sandbox running.
- **Explicit filesystem-only resume**: `memory: false` on resume/connect demands a cold boot
  (`RebootSandbox`) even when the snapshot includes memory, as a self-serve rescue when the
  restored memory state is unusable. Gated per team by the `fs-only-resume-api` flag; when off
  the request is rejected with an explicit error, never silently downgraded to a memory restore.
  The disk has crash-recovery semantics (unflushed pre-pause writes are lost), nothing durable
  is mutated (the memory snapshot survives untouched), and auto-resume never takes this path —
  traffic always memory-resumes. The one way a memory-preferring sandbox ends up with a
  filesystem-only snapshot is at pause time, not resume time: when the node keeps refusing a
  timeout auto-pause (its parent memfile header still deduplicating after the admission grace)
  for longer than the retry budget counted from the first refusal
  (`auto-pause-overstay-budget-milliseconds`, 120 s by default; 0 degrades at the first
  refusal, negative never degrades), the evictor requests a filesystem-only snapshot instead, so
  the sandbox stops overstaying its expiry; the next resume of that snapshot is a cold boot. The
  degrade is only ever decided on a refusal in the same sweep, so eviction lag alone never
  degrades anything, and refusals only survive with `pause-refusal-restore` on. On a
  BYOC cluster that flag also needs every edge replica on a release that restores the route
  after a refusal: roll the edge fully first, and turn the flag off for the cluster before any
  edge rollback — nothing in the cluster model can check the edge's version.
- **Pre-boot filesystem recovery**: every cold boot of a rootfs that was not frozen at pause
  (`fs_quiesced` false/absent) runs a jailed `e2fsck -p -E journal_only` before the VM starts —
  journal replay only, the same recovery the guest kernel would do at mount — so a `memory: false`
  rescue and a legacy sync-fallback filesystem-only snapshot both mount a consistent disk. It
  replays and exits without a full consistency scan, so the cost is bounded by journal content,
  not filesystem size. Runs under the same confinement as the offline envd swap (unprivileged
  transient unit, device access pinned to the sandbox's own NBD node). A clean replay boots;
  anything else fails the start with the snapshot untouched but stays retryable. Journal replay
  never condemns a snapshot: its exit codes cannot tell an unmountable filesystem apart from a
  transient device fault, so every non-replayed outcome — an operational failure (timeout, I/O,
  device error) or an e2fsck exit that is not a clean replay — is retryable, never a permanent
  customer-facing verdict. In-file corruption that still mounts is likewise not condemned — replay
  does not scan, so it boots. Whole-filesystem repair and condemning a rootfs are left to a
  separate opt-in full-filesystem repair path. Gated by the `preboot-fs-recovery` flag, separate
  from `fs-only-resume-api` because it also changes the behavior of existing filesystem-only cold
  boots.
- **Envd live-upgrade on resume**: the orchestrator can upgrade the sandbox's envd to a newer
  node-local build during resume (gated by the `envd-upgrade-target` flag in
  `packages/shared/pkg/featureflags`), via envd's `POST /upgrade` (see the envd section). It is
  best-effort — a delivery failure before the `exec` leaves the old envd serving — except an
  unrecoverable post-`exec` failure (the new envd never re-initializes), which fails the resume
  rather than return a permanently unusable sandbox.
- **Envd offline-upgrade on cold-boot resume**: reaches envd too old for the live `/upgrade`
  handover (below `MinEnvdVersionForUpgrade`). When a *filesystem-only* snapshot cold-boots
  (`RebootSandbox`), the orchestrator rewrites `/usr/bin/envd` in the rootfs **before** the VM
  boots (`PreBootFn` → `pkg/sandbox/rootfs.SwapEnvdBinary`), entirely in userspace via a jailed
  `debugfs` — never a host-kernel mount of the tenant image. The old envd never participates, so
  the method is version-agnostic. Gated by the `envd-offline-upgrade-target` flag (a sibling of
  `envd-upgrade-target` sharing the same version-remap resolver), and applied only when the
  snapshot's rootfs was captured frozen (`fs_quiesced`, so it is crash-consistent); best-effort
  (a swap failure boots the original envd). Because the swap keys on the snapshot's *built-with*
  version, which it does not advance, it re-fires idempotently on each cold-boot resume until a
  re-pause re-bakes the running version.
- **Both upgrade paths read the host envd binary through a node-local cache** (gated by the
  `envd-binary-cache` flag): the version probe, the live delivery and the offline swap's staging
  copy all read a local copy rather than the read-only artifact mount, which an `exec` would
  otherwise demand-page in small random reads. The cache is why neither path performs bulk I/O
  against that mount on the resume path.

  It adds a third condition to both paths, so an eligible snapshot is not unconditionally
  upgraded: an upgrade whose binary is not cached on this node is **deferred to a later resume**
  rather than served from the mount, since reading it there would put tens of seconds on a path a
  customer is waiting on. A node whose cache is cold — any miss, and in particular a ramp keyed on
  sandbox or team, where the boot-time warm does not evaluate — defers that upgrade while a
  background warm populates the cache; on a node-scoped rule the boot warm normally lands before
  the first resume. Both upgrades are idempotent and re-fire per resume, so a deferral costs one
  cycle. Ramp the flag on a node-scoped context kind — the cache is
  shared by every sandbox on the node.
- Auto-pause/auto-resume make sandboxes effectively serverless: idle sandboxes pause, traffic
  resumes them (see traffic flow above).

### Template build

```mermaid
sequenceDiagram
    autonumber
    participant C as SDK
    participant API as API
    participant TM as template-manager (build node)
    participant FC as Firecracker build VMs
    participant OS as Object storage

    C->>API: POST /v3/templates (register build: cpu, ram, free-disk target) → Postgres env_builds
    C->>API: POST /v2/templates/{id}/builds/{buildID} (recipe: steps, start/ready cmd)
    API->>TM: gRPC TemplateCreate(TemplateConfig)
    TM->>TM: pull image → inject envd/provisioning → extract ext4 rootfs
    TM->>FC: boot VM per phase: provision → user steps
    TM->>TM: resize disk on host
    TM->>FC: boot VM per phase: finalize → optimize
    TM->>OS: upload layers + final {buildID}/memfile, rootfs.ext4, snapfile, metadata
    API->>TM: poll TemplateBuildStatus
    API->>API: mark build ready in Postgres
```

Builds are **layered** (`pkg/template/build/phases/`): base → user → one layer per recipe step →
resize disk → finalize → optimize. Each layer is hashed and cached, so rebuilds only re-run changed
steps. Resize disk grows the quiescent rootfs on the host; the other non-cached phases run in a real
Firecracker VM and their pause-diffs become layers. The optimize phase records which memory pages a
fresh resume touches, producing prefetch hints that speed up future sandbox starts.

Template creation holds node work through foreground setup and asynchronous build completion,
including layer uploads, synchronous cleanup, and final status publication. Cancellation can mark
a build failed before execution finishes; the work hold remains until execution unwinds. Template
deletion owns a separate hold until artifact cleanup returns.

## Deployment topology

The services are scheduler-agnostic binaries and containers; the supported way to run them is the
Kubernetes-based distribution. The roles below hold regardless of how the nodes are provisioned.

```mermaid
flowchart TB
    LB["Load balancer + TLS<br/>api.* → API | *.domain → client-proxy"]

    subgraph control["control plane"]
        AJ["api, dashboard-api, client-proxy,<br/>redis, otel-collector"]
    end
    subgraph sandboxnodes["sandbox nodes (autoscaled)"]
        OJ["orchestrator<br/>+ Firecracker sandboxes"]
    end
    subgraph buildnodes["build nodes (autoscaled)"]
        TJ["template-manager"]
    end
    subgraph ch["analytics"]
        CJ["clickhouse"]
    end

    LB --> control
    AJ -->|gRPC| OJ & TJ
```

- **Control-plane services** (api, dashboard-api, client-proxy) are stateless containers and the
  only LB backends. The api discovers orchestrator nodes through
  `packages/shared/pkg/servicediscovery` (Kubernetes, DNS, a static list, or the legacy Nomad
  backend the code still carries).
- **Sandbox nodes** run the orchestrator directly on the host (it needs root for Firecracker,
  namespaces, NBD, cgroups), configured with hugepages and local template caches. Autoscaled.
- **Build nodes** run the same binary in template-manager mode.
- PostgreSQL is external (connection string via secrets); Redis is a managed service or a
  single in-cluster instance; ClickHouse runs on its own nodes.
- Observability: everything exports OTel; the collector fans out to ClickHouse (product metrics)
  and Grafana Cloud/stack. Logs default to the legacy Vector → Loki path; dynamic log routing can
  select a primary collector and shadow collectors, and local-cluster log reads can be switched to
  ClickHouse with `logs-read-config` after `sandbox_logs` is populated (`LOGS_READ_CONFIG` is the
  flag's fallback where LaunchDarkly has no value). Once reads are on ClickHouse, Loki can be left
  out of a deployment and the api started without `LOKI_URL`.

## Repository layout

```
packages/
  api/                  Control-plane REST API
  orchestrator/         Sandbox runtime + template builder (one binary, per-node)
  client-proxy/         Edge router for sandbox traffic
  envd/                 In-VM agent (bump pkg/version.go on behavior change!)
  dashboard-api/        Web-dashboard backend
  shared/               Protos, telemetry, storage clients, proxy engine, feature flags
  auth/                 AuthN library (API keys, JWT/OIDC) used by api + dashboard-api
  db/                   Postgres migrations (goose) + queries (sqlc)
  clickhouse/           ClickHouse schema, batching writers, query clients
  otel-collector/       Collector config
  nomad-nodepool-apm/   Nomad autoscaler metric and deployment-aware target plugins
  local-dev/            docker-compose local stack + DB seeding
spec/                   OpenAPI specs (public, edge, dashboard) — codegen sources
tests/integration/      Integration tests against a live deployment
```

Cross-service contracts are all generated: OpenAPI specs in `spec/`, gRPC protos in
`packages/orchestrator/*.proto` and `packages/envd/spec/`, SQL in `packages/db/queries/`.
Run `make generate` after changing any of them.