# SPACE: Architecture
[Documentation](../README.md)
A sandbox, a lifecycle operation, and a command stream have separate lifetimes. A sandbox can outlive both the request that created it and the connection carrying its output.
## Core concepts
| Concept | Meaning |
|---|---|
| Template | The starting filesystem and runtime configuration. Many sandboxes can use the same template without changing it. |
| Sandbox | An isolated microVM with its own writable filesystem and lifecycle. |
| Process | A program executing inside a sandbox. Processes in the same sandbox share its environment and filesystem. |
| Command session | Connection information and authorization for one command. Creating the session assigns a process ID; attaching starts the process. |
| Operation | A record tracking an asynchronous create, pause, suspend, wake, or delete request. |
A template reference such as `seed:latest` selects a published release; a digest reference selects specific content. A sandbox response reports the resolved template digest so callers can record what they ran.
Operators can restrict newly registered template artifacts with
`SPACE_TRUSTED_TEMPLATE_STORAGE_BUCKETS` on the internal API: a comma-separated
list of exact S3 bucket names. A configured policy rejects other buckets and
all endpoint overrides before writing the release or its tags. An unset or
empty policy preserves local S3-compatible storage. This policy does not audit
existing releases or replace publisher authorization; provision the allowlist
before promoting an enforcement image. Set the same list on the cluster
controller, whose template campaign driver uses only the storage locations it
admits. With the list empty there, the driver refuses locations that override
the endpoint unless `SPACE_TEMPLATE_STORAGE_ALLOW_ENDPOINT_OVERRIDES` opts in
for S3-compatible storage at an `https://` endpoint, and the controller logs a
warning naming the variable. It refuses a plain `http://` endpoint unless
`SPACE_TEMPLATE_STORAGE_ALLOW_PLAINTEXT_ENDPOINTS` opts in for local storage
such as the Kind object store. A twin keeps only the locations the driver admits, and a release
with none fails its item once its attempts run out.
## Template lookup rollout
Template lookup clients attach scope metadata and compatible servers return a
receipt. A missing receipt preserves the existing lookup behavior during a
rolling upgrade. A mismatched receipt fails the request. Authentication,
organization checks, and cache decisions continue to use the existing rules.
The `auth.template_lookup.compatibility` counter records `missing_scope`,
`missing_receipt`, and `received`, under the emitting service's identity. A
receipt confirms only that one peer received the metadata. It does not certify
that a downstream peer received it or that the scope was authorized. Check
all participating services and exercise cache misses as well as hits; an idle
service or a cached lookup cannot prove the catalog supports the protocol.
Deploy this compatibility phase before introducing stricter lookup rules.
Strict enforcement requires a separate authorization acknowledgement; it must
not accept the compatibility receipt as proof. Existing clients that omit scope
must be migrated before enforcing it. Rollback to the previous version remains
compatible with these requests and replies.
## Request and execution flow
The API records desired state in PostgreSQL. Controllers and worker services reconcile the running system with that state, so an accepted request is not a readiness signal.
SPACE can manage workers directly or through Kubernetes. The [direct adapter](../crates/l40_api/api/src/controllers/substrate/direct.rs) calls worker services without creating Kubernetes sandbox pods. The [Kubernetes adapter](../crates/l40_api/api/src/controllers/substrate/kubernetes/mod.rs) is used by the local Kind installation in these guides.
### Local Kind topology
The diagram below shows the Kind deployment used in these guides, not a requirement for every SPACE deployment.
Scroll horizontally to view the full diagram.
```mermaid
flowchart TB
accTitle: SPACE local Kind request and execution flow
accDescr: Management requests and command streams follow separate paths. The API stores desired state in PostgreSQL. The cluster controller, Kubernetes sandbox pod, runtime shim and worker services arrange the Firecracker microVM and guest process. Worker services exchange templates and snapshots with object storage. The data plane connects the application to the guest process.
application["Application"]
api["API"]
postgres[("PostgreSQL")]
controller["Cluster controller"]
pod["Kubernetes sandbox pod"]
shim["SPACE runtime shim"]
workers["Worker services"]
storage[("Object storage
templates / snapshots")]
vm["Firecracker microVM"]
data["Data plane"]
guest["Guest process"]
application -->|Management| api
api --> postgres
postgres -->|desired state| controller
controller --> pod
pod --> shim
shim --> workers
workers <--> storage
workers --> vm
vm --> guest
application -->|Command stream| data
data <--> guest
```
Diagram text description
Management follows **Application → API → PostgreSQL**. Desired state flows from
**PostgreSQL → Cluster controller → Kubernetes sandbox pod → SPACE runtime shim
→ Worker services → Firecracker microVM → Guest process**. Worker services
exchange **templates/snapshots** with **Object storage** in both directions.
The separate command-stream path is **Application → Data plane ↔ Guest process**.
- **Create:** The API resolves the template and accepts the requested sandbox. The client retains the sandbox ID and operation ID.
- **Become ready:** Controllers arrange execution through the selected adapter and worker services. In Kind, this includes Kubernetes sandbox pods and the runtime shim. The client waits for operation success, a `running` sandbox, and a reachable process endpoint.
- **Execute:** The API creates a command session. The client attaches to the returned events endpoint, starting the guest process and receiving output.
- **Finish:** The client checks the process exit result, then keeps the sandbox for another command or deletes it. A completed command does not itself delete the sandbox.
An operation's `state` describes the lifecycle request; the sandbox's `state` describes the environment. Neither is the exit status of a command. A create, or a wake or start that leaves the sandbox `running`, succeeds once the sandbox's guest agent answers, so a command or file call sent after it does not wait for the guest to boot. On Kubernetes, a create whose guest does not answer within its readiness budget can still succeed with the sandbox `running`, and the first call then waits for the guest. A Native create's budget for that is 5 seconds. A create retried after an earlier attempt recorded its sandbox can also succeed before the guest answers.
Native create, pause, suspend, wake, and delete operations have a 30-minute deadline from when they were requested. One that has started and is still retrying when it passes fails with `operation_failed` and the reason `operation_deadline_exceeded`. A create, pause, suspend, or wake that has not started by then, because it waited behind earlier work on its sandbox, for a free execution slot, or for the cluster controller to come back, ends `expired` with the same code and reason instead of starting late. Inspect the sandbox before requesting it again. A delete still runs however late it starts: expiring it would leave running a sandbox its owner asked to remove.
Requesting a delete cancels the sandbox's create if that create is still queued or waiting to retry: the create ends `canceled`, with `operation_failed` and the reason `superseded_by_delete`, and the delete proceeds. A create the delete cannot cancel, because its attempt was running or its claim had gone stale, still holds the delete back until that attempt ends; if the attempt would retry, or the dispatcher hands the create back without running it, the create ends `canceled` instead, so it never runs after the delete, and its error keeps the attempt's last progress message and error fields beside `superseded_by_delete`. The delete's request and a create's requeue each lock the create first, so a delete arriving as an attempt ends is always seen by one of them. One case still requeues the create: a dispatcher that reclaims a create whose earlier execution lost its claim but still runs in the same process hands it back unchanged, since canceling it would let the delete start under that execution; the delete then waits for one more attempt.
A Native create can answer `202` before its execution records the sandbox. A `DELETE /v1/sandboxes/{sandbox_id}` in that window is accepted while the create is queued or running, for a caller who could see that create's operation: it answers with a delete operation, as for a sandbox already `creating`, and cancels or waits for the create as described above. A create that succeeds records the sandbox before it is marked succeeded, so a delete that races it still finds the sandbox. A create that fails, expires, or is canceled by an earlier delete can end without recording the sandbox, and a delete with a new `Idempotency-Key` then answers `404`. Every other Native request naming the sandbox, such as `GET /v1/sandboxes/{sandbox_id}`, answers `404` until the sandbox is recorded; poll the create operation instead.
### Inline-durable execution
Native lifecycle operations are durable records in PostgreSQL. By default the
cluster controller's dispatcher claims and runs each one: it runs on the
elected leader only, with a bounded number of execution slots and a
per-organization slot limit. With inline execution on, the API process that
admits a cold create, a `:wake`, `:resume` or `:start`, or the wake a command,
process session or attach queues, can claim and run that operation itself, in
the same transaction that records it. It does so only when:
- the env master `SPACE_NATIVE_INLINE_OPERATIONS_ENABLED` is on, the kind is in
`SPACE_NATIVE_INLINE_KINDS`, and the `space-native-inline-operations` flag
allows the organization;
- nothing else for the sandbox is queued or running;
- the process has one of its `SPACE_NATIVE_INLINE_MAX_CONCURRENT` permits free
and is not draining. Admission takes the permit only here, after it has
created the operation and found nothing ahead of it, so a request that joins
other work, or waits on the sandbox's locks, holds none;
- the organization is under `SPACE_NATIVE_INLINE_ORG_LIMIT`.
Anything else records the operation queued exactly as before, for the
dispatcher. [`configuration.md`](configuration.md) describes each setting.
A create that wins an admission-time warm-pool claim runs inline under its
own switch, `SPACE_NATIVE_CREATE_INLINE_ENABLED`, not these.
The master goes on only once every API process, on every surface, takes the
per-sandbox queue lock below, and off before the dispatcher's keyed claim:
the admission claim is sound only while both hold.
The record, the claim and the execution are the dispatcher's. The API runs the
same executors through the same driver, so claim renewal, retry deferral and
the operation deadline behave the same, and every retry is requeued for the
dispatcher. The claim commits in the task that runs the operation, not in
the request, so a client that disconnects mid-commit cannot leave a claim
that nothing runs. If the API process dies mid-execution, its claim stops being
renewed. After the claim staleness window (60 s) the dispatcher claims the
same operation again, so nothing is lost, though that operation finishes later
than a queued one would have. A graceful API shutdown waits for running inline
work within its drain budget and queues new work meanwhile.
Inline claims keep one execution per sandbox, in request order:
- Every insert for a sandbox takes a per-sandbox queue lock until it commits.
So the admission's check that nothing else is pending sees everything
inserted before it. A later insert commits only after the claim, and the
dispatcher's claim then sees that claim live and holds the later operation
back.
- A caller's `:wake` joins a system wake only while that wake is still queued.
Behind one the API is already running, the `:wake` is queued and completes as
already done.
Two limits are counted separately. The organization limit counts the
organization's live inline claims and is soft: admissions that commit at the
same moment can pass it by up to their number less one. The dispatcher's
per-organization slot limit does not count inline claims, and the inline limit
does not count dispatcher claims. Suspend, pause, stop, delete and kill always
run on the dispatcher.
In Kind, sandbox pods run in their own namespace, `sandboxes`, apart from the control plane in `space`. NetworkPolicies there admit only the API, the cluster controller and the data plane, on the port they dial, and limit sandbox egress to the public IPv4 internet and cluster DNS; the [Kind harness README](../infra/space/kind/README.md#sandbox-namespace-and-network-policy) lists each rule's source.
## Lifecycle and persistence
[Snapshot upload streaming](snapshot-upload-streaming.md) describes the worker
transfer pipeline, memory bounds, failure ownership, and rollout behavior.
Pause and suspend both stop execution, but retain state differently. Use wake to return a paused or suspended sandbox to running, and wait for its operation and readiness before issuing another command.
| State | Process execution | What is retained | Next action |
|---|---|---|---|
| Running | Processes can execute. | Live VM memory and writable filesystem. | Run more commands, pause, suspend, or delete. |
| Paused | Execution is paused. | VM memory remains resident and the filesystem remains on the worker. This is not a durable snapshot. | Wake to resume. |
| Suspended | No guest execution. | A snapshot of VM memory, runtime state, and root filesystem; active VM resources are released. | Wake to restore. |
| Deleted | No execution. | The sandbox is no longer available through the API. | Create another sandbox if needed. |
The stored vCPU shape is the sandbox's activation allocation. A live CPU update
changes the current execution's scheduling quota within its admitted ceiling;
it does not resize the guest or replace that stored shape. Pause/resume keeps
the live quota. A new start or restore returns to the activation allocation,
so a temporary reduction does not permanently prevent raising the quota again.
By default, sandboxes created through the SPACE API suspend after six hours without access or command activity. This is an idle limit, not a maximum lifetime.
A created or overwritten file that can replace its destination is published
atomically when its writes complete, and persisted at lifecycle boundaries,
not per write: an acknowledged one is not guaranteed to survive a sandbox
crash, forced stop or host failure. [Guest file and disk
I/O](space/design/guest-io.md) describes the guest's disk, its per-sandbox
limits, the upload and read paths, and what each lifecycle operation persists.
The **E2B Compatibility Wrapper (ECW)** is SPACE's E2B-compatible API surface.
It translates E2B-compatible requests and responses at the edge and is distinct
from the SPACE API. The [ECW guide](ecw.md) defines its tested compatibility
contract and configuration. ECW uses different lifecycle names: its pause maps
to SPACE suspend, and its resume maps to SPACE restore.
Service-port ingress has a separate connection lifetime. With ingress protection
enabled and completion framing negotiated, automatic lifecycle transitions defer until the HTTP response body or
WebSocket connection completes; explicit lifecycle operations retain their
existing authority. Ingress wake authorizes access and respects the
sandbox's access-wake policy before preparing a dormant backend. Route resolution
first prepares the current worker path. If activation or recovery is needed,
the cluster API delegates through the authenticated `IngressActivationApi` to
the API process's existing lifecycle controller. It re-resolves and reauthorizes
after activation, within the original request deadline. The cluster API does
not construct a Kubernetes substrate or retain the sandbox-pod reflector cache.
Accepted activation remains owned by the API's tracked lifecycle tasks and
existing execution leases; a caller timeout only ends its wait. Transport idle
limits still apply, and SPACE does not generate keepalives on proxied
service-port WebSockets.
During coordinated deployment, a temporary metadata opt-in falls back to ordinary
stream termination for older callers or gateways. Drain existing attachments
before rollback; old decoding forwarders are not a supported steady state. The
internal API retains explicit loopback preparation until all workers provide
attachment readiness. Fallback connections do not provide full downstream
lifetime protection, so retain caller renewals. Remove this metadata opt-in and
the fallback branches after deployment is complete. Reconnect deadline updates require a
specific worker response-metadata acknowledgement; ordinary touch success cannot
stand in for a successful extension. Reconnect extends a deadline floor, not
the configured idle window; changing that window requires an explicit timeout
update. When API identity publication lags, deadline operations obtain the
current execution generation from the worker before applying a fenced mutation.
This lookup remains necessary after rollout; missing API identity is not proof
of a dormant execution. Ingress protection does not cover unrelated background
operations.
Accepted ingress activation survives request cancellation. The API's HTTP and
activation listeners share its shutdown signal and drain budget. Draining is
best-effort within the deployment termination grace; existing activation leases
provide recovery if the process exits before completion.
### Lifecycle policy at activation
Timeout updates use durable execution ownership rather than a lifecycle label
alone. A Kubernetes sandbox can retain lifecycle `started` after its execution
has retired and placement has cleared. With no execution, activation lease,
route, or node assignment, the policy update succeeds without waking it.
Direct-mode missing placement does not provide the same proof of absence.
The next activation reads policy after claiming execution ownership. Start
(including snapshot restore), Wake, and resident Resume can carry that policy
to the worker. The worker installs it atomically with activation admission,
after snapshot materialization and before arming inactivity timers. Omission
preserves the worker policy; local overrides remain authoritative. Retrying an
already-running activation does not replace its policy or reset its deadline.
Resident policy updates compare the observed execution generation atomically
with resume admission. An absent generation matches only a legacy resident
that still has none. Wake carries this resume fence separately from fresh
activation authority. If the resident retires after observation, Wake requests
a fresh identity through the normal activation handshake. A fence without
policy is invalid; policy omission preserves legacy behavior without a resume
fence. Capability checks use the resolved runtime connection rather than
asynchronous placement observations.
An identified execution still starting may be observed within the original
operation deadline before a timeout reset is sent. Observation does not start
another execution or retarget a replacement generation. The reset is sent once:
an ambiguous transport failure does not establish that repeating a clock reset
is safe.
### Native files on the data plane
The data plane can also answer Native `files:*` requests (`SPACE_DATA_PLANE_NATIVE_FILES=serve`), under `/v1` and `/api/v1` as the API mounts them. A deployment routes those paths to it with its load balancer. The API keeps its own `files:*` routes, so any request that reaches it is still answered there. The data plane carries the request's bytes and the API makes every decision, through two RPCs on its activation listener ([native file access](../proto/node/api/v1/native_file_access.proto)):
1. A `files:write` or `files:write_batch`, or any request whose body size is not known in advance, is more than 16 KiB, or that sends `Expect: 100-continue`, is authorized before its body is read: the data plane asks the API to authorize it from its headers. The API runs its public route's own credential check, owner scope and error rendering, so such a request without valid credentials is refused before its body is read. A refusal keeps the status, body and headers the API rendered, except the hop-by-hop headers, `Content-Length` and `Transfer-Encoding`, which the data plane sets for its own connection. A smaller body that has not fully arrived 100 ms after its request did is authorized the same way before the data plane waits for the rest of it, so a client that stalls its body is refused as promptly; one that is authorized continues with the bytes already read.
2. With Native upload admission on (`SPACE_DATA_PLANE_NATIVE_UPLOAD_ADMISSION`), a `files:write` or `files:write_batch` then reserves its declared `Content-Length` from the data plane's Native upload budgets before its body is read.
3. The data plane reads and checks the body with the same rules as the API's handler. For write, write_batch, mkdir, remove and rename it computes the request's fingerprint. When a body it checked before asking the API fails, it asks the API to authorize the request and answers with the API's refusal if there is one, so a request is answered as the API's route answers it.
4. It asks the API to prepare the call, which authorizes the request in the same call; for a request with a small body this is the only call to the API. For those five operations the API binds the `Idempotency-Key` to the fingerprint, refusing a different request under it with 409 `idempotency_conflict`; `files:changes_enable` and `files:changes_disable` require a key but have no fingerprint, so nothing is bound for them. The API runs the drain gate and the admission charge, and wakes the sandbox if its policy allows. It answers with the runtime to call and a route token.
5. The data plane calls the sandbox's runtime directly and maps a failure as the API does. A call whose connection to the runtime fails before the runtime answers gets the retryable 503 `service_unavailable` with `Retry-After`, from either server; a read-only call (`stat`, `list`, `read`, `modified`) is first sent once more over a new connection, since a connection kept to a sandbox endpoint can be dead when used. A mutation is not sent again, and one that nothing deduplicates is answered as not retryable, since it may have reached the runtime.
Nothing is cached between calls, so a revoked credential is refused on its next request. A key that `files:write`, `files:write_batch`, `files:mkdir`, `files:remove` or `files:rename` bound through either server binds the same record, so a retry of those may land on either. A data-plane replica that is draining answers new files requests with the Native 503 `service_unavailable` and lets the ones it took finish. A request it took may first take as long as its client needs to send its body, then wait up to 135 s for its sandbox to wake, and then transfer its body, so size its drain deadline and the load balancer's deregistration delay to cover all three. If the API cannot be reached, or predates these RPCs, the data plane answers a retryable 503. It does not wait out the RPC's deadline for that: a dial to the listener that goes unanswered gives up after a second, and the data plane closes a connection to it that stops answering HTTP/2 pings within about four seconds of its last answer.
## Service responsibilities
| Component | Responsibility |
|---|---|
| API and PostgreSQL | Authenticate and authorize requests; store control-plane records and desired state. |
| Controllers and adapters | Coordinate sandbox lifecycle changes with worker services. In Kind, the Kubernetes adapter, cluster controller, and runtime shim create and manage sandbox pods. |
| Worker services | Manage VMs, networking, volumes, templates, and credentials. btrfs copy-on-write supports root filesystem creation. |
| Data plane | Connect clients to guest processes and forward events using scoped tokens. Optionally carry Native `files:*` requests that the API authorizes and prepares. |
| Builders and object storage | Build templates separately from user sandboxes and store template and snapshot artifacts. The Kind deployment uses Versity S3 Gateway (versitygw). |
| Internal API and UI | Receive node state publication and template-release publication, and provide a browser interface. API clients do not require the UI. |
Sandbox-manager waits for node-manager before advertising takeover readiness
and again before acquiring the startup-recovery work gate. Temporary dependency
failures keep the process waiting with bounded individual attempts and periodic
warnings; there is no overall dependency-wait deadline. Its optional local health
socket serves liveness while withholding readiness during these waits. Deployment
probes must distinguish liveness from readiness; external startup deadlines still
apply. Local execution-authority validation remains under recovery admission.
These availability checks do not guarantee that node-manager will stay available
through recovery, and do not extend any caller's request deadline.
Existing namespace containment runs with at most four workers, preserving the
order of checks within each namespace. All checks finish before admission;
any failure prevents activation. Shutdown stops scheduling new namespaces and
waits for in-flight host commands before releasing service ownership. The pool
opens on tracked blocking work so its host commands do not stall local liveness.
An opt-in Linux benchmark exercises production containment against synthetic
network namespaces and nftables rules, comparing sequential and bounded parallel
passes. Run only inside a disposable container with network namespace capabilities:
```sh
SPACE_RUN_CONTAINMENT_BENCHMARK=1 cargo test -p pplx-space-node-mgr --lib \
linux_containment_benchmark -- --ignored --nocapture
```
## Crate layout
The Rust workspace is layered, bottom-up: generic building blocks
(`l10_foundation`), SPACE models and wire protocols without I/O (`l20_domain`),
stores (`l30_storage`), use cases and service entry points with their clients
(`l40_api`), and thin binaries (`l50_app`), with `test_support` and `tools`
beside them. Each is a directory under `crates/` whose `AGENTS.md` says what
belongs there; "Layers" in
[archcheck's README](../crates/tools/archcheck/README.md#layers) gives the
dependency rules CI enforces.
## Security boundaries
Firecracker isolates guest execution. Host-side services still require privileges to manage VMs, storage, and networking.
The API authorizes management requests with a user-session or service bearer token. The data plane authorizes command connections with session-scoped attachment tokens. Management and attachment tokens are not interchangeable. A Native `files:*` request that reaches the data plane carries a management token, which the data plane passes to the API to authorize; it never decides one itself.
A data-plane replica that receives SIGTERM drains rather than dropping its
streams. The listener closes, so new connections, readiness probes included,
are refused, and idle keep-alive connections close; `/readyz` reports 503 only
to a probe already in flight on an open connection. Behind a load balancer, use
a preStop delay so it stops routing to the replica before the listener closes.
A stream request whose headers had already arrived when the drain started gets
503 with `Retry-After` (plus `Connection: close` on HTTP/1); that includes a
streamed ECW `Process.StreamInput`. A native WebSocket that was accepted but
hadn't attached yet is closed with code 1012 (service restart). Streams that
were already attached keep running, and the replica exits once none remain and
in-flight requests finish. With a drain deadline set (`--shutdown-grace-secs`),
streams still open when it passes are cancelled; without one, they run until
they end or the pod's termination grace period kills the process. A second stop
signal ends the drain the way the deadline does, but at once: a `Process.Start`
still opening is cut the way SIGKILL would cut it. At the deadline: native
event streams end, and a client that reconnects with `Last-Event-ID` resumes
a process's output after the last event it received while the guest still
retains it, and a file watch's live stream; native WebSockets close
with 1012, including ones still opening their backend stream, and the client
re-attaches the same session (a process or terminal session doesn't start its
process again); ECW connect streams end with `unavailable` and need a new
client request, a process stream after first delivering the output the backend
has already produced; a `Process.Start` that hasn't reported itself gets up to
2 seconds to, and otherwise ends with `unknown`, since the command may be
running and a retried start would run it twice; a streamed
`Process.StreamInput` gets a 409, since some of its input may already have been
delivered; other stream requests still opening their backend stream, and
proxied sandbox-service requests still preparing their route, get the draining
503; proxied sandbox-service responses and other upgraded connections are cut.
Other responses still in flight, such as a large file download, end when the
process exits, at most about 5 seconds after the deadline, unless they finish
first. Keep the deadline plus those 5 seconds within the pod's termination
grace period minus its preStop delay; the telemetry flush on exit gets whatever
time remains. The service rejects a deadline under 8 seconds, the stream-open
budget.
Continue with [Your first sandbox](first-sandbox.md) for API examples and [recovery steps](first-sandbox.md#troubleshoot-failures).
Internal TCP transport configuration and coordinated rollout requirements are
documented in [Internal RPC TLS](internal-rpc-tls.md).
### Sandbox artifact transfer administration
Node-api exposes no sandbox or snapshot transfer RPCs to remote callers.
Template transfer remains available through its existing API.
Sandbox and snapshot export/import remain available through the sandbox-manager
Unix socket to root or an account with the local operator role. No transfer
command is shipped in the `sbx` CLI. Performing a transfer requires custom code
using the sandbox-manager client library: `ExportSandbox` / `ExportSnapshot` on
the source and `ImportSandbox` / `ImportSnapshot` on the destination. That code
must supply the destination RPC's existing import header and file chunks.
Source lifecycle, encryption restrictions, and destination validation still
apply. The node-api service account cannot invoke these local artifact methods.
A transfer with `use_delta` sends the rootfs as a delta against the cached
template build it was created from. A build directory alone says nothing about
that rootfs: one can exist without a catalog row, as a transient recovery
failure or an interrupted pull leaves it. So sandbox-manager takes the delta
base, for sandbox and snapshot export and import, from a `GetTemplate` lookup
with `cached_only` set. Template-manager answers it from a ready catalog row
whose rootfs has the size its build declares and is a regular file, never
pulls, and marks its answer as a `cached_only` one; sandbox-manager takes no
base from an answer without that mark. Each lookup is bounded at 5 seconds.
Exports, and a sandbox import's check before its body arrives, look the base up
before taking the sandbox's locks. A delta import, sandbox or snapshot, looks
it up again under the sandbox's locks just before it applies the diff, and once
more after: it keeps the result only if the build still answers and its rootfs
is the file the apply read, held open throughout, and is refused with a
retryable `ABORTED` otherwise. A snapshot import makes only those two lookups,
after its body. An export likewise looks the base up again after computing its
diff and keeps the diff only for the file it read. Without a verified base an
export sends the full rootfs. A delta import is refused with
`FAILED_PRECONDITION` when template-manager answers that the node has no such
build, holds one that can't serve as a base, or refuses the lookup, and with a
retryable `UNAVAILABLE` when template-manager can't be reached, doesn't answer
in time or fails transiently. A delta import of a sandbox that already exists,
or whose cleanup is in progress, is refused with `ALREADY_EXISTS` or `ABORTED`
before any lookup.
## TLS interception exemptions
`NetworkPolicyV1.tls_interception_exemptions` uses the same destination selectors
as egress rules. After normal destination admission, the gateway relays matching
SNI hostnames or destination IPs without terminating TLS, even when
`tls_interception` is `on`. A new connection using a hostname exemption requires
a live observed DNS binding from that hostname to the destination IP, including
under default-allow or independent IP allowances. Without that binding, normal
interception applies.
IP exemptions also support TLS without SNI and apply regardless of the hostname
served at that IP. Domain-only egress allowances still require the guest's DNS
attribution and matching SNI.
Explicit destination denies and platform guardrails take precedence. Exemptions
grant no egress permission and bypass HTTP inspection and credential injection.
Other destinations retain the policy's interception mode.
The API composes [operator egress profiles](egress-profiles.md) into the tenant's
policy on authenticated federated creates. A profile independently supplies
allowances and TLS exemptions. DNS resolver allowances are not exempted.
Selected names are persisted separately from the effective policy. Fresh creates,
policy replacement, and explicit template inheritance resolve current definitions.
Forks retain the selected names and saved effective policy; resume and restart do
not refresh profile destinations.
An established exempt flow retains its verified hostname binding for its lifetime,
so DNS expiry or cache eviction does not disconnect it on an unrelated policy
update. Each update still rechecks current egress permissions and exemptions;
removing an exemption closes the flow when the updated policy requires interception.
New connections require fresh DNS evidence.
Before configuring nonempty profile exemptions, upgrade every policy reader and
running gateway to support TLS exemptions. Native sandbox and template policy
writers must observe the same reader compatibility requirement. Upgrade queued
operation executors before APIs begin admitting profile selections.
Before downgrading a reader, remove exemptions from stored policies and operator
profile selections using compatible binaries, or retire the affected resources.
Disabling configuration does not strip an existing sandbox's profile selection.
Removing an exemption restores interception, so pinned clients need an
appropriate operational fallback before rollback.