---
title: Delivery service contracts
description: Fan-out, trusted reads, leases, membership retirement, credential adoption, and daemon lifetime.
kind: Reference
audience: integrators
---

## The three pieces

- **Fan-out writer.** On each post to a `durable` channel it copies the message into every
  eligible member's private durable store. For an `@mention` on a *`live`* channel it also writes
  a copy for each mentioned peer authorized to read that channel, which is how a mention reaches
  an authorized peer who isn't currently joined ([SPEC §4](../SPEC.md#4-delivery-modes)). Fan-out
  handles routing; authorization remains with the broker policy. A post with an empty id is copied
  without a duplicate-suppression key, so two distinct id-less posts are both delivered and a
  redelivery of one may surface twice.
- **Trusted reader.** It pulls each pending entry, re-checks that the member is still allowed to
  read it, and hands the authorized copy to the member's own `dlv_<owner>-<actor>-<uid>` DELIVER
  consumer, which has the same ack semantics. The copy does not ride the member's reply inbox. The
  entry stays pending until the member confirms it was surfaced. A crash between handing off
  and surfacing does not lose the message; the entry redelivers ([SPEC §8](../SPEC.md#8-nats--jetstream-binding)).
  An entry addressed to a retired lifecycle is dropped and removed from the store the first time
  the reader meets it. Retirement leaves a tombstone on that lifecycle's read-ACL row, and a retired
  lifecycle never comes back, so a later reader with a fresh cursor does not pay for it again. The
  reader acks such an entry only after the delete succeeds. A failed delete leaves the entry pending,
  so it is retried and given up after ten redeliveries, and an entry the stream no longer holds counts
  as removed. A daemon that stops serving while the delete is in flight neither acks nor gives up the
  entry, so the daemon that serves next retries it. When the entry came from a durable channel, the
  reader also removes the retired lifecycle's member row for that channel, when one exists: a daemon
  that died between creating a row and its post-write ACL read leaves one, and until it goes every
  later message on the channel is copied into the retired inbox and dropped again. The removal is
  best effort; a failure is a warning, and the next entry retries it. An entry whose owner has no ACL row at all is
  retried, then given up after ten redeliveries, and kept, because a missing row does not prove the
  owner is gone.
- **Membership registry.** A privileged-written record of who is a durable member of each
  channel, carrying per-member join and leave cursors so a post concurrent with a join or leave
  orders deterministically ([SPEC §7](../SPEC.md#7-channels)). It is broker-known truth, not
  self-reported: an agent cannot assert its own membership.

## Anycast request records

On an authenticated mesh the daemon also records which holder took each anycast request and which
requests their senders cancelled ([SPEC §8.1](../SPEC.md#81-anycast-request-status)). Just before a
role holder's connector shows a role request, it asks the daemon on its own take subject to record
the take. The daemon reads the request at that seq on the task stream and writes one row per
request to the `cotal_anycast_<space>` bucket. A take names the task stream's creation time as the
holder saw it, and the daemon refuses one from a stream since recreated, because the new stream
reuses seqs. The holder then reads the creation time again, and when it moved, it naks its copy, so
the broker delivers the request again at once on the new stream. A take and a cancel of one
request race on that row's create and revision check, so only one of them wins. When a take wins,
the daemon sends the request's sender a
DM from its own principal naming the holder. It sends a second one when the request's ack wait ran
out and a later delivery moved the take to another holder.

A sender reads or cancels its own request by seq with the `anycastStatus` and `anycastCancel`
commands on the `ctl.delivery` rail. A cancel succeeds only while no holder has taken the request:
the daemon writes a cancelled row and deletes the request from the task stream, and a holder that
then asks to take its buffered copy is refused and forgets it. The daemon removes a row a day after
the request left the queue. It checks every row when it starts and every ten minutes, and resends
any notice a crash left unsent.

Before you delete or recreate the task stream, stop every delivery daemon that serves the space and
wait until each has exited and its broker connection has closed, then start the daemon against the
new stream. A cancel checks the stream's creation time and then deletes by seq, and JetStream has no
delete that checks the stream it lands on. A delete still in flight when the stream is recreated
removes the request the new stream stores at that seq, and the cancel still reads deleted.

For this the `delivery` credential can read every request body on the task stream and delete any
request there. Only the trust placed in the daemon bounds those two grants
([SPEC Appendix B](../SPEC.md#appendix-b-profile-acls)).

## Why a *trusted* reader

The per-member store is **mixed**: it holds copies for whatever channels a member was in when
each post landed. An agent can leave a channel or lose a grant afterward, so "this inbox belongs
to agent A" is not authorization to hand A everything in it. Agents therefore hold **no
content-bearing read** on the store; the daemon reads it on their behalf and re-authorizes every
`(instance, channel, message)` entry against the member's **current read ACL** and, for
`durable`-channel entries, its **membership interval** (the post's sequence sits between the
member's join and leave cursors) before releasing content ([SPEC §7](../SPEC.md#7-channels),
[§8](../SPEC.md#8-nats--jetstream-binding), [§9](../SPEC.md#9-nats--jetstream-security-and-authorization)).

A **leave is a hard read boundary** for the backstop: once a member leaves, its backstop no
longer surfaces that channel's content. (Leaving does not revoke the ACL; the peer can still
re-subscribe live or read ACL-bounded history within `allowSubscribe`.) See
[identity-and-auth.md](identity-and-auth.mdx) for how the ACLs are minted and
[presence-and-delivery.md](presence-and-delivery.mdx) for the delivery-class model.

## Where it runs

`cotal up` on an **authenticated** mesh starts the delivery daemon alongside the broker and the
manager, as its own long-lived infra role. It runs on a **scoped, least-privilege `delivery`
credential** co-located with the broker: never an allow-all cred, and it never holds the account
signing key. One daemon serves a space (a single-flight lease guards against a second binding the
same durables).

One process can also serve every space on a host: `cotal deliver --spaces <manifest> --server <url>`
holds one lane per space the manifest names ([server daemons](cli.mdx#server-daemons)). A lane is
that space's daemon. It reads the space's own credential from the root, dials on its own
connections, which authenticate into that space's account, and holds its own lease, broker
watchdog, timer writer and membership feed, so a second daemon for the space is refused by the lease
as before. A lane that loses the broker, its credential or its lease closes alone and the host
reopens it; the other lanes keep serving.

### Standby host

Two provisioned multi-space hosts can stand by for the same space, each with its own root and
scoped credentials for that space account. The host that takes the lease serves. The other host reports the held lease, opens no lane and retries every 5
seconds. If the holder dies, its lease row expires with the bucket TTL, and the standby takes the
row on a retry and serves. A host that comes back finds the new holder's row, does not serve, and
does not delete or rewrite that row. There is no election, placement or failover command: the lease
is the only arbiter, and nothing moves a space back to the first host.

A host that is not provisioned for the space, holds a credential for another account, holds a
malformed credential, or lacks the `$SYS` observer credential refuses on its own checks and never
touches the lease. The `smoke:delivery-standby` suite runs two complete hosts on one machine and
one broker through the kill, return and graceful-stop cases and prints observed takeover times.
Those times are not an SLA. One machine with two host processes is not a test of two physical
hosts or of a network partition. The fixture also replaces each host's delivery credential through
its own store, crosses the initial expiry while the holder still serves, and refuses an invalid
manifest without disturbing that holder. It does not test membership-credential renewal or
system-account rotation.

### Standby provisioning

This is an operator-provisioned or embedded topology; the CLI has no cross-host standby enrollment
command. Prepare two distinct roots with `.cotal`, one version-1 manifest per host, and the same
broker URL and space account. Start each with `cotal deliver --spaces <manifest> --root <root>
--server <url>` and the broker's required TLS setting. The manifests name spaces, not identities or
credentials. A malformed manifest refuses the whole start; an invalid lane credential refuses that
lane and is retried. Never copy another account's credential to make a lane open.

A trusted provisioner uses the public mint helpers and writes each host's assigned SecretStore keys:

| Input | Mint helper | Store key helper |
| --- | --- | --- |
| Delivery | `mintCreds` with profile `delivery` | `deliveryCredsKey` |
| Membership writer | `mintCreds` with profile `membership-rw` | `membershipRwCredsKey` |
| Account observer | `mintMembershipObserverCreds` | `membershipObserverCredsKey` |
| Connection eviction | `mintConnectionEvictorCreds` | `connectionEvictorCredsKey` |

The mint helpers come from `@cotal-ai/core`; the store key helpers and `workspaceSecretStore` come
from `@cotal-ai/workspace`. A CLI host's keys use `{ injected: false, root }` with that host's root.
Keep each host's delivery nkey stable across renewal and restart, and distinct from the other host's.
The provisioning owner retains identity/signing custody and atomically replaces bounded credentials
in each host's store before expiry. Delivery rereads its own key and proves renewed credentials
against the broker; the membership feed uses its own renewed writer key. Neither host needs the
account signer. A copied static credential file is not a renewal owner. See
[delivery renewal](embedding-daemons.mdx#delivery) for the stock Manager/store continuity contract.

The observer must be pinned to the same data account and broker the lane serves. Its scan results,
not a caller-supplied connection ID, select eviction targets. The separate evictor is high-power:
its native KICK permission is broker-wide, not account-confined. Keep it on trusted delivery hosts,
never in a participant or tenant-controlled sandbox. On a broker that hosts several spaces, no space
daemon holds it. The multi-space delivery host (`cotal deliver --spaces`) is the one process that
kicks: when a space on its root retained the system-account signing seed, the host mints a kick-only
user per kick from that seed, and a lane asks the host to kick each live match of its own sweep. The
host kicks a connection only after its own complete census of the lane's account lists it, and the
lane opens no kick credential. Observer and evictor credentials have their
own provision/rotation lifecycle; do not infer their renewal from a delivery-credential refresh.
Missing or inconsistent scan authority refuses admission or verified eviction. An operator must
resolve that provisioning boundary explicitly, without clearing leases or fabricating scan results.

The manager needs this daemon while it starts. Its SecretStore challenge, its boot repair of a frozen
registration gate, and the verified eviction a restart performs all go over the daemon's
`ctl.delivery-admin` rail. Agents' durable join, leave and membership-list requests go over its
`ctl.delivery` rail. Both are `ctl` subjects that [SPEC §13.11](../SPEC.md#1311-the-hard-cut)
retires; the reference implementation still serves them (see
[the control surface](control-surface.mdx)). A manager that starts while the daemon is still binding, or while the
daemon re-checks who owns its lease, waits up to 60 seconds for the rail to answer. Only a request
that times out or finds no responder is retried. The first retry runs at once. Later retries are at
most one second apart, shortened as the wait runs out, so a daemon that binds in its last seconds is
still asked. The wait ends on time even while a retry is still connecting, and nothing is sent after
it. A daemon that answers fails the start at once if it refuses or its reply cannot be read. One
that stays silent for the whole wait fails it, and the manager log names the rail.

A manager retiring a static agent uses the same retry cadence for an unanswered verified-eviction
request, within a 15-second total wait. A credential reload briefly removes the daemon's responder
while it reconnects; that gap must not abandon an otherwise healthy retirement. A refusal or an
unusable eviction reply still fails at once. The lifecycle stays held until eviction is verified
and cleanup completes.

The daemon answers `ctl.delivery-admin` requests one at a time, in the order they arrive, with one
exception: the `lifecycleMemberships` inventory a retirement reads before it deletes the retired
agent's membership rows. That request is answered without waiting for the others, so a manager stop
that retires many seats does not queue each inventory behind the other seats' evictions. Inventories
that arrive together share one read of the members bucket, and each one's read begins after it
arrived. Before the manager, or the auth service retiring a chat session or a managed agent, asks for
a lifecycle's inventory, it narrows that lifecycle's read-ACL row to no reads. A durable join reads that row again just before it writes its membership row, and
once more after. When the channel is no longer allowed, the join writes nothing, or removes the row it
just wrote, and is refused. A leave or an activation only updates a row that still exists. A row of the
retiring lifecycle is then either in the inventory or gone, whichever daemon process wrote it. The
daemon also fences on its own, for a manager that does not narrow the row: when the inventory request
arrives with the lifecycle's ACL row still allowing reads, it refuses every durable join and leave of
that lifecycle from that moment, on that daemon process. A request whose row is already narrowed, or
retired, leaves no such mark, since the row refuses those writes by itself. Either way the read starts
only after the writes already running have finished.

A reply on the `ctl.delivery` or `ctl.delivery-admin` rail that is larger than the broker's
`max_payload` cannot be published. The daemon answers that request with a refusal giving the reply's
size in bytes, so the caller fails at once and does not wait out its timeout.

An endpoint's pending control and delivery requests reject if their connection closes, including
when the endpoint stops or replaces it. The request is not replayed on the new connection: a lost
reply does not prove that the operation did not run. A liveness probe interrupted this way returns
`unknown`, not `unbound`. Normal replies, timeouts and broker no-responder answers keep their usual
classification.

A restart verify-evicts every holder in the manager's credential family, and the family keeps a
ledger row for every credential an earlier incarnation was issued. The manager keeps its serve,
goal-writer and session-ledger identities across restarts, so restarts add no holders; each attach
session adds one serving holder. The manager sends those holders
as one `evictPrincipals` request per 256, and the daemon answers each request with one shared sweep
of the broker. The manager records the holders each request verified before it sends the next, so a
restart cut short by its executor window resumes after the last recorded request. A daemon that does
not serve that verb refuses it, and the restart leaves the gate frozen.

An agent binds its per-member delivery durable even when the plane reached by its connection has no
ready delivery lease, so a daemon that starts later can deliver through it. A missing or not-ready
lease emits a warning that names the durable, space, and condition. It tells the agent to reconnect
against another plane if that plane serves the space, which re-binds the durable there.

Before it constructs its endpoint or claims that lease, the daemon reads the account-scoped `$SYS`
observer from the same source it will use for scans, whether that source is the workstation store or
an injected hosted store. It refuses if the observer belongs to another account, is missing, or is
part of a present but torn observer/evictor rotation. An absent evictor keeps the documented
pre-eviction, deny-new-only posture; eviction itself remains unavailable until it is provisioned.

It inherits the mesh's transport on **every** launch, including the relaunch a bare `cotal up`
performs when the daemon is missing. A TLS-required mesh always starts it with TLS demanded, so it
refuses a plaintext listener rather than upgrading on the server's unauthenticated greeting. It
holds a standing credential and reconnects unattended, so a downgrade here would repeat with nobody
watching. See [transport.md](transport.mdx).

After a reachability check at start-up, the daemon watches the broker only through its resident
connection's events. It opens no other connection to check the broker. A disconnect starts a clock,
and a reconnect stops it. When the time since the disconnect, less any time the daemon's own
event loop was stalled, reaches the window (`COTAL_DELIVERY_BROKER_GONE_MS`, 15 seconds by
default), the daemon logs
`✗ delivery: broker connection unavailable (broker unreachable), exiting (coupled to the broker)`
and exits. Wall time since the disconnect also has a backstop
(`COTAL_DELIVERY_BROKER_GONE_BACKSTOP_MS`, four times the window by default and never less than it).
Reaching it logs
`✗ delivery: broker connection unavailable past backstop (broker unreachable), exiting (coupled to the broker)`
and exits, however long the daemon was stalled. A broker that stops answering without closing the
socket counts as disconnected only once the client's pings go unanswered, so the clock starts a few
seconds after the broker stops.

An authentication-expiry error is not a broker disconnect. The daemon logs
`! delivery: credential expired, awaiting proved renewal` at once and waits for a renewal it has
proved against the broker; a reconnect alone does not clear it. If no such renewal arrives by the
backstop, it logs `✗ delivery: credential expired without renewal` and exits.

The daemon refuses a `reloadCreds` adoption until it has finished starting, which is after its lease
watch is bound. Its lease turns ready earlier than that, so a renewal owner can ask before start-up
is done. The refusal says the daemon has not finished starting and adopts nothing. The next renewal
pass or the daemon's own 75% re-read adopts the re-signed credentials.

After an adoption the daemon reconnects with the new credentials 250ms after its reply. A
delivery-admin request it reads in the meantime is answered once that reconnect completes, or
after 5s. A reply sent while the connection is down would be lost, and its caller, such as a
manager evicting its predecessor right after its boot renewal, would wait out its whole 15s request
timeout.

`cotal up` reports the daemon **only when it is actually serving**. A daemon it started can exit
without taking the single-flight lease because another daemon holds it, or because a crashed
holder's lease has not expired yet. `up` then says so and exits non-zero instead of printing a
healthy control plane over a daemon that is not there. A lease write the credential is not allowed to make is reported
as a denial naming the refused subject and operation, never as another daemon holding the lease.
A lease write the broker refuses for another reason, such as the insufficient-resources answer of a
store over its `--max-file-store` cap, is quoted as the broker gave it. A running daemon whose lease
key expired creates it again. Only a create refused because a live lease exists makes it say that
another daemon has taken the shard. Any other refusal is quoted, with the line saying it could not
re-acquire the lease, and the daemon exits all the same, because it serves only under its lease.
The daemon writes its own reason to `.cotal/delivery.<key>.log`, the log
for the space it serves ([Config](config.mdx#project-files)). That path is project-local. Detached
`up` redirects the daemon's stdout and stderr onto the file, so wrapping the launcher in a
systemd unit does not put those lines in that unit's journal. A daemon stopped by SIGTERM or
SIGINT, which is what `cotal down`, a service stop and Ctrl-C send, writes `received <signal>,
exiting` to that log before it releases its lease. A signal that arrives while the daemon is
still claiming the lease waits for the broker's answer, and a claim that won is then released the
same way. A signal that arrives while the daemon's connection redials, as it does when the daemon
moves onto a renewed credential, waits up to five seconds from the drop for the redial before the
release, because a request sent while the connection is down is never answered. A SIGKILL,
including one from the kernel OOM killer, ends the daemon with no line.

A foreground `cotal up` restarts a daemon it started when that daemon dies while the broker that
`up` started is still running. It logs
`delivery daemon exited (<cause>) while nats-server is running - restarting it`, then
`delivery daemon running again` once the replacement's responder is bound. A replacement whose
responder does not bind gets the same not-bound warning as at startup instead. The daemon ends
itself when it cannot reach the broker, and a starved host can make a running broker look
unreachable. Without the restart, every retirement that needs the daemon would fail until someone
ran `cotal up` again. A failed restart is logged and retried after the 30-second lease TTL. A daemon
that exits cleanly or on SIGTERM or SIGINT stays stopped. So does one that bare `cotal down`
stops, also when the daemon is too starved to exit on SIGTERM and `down` kills it, when it is a
replacement that is still starting, and when the stop lands between two restart attempts. A legacy
stack has no command that stops only the daemon. Detached
`up` exits after launching and restarts nothing; a bare `cotal up` relaunches a missing daemon there.

Ctrl-C on a foreground `up`, and a broker that exits under it, stop the daemon with the same stop
bare `cotal down` uses for it. It holds the reservation `down` takes, so the delivery stop of a
concurrent `cotal down` is refused while it runs. It sends SIGKILL to a daemon that has not exited 15
seconds after SIGTERM, and removes the daemon's credential only once its death is confirmed. A
record whose pid now belongs to another program is cleared without a signal. If the broker exits
while that stop is running, `up` waits for the stop to finish before it exits.

The daemon **records itself** in `.cotal/delivery.<key>.pid`, whichever way it was started, and
removes that record when it exits cleanly. A lane of `cotal deliver --spaces` is the exception: it
writes no record, because the record is what `cotal down` signals for one space and the host serves
several. The launcher is not the only route to a running daemon: a
container entrypoint, a systemd unit, or `cotal deliver --space <space>` typed by hand all reach one
too, and a record written only by the launcher goes stale the moment any of those restarts it.
`cotal deliver` is a deprecated entry kept for existing units and scripts; new setups use the
`delivery` service preset ([deprecated daemon entries](UPGRADING.md#deprecated-daemon-entries)). Typed
by hand on the workstation, the daemon dials the broker recorded for the space in the mesh registry
(a mismatching `--server` is refused before any dial); with no record for the space it uses the
local mesh default, and a daemon with an injected store never consults the registry at all. The
write happens once the daemon holds the single-flight lease, because that is the point at which it is
the space's daemon: one that loses the lease refuses to bind and exits, and must not overwrite the
live holder's record on its way out.

The daemon serves one workspace root, chosen at start: the one `cotal deliver --root <dir>` names,
or the nearest `.cotal/` above its working directory. A workstation daemon with neither refuses at
start and names the directory it searched from, before it reads a credential or dials a broker,
because a directory nobody set up holds none of its credentials. A daemon with an injected store
takes its credentials from that store and needs no `.cotal/`.

Readers verify the record before believing it. A recorded pid is trusted only when the process behind
it is alive **and** its command line names a delivery daemon, so a record that outlived its process
and had its number reused is reported as stale rather than as a healthy daemon. `cotal down` never
signals such a process. Where a command line cannot be read, the record is trusted as before: the
check only ever downgrades on proof.

The daemon also hosts the space's **checkpoint timer writer** ([SPEC §13.9](../SPEC.md#139-authority-boundary)):
the standing pump that turns workflow `.schedule` requests into armed broker schedules, on its own
connection under the same delivery credential. That connection presents the credential the daemon
holds now and reconnects onto each renewed one the daemon adopts, so the broker never closes it for
an expired credential and it never dials with one. Without a running writer no workflow pause on the
space ever expires. The writer restarts itself with backoff and logs while it is down; a fault
there never takes delivery down. Stopping the daemon ends the writer's pending fetch at once and
wakes a retry backoff it is sleeping through, so a close never waits for a retry it will not make.

**Open dev mode has no delivery daemon.** Open mode is deliberately **live-only**: there is no
trusted reader, so there is no durable backstop. Run an auth mesh if you need durable channels.

## Without it

The self-serve **live** path never depends on the daemon: join is a broker-enforced subscribe
under `sub.allow`, so a `durable` channel still delivers live with no daemon present ([SPEC §7](../SPEC.md#7-channels)).
Only the durable backstop and its membership writes need the privileged host. If a peer joins a
`durable` channel while the backstop can't be established, it is **joined live with the durable
backstop unestablished**: the live subscription is active, and the shortfall is surfaced as an
exceptional delivery state, never reported as `joined durable` and never silently dropped ([SPEC §7](../SPEC.md#7-channels)).

Role holders on an authenticated mesh do need the daemon. A holder's connector shows a role request
only after the daemon records its take, so while no daemon runs, role requests wait on the queue
and each holder reports that no delivery daemon is answering. A daemon that predates takes, or a
seat credential minted before them, holds requests the same way, and the holder says to restart
the daemon or respawn the seat.