# Federation How independent StrafeChat instances talk to each other, what was true about the codebase before this existed, and what the design does and doesn't cover. ## Starting point (codebase analysis) StrafeChat is three services: **equinox** (Go/Fiber REST API and the **stargate** WebSocket gateway, sharing ScyllaDB + Redis), **nebula** (a small object store serving avatars, emoji and attachments over HTTP), and the **web** client (SolidJS). Before this work the system was strictly single-instance: - **Identity** was a local snowflake id plus a username (then paired with a #0001 discriminator; usernames have since become unique on their own), with lookup tables keyed by email and username. Nothing carried a "which server" notion. - **Rooms** (PMs, group PMs, space channels) stored participants as local user ids; messages lived in one `messages` partition per room with local snowflake ids. - **E2EE** (Olm/Megolm through matrix-sdk-crypto) already used Matrix-shaped identifiers, but with a synthetic server name (`@:strafe.internal`, `!:strafe.internal`). The key-exchange endpoints (`/devices/keys/query`, `/keys/claim`, `/send_to_device`) parsed and discarded the server part. - **Deployment** had a development compose file (no web image, no TLS, secrets in a mounted `.env`), and both Go entrypoints aborted if `.env` was missing, which made configuring containers purely from the environment impossible. The E2EE layer turned out to be the load-bearing decision: because it already spoke in `@user:server` / `!room:server`, federation could be built by making those names *real* rather than by re-plumbing the crypto. ## Design ### Instance identity and discovery An instance is identified by its **domain** (`FEDERATION_DOMAIN`, e.g. `chat.example.com`) and an **Ed25519 signing key** (auto-generated into `FEDERATION_KEY_FILE`, or supplied via `FEDERATION_SIGNING_KEY`). It publishes an instance document at `https:///.well-known/strafe` (also at `/federation/v1/instance`): ```json { "domain": "chat.example.com", "version": 1, "api_url": "https://chat.example.com/api", "gateway_url": "wss://chat.example.com/gateway/events", "federation_url": "https://chat.example.com/api/federation/v1", "key_id": "ed25519:1", "public_key": "", "software": { "name": "strafe-equinox", "version": "..." } } ``` Peers cache this for an hour and re-fetch on a signature failure (key rotation). ### Naming | Thing | Local form | Federated form | | --- | --- | --- | | User, as typed by people | `alice` | `alice@chat.example.com` | | User, for the crypto engine and S2S payloads (FID) | `@:` | `@:` | | Room, for the crypto engine | `!:` | `!:` | The *origin* id of a user is what their home instance calls them; the origin of a room is the instance that created it. Every participating instance must agree on these strings so Olm sessions and Megolm room keys (which embed the room id) line up. ### Shadow users A remote user gets a local **shadow row** in `users` with `home_domain` and `remote_id` set and no email/password. It is never in the email or username lookup tables and can't log in, but every existing code path that resolves a user id (participants, mentions, message senders, key-exchange reachability checks) works unchanged. `users_by_remote` maps `(home_domain, remote_id)` back to the shadow. Profiles (username, display name, avatar, banner, bio, about me, pronouns, bot flag) are refreshed whenever the home instance relays a change (`POST /federation/v1/users/update`). Profile badges stay instance-local: they are something an instance's administrators grant, not something a peer asserts. ### Mirrored rooms Each participating instance keeps its **own** room row, participant rows and message rows under local ids, so listing, read state, mention counts and the gateway all keep working per instance. Two mapping tables tie the copies together: - `federated_rooms` / `federated_rooms_by_origin`: local room id ↔ `(origin_domain, origin_room_id)`. - `federated_messages` / `federated_messages_by_origin`: local message id ↔ `(origin_domain, origin_message_id)` for messages that crossed an instance boundary (edits, deletes and replies reference messages by origin id). PMs and group PMs are mirrored this way between the instances of their participants. Space channels are mirrored too, but with one difference: a channel keeps the **origin's message ids** on every instance (every message in it is stored by the origin first), so replies, reactions and ordering agree everywhere without a lookup. See "Spaces across instances". ### Server-to-server protocol All S2S calls are JSON over HTTPS under `` and are signed: ``` X-Strafe-Instance: sender domain X-Strafe-Timestamp: unix seconds (rejected if more than 5 minutes off) X-Strafe-Nonce: random per request (rejected if seen before, 10 minute window) X-Strafe-Signature: ed25519:1= ``` The signature covers `strafe-fed-v1 \n METHOD \n \n timestamp \n nonce \n sha256(body)`. Signing the federation-relative path (not the full URL path) means a reverse proxy may mount the API under any prefix. | Endpoint | Purpose | | --- | --- | | `GET /users/lookup?username` | resolve a handle to a profile | | `GET /users/:id` | profile by origin id | | `POST /users/update` | a user's profile changed | | `POST /users/presence` | how a user now appears to others - their status and what they are playing - for the receiver's users who are their friends or share a space with them | | `POST /relationships` | a friend request, acceptance or teardown from the sender's user to one of the receiver's | | `POST /rooms` | the origin announces a new room and its participants. The receiver admits each of its own users through the same gate a local account passes to start a conversation (never across a block; under `PM_POLICY=shared` only friends or people sharing a space) and answers `403` when none of them may be reached | | `PUT /rooms/participants` | full member set after add/remove; accepted from the room's origin only (only a group's creator may change membership, and the creator lives on the origin). Newly added users of the receiver pass the same gate as above | | `PATCH /rooms` | name / E2EE setting changed; accepted from the origin only. `e2ee_enabled` may change on a group only, never on a 1:1 PM, which is encrypted for life | | `POST /rooms/typing` | typing indicator | | `POST /rooms/messages`, `PATCH /rooms/messages`, `POST /rooms/messages/delete` | message lifecycle (including system messages); in a space's channel accepted only from its origin, `403` from anyone else | | `POST /rooms/reactions`, `POST /rooms/reactions/delete` | a reaction added to / withdrawn from a message | | `POST /rooms/pins` | a user pinned or unpinned a message in a shared room | | `POST /rooms/voice/join`, `/leave`, `/self`, `/ring`, `/decline` | asked of the room's origin: a token for the call it hosts, and the caller's later actions. A leave may name its `session_id`; if that is not the current state's session the origin answers `204` and does nothing, so a leave still in flight cannot end a newer join | | `POST /rooms/voice/states` | asked of a room's origin by an instance holding a copy of its voice states: who is really in these rooms, at most 64 (see "Calls across instances") | | `POST /rooms/voice/state`, `POST /rooms/voice/call` | pushed by the origin: a voice state changed or left (`moved_to`, a room ref in the origin's ids, on a left caused by a moderator's move), the call started / changed / ended | | `GET /spaces/invites/:code`, `POST /spaces/join`, `POST /spaces/leave`, `POST /spaces/invites` | asked of a space's origin: preview an invite, redeem it for one of the asking instance's users (the answer is the whole space), leave, mint an invite | | `POST /spaces/messages`, `PATCH /spaces/messages`, `POST /spaces/messages/delete`, `POST /spaces/reactions[/delete]`, `POST /spaces/pins[/list]`, `POST /spaces/messages/list`, `POST /spaces/messages/get` | asked of a space's origin by a mirror on behalf of a member: write into, react and pin in, and read a channel (history and its pin list) | | `POST /spaces/manage` | asked of a space's origin by a mirror on behalf of a member who may manage it: one of `space.patch`, `space.image`, `role.create/update/delete`, `role.positions`, `member.roles/role/kick/ban/unban`, `bans.list`, `invites.list`, `invite.delete`, `audit.list`, `room.create/update/delete`, `rooms.reorder`, `room.move`, `override.put/delete`, `emoji.create/rename/delete`, `space.transfer`, `space.delete`, `bot.install`, `member.add`, `member.profile`, `member.birthday`, `voice.moderate`, `voice.disconnect`; answers with the result and the relays captured for the asker | | `POST /spaces/sync` | asked of a space's origin by a mirror: a fresh snapshot to reconcile against | | `POST /spaces/members/list` | asked of a space's origin by a mirror: the next page of members (the join and sync replies carry the first page and a cursor) | | `POST /spaces/update`, `/spaces/members`, `/spaces/roles`, `/spaces/rooms`, `/spaces/emoji`, `/spaces/peers`, `/spaces/delete` | pushed by the origin to every instance mirroring the space: settings, membership and roles, roles, channels and overrides (and their order), custom emoji, which instances are in the space, deletion; threads travel as rooms of type 6 with their `thread` state, but thread membership is not mirrored, so on a mirror a private thread reaches only the owner, Administrators and holders of Manage Threads, and threads are created and edited on the origin | | `POST /keys/query`, `POST /keys/claim`, `POST /to_device` | E2EE key exchange for the receiver's users. The body names the requesting user (`requester`, a FID on the signing instance, for the key endpoints; `sender` for to-device); the receiver serves each of its users only when the two may exchange keys - a shared room, or friends who do not block each other, the gate its own clients go through - and refuses a request without a requester. Per-peer rate limits apply (600 queries and 300 claims a minute) | | `GET /discover` | the spaces this instance lists publicly and shares, for the asker's Discover page | | `POST /spaces/join_listed` | asked of a space's origin: add one of the asking instance's users to a space it lists publicly, no invite (the answer is the whole space) | Authorization rules on the receiving side: an instance may only announce rooms it created, relay messages/edits/deletes/typing for its **own** users, touch rooms one of its users participates in, and act on relationships between its **own** user and one of ours. For a space, only the origin may push changes to a mirror, and only an instance that mirrors the space (one of its users is a member) may ask the origin for anything in it; the acting user must belong to the asking instance. Profiles in payloads are trusted only for the sender's own users; a third instance's user is fetched from their home before a shadow is created (the origin of a space relays messages from members of any instance, so a mirror may fetch a sender's profile from a third instance the first time). Relays are fire-and-forget from the user's point of view: the local write and gateway event happen first, then the relay is queued, and a slow or absent peer never fails or delays the user's own request. Every relay but typing, presence and call state is written to `federation_outbox` before it is sent and deleted once the peer has taken it. One worker per peer sends that peer's entries in order (so a message sent right after its room was created cannot overtake the room announce) and, when the peer cannot be reached, retries with backoff (1 s doubling to 30 s) until it can - so a peer that was down gets everything it missed, in order, when it comes back, and a mirror converges without a resync. A reply the peer will only repeat (a 4xx other than 408, 425 and 429) drops the entry; an entry nobody could deliver in seven days expires. While a peer is unreachable its row in `GET /federation/peers` carries `last_error`, `last_failure` and `pending` (entries waiting), cleared by the next relay it takes. The API process drains the outbox; the gateway, which relays presence only, writes entries and wakes the API over Redis. Typing, presence and call state describe the moment and are superseded within seconds, so they are sent once, from memory, and dropped if the peer is away. A voice state lost this way is put right by the pull described under "Calls across instances". ### Friends across instances A friend request to `alice@chat.example.com` resolves the handle through `GET /users/lookup` on her instance (creating or refreshing her shadow row), stores the request locally against the shadow id, and relays it with `POST /relationships` (`action: request`). Her instance stores it the other way round - from *your* shadow row to her - and shows it to her like any local request. Accepting relays `accept`; declining, withdrawing, unfriending and blocking all relay a single `remove`, and the receiving side tears down whatever still stands (its own state says which it was). Two requests that cross become a friendship on both sides. Blocks are never announced: a request from someone the recipient has blocked is accepted with `204` and dropped, so the sender's instance learns nothing. Because shadows are ordinary `users` rows, the friends list, pending requests and nicknames need no extra tables; `home_domain` on the relationship's user object is what the client renders as `@domain`. Profile changes are relayed to the home instance of every remote friend, not only to instances sharing a room. ### Presence across instances Whenever a user's presence changes - the gateway marking them online or offline, or a status/custom-status change through `PATCH /users/@me` - the instance tells the home instance of each remote *friend*, and every instance that shares a *space* with the user (see "Spaces across instances"), how the user now appears to others (`POST /users/presence`; invisible already reads as offline). The receiver writes it to the shadow row and publishes `PRESENCE_UPDATE` to the shadow's local friends and shared spaces, so the status dot moves live and the next READY carries it. A freshly accepted friendship exchanges presence in both directions so the new friend starts with a real status. Friends and fellow space members only, which is who the local gateway tells as well; people who merely share a PM are not told, matching local behaviour. The gateway process therefore needs the same federation configuration and signing key as the API (the compose file gives both services the `federation-data` volume); it loads the key but never creates it, so the API always owns the identity. If a peer goes down mid-session its users stay at their last known status. What someone is playing travels with their presence. A presence carries `activities`: at most one entry, `{type: "playing", name, started_at}` (`started_at` in unix milliseconds), present only while the user does not read offline to others - so an invisible user's game never leaves their instance, and changing game while invisible sends nothing at all. The user's home instance relays a change of activity like a change of status. The receiver takes an activity from `POST /users/presence` only - the user's own instance speaking - and checks it the way its own gateway checks a client's report: more than one entry is refused with `400`; control, line-breaking and bidi characters are stripped, the name must keep a visible character and at most 128 code points, and an entry of a type it does not know, or with a name that does not survive that, is dropped while the status still applies; `started_at` is clamped into the last seven days. A relay without `activities` means "playing nothing", which is also what an instance from before activities sends. The receiver keeps the activity in Redis only, never on the shadow row: one key per remote user (`presence:activity:fed:`), expiring after 12 hours, deleted when a relay clears it or says the user is offline. A game can outlast that, so while the user keeps playing their home relays their presence again every four hours (its gateway renews its own copy hourly), and each relay renews the receiver's 12 hours. If an entry does lapse on a receiver - the home went quiet for half a day - the receiver's clients may still show the game, so a later relay of "playing nothing" from a user who is online is published even though it changes nothing in the receiver's store. Snapshots, member pages, member relays and joins carry status only and never set or clear an activity - they describe members of any instance, and member relays are written to `federation_outbox`, a ScyllaDB table. Redis's periodic snapshot (the compose file runs it with `--save 60 1`) can copy these short-lived entries, like the gateway's own per-connection ones, into the snapshot file on the instance's disk, where each stays until a later snapshot is taken without it; nothing else writes them down. ### Reactions across instances Reactions on messages in federated PMs and groups are relayed (`POST /rooms/reactions`, `POST /rooms/reactions/delete`) with the message referenced by origin id, exactly like edits and deletes, and applied with the same per-message cap and gateway events as a local reaction. A `custom:` reaction is the origin's custom emoji id; a client that does not know that emoji shows an empty pill, as it already does for a local reaction with an emoji from a space the viewer is not in. ### Calls across instances LiveKit servers do not federate, so a call in a federated PM or group runs on **one** LiveKit: the one belonging to the room's *origin* instance (every peer knows it from the room mapping). The origin mints every token, owns the voice states and the ringing call object, and relays each change to the other instances in the room, which mirror them so their READY payload, `GET /rooms/:id/voice/states` and gateway events work unchanged. A user on another instance joins through their own API as usual: it asks the origin (`POST /rooms/voice/join`, synchronous) and hands back the origin's public LiveKit URL and token; leave, mute/camera flags, ring and decline are relayed to the origin the same way. Participants in such rooms are named `.` towards LiveKit, so a client on any instance can tell who a participant is; the E2EE media keys reach them over the federated to-device channel like all other Olm traffic, and a key announcement names the call's room by its federated identity (`!origin:domain`), never by a local id - a PM or a mirrored channel has a different local id on every instance, and a key filed under the sender's id would never match the receiver's call. Consequences: the origin's `LIVEKIT_URL` must be reachable from the other instance's users, both instances need voice configured (an instance without LiveKit has no voice routes at all), and if the origin has no LiveKit the call cannot happen in that room. Voice rooms of a federated space work the same way, on the space origin's LiveKit. **Relays.** The origin relays every voice state change (`POST /rooms/voice/state`), in a space's voice rooms as well as in PMs and groups. Voice relays and `/rooms/voice/call` also reach the instance whose request caused the change: the reply to that request says nothing about voice, so the usual skip of the asker and the capture in `/spaces/manage` do not apply to them. In a space, both go only to peers with a member who can view the room (see "Spaces across instances"). A state's relay also always reaches that user's home instance when it is a peer, because that instance has to hang them up. `moved_to` goes only to peers that can view the destination and to the moved user's home; naming a private room to anyone else would say who is in it, so they get the same left without it. A mirror maps `moved_to` to its own room id and, if it has none, omits it and still applies the left. **The pull.** A relay is one attempt, a lost left would leave a ghost, and clients key their media to whoever the states list. So on every voice sweep (30 s) an instance asks each origin about the rooms it holds states or a call for: ```json POST /rooms/voice/states {"rooms": [{"origin_domain": "...", "origin_room_id": "..."}]} {"rooms": [{"room": {...}, "gone": false, "states": [...], "call": {...}}]} ``` A request names at most 64 rooms (`voice.MaxSyncRooms`; more answers `400`), and a mirror with more sends several; a Redis lock keeps API replicas from each asking in the same sweep. The route answers `200` whenever it exists, so a `404` for the whole request can only mean an origin too old to have it. `gone` is per room: not found, the asker is not a peer, not hosted here, an invalid ref, or voice disabled. A room may be left out of the reply (it stops at 3 MiB, or the room's viewers could not be read) and is simply asked about again. Anything else is a 5xx, and the asker keeps its states. A pull, a join's reply and a snapshot all go through one function on the mirror (`applyOriginAnswer`): a state written after the question went out stands, a user a left relay has removed since stays removed, and it only ever changes the local copy. It never goes through a leave, which would ask the origin to remove users it may be hosting itself. How long a stale state can last is in `VOICE.md`, "Calls across instances": under a minute while the origin answers, about five minutes when it does not. **Private rooms.** In a space, what the pull and a snapshot say depends on the asker: everyone in the room when it is open or one of the peer's users can view it, otherwise only the peer's own users. A peer with no viewer gets no relays for the room, not even lefts, since a left is occupancy too; only its own users' states still reach it. So a peer that loses its last viewer in the middle of a call has everyone else's states cleared by its next pull, within about a minute, and its own users still in the call stay listed. A peer that gains its first viewer in the middle of a call hears of it at the next relay for that room or the next resync, up to 30 minutes later. **Running calls.** Relays only carry changes, so a snapshot (`/spaces/join`, `/spaces/join_listed`, `/spaces/sync`) carries `voice_rooms: [{room, states}]`, optional, filtered per peer as above and capped at 1 MiB. It is absent when nobody is in voice. A mirror only adds what a snapshot says; removal is left to the pull. Ending a room ends its call: deleting a room or a space, an instance administrator's takedown and a mirror being dropped all publish a left for each state, on the origin and on mirrors, and the origin removes the LiveKit participants. Removing a member of a mirrored space clears their copy at once instead of waiting for the relay. ### Spaces across instances A space is hosted by the instance that created it, its **origin**, which stays the one authority for membership, roles, permissions, channels, bans and settings - the Discord-shaped model, chosen over Matrix-style distributed state because every check a space needs (role hierarchy, channel overrides, slowmode, bans) already runs on one server here and would otherwise have to be resolved between servers that disagree. Every other instance with a member keeps a **mirror**: its own `spaces` row, room rows, roles, overrides and member rows under local ids, so READY, local permission checks, unread state, search and the gateway work unchanged for its users. `federated_spaces` (and the room mapping tables for each channel) tie the mirror to the origin; `federated_space_peers` is kept by the origin: which instances mirror each space. Role ids are the origin's on every instance (they only need to be unique within the space), room and space ids are local and mapped, members are shadow users, and - unlike a PM - a channel's messages keep the origin's ids everywhere. - **Joining.** An invite for a space on another instance is `code@domain`. The joining instance asks the origin to redeem it (`POST /spaces/join`) on behalf of its user; the origin applies the same checks a local join gets (invite validity, bans), adds the user as a shadow member, and answers with the whole space - settings, roles, channels with overrides, custom emoji, the other instances in the space, any calls already running, and the first page of members with profiles plus a cursor for the rest (`POST /spaces/members/list`, `FEDERATION_MEMBER_PAGE` members per page, 1000 by default, up to 100,000 members). The joiner builds the mirror from that and is a member from then on. A second user of the same instance just gets added to the existing mirror. Only the origin admits anyone: a mirror's space widget carries no `instant_invite`, and a code stored for a mirror (its widget used to mint them) answers `404` and goes at the next resync. - **Writes go to the origin.** A member on a mirror sending a message, editing or deleting one, reacting, leaving, or minting an invite has their instance ask the origin (`/spaces/messages`, `/spaces/reactions`, `/spaces/leave`, `/spaces/invites`). The origin runs its usual checks - channel permissions, slowmode, @everyone gating, message ownership for edits, bans - stores the result, relays it to every *other* mirror, and answers; the asking instance applies the answer itself (it is skipped in that fan-out so nothing arrives twice). A system message the request caused (a pin, a join, a leave) is not in that answer, so it is relayed to the asking instance as well, and receivers are idempotent on the origin's id. On a first join the mirror may not exist yet: it answers `404`, the relay is dropped, and the notice arrives with history. The origin's refusal reaches the member as a local refusal would: `/spaces/messages`, `/spaces/reactions` and `/spaces/pins` answer with its status and reason, plus the optional fields `code`, `rule`, `requirement`, `level` and `retry_after` (and a `Retry-After` header on slowmode). A 4xx is the origin's decision. A 5xx or no answer decided nothing and counts as the origin being unreachable, which is `502` on a write. Typing goes to the origin the same way and is passed on; the origin judges the typer by View and answers `403` to someone who lacks it. A mirror never sends `/rooms/messages` relays for a channel, edits and deletes included: they go through `/spaces/messages`, and the origin's relays carry them to everyone else. - **Reads come from the origin.** A mirror only holds what was relayed since one of its users joined, so a channel's history (`GET /rooms/:id/messages` with any of the `before`, `after` and `around` cursors - paging, jump-to-message, reading downward out of a jumped-to window) is read from the origin, as the asking member may see it, and kept locally as it is read. A refusal stands, because a copy must not show what the origin just said no to; only when the origin cannot be reached is the local copy served. `GET /rooms/:id/messages/:msg_id` follows the same rule, at one S2S call per GET, and answers `502` with no copy and no origin. Search works the same way: a channel or space-wide search on a mirror runs on the origin (`POST /spaces/messages/search`, the member's channel permissions and the E2EE skip applied there), so it covers history from before the mirror existed. Only a relayed message must name a sender still in the room; history, pins, search hits and a single get keep the messages of people who have left. In a channel a message reference's id is used as it stands, stored here or not: a reply can reach a mirror before its parent, and a pin can name a message older than the mirror. Edits and reactions on a message not stored here still answer `404`. The member list, roles and channel structure are read locally - they are complete. - **Changes flow down.** Settings, icon/banner, role create/update/delete and order, member roles, channel create/update/delete and reorder, overrides, kicks, bans and the space's deletion are pushed by the origin to every mirror, which applies them and emits the same gateway events its own clients would see for a local change. A kicked, banned or departing member's instance drops the mirror once no local member is left; a banned user's rejoin is refused by the origin. Settings on the wire (`SpaceInfo`) gained `automod_flags`, `automod_mention_limit`, `birthday_room` (the origin's room id, `""` clears it) and `birthday_message`: absent means unchanged, while zero and `""` are real values. Order matters to a mirror working out who can see what, so a moved room's upsert goes before the reorder, a deleted Section's children are upserted before its delete, and a deleted room's threads are deleted before it. A message that unarchives a thread sends a `/spaces/rooms` upsert to every mirror, and a new thread's upsert goes before its `thread_created` notice. An instance administrator's takedown is relayed like the owner's delete; it runs only on the origin, and a mirror is refused one. - **Management goes to the origin too.** A member on a mirror who may manage the space (by the mirrored roles, which are the origin's) changes settings, icon and banner, roles and members' roles, channels, their order and overrides, custom emoji, invites, kicks, bans and ownership, moderates people in voice, or deletes the space, through one call: `POST /spaces/manage {op, params}`. The origin runs the operation as that member with every check a local member gets (permissions, role hierarchy, owner-only rules, the typed name on delete), and answers with the operation's result **plus the relays the operation produced for the asking instance** - captured instead of sent. The mirror applies those exactly as it applies relays that arrive on their own, so after the call its state is what every other mirror has and the REST reply is served from it. Lists the mirror does not hold (bans, invites, the audit log) are read the same way, with users mapped to local ids and invite codes returned as `code@origin`. `space.patch` also takes `verification_level`, `automod_flags`, `automod_mention_limit`, `birthday_room` and `birthday_message`; an older origin ignores them, so a patch of only these gets its `400` "no fields to update". - **Who sees a room.** A mirror never asks the origin whether one of its members may see a room. It runs the origin's own rule (`Snapshot.roomPermissions`) over its copy of the roles, overrides and members: the owner and Administrators see everything, anyone else gets the override stack on the room's effective overrides (a synced room is judged by its Section, a thread by its parent room), and a private thread also needs membership or Manage Threads. Everything that rule reads therefore has to be on the wire. `permissions_synced` on a room is optional; a current origin always sends it in `/spaces/rooms` and snapshots, `false` included so that an unsync reaches mirrors, and when an older origin leaves it out the mirror's value stands. A mirror writes first, drops its permission snapshot, and only then tells its clients, and it says nothing about what did not change. A new room or an update goes space-wide only when `ViewersOf` says everyone, meaning @everyone can view it and no role or user override denies View; otherwise it goes to the viewers one by one. Where visibility may have changed, members who can view get `SPACE_ROOM_CREATE` and the rest `SPACE_ROOM_DELETE`, which also ends their gateway subscription to the room. A role given or taken, a role deleted, or a change to a role's View, Administrator or Manage Threads bit is re-evaluated the same way. A resync does this in two phases, everything written silently and then announced once, which is what heals a mirror on its first pass after an upgrade. - **What reaches a peer.** In a space hosted here, `POST` and `PATCH /rooms/messages`, `/rooms/reactions[/delete]`, `/rooms/pins`, `/rooms/typing`, `/rooms/voice/state` and `/rooms/voice/call` go only to instances with a member who can view the room (`channelPeers`). It fails closed: if the viewers cannot be read, nothing is relayed. A peer that gains a viewer later loses nothing, since history is read from the origin. `POST /rooms/messages/delete` still goes to every peer, because it carries no content and a peer may hold a copy from when it had a viewer; a peer that never stored the message writes nothing. What this does not cover: room names and overrides reach every mirror, through snapshots and `/spaces/rooms`, whether or not it has a viewer. The mirror hides them from its clients, but it holds them. - **Birthdays.** A date of birth stays on the member's home instance, which decides when the day has come. On a mirrored space it then asks the origin to post the greeting (`member.birthday`, no parameters), so it lands in the room everyone reads rather than in a local copy. The date is never sent. The origin limits it to one per member, space and day and otherwise takes the home instance's word for it. An origin that predates the op answers `400` "unknown action", which the mirror logs. - **Role order.** A custom role's position is its rank, as on Discord, and the origin keeps positions unique and at least 1 (@everyone is 0). A new role goes in at position 1, just above @everyone (a bot's managed role on install too), and the roles above it that would collide move up one. One role moved with `role.update {position}` (the single-role REST route) swaps: it takes that slot, and the role holding it takes the moved role's old position; a free position is written as asked. That keeps the clients that reorder with one such request per role, reusing the positions the roles already hold, ending in the order they meant, whatever order the requests arrive in. (While a space still has two roles at one position, from before positions were kept unique, a move inserts and pushes up instead, as `role.positions` does.) A mirror moves several roles at once with `role.positions` (`{positions: [{role, position}]}`, 1 to 250 entries, Discord's Modify Guild Role Positions): each listed role takes the slot it asks for and the roles above it that would collide move up one. The origin runs the whole list as one request. Unless the asking member is the owner, every listed role and every requested position must be below their highest role; if any is not, nothing is written. The answer is every role of the space, in the wire form `/spaces/roles` carries. Each role that moved, the ones pushed up included, is relayed in the order the origin wrote them, which is how every mirror learns the new order. The mirror refuses a list naming @everyone, or a role it does not have, before asking, with the answer the origin's own REST API gives. An origin that predates `role.positions` answers "unknown action". The mirror then sends one `role.update {position}` per move, lowest requested position first. Such an origin writes each position as given, so the lists clients send (the moved roles' own positions, reassigned among them) end in the requested order once the last one is in. They are separate writes, though: if one is refused part-way, the moves before it stay applied. - **Member roles.** Whether a member's role can be given or taken away depends on the role alone, as on Discord. A role below the actor's highest (for the owner, any role) can be given to anyone, or taken from anyone: the space owner and members who outrank the actor included. Roles at or above the actor's own that the member holds are kept as they are, and a bot's managed role is never given to anyone else or taken from its bot. Kicking, banning, renaming a member and resetting their space profile still need the member to rank below the actor. A mirror sends the single-role routes (`PUT`/`DELETE /spaces/:id/members/:userId/roles/:roleId`) as `member.role` (`{user, role, add}`: the member's FID, the role's id, `true` to give it and `false` to take it away; `add` is required). The origin applies it with the routes' own rules, as one set operation on the member's roles, so two changes racing each other both land, where a whole list read on the mirror and written back could undo one. A member who already holds the role (or, removing, does not) is success with nothing written or relayed. Otherwise the member relay is in the reply. An origin that predates `member.role` answers "unknown action", and the mirror then sends the member's whole new list as `member.roles`, which every origin understands, built from its own copy of the member. An origin that predates this section still refuses changes to the owner's roles by anyone but the owner, and to the roles of a member at or above the actor. - **Space profiles.** A member's per-space nickname, avatar, banner, About Me, bio and pronouns live on their member row at the origin and travel with it: the snapshot, the member pages and every member relay carry all six as full state (a missing field means no override), and a mirror clamps what it applies to the origin's limits (nickname 32 characters with control and bidirectional-override characters removed, About Me and bio 190, pronouns 40, images http(s) only). A member on a mirror changes their own space profile, and a moderator with Manage Nicknames changes someone's nickname or clears their space profile, through `member.profile` (`{user, nick?, avatar?, banner?, about_me?, bio?, pronouns?}`: a missing key is unchanged, `""` clears). The origin runs the same checks as for a local member - Change Nickname for your own nickname, Manage Nicknames and a higher role for anyone else's; a member's other profile fields can be set only by that member, and a moderator may only clear them - writes the audit entry for a moderator's change, and relays the member. A space profile's images stay on the member's home instance's CDN; an instance only ever deletes images under its own `spaces/members/` keys. An origin that predates space profiles answers `member.profile` with "unknown action", and clients say the space's instance does not support them yet. - **Bots.** A bot is installed into a federated space by a member whose account lives on the bot's instance: they approve the bot's consent screen there (the mirrored space is listed as a target, marked with where it is hosted), and that instance asks the origin (`bot.install`) to add the bot as that member, with the member's own grantable permissions. The origin creates the bot's shadow, runs the install exactly as a local one (Manage Space or Administrator required, the managed role at position 1, the audit entry) and relays the member and the role to every mirror; on the bot's own instance the member row is the bot itself, so its gateway connection gets the space in READY and every relayed message in it, and what it posts goes to the origin like any member's. Re-authorising re-applies permissions, kicking the bot (from any instance with the right to) removes it and its role everywhere, and an app's `spaces.join` (`member.add`) adds one of its instance's users through the origin the same way. No instance can install another instance's bot or add another instance's users; a bot cannot be added to a space hosted elsewhere by someone from a third instance, because the consent happens where both the person and the bot live. - **Custom emoji** are part of the snapshot and relayed on create/rename/delete, keeping the origin's ids, so `<:name:id>` renders on every instance and the picker shows them; an emoji uploaded from a mirror is stored on that instance's CDN and recorded by the origin with its URL. - **Resync.** A mirror asks its origin for a fresh snapshot (`POST /spaces/sync`) soon after the API starts and every half hour, and reconciles settings, roles, channels, members (fetched page by page; if a page cannot be fetched the reconcile adds members but removes none), emoji and the list of instances against it - the catch-up for anything that went wrong while the instance was down, on top of the outbox replaying the relays themselves. A member can ask for one any time (`POST /spaces/:id/resync` on their own instance). A space the origin no longer has is dropped: a `404`, or a `403` carrying the peer refusal (`ErrPeerNotAllowed`). The origin looks for the space before it checks the peer, so a deleted space always reads as `404`. `RequireInstance`'s "instance not allowed" is a policy on this whole instance and may be lifted, so a mirror survives it. - **Presence and profiles.** The origin keeps the list of instances in each space and hands it to every mirror - in the snapshot, then `POST /spaces/peers` as instances come and go - and a mirror stores it under its own space id. A member's home instance therefore sends their presence and profile changes to every instance in the space itself, the origin and the other mirrors alike, and no instance ever relays what another instance's user is doing: `POST /users/presence` and `/users/update` keep the rule that a peer speaks only for its own users with any number of instances, so a space's origin cannot make one of its members look different elsewhere. A joining member's status travels with the join, so the member-add every mirror gets already carries it, and the snapshot carries every member's; what a member is playing comes only from their home instance's `POST /users/presence`. - **Voice rooms** of a federated space run on the origin's LiveKit exactly like a call in a federated PM (the origin mints the token, mirrors keep the states; see "Calls across instances"). Moderating from a mirror happens at the origin, ahead of anything read locally, because a mirror's copy may be stale and only the origin's LiveKit can act: `voice.moderate` (`{user, mute?, deaf?, room?}`, `room` in the origin's ids) and `voice.disconnect` (`{user}`) run as the actor's shadow, so the permission checks and the audit entry are the origin's, and the relay that follows is what updates the mirror. `PATCH` and `DELETE /spaces/:id/members/:userId/voice` on a mirror return the origin's status and message, or `502` when it cannot be reached. - **Clients** see a `federation: {origin_domain, origin_id}` field on a mirrored space and on each of its rooms, show "hosted on domain" from it, render remote members' handles with their domain, and accept `code@domain` wherever an invite code goes (invite page, pasted links, the "have an invite?" box). Managing a mirrored space needs no client changes: the same settings pages work, with the origin's answer. An invite link opened on an instance where the visitor has no account offers "continue on your instance": they name the instance their account is on (remembered for next time) and are sent to `/invite/@` there, where they are logged in - so a plain link to a space on one instance works for someone whose account is on another. What the origin does not do: it never trusts a mirror's permission decisions (a mirror's local checks are a convenience), never lets one instance act for another's users, and never accepts a mirror's claims about structure. What a mirror cannot do: read pre-join history while the origin is down. ### End-to-end encryption across instances The client builds the crypto engine's ids from the identity registry (`web.strafe.chat/src/stores/federationIds.ts`), populated from the `home_domain` / `origin_id` fields every participant now carries and the `federation` field on rooms. On the server, `/devices/keys/query`, `/keys/claim` and `/send_to_device` split requests by the domain in each `@id:domain` key: local users are answered locally, the rest is forwarded (signed) to their home instance, which serves only its own users. The forwarded request carries the asking user (`requester`), and the home instance answers for each of its users only when the two share a room (or are friends, neither blocking the other), exactly as it would for one of its own clients - a peer cannot enumerate device lists or drain one-time-key pools of users its people have nothing to do with, and an instance running an older version, which sends no `requester`, is refused with `403`. Inbound to-device traffic is gated the same way, by the sender it names. Inbound to-device traffic is stored for the local recipient with the sender's shadow id and pushed with `sender_fid` so the recipient's engine keys the Olm session by the right name. Consequence for existing installs: enabling federation renames local users from `@id:strafe.internal` to `@id:`. The crypto engine refuses to reuse an account under a different name, so the client keys its local store and device id by the federated identity (`@`) and, the first time a browser connects afterwards, retires the pre-federation device: its store is deleted, the device revoked, and a fresh one provisioned (`lib/e2ee/machine.ts`, `migrateLegacyIdentity`). A recovery backup that still names the retired device is ignored rather than restored. Peers re-share room keys on their next message; history encrypted before that stays readable only where the old sessions still exist. Do this once, before you have users who care. To-device messages record the sender's federated id (`sender_fid`, migration 022) so the REST poll and the gateway push agree on who an Olm session belongs to. ### Attachments and emoji Attachment URLs are absolute (the nebula of the sender's instance), so they render on any instance. Encrypted attachments are fetched with `fetch()` and decrypted client-side, which needs the CDN to allow cross-origin reads: nebula's `CORS_ORIGINS` must be empty (any origin) or include the other instances. A member writing into a space hosted elsewhere uploads to their own instance, which claims the file when the origin accepts the message (so the orphan sweep leaves it), keeps it when the origin's copy of the message comes back, and deletes it with the message. Custom emoji are referenced by id (`<:name:id>` in text, `custom:` as a reaction) and the client resolves an id it does not know through `GET /emojis/:id` on its own instance. So that this works for an emoji from a space the reader's instance has never mirrored, every message, edit and reaction that crosses instances carries `emojis: [{id, name, url, animated}]` describing the custom emoji it uses, and the receiving instance records the ones it does not know (under "no space of ours"); an emoji it already has - its own or a mirrored space's - is never overwritten by a peer's description. ### Mentions in text Message text is written with instance-local ids (`<@123>`), which mean nothing elsewhere: the same person has a different shadow id on every instance. Text is translated at the boundary - mentions go out as `<@@id:domain>` (the person's federated id) and are rewritten to the receiver's local id on the way in, fetching a shadow for anyone not yet known. This applies to every path text takes (relays, a mirror's writes to the origin and the origin's answers, history and search pages), so a mention typed on one instance highlights the right person on all of them. Role mentions already use the origin's role ids everywhere. In end-to-end-encrypted rooms the server never sees the text; there the declared `mentions` list (federated ids on the wire) is what crosses. A system message's `system_payload` is translated at the same boundary. In `actor_id` and `user_id` a user goes out as an FID, `thread_id` goes out as a room FID (`!id:domain`), and the receiver maps them to its local ids, fetching a shadow when it has to. `message_id` and keys it does not know stay as they are. What cannot be resolved stays as it came, which a client shows as "Someone". ### Discover across instances Every instance serves the spaces it lists at `GET /discover` and asks each peer for theirs on a 15-minute timer, keeping the last answer until the next round. The directory is pulled, not pushed: nothing has to be relayed, retried or deleted in a fan-out, a peer that is down only means a slightly stale card, and the instance that hosts a space stays the only authority on whether it is listed at all. A peer's cards sit on the same Discover page as the local ones, each badged with the instance that hosts it, and the page can be narrowed to this instance. Joining one goes to the origin (`POST /spaces/join_listed`), which checks for itself that the space really is listed and shared before adding the member - a peer holding a card from before the managers changed their mind gets a plain "not listed", not a way in. From there it is an ordinary mirror, exactly as an invite join would have built. Sharing is per listing and on by default (`federate_opt_out` on the listing); operators can turn the whole exchange off with `DISCOVER_FEDERATION=false`, which also makes this instance answer `GET /discover` with nothing and refuse every listed join. Only approved space listings are shared: a pending or declined application never leaves the instance, and bots are never shared because installing one is an OAuth flow on the instance the application lives on. What a peer stores is capped at 100 cards per instance, and every text field is clamped on arrival. ### Policy `FEDERATION_ALLOWLIST` (when non-empty, the only peers) and `FEDERATION_BLOCKLIST` apply to both directions. `GET /federation/peers` (session auth) lists instances seen so far. `FEDERATION_STATIC_PEERS` and `FEDERATION_ALLOW_INSECURE` exist for development only. `FEDERATION_MEMBER_PAGE` (default 1000, at most 1000) is how many members go in one page when another instance builds or resyncs a mirror of a space hosted here. ### Mixed versions Instances upgrade on their own schedule. Everything added to the wire is optional: an older peer ignores it, and absent means unchanged. Nothing breaks across versions, but a fix only takes effect where the instance that has to act on it is new enough. **Both sides have to be upgraded for:** | Behaviour | With an older origin | With an older mirror | | --- | --- | --- | | A room synced to a private Section stays hidden (`permissions_synced`) | Never sent, so the mirror keeps its value and the room shows there | Ignored, so the room shows there. Heals on the first resync once both are up | | Voice states in a space's rooms stay correct | No relays for space rooms and no pull (`404`, asked again after an hour): the mirror knows what join replies told it, ghosts included | Relays apply, so whoever leaves is cleared, but it never pulls, so a lost left stays a ghost | | Moderating voice from a mirror (`voice.moderate`, `voice.disconnect`) | The moderator gets `400` "unknown action" | Acts on its own copy, as before: `409` when that copy does not list the target, otherwise a change the origin never hears of | | A moved member follows the move (`moved_to`) | Not sent | Ignored: the member is dropped from the call instead | | A mirror sees calls already running when it joins or resyncs (`voice_rooms`) | Not sent | Ignored | | A leave cannot end a newer join (`session_id`) | Not read | Not sent, so a leave ends whatever is there, as before | | Automod and birthday settings cross; its members get birthday greetings | Not sent, so the mirror has no birthday room and greets nobody | Ignored, with the same result, as before | | A mirror shows refusals properly | Automod and similar refusals arrive as `500`, which a current mirror reads as the origin being unreachable: `502` | Status and reason pass through; the extra fields are ignored | | Notices name people and threads (`system_payload` ids) | Plain ids stay as they came and read "Someone" | Stores the FIDs verbatim, with the same result | So a new mirror behind an older origin gets none of these fixes and keeps the old behaviour. A new origin's relays do clear people who leave from an older mirror's copy (only a lost left stays a ghost there), but that mirror still shows rooms synced to a private Section. **The origin alone is enough for** narrowing relays to peers with a viewer, voice relays for space rooms (older mirrors already accept them), ending a call when its room or space goes, the relay order of moved and deleted rooms, and `404` for a deleted space. **The mirror alone is enough for** scoping its own gateway events to viewers, letting an origin's refusal stand rather than serving the local copy, role-mention badges, keeping uploads made from a mirror, and treating a peer refusal as "this space is gone". ## Not covered (yet) - **Spaces, the rest of it**: a mirror that cannot reach the origin serves only the history it has seen; when the last local member leaves, the mirror and its local copy of the history are dropped; a mirror holds at most 100,000 members. A peer that lost its last viewer of a room keeps the messages it stored, and edits and reaction changes stop reaching that copy; only deletes still arrive. A former member from a third instance that has no shadow on the origin, and whose home cannot be reached, is dropped from a history page. - **Read receipts** are not relayed, and presence reaches remote friends and instances sharing a space only (not people who merely share a PM). - **Several API replicas** would each deliver the same outbox entry (every receiver is idempotent for messages, rooms and reactions, so nothing breaks, but a per-entry claim belongs in front of that step before the API is scaled out). - **Media proxying**: attachments are loaded directly from the origin CDN, which exposes the reader's IP to that instance. A caching proxy on the reader's instance would fix that. - **Key gating** on the S2S key endpoints is by signature only (any allowed instance can query/claim keys for your users), matching what Matrix does; a stricter "must share a room" check is possible with the mapping tables. - **Account migration** between instances (Mastodon's "move"), **moderating a single remote account** instance-wide (today only a whole peer can be blocked), **forwarding a report** to the reported user's home instance, and **per-space federation policy** (a space that admits members from this instance only, or from a list of instances) are not built; they are the next layer, see the project tracker. ## Files - `equinox/internal/federation/` - the engine: identity parsing, signing key, discovery, signed client, request-verifying middleware, mapping repository, outbound relays (`outbox.go`: the durable per-peer queue), inbound handlers; `spaces_peers.go` holds the instance list of a space and the paged member sync. - `equinox/internal/modules/{rooms,messages,devices,users,relationships,spaces,voice}` - small `Federator`/`KeyRouter` hook interfaces the engine implements; nil when federation is off. `spaces/federation.go` holds the mirror side (building and updating a mirror), `federation/spaces.go` the wire types and both directions of the space protocol; `voice/mirror.go` holds a mirror's copy of a call (the pull and everything that applies an origin's answer), `federation/voice.go` its wire types and relays. - `equinox/migrations/021_federation.cql` - shadow-user columns and mapping tables; `035_federated_spaces.cql` - space mappings and the space peer list; `036_federation_outbox.cql` - the relay outbox and the peer failure marks. - `web.strafe.chat/src/stores/{instance,federationIds}.ts`, `src/lib/e2ee/constants.ts` - client-side identity. - `deploy/` - production compose, Caddyfile, env template, runbook.