# Documents & Collaborative Editing Map The dual-state storage model, the Yjs WebSocket protocol, and every path that can write a document's content. This is the subsystem where "it looked right in the editor" and "it is right in the database" most easily diverge. --- ## 1. Dual-state storage A `logs` row carries **three** representations of the same document, plus one generated column: | Column | Type | Written by | Read by | |---|---|---|---| | `html_content` | `MEDIUMTEXT` | REST save, publish, restore, explicit WS save, GitHub pull | everything: rendering, export, search (via generated column), GitHub push | | `markdown_content` | `MEDIUMTEXT` | REST save when the client sends it, WS save, GitHub pull/resolve/import | GitHub push (`github.js:1035-1043`), markdown-mode editing | | `ydoc_state` | `LONGBLOB` | collab autosave and explicit save (`collab.js:126-132`) | collab session restore only (`collab.js:84-91`) | | `plain_content` | generated `STORED` | MySQL, from `html_content` (`init.sql:293`) | the FULLTEXT index | `plain_content` is `REGEXP_REPLACE(html_content, '<[^>]+>', '')`, computed by MySQL on every write of `html_content`. It is the *only* thing search matches on besides the title, which means **a document edited only over the CRDT is not searchable until an explicit save writes `html_content`.** ### The invariant, and where it breaks The intended invariant is that `html_content` and `ydoc_state` describe the same document. It is maintained by convention, not enforced. The debounced autosave is explicit about this (`collab.js:135-137`): > HTML is NOT updated here. Clients send HTML during explicit save/publish. So a 3-second-debounce autosave writes **only** the blob. Between autosave and the next explicit save, `html_content` is stale, and every consumer of `html_content` sees the older document: search, all four export formats, GitHub push, and the version snapshot taken by REST publish (`documents.js:216`, which reads `pg.html_content` straight from the row). **`markdown_content` semantics:** `undefined` leaves the column untouched, an explicit `null` clears it, a string replaces it (`documents.js:129-140`, `collab.js:573`). Rich-text-mode saves send `null` to signal "HTML is now the canonical source"; markdown-mode saves send the raw markdown. `localMarkdown()` in the GitHub path (`github.js:1035-1043`) prefers `markdown_content` when non-empty and otherwise round-trips the HTML through turndown, so a stale `markdown_content` silently wins over fresher HTML on push. ### Every writer of document content | Path | Writes | Clears `ydoc_state`? | |---|---|---| | `POST /api/save-document` (`documents.js:84`) | `html_content`, optionally `markdown_content` | no | | `POST /api/document/:logId/publish` (`documents.js:173`) | `version`, `versions` row from existing `html_content` | no | | `POST .../versions/:versionId/restore` (`documents.js:426`) | `html_content`, `version`, new `versions` row | **yes**, `NULL` (`documents.js:465`) | | WS `{type:'save'}` (`collab.js:555-623`) | `html_content` if changed, `markdown_content`, `ydoc_state` | no | | WS `{type:'publish'}` (`collab.js:654-731`) | `html_content`, `ydoc_state`, `version`, `versions` row | no | | WS `{type:'title'}` (`collab.js:629-652`) | `title` only | no | | collab autosave (`collab.js:139-149`), and the shutdown flush (`flushPendingSaves`, `collab.js:165-187`) | `ydoc_state` only | no | | GitHub pull / resolve / overwrite (`github.js:1183-1192`, `1225-1234`, `1407-1416`) | `html_content`, `markdown_content` | **yes**, `NULL` | | GitHub bulk import (`github.js:1597-1606`) | new `logs` row | n/a | | First-boot seed (`bootstrapInstance`, `routes/admin.js`) | new `logs` row, the welcome document | n/a | Clearing `ydoc_state` to `NULL` is the deliberate mechanism for "the HTML just changed underneath the CRDT": on next connect the Y.Doc starts empty and the first client's Tiptap `Collaboration` extension re-initialises it from the REST-loaded HTML (`collab.js:92-94`). Any external writer of `html_content` **must** null the blob or live editors will keep resurrecting the old content. `POST /api/save-document` notably does not, which is correct only because it is the editor's own save path. ## 2. The collab WebSocket `services/collab.js`. One `Map` of `logId -> entry` (`collab.js:45`), where an entry is `{ doc, conns, saveTimer, saving, cleanupTimer, lastSavedHtml, credits, logId }` (`collab.js:71-81`). ### Constants (`collab.js:53-59`) ``` SAVE_DEBOUNCE_MS 3000 autosave debounce after any Y.Doc mutation CLEANUP_DELAY_MS 30000 drop the in-memory doc this long after the last close MAX_MESSAGE_SIZE 5 MiB ws maxPayload MAX_HTML_SIZE 2 MiB largest HTML accepted in a save/publish message MAX_CONNECTIONS_PER_USER 10 across all documents, tracked in userConnectionCounts RATE_LIMIT_WINDOW_MS 1000 per-connection RATE_LIMIT_MAX_MESSAGES 60 per window, applies to binary and text alike ``` ### Session setup `setupDocSession` (`collab.js:441-780`) runs after auth succeeds: 1. `getOrCreateDoc(logId)` (`collab.js:67-121`) loads `ydoc_state` and applies it to a fresh `Y.Doc`, and caches `html_content` into `entry.lastSavedHtml`. 2. Cancels any pending cleanup timer (`collab.js:452-455`). 3. Assigns a cursor colour round-robin from a 10-entry palette (`collab.js:297-307`). `colorIndex` is a module global, so colours are assigned per-connection in arrival order, never per-user. 4. Sends **sync step 1** then **sync step 2** immediately (`collab.js:469-479`). Step 2 is proactive so a joining client gets content without a round trip. 5. Sends a JSON `{type:'sync', canWrite, user}` frame carrying permissions and identity. **No HTML travels over this frame**, despite the older protocol comment at `collab.js:315` saying it does. The doc arrives as binary CRDT state. 6. Broadcasts awareness to everyone. `canWrite` is computed once at connect time (`collab.js:430`) and never re-checked. Revoking someone's write access does not take effect until they reconnect. ### Message taxonomy Binary frames are Yjs sync protocol; text frames are JSON. The JSON `type` is allow-listed to exactly five values (`collab.js:526-528`): | Type | Gate | Effect | |---|---|---| | `cursor` | `canWrite` | position is validated and coerced to safe integers (`collab.js:532-543`), then broadcast to others | | `save` | `canWrite` | immediate save, see below | | `publish` | `canWrite` | permission-checked snapshot, see below | | `comment` | `canWrite` | relays only ids, never content (`collab.js:736-750`); the actual CRUD is REST | | `title` | `canWrite` | updates `logs.title`, broadcasts, and logs `log.rename` | All five are gated the same way. `title` was the exception until 2026-08-09 (B6 in `open-questions.md`): a read-only participant could rename the document, and the `log.rename` activity that followed also mailed every watcher. `comment` broadcasts are id-only by design: the receiving client refetches over REST, which keeps the WS from becoming a second authorisation surface for comment content. Binary frames are dropped outright for read-only clients (`collab.js:508`). Updates are echoed to every peer except the origin, using the sender's `ws` as the Yjs transaction origin (`collab.js:106-117`, `collab.js:225`), which is what prevents an echo loop. **Image credits.** Every client's save carries the whole shared document, so the client that sends a save is not necessarily the writer who added what it carries. `applySyncFrame` (`collab.js:217-231`) scans each binary frame's bytes for `/doc-images/.webp`, and only for a frame that names one renders the `default` fragment before and after applying it. `entry.credits`, a `DocImageCredits` from `routes/helpers/images.js` opened with the entry (`collab.js:79`, seeded with the stored HTML and any loaded CRDT state at `collab.js:101`, closed on cleanup at `collab.js:258`), credits a reference to the frame's writer only when their own frame named it, it newly appeared, the stored HTML does not already show it, and nobody holds the credit yet. Saves then record references per credited writer, never per sender (access-control.md section 3f). ### Save (`collab.js:555-623`) Cancels the debounce, encodes the CRDT state, sanitises the client HTML through `sanitizeHtml` then `extractImagesFromHtml`, and writes. It only writes `html_content` when the HTML actually changed against `entry.lastSavedHtml` (`collab.js:576`); otherwise it writes the blob alone. On an HTML change it then records the images (`recordLiveDocImages`, `collab.js:239-246`, which logs its own failure so nothing after it is skipped), and runs `processMentionsOnSave` and `logActivity('log.update')` (`collab.js:599-618`). ### Publish (`collab.js:654-731`) Loads the log's squad context, calls the shared `canPublish` (`collab.js:670`), bumps `logs.version`, writes HTML plus blob plus version, inserts the `versions` row, fires mentions and `log.publish` activity, then broadcasts `{type:'published', version, title}` to **all** connections including the publisher. Title is capped at 255 chars, notes at 5000 (`collab.js:656-657`). ### Lifecycle and teardown On close (`collab.js:753-768`): drop the connection, decrement the per-user count, rebroadcast awareness, and if this was the last connection schedule both a final save and cleanup. Cleanup after 30s destroys the `Y.Doc` and removes the map entry, but only if no one reconnected (`collab.js:251-261`). **Sign-out closes the sockets of one session (W6-CDX-6).** Each socket records the digest of the session it authenticates with (`ws.sessionId = digest`, `collab.js:431`, the SHA-256 of the token it presented) before its auth read's first await, so a sign-out that lands at any point finds it. After the read, a socket whose sign-out already closed it (`sessionClosedRecently`, `services/closed-sessions.js`, a five-minute set) or that is no longer open is closed 4004 before it registers. `closeSocketsForSessions(digests)` (`collab.js:197`) closes, with 4004 "Signed out", exactly the sockets whose digest is in the set, across every collab server, and records the digests. `POST /api/logout` calls it for the one session it deleted, and the back-channel receiver calls it for every row a logout token ended (`routes/auth.js:608`, `routes/oidc.js`, section 3 of `request-lifecycle.md`). The same user's other devices stay open, which the tests pin. The user channel has the same closer (`services/user-channel.js:88`, socket recorded at `user-channel.js:183`). Rotations (password and email changes) close the sockets of every session they end, through the same closers (`routes/auth.js:57`). **The open sockets are re-checked every minute (W6-CDX-8).** Auth is read once, on a socket's first message, so a session ended later (a deactivation, a row deleted by hand) would keep its socket. The sweep (`services/socket-sweep.js`, started at `server.js:293`) runs every 60 s, collects the digests of the authenticated sockets on both servers (`openSessionDigests`, `collab.js:196`, `user-channel.js:87`), asks `liveSessionDigests` (`socket-sweep.js:30`) which of them still hold a live session for an active user (OIDC only on an OIDC-only instance, batched 500 at a time), and closes the rest with 4004 through `closeSocketsForSessions` (`collab.js:212`). A failed pass is logged and the timer keeps running. **All of this is per-process, in-memory. Restarting the server drops every in-memory doc; state survives only through `ydoc_state`. A graceful stop (SIGTERM or SIGINT) writes every pending debounced save first (`flushPendingSaves`, `request-lifecycle.md` section 7), so `ydoc_state` is current after one; only a crash or a SIGKILL can leave it up to 3 seconds stale. This is the load-bearing reason the single-process architecture is not negotiable: a second replica would maintain a second, divergent `Y.Doc` for the same log, and the instance lock (`services/instance-lock.js`) now makes a second process on the same schema refuse to start. ### The REST side-channel `broadcastToDoc(logId, message)` (`collab.js:787-795`) lets REST handlers push arbitrary JSON to live editors without going through Yjs. Its only current callers are the GitHub pull and resolve routes, which emit `{type:'github-pulled', ...}` (`github.js:1193`, `1235`, `1417`). It returns `false` when no one has the doc open. Three read-only accessors feed the admin and presence surfaces: `getActiveDocCount` (`collab.js:800`), `getActiveUsers(logId)` (`collab.js:807`), `getAllPresence()` (`collab.js:820`). ## 3. Versions `versions` rows are snapshots of `html_content` with an optional title and release notes (`init.sql:360-375`). Four operations: - **Publish** bumps `logs.version` and inserts a row. Two entry points, REST (`documents.js:173`) and WS (`collab.js:654`), sharing `canPublish`. - **List / fetch** (`documents.js:350`, `documents.js:386`) behind read access. - **Restore** (`documents.js:426`) writes the old HTML back as a *new* version, so history is append-only and nothing is lost. It nulls `ydoc_state`. - **Delete** (`documents.js:500`) allows admin or the version's own author first, then falls back to `squad_members.can_delete_version` or squad `role = 'owner'`. The publish route also accepts `create_github_release`, `target_repo`, and `tag_name` (`documents.js:179`), tying a version to a GitHub Release through `versions.github_release_id` / `github_tag_name` / `github_target_repo`. ## 4. The editor `src/pages/Editor.jsx` (1356 lines) is the client. The Tiptap extension stack is assembled at `Editor.jsx:341-361`: `StarterKit` with `codeBlock`, `underline` and `undoRedo` disabled (`Editor.jsx:342`, undo/redo because the `Collaboration` extension supplies its own CRDT-aware history), plus `ResizableImage`, `Placeholder`, `Underline`, `TextAlign`, `Link`, the four table extensions, `CodeBlockWithLanguage` (lowlight), `DrawioBlock`, `GitHubCodeEmbed`, `GitHubIssueEmbed`, `Mention`, and `Collaboration.configure({ document: ydoc })`. Initial content is deliberately empty (`Editor.jsx:362-365`); the `Collaboration` extension populates the editor from the shared Y.Doc. Its `onUpdate` fires for **both** local and remote changes (`Editor.jsx:393-407`), which is why state updates there are deferred rather than applied inline. `useCollab(logId, onRemoteUpdate, onRemoteComment, onPublished, onRemoteTitle)` (`src/hooks/useCollab.js:31`) owns the socket: it memoises the `Y.Doc` (`useCollab.js:49`), connects to `${proto}//${host}/collab?logId=${logId}` (`useCollab.js:71-72`), sets `binaryType = 'arraybuffer'`, sends the auth frame on open (`useCollab.js:103`), queues local updates until the server's sync step completes and flushes them afterwards (`useCollab.js:83`, `useCollab.js:126-129`), and auto-reconnects on close. It exposes `sendCursor`, `sendSave`, `sendPublish`, `sendTitle`, `sendCommentEvent`. `sendPublish` waits for the `published` frame with a 10-second timeout (`useCollab.js:258`). Mobile visitors are redirected away from `/editor/:logId` entirely by `MobileEditorGuard` (`src/App.jsx:43-47`); on mobile, documents are view-only through the archive view. --- ## Related - [access-control.md](access-control.md) for what `canWrite` at `collab.js:430` actually resolves. - [notifications-and-activity.md](notifications-and-activity.md) for `processMentionsOnSave` and `logActivity`, called from every save path here. - [github-integration.md](github-integration.md) for the pull/push paths that write `html_content` from outside the editor.