# Digarr Architecture ## Overview Single Bun process serving a Hono API backend + React SPA frontend. PostgreSQL via Drizzle ORM -- either an external server or the embedded PGlite backend (see [Database backend](#database-backend)). Frontend is a Vite SPA served by Hono in production, proxied via Vite dev server in development. ## Authentication boundary The SPA authenticates with an `HttpOnly; SameSite=Lax` session cookie and never reads the raw token. The CSRF trust origin, CORS, and the OIDC callback all derive from the public `ALLOWED_ORIGIN` when it is set; reverse-proxy deployments must therefore configure the exact external origin, and TLS termination requires an `https://` value for correct CSRF and public-URL behavior. The cookie's `Secure` flag reads the public origin protocol only (never `X-Forwarded-Proto`): in production it fails closed to `Secure` even when the backend request arrives over HTTP, so only an explicit `DIGARR_ALLOW_INSECURE_COOKIES=true` on an `http:` public origin drops it. That override is intended for a production instance served directly over plain HTTP, which is vulnerable to network interception. Unsafe `/api/v1/*` requests using cookie or proxy auth require the fixed `X-Digarr-CSRF: 1` header plus exact same-origin browser evidence. Verified bearer sessions remain supported for API compatibility and bypass the ambient-credential CSRF check. Query-token auth remains restricted to the two safe GET surfaces that need it: pipeline SSE and preview audio. Password login and registration negotiate cookie mode through `X-Digarr-Auth-Mode: cookie`; calls without it retain the bearer-token response contract. The SPA rotates an old stored bearer into a cookie through an atomic, single-use migration endpoint. OIDC and trusted-proxy auth mint cookies directly, and the OIDC callback redirects without putting the session token in the URL. OIDC login state is browser-bound in a state-scoped `HttpOnly` transaction cookie that the callback consumes: one-time, 10-minute TTL, multi-tab safe, capacity-capped, and login-rate-limited (10/min/IP on the login route; the callback is not limited). Password change and session replacement run as one database transaction under a user-row lock, so a password verified before a concurrent reset cannot mint a post-reset session. OIDC account linking reuses that callback with a server-owned transaction purpose. Initiation requires a cookie session and fresh password proof. The link callback locks the user and initiating session, rechecks the password fingerprint and session validity, and updates only the OIDC subject. It never creates an account or session; the unique subject index prevents linking one identity to two accounts. ## Database backend Digarr runs on PostgreSQL through Drizzle either way, but the backend is chosen at boot: - **External PostgreSQL** when a DSN is present -- `DATABASE_URL`, or the `DB_HOST` + `DB_USER` + `DB_NAME` triple. Uses a connection pool. - **Embedded PGlite** otherwise -- PostgreSQL compiled to Wasm, running in-process, with the whole database persisted to a single directory at `DB_PATH` (image default `/app/data`). No separate database server or container. The DB module resolves the backend once and exposes an eager Drizzle singleton. On shutdown `closeDb()` flushes the PGlite data to disk (and closes the pool on the external path), so the data directory is consistent across restarts. The selected backend is surfaced at `GET /health` (`"dbBackend": "pglite" | "postgres"`) and printed at startup as `[db] backend=...`. Boot interaction (see [Boot order](#boot-order)): `waitForDatabase()` only runs for the external pool (`if (pool)`) -- PGlite is in-process and always ready -- then the same `preFlightCheck()` -> `runMigrations()` path runs for both backends. **Invariants and limits.** PGlite is single-writer: the database runs in Wasm and persists in a data directory, so exactly one replica may own it. The Helm/k8s opt-in pins `replicaCount=1` and forces the `Recreate` rollout strategy (no two pods touching the file at once). Because the working set sits in Wasm memory, PGlite is a scale ceiling -- it fits digarr's small-data, single-writer profile. Switch to external PostgreSQL for a managed database or larger datasets, but keep the app at one replica: pipeline coordination, schedulers, rate limits, and migration locks remain process-local, so a DSN alone does not make horizontal scaling safe. **Per-platform defaults.** The container image and the Unraid template default to embedded PGlite (bare `docker run` with no DB env, or `deploy/docker/docker-compose.pglite.yml`). The default `deploy/docker/docker-compose.yml`, the Helm chart, and the raw k8s manifests default to external PostgreSQL; PGlite is opt-in there (Helm `--set database.backend=pglite`, which requires a PVC plus `replicaCount=1` and `Recreate`). **In-app backend migration.** Admins can switch between PGlite and external PostgreSQL through Settings -> Administration -> Migrate Database Backend without stopping the server or writing SQL. The tool (`src/core/ops/migrate-backend.ts`) runs schema migrations on the target, then opens a consistent source view inside a `REPEATABLE READ READ ONLY` transaction and copies the restore registry in foreign key order inside one target transaction. Each table is selected, restored in chunks, and verified by row count and SHA-256 content hash before the next table is loaded. The process does not retain whole-source or whole-target backup objects, so its working set follows the largest individual table instead of the whole database. A write failure rolls back the target copy transaction before a `MigrationReport` is returned. During the copy, `maintenanceMiddleware` blocks all write methods (`POST/PUT/PATCH/DELETE`) on non-migration routes, returning `503 Maintenance in progress`; reads pass through. Background schedulers check the same flag (`isMaintenance()` in `src/core/ops/maintenance.ts`) and skip their ticks while it is set. The routes are `POST /api/v1/admin/migrate-backend/test` (validate target, non-destructive) and `POST /api/v1/admin/migrate-backend` (run copy). See [`docs/guides/switching-backends.md`](guides/switching-backends.md) for the operator walkthrough. ## Dashboard listening history Listening routes preserve configuration, empty-result, and failure outcomes separately from their returned entries. A successful fallback with entries wins; without entries, any attempted source failure produces an error outcome. ListenBrainz artist statistics map HTTP 204 to an empty result locally, while the shared JSON transport continues to reject missing response bodies elsewhere. The dashboard distinguishes loading, unconfigured, empty, and failed history and retries failed queries on request. Failed refreshes keep cached entries visible with a failure notice. ## Pipeline Seven stages: 1. **Collect** - gather seed artists from the user's library and listening history 2. **Analyze** - extract profile features (preferred genres, eras, popularity) 3. **Discover** - ask providers (AI + similarity sources) for candidates 4. **Resolve** - canonicalize candidates to MusicBrainz IDs 5. **Score** - weighted feature scoring, clamped to [0, 1] 6. **Filter** - dedupe across batches, apply rejection cooldown, threshold 7. **Store** - persist recommendations with status = 'discovered' Pure functions live in `src/core/pipeline/`. The orchestrator (`src/core/pipeline/orchestrator.ts`) composes the stages and emits SSE progress. Analyze hydrates listening-artist genres from native source payloads, `library_artists`, and the `artists` cache before computing the taste profile. After foreground pipeline work completes, a maintenance-aware warmer queues at most 10 stale or missing artists through the shared MusicBrainz rate gate; the next scan consumes the refreshed cache through an ambiguity-checked source/name alias when the listening source has no MBID. Listening-artist genre data in the artist cache uses its own freshness timestamp, so unrelated image or metadata refreshes cannot extend the 180-day genre TTL. Enrichment from `artist_metadata` still runs between resolve and score. Discovery queries only listening sources declaring `similarArtists`. Job results distinguish unsupported capabilities, explicit discovery modes, missing seeds, successful empty lookups, and upstream failures. A seed lookup failure remains visible even when other seeds contribute candidates. These outcomes describe discovery, independently of profile analysis and library sync. Analyze deduplicates each source's artist evidence before normalizing its positive weights to a total of one. Spotify contributes reciprocal best position across the existing personal top-artist windows; this is an ordinal estimate, not a published affinity score. Subsonic starred artists contribute equal membership evidence. Other adapters retain their source-local numeric signals, including favorite boosts or collection counts. Raw values remain separate from normalized `tasteWeight` values. Artists appearing in several sources keep their maximum contribution rather than summing overlapping history. Aggregate genres and the analyzed profile use these relative weights with deterministic ties. Before similarity lookup, discovery shuffles exact positive finite taste-weight ties on a copied list; unequal weights and legacy ordering remain intact. Library mixing deduplicates known identities, preserves a uniquely matching catalog ID on copied seeds, and backfills unavailable slots from remaining listening artists without exceeding the configured cap. This changes seed opportunity, not genre quotas or scoring. Empty, invalid, or zero numeric evidence contributes no positive weight; missing genre tags stay unknown. Recommendation prompts retain per-artist genre context for at most 20 seeds and eight genre tags per seed. Raw seed values remain source-dependent, not comparable play counts; relative taste weights are neither probabilities nor evidence of a single dominant taste. Guidance preserves distinct evidenced interests without recommendation quotas. A sparse source can give its single artist a strong relative weight, and bounded seed selection cannot guarantee representation of every interest. Scoring, stored recommendations, and source history windows are unchanged. AI discovery retains comparisons to listening-profile artists. Its description guard only checks likely shared-name collisions: an unquoted seed name without the recommended name can be rejected. Prompts ask for the exact recommended name in the first sentence. This heuristic cannot establish artist identity or factual accuracy; MusicBrainz resolution remains a separate stage. Name-only resolution checks at most five MusicBrainz hits against the requested name or returned catalog aliases before comparing genres. Name normalization preserves accents and punctuation. A uniquely best matching identity can resolve; ties and unrelated hits are dropped. Known-MBID candidates retain their explicit identity path. Older stored recommendations are not rewritten. The filter stage partitions candidates by `kind`. Artist-kind candidates run the full artist-existence / library / top-artist filters. Album-kind candidates bypass those artist-oriented filters (a new release from a tracked artist is the point, not a rejection cause) but still pass the album block layer, cross-batch dedup, and the score threshold. ## Registry patterns Seven extension points, each registry-based: - `DestinationTarget` - where recommendations are pushed (Lidarr, Emby, `slskd`, ...) - `SubscriptionAdapter` - how recurring seeds are sourced (CSV, Spotify saved, ...) - `SearchSource` - multi-source artist / track search (Lidarr, MusicBrainz, Deezer, ...) - `RecommendationProvider` - AI backends (Anthropic, OpenAI, Gemini, Ollama, ...) - `DiscoveryMode` - on-demand / savable discovery flows, registered in `src/core/discovery-modes/registry.ts` (ListenBrainz radio, Release Radar, Library Gap-Fill, Charts, Deezer Flow, Spotify Saved Albums, TIDAL Favorite Artists, ...). A new mode is a factory plus a `registry.register` line plus an availability entry; the frontend renders modes generically, so no frontend change is needed. Modes that just read a user's artist collection from an OAuth-connected provider are one `createUserArtistCollectionMode({ id, label, description, provider, fetchArtists })` spec (`modes/user-artist-collection.ts`), and modes gated on a single connection flag are one row in `SINGLE_FLAG_MODES` in `availability.ts` rather than a hand-written branch. An optional `stability: 'experimental'` on the definition (serialized by `GET /api/v1/discovery-modes`, defaulting to `stable`) badges the mode card without a per-mode frontend branch. TIDAL Favorite Artists remains experimental while live-account connect, refresh, and populated collection-result validation is deferred; see [TIDAL feedback](../README.md#tidal-feedback). - `NotificationChannel` - where notifications are delivered (webhook, ntfy, Telegram, Apprise), in `src/core/notifications/`. `registry.ts` fans one event out to every enabled, subscribed channel via `Promise.allSettled` (one channel down never blocks the others); each `channels/.ts` formats its payload and calls the single SSRF-guarded `transport.ts`. A new type is a `channels/.ts` module plus a union arm on `NotificationChannel`. The transport does DNS-pinned resolution, `redirect: manual`, and blocks private/link-local/cloud-metadata targets; a per-channel admin-only `allowPrivateTarget` waives only the RFC1918 set. Channel secrets are encrypted at rest and masked (`***`) through the settings API - `ProviderAuth` - how a streaming provider's stored OAuth token is resolved and refreshed, as a `PROVIDER_AUTH` map in `src/core/provider-auth.ts` keyed by `OAuthProvider`. `resolveProviderToken(db, userId, provider)` is the single entry point for Spotify, Deezer, and TIDAL; a provider without a `tokenEndpoint` (Deezer) is simply one that cannot refresh, rather than a separate code path. `authStyle` (`basic` or `body`) must match how that provider's authorization-code exchange authenticates, since a client accepts one style and not both. Failures raise `ProviderAuthError` with `reason: 'not_connected' | 'token_unusable'`, which is what lets discovery modes tell "never connected" from "token dead" instead of flattening both into one message. A new provider is one row here plus a callback handler in `src/server/routes/oauth-callbacks.ts` Adding a new implementation means: 1. Implement the interface in `src/core//adapters/.ts` 2. Register in `src/core//registry.ts` 3. Add a settings schema and UI when the adapter is user-configurable ## Preview playback Recommendation-card tracks and the Discover audition queue share the global preview context, so starting one preview stops the other audio surface. Deezer clips advance from the native `ended` event. Spotify audition playback keeps one persistent local bridge iframe with `sandbox="allow-scripts"` and no same-origin capability. Spotify's controller script and embed run only inside that opaque-origin document; the authenticated SPA transfers a private `MessageChannel` and accepts exact, token-bound playback events. The narrow protocol carries load/play/pause/destroy commands plus ready, started, state, and failure events. Playback start, pause, completion deduplication, controller reuse, and queue advancement remain in the SPA. Bridge initialization is bounded and falls back to the standard Spotify iframe for standalone previews; an active audition queue skips the unavailable item. YouTube embeds have no equivalent completion signal in this integration and use a bounded 30-second fallback. Audition retains per-item unavailable reasons after skipping or queue completion, independently of playback state. New runs reset the summary and retrying an item replaces its prior failure. Reasons describe observable lookup and playback outcomes; browser fetch rejection does not establish CORS or provider outage, and Spotify controller failure does not establish account access. Stale audio callbacks and superseded resolutions cannot advance a newer queue item. No approval writes occur during playback. ## Boot order Startup in `src/index.ts` has two phases: 1. Before the HTTP listener starts, initialize encryption, wait for external PostgreSQL if configured, run the pre-flight backup check, and apply schema migrations. Then wire the session store, library services, job recorder, and application dependencies. Startup stuck-job detection runs after migrations. 2. After the HTTP listener starts, an async initializer completes env-based setup when configured, creates the initial admin if no users exist, migrates legacy connections, backfills targets, and starts the pipeline, subscription, playlist, library, slskd, stuck-job, and notification-digest schedulers. The digest bookmark is persisted after successful delivery. This provides at-least-once delivery: a crash after sending but before saving the bookmark can repeat a digest. Schedulers skip work during maintenance; jobs already running must finish before a backend migration. ## Album-level discovery Albums are a first-class recommendation unit. Key additions: - **`kind` discriminator** on the `recommendations` table (`'artist' | 'album'`, default `'artist'`). All recommendation queries and API responses include `kind`; the list endpoint accepts a `?kind=` filter. - **`album_blocks` table** -- per-user, forever-block layer for albums, keyed on release-group MBID. Independent of `artist_blocks`; the filter stage drops candidates matching either block layer. - **`applyAlbumModifier`** in `src/core/pipeline/score.ts` -- computes a bounded recency / popularity / gap-priority modifier added to the artist-similarity base score, then clamps the result to `[0, 1]`. - **`addAlbum` target capability** -- approving an album recommendation calls the Lidarr target's `addAlbum` method: adds the artist unmonitored (no whole-discography grab) and monitors + searches only the approved album. If the artist already exists in Lidarr, the existing record is reused (gap-fill safe). - **Release-radar producer** -- the release-radar discovery mode is the first producer that populates the album substrate. It emits first-class `kind='album'` recommendations for new releases from artists the user already tracks, instead of collapsing them into artist rows, and these land in the Albums tab. With the kind-aware dedup change (below), a tracked artist that drops several releases in one scan window now yields one album recommendation per release in the same run, rather than one per run. - **Library gap-fill producer** -- a discovery mode whose executor iterates a rotated, bounded slice of the user's tracked artists. The cursor is the `library_artists.last_gap_check_at` column, ordered `asc nulls first` so never-checked artists go first; the slice is bounded (default 25 per run, overridable via the mode's `maxArtistsPerRun` setting) and walked with a p-queue (concurrency 2, 200ms interval) so a large library does not starve the event loop. For each artist it calls the album-coverage engine (`src/core/library/album-coverage.ts`) and emits one `kind='album'` candidate per missing studio album, carrying the release-group MBID and the release year as the recency signal. After the slice runs, the checked artists' `last_gap_check_at` is stamped so the next run advances the cursor. This fills the Albums tab from missing studio albums of artists already in the library. - **Net-new album discovery producer** -- when `netNewAlbumDiscovery` is enabled (default off), `resolve()` tries to match the AI's `suggestedAlbum` to a MusicBrainz release group. A match becomes an album recommendation with its release-group MBID and first-release date, then follows the normal album scoring, filtering, and storage paths. An unmatched title stays an artist recommendation. - **Album empty-state routing** -- a normal pipeline scan remains artist-focused. When the album-filtered recommendation list is empty, the frontend links to the two explicit album discovery modes (`gap-fill` and `release-radar`) and to the default-off `netNewAlbumDiscovery` preference. Discovery-mode deep links focus the requested generic mode card; the preference link opens its collapsed settings section and focuses the target. - **Kind-aware dedup** -- album candidates dedup and group by release-group MBID instead of artist MBID at three points, which is what lets multiple albums per artist survive a single run (and lifted the release-radar one-album-per-artist-per-run cap): the discover-stage dedup keys album candidates on `rg::{releaseGroupMbid}` while artist candidates still key on artist MBID/name (`src/core/pipeline/discover.ts`); `resolve()` partitions album-kind discoveries out of the artist-MBID grouping and groups them by release group, one resolved recommendation per release group (`src/core/pipeline/resolve.ts`); and the resolve final dedup keys album-kind recommendations on `{artistMbid}::{releaseGroupMbid}` so distinct albums for the same artist are kept. ## Key invariants - Plex listening uses a per-user server-account mapping bound to the server machine identifier. Every history page is account-filtered and checked before aggregation; missing mappings disable listening without disabling shared-library sync. Plex similarity candidates use artist metadata from that listener's history. - Audition playlists select only the owner's pending recommendations, deduplicate artists before limiting, and resolve one real track per artist. Generation does not approve recommendations or acquire missing media. - slskd linked imports retain the queued release manifest, require every expected transfer to succeed, and validate Lidarr's per-file artist, album, and track identifications before moving files. Completion requires track-file verification. Failed work keys remain unique through cooldown retries; superseded historical duplicates are preserved by migrations and backup restore. - Library sync replaces a source snapshot only after all source album fetches succeed. A failed fetch retains the previous snapshot and marks the run failed; MusicBrainz reconciliation failures remain separately counted. - Config precedence: for settings stored in the DB (single row, `id=1`), saved values override env defaults. Deployment-only options such as `DIGARR_MUSICBRAINZ_URL` and `DIGARR_MUSICBRAINZ_INTERVAL_MS` come from the environment and require a restart. Direct per-user service credentials live on `users`, with global settings as the fallback where supported; Spotify, Deezer, and TIDAL OAuth credentials live in `oauth_tokens`. - Provider, metadata, and playlist-target requests go through `createHttpClient()` in `src/core/clients/http.ts` for timeout, retry/backoff, JSON parsing, response-body errors, redaction, and optional TLS-skip behavior. Read-only calls retain the client retry default. Duplicate-producing playlist creation and song-add calls pass `retries: 0`; this classification is based on endpoint semantics because Subsonic mutations use GET-shaped endpoints. - Playlist resolution records a disposition for every selected artist: resolved, unmatched, unavailable, error, or excluded by the size cap. Returned local, Spotify and Deezer artist names must match after Unicode/case/whitespace normalization before tracks are selected. Configured fallback sources remain available; no fake playable entries fill unresolved artists. Counts distinguish selected artists, artists with resolved tracks, artists included after truncation, and included tracks. Resolution metadata is saved in the existing job record after local tracks are saved and before remote exports, so a later target failure preserves the local result. Owned playlist details expose only their latest job projection; legacy playlists have no fabricated historical summary. - Playlist generation stores its local tracks before pushing to selected enabled Navidrome, Jellyfin, Emby, Plex, and Spotify targets. A target error, including a returned failed playlist result, does not stop later selected targets; after all attempts it fails the playlist job for Job History. There is no remote rollback, and the locally generated playlist remains available. - Spotify playlist exports retain explicit track URIs, or resolve artist/title pairs with exact matching. Artist-only approvals take up to three artist-matching track search results. Writes use `/me/playlists` and `/playlists/{id}/items`, with at most 100 URIs per request; failures are not retried as duplicate writes. - Emby, Jellyfin, and Subsonic source clients each own a media-server request queue capped at three concurrent requests and ten starts per second. This is an internal load-smoothing policy for self-hosted servers, not a claimed vendor limit. It is deliberately per client instance; configurable overrides and queue metrics remain deferred until an operational need is demonstrated. - Field-level encryption uses AES-256-GCM with HKDF-derived keys (`src/core/crypto.ts`). Encrypted DB values are prefixed `enc:v1:`. Legacy SHA-256 decryption is retained as a read-path fallback for pre-migration values. - Tests run in Node.js (vitest), not Bun. `Bun.serve()`, `Bun.file()` and similar Bun-only APIs are unavailable in tests; password hashing uses `node:crypto` `scrypt`. - Migrations are idempotent. Drizzle generates bare DDL, so every generated migration must add `IF NOT EXISTS` / `IF EXISTS` clauses by hand. - Backup restore runs in a single DB transaction. Upsert conflict targets are natural keys (`mbid`, `slug`, `nameNormalized`, `token`), not generated IDs. - Primary keys are `integer GENERATED BY DEFAULT AS IDENTITY` (not legacy `serial`). BY DEFAULT is deliberate: backup restore re-inserts rows with their original `id`, which `GENERATED ALWAYS` would reject. - Backend migration never modifies the source database. Verification (row count + content hash) must pass before `ok: true` is returned; any mismatch surfaces in `MigrationReport.mismatches`. - Optional genre-priority ordering is a user-scoped read concern in `listRecommendations`, independent of score computation. Exact genre matches select primary, secondary, and other groups before score ordering and pagination. Secondary browsing excludes primary matches; missing preferences preserve score ordering. No recommendation rows are rewritten. - Scoring uses the shared `computeWeightedScore()` in `src/core/pipeline/score.ts`. All callers (main pipeline + hygiene rescorer) clamp results to `[0, 1]` regardless of user weight sums. Maintenance rescoring reuses stored components and album modifiers, scopes reads and writes to the current user, and skips incompatible evidence or rows changed since selection. - Listening profiles clean semicolon-separated genres, blank values, and numeric artifacts before hydration and after reading cached genres. Coverage counts usable genres; pending-cache counts retain their freshness semantics. This does not rewrite library or cache metadata. See `AGENTS.md` for the gotchas, external-API quirks, and CI notes that accumulate faster than this doc should; `AGENTS.md` stays the living ops file.